2024-09-16T03:28:28.382 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-16T03:28:28.399 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-16T03:28:28.482 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7905897 branch: squid description: rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} thrashers/one workloads/rados_api_tests} email: ceph-qa@ceph.com first_in_suite: false job_id: '7905897' kernel: kdb: 1 sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 3 size: 10 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: squid ceph: conf: client: debug monc: 20 debug ms: 1 global: debug ms: 1 debug objecter: 20 debug rados: 20 mon client directed command retry: 5 mon election default strategy: 1 ms bind msgr1: false ms bind msgr2: true ms inject socket failures: 5000 ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 25 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub inject crc mismatch: 0.01 mon scrub inject missing keys: 0.05 mon scrub interval: 300 mon warn on pool no app: false paxos service trim min: 5 osd: bluestore block size: 96636764160 bluestore compression algorithm: lz4 bluestore compression mode: aggressive bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd class default list: '*' osd class load list: '*' osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - ScrubResult - scrub mismatch - overall HEALTH_ - \(MGR_DOWN\) - \(PG_AVAILABILITY\) - \(SLOW_OPS\) - slow request - \(OSD_SLOW_PING_TIME - overall HEALTH_ - \(MON_DOWN\) - \(POOL_APP_NOT_ENABLED\) - reached quota - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_FULL\) - \(SLOW_OPS\) - \(MON_DOWN\) - \(PG_ - \(SMALLER_PGP_NUM\) - slow request sha1: 8921c8ef947807b717db041d85f0761b7914d5fc ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 8921c8ef947807b717db041d85f0761b7914d5fc thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: squid sha1: ad35db06e8924912f01e32b35576297c0f1352b4 owner: scheduled_teuthology@teuthology priority: 700 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mon.c - osd.0 - osd.1 - osd.2 - - mon.b - mgr.x - osd.3 - osd.4 - osd.5 - client.0 seed: 5616 sha1: 8921c8ef947807b717db041d85f0761b7914d5fc sleep_before_teardown: 0 subset: 15927/100000 suite: rados suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_ad35db06e8924912f01e32b35576297c0f1352b4/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: ad35db06e8924912f01e32b35576297c0f1352b4 targets: smithi042.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBXDiWN6PE9DDlOAE2tzOYpWt1M0jEUAWTMW+5GD7AygeyBTve7KeeVgVAiORSfXTBXVrAwHZS2HPCavrwkjJ+0= smithi154.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGWwvZ/6aduK1wMTWLfwMlqnrSonkUJTL7pZ4qzv27JiPXURKoUx+YOgY96cMRLv5YgRRJg+qgSaAicMAKckoXs= tasks: - install: null - ceph: null - mon_thrash: revive_delay: 20 thrash_delay: 1 - workunit: clients: client.0: - rados/test.sh unit_test_scan teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-15_21:00:02 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-16T03:28:28.482 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_ad35db06e8924912f01e32b35576297c0f1352b4/qa; will attempt to use it 2024-09-16T03:28:28.483 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_ad35db06e8924912f01e32b35576297c0f1352b4/qa/tasks 2024-09-16T03:28:28.483 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-16T03:28:28.485 INFO:teuthology.task.internal:Checking packages... 2024-09-16T03:28:28.505 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '8921c8ef947807b717db041d85f0761b7914d5fc' 2024-09-16T03:28:28.505 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-16T03:28:28.505 INFO:teuthology.packaging:ref: None 2024-09-16T03:28:28.505 INFO:teuthology.packaging:tag: None 2024-09-16T03:28:28.505 INFO:teuthology.packaging:branch: squid 2024-09-16T03:28:28.505 INFO:teuthology.packaging:sha1: 8921c8ef947807b717db041d85f0761b7914d5fc 2024-09-16T03:28:28.505 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=squid 2024-09-16T03:28:28.696 INFO:teuthology.task.internal:Found packages for ceph version 19.1.1-326-g8921c8ef-1jammy 2024-09-16T03:28:28.698 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-16T03:28:28.705 INFO:teuthology.task.internal:no buildpackages task found 2024-09-16T03:28:28.706 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-16T03:28:28.749 INFO:teuthology.task.internal:Saving configuration 2024-09-16T03:28:28.763 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-16T03:28:28.783 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-16T03:28:28.810 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi042.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7905897', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-16 03:22:33.923976', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIK6oIPu9rgit7gcKwC52hisWUROpy4u/s3Uchx5btp9N'} 2024-09-16T03:28:28.833 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi154.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7905897', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-16 03:22:33.922198', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDoK07BgJclwZC4sV3wp/7ZABe6EJkVsFn8L0ffqpR2b'} 2024-09-16T03:28:28.833 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-16T03:28:28.844 INFO:teuthology.task.internal:roles: ubuntu@smithi042.front.sepia.ceph.com - ['mon.a', 'mon.c', 'osd.0', 'osd.1', 'osd.2'] 2024-09-16T03:28:28.844 INFO:teuthology.task.internal:roles: ubuntu@smithi154.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.3', 'osd.4', 'osd.5', 'client.0'] 2024-09-16T03:28:28.844 INFO:teuthology.run_tasks:Running task console_log... 2024-09-16T03:28:28.929 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f77147cae60>, signals=[15]) 2024-09-16T03:28:28.930 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-16T03:28:28.940 INFO:teuthology.task.internal:Opening connections... 2024-09-16T03:28:28.940 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi042.front.sepia.ceph.com 2024-09-16T03:28:28.941 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi042.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T03:28:29.026 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi154.front.sepia.ceph.com 2024-09-16T03:28:29.026 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi154.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T03:28:29.105 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-16T03:28:29.114 DEBUG:teuthology.orchestra.run.smithi042:> uname -m 2024-09-16T03:28:29.119 INFO:teuthology.orchestra.run.smithi042.stdout:x86_64 2024-09-16T03:28:29.119 DEBUG:teuthology.orchestra.run.smithi042:> cat /etc/os-release 2024-09-16T03:28:29.166 INFO:teuthology.orchestra.run.smithi042.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-09-16T03:28:29.166 INFO:teuthology.orchestra.run.smithi042.stdout:NAME="Ubuntu" 2024-09-16T03:28:29.166 INFO:teuthology.orchestra.run.smithi042.stdout:VERSION_ID="22.04" 2024-09-16T03:28:29.166 INFO:teuthology.orchestra.run.smithi042.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-09-16T03:28:29.167 INFO:teuthology.orchestra.run.smithi042.stdout:VERSION_CODENAME=jammy 2024-09-16T03:28:29.167 INFO:teuthology.orchestra.run.smithi042.stdout:ID=ubuntu 2024-09-16T03:28:29.167 INFO:teuthology.orchestra.run.smithi042.stdout:ID_LIKE=debian 2024-09-16T03:28:29.167 INFO:teuthology.orchestra.run.smithi042.stdout:HOME_URL="https://www.ubuntu.com/" 2024-09-16T03:28:29.167 INFO:teuthology.orchestra.run.smithi042.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-09-16T03:28:29.167 INFO:teuthology.orchestra.run.smithi042.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-09-16T03:28:29.168 INFO:teuthology.orchestra.run.smithi042.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-09-16T03:28:29.168 INFO:teuthology.orchestra.run.smithi042.stdout:UBUNTU_CODENAME=jammy 2024-09-16T03:28:29.169 INFO:teuthology.lock.ops:Updating smithi042.front.sepia.ceph.com on lock server 2024-09-16T03:28:29.192 DEBUG:teuthology.orchestra.run.smithi154:> uname -m 2024-09-16T03:28:29.198 INFO:teuthology.orchestra.run.smithi154.stdout:x86_64 2024-09-16T03:28:29.199 DEBUG:teuthology.orchestra.run.smithi154:> cat /etc/os-release 2024-09-16T03:28:29.245 INFO:teuthology.orchestra.run.smithi154.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-09-16T03:28:29.245 INFO:teuthology.orchestra.run.smithi154.stdout:NAME="Ubuntu" 2024-09-16T03:28:29.245 INFO:teuthology.orchestra.run.smithi154.stdout:VERSION_ID="22.04" 2024-09-16T03:28:29.246 INFO:teuthology.orchestra.run.smithi154.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-09-16T03:28:29.246 INFO:teuthology.orchestra.run.smithi154.stdout:VERSION_CODENAME=jammy 2024-09-16T03:28:29.246 INFO:teuthology.orchestra.run.smithi154.stdout:ID=ubuntu 2024-09-16T03:28:29.246 INFO:teuthology.orchestra.run.smithi154.stdout:ID_LIKE=debian 2024-09-16T03:28:29.246 INFO:teuthology.orchestra.run.smithi154.stdout:HOME_URL="https://www.ubuntu.com/" 2024-09-16T03:28:29.246 INFO:teuthology.orchestra.run.smithi154.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-09-16T03:28:29.246 INFO:teuthology.orchestra.run.smithi154.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-09-16T03:28:29.246 INFO:teuthology.orchestra.run.smithi154.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-09-16T03:28:29.246 INFO:teuthology.orchestra.run.smithi154.stdout:UBUNTU_CODENAME=jammy 2024-09-16T03:28:29.247 INFO:teuthology.lock.ops:Updating smithi154.front.sepia.ceph.com on lock server 2024-09-16T03:28:29.275 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-16T03:28:29.286 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-16T03:28:29.307 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-16T03:28:29.307 DEBUG:teuthology.orchestra.run.smithi042:> test '!' -e /home/ubuntu/cephtest 2024-09-16T03:28:29.310 DEBUG:teuthology.orchestra.run.smithi154:> test '!' -e /home/ubuntu/cephtest 2024-09-16T03:28:29.314 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-16T03:28:29.338 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-16T03:28:29.338 DEBUG:teuthology.orchestra.run.smithi042:> test -z $(ls -A /var/lib/ceph) 2024-09-16T03:28:29.359 DEBUG:teuthology.orchestra.run.smithi154:> test -z $(ls -A /var/lib/ceph) 2024-09-16T03:28:29.381 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-16T03:28:29.429 INFO:teuthology.run_tasks:Running task kernel... 2024-09-16T03:28:29.467 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-16T03:28:29.467 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-16T03:28:29.467 DEBUG:teuthology.orchestra.run.smithi042:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-16T03:28:29.468 DEBUG:teuthology.orchestra.run.smithi154:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-16T03:28:29.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T03:28:29.472 DEBUG:teuthology.orchestra.run.smithi042:> uname -r 2024-09-16T03:28:29.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T03:28:29.473 DEBUG:teuthology.orchestra.run.smithi154:> uname -r 2024-09-16T03:28:29.521 INFO:teuthology.orchestra.run.smithi042.stdout:5.15.0-56-generic 2024-09-16T03:28:29.521 INFO:teuthology.task.kernel:Running kernel on smithi042: 5.15.0-56-generic 2024-09-16T03:28:29.522 DEBUG:teuthology.orchestra.run.smithi042:> sudo apt-get clean 2024-09-16T03:28:29.522 INFO:teuthology.orchestra.run.smithi154.stdout:5.15.0-56-generic 2024-09-16T03:28:29.523 INFO:teuthology.task.kernel:Running kernel on smithi154: 5.15.0-56-generic 2024-09-16T03:28:29.523 DEBUG:teuthology.orchestra.run.smithi154:> sudo apt-get clean 2024-09-16T03:28:29.664 DEBUG:teuthology.orchestra.run.smithi154:> sudo apt-get update 2024-09-16T03:28:29.678 DEBUG:teuthology.orchestra.run.smithi042:> sudo apt-get update 2024-09-16T03:28:29.831 INFO:teuthology.orchestra.run.smithi154.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-16T03:28:29.831 INFO:teuthology.orchestra.run.smithi154.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [128 kB] 2024-09-16T03:28:29.831 INFO:teuthology.orchestra.run.smithi154.stdout:Get:3 http://security.ubuntu.com/ubuntu jammy-security InRelease [129 kB] 2024-09-16T03:28:29.840 INFO:teuthology.orchestra.run.smithi042.stdout:Get:1 http://security.ubuntu.com/ubuntu jammy-security InRelease [129 kB] 2024-09-16T03:28:29.881 INFO:teuthology.orchestra.run.smithi154.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [127 kB] 2024-09-16T03:28:29.952 INFO:teuthology.orchestra.run.smithi042.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-16T03:28:30.036 INFO:teuthology.orchestra.run.smithi042.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [128 kB] 2024-09-16T03:28:30.076 INFO:teuthology.orchestra.run.smithi042.stdout:Get:4 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [539 kB] 2024-09-16T03:28:30.079 INFO:teuthology.orchestra.run.smithi154.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [539 kB] 2024-09-16T03:28:30.150 INFO:teuthology.orchestra.run.smithi154.stdout:Get:6 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,806 kB] 2024-09-16T03:28:30.151 INFO:teuthology.orchestra.run.smithi042.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,806 kB] 2024-09-16T03:28:30.153 INFO:teuthology.orchestra.run.smithi154.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [2,023 kB] 2024-09-16T03:28:30.189 INFO:teuthology.orchestra.run.smithi042.stdout:Get:6 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [295 kB] 2024-09-16T03:28:30.190 INFO:teuthology.orchestra.run.smithi154.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [295 kB] 2024-09-16T03:28:30.191 INFO:teuthology.orchestra.run.smithi042.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [13.3 kB] 2024-09-16T03:28:30.192 INFO:teuthology.orchestra.run.smithi042.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [2,377 kB] 2024-09-16T03:28:30.193 INFO:teuthology.orchestra.run.smithi154.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [13.3 kB] 2024-09-16T03:28:30.193 INFO:teuthology.orchestra.run.smithi154.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [37.7 kB] 2024-09-16T03:28:30.194 INFO:teuthology.orchestra.run.smithi154.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [2,377 kB] 2024-09-16T03:28:30.211 INFO:teuthology.orchestra.run.smithi042.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [37.7 kB] 2024-09-16T03:28:30.212 INFO:teuthology.orchestra.run.smithi042.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [409 kB] 2024-09-16T03:28:30.214 INFO:teuthology.orchestra.run.smithi154.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [409 kB] 2024-09-16T03:28:30.215 INFO:teuthology.orchestra.run.smithi042.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [584 B] 2024-09-16T03:28:30.216 INFO:teuthology.orchestra.run.smithi042.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [902 kB] 2024-09-16T03:28:30.219 INFO:teuthology.orchestra.run.smithi154.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [584 B] 2024-09-16T03:28:30.219 INFO:teuthology.orchestra.run.smithi154.stdout:Get:14 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [627 kB] 2024-09-16T03:28:30.223 INFO:teuthology.orchestra.run.smithi042.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [627 kB] 2024-09-16T03:28:30.225 INFO:teuthology.orchestra.run.smithi154.stdout:Get:15 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [902 kB] 2024-09-16T03:28:30.227 INFO:teuthology.orchestra.run.smithi042.stdout:Get:14 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [176 kB] 2024-09-16T03:28:30.228 INFO:teuthology.orchestra.run.smithi042.stdout:Get:15 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [19.2 kB] 2024-09-16T03:28:30.229 INFO:teuthology.orchestra.run.smithi042.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-09-16T03:28:30.229 INFO:teuthology.orchestra.run.smithi042.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-09-16T03:28:30.234 INFO:teuthology.orchestra.run.smithi154.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [176 kB] 2024-09-16T03:28:30.236 INFO:teuthology.orchestra.run.smithi154.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [19.2 kB] 2024-09-16T03:28:30.237 INFO:teuthology.orchestra.run.smithi154.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-09-16T03:28:30.237 INFO:teuthology.orchestra.run.smithi154.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-09-16T03:28:30.238 INFO:teuthology.orchestra.run.smithi154.stdout:Get:20 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-09-16T03:28:30.239 INFO:teuthology.orchestra.run.smithi154.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [696 kB] 2024-09-16T03:28:30.243 INFO:teuthology.orchestra.run.smithi042.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-09-16T03:28:30.244 INFO:teuthology.orchestra.run.smithi042.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [228 B] 2024-09-16T03:28:30.244 INFO:teuthology.orchestra.run.smithi154.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [352 kB] 2024-09-16T03:28:30.246 INFO:teuthology.orchestra.run.smithi154.stdout:Get:23 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [228 B] 2024-09-16T03:28:30.247 INFO:teuthology.orchestra.run.smithi154.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [17.8 kB] 2024-09-16T03:28:30.248 INFO:teuthology.orchestra.run.smithi154.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [2,437 kB] 2024-09-16T03:28:30.265 INFO:teuthology.orchestra.run.smithi154.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [39.3 kB] 2024-09-16T03:28:30.266 INFO:teuthology.orchestra.run.smithi154.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [419 kB] 2024-09-16T03:28:30.269 INFO:teuthology.orchestra.run.smithi154.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [616 B] 2024-09-16T03:28:30.269 INFO:teuthology.orchestra.run.smithi154.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,124 kB] 2024-09-16T03:28:30.276 INFO:teuthology.orchestra.run.smithi154.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [732 kB] 2024-09-16T03:28:30.280 INFO:teuthology.orchestra.run.smithi154.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [261 kB] 2024-09-16T03:28:30.282 INFO:teuthology.orchestra.run.smithi154.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [26.1 kB] 2024-09-16T03:28:30.282 INFO:teuthology.orchestra.run.smithi154.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [43.3 kB] 2024-09-16T03:28:30.283 INFO:teuthology.orchestra.run.smithi154.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,752 B] 2024-09-16T03:28:30.298 INFO:teuthology.orchestra.run.smithi154.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.8 kB] 2024-09-16T03:28:30.298 INFO:teuthology.orchestra.run.smithi154.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [444 B] 2024-09-16T03:28:30.298 INFO:teuthology.orchestra.run.smithi154.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.9 kB] 2024-09-16T03:28:30.299 INFO:teuthology.orchestra.run.smithi154.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.8 kB] 2024-09-16T03:28:30.299 INFO:teuthology.orchestra.run.smithi154.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.1 kB] 2024-09-16T03:28:30.315 INFO:teuthology.orchestra.run.smithi154.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-09-16T03:28:30.315 INFO:teuthology.orchestra.run.smithi154.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [28.8 kB] 2024-09-16T03:28:30.315 INFO:teuthology.orchestra.run.smithi154.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [17.3 kB] 2024-09-16T03:28:30.316 INFO:teuthology.orchestra.run.smithi154.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.5 kB] 2024-09-16T03:28:30.316 INFO:teuthology.orchestra.run.smithi154.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [672 B] 2024-09-16T03:28:30.387 INFO:teuthology.orchestra.run.smithi042.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [127 kB] 2024-09-16T03:28:30.525 INFO:teuthology.orchestra.run.smithi042.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [2,023 kB] 2024-09-16T03:28:30.841 INFO:teuthology.orchestra.run.smithi042.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [696 kB] 2024-09-16T03:28:30.876 INFO:teuthology.orchestra.run.smithi042.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [352 kB] 2024-09-16T03:28:30.890 INFO:teuthology.orchestra.run.smithi042.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [17.8 kB] 2024-09-16T03:28:30.891 INFO:teuthology.orchestra.run.smithi042.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [2,437 kB] 2024-09-16T03:28:30.960 INFO:teuthology.orchestra.run.smithi042.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [39.3 kB] 2024-09-16T03:28:30.961 INFO:teuthology.orchestra.run.smithi042.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [419 kB] 2024-09-16T03:28:30.977 INFO:teuthology.orchestra.run.smithi042.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [616 B] 2024-09-16T03:28:30.977 INFO:teuthology.orchestra.run.smithi042.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [732 kB] 2024-09-16T03:28:30.999 INFO:teuthology.orchestra.run.smithi042.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,124 kB] 2024-09-16T03:28:31.028 INFO:teuthology.orchestra.run.smithi042.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [261 kB] 2024-09-16T03:28:31.034 INFO:teuthology.orchestra.run.smithi042.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [26.1 kB] 2024-09-16T03:28:31.034 INFO:teuthology.orchestra.run.smithi042.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,752 B] 2024-09-16T03:28:31.035 INFO:teuthology.orchestra.run.smithi042.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [43.3 kB] 2024-09-16T03:28:31.085 INFO:teuthology.orchestra.run.smithi042.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.8 kB] 2024-09-16T03:28:31.085 INFO:teuthology.orchestra.run.smithi042.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [444 B] 2024-09-16T03:28:31.086 INFO:teuthology.orchestra.run.smithi042.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.9 kB] 2024-09-16T03:28:31.087 INFO:teuthology.orchestra.run.smithi042.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.8 kB] 2024-09-16T03:28:31.088 INFO:teuthology.orchestra.run.smithi042.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.1 kB] 2024-09-16T03:28:31.166 INFO:teuthology.orchestra.run.smithi042.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-09-16T03:28:31.166 INFO:teuthology.orchestra.run.smithi042.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [28.8 kB] 2024-09-16T03:28:31.167 INFO:teuthology.orchestra.run.smithi042.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [17.3 kB] 2024-09-16T03:28:31.167 INFO:teuthology.orchestra.run.smithi042.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.5 kB] 2024-09-16T03:28:31.168 INFO:teuthology.orchestra.run.smithi042.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [672 B] 2024-09-16T03:28:36.468 INFO:teuthology.orchestra.run.smithi154.stdout:Fetched 16.0 MB in 2s (7,271 kB/s) 2024-09-16T03:28:36.996 INFO:teuthology.orchestra.run.smithi042.stdout:Fetched 16.0 MB in 2s (7,201 kB/s) 2024-09-16T03:28:37.940 INFO:teuthology.orchestra.run.smithi154.stdout:Reading package lists... 2024-09-16T03:28:37.960 DEBUG:teuthology.orchestra.run.smithi154:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-16T03:28:38.022 INFO:teuthology.orchestra.run.smithi154.stdout:Reading package lists... 2024-09-16T03:28:38.213 INFO:teuthology.orchestra.run.smithi154.stdout:Building dependency tree... 2024-09-16T03:28:38.213 INFO:teuthology.orchestra.run.smithi154.stdout:Reading state information... 2024-09-16T03:28:38.383 INFO:teuthology.orchestra.run.smithi154.stdout:The following packages were automatically installed and are no longer required: 2024-09-16T03:28:38.383 INFO:teuthology.orchestra.run.smithi154.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-16T03:28:38.383 INFO:teuthology.orchestra.run.smithi154.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-16T03:28:38.384 INFO:teuthology.orchestra.run.smithi154.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-16T03:28:38.384 INFO:teuthology.orchestra.run.smithi154.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-09-16T03:28:38.384 INFO:teuthology.orchestra.run.smithi154.stdout: python2.7-minimal 2024-09-16T03:28:38.384 INFO:teuthology.orchestra.run.smithi154.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-16T03:28:38.385 INFO:teuthology.orchestra.run.smithi154.stdout:The following additional packages will be installed: 2024-09-16T03:28:38.385 INFO:teuthology.orchestra.run.smithi154.stdout: linux-generic linux-headers-5.15.0-121 linux-headers-5.15.0-121-generic 2024-09-16T03:28:38.385 INFO:teuthology.orchestra.run.smithi154.stdout: linux-headers-generic linux-image-5.15.0-121-generic 2024-09-16T03:28:38.386 INFO:teuthology.orchestra.run.smithi154.stdout: linux-modules-5.15.0-121-generic linux-modules-extra-5.15.0-121-generic 2024-09-16T03:28:38.386 INFO:teuthology.orchestra.run.smithi154.stdout:Suggested packages: 2024-09-16T03:28:38.387 INFO:teuthology.orchestra.run.smithi154.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-09-16T03:28:38.387 INFO:teuthology.orchestra.run.smithi154.stdout:Recommended packages: 2024-09-16T03:28:38.387 INFO:teuthology.orchestra.run.smithi154.stdout: thermald 2024-09-16T03:28:38.420 INFO:teuthology.orchestra.run.smithi042.stdout:Reading package lists... 2024-09-16T03:28:38.437 DEBUG:teuthology.orchestra.run.smithi042:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-16T03:28:38.441 INFO:teuthology.orchestra.run.smithi154.stdout:The following NEW packages will be installed: 2024-09-16T03:28:38.441 INFO:teuthology.orchestra.run.smithi154.stdout: linux-headers-5.15.0-121 linux-headers-5.15.0-121-generic 2024-09-16T03:28:38.441 INFO:teuthology.orchestra.run.smithi154.stdout: linux-image-5.15.0-121-generic linux-modules-5.15.0-121-generic 2024-09-16T03:28:38.441 INFO:teuthology.orchestra.run.smithi154.stdout: linux-modules-extra-5.15.0-121-generic 2024-09-16T03:28:38.442 INFO:teuthology.orchestra.run.smithi154.stdout:The following packages will be upgraded: 2024-09-16T03:28:38.443 INFO:teuthology.orchestra.run.smithi154.stdout: linux-generic linux-headers-generic linux-image-generic 2024-09-16T03:28:38.493 INFO:teuthology.orchestra.run.smithi154.stdout:3 upgraded, 5 newly installed, 0 to remove and 344 not upgraded. 2024-09-16T03:28:38.493 INFO:teuthology.orchestra.run.smithi154.stdout:Need to get 113 MB of archives. 2024-09-16T03:28:38.493 INFO:teuthology.orchestra.run.smithi154.stdout:After this operation, 584 MB of additional disk space will be used. 2024-09-16T03:28:38.493 INFO:teuthology.orchestra.run.smithi154.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-121-generic amd64 5.15.0-121.131 [22.7 MB] 2024-09-16T03:28:38.504 INFO:teuthology.orchestra.run.smithi042.stdout:Reading package lists... 2024-09-16T03:28:38.718 INFO:teuthology.orchestra.run.smithi042.stdout:Building dependency tree... 2024-09-16T03:28:38.719 INFO:teuthology.orchestra.run.smithi042.stdout:Reading state information... 2024-09-16T03:28:38.912 INFO:teuthology.orchestra.run.smithi042.stdout:The following packages were automatically installed and are no longer required: 2024-09-16T03:28:38.913 INFO:teuthology.orchestra.run.smithi042.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-16T03:28:38.913 INFO:teuthology.orchestra.run.smithi042.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-16T03:28:38.913 INFO:teuthology.orchestra.run.smithi042.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-16T03:28:38.913 INFO:teuthology.orchestra.run.smithi042.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-09-16T03:28:38.913 INFO:teuthology.orchestra.run.smithi042.stdout: python2.7-minimal 2024-09-16T03:28:38.914 INFO:teuthology.orchestra.run.smithi042.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-16T03:28:38.915 INFO:teuthology.orchestra.run.smithi042.stdout:The following additional packages will be installed: 2024-09-16T03:28:38.915 INFO:teuthology.orchestra.run.smithi042.stdout: linux-generic linux-headers-5.15.0-121 linux-headers-5.15.0-121-generic 2024-09-16T03:28:38.915 INFO:teuthology.orchestra.run.smithi042.stdout: linux-headers-generic linux-image-5.15.0-121-generic 2024-09-16T03:28:38.915 INFO:teuthology.orchestra.run.smithi042.stdout: linux-modules-5.15.0-121-generic linux-modules-extra-5.15.0-121-generic 2024-09-16T03:28:38.917 INFO:teuthology.orchestra.run.smithi042.stdout:Suggested packages: 2024-09-16T03:28:38.917 INFO:teuthology.orchestra.run.smithi042.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-09-16T03:28:38.917 INFO:teuthology.orchestra.run.smithi042.stdout:Recommended packages: 2024-09-16T03:28:38.917 INFO:teuthology.orchestra.run.smithi042.stdout: thermald 2024-09-16T03:28:38.924 INFO:teuthology.orchestra.run.smithi154.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-121-generic amd64 5.15.0-121.131 [11.5 MB] 2024-09-16T03:28:38.972 INFO:teuthology.orchestra.run.smithi042.stdout:The following NEW packages will be installed: 2024-09-16T03:28:38.972 INFO:teuthology.orchestra.run.smithi042.stdout: linux-headers-5.15.0-121 linux-headers-5.15.0-121-generic 2024-09-16T03:28:38.972 INFO:teuthology.orchestra.run.smithi042.stdout: linux-image-5.15.0-121-generic linux-modules-5.15.0-121-generic 2024-09-16T03:28:38.973 INFO:teuthology.orchestra.run.smithi042.stdout: linux-modules-extra-5.15.0-121-generic 2024-09-16T03:28:38.975 INFO:teuthology.orchestra.run.smithi042.stdout:The following packages will be upgraded: 2024-09-16T03:28:38.975 INFO:teuthology.orchestra.run.smithi042.stdout: linux-generic linux-headers-generic linux-image-generic 2024-09-16T03:28:39.024 INFO:teuthology.orchestra.run.smithi154.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-121-generic amd64 5.15.0-121.131 [63.9 MB] 2024-09-16T03:28:39.155 INFO:teuthology.orchestra.run.smithi042.stdout:3 upgraded, 5 newly installed, 0 to remove and 344 not upgraded. 2024-09-16T03:28:39.155 INFO:teuthology.orchestra.run.smithi042.stdout:Need to get 113 MB of archives. 2024-09-16T03:28:39.155 INFO:teuthology.orchestra.run.smithi042.stdout:After this operation, 584 MB of additional disk space will be used. 2024-09-16T03:28:39.155 INFO:teuthology.orchestra.run.smithi042.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-121-generic amd64 5.15.0-121.131 [22.7 MB] 2024-09-16T03:28:39.806 INFO:teuthology.orchestra.run.smithi154.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.121.121 [1,694 B] 2024-09-16T03:28:39.806 INFO:teuthology.orchestra.run.smithi154.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.121.121 [2,464 B] 2024-09-16T03:28:39.806 INFO:teuthology.orchestra.run.smithi154.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-121 all 5.15.0-121.131 [12.3 MB] 2024-09-16T03:28:39.925 INFO:teuthology.orchestra.run.smithi154.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-121-generic amd64 5.15.0-121.131 [2,827 kB] 2024-09-16T03:28:40.026 INFO:teuthology.orchestra.run.smithi154.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.121.121 [2,316 B] 2024-09-16T03:28:40.425 INFO:teuthology.orchestra.run.smithi154.stdout:Fetched 113 MB in 2s (71.9 MB/s) 2024-09-16T03:28:40.554 INFO:teuthology.orchestra.run.smithi042.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-121-generic amd64 5.15.0-121.131 [11.5 MB] 2024-09-16T03:28:40.607 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package linux-modules-5.15.0-121-generic. 2024-09-16T03:28:40.909 INFO:teuthology.orchestra.run.smithi042.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-121-generic amd64 5.15.0-121.131 [63.9 MB] 2024-09-16T03:28:42.688 INFO:teuthology.orchestra.run.smithi154.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-09-16T03:28:42.693 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../0-linux-modules-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-16T03:28:42.788 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking linux-modules-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T03:28:43.171 INFO:teuthology.orchestra.run.smithi042.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.121.121 [1,694 B] 2024-09-16T03:28:43.171 INFO:teuthology.orchestra.run.smithi042.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.121.121 [2,464 B] 2024-09-16T03:28:43.172 INFO:teuthology.orchestra.run.smithi042.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-121 all 5.15.0-121.131 [12.3 MB] 2024-09-16T03:28:43.531 INFO:teuthology.orchestra.run.smithi042.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-121-generic amd64 5.15.0-121.131 [2,827 kB] 2024-09-16T03:28:43.616 INFO:teuthology.orchestra.run.smithi042.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.121.121 [2,316 B] 2024-09-16T03:28:44.016 INFO:teuthology.orchestra.run.smithi042.stdout:Fetched 113 MB in 5s (24.4 MB/s) 2024-09-16T03:28:44.201 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package linux-modules-5.15.0-121-generic. 2024-09-16T03:28:44.931 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package linux-image-5.15.0-121-generic. 2024-09-16T03:28:44.953 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../1-linux-image-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-16T03:28:45.059 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking linux-image-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T03:28:45.484 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-121-generic. 2024-09-16T03:28:45.505 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-16T03:28:45.534 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking linux-modules-extra-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T03:28:46.258 INFO:teuthology.orchestra.run.smithi042.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-09-16T03:28:46.262 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../0-linux-modules-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-16T03:28:46.473 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking linux-modules-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T03:28:48.924 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package linux-image-5.15.0-121-generic. 2024-09-16T03:28:48.946 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../1-linux-image-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-16T03:28:49.030 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking linux-image-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T03:28:49.486 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-121-generic. 2024-09-16T03:28:49.507 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-16T03:28:49.544 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking linux-modules-extra-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T03:28:52.669 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../3-linux-generic_5.15.0.121.121_amd64.deb ... 2024-09-16T03:28:52.797 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking linux-generic (5.15.0.121.121) over (5.15.0.56.54) ... 2024-09-16T03:28:53.121 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.121.121_amd64.deb ... 2024-09-16T03:28:53.225 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking linux-image-generic (5.15.0.121.121) over (5.15.0.56.54) ... 2024-09-16T03:28:53.478 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package linux-headers-5.15.0-121. 2024-09-16T03:28:53.489 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../5-linux-headers-5.15.0-121_5.15.0-121.131_all.deb ... 2024-09-16T03:28:53.553 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking linux-headers-5.15.0-121 (5.15.0-121.131) ... 2024-09-16T03:28:56.186 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../3-linux-generic_5.15.0.121.121_amd64.deb ... 2024-09-16T03:28:56.304 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking linux-generic (5.15.0.121.121) over (5.15.0.56.54) ... 2024-09-16T03:28:56.688 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.121.121_amd64.deb ... 2024-09-16T03:28:56.799 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking linux-image-generic (5.15.0.121.121) over (5.15.0.56.54) ... 2024-09-16T03:28:57.052 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package linux-headers-5.15.0-121. 2024-09-16T03:28:57.063 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../5-linux-headers-5.15.0-121_5.15.0-121.131_all.deb ... 2024-09-16T03:28:57.102 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking linux-headers-5.15.0-121 (5.15.0-121.131) ... 2024-09-16T03:28:59.616 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package linux-headers-5.15.0-121-generic. 2024-09-16T03:28:59.653 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../6-linux-headers-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-16T03:28:59.691 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking linux-headers-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T03:29:02.295 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.121.121_amd64.deb ... 2024-09-16T03:29:02.435 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking linux-headers-generic (5.15.0.121.121) over (5.15.0.56.54) ... 2024-09-16T03:29:02.768 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up linux-headers-5.15.0-121 (5.15.0-121.131) ... 2024-09-16T03:29:02.872 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up linux-headers-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T03:29:02.930 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package linux-headers-5.15.0-121-generic. 2024-09-16T03:29:02.967 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../6-linux-headers-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-16T03:29:03.013 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking linux-headers-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T03:29:03.024 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up linux-headers-generic (5.15.0.121.121) ... 2024-09-16T03:29:03.151 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up linux-image-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T03:29:04.490 INFO:teuthology.orchestra.run.smithi154.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-09-16T03:29:04.491 INFO:teuthology.orchestra.run.smithi154.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-09-16T03:29:04.491 INFO:teuthology.orchestra.run.smithi154.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-121-generic 2024-09-16T03:29:04.491 INFO:teuthology.orchestra.run.smithi154.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-121-generic 2024-09-16T03:29:04.679 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up linux-modules-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T03:29:06.606 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up linux-modules-extra-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T03:29:06.689 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.121.121_amd64.deb ... 2024-09-16T03:29:06.791 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking linux-headers-generic (5.15.0.121.121) over (5.15.0.56.54) ... 2024-09-16T03:29:07.176 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up linux-headers-5.15.0-121 (5.15.0-121.131) ... 2024-09-16T03:29:07.280 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up linux-headers-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T03:29:07.422 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up linux-headers-generic (5.15.0.121.121) ... 2024-09-16T03:29:07.550 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up linux-image-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T03:29:07.973 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up linux-image-generic (5.15.0.121.121) ... 2024-09-16T03:29:08.100 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up linux-generic (5.15.0.121.121) ... 2024-09-16T03:29:08.242 INFO:teuthology.orchestra.run.smithi154.stdout:Processing triggers for linux-image-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T03:29:08.313 INFO:teuthology.orchestra.run.smithi154.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-09-16T03:29:08.313 INFO:teuthology.orchestra.run.smithi154.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-121-generic 2024-09-16T03:29:08.910 INFO:teuthology.orchestra.run.smithi042.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-09-16T03:29:08.910 INFO:teuthology.orchestra.run.smithi042.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-09-16T03:29:08.910 INFO:teuthology.orchestra.run.smithi042.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-121-generic 2024-09-16T03:29:08.910 INFO:teuthology.orchestra.run.smithi042.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-121-generic 2024-09-16T03:29:09.087 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up linux-modules-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T03:29:10.546 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up linux-modules-extra-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T03:29:12.414 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up linux-image-generic (5.15.0.121.121) ... 2024-09-16T03:29:12.523 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up linux-generic (5.15.0.121.121) ... 2024-09-16T03:29:12.675 INFO:teuthology.orchestra.run.smithi042.stdout:Processing triggers for linux-image-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T03:29:12.731 INFO:teuthology.orchestra.run.smithi042.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-09-16T03:29:12.731 INFO:teuthology.orchestra.run.smithi042.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-121-generic 2024-09-16T03:29:23.234 INFO:teuthology.orchestra.run.smithi154.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-09-16T03:29:23.234 INFO:teuthology.orchestra.run.smithi154.stdout:Sourcing file `/etc/default/grub' 2024-09-16T03:29:23.265 INFO:teuthology.orchestra.run.smithi154.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-16T03:29:23.288 INFO:teuthology.orchestra.run.smithi154.stdout:Generating grub configuration file ... 2024-09-16T03:29:23.580 INFO:teuthology.orchestra.run.smithi154.stdout:Found linux image: /boot/vmlinuz-5.15.0-121-generic 2024-09-16T03:29:23.594 INFO:teuthology.orchestra.run.smithi154.stdout:Found initrd image: /boot/initrd.img-5.15.0-121-generic 2024-09-16T03:29:23.878 INFO:teuthology.orchestra.run.smithi154.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-16T03:29:23.884 INFO:teuthology.orchestra.run.smithi154.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-16T03:29:23.951 INFO:teuthology.orchestra.run.smithi154.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-16T03:29:23.956 INFO:teuthology.orchestra.run.smithi154.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-16T03:29:24.109 INFO:teuthology.orchestra.run.smithi154.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-16T03:29:24.109 INFO:teuthology.orchestra.run.smithi154.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-09-16T03:29:24.110 INFO:teuthology.orchestra.run.smithi154.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-16T03:29:24.152 INFO:teuthology.orchestra.run.smithi154.stdout:done 2024-09-16T03:29:24.607 DEBUG:teuthology.orchestra.run.smithi154:> dpkg -s linux-image-generic 2024-09-16T03:29:24.626 INFO:teuthology.orchestra.run.smithi154.stdout:Package: linux-image-generic 2024-09-16T03:29:24.626 INFO:teuthology.orchestra.run.smithi154.stdout:Status: install ok installed 2024-09-16T03:29:24.626 INFO:teuthology.orchestra.run.smithi154.stdout:Priority: optional 2024-09-16T03:29:24.626 INFO:teuthology.orchestra.run.smithi154.stdout:Section: kernel 2024-09-16T03:29:24.626 INFO:teuthology.orchestra.run.smithi154.stdout:Installed-Size: 21 2024-09-16T03:29:24.626 INFO:teuthology.orchestra.run.smithi154.stdout:Maintainer: Ubuntu Kernel Team 2024-09-16T03:29:24.626 INFO:teuthology.orchestra.run.smithi154.stdout:Architecture: amd64 2024-09-16T03:29:24.626 INFO:teuthology.orchestra.run.smithi154.stdout:Source: linux-meta 2024-09-16T03:29:24.627 INFO:teuthology.orchestra.run.smithi154.stdout:Version: 5.15.0.121.121 2024-09-16T03:29:24.627 INFO:teuthology.orchestra.run.smithi154.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-121), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-16T03:29:24.627 INFO:teuthology.orchestra.run.smithi154.stdout:Depends: linux-image-5.15.0-121-generic, linux-modules-extra-5.15.0-121-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-16T03:29:24.627 INFO:teuthology.orchestra.run.smithi154.stdout:Recommends: thermald 2024-09-16T03:29:24.627 INFO:teuthology.orchestra.run.smithi154.stdout:Description: Generic Linux kernel image 2024-09-16T03:29:24.627 INFO:teuthology.orchestra.run.smithi154.stdout: This package will always depend on the latest generic kernel image 2024-09-16T03:29:24.627 INFO:teuthology.orchestra.run.smithi154.stdout: available. 2024-09-16T03:29:24.627 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-121-generic 2024-09-16T03:29:24.627 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-16T03:29:24.627 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-16T03:29:24.628 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-09-16T03:29:24.628 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi154.front.sepia.ceph.com, path=None, version=distro) 2024-09-16T03:29:24.628 DEBUG:teuthology.orchestra.run.smithi154:> sudo apt-get clean 2024-09-16T03:29:24.791 DEBUG:teuthology.orchestra.run.smithi154:> sudo apt-get update 2024-09-16T03:29:24.942 INFO:teuthology.orchestra.run.smithi154.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-16T03:29:24.949 INFO:teuthology.orchestra.run.smithi154.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-16T03:29:24.966 INFO:teuthology.orchestra.run.smithi154.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-16T03:29:25.045 INFO:teuthology.orchestra.run.smithi154.stdout:Hit:4 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-16T03:29:26.385 INFO:teuthology.orchestra.run.smithi154.stdout:Reading package lists... 2024-09-16T03:29:26.402 DEBUG:teuthology.orchestra.run.smithi154:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-16T03:29:26.464 INFO:teuthology.orchestra.run.smithi154.stdout:Reading package lists... 2024-09-16T03:29:26.669 INFO:teuthology.orchestra.run.smithi154.stdout:Building dependency tree... 2024-09-16T03:29:26.670 INFO:teuthology.orchestra.run.smithi154.stdout:Reading state information... 2024-09-16T03:29:26.849 INFO:teuthology.orchestra.run.smithi154.stdout:linux-image-generic is already the newest version (5.15.0.121.121). 2024-09-16T03:29:26.849 INFO:teuthology.orchestra.run.smithi154.stdout:The following packages were automatically installed and are no longer required: 2024-09-16T03:29:26.850 INFO:teuthology.orchestra.run.smithi154.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-16T03:29:26.850 INFO:teuthology.orchestra.run.smithi154.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-16T03:29:26.850 INFO:teuthology.orchestra.run.smithi154.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-16T03:29:26.850 INFO:teuthology.orchestra.run.smithi154.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-16T03:29:26.850 INFO:teuthology.orchestra.run.smithi154.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-16T03:29:26.850 INFO:teuthology.orchestra.run.smithi154.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-16T03:29:26.891 INFO:teuthology.orchestra.run.smithi154.stdout:0 upgraded, 0 newly installed, 0 to remove and 344 not upgraded. 2024-09-16T03:29:26.892 DEBUG:teuthology.orchestra.run.smithi154:> dpkg -s linux-image-generic 2024-09-16T03:29:26.907 INFO:teuthology.orchestra.run.smithi154.stdout:Package: linux-image-generic 2024-09-16T03:29:26.907 INFO:teuthology.orchestra.run.smithi154.stdout:Status: install ok installed 2024-09-16T03:29:26.907 INFO:teuthology.orchestra.run.smithi154.stdout:Priority: optional 2024-09-16T03:29:26.908 INFO:teuthology.orchestra.run.smithi154.stdout:Section: kernel 2024-09-16T03:29:26.908 INFO:teuthology.orchestra.run.smithi154.stdout:Installed-Size: 21 2024-09-16T03:29:26.908 INFO:teuthology.orchestra.run.smithi154.stdout:Maintainer: Ubuntu Kernel Team 2024-09-16T03:29:26.908 INFO:teuthology.orchestra.run.smithi154.stdout:Architecture: amd64 2024-09-16T03:29:26.908 INFO:teuthology.orchestra.run.smithi154.stdout:Source: linux-meta 2024-09-16T03:29:26.908 INFO:teuthology.orchestra.run.smithi154.stdout:Version: 5.15.0.121.121 2024-09-16T03:29:26.908 INFO:teuthology.orchestra.run.smithi154.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-121), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-16T03:29:26.908 INFO:teuthology.orchestra.run.smithi154.stdout:Depends: linux-image-5.15.0-121-generic, linux-modules-extra-5.15.0-121-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-16T03:29:26.908 INFO:teuthology.orchestra.run.smithi154.stdout:Recommends: thermald 2024-09-16T03:29:26.908 INFO:teuthology.orchestra.run.smithi154.stdout:Description: Generic Linux kernel image 2024-09-16T03:29:26.908 INFO:teuthology.orchestra.run.smithi154.stdout: This package will always depend on the latest generic kernel image 2024-09-16T03:29:26.908 INFO:teuthology.orchestra.run.smithi154.stdout: available. 2024-09-16T03:29:26.909 DEBUG:teuthology.orchestra.run.smithi154:> mktemp 2024-09-16T03:29:26.956 INFO:teuthology.orchestra.run.smithi154.stdout:/tmp/tmp.Wu8ANYdjb2 2024-09-16T03:29:26.956 DEBUG:teuthology.orchestra.run.smithi154:> sudo cp /boot/grub/grub.cfg /tmp/tmp.Wu8ANYdjb2 2024-09-16T03:29:27.013 DEBUG:teuthology.orchestra.run.smithi154:> sudo chmod 0666 /tmp/tmp.Wu8ANYdjb2 2024-09-16T03:29:27.139 DEBUG:teuthology.orchestra.remote:smithi154:/tmp/tmp.Wu8ANYdjb2 is 10KB 2024-09-16T03:29:27.151 DEBUG:teuthology.orchestra.run.smithi154:> rm -fr /tmp/tmp.Wu8ANYdjb2 2024-09-16T03:29:27.157 DEBUG:teuthology.orchestra.run.smithi154:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-09-16T03:29:27.211 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-09-16T03:29:27.211 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-09-16T03:29:27.211 DEBUG:teuthology.orchestra.run.smithi154:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-09-16T03:29:27.275 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-121-generic 2024-09-16T03:29:27.275 DEBUG:teuthology.orchestra.run.smithi154:> sudo update-grub 2024-09-16T03:29:27.289 INFO:teuthology.orchestra.run.smithi042.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-09-16T03:29:27.289 INFO:teuthology.orchestra.run.smithi042.stdout:Sourcing file `/etc/default/grub' 2024-09-16T03:29:27.309 INFO:teuthology.orchestra.run.smithi042.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-16T03:29:27.335 INFO:teuthology.orchestra.run.smithi042.stdout:Generating grub configuration file ... 2024-09-16T03:29:27.627 INFO:teuthology.orchestra.run.smithi042.stdout:Found linux image: /boot/vmlinuz-5.15.0-121-generic 2024-09-16T03:29:27.647 INFO:teuthology.orchestra.run.smithi042.stdout:Found initrd image: /boot/initrd.img-5.15.0-121-generic 2024-09-16T03:29:27.944 INFO:teuthology.orchestra.run.smithi042.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-16T03:29:27.949 INFO:teuthology.orchestra.run.smithi042.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-16T03:29:28.021 INFO:teuthology.orchestra.run.smithi042.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-16T03:29:28.026 INFO:teuthology.orchestra.run.smithi042.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-16T03:29:28.182 INFO:teuthology.orchestra.run.smithi042.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-16T03:29:28.182 INFO:teuthology.orchestra.run.smithi042.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-09-16T03:29:28.182 INFO:teuthology.orchestra.run.smithi042.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-16T03:29:28.230 INFO:teuthology.orchestra.run.smithi042.stdout:done 2024-09-16T03:29:28.539 INFO:teuthology.orchestra.run.smithi154.stderr:Sourcing file `/etc/default/grub' 2024-09-16T03:29:28.541 INFO:teuthology.orchestra.run.smithi154.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-16T03:29:28.544 INFO:teuthology.orchestra.run.smithi154.stderr:Generating grub configuration file ... 2024-09-16T03:29:28.698 DEBUG:teuthology.orchestra.run.smithi042:> dpkg -s linux-image-generic 2024-09-16T03:29:28.729 INFO:teuthology.orchestra.run.smithi154.stderr:Found linux image: /boot/vmlinuz-5.15.0-121-generic 2024-09-16T03:29:28.730 INFO:teuthology.orchestra.run.smithi042.stdout:Package: linux-image-generic 2024-09-16T03:29:28.730 INFO:teuthology.orchestra.run.smithi042.stdout:Status: install ok installed 2024-09-16T03:29:28.730 INFO:teuthology.orchestra.run.smithi042.stdout:Priority: optional 2024-09-16T03:29:28.730 INFO:teuthology.orchestra.run.smithi042.stdout:Section: kernel 2024-09-16T03:29:28.731 INFO:teuthology.orchestra.run.smithi042.stdout:Installed-Size: 21 2024-09-16T03:29:28.731 INFO:teuthology.orchestra.run.smithi042.stdout:Maintainer: Ubuntu Kernel Team 2024-09-16T03:29:28.731 INFO:teuthology.orchestra.run.smithi042.stdout:Architecture: amd64 2024-09-16T03:29:28.731 INFO:teuthology.orchestra.run.smithi042.stdout:Source: linux-meta 2024-09-16T03:29:28.731 INFO:teuthology.orchestra.run.smithi042.stdout:Version: 5.15.0.121.121 2024-09-16T03:29:28.731 INFO:teuthology.orchestra.run.smithi042.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-121), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-16T03:29:28.731 INFO:teuthology.orchestra.run.smithi042.stdout:Depends: linux-image-5.15.0-121-generic, linux-modules-extra-5.15.0-121-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-16T03:29:28.731 INFO:teuthology.orchestra.run.smithi042.stdout:Recommends: thermald 2024-09-16T03:29:28.731 INFO:teuthology.orchestra.run.smithi042.stdout:Description: Generic Linux kernel image 2024-09-16T03:29:28.731 INFO:teuthology.orchestra.run.smithi042.stdout: This package will always depend on the latest generic kernel image 2024-09-16T03:29:28.731 INFO:teuthology.orchestra.run.smithi042.stdout: available. 2024-09-16T03:29:28.732 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-121-generic 2024-09-16T03:29:28.732 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-16T03:29:28.732 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-16T03:29:28.732 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-16T03:29:28.732 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi042.front.sepia.ceph.com, path=None, version=distro) 2024-09-16T03:29:28.732 DEBUG:teuthology.orchestra.run.smithi042:> sudo apt-get clean 2024-09-16T03:29:28.734 INFO:teuthology.orchestra.run.smithi154.stderr:Found initrd image: /boot/initrd.img-5.15.0-121-generic 2024-09-16T03:29:28.924 DEBUG:teuthology.orchestra.run.smithi042:> sudo apt-get update 2024-09-16T03:29:29.022 INFO:teuthology.orchestra.run.smithi154.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-16T03:29:29.028 INFO:teuthology.orchestra.run.smithi154.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-16T03:29:29.052 INFO:teuthology.orchestra.run.smithi042.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-16T03:29:29.052 INFO:teuthology.orchestra.run.smithi042.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-16T03:29:29.061 INFO:teuthology.orchestra.run.smithi042.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-16T03:29:29.070 INFO:teuthology.orchestra.run.smithi154.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-16T03:29:29.076 INFO:teuthology.orchestra.run.smithi154.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-16T03:29:29.079 INFO:teuthology.orchestra.run.smithi042.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-16T03:29:29.197 INFO:teuthology.orchestra.run.smithi154.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-16T03:29:29.197 INFO:teuthology.orchestra.run.smithi154.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-09-16T03:29:29.197 INFO:teuthology.orchestra.run.smithi154.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-16T03:29:29.223 INFO:teuthology.orchestra.run.smithi154.stderr:done 2024-09-16T03:29:29.232 DEBUG:teuthology.orchestra.run.smithi154:> sudo shutdown -r now 2024-09-16T03:29:30.498 INFO:teuthology.orchestra.run.smithi042.stdout:Reading package lists... 2024-09-16T03:29:30.518 DEBUG:teuthology.orchestra.run.smithi042:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-16T03:29:30.583 INFO:teuthology.orchestra.run.smithi042.stdout:Reading package lists... 2024-09-16T03:29:30.798 INFO:teuthology.orchestra.run.smithi042.stdout:Building dependency tree... 2024-09-16T03:29:30.798 INFO:teuthology.orchestra.run.smithi042.stdout:Reading state information... 2024-09-16T03:29:30.995 INFO:teuthology.orchestra.run.smithi042.stdout:linux-image-generic is already the newest version (5.15.0.121.121). 2024-09-16T03:29:30.995 INFO:teuthology.orchestra.run.smithi042.stdout:The following packages were automatically installed and are no longer required: 2024-09-16T03:29:30.995 INFO:teuthology.orchestra.run.smithi042.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-16T03:29:30.995 INFO:teuthology.orchestra.run.smithi042.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-16T03:29:30.995 INFO:teuthology.orchestra.run.smithi042.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-16T03:29:30.996 INFO:teuthology.orchestra.run.smithi042.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-16T03:29:30.996 INFO:teuthology.orchestra.run.smithi042.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-16T03:29:30.996 INFO:teuthology.orchestra.run.smithi042.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-16T03:29:31.040 INFO:teuthology.orchestra.run.smithi042.stdout:0 upgraded, 0 newly installed, 0 to remove and 344 not upgraded. 2024-09-16T03:29:31.042 DEBUG:teuthology.orchestra.run.smithi042:> dpkg -s linux-image-generic 2024-09-16T03:29:31.058 INFO:teuthology.orchestra.run.smithi042.stdout:Package: linux-image-generic 2024-09-16T03:29:31.059 INFO:teuthology.orchestra.run.smithi042.stdout:Status: install ok installed 2024-09-16T03:29:31.059 INFO:teuthology.orchestra.run.smithi042.stdout:Priority: optional 2024-09-16T03:29:31.059 INFO:teuthology.orchestra.run.smithi042.stdout:Section: kernel 2024-09-16T03:29:31.059 INFO:teuthology.orchestra.run.smithi042.stdout:Installed-Size: 21 2024-09-16T03:29:31.060 INFO:teuthology.orchestra.run.smithi042.stdout:Maintainer: Ubuntu Kernel Team 2024-09-16T03:29:31.060 INFO:teuthology.orchestra.run.smithi042.stdout:Architecture: amd64 2024-09-16T03:29:31.060 INFO:teuthology.orchestra.run.smithi042.stdout:Source: linux-meta 2024-09-16T03:29:31.060 INFO:teuthology.orchestra.run.smithi042.stdout:Version: 5.15.0.121.121 2024-09-16T03:29:31.060 INFO:teuthology.orchestra.run.smithi042.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-121), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-16T03:29:31.060 INFO:teuthology.orchestra.run.smithi042.stdout:Depends: linux-image-5.15.0-121-generic, linux-modules-extra-5.15.0-121-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-16T03:29:31.060 INFO:teuthology.orchestra.run.smithi042.stdout:Recommends: thermald 2024-09-16T03:29:31.060 INFO:teuthology.orchestra.run.smithi042.stdout:Description: Generic Linux kernel image 2024-09-16T03:29:31.060 INFO:teuthology.orchestra.run.smithi042.stdout: This package will always depend on the latest generic kernel image 2024-09-16T03:29:31.060 INFO:teuthology.orchestra.run.smithi042.stdout: available. 2024-09-16T03:29:31.061 DEBUG:teuthology.orchestra.run.smithi042:> mktemp 2024-09-16T03:29:31.105 INFO:teuthology.orchestra.run.smithi042.stdout:/tmp/tmp.GZ924gOK3l 2024-09-16T03:29:31.106 DEBUG:teuthology.orchestra.run.smithi042:> sudo cp /boot/grub/grub.cfg /tmp/tmp.GZ924gOK3l 2024-09-16T03:29:31.163 DEBUG:teuthology.orchestra.run.smithi042:> sudo chmod 0666 /tmp/tmp.GZ924gOK3l 2024-09-16T03:29:31.280 DEBUG:teuthology.orchestra.remote:smithi042:/tmp/tmp.GZ924gOK3l is 10KB 2024-09-16T03:29:31.293 DEBUG:teuthology.orchestra.run.smithi042:> rm -fr /tmp/tmp.GZ924gOK3l 2024-09-16T03:29:31.299 DEBUG:teuthology.orchestra.run.smithi042:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-09-16T03:29:31.354 DEBUG:teuthology.orchestra.run.smithi042:> set -ex 2024-09-16T03:29:31.354 DEBUG:teuthology.orchestra.run.smithi042:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-09-16T03:29:31.354 DEBUG:teuthology.orchestra.run.smithi042:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-09-16T03:29:31.441 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-121-generic 2024-09-16T03:29:31.441 DEBUG:teuthology.orchestra.run.smithi042:> sudo update-grub 2024-09-16T03:29:32.749 INFO:teuthology.orchestra.run.smithi042.stderr:Sourcing file `/etc/default/grub' 2024-09-16T03:29:32.750 INFO:teuthology.orchestra.run.smithi042.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-16T03:29:32.752 INFO:teuthology.orchestra.run.smithi042.stderr:Generating grub configuration file ... 2024-09-16T03:29:32.930 INFO:teuthology.orchestra.run.smithi042.stderr:Found linux image: /boot/vmlinuz-5.15.0-121-generic 2024-09-16T03:29:32.936 INFO:teuthology.orchestra.run.smithi042.stderr:Found initrd image: /boot/initrd.img-5.15.0-121-generic 2024-09-16T03:29:33.176 INFO:teuthology.orchestra.run.smithi042.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-16T03:29:33.179 INFO:teuthology.orchestra.run.smithi042.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-16T03:29:33.221 INFO:teuthology.orchestra.run.smithi042.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-16T03:29:33.226 INFO:teuthology.orchestra.run.smithi042.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-16T03:29:33.355 INFO:teuthology.orchestra.run.smithi042.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-16T03:29:33.355 INFO:teuthology.orchestra.run.smithi042.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-09-16T03:29:33.355 INFO:teuthology.orchestra.run.smithi042.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-16T03:29:33.367 INFO:teuthology.orchestra.run.smithi042.stderr:done 2024-09-16T03:29:33.369 DEBUG:teuthology.orchestra.run.smithi042:> sudo shutdown -r now 2024-09-16T03:29:59.261 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-09-16T03:29:59.262 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi154.front.sepia.ceph.com' 2024-09-16T03:29:59.263 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi154.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T03:30:03.374 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-16T03:30:03.374 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi042.front.sepia.ceph.com' 2024-09-16T03:30:03.375 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi042.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T03:30:17.789 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.154 2024-09-16T03:30:21.885 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.42 2024-09-16T03:30:26.795 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi154.front.sepia.ceph.com' 2024-09-16T03:30:26.796 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi154.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T03:30:29.853 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.154 2024-09-16T03:30:30.887 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi042.front.sepia.ceph.com' 2024-09-16T03:30:30.888 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi042.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T03:30:41.865 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi154.front.sepia.ceph.com' 2024-09-16T03:30:41.868 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi154.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T03:31:30.937 DEBUG:teuthology.orchestra.remote:timed out 2024-09-16T03:31:41.881 DEBUG:teuthology.orchestra.remote:timed out 2024-09-16T03:31:42.940 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi042.front.sepia.ceph.com' 2024-09-16T03:31:42.941 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi042.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T03:31:43.219 DEBUG:teuthology.orchestra.run.smithi042:> true 2024-09-16T03:31:43.910 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi042.front.sepia.ceph.com' 2024-09-16T03:31:43.910 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.15.0-121-generic"... 2024-09-16T03:31:43.910 DEBUG:teuthology.orchestra.run.smithi042:> uname -r 2024-09-16T03:31:43.959 INFO:teuthology.orchestra.run.smithi042.stdout:5.15.0-121-generic 2024-09-16T03:31:43.959 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-121-generic vs 5.15.0-121-generic 2024-09-16T03:31:43.959 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-16T03:31:43.959 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-09-16T03:31:44.961 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-16T03:31:44.961 DEBUG:teuthology.orchestra.run.smithi042:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-16T03:31:45.050 INFO:teuthology.orchestra.run.smithi042.stdout:ttyS1 2024-09-16T03:31:45.079 DEBUG:teuthology.parallel:result is None 2024-09-16T03:31:56.882 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi154.front.sepia.ceph.com' 2024-09-16T03:31:56.883 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi154.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T03:31:57.101 DEBUG:teuthology.orchestra.run.smithi154:> true 2024-09-16T03:31:57.707 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi154.front.sepia.ceph.com' 2024-09-16T03:31:57.707 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.15.0-121-generic"... 2024-09-16T03:31:57.707 DEBUG:teuthology.orchestra.run.smithi154:> uname -r 2024-09-16T03:31:57.752 INFO:teuthology.orchestra.run.smithi154.stdout:5.15.0-121-generic 2024-09-16T03:31:57.753 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-121-generic vs 5.15.0-121-generic 2024-09-16T03:31:57.753 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-16T03:31:57.753 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-09-16T03:31:58.754 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-09-16T03:31:58.754 DEBUG:teuthology.orchestra.run.smithi154:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-16T03:31:58.835 INFO:teuthology.orchestra.run.smithi154.stdout:ttyS1 2024-09-16T03:31:58.871 DEBUG:teuthology.parallel:result is None 2024-09-16T03:31:58.871 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-16T03:31:58.879 INFO:teuthology.task.internal:Creating test directory... 2024-09-16T03:31:58.879 DEBUG:teuthology.orchestra.run.smithi042:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-16T03:31:58.881 DEBUG:teuthology.orchestra.run.smithi154:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-16T03:31:58.887 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-16T03:31:58.906 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-16T03:31:58.956 INFO:teuthology.task.internal:Creating archive directory... 2024-09-16T03:31:58.957 DEBUG:teuthology.orchestra.run.smithi042:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-16T03:31:58.959 DEBUG:teuthology.orchestra.run.smithi154:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-16T03:31:58.976 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-16T03:31:59.006 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-16T03:31:59.007 DEBUG:teuthology.orchestra.run.smithi042:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-16T03:31:59.011 DEBUG:teuthology.orchestra.run.smithi154:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-16T03:31:59.031 INFO:teuthology.orchestra.run.smithi154.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T03:31:59.039 INFO:teuthology.orchestra.run.smithi154.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T03:31:59.041 INFO:teuthology.orchestra.run.smithi042.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T03:31:59.050 INFO:teuthology.orchestra.run.smithi042.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T03:31:59.052 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-16T03:31:59.058 INFO:teuthology.task.internal:Configuring sudo... 2024-09-16T03:31:59.058 DEBUG:teuthology.orchestra.run.smithi042:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-16T03:31:59.095 DEBUG:teuthology.orchestra.run.smithi154:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-16T03:31:59.121 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-16T03:31:59.130 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-16T03:31:59.130 DEBUG:teuthology.orchestra.run.smithi042:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-16T03:31:59.167 DEBUG:teuthology.orchestra.run.smithi154:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-16T03:31:59.173 DEBUG:teuthology.orchestra.run.smithi042:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-16T03:31:59.218 DEBUG:teuthology.orchestra.run.smithi042:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-16T03:31:59.266 DEBUG:teuthology.orchestra.run.smithi042:> set -ex 2024-09-16T03:31:59.267 DEBUG:teuthology.orchestra.run.smithi042:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-16T03:31:59.339 DEBUG:teuthology.orchestra.run.smithi154:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-16T03:31:59.345 DEBUG:teuthology.orchestra.run.smithi154:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-16T03:31:59.393 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-09-16T03:31:59.393 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-16T03:31:59.462 DEBUG:teuthology.orchestra.run.smithi042:> sudo service rsyslog restart 2024-09-16T03:31:59.465 DEBUG:teuthology.orchestra.run.smithi154:> sudo service rsyslog restart 2024-09-16T03:31:59.540 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-16T03:31:59.547 INFO:teuthology.task.internal:Starting timer... 2024-09-16T03:31:59.547 INFO:teuthology.run_tasks:Running task pcp... 2024-09-16T03:31:59.602 INFO:teuthology.run_tasks:Running task selinux... 2024-09-16T03:31:59.651 DEBUG:teuthology.task.selinux:Excluding smithi042: OS 'ubuntu' does not support SELinux 2024-09-16T03:31:59.651 DEBUG:teuthology.task.selinux:Excluding smithi154: OS 'ubuntu' does not support SELinux 2024-09-16T03:31:59.651 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-16T03:31:59.651 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2024-09-16T03:31:59.651 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-16T03:31:59.651 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-16T03:31:59.700 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-16T03:31:59.715 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-16T03:31:59.717 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi042.front.sepia.ceph.com,smithi154.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-16T03:39:39.898 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi042.front.sepia.ceph.com'), Remote(name='ubuntu@smithi154.front.sepia.ceph.com')] 2024-09-16T03:39:39.899 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi042.front.sepia.ceph.com' 2024-09-16T03:39:39.900 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi042.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T03:39:39.989 DEBUG:teuthology.orchestra.run.smithi042:> true 2024-09-16T03:39:40.065 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi042.front.sepia.ceph.com' 2024-09-16T03:39:40.066 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi154.front.sepia.ceph.com' 2024-09-16T03:39:40.066 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi154.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T03:39:40.146 DEBUG:teuthology.orchestra.run.smithi154:> true 2024-09-16T03:39:40.221 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi154.front.sepia.ceph.com' 2024-09-16T03:39:40.221 INFO:teuthology.run_tasks:Running task clock... 2024-09-16T03:39:40.231 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-16T03:39:40.231 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-16T03:39:40.232 DEBUG:teuthology.orchestra.run.smithi042:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T03:39:40.234 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-16T03:39:40.234 DEBUG:teuthology.orchestra.run.smithi154:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T03:39:40.263 INFO:teuthology.orchestra.run.smithi042.stdout:16 Sep 03:39:40 ntpd[18357]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-09-16T03:39:40.264 INFO:teuthology.orchestra.run.smithi042.stdout:16 Sep 03:39:40 ntpd[18357]: Command line: ntpd -gq 2024-09-16T03:39:40.264 INFO:teuthology.orchestra.run.smithi042.stdout:16 Sep 03:39:40 ntpd[18357]: ---------------------------------------------------- 2024-09-16T03:39:40.264 INFO:teuthology.orchestra.run.smithi042.stdout:16 Sep 03:39:40 ntpd[18357]: ntp-4 is maintained by Network Time Foundation, 2024-09-16T03:39:40.264 INFO:teuthology.orchestra.run.smithi042.stdout:16 Sep 03:39:40 ntpd[18357]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-09-16T03:39:40.264 INFO:teuthology.orchestra.run.smithi042.stdout:16 Sep 03:39:40 ntpd[18357]: corporation. Support and training for ntp-4 are 2024-09-16T03:39:40.264 INFO:teuthology.orchestra.run.smithi042.stdout:16 Sep 03:39:40 ntpd[18357]: available at https://www.nwtime.org/support 2024-09-16T03:39:40.264 INFO:teuthology.orchestra.run.smithi042.stdout:16 Sep 03:39:40 ntpd[18357]: ---------------------------------------------------- 2024-09-16T03:39:40.265 INFO:teuthology.orchestra.run.smithi042.stdout:16 Sep 03:39:40 ntpd[18357]: proto: precision = 0.065 usec (-24) 2024-09-16T03:39:40.265 INFO:teuthology.orchestra.run.smithi042.stdout:16 Sep 03:39:40 ntpd[18357]: basedate set to 2022-02-04 2024-09-16T03:39:40.265 INFO:teuthology.orchestra.run.smithi042.stdout:16 Sep 03:39:40 ntpd[18357]: gps base set to 2022-02-06 (week 2196) 2024-09-16T03:39:40.266 INFO:teuthology.orchestra.run.smithi042.stdout:16 Sep 03:39:40 ntpd[18357]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-16T03:39:40.266 INFO:teuthology.orchestra.run.smithi042.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-16T03:39:40.266 INFO:teuthology.orchestra.run.smithi042.stderr:restrict ::: KOD does nothing without LIMITED. 2024-09-16T03:39:40.267 INFO:teuthology.orchestra.run.smithi042.stdout:16 Sep 03:39:40 ntpd[18357]: restrict ::: KOD does nothing without LIMITED. 2024-09-16T03:39:40.267 INFO:teuthology.orchestra.run.smithi042.stdout:16 Sep 03:39:40 ntpd[18357]: Listen and drop on 0 v6wildcard [::]:123 2024-09-16T03:39:40.267 INFO:teuthology.orchestra.run.smithi042.stdout:16 Sep 03:39:40 ntpd[18357]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-09-16T03:39:40.267 INFO:teuthology.orchestra.run.smithi042.stdout:16 Sep 03:39:40 ntpd[18357]: Listen normally on 2 lo 127.0.0.1:123 2024-09-16T03:39:40.268 INFO:teuthology.orchestra.run.smithi042.stdout:16 Sep 03:39:40 ntpd[18357]: Listen normally on 3 ens1f0 172.21.15.42:123 2024-09-16T03:39:40.268 INFO:teuthology.orchestra.run.smithi042.stdout:16 Sep 03:39:40 ntpd[18357]: Listen normally on 4 lo [::1]:123 2024-09-16T03:39:40.268 INFO:teuthology.orchestra.run.smithi042.stdout:16 Sep 03:39:40 ntpd[18357]: Listen normally on 5 ens1f0 [fe80::ec4:7aff:febd:1440%4]:123 2024-09-16T03:39:40.268 INFO:teuthology.orchestra.run.smithi042.stdout:16 Sep 03:39:40 ntpd[18357]: Listening on routing socket on fd #22 for interface updates 2024-09-16T03:39:40.291 INFO:teuthology.orchestra.run.smithi154.stdout:16 Sep 03:39:40 ntpd[18460]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-09-16T03:39:40.291 INFO:teuthology.orchestra.run.smithi154.stdout:16 Sep 03:39:40 ntpd[18460]: Command line: ntpd -gq 2024-09-16T03:39:40.291 INFO:teuthology.orchestra.run.smithi154.stdout:16 Sep 03:39:40 ntpd[18460]: ---------------------------------------------------- 2024-09-16T03:39:40.291 INFO:teuthology.orchestra.run.smithi154.stdout:16 Sep 03:39:40 ntpd[18460]: ntp-4 is maintained by Network Time Foundation, 2024-09-16T03:39:40.291 INFO:teuthology.orchestra.run.smithi154.stdout:16 Sep 03:39:40 ntpd[18460]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-09-16T03:39:40.291 INFO:teuthology.orchestra.run.smithi154.stdout:16 Sep 03:39:40 ntpd[18460]: corporation. Support and training for ntp-4 are 2024-09-16T03:39:40.292 INFO:teuthology.orchestra.run.smithi154.stdout:16 Sep 03:39:40 ntpd[18460]: available at https://www.nwtime.org/support 2024-09-16T03:39:40.292 INFO:teuthology.orchestra.run.smithi154.stdout:16 Sep 03:39:40 ntpd[18460]: ---------------------------------------------------- 2024-09-16T03:39:40.292 INFO:teuthology.orchestra.run.smithi154.stdout:16 Sep 03:39:40 ntpd[18460]: proto: precision = 0.043 usec (-24) 2024-09-16T03:39:40.292 INFO:teuthology.orchestra.run.smithi154.stdout:16 Sep 03:39:40 ntpd[18460]: basedate set to 2022-02-04 2024-09-16T03:39:40.292 INFO:teuthology.orchestra.run.smithi154.stdout:16 Sep 03:39:40 ntpd[18460]: gps base set to 2022-02-06 (week 2196) 2024-09-16T03:39:40.293 INFO:teuthology.orchestra.run.smithi154.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-16T03:39:40.293 INFO:teuthology.orchestra.run.smithi154.stdout:16 Sep 03:39:40 ntpd[18460]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-16T03:39:40.293 INFO:teuthology.orchestra.run.smithi154.stdout:16 Sep 03:39:40 ntpd[18460]: restrict ::: KOD does nothing without LIMITED. 2024-09-16T03:39:40.293 INFO:teuthology.orchestra.run.smithi154.stderr:restrict ::: KOD does nothing without LIMITED. 2024-09-16T03:39:40.294 INFO:teuthology.orchestra.run.smithi154.stdout:16 Sep 03:39:40 ntpd[18460]: Listen and drop on 0 v6wildcard [::]:123 2024-09-16T03:39:40.294 INFO:teuthology.orchestra.run.smithi154.stdout:16 Sep 03:39:40 ntpd[18460]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-09-16T03:39:40.295 INFO:teuthology.orchestra.run.smithi154.stdout:16 Sep 03:39:40 ntpd[18460]: Listen normally on 2 lo 127.0.0.1:123 2024-09-16T03:39:40.295 INFO:teuthology.orchestra.run.smithi154.stdout:16 Sep 03:39:40 ntpd[18460]: Listen normally on 3 enp3s0f1 172.21.15.154:123 2024-09-16T03:39:40.295 INFO:teuthology.orchestra.run.smithi154.stdout:16 Sep 03:39:40 ntpd[18460]: Listen normally on 4 lo [::1]:123 2024-09-16T03:39:40.295 INFO:teuthology.orchestra.run.smithi154.stdout:16 Sep 03:39:40 ntpd[18460]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:7761%5]:123 2024-09-16T03:39:40.295 INFO:teuthology.orchestra.run.smithi154.stdout:16 Sep 03:39:40 ntpd[18460]: Listening on routing socket on fd #22 for interface updates 2024-09-16T03:39:41.265 INFO:teuthology.orchestra.run.smithi042.stderr:16 Sep 03:39:41 ntpd[18357]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:39:41.265 INFO:teuthology.orchestra.run.smithi042.stderr:16 Sep 03:39:41 ntpd[18357]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:39:41.265 INFO:teuthology.orchestra.run.smithi042.stderr:16 Sep 03:39:41 ntpd[18357]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:39:41.265 INFO:teuthology.orchestra.run.smithi042.stderr:16 Sep 03:39:41 ntpd[18357]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:39:41.265 INFO:teuthology.orchestra.run.smithi042.stderr:16 Sep 03:39:41 ntpd[18357]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:39:41.265 INFO:teuthology.orchestra.run.smithi042.stderr:16 Sep 03:39:41 ntpd[18357]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:39:41.265 INFO:teuthology.orchestra.run.smithi042.stderr:16 Sep 03:39:41 ntpd[18357]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T03:39:41.266 INFO:teuthology.orchestra.run.smithi042.stderr:16 Sep 03:39:41 ntpd[18357]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T03:39:41.291 INFO:teuthology.orchestra.run.smithi154.stderr:16 Sep 03:39:41 ntpd[18460]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:39:41.291 INFO:teuthology.orchestra.run.smithi154.stderr:16 Sep 03:39:41 ntpd[18460]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:39:41.292 INFO:teuthology.orchestra.run.smithi154.stderr:16 Sep 03:39:41 ntpd[18460]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:39:41.292 INFO:teuthology.orchestra.run.smithi154.stderr:16 Sep 03:39:41 ntpd[18460]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:39:41.292 INFO:teuthology.orchestra.run.smithi154.stderr:16 Sep 03:39:41 ntpd[18460]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T03:39:41.292 INFO:teuthology.orchestra.run.smithi154.stderr:16 Sep 03:39:41 ntpd[18460]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T03:39:42.292 INFO:teuthology.orchestra.run.smithi154.stderr:16 Sep 03:39:42 ntpd[18460]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:39:42.292 INFO:teuthology.orchestra.run.smithi154.stderr:16 Sep 03:39:42 ntpd[18460]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:39:42.292 INFO:teuthology.orchestra.run.smithi154.stderr:16 Sep 03:39:42 ntpd[18460]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:39:42.292 INFO:teuthology.orchestra.run.smithi154.stderr:16 Sep 03:39:42 ntpd[18460]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:39:42.292 INFO:teuthology.orchestra.run.smithi154.stderr:16 Sep 03:39:42 ntpd[18460]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T03:39:42.292 INFO:teuthology.orchestra.run.smithi154.stderr:16 Sep 03:39:42 ntpd[18460]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T03:39:43.265 INFO:teuthology.orchestra.run.smithi042.stderr:16 Sep 03:39:43 ntpd[18357]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:39:43.265 INFO:teuthology.orchestra.run.smithi042.stderr:16 Sep 03:39:43 ntpd[18357]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:39:43.265 INFO:teuthology.orchestra.run.smithi042.stderr:16 Sep 03:39:43 ntpd[18357]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:39:43.265 INFO:teuthology.orchestra.run.smithi042.stderr:16 Sep 03:39:43 ntpd[18357]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:39:43.265 INFO:teuthology.orchestra.run.smithi042.stderr:16 Sep 03:39:43 ntpd[18357]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T03:39:43.265 INFO:teuthology.orchestra.run.smithi042.stderr:16 Sep 03:39:43 ntpd[18357]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T03:39:43.292 INFO:teuthology.orchestra.run.smithi154.stderr:16 Sep 03:39:43 ntpd[18460]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:39:43.292 INFO:teuthology.orchestra.run.smithi154.stderr:16 Sep 03:39:43 ntpd[18460]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:39:43.292 INFO:teuthology.orchestra.run.smithi154.stderr:16 Sep 03:39:43 ntpd[18460]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:39:43.292 INFO:teuthology.orchestra.run.smithi154.stderr:16 Sep 03:39:43 ntpd[18460]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:39:43.292 INFO:teuthology.orchestra.run.smithi154.stderr:16 Sep 03:39:43 ntpd[18460]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T03:39:43.292 INFO:teuthology.orchestra.run.smithi154.stderr:16 Sep 03:39:43 ntpd[18460]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T03:39:44.265 INFO:teuthology.orchestra.run.smithi042.stderr:16 Sep 03:39:44 ntpd[18357]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:39:44.265 INFO:teuthology.orchestra.run.smithi042.stderr:16 Sep 03:39:44 ntpd[18357]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:39:44.265 INFO:teuthology.orchestra.run.smithi042.stderr:16 Sep 03:39:44 ntpd[18357]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:39:44.266 INFO:teuthology.orchestra.run.smithi042.stderr:16 Sep 03:39:44 ntpd[18357]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:39:44.266 INFO:teuthology.orchestra.run.smithi042.stderr:16 Sep 03:39:44 ntpd[18357]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T03:39:44.266 INFO:teuthology.orchestra.run.smithi042.stderr:16 Sep 03:39:44 ntpd[18357]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T03:39:44.293 INFO:teuthology.orchestra.run.smithi154.stderr:16 Sep 03:39:44 ntpd[18460]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:39:44.293 INFO:teuthology.orchestra.run.smithi154.stderr:16 Sep 03:39:44 ntpd[18460]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:39:44.294 INFO:teuthology.orchestra.run.smithi154.stderr:16 Sep 03:39:44 ntpd[18460]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:39:44.294 INFO:teuthology.orchestra.run.smithi154.stderr:16 Sep 03:39:44 ntpd[18460]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:39:44.294 INFO:teuthology.orchestra.run.smithi154.stderr:16 Sep 03:39:44 ntpd[18460]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:39:44.294 INFO:teuthology.orchestra.run.smithi154.stderr:16 Sep 03:39:44 ntpd[18460]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:39:44.294 INFO:teuthology.orchestra.run.smithi154.stderr:16 Sep 03:39:44 ntpd[18460]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T03:39:44.295 INFO:teuthology.orchestra.run.smithi154.stderr:16 Sep 03:39:44 ntpd[18460]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T03:39:45.265 INFO:teuthology.orchestra.run.smithi042.stderr:16 Sep 03:39:45 ntpd[18357]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:39:45.265 INFO:teuthology.orchestra.run.smithi042.stderr:16 Sep 03:39:45 ntpd[18357]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:39:45.265 INFO:teuthology.orchestra.run.smithi042.stderr:16 Sep 03:39:45 ntpd[18357]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:39:45.265 INFO:teuthology.orchestra.run.smithi042.stderr:16 Sep 03:39:45 ntpd[18357]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:39:45.266 INFO:teuthology.orchestra.run.smithi042.stderr:16 Sep 03:39:45 ntpd[18357]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:39:45.266 INFO:teuthology.orchestra.run.smithi042.stderr:16 Sep 03:39:45 ntpd[18357]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:39:45.266 INFO:teuthology.orchestra.run.smithi042.stderr:16 Sep 03:39:45 ntpd[18357]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T03:39:45.266 INFO:teuthology.orchestra.run.smithi042.stderr:16 Sep 03:39:45 ntpd[18357]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T03:39:45.266 INFO:teuthology.orchestra.run.smithi042.stderr:16 Sep 03:39:45 ntpd[18357]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:39:45.266 INFO:teuthology.orchestra.run.smithi042.stderr:16 Sep 03:39:45 ntpd[18357]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:39:45.266 INFO:teuthology.orchestra.run.smithi042.stderr:16 Sep 03:39:45 ntpd[18357]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T03:39:45.266 INFO:teuthology.orchestra.run.smithi042.stderr:16 Sep 03:39:45 ntpd[18357]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T03:39:45.293 INFO:teuthology.orchestra.run.smithi154.stderr:16 Sep 03:39:45 ntpd[18460]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:39:45.293 INFO:teuthology.orchestra.run.smithi154.stderr:16 Sep 03:39:45 ntpd[18460]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:39:45.293 INFO:teuthology.orchestra.run.smithi154.stderr:16 Sep 03:39:45 ntpd[18460]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:39:45.293 INFO:teuthology.orchestra.run.smithi154.stderr:16 Sep 03:39:45 ntpd[18460]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:39:45.293 INFO:teuthology.orchestra.run.smithi154.stderr:16 Sep 03:39:45 ntpd[18460]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:39:45.293 INFO:teuthology.orchestra.run.smithi154.stderr:16 Sep 03:39:45 ntpd[18460]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:39:45.293 INFO:teuthology.orchestra.run.smithi154.stderr:16 Sep 03:39:45 ntpd[18460]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T03:39:45.293 INFO:teuthology.orchestra.run.smithi154.stderr:16 Sep 03:39:45 ntpd[18460]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T03:39:45.293 INFO:teuthology.orchestra.run.smithi154.stderr:16 Sep 03:39:45 ntpd[18460]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:39:45.294 INFO:teuthology.orchestra.run.smithi154.stderr:16 Sep 03:39:45 ntpd[18460]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:39:45.294 INFO:teuthology.orchestra.run.smithi154.stderr:16 Sep 03:39:45 ntpd[18460]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T03:39:45.294 INFO:teuthology.orchestra.run.smithi154.stderr:16 Sep 03:39:45 ntpd[18460]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T03:39:46.264 INFO:teuthology.orchestra.run.smithi042.stderr:16 Sep 03:39:46 ntpd[18357]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:39:46.265 INFO:teuthology.orchestra.run.smithi042.stderr:16 Sep 03:39:46 ntpd[18357]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:39:46.265 INFO:teuthology.orchestra.run.smithi042.stderr:16 Sep 03:39:46 ntpd[18357]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:39:46.265 INFO:teuthology.orchestra.run.smithi042.stderr:16 Sep 03:39:46 ntpd[18357]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:39:46.265 INFO:teuthology.orchestra.run.smithi042.stderr:16 Sep 03:39:46 ntpd[18357]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T03:39:46.265 INFO:teuthology.orchestra.run.smithi042.stderr:16 Sep 03:39:46 ntpd[18357]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T03:39:46.292 INFO:teuthology.orchestra.run.smithi154.stderr:16 Sep 03:39:46 ntpd[18460]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:39:46.292 INFO:teuthology.orchestra.run.smithi154.stderr:16 Sep 03:39:46 ntpd[18460]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:39:46.292 INFO:teuthology.orchestra.run.smithi154.stderr:16 Sep 03:39:46 ntpd[18460]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:39:46.292 INFO:teuthology.orchestra.run.smithi154.stderr:16 Sep 03:39:46 ntpd[18460]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:39:46.292 INFO:teuthology.orchestra.run.smithi154.stderr:16 Sep 03:39:46 ntpd[18460]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T03:39:46.292 INFO:teuthology.orchestra.run.smithi154.stderr:16 Sep 03:39:46 ntpd[18460]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T03:39:47.265 INFO:teuthology.orchestra.run.smithi042.stderr:16 Sep 03:39:47 ntpd[18357]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:39:47.265 INFO:teuthology.orchestra.run.smithi042.stderr:16 Sep 03:39:47 ntpd[18357]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:39:47.265 INFO:teuthology.orchestra.run.smithi042.stderr:16 Sep 03:39:47 ntpd[18357]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:39:47.265 INFO:teuthology.orchestra.run.smithi042.stderr:16 Sep 03:39:47 ntpd[18357]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:39:47.265 INFO:teuthology.orchestra.run.smithi042.stderr:16 Sep 03:39:47 ntpd[18357]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:39:47.265 INFO:teuthology.orchestra.run.smithi042.stderr:16 Sep 03:39:47 ntpd[18357]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:39:47.265 INFO:teuthology.orchestra.run.smithi042.stderr:16 Sep 03:39:47 ntpd[18357]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T03:39:47.265 INFO:teuthology.orchestra.run.smithi042.stderr:16 Sep 03:39:47 ntpd[18357]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T03:39:47.265 INFO:teuthology.orchestra.run.smithi042.stdout:16 Sep 03:39:47 ntpd[18357]: ntpd: time slew -0.001325 s 2024-09-16T03:39:47.265 INFO:teuthology.orchestra.run.smithi042.stdout:ntpd: time slew -0.001325s 2024-09-16T03:39:47.266 INFO:teuthology.orchestra.run.smithi042.stderr:16 Sep 03:39:47 ntpd[18357]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-09-16T03:39:47.266 INFO:teuthology.orchestra.run.smithi042.stderr:16 Sep 03:39:47 ntpd[18357]: can't open /var/log/ntpstats/loopstats.20240916: Permission denied 2024-09-16T03:39:47.293 INFO:teuthology.orchestra.run.smithi154.stderr:16 Sep 03:39:47 ntpd[18460]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:39:47.294 INFO:teuthology.orchestra.run.smithi154.stderr:16 Sep 03:39:47 ntpd[18460]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:39:47.294 INFO:teuthology.orchestra.run.smithi154.stderr:16 Sep 03:39:47 ntpd[18460]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:39:47.294 INFO:teuthology.orchestra.run.smithi154.stderr:16 Sep 03:39:47 ntpd[18460]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:39:47.294 INFO:teuthology.orchestra.run.smithi154.stderr:16 Sep 03:39:47 ntpd[18460]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:39:47.294 INFO:teuthology.orchestra.run.smithi154.stderr:16 Sep 03:39:47 ntpd[18460]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:39:47.295 INFO:teuthology.orchestra.run.smithi154.stderr:16 Sep 03:39:47 ntpd[18460]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T03:39:47.295 INFO:teuthology.orchestra.run.smithi154.stderr:16 Sep 03:39:47 ntpd[18460]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T03:39:47.295 INFO:teuthology.orchestra.run.smithi154.stderr:16 Sep 03:39:47 ntpd[18460]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T03:39:47.295 INFO:teuthology.orchestra.run.smithi154.stderr:16 Sep 03:39:47 ntpd[18460]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T03:39:47.295 INFO:teuthology.orchestra.run.smithi154.stderr:16 Sep 03:39:47 ntpd[18460]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T03:39:47.296 INFO:teuthology.orchestra.run.smithi154.stderr:16 Sep 03:39:47 ntpd[18460]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T03:39:47.296 INFO:teuthology.orchestra.run.smithi154.stdout:16 Sep 03:39:47 ntpd[18460]: ntpd: time slew -0.000718 s 2024-09-16T03:39:47.296 INFO:teuthology.orchestra.run.smithi154.stdout:ntpd: time slew -0.000718s 2024-09-16T03:39:47.297 INFO:teuthology.orchestra.run.smithi154.stderr:16 Sep 03:39:47 ntpd[18460]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-09-16T03:39:47.297 INFO:teuthology.orchestra.run.smithi154.stderr:16 Sep 03:39:47 ntpd[18460]: can't open /var/log/ntpstats/loopstats.20240916: Permission denied 2024-09-16T03:39:47.316 INFO:teuthology.orchestra.run.smithi042.stdout: remote refid st t when poll reach delay offset jitter 2024-09-16T03:39:47.316 INFO:teuthology.orchestra.run.smithi042.stdout:============================================================================== 2024-09-16T03:39:47.316 INFO:teuthology.orchestra.run.smithi042.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-16T03:39:47.317 INFO:teuthology.orchestra.run.smithi042.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-16T03:39:47.317 INFO:teuthology.orchestra.run.smithi042.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-16T03:39:47.317 INFO:teuthology.orchestra.run.smithi042.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-16T03:39:47.340 INFO:teuthology.orchestra.run.smithi154.stdout: remote refid st t when poll reach delay offset jitter 2024-09-16T03:39:47.341 INFO:teuthology.orchestra.run.smithi154.stdout:============================================================================== 2024-09-16T03:39:47.341 INFO:teuthology.orchestra.run.smithi154.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-16T03:39:47.341 INFO:teuthology.orchestra.run.smithi154.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-16T03:39:47.341 INFO:teuthology.orchestra.run.smithi154.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-16T03:39:47.341 INFO:teuthology.orchestra.run.smithi154.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-16T03:39:47.342 INFO:teuthology.run_tasks:Running task install... 2024-09-16T03:39:47.354 DEBUG:teuthology.task.install:project ceph 2024-09-16T03:39:47.354 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '8921c8ef947807b717db041d85f0761b7914d5fc'}} 2024-09-16T03:39:47.354 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '8921c8ef947807b717db041d85f0761b7914d5fc'} 2024-09-16T03:39:47.354 INFO:teuthology.task.install:Using flavor: default 2024-09-16T03:39:47.359 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-16T03:39:47.359 INFO:teuthology.task.install:extra packages: [] 2024-09-16T03:39:47.360 DEBUG:teuthology.orchestra.run.smithi042:> sudo apt-key list | grep Ceph 2024-09-16T03:39:47.360 DEBUG:teuthology.orchestra.run.smithi154:> sudo apt-key list | grep Ceph 2024-09-16T03:39:47.423 INFO:teuthology.orchestra.run.smithi042.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-09-16T03:39:47.438 INFO:teuthology.orchestra.run.smithi154.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-09-16T03:39:47.452 INFO:teuthology.orchestra.run.smithi042.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-09-16T03:39:47.452 INFO:teuthology.orchestra.run.smithi042.stdout:uid [ unknown] Ceph.com (release key) 2024-09-16T03:39:47.453 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-09-16T03:39:47.453 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=8921c8ef947807b717db041d85f0761b7914d5fc 2024-09-16T03:39:47.464 INFO:teuthology.orchestra.run.smithi154.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-09-16T03:39:47.464 INFO:teuthology.orchestra.run.smithi154.stdout:uid [ unknown] Ceph.com (release key) 2024-09-16T03:39:47.465 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-09-16T03:39:47.465 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=8921c8ef947807b717db041d85f0761b7914d5fc 2024-09-16T03:39:47.604 INFO:teuthology.task.install.deb:Pulling from https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default/ 2024-09-16T03:39:47.604 INFO:teuthology.task.install.deb:Package version is 19.1.1-326-g8921c8ef-1jammy 2024-09-16T03:39:47.608 INFO:teuthology.task.install.deb:Pulling from https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default/ 2024-09-16T03:39:47.608 INFO:teuthology.task.install.deb:Package version is 19.1.1-326-g8921c8ef-1jammy 2024-09-16T03:39:47.718 DEBUG:teuthology.orchestra.run.smithi042:> set -ex 2024-09-16T03:39:47.718 DEBUG:teuthology.orchestra.run.smithi042:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-09-16T03:39:47.724 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-09-16T03:39:47.724 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-09-16T03:39:47.733 DEBUG:teuthology.orchestra.run.smithi042:> sudo apt-get update 2024-09-16T03:39:47.738 DEBUG:teuthology.orchestra.run.smithi154:> sudo apt-get update 2024-09-16T03:39:47.922 INFO:teuthology.orchestra.run.smithi042.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-16T03:39:47.931 INFO:teuthology.orchestra.run.smithi042.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-16T03:39:47.948 INFO:teuthology.orchestra.run.smithi042.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-16T03:39:47.958 INFO:teuthology.orchestra.run.smithi042.stdout:Ign:4 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy InRelease 2024-09-16T03:39:47.959 INFO:teuthology.orchestra.run.smithi154.stdout:Ign:1 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy InRelease 2024-09-16T03:39:47.980 INFO:teuthology.orchestra.run.smithi042.stdout:Get:5 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-09-16T03:39:47.980 INFO:teuthology.orchestra.run.smithi154.stdout:Get:2 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-09-16T03:39:48.002 INFO:teuthology.orchestra.run.smithi042.stdout:Ign:6 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy Release.gpg 2024-09-16T03:39:48.003 INFO:teuthology.orchestra.run.smithi154.stdout:Ign:3 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy Release.gpg 2024-09-16T03:39:48.023 INFO:teuthology.orchestra.run.smithi042.stdout:Get:7 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,422 B] 2024-09-16T03:39:48.030 INFO:teuthology.orchestra.run.smithi042.stdout:Hit:8 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-16T03:39:48.035 INFO:teuthology.orchestra.run.smithi154.stdout:Hit:4 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-16T03:39:48.035 INFO:teuthology.orchestra.run.smithi154.stdout:Get:5 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.9 kB] 2024-09-16T03:39:48.043 INFO:teuthology.orchestra.run.smithi154.stdout:Hit:6 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-16T03:39:48.052 INFO:teuthology.orchestra.run.smithi042.stdout:Get:9 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.9 kB] 2024-09-16T03:39:48.065 INFO:teuthology.orchestra.run.smithi154.stdout:Get:7 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,422 B] 2024-09-16T03:39:48.125 INFO:teuthology.orchestra.run.smithi154.stdout:Hit:8 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-16T03:39:48.206 INFO:teuthology.orchestra.run.smithi154.stdout:Hit:9 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-16T03:39:48.314 INFO:teuthology.orchestra.run.smithi042.stdout:Fetched 29.0 kB in 0s (73.5 kB/s) 2024-09-16T03:39:48.405 INFO:teuthology.orchestra.run.smithi154.stdout:Fetched 29.0 kB in 0s (59.3 kB/s) 2024-09-16T03:39:49.348 INFO:teuthology.orchestra.run.smithi042.stdout:Reading package lists... 2024-09-16T03:39:49.366 DEBUG:teuthology.orchestra.run.smithi042:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.1.1-326-g8921c8ef-1jammy cephadm=19.1.1-326-g8921c8ef-1jammy ceph-mds=19.1.1-326-g8921c8ef-1jammy ceph-mgr=19.1.1-326-g8921c8ef-1jammy ceph-common=19.1.1-326-g8921c8ef-1jammy ceph-fuse=19.1.1-326-g8921c8ef-1jammy ceph-test=19.1.1-326-g8921c8ef-1jammy ceph-volume=19.1.1-326-g8921c8ef-1jammy radosgw=19.1.1-326-g8921c8ef-1jammy python3-rados=19.1.1-326-g8921c8ef-1jammy python3-rgw=19.1.1-326-g8921c8ef-1jammy python3-cephfs=19.1.1-326-g8921c8ef-1jammy python3-rbd=19.1.1-326-g8921c8ef-1jammy libcephfs2=19.1.1-326-g8921c8ef-1jammy libcephfs-dev=19.1.1-326-g8921c8ef-1jammy librados2=19.1.1-326-g8921c8ef-1jammy librbd1=19.1.1-326-g8921c8ef-1jammy rbd-fuse=19.1.1-326-g8921c8ef-1jammy 2024-09-16T03:39:49.385 INFO:teuthology.orchestra.run.smithi154.stdout:Reading package lists... 2024-09-16T03:39:49.402 DEBUG:teuthology.orchestra.run.smithi154:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.1.1-326-g8921c8ef-1jammy cephadm=19.1.1-326-g8921c8ef-1jammy ceph-mds=19.1.1-326-g8921c8ef-1jammy ceph-mgr=19.1.1-326-g8921c8ef-1jammy ceph-common=19.1.1-326-g8921c8ef-1jammy ceph-fuse=19.1.1-326-g8921c8ef-1jammy ceph-test=19.1.1-326-g8921c8ef-1jammy ceph-volume=19.1.1-326-g8921c8ef-1jammy radosgw=19.1.1-326-g8921c8ef-1jammy python3-rados=19.1.1-326-g8921c8ef-1jammy python3-rgw=19.1.1-326-g8921c8ef-1jammy python3-cephfs=19.1.1-326-g8921c8ef-1jammy python3-rbd=19.1.1-326-g8921c8ef-1jammy libcephfs2=19.1.1-326-g8921c8ef-1jammy libcephfs-dev=19.1.1-326-g8921c8ef-1jammy librados2=19.1.1-326-g8921c8ef-1jammy librbd1=19.1.1-326-g8921c8ef-1jammy rbd-fuse=19.1.1-326-g8921c8ef-1jammy 2024-09-16T03:39:49.432 INFO:teuthology.orchestra.run.smithi042.stdout:Reading package lists... 2024-09-16T03:39:49.464 INFO:teuthology.orchestra.run.smithi154.stdout:Reading package lists... 2024-09-16T03:39:49.644 INFO:teuthology.orchestra.run.smithi042.stdout:Building dependency tree... 2024-09-16T03:39:49.645 INFO:teuthology.orchestra.run.smithi042.stdout:Reading state information... 2024-09-16T03:39:49.672 INFO:teuthology.orchestra.run.smithi154.stdout:Building dependency tree... 2024-09-16T03:39:49.673 INFO:teuthology.orchestra.run.smithi154.stdout:Reading state information... 2024-09-16T03:39:49.861 INFO:teuthology.orchestra.run.smithi042.stdout:The following packages were automatically installed and are no longer required: 2024-09-16T03:39:49.861 INFO:teuthology.orchestra.run.smithi042.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-16T03:39:49.861 INFO:teuthology.orchestra.run.smithi042.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-16T03:39:49.861 INFO:teuthology.orchestra.run.smithi042.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-16T03:39:49.862 INFO:teuthology.orchestra.run.smithi042.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-16T03:39:49.862 INFO:teuthology.orchestra.run.smithi042.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-16T03:39:49.862 INFO:teuthology.orchestra.run.smithi042.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-16T03:39:49.862 INFO:teuthology.orchestra.run.smithi042.stdout:The following additional packages will be installed: 2024-09-16T03:39:49.862 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-09-16T03:39:49.863 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-09-16T03:39:49.863 INFO:teuthology.orchestra.run.smithi042.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-09-16T03:39:49.863 INFO:teuthology.orchestra.run.smithi042.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-09-16T03:39:49.864 INFO:teuthology.orchestra.run.smithi042.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-09-16T03:39:49.864 INFO:teuthology.orchestra.run.smithi042.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-09-16T03:39:49.865 INFO:teuthology.orchestra.run.smithi042.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-09-16T03:39:49.865 INFO:teuthology.orchestra.run.smithi042.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-09-16T03:39:49.865 INFO:teuthology.orchestra.run.smithi042.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-09-16T03:39:49.865 INFO:teuthology.orchestra.run.smithi042.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-09-16T03:39:49.865 INFO:teuthology.orchestra.run.smithi042.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-09-16T03:39:49.865 INFO:teuthology.orchestra.run.smithi042.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-09-16T03:39:49.865 INFO:teuthology.orchestra.run.smithi042.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-09-16T03:39:49.865 INFO:teuthology.orchestra.run.smithi042.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-09-16T03:39:49.865 INFO:teuthology.orchestra.run.smithi042.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-09-16T03:39:49.865 INFO:teuthology.orchestra.run.smithi042.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-09-16T03:39:49.865 INFO:teuthology.orchestra.run.smithi042.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2024-09-16T03:39:49.865 INFO:teuthology.orchestra.run.smithi042.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2024-09-16T03:39:49.865 INFO:teuthology.orchestra.run.smithi042.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-09-16T03:39:49.866 INFO:teuthology.orchestra.run.smithi042.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-09-16T03:39:49.866 INFO:teuthology.orchestra.run.smithi042.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-09-16T03:39:49.866 INFO:teuthology.orchestra.run.smithi042.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-09-16T03:39:49.866 INFO:teuthology.orchestra.run.smithi042.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-09-16T03:39:49.866 INFO:teuthology.orchestra.run.smithi042.stdout: xmlstarlet zip 2024-09-16T03:39:49.866 INFO:teuthology.orchestra.run.smithi042.stdout:Suggested packages: 2024-09-16T03:39:49.866 INFO:teuthology.orchestra.run.smithi042.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-09-16T03:39:49.866 INFO:teuthology.orchestra.run.smithi042.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-09-16T03:39:49.866 INFO:teuthology.orchestra.run.smithi042.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-09-16T03:39:49.867 INFO:teuthology.orchestra.run.smithi042.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-09-16T03:39:49.867 INFO:teuthology.orchestra.run.smithi042.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-09-16T03:39:49.867 INFO:teuthology.orchestra.run.smithi042.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-09-16T03:39:49.867 INFO:teuthology.orchestra.run.smithi042.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-09-16T03:39:49.867 INFO:teuthology.orchestra.run.smithi042.stdout:Recommended packages: 2024-09-16T03:39:49.867 INFO:teuthology.orchestra.run.smithi042.stdout: btrfs-tools 2024-09-16T03:39:49.868 INFO:teuthology.orchestra.run.smithi154.stdout:The following packages were automatically installed and are no longer required: 2024-09-16T03:39:49.868 INFO:teuthology.orchestra.run.smithi154.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-16T03:39:49.869 INFO:teuthology.orchestra.run.smithi154.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-16T03:39:49.869 INFO:teuthology.orchestra.run.smithi154.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-16T03:39:49.869 INFO:teuthology.orchestra.run.smithi154.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-16T03:39:49.869 INFO:teuthology.orchestra.run.smithi154.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-16T03:39:49.869 INFO:teuthology.orchestra.run.smithi154.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-16T03:39:49.870 INFO:teuthology.orchestra.run.smithi154.stdout:The following additional packages will be installed: 2024-09-16T03:39:49.870 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-09-16T03:39:49.870 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-09-16T03:39:49.871 INFO:teuthology.orchestra.run.smithi154.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-09-16T03:39:49.871 INFO:teuthology.orchestra.run.smithi154.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-09-16T03:39:49.872 INFO:teuthology.orchestra.run.smithi154.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-09-16T03:39:49.872 INFO:teuthology.orchestra.run.smithi154.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-09-16T03:39:49.872 INFO:teuthology.orchestra.run.smithi154.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-09-16T03:39:49.872 INFO:teuthology.orchestra.run.smithi154.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-09-16T03:39:49.872 INFO:teuthology.orchestra.run.smithi154.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-09-16T03:39:49.873 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-09-16T03:39:49.873 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-09-16T03:39:49.873 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-09-16T03:39:49.873 INFO:teuthology.orchestra.run.smithi154.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-09-16T03:39:49.873 INFO:teuthology.orchestra.run.smithi154.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-09-16T03:39:49.873 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-09-16T03:39:49.873 INFO:teuthology.orchestra.run.smithi154.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-09-16T03:39:49.873 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2024-09-16T03:39:49.873 INFO:teuthology.orchestra.run.smithi154.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2024-09-16T03:39:49.873 INFO:teuthology.orchestra.run.smithi154.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-09-16T03:39:49.873 INFO:teuthology.orchestra.run.smithi154.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-09-16T03:39:49.874 INFO:teuthology.orchestra.run.smithi154.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-09-16T03:39:49.874 INFO:teuthology.orchestra.run.smithi154.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-09-16T03:39:49.874 INFO:teuthology.orchestra.run.smithi154.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-09-16T03:39:49.874 INFO:teuthology.orchestra.run.smithi154.stdout: xmlstarlet zip 2024-09-16T03:39:49.874 INFO:teuthology.orchestra.run.smithi154.stdout:Suggested packages: 2024-09-16T03:39:49.874 INFO:teuthology.orchestra.run.smithi154.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-09-16T03:39:49.875 INFO:teuthology.orchestra.run.smithi154.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-09-16T03:39:49.875 INFO:teuthology.orchestra.run.smithi154.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-09-16T03:39:49.875 INFO:teuthology.orchestra.run.smithi154.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-09-16T03:39:49.875 INFO:teuthology.orchestra.run.smithi154.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-09-16T03:39:49.875 INFO:teuthology.orchestra.run.smithi154.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-09-16T03:39:49.876 INFO:teuthology.orchestra.run.smithi154.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-09-16T03:39:49.876 INFO:teuthology.orchestra.run.smithi154.stdout:Recommended packages: 2024-09-16T03:39:49.876 INFO:teuthology.orchestra.run.smithi154.stdout: btrfs-tools 2024-09-16T03:39:49.977 INFO:teuthology.orchestra.run.smithi154.stdout:The following NEW packages will be installed: 2024-09-16T03:39:49.977 INFO:teuthology.orchestra.run.smithi154.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-09-16T03:39:49.977 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-09-16T03:39:49.977 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-09-16T03:39:49.977 INFO:teuthology.orchestra.run.smithi154.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-09-16T03:39:49.978 INFO:teuthology.orchestra.run.smithi154.stdout: liblua5.3-dev liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-09-16T03:39:49.978 INFO:teuthology.orchestra.run.smithi154.stdout: libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-09-16T03:39:49.978 INFO:teuthology.orchestra.run.smithi154.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-09-16T03:39:49.978 INFO:teuthology.orchestra.run.smithi154.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-09-16T03:39:49.978 INFO:teuthology.orchestra.run.smithi154.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-09-16T03:39:49.978 INFO:teuthology.orchestra.run.smithi154.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-09-16T03:39:49.978 INFO:teuthology.orchestra.run.smithi154.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-09-16T03:39:49.978 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-09-16T03:39:49.978 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-09-16T03:39:49.978 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-09-16T03:39:49.978 INFO:teuthology.orchestra.run.smithi154.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-09-16T03:39:49.978 INFO:teuthology.orchestra.run.smithi154.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-09-16T03:39:49.978 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-09-16T03:39:49.978 INFO:teuthology.orchestra.run.smithi154.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-09-16T03:39:49.979 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-09-16T03:39:49.979 INFO:teuthology.orchestra.run.smithi154.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-09-16T03:39:49.979 INFO:teuthology.orchestra.run.smithi154.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-09-16T03:39:49.979 INFO:teuthology.orchestra.run.smithi154.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-09-16T03:39:49.979 INFO:teuthology.orchestra.run.smithi154.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-09-16T03:39:49.979 INFO:teuthology.orchestra.run.smithi154.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-09-16T03:39:49.979 INFO:teuthology.orchestra.run.smithi154.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-09-16T03:39:49.979 INFO:teuthology.orchestra.run.smithi154.stdout: radosgw rbd-fuse socat unzip xmlstarlet zip 2024-09-16T03:39:49.979 INFO:teuthology.orchestra.run.smithi042.stdout:The following NEW packages will be installed: 2024-09-16T03:39:49.979 INFO:teuthology.orchestra.run.smithi042.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-09-16T03:39:49.979 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-09-16T03:39:49.979 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-09-16T03:39:49.980 INFO:teuthology.orchestra.run.smithi042.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-09-16T03:39:49.980 INFO:teuthology.orchestra.run.smithi042.stdout: liblua5.3-dev liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-09-16T03:39:49.980 INFO:teuthology.orchestra.run.smithi042.stdout: libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-09-16T03:39:49.980 INFO:teuthology.orchestra.run.smithi042.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-09-16T03:39:49.980 INFO:teuthology.orchestra.run.smithi042.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-09-16T03:39:49.980 INFO:teuthology.orchestra.run.smithi042.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-09-16T03:39:49.980 INFO:teuthology.orchestra.run.smithi042.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-09-16T03:39:49.980 INFO:teuthology.orchestra.run.smithi042.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-09-16T03:39:49.980 INFO:teuthology.orchestra.run.smithi042.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-09-16T03:39:49.980 INFO:teuthology.orchestra.run.smithi042.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-09-16T03:39:49.980 INFO:teuthology.orchestra.run.smithi042.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-09-16T03:39:49.981 INFO:teuthology.orchestra.run.smithi042.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-09-16T03:39:49.981 INFO:teuthology.orchestra.run.smithi042.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-09-16T03:39:49.981 INFO:teuthology.orchestra.run.smithi042.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-09-16T03:39:49.981 INFO:teuthology.orchestra.run.smithi042.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-09-16T03:39:49.981 INFO:teuthology.orchestra.run.smithi042.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-09-16T03:39:49.981 INFO:teuthology.orchestra.run.smithi042.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-09-16T03:39:49.981 INFO:teuthology.orchestra.run.smithi042.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-09-16T03:39:49.981 INFO:teuthology.orchestra.run.smithi042.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-09-16T03:39:49.981 INFO:teuthology.orchestra.run.smithi042.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-09-16T03:39:49.981 INFO:teuthology.orchestra.run.smithi042.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-09-16T03:39:49.981 INFO:teuthology.orchestra.run.smithi042.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-09-16T03:39:49.981 INFO:teuthology.orchestra.run.smithi042.stdout: radosgw rbd-fuse socat unzip xmlstarlet zip 2024-09-16T03:39:50.120 INFO:teuthology.orchestra.run.smithi042.stdout:0 upgraded, 118 newly installed, 0 to remove and 331 not upgraded. 2024-09-16T03:39:50.120 INFO:teuthology.orchestra.run.smithi042.stdout:Need to get 181 MB of archives. 2024-09-16T03:39:50.120 INFO:teuthology.orchestra.run.smithi042.stdout:After this operation, 829 MB of additional disk space will be used. 2024-09-16T03:39:50.120 INFO:teuthology.orchestra.run.smithi042.stdout:Get:1 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.1.1-326-g8921c8ef-1jammy [3,590 kB] 2024-09-16T03:39:50.125 INFO:teuthology.orchestra.run.smithi154.stdout:0 upgraded, 118 newly installed, 0 to remove and 331 not upgraded. 2024-09-16T03:39:50.125 INFO:teuthology.orchestra.run.smithi154.stdout:Need to get 181 MB of archives. 2024-09-16T03:39:50.125 INFO:teuthology.orchestra.run.smithi154.stdout:After this operation, 829 MB of additional disk space will be used. 2024-09-16T03:39:50.125 INFO:teuthology.orchestra.run.smithi154.stdout:Get:1 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.1.1-326-g8921c8ef-1jammy [3,590 kB] 2024-09-16T03:39:50.164 INFO:teuthology.orchestra.run.smithi042.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-09-16T03:39:50.167 INFO:teuthology.orchestra.run.smithi154.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-09-16T03:39:50.318 INFO:teuthology.orchestra.run.smithi042.stdout:Get:3 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.1.1-326-g8921c8ef-1jammy [3,241 kB] 2024-09-16T03:39:50.335 INFO:teuthology.orchestra.run.smithi154.stdout:Get:3 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.1.1-326-g8921c8ef-1jammy [3,241 kB] 2024-09-16T03:39:50.393 INFO:teuthology.orchestra.run.smithi042.stdout:Get:4 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.1.1-326-g8921c8ef-1jammy [719 kB] 2024-09-16T03:39:50.420 INFO:teuthology.orchestra.run.smithi154.stdout:Get:4 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.1.1-326-g8921c8ef-1jammy [719 kB] 2024-09-16T03:39:50.429 INFO:teuthology.orchestra.run.smithi042.stdout:Get:5 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.1.1-326-g8921c8ef-1jammy [356 kB] 2024-09-16T03:39:50.436 INFO:teuthology.orchestra.run.smithi042.stdout:Get:6 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.1.1-326-g8921c8ef-1jammy [32.8 kB] 2024-09-16T03:39:50.436 INFO:teuthology.orchestra.run.smithi042.stdout:Get:7 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.1.1-326-g8921c8ef-1jammy [180 kB] 2024-09-16T03:39:50.440 INFO:teuthology.orchestra.run.smithi042.stdout:Get:8 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.1.1-326-g8921c8ef-1jammy [67.8 kB] 2024-09-16T03:39:50.441 INFO:teuthology.orchestra.run.smithi042.stdout:Get:9 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.1.1-326-g8921c8ef-1jammy [333 kB] 2024-09-16T03:39:50.447 INFO:teuthology.orchestra.run.smithi042.stdout:Get:10 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.1.1-326-g8921c8ef-1jammy [6,875 kB] 2024-09-16T03:39:50.458 INFO:teuthology.orchestra.run.smithi154.stdout:Get:5 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.1.1-326-g8921c8ef-1jammy [356 kB] 2024-09-16T03:39:50.465 INFO:teuthology.orchestra.run.smithi154.stdout:Get:6 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.1.1-326-g8921c8ef-1jammy [32.8 kB] 2024-09-16T03:39:50.466 INFO:teuthology.orchestra.run.smithi154.stdout:Get:7 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.1.1-326-g8921c8ef-1jammy [180 kB] 2024-09-16T03:39:50.470 INFO:teuthology.orchestra.run.smithi154.stdout:Get:8 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.1.1-326-g8921c8ef-1jammy [67.8 kB] 2024-09-16T03:39:50.472 INFO:teuthology.orchestra.run.smithi154.stdout:Get:9 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.1.1-326-g8921c8ef-1jammy [333 kB] 2024-09-16T03:39:50.478 INFO:teuthology.orchestra.run.smithi154.stdout:Get:10 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.1.1-326-g8921c8ef-1jammy [6,875 kB] 2024-09-16T03:39:50.584 INFO:teuthology.orchestra.run.smithi042.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-09-16T03:39:50.587 INFO:teuthology.orchestra.run.smithi154.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-09-16T03:39:50.597 INFO:teuthology.orchestra.run.smithi042.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-09-16T03:39:50.600 INFO:teuthology.orchestra.run.smithi154.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-09-16T03:39:50.678 INFO:teuthology.orchestra.run.smithi042.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-09-16T03:39:50.681 INFO:teuthology.orchestra.run.smithi154.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-09-16T03:39:50.687 INFO:teuthology.orchestra.run.smithi042.stdout:Get:14 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.1.1-326-g8921c8ef-1jammy [112 kB] 2024-09-16T03:39:50.690 INFO:teuthology.orchestra.run.smithi042.stdout:Get:15 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.1.1-326-g8921c8ef-1jammy [472 kB] 2024-09-16T03:39:50.699 INFO:teuthology.orchestra.run.smithi042.stdout:Get:16 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.1.1-326-g8921c8ef-1jammy [25.9 MB] 2024-09-16T03:39:50.721 INFO:teuthology.orchestra.run.smithi154.stdout:Get:14 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.1.1-326-g8921c8ef-1jammy [112 kB] 2024-09-16T03:39:50.724 INFO:teuthology.orchestra.run.smithi154.stdout:Get:15 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.1.1-326-g8921c8ef-1jammy [472 kB] 2024-09-16T03:39:50.733 INFO:teuthology.orchestra.run.smithi154.stdout:Get:16 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.1.1-326-g8921c8ef-1jammy [25.9 MB] 2024-09-16T03:39:50.907 INFO:teuthology.orchestra.run.smithi042.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-09-16T03:39:50.909 INFO:teuthology.orchestra.run.smithi154.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-09-16T03:39:50.917 INFO:teuthology.orchestra.run.smithi042.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-09-16T03:39:50.920 INFO:teuthology.orchestra.run.smithi154.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-09-16T03:39:50.947 INFO:teuthology.orchestra.run.smithi042.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-09-16T03:39:50.950 INFO:teuthology.orchestra.run.smithi154.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-09-16T03:39:50.956 INFO:teuthology.orchestra.run.smithi042.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-09-16T03:39:50.957 INFO:teuthology.orchestra.run.smithi042.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-09-16T03:39:50.958 INFO:teuthology.orchestra.run.smithi042.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-09-16T03:39:50.958 INFO:teuthology.orchestra.run.smithi154.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-09-16T03:39:50.959 INFO:teuthology.orchestra.run.smithi154.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-09-16T03:39:50.961 INFO:teuthology.orchestra.run.smithi154.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-09-16T03:39:50.976 INFO:teuthology.orchestra.run.smithi042.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-09-16T03:39:50.978 INFO:teuthology.orchestra.run.smithi154.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-09-16T03:39:50.980 INFO:teuthology.orchestra.run.smithi042.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-09-16T03:39:50.983 INFO:teuthology.orchestra.run.smithi154.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-09-16T03:39:50.985 INFO:teuthology.orchestra.run.smithi042.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-09-16T03:39:50.987 INFO:teuthology.orchestra.run.smithi154.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-09-16T03:39:50.991 INFO:teuthology.orchestra.run.smithi042.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-09-16T03:39:50.992 INFO:teuthology.orchestra.run.smithi154.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-09-16T03:39:51.069 INFO:teuthology.orchestra.run.smithi042.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-09-16T03:39:51.074 INFO:teuthology.orchestra.run.smithi042.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-09-16T03:39:51.074 INFO:teuthology.orchestra.run.smithi154.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-09-16T03:39:51.078 INFO:teuthology.orchestra.run.smithi154.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-09-16T03:39:51.078 INFO:teuthology.orchestra.run.smithi042.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-09-16T03:39:51.082 INFO:teuthology.orchestra.run.smithi042.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-09-16T03:39:51.082 INFO:teuthology.orchestra.run.smithi154.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-09-16T03:39:51.083 INFO:teuthology.orchestra.run.smithi042.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-09-16T03:39:51.084 INFO:teuthology.orchestra.run.smithi042.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-09-16T03:39:51.086 INFO:teuthology.orchestra.run.smithi042.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-09-16T03:39:51.086 INFO:teuthology.orchestra.run.smithi042.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-09-16T03:39:51.087 INFO:teuthology.orchestra.run.smithi154.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-09-16T03:39:51.087 INFO:teuthology.orchestra.run.smithi042.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-09-16T03:39:51.088 INFO:teuthology.orchestra.run.smithi154.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-09-16T03:39:51.088 INFO:teuthology.orchestra.run.smithi154.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-09-16T03:39:51.090 INFO:teuthology.orchestra.run.smithi154.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-09-16T03:39:51.091 INFO:teuthology.orchestra.run.smithi154.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-09-16T03:39:51.156 INFO:teuthology.orchestra.run.smithi042.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-09-16T03:39:51.157 INFO:teuthology.orchestra.run.smithi154.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-09-16T03:39:51.157 INFO:teuthology.orchestra.run.smithi154.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-09-16T03:39:51.170 INFO:teuthology.orchestra.run.smithi042.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-09-16T03:39:51.170 INFO:teuthology.orchestra.run.smithi042.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-09-16T03:39:51.171 INFO:teuthology.orchestra.run.smithi042.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-09-16T03:39:51.176 INFO:teuthology.orchestra.run.smithi042.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-09-16T03:39:51.177 INFO:teuthology.orchestra.run.smithi042.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-09-16T03:39:51.236 INFO:teuthology.orchestra.run.smithi154.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-09-16T03:39:51.237 INFO:teuthology.orchestra.run.smithi042.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-09-16T03:39:51.237 INFO:teuthology.orchestra.run.smithi154.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-09-16T03:39:51.238 INFO:teuthology.orchestra.run.smithi042.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-09-16T03:39:51.238 INFO:teuthology.orchestra.run.smithi154.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-09-16T03:39:51.240 INFO:teuthology.orchestra.run.smithi042.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-09-16T03:39:51.240 INFO:teuthology.orchestra.run.smithi042.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-09-16T03:39:51.243 INFO:teuthology.orchestra.run.smithi154.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-09-16T03:39:51.245 INFO:teuthology.orchestra.run.smithi154.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-09-16T03:39:51.246 INFO:teuthology.orchestra.run.smithi154.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-09-16T03:39:51.246 INFO:teuthology.orchestra.run.smithi154.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-09-16T03:39:51.249 INFO:teuthology.orchestra.run.smithi154.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-09-16T03:39:51.318 INFO:teuthology.orchestra.run.smithi154.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-09-16T03:39:51.318 INFO:teuthology.orchestra.run.smithi042.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-webob all 1:1.8.6-1.1ubuntu0.1 [86.7 kB] 2024-09-16T03:39:51.318 INFO:teuthology.orchestra.run.smithi154.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-webob all 1:1.8.6-1.1ubuntu0.1 [86.7 kB] 2024-09-16T03:39:51.320 INFO:teuthology.orchestra.run.smithi042.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-09-16T03:39:51.322 INFO:teuthology.orchestra.run.smithi042.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-09-16T03:39:51.323 INFO:teuthology.orchestra.run.smithi042.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-09-16T03:39:51.333 INFO:teuthology.orchestra.run.smithi042.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-09-16T03:39:51.399 INFO:teuthology.orchestra.run.smithi154.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-09-16T03:39:51.399 INFO:teuthology.orchestra.run.smithi042.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-09-16T03:39:51.400 INFO:teuthology.orchestra.run.smithi042.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-09-16T03:39:51.401 INFO:teuthology.orchestra.run.smithi154.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-09-16T03:39:51.402 INFO:teuthology.orchestra.run.smithi042.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-09-16T03:39:51.402 INFO:teuthology.orchestra.run.smithi154.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-09-16T03:39:51.404 INFO:teuthology.orchestra.run.smithi042.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-09-16T03:39:51.408 INFO:teuthology.orchestra.run.smithi042.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-idna all 3.3-1ubuntu0.1 [52.1 kB] 2024-09-16T03:39:51.413 INFO:teuthology.orchestra.run.smithi154.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-09-16T03:39:51.413 INFO:teuthology.orchestra.run.smithi154.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-09-16T03:39:51.414 INFO:teuthology.orchestra.run.smithi154.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-09-16T03:39:51.415 INFO:teuthology.orchestra.run.smithi154.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-09-16T03:39:51.418 INFO:teuthology.orchestra.run.smithi154.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-09-16T03:39:51.480 INFO:teuthology.orchestra.run.smithi154.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-idna all 3.3-1ubuntu0.1 [52.1 kB] 2024-09-16T03:39:51.481 INFO:teuthology.orchestra.run.smithi042.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-09-16T03:39:51.482 INFO:teuthology.orchestra.run.smithi154.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-09-16T03:39:51.483 INFO:teuthology.orchestra.run.smithi042.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-09-16T03:39:51.485 INFO:teuthology.orchestra.run.smithi042.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.2 [181 kB] 2024-09-16T03:39:51.489 INFO:teuthology.orchestra.run.smithi042.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-09-16T03:39:51.491 INFO:teuthology.orchestra.run.smithi042.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-09-16T03:39:51.504 INFO:teuthology.orchestra.run.smithi154.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-09-16T03:39:51.505 INFO:teuthology.orchestra.run.smithi154.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.2 [181 kB] 2024-09-16T03:39:51.510 INFO:teuthology.orchestra.run.smithi154.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-09-16T03:39:51.511 INFO:teuthology.orchestra.run.smithi154.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-09-16T03:39:51.512 INFO:teuthology.orchestra.run.smithi154.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-09-16T03:39:51.562 INFO:teuthology.orchestra.run.smithi042.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-09-16T03:39:51.567 INFO:teuthology.orchestra.run.smithi042.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-09-16T03:39:51.588 INFO:teuthology.orchestra.run.smithi154.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-09-16T03:39:51.652 INFO:teuthology.orchestra.run.smithi154.stdout:Get:63 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.1.1-326-g8921c8ef-1jammy [5,145 kB] 2024-09-16T03:39:51.691 INFO:teuthology.orchestra.run.smithi042.stdout:Get:63 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.1.1-326-g8921c8ef-1jammy [5,145 kB] 2024-09-16T03:39:51.731 INFO:teuthology.orchestra.run.smithi154.stdout:Get:64 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.1.1-326-g8921c8ef-1jammy [245 kB] 2024-09-16T03:39:51.731 INFO:teuthology.orchestra.run.smithi042.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-09-16T03:39:51.732 INFO:teuthology.orchestra.run.smithi154.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-09-16T03:39:51.733 INFO:teuthology.orchestra.run.smithi042.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.2 [108 kB] 2024-09-16T03:39:51.733 INFO:teuthology.orchestra.run.smithi154.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.2 [108 kB] 2024-09-16T03:39:51.736 INFO:teuthology.orchestra.run.smithi042.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-09-16T03:39:51.736 INFO:teuthology.orchestra.run.smithi042.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-09-16T03:39:51.776 INFO:teuthology.orchestra.run.smithi154.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-09-16T03:39:51.776 INFO:teuthology.orchestra.run.smithi154.stdout:Get:68 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.1.1-326-g8921c8ef-1jammy [124 kB] 2024-09-16T03:39:51.777 INFO:teuthology.orchestra.run.smithi154.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-09-16T03:39:51.777 INFO:teuthology.orchestra.run.smithi154.stdout:Get:70 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.1.1-326-g8921c8ef-1jammy [1,555 kB] 2024-09-16T03:39:51.778 INFO:teuthology.orchestra.run.smithi154.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-09-16T03:39:51.788 INFO:teuthology.orchestra.run.smithi042.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-09-16T03:39:51.792 INFO:teuthology.orchestra.run.smithi154.stdout:Get:72 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.1.1-326-g8921c8ef-1jammy [6,143 kB] 2024-09-16T03:39:51.795 INFO:teuthology.orchestra.run.smithi042.stdout:Get:69 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.1.1-326-g8921c8ef-1jammy [245 kB] 2024-09-16T03:39:51.799 INFO:teuthology.orchestra.run.smithi042.stdout:Get:70 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.1.1-326-g8921c8ef-1jammy [124 kB] 2024-09-16T03:39:51.811 INFO:teuthology.orchestra.run.smithi154.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-09-16T03:39:51.815 INFO:teuthology.orchestra.run.smithi154.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-09-16T03:39:51.816 INFO:teuthology.orchestra.run.smithi154.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-09-16T03:39:51.822 INFO:teuthology.orchestra.run.smithi042.stdout:Get:71 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.1.1-326-g8921c8ef-1jammy [1,555 kB] 2024-09-16T03:39:51.834 INFO:teuthology.orchestra.run.smithi042.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-09-16T03:39:51.838 INFO:teuthology.orchestra.run.smithi042.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-09-16T03:39:51.839 INFO:teuthology.orchestra.run.smithi042.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-09-16T03:39:51.873 INFO:teuthology.orchestra.run.smithi042.stdout:Get:75 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.1.1-326-g8921c8ef-1jammy [6,143 kB] 2024-09-16T03:39:51.890 INFO:teuthology.orchestra.run.smithi154.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-09-16T03:39:51.891 INFO:teuthology.orchestra.run.smithi154.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-09-16T03:39:51.895 INFO:teuthology.orchestra.run.smithi042.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-09-16T03:39:51.895 INFO:teuthology.orchestra.run.smithi042.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-09-16T03:39:51.897 INFO:teuthology.orchestra.run.smithi154.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-09-16T03:39:51.898 INFO:teuthology.orchestra.run.smithi154.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-09-16T03:39:51.899 INFO:teuthology.orchestra.run.smithi042.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-09-16T03:39:51.899 INFO:teuthology.orchestra.run.smithi154.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-09-16T03:39:51.899 INFO:teuthology.orchestra.run.smithi042.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-09-16T03:39:51.900 INFO:teuthology.orchestra.run.smithi154.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-09-16T03:39:51.900 INFO:teuthology.orchestra.run.smithi042.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-09-16T03:39:51.900 INFO:teuthology.orchestra.run.smithi154.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-09-16T03:39:51.902 INFO:teuthology.orchestra.run.smithi042.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-09-16T03:39:51.955 INFO:teuthology.orchestra.run.smithi154.stdout:Get:83 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.1.1-326-g8921c8ef-1jammy [22.8 MB] 2024-09-16T03:39:51.979 INFO:teuthology.orchestra.run.smithi154.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-09-16T03:39:51.980 INFO:teuthology.orchestra.run.smithi042.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-09-16T03:39:51.981 INFO:teuthology.orchestra.run.smithi042.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-09-16T03:39:51.986 INFO:teuthology.orchestra.run.smithi154.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-09-16T03:39:51.990 INFO:teuthology.orchestra.run.smithi042.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-09-16T03:39:51.990 INFO:teuthology.orchestra.run.smithi154.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-09-16T03:39:51.994 INFO:teuthology.orchestra.run.smithi042.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-09-16T03:39:51.994 INFO:teuthology.orchestra.run.smithi154.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-09-16T03:39:51.995 INFO:teuthology.orchestra.run.smithi154.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-09-16T03:39:51.997 INFO:teuthology.orchestra.run.smithi042.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-09-16T03:39:51.999 INFO:teuthology.orchestra.run.smithi042.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-09-16T03:39:52.061 INFO:teuthology.orchestra.run.smithi154.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-09-16T03:39:52.061 INFO:teuthology.orchestra.run.smithi042.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-09-16T03:39:52.067 INFO:teuthology.orchestra.run.smithi154.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-09-16T03:39:52.068 INFO:teuthology.orchestra.run.smithi042.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-09-16T03:39:52.069 INFO:teuthology.orchestra.run.smithi154.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-09-16T03:39:52.070 INFO:teuthology.orchestra.run.smithi042.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-09-16T03:39:52.070 INFO:teuthology.orchestra.run.smithi154.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.3 [474 kB] 2024-09-16T03:39:52.071 INFO:teuthology.orchestra.run.smithi042.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.3 [474 kB] 2024-09-16T03:39:52.082 INFO:teuthology.orchestra.run.smithi154.stdout:Get:93 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-09-16T03:39:52.100 INFO:teuthology.orchestra.run.smithi042.stdout:Get:92 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.1.1-326-g8921c8ef-1jammy [22.8 MB] 2024-09-16T03:39:52.142 INFO:teuthology.orchestra.run.smithi154.stdout:Get:94 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-09-16T03:39:52.143 INFO:teuthology.orchestra.run.smithi042.stdout:Get:93 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-09-16T03:39:52.144 INFO:teuthology.orchestra.run.smithi042.stdout:Get:94 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-09-16T03:39:52.149 INFO:teuthology.orchestra.run.smithi154.stdout:Get:95 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-09-16T03:39:52.150 INFO:teuthology.orchestra.run.smithi154.stdout:Get:96 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-09-16T03:39:52.151 INFO:teuthology.orchestra.run.smithi154.stdout:Get:97 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-09-16T03:39:52.152 INFO:teuthology.orchestra.run.smithi042.stdout:Get:95 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-09-16T03:39:52.152 INFO:teuthology.orchestra.run.smithi154.stdout:Get:98 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-09-16T03:39:52.153 INFO:teuthology.orchestra.run.smithi042.stdout:Get:96 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-09-16T03:39:52.154 INFO:teuthology.orchestra.run.smithi042.stdout:Get:97 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-09-16T03:39:52.155 INFO:teuthology.orchestra.run.smithi042.stdout:Get:98 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-09-16T03:39:52.223 INFO:teuthology.orchestra.run.smithi154.stdout:Get:99 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-09-16T03:39:52.225 INFO:teuthology.orchestra.run.smithi042.stdout:Get:99 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-09-16T03:39:52.225 INFO:teuthology.orchestra.run.smithi154.stdout:Get:100 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-09-16T03:39:52.227 INFO:teuthology.orchestra.run.smithi042.stdout:Get:100 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-09-16T03:39:52.242 INFO:teuthology.orchestra.run.smithi154.stdout:Get:101 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-09-16T03:39:52.243 INFO:teuthology.orchestra.run.smithi154.stdout:Get:102 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-09-16T03:39:52.244 INFO:teuthology.orchestra.run.smithi154.stdout:Get:103 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-09-16T03:39:52.244 INFO:teuthology.orchestra.run.smithi042.stdout:Get:101 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-09-16T03:39:52.245 INFO:teuthology.orchestra.run.smithi042.stdout:Get:102 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-09-16T03:39:52.305 INFO:teuthology.orchestra.run.smithi154.stdout:Get:104 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-09-16T03:39:52.306 INFO:teuthology.orchestra.run.smithi154.stdout:Get:105 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-09-16T03:39:52.307 INFO:teuthology.orchestra.run.smithi042.stdout:Get:103 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-09-16T03:39:52.311 INFO:teuthology.orchestra.run.smithi042.stdout:Get:104 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-09-16T03:39:52.313 INFO:teuthology.orchestra.run.smithi042.stdout:Get:105 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-09-16T03:39:52.727 INFO:teuthology.orchestra.run.smithi154.stdout:Get:106 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.1.1-326-g8921c8ef-1jammy [14.1 kB] 2024-09-16T03:39:52.728 INFO:teuthology.orchestra.run.smithi154.stdout:Get:107 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.1.1-326-g8921c8ef-1jammy [877 kB] 2024-09-16T03:39:52.745 INFO:teuthology.orchestra.run.smithi154.stdout:Get:108 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.1.1-326-g8921c8ef-1jammy [2,462 kB] 2024-09-16T03:39:52.785 INFO:teuthology.orchestra.run.smithi154.stdout:Get:109 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.1.1-326-g8921c8ef-1jammy [794 kB] 2024-09-16T03:39:52.795 INFO:teuthology.orchestra.run.smithi154.stdout:Get:110 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.1.1-326-g8921c8ef-1jammy [152 kB] 2024-09-16T03:39:52.810 INFO:teuthology.orchestra.run.smithi154.stdout:Get:111 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.1.1-326-g8921c8ef-1jammy [2,275 kB] 2024-09-16T03:39:52.873 INFO:teuthology.orchestra.run.smithi154.stdout:Get:112 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.1.1-326-g8921c8ef-1jammy [8,625 kB] 2024-09-16T03:39:53.013 INFO:teuthology.orchestra.run.smithi042.stdout:Get:106 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.1.1-326-g8921c8ef-1jammy [14.1 kB] 2024-09-16T03:39:53.013 INFO:teuthology.orchestra.run.smithi042.stdout:Get:107 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.1.1-326-g8921c8ef-1jammy [877 kB] 2024-09-16T03:39:53.031 INFO:teuthology.orchestra.run.smithi042.stdout:Get:108 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.1.1-326-g8921c8ef-1jammy [2,462 kB] 2024-09-16T03:39:53.074 INFO:teuthology.orchestra.run.smithi042.stdout:Get:109 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.1.1-326-g8921c8ef-1jammy [794 kB] 2024-09-16T03:39:53.085 INFO:teuthology.orchestra.run.smithi042.stdout:Get:110 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.1.1-326-g8921c8ef-1jammy [152 kB] 2024-09-16T03:39:53.087 INFO:teuthology.orchestra.run.smithi042.stdout:Get:111 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.1.1-326-g8921c8ef-1jammy [2,275 kB] 2024-09-16T03:39:53.095 INFO:teuthology.orchestra.run.smithi154.stdout:Get:113 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.1.1-326-g8921c8ef-1jammy [14.2 kB] 2024-09-16T03:39:53.095 INFO:teuthology.orchestra.run.smithi154.stdout:Get:114 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.1.1-326-g8921c8ef-1jammy [51.4 MB] 2024-09-16T03:39:53.165 INFO:teuthology.orchestra.run.smithi042.stdout:Get:112 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.1.1-326-g8921c8ef-1jammy [8,625 kB] 2024-09-16T03:39:53.463 INFO:teuthology.orchestra.run.smithi042.stdout:Get:113 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.1.1-326-g8921c8ef-1jammy [14.2 kB] 2024-09-16T03:39:53.464 INFO:teuthology.orchestra.run.smithi042.stdout:Get:114 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.1.1-326-g8921c8ef-1jammy [51.4 MB] 2024-09-16T03:39:54.953 INFO:teuthology.orchestra.run.smithi154.stdout:Get:115 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.1.1-326-g8921c8ef-1jammy [130 kB] 2024-09-16T03:39:54.955 INFO:teuthology.orchestra.run.smithi154.stdout:Get:116 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.1.1-326-g8921c8ef-1jammy [39.0 kB] 2024-09-16T03:39:54.956 INFO:teuthology.orchestra.run.smithi154.stdout:Get:117 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.1.1-326-g8921c8ef-1jammy [13.6 MB] 2024-09-16T03:39:55.246 INFO:teuthology.orchestra.run.smithi154.stdout:Get:118 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.1.1-326-g8921c8ef-1jammy [91.5 kB] 2024-09-16T03:39:55.290 INFO:teuthology.orchestra.run.smithi042.stdout:Get:115 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.1.1-326-g8921c8ef-1jammy [130 kB] 2024-09-16T03:39:55.293 INFO:teuthology.orchestra.run.smithi042.stdout:Get:116 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.1.1-326-g8921c8ef-1jammy [39.0 kB] 2024-09-16T03:39:55.294 INFO:teuthology.orchestra.run.smithi042.stdout:Get:117 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.1.1-326-g8921c8ef-1jammy [13.6 MB] 2024-09-16T03:39:55.528 INFO:teuthology.orchestra.run.smithi042.stdout:Get:118 https://1.chacra.ceph.com/r/ceph/squid/8921c8ef947807b717db041d85f0761b7914d5fc/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.1.1-326-g8921c8ef-1jammy [91.5 kB] 2024-09-16T03:39:55.824 INFO:teuthology.orchestra.run.smithi154.stdout:Fetched 181 MB in 5s (34.4 MB/s) 2024-09-16T03:39:55.969 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-09-16T03:39:56.068 INFO:teuthology.orchestra.run.smithi154.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165990 files and directories currently installed.) 2024-09-16T03:39:56.074 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-09-16T03:39:56.111 INFO:teuthology.orchestra.run.smithi042.stdout:Fetched 181 MB in 6s (32.6 MB/s) 2024-09-16T03:39:56.143 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-16T03:39:56.273 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-09-16T03:39:56.376 INFO:teuthology.orchestra.run.smithi042.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165990 files and directories currently installed.) 2024-09-16T03:39:56.381 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-09-16T03:39:56.471 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-16T03:39:56.472 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-09-16T03:39:56.498 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-09-16T03:39:56.538 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-16T03:39:56.792 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-09-16T03:39:56.819 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-09-16T03:39:56.858 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-16T03:39:56.874 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-09-16T03:39:56.901 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-09-16T03:39:56.941 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-16T03:39:57.203 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-09-16T03:39:57.230 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-09-16T03:39:57.270 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-16T03:39:57.428 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-09-16T03:39:57.455 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-16T03:39:57.503 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T03:39:57.723 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-09-16T03:39:57.750 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-16T03:39:57.798 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T03:39:57.923 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-09-16T03:39:57.949 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-16T03:39:57.989 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T03:39:58.210 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-09-16T03:39:58.237 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-16T03:39:58.277 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T03:39:58.326 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-09-16T03:39:58.352 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-16T03:39:58.392 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T03:39:58.630 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-09-16T03:39:58.657 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-16T03:39:58.696 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T03:39:58.746 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-09-16T03:39:58.772 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-09-16T03:39:58.812 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-16T03:39:59.025 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-09-16T03:39:59.052 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-09-16T03:39:59.091 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-16T03:39:59.132 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package librados2. 2024-09-16T03:39:59.158 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../007-librados2_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T03:39:59.198 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking librados2 (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:39:59.419 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package librados2. 2024-09-16T03:39:59.446 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../007-librados2_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T03:39:59.486 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking librados2 (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:39:59.643 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package librbd1. 2024-09-16T03:39:59.669 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../008-librbd1_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T03:39:59.709 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking librbd1 (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:39:59.964 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package librbd1. 2024-09-16T03:39:59.991 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../008-librbd1_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T03:40:00.030 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking librbd1 (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:40:00.163 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package libcephfs2. 2024-09-16T03:40:00.189 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../009-libcephfs2_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T03:40:00.229 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking libcephfs2 (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:40:00.450 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package libcephfs2. 2024-09-16T03:40:00.477 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../009-libcephfs2_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T03:40:00.507 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-rados. 2024-09-16T03:40:00.517 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking libcephfs2 (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:40:00.522 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../010-python3-rados_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T03:40:00.565 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-rados (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:40:00.803 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-rados. 2024-09-16T03:40:00.819 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../010-python3-rados_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T03:40:00.862 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-rados (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:40:00.901 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-09-16T03:40:00.928 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../011-python3-ceph-argparse_19.1.1-326-g8921c8ef-1jammy_all.deb ... 2024-09-16T03:40:00.959 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-ceph-argparse (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:40:01.122 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-09-16T03:40:01.138 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../011-python3-ceph-argparse_19.1.1-326-g8921c8ef-1jammy_all.deb ... 2024-09-16T03:40:01.172 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-ceph-argparse (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:40:01.212 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-cephfs. 2024-09-16T03:40:01.238 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../012-python3-cephfs_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T03:40:01.279 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-cephfs (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:40:01.425 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-cephfs. 2024-09-16T03:40:01.441 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../012-python3-cephfs_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T03:40:01.483 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-cephfs (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:40:01.590 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-ceph-common. 2024-09-16T03:40:01.616 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../013-python3-ceph-common_19.1.1-326-g8921c8ef-1jammy_all.deb ... 2024-09-16T03:40:01.656 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-ceph-common (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:40:01.803 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-ceph-common. 2024-09-16T03:40:01.830 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../013-python3-ceph-common_19.1.1-326-g8921c8ef-1jammy_all.deb ... 2024-09-16T03:40:01.869 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-ceph-common (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:40:01.993 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-wcwidth. 2024-09-16T03:40:02.019 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../014-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-09-16T03:40:02.060 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-16T03:40:02.206 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-wcwidth. 2024-09-16T03:40:02.232 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../014-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-09-16T03:40:02.272 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-16T03:40:02.371 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-prettytable. 2024-09-16T03:40:02.397 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../015-python3-prettytable_2.5.0-2_all.deb ... 2024-09-16T03:40:02.437 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-09-16T03:40:02.551 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-prettytable. 2024-09-16T03:40:02.577 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../015-python3-prettytable_2.5.0-2_all.deb ... 2024-09-16T03:40:02.609 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-09-16T03:40:02.690 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-rbd. 2024-09-16T03:40:02.705 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../016-python3-rbd_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T03:40:02.748 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-rbd (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:40:02.870 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-rbd. 2024-09-16T03:40:02.886 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../016-python3-rbd_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T03:40:02.928 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-rbd (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:40:03.281 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-09-16T03:40:03.308 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../017-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-09-16T03:40:03.310 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-09-16T03:40:03.336 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../017-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-09-16T03:40:03.350 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-16T03:40:03.376 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-16T03:40:03.629 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-09-16T03:40:03.655 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../018-libreadline-dev_8.1.2-1_amd64.deb ... 2024-09-16T03:40:03.695 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-09-16T03:40:03.859 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-09-16T03:40:03.875 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../018-libreadline-dev_8.1.2-1_amd64.deb ... 2024-09-16T03:40:03.909 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-09-16T03:40:03.999 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-09-16T03:40:04.014 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../019-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-09-16T03:40:04.057 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-16T03:40:04.196 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-09-16T03:40:04.222 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../019-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-09-16T03:40:04.262 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-16T03:40:04.435 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package lua5.1. 2024-09-16T03:40:04.461 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../020-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-09-16T03:40:04.502 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-09-16T03:40:04.624 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package lua5.1. 2024-09-16T03:40:04.650 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../020-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-09-16T03:40:04.690 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-09-16T03:40:04.822 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package lua-any. 2024-09-16T03:40:04.848 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../021-lua-any_27ubuntu1_all.deb ... 2024-09-16T03:40:04.888 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking lua-any (27ubuntu1) ... 2024-09-16T03:40:04.994 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package lua-any. 2024-09-16T03:40:05.010 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../021-lua-any_27ubuntu1_all.deb ... 2024-09-16T03:40:05.044 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking lua-any (27ubuntu1) ... 2024-09-16T03:40:05.141 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package zip. 2024-09-16T03:40:05.167 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../022-zip_3.0-12build2_amd64.deb ... 2024-09-16T03:40:05.208 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking zip (3.0-12build2) ... 2024-09-16T03:40:05.254 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package zip. 2024-09-16T03:40:05.270 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../022-zip_3.0-12build2_amd64.deb ... 2024-09-16T03:40:05.313 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking zip (3.0-12build2) ... 2024-09-16T03:40:05.477 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package unzip. 2024-09-16T03:40:05.493 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../023-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-09-16T03:40:05.552 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-09-16T03:40:05.608 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package unzip. 2024-09-16T03:40:05.635 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../023-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-09-16T03:40:05.674 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-09-16T03:40:05.965 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package luarocks. 2024-09-16T03:40:05.992 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../024-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-09-16T03:40:06.031 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-09-16T03:40:06.087 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package luarocks. 2024-09-16T03:40:06.114 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../024-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-09-16T03:40:06.153 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-09-16T03:40:06.389 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package librgw2. 2024-09-16T03:40:06.416 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../025-librgw2_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T03:40:06.456 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking librgw2 (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:40:06.480 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package librgw2. 2024-09-16T03:40:06.496 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../025-librgw2_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T03:40:06.530 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking librgw2 (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:40:07.017 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-rgw. 2024-09-16T03:40:07.044 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../026-python3-rgw_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T03:40:07.083 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-rgw (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:40:07.099 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-rgw. 2024-09-16T03:40:07.116 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../026-python3-rgw_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T03:40:07.157 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-rgw (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:40:07.386 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package liboath0:amd64. 2024-09-16T03:40:07.413 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../027-liboath0_2.6.7-3build1_amd64.deb ... 2024-09-16T03:40:07.453 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-09-16T03:40:07.494 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package liboath0:amd64. 2024-09-16T03:40:07.521 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../027-liboath0_2.6.7-3build1_amd64.deb ... 2024-09-16T03:40:07.560 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-09-16T03:40:07.756 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package libradosstriper1. 2024-09-16T03:40:07.783 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../028-libradosstriper1_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T03:40:07.822 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking libradosstriper1 (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:40:07.830 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package libradosstriper1. 2024-09-16T03:40:07.845 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../028-libradosstriper1_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T03:40:07.880 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking libradosstriper1 (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:40:08.283 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package ceph-common. 2024-09-16T03:40:08.310 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../029-ceph-common_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T03:40:08.349 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking ceph-common (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:40:08.760 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package ceph-common. 2024-09-16T03:40:08.787 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../029-ceph-common_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T03:40:08.826 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking ceph-common (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:40:09.847 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package ceph-base. 2024-09-16T03:40:09.874 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../030-ceph-base_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T03:40:09.923 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking ceph-base (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:40:10.123 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package ceph-base. 2024-09-16T03:40:10.150 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../030-ceph-base_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T03:40:10.190 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking ceph-base (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:40:10.484 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-09-16T03:40:10.511 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../031-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-09-16T03:40:10.551 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-09-16T03:40:10.761 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-09-16T03:40:10.788 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../031-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-09-16T03:40:10.812 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-cheroot. 2024-09-16T03:40:10.828 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-09-16T03:40:10.839 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../032-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-09-16T03:40:10.879 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-16T03:40:11.139 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-cheroot. 2024-09-16T03:40:11.165 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../032-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-09-16T03:40:11.205 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-16T03:40:11.215 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-09-16T03:40:11.242 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../033-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-09-16T03:40:11.281 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-09-16T03:40:11.542 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-09-16T03:40:11.569 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../033-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-09-16T03:40:11.584 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-jaraco.text. 2024-09-16T03:40:11.608 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-09-16T03:40:11.611 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../034-python3-jaraco.text_3.6.0-2_all.deb ... 2024-09-16T03:40:11.651 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-09-16T03:40:11.903 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-jaraco.text. 2024-09-16T03:40:11.930 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../034-python3-jaraco.text_3.6.0-2_all.deb ... 2024-09-16T03:40:11.937 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-09-16T03:40:11.953 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../035-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-09-16T03:40:11.970 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-09-16T03:40:11.995 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-09-16T03:40:12.272 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-09-16T03:40:12.273 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-tempora. 2024-09-16T03:40:12.299 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../035-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-09-16T03:40:12.300 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../036-python3-tempora_4.1.2-1_all.deb ... 2024-09-16T03:40:12.339 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-09-16T03:40:12.340 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-09-16T03:40:12.617 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-tempora. 2024-09-16T03:40:12.642 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-portend. 2024-09-16T03:40:12.643 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../036-python3-tempora_4.1.2-1_all.deb ... 2024-09-16T03:40:12.670 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../037-python3-portend_3.0.0-1_all.deb ... 2024-09-16T03:40:12.675 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-09-16T03:40:12.709 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-09-16T03:40:12.987 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-portend. 2024-09-16T03:40:13.012 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-09-16T03:40:13.013 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../037-python3-portend_3.0.0-1_all.deb ... 2024-09-16T03:40:13.040 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../038-python3-zc.lockfile_2.0-1_all.deb ... 2024-09-16T03:40:13.053 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-09-16T03:40:13.078 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-09-16T03:40:13.348 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-cherrypy3. 2024-09-16T03:40:13.356 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-09-16T03:40:13.376 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../039-python3-cherrypy3_18.6.1-4_all.deb ... 2024-09-16T03:40:13.383 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../038-python3-zc.lockfile_2.0-1_all.deb ... 2024-09-16T03:40:13.406 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-09-16T03:40:13.423 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-09-16T03:40:13.734 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-natsort. 2024-09-16T03:40:13.759 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../040-python3-natsort_8.0.2-1_all.deb ... 2024-09-16T03:40:13.793 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-09-16T03:40:13.876 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-cherrypy3. 2024-09-16T03:40:13.902 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../039-python3-cherrypy3_18.6.1-4_all.deb ... 2024-09-16T03:40:13.942 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-09-16T03:40:14.088 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-logutils. 2024-09-16T03:40:14.115 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../041-python3-logutils_0.3.3-8_all.deb ... 2024-09-16T03:40:14.154 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-09-16T03:40:14.305 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-natsort. 2024-09-16T03:40:14.331 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../040-python3-natsort_8.0.2-1_all.deb ... 2024-09-16T03:40:14.371 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-09-16T03:40:14.400 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-markupsafe. 2024-09-16T03:40:14.426 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../042-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-09-16T03:40:14.457 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-09-16T03:40:14.699 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-logutils. 2024-09-16T03:40:14.726 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../041-python3-logutils_0.3.3-8_all.deb ... 2024-09-16T03:40:14.766 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-09-16T03:40:14.935 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-mako. 2024-09-16T03:40:14.962 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../043-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-09-16T03:40:15.001 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-16T03:40:15.052 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-markupsafe. 2024-09-16T03:40:15.078 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../042-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-09-16T03:40:15.127 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-09-16T03:40:15.296 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-simplegeneric. 2024-09-16T03:40:15.312 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../044-python3-simplegeneric_0.8.1-3_all.deb ... 2024-09-16T03:40:15.346 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-09-16T03:40:15.422 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-mako. 2024-09-16T03:40:15.448 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../043-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-09-16T03:40:15.488 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-16T03:40:15.641 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-singledispatch. 2024-09-16T03:40:15.668 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../045-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-09-16T03:40:15.707 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-09-16T03:40:15.816 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-simplegeneric. 2024-09-16T03:40:15.843 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../044-python3-simplegeneric_0.8.1-3_all.deb ... 2024-09-16T03:40:15.883 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-09-16T03:40:16.010 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-webob. 2024-09-16T03:40:16.037 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../046-python3-webob_1%3a1.8.6-1.1ubuntu0.1_all.deb ... 2024-09-16T03:40:16.076 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-16T03:40:16.169 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-singledispatch. 2024-09-16T03:40:16.195 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../045-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-09-16T03:40:16.235 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-09-16T03:40:16.438 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-waitress. 2024-09-16T03:40:16.459 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../047-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-09-16T03:40:16.499 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-16T03:40:16.522 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-webob. 2024-09-16T03:40:16.548 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../046-python3-webob_1%3a1.8.6-1.1ubuntu0.1_all.deb ... 2024-09-16T03:40:16.588 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-16T03:40:16.807 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-tempita. 2024-09-16T03:40:16.834 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../048-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-09-16T03:40:16.874 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-09-16T03:40:16.933 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-waitress. 2024-09-16T03:40:16.960 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../047-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-09-16T03:40:17.003 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-16T03:40:17.160 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-paste. 2024-09-16T03:40:17.187 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../049-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-09-16T03:40:17.226 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-09-16T03:40:17.319 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-tempita. 2024-09-16T03:40:17.346 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../048-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-09-16T03:40:17.386 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-09-16T03:40:17.571 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-09-16T03:40:17.587 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../050-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-09-16T03:40:17.630 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-09-16T03:40:17.706 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-paste. 2024-09-16T03:40:17.733 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../049-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-09-16T03:40:17.772 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-09-16T03:40:17.916 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-pastedeploy. 2024-09-16T03:40:17.943 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../051-python3-pastedeploy_2.1.1-1_all.deb ... 2024-09-16T03:40:17.991 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-09-16T03:40:18.101 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-09-16T03:40:18.128 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../050-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-09-16T03:40:18.192 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-09-16T03:40:18.293 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-webtest. 2024-09-16T03:40:18.310 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../052-python3-webtest_2.0.35-1_all.deb ... 2024-09-16T03:40:18.343 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-09-16T03:40:18.504 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-pastedeploy. 2024-09-16T03:40:18.531 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../051-python3-pastedeploy_2.1.1-1_all.deb ... 2024-09-16T03:40:18.570 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-09-16T03:40:18.647 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-pecan. 2024-09-16T03:40:18.673 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../053-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-09-16T03:40:18.713 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-09-16T03:40:18.865 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-webtest. 2024-09-16T03:40:18.892 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../052-python3-webtest_2.0.35-1_all.deb ... 2024-09-16T03:40:18.923 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-09-16T03:40:19.068 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-certifi. 2024-09-16T03:40:19.095 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../054-python3-certifi_2020.6.20-1_all.deb ... 2024-09-16T03:40:19.134 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-09-16T03:40:19.201 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-pecan. 2024-09-16T03:40:19.227 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../053-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-09-16T03:40:19.267 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-09-16T03:40:19.471 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-idna. 2024-09-16T03:40:19.487 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../055-python3-idna_3.3-1ubuntu0.1_all.deb ... 2024-09-16T03:40:19.529 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-idna (3.3-1ubuntu0.1) ... 2024-09-16T03:40:19.641 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-certifi. 2024-09-16T03:40:19.668 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../054-python3-certifi_2020.6.20-1_all.deb ... 2024-09-16T03:40:19.708 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-09-16T03:40:19.949 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-urllib3. 2024-09-16T03:40:19.976 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../056-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-09-16T03:40:20.015 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-16T03:40:20.019 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-idna. 2024-09-16T03:40:20.038 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../055-python3-idna_3.3-1ubuntu0.1_all.deb ... 2024-09-16T03:40:20.077 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-idna (3.3-1ubuntu0.1) ... 2024-09-16T03:40:20.285 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-requests. 2024-09-16T03:40:20.312 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../057-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-09-16T03:40:20.351 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-16T03:40:20.355 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-urllib3. 2024-09-16T03:40:20.381 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../056-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-09-16T03:40:20.413 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-16T03:40:20.655 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-werkzeug. 2024-09-16T03:40:20.682 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../058-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.2_all.deb ... 2024-09-16T03:40:20.691 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-requests. 2024-09-16T03:40:20.718 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../057-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-09-16T03:40:20.721 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-16T03:40:20.758 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-16T03:40:21.032 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-09-16T03:40:21.048 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../059-ceph-mgr-modules-core_19.1.1-326-g8921c8ef-1jammy_all.deb ... 2024-09-16T03:40:21.053 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-werkzeug. 2024-09-16T03:40:21.079 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../058-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.2_all.deb ... 2024-09-16T03:40:21.090 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking ceph-mgr-modules-core (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:40:21.119 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-16T03:40:21.406 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-09-16T03:40:21.433 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../059-ceph-mgr-modules-core_19.1.1-326-g8921c8ef-1jammy_all.deb ... 2024-09-16T03:40:21.464 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking ceph-mgr-modules-core (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:40:21.479 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-09-16T03:40:21.506 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../060-libsqlite3-mod-ceph_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T03:40:21.548 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking libsqlite3-mod-ceph (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:40:21.861 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-09-16T03:40:21.865 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-bcrypt. 2024-09-16T03:40:21.887 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../060-libsqlite3-mod-ceph_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T03:40:21.889 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../061-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-09-16T03:40:21.927 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking libsqlite3-mod-ceph (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:40:21.931 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-09-16T03:40:22.197 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-bcrypt. 2024-09-16T03:40:22.209 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-openssl. 2024-09-16T03:40:22.223 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../061-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-09-16T03:40:22.225 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../062-python3-openssl_21.0.0-1_all.deb ... 2024-09-16T03:40:22.259 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-09-16T03:40:22.280 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-09-16T03:40:22.583 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-openssl. 2024-09-16T03:40:22.587 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package ceph-mgr. 2024-09-16T03:40:22.610 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../062-python3-openssl_21.0.0-1_all.deb ... 2024-09-16T03:40:22.612 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../063-ceph-mgr_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T03:40:22.649 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-09-16T03:40:22.662 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking ceph-mgr (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:40:22.986 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package ceph-mgr. 2024-09-16T03:40:23.013 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../063-ceph-mgr_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T03:40:23.052 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking ceph-mgr (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:40:23.449 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package ceph-mon. 2024-09-16T03:40:23.476 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../064-ceph-mon_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T03:40:23.899 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking ceph-mon (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:40:24.424 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package ceph-mon. 2024-09-16T03:40:24.450 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../064-ceph-mon_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T03:40:24.489 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking ceph-mon (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:40:24.595 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package ceph-osd. 2024-09-16T03:40:24.621 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../065-ceph-osd_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T03:40:24.661 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking ceph-osd (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:40:25.068 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package ceph-osd. 2024-09-16T03:40:25.095 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../065-ceph-osd_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T03:40:25.135 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking ceph-osd (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:40:25.739 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package ceph. 2024-09-16T03:40:25.755 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../066-ceph_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T03:40:25.789 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking ceph (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:40:26.083 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package ceph-fuse. 2024-09-16T03:40:26.110 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../067-ceph-fuse_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T03:40:26.146 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package ceph. 2024-09-16T03:40:26.150 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking ceph-fuse (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:40:26.161 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../066-ceph_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T03:40:26.204 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking ceph (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:40:26.494 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package ceph-mds. 2024-09-16T03:40:26.507 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../068-ceph-mds_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T03:40:26.508 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package ceph-fuse. 2024-09-16T03:40:26.534 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../067-ceph-fuse_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T03:40:26.544 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking ceph-mds (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:40:26.574 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking ceph-fuse (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:40:26.931 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package cephadm. 2024-09-16T03:40:26.944 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package ceph-mds. 2024-09-16T03:40:26.958 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../069-cephadm_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T03:40:26.971 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../068-ceph-mds_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T03:40:26.997 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking cephadm (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:40:27.010 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking ceph-mds (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:40:27.300 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-asyncssh. 2024-09-16T03:40:27.327 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../070-python3-asyncssh_2.5.0-1_all.deb ... 2024-09-16T03:40:27.366 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-09-16T03:40:27.413 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package cephadm. 2024-09-16T03:40:27.440 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../069-cephadm_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T03:40:27.479 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking cephadm (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:40:27.719 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python-babel-localedata. 2024-09-16T03:40:27.735 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../071-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-09-16T03:40:27.749 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-asyncssh. 2024-09-16T03:40:27.776 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../070-python3-asyncssh_2.5.0-1_all.deb ... 2024-09-16T03:40:27.777 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-16T03:40:27.816 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-09-16T03:40:28.110 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python-babel-localedata. 2024-09-16T03:40:28.137 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../071-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-09-16T03:40:28.168 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-16T03:40:28.783 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-babel. 2024-09-16T03:40:28.810 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../072-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-09-16T03:40:28.841 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-09-16T03:40:29.110 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-jinja2. 2024-09-16T03:40:29.126 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../073-python3-jinja2_3.0.3-1ubuntu0.2_all.deb ... 2024-09-16T03:40:29.160 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-16T03:40:29.232 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-babel. 2024-09-16T03:40:29.259 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../072-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-09-16T03:40:29.298 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-09-16T03:40:29.580 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-09-16T03:40:29.608 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../074-ceph-mgr-cephadm_19.1.1-326-g8921c8ef-1jammy_all.deb ... 2024-09-16T03:40:29.626 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-jinja2. 2024-09-16T03:40:29.653 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../073-python3-jinja2_3.0.3-1ubuntu0.2_all.deb ... 2024-09-16T03:40:29.655 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking ceph-mgr-cephadm (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:40:29.693 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-16T03:40:30.021 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-09-16T03:40:30.042 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-repoze.lru. 2024-09-16T03:40:30.048 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../074-ceph-mgr-cephadm_19.1.1-326-g8921c8ef-1jammy_all.deb ... 2024-09-16T03:40:30.070 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../075-python3-repoze.lru_0.7-2_all.deb ... 2024-09-16T03:40:30.087 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking ceph-mgr-cephadm (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:40:30.116 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-09-16T03:40:30.408 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-repoze.lru. 2024-09-16T03:40:30.435 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../075-python3-repoze.lru_0.7-2_all.deb ... 2024-09-16T03:40:30.474 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-09-16T03:40:30.686 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-routes. 2024-09-16T03:40:30.714 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../076-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-09-16T03:40:30.761 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-routes. 2024-09-16T03:40:30.761 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-09-16T03:40:30.787 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../076-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-09-16T03:40:30.827 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-09-16T03:40:31.121 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-09-16T03:40:31.137 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../077-ceph-mgr-dashboard_19.1.1-326-g8921c8ef-1jammy_all.deb ... 2024-09-16T03:40:31.164 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-09-16T03:40:31.171 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking ceph-mgr-dashboard (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:40:31.192 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../077-ceph-mgr-dashboard_19.1.1-326-g8921c8ef-1jammy_all.deb ... 2024-09-16T03:40:31.239 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking ceph-mgr-dashboard (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:40:32.460 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-09-16T03:40:32.475 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../078-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-09-16T03:40:32.518 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-16T03:40:32.519 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-09-16T03:40:32.535 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../078-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-09-16T03:40:32.569 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-16T03:40:33.030 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-joblib. 2024-09-16T03:40:33.057 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../079-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-09-16T03:40:33.081 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-joblib. 2024-09-16T03:40:33.096 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-09-16T03:40:33.105 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../079-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-09-16T03:40:33.147 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-09-16T03:40:33.459 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-09-16T03:40:33.486 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../080-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-09-16T03:40:33.501 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-09-16T03:40:33.525 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-09-16T03:40:33.528 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../080-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-09-16T03:40:33.567 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-09-16T03:40:33.820 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-sklearn. 2024-09-16T03:40:33.828 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-sklearn. 2024-09-16T03:40:33.847 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../081-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-09-16T03:40:33.854 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../081-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-09-16T03:40:33.886 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-16T03:40:33.894 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-16T03:40:34.637 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-09-16T03:40:34.665 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../082-ceph-mgr-diskprediction-local_19.1.1-326-g8921c8ef-1jammy_all.deb ... 2024-09-16T03:40:34.795 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-09-16T03:40:34.797 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking ceph-mgr-diskprediction-local (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:40:34.822 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../082-ceph-mgr-diskprediction-local_19.1.1-326-g8921c8ef-1jammy_all.deb ... 2024-09-16T03:40:34.930 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking ceph-mgr-diskprediction-local (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:40:35.726 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-cachetools. 2024-09-16T03:40:35.743 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../083-python3-cachetools_5.0.0-1_all.deb ... 2024-09-16T03:40:35.776 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-09-16T03:40:35.883 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-cachetools. 2024-09-16T03:40:35.911 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../083-python3-cachetools_5.0.0-1_all.deb ... 2024-09-16T03:40:35.950 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-09-16T03:40:36.053 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-pyasn1. 2024-09-16T03:40:36.070 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../084-python3-pyasn1_0.4.8-1_all.deb ... 2024-09-16T03:40:36.112 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-09-16T03:40:36.211 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-pyasn1. 2024-09-16T03:40:36.227 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../084-python3-pyasn1_0.4.8-1_all.deb ... 2024-09-16T03:40:36.264 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-09-16T03:40:36.432 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-09-16T03:40:36.460 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../085-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-09-16T03:40:36.498 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-09-16T03:40:36.573 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-09-16T03:40:36.600 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../085-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-09-16T03:40:36.648 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-09-16T03:40:36.793 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-rsa. 2024-09-16T03:40:36.821 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../086-python3-rsa_4.8-1_all.deb ... 2024-09-16T03:40:36.859 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-rsa (4.8-1) ... 2024-09-16T03:40:36.959 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-rsa. 2024-09-16T03:40:36.987 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../086-python3-rsa_4.8-1_all.deb ... 2024-09-16T03:40:37.026 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-rsa (4.8-1) ... 2024-09-16T03:40:37.214 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-google-auth. 2024-09-16T03:40:37.241 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../087-python3-google-auth_1.5.1-3_all.deb ... 2024-09-16T03:40:37.280 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-09-16T03:40:37.346 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-google-auth. 2024-09-16T03:40:37.374 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../087-python3-google-auth_1.5.1-3_all.deb ... 2024-09-16T03:40:37.412 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-09-16T03:40:37.600 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-09-16T03:40:37.628 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../088-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-09-16T03:40:37.666 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-16T03:40:37.741 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-09-16T03:40:37.769 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../088-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-09-16T03:40:37.807 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-16T03:40:37.953 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-websocket. 2024-09-16T03:40:37.980 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../089-python3-websocket_1.2.3-1_all.deb ... 2024-09-16T03:40:38.019 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-09-16T03:40:38.094 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-websocket. 2024-09-16T03:40:38.121 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../089-python3-websocket_1.2.3-1_all.deb ... 2024-09-16T03:40:38.152 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-09-16T03:40:38.314 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-kubernetes. 2024-09-16T03:40:38.342 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../090-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-09-16T03:40:38.381 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-16T03:40:38.438 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-kubernetes. 2024-09-16T03:40:38.466 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../090-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-09-16T03:40:38.505 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-16T03:40:39.028 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-09-16T03:40:39.056 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../091-ceph-mgr-k8sevents_19.1.1-326-g8921c8ef-1jammy_all.deb ... 2024-09-16T03:40:39.094 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking ceph-mgr-k8sevents (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:40:39.136 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-09-16T03:40:39.152 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../091-ceph-mgr-k8sevents_19.1.1-326-g8921c8ef-1jammy_all.deb ... 2024-09-16T03:40:39.194 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking ceph-mgr-k8sevents (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:40:39.414 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package libonig5:amd64. 2024-09-16T03:40:39.441 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../092-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-09-16T03:40:39.480 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-16T03:40:39.513 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package libonig5:amd64. 2024-09-16T03:40:39.541 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../092-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-09-16T03:40:39.580 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-16T03:40:39.825 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package libjq1:amd64. 2024-09-16T03:40:39.853 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../093-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-09-16T03:40:39.866 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package libjq1:amd64. 2024-09-16T03:40:39.891 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-16T03:40:39.894 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../093-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-09-16T03:40:39.933 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-16T03:40:40.111 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package jq. 2024-09-16T03:40:40.127 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../094-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-09-16T03:40:40.161 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-09-16T03:40:40.219 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package jq. 2024-09-16T03:40:40.247 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../094-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-09-16T03:40:40.285 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-09-16T03:40:40.388 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package socat. 2024-09-16T03:40:40.405 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../095-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-09-16T03:40:40.447 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-09-16T03:40:40.538 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package socat. 2024-09-16T03:40:40.557 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../095-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-09-16T03:40:40.596 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-09-16T03:40:40.875 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package xmlstarlet. 2024-09-16T03:40:40.900 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package xmlstarlet. 2024-09-16T03:40:40.903 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../096-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-09-16T03:40:40.928 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../096-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-09-16T03:40:40.966 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-09-16T03:40:40.974 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-09-16T03:40:41.329 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package ceph-test. 2024-09-16T03:40:41.357 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../097-ceph-test_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T03:40:41.362 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package ceph-test. 2024-09-16T03:40:41.378 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../097-ceph-test_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T03:40:41.387 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking ceph-test (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:40:41.420 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking ceph-test (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:40:43.650 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package ceph-volume. 2024-09-16T03:40:43.677 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../098-ceph-volume_19.1.1-326-g8921c8ef-1jammy_all.deb ... 2024-09-16T03:40:43.716 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking ceph-volume (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:40:43.834 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package ceph-volume. 2024-09-16T03:40:43.861 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../098-ceph-volume_19.1.1-326-g8921c8ef-1jammy_all.deb ... 2024-09-16T03:40:43.900 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking ceph-volume (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:40:44.029 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package libcephfs-dev. 2024-09-16T03:40:44.045 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../099-libcephfs-dev_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T03:40:44.087 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking libcephfs-dev (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:40:44.245 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package libcephfs-dev. 2024-09-16T03:40:44.273 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../099-libcephfs-dev_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T03:40:44.320 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking libcephfs-dev (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:40:44.415 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package lua-socket:amd64. 2024-09-16T03:40:44.443 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../100-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-09-16T03:40:44.481 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-16T03:40:44.832 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package lua-socket:amd64. 2024-09-16T03:40:44.860 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../100-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-09-16T03:40:44.898 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-16T03:40:44.947 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package lua-sec:amd64. 2024-09-16T03:40:44.975 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../101-lua-sec_1.0.2-1_amd64.deb ... 2024-09-16T03:40:45.011 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-09-16T03:40:45.227 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package lua-sec:amd64. 2024-09-16T03:40:45.256 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../101-lua-sec_1.0.2-1_amd64.deb ... 2024-09-16T03:40:45.294 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-09-16T03:40:45.607 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package nvme-cli. 2024-09-16T03:40:45.634 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../102-nvme-cli_1.16-3ubuntu0.3_amd64.deb ... 2024-09-16T03:40:45.673 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking nvme-cli (1.16-3ubuntu0.3) ... 2024-09-16T03:40:45.748 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package nvme-cli. 2024-09-16T03:40:45.776 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../102-nvme-cli_1.16-3ubuntu0.3_amd64.deb ... 2024-09-16T03:40:45.823 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking nvme-cli (1.16-3ubuntu0.3) ... 2024-09-16T03:40:46.087 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package pkg-config. 2024-09-16T03:40:46.114 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../103-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-09-16T03:40:46.153 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-09-16T03:40:46.286 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package pkg-config. 2024-09-16T03:40:46.315 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../103-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-09-16T03:40:46.353 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-09-16T03:40:46.389 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-09-16T03:40:46.405 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../104-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-09-16T03:40:46.439 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-09-16T03:40:46.614 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-09-16T03:40:46.643 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../104-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-09-16T03:40:46.681 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-09-16T03:40:46.818 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-iniconfig. 2024-09-16T03:40:46.845 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../105-python3-iniconfig_1.1.1-2_all.deb ... 2024-09-16T03:40:46.875 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-09-16T03:40:47.117 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-iniconfig. 2024-09-16T03:40:47.134 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../105-python3-iniconfig_1.1.1-2_all.deb ... 2024-09-16T03:40:47.154 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-pastescript. 2024-09-16T03:40:47.175 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-09-16T03:40:47.181 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../106-python3-pastescript_2.0.2-4_all.deb ... 2024-09-16T03:40:47.220 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-09-16T03:40:47.478 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-pastescript. 2024-09-16T03:40:47.506 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../106-python3-pastescript_2.0.2-4_all.deb ... 2024-09-16T03:40:47.544 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-09-16T03:40:47.557 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-pluggy. 2024-09-16T03:40:47.584 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../107-python3-pluggy_0.13.0-7.1_all.deb ... 2024-09-16T03:40:47.623 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-09-16T03:40:47.842 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-pluggy. 2024-09-16T03:40:47.870 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../107-python3-pluggy_0.13.0-7.1_all.deb ... 2024-09-16T03:40:47.900 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-09-16T03:40:47.934 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-psutil. 2024-09-16T03:40:47.962 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../108-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-09-16T03:40:48.001 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-09-16T03:40:48.194 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-psutil. 2024-09-16T03:40:48.222 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../108-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-09-16T03:40:48.261 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-09-16T03:40:48.321 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-py. 2024-09-16T03:40:48.348 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../109-python3-py_1.10.0-1_all.deb ... 2024-09-16T03:40:48.387 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-py (1.10.0-1) ... 2024-09-16T03:40:48.622 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-py. 2024-09-16T03:40:48.638 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../109-python3-py_1.10.0-1_all.deb ... 2024-09-16T03:40:48.680 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-py (1.10.0-1) ... 2024-09-16T03:40:48.716 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-pygments. 2024-09-16T03:40:48.744 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../110-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-09-16T03:40:48.782 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-09-16T03:40:48.984 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-pygments. 2024-09-16T03:40:49.012 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../110-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-09-16T03:40:49.050 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-09-16T03:40:49.262 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-pyinotify. 2024-09-16T03:40:49.289 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../111-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-09-16T03:40:49.319 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-09-16T03:40:49.488 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-pyinotify. 2024-09-16T03:40:49.516 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../111-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-09-16T03:40:49.554 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-09-16T03:40:49.639 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-toml. 2024-09-16T03:40:49.669 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../112-python3-toml_0.10.2-1_all.deb ... 2024-09-16T03:40:49.705 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-09-16T03:40:49.873 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-toml. 2024-09-16T03:40:49.901 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../112-python3-toml_0.10.2-1_all.deb ... 2024-09-16T03:40:49.940 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-09-16T03:40:50.026 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-pytest. 2024-09-16T03:40:50.053 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../113-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-09-16T03:40:50.092 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-09-16T03:40:50.243 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-pytest. 2024-09-16T03:40:50.271 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../113-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-09-16T03:40:50.309 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-09-16T03:40:50.437 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package python3-simplejson. 2024-09-16T03:40:50.465 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../114-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-09-16T03:40:50.504 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-09-16T03:40:50.663 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package python3-simplejson. 2024-09-16T03:40:50.691 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../114-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-09-16T03:40:50.756 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-09-16T03:40:50.762 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-09-16T03:40:50.784 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../115-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-09-16T03:40:50.831 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-09-16T03:40:51.032 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-09-16T03:40:51.048 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../115-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-09-16T03:40:51.090 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-09-16T03:40:51.594 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package radosgw. 2024-09-16T03:40:51.621 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../116-radosgw_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T03:40:51.660 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking radosgw (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:40:52.053 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package radosgw. 2024-09-16T03:40:52.081 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../116-radosgw_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T03:40:52.119 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking radosgw (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:40:52.463 INFO:teuthology.orchestra.run.smithi154.stdout:Selecting previously unselected package rbd-fuse. 2024-09-16T03:40:52.491 INFO:teuthology.orchestra.run.smithi154.stdout:Preparing to unpack .../117-rbd-fuse_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T03:40:52.529 INFO:teuthology.orchestra.run.smithi154.stdout:Unpacking rbd-fuse (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:40:52.834 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-09-16T03:40:52.955 INFO:teuthology.orchestra.run.smithi042.stdout:Selecting previously unselected package rbd-fuse. 2024-09-16T03:40:52.972 INFO:teuthology.orchestra.run.smithi042.stdout:Preparing to unpack .../117-rbd-fuse_19.1.1-326-g8921c8ef-1jammy_amd64.deb ... 2024-09-16T03:40:53.013 INFO:teuthology.orchestra.run.smithi042.stdout:Unpacking rbd-fuse (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:40:53.093 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-16T03:40:53.194 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up nvme-cli (1.16-3ubuntu0.3) ... 2024-09-16T03:40:53.327 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-09-16T03:40:53.463 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-09-16T03:40:53.577 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-16T03:40:53.703 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up nvme-cli (1.16-3ubuntu0.3) ... 2024-09-16T03:40:53.776 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-09-16T03:40:53.955 INFO:teuthology.orchestra.run.smithi042.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-09-16T03:40:54.213 INFO:teuthology.orchestra.run.smithi154.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-09-16T03:40:54.271 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up cephadm (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:40:54.278 INFO:teuthology.orchestra.run.smithi042.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-09-16T03:40:54.685 INFO:teuthology.orchestra.run.smithi042.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-09-16T03:40:54.754 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up cephadm (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:40:54.829 INFO:teuthology.orchestra.run.smithi154.stdout:Adding system user cephadm....done 2024-09-16T03:40:54.977 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-16T03:40:55.246 INFO:teuthology.orchestra.run.smithi042.stdout:Adding system user cephadm....done 2024-09-16T03:40:55.259 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-09-16T03:40:55.394 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-16T03:40:55.678 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-09-16T03:40:55.771 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-09-16T03:40:55.906 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-09-16T03:40:55.946 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-09-16T03:40:56.047 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-09-16T03:40:56.157 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-09-16T03:40:56.307 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-09-16T03:40:56.417 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-09-16T03:40:56.518 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-py (1.10.0-1) ... 2024-09-16T03:40:56.550 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-09-16T03:40:56.676 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-py (1.10.0-1) ... 2024-09-16T03:40:56.811 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-09-16T03:40:57.002 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-09-16T03:40:57.155 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-09-16T03:40:57.406 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-09-16T03:40:57.429 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-09-16T03:40:57.682 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-09-16T03:40:57.714 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-09-16T03:40:57.959 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-09-16T03:40:58.015 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-09-16T03:40:58.202 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-09-16T03:40:58.283 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-09-16T03:40:58.453 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-09-16T03:40:58.542 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-09-16T03:40:58.729 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-ceph-argparse (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:40:58.810 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-09-16T03:40:58.971 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-16T03:40:59.078 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-ceph-argparse (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:40:59.106 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-16T03:40:59.232 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-09-16T03:40:59.358 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-09-16T03:40:59.387 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-16T03:40:59.538 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-16T03:40:59.617 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-09-16T03:40:59.698 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-09-16T03:40:59.849 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-09-16T03:41:00.135 INFO:teuthology.orchestra.run.smithi154.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-09-16T03:41:00.150 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-09-16T03:41:00.157 INFO:teuthology.orchestra.run.smithi154.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-09-16T03:41:00.212 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-16T03:41:00.313 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-09-16T03:41:00.551 INFO:teuthology.orchestra.run.smithi042.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-09-16T03:41:00.850 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-16T03:41:00.850 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-09-16T03:41:00.850 INFO:teuthology.orchestra.run.smithi042.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-09-16T03:41:00.850 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-16T03:41:00.851 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-09-16T03:41:01.067 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-09-16T03:41:01.222 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-16T03:41:01.326 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-09-16T03:41:01.373 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-09-16T03:41:01.636 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up zip (3.0-12build2) ... 2024-09-16T03:41:01.674 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-09-16T03:41:01.770 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-09-16T03:41:01.950 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-09-16T03:41:02.260 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up zip (3.0-12build2) ... 2024-09-16T03:41:02.394 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-09-16T03:41:02.463 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-09-16T03:41:02.725 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-09-16T03:41:02.851 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-09-16T03:41:02.977 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-16T03:41:03.079 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-09-16T03:41:03.303 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-09-16T03:41:03.356 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-09-16T03:41:03.465 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-09-16T03:41:03.574 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-16T03:41:03.687 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-09-16T03:41:03.859 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-09-16T03:41:03.922 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-09-16T03:41:04.252 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-09-16T03:41:04.298 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-16T03:41:04.503 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-09-16T03:41:04.600 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-16T03:41:04.921 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-16T03:41:04.934 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-09-16T03:41:05.185 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-09-16T03:41:05.231 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-16T03:41:05.345 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-ceph-common (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:41:05.590 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-09-16T03:41:05.646 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-idna (3.3-1ubuntu0.1) ... 2024-09-16T03:41:05.841 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-09-16T03:41:05.955 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-16T03:41:05.967 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-ceph-common (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:41:06.277 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-idna (3.3-1ubuntu0.1) ... 2024-09-16T03:41:06.628 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-16T03:41:07.145 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-09-16T03:41:07.396 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T03:41:07.530 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-toml (0.10.2-1) ... 2024-09-16T03:41:07.807 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-16T03:41:07.842 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-09-16T03:41:07.941 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-09-16T03:41:08.067 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-16T03:41:08.102 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T03:41:08.253 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-toml (0.10.2-1) ... 2024-09-16T03:41:08.368 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-09-16T03:41:08.538 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-16T03:41:08.628 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-09-16T03:41:08.689 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-09-16T03:41:08.840 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-16T03:41:08.887 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T03:41:09.013 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-09-16T03:41:09.200 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-09-16T03:41:09.315 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-09-16T03:41:09.468 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-09-16T03:41:09.566 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-09-16T03:41:09.727 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T03:41:09.825 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-09-16T03:41:09.861 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-09-16T03:41:10.084 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-09-16T03:41:10.213 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-09-16T03:41:10.327 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-09-16T03:41:10.489 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-09-16T03:41:10.620 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-16T03:41:10.746 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-09-16T03:41:10.756 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-09-16T03:41:11.024 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-09-16T03:41:11.030 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-16T03:41:11.157 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-16T03:41:11.292 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-09-16T03:41:11.466 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-16T03:41:11.568 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-16T03:41:11.694 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-09-16T03:41:11.759 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-09-16T03:41:12.002 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-16T03:41:12.003 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-16T03:41:12.119 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-09-16T03:41:12.154 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-16T03:41:12.220 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-09-16T03:41:12.293 INFO:teuthology.orchestra.run.smithi154.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-09-16T03:41:12.472 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-16T03:41:12.555 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-16T03:41:12.681 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-09-16T03:41:12.823 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-09-16T03:41:13.091 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-09-16T03:41:13.108 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-16T03:41:13.251 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-09-16T03:41:13.342 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up lua-any (27ubuntu1) ... 2024-09-16T03:41:13.377 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-09-16T03:41:13.452 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-portend (3.0.0-1) ... 2024-09-16T03:41:13.467 INFO:teuthology.orchestra.run.smithi042.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-09-16T03:41:13.704 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T03:41:13.720 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-16T03:41:13.804 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-16T03:41:13.871 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-09-16T03:41:14.121 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-09-16T03:41:14.281 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-09-16T03:41:14.414 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-16T03:41:14.557 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up lua-any (27ubuntu1) ... 2024-09-16T03:41:14.658 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-portend (3.0.0-1) ... 2024-09-16T03:41:14.690 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-09-16T03:41:14.808 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-rsa (4.8-1) ... 2024-09-16T03:41:14.917 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T03:41:15.043 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-16T03:41:15.076 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-09-16T03:41:15.344 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-16T03:41:15.369 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-09-16T03:41:15.672 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-09-16T03:41:15.704 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-16T03:41:15.988 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-09-16T03:41:16.082 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-09-16T03:41:16.139 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-rsa (4.8-1) ... 2024-09-16T03:41:16.359 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-09-16T03:41:16.424 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-09-16T03:41:16.692 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-16T03:41:16.694 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-16T03:41:16.821 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up librados2 (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:41:16.947 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up libsqlite3-mod-ceph (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:41:17.062 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-09-16T03:41:17.073 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-09-16T03:41:17.401 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up libcephfs2 (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:41:17.488 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-09-16T03:41:17.510 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up libradosstriper1 (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:41:17.636 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-09-16T03:41:17.798 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-09-16T03:41:17.913 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up librbd1 (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:41:18.039 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up ceph-mgr-modules-core (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:41:18.151 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-16T03:41:18.165 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up ceph-fuse (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:41:18.280 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up librados2 (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:41:18.381 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-16T03:41:18.381 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-16T03:41:18.420 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up libsqlite3-mod-ceph (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:41:18.546 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-09-16T03:41:18.857 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up libcephfs2 (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:41:18.858 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up libcephfs-dev (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:41:18.958 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up libradosstriper1 (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:41:19.017 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-rados (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:41:19.084 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-09-16T03:41:19.144 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up librgw2 (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:41:19.278 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-16T03:41:19.369 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up librbd1 (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:41:19.503 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up ceph-mgr-modules-core (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:41:19.654 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up ceph-fuse (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:41:19.890 INFO:teuthology.orchestra.run.smithi042.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-16T03:41:19.890 INFO:teuthology.orchestra.run.smithi042.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-16T03:41:20.381 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up libcephfs-dev (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:41:20.466 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-rbd (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:41:20.541 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-rados (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:41:20.576 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up rbd-fuse (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:41:20.676 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-rgw (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:41:20.794 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up python3-cephfs (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:41:20.920 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up ceph-common (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:41:21.051 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up librgw2 (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:41:21.177 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-16T03:41:21.772 INFO:teuthology.orchestra.run.smithi154.stdout:Adding group ceph....done 2024-09-16T03:41:22.131 INFO:teuthology.orchestra.run.smithi154.stdout:Adding system user ceph....done 2024-09-16T03:41:22.300 INFO:teuthology.orchestra.run.smithi154.stdout:Setting system user ceph properties....done 2024-09-16T03:41:22.314 INFO:teuthology.orchestra.run.smithi154.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-09-16T03:41:22.451 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-09-16T03:41:22.455 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-rbd (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:41:22.564 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up rbd-fuse (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:41:22.666 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-rgw (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:41:22.775 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up python3-cephfs (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:41:22.796 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-09-16T03:41:22.901 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up ceph-common (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:41:23.263 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up ceph-test (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:41:23.389 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up radosgw (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:41:23.594 INFO:teuthology.orchestra.run.smithi042.stdout:Adding group ceph....done 2024-09-16T03:41:23.979 INFO:teuthology.orchestra.run.smithi042.stdout:Adding system user ceph....done 2024-09-16T03:41:24.011 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-16T03:41:24.011 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-16T03:41:24.148 INFO:teuthology.orchestra.run.smithi042.stdout:Setting system user ceph properties....done 2024-09-16T03:41:24.165 INFO:teuthology.orchestra.run.smithi042.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-09-16T03:41:24.320 INFO:teuthology.orchestra.run.smithi042.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-09-16T03:41:24.500 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up ceph-base (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:41:24.681 INFO:teuthology.orchestra.run.smithi042.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-09-16T03:41:24.839 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-09-16T03:41:25.136 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up ceph-test (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:41:25.262 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up radosgw (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:41:25.301 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up ceph-mds (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:41:25.512 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-16T03:41:25.512 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-16T03:41:25.737 INFO:teuthology.orchestra.run.smithi042.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-16T03:41:25.737 INFO:teuthology.orchestra.run.smithi042.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-16T03:41:26.003 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up ceph-mgr (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:41:26.206 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up ceph-base (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:41:26.221 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-16T03:41:26.221 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-16T03:41:26.649 INFO:teuthology.orchestra.run.smithi042.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-09-16T03:41:27.099 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up ceph-mds (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:41:27.214 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up ceph-osd (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:41:27.342 INFO:teuthology.orchestra.run.smithi042.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-16T03:41:27.342 INFO:teuthology.orchestra.run.smithi042.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-16T03:41:27.479 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-16T03:41:27.479 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-16T03:41:27.802 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up ceph-mgr (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:41:28.021 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up ceph-mgr-k8sevents (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:41:28.028 INFO:teuthology.orchestra.run.smithi042.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-16T03:41:28.028 INFO:teuthology.orchestra.run.smithi042.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-16T03:41:28.147 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up ceph-mgr-diskprediction-local (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:41:28.315 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up ceph-mon (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:41:28.470 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up ceph-osd (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:41:28.512 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-16T03:41:28.512 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-16T03:41:28.766 INFO:teuthology.orchestra.run.smithi042.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-16T03:41:28.766 INFO:teuthology.orchestra.run.smithi042.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-16T03:41:29.046 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up ceph-mgr-cephadm (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:41:29.213 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up ceph-mgr-k8sevents (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:41:29.306 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up ceph (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:41:29.364 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up ceph-mgr-diskprediction-local (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:41:29.497 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up ceph-mgr-dashboard (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:41:29.523 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up ceph-mon (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:41:29.725 INFO:teuthology.orchestra.run.smithi042.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-16T03:41:29.725 INFO:teuthology.orchestra.run.smithi042.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-16T03:41:29.750 INFO:teuthology.orchestra.run.smithi154.stdout:Setting up ceph-volume (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:41:30.158 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up ceph-mgr-cephadm (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:41:30.208 INFO:teuthology.orchestra.run.smithi154.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-09-16T03:41:30.309 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up ceph (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:41:30.317 INFO:teuthology.orchestra.run.smithi154.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-09-16T03:41:30.435 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up ceph-mgr-dashboard (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:41:30.453 INFO:teuthology.orchestra.run.smithi154.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-09-16T03:41:30.586 INFO:teuthology.orchestra.run.smithi042.stdout:Setting up ceph-volume (19.1.1-326-g8921c8ef-1jammy) ... 2024-09-16T03:41:30.964 INFO:teuthology.orchestra.run.smithi042.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-09-16T03:41:31.073 INFO:teuthology.orchestra.run.smithi042.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-09-16T03:41:31.217 INFO:teuthology.orchestra.run.smithi042.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-09-16T03:41:31.219 INFO:teuthology.orchestra.run.smithi154.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-09-16T03:41:31.865 INFO:teuthology.orchestra.run.smithi154.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-09-16T03:41:32.327 DEBUG:teuthology.parallel:result is None 2024-09-16T03:41:32.401 INFO:teuthology.orchestra.run.smithi042.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-09-16T03:41:33.098 INFO:teuthology.orchestra.run.smithi042.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-09-16T03:41:33.103 DEBUG:teuthology.parallel:result is None 2024-09-16T03:41:33.104 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=8921c8ef947807b717db041d85f0761b7914d5fc 2024-09-16T03:41:33.249 DEBUG:teuthology.orchestra.run.smithi042:> dpkg-query -W -f '${Version}' ceph 2024-09-16T03:41:33.268 INFO:teuthology.orchestra.run.smithi042.stdout:19.1.1-326-g8921c8ef-1jammy 2024-09-16T03:41:33.269 INFO:teuthology.packaging:The installed version of ceph is 19.1.1-326-g8921c8ef-1jammy 2024-09-16T03:41:33.269 INFO:teuthology.task.install:The correct ceph version 19.1.1-326-g8921c8ef-1jammy is installed. 2024-09-16T03:41:33.271 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=8921c8ef947807b717db041d85f0761b7914d5fc 2024-09-16T03:41:33.411 DEBUG:teuthology.orchestra.run.smithi154:> dpkg-query -W -f '${Version}' ceph 2024-09-16T03:41:33.430 INFO:teuthology.orchestra.run.smithi154.stdout:19.1.1-326-g8921c8ef-1jammy 2024-09-16T03:41:33.430 INFO:teuthology.packaging:The installed version of ceph is 19.1.1-326-g8921c8ef-1jammy 2024-09-16T03:41:33.430 INFO:teuthology.task.install:The correct ceph version 19.1.1-326-g8921c8ef-1jammy is installed. 2024-09-16T03:41:33.432 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-16T03:41:33.432 DEBUG:teuthology.orchestra.run.smithi042:> set -ex 2024-09-16T03:41:33.432 DEBUG:teuthology.orchestra.run.smithi042:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-16T03:41:33.447 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-09-16T03:41:33.447 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-16T03:41:33.484 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-16T03:41:33.484 DEBUG:teuthology.orchestra.run.smithi042:> set -ex 2024-09-16T03:41:33.485 DEBUG:teuthology.orchestra.run.smithi042:> sudo dd of=/usr/bin/daemon-helper 2024-09-16T03:41:33.502 DEBUG:teuthology.orchestra.run.smithi042:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-16T03:41:33.558 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-09-16T03:41:33.558 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd of=/usr/bin/daemon-helper 2024-09-16T03:41:33.572 DEBUG:teuthology.orchestra.run.smithi154:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-16T03:41:33.627 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-16T03:41:33.628 DEBUG:teuthology.orchestra.run.smithi042:> set -ex 2024-09-16T03:41:33.628 DEBUG:teuthology.orchestra.run.smithi042:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-16T03:41:33.641 DEBUG:teuthology.orchestra.run.smithi042:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-16T03:41:33.694 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-09-16T03:41:33.694 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-16T03:41:33.707 DEBUG:teuthology.orchestra.run.smithi154:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-16T03:41:33.767 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-16T03:41:33.768 DEBUG:teuthology.orchestra.run.smithi042:> set -ex 2024-09-16T03:41:33.768 DEBUG:teuthology.orchestra.run.smithi042:> sudo dd of=/usr/bin/stdin-killer 2024-09-16T03:41:33.781 DEBUG:teuthology.orchestra.run.smithi042:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-16T03:41:33.838 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-09-16T03:41:33.838 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd of=/usr/bin/stdin-killer 2024-09-16T03:41:33.851 DEBUG:teuthology.orchestra.run.smithi154:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-16T03:41:33.907 INFO:teuthology.run_tasks:Running task ceph... 2024-09-16T03:41:33.988 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-09-16T03:41:33.988 DEBUG:teuthology.orchestra.run.smithi042:> sudo chmod 777 /var/log/ceph 2024-09-16T03:41:33.991 DEBUG:teuthology.orchestra.run.smithi154:> sudo chmod 777 /var/log/ceph 2024-09-16T03:41:34.004 INFO:tasks.ceph:Disabling ceph logrotate... 2024-09-16T03:41:34.004 DEBUG:teuthology.orchestra.run.smithi042:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-16T03:41:34.048 DEBUG:teuthology.orchestra.run.smithi154:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-16T03:41:34.059 INFO:tasks.ceph:Creating extra log directories... 2024-09-16T03:41:34.060 DEBUG:teuthology.orchestra.run.smithi042:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-16T03:41:34.107 DEBUG:teuthology.orchestra.run.smithi154:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-16T03:41:34.129 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-09-16T03:41:34.129 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1}, 'global': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 5}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'lz4', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'ScrubResult', 'scrub mismatch', 'overall HEALTH_', '\\(MGR_DOWN\\)', '\\(PG_AVAILABILITY\\)', '\\(SLOW_OPS\\)', 'slow request', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(MON_DOWN\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_FULL\\)', '\\(SLOW_OPS\\)', '\\(MON_DOWN\\)', '\\(PG_', '\\(SMALLER_PGP_NUM\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-09-16T03:41:34.130 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7905897', 'branch': 'squid', 'description': 'rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} thrashers/one workloads/rados_api_tests}', 'email': 'ceph-qa@ceph.com', 'first_in_suite': False, 'job_id': '7905897', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 3, 'size': 10}}], 'os_type': 'ubuntu', 'os_version': '22.04', 'overrides': {'admin_socket': {'branch': 'squid'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1}, 'global': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 5}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'lz4', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'ScrubResult', 'scrub mismatch', 'overall HEALTH_', '\\(MGR_DOWN\\)', '\\(PG_AVAILABILITY\\)', '\\(SLOW_OPS\\)', 'slow request', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(MON_DOWN\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_FULL\\)', '\\(SLOW_OPS\\)', '\\(MON_DOWN\\)', '\\(PG_', '\\(SMALLER_PGP_NUM\\)', 'slow request'], 'sha1': '8921c8ef947807b717db041d85f0761b7914d5fc'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '8921c8ef947807b717db041d85f0761b7914d5fc'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'squid', 'sha1': 'ad35db06e8924912f01e32b35576297c0f1352b4'}}, 'owner': 'scheduled_teuthology@teuthology', 'priority': 700, 'repo': 'https://git.ceph.com/ceph.git', 'roles': [['mon.a', 'mon.c', 'osd.0', 'osd.1', 'osd.2'], ['mon.b', 'mgr.x', 'osd.3', 'osd.4', 'osd.5', 'client.0']], 'seed': 5616, 'sha1': '8921c8ef947807b717db041d85f0761b7914d5fc', 'sleep_before_teardown': 0, 'subset': '15927/100000', 'suite': 'rados', 'suite_branch': 'squid', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph_ad35db06e8924912f01e32b35576297c0f1352b4/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph.git', 'suite_sha1': 'ad35db06e8924912f01e32b35576297c0f1352b4', 'targets': {'smithi042.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBXDiWN6PE9DDlOAE2tzOYpWt1M0jEUAWTMW+5GD7AygeyBTve7KeeVgVAiORSfXTBXVrAwHZS2HPCavrwkjJ+0=', 'smithi154.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGWwvZ/6aduK1wMTWLfwMlqnrSonkUJTL7pZ4qzv27JiPXURKoUx+YOgY96cMRLv5YgRRJg+qgSaAicMAKckoXs='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'mon_thrash': {'revive_delay': 20, 'thrash_delay': 1}}, {'workunit': {'clients': {'client.0': ['rados/test.sh unit_test_scan']}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-09-15_21:00:02', 'tube': 'smithi', 'user': 'teuthology', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856'} 2024-09-16T03:41:34.130 DEBUG:teuthology.orchestra.run.smithi042:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-16T03:41:34.163 DEBUG:teuthology.orchestra.run.smithi154:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-16T03:41:34.169 DEBUG:teuthology.orchestra.run.smithi042:> sudo install -d -m0777 -- /var/run/ceph 2024-09-16T03:41:34.211 DEBUG:teuthology.orchestra.run.smithi154:> sudo install -d -m0777 -- /var/run/ceph 2024-09-16T03:41:34.224 DEBUG:teuthology.orchestra.run.smithi042:> set -ex 2024-09-16T03:41:34.225 DEBUG:teuthology.orchestra.run.smithi042:> dd if=/scratch_devs of=/dev/stdout 2024-09-16T03:41:34.270 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-16T03:41:34.271 DEBUG:teuthology.orchestra.run.smithi042:> stat /dev/vg_nvme/lv_1 2024-09-16T03:41:34.319 INFO:teuthology.orchestra.run.smithi042.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-16T03:41:34.319 INFO:teuthology.orchestra.run.smithi042.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T03:41:34.319 INFO:teuthology.orchestra.run.smithi042.stdout:Device: 5h/5d Inode: 889 Links: 1 2024-09-16T03:41:34.319 INFO:teuthology.orchestra.run.smithi042.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T03:41:34.319 INFO:teuthology.orchestra.run.smithi042.stdout:Access: 2024-09-16 03:36:04.764257684 +0000 2024-09-16T03:41:34.319 INFO:teuthology.orchestra.run.smithi042.stdout:Modify: 2024-09-16 03:36:04.512259936 +0000 2024-09-16T03:41:34.319 INFO:teuthology.orchestra.run.smithi042.stdout:Change: 2024-09-16 03:36:04.512259936 +0000 2024-09-16T03:41:34.319 INFO:teuthology.orchestra.run.smithi042.stdout: Birth: - 2024-09-16T03:41:34.320 DEBUG:teuthology.orchestra.run.smithi042:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-16T03:41:34.373 INFO:teuthology.orchestra.run.smithi042.stderr:1+0 records in 2024-09-16T03:41:34.373 INFO:teuthology.orchestra.run.smithi042.stderr:1+0 records out 2024-09-16T03:41:34.373 INFO:teuthology.orchestra.run.smithi042.stderr:512 bytes copied, 0.000327034 s, 1.6 MB/s 2024-09-16T03:41:34.375 DEBUG:teuthology.orchestra.run.smithi042:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-16T03:41:34.424 DEBUG:teuthology.orchestra.run.smithi042:> stat /dev/vg_nvme/lv_2 2024-09-16T03:41:34.474 INFO:teuthology.orchestra.run.smithi042.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-16T03:41:34.475 INFO:teuthology.orchestra.run.smithi042.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T03:41:34.475 INFO:teuthology.orchestra.run.smithi042.stdout:Device: 5h/5d Inode: 899 Links: 1 2024-09-16T03:41:34.475 INFO:teuthology.orchestra.run.smithi042.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T03:41:34.475 INFO:teuthology.orchestra.run.smithi042.stdout:Access: 2024-09-16 03:36:05.252253327 +0000 2024-09-16T03:41:34.475 INFO:teuthology.orchestra.run.smithi042.stdout:Modify: 2024-09-16 03:36:05.004255542 +0000 2024-09-16T03:41:34.475 INFO:teuthology.orchestra.run.smithi042.stdout:Change: 2024-09-16 03:36:05.004255542 +0000 2024-09-16T03:41:34.475 INFO:teuthology.orchestra.run.smithi042.stdout: Birth: - 2024-09-16T03:41:34.475 DEBUG:teuthology.orchestra.run.smithi042:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-16T03:41:34.528 INFO:teuthology.orchestra.run.smithi042.stderr:1+0 records in 2024-09-16T03:41:34.528 INFO:teuthology.orchestra.run.smithi042.stderr:1+0 records out 2024-09-16T03:41:34.529 INFO:teuthology.orchestra.run.smithi042.stderr:512 bytes copied, 0.000208612 s, 2.5 MB/s 2024-09-16T03:41:34.530 DEBUG:teuthology.orchestra.run.smithi042:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-16T03:41:34.579 DEBUG:teuthology.orchestra.run.smithi042:> stat /dev/vg_nvme/lv_3 2024-09-16T03:41:34.626 INFO:teuthology.orchestra.run.smithi042.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-16T03:41:34.626 INFO:teuthology.orchestra.run.smithi042.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T03:41:34.626 INFO:teuthology.orchestra.run.smithi042.stdout:Device: 5h/5d Inode: 923 Links: 1 2024-09-16T03:41:34.627 INFO:teuthology.orchestra.run.smithi042.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T03:41:34.627 INFO:teuthology.orchestra.run.smithi042.stdout:Access: 2024-09-16 03:36:05.744248931 +0000 2024-09-16T03:41:34.627 INFO:teuthology.orchestra.run.smithi042.stdout:Modify: 2024-09-16 03:36:05.500251111 +0000 2024-09-16T03:41:34.627 INFO:teuthology.orchestra.run.smithi042.stdout:Change: 2024-09-16 03:36:05.500251111 +0000 2024-09-16T03:41:34.627 INFO:teuthology.orchestra.run.smithi042.stdout: Birth: - 2024-09-16T03:41:34.627 DEBUG:teuthology.orchestra.run.smithi042:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-16T03:41:34.681 INFO:teuthology.orchestra.run.smithi042.stderr:1+0 records in 2024-09-16T03:41:34.681 INFO:teuthology.orchestra.run.smithi042.stderr:1+0 records out 2024-09-16T03:41:34.681 INFO:teuthology.orchestra.run.smithi042.stderr:512 bytes copied, 0.000296591 s, 1.7 MB/s 2024-09-16T03:41:34.683 DEBUG:teuthology.orchestra.run.smithi042:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-16T03:41:34.732 DEBUG:teuthology.orchestra.run.smithi042:> stat /dev/vg_nvme/lv_4 2024-09-16T03:41:34.778 INFO:teuthology.orchestra.run.smithi042.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-16T03:41:34.778 INFO:teuthology.orchestra.run.smithi042.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T03:41:34.778 INFO:teuthology.orchestra.run.smithi042.stdout:Device: 5h/5d Inode: 934 Links: 1 2024-09-16T03:41:34.778 INFO:teuthology.orchestra.run.smithi042.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T03:41:34.779 INFO:teuthology.orchestra.run.smithi042.stdout:Access: 2024-09-16 03:36:06.236244537 +0000 2024-09-16T03:41:34.779 INFO:teuthology.orchestra.run.smithi042.stdout:Modify: 2024-09-16 03:36:05.984246788 +0000 2024-09-16T03:41:34.779 INFO:teuthology.orchestra.run.smithi042.stdout:Change: 2024-09-16 03:36:05.984246788 +0000 2024-09-16T03:41:34.779 INFO:teuthology.orchestra.run.smithi042.stdout: Birth: - 2024-09-16T03:41:34.779 DEBUG:teuthology.orchestra.run.smithi042:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-16T03:41:34.833 INFO:teuthology.orchestra.run.smithi042.stderr:1+0 records in 2024-09-16T03:41:34.833 INFO:teuthology.orchestra.run.smithi042.stderr:1+0 records out 2024-09-16T03:41:34.833 INFO:teuthology.orchestra.run.smithi042.stderr:512 bytes copied, 0.000234433 s, 2.2 MB/s 2024-09-16T03:41:34.834 DEBUG:teuthology.orchestra.run.smithi042:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-16T03:41:34.884 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-16T03:41:34.884 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-09-16T03:41:34.884 DEBUG:teuthology.orchestra.run.smithi154:> dd if=/scratch_devs of=/dev/stdout 2024-09-16T03:41:34.890 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-16T03:41:34.890 DEBUG:teuthology.orchestra.run.smithi154:> stat /dev/vg_nvme/lv_1 2024-09-16T03:41:34.936 INFO:teuthology.orchestra.run.smithi154.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-16T03:41:34.936 INFO:teuthology.orchestra.run.smithi154.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T03:41:34.936 INFO:teuthology.orchestra.run.smithi154.stdout:Device: 5h/5d Inode: 887 Links: 1 2024-09-16T03:41:34.936 INFO:teuthology.orchestra.run.smithi154.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T03:41:34.937 INFO:teuthology.orchestra.run.smithi154.stdout:Access: 2024-09-16 03:35:59.954323546 +0000 2024-09-16T03:41:34.937 INFO:teuthology.orchestra.run.smithi154.stdout:Modify: 2024-09-16 03:35:59.682325975 +0000 2024-09-16T03:41:34.937 INFO:teuthology.orchestra.run.smithi154.stdout:Change: 2024-09-16 03:35:59.682325975 +0000 2024-09-16T03:41:34.937 INFO:teuthology.orchestra.run.smithi154.stdout: Birth: - 2024-09-16T03:41:34.937 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-16T03:41:34.990 INFO:teuthology.orchestra.run.smithi154.stderr:1+0 records in 2024-09-16T03:41:34.991 INFO:teuthology.orchestra.run.smithi154.stderr:1+0 records out 2024-09-16T03:41:34.991 INFO:teuthology.orchestra.run.smithi154.stderr:512 bytes copied, 0.000421134 s, 1.2 MB/s 2024-09-16T03:41:34.992 DEBUG:teuthology.orchestra.run.smithi154:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-16T03:41:35.041 DEBUG:teuthology.orchestra.run.smithi154:> stat /dev/vg_nvme/lv_2 2024-09-16T03:41:35.091 INFO:teuthology.orchestra.run.smithi154.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-16T03:41:35.092 INFO:teuthology.orchestra.run.smithi154.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T03:41:35.092 INFO:teuthology.orchestra.run.smithi154.stdout:Device: 5h/5d Inode: 903 Links: 1 2024-09-16T03:41:35.092 INFO:teuthology.orchestra.run.smithi154.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T03:41:35.092 INFO:teuthology.orchestra.run.smithi154.stdout:Access: 2024-09-16 03:36:00.462319008 +0000 2024-09-16T03:41:35.092 INFO:teuthology.orchestra.run.smithi154.stdout:Modify: 2024-09-16 03:36:00.214321224 +0000 2024-09-16T03:41:35.092 INFO:teuthology.orchestra.run.smithi154.stdout:Change: 2024-09-16 03:36:00.214321224 +0000 2024-09-16T03:41:35.092 INFO:teuthology.orchestra.run.smithi154.stdout: Birth: - 2024-09-16T03:41:35.092 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-16T03:41:35.146 INFO:teuthology.orchestra.run.smithi154.stderr:1+0 records in 2024-09-16T03:41:35.146 INFO:teuthology.orchestra.run.smithi154.stderr:1+0 records out 2024-09-16T03:41:35.146 INFO:teuthology.orchestra.run.smithi154.stderr:512 bytes copied, 0.000408497 s, 1.3 MB/s 2024-09-16T03:41:35.147 DEBUG:teuthology.orchestra.run.smithi154:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-16T03:41:35.196 DEBUG:teuthology.orchestra.run.smithi154:> stat /dev/vg_nvme/lv_3 2024-09-16T03:41:35.243 INFO:teuthology.orchestra.run.smithi154.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-16T03:41:35.243 INFO:teuthology.orchestra.run.smithi154.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T03:41:35.243 INFO:teuthology.orchestra.run.smithi154.stdout:Device: 5h/5d Inode: 923 Links: 1 2024-09-16T03:41:35.244 INFO:teuthology.orchestra.run.smithi154.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T03:41:35.244 INFO:teuthology.orchestra.run.smithi154.stdout:Access: 2024-09-16 03:36:00.950314649 +0000 2024-09-16T03:41:35.244 INFO:teuthology.orchestra.run.smithi154.stdout:Modify: 2024-09-16 03:36:00.698316900 +0000 2024-09-16T03:41:35.244 INFO:teuthology.orchestra.run.smithi154.stdout:Change: 2024-09-16 03:36:00.698316900 +0000 2024-09-16T03:41:35.244 INFO:teuthology.orchestra.run.smithi154.stdout: Birth: - 2024-09-16T03:41:35.244 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-16T03:41:35.298 INFO:teuthology.orchestra.run.smithi154.stderr:1+0 records in 2024-09-16T03:41:35.298 INFO:teuthology.orchestra.run.smithi154.stderr:1+0 records out 2024-09-16T03:41:35.298 INFO:teuthology.orchestra.run.smithi154.stderr:512 bytes copied, 0.000430276 s, 1.2 MB/s 2024-09-16T03:41:35.299 DEBUG:teuthology.orchestra.run.smithi154:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-16T03:41:35.349 DEBUG:teuthology.orchestra.run.smithi154:> stat /dev/vg_nvme/lv_4 2024-09-16T03:41:35.395 INFO:teuthology.orchestra.run.smithi154.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-16T03:41:35.396 INFO:teuthology.orchestra.run.smithi154.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T03:41:35.396 INFO:teuthology.orchestra.run.smithi154.stdout:Device: 5h/5d Inode: 937 Links: 1 2024-09-16T03:41:35.396 INFO:teuthology.orchestra.run.smithi154.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T03:41:35.396 INFO:teuthology.orchestra.run.smithi154.stdout:Access: 2024-09-16 03:36:01.442310255 +0000 2024-09-16T03:41:35.396 INFO:teuthology.orchestra.run.smithi154.stdout:Modify: 2024-09-16 03:36:01.178312613 +0000 2024-09-16T03:41:35.396 INFO:teuthology.orchestra.run.smithi154.stdout:Change: 2024-09-16 03:36:01.178312613 +0000 2024-09-16T03:41:35.396 INFO:teuthology.orchestra.run.smithi154.stdout: Birth: - 2024-09-16T03:41:35.396 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-16T03:41:35.450 INFO:teuthology.orchestra.run.smithi154.stderr:1+0 records in 2024-09-16T03:41:35.450 INFO:teuthology.orchestra.run.smithi154.stderr:1+0 records out 2024-09-16T03:41:35.450 INFO:teuthology.orchestra.run.smithi154.stderr:512 bytes copied, 0.000384281 s, 1.3 MB/s 2024-09-16T03:41:35.451 DEBUG:teuthology.orchestra.run.smithi154:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-16T03:41:35.501 INFO:tasks.ceph:osd dev map: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-09-16T03:41:35.501 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi042.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi154.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'}} 2024-09-16T03:41:35.501 INFO:tasks.ceph:Generating config... 2024-09-16T03:41:35.502 INFO:tasks.ceph:[client] debug monc = 20 2024-09-16T03:41:35.502 INFO:tasks.ceph:[client] debug ms = 1 2024-09-16T03:41:35.502 INFO:tasks.ceph:[global] debug ms = 1 2024-09-16T03:41:35.502 INFO:tasks.ceph:[global] debug objecter = 20 2024-09-16T03:41:35.502 INFO:tasks.ceph:[global] debug rados = 20 2024-09-16T03:41:35.503 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-09-16T03:41:35.503 INFO:tasks.ceph:[global] mon election default strategy = 1 2024-09-16T03:41:35.503 INFO:tasks.ceph:[global] ms bind msgr1 = False 2024-09-16T03:41:35.503 INFO:tasks.ceph:[global] ms bind msgr2 = True 2024-09-16T03:41:35.503 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-09-16T03:41:35.503 INFO:tasks.ceph:[global] ms type = async 2024-09-16T03:41:35.503 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-09-16T03:41:35.503 INFO:tasks.ceph:[mgr] debug ms = 1 2024-09-16T03:41:35.503 INFO:tasks.ceph:[mon] debug mon = 20 2024-09-16T03:41:35.503 INFO:tasks.ceph:[mon] debug ms = 1 2024-09-16T03:41:35.503 INFO:tasks.ceph:[mon] debug paxos = 20 2024-09-16T03:41:35.503 INFO:tasks.ceph:[mon] mon min osdmap epochs = 25 2024-09-16T03:41:35.503 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-09-16T03:41:35.503 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-09-16T03:41:35.503 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-09-16T03:41:35.504 INFO:tasks.ceph:[mon] mon scrub inject crc mismatch = 0.01 2024-09-16T03:41:35.504 INFO:tasks.ceph:[mon] mon scrub inject missing keys = 0.05 2024-09-16T03:41:35.504 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-09-16T03:41:35.504 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2024-09-16T03:41:35.504 INFO:tasks.ceph:[mon] paxos service trim min = 5 2024-09-16T03:41:35.504 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-09-16T03:41:35.504 INFO:tasks.ceph:[osd] bluestore compression algorithm = lz4 2024-09-16T03:41:35.504 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-09-16T03:41:35.504 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-09-16T03:41:35.504 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-09-16T03:41:35.504 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-09-16T03:41:35.504 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-09-16T03:41:35.505 INFO:tasks.ceph:[osd] debug ms = 1 2024-09-16T03:41:35.505 INFO:tasks.ceph:[osd] debug osd = 20 2024-09-16T03:41:35.505 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-09-16T03:41:35.505 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-09-16T03:41:35.505 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-09-16T03:41:35.505 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-09-16T03:41:35.505 INFO:tasks.ceph:[osd] osd class default list = * 2024-09-16T03:41:35.505 INFO:tasks.ceph:[osd] osd class load list = * 2024-09-16T03:41:35.505 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-09-16T03:41:35.505 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-09-16T03:41:35.505 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-09-16T03:41:35.506 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-09-16T03:41:35.506 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-09-16T03:41:35.506 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-09-16T03:41:35.506 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-09-16T03:41:35.506 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-09-16T03:41:35.506 INFO:tasks.ceph:Setting up mon.a... 2024-09-16T03:41:35.506 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-09-16T03:41:35.540 INFO:teuthology.orchestra.run.smithi042.stdout:creating /etc/ceph/ceph.keyring 2024-09-16T03:41:35.555 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-09-16T03:41:35.635 DEBUG:teuthology.orchestra.run.smithi042:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-16T03:41:35.690 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.42'), ('mon.c', '[v2:172.21.15.42:3301,v1:172.21.15.42:6790]'), ('mon.b', '172.21.15.154')] 2024-09-16T03:41:35.691 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.42,[v2:172.21.15.42:3301,v1:172.21.15.42:6790],172.21.15.154', 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'lz4', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 5}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-09-16T03:41:35.691 DEBUG:teuthology.orchestra.run.smithi042:> set -ex 2024-09-16T03:41:35.692 DEBUG:teuthology.orchestra.run.smithi042:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-09-16T03:41:35.739 DEBUG:teuthology.orchestra.run.smithi042:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.42 --addv c '[v2:172.21.15.42:3301,v1:172.21.15.42:6790]' --add b 172.21.15.154 --print /home/ubuntu/cephtest/ceph.monmap 2024-09-16T03:41:35.826 INFO:teuthology.orchestra.run.smithi042.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-09-16T03:41:35.826 INFO:teuthology.orchestra.run.smithi042.stdout:monmaptool: generated fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:41:35.826 INFO:teuthology.orchestra.run.smithi042.stdout:setting min_mon_release = quincy 2024-09-16T03:41:35.826 INFO:teuthology.orchestra.run.smithi042.stdout:epoch 0 2024-09-16T03:41:35.826 INFO:teuthology.orchestra.run.smithi042.stdout:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:41:35.826 INFO:teuthology.orchestra.run.smithi042.stdout:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:41:35.826 INFO:teuthology.orchestra.run.smithi042.stdout:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:41:35.827 INFO:teuthology.orchestra.run.smithi042.stdout:min_mon_release 17 (quincy) 2024-09-16T03:41:35.827 INFO:teuthology.orchestra.run.smithi042.stdout:election_strategy: 1 2024-09-16T03:41:35.827 INFO:teuthology.orchestra.run.smithi042.stdout:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:41:35.827 INFO:teuthology.orchestra.run.smithi042.stdout:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:41:35.827 INFO:teuthology.orchestra.run.smithi042.stdout:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:41:35.827 INFO:teuthology.orchestra.run.smithi042.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-09-16T03:41:35.828 DEBUG:teuthology.orchestra.run.smithi042:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-09-16T03:41:35.832 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 15853af6-d0b2-472c-b2ac-8d7ad428641f... 2024-09-16T03:41:35.833 DEBUG:teuthology.orchestra.run.smithi042:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-16T03:41:35.875 DEBUG:teuthology.orchestra.run.smithi154:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-16T03:41:35.903 INFO:teuthology.orchestra.run.smithi042.stdout:[global] 2024-09-16T03:41:35.903 INFO:teuthology.orchestra.run.smithi042.stdout: chdir = "" 2024-09-16T03:41:35.904 INFO:teuthology.orchestra.run.smithi042.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-16T03:41:35.904 INFO:teuthology.orchestra.run.smithi042.stdout: auth supported = cephx 2024-09-16T03:41:35.904 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-09-16T03:41:35.904 INFO:teuthology.orchestra.run.smithi042.stdout: filestore xattr use omap = true 2024-09-16T03:41:35.904 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-09-16T03:41:35.904 INFO:teuthology.orchestra.run.smithi042.stdout: mon clock drift allowed = 1.000 2024-09-16T03:41:35.904 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-09-16T03:41:35.904 INFO:teuthology.orchestra.run.smithi042.stdout: osd crush chooseleaf type = 0 2024-09-16T03:41:35.904 INFO:teuthology.orchestra.run.smithi042.stdout: auth debug = true 2024-09-16T03:41:35.904 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-09-16T03:41:35.904 INFO:teuthology.orchestra.run.smithi042.stdout: ms die on old message = true 2024-09-16T03:41:35.904 INFO:teuthology.orchestra.run.smithi042.stdout: ms die on bug = true 2024-09-16T03:41:35.904 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-09-16T03:41:35.905 INFO:teuthology.orchestra.run.smithi042.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-16T03:41:35.905 INFO:teuthology.orchestra.run.smithi042.stdout: mon pg warn max object skew = 0 2024-09-16T03:41:35.905 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-09-16T03:41:35.905 INFO:teuthology.orchestra.run.smithi042.stdout: # disable pg_autoscaler by default for new pools 2024-09-16T03:41:35.905 INFO:teuthology.orchestra.run.smithi042.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-16T03:41:35.905 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-09-16T03:41:35.905 INFO:teuthology.orchestra.run.smithi042.stdout: osd pool default size = 2 2024-09-16T03:41:35.905 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-09-16T03:41:35.905 INFO:teuthology.orchestra.run.smithi042.stdout: mon osd allow primary affinity = true 2024-09-16T03:41:35.905 INFO:teuthology.orchestra.run.smithi042.stdout: mon osd allow pg remap = true 2024-09-16T03:41:35.905 INFO:teuthology.orchestra.run.smithi042.stdout: mon warn on legacy crush tunables = false 2024-09-16T03:41:35.905 INFO:teuthology.orchestra.run.smithi042.stdout: mon warn on crush straw calc version zero = false 2024-09-16T03:41:35.906 INFO:teuthology.orchestra.run.smithi042.stdout: mon warn on no sortbitwise = false 2024-09-16T03:41:35.906 INFO:teuthology.orchestra.run.smithi042.stdout: mon warn on osd down out interval zero = false 2024-09-16T03:41:35.906 INFO:teuthology.orchestra.run.smithi042.stdout: mon warn on too few osds = false 2024-09-16T03:41:35.906 INFO:teuthology.orchestra.run.smithi042.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-16T03:41:35.906 INFO:teuthology.orchestra.run.smithi042.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-16T03:41:35.906 INFO:teuthology.orchestra.run.smithi042.stdout: mon_allow_pool_size_one = true 2024-09-16T03:41:35.906 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-09-16T03:41:35.906 INFO:teuthology.orchestra.run.smithi042.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-16T03:41:35.906 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-09-16T03:41:35.906 INFO:teuthology.orchestra.run.smithi042.stdout: osd default data pool replay window = 5 2024-09-16T03:41:35.906 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-09-16T03:41:35.906 INFO:teuthology.orchestra.run.smithi042.stdout: mon allow pool delete = true 2024-09-16T03:41:35.906 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-09-16T03:41:35.907 INFO:teuthology.orchestra.run.smithi042.stdout: mon cluster log file level = debug 2024-09-16T03:41:35.907 INFO:teuthology.orchestra.run.smithi042.stdout: debug asserts on shutdown = true 2024-09-16T03:41:35.907 INFO:teuthology.orchestra.run.smithi042.stdout: mon health detail to clog = false 2024-09-16T03:41:35.907 INFO:teuthology.orchestra.run.smithi042.stdout: mon host = "172.21.15.42,[v2:172.21.15.42:3301,v1:172.21.15.42:6790],172.21.15.154" 2024-09-16T03:41:35.907 INFO:teuthology.orchestra.run.smithi042.stdout: debug ms = 1 2024-09-16T03:41:35.907 INFO:teuthology.orchestra.run.smithi042.stdout: debug objecter = 20 2024-09-16T03:41:35.907 INFO:teuthology.orchestra.run.smithi042.stdout: debug rados = 20 2024-09-16T03:41:35.907 INFO:teuthology.orchestra.run.smithi042.stdout: mon client directed command retry = 5 2024-09-16T03:41:35.907 INFO:teuthology.orchestra.run.smithi042.stdout: mon election default strategy = 1 2024-09-16T03:41:35.907 INFO:teuthology.orchestra.run.smithi042.stdout: ms bind msgr1 = False 2024-09-16T03:41:35.907 INFO:teuthology.orchestra.run.smithi042.stdout: ms bind msgr2 = True 2024-09-16T03:41:35.907 INFO:teuthology.orchestra.run.smithi042.stdout: ms inject socket failures = 5000 2024-09-16T03:41:35.908 INFO:teuthology.orchestra.run.smithi042.stdout: ms type = async 2024-09-16T03:41:35.908 INFO:teuthology.orchestra.run.smithi042.stdout: fsid = 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:41:35.908 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-09-16T03:41:35.908 INFO:teuthology.orchestra.run.smithi042.stdout:[osd] 2024-09-16T03:41:35.908 INFO:teuthology.orchestra.run.smithi042.stdout: osd journal size = 100 2024-09-16T03:41:35.908 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-09-16T03:41:35.908 INFO:teuthology.orchestra.run.smithi042.stdout: osd scrub load threshold = 5.0 2024-09-16T03:41:35.908 INFO:teuthology.orchestra.run.smithi042.stdout: osd scrub max interval = 600 2024-09-16T03:41:35.908 INFO:teuthology.orchestra.run.smithi042.stdout: osd mclock profile = high_recovery_ops 2024-09-16T03:41:35.908 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-09-16T03:41:35.908 INFO:teuthology.orchestra.run.smithi042.stdout: osd recover clone overlap = true 2024-09-16T03:41:35.908 INFO:teuthology.orchestra.run.smithi042.stdout: osd recovery max chunk = 1048576 2024-09-16T03:41:35.908 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-09-16T03:41:35.909 INFO:teuthology.orchestra.run.smithi042.stdout: osd debug shutdown = true 2024-09-16T03:41:35.909 INFO:teuthology.orchestra.run.smithi042.stdout: osd debug op order = true 2024-09-16T03:41:35.909 INFO:teuthology.orchestra.run.smithi042.stdout: osd debug verify stray on activate = true 2024-09-16T03:41:35.909 INFO:teuthology.orchestra.run.smithi042.stdout: osd debug trim objects = true 2024-09-16T03:41:35.909 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-09-16T03:41:35.909 INFO:teuthology.orchestra.run.smithi042.stdout: osd open classes on start = true 2024-09-16T03:41:35.909 INFO:teuthology.orchestra.run.smithi042.stdout: osd debug pg log writeout = true 2024-09-16T03:41:35.909 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-09-16T03:41:35.909 INFO:teuthology.orchestra.run.smithi042.stdout: osd deep scrub update digest min age = 30 2024-09-16T03:41:35.909 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-09-16T03:41:35.909 INFO:teuthology.orchestra.run.smithi042.stdout: osd map max advance = 10 2024-09-16T03:41:35.909 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-09-16T03:41:35.909 INFO:teuthology.orchestra.run.smithi042.stdout: journal zero on create = true 2024-09-16T03:41:35.909 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-09-16T03:41:35.910 INFO:teuthology.orchestra.run.smithi042.stdout: filestore ondisk finisher threads = 3 2024-09-16T03:41:35.910 INFO:teuthology.orchestra.run.smithi042.stdout: filestore apply finisher threads = 3 2024-09-16T03:41:35.910 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-09-16T03:41:35.910 INFO:teuthology.orchestra.run.smithi042.stdout: bdev debug aio = true 2024-09-16T03:41:35.910 INFO:teuthology.orchestra.run.smithi042.stdout: osd debug misdirected ops = true 2024-09-16T03:41:35.910 INFO:teuthology.orchestra.run.smithi042.stdout: bluestore block size = 96636764160 2024-09-16T03:41:35.910 INFO:teuthology.orchestra.run.smithi042.stdout: bluestore compression algorithm = lz4 2024-09-16T03:41:35.910 INFO:teuthology.orchestra.run.smithi042.stdout: bluestore compression mode = aggressive 2024-09-16T03:41:35.910 INFO:teuthology.orchestra.run.smithi042.stdout: bluestore fsck on mount = True 2024-09-16T03:41:35.910 INFO:teuthology.orchestra.run.smithi042.stdout: bluestore zero block detection = True 2024-09-16T03:41:35.910 INFO:teuthology.orchestra.run.smithi042.stdout: debug bluefs = 20 2024-09-16T03:41:35.910 INFO:teuthology.orchestra.run.smithi042.stdout: debug bluestore = 20 2024-09-16T03:41:35.911 INFO:teuthology.orchestra.run.smithi042.stdout: debug ms = 1 2024-09-16T03:41:35.911 INFO:teuthology.orchestra.run.smithi042.stdout: debug osd = 20 2024-09-16T03:41:35.911 INFO:teuthology.orchestra.run.smithi042.stdout: debug rocksdb = 10 2024-09-16T03:41:35.911 INFO:teuthology.orchestra.run.smithi042.stdout: mon osd backfillfull_ratio = 0.85 2024-09-16T03:41:35.911 INFO:teuthology.orchestra.run.smithi042.stdout: mon osd full ratio = 0.9 2024-09-16T03:41:35.911 INFO:teuthology.orchestra.run.smithi042.stdout: mon osd nearfull ratio = 0.8 2024-09-16T03:41:35.911 INFO:teuthology.orchestra.run.smithi042.stdout: osd class default list = * 2024-09-16T03:41:35.911 INFO:teuthology.orchestra.run.smithi042.stdout: osd class load list = * 2024-09-16T03:41:35.911 INFO:teuthology.orchestra.run.smithi042.stdout: osd debug verify cached snaps = True 2024-09-16T03:41:35.911 INFO:teuthology.orchestra.run.smithi042.stdout: osd debug verify missing on start = True 2024-09-16T03:41:35.911 INFO:teuthology.orchestra.run.smithi042.stdout: osd failsafe full ratio = 0.95 2024-09-16T03:41:35.911 INFO:teuthology.orchestra.run.smithi042.stdout: osd mclock override recovery settings = True 2024-09-16T03:41:35.911 INFO:teuthology.orchestra.run.smithi042.stdout: osd objectstore = bluestore 2024-09-16T03:41:35.912 INFO:teuthology.orchestra.run.smithi042.stdout: osd op queue = debug_random 2024-09-16T03:41:35.912 INFO:teuthology.orchestra.run.smithi042.stdout: osd op queue cut off = debug_random 2024-09-16T03:41:35.912 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-09-16T03:41:35.912 INFO:teuthology.orchestra.run.smithi042.stdout:[mgr] 2024-09-16T03:41:35.912 INFO:teuthology.orchestra.run.smithi042.stdout: debug ms = 1 2024-09-16T03:41:35.912 INFO:teuthology.orchestra.run.smithi042.stdout: debug mgr = 20 2024-09-16T03:41:35.912 INFO:teuthology.orchestra.run.smithi042.stdout: debug mon = 20 2024-09-16T03:41:35.912 INFO:teuthology.orchestra.run.smithi042.stdout: debug auth = 20 2024-09-16T03:41:35.912 INFO:teuthology.orchestra.run.smithi042.stdout: mon reweight min pgs per osd = 4 2024-09-16T03:41:35.912 INFO:teuthology.orchestra.run.smithi042.stdout: mon reweight min bytes per osd = 10 2024-09-16T03:41:35.912 INFO:teuthology.orchestra.run.smithi042.stdout: mgr/telemetry/nag = false 2024-09-16T03:41:35.912 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-09-16T03:41:35.912 INFO:teuthology.orchestra.run.smithi042.stdout:[mon] 2024-09-16T03:41:35.912 INFO:teuthology.orchestra.run.smithi042.stdout: debug ms = 1 2024-09-16T03:41:35.913 INFO:teuthology.orchestra.run.smithi042.stdout: debug mon = 20 2024-09-16T03:41:35.913 INFO:teuthology.orchestra.run.smithi042.stdout: debug paxos = 20 2024-09-16T03:41:35.913 INFO:teuthology.orchestra.run.smithi042.stdout: debug auth = 20 2024-09-16T03:41:35.913 INFO:teuthology.orchestra.run.smithi042.stdout: mon data avail warn = 5 2024-09-16T03:41:35.913 INFO:teuthology.orchestra.run.smithi042.stdout: mon mgr mkfs grace = 240 2024-09-16T03:41:35.913 INFO:teuthology.orchestra.run.smithi042.stdout: mon reweight min pgs per osd = 4 2024-09-16T03:41:35.913 INFO:teuthology.orchestra.run.smithi042.stdout: mon osd reporter subtree level = osd 2024-09-16T03:41:35.913 INFO:teuthology.orchestra.run.smithi042.stdout: mon osd prime pg temp = true 2024-09-16T03:41:35.913 INFO:teuthology.orchestra.run.smithi042.stdout: mon reweight min bytes per osd = 10 2024-09-16T03:41:35.913 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-09-16T03:41:35.913 INFO:teuthology.orchestra.run.smithi042.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-16T03:41:35.913 INFO:teuthology.orchestra.run.smithi042.stdout: auth mon ticket ttl = 660 # 11m 2024-09-16T03:41:35.913 INFO:teuthology.orchestra.run.smithi042.stdout: auth service ticket ttl = 240 # 4m 2024-09-16T03:41:35.913 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-09-16T03:41:35.914 INFO:teuthology.orchestra.run.smithi042.stdout: # don't complain about insecure global_id in the test suite 2024-09-16T03:41:35.914 INFO:teuthology.orchestra.run.smithi042.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-16T03:41:35.914 INFO:teuthology.orchestra.run.smithi042.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-16T03:41:35.914 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-09-16T03:41:35.914 INFO:teuthology.orchestra.run.smithi042.stdout: # 1m isn't quite enough 2024-09-16T03:41:35.914 INFO:teuthology.orchestra.run.smithi042.stdout: mon_down_mkfs_grace = 2m 2024-09-16T03:41:35.914 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-09-16T03:41:35.914 INFO:teuthology.orchestra.run.smithi042.stdout: mon_warn_on_filestore_osds = false 2024-09-16T03:41:35.914 INFO:teuthology.orchestra.run.smithi042.stdout: mon min osdmap epochs = 25 2024-09-16T03:41:35.914 INFO:teuthology.orchestra.run.smithi042.stdout: mon osdmap full prune interval = 2 2024-09-16T03:41:35.914 INFO:teuthology.orchestra.run.smithi042.stdout: mon osdmap full prune min = 15 2024-09-16T03:41:35.914 INFO:teuthology.orchestra.run.smithi042.stdout: mon osdmap full prune txsize = 2 2024-09-16T03:41:35.914 INFO:teuthology.orchestra.run.smithi042.stdout: mon scrub inject crc mismatch = 0.01 2024-09-16T03:41:35.915 INFO:teuthology.orchestra.run.smithi042.stdout: mon scrub inject missing keys = 0.05 2024-09-16T03:41:35.915 INFO:teuthology.orchestra.run.smithi042.stdout: mon scrub interval = 300 2024-09-16T03:41:35.915 INFO:teuthology.orchestra.run.smithi042.stdout: mon warn on pool no app = False 2024-09-16T03:41:35.915 INFO:teuthology.orchestra.run.smithi042.stdout: paxos service trim min = 5 2024-09-16T03:41:35.915 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-09-16T03:41:35.915 INFO:teuthology.orchestra.run.smithi042.stdout:[client] 2024-09-16T03:41:35.915 INFO:teuthology.orchestra.run.smithi042.stdout: rgw cache enabled = true 2024-09-16T03:41:35.915 INFO:teuthology.orchestra.run.smithi042.stdout: rgw enable ops log = true 2024-09-16T03:41:35.915 INFO:teuthology.orchestra.run.smithi042.stdout: rgw enable usage log = true 2024-09-16T03:41:35.915 INFO:teuthology.orchestra.run.smithi042.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-16T03:41:35.915 INFO:teuthology.orchestra.run.smithi042.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-16T03:41:35.915 INFO:teuthology.orchestra.run.smithi042.stdout: debug monc = 20 2024-09-16T03:41:35.915 INFO:teuthology.orchestra.run.smithi042.stdout: debug ms = 1 2024-09-16T03:41:35.915 INFO:teuthology.orchestra.run.smithi042.stdout:[mon.a] 2024-09-16T03:41:35.916 INFO:teuthology.orchestra.run.smithi042.stdout:[mon.c] 2024-09-16T03:41:35.916 INFO:teuthology.orchestra.run.smithi042.stdout:[mon.b] 2024-09-16T03:41:35.917 INFO:teuthology.orchestra.run.smithi154.stdout:[global] 2024-09-16T03:41:35.917 INFO:teuthology.orchestra.run.smithi154.stdout: chdir = "" 2024-09-16T03:41:35.917 INFO:teuthology.orchestra.run.smithi154.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-16T03:41:35.917 INFO:teuthology.orchestra.run.smithi154.stdout: auth supported = cephx 2024-09-16T03:41:35.917 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-16T03:41:35.918 INFO:teuthology.orchestra.run.smithi154.stdout: filestore xattr use omap = true 2024-09-16T03:41:35.918 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-16T03:41:35.918 INFO:teuthology.orchestra.run.smithi154.stdout: mon clock drift allowed = 1.000 2024-09-16T03:41:35.918 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-16T03:41:35.918 INFO:teuthology.orchestra.run.smithi154.stdout: osd crush chooseleaf type = 0 2024-09-16T03:41:35.918 INFO:teuthology.orchestra.run.smithi154.stdout: auth debug = true 2024-09-16T03:41:35.918 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-16T03:41:35.918 INFO:teuthology.orchestra.run.smithi154.stdout: ms die on old message = true 2024-09-16T03:41:35.918 INFO:teuthology.orchestra.run.smithi154.stdout: ms die on bug = true 2024-09-16T03:41:35.918 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-16T03:41:35.918 INFO:teuthology.orchestra.run.smithi154.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-16T03:41:35.918 INFO:teuthology.orchestra.run.smithi154.stdout: mon pg warn max object skew = 0 2024-09-16T03:41:35.918 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-16T03:41:35.918 INFO:teuthology.orchestra.run.smithi154.stdout: # disable pg_autoscaler by default for new pools 2024-09-16T03:41:35.919 INFO:teuthology.orchestra.run.smithi154.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-16T03:41:35.919 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-16T03:41:35.919 INFO:teuthology.orchestra.run.smithi154.stdout: osd pool default size = 2 2024-09-16T03:41:35.919 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-16T03:41:35.919 INFO:teuthology.orchestra.run.smithi154.stdout: mon osd allow primary affinity = true 2024-09-16T03:41:35.919 INFO:teuthology.orchestra.run.smithi154.stdout: mon osd allow pg remap = true 2024-09-16T03:41:35.919 INFO:teuthology.orchestra.run.smithi154.stdout: mon warn on legacy crush tunables = false 2024-09-16T03:41:35.919 INFO:teuthology.orchestra.run.smithi154.stdout: mon warn on crush straw calc version zero = false 2024-09-16T03:41:35.919 INFO:teuthology.orchestra.run.smithi154.stdout: mon warn on no sortbitwise = false 2024-09-16T03:41:35.919 INFO:teuthology.orchestra.run.smithi154.stdout: mon warn on osd down out interval zero = false 2024-09-16T03:41:35.919 INFO:teuthology.orchestra.run.smithi154.stdout: mon warn on too few osds = false 2024-09-16T03:41:35.919 INFO:teuthology.orchestra.run.smithi154.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-16T03:41:35.919 INFO:teuthology.orchestra.run.smithi154.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-16T03:41:35.919 INFO:teuthology.orchestra.run.smithi154.stdout: mon_allow_pool_size_one = true 2024-09-16T03:41:35.920 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-16T03:41:35.920 INFO:teuthology.orchestra.run.smithi154.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-16T03:41:35.920 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-16T03:41:35.920 INFO:teuthology.orchestra.run.smithi154.stdout: osd default data pool replay window = 5 2024-09-16T03:41:35.920 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-16T03:41:35.920 INFO:teuthology.orchestra.run.smithi154.stdout: mon allow pool delete = true 2024-09-16T03:41:35.920 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-16T03:41:35.920 INFO:teuthology.orchestra.run.smithi154.stdout: mon cluster log file level = debug 2024-09-16T03:41:35.920 INFO:teuthology.orchestra.run.smithi154.stdout: debug asserts on shutdown = true 2024-09-16T03:41:35.920 INFO:teuthology.orchestra.run.smithi154.stdout: mon health detail to clog = false 2024-09-16T03:41:35.920 INFO:teuthology.orchestra.run.smithi154.stdout: mon host = "172.21.15.42,[v2:172.21.15.42:3301,v1:172.21.15.42:6790],172.21.15.154" 2024-09-16T03:41:35.920 INFO:teuthology.orchestra.run.smithi154.stdout: debug ms = 1 2024-09-16T03:41:35.920 INFO:teuthology.orchestra.run.smithi154.stdout: debug objecter = 20 2024-09-16T03:41:35.920 INFO:teuthology.orchestra.run.smithi154.stdout: debug rados = 20 2024-09-16T03:41:35.921 INFO:teuthology.orchestra.run.smithi154.stdout: mon client directed command retry = 5 2024-09-16T03:41:35.921 INFO:teuthology.orchestra.run.smithi154.stdout: mon election default strategy = 1 2024-09-16T03:41:35.921 INFO:teuthology.orchestra.run.smithi154.stdout: ms bind msgr1 = False 2024-09-16T03:41:35.921 INFO:teuthology.orchestra.run.smithi154.stdout: ms bind msgr2 = True 2024-09-16T03:41:35.921 INFO:teuthology.orchestra.run.smithi154.stdout: ms inject socket failures = 5000 2024-09-16T03:41:35.921 INFO:teuthology.orchestra.run.smithi154.stdout: ms type = async 2024-09-16T03:41:35.921 INFO:teuthology.orchestra.run.smithi154.stdout: fsid = 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:41:35.921 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-16T03:41:35.921 INFO:teuthology.orchestra.run.smithi154.stdout:[osd] 2024-09-16T03:41:35.921 INFO:teuthology.orchestra.run.smithi154.stdout: osd journal size = 100 2024-09-16T03:41:35.921 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-16T03:41:35.921 INFO:teuthology.orchestra.run.smithi154.stdout: osd scrub load threshold = 5.0 2024-09-16T03:41:35.921 INFO:teuthology.orchestra.run.smithi154.stdout: osd scrub max interval = 600 2024-09-16T03:41:35.922 INFO:teuthology.orchestra.run.smithi154.stdout: osd mclock profile = high_recovery_ops 2024-09-16T03:41:35.922 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-16T03:41:35.922 INFO:teuthology.orchestra.run.smithi154.stdout: osd recover clone overlap = true 2024-09-16T03:41:35.922 INFO:teuthology.orchestra.run.smithi154.stdout: osd recovery max chunk = 1048576 2024-09-16T03:41:35.922 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-16T03:41:35.922 INFO:teuthology.orchestra.run.smithi154.stdout: osd debug shutdown = true 2024-09-16T03:41:35.922 INFO:teuthology.orchestra.run.smithi154.stdout: osd debug op order = true 2024-09-16T03:41:35.922 INFO:teuthology.orchestra.run.smithi154.stdout: osd debug verify stray on activate = true 2024-09-16T03:41:35.922 INFO:teuthology.orchestra.run.smithi154.stdout: osd debug trim objects = true 2024-09-16T03:41:35.922 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-16T03:41:35.922 INFO:teuthology.orchestra.run.smithi154.stdout: osd open classes on start = true 2024-09-16T03:41:35.922 INFO:teuthology.orchestra.run.smithi154.stdout: osd debug pg log writeout = true 2024-09-16T03:41:35.922 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-16T03:41:35.922 INFO:teuthology.orchestra.run.smithi154.stdout: osd deep scrub update digest min age = 30 2024-09-16T03:41:35.922 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-16T03:41:35.923 INFO:teuthology.orchestra.run.smithi154.stdout: osd map max advance = 10 2024-09-16T03:41:35.923 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-16T03:41:35.923 INFO:teuthology.orchestra.run.smithi154.stdout: journal zero on create = true 2024-09-16T03:41:35.923 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-16T03:41:35.923 INFO:teuthology.orchestra.run.smithi154.stdout: filestore ondisk finisher threads = 3 2024-09-16T03:41:35.923 INFO:teuthology.orchestra.run.smithi154.stdout: filestore apply finisher threads = 3 2024-09-16T03:41:35.923 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-16T03:41:35.923 INFO:teuthology.orchestra.run.smithi154.stdout: bdev debug aio = true 2024-09-16T03:41:35.923 INFO:teuthology.orchestra.run.smithi154.stdout: osd debug misdirected ops = true 2024-09-16T03:41:35.923 INFO:teuthology.orchestra.run.smithi154.stdout: bluestore block size = 96636764160 2024-09-16T03:41:35.923 INFO:teuthology.orchestra.run.smithi154.stdout: bluestore compression algorithm = lz4 2024-09-16T03:41:35.923 INFO:teuthology.orchestra.run.smithi154.stdout: bluestore compression mode = aggressive 2024-09-16T03:41:35.923 INFO:teuthology.orchestra.run.smithi154.stdout: bluestore fsck on mount = True 2024-09-16T03:41:35.923 INFO:teuthology.orchestra.run.smithi154.stdout: bluestore zero block detection = True 2024-09-16T03:41:35.924 INFO:teuthology.orchestra.run.smithi154.stdout: debug bluefs = 20 2024-09-16T03:41:35.924 INFO:teuthology.orchestra.run.smithi154.stdout: debug bluestore = 20 2024-09-16T03:41:35.924 INFO:teuthology.orchestra.run.smithi154.stdout: debug ms = 1 2024-09-16T03:41:35.924 INFO:teuthology.orchestra.run.smithi154.stdout: debug osd = 20 2024-09-16T03:41:35.924 INFO:teuthology.orchestra.run.smithi154.stdout: debug rocksdb = 10 2024-09-16T03:41:35.924 INFO:teuthology.orchestra.run.smithi154.stdout: mon osd backfillfull_ratio = 0.85 2024-09-16T03:41:35.924 INFO:teuthology.orchestra.run.smithi154.stdout: mon osd full ratio = 0.9 2024-09-16T03:41:35.924 INFO:teuthology.orchestra.run.smithi154.stdout: mon osd nearfull ratio = 0.8 2024-09-16T03:41:35.924 INFO:teuthology.orchestra.run.smithi154.stdout: osd class default list = * 2024-09-16T03:41:35.924 INFO:teuthology.orchestra.run.smithi154.stdout: osd class load list = * 2024-09-16T03:41:35.924 INFO:teuthology.orchestra.run.smithi154.stdout: osd debug verify cached snaps = True 2024-09-16T03:41:35.924 INFO:teuthology.orchestra.run.smithi154.stdout: osd debug verify missing on start = True 2024-09-16T03:41:35.924 INFO:teuthology.orchestra.run.smithi154.stdout: osd failsafe full ratio = 0.95 2024-09-16T03:41:35.925 INFO:teuthology.orchestra.run.smithi154.stdout: osd mclock override recovery settings = True 2024-09-16T03:41:35.925 INFO:teuthology.orchestra.run.smithi154.stdout: osd objectstore = bluestore 2024-09-16T03:41:35.925 INFO:teuthology.orchestra.run.smithi154.stdout: osd op queue = debug_random 2024-09-16T03:41:35.925 INFO:teuthology.orchestra.run.smithi154.stdout: osd op queue cut off = debug_random 2024-09-16T03:41:35.925 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-16T03:41:35.925 INFO:teuthology.orchestra.run.smithi154.stdout:[mgr] 2024-09-16T03:41:35.925 INFO:teuthology.orchestra.run.smithi154.stdout: debug ms = 1 2024-09-16T03:41:35.925 INFO:teuthology.orchestra.run.smithi154.stdout: debug mgr = 20 2024-09-16T03:41:35.925 INFO:teuthology.orchestra.run.smithi154.stdout: debug mon = 20 2024-09-16T03:41:35.925 INFO:teuthology.orchestra.run.smithi154.stdout: debug auth = 20 2024-09-16T03:41:35.925 INFO:teuthology.orchestra.run.smithi154.stdout: mon reweight min pgs per osd = 4 2024-09-16T03:41:35.925 INFO:teuthology.orchestra.run.smithi154.stdout: mon reweight min bytes per osd = 10 2024-09-16T03:41:35.925 INFO:teuthology.orchestra.run.smithi154.stdout: mgr/telemetry/nag = false 2024-09-16T03:41:35.925 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-16T03:41:35.926 INFO:teuthology.orchestra.run.smithi154.stdout:[mon] 2024-09-16T03:41:35.926 INFO:teuthology.orchestra.run.smithi154.stdout: debug ms = 1 2024-09-16T03:41:35.926 INFO:teuthology.orchestra.run.smithi154.stdout: debug mon = 20 2024-09-16T03:41:35.926 INFO:teuthology.orchestra.run.smithi154.stdout: debug paxos = 20 2024-09-16T03:41:35.926 INFO:teuthology.orchestra.run.smithi154.stdout: debug auth = 20 2024-09-16T03:41:35.926 INFO:teuthology.orchestra.run.smithi154.stdout: mon data avail warn = 5 2024-09-16T03:41:35.926 INFO:teuthology.orchestra.run.smithi154.stdout: mon mgr mkfs grace = 240 2024-09-16T03:41:35.926 INFO:teuthology.orchestra.run.smithi154.stdout: mon reweight min pgs per osd = 4 2024-09-16T03:41:35.926 INFO:teuthology.orchestra.run.smithi154.stdout: mon osd reporter subtree level = osd 2024-09-16T03:41:35.926 INFO:teuthology.orchestra.run.smithi154.stdout: mon osd prime pg temp = true 2024-09-16T03:41:35.926 INFO:teuthology.orchestra.run.smithi154.stdout: mon reweight min bytes per osd = 10 2024-09-16T03:41:35.926 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-16T03:41:35.926 INFO:teuthology.orchestra.run.smithi154.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-16T03:41:35.926 INFO:teuthology.orchestra.run.smithi154.stdout: auth mon ticket ttl = 660 # 11m 2024-09-16T03:41:35.927 INFO:teuthology.orchestra.run.smithi154.stdout: auth service ticket ttl = 240 # 4m 2024-09-16T03:41:35.927 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-16T03:41:35.927 INFO:teuthology.orchestra.run.smithi154.stdout: # don't complain about insecure global_id in the test suite 2024-09-16T03:41:35.927 INFO:teuthology.orchestra.run.smithi154.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-16T03:41:35.927 INFO:teuthology.orchestra.run.smithi154.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-16T03:41:35.927 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-16T03:41:35.927 INFO:teuthology.orchestra.run.smithi154.stdout: # 1m isn't quite enough 2024-09-16T03:41:35.927 INFO:teuthology.orchestra.run.smithi154.stdout: mon_down_mkfs_grace = 2m 2024-09-16T03:41:35.927 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-16T03:41:35.927 INFO:teuthology.orchestra.run.smithi154.stdout: mon_warn_on_filestore_osds = false 2024-09-16T03:41:35.927 INFO:teuthology.orchestra.run.smithi154.stdout: mon min osdmap epochs = 25 2024-09-16T03:41:35.927 INFO:teuthology.orchestra.run.smithi154.stdout: mon osdmap full prune interval = 2 2024-09-16T03:41:35.927 INFO:teuthology.orchestra.run.smithi154.stdout: mon osdmap full prune min = 15 2024-09-16T03:41:35.927 INFO:teuthology.orchestra.run.smithi154.stdout: mon osdmap full prune txsize = 2 2024-09-16T03:41:35.927 INFO:teuthology.orchestra.run.smithi154.stdout: mon scrub inject crc mismatch = 0.01 2024-09-16T03:41:35.928 INFO:teuthology.orchestra.run.smithi154.stdout: mon scrub inject missing keys = 0.05 2024-09-16T03:41:35.928 INFO:teuthology.orchestra.run.smithi154.stdout: mon scrub interval = 300 2024-09-16T03:41:35.928 INFO:teuthology.orchestra.run.smithi154.stdout: mon warn on pool no app = False 2024-09-16T03:41:35.928 INFO:teuthology.orchestra.run.smithi154.stdout: paxos service trim min = 5 2024-09-16T03:41:35.928 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-16T03:41:35.928 INFO:teuthology.orchestra.run.smithi154.stdout:[client] 2024-09-16T03:41:35.928 INFO:teuthology.orchestra.run.smithi154.stdout: rgw cache enabled = true 2024-09-16T03:41:35.928 INFO:teuthology.orchestra.run.smithi154.stdout: rgw enable ops log = true 2024-09-16T03:41:35.928 INFO:teuthology.orchestra.run.smithi154.stdout: rgw enable usage log = true 2024-09-16T03:41:35.928 INFO:teuthology.orchestra.run.smithi154.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-16T03:41:35.928 INFO:teuthology.orchestra.run.smithi154.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-16T03:41:35.928 INFO:teuthology.orchestra.run.smithi154.stdout: debug monc = 20 2024-09-16T03:41:35.928 INFO:teuthology.orchestra.run.smithi154.stdout: debug ms = 1 2024-09-16T03:41:35.928 INFO:teuthology.orchestra.run.smithi154.stdout:[mon.a] 2024-09-16T03:41:35.929 INFO:teuthology.orchestra.run.smithi154.stdout:[mon.c] 2024-09-16T03:41:35.929 INFO:teuthology.orchestra.run.smithi154.stdout:[mon.b] 2024-09-16T03:41:35.929 INFO:tasks.ceph:Creating admin key on mon.a... 2024-09-16T03:41:35.929 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-09-16T03:41:36.001 INFO:tasks.ceph:Copying monmap to all nodes... 2024-09-16T03:41:36.002 DEBUG:teuthology.orchestra.run.smithi042:> set -ex 2024-09-16T03:41:36.002 DEBUG:teuthology.orchestra.run.smithi042:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-09-16T03:41:36.051 DEBUG:teuthology.orchestra.run.smithi042:> set -ex 2024-09-16T03:41:36.051 DEBUG:teuthology.orchestra.run.smithi042:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-09-16T03:41:36.098 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi042.front.sepia.ceph.com 2024-09-16T03:41:36.098 DEBUG:teuthology.orchestra.run.smithi042:> set -ex 2024-09-16T03:41:36.099 DEBUG:teuthology.orchestra.run.smithi042:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-16T03:41:36.099 DEBUG:teuthology.orchestra.run.smithi042:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-16T03:41:36.164 DEBUG:teuthology.orchestra.run.smithi042:> set -ex 2024-09-16T03:41:36.164 DEBUG:teuthology.orchestra.run.smithi042:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-16T03:41:36.211 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi154.front.sepia.ceph.com 2024-09-16T03:41:36.211 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-09-16T03:41:36.211 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-16T03:41:36.211 DEBUG:teuthology.orchestra.run.smithi154:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-16T03:41:36.233 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-09-16T03:41:36.233 DEBUG:teuthology.orchestra.run.smithi154:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-16T03:41:36.279 INFO:tasks.ceph:Setting up mon nodes... 2024-09-16T03:41:36.280 INFO:tasks.ceph:Setting up mgr nodes... 2024-09-16T03:41:36.280 DEBUG:teuthology.orchestra.run.smithi154:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-09-16T03:41:36.364 INFO:teuthology.orchestra.run.smithi154.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-09-16T03:41:36.377 INFO:tasks.ceph:Setting up mds nodes... 2024-09-16T03:41:36.377 INFO:tasks.ceph_client:Setting up client nodes... 2024-09-16T03:41:36.378 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-16T03:41:36.452 INFO:teuthology.orchestra.run.smithi154.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-09-16T03:41:36.463 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-09-16T03:41:36.463 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi042.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi154.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'}} 2024-09-16T03:41:36.463 DEBUG:teuthology.orchestra.run.smithi042:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-09-16T03:41:36.477 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-16T03:41:36.477 INFO:tasks.ceph:role: osd.0 2024-09-16T03:41:36.477 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi042.front.sepia.ceph.com 2024-09-16T03:41:36.477 DEBUG:teuthology.orchestra.run.smithi042:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-16T03:41:36.533 INFO:teuthology.orchestra.run.smithi042.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T03:41:36.534 INFO:teuthology.orchestra.run.smithi042.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T03:41:36.534 INFO:teuthology.orchestra.run.smithi042.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T03:41:36.534 INFO:teuthology.orchestra.run.smithi042.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-16T03:41:36.534 INFO:teuthology.orchestra.run.smithi042.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T03:41:36.534 INFO:teuthology.orchestra.run.smithi042.stdout: = sunit=0 swidth=0 blks 2024-09-16T03:41:36.534 INFO:teuthology.orchestra.run.smithi042.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T03:41:36.534 INFO:teuthology.orchestra.run.smithi042.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-16T03:41:36.534 INFO:teuthology.orchestra.run.smithi042.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T03:41:36.534 INFO:teuthology.orchestra.run.smithi042.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T03:41:36.557 INFO:teuthology.orchestra.run.smithi042.stdout:Discarding blocks...Done. 2024-09-16T03:41:36.559 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi042.front.sepia.ceph.com -o noatime 2024-09-16T03:41:36.559 DEBUG:teuthology.orchestra.run.smithi042:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-09-16T03:41:36.619 DEBUG:teuthology.orchestra.run.smithi042:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-09-16T03:41:36.672 INFO:teuthology.orchestra.run.smithi042.stderr:sudo: /sbin/restorecon: command not found 2024-09-16T03:41:36.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T03:41:36.674 DEBUG:teuthology.orchestra.run.smithi042:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-09-16T03:41:36.730 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-16T03:41:36.730 INFO:tasks.ceph:role: osd.1 2024-09-16T03:41:36.730 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi042.front.sepia.ceph.com 2024-09-16T03:41:36.731 DEBUG:teuthology.orchestra.run.smithi042:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-16T03:41:36.786 INFO:teuthology.orchestra.run.smithi042.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T03:41:36.787 INFO:teuthology.orchestra.run.smithi042.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T03:41:36.787 INFO:teuthology.orchestra.run.smithi042.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T03:41:36.787 INFO:teuthology.orchestra.run.smithi042.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-16T03:41:36.787 INFO:teuthology.orchestra.run.smithi042.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T03:41:36.787 INFO:teuthology.orchestra.run.smithi042.stdout: = sunit=0 swidth=0 blks 2024-09-16T03:41:36.787 INFO:teuthology.orchestra.run.smithi042.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T03:41:36.787 INFO:teuthology.orchestra.run.smithi042.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-16T03:41:36.787 INFO:teuthology.orchestra.run.smithi042.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T03:41:36.787 INFO:teuthology.orchestra.run.smithi042.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T03:41:36.810 INFO:teuthology.orchestra.run.smithi042.stdout:Discarding blocks...Done. 2024-09-16T03:41:36.812 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi042.front.sepia.ceph.com -o noatime 2024-09-16T03:41:36.812 DEBUG:teuthology.orchestra.run.smithi042:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-09-16T03:41:36.870 DEBUG:teuthology.orchestra.run.smithi042:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-09-16T03:41:36.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T03:41:36.926 INFO:teuthology.orchestra.run.smithi042.stderr:sudo: /sbin/restorecon: command not found 2024-09-16T03:41:36.926 DEBUG:teuthology.orchestra.run.smithi042:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-09-16T03:41:36.983 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-16T03:41:36.983 INFO:tasks.ceph:role: osd.2 2024-09-16T03:41:36.983 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi042.front.sepia.ceph.com 2024-09-16T03:41:36.983 DEBUG:teuthology.orchestra.run.smithi042:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-16T03:41:37.039 INFO:teuthology.orchestra.run.smithi042.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T03:41:37.039 INFO:teuthology.orchestra.run.smithi042.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T03:41:37.039 INFO:teuthology.orchestra.run.smithi042.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T03:41:37.039 INFO:teuthology.orchestra.run.smithi042.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-16T03:41:37.039 INFO:teuthology.orchestra.run.smithi042.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T03:41:37.039 INFO:teuthology.orchestra.run.smithi042.stdout: = sunit=0 swidth=0 blks 2024-09-16T03:41:37.039 INFO:teuthology.orchestra.run.smithi042.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T03:41:37.039 INFO:teuthology.orchestra.run.smithi042.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-16T03:41:37.039 INFO:teuthology.orchestra.run.smithi042.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T03:41:37.040 INFO:teuthology.orchestra.run.smithi042.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T03:41:37.062 INFO:teuthology.orchestra.run.smithi042.stdout:Discarding blocks...Done. 2024-09-16T03:41:37.064 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi042.front.sepia.ceph.com -o noatime 2024-09-16T03:41:37.065 DEBUG:teuthology.orchestra.run.smithi042:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-09-16T03:41:37.124 DEBUG:teuthology.orchestra.run.smithi042:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-09-16T03:41:37.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T03:41:37.178 INFO:teuthology.orchestra.run.smithi042.stderr:sudo: /sbin/restorecon: command not found 2024-09-16T03:41:37.178 DEBUG:teuthology.orchestra.run.smithi042:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T03:41:37.294 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:41:37.289+0000 7fbc28e597c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-09-16T03:41:37.295 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:41:37.289+0000 7fbc28e597c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-09-16T03:41:37.295 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:41:37.289+0000 7fbc28e597c0 -1 bdev(0x55b2460c5800 /var/lib/ceph/osd/ceph-0/block) open stat got: (1) Operation not permitted 2024-09-16T03:41:37.295 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:41:37.289+0000 7fbc28e597c0 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-09-16T03:41:40.459 DEBUG:teuthology.orchestra.run.smithi042:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-16T03:41:40.474 DEBUG:teuthology.orchestra.run.smithi042:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T03:41:40.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:41:40.549+0000 7fe98dc827c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-09-16T03:41:40.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:41:40.553+0000 7fe98dc827c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-09-16T03:41:40.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:41:40.553+0000 7fe98dc827c0 -1 bdev(0x56450c415800 /var/lib/ceph/osd/ceph-1/block) open stat got: (1) Operation not permitted 2024-09-16T03:41:40.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:41:40.553+0000 7fe98dc827c0 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-09-16T03:41:43.717 DEBUG:teuthology.orchestra.run.smithi042:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-16T03:41:43.772 DEBUG:teuthology.orchestra.run.smithi042:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T03:41:43.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:41:43.845+0000 7f44c8c337c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-09-16T03:41:43.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:41:43.849+0000 7f44c8c337c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-09-16T03:41:43.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:41:43.849+0000 7f44c8c337c0 -1 bdev(0x55970951f800 /var/lib/ceph/osd/ceph-2/block) open stat got: (1) Operation not permitted 2024-09-16T03:41:43.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:41:43.849+0000 7f44c8c337c0 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-09-16T03:41:47.005 DEBUG:teuthology.orchestra.run.smithi042:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-16T03:41:47.063 DEBUG:teuthology.orchestra.run.smithi154:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-09-16T03:41:47.078 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-09-16T03:41:47.078 INFO:tasks.ceph:role: osd.3 2024-09-16T03:41:47.079 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi154.front.sepia.ceph.com 2024-09-16T03:41:47.079 DEBUG:teuthology.orchestra.run.smithi154:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-16T03:41:47.143 INFO:teuthology.orchestra.run.smithi154.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T03:41:47.143 INFO:teuthology.orchestra.run.smithi154.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T03:41:47.143 INFO:teuthology.orchestra.run.smithi154.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T03:41:47.143 INFO:teuthology.orchestra.run.smithi154.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-16T03:41:47.143 INFO:teuthology.orchestra.run.smithi154.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T03:41:47.144 INFO:teuthology.orchestra.run.smithi154.stdout: = sunit=0 swidth=0 blks 2024-09-16T03:41:47.144 INFO:teuthology.orchestra.run.smithi154.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T03:41:47.144 INFO:teuthology.orchestra.run.smithi154.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-16T03:41:47.144 INFO:teuthology.orchestra.run.smithi154.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T03:41:47.144 INFO:teuthology.orchestra.run.smithi154.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T03:41:47.635 INFO:teuthology.orchestra.run.smithi154.stdout:Discarding blocks...Done. 2024-09-16T03:41:47.638 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi154.front.sepia.ceph.com -o noatime 2024-09-16T03:41:47.638 DEBUG:teuthology.orchestra.run.smithi154:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-3 2024-09-16T03:41:47.748 DEBUG:teuthology.orchestra.run.smithi154:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-09-16T03:41:47.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T03:41:47.761 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: /sbin/restorecon: command not found 2024-09-16T03:41:47.761 DEBUG:teuthology.orchestra.run.smithi154:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-09-16T03:41:47.815 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-09-16T03:41:47.815 INFO:tasks.ceph:role: osd.4 2024-09-16T03:41:47.815 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi154.front.sepia.ceph.com 2024-09-16T03:41:47.815 DEBUG:teuthology.orchestra.run.smithi154:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-16T03:41:47.878 INFO:teuthology.orchestra.run.smithi154.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T03:41:47.879 INFO:teuthology.orchestra.run.smithi154.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T03:41:47.879 INFO:teuthology.orchestra.run.smithi154.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T03:41:47.879 INFO:teuthology.orchestra.run.smithi154.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-16T03:41:47.879 INFO:teuthology.orchestra.run.smithi154.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T03:41:47.879 INFO:teuthology.orchestra.run.smithi154.stdout: = sunit=0 swidth=0 blks 2024-09-16T03:41:47.879 INFO:teuthology.orchestra.run.smithi154.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T03:41:47.879 INFO:teuthology.orchestra.run.smithi154.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-16T03:41:47.880 INFO:teuthology.orchestra.run.smithi154.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T03:41:47.880 INFO:teuthology.orchestra.run.smithi154.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T03:41:48.312 INFO:teuthology.orchestra.run.smithi154.stdout:Discarding blocks...Done. 2024-09-16T03:41:48.313 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi154.front.sepia.ceph.com -o noatime 2024-09-16T03:41:48.313 DEBUG:teuthology.orchestra.run.smithi154:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-4 2024-09-16T03:41:48.375 DEBUG:teuthology.orchestra.run.smithi154:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-09-16T03:41:48.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T03:41:48.429 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: /sbin/restorecon: command not found 2024-09-16T03:41:48.429 DEBUG:teuthology.orchestra.run.smithi154:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-09-16T03:41:48.481 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-09-16T03:41:48.481 INFO:tasks.ceph:role: osd.5 2024-09-16T03:41:48.481 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi154.front.sepia.ceph.com 2024-09-16T03:41:48.481 DEBUG:teuthology.orchestra.run.smithi154:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-16T03:41:48.542 INFO:teuthology.orchestra.run.smithi154.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T03:41:48.542 INFO:teuthology.orchestra.run.smithi154.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T03:41:48.542 INFO:teuthology.orchestra.run.smithi154.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T03:41:48.542 INFO:teuthology.orchestra.run.smithi154.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-16T03:41:48.542 INFO:teuthology.orchestra.run.smithi154.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T03:41:48.542 INFO:teuthology.orchestra.run.smithi154.stdout: = sunit=0 swidth=0 blks 2024-09-16T03:41:48.542 INFO:teuthology.orchestra.run.smithi154.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T03:41:48.542 INFO:teuthology.orchestra.run.smithi154.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-16T03:41:48.543 INFO:teuthology.orchestra.run.smithi154.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T03:41:48.543 INFO:teuthology.orchestra.run.smithi154.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T03:41:49.010 INFO:teuthology.orchestra.run.smithi154.stdout:Discarding blocks...Done. 2024-09-16T03:41:49.012 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi154.front.sepia.ceph.com -o noatime 2024-09-16T03:41:49.012 DEBUG:teuthology.orchestra.run.smithi154:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-5 2024-09-16T03:41:49.100 DEBUG:teuthology.orchestra.run.smithi154:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-09-16T03:41:49.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T03:41:49.111 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: /sbin/restorecon: command not found 2024-09-16T03:41:49.112 DEBUG:teuthology.orchestra.run.smithi154:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T03:41:49.221 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:41:49.218+0000 7f00997b47c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-09-16T03:41:49.221 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:41:49.218+0000 7f00997b47c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-09-16T03:41:49.221 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:41:49.218+0000 7f00997b47c0 -1 bdev(0x557a6f849800 /var/lib/ceph/osd/ceph-3/block) open stat got: (1) Operation not permitted 2024-09-16T03:41:49.221 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:41:49.218+0000 7f00997b47c0 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-09-16T03:41:52.359 DEBUG:teuthology.orchestra.run.smithi154:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-16T03:41:52.373 DEBUG:teuthology.orchestra.run.smithi154:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T03:41:52.456 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:41:52.454+0000 7efc399b37c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-09-16T03:41:52.457 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:41:52.454+0000 7efc399b37c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-09-16T03:41:52.457 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:41:52.454+0000 7efc399b37c0 -1 bdev(0x5596a7941800 /var/lib/ceph/osd/ceph-4/block) open stat got: (1) Operation not permitted 2024-09-16T03:41:52.457 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:41:52.454+0000 7efc399b37c0 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-09-16T03:41:55.642 DEBUG:teuthology.orchestra.run.smithi154:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-09-16T03:41:55.695 DEBUG:teuthology.orchestra.run.smithi154:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T03:41:55.778 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:41:55.774+0000 7fa5aa7577c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-09-16T03:41:55.778 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:41:55.774+0000 7fa5aa7577c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-09-16T03:41:55.778 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:41:55.774+0000 7fa5aa7577c0 -1 bdev(0x55f576273800 /var/lib/ceph/osd/ceph-5/block) open stat got: (1) Operation not permitted 2024-09-16T03:41:55.779 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:41:55.778+0000 7fa5aa7577c0 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-09-16T03:41:58.898 DEBUG:teuthology.orchestra.run.smithi154:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-09-16T03:41:58.961 INFO:tasks.ceph:Reading keys from all nodes... 2024-09-16T03:41:58.961 DEBUG:teuthology.orchestra.run.smithi042:> set -ex 2024-09-16T03:41:58.962 DEBUG:teuthology.orchestra.run.smithi042:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-09-16T03:41:58.977 DEBUG:teuthology.orchestra.run.smithi042:> set -ex 2024-09-16T03:41:58.977 DEBUG:teuthology.orchestra.run.smithi042:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-09-16T03:41:59.034 DEBUG:teuthology.orchestra.run.smithi042:> set -ex 2024-09-16T03:41:59.034 DEBUG:teuthology.orchestra.run.smithi042:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-09-16T03:41:59.090 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-09-16T03:41:59.090 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-09-16T03:41:59.104 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-09-16T03:41:59.104 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-09-16T03:41:59.163 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-09-16T03:41:59.163 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-09-16T03:41:59.218 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-09-16T03:41:59.218 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-09-16T03:41:59.274 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-09-16T03:41:59.275 DEBUG:teuthology.orchestra.run.smithi154:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-09-16T03:41:59.324 INFO:tasks.ceph:Adding keys to all mons... 2024-09-16T03:41:59.325 DEBUG:teuthology.orchestra.run.smithi042:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-16T03:41:59.329 DEBUG:teuthology.orchestra.run.smithi154:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-16T03:41:59.369 INFO:teuthology.orchestra.run.smithi042.stdout:[osd.0] 2024-09-16T03:41:59.369 INFO:teuthology.orchestra.run.smithi042.stdout: key = AQDxqOdm7qNtERAAaBdyw5Zjh4dVsZOeswXz8w== 2024-09-16T03:41:59.369 INFO:teuthology.orchestra.run.smithi042.stdout:[osd.1] 2024-09-16T03:41:59.369 INFO:teuthology.orchestra.run.smithi042.stdout: key = AQD0qOdmO7YiIRAAl0ZtWasF0rFLCnZ/ggBOtA== 2024-09-16T03:41:59.369 INFO:teuthology.orchestra.run.smithi042.stdout:[osd.2] 2024-09-16T03:41:59.369 INFO:teuthology.orchestra.run.smithi042.stdout: key = AQD3qOdm3EHJMhAAdT+efEMq25FyjspTlzA6+g== 2024-09-16T03:41:59.370 INFO:teuthology.orchestra.run.smithi042.stdout:[mgr.x] 2024-09-16T03:41:59.370 INFO:teuthology.orchestra.run.smithi042.stdout: key = AQDwqOdmrIDSFRAAGmba/pPBxe3dxwCP82HR5A== 2024-09-16T03:41:59.370 INFO:teuthology.orchestra.run.smithi042.stdout:[osd.3] 2024-09-16T03:41:59.370 INFO:teuthology.orchestra.run.smithi042.stdout: key = AQD9qOdmJKUzDRAAbe0J1vlvnLNGvL4u+dWcEA== 2024-09-16T03:41:59.370 INFO:teuthology.orchestra.run.smithi042.stdout:[osd.4] 2024-09-16T03:41:59.370 INFO:teuthology.orchestra.run.smithi042.stdout: key = AQAAqedmOY9RGxAALwmijfjmoH9BoZByzGEaWA== 2024-09-16T03:41:59.370 INFO:teuthology.orchestra.run.smithi042.stdout:[osd.5] 2024-09-16T03:41:59.370 INFO:teuthology.orchestra.run.smithi042.stdout: key = AQADqedmGad9LhAAlGgvG4wPWvY5jE6eqeZupQ== 2024-09-16T03:41:59.370 INFO:teuthology.orchestra.run.smithi042.stdout:[client.0] 2024-09-16T03:41:59.370 INFO:teuthology.orchestra.run.smithi042.stdout: key = AQDwqOdmfOoLGxAAcLOjrRbE7EVpBjrZ83Zucw== 2024-09-16T03:41:59.377 INFO:teuthology.orchestra.run.smithi154.stdout:[osd.0] 2024-09-16T03:41:59.377 INFO:teuthology.orchestra.run.smithi154.stdout: key = AQDxqOdm7qNtERAAaBdyw5Zjh4dVsZOeswXz8w== 2024-09-16T03:41:59.377 INFO:teuthology.orchestra.run.smithi154.stdout:[osd.1] 2024-09-16T03:41:59.377 INFO:teuthology.orchestra.run.smithi154.stdout: key = AQD0qOdmO7YiIRAAl0ZtWasF0rFLCnZ/ggBOtA== 2024-09-16T03:41:59.377 INFO:teuthology.orchestra.run.smithi154.stdout:[osd.2] 2024-09-16T03:41:59.378 INFO:teuthology.orchestra.run.smithi154.stdout: key = AQD3qOdm3EHJMhAAdT+efEMq25FyjspTlzA6+g== 2024-09-16T03:41:59.378 INFO:teuthology.orchestra.run.smithi154.stdout:[mgr.x] 2024-09-16T03:41:59.378 INFO:teuthology.orchestra.run.smithi154.stdout: key = AQDwqOdmrIDSFRAAGmba/pPBxe3dxwCP82HR5A== 2024-09-16T03:41:59.378 INFO:teuthology.orchestra.run.smithi154.stdout:[osd.3] 2024-09-16T03:41:59.378 INFO:teuthology.orchestra.run.smithi154.stdout: key = AQD9qOdmJKUzDRAAbe0J1vlvnLNGvL4u+dWcEA== 2024-09-16T03:41:59.378 INFO:teuthology.orchestra.run.smithi154.stdout:[osd.4] 2024-09-16T03:41:59.378 INFO:teuthology.orchestra.run.smithi154.stdout: key = AQAAqedmOY9RGxAALwmijfjmoH9BoZByzGEaWA== 2024-09-16T03:41:59.378 INFO:teuthology.orchestra.run.smithi154.stdout:[osd.5] 2024-09-16T03:41:59.378 INFO:teuthology.orchestra.run.smithi154.stdout: key = AQADqedmGad9LhAAlGgvG4wPWvY5jE6eqeZupQ== 2024-09-16T03:41:59.378 INFO:teuthology.orchestra.run.smithi154.stdout:[client.0] 2024-09-16T03:41:59.378 INFO:teuthology.orchestra.run.smithi154.stdout: key = AQDwqOdmfOoLGxAAcLOjrRbE7EVpBjrZ83Zucw== 2024-09-16T03:41:59.380 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T03:41:59.416 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T03:41:59.470 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T03:41:59.472 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T03:41:59.556 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T03:41:59.558 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T03:41:59.643 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-16T03:41:59.645 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-16T03:41:59.733 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T03:41:59.736 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T03:41:59.818 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T03:41:59.823 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T03:41:59.901 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T03:41:59.911 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T03:41:59.985 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T03:41:59.999 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T03:42:00.068 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-09-16T03:42:00.068 DEBUG:teuthology.orchestra.run.smithi042:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-09-16T03:42:00.080 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-16T03:42:00.172 DEBUG:teuthology.orchestra.run.smithi042:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-09-16T03:42:00.224 DEBUG:teuthology.orchestra.run.smithi042:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-09-16T03:42:00.276 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-16T03:42:00.370 DEBUG:teuthology.orchestra.run.smithi042:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-09-16T03:42:00.381 DEBUG:teuthology.orchestra.run.smithi154:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-09-16T03:42:00.395 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-16T03:42:00.494 DEBUG:teuthology.orchestra.run.smithi154:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-09-16T03:42:00.504 DEBUG:teuthology.orchestra.run.smithi042:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-16T03:42:00.506 DEBUG:teuthology.orchestra.run.smithi154:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-16T03:42:00.551 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-09-16T03:42:00.551 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-16T03:42:00.552 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-16T03:42:00.555 INFO:tasks.ceph.mon.a:Started 2024-09-16T03:42:00.555 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-16T03:42:00.555 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-16T03:42:00.558 INFO:tasks.ceph.mon.c:Started 2024-09-16T03:42:00.558 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-16T03:42:00.558 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-16T03:42:00.596 INFO:tasks.ceph.mon.b:Started 2024-09-16T03:42:00.596 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-09-16T03:42:00.597 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-09-16T03:42:00.597 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-09-16T03:42:00.599 INFO:tasks.ceph.mgr.x:Started 2024-09-16T03:42:00.599 INFO:tasks.ceph:Setting crush tunables to default 2024-09-16T03:42:00.599 DEBUG:teuthology.orchestra.run.smithi042:> sudo ceph --cluster ceph osd crush tunables default 2024-09-16T03:42:00.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:00.709+0000 7f3e96735640 10 monclient: get_monmap_and_config 2024-09-16T03:42:00.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:00.709+0000 7f3e96735640 10 monclient: build_initial_monmap 2024-09-16T03:42:00.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:00.709+0000 7f3e96735640 10 monclient: monmap: 2024-09-16T03:42:00.716 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:42:00.716 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:00.716 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:42:00.713192+0000 2024-09-16T03:42:00.716 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:42:00.713192+0000 2024-09-16T03:42:00.716 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:42:00.716 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:00.716 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:42:00.716 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:42:00.716 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:42:00.716 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:42:00.716 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:42:00.716 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:00.716 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:00.709+0000 7f3e96735640 1 Processor -- start 2024-09-16T03:42:00.716 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:00.709+0000 7f3e96735640 1 -- start start 2024-09-16T03:42:00.717 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:00.709+0000 7f3e96735640 10 monclient: init 2024-09-16T03:42:00.718 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:00.709+0000 7f3e96735640 10 monclient: _reopen_session rank -1 2024-09-16T03:42:00.718 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:00.709+0000 7f3e96735640 10 monclient: _add_conns ranks=[3,4,1,2,0] 2024-09-16T03:42:00.718 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:00.709+0000 7f3e96735640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f3e9006e7f0 0x7f3e900a4a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:00.719 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:00.709+0000 7f3e96735640 10 monclient(hunting): picked mon.noname-a con 0x7f3e9006e7f0 addr v2:172.21.15.42:3300/0 2024-09-16T03:42:00.719 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:00.709+0000 7f3e96735640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f3e900a5a00 0x7f3e900a4fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:00.719 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:00.709+0000 7f3e96735640 10 monclient(hunting): picked mon.noname-c con 0x7f3e900a5a00 addr v2:172.21.15.154:3300/0 2024-09-16T03:42:00.719 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:00.709+0000 7f3e96735640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3e90136f80 addr v1:172.21.15.154:6789/0 2024-09-16T03:42:00.719 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:00.709+0000 7f3e96735640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3e900a40d0 con 0x7f3e90136f80 2024-09-16T03:42:00.719 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:00.709+0000 7f3e96735640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:00.719 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:00.709+0000 7f3e96735640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f3e900a5510 con 0x7f3e9006e7f0 2024-09-16T03:42:00.719 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:00.709+0000 7f3e96735640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:00.719 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:00.709+0000 7f3e96735640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f3e9006ebd0 con 0x7f3e900a5a00 2024-09-16T03:42:00.719 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:00.709+0000 7f3e96735640 10 monclient(hunting): _renew_subs 2024-09-16T03:42:00.719 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:00.709+0000 7f3e96735640 10 monclient(hunting): authenticate will time out at 967.682341s 2024-09-16T03:42:00.719 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:00.709+0000 7f3e8ffff640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f3e9006e7f0 0x7f3e900a4a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:00.720 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:00.709+0000 7f3e8ffff640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f3e9006e7f0 0x7f3e900a4a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:39410/0 (socket says 172.21.15.42:39410) 2024-09-16T03:42:00.720 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:00.709+0000 7f3e8ffff640 1 -- 172.21.15.42:0/2002686821 learned_addr learned my addr 172.21.15.42:0/2002686821 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:42:00.720 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:00.709+0000 7f3e8ffff640 10 monclient(hunting): get_auth_request con 0x7f3e9006e7f0 auth_method 0 2024-09-16T03:42:00.720 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:00.709+0000 7f3e8ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:00.720 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:00.709+0000 7f3e8ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:00.720 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:00.709+0000 7f3e8ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:00.720 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:00.709+0000 7f3e8ffff640 1 -- 172.21.15.42:0/2002686821 >> v2:172.21.15.42:3300/0 conn(0x7f3e9006e7f0 msgr2=0x7f3e900a4a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T03:42:00.720 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:00.709+0000 7f3e8ffff640 1 -- 172.21.15.42:0/2002686821 >> v2:172.21.15.42:3300/0 conn(0x7f3e9006e7f0 msgr2=0x7f3e900a4a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:42:00.720 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:00.709+0000 7f3e8ffff640 1 --2- 172.21.15.42:0/2002686821 >> v2:172.21.15.42:3300/0 conn(0x7f3e9006e7f0 0x7f3e900a4a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:42:00.720 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:00.709+0000 7f3e8f7fe640 1 --2- 172.21.15.42:0/2002686821 >> v2:172.21.15.154:3300/0 conn(0x7f3e900a5a00 0x7f3e900a4fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:00.720 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:00.709+0000 7f3e8f7fe640 10 monclient(hunting): get_auth_request con 0x7f3e900a5a00 auth_method 0 2024-09-16T03:42:00.720 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:00.709+0000 7f3e8f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:00.720 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:00.709+0000 7f3e8f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:00.720 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:00.709+0000 7f3e8f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:00.721 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:00.709+0000 7f3e8ffff640 1 --2- 172.21.15.42:0/2002686821 >> v2:172.21.15.42:3300/0 conn(0x7f3e9006e7f0 0x7f3e900a4a90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:42:00.721 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:00.709+0000 7f3e8f7fe640 1 -- 172.21.15.42:0/2002686821 >> v2:172.21.15.154:3300/0 conn(0x7f3e900a5a00 msgr2=0x7f3e900a4fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T03:42:00.721 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:00.709+0000 7f3e8f7fe640 1 -- 172.21.15.42:0/2002686821 >> v2:172.21.15.154:3300/0 conn(0x7f3e900a5a00 msgr2=0x7f3e900a4fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:42:00.721 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:00.709+0000 7f3e8f7fe640 1 --2- 172.21.15.42:0/2002686821 >> v2:172.21.15.154:3300/0 conn(0x7f3e900a5a00 0x7f3e900a4fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:42:00.721 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:00.709+0000 7f3e8f7fe640 1 --2- 172.21.15.42:0/2002686821 >> v2:172.21.15.154:3300/0 conn(0x7f3e900a5a00 0x7f3e900a4fd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:42:00.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:00.913+0000 7f3e8ffff640 1 --2- 172.21.15.42:0/2002686821 >> v2:172.21.15.42:3300/0 conn(0x7f3e9006e7f0 0x7f3e900a4a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:00.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:00.913+0000 7f3e8ffff640 10 monclient(hunting): get_auth_request con 0x7f3e9006e7f0 auth_method 0 2024-09-16T03:42:00.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:00.913+0000 7f3e8ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:00.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:00.913+0000 7f3e8ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:00.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:00.913+0000 7f3e8ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:42:00.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:00.913+0000 7f3e8f7fe640 1 --2- 172.21.15.42:0/2002686821 >> v2:172.21.15.154:3300/0 conn(0x7f3e900a5a00 0x7f3e900a4fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:00.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:00.913+0000 7f3e8ffff640 1 -- 172.21.15.42:0/2002686821 >> v2:172.21.15.42:3300/0 conn(0x7f3e9006e7f0 msgr2=0x7f3e900a4a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T03:42:00.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:00.913+0000 7f3e8ffff640 1 -- 172.21.15.42:0/2002686821 >> v2:172.21.15.42:3300/0 conn(0x7f3e9006e7f0 msgr2=0x7f3e900a4a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:42:00.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:00.913+0000 7f3e8ffff640 1 --2- 172.21.15.42:0/2002686821 >> v2:172.21.15.42:3300/0 conn(0x7f3e9006e7f0 0x7f3e900a4a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:42:00.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:00.913+0000 7f3e8f7fe640 10 monclient(hunting): get_auth_request con 0x7f3e900a5a00 auth_method 0 2024-09-16T03:42:00.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:00.913+0000 7f3e8f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:00.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:00.913+0000 7f3e8f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:00.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:00.913+0000 7f3e8f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:42:00.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:00.913+0000 7f3e8ffff640 1 --2- 172.21.15.42:0/2002686821 >> v2:172.21.15.42:3300/0 conn(0x7f3e9006e7f0 0x7f3e900a4a90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T03:42:00.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:00.913+0000 7f3e8f7fe640 1 -- 172.21.15.42:0/2002686821 >> v2:172.21.15.154:3300/0 conn(0x7f3e900a5a00 msgr2=0x7f3e900a4fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T03:42:00.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:00.913+0000 7f3e8f7fe640 1 -- 172.21.15.42:0/2002686821 >> v2:172.21.15.154:3300/0 conn(0x7f3e900a5a00 msgr2=0x7f3e900a4fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:42:00.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:00.913+0000 7f3e8f7fe640 1 --2- 172.21.15.42:0/2002686821 >> v2:172.21.15.154:3300/0 conn(0x7f3e900a5a00 0x7f3e900a4fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:42:00.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:00.913+0000 7f3e8f7fe640 1 --2- 172.21.15.42:0/2002686821 >> v2:172.21.15.154:3300/0 conn(0x7f3e900a5a00 0x7f3e900a4fd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T03:42:01.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:01.317+0000 7f3e8ffff640 1 --2- 172.21.15.42:0/2002686821 >> v2:172.21.15.42:3300/0 conn(0x7f3e9006e7f0 0x7f3e900a4a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:01.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:01.317+0000 7f3e8ffff640 10 monclient(hunting): get_auth_request con 0x7f3e9006e7f0 auth_method 0 2024-09-16T03:42:01.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:01.317+0000 7f3e8ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:01.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:01.317+0000 7f3e8ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:01.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:01.317+0000 7f3e8ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:42:01.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:01.317+0000 7f3e8f7fe640 1 --2- 172.21.15.42:0/2002686821 >> v2:172.21.15.154:3300/0 conn(0x7f3e900a5a00 0x7f3e900a4fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:01.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:01.317+0000 7f3e8ffff640 1 -- 172.21.15.42:0/2002686821 >> v2:172.21.15.42:3300/0 conn(0x7f3e9006e7f0 msgr2=0x7f3e900a4a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T03:42:01.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:01.317+0000 7f3e8ffff640 1 -- 172.21.15.42:0/2002686821 >> v2:172.21.15.42:3300/0 conn(0x7f3e9006e7f0 msgr2=0x7f3e900a4a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:42:01.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:01.317+0000 7f3e8ffff640 1 --2- 172.21.15.42:0/2002686821 >> v2:172.21.15.42:3300/0 conn(0x7f3e9006e7f0 0x7f3e900a4a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:42:01.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:01.317+0000 7f3e8f7fe640 10 monclient(hunting): get_auth_request con 0x7f3e900a5a00 auth_method 0 2024-09-16T03:42:01.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:01.317+0000 7f3e8f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:01.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:01.317+0000 7f3e8f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:01.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:01.317+0000 7f3e8f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:42:01.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:01.317+0000 7f3e8ffff640 1 --2- 172.21.15.42:0/2002686821 >> v2:172.21.15.42:3300/0 conn(0x7f3e9006e7f0 0x7f3e900a4a90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T03:42:01.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:01.317+0000 7f3e8f7fe640 1 -- 172.21.15.42:0/2002686821 >> v2:172.21.15.154:3300/0 conn(0x7f3e900a5a00 msgr2=0x7f3e900a4fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T03:42:01.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:01.317+0000 7f3e8f7fe640 1 -- 172.21.15.42:0/2002686821 >> v2:172.21.15.154:3300/0 conn(0x7f3e900a5a00 msgr2=0x7f3e900a4fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:42:01.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:01.317+0000 7f3e8f7fe640 1 --2- 172.21.15.42:0/2002686821 >> v2:172.21.15.154:3300/0 conn(0x7f3e900a5a00 0x7f3e900a4fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:42:01.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:01.317+0000 7f3e8f7fe640 1 --2- 172.21.15.42:0/2002686821 >> v2:172.21.15.154:3300/0 conn(0x7f3e900a5a00 0x7f3e900a4fd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T03:42:02.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:02.121+0000 7f3e8ffff640 1 --2- 172.21.15.42:0/2002686821 >> v2:172.21.15.42:3300/0 conn(0x7f3e9006e7f0 0x7f3e900a4a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:02.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:02.121+0000 7f3e8ffff640 10 monclient(hunting): get_auth_request con 0x7f3e9006e7f0 auth_method 0 2024-09-16T03:42:02.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:02.121+0000 7f3e8ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:02.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:02.121+0000 7f3e8ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:02.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:02.121+0000 7f3e8ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:42:02.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:02.121+0000 7f3e8f7fe640 1 --2- 172.21.15.42:0/2002686821 >> v2:172.21.15.154:3300/0 conn(0x7f3e900a5a00 0x7f3e900a4fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:02.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:02.121+0000 7f3e8ffff640 1 -- 172.21.15.42:0/2002686821 >> v2:172.21.15.42:3300/0 conn(0x7f3e9006e7f0 msgr2=0x7f3e900a4a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T03:42:02.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:02.121+0000 7f3e8ffff640 1 -- 172.21.15.42:0/2002686821 >> v2:172.21.15.42:3300/0 conn(0x7f3e9006e7f0 msgr2=0x7f3e900a4a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:42:02.128 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:02.121+0000 7f3e8ffff640 1 --2- 172.21.15.42:0/2002686821 >> v2:172.21.15.42:3300/0 conn(0x7f3e9006e7f0 0x7f3e900a4a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:42:02.128 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:02.121+0000 7f3e8f7fe640 10 monclient(hunting): get_auth_request con 0x7f3e900a5a00 auth_method 0 2024-09-16T03:42:02.128 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:02.121+0000 7f3e8f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:02.128 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:02.121+0000 7f3e8f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:02.128 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:02.121+0000 7f3e8f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:42:02.128 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:02.121+0000 7f3e8ffff640 1 --2- 172.21.15.42:0/2002686821 >> v2:172.21.15.42:3300/0 conn(0x7f3e9006e7f0 0x7f3e900a4a90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T03:42:02.128 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:02.121+0000 7f3e8f7fe640 1 -- 172.21.15.42:0/2002686821 >> v2:172.21.15.154:3300/0 conn(0x7f3e900a5a00 msgr2=0x7f3e900a4fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T03:42:02.128 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:02.121+0000 7f3e8f7fe640 1 -- 172.21.15.42:0/2002686821 >> v2:172.21.15.154:3300/0 conn(0x7f3e900a5a00 msgr2=0x7f3e900a4fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:42:02.128 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:02.121+0000 7f3e8f7fe640 1 --2- 172.21.15.42:0/2002686821 >> v2:172.21.15.154:3300/0 conn(0x7f3e900a5a00 0x7f3e900a4fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:42:02.128 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:02.121+0000 7f3e8f7fe640 1 --2- 172.21.15.42:0/2002686821 >> v2:172.21.15.154:3300/0 conn(0x7f3e900a5a00 0x7f3e900a4fd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T03:42:03.717 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:03.709+0000 7f3e8dffb640 10 monclient(hunting): tick 2024-09-16T03:42:03.717 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:03.709+0000 7f3e8dffb640 10 monclient(hunting): _check_auth_tickets 2024-09-16T03:42:03.717 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:03.709+0000 7f3e8dffb640 1 monclient(hunting): continuing hunt 2024-09-16T03:42:03.717 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:03.709+0000 7f3e8dffb640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T03:42:03.717 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:03.709+0000 7f3e8dffb640 1 -- 172.21.15.42:0/2002686821 >> v2:172.21.15.154:3300/0 conn(0x7f3e900a5a00 msgr2=0x7f3e900a4fd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:42:03.717 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:03.709+0000 7f3e8dffb640 1 --2- 172.21.15.42:0/2002686821 >> v2:172.21.15.154:3300/0 conn(0x7f3e900a5a00 0x7f3e900a4fd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:03.717 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:03.709+0000 7f3e8dffb640 1 -- 172.21.15.42:0/2002686821 >> v2:172.21.15.42:3300/0 conn(0x7f3e9006e7f0 msgr2=0x7f3e900a4a90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:42:03.717 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:03.709+0000 7f3e8dffb640 1 --2- 172.21.15.42:0/2002686821 >> v2:172.21.15.42:3300/0 conn(0x7f3e9006e7f0 0x7f3e900a4a90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:03.718 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:03.709+0000 7f3e8dffb640 1 -- 172.21.15.42:0/2002686821 >> v1:172.21.15.154:6789/0 conn(0x7f3e90136f80 legacy=0x7f3e90137410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:03.718 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:03.709+0000 7f3e8dffb640 10 monclient: _add_conns ranks=[2,0] 2024-09-16T03:42:03.718 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:03.709+0000 7f3e8dffb640 1 --2- 172.21.15.42:0/2002686821 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3e7c0036d0 0x7f3e7c005b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:03.718 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:03.709+0000 7f3e8dffb640 10 monclient(hunting): picked mon.noname-b con 0x7f3e7c0036d0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:42:03.718 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:03.709+0000 7f3e8dffb640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3e7c006230 addr v1:172.21.15.42:6789/0 2024-09-16T03:42:03.718 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:03.709+0000 7f3e8dffb640 1 -- 172.21.15.42:0/2002686821 --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3e9009be60 con 0x7f3e7c006230 2024-09-16T03:42:03.718 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:03.709+0000 7f3e8dffb640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:03.718 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:03.709+0000 7f3e8dffb640 1 -- 172.21.15.42:0/2002686821 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f3e9006ebd0 con 0x7f3e7c0036d0 2024-09-16T03:42:03.718 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:03.709+0000 7f3e8dffb640 10 monclient(hunting): _renew_subs 2024-09-16T03:42:03.718 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:03.709+0000 7f3e8ffff640 1 --2- 172.21.15.42:0/2002686821 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3e7c0036d0 0x7f3e7c005b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:03.718 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:03.709+0000 7f3e8ffff640 10 monclient(hunting): get_auth_request con 0x7f3e7c0036d0 auth_method 0 2024-09-16T03:42:03.718 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:03.709+0000 7f3e8ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:03.718 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:03.709+0000 7f3e8ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:03.719 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:03.709+0000 7f3e8ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:03.719 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:03.709+0000 7f3e8ffff640 1 -- 172.21.15.42:0/2002686821 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3e7c0036d0 msgr2=0x7f3e7c005b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T03:42:03.719 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:03.709+0000 7f3e8ffff640 1 -- 172.21.15.42:0/2002686821 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3e7c0036d0 msgr2=0x7f3e7c005b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:42:03.719 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:03.709+0000 7f3e8ffff640 1 --2- 172.21.15.42:0/2002686821 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3e7c0036d0 0x7f3e7c005b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:42:03.719 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:03.709+0000 7f3e8ffff640 1 --2- 172.21.15.42:0/2002686821 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3e7c0036d0 0x7f3e7c005b00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:42:03.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:03.913+0000 7f3e8ffff640 1 --2- 172.21.15.42:0/2002686821 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3e7c0036d0 0x7f3e7c005b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:03.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:03.913+0000 7f3e8ffff640 10 monclient(hunting): get_auth_request con 0x7f3e7c0036d0 auth_method 0 2024-09-16T03:42:03.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:03.913+0000 7f3e8ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:03.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:03.913+0000 7f3e8ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:03.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:03.913+0000 7f3e8ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:42:03.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:03.913+0000 7f3e8ffff640 1 -- 172.21.15.42:0/2002686821 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3e7c0036d0 msgr2=0x7f3e7c005b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T03:42:03.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:03.913+0000 7f3e8ffff640 1 -- 172.21.15.42:0/2002686821 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3e7c0036d0 msgr2=0x7f3e7c005b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:42:03.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:03.913+0000 7f3e8ffff640 1 --2- 172.21.15.42:0/2002686821 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3e7c0036d0 0x7f3e7c005b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:42:03.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:03.913+0000 7f3e8ffff640 1 --2- 172.21.15.42:0/2002686821 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3e7c0036d0 0x7f3e7c005b00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T03:42:04.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:04.317+0000 7f3e8ffff640 1 --2- 172.21.15.42:0/2002686821 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3e7c0036d0 0x7f3e7c005b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:04.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:04.317+0000 7f3e8ffff640 10 monclient(hunting): get_auth_request con 0x7f3e7c0036d0 auth_method 0 2024-09-16T03:42:04.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:04.317+0000 7f3e8ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:04.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:04.317+0000 7f3e8ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:04.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:04.317+0000 7f3e8ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:42:04.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:04.317+0000 7f3e8ffff640 1 -- 172.21.15.42:0/2002686821 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3e7c0036d0 msgr2=0x7f3e7c005b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T03:42:04.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:04.317+0000 7f3e8ffff640 1 -- 172.21.15.42:0/2002686821 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3e7c0036d0 msgr2=0x7f3e7c005b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:42:04.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:04.317+0000 7f3e8ffff640 1 --2- 172.21.15.42:0/2002686821 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3e7c0036d0 0x7f3e7c005b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:42:04.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:04.317+0000 7f3e8ffff640 1 --2- 172.21.15.42:0/2002686821 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3e7c0036d0 0x7f3e7c005b00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T03:42:05.126 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:05.121+0000 7f3e8ffff640 1 --2- 172.21.15.42:0/2002686821 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3e7c0036d0 0x7f3e7c005b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:05.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:05.121+0000 7f3e8ffff640 10 monclient(hunting): get_auth_request con 0x7f3e7c0036d0 auth_method 0 2024-09-16T03:42:05.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:05.121+0000 7f3e8ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:05.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:05.121+0000 7f3e8ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:05.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:05.121+0000 7f3e8ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:42:05.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:05.121+0000 7f3e8ffff640 1 -- 172.21.15.42:0/2002686821 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3e7c0036d0 msgr2=0x7f3e7c005b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T03:42:05.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:05.121+0000 7f3e8ffff640 1 -- 172.21.15.42:0/2002686821 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3e7c0036d0 msgr2=0x7f3e7c005b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:42:05.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:05.121+0000 7f3e8ffff640 1 --2- 172.21.15.42:0/2002686821 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3e7c0036d0 0x7f3e7c005b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:42:05.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:05.121+0000 7f3e8ffff640 1 --2- 172.21.15.42:0/2002686821 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3e7c0036d0 0x7f3e7c005b00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T03:42:06.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:05.997+0000 7f3e8effd640 1 -- 172.21.15.42:0/2002686821 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2384415072 0 0) 0x7f3e9009be60 con 0x7f3e7c006230 2024-09-16T03:42:06.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:05.997+0000 7f3e8effd640 10 monclient(hunting): _negotiate 2024-09-16T03:42:06.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:05.997+0000 7f3e8effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:06.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:05.997+0000 7f3e8effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:06.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:05.997+0000 7f3e8effd640 10 monclient(hunting): my global_id is 4098 2024-09-16T03:42:06.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:05.997+0000 7f3e8effd640 1 -- 172.21.15.42:0/2002686821 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3e78003530 con 0x7f3e7c006230 2024-09-16T03:42:06.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:05.997+0000 7f3e8f7fe640 1 -- 172.21.15.42:0/2002686821 >> v1:172.21.15.42:6789/0 conn(0x7f3e7c006230 legacy=0x7f3e7c0086f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-09-16T03:42:06.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:05.997+0000 7f3e8f7fe640 1 -- 172.21.15.42:0/2002686821 >> v1:172.21.15.42:6789/0 conn(0x7f3e7c006230 legacy=0x7f3e7c0086f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T03:42:06.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:05.997+0000 7f3e8f7fe640 1 --1- 172.21.15.42:0/2002686821 >> v1:172.21.15.42:6789/0 conn(0x7f3e7c006230 0x7f3e7c0086f0 :-1 s=OPENED pgs=5 cs=1 l=1).handle_message read tag failed 2024-09-16T03:42:06.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:05.997+0000 7f3e8f7fe640 1 --1- 172.21.15.42:0/2002686821 >> v1:172.21.15.42:6789/0 conn(0x7f3e7c006230 0x7f3e7c0086f0 :-1 s=OPENED pgs=5 cs=1 l=1).fault on lossy channel, failing 2024-09-16T03:42:06.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:05.997+0000 7f3e8effd640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.42:6789/0 2024-09-16T03:42:06.718 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.709+0000 7f3e8dffb640 10 monclient(hunting): tick 2024-09-16T03:42:06.718 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.709+0000 7f3e8dffb640 10 monclient(hunting): _check_auth_tickets 2024-09-16T03:42:06.719 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.709+0000 7f3e8dffb640 1 monclient(hunting): continuing hunt 2024-09-16T03:42:06.719 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.709+0000 7f3e8dffb640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T03:42:06.719 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.709+0000 7f3e8dffb640 1 -- 172.21.15.42:0/2002686821 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3e7c0036d0 msgr2=0x7f3e7c005b00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:42:06.719 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.709+0000 7f3e8dffb640 1 --2- 172.21.15.42:0/2002686821 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3e7c0036d0 0x7f3e7c005b00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:06.719 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.709+0000 7f3e8dffb640 1 -- 172.21.15.42:0/2002686821 >> v1:172.21.15.42:6789/0 conn(0x7f3e7c006230 legacy=0x7f3e7c0086f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:42:06.719 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.709+0000 7f3e94cd2640 1 -- 172.21.15.42:0/2002686821 reap_dead start 2024-09-16T03:42:06.719 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.709+0000 7f3e8dffb640 10 monclient: _add_conns ranks=[0,1,3,2,4] 2024-09-16T03:42:06.719 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.709+0000 7f3e8dffb640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3e7c009fe0 addr v1:172.21.15.42:6789/0 2024-09-16T03:42:06.719 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.709+0000 7f3e8dffb640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3e7c00ab70 addr v1:172.21.15.154:6789/0 2024-09-16T03:42:06.719 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.709+0000 7f3e8dffb640 1 --2- 172.21.15.42:0/2002686821 >> v2:172.21.15.42:3300/0 conn(0x7f3e7c00c370 0x7f3e7c00e760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:06.719 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.709+0000 7f3e8dffb640 10 monclient(hunting): picked mon.noname-a con 0x7f3e7c00c370 addr v2:172.21.15.42:3300/0 2024-09-16T03:42:06.719 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.709+0000 7f3e8dffb640 1 -- 172.21.15.42:0/2002686821 --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3e7c00b050 con 0x7f3e7c009fe0 2024-09-16T03:42:06.719 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.709+0000 7f3e8dffb640 1 -- 172.21.15.42:0/2002686821 --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3e7c00fcb0 con 0x7f3e7c00ab70 2024-09-16T03:42:06.720 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.709+0000 7f3e8dffb640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:06.720 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.709+0000 7f3e8dffb640 1 -- 172.21.15.42:0/2002686821 --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f3e9006ebd0 con 0x7f3e7c00c370 2024-09-16T03:42:06.720 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.709+0000 7f3e8dffb640 10 monclient(hunting): _renew_subs 2024-09-16T03:42:06.720 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.709+0000 7f3e94cd2640 1 --2- 172.21.15.42:0/2002686821 >> v2:172.21.15.42:3300/0 conn(0x7f3e7c00c370 0x7f3e7c00e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:06.720 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.709+0000 7f3e94cd2640 10 monclient(hunting): get_auth_request con 0x7f3e7c00c370 auth_method 0 2024-09-16T03:42:06.720 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.709+0000 7f3e94cd2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:06.720 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.709+0000 7f3e94cd2640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:06.720 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.709+0000 7f3e94cd2640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:06.720 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.713+0000 7f3e94cd2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:06.720 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.713+0000 7f3e94cd2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:06.720 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.713+0000 7f3e94cd2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:06.720 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.713+0000 7f3e8effd640 1 -- 172.21.15.42:0/2002686821 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3762194019 0 0) 0x7f3e7c00fcb0 con 0x7f3e7c00ab70 2024-09-16T03:42:06.720 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.713+0000 7f3e8effd640 10 monclient(hunting): _negotiate 2024-09-16T03:42:06.721 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.713+0000 7f3e8effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:06.721 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.713+0000 7f3e8effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:06.721 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.713+0000 7f3e8effd640 10 monclient(hunting): my global_id is 4105 2024-09-16T03:42:06.721 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.713+0000 7f3e8effd640 1 -- 172.21.15.42:0/2002686821 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3e9009be60 con 0x7f3e7c00ab70 2024-09-16T03:42:06.721 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.713+0000 7f3e94cd2640 10 monclient(hunting): handle_auth_done global_id 4110 payload 274 2024-09-16T03:42:06.721 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.713+0000 7f3e94cd2640 1 -- 172.21.15.42:0/2002686821 >> v1:172.21.15.154:6789/0 conn(0x7f3e7c00ab70 legacy=0x7f3e7c0086f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:06.721 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.713+0000 7f3e94cd2640 1 -- 172.21.15.42:0/2002686821 >> v1:172.21.15.42:6789/0 conn(0x7f3e7c009fe0 legacy=0x7f3e7c00a3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:06.721 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.713+0000 7f3e8effd640 1 -- 172.21.15.42:0/2002686821 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1487967697 0 0) 0x7f3e7c00b050 con 0x7f3e7c009fe0 2024-09-16T03:42:06.721 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.713+0000 7f3e94cd2640 10 monclient: _finish_hunting 0 2024-09-16T03:42:06.721 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.713+0000 7f3e94cd2640 1 monclient: found mon.noname-a 2024-09-16T03:42:06.721 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.713+0000 7f3e94cd2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:42:06.721 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.713+0000 7f3e94cd2640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:42:06.722 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.713+0000 7f3e94cd2640 1 -- 172.21.15.42:0/2002686821 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e7c010e90 con 0x7f3e7c00c370 2024-09-16T03:42:06.722 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.713+0000 7f3e8effd640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T03:42:06.724 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.717+0000 7f3e94cd2640 1 --2- 172.21.15.42:0/2002686821 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3e7c00c370 0x7f3e7c00e760 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f3e900a56e0 tx=0x7f3e8801c360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44e197fd59fdb362 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:06.724 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.717+0000 7f3e8effd640 1 -- 172.21.15.42:0/2002686821 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e88005d50 con 0x7f3e7c00c370 2024-09-16T03:42:06.725 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.717+0000 7f3e8effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:06.725 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.717+0000 7f3e8effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:42:06.725 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.717+0000 7f3e8effd640 10 monclient: dump: 2024-09-16T03:42:06.725 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:06.725 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:06.725 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:06.725 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:06.725 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:06.725 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:06.725 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:06.725 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:06.725 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:06.726 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:06.726 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.717+0000 7f3e8effd640 10 monclient: _finish_auth 0 2024-09-16T03:42:06.726 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.717+0000 7f3e8effd640 10 monclient: _check_auth_tickets 2024-09-16T03:42:06.726 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.717+0000 7f3e8effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:42:06.726 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.717+0000 7f3e8effd640 1 -- 172.21.15.42:0/2002686821 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e88003980 con 0x7f3e7c00c370 2024-09-16T03:42:06.726 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.717+0000 7f3e96735640 5 monclient: authenticate success, global_id 4110 2024-09-16T03:42:06.726 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.717+0000 7f3e96735640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:42:06.726 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.717+0000 7f3e8effd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:42:06.726 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.717+0000 7f3e96735640 10 monclient: get_monmap_and_config success 2024-09-16T03:42:06.726 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.717+0000 7f3e96735640 4 set_mon_vals no callback set 2024-09-16T03:42:06.726 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.717+0000 7f3e8effd640 1 -- 172.21.15.42:0/2002686821 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e88003d70 con 0x7f3e7c00c370 2024-09-16T03:42:06.726 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.717+0000 7f3e8effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:06.726 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.717+0000 7f3e8effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:42:06.727 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.717+0000 7f3e8effd640 10 monclient: dump: 2024-09-16T03:42:06.727 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:06.727 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:06.727 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:06.727 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:06.727 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:06.727 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:06.727 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:06.727 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:06.727 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:06.727 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:06.728 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.721+0000 7f3e96735640 10 monclient: shutdown 2024-09-16T03:42:06.728 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.721+0000 7f3e96735640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:42:06.728 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.721+0000 7f3e96735640 1 -- 172.21.15.42:0/2002686821 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3e7c00c370 msgr2=0x7f3e7c00e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:06.728 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.721+0000 7f3e96735640 1 --2- 172.21.15.42:0/2002686821 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3e7c00c370 0x7f3e7c00e760 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f3e900a56e0 tx=0x7f3e8801c360 comp rx=0 tx=0).stop 2024-09-16T03:42:06.728 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.721+0000 7f3e96735640 1 -- 172.21.15.42:0/2002686821 shutdown_connections 2024-09-16T03:42:06.729 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.721+0000 7f3e96735640 1 --2- 172.21.15.42:0/2002686821 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3e7c00c370 0x7f3e7c00e760 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:06.729 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.721+0000 7f3e96735640 1 -- 172.21.15.42:0/2002686821 >> 172.21.15.42:0/2002686821 conn(0x7f3e900aa1b0 msgr2=0x7f3e900a66b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:42:06.729 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.721+0000 7f3e96735640 1 -- 172.21.15.42:0/2002686821 shutdown_connections 2024-09-16T03:42:06.729 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.721+0000 7f3e96735640 1 -- 172.21.15.42:0/2002686821 wait complete. 2024-09-16T03:42:06.729 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.721+0000 7f3e96735640 10 monclient: build_initial_monmap 2024-09-16T03:42:06.729 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.721+0000 7f3e96735640 10 monclient: monmap: 2024-09-16T03:42:06.729 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:42:06.729 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:42:06.729 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:42:06.730 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:42:06.730 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:42:06.730 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:06.730 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:42:06.730 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:42:06.730 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:42:06.730 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:06.737 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.721+0000 7f3e96735640 1 librados: starting msgr at 2024-09-16T03:42:06.737 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.721+0000 7f3e96735640 1 librados: starting objecter 2024-09-16T03:42:06.737 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.721+0000 7f3e96735640 1 Processor -- start 2024-09-16T03:42:06.738 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.721+0000 7f3e96735640 1 -- start start 2024-09-16T03:42:06.738 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.721+0000 7f3e96735640 1 librados: setting wanted keys 2024-09-16T03:42:06.738 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.721+0000 7f3e96735640 1 librados: calling monclient init 2024-09-16T03:42:06.738 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.721+0000 7f3e96735640 10 monclient: init 2024-09-16T03:42:06.738 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.725+0000 7f3e96735640 10 monclient: _reopen_session rank -1 2024-09-16T03:42:06.738 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.725+0000 7f3e96735640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:42:06.738 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.725+0000 7f3e96735640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3e7c009fe0 0x7f3e901cf6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:06.738 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.725+0000 7f3e96735640 10 monclient(hunting): picked mon.noname-b con 0x7f3e7c009fe0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:42:06.738 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.725+0000 7f3e96735640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3e7c00ab70 0x7f3e9012d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:06.738 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.725+0000 7f3e96735640 10 monclient(hunting): picked mon.noname-a con 0x7f3e7c00ab70 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:42:06.738 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.725+0000 7f3e96735640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3e7c00c370 0x7f3e9012dca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:06.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.725+0000 7f3e96735640 10 monclient(hunting): picked mon.noname-c con 0x7f3e7c00c370 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:42:06.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.725+0000 7f3e96735640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:06.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.725+0000 7f3e96735640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f3e90137510 con 0x7f3e7c00ab70 2024-09-16T03:42:06.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.725+0000 7f3e96735640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:06.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.725+0000 7f3e96735640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f3e90137390 con 0x7f3e7c009fe0 2024-09-16T03:42:06.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.725+0000 7f3e96735640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:06.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.725+0000 7f3e96735640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f3e90137690 con 0x7f3e7c00c370 2024-09-16T03:42:06.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.725+0000 7f3e96735640 10 monclient(hunting): _renew_subs 2024-09-16T03:42:06.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.725+0000 7f3e96735640 10 monclient(hunting): authenticate will time out at 973.696716s 2024-09-16T03:42:06.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.725+0000 7f3e8f7fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3e7c00ab70 0x7f3e9012d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:06.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.725+0000 7f3e94cd2640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3e7c00c370 0x7f3e9012dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:06.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.725+0000 7f3e8f7fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3e7c00ab70 0x7f3e9012d760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:53410/0 (socket says 172.21.15.42:53410) 2024-09-16T03:42:06.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.725+0000 7f3e94cd2640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3e7c00c370 0x7f3e9012dca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:54898/0 (socket says 172.21.15.42:54898) 2024-09-16T03:42:06.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.725+0000 7f3e8f7fe640 1 -- 172.21.15.42:0/222951544 learned_addr learned my addr 172.21.15.42:0/222951544 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:42:06.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.725+0000 7f3e8f7fe640 10 monclient(hunting): get_auth_request con 0x7f3e7c00ab70 auth_method 0 2024-09-16T03:42:06.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.725+0000 7f3e8f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:06.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.725+0000 7f3e8f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:06.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.725+0000 7f3e8f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:06.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.725+0000 7f3e94cd2640 10 monclient(hunting): get_auth_request con 0x7f3e7c00c370 auth_method 0 2024-09-16T03:42:06.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.725+0000 7f3e94cd2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:06.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.725+0000 7f3e94cd2640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:06.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.725+0000 7f3e94cd2640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:06.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.725+0000 7f3e8ffff640 1 --2- 172.21.15.42:0/222951544 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3e7c009fe0 0x7f3e901cf6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:06.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.725+0000 7f3e8f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:06.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.725+0000 7f3e8f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:06.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.725+0000 7f3e8f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:06.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.725+0000 7f3e94cd2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:06.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.725+0000 7f3e94cd2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:06.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.725+0000 7f3e94cd2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:06.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.725+0000 7f3e8ffff640 10 monclient(hunting): get_auth_request con 0x7f3e7c009fe0 auth_method 0 2024-09-16T03:42:06.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.725+0000 7f3e8ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:06.742 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.725+0000 7f3e8ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:06.742 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.725+0000 7f3e8ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:06.742 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.725+0000 7f3e8ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:06.742 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.725+0000 7f3e8ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:06.742 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.725+0000 7f3e8ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:06.742 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.725+0000 7f3e8f7fe640 10 monclient(hunting): handle_auth_done global_id 4116 payload 824 2024-09-16T03:42:06.742 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.725+0000 7f3e8f7fe640 1 -- 172.21.15.42:0/222951544 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3e7c00c370 msgr2=0x7f3e9012dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:06.742 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.725+0000 7f3e8f7fe640 1 --2- 172.21.15.42:0/222951544 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3e7c00c370 0x7f3e9012dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:06.742 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.725+0000 7f3e8f7fe640 1 -- 172.21.15.42:0/222951544 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3e7c009fe0 msgr2=0x7f3e901cf6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:06.742 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.725+0000 7f3e8f7fe640 1 --2- 172.21.15.42:0/222951544 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3e7c009fe0 0x7f3e901cf6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:06.742 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.725+0000 7f3e8f7fe640 10 monclient: _finish_hunting 0 2024-09-16T03:42:06.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.725+0000 7f3e8f7fe640 1 monclient: found mon.noname-a 2024-09-16T03:42:06.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.725+0000 7f3e8f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:42:06.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.725+0000 7f3e8f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:42:06.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.725+0000 7f3e8f7fe640 1 -- 172.21.15.42:0/222951544 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e900a37a0 con 0x7f3e7c00ab70 2024-09-16T03:42:06.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.725+0000 7f3e8f7fe640 1 --2- 172.21.15.42:0/222951544 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3e7c00ab70 0x7f3e9012d760 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f3e84000c10 tx=0x7f3e84011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:06.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.725+0000 7f3e8cff9640 10 client.?.objecter ms_handle_connect 0x7f3e7c00ab70 2024-09-16T03:42:06.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.725+0000 7f3e8cff9640 10 client.?.objecter resend_mon_ops 2024-09-16T03:42:06.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.725+0000 7f3e8cff9640 1 -- 172.21.15.42:0/222951544 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e8400f040 con 0x7f3e7c00ab70 2024-09-16T03:42:06.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.725+0000 7f3e8cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:06.744 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.725+0000 7f3e8cff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:42:06.744 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.725+0000 7f3e8cff9640 10 monclient: dump: 2024-09-16T03:42:06.744 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:06.744 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:06.744 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:06.744 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:06.744 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:06.744 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:06.744 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:06.744 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:06.744 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:06.744 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:06.744 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.725+0000 7f3e8cff9640 10 monclient: _finish_auth 0 2024-09-16T03:42:06.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.725+0000 7f3e8cff9640 10 monclient: _check_auth_tickets 2024-09-16T03:42:06.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.725+0000 7f3e8cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:42:06.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.725+0000 7f3e8cff9640 1 -- 172.21.15.42:0/222951544 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e8401a070 con 0x7f3e7c00ab70 2024-09-16T03:42:06.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.725+0000 7f3e8cff9640 10 monclient: handle_config config(0 keys) 2024-09-16T03:42:06.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.725+0000 7f3e8cff9640 1 -- 172.21.15.42:0/222951544 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e84003620 con 0x7f3e7c00ab70 2024-09-16T03:42:06.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.725+0000 7f3e8cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:06.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.725+0000 7f3e8cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:42:06.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.725+0000 7f3e8cff9640 10 monclient: dump: 2024-09-16T03:42:06.745 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:06.745 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:06.745 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:06.745 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:06.745 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:06.746 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:06.746 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:06.746 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:06.746 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:06.746 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:06.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.725+0000 7f3e96735640 5 monclient: authenticate success, global_id 4116 2024-09-16T03:42:06.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.725+0000 7f3e96735640 10 monclient: _renew_subs 2024-09-16T03:42:06.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.725+0000 7f3e8e7fc640 4 set_mon_vals no callback set 2024-09-16T03:42:06.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.725+0000 7f3e96735640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:42:06.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.725+0000 7f3e96735640 1 -- 172.21.15.42:0/222951544 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e9012e1e0 con 0x7f3e7c00ab70 2024-09-16T03:42:06.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.725+0000 7f3e96735640 10 client.4116.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:42:06.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.725+0000 7f3e96735640 10 monclient: _renew_subs 2024-09-16T03:42:06.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.725+0000 7f3e96735640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:42:06.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.725+0000 7f3e96735640 1 -- 172.21.15.42:0/222951544 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e90135740 con 0x7f3e7c00ab70 2024-09-16T03:42:06.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.725+0000 7f3e96735640 1 librados: init done 2024-09-16T03:42:06.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.725+0000 7f3e8cff9640 1 -- 172.21.15.42:0/222951544 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f3e840037c0 con 0x7f3e7c00ab70 2024-09-16T03:42:06.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.725+0000 7f3e8f7fe640 1 -- 172.21.15.42:0/222951544 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3e7c00ab70 msgr2=0x7f3e9012d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-16T03:42:06.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.725+0000 7f3e8f7fe640 1 -- 172.21.15.42:0/222951544 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3e7c00ab70 msgr2=0x7f3e9012d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T03:42:06.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.725+0000 7f3e8f7fe640 1 --2- 172.21.15.42:0/222951544 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3e7c00ab70 0x7f3e9012d760 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f3e84000c10 tx=0x7f3e84011a30 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:42:06.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.725+0000 7f3e8f7fe640 1 --2- 172.21.15.42:0/222951544 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3e7c00ab70 0x7f3e9012d760 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f3e84000c10 tx=0x7f3e84011a30 comp rx=0 tx=0).stop 2024-09-16T03:42:06.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.725+0000 7f3e8cff9640 10 monclient: ms_handle_reset current mon [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:42:06.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.725+0000 7f3e8cff9640 10 monclient: _reopen_session rank -1 2024-09-16T03:42:06.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.725+0000 7f3e8cff9640 1 -- 172.21.15.42:0/222951544 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3e7c00ab70 msgr2=0x7f3e9012d760 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:42:06.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.725+0000 7f3e8cff9640 1 --2- 172.21.15.42:0/222951544 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3e7c00ab70 0x7f3e9012d760 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:06.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.725+0000 7f3e8cff9640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:42:06.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.725+0000 7f3e8cff9640 1 --2- 172.21.15.42:0/222951544 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3e7c00c370 0x7f3e7c008620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:06.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.725+0000 7f3e8cff9640 10 monclient(hunting): picked mon.c con 0x7f3e7c00c370 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:42:06.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.725+0000 7f3e8cff9640 1 --2- 172.21.15.42:0/222951544 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3e7c010400 0x7f3e7c010880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:06.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.725+0000 7f3e8cff9640 10 monclient(hunting): picked mon.a con 0x7f3e7c010400 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:42:06.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.725+0000 7f3e8cff9640 1 --2- 172.21.15.42:0/222951544 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3e7c009fe0 0x7f3e7c002fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:06.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.725+0000 7f3e8cff9640 10 monclient(hunting): picked mon.b con 0x7f3e7c009fe0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:42:06.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.725+0000 7f3e8cff9640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:06.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.725+0000 7f3e8cff9640 1 -- 172.21.15.42:0/222951544 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f3e7c0050e0 con 0x7f3e7c010400 2024-09-16T03:42:06.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.725+0000 7f3e8cff9640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:06.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.725+0000 7f3e8cff9640 1 -- 172.21.15.42:0/222951544 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f3e7c005260 con 0x7f3e7c009fe0 2024-09-16T03:42:06.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.725+0000 7f3e8cff9640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:06.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.725+0000 7f3e8cff9640 1 -- 172.21.15.42:0/222951544 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f3e7c0053e0 con 0x7f3e7c00c370 2024-09-16T03:42:06.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.725+0000 7f3e8cff9640 10 monclient(hunting): _renew_subs 2024-09-16T03:42:06.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.725+0000 7f3e8ffff640 1 --2- 172.21.15.42:0/222951544 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3e7c00c370 0x7f3e7c008620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:06.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.725+0000 7f3e8ffff640 10 monclient(hunting): get_auth_request con 0x7f3e7c00c370 auth_method 0 2024-09-16T03:42:06.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.725+0000 7f3e8ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:06.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.725+0000 7f3e8ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:06.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.725+0000 7f3e8ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:42:06.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.725+0000 7f3e8f7fe640 1 --2- 172.21.15.42:0/222951544 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3e7c010400 0x7f3e7c010880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:06.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.725+0000 7f3e8f7fe640 10 monclient(hunting): get_auth_request con 0x7f3e7c010400 auth_method 0 2024-09-16T03:42:06.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.725+0000 7f3e8f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:06.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.725+0000 7f3e8f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:06.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.725+0000 7f3e8f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:42:06.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.725+0000 7f3e8ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:06.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.725+0000 7f3e8ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:06.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.725+0000 7f3e8ffff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T03:42:06.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.725+0000 7f3e8f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:06.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.725+0000 7f3e8f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:06.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.725+0000 7f3e8f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T03:42:06.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.725+0000 7f3e94cd2640 1 --2- 172.21.15.42:0/222951544 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3e7c009fe0 0x7f3e7c002fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:06.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.725+0000 7f3e94cd2640 10 monclient(hunting): get_auth_request con 0x7f3e7c009fe0 auth_method 0 2024-09-16T03:42:06.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.725+0000 7f3e94cd2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:06.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.725+0000 7f3e94cd2640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:06.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.725+0000 7f3e94cd2640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:42:06.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.725+0000 7f3e8ffff640 10 monclient(hunting): handle_auth_done global_id 4116 payload 293 2024-09-16T03:42:06.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.725+0000 7f3e8ffff640 1 -- 172.21.15.42:0/222951544 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3e7c009fe0 msgr2=0x7f3e7c002fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:06.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.725+0000 7f3e8ffff640 1 --2- 172.21.15.42:0/222951544 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3e7c009fe0 0x7f3e7c002fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:06.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.725+0000 7f3e8ffff640 1 -- 172.21.15.42:0/222951544 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3e7c010400 msgr2=0x7f3e7c010880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:06.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.725+0000 7f3e8ffff640 1 --2- 172.21.15.42:0/222951544 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3e7c010400 0x7f3e7c010880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:06.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.725+0000 7f3e8ffff640 10 monclient: _finish_hunting 0 2024-09-16T03:42:06.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.725+0000 7f3e8ffff640 1 monclient: found mon.c 2024-09-16T03:42:06.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.725+0000 7f3e96735640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:42:06.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.725+0000 7f3e8ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:42:06.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.725+0000 7f3e8ffff640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:42:06.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.725+0000 7f3e8ffff640 1 -- 172.21.15.42:0/222951544 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+,osdmap=0}) -- 0x7f3e7c005590 con 0x7f3e7c00c370 2024-09-16T03:42:06.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.725+0000 7f3e8ffff640 10 monclient: _finish_auth 0 2024-09-16T03:42:06.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.725+0000 7f3e8ffff640 10 monclient: _check_auth_tickets 2024-09-16T03:42:06.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.725+0000 7f3e8ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:42:06.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.725+0000 7f3e96735640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:42:06.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.725+0000 7f3e96735640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:42:06.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.725+0000 7f3e96735640 1 -- 172.21.15.42:0/222951544 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3e900a9640 con 0x7f3e7c00c370 2024-09-16T03:42:06.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.729+0000 7f3e8cff9640 10 client.4116.objecter ms_handle_connect 0x7f3e7c00c370 2024-09-16T03:42:06.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.729+0000 7f3e8cff9640 10 client.4116.objecter resend_mon_ops 2024-09-16T03:42:06.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.729+0000 7f3e8ffff640 1 --2- 172.21.15.42:0/222951544 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3e7c00c370 0x7f3e7c008620 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f3e80006bb0 tx=0x7f3e8400e3c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:06.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.729+0000 7f3e8cff9640 1 -- 172.21.15.42:0/222951544 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e8000f040 con 0x7f3e7c00c370 2024-09-16T03:42:06.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.729+0000 7f3e8cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:06.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.729+0000 7f3e8cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:42:06.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.729+0000 7f3e8cff9640 10 monclient: dump: 2024-09-16T03:42:06.752 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:06.752 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:06.752 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:06.752 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:06.752 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:06.752 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:06.752 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:06.752 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:06.752 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:06.753 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:06.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.729+0000 7f3e8cff9640 1 -- 172.21.15.42:0/222951544 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e8002a020 con 0x7f3e7c00c370 2024-09-16T03:42:06.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.729+0000 7f3e8cff9640 10 monclient: handle_config config(0 keys) 2024-09-16T03:42:06.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.729+0000 7f3e8cff9640 1 -- 172.21.15.42:0/222951544 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f3e80018d00 con 0x7f3e7c00c370 2024-09-16T03:42:06.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.729+0000 7f3e8cff9640 1 -- 172.21.15.42:0/222951544 <== mon.2 v2:172.21.15.42:3301/0 4 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f3e80024480 con 0x7f3e7c00c370 2024-09-16T03:42:06.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.729+0000 7f3e8dffb640 4 set_mon_vals no callback set 2024-09-16T03:42:06.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.729+0000 7f3e8cff9640 10 client.4116.objecter ms_dispatch 0x7f3e90000ba0 osd_map(1..1 src has 1..1) 2024-09-16T03:42:06.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.729+0000 7f3e8cff9640 3 client.4116.objecter handle_osd_map got epochs [1,1] > 0 2024-09-16T03:42:06.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.729+0000 7f3e8cff9640 3 client.4116.objecter handle_osd_map decoding full epoch 1 2024-09-16T03:42:06.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.729+0000 7f3e8cff9640 20 client.4116.objecter dump_active .. 0 homeless 2024-09-16T03:42:06.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.745+0000 7f3e8cff9640 1 -- 172.21.15.42:0/222951544 <== mon.2 v2:172.21.15.42:3301/0 5 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (secure 0 0 0) 0x7f3e80003620 con 0x7f3e7c00c370 2024-09-16T03:42:06.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.745+0000 7f3e8cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:42:06.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.745+0000 7f3e8cff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:42:06.828 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.821+0000 7f3e96735640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-09-16T03:42:06.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.821+0000 7f3e96735640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-09-16T03:42:06.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.821+0000 7f3e96735640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:42:06.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.821+0000 7f3e96735640 1 -- 172.21.15.42:0/222951544 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7f3e90137a20 con 0x7f3e7c00c370 2024-09-16T03:42:06.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.825+0000 7f3e8cff9640 1 -- 172.21.15.42:0/222951544 <== mon.2 v2:172.21.15.42:3301/0 6 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) ==== 124+0+0 (secure 0 0 0) 0x7f3e80003850 con 0x7f3e7c00c370 2024-09-16T03:42:06.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.825+0000 7f3e8cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-09-16T03:42:06.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.825+0000 7f3e8cff9640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-09-16T03:42:06.833 INFO:teuthology.orchestra.run.smithi042.stderr:adjusted tunables profile to default 2024-09-16T03:42:06.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.825+0000 7f3e96735640 10 librados: watch_flush enter 2024-09-16T03:42:06.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.825+0000 7f3e96735640 10 librados: watch_flush exit 2024-09-16T03:42:06.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.825+0000 7f3e96735640 20 client.4116.objecter shutdown clearing up homeless session... 2024-09-16T03:42:06.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.825+0000 7f3e96735640 10 client.4116.objecter successfully canceled tick 2024-09-16T03:42:06.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.825+0000 7f3e96735640 10 monclient: shutdown 2024-09-16T03:42:06.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.825+0000 7f3e96735640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:42:06.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.825+0000 7f3e96735640 1 -- 172.21.15.42:0/222951544 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3e7c00c370 msgr2=0x7f3e7c008620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:06.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.825+0000 7f3e96735640 1 --2- 172.21.15.42:0/222951544 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3e7c00c370 0x7f3e7c008620 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f3e80006bb0 tx=0x7f3e8400e3c0 comp rx=0 tx=0).stop 2024-09-16T03:42:06.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.829+0000 7f3e96735640 1 -- 172.21.15.42:0/222951544 shutdown_connections 2024-09-16T03:42:06.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.829+0000 7f3e96735640 1 --2- 172.21.15.42:0/222951544 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3e7c009fe0 0x7f3e7c002fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:06.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.829+0000 7f3e96735640 1 --2- 172.21.15.42:0/222951544 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3e7c010400 0x7f3e7c010880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:06.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.829+0000 7f3e96735640 1 --2- 172.21.15.42:0/222951544 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3e7c00c370 0x7f3e7c008620 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:06.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.829+0000 7f3e96735640 1 -- 172.21.15.42:0/222951544 >> 172.21.15.42:0/222951544 conn(0x7f3e900aa1b0 msgr2=0x7f3e90130db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:42:06.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.829+0000 7f3e96735640 1 -- 172.21.15.42:0/222951544 shutdown_connections 2024-09-16T03:42:06.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.829+0000 7f3e96735640 1 -- 172.21.15.42:0/222951544 wait complete. 2024-09-16T03:42:06.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:06.829+0000 7f3e96735640 1 librados: shutdown 2024-09-16T03:42:06.854 INFO:tasks.ceph:check_enable_crimson: False 2024-09-16T03:42:06.854 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-09-16T03:42:06.854 DEBUG:teuthology.orchestra.run.smithi042:> set -ex 2024-09-16T03:42:06.854 DEBUG:teuthology.orchestra.run.smithi042:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-09-16T03:42:06.865 DEBUG:teuthology.orchestra.run.smithi042:> set -ex 2024-09-16T03:42:06.865 DEBUG:teuthology.orchestra.run.smithi042:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-09-16T03:42:06.920 DEBUG:teuthology.orchestra.run.smithi042:> set -ex 2024-09-16T03:42:06.920 DEBUG:teuthology.orchestra.run.smithi042:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-09-16T03:42:06.952 INFO:tasks.ceph.mgr.x.smithi154.stderr:2024-09-16T03:42:06.950+0000 7f8fbbf11180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-16T03:42:06.973 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-09-16T03:42:06.973 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-09-16T03:42:06.986 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-09-16T03:42:06.986 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-09-16T03:42:07.038 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-09-16T03:42:07.038 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-09-16T03:42:07.093 DEBUG:teuthology.orchestra.run.smithi154:> sudo ceph --cluster ceph osd new 6b9fc058-2cff-46af-b95c-b78c72e1dc2b 0 2024-09-16T03:42:07.254 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.250+0000 7fb8c2730640 10 monclient: get_monmap_and_config 2024-09-16T03:42:07.254 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.250+0000 7fb8c2730640 10 monclient: build_initial_monmap 2024-09-16T03:42:07.254 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.250+0000 7fb8c2730640 10 monclient: monmap: 2024-09-16T03:42:07.255 INFO:teuthology.orchestra.run.smithi154.stderr:epoch 0 2024-09-16T03:42:07.255 INFO:teuthology.orchestra.run.smithi154.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:07.255 INFO:teuthology.orchestra.run.smithi154.stderr:last_changed 2024-09-16T03:42:07.256516+0000 2024-09-16T03:42:07.255 INFO:teuthology.orchestra.run.smithi154.stderr:created 2024-09-16T03:42:07.256516+0000 2024-09-16T03:42:07.255 INFO:teuthology.orchestra.run.smithi154.stderr:min_mon_release 0 (unknown) 2024-09-16T03:42:07.255 INFO:teuthology.orchestra.run.smithi154.stderr:election_strategy: 1 2024-09-16T03:42:07.255 INFO:teuthology.orchestra.run.smithi154.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:42:07.255 INFO:teuthology.orchestra.run.smithi154.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:42:07.255 INFO:teuthology.orchestra.run.smithi154.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:42:07.255 INFO:teuthology.orchestra.run.smithi154.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:42:07.255 INFO:teuthology.orchestra.run.smithi154.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:42:07.255 INFO:teuthology.orchestra.run.smithi154.stderr: 2024-09-16T03:42:07.258 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.254+0000 7fb8c2730640 1 Processor -- start 2024-09-16T03:42:07.258 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.254+0000 7fb8c2730640 1 -- start start 2024-09-16T03:42:07.258 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.254+0000 7fb8c2730640 10 monclient: init 2024-09-16T03:42:07.258 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.254+0000 7fb8c2730640 10 monclient: _reopen_session rank -1 2024-09-16T03:42:07.258 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.254+0000 7fb8c2730640 10 monclient: _add_conns ranks=[1,2,4,3,0] 2024-09-16T03:42:07.258 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.254+0000 7fb8c2730640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb8bc137680 addr v1:172.21.15.154:6789/0 2024-09-16T03:42:07.258 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.254+0000 7fb8c2730640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb8bc09ff60 0x7fb8bc13c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:07.258 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.254+0000 7fb8c2730640 10 monclient(hunting): picked mon.noname-b con 0x7fb8bc09ff60 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:42:07.258 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.254+0000 7fb8c2730640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fb8bc13c9f0 0x7fb8bc13ede0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:07.258 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.254+0000 7fb8c2730640 10 monclient(hunting): picked mon.noname-c con 0x7fb8bc13c9f0 addr v2:172.21.15.154:3300/0 2024-09-16T03:42:07.258 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.254+0000 7fb8c2730640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb8bc0a0f30 con 0x7fb8bc137680 2024-09-16T03:42:07.258 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.254+0000 7fb8c2730640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:07.259 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.254+0000 7fb8c2730640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fb8bc06c100 con 0x7fb8bc13c9f0 2024-09-16T03:42:07.259 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.254+0000 7fb8c2730640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:07.259 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.254+0000 7fb8c2730640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fb8bc06c4d0 con 0x7fb8bc09ff60 2024-09-16T03:42:07.259 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.254+0000 7fb8c2730640 10 monclient(hunting): _renew_subs 2024-09-16T03:42:07.259 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.254+0000 7fb8c2730640 10 monclient(hunting): authenticate will time out at 966.085254s 2024-09-16T03:42:07.259 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.254+0000 7fb8c0ccd640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fb8bc13c9f0 0x7fb8bc13ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:07.259 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.254+0000 7fb8bbfff640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7fb8bc137680 0x7fb8bc137a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.154:59294/0 (socket says 172.21.15.154:59294) 2024-09-16T03:42:07.259 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.254+0000 7fb8bbfff640 1 -- 172.21.15.154:0/270191236 learned_addr learned my addr 172.21.15.154:0/270191236 (peer_addr_for_me v1:172.21.15.154:0/0) 2024-09-16T03:42:07.259 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.254+0000 7fb8c0ccd640 10 monclient(hunting): get_auth_request con 0x7fb8bc13c9f0 auth_method 0 2024-09-16T03:42:07.259 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.254+0000 7fb8c0ccd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:07.259 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.254+0000 7fb8c0ccd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:07.259 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.254+0000 7fb8c0ccd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:07.259 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.254+0000 7fb8c0ccd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:07.259 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.254+0000 7fb8c0ccd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:07.260 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.254+0000 7fb8c0ccd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:07.260 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.254+0000 7fb8bb7fe640 1 --2- 172.21.15.154:0/270191236 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb8bc09ff60 0x7fb8bc13c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:07.260 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.254+0000 7fb8bb7fe640 10 monclient(hunting): get_auth_request con 0x7fb8bc09ff60 auth_method 0 2024-09-16T03:42:07.260 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.254+0000 7fb8bb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:07.260 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.254+0000 7fb8bb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:07.260 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.254+0000 7fb8bb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:07.260 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.254+0000 7fb8c0ccd640 10 monclient(hunting): handle_auth_done global_id 4111 payload 274 2024-09-16T03:42:07.260 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.254+0000 7fb8c0ccd640 1 -- 172.21.15.154:0/270191236 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb8bc09ff60 msgr2=0x7fb8bc13c4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:42:07.260 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.254+0000 7fb8c0ccd640 1 --2- 172.21.15.154:0/270191236 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb8bc09ff60 0x7fb8bc13c4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:07.260 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.254+0000 7fb8c0ccd640 1 -- 172.21.15.154:0/270191236 >> v1:172.21.15.154:6789/0 conn(0x7fb8bc137680 legacy=0x7fb8bc137a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:07.260 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.254+0000 7fb8c0ccd640 10 monclient: _finish_hunting 0 2024-09-16T03:42:07.260 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.254+0000 7fb8c0ccd640 1 monclient: found mon.noname-c 2024-09-16T03:42:07.261 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.254+0000 7fb8baffd640 1 -- 172.21.15.154:0/270191236 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3992325716 0 0) 0x7fb8bc0a0f30 con 0x7fb8bc137680 2024-09-16T03:42:07.261 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.254+0000 7fb8c0ccd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:42:07.261 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.254+0000 7fb8c0ccd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.154:3300/0 2024-09-16T03:42:07.261 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.254+0000 7fb8c0ccd640 1 -- 172.21.15.154:0/270191236 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8bc0a0890 con 0x7fb8bc13c9f0 2024-09-16T03:42:07.261 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.254+0000 7fb8baffd640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T03:42:07.266 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.254+0000 7fb8c0ccd640 1 --2- 172.21.15.154:0/270191236 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb8bc13c9f0 0x7fb8bc13ede0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fb8b4009d90 tx=0x7fb8b401c830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6542b095e5a53bd3 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:07.266 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.258+0000 7fb8baffd640 1 -- 172.21.15.154:0/270191236 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8b40a6700 con 0x7fb8bc13c9f0 2024-09-16T03:42:07.266 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.258+0000 7fb8baffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:07.266 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.258+0000 7fb8baffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:42:07.266 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.258+0000 7fb8baffd640 10 monclient: dump: 2024-09-16T03:42:07.266 INFO:teuthology.orchestra.run.smithi154.stderr:epoch 1 2024-09-16T03:42:07.266 INFO:teuthology.orchestra.run.smithi154.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:07.267 INFO:teuthology.orchestra.run.smithi154.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:07.267 INFO:teuthology.orchestra.run.smithi154.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:07.267 INFO:teuthology.orchestra.run.smithi154.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:07.267 INFO:teuthology.orchestra.run.smithi154.stderr:election_strategy: 1 2024-09-16T03:42:07.267 INFO:teuthology.orchestra.run.smithi154.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:07.267 INFO:teuthology.orchestra.run.smithi154.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:07.267 INFO:teuthology.orchestra.run.smithi154.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:07.267 INFO:teuthology.orchestra.run.smithi154.stderr: 2024-09-16T03:42:07.267 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.258+0000 7fb8baffd640 10 monclient: _finish_auth 0 2024-09-16T03:42:07.267 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.258+0000 7fb8baffd640 10 monclient: _check_auth_tickets 2024-09-16T03:42:07.267 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.258+0000 7fb8baffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:42:07.267 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.258+0000 7fb8baffd640 1 -- 172.21.15.154:0/270191236 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb8b40a68a0 con 0x7fb8bc13c9f0 2024-09-16T03:42:07.267 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.258+0000 7fb8c2730640 5 monclient: authenticate success, global_id 4111 2024-09-16T03:42:07.268 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.258+0000 7fb8c2730640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:42:07.268 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.258+0000 7fb8baffd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:42:07.268 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.258+0000 7fb8c2730640 10 monclient: get_monmap_and_config success 2024-09-16T03:42:07.268 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.258+0000 7fb8c2730640 4 set_mon_vals no callback set 2024-09-16T03:42:07.268 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.258+0000 7fb8baffd640 1 -- 172.21.15.154:0/270191236 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8b40a6c70 con 0x7fb8bc13c9f0 2024-09-16T03:42:07.268 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.258+0000 7fb8baffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:07.268 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.258+0000 7fb8baffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:42:07.268 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.258+0000 7fb8baffd640 10 monclient: dump: 2024-09-16T03:42:07.268 INFO:teuthology.orchestra.run.smithi154.stderr:epoch 1 2024-09-16T03:42:07.268 INFO:teuthology.orchestra.run.smithi154.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:07.268 INFO:teuthology.orchestra.run.smithi154.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:07.268 INFO:teuthology.orchestra.run.smithi154.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:07.268 INFO:teuthology.orchestra.run.smithi154.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:07.268 INFO:teuthology.orchestra.run.smithi154.stderr:election_strategy: 1 2024-09-16T03:42:07.269 INFO:teuthology.orchestra.run.smithi154.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:07.269 INFO:teuthology.orchestra.run.smithi154.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:07.269 INFO:teuthology.orchestra.run.smithi154.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:07.269 INFO:teuthology.orchestra.run.smithi154.stderr: 2024-09-16T03:42:07.269 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.258+0000 7fb8c2730640 10 monclient: shutdown 2024-09-16T03:42:07.269 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.258+0000 7fb8c2730640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:42:07.269 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.258+0000 7fb8c2730640 1 -- 172.21.15.154:0/270191236 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb8bc13c9f0 msgr2=0x7fb8bc13ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:07.269 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.258+0000 7fb8c2730640 1 --2- 172.21.15.154:0/270191236 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb8bc13c9f0 0x7fb8bc13ede0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fb8b4009d90 tx=0x7fb8b401c830 comp rx=0 tx=0).stop 2024-09-16T03:42:07.269 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.258+0000 7fb8c2730640 1 -- 172.21.15.154:0/270191236 shutdown_connections 2024-09-16T03:42:07.269 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.258+0000 7fb8c2730640 1 --2- 172.21.15.154:0/270191236 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb8bc13c9f0 0x7fb8bc13ede0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:07.269 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.258+0000 7fb8c2730640 1 --2- 172.21.15.154:0/270191236 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb8bc09ff60 0x7fb8bc13c4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:07.269 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.258+0000 7fb8c2730640 1 -- 172.21.15.154:0/270191236 >> 172.21.15.154:0/270191236 conn(0x7fb8bc0a6ec0 msgr2=0x7fb8bc0a33c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:42:07.269 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.258+0000 7fb8c2730640 1 -- 172.21.15.154:0/270191236 shutdown_connections 2024-09-16T03:42:07.269 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.258+0000 7fb8c2730640 1 -- 172.21.15.154:0/270191236 wait complete. 2024-09-16T03:42:07.270 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.258+0000 7fb8c2730640 10 monclient: build_initial_monmap 2024-09-16T03:42:07.270 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.258+0000 7fb8c2730640 10 monclient: monmap: 2024-09-16T03:42:07.270 INFO:teuthology.orchestra.run.smithi154.stderr:epoch 0 2024-09-16T03:42:07.270 INFO:teuthology.orchestra.run.smithi154.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:42:07.270 INFO:teuthology.orchestra.run.smithi154.stderr:last_changed 0.000000 2024-09-16T03:42:07.270 INFO:teuthology.orchestra.run.smithi154.stderr:created 0.000000 2024-09-16T03:42:07.270 INFO:teuthology.orchestra.run.smithi154.stderr:min_mon_release 0 (unknown) 2024-09-16T03:42:07.270 INFO:teuthology.orchestra.run.smithi154.stderr:election_strategy: 1 2024-09-16T03:42:07.270 INFO:teuthology.orchestra.run.smithi154.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:42:07.270 INFO:teuthology.orchestra.run.smithi154.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:42:07.270 INFO:teuthology.orchestra.run.smithi154.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:42:07.270 INFO:teuthology.orchestra.run.smithi154.stderr: 2024-09-16T03:42:07.270 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.258+0000 7fb8c2730640 1 librados: starting msgr at 2024-09-16T03:42:07.270 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.258+0000 7fb8c2730640 1 librados: starting objecter 2024-09-16T03:42:07.271 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.258+0000 7fb8c2730640 1 Processor -- start 2024-09-16T03:42:07.271 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.258+0000 7fb8c2730640 1 -- start start 2024-09-16T03:42:07.271 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.258+0000 7fb8c2730640 1 librados: setting wanted keys 2024-09-16T03:42:07.271 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.258+0000 7fb8c2730640 1 librados: calling monclient init 2024-09-16T03:42:07.271 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.258+0000 7fb8c2730640 10 monclient: init 2024-09-16T03:42:07.271 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.258+0000 7fb8c2730640 10 monclient: _reopen_session rank -1 2024-09-16T03:42:07.271 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.258+0000 7fb8c2730640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:42:07.271 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.258+0000 7fb8c2730640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb8bc09ff60 0x7fb8bc135850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:07.271 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.258+0000 7fb8c2730640 10 monclient(hunting): picked mon.noname-a con 0x7fb8bc09ff60 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:42:07.271 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.258+0000 7fb8c2730640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb8bc137680 0x7fb8bc135d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:07.271 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.258+0000 7fb8c2730640 10 monclient(hunting): picked mon.noname-c con 0x7fb8bc137680 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:42:07.271 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.258+0000 7fb8c2730640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb8bc13c9f0 0x7fb8bc11cf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:07.271 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.258+0000 7fb8c2730640 10 monclient(hunting): picked mon.noname-b con 0x7fb8bc13c9f0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:42:07.272 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.258+0000 7fb8c2730640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:07.272 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.258+0000 7fb8c2730640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fb8bc141cc0 con 0x7fb8bc09ff60 2024-09-16T03:42:07.272 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.258+0000 7fb8c2730640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:07.272 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.258+0000 7fb8c2730640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fb8bc141b40 con 0x7fb8bc13c9f0 2024-09-16T03:42:07.272 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.258+0000 7fb8c2730640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:07.272 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.258+0000 7fb8c2730640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fb8bc141e40 con 0x7fb8bc137680 2024-09-16T03:42:07.272 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.258+0000 7fb8c2730640 10 monclient(hunting): _renew_subs 2024-09-16T03:42:07.272 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.258+0000 7fb8c2730640 10 monclient(hunting): authenticate will time out at 966.090326s 2024-09-16T03:42:07.272 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.258+0000 7fb8c0ccd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb8bc13c9f0 0x7fb8bc11cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:07.272 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.258+0000 7fb8c0ccd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb8bc13c9f0 0x7fb8bc11cf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.154:50128/0 (socket says 172.21.15.154:50128) 2024-09-16T03:42:07.272 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.258+0000 7fb8c0ccd640 1 -- 172.21.15.154:0/330771869 learned_addr learned my addr 172.21.15.154:0/330771869 (peer_addr_for_me v2:172.21.15.154:0/0) 2024-09-16T03:42:07.273 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.258+0000 7fb8c0ccd640 10 monclient(hunting): get_auth_request con 0x7fb8bc13c9f0 auth_method 0 2024-09-16T03:42:07.273 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.258+0000 7fb8c0ccd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:07.273 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.258+0000 7fb8c0ccd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:07.273 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.258+0000 7fb8c0ccd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:07.273 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.258+0000 7fb8bb7fe640 1 --2- 172.21.15.154:0/330771869 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb8bc137680 0x7fb8bc135d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:07.273 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.258+0000 7fb8c0ccd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:07.273 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.258+0000 7fb8c0ccd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:07.273 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.258+0000 7fb8c0ccd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:07.273 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.258+0000 7fb8bb7fe640 10 monclient(hunting): get_auth_request con 0x7fb8bc137680 auth_method 0 2024-09-16T03:42:07.273 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.258+0000 7fb8bb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:07.274 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.258+0000 7fb8bb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:07.274 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.258+0000 7fb8bb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:07.274 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.258+0000 7fb8bb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:07.274 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.258+0000 7fb8bb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:07.274 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.258+0000 7fb8bb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:07.274 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.258+0000 7fb8bbfff640 1 --2- 172.21.15.154:0/330771869 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb8bc09ff60 0x7fb8bc135850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:07.274 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.258+0000 7fb8bbfff640 10 monclient(hunting): get_auth_request con 0x7fb8bc09ff60 auth_method 0 2024-09-16T03:42:07.274 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.258+0000 7fb8bbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:07.274 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.258+0000 7fb8bbfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:07.274 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.258+0000 7fb8bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:07.274 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.258+0000 7fb8c0ccd640 10 monclient(hunting): handle_auth_done global_id 4117 payload 824 2024-09-16T03:42:07.274 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.258+0000 7fb8c0ccd640 1 -- 172.21.15.154:0/330771869 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb8bc137680 msgr2=0x7fb8bc135d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:07.274 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.258+0000 7fb8c0ccd640 1 --2- 172.21.15.154:0/330771869 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb8bc137680 0x7fb8bc135d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:07.274 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.258+0000 7fb8c0ccd640 1 -- 172.21.15.154:0/330771869 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb8bc09ff60 msgr2=0x7fb8bc135850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:07.275 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.258+0000 7fb8c0ccd640 1 --2- 172.21.15.154:0/330771869 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb8bc09ff60 0x7fb8bc135850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:07.275 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.258+0000 7fb8c0ccd640 10 monclient: _finish_hunting 0 2024-09-16T03:42:07.275 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.258+0000 7fb8c0ccd640 1 monclient: found mon.noname-b 2024-09-16T03:42:07.275 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.258+0000 7fb8c0ccd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:42:07.275 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.258+0000 7fb8c0ccd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.154:3300/0 2024-09-16T03:42:07.275 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.258+0000 7fb8c0ccd640 1 -- 172.21.15.154:0/330771869 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8bc1202f0 con 0x7fb8bc13c9f0 2024-09-16T03:42:07.275 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.258+0000 7fb8c0ccd640 1 --2- 172.21.15.154:0/330771869 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb8bc13c9f0 0x7fb8bc11cf10 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fb8b4009d90 tx=0x7fb8b40195d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:07.275 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.258+0000 7fb8b8ff9640 10 client.?.objecter ms_handle_connect 0x7fb8bc13c9f0 2024-09-16T03:42:07.275 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.258+0000 7fb8b8ff9640 10 client.?.objecter resend_mon_ops 2024-09-16T03:42:07.276 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.258+0000 7fb8b8ff9640 1 -- 172.21.15.154:0/330771869 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8b40be070 con 0x7fb8bc13c9f0 2024-09-16T03:42:07.276 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.258+0000 7fb8b8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:07.276 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.258+0000 7fb8b8ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:42:07.276 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.258+0000 7fb8b8ff9640 10 monclient: dump: 2024-09-16T03:42:07.276 INFO:teuthology.orchestra.run.smithi154.stderr:epoch 1 2024-09-16T03:42:07.276 INFO:teuthology.orchestra.run.smithi154.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:07.276 INFO:teuthology.orchestra.run.smithi154.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:07.276 INFO:teuthology.orchestra.run.smithi154.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:07.276 INFO:teuthology.orchestra.run.smithi154.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:07.276 INFO:teuthology.orchestra.run.smithi154.stderr:election_strategy: 1 2024-09-16T03:42:07.276 INFO:teuthology.orchestra.run.smithi154.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:07.276 INFO:teuthology.orchestra.run.smithi154.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:07.276 INFO:teuthology.orchestra.run.smithi154.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:07.277 INFO:teuthology.orchestra.run.smithi154.stderr: 2024-09-16T03:42:07.277 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.258+0000 7fb8b8ff9640 10 monclient: _finish_auth 0 2024-09-16T03:42:07.277 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.258+0000 7fb8b8ff9640 10 monclient: _check_auth_tickets 2024-09-16T03:42:07.277 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.258+0000 7fb8b8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:42:07.277 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.258+0000 7fb8c2730640 5 monclient: authenticate success, global_id 4117 2024-09-16T03:42:07.277 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.258+0000 7fb8b8ff9640 1 -- 172.21.15.154:0/330771869 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb8b40b8840 con 0x7fb8bc13c9f0 2024-09-16T03:42:07.277 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.258+0000 7fb8c2730640 10 monclient: _renew_subs 2024-09-16T03:42:07.277 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.258+0000 7fb8c2730640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2024-09-16T03:42:07.277 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.258+0000 7fb8c2730640 1 -- 172.21.15.154:0/330771869 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8bc1205d0 con 0x7fb8bc13c9f0 2024-09-16T03:42:07.277 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.258+0000 7fb8b8ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T03:42:07.278 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.258+0000 7fb8c2730640 10 client.4117.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:42:07.278 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.258+0000 7fb8b8ff9640 1 -- 172.21.15.154:0/330771869 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8b40b8c30 con 0x7fb8bc13c9f0 2024-09-16T03:42:07.278 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.258+0000 7fb8c2730640 10 monclient: _renew_subs 2024-09-16T03:42:07.278 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.258+0000 7fb8c2730640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2024-09-16T03:42:07.278 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.258+0000 7fb8ba7fc640 4 set_mon_vals no callback set 2024-09-16T03:42:07.278 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.258+0000 7fb8c2730640 1 -- 172.21.15.154:0/330771869 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8bc0a2cb0 con 0x7fb8bc13c9f0 2024-09-16T03:42:07.278 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.258+0000 7fb8c2730640 1 librados: init done 2024-09-16T03:42:07.278 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.258+0000 7fb8b8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:07.278 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.258+0000 7fb8b8ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:42:07.278 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.258+0000 7fb8b8ff9640 10 monclient: dump: 2024-09-16T03:42:07.278 INFO:teuthology.orchestra.run.smithi154.stderr:epoch 1 2024-09-16T03:42:07.278 INFO:teuthology.orchestra.run.smithi154.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:07.278 INFO:teuthology.orchestra.run.smithi154.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:07.279 INFO:teuthology.orchestra.run.smithi154.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:07.279 INFO:teuthology.orchestra.run.smithi154.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:07.279 INFO:teuthology.orchestra.run.smithi154.stderr:election_strategy: 1 2024-09-16T03:42:07.279 INFO:teuthology.orchestra.run.smithi154.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:07.279 INFO:teuthology.orchestra.run.smithi154.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:07.279 INFO:teuthology.orchestra.run.smithi154.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:07.279 INFO:teuthology.orchestra.run.smithi154.stderr: 2024-09-16T03:42:07.279 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.258+0000 7fb8b8ff9640 1 -- 172.21.15.154:0/330771869 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fb8b40c1070 con 0x7fb8bc13c9f0 2024-09-16T03:42:07.280 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.258+0000 7fb8b8ff9640 1 -- 172.21.15.154:0/330771869 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 725+0+0 (secure 0 0 0) 0x7fb8b40ae420 con 0x7fb8bc13c9f0 2024-09-16T03:42:07.280 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.258+0000 7fb8b8ff9640 10 client.4117.objecter ms_dispatch 0x7fb8bc000ba0 osd_map(2..2 src has 1..2) 2024-09-16T03:42:07.280 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.258+0000 7fb8b8ff9640 3 client.4117.objecter handle_osd_map got epochs [2,2] > 0 2024-09-16T03:42:07.280 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.258+0000 7fb8b8ff9640 3 client.4117.objecter handle_osd_map decoding full epoch 2 2024-09-16T03:42:07.280 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.258+0000 7fb8b8ff9640 20 client.4117.objecter dump_active .. 0 homeless 2024-09-16T03:42:07.280 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.262+0000 7fb8c2730640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:42:07.280 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.262+0000 7fb8c2730640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:42:07.280 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.262+0000 7fb8c2730640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2024-09-16T03:42:07.280 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.262+0000 7fb8c2730640 1 -- 172.21.15.154:0/330771869 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb8bc137a60 con 0x7fb8bc13c9f0 2024-09-16T03:42:07.281 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.278+0000 7fb8b8ff9640 1 -- 172.21.15.154:0/330771869 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (secure 0 0 0) 0x7fb8b4016540 con 0x7fb8bc13c9f0 2024-09-16T03:42:07.281 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.278+0000 7fb8b8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:42:07.281 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.278+0000 7fb8b8ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:42:07.349 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.346+0000 7fb8c2730640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "6b9fc058-2cff-46af-b95c-b78c72e1dc2b", "id": 0}] 2024-09-16T03:42:07.349 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.346+0000 7fb8c2730640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "6b9fc058-2cff-46af-b95c-b78c72e1dc2b", "id": 0}] 2024-09-16T03:42:07.350 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.346+0000 7fb8c2730640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2024-09-16T03:42:07.350 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.346+0000 7fb8c2730640 1 -- 172.21.15.154:0/330771869 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "osd new", "uuid": "6b9fc058-2cff-46af-b95c-b78c72e1dc2b", "id": 0} v 0) -- 0x7fb8bc120010 con 0x7fb8bc13c9f0 2024-09-16T03:42:07.356 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.354+0000 7fb8b8ff9640 1 -- 172.21.15.154:0/330771869 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "6b9fc058-2cff-46af-b95c-b78c72e1dc2b", "id": 0}]=0 v3) ==== 112+0+2 (secure 0 0 0) 0x7fb8b40b8dd0 con 0x7fb8bc13c9f0 2024-09-16T03:42:07.356 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.354+0000 7fb8b8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "6b9fc058-2cff-46af-b95c-b78c72e1dc2b", "id": 0}] 2024-09-16T03:42:07.356 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.354+0000 7fb8b8ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:42:07.357 INFO:teuthology.orchestra.run.smithi154.stdout:0 2024-09-16T03:42:07.358 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.354+0000 7fb8c2730640 10 librados: watch_flush enter 2024-09-16T03:42:07.358 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.354+0000 7fb8c2730640 10 librados: watch_flush exit 2024-09-16T03:42:07.358 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.354+0000 7fb8c2730640 20 client.4117.objecter shutdown clearing up homeless session... 2024-09-16T03:42:07.359 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.354+0000 7fb8c2730640 10 client.4117.objecter successfully canceled tick 2024-09-16T03:42:07.359 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.354+0000 7fb8c2730640 10 monclient: shutdown 2024-09-16T03:42:07.359 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.354+0000 7fb8c2730640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:42:07.359 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.354+0000 7fb8c2730640 1 -- 172.21.15.154:0/330771869 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb8bc13c9f0 msgr2=0x7fb8bc11cf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:07.359 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.354+0000 7fb8c2730640 1 --2- 172.21.15.154:0/330771869 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb8bc13c9f0 0x7fb8bc11cf10 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fb8b4009d90 tx=0x7fb8b40195d0 comp rx=0 tx=0).stop 2024-09-16T03:42:07.359 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.354+0000 7fb8c2730640 1 -- 172.21.15.154:0/330771869 shutdown_connections 2024-09-16T03:42:07.359 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.354+0000 7fb8c2730640 1 --2- 172.21.15.154:0/330771869 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb8bc13c9f0 0x7fb8bc11cf10 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:07.359 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.354+0000 7fb8c2730640 1 --2- 172.21.15.154:0/330771869 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb8bc137680 0x7fb8bc135d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:07.359 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.354+0000 7fb8c2730640 1 --2- 172.21.15.154:0/330771869 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb8bc09ff60 0x7fb8bc135850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:07.359 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.354+0000 7fb8c2730640 1 -- 172.21.15.154:0/330771869 >> 172.21.15.154:0/330771869 conn(0x7fb8bc0a6ec0 msgr2=0x7fb8bc13e210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:42:07.359 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.354+0000 7fb8c2730640 1 -- 172.21.15.154:0/330771869 shutdown_connections 2024-09-16T03:42:07.359 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.354+0000 7fb8c2730640 1 -- 172.21.15.154:0/330771869 wait complete. 2024-09-16T03:42:07.359 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.354+0000 7fb8c2730640 1 librados: shutdown 2024-09-16T03:42:07.373 DEBUG:teuthology.orchestra.run.smithi154:> sudo ceph --cluster ceph osd new ee31f807-cb16-4c33-8e23-2573e1a6b5c9 1 2024-09-16T03:42:07.447 INFO:tasks.ceph.mgr.x.smithi154.stderr:2024-09-16T03:42:07.446+0000 7f8fbbf11180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-16T03:42:07.485 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.482+0000 7f5abf684640 10 monclient: get_monmap_and_config 2024-09-16T03:42:07.485 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.482+0000 7f5abf684640 10 monclient: build_initial_monmap 2024-09-16T03:42:07.485 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.482+0000 7f5abf684640 10 monclient: monmap: 2024-09-16T03:42:07.486 INFO:teuthology.orchestra.run.smithi154.stderr:epoch 0 2024-09-16T03:42:07.486 INFO:teuthology.orchestra.run.smithi154.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:07.486 INFO:teuthology.orchestra.run.smithi154.stderr:last_changed 2024-09-16T03:42:07.486681+0000 2024-09-16T03:42:07.486 INFO:teuthology.orchestra.run.smithi154.stderr:created 2024-09-16T03:42:07.486681+0000 2024-09-16T03:42:07.486 INFO:teuthology.orchestra.run.smithi154.stderr:min_mon_release 0 (unknown) 2024-09-16T03:42:07.486 INFO:teuthology.orchestra.run.smithi154.stderr:election_strategy: 1 2024-09-16T03:42:07.486 INFO:teuthology.orchestra.run.smithi154.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:42:07.486 INFO:teuthology.orchestra.run.smithi154.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:42:07.487 INFO:teuthology.orchestra.run.smithi154.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:42:07.487 INFO:teuthology.orchestra.run.smithi154.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:42:07.487 INFO:teuthology.orchestra.run.smithi154.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:42:07.487 INFO:teuthology.orchestra.run.smithi154.stderr: 2024-09-16T03:42:07.487 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.482+0000 7f5abf684640 1 Processor -- start 2024-09-16T03:42:07.487 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.482+0000 7f5abf684640 1 -- start start 2024-09-16T03:42:07.487 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.482+0000 7f5abf684640 10 monclient: init 2024-09-16T03:42:07.494 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.482+0000 7f5abf684640 10 monclient: _reopen_session rank -1 2024-09-16T03:42:07.494 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.482+0000 7f5abf684640 10 monclient: _add_conns ranks=[4,0,3,1,2] 2024-09-16T03:42:07.494 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.482+0000 7f5abf684640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f5ab806e7f0 0x7f5ab80a4a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:07.494 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.482+0000 7f5abf684640 10 monclient(hunting): picked mon.noname-c con 0x7f5ab806e7f0 addr v2:172.21.15.154:3300/0 2024-09-16T03:42:07.494 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.482+0000 7f5abf684640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5ab80a5a00 addr v1:172.21.15.42:6789/0 2024-09-16T03:42:07.495 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.482+0000 7f5abf684640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f5ab80a4fd0 0x7f5ab8137690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:07.495 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.482+0000 7f5abf684640 10 monclient(hunting): picked mon.noname-a con 0x7f5ab80a4fd0 addr v2:172.21.15.42:3300/0 2024-09-16T03:42:07.495 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.482+0000 7f5abf684640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5ab80a40d0 con 0x7f5ab80a5a00 2024-09-16T03:42:07.495 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.482+0000 7f5abf684640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:07.495 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.482+0000 7f5abf684640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f5ab806ebd0 con 0x7f5ab80a4fd0 2024-09-16T03:42:07.495 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.482+0000 7f5abf684640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:07.495 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.482+0000 7f5abf684640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f5ab809be60 con 0x7f5ab806e7f0 2024-09-16T03:42:07.495 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.482+0000 7f5abf684640 10 monclient(hunting): _renew_subs 2024-09-16T03:42:07.495 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.482+0000 7f5abf684640 10 monclient(hunting): authenticate will time out at 966.315497s 2024-09-16T03:42:07.495 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.482+0000 7f5abd420640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f5ab806e7f0 0x7f5ab80a4a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:07.495 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.482+0000 7f5abd420640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f5ab806e7f0 0x7f5ab80a4a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.154:50142/0 (socket says 172.21.15.154:50142) 2024-09-16T03:42:07.495 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.482+0000 7f5abd420640 1 -- 172.21.15.154:0/312486310 learned_addr learned my addr 172.21.15.154:0/312486310 (peer_addr_for_me v2:172.21.15.154:0/0) 2024-09-16T03:42:07.495 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.482+0000 7f5abd420640 10 monclient(hunting): get_auth_request con 0x7f5ab806e7f0 auth_method 0 2024-09-16T03:42:07.495 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.482+0000 7f5abd420640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:07.496 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.482+0000 7f5abd420640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:07.496 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.482+0000 7f5abd420640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:07.496 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.482+0000 7f5abd420640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:07.496 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.482+0000 7f5abd420640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:07.496 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.482+0000 7f5abd420640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:07.496 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.482+0000 7f5abd420640 10 monclient(hunting): handle_auth_done global_id 4120 payload 274 2024-09-16T03:42:07.496 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.482+0000 7f5abdc21640 1 --2- 172.21.15.154:0/312486310 >> v2:172.21.15.42:3300/0 conn(0x7f5ab80a4fd0 0x7f5ab8137690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:07.496 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.482+0000 7f5abd420640 1 -- 172.21.15.154:0/312486310 >> v2:172.21.15.42:3300/0 conn(0x7f5ab80a4fd0 msgr2=0x7f5ab8137690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:42:07.496 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.482+0000 7f5abd420640 1 --2- 172.21.15.154:0/312486310 >> v2:172.21.15.42:3300/0 conn(0x7f5ab80a4fd0 0x7f5ab8137690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:07.496 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.486+0000 7f5abd420640 1 -- 172.21.15.154:0/312486310 >> v1:172.21.15.42:6789/0 conn(0x7f5ab80a5a00 legacy=0x7f5ab8136f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:42:07.496 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.486+0000 7f5abd420640 10 monclient: _finish_hunting 0 2024-09-16T03:42:07.496 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.486+0000 7f5abd420640 1 monclient: found mon.noname-c 2024-09-16T03:42:07.496 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.486+0000 7f5abd420640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:42:07.497 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.486+0000 7f5abd420640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.154:3300/0 2024-09-16T03:42:07.497 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.486+0000 7f5abd420640 1 -- 172.21.15.154:0/312486310 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ab809c140 con 0x7f5ab806e7f0 2024-09-16T03:42:07.497 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.486+0000 7f5abd420640 1 --2- 172.21.15.154:0/312486310 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5ab806e7f0 0x7f5ab80a4a90 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f5ab80745d0 tx=0x7f5ab401c370 comp rx=0 tx=0).ready entity=mon.1 client_cookie=292ac24ac7c8f68f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:07.497 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.486+0000 7f5aaffff640 1 -- 172.21.15.154:0/312486310 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ab40b2070 con 0x7f5ab806e7f0 2024-09-16T03:42:07.497 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.486+0000 7f5aaffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:07.497 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.486+0000 7f5aaffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:42:07.497 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.486+0000 7f5aaffff640 10 monclient: dump: 2024-09-16T03:42:07.497 INFO:teuthology.orchestra.run.smithi154.stderr:epoch 1 2024-09-16T03:42:07.497 INFO:teuthology.orchestra.run.smithi154.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:07.497 INFO:teuthology.orchestra.run.smithi154.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:07.497 INFO:teuthology.orchestra.run.smithi154.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:07.498 INFO:teuthology.orchestra.run.smithi154.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:07.498 INFO:teuthology.orchestra.run.smithi154.stderr:election_strategy: 1 2024-09-16T03:42:07.498 INFO:teuthology.orchestra.run.smithi154.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:07.498 INFO:teuthology.orchestra.run.smithi154.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:07.498 INFO:teuthology.orchestra.run.smithi154.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:07.498 INFO:teuthology.orchestra.run.smithi154.stderr: 2024-09-16T03:42:07.498 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.486+0000 7f5aaffff640 10 monclient: _finish_auth 0 2024-09-16T03:42:07.498 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.486+0000 7f5aaffff640 10 monclient: _check_auth_tickets 2024-09-16T03:42:07.499 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.486+0000 7f5aaffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:42:07.499 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.486+0000 7f5aaffff640 1 -- 172.21.15.154:0/312486310 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ab40ad640 con 0x7f5ab806e7f0 2024-09-16T03:42:07.499 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.486+0000 7f5aaffff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:42:07.499 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.486+0000 7f5aaffff640 1 -- 172.21.15.154:0/312486310 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ab40ada10 con 0x7f5ab806e7f0 2024-09-16T03:42:07.499 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.486+0000 7f5abf684640 5 monclient: authenticate success, global_id 4120 2024-09-16T03:42:07.499 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.486+0000 7f5abf684640 10 monclient: get_monmap_and_config success 2024-09-16T03:42:07.499 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.486+0000 7f5abf684640 4 set_mon_vals no callback set 2024-09-16T03:42:07.499 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.486+0000 7f5aaffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:07.499 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.486+0000 7f5aaffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:42:07.499 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.486+0000 7f5aaffff640 10 monclient: dump: 2024-09-16T03:42:07.499 INFO:teuthology.orchestra.run.smithi154.stderr:epoch 1 2024-09-16T03:42:07.499 INFO:teuthology.orchestra.run.smithi154.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:07.499 INFO:teuthology.orchestra.run.smithi154.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:07.499 INFO:teuthology.orchestra.run.smithi154.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:07.500 INFO:teuthology.orchestra.run.smithi154.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:07.500 INFO:teuthology.orchestra.run.smithi154.stderr:election_strategy: 1 2024-09-16T03:42:07.500 INFO:teuthology.orchestra.run.smithi154.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:07.500 INFO:teuthology.orchestra.run.smithi154.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:07.500 INFO:teuthology.orchestra.run.smithi154.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:07.500 INFO:teuthology.orchestra.run.smithi154.stderr: 2024-09-16T03:42:07.500 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.486+0000 7f5abf684640 10 monclient: shutdown 2024-09-16T03:42:07.500 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.486+0000 7f5abf684640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:42:07.500 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.486+0000 7f5abf684640 1 -- 172.21.15.154:0/312486310 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5ab806e7f0 msgr2=0x7f5ab80a4a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:07.501 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.486+0000 7f5abf684640 1 --2- 172.21.15.154:0/312486310 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5ab806e7f0 0x7f5ab80a4a90 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f5ab80745d0 tx=0x7f5ab401c370 comp rx=0 tx=0).stop 2024-09-16T03:42:07.501 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.486+0000 7f5abf684640 1 -- 172.21.15.154:0/312486310 shutdown_connections 2024-09-16T03:42:07.501 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.486+0000 7f5abf684640 1 --2- 172.21.15.154:0/312486310 >> v2:172.21.15.42:3300/0 conn(0x7f5ab80a4fd0 0x7f5ab8137690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:07.501 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.486+0000 7f5abf684640 1 --2- 172.21.15.154:0/312486310 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5ab806e7f0 0x7f5ab80a4a90 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:07.501 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.486+0000 7f5abf684640 1 -- 172.21.15.154:0/312486310 >> 172.21.15.154:0/312486310 conn(0x7f5ab80aa1b0 msgr2=0x7f5ab80a66b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:42:07.501 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.486+0000 7f5abf684640 1 -- 172.21.15.154:0/312486310 shutdown_connections 2024-09-16T03:42:07.501 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.486+0000 7f5abf684640 1 -- 172.21.15.154:0/312486310 wait complete. 2024-09-16T03:42:07.501 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.486+0000 7f5abf684640 10 monclient: build_initial_monmap 2024-09-16T03:42:07.501 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.486+0000 7f5abf684640 10 monclient: monmap: 2024-09-16T03:42:07.501 INFO:teuthology.orchestra.run.smithi154.stderr:epoch 0 2024-09-16T03:42:07.501 INFO:teuthology.orchestra.run.smithi154.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:42:07.501 INFO:teuthology.orchestra.run.smithi154.stderr:last_changed 0.000000 2024-09-16T03:42:07.501 INFO:teuthology.orchestra.run.smithi154.stderr:created 0.000000 2024-09-16T03:42:07.502 INFO:teuthology.orchestra.run.smithi154.stderr:min_mon_release 0 (unknown) 2024-09-16T03:42:07.502 INFO:teuthology.orchestra.run.smithi154.stderr:election_strategy: 1 2024-09-16T03:42:07.502 INFO:teuthology.orchestra.run.smithi154.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:42:07.502 INFO:teuthology.orchestra.run.smithi154.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:42:07.502 INFO:teuthology.orchestra.run.smithi154.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:42:07.502 INFO:teuthology.orchestra.run.smithi154.stderr: 2024-09-16T03:42:07.502 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.490+0000 7f5abf684640 1 librados: starting msgr at 2024-09-16T03:42:07.502 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.490+0000 7f5abf684640 1 librados: starting objecter 2024-09-16T03:42:07.502 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.490+0000 7f5abf684640 1 Processor -- start 2024-09-16T03:42:07.502 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.490+0000 7f5abf684640 1 -- start start 2024-09-16T03:42:07.502 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.490+0000 7f5abf684640 1 librados: setting wanted keys 2024-09-16T03:42:07.502 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.490+0000 7f5abf684640 1 librados: calling monclient init 2024-09-16T03:42:07.502 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.490+0000 7f5abf684640 10 monclient: init 2024-09-16T03:42:07.502 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.490+0000 7f5abf684640 10 monclient: _reopen_session rank -1 2024-09-16T03:42:07.503 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.490+0000 7f5abf684640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T03:42:07.503 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.490+0000 7f5abf684640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5ab806e7f0 0x7f5ab812d820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:07.503 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.490+0000 7f5abf684640 10 monclient(hunting): picked mon.noname-a con 0x7f5ab806e7f0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:42:07.503 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.490+0000 7f5abf684640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5ab80a4fd0 0x7f5ab812dd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:07.503 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.490+0000 7f5abf684640 10 monclient(hunting): picked mon.noname-b con 0x7f5ab80a4fd0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:42:07.503 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.490+0000 7f5abf684640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5ab80a5a00 0x7f5ab81cedf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:07.503 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.490+0000 7f5abf684640 10 monclient(hunting): picked mon.noname-c con 0x7f5ab80a5a00 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:42:07.503 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.490+0000 7f5abf684640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:07.503 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.490+0000 7f5abf684640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f5ab8137d50 con 0x7f5ab806e7f0 2024-09-16T03:42:07.503 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.490+0000 7f5abf684640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:07.503 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.490+0000 7f5abf684640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f5ab8137bd0 con 0x7f5ab80a4fd0 2024-09-16T03:42:07.503 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.490+0000 7f5abf684640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:07.503 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.490+0000 7f5abf684640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f5ab8141b40 con 0x7f5ab80a5a00 2024-09-16T03:42:07.503 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.490+0000 7f5abf684640 10 monclient(hunting): _renew_subs 2024-09-16T03:42:07.504 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.490+0000 7f5abf684640 10 monclient(hunting): authenticate will time out at 966.320826s 2024-09-16T03:42:07.504 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.490+0000 7f5abcc1f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5ab80a4fd0 0x7f5ab812dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:07.504 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.490+0000 7f5abcc1f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5ab80a4fd0 0x7f5ab812dd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.154:50144/0 (socket says 172.21.15.154:50144) 2024-09-16T03:42:07.504 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.490+0000 7f5abcc1f640 1 -- 172.21.15.154:0/3087816044 learned_addr learned my addr 172.21.15.154:0/3087816044 (peer_addr_for_me v2:172.21.15.154:0/0) 2024-09-16T03:42:07.504 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.490+0000 7f5abcc1f640 10 monclient(hunting): get_auth_request con 0x7f5ab80a4fd0 auth_method 0 2024-09-16T03:42:07.504 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.490+0000 7f5abcc1f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:07.504 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.490+0000 7f5abcc1f640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:07.504 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.490+0000 7f5abcc1f640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:07.504 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.490+0000 7f5abcc1f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:07.504 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.490+0000 7f5abcc1f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:07.504 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.490+0000 7f5abcc1f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:07.504 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.490+0000 7f5abcc1f640 10 monclient(hunting): handle_auth_done global_id 4123 payload 824 2024-09-16T03:42:07.504 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.490+0000 7f5abcc1f640 1 -- 172.21.15.154:0/3087816044 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5ab80a5a00 msgr2=0x7f5ab81cedf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:07.505 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.490+0000 7f5abcc1f640 1 --2- 172.21.15.154:0/3087816044 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5ab80a5a00 0x7f5ab81cedf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:07.505 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.490+0000 7f5abcc1f640 1 -- 172.21.15.154:0/3087816044 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5ab806e7f0 msgr2=0x7f5ab812d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:07.505 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.490+0000 7f5abcc1f640 1 --2- 172.21.15.154:0/3087816044 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5ab806e7f0 0x7f5ab812d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:07.505 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.490+0000 7f5abcc1f640 10 monclient: _finish_hunting 0 2024-09-16T03:42:07.505 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.490+0000 7f5abcc1f640 1 monclient: found mon.noname-b 2024-09-16T03:42:07.505 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.490+0000 7f5abcc1f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:42:07.505 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.490+0000 7f5abcc1f640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.154:3300/0 2024-09-16T03:42:07.505 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.490+0000 7f5abcc1f640 1 -- 172.21.15.154:0/3087816044 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ab813c0b0 con 0x7f5ab80a4fd0 2024-09-16T03:42:07.505 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.490+0000 7f5abcc1f640 1 --2- 172.21.15.154:0/3087816044 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5ab80a4fd0 0x7f5ab812dd60 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f5aa800c480 tx=0x7f5aa800c940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:07.505 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.490+0000 7f5aadffb640 10 client.?.objecter ms_handle_connect 0x7f5ab80a4fd0 2024-09-16T03:42:07.505 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.490+0000 7f5aadffb640 10 client.?.objecter resend_mon_ops 2024-09-16T03:42:07.505 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.490+0000 7f5aadffb640 1 -- 172.21.15.154:0/3087816044 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5aa8012830 con 0x7f5ab80a4fd0 2024-09-16T03:42:07.505 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.490+0000 7f5aadffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:07.505 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.490+0000 7f5aadffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:42:07.506 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.490+0000 7f5aadffb640 10 monclient: dump: 2024-09-16T03:42:07.506 INFO:teuthology.orchestra.run.smithi154.stderr:epoch 1 2024-09-16T03:42:07.506 INFO:teuthology.orchestra.run.smithi154.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:07.506 INFO:teuthology.orchestra.run.smithi154.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:07.506 INFO:teuthology.orchestra.run.smithi154.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:07.506 INFO:teuthology.orchestra.run.smithi154.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:07.506 INFO:teuthology.orchestra.run.smithi154.stderr:election_strategy: 1 2024-09-16T03:42:07.506 INFO:teuthology.orchestra.run.smithi154.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:07.506 INFO:teuthology.orchestra.run.smithi154.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:07.506 INFO:teuthology.orchestra.run.smithi154.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:07.506 INFO:teuthology.orchestra.run.smithi154.stderr: 2024-09-16T03:42:07.506 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.490+0000 7f5aadffb640 10 monclient: _finish_auth 0 2024-09-16T03:42:07.506 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.490+0000 7f5aadffb640 10 monclient: _check_auth_tickets 2024-09-16T03:42:07.507 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.490+0000 7f5aadffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:42:07.507 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.490+0000 7f5aadffb640 1 -- 172.21.15.154:0/3087816044 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5aa8017070 con 0x7f5ab80a4fd0 2024-09-16T03:42:07.507 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.490+0000 7f5aadffb640 10 monclient: handle_config config(0 keys) 2024-09-16T03:42:07.507 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.490+0000 7f5aadffb640 1 -- 172.21.15.154:0/3087816044 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5aa8012c30 con 0x7f5ab80a4fd0 2024-09-16T03:42:07.507 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.490+0000 7f5aadffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:07.507 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.490+0000 7f5aadffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:42:07.507 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.490+0000 7f5aadffb640 10 monclient: dump: 2024-09-16T03:42:07.507 INFO:teuthology.orchestra.run.smithi154.stderr:epoch 1 2024-09-16T03:42:07.507 INFO:teuthology.orchestra.run.smithi154.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:07.507 INFO:teuthology.orchestra.run.smithi154.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:07.507 INFO:teuthology.orchestra.run.smithi154.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:07.507 INFO:teuthology.orchestra.run.smithi154.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:07.507 INFO:teuthology.orchestra.run.smithi154.stderr:election_strategy: 1 2024-09-16T03:42:07.507 INFO:teuthology.orchestra.run.smithi154.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:07.508 INFO:teuthology.orchestra.run.smithi154.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:07.508 INFO:teuthology.orchestra.run.smithi154.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:07.508 INFO:teuthology.orchestra.run.smithi154.stderr: 2024-09-16T03:42:07.508 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.490+0000 7f5abf684640 5 monclient: authenticate success, global_id 4123 2024-09-16T03:42:07.508 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.490+0000 7f5abf684640 10 monclient: _renew_subs 2024-09-16T03:42:07.508 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.490+0000 7f5abf684640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2024-09-16T03:42:07.508 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.490+0000 7f5abf684640 1 -- 172.21.15.154:0/3087816044 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ab813df40 con 0x7f5ab80a4fd0 2024-09-16T03:42:07.508 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.490+0000 7f5aaf7fe640 4 set_mon_vals no callback set 2024-09-16T03:42:07.508 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.490+0000 7f5abf684640 10 client.4123.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:42:07.508 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.490+0000 7f5abf684640 10 monclient: _renew_subs 2024-09-16T03:42:07.508 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.490+0000 7f5abf684640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2024-09-16T03:42:07.508 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.490+0000 7f5abf684640 1 -- 172.21.15.154:0/3087816044 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ab8137920 con 0x7f5ab80a4fd0 2024-09-16T03:42:07.508 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.490+0000 7f5abf684640 1 librados: init done 2024-09-16T03:42:07.508 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.490+0000 7f5aadffb640 1 -- 172.21.15.154:0/3087816044 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f5aa801c460 con 0x7f5ab80a4fd0 2024-09-16T03:42:07.509 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.490+0000 7f5aadffb640 1 -- 172.21.15.154:0/3087816044 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 840+0+0 (secure 0 0 0) 0x7f5aa801b6e0 con 0x7f5ab80a4fd0 2024-09-16T03:42:07.509 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.490+0000 7f5aadffb640 10 client.4123.objecter ms_dispatch 0x7f5ab8000ba0 osd_map(3..3 src has 1..3) 2024-09-16T03:42:07.509 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.490+0000 7f5aadffb640 3 client.4123.objecter handle_osd_map got epochs [3,3] > 0 2024-09-16T03:42:07.509 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.490+0000 7f5aadffb640 3 client.4123.objecter handle_osd_map decoding full epoch 3 2024-09-16T03:42:07.509 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.490+0000 7f5aadffb640 20 client.4123.objecter dump_active .. 0 homeless 2024-09-16T03:42:07.509 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.490+0000 7f5abf684640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:42:07.509 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.490+0000 7f5abf684640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:42:07.509 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.490+0000 7f5abf684640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2024-09-16T03:42:07.509 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.490+0000 7f5abf684640 1 -- 172.21.15.154:0/3087816044 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ab80a4a90 con 0x7f5ab80a4fd0 2024-09-16T03:42:07.510 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.494+0000 7f5aadffb640 1 -- 172.21.15.154:0/3087816044 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (secure 0 0 0) 0x7f5aa8012dd0 con 0x7f5ab80a4fd0 2024-09-16T03:42:07.510 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.494+0000 7f5aadffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:42:07.510 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.494+0000 7f5aadffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:42:07.564 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.562+0000 7f5abf684640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "ee31f807-cb16-4c33-8e23-2573e1a6b5c9", "id": 1}] 2024-09-16T03:42:07.564 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.562+0000 7f5abf684640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "ee31f807-cb16-4c33-8e23-2573e1a6b5c9", "id": 1}] 2024-09-16T03:42:07.564 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.562+0000 7f5abf684640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2024-09-16T03:42:07.564 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.562+0000 7f5abf684640 1 -- 172.21.15.154:0/3087816044 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "osd new", "uuid": "ee31f807-cb16-4c33-8e23-2573e1a6b5c9", "id": 1} v 0) -- 0x7f5ab8068e40 con 0x7f5ab80a4fd0 2024-09-16T03:42:07.571 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.570+0000 7f5aadffb640 1 -- 172.21.15.154:0/3087816044 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "ee31f807-cb16-4c33-8e23-2573e1a6b5c9", "id": 1}]=0 v4) ==== 112+0+2 (secure 0 0 0) 0x7f5aa801ca60 con 0x7f5ab80a4fd0 2024-09-16T03:42:07.571 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.570+0000 7f5aadffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "ee31f807-cb16-4c33-8e23-2573e1a6b5c9", "id": 1}] 2024-09-16T03:42:07.571 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.570+0000 7f5aadffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:42:07.572 INFO:teuthology.orchestra.run.smithi154.stdout:1 2024-09-16T03:42:07.573 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.570+0000 7f5abf684640 10 librados: watch_flush enter 2024-09-16T03:42:07.573 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.570+0000 7f5abf684640 10 librados: watch_flush exit 2024-09-16T03:42:07.573 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.570+0000 7f5abf684640 20 client.4123.objecter shutdown clearing up homeless session... 2024-09-16T03:42:07.573 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.570+0000 7f5abf684640 10 client.4123.objecter successfully canceled tick 2024-09-16T03:42:07.573 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.570+0000 7f5abf684640 10 monclient: shutdown 2024-09-16T03:42:07.573 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.570+0000 7f5abf684640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:42:07.573 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.570+0000 7f5abf684640 1 -- 172.21.15.154:0/3087816044 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5ab80a4fd0 msgr2=0x7f5ab812dd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:07.573 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.570+0000 7f5abf684640 1 --2- 172.21.15.154:0/3087816044 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5ab80a4fd0 0x7f5ab812dd60 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f5aa800c480 tx=0x7f5aa800c940 comp rx=0 tx=0).stop 2024-09-16T03:42:07.573 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.570+0000 7f5abf684640 1 -- 172.21.15.154:0/3087816044 shutdown_connections 2024-09-16T03:42:07.573 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.570+0000 7f5abf684640 1 --2- 172.21.15.154:0/3087816044 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5ab80a5a00 0x7f5ab81cedf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:07.574 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.570+0000 7f5abf684640 1 --2- 172.21.15.154:0/3087816044 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5ab80a4fd0 0x7f5ab812dd60 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:07.574 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.570+0000 7f5abf684640 1 --2- 172.21.15.154:0/3087816044 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5ab806e7f0 0x7f5ab812d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:07.574 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.570+0000 7f5abf684640 1 -- 172.21.15.154:0/3087816044 >> 172.21.15.154:0/3087816044 conn(0x7f5ab80aa1b0 msgr2=0x7f5ab8136f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:42:07.574 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.570+0000 7f5abf684640 1 -- 172.21.15.154:0/3087816044 shutdown_connections 2024-09-16T03:42:07.574 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.570+0000 7f5abf684640 1 -- 172.21.15.154:0/3087816044 wait complete. 2024-09-16T03:42:07.574 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.570+0000 7f5abf684640 1 librados: shutdown 2024-09-16T03:42:07.588 DEBUG:teuthology.orchestra.run.smithi154:> sudo ceph --cluster ceph osd new a26f3917-6702-43db-9355-e96da86a3365 2 2024-09-16T03:42:07.638 INFO:tasks.ceph.mgr.x.smithi154.stderr:2024-09-16T03:42:07.638+0000 7f8fbbf11180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-16T03:42:07.705 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.702+0000 7f68d179c640 10 monclient: get_monmap_and_config 2024-09-16T03:42:07.705 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.702+0000 7f68d179c640 10 monclient: build_initial_monmap 2024-09-16T03:42:07.705 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.702+0000 7f68d179c640 10 monclient: monmap: 2024-09-16T03:42:07.705 INFO:teuthology.orchestra.run.smithi154.stderr:epoch 0 2024-09-16T03:42:07.705 INFO:teuthology.orchestra.run.smithi154.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:07.705 INFO:teuthology.orchestra.run.smithi154.stderr:last_changed 2024-09-16T03:42:07.706486+0000 2024-09-16T03:42:07.705 INFO:teuthology.orchestra.run.smithi154.stderr:created 2024-09-16T03:42:07.706486+0000 2024-09-16T03:42:07.705 INFO:teuthology.orchestra.run.smithi154.stderr:min_mon_release 0 (unknown) 2024-09-16T03:42:07.705 INFO:teuthology.orchestra.run.smithi154.stderr:election_strategy: 1 2024-09-16T03:42:07.705 INFO:teuthology.orchestra.run.smithi154.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:42:07.705 INFO:teuthology.orchestra.run.smithi154.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:42:07.705 INFO:teuthology.orchestra.run.smithi154.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:42:07.706 INFO:teuthology.orchestra.run.smithi154.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:42:07.706 INFO:teuthology.orchestra.run.smithi154.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:42:07.706 INFO:teuthology.orchestra.run.smithi154.stderr: 2024-09-16T03:42:07.706 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.702+0000 7f68d179c640 1 Processor -- start 2024-09-16T03:42:07.706 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.702+0000 7f68d179c640 1 -- start start 2024-09-16T03:42:07.706 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.702+0000 7f68d179c640 10 monclient: init 2024-09-16T03:42:07.708 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.702+0000 7f68d179c640 10 monclient: _reopen_session rank -1 2024-09-16T03:42:07.708 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.702+0000 7f68d179c640 10 monclient: _add_conns ranks=[0,3,4,1,2] 2024-09-16T03:42:07.708 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.702+0000 7f68d179c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f68cc134450 addr v1:172.21.15.42:6789/0 2024-09-16T03:42:07.708 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.702+0000 7f68d179c640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f68cc071440 0x7f68cc0a27e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:07.708 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.702+0000 7f68d179c640 10 monclient(hunting): picked mon.noname-a con 0x7f68cc071440 addr v2:172.21.15.42:3300/0 2024-09-16T03:42:07.708 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.702+0000 7f68d179c640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f68cc139430 0x7f68cc13b820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:07.708 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.702+0000 7f68d179c640 10 monclient(hunting): picked mon.noname-c con 0x7f68cc139430 addr v2:172.21.15.154:3300/0 2024-09-16T03:42:07.708 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.702+0000 7f68d179c640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f68cc06ea60 con 0x7f68cc134450 2024-09-16T03:42:07.708 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.702+0000 7f68d179c640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:07.709 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.702+0000 7f68d179c640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f68cc070af0 con 0x7f68cc071440 2024-09-16T03:42:07.709 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.702+0000 7f68d179c640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:07.709 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.702+0000 7f68d179c640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f68cc06eea0 con 0x7f68cc139430 2024-09-16T03:42:07.709 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.702+0000 7f68d179c640 10 monclient(hunting): _renew_subs 2024-09-16T03:42:07.709 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.702+0000 7f68d179c640 10 monclient(hunting): authenticate will time out at 966.535054s 2024-09-16T03:42:07.709 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.702+0000 7f68cb7fe640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f68cc139430 0x7f68cc13b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:07.709 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.702+0000 7f68cb7fe640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f68cc139430 0x7f68cc13b820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.154:50146/0 (socket says 172.21.15.154:50146) 2024-09-16T03:42:07.709 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.702+0000 7f68cb7fe640 1 -- 172.21.15.154:0/4254642313 learned_addr learned my addr 172.21.15.154:0/4254642313 (peer_addr_for_me v2:172.21.15.154:0/0) 2024-09-16T03:42:07.709 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.702+0000 7f68cb7fe640 10 monclient(hunting): get_auth_request con 0x7f68cc139430 auth_method 0 2024-09-16T03:42:07.709 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.702+0000 7f68cb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:07.709 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.702+0000 7f68cb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:07.709 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.702+0000 7f68cb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:07.709 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.702+0000 7f68cb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:07.709 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.702+0000 7f68cb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:07.710 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.702+0000 7f68cb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:07.710 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.702+0000 7f68ca7fc640 1 --2- 172.21.15.154:0/4254642313 >> v2:172.21.15.42:3300/0 conn(0x7f68cc071440 0x7f68cc0a27e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:07.710 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.702+0000 7f68ca7fc640 10 monclient(hunting): get_auth_request con 0x7f68cc071440 auth_method 0 2024-09-16T03:42:07.710 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.702+0000 7f68ca7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:07.710 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.702+0000 7f68ca7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:07.710 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.702+0000 7f68ca7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:07.710 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.702+0000 7f68cb7fe640 10 monclient(hunting): handle_auth_done global_id 4126 payload 274 2024-09-16T03:42:07.710 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.702+0000 7f68cb7fe640 1 -- 172.21.15.154:0/4254642313 >> v2:172.21.15.42:3300/0 conn(0x7f68cc071440 msgr2=0x7f68cc0a27e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:42:07.710 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.702+0000 7f68cb7fe640 1 --2- 172.21.15.154:0/4254642313 >> v2:172.21.15.42:3300/0 conn(0x7f68cc071440 0x7f68cc0a27e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:07.710 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.702+0000 7f68cb7fe640 1 -- 172.21.15.154:0/4254642313 >> v1:172.21.15.42:6789/0 conn(0x7f68cc134450 legacy=0x7f68cc134830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:42:07.710 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.702+0000 7f68cb7fe640 10 monclient: _finish_hunting 0 2024-09-16T03:42:07.710 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.702+0000 7f68cb7fe640 1 monclient: found mon.noname-c 2024-09-16T03:42:07.710 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.702+0000 7f68cb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:42:07.710 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.702+0000 7f68cb7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.154:3300/0 2024-09-16T03:42:07.711 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.702+0000 7f68cb7fe640 1 -- 172.21.15.154:0/4254642313 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68cc0701c0 con 0x7f68cc139430 2024-09-16T03:42:07.715 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.706+0000 7f68cb7fe640 1 --2- 172.21.15.154:0/4254642313 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f68cc139430 0x7f68cc13b820 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f68cc0a34c0 tx=0x7f68c401ea00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d5a6aa6f8b67be5f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:07.715 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.706+0000 7f68c9ffb640 1 -- 172.21.15.154:0/4254642313 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68c400b2d0 con 0x7f68cc139430 2024-09-16T03:42:07.715 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.706+0000 7f68c9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:07.715 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.706+0000 7f68c9ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:42:07.715 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.706+0000 7f68c9ffb640 10 monclient: dump: 2024-09-16T03:42:07.715 INFO:teuthology.orchestra.run.smithi154.stderr:epoch 1 2024-09-16T03:42:07.715 INFO:teuthology.orchestra.run.smithi154.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:07.715 INFO:teuthology.orchestra.run.smithi154.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:07.716 INFO:teuthology.orchestra.run.smithi154.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:07.716 INFO:teuthology.orchestra.run.smithi154.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:07.716 INFO:teuthology.orchestra.run.smithi154.stderr:election_strategy: 1 2024-09-16T03:42:07.716 INFO:teuthology.orchestra.run.smithi154.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:07.716 INFO:teuthology.orchestra.run.smithi154.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:07.716 INFO:teuthology.orchestra.run.smithi154.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:07.716 INFO:teuthology.orchestra.run.smithi154.stderr: 2024-09-16T03:42:07.716 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.706+0000 7f68c9ffb640 10 monclient: _finish_auth 0 2024-09-16T03:42:07.716 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.706+0000 7f68c9ffb640 10 monclient: _check_auth_tickets 2024-09-16T03:42:07.716 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.706+0000 7f68c9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:42:07.716 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.706+0000 7f68c9ffb640 1 -- 172.21.15.154:0/4254642313 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f68c400b470 con 0x7f68cc139430 2024-09-16T03:42:07.716 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.706+0000 7f68d179c640 5 monclient: authenticate success, global_id 4126 2024-09-16T03:42:07.716 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.706+0000 7f68d179c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:42:07.717 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.706+0000 7f68c9ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T03:42:07.717 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.706+0000 7f68d179c640 10 monclient: get_monmap_and_config success 2024-09-16T03:42:07.717 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.706+0000 7f68c9ffb640 1 -- 172.21.15.154:0/4254642313 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68c40a7a10 con 0x7f68cc139430 2024-09-16T03:42:07.717 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.706+0000 7f68c9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:07.717 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.706+0000 7f68d179c640 4 set_mon_vals no callback set 2024-09-16T03:42:07.717 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.706+0000 7f68c9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:42:07.717 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.706+0000 7f68c9ffb640 10 monclient: dump: 2024-09-16T03:42:07.717 INFO:teuthology.orchestra.run.smithi154.stderr:epoch 1 2024-09-16T03:42:07.717 INFO:teuthology.orchestra.run.smithi154.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:07.717 INFO:teuthology.orchestra.run.smithi154.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:07.717 INFO:teuthology.orchestra.run.smithi154.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:07.717 INFO:teuthology.orchestra.run.smithi154.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:07.717 INFO:teuthology.orchestra.run.smithi154.stderr:election_strategy: 1 2024-09-16T03:42:07.717 INFO:teuthology.orchestra.run.smithi154.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:07.718 INFO:teuthology.orchestra.run.smithi154.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:07.718 INFO:teuthology.orchestra.run.smithi154.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:07.718 INFO:teuthology.orchestra.run.smithi154.stderr: 2024-09-16T03:42:07.718 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.706+0000 7f68d179c640 10 monclient: shutdown 2024-09-16T03:42:07.718 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.706+0000 7f68d179c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:42:07.718 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.706+0000 7f68d179c640 1 -- 172.21.15.154:0/4254642313 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f68cc139430 msgr2=0x7f68cc13b820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:07.718 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.706+0000 7f68d179c640 1 --2- 172.21.15.154:0/4254642313 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f68cc139430 0x7f68cc13b820 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f68cc0a34c0 tx=0x7f68c401ea00 comp rx=0 tx=0).stop 2024-09-16T03:42:07.718 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.706+0000 7f68d179c640 1 -- 172.21.15.154:0/4254642313 shutdown_connections 2024-09-16T03:42:07.718 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.706+0000 7f68d179c640 1 --2- 172.21.15.154:0/4254642313 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f68cc139430 0x7f68cc13b820 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:07.718 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.706+0000 7f68d179c640 1 --2- 172.21.15.154:0/4254642313 >> v2:172.21.15.42:3300/0 conn(0x7f68cc071440 0x7f68cc0a27e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:07.718 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.706+0000 7f68d179c640 1 -- 172.21.15.154:0/4254642313 >> 172.21.15.154:0/4254642313 conn(0x7f68cc0a6e30 msgr2=0x7f68cc0a7240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:42:07.718 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.706+0000 7f68d179c640 1 -- 172.21.15.154:0/4254642313 shutdown_connections 2024-09-16T03:42:07.718 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.706+0000 7f68d179c640 1 -- 172.21.15.154:0/4254642313 wait complete. 2024-09-16T03:42:07.718 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.706+0000 7f68d179c640 10 monclient: build_initial_monmap 2024-09-16T03:42:07.719 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.706+0000 7f68d179c640 10 monclient: monmap: 2024-09-16T03:42:07.719 INFO:teuthology.orchestra.run.smithi154.stderr:epoch 0 2024-09-16T03:42:07.719 INFO:teuthology.orchestra.run.smithi154.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:42:07.719 INFO:teuthology.orchestra.run.smithi154.stderr:last_changed 0.000000 2024-09-16T03:42:07.719 INFO:teuthology.orchestra.run.smithi154.stderr:created 0.000000 2024-09-16T03:42:07.719 INFO:teuthology.orchestra.run.smithi154.stderr:min_mon_release 0 (unknown) 2024-09-16T03:42:07.719 INFO:teuthology.orchestra.run.smithi154.stderr:election_strategy: 1 2024-09-16T03:42:07.719 INFO:teuthology.orchestra.run.smithi154.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:42:07.719 INFO:teuthology.orchestra.run.smithi154.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:42:07.719 INFO:teuthology.orchestra.run.smithi154.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:42:07.719 INFO:teuthology.orchestra.run.smithi154.stderr: 2024-09-16T03:42:07.719 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.706+0000 7f68d179c640 1 librados: starting msgr at 2024-09-16T03:42:07.719 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.706+0000 7f68d179c640 1 librados: starting objecter 2024-09-16T03:42:07.719 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.706+0000 7f68d179c640 1 Processor -- start 2024-09-16T03:42:07.720 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.706+0000 7f68d179c640 1 -- start start 2024-09-16T03:42:07.720 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.706+0000 7f68d179c640 1 librados: setting wanted keys 2024-09-16T03:42:07.720 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.706+0000 7f68d179c640 1 librados: calling monclient init 2024-09-16T03:42:07.720 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.706+0000 7f68d179c640 10 monclient: init 2024-09-16T03:42:07.720 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.706+0000 7f68d179c640 10 monclient: _reopen_session rank -1 2024-09-16T03:42:07.720 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.706+0000 7f68d179c640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:42:07.720 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.706+0000 7f68d179c640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f68cc071440 0x7f68cc1330c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:07.720 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.706+0000 7f68d179c640 10 monclient(hunting): picked mon.noname-c con 0x7f68cc071440 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:42:07.720 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.706+0000 7f68d179c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f68cc134450 0x7f68cc09d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:07.720 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.706+0000 7f68d179c640 10 monclient(hunting): picked mon.noname-a con 0x7f68cc134450 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:42:07.720 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.706+0000 7f68d179c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f68cc139430 0x7f68cc09db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:07.720 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.706+0000 7f68d179c640 10 monclient(hunting): picked mon.noname-b con 0x7f68cc139430 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:42:07.720 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.706+0000 7f68d179c640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:07.720 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.706+0000 7f68d179c640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f68cc06c5d0 con 0x7f68cc134450 2024-09-16T03:42:07.721 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.706+0000 7f68d179c640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:07.721 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.706+0000 7f68d179c640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f68cc06c450 con 0x7f68cc139430 2024-09-16T03:42:07.721 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.706+0000 7f68d179c640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:07.721 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.706+0000 7f68d179c640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f68cc06c750 con 0x7f68cc071440 2024-09-16T03:42:07.721 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.706+0000 7f68d179c640 10 monclient(hunting): _renew_subs 2024-09-16T03:42:07.721 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.706+0000 7f68d179c640 10 monclient(hunting): authenticate will time out at 966.539913s 2024-09-16T03:42:07.721 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.710+0000 7f68cb7fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f68cc139430 0x7f68cc09db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:07.721 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.710+0000 7f68cb7fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f68cc139430 0x7f68cc09db90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.154:50158/0 (socket says 172.21.15.154:50158) 2024-09-16T03:42:07.721 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.710+0000 7f68cb7fe640 1 -- 172.21.15.154:0/3589996822 learned_addr learned my addr 172.21.15.154:0/3589996822 (peer_addr_for_me v2:172.21.15.154:0/0) 2024-09-16T03:42:07.721 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.710+0000 7f68cb7fe640 10 monclient(hunting): get_auth_request con 0x7f68cc139430 auth_method 0 2024-09-16T03:42:07.721 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.710+0000 7f68cb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:07.721 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.710+0000 7f68cb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:07.721 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.710+0000 7f68cb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:07.721 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.710+0000 7f68cb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:07.722 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.710+0000 7f68cb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:07.722 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.710+0000 7f68cb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:07.722 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.710+0000 7f68ca7fc640 1 --2- 172.21.15.154:0/3589996822 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f68cc134450 0x7f68cc09d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:07.722 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.710+0000 7f68caffd640 1 --2- 172.21.15.154:0/3589996822 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f68cc071440 0x7f68cc1330c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:07.722 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.710+0000 7f68cb7fe640 10 monclient(hunting): handle_auth_done global_id 4129 payload 824 2024-09-16T03:42:07.722 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.710+0000 7f68cb7fe640 1 -- 172.21.15.154:0/3589996822 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f68cc071440 msgr2=0x7f68cc1330c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:07.722 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.710+0000 7f68cb7fe640 1 --2- 172.21.15.154:0/3589996822 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f68cc071440 0x7f68cc1330c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:07.722 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.710+0000 7f68cb7fe640 1 -- 172.21.15.154:0/3589996822 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f68cc134450 msgr2=0x7f68cc09d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:07.722 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.710+0000 7f68cb7fe640 1 --2- 172.21.15.154:0/3589996822 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f68cc134450 0x7f68cc09d650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:07.722 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.710+0000 7f68cb7fe640 10 monclient: _finish_hunting 0 2024-09-16T03:42:07.722 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.710+0000 7f68cb7fe640 1 monclient: found mon.noname-b 2024-09-16T03:42:07.722 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.710+0000 7f68cb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:42:07.722 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.710+0000 7f68cb7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.154:3300/0 2024-09-16T03:42:07.722 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.710+0000 7f68cb7fe640 1 -- 172.21.15.154:0/3589996822 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68cc11d590 con 0x7f68cc139430 2024-09-16T03:42:07.723 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.710+0000 7f68ca7fc640 10 monclient: get_auth_request con 0x7f68cc134450 auth_method 0 2024-09-16T03:42:07.723 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.710+0000 7f68caffd640 10 monclient: get_auth_request con 0x7f68cc071440 auth_method 0 2024-09-16T03:42:07.723 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.710+0000 7f68ca7fc640 1 --2- 172.21.15.154:0/3589996822 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f68cc134450 0x7f68cc09d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T03:42:07.723 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.710+0000 7f68caffd640 1 --2- 172.21.15.154:0/3589996822 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f68cc071440 0x7f68cc1330c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T03:42:07.723 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.710+0000 7f68cb7fe640 1 --2- 172.21.15.154:0/3589996822 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f68cc139430 0x7f68cc09db90 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f68c401c630 tx=0x7f68c401b7a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:07.723 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.710+0000 7f68b37fe640 10 client.?.objecter ms_handle_connect 0x7f68cc139430 2024-09-16T03:42:07.723 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.710+0000 7f68b37fe640 10 client.?.objecter resend_mon_ops 2024-09-16T03:42:07.723 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.710+0000 7f68b37fe640 1 -- 172.21.15.154:0/3589996822 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68c40be020 con 0x7f68cc139430 2024-09-16T03:42:07.723 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.710+0000 7f68b37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:07.723 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.710+0000 7f68b37fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:42:07.723 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.710+0000 7f68b37fe640 10 monclient: dump: 2024-09-16T03:42:07.723 INFO:teuthology.orchestra.run.smithi154.stderr:epoch 1 2024-09-16T03:42:07.723 INFO:teuthology.orchestra.run.smithi154.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:07.723 INFO:teuthology.orchestra.run.smithi154.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:07.723 INFO:teuthology.orchestra.run.smithi154.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:07.724 INFO:teuthology.orchestra.run.smithi154.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:07.724 INFO:teuthology.orchestra.run.smithi154.stderr:election_strategy: 1 2024-09-16T03:42:07.724 INFO:teuthology.orchestra.run.smithi154.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:07.724 INFO:teuthology.orchestra.run.smithi154.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:07.724 INFO:teuthology.orchestra.run.smithi154.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:07.724 INFO:teuthology.orchestra.run.smithi154.stderr: 2024-09-16T03:42:07.724 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.710+0000 7f68b37fe640 10 monclient: _finish_auth 0 2024-09-16T03:42:07.724 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.710+0000 7f68b37fe640 10 monclient: _check_auth_tickets 2024-09-16T03:42:07.724 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.710+0000 7f68b37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:42:07.724 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.710+0000 7f68b37fe640 1 -- 172.21.15.154:0/3589996822 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f68c4009800 con 0x7f68cc139430 2024-09-16T03:42:07.724 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.710+0000 7f68d179c640 5 monclient: authenticate success, global_id 4129 2024-09-16T03:42:07.724 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.710+0000 7f68d179c640 10 monclient: _renew_subs 2024-09-16T03:42:07.724 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.710+0000 7f68d179c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2024-09-16T03:42:07.724 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.710+0000 7f68d179c640 1 -- 172.21.15.154:0/3589996822 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68cc0a0090 con 0x7f68cc139430 2024-09-16T03:42:07.725 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.710+0000 7f68b37fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:42:07.725 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.710+0000 7f68b37fe640 1 -- 172.21.15.154:0/3589996822 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68c400b720 con 0x7f68cc139430 2024-09-16T03:42:07.725 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.710+0000 7f68b37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:07.725 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.710+0000 7f68b37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:42:07.725 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.710+0000 7f68b37fe640 10 monclient: dump: 2024-09-16T03:42:07.725 INFO:teuthology.orchestra.run.smithi154.stderr:epoch 1 2024-09-16T03:42:07.725 INFO:teuthology.orchestra.run.smithi154.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:07.725 INFO:teuthology.orchestra.run.smithi154.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:07.725 INFO:teuthology.orchestra.run.smithi154.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:07.725 INFO:teuthology.orchestra.run.smithi154.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:07.725 INFO:teuthology.orchestra.run.smithi154.stderr:election_strategy: 1 2024-09-16T03:42:07.725 INFO:teuthology.orchestra.run.smithi154.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:07.725 INFO:teuthology.orchestra.run.smithi154.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:07.725 INFO:teuthology.orchestra.run.smithi154.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:07.726 INFO:teuthology.orchestra.run.smithi154.stderr: 2024-09-16T03:42:07.726 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.710+0000 7f68c97fa640 4 set_mon_vals no callback set 2024-09-16T03:42:07.726 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.710+0000 7f68d179c640 10 client.4129.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:42:07.726 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.710+0000 7f68d179c640 10 monclient: _renew_subs 2024-09-16T03:42:07.726 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.710+0000 7f68d179c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2024-09-16T03:42:07.726 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.710+0000 7f68d179c640 1 -- 172.21.15.154:0/3589996822 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68cc0a63a0 con 0x7f68cc139430 2024-09-16T03:42:07.726 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.710+0000 7f68d179c640 1 librados: init done 2024-09-16T03:42:07.726 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.710+0000 7f68b37fe640 1 -- 172.21.15.154:0/3589996822 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f68c40c1020 con 0x7f68cc139430 2024-09-16T03:42:07.726 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.710+0000 7f68b37fe640 1 -- 172.21.15.154:0/3589996822 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 955+0+0 (secure 0 0 0) 0x7f68c40bac40 con 0x7f68cc139430 2024-09-16T03:42:07.726 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.710+0000 7f68b37fe640 10 client.4129.objecter ms_dispatch 0x7f68cc000ba0 osd_map(4..4 src has 1..4) 2024-09-16T03:42:07.726 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.710+0000 7f68b37fe640 3 client.4129.objecter handle_osd_map got epochs [4,4] > 0 2024-09-16T03:42:07.726 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.710+0000 7f68b37fe640 3 client.4129.objecter handle_osd_map decoding full epoch 4 2024-09-16T03:42:07.726 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.710+0000 7f68b37fe640 20 client.4129.objecter dump_active .. 0 homeless 2024-09-16T03:42:07.726 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.710+0000 7f68d179c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:42:07.727 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.710+0000 7f68d179c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:42:07.727 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.710+0000 7f68d179c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2024-09-16T03:42:07.727 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.710+0000 7f68d179c640 1 -- 172.21.15.154:0/3589996822 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f68cc134830 con 0x7f68cc139430 2024-09-16T03:42:07.727 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.710+0000 7f68b37fe640 1 -- 172.21.15.154:0/3589996822 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (secure 0 0 0) 0x7f68c40c8030 con 0x7f68cc139430 2024-09-16T03:42:07.727 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.710+0000 7f68b37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:42:07.727 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.710+0000 7f68b37fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:42:07.783 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.782+0000 7f68d179c640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "a26f3917-6702-43db-9355-e96da86a3365", "id": 2}] 2024-09-16T03:42:07.784 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.782+0000 7f68d179c640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "a26f3917-6702-43db-9355-e96da86a3365", "id": 2}] 2024-09-16T03:42:07.784 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.782+0000 7f68d179c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2024-09-16T03:42:07.784 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.782+0000 7f68d179c640 1 -- 172.21.15.154:0/3589996822 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "osd new", "uuid": "a26f3917-6702-43db-9355-e96da86a3365", "id": 2} v 0) -- 0x7f68cc11dab0 con 0x7f68cc139430 2024-09-16T03:42:07.790 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.786+0000 7f68b37fe640 1 -- 172.21.15.154:0/3589996822 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "a26f3917-6702-43db-9355-e96da86a3365", "id": 2}]=0 v5) ==== 112+0+2 (secure 0 0 0) 0x7f68c4018740 con 0x7f68cc139430 2024-09-16T03:42:07.790 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.786+0000 7f68b37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "a26f3917-6702-43db-9355-e96da86a3365", "id": 2}] 2024-09-16T03:42:07.791 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.786+0000 7f68b37fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:42:07.791 INFO:teuthology.orchestra.run.smithi154.stdout:2 2024-09-16T03:42:07.792 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.790+0000 7f68d179c640 10 librados: watch_flush enter 2024-09-16T03:42:07.792 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.790+0000 7f68d179c640 10 librados: watch_flush exit 2024-09-16T03:42:07.792 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.790+0000 7f68d179c640 20 client.4129.objecter shutdown clearing up homeless session... 2024-09-16T03:42:07.792 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.790+0000 7f68d179c640 10 client.4129.objecter successfully canceled tick 2024-09-16T03:42:07.792 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.790+0000 7f68d179c640 10 monclient: shutdown 2024-09-16T03:42:07.792 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.790+0000 7f68d179c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:42:07.792 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.790+0000 7f68d179c640 1 -- 172.21.15.154:0/3589996822 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f68cc139430 msgr2=0x7f68cc09db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:07.792 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.790+0000 7f68d179c640 1 --2- 172.21.15.154:0/3589996822 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f68cc139430 0x7f68cc09db90 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f68c401c630 tx=0x7f68c401b7a0 comp rx=0 tx=0).stop 2024-09-16T03:42:07.792 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.790+0000 7f68d179c640 1 -- 172.21.15.154:0/3589996822 shutdown_connections 2024-09-16T03:42:07.793 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.790+0000 7f68d179c640 1 --2- 172.21.15.154:0/3589996822 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f68cc139430 0x7f68cc09db90 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:07.793 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.790+0000 7f68d179c640 1 --2- 172.21.15.154:0/3589996822 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f68cc134450 0x7f68cc09d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:07.793 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.790+0000 7f68d179c640 1 --2- 172.21.15.154:0/3589996822 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f68cc071440 0x7f68cc1330c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:07.793 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.790+0000 7f68d179c640 1 -- 172.21.15.154:0/3589996822 >> 172.21.15.154:0/3589996822 conn(0x7f68cc0a6e30 msgr2=0x7f68cc0a16e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:42:07.793 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.790+0000 7f68d179c640 1 -- 172.21.15.154:0/3589996822 shutdown_connections 2024-09-16T03:42:07.793 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.790+0000 7f68d179c640 1 -- 172.21.15.154:0/3589996822 wait complete. 2024-09-16T03:42:07.793 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.790+0000 7f68d179c640 1 librados: shutdown 2024-09-16T03:42:07.806 DEBUG:teuthology.orchestra.run.smithi154:> sudo ceph --cluster ceph osd new 4e809f2a-2ffd-4791-9f2d-18f1bfe7d2ea 3 2024-09-16T03:42:07.912 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.910+0000 7faccd824640 10 monclient: get_monmap_and_config 2024-09-16T03:42:07.912 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.910+0000 7faccd824640 10 monclient: build_initial_monmap 2024-09-16T03:42:07.912 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.910+0000 7faccd824640 10 monclient: monmap: 2024-09-16T03:42:07.912 INFO:teuthology.orchestra.run.smithi154.stderr:epoch 0 2024-09-16T03:42:07.912 INFO:teuthology.orchestra.run.smithi154.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:07.912 INFO:teuthology.orchestra.run.smithi154.stderr:last_changed 2024-09-16T03:42:07.914369+0000 2024-09-16T03:42:07.912 INFO:teuthology.orchestra.run.smithi154.stderr:created 2024-09-16T03:42:07.914369+0000 2024-09-16T03:42:07.913 INFO:teuthology.orchestra.run.smithi154.stderr:min_mon_release 0 (unknown) 2024-09-16T03:42:07.913 INFO:teuthology.orchestra.run.smithi154.stderr:election_strategy: 1 2024-09-16T03:42:07.913 INFO:teuthology.orchestra.run.smithi154.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:42:07.913 INFO:teuthology.orchestra.run.smithi154.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:42:07.913 INFO:teuthology.orchestra.run.smithi154.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:42:07.913 INFO:teuthology.orchestra.run.smithi154.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:42:07.913 INFO:teuthology.orchestra.run.smithi154.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:42:07.913 INFO:teuthology.orchestra.run.smithi154.stderr: 2024-09-16T03:42:07.925 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.910+0000 7faccd824640 1 Processor -- start 2024-09-16T03:42:07.925 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.910+0000 7faccd824640 1 -- start start 2024-09-16T03:42:07.925 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.910+0000 7faccd824640 10 monclient: init 2024-09-16T03:42:07.925 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.910+0000 7faccd824640 10 monclient: _reopen_session rank -1 2024-09-16T03:42:07.926 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.910+0000 7faccd824640 10 monclient: _add_conns ranks=[1,2,0,3,4] 2024-09-16T03:42:07.926 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.910+0000 7faccd824640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7facc806bc40 addr v1:172.21.15.154:6789/0 2024-09-16T03:42:07.926 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.910+0000 7faccd824640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7facc809fef0 0x7facc813c670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:07.926 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.910+0000 7faccd824640 10 monclient(hunting): picked mon.noname-b con 0x7facc809fef0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:42:07.926 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.910+0000 7faccd824640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7facc806c730 addr v1:172.21.15.42:6789/0 2024-09-16T03:42:07.926 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.910+0000 7faccd824640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7facc80a2ac0 con 0x7facc806c730 2024-09-16T03:42:07.926 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.910+0000 7faccd824640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7facc80a07b0 con 0x7facc806bc40 2024-09-16T03:42:07.926 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.910+0000 7faccd824640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:07.926 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.910+0000 7faccd824640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7facc80a0a90 con 0x7facc809fef0 2024-09-16T03:42:07.926 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.910+0000 7faccd824640 10 monclient(hunting): _renew_subs 2024-09-16T03:42:07.926 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.910+0000 7faccd824640 10 monclient(hunting): authenticate will time out at 966.743122s 2024-09-16T03:42:07.926 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.910+0000 7facc6ffd640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7facc806bc40 0x7facc806c020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.154:59302/0 (socket says 172.21.15.154:59302) 2024-09-16T03:42:07.927 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.910+0000 7facc6ffd640 1 -- 172.21.15.154:0/597716493 learned_addr learned my addr 172.21.15.154:0/597716493 (peer_addr_for_me v1:172.21.15.154:0/0) 2024-09-16T03:42:07.927 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.910+0000 7facc5ffb640 1 -- 172.21.15.154:0/597716493 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1036870127 0 0) 0x7facc80a07b0 con 0x7facc806bc40 2024-09-16T03:42:07.927 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.910+0000 7facc5ffb640 10 monclient(hunting): _negotiate 2024-09-16T03:42:07.927 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.910+0000 7facc5ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:07.927 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.910+0000 7facc5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:07.927 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.910+0000 7facc5ffb640 10 monclient(hunting): my global_id is 4132 2024-09-16T03:42:07.927 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.910+0000 7facc5ffb640 1 -- 172.21.15.154:0/597716493 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7facb4003530 con 0x7facc806bc40 2024-09-16T03:42:07.927 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.910+0000 7facc67fc640 1 --2- 172.21.15.154:0/597716493 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7facc809fef0 0x7facc813c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:07.927 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.910+0000 7facc67fc640 10 monclient(hunting): get_auth_request con 0x7facc809fef0 auth_method 0 2024-09-16T03:42:07.927 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.910+0000 7facc67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:07.927 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.910+0000 7facc67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:07.927 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.910+0000 7facc67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:07.927 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.910+0000 7facc5ffb640 1 -- 172.21.15.154:0/597716493 <== mon.1 v1:172.21.15.154:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1283026319 0 0) 0x7facb4003530 con 0x7facc806bc40 2024-09-16T03:42:07.928 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.910+0000 7facc5ffb640 1 -- 172.21.15.154:0/597716493 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7facc809fef0 msgr2=0x7facc813c670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:42:07.928 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.910+0000 7facc5ffb640 1 --2- 172.21.15.154:0/597716493 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7facc809fef0 0x7facc813c670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:07.928 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.910+0000 7facc5ffb640 1 -- 172.21.15.154:0/597716493 >> v1:172.21.15.42:6789/0 conn(0x7facc806c730 legacy=0x7facc813ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:07.928 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.910+0000 7facc5ffb640 10 monclient: _finish_hunting 0 2024-09-16T03:42:07.928 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.910+0000 7facc5ffb640 1 monclient: found mon.noname-c-legacy 2024-09-16T03:42:07.928 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.910+0000 7facc5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:42:07.928 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.910+0000 7facc5ffb640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.154:6789/0 2024-09-16T03:42:07.928 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.910+0000 7facc5ffb640 1 -- 172.21.15.154:0/597716493 --> v1:172.21.15.154:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7facc80a32a0 con 0x7facc806bc40 2024-09-16T03:42:07.928 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.910+0000 7facc5ffb640 10 monclient: _finish_auth 0 2024-09-16T03:42:07.928 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.910+0000 7facc5ffb640 10 monclient: _check_auth_tickets 2024-09-16T03:42:07.929 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.910+0000 7facc5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:42:07.929 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.910+0000 7faccd824640 5 monclient: authenticate success, global_id 4132 2024-09-16T03:42:07.929 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.910+0000 7facc5ffb640 1 -- 172.21.15.154:0/597716493 <== mon.1 v1:172.21.15.154:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2177413734 0 0) 0x7facb0001680 con 0x7facc806bc40 2024-09-16T03:42:07.929 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.910+0000 7faccd824640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:42:07.929 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.910+0000 7facc5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:07.929 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.910+0000 7facc5ffb640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-16T03:42:07.929 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.910+0000 7facc5ffb640 10 monclient: dump: 2024-09-16T03:42:07.929 INFO:teuthology.orchestra.run.smithi154.stderr:epoch 1 2024-09-16T03:42:07.929 INFO:teuthology.orchestra.run.smithi154.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:07.929 INFO:teuthology.orchestra.run.smithi154.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:07.929 INFO:teuthology.orchestra.run.smithi154.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:07.930 INFO:teuthology.orchestra.run.smithi154.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:07.930 INFO:teuthology.orchestra.run.smithi154.stderr:election_strategy: 1 2024-09-16T03:42:07.930 INFO:teuthology.orchestra.run.smithi154.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:07.930 INFO:teuthology.orchestra.run.smithi154.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:07.930 INFO:teuthology.orchestra.run.smithi154.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:07.930 INFO:teuthology.orchestra.run.smithi154.stderr: 2024-09-16T03:42:07.930 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.910+0000 7facc5ffb640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] but i'm connected to v1:172.21.15.154:6789/0, reconnecting 2024-09-16T03:42:07.930 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.910+0000 7facc5ffb640 10 monclient: _reopen_session rank -1 2024-09-16T03:42:07.930 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.910+0000 7facc5ffb640 1 -- 172.21.15.154:0/597716493 >> v1:172.21.15.154:6789/0 conn(0x7facc806bc40 legacy=0x7facc806c020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:07.930 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.910+0000 7facc5ffb640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:42:07.930 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.910+0000 7facc5ffb640 1 --2- 172.21.15.154:0/597716493 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7facc809fef0 0x7facb40039c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:07.930 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.910+0000 7facc5ffb640 10 monclient(hunting): picked mon.c con 0x7facc809fef0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:42:07.930 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.910+0000 7facc5ffb640 1 --2- 172.21.15.154:0/597716493 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7facb4003ff0 0x7facb4004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:07.930 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.910+0000 7facc5ffb640 10 monclient(hunting): picked mon.b con 0x7facb4003ff0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:42:07.931 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.910+0000 7facc5ffb640 1 --2- 172.21.15.154:0/597716493 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7facb4004da0 0x7facb4005220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:07.931 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.910+0000 7facc5ffb640 10 monclient(hunting): picked mon.a con 0x7facb4004da0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:42:07.931 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.910+0000 7facc5ffb640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:07.931 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.910+0000 7facc5ffb640 1 -- 172.21.15.154:0/597716493 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7facc80a0a90 con 0x7facb4004da0 2024-09-16T03:42:07.931 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.910+0000 7facc5ffb640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:07.931 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.910+0000 7facc5ffb640 1 -- 172.21.15.154:0/597716493 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7facb400bc80 con 0x7facb4003ff0 2024-09-16T03:42:07.931 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.910+0000 7facc5ffb640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:07.931 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.910+0000 7facc5ffb640 1 -- 172.21.15.154:0/597716493 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7facb400be80 con 0x7facc809fef0 2024-09-16T03:42:07.931 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.910+0000 7facc5ffb640 10 monclient(hunting): _renew_subs 2024-09-16T03:42:07.931 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.910+0000 7facc5ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:42:07.931 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.910+0000 7faccd824640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-16T03:42:07.931 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.910+0000 7facc67fc640 1 --2- 172.21.15.154:0/597716493 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7facb4003ff0 0x7facb4004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:07.931 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.910+0000 7facc67fc640 10 monclient(hunting): get_auth_request con 0x7facb4003ff0 auth_method 0 2024-09-16T03:42:07.932 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.910+0000 7facc67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:07.932 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.910+0000 7facc67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:07.932 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.910+0000 7facc67fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:42:07.932 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.914+0000 7facc67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:07.932 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.914+0000 7facc67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:07.932 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.914+0000 7facc67fc640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T03:42:07.932 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.914+0000 7facc6ffd640 1 --2- 172.21.15.154:0/597716493 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7facc809fef0 0x7facb40039c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:07.932 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.914+0000 7facc77fe640 1 --2- 172.21.15.154:0/597716493 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7facb4004da0 0x7facb4005220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:07.932 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.914+0000 7facc6ffd640 10 monclient(hunting): get_auth_request con 0x7facc809fef0 auth_method 0 2024-09-16T03:42:07.932 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.914+0000 7facc6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:07.932 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.914+0000 7facc6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:07.932 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.914+0000 7facc6ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:42:07.932 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.914+0000 7facc67fc640 10 monclient(hunting): handle_auth_done global_id 4132 payload 293 2024-09-16T03:42:07.933 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.914+0000 7facc67fc640 1 -- 172.21.15.154:0/597716493 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7facc809fef0 msgr2=0x7facb40039c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:42:07.933 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.914+0000 7facc67fc640 1 --2- 172.21.15.154:0/597716493 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7facc809fef0 0x7facb40039c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:07.933 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.914+0000 7facc67fc640 1 -- 172.21.15.154:0/597716493 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7facb4004da0 msgr2=0x7facb4005220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:42:07.933 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.914+0000 7facc67fc640 1 --2- 172.21.15.154:0/597716493 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7facb4004da0 0x7facb4005220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:07.933 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.914+0000 7facc67fc640 10 monclient: _finish_hunting 0 2024-09-16T03:42:07.933 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.914+0000 7facc67fc640 1 monclient: found mon.b 2024-09-16T03:42:07.933 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.914+0000 7facc67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:42:07.933 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.914+0000 7facc67fc640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2024-09-16T03:42:07.933 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.914+0000 7facc67fc640 1 -- 172.21.15.154:0/597716493 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7facb4003530 con 0x7facb4003ff0 2024-09-16T03:42:07.933 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.914+0000 7facc67fc640 10 monclient: _finish_auth 0 2024-09-16T03:42:07.933 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.914+0000 7facc67fc640 10 monclient: _check_auth_tickets 2024-09-16T03:42:07.933 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.914+0000 7facc67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:42:07.933 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.914+0000 7facc77fe640 10 monclient: get_auth_request con 0x7facb4004da0 auth_method 0 2024-09-16T03:42:07.934 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.914+0000 7facc77fe640 1 --2- 172.21.15.154:0/597716493 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7facb4004da0 0x7facb4005220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T03:42:07.934 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.914+0000 7facc67fc640 1 --2- 172.21.15.154:0/597716493 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7facb4003ff0 0x7facb4004810 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7facb800ba30 tx=0x7facb801c360 comp rx=0 tx=0).ready entity=mon.1 client_cookie=763920e16858a732 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:07.934 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.914+0000 7facc5ffb640 1 -- 172.21.15.154:0/597716493 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facb80aa070 con 0x7facb4003ff0 2024-09-16T03:42:07.934 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.914+0000 7facc5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:07.934 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.914+0000 7facc5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:42:07.934 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.914+0000 7facc5ffb640 10 monclient: dump: 2024-09-16T03:42:07.934 INFO:teuthology.orchestra.run.smithi154.stderr:epoch 1 2024-09-16T03:42:07.934 INFO:teuthology.orchestra.run.smithi154.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:07.934 INFO:teuthology.orchestra.run.smithi154.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:07.934 INFO:teuthology.orchestra.run.smithi154.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:07.934 INFO:teuthology.orchestra.run.smithi154.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:07.934 INFO:teuthology.orchestra.run.smithi154.stderr:election_strategy: 1 2024-09-16T03:42:07.934 INFO:teuthology.orchestra.run.smithi154.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:07.934 INFO:teuthology.orchestra.run.smithi154.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:07.935 INFO:teuthology.orchestra.run.smithi154.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:07.935 INFO:teuthology.orchestra.run.smithi154.stderr: 2024-09-16T03:42:07.935 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.914+0000 7faccd824640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:42:07.935 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.914+0000 7facc5ffb640 1 -- 172.21.15.154:0/597716493 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7facb80a49b0 con 0x7facb4003ff0 2024-09-16T03:42:07.935 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.914+0000 7facc5ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T03:42:07.935 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.914+0000 7faccd824640 10 monclient: get_monmap_and_config success 2024-09-16T03:42:07.935 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.914+0000 7faccd824640 4 set_mon_vals no callback set 2024-09-16T03:42:07.935 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.914+0000 7facc5ffb640 1 -- 172.21.15.154:0/597716493 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facb800b600 con 0x7facb4003ff0 2024-09-16T03:42:07.935 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.914+0000 7facc5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:07.935 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.914+0000 7facc5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:42:07.935 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.914+0000 7facc5ffb640 10 monclient: dump: 2024-09-16T03:42:07.935 INFO:teuthology.orchestra.run.smithi154.stderr:epoch 1 2024-09-16T03:42:07.935 INFO:teuthology.orchestra.run.smithi154.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:07.935 INFO:teuthology.orchestra.run.smithi154.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:07.936 INFO:teuthology.orchestra.run.smithi154.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:07.936 INFO:teuthology.orchestra.run.smithi154.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:07.936 INFO:teuthology.orchestra.run.smithi154.stderr:election_strategy: 1 2024-09-16T03:42:07.936 INFO:teuthology.orchestra.run.smithi154.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:07.936 INFO:teuthology.orchestra.run.smithi154.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:07.936 INFO:teuthology.orchestra.run.smithi154.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:07.936 INFO:teuthology.orchestra.run.smithi154.stderr: 2024-09-16T03:42:07.936 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.914+0000 7faccd824640 10 monclient: shutdown 2024-09-16T03:42:07.936 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.914+0000 7faccd824640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:42:07.936 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.914+0000 7faccd824640 1 -- 172.21.15.154:0/597716493 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7facb4003ff0 msgr2=0x7facb4004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:07.936 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.914+0000 7faccd824640 1 --2- 172.21.15.154:0/597716493 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7facb4003ff0 0x7facb4004810 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7facb800ba30 tx=0x7facb801c360 comp rx=0 tx=0).stop 2024-09-16T03:42:07.936 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.914+0000 7facc77fe640 1 -- 172.21.15.154:0/597716493 reap_dead start 2024-09-16T03:42:07.936 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.914+0000 7faccd824640 1 -- 172.21.15.154:0/597716493 shutdown_connections 2024-09-16T03:42:07.937 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.914+0000 7faccd824640 1 -- 172.21.15.154:0/597716493 >> 172.21.15.154:0/597716493 conn(0x7facc80a3530 msgr2=0x7facc80a3940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:42:07.937 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.914+0000 7faccd824640 1 -- 172.21.15.154:0/597716493 shutdown_connections 2024-09-16T03:42:07.937 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.914+0000 7faccd824640 1 -- 172.21.15.154:0/597716493 wait complete. 2024-09-16T03:42:07.937 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.914+0000 7faccd824640 10 monclient: build_initial_monmap 2024-09-16T03:42:07.937 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.914+0000 7faccd824640 10 monclient: monmap: 2024-09-16T03:42:07.937 INFO:teuthology.orchestra.run.smithi154.stderr:epoch 0 2024-09-16T03:42:07.937 INFO:teuthology.orchestra.run.smithi154.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:42:07.937 INFO:teuthology.orchestra.run.smithi154.stderr:last_changed 0.000000 2024-09-16T03:42:07.937 INFO:teuthology.orchestra.run.smithi154.stderr:created 0.000000 2024-09-16T03:42:07.937 INFO:teuthology.orchestra.run.smithi154.stderr:min_mon_release 0 (unknown) 2024-09-16T03:42:07.937 INFO:teuthology.orchestra.run.smithi154.stderr:election_strategy: 1 2024-09-16T03:42:07.937 INFO:teuthology.orchestra.run.smithi154.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:42:07.937 INFO:teuthology.orchestra.run.smithi154.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:42:07.937 INFO:teuthology.orchestra.run.smithi154.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:42:07.938 INFO:teuthology.orchestra.run.smithi154.stderr: 2024-09-16T03:42:07.938 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.918+0000 7faccd824640 1 librados: starting msgr at 2024-09-16T03:42:07.938 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.918+0000 7faccd824640 1 librados: starting objecter 2024-09-16T03:42:07.938 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.918+0000 7faccd824640 1 Processor -- start 2024-09-16T03:42:07.938 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.918+0000 7faccd824640 1 -- start start 2024-09-16T03:42:07.938 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.918+0000 7faccd824640 1 librados: setting wanted keys 2024-09-16T03:42:07.938 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.918+0000 7faccd824640 1 librados: calling monclient init 2024-09-16T03:42:07.938 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.918+0000 7faccd824640 10 monclient: init 2024-09-16T03:42:07.938 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.918+0000 7faccd824640 10 monclient: _reopen_session rank -1 2024-09-16T03:42:07.938 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.918+0000 7faccd824640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:42:07.938 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.918+0000 7faccd824640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7facb4003ff0 0x7facc813ee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:07.938 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.918+0000 7faccd824640 10 monclient(hunting): picked mon.noname-c con 0x7facb4003ff0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:42:07.938 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.918+0000 7faccd824640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7facc811ea40 0x7facc8134830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:07.938 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.918+0000 7faccd824640 10 monclient(hunting): picked mon.noname-a con 0x7facc811ea40 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:42:07.939 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.918+0000 7faccd824640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7facc8134d70 0x7facc811d080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:07.939 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.918+0000 7faccd824640 10 monclient(hunting): picked mon.noname-b con 0x7facc8134d70 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:42:07.939 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.918+0000 7faccd824640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:07.939 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.918+0000 7faccd824640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7facc8141290 con 0x7facc811ea40 2024-09-16T03:42:07.939 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.918+0000 7faccd824640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:07.939 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.918+0000 7faccd824640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7facc8141110 con 0x7facc8134d70 2024-09-16T03:42:07.939 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.918+0000 7faccd824640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:07.939 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.918+0000 7faccd824640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7facc8141410 con 0x7facb4003ff0 2024-09-16T03:42:07.939 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.918+0000 7faccd824640 10 monclient(hunting): _renew_subs 2024-09-16T03:42:07.939 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.918+0000 7faccd824640 10 monclient(hunting): authenticate will time out at 966.749264s 2024-09-16T03:42:07.939 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.918+0000 7facc77fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7facc8134d70 0x7facc811d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:07.939 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.918+0000 7facc77fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7facc8134d70 0x7facc811d080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.154:50180/0 (socket says 172.21.15.154:50180) 2024-09-16T03:42:07.939 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.918+0000 7facc77fe640 1 -- 172.21.15.154:0/2067794350 learned_addr learned my addr 172.21.15.154:0/2067794350 (peer_addr_for_me v2:172.21.15.154:0/0) 2024-09-16T03:42:07.939 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.918+0000 7facc77fe640 10 monclient(hunting): get_auth_request con 0x7facc8134d70 auth_method 0 2024-09-16T03:42:07.940 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.918+0000 7facc77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:07.940 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.918+0000 7facc77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:07.940 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.918+0000 7facc77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:07.940 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.918+0000 7facc77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:07.940 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.918+0000 7facc77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:07.940 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.918+0000 7facc77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:07.940 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.918+0000 7facc6ffd640 1 --2- 172.21.15.154:0/2067794350 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7facb4003ff0 0x7facc813ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:07.940 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.918+0000 7facc67fc640 1 --2- 172.21.15.154:0/2067794350 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7facc811ea40 0x7facc8134830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:07.940 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.918+0000 7facc6ffd640 10 monclient(hunting): get_auth_request con 0x7facb4003ff0 auth_method 0 2024-09-16T03:42:07.940 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.918+0000 7facc6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:07.940 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.918+0000 7facc6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:07.940 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.918+0000 7facc6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:07.940 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.918+0000 7facc67fc640 10 monclient(hunting): get_auth_request con 0x7facc811ea40 auth_method 0 2024-09-16T03:42:07.941 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.918+0000 7facc67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:07.941 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.918+0000 7facc67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:07.941 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.918+0000 7facc67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:07.941 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.918+0000 7facc77fe640 10 monclient(hunting): handle_auth_done global_id 4135 payload 824 2024-09-16T03:42:07.941 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.918+0000 7facc77fe640 1 -- 172.21.15.154:0/2067794350 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7facb4003ff0 msgr2=0x7facc813ee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:07.941 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.918+0000 7facc77fe640 1 --2- 172.21.15.154:0/2067794350 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7facb4003ff0 0x7facc813ee50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:07.941 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.918+0000 7facc77fe640 1 -- 172.21.15.154:0/2067794350 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7facc811ea40 msgr2=0x7facc8134830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:07.941 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.918+0000 7facc77fe640 1 --2- 172.21.15.154:0/2067794350 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7facc811ea40 0x7facc8134830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:07.941 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.918+0000 7facc77fe640 10 monclient: _finish_hunting 0 2024-09-16T03:42:07.941 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.918+0000 7facc77fe640 1 monclient: found mon.noname-b 2024-09-16T03:42:07.941 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.918+0000 7facc77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:42:07.941 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.918+0000 7facc77fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.154:3300/0 2024-09-16T03:42:07.941 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.918+0000 7facc77fe640 1 -- 172.21.15.154:0/2067794350 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7facc80a4d00 con 0x7facc8134d70 2024-09-16T03:42:07.942 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.918+0000 7facc77fe640 1 --2- 172.21.15.154:0/2067794350 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7facc8134d70 0x7facc811d080 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7facc80a04b0 tx=0x7facc000dc00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:07.942 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.918+0000 7facbffff640 10 client.?.objecter ms_handle_connect 0x7facc8134d70 2024-09-16T03:42:07.942 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.918+0000 7facbffff640 10 client.?.objecter resend_mon_ops 2024-09-16T03:42:07.942 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.918+0000 7facbffff640 1 -- 172.21.15.154:0/2067794350 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facc0006d60 con 0x7facc8134d70 2024-09-16T03:42:07.942 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.918+0000 7facbffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:07.942 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.918+0000 7facbffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:42:07.942 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.918+0000 7facbffff640 10 monclient: dump: 2024-09-16T03:42:07.942 INFO:teuthology.orchestra.run.smithi154.stderr:epoch 1 2024-09-16T03:42:07.942 INFO:teuthology.orchestra.run.smithi154.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:07.942 INFO:teuthology.orchestra.run.smithi154.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:07.943 INFO:teuthology.orchestra.run.smithi154.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:07.943 INFO:teuthology.orchestra.run.smithi154.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:07.943 INFO:teuthology.orchestra.run.smithi154.stderr:election_strategy: 1 2024-09-16T03:42:07.943 INFO:teuthology.orchestra.run.smithi154.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:07.943 INFO:teuthology.orchestra.run.smithi154.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:07.943 INFO:teuthology.orchestra.run.smithi154.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:07.943 INFO:teuthology.orchestra.run.smithi154.stderr: 2024-09-16T03:42:07.943 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.918+0000 7facbffff640 10 monclient: _finish_auth 0 2024-09-16T03:42:07.943 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.918+0000 7facbffff640 10 monclient: _check_auth_tickets 2024-09-16T03:42:07.943 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.918+0000 7facbffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:42:07.943 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.918+0000 7facbffff640 1 -- 172.21.15.154:0/2067794350 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7facc0003980 con 0x7facc8134d70 2024-09-16T03:42:07.943 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.918+0000 7facbffff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:42:07.943 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.918+0000 7facbffff640 1 -- 172.21.15.154:0/2067794350 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facc0003db0 con 0x7facc8134d70 2024-09-16T03:42:07.943 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.918+0000 7facbffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:07.944 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.918+0000 7facbffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:42:07.944 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.918+0000 7facbffff640 10 monclient: dump: 2024-09-16T03:42:07.944 INFO:teuthology.orchestra.run.smithi154.stderr:epoch 1 2024-09-16T03:42:07.944 INFO:teuthology.orchestra.run.smithi154.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:07.944 INFO:teuthology.orchestra.run.smithi154.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:07.944 INFO:teuthology.orchestra.run.smithi154.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:07.944 INFO:teuthology.orchestra.run.smithi154.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:07.944 INFO:teuthology.orchestra.run.smithi154.stderr:election_strategy: 1 2024-09-16T03:42:07.944 INFO:teuthology.orchestra.run.smithi154.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:07.944 INFO:teuthology.orchestra.run.smithi154.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:07.944 INFO:teuthology.orchestra.run.smithi154.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:07.944 INFO:teuthology.orchestra.run.smithi154.stderr: 2024-09-16T03:42:07.944 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.918+0000 7facc57fa640 4 set_mon_vals no callback set 2024-09-16T03:42:07.945 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.918+0000 7faccd824640 5 monclient: authenticate success, global_id 4135 2024-09-16T03:42:07.945 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.918+0000 7faccd824640 10 monclient: _renew_subs 2024-09-16T03:42:07.945 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.918+0000 7faccd824640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2024-09-16T03:42:07.945 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.918+0000 7faccd824640 1 -- 172.21.15.154:0/2067794350 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7facc8073ad0 con 0x7facc8134d70 2024-09-16T03:42:07.945 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.918+0000 7faccd824640 10 client.4135.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:42:07.945 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.918+0000 7faccd824640 10 monclient: _renew_subs 2024-09-16T03:42:07.945 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.918+0000 7faccd824640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2024-09-16T03:42:07.945 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.918+0000 7faccd824640 1 -- 172.21.15.154:0/2067794350 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7facc81201d0 con 0x7facc8134d70 2024-09-16T03:42:07.945 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.918+0000 7faccd824640 1 librados: init done 2024-09-16T03:42:07.945 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.918+0000 7facbffff640 1 -- 172.21.15.154:0/2067794350 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7facc001b020 con 0x7facc8134d70 2024-09-16T03:42:07.945 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.918+0000 7facbffff640 1 -- 172.21.15.154:0/2067794350 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1070+0+0 (secure 0 0 0) 0x7facc000b680 con 0x7facc8134d70 2024-09-16T03:42:07.945 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.918+0000 7facbffff640 10 client.4135.objecter ms_dispatch 0x7facc8000ba0 osd_map(5..5 src has 1..5) 2024-09-16T03:42:07.945 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.918+0000 7facbffff640 3 client.4135.objecter handle_osd_map got epochs [5,5] > 0 2024-09-16T03:42:07.945 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.918+0000 7facbffff640 3 client.4135.objecter handle_osd_map decoding full epoch 5 2024-09-16T03:42:07.946 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.918+0000 7facbffff640 20 client.4135.objecter dump_active .. 0 homeless 2024-09-16T03:42:07.946 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.918+0000 7faccd824640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:42:07.946 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.918+0000 7faccd824640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:42:07.946 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.918+0000 7faccd824640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2024-09-16T03:42:07.946 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.918+0000 7faccd824640 1 -- 172.21.15.154:0/2067794350 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7facc80a07b0 con 0x7facc8134d70 2024-09-16T03:42:07.946 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.922+0000 7facbffff640 1 -- 172.21.15.154:0/2067794350 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (secure 0 0 0) 0x7facc0022070 con 0x7facc8134d70 2024-09-16T03:42:07.946 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.922+0000 7facbffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:42:07.946 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.922+0000 7facbffff640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:42:07.993 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.990+0000 7faccd824640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "4e809f2a-2ffd-4791-9f2d-18f1bfe7d2ea", "id": 3}] 2024-09-16T03:42:07.994 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.990+0000 7faccd824640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "4e809f2a-2ffd-4791-9f2d-18f1bfe7d2ea", "id": 3}] 2024-09-16T03:42:07.994 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.990+0000 7faccd824640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2024-09-16T03:42:07.994 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.990+0000 7faccd824640 1 -- 172.21.15.154:0/2067794350 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "osd new", "uuid": "4e809f2a-2ffd-4791-9f2d-18f1bfe7d2ea", "id": 3} v 0) -- 0x7facc806b3e0 con 0x7facc8134d70 2024-09-16T03:42:07.999 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.998+0000 7facbffff640 1 -- 172.21.15.154:0/2067794350 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "4e809f2a-2ffd-4791-9f2d-18f1bfe7d2ea", "id": 3}]=0 v6) ==== 112+0+2 (secure 0 0 0) 0x7facc000bca0 con 0x7facc8134d70 2024-09-16T03:42:08.000 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.998+0000 7facbffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "4e809f2a-2ffd-4791-9f2d-18f1bfe7d2ea", "id": 3}] 2024-09-16T03:42:08.000 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.998+0000 7facbffff640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:42:08.000 INFO:teuthology.orchestra.run.smithi154.stdout:3 2024-09-16T03:42:08.002 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.998+0000 7faccd824640 10 librados: watch_flush enter 2024-09-16T03:42:08.003 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.998+0000 7faccd824640 10 librados: watch_flush exit 2024-09-16T03:42:08.003 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.998+0000 7faccd824640 20 client.4135.objecter shutdown clearing up homeless session... 2024-09-16T03:42:08.003 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.998+0000 7faccd824640 10 client.4135.objecter successfully canceled tick 2024-09-16T03:42:08.003 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.998+0000 7faccd824640 10 monclient: shutdown 2024-09-16T03:42:08.003 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.998+0000 7faccd824640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:42:08.004 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.998+0000 7faccd824640 1 -- 172.21.15.154:0/2067794350 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7facc8134d70 msgr2=0x7facc811d080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:08.004 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.998+0000 7faccd824640 1 --2- 172.21.15.154:0/2067794350 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7facc8134d70 0x7facc811d080 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7facc80a04b0 tx=0x7facc000dc00 comp rx=0 tx=0).stop 2024-09-16T03:42:08.004 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.998+0000 7faccd824640 1 -- 172.21.15.154:0/2067794350 shutdown_connections 2024-09-16T03:42:08.004 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.998+0000 7faccd824640 1 --2- 172.21.15.154:0/2067794350 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7facc8134d70 0x7facc811d080 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:08.004 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.998+0000 7faccd824640 1 --2- 172.21.15.154:0/2067794350 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7facc811ea40 0x7facc8134830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:08.004 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.998+0000 7faccd824640 1 --2- 172.21.15.154:0/2067794350 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7facb4003ff0 0x7facc813ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:08.004 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.998+0000 7faccd824640 1 -- 172.21.15.154:0/2067794350 >> 172.21.15.154:0/2067794350 conn(0x7facc80a3530 msgr2=0x7facc8071b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:42:08.005 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.998+0000 7faccd824640 1 -- 172.21.15.154:0/2067794350 shutdown_connections 2024-09-16T03:42:08.005 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.998+0000 7faccd824640 1 -- 172.21.15.154:0/2067794350 wait complete. 2024-09-16T03:42:08.005 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:07.998+0000 7faccd824640 1 librados: shutdown 2024-09-16T03:42:08.017 DEBUG:teuthology.orchestra.run.smithi154:> sudo ceph --cluster ceph osd new dcf637ab-5f72-4ed2-a99b-69083f0b6b15 4 2024-09-16T03:42:08.127 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.126+0000 7f06c00b7640 10 monclient: get_monmap_and_config 2024-09-16T03:42:08.127 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.126+0000 7f06c00b7640 10 monclient: build_initial_monmap 2024-09-16T03:42:08.127 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.126+0000 7f06c00b7640 10 monclient: monmap: 2024-09-16T03:42:08.128 INFO:teuthology.orchestra.run.smithi154.stderr:epoch 0 2024-09-16T03:42:08.128 INFO:teuthology.orchestra.run.smithi154.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:08.128 INFO:teuthology.orchestra.run.smithi154.stderr:last_changed 2024-09-16T03:42:08.129726+0000 2024-09-16T03:42:08.128 INFO:teuthology.orchestra.run.smithi154.stderr:created 2024-09-16T03:42:08.129726+0000 2024-09-16T03:42:08.128 INFO:teuthology.orchestra.run.smithi154.stderr:min_mon_release 0 (unknown) 2024-09-16T03:42:08.128 INFO:teuthology.orchestra.run.smithi154.stderr:election_strategy: 1 2024-09-16T03:42:08.128 INFO:teuthology.orchestra.run.smithi154.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:42:08.128 INFO:teuthology.orchestra.run.smithi154.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:42:08.128 INFO:teuthology.orchestra.run.smithi154.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:42:08.128 INFO:teuthology.orchestra.run.smithi154.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:42:08.128 INFO:teuthology.orchestra.run.smithi154.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:42:08.128 INFO:teuthology.orchestra.run.smithi154.stderr: 2024-09-16T03:42:08.134 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.126+0000 7f06c00b7640 1 Processor -- start 2024-09-16T03:42:08.134 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.126+0000 7f06c00b7640 1 -- start start 2024-09-16T03:42:08.134 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.126+0000 7f06c00b7640 10 monclient: init 2024-09-16T03:42:08.134 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.126+0000 7f06c00b7640 10 monclient: _reopen_session rank -1 2024-09-16T03:42:08.134 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.126+0000 7f06c00b7640 10 monclient: _add_conns ranks=[2,1,3,0,4] 2024-09-16T03:42:08.134 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.126+0000 7f06c00b7640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f06b80a35a0 0x7f06b80a3980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:08.134 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.126+0000 7f06c00b7640 10 monclient(hunting): picked mon.noname-b con 0x7f06b80a35a0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:42:08.134 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.126+0000 7f06c00b7640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f06b8072940 addr v1:172.21.15.154:6789/0 2024-09-16T03:42:08.134 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.126+0000 7f06c00b7640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f06b80a3ec0 0x7f06b8135a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:08.134 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.126+0000 7f06c00b7640 10 monclient(hunting): picked mon.noname-a con 0x7f06b80a3ec0 addr v2:172.21.15.42:3300/0 2024-09-16T03:42:08.134 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.126+0000 7f06c00b7640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f06b80731b0 con 0x7f06b8072940 2024-09-16T03:42:08.134 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.126+0000 7f06c00b7640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:08.134 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.126+0000 7f06c00b7640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f06b80716c0 con 0x7f06b80a3ec0 2024-09-16T03:42:08.135 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.126+0000 7f06c00b7640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:08.135 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.126+0000 7f06c00b7640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f06b80719a0 con 0x7f06b80a35a0 2024-09-16T03:42:08.135 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.126+0000 7f06c00b7640 10 monclient(hunting): _renew_subs 2024-09-16T03:42:08.135 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.126+0000 7f06c00b7640 10 monclient(hunting): authenticate will time out at 966.958880s 2024-09-16T03:42:08.135 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.126+0000 7f06bd652640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f06b8072940 0x7f06b8135350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.154:59314/0 (socket says 172.21.15.154:59314) 2024-09-16T03:42:08.135 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.126+0000 7f06bd652640 1 -- 172.21.15.154:0/209948105 learned_addr learned my addr 172.21.15.154:0/209948105 (peer_addr_for_me v1:172.21.15.154:0/0) 2024-09-16T03:42:08.135 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.126+0000 7f06bde53640 1 --2- 172.21.15.154:0/209948105 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f06b80a35a0 0x7f06b80a3980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:08.135 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.126+0000 7f06be654640 1 --2- 172.21.15.154:0/209948105 >> v2:172.21.15.42:3300/0 conn(0x7f06b80a3ec0 0x7f06b8135a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:08.135 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.126+0000 7f06bde53640 10 monclient(hunting): get_auth_request con 0x7f06b80a35a0 auth_method 0 2024-09-16T03:42:08.135 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.126+0000 7f06bde53640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:08.135 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.126+0000 7f06bde53640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:08.135 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.126+0000 7f06bde53640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:08.135 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.126+0000 7f06bce51640 1 -- 172.21.15.154:0/209948105 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1085123471 0 0) 0x7f06b80731b0 con 0x7f06b8072940 2024-09-16T03:42:08.136 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.126+0000 7f06be654640 10 monclient(hunting): get_auth_request con 0x7f06b80a3ec0 auth_method 0 2024-09-16T03:42:08.136 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.126+0000 7f06be654640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:08.136 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.126+0000 7f06be654640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:08.136 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.126+0000 7f06be654640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:08.136 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.126+0000 7f06bce51640 10 monclient(hunting): _negotiate 2024-09-16T03:42:08.136 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.126+0000 7f06bce51640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:08.136 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.126+0000 7f06bce51640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:08.136 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.126+0000 7f06bce51640 10 monclient(hunting): my global_id is 4138 2024-09-16T03:42:08.136 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.126+0000 7f06bce51640 1 -- 172.21.15.154:0/209948105 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f06a4003530 con 0x7f06b8072940 2024-09-16T03:42:08.136 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.126+0000 7f06bde53640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:08.136 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.126+0000 7f06bde53640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:08.136 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.126+0000 7f06bde53640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:08.137 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.126+0000 7f06be654640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:08.137 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.126+0000 7f06be654640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:08.137 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.126+0000 7f06be654640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:08.137 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.126+0000 7f06bce51640 1 -- 172.21.15.154:0/209948105 <== mon.1 v1:172.21.15.154:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3024106866 0 0) 0x7f06a4003530 con 0x7f06b8072940 2024-09-16T03:42:08.137 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.126+0000 7f06bce51640 1 -- 172.21.15.154:0/209948105 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f06b80a35a0 msgr2=0x7f06b80a3980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:42:08.137 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.126+0000 7f06bce51640 1 --2- 172.21.15.154:0/209948105 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f06b80a35a0 0x7f06b80a3980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:08.137 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.126+0000 7f06bce51640 1 -- 172.21.15.154:0/209948105 >> v2:172.21.15.42:3300/0 conn(0x7f06b80a3ec0 msgr2=0x7f06b8135a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:42:08.137 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.126+0000 7f06bce51640 1 --2- 172.21.15.154:0/209948105 >> v2:172.21.15.42:3300/0 conn(0x7f06b80a3ec0 0x7f06b8135a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:08.137 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.126+0000 7f06bce51640 10 monclient: _finish_hunting 0 2024-09-16T03:42:08.137 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.126+0000 7f06bce51640 1 monclient: found mon.noname-c-legacy 2024-09-16T03:42:08.137 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.126+0000 7f06bce51640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:42:08.137 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.126+0000 7f06bce51640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.154:6789/0 2024-09-16T03:42:08.137 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.126+0000 7f06bce51640 1 -- 172.21.15.154:0/209948105 --> v1:172.21.15.154:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06b80a42e0 con 0x7f06b8072940 2024-09-16T03:42:08.138 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.126+0000 7f06bce51640 10 monclient: _finish_auth 0 2024-09-16T03:42:08.138 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.126+0000 7f06bce51640 10 monclient: _check_auth_tickets 2024-09-16T03:42:08.138 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.126+0000 7f06bce51640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:42:08.138 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.126+0000 7f06bce51640 1 -- 172.21.15.154:0/209948105 <== mon.1 v1:172.21.15.154:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2177413734 0 0) 0x7f06a8001680 con 0x7f06b8072940 2024-09-16T03:42:08.138 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.126+0000 7f06bce51640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:08.138 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.126+0000 7f06bce51640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-16T03:42:08.138 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.126+0000 7f06bce51640 10 monclient: dump: 2024-09-16T03:42:08.138 INFO:teuthology.orchestra.run.smithi154.stderr:epoch 1 2024-09-16T03:42:08.138 INFO:teuthology.orchestra.run.smithi154.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:08.138 INFO:teuthology.orchestra.run.smithi154.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:08.138 INFO:teuthology.orchestra.run.smithi154.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:08.138 INFO:teuthology.orchestra.run.smithi154.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:08.138 INFO:teuthology.orchestra.run.smithi154.stderr:election_strategy: 1 2024-09-16T03:42:08.138 INFO:teuthology.orchestra.run.smithi154.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:08.138 INFO:teuthology.orchestra.run.smithi154.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:08.139 INFO:teuthology.orchestra.run.smithi154.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:08.139 INFO:teuthology.orchestra.run.smithi154.stderr: 2024-09-16T03:42:08.139 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.126+0000 7f06bce51640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] but i'm connected to v1:172.21.15.154:6789/0, reconnecting 2024-09-16T03:42:08.139 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.126+0000 7f06bce51640 10 monclient: _reopen_session rank -1 2024-09-16T03:42:08.139 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.126+0000 7f06bce51640 1 -- 172.21.15.154:0/209948105 >> v1:172.21.15.154:6789/0 conn(0x7f06b8072940 legacy=0x7f06b8135350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:08.139 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.126+0000 7f06bce51640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:42:08.139 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.126+0000 7f06bce51640 1 --2- 172.21.15.154:0/209948105 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f06a40039c0 0x7f06a4003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:08.139 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.126+0000 7f06bce51640 10 monclient(hunting): picked mon.b con 0x7f06a40039c0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:42:08.139 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.126+0000 7f06bce51640 1 --2- 172.21.15.154:0/209948105 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f06a4004810 0x7f06a4004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:08.139 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.126+0000 7f06bce51640 10 monclient(hunting): picked mon.a con 0x7f06a4004810 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:42:08.139 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.126+0000 7f06bce51640 1 --2- 172.21.15.154:0/209948105 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f06b80a35a0 0x7f06a40051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:08.139 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.126+0000 7f06bce51640 10 monclient(hunting): picked mon.c con 0x7f06b80a35a0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:42:08.139 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.126+0000 7f06bce51640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:08.140 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.126+0000 7f06bce51640 1 -- 172.21.15.154:0/209948105 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f06b80716c0 con 0x7f06a4004810 2024-09-16T03:42:08.140 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.126+0000 7f06bce51640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:08.140 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.126+0000 7f06bce51640 1 -- 172.21.15.154:0/209948105 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f06b80719a0 con 0x7f06a40039c0 2024-09-16T03:42:08.140 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.126+0000 7f06bce51640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:08.140 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.126+0000 7f06bce51640 1 -- 172.21.15.154:0/209948105 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f06a400b9d0 con 0x7f06b80a35a0 2024-09-16T03:42:08.140 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.126+0000 7f06bce51640 10 monclient(hunting): _renew_subs 2024-09-16T03:42:08.140 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.126+0000 7f06bce51640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:42:08.140 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.126+0000 7f06bde53640 1 --2- 172.21.15.154:0/209948105 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f06a40039c0 0x7f06a4003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:08.140 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.126+0000 7f06bde53640 10 monclient(hunting): get_auth_request con 0x7f06a40039c0 auth_method 0 2024-09-16T03:42:08.140 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.126+0000 7f06bde53640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:08.140 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.126+0000 7f06bde53640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:08.140 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.126+0000 7f06bde53640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:42:08.140 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.130+0000 7f06bde53640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:08.140 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.130+0000 7f06bde53640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:08.141 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.130+0000 7f06bde53640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T03:42:08.141 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.130+0000 7f06be654640 1 --2- 172.21.15.154:0/209948105 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f06b80a35a0 0x7f06a40051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:08.141 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.130+0000 7f06bde53640 10 monclient(hunting): handle_auth_done global_id 4138 payload 293 2024-09-16T03:42:08.141 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.130+0000 7f06bde53640 1 -- 172.21.15.154:0/209948105 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f06b80a35a0 msgr2=0x7f06a40051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:42:08.141 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.130+0000 7f06bde53640 1 --2- 172.21.15.154:0/209948105 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f06b80a35a0 0x7f06a40051d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:08.141 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.130+0000 7f06bd652640 1 --2- 172.21.15.154:0/209948105 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f06a4004810 0x7f06a4004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:08.141 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.130+0000 7f06bde53640 1 -- 172.21.15.154:0/209948105 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f06a4004810 msgr2=0x7f06a4004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:42:08.141 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.130+0000 7f06bde53640 1 --2- 172.21.15.154:0/209948105 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f06a4004810 0x7f06a4004c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:08.141 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.130+0000 7f06bde53640 10 monclient: _finish_hunting 0 2024-09-16T03:42:08.141 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.130+0000 7f06bde53640 1 monclient: found mon.b 2024-09-16T03:42:08.141 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.130+0000 7f06bde53640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:42:08.141 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.130+0000 7f06bde53640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2024-09-16T03:42:08.141 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.130+0000 7f06bde53640 1 -- 172.21.15.154:0/209948105 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06a4003530 con 0x7f06a40039c0 2024-09-16T03:42:08.142 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.130+0000 7f06bde53640 10 monclient: _finish_auth 0 2024-09-16T03:42:08.142 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.130+0000 7f06bde53640 10 monclient: _check_auth_tickets 2024-09-16T03:42:08.142 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.130+0000 7f06bde53640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:42:08.142 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.130+0000 7f06c00b7640 5 monclient: authenticate success, global_id 4138 2024-09-16T03:42:08.142 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.130+0000 7f06c00b7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:42:08.142 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.130+0000 7f06bde53640 1 --2- 172.21.15.154:0/209948105 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f06a40039c0 0x7f06a4003de0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f06b4009ac0 tx=0x7f06b401ca30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aad8d12755b6685b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:08.142 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.130+0000 7f06bce51640 1 -- 172.21.15.154:0/209948105 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06b40b2070 con 0x7f06a40039c0 2024-09-16T03:42:08.142 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.130+0000 7f06bce51640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:08.142 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.130+0000 7f06bce51640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:42:08.142 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.130+0000 7f06bce51640 10 monclient: dump: 2024-09-16T03:42:08.142 INFO:teuthology.orchestra.run.smithi154.stderr:epoch 1 2024-09-16T03:42:08.142 INFO:teuthology.orchestra.run.smithi154.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:08.142 INFO:teuthology.orchestra.run.smithi154.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:08.142 INFO:teuthology.orchestra.run.smithi154.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:08.143 INFO:teuthology.orchestra.run.smithi154.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:08.143 INFO:teuthology.orchestra.run.smithi154.stderr:election_strategy: 1 2024-09-16T03:42:08.143 INFO:teuthology.orchestra.run.smithi154.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:08.143 INFO:teuthology.orchestra.run.smithi154.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:08.143 INFO:teuthology.orchestra.run.smithi154.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:08.143 INFO:teuthology.orchestra.run.smithi154.stderr: 2024-09-16T03:42:08.143 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.130+0000 7f06c00b7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:42:08.143 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.130+0000 7f06bce51640 1 -- 172.21.15.154:0/209948105 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f06b40a5540 con 0x7f06a40039c0 2024-09-16T03:42:08.143 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.130+0000 7f06bce51640 10 monclient: handle_config config(0 keys) 2024-09-16T03:42:08.143 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.130+0000 7f06c00b7640 10 monclient: get_monmap_and_config success 2024-09-16T03:42:08.144 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.130+0000 7f06bce51640 1 -- 172.21.15.154:0/209948105 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06b4009690 con 0x7f06a40039c0 2024-09-16T03:42:08.144 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.130+0000 7f06c00b7640 4 set_mon_vals no callback set 2024-09-16T03:42:08.144 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.130+0000 7f06bce51640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:08.144 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.130+0000 7f06bce51640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:42:08.144 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.130+0000 7f06bce51640 10 monclient: dump: 2024-09-16T03:42:08.144 INFO:teuthology.orchestra.run.smithi154.stderr:epoch 1 2024-09-16T03:42:08.144 INFO:teuthology.orchestra.run.smithi154.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:08.144 INFO:teuthology.orchestra.run.smithi154.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:08.144 INFO:teuthology.orchestra.run.smithi154.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:08.144 INFO:teuthology.orchestra.run.smithi154.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:08.144 INFO:teuthology.orchestra.run.smithi154.stderr:election_strategy: 1 2024-09-16T03:42:08.144 INFO:teuthology.orchestra.run.smithi154.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:08.144 INFO:teuthology.orchestra.run.smithi154.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:08.145 INFO:teuthology.orchestra.run.smithi154.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:08.145 INFO:teuthology.orchestra.run.smithi154.stderr: 2024-09-16T03:42:08.145 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.130+0000 7f06c00b7640 10 monclient: shutdown 2024-09-16T03:42:08.145 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.130+0000 7f06c00b7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:42:08.145 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.130+0000 7f06c00b7640 1 -- 172.21.15.154:0/209948105 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f06a40039c0 msgr2=0x7f06a4003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:08.145 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.130+0000 7f06c00b7640 1 --2- 172.21.15.154:0/209948105 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f06a40039c0 0x7f06a4003de0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f06b4009ac0 tx=0x7f06b401ca30 comp rx=0 tx=0).stop 2024-09-16T03:42:08.145 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.130+0000 7f06be654640 1 -- 172.21.15.154:0/209948105 reap_dead start 2024-09-16T03:42:08.145 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.130+0000 7f06c00b7640 1 -- 172.21.15.154:0/209948105 shutdown_connections 2024-09-16T03:42:08.145 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.130+0000 7f06c00b7640 1 -- 172.21.15.154:0/209948105 >> 172.21.15.154:0/209948105 conn(0x7f06b80a81f0 msgr2=0x7f06b80a4690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:42:08.145 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.130+0000 7f06c00b7640 1 -- 172.21.15.154:0/209948105 shutdown_connections 2024-09-16T03:42:08.145 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.130+0000 7f06c00b7640 1 -- 172.21.15.154:0/209948105 wait complete. 2024-09-16T03:42:08.150 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.130+0000 7f06c00b7640 10 monclient: build_initial_monmap 2024-09-16T03:42:08.150 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.130+0000 7f06c00b7640 10 monclient: monmap: 2024-09-16T03:42:08.150 INFO:teuthology.orchestra.run.smithi154.stderr:epoch 0 2024-09-16T03:42:08.151 INFO:teuthology.orchestra.run.smithi154.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:42:08.151 INFO:teuthology.orchestra.run.smithi154.stderr:last_changed 0.000000 2024-09-16T03:42:08.151 INFO:teuthology.orchestra.run.smithi154.stderr:created 0.000000 2024-09-16T03:42:08.151 INFO:teuthology.orchestra.run.smithi154.stderr:min_mon_release 0 (unknown) 2024-09-16T03:42:08.151 INFO:teuthology.orchestra.run.smithi154.stderr:election_strategy: 1 2024-09-16T03:42:08.151 INFO:teuthology.orchestra.run.smithi154.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:42:08.151 INFO:teuthology.orchestra.run.smithi154.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:42:08.151 INFO:teuthology.orchestra.run.smithi154.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:42:08.151 INFO:teuthology.orchestra.run.smithi154.stderr: 2024-09-16T03:42:08.151 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.134+0000 7f06c00b7640 1 librados: starting msgr at 2024-09-16T03:42:08.151 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.134+0000 7f06c00b7640 1 librados: starting objecter 2024-09-16T03:42:08.151 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.134+0000 7f06c00b7640 1 Processor -- start 2024-09-16T03:42:08.151 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.134+0000 7f06c00b7640 1 -- start start 2024-09-16T03:42:08.151 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.134+0000 7f06c00b7640 1 librados: setting wanted keys 2024-09-16T03:42:08.152 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.134+0000 7f06c00b7640 1 librados: calling monclient init 2024-09-16T03:42:08.152 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.134+0000 7f06c00b7640 10 monclient: init 2024-09-16T03:42:08.152 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.134+0000 7f06c00b7640 10 monclient: _reopen_session rank -1 2024-09-16T03:42:08.152 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.134+0000 7f06c00b7640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:42:08.152 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.134+0000 7f06c00b7640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f06a40039c0 0x7f06b81ca020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:08.152 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.134+0000 7f06c00b7640 10 monclient(hunting): picked mon.noname-c con 0x7f06a40039c0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:42:08.152 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.134+0000 7f06c00b7640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f06b81349d0 0x7f06b8134df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:08.152 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.134+0000 7f06c00b7640 10 monclient(hunting): picked mon.noname-b con 0x7f06b81349d0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:42:08.152 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.134+0000 7f06c00b7640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f06b811e450 0x7f06b811e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:08.152 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.134+0000 7f06c00b7640 10 monclient(hunting): picked mon.noname-a con 0x7f06b811e450 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:42:08.152 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.134+0000 7f06c00b7640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:08.152 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.134+0000 7f06c00b7640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f06b80a5180 con 0x7f06b811e450 2024-09-16T03:42:08.152 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.134+0000 7f06c00b7640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:08.152 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.134+0000 7f06c00b7640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f06b80a5000 con 0x7f06b81349d0 2024-09-16T03:42:08.153 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.134+0000 7f06c00b7640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:08.153 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.134+0000 7f06c00b7640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f06b8135fa0 con 0x7f06a40039c0 2024-09-16T03:42:08.153 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.134+0000 7f06c00b7640 10 monclient(hunting): _renew_subs 2024-09-16T03:42:08.153 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.134+0000 7f06c00b7640 10 monclient(hunting): authenticate will time out at 966.964848s 2024-09-16T03:42:08.153 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.134+0000 7f06bd652640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f06b81349d0 0x7f06b8134df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:08.153 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.134+0000 7f06bd652640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f06b81349d0 0x7f06b8134df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.154:50212/0 (socket says 172.21.15.154:50212) 2024-09-16T03:42:08.153 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.134+0000 7f06bd652640 1 -- 172.21.15.154:0/3337699570 learned_addr learned my addr 172.21.15.154:0/3337699570 (peer_addr_for_me v2:172.21.15.154:0/0) 2024-09-16T03:42:08.153 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.134+0000 7f06bd652640 10 monclient(hunting): get_auth_request con 0x7f06b81349d0 auth_method 0 2024-09-16T03:42:08.153 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.134+0000 7f06bd652640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:08.153 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.134+0000 7f06bd652640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:08.153 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.134+0000 7f06bd652640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:08.153 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.134+0000 7f06bd652640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:08.153 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.134+0000 7f06bd652640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:08.153 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.134+0000 7f06bd652640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:08.154 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.134+0000 7f06bde53640 1 --2- 172.21.15.154:0/3337699570 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f06a40039c0 0x7f06b81ca020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:08.154 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.134+0000 7f06be654640 1 --2- 172.21.15.154:0/3337699570 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f06b811e450 0x7f06b811e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:08.154 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.134+0000 7f06be654640 10 monclient(hunting): get_auth_request con 0x7f06b811e450 auth_method 0 2024-09-16T03:42:08.154 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.134+0000 7f06be654640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:08.154 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.134+0000 7f06be654640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:08.154 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.134+0000 7f06be654640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:08.154 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.134+0000 7f06bde53640 10 monclient(hunting): get_auth_request con 0x7f06a40039c0 auth_method 0 2024-09-16T03:42:08.154 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.134+0000 7f06bde53640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:08.154 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.134+0000 7f06bde53640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:08.154 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.134+0000 7f06bde53640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:08.154 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.134+0000 7f06bd652640 10 monclient(hunting): handle_auth_done global_id 4141 payload 824 2024-09-16T03:42:08.154 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.134+0000 7f06bd652640 1 -- 172.21.15.154:0/3337699570 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f06a40039c0 msgr2=0x7f06b81ca020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:08.154 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.134+0000 7f06bd652640 1 --2- 172.21.15.154:0/3337699570 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f06a40039c0 0x7f06b81ca020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:08.154 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.134+0000 7f06bd652640 1 -- 172.21.15.154:0/3337699570 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f06b811e450 msgr2=0x7f06b811e8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:08.155 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.134+0000 7f06bd652640 1 --2- 172.21.15.154:0/3337699570 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f06b811e450 0x7f06b811e8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:08.155 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.134+0000 7f06bd652640 10 monclient: _finish_hunting 0 2024-09-16T03:42:08.155 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.134+0000 7f06bd652640 1 monclient: found mon.noname-b 2024-09-16T03:42:08.155 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.134+0000 7f06bd652640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:42:08.155 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.134+0000 7f06bd652640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.154:3300/0 2024-09-16T03:42:08.155 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.134+0000 7f06bd652640 1 -- 172.21.15.154:0/3337699570 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06b8121dd0 con 0x7f06b81349d0 2024-09-16T03:42:08.155 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.134+0000 7f06bd652640 1 --2- 172.21.15.154:0/3337699570 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f06b81349d0 0x7f06b8134df0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f06a8010b10 tx=0x7f06a800e100 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:08.155 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.134+0000 7f06ae7fc640 10 client.?.objecter ms_handle_connect 0x7f06b81349d0 2024-09-16T03:42:08.155 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.134+0000 7f06ae7fc640 10 client.?.objecter resend_mon_ops 2024-09-16T03:42:08.155 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.134+0000 7f06ae7fc640 1 -- 172.21.15.154:0/3337699570 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06a801b070 con 0x7f06b81349d0 2024-09-16T03:42:08.155 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.134+0000 7f06ae7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:08.155 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.134+0000 7f06ae7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:42:08.155 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.134+0000 7f06ae7fc640 10 monclient: dump: 2024-09-16T03:42:08.155 INFO:teuthology.orchestra.run.smithi154.stderr:epoch 1 2024-09-16T03:42:08.156 INFO:teuthology.orchestra.run.smithi154.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:08.156 INFO:teuthology.orchestra.run.smithi154.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:08.156 INFO:teuthology.orchestra.run.smithi154.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:08.156 INFO:teuthology.orchestra.run.smithi154.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:08.156 INFO:teuthology.orchestra.run.smithi154.stderr:election_strategy: 1 2024-09-16T03:42:08.156 INFO:teuthology.orchestra.run.smithi154.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:08.156 INFO:teuthology.orchestra.run.smithi154.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:08.156 INFO:teuthology.orchestra.run.smithi154.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:08.156 INFO:teuthology.orchestra.run.smithi154.stderr: 2024-09-16T03:42:08.156 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.134+0000 7f06ae7fc640 10 monclient: _finish_auth 0 2024-09-16T03:42:08.156 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.134+0000 7f06ae7fc640 10 monclient: _check_auth_tickets 2024-09-16T03:42:08.156 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.134+0000 7f06ae7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:42:08.156 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.134+0000 7f06ae7fc640 1 -- 172.21.15.154:0/3337699570 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f06a8011040 con 0x7f06b81349d0 2024-09-16T03:42:08.157 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.134+0000 7f06ae7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:42:08.157 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.134+0000 7f06ae7fc640 1 -- 172.21.15.154:0/3337699570 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06a80143a0 con 0x7f06b81349d0 2024-09-16T03:42:08.157 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.134+0000 7f06ae7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:08.157 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.134+0000 7f06ae7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:42:08.157 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.134+0000 7f06ae7fc640 10 monclient: dump: 2024-09-16T03:42:08.157 INFO:teuthology.orchestra.run.smithi154.stderr:epoch 1 2024-09-16T03:42:08.157 INFO:teuthology.orchestra.run.smithi154.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:08.157 INFO:teuthology.orchestra.run.smithi154.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:08.157 INFO:teuthology.orchestra.run.smithi154.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:08.157 INFO:teuthology.orchestra.run.smithi154.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:08.157 INFO:teuthology.orchestra.run.smithi154.stderr:election_strategy: 1 2024-09-16T03:42:08.157 INFO:teuthology.orchestra.run.smithi154.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:08.157 INFO:teuthology.orchestra.run.smithi154.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:08.157 INFO:teuthology.orchestra.run.smithi154.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:08.158 INFO:teuthology.orchestra.run.smithi154.stderr: 2024-09-16T03:42:08.158 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.134+0000 7f06c00b7640 5 monclient: authenticate success, global_id 4141 2024-09-16T03:42:08.158 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.134+0000 7f06affff640 4 set_mon_vals no callback set 2024-09-16T03:42:08.158 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.134+0000 7f06c00b7640 10 monclient: _renew_subs 2024-09-16T03:42:08.158 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.134+0000 7f06c00b7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2024-09-16T03:42:08.158 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.134+0000 7f06c00b7640 1 -- 172.21.15.154:0/3337699570 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06b809bed0 con 0x7f06b81349d0 2024-09-16T03:42:08.158 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.134+0000 7f06c00b7640 10 client.4141.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:42:08.158 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.134+0000 7f06c00b7640 10 monclient: _renew_subs 2024-09-16T03:42:08.158 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.134+0000 7f06c00b7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2024-09-16T03:42:08.158 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.134+0000 7f06c00b7640 1 -- 172.21.15.154:0/3337699570 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06b809ba80 con 0x7f06b81349d0 2024-09-16T03:42:08.158 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.134+0000 7f06c00b7640 1 librados: init done 2024-09-16T03:42:08.158 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.134+0000 7f06ae7fc640 1 -- 172.21.15.154:0/3337699570 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f06a8014540 con 0x7f06b81349d0 2024-09-16T03:42:08.158 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.134+0000 7f06ae7fc640 1 -- 172.21.15.154:0/3337699570 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(6..6 src has 1..6) ==== 1185+0+0 (secure 0 0 0) 0x7f06a8015d70 con 0x7f06b81349d0 2024-09-16T03:42:08.158 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.134+0000 7f06ae7fc640 10 client.4141.objecter ms_dispatch 0x7f06b8000ba0 osd_map(6..6 src has 1..6) 2024-09-16T03:42:08.159 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.134+0000 7f06ae7fc640 3 client.4141.objecter handle_osd_map got epochs [6,6] > 0 2024-09-16T03:42:08.159 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.134+0000 7f06ae7fc640 3 client.4141.objecter handle_osd_map decoding full epoch 6 2024-09-16T03:42:08.159 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.134+0000 7f06ae7fc640 20 client.4141.objecter dump_active .. 0 homeless 2024-09-16T03:42:08.159 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.134+0000 7f06c00b7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:42:08.159 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.134+0000 7f06c00b7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:42:08.159 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.134+0000 7f06c00b7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2024-09-16T03:42:08.159 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.134+0000 7f06c00b7640 1 -- 172.21.15.154:0/3337699570 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f06b80a3980 con 0x7f06b81349d0 2024-09-16T03:42:08.159 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.138+0000 7f06ae7fc640 1 -- 172.21.15.154:0/3337699570 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (secure 0 0 0) 0x7f06a8024070 con 0x7f06b81349d0 2024-09-16T03:42:08.159 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.138+0000 7f06ae7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:42:08.159 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.138+0000 7f06ae7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:42:08.217 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.214+0000 7f06c00b7640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "dcf637ab-5f72-4ed2-a99b-69083f0b6b15", "id": 4}] 2024-09-16T03:42:08.217 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.214+0000 7f06c00b7640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "dcf637ab-5f72-4ed2-a99b-69083f0b6b15", "id": 4}] 2024-09-16T03:42:08.217 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.214+0000 7f06c00b7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2024-09-16T03:42:08.217 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.214+0000 7f06c00b7640 1 -- 172.21.15.154:0/3337699570 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "osd new", "uuid": "dcf637ab-5f72-4ed2-a99b-69083f0b6b15", "id": 4} v 0) -- 0x7f06b8120980 con 0x7f06b81349d0 2024-09-16T03:42:08.221 INFO:tasks.ceph.mgr.x.smithi154.stderr:2024-09-16T03:42:08.218+0000 7f8fbbf11180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-16T03:42:08.224 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.222+0000 7f06ae7fc640 1 -- 172.21.15.154:0/3337699570 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "dcf637ab-5f72-4ed2-a99b-69083f0b6b15", "id": 4}]=0 v7) ==== 112+0+2 (secure 0 0 0) 0x7f06a8015860 con 0x7f06b81349d0 2024-09-16T03:42:08.224 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.222+0000 7f06ae7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "dcf637ab-5f72-4ed2-a99b-69083f0b6b15", "id": 4}] 2024-09-16T03:42:08.224 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.222+0000 7f06ae7fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:42:08.224 INFO:teuthology.orchestra.run.smithi154.stdout:4 2024-09-16T03:42:08.226 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.222+0000 7f06c00b7640 10 librados: watch_flush enter 2024-09-16T03:42:08.226 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.222+0000 7f06c00b7640 10 librados: watch_flush exit 2024-09-16T03:42:08.226 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.222+0000 7f06c00b7640 20 client.4141.objecter shutdown clearing up homeless session... 2024-09-16T03:42:08.226 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.222+0000 7f06c00b7640 10 client.4141.objecter successfully canceled tick 2024-09-16T03:42:08.226 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.222+0000 7f06c00b7640 10 monclient: shutdown 2024-09-16T03:42:08.226 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.222+0000 7f06c00b7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:42:08.226 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.222+0000 7f06c00b7640 1 -- 172.21.15.154:0/3337699570 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f06b81349d0 msgr2=0x7f06b8134df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:08.226 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.222+0000 7f06c00b7640 1 --2- 172.21.15.154:0/3337699570 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f06b81349d0 0x7f06b8134df0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f06a8010b10 tx=0x7f06a800e100 comp rx=0 tx=0).stop 2024-09-16T03:42:08.226 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.222+0000 7f06c00b7640 1 -- 172.21.15.154:0/3337699570 shutdown_connections 2024-09-16T03:42:08.226 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.222+0000 7f06c00b7640 1 --2- 172.21.15.154:0/3337699570 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f06b811e450 0x7f06b811e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:08.226 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.222+0000 7f06c00b7640 1 --2- 172.21.15.154:0/3337699570 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f06b81349d0 0x7f06b8134df0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:08.227 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.222+0000 7f06c00b7640 1 --2- 172.21.15.154:0/3337699570 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f06a40039c0 0x7f06b81ca020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:08.227 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.222+0000 7f06c00b7640 1 -- 172.21.15.154:0/3337699570 >> 172.21.15.154:0/3337699570 conn(0x7f06b80a81f0 msgr2=0x7f06b81355e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:42:08.227 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.222+0000 7f06c00b7640 1 -- 172.21.15.154:0/3337699570 shutdown_connections 2024-09-16T03:42:08.227 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.222+0000 7f06c00b7640 1 -- 172.21.15.154:0/3337699570 wait complete. 2024-09-16T03:42:08.227 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.222+0000 7f06c00b7640 1 librados: shutdown 2024-09-16T03:42:08.241 DEBUG:teuthology.orchestra.run.smithi154:> sudo ceph --cluster ceph osd new 83c05e44-861b-4bb2-9657-19196e5ac1f1 5 2024-09-16T03:42:08.278 INFO:tasks.ceph.mgr.x.smithi154.stderr:2024-09-16T03:42:08.274+0000 7f8fbbf11180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-16T03:42:08.332 INFO:tasks.ceph.mgr.x.smithi154.stderr:2024-09-16T03:42:08.330+0000 7f8fbbf11180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-16T03:42:08.357 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.354+0000 7f0e98dfb640 10 monclient: get_monmap_and_config 2024-09-16T03:42:08.357 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.354+0000 7f0e98dfb640 10 monclient: build_initial_monmap 2024-09-16T03:42:08.357 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.354+0000 7f0e98dfb640 10 monclient: monmap: 2024-09-16T03:42:08.357 INFO:teuthology.orchestra.run.smithi154.stderr:epoch 0 2024-09-16T03:42:08.357 INFO:teuthology.orchestra.run.smithi154.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:08.357 INFO:teuthology.orchestra.run.smithi154.stderr:last_changed 2024-09-16T03:42:08.358957+0000 2024-09-16T03:42:08.357 INFO:teuthology.orchestra.run.smithi154.stderr:created 2024-09-16T03:42:08.358957+0000 2024-09-16T03:42:08.357 INFO:teuthology.orchestra.run.smithi154.stderr:min_mon_release 0 (unknown) 2024-09-16T03:42:08.357 INFO:teuthology.orchestra.run.smithi154.stderr:election_strategy: 1 2024-09-16T03:42:08.357 INFO:teuthology.orchestra.run.smithi154.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:42:08.358 INFO:teuthology.orchestra.run.smithi154.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:42:08.358 INFO:teuthology.orchestra.run.smithi154.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:42:08.358 INFO:teuthology.orchestra.run.smithi154.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:42:08.358 INFO:teuthology.orchestra.run.smithi154.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:42:08.358 INFO:teuthology.orchestra.run.smithi154.stderr: 2024-09-16T03:42:08.364 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.354+0000 7f0e98dfb640 1 Processor -- start 2024-09-16T03:42:08.364 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.354+0000 7f0e98dfb640 1 -- start start 2024-09-16T03:42:08.364 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.354+0000 7f0e98dfb640 10 monclient: init 2024-09-16T03:42:08.364 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.354+0000 7f0e98dfb640 10 monclient: _reopen_session rank -1 2024-09-16T03:42:08.364 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.354+0000 7f0e98dfb640 10 monclient: _add_conns ranks=[2,1,0,3,4] 2024-09-16T03:42:08.364 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.354+0000 7f0e98dfb640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0e9406e7f0 0x7f0e940a4a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:08.364 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.354+0000 7f0e98dfb640 10 monclient(hunting): picked mon.noname-b con 0x7f0e9406e7f0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:42:08.364 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.354+0000 7f0e98dfb640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0e940a5a00 addr v1:172.21.15.154:6789/0 2024-09-16T03:42:08.364 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.358+0000 7f0e98dfb640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0e940a4fd0 addr v1:172.21.15.42:6789/0 2024-09-16T03:42:08.364 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.358+0000 7f0e98dfb640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0e940a40d0 con 0x7f0e940a4fd0 2024-09-16T03:42:08.364 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.358+0000 7f0e98dfb640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0e94137da0 con 0x7f0e940a5a00 2024-09-16T03:42:08.365 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.358+0000 7f0e98dfb640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:08.365 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.358+0000 7f0e98dfb640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f0e9406ebd0 con 0x7f0e9406e7f0 2024-09-16T03:42:08.365 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.358+0000 7f0e98dfb640 10 monclient(hunting): _renew_subs 2024-09-16T03:42:08.365 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.358+0000 7f0e98dfb640 10 monclient(hunting): authenticate will time out at 967.188078s 2024-09-16T03:42:08.365 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.358+0000 7f0e91d9b640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f0e940a5a00 0x7f0e94136f80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.154:59330/0 (socket says 172.21.15.154:59330) 2024-09-16T03:42:08.365 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.358+0000 7f0e91d9b640 1 -- 172.21.15.154:0/870029975 learned_addr learned my addr 172.21.15.154:0/870029975 (peer_addr_for_me v1:172.21.15.154:0/0) 2024-09-16T03:42:08.365 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.358+0000 7f0e9259c640 1 --2- 172.21.15.154:0/870029975 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0e9406e7f0 0x7f0e940a4a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:08.365 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.358+0000 7f0e9159a640 1 -- 172.21.15.154:0/870029975 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1017208725 0 0) 0x7f0e94137da0 con 0x7f0e940a5a00 2024-09-16T03:42:08.366 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.358+0000 7f0e9159a640 10 monclient(hunting): _negotiate 2024-09-16T03:42:08.366 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.358+0000 7f0e9159a640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:08.366 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.358+0000 7f0e9159a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:08.366 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.358+0000 7f0e9159a640 10 monclient(hunting): my global_id is 4144 2024-09-16T03:42:08.366 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.358+0000 7f0e9159a640 1 -- 172.21.15.154:0/870029975 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0e7c003530 con 0x7f0e940a5a00 2024-09-16T03:42:08.366 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.358+0000 7f0e9259c640 10 monclient(hunting): get_auth_request con 0x7f0e9406e7f0 auth_method 0 2024-09-16T03:42:08.366 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.358+0000 7f0e9259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:08.366 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.358+0000 7f0e9259c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:08.367 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.358+0000 7f0e9259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:08.367 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.358+0000 7f0e9259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:08.367 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.358+0000 7f0e9259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:08.367 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.358+0000 7f0e9259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:08.367 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.358+0000 7f0e9159a640 1 -- 172.21.15.154:0/870029975 <== mon.1 v1:172.21.15.154:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1630286825 0 0) 0x7f0e7c003530 con 0x7f0e940a5a00 2024-09-16T03:42:08.367 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.358+0000 7f0e9159a640 1 -- 172.21.15.154:0/870029975 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0e9406e7f0 msgr2=0x7f0e940a4a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:42:08.367 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.358+0000 7f0e9159a640 1 --2- 172.21.15.154:0/870029975 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0e9406e7f0 0x7f0e940a4a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:08.367 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.358+0000 7f0e9159a640 1 -- 172.21.15.154:0/870029975 >> v1:172.21.15.42:6789/0 conn(0x7f0e940a4fd0 legacy=0x7f0e94137690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:08.367 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.358+0000 7f0e9159a640 10 monclient: _finish_hunting 0 2024-09-16T03:42:08.367 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.358+0000 7f0e9159a640 1 monclient: found mon.noname-c-legacy 2024-09-16T03:42:08.367 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.358+0000 7f0e9159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:42:08.367 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.358+0000 7f0e9159a640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.154:6789/0 2024-09-16T03:42:08.368 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.358+0000 7f0e9159a640 1 -- 172.21.15.154:0/870029975 --> v1:172.21.15.154:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e940a5480 con 0x7f0e940a5a00 2024-09-16T03:42:08.368 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.358+0000 7f0e9159a640 10 monclient: _finish_auth 0 2024-09-16T03:42:08.368 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.358+0000 7f0e9159a640 10 monclient: _check_auth_tickets 2024-09-16T03:42:08.368 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.358+0000 7f0e9159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:42:08.368 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.358+0000 7f0e9159a640 1 -- 172.21.15.154:0/870029975 <== mon.1 v1:172.21.15.154:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2177413734 0 0) 0x7f0e88001680 con 0x7f0e940a5a00 2024-09-16T03:42:08.368 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.358+0000 7f0e9159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:08.368 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.358+0000 7f0e9159a640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-16T03:42:08.368 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.358+0000 7f0e9159a640 10 monclient: dump: 2024-09-16T03:42:08.368 INFO:teuthology.orchestra.run.smithi154.stderr:epoch 1 2024-09-16T03:42:08.369 INFO:teuthology.orchestra.run.smithi154.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:08.369 INFO:teuthology.orchestra.run.smithi154.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:08.369 INFO:teuthology.orchestra.run.smithi154.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:08.369 INFO:teuthology.orchestra.run.smithi154.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:08.369 INFO:teuthology.orchestra.run.smithi154.stderr:election_strategy: 1 2024-09-16T03:42:08.370 INFO:teuthology.orchestra.run.smithi154.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:08.370 INFO:teuthology.orchestra.run.smithi154.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:08.370 INFO:teuthology.orchestra.run.smithi154.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:08.370 INFO:teuthology.orchestra.run.smithi154.stderr: 2024-09-16T03:42:08.370 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.358+0000 7f0e9159a640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] but i'm connected to v1:172.21.15.154:6789/0, reconnecting 2024-09-16T03:42:08.370 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.358+0000 7f0e9159a640 10 monclient: _reopen_session rank -1 2024-09-16T03:42:08.370 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.358+0000 7f0e9159a640 1 -- 172.21.15.154:0/870029975 >> v1:172.21.15.154:6789/0 conn(0x7f0e940a5a00 legacy=0x7f0e94136f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:08.370 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.358+0000 7f0e9159a640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T03:42:08.370 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.358+0000 7f0e9159a640 1 --2- 172.21.15.154:0/870029975 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0e7c0039c0 0x7f0e7c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:08.370 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.358+0000 7f0e9159a640 10 monclient(hunting): picked mon.a con 0x7f0e7c0039c0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:42:08.370 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.358+0000 7f0e9159a640 1 --2- 172.21.15.154:0/870029975 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0e7c004810 0x7f0e7c004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:08.370 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.358+0000 7f0e9159a640 10 monclient(hunting): picked mon.b con 0x7f0e7c004810 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:42:08.370 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.358+0000 7f0e9159a640 1 --2- 172.21.15.154:0/870029975 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0e9406e7f0 0x7f0e7c005280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:08.371 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.358+0000 7f0e9159a640 10 monclient(hunting): picked mon.c con 0x7f0e9406e7f0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:42:08.371 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.358+0000 7f0e9159a640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:08.371 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.358+0000 7f0e9159a640 1 -- 172.21.15.154:0/870029975 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f0e9406ebd0 con 0x7f0e7c0039c0 2024-09-16T03:42:08.371 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.358+0000 7f0e9159a640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:08.371 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.358+0000 7f0e9159a640 1 -- 172.21.15.154:0/870029975 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f0e7c00bb90 con 0x7f0e7c004810 2024-09-16T03:42:08.371 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.358+0000 7f0e9159a640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:08.371 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.358+0000 7f0e9159a640 1 -- 172.21.15.154:0/870029975 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f0e7c00bd90 con 0x7f0e9406e7f0 2024-09-16T03:42:08.371 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.358+0000 7f0e9159a640 10 monclient(hunting): _renew_subs 2024-09-16T03:42:08.371 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.358+0000 7f0e9159a640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:42:08.371 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.358+0000 7f0e91d9b640 1 --2- 172.21.15.154:0/870029975 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0e7c004810 0x7f0e7c004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:08.371 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.358+0000 7f0e91d9b640 10 monclient(hunting): get_auth_request con 0x7f0e7c004810 auth_method 0 2024-09-16T03:42:08.371 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.358+0000 7f0e91d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:08.371 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.358+0000 7f0e91d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:08.371 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.358+0000 7f0e91d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:42:08.372 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.358+0000 7f0e91d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:08.372 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.358+0000 7f0e91d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:08.372 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.358+0000 7f0e91d9b640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T03:42:08.372 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.358+0000 7f0e91d9b640 10 monclient(hunting): handle_auth_done global_id 4144 payload 293 2024-09-16T03:42:08.372 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.358+0000 7f0e91d9b640 1 -- 172.21.15.154:0/870029975 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0e9406e7f0 msgr2=0x7f0e7c005280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:42:08.372 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.358+0000 7f0e91d9b640 1 --2- 172.21.15.154:0/870029975 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0e9406e7f0 0x7f0e7c005280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:08.372 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.358+0000 7f0e91d9b640 1 -- 172.21.15.154:0/870029975 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0e7c0039c0 msgr2=0x7f0e7c003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:42:08.372 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.358+0000 7f0e9259c640 1 --2- 172.21.15.154:0/870029975 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0e7c0039c0 0x7f0e7c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:08.372 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.358+0000 7f0e91d9b640 1 --2- 172.21.15.154:0/870029975 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0e7c0039c0 0x7f0e7c003de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:08.372 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.358+0000 7f0e91d9b640 10 monclient: _finish_hunting 0 2024-09-16T03:42:08.372 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.358+0000 7f0e91d9b640 1 monclient: found mon.b 2024-09-16T03:42:08.372 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.358+0000 7f0e91d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:42:08.372 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.358+0000 7f0e91d9b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2024-09-16T03:42:08.373 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.358+0000 7f0e91d9b640 1 -- 172.21.15.154:0/870029975 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e7c003530 con 0x7f0e7c004810 2024-09-16T03:42:08.373 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.358+0000 7f0e91d9b640 10 monclient: _finish_auth 0 2024-09-16T03:42:08.373 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.358+0000 7f0e91d9b640 10 monclient: _check_auth_tickets 2024-09-16T03:42:08.373 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.358+0000 7f0e91d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:42:08.373 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.358+0000 7f0e98dfb640 5 monclient: authenticate success, global_id 4144 2024-09-16T03:42:08.373 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.358+0000 7f0e98dfb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:42:08.373 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.358+0000 7f0e91d9b640 1 --2- 172.21.15.154:0/870029975 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0e7c004810 0x7f0e7c004c90 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f0e88001060 tx=0x7f0e8801cc10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b89ad16a95ecd2d5 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:08.373 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.358+0000 7f0e9159a640 1 -- 172.21.15.154:0/870029975 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e88007d40 con 0x7f0e7c004810 2024-09-16T03:42:08.373 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.358+0000 7f0e9159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:08.373 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.358+0000 7f0e9159a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:42:08.373 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.358+0000 7f0e9159a640 10 monclient: dump: 2024-09-16T03:42:08.373 INFO:teuthology.orchestra.run.smithi154.stderr:epoch 1 2024-09-16T03:42:08.373 INFO:teuthology.orchestra.run.smithi154.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:08.373 INFO:teuthology.orchestra.run.smithi154.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:08.374 INFO:teuthology.orchestra.run.smithi154.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:08.374 INFO:teuthology.orchestra.run.smithi154.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:08.374 INFO:teuthology.orchestra.run.smithi154.stderr:election_strategy: 1 2024-09-16T03:42:08.374 INFO:teuthology.orchestra.run.smithi154.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:08.374 INFO:teuthology.orchestra.run.smithi154.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:08.374 INFO:teuthology.orchestra.run.smithi154.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:08.374 INFO:teuthology.orchestra.run.smithi154.stderr: 2024-09-16T03:42:08.374 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.358+0000 7f0e98dfb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:42:08.374 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.358+0000 7f0e9159a640 1 -- 172.21.15.154:0/870029975 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0e880a9020 con 0x7f0e7c004810 2024-09-16T03:42:08.374 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.358+0000 7f0e9159a640 10 monclient: handle_config config(0 keys) 2024-09-16T03:42:08.374 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.358+0000 7f0e98dfb640 10 monclient: get_monmap_and_config success 2024-09-16T03:42:08.374 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.358+0000 7f0e9159a640 1 -- 172.21.15.154:0/870029975 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e88005b00 con 0x7f0e7c004810 2024-09-16T03:42:08.374 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.358+0000 7f0e98dfb640 4 set_mon_vals no callback set 2024-09-16T03:42:08.375 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.358+0000 7f0e9159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:08.375 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.358+0000 7f0e9159a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:42:08.375 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.358+0000 7f0e9159a640 10 monclient: dump: 2024-09-16T03:42:08.375 INFO:teuthology.orchestra.run.smithi154.stderr:epoch 1 2024-09-16T03:42:08.375 INFO:teuthology.orchestra.run.smithi154.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:08.375 INFO:teuthology.orchestra.run.smithi154.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:08.375 INFO:teuthology.orchestra.run.smithi154.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:08.375 INFO:teuthology.orchestra.run.smithi154.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:08.375 INFO:teuthology.orchestra.run.smithi154.stderr:election_strategy: 1 2024-09-16T03:42:08.375 INFO:teuthology.orchestra.run.smithi154.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:08.375 INFO:teuthology.orchestra.run.smithi154.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:08.375 INFO:teuthology.orchestra.run.smithi154.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:08.375 INFO:teuthology.orchestra.run.smithi154.stderr: 2024-09-16T03:42:08.375 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.362+0000 7f0e98dfb640 10 monclient: shutdown 2024-09-16T03:42:08.375 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.362+0000 7f0e98dfb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:42:08.376 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.362+0000 7f0e98dfb640 1 -- 172.21.15.154:0/870029975 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0e7c004810 msgr2=0x7f0e7c004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:08.376 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.362+0000 7f0e98dfb640 1 --2- 172.21.15.154:0/870029975 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0e7c004810 0x7f0e7c004c90 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f0e88001060 tx=0x7f0e8801cc10 comp rx=0 tx=0).stop 2024-09-16T03:42:08.376 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.362+0000 7f0e92d9d640 1 -- 172.21.15.154:0/870029975 reap_dead start 2024-09-16T03:42:08.376 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.362+0000 7f0e98dfb640 1 -- 172.21.15.154:0/870029975 shutdown_connections 2024-09-16T03:42:08.376 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.362+0000 7f0e98dfb640 1 -- 172.21.15.154:0/870029975 >> 172.21.15.154:0/870029975 conn(0x7f0e940aa1b0 msgr2=0x7f0e940a66b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:42:08.376 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.362+0000 7f0e98dfb640 1 -- 172.21.15.154:0/870029975 shutdown_connections 2024-09-16T03:42:08.376 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.362+0000 7f0e98dfb640 1 -- 172.21.15.154:0/870029975 wait complete. 2024-09-16T03:42:08.376 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.362+0000 7f0e98dfb640 10 monclient: build_initial_monmap 2024-09-16T03:42:08.376 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.362+0000 7f0e98dfb640 10 monclient: monmap: 2024-09-16T03:42:08.376 INFO:teuthology.orchestra.run.smithi154.stderr:epoch 0 2024-09-16T03:42:08.376 INFO:teuthology.orchestra.run.smithi154.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:42:08.376 INFO:teuthology.orchestra.run.smithi154.stderr:last_changed 0.000000 2024-09-16T03:42:08.376 INFO:teuthology.orchestra.run.smithi154.stderr:created 0.000000 2024-09-16T03:42:08.377 INFO:teuthology.orchestra.run.smithi154.stderr:min_mon_release 0 (unknown) 2024-09-16T03:42:08.377 INFO:teuthology.orchestra.run.smithi154.stderr:election_strategy: 1 2024-09-16T03:42:08.377 INFO:teuthology.orchestra.run.smithi154.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:42:08.377 INFO:teuthology.orchestra.run.smithi154.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:42:08.377 INFO:teuthology.orchestra.run.smithi154.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:42:08.377 INFO:teuthology.orchestra.run.smithi154.stderr: 2024-09-16T03:42:08.377 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.362+0000 7f0e98dfb640 1 librados: starting msgr at 2024-09-16T03:42:08.377 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.362+0000 7f0e98dfb640 1 librados: starting objecter 2024-09-16T03:42:08.377 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.362+0000 7f0e98dfb640 1 Processor -- start 2024-09-16T03:42:08.377 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.362+0000 7f0e98dfb640 1 -- start start 2024-09-16T03:42:08.377 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.362+0000 7f0e98dfb640 1 librados: setting wanted keys 2024-09-16T03:42:08.377 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.362+0000 7f0e98dfb640 1 librados: calling monclient init 2024-09-16T03:42:08.377 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.362+0000 7f0e98dfb640 10 monclient: init 2024-09-16T03:42:08.382 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.362+0000 7f0e98dfb640 10 monclient: _reopen_session rank -1 2024-09-16T03:42:08.382 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.362+0000 7f0e98dfb640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T03:42:08.382 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.362+0000 7f0e98dfb640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0e7c004810 0x7f0e94134190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:08.382 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.362+0000 7f0e98dfb640 10 monclient(hunting): picked mon.noname-a con 0x7f0e7c004810 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:42:08.382 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.362+0000 7f0e98dfb640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0e9413ec80 0x7f0e9413a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:08.382 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.362+0000 7f0e98dfb640 10 monclient(hunting): picked mon.noname-b con 0x7f0e9413ec80 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:42:08.382 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.362+0000 7f0e98dfb640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0e9413f100 0x7f0e9413a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:08.383 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.362+0000 7f0e98dfb640 10 monclient(hunting): picked mon.noname-c con 0x7f0e9413f100 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:42:08.383 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.362+0000 7f0e98dfb640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:08.383 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.362+0000 7f0e98dfb640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f0e940a60d0 con 0x7f0e7c004810 2024-09-16T03:42:08.383 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.362+0000 7f0e98dfb640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:08.383 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.362+0000 7f0e98dfb640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f0e940a5f50 con 0x7f0e9413ec80 2024-09-16T03:42:08.383 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.362+0000 7f0e98dfb640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:08.383 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.362+0000 7f0e98dfb640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f0e94140b40 con 0x7f0e9413f100 2024-09-16T03:42:08.383 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.362+0000 7f0e98dfb640 10 monclient(hunting): _renew_subs 2024-09-16T03:42:08.383 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.362+0000 7f0e98dfb640 10 monclient(hunting): authenticate will time out at 967.193939s 2024-09-16T03:42:08.383 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.362+0000 7f0e91d9b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0e9413ec80 0x7f0e9413a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:08.383 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.362+0000 7f0e91d9b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0e9413ec80 0x7f0e9413a2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.154:50224/0 (socket says 172.21.15.154:50224) 2024-09-16T03:42:08.383 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.362+0000 7f0e91d9b640 1 -- 172.21.15.154:0/388682041 learned_addr learned my addr 172.21.15.154:0/388682041 (peer_addr_for_me v2:172.21.15.154:0/0) 2024-09-16T03:42:08.383 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.362+0000 7f0e91d9b640 10 monclient(hunting): get_auth_request con 0x7f0e9413ec80 auth_method 0 2024-09-16T03:42:08.383 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.362+0000 7f0e91d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:08.384 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.362+0000 7f0e91d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:08.384 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.362+0000 7f0e91d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:08.384 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.362+0000 7f0e91d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:08.384 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.362+0000 7f0e91d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:08.384 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.362+0000 7f0e91d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:08.384 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.362+0000 7f0e9259c640 1 --2- 172.21.15.154:0/388682041 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0e7c004810 0x7f0e94134190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:08.384 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.362+0000 7f0e91d9b640 10 monclient(hunting): handle_auth_done global_id 4147 payload 824 2024-09-16T03:42:08.384 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.362+0000 7f0e91d9b640 1 -- 172.21.15.154:0/388682041 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0e9413f100 msgr2=0x7f0e9413a830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:08.384 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.362+0000 7f0e91d9b640 1 --2- 172.21.15.154:0/388682041 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0e9413f100 0x7f0e9413a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:08.384 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.362+0000 7f0e91d9b640 1 -- 172.21.15.154:0/388682041 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0e7c004810 msgr2=0x7f0e94134190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:08.384 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.362+0000 7f0e91d9b640 1 --2- 172.21.15.154:0/388682041 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0e7c004810 0x7f0e94134190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:08.384 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.362+0000 7f0e91d9b640 10 monclient: _finish_hunting 0 2024-09-16T03:42:08.384 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.362+0000 7f0e91d9b640 1 monclient: found mon.noname-b 2024-09-16T03:42:08.385 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.362+0000 7f0e91d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:42:08.385 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.362+0000 7f0e91d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.154:3300/0 2024-09-16T03:42:08.385 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.362+0000 7f0e91d9b640 1 -- 172.21.15.154:0/388682041 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e9406f6e0 con 0x7f0e9413ec80 2024-09-16T03:42:08.385 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.362+0000 7f0e91d9b640 1 --2- 172.21.15.154:0/388682041 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0e9413ec80 0x7f0e9413a2f0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f0e880ab040 tx=0x7f0e7c005840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:08.385 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.362+0000 7f0e82ffd640 10 client.?.objecter ms_handle_connect 0x7f0e9413ec80 2024-09-16T03:42:08.385 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.362+0000 7f0e82ffd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:42:08.385 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.362+0000 7f0e82ffd640 1 -- 172.21.15.154:0/388682041 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e880049f0 con 0x7f0e9413ec80 2024-09-16T03:42:08.385 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.362+0000 7f0e82ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:08.385 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.362+0000 7f0e82ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:42:08.385 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.362+0000 7f0e82ffd640 10 monclient: dump: 2024-09-16T03:42:08.385 INFO:teuthology.orchestra.run.smithi154.stderr:epoch 1 2024-09-16T03:42:08.385 INFO:teuthology.orchestra.run.smithi154.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:08.385 INFO:teuthology.orchestra.run.smithi154.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:08.385 INFO:teuthology.orchestra.run.smithi154.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:08.386 INFO:teuthology.orchestra.run.smithi154.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:08.386 INFO:teuthology.orchestra.run.smithi154.stderr:election_strategy: 1 2024-09-16T03:42:08.386 INFO:teuthology.orchestra.run.smithi154.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:08.386 INFO:teuthology.orchestra.run.smithi154.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:08.386 INFO:teuthology.orchestra.run.smithi154.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:08.386 INFO:teuthology.orchestra.run.smithi154.stderr: 2024-09-16T03:42:08.386 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.362+0000 7f0e82ffd640 10 monclient: _finish_auth 0 2024-09-16T03:42:08.386 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.362+0000 7f0e82ffd640 10 monclient: _check_auth_tickets 2024-09-16T03:42:08.386 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.362+0000 7f0e82ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:42:08.387 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.362+0000 7f0e82ffd640 1 -- 172.21.15.154:0/388682041 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0e88004b90 con 0x7f0e9413ec80 2024-09-16T03:42:08.387 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.362+0000 7f0e82ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:42:08.387 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.362+0000 7f0e82ffd640 1 -- 172.21.15.154:0/388682041 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e880070d0 con 0x7f0e9413ec80 2024-09-16T03:42:08.387 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.362+0000 7f0e82ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:08.387 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.362+0000 7f0e82ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:42:08.387 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.362+0000 7f0e82ffd640 10 monclient: dump: 2024-09-16T03:42:08.387 INFO:teuthology.orchestra.run.smithi154.stderr:epoch 1 2024-09-16T03:42:08.387 INFO:teuthology.orchestra.run.smithi154.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:08.388 INFO:teuthology.orchestra.run.smithi154.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:08.388 INFO:teuthology.orchestra.run.smithi154.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:08.388 INFO:teuthology.orchestra.run.smithi154.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:08.388 INFO:teuthology.orchestra.run.smithi154.stderr:election_strategy: 1 2024-09-16T03:42:08.388 INFO:teuthology.orchestra.run.smithi154.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:08.388 INFO:teuthology.orchestra.run.smithi154.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:08.389 INFO:teuthology.orchestra.run.smithi154.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:08.389 INFO:teuthology.orchestra.run.smithi154.stderr: 2024-09-16T03:42:08.389 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.362+0000 7f0e90d99640 4 set_mon_vals no callback set 2024-09-16T03:42:08.389 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.362+0000 7f0e98dfb640 5 monclient: authenticate success, global_id 4147 2024-09-16T03:42:08.389 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.362+0000 7f0e98dfb640 10 monclient: _renew_subs 2024-09-16T03:42:08.389 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.362+0000 7f0e98dfb640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2024-09-16T03:42:08.389 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.362+0000 7f0e98dfb640 1 -- 172.21.15.154:0/388682041 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e9413ceb0 con 0x7f0e9413ec80 2024-09-16T03:42:08.389 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.362+0000 7f0e98dfb640 10 client.4147.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:42:08.389 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.362+0000 7f0e98dfb640 10 monclient: _renew_subs 2024-09-16T03:42:08.389 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.362+0000 7f0e98dfb640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2024-09-16T03:42:08.389 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.362+0000 7f0e98dfb640 1 -- 172.21.15.154:0/388682041 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e9413d470 con 0x7f0e9413ec80 2024-09-16T03:42:08.389 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.362+0000 7f0e98dfb640 1 librados: init done 2024-09-16T03:42:08.390 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.362+0000 7f0e82ffd640 1 -- 172.21.15.154:0/388682041 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f0e88007600 con 0x7f0e9413ec80 2024-09-16T03:42:08.390 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.362+0000 7f0e82ffd640 1 -- 172.21.15.154:0/388682041 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(7..7 src has 1..7) ==== 1300+0+0 (secure 0 0 0) 0x7f0e880b8bf0 con 0x7f0e9413ec80 2024-09-16T03:42:08.390 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.362+0000 7f0e82ffd640 10 client.4147.objecter ms_dispatch 0x7f0e94000ba0 osd_map(7..7 src has 1..7) 2024-09-16T03:42:08.390 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.362+0000 7f0e82ffd640 3 client.4147.objecter handle_osd_map got epochs [7,7] > 0 2024-09-16T03:42:08.390 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.362+0000 7f0e82ffd640 3 client.4147.objecter handle_osd_map decoding full epoch 7 2024-09-16T03:42:08.390 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.362+0000 7f0e82ffd640 20 client.4147.objecter dump_active .. 0 homeless 2024-09-16T03:42:08.390 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.366+0000 7f0e98dfb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:42:08.390 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.366+0000 7f0e98dfb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:42:08.390 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.366+0000 7f0e98dfb640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2024-09-16T03:42:08.390 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.366+0000 7f0e98dfb640 1 -- 172.21.15.154:0/388682041 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0e94136f80 con 0x7f0e9413ec80 2024-09-16T03:42:08.391 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.366+0000 7f0e82ffd640 1 -- 172.21.15.154:0/388682041 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (secure 0 0 0) 0x7f0e88016950 con 0x7f0e9413ec80 2024-09-16T03:42:08.391 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.366+0000 7f0e82ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:42:08.391 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.366+0000 7f0e82ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:42:08.412 INFO:tasks.ceph.mgr.x.smithi154.stderr:2024-09-16T03:42:08.410+0000 7f8fbbf11180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-16T03:42:08.437 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.434+0000 7f0e98dfb640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "83c05e44-861b-4bb2-9657-19196e5ac1f1", "id": 5}] 2024-09-16T03:42:08.437 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.434+0000 7f0e98dfb640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "83c05e44-861b-4bb2-9657-19196e5ac1f1", "id": 5}] 2024-09-16T03:42:08.437 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.434+0000 7f0e98dfb640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2024-09-16T03:42:08.437 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.434+0000 7f0e98dfb640 1 -- 172.21.15.154:0/388682041 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "osd new", "uuid": "83c05e44-861b-4bb2-9657-19196e5ac1f1", "id": 5} v 0) -- 0x7f0e9413c0c0 con 0x7f0e9413ec80 2024-09-16T03:42:08.445 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.442+0000 7f0e82ffd640 1 -- 172.21.15.154:0/388682041 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "83c05e44-861b-4bb2-9657-19196e5ac1f1", "id": 5}]=0 v8) ==== 112+0+2 (secure 0 0 0) 0x7f0e880b8670 con 0x7f0e9413ec80 2024-09-16T03:42:08.445 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.442+0000 7f0e82ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "83c05e44-861b-4bb2-9657-19196e5ac1f1", "id": 5}] 2024-09-16T03:42:08.445 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.442+0000 7f0e82ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:42:08.445 INFO:teuthology.orchestra.run.smithi154.stdout:5 2024-09-16T03:42:08.447 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.442+0000 7f0e98dfb640 10 librados: watch_flush enter 2024-09-16T03:42:08.447 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.442+0000 7f0e98dfb640 10 librados: watch_flush exit 2024-09-16T03:42:08.447 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.442+0000 7f0e98dfb640 20 client.4147.objecter shutdown clearing up homeless session... 2024-09-16T03:42:08.447 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.442+0000 7f0e98dfb640 10 client.4147.objecter successfully canceled tick 2024-09-16T03:42:08.448 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.442+0000 7f0e98dfb640 10 monclient: shutdown 2024-09-16T03:42:08.448 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.442+0000 7f0e98dfb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:42:08.448 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.442+0000 7f0e98dfb640 1 -- 172.21.15.154:0/388682041 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0e9413ec80 msgr2=0x7f0e9413a2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:08.448 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.442+0000 7f0e98dfb640 1 --2- 172.21.15.154:0/388682041 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0e9413ec80 0x7f0e9413a2f0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f0e880ab040 tx=0x7f0e7c005840 comp rx=0 tx=0).stop 2024-09-16T03:42:08.448 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.442+0000 7f0e98dfb640 1 -- 172.21.15.154:0/388682041 shutdown_connections 2024-09-16T03:42:08.448 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.442+0000 7f0e98dfb640 1 --2- 172.21.15.154:0/388682041 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0e9413f100 0x7f0e9413a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:08.448 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.442+0000 7f0e98dfb640 1 --2- 172.21.15.154:0/388682041 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0e9413ec80 0x7f0e9413a2f0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:08.448 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.442+0000 7f0e98dfb640 1 --2- 172.21.15.154:0/388682041 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0e7c004810 0x7f0e94134190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:08.448 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.442+0000 7f0e98dfb640 1 -- 172.21.15.154:0/388682041 >> 172.21.15.154:0/388682041 conn(0x7f0e940aa1b0 msgr2=0x7f0e9412e450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:42:08.449 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.442+0000 7f0e98dfb640 1 -- 172.21.15.154:0/388682041 shutdown_connections 2024-09-16T03:42:08.449 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.442+0000 7f0e98dfb640 1 -- 172.21.15.154:0/388682041 wait complete. 2024-09-16T03:42:08.449 INFO:teuthology.orchestra.run.smithi154.stderr:2024-09-16T03:42:08.442+0000 7f0e98dfb640 1 librados: shutdown 2024-09-16T03:42:08.461 INFO:tasks.ceph.osd.0:Restarting daemon 2024-09-16T03:42:08.461 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-09-16T03:42:08.464 INFO:tasks.ceph.osd.0:Started 2024-09-16T03:42:08.464 INFO:tasks.ceph.osd.1:Restarting daemon 2024-09-16T03:42:08.464 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-09-16T03:42:08.467 INFO:tasks.ceph.osd.1:Started 2024-09-16T03:42:08.467 INFO:tasks.ceph.osd.2:Restarting daemon 2024-09-16T03:42:08.467 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-09-16T03:42:08.468 INFO:tasks.ceph.mgr.x.smithi154.stderr:2024-09-16T03:42:08.466+0000 7f8fbbf11180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-16T03:42:08.469 INFO:tasks.ceph.osd.2:Started 2024-09-16T03:42:08.470 INFO:tasks.ceph.osd.3:Restarting daemon 2024-09-16T03:42:08.470 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-09-16T03:42:08.472 INFO:tasks.ceph.osd.3:Started 2024-09-16T03:42:08.472 INFO:tasks.ceph.osd.4:Restarting daemon 2024-09-16T03:42:08.472 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-09-16T03:42:08.474 INFO:tasks.ceph.osd.4:Started 2024-09-16T03:42:08.474 INFO:tasks.ceph.osd.5:Restarting daemon 2024-09-16T03:42:08.474 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-09-16T03:42:08.475 INFO:tasks.ceph.osd.5:Started 2024-09-16T03:42:08.476 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T03:42:08.555 INFO:tasks.ceph.mgr.x.smithi154.stderr:2024-09-16T03:42:08.554+0000 7f8fbbf11180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-16T03:42:08.596 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.589+0000 7fd72e40a640 10 monclient: get_monmap_and_config 2024-09-16T03:42:08.596 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.589+0000 7fd72e40a640 10 monclient: build_initial_monmap 2024-09-16T03:42:08.597 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.589+0000 7fd72e40a640 10 monclient: monmap: 2024-09-16T03:42:08.597 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:42:08.597 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:08.597 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:42:08.594591+0000 2024-09-16T03:42:08.597 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:42:08.594591+0000 2024-09-16T03:42:08.597 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:42:08.597 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:08.597 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:42:08.597 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:42:08.597 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:42:08.597 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:42:08.597 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:42:08.597 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:08.600 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.589+0000 7fd72e40a640 1 Processor -- start 2024-09-16T03:42:08.600 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.589+0000 7fd72e40a640 1 -- start start 2024-09-16T03:42:08.600 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.589+0000 7fd72e40a640 10 monclient: init 2024-09-16T03:42:08.600 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.589+0000 7fd72e40a640 10 monclient: _reopen_session rank -1 2024-09-16T03:42:08.600 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.589+0000 7fd72e40a640 10 monclient: _add_conns ranks=[2,0,3,1,4] 2024-09-16T03:42:08.600 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.589+0000 7fd72e40a640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd72806db90 0x7fd7281340c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:08.600 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.589+0000 7fd72e40a640 10 monclient(hunting): picked mon.noname-b con 0x7fd72806db90 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:42:08.600 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.589+0000 7fd72e40a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd7280a37f0 addr v1:172.21.15.42:6789/0 2024-09-16T03:42:08.600 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.589+0000 7fd72e40a640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd7280a3bd0 0x7fd728137520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:08.600 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.589+0000 7fd72e40a640 10 monclient(hunting): picked mon.noname-a con 0x7fd7280a3bd0 addr v2:172.21.15.42:3300/0 2024-09-16T03:42:08.600 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.589+0000 7fd72e40a640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd72806ece0 con 0x7fd7280a37f0 2024-09-16T03:42:08.601 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.589+0000 7fd72e40a640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:08.601 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.589+0000 7fd72e40a640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7fd72806df70 con 0x7fd7280a3bd0 2024-09-16T03:42:08.601 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.589+0000 7fd72e40a640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:08.601 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.589+0000 7fd72e40a640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fd72809bf80 con 0x7fd72806db90 2024-09-16T03:42:08.601 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.589+0000 7fd72e40a640 10 monclient(hunting): _renew_subs 2024-09-16T03:42:08.601 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.589+0000 7fd72e40a640 10 monclient(hunting): authenticate will time out at 975.563871s 2024-09-16T03:42:08.601 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.589+0000 7fd72c9a7640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd7280a3bd0 0x7fd728137520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:08.601 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.593+0000 7fd727fff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd72806db90 0x7fd7281340c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:08.601 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.593+0000 7fd7277fe640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fd7280a37f0 0x7fd728136e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:33560/0 (socket says 172.21.15.42:33560) 2024-09-16T03:42:08.601 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.593+0000 7fd7277fe640 1 -- 172.21.15.42:0/2059306726 learned_addr learned my addr 172.21.15.42:0/2059306726 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:42:08.601 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.593+0000 7fd72c9a7640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd7280a3bd0 0x7fd728137520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:53428/0 (socket says 172.21.15.42:53428) 2024-09-16T03:42:08.601 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.593+0000 7fd727fff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd72806db90 0x7fd7281340c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:54940/0 (socket says 172.21.15.42:54940) 2024-09-16T03:42:08.601 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.593+0000 7fd72c9a7640 10 monclient(hunting): get_auth_request con 0x7fd7280a3bd0 auth_method 0 2024-09-16T03:42:08.601 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.593+0000 7fd72c9a7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:08.602 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.593+0000 7fd72c9a7640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:08.602 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.593+0000 7fd72c9a7640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:08.602 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.593+0000 7fd727fff640 10 monclient(hunting): get_auth_request con 0x7fd72806db90 auth_method 0 2024-09-16T03:42:08.602 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.593+0000 7fd727fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:08.602 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.593+0000 7fd727fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:08.602 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.593+0000 7fd727fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:08.602 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.593+0000 7fd72c9a7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:08.602 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.593+0000 7fd72c9a7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:08.602 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.593+0000 7fd72c9a7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:08.602 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.593+0000 7fd727fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:08.603 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.593+0000 7fd727fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:08.603 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.593+0000 7fd727fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:08.603 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.593+0000 7fd72c9a7640 10 monclient(hunting): handle_auth_done global_id 4158 payload 274 2024-09-16T03:42:08.603 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.593+0000 7fd72c9a7640 1 -- 172.21.15.42:0/2059306726 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd72806db90 msgr2=0x7fd7281340c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:42:08.603 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.593+0000 7fd72c9a7640 1 --2- 172.21.15.42:0/2059306726 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd72806db90 0x7fd7281340c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:08.603 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.593+0000 7fd72c9a7640 1 -- 172.21.15.42:0/2059306726 >> v1:172.21.15.42:6789/0 conn(0x7fd7280a37f0 legacy=0x7fd728136e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:08.603 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.593+0000 7fd72c9a7640 10 monclient: _finish_hunting 0 2024-09-16T03:42:08.603 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.593+0000 7fd72c9a7640 1 monclient: found mon.noname-a 2024-09-16T03:42:08.603 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.593+0000 7fd72c9a7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:42:08.603 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.593+0000 7fd72c9a7640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:42:08.603 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.593+0000 7fd72c9a7640 1 -- 172.21.15.42:0/2059306726 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd728134780 con 0x7fd7280a3bd0 2024-09-16T03:42:08.603 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.593+0000 7fd726ffd640 1 -- 172.21.15.42:0/2059306726 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 338292409 0 0) 0x7fd72806ece0 con 0x7fd7280a37f0 2024-09-16T03:42:08.603 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.593+0000 7fd726ffd640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T03:42:08.609 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.593+0000 7fd72c9a7640 1 --2- 172.21.15.42:0/2059306726 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd7280a3bd0 0x7fd728137520 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fd720009d90 tx=0x7fd72001d290 comp rx=0 tx=0).ready entity=mon.0 client_cookie=637afee531fe8b7b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:08.609 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.593+0000 7fd726ffd640 1 -- 172.21.15.42:0/2059306726 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7200b2070 con 0x7fd7280a3bd0 2024-09-16T03:42:08.609 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.593+0000 7fd726ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:08.609 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.593+0000 7fd726ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:42:08.609 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.593+0000 7fd726ffd640 10 monclient: dump: 2024-09-16T03:42:08.609 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:08.609 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:08.610 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:08.610 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:08.610 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:08.610 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:08.610 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:08.610 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:08.610 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:08.610 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:08.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.593+0000 7fd726ffd640 10 monclient: _finish_auth 0 2024-09-16T03:42:08.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.593+0000 7fd726ffd640 10 monclient: _check_auth_tickets 2024-09-16T03:42:08.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.593+0000 7fd726ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:42:08.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.593+0000 7fd726ffd640 1 -- 172.21.15.42:0/2059306726 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7200a5800 con 0x7fd7280a3bd0 2024-09-16T03:42:08.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.593+0000 7fd72e40a640 5 monclient: authenticate success, global_id 4158 2024-09-16T03:42:08.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.593+0000 7fd72e40a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:42:08.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.593+0000 7fd726ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:42:08.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.593+0000 7fd72e40a640 10 monclient: get_monmap_and_config success 2024-09-16T03:42:08.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.593+0000 7fd726ffd640 1 -- 172.21.15.42:0/2059306726 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7200a5bd0 con 0x7fd7280a3bd0 2024-09-16T03:42:08.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.593+0000 7fd726ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:08.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.593+0000 7fd72e40a640 4 set_mon_vals no callback set 2024-09-16T03:42:08.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.593+0000 7fd726ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:42:08.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.593+0000 7fd726ffd640 10 monclient: dump: 2024-09-16T03:42:08.611 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:08.611 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:08.611 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:08.611 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:08.611 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:08.611 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:08.612 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:08.612 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:08.612 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:08.612 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:08.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.593+0000 7fd72e40a640 10 monclient: shutdown 2024-09-16T03:42:08.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.593+0000 7fd72e40a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:42:08.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.593+0000 7fd72e40a640 1 -- 172.21.15.42:0/2059306726 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd7280a3bd0 msgr2=0x7fd728137520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:08.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.593+0000 7fd72e40a640 1 --2- 172.21.15.42:0/2059306726 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd7280a3bd0 0x7fd728137520 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fd720009d90 tx=0x7fd72001d290 comp rx=0 tx=0).stop 2024-09-16T03:42:08.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.593+0000 7fd72e40a640 1 -- 172.21.15.42:0/2059306726 shutdown_connections 2024-09-16T03:42:08.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.593+0000 7fd72e40a640 1 --2- 172.21.15.42:0/2059306726 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd7280a3bd0 0x7fd728137520 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:08.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.593+0000 7fd72e40a640 1 --2- 172.21.15.42:0/2059306726 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd72806db90 0x7fd7281340c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:08.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.593+0000 7fd72e40a640 1 -- 172.21.15.42:0/2059306726 >> 172.21.15.42:0/2059306726 conn(0x7fd7280a9fb0 msgr2=0x7fd7280aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:42:08.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.593+0000 7fd72e40a640 1 -- 172.21.15.42:0/2059306726 shutdown_connections 2024-09-16T03:42:08.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.593+0000 7fd72e40a640 1 -- 172.21.15.42:0/2059306726 wait complete. 2024-09-16T03:42:08.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.593+0000 7fd72e40a640 10 monclient: build_initial_monmap 2024-09-16T03:42:08.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.593+0000 7fd72e40a640 10 monclient: monmap: 2024-09-16T03:42:08.613 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:42:08.613 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:42:08.613 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:42:08.613 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:42:08.613 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:42:08.613 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:08.613 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:42:08.613 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:42:08.613 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:42:08.613 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:08.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.597+0000 7fd72e40a640 1 librados: starting msgr at 2024-09-16T03:42:08.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.597+0000 7fd72e40a640 1 librados: starting objecter 2024-09-16T03:42:08.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.597+0000 7fd72e40a640 1 Processor -- start 2024-09-16T03:42:08.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.597+0000 7fd72e40a640 1 -- start start 2024-09-16T03:42:08.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.597+0000 7fd72e40a640 1 librados: setting wanted keys 2024-09-16T03:42:08.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.597+0000 7fd72e40a640 1 librados: calling monclient init 2024-09-16T03:42:08.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.597+0000 7fd72e40a640 10 monclient: init 2024-09-16T03:42:08.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.597+0000 7fd72e40a640 10 monclient: _reopen_session rank -1 2024-09-16T03:42:08.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.597+0000 7fd72e40a640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:42:08.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.597+0000 7fd72e40a640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd72806db90 0x7fd72813e310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:08.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.597+0000 7fd72e40a640 10 monclient(hunting): picked mon.noname-c con 0x7fd72806db90 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:42:08.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.597+0000 7fd72e40a640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd7280a37f0 0x7fd72813e850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:08.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.597+0000 7fd72e40a640 10 monclient(hunting): picked mon.noname-b con 0x7fd7280a37f0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:42:08.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.597+0000 7fd72e40a640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd7280a3bd0 0x7fd7280a1710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:08.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.597+0000 7fd72e40a640 10 monclient(hunting): picked mon.noname-a con 0x7fd7280a3bd0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:42:08.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.597+0000 7fd72e40a640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:08.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.597+0000 7fd72e40a640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fd7281419d0 con 0x7fd7280a3bd0 2024-09-16T03:42:08.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.597+0000 7fd72e40a640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:08.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.597+0000 7fd72e40a640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd7280a4100 con 0x7fd7280a37f0 2024-09-16T03:42:08.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.597+0000 7fd72e40a640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:08.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.597+0000 7fd72e40a640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fd728141b50 con 0x7fd72806db90 2024-09-16T03:42:08.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.597+0000 7fd72e40a640 10 monclient(hunting): _renew_subs 2024-09-16T03:42:08.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.597+0000 7fd72e40a640 10 monclient(hunting): authenticate will time out at 975.568939s 2024-09-16T03:42:08.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.597+0000 7fd727fff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd72806db90 0x7fd72813e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:08.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.597+0000 7fd727fff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd72806db90 0x7fd72813e310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:54952/0 (socket says 172.21.15.42:54952) 2024-09-16T03:42:08.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.597+0000 7fd72c9a7640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd7280a3bd0 0x7fd7280a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:08.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.597+0000 7fd727fff640 1 -- 172.21.15.42:0/4050090737 learned_addr learned my addr 172.21.15.42:0/4050090737 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:42:08.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.597+0000 7fd727fff640 10 monclient(hunting): get_auth_request con 0x7fd72806db90 auth_method 0 2024-09-16T03:42:08.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.597+0000 7fd727fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:08.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.597+0000 7fd727fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:08.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.597+0000 7fd727fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:08.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.597+0000 7fd72c9a7640 10 monclient(hunting): get_auth_request con 0x7fd7280a3bd0 auth_method 0 2024-09-16T03:42:08.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.597+0000 7fd72c9a7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:08.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.597+0000 7fd72c9a7640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:08.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.597+0000 7fd72c9a7640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:08.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.597+0000 7fd727fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:08.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.597+0000 7fd727fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:08.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.597+0000 7fd727fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:08.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.597+0000 7fd72c9a7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:08.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.597+0000 7fd72c9a7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:08.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.597+0000 7fd72c9a7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:08.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.597+0000 7fd7277fe640 1 --2- 172.21.15.42:0/4050090737 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd7280a37f0 0x7fd72813e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:08.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.597+0000 7fd7277fe640 10 monclient(hunting): get_auth_request con 0x7fd7280a37f0 auth_method 0 2024-09-16T03:42:08.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.597+0000 7fd7277fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:08.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.597+0000 7fd7277fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:08.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.597+0000 7fd7277fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:08.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.597+0000 7fd727fff640 10 monclient(hunting): handle_auth_done global_id 4142 payload 824 2024-09-16T03:42:08.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.597+0000 7fd727fff640 1 -- 172.21.15.42:0/4050090737 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd7280a37f0 msgr2=0x7fd72813e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:08.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.597+0000 7fd727fff640 1 --2- 172.21.15.42:0/4050090737 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd7280a37f0 0x7fd72813e850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:08.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.597+0000 7fd727fff640 1 -- 172.21.15.42:0/4050090737 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd7280a3bd0 msgr2=0x7fd7280a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:08.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.597+0000 7fd727fff640 1 --2- 172.21.15.42:0/4050090737 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd7280a3bd0 0x7fd7280a1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:08.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.597+0000 7fd727fff640 10 monclient: _finish_hunting 0 2024-09-16T03:42:08.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.597+0000 7fd727fff640 1 monclient: found mon.noname-c 2024-09-16T03:42:08.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.597+0000 7fd727fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:42:08.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.597+0000 7fd727fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:42:08.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.597+0000 7fd727fff640 1 -- 172.21.15.42:0/4050090737 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd728139db0 con 0x7fd72806db90 2024-09-16T03:42:08.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.597+0000 7fd72c9a7640 1 --2- 172.21.15.42:0/4050090737 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd7280a3bd0 0x7fd7280a1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:42:08.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.597+0000 7fd727fff640 1 --2- 172.21.15.42:0/4050090737 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd72806db90 0x7fd72813e310 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fd71800e8a0 tx=0x7fd71800eca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:08.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.597+0000 7fd724ff9640 10 client.?.objecter ms_handle_connect 0x7fd72806db90 2024-09-16T03:42:08.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.597+0000 7fd724ff9640 10 client.?.objecter resend_mon_ops 2024-09-16T03:42:08.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.597+0000 7fd724ff9640 1 -- 172.21.15.42:0/4050090737 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd718017070 con 0x7fd72806db90 2024-09-16T03:42:08.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.597+0000 7fd724ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:08.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.597+0000 7fd724ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:42:08.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.597+0000 7fd724ff9640 10 monclient: dump: 2024-09-16T03:42:08.618 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:08.618 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:08.618 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:08.618 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:08.619 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:08.619 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:08.619 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:08.619 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:08.619 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:08.619 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:08.619 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.597+0000 7fd724ff9640 10 monclient: _finish_auth 0 2024-09-16T03:42:08.619 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.597+0000 7fd724ff9640 10 monclient: _check_auth_tickets 2024-09-16T03:42:08.619 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.597+0000 7fd724ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:42:08.619 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.597+0000 7fd724ff9640 1 -- 172.21.15.42:0/4050090737 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd71800c940 con 0x7fd72806db90 2024-09-16T03:42:08.619 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.597+0000 7fd72e40a640 5 monclient: authenticate success, global_id 4142 2024-09-16T03:42:08.619 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.597+0000 7fd72e40a640 10 monclient: _renew_subs 2024-09-16T03:42:08.619 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.597+0000 7fd72e40a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:42:08.619 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.597+0000 7fd72e40a640 1 -- 172.21.15.42:0/4050090737 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd728133110 con 0x7fd72806db90 2024-09-16T03:42:08.620 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.597+0000 7fd724ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T03:42:08.620 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.597+0000 7fd724ff9640 1 -- 172.21.15.42:0/4050090737 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd71800ccf0 con 0x7fd72806db90 2024-09-16T03:42:08.620 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.597+0000 7fd724ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:08.620 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.597+0000 7fd724ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:42:08.620 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.597+0000 7fd724ff9640 10 monclient: dump: 2024-09-16T03:42:08.620 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:08.620 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:08.620 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:08.620 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:08.620 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:08.620 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:08.620 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:08.620 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:08.620 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:08.621 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:08.621 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.597+0000 7fd72e40a640 10 client.4142.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:42:08.621 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.597+0000 7fd72e40a640 10 monclient: _renew_subs 2024-09-16T03:42:08.621 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.597+0000 7fd72e40a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:42:08.621 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.597+0000 7fd72e40a640 1 -- 172.21.15.42:0/4050090737 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7281397f0 con 0x7fd72806db90 2024-09-16T03:42:08.621 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.597+0000 7fd72e40a640 1 librados: init done 2024-09-16T03:42:08.621 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.597+0000 7fd7267fc640 4 set_mon_vals no callback set 2024-09-16T03:42:08.621 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.597+0000 7fd724ff9640 1 -- 172.21.15.42:0/4050090737 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fd71801a070 con 0x7fd72806db90 2024-09-16T03:42:08.621 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.597+0000 7fd724ff9640 1 -- 172.21.15.42:0/4050090737 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(8..8 src has 1..8) ==== 1415+0+0 (secure 0 0 0) 0x7fd718013470 con 0x7fd72806db90 2024-09-16T03:42:08.621 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.597+0000 7fd724ff9640 10 client.4142.objecter ms_dispatch 0x7fd728000ba0 osd_map(8..8 src has 1..8) 2024-09-16T03:42:08.621 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.597+0000 7fd724ff9640 3 client.4142.objecter handle_osd_map got epochs [8,8] > 0 2024-09-16T03:42:08.621 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.597+0000 7fd724ff9640 3 client.4142.objecter handle_osd_map decoding full epoch 8 2024-09-16T03:42:08.621 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.597+0000 7fd724ff9640 20 client.4142.objecter dump_active .. 0 homeless 2024-09-16T03:42:08.621 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.597+0000 7fd72e40a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:42:08.622 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.597+0000 7fd72e40a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:42:08.622 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.597+0000 7fd72e40a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:42:08.622 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.597+0000 7fd72e40a640 1 -- 172.21.15.42:0/4050090737 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd7281340c0 con 0x7fd72806db90 2024-09-16T03:42:08.622 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.601+0000 7fd724ff9640 1 -- 172.21.15.42:0/4050090737 <== mon.2 v2:172.21.15.42:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (secure 0 0 0) 0x7fd718006430 con 0x7fd72806db90 2024-09-16T03:42:08.622 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.601+0000 7fd724ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:42:08.622 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.601+0000 7fd724ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:42:08.624 INFO:tasks.ceph.mgr.x.smithi154.stderr:2024-09-16T03:42:08.606+0000 7f8fbbf11180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-16T03:42:08.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.657+0000 7fd72e40a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-16T03:42:08.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.657+0000 7fd72e40a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T03:42:08.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.657+0000 7fd72e40a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:42:08.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.657+0000 7fd72e40a640 1 -- 172.21.15.42:0/4050090737 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd728068e40 con 0x7fd72806db90 2024-09-16T03:42:08.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.661+0000 7fd724ff9640 1 -- 172.21.15.42:0/4050090737 <== mon.2 v2:172.21.15.42:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v8) ==== 74+0+5605 (secure 0 0 0) 0x7fd718006610 con 0x7fd72806db90 2024-09-16T03:42:08.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.661+0000 7fd724ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T03:42:08.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.661+0000 7fd724ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:42:08.667 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-09-16T03:42:08.667 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":8,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","created":"2024-09-16T03:42:05.977277+0000","modified":"2024-09-16T03:42:08.437271+0000","last_up_change":"0.000000","last_in_change":"2024-09-16T03:42:08.437271+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6b9fc058-2cff-46af-b95c-b78c72e1dc2b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ee31f807-cb16-4c33-8e23-2573e1a6b5c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a26f3917-6702-43db-9355-e96da86a3365","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4e809f2a-2ffd-4791-9f2d-18f1bfe7d2ea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"dcf637ab-5f72-4ed2-a99b-69083f0b6b15","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"83c05e44-861b-4bb2-9657-19196e5ac1f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T03:42:08.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.661+0000 7fd72e40a640 10 librados: watch_flush enter 2024-09-16T03:42:08.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.661+0000 7fd72e40a640 10 librados: watch_flush exit 2024-09-16T03:42:08.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.661+0000 7fd72e40a640 20 client.4142.objecter shutdown clearing up homeless session... 2024-09-16T03:42:08.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.661+0000 7fd72e40a640 10 client.4142.objecter successfully canceled tick 2024-09-16T03:42:08.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.661+0000 7fd72e40a640 10 monclient: shutdown 2024-09-16T03:42:08.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.661+0000 7fd72e40a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:42:08.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.661+0000 7fd72e40a640 1 -- 172.21.15.42:0/4050090737 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd72806db90 msgr2=0x7fd72813e310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:08.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.661+0000 7fd72e40a640 1 --2- 172.21.15.42:0/4050090737 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd72806db90 0x7fd72813e310 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fd71800e8a0 tx=0x7fd71800eca0 comp rx=0 tx=0).stop 2024-09-16T03:42:08.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.661+0000 7fd72e40a640 1 -- 172.21.15.42:0/4050090737 shutdown_connections 2024-09-16T03:42:08.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.661+0000 7fd72e40a640 1 --2- 172.21.15.42:0/4050090737 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd7280a3bd0 0x7fd7280a1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:08.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.661+0000 7fd72e40a640 1 --2- 172.21.15.42:0/4050090737 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd7280a37f0 0x7fd72813e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:08.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.661+0000 7fd72e40a640 1 --2- 172.21.15.42:0/4050090737 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd72806db90 0x7fd72813e310 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:08.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.661+0000 7fd72e40a640 1 -- 172.21.15.42:0/4050090737 >> 172.21.15.42:0/4050090737 conn(0x7fd7280a9fb0 msgr2=0x7fd728136a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:42:08.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.661+0000 7fd72e40a640 1 -- 172.21.15.42:0/4050090737 shutdown_connections 2024-09-16T03:42:08.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.661+0000 7fd72e40a640 1 -- 172.21.15.42:0/4050090737 wait complete. 2024-09-16T03:42:08.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:08.661+0000 7fd72e40a640 1 librados: shutdown 2024-09-16T03:42:08.684 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-09-16T03:42:08.684 INFO:tasks.ceph:Waiting for OSDs to come up 2024-09-16T03:42:08.757 INFO:tasks.ceph.mgr.x.smithi154.stderr:2024-09-16T03:42:08.754+0000 7f8fbbf11180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-16T03:42:08.818 INFO:tasks.ceph.mgr.x.smithi154.stderr:2024-09-16T03:42:08.814+0000 7f8fbbf11180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-16T03:42:09.288 DEBUG:teuthology.orchestra.run.smithi042:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-16T03:42:09.409 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.401+0000 7f6a898dc640 10 monclient: get_monmap_and_config 2024-09-16T03:42:09.409 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.401+0000 7f6a898dc640 10 monclient: build_initial_monmap 2024-09-16T03:42:09.409 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.401+0000 7f6a898dc640 10 monclient: monmap: 2024-09-16T03:42:09.409 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-09-16T03:42:09.409 INFO:teuthology.misc.health.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:09.409 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-09-16T03:42:09.407091+0000 2024-09-16T03:42:09.410 INFO:teuthology.misc.health.smithi042.stderr:created 2024-09-16T03:42:09.407091+0000 2024-09-16T03:42:09.410 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:42:09.410 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:09.410 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:42:09.410 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:42:09.410 INFO:teuthology.misc.health.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:42:09.410 INFO:teuthology.misc.health.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:42:09.410 INFO:teuthology.misc.health.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:42:09.410 INFO:teuthology.misc.health.smithi042.stderr: 2024-09-16T03:42:09.412 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.401+0000 7f6a898dc640 1 Processor -- start 2024-09-16T03:42:09.412 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.401+0000 7f6a898dc640 1 -- start start 2024-09-16T03:42:09.413 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.401+0000 7f6a898dc640 10 monclient: init 2024-09-16T03:42:09.413 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.405+0000 7f6a898dc640 10 monclient: _reopen_session rank -1 2024-09-16T03:42:09.413 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.405+0000 7f6a898dc640 10 monclient: _add_conns ranks=[2,3,4,1,0] 2024-09-16T03:42:09.413 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.405+0000 7f6a898dc640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f6a840a1230 0x7f6a8409dc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:09.413 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.405+0000 7f6a898dc640 10 monclient(hunting): picked mon.noname-b con 0x7f6a840a1230 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:42:09.413 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.405+0000 7f6a898dc640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f6a8409a100 0x7f6a8409e140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:09.413 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.405+0000 7f6a898dc640 10 monclient(hunting): picked mon.noname-a con 0x7f6a8409a100 addr v2:172.21.15.42:3300/0 2024-09-16T03:42:09.413 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.405+0000 7f6a898dc640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f6a840a1ac0 0x7f6a8409e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:09.413 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.405+0000 7f6a898dc640 10 monclient(hunting): picked mon.noname-c con 0x7f6a840a1ac0 addr v2:172.21.15.154:3300/0 2024-09-16T03:42:09.413 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.405+0000 7f6a898dc640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:09.413 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.405+0000 7f6a898dc640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f6a840a1610 con 0x7f6a8409a100 2024-09-16T03:42:09.413 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.405+0000 7f6a898dc640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:09.413 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.405+0000 7f6a898dc640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f6a840a1840 con 0x7f6a840a1ac0 2024-09-16T03:42:09.414 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.405+0000 7f6a898dc640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:09.414 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.405+0000 7f6a898dc640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f6a8409eea0 con 0x7f6a840a1230 2024-09-16T03:42:09.414 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.405+0000 7f6a898dc640 10 monclient(hunting): _renew_subs 2024-09-16T03:42:09.414 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.405+0000 7f6a898dc640 10 monclient(hunting): authenticate will time out at 976.376567s 2024-09-16T03:42:09.414 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.405+0000 7f6a827fc640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f6a8409a100 0x7f6a8409e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:09.414 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.405+0000 7f6a82ffd640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f6a840a1230 0x7f6a8409dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:09.414 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.405+0000 7f6a827fc640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f6a8409a100 0x7f6a8409e140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:53452/0 (socket says 172.21.15.42:53452) 2024-09-16T03:42:09.414 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.405+0000 7f6a827fc640 1 -- 172.21.15.42:0/1999992207 learned_addr learned my addr 172.21.15.42:0/1999992207 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:42:09.414 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.405+0000 7f6a827fc640 10 monclient(hunting): get_auth_request con 0x7f6a8409a100 auth_method 0 2024-09-16T03:42:09.414 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.405+0000 7f6a827fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:09.414 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.405+0000 7f6a827fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:09.414 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.405+0000 7f6a827fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:09.415 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.405+0000 7f6a82ffd640 10 monclient(hunting): get_auth_request con 0x7f6a840a1230 auth_method 0 2024-09-16T03:42:09.415 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.405+0000 7f6a82ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:09.415 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.405+0000 7f6a82ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:09.415 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.405+0000 7f6a82ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:09.415 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.405+0000 7f6a827fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:09.415 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.405+0000 7f6a827fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:09.415 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.405+0000 7f6a827fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:09.415 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.405+0000 7f6a82ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:09.415 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.405+0000 7f6a82ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:09.415 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.405+0000 7f6a82ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:09.415 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.405+0000 7f6a827fc640 10 monclient(hunting): handle_auth_done global_id 4167 payload 274 2024-09-16T03:42:09.416 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.405+0000 7f6a827fc640 1 -- 172.21.15.42:0/1999992207 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f6a840a1230 msgr2=0x7f6a8409dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:42:09.416 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.405+0000 7f6a827fc640 1 --2- 172.21.15.42:0/1999992207 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f6a840a1230 0x7f6a8409dc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:09.416 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.405+0000 7f6a827fc640 1 -- 172.21.15.42:0/1999992207 >> v2:172.21.15.154:3300/0 conn(0x7f6a840a1ac0 msgr2=0x7f6a8409e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:42:09.416 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.405+0000 7f6a827fc640 1 --2- 172.21.15.42:0/1999992207 >> v2:172.21.15.154:3300/0 conn(0x7f6a840a1ac0 0x7f6a8409e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:09.416 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.405+0000 7f6a827fc640 10 monclient: _finish_hunting 0 2024-09-16T03:42:09.416 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.405+0000 7f6a827fc640 1 monclient: found mon.noname-a 2024-09-16T03:42:09.416 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.405+0000 7f6a827fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:42:09.416 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.405+0000 7f6a827fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:42:09.416 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.405+0000 7f6a827fc640 1 -- 172.21.15.42:0/1999992207 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a84099ce0 con 0x7f6a8409a100 2024-09-16T03:42:09.416 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.405+0000 7f6a82ffd640 1 --2- 172.21.15.42:0/1999992207 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f6a840a1230 0x7f6a8409dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:42:09.420 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.405+0000 7f6a827fc640 1 --2- 172.21.15.42:0/1999992207 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6a8409a100 0x7f6a8409e140 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f6a84099220 tx=0x7f6a6c01c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c934a36018d6429 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:09.421 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.405+0000 7f6a81ffb640 1 -- 172.21.15.42:0/1999992207 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a6c0b0070 con 0x7f6a8409a100 2024-09-16T03:42:09.421 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.405+0000 7f6a81ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:09.421 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.405+0000 7f6a81ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:42:09.421 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.405+0000 7f6a81ffb640 10 monclient: dump: 2024-09-16T03:42:09.421 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-09-16T03:42:09.421 INFO:teuthology.misc.health.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:09.421 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:09.421 INFO:teuthology.misc.health.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:09.421 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:09.421 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:09.421 INFO:teuthology.misc.health.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:09.421 INFO:teuthology.misc.health.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:09.422 INFO:teuthology.misc.health.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:09.422 INFO:teuthology.misc.health.smithi042.stderr: 2024-09-16T03:42:09.422 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.405+0000 7f6a81ffb640 10 monclient: _finish_auth 0 2024-09-16T03:42:09.422 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.405+0000 7f6a81ffb640 10 monclient: _check_auth_tickets 2024-09-16T03:42:09.422 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.405+0000 7f6a81ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:42:09.422 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.405+0000 7f6a81ffb640 1 -- 172.21.15.42:0/1999992207 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6a6c0ab600 con 0x7f6a8409a100 2024-09-16T03:42:09.422 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.405+0000 7f6a898dc640 5 monclient: authenticate success, global_id 4167 2024-09-16T03:42:09.422 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.405+0000 7f6a898dc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:42:09.422 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.405+0000 7f6a81ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T03:42:09.422 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.405+0000 7f6a898dc640 10 monclient: get_monmap_and_config success 2024-09-16T03:42:09.422 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.405+0000 7f6a81ffb640 1 -- 172.21.15.42:0/1999992207 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a6c0ab9d0 con 0x7f6a8409a100 2024-09-16T03:42:09.422 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.405+0000 7f6a898dc640 4 set_mon_vals no callback set 2024-09-16T03:42:09.422 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.405+0000 7f6a81ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:09.422 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.405+0000 7f6a81ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:42:09.423 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.405+0000 7f6a81ffb640 10 monclient: dump: 2024-09-16T03:42:09.423 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-09-16T03:42:09.423 INFO:teuthology.misc.health.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:09.423 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:09.423 INFO:teuthology.misc.health.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:09.423 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:09.423 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:09.423 INFO:teuthology.misc.health.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:09.423 INFO:teuthology.misc.health.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:09.423 INFO:teuthology.misc.health.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:09.423 INFO:teuthology.misc.health.smithi042.stderr: 2024-09-16T03:42:09.423 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.405+0000 7f6a898dc640 10 monclient: shutdown 2024-09-16T03:42:09.423 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.405+0000 7f6a898dc640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:42:09.424 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.405+0000 7f6a898dc640 1 -- 172.21.15.42:0/1999992207 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6a8409a100 msgr2=0x7f6a8409e140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:09.424 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.405+0000 7f6a898dc640 1 --2- 172.21.15.42:0/1999992207 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6a8409a100 0x7f6a8409e140 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f6a84099220 tx=0x7f6a6c01c400 comp rx=0 tx=0).stop 2024-09-16T03:42:09.424 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.409+0000 7f6a898dc640 1 -- 172.21.15.42:0/1999992207 shutdown_connections 2024-09-16T03:42:09.424 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.409+0000 7f6a898dc640 1 --2- 172.21.15.42:0/1999992207 >> v2:172.21.15.154:3300/0 conn(0x7f6a840a1ac0 0x7f6a8409e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:09.424 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.409+0000 7f6a898dc640 1 --2- 172.21.15.42:0/1999992207 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6a8409a100 0x7f6a8409e140 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:09.424 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.409+0000 7f6a898dc640 1 --2- 172.21.15.42:0/1999992207 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f6a840a1230 0x7f6a8409dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:09.424 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.409+0000 7f6a898dc640 1 -- 172.21.15.42:0/1999992207 >> 172.21.15.42:0/1999992207 conn(0x7f6a840a24f0 msgr2=0x7f6a840a2900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:42:09.424 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.409+0000 7f6a898dc640 1 -- 172.21.15.42:0/1999992207 shutdown_connections 2024-09-16T03:42:09.424 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.409+0000 7f6a898dc640 1 -- 172.21.15.42:0/1999992207 wait complete. 2024-09-16T03:42:09.424 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.409+0000 7f6a898dc640 10 monclient: build_initial_monmap 2024-09-16T03:42:09.424 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.409+0000 7f6a898dc640 10 monclient: monmap: 2024-09-16T03:42:09.424 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-09-16T03:42:09.424 INFO:teuthology.misc.health.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:42:09.425 INFO:teuthology.misc.health.smithi042.stderr:last_changed 0.000000 2024-09-16T03:42:09.425 INFO:teuthology.misc.health.smithi042.stderr:created 0.000000 2024-09-16T03:42:09.425 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:42:09.425 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:09.425 INFO:teuthology.misc.health.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:42:09.425 INFO:teuthology.misc.health.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:42:09.425 INFO:teuthology.misc.health.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:42:09.425 INFO:teuthology.misc.health.smithi042.stderr: 2024-09-16T03:42:09.426 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.409+0000 7f6a898dc640 1 librados: starting msgr at 2024-09-16T03:42:09.426 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.409+0000 7f6a898dc640 1 librados: starting objecter 2024-09-16T03:42:09.426 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.409+0000 7f6a898dc640 1 Processor -- start 2024-09-16T03:42:09.426 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.409+0000 7f6a898dc640 1 -- start start 2024-09-16T03:42:09.426 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.409+0000 7f6a898dc640 1 librados: setting wanted keys 2024-09-16T03:42:09.426 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.409+0000 7f6a898dc640 1 librados: calling monclient init 2024-09-16T03:42:09.426 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.409+0000 7f6a898dc640 10 monclient: init 2024-09-16T03:42:09.426 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.409+0000 7f6a898dc640 10 monclient: _reopen_session rank -1 2024-09-16T03:42:09.426 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.409+0000 7f6a898dc640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:42:09.426 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.409+0000 7f6a898dc640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f6a8409a100 0x7f6a84097f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:09.426 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.409+0000 7f6a898dc640 10 monclient(hunting): picked mon.noname-c con 0x7f6a8409a100 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:42:09.426 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.409+0000 7f6a898dc640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6a840a1230 0x7f6a84098480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:09.427 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.409+0000 7f6a898dc640 10 monclient(hunting): picked mon.noname-a con 0x7f6a840a1230 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:42:09.427 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.409+0000 7f6a898dc640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6a840a1ac0 0x7f6a8408fa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:09.427 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.409+0000 7f6a898dc640 10 monclient(hunting): picked mon.noname-b con 0x7f6a840a1ac0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:42:09.427 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.409+0000 7f6a898dc640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:09.427 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.409+0000 7f6a898dc640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f6a8409ceb0 con 0x7f6a840a1230 2024-09-16T03:42:09.427 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.409+0000 7f6a898dc640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:09.427 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.409+0000 7f6a898dc640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f6a8409cd30 con 0x7f6a840a1ac0 2024-09-16T03:42:09.427 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.409+0000 7f6a898dc640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:09.427 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.409+0000 7f6a898dc640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f6a8409d030 con 0x7f6a8409a100 2024-09-16T03:42:09.427 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.409+0000 7f6a898dc640 10 monclient(hunting): _renew_subs 2024-09-16T03:42:09.427 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.409+0000 7f6a898dc640 10 monclient(hunting): authenticate will time out at 976.381711s 2024-09-16T03:42:09.427 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.409+0000 7f6a827fc640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6a840a1230 0x7f6a84098480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:09.428 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.409+0000 7f6a82ffd640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f6a8409a100 0x7f6a84097f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:09.428 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.409+0000 7f6a827fc640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6a840a1230 0x7f6a84098480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:53464/0 (socket says 172.21.15.42:53464) 2024-09-16T03:42:09.428 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.409+0000 7f6a827fc640 1 -- 172.21.15.42:0/4247183053 learned_addr learned my addr 172.21.15.42:0/4247183053 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:42:09.428 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.409+0000 7f6a827fc640 10 monclient(hunting): get_auth_request con 0x7f6a840a1230 auth_method 0 2024-09-16T03:42:09.428 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.409+0000 7f6a827fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:09.428 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.409+0000 7f6a827fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:09.428 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.409+0000 7f6a827fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:09.428 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.409+0000 7f6a82ffd640 10 monclient(hunting): get_auth_request con 0x7f6a8409a100 auth_method 0 2024-09-16T03:42:09.428 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.409+0000 7f6a82ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:09.428 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.409+0000 7f6a82ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:09.428 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.409+0000 7f6a82ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:09.429 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.409+0000 7f6a827fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:09.429 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.409+0000 7f6a827fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:09.429 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.409+0000 7f6a827fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:09.429 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.409+0000 7f6a82ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:09.429 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.409+0000 7f6a82ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:09.429 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.409+0000 7f6a82ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:09.429 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.409+0000 7f6a827fc640 10 monclient(hunting): handle_auth_done global_id 4170 payload 824 2024-09-16T03:42:09.429 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.409+0000 7f6a827fc640 1 -- 172.21.15.42:0/4247183053 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f6a8409a100 msgr2=0x7f6a84097f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:09.429 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.409+0000 7f6a837fe640 1 --2- 172.21.15.42:0/4247183053 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6a840a1ac0 0x7f6a8408fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:09.429 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.409+0000 7f6a827fc640 1 --2- 172.21.15.42:0/4247183053 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f6a8409a100 0x7f6a84097f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:09.430 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.409+0000 7f6a827fc640 1 -- 172.21.15.42:0/4247183053 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6a840a1ac0 msgr2=0x7f6a8408fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:09.430 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.409+0000 7f6a827fc640 1 --2- 172.21.15.42:0/4247183053 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6a840a1ac0 0x7f6a8408fa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:09.430 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.409+0000 7f6a827fc640 10 monclient: _finish_hunting 0 2024-09-16T03:42:09.430 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.409+0000 7f6a827fc640 1 monclient: found mon.noname-a 2024-09-16T03:42:09.430 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.409+0000 7f6a827fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:42:09.430 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.409+0000 7f6a827fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:42:09.430 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.409+0000 7f6a827fc640 1 -- 172.21.15.42:0/4247183053 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a84093700 con 0x7f6a840a1230 2024-09-16T03:42:09.430 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.409+0000 7f6a827fc640 1 --2- 172.21.15.42:0/4247183053 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6a840a1230 0x7f6a84098480 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f6a6c01aab0 tx=0x7f6a6c0191a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:09.430 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.409+0000 7f6a7bfff640 10 client.?.objecter ms_handle_connect 0x7f6a840a1230 2024-09-16T03:42:09.430 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.409+0000 7f6a7bfff640 10 client.?.objecter resend_mon_ops 2024-09-16T03:42:09.430 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.409+0000 7f6a7bfff640 1 -- 172.21.15.42:0/4247183053 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a6c0b0040 con 0x7f6a840a1230 2024-09-16T03:42:09.430 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.409+0000 7f6a7bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:09.431 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.409+0000 7f6a7bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:42:09.431 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.409+0000 7f6a7bfff640 10 monclient: dump: 2024-09-16T03:42:09.431 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-09-16T03:42:09.431 INFO:teuthology.misc.health.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:09.431 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:09.431 INFO:teuthology.misc.health.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:09.431 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:09.431 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:09.431 INFO:teuthology.misc.health.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:09.431 INFO:teuthology.misc.health.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:09.431 INFO:teuthology.misc.health.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:09.432 INFO:teuthology.misc.health.smithi042.stderr: 2024-09-16T03:42:09.432 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.409+0000 7f6a7bfff640 10 monclient: _finish_auth 0 2024-09-16T03:42:09.432 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.409+0000 7f6a7bfff640 10 monclient: _check_auth_tickets 2024-09-16T03:42:09.432 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.409+0000 7f6a7bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:42:09.432 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.409+0000 7f6a7bfff640 1 -- 172.21.15.42:0/4247183053 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6a6c0b4420 con 0x7f6a840a1230 2024-09-16T03:42:09.432 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.409+0000 7f6a898dc640 5 monclient: authenticate success, global_id 4170 2024-09-16T03:42:09.432 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.409+0000 7f6a898dc640 10 monclient: _renew_subs 2024-09-16T03:42:09.432 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.409+0000 7f6a898dc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:42:09.432 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.409+0000 7f6a898dc640 1 -- 172.21.15.42:0/4247183053 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a840939e0 con 0x7f6a840a1230 2024-09-16T03:42:09.432 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.409+0000 7f6a7bfff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:42:09.432 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.409+0000 7f6a7bfff640 1 -- 172.21.15.42:0/4247183053 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a6c0b47f0 con 0x7f6a840a1230 2024-09-16T03:42:09.432 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.409+0000 7f6a7bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:09.433 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.409+0000 7f6a7bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:42:09.433 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.409+0000 7f6a7bfff640 10 monclient: dump: 2024-09-16T03:42:09.433 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-09-16T03:42:09.433 INFO:teuthology.misc.health.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:09.433 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:09.433 INFO:teuthology.misc.health.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:09.433 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:09.433 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:09.433 INFO:teuthology.misc.health.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:09.433 INFO:teuthology.misc.health.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:09.434 INFO:teuthology.misc.health.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:09.434 INFO:teuthology.misc.health.smithi042.stderr: 2024-09-16T03:42:09.434 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.409+0000 7f6a817fa640 4 set_mon_vals no callback set 2024-09-16T03:42:09.434 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.409+0000 7f6a898dc640 10 client.4170.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:42:09.434 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.409+0000 7f6a898dc640 10 monclient: _renew_subs 2024-09-16T03:42:09.434 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.409+0000 7f6a898dc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:42:09.434 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.409+0000 7f6a898dc640 1 -- 172.21.15.42:0/4247183053 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a8409c7e0 con 0x7f6a840a1230 2024-09-16T03:42:09.434 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.409+0000 7f6a898dc640 1 librados: init done 2024-09-16T03:42:09.434 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.409+0000 7f6a7bfff640 1 -- 172.21.15.42:0/4247183053 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f6a6c0b4990 con 0x7f6a840a1230 2024-09-16T03:42:09.434 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.409+0000 7f6a7bfff640 1 -- 172.21.15.42:0/4247183053 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(8..8 src has 1..8) ==== 1415+0+0 (secure 0 0 0) 0x7f6a6c0bda40 con 0x7f6a840a1230 2024-09-16T03:42:09.434 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.409+0000 7f6a7bfff640 10 client.4170.objecter ms_dispatch 0x7f6a84000ba0 osd_map(8..8 src has 1..8) 2024-09-16T03:42:09.435 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.409+0000 7f6a7bfff640 3 client.4170.objecter handle_osd_map got epochs [8,8] > 0 2024-09-16T03:42:09.435 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.409+0000 7f6a7bfff640 3 client.4170.objecter handle_osd_map decoding full epoch 8 2024-09-16T03:42:09.435 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.409+0000 7f6a7bfff640 20 client.4170.objecter dump_active .. 0 homeless 2024-09-16T03:42:09.435 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.409+0000 7f6a898dc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:42:09.435 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.409+0000 7f6a898dc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:42:09.435 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.409+0000 7f6a898dc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:42:09.435 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.409+0000 7f6a898dc640 1 -- 172.21.15.42:0/4247183053 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6a84072c30 con 0x7f6a840a1230 2024-09-16T03:42:09.435 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.413+0000 7f6a7bfff640 1 -- 172.21.15.42:0/4247183053 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+74432 (secure 0 0 0) 0x7f6a6c016110 con 0x7f6a840a1230 2024-09-16T03:42:09.435 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.413+0000 7f6a7bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:42:09.435 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.413+0000 7f6a7bfff640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:42:09.478 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.473+0000 7f6a898dc640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-16T03:42:09.478 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.473+0000 7f6a898dc640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T03:42:09.478 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.473+0000 7f6a898dc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:42:09.479 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.473+0000 7f6a898dc640 1 -- 172.21.15.42:0/4247183053 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6a8409d440 con 0x7f6a840a1230 2024-09-16T03:42:09.479 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.473+0000 7f6a7bfff640 1 -- 172.21.15.42:0/4247183053 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v8) ==== 74+0+5605 (secure 0 0 0) 0x7f6a6c0bd400 con 0x7f6a840a1230 2024-09-16T03:42:09.479 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.473+0000 7f6a7bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T03:42:09.479 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.473+0000 7f6a7bfff640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:42:09.480 INFO:teuthology.misc.health.smithi042.stdout: 2024-09-16T03:42:09.480 INFO:teuthology.misc.health.smithi042.stdout:{"epoch":8,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","created":"2024-09-16T03:42:05.977277+0000","modified":"2024-09-16T03:42:08.437271+0000","last_up_change":"0.000000","last_in_change":"2024-09-16T03:42:08.437271+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6b9fc058-2cff-46af-b95c-b78c72e1dc2b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ee31f807-cb16-4c33-8e23-2573e1a6b5c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a26f3917-6702-43db-9355-e96da86a3365","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4e809f2a-2ffd-4791-9f2d-18f1bfe7d2ea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"dcf637ab-5f72-4ed2-a99b-69083f0b6b15","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"83c05e44-861b-4bb2-9657-19196e5ac1f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T03:42:09.481 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.473+0000 7f6a898dc640 10 librados: watch_flush enter 2024-09-16T03:42:09.481 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.473+0000 7f6a898dc640 10 librados: watch_flush exit 2024-09-16T03:42:09.481 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.473+0000 7f6a898dc640 20 client.4170.objecter shutdown clearing up homeless session... 2024-09-16T03:42:09.481 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.473+0000 7f6a898dc640 10 client.4170.objecter successfully canceled tick 2024-09-16T03:42:09.481 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.473+0000 7f6a898dc640 10 monclient: shutdown 2024-09-16T03:42:09.481 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.473+0000 7f6a898dc640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:42:09.481 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.473+0000 7f6a898dc640 1 -- 172.21.15.42:0/4247183053 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6a840a1230 msgr2=0x7f6a84098480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:09.481 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.473+0000 7f6a898dc640 1 --2- 172.21.15.42:0/4247183053 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6a840a1230 0x7f6a84098480 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f6a6c01aab0 tx=0x7f6a6c0191a0 comp rx=0 tx=0).stop 2024-09-16T03:42:09.481 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.473+0000 7f6a898dc640 1 -- 172.21.15.42:0/4247183053 shutdown_connections 2024-09-16T03:42:09.482 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.473+0000 7f6a898dc640 1 --2- 172.21.15.42:0/4247183053 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6a840a1ac0 0x7f6a8408fa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:09.482 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.473+0000 7f6a898dc640 1 --2- 172.21.15.42:0/4247183053 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6a840a1230 0x7f6a84098480 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:09.482 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.473+0000 7f6a898dc640 1 --2- 172.21.15.42:0/4247183053 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f6a8409a100 0x7f6a84097f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:09.482 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.473+0000 7f6a898dc640 1 -- 172.21.15.42:0/4247183053 >> 172.21.15.42:0/4247183053 conn(0x7f6a840a24f0 msgr2=0x7f6a84073180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:42:09.482 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.473+0000 7f6a898dc640 1 -- 172.21.15.42:0/4247183053 shutdown_connections 2024-09-16T03:42:09.482 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.473+0000 7f6a898dc640 1 -- 172.21.15.42:0/4247183053 wait complete. 2024-09-16T03:42:09.482 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:09.473+0000 7f6a898dc640 1 librados: shutdown 2024-09-16T03:42:09.491 INFO:tasks.ceph.mgr.x.smithi154.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2024-09-16T03:42:09.491 INFO:tasks.ceph.mgr.x.smithi154.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2024-09-16T03:42:09.491 INFO:tasks.ceph.mgr.x.smithi154.stderr: from numpy import show_config as show_numpy_config 2024-09-16T03:42:09.495 DEBUG:teuthology.misc:0 of 6 OSDs are up 2024-09-16T03:42:09.567 INFO:tasks.ceph.mgr.x.smithi154.stderr:2024-09-16T03:42:09.566+0000 7f8fbbf11180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-16T03:42:09.620 INFO:tasks.ceph.osd.3.smithi154.stderr:2024-09-16T03:42:09.618+0000 7fa63ba6f7c0 -1 Falling back to public interface 2024-09-16T03:42:09.635 INFO:tasks.ceph.osd.1.smithi042.stderr:2024-09-16T03:42:09.629+0000 7f457cef17c0 -1 Falling back to public interface 2024-09-16T03:42:09.661 INFO:tasks.ceph.osd.2.smithi042.stderr:2024-09-16T03:42:09.653+0000 7fe6faffe7c0 -1 Falling back to public interface 2024-09-16T03:42:09.753 INFO:tasks.ceph.mgr.x.smithi154.stderr:2024-09-16T03:42:09.750+0000 7f8fbbf11180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-16T03:42:09.860 INFO:tasks.ceph.osd.0.smithi042.stderr:2024-09-16T03:42:09.853+0000 7f7de69347c0 -1 Falling back to public interface 2024-09-16T03:42:09.916 INFO:tasks.ceph.mgr.x.smithi154.stderr:2024-09-16T03:42:09.914+0000 7f8fbbf11180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-16T03:42:09.987 INFO:tasks.ceph.mgr.x.smithi154.stderr:2024-09-16T03:42:09.986+0000 7f8fbbf11180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-16T03:42:10.137 INFO:tasks.ceph.mgr.x.smithi154.stderr:2024-09-16T03:42:10.134+0000 7f8fbbf11180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-16T03:42:10.148 INFO:tasks.ceph.osd.4.smithi154.stderr:2024-09-16T03:42:10.146+0000 7fddc26577c0 -1 Falling back to public interface 2024-09-16T03:42:10.588 INFO:tasks.ceph.osd.5.smithi154.stderr:2024-09-16T03:42:10.586+0000 7f058f2327c0 -1 Falling back to public interface 2024-09-16T03:42:10.753 INFO:tasks.ceph.mgr.x.smithi154.stderr:2024-09-16T03:42:10.750+0000 7f8fbbf11180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-16T03:42:10.829 INFO:tasks.ceph.mgr.x.smithi154.stderr:2024-09-16T03:42:10.826+0000 7f8fbbf11180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-16T03:42:11.012 INFO:tasks.ceph.mgr.x.smithi154.stderr:2024-09-16T03:42:11.010+0000 7f8fbbf11180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-16T03:42:11.084 INFO:tasks.ceph.mgr.x.smithi154.stderr:2024-09-16T03:42:11.082+0000 7f8fbbf11180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-16T03:42:11.374 INFO:tasks.ceph.mgr.x.smithi154.stderr:2024-09-16T03:42:11.374+0000 7f8fbbf11180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-16T03:42:11.436 INFO:tasks.ceph.mgr.x.smithi154.stderr:2024-09-16T03:42:11.434+0000 7f8fbbf11180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-16T03:42:12.472 INFO:tasks.ceph.mgr.x.smithi154.stderr:2024-09-16T03:42:12.470+0000 7f8f62a48640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-09-16T03:42:12.472 INFO:tasks.ceph.mgr.x.smithi154.stderr:2024-09-16T03:42:12.470+0000 7f8f62a48640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-09-16T03:42:13.575 INFO:tasks.ceph.osd.3.smithi154.stderr:2024-09-16T03:42:13.574+0000 7fa63ba6f7c0 -1 osd.3 0 log_to_monitors true 2024-09-16T03:42:13.639 INFO:tasks.ceph.osd.1.smithi042.stderr:2024-09-16T03:42:13.633+0000 7f457cef17c0 -1 osd.1 0 log_to_monitors true 2024-09-16T03:42:13.900 INFO:tasks.ceph.osd.2.smithi042.stderr:2024-09-16T03:42:13.893+0000 7fe6faffe7c0 -1 osd.2 0 log_to_monitors true 2024-09-16T03:42:14.087 INFO:tasks.ceph.osd.0.smithi042.stderr:2024-09-16T03:42:14.081+0000 7f7de69347c0 -1 osd.0 0 log_to_monitors true 2024-09-16T03:42:14.160 INFO:tasks.ceph.osd.4.smithi154.stderr:2024-09-16T03:42:14.158+0000 7fddc26577c0 -1 osd.4 0 log_to_monitors true 2024-09-16T03:42:14.901 INFO:tasks.ceph.osd.5.smithi154.stderr:2024-09-16T03:42:14.898+0000 7f058f2327c0 -1 osd.5 0 log_to_monitors true 2024-09-16T03:42:15.538 INFO:tasks.ceph.osd.2.smithi042.stderr:2024-09-16T03:42:15.533+0000 7fe6f6fa7640 -1 osd.2 0 waiting for initial osdmap 2024-09-16T03:42:15.541 INFO:tasks.ceph.osd.0.smithi042.stderr:2024-09-16T03:42:15.533+0000 7f7de28dd640 -1 osd.0 0 waiting for initial osdmap 2024-09-16T03:42:15.544 INFO:tasks.ceph.osd.3.smithi154.stderr:2024-09-16T03:42:15.542+0000 7fa637a18640 -1 osd.3 0 waiting for initial osdmap 2024-09-16T03:42:15.544 INFO:tasks.ceph.osd.4.smithi154.stderr:2024-09-16T03:42:15.542+0000 7fddbe600640 -1 osd.4 0 waiting for initial osdmap 2024-09-16T03:42:15.548 INFO:tasks.ceph.osd.0.smithi042.stderr:2024-09-16T03:42:15.541+0000 7f7dde71a640 -1 osd.0 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T03:42:15.548 INFO:tasks.ceph.osd.2.smithi042.stderr:2024-09-16T03:42:15.541+0000 7fe6f2de4640 -1 osd.2 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T03:42:15.550 INFO:tasks.ceph.osd.3.smithi154.stderr:2024-09-16T03:42:15.546+0000 7fa633855640 -1 osd.3 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T03:42:15.551 INFO:tasks.ceph.osd.4.smithi154.stderr:2024-09-16T03:42:15.550+0000 7fddba43d640 -1 osd.4 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T03:42:16.100 DEBUG:teuthology.orchestra.run.smithi042:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-16T03:42:16.216 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.209+0000 7f250418a640 10 monclient: get_monmap_and_config 2024-09-16T03:42:16.216 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.209+0000 7f250418a640 10 monclient: build_initial_monmap 2024-09-16T03:42:16.216 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.209+0000 7f250418a640 10 monclient: monmap: 2024-09-16T03:42:16.216 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-09-16T03:42:16.216 INFO:teuthology.misc.health.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:16.216 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-09-16T03:42:16.214264+0000 2024-09-16T03:42:16.217 INFO:teuthology.misc.health.smithi042.stderr:created 2024-09-16T03:42:16.214264+0000 2024-09-16T03:42:16.217 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:42:16.217 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:16.217 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:42:16.217 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:42:16.217 INFO:teuthology.misc.health.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:42:16.217 INFO:teuthology.misc.health.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:42:16.217 INFO:teuthology.misc.health.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:42:16.217 INFO:teuthology.misc.health.smithi042.stderr: 2024-09-16T03:42:16.220 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.209+0000 7f250418a640 1 Processor -- start 2024-09-16T03:42:16.220 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.209+0000 7f250418a640 1 -- start start 2024-09-16T03:42:16.220 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.209+0000 7f250418a640 10 monclient: init 2024-09-16T03:42:16.220 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.209+0000 7f250418a640 10 monclient: _reopen_session rank -1 2024-09-16T03:42:16.220 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.209+0000 7f250418a640 10 monclient: _add_conns ranks=[4,2,0,1,3] 2024-09-16T03:42:16.220 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.209+0000 7f250418a640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f24fc0a4450 0x7f24fc0a4830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:16.220 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.209+0000 7f250418a640 10 monclient(hunting): picked mon.noname-c con 0x7f24fc0a4450 addr v2:172.21.15.154:3300/0 2024-09-16T03:42:16.221 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.209+0000 7f250418a640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f24fc0a5c30 0x7f24fc0a4d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:16.221 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.209+0000 7f250418a640 10 monclient(hunting): picked mon.noname-b con 0x7f24fc0a5c30 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:42:16.221 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.209+0000 7f250418a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f24fc137210 addr v1:172.21.15.42:6789/0 2024-09-16T03:42:16.221 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.209+0000 7f250418a640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f24fc09c2e0 con 0x7f24fc137210 2024-09-16T03:42:16.221 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.209+0000 7f250418a640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:16.221 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.209+0000 7f250418a640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f24fc0a52b0 con 0x7f24fc0a4450 2024-09-16T03:42:16.221 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.209+0000 7f250418a640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:16.221 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.209+0000 7f250418a640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f24fc073790 con 0x7f24fc0a5c30 2024-09-16T03:42:16.221 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.209+0000 7f250418a640 10 monclient(hunting): _renew_subs 2024-09-16T03:42:16.221 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.209+0000 7f250418a640 10 monclient(hunting): authenticate will time out at 983.183352s 2024-09-16T03:42:16.221 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.209+0000 7f2502727640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f24fc137210 0x7f24fc1376a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:49316/0 (socket says 172.21.15.42:49316) 2024-09-16T03:42:16.222 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.209+0000 7f2502727640 1 -- 172.21.15.42:0/3780033143 learned_addr learned my addr 172.21.15.42:0/3780033143 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:42:16.222 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.209+0000 7f2501725640 1 --2- 172.21.15.42:0/3780033143 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f24fc0a5c30 0x7f24fc0a4d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:16.222 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.209+0000 7f2501725640 10 monclient(hunting): get_auth_request con 0x7f24fc0a5c30 auth_method 0 2024-09-16T03:42:16.222 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.209+0000 7f2501725640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:16.222 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.209+0000 7f2501725640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:16.222 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.209+0000 7f2501725640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:16.222 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.209+0000 7f2501725640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:16.222 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.209+0000 7f2501725640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:16.222 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.209+0000 7f2501725640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:16.222 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.209+0000 7f2501725640 10 monclient(hunting): handle_auth_done global_id 4175 payload 274 2024-09-16T03:42:16.222 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.209+0000 7f2501725640 1 -- 172.21.15.42:0/3780033143 >> v2:172.21.15.154:3300/0 conn(0x7f24fc0a4450 msgr2=0x7f24fc0a4830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:42:16.222 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.209+0000 7f2501f26640 1 --2- 172.21.15.42:0/3780033143 >> v2:172.21.15.154:3300/0 conn(0x7f24fc0a4450 0x7f24fc0a4830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:16.222 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.209+0000 7f2500f24640 1 -- 172.21.15.42:0/3780033143 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2614734329 0 0) 0x7f24fc09c2e0 con 0x7f24fc137210 2024-09-16T03:42:16.223 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.209+0000 7f2501725640 1 --2- 172.21.15.42:0/3780033143 >> v2:172.21.15.154:3300/0 conn(0x7f24fc0a4450 0x7f24fc0a4830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:16.223 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.209+0000 7f2501725640 1 -- 172.21.15.42:0/3780033143 >> v1:172.21.15.42:6789/0 conn(0x7f24fc137210 legacy=0x7f24fc1376a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:16.223 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.209+0000 7f2501725640 10 monclient: _finish_hunting 0 2024-09-16T03:42:16.223 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.209+0000 7f2501725640 1 monclient: found mon.noname-b 2024-09-16T03:42:16.223 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.209+0000 7f2501725640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:42:16.223 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.209+0000 7f2501725640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T03:42:16.223 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.209+0000 7f2501725640 1 -- 172.21.15.42:0/3780033143 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24fc09ba40 con 0x7f24fc0a5c30 2024-09-16T03:42:16.223 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.209+0000 7f2501f26640 10 monclient: get_auth_request con 0x7f24fc0a4450 auth_method 0 2024-09-16T03:42:16.223 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.209+0000 7f2501f26640 1 --2- 172.21.15.42:0/3780033143 >> v2:172.21.15.154:3300/0 conn(0x7f24fc0a4450 0x7f24fc0a4830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T03:42:16.223 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.213+0000 7f2500f24640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T03:42:16.223 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.213+0000 7f2501725640 1 --2- 172.21.15.42:0/3780033143 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f24fc0a5c30 0x7f24fc0a4d70 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f24ec0097c0 tx=0x7f24ec01cd90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1e6eb0e77085fa49 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:16.223 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.213+0000 7f2500f24640 1 -- 172.21.15.42:0/3780033143 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24ec0b0070 con 0x7f24fc0a5c30 2024-09-16T03:42:16.223 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.213+0000 7f2500f24640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:16.223 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.213+0000 7f2500f24640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:42:16.224 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.213+0000 7f2500f24640 10 monclient: dump: 2024-09-16T03:42:16.224 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-09-16T03:42:16.224 INFO:teuthology.misc.health.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:16.224 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:16.224 INFO:teuthology.misc.health.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:16.224 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:16.224 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:16.224 INFO:teuthology.misc.health.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:16.224 INFO:teuthology.misc.health.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:16.224 INFO:teuthology.misc.health.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:16.224 INFO:teuthology.misc.health.smithi042.stderr: 2024-09-16T03:42:16.224 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.213+0000 7f2500f24640 10 monclient: _finish_auth 0 2024-09-16T03:42:16.224 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.213+0000 7f2500f24640 10 monclient: _check_auth_tickets 2024-09-16T03:42:16.224 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.213+0000 7f2500f24640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:42:16.225 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.213+0000 7f2500f24640 1 -- 172.21.15.42:0/3780033143 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f24ec004230 con 0x7f24fc0a5c30 2024-09-16T03:42:16.225 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.213+0000 7f250418a640 5 monclient: authenticate success, global_id 4175 2024-09-16T03:42:16.225 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.213+0000 7f250418a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:42:16.225 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.213+0000 7f2500f24640 10 monclient: handle_config config(0 keys) 2024-09-16T03:42:16.225 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.213+0000 7f250418a640 10 monclient: get_monmap_and_config success 2024-09-16T03:42:16.225 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.213+0000 7f250418a640 4 set_mon_vals no callback set 2024-09-16T03:42:16.225 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.213+0000 7f2500f24640 1 -- 172.21.15.42:0/3780033143 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24ec003620 con 0x7f24fc0a5c30 2024-09-16T03:42:16.225 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.213+0000 7f2500f24640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:16.225 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.213+0000 7f2500f24640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:42:16.225 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.213+0000 7f2500f24640 10 monclient: dump: 2024-09-16T03:42:16.225 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-09-16T03:42:16.225 INFO:teuthology.misc.health.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:16.225 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:16.226 INFO:teuthology.misc.health.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:16.226 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:16.226 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:16.226 INFO:teuthology.misc.health.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:16.226 INFO:teuthology.misc.health.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:16.226 INFO:teuthology.misc.health.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:16.226 INFO:teuthology.misc.health.smithi042.stderr: 2024-09-16T03:42:16.230 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.213+0000 7f250418a640 10 monclient: shutdown 2024-09-16T03:42:16.230 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.213+0000 7f250418a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:42:16.230 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.213+0000 7f250418a640 1 -- 172.21.15.42:0/3780033143 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f24fc0a5c30 msgr2=0x7f24fc0a4d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:16.230 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.213+0000 7f250418a640 1 --2- 172.21.15.42:0/3780033143 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f24fc0a5c30 0x7f24fc0a4d70 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f24ec0097c0 tx=0x7f24ec01cd90 comp rx=0 tx=0).stop 2024-09-16T03:42:16.230 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.213+0000 7f250418a640 1 -- 172.21.15.42:0/3780033143 shutdown_connections 2024-09-16T03:42:16.230 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.213+0000 7f250418a640 1 --2- 172.21.15.42:0/3780033143 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f24fc0a5c30 0x7f24fc0a4d70 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:16.231 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.213+0000 7f250418a640 1 --2- 172.21.15.42:0/3780033143 >> v2:172.21.15.154:3300/0 conn(0x7f24fc0a4450 0x7f24fc0a4830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:16.231 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.213+0000 7f250418a640 1 -- 172.21.15.42:0/3780033143 >> 172.21.15.42:0/3780033143 conn(0x7f24fc0a9e50 msgr2=0x7f24fc0aa260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:42:16.231 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.213+0000 7f250418a640 1 -- 172.21.15.42:0/3780033143 shutdown_connections 2024-09-16T03:42:16.231 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.213+0000 7f250418a640 1 -- 172.21.15.42:0/3780033143 wait complete. 2024-09-16T03:42:16.231 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.213+0000 7f250418a640 10 monclient: build_initial_monmap 2024-09-16T03:42:16.231 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.213+0000 7f250418a640 10 monclient: monmap: 2024-09-16T03:42:16.231 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-09-16T03:42:16.231 INFO:teuthology.misc.health.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:42:16.231 INFO:teuthology.misc.health.smithi042.stderr:last_changed 0.000000 2024-09-16T03:42:16.231 INFO:teuthology.misc.health.smithi042.stderr:created 0.000000 2024-09-16T03:42:16.232 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:42:16.232 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:16.232 INFO:teuthology.misc.health.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:42:16.232 INFO:teuthology.misc.health.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:42:16.232 INFO:teuthology.misc.health.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:42:16.232 INFO:teuthology.misc.health.smithi042.stderr: 2024-09-16T03:42:16.232 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.213+0000 7f250418a640 1 librados: starting msgr at 2024-09-16T03:42:16.232 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.213+0000 7f250418a640 1 librados: starting objecter 2024-09-16T03:42:16.232 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.217+0000 7f250418a640 1 Processor -- start 2024-09-16T03:42:16.232 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.217+0000 7f250418a640 1 -- start start 2024-09-16T03:42:16.232 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.217+0000 7f250418a640 1 librados: setting wanted keys 2024-09-16T03:42:16.232 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.217+0000 7f250418a640 1 librados: calling monclient init 2024-09-16T03:42:16.232 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.217+0000 7f250418a640 10 monclient: init 2024-09-16T03:42:16.233 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.217+0000 7f250418a640 10 monclient: _reopen_session rank -1 2024-09-16T03:42:16.233 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.217+0000 7f250418a640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:42:16.233 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.217+0000 7f250418a640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f24fc0a4450 0x7f24fc1359d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:16.233 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.217+0000 7f250418a640 10 monclient(hunting): picked mon.noname-a con 0x7f24fc0a4450 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:42:16.233 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.217+0000 7f250418a640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f24fc0a5c30 0x7f24fc135f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:16.233 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.217+0000 7f250418a640 10 monclient(hunting): picked mon.noname-c con 0x7f24fc0a5c30 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:42:16.233 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.217+0000 7f250418a640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f24fc137210 0x7f24fc13c050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:16.233 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.217+0000 7f250418a640 10 monclient(hunting): picked mon.noname-b con 0x7f24fc137210 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:42:16.233 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.217+0000 7f250418a640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:16.233 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.217+0000 7f250418a640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f24fc141ea0 con 0x7f24fc0a4450 2024-09-16T03:42:16.233 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.217+0000 7f250418a640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:16.233 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.217+0000 7f250418a640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f24fc072b20 con 0x7f24fc137210 2024-09-16T03:42:16.233 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.217+0000 7f250418a640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:16.233 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.217+0000 7f250418a640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f24fc142020 con 0x7f24fc0a5c30 2024-09-16T03:42:16.234 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.217+0000 7f250418a640 10 monclient(hunting): _renew_subs 2024-09-16T03:42:16.234 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.217+0000 7f250418a640 10 monclient(hunting): authenticate will time out at 983.188634s 2024-09-16T03:42:16.234 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.217+0000 7f2501f26640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f24fc0a4450 0x7f24fc1359d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:16.234 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.217+0000 7f2501725640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f24fc0a5c30 0x7f24fc135f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:16.234 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.217+0000 7f2501f26640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f24fc0a4450 0x7f24fc1359d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:35708/0 (socket says 172.21.15.42:35708) 2024-09-16T03:42:16.234 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.217+0000 7f2501725640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f24fc0a5c30 0x7f24fc135f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:36124/0 (socket says 172.21.15.42:36124) 2024-09-16T03:42:16.234 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.217+0000 7f2501f26640 1 -- 172.21.15.42:0/2562331424 learned_addr learned my addr 172.21.15.42:0/2562331424 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:42:16.234 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.217+0000 7f2501f26640 10 monclient(hunting): get_auth_request con 0x7f24fc0a4450 auth_method 0 2024-09-16T03:42:16.234 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.217+0000 7f2501f26640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:16.234 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.217+0000 7f2501f26640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:16.234 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.217+0000 7f2501f26640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:16.234 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.217+0000 7f2501725640 10 monclient(hunting): get_auth_request con 0x7f24fc0a5c30 auth_method 0 2024-09-16T03:42:16.234 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.217+0000 7f2501725640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:16.234 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.217+0000 7f2501725640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:16.235 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.217+0000 7f2501725640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:16.235 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.217+0000 7f2501725640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:16.235 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.217+0000 7f2501725640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:16.235 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.217+0000 7f2501725640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:16.235 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.217+0000 7f2501f26640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:16.235 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.217+0000 7f2501f26640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:16.235 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.217+0000 7f2501f26640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:16.236 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.217+0000 7f2502727640 1 --2- 172.21.15.42:0/2562331424 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f24fc137210 0x7f24fc13c050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:16.236 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.217+0000 7f2502727640 10 monclient(hunting): get_auth_request con 0x7f24fc137210 auth_method 0 2024-09-16T03:42:16.236 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.217+0000 7f2502727640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:16.236 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.217+0000 7f2502727640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:16.236 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.217+0000 7f2502727640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:16.236 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.217+0000 7f2501725640 10 monclient(hunting): handle_auth_done global_id 4178 payload 824 2024-09-16T03:42:16.236 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.217+0000 7f2501725640 1 -- 172.21.15.42:0/2562331424 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f24fc137210 msgr2=0x7f24fc13c050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:16.236 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.217+0000 7f2501725640 1 --2- 172.21.15.42:0/2562331424 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f24fc137210 0x7f24fc13c050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:16.237 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.217+0000 7f2501725640 1 -- 172.21.15.42:0/2562331424 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f24fc0a4450 msgr2=0x7f24fc1359d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:16.237 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.217+0000 7f2501725640 1 --2- 172.21.15.42:0/2562331424 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f24fc0a4450 0x7f24fc1359d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:16.237 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.217+0000 7f2501725640 10 monclient: _finish_hunting 0 2024-09-16T03:42:16.237 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.217+0000 7f2501725640 1 monclient: found mon.noname-c 2024-09-16T03:42:16.237 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.217+0000 7f2501725640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:42:16.237 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.217+0000 7f2501725640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:42:16.237 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.217+0000 7f2501725640 1 -- 172.21.15.42:0/2562331424 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24fc0a9400 con 0x7f24fc0a5c30 2024-09-16T03:42:16.237 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.217+0000 7f2501725640 1 --2- 172.21.15.42:0/2562331424 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f24fc0a5c30 0x7f24fc135f10 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f24fc0a5430 tx=0x7f24ec0190d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:16.237 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.217+0000 7f24f27fc640 10 client.?.objecter ms_handle_connect 0x7f24fc0a5c30 2024-09-16T03:42:16.237 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.217+0000 7f24f27fc640 10 client.?.objecter resend_mon_ops 2024-09-16T03:42:16.237 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.217+0000 7f24f27fc640 1 -- 172.21.15.42:0/2562331424 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24ec0b9020 con 0x7f24fc0a5c30 2024-09-16T03:42:16.237 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.217+0000 7f24f27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:16.237 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.217+0000 7f24f27fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:42:16.238 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.217+0000 7f24f27fc640 10 monclient: dump: 2024-09-16T03:42:16.238 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-09-16T03:42:16.238 INFO:teuthology.misc.health.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:16.238 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:16.238 INFO:teuthology.misc.health.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:16.238 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:16.238 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:16.238 INFO:teuthology.misc.health.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:16.238 INFO:teuthology.misc.health.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:16.238 INFO:teuthology.misc.health.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:16.238 INFO:teuthology.misc.health.smithi042.stderr: 2024-09-16T03:42:16.238 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.217+0000 7f24f27fc640 10 monclient: _finish_auth 0 2024-09-16T03:42:16.238 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.217+0000 7f24f27fc640 10 monclient: _check_auth_tickets 2024-09-16T03:42:16.238 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.217+0000 7f24f27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:42:16.239 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.217+0000 7f24f27fc640 1 -- 172.21.15.42:0/2562331424 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f24ec003ee0 con 0x7f24fc0a5c30 2024-09-16T03:42:16.239 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.217+0000 7f250418a640 5 monclient: authenticate success, global_id 4178 2024-09-16T03:42:16.239 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.217+0000 7f250418a640 10 monclient: _renew_subs 2024-09-16T03:42:16.239 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.217+0000 7f250418a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:42:16.239 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.217+0000 7f250418a640 1 -- 172.21.15.42:0/2562331424 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24fc120700 con 0x7f24fc0a5c30 2024-09-16T03:42:16.239 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.217+0000 7f24f27fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:42:16.239 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.217+0000 7f24f27fc640 1 -- 172.21.15.42:0/2562331424 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24ec0b0040 con 0x7f24fc0a5c30 2024-09-16T03:42:16.239 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.217+0000 7f24f27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:16.239 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.217+0000 7f24f27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:42:16.239 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.217+0000 7f24f27fc640 10 monclient: dump: 2024-09-16T03:42:16.239 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-09-16T03:42:16.239 INFO:teuthology.misc.health.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:16.239 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:16.239 INFO:teuthology.misc.health.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:16.240 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:16.240 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:16.240 INFO:teuthology.misc.health.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:16.240 INFO:teuthology.misc.health.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:16.240 INFO:teuthology.misc.health.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:16.240 INFO:teuthology.misc.health.smithi042.stderr: 2024-09-16T03:42:16.240 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.217+0000 7f24f3fff640 4 set_mon_vals no callback set 2024-09-16T03:42:16.240 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.217+0000 7f250418a640 10 client.4178.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:42:16.240 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.217+0000 7f250418a640 10 monclient: _renew_subs 2024-09-16T03:42:16.240 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.217+0000 7f250418a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:42:16.240 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.217+0000 7f250418a640 1 -- 172.21.15.42:0/2562331424 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f24fc0a2420 con 0x7f24fc0a5c30 2024-09-16T03:42:16.240 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.217+0000 7f250418a640 1 librados: init done 2024-09-16T03:42:16.240 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.217+0000 7f24f27fc640 1 -- 172.21.15.42:0/2562331424 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 4) ==== 43962+0+0 (secure 0 0 0) 0x7f24ec0bc020 con 0x7f24fc0a5c30 2024-09-16T03:42:16.241 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.217+0000 7f250418a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:42:16.241 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.217+0000 7f250418a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:42:16.241 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.217+0000 7f250418a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:42:16.241 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.217+0000 7f250418a640 1 -- 172.21.15.42:0/2562331424 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f24fc1376a0 con 0x7f24fc0a5c30 2024-09-16T03:42:16.241 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.217+0000 7f24f27fc640 1 --2- 172.21.15.42:0/2562331424 >> v2:172.21.15.154:6812/254845342 conn(0x7f24e0037780 0x7f24e0039c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:16.241 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.217+0000 7f24f27fc640 1 -- 172.21.15.42:0/2562331424 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(10..10 src has 1..10) ==== 1911+0+0 (secure 0 0 0) 0x7f24ec0e1f50 con 0x7f24fc0a5c30 2024-09-16T03:42:16.241 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.217+0000 7f24f27fc640 10 client.4178.objecter ms_dispatch 0x7f24fc000ba0 osd_map(10..10 src has 1..10) 2024-09-16T03:42:16.241 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.217+0000 7f24f27fc640 3 client.4178.objecter handle_osd_map got epochs [10,10] > 0 2024-09-16T03:42:16.241 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.217+0000 7f24f27fc640 3 client.4178.objecter handle_osd_map decoding full epoch 10 2024-09-16T03:42:16.241 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.217+0000 7f24f27fc640 20 client.4178.objecter dump_active .. 0 homeless 2024-09-16T03:42:16.241 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.217+0000 7f2501f26640 1 --2- 172.21.15.42:0/2562331424 >> v2:172.21.15.154:6812/254845342 conn(0x7f24e0037780 0x7f24e0039c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:16.241 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.217+0000 7f2501f26640 10 monclient: get_auth_request con 0x7f24e0037780 auth_method 0 2024-09-16T03:42:16.241 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.221+0000 7f24f27fc640 10 client.4178.objecter ms_handle_connect 0x7f24e0037780 2024-09-16T03:42:16.242 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.221+0000 7f2501f26640 1 --2- 172.21.15.42:0/2562331424 >> v2:172.21.15.154:6812/254845342 conn(0x7f24e0037780 0x7f24e0039c40 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f24f8006fd0 tx=0x7f24f8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:16.242 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.229+0000 7f24f27fc640 1 -- 172.21.15.42:0/2562331424 <== mon.2 v2:172.21.15.42:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f24ec0c3030 con 0x7f24fc0a5c30 2024-09-16T03:42:16.242 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.229+0000 7f24f27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:42:16.242 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.229+0000 7f24f27fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:42:16.247 INFO:tasks.ceph.osd.1.smithi042.stderr:2024-09-16T03:42:16.241+0000 7f4578e9a640 -1 osd.1 0 waiting for initial osdmap 2024-09-16T03:42:16.250 INFO:tasks.ceph.osd.1.smithi042.stderr:2024-09-16T03:42:16.245+0000 7f4570ccf640 -1 osd.1 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T03:42:16.401 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.393+0000 7f250418a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-16T03:42:16.401 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.393+0000 7f250418a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T03:42:16.401 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.393+0000 7f250418a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:42:16.401 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.393+0000 7f250418a640 1 -- 172.21.15.42:0/2562331424 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f24fc13e5d0 con 0x7f24fc0a5c30 2024-09-16T03:42:16.401 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.393+0000 7f24f27fc640 1 -- 172.21.15.42:0/2562331424 <== mon.2 v2:172.21.15.42:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+5606 (secure 0 0 0) 0x7f24fc0a9400 con 0x7f24fc0a5c30 2024-09-16T03:42:16.401 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.393+0000 7f24f27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T03:42:16.402 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.393+0000 7f24f27fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:42:16.402 INFO:teuthology.misc.health.smithi042.stdout: 2024-09-16T03:42:16.402 INFO:teuthology.misc.health.smithi042.stdout:{"epoch":10,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","created":"2024-09-16T03:42:05.977277+0000","modified":"2024-09-16T03:42:15.524295+0000","last_up_change":"0.000000","last_in_change":"2024-09-16T03:42:08.437271+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6b9fc058-2cff-46af-b95c-b78c72e1dc2b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ee31f807-cb16-4c33-8e23-2573e1a6b5c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a26f3917-6702-43db-9355-e96da86a3365","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4e809f2a-2ffd-4791-9f2d-18f1bfe7d2ea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"dcf637ab-5f72-4ed2-a99b-69083f0b6b15","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"83c05e44-861b-4bb2-9657-19196e5ac1f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T03:42:16.403 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.397+0000 7f250418a640 10 librados: watch_flush enter 2024-09-16T03:42:16.404 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.397+0000 7f250418a640 10 librados: watch_flush exit 2024-09-16T03:42:16.404 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.397+0000 7f250418a640 20 client.4178.objecter shutdown clearing up homeless session... 2024-09-16T03:42:16.404 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.397+0000 7f250418a640 10 client.4178.objecter successfully canceled tick 2024-09-16T03:42:16.404 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.397+0000 7f250418a640 1 -- 172.21.15.42:0/2562331424 >> v2:172.21.15.154:6812/254845342 conn(0x7f24e0037780 msgr2=0x7f24e0039c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:16.404 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.397+0000 7f250418a640 1 --2- 172.21.15.42:0/2562331424 >> v2:172.21.15.154:6812/254845342 conn(0x7f24e0037780 0x7f24e0039c40 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f24f8006fd0 tx=0x7f24f8008040 comp rx=0 tx=0).stop 2024-09-16T03:42:16.404 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.397+0000 7f250418a640 10 monclient: shutdown 2024-09-16T03:42:16.404 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.397+0000 7f250418a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:42:16.404 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.397+0000 7f250418a640 1 -- 172.21.15.42:0/2562331424 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f24fc0a5c30 msgr2=0x7f24fc135f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:16.404 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.397+0000 7f250418a640 1 --2- 172.21.15.42:0/2562331424 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f24fc0a5c30 0x7f24fc135f10 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f24fc0a5430 tx=0x7f24ec0190d0 comp rx=0 tx=0).stop 2024-09-16T03:42:16.404 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.397+0000 7f250418a640 1 -- 172.21.15.42:0/2562331424 shutdown_connections 2024-09-16T03:42:16.404 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.397+0000 7f250418a640 1 --2- 172.21.15.42:0/2562331424 >> v2:172.21.15.154:6812/254845342 conn(0x7f24e0037780 0x7f24e0039c40 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:16.405 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.397+0000 7f250418a640 1 --2- 172.21.15.42:0/2562331424 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f24fc137210 0x7f24fc13c050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:16.405 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.397+0000 7f250418a640 1 --2- 172.21.15.42:0/2562331424 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f24fc0a5c30 0x7f24fc135f10 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:16.405 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.397+0000 7f250418a640 1 --2- 172.21.15.42:0/2562331424 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f24fc0a4450 0x7f24fc1359d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:16.405 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.397+0000 7f250418a640 1 -- 172.21.15.42:0/2562331424 >> 172.21.15.42:0/2562331424 conn(0x7f24fc0a9e50 msgr2=0x7f24fc09fb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:42:16.405 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.397+0000 7f250418a640 1 -- 172.21.15.42:0/2562331424 shutdown_connections 2024-09-16T03:42:16.405 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.397+0000 7f250418a640 1 -- 172.21.15.42:0/2562331424 wait complete. 2024-09-16T03:42:16.405 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:16.397+0000 7f250418a640 1 librados: shutdown 2024-09-16T03:42:16.419 DEBUG:teuthology.misc:0 of 6 OSDs are up 2024-09-16T03:42:17.184 INFO:tasks.ceph.osd.5.smithi154.stderr:2024-09-16T03:42:17.182+0000 7f058b1db640 -1 osd.5 0 waiting for initial osdmap 2024-09-16T03:42:17.189 INFO:tasks.ceph.osd.5.smithi154.stderr:2024-09-16T03:42:17.186+0000 7f0583010640 -1 osd.5 11 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T03:42:23.025 DEBUG:teuthology.orchestra.run.smithi042:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-16T03:42:23.152 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.145+0000 7f9e2fc36640 10 monclient: get_monmap_and_config 2024-09-16T03:42:23.152 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.145+0000 7f9e2fc36640 10 monclient: build_initial_monmap 2024-09-16T03:42:23.153 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.145+0000 7f9e2fc36640 10 monclient: monmap: 2024-09-16T03:42:23.153 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-09-16T03:42:23.153 INFO:teuthology.misc.health.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:23.153 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-09-16T03:42:23.150161+0000 2024-09-16T03:42:23.153 INFO:teuthology.misc.health.smithi042.stderr:created 2024-09-16T03:42:23.150161+0000 2024-09-16T03:42:23.153 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:42:23.153 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:23.153 INFO:teuthology.misc.health.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:42:23.154 INFO:teuthology.misc.health.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:42:23.154 INFO:teuthology.misc.health.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:42:23.154 INFO:teuthology.misc.health.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:42:23.154 INFO:teuthology.misc.health.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:42:23.154 INFO:teuthology.misc.health.smithi042.stderr: 2024-09-16T03:42:23.156 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.145+0000 7f9e2fc36640 1 Processor -- start 2024-09-16T03:42:23.156 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.145+0000 7f9e2fc36640 1 -- start start 2024-09-16T03:42:23.156 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.145+0000 7f9e2fc36640 10 monclient: init 2024-09-16T03:42:23.157 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.145+0000 7f9e2fc36640 10 monclient: _reopen_session rank -1 2024-09-16T03:42:23.157 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.145+0000 7f9e2fc36640 10 monclient: _add_conns ranks=[3,1,2,0,4] 2024-09-16T03:42:23.157 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.145+0000 7f9e2fc36640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f9e280a1230 0x7f9e2809caf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:23.157 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.145+0000 7f9e2fc36640 10 monclient(hunting): picked mon.noname-a con 0x7f9e280a1230 addr v2:172.21.15.42:3300/0 2024-09-16T03:42:23.157 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.145+0000 7f9e2fc36640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9e2809e730 addr v1:172.21.15.154:6789/0 2024-09-16T03:42:23.157 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.145+0000 7f9e2fc36640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9e2809d740 0x7f9e2813c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:23.157 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.145+0000 7f9e2fc36640 10 monclient(hunting): picked mon.noname-b con 0x7f9e2809d740 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:42:23.157 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.145+0000 7f9e2fc36640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9e2806d050 con 0x7f9e2809e730 2024-09-16T03:42:23.157 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.145+0000 7f9e2fc36640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:23.157 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.145+0000 7f9e2fc36640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f9e280a1880 con 0x7f9e280a1230 2024-09-16T03:42:23.157 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.145+0000 7f9e2fc36640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:23.157 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.145+0000 7f9e2fc36640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f9e280a1610 con 0x7f9e2809d740 2024-09-16T03:42:23.158 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.145+0000 7f9e2fc36640 10 monclient(hunting): _renew_subs 2024-09-16T03:42:23.158 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.145+0000 7f9e2fc36640 10 monclient(hunting): authenticate will time out at 990.119551s 2024-09-16T03:42:23.158 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.145+0000 7f9e2d9d2640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f9e280a1230 0x7f9e2809caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:23.158 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.145+0000 7f9e2e1d3640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9e2809d740 0x7f9e2813c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:23.158 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.145+0000 7f9e2e1d3640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9e2809d740 0x7f9e2813c7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:36128/0 (socket says 172.21.15.42:36128) 2024-09-16T03:42:23.158 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.145+0000 7f9e2d9d2640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f9e280a1230 0x7f9e2809caf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:35722/0 (socket says 172.21.15.42:35722) 2024-09-16T03:42:23.158 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.145+0000 7f9e2e1d3640 1 -- 172.21.15.42:0/235195278 learned_addr learned my addr 172.21.15.42:0/235195278 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:42:23.158 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.145+0000 7f9e2e1d3640 10 monclient(hunting): get_auth_request con 0x7f9e2809d740 auth_method 0 2024-09-16T03:42:23.158 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.145+0000 7f9e2e1d3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:23.158 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.145+0000 7f9e2e1d3640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:23.158 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.145+0000 7f9e2e1d3640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:23.158 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.145+0000 7f9e2d9d2640 10 monclient(hunting): get_auth_request con 0x7f9e280a1230 auth_method 0 2024-09-16T03:42:23.158 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.145+0000 7f9e2d9d2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:23.158 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.145+0000 7f9e2d9d2640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:23.159 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.145+0000 7f9e2d9d2640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:23.159 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.145+0000 7f9e2e1d3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:23.159 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.145+0000 7f9e2e1d3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:23.159 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.145+0000 7f9e2e1d3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:23.159 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.145+0000 7f9e2d9d2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:23.159 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.145+0000 7f9e2d9d2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:23.159 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.149+0000 7f9e2d9d2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:23.159 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.149+0000 7f9e2d9d2640 10 monclient(hunting): handle_auth_done global_id 4209 payload 274 2024-09-16T03:42:23.159 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.149+0000 7f9e2d9d2640 1 -- 172.21.15.42:0/235195278 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9e2809d740 msgr2=0x7f9e2813c7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:42:23.159 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.149+0000 7f9e2d9d2640 1 --2- 172.21.15.42:0/235195278 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9e2809d740 0x7f9e2813c7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:23.159 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.149+0000 7f9e2d9d2640 1 -- 172.21.15.42:0/235195278 >> v1:172.21.15.154:6789/0 conn(0x7f9e2809e730 legacy=0x7f9e2809d030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:42:23.159 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.149+0000 7f9e2d9d2640 10 monclient: _finish_hunting 0 2024-09-16T03:42:23.160 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.149+0000 7f9e2d9d2640 1 monclient: found mon.noname-a 2024-09-16T03:42:23.160 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.149+0000 7f9e2d9d2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:42:23.160 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.149+0000 7f9e2d9d2640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:42:23.160 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.149+0000 7f9e2d9d2640 1 -- 172.21.15.42:0/235195278 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e280a1ac0 con 0x7f9e280a1230 2024-09-16T03:42:23.160 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.149+0000 7f9e2e1d3640 1 --2- 172.21.15.42:0/235195278 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9e2809d740 0x7f9e2813c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:42:23.160 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.149+0000 7f9e2d9d2640 1 --2- 172.21.15.42:0/235195278 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9e280a1230 0x7f9e2809caf0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f9e240097c0 tx=0x7f9e2401c320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b20fca038d19290 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:23.168 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.149+0000 7f9e2c9d0640 1 -- 172.21.15.42:0/235195278 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e240b0070 con 0x7f9e280a1230 2024-09-16T03:42:23.169 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.149+0000 7f9e2c9d0640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:23.169 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.149+0000 7f9e2c9d0640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:42:23.169 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.149+0000 7f9e2c9d0640 10 monclient: dump: 2024-09-16T03:42:23.169 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-09-16T03:42:23.169 INFO:teuthology.misc.health.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:23.169 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:23.169 INFO:teuthology.misc.health.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:23.169 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:23.169 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:23.169 INFO:teuthology.misc.health.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:23.169 INFO:teuthology.misc.health.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:23.169 INFO:teuthology.misc.health.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:23.170 INFO:teuthology.misc.health.smithi042.stderr: 2024-09-16T03:42:23.170 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.149+0000 7f9e2c9d0640 10 monclient: _finish_auth 0 2024-09-16T03:42:23.170 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.149+0000 7f9e2c9d0640 10 monclient: _check_auth_tickets 2024-09-16T03:42:23.170 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.149+0000 7f9e2c9d0640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:42:23.170 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.149+0000 7f9e2c9d0640 1 -- 172.21.15.42:0/235195278 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9e240ab600 con 0x7f9e280a1230 2024-09-16T03:42:23.170 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.149+0000 7f9e2c9d0640 10 monclient: handle_config config(0 keys) 2024-09-16T03:42:23.170 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.149+0000 7f9e2c9d0640 1 -- 172.21.15.42:0/235195278 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e240ab9d0 con 0x7f9e280a1230 2024-09-16T03:42:23.170 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.149+0000 7f9e2c9d0640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:23.170 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.149+0000 7f9e2c9d0640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:42:23.170 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.149+0000 7f9e2c9d0640 10 monclient: dump: 2024-09-16T03:42:23.170 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-09-16T03:42:23.170 INFO:teuthology.misc.health.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:23.171 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:23.171 INFO:teuthology.misc.health.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:23.171 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:23.171 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:23.171 INFO:teuthology.misc.health.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:23.171 INFO:teuthology.misc.health.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:23.171 INFO:teuthology.misc.health.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:23.171 INFO:teuthology.misc.health.smithi042.stderr: 2024-09-16T03:42:23.171 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.149+0000 7f9e2fc36640 5 monclient: authenticate success, global_id 4209 2024-09-16T03:42:23.171 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.149+0000 7f9e2fc36640 10 monclient: get_monmap_and_config success 2024-09-16T03:42:23.171 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.149+0000 7f9e2fc36640 4 set_mon_vals no callback set 2024-09-16T03:42:23.171 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.149+0000 7f9e2fc36640 10 monclient: shutdown 2024-09-16T03:42:23.171 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.149+0000 7f9e2fc36640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:42:23.171 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.149+0000 7f9e2fc36640 1 -- 172.21.15.42:0/235195278 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9e280a1230 msgr2=0x7f9e2809caf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:23.172 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.149+0000 7f9e2fc36640 1 --2- 172.21.15.42:0/235195278 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9e280a1230 0x7f9e2809caf0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f9e240097c0 tx=0x7f9e2401c320 comp rx=0 tx=0).stop 2024-09-16T03:42:23.172 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.149+0000 7f9e2fc36640 1 -- 172.21.15.42:0/235195278 shutdown_connections 2024-09-16T03:42:23.172 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.149+0000 7f9e2fc36640 1 --2- 172.21.15.42:0/235195278 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9e2809d740 0x7f9e2813c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:23.172 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.149+0000 7f9e2fc36640 1 --2- 172.21.15.42:0/235195278 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9e280a1230 0x7f9e2809caf0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:23.172 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.149+0000 7f9e2fc36640 1 -- 172.21.15.42:0/235195278 >> 172.21.15.42:0/235195278 conn(0x7f9e280a24f0 msgr2=0x7f9e280a2900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:42:23.172 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.149+0000 7f9e2fc36640 1 -- 172.21.15.42:0/235195278 shutdown_connections 2024-09-16T03:42:23.172 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.149+0000 7f9e2fc36640 1 -- 172.21.15.42:0/235195278 wait complete. 2024-09-16T03:42:23.172 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.153+0000 7f9e2fc36640 10 monclient: build_initial_monmap 2024-09-16T03:42:23.172 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.153+0000 7f9e2fc36640 10 monclient: monmap: 2024-09-16T03:42:23.173 INFO:teuthology.misc.health.smithi042.stderr:epoch 0 2024-09-16T03:42:23.173 INFO:teuthology.misc.health.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:42:23.173 INFO:teuthology.misc.health.smithi042.stderr:last_changed 0.000000 2024-09-16T03:42:23.173 INFO:teuthology.misc.health.smithi042.stderr:created 0.000000 2024-09-16T03:42:23.173 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:42:23.173 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:23.173 INFO:teuthology.misc.health.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:42:23.173 INFO:teuthology.misc.health.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:42:23.173 INFO:teuthology.misc.health.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:42:23.173 INFO:teuthology.misc.health.smithi042.stderr: 2024-09-16T03:42:23.173 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.153+0000 7f9e2fc36640 1 librados: starting msgr at 2024-09-16T03:42:23.173 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.153+0000 7f9e2fc36640 1 librados: starting objecter 2024-09-16T03:42:23.173 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.153+0000 7f9e2fc36640 1 Processor -- start 2024-09-16T03:42:23.174 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.153+0000 7f9e2fc36640 1 -- start start 2024-09-16T03:42:23.174 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.153+0000 7f9e2fc36640 1 librados: setting wanted keys 2024-09-16T03:42:23.174 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.153+0000 7f9e2fc36640 1 librados: calling monclient init 2024-09-16T03:42:23.174 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.153+0000 7f9e2fc36640 10 monclient: init 2024-09-16T03:42:23.174 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.153+0000 7f9e2fc36640 10 monclient: _reopen_session rank -1 2024-09-16T03:42:23.174 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.153+0000 7f9e2fc36640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T03:42:23.174 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.153+0000 7f9e2fc36640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9e2809d740 0x7f9e28097cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:23.174 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.153+0000 7f9e2fc36640 10 monclient(hunting): picked mon.noname-b con 0x7f9e2809d740 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:42:23.174 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.153+0000 7f9e2fc36640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9e2809e730 0x7f9e2808e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:23.175 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.153+0000 7f9e2fc36640 10 monclient(hunting): picked mon.noname-c con 0x7f9e2809e730 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:42:23.175 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.153+0000 7f9e2fc36640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9e280a1230 0x7f9e2808e560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:23.175 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.153+0000 7f9e2fc36640 10 monclient(hunting): picked mon.noname-a con 0x7f9e280a1230 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:42:23.175 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.153+0000 7f9e2fc36640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:23.175 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.153+0000 7f9e2fc36640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f9e2813f710 con 0x7f9e280a1230 2024-09-16T03:42:23.175 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.153+0000 7f9e2fc36640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:23.175 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.153+0000 7f9e2fc36640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f9e280a1cc0 con 0x7f9e2809d740 2024-09-16T03:42:23.175 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.153+0000 7f9e2fc36640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:23.175 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.153+0000 7f9e2fc36640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f9e2813f890 con 0x7f9e2809e730 2024-09-16T03:42:23.175 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.153+0000 7f9e2fc36640 10 monclient(hunting): _renew_subs 2024-09-16T03:42:23.175 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.153+0000 7f9e2fc36640 10 monclient(hunting): authenticate will time out at 990.125311s 2024-09-16T03:42:23.175 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.153+0000 7f9e2e1d3640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9e280a1230 0x7f9e2808e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:23.175 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.153+0000 7f9e2d1d1640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9e2809e730 0x7f9e2808e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:23.176 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.153+0000 7f9e2e1d3640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9e280a1230 0x7f9e2808e560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:35732/0 (socket says 172.21.15.42:35732) 2024-09-16T03:42:23.176 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.153+0000 7f9e2d1d1640 10 monclient(hunting): get_auth_request con 0x7f9e2809e730 auth_method 0 2024-09-16T03:42:23.176 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.153+0000 7f9e2e1d3640 1 -- 172.21.15.42:0/1963112364 learned_addr learned my addr 172.21.15.42:0/1963112364 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:42:23.176 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.153+0000 7f9e2d1d1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:23.176 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.153+0000 7f9e2d1d1640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:23.176 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.153+0000 7f9e2d1d1640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:23.176 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.153+0000 7f9e2e1d3640 10 monclient(hunting): get_auth_request con 0x7f9e280a1230 auth_method 0 2024-09-16T03:42:23.176 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.153+0000 7f9e2e1d3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:23.176 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.153+0000 7f9e2e1d3640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:23.176 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.153+0000 7f9e2e1d3640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:23.176 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.153+0000 7f9e2d1d1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:23.176 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.153+0000 7f9e2d1d1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:23.176 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.153+0000 7f9e2d1d1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:23.177 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.153+0000 7f9e2e1d3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:23.177 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.153+0000 7f9e2e1d3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:23.177 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.153+0000 7f9e2e1d3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:23.177 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.153+0000 7f9e2d9d2640 1 --2- 172.21.15.42:0/1963112364 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9e2809d740 0x7f9e28097cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:23.177 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.153+0000 7f9e2d1d1640 10 monclient(hunting): handle_auth_done global_id 4193 payload 824 2024-09-16T03:42:23.177 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.153+0000 7f9e2d1d1640 1 -- 172.21.15.42:0/1963112364 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9e2809d740 msgr2=0x7f9e28097cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:23.177 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.153+0000 7f9e2d1d1640 1 --2- 172.21.15.42:0/1963112364 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9e2809d740 0x7f9e28097cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:23.177 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.153+0000 7f9e2d1d1640 1 -- 172.21.15.42:0/1963112364 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9e280a1230 msgr2=0x7f9e2808e560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:23.177 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.153+0000 7f9e2d1d1640 1 --2- 172.21.15.42:0/1963112364 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9e280a1230 0x7f9e2808e560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:23.177 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.153+0000 7f9e2d1d1640 10 monclient: _finish_hunting 0 2024-09-16T03:42:23.177 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.153+0000 7f9e2d1d1640 1 monclient: found mon.noname-c 2024-09-16T03:42:23.177 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.153+0000 7f9e2d1d1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:42:23.177 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.153+0000 7f9e2d1d1640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:42:23.177 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.153+0000 7f9e2d1d1640 1 -- 172.21.15.42:0/1963112364 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e28090f30 con 0x7f9e2809e730 2024-09-16T03:42:23.178 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.153+0000 7f9e2d9d2640 10 monclient: get_auth_request con 0x7f9e2809d740 auth_method 0 2024-09-16T03:42:23.178 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.153+0000 7f9e2e1d3640 1 --2- 172.21.15.42:0/1963112364 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9e280a1230 0x7f9e2808e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:42:23.178 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.153+0000 7f9e2d9d2640 1 --2- 172.21.15.42:0/1963112364 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9e2809d740 0x7f9e28097cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T03:42:23.178 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.153+0000 7f9e2d1d1640 1 --2- 172.21.15.42:0/1963112364 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9e2809e730 0x7f9e2808e020 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f9e1c00c360 tx=0x7f9e1c00c820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:23.178 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.153+0000 7f9e1a7fc640 10 client.?.objecter ms_handle_connect 0x7f9e2809e730 2024-09-16T03:42:23.178 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.153+0000 7f9e1a7fc640 10 client.?.objecter resend_mon_ops 2024-09-16T03:42:23.178 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.153+0000 7f9e1a7fc640 1 -- 172.21.15.42:0/1963112364 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e1c017070 con 0x7f9e2809e730 2024-09-16T03:42:23.178 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.153+0000 7f9e1a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:23.178 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.153+0000 7f9e1a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:42:23.178 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.153+0000 7f9e1a7fc640 10 monclient: dump: 2024-09-16T03:42:23.178 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-09-16T03:42:23.178 INFO:teuthology.misc.health.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:23.178 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:23.179 INFO:teuthology.misc.health.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:23.179 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:23.179 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:23.179 INFO:teuthology.misc.health.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:23.179 INFO:teuthology.misc.health.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:23.179 INFO:teuthology.misc.health.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:23.179 INFO:teuthology.misc.health.smithi042.stderr: 2024-09-16T03:42:23.179 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.153+0000 7f9e1a7fc640 10 monclient: _finish_auth 0 2024-09-16T03:42:23.179 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.153+0000 7f9e1a7fc640 10 monclient: _check_auth_tickets 2024-09-16T03:42:23.179 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.153+0000 7f9e1a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:42:23.179 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.153+0000 7f9e2fc36640 5 monclient: authenticate success, global_id 4193 2024-09-16T03:42:23.179 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.153+0000 7f9e1a7fc640 1 -- 172.21.15.42:0/1963112364 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9e1c003600 con 0x7f9e2809e730 2024-09-16T03:42:23.179 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.153+0000 7f9e1a7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:42:23.179 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.153+0000 7f9e1a7fc640 1 -- 172.21.15.42:0/1963112364 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e1c003a30 con 0x7f9e2809e730 2024-09-16T03:42:23.180 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.153+0000 7f9e1a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:23.180 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.153+0000 7f9e1a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:42:23.180 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.153+0000 7f9e1a7fc640 10 monclient: dump: 2024-09-16T03:42:23.180 INFO:teuthology.misc.health.smithi042.stderr:epoch 1 2024-09-16T03:42:23.180 INFO:teuthology.misc.health.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:23.180 INFO:teuthology.misc.health.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:23.180 INFO:teuthology.misc.health.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:23.180 INFO:teuthology.misc.health.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:23.180 INFO:teuthology.misc.health.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:23.180 INFO:teuthology.misc.health.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:23.180 INFO:teuthology.misc.health.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:23.180 INFO:teuthology.misc.health.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:23.180 INFO:teuthology.misc.health.smithi042.stderr: 2024-09-16T03:42:23.180 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.153+0000 7f9e1bfff640 4 set_mon_vals no callback set 2024-09-16T03:42:23.180 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.153+0000 7f9e2fc36640 10 monclient: _renew_subs 2024-09-16T03:42:23.181 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.153+0000 7f9e2fc36640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:42:23.181 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.153+0000 7f9e2fc36640 1 -- 172.21.15.42:0/1963112364 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e2806e270 con 0x7f9e2809e730 2024-09-16T03:42:23.181 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.153+0000 7f9e2fc36640 10 client.4193.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:42:23.181 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.153+0000 7f9e2fc36640 10 monclient: _renew_subs 2024-09-16T03:42:23.181 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.153+0000 7f9e2fc36640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:42:23.181 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.153+0000 7f9e2fc36640 1 -- 172.21.15.42:0/1963112364 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e28091480 con 0x7f9e2809e730 2024-09-16T03:42:23.181 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.153+0000 7f9e2fc36640 1 librados: init done 2024-09-16T03:42:23.181 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.153+0000 7f9e2fc36640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:42:23.181 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.153+0000 7f9e2fc36640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:42:23.181 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.153+0000 7f9e2fc36640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:42:23.181 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.153+0000 7f9e2fc36640 1 -- 172.21.15.42:0/1963112364 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e28090c30 con 0x7f9e2809e730 2024-09-16T03:42:23.181 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.153+0000 7f9e1a7fc640 1 -- 172.21.15.42:0/1963112364 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f9e1c003bd0 con 0x7f9e2809e730 2024-09-16T03:42:23.181 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.153+0000 7f9e1a7fc640 1 --2- 172.21.15.42:0/1963112364 >> v2:172.21.15.154:6812/254845342 conn(0x7f9e0c037810 0x7f9e0c039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:23.182 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.153+0000 7f9e1a7fc640 1 -- 172.21.15.42:0/1963112364 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(14..14 src has 1..14) ==== 3153+0+0 (secure 0 0 0) 0x7f9e1c0478b0 con 0x7f9e2809e730 2024-09-16T03:42:23.182 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.153+0000 7f9e1a7fc640 10 client.4193.objecter ms_dispatch 0x7f9e28000ba0 osd_map(14..14 src has 1..14) 2024-09-16T03:42:23.182 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.153+0000 7f9e1a7fc640 3 client.4193.objecter handle_osd_map got epochs [14,14] > 0 2024-09-16T03:42:23.182 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.153+0000 7f9e1a7fc640 3 client.4193.objecter handle_osd_map decoding full epoch 14 2024-09-16T03:42:23.182 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.153+0000 7f9e1a7fc640 20 client.4193.objecter dump_active .. 0 homeless 2024-09-16T03:42:23.182 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.157+0000 7f9e2d9d2640 1 --2- 172.21.15.42:0/1963112364 >> v2:172.21.15.154:6812/254845342 conn(0x7f9e0c037810 0x7f9e0c039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:23.182 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.157+0000 7f9e2d9d2640 10 monclient: get_auth_request con 0x7f9e0c037810 auth_method 0 2024-09-16T03:42:23.182 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.157+0000 7f9e2d9d2640 1 --2- 172.21.15.42:0/1963112364 >> v2:172.21.15.154:6812/254845342 conn(0x7f9e0c037810 0x7f9e0c039cd0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f9e240ad000 tx=0x7f9e24008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:23.182 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.157+0000 7f9e1a7fc640 10 client.4193.objecter ms_handle_connect 0x7f9e0c037810 2024-09-16T03:42:23.182 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.161+0000 7f9e1a7fc640 1 -- 172.21.15.42:0/1963112364 <== mon.2 v2:172.21.15.42:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f9e1c019740 con 0x7f9e2809e730 2024-09-16T03:42:23.182 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.161+0000 7f9e1a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:42:23.182 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.161+0000 7f9e1a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:42:23.331 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.325+0000 7f9e2fc36640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-16T03:42:23.331 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.325+0000 7f9e2fc36640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T03:42:23.331 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.325+0000 7f9e2fc36640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:42:23.332 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.325+0000 7f9e2fc36640 1 -- 172.21.15.42:0/1963112364 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9e2809caf0 con 0x7f9e2809e730 2024-09-16T03:42:23.332 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.325+0000 7f9e1a7fc640 1 -- 172.21.15.42:0/1963112364 <== mon.2 v2:172.21.15.42:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v14) ==== 74+0+9049 (secure 0 0 0) 0x7f9e1c019960 con 0x7f9e2809e730 2024-09-16T03:42:23.333 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.325+0000 7f9e1a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T03:42:23.333 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.325+0000 7f9e1a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:42:23.333 INFO:teuthology.misc.health.smithi042.stdout: 2024-09-16T03:42:23.333 INFO:teuthology.misc.health.smithi042.stdout:{"epoch":14,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","created":"2024-09-16T03:42:05.977277+0000","modified":"2024-09-16T03:42:19.562804+0000","last_up_change":"2024-09-16T03:42:17.544237+0000","last_in_change":"2024-09-16T03:42:08.437271+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T03:42:17.542498+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6b9fc058-2cff-46af-b95c-b78c72e1dc2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6808","nonce":1401549752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6809","nonce":1401549752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6811","nonce":1401549752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6810","nonce":1401549752}]},"public_addr":"172.21.15.42:6808/1401549752","cluster_addr":"172.21.15.42:6809/1401549752","heartbeat_back_addr":"172.21.15.42:6811/1401549752","heartbeat_front_addr":"172.21.15.42:6810/1401549752","state":["exists","up"]},{"osd":1,"uuid":"ee31f807-cb16-4c33-8e23-2573e1a6b5c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6800","nonce":4217863575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6801","nonce":4217863575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6803","nonce":4217863575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6802","nonce":4217863575}]},"public_addr":"172.21.15.42:6800/4217863575","cluster_addr":"172.21.15.42:6801/4217863575","heartbeat_back_addr":"172.21.15.42:6803/4217863575","heartbeat_front_addr":"172.21.15.42:6802/4217863575","state":["exists","up"]},{"osd":2,"uuid":"a26f3917-6702-43db-9355-e96da86a3365","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6804","nonce":263835041}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6805","nonce":263835041}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6807","nonce":263835041}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6806","nonce":263835041}]},"public_addr":"172.21.15.42:6804/263835041","cluster_addr":"172.21.15.42:6805/263835041","heartbeat_back_addr":"172.21.15.42:6807/263835041","heartbeat_front_addr":"172.21.15.42:6806/263835041","state":["exists","up"]},{"osd":3,"uuid":"4e809f2a-2ffd-4791-9f2d-18f1bfe7d2ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":4043102363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6801","nonce":4043102363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6803","nonce":4043102363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":4043102363}]},"public_addr":"172.21.15.154:6800/4043102363","cluster_addr":"172.21.15.154:6801/4043102363","heartbeat_back_addr":"172.21.15.154:6803/4043102363","heartbeat_front_addr":"172.21.15.154:6802/4043102363","state":["exists","up"]},{"osd":4,"uuid":"dcf637ab-5f72-4ed2-a99b-69083f0b6b15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":1038452759}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6805","nonce":1038452759}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6807","nonce":1038452759}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":1038452759}]},"public_addr":"172.21.15.154:6804/1038452759","cluster_addr":"172.21.15.154:6805/1038452759","heartbeat_back_addr":"172.21.15.154:6807/1038452759","heartbeat_front_addr":"172.21.15.154:6806/1038452759","state":["exists","up"]},{"osd":5,"uuid":"83c05e44-861b-4bb2-9657-19196e5ac1f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":3026127822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6809","nonce":3026127822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6811","nonce":3026127822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":3026127822}]},"public_addr":"172.21.15.154:6808/3026127822","cluster_addr":"172.21.15.154:6809/3026127822","heartbeat_back_addr":"172.21.15.154:6811/3026127822","heartbeat_front_addr":"172.21.15.154:6810/3026127822","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:42:15.086682+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:42:14.665142+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:42:14.860865+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:42:14.620446+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:42:15.163090+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:42:15.856205+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T03:42:23.335 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.329+0000 7f9e2fc36640 10 librados: watch_flush enter 2024-09-16T03:42:23.335 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.329+0000 7f9e2fc36640 10 librados: watch_flush exit 2024-09-16T03:42:23.335 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.329+0000 7f9e2fc36640 20 client.4193.objecter shutdown clearing up homeless session... 2024-09-16T03:42:23.335 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.329+0000 7f9e2fc36640 10 client.4193.objecter successfully canceled tick 2024-09-16T03:42:23.335 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.329+0000 7f9e2fc36640 1 -- 172.21.15.42:0/1963112364 >> v2:172.21.15.154:6812/254845342 conn(0x7f9e0c037810 msgr2=0x7f9e0c039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:23.335 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.329+0000 7f9e2fc36640 1 --2- 172.21.15.42:0/1963112364 >> v2:172.21.15.154:6812/254845342 conn(0x7f9e0c037810 0x7f9e0c039cd0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f9e240ad000 tx=0x7f9e24008040 comp rx=0 tx=0).stop 2024-09-16T03:42:23.335 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.329+0000 7f9e2fc36640 10 monclient: shutdown 2024-09-16T03:42:23.335 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.329+0000 7f9e2fc36640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:42:23.335 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.329+0000 7f9e2fc36640 1 -- 172.21.15.42:0/1963112364 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9e2809e730 msgr2=0x7f9e2808e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:23.335 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.329+0000 7f9e2fc36640 1 --2- 172.21.15.42:0/1963112364 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9e2809e730 0x7f9e2808e020 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f9e1c00c360 tx=0x7f9e1c00c820 comp rx=0 tx=0).stop 2024-09-16T03:42:23.335 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.329+0000 7f9e2fc36640 1 -- 172.21.15.42:0/1963112364 shutdown_connections 2024-09-16T03:42:23.336 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.329+0000 7f9e2fc36640 1 --2- 172.21.15.42:0/1963112364 >> v2:172.21.15.154:6812/254845342 conn(0x7f9e0c037810 0x7f9e0c039cd0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:23.336 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.329+0000 7f9e2fc36640 1 --2- 172.21.15.42:0/1963112364 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9e280a1230 0x7f9e2808e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:23.336 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.329+0000 7f9e2fc36640 1 --2- 172.21.15.42:0/1963112364 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9e2809e730 0x7f9e2808e020 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:23.336 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.329+0000 7f9e2fc36640 1 --2- 172.21.15.42:0/1963112364 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9e2809d740 0x7f9e28097cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:23.336 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.329+0000 7f9e2fc36640 1 -- 172.21.15.42:0/1963112364 >> 172.21.15.42:0/1963112364 conn(0x7f9e280a24f0 msgr2=0x7f9e2813bf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:42:23.336 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.329+0000 7f9e2fc36640 1 -- 172.21.15.42:0/1963112364 shutdown_connections 2024-09-16T03:42:23.336 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.329+0000 7f9e2fc36640 1 -- 172.21.15.42:0/1963112364 wait complete. 2024-09-16T03:42:23.336 INFO:teuthology.misc.health.smithi042.stderr:2024-09-16T03:42:23.329+0000 7f9e2fc36640 1 librados: shutdown 2024-09-16T03:42:23.350 DEBUG:teuthology.misc:6 of 6 OSDs are up 2024-09-16T03:42:23.350 INFO:tasks.ceph:Creating RBD pool 2024-09-16T03:42:23.350 DEBUG:teuthology.orchestra.run.smithi042:> sudo ceph --cluster ceph osd pool create rbd 8 2024-09-16T03:42:23.465 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.457+0000 7fc4f1d08640 10 monclient: get_monmap_and_config 2024-09-16T03:42:23.466 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.457+0000 7fc4f1d08640 10 monclient: build_initial_monmap 2024-09-16T03:42:23.466 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.457+0000 7fc4f1d08640 10 monclient: monmap: 2024-09-16T03:42:23.466 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:42:23.466 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:23.466 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:42:23.463692+0000 2024-09-16T03:42:23.466 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:42:23.463692+0000 2024-09-16T03:42:23.466 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:42:23.466 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:23.466 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:42:23.466 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:42:23.466 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:42:23.467 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:42:23.467 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:42:23.467 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:23.469 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.461+0000 7fc4f1d08640 1 Processor -- start 2024-09-16T03:42:23.469 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.461+0000 7fc4f1d08640 1 -- start start 2024-09-16T03:42:23.469 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.461+0000 7fc4f1d08640 10 monclient: init 2024-09-16T03:42:23.469 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.461+0000 7fc4f1d08640 10 monclient: _reopen_session rank -1 2024-09-16T03:42:23.469 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.461+0000 7fc4f1d08640 10 monclient: _add_conns ranks=[4,3,2,0,1] 2024-09-16T03:42:23.469 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.461+0000 7fc4f1d08640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fc4ec0a2d60 0x7fc4ec134d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:23.469 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.461+0000 7fc4f1d08640 10 monclient(hunting): picked mon.noname-c con 0x7fc4ec0a2d60 addr v2:172.21.15.154:3300/0 2024-09-16T03:42:23.469 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.461+0000 7fc4f1d08640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fc4ec072d70 0x7fc4ec0a1330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:23.469 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.461+0000 7fc4f1d08640 10 monclient(hunting): picked mon.noname-a con 0x7fc4ec072d70 addr v2:172.21.15.42:3300/0 2024-09-16T03:42:23.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.461+0000 7fc4f1d08640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc4ec073200 0x7fc4ec0a1870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:23.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.461+0000 7fc4f1d08640 10 monclient(hunting): picked mon.noname-b con 0x7fc4ec073200 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:42:23.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.461+0000 7fc4f1d08640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:23.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.461+0000 7fc4f1d08640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7fc4ec09ee90 con 0x7fc4ec072d70 2024-09-16T03:42:23.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.461+0000 7fc4f1d08640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:23.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.461+0000 7fc4f1d08640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fc4ec0a3140 con 0x7fc4ec0a2d60 2024-09-16T03:42:23.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.461+0000 7fc4f1d08640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:23.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.461+0000 7fc4f1d08640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fc4ec135390 con 0x7fc4ec073200 2024-09-16T03:42:23.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.461+0000 7fc4f1d08640 10 monclient(hunting): _renew_subs 2024-09-16T03:42:23.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.461+0000 7fc4f1d08640 10 monclient(hunting): authenticate will time out at 990.433489s 2024-09-16T03:42:23.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.461+0000 7fc4eaffd640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fc4ec072d70 0x7fc4ec0a1330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:23.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.461+0000 7fc4ebfff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc4ec073200 0x7fc4ec0a1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:23.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.461+0000 7fc4eaffd640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fc4ec072d70 0x7fc4ec0a1330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:35734/0 (socket says 172.21.15.42:35734) 2024-09-16T03:42:23.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.461+0000 7fc4eaffd640 1 -- 172.21.15.42:0/3770509009 learned_addr learned my addr 172.21.15.42:0/3770509009 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:42:23.471 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.461+0000 7fc4eaffd640 10 monclient(hunting): get_auth_request con 0x7fc4ec072d70 auth_method 0 2024-09-16T03:42:23.471 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.461+0000 7fc4eaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:23.471 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.461+0000 7fc4eaffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:23.471 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.461+0000 7fc4eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:23.471 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.461+0000 7fc4ebfff640 10 monclient(hunting): get_auth_request con 0x7fc4ec073200 auth_method 0 2024-09-16T03:42:23.471 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.461+0000 7fc4ebfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:23.471 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.461+0000 7fc4ebfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:23.471 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.461+0000 7fc4ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:23.471 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.461+0000 7fc4eb7fe640 1 --2- 172.21.15.42:0/3770509009 >> v2:172.21.15.154:3300/0 conn(0x7fc4ec0a2d60 0x7fc4ec134d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:23.471 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.461+0000 7fc4eb7fe640 10 monclient(hunting): get_auth_request con 0x7fc4ec0a2d60 auth_method 0 2024-09-16T03:42:23.471 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.461+0000 7fc4eb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:23.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.461+0000 7fc4eb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:23.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.461+0000 7fc4eb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:23.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.461+0000 7fc4eaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:23.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.461+0000 7fc4eaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:23.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.461+0000 7fc4eaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:23.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.461+0000 7fc4ebfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:23.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.461+0000 7fc4ebfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:23.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.461+0000 7fc4ebfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:23.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.461+0000 7fc4eb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:23.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.461+0000 7fc4eb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:23.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.461+0000 7fc4eb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:23.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.461+0000 7fc4eaffd640 10 monclient(hunting): handle_auth_done global_id 4215 payload 274 2024-09-16T03:42:23.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.461+0000 7fc4eaffd640 1 -- 172.21.15.42:0/3770509009 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc4ec073200 msgr2=0x7fc4ec0a1870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:42:23.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.461+0000 7fc4eaffd640 1 --2- 172.21.15.42:0/3770509009 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc4ec073200 0x7fc4ec0a1870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:23.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.461+0000 7fc4eaffd640 1 -- 172.21.15.42:0/3770509009 >> v2:172.21.15.154:3300/0 conn(0x7fc4ec0a2d60 msgr2=0x7fc4ec134d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:42:23.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.461+0000 7fc4eaffd640 1 --2- 172.21.15.42:0/3770509009 >> v2:172.21.15.154:3300/0 conn(0x7fc4ec0a2d60 0x7fc4ec134d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:23.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.461+0000 7fc4eaffd640 10 monclient: _finish_hunting 0 2024-09-16T03:42:23.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.461+0000 7fc4eaffd640 1 monclient: found mon.noname-a 2024-09-16T03:42:23.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.461+0000 7fc4eaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:42:23.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.461+0000 7fc4eaffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:42:23.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.461+0000 7fc4eaffd640 1 -- 172.21.15.42:0/3770509009 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4ec09fa40 con 0x7fc4ec072d70 2024-09-16T03:42:23.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.461+0000 7fc4ebfff640 1 --2- 172.21.15.42:0/3770509009 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc4ec073200 0x7fc4ec0a1870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:42:23.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.465+0000 7fc4eaffd640 1 --2- 172.21.15.42:0/3770509009 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc4ec072d70 0x7fc4ec0a1330 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fc4ec0a3330 tx=0x7fc4dc01c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5fdbf20f429ea68 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:23.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.465+0000 7fc4ea7fc640 1 -- 172.21.15.42:0/3770509009 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4dc0b1070 con 0x7fc4ec072d70 2024-09-16T03:42:23.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.465+0000 7fc4ea7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:23.479 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.465+0000 7fc4ea7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:42:23.479 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.465+0000 7fc4ea7fc640 10 monclient: dump: 2024-09-16T03:42:23.479 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:23.479 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:23.479 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:23.479 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:23.479 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:23.479 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:23.479 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:23.479 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:23.479 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:23.479 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:23.479 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.465+0000 7fc4ea7fc640 10 monclient: _finish_auth 0 2024-09-16T03:42:23.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.465+0000 7fc4ea7fc640 10 monclient: _check_auth_tickets 2024-09-16T03:42:23.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.465+0000 7fc4ea7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:42:23.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.465+0000 7fc4f1d08640 5 monclient: authenticate success, global_id 4215 2024-09-16T03:42:23.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.465+0000 7fc4f1d08640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:42:23.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.465+0000 7fc4ea7fc640 1 -- 172.21.15.42:0/3770509009 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc4dc0ac640 con 0x7fc4ec072d70 2024-09-16T03:42:23.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.465+0000 7fc4ea7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:42:23.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.465+0000 7fc4ea7fc640 1 -- 172.21.15.42:0/3770509009 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4dc0aca10 con 0x7fc4ec072d70 2024-09-16T03:42:23.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.465+0000 7fc4ea7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:23.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.465+0000 7fc4ea7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:42:23.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.465+0000 7fc4ea7fc640 10 monclient: dump: 2024-09-16T03:42:23.480 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:23.480 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:23.480 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:23.480 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:23.481 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:23.481 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:23.481 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:23.481 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:23.481 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:23.481 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:23.481 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.465+0000 7fc4f1d08640 10 monclient: get_monmap_and_config success 2024-09-16T03:42:23.481 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.465+0000 7fc4f1d08640 4 set_mon_vals no callback set 2024-09-16T03:42:23.481 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.465+0000 7fc4f1d08640 10 monclient: shutdown 2024-09-16T03:42:23.481 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.465+0000 7fc4f1d08640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:42:23.481 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.465+0000 7fc4f1d08640 1 -- 172.21.15.42:0/3770509009 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc4ec072d70 msgr2=0x7fc4ec0a1330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:23.481 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.465+0000 7fc4f1d08640 1 --2- 172.21.15.42:0/3770509009 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc4ec072d70 0x7fc4ec0a1330 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fc4ec0a3330 tx=0x7fc4dc01c440 comp rx=0 tx=0).stop 2024-09-16T03:42:23.481 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.465+0000 7fc4f1d08640 1 -- 172.21.15.42:0/3770509009 shutdown_connections 2024-09-16T03:42:23.481 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.465+0000 7fc4f1d08640 1 --2- 172.21.15.42:0/3770509009 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc4ec073200 0x7fc4ec0a1870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:23.482 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.465+0000 7fc4f1d08640 1 --2- 172.21.15.42:0/3770509009 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc4ec072d70 0x7fc4ec0a1330 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:23.482 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.465+0000 7fc4f1d08640 1 --2- 172.21.15.42:0/3770509009 >> v2:172.21.15.154:3300/0 conn(0x7fc4ec0a2d60 0x7fc4ec134d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:23.482 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.465+0000 7fc4f1d08640 1 -- 172.21.15.42:0/3770509009 >> 172.21.15.42:0/3770509009 conn(0x7fc4ec0a62b0 msgr2=0x7fc4ec0a2420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:42:23.482 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.465+0000 7fc4f1d08640 1 -- 172.21.15.42:0/3770509009 shutdown_connections 2024-09-16T03:42:23.482 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.465+0000 7fc4f1d08640 1 -- 172.21.15.42:0/3770509009 wait complete. 2024-09-16T03:42:23.482 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.465+0000 7fc4f1d08640 10 monclient: build_initial_monmap 2024-09-16T03:42:23.482 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.465+0000 7fc4f1d08640 10 monclient: monmap: 2024-09-16T03:42:23.482 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:42:23.482 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:42:23.482 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:42:23.482 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:42:23.482 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:42:23.482 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:23.483 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:42:23.483 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:42:23.483 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:42:23.483 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:23.483 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.465+0000 7fc4f1d08640 1 librados: starting msgr at 2024-09-16T03:42:23.483 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.465+0000 7fc4f1d08640 1 librados: starting objecter 2024-09-16T03:42:23.483 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.465+0000 7fc4f1d08640 1 Processor -- start 2024-09-16T03:42:23.483 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.465+0000 7fc4f1d08640 1 -- start start 2024-09-16T03:42:23.483 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.465+0000 7fc4f1d08640 1 librados: setting wanted keys 2024-09-16T03:42:23.483 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.465+0000 7fc4f1d08640 1 librados: calling monclient init 2024-09-16T03:42:23.483 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.465+0000 7fc4f1d08640 10 monclient: init 2024-09-16T03:42:23.483 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.465+0000 7fc4f1d08640 10 monclient: _reopen_session rank -1 2024-09-16T03:42:23.483 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.465+0000 7fc4f1d08640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T03:42:23.483 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.465+0000 7fc4f1d08640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc4ec072d70 0x7fc4ec13b990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:23.483 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.465+0000 7fc4f1d08640 10 monclient(hunting): picked mon.noname-b con 0x7fc4ec072d70 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:42:23.484 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.465+0000 7fc4f1d08640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc4ec073200 0x7fc4ec136e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:23.484 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.465+0000 7fc4f1d08640 10 monclient(hunting): picked mon.noname-c con 0x7fc4ec073200 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:42:23.484 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.465+0000 7fc4f1d08640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc4ec0a2d60 0x7fc4ec137390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:23.484 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.465+0000 7fc4f1d08640 10 monclient(hunting): picked mon.noname-a con 0x7fc4ec0a2d60 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:42:23.484 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.465+0000 7fc4f1d08640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:23.484 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.465+0000 7fc4f1d08640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fc4ec13ebf0 con 0x7fc4ec0a2d60 2024-09-16T03:42:23.484 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.465+0000 7fc4f1d08640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:23.484 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.465+0000 7fc4f1d08640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc4ec13ea70 con 0x7fc4ec072d70 2024-09-16T03:42:23.484 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.465+0000 7fc4f1d08640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:23.484 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.465+0000 7fc4f1d08640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fc4ec13ed70 con 0x7fc4ec073200 2024-09-16T03:42:23.484 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.465+0000 7fc4f1d08640 10 monclient(hunting): _renew_subs 2024-09-16T03:42:23.484 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.465+0000 7fc4f1d08640 10 monclient(hunting): authenticate will time out at 990.439161s 2024-09-16T03:42:23.484 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.465+0000 7fc4eaffd640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc4ec073200 0x7fc4ec136e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:23.485 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.465+0000 7fc4eaffd640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc4ec073200 0x7fc4ec136e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:36152/0 (socket says 172.21.15.42:36152) 2024-09-16T03:42:23.485 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.465+0000 7fc4eaffd640 1 -- 172.21.15.42:0/2644748406 learned_addr learned my addr 172.21.15.42:0/2644748406 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:42:23.485 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.465+0000 7fc4eaffd640 10 monclient(hunting): get_auth_request con 0x7fc4ec073200 auth_method 0 2024-09-16T03:42:23.485 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.465+0000 7fc4eaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:23.485 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.465+0000 7fc4eaffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:23.485 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.465+0000 7fc4eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:23.485 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.465+0000 7fc4ebfff640 1 --2- 172.21.15.42:0/2644748406 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc4ec0a2d60 0x7fc4ec137390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:23.485 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.465+0000 7fc4ebfff640 10 monclient(hunting): get_auth_request con 0x7fc4ec0a2d60 auth_method 0 2024-09-16T03:42:23.485 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.465+0000 7fc4ebfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:23.485 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.465+0000 7fc4ebfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:23.485 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.465+0000 7fc4ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:23.485 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.465+0000 7fc4eaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:23.485 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.465+0000 7fc4eaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:23.485 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.465+0000 7fc4eaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:23.486 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.465+0000 7fc4ebfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:23.486 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.465+0000 7fc4ebfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:23.486 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.465+0000 7fc4ebfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:23.486 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.465+0000 7fc4eb7fe640 1 --2- 172.21.15.42:0/2644748406 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc4ec072d70 0x7fc4ec13b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:23.486 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.465+0000 7fc4eb7fe640 10 monclient(hunting): get_auth_request con 0x7fc4ec072d70 auth_method 0 2024-09-16T03:42:23.486 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.465+0000 7fc4eb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:23.486 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.465+0000 7fc4eb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:23.486 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.465+0000 7fc4eb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:23.486 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.465+0000 7fc4eaffd640 10 monclient(hunting): handle_auth_done global_id 4199 payload 824 2024-09-16T03:42:23.486 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.465+0000 7fc4eaffd640 1 -- 172.21.15.42:0/2644748406 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc4ec072d70 msgr2=0x7fc4ec13b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:23.486 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.465+0000 7fc4eaffd640 1 --2- 172.21.15.42:0/2644748406 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc4ec072d70 0x7fc4ec13b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:23.486 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.465+0000 7fc4eaffd640 1 -- 172.21.15.42:0/2644748406 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc4ec0a2d60 msgr2=0x7fc4ec137390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:23.486 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.465+0000 7fc4eaffd640 1 --2- 172.21.15.42:0/2644748406 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc4ec0a2d60 0x7fc4ec137390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:23.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.465+0000 7fc4eaffd640 10 monclient: _finish_hunting 0 2024-09-16T03:42:23.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.465+0000 7fc4eaffd640 1 monclient: found mon.noname-c 2024-09-16T03:42:23.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.465+0000 7fc4eaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:42:23.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.465+0000 7fc4eaffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:42:23.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.465+0000 7fc4eaffd640 1 -- 172.21.15.42:0/2644748406 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4ec13a210 con 0x7fc4ec073200 2024-09-16T03:42:23.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.465+0000 7fc4eaffd640 1 --2- 172.21.15.42:0/2644748406 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc4ec073200 0x7fc4ec136e50 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fc4dc000c10 tx=0x7fc4dc0191e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:23.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.465+0000 7fc4d3fff640 10 client.?.objecter ms_handle_connect 0x7fc4ec073200 2024-09-16T03:42:23.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.465+0000 7fc4d3fff640 10 client.?.objecter resend_mon_ops 2024-09-16T03:42:23.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.469+0000 7fc4d3fff640 1 -- 172.21.15.42:0/2644748406 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4dc0b1040 con 0x7fc4ec073200 2024-09-16T03:42:23.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.469+0000 7fc4d3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:23.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.469+0000 7fc4d3fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:42:23.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.469+0000 7fc4d3fff640 10 monclient: dump: 2024-09-16T03:42:23.487 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:23.487 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:23.488 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:23.488 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:23.488 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:23.488 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:23.488 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:23.488 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:23.488 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:23.488 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:23.488 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.469+0000 7fc4d3fff640 10 monclient: _finish_auth 0 2024-09-16T03:42:23.488 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.469+0000 7fc4d3fff640 10 monclient: _check_auth_tickets 2024-09-16T03:42:23.488 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.469+0000 7fc4d3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:42:23.488 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.469+0000 7fc4d3fff640 1 -- 172.21.15.42:0/2644748406 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc4dc0ba070 con 0x7fc4ec073200 2024-09-16T03:42:23.488 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.469+0000 7fc4d3fff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:42:23.488 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.469+0000 7fc4d3fff640 1 -- 172.21.15.42:0/2644748406 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4dc0b5630 con 0x7fc4ec073200 2024-09-16T03:42:23.489 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.469+0000 7fc4f1d08640 5 monclient: authenticate success, global_id 4199 2024-09-16T03:42:23.489 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.469+0000 7fc4d3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:23.489 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.469+0000 7fc4d3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:42:23.489 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.469+0000 7fc4d3fff640 10 monclient: dump: 2024-09-16T03:42:23.489 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:23.489 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:23.489 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:23.489 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:23.489 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:23.489 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:23.489 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:23.489 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:23.489 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:23.489 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:23.489 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.469+0000 7fc4f1d08640 10 monclient: _renew_subs 2024-09-16T03:42:23.490 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.469+0000 7fc4e97fa640 4 set_mon_vals no callback set 2024-09-16T03:42:23.490 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.469+0000 7fc4f1d08640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:42:23.490 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.469+0000 7fc4f1d08640 1 -- 172.21.15.42:0/2644748406 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4ec073a10 con 0x7fc4ec073200 2024-09-16T03:42:23.490 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.469+0000 7fc4f1d08640 10 client.4199.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:42:23.490 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.469+0000 7fc4f1d08640 10 monclient: _renew_subs 2024-09-16T03:42:23.490 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.469+0000 7fc4f1d08640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:42:23.490 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.469+0000 7fc4f1d08640 1 -- 172.21.15.42:0/2644748406 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4ec138a80 con 0x7fc4ec073200 2024-09-16T03:42:23.490 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.469+0000 7fc4f1d08640 1 librados: init done 2024-09-16T03:42:23.490 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.469+0000 7fc4f1d08640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:42:23.490 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.469+0000 7fc4f1d08640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:42:23.490 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.469+0000 7fc4f1d08640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:42:23.490 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.469+0000 7fc4f1d08640 1 -- 172.21.15.42:0/2644748406 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc4ec134d10 con 0x7fc4ec073200 2024-09-16T03:42:23.490 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.469+0000 7fc4d3fff640 1 -- 172.21.15.42:0/2644748406 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fc4dc0b57d0 con 0x7fc4ec073200 2024-09-16T03:42:23.490 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.469+0000 7fc4d3fff640 1 --2- 172.21.15.42:0/2644748406 >> v2:172.21.15.154:6812/254845342 conn(0x7fc4d8037840 0x7fc4d8039d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:23.491 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.469+0000 7fc4d3fff640 1 -- 172.21.15.42:0/2644748406 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(14..14 src has 1..14) ==== 3153+0+0 (secure 0 0 0) 0x7fc4dc0e2cc0 con 0x7fc4ec073200 2024-09-16T03:42:23.491 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.469+0000 7fc4d3fff640 10 client.4199.objecter ms_dispatch 0x7fc4ec000ba0 osd_map(14..14 src has 1..14) 2024-09-16T03:42:23.491 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.469+0000 7fc4d3fff640 3 client.4199.objecter handle_osd_map got epochs [14,14] > 0 2024-09-16T03:42:23.491 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.469+0000 7fc4d3fff640 3 client.4199.objecter handle_osd_map decoding full epoch 14 2024-09-16T03:42:23.491 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.469+0000 7fc4d3fff640 20 client.4199.objecter dump_active .. 0 homeless 2024-09-16T03:42:23.491 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.469+0000 7fc4eb7fe640 1 --2- 172.21.15.42:0/2644748406 >> v2:172.21.15.154:6812/254845342 conn(0x7fc4d8037840 0x7fc4d8039d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:23.491 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.469+0000 7fc4eb7fe640 10 monclient: get_auth_request con 0x7fc4d8037840 auth_method 0 2024-09-16T03:42:23.491 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.469+0000 7fc4d3fff640 10 client.4199.objecter ms_handle_connect 0x7fc4d8037840 2024-09-16T03:42:23.491 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.469+0000 7fc4eb7fe640 1 --2- 172.21.15.42:0/2644748406 >> v2:172.21.15.154:6812/254845342 conn(0x7fc4d8037840 0x7fc4d8039d00 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fc4e0009a70 tx=0x7fc4e0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:23.491 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.473+0000 7fc4d3fff640 1 -- 172.21.15.42:0/2644748406 <== mon.2 v2:172.21.15.42:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7fc4dc0160f0 con 0x7fc4ec073200 2024-09-16T03:42:23.492 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.473+0000 7fc4d3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:42:23.492 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.473+0000 7fc4d3fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:42:23.641 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.633+0000 7fc4f1d08640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-09-16T03:42:23.641 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.633+0000 7fc4f1d08640 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-09-16T03:42:23.641 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.633+0000 7fc4f1d08640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:42:23.641 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:23.633+0000 7fc4f1d08640 1 -- 172.21.15.42:0/2644748406 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) -- 0x7fc4ec069040 con 0x7fc4ec073200 2024-09-16T03:42:24.599 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.593+0000 7fc4d3fff640 1 -- 172.21.15.42:0/2644748406 <== mon.2 v2:172.21.15.42:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]=0 pool 'rbd' created v15) ==== 109+0+0 (secure 0 0 0) 0x7fc4dc0be580 con 0x7fc4ec073200 2024-09-16T03:42:24.599 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.593+0000 7fc4d3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-09-16T03:42:24.599 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.593+0000 7fc4d3fff640 10 monclient: _finish_command 2 = system:0 pool 'rbd' created 2024-09-16T03:42:24.601 INFO:teuthology.orchestra.run.smithi042.stderr:pool 'rbd' created 2024-09-16T03:42:24.605 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.597+0000 7fc4f1d08640 10 librados: watch_flush enter 2024-09-16T03:42:24.605 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.597+0000 7fc4f1d08640 10 librados: watch_flush exit 2024-09-16T03:42:24.605 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.597+0000 7fc4f1d08640 20 client.4199.objecter shutdown clearing up homeless session... 2024-09-16T03:42:24.605 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.597+0000 7fc4f1d08640 10 client.4199.objecter successfully canceled tick 2024-09-16T03:42:24.605 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.597+0000 7fc4f1d08640 1 -- 172.21.15.42:0/2644748406 >> v2:172.21.15.154:6812/254845342 conn(0x7fc4d8037840 msgr2=0x7fc4d8039d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:24.605 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.597+0000 7fc4f1d08640 1 --2- 172.21.15.42:0/2644748406 >> v2:172.21.15.154:6812/254845342 conn(0x7fc4d8037840 0x7fc4d8039d00 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fc4e0009a70 tx=0x7fc4e0008040 comp rx=0 tx=0).stop 2024-09-16T03:42:24.605 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.597+0000 7fc4f1d08640 10 monclient: shutdown 2024-09-16T03:42:24.605 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.597+0000 7fc4f1d08640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:42:24.605 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.597+0000 7fc4f1d08640 1 -- 172.21.15.42:0/2644748406 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc4ec073200 msgr2=0x7fc4ec136e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:24.605 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.597+0000 7fc4f1d08640 1 --2- 172.21.15.42:0/2644748406 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc4ec073200 0x7fc4ec136e50 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fc4dc000c10 tx=0x7fc4dc0191e0 comp rx=0 tx=0).stop 2024-09-16T03:42:24.605 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.597+0000 7fc4f1d08640 1 -- 172.21.15.42:0/2644748406 shutdown_connections 2024-09-16T03:42:24.606 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.597+0000 7fc4f1d08640 1 --2- 172.21.15.42:0/2644748406 >> v2:172.21.15.154:6812/254845342 conn(0x7fc4d8037840 0x7fc4d8039d00 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:24.606 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.597+0000 7fc4f1d08640 1 --2- 172.21.15.42:0/2644748406 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc4ec0a2d60 0x7fc4ec137390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:24.606 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.597+0000 7fc4f1d08640 1 --2- 172.21.15.42:0/2644748406 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc4ec073200 0x7fc4ec136e50 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:24.606 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.597+0000 7fc4f1d08640 1 --2- 172.21.15.42:0/2644748406 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc4ec072d70 0x7fc4ec13b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:24.606 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.597+0000 7fc4f1d08640 1 -- 172.21.15.42:0/2644748406 >> 172.21.15.42:0/2644748406 conn(0x7fc4ec0a62b0 msgr2=0x7fc4ec131720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:42:24.606 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.597+0000 7fc4f1d08640 1 -- 172.21.15.42:0/2644748406 shutdown_connections 2024-09-16T03:42:24.606 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.597+0000 7fc4f1d08640 1 -- 172.21.15.42:0/2644748406 wait complete. 2024-09-16T03:42:24.606 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.597+0000 7fc4f1d08640 1 librados: shutdown 2024-09-16T03:42:24.626 DEBUG:teuthology.orchestra.run.smithi042:> rbd --cluster ceph pool init rbd 2024-09-16T03:42:24.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.677+0000 7f6be7d5ff80 10 monclient: get_monmap_and_config 2024-09-16T03:42:24.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.677+0000 7f6be7d5ff80 10 monclient: build_initial_monmap 2024-09-16T03:42:24.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.677+0000 7f6be7d5ff80 10 monclient: monmap: 2024-09-16T03:42:24.685 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:42:24.685 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:24.685 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:42:24.683231+0000 2024-09-16T03:42:24.686 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:42:24.683231+0000 2024-09-16T03:42:24.686 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:42:24.686 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:24.686 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:42:24.686 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:42:24.686 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:42:24.686 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:42:24.686 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:42:24.686 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:24.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.681+0000 7f6be7d5ff80 1 Processor -- start 2024-09-16T03:42:24.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.681+0000 7f6be7d5ff80 1 -- start start 2024-09-16T03:42:24.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.681+0000 7f6be7d5ff80 10 monclient: init 2024-09-16T03:42:24.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.681+0000 7f6be7d5ff80 10 monclient: _reopen_session rank -1 2024-09-16T03:42:24.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.681+0000 7f6be7d5ff80 10 monclient: _add_conns ranks=[0,1,4,3,2] 2024-09-16T03:42:24.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.681+0000 7f6be7d5ff80 10 monclient(hunting): picked mon.noname-a-legacy con 0x56034ec6c150 addr v1:172.21.15.42:6789/0 2024-09-16T03:42:24.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.681+0000 7f6be7d5ff80 10 monclient(hunting): picked mon.noname-c-legacy con 0x56034ec6bb10 addr v1:172.21.15.154:6789/0 2024-09-16T03:42:24.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.681+0000 7f6be7d5ff80 1 --2- >> v2:172.21.15.154:3300/0 conn(0x56034ebba140 0x56034ec6c530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:24.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.681+0000 7f6be7d5ff80 10 monclient(hunting): picked mon.noname-c con 0x56034ebba140 addr v2:172.21.15.154:3300/0 2024-09-16T03:42:24.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.681+0000 7f6be7d5ff80 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x56034ea55ea0 con 0x56034ec6c150 2024-09-16T03:42:24.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.681+0000 7f6be7d5ff80 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x56034ea3c180 con 0x56034ec6bb10 2024-09-16T03:42:24.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.681+0000 7f6be7d5ff80 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:24.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.681+0000 7f6be7d5ff80 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x56034ea4ae20 con 0x56034ebba140 2024-09-16T03:42:24.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.681+0000 7f6be7d5ff80 10 monclient(hunting): _renew_subs 2024-09-16T03:42:24.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.681+0000 7f6be7d5ff80 10 monclient(hunting): authenticate will time out at 991.652917s 2024-09-16T03:42:24.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.681+0000 7f6be6873640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x56034ec6c150 0x56034ebd7a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:46718/0 (socket says 172.21.15.42:46718) 2024-09-16T03:42:24.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.681+0000 7f6be6873640 1 -- 172.21.15.42:0/872103834 learned_addr learned my addr 172.21.15.42:0/872103834 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:42:24.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.681+0000 7f6be7ad5640 1 --2- 172.21.15.42:0/872103834 >> v2:172.21.15.154:3300/0 conn(0x56034ebba140 0x56034ec6c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:24.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.681+0000 7f6be7ad5640 10 monclient(hunting): get_auth_request con 0x56034ebba140 auth_method 0 2024-09-16T03:42:24.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.681+0000 7f6be7ad5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:24.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.681+0000 7f6be7ad5640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:24.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.681+0000 7f6be7ad5640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:24.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.681+0000 7f6be5871640 1 -- 172.21.15.42:0/872103834 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 475315507 0 0) 0x56034ea55ea0 con 0x56034ec6c150 2024-09-16T03:42:24.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.681+0000 7f6be5871640 10 monclient(hunting): _negotiate 2024-09-16T03:42:24.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.681+0000 7f6be5871640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:24.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.681+0000 7f6be5871640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:24.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.681+0000 7f6be5871640 10 monclient(hunting): my global_id is 4221 2024-09-16T03:42:24.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.681+0000 7f6be5871640 1 -- 172.21.15.42:0/872103834 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6bd4001e20 con 0x56034ec6c150 2024-09-16T03:42:24.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.681+0000 7f6be7ad5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:24.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.681+0000 7f6be7ad5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:24.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.681+0000 7f6be7ad5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:24.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.681+0000 7f6be5871640 1 -- 172.21.15.42:0/872103834 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1507978980 0 0) 0x7f6bd4001e20 con 0x56034ec6c150 2024-09-16T03:42:24.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.681+0000 7f6be5871640 1 -- 172.21.15.42:0/872103834 >> v2:172.21.15.154:3300/0 conn(0x56034ebba140 msgr2=0x56034ec6c530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:42:24.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.681+0000 7f6be5871640 1 --2- 172.21.15.42:0/872103834 >> v2:172.21.15.154:3300/0 conn(0x56034ebba140 0x56034ec6c530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:24.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.681+0000 7f6be5871640 1 -- 172.21.15.42:0/872103834 >> v1:172.21.15.154:6789/0 conn(0x56034ec6bb10 legacy=0x56034ebd8190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:24.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.681+0000 7f6be5871640 10 monclient: _finish_hunting 0 2024-09-16T03:42:24.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.681+0000 7f6be5871640 1 monclient: found mon.noname-a-legacy 2024-09-16T03:42:24.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.681+0000 7f6be5871640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:42:24.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.681+0000 7f6be5871640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.42:6789/0 2024-09-16T03:42:24.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.681+0000 7f6be5871640 1 -- 172.21.15.42:0/872103834 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x56034ea26990 con 0x56034ec6c150 2024-09-16T03:42:24.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.681+0000 7f6be5871640 10 monclient: _finish_auth 0 2024-09-16T03:42:24.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.681+0000 7f6be5871640 10 monclient: _check_auth_tickets 2024-09-16T03:42:24.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.681+0000 7f6be5871640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:42:24.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.681+0000 7f6be5871640 1 -- 172.21.15.42:0/872103834 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2177413734 0 0) 0x7f6bdc001680 con 0x56034ec6c150 2024-09-16T03:42:24.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.681+0000 7f6be5871640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:24.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.681+0000 7f6be5871640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T03:42:24.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.681+0000 7f6be5871640 10 monclient: dump: 2024-09-16T03:42:24.693 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:24.693 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:24.693 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:24.693 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:24.693 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:24.694 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:24.694 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:24.694 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:24.694 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:24.694 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:24.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.681+0000 7f6be5871640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] but i'm connected to v1:172.21.15.42:6789/0, reconnecting 2024-09-16T03:42:24.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.681+0000 7f6be5871640 10 monclient: _reopen_session rank -1 2024-09-16T03:42:24.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.681+0000 7f6be5871640 1 -- 172.21.15.42:0/872103834 >> v1:172.21.15.42:6789/0 conn(0x56034ec6c150 legacy=0x56034ebd7a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:24.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.681+0000 7f6be5871640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:42:24.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.681+0000 7f6be5871640 1 --2- 172.21.15.42:0/872103834 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f6bd40022b0 0x7f6bd40026d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:24.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.681+0000 7f6be5871640 10 monclient(hunting): picked mon.c con 0x7f6bd40022b0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:42:24.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.681+0000 7f6be5871640 1 --2- 172.21.15.42:0/872103834 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6bd4003100 0x7f6bd4003580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:24.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.681+0000 7f6be5871640 10 monclient(hunting): picked mon.a con 0x7f6bd4003100 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:42:24.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.681+0000 7f6be5871640 1 --2- 172.21.15.42:0/872103834 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6bd4003ac0 0x7f6bd400a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:24.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.681+0000 7f6be5871640 10 monclient(hunting): picked mon.b con 0x7f6bd4003ac0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:42:24.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.681+0000 7f6be5871640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:24.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.681+0000 7f6be5871640 1 -- 172.21.15.42:0/872103834 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x56034ea4ae20 con 0x7f6bd4003100 2024-09-16T03:42:24.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.681+0000 7f6be5871640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:24.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.681+0000 7f6be5871640 1 -- 172.21.15.42:0/872103834 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f6bd400aa60 con 0x7f6bd4003ac0 2024-09-16T03:42:24.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.681+0000 7f6be5871640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:24.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.681+0000 7f6be5871640 1 -- 172.21.15.42:0/872103834 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f6bd400ac60 con 0x7f6bd40022b0 2024-09-16T03:42:24.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.681+0000 7f6be5871640 10 monclient(hunting): _renew_subs 2024-09-16T03:42:24.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.681+0000 7f6be5871640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:42:24.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.681+0000 7f6be6072640 1 --2- 172.21.15.42:0/872103834 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6bd4003100 0x7f6bd4003580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:24.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.681+0000 7f6be6873640 1 --2- 172.21.15.42:0/872103834 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f6bd40022b0 0x7f6bd40026d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:24.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.681+0000 7f6be6072640 10 monclient(hunting): get_auth_request con 0x7f6bd4003100 auth_method 0 2024-09-16T03:42:24.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.681+0000 7f6be6072640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:24.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.681+0000 7f6be6072640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:24.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.681+0000 7f6be6072640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:42:24.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.681+0000 7f6be6873640 10 monclient(hunting): get_auth_request con 0x7f6bd40022b0 auth_method 0 2024-09-16T03:42:24.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.681+0000 7f6be6873640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:24.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.681+0000 7f6be6873640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:24.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.681+0000 7f6be6873640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:42:24.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.681+0000 7f6be6072640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:24.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.681+0000 7f6be6072640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:24.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.681+0000 7f6be6072640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T03:42:24.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.681+0000 7f6be7ad5640 1 --2- 172.21.15.42:0/872103834 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6bd4003ac0 0x7f6bd400a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:24.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.681+0000 7f6be7ad5640 10 monclient(hunting): get_auth_request con 0x7f6bd4003ac0 auth_method 0 2024-09-16T03:42:24.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.681+0000 7f6be7ad5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:24.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.681+0000 7f6be7ad5640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:24.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.681+0000 7f6be7ad5640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:42:24.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.681+0000 7f6be6873640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:24.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.681+0000 7f6be6873640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:24.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.681+0000 7f6be6873640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T03:42:24.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.681+0000 7f6be6072640 10 monclient(hunting): handle_auth_done global_id 4221 payload 293 2024-09-16T03:42:24.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.681+0000 7f6be6072640 1 -- 172.21.15.42:0/872103834 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f6bd40022b0 msgr2=0x7f6bd40026d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:42:24.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.681+0000 7f6be6072640 1 --2- 172.21.15.42:0/872103834 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f6bd40022b0 0x7f6bd40026d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:24.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.681+0000 7f6be6072640 1 -- 172.21.15.42:0/872103834 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6bd4003ac0 msgr2=0x7f6bd400a2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:42:24.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.681+0000 7f6be6072640 1 --2- 172.21.15.42:0/872103834 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6bd4003ac0 0x7f6bd400a2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:24.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.681+0000 7f6be6072640 10 monclient: _finish_hunting 0 2024-09-16T03:42:24.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.681+0000 7f6be6072640 1 monclient: found mon.a 2024-09-16T03:42:24.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.681+0000 7f6be6072640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:42:24.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.681+0000 7f6be6072640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:42:24.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.681+0000 7f6be6072640 1 -- 172.21.15.42:0/872103834 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6bd4001e20 con 0x7f6bd4003100 2024-09-16T03:42:24.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.681+0000 7f6be6072640 10 monclient: _finish_auth 0 2024-09-16T03:42:24.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.681+0000 7f6be6072640 10 monclient: _check_auth_tickets 2024-09-16T03:42:24.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.681+0000 7f6be6072640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:42:24.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.681+0000 7f6be7d5ff80 5 monclient: authenticate success, global_id 4221 2024-09-16T03:42:24.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.681+0000 7f6be7d5ff80 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:42:24.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.681+0000 7f6be7ad5640 1 --2- 172.21.15.42:0/872103834 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6bd4003ac0 0x7f6bd400a2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T03:42:24.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.681+0000 7f6be7ad5640 1 -- 172.21.15.42:0/872103834 reap_dead start 2024-09-16T03:42:24.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.685+0000 7f6be6072640 1 --2- 172.21.15.42:0/872103834 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6bd4003100 0x7f6bd4003580 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f6bd0001e40 tx=0x7f6bd001aeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3630689a6fb9927 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:24.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.685+0000 7f6be5871640 1 -- 172.21.15.42:0/872103834 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bd000a040 con 0x7f6bd4003100 2024-09-16T03:42:24.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.685+0000 7f6be5871640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:24.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.685+0000 7f6be5871640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:42:24.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.685+0000 7f6be5871640 10 monclient: dump: 2024-09-16T03:42:24.704 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:24.704 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:24.704 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:24.704 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:24.704 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:24.704 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:24.705 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:24.705 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:24.705 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:24.705 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:24.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.685+0000 7f6be7d5ff80 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:42:24.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.685+0000 7f6be5871640 1 -- 172.21.15.42:0/872103834 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6bd00a3b80 con 0x7f6bd4003100 2024-09-16T03:42:24.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.685+0000 7f6be5871640 10 monclient: handle_config config(0 keys) 2024-09-16T03:42:24.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.685+0000 7f6be7d5ff80 10 monclient: get_monmap_and_config success 2024-09-16T03:42:24.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.685+0000 7f6be7d5ff80 4 set_mon_vals no callback set 2024-09-16T03:42:24.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.685+0000 7f6be5871640 1 -- 172.21.15.42:0/872103834 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bd00ab3f0 con 0x7f6bd4003100 2024-09-16T03:42:24.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.685+0000 7f6be5871640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:24.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.685+0000 7f6be5871640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:42:24.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.685+0000 7f6be5871640 10 monclient: dump: 2024-09-16T03:42:24.706 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:24.706 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:24.706 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:24.706 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:24.706 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:24.706 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:24.706 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:24.706 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:24.706 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:24.706 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:24.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.685+0000 7f6be7d5ff80 10 monclient: shutdown 2024-09-16T03:42:24.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.685+0000 7f6be7d5ff80 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:42:24.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.685+0000 7f6be7d5ff80 1 -- 172.21.15.42:0/872103834 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6bd4003100 msgr2=0x7f6bd4003580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:24.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.685+0000 7f6be7d5ff80 1 --2- 172.21.15.42:0/872103834 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6bd4003100 0x7f6bd4003580 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f6bd0001e40 tx=0x7f6bd001aeb0 comp rx=0 tx=0).stop 2024-09-16T03:42:24.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.685+0000 7f6be7d5ff80 1 -- 172.21.15.42:0/872103834 shutdown_connections 2024-09-16T03:42:24.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.685+0000 7f6be7d5ff80 1 --2- 172.21.15.42:0/872103834 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6bd4003100 0x7f6bd4003580 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:24.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.685+0000 7f6be7d5ff80 1 -- 172.21.15.42:0/872103834 >> 172.21.15.42:0/872103834 conn(0x56034ebdc7b0 msgr2=0x56034ebd8b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:42:24.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.685+0000 7f6be7d5ff80 1 -- 172.21.15.42:0/872103834 shutdown_connections 2024-09-16T03:42:24.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.685+0000 7f6be7d5ff80 1 -- 172.21.15.42:0/872103834 wait complete. 2024-09-16T03:42:24.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.685+0000 7f6be7d5ff80 10 monclient: build_initial_monmap 2024-09-16T03:42:24.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.685+0000 7f6be7d5ff80 10 monclient: monmap: 2024-09-16T03:42:24.707 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:42:24.707 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:42:24.707 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:42:24.707 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:42:24.707 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:42:24.707 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:24.707 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:42:24.708 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:42:24.708 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:42:24.708 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:24.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.685+0000 7f6be7d5ff80 1 librados: starting msgr at 2024-09-16T03:42:24.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.685+0000 7f6be7d5ff80 1 librados: starting objecter 2024-09-16T03:42:24.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.685+0000 7f6be7d5ff80 1 Processor -- start 2024-09-16T03:42:24.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.685+0000 7f6be7d5ff80 1 -- start start 2024-09-16T03:42:24.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.685+0000 7f6be7d5ff80 1 librados: setting wanted keys 2024-09-16T03:42:24.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.685+0000 7f6be7d5ff80 1 librados: calling monclient init 2024-09-16T03:42:24.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.685+0000 7f6be7d5ff80 10 monclient: init 2024-09-16T03:42:24.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.685+0000 7f6be7d5ff80 10 monclient: _reopen_session rank -1 2024-09-16T03:42:24.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.685+0000 7f6be7d5ff80 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T03:42:24.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.685+0000 7f6be7d5ff80 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6bd4003100 0x56034ec7e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:24.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.685+0000 7f6be7d5ff80 10 monclient(hunting): picked mon.noname-a con 0x7f6bd4003100 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:42:24.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.685+0000 7f6be7d5ff80 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56034ec7eca0 0x56034ec54160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:24.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.685+0000 7f6be7d5ff80 10 monclient(hunting): picked mon.noname-b con 0x56034ec7eca0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:42:24.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.685+0000 7f6be7d5ff80 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x56034ec7f0e0 0x56034ec546a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:24.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.685+0000 7f6be7d5ff80 10 monclient(hunting): picked mon.noname-c con 0x56034ec7f0e0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:42:24.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.685+0000 7f6be7d5ff80 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:24.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.685+0000 7f6be7d5ff80 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x56034eaa1800 con 0x7f6bd4003100 2024-09-16T03:42:24.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.685+0000 7f6be7d5ff80 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:24.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.685+0000 7f6be7d5ff80 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x56034eaa5840 con 0x56034ec7eca0 2024-09-16T03:42:24.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.685+0000 7f6be7d5ff80 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:24.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.685+0000 7f6be7d5ff80 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x56034ebd1c40 con 0x56034ec7f0e0 2024-09-16T03:42:24.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.685+0000 7f6be7d5ff80 10 monclient(hunting): _renew_subs 2024-09-16T03:42:24.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.685+0000 7f6be7d5ff80 10 monclient(hunting): authenticate will time out at 991.659785s 2024-09-16T03:42:24.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.685+0000 7f6be6873640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6bd4003100 0x56034ec7e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:24.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.685+0000 7f6be6873640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6bd4003100 0x56034ec7e760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:37512/0 (socket says 172.21.15.42:37512) 2024-09-16T03:42:24.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.685+0000 7f6be6873640 1 -- 172.21.15.42:0/3600569605 learned_addr learned my addr 172.21.15.42:0/3600569605 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:42:24.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.685+0000 7f6be6873640 10 monclient(hunting): get_auth_request con 0x7f6bd4003100 auth_method 0 2024-09-16T03:42:24.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.685+0000 7f6be6873640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:24.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.685+0000 7f6be6873640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:24.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.685+0000 7f6be6873640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:24.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.685+0000 7f6be7ad5640 1 --2- 172.21.15.42:0/3600569605 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x56034ec7f0e0 0x56034ec546a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:24.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.685+0000 7f6be7ad5640 10 monclient(hunting): get_auth_request con 0x56034ec7f0e0 auth_method 0 2024-09-16T03:42:24.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.685+0000 7f6be7ad5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:24.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.685+0000 7f6be7ad5640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:24.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.685+0000 7f6be7ad5640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:24.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.689+0000 7f6be6873640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:24.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.689+0000 7f6be6873640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:24.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.689+0000 7f6be6873640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:24.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.689+0000 7f6be7ad5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:24.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.689+0000 7f6be7ad5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:24.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.689+0000 7f6be7ad5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:24.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.689+0000 7f6be6072640 1 --2- 172.21.15.42:0/3600569605 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56034ec7eca0 0x56034ec54160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:24.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.689+0000 7f6be6873640 10 monclient(hunting): handle_auth_done global_id 4224 payload 824 2024-09-16T03:42:24.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.689+0000 7f6be6873640 1 -- 172.21.15.42:0/3600569605 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x56034ec7f0e0 msgr2=0x56034ec546a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:24.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.689+0000 7f6be6873640 1 --2- 172.21.15.42:0/3600569605 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x56034ec7f0e0 0x56034ec546a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:24.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.689+0000 7f6be6873640 1 -- 172.21.15.42:0/3600569605 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56034ec7eca0 msgr2=0x56034ec54160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:24.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.689+0000 7f6be6873640 1 --2- 172.21.15.42:0/3600569605 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56034ec7eca0 0x56034ec54160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:24.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.689+0000 7f6be6873640 10 monclient: _finish_hunting 0 2024-09-16T03:42:24.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.689+0000 7f6be6873640 1 monclient: found mon.noname-a 2024-09-16T03:42:24.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.689+0000 7f6be6873640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:42:24.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.689+0000 7f6be6873640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:42:24.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.689+0000 7f6be6873640 1 -- 172.21.15.42:0/3600569605 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56034ec58050 con 0x7f6bd4003100 2024-09-16T03:42:24.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.689+0000 7f6be6072640 10 monclient: get_auth_request con 0x56034ec7eca0 auth_method 0 2024-09-16T03:42:24.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.689+0000 7f6be6072640 1 --2- 172.21.15.42:0/3600569605 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x56034ec7eca0 0x56034ec54160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T03:42:24.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.689+0000 7f6be6873640 1 --2- 172.21.15.42:0/3600569605 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6bd4003100 0x56034ec7e760 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f6bdc000fc0 tx=0x7f6bdc009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:24.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.689+0000 7f6bceffd640 10 client.?.objecter ms_handle_connect 0x7f6bd4003100 2024-09-16T03:42:24.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.689+0000 7f6bceffd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:42:24.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.689+0000 7f6bceffd640 1 -- 172.21.15.42:0/3600569605 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bdc00ca80 con 0x7f6bd4003100 2024-09-16T03:42:24.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.689+0000 7f6bceffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:24.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.689+0000 7f6bceffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:42:24.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.689+0000 7f6bceffd640 10 monclient: dump: 2024-09-16T03:42:24.712 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:24.712 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:24.712 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:24.713 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:24.713 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:24.713 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:24.713 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:24.713 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:24.713 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:24.713 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:24.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.689+0000 7f6bceffd640 10 monclient: _finish_auth 0 2024-09-16T03:42:24.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.689+0000 7f6bceffd640 10 monclient: _check_auth_tickets 2024-09-16T03:42:24.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.689+0000 7f6bceffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:42:24.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.689+0000 7f6bceffd640 1 -- 172.21.15.42:0/3600569605 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6bdc00cc40 con 0x7f6bd4003100 2024-09-16T03:42:24.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.689+0000 7f6be7d5ff80 5 monclient: authenticate success, global_id 4224 2024-09-16T03:42:24.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.689+0000 7f6bceffd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:42:24.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.689+0000 7f6bceffd640 1 -- 172.21.15.42:0/3600569605 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bdc017440 con 0x7f6bd4003100 2024-09-16T03:42:24.714 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.689+0000 7f6bceffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:24.714 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.689+0000 7f6bceffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:42:24.714 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.689+0000 7f6bceffd640 10 monclient: dump: 2024-09-16T03:42:24.714 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:24.714 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:24.714 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:24.714 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:24.714 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:24.714 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:24.714 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:24.714 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:24.714 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:24.714 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:24.714 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.689+0000 7f6be5070640 4 set_mon_vals no callback set 2024-09-16T03:42:24.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.689+0000 7f6be7d5ff80 10 monclient: _renew_subs 2024-09-16T03:42:24.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.689+0000 7f6be7d5ff80 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:42:24.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.689+0000 7f6be7d5ff80 1 -- 172.21.15.42:0/3600569605 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x56034ebbca80 con 0x7f6bd4003100 2024-09-16T03:42:24.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.689+0000 7f6be7d5ff80 10 client.4224.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:42:24.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.689+0000 7f6be7d5ff80 10 monclient: _renew_subs 2024-09-16T03:42:24.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.689+0000 7f6be7d5ff80 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:42:24.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.689+0000 7f6be7d5ff80 1 -- 172.21.15.42:0/3600569605 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56034ec56640 con 0x7f6bd4003100 2024-09-16T03:42:24.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.689+0000 7f6be7d5ff80 1 librados: init done 2024-09-16T03:42:24.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.689+0000 7f6be7d5ff80 10 librados: wait_for_osdmap waiting 2024-09-16T03:42:24.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.689+0000 7f6bceffd640 1 -- 172.21.15.42:0/3600569605 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f6bdc0213f0 con 0x7f6bd4003100 2024-09-16T03:42:24.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.689+0000 7f6bceffd640 1 --2- 172.21.15.42:0/3600569605 >> v2:172.21.15.154:6812/254845342 conn(0x7f6bc8035940 0x7f6bc8037e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:24.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.689+0000 7f6bceffd640 1 -- 172.21.15.42:0/3600569605 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(15..15 src has 1..15) ==== 3495+0+0 (secure 0 0 0) 0x7f6bdc047f30 con 0x7f6bd4003100 2024-09-16T03:42:24.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.689+0000 7f6bceffd640 10 client.4224.objecter ms_dispatch 0x56034eabc7b0 osd_map(15..15 src has 1..15) 2024-09-16T03:42:24.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.689+0000 7f6bceffd640 3 client.4224.objecter handle_osd_map got epochs [15,15] > 0 2024-09-16T03:42:24.716 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.689+0000 7f6bceffd640 3 client.4224.objecter handle_osd_map decoding full epoch 15 2024-09-16T03:42:24.716 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.689+0000 7f6bceffd640 20 client.4224.objecter dump_active .. 0 homeless 2024-09-16T03:42:24.716 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.689+0000 7f6be7d5ff80 10 librados: wait_for_osdmap done waiting 2024-09-16T03:42:24.716 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.689+0000 7f6be7d5ff80 10 start_mon_command cmd=[{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}] 2024-09-16T03:42:24.716 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.689+0000 7f6be7d5ff80 10 monclient: _send_command 1 [{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}] 2024-09-16T03:42:24.716 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.689+0000 7f6be7d5ff80 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:42:24.716 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.689+0000 7f6be7d5ff80 1 -- 172.21.15.42:0/3600569605 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "osd pool application enable","pool": "rbd","app": "rbd"} v 0) -- 0x56034ec54be0 con 0x7f6bd4003100 2024-09-16T03:42:24.716 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.689+0000 7f6be6072640 1 --2- 172.21.15.42:0/3600569605 >> v2:172.21.15.154:6812/254845342 conn(0x7f6bc8035940 0x7f6bc8037e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:24.716 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.689+0000 7f6be6072640 10 monclient: get_auth_request con 0x7f6bc8035940 auth_method 0 2024-09-16T03:42:24.716 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.689+0000 7f6bceffd640 10 client.4224.objecter ms_handle_connect 0x7f6bc8035940 2024-09-16T03:42:24.716 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:24.689+0000 7f6be6072640 1 --2- 172.21.15.42:0/3600569605 >> v2:172.21.15.154:6812/254845342 conn(0x7f6bc8035940 0x7f6bc8037e00 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f6bd00ab000 tx=0x7f6bd00011a0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:25.601 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:25.593+0000 7f6bceffd640 1 -- 172.21.15.42:0/3600569605 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}]=0 enabled application 'rbd' on pool 'rbd' v16) ==== 141+0+0 (secure 0 0 0) 0x7f6bdc021df0 con 0x7f6bd4003100 2024-09-16T03:42:25.601 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:25.593+0000 7f6bceffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}] 2024-09-16T03:42:25.601 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:25.593+0000 7f6bceffd640 10 monclient: _finish_command 1 = system:0 enabled application 'rbd' on pool 'rbd' 2024-09-16T03:42:25.601 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:25.593+0000 7f6be7d5ff80 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:42:25.601 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:25.593+0000 7f6be7d5ff80 1 -- 172.21.15.42:0/3600569605 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f6bd00a3b80 con 0x7f6bd4003100 2024-09-16T03:42:25.609 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:25.601+0000 7f6bceffd640 1 -- 172.21.15.42:0/3600569605 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_get_version_reply(handle=1 version=16) ==== 24+0+0 (secure 0 0 0) 0x7f6bdc01c030 con 0x7f6bd4003100 2024-09-16T03:42:25.609 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:25.601+0000 7f6bceffd640 10 monclient: handle_get_version_reply finishing 1 version 16 2024-09-16T03:42:25.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:25.601+0000 7f6be5070640 10 client.4224.objecter _get_latest_version latest 16, waiting 2024-09-16T03:42:25.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:25.601+0000 7f6be5070640 10 client.4224.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:42:25.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:25.601+0000 7f6be5070640 10 monclient: _renew_subs 2024-09-16T03:42:25.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:25.601+0000 7f6be5070640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:42:25.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:25.601+0000 7f6be5070640 1 -- 172.21.15.42:0/3600569605 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=16}) -- 0x7f6bc0003ca0 con 0x7f6bd4003100 2024-09-16T03:42:25.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:25.605+0000 7f6bceffd640 1 -- 172.21.15.42:0/3600569605 <== mon.0 v2:172.21.15.42:3300/0 8 ==== osd_map(16..16 src has 1..16) ==== 658+0+0 (secure 0 0 0) 0x7f6bdc015500 con 0x7f6bd4003100 2024-09-16T03:42:25.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:25.605+0000 7f6bceffd640 10 client.4224.objecter ms_dispatch 0x56034eabc7b0 osd_map(16..16 src has 1..16) 2024-09-16T03:42:25.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:25.605+0000 7f6bceffd640 3 client.4224.objecter handle_osd_map got epochs [16,16] > 15 2024-09-16T03:42:25.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:25.605+0000 7f6bceffd640 3 client.4224.objecter handle_osd_map decoding incremental epoch 16 2024-09-16T03:42:25.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:25.605+0000 7f6bceffd640 20 client.4224.objecter dump_active .. 0 homeless 2024-09-16T03:42:25.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:25.605+0000 7f6be7d5ff80 10 librados: create oid=rbd_trash nspace= 2024-09-16T03:42:25.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:25.605+0000 7f6be7d5ff80 10 client.4224.objecter _op_submit op 0x56034ec85780 2024-09-16T03:42:25.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:25.605+0000 7f6be7d5ff80 20 client.4224.objecter _calc_target epoch 16 base rbd_trash @2 precalc_pgid 0 pgid 0.0 is_write 2024-09-16T03:42:25.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:25.605+0000 7f6be7d5ff80 20 client.4224.objecter _calc_target target rbd_trash @2 -> pgid 2.5a749323 2024-09-16T03:42:25.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:25.605+0000 7f6be7d5ff80 10 client.4224.objecter _calc_target raw pgid 2.5a749323 -> actual 2.3 acting [5,2] primary 5 2024-09-16T03:42:25.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:25.605+0000 7f6be7d5ff80 1 --2- 172.21.15.42:0/3600569605 >> v2:172.21.15.154:6808/3026127822 conn(0x56034ec86140 0x56034ec88530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:25.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:25.605+0000 7f6be7d5ff80 20 client.4224.objecter _get_session s=0x56034ec552e0 osd=5 3 2024-09-16T03:42:25.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:25.605+0000 7f6be7d5ff80 10 client.4224.objecter _op_submit oid rbd_trash '@2' '@2' [create] tid 1 osd.5 2024-09-16T03:42:25.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:25.605+0000 7f6be7d5ff80 20 client.4224.objecter get_session s=0x56034ec552e0 osd=5 3 2024-09-16T03:42:25.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:25.605+0000 7f6be7d5ff80 15 client.4224.objecter _session_op_assign 5 1 2024-09-16T03:42:25.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:25.605+0000 7f6be7d5ff80 15 client.4224.objecter _send_op 1 to 2.3 on osd.5 2024-09-16T03:42:25.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:25.605+0000 7f6be7d5ff80 1 -- 172.21.15.42:0/3600569605 --> v2:172.21.15.154:6808/3026127822 -- osd_op(unknown.0.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16) -- 0x56034ec88a70 con 0x56034ec86140 2024-09-16T03:42:25.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:25.605+0000 7f6be7d5ff80 20 client.4224.objecter put_session s=0x56034ec552e0 osd=5 4 2024-09-16T03:42:25.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:25.605+0000 7f6be7d5ff80 5 client.4224.objecter 1 in flight 2024-09-16T03:42:25.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:25.605+0000 7f6be7ad5640 1 --2- 172.21.15.42:0/3600569605 >> v2:172.21.15.154:6808/3026127822 conn(0x56034ec86140 0x56034ec88530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:25.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:25.609+0000 7f6be7ad5640 10 monclient: get_auth_request con 0x56034ec86140 auth_method 0 2024-09-16T03:42:25.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:25.609+0000 7f6bceffd640 10 client.4224.objecter ms_handle_connect 0x56034ec86140 2024-09-16T03:42:25.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:25.609+0000 7f6be7ad5640 1 --2- 172.21.15.42:0/3600569605 >> v2:172.21.15.154:6808/3026127822 conn(0x56034ec86140 0x56034ec88530 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:25.621 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:25.613+0000 7f6be7ad5640 1 -- 172.21.15.42:0/3600569605 <== osd.5 v2:172.21.15.154:6808/3026127822 1 ==== osd_op_reply(1 rbd_trash [create] v16'1 uv1 ondisk = 0) ==== 153+0+0 (crc 0 0 0) 0x7f6bd80106a0 con 0x56034ec86140 2024-09-16T03:42:25.621 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:25.613+0000 7f6be7ad5640 10 client.4224.objecter ms_dispatch 0x56034eabc7b0 osd_op_reply(1 rbd_trash [create] v16'1 uv1 ondisk = 0) 2024-09-16T03:42:25.621 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:25.613+0000 7f6be7ad5640 10 client.4224.objecter in handle_osd_op_reply 2024-09-16T03:42:25.621 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:25.613+0000 7f6be7ad5640 7 client.4224.objecter handle_osd_op_reply 1 ondisk uv 1 in 2.3 attempt 0 2024-09-16T03:42:25.622 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:25.613+0000 7f6be7ad5640 10 client.4224.objecter op 0 rval 0 len 0 2024-09-16T03:42:25.622 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:25.613+0000 7f6be7ad5640 15 client.4224.objecter handle_osd_op_reply completed tid 1 2024-09-16T03:42:25.622 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:25.613+0000 7f6be7ad5640 15 client.4224.objecter _finish_op 1 2024-09-16T03:42:25.622 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:25.613+0000 7f6be7ad5640 20 client.4224.objecter put_session s=0x56034ec552e0 osd=5 4 2024-09-16T03:42:25.622 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:25.613+0000 7f6be7ad5640 15 client.4224.objecter _session_op_remove 5 1 2024-09-16T03:42:25.622 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:25.613+0000 7f6be7ad5640 5 client.4224.objecter 0 in flight 2024-09-16T03:42:25.622 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:25.613+0000 7f6be7d5ff80 10 librados: Objecter returned from create r=0 2024-09-16T03:42:25.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:25.617+0000 7f6be7d5ff80 10 client.4224.objecter _op_submit op 0x56034ecda5b0 2024-09-16T03:42:25.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:25.617+0000 7f6be7d5ff80 20 client.4224.objecter _calc_target epoch 16 base rbd_info @2 precalc_pgid 0 pgid 0.0 is_read 2024-09-16T03:42:25.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:25.617+0000 7f6be7d5ff80 20 client.4224.objecter _calc_target target rbd_info @2 -> pgid 2.ac0e573a 2024-09-16T03:42:25.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:25.617+0000 7f6be7d5ff80 10 client.4224.objecter _calc_target raw pgid 2.ac0e573a -> actual 2.2 acting [5,1] primary 5 2024-09-16T03:42:25.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:25.617+0000 7f6be7d5ff80 20 client.4224.objecter _get_session s=0x56034ec552e0 osd=5 3 2024-09-16T03:42:25.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:25.617+0000 7f6be7d5ff80 10 client.4224.objecter _op_submit oid rbd_info '@2' '@2' [call rbd.metadata_list in=33b] tid 2 osd.5 2024-09-16T03:42:25.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:25.617+0000 7f6be7d5ff80 20 client.4224.objecter get_session s=0x56034ec552e0 osd=5 3 2024-09-16T03:42:25.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:25.617+0000 7f6be7d5ff80 15 client.4224.objecter _session_op_assign 5 2 2024-09-16T03:42:25.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:25.617+0000 7f6be7d5ff80 15 client.4224.objecter _send_op 2 to 2.2 on osd.5 2024-09-16T03:42:25.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:25.617+0000 7f6be7d5ff80 1 -- 172.21.15.42:0/3600569605 --> v2:172.21.15.154:6808/3026127822 -- osd_op(unknown.0.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=33b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e16) -- 0x56034ecdab00 con 0x56034ec86140 2024-09-16T03:42:25.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:25.617+0000 7f6be7d5ff80 20 client.4224.objecter put_session s=0x56034ec552e0 osd=5 4 2024-09-16T03:42:25.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:25.617+0000 7f6be7d5ff80 5 client.4224.objecter 1 in flight 2024-09-16T03:42:25.626 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:25.617+0000 7f6be7ad5640 1 -- 172.21.15.42:0/3600569605 <== osd.5 v2:172.21.15.154:6808/3026127822 2 ==== osd_op_reply(2 rbd_info [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 152+0+0 (crc 0 0 0) 0x7f6bd80106a0 con 0x56034ec86140 2024-09-16T03:42:25.626 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:25.617+0000 7f6be7ad5640 10 client.4224.objecter ms_dispatch 0x56034eabc7b0 osd_op_reply(2 rbd_info [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) 2024-09-16T03:42:25.626 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:25.617+0000 7f6be7ad5640 10 client.4224.objecter in handle_osd_op_reply 2024-09-16T03:42:25.626 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:25.617+0000 7f6be7ad5640 7 client.4224.objecter handle_osd_op_reply 2 ondisk uv 0 in 2.2 attempt 0 2024-09-16T03:42:25.626 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:25.617+0000 7f6be7ad5640 10 client.4224.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-09-16T03:42:25.626 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:25.617+0000 7f6be7ad5640 10 client.4224.objecter op 0 rval 0 len 0 2024-09-16T03:42:25.626 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:25.617+0000 7f6be7ad5640 15 client.4224.objecter handle_osd_op_reply completed tid 2 2024-09-16T03:42:25.626 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:25.617+0000 7f6be7ad5640 15 client.4224.objecter _finish_op 2 2024-09-16T03:42:25.626 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:25.617+0000 7f6be7ad5640 20 client.4224.objecter put_session s=0x56034ec552e0 osd=5 4 2024-09-16T03:42:25.626 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:25.617+0000 7f6be7ad5640 15 client.4224.objecter _session_op_remove 5 2 2024-09-16T03:42:25.626 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:25.617+0000 7f6be7ad5640 5 client.4224.objecter 0 in flight 2024-09-16T03:42:25.626 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:25.617+0000 7f6be7d5ff80 10 client.4224.objecter _op_submit op 0x56034ecd9710 2024-09-16T03:42:25.626 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:25.617+0000 7f6be7d5ff80 20 client.4224.objecter _calc_target epoch 16 base rbd_info @2 precalc_pgid 0 pgid 0.0 is_read 2024-09-16T03:42:25.627 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:25.617+0000 7f6be7d5ff80 20 client.4224.objecter _calc_target target rbd_info @2 -> pgid 2.ac0e573a 2024-09-16T03:42:25.627 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:25.617+0000 7f6be7d5ff80 10 client.4224.objecter _calc_target raw pgid 2.ac0e573a -> actual 2.2 acting [5,1] primary 5 2024-09-16T03:42:25.627 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:25.617+0000 7f6be7d5ff80 20 client.4224.objecter _get_session s=0x56034ec552e0 osd=5 3 2024-09-16T03:42:25.627 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:25.617+0000 7f6be7d5ff80 10 client.4224.objecter _op_submit oid rbd_info '@2' '@2' [read 0~0] tid 3 osd.5 2024-09-16T03:42:25.627 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:25.617+0000 7f6be7d5ff80 20 client.4224.objecter get_session s=0x56034ec552e0 osd=5 3 2024-09-16T03:42:25.627 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:25.617+0000 7f6be7d5ff80 15 client.4224.objecter _session_op_assign 5 3 2024-09-16T03:42:25.627 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:25.617+0000 7f6be7d5ff80 15 client.4224.objecter _send_op 3 to 2.2 on osd.5 2024-09-16T03:42:25.627 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:25.617+0000 7f6be7d5ff80 1 -- 172.21.15.42:0/3600569605 --> v2:172.21.15.154:6808/3026127822 -- osd_op(unknown.0.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e16) -- 0x56034ecd9bc0 con 0x56034ec86140 2024-09-16T03:42:25.627 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:25.617+0000 7f6be7d5ff80 20 client.4224.objecter put_session s=0x56034ec552e0 osd=5 4 2024-09-16T03:42:25.627 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:25.617+0000 7f6be7d5ff80 5 client.4224.objecter 1 in flight 2024-09-16T03:42:25.627 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:25.617+0000 7f6be7ad5640 1 -- 172.21.15.42:0/3600569605 <== osd.5 v2:172.21.15.154:6808/3026127822 3 ==== osd_op_reply(3 rbd_info [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 152+0+0 (crc 0 0 0) 0x7f6bd80106a0 con 0x56034ec86140 2024-09-16T03:42:25.627 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:25.617+0000 7f6be7ad5640 10 client.4224.objecter ms_dispatch 0x56034eabc7b0 osd_op_reply(3 rbd_info [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) 2024-09-16T03:42:25.627 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:25.617+0000 7f6be7ad5640 10 client.4224.objecter in handle_osd_op_reply 2024-09-16T03:42:25.628 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:25.617+0000 7f6be7ad5640 7 client.4224.objecter handle_osd_op_reply 3 ondisk uv 0 in 2.2 attempt 0 2024-09-16T03:42:25.628 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:25.617+0000 7f6be7ad5640 10 client.4224.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-09-16T03:42:25.628 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:25.617+0000 7f6be7ad5640 10 client.4224.objecter op 0 rval 0 len 0 2024-09-16T03:42:25.628 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:25.617+0000 7f6be7ad5640 15 client.4224.objecter handle_osd_op_reply completed tid 3 2024-09-16T03:42:25.628 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:25.617+0000 7f6be7ad5640 15 client.4224.objecter _finish_op 3 2024-09-16T03:42:25.628 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:25.617+0000 7f6be7ad5640 20 client.4224.objecter put_session s=0x56034ec552e0 osd=5 4 2024-09-16T03:42:25.628 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:25.617+0000 7f6be7ad5640 15 client.4224.objecter _session_op_remove 5 3 2024-09-16T03:42:25.628 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:25.617+0000 7f6be7ad5640 5 client.4224.objecter 0 in flight 2024-09-16T03:42:25.628 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:25.617+0000 7f6be486f640 10 client.4224.objecter allocate_selfmanaged_snap; pool: 2 2024-09-16T03:42:25.628 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:25.617+0000 7f6be486f640 10 client.4224.objecter pool_op_submit 4 2024-09-16T03:42:25.628 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:25.617+0000 7f6be486f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:42:25.628 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:25.617+0000 7f6be486f640 1 -- 172.21.15.42:0/3600569605 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- pool_op(create unmanaged snap pool 2 tid 4 name v0) -- 0x7f6bc40030e0 con 0x7f6bd4003100 2024-09-16T03:42:26.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:26.601+0000 7f6bceffd640 1 -- 172.21.15.42:0/3600569605 <== mon.0 v2:172.21.15.42:3300/0 9 ==== pool_op_reply(tid 4 (0) Success v17) ==== 55+0+0 (secure 0 0 0) 0x7f6bdc045030 con 0x7f6bd4003100 2024-09-16T03:42:26.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:26.601+0000 7f6bceffd640 10 client.4224.objecter ms_dispatch 0x56034eabc7b0 pool_op_reply(tid 4 (0) Success v17) 2024-09-16T03:42:26.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:26.601+0000 7f6bceffd640 10 client.4224.objecter handle_pool_op_reply pool_op_reply(tid 4 (0) Success v17) 2024-09-16T03:42:26.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:26.601+0000 7f6bceffd640 10 client.4224.objecter have request 4 at 0x7f6bc4003070 Op: create unmanaged snap 2024-09-16T03:42:26.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:26.601+0000 7f6bceffd640 20 client.4224.objecter waiting for client to reach epoch 17 before calling back 2024-09-16T03:42:26.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:26.601+0000 7f6bceffd640 10 client.4224.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:42:26.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:26.601+0000 7f6bceffd640 10 monclient: _renew_subs 2024-09-16T03:42:26.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:26.601+0000 7f6bceffd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:42:26.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:26.601+0000 7f6bceffd640 1 -- 172.21.15.42:0/3600569605 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=17}) -- 0x7f6bc803b7f0 con 0x7f6bd4003100 2024-09-16T03:42:26.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:26.601+0000 7f6bceffd640 10 client.4224.objecter done 2024-09-16T03:42:26.609 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:26.601+0000 7f6bceffd640 1 -- 172.21.15.42:0/3600569605 <== mon.0 v2:172.21.15.42:3300/0 10 ==== osd_map(17..17 src has 1..17) ==== 626+0+0 (secure 0 0 0) 0x7f6bdc016af0 con 0x7f6bd4003100 2024-09-16T03:42:26.609 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:26.601+0000 7f6bceffd640 10 client.4224.objecter ms_dispatch 0x56034eabc7b0 osd_map(17..17 src has 1..17) 2024-09-16T03:42:26.609 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:26.601+0000 7f6bceffd640 3 client.4224.objecter handle_osd_map got epochs [17,17] > 16 2024-09-16T03:42:26.609 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:26.601+0000 7f6bceffd640 3 client.4224.objecter handle_osd_map decoding incremental epoch 17 2024-09-16T03:42:26.609 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:26.601+0000 7f6bceffd640 20 client.4224.objecter dump_active .. 0 homeless 2024-09-16T03:42:26.609 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:26.601+0000 7f6be5070640 20 librados: queue_aio_write 0x56034ecd5b40 completion 0x56034ea65a30 write_seq 1 2024-09-16T03:42:26.609 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:26.601+0000 7f6be5070640 10 client.4224.objecter _op_submit op 0x7f6bc0006390 2024-09-16T03:42:26.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:26.601+0000 7f6be5070640 20 client.4224.objecter _calc_target epoch 17 base rbd_info @2 precalc_pgid 0 pgid 0.0 is_write 2024-09-16T03:42:26.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:26.601+0000 7f6be5070640 20 client.4224.objecter _calc_target target rbd_info @2 -> pgid 2.ac0e573a 2024-09-16T03:42:26.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:26.601+0000 7f6be5070640 10 client.4224.objecter _calc_target raw pgid 2.ac0e573a -> actual 2.2 acting [5,1] primary 5 2024-09-16T03:42:26.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:26.601+0000 7f6be5070640 20 client.4224.objecter _get_session s=0x56034ec552e0 osd=5 3 2024-09-16T03:42:26.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:26.601+0000 7f6be5070640 10 client.4224.objecter _op_submit oid rbd_info '@2' '@2' [create,write 0~8 in=8b] tid 5 osd.5 2024-09-16T03:42:26.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:26.601+0000 7f6be5070640 20 client.4224.objecter get_session s=0x56034ec552e0 osd=5 3 2024-09-16T03:42:26.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:26.601+0000 7f6be5070640 15 client.4224.objecter _session_op_assign 5 5 2024-09-16T03:42:26.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:26.601+0000 7f6be5070640 15 client.4224.objecter _send_op 5 to 2.2 on osd.5 2024-09-16T03:42:26.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:26.601+0000 7f6be5070640 1 -- 172.21.15.42:0/3600569605 --> v2:172.21.15.154:6808/3026127822 -- osd_op(unknown.0.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17) -- 0x7f6bc0006840 con 0x56034ec86140 2024-09-16T03:42:26.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:26.601+0000 7f6be5070640 20 client.4224.objecter put_session s=0x56034ec552e0 osd=5 4 2024-09-16T03:42:26.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:26.601+0000 7f6be5070640 5 client.4224.objecter 1 in flight 2024-09-16T03:42:26.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:26.609+0000 7f6be7ad5640 1 -- 172.21.15.42:0/3600569605 <== osd.5 v2:172.21.15.154:6808/3026127822 4 ==== osd_op_reply(5 rbd_info [create,write 0~8] v17'1 uv1 ondisk = 0) ==== 194+0+0 (crc 0 0 0) 0x7f6bd80106a0 con 0x56034ec86140 2024-09-16T03:42:26.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:26.609+0000 7f6be7ad5640 10 client.4224.objecter ms_dispatch 0x56034eabc7b0 osd_op_reply(5 rbd_info [create,write 0~8] v17'1 uv1 ondisk = 0) 2024-09-16T03:42:26.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:26.609+0000 7f6be7ad5640 10 client.4224.objecter in handle_osd_op_reply 2024-09-16T03:42:26.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:26.609+0000 7f6be7ad5640 7 client.4224.objecter handle_osd_op_reply 5 ondisk uv 1 in 2.2 attempt 0 2024-09-16T03:42:26.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:26.609+0000 7f6be7ad5640 10 client.4224.objecter op 0 rval 0 len 0 2024-09-16T03:42:26.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:26.609+0000 7f6be7ad5640 10 client.4224.objecter op 1 rval 0 len 0 2024-09-16T03:42:26.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:26.609+0000 7f6be7ad5640 15 client.4224.objecter handle_osd_op_reply completed tid 5 2024-09-16T03:42:26.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:26.609+0000 7f6be7ad5640 15 client.4224.objecter _finish_op 5 2024-09-16T03:42:26.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:26.609+0000 7f6be7ad5640 20 client.4224.objecter put_session s=0x56034ec552e0 osd=5 4 2024-09-16T03:42:26.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:26.609+0000 7f6be7ad5640 15 client.4224.objecter _session_op_remove 5 5 2024-09-16T03:42:26.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:26.609+0000 7f6be7ad5640 5 client.4224.objecter 0 in flight 2024-09-16T03:42:26.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:26.609+0000 7f6be7ad5640 20 librados: complete_aio_write 0x56034ea65a30 2024-09-16T03:42:26.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:26.609+0000 7f6be486f640 10 client.4224.objecter delete_selfmanaged_snap; pool: 2; snap: 2 2024-09-16T03:42:26.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:26.609+0000 7f6be486f640 10 client.4224.objecter pool_op_submit 6 2024-09-16T03:42:26.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:26.609+0000 7f6be486f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:42:26.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:26.609+0000 7f6be486f640 1 -- 172.21.15.42:0/3600569605 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- pool_op(delete unmanaged snap pool 2 tid 6 name v17) -- 0x7f6bc4004530 con 0x7f6bd4003100 2024-09-16T03:42:27.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.617+0000 7f6bceffd640 1 -- 172.21.15.42:0/3600569605 <== mon.0 v2:172.21.15.42:3300/0 11 ==== pool_op_reply(tid 6 (0) Success v18) ==== 43+0+0 (secure 0 0 0) 0x7f6bdc016cf0 con 0x7f6bd4003100 2024-09-16T03:42:27.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.617+0000 7f6bceffd640 10 client.4224.objecter ms_dispatch 0x56034eabc7b0 pool_op_reply(tid 6 (0) Success v18) 2024-09-16T03:42:27.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.617+0000 7f6bceffd640 10 client.4224.objecter handle_pool_op_reply pool_op_reply(tid 6 (0) Success v18) 2024-09-16T03:42:27.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.617+0000 7f6bceffd640 10 client.4224.objecter have request 6 at 0x7f6bc4003070 Op: delete unmanaged snap 2024-09-16T03:42:27.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.617+0000 7f6bceffd640 20 client.4224.objecter waiting for client to reach epoch 18 before calling back 2024-09-16T03:42:27.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.617+0000 7f6bceffd640 10 client.4224.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:42:27.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.617+0000 7f6bceffd640 10 monclient: _renew_subs 2024-09-16T03:42:27.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.617+0000 7f6bceffd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:42:27.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.617+0000 7f6bceffd640 1 -- 172.21.15.42:0/3600569605 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=18}) -- 0x7f6bc803ba70 con 0x7f6bd4003100 2024-09-16T03:42:27.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.617+0000 7f6bceffd640 10 client.4224.objecter done 2024-09-16T03:42:27.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.617+0000 7f6bceffd640 1 -- 172.21.15.42:0/3600569605 <== mon.0 v2:172.21.15.42:3300/0 12 ==== osd_map(18..18 src has 1..18) ==== 654+0+0 (secure 0 0 0) 0x7f6bdc02f070 con 0x7f6bd4003100 2024-09-16T03:42:27.625 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.617+0000 7f6bceffd640 10 client.4224.objecter ms_dispatch 0x56034eabc7b0 osd_map(18..18 src has 1..18) 2024-09-16T03:42:27.625 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.617+0000 7f6bceffd640 3 client.4224.objecter handle_osd_map got epochs [18,18] > 17 2024-09-16T03:42:27.625 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.617+0000 7f6bceffd640 3 client.4224.objecter handle_osd_map decoding incremental epoch 18 2024-09-16T03:42:27.625 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.617+0000 7f6bceffd640 20 client.4224.objecter dump_active .. 0 homeless 2024-09-16T03:42:27.625 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.617+0000 7f6be5070640 20 librados: queue_aio_write 0x56034ecd5b40 completion 0x7f6bc4001d70 write_seq 2 2024-09-16T03:42:27.625 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.617+0000 7f6be5070640 10 client.4224.objecter _op_submit op 0x7f6bc0006390 2024-09-16T03:42:27.625 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.617+0000 7f6be5070640 20 client.4224.objecter _calc_target epoch 18 base rbd_info @2 precalc_pgid 0 pgid 0.0 is_write 2024-09-16T03:42:27.625 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.617+0000 7f6be5070640 20 client.4224.objecter _calc_target target rbd_info @2 -> pgid 2.ac0e573a 2024-09-16T03:42:27.625 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.617+0000 7f6be5070640 10 client.4224.objecter _calc_target raw pgid 2.ac0e573a -> actual 2.2 acting [5,1] primary 5 2024-09-16T03:42:27.625 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.617+0000 7f6be5070640 20 client.4224.objecter _get_session s=0x56034ec552e0 osd=5 3 2024-09-16T03:42:27.625 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.617+0000 7f6be5070640 10 client.4224.objecter _op_submit oid rbd_info '@2' '@2' [write 0~19 in=19b] tid 7 osd.5 2024-09-16T03:42:27.625 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.617+0000 7f6be5070640 20 client.4224.objecter get_session s=0x56034ec552e0 osd=5 3 2024-09-16T03:42:27.625 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.617+0000 7f6be5070640 15 client.4224.objecter _session_op_assign 5 7 2024-09-16T03:42:27.625 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.617+0000 7f6be5070640 15 client.4224.objecter _send_op 7 to 2.2 on osd.5 2024-09-16T03:42:27.626 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.617+0000 7f6be5070640 1 -- 172.21.15.42:0/3600569605 --> v2:172.21.15.154:6808/3026127822 -- osd_op(unknown.0.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19 in=19b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18) -- 0x7f6bc0006be0 con 0x56034ec86140 2024-09-16T03:42:27.626 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.617+0000 7f6be5070640 20 client.4224.objecter put_session s=0x56034ec552e0 osd=5 4 2024-09-16T03:42:27.626 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.617+0000 7f6be5070640 5 client.4224.objecter 1 in flight 2024-09-16T03:42:27.631 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.625+0000 7f6be7ad5640 1 -- 172.21.15.42:0/3600569605 <== osd.5 v2:172.21.15.154:6808/3026127822 5 ==== osd_op_reply(7 rbd_info [write 0~19] v18'2 uv2 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f6bd80106a0 con 0x56034ec86140 2024-09-16T03:42:27.631 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.625+0000 7f6be7ad5640 10 client.4224.objecter ms_dispatch 0x56034eabc7b0 osd_op_reply(7 rbd_info [write 0~19] v18'2 uv2 ondisk = 0) 2024-09-16T03:42:27.631 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.625+0000 7f6be7ad5640 10 client.4224.objecter in handle_osd_op_reply 2024-09-16T03:42:27.631 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.625+0000 7f6be7ad5640 7 client.4224.objecter handle_osd_op_reply 7 ondisk uv 2 in 2.2 attempt 0 2024-09-16T03:42:27.631 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.625+0000 7f6be7ad5640 10 client.4224.objecter op 0 rval 0 len 0 2024-09-16T03:42:27.631 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.625+0000 7f6be7ad5640 15 client.4224.objecter handle_osd_op_reply completed tid 7 2024-09-16T03:42:27.631 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.625+0000 7f6be7ad5640 15 client.4224.objecter _finish_op 7 2024-09-16T03:42:27.632 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.625+0000 7f6be7ad5640 20 client.4224.objecter put_session s=0x56034ec552e0 osd=5 4 2024-09-16T03:42:27.632 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.625+0000 7f6be7ad5640 15 client.4224.objecter _session_op_remove 5 7 2024-09-16T03:42:27.632 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.625+0000 7f6be7ad5640 5 client.4224.objecter 0 in flight 2024-09-16T03:42:27.632 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.625+0000 7f6be7ad5640 20 librados: complete_aio_write 0x7f6bc4001d70 2024-09-16T03:42:27.633 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.625+0000 7f6be7d5ff80 10 librados: watch_flush enter 2024-09-16T03:42:27.633 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.625+0000 7f6be7d5ff80 10 librados: watch_flush exit 2024-09-16T03:42:27.633 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.625+0000 7f6be7d5ff80 10 client.4224.objecter close_session for osd.5 2024-09-16T03:42:27.633 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.625+0000 7f6be7d5ff80 1 -- 172.21.15.42:0/3600569605 >> v2:172.21.15.154:6808/3026127822 conn(0x56034ec86140 msgr2=0x56034ec88530 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:27.633 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.625+0000 7f6be7d5ff80 1 --2- 172.21.15.42:0/3600569605 >> v2:172.21.15.154:6808/3026127822 conn(0x56034ec86140 0x56034ec88530 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:27.633 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.625+0000 7f6be7d5ff80 20 client.4224.objecter put_session s=0x56034ec552e0 osd=5 1 2024-09-16T03:42:27.633 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.625+0000 7f6be7d5ff80 20 client.4224.objecter shutdown clearing up homeless session... 2024-09-16T03:42:27.633 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.625+0000 7f6be7d5ff80 10 client.4224.objecter successfully canceled tick 2024-09-16T03:42:27.634 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.625+0000 7f6be7d5ff80 1 -- 172.21.15.42:0/3600569605 >> v2:172.21.15.154:6812/254845342 conn(0x7f6bc8035940 msgr2=0x7f6bc8037e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:27.634 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.625+0000 7f6be7d5ff80 1 --2- 172.21.15.42:0/3600569605 >> v2:172.21.15.154:6812/254845342 conn(0x7f6bc8035940 0x7f6bc8037e00 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f6bd00ab000 tx=0x7f6bd00011a0 comp rx=0 tx=0).stop 2024-09-16T03:42:27.634 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.625+0000 7f6be7d5ff80 10 monclient: shutdown 2024-09-16T03:42:27.634 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.625+0000 7f6be7d5ff80 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:42:27.634 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.625+0000 7f6be7d5ff80 1 -- 172.21.15.42:0/3600569605 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6bd4003100 msgr2=0x56034ec7e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:27.634 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.625+0000 7f6be7d5ff80 1 --2- 172.21.15.42:0/3600569605 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6bd4003100 0x56034ec7e760 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f6bdc000fc0 tx=0x7f6bdc009f90 comp rx=0 tx=0).stop 2024-09-16T03:42:27.634 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.625+0000 7f6be7ad5640 1 -- 172.21.15.42:0/3600569605 reap_dead start 2024-09-16T03:42:27.634 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.625+0000 7f6be7d5ff80 1 -- 172.21.15.42:0/3600569605 shutdown_connections 2024-09-16T03:42:27.634 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.625+0000 7f6be7d5ff80 1 -- 172.21.15.42:0/3600569605 >> 172.21.15.42:0/3600569605 conn(0x56034ebdc7b0 msgr2=0x56034ec6ee20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:42:27.634 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.625+0000 7f6be7d5ff80 1 -- 172.21.15.42:0/3600569605 shutdown_connections 2024-09-16T03:42:27.634 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.625+0000 7f6be7d5ff80 1 -- 172.21.15.42:0/3600569605 wait complete. 2024-09-16T03:42:27.634 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.625+0000 7f6be7d5ff80 1 librados: shutdown 2024-09-16T03:42:27.639 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-09-16T03:42:27.639 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-16T03:42:27.640 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-09-16T03:42:27.640 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-09-16T03:42:27.640 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-16T03:42:27.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.765+0000 7f98ef2fd640 10 monclient: get_monmap_and_config 2024-09-16T03:42:27.773 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.765+0000 7f98ef2fd640 10 monclient: build_initial_monmap 2024-09-16T03:42:27.773 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.765+0000 7f98ef2fd640 10 monclient: monmap: 2024-09-16T03:42:27.773 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:42:27.773 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:27.773 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:42:27.770722+0000 2024-09-16T03:42:27.773 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:42:27.770722+0000 2024-09-16T03:42:27.773 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:42:27.773 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:27.773 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:42:27.773 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:42:27.773 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:42:27.773 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:42:27.774 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:42:27.774 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:27.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.765+0000 7f98ef2fd640 1 Processor -- start 2024-09-16T03:42:27.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.769+0000 7f98ef2fd640 1 -- start start 2024-09-16T03:42:27.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.769+0000 7f98ef2fd640 10 monclient: init 2024-09-16T03:42:27.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.769+0000 7f98ef2fd640 10 monclient: _reopen_session rank -1 2024-09-16T03:42:27.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.769+0000 7f98ef2fd640 10 monclient: _add_conns ranks=[2,1,0,3,4] 2024-09-16T03:42:27.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.769+0000 7f98ef2fd640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f98e8073080 0x7f98e8134ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:27.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.769+0000 7f98ef2fd640 10 monclient(hunting): picked mon.noname-b con 0x7f98e8073080 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:42:27.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.769+0000 7f98ef2fd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f98e8073460 addr v1:172.21.15.154:6789/0 2024-09-16T03:42:27.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.769+0000 7f98ef2fd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f98e8073860 addr v1:172.21.15.42:6789/0 2024-09-16T03:42:27.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.769+0000 7f98ef2fd640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f98e80a1780 con 0x7f98e8073860 2024-09-16T03:42:27.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.769+0000 7f98ef2fd640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f98e80a1cd0 con 0x7f98e8073460 2024-09-16T03:42:27.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.769+0000 7f98ef2fd640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:27.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.769+0000 7f98ef2fd640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f98e80a1a10 con 0x7f98e8073080 2024-09-16T03:42:27.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.769+0000 7f98ef2fd640 10 monclient(hunting): _renew_subs 2024-09-16T03:42:27.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.769+0000 7f98ef2fd640 10 monclient(hunting): authenticate will time out at 994.740774s 2024-09-16T03:42:27.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.769+0000 7f98ed099640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f98e8073080 0x7f98e8134ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:27.777 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.769+0000 7f98ed89a640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f98e8073860 0x7f98e80a1070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:46732/0 (socket says 172.21.15.42:46732) 2024-09-16T03:42:27.777 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.769+0000 7f98ed89a640 1 -- 172.21.15.42:0/2536579667 learned_addr learned my addr 172.21.15.42:0/2536579667 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:42:27.777 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.769+0000 7f98ed099640 10 monclient(hunting): get_auth_request con 0x7f98e8073080 auth_method 0 2024-09-16T03:42:27.777 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.769+0000 7f98ed099640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:27.777 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.769+0000 7f98ed099640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:27.777 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.769+0000 7f98ed099640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:27.777 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.769+0000 7f98ed099640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:27.777 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.769+0000 7f98ed099640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:27.777 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.769+0000 7f98ed099640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:27.777 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.769+0000 7f98ed099640 10 monclient(hunting): handle_auth_done global_id 4205 payload 274 2024-09-16T03:42:27.777 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.769+0000 7f98ed099640 1 -- 172.21.15.42:0/2536579667 >> v1:172.21.15.154:6789/0 conn(0x7f98e8073460 legacy=0x7f98e80a0960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:42:27.777 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.769+0000 7f98ed099640 1 -- 172.21.15.42:0/2536579667 >> v1:172.21.15.42:6789/0 conn(0x7f98e8073860 legacy=0x7f98e80a1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:27.777 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.769+0000 7f98dffff640 1 -- 172.21.15.42:0/2536579667 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 977508064 0 0) 0x7f98e80a1780 con 0x7f98e8073860 2024-09-16T03:42:27.777 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.769+0000 7f98ed099640 10 monclient: _finish_hunting 0 2024-09-16T03:42:27.778 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.769+0000 7f98ed099640 1 monclient: found mon.noname-b 2024-09-16T03:42:27.778 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.769+0000 7f98ed099640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:42:27.778 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.769+0000 7f98ed099640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T03:42:27.778 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.769+0000 7f98ed099640 1 -- 172.21.15.42:0/2536579667 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98e80a1fb0 con 0x7f98e8073080 2024-09-16T03:42:27.778 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.769+0000 7f98dffff640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T03:42:27.783 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.769+0000 7f98ed099640 1 --2- 172.21.15.42:0/2536579667 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f98e8073080 0x7f98e8134ee0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f98e40097f0 tx=0x7f98e401c3f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8957473c5cb0e569 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:27.783 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.769+0000 7f98dffff640 1 -- 172.21.15.42:0/2536579667 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98e40b0070 con 0x7f98e8073080 2024-09-16T03:42:27.783 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.769+0000 7f98dffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:27.783 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.769+0000 7f98dffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:42:27.783 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.769+0000 7f98dffff640 10 monclient: dump: 2024-09-16T03:42:27.784 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:27.784 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:27.784 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:27.784 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:27.784 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:27.784 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:27.784 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:27.784 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:27.784 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:27.785 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:27.785 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.769+0000 7f98dffff640 10 monclient: _finish_auth 0 2024-09-16T03:42:27.785 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.769+0000 7f98dffff640 10 monclient: _check_auth_tickets 2024-09-16T03:42:27.785 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.769+0000 7f98dffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:42:27.785 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.769+0000 7f98dffff640 1 -- 172.21.15.42:0/2536579667 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f98e40ab600 con 0x7f98e8073080 2024-09-16T03:42:27.785 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.769+0000 7f98dffff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:42:27.785 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.769+0000 7f98dffff640 1 -- 172.21.15.42:0/2536579667 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98e40ab9d0 con 0x7f98e8073080 2024-09-16T03:42:27.785 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.769+0000 7f98ef2fd640 5 monclient: authenticate success, global_id 4205 2024-09-16T03:42:27.785 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.769+0000 7f98ef2fd640 10 monclient: get_monmap_and_config success 2024-09-16T03:42:27.785 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.769+0000 7f98ef2fd640 4 set_mon_vals no callback set 2024-09-16T03:42:27.785 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.769+0000 7f98dffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:27.785 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.769+0000 7f98dffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:42:27.785 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.769+0000 7f98dffff640 10 monclient: dump: 2024-09-16T03:42:27.785 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:27.786 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:27.786 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:27.786 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:27.786 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:27.786 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:27.786 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:27.786 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:27.786 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:27.786 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:27.786 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.773+0000 7f98ef2fd640 10 monclient: shutdown 2024-09-16T03:42:27.786 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.773+0000 7f98ef2fd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:42:27.786 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.773+0000 7f98ef2fd640 1 -- 172.21.15.42:0/2536579667 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f98e8073080 msgr2=0x7f98e8134ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:27.786 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.773+0000 7f98ef2fd640 1 --2- 172.21.15.42:0/2536579667 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f98e8073080 0x7f98e8134ee0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f98e40097f0 tx=0x7f98e401c3f0 comp rx=0 tx=0).stop 2024-09-16T03:42:27.787 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.773+0000 7f98ef2fd640 1 -- 172.21.15.42:0/2536579667 shutdown_connections 2024-09-16T03:42:27.787 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.773+0000 7f98ef2fd640 1 --2- 172.21.15.42:0/2536579667 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f98e8073080 0x7f98e8134ee0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:27.787 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.773+0000 7f98ef2fd640 1 -- 172.21.15.42:0/2536579667 >> 172.21.15.42:0/2536579667 conn(0x7f98e806bc40 msgr2=0x7f98e806c050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:42:27.787 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.773+0000 7f98ef2fd640 1 -- 172.21.15.42:0/2536579667 shutdown_connections 2024-09-16T03:42:27.787 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.773+0000 7f98ef2fd640 1 -- 172.21.15.42:0/2536579667 wait complete. 2024-09-16T03:42:27.787 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.773+0000 7f98ef2fd640 10 monclient: build_initial_monmap 2024-09-16T03:42:27.787 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.773+0000 7f98ef2fd640 10 monclient: monmap: 2024-09-16T03:42:27.787 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:42:27.787 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:42:27.787 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:42:27.787 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:42:27.787 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:42:27.787 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:27.787 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:42:27.788 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:42:27.788 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:42:27.788 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:27.788 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.773+0000 7f98ef2fd640 1 librados: starting msgr at 2024-09-16T03:42:27.788 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.773+0000 7f98ef2fd640 1 librados: starting objecter 2024-09-16T03:42:27.788 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.773+0000 7f98ef2fd640 1 Processor -- start 2024-09-16T03:42:27.788 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.773+0000 7f98ef2fd640 1 -- start start 2024-09-16T03:42:27.788 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.773+0000 7f98ef2fd640 1 librados: setting wanted keys 2024-09-16T03:42:27.788 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.773+0000 7f98ef2fd640 1 librados: calling monclient init 2024-09-16T03:42:27.788 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.773+0000 7f98ef2fd640 10 monclient: init 2024-09-16T03:42:27.788 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.773+0000 7f98ef2fd640 10 monclient: _reopen_session rank -1 2024-09-16T03:42:27.788 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.773+0000 7f98ef2fd640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:42:27.788 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.773+0000 7f98ef2fd640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f98e8073080 0x7f98e8135e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:27.789 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.773+0000 7f98ef2fd640 10 monclient(hunting): picked mon.noname-a con 0x7f98e8073080 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:42:27.789 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.773+0000 7f98ef2fd640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f98e8073460 0x7f98e81363a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:27.789 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.773+0000 7f98ef2fd640 10 monclient(hunting): picked mon.noname-c con 0x7f98e8073460 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:42:27.789 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.773+0000 7f98ef2fd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f98e8073860 0x7f98e81368e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:27.789 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.773+0000 7f98ef2fd640 10 monclient(hunting): picked mon.noname-b con 0x7f98e8073860 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:42:27.789 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.773+0000 7f98ef2fd640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:27.789 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.773+0000 7f98ef2fd640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f98e8140b60 con 0x7f98e8073080 2024-09-16T03:42:27.789 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.773+0000 7f98ef2fd640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:27.789 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.773+0000 7f98ef2fd640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f98e8140ce0 con 0x7f98e8073860 2024-09-16T03:42:27.789 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.773+0000 7f98ef2fd640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:27.789 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.773+0000 7f98ef2fd640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f98e8140e60 con 0x7f98e8073460 2024-09-16T03:42:27.789 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.773+0000 7f98ef2fd640 10 monclient(hunting): _renew_subs 2024-09-16T03:42:27.789 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.773+0000 7f98ef2fd640 10 monclient(hunting): authenticate will time out at 994.745704s 2024-09-16T03:42:27.789 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.773+0000 7f98ec898640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f98e8073460 0x7f98e81363a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:27.790 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.773+0000 7f98ed099640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f98e8073080 0x7f98e8135e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:27.790 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.773+0000 7f98ed099640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f98e8073080 0x7f98e8135e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:37524/0 (socket says 172.21.15.42:37524) 2024-09-16T03:42:27.790 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.773+0000 7f98ed099640 1 -- 172.21.15.42:0/622961656 learned_addr learned my addr 172.21.15.42:0/622961656 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:42:27.790 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.773+0000 7f98ed099640 10 monclient(hunting): get_auth_request con 0x7f98e8073080 auth_method 0 2024-09-16T03:42:27.790 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.773+0000 7f98ed099640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:27.790 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.773+0000 7f98ed099640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:27.790 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.773+0000 7f98ed099640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:27.790 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.773+0000 7f98ec898640 10 monclient(hunting): get_auth_request con 0x7f98e8073460 auth_method 0 2024-09-16T03:42:27.790 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.773+0000 7f98ec898640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:27.790 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.773+0000 7f98ec898640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:27.790 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.773+0000 7f98ec898640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:27.790 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.773+0000 7f98ed099640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:27.790 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.773+0000 7f98ed099640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:27.791 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.773+0000 7f98ed099640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:27.791 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.773+0000 7f98ec898640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:27.791 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.773+0000 7f98ec898640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:27.791 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.773+0000 7f98ec898640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:27.791 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.773+0000 7f98ed89a640 1 --2- 172.21.15.42:0/622961656 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f98e8073860 0x7f98e81368e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:27.791 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.773+0000 7f98ed89a640 10 monclient(hunting): get_auth_request con 0x7f98e8073860 auth_method 0 2024-09-16T03:42:27.791 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.773+0000 7f98ed89a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:27.791 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.773+0000 7f98ed89a640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:27.791 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.773+0000 7f98ed89a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:27.791 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.773+0000 7f98ec898640 10 monclient(hunting): handle_auth_done global_id 4208 payload 824 2024-09-16T03:42:27.791 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.773+0000 7f98ec898640 1 -- 172.21.15.42:0/622961656 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f98e8073860 msgr2=0x7f98e81368e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:27.791 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.773+0000 7f98ec898640 1 --2- 172.21.15.42:0/622961656 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f98e8073860 0x7f98e81368e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:27.791 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.773+0000 7f98ec898640 1 -- 172.21.15.42:0/622961656 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f98e8073080 msgr2=0x7f98e8135e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:27.792 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.773+0000 7f98ec898640 1 --2- 172.21.15.42:0/622961656 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f98e8073080 0x7f98e8135e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:27.792 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.773+0000 7f98ec898640 10 monclient: _finish_hunting 0 2024-09-16T03:42:27.792 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.773+0000 7f98ec898640 1 monclient: found mon.noname-c 2024-09-16T03:42:27.792 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.773+0000 7f98ec898640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:42:27.792 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.773+0000 7f98ec898640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:42:27.792 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.773+0000 7f98ec898640 1 -- 172.21.15.42:0/622961656 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98e8139420 con 0x7f98e8073460 2024-09-16T03:42:27.792 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.773+0000 7f98ed099640 1 --2- 172.21.15.42:0/622961656 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f98e8073080 0x7f98e8135e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:42:27.792 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.773+0000 7f98ec898640 1 --2- 172.21.15.42:0/622961656 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f98e8073460 0x7f98e81363a0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f98e809d070 tx=0x7f98d800c870 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:27.792 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.773+0000 7f98ddffb640 10 client.?.objecter ms_handle_connect 0x7f98e8073460 2024-09-16T03:42:27.792 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.773+0000 7f98ddffb640 10 client.?.objecter resend_mon_ops 2024-09-16T03:42:27.792 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.773+0000 7f98ddffb640 1 -- 172.21.15.42:0/622961656 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98d8011790 con 0x7f98e8073460 2024-09-16T03:42:27.792 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.773+0000 7f98ddffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:27.792 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.773+0000 7f98ddffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:42:27.792 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.773+0000 7f98ddffb640 10 monclient: dump: 2024-09-16T03:42:27.793 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:27.793 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:27.793 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:27.793 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:27.793 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:27.793 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:27.793 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:27.793 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:27.793 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:27.793 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:27.793 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.773+0000 7f98ddffb640 10 monclient: _finish_auth 0 2024-09-16T03:42:27.793 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.773+0000 7f98ddffb640 10 monclient: _check_auth_tickets 2024-09-16T03:42:27.793 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.773+0000 7f98ddffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:42:27.793 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.773+0000 7f98ddffb640 1 -- 172.21.15.42:0/622961656 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f98d8017070 con 0x7f98e8073460 2024-09-16T03:42:27.794 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.773+0000 7f98ef2fd640 5 monclient: authenticate success, global_id 4208 2024-09-16T03:42:27.794 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.773+0000 7f98ef2fd640 10 monclient: _renew_subs 2024-09-16T03:42:27.794 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.773+0000 7f98ef2fd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:42:27.794 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.773+0000 7f98ef2fd640 1 -- 172.21.15.42:0/622961656 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98e80a5ca0 con 0x7f98e8073460 2024-09-16T03:42:27.794 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.773+0000 7f98ddffb640 10 monclient: handle_config config(0 keys) 2024-09-16T03:42:27.794 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.773+0000 7f98ddffb640 1 -- 172.21.15.42:0/622961656 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98d8011bb0 con 0x7f98e8073460 2024-09-16T03:42:27.794 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.773+0000 7f98ddffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:27.794 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.773+0000 7f98ddffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:42:27.794 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.773+0000 7f98ddffb640 10 monclient: dump: 2024-09-16T03:42:27.794 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:27.794 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:27.794 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:27.794 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:27.794 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:27.795 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:27.795 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:27.795 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:27.795 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:27.795 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:27.795 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.773+0000 7f98deffd640 4 set_mon_vals no callback set 2024-09-16T03:42:27.795 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.773+0000 7f98ef2fd640 10 client.4208.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:42:27.795 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.773+0000 7f98ef2fd640 10 monclient: _renew_subs 2024-09-16T03:42:27.795 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.773+0000 7f98ef2fd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:42:27.795 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.773+0000 7f98ef2fd640 1 -- 172.21.15.42:0/622961656 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f98e8139140 con 0x7f98e8073460 2024-09-16T03:42:27.795 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.773+0000 7f98ef2fd640 1 librados: init done 2024-09-16T03:42:27.795 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.773+0000 7f98ef2fd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:42:27.795 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.773+0000 7f98ef2fd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:42:27.795 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.773+0000 7f98ef2fd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:42:27.796 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.773+0000 7f98ef2fd640 1 -- 172.21.15.42:0/622961656 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f98e8134ee0 con 0x7f98e8073460 2024-09-16T03:42:27.796 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.773+0000 7f98ddffb640 1 -- 172.21.15.42:0/622961656 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f98d801c460 con 0x7f98e8073460 2024-09-16T03:42:27.796 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.773+0000 7f98ddffb640 1 --2- 172.21.15.42:0/622961656 >> v2:172.21.15.154:6812/254845342 conn(0x7f98cc037810 0x7f98cc039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:27.796 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.773+0000 7f98ddffb640 1 -- 172.21.15.42:0/622961656 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(18..18 src has 1..18) ==== 3562+0+0 (secure 0 0 0) 0x7f98d80484a0 con 0x7f98e8073460 2024-09-16T03:42:27.796 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.773+0000 7f98ddffb640 10 client.4208.objecter ms_dispatch 0x7f98e8000ba0 osd_map(18..18 src has 1..18) 2024-09-16T03:42:27.796 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.773+0000 7f98ddffb640 3 client.4208.objecter handle_osd_map got epochs [18,18] > 0 2024-09-16T03:42:27.796 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.773+0000 7f98ddffb640 3 client.4208.objecter handle_osd_map decoding full epoch 18 2024-09-16T03:42:27.796 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.777+0000 7f98ddffb640 20 client.4208.objecter dump_active .. 0 homeless 2024-09-16T03:42:27.796 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.777+0000 7f98ed099640 1 --2- 172.21.15.42:0/622961656 >> v2:172.21.15.154:6812/254845342 conn(0x7f98cc037810 0x7f98cc039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:27.796 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.777+0000 7f98ed099640 10 monclient: get_auth_request con 0x7f98cc037810 auth_method 0 2024-09-16T03:42:27.796 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.777+0000 7f98ddffb640 10 client.4208.objecter ms_handle_connect 0x7f98cc037810 2024-09-16T03:42:27.796 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.777+0000 7f98ed099640 1 --2- 172.21.15.42:0/622961656 >> v2:172.21.15.154:6812/254845342 conn(0x7f98cc037810 0x7f98cc039cd0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f98e401aaa0 tx=0x7f98e40a4f90 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:27.797 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.781+0000 7f98ddffb640 1 -- 172.21.15.42:0/622961656 <== mon.2 v2:172.21.15.42:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f98d801a760 con 0x7f98e8073460 2024-09-16T03:42:27.797 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.781+0000 7f98ddffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:42:27.797 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.781+0000 7f98ddffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:42:27.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.985+0000 7f98ef2fd640 10 start_mon_command cmd=[{"prefix": "mgr dump", "format": "json"}] 2024-09-16T03:42:27.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.985+0000 7f98ef2fd640 10 monclient: _send_command 2 [{"prefix": "mgr dump", "format": "json"}] 2024-09-16T03:42:27.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.985+0000 7f98ef2fd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:42:27.992 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.985+0000 7f98ef2fd640 1 -- 172.21.15.42:0/622961656 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7f98e809d150 con 0x7f98e8073460 2024-09-16T03:42:27.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.989+0000 7f98ddffb640 1 -- 172.21.15.42:0/622961656 <== mon.2 v2:172.21.15.42:3301/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) ==== 74+0+84117 (secure 0 0 0) 0x7f98d8012420 con 0x7f98e8073460 2024-09-16T03:42:27.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.989+0000 7f98ddffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mgr dump", "format": "json"}] 2024-09-16T03:42:27.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.989+0000 7f98ddffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:42:27.997 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-09-16T03:42:27.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.989+0000 7f98ef2fd640 10 librados: watch_flush enter 2024-09-16T03:42:27.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.989+0000 7f98ef2fd640 10 librados: watch_flush exit 2024-09-16T03:42:27.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.989+0000 7f98ef2fd640 20 client.4208.objecter shutdown clearing up homeless session... 2024-09-16T03:42:27.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.989+0000 7f98ef2fd640 10 client.4208.objecter successfully canceled tick 2024-09-16T03:42:27.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.993+0000 7f98ef2fd640 1 -- 172.21.15.42:0/622961656 >> v2:172.21.15.154:6812/254845342 conn(0x7f98cc037810 msgr2=0x7f98cc039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:27.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.993+0000 7f98ef2fd640 1 --2- 172.21.15.42:0/622961656 >> v2:172.21.15.154:6812/254845342 conn(0x7f98cc037810 0x7f98cc039cd0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f98e401aaa0 tx=0x7f98e40a4f90 comp rx=0 tx=0).stop 2024-09-16T03:42:27.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.993+0000 7f98ef2fd640 10 monclient: shutdown 2024-09-16T03:42:27.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.993+0000 7f98ef2fd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:42:27.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.993+0000 7f98ef2fd640 1 -- 172.21.15.42:0/622961656 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f98e8073460 msgr2=0x7f98e81363a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:27.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.993+0000 7f98ef2fd640 1 --2- 172.21.15.42:0/622961656 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f98e8073460 0x7f98e81363a0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f98e809d070 tx=0x7f98d800c870 comp rx=0 tx=0).stop 2024-09-16T03:42:27.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.993+0000 7f98ef2fd640 1 -- 172.21.15.42:0/622961656 shutdown_connections 2024-09-16T03:42:27.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.993+0000 7f98ef2fd640 1 --2- 172.21.15.42:0/622961656 >> v2:172.21.15.154:6812/254845342 conn(0x7f98cc037810 0x7f98cc039cd0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:27.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.993+0000 7f98ef2fd640 1 --2- 172.21.15.42:0/622961656 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f98e8073860 0x7f98e81368e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:27.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.993+0000 7f98ef2fd640 1 --2- 172.21.15.42:0/622961656 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f98e8073460 0x7f98e81363a0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:27.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.993+0000 7f98ef2fd640 1 --2- 172.21.15.42:0/622961656 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f98e8073080 0x7f98e8135e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:27.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.993+0000 7f98ef2fd640 1 -- 172.21.15.42:0/622961656 >> 172.21.15.42:0/622961656 conn(0x7f98e806bc40 msgr2=0x7f98e80a3e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:42:28.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.993+0000 7f98ef2fd640 1 -- 172.21.15.42:0/622961656 shutdown_connections 2024-09-16T03:42:28.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.993+0000 7f98ef2fd640 1 -- 172.21.15.42:0/622961656 wait complete. 2024-09-16T03:42:28.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:27.993+0000 7f98ef2fd640 1 librados: shutdown 2024-09-16T03:42:28.014 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":5,"flags":0,"active_gid":4102,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":254845342}]},"active_addr":"172.21.15.154:6812/254845342","active_change":"2024-09-16T03:42:11.455906+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.154:0","nonce":1950402610}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.154:0","nonce":677043324}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.154:0","nonce":2364945363}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.154:0","nonce":1016600757}]}]} 2024-09-16T03:42:28.017 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-09-16T03:42:28.017 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-09-16T03:42:28.017 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T03:42:28.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.125+0000 7f53f7adc640 10 monclient: get_monmap_and_config 2024-09-16T03:42:28.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.125+0000 7f53f7adc640 10 monclient: build_initial_monmap 2024-09-16T03:42:28.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.125+0000 7f53f7adc640 10 monclient: monmap: 2024-09-16T03:42:28.134 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:42:28.134 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:28.134 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:42:28.131842+0000 2024-09-16T03:42:28.134 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:42:28.131842+0000 2024-09-16T03:42:28.134 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:42:28.134 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:28.134 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:42:28.134 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:42:28.134 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:42:28.135 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:42:28.135 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:42:28.135 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:28.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.129+0000 7f53f7adc640 1 Processor -- start 2024-09-16T03:42:28.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.129+0000 7f53f7adc640 1 -- start start 2024-09-16T03:42:28.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.129+0000 7f53f7adc640 10 monclient: init 2024-09-16T03:42:28.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.129+0000 7f53f7adc640 10 monclient: _reopen_session rank -1 2024-09-16T03:42:28.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.129+0000 7f53f7adc640 10 monclient: _add_conns ranks=[4,0,3,2,1] 2024-09-16T03:42:28.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.129+0000 7f53f7adc640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f53f006db90 0x7f53f01340c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:28.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.129+0000 7f53f7adc640 10 monclient(hunting): picked mon.noname-c con 0x7f53f006db90 addr v2:172.21.15.154:3300/0 2024-09-16T03:42:28.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.129+0000 7f53f7adc640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f53f00a37f0 addr v1:172.21.15.42:6789/0 2024-09-16T03:42:28.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.129+0000 7f53f7adc640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f53f00a3c00 0x7f53f0137520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:28.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.129+0000 7f53f7adc640 10 monclient(hunting): picked mon.noname-a con 0x7f53f00a3c00 addr v2:172.21.15.42:3300/0 2024-09-16T03:42:28.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.129+0000 7f53f7adc640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53f006ece0 con 0x7f53f00a37f0 2024-09-16T03:42:28.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.129+0000 7f53f7adc640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:28.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.129+0000 7f53f7adc640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f53f006df70 con 0x7f53f00a3c00 2024-09-16T03:42:28.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.129+0000 7f53f7adc640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:28.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.129+0000 7f53f7adc640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f53f009bf80 con 0x7f53f006db90 2024-09-16T03:42:28.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.129+0000 7f53f7adc640 10 monclient(hunting): _renew_subs 2024-09-16T03:42:28.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.129+0000 7f53f7adc640 10 monclient(hunting): authenticate will time out at 995.101318s 2024-09-16T03:42:28.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.129+0000 7f53f6079640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f53f00a3c00 0x7f53f0137520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:28.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.129+0000 7f53f5077640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f53f00a37f0 0x7f53f0136e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:46744/0 (socket says 172.21.15.42:46744) 2024-09-16T03:42:28.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.129+0000 7f53f5077640 1 -- 172.21.15.42:0/3561316484 learned_addr learned my addr 172.21.15.42:0/3561316484 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:42:28.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.129+0000 7f53f6079640 10 monclient(hunting): get_auth_request con 0x7f53f00a3c00 auth_method 0 2024-09-16T03:42:28.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.129+0000 7f53f6079640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:28.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.129+0000 7f53f6079640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:28.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.129+0000 7f53f6079640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:28.140 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.129+0000 7f53f6079640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:28.140 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.129+0000 7f53f6079640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:28.140 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.129+0000 7f53f6079640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:28.140 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.129+0000 7f53f5878640 1 --2- 172.21.15.42:0/3561316484 >> v2:172.21.15.154:3300/0 conn(0x7f53f006db90 0x7f53f01340c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:28.140 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.129+0000 7f53f5878640 10 monclient(hunting): get_auth_request con 0x7f53f006db90 auth_method 0 2024-09-16T03:42:28.140 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.129+0000 7f53f5878640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:28.140 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.129+0000 7f53f5878640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:28.140 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.129+0000 7f53f5878640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:28.140 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.129+0000 7f53f6079640 10 monclient(hunting): handle_auth_done global_id 4233 payload 274 2024-09-16T03:42:28.140 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.129+0000 7f53f6079640 1 -- 172.21.15.42:0/3561316484 >> v2:172.21.15.154:3300/0 conn(0x7f53f006db90 msgr2=0x7f53f01340c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:42:28.140 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.129+0000 7f53f6079640 1 --2- 172.21.15.42:0/3561316484 >> v2:172.21.15.154:3300/0 conn(0x7f53f006db90 0x7f53f01340c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:28.140 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.129+0000 7f53f6079640 1 -- 172.21.15.42:0/3561316484 >> v1:172.21.15.42:6789/0 conn(0x7f53f00a37f0 legacy=0x7f53f0136e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:28.140 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.129+0000 7f53f4876640 1 -- 172.21.15.42:0/3561316484 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1990967850 0 0) 0x7f53f006ece0 con 0x7f53f00a37f0 2024-09-16T03:42:28.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.129+0000 7f53f6079640 10 monclient: _finish_hunting 0 2024-09-16T03:42:28.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.129+0000 7f53f6079640 1 monclient: found mon.noname-a 2024-09-16T03:42:28.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.129+0000 7f53f6079640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:42:28.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.129+0000 7f53f6079640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:42:28.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.129+0000 7f53f6079640 1 -- 172.21.15.42:0/3561316484 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53f0134780 con 0x7f53f00a3c00 2024-09-16T03:42:28.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.129+0000 7f53f4876640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T03:42:28.145 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.133+0000 7f53f6079640 1 --2- 172.21.15.42:0/3561316484 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f53f00a3c00 0x7f53f0137520 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f53f00a76f0 tx=0x7f53e801d220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=535d589add5449ea server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:28.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.133+0000 7f53f4876640 1 -- 172.21.15.42:0/3561316484 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53e80b2070 con 0x7f53f00a3c00 2024-09-16T03:42:28.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.133+0000 7f53f4876640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:28.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.133+0000 7f53f4876640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:42:28.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.133+0000 7f53f4876640 10 monclient: dump: 2024-09-16T03:42:28.146 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:28.146 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:28.146 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:28.146 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:28.146 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:28.146 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:28.146 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:28.146 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:28.146 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:28.146 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:28.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.133+0000 7f53f4876640 10 monclient: _finish_auth 0 2024-09-16T03:42:28.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.133+0000 7f53f4876640 10 monclient: _check_auth_tickets 2024-09-16T03:42:28.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.133+0000 7f53f4876640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:42:28.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.133+0000 7f53f4876640 1 -- 172.21.15.42:0/3561316484 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f53e80a58a0 con 0x7f53f00a3c00 2024-09-16T03:42:28.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.133+0000 7f53f7adc640 5 monclient: authenticate success, global_id 4233 2024-09-16T03:42:28.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.133+0000 7f53f7adc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:42:28.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.133+0000 7f53f4876640 10 monclient: handle_config config(0 keys) 2024-09-16T03:42:28.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.133+0000 7f53f7adc640 10 monclient: get_monmap_and_config success 2024-09-16T03:42:28.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.133+0000 7f53f7adc640 4 set_mon_vals no callback set 2024-09-16T03:42:28.148 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.133+0000 7f53f4876640 1 -- 172.21.15.42:0/3561316484 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53e80a5c70 con 0x7f53f00a3c00 2024-09-16T03:42:28.148 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.133+0000 7f53f4876640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:28.148 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.133+0000 7f53f4876640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:42:28.148 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.133+0000 7f53f4876640 10 monclient: dump: 2024-09-16T03:42:28.149 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:28.149 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:28.149 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:28.149 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:28.149 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:28.149 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:28.150 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:28.150 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:28.150 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:28.150 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:28.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.133+0000 7f53f7adc640 10 monclient: shutdown 2024-09-16T03:42:28.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.133+0000 7f53f7adc640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:42:28.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.133+0000 7f53f7adc640 1 -- 172.21.15.42:0/3561316484 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f53f00a3c00 msgr2=0x7f53f0137520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:28.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.133+0000 7f53f7adc640 1 --2- 172.21.15.42:0/3561316484 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f53f00a3c00 0x7f53f0137520 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f53f00a76f0 tx=0x7f53e801d220 comp rx=0 tx=0).stop 2024-09-16T03:42:28.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.133+0000 7f53f7adc640 1 -- 172.21.15.42:0/3561316484 shutdown_connections 2024-09-16T03:42:28.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.133+0000 7f53f7adc640 1 --2- 172.21.15.42:0/3561316484 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f53f00a3c00 0x7f53f0137520 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:28.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.133+0000 7f53f7adc640 1 --2- 172.21.15.42:0/3561316484 >> v2:172.21.15.154:3300/0 conn(0x7f53f006db90 0x7f53f01340c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:28.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.133+0000 7f53f7adc640 1 -- 172.21.15.42:0/3561316484 >> 172.21.15.42:0/3561316484 conn(0x7f53f00a9fb0 msgr2=0x7f53f00aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:42:28.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.133+0000 7f53f7adc640 1 -- 172.21.15.42:0/3561316484 shutdown_connections 2024-09-16T03:42:28.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.133+0000 7f53f7adc640 1 -- 172.21.15.42:0/3561316484 wait complete. 2024-09-16T03:42:28.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.133+0000 7f53f7adc640 10 monclient: build_initial_monmap 2024-09-16T03:42:28.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.133+0000 7f53f7adc640 10 monclient: monmap: 2024-09-16T03:42:28.152 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:42:28.152 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:42:28.153 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:42:28.153 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:42:28.153 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:42:28.153 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:28.153 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:42:28.154 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:42:28.154 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:42:28.154 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:28.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.133+0000 7f53f7adc640 1 librados: starting msgr at 2024-09-16T03:42:28.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.133+0000 7f53f7adc640 1 librados: starting objecter 2024-09-16T03:42:28.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.133+0000 7f53f7adc640 1 Processor -- start 2024-09-16T03:42:28.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.133+0000 7f53f7adc640 1 -- start start 2024-09-16T03:42:28.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.133+0000 7f53f7adc640 1 librados: setting wanted keys 2024-09-16T03:42:28.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.133+0000 7f53f7adc640 1 librados: calling monclient init 2024-09-16T03:42:28.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.133+0000 7f53f7adc640 10 monclient: init 2024-09-16T03:42:28.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.133+0000 7f53f7adc640 10 monclient: _reopen_session rank -1 2024-09-16T03:42:28.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.133+0000 7f53f7adc640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:42:28.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.133+0000 7f53f7adc640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f53f006db90 0x7f53f01cec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:28.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.133+0000 7f53f7adc640 10 monclient(hunting): picked mon.noname-a con 0x7f53f006db90 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:42:28.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.133+0000 7f53f7adc640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f53f00a37f0 0x7f53f01cf1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:28.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.133+0000 7f53f7adc640 10 monclient(hunting): picked mon.noname-c con 0x7f53f00a37f0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:42:28.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.133+0000 7f53f7adc640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f53f00a3c00 0x7f53f01cf6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:28.157 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.133+0000 7f53f7adc640 10 monclient(hunting): picked mon.noname-b con 0x7f53f00a3c00 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:42:28.157 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.133+0000 7f53f7adc640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:28.157 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.133+0000 7f53f7adc640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f53f00a40d0 con 0x7f53f006db90 2024-09-16T03:42:28.157 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.133+0000 7f53f7adc640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:28.157 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.133+0000 7f53f7adc640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f53f0137c70 con 0x7f53f00a3c00 2024-09-16T03:42:28.157 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.133+0000 7f53f7adc640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:28.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.133+0000 7f53f7adc640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f53f01419d0 con 0x7f53f00a37f0 2024-09-16T03:42:28.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.133+0000 7f53f7adc640 10 monclient(hunting): _renew_subs 2024-09-16T03:42:28.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.133+0000 7f53f7adc640 10 monclient(hunting): authenticate will time out at 995.107234s 2024-09-16T03:42:28.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.133+0000 7f53f5878640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f53f006db90 0x7f53f01cec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:28.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.133+0000 7f53f5878640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f53f006db90 0x7f53f01cec60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:37540/0 (socket says 172.21.15.42:37540) 2024-09-16T03:42:28.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.133+0000 7f53f5878640 1 -- 172.21.15.42:0/4289003876 learned_addr learned my addr 172.21.15.42:0/4289003876 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:42:28.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.133+0000 7f53f5878640 10 monclient(hunting): get_auth_request con 0x7f53f006db90 auth_method 0 2024-09-16T03:42:28.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.133+0000 7f53f5878640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:28.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.133+0000 7f53f5878640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:28.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.133+0000 7f53f5077640 1 --2- 172.21.15.42:0/4289003876 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f53f00a37f0 0x7f53f01cf1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:28.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.133+0000 7f53f5878640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:28.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.133+0000 7f53f5077640 10 monclient(hunting): get_auth_request con 0x7f53f00a37f0 auth_method 0 2024-09-16T03:42:28.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.133+0000 7f53f5077640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:28.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.133+0000 7f53f5077640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:28.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.133+0000 7f53f5077640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:28.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.133+0000 7f53f5878640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:28.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.133+0000 7f53f5878640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:28.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.133+0000 7f53f5878640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:28.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.133+0000 7f53f5077640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:28.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.133+0000 7f53f5077640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:28.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.133+0000 7f53f5077640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:28.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.133+0000 7f53f6079640 1 --2- 172.21.15.42:0/4289003876 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f53f00a3c00 0x7f53f01cf6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:28.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.133+0000 7f53f5878640 10 monclient(hunting): handle_auth_done global_id 4239 payload 824 2024-09-16T03:42:28.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.133+0000 7f53f5878640 1 -- 172.21.15.42:0/4289003876 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f53f00a37f0 msgr2=0x7f53f01cf1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:28.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.133+0000 7f53f5878640 1 --2- 172.21.15.42:0/4289003876 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f53f00a37f0 0x7f53f01cf1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:28.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.133+0000 7f53f5878640 1 -- 172.21.15.42:0/4289003876 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f53f00a3c00 msgr2=0x7f53f01cf6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:28.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.133+0000 7f53f5878640 1 --2- 172.21.15.42:0/4289003876 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f53f00a3c00 0x7f53f01cf6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:28.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.133+0000 7f53f5878640 10 monclient: _finish_hunting 0 2024-09-16T03:42:28.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.133+0000 7f53f5878640 1 monclient: found mon.noname-a 2024-09-16T03:42:28.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.133+0000 7f53f5878640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:42:28.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.133+0000 7f53f5878640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:42:28.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.133+0000 7f53f5878640 1 -- 172.21.15.42:0/4289003876 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53f00a21f0 con 0x7f53f006db90 2024-09-16T03:42:28.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.133+0000 7f53f6079640 10 monclient: get_auth_request con 0x7f53f00a3c00 auth_method 0 2024-09-16T03:42:28.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.133+0000 7f53f6079640 1 --2- 172.21.15.42:0/4289003876 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f53f00a3c00 0x7f53f01cf6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T03:42:28.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.133+0000 7f53f5878640 1 --2- 172.21.15.42:0/4289003876 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f53f006db90 0x7f53f01cec60 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f53ec00d5f0 tx=0x7f53ec00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:28.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.133+0000 7f53e67fc640 10 client.?.objecter ms_handle_connect 0x7f53f006db90 2024-09-16T03:42:28.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.133+0000 7f53e67fc640 10 client.?.objecter resend_mon_ops 2024-09-16T03:42:28.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.137+0000 7f53e67fc640 1 -- 172.21.15.42:0/4289003876 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53ec0125f0 con 0x7f53f006db90 2024-09-16T03:42:28.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.137+0000 7f53e67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:28.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.137+0000 7f53e67fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:42:28.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.137+0000 7f53e67fc640 10 monclient: dump: 2024-09-16T03:42:28.162 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:28.162 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:28.163 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:28.163 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:28.163 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:28.163 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:28.163 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:28.163 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:28.163 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:28.163 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:28.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.137+0000 7f53e67fc640 10 monclient: _finish_auth 0 2024-09-16T03:42:28.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.137+0000 7f53e67fc640 10 monclient: _check_auth_tickets 2024-09-16T03:42:28.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.137+0000 7f53e67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:42:28.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.137+0000 7f53e67fc640 1 -- 172.21.15.42:0/4289003876 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f53ec019070 con 0x7f53f006db90 2024-09-16T03:42:28.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.137+0000 7f53f7adc640 5 monclient: authenticate success, global_id 4239 2024-09-16T03:42:28.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.137+0000 7f53f7adc640 10 monclient: _renew_subs 2024-09-16T03:42:28.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.137+0000 7f53f7adc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:42:28.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.137+0000 7f53f7adc640 1 -- 172.21.15.42:0/4289003876 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53f0132d80 con 0x7f53f006db90 2024-09-16T03:42:28.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.137+0000 7f53e67fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:42:28.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.137+0000 7f53e67fc640 1 -- 172.21.15.42:0/4289003876 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53ec0129b0 con 0x7f53f006db90 2024-09-16T03:42:28.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.137+0000 7f53e67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:28.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.137+0000 7f53e67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:42:28.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.137+0000 7f53e67fc640 10 monclient: dump: 2024-09-16T03:42:28.164 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:28.164 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:28.164 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:28.164 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:28.164 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:28.164 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:28.164 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:28.164 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:28.165 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:28.165 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:28.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.137+0000 7f53e7fff640 4 set_mon_vals no callback set 2024-09-16T03:42:28.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.137+0000 7f53f7adc640 10 client.4239.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:42:28.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.137+0000 7f53f7adc640 10 monclient: _renew_subs 2024-09-16T03:42:28.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.137+0000 7f53f7adc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:42:28.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.137+0000 7f53f7adc640 1 -- 172.21.15.42:0/4289003876 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53f00a1f10 con 0x7f53f006db90 2024-09-16T03:42:28.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.137+0000 7f53f7adc640 1 librados: init done 2024-09-16T03:42:28.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.137+0000 7f53f7adc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:42:28.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.137+0000 7f53f7adc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:42:28.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.137+0000 7f53f7adc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:42:28.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.137+0000 7f53f7adc640 1 -- 172.21.15.42:0/4289003876 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f53f01340c0 con 0x7f53f006db90 2024-09-16T03:42:28.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.137+0000 7f53e67fc640 1 -- 172.21.15.42:0/4289003876 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f53ec012b50 con 0x7f53f006db90 2024-09-16T03:42:28.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.137+0000 7f53e67fc640 1 --2- 172.21.15.42:0/4289003876 >> v2:172.21.15.154:6812/254845342 conn(0x7f53d4037810 0x7f53d4039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:28.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.137+0000 7f53e67fc640 1 -- 172.21.15.42:0/4289003876 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 3562+0+0 (secure 0 0 0) 0x7f53ec048b20 con 0x7f53f006db90 2024-09-16T03:42:28.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.137+0000 7f53e67fc640 10 client.4239.objecter ms_dispatch 0x7f53f0000ba0 osd_map(18..18 src has 1..18) 2024-09-16T03:42:28.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.137+0000 7f53e67fc640 3 client.4239.objecter handle_osd_map got epochs [18,18] > 0 2024-09-16T03:42:28.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.137+0000 7f53e67fc640 3 client.4239.objecter handle_osd_map decoding full epoch 18 2024-09-16T03:42:28.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.137+0000 7f53e67fc640 20 client.4239.objecter dump_active .. 0 homeless 2024-09-16T03:42:28.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.137+0000 7f53f5077640 1 --2- 172.21.15.42:0/4289003876 >> v2:172.21.15.154:6812/254845342 conn(0x7f53d4037810 0x7f53d4039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:28.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.137+0000 7f53f5077640 10 monclient: get_auth_request con 0x7f53d4037810 auth_method 0 2024-09-16T03:42:28.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.137+0000 7f53e67fc640 10 client.4239.objecter ms_handle_connect 0x7f53d4037810 2024-09-16T03:42:28.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.137+0000 7f53f5077640 1 --2- 172.21.15.42:0/4289003876 >> v2:172.21.15.154:6812/254845342 conn(0x7f53d4037810 0x7f53d4039cd0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f53f0137630 tx=0x7f53e0007d60 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:28.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.141+0000 7f53e67fc640 1 -- 172.21.15.42:0/4289003876 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f53ec01d3c0 con 0x7f53f006db90 2024-09-16T03:42:28.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.141+0000 7f53e67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:42:28.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.141+0000 7f53e67fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:42:28.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.305+0000 7f53f7adc640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-16T03:42:28.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.305+0000 7f53f7adc640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T03:42:28.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.305+0000 7f53f7adc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:42:28.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.305+0000 7f53f7adc640 1 -- 172.21.15.42:0/4289003876 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f53f00a0d40 con 0x7f53f006db90 2024-09-16T03:42:28.314 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.305+0000 7f53e67fc640 1 -- 172.21.15.42:0/4289003876 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v18) ==== 74+0+10953 (secure 0 0 0) 0x7f53ec039090 con 0x7f53f006db90 2024-09-16T03:42:28.314 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.305+0000 7f53e67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T03:42:28.314 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.305+0000 7f53e67fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:42:28.314 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-09-16T03:42:28.314 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":18,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","created":"2024-09-16T03:42:05.977277+0000","modified":"2024-09-16T03:42:27.608372+0000","last_up_change":"2024-09-16T03:42:17.544237+0000","last_in_change":"2024-09-16T03:42:08.437271+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T03:42:17.542498+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-16T03:42:23.677980+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":18,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6b9fc058-2cff-46af-b95c-b78c72e1dc2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6808","nonce":1401549752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6809","nonce":1401549752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6811","nonce":1401549752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6810","nonce":1401549752}]},"public_addr":"172.21.15.42:6808/1401549752","cluster_addr":"172.21.15.42:6809/1401549752","heartbeat_back_addr":"172.21.15.42:6811/1401549752","heartbeat_front_addr":"172.21.15.42:6810/1401549752","state":["exists","up"]},{"osd":1,"uuid":"ee31f807-cb16-4c33-8e23-2573e1a6b5c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6800","nonce":4217863575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6801","nonce":4217863575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6803","nonce":4217863575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6802","nonce":4217863575}]},"public_addr":"172.21.15.42:6800/4217863575","cluster_addr":"172.21.15.42:6801/4217863575","heartbeat_back_addr":"172.21.15.42:6803/4217863575","heartbeat_front_addr":"172.21.15.42:6802/4217863575","state":["exists","up"]},{"osd":2,"uuid":"a26f3917-6702-43db-9355-e96da86a3365","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6804","nonce":263835041}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6805","nonce":263835041}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6807","nonce":263835041}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6806","nonce":263835041}]},"public_addr":"172.21.15.42:6804/263835041","cluster_addr":"172.21.15.42:6805/263835041","heartbeat_back_addr":"172.21.15.42:6807/263835041","heartbeat_front_addr":"172.21.15.42:6806/263835041","state":["exists","up"]},{"osd":3,"uuid":"4e809f2a-2ffd-4791-9f2d-18f1bfe7d2ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":4043102363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6801","nonce":4043102363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6803","nonce":4043102363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":4043102363}]},"public_addr":"172.21.15.154:6800/4043102363","cluster_addr":"172.21.15.154:6801/4043102363","heartbeat_back_addr":"172.21.15.154:6803/4043102363","heartbeat_front_addr":"172.21.15.154:6802/4043102363","state":["exists","up"]},{"osd":4,"uuid":"dcf637ab-5f72-4ed2-a99b-69083f0b6b15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":1038452759}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6805","nonce":1038452759}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6807","nonce":1038452759}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":1038452759}]},"public_addr":"172.21.15.154:6804/1038452759","cluster_addr":"172.21.15.154:6805/1038452759","heartbeat_back_addr":"172.21.15.154:6807/1038452759","heartbeat_front_addr":"172.21.15.154:6806/1038452759","state":["exists","up"]},{"osd":5,"uuid":"83c05e44-861b-4bb2-9657-19196e5ac1f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":3026127822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6809","nonce":3026127822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6811","nonce":3026127822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":3026127822}]},"public_addr":"172.21.15.154:6808/3026127822","cluster_addr":"172.21.15.154:6809/3026127822","heartbeat_back_addr":"172.21.15.154:6811/3026127822","heartbeat_front_addr":"172.21.15.154:6810/3026127822","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:42:15.086682+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:42:14.665142+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:42:14.860865+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:42:14.620446+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:42:15.163090+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:42:15.856205+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T03:42:28.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.309+0000 7f53f7adc640 10 librados: watch_flush enter 2024-09-16T03:42:28.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.309+0000 7f53f7adc640 10 librados: watch_flush exit 2024-09-16T03:42:28.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.309+0000 7f53f7adc640 20 client.4239.objecter shutdown clearing up homeless session... 2024-09-16T03:42:28.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.309+0000 7f53f7adc640 10 client.4239.objecter successfully canceled tick 2024-09-16T03:42:28.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.309+0000 7f53f7adc640 1 -- 172.21.15.42:0/4289003876 >> v2:172.21.15.154:6812/254845342 conn(0x7f53d4037810 msgr2=0x7f53d4039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:28.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.309+0000 7f53f7adc640 1 --2- 172.21.15.42:0/4289003876 >> v2:172.21.15.154:6812/254845342 conn(0x7f53d4037810 0x7f53d4039cd0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f53f0137630 tx=0x7f53e0007d60 comp rx=0 tx=0).stop 2024-09-16T03:42:28.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.309+0000 7f53f7adc640 10 monclient: shutdown 2024-09-16T03:42:28.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.309+0000 7f53f7adc640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:42:28.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.309+0000 7f53f7adc640 1 -- 172.21.15.42:0/4289003876 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f53f006db90 msgr2=0x7f53f01cec60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:28.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.309+0000 7f53f7adc640 1 --2- 172.21.15.42:0/4289003876 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f53f006db90 0x7f53f01cec60 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f53ec00d5f0 tx=0x7f53ec00dab0 comp rx=0 tx=0).stop 2024-09-16T03:42:28.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.309+0000 7f53f7adc640 1 -- 172.21.15.42:0/4289003876 shutdown_connections 2024-09-16T03:42:28.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.309+0000 7f53f7adc640 1 --2- 172.21.15.42:0/4289003876 >> v2:172.21.15.154:6812/254845342 conn(0x7f53d4037810 0x7f53d4039cd0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:28.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.309+0000 7f53f7adc640 1 --2- 172.21.15.42:0/4289003876 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f53f00a3c00 0x7f53f01cf6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:28.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.309+0000 7f53f7adc640 1 --2- 172.21.15.42:0/4289003876 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f53f00a37f0 0x7f53f01cf1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:28.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.309+0000 7f53f7adc640 1 --2- 172.21.15.42:0/4289003876 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f53f006db90 0x7f53f01cec60 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:28.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.309+0000 7f53f7adc640 1 -- 172.21.15.42:0/4289003876 >> 172.21.15.42:0/4289003876 conn(0x7f53f00a9fb0 msgr2=0x7f53f00a7a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:42:28.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.309+0000 7f53f7adc640 1 -- 172.21.15.42:0/4289003876 shutdown_connections 2024-09-16T03:42:28.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.309+0000 7f53f7adc640 1 -- 172.21.15.42:0/4289003876 wait complete. 2024-09-16T03:42:28.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.309+0000 7f53f7adc640 1 librados: shutdown 2024-09-16T03:42:28.332 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-09-16T03:42:28.333 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T03:42:28.441 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.433+0000 7f99843c8640 10 monclient: get_monmap_and_config 2024-09-16T03:42:28.441 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.433+0000 7f99843c8640 10 monclient: build_initial_monmap 2024-09-16T03:42:28.441 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.433+0000 7f99843c8640 10 monclient: monmap: 2024-09-16T03:42:28.441 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:42:28.442 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:28.442 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:42:28.439372+0000 2024-09-16T03:42:28.442 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:42:28.439372+0000 2024-09-16T03:42:28.442 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:42:28.442 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:28.442 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:42:28.442 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:42:28.442 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:42:28.442 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:42:28.442 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:42:28.442 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:28.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.437+0000 7f99843c8640 1 Processor -- start 2024-09-16T03:42:28.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.437+0000 7f99843c8640 1 -- start start 2024-09-16T03:42:28.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.437+0000 7f99843c8640 10 monclient: init 2024-09-16T03:42:28.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.437+0000 7f99843c8640 10 monclient: _reopen_session rank -1 2024-09-16T03:42:28.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.437+0000 7f99843c8640 10 monclient: _add_conns ranks=[2,0,3,1,4] 2024-09-16T03:42:28.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.437+0000 7f99843c8640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f997c0a1330 0x7f997c0a1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:28.448 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.437+0000 7f99843c8640 10 monclient(hunting): picked mon.noname-b con 0x7f997c0a1330 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:42:28.448 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.437+0000 7f99843c8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f997c06dac0 addr v1:172.21.15.42:6789/0 2024-09-16T03:42:28.448 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.437+0000 7f99843c8640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f997c0a1c50 0x7f997c135810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:28.449 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.437+0000 7f99843c8640 10 monclient(hunting): picked mon.noname-a con 0x7f997c0a1c50 addr v2:172.21.15.42:3300/0 2024-09-16T03:42:28.449 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.437+0000 7f99843c8640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f997c09c090 con 0x7f997c06dac0 2024-09-16T03:42:28.449 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.437+0000 7f99843c8640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:28.449 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.437+0000 7f99843c8640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f997c09c890 con 0x7f997c0a1c50 2024-09-16T03:42:28.449 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.437+0000 7f99843c8640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:28.449 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.437+0000 7f99843c8640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f997c09f610 con 0x7f997c0a1330 2024-09-16T03:42:28.449 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.437+0000 7f99843c8640 10 monclient(hunting): _renew_subs 2024-09-16T03:42:28.449 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.437+0000 7f99843c8640 10 monclient(hunting): authenticate will time out at 995.409045s 2024-09-16T03:42:28.449 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.437+0000 7f9981963640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f997c06dac0 0x7f997c135100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:46748/0 (socket says 172.21.15.42:46748) 2024-09-16T03:42:28.449 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.437+0000 7f9981963640 1 -- 172.21.15.42:0/4044526761 learned_addr learned my addr 172.21.15.42:0/4044526761 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:42:28.449 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.437+0000 7f9982164640 1 --2- 172.21.15.42:0/4044526761 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f997c0a1330 0x7f997c0a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:28.450 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.437+0000 7f9982965640 1 --2- 172.21.15.42:0/4044526761 >> v2:172.21.15.42:3300/0 conn(0x7f997c0a1c50 0x7f997c135810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:28.450 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.437+0000 7f9982164640 10 monclient(hunting): get_auth_request con 0x7f997c0a1330 auth_method 0 2024-09-16T03:42:28.450 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.437+0000 7f9982164640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:28.450 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.437+0000 7f9982164640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:28.450 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.437+0000 7f9982164640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:28.450 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.437+0000 7f9982965640 10 monclient(hunting): get_auth_request con 0x7f997c0a1c50 auth_method 0 2024-09-16T03:42:28.450 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.437+0000 7f9982965640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:28.450 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.437+0000 7f9982965640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:28.450 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.437+0000 7f9982965640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:28.450 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.437+0000 7f9982164640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:28.450 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.437+0000 7f9982164640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:28.450 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.437+0000 7f9982164640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:28.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.437+0000 7f9982965640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:28.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.437+0000 7f9982965640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:28.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.437+0000 7f9982965640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:28.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.437+0000 7f9981162640 1 -- 172.21.15.42:0/4044526761 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3159180628 0 0) 0x7f997c09c090 con 0x7f997c06dac0 2024-09-16T03:42:28.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.437+0000 7f9981162640 10 monclient(hunting): _negotiate 2024-09-16T03:42:28.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.437+0000 7f9981162640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:28.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.437+0000 7f9981162640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:28.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.437+0000 7f9981162640 10 monclient(hunting): my global_id is 4245 2024-09-16T03:42:28.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.437+0000 7f9981162640 1 -- 172.21.15.42:0/4044526761 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9968003530 con 0x7f997c06dac0 2024-09-16T03:42:28.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.437+0000 7f9982965640 10 monclient(hunting): handle_auth_done global_id 4242 payload 274 2024-09-16T03:42:28.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.437+0000 7f9982965640 1 -- 172.21.15.42:0/4044526761 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f997c0a1330 msgr2=0x7f997c0a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:42:28.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.437+0000 7f9982965640 1 --2- 172.21.15.42:0/4044526761 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f997c0a1330 0x7f997c0a1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:28.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.437+0000 7f9982965640 1 -- 172.21.15.42:0/4044526761 >> v1:172.21.15.42:6789/0 conn(0x7f997c06dac0 legacy=0x7f997c135100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:28.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.437+0000 7f9982965640 10 monclient: _finish_hunting 0 2024-09-16T03:42:28.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.437+0000 7f9982965640 1 monclient: found mon.noname-a 2024-09-16T03:42:28.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.437+0000 7f9982965640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:42:28.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.437+0000 7f9982965640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:42:28.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.437+0000 7f9982965640 1 -- 172.21.15.42:0/4044526761 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f997c09f8f0 con 0x7f997c0a1c50 2024-09-16T03:42:28.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.437+0000 7f9982164640 1 --2- 172.21.15.42:0/4044526761 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f997c0a1330 0x7f997c0a1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:42:28.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.441+0000 7f9982965640 1 --2- 172.21.15.42:0/4044526761 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f997c0a1c50 0x7f997c135810 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f9974009d90 tx=0x7f997401c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e046cee5105ab562 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:28.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.441+0000 7f9981162640 1 -- 172.21.15.42:0/4044526761 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99740b2070 con 0x7f997c0a1c50 2024-09-16T03:42:28.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.441+0000 7f9981162640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:28.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.441+0000 7f9981162640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:42:28.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.441+0000 7f9981162640 10 monclient: dump: 2024-09-16T03:42:28.458 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:28.458 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:28.459 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:28.459 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:28.459 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:28.459 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:28.459 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:28.459 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:28.459 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:28.459 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:28.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.441+0000 7f9981162640 10 monclient: _finish_auth 0 2024-09-16T03:42:28.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.441+0000 7f9981162640 10 monclient: _check_auth_tickets 2024-09-16T03:42:28.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.441+0000 7f9981162640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:42:28.460 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.441+0000 7f9981162640 1 -- 172.21.15.42:0/4044526761 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f99740a58e0 con 0x7f997c0a1c50 2024-09-16T03:42:28.460 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.441+0000 7f9981162640 10 monclient: handle_config config(0 keys) 2024-09-16T03:42:28.460 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.441+0000 7f9981162640 1 -- 172.21.15.42:0/4044526761 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99740a5cb0 con 0x7f997c0a1c50 2024-09-16T03:42:28.460 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.441+0000 7f99843c8640 5 monclient: authenticate success, global_id 4242 2024-09-16T03:42:28.460 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.441+0000 7f99843c8640 10 monclient: get_monmap_and_config success 2024-09-16T03:42:28.460 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.441+0000 7f99843c8640 4 set_mon_vals no callback set 2024-09-16T03:42:28.460 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.441+0000 7f9981162640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:28.460 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.441+0000 7f9981162640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:42:28.460 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.441+0000 7f9981162640 10 monclient: dump: 2024-09-16T03:42:28.460 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:28.460 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:28.461 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:28.461 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:28.461 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:28.461 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:28.461 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:28.461 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:28.461 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:28.461 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:28.461 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.441+0000 7f99843c8640 10 monclient: shutdown 2024-09-16T03:42:28.461 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.441+0000 7f99843c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:42:28.461 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.441+0000 7f99843c8640 1 -- 172.21.15.42:0/4044526761 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f997c0a1c50 msgr2=0x7f997c135810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:28.462 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.441+0000 7f99843c8640 1 --2- 172.21.15.42:0/4044526761 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f997c0a1c50 0x7f997c135810 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f9974009d90 tx=0x7f997401c890 comp rx=0 tx=0).stop 2024-09-16T03:42:28.462 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.441+0000 7f99843c8640 1 -- 172.21.15.42:0/4044526761 shutdown_connections 2024-09-16T03:42:28.462 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.441+0000 7f99843c8640 1 --2- 172.21.15.42:0/4044526761 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f997c0a1c50 0x7f997c135810 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:28.462 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.441+0000 7f99843c8640 1 --2- 172.21.15.42:0/4044526761 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f997c0a1330 0x7f997c0a1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:28.462 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.441+0000 7f99843c8640 1 -- 172.21.15.42:0/4044526761 >> 172.21.15.42:0/4044526761 conn(0x7f997c0a2420 msgr2=0x7f997c0a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:42:28.462 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.441+0000 7f99843c8640 1 -- 172.21.15.42:0/4044526761 shutdown_connections 2024-09-16T03:42:28.462 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.441+0000 7f99843c8640 1 -- 172.21.15.42:0/4044526761 wait complete. 2024-09-16T03:42:28.462 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.441+0000 7f99843c8640 10 monclient: build_initial_monmap 2024-09-16T03:42:28.462 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.441+0000 7f99843c8640 10 monclient: monmap: 2024-09-16T03:42:28.462 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:42:28.462 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:42:28.463 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:42:28.463 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:42:28.463 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:42:28.463 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:28.463 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:42:28.463 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:42:28.463 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:42:28.463 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:28.463 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.441+0000 7f99843c8640 1 librados: starting msgr at 2024-09-16T03:42:28.463 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.441+0000 7f99843c8640 1 librados: starting objecter 2024-09-16T03:42:28.463 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.441+0000 7f99843c8640 1 Processor -- start 2024-09-16T03:42:28.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.441+0000 7f99843c8640 1 -- start start 2024-09-16T03:42:28.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.441+0000 7f99843c8640 1 librados: setting wanted keys 2024-09-16T03:42:28.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.441+0000 7f99843c8640 1 librados: calling monclient init 2024-09-16T03:42:28.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.441+0000 7f99843c8640 10 monclient: init 2024-09-16T03:42:28.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.441+0000 7f99843c8640 10 monclient: _reopen_session rank -1 2024-09-16T03:42:28.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.441+0000 7f99843c8640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:42:28.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.441+0000 7f99843c8640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f997c0a1330 0x7f997c13cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:28.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.441+0000 7f99843c8640 10 monclient(hunting): picked mon.noname-b con 0x7f997c0a1330 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:42:28.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.441+0000 7f99843c8640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f997c0a1c50 0x7f997c134fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:28.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.441+0000 7f99843c8640 10 monclient(hunting): picked mon.noname-a con 0x7f997c0a1c50 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:42:28.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.441+0000 7f99843c8640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f997c135510 0x7f997c09cb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:28.465 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.441+0000 7f99843c8640 10 monclient(hunting): picked mon.noname-c con 0x7f997c135510 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:42:28.465 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.441+0000 7f99843c8640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:28.465 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.441+0000 7f99843c8640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f997c09ef30 con 0x7f997c0a1c50 2024-09-16T03:42:28.465 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.441+0000 7f99843c8640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:28.465 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.441+0000 7f99843c8640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f997c09edb0 con 0x7f997c0a1330 2024-09-16T03:42:28.465 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.441+0000 7f99843c8640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:28.465 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.441+0000 7f99843c8640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f997c135d50 con 0x7f997c135510 2024-09-16T03:42:28.465 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.441+0000 7f99843c8640 10 monclient(hunting): _renew_subs 2024-09-16T03:42:28.465 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.441+0000 7f99843c8640 10 monclient(hunting): authenticate will time out at 995.415220s 2024-09-16T03:42:28.465 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.441+0000 7f9981963640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f997c0a1c50 0x7f997c134fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:28.465 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.441+0000 7f9982965640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f997c135510 0x7f997c09cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:28.466 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.441+0000 7f9982965640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f997c135510 0x7f997c09cb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:41244/0 (socket says 172.21.15.42:41244) 2024-09-16T03:42:28.466 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.441+0000 7f9981963640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f997c0a1c50 0x7f997c134fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:37550/0 (socket says 172.21.15.42:37550) 2024-09-16T03:42:28.466 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.441+0000 7f9982965640 1 -- 172.21.15.42:0/2897852794 learned_addr learned my addr 172.21.15.42:0/2897852794 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:42:28.466 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.441+0000 7f9982965640 10 monclient(hunting): get_auth_request con 0x7f997c135510 auth_method 0 2024-09-16T03:42:28.466 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.441+0000 7f9982965640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:28.466 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.441+0000 7f9982965640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:28.466 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.441+0000 7f9982965640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:28.466 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.441+0000 7f9981963640 10 monclient(hunting): get_auth_request con 0x7f997c0a1c50 auth_method 0 2024-09-16T03:42:28.466 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.441+0000 7f9981963640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:28.466 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.441+0000 7f9981963640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:28.466 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.441+0000 7f9981963640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:28.466 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.441+0000 7f9982965640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:28.467 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.441+0000 7f9982965640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:28.467 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.441+0000 7f9982965640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:28.467 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.441+0000 7f9981963640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:28.467 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.441+0000 7f9981963640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:28.467 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.441+0000 7f9981963640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:28.467 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.441+0000 7f9982164640 1 --2- 172.21.15.42:0/2897852794 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f997c0a1330 0x7f997c13cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:28.467 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.441+0000 7f9982164640 10 monclient(hunting): get_auth_request con 0x7f997c0a1330 auth_method 0 2024-09-16T03:42:28.467 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.441+0000 7f9982164640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:28.467 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.441+0000 7f9982164640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:28.467 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.441+0000 7f9982164640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:28.467 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.441+0000 7f9982965640 10 monclient(hunting): handle_auth_done global_id 4217 payload 824 2024-09-16T03:42:28.468 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.441+0000 7f9982965640 1 -- 172.21.15.42:0/2897852794 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f997c0a1330 msgr2=0x7f997c13cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:28.468 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.441+0000 7f9982965640 1 --2- 172.21.15.42:0/2897852794 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f997c0a1330 0x7f997c13cbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:28.468 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.441+0000 7f9982965640 1 -- 172.21.15.42:0/2897852794 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f997c0a1c50 msgr2=0x7f997c134fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:28.468 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.441+0000 7f9982965640 1 --2- 172.21.15.42:0/2897852794 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f997c0a1c50 0x7f997c134fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:28.468 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.441+0000 7f9982965640 10 monclient: _finish_hunting 0 2024-09-16T03:42:28.468 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.441+0000 7f9982965640 1 monclient: found mon.noname-c 2024-09-16T03:42:28.468 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.441+0000 7f9982965640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:42:28.468 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.441+0000 7f9982965640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:42:28.468 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.441+0000 7f9982965640 1 -- 172.21.15.42:0/2897852794 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f997c1cf100 con 0x7f997c135510 2024-09-16T03:42:28.468 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.441+0000 7f9982965640 1 --2- 172.21.15.42:0/2897852794 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f997c135510 0x7f997c09cb60 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f9974009d90 tx=0x7f9974019630 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:28.468 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.441+0000 7f996effd640 10 client.?.objecter ms_handle_connect 0x7f997c135510 2024-09-16T03:42:28.469 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.441+0000 7f996effd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:42:28.469 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.445+0000 7f996effd640 1 -- 172.21.15.42:0/2897852794 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99740b2050 con 0x7f997c135510 2024-09-16T03:42:28.469 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.445+0000 7f996effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:28.469 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.445+0000 7f996effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:42:28.469 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.445+0000 7f996effd640 10 monclient: dump: 2024-09-16T03:42:28.469 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:28.469 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:28.469 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:28.469 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:28.469 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:28.469 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:28.470 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:28.470 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:28.470 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:28.470 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:28.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.445+0000 7f996effd640 10 monclient: _finish_auth 0 2024-09-16T03:42:28.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.445+0000 7f996effd640 10 monclient: _check_auth_tickets 2024-09-16T03:42:28.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.445+0000 7f996effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:42:28.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.445+0000 7f996effd640 1 -- 172.21.15.42:0/2897852794 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f99740be070 con 0x7f997c135510 2024-09-16T03:42:28.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.445+0000 7f99843c8640 5 monclient: authenticate success, global_id 4217 2024-09-16T03:42:28.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.445+0000 7f99843c8640 10 monclient: _renew_subs 2024-09-16T03:42:28.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.445+0000 7f99843c8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:42:28.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.445+0000 7f99843c8640 1 -- 172.21.15.42:0/2897852794 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f997c121980 con 0x7f997c135510 2024-09-16T03:42:28.471 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.445+0000 7f996effd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:42:28.471 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.445+0000 7f996effd640 1 -- 172.21.15.42:0/2897852794 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99740ada90 con 0x7f997c135510 2024-09-16T03:42:28.471 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.445+0000 7f996effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:28.471 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.445+0000 7f996effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:42:28.471 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.445+0000 7f996effd640 10 monclient: dump: 2024-09-16T03:42:28.471 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:28.471 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:28.471 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:28.471 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:28.471 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:28.471 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:28.472 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:28.472 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:28.472 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:28.472 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:28.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.445+0000 7f99843c8640 10 client.4217.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:42:28.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.445+0000 7f99843c8640 10 monclient: _renew_subs 2024-09-16T03:42:28.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.445+0000 7f99843c8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:42:28.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.445+0000 7f9980961640 4 set_mon_vals no callback set 2024-09-16T03:42:28.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.445+0000 7f99843c8640 1 -- 172.21.15.42:0/2897852794 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f997c073a30 con 0x7f997c135510 2024-09-16T03:42:28.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.445+0000 7f99843c8640 1 librados: init done 2024-09-16T03:42:28.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.445+0000 7f996effd640 1 -- 172.21.15.42:0/2897852794 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f99740a5760 con 0x7f997c135510 2024-09-16T03:42:28.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.445+0000 7f99843c8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:42:28.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.445+0000 7f99843c8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:42:28.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.445+0000 7f99843c8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:42:28.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.445+0000 7f99843c8640 1 -- 172.21.15.42:0/2897852794 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f997c1210e0 con 0x7f997c135510 2024-09-16T03:42:28.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.445+0000 7f996effd640 1 --2- 172.21.15.42:0/2897852794 >> v2:172.21.15.154:6812/254845342 conn(0x7f9960037810 0x7f9960039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:28.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.445+0000 7f996effd640 1 -- 172.21.15.42:0/2897852794 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(18..18 src has 1..18) ==== 3562+0+0 (secure 0 0 0) 0x7f99740e38a0 con 0x7f997c135510 2024-09-16T03:42:28.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.445+0000 7f996effd640 10 client.4217.objecter ms_dispatch 0x7f997c000ba0 osd_map(18..18 src has 1..18) 2024-09-16T03:42:28.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.445+0000 7f996effd640 3 client.4217.objecter handle_osd_map got epochs [18,18] > 0 2024-09-16T03:42:28.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.445+0000 7f996effd640 3 client.4217.objecter handle_osd_map decoding full epoch 18 2024-09-16T03:42:28.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.445+0000 7f996effd640 20 client.4217.objecter dump_active .. 0 homeless 2024-09-16T03:42:28.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.445+0000 7f9982164640 1 --2- 172.21.15.42:0/2897852794 >> v2:172.21.15.154:6812/254845342 conn(0x7f9960037810 0x7f9960039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:28.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.445+0000 7f9982164640 10 monclient: get_auth_request con 0x7f9960037810 auth_method 0 2024-09-16T03:42:28.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.445+0000 7f9982164640 1 --2- 172.21.15.42:0/2897852794 >> v2:172.21.15.154:6812/254845342 conn(0x7f9960037810 0x7f9960039cd0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f9970009580 tx=0x7f9970008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:28.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.445+0000 7f996effd640 10 client.4217.objecter ms_handle_connect 0x7f9960037810 2024-09-16T03:42:28.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.449+0000 7f996effd640 1 -- 172.21.15.42:0/2897852794 <== mon.2 v2:172.21.15.42:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f997c1210e0 con 0x7f997c135510 2024-09-16T03:42:28.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.449+0000 7f996effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:42:28.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.449+0000 7f996effd640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:42:28.619 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.613+0000 7f99843c8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-16T03:42:28.619 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.613+0000 7f99843c8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T03:42:28.619 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.613+0000 7f99843c8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:42:28.619 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.613+0000 7f99843c8640 1 -- 172.21.15.42:0/2897852794 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f997c070fe0 con 0x7f997c135510 2024-09-16T03:42:28.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.617+0000 7f996effd640 1 -- 172.21.15.42:0/2897852794 <== mon.2 v2:172.21.15.42:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v18) ==== 74+0+10953 (secure 0 0 0) 0x7f997c1cf100 con 0x7f997c135510 2024-09-16T03:42:28.625 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.617+0000 7f996effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T03:42:28.625 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.617+0000 7f996effd640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:42:28.625 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-09-16T03:42:28.625 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":18,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","created":"2024-09-16T03:42:05.977277+0000","modified":"2024-09-16T03:42:27.608372+0000","last_up_change":"2024-09-16T03:42:17.544237+0000","last_in_change":"2024-09-16T03:42:08.437271+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T03:42:17.542498+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-16T03:42:23.677980+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":18,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6b9fc058-2cff-46af-b95c-b78c72e1dc2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6808","nonce":1401549752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6809","nonce":1401549752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6811","nonce":1401549752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6810","nonce":1401549752}]},"public_addr":"172.21.15.42:6808/1401549752","cluster_addr":"172.21.15.42:6809/1401549752","heartbeat_back_addr":"172.21.15.42:6811/1401549752","heartbeat_front_addr":"172.21.15.42:6810/1401549752","state":["exists","up"]},{"osd":1,"uuid":"ee31f807-cb16-4c33-8e23-2573e1a6b5c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6800","nonce":4217863575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6801","nonce":4217863575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6803","nonce":4217863575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6802","nonce":4217863575}]},"public_addr":"172.21.15.42:6800/4217863575","cluster_addr":"172.21.15.42:6801/4217863575","heartbeat_back_addr":"172.21.15.42:6803/4217863575","heartbeat_front_addr":"172.21.15.42:6802/4217863575","state":["exists","up"]},{"osd":2,"uuid":"a26f3917-6702-43db-9355-e96da86a3365","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6804","nonce":263835041}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6805","nonce":263835041}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6807","nonce":263835041}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6806","nonce":263835041}]},"public_addr":"172.21.15.42:6804/263835041","cluster_addr":"172.21.15.42:6805/263835041","heartbeat_back_addr":"172.21.15.42:6807/263835041","heartbeat_front_addr":"172.21.15.42:6806/263835041","state":["exists","up"]},{"osd":3,"uuid":"4e809f2a-2ffd-4791-9f2d-18f1bfe7d2ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":4043102363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6801","nonce":4043102363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6803","nonce":4043102363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":4043102363}]},"public_addr":"172.21.15.154:6800/4043102363","cluster_addr":"172.21.15.154:6801/4043102363","heartbeat_back_addr":"172.21.15.154:6803/4043102363","heartbeat_front_addr":"172.21.15.154:6802/4043102363","state":["exists","up"]},{"osd":4,"uuid":"dcf637ab-5f72-4ed2-a99b-69083f0b6b15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":1038452759}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6805","nonce":1038452759}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6807","nonce":1038452759}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":1038452759}]},"public_addr":"172.21.15.154:6804/1038452759","cluster_addr":"172.21.15.154:6805/1038452759","heartbeat_back_addr":"172.21.15.154:6807/1038452759","heartbeat_front_addr":"172.21.15.154:6806/1038452759","state":["exists","up"]},{"osd":5,"uuid":"83c05e44-861b-4bb2-9657-19196e5ac1f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":3026127822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6809","nonce":3026127822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6811","nonce":3026127822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":3026127822}]},"public_addr":"172.21.15.154:6808/3026127822","cluster_addr":"172.21.15.154:6809/3026127822","heartbeat_back_addr":"172.21.15.154:6811/3026127822","heartbeat_front_addr":"172.21.15.154:6810/3026127822","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:42:15.086682+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:42:14.665142+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:42:14.860865+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:42:14.620446+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:42:15.163090+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:42:15.856205+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T03:42:28.627 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.621+0000 7f99843c8640 10 librados: watch_flush enter 2024-09-16T03:42:28.627 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.621+0000 7f99843c8640 10 librados: watch_flush exit 2024-09-16T03:42:28.628 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.621+0000 7f99843c8640 20 client.4217.objecter shutdown clearing up homeless session... 2024-09-16T03:42:28.628 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.621+0000 7f99843c8640 10 client.4217.objecter successfully canceled tick 2024-09-16T03:42:28.628 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.621+0000 7f99843c8640 1 -- 172.21.15.42:0/2897852794 >> v2:172.21.15.154:6812/254845342 conn(0x7f9960037810 msgr2=0x7f9960039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:28.628 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.621+0000 7f99843c8640 1 --2- 172.21.15.42:0/2897852794 >> v2:172.21.15.154:6812/254845342 conn(0x7f9960037810 0x7f9960039cd0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f9970009580 tx=0x7f9970008040 comp rx=0 tx=0).stop 2024-09-16T03:42:28.628 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.621+0000 7f99843c8640 10 monclient: shutdown 2024-09-16T03:42:28.628 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.621+0000 7f99843c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:42:28.628 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.621+0000 7f99843c8640 1 -- 172.21.15.42:0/2897852794 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f997c135510 msgr2=0x7f997c09cb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:28.628 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.621+0000 7f99843c8640 1 --2- 172.21.15.42:0/2897852794 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f997c135510 0x7f997c09cb60 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f9974009d90 tx=0x7f9974019630 comp rx=0 tx=0).stop 2024-09-16T03:42:28.628 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.621+0000 7f99843c8640 1 -- 172.21.15.42:0/2897852794 shutdown_connections 2024-09-16T03:42:28.628 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.621+0000 7f99843c8640 1 --2- 172.21.15.42:0/2897852794 >> v2:172.21.15.154:6812/254845342 conn(0x7f9960037810 0x7f9960039cd0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:28.628 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.621+0000 7f99843c8640 1 --2- 172.21.15.42:0/2897852794 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f997c135510 0x7f997c09cb60 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:28.628 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.621+0000 7f99843c8640 1 --2- 172.21.15.42:0/2897852794 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f997c0a1c50 0x7f997c134fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:28.628 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.621+0000 7f99843c8640 1 --2- 172.21.15.42:0/2897852794 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f997c0a1330 0x7f997c13cbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:28.629 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.621+0000 7f99843c8640 1 -- 172.21.15.42:0/2897852794 >> 172.21.15.42:0/2897852794 conn(0x7f997c0a2420 msgr2=0x7f997c0707c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:42:28.629 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.621+0000 7f99843c8640 1 -- 172.21.15.42:0/2897852794 shutdown_connections 2024-09-16T03:42:28.629 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.621+0000 7f99843c8640 1 -- 172.21.15.42:0/2897852794 wait complete. 2024-09-16T03:42:28.629 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.621+0000 7f99843c8640 1 librados: shutdown 2024-09-16T03:42:28.644 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-16T03:42:28.644 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-16T03:42:28.645 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-16T03:42:28.645 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-16T03:42:28.646 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-09-16T03:42:28.646 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-09-16T03:42:28.764 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.757+0000 7ff1d0aad640 10 monclient: get_monmap_and_config 2024-09-16T03:42:28.764 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.757+0000 7ff1d0aad640 10 monclient: build_initial_monmap 2024-09-16T03:42:28.764 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.757+0000 7ff1d0aad640 10 monclient: monmap: 2024-09-16T03:42:28.764 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:42:28.764 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:28.764 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:42:28.761404+0000 2024-09-16T03:42:28.764 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:42:28.761404+0000 2024-09-16T03:42:28.764 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:42:28.765 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:28.765 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:42:28.765 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:42:28.765 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:42:28.765 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:42:28.765 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:42:28.765 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:28.765 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.757+0000 7ff1d0aad640 1 Processor -- start 2024-09-16T03:42:28.765 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.757+0000 7ff1d0aad640 1 -- start start 2024-09-16T03:42:28.765 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.757+0000 7ff1d0aad640 10 monclient: init 2024-09-16T03:42:28.766 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.757+0000 7ff1d0aad640 10 monclient: _reopen_session rank -1 2024-09-16T03:42:28.767 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.757+0000 7ff1d0aad640 10 monclient: _add_conns ranks=[1,4,3,0,2] 2024-09-16T03:42:28.767 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.757+0000 7ff1d0aad640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff1cc0a39b0 addr v1:172.21.15.154:6789/0 2024-09-16T03:42:28.767 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.757+0000 7ff1d0aad640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7ff1cc0a3d90 0x7ff1cc13c380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:28.767 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.757+0000 7ff1d0aad640 10 monclient(hunting): picked mon.noname-c con 0x7ff1cc0a3d90 addr v2:172.21.15.154:3300/0 2024-09-16T03:42:28.767 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.757+0000 7ff1d0aad640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff1cc13c8c0 0x7ff1cc13ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:28.767 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.757+0000 7ff1d0aad640 10 monclient(hunting): picked mon.noname-a con 0x7ff1cc13c8c0 addr v2:172.21.15.42:3300/0 2024-09-16T03:42:28.767 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.757+0000 7ff1cad9d640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff1cc13c8c0 0x7ff1cc13ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:28.767 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.757+0000 7ff1cad9d640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff1cc13c8c0 0x7ff1cc13ecb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:37560/0 (socket says 172.21.15.42:37560) 2024-09-16T03:42:28.767 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.757+0000 7ff1cad9d640 1 -- 172.21.15.42:0/2595147914 learned_addr learned my addr 172.21.15.42:0/2595147914 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:42:28.768 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.757+0000 7fa6ff288640 10 monclient: get_monmap_and_config 2024-09-16T03:42:28.768 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.757+0000 7fa6ff288640 10 monclient: build_initial_monmap 2024-09-16T03:42:28.768 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.757+0000 7fa6ff288640 10 monclient: monmap: 2024-09-16T03:42:28.768 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:42:28.768 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:28.768 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:42:28.762986+0000 2024-09-16T03:42:28.768 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:42:28.762986+0000 2024-09-16T03:42:28.768 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:42:28.768 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:28.768 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:42:28.769 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:42:28.769 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:42:28.769 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:42:28.769 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:42:28.769 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:28.770 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.757+0000 7ff1d0aad640 1 -- 172.21.15.42:0/2595147914 --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff1cc134290 con 0x7ff1cc0a39b0 2024-09-16T03:42:28.770 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.757+0000 7ff1d0aad640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:28.771 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.757+0000 7ff1d0aad640 1 -- 172.21.15.42:0/2595147914 --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7ff1cc09c140 con 0x7ff1cc13c8c0 2024-09-16T03:42:28.771 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.757+0000 7ff1d0aad640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:28.771 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.757+0000 7ff1d0aad640 1 -- 172.21.15.42:0/2595147914 --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7ff1cc09c420 con 0x7ff1cc0a3d90 2024-09-16T03:42:28.771 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.757+0000 7ff1d0aad640 10 monclient(hunting): _renew_subs 2024-09-16T03:42:28.771 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.757+0000 7ff1d0aad640 10 monclient(hunting): authenticate will time out at 995.731269s 2024-09-16T03:42:28.771 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.757+0000 7ff1c9d9b640 1 --2- 172.21.15.42:0/2595147914 >> v2:172.21.15.154:3300/0 conn(0x7ff1cc0a3d90 0x7ff1cc13c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:28.771 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.757+0000 7ff1c9d9b640 10 monclient(hunting): get_auth_request con 0x7ff1cc0a3d90 auth_method 0 2024-09-16T03:42:28.771 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.757+0000 7ff1c9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:28.771 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.757+0000 7ff1c9d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:28.771 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.757+0000 7ff1c9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:28.771 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.757+0000 7ff1cad9d640 10 monclient(hunting): get_auth_request con 0x7ff1cc13c8c0 auth_method 0 2024-09-16T03:42:28.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.757+0000 7ff1cad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:28.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.757+0000 7ff1cad9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:28.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.757+0000 7ff1cad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:28.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.757+0000 7ff1c9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:28.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.757+0000 7ff1c9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:28.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.757+0000 7ff1c9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:28.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.757+0000 7ff1cad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:28.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.757+0000 7ff1cad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:28.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.757+0000 7ff1cad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:28.774 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.757+0000 7fa6ff288640 1 Processor -- start 2024-09-16T03:42:28.774 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.757+0000 7fa6ff288640 1 -- start start 2024-09-16T03:42:28.774 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.757+0000 7fa6ff288640 10 monclient: init 2024-09-16T03:42:28.774 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.761+0000 7fa6ff288640 10 monclient: _reopen_session rank -1 2024-09-16T03:42:28.774 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.761+0000 7fa6ff288640 10 monclient: _add_conns ranks=[0,3,2,4,1] 2024-09-16T03:42:28.774 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.761+0000 7fa6ff288640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa6f80a1330 addr v1:172.21.15.42:6789/0 2024-09-16T03:42:28.774 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.761+0000 7fa6ff288640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fa6f809c920 0x7fa6f813c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:28.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.761+0000 7fa6fc823640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fa6f809c920 0x7fa6f813c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:28.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.761+0000 7fa6fd024640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fa6f80a1330 0x7fa6f80a1710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:46762/0 (socket says 172.21.15.42:46762) 2024-09-16T03:42:28.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.761+0000 7fa6fd024640 1 -- 172.21.15.42:0/3666849990 learned_addr learned my addr 172.21.15.42:0/3666849990 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:42:28.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.761+0000 7fa6ff288640 10 monclient(hunting): picked mon.noname-a con 0x7fa6f809c920 addr v2:172.21.15.42:3300/0 2024-09-16T03:42:28.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.761+0000 7fa6ff288640 1 --2- 172.21.15.42:0/3666849990 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa6f8072d40 0x7fa6f813eac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:28.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.761+0000 7fa6ff288640 10 monclient(hunting): picked mon.noname-b con 0x7fa6f8072d40 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:42:28.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.761+0000 7fa6ff288640 1 -- 172.21.15.42:0/3666849990 --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa6f8072350 con 0x7fa6f80a1330 2024-09-16T03:42:28.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.761+0000 7fa6fd825640 1 --2- 172.21.15.42:0/3666849990 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa6f8072d40 0x7fa6f813eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:28.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.761+0000 7fa6ff288640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:28.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.761+0000 7fa6ff288640 1 -- 172.21.15.42:0/3666849990 --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7fa6f8072b50 con 0x7fa6f809c920 2024-09-16T03:42:28.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.761+0000 7fa6ff288640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:28.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.761+0000 7fa6ff288640 1 -- 172.21.15.42:0/3666849990 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fa6f809bc60 con 0x7fa6f8072d40 2024-09-16T03:42:28.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.761+0000 7fa6ff288640 10 monclient(hunting): _renew_subs 2024-09-16T03:42:28.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.761+0000 7fa6ff288640 10 monclient(hunting): authenticate will time out at 995.733138s 2024-09-16T03:42:28.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.761+0000 7fa6fc823640 10 monclient(hunting): get_auth_request con 0x7fa6f809c920 auth_method 0 2024-09-16T03:42:28.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.761+0000 7fa6fc823640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:28.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.761+0000 7fa6fc823640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:28.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.761+0000 7fa6fc823640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:28.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.761+0000 7fa6fd825640 10 monclient(hunting): get_auth_request con 0x7fa6f8072d40 auth_method 0 2024-09-16T03:42:28.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.761+0000 7fa6fd825640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:28.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.761+0000 7fa6fd825640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:28.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.761+0000 7fa6fd825640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:28.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.761+0000 7fa6effff640 1 -- 172.21.15.42:0/3666849990 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1200406231 0 0) 0x7fa6f8072350 con 0x7fa6f80a1330 2024-09-16T03:42:28.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.761+0000 7fa6effff640 10 monclient(hunting): _negotiate 2024-09-16T03:42:28.777 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.761+0000 7fa6effff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:28.777 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.761+0000 7fa6effff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:28.777 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.761+0000 7fa6effff640 10 monclient(hunting): my global_id is 4254 2024-09-16T03:42:28.777 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.761+0000 7fa6effff640 1 -- 172.21.15.42:0/3666849990 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa6e4003530 con 0x7fa6f80a1330 2024-09-16T03:42:28.777 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.761+0000 7fa6fd825640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:28.777 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.761+0000 7fa6fd825640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:28.777 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.761+0000 7fa6fd825640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:28.777 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.761+0000 7fa6fc823640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:28.777 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.761+0000 7fa6fc823640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:28.777 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.761+0000 7fa6fc823640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:28.778 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.761+0000 7fa6fd825640 10 monclient(hunting): handle_auth_done global_id 4220 payload 274 2024-09-16T03:42:28.778 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.761+0000 7fa6fd825640 1 -- 172.21.15.42:0/3666849990 >> v2:172.21.15.42:3300/0 conn(0x7fa6f809c920 msgr2=0x7fa6f813c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:42:28.778 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.761+0000 7fa6fd825640 1 --2- 172.21.15.42:0/3666849990 >> v2:172.21.15.42:3300/0 conn(0x7fa6f809c920 0x7fa6f813c570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:28.778 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.761+0000 7fa6fd825640 1 -- 172.21.15.42:0/3666849990 >> v1:172.21.15.42:6789/0 conn(0x7fa6f80a1330 legacy=0x7fa6f80a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:28.778 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.761+0000 7fa6fd825640 10 monclient: _finish_hunting 0 2024-09-16T03:42:28.778 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.761+0000 7fa6fd825640 1 monclient: found mon.noname-b 2024-09-16T03:42:28.778 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.761+0000 7fa6fd825640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:42:28.778 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.761+0000 7fa6fd825640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T03:42:28.778 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.761+0000 7fa6fd825640 1 -- 172.21.15.42:0/3666849990 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6f809bf40 con 0x7fa6f8072d40 2024-09-16T03:42:28.778 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.761+0000 7fa6fc823640 1 --2- 172.21.15.42:0/3666849990 >> v2:172.21.15.42:3300/0 conn(0x7fa6f809c920 0x7fa6f813c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:42:28.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.757+0000 7ff1cad9d640 10 monclient(hunting): handle_auth_done global_id 4251 payload 274 2024-09-16T03:42:28.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.757+0000 7ff1cad9d640 1 -- 172.21.15.42:0/2595147914 >> v2:172.21.15.154:3300/0 conn(0x7ff1cc0a3d90 msgr2=0x7ff1cc13c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:42:28.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.757+0000 7ff1cad9d640 1 --2- 172.21.15.42:0/2595147914 >> v2:172.21.15.154:3300/0 conn(0x7ff1cc0a3d90 0x7ff1cc13c380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:28.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.757+0000 7ff1cad9d640 1 -- 172.21.15.42:0/2595147914 >> v1:172.21.15.154:6789/0 conn(0x7ff1cc0a39b0 legacy=0x7ff1cc133b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:28.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.757+0000 7ff1cad9d640 10 monclient: _finish_hunting 0 2024-09-16T03:42:28.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.757+0000 7ff1cad9d640 1 monclient: found mon.noname-a 2024-09-16T03:42:28.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.757+0000 7ff1cad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:42:28.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.757+0000 7ff1cad9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:42:28.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.757+0000 7ff1cad9d640 1 -- 172.21.15.42:0/2595147914 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1cc06dd60 con 0x7ff1cc13c8c0 2024-09-16T03:42:28.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.761+0000 7ff1cad9d640 1 --2- 172.21.15.42:0/2595147914 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff1cc13c8c0 0x7ff1cc13ecb0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7ff1cc0a56c0 tx=0x7ff1c401c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3faee6c6e8a74f25 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:28.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.761+0000 7ff1c959a640 1 -- 172.21.15.42:0/2595147914 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1c40a5590 con 0x7ff1cc13c8c0 2024-09-16T03:42:28.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.761+0000 7ff1c959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:28.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.761+0000 7ff1c959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:42:28.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.761+0000 7ff1c959a640 10 monclient: dump: 2024-09-16T03:42:28.781 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:28.781 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:28.781 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:28.782 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:28.782 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:28.782 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:28.782 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:28.782 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:28.782 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:28.782 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:28.782 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.761+0000 7ff1c959a640 10 monclient: _finish_auth 0 2024-09-16T03:42:28.782 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.761+0000 7ff1c959a640 10 monclient: _check_auth_tickets 2024-09-16T03:42:28.782 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.761+0000 7ff1c959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:42:28.782 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.761+0000 7ff1c959a640 1 -- 172.21.15.42:0/2595147914 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff1c40a5730 con 0x7ff1cc13c8c0 2024-09-16T03:42:28.782 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.761+0000 7ff1c959a640 10 monclient: handle_config config(0 keys) 2024-09-16T03:42:28.782 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.761+0000 7ff1c959a640 1 -- 172.21.15.42:0/2595147914 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1c40a5b40 con 0x7ff1cc13c8c0 2024-09-16T03:42:28.783 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.761+0000 7ff1c959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:28.783 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.761+0000 7ff1c959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:42:28.783 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.761+0000 7ff1c959a640 10 monclient: dump: 2024-09-16T03:42:28.783 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:28.783 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:28.783 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:28.783 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:28.783 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:28.783 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:28.783 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:28.783 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:28.783 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:28.783 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:28.783 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.761+0000 7ff1d0aad640 5 monclient: authenticate success, global_id 4251 2024-09-16T03:42:28.783 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.761+0000 7ff1d0aad640 10 monclient: get_monmap_and_config success 2024-09-16T03:42:28.784 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.761+0000 7ff1d0aad640 4 set_mon_vals no callback set 2024-09-16T03:42:28.784 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.761+0000 7ff1d0aad640 10 monclient: shutdown 2024-09-16T03:42:28.784 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.761+0000 7ff1d0aad640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:42:28.784 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.761+0000 7ff1d0aad640 1 -- 172.21.15.42:0/2595147914 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff1cc13c8c0 msgr2=0x7ff1cc13ecb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:28.784 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.761+0000 7ff1d0aad640 1 --2- 172.21.15.42:0/2595147914 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff1cc13c8c0 0x7ff1cc13ecb0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7ff1cc0a56c0 tx=0x7ff1c401c8d0 comp rx=0 tx=0).stop 2024-09-16T03:42:28.784 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.765+0000 7ff1d0aad640 1 -- 172.21.15.42:0/2595147914 shutdown_connections 2024-09-16T03:42:28.784 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.765+0000 7ff1d0aad640 1 --2- 172.21.15.42:0/2595147914 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff1cc13c8c0 0x7ff1cc13ecb0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:28.784 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.765+0000 7ff1d0aad640 1 --2- 172.21.15.42:0/2595147914 >> v2:172.21.15.154:3300/0 conn(0x7ff1cc0a3d90 0x7ff1cc13c380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:28.784 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.765+0000 7ff1d0aad640 1 -- 172.21.15.42:0/2595147914 >> 172.21.15.42:0/2595147914 conn(0x7ff1cc0a9fb0 msgr2=0x7ff1cc0aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:42:28.784 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.765+0000 7ff1d0aad640 1 -- 172.21.15.42:0/2595147914 shutdown_connections 2024-09-16T03:42:28.784 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.765+0000 7ff1d0aad640 1 -- 172.21.15.42:0/2595147914 wait complete. 2024-09-16T03:42:28.785 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.765+0000 7fa6fd825640 1 --2- 172.21.15.42:0/3666849990 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa6f8072d40 0x7fa6f813eac0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fa6f0009d90 tx=0x7fa6f001c670 comp rx=0 tx=0).ready entity=mon.2 client_cookie=619f053d17a76703 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:28.785 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.765+0000 7fa6effff640 1 -- 172.21.15.42:0/3666849990 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6f4001680 con 0x7fa6f8072d40 2024-09-16T03:42:28.785 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.765+0000 7fa6effff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:28.785 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.765+0000 7fa6effff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:42:28.785 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.765+0000 7fa6effff640 10 monclient: dump: 2024-09-16T03:42:28.785 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:28.785 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:28.786 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:28.786 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:28.786 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:28.786 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:28.786 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:28.786 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:28.786 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:28.786 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:28.786 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.765+0000 7fa6effff640 10 monclient: _finish_auth 0 2024-09-16T03:42:28.786 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.765+0000 7fa6effff640 10 monclient: _check_auth_tickets 2024-09-16T03:42:28.786 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.765+0000 7fa6effff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:42:28.786 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.765+0000 7fa6effff640 1 -- 172.21.15.42:0/3666849990 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa6f00a8430 con 0x7fa6f8072d40 2024-09-16T03:42:28.786 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.765+0000 7fa6effff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:42:28.786 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.765+0000 7fa6effff640 1 -- 172.21.15.42:0/3666849990 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6f00a8800 con 0x7fa6f8072d40 2024-09-16T03:42:28.787 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.765+0000 7fa6effff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:28.787 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.765+0000 7fa6effff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:42:28.787 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.765+0000 7fa6effff640 10 monclient: dump: 2024-09-16T03:42:28.787 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:28.787 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:28.787 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:28.787 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:28.787 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:28.787 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:28.787 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:28.787 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:28.787 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:28.787 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:28.788 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.765+0000 7ff1d0aad640 10 monclient: build_initial_monmap 2024-09-16T03:42:28.788 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.765+0000 7ff1d0aad640 10 monclient: monmap: 2024-09-16T03:42:28.788 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:42:28.788 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:42:28.788 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:42:28.788 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:42:28.789 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:42:28.789 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:28.789 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:42:28.789 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:42:28.789 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:42:28.789 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:28.789 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.765+0000 7ff1d0aad640 1 librados: starting msgr at 2024-09-16T03:42:28.789 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.765+0000 7ff1d0aad640 1 librados: starting objecter 2024-09-16T03:42:28.789 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.765+0000 7fa6ff288640 5 monclient: authenticate success, global_id 4220 2024-09-16T03:42:28.789 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.765+0000 7fa6ff288640 10 monclient: get_monmap_and_config success 2024-09-16T03:42:28.789 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.765+0000 7fa6ff288640 4 set_mon_vals no callback set 2024-09-16T03:42:28.790 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.765+0000 7fa6ff288640 10 monclient: shutdown 2024-09-16T03:42:28.790 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.765+0000 7fa6ff288640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:42:28.790 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.765+0000 7fa6ff288640 1 -- 172.21.15.42:0/3666849990 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa6f8072d40 msgr2=0x7fa6f813eac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:28.790 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.765+0000 7fa6ff288640 1 --2- 172.21.15.42:0/3666849990 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa6f8072d40 0x7fa6f813eac0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fa6f0009d90 tx=0x7fa6f001c670 comp rx=0 tx=0).stop 2024-09-16T03:42:28.790 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.765+0000 7ff1d0aad640 1 Processor -- start 2024-09-16T03:42:28.790 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.765+0000 7fa6ff288640 1 -- 172.21.15.42:0/3666849990 shutdown_connections 2024-09-16T03:42:28.791 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.765+0000 7fa6ff288640 1 --2- 172.21.15.42:0/3666849990 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa6f8072d40 0x7fa6f813eac0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:28.791 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.765+0000 7fa6ff288640 1 --2- 172.21.15.42:0/3666849990 >> v2:172.21.15.42:3300/0 conn(0x7fa6f809c920 0x7fa6f813c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:28.791 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.765+0000 7fa6ff288640 1 -- 172.21.15.42:0/3666849990 >> 172.21.15.42:0/3666849990 conn(0x7fa6f80a2420 msgr2=0x7fa6f80a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:42:28.791 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.765+0000 7fa6ff288640 1 -- 172.21.15.42:0/3666849990 shutdown_connections 2024-09-16T03:42:28.791 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.765+0000 7ff1d0aad640 1 -- start start 2024-09-16T03:42:28.791 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.765+0000 7ff1d0aad640 1 librados: setting wanted keys 2024-09-16T03:42:28.791 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.765+0000 7ff1d0aad640 1 librados: calling monclient init 2024-09-16T03:42:28.791 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.765+0000 7ff1d0aad640 10 monclient: init 2024-09-16T03:42:28.792 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.765+0000 7fa6ff288640 1 -- 172.21.15.42:0/3666849990 wait complete. 2024-09-16T03:42:28.793 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.765+0000 7ff1d0aad640 10 monclient: _reopen_session rank -1 2024-09-16T03:42:28.793 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.765+0000 7ff1d0aad640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T03:42:28.793 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.765+0000 7ff1d0aad640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff1cc0a39b0 0x7ff1cc130560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:28.794 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.765+0000 7ff1d0aad640 10 monclient(hunting): picked mon.noname-a con 0x7ff1cc0a39b0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:42:28.794 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.765+0000 7ff1d0aad640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff1cc0a3d90 0x7ff1cc130aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:28.794 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.765+0000 7ff1d0aad640 10 monclient(hunting): picked mon.noname-b con 0x7ff1cc0a3d90 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:42:28.794 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.765+0000 7ff1d0aad640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff1cc13c8c0 0x7ff1cc139dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:28.794 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.765+0000 7ff1ca59c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff1cc0a39b0 0x7ff1cc130560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:28.794 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.765+0000 7ff1ca59c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff1cc0a39b0 0x7ff1cc130560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:37582/0 (socket says 172.21.15.42:37582) 2024-09-16T03:42:28.794 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.765+0000 7ff1d0aad640 10 monclient(hunting): picked mon.noname-c con 0x7ff1cc13c8c0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:42:28.794 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.765+0000 7ff1ca59c640 1 -- 172.21.15.42:0/1615335561 learned_addr learned my addr 172.21.15.42:0/1615335561 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:42:28.794 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.765+0000 7ff1c9d9b640 1 --2- 172.21.15.42:0/1615335561 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff1cc0a3d90 0x7ff1cc130aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:28.794 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.765+0000 7ff1cad9d640 1 --2- 172.21.15.42:0/1615335561 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff1cc13c8c0 0x7ff1cc139dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:28.794 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.765+0000 7ff1d0aad640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:28.794 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.765+0000 7ff1d0aad640 1 -- 172.21.15.42:0/1615335561 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7ff1cc141da0 con 0x7ff1cc0a39b0 2024-09-16T03:42:28.795 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.765+0000 7ff1d0aad640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:28.795 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.765+0000 7ff1d0aad640 1 -- 172.21.15.42:0/1615335561 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff1cc141c20 con 0x7ff1cc0a3d90 2024-09-16T03:42:28.795 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.765+0000 7ff1d0aad640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:28.795 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.765+0000 7ff1d0aad640 1 -- 172.21.15.42:0/1615335561 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7ff1cc141f20 con 0x7ff1cc13c8c0 2024-09-16T03:42:28.795 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.765+0000 7ff1d0aad640 10 monclient(hunting): _renew_subs 2024-09-16T03:42:28.795 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.765+0000 7ff1d0aad640 10 monclient(hunting): authenticate will time out at 995.739133s 2024-09-16T03:42:28.795 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.765+0000 7ff1ca59c640 10 monclient(hunting): get_auth_request con 0x7ff1cc0a39b0 auth_method 0 2024-09-16T03:42:28.795 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.765+0000 7ff1ca59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:28.795 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.765+0000 7ff1ca59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:28.795 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.765+0000 7ff1ca59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:28.795 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.765+0000 7ff1c9d9b640 10 monclient(hunting): get_auth_request con 0x7ff1cc0a3d90 auth_method 0 2024-09-16T03:42:28.795 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.765+0000 7ff1c9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:28.796 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.765+0000 7ff1c9d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:28.796 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.765+0000 7ff1c9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:28.796 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.765+0000 7ff1cad9d640 10 monclient(hunting): get_auth_request con 0x7ff1cc13c8c0 auth_method 0 2024-09-16T03:42:28.796 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.765+0000 7ff1cad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:28.796 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.765+0000 7ff1cad9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:28.796 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.765+0000 7ff1cad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:28.796 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.765+0000 7fa6ff288640 10 monclient: build_initial_monmap 2024-09-16T03:42:28.796 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.765+0000 7fa6ff288640 10 monclient: monmap: 2024-09-16T03:42:28.796 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:42:28.797 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:42:28.797 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:42:28.797 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:42:28.797 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:42:28.797 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:28.797 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:42:28.797 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:42:28.797 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:42:28.797 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:28.797 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.765+0000 7fa6ff288640 1 librados: starting msgr at 2024-09-16T03:42:28.797 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.765+0000 7fa6ff288640 1 librados: starting objecter 2024-09-16T03:42:28.797 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.765+0000 7fa6ff288640 1 Processor -- start 2024-09-16T03:42:28.798 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.765+0000 7ff1ca59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:28.798 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.765+0000 7ff1ca59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:28.798 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.765+0000 7ff1ca59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:28.799 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.765+0000 7ff1c9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:28.799 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.765+0000 7ff1c9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:28.799 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.765+0000 7ff1c9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:28.799 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.765+0000 7ff1cad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:28.799 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.765+0000 7ff1cad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:28.799 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.765+0000 7ff1cad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:28.799 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.765+0000 7ff1ca59c640 10 monclient(hunting): handle_auth_done global_id 4260 payload 824 2024-09-16T03:42:28.799 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.765+0000 7ff1ca59c640 1 -- 172.21.15.42:0/1615335561 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff1cc13c8c0 msgr2=0x7ff1cc139dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:28.799 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.765+0000 7ff1ca59c640 1 --2- 172.21.15.42:0/1615335561 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff1cc13c8c0 0x7ff1cc139dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:28.799 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.765+0000 7ff1ca59c640 1 -- 172.21.15.42:0/1615335561 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff1cc0a3d90 msgr2=0x7ff1cc130aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:28.799 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.765+0000 7ff1ca59c640 1 --2- 172.21.15.42:0/1615335561 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff1cc0a3d90 0x7ff1cc130aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:28.799 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.765+0000 7ff1ca59c640 10 monclient: _finish_hunting 0 2024-09-16T03:42:28.799 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.765+0000 7ff1ca59c640 1 monclient: found mon.noname-a 2024-09-16T03:42:28.799 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.765+0000 7ff1ca59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:42:28.800 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.765+0000 7ff1ca59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:42:28.800 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.765+0000 7ff1ca59c640 1 -- 172.21.15.42:0/1615335561 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1cc13c420 con 0x7ff1cc0a39b0 2024-09-16T03:42:28.800 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.765+0000 7ff1ca59c640 1 --2- 172.21.15.42:0/1615335561 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff1cc0a39b0 0x7ff1cc130560 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7ff1bc00c500 tx=0x7ff1bc00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:28.800 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.765+0000 7ff1baffd640 10 client.?.objecter ms_handle_connect 0x7ff1cc0a39b0 2024-09-16T03:42:28.800 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.765+0000 7ff1baffd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:42:28.800 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.765+0000 7ff1baffd640 1 -- 172.21.15.42:0/1615335561 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1bc016020 con 0x7ff1cc0a39b0 2024-09-16T03:42:28.800 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.765+0000 7ff1baffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:28.800 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.765+0000 7ff1baffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:42:28.800 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.765+0000 7ff1baffd640 10 monclient: dump: 2024-09-16T03:42:28.800 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:28.800 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:28.800 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:28.800 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:28.801 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:28.801 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:28.801 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:28.801 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:28.801 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:28.801 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:28.801 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.765+0000 7ff1baffd640 10 monclient: _finish_auth 0 2024-09-16T03:42:28.801 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.765+0000 7ff1baffd640 10 monclient: _check_auth_tickets 2024-09-16T03:42:28.801 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.765+0000 7ff1baffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:42:28.801 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.765+0000 7ff1baffd640 1 -- 172.21.15.42:0/1615335561 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff1bc005220 con 0x7ff1cc0a39b0 2024-09-16T03:42:28.801 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.765+0000 7ff1baffd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:42:28.801 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.765+0000 7ff1baffd640 1 -- 172.21.15.42:0/1615335561 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1bc007030 con 0x7ff1cc0a39b0 2024-09-16T03:42:28.801 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.765+0000 7ff1baffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:28.801 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.765+0000 7ff1baffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:42:28.802 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.769+0000 7ff1c8d99640 4 set_mon_vals no callback set 2024-09-16T03:42:28.803 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.765+0000 7fa6ff288640 1 -- start start 2024-09-16T03:42:28.803 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.765+0000 7fa6ff288640 1 librados: setting wanted keys 2024-09-16T03:42:28.803 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.765+0000 7fa6ff288640 1 librados: calling monclient init 2024-09-16T03:42:28.803 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.765+0000 7fa6ff288640 10 monclient: init 2024-09-16T03:42:28.803 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.765+0000 7fa6ff288640 10 monclient: _reopen_session rank -1 2024-09-16T03:42:28.803 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.765+0000 7fa6ff288640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:42:28.803 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.765+0000 7fa6ff288640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa6f8072d40 0x7fa6f81d32d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:28.803 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.765+0000 7fa6ff288640 10 monclient(hunting): picked mon.noname-c con 0x7fa6f8072d40 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:42:28.803 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.769+0000 7fa6ff288640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa6f809c920 0x7fa6f8137610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:28.803 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.769+0000 7fa6ff288640 10 monclient(hunting): picked mon.noname-a con 0x7fa6f809c920 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:42:28.803 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.769+0000 7fa6ff288640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa6f80a1330 0x7fa6f806d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:28.803 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.769+0000 7fa6ff288640 10 monclient(hunting): picked mon.noname-b con 0x7fa6f80a1330 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:42:28.803 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.769+0000 7fa6ff288640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:28.804 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.769+0000 7fa6ff288640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fa6f8141ca0 con 0x7fa6f809c920 2024-09-16T03:42:28.804 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.769+0000 7fa6ff288640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:28.804 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.769+0000 7fa6ff288640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa6f8141b20 con 0x7fa6f80a1330 2024-09-16T03:42:28.804 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.769+0000 7fa6ff288640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:28.804 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.769+0000 7fa6ff288640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fa6f8141e20 con 0x7fa6f8072d40 2024-09-16T03:42:28.804 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.769+0000 7fa6ff288640 10 monclient(hunting): _renew_subs 2024-09-16T03:42:28.804 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.769+0000 7fa6ff288640 10 monclient(hunting): authenticate will time out at 995.740086s 2024-09-16T03:42:28.804 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.769+0000 7fa6fd024640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa6f8072d40 0x7fa6f81d32d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:28.804 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.769+0000 7fa6fd024640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa6f8072d40 0x7fa6f81d32d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:41276/0 (socket says 172.21.15.42:41276) 2024-09-16T03:42:28.804 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.769+0000 7fa6fd024640 1 -- 172.21.15.42:0/1767692205 learned_addr learned my addr 172.21.15.42:0/1767692205 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:42:28.804 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.769+0000 7fa6fd024640 10 monclient(hunting): get_auth_request con 0x7fa6f8072d40 auth_method 0 2024-09-16T03:42:28.804 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.769+0000 7fa6fd024640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:28.804 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.769+0000 7fa6fd024640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:28.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.769+0000 7fa6fd024640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:28.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.769+0000 7fa6fd024640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:28.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.769+0000 7fa6fd024640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:28.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.769+0000 7fa6fd024640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:28.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.769+0000 7fa6fc823640 1 --2- 172.21.15.42:0/1767692205 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa6f809c920 0x7fa6f8137610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:28.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.769+0000 7fa6fd825640 1 --2- 172.21.15.42:0/1767692205 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa6f80a1330 0x7fa6f806d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:28.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.769+0000 7fa6fc823640 10 monclient(hunting): get_auth_request con 0x7fa6f809c920 auth_method 0 2024-09-16T03:42:28.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.769+0000 7fa6fc823640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:28.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.769+0000 7fa6fc823640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:28.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.769+0000 7fa6fc823640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:28.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.769+0000 7fa6fd825640 10 monclient(hunting): get_auth_request con 0x7fa6f80a1330 auth_method 0 2024-09-16T03:42:28.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.769+0000 7fa6fd825640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:28.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.769+0000 7fa6fd825640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:28.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.769+0000 7fa6fd825640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:28.806 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.769+0000 7fa6fd024640 10 monclient(hunting): handle_auth_done global_id 4226 payload 824 2024-09-16T03:42:28.806 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.769+0000 7fa6fd024640 1 -- 172.21.15.42:0/1767692205 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa6f80a1330 msgr2=0x7fa6f806d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:28.806 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.769+0000 7fa6fd024640 1 --2- 172.21.15.42:0/1767692205 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa6f80a1330 0x7fa6f806d930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:28.806 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.769+0000 7fa6fd024640 1 -- 172.21.15.42:0/1767692205 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa6f809c920 msgr2=0x7fa6f8137610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:28.806 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.769+0000 7fa6fd024640 1 --2- 172.21.15.42:0/1767692205 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa6f809c920 0x7fa6f8137610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:28.806 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.769+0000 7fa6fd024640 10 monclient: _finish_hunting 0 2024-09-16T03:42:28.806 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.769+0000 7fa6fd024640 1 monclient: found mon.noname-c 2024-09-16T03:42:28.806 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.769+0000 7fa6fd024640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:42:28.806 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.769+0000 7fa6fd024640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:42:28.806 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.769+0000 7fa6fd024640 1 -- 172.21.15.42:0/1767692205 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6f81d41e0 con 0x7fa6f8072d40 2024-09-16T03:42:28.806 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.769+0000 7fa6fd024640 1 --2- 172.21.15.42:0/1767692205 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa6f8072d40 0x7fa6f81d32d0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fa6f400c810 tx=0x7fa6f400ccd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:28.807 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.765+0000 7ff1baffd640 10 monclient: dump: 2024-09-16T03:42:28.807 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:28.807 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:28.807 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:28.807 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:28.807 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:28.807 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:28.807 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:28.807 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:28.807 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:28.807 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:28.807 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.769+0000 7ff1d0aad640 5 monclient: authenticate success, global_id 4260 2024-09-16T03:42:28.807 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.769+0000 7ff1d0aad640 10 monclient: _renew_subs 2024-09-16T03:42:28.808 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.769+0000 7ff1d0aad640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:42:28.808 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.769+0000 7ff1d0aad640 1 -- 172.21.15.42:0/1615335561 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1cc09e1e0 con 0x7ff1cc0a39b0 2024-09-16T03:42:28.808 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.769+0000 7ff1d0aad640 10 client.4260.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:42:28.808 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.769+0000 7ff1d0aad640 10 monclient: _renew_subs 2024-09-16T03:42:28.808 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.769+0000 7ff1d0aad640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:42:28.808 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.769+0000 7ff1d0aad640 1 -- 172.21.15.42:0/1615335561 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1cc09df00 con 0x7ff1cc0a39b0 2024-09-16T03:42:28.808 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.769+0000 7ff1d0aad640 1 librados: init done 2024-09-16T03:42:28.808 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.769+0000 7fa6edffb640 10 client.?.objecter ms_handle_connect 0x7fa6f8072d40 2024-09-16T03:42:28.808 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.769+0000 7fa6edffb640 10 client.?.objecter resend_mon_ops 2024-09-16T03:42:28.808 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.769+0000 7fa6edffb640 1 -- 172.21.15.42:0/1767692205 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6f4018070 con 0x7fa6f8072d40 2024-09-16T03:42:28.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.769+0000 7fa6edffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:28.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.769+0000 7fa6edffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:42:28.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.769+0000 7fa6edffb640 10 monclient: dump: 2024-09-16T03:42:28.809 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:28.809 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:28.809 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:28.809 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:28.809 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:28.809 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:28.809 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:28.809 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:28.809 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:28.809 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:28.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.769+0000 7fa6edffb640 10 monclient: _finish_auth 0 2024-09-16T03:42:28.810 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.769+0000 7fa6edffb640 10 monclient: _check_auth_tickets 2024-09-16T03:42:28.810 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.769+0000 7fa6edffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:42:28.810 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.769+0000 7fa6edffb640 1 -- 172.21.15.42:0/1767692205 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa6f40074e0 con 0x7fa6f8072d40 2024-09-16T03:42:28.810 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.769+0000 7fa6edffb640 10 monclient: handle_config config(0 keys) 2024-09-16T03:42:28.810 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.769+0000 7fa6edffb640 1 -- 172.21.15.42:0/1767692205 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6f40049f0 con 0x7fa6f8072d40 2024-09-16T03:42:28.810 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.769+0000 7fa6edffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:28.810 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.769+0000 7fa6edffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:42:28.810 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.769+0000 7fa6edffb640 10 monclient: dump: 2024-09-16T03:42:28.810 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:28.810 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:28.810 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:28.810 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:28.811 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:28.811 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:28.811 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:28.811 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:28.811 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:28.811 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:28.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.769+0000 7fa6ef7fe640 4 set_mon_vals no callback set 2024-09-16T03:42:28.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.769+0000 7fa6ff288640 5 monclient: authenticate success, global_id 4226 2024-09-16T03:42:28.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.769+0000 7fa6ff288640 10 monclient: _renew_subs 2024-09-16T03:42:28.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.769+0000 7fa6ff288640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:42:28.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.769+0000 7fa6ff288640 1 -- 172.21.15.42:0/1767692205 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6f80715e0 con 0x7fa6f8072d40 2024-09-16T03:42:28.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.769+0000 7fa6ff288640 10 client.4226.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:42:28.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.769+0000 7fa6ff288640 10 monclient: _renew_subs 2024-09-16T03:42:28.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.769+0000 7fa6ff288640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:42:28.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.769+0000 7fa6ff288640 1 -- 172.21.15.42:0/1767692205 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6f8073a30 con 0x7fa6f8072d40 2024-09-16T03:42:28.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.769+0000 7fa6ff288640 1 librados: init done 2024-09-16T03:42:28.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.769+0000 7ff1d0aad640 10 client.4260.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:42:28.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.769+0000 7ff1d0aad640 15 client.4260.objecter _session_command_op_assign -1 1 2024-09-16T03:42:28.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.769+0000 7ff1d0aad640 20 client.4260.objecter _get_session osd=-1 returning homeless 2024-09-16T03:42:28.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.769+0000 7ff1d0aad640 10 client.4260.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:42:28.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.769+0000 7ff1d0aad640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:42:28.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.769+0000 7ff1d0aad640 1 -- 172.21.15.42:0/1615335561 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7ff1cc0a5750 con 0x7ff1cc0a39b0 2024-09-16T03:42:28.813 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.769+0000 7fa6ff288640 10 client.4226.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:42:28.813 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.769+0000 7fa6ff288640 15 client.4226.objecter _session_command_op_assign -1 1 2024-09-16T03:42:28.813 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.769+0000 7fa6ff288640 20 client.4226.objecter _get_session osd=-1 returning homeless 2024-09-16T03:42:28.813 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.769+0000 7fa6ff288640 10 client.4226.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:42:28.813 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.769+0000 7fa6ff288640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:42:28.813 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.769+0000 7fa6ff288640 1 -- 172.21.15.42:0/1767692205 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fa6f80a1710 con 0x7fa6f8072d40 2024-09-16T03:42:28.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.769+0000 7ff1baffd640 1 -- 172.21.15.42:0/1615335561 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7ff1bc0071d0 con 0x7ff1cc0a39b0 2024-09-16T03:42:28.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.769+0000 7ff1baffd640 1 --2- 172.21.15.42:0/1615335561 >> v2:172.21.15.154:6812/254845342 conn(0x7ff1ac037840 0x7ff1ac039d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:28.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.769+0000 7ff1baffd640 1 -- 172.21.15.42:0/1615335561 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 3562+0+0 (secure 0 0 0) 0x7ff1bc047be0 con 0x7ff1cc0a39b0 2024-09-16T03:42:28.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.769+0000 7ff1baffd640 10 client.4260.objecter ms_dispatch 0x7ff1cc000ba0 osd_map(18..18 src has 1..18) 2024-09-16T03:42:28.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.769+0000 7ff1baffd640 3 client.4260.objecter handle_osd_map got epochs [18,18] > 0 2024-09-16T03:42:28.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.769+0000 7ff1baffd640 3 client.4260.objecter handle_osd_map decoding full epoch 18 2024-09-16T03:42:28.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.769+0000 7ff1baffd640 10 client.4260.objecter checking command 1 2024-09-16T03:42:28.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.769+0000 7ff1baffd640 1 --2- 172.21.15.42:0/1615335561 >> v2:172.21.15.42:6804/263835041 conn(0x7ff1ac03d350 0x7ff1ac03f790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:28.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.769+0000 7ff1baffd640 20 client.4260.objecter _get_session s=0x7ff1ac03cca0 osd=2 3 2024-09-16T03:42:28.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.769+0000 7ff1baffd640 20 client.4260.objecter put_session s=0x7ff1ac03cca0 osd=2 3 2024-09-16T03:42:28.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.769+0000 7ff1baffd640 15 client.4260.objecter _session_command_op_remove -1 1 2024-09-16T03:42:28.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.769+0000 7ff1baffd640 20 client.4260.objecter _get_session s=0x7ff1ac03cca0 osd=2 3 2024-09-16T03:42:28.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.769+0000 7ff1baffd640 20 client.4260.objecter get_session s=0x7ff1ac03cca0 osd=2 3 2024-09-16T03:42:28.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.769+0000 7ff1baffd640 15 client.4260.objecter _session_command_op_assign 2 1 2024-09-16T03:42:28.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.769+0000 7ff1baffd640 20 client.4260.objecter put_session s=0x7ff1ac03cca0 osd=2 4 2024-09-16T03:42:28.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.769+0000 7ff1baffd640 10 client.4260.objecter _send_command 1 2024-09-16T03:42:28.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.769+0000 7ff1baffd640 1 -- 172.21.15.42:0/1615335561 --> v2:172.21.15.42:6804/263835041 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff1bc010e50 con 0x7ff1ac03d350 2024-09-16T03:42:28.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.769+0000 7ff1baffd640 20 client.4260.objecter dump_active .. 0 homeless 2024-09-16T03:42:28.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.769+0000 7ff1baffd640 1 -- 172.21.15.42:0/1615335561 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_get_version_reply(handle=1 version=18) ==== 24+0+0 (secure 0 0 0) 0x7ff1bc04c220 con 0x7ff1cc0a39b0 2024-09-16T03:42:28.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.769+0000 7ff1baffd640 10 monclient: handle_get_version_reply finishing 1 version 18 2024-09-16T03:42:28.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.769+0000 7fa6edffb640 1 -- 172.21.15.42:0/1767692205 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fa6f4004b90 con 0x7fa6f8072d40 2024-09-16T03:42:28.816 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.769+0000 7ff1cad9d640 1 --2- 172.21.15.42:0/1615335561 >> v2:172.21.15.42:6804/263835041 conn(0x7ff1ac03d350 0x7ff1ac03f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:28.816 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.769+0000 7ff1cad9d640 10 monclient: get_auth_request con 0x7ff1ac03d350 auth_method 0 2024-09-16T03:42:28.816 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.769+0000 7ff1c9d9b640 1 --2- 172.21.15.42:0/1615335561 >> v2:172.21.15.154:6812/254845342 conn(0x7ff1ac037840 0x7ff1ac039d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:28.816 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.769+0000 7ff1c9d9b640 10 monclient: get_auth_request con 0x7ff1ac037840 auth_method 0 2024-09-16T03:42:28.817 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.769+0000 7fa6edffb640 1 --2- 172.21.15.42:0/1767692205 >> v2:172.21.15.154:6812/254845342 conn(0x7fa6dc037810 0x7fa6dc039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:28.817 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.769+0000 7fa6edffb640 1 -- 172.21.15.42:0/1767692205 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(18..18 src has 1..18) ==== 3562+0+0 (secure 0 0 0) 0x7fa6f4047e80 con 0x7fa6f8072d40 2024-09-16T03:42:28.817 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.769+0000 7fa6edffb640 10 client.4226.objecter ms_dispatch 0x7fa6f8000ba0 osd_map(18..18 src has 1..18) 2024-09-16T03:42:28.817 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.769+0000 7fa6edffb640 3 client.4226.objecter handle_osd_map got epochs [18,18] > 0 2024-09-16T03:42:28.817 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.769+0000 7fa6edffb640 3 client.4226.objecter handle_osd_map decoding full epoch 18 2024-09-16T03:42:28.817 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.769+0000 7fa6edffb640 10 client.4226.objecter checking command 1 2024-09-16T03:42:28.817 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.769+0000 7fa6edffb640 1 --2- 172.21.15.42:0/1767692205 >> v2:172.21.15.154:6808/3026127822 conn(0x7fa6dc03d320 0x7fa6dc03f760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:28.817 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.769+0000 7fa6edffb640 20 client.4226.objecter _get_session s=0x7fa6dc03cc70 osd=5 3 2024-09-16T03:42:28.817 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.769+0000 7fa6edffb640 20 client.4226.objecter put_session s=0x7fa6dc03cc70 osd=5 3 2024-09-16T03:42:28.817 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.769+0000 7fa6edffb640 15 client.4226.objecter _session_command_op_remove -1 1 2024-09-16T03:42:28.818 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.769+0000 7fa6edffb640 20 client.4226.objecter _get_session s=0x7fa6dc03cc70 osd=5 3 2024-09-16T03:42:28.818 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.769+0000 7fa6edffb640 20 client.4226.objecter get_session s=0x7fa6dc03cc70 osd=5 3 2024-09-16T03:42:28.818 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.769+0000 7fa6edffb640 15 client.4226.objecter _session_command_op_assign 5 1 2024-09-16T03:42:28.818 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.769+0000 7fa6edffb640 20 client.4226.objecter put_session s=0x7fa6dc03cc70 osd=5 4 2024-09-16T03:42:28.818 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.769+0000 7fa6edffb640 10 client.4226.objecter _send_command 1 2024-09-16T03:42:28.818 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.769+0000 7fa6edffb640 1 -- 172.21.15.42:0/1767692205 --> v2:172.21.15.154:6808/3026127822 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa6dc03fdf0 con 0x7fa6dc03d320 2024-09-16T03:42:28.818 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.769+0000 7fa6edffb640 20 client.4226.objecter dump_active .. 0 homeless 2024-09-16T03:42:28.818 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.769+0000 7fa6edffb640 1 -- 172.21.15.42:0/1767692205 <== mon.2 v2:172.21.15.42:3301/0 6 ==== mon_get_version_reply(handle=1 version=18) ==== 24+0+0 (secure 0 0 0) 0x7fa6f404a400 con 0x7fa6f8072d40 2024-09-16T03:42:28.818 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.769+0000 7fa6edffb640 10 monclient: handle_get_version_reply finishing 1 version 18 2024-09-16T03:42:28.818 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.769+0000 7fa6fc823640 1 --2- 172.21.15.42:0/1767692205 >> v2:172.21.15.154:6812/254845342 conn(0x7fa6dc037810 0x7fa6dc039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:28.818 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.769+0000 7fa6fc823640 10 monclient: get_auth_request con 0x7fa6dc037810 auth_method 0 2024-09-16T03:42:28.819 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.769+0000 7ff1baffd640 10 client.4260.objecter ms_handle_connect 0x7ff1ac03d350 2024-09-16T03:42:28.819 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.769+0000 7ff1cad9d640 1 --2- 172.21.15.42:0/1615335561 >> v2:172.21.15.42:6804/263835041 conn(0x7ff1ac03d350 0x7ff1ac03f790 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:28.819 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.769+0000 7fa6fd825640 1 --2- 172.21.15.42:0/1767692205 >> v2:172.21.15.154:6808/3026127822 conn(0x7fa6dc03d320 0x7fa6dc03f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:28.819 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.769+0000 7fa6fd825640 10 monclient: get_auth_request con 0x7fa6dc03d320 auth_method 0 2024-09-16T03:42:28.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.769+0000 7ff1baffd640 10 client.4260.objecter ms_handle_connect 0x7ff1ac037840 2024-09-16T03:42:28.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.769+0000 7ff1c9d9b640 1 --2- 172.21.15.42:0/1615335561 >> v2:172.21.15.154:6812/254845342 conn(0x7ff1ac037840 0x7ff1ac039d00 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7ff1cc13b670 tx=0x7ff1c0009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:28.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.773+0000 7ff1baffd640 1 -- 172.21.15.42:0/1615335561 <== osd.2 v2:172.21.15.42:6804/263835041 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff1bc010e50 con 0x7ff1ac03d350 2024-09-16T03:42:28.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.773+0000 7ff1baffd640 10 client.4260.objecter ms_dispatch 0x7ff1cc000ba0 command_reply(tid 1: 0 ) 2024-09-16T03:42:28.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.773+0000 7ff1baffd640 10 client.4260.objecter _finish_command 1 = system:0 2024-09-16T03:42:28.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.773+0000 7ff1baffd640 20 client.4260.objecter put_session s=0x7ff1ac03cca0 osd=2 4 2024-09-16T03:42:28.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.773+0000 7ff1baffd640 15 client.4260.objecter _session_command_op_remove 2 1 2024-09-16T03:42:28.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.773+0000 7fa6fc823640 1 --2- 172.21.15.42:0/1767692205 >> v2:172.21.15.154:6812/254845342 conn(0x7fa6dc037810 0x7fa6dc039cd0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fa6e8009580 tx=0x7fa6e8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:28.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.773+0000 7fa6edffb640 10 client.4226.objecter ms_handle_connect 0x7fa6dc037810 2024-09-16T03:42:28.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.773+0000 7fa6edffb640 10 client.4226.objecter ms_handle_connect 0x7fa6dc03d320 2024-09-16T03:42:28.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.773+0000 7fa6fd825640 1 --2- 172.21.15.42:0/1767692205 >> v2:172.21.15.154:6808/3026127822 conn(0x7fa6dc03d320 0x7fa6dc03f760 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:28.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.773+0000 7fa6edffb640 1 -- 172.21.15.42:0/1767692205 <== osd.5 v2:172.21.15.154:6808/3026127822 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa6dc03fdf0 con 0x7fa6dc03d320 2024-09-16T03:42:28.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.773+0000 7fa6edffb640 10 client.4226.objecter ms_dispatch 0x7fa6f8000ba0 command_reply(tid 1: 0 ) 2024-09-16T03:42:28.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.773+0000 7fa6edffb640 10 client.4226.objecter _finish_command 1 = system:0 2024-09-16T03:42:28.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.773+0000 7fa6edffb640 20 client.4226.objecter put_session s=0x7fa6dc03cc70 osd=5 4 2024-09-16T03:42:28.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.773+0000 7fa6edffb640 15 client.4226.objecter _session_command_op_remove 5 1 2024-09-16T03:42:28.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.785+0000 7ff1d0aad640 10 client.4260.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-09-16T03:42:28.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.785+0000 7ff1d0aad640 15 client.4260.objecter _session_command_op_assign -1 2 2024-09-16T03:42:28.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.785+0000 7ff1d0aad640 20 client.4260.objecter _get_session s=0x7ff1ac03cca0 osd=2 3 2024-09-16T03:42:28.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.785+0000 7ff1d0aad640 20 client.4260.objecter put_session s=0x7ff1ac03cca0 osd=2 3 2024-09-16T03:42:28.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.785+0000 7ff1d0aad640 20 client.4260.objecter _get_session s=0x7ff1ac03cca0 osd=2 3 2024-09-16T03:42:28.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.785+0000 7ff1d0aad640 15 client.4260.objecter _session_command_op_remove -1 2 2024-09-16T03:42:28.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.785+0000 7ff1d0aad640 20 client.4260.objecter get_session s=0x7ff1ac03cca0 osd=2 3 2024-09-16T03:42:28.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.785+0000 7ff1d0aad640 15 client.4260.objecter _session_command_op_assign 2 2 2024-09-16T03:42:28.823 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.785+0000 7ff1d0aad640 20 client.4260.objecter put_session s=0x7ff1ac03cca0 osd=2 4 2024-09-16T03:42:28.823 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.785+0000 7ff1d0aad640 10 client.4260.objecter _send_command 2 2024-09-16T03:42:28.823 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.785+0000 7ff1d0aad640 1 -- 172.21.15.42:0/1615335561 --> v2:172.21.15.42:6804/263835041 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7ff1cc09cab0 con 0x7ff1ac03d350 2024-09-16T03:42:28.823 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.785+0000 7ff1baffd640 1 -- 172.21.15.42:0/1615335561 <== osd.2 v2:172.21.15.42:6804/263835041 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7ff1cc09cab0 con 0x7ff1ac03d350 2024-09-16T03:42:28.823 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.785+0000 7ff1baffd640 10 client.4260.objecter ms_dispatch 0x7ff1cc000ba0 command_reply(tid 2: 0 ) 2024-09-16T03:42:28.823 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.785+0000 7ff1baffd640 10 client.4260.objecter _finish_command 2 = system:0 2024-09-16T03:42:28.823 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.785+0000 7ff1baffd640 20 client.4260.objecter put_session s=0x7ff1ac03cca0 osd=2 4 2024-09-16T03:42:28.823 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.785+0000 7ff1baffd640 15 client.4260.objecter _session_command_op_remove 2 2 2024-09-16T03:42:28.824 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.785+0000 7ff1d0aad640 10 librados: watch_flush enter 2024-09-16T03:42:28.824 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.785+0000 7ff1d0aad640 10 librados: watch_flush exit 2024-09-16T03:42:28.825 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.785+0000 7ff1d0aad640 10 client.4260.objecter close_session for osd.2 2024-09-16T03:42:28.825 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.785+0000 7ff1d0aad640 1 -- 172.21.15.42:0/1615335561 >> v2:172.21.15.42:6804/263835041 conn(0x7ff1ac03d350 msgr2=0x7ff1ac03f790 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:28.825 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.785+0000 7ff1d0aad640 1 --2- 172.21.15.42:0/1615335561 >> v2:172.21.15.42:6804/263835041 conn(0x7ff1ac03d350 0x7ff1ac03f790 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:28.825 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.785+0000 7ff1d0aad640 20 client.4260.objecter put_session s=0x7ff1ac03cca0 osd=2 1 2024-09-16T03:42:28.825 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.785+0000 7ff1d0aad640 20 client.4260.objecter shutdown clearing up homeless session... 2024-09-16T03:42:28.825 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.785+0000 7ff1d0aad640 10 client.4260.objecter successfully canceled tick 2024-09-16T03:42:28.825 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.785+0000 7ff1d0aad640 1 -- 172.21.15.42:0/1615335561 >> v2:172.21.15.154:6812/254845342 conn(0x7ff1ac037840 msgr2=0x7ff1ac039d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:28.825 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.785+0000 7ff1d0aad640 1 --2- 172.21.15.42:0/1615335561 >> v2:172.21.15.154:6812/254845342 conn(0x7ff1ac037840 0x7ff1ac039d00 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7ff1cc13b670 tx=0x7ff1c0009190 comp rx=0 tx=0).stop 2024-09-16T03:42:28.825 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.785+0000 7ff1d0aad640 10 monclient: shutdown 2024-09-16T03:42:28.825 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.785+0000 7ff1d0aad640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:42:28.825 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.785+0000 7ff1d0aad640 1 -- 172.21.15.42:0/1615335561 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff1cc0a39b0 msgr2=0x7ff1cc130560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:28.825 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.785+0000 7ff1d0aad640 1 --2- 172.21.15.42:0/1615335561 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff1cc0a39b0 0x7ff1cc130560 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7ff1bc00c500 tx=0x7ff1bc00c9c0 comp rx=0 tx=0).stop 2024-09-16T03:42:28.826 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.785+0000 7ff1cad9d640 1 -- 172.21.15.42:0/1615335561 reap_dead start 2024-09-16T03:42:28.826 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.785+0000 7ff1d0aad640 1 -- 172.21.15.42:0/1615335561 shutdown_connections 2024-09-16T03:42:28.826 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.785+0000 7ff1d0aad640 1 -- 172.21.15.42:0/1615335561 >> 172.21.15.42:0/1615335561 conn(0x7ff1cc0a9fb0 msgr2=0x7ff1cc0a8bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:42:28.826 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.789+0000 7ff1d0aad640 1 -- 172.21.15.42:0/1615335561 shutdown_connections 2024-09-16T03:42:28.826 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.789+0000 7ff1d0aad640 1 -- 172.21.15.42:0/1615335561 wait complete. 2024-09-16T03:42:28.826 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.789+0000 7ff1d0aad640 1 librados: shutdown 2024-09-16T03:42:28.826 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.789+0000 7fa6ff288640 10 client.4226.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-09-16T03:42:28.826 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.789+0000 7fa6ff288640 15 client.4226.objecter _session_command_op_assign -1 2 2024-09-16T03:42:28.826 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.789+0000 7fa6ff288640 20 client.4226.objecter _get_session s=0x7fa6dc03cc70 osd=5 3 2024-09-16T03:42:28.826 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.789+0000 7fa6ff288640 20 client.4226.objecter put_session s=0x7fa6dc03cc70 osd=5 3 2024-09-16T03:42:28.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.789+0000 7fa6ff288640 20 client.4226.objecter _get_session s=0x7fa6dc03cc70 osd=5 3 2024-09-16T03:42:28.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.789+0000 7fa6ff288640 15 client.4226.objecter _session_command_op_remove -1 2 2024-09-16T03:42:28.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.789+0000 7fa6ff288640 20 client.4226.objecter get_session s=0x7fa6dc03cc70 osd=5 3 2024-09-16T03:42:28.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.789+0000 7fa6ff288640 15 client.4226.objecter _session_command_op_assign 5 2 2024-09-16T03:42:28.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.789+0000 7fa6ff288640 20 client.4226.objecter put_session s=0x7fa6dc03cc70 osd=5 4 2024-09-16T03:42:28.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.789+0000 7fa6ff288640 10 client.4226.objecter _send_command 2 2024-09-16T03:42:28.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.789+0000 7fa6ff288640 1 -- 172.21.15.42:0/1767692205 --> v2:172.21.15.154:6808/3026127822 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fa6f8137f80 con 0x7fa6dc03d320 2024-09-16T03:42:28.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.789+0000 7fa6edffb640 1 -- 172.21.15.42:0/1767692205 <== osd.5 v2:172.21.15.154:6808/3026127822 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fa6f8137f80 con 0x7fa6dc03d320 2024-09-16T03:42:28.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.789+0000 7fa6edffb640 10 client.4226.objecter ms_dispatch 0x7fa6f8000ba0 command_reply(tid 2: 0 ) 2024-09-16T03:42:28.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.789+0000 7fa6edffb640 10 client.4226.objecter _finish_command 2 = system:0 2024-09-16T03:42:28.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.789+0000 7fa6edffb640 20 client.4226.objecter put_session s=0x7fa6dc03cc70 osd=5 4 2024-09-16T03:42:28.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.789+0000 7fa6edffb640 15 client.4226.objecter _session_command_op_remove 5 2 2024-09-16T03:42:28.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.789+0000 7fa6ff288640 10 librados: watch_flush enter 2024-09-16T03:42:28.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.789+0000 7fa6ff288640 10 librados: watch_flush exit 2024-09-16T03:42:28.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.789+0000 7fa6ff288640 10 client.4226.objecter close_session for osd.5 2024-09-16T03:42:28.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.789+0000 7fa6ff288640 1 -- 172.21.15.42:0/1767692205 >> v2:172.21.15.154:6808/3026127822 conn(0x7fa6dc03d320 msgr2=0x7fa6dc03f760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:28.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.789+0000 7fa6ff288640 1 --2- 172.21.15.42:0/1767692205 >> v2:172.21.15.154:6808/3026127822 conn(0x7fa6dc03d320 0x7fa6dc03f760 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:28.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.789+0000 7fa6ff288640 20 client.4226.objecter put_session s=0x7fa6dc03cc70 osd=5 1 2024-09-16T03:42:28.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.789+0000 7fa6ff288640 20 client.4226.objecter shutdown clearing up homeless session... 2024-09-16T03:42:28.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.789+0000 7fa6ff288640 10 client.4226.objecter successfully canceled tick 2024-09-16T03:42:28.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.789+0000 7fa6ff288640 1 -- 172.21.15.42:0/1767692205 >> v2:172.21.15.154:6812/254845342 conn(0x7fa6dc037810 msgr2=0x7fa6dc039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:28.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.789+0000 7fa6ff288640 1 --2- 172.21.15.42:0/1767692205 >> v2:172.21.15.154:6812/254845342 conn(0x7fa6dc037810 0x7fa6dc039cd0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fa6e8009580 tx=0x7fa6e8008040 comp rx=0 tx=0).stop 2024-09-16T03:42:28.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.789+0000 7fa6ff288640 10 monclient: shutdown 2024-09-16T03:42:28.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.789+0000 7fa6ff288640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:42:28.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.789+0000 7fa6ff288640 1 -- 172.21.15.42:0/1767692205 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa6f8072d40 msgr2=0x7fa6f81d32d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:28.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.789+0000 7fa6ff288640 1 --2- 172.21.15.42:0/1767692205 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa6f8072d40 0x7fa6f81d32d0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fa6f400c810 tx=0x7fa6f400ccd0 comp rx=0 tx=0).stop 2024-09-16T03:42:28.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.789+0000 7fa6fd825640 1 -- 172.21.15.42:0/1767692205 reap_dead start 2024-09-16T03:42:28.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.789+0000 7fa6ff288640 1 -- 172.21.15.42:0/1767692205 shutdown_connections 2024-09-16T03:42:28.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.789+0000 7fa6ff288640 1 -- 172.21.15.42:0/1767692205 >> 172.21.15.42:0/1767692205 conn(0x7fa6f80a2420 msgr2=0x7fa6f809e620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:42:28.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.789+0000 7fa6ff288640 1 -- 172.21.15.42:0/1767692205 shutdown_connections 2024-09-16T03:42:28.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.789+0000 7fa6ff288640 1 -- 172.21.15.42:0/1767692205 wait complete. 2024-09-16T03:42:28.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.789+0000 7fa6ff288640 1 librados: shutdown 2024-09-16T03:42:28.831 INFO:teuthology.orchestra.run.smithi042.stdout:47244640260 2024-09-16T03:42:28.832 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-16T03:42:28.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.809+0000 7fcfee6aa640 10 monclient: get_monmap_and_config 2024-09-16T03:42:28.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.809+0000 7fcfee6aa640 10 monclient: build_initial_monmap 2024-09-16T03:42:28.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.809+0000 7fcfee6aa640 10 monclient: monmap: 2024-09-16T03:42:28.833 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:42:28.833 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:28.833 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:42:28.814185+0000 2024-09-16T03:42:28.833 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:42:28.814185+0000 2024-09-16T03:42:28.833 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:42:28.833 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:28.833 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:42:28.834 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:42:28.834 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:42:28.834 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:42:28.834 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:42:28.834 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:28.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.809+0000 7fcfee6aa640 1 Processor -- start 2024-09-16T03:42:28.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.809+0000 7fcfee6aa640 1 -- start start 2024-09-16T03:42:28.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.809+0000 7fcfee6aa640 10 monclient: init 2024-09-16T03:42:28.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.809+0000 7faa466ad640 10 monclient: get_monmap_and_config 2024-09-16T03:42:28.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.809+0000 7faa466ad640 10 monclient: build_initial_monmap 2024-09-16T03:42:28.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.809+0000 7faa466ad640 10 monclient: monmap: 2024-09-16T03:42:28.835 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:42:28.835 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:28.835 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:42:28.815485+0000 2024-09-16T03:42:28.835 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:42:28.815485+0000 2024-09-16T03:42:28.835 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:42:28.835 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:28.835 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:42:28.835 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:42:28.836 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:42:28.836 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:42:28.836 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:42:28.836 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:28.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.809+0000 7fcfee6aa640 10 monclient: _reopen_session rank -1 2024-09-16T03:42:28.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.809+0000 7fcfee6aa640 10 monclient: _add_conns ranks=[3,4,1,0,2] 2024-09-16T03:42:28.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.809+0000 7fcfee6aa640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fcfe80a39b0 0x7fcfe8133b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:28.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.809+0000 7fcfee6aa640 10 monclient(hunting): picked mon.noname-a con 0x7fcfe80a39b0 addr v2:172.21.15.42:3300/0 2024-09-16T03:42:28.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.809+0000 7fcfee6aa640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fcfe80a3d90 0x7fcfe8137000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:28.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.809+0000 7fcfee6aa640 10 monclient(hunting): picked mon.noname-c con 0x7fcfe80a3d90 addr v2:172.21.15.154:3300/0 2024-09-16T03:42:28.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.809+0000 7fcfee6aa640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fcfe8137540 addr v1:172.21.15.154:6789/0 2024-09-16T03:42:28.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.809+0000 7fcfee6aa640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcfe809c140 con 0x7fcfe8137540 2024-09-16T03:42:28.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.809+0000 7fcfee6aa640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:28.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.809+0000 7fcfee6aa640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7fcfe809c420 con 0x7fcfe80a39b0 2024-09-16T03:42:28.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.809+0000 7fcfe7fff640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fcfe80a39b0 0x7fcfe8133b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:28.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.809+0000 7fcfee6aa640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:28.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.809+0000 7fcfee6aa640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fcfe806dd60 con 0x7fcfe80a3d90 2024-09-16T03:42:28.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.809+0000 7fcfe7fff640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fcfe80a39b0 0x7fcfe8133b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:37608/0 (socket says 172.21.15.42:37608) 2024-09-16T03:42:28.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.809+0000 7fcfe7fff640 1 -- 172.21.15.42:0/115499394 learned_addr learned my addr 172.21.15.42:0/115499394 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:42:28.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.809+0000 7fcfee6aa640 10 monclient(hunting): _renew_subs 2024-09-16T03:42:28.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.809+0000 7fcfee6aa640 10 monclient(hunting): authenticate will time out at 995.783892s 2024-09-16T03:42:28.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.809+0000 7fcfe7fff640 10 monclient(hunting): get_auth_request con 0x7fcfe80a39b0 auth_method 0 2024-09-16T03:42:28.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.809+0000 7fcfe7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:28.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.809+0000 7fcfe7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:28.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.809+0000 7fcfe7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:28.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.813+0000 7faa466ad640 1 Processor -- start 2024-09-16T03:42:28.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.813+0000 7fcfe7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:28.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.813+0000 7fcfe7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:28.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.813+0000 7fcfe7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:28.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.813+0000 7faa466ad640 1 -- start start 2024-09-16T03:42:28.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.813+0000 7faa466ad640 10 monclient: init 2024-09-16T03:42:28.841 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.813+0000 7fcfe7fff640 10 monclient(hunting): handle_auth_done global_id 4266 payload 274 2024-09-16T03:42:28.841 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.813+0000 7fcfe7fff640 1 -- 172.21.15.42:0/115499394 >> v2:172.21.15.154:3300/0 conn(0x7fcfe80a3d90 msgr2=0x7fcfe8137000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:42:28.841 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.813+0000 7fcfe7fff640 1 --2- 172.21.15.42:0/115499394 >> v2:172.21.15.154:3300/0 conn(0x7fcfe80a3d90 0x7fcfe8137000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:28.841 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.813+0000 7fcfe7fff640 1 -- 172.21.15.42:0/115499394 >> v1:172.21.15.154:6789/0 conn(0x7fcfe8137540 legacy=0x7fcfe813e390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:42:28.841 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.813+0000 7fcfe7fff640 10 monclient: _finish_hunting 0 2024-09-16T03:42:28.841 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.813+0000 7fcfe7fff640 1 monclient: found mon.noname-a 2024-09-16T03:42:28.841 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.813+0000 7fcfe7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:42:28.841 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.813+0000 7fcfe7fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:42:28.841 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.813+0000 7fcfe7fff640 1 -- 172.21.15.42:0/115499394 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfe80a5e20 con 0x7fcfe80a39b0 2024-09-16T03:42:28.842 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.813+0000 7fcff6396640 10 monclient: get_monmap_and_config 2024-09-16T03:42:28.842 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.813+0000 7fcff6396640 10 monclient: build_initial_monmap 2024-09-16T03:42:28.842 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.813+0000 7fcff6396640 10 monclient: monmap: 2024-09-16T03:42:28.842 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:42:28.842 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:28.842 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:42:28.816816+0000 2024-09-16T03:42:28.842 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:42:28.816816+0000 2024-09-16T03:42:28.842 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:42:28.843 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:28.843 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:42:28.843 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:42:28.843 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:42:28.843 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:42:28.843 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:42:28.843 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:28.843 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.813+0000 7faa466ad640 10 monclient: _reopen_session rank -1 2024-09-16T03:42:28.843 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.813+0000 7faa466ad640 10 monclient: _add_conns ranks=[2,1,4,3,0] 2024-09-16T03:42:28.843 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.813+0000 7faa466ad640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7faa400a67e0 0x7faa400a6bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:28.843 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.813+0000 7faa466ad640 10 monclient(hunting): picked mon.noname-b con 0x7faa400a67e0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:42:28.844 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.813+0000 7faa466ad640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7faa40072560 addr v1:172.21.15.154:6789/0 2024-09-16T03:42:28.844 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.813+0000 7faa466ad640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7faa400a7100 0x7faa401357a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:28.844 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.813+0000 7faa466ad640 10 monclient(hunting): picked mon.noname-c con 0x7faa400a7100 addr v2:172.21.15.154:3300/0 2024-09-16T03:42:28.844 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.813+0000 7faa466ad640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa40071b80 con 0x7faa40072560 2024-09-16T03:42:28.844 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.813+0000 7faa466ad640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:28.844 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.813+0000 7faa466ad640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7faa40072ed0 con 0x7faa400a7100 2024-09-16T03:42:28.844 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.813+0000 7faa466ad640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:28.844 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.813+0000 7faa466ad640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7faa40070030 con 0x7faa400a67e0 2024-09-16T03:42:28.844 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.813+0000 7faa466ad640 10 monclient(hunting): _renew_subs 2024-09-16T03:42:28.844 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.813+0000 7faa466ad640 10 monclient(hunting): authenticate will time out at 995.784976s 2024-09-16T03:42:28.844 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.813+0000 7faa3ffff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7faa400a67e0 0x7faa400a6bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:28.844 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.813+0000 7faa3ffff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7faa400a67e0 0x7faa400a6bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:41288/0 (socket says 172.21.15.42:41288) 2024-09-16T03:42:28.844 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.813+0000 7faa3ffff640 1 -- 172.21.15.42:0/3856101560 learned_addr learned my addr 172.21.15.42:0/3856101560 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:42:28.844 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.813+0000 7faa3ffff640 10 monclient(hunting): get_auth_request con 0x7faa400a67e0 auth_method 0 2024-09-16T03:42:28.845 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.813+0000 7faa3ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:28.845 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.813+0000 7faa3ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:28.845 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.813+0000 7faa3ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:28.845 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.813+0000 7faa3ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:28.845 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.813+0000 7faa3ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:28.845 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.813+0000 7faa3ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:28.845 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.813+0000 7faa44c4a640 1 --2- 172.21.15.42:0/3856101560 >> v2:172.21.15.154:3300/0 conn(0x7faa400a7100 0x7faa401357a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:28.845 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.813+0000 7faa44c4a640 10 monclient(hunting): get_auth_request con 0x7faa400a7100 auth_method 0 2024-09-16T03:42:28.845 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.813+0000 7faa44c4a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:28.845 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.813+0000 7faa44c4a640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:28.845 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.813+0000 7faa44c4a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:28.846 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.813+0000 7faa3ffff640 10 monclient(hunting): handle_auth_done global_id 4229 payload 274 2024-09-16T03:42:28.846 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.813+0000 7faa3ffff640 1 -- 172.21.15.42:0/3856101560 >> v2:172.21.15.154:3300/0 conn(0x7faa400a7100 msgr2=0x7faa401357a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:42:28.846 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.813+0000 7faa3ffff640 1 --2- 172.21.15.42:0/3856101560 >> v2:172.21.15.154:3300/0 conn(0x7faa400a7100 0x7faa401357a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:28.846 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.813+0000 7faa3ffff640 1 -- 172.21.15.42:0/3856101560 >> v1:172.21.15.154:6789/0 conn(0x7faa40072560 legacy=0x7faa40135090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:28.846 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.813+0000 7faa3ffff640 10 monclient: _finish_hunting 0 2024-09-16T03:42:28.846 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.813+0000 7faa3ffff640 1 monclient: found mon.noname-b 2024-09-16T03:42:28.846 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.813+0000 7faa3ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:42:28.846 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.813+0000 7faa3ffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T03:42:28.846 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.813+0000 7faa3ffff640 1 -- 172.21.15.42:0/3856101560 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa400712e0 con 0x7faa400a67e0 2024-09-16T03:42:28.847 INFO:teuthology.orchestra.run.smithi042.stdout:51539607558 2024-09-16T03:42:28.847 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-16T03:42:28.849 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.813+0000 7fcff6396640 1 Processor -- start 2024-09-16T03:42:28.849 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.813+0000 7fcff6396640 1 -- start start 2024-09-16T03:42:28.849 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.813+0000 7fcff6396640 10 monclient: init 2024-09-16T03:42:28.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.813+0000 7fcff6396640 10 monclient: _reopen_session rank -1 2024-09-16T03:42:28.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.813+0000 7fcff6396640 10 monclient: _add_conns ranks=[2,4,1,0,3] 2024-09-16T03:42:28.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.813+0000 7fcff6396640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fcff00a39b0 0x7fcff0133b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:28.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.813+0000 7fcff6396640 10 monclient(hunting): picked mon.noname-b con 0x7fcff00a39b0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:42:28.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.813+0000 7fcff6396640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fcff00a3d90 0x7fcff0137000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:28.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.813+0000 7fcff6396640 10 monclient(hunting): picked mon.noname-c con 0x7fcff00a3d90 addr v2:172.21.15.154:3300/0 2024-09-16T03:42:28.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.813+0000 7fcff6396640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fcff0137540 addr v1:172.21.15.154:6789/0 2024-09-16T03:42:28.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.813+0000 7fcff6396640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcff009c140 con 0x7fcff0137540 2024-09-16T03:42:28.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.813+0000 7fcff6396640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:28.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.813+0000 7fcff6396640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fcff009c420 con 0x7fcff00a3d90 2024-09-16T03:42:28.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.813+0000 7fcff6396640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:28.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.813+0000 7fcff6396640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fcff006dd60 con 0x7fcff00a39b0 2024-09-16T03:42:28.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.813+0000 7fcff6396640 10 monclient(hunting): _renew_subs 2024-09-16T03:42:28.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.813+0000 7fcff6396640 10 monclient(hunting): authenticate will time out at 995.786424s 2024-09-16T03:42:28.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.813+0000 7fcfeffff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fcff00a39b0 0x7fcff0133b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:28.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.813+0000 7fcfeffff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fcff00a39b0 0x7fcff0133b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:41296/0 (socket says 172.21.15.42:41296) 2024-09-16T03:42:28.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.813+0000 7fcfeffff640 1 -- 172.21.15.42:0/3434134151 learned_addr learned my addr 172.21.15.42:0/3434134151 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:42:28.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.813+0000 7fcfeffff640 10 monclient(hunting): get_auth_request con 0x7fcff00a39b0 auth_method 0 2024-09-16T03:42:28.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.813+0000 7fcfeffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:28.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.813+0000 7fcfeffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:28.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.813+0000 7fcfeffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:28.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.813+0000 7fcfeffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:28.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.813+0000 7fcfeffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:28.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.813+0000 7fcfeffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:28.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.813+0000 7fcfeffff640 10 monclient(hunting): handle_auth_done global_id 4232 payload 274 2024-09-16T03:42:28.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.813+0000 7fcfef7fe640 1 --2- 172.21.15.42:0/3434134151 >> v2:172.21.15.154:3300/0 conn(0x7fcff00a3d90 0x7fcff0137000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:28.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.813+0000 7fcfeffff640 1 -- 172.21.15.42:0/3434134151 >> v2:172.21.15.154:3300/0 conn(0x7fcff00a3d90 msgr2=0x7fcff0137000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:42:28.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.813+0000 7fcfeffff640 1 --2- 172.21.15.42:0/3434134151 >> v2:172.21.15.154:3300/0 conn(0x7fcff00a3d90 0x7fcff0137000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:28.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.813+0000 7fcfeffff640 1 -- 172.21.15.42:0/3434134151 >> v1:172.21.15.154:6789/0 conn(0x7fcff0137540 legacy=0x7fcff013e390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:42:28.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.813+0000 7fcfeffff640 10 monclient: _finish_hunting 0 2024-09-16T03:42:28.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.813+0000 7fcfeffff640 1 monclient: found mon.noname-b 2024-09-16T03:42:28.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.813+0000 7fcfeffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:42:28.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.813+0000 7fcfeffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T03:42:28.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.813+0000 7fcfeffff640 1 -- 172.21.15.42:0/3434134151 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcff00a5e20 con 0x7fcff00a39b0 2024-09-16T03:42:28.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.813+0000 7fcfef7fe640 10 monclient: get_auth_request con 0x7fcff00a3d90 auth_method 0 2024-09-16T03:42:28.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.813+0000 7fcfef7fe640 1 --2- 172.21.15.42:0/3434134151 >> v2:172.21.15.154:3300/0 conn(0x7fcff00a3d90 0x7fcff0137000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T03:42:28.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.813+0000 7fcfe7fff640 1 --2- 172.21.15.42:0/115499394 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcfe80a39b0 0x7fcfe8133b80 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fcfe80a56c0 tx=0x7fcfd801c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4267bfae4914bd02 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:28.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.813+0000 7fcfe6ffd640 1 -- 172.21.15.42:0/115499394 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfd80b0070 con 0x7fcfe80a39b0 2024-09-16T03:42:28.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.813+0000 7fcfe6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:28.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.813+0000 7fcfe6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:42:28.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.813+0000 7fcfe6ffd640 10 monclient: dump: 2024-09-16T03:42:28.853 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:28.854 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:28.854 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:28.854 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:28.854 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:28.854 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:28.854 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:28.854 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:28.854 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:28.854 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:28.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.813+0000 7fcfe6ffd640 10 monclient: _finish_auth 0 2024-09-16T03:42:28.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.813+0000 7fcfe6ffd640 10 monclient: _check_auth_tickets 2024-09-16T03:42:28.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.813+0000 7fcfe6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:42:28.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.813+0000 7fcfe6ffd640 1 -- 172.21.15.42:0/115499394 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcfd80ab600 con 0x7fcfe80a39b0 2024-09-16T03:42:28.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.813+0000 7fcfee6aa640 5 monclient: authenticate success, global_id 4266 2024-09-16T03:42:28.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.813+0000 7fcfee6aa640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:42:28.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.813+0000 7fcfe6ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:42:28.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.813+0000 7fcfee6aa640 10 monclient: get_monmap_and_config success 2024-09-16T03:42:28.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.813+0000 7fcfee6aa640 4 set_mon_vals no callback set 2024-09-16T03:42:28.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.813+0000 7fcfe6ffd640 1 -- 172.21.15.42:0/115499394 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfd80ab9d0 con 0x7fcfe80a39b0 2024-09-16T03:42:28.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.813+0000 7fcfe6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:28.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.813+0000 7fcfe6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:42:28.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.813+0000 7fcfe6ffd640 10 monclient: dump: 2024-09-16T03:42:28.855 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:28.855 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:28.855 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:28.855 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:28.855 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:28.856 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:28.856 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:28.856 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:28.856 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:28.856 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:28.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7efd2cc9d640 10 monclient: get_monmap_and_config 2024-09-16T03:42:28.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7efd2cc9d640 10 monclient: build_initial_monmap 2024-09-16T03:42:28.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7fcfee6aa640 10 monclient: shutdown 2024-09-16T03:42:28.857 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7fcfee6aa640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:42:28.857 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7efd2cc9d640 10 monclient: monmap: 2024-09-16T03:42:28.857 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:42:28.857 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:28.857 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:42:28.820404+0000 2024-09-16T03:42:28.857 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:42:28.820404+0000 2024-09-16T03:42:28.857 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:42:28.857 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:28.857 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:42:28.857 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:42:28.858 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:42:28.858 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:42:28.858 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:42:28.858 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:28.858 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7fcfee6aa640 1 -- 172.21.15.42:0/115499394 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcfe80a39b0 msgr2=0x7fcfe8133b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:28.858 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7fcfee6aa640 1 --2- 172.21.15.42:0/115499394 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcfe80a39b0 0x7fcfe8133b80 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fcfe80a56c0 tx=0x7fcfd801c3a0 comp rx=0 tx=0).stop 2024-09-16T03:42:28.859 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7fcfee6aa640 1 -- 172.21.15.42:0/115499394 shutdown_connections 2024-09-16T03:42:28.859 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7fcfee6aa640 1 --2- 172.21.15.42:0/115499394 >> v2:172.21.15.154:3300/0 conn(0x7fcfe80a3d90 0x7fcfe8137000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:28.859 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7fcfee6aa640 1 --2- 172.21.15.42:0/115499394 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcfe80a39b0 0x7fcfe8133b80 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:28.859 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7fcfee6aa640 1 -- 172.21.15.42:0/115499394 >> 172.21.15.42:0/115499394 conn(0x7fcfe80a9fb0 msgr2=0x7fcfe80aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:42:28.859 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7fcfee6aa640 1 -- 172.21.15.42:0/115499394 shutdown_connections 2024-09-16T03:42:28.859 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7fcfee6aa640 1 -- 172.21.15.42:0/115499394 wait complete. 2024-09-16T03:42:28.860 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7faa3ffff640 1 --2- 172.21.15.42:0/3856101560 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7faa400a67e0 0x7faa400a6bc0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7faa30009870 tx=0x7faa3001c3f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=618c647689ff5a0f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:28.860 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7faa3effd640 1 -- 172.21.15.42:0/3856101560 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa300ab410 con 0x7faa400a67e0 2024-09-16T03:42:28.860 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7faa3effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:28.860 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7faa3effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:42:28.860 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7faa3effd640 10 monclient: dump: 2024-09-16T03:42:28.860 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:28.860 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:28.860 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:28.860 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:28.860 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:28.860 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:28.861 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:28.861 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:28.861 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:28.861 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:28.861 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7faa3effd640 10 monclient: _finish_auth 0 2024-09-16T03:42:28.861 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7faa3effd640 10 monclient: _check_auth_tickets 2024-09-16T03:42:28.861 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7faa3effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:42:28.861 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7faa3effd640 1 -- 172.21.15.42:0/3856101560 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa300b0070 con 0x7faa400a67e0 2024-09-16T03:42:28.861 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7faa3effd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:42:28.861 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7faa3effd640 1 -- 172.21.15.42:0/3856101560 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa300ab7d0 con 0x7faa400a67e0 2024-09-16T03:42:28.861 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7faa466ad640 5 monclient: authenticate success, global_id 4229 2024-09-16T03:42:28.861 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7faa466ad640 10 monclient: get_monmap_and_config success 2024-09-16T03:42:28.861 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7faa466ad640 4 set_mon_vals no callback set 2024-09-16T03:42:28.861 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7faa3effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:28.862 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7faa3effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:42:28.862 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7faa3effd640 10 monclient: dump: 2024-09-16T03:42:28.862 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:28.862 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:28.862 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:28.862 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:28.862 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:28.862 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:28.862 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:28.862 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:28.862 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:28.862 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:28.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7efd2cc9d640 1 Processor -- start 2024-09-16T03:42:28.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7fcfee6aa640 10 monclient: build_initial_monmap 2024-09-16T03:42:28.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7fcfee6aa640 10 monclient: monmap: 2024-09-16T03:42:28.863 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:42:28.863 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:42:28.863 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:42:28.863 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:42:28.863 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:42:28.863 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:28.864 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:42:28.864 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:42:28.864 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:42:28.864 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:28.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7efd2cc9d640 1 -- start start 2024-09-16T03:42:28.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7efd2cc9d640 10 monclient: init 2024-09-16T03:42:28.865 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7fcfee6aa640 1 librados: starting msgr at 2024-09-16T03:42:28.865 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7fcfee6aa640 1 librados: starting objecter 2024-09-16T03:42:28.865 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7fcfee6aa640 1 Processor -- start 2024-09-16T03:42:28.865 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7fcfee6aa640 1 -- start start 2024-09-16T03:42:28.865 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7fcfee6aa640 1 librados: setting wanted keys 2024-09-16T03:42:28.865 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7fcfee6aa640 1 librados: calling monclient init 2024-09-16T03:42:28.865 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7fcfee6aa640 10 monclient: init 2024-09-16T03:42:28.865 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7fcfeffff640 1 --2- 172.21.15.42:0/3434134151 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fcff00a39b0 0x7fcff0133b80 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fcfe0009870 tx=0x7fcfe001c3e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ac830f15ee9d5c81 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:28.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7faa466ad640 10 monclient: shutdown 2024-09-16T03:42:28.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7faa466ad640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:42:28.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7faa466ad640 1 -- 172.21.15.42:0/3856101560 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7faa400a67e0 msgr2=0x7faa400a6bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:28.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7faa466ad640 1 --2- 172.21.15.42:0/3856101560 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7faa400a67e0 0x7faa400a6bc0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7faa30009870 tx=0x7faa3001c3f0 comp rx=0 tx=0).stop 2024-09-16T03:42:28.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7faa466ad640 1 -- 172.21.15.42:0/3856101560 shutdown_connections 2024-09-16T03:42:28.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7faa466ad640 1 --2- 172.21.15.42:0/3856101560 >> v2:172.21.15.154:3300/0 conn(0x7faa400a7100 0x7faa401357a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:28.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7faa466ad640 1 --2- 172.21.15.42:0/3856101560 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7faa400a67e0 0x7faa400a6bc0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:28.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7faa466ad640 1 -- 172.21.15.42:0/3856101560 >> 172.21.15.42:0/3856101560 conn(0x7faa400a7ed0 msgr2=0x7faa400a43d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:42:28.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7efd2cc9d640 10 monclient: _reopen_session rank -1 2024-09-16T03:42:28.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7efd2cc9d640 10 monclient: _add_conns ranks=[2,0,1,4,3] 2024-09-16T03:42:28.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7efd2cc9d640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7efd280a39b0 0x7efd28133b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:28.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7efd2cc9d640 10 monclient(hunting): picked mon.noname-b con 0x7efd280a39b0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:42:28.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7faa466ad640 1 -- 172.21.15.42:0/3856101560 shutdown_connections 2024-09-16T03:42:28.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7faa466ad640 1 -- 172.21.15.42:0/3856101560 wait complete. 2024-09-16T03:42:28.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7fcfeeffd640 1 -- 172.21.15.42:0/3434134151 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfe00b0070 con 0x7fcff00a39b0 2024-09-16T03:42:28.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7fcfeeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:28.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7fcfeeffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:42:28.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7fcfeeffd640 10 monclient: dump: 2024-09-16T03:42:28.869 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:28.869 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:28.869 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:28.869 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:28.869 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:28.869 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:28.869 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:28.869 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:28.869 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:28.869 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:28.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7fcfeeffd640 10 monclient: _finish_auth 0 2024-09-16T03:42:28.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7fcfeeffd640 10 monclient: _check_auth_tickets 2024-09-16T03:42:28.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7fcfeeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:42:28.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7fcfeeffd640 1 -- 172.21.15.42:0/3434134151 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcfe00ab600 con 0x7fcff00a39b0 2024-09-16T03:42:28.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7fcfeeffd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:42:28.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7fcfeeffd640 1 -- 172.21.15.42:0/3434134151 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfe00ab9d0 con 0x7fcff00a39b0 2024-09-16T03:42:28.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7fcfeeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:28.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7fcfeeffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:42:28.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7fcfeeffd640 10 monclient: dump: 2024-09-16T03:42:28.871 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:28.871 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:28.871 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:28.871 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:28.871 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:28.871 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:28.871 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:28.871 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:28.871 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:28.871 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:28.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7fcff6396640 5 monclient: authenticate success, global_id 4232 2024-09-16T03:42:28.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7fcff6396640 10 monclient: get_monmap_and_config success 2024-09-16T03:42:28.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7fcff6396640 4 set_mon_vals no callback set 2024-09-16T03:42:28.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7efd2cc9d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7efd280a3d90 addr v1:172.21.15.42:6789/0 2024-09-16T03:42:28.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7efd2659c640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7efd280a39b0 0x7efd28133b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:28.873 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7efd2659c640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7efd280a39b0 0x7efd28133b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:41298/0 (socket says 172.21.15.42:41298) 2024-09-16T03:42:28.873 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7efd2cc9d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7efd28137710 addr v1:172.21.15.154:6789/0 2024-09-16T03:42:28.873 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7efd2659c640 1 -- 172.21.15.42:0/491001221 learned_addr learned my addr 172.21.15.42:0/491001221 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:42:28.873 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7efd2cc9d640 1 -- 172.21.15.42:0/491001221 --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efd2809c140 con 0x7efd280a3d90 2024-09-16T03:42:28.873 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7fcfee6aa640 10 monclient: _reopen_session rank -1 2024-09-16T03:42:28.874 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7fcfee6aa640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:42:28.874 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7fcfee6aa640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fcfe80a39b0 0x7fcfe813d250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:28.874 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7fcfee6aa640 10 monclient(hunting): picked mon.noname-c con 0x7fcfe80a39b0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:42:28.874 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7fcfee6aa640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcfe80a3d90 0x7fcfe813d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:28.874 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7fcfee6aa640 10 monclient(hunting): picked mon.noname-a con 0x7fcfe80a3d90 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:42:28.874 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7fcfee6aa640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcfe8137540 0x7fcfe813dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:28.874 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7fcfee6aa640 10 monclient(hunting): picked mon.noname-b con 0x7fcfe8137540 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:42:28.874 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7fcfee6aa640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:28.874 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7fcfee6aa640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fcfe81424e0 con 0x7fcfe80a3d90 2024-09-16T03:42:28.874 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7fcfee6aa640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:28.875 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7fcfee6aa640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fcfe8142660 con 0x7fcfe8137540 2024-09-16T03:42:28.875 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7fcfee6aa640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:28.875 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7fcfee6aa640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fcfe81427e0 con 0x7fcfe80a39b0 2024-09-16T03:42:28.875 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7fcfee6aa640 10 monclient(hunting): _renew_subs 2024-09-16T03:42:28.875 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7fcfee6aa640 10 monclient(hunting): authenticate will time out at 995.790197s 2024-09-16T03:42:28.875 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7efd2cc9d640 1 -- 172.21.15.42:0/491001221 --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efd2809c420 con 0x7efd28137710 2024-09-16T03:42:28.876 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7efd2cc9d640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:28.876 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7fcfe7fff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fcfe80a39b0 0x7fcfe813d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:28.876 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7efd2cc9d640 1 -- 172.21.15.42:0/491001221 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7efd2806dd60 con 0x7efd280a39b0 2024-09-16T03:42:28.876 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7efd2cc9d640 10 monclient(hunting): _renew_subs 2024-09-16T03:42:28.876 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7fcfe7fff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fcfe80a39b0 0x7fcfe813d250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:41314/0 (socket says 172.21.15.42:41314) 2024-09-16T03:42:28.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7fcfe7fff640 1 -- 172.21.15.42:0/131202167 learned_addr learned my addr 172.21.15.42:0/131202167 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:42:28.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7fcfe7fff640 10 monclient(hunting): get_auth_request con 0x7fcfe80a39b0 auth_method 0 2024-09-16T03:42:28.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7fcfe7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:28.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7fcfe7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:28.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7fcfe7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:28.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7fcfe77fe640 1 --2- 172.21.15.42:0/131202167 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcfe80a3d90 0x7fcfe813d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:28.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7efd2cc9d640 10 monclient(hunting): authenticate will time out at 995.790420s 2024-09-16T03:42:28.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7fcfe77fe640 10 monclient(hunting): get_auth_request con 0x7fcfe80a3d90 auth_method 0 2024-09-16T03:42:28.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7fcfe77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:28.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7fcfe77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:28.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7fcfe77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:28.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7fcfe7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:28.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7fcfe7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:28.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7fcfe7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:28.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7fcfe77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:28.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7fcfe77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:28.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7efd2659c640 10 monclient(hunting): get_auth_request con 0x7efd280a39b0 auth_method 0 2024-09-16T03:42:28.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7efd2659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:28.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7efd2659c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:28.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7efd2659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:28.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7fcfe77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:28.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7fcfecc47640 1 --2- 172.21.15.42:0/131202167 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcfe8137540 0x7fcfe813dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:28.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7fcfecc47640 10 monclient(hunting): get_auth_request con 0x7fcfe8137540 auth_method 0 2024-09-16T03:42:28.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7fcfecc47640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:28.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7fcfecc47640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:28.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7fcfecc47640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:28.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7faa466ad640 10 monclient: build_initial_monmap 2024-09-16T03:42:28.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7faa466ad640 10 monclient: monmap: 2024-09-16T03:42:28.880 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:42:28.880 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:42:28.880 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:42:28.880 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:42:28.880 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:42:28.880 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:28.881 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:42:28.881 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:42:28.881 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:42:28.881 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:28.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7fcff6396640 10 monclient: shutdown 2024-09-16T03:42:28.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7fcff6396640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:42:28.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7fcff6396640 1 -- 172.21.15.42:0/3434134151 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fcff00a39b0 msgr2=0x7fcff0133b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:28.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7fcff6396640 1 --2- 172.21.15.42:0/3434134151 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fcff00a39b0 0x7fcff0133b80 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fcfe0009870 tx=0x7fcfe001c3e0 comp rx=0 tx=0).stop 2024-09-16T03:42:28.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7efd2659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:28.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7efd2659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:28.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7efd2659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:28.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7fcfe77fe640 10 monclient(hunting): handle_auth_done global_id 4269 payload 824 2024-09-16T03:42:28.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7fcfe77fe640 1 -- 172.21.15.42:0/131202167 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fcfe80a39b0 msgr2=0x7fcfe813d250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:28.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7fcfe77fe640 1 --2- 172.21.15.42:0/131202167 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fcfe80a39b0 0x7fcfe813d250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:28.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7fcfe77fe640 1 -- 172.21.15.42:0/131202167 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcfe8137540 msgr2=0x7fcfe813dcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:28.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7fcfe77fe640 1 --2- 172.21.15.42:0/131202167 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcfe8137540 0x7fcfe813dcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:28.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7fcfe77fe640 10 monclient: _finish_hunting 0 2024-09-16T03:42:28.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7fcfe77fe640 1 monclient: found mon.noname-a 2024-09-16T03:42:28.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7fcfe77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:42:28.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7fcfe77fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:42:28.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7fcfe77fe640 1 -- 172.21.15.42:0/131202167 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfe809e5d0 con 0x7fcfe80a3d90 2024-09-16T03:42:28.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7efd25d9b640 1 -- 172.21.15.42:0/491001221 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1472261313 0 0) 0x7efd2809c420 con 0x7efd28137710 2024-09-16T03:42:28.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7efd25d9b640 10 monclient(hunting): _negotiate 2024-09-16T03:42:28.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7efd25d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:28.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7efd25d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:28.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7efd25d9b640 10 monclient(hunting): my global_id is 4258 2024-09-16T03:42:28.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7fcfecc47640 1 --2- 172.21.15.42:0/131202167 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcfe8137540 0x7fcfe813dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T03:42:28.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7faa466ad640 1 librados: starting msgr at 2024-09-16T03:42:28.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7faa466ad640 1 librados: starting objecter 2024-09-16T03:42:28.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7efd25d9b640 1 -- 172.21.15.42:0/491001221 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efd14003530 con 0x7efd28137710 2024-09-16T03:42:28.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7efd2659c640 10 monclient(hunting): handle_auth_done global_id 4238 payload 274 2024-09-16T03:42:28.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7efd2659c640 1 -- 172.21.15.42:0/491001221 >> v1:172.21.15.154:6789/0 conn(0x7efd28137710 legacy=0x7efd2813f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:28.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7efd2659c640 1 -- 172.21.15.42:0/491001221 >> v1:172.21.15.42:6789/0 conn(0x7efd280a3d90 legacy=0x7efd28137000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:28.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7efd2659c640 10 monclient: _finish_hunting 0 2024-09-16T03:42:28.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7efd2659c640 1 monclient: found mon.noname-b 2024-09-16T03:42:28.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7efd2659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:42:28.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7efd2659c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T03:42:28.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7efd2659c640 1 -- 172.21.15.42:0/491001221 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd280a5e20 con 0x7efd280a39b0 2024-09-16T03:42:28.888 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7faa466ad640 1 Processor -- start 2024-09-16T03:42:28.888 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7fcff6396640 1 -- 172.21.15.42:0/3434134151 shutdown_connections 2024-09-16T03:42:28.888 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7fcff6396640 1 --2- 172.21.15.42:0/3434134151 >> v2:172.21.15.154:3300/0 conn(0x7fcff00a3d90 0x7fcff0137000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:28.888 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7fcff6396640 1 --2- 172.21.15.42:0/3434134151 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fcff00a39b0 0x7fcff0133b80 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:28.888 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7fcff6396640 1 -- 172.21.15.42:0/3434134151 >> 172.21.15.42:0/3434134151 conn(0x7fcff00a9fb0 msgr2=0x7fcff00aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:42:28.888 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7fcfe77fe640 1 --2- 172.21.15.42:0/131202167 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcfe80a3d90 0x7fcfe813d790 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fcfdc00b3f0 tx=0x7fcfdc00b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:28.889 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7fcfe4ff9640 10 client.?.objecter ms_handle_connect 0x7fcfe80a3d90 2024-09-16T03:42:28.889 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7fcfe4ff9640 10 client.?.objecter resend_mon_ops 2024-09-16T03:42:28.889 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7faa466ad640 1 -- start start 2024-09-16T03:42:28.889 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7faa466ad640 1 librados: setting wanted keys 2024-09-16T03:42:28.889 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7faa466ad640 1 librados: calling monclient init 2024-09-16T03:42:28.889 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7faa466ad640 10 monclient: init 2024-09-16T03:42:28.890 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7fcfe4ff9640 1 -- 172.21.15.42:0/131202167 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfdc015020 con 0x7fcfe80a3d90 2024-09-16T03:42:28.890 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7fcfe4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:28.890 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7fcfe4ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:42:28.890 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7fcfe4ff9640 10 monclient: dump: 2024-09-16T03:42:28.890 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:28.890 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:28.891 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:28.891 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:28.891 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:28.891 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:28.891 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:28.891 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:28.891 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:28.891 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:28.891 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7fcfe4ff9640 10 monclient: _finish_auth 0 2024-09-16T03:42:28.891 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7fcfe4ff9640 10 monclient: _check_auth_tickets 2024-09-16T03:42:28.891 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7fcfe4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:42:28.891 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7fcfee6aa640 5 monclient: authenticate success, global_id 4269 2024-09-16T03:42:28.891 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7fcfe4ff9640 1 -- 172.21.15.42:0/131202167 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcfdc011830 con 0x7fcfe80a3d90 2024-09-16T03:42:28.891 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7fcfee6aa640 10 monclient: _renew_subs 2024-09-16T03:42:28.892 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7fcfee6aa640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:42:28.892 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7fcfee6aa640 1 -- 172.21.15.42:0/131202167 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcfe809e8b0 con 0x7fcfe80a3d90 2024-09-16T03:42:28.892 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7fcfe4ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T03:42:28.892 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7fcfe4ff9640 1 -- 172.21.15.42:0/131202167 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfdc011c20 con 0x7fcfe80a3d90 2024-09-16T03:42:28.892 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7fcfe4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:28.892 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7fcfe4ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:42:28.892 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7fcfe4ff9640 10 monclient: dump: 2024-09-16T03:42:28.892 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:28.892 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:28.892 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:28.892 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:28.892 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:28.892 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:28.893 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:28.893 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:28.893 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:28.893 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:28.893 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7fcfee6aa640 10 client.4269.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:42:28.893 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7fcfee6aa640 10 monclient: _renew_subs 2024-09-16T03:42:28.893 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7fcfee6aa640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:42:28.893 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7fcfee6aa640 1 -- 172.21.15.42:0/131202167 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcfe806f680 con 0x7fcfe80a3d90 2024-09-16T03:42:28.893 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7fcfee6aa640 1 librados: init done 2024-09-16T03:42:28.893 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7fcfe67fc640 4 set_mon_vals no callback set 2024-09-16T03:42:28.894 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7faa466ad640 10 monclient: _reopen_session rank -1 2024-09-16T03:42:28.894 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7faa466ad640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:42:28.894 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7faa466ad640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faa40072560 0x7faa40135430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:28.894 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7faa466ad640 10 monclient(hunting): picked mon.noname-a con 0x7faa40072560 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:42:28.894 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7faa466ad640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7faa400a67e0 0x7faa40136ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:28.894 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7faa466ad640 10 monclient(hunting): picked mon.noname-c con 0x7faa400a67e0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:42:28.894 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7faa466ad640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faa400a7100 0x7faa40137430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:28.894 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7faa466ad640 10 monclient(hunting): picked mon.noname-b con 0x7faa400a7100 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:42:28.894 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7faa466ad640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:28.895 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7faa466ad640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7faa400a4ec0 con 0x7faa40072560 2024-09-16T03:42:28.895 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7faa466ad640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:28.895 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7faa466ad640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7faa400a4d40 con 0x7faa400a7100 2024-09-16T03:42:28.895 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7faa466ad640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:28.895 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7faa466ad640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7faa40135ce0 con 0x7faa400a67e0 2024-09-16T03:42:28.895 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7faa466ad640 10 monclient(hunting): _renew_subs 2024-09-16T03:42:28.895 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.817+0000 7faa466ad640 10 monclient(hunting): authenticate will time out at 995.791990s 2024-09-16T03:42:28.895 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7fcff6396640 1 -- 172.21.15.42:0/3434134151 shutdown_connections 2024-09-16T03:42:28.895 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7fcff6396640 1 -- 172.21.15.42:0/3434134151 wait complete. 2024-09-16T03:42:28.896 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7faa3ffff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faa40072560 0x7faa40135430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:28.896 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7faa3f7fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7faa400a67e0 0x7faa40136ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:28.896 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7faa3ffff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faa40072560 0x7faa40135430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:37632/0 (socket says 172.21.15.42:37632) 2024-09-16T03:42:28.896 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7faa3ffff640 1 -- 172.21.15.42:0/2993064549 learned_addr learned my addr 172.21.15.42:0/2993064549 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:42:28.896 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7faa3ffff640 10 monclient(hunting): get_auth_request con 0x7faa40072560 auth_method 0 2024-09-16T03:42:28.896 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7faa3ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:28.896 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7faa3ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:28.896 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7faa3ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:28.897 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7faa44c4a640 1 --2- 172.21.15.42:0/2993064549 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faa400a7100 0x7faa40137430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:28.897 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7faa3f7fe640 10 monclient(hunting): get_auth_request con 0x7faa400a67e0 auth_method 0 2024-09-16T03:42:28.897 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7faa3f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:28.897 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7faa3f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:28.897 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7faa3f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:28.897 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7faa44c4a640 10 monclient(hunting): get_auth_request con 0x7faa400a7100 auth_method 0 2024-09-16T03:42:28.897 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7faa44c4a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:28.897 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7faa44c4a640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:28.897 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7faa44c4a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:28.897 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7faa3f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:28.897 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7faa3f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:28.897 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7faa3f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:28.897 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7faa3ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:28.897 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7faa3ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:28.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7faa3ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:28.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7fcfe4ff9640 1 -- 172.21.15.42:0/131202167 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fcfdc018020 con 0x7fcfe80a3d90 2024-09-16T03:42:28.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7fcfe4ff9640 1 --2- 172.21.15.42:0/131202167 >> v2:172.21.15.154:6812/254845342 conn(0x7fcfd4037810 0x7fcfd4039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:28.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7faa44c4a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:28.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7faa44c4a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:28.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7faa44c4a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:28.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7faa3f7fe640 10 monclient(hunting): handle_auth_done global_id 4241 payload 824 2024-09-16T03:42:28.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7faa3f7fe640 1 -- 172.21.15.42:0/2993064549 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faa400a7100 msgr2=0x7faa40137430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:28.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7faa3f7fe640 1 --2- 172.21.15.42:0/2993064549 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faa400a7100 0x7faa40137430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:28.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7faa3f7fe640 1 -- 172.21.15.42:0/2993064549 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faa40072560 msgr2=0x7faa40135430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:28.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7faa3f7fe640 1 --2- 172.21.15.42:0/2993064549 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faa40072560 0x7faa40135430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:28.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7faa3f7fe640 10 monclient: _finish_hunting 0 2024-09-16T03:42:28.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7faa3f7fe640 1 monclient: found mon.noname-c 2024-09-16T03:42:28.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7faa3f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:42:28.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7faa3f7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:42:28.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7faa3f7fe640 1 -- 172.21.15.42:0/2993064549 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa4013a8c0 con 0x7faa400a67e0 2024-09-16T03:42:28.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7faa3ffff640 1 --2- 172.21.15.42:0/2993064549 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faa40072560 0x7faa40135430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:42:28.900 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7fcfe4ff9640 1 -- 172.21.15.42:0/131202167 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 3562+0+0 (secure 0 0 0) 0x7fcfdc048660 con 0x7fcfe80a3d90 2024-09-16T03:42:28.900 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7fcfe4ff9640 10 client.4269.objecter ms_dispatch 0x7fcfe8000ba0 osd_map(18..18 src has 1..18) 2024-09-16T03:42:28.900 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7fcfe4ff9640 3 client.4269.objecter handle_osd_map got epochs [18,18] > 0 2024-09-16T03:42:28.900 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7fcfe4ff9640 3 client.4269.objecter handle_osd_map decoding full epoch 18 2024-09-16T03:42:28.900 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7fcfe4ff9640 20 client.4269.objecter dump_active .. 0 homeless 2024-09-16T03:42:28.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7faa3f7fe640 1 --2- 172.21.15.42:0/2993064549 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7faa400a67e0 0x7faa40136ef0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7faa34000f70 tx=0x7faa3400c820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:28.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7faa3cff9640 10 client.?.objecter ms_handle_connect 0x7faa400a67e0 2024-09-16T03:42:28.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7faa3cff9640 10 client.?.objecter resend_mon_ops 2024-09-16T03:42:28.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7fcfee6aa640 10 client.4269.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:42:28.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7fcfee6aa640 15 client.4269.objecter _session_command_op_assign -1 1 2024-09-16T03:42:28.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7fcfee6aa640 1 --2- 172.21.15.42:0/131202167 >> v2:172.21.15.42:6808/1401549752 conn(0x7fcfe81d05c0 0x7fcfe802c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:28.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7fcfee6aa640 20 client.4269.objecter _get_session s=0x7fcfe81342d0 osd=0 3 2024-09-16T03:42:28.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7fcfee6aa640 20 client.4269.objecter put_session s=0x7fcfe81342d0 osd=0 3 2024-09-16T03:42:28.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7fcfee6aa640 20 client.4269.objecter _get_session s=0x7fcfe81342d0 osd=0 3 2024-09-16T03:42:28.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7fcfee6aa640 15 client.4269.objecter _session_command_op_remove -1 1 2024-09-16T03:42:28.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7fcfee6aa640 20 client.4269.objecter get_session s=0x7fcfe81342d0 osd=0 3 2024-09-16T03:42:28.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7fcfee6aa640 15 client.4269.objecter _session_command_op_assign 0 1 2024-09-16T03:42:28.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7fcfee6aa640 20 client.4269.objecter put_session s=0x7fcfe81342d0 osd=0 4 2024-09-16T03:42:28.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7fcfee6aa640 10 client.4269.objecter _send_command 1 2024-09-16T03:42:28.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7fcfee6aa640 1 -- 172.21.15.42:0/131202167 --> v2:172.21.15.42:6808/1401549752 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fcfe8133b80 con 0x7fcfe81d05c0 2024-09-16T03:42:28.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7faa3cff9640 1 -- 172.21.15.42:0/2993064549 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa34017070 con 0x7faa400a67e0 2024-09-16T03:42:28.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7faa3cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:28.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7faa3cff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:42:28.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7faa3cff9640 10 monclient: dump: 2024-09-16T03:42:28.903 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:28.903 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:28.903 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:28.903 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:28.903 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:28.903 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:28.904 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:28.904 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:28.904 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:28.904 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:28.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7faa3cff9640 10 monclient: _finish_auth 0 2024-09-16T03:42:28.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7faa3cff9640 10 monclient: _check_auth_tickets 2024-09-16T03:42:28.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7faa3cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:42:28.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7faa3cff9640 1 -- 172.21.15.42:0/2993064549 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa340052d0 con 0x7faa400a67e0 2024-09-16T03:42:28.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7faa466ad640 5 monclient: authenticate success, global_id 4241 2024-09-16T03:42:28.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7faa466ad640 10 monclient: _renew_subs 2024-09-16T03:42:28.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7faa466ad640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:42:28.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7faa466ad640 1 -- 172.21.15.42:0/2993064549 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa4013dcb0 con 0x7faa400a67e0 2024-09-16T03:42:28.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7faa3cff9640 10 monclient: handle_config config(0 keys) 2024-09-16T03:42:28.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7faa3cff9640 1 -- 172.21.15.42:0/2993064549 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa34003840 con 0x7faa400a67e0 2024-09-16T03:42:28.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7faa3cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:28.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7faa3cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:42:28.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7faa3cff9640 10 monclient: dump: 2024-09-16T03:42:28.905 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:28.905 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:28.905 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:28.905 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:28.905 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:28.905 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:28.905 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:28.905 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:28.905 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:28.905 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:28.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7faa466ad640 10 client.4241.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:42:28.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7faa466ad640 10 monclient: _renew_subs 2024-09-16T03:42:28.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7faa466ad640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:42:28.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7fcff6396640 10 monclient: build_initial_monmap 2024-09-16T03:42:28.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7fcff6396640 10 monclient: monmap: 2024-09-16T03:42:28.906 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:42:28.906 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:42:28.906 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:42:28.906 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:42:28.906 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:42:28.907 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:28.907 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:42:28.907 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:42:28.907 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:42:28.907 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:28.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7fcff6396640 1 librados: starting msgr at 2024-09-16T03:42:28.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7fcff6396640 1 librados: starting objecter 2024-09-16T03:42:28.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7fcff6396640 1 Processor -- start 2024-09-16T03:42:28.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7faa466ad640 1 -- 172.21.15.42:0/2993064549 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7faa4013a5e0 con 0x7faa400a67e0 2024-09-16T03:42:28.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7faa466ad640 1 librados: init done 2024-09-16T03:42:28.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7fcfecc47640 1 --2- 172.21.15.42:0/131202167 >> v2:172.21.15.42:6808/1401549752 conn(0x7fcfe81d05c0 0x7fcfe802c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:28.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7fcfecc47640 10 monclient: get_auth_request con 0x7fcfe81d05c0 auth_method 0 2024-09-16T03:42:28.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7fcfe7fff640 1 --2- 172.21.15.42:0/131202167 >> v2:172.21.15.154:6812/254845342 conn(0x7fcfd4037810 0x7fcfd4039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:28.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7fcfe7fff640 10 monclient: get_auth_request con 0x7fcfd4037810 auth_method 0 2024-09-16T03:42:28.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7fcfecc47640 1 --2- 172.21.15.42:0/131202167 >> v2:172.21.15.42:6808/1401549752 conn(0x7fcfe81d05c0 0x7fcfe802c650 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:28.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7faa3dffb640 4 set_mon_vals no callback set 2024-09-16T03:42:28.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7efd2659c640 1 --2- 172.21.15.42:0/491001221 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7efd280a39b0 0x7efd28133b80 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7efd180098a0 tx=0x7efd1801c400 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5f49fd9768293ca2 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:28.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7faa466ad640 10 client.4241.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:42:28.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7faa466ad640 15 client.4241.objecter _session_command_op_assign -1 1 2024-09-16T03:42:28.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7faa466ad640 20 client.4241.objecter _get_session osd=-1 returning homeless 2024-09-16T03:42:28.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7faa466ad640 10 client.4241.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:42:28.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7faa466ad640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:42:28.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7faa466ad640 1 -- 172.21.15.42:0/2993064549 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7faa20002920 con 0x7faa400a67e0 2024-09-16T03:42:28.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7fcff6396640 1 -- start start 2024-09-16T03:42:28.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7fcff6396640 1 librados: setting wanted keys 2024-09-16T03:42:28.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7fcff6396640 1 librados: calling monclient init 2024-09-16T03:42:28.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7fcff6396640 10 monclient: init 2024-09-16T03:42:28.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7fcff6396640 10 monclient: _reopen_session rank -1 2024-09-16T03:42:28.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7fcff6396640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:42:28.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7fcff6396640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcff00a39b0 0x7fcff0130e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:28.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7fcff6396640 10 monclient(hunting): picked mon.noname-b con 0x7fcff00a39b0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:42:28.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7fcff6396640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcff00a3d90 0x7fcff009d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:28.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7fcff6396640 10 monclient(hunting): picked mon.noname-a con 0x7fcff00a3d90 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:42:28.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7fcff6396640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fcff0137540 0x7fcff009db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:28.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7fcff6396640 10 monclient(hunting): picked mon.noname-c con 0x7fcff0137540 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:42:28.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7fcff6396640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:28.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7fcff6396640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fcff01424e0 con 0x7fcff00a3d90 2024-09-16T03:42:28.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7fcff6396640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:28.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7fcff6396640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fcff0142660 con 0x7fcff00a39b0 2024-09-16T03:42:28.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7fcff6396640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:28.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7fcff6396640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fcff0131370 con 0x7fcff0137540 2024-09-16T03:42:28.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7fcff6396640 10 monclient(hunting): _renew_subs 2024-09-16T03:42:28.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7fcff6396640 10 monclient(hunting): authenticate will time out at 995.794169s 2024-09-16T03:42:28.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7faa3cff9640 1 -- 172.21.15.42:0/2993064549 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7faa3401a070 con 0x7faa400a67e0 2024-09-16T03:42:28.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7fcff4933640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fcff0137540 0x7fcff009db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:28.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7fcff4933640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fcff0137540 0x7fcff009db90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:41334/0 (socket says 172.21.15.42:41334) 2024-09-16T03:42:28.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7fcff4933640 1 -- 172.21.15.42:0/677881880 learned_addr learned my addr 172.21.15.42:0/677881880 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:42:28.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7fcff4933640 10 monclient(hunting): get_auth_request con 0x7fcff0137540 auth_method 0 2024-09-16T03:42:28.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7fcff4933640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:28.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7fcff4933640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:28.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7fcff4933640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:28.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7fcfeffff640 1 --2- 172.21.15.42:0/677881880 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcff00a39b0 0x7fcff0130e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:28.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7fcff4933640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:28.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7fcff4933640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:28.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7fcff4933640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:28.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7fcfeffff640 10 monclient(hunting): get_auth_request con 0x7fcff00a39b0 auth_method 0 2024-09-16T03:42:28.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7fcfeffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:28.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7fcfeffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:28.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7fcfeffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:28.916 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7efd25d9b640 1 -- 172.21.15.42:0/491001221 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd180b0070 con 0x7efd280a39b0 2024-09-16T03:42:28.916 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7efd25d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:28.916 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7efd25d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:42:28.916 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7efd25d9b640 10 monclient: dump: 2024-09-16T03:42:28.916 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:28.916 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:28.916 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:28.916 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:28.916 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:28.916 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:28.916 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:28.916 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:28.917 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:28.917 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:28.917 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7efd25d9b640 10 monclient: _finish_auth 0 2024-09-16T03:42:28.917 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7fcfeffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:28.917 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7fcfeffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:28.917 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7fcfeffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:28.917 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7faa3cff9640 1 --2- 172.21.15.42:0/2993064549 >> v2:172.21.15.154:6812/254845342 conn(0x7faa2c037870 0x7faa2c039d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:28.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7efd25d9b640 10 monclient: _check_auth_tickets 2024-09-16T03:42:28.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7efd25d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:42:28.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7efd25d9b640 1 -- 172.21.15.42:0/491001221 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efd180ab600 con 0x7efd280a39b0 2024-09-16T03:42:28.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7efd25d9b640 10 monclient: handle_config config(0 keys) 2024-09-16T03:42:28.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7efd25d9b640 1 -- 172.21.15.42:0/491001221 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd180ab9d0 con 0x7efd280a39b0 2024-09-16T03:42:28.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7efd25d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:28.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7efd25d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:42:28.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7fcff4933640 10 monclient(hunting): handle_auth_done global_id 4244 payload 824 2024-09-16T03:42:28.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7efd25d9b640 10 monclient: dump: 2024-09-16T03:42:28.919 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:28.919 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:28.919 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:28.919 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:28.919 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:28.919 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:28.919 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:28.919 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:28.919 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:28.919 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:28.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7fcff4933640 1 -- 172.21.15.42:0/677881880 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcff00a39b0 msgr2=0x7fcff0130e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:28.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7faa3cff9640 1 -- 172.21.15.42:0/2993064549 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(18..18 src has 1..18) ==== 3562+0+0 (secure 0 0 0) 0x7faa34047a20 con 0x7faa400a67e0 2024-09-16T03:42:28.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7faa3cff9640 10 client.4241.objecter ms_dispatch 0x7faa40000ba0 osd_map(18..18 src has 1..18) 2024-09-16T03:42:28.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7faa3cff9640 3 client.4241.objecter handle_osd_map got epochs [18,18] > 0 2024-09-16T03:42:28.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7faa3cff9640 3 client.4241.objecter handle_osd_map decoding full epoch 18 2024-09-16T03:42:28.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7fcff4933640 1 --2- 172.21.15.42:0/677881880 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcff00a39b0 0x7fcff0130e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:28.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7fcff4933640 1 -- 172.21.15.42:0/677881880 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcff00a3d90 msgr2=0x7fcff009d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:28.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7fcff4933640 1 --2- 172.21.15.42:0/677881880 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcff00a3d90 0x7fcff009d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:28.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7fcff4933640 10 monclient: _finish_hunting 0 2024-09-16T03:42:28.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7fcff4933640 1 monclient: found mon.noname-c 2024-09-16T03:42:28.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7fcff4933640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:42:28.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7fcff4933640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:42:28.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7fcff4933640 1 -- 172.21.15.42:0/677881880 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcff009f6d0 con 0x7fcff0137540 2024-09-16T03:42:28.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7faa3cff9640 10 client.4241.objecter checking command 1 2024-09-16T03:42:28.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7fcfe4ff9640 10 client.4269.objecter ms_handle_connect 0x7fcfe81d05c0 2024-09-16T03:42:28.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7fcfe7fff640 1 --2- 172.21.15.42:0/131202167 >> v2:172.21.15.154:6812/254845342 conn(0x7fcfd4037810 0x7fcfd4039cd0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fcfd801c3e0 tx=0x7fcfd8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:28.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7faa3cff9640 1 --2- 172.21.15.42:0/2993064549 >> v2:172.21.15.42:6800/4217863575 conn(0x7faa2c03d380 0x7faa2c03f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:28.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7faa3cff9640 20 client.4241.objecter _get_session s=0x7faa2c03ccd0 osd=1 3 2024-09-16T03:42:28.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7faa3cff9640 20 client.4241.objecter put_session s=0x7faa2c03ccd0 osd=1 3 2024-09-16T03:42:28.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7faa3cff9640 15 client.4241.objecter _session_command_op_remove -1 1 2024-09-16T03:42:28.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7faa3cff9640 20 client.4241.objecter _get_session s=0x7faa2c03ccd0 osd=1 3 2024-09-16T03:42:28.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7faa3cff9640 20 client.4241.objecter get_session s=0x7faa2c03ccd0 osd=1 3 2024-09-16T03:42:28.923 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7faa3cff9640 15 client.4241.objecter _session_command_op_assign 1 1 2024-09-16T03:42:28.923 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7faa3cff9640 20 client.4241.objecter put_session s=0x7faa2c03ccd0 osd=1 4 2024-09-16T03:42:28.923 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7faa3cff9640 10 client.4241.objecter _send_command 1 2024-09-16T03:42:28.923 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7faa3cff9640 1 -- 172.21.15.42:0/2993064549 --> v2:172.21.15.42:6800/4217863575 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7faa34016ec0 con 0x7faa2c03d380 2024-09-16T03:42:28.923 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7faa3cff9640 20 client.4241.objecter dump_active .. 0 homeless 2024-09-16T03:42:28.923 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7faa3cff9640 1 -- 172.21.15.42:0/2993064549 <== mon.2 v2:172.21.15.42:3301/0 6 ==== mon_get_version_reply(handle=1 version=18) ==== 24+0+0 (secure 0 0 0) 0x7faa3404c2c0 con 0x7faa400a67e0 2024-09-16T03:42:28.923 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7faa3cff9640 10 monclient: handle_get_version_reply finishing 1 version 18 2024-09-16T03:42:28.923 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7efd2cc9d640 5 monclient: authenticate success, global_id 4238 2024-09-16T03:42:28.923 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7efd2cc9d640 10 monclient: get_monmap_and_config success 2024-09-16T03:42:28.923 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7efd2cc9d640 4 set_mon_vals no callback set 2024-09-16T03:42:28.924 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7faa3ffff640 1 --2- 172.21.15.42:0/2993064549 >> v2:172.21.15.154:6812/254845342 conn(0x7faa2c037870 0x7faa2c039d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:28.924 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7faa3ffff640 10 monclient: get_auth_request con 0x7faa2c037870 auth_method 0 2024-09-16T03:42:28.924 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7faa44c4a640 1 --2- 172.21.15.42:0/2993064549 >> v2:172.21.15.42:6800/4217863575 conn(0x7faa2c03d380 0x7faa2c03f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:28.924 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7faa44c4a640 10 monclient: get_auth_request con 0x7faa2c03d380 auth_method 0 2024-09-16T03:42:28.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7fcff4933640 1 --2- 172.21.15.42:0/677881880 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fcff0137540 0x7fcff009db90 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fcfe800c780 tx=0x7fcfe800cc40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:28.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7fcfecff9640 10 client.?.objecter ms_handle_connect 0x7fcff0137540 2024-09-16T03:42:28.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7fcfecff9640 10 client.?.objecter resend_mon_ops 2024-09-16T03:42:28.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7fcfecff9640 1 -- 172.21.15.42:0/677881880 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfe8004b70 con 0x7fcff0137540 2024-09-16T03:42:28.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7fcfecff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:28.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7fcfecff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:42:28.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7fcfecff9640 10 monclient: dump: 2024-09-16T03:42:28.925 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:28.926 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:28.926 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:28.926 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:28.926 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:28.926 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:28.926 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:28.926 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:28.926 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:28.926 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:28.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7fcfecff9640 10 monclient: _finish_auth 0 2024-09-16T03:42:28.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7fcfecff9640 10 monclient: _check_auth_tickets 2024-09-16T03:42:28.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7fcfecff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:42:28.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7fcfecff9640 1 -- 172.21.15.42:0/677881880 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcfe8004d10 con 0x7fcff0137540 2024-09-16T03:42:28.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7fcff6396640 5 monclient: authenticate success, global_id 4244 2024-09-16T03:42:28.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7fcff6396640 10 monclient: _renew_subs 2024-09-16T03:42:28.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7fcff6396640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:42:28.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7fcff6396640 1 -- 172.21.15.42:0/677881880 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcff00a1520 con 0x7fcff0137540 2024-09-16T03:42:28.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7fcfecff9640 10 monclient: handle_config config(0 keys) 2024-09-16T03:42:28.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7fcfecff9640 1 -- 172.21.15.42:0/677881880 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfe800aa90 con 0x7fcff0137540 2024-09-16T03:42:28.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7fcfecff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:28.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7fcfecff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:42:28.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7fcfecff9640 10 monclient: dump: 2024-09-16T03:42:28.927 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:28.927 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:28.927 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:28.927 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:28.927 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:28.928 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:28.928 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:28.928 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:28.928 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:28.928 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:28.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7faa44c4a640 1 --2- 172.21.15.42:0/2993064549 >> v2:172.21.15.42:6800/4217863575 conn(0x7faa2c03d380 0x7faa2c03f7c0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:28.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7fcff6396640 10 client.4244.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:42:28.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7fcff6396640 10 monclient: _renew_subs 2024-09-16T03:42:28.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7fcff6396640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:42:28.929 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7fcff6396640 1 -- 172.21.15.42:0/677881880 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fcff0136e90 con 0x7fcff0137540 2024-09-16T03:42:28.929 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7fcff6396640 1 librados: init done 2024-09-16T03:42:28.929 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7fcfee7fc640 4 set_mon_vals no callback set 2024-09-16T03:42:28.929 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7efd2cc9d640 10 monclient: shutdown 2024-09-16T03:42:28.929 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7efd2cc9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:42:28.929 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7efd2cc9d640 1 -- 172.21.15.42:0/491001221 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7efd280a39b0 msgr2=0x7efd28133b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:28.929 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7efd2cc9d640 1 --2- 172.21.15.42:0/491001221 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7efd280a39b0 0x7efd28133b80 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7efd180098a0 tx=0x7efd1801c400 comp rx=0 tx=0).stop 2024-09-16T03:42:28.930 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.825+0000 7faa3cff9640 10 client.4241.objecter ms_handle_connect 0x7faa2c03d380 2024-09-16T03:42:28.930 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7fcfe4ff9640 10 client.4269.objecter ms_handle_connect 0x7fcfd4037810 2024-09-16T03:42:28.930 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7fcfe4ff9640 1 -- 172.21.15.42:0/131202167 <== osd.0 v2:172.21.15.42:6808/1401549752 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fcfe8133b80 con 0x7fcfe81d05c0 2024-09-16T03:42:28.930 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7fcfe4ff9640 10 client.4269.objecter ms_dispatch 0x7fcfe8000ba0 command_reply(tid 1: 0 ) 2024-09-16T03:42:28.930 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7fcfe4ff9640 10 client.4269.objecter _finish_command 1 = system:0 2024-09-16T03:42:28.930 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7fcfe4ff9640 20 client.4269.objecter put_session s=0x7fcfe81342d0 osd=0 4 2024-09-16T03:42:28.930 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.821+0000 7fcfe4ff9640 15 client.4269.objecter _session_command_op_remove 0 1 2024-09-16T03:42:28.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.825+0000 7efd2cc9d640 1 -- 172.21.15.42:0/491001221 shutdown_connections 2024-09-16T03:42:28.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.825+0000 7efd2cc9d640 1 --2- 172.21.15.42:0/491001221 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7efd280a39b0 0x7efd28133b80 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:28.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.825+0000 7efd2cc9d640 1 -- 172.21.15.42:0/491001221 >> 172.21.15.42:0/491001221 conn(0x7efd280a9fb0 msgr2=0x7efd280aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:42:28.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.825+0000 7faa3ffff640 1 --2- 172.21.15.42:0/2993064549 >> v2:172.21.15.154:6812/254845342 conn(0x7faa2c037870 0x7faa2c039d30 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7faa3001c430 tx=0x7faa300a7000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:28.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.825+0000 7faa3cff9640 10 client.4241.objecter ms_handle_connect 0x7faa2c037870 2024-09-16T03:42:28.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.825+0000 7efd2cc9d640 1 -- 172.21.15.42:0/491001221 shutdown_connections 2024-09-16T03:42:28.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.825+0000 7efd2cc9d640 1 -- 172.21.15.42:0/491001221 wait complete. 2024-09-16T03:42:28.933 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.825+0000 7fcfecff9640 1 -- 172.21.15.42:0/677881880 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fcfe800ac30 con 0x7fcff0137540 2024-09-16T03:42:28.933 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.825+0000 7fcff6396640 10 client.4244.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:42:28.933 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.825+0000 7fcff6396640 15 client.4244.objecter _session_command_op_assign -1 1 2024-09-16T03:42:28.933 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.825+0000 7fcff6396640 20 client.4244.objecter _get_session osd=-1 returning homeless 2024-09-16T03:42:28.933 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.825+0000 7fcff6396640 10 client.4244.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:42:28.933 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.825+0000 7fcff6396640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:42:28.934 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.825+0000 7fcff6396640 1 -- 172.21.15.42:0/677881880 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fcff0133b80 con 0x7fcff0137540 2024-09-16T03:42:28.934 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.825+0000 7fcfecff9640 1 --2- 172.21.15.42:0/677881880 >> v2:172.21.15.154:6812/254845342 conn(0x7fcfdc037810 0x7fcfdc039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:28.934 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.825+0000 7fcfecff9640 1 -- 172.21.15.42:0/677881880 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(18..18 src has 1..18) ==== 3562+0+0 (secure 0 0 0) 0x7fcfe80488d0 con 0x7fcff0137540 2024-09-16T03:42:28.934 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.825+0000 7fcfecff9640 10 client.4244.objecter ms_dispatch 0x7fcff0000ba0 osd_map(18..18 src has 1..18) 2024-09-16T03:42:28.934 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.825+0000 7fcfecff9640 3 client.4244.objecter handle_osd_map got epochs [18,18] > 0 2024-09-16T03:42:28.934 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.825+0000 7fcfecff9640 3 client.4244.objecter handle_osd_map decoding full epoch 18 2024-09-16T03:42:28.934 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.825+0000 7fcfecff9640 10 client.4244.objecter checking command 1 2024-09-16T03:42:28.934 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.825+0000 7fcfecff9640 1 --2- 172.21.15.42:0/677881880 >> v2:172.21.15.154:6804/1038452759 conn(0x7fcfdc03d2b0 0x7fcfdc03f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:28.934 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.825+0000 7fcfecff9640 20 client.4244.objecter _get_session s=0x7fcfdc03cc00 osd=4 3 2024-09-16T03:42:28.934 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.825+0000 7fcfecff9640 20 client.4244.objecter put_session s=0x7fcfdc03cc00 osd=4 3 2024-09-16T03:42:28.934 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.825+0000 7fcfecff9640 15 client.4244.objecter _session_command_op_remove -1 1 2024-09-16T03:42:28.934 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.825+0000 7fcfecff9640 20 client.4244.objecter _get_session s=0x7fcfdc03cc00 osd=4 3 2024-09-16T03:42:28.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.825+0000 7fcfecff9640 20 client.4244.objecter get_session s=0x7fcfdc03cc00 osd=4 3 2024-09-16T03:42:28.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.825+0000 7fcfecff9640 15 client.4244.objecter _session_command_op_assign 4 1 2024-09-16T03:42:28.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.825+0000 7fcfecff9640 20 client.4244.objecter put_session s=0x7fcfdc03cc00 osd=4 4 2024-09-16T03:42:28.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.825+0000 7fcfecff9640 10 client.4244.objecter _send_command 1 2024-09-16T03:42:28.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.825+0000 7fcfecff9640 1 -- 172.21.15.42:0/677881880 --> v2:172.21.15.154:6804/1038452759 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fcfe8010e50 con 0x7fcfdc03d2b0 2024-09-16T03:42:28.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.825+0000 7fcfecff9640 20 client.4244.objecter dump_active .. 0 homeless 2024-09-16T03:42:28.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.825+0000 7fcfecff9640 1 -- 172.21.15.42:0/677881880 <== mon.2 v2:172.21.15.42:3301/0 6 ==== mon_get_version_reply(handle=1 version=18) ==== 24+0+0 (secure 0 0 0) 0x7fcfe8019710 con 0x7fcff0137540 2024-09-16T03:42:28.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.825+0000 7fcfecff9640 10 monclient: handle_get_version_reply finishing 1 version 18 2024-09-16T03:42:28.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.825+0000 7fcfeffff640 1 --2- 172.21.15.42:0/677881880 >> v2:172.21.15.154:6812/254845342 conn(0x7fcfdc037810 0x7fcfdc039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:28.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.825+0000 7efd2cc9d640 10 monclient: build_initial_monmap 2024-09-16T03:42:28.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.825+0000 7efd2cc9d640 10 monclient: monmap: 2024-09-16T03:42:28.936 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:42:28.936 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:42:28.936 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:42:28.936 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:42:28.936 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:42:28.936 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:28.936 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:42:28.936 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:42:28.936 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:42:28.937 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:28.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.825+0000 7fcfeffff640 10 monclient: get_auth_request con 0x7fcfdc037810 auth_method 0 2024-09-16T03:42:28.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.825+0000 7efd2cc9d640 1 librados: starting msgr at 2024-09-16T03:42:28.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.825+0000 7efd2cc9d640 1 librados: starting objecter 2024-09-16T03:42:28.938 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.825+0000 7efd2cc9d640 1 Processor -- start 2024-09-16T03:42:28.938 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.825+0000 7faa3cff9640 1 -- 172.21.15.42:0/2993064549 <== osd.1 v2:172.21.15.42:6800/4217863575 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7faa34016ec0 con 0x7faa2c03d380 2024-09-16T03:42:28.938 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.825+0000 7faa3cff9640 10 client.4241.objecter ms_dispatch 0x7faa40000ba0 command_reply(tid 1: 0 ) 2024-09-16T03:42:28.938 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.825+0000 7faa3cff9640 10 client.4241.objecter _finish_command 1 = system:0 2024-09-16T03:42:28.938 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.825+0000 7faa3cff9640 20 client.4241.objecter put_session s=0x7faa2c03ccd0 osd=1 4 2024-09-16T03:42:28.938 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.825+0000 7faa3cff9640 15 client.4241.objecter _session_command_op_remove 1 1 2024-09-16T03:42:28.938 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.825+0000 7efd2cc9d640 1 -- start start 2024-09-16T03:42:28.938 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.825+0000 7efd2cc9d640 1 librados: setting wanted keys 2024-09-16T03:42:28.939 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.825+0000 7efd2cc9d640 1 librados: calling monclient init 2024-09-16T03:42:28.939 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.825+0000 7efd2cc9d640 10 monclient: init 2024-09-16T03:42:28.939 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.825+0000 7fcfef7fe640 1 --2- 172.21.15.42:0/677881880 >> v2:172.21.15.154:6804/1038452759 conn(0x7fcfdc03d2b0 0x7fcfdc03f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:28.939 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.825+0000 7fcfef7fe640 10 monclient: get_auth_request con 0x7fcfdc03d2b0 auth_method 0 2024-09-16T03:42:28.940 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.825+0000 7efd2cc9d640 10 monclient: _reopen_session rank -1 2024-09-16T03:42:28.940 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.825+0000 7efd2cc9d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:42:28.940 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.825+0000 7efd2cc9d640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efd280a39b0 0x7efd28130cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:28.941 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.825+0000 7efd2cc9d640 10 monclient(hunting): picked mon.noname-a con 0x7efd280a39b0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:42:28.941 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.825+0000 7efd2cc9d640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7efd280a3d90 0x7efd280a1640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:28.941 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.825+0000 7efd2cc9d640 10 monclient(hunting): picked mon.noname-c con 0x7efd280a3d90 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:42:28.941 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.825+0000 7efd2cc9d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efd28137710 0x7efd280a1b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:28.941 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.825+0000 7efd2cc9d640 10 monclient(hunting): picked mon.noname-b con 0x7efd28137710 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:42:28.941 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.825+0000 7efd2cc9d640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:28.941 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.825+0000 7efd2cc9d640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7efd28143900 con 0x7efd280a39b0 2024-09-16T03:42:28.941 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.825+0000 7efd2cc9d640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:28.941 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.825+0000 7efd2cc9d640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7efd2806dee0 con 0x7efd28137710 2024-09-16T03:42:28.941 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.825+0000 7efd2cc9d640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:28.941 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.825+0000 7efd2cc9d640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7efd28143a80 con 0x7efd280a3d90 2024-09-16T03:42:28.941 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.825+0000 7efd2cc9d640 10 monclient(hunting): _renew_subs 2024-09-16T03:42:28.941 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.825+0000 7efd2cc9d640 10 monclient(hunting): authenticate will time out at 995.798095s 2024-09-16T03:42:28.941 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.825+0000 7efd2659c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efd280a39b0 0x7efd28130cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:28.942 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.825+0000 7efd2659c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efd280a39b0 0x7efd28130cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:37646/0 (socket says 172.21.15.42:37646) 2024-09-16T03:42:28.942 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.825+0000 7efd2659c640 1 -- 172.21.15.42:0/2080891320 learned_addr learned my addr 172.21.15.42:0/2080891320 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:42:28.942 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.825+0000 7efd2659c640 10 monclient(hunting): get_auth_request con 0x7efd280a39b0 auth_method 0 2024-09-16T03:42:28.942 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.825+0000 7efd2659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:28.942 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.825+0000 7efd2659c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:28.942 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.825+0000 7efd2659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:28.942 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.825+0000 7efd2659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:28.942 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.825+0000 7efd2659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:28.942 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.825+0000 7efd1dd9b640 1 --2- 172.21.15.42:0/2080891320 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7efd280a3d90 0x7efd280a1640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:28.942 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.825+0000 7efd2659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:28.942 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.825+0000 7efd1dd9b640 10 monclient(hunting): get_auth_request con 0x7efd280a3d90 auth_method 0 2024-09-16T03:42:28.942 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.825+0000 7efd1dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:28.942 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.825+0000 7efd1dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:28.942 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.825+0000 7efd1dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:28.943 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.825+0000 7efd1dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:28.943 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.825+0000 7efd1dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:28.943 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.825+0000 7efd1dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:28.943 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.825+0000 7efd26d9d640 1 --2- 172.21.15.42:0/2080891320 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efd28137710 0x7efd280a1b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:28.943 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.825+0000 7efd26d9d640 10 monclient(hunting): get_auth_request con 0x7efd28137710 auth_method 0 2024-09-16T03:42:28.943 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.825+0000 7efd26d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:28.943 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.825+0000 7efd26d9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:28.943 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.825+0000 7efd26d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:28.943 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.825+0000 7efd2659c640 10 monclient(hunting): handle_auth_done global_id 4278 payload 824 2024-09-16T03:42:28.943 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.825+0000 7efd2659c640 1 -- 172.21.15.42:0/2080891320 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7efd280a3d90 msgr2=0x7efd280a1640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:28.943 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.825+0000 7efd2659c640 1 --2- 172.21.15.42:0/2080891320 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7efd280a3d90 0x7efd280a1640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:28.943 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.825+0000 7efd2659c640 1 -- 172.21.15.42:0/2080891320 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efd28137710 msgr2=0x7efd280a1b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:28.943 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.825+0000 7efd2659c640 1 --2- 172.21.15.42:0/2080891320 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efd28137710 0x7efd280a1b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:28.943 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.825+0000 7efd2659c640 10 monclient: _finish_hunting 0 2024-09-16T03:42:28.944 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.825+0000 7efd2659c640 1 monclient: found mon.noname-a 2024-09-16T03:42:28.944 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.825+0000 7efd2659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:42:28.944 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.825+0000 7efd2659c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:42:28.944 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.825+0000 7efd2659c640 1 -- 172.21.15.42:0/2080891320 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd28071090 con 0x7efd280a39b0 2024-09-16T03:42:28.944 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.825+0000 7fcfef7fe640 1 --2- 172.21.15.42:0/677881880 >> v2:172.21.15.154:6804/1038452759 conn(0x7fcfdc03d2b0 0x7fcfdc03f6f0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:28.945 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.825+0000 7efd2659c640 1 --2- 172.21.15.42:0/2080891320 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efd280a39b0 0x7efd28130cf0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7efd18019010 tx=0x7efd180191a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:28.945 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.825+0000 7efd1f7fe640 10 client.?.objecter ms_handle_connect 0x7efd280a39b0 2024-09-16T03:42:28.945 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.825+0000 7efd1f7fe640 10 client.?.objecter resend_mon_ops 2024-09-16T03:42:28.945 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.825+0000 7fcfecff9640 10 client.4244.objecter ms_handle_connect 0x7fcfdc03d2b0 2024-09-16T03:42:28.946 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.825+0000 7efd1f7fe640 1 -- 172.21.15.42:0/2080891320 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd180b0050 con 0x7efd280a39b0 2024-09-16T03:42:28.947 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.825+0000 7efd1f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:28.947 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.825+0000 7efd1f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:42:28.947 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.825+0000 7efd1f7fe640 10 monclient: dump: 2024-09-16T03:42:28.947 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:28.947 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:28.947 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:28.947 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:28.947 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:28.947 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:28.947 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:28.947 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:28.947 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:28.947 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:28.948 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.825+0000 7efd1f7fe640 10 monclient: _finish_auth 0 2024-09-16T03:42:28.948 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.825+0000 7efd1f7fe640 10 monclient: _check_auth_tickets 2024-09-16T03:42:28.948 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.825+0000 7efd1f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:42:28.948 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.825+0000 7efd2cc9d640 5 monclient: authenticate success, global_id 4278 2024-09-16T03:42:28.948 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.825+0000 7efd2cc9d640 10 monclient: _renew_subs 2024-09-16T03:42:28.948 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.825+0000 7efd2cc9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:42:28.948 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.825+0000 7efd2cc9d640 1 -- 172.21.15.42:0/2080891320 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd280a36e0 con 0x7efd280a39b0 2024-09-16T03:42:28.948 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.825+0000 7efd2cc9d640 10 client.4278.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:42:28.948 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.825+0000 7efd2cc9d640 10 monclient: _renew_subs 2024-09-16T03:42:28.948 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.825+0000 7efd2cc9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:42:28.948 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.825+0000 7efd2cc9d640 1 -- 172.21.15.42:0/2080891320 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd28136e90 con 0x7efd280a39b0 2024-09-16T03:42:28.948 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.825+0000 7efd2cc9d640 1 librados: init done 2024-09-16T03:42:28.948 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.825+0000 7efd1f7fe640 1 -- 172.21.15.42:0/2080891320 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efd180b1040 con 0x7efd280a39b0 2024-09-16T03:42:28.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.829+0000 7efd1f7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:42:28.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.829+0000 7efd2559a640 4 set_mon_vals no callback set 2024-09-16T03:42:28.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.829+0000 7efd1cd99640 10 client.4278.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:42:28.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.829+0000 7efd1cd99640 15 client.4278.objecter _session_command_op_assign -1 1 2024-09-16T03:42:28.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.829+0000 7efd1cd99640 20 client.4278.objecter _get_session osd=-1 returning homeless 2024-09-16T03:42:28.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.829+0000 7efd1cd99640 10 client.4278.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:42:28.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.829+0000 7efd1f7fe640 1 -- 172.21.15.42:0/2080891320 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd180b4810 con 0x7efd280a39b0 2024-09-16T03:42:28.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.829+0000 7fcfecff9640 1 -- 172.21.15.42:0/677881880 <== osd.4 v2:172.21.15.154:6804/1038452759 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fcfe8010e50 con 0x7fcfdc03d2b0 2024-09-16T03:42:28.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.829+0000 7fcfecff9640 10 client.4244.objecter ms_dispatch 0x7fcff0000ba0 command_reply(tid 1: 0 ) 2024-09-16T03:42:28.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.829+0000 7fcfecff9640 10 client.4244.objecter _finish_command 1 = system:0 2024-09-16T03:42:28.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.829+0000 7efd1cd99640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:42:28.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.829+0000 7efd1cd99640 1 -- 172.21.15.42:0/2080891320 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7efd0c002920 con 0x7efd280a39b0 2024-09-16T03:42:28.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.829+0000 7efd1f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:28.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.829+0000 7fcfecff9640 20 client.4244.objecter put_session s=0x7fcfdc03cc00 osd=4 4 2024-09-16T03:42:28.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.829+0000 7fcfecff9640 15 client.4244.objecter _session_command_op_remove 4 1 2024-09-16T03:42:28.952 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.829+0000 7efd1f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:42:28.952 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.829+0000 7efd1f7fe640 10 monclient: dump: 2024-09-16T03:42:28.953 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:28.953 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:28.953 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:28.953 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:28.953 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:28.953 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:28.953 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:28.953 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:28.953 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:28.953 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:28.953 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.829+0000 7efd1f7fe640 1 -- 172.21.15.42:0/2080891320 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7efd180bc020 con 0x7efd280a39b0 2024-09-16T03:42:28.953 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.829+0000 7efd1f7fe640 1 --2- 172.21.15.42:0/2080891320 >> v2:172.21.15.154:6812/254845342 conn(0x7efd08037840 0x7efd08039d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:28.953 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.829+0000 7efd1f7fe640 1 -- 172.21.15.42:0/2080891320 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 3562+0+0 (secure 0 0 0) 0x7efd180e2280 con 0x7efd280a39b0 2024-09-16T03:42:28.954 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.829+0000 7efd1f7fe640 10 client.4278.objecter ms_dispatch 0x7efd28000ba0 osd_map(18..18 src has 1..18) 2024-09-16T03:42:28.954 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.829+0000 7efd1f7fe640 3 client.4278.objecter handle_osd_map got epochs [18,18] > 0 2024-09-16T03:42:28.954 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.829+0000 7efd1f7fe640 3 client.4278.objecter handle_osd_map decoding full epoch 18 2024-09-16T03:42:28.954 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.829+0000 7efd1f7fe640 10 client.4278.objecter checking command 1 2024-09-16T03:42:28.954 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.829+0000 7efd1f7fe640 1 --2- 172.21.15.42:0/2080891320 >> v2:172.21.15.154:6800/4043102363 conn(0x7efd0803d450 0x7efd0803f890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:28.954 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.829+0000 7efd1f7fe640 20 client.4278.objecter _get_session s=0x7efd0803cda0 osd=3 3 2024-09-16T03:42:28.954 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.829+0000 7efd1f7fe640 20 client.4278.objecter put_session s=0x7efd0803cda0 osd=3 3 2024-09-16T03:42:28.954 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.829+0000 7efd1f7fe640 15 client.4278.objecter _session_command_op_remove -1 1 2024-09-16T03:42:28.954 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.829+0000 7efd1f7fe640 20 client.4278.objecter _get_session s=0x7efd0803cda0 osd=3 3 2024-09-16T03:42:28.954 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.829+0000 7efd1f7fe640 20 client.4278.objecter get_session s=0x7efd0803cda0 osd=3 3 2024-09-16T03:42:28.954 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.829+0000 7efd1f7fe640 15 client.4278.objecter _session_command_op_assign 3 1 2024-09-16T03:42:28.954 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.829+0000 7efd1f7fe640 20 client.4278.objecter put_session s=0x7efd0803cda0 osd=3 4 2024-09-16T03:42:28.955 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.829+0000 7efd1f7fe640 10 client.4278.objecter _send_command 1 2024-09-16T03:42:28.955 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.829+0000 7efd1f7fe640 1 -- 172.21.15.42:0/2080891320 --> v2:172.21.15.154:6800/4043102363 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7efd180ab600 con 0x7efd0803d450 2024-09-16T03:42:28.955 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.829+0000 7efd1f7fe640 20 client.4278.objecter dump_active .. 0 homeless 2024-09-16T03:42:28.955 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.829+0000 7efd1f7fe640 1 -- 172.21.15.42:0/2080891320 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_get_version_reply(handle=1 version=18) ==== 24+0+0 (secure 0 0 0) 0x7efd180e9050 con 0x7efd280a39b0 2024-09-16T03:42:28.955 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.829+0000 7efd1dd9b640 1 --2- 172.21.15.42:0/2080891320 >> v2:172.21.15.154:6812/254845342 conn(0x7efd08037840 0x7efd08039d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:28.955 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.829+0000 7efd1f7fe640 10 monclient: handle_get_version_reply finishing 1 version 18 2024-09-16T03:42:28.955 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.829+0000 7efd1dd9b640 10 monclient: get_auth_request con 0x7efd08037840 auth_method 0 2024-09-16T03:42:28.955 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.829+0000 7efd26d9d640 1 --2- 172.21.15.42:0/2080891320 >> v2:172.21.15.154:6800/4043102363 conn(0x7efd0803d450 0x7efd0803f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:28.955 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.829+0000 7efd26d9d640 10 monclient: get_auth_request con 0x7efd0803d450 auth_method 0 2024-09-16T03:42:28.955 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.829+0000 7efd1f7fe640 10 client.4278.objecter ms_handle_connect 0x7efd08037840 2024-09-16T03:42:28.955 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.829+0000 7efd1dd9b640 1 --2- 172.21.15.42:0/2080891320 >> v2:172.21.15.154:6812/254845342 conn(0x7efd08037840 0x7efd08039d00 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7efd28132a40 tx=0x7efd10009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:28.955 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.829+0000 7efd26d9d640 1 --2- 172.21.15.42:0/2080891320 >> v2:172.21.15.154:6800/4043102363 conn(0x7efd0803d450 0x7efd0803f890 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:28.955 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.829+0000 7efd1f7fe640 10 client.4278.objecter ms_handle_connect 0x7efd0803d450 2024-09-16T03:42:28.956 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.833+0000 7efd1f7fe640 1 -- 172.21.15.42:0/2080891320 <== osd.3 v2:172.21.15.154:6800/4043102363 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7efd180ab600 con 0x7efd0803d450 2024-09-16T03:42:28.956 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.833+0000 7fcfecff9640 10 client.4244.objecter ms_handle_connect 0x7fcfdc037810 2024-09-16T03:42:28.956 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.833+0000 7fcfeffff640 1 --2- 172.21.15.42:0/677881880 >> v2:172.21.15.154:6812/254845342 conn(0x7fcfdc037810 0x7fcfdc039cd0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fcfe00ad000 tx=0x7fcfe00a6000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:28.956 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.833+0000 7efd1f7fe640 10 client.4278.objecter ms_dispatch 0x7efd28000ba0 command_reply(tid 1: 0 ) 2024-09-16T03:42:28.956 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.833+0000 7efd1f7fe640 10 client.4278.objecter _finish_command 1 = system:0 2024-09-16T03:42:28.956 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.833+0000 7efd1f7fe640 20 client.4278.objecter put_session s=0x7efd0803cda0 osd=3 4 2024-09-16T03:42:28.956 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.833+0000 7efd1f7fe640 15 client.4278.objecter _session_command_op_remove 3 1 2024-09-16T03:42:28.957 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.837+0000 7fcfee6aa640 10 client.4269.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-09-16T03:42:28.957 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.837+0000 7fcfee6aa640 15 client.4269.objecter _session_command_op_assign -1 2 2024-09-16T03:42:28.957 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.837+0000 7fcfee6aa640 20 client.4269.objecter _get_session s=0x7fcfe81342d0 osd=0 3 2024-09-16T03:42:28.957 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.837+0000 7fcfee6aa640 20 client.4269.objecter put_session s=0x7fcfe81342d0 osd=0 3 2024-09-16T03:42:28.957 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.837+0000 7fcfee6aa640 20 client.4269.objecter _get_session s=0x7fcfe81342d0 osd=0 3 2024-09-16T03:42:28.957 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.837+0000 7fcfee6aa640 15 client.4269.objecter _session_command_op_remove -1 2 2024-09-16T03:42:28.958 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.837+0000 7fcfee6aa640 20 client.4269.objecter get_session s=0x7fcfe81342d0 osd=0 3 2024-09-16T03:42:28.958 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.837+0000 7fcfee6aa640 15 client.4269.objecter _session_command_op_assign 0 2 2024-09-16T03:42:28.958 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.837+0000 7fcfee6aa640 20 client.4269.objecter put_session s=0x7fcfe81342d0 osd=0 4 2024-09-16T03:42:28.958 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.837+0000 7fcfee6aa640 10 client.4269.objecter _send_command 2 2024-09-16T03:42:28.958 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.837+0000 7fcfee6aa640 1 -- 172.21.15.42:0/131202167 --> v2:172.21.15.42:6808/1401549752 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fcfe80a5750 con 0x7fcfe81d05c0 2024-09-16T03:42:28.958 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.841+0000 7fcfe4ff9640 1 -- 172.21.15.42:0/131202167 <== osd.0 v2:172.21.15.42:6808/1401549752 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fcfe80a5750 con 0x7fcfe81d05c0 2024-09-16T03:42:28.958 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.841+0000 7fcfe4ff9640 10 client.4269.objecter ms_dispatch 0x7fcfe8000ba0 command_reply(tid 2: 0 ) 2024-09-16T03:42:28.958 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.841+0000 7fcfe4ff9640 10 client.4269.objecter _finish_command 2 = system:0 2024-09-16T03:42:28.958 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.841+0000 7fcfe4ff9640 20 client.4269.objecter put_session s=0x7fcfe81342d0 osd=0 4 2024-09-16T03:42:28.958 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.841+0000 7fcfe4ff9640 15 client.4269.objecter _session_command_op_remove 0 2 2024-09-16T03:42:28.959 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.841+0000 7fcfee6aa640 10 librados: watch_flush enter 2024-09-16T03:42:28.959 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.841+0000 7fcfee6aa640 10 librados: watch_flush exit 2024-09-16T03:42:28.959 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.841+0000 7fcfee6aa640 10 client.4269.objecter close_session for osd.0 2024-09-16T03:42:28.959 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.841+0000 7fcfee6aa640 1 -- 172.21.15.42:0/131202167 >> v2:172.21.15.42:6808/1401549752 conn(0x7fcfe81d05c0 msgr2=0x7fcfe802c650 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:28.959 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.841+0000 7fcfee6aa640 1 --2- 172.21.15.42:0/131202167 >> v2:172.21.15.42:6808/1401549752 conn(0x7fcfe81d05c0 0x7fcfe802c650 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:28.959 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.841+0000 7fcfee6aa640 20 client.4269.objecter put_session s=0x7fcfe81342d0 osd=0 1 2024-09-16T03:42:28.959 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.841+0000 7fcfee6aa640 20 client.4269.objecter shutdown clearing up homeless session... 2024-09-16T03:42:28.960 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.841+0000 7fcfee6aa640 10 client.4269.objecter successfully canceled tick 2024-09-16T03:42:28.960 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.841+0000 7fcfee6aa640 1 -- 172.21.15.42:0/131202167 >> v2:172.21.15.154:6812/254845342 conn(0x7fcfd4037810 msgr2=0x7fcfd4039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:28.960 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.841+0000 7fcfee6aa640 1 --2- 172.21.15.42:0/131202167 >> v2:172.21.15.154:6812/254845342 conn(0x7fcfd4037810 0x7fcfd4039cd0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fcfd801c3e0 tx=0x7fcfd8008040 comp rx=0 tx=0).stop 2024-09-16T03:42:28.960 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.841+0000 7fcfee6aa640 10 monclient: shutdown 2024-09-16T03:42:28.960 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.841+0000 7fcfee6aa640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:42:28.960 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.841+0000 7fcfee6aa640 1 -- 172.21.15.42:0/131202167 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcfe80a3d90 msgr2=0x7fcfe813d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:28.960 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.841+0000 7fcfee6aa640 1 --2- 172.21.15.42:0/131202167 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcfe80a3d90 0x7fcfe813d790 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fcfdc00b3f0 tx=0x7fcfdc00b8b0 comp rx=0 tx=0).stop 2024-09-16T03:42:28.960 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.841+0000 7fcfecc47640 1 -- 172.21.15.42:0/131202167 reap_dead start 2024-09-16T03:42:28.960 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.841+0000 7fcfee6aa640 1 -- 172.21.15.42:0/131202167 shutdown_connections 2024-09-16T03:42:28.960 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.841+0000 7fcfee6aa640 1 -- 172.21.15.42:0/131202167 >> 172.21.15.42:0/131202167 conn(0x7fcfe80a9fb0 msgr2=0x7fcfe8131db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:42:28.960 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.841+0000 7fcfee6aa640 1 -- 172.21.15.42:0/131202167 shutdown_connections 2024-09-16T03:42:28.960 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.841+0000 7fcfee6aa640 1 -- 172.21.15.42:0/131202167 wait complete. 2024-09-16T03:42:28.960 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.841+0000 7fcfee6aa640 1 librados: shutdown 2024-09-16T03:42:28.961 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.841+0000 7faa466ad640 10 client.4241.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-09-16T03:42:28.961 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.841+0000 7faa466ad640 15 client.4241.objecter _session_command_op_assign -1 2 2024-09-16T03:42:28.961 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.841+0000 7faa466ad640 20 client.4241.objecter _get_session s=0x7faa2c03ccd0 osd=1 3 2024-09-16T03:42:28.961 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.841+0000 7faa466ad640 20 client.4241.objecter put_session s=0x7faa2c03ccd0 osd=1 3 2024-09-16T03:42:28.961 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.841+0000 7faa466ad640 20 client.4241.objecter _get_session s=0x7faa2c03ccd0 osd=1 3 2024-09-16T03:42:28.961 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.841+0000 7faa466ad640 15 client.4241.objecter _session_command_op_remove -1 2 2024-09-16T03:42:28.961 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.841+0000 7faa466ad640 20 client.4241.objecter get_session s=0x7faa2c03ccd0 osd=1 3 2024-09-16T03:42:28.962 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.841+0000 7faa466ad640 15 client.4241.objecter _session_command_op_assign 1 2 2024-09-16T03:42:28.962 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.841+0000 7faa466ad640 20 client.4241.objecter put_session s=0x7faa2c03ccd0 osd=1 4 2024-09-16T03:42:28.962 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.841+0000 7faa466ad640 10 client.4241.objecter _send_command 2 2024-09-16T03:42:28.962 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.841+0000 7faa466ad640 1 -- 172.21.15.42:0/2993064549 --> v2:172.21.15.42:6800/4217863575 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7faa20001230 con 0x7faa2c03d380 2024-09-16T03:42:28.962 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.841+0000 7faa3cff9640 1 -- 172.21.15.42:0/2993064549 <== osd.1 v2:172.21.15.42:6800/4217863575 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7faa20001230 con 0x7faa2c03d380 2024-09-16T03:42:28.962 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.841+0000 7faa3cff9640 10 client.4241.objecter ms_dispatch 0x7faa40000ba0 command_reply(tid 2: 0 ) 2024-09-16T03:42:28.962 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.841+0000 7faa3cff9640 10 client.4241.objecter _finish_command 2 = system:0 2024-09-16T03:42:28.962 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.841+0000 7faa3cff9640 20 client.4241.objecter put_session s=0x7faa2c03ccd0 osd=1 4 2024-09-16T03:42:28.962 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.841+0000 7faa3cff9640 15 client.4241.objecter _session_command_op_remove 1 2 2024-09-16T03:42:28.964 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.841+0000 7faa466ad640 10 librados: watch_flush enter 2024-09-16T03:42:28.964 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.841+0000 7faa466ad640 10 librados: watch_flush exit 2024-09-16T03:42:28.964 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.841+0000 7faa466ad640 10 client.4241.objecter close_session for osd.1 2024-09-16T03:42:28.964 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.841+0000 7faa466ad640 1 -- 172.21.15.42:0/2993064549 >> v2:172.21.15.42:6800/4217863575 conn(0x7faa2c03d380 msgr2=0x7faa2c03f7c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:28.964 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.841+0000 7faa466ad640 1 --2- 172.21.15.42:0/2993064549 >> v2:172.21.15.42:6800/4217863575 conn(0x7faa2c03d380 0x7faa2c03f7c0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:28.964 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.841+0000 7faa466ad640 20 client.4241.objecter put_session s=0x7faa2c03ccd0 osd=1 1 2024-09-16T03:42:28.964 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.841+0000 7faa466ad640 20 client.4241.objecter shutdown clearing up homeless session... 2024-09-16T03:42:28.964 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.841+0000 7faa466ad640 10 client.4241.objecter successfully canceled tick 2024-09-16T03:42:28.964 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.841+0000 7faa466ad640 1 -- 172.21.15.42:0/2993064549 >> v2:172.21.15.154:6812/254845342 conn(0x7faa2c037870 msgr2=0x7faa2c039d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:28.964 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.841+0000 7faa466ad640 1 --2- 172.21.15.42:0/2993064549 >> v2:172.21.15.154:6812/254845342 conn(0x7faa2c037870 0x7faa2c039d30 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7faa3001c430 tx=0x7faa300a7000 comp rx=0 tx=0).stop 2024-09-16T03:42:28.964 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.841+0000 7faa466ad640 10 monclient: shutdown 2024-09-16T03:42:28.964 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.841+0000 7faa466ad640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:42:28.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.841+0000 7faa466ad640 1 -- 172.21.15.42:0/2993064549 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7faa400a67e0 msgr2=0x7faa40136ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:28.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.841+0000 7faa466ad640 1 --2- 172.21.15.42:0/2993064549 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7faa400a67e0 0x7faa40136ef0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7faa34000f70 tx=0x7faa3400c820 comp rx=0 tx=0).stop 2024-09-16T03:42:28.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.841+0000 7faa44c4a640 1 -- 172.21.15.42:0/2993064549 reap_dead start 2024-09-16T03:42:28.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.841+0000 7faa466ad640 1 -- 172.21.15.42:0/2993064549 shutdown_connections 2024-09-16T03:42:28.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.841+0000 7faa466ad640 1 -- 172.21.15.42:0/2993064549 >> 172.21.15.42:0/2993064549 conn(0x7faa400a7ed0 msgr2=0x7faa4013f9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:42:28.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.841+0000 7faa466ad640 1 -- 172.21.15.42:0/2993064549 shutdown_connections 2024-09-16T03:42:28.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.841+0000 7faa466ad640 1 -- 172.21.15.42:0/2993064549 wait complete. 2024-09-16T03:42:28.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.841+0000 7faa466ad640 1 librados: shutdown 2024-09-16T03:42:28.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.841+0000 7fcff6396640 10 client.4244.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-09-16T03:42:28.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.841+0000 7fcff6396640 15 client.4244.objecter _session_command_op_assign -1 2 2024-09-16T03:42:28.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.841+0000 7fcff6396640 20 client.4244.objecter _get_session s=0x7fcfdc03cc00 osd=4 3 2024-09-16T03:42:28.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.841+0000 7fcff6396640 20 client.4244.objecter put_session s=0x7fcfdc03cc00 osd=4 3 2024-09-16T03:42:28.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.841+0000 7fcff6396640 20 client.4244.objecter _get_session s=0x7fcfdc03cc00 osd=4 3 2024-09-16T03:42:28.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.841+0000 7fcff6396640 15 client.4244.objecter _session_command_op_remove -1 2 2024-09-16T03:42:28.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.841+0000 7fcff6396640 20 client.4244.objecter get_session s=0x7fcfdc03cc00 osd=4 3 2024-09-16T03:42:28.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.841+0000 7fcff6396640 15 client.4244.objecter _session_command_op_assign 4 2 2024-09-16T03:42:28.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.841+0000 7fcff6396640 20 client.4244.objecter put_session s=0x7fcfdc03cc00 osd=4 4 2024-09-16T03:42:28.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.841+0000 7fcff6396640 10 client.4244.objecter _send_command 2 2024-09-16T03:42:28.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.841+0000 7fcff6396640 1 -- 172.21.15.42:0/677881880 --> v2:172.21.15.154:6804/1038452759 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fcff00a5750 con 0x7fcfdc03d2b0 2024-09-16T03:42:28.967 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.841+0000 7fcfecff9640 1 -- 172.21.15.42:0/677881880 <== osd.4 v2:172.21.15.154:6804/1038452759 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fcff00a5750 con 0x7fcfdc03d2b0 2024-09-16T03:42:28.967 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.841+0000 7fcfecff9640 10 client.4244.objecter ms_dispatch 0x7fcff0000ba0 command_reply(tid 2: 0 ) 2024-09-16T03:42:28.967 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.841+0000 7fcfecff9640 10 client.4244.objecter _finish_command 2 = system:0 2024-09-16T03:42:28.967 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.845+0000 7fcfecff9640 20 client.4244.objecter put_session s=0x7fcfdc03cc00 osd=4 4 2024-09-16T03:42:28.967 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.845+0000 7fcfecff9640 15 client.4244.objecter _session_command_op_remove 4 2 2024-09-16T03:42:28.968 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.845+0000 7fcff6396640 10 librados: watch_flush enter 2024-09-16T03:42:28.968 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.845+0000 7fcff6396640 10 librados: watch_flush exit 2024-09-16T03:42:28.968 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.845+0000 7fcff6396640 10 client.4244.objecter close_session for osd.4 2024-09-16T03:42:28.969 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.845+0000 7fcff6396640 1 -- 172.21.15.42:0/677881880 >> v2:172.21.15.154:6804/1038452759 conn(0x7fcfdc03d2b0 msgr2=0x7fcfdc03f6f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:28.969 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.845+0000 7fcff6396640 1 --2- 172.21.15.42:0/677881880 >> v2:172.21.15.154:6804/1038452759 conn(0x7fcfdc03d2b0 0x7fcfdc03f6f0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:28.969 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.845+0000 7fcff6396640 20 client.4244.objecter put_session s=0x7fcfdc03cc00 osd=4 1 2024-09-16T03:42:28.969 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.845+0000 7fcff6396640 20 client.4244.objecter shutdown clearing up homeless session... 2024-09-16T03:42:28.969 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.845+0000 7fcff6396640 10 client.4244.objecter successfully canceled tick 2024-09-16T03:42:28.969 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.845+0000 7fcff6396640 1 -- 172.21.15.42:0/677881880 >> v2:172.21.15.154:6812/254845342 conn(0x7fcfdc037810 msgr2=0x7fcfdc039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:28.969 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.845+0000 7fcff6396640 1 --2- 172.21.15.42:0/677881880 >> v2:172.21.15.154:6812/254845342 conn(0x7fcfdc037810 0x7fcfdc039cd0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fcfe00ad000 tx=0x7fcfe00a6000 comp rx=0 tx=0).stop 2024-09-16T03:42:28.969 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.845+0000 7fcff6396640 10 monclient: shutdown 2024-09-16T03:42:28.969 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.845+0000 7fcff6396640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:42:28.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.845+0000 7fcff6396640 1 -- 172.21.15.42:0/677881880 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fcff0137540 msgr2=0x7fcff009db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:28.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.845+0000 7fcff6396640 1 --2- 172.21.15.42:0/677881880 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fcff0137540 0x7fcff009db90 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fcfe800c780 tx=0x7fcfe800cc40 comp rx=0 tx=0).stop 2024-09-16T03:42:28.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.845+0000 7fcff4933640 1 -- 172.21.15.42:0/677881880 reap_dead start 2024-09-16T03:42:28.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.845+0000 7fcff6396640 1 -- 172.21.15.42:0/677881880 shutdown_connections 2024-09-16T03:42:28.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.845+0000 7fcff6396640 1 -- 172.21.15.42:0/677881880 >> 172.21.15.42:0/677881880 conn(0x7fcff00a9fb0 msgr2=0x7fcff0131980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:42:28.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.845+0000 7fcff6396640 1 -- 172.21.15.42:0/677881880 shutdown_connections 2024-09-16T03:42:28.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.845+0000 7fcff6396640 1 -- 172.21.15.42:0/677881880 wait complete. 2024-09-16T03:42:28.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.845+0000 7fcff6396640 1 librados: shutdown 2024-09-16T03:42:28.971 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.849+0000 7efd1cd99640 10 client.4278.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-09-16T03:42:28.971 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.849+0000 7efd1cd99640 15 client.4278.objecter _session_command_op_assign -1 2 2024-09-16T03:42:28.971 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.849+0000 7efd1cd99640 20 client.4278.objecter _get_session s=0x7efd0803cda0 osd=3 3 2024-09-16T03:42:28.971 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.849+0000 7efd1cd99640 20 client.4278.objecter put_session s=0x7efd0803cda0 osd=3 3 2024-09-16T03:42:28.971 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.849+0000 7efd1cd99640 20 client.4278.objecter _get_session s=0x7efd0803cda0 osd=3 3 2024-09-16T03:42:28.971 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.849+0000 7efd1cd99640 15 client.4278.objecter _session_command_op_remove -1 2 2024-09-16T03:42:28.971 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.849+0000 7efd1cd99640 20 client.4278.objecter get_session s=0x7efd0803cda0 osd=3 3 2024-09-16T03:42:28.972 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.849+0000 7efd1cd99640 15 client.4278.objecter _session_command_op_assign 3 2 2024-09-16T03:42:28.972 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.849+0000 7efd1cd99640 20 client.4278.objecter put_session s=0x7efd0803cda0 osd=3 4 2024-09-16T03:42:28.972 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.849+0000 7efd1cd99640 10 client.4278.objecter _send_command 2 2024-09-16T03:42:28.972 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.849+0000 7efd1cd99640 1 -- 172.21.15.42:0/2080891320 --> v2:172.21.15.154:6800/4043102363 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7efd0c001230 con 0x7efd0803d450 2024-09-16T03:42:28.972 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.853+0000 7efd1f7fe640 1 -- 172.21.15.42:0/2080891320 <== osd.3 v2:172.21.15.154:6800/4043102363 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7efd0c001230 con 0x7efd0803d450 2024-09-16T03:42:28.972 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.853+0000 7efd1f7fe640 10 client.4278.objecter ms_dispatch 0x7efd28000ba0 command_reply(tid 2: 0 ) 2024-09-16T03:42:28.972 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.853+0000 7efd1f7fe640 10 client.4278.objecter _finish_command 2 = system:0 2024-09-16T03:42:28.972 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.853+0000 7efd1f7fe640 20 client.4278.objecter put_session s=0x7efd0803cda0 osd=3 4 2024-09-16T03:42:28.972 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.853+0000 7efd1f7fe640 15 client.4278.objecter _session_command_op_remove 3 2 2024-09-16T03:42:28.973 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.853+0000 7efd1cd99640 10 librados: watch_flush enter 2024-09-16T03:42:28.974 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.853+0000 7efd1cd99640 10 librados: watch_flush exit 2024-09-16T03:42:28.974 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.853+0000 7efd1cd99640 10 client.4278.objecter close_session for osd.3 2024-09-16T03:42:28.974 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.853+0000 7efd1cd99640 1 -- 172.21.15.42:0/2080891320 >> v2:172.21.15.154:6800/4043102363 conn(0x7efd0803d450 msgr2=0x7efd0803f890 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:28.974 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.853+0000 7efd1cd99640 1 --2- 172.21.15.42:0/2080891320 >> v2:172.21.15.154:6800/4043102363 conn(0x7efd0803d450 0x7efd0803f890 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:28.974 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.853+0000 7efd1cd99640 20 client.4278.objecter put_session s=0x7efd0803cda0 osd=3 1 2024-09-16T03:42:28.974 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.853+0000 7efd1cd99640 20 client.4278.objecter shutdown clearing up homeless session... 2024-09-16T03:42:28.974 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.853+0000 7efd1cd99640 10 client.4278.objecter successfully canceled tick 2024-09-16T03:42:28.974 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.853+0000 7efd1cd99640 1 -- 172.21.15.42:0/2080891320 >> v2:172.21.15.154:6812/254845342 conn(0x7efd08037840 msgr2=0x7efd08039d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:28.974 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.853+0000 7efd1cd99640 1 --2- 172.21.15.42:0/2080891320 >> v2:172.21.15.154:6812/254845342 conn(0x7efd08037840 0x7efd08039d00 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7efd28132a40 tx=0x7efd10009040 comp rx=0 tx=0).stop 2024-09-16T03:42:28.974 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.853+0000 7efd1cd99640 10 monclient: shutdown 2024-09-16T03:42:28.974 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.853+0000 7efd1cd99640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:42:28.975 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.853+0000 7efd1cd99640 1 -- 172.21.15.42:0/2080891320 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efd280a39b0 msgr2=0x7efd28130cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:28.975 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.853+0000 7efd1cd99640 1 --2- 172.21.15.42:0/2080891320 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efd280a39b0 0x7efd28130cf0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7efd18019010 tx=0x7efd180191a0 comp rx=0 tx=0).stop 2024-09-16T03:42:28.975 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.853+0000 7efd26d9d640 1 -- 172.21.15.42:0/2080891320 reap_dead start 2024-09-16T03:42:28.975 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.853+0000 7efd1cd99640 1 -- 172.21.15.42:0/2080891320 shutdown_connections 2024-09-16T03:42:28.975 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.853+0000 7efd1cd99640 1 -- 172.21.15.42:0/2080891320 >> 172.21.15.42:0/2080891320 conn(0x7efd280a9fb0 msgr2=0x7efd28132b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:42:28.975 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.853+0000 7efd1cd99640 1 -- 172.21.15.42:0/2080891320 shutdown_connections 2024-09-16T03:42:28.975 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.853+0000 7efd1cd99640 1 -- 172.21.15.42:0/2080891320 wait complete. 2024-09-16T03:42:28.975 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:28.853+0000 7efd1cd99640 1 librados: shutdown 2024-09-16T03:42:28.976 INFO:teuthology.orchestra.run.smithi042.stdout:47244640260 2024-09-16T03:42:28.976 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-16T03:42:28.977 INFO:teuthology.orchestra.run.smithi042.stdout:47244640262 2024-09-16T03:42:28.977 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-16T03:42:28.978 INFO:teuthology.orchestra.run.smithi042.stdout:47244640260 2024-09-16T03:42:28.978 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-16T03:42:28.979 INFO:teuthology.orchestra.run.smithi042.stdout:47244640260 2024-09-16T03:42:28.980 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-16T03:42:29.069 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.061+0000 7f58cb500640 10 monclient: get_monmap_and_config 2024-09-16T03:42:29.069 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.061+0000 7f58cb500640 10 monclient: build_initial_monmap 2024-09-16T03:42:29.069 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.061+0000 7f58cb500640 10 monclient: monmap: 2024-09-16T03:42:29.070 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:42:29.070 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:29.070 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:42:29.067233+0000 2024-09-16T03:42:29.070 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:42:29.067233+0000 2024-09-16T03:42:29.070 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:42:29.070 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:29.070 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:42:29.070 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:42:29.070 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:42:29.070 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:42:29.070 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:42:29.071 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:29.072 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.061+0000 7f58cb500640 1 Processor -- start 2024-09-16T03:42:29.072 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.065+0000 7f58cb500640 1 -- start start 2024-09-16T03:42:29.072 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.065+0000 7f58cb500640 10 monclient: init 2024-09-16T03:42:29.072 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.065+0000 7f58cb500640 10 monclient: _reopen_session rank -1 2024-09-16T03:42:29.072 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.065+0000 7f58cb500640 10 monclient: _add_conns ranks=[2,1,3,0,4] 2024-09-16T03:42:29.072 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.065+0000 7f58cb500640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f58c40a39b0 0x7f58c4133b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:29.072 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.065+0000 7f58cb500640 10 monclient(hunting): picked mon.noname-b con 0x7f58c40a39b0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:42:29.072 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.065+0000 7f58cb500640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f58c40a3d90 addr v1:172.21.15.154:6789/0 2024-09-16T03:42:29.072 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.065+0000 7f58cb500640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f58c4137710 0x7f58c413f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:29.072 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.065+0000 7f58cb500640 10 monclient(hunting): picked mon.noname-a con 0x7f58c4137710 addr v2:172.21.15.42:3300/0 2024-09-16T03:42:29.072 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.065+0000 7f58cb500640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f58c409c140 con 0x7f58c40a3d90 2024-09-16T03:42:29.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.065+0000 7f58cb500640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:29.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.065+0000 7f58cb500640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f58c409c420 con 0x7f58c4137710 2024-09-16T03:42:29.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.065+0000 7f58cb500640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:29.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.065+0000 7f58cb500640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f58c406dd60 con 0x7f58c40a39b0 2024-09-16T03:42:29.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.065+0000 7f58cb500640 10 monclient(hunting): _renew_subs 2024-09-16T03:42:29.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.065+0000 7f58cb500640 10 monclient(hunting): authenticate will time out at 996.036917s 2024-09-16T03:42:29.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.065+0000 7f58c929c640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f58c40a39b0 0x7f58c4133b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:29.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.065+0000 7f58c929c640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f58c40a39b0 0x7f58c4133b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:41354/0 (socket says 172.21.15.42:41354) 2024-09-16T03:42:29.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.065+0000 7f58c929c640 1 -- 172.21.15.42:0/3573601050 learned_addr learned my addr 172.21.15.42:0/3573601050 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:42:29.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.065+0000 7f58c929c640 10 monclient(hunting): get_auth_request con 0x7f58c40a39b0 auth_method 0 2024-09-16T03:42:29.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.065+0000 7f58c929c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:29.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.065+0000 7f58c929c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:29.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.065+0000 7f58c929c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:29.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.065+0000 7f58c9a9d640 1 --2- 172.21.15.42:0/3573601050 >> v2:172.21.15.42:3300/0 conn(0x7f58c4137710 0x7f58c413f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:29.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.065+0000 7f58c9a9d640 10 monclient(hunting): get_auth_request con 0x7f58c4137710 auth_method 0 2024-09-16T03:42:29.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.065+0000 7f58c9a9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:29.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.065+0000 7f58c9a9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:29.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.065+0000 7f58c9a9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:29.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.065+0000 7f58c929c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:29.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.065+0000 7f58c929c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:29.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.065+0000 7f58c929c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:29.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.065+0000 7f58c9a9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:29.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.065+0000 7f58c9a9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:29.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.065+0000 7f58c9a9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:29.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.065+0000 7f58c929c640 10 monclient(hunting): handle_auth_done global_id 4250 payload 274 2024-09-16T03:42:29.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.065+0000 7f58c929c640 1 -- 172.21.15.42:0/3573601050 >> v2:172.21.15.42:3300/0 conn(0x7f58c4137710 msgr2=0x7f58c413f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:42:29.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.065+0000 7f58c929c640 1 --2- 172.21.15.42:0/3573601050 >> v2:172.21.15.42:3300/0 conn(0x7f58c4137710 0x7f58c413f3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:29.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.065+0000 7f58c929c640 1 -- 172.21.15.42:0/3573601050 >> v1:172.21.15.154:6789/0 conn(0x7f58c40a3d90 legacy=0x7f58c4137000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:42:29.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.065+0000 7f58c929c640 10 monclient: _finish_hunting 0 2024-09-16T03:42:29.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.065+0000 7f58c929c640 1 monclient: found mon.noname-b 2024-09-16T03:42:29.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.065+0000 7f58c929c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:42:29.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.065+0000 7f58c929c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T03:42:29.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.065+0000 7f58c929c640 1 -- 172.21.15.42:0/3573601050 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58c40a5e20 con 0x7f58c40a39b0 2024-09-16T03:42:29.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.069+0000 7f58c929c640 1 --2- 172.21.15.42:0/3573601050 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f58c40a39b0 0x7f58c4133b80 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f58c0009870 tx=0x7f58c001c3e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bd8ac1eecb5c2bb3 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:29.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.069+0000 7f58bbfff640 1 -- 172.21.15.42:0/3573601050 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58c00b0070 con 0x7f58c40a39b0 2024-09-16T03:42:29.077 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.069+0000 7f58bbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:29.077 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.069+0000 7f58bbfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:42:29.077 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.069+0000 7f58bbfff640 10 monclient: dump: 2024-09-16T03:42:29.077 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:29.077 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:29.077 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:29.077 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:29.077 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:29.077 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:29.077 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:29.077 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:29.077 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:29.078 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:29.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.069+0000 7f58bbfff640 10 monclient: _finish_auth 0 2024-09-16T03:42:29.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.069+0000 7f58bbfff640 10 monclient: _check_auth_tickets 2024-09-16T03:42:29.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.069+0000 7f58bbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:42:29.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.069+0000 7f58bbfff640 1 -- 172.21.15.42:0/3573601050 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f58c00ab600 con 0x7f58c40a39b0 2024-09-16T03:42:29.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.069+0000 7f58bbfff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:42:29.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.069+0000 7f58bbfff640 1 -- 172.21.15.42:0/3573601050 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58c00ab9d0 con 0x7f58c40a39b0 2024-09-16T03:42:29.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.069+0000 7f58cb500640 5 monclient: authenticate success, global_id 4250 2024-09-16T03:42:29.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.069+0000 7f58cb500640 10 monclient: get_monmap_and_config success 2024-09-16T03:42:29.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.069+0000 7f58cb500640 4 set_mon_vals no callback set 2024-09-16T03:42:29.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.069+0000 7f58bbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:29.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.069+0000 7f58bbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:42:29.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.069+0000 7f58bbfff640 10 monclient: dump: 2024-09-16T03:42:29.079 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:29.079 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:29.079 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:29.079 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:29.079 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:29.080 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:29.080 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:29.080 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:29.080 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:29.080 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:29.080 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.069+0000 7f58cb500640 10 monclient: shutdown 2024-09-16T03:42:29.080 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.069+0000 7f58cb500640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:42:29.080 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.069+0000 7f58cb500640 1 -- 172.21.15.42:0/3573601050 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f58c40a39b0 msgr2=0x7f58c4133b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:29.080 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.069+0000 7f58cb500640 1 --2- 172.21.15.42:0/3573601050 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f58c40a39b0 0x7f58c4133b80 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f58c0009870 tx=0x7f58c001c3e0 comp rx=0 tx=0).stop 2024-09-16T03:42:29.080 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.069+0000 7f58cb500640 1 -- 172.21.15.42:0/3573601050 shutdown_connections 2024-09-16T03:42:29.081 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.069+0000 7f58cb500640 1 --2- 172.21.15.42:0/3573601050 >> v2:172.21.15.42:3300/0 conn(0x7f58c4137710 0x7f58c413f3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:29.081 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.069+0000 7f58cb500640 1 --2- 172.21.15.42:0/3573601050 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f58c40a39b0 0x7f58c4133b80 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:29.081 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.069+0000 7f58cb500640 1 -- 172.21.15.42:0/3573601050 >> 172.21.15.42:0/3573601050 conn(0x7f58c40a9fb0 msgr2=0x7f58c40aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:42:29.081 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.069+0000 7f58cb500640 1 -- 172.21.15.42:0/3573601050 shutdown_connections 2024-09-16T03:42:29.081 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.069+0000 7f58cb500640 1 -- 172.21.15.42:0/3573601050 wait complete. 2024-09-16T03:42:29.085 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.069+0000 7f58cb500640 10 monclient: build_initial_monmap 2024-09-16T03:42:29.085 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.069+0000 7f58cb500640 10 monclient: monmap: 2024-09-16T03:42:29.085 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:42:29.085 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:42:29.085 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:42:29.086 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:42:29.086 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:42:29.086 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:29.086 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:42:29.086 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:42:29.086 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:42:29.086 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:29.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.069+0000 7f58cb500640 1 librados: starting msgr at 2024-09-16T03:42:29.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.069+0000 7f58cb500640 1 librados: starting objecter 2024-09-16T03:42:29.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.073+0000 7f58cb500640 1 Processor -- start 2024-09-16T03:42:29.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.073+0000 7f58cb500640 1 -- start start 2024-09-16T03:42:29.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.073+0000 7f58cb500640 1 librados: setting wanted keys 2024-09-16T03:42:29.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.073+0000 7f58cb500640 1 librados: calling monclient init 2024-09-16T03:42:29.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.073+0000 7f58cb500640 10 monclient: init 2024-09-16T03:42:29.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.073+0000 7f58cb500640 10 monclient: _reopen_session rank -1 2024-09-16T03:42:29.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.073+0000 7f58cb500640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:42:29.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.073+0000 7f58cb500640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f58c40a39b0 0x7f58c413b990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:29.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.073+0000 7f58cb500640 10 monclient(hunting): picked mon.noname-a con 0x7f58c40a39b0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:42:29.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.073+0000 7f58cb500640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f58c40a3d90 0x7f58c4139180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:29.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.073+0000 7f58cb500640 10 monclient(hunting): picked mon.noname-c con 0x7f58c40a3d90 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:42:29.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.073+0000 7f58cb500640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f58c4137710 0x7f58c41396c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:29.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.073+0000 7f58c929c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f58c40a39b0 0x7f58c413b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:29.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.073+0000 7f58c929c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f58c40a39b0 0x7f58c413b990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:37672/0 (socket says 172.21.15.42:37672) 2024-09-16T03:42:29.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.073+0000 7f58c8a9b640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f58c40a3d90 0x7f58c4139180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:29.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.073+0000 7f58c8a9b640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f58c40a3d90 0x7f58c4139180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:41358/0 (socket says 172.21.15.42:41358) 2024-09-16T03:42:29.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.073+0000 7f58cb500640 10 monclient(hunting): picked mon.noname-b con 0x7f58c4137710 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:42:29.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.073+0000 7f58c929c640 1 -- 172.21.15.42:0/3729999718 learned_addr learned my addr 172.21.15.42:0/3729999718 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:42:29.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.073+0000 7f58cb500640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:29.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.073+0000 7f58cb500640 1 -- 172.21.15.42:0/3729999718 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f58c4142310 con 0x7f58c40a39b0 2024-09-16T03:42:29.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.073+0000 7f58cb500640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:29.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.073+0000 7f58cb500640 1 -- 172.21.15.42:0/3729999718 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f58c4142490 con 0x7f58c4137710 2024-09-16T03:42:29.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.073+0000 7f58cb500640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:29.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.073+0000 7f58cb500640 1 -- 172.21.15.42:0/3729999718 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f58c413a8e0 con 0x7f58c40a3d90 2024-09-16T03:42:29.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.073+0000 7f58cb500640 10 monclient(hunting): _renew_subs 2024-09-16T03:42:29.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.073+0000 7f58c9a9d640 1 --2- 172.21.15.42:0/3729999718 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f58c4137710 0x7f58c41396c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:29.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.073+0000 7f58cb500640 10 monclient(hunting): authenticate will time out at 996.045322s 2024-09-16T03:42:29.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.073+0000 7f58c929c640 10 monclient(hunting): get_auth_request con 0x7f58c40a39b0 auth_method 0 2024-09-16T03:42:29.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.073+0000 7f58c929c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:29.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.073+0000 7f58c929c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:29.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.073+0000 7f58c929c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:29.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.073+0000 7f58c8a9b640 10 monclient(hunting): get_auth_request con 0x7f58c40a3d90 auth_method 0 2024-09-16T03:42:29.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.073+0000 7f58c8a9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:29.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.073+0000 7f58c8a9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:29.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.073+0000 7f58c8a9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:29.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.073+0000 7f58c929c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:29.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.073+0000 7f58c929c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:29.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.073+0000 7f58c929c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:29.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.073+0000 7f58c9a9d640 10 monclient(hunting): get_auth_request con 0x7f58c4137710 auth_method 0 2024-09-16T03:42:29.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.073+0000 7f58c9a9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:29.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.073+0000 7f58c9a9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:29.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.073+0000 7f58c9a9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:29.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.073+0000 7f58c8a9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:29.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.073+0000 7f58c8a9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:29.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.073+0000 7f58c8a9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:29.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.073+0000 7f58c929c640 10 monclient(hunting): handle_auth_done global_id 4284 payload 824 2024-09-16T03:42:29.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.073+0000 7f58c929c640 1 -- 172.21.15.42:0/3729999718 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f58c40a3d90 msgr2=0x7f58c4139180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:29.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.073+0000 7f58c929c640 1 --2- 172.21.15.42:0/3729999718 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f58c40a3d90 0x7f58c4139180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:29.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.073+0000 7f58c929c640 1 -- 172.21.15.42:0/3729999718 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f58c4137710 msgr2=0x7f58c41396c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:29.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.073+0000 7f58c929c640 1 --2- 172.21.15.42:0/3729999718 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f58c4137710 0x7f58c41396c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:29.093 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.073+0000 7f58c929c640 10 monclient: _finish_hunting 0 2024-09-16T03:42:29.093 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.073+0000 7f58c929c640 1 monclient: found mon.noname-a 2024-09-16T03:42:29.093 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.073+0000 7f58c929c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:42:29.093 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.073+0000 7f58c929c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:42:29.093 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.073+0000 7f58c929c640 1 -- 172.21.15.42:0/3729999718 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58c413c680 con 0x7f58c40a39b0 2024-09-16T03:42:29.093 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.073+0000 7f58c929c640 1 --2- 172.21.15.42:0/3729999718 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f58c40a39b0 0x7f58c413b990 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f58c001cc80 tx=0x7f58c0019180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:29.093 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.077+0000 7f58b9ffb640 10 client.?.objecter ms_handle_connect 0x7f58c40a39b0 2024-09-16T03:42:29.093 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.077+0000 7f58b9ffb640 10 client.?.objecter resend_mon_ops 2024-09-16T03:42:29.093 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.077+0000 7f58b9ffb640 1 -- 172.21.15.42:0/3729999718 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58c00b0050 con 0x7f58c40a39b0 2024-09-16T03:42:29.093 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.077+0000 7f58b9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:29.093 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.077+0000 7f58b9ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:42:29.093 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.077+0000 7f58b9ffb640 10 monclient: dump: 2024-09-16T03:42:29.093 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:29.094 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:29.094 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:29.094 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:29.094 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:29.094 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:29.094 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:29.094 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:29.094 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:29.094 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:29.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.077+0000 7f58b9ffb640 10 monclient: _finish_auth 0 2024-09-16T03:42:29.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.077+0000 7f58b9ffb640 10 monclient: _check_auth_tickets 2024-09-16T03:42:29.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.077+0000 7f58b9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:42:29.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.077+0000 7f58cb500640 5 monclient: authenticate success, global_id 4284 2024-09-16T03:42:29.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.077+0000 7f58cb500640 10 monclient: _renew_subs 2024-09-16T03:42:29.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.077+0000 7f58cb500640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:42:29.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.077+0000 7f58cb500640 1 -- 172.21.15.42:0/3729999718 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58c413c3a0 con 0x7f58c40a39b0 2024-09-16T03:42:29.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.077+0000 7f58cb500640 10 client.4284.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:42:29.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.077+0000 7f58cb500640 10 monclient: _renew_subs 2024-09-16T03:42:29.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.077+0000 7f58cb500640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:42:29.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.077+0000 7f58cb500640 1 -- 172.21.15.42:0/3729999718 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58c413ea50 con 0x7f58c40a39b0 2024-09-16T03:42:29.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.077+0000 7f58cb500640 1 librados: init done 2024-09-16T03:42:29.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.077+0000 7f58b9ffb640 1 -- 172.21.15.42:0/3729999718 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f58c00ba070 con 0x7f58c40a39b0 2024-09-16T03:42:29.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.077+0000 7f58b9ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T03:42:29.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.077+0000 7f58cb500640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:42:29.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.077+0000 7f58bb7fe640 4 set_mon_vals no callback set 2024-09-16T03:42:29.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.077+0000 7f58b9ffb640 1 -- 172.21.15.42:0/3729999718 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58c00b49f0 con 0x7f58c40a39b0 2024-09-16T03:42:29.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.077+0000 7f58cb500640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:42:29.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.077+0000 7f58cb500640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:42:29.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.077+0000 7f58cb500640 1 -- 172.21.15.42:0/3729999718 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f58c4133b80 con 0x7f58c40a39b0 2024-09-16T03:42:29.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.077+0000 7f58b9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:29.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.077+0000 7f58b9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:42:29.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.077+0000 7f58b9ffb640 10 monclient: dump: 2024-09-16T03:42:29.096 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:29.096 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:29.096 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:29.096 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:29.096 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:29.096 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:29.096 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:29.096 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:29.096 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:29.096 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:29.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.077+0000 7f58b9ffb640 1 -- 172.21.15.42:0/3729999718 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f58c00b4bd0 con 0x7f58c40a39b0 2024-09-16T03:42:29.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.077+0000 7f58b9ffb640 1 --2- 172.21.15.42:0/3729999718 >> v2:172.21.15.154:6812/254845342 conn(0x7f58a8037840 0x7f58a8039d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:29.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.077+0000 7f58b9ffb640 1 -- 172.21.15.42:0/3729999718 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 3562+0+0 (secure 0 0 0) 0x7f58c00e2a70 con 0x7f58c40a39b0 2024-09-16T03:42:29.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.077+0000 7f58b9ffb640 10 client.4284.objecter ms_dispatch 0x7f58c4000ba0 osd_map(18..18 src has 1..18) 2024-09-16T03:42:29.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.077+0000 7f58b9ffb640 3 client.4284.objecter handle_osd_map got epochs [18,18] > 0 2024-09-16T03:42:29.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.077+0000 7f58b9ffb640 3 client.4284.objecter handle_osd_map decoding full epoch 18 2024-09-16T03:42:29.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.077+0000 7f58b9ffb640 20 client.4284.objecter dump_active .. 0 homeless 2024-09-16T03:42:29.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.077+0000 7f58c8a9b640 1 --2- 172.21.15.42:0/3729999718 >> v2:172.21.15.154:6812/254845342 conn(0x7f58a8037840 0x7f58a8039d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:29.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.077+0000 7f58c8a9b640 10 monclient: get_auth_request con 0x7f58a8037840 auth_method 0 2024-09-16T03:42:29.098 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.081+0000 7f58b9ffb640 10 client.4284.objecter ms_handle_connect 0x7f58a8037840 2024-09-16T03:42:29.098 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.081+0000 7f58c8a9b640 1 --2- 172.21.15.42:0/3729999718 >> v2:172.21.15.154:6812/254845342 conn(0x7f58a8037840 0x7f58a8039d00 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f58c413c2d0 tx=0x7f58b4007e40 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:29.098 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.085+0000 7f58b9ffb640 1 -- 172.21.15.42:0/3729999718 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f58c413c680 con 0x7f58c40a39b0 2024-09-16T03:42:29.098 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.085+0000 7f58b9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:42:29.098 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.085+0000 7f58b9ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:42:29.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.097+0000 7fd71a637640 10 monclient: get_monmap_and_config 2024-09-16T03:42:29.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.097+0000 7fd71a637640 10 monclient: build_initial_monmap 2024-09-16T03:42:29.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.097+0000 7fd71a637640 10 monclient: monmap: 2024-09-16T03:42:29.104 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:42:29.104 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:29.104 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:42:29.101778+0000 2024-09-16T03:42:29.104 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:42:29.101778+0000 2024-09-16T03:42:29.104 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:42:29.104 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:29.104 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:42:29.104 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:42:29.104 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:42:29.104 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:42:29.104 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:42:29.105 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:29.106 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.097+0000 7fd71a637640 1 Processor -- start 2024-09-16T03:42:29.106 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.097+0000 7fd71a637640 1 -- start start 2024-09-16T03:42:29.106 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.097+0000 7fd71a637640 10 monclient: init 2024-09-16T03:42:29.106 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.097+0000 7fd71a637640 10 monclient: _reopen_session rank -1 2024-09-16T03:42:29.106 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.097+0000 7fd71a637640 10 monclient: _add_conns ranks=[3,0,1,4,2] 2024-09-16T03:42:29.106 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.097+0000 7fd71a637640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd7140a39b0 0x7fd714133b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:29.106 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.097+0000 7fd71a637640 10 monclient(hunting): picked mon.noname-a con 0x7fd7140a39b0 addr v2:172.21.15.42:3300/0 2024-09-16T03:42:29.106 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.097+0000 7fd71a637640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd7140a3d90 addr v1:172.21.15.42:6789/0 2024-09-16T03:42:29.106 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.097+0000 7fd71a637640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd714137710 addr v1:172.21.15.154:6789/0 2024-09-16T03:42:29.106 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.097+0000 7fd71a637640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd71409c140 con 0x7fd7140a3d90 2024-09-16T03:42:29.106 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.097+0000 7fd71a637640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd71409c420 con 0x7fd714137710 2024-09-16T03:42:29.106 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.097+0000 7fd71a637640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:29.107 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.097+0000 7fd71a637640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7fd71406dd60 con 0x7fd7140a39b0 2024-09-16T03:42:29.107 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.097+0000 7fd71a637640 10 monclient(hunting): _renew_subs 2024-09-16T03:42:29.107 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.097+0000 7fd71a637640 10 monclient(hunting): authenticate will time out at 996.071055s 2024-09-16T03:42:29.107 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.097+0000 7fd713fff640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd7140a39b0 0x7fd714133b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:29.107 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.097+0000 7fd713fff640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd7140a39b0 0x7fd714133b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:37688/0 (socket says 172.21.15.42:37688) 2024-09-16T03:42:29.107 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.097+0000 7fd713fff640 1 -- 172.21.15.42:0/3766708113 learned_addr learned my addr 172.21.15.42:0/3766708113 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:42:29.107 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.097+0000 7fd713fff640 10 monclient(hunting): get_auth_request con 0x7fd7140a39b0 auth_method 0 2024-09-16T03:42:29.107 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.097+0000 7fd713fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:29.107 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.097+0000 7fd713fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:29.107 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.097+0000 7fd713fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:29.107 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.097+0000 7fd713fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:29.107 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.097+0000 7fd713fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:29.107 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.097+0000 7fd713fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:29.108 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.097+0000 7fd713fff640 10 monclient(hunting): handle_auth_done global_id 4287 payload 274 2024-09-16T03:42:29.108 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.097+0000 7fd713fff640 1 -- 172.21.15.42:0/3766708113 >> v1:172.21.15.154:6789/0 conn(0x7fd714137710 legacy=0x7fd71413f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:42:29.108 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.097+0000 7fd713fff640 1 -- 172.21.15.42:0/3766708113 >> v1:172.21.15.42:6789/0 conn(0x7fd7140a3d90 legacy=0x7fd714137000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:42:29.108 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.097+0000 7fd713fff640 10 monclient: _finish_hunting 0 2024-09-16T03:42:29.108 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.097+0000 7fd713fff640 1 monclient: found mon.noname-a 2024-09-16T03:42:29.108 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.097+0000 7fd713fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:42:29.108 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.097+0000 7fd713fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:42:29.108 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.097+0000 7fd713fff640 1 -- 172.21.15.42:0/3766708113 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7140a5e20 con 0x7fd7140a39b0 2024-09-16T03:42:29.109 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.101+0000 7fd713fff640 1 --2- 172.21.15.42:0/3766708113 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd7140a39b0 0x7fd714133b80 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fd704009870 tx=0x7fd70401c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d137ee0018f83318 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:29.109 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.101+0000 7fd712ffd640 1 -- 172.21.15.42:0/3766708113 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7040b0070 con 0x7fd7140a39b0 2024-09-16T03:42:29.109 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.101+0000 7fd712ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:29.109 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.101+0000 7fd712ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:42:29.110 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.101+0000 7fd712ffd640 10 monclient: dump: 2024-09-16T03:42:29.110 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:29.110 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:29.110 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:29.110 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:29.110 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:29.110 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:29.110 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:29.110 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:29.110 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:29.110 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:29.110 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.101+0000 7fd712ffd640 10 monclient: _finish_auth 0 2024-09-16T03:42:29.110 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.101+0000 7fd712ffd640 10 monclient: _check_auth_tickets 2024-09-16T03:42:29.110 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.101+0000 7fd712ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:42:29.111 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.101+0000 7fd712ffd640 1 -- 172.21.15.42:0/3766708113 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7040ab600 con 0x7fd7140a39b0 2024-09-16T03:42:29.111 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.101+0000 7fd712ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:42:29.111 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.101+0000 7fd712ffd640 1 -- 172.21.15.42:0/3766708113 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7040ab9d0 con 0x7fd7140a39b0 2024-09-16T03:42:29.111 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.101+0000 7fd712ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:29.111 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.101+0000 7fd712ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:42:29.111 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.101+0000 7fd712ffd640 10 monclient: dump: 2024-09-16T03:42:29.111 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:29.111 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:29.111 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:29.111 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:29.111 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:29.111 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:29.112 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:29.112 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:29.112 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:29.112 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:29.112 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.101+0000 7fd71a637640 5 monclient: authenticate success, global_id 4287 2024-09-16T03:42:29.112 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.101+0000 7fd71a637640 10 monclient: get_monmap_and_config success 2024-09-16T03:42:29.112 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.101+0000 7fd71a637640 4 set_mon_vals no callback set 2024-09-16T03:42:29.112 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.101+0000 7fd71a637640 10 monclient: shutdown 2024-09-16T03:42:29.112 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.101+0000 7fd71a637640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:42:29.112 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.101+0000 7fd71a637640 1 -- 172.21.15.42:0/3766708113 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd7140a39b0 msgr2=0x7fd714133b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:29.113 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.101+0000 7fd71a637640 1 --2- 172.21.15.42:0/3766708113 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd7140a39b0 0x7fd714133b80 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fd704009870 tx=0x7fd70401c3a0 comp rx=0 tx=0).stop 2024-09-16T03:42:29.113 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.101+0000 7fd71a637640 1 -- 172.21.15.42:0/3766708113 shutdown_connections 2024-09-16T03:42:29.113 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.101+0000 7fd71a637640 1 --2- 172.21.15.42:0/3766708113 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd7140a39b0 0x7fd714133b80 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:29.113 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.101+0000 7fd71a637640 1 -- 172.21.15.42:0/3766708113 >> 172.21.15.42:0/3766708113 conn(0x7fd7140a9fb0 msgr2=0x7fd7140aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:42:29.113 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.101+0000 7fd71a637640 1 -- 172.21.15.42:0/3766708113 shutdown_connections 2024-09-16T03:42:29.115 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.101+0000 7fd71a637640 1 -- 172.21.15.42:0/3766708113 wait complete. 2024-09-16T03:42:29.115 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.101+0000 7fd71a637640 10 monclient: build_initial_monmap 2024-09-16T03:42:29.116 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.101+0000 7fd71a637640 10 monclient: monmap: 2024-09-16T03:42:29.116 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:42:29.116 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:42:29.116 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:42:29.116 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:42:29.116 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:42:29.116 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:29.116 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:42:29.116 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:42:29.117 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:42:29.117 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:29.117 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.105+0000 7fd71a637640 1 librados: starting msgr at 2024-09-16T03:42:29.117 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.105+0000 7fd71a637640 1 librados: starting objecter 2024-09-16T03:42:29.117 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.105+0000 7fd71a637640 1 Processor -- start 2024-09-16T03:42:29.117 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.105+0000 7fd71a637640 1 -- start start 2024-09-16T03:42:29.117 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.105+0000 7fd71a637640 1 librados: setting wanted keys 2024-09-16T03:42:29.117 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.105+0000 7fd71a637640 1 librados: calling monclient init 2024-09-16T03:42:29.117 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.105+0000 7fd71a637640 10 monclient: init 2024-09-16T03:42:29.117 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.105+0000 7fd71a637640 10 monclient: _reopen_session rank -1 2024-09-16T03:42:29.117 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.105+0000 7fd71a637640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T03:42:29.117 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.105+0000 7fd71a637640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd7140a39b0 0x7fd7141dceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:29.118 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.105+0000 7fd71a637640 10 monclient(hunting): picked mon.noname-b con 0x7fd7140a39b0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:42:29.118 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.105+0000 7fd71a637640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd7140a3d90 0x7fd7141dd3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:29.118 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.105+0000 7fd71a637640 10 monclient(hunting): picked mon.noname-c con 0x7fd7140a3d90 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:42:29.118 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.105+0000 7fd71a637640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd714137710 0x7fd71409d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:29.118 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.105+0000 7fd71a637640 10 monclient(hunting): picked mon.noname-a con 0x7fd714137710 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:42:29.118 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.105+0000 7fd71a637640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:29.118 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.105+0000 7fd71a637640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fd714143900 con 0x7fd714137710 2024-09-16T03:42:29.118 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.105+0000 7fd71a637640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:29.118 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.105+0000 7fd71a637640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd71406dee0 con 0x7fd7140a39b0 2024-09-16T03:42:29.118 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.105+0000 7fd71a637640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:29.119 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.105+0000 7fd71a637640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fd714143a80 con 0x7fd7140a3d90 2024-09-16T03:42:29.119 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.105+0000 7fd71a637640 10 monclient(hunting): _renew_subs 2024-09-16T03:42:29.119 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.105+0000 7fd71a637640 10 monclient(hunting): authenticate will time out at 996.076791s 2024-09-16T03:42:29.119 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.105+0000 7fd718bd4640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd714137710 0x7fd71409d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:29.119 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.105+0000 7fd7137fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd7140a3d90 0x7fd7141dd3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:29.119 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.105+0000 7fd7137fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd7140a3d90 0x7fd7141dd3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:41372/0 (socket says 172.21.15.42:41372) 2024-09-16T03:42:29.119 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.105+0000 7fd718bd4640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd714137710 0x7fd71409d210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:37704/0 (socket says 172.21.15.42:37704) 2024-09-16T03:42:29.119 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.105+0000 7fd7137fe640 1 -- 172.21.15.42:0/56590665 learned_addr learned my addr 172.21.15.42:0/56590665 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:42:29.119 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.105+0000 7fd7137fe640 10 monclient(hunting): get_auth_request con 0x7fd7140a3d90 auth_method 0 2024-09-16T03:42:29.119 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.105+0000 7fd7137fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:29.119 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.105+0000 7fd7137fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:29.120 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.105+0000 7fd7137fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:29.120 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.105+0000 7fd718bd4640 10 monclient(hunting): get_auth_request con 0x7fd714137710 auth_method 0 2024-09-16T03:42:29.120 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.105+0000 7fd718bd4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:29.120 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.105+0000 7fd718bd4640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:29.120 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.105+0000 7fd718bd4640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:29.120 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.105+0000 7fd718bd4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:29.120 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.105+0000 7fd718bd4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:29.120 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.105+0000 7fd718bd4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:29.120 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.105+0000 7fd7137fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:29.120 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.105+0000 7fd7137fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:29.120 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.105+0000 7fd713fff640 1 --2- 172.21.15.42:0/56590665 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd7140a39b0 0x7fd7141dceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:29.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.105+0000 7fd7137fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:29.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.105+0000 7fd713fff640 10 monclient(hunting): get_auth_request con 0x7fd7140a39b0 auth_method 0 2024-09-16T03:42:29.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.105+0000 7fd713fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:29.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.105+0000 7fd713fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:29.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.105+0000 7fd713fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:29.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.105+0000 7fd713fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:29.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.105+0000 7fd713fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:29.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.105+0000 7fd713fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:29.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.105+0000 7fd7137fe640 10 monclient(hunting): handle_auth_done global_id 4256 payload 824 2024-09-16T03:42:29.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.105+0000 7fd7137fe640 1 -- 172.21.15.42:0/56590665 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd7140a39b0 msgr2=0x7fd7141dceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:29.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.105+0000 7fd7137fe640 1 --2- 172.21.15.42:0/56590665 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd7140a39b0 0x7fd7141dceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:29.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.105+0000 7fd7137fe640 1 -- 172.21.15.42:0/56590665 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd714137710 msgr2=0x7fd71409d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:29.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.105+0000 7fd7137fe640 1 --2- 172.21.15.42:0/56590665 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd714137710 0x7fd71409d210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:29.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.105+0000 7fd7137fe640 10 monclient: _finish_hunting 0 2024-09-16T03:42:29.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.105+0000 7fd7137fe640 1 monclient: found mon.noname-c 2024-09-16T03:42:29.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.105+0000 7fd7137fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:42:29.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.105+0000 7fd7137fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:42:29.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.105+0000 7fd7137fe640 1 -- 172.21.15.42:0/56590665 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7140a0ec0 con 0x7fd7140a3d90 2024-09-16T03:42:29.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.105+0000 7fd718bd4640 1 --2- 172.21.15.42:0/56590665 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd714137710 0x7fd71409d210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:42:29.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.105+0000 7fd7137fe640 1 --2- 172.21.15.42:0/56590665 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd7140a3d90 0x7fd7141dd3f0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fd70800c450 tx=0x7fd70800c910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:29.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.105+0000 7fd710ff9640 10 client.?.objecter ms_handle_connect 0x7fd7140a3d90 2024-09-16T03:42:29.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.105+0000 7fd710ff9640 10 client.?.objecter resend_mon_ops 2024-09-16T03:42:29.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.105+0000 7fd710ff9640 1 -- 172.21.15.42:0/56590665 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd708012830 con 0x7fd7140a3d90 2024-09-16T03:42:29.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.105+0000 7fd710ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:29.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.105+0000 7fd710ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:42:29.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.105+0000 7fd710ff9640 10 monclient: dump: 2024-09-16T03:42:29.123 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:29.123 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:29.123 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:29.123 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:29.123 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:29.123 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:29.123 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:29.123 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:29.123 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:29.123 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:29.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.105+0000 7fd710ff9640 10 monclient: _finish_auth 0 2024-09-16T03:42:29.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.105+0000 7fd710ff9640 10 monclient: _check_auth_tickets 2024-09-16T03:42:29.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.105+0000 7fd710ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:42:29.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.105+0000 7fd710ff9640 1 -- 172.21.15.42:0/56590665 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd708017070 con 0x7fd7140a3d90 2024-09-16T03:42:29.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.105+0000 7fd710ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T03:42:29.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.105+0000 7fd7127fc640 4 set_mon_vals no callback set 2024-09-16T03:42:29.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.105+0000 7fd71a637640 5 monclient: authenticate success, global_id 4256 2024-09-16T03:42:29.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.105+0000 7fd710ff9640 1 -- 172.21.15.42:0/56590665 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd708012c30 con 0x7fd7140a3d90 2024-09-16T03:42:29.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.105+0000 7fd71a637640 10 monclient: _renew_subs 2024-09-16T03:42:29.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.105+0000 7fd71a637640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:42:29.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.105+0000 7fd71a637640 1 -- 172.21.15.42:0/56590665 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7140a11a0 con 0x7fd7140a3d90 2024-09-16T03:42:29.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.105+0000 7fd71a637640 10 client.4256.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:42:29.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.105+0000 7fd71a637640 10 monclient: _renew_subs 2024-09-16T03:42:29.125 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.105+0000 7fd71a637640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:42:29.125 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.105+0000 7fd71a637640 1 -- 172.21.15.42:0/56590665 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd714133420 con 0x7fd7140a3d90 2024-09-16T03:42:29.125 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.105+0000 7fd71a637640 1 librados: init done 2024-09-16T03:42:29.125 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.105+0000 7fd71a637640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:42:29.125 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.105+0000 7fd71a637640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:42:29.125 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.105+0000 7fd71a637640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:42:29.125 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.109+0000 7fd71a637640 1 -- 172.21.15.42:0/56590665 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd714133b80 con 0x7fd7140a3d90 2024-09-16T03:42:29.126 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.113+0000 7fd710ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:29.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.113+0000 7fd710ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:42:29.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.113+0000 7fd710ff9640 10 monclient: dump: 2024-09-16T03:42:29.127 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:29.127 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:29.127 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:29.127 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:29.127 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:29.127 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:29.127 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:29.127 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:29.127 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:29.127 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:29.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.113+0000 7fd710ff9640 1 -- 172.21.15.42:0/56590665 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fd70801c460 con 0x7fd7140a3d90 2024-09-16T03:42:29.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.113+0000 7fd710ff9640 1 --2- 172.21.15.42:0/56590665 >> v2:172.21.15.154:6812/254845342 conn(0x7fd700037860 0x7fd700039d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:29.128 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.113+0000 7fd710ff9640 1 -- 172.21.15.42:0/56590665 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(18..18 src has 1..18) ==== 3562+0+0 (secure 0 0 0) 0x7fd708049000 con 0x7fd7140a3d90 2024-09-16T03:42:29.128 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.113+0000 7fd710ff9640 10 client.4256.objecter ms_dispatch 0x7fd714000ba0 osd_map(18..18 src has 1..18) 2024-09-16T03:42:29.128 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.113+0000 7fd710ff9640 3 client.4256.objecter handle_osd_map got epochs [18,18] > 0 2024-09-16T03:42:29.128 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.113+0000 7fd710ff9640 3 client.4256.objecter handle_osd_map decoding full epoch 18 2024-09-16T03:42:29.128 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.113+0000 7fd710ff9640 20 client.4256.objecter dump_active .. 0 homeless 2024-09-16T03:42:29.128 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.113+0000 7fd713fff640 1 --2- 172.21.15.42:0/56590665 >> v2:172.21.15.154:6812/254845342 conn(0x7fd700037860 0x7fd700039d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:29.128 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.113+0000 7fd713fff640 10 monclient: get_auth_request con 0x7fd700037860 auth_method 0 2024-09-16T03:42:29.128 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.113+0000 7fd710ff9640 1 -- 172.21.15.42:0/56590665 <== mon.2 v2:172.21.15.42:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7fd708046ed0 con 0x7fd7140a3d90 2024-09-16T03:42:29.128 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.113+0000 7fd710ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:42:29.128 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.113+0000 7fd710ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:42:29.128 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.113+0000 7fd713fff640 1 --2- 172.21.15.42:0/56590665 >> v2:172.21.15.154:6812/254845342 conn(0x7fd700037860 0x7fd700039d20 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fd7040ad000 tx=0x7fd7040a6000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:29.128 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.113+0000 7fd710ff9640 10 client.4256.objecter ms_handle_connect 0x7fd700037860 2024-09-16T03:42:29.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.133+0000 7f49c8e83640 10 monclient: get_monmap_and_config 2024-09-16T03:42:29.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.133+0000 7f49c8e83640 10 monclient: build_initial_monmap 2024-09-16T03:42:29.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.133+0000 7f49c8e83640 10 monclient: monmap: 2024-09-16T03:42:29.139 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:42:29.139 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:29.139 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:42:29.136605+0000 2024-09-16T03:42:29.139 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:42:29.136605+0000 2024-09-16T03:42:29.139 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:42:29.139 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:29.139 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:42:29.139 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:42:29.139 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:42:29.139 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:42:29.139 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:42:29.139 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:29.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.133+0000 7f49c8e83640 1 Processor -- start 2024-09-16T03:42:29.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.133+0000 7f49c8e83640 1 -- start start 2024-09-16T03:42:29.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.133+0000 7f49c8e83640 10 monclient: init 2024-09-16T03:42:29.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.133+0000 7f49c8e83640 10 monclient: _reopen_session rank -1 2024-09-16T03:42:29.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.133+0000 7f49c8e83640 10 monclient: _add_conns ranks=[0,1,4,3,2] 2024-09-16T03:42:29.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.133+0000 7f49c8e83640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f49c40a39b0 addr v1:172.21.15.42:6789/0 2024-09-16T03:42:29.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.133+0000 7f49c8e83640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f49c40a3d90 addr v1:172.21.15.154:6789/0 2024-09-16T03:42:29.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.133+0000 7f49c8e83640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f49c413daa0 0x7f49c413fe90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:29.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.133+0000 7f49c8e83640 10 monclient(hunting): picked mon.noname-c con 0x7f49c413daa0 addr v2:172.21.15.154:3300/0 2024-09-16T03:42:29.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.133+0000 7f49c8e83640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f49c4134290 con 0x7f49c40a39b0 2024-09-16T03:42:29.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.133+0000 7f49c8e83640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f49c409c140 con 0x7f49c40a3d90 2024-09-16T03:42:29.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.133+0000 7f49c8e83640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:29.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.133+0000 7f49c8e83640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f49c409c420 con 0x7f49c413daa0 2024-09-16T03:42:29.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.133+0000 7f49c8e83640 10 monclient(hunting): _renew_subs 2024-09-16T03:42:29.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.133+0000 7f49c8e83640 10 monclient(hunting): authenticate will time out at 996.105787s 2024-09-16T03:42:29.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.133+0000 7f49c259c640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f49c40a39b0 0x7f49c4133b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:46794/0 (socket says 172.21.15.42:46794) 2024-09-16T03:42:29.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.133+0000 7f49c259c640 1 -- 172.21.15.42:0/3382682086 learned_addr learned my addr 172.21.15.42:0/3382682086 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:42:29.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.133+0000 7f49c2d9d640 1 --2- 172.21.15.42:0/3382682086 >> v2:172.21.15.154:3300/0 conn(0x7f49c413daa0 0x7f49c413fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:29.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.133+0000 7f49c2d9d640 10 monclient(hunting): get_auth_request con 0x7f49c413daa0 auth_method 0 2024-09-16T03:42:29.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.133+0000 7f49c2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:29.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.133+0000 7f49c2d9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:29.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.133+0000 7f49c2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:29.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.133+0000 7f49c2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:29.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.133+0000 7f49c2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:29.143 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.133+0000 7f49c2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:29.143 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.133+0000 7f49c2d9d640 10 monclient(hunting): handle_auth_done global_id 4276 payload 274 2024-09-16T03:42:29.143 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.133+0000 7f49c1d9b640 1 -- 172.21.15.42:0/3382682086 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1337626764 0 0) 0x7f49c4134290 con 0x7f49c40a39b0 2024-09-16T03:42:29.143 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.133+0000 7f49c2d9d640 1 -- 172.21.15.42:0/3382682086 >> v1:172.21.15.154:6789/0 conn(0x7f49c40a3d90 legacy=0x7f49c413c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:29.143 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.133+0000 7f49c2d9d640 1 -- 172.21.15.42:0/3382682086 >> v1:172.21.15.42:6789/0 conn(0x7f49c40a39b0 legacy=0x7f49c4133b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:29.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.133+0000 7f49c2d9d640 10 monclient: _finish_hunting 0 2024-09-16T03:42:29.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.133+0000 7f49c2d9d640 1 monclient: found mon.noname-c 2024-09-16T03:42:29.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.133+0000 7f49c2d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:42:29.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.133+0000 7f49c2d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.154:3300/0 2024-09-16T03:42:29.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.133+0000 7f49c2d9d640 1 -- 172.21.15.42:0/3382682086 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49c406dd60 con 0x7f49c413daa0 2024-09-16T03:42:29.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.133+0000 7f49c1d9b640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T03:42:29.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.137+0000 7f49c2d9d640 1 --2- 172.21.15.42:0/3382682086 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f49c413daa0 0x7f49c413fe90 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f49bc009d90 tx=0x7f49bc01c830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d7c7b43a90f7763c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:29.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.137+0000 7f49c1d9b640 1 -- 172.21.15.42:0/3382682086 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49bc0a6700 con 0x7f49c413daa0 2024-09-16T03:42:29.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.137+0000 7f49c1d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:29.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.137+0000 7f49c1d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:42:29.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.137+0000 7f49c1d9b640 10 monclient: dump: 2024-09-16T03:42:29.145 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:29.145 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:29.145 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:29.145 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:29.145 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:29.145 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:29.145 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:29.145 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:29.145 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:29.145 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:29.145 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.137+0000 7f49c1d9b640 10 monclient: _finish_auth 0 2024-09-16T03:42:29.145 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.137+0000 7f49c1d9b640 10 monclient: _check_auth_tickets 2024-09-16T03:42:29.145 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.137+0000 7f49c1d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:42:29.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.137+0000 7f49c8e83640 5 monclient: authenticate success, global_id 4276 2024-09-16T03:42:29.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.137+0000 7f49c8e83640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:42:29.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.137+0000 7f49c1d9b640 1 -- 172.21.15.42:0/3382682086 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f49bc0a68a0 con 0x7f49c413daa0 2024-09-16T03:42:29.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.137+0000 7f49c1d9b640 10 monclient: handle_config config(0 keys) 2024-09-16T03:42:29.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.137+0000 7f49c1d9b640 1 -- 172.21.15.42:0/3382682086 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49bc0a6c70 con 0x7f49c413daa0 2024-09-16T03:42:29.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.137+0000 7f49c1d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:29.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.137+0000 7f49c1d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:42:29.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.137+0000 7f49c1d9b640 10 monclient: dump: 2024-09-16T03:42:29.146 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:29.146 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:29.146 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:29.146 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:29.146 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:29.146 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:29.147 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:29.147 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:29.147 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:29.147 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:29.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.137+0000 7f49c8e83640 10 monclient: get_monmap_and_config success 2024-09-16T03:42:29.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.137+0000 7f49c8e83640 4 set_mon_vals no callback set 2024-09-16T03:42:29.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.137+0000 7f49c8e83640 10 monclient: shutdown 2024-09-16T03:42:29.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.137+0000 7f49c8e83640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:42:29.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.137+0000 7f49c8e83640 1 -- 172.21.15.42:0/3382682086 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f49c413daa0 msgr2=0x7f49c413fe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:29.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.137+0000 7f49c8e83640 1 --2- 172.21.15.42:0/3382682086 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f49c413daa0 0x7f49c413fe90 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f49bc009d90 tx=0x7f49bc01c830 comp rx=0 tx=0).stop 2024-09-16T03:42:29.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.137+0000 7f49c8e83640 1 -- 172.21.15.42:0/3382682086 shutdown_connections 2024-09-16T03:42:29.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.137+0000 7f49c8e83640 1 --2- 172.21.15.42:0/3382682086 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f49c413daa0 0x7f49c413fe90 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:29.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.137+0000 7f49c8e83640 1 -- 172.21.15.42:0/3382682086 >> 172.21.15.42:0/3382682086 conn(0x7f49c40a9fb0 msgr2=0x7f49c40aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:42:29.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.137+0000 7f49c8e83640 1 -- 172.21.15.42:0/3382682086 shutdown_connections 2024-09-16T03:42:29.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.137+0000 7f49c8e83640 1 -- 172.21.15.42:0/3382682086 wait complete. 2024-09-16T03:42:29.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.137+0000 7f49c8e83640 10 monclient: build_initial_monmap 2024-09-16T03:42:29.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.137+0000 7f49c8e83640 10 monclient: monmap: 2024-09-16T03:42:29.157 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:42:29.157 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:42:29.157 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:42:29.157 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:42:29.157 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:42:29.157 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:29.157 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:42:29.157 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:42:29.157 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:42:29.157 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:29.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.137+0000 7f49c8e83640 1 librados: starting msgr at 2024-09-16T03:42:29.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.137+0000 7f49c8e83640 1 librados: starting objecter 2024-09-16T03:42:29.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.137+0000 7f49c8e83640 1 Processor -- start 2024-09-16T03:42:29.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.137+0000 7f49c8e83640 1 -- start start 2024-09-16T03:42:29.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.137+0000 7f49c8e83640 1 librados: setting wanted keys 2024-09-16T03:42:29.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.137+0000 7f49c8e83640 1 librados: calling monclient init 2024-09-16T03:42:29.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.137+0000 7f49c8e83640 10 monclient: init 2024-09-16T03:42:29.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.137+0000 7f49c8e83640 10 monclient: _reopen_session rank -1 2024-09-16T03:42:29.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.141+0000 7f49c8e83640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T03:42:29.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.141+0000 7f49c8e83640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f49c40a39b0 0x7f49c4130560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:29.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.141+0000 7f49c8e83640 10 monclient(hunting): picked mon.noname-a con 0x7f49c40a39b0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:42:29.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.141+0000 7f49c8e83640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f49c40a3d90 0x7f49c4130aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:29.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.141+0000 7f49c8e83640 10 monclient(hunting): picked mon.noname-b con 0x7f49c40a3d90 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:42:29.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.141+0000 7f49c8e83640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f49c413daa0 0x7f49c413a070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:29.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.141+0000 7f49c8e83640 10 monclient(hunting): picked mon.noname-c con 0x7f49c413daa0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:42:29.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.141+0000 7f49c8e83640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:29.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.141+0000 7f49c8e83640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f49c4142f80 con 0x7f49c40a39b0 2024-09-16T03:42:29.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.141+0000 7f49c8e83640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:29.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.141+0000 7f49c8e83640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f49c4142e00 con 0x7f49c40a3d90 2024-09-16T03:42:29.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.141+0000 7f49c8e83640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:29.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.141+0000 7f49c259c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f49c40a39b0 0x7f49c4130560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:29.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.141+0000 7f49c8e83640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f49c4143100 con 0x7f49c413daa0 2024-09-16T03:42:29.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.141+0000 7f49c8e83640 10 monclient(hunting): _renew_subs 2024-09-16T03:42:29.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.141+0000 7f49c259c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f49c40a39b0 0x7f49c4130560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:37708/0 (socket says 172.21.15.42:37708) 2024-09-16T03:42:29.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.141+0000 7f49c259c640 1 -- 172.21.15.42:0/3023943085 learned_addr learned my addr 172.21.15.42:0/3023943085 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:42:29.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.141+0000 7f49c8e83640 10 monclient(hunting): authenticate will time out at 996.112474s 2024-09-16T03:42:29.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.141+0000 7f49c2d9d640 1 --2- 172.21.15.42:0/3023943085 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f49c413daa0 0x7f49c413a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:29.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.141+0000 7f49c259c640 10 monclient(hunting): get_auth_request con 0x7f49c40a39b0 auth_method 0 2024-09-16T03:42:29.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.141+0000 7f49c259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:29.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.141+0000 7f49c259c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:29.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.141+0000 7f49c259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:29.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.141+0000 7f49c2d9d640 10 monclient(hunting): get_auth_request con 0x7f49c413daa0 auth_method 0 2024-09-16T03:42:29.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.141+0000 7f49c2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:29.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.141+0000 7f49c2d9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:29.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.141+0000 7f49c2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:29.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.141+0000 7f49bbfff640 1 --2- 172.21.15.42:0/3023943085 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f49c40a3d90 0x7f49c4130aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:29.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.141+0000 7f49bbfff640 10 monclient(hunting): get_auth_request con 0x7f49c40a3d90 auth_method 0 2024-09-16T03:42:29.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.141+0000 7f49bbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:29.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.141+0000 7f49bbfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:29.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.141+0000 7f49bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:29.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.141+0000 7f49c2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:29.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.141+0000 7f49c2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:29.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.141+0000 7f49c2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:29.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.141+0000 7f49c259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:29.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.141+0000 7f49c259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:29.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.141+0000 7f49c259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:29.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.141+0000 7f49bbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:29.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.141+0000 7f49bbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:29.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.141+0000 7f49bbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:29.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.141+0000 7f49c2d9d640 10 monclient(hunting): handle_auth_done global_id 4259 payload 824 2024-09-16T03:42:29.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.141+0000 7f49c2d9d640 1 -- 172.21.15.42:0/3023943085 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f49c40a3d90 msgr2=0x7f49c4130aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:29.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.141+0000 7f49c2d9d640 1 --2- 172.21.15.42:0/3023943085 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f49c40a3d90 0x7f49c4130aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:29.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.141+0000 7f49c2d9d640 1 -- 172.21.15.42:0/3023943085 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f49c40a39b0 msgr2=0x7f49c4130560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:29.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.141+0000 7f49c2d9d640 1 --2- 172.21.15.42:0/3023943085 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f49c40a39b0 0x7f49c4130560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:29.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.141+0000 7f49c2d9d640 10 monclient: _finish_hunting 0 2024-09-16T03:42:29.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.141+0000 7f49c2d9d640 1 monclient: found mon.noname-c 2024-09-16T03:42:29.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.141+0000 7f49c2d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:42:29.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.141+0000 7f49c2d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:42:29.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.141+0000 7f49c2d9d640 1 -- 172.21.15.42:0/3023943085 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49c409d4b0 con 0x7f49c413daa0 2024-09-16T03:42:29.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.141+0000 7f49c2d9d640 1 --2- 172.21.15.42:0/3023943085 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f49c413daa0 0x7f49c413a070 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f49bc0ac000 tx=0x7f49bc0195d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:29.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.141+0000 7f49baffd640 10 client.?.objecter ms_handle_connect 0x7f49c413daa0 2024-09-16T03:42:29.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.141+0000 7f49baffd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:42:29.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.141+0000 7f49baffd640 1 -- 172.21.15.42:0/3023943085 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49bc003ba0 con 0x7f49c413daa0 2024-09-16T03:42:29.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.141+0000 7f49baffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:29.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.141+0000 7f49baffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:42:29.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.141+0000 7f49baffd640 10 monclient: dump: 2024-09-16T03:42:29.163 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:29.163 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:29.164 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:29.164 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:29.164 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:29.164 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:29.164 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:29.164 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:29.164 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:29.164 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:29.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.141+0000 7f49baffd640 10 monclient: _finish_auth 0 2024-09-16T03:42:29.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.141+0000 7f49baffd640 10 monclient: _check_auth_tickets 2024-09-16T03:42:29.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.141+0000 7f49baffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:42:29.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.141+0000 7f49baffd640 1 -- 172.21.15.42:0/3023943085 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f49bc003d40 con 0x7f49c413daa0 2024-09-16T03:42:29.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.141+0000 7f49c8e83640 5 monclient: authenticate success, global_id 4259 2024-09-16T03:42:29.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.141+0000 7f49c8e83640 10 monclient: _renew_subs 2024-09-16T03:42:29.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.141+0000 7f49c8e83640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:42:29.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.141+0000 7f49c8e83640 1 -- 172.21.15.42:0/3023943085 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49c409d790 con 0x7f49c413daa0 2024-09-16T03:42:29.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.141+0000 7f49c8e83640 10 client.4259.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:42:29.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.141+0000 7f49c8e83640 10 monclient: _renew_subs 2024-09-16T03:42:29.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.141+0000 7f49c8e83640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:42:29.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.141+0000 7f49c8e83640 1 -- 172.21.15.42:0/3023943085 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f49c406f680 con 0x7f49c413daa0 2024-09-16T03:42:29.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.141+0000 7f49c8e83640 1 librados: init done 2024-09-16T03:42:29.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.141+0000 7f49baffd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:42:29.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.141+0000 7f49baffd640 1 -- 172.21.15.42:0/3023943085 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49bc0ae970 con 0x7f49c413daa0 2024-09-16T03:42:29.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.141+0000 7f49baffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:29.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.141+0000 7f49c159a640 4 set_mon_vals no callback set 2024-09-16T03:42:29.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.141+0000 7f49baffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:42:29.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.141+0000 7f49baffd640 10 monclient: dump: 2024-09-16T03:42:29.166 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:29.166 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:29.166 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:29.166 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:29.166 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:29.166 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:29.166 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:29.166 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:29.166 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:29.166 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:29.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.141+0000 7f49baffd640 1 -- 172.21.15.42:0/3023943085 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f49bc0aeb10 con 0x7f49c413daa0 2024-09-16T03:42:29.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.141+0000 7f49baffd640 1 --2- 172.21.15.42:0/3023943085 >> v2:172.21.15.154:6812/254845342 conn(0x7f49a4037810 0x7f49a4039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:29.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.141+0000 7f49baffd640 1 -- 172.21.15.42:0/3023943085 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(18..18 src has 1..18) ==== 3562+0+0 (secure 0 0 0) 0x7f49bc0e3dc0 con 0x7f49c413daa0 2024-09-16T03:42:29.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.141+0000 7f49baffd640 10 client.4259.objecter ms_dispatch 0x7f49c4000ba0 osd_map(18..18 src has 1..18) 2024-09-16T03:42:29.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.141+0000 7f49baffd640 3 client.4259.objecter handle_osd_map got epochs [18,18] > 0 2024-09-16T03:42:29.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.141+0000 7f49baffd640 3 client.4259.objecter handle_osd_map decoding full epoch 18 2024-09-16T03:42:29.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.141+0000 7f49c8e83640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:42:29.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.141+0000 7f49c8e83640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:42:29.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.141+0000 7f49c8e83640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:42:29.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.141+0000 7f49c259c640 1 --2- 172.21.15.42:0/3023943085 >> v2:172.21.15.154:6812/254845342 conn(0x7f49a4037810 0x7f49a4039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:29.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.141+0000 7f49c8e83640 1 -- 172.21.15.42:0/3023943085 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f49a8005090 con 0x7f49c413daa0 2024-09-16T03:42:29.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.141+0000 7f49c259c640 10 monclient: get_auth_request con 0x7f49a4037810 auth_method 0 2024-09-16T03:42:29.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.145+0000 7f49baffd640 20 client.4259.objecter dump_active .. 0 homeless 2024-09-16T03:42:29.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.145+0000 7f49c259c640 1 --2- 172.21.15.42:0/3023943085 >> v2:172.21.15.154:6812/254845342 conn(0x7f49a4037810 0x7f49a4039cd0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f49b4000f70 tx=0x7f49b4007ee0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:29.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.145+0000 7f49baffd640 10 client.4259.objecter ms_handle_connect 0x7f49a4037810 2024-09-16T03:42:29.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.153+0000 7f49baffd640 1 -- 172.21.15.42:0/3023943085 <== mon.2 v2:172.21.15.42:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f49bc0c7030 con 0x7f49c413daa0 2024-09-16T03:42:29.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.153+0000 7f49baffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:42:29.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.153+0000 7f49baffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:42:29.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.157+0000 7f3c72ade640 10 monclient: get_monmap_and_config 2024-09-16T03:42:29.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.157+0000 7f3c72ade640 10 monclient: build_initial_monmap 2024-09-16T03:42:29.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.157+0000 7f3c72ade640 10 monclient: monmap: 2024-09-16T03:42:29.171 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:42:29.171 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:29.171 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:42:29.163099+0000 2024-09-16T03:42:29.171 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:42:29.163099+0000 2024-09-16T03:42:29.171 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:42:29.171 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:29.171 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:42:29.172 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:42:29.172 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:42:29.172 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:42:29.172 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:42:29.172 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:29.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.157+0000 7f3c72ade640 1 Processor -- start 2024-09-16T03:42:29.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.157+0000 7f3c72ade640 1 -- start start 2024-09-16T03:42:29.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.157+0000 7f3c72ade640 10 monclient: init 2024-09-16T03:42:29.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.161+0000 7f3c72ade640 10 monclient: _reopen_session rank -1 2024-09-16T03:42:29.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.161+0000 7f3c72ade640 10 monclient: _add_conns ranks=[1,3,4,0,2] 2024-09-16T03:42:29.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.161+0000 7f3c72ade640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3c6c0a39b0 addr v1:172.21.15.154:6789/0 2024-09-16T03:42:29.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.161+0000 7f3c72ade640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f3c6c0a3d90 0x7f3c6c13c380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:29.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.161+0000 7f3c72ade640 10 monclient(hunting): picked mon.noname-a con 0x7f3c6c0a3d90 addr v2:172.21.15.42:3300/0 2024-09-16T03:42:29.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.161+0000 7f3c72ade640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f3c6c13c8c0 0x7f3c6c13ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:29.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.161+0000 7f3c72ade640 10 monclient(hunting): picked mon.noname-c con 0x7f3c6c13c8c0 addr v2:172.21.15.154:3300/0 2024-09-16T03:42:29.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.161+0000 7f3c72ade640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3c6c134290 con 0x7f3c6c0a39b0 2024-09-16T03:42:29.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.161+0000 7f3c72ade640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:29.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.161+0000 7f3c72ade640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f3c6c09c140 con 0x7f3c6c0a3d90 2024-09-16T03:42:29.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.161+0000 7f3c72ade640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:29.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.161+0000 7f3c72ade640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f3c6c09c420 con 0x7f3c6c13c8c0 2024-09-16T03:42:29.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.161+0000 7f3c72ade640 10 monclient(hunting): _renew_subs 2024-09-16T03:42:29.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.161+0000 7f3c72ade640 10 monclient(hunting): authenticate will time out at 996.132632s 2024-09-16T03:42:29.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.161+0000 7f3c6bfff640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f3c6c0a3d90 0x7f3c6c13c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:29.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.161+0000 7f3c6bfff640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f3c6c0a3d90 0x7f3c6c13c380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:37710/0 (socket says 172.21.15.42:37710) 2024-09-16T03:42:29.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.161+0000 7f3c6bfff640 1 -- 172.21.15.42:0/1739641781 learned_addr learned my addr 172.21.15.42:0/1739641781 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:42:29.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.161+0000 7f3c6bfff640 10 monclient(hunting): get_auth_request con 0x7f3c6c0a3d90 auth_method 0 2024-09-16T03:42:29.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.161+0000 7f3c6bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:29.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.161+0000 7f3c6bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:29.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.161+0000 7f3c6bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:29.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.161+0000 7f3c7107b640 1 --2- 172.21.15.42:0/1739641781 >> v2:172.21.15.154:3300/0 conn(0x7f3c6c13c8c0 0x7f3c6c13ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:29.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.161+0000 7f3c7107b640 10 monclient(hunting): get_auth_request con 0x7f3c6c13c8c0 auth_method 0 2024-09-16T03:42:29.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.161+0000 7f3c7107b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:29.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.161+0000 7f3c7107b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:29.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.161+0000 7f3c7107b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:29.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.161+0000 7f3c6bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:29.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.161+0000 7f3c6bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:29.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.161+0000 7f3c6bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:29.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.161+0000 7f3c7107b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:29.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.161+0000 7f3c7107b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:29.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.161+0000 7f3c7107b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:29.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.161+0000 7f3c6bfff640 10 monclient(hunting): handle_auth_done global_id 4299 payload 274 2024-09-16T03:42:29.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.161+0000 7f3c6bfff640 1 -- 172.21.15.42:0/1739641781 >> v2:172.21.15.154:3300/0 conn(0x7f3c6c13c8c0 msgr2=0x7f3c6c13ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:42:29.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.161+0000 7f3c6bfff640 1 --2- 172.21.15.42:0/1739641781 >> v2:172.21.15.154:3300/0 conn(0x7f3c6c13c8c0 0x7f3c6c13ecb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:29.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.161+0000 7f3c6bfff640 1 -- 172.21.15.42:0/1739641781 >> v1:172.21.15.154:6789/0 conn(0x7f3c6c0a39b0 legacy=0x7f3c6c133b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:29.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.161+0000 7f3c6bfff640 10 monclient: _finish_hunting 0 2024-09-16T03:42:29.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.161+0000 7f3c6bfff640 1 monclient: found mon.noname-a 2024-09-16T03:42:29.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.161+0000 7f3c6bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:42:29.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.161+0000 7f3c6bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:42:29.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.161+0000 7f3c6bfff640 1 -- 172.21.15.42:0/1739641781 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c6c06dd60 con 0x7f3c6c0a3d90 2024-09-16T03:42:29.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.165+0000 7f3c6bfff640 1 --2- 172.21.15.42:0/1739641781 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3c6c0a3d90 0x7f3c6c13c380 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f3c6c06b540 tx=0x7f3c5801c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d91b83ae5659cb21 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:29.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.165+0000 7f3c6b7fe640 1 -- 172.21.15.42:0/1739641781 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c580b0070 con 0x7f3c6c0a3d90 2024-09-16T03:42:29.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.165+0000 7f3c6b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:29.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.165+0000 7f3c6b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:42:29.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.165+0000 7f3c6b7fe640 10 monclient: dump: 2024-09-16T03:42:29.176 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:29.176 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:29.176 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:29.176 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:29.177 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:29.177 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:29.177 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:29.177 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:29.177 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:29.177 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:29.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.165+0000 7f3c6b7fe640 10 monclient: _finish_auth 0 2024-09-16T03:42:29.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.165+0000 7f3c6b7fe640 10 monclient: _check_auth_tickets 2024-09-16T03:42:29.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.165+0000 7f3c6b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:42:29.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.165+0000 7f3c6b7fe640 1 -- 172.21.15.42:0/1739641781 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3c580ab600 con 0x7f3c6c0a3d90 2024-09-16T03:42:29.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.165+0000 7f3c72ade640 5 monclient: authenticate success, global_id 4299 2024-09-16T03:42:29.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.165+0000 7f3c72ade640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:42:29.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.165+0000 7f3c6b7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:42:29.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.165+0000 7f3c72ade640 10 monclient: get_monmap_and_config success 2024-09-16T03:42:29.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.165+0000 7f3c72ade640 4 set_mon_vals no callback set 2024-09-16T03:42:29.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.165+0000 7f3c6b7fe640 1 -- 172.21.15.42:0/1739641781 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c580ab9d0 con 0x7f3c6c0a3d90 2024-09-16T03:42:29.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.165+0000 7f3c6b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:29.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.165+0000 7f3c6b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:42:29.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.165+0000 7f3c6b7fe640 10 monclient: dump: 2024-09-16T03:42:29.178 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:29.178 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:29.178 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:29.178 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:29.179 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:29.179 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:29.179 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:29.179 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:29.179 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:29.179 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:29.180 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.165+0000 7f3c72ade640 10 monclient: shutdown 2024-09-16T03:42:29.180 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.165+0000 7f3c72ade640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:42:29.180 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.165+0000 7f3c72ade640 1 -- 172.21.15.42:0/1739641781 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3c6c0a3d90 msgr2=0x7f3c6c13c380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:29.180 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.165+0000 7f3c72ade640 1 --2- 172.21.15.42:0/1739641781 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3c6c0a3d90 0x7f3c6c13c380 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f3c6c06b540 tx=0x7f3c5801c3e0 comp rx=0 tx=0).stop 2024-09-16T03:42:29.180 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.165+0000 7f3c72ade640 1 -- 172.21.15.42:0/1739641781 shutdown_connections 2024-09-16T03:42:29.180 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.165+0000 7f3c72ade640 1 --2- 172.21.15.42:0/1739641781 >> v2:172.21.15.154:3300/0 conn(0x7f3c6c13c8c0 0x7f3c6c13ecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:29.180 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.165+0000 7f3c72ade640 1 --2- 172.21.15.42:0/1739641781 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3c6c0a3d90 0x7f3c6c13c380 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:29.180 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.165+0000 7f3c72ade640 1 -- 172.21.15.42:0/1739641781 >> 172.21.15.42:0/1739641781 conn(0x7f3c6c0a9fb0 msgr2=0x7f3c6c0aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:42:29.180 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.165+0000 7f3c72ade640 1 -- 172.21.15.42:0/1739641781 shutdown_connections 2024-09-16T03:42:29.180 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.165+0000 7f3c72ade640 1 -- 172.21.15.42:0/1739641781 wait complete. 2024-09-16T03:42:29.180 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.165+0000 7f3c72ade640 10 monclient: build_initial_monmap 2024-09-16T03:42:29.181 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.165+0000 7f3c72ade640 10 monclient: monmap: 2024-09-16T03:42:29.181 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:42:29.181 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:42:29.181 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:42:29.181 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:42:29.181 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:42:29.181 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:29.181 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:42:29.181 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:42:29.181 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:42:29.181 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:29.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.165+0000 7fbbc1ba6640 10 monclient: get_monmap_and_config 2024-09-16T03:42:29.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.165+0000 7fbbc1ba6640 10 monclient: build_initial_monmap 2024-09-16T03:42:29.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.165+0000 7fbbc1ba6640 10 monclient: monmap: 2024-09-16T03:42:29.182 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:42:29.182 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:29.182 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:42:29.171413+0000 2024-09-16T03:42:29.182 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:42:29.171413+0000 2024-09-16T03:42:29.182 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:42:29.182 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:29.183 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:42:29.183 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:42:29.183 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:42:29.183 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:42:29.183 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:42:29.183 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:29.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.165+0000 7f3c72ade640 1 librados: starting msgr at 2024-09-16T03:42:29.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.165+0000 7f3c72ade640 1 librados: starting objecter 2024-09-16T03:42:29.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.165+0000 7f3c72ade640 1 Processor -- start 2024-09-16T03:42:29.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.165+0000 7f3c72ade640 1 -- start start 2024-09-16T03:42:29.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.165+0000 7f3c72ade640 1 librados: setting wanted keys 2024-09-16T03:42:29.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.165+0000 7f3c72ade640 1 librados: calling monclient init 2024-09-16T03:42:29.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.165+0000 7f3c72ade640 10 monclient: init 2024-09-16T03:42:29.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.169+0000 7fbbc1ba6640 1 Processor -- start 2024-09-16T03:42:29.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.169+0000 7fbbc1ba6640 1 -- start start 2024-09-16T03:42:29.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.169+0000 7fbbc1ba6640 10 monclient: init 2024-09-16T03:42:29.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.169+0000 7f3c72ade640 10 monclient: _reopen_session rank -1 2024-09-16T03:42:29.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.169+0000 7f3c72ade640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T03:42:29.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.169+0000 7f3c72ade640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3c6c0a39b0 0x7f3c6c13a080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:29.187 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.169+0000 7f3c72ade640 10 monclient(hunting): picked mon.noname-b con 0x7f3c6c0a39b0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:42:29.187 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.169+0000 7f3c72ade640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3c6c0a3d90 0x7f3c6c13a5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:29.187 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.169+0000 7f3c72ade640 10 monclient(hunting): picked mon.noname-c con 0x7f3c6c0a3d90 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:42:29.187 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.169+0000 7f3c72ade640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3c6c13c8c0 0x7f3c6c13ab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:29.187 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.169+0000 7f3c72ade640 10 monclient(hunting): picked mon.noname-a con 0x7f3c6c13c8c0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:42:29.187 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.169+0000 7f3c72ade640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:29.187 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.169+0000 7f3c72ade640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f3c6c141da0 con 0x7f3c6c13c8c0 2024-09-16T03:42:29.187 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.169+0000 7f3c72ade640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:29.187 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.169+0000 7f3c72ade640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f3c6c141c20 con 0x7f3c6c0a39b0 2024-09-16T03:42:29.187 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.169+0000 7f3c72ade640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:29.188 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.169+0000 7f3c72ade640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f3c6c141f20 con 0x7f3c6c0a3d90 2024-09-16T03:42:29.188 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.169+0000 7f3c72ade640 10 monclient(hunting): _renew_subs 2024-09-16T03:42:29.188 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.169+0000 7f3c72ade640 10 monclient(hunting): authenticate will time out at 996.140459s 2024-09-16T03:42:29.188 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.169+0000 7f3c7107b640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3c6c13c8c0 0x7f3c6c13ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:29.188 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.169+0000 7f3c7107b640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3c6c13c8c0 0x7f3c6c13ab00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:37714/0 (socket says 172.21.15.42:37714) 2024-09-16T03:42:29.188 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.169+0000 7f3c6bfff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3c6c0a3d90 0x7f3c6c13a5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:29.188 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.169+0000 7f3c7107b640 1 -- 172.21.15.42:0/3989964322 learned_addr learned my addr 172.21.15.42:0/3989964322 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:42:29.188 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.169+0000 7f3c7107b640 10 monclient(hunting): get_auth_request con 0x7f3c6c13c8c0 auth_method 0 2024-09-16T03:42:29.188 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.169+0000 7f3c7107b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:29.188 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.169+0000 7f3c7107b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:29.188 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.169+0000 7f3c7107b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:29.188 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.169+0000 7f3c6bfff640 10 monclient(hunting): get_auth_request con 0x7f3c6c0a3d90 auth_method 0 2024-09-16T03:42:29.189 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.169+0000 7f3c6bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:29.189 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.169+0000 7f3c6bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:29.189 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.169+0000 7f3c6bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:29.189 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.169+0000 7f3c6bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:29.189 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.169+0000 7f3c6bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:29.189 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.169+0000 7f3c6bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:29.190 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.169+0000 7fbbc1ba6640 10 monclient: _reopen_session rank -1 2024-09-16T03:42:29.190 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.169+0000 7fbbc1ba6640 10 monclient: _add_conns ranks=[3,1,4,2,0] 2024-09-16T03:42:29.190 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.169+0000 7fbbc1ba6640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fbbbc0a39b0 0x7fbbbc133b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:29.190 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.169+0000 7fbbc1ba6640 10 monclient(hunting): picked mon.noname-a con 0x7fbbbc0a39b0 addr v2:172.21.15.42:3300/0 2024-09-16T03:42:29.190 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.169+0000 7fbbc1ba6640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbbbc0a3d90 addr v1:172.21.15.154:6789/0 2024-09-16T03:42:29.190 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.169+0000 7fbbc1ba6640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fbbbc137710 0x7fbbbc13f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:29.190 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.169+0000 7fbbbb7fe640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fbbbc0a39b0 0x7fbbbc133b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:29.190 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.169+0000 7fbbc1ba6640 10 monclient(hunting): picked mon.noname-c con 0x7fbbbc137710 addr v2:172.21.15.154:3300/0 2024-09-16T03:42:29.191 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.169+0000 7f3c7107b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:29.191 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.169+0000 7f3c7107b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:29.191 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.169+0000 7f3c7107b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:29.191 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.169+0000 7f3c7087a640 1 --2- 172.21.15.42:0/3989964322 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3c6c0a39b0 0x7f3c6c13a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:29.191 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.169+0000 7f3c7087a640 10 monclient(hunting): get_auth_request con 0x7f3c6c0a39b0 auth_method 0 2024-09-16T03:42:29.191 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.169+0000 7f3c7087a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:29.191 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.169+0000 7f3c7087a640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:29.191 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.169+0000 7f3c7087a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:29.191 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.169+0000 7f3c6bfff640 10 monclient(hunting): handle_auth_done global_id 4262 payload 824 2024-09-16T03:42:29.191 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.169+0000 7f3c6bfff640 1 -- 172.21.15.42:0/3989964322 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3c6c0a39b0 msgr2=0x7f3c6c13a080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:29.192 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.169+0000 7f3c6bfff640 1 --2- 172.21.15.42:0/3989964322 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3c6c0a39b0 0x7f3c6c13a080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:29.192 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.169+0000 7f3c6bfff640 1 -- 172.21.15.42:0/3989964322 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3c6c13c8c0 msgr2=0x7f3c6c13ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:29.192 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.169+0000 7f3c6bfff640 1 --2- 172.21.15.42:0/3989964322 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3c6c13c8c0 0x7f3c6c13ab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:29.192 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.169+0000 7f3c6bfff640 10 monclient: _finish_hunting 0 2024-09-16T03:42:29.192 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.169+0000 7f3c6bfff640 1 monclient: found mon.noname-c 2024-09-16T03:42:29.192 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.169+0000 7f3c6bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:42:29.192 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.169+0000 7f3c6bfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:42:29.192 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.169+0000 7f3c6bfff640 1 -- 172.21.15.42:0/3989964322 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c6c06f680 con 0x7f3c6c0a3d90 2024-09-16T03:42:29.193 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.169+0000 7fbbbb7fe640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fbbbc0a39b0 0x7fbbbc133b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:37724/0 (socket says 172.21.15.42:37724) 2024-09-16T03:42:29.193 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.169+0000 7fbbc1ba6640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbbbc09c140 con 0x7fbbbc0a3d90 2024-09-16T03:42:29.193 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.169+0000 7fbbc1ba6640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:29.193 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.169+0000 7fbbc1ba6640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7fbbbc09c420 con 0x7fbbbc0a39b0 2024-09-16T03:42:29.193 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.169+0000 7fbbc1ba6640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:29.193 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.169+0000 7fbbc1ba6640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fbbbc06dd60 con 0x7fbbbc137710 2024-09-16T03:42:29.193 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.169+0000 7fbbc1ba6640 10 monclient(hunting): _renew_subs 2024-09-16T03:42:29.193 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.169+0000 7fbbc1ba6640 10 monclient(hunting): authenticate will time out at 996.141252s 2024-09-16T03:42:29.193 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.169+0000 7fbbbb7fe640 1 -- 172.21.15.42:0/2579966335 learned_addr learned my addr 172.21.15.42:0/2579966335 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:42:29.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.169+0000 7fbbbb7fe640 10 monclient(hunting): get_auth_request con 0x7fbbbc0a39b0 auth_method 0 2024-09-16T03:42:29.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.169+0000 7fbbbb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:29.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.169+0000 7fbbbb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:29.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.169+0000 7fbbbb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:29.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.169+0000 7f3c6bfff640 1 --2- 172.21.15.42:0/3989964322 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3c6c0a3d90 0x7f3c6c13a5c0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f3c580ab000 tx=0x7f3c58019180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:29.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.169+0000 7fbbbb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:29.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.169+0000 7fbbbbfff640 1 --2- 172.21.15.42:0/2579966335 >> v2:172.21.15.154:3300/0 conn(0x7fbbbc137710 0x7fbbbc13f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:29.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.169+0000 7fbbbb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:29.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.169+0000 7fbbbb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:29.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.169+0000 7f3c697fa640 10 client.?.objecter ms_handle_connect 0x7f3c6c0a3d90 2024-09-16T03:42:29.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.169+0000 7f3c697fa640 10 client.?.objecter resend_mon_ops 2024-09-16T03:42:29.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.169+0000 7fbbbbfff640 10 monclient(hunting): get_auth_request con 0x7fbbbc137710 auth_method 0 2024-09-16T03:42:29.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.169+0000 7fbbbbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:29.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.169+0000 7fbbbbfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:29.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.169+0000 7fbbbbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:29.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.169+0000 7fbbbbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:29.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.169+0000 7fbbbbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:29.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.169+0000 7fbbbbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:29.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.169+0000 7f3c697fa640 1 -- 172.21.15.42:0/3989964322 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c580b0040 con 0x7f3c6c0a3d90 2024-09-16T03:42:29.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.169+0000 7f3c697fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:29.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.169+0000 7f3c697fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:42:29.197 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.169+0000 7f3c697fa640 10 monclient: dump: 2024-09-16T03:42:29.197 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:29.197 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:29.197 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:29.197 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:29.197 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:29.197 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:29.197 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:29.197 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:29.197 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:29.197 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:29.197 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.169+0000 7f3c697fa640 10 monclient: _finish_auth 0 2024-09-16T03:42:29.197 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.169+0000 7f3c697fa640 10 monclient: _check_auth_tickets 2024-09-16T03:42:29.197 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.169+0000 7f3c697fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:42:29.198 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.169+0000 7f3c72ade640 5 monclient: authenticate success, global_id 4262 2024-09-16T03:42:29.198 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.169+0000 7f3c72ade640 10 monclient: _renew_subs 2024-09-16T03:42:29.198 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.169+0000 7f3c72ade640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:42:29.198 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.169+0000 7f3c72ade640 1 -- 172.21.15.42:0/3989964322 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c6c09e6a0 con 0x7f3c6c0a3d90 2024-09-16T03:42:29.198 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.169+0000 7fbbba7fc640 1 -- 172.21.15.42:0/2579966335 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 219923335 0 0) 0x7fbbbc09c140 con 0x7fbbbc0a3d90 2024-09-16T03:42:29.198 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.169+0000 7fbbba7fc640 10 monclient(hunting): _negotiate 2024-09-16T03:42:29.198 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.169+0000 7fbbba7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:29.198 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.169+0000 7fbbba7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:29.198 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.169+0000 7fbbba7fc640 10 monclient(hunting): my global_id is 4291 2024-09-16T03:42:29.199 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.169+0000 7fbbba7fc640 1 -- 172.21.15.42:0/2579966335 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbba4003530 con 0x7fbbbc0a3d90 2024-09-16T03:42:29.199 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.169+0000 7f3c72ade640 10 client.4262.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:42:29.199 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.169+0000 7f3c72ade640 10 monclient: _renew_subs 2024-09-16T03:42:29.199 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.169+0000 7f3c72ade640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:42:29.199 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.169+0000 7f3c72ade640 1 -- 172.21.15.42:0/3989964322 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c6c09d090 con 0x7f3c6c0a3d90 2024-09-16T03:42:29.199 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.169+0000 7f3c72ade640 1 librados: init done 2024-09-16T03:42:29.200 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.169+0000 7fbbbbfff640 10 monclient(hunting): handle_auth_done global_id 4294 payload 274 2024-09-16T03:42:29.200 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.169+0000 7fbbbbfff640 1 -- 172.21.15.42:0/2579966335 >> v2:172.21.15.42:3300/0 conn(0x7fbbbc0a39b0 msgr2=0x7fbbbc133b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:42:29.200 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.169+0000 7fbbbbfff640 1 --2- 172.21.15.42:0/2579966335 >> v2:172.21.15.42:3300/0 conn(0x7fbbbc0a39b0 0x7fbbbc133b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:29.200 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.169+0000 7fbbbbfff640 1 -- 172.21.15.42:0/2579966335 >> v1:172.21.15.154:6789/0 conn(0x7fbbbc0a3d90 legacy=0x7fbbbc137000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:29.200 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.169+0000 7fbbbbfff640 10 monclient: _finish_hunting 0 2024-09-16T03:42:29.200 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.169+0000 7fbbbbfff640 1 monclient: found mon.noname-c 2024-09-16T03:42:29.200 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.169+0000 7fbbbbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:42:29.200 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.169+0000 7fbbbbfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.154:3300/0 2024-09-16T03:42:29.200 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.169+0000 7fbbbbfff640 1 -- 172.21.15.42:0/2579966335 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbbc0a5e20 con 0x7fbbbc137710 2024-09-16T03:42:29.201 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.169+0000 7f3c697fa640 1 -- 172.21.15.42:0/3989964322 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3c580ab580 con 0x7f3c6c0a3d90 2024-09-16T03:42:29.201 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.169+0000 7f3c697fa640 10 monclient: handle_config config(0 keys) 2024-09-16T03:42:29.201 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.169+0000 7f3c697fa640 1 -- 172.21.15.42:0/3989964322 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c580b4660 con 0x7f3c6c0a3d90 2024-09-16T03:42:29.201 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.169+0000 7f3c697fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:29.201 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.169+0000 7f3c697fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:42:29.201 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.169+0000 7f3c697fa640 10 monclient: dump: 2024-09-16T03:42:29.202 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:29.202 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:29.202 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:29.202 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:29.202 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:29.202 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:29.202 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:29.202 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:29.202 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:29.202 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:29.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.169+0000 7f3c6affd640 4 set_mon_vals no callback set 2024-09-16T03:42:29.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.173+0000 7f3c697fa640 1 -- 172.21.15.42:0/3989964322 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f3c580bb020 con 0x7f3c6c0a3d90 2024-09-16T03:42:29.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.173+0000 7f3c697fa640 1 --2- 172.21.15.42:0/3989964322 >> v2:172.21.15.154:6812/254845342 conn(0x7f3c50037810 0x7f3c50039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:29.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.173+0000 7f3c697fa640 1 -- 172.21.15.42:0/3989964322 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(18..18 src has 1..18) ==== 3562+0+0 (secure 0 0 0) 0x7f3c580e1bf0 con 0x7f3c6c0a3d90 2024-09-16T03:42:29.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.173+0000 7f3c697fa640 10 client.4262.objecter ms_dispatch 0x7f3c6c000ba0 osd_map(18..18 src has 1..18) 2024-09-16T03:42:29.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.173+0000 7f3c697fa640 3 client.4262.objecter handle_osd_map got epochs [18,18] > 0 2024-09-16T03:42:29.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.173+0000 7f3c697fa640 3 client.4262.objecter handle_osd_map decoding full epoch 18 2024-09-16T03:42:29.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.173+0000 7f3c697fa640 20 client.4262.objecter dump_active .. 0 homeless 2024-09-16T03:42:29.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.173+0000 7f3c7087a640 1 --2- 172.21.15.42:0/3989964322 >> v2:172.21.15.154:6812/254845342 conn(0x7f3c50037810 0x7f3c50039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:29.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.173+0000 7f3c7087a640 10 monclient: get_auth_request con 0x7f3c50037810 auth_method 0 2024-09-16T03:42:29.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.173+0000 7fbbbbfff640 1 --2- 172.21.15.42:0/2579966335 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbbbc137710 0x7fbbbc13f3a0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fbbbc06b540 tx=0x7fbbb401c810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=450c5174433f34f4 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:29.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.173+0000 7f3c697fa640 10 client.4262.objecter ms_handle_connect 0x7f3c50037810 2024-09-16T03:42:29.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.173+0000 7f3c7087a640 1 --2- 172.21.15.42:0/3989964322 >> v2:172.21.15.154:6812/254845342 conn(0x7f3c50037810 0x7f3c50039cd0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f3c60000f70 tx=0x7f3c60009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:29.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.173+0000 7fbbba7fc640 1 -- 172.21.15.42:0/2579966335 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbb40a66c0 con 0x7fbbbc137710 2024-09-16T03:42:29.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.173+0000 7fbbba7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:29.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.173+0000 7fbbba7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:42:29.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.173+0000 7fbbba7fc640 10 monclient: dump: 2024-09-16T03:42:29.208 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:29.208 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:29.208 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:29.208 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:29.208 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:29.208 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:29.208 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:29.208 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:29.208 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:29.209 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:29.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.173+0000 7fbbba7fc640 10 monclient: _finish_auth 0 2024-09-16T03:42:29.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.173+0000 7fbbba7fc640 10 monclient: _check_auth_tickets 2024-09-16T03:42:29.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.173+0000 7fbbba7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:42:29.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.173+0000 7fbbba7fc640 1 -- 172.21.15.42:0/2579966335 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbbb40a6860 con 0x7fbbbc137710 2024-09-16T03:42:29.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.173+0000 7fbbba7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:42:29.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.173+0000 7fbbba7fc640 1 -- 172.21.15.42:0/2579966335 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbb40a6c10 con 0x7fbbbc137710 2024-09-16T03:42:29.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.173+0000 7fbbc1ba6640 5 monclient: authenticate success, global_id 4294 2024-09-16T03:42:29.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.173+0000 7fbbc1ba6640 10 monclient: get_monmap_and_config success 2024-09-16T03:42:29.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.173+0000 7fbbc1ba6640 4 set_mon_vals no callback set 2024-09-16T03:42:29.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.173+0000 7fbbba7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:29.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.173+0000 7fbbba7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:42:29.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.173+0000 7fbbba7fc640 10 monclient: dump: 2024-09-16T03:42:29.210 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:29.210 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:29.210 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:29.210 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:29.210 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:29.210 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:29.210 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:29.210 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:29.210 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:29.210 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:29.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.173+0000 7fbbc1ba6640 10 monclient: shutdown 2024-09-16T03:42:29.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.173+0000 7fbbc1ba6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:42:29.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.173+0000 7fbbc1ba6640 1 -- 172.21.15.42:0/2579966335 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbbbc137710 msgr2=0x7fbbbc13f3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:29.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.173+0000 7fbbc1ba6640 1 --2- 172.21.15.42:0/2579966335 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbbbc137710 0x7fbbbc13f3a0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fbbbc06b540 tx=0x7fbbb401c810 comp rx=0 tx=0).stop 2024-09-16T03:42:29.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.173+0000 7fbbc1ba6640 1 -- 172.21.15.42:0/2579966335 shutdown_connections 2024-09-16T03:42:29.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.173+0000 7fbbc1ba6640 1 --2- 172.21.15.42:0/2579966335 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbbbc137710 0x7fbbbc13f3a0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:29.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.173+0000 7fbbc1ba6640 1 --2- 172.21.15.42:0/2579966335 >> v2:172.21.15.42:3300/0 conn(0x7fbbbc0a39b0 0x7fbbbc133b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:29.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.173+0000 7fbbc1ba6640 1 -- 172.21.15.42:0/2579966335 >> 172.21.15.42:0/2579966335 conn(0x7fbbbc0a9fb0 msgr2=0x7fbbbc0aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:42:29.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.173+0000 7fbbc1ba6640 1 -- 172.21.15.42:0/2579966335 shutdown_connections 2024-09-16T03:42:29.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.173+0000 7fbbc1ba6640 1 -- 172.21.15.42:0/2579966335 wait complete. 2024-09-16T03:42:29.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.173+0000 7fbbc1ba6640 10 monclient: build_initial_monmap 2024-09-16T03:42:29.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.173+0000 7fbbc1ba6640 10 monclient: monmap: 2024-09-16T03:42:29.211 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:42:29.211 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:42:29.211 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:42:29.211 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:42:29.211 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:42:29.211 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:29.212 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:42:29.212 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:42:29.212 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:42:29.212 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:29.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.173+0000 7fbbc1ba6640 1 librados: starting msgr at 2024-09-16T03:42:29.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.173+0000 7fbbc1ba6640 1 librados: starting objecter 2024-09-16T03:42:29.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.177+0000 7fbbc1ba6640 1 Processor -- start 2024-09-16T03:42:29.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.177+0000 7fbbc1ba6640 1 -- start start 2024-09-16T03:42:29.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.177+0000 7fbbc1ba6640 1 librados: setting wanted keys 2024-09-16T03:42:29.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.177+0000 7fbbc1ba6640 1 librados: calling monclient init 2024-09-16T03:42:29.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.177+0000 7fbbc1ba6640 10 monclient: init 2024-09-16T03:42:29.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.177+0000 7fbbc1ba6640 10 monclient: _reopen_session rank -1 2024-09-16T03:42:29.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.177+0000 7fbbc1ba6640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T03:42:29.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.177+0000 7fbbc1ba6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbbbc0a39b0 0x7fbbbc130560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:29.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.177+0000 7fbbc1ba6640 10 monclient(hunting): picked mon.noname-b con 0x7fbbbc0a39b0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:42:29.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.177+0000 7fbbc1ba6640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbbbc0a3d90 0x7fbbbc130aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:29.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.177+0000 7fbbc1ba6640 10 monclient(hunting): picked mon.noname-c con 0x7fbbbc0a3d90 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:42:29.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.177+0000 7fbbc1ba6640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbbbc137710 0x7fbbbc0a1910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:29.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.177+0000 7fbbc1ba6640 10 monclient(hunting): picked mon.noname-a con 0x7fbbbc137710 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:42:29.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.177+0000 7fbbc1ba6640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:29.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.177+0000 7fbbc1ba6640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fbbbc142310 con 0x7fbbbc137710 2024-09-16T03:42:29.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.177+0000 7fbbc1ba6640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:29.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.177+0000 7fbbc1ba6640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fbbbc142490 con 0x7fbbbc0a39b0 2024-09-16T03:42:29.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.177+0000 7fbbc1ba6640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:29.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.177+0000 7fbbc1ba6640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fbbbc142610 con 0x7fbbbc0a3d90 2024-09-16T03:42:29.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.177+0000 7fbbc1ba6640 10 monclient(hunting): _renew_subs 2024-09-16T03:42:29.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.177+0000 7fbbc1ba6640 10 monclient(hunting): authenticate will time out at 996.148980s 2024-09-16T03:42:29.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.177+0000 7fbbbaffd640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbbbc0a3d90 0x7fbbbc130aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:29.214 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.177+0000 7fbbbaffd640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbbbc0a3d90 0x7fbbbc130aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:41392/0 (socket says 172.21.15.42:41392) 2024-09-16T03:42:29.214 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.177+0000 7fbbbaffd640 1 -- 172.21.15.42:0/497227334 learned_addr learned my addr 172.21.15.42:0/497227334 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:42:29.214 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.177+0000 7fbbbaffd640 10 monclient(hunting): get_auth_request con 0x7fbbbc0a3d90 auth_method 0 2024-09-16T03:42:29.214 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.177+0000 7fbbbaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:29.214 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.177+0000 7fbbbaffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:29.214 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.177+0000 7fbbbaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:29.214 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.177+0000 7fbbbaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:29.214 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.177+0000 7fbbbaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:29.214 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.177+0000 7fbbbaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:29.214 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.177+0000 7fbbbbfff640 1 --2- 172.21.15.42:0/497227334 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbbbc137710 0x7fbbbc0a1910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:29.214 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.177+0000 7fbbbb7fe640 1 --2- 172.21.15.42:0/497227334 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbbbc0a39b0 0x7fbbbc130560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:29.214 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.177+0000 7fbbbbfff640 10 monclient(hunting): get_auth_request con 0x7fbbbc137710 auth_method 0 2024-09-16T03:42:29.214 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.177+0000 7fbbbbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:29.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.177+0000 7fbbbbfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:29.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.177+0000 7fbbbbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:29.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.177+0000 7fbbbb7fe640 10 monclient(hunting): get_auth_request con 0x7fbbbc0a39b0 auth_method 0 2024-09-16T03:42:29.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.177+0000 7fbbbb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:29.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.177+0000 7fbbbb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:29.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.177+0000 7fbbbb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:29.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.177+0000 7fbbbb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:29.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.177+0000 7fbbbb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:29.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.177+0000 7fbbbb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:29.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.177+0000 7fbbbaffd640 10 monclient(hunting): handle_auth_done global_id 4265 payload 824 2024-09-16T03:42:29.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.177+0000 7fbbbaffd640 1 -- 172.21.15.42:0/497227334 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbbbc0a39b0 msgr2=0x7fbbbc130560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:29.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.177+0000 7fbbbaffd640 1 --2- 172.21.15.42:0/497227334 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbbbc0a39b0 0x7fbbbc130560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:29.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.177+0000 7fbbbaffd640 1 -- 172.21.15.42:0/497227334 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbbbc137710 msgr2=0x7fbbbc0a1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:29.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.177+0000 7fbbbaffd640 1 --2- 172.21.15.42:0/497227334 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbbbc137710 0x7fbbbc0a1910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:29.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.177+0000 7fbbbaffd640 10 monclient: _finish_hunting 0 2024-09-16T03:42:29.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.177+0000 7fbbbaffd640 1 monclient: found mon.noname-c 2024-09-16T03:42:29.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.177+0000 7fbbbaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:42:29.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.177+0000 7fbbbaffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:42:29.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.177+0000 7fbbbaffd640 1 -- 172.21.15.42:0/497227334 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbbc139a70 con 0x7fbbbc0a3d90 2024-09-16T03:42:29.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.177+0000 7f3c72ade640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:42:29.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.177+0000 7f3c72ade640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:42:29.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.177+0000 7f3c72ade640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:42:29.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.177+0000 7fbbbaffd640 1 --2- 172.21.15.42:0/497227334 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbbbc0a3d90 0x7fbbbc130aa0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fbbb000c4c0 tx=0x7fbbb000c980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:29.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.177+0000 7fbba3fff640 10 client.?.objecter ms_handle_connect 0x7fbbbc0a3d90 2024-09-16T03:42:29.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.177+0000 7fbba3fff640 10 client.?.objecter resend_mon_ops 2024-09-16T03:42:29.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.177+0000 7f94e3715640 10 monclient: get_monmap_and_config 2024-09-16T03:42:29.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.177+0000 7f3c72ade640 1 -- 172.21.15.42:0/3989964322 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3c4c005090 con 0x7f3c6c0a3d90 2024-09-16T03:42:29.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.177+0000 7f94e3715640 10 monclient: build_initial_monmap 2024-09-16T03:42:29.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.177+0000 7f94e3715640 10 monclient: monmap: 2024-09-16T03:42:29.218 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:42:29.218 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:29.218 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:42:29.182303+0000 2024-09-16T03:42:29.218 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:42:29.182303+0000 2024-09-16T03:42:29.219 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:42:29.219 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:29.219 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:42:29.219 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:42:29.219 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:42:29.219 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:42:29.219 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:42:29.219 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:29.220 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.177+0000 7fbba3fff640 1 -- 172.21.15.42:0/497227334 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbb00052d0 con 0x7fbbbc0a3d90 2024-09-16T03:42:29.220 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.177+0000 7fbba3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:29.220 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.177+0000 7fbba3fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:42:29.220 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.177+0000 7fbba3fff640 10 monclient: dump: 2024-09-16T03:42:29.220 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:29.220 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:29.221 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:29.221 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:29.221 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:29.221 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:29.221 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:29.221 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:29.221 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:29.221 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:29.221 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.177+0000 7fbba3fff640 10 monclient: _finish_auth 0 2024-09-16T03:42:29.221 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.177+0000 7fbba3fff640 10 monclient: _check_auth_tickets 2024-09-16T03:42:29.221 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.177+0000 7fbba3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:42:29.221 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.177+0000 7fbba3fff640 1 -- 172.21.15.42:0/497227334 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbbb0005470 con 0x7fbbbc0a3d90 2024-09-16T03:42:29.222 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.177+0000 7fbba3fff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:42:29.222 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.177+0000 7fbba3fff640 1 -- 172.21.15.42:0/497227334 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbb0012870 con 0x7fbbbc0a3d90 2024-09-16T03:42:29.222 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.177+0000 7fbba3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:29.222 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.177+0000 7fbba3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:42:29.222 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.177+0000 7fbba3fff640 10 monclient: dump: 2024-09-16T03:42:29.222 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:29.222 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:29.222 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:29.222 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:29.222 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:29.222 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:29.222 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:29.222 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:29.222 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:29.223 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:29.223 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.177+0000 7fbbc1ba6640 5 monclient: authenticate success, global_id 4265 2024-09-16T03:42:29.223 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.177+0000 7fbbc1ba6640 10 monclient: _renew_subs 2024-09-16T03:42:29.223 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.177+0000 7fbbc1ba6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:42:29.223 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.177+0000 7fbbc1ba6640 1 -- 172.21.15.42:0/497227334 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbbbc139d50 con 0x7fbbbc0a3d90 2024-09-16T03:42:29.223 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.177+0000 7fbbb9ffb640 4 set_mon_vals no callback set 2024-09-16T03:42:29.223 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.177+0000 7fbbc1ba6640 10 client.4265.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:42:29.223 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.177+0000 7fbbc1ba6640 10 monclient: _renew_subs 2024-09-16T03:42:29.223 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.177+0000 7fbbc1ba6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:42:29.223 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.177+0000 7fbbc1ba6640 1 -- 172.21.15.42:0/497227334 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fbbbc06f680 con 0x7fbbbc0a3d90 2024-09-16T03:42:29.223 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.177+0000 7fbbc1ba6640 1 librados: init done 2024-09-16T03:42:29.223 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.177+0000 7fbbc1ba6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:42:29.223 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.177+0000 7fbbc1ba6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:42:29.223 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.177+0000 7fbbc1ba6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:42:29.224 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.177+0000 7fbbc1ba6640 1 -- 172.21.15.42:0/497227334 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbbbc133b80 con 0x7fbbbc0a3d90 2024-09-16T03:42:29.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.177+0000 7f94e3715640 1 Processor -- start 2024-09-16T03:42:29.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.177+0000 7f94e3715640 1 -- start start 2024-09-16T03:42:29.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.177+0000 7f94e3715640 10 monclient: init 2024-09-16T03:42:29.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.181+0000 7f94e3715640 10 monclient: _reopen_session rank -1 2024-09-16T03:42:29.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.181+0000 7f94e3715640 10 monclient: _add_conns ranks=[4,1,3,2,0] 2024-09-16T03:42:29.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.181+0000 7f94e3715640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f94dc0a2f10 0x7f94dc0a1330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:29.230 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.181+0000 7f94e3715640 10 monclient(hunting): picked mon.noname-c con 0x7f94dc0a2f10 addr v2:172.21.15.154:3300/0 2024-09-16T03:42:29.230 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.181+0000 7f94e3715640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f94dc09e070 addr v1:172.21.15.154:6789/0 2024-09-16T03:42:29.230 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.181+0000 7f94e3715640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f94dc140b30 0x7f94dc142f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:29.230 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.181+0000 7f94e3715640 10 monclient(hunting): picked mon.noname-a con 0x7f94dc140b30 addr v2:172.21.15.42:3300/0 2024-09-16T03:42:29.230 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.181+0000 7f94e3715640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f94dc09dc50 con 0x7f94dc09e070 2024-09-16T03:42:29.230 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.181+0000 7f94e3715640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:29.230 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.181+0000 7f94e3715640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f94dc0a32f0 con 0x7f94dc140b30 2024-09-16T03:42:29.230 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.181+0000 7f94e3715640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:29.230 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.181+0000 7f94e3715640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f94dc073290 con 0x7f94dc0a2f10 2024-09-16T03:42:29.230 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.181+0000 7f94e3715640 10 monclient(hunting): _renew_subs 2024-09-16T03:42:29.230 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.181+0000 7f94e3715640 10 monclient(hunting): authenticate will time out at 996.152181s 2024-09-16T03:42:29.230 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.181+0000 7f94e1cb2640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f94dc140b30 0x7f94dc142f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:29.230 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.181+0000 7f94e1cb2640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f94dc140b30 0x7f94dc142f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:37730/0 (socket says 172.21.15.42:37730) 2024-09-16T03:42:29.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.181+0000 7f94e1cb2640 1 -- 172.21.15.42:0/621723256 learned_addr learned my addr 172.21.15.42:0/621723256 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:42:29.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.181+0000 7f94e1cb2640 10 monclient(hunting): get_auth_request con 0x7f94dc140b30 auth_method 0 2024-09-16T03:42:29.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.181+0000 7f94e1cb2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:29.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.181+0000 7f94e1cb2640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:29.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.181+0000 7f94e1cb2640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:29.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.181+0000 7f94e14b1640 1 --2- 172.21.15.42:0/621723256 >> v2:172.21.15.154:3300/0 conn(0x7f94dc0a2f10 0x7f94dc0a1330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:29.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.181+0000 7f94e14b1640 10 monclient(hunting): get_auth_request con 0x7f94dc0a2f10 auth_method 0 2024-09-16T03:42:29.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.181+0000 7f94e14b1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:29.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.181+0000 7f94e14b1640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:29.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.181+0000 7f94e14b1640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:29.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.181+0000 7f94e1cb2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:29.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.181+0000 7f94e1cb2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:29.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.181+0000 7f94e1cb2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:29.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.181+0000 7f94e14b1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:29.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.181+0000 7f94e14b1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:29.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.181+0000 7f94e14b1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:29.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.181+0000 7f94e1cb2640 10 monclient(hunting): handle_auth_done global_id 4311 payload 274 2024-09-16T03:42:29.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.181+0000 7f94e1cb2640 1 -- 172.21.15.42:0/621723256 >> v2:172.21.15.154:3300/0 conn(0x7f94dc0a2f10 msgr2=0x7f94dc0a1330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:42:29.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.181+0000 7f94e1cb2640 1 --2- 172.21.15.42:0/621723256 >> v2:172.21.15.154:3300/0 conn(0x7f94dc0a2f10 0x7f94dc0a1330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:29.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.181+0000 7f94e1cb2640 1 -- 172.21.15.42:0/621723256 >> v1:172.21.15.154:6789/0 conn(0x7f94dc09e070 legacy=0x7f94dc0a1870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:29.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.181+0000 7f94e1cb2640 10 monclient: _finish_hunting 0 2024-09-16T03:42:29.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.181+0000 7f94e1cb2640 1 monclient: found mon.noname-a 2024-09-16T03:42:29.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.181+0000 7f94e1cb2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:42:29.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.181+0000 7f94e1cb2640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:42:29.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.181+0000 7f94e1cb2640 1 -- 172.21.15.42:0/621723256 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94dc09e600 con 0x7f94dc140b30 2024-09-16T03:42:29.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.181+0000 7f94e1cb2640 1 --2- 172.21.15.42:0/621723256 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f94dc140b30 0x7f94dc142f20 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f94dc06add0 tx=0x7f94cc01c3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e548f2e09b7131 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:29.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.181+0000 7f94d3fff640 1 -- 172.21.15.42:0/621723256 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94cc0b1070 con 0x7f94dc140b30 2024-09-16T03:42:29.233 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.181+0000 7f94d3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:29.233 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.181+0000 7f94d3fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:42:29.233 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.181+0000 7f94d3fff640 10 monclient: dump: 2024-09-16T03:42:29.233 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:29.233 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:29.233 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:29.233 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:29.233 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:29.233 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:29.233 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:29.233 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:29.233 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:29.233 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:29.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.181+0000 7f94d3fff640 10 monclient: _finish_auth 0 2024-09-16T03:42:29.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.181+0000 7f94d3fff640 10 monclient: _check_auth_tickets 2024-09-16T03:42:29.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.181+0000 7f94d3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:42:29.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.181+0000 7f94d3fff640 1 -- 172.21.15.42:0/621723256 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f94cc0ac410 con 0x7f94dc140b30 2024-09-16T03:42:29.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.181+0000 7f94d3fff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:42:29.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.181+0000 7f94d3fff640 1 -- 172.21.15.42:0/621723256 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94cc0ac7e0 con 0x7f94dc140b30 2024-09-16T03:42:29.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.181+0000 7f94e3715640 5 monclient: authenticate success, global_id 4311 2024-09-16T03:42:29.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.181+0000 7f94e3715640 10 monclient: get_monmap_and_config success 2024-09-16T03:42:29.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.181+0000 7f94e3715640 4 set_mon_vals no callback set 2024-09-16T03:42:29.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.181+0000 7f94d3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:29.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.181+0000 7f94d3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:42:29.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.181+0000 7f94d3fff640 10 monclient: dump: 2024-09-16T03:42:29.234 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:29.235 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:29.235 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:29.235 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:29.235 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:29.235 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:29.235 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:29.235 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:29.235 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:29.235 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:29.235 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.181+0000 7f94e3715640 10 monclient: shutdown 2024-09-16T03:42:29.235 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.181+0000 7f94e3715640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:42:29.235 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.181+0000 7f94e3715640 1 -- 172.21.15.42:0/621723256 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f94dc140b30 msgr2=0x7f94dc142f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:29.235 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.181+0000 7f94e3715640 1 --2- 172.21.15.42:0/621723256 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f94dc140b30 0x7f94dc142f20 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f94dc06add0 tx=0x7f94cc01c3b0 comp rx=0 tx=0).stop 2024-09-16T03:42:29.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.181+0000 7f94e3715640 1 -- 172.21.15.42:0/621723256 shutdown_connections 2024-09-16T03:42:29.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.181+0000 7f94e3715640 1 --2- 172.21.15.42:0/621723256 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f94dc140b30 0x7f94dc142f20 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:29.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.181+0000 7f94e3715640 1 --2- 172.21.15.42:0/621723256 >> v2:172.21.15.154:3300/0 conn(0x7f94dc0a2f10 0x7f94dc0a1330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:29.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.181+0000 7f94e3715640 1 -- 172.21.15.42:0/621723256 >> 172.21.15.42:0/621723256 conn(0x7f94dc0a2420 msgr2=0x7f94dc0a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:42:29.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.181+0000 7f94e3715640 1 -- 172.21.15.42:0/621723256 shutdown_connections 2024-09-16T03:42:29.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.181+0000 7f94e3715640 1 -- 172.21.15.42:0/621723256 wait complete. 2024-09-16T03:42:29.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.185+0000 7f94e3715640 10 monclient: build_initial_monmap 2024-09-16T03:42:29.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.185+0000 7f94e3715640 10 monclient: monmap: 2024-09-16T03:42:29.236 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:42:29.236 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:42:29.236 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:42:29.236 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:42:29.236 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:42:29.237 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:29.237 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:42:29.237 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:42:29.237 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:42:29.237 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:29.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.185+0000 7f94e3715640 1 librados: starting msgr at 2024-09-16T03:42:29.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.185+0000 7f94e3715640 1 librados: starting objecter 2024-09-16T03:42:29.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.185+0000 7f94e3715640 1 Processor -- start 2024-09-16T03:42:29.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.185+0000 7f94e3715640 1 -- start start 2024-09-16T03:42:29.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.185+0000 7f94e3715640 1 librados: setting wanted keys 2024-09-16T03:42:29.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.185+0000 7f94e3715640 1 librados: calling monclient init 2024-09-16T03:42:29.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.185+0000 7f94e3715640 10 monclient: init 2024-09-16T03:42:29.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.185+0000 7f94e3715640 10 monclient: _reopen_session rank -1 2024-09-16T03:42:29.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.185+0000 7f94e3715640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:42:29.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.185+0000 7f94e3715640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f94dc09e070 0x7f94dc1d2f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:29.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.185+0000 7f94e3715640 10 monclient(hunting): picked mon.noname-a con 0x7f94dc09e070 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:42:29.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.185+0000 7f94e3715640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f94dc0a2f10 0x7f94dc13bab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:29.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.185+0000 7f94e3715640 10 monclient(hunting): picked mon.noname-c con 0x7f94dc0a2f10 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:42:29.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.185+0000 7f94e3715640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f94dc140b30 0x7f94dc070750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:29.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.185+0000 7f94e3715640 10 monclient(hunting): picked mon.noname-b con 0x7f94dc140b30 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:42:29.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.185+0000 7f94e3715640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:29.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.185+0000 7f94e3715640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f94dc145e00 con 0x7f94dc09e070 2024-09-16T03:42:29.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.185+0000 7f94e0cb0640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f94dc0a2f10 0x7f94dc13bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:29.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.185+0000 7f94e14b1640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f94dc09e070 0x7f94dc1d2f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:29.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.185+0000 7f94e0cb0640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f94dc0a2f10 0x7f94dc13bab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:41402/0 (socket says 172.21.15.42:41402) 2024-09-16T03:42:29.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.185+0000 7f94e0cb0640 1 -- 172.21.15.42:0/1882887614 learned_addr learned my addr 172.21.15.42:0/1882887614 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:42:29.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.185+0000 7f94e3715640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:29.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.185+0000 7f94e3715640 1 -- 172.21.15.42:0/1882887614 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f94dc145c80 con 0x7f94dc140b30 2024-09-16T03:42:29.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.185+0000 7f94e3715640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:29.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.185+0000 7f94e3715640 1 -- 172.21.15.42:0/1882887614 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f94dc145f80 con 0x7f94dc0a2f10 2024-09-16T03:42:29.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.185+0000 7f94e3715640 10 monclient(hunting): _renew_subs 2024-09-16T03:42:29.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.185+0000 7f94e3715640 10 monclient(hunting): authenticate will time out at 996.157974s 2024-09-16T03:42:29.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.185+0000 7f94e0cb0640 10 monclient(hunting): get_auth_request con 0x7f94dc0a2f10 auth_method 0 2024-09-16T03:42:29.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.185+0000 7f94e0cb0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:29.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.185+0000 7f94e0cb0640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:29.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.185+0000 7f94e0cb0640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:29.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.185+0000 7f94e14b1640 10 monclient(hunting): get_auth_request con 0x7f94dc09e070 auth_method 0 2024-09-16T03:42:29.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.185+0000 7f94e14b1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:29.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.185+0000 7f94e14b1640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:29.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.185+0000 7f94e14b1640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:29.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.185+0000 7f94e0cb0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:29.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.185+0000 7f94e0cb0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:29.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.185+0000 7f94e0cb0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:29.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.185+0000 7f94e1cb2640 1 --2- 172.21.15.42:0/1882887614 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f94dc140b30 0x7f94dc070750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:29.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.185+0000 7f94e14b1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:29.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.185+0000 7f94e14b1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:29.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.185+0000 7f94e14b1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:29.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.185+0000 7f94e1cb2640 10 monclient(hunting): get_auth_request con 0x7f94dc140b30 auth_method 0 2024-09-16T03:42:29.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.185+0000 7f94e1cb2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:29.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.185+0000 7f94e1cb2640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:29.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.185+0000 7f94e1cb2640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:29.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.185+0000 7f94e0cb0640 10 monclient(hunting): handle_auth_done global_id 4268 payload 824 2024-09-16T03:42:29.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.185+0000 7f94e0cb0640 1 -- 172.21.15.42:0/1882887614 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f94dc140b30 msgr2=0x7f94dc070750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:29.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.185+0000 7f94e0cb0640 1 --2- 172.21.15.42:0/1882887614 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f94dc140b30 0x7f94dc070750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:29.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.185+0000 7f94e0cb0640 1 -- 172.21.15.42:0/1882887614 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f94dc09e070 msgr2=0x7f94dc1d2f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:29.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.185+0000 7f94e0cb0640 1 --2- 172.21.15.42:0/1882887614 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f94dc09e070 0x7f94dc1d2f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:29.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.185+0000 7f94e0cb0640 10 monclient: _finish_hunting 0 2024-09-16T03:42:29.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.185+0000 7f94e0cb0640 1 monclient: found mon.noname-c 2024-09-16T03:42:29.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.185+0000 7f94e0cb0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:42:29.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.185+0000 7f94e0cb0640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:42:29.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.185+0000 7f94e0cb0640 1 -- 172.21.15.42:0/1882887614 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94dc126440 con 0x7f94dc0a2f10 2024-09-16T03:42:29.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.185+0000 7f94e0cb0640 1 --2- 172.21.15.42:0/1882887614 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f94dc0a2f10 0x7f94dc13bab0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f94d800c3f0 tx=0x7f94d800c8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:29.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.185+0000 7f94d1ffb640 10 client.?.objecter ms_handle_connect 0x7f94dc0a2f10 2024-09-16T03:42:29.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.185+0000 7f94d1ffb640 10 client.?.objecter resend_mon_ops 2024-09-16T03:42:29.242 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.189+0000 7fbba3fff640 1 -- 172.21.15.42:0/497227334 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fbbb0012a10 con 0x7fbbbc0a3d90 2024-09-16T03:42:29.242 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.189+0000 7fbba3fff640 1 --2- 172.21.15.42:0/497227334 >> v2:172.21.15.154:6812/254845342 conn(0x7fbba8037810 0x7fbba8039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:29.242 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.189+0000 7fbba3fff640 1 -- 172.21.15.42:0/497227334 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(18..18 src has 1..18) ==== 3562+0+0 (secure 0 0 0) 0x7fbbb0048f10 con 0x7fbbbc0a3d90 2024-09-16T03:42:29.242 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.189+0000 7fbba3fff640 10 client.4265.objecter ms_dispatch 0x7fbbbc000ba0 osd_map(18..18 src has 1..18) 2024-09-16T03:42:29.242 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.189+0000 7fbba3fff640 3 client.4265.objecter handle_osd_map got epochs [18,18] > 0 2024-09-16T03:42:29.242 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.189+0000 7fbba3fff640 3 client.4265.objecter handle_osd_map decoding full epoch 18 2024-09-16T03:42:29.242 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.189+0000 7fbba3fff640 20 client.4265.objecter dump_active .. 0 homeless 2024-09-16T03:42:29.242 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.189+0000 7fbbbb7fe640 1 --2- 172.21.15.42:0/497227334 >> v2:172.21.15.154:6812/254845342 conn(0x7fbba8037810 0x7fbba8039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:29.242 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.189+0000 7fbbbb7fe640 10 monclient: get_auth_request con 0x7fbba8037810 auth_method 0 2024-09-16T03:42:29.242 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.189+0000 7fbbbb7fe640 1 --2- 172.21.15.42:0/497227334 >> v2:172.21.15.154:6812/254845342 conn(0x7fbba8037810 0x7fbba8039cd0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fbbac009b50 tx=0x7fbbac009240 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:29.242 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.189+0000 7fbba3fff640 10 client.4265.objecter ms_handle_connect 0x7fbba8037810 2024-09-16T03:42:29.243 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.189+0000 7f3c697fa640 1 -- 172.21.15.42:0/3989964322 <== mon.2 v2:172.21.15.42:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f3c580c2020 con 0x7f3c6c0a3d90 2024-09-16T03:42:29.243 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.189+0000 7f3c697fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:42:29.243 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.189+0000 7f3c697fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:42:29.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.193+0000 7f94d1ffb640 1 -- 172.21.15.42:0/1882887614 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94d8017070 con 0x7f94dc0a2f10 2024-09-16T03:42:29.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.193+0000 7f94d1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:29.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.193+0000 7f94d1ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:42:29.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.193+0000 7f94d1ffb640 10 monclient: dump: 2024-09-16T03:42:29.244 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:29.244 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:29.244 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:29.245 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:29.245 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:29.245 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:29.245 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:29.245 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:29.245 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:29.245 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:29.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.193+0000 7f94d1ffb640 10 monclient: _finish_auth 0 2024-09-16T03:42:29.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.193+0000 7f94d1ffb640 10 monclient: _check_auth_tickets 2024-09-16T03:42:29.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.193+0000 7f94d1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:42:29.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.193+0000 7f94d1ffb640 1 -- 172.21.15.42:0/1882887614 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f94d80052d0 con 0x7f94dc0a2f10 2024-09-16T03:42:29.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.193+0000 7f94d1ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T03:42:29.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.193+0000 7f94d1ffb640 1 -- 172.21.15.42:0/1882887614 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94d8003840 con 0x7f94dc0a2f10 2024-09-16T03:42:29.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.193+0000 7f94d1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:29.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.193+0000 7f94d1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:42:29.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.193+0000 7f94d1ffb640 10 monclient: dump: 2024-09-16T03:42:29.246 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:29.246 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:29.246 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:29.246 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:29.246 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:29.246 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:29.246 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:29.246 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:29.246 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:29.246 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:29.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.193+0000 7f94d37fe640 4 set_mon_vals no callback set 2024-09-16T03:42:29.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.193+0000 7f94e3715640 5 monclient: authenticate success, global_id 4268 2024-09-16T03:42:29.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.193+0000 7f94e3715640 10 monclient: _renew_subs 2024-09-16T03:42:29.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.193+0000 7f94e3715640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:42:29.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.193+0000 7f94e3715640 1 -- 172.21.15.42:0/1882887614 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94dc0a47a0 con 0x7f94dc0a2f10 2024-09-16T03:42:29.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.193+0000 7f94e3715640 10 client.4268.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:42:29.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.193+0000 7f94e3715640 10 monclient: _renew_subs 2024-09-16T03:42:29.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.193+0000 7f94e3715640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:42:29.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.193+0000 7f94e3715640 1 -- 172.21.15.42:0/1882887614 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f94dc0723f0 con 0x7f94dc0a2f10 2024-09-16T03:42:29.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.193+0000 7f94e3715640 1 librados: init done 2024-09-16T03:42:29.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.193+0000 7f94bf7fe640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:42:29.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.193+0000 7f94bf7fe640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:42:29.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.193+0000 7f94bf7fe640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:42:29.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.193+0000 7f94bf7fe640 1 -- 172.21.15.42:0/1882887614 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f94b8005090 con 0x7f94dc0a2f10 2024-09-16T03:42:29.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.201+0000 7fbba3fff640 1 -- 172.21.15.42:0/497227334 <== mon.2 v2:172.21.15.42:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7fbbb0003db0 con 0x7fbbbc0a3d90 2024-09-16T03:42:29.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.201+0000 7fbba3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:42:29.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.201+0000 7fbba3fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:42:29.249 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.201+0000 7f94d1ffb640 1 -- 172.21.15.42:0/1882887614 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f94d801a070 con 0x7f94dc0a2f10 2024-09-16T03:42:29.249 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.201+0000 7f94d1ffb640 1 --2- 172.21.15.42:0/1882887614 >> v2:172.21.15.154:6812/254845342 conn(0x7f94c0037810 0x7f94c0039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:29.249 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.201+0000 7f94d1ffb640 1 -- 172.21.15.42:0/1882887614 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(18..18 src has 1..18) ==== 3562+0+0 (secure 0 0 0) 0x7f94d8011590 con 0x7f94dc0a2f10 2024-09-16T03:42:29.249 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.201+0000 7f94d1ffb640 10 client.4268.objecter ms_dispatch 0x7f94dc000ba0 osd_map(18..18 src has 1..18) 2024-09-16T03:42:29.249 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.201+0000 7f94d1ffb640 3 client.4268.objecter handle_osd_map got epochs [18,18] > 0 2024-09-16T03:42:29.249 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.201+0000 7f94d1ffb640 3 client.4268.objecter handle_osd_map decoding full epoch 18 2024-09-16T03:42:29.249 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.201+0000 7f94d1ffb640 20 client.4268.objecter dump_active .. 0 homeless 2024-09-16T03:42:29.249 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.201+0000 7f94e14b1640 1 --2- 172.21.15.42:0/1882887614 >> v2:172.21.15.154:6812/254845342 conn(0x7f94c0037810 0x7f94c0039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:29.249 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.201+0000 7f94e14b1640 10 monclient: get_auth_request con 0x7f94c0037810 auth_method 0 2024-09-16T03:42:29.249 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.201+0000 7f94d1ffb640 10 client.4268.objecter ms_handle_connect 0x7f94c0037810 2024-09-16T03:42:29.249 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.201+0000 7f94e14b1640 1 --2- 172.21.15.42:0/1882887614 >> v2:172.21.15.154:6812/254845342 conn(0x7f94c0037810 0x7f94c0039cd0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f94d40113e0 tx=0x7f94d400b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:29.249 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.209+0000 7f94d1ffb640 1 -- 172.21.15.42:0/1882887614 <== mon.2 v2:172.21.15.42:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f94d8021020 con 0x7f94dc0a2f10 2024-09-16T03:42:29.250 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.209+0000 7f94d1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:42:29.250 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.209+0000 7f94d1ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:42:29.301 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.293+0000 7f58cb500640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-16T03:42:29.301 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.293+0000 7f58cb500640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-16T03:42:29.301 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.293+0000 7f58cb500640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:42:29.301 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.293+0000 7f58cb500640 1 -- 172.21.15.42:0/3729999718 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f58c413cba0 con 0x7f58c40a39b0 2024-09-16T03:42:29.301 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.293+0000 7f58b9ffb640 1 -- 172.21.15.42:0/3729999718 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f58c00160f0 con 0x7f58c40a39b0 2024-09-16T03:42:29.301 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.293+0000 7f58b9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-16T03:42:29.301 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.293+0000 7f58b9ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:42:29.302 INFO:teuthology.orchestra.run.smithi042.stdout:47244640258 2024-09-16T03:42:29.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.297+0000 7f58cb500640 10 librados: watch_flush enter 2024-09-16T03:42:29.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.297+0000 7f58cb500640 10 librados: watch_flush exit 2024-09-16T03:42:29.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.297+0000 7f58cb500640 20 client.4284.objecter shutdown clearing up homeless session... 2024-09-16T03:42:29.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.297+0000 7f58cb500640 10 client.4284.objecter successfully canceled tick 2024-09-16T03:42:29.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.297+0000 7f58cb500640 1 -- 172.21.15.42:0/3729999718 >> v2:172.21.15.154:6812/254845342 conn(0x7f58a8037840 msgr2=0x7f58a8039d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:29.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.297+0000 7f58cb500640 1 --2- 172.21.15.42:0/3729999718 >> v2:172.21.15.154:6812/254845342 conn(0x7f58a8037840 0x7f58a8039d00 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f58c413c2d0 tx=0x7f58b4007e40 comp rx=0 tx=0).stop 2024-09-16T03:42:29.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.297+0000 7f58cb500640 10 monclient: shutdown 2024-09-16T03:42:29.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.297+0000 7f58cb500640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:42:29.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.297+0000 7f58cb500640 1 -- 172.21.15.42:0/3729999718 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f58c40a39b0 msgr2=0x7f58c413b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:29.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.297+0000 7f58cb500640 1 --2- 172.21.15.42:0/3729999718 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f58c40a39b0 0x7f58c413b990 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f58c001cc80 tx=0x7f58c0019180 comp rx=0 tx=0).stop 2024-09-16T03:42:29.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.297+0000 7f58cb500640 1 -- 172.21.15.42:0/3729999718 shutdown_connections 2024-09-16T03:42:29.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.297+0000 7f58cb500640 1 --2- 172.21.15.42:0/3729999718 >> v2:172.21.15.154:6812/254845342 conn(0x7f58a8037840 0x7f58a8039d00 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:29.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.297+0000 7f58cb500640 1 --2- 172.21.15.42:0/3729999718 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f58c4137710 0x7f58c41396c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:29.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.297+0000 7f58cb500640 1 --2- 172.21.15.42:0/3729999718 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f58c40a3d90 0x7f58c4139180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:29.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.297+0000 7f58cb500640 1 --2- 172.21.15.42:0/3729999718 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f58c40a39b0 0x7f58c413b990 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:29.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.297+0000 7f58cb500640 1 -- 172.21.15.42:0/3729999718 >> 172.21.15.42:0/3729999718 conn(0x7f58c40a9fb0 msgr2=0x7f58c4131980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:42:29.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.297+0000 7f58cb500640 1 -- 172.21.15.42:0/3729999718 shutdown_connections 2024-09-16T03:42:29.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.297+0000 7f58cb500640 1 -- 172.21.15.42:0/3729999718 wait complete. 2024-09-16T03:42:29.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.297+0000 7f58cb500640 1 librados: shutdown 2024-09-16T03:42:29.320 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640260 got 47244640258 for osd.2 2024-09-16T03:42:29.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.345+0000 7f49c8e83640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-16T03:42:29.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.345+0000 7f49c8e83640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-16T03:42:29.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.345+0000 7f49c8e83640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:42:29.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.345+0000 7f49c8e83640 1 -- 172.21.15.42:0/3023943085 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f49a8005650 con 0x7f49c413daa0 2024-09-16T03:42:29.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.349+0000 7f49baffd640 1 -- 172.21.15.42:0/3023943085 <== mon.2 v2:172.21.15.42:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f49bc0be030 con 0x7f49c413daa0 2024-09-16T03:42:29.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.349+0000 7f49baffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-16T03:42:29.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.349+0000 7f49baffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:42:29.355 INFO:teuthology.orchestra.run.smithi042.stdout:47244640258 2024-09-16T03:42:29.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.349+0000 7f49c8e83640 10 librados: watch_flush enter 2024-09-16T03:42:29.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.349+0000 7f49c8e83640 10 librados: watch_flush exit 2024-09-16T03:42:29.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.349+0000 7f49c8e83640 20 client.4259.objecter shutdown clearing up homeless session... 2024-09-16T03:42:29.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.349+0000 7f49c8e83640 10 client.4259.objecter successfully canceled tick 2024-09-16T03:42:29.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.349+0000 7f49c8e83640 1 -- 172.21.15.42:0/3023943085 >> v2:172.21.15.154:6812/254845342 conn(0x7f49a4037810 msgr2=0x7f49a4039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:29.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.349+0000 7f49c8e83640 1 --2- 172.21.15.42:0/3023943085 >> v2:172.21.15.154:6812/254845342 conn(0x7f49a4037810 0x7f49a4039cd0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f49b4000f70 tx=0x7f49b4007ee0 comp rx=0 tx=0).stop 2024-09-16T03:42:29.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.349+0000 7f49c8e83640 10 monclient: shutdown 2024-09-16T03:42:29.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.349+0000 7f49c8e83640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:42:29.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.349+0000 7f49c8e83640 1 -- 172.21.15.42:0/3023943085 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f49c413daa0 msgr2=0x7f49c413a070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:29.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.349+0000 7f49c8e83640 1 --2- 172.21.15.42:0/3023943085 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f49c413daa0 0x7f49c413a070 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f49bc0ac000 tx=0x7f49bc0195d0 comp rx=0 tx=0).stop 2024-09-16T03:42:29.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.349+0000 7f49c8e83640 1 -- 172.21.15.42:0/3023943085 shutdown_connections 2024-09-16T03:42:29.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.349+0000 7f49c8e83640 1 --2- 172.21.15.42:0/3023943085 >> v2:172.21.15.154:6812/254845342 conn(0x7f49a4037810 0x7f49a4039cd0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:29.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.349+0000 7f49c8e83640 1 --2- 172.21.15.42:0/3023943085 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f49c413daa0 0x7f49c413a070 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:29.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.349+0000 7f49c8e83640 1 --2- 172.21.15.42:0/3023943085 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f49c40a3d90 0x7f49c4130aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:29.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.349+0000 7f49c8e83640 1 --2- 172.21.15.42:0/3023943085 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f49c40a39b0 0x7f49c4130560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:29.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.349+0000 7f49c8e83640 1 -- 172.21.15.42:0/3023943085 >> 172.21.15.42:0/3023943085 conn(0x7f49c40a9fb0 msgr2=0x7f49c40a7ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:42:29.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.349+0000 7f49c8e83640 1 -- 172.21.15.42:0/3023943085 shutdown_connections 2024-09-16T03:42:29.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.349+0000 7f49c8e83640 1 -- 172.21.15.42:0/3023943085 wait complete. 2024-09-16T03:42:29.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.349+0000 7f49c8e83640 1 librados: shutdown 2024-09-16T03:42:29.372 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640260 got 47244640258 for osd.0 2024-09-16T03:42:29.418 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.409+0000 7fd71a637640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-16T03:42:29.418 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.409+0000 7fd71a637640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-16T03:42:29.418 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.409+0000 7fd71a637640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:42:29.418 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.409+0000 7fd71a637640 1 -- 172.21.15.42:0/56590665 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7fd7140a5750 con 0x7fd7140a3d90 2024-09-16T03:42:29.418 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.413+0000 7fd710ff9640 1 -- 172.21.15.42:0/56590665 <== mon.2 v2:172.21.15.42:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fd70801c940 con 0x7fd7140a3d90 2024-09-16T03:42:29.419 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.413+0000 7fd710ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-16T03:42:29.419 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.413+0000 7fd710ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:42:29.419 INFO:teuthology.orchestra.run.smithi042.stdout:51539607556 2024-09-16T03:42:29.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.413+0000 7fd71a637640 10 librados: watch_flush enter 2024-09-16T03:42:29.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.413+0000 7fd71a637640 10 librados: watch_flush exit 2024-09-16T03:42:29.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.413+0000 7fd71a637640 20 client.4256.objecter shutdown clearing up homeless session... 2024-09-16T03:42:29.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.413+0000 7fd71a637640 10 client.4256.objecter successfully canceled tick 2024-09-16T03:42:29.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.413+0000 7fd71a637640 1 -- 172.21.15.42:0/56590665 >> v2:172.21.15.154:6812/254845342 conn(0x7fd700037860 msgr2=0x7fd700039d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:29.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.413+0000 7fd71a637640 1 --2- 172.21.15.42:0/56590665 >> v2:172.21.15.154:6812/254845342 conn(0x7fd700037860 0x7fd700039d20 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fd7040ad000 tx=0x7fd7040a6000 comp rx=0 tx=0).stop 2024-09-16T03:42:29.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.413+0000 7fd71a637640 10 monclient: shutdown 2024-09-16T03:42:29.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.413+0000 7fd71a637640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:42:29.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.413+0000 7fd71a637640 1 -- 172.21.15.42:0/56590665 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd7140a3d90 msgr2=0x7fd7141dd3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:29.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.413+0000 7fd71a637640 1 --2- 172.21.15.42:0/56590665 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd7140a3d90 0x7fd7141dd3f0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fd70800c450 tx=0x7fd70800c910 comp rx=0 tx=0).stop 2024-09-16T03:42:29.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.413+0000 7fd71a637640 1 -- 172.21.15.42:0/56590665 shutdown_connections 2024-09-16T03:42:29.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.413+0000 7fd71a637640 1 --2- 172.21.15.42:0/56590665 >> v2:172.21.15.154:6812/254845342 conn(0x7fd700037860 0x7fd700039d20 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:29.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.413+0000 7fd71a637640 1 --2- 172.21.15.42:0/56590665 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd714137710 0x7fd71409d210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:29.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.413+0000 7fd71a637640 1 --2- 172.21.15.42:0/56590665 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd7140a3d90 0x7fd7141dd3f0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:29.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.413+0000 7fd71a637640 1 --2- 172.21.15.42:0/56590665 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd7140a39b0 0x7fd7141dceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:29.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.413+0000 7fd71a637640 1 -- 172.21.15.42:0/56590665 >> 172.21.15.42:0/56590665 conn(0x7fd7140a9fb0 msgr2=0x7fd714131980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:42:29.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.413+0000 7fd71a637640 1 -- 172.21.15.42:0/56590665 shutdown_connections 2024-09-16T03:42:29.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.413+0000 7fd71a637640 1 -- 172.21.15.42:0/56590665 wait complete. 2024-09-16T03:42:29.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.413+0000 7fd71a637640 1 librados: shutdown 2024-09-16T03:42:29.440 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.433+0000 7fbbc1ba6640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-16T03:42:29.440 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.433+0000 7fbbc1ba6640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-16T03:42:29.440 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.433+0000 7fbbc1ba6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:42:29.440 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.433+0000 7fbbc1ba6640 1 -- 172.21.15.42:0/497227334 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7fbbbc13f6c0 con 0x7fbbbc0a3d90 2024-09-16T03:42:29.440 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.433+0000 7fbba3fff640 1 -- 172.21.15.42:0/497227334 <== mon.2 v2:172.21.15.42:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fbbb00245a0 con 0x7fbbbc0a3d90 2024-09-16T03:42:29.440 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.433+0000 7fbba3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-16T03:42:29.440 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.433+0000 7fbba3fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:42:29.441 INFO:teuthology.orchestra.run.smithi042.stdout:47244640258 2024-09-16T03:42:29.442 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607558 got 51539607556 for osd.5 2024-09-16T03:42:29.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.433+0000 7fbbc1ba6640 10 librados: watch_flush enter 2024-09-16T03:42:29.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.433+0000 7fbbc1ba6640 10 librados: watch_flush exit 2024-09-16T03:42:29.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.433+0000 7fbbc1ba6640 20 client.4265.objecter shutdown clearing up homeless session... 2024-09-16T03:42:29.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.433+0000 7fbbc1ba6640 10 client.4265.objecter successfully canceled tick 2024-09-16T03:42:29.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.433+0000 7fbbc1ba6640 1 -- 172.21.15.42:0/497227334 >> v2:172.21.15.154:6812/254845342 conn(0x7fbba8037810 msgr2=0x7fbba8039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:29.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.433+0000 7fbbc1ba6640 1 --2- 172.21.15.42:0/497227334 >> v2:172.21.15.154:6812/254845342 conn(0x7fbba8037810 0x7fbba8039cd0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fbbac009b50 tx=0x7fbbac009240 comp rx=0 tx=0).stop 2024-09-16T03:42:29.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.433+0000 7fbbc1ba6640 10 monclient: shutdown 2024-09-16T03:42:29.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.433+0000 7fbbc1ba6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:42:29.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.433+0000 7fbbc1ba6640 1 -- 172.21.15.42:0/497227334 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbbbc0a3d90 msgr2=0x7fbbbc130aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:29.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.433+0000 7fbbc1ba6640 1 --2- 172.21.15.42:0/497227334 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbbbc0a3d90 0x7fbbbc130aa0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fbbb000c4c0 tx=0x7fbbb000c980 comp rx=0 tx=0).stop 2024-09-16T03:42:29.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.433+0000 7fbbc1ba6640 1 -- 172.21.15.42:0/497227334 shutdown_connections 2024-09-16T03:42:29.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.433+0000 7fbbc1ba6640 1 --2- 172.21.15.42:0/497227334 >> v2:172.21.15.154:6812/254845342 conn(0x7fbba8037810 0x7fbba8039cd0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:29.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.433+0000 7fbbc1ba6640 1 --2- 172.21.15.42:0/497227334 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbbbc137710 0x7fbbbc0a1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:29.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.433+0000 7fbbc1ba6640 1 --2- 172.21.15.42:0/497227334 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbbbc0a3d90 0x7fbbbc130aa0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:29.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.433+0000 7fbbc1ba6640 1 --2- 172.21.15.42:0/497227334 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbbbc0a39b0 0x7fbbbc130560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:29.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.433+0000 7fbbc1ba6640 1 -- 172.21.15.42:0/497227334 >> 172.21.15.42:0/497227334 conn(0x7fbbbc0a9fb0 msgr2=0x7fbbbc0a79b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:42:29.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.433+0000 7fbbc1ba6640 1 -- 172.21.15.42:0/497227334 shutdown_connections 2024-09-16T03:42:29.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.433+0000 7fbbc1ba6640 1 -- 172.21.15.42:0/497227334 wait complete. 2024-09-16T03:42:29.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.433+0000 7fbbc1ba6640 1 librados: shutdown 2024-09-16T03:42:29.445 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.437+0000 7f94bf7fe640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-16T03:42:29.445 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.437+0000 7f94bf7fe640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-16T03:42:29.445 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.437+0000 7f94bf7fe640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:42:29.446 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.437+0000 7f94bf7fe640 1 -- 172.21.15.42:0/1882887614 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f94b8005650 con 0x7f94dc0a2f10 2024-09-16T03:42:29.446 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.441+0000 7f94d1ffb640 1 -- 172.21.15.42:0/1882887614 <== mon.2 v2:172.21.15.42:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f94d8045f00 con 0x7f94dc0a2f10 2024-09-16T03:42:29.446 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.441+0000 7f94d1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-16T03:42:29.446 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.441+0000 7f94d1ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:42:29.446 INFO:teuthology.orchestra.run.smithi042.stdout:47244640260 2024-09-16T03:42:29.448 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.441+0000 7f94bf7fe640 10 librados: watch_flush enter 2024-09-16T03:42:29.448 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.441+0000 7f94bf7fe640 10 librados: watch_flush exit 2024-09-16T03:42:29.448 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.441+0000 7f94bf7fe640 20 client.4268.objecter shutdown clearing up homeless session... 2024-09-16T03:42:29.448 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.441+0000 7f94bf7fe640 10 client.4268.objecter successfully canceled tick 2024-09-16T03:42:29.448 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.441+0000 7f94bf7fe640 1 -- 172.21.15.42:0/1882887614 >> v2:172.21.15.154:6812/254845342 conn(0x7f94c0037810 msgr2=0x7f94c0039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:29.449 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.441+0000 7f94bf7fe640 1 --2- 172.21.15.42:0/1882887614 >> v2:172.21.15.154:6812/254845342 conn(0x7f94c0037810 0x7f94c0039cd0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f94d40113e0 tx=0x7f94d400b040 comp rx=0 tx=0).stop 2024-09-16T03:42:29.449 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.441+0000 7f94bf7fe640 10 monclient: shutdown 2024-09-16T03:42:29.449 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.441+0000 7f94bf7fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:42:29.449 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.441+0000 7f94bf7fe640 1 -- 172.21.15.42:0/1882887614 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f94dc0a2f10 msgr2=0x7f94dc13bab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:29.449 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.441+0000 7f94bf7fe640 1 --2- 172.21.15.42:0/1882887614 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f94dc0a2f10 0x7f94dc13bab0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f94d800c3f0 tx=0x7f94d800c8b0 comp rx=0 tx=0).stop 2024-09-16T03:42:29.449 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.441+0000 7f94bf7fe640 1 -- 172.21.15.42:0/1882887614 shutdown_connections 2024-09-16T03:42:29.449 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.441+0000 7f94bf7fe640 1 --2- 172.21.15.42:0/1882887614 >> v2:172.21.15.154:6812/254845342 conn(0x7f94c0037810 0x7f94c0039cd0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:29.449 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.441+0000 7f94bf7fe640 1 --2- 172.21.15.42:0/1882887614 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f94dc140b30 0x7f94dc070750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:29.449 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.441+0000 7f94bf7fe640 1 --2- 172.21.15.42:0/1882887614 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f94dc0a2f10 0x7f94dc13bab0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:29.449 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.441+0000 7f94bf7fe640 1 --2- 172.21.15.42:0/1882887614 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f94dc09e070 0x7f94dc1d2f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:29.449 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.441+0000 7f94bf7fe640 1 -- 172.21.15.42:0/1882887614 >> 172.21.15.42:0/1882887614 conn(0x7f94dc0a2420 msgr2=0x7f94dc142070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:42:29.449 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.441+0000 7f94bf7fe640 1 -- 172.21.15.42:0/1882887614 shutdown_connections 2024-09-16T03:42:29.449 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.441+0000 7f94bf7fe640 1 -- 172.21.15.42:0/1882887614 wait complete. 2024-09-16T03:42:29.450 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.441+0000 7f94bf7fe640 1 librados: shutdown 2024-09-16T03:42:29.458 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640260 got 47244640258 for osd.3 2024-09-16T03:42:29.464 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640262 got 47244640260 for osd.1 2024-09-16T03:42:29.476 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.469+0000 7f3c72ade640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-16T03:42:29.476 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.469+0000 7f3c72ade640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-16T03:42:29.476 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.469+0000 7f3c72ade640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:42:29.476 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.469+0000 7f3c72ade640 1 -- 172.21.15.42:0/3989964322 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f3c4c005650 con 0x7f3c6c0a3d90 2024-09-16T03:42:29.476 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.469+0000 7f3c697fa640 1 -- 172.21.15.42:0/3989964322 <== mon.2 v2:172.21.15.42:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f3c6c06f680 con 0x7f3c6c0a3d90 2024-09-16T03:42:29.476 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.469+0000 7f3c697fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-16T03:42:29.476 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.469+0000 7f3c697fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:42:29.476 INFO:teuthology.orchestra.run.smithi042.stdout:47244640258 2024-09-16T03:42:29.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.469+0000 7f3c72ade640 10 librados: watch_flush enter 2024-09-16T03:42:29.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.469+0000 7f3c72ade640 10 librados: watch_flush exit 2024-09-16T03:42:29.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.469+0000 7f3c72ade640 20 client.4262.objecter shutdown clearing up homeless session... 2024-09-16T03:42:29.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.469+0000 7f3c72ade640 10 client.4262.objecter successfully canceled tick 2024-09-16T03:42:29.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.469+0000 7f3c72ade640 1 -- 172.21.15.42:0/3989964322 >> v2:172.21.15.154:6812/254845342 conn(0x7f3c50037810 msgr2=0x7f3c50039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:29.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.469+0000 7f3c72ade640 1 --2- 172.21.15.42:0/3989964322 >> v2:172.21.15.154:6812/254845342 conn(0x7f3c50037810 0x7f3c50039cd0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f3c60000f70 tx=0x7f3c60009040 comp rx=0 tx=0).stop 2024-09-16T03:42:29.479 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.469+0000 7f3c72ade640 10 monclient: shutdown 2024-09-16T03:42:29.479 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.469+0000 7f3c72ade640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:42:29.479 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.469+0000 7f3c72ade640 1 -- 172.21.15.42:0/3989964322 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3c6c0a3d90 msgr2=0x7f3c6c13a5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:29.479 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.469+0000 7f3c72ade640 1 --2- 172.21.15.42:0/3989964322 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3c6c0a3d90 0x7f3c6c13a5c0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f3c580ab000 tx=0x7f3c58019180 comp rx=0 tx=0).stop 2024-09-16T03:42:29.479 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.469+0000 7f3c72ade640 1 -- 172.21.15.42:0/3989964322 shutdown_connections 2024-09-16T03:42:29.479 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.469+0000 7f3c72ade640 1 --2- 172.21.15.42:0/3989964322 >> v2:172.21.15.154:6812/254845342 conn(0x7f3c50037810 0x7f3c50039cd0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:29.479 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.469+0000 7f3c72ade640 1 --2- 172.21.15.42:0/3989964322 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3c6c13c8c0 0x7f3c6c13ab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:29.479 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.469+0000 7f3c72ade640 1 --2- 172.21.15.42:0/3989964322 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3c6c0a3d90 0x7f3c6c13a5c0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:29.479 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.469+0000 7f3c72ade640 1 --2- 172.21.15.42:0/3989964322 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3c6c0a39b0 0x7f3c6c13a080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:29.479 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.469+0000 7f3c72ade640 1 -- 172.21.15.42:0/3989964322 >> 172.21.15.42:0/3989964322 conn(0x7f3c6c0a9fb0 msgr2=0x7f3c6c1331d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:42:29.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.469+0000 7f3c72ade640 1 -- 172.21.15.42:0/3989964322 shutdown_connections 2024-09-16T03:42:29.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.469+0000 7f3c72ade640 1 -- 172.21.15.42:0/3989964322 wait complete. 2024-09-16T03:42:29.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:29.473+0000 7f3c72ade640 1 librados: shutdown 2024-09-16T03:42:29.492 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640260 got 47244640258 for osd.4 2024-09-16T03:42:30.320 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-16T03:42:30.372 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-16T03:42:30.442 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-16T03:42:30.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.445+0000 7fd86fc78640 10 monclient: get_monmap_and_config 2024-09-16T03:42:30.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.445+0000 7fd86fc78640 10 monclient: build_initial_monmap 2024-09-16T03:42:30.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.445+0000 7fd86fc78640 10 monclient: monmap: 2024-09-16T03:42:30.453 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:42:30.453 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:30.453 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:42:30.451461+0000 2024-09-16T03:42:30.453 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:42:30.451461+0000 2024-09-16T03:42:30.454 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:42:30.454 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:30.454 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:42:30.454 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:42:30.454 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:42:30.454 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:42:30.454 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:42:30.454 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:30.455 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.449+0000 7fd86fc78640 1 Processor -- start 2024-09-16T03:42:30.456 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.449+0000 7fd86fc78640 1 -- start start 2024-09-16T03:42:30.456 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.449+0000 7fd86fc78640 10 monclient: init 2024-09-16T03:42:30.456 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.449+0000 7fd86fc78640 10 monclient: _reopen_session rank -1 2024-09-16T03:42:30.456 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.449+0000 7fd86fc78640 10 monclient: _add_conns ranks=[2,0,1,3,4] 2024-09-16T03:42:30.456 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.449+0000 7fd86fc78640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd8680a67e0 0x7fd8680a0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:30.456 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.449+0000 7fd86fc78640 10 monclient(hunting): picked mon.noname-b con 0x7fd8680a67e0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:42:30.456 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.449+0000 7fd86fc78640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd8680a6bc0 addr v1:172.21.15.42:6789/0 2024-09-16T03:42:30.456 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.449+0000 7fd86fc78640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd86813c2f0 addr v1:172.21.15.154:6789/0 2024-09-16T03:42:30.456 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.449+0000 7fd86fc78640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd868072600 con 0x7fd8680a6bc0 2024-09-16T03:42:30.456 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.449+0000 7fd86fc78640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd8680a0430 con 0x7fd86813c2f0 2024-09-16T03:42:30.456 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.449+0000 7fd86fc78640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:30.456 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.449+0000 7fd86fc78640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fd8680a19c0 con 0x7fd8680a67e0 2024-09-16T03:42:30.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.449+0000 7fd86fc78640 10 monclient(hunting): _renew_subs 2024-09-16T03:42:30.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.449+0000 7fd86fc78640 10 monclient(hunting): authenticate will time out at 997.420851s 2024-09-16T03:42:30.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.449+0000 7fd86d213640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fd8680a6bc0 0x7fd8680a6fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:46802/0 (socket says 172.21.15.42:46802) 2024-09-16T03:42:30.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.449+0000 7fd86d213640 1 -- 172.21.15.42:0/2995383292 learned_addr learned my addr 172.21.15.42:0/2995383292 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:42:30.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.449+0000 7fd86da14640 1 --2- 172.21.15.42:0/2995383292 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd8680a67e0 0x7fd8680a0bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:30.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.449+0000 7fd86da14640 10 monclient(hunting): get_auth_request con 0x7fd8680a67e0 auth_method 0 2024-09-16T03:42:30.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.449+0000 7fd86da14640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:30.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.449+0000 7fd86da14640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:30.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.449+0000 7fd86da14640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:30.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.449+0000 7fd86da14640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:30.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.449+0000 7fd86da14640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:30.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.449+0000 7fd86da14640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:30.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.449+0000 7fd86da14640 10 monclient(hunting): handle_auth_done global_id 4271 payload 274 2024-09-16T03:42:30.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.449+0000 7fd86da14640 1 -- 172.21.15.42:0/2995383292 >> v1:172.21.15.154:6789/0 conn(0x7fd86813c2f0 legacy=0x7fd86813e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:42:30.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.449+0000 7fd86da14640 1 -- 172.21.15.42:0/2995383292 >> v1:172.21.15.42:6789/0 conn(0x7fd8680a6bc0 legacy=0x7fd8680a6fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:30.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.449+0000 7fd86da14640 10 monclient: _finish_hunting 0 2024-09-16T03:42:30.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.449+0000 7fd86da14640 1 monclient: found mon.noname-b 2024-09-16T03:42:30.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.449+0000 7fd86da14640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:42:30.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.449+0000 7fd86da14640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T03:42:30.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.449+0000 7fd86da14640 1 -- 172.21.15.42:0/2995383292 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8680a16e0 con 0x7fd8680a67e0 2024-09-16T03:42:30.458 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-16T03:42:30.460 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.449+0000 7fd86da14640 1 --2- 172.21.15.42:0/2995383292 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd8680a67e0 0x7fd8680a0bc0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fd8640097f0 tx=0x7fd86401bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b0e2349862500036 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:30.460 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.449+0000 7fd86ca12640 1 -- 172.21.15.42:0/2995383292 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8640ab070 con 0x7fd8680a67e0 2024-09-16T03:42:30.461 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.449+0000 7fd86ca12640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:30.461 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.449+0000 7fd86ca12640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:42:30.461 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.449+0000 7fd86ca12640 10 monclient: dump: 2024-09-16T03:42:30.461 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:30.461 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:30.461 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:30.461 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:30.461 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:30.461 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:30.461 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:30.461 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:30.461 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:30.461 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:30.461 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.449+0000 7fd86ca12640 10 monclient: _finish_auth 0 2024-09-16T03:42:30.462 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.449+0000 7fd86ca12640 10 monclient: _check_auth_tickets 2024-09-16T03:42:30.462 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.449+0000 7fd86ca12640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:42:30.462 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.449+0000 7fd86fc78640 5 monclient: authenticate success, global_id 4271 2024-09-16T03:42:30.462 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.449+0000 7fd86fc78640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:42:30.462 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.449+0000 7fd86ca12640 1 -- 172.21.15.42:0/2995383292 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd8640a5630 con 0x7fd8680a67e0 2024-09-16T03:42:30.462 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.449+0000 7fd86ca12640 10 monclient: handle_config config(0 keys) 2024-09-16T03:42:30.462 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.449+0000 7fd86ca12640 1 -- 172.21.15.42:0/2995383292 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8640a5a00 con 0x7fd8680a67e0 2024-09-16T03:42:30.462 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.449+0000 7fd86ca12640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:30.462 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.449+0000 7fd86ca12640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:42:30.462 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.449+0000 7fd86ca12640 10 monclient: dump: 2024-09-16T03:42:30.462 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:30.462 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:30.462 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:30.462 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:30.463 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:30.463 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:30.463 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:30.463 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:30.463 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:30.463 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:30.463 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.449+0000 7fd86fc78640 10 monclient: get_monmap_and_config success 2024-09-16T03:42:30.463 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.449+0000 7fd86fc78640 4 set_mon_vals no callback set 2024-09-16T03:42:30.463 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.453+0000 7fd86fc78640 10 monclient: shutdown 2024-09-16T03:42:30.463 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.453+0000 7fd86fc78640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:42:30.463 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.453+0000 7fd86fc78640 1 -- 172.21.15.42:0/2995383292 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd8680a67e0 msgr2=0x7fd8680a0bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:30.463 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.453+0000 7fd86fc78640 1 --2- 172.21.15.42:0/2995383292 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd8680a67e0 0x7fd8680a0bc0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fd8640097f0 tx=0x7fd86401bf60 comp rx=0 tx=0).stop 2024-09-16T03:42:30.463 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.453+0000 7fd86fc78640 1 -- 172.21.15.42:0/2995383292 shutdown_connections 2024-09-16T03:42:30.463 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.453+0000 7fd86fc78640 1 --2- 172.21.15.42:0/2995383292 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd8680a67e0 0x7fd8680a0bc0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:30.463 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.453+0000 7fd86fc78640 1 -- 172.21.15.42:0/2995383292 >> 172.21.15.42:0/2995383292 conn(0x7fd8680a7ed0 msgr2=0x7fd8680a43d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:42:30.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.453+0000 7fd86fc78640 1 -- 172.21.15.42:0/2995383292 shutdown_connections 2024-09-16T03:42:30.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.453+0000 7fd86fc78640 1 -- 172.21.15.42:0/2995383292 wait complete. 2024-09-16T03:42:30.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.453+0000 7fd86fc78640 10 monclient: build_initial_monmap 2024-09-16T03:42:30.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.453+0000 7fd86fc78640 10 monclient: monmap: 2024-09-16T03:42:30.464 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:42:30.464 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:42:30.464 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:42:30.464 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:42:30.464 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:42:30.464 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:30.464 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:42:30.464 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:42:30.464 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:42:30.464 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:30.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.453+0000 7fd86fc78640 1 librados: starting msgr at 2024-09-16T03:42:30.465 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.453+0000 7fd86fc78640 1 librados: starting objecter 2024-09-16T03:42:30.465 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.453+0000 7fd86fc78640 1 Processor -- start 2024-09-16T03:42:30.465 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.453+0000 7fd86fc78640 1 -- start start 2024-09-16T03:42:30.465 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.453+0000 7fd86fc78640 1 librados: setting wanted keys 2024-09-16T03:42:30.465 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.453+0000 7fd86fc78640 1 librados: calling monclient init 2024-09-16T03:42:30.465 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.453+0000 7fd86fc78640 10 monclient: init 2024-09-16T03:42:30.465 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-16T03:42:30.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.453+0000 7fd86fc78640 10 monclient: _reopen_session rank -1 2024-09-16T03:42:30.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.453+0000 7fd86fc78640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:42:30.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.453+0000 7fd86fc78640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd8680a67e0 0x7fd8681cf760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:30.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.453+0000 7fd86fc78640 10 monclient(hunting): picked mon.noname-c con 0x7fd8680a67e0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:42:30.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.453+0000 7fd86fc78640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd8680a6bc0 0x7fd86813b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:30.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.453+0000 7fd86fc78640 10 monclient(hunting): picked mon.noname-a con 0x7fd8680a6bc0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:42:30.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.453+0000 7fd86fc78640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd86813c2f0 0x7fd868120ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:30.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.453+0000 7fd86fc78640 10 monclient(hunting): picked mon.noname-b con 0x7fd86813c2f0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:42:30.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.453+0000 7fd86fc78640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:30.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.453+0000 7fd86fc78640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fd868143230 con 0x7fd8680a6bc0 2024-09-16T03:42:30.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.453+0000 7fd86fc78640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:30.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.453+0000 7fd86fc78640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd8681430b0 con 0x7fd86813c2f0 2024-09-16T03:42:30.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.453+0000 7fd86fc78640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:30.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.453+0000 7fd86fc78640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fd8681433b0 con 0x7fd8680a67e0 2024-09-16T03:42:30.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.453+0000 7fd86fc78640 10 monclient(hunting): _renew_subs 2024-09-16T03:42:30.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.453+0000 7fd86fc78640 10 monclient(hunting): authenticate will time out at 997.427136s 2024-09-16T03:42:30.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.453+0000 7fd86d213640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd8680a6bc0 0x7fd86813b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:30.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.453+0000 7fd86d213640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd8680a6bc0 0x7fd86813b4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:37748/0 (socket says 172.21.15.42:37748) 2024-09-16T03:42:30.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.453+0000 7fd86d213640 1 -- 172.21.15.42:0/1589220242 learned_addr learned my addr 172.21.15.42:0/1589220242 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:42:30.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.453+0000 7fd86da14640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd8680a67e0 0x7fd8681cf760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:30.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.453+0000 7fd86d213640 10 monclient(hunting): get_auth_request con 0x7fd8680a6bc0 auth_method 0 2024-09-16T03:42:30.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.453+0000 7fd86d213640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:30.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.453+0000 7fd86d213640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:30.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.453+0000 7fd86d213640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:30.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.453+0000 7fd86da14640 10 monclient(hunting): get_auth_request con 0x7fd8680a67e0 auth_method 0 2024-09-16T03:42:30.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.453+0000 7fd86da14640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:30.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.453+0000 7fd86da14640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:30.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.453+0000 7fd86da14640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:30.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.453+0000 7fd86d213640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:30.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.453+0000 7fd86d213640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:30.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.453+0000 7fd86d213640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:30.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.453+0000 7fd86da14640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:30.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.453+0000 7fd86da14640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:30.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.453+0000 7fd86da14640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:30.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.453+0000 7fd86d213640 10 monclient(hunting): handle_auth_done global_id 4320 payload 824 2024-09-16T03:42:30.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.453+0000 7fd86d213640 1 -- 172.21.15.42:0/1589220242 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd8680a67e0 msgr2=0x7fd8681cf760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:30.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.453+0000 7fd86d213640 1 --2- 172.21.15.42:0/1589220242 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd8680a67e0 0x7fd8681cf760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:30.475 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.453+0000 7fd86d213640 1 -- 172.21.15.42:0/1589220242 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd86813c2f0 msgr2=0x7fd868120ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:30.475 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.453+0000 7fd86d213640 1 --2- 172.21.15.42:0/1589220242 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd86813c2f0 0x7fd868120ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:30.475 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.453+0000 7fd86d213640 10 monclient: _finish_hunting 0 2024-09-16T03:42:30.475 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.453+0000 7fd86d213640 1 monclient: found mon.noname-a 2024-09-16T03:42:30.475 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.453+0000 7fd86d213640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:42:30.475 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.453+0000 7fd86d213640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:42:30.475 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.453+0000 7fd86d213640 1 -- 172.21.15.42:0/1589220242 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd868124b90 con 0x7fd8680a6bc0 2024-09-16T03:42:30.475 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.453+0000 7fd86da14640 1 --2- 172.21.15.42:0/1589220242 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd8680a67e0 0x7fd8681cf760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:42:30.475 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.457+0000 7fd86d213640 1 --2- 172.21.15.42:0/1589220242 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd8680a6bc0 0x7fd86813b4f0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fd85800c780 tx=0x7fd85800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:30.475 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.457+0000 7fd85e7fc640 10 client.?.objecter ms_handle_connect 0x7fd8680a6bc0 2024-09-16T03:42:30.475 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.457+0000 7fd85e7fc640 10 client.?.objecter resend_mon_ops 2024-09-16T03:42:30.476 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.457+0000 7fd85e7fc640 1 -- 172.21.15.42:0/1589220242 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd858018070 con 0x7fd8680a6bc0 2024-09-16T03:42:30.476 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.457+0000 7fd85e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:30.476 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.457+0000 7fd85e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:42:30.476 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.457+0000 7fd85e7fc640 10 monclient: dump: 2024-09-16T03:42:30.476 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:30.476 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:30.476 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:30.476 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:30.476 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:30.476 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:30.476 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:30.476 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:30.476 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:30.476 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:30.477 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.457+0000 7fd85e7fc640 10 monclient: _finish_auth 0 2024-09-16T03:42:30.477 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.457+0000 7fd85e7fc640 10 monclient: _check_auth_tickets 2024-09-16T03:42:30.477 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.457+0000 7fd85e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:42:30.477 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.457+0000 7fd85e7fc640 1 -- 172.21.15.42:0/1589220242 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd8580072d0 con 0x7fd8680a6bc0 2024-09-16T03:42:30.477 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.457+0000 7fd86fc78640 5 monclient: authenticate success, global_id 4320 2024-09-16T03:42:30.477 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.457+0000 7fd85e7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:42:30.477 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.457+0000 7fd85e7fc640 1 -- 172.21.15.42:0/1589220242 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8580047f0 con 0x7fd8680a6bc0 2024-09-16T03:42:30.477 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.457+0000 7fd85e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:30.477 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.457+0000 7fd85e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:42:30.477 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.457+0000 7fd85e7fc640 10 monclient: dump: 2024-09-16T03:42:30.477 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:30.477 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:30.477 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:30.478 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:30.478 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:30.478 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:30.478 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:30.478 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:30.478 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:30.478 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:30.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.457+0000 7fd85ffff640 4 set_mon_vals no callback set 2024-09-16T03:42:30.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.457+0000 7fd86fc78640 10 monclient: _renew_subs 2024-09-16T03:42:30.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.457+0000 7fd86fc78640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:42:30.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.457+0000 7fd86fc78640 1 -- 172.21.15.42:0/1589220242 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd86813c940 con 0x7fd8680a6bc0 2024-09-16T03:42:30.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.457+0000 7fd86fc78640 10 client.4320.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:42:30.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.457+0000 7fd86fc78640 10 monclient: _renew_subs 2024-09-16T03:42:30.479 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.457+0000 7fd86fc78640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:42:30.479 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.457+0000 7fd86fc78640 1 -- 172.21.15.42:0/1589220242 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd868123900 con 0x7fd8680a6bc0 2024-09-16T03:42:30.479 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.457+0000 7fd86fc78640 1 librados: init done 2024-09-16T03:42:30.479 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.457+0000 7fd86fc78640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:42:30.479 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.457+0000 7fd86fc78640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:42:30.479 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.457+0000 7fd86fc78640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:42:30.479 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.457+0000 7fd86fc78640 1 -- 172.21.15.42:0/1589220242 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd8680a0bc0 con 0x7fd8680a6bc0 2024-09-16T03:42:30.479 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.457+0000 7fd85e7fc640 1 -- 172.21.15.42:0/1589220242 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fd858004990 con 0x7fd8680a6bc0 2024-09-16T03:42:30.479 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.457+0000 7fd85e7fc640 1 --2- 172.21.15.42:0/1589220242 >> v2:172.21.15.154:6812/254845342 conn(0x7fd84c037810 0x7fd84c039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:30.479 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.457+0000 7fd85e7fc640 1 -- 172.21.15.42:0/1589220242 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 3562+0+0 (secure 0 0 0) 0x7fd858047b50 con 0x7fd8680a6bc0 2024-09-16T03:42:30.479 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.457+0000 7fd85e7fc640 10 client.4320.objecter ms_dispatch 0x7fd868000ba0 osd_map(18..18 src has 1..18) 2024-09-16T03:42:30.479 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.457+0000 7fd85e7fc640 3 client.4320.objecter handle_osd_map got epochs [18,18] > 0 2024-09-16T03:42:30.479 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.457+0000 7fd85e7fc640 3 client.4320.objecter handle_osd_map decoding full epoch 18 2024-09-16T03:42:30.479 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.457+0000 7fd85e7fc640 20 client.4320.objecter dump_active .. 0 homeless 2024-09-16T03:42:30.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.457+0000 7fd86da14640 1 --2- 172.21.15.42:0/1589220242 >> v2:172.21.15.154:6812/254845342 conn(0x7fd84c037810 0x7fd84c039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:30.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.457+0000 7fd86da14640 10 monclient: get_auth_request con 0x7fd84c037810 auth_method 0 2024-09-16T03:42:30.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.461+0000 7fd85e7fc640 10 client.4320.objecter ms_handle_connect 0x7fd84c037810 2024-09-16T03:42:30.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.461+0000 7fd86da14640 1 --2- 172.21.15.42:0/1589220242 >> v2:172.21.15.154:6812/254845342 conn(0x7fd84c037810 0x7fd84c039cd0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fd8640ad000 tx=0x7fd8640b2040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:30.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.465+0000 7fd85e7fc640 1 -- 172.21.15.42:0/1589220242 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7fd858010300 con 0x7fd8680a6bc0 2024-09-16T03:42:30.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.465+0000 7fd85e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:42:30.481 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.465+0000 7fd85e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:42:30.493 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-16T03:42:30.500 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.493+0000 7f9a0a895640 10 monclient: get_monmap_and_config 2024-09-16T03:42:30.500 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.493+0000 7f9a0a895640 10 monclient: build_initial_monmap 2024-09-16T03:42:30.500 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.493+0000 7f9a0a895640 10 monclient: monmap: 2024-09-16T03:42:30.500 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:42:30.501 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:30.501 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:42:30.497764+0000 2024-09-16T03:42:30.501 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:42:30.497764+0000 2024-09-16T03:42:30.501 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:42:30.501 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:30.501 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:42:30.501 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:42:30.501 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:42:30.501 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:42:30.501 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:42:30.501 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:30.502 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.493+0000 7f9a0a895640 1 Processor -- start 2024-09-16T03:42:30.502 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.493+0000 7f9a0a895640 1 -- start start 2024-09-16T03:42:30.502 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.493+0000 7f9a0a895640 10 monclient: init 2024-09-16T03:42:30.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.493+0000 7f9a0a895640 10 monclient: _reopen_session rank -1 2024-09-16T03:42:30.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.493+0000 7f9a0a895640 10 monclient: _add_conns ranks=[1,2,0,3,4] 2024-09-16T03:42:30.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.493+0000 7f9a0a895640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9a040a39b0 addr v1:172.21.15.154:6789/0 2024-09-16T03:42:30.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.493+0000 7f9a0a895640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9a040a3d90 0x7f9a0413c380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:30.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.493+0000 7f9a0a895640 10 monclient(hunting): picked mon.noname-b con 0x7f9a040a3d90 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:42:30.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.493+0000 7f9a0a895640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9a0413c8c0 addr v1:172.21.15.42:6789/0 2024-09-16T03:42:30.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.493+0000 7f9a0a895640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9a04134290 con 0x7f9a0413c8c0 2024-09-16T03:42:30.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.493+0000 7f9a0a895640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9a0409c140 con 0x7f9a040a39b0 2024-09-16T03:42:30.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.493+0000 7f9a0a895640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:30.505 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.493+0000 7f9a0a895640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f9a0409c420 con 0x7f9a040a3d90 2024-09-16T03:42:30.505 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.493+0000 7f9a0a895640 10 monclient(hunting): _renew_subs 2024-09-16T03:42:30.505 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.493+0000 7f9a0a895640 10 monclient(hunting): authenticate will time out at 997.467007s 2024-09-16T03:42:30.505 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.493+0000 7f9a08e32640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f9a0413c8c0 0x7f9a0413ecb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:46808/0 (socket says 172.21.15.42:46808) 2024-09-16T03:42:30.505 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.493+0000 7f9a08e32640 1 -- 172.21.15.42:0/3136102632 learned_addr learned my addr 172.21.15.42:0/3136102632 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:42:30.505 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.493+0000 7f9a02ffd640 1 -- 172.21.15.42:0/3136102632 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2923012485 0 0) 0x7f9a04134290 con 0x7f9a0413c8c0 2024-09-16T03:42:30.505 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.493+0000 7f9a02ffd640 10 monclient(hunting): _negotiate 2024-09-16T03:42:30.505 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.493+0000 7f9a02ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:30.505 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.493+0000 7f9a02ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:30.505 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.493+0000 7f9a02ffd640 10 monclient(hunting): my global_id is 4323 2024-09-16T03:42:30.505 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.493+0000 7f9a02ffd640 1 -- 172.21.15.42:0/3136102632 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f99ec003530 con 0x7f9a0413c8c0 2024-09-16T03:42:30.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.493+0000 7f9a037fe640 1 --2- 172.21.15.42:0/3136102632 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9a040a3d90 0x7f9a0413c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:30.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.493+0000 7f9a037fe640 10 monclient(hunting): get_auth_request con 0x7f9a040a3d90 auth_method 0 2024-09-16T03:42:30.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.493+0000 7f9a037fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:30.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.493+0000 7f9a037fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:30.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.493+0000 7f9a037fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:30.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.493+0000 7f9a037fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:30.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.493+0000 7f9a037fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:30.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.493+0000 7f9a037fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:30.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.493+0000 7f9a02ffd640 1 -- 172.21.15.42:0/3136102632 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4112804068 0 0) 0x7f99ec003530 con 0x7f9a0413c8c0 2024-09-16T03:42:30.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.493+0000 7f9a02ffd640 1 -- 172.21.15.42:0/3136102632 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9a040a3d90 msgr2=0x7f9a0413c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:42:30.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.493+0000 7f9a02ffd640 1 --2- 172.21.15.42:0/3136102632 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9a040a3d90 0x7f9a0413c380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:30.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.493+0000 7f9a02ffd640 1 -- 172.21.15.42:0/3136102632 >> v1:172.21.15.154:6789/0 conn(0x7f9a040a39b0 legacy=0x7f9a04133b80 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-16T03:42:30.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.493+0000 7f9a02ffd640 10 monclient: _finish_hunting 0 2024-09-16T03:42:30.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.493+0000 7f9a02ffd640 1 monclient: found mon.noname-a-legacy 2024-09-16T03:42:30.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.493+0000 7f9a02ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:42:30.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.493+0000 7f9a02ffd640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.42:6789/0 2024-09-16T03:42:30.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.493+0000 7f9a02ffd640 1 -- 172.21.15.42:0/3136102632 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a0406dd60 con 0x7f9a0413c8c0 2024-09-16T03:42:30.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.493+0000 7f9a02ffd640 10 monclient: _finish_auth 0 2024-09-16T03:42:30.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.493+0000 7f9a02ffd640 10 monclient: _check_auth_tickets 2024-09-16T03:42:30.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.493+0000 7f9a02ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:42:30.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.493+0000 7f9a02ffd640 1 -- 172.21.15.42:0/3136102632 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2177413734 0 0) 0x7f99fc001a00 con 0x7f9a0413c8c0 2024-09-16T03:42:30.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.497+0000 7f9a02ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:30.508 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.497+0000 7f9a02ffd640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T03:42:30.508 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.497+0000 7f9a02ffd640 10 monclient: dump: 2024-09-16T03:42:30.508 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:30.508 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:30.508 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:30.508 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:30.508 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:30.509 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:30.509 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:30.509 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:30.509 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:30.509 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:30.509 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.497+0000 7f9a02ffd640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] but i'm connected to v1:172.21.15.42:6789/0, reconnecting 2024-09-16T03:42:30.509 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.497+0000 7f9a02ffd640 10 monclient: _reopen_session rank -1 2024-09-16T03:42:30.509 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.497+0000 7f9a02ffd640 1 -- 172.21.15.42:0/3136102632 >> v1:172.21.15.42:6789/0 conn(0x7f9a0413c8c0 legacy=0x7f9a0413ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:30.509 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.497+0000 7f9a02ffd640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T03:42:30.509 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.497+0000 7f9a02ffd640 1 --2- 172.21.15.42:0/3136102632 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f99ec0039f0 0x7f99ec003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:30.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.497+0000 7f9a02ffd640 10 monclient(hunting): picked mon.a con 0x7f99ec0039f0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:42:30.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.497+0000 7f9a02ffd640 1 --2- 172.21.15.42:0/3136102632 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f99ec004810 0x7f99ec004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:30.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.497+0000 7f9a02ffd640 10 monclient(hunting): picked mon.b con 0x7f99ec004810 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:42:30.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.497+0000 7f9a02ffd640 1 --2- 172.21.15.42:0/3136102632 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9a040a3d90 0x7f99ec00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:30.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.497+0000 7f9a02ffd640 10 monclient(hunting): picked mon.c con 0x7f9a040a3d90 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:42:30.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.497+0000 7f9a02ffd640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:30.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.497+0000 7f9a02ffd640 1 -- 172.21.15.42:0/3136102632 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f9a0409c420 con 0x7f99ec0039f0 2024-09-16T03:42:30.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.497+0000 7f9a02ffd640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:30.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.497+0000 7f9a02ffd640 1 -- 172.21.15.42:0/3136102632 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f99ec005560 con 0x7f99ec004810 2024-09-16T03:42:30.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.497+0000 7f9a02ffd640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:30.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.497+0000 7f9a02ffd640 1 -- 172.21.15.42:0/3136102632 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f99ec00bf10 con 0x7f9a040a3d90 2024-09-16T03:42:30.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.497+0000 7f9a02ffd640 10 monclient(hunting): _renew_subs 2024-09-16T03:42:30.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.497+0000 7f9a02ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:42:30.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.497+0000 7f9a08e32640 1 --2- 172.21.15.42:0/3136102632 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9a040a3d90 0x7f99ec00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:30.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.497+0000 7f9a03fff640 1 --2- 172.21.15.42:0/3136102632 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f99ec0039f0 0x7f99ec003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:30.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.497+0000 7f9a08e32640 10 monclient(hunting): get_auth_request con 0x7f9a040a3d90 auth_method 0 2024-09-16T03:42:30.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.497+0000 7f9a08e32640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:30.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.497+0000 7f9a08e32640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:30.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.497+0000 7f9a08e32640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:42:30.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.497+0000 7f9a03fff640 10 monclient(hunting): get_auth_request con 0x7f99ec0039f0 auth_method 0 2024-09-16T03:42:30.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.497+0000 7f9a03fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:30.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.497+0000 7f9a03fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:30.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.497+0000 7f9a03fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:42:30.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.497+0000 7f9a08e32640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:30.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.497+0000 7f9a08e32640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:30.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.497+0000 7f9a08e32640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T03:42:30.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.497+0000 7f9a03fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:30.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.497+0000 7f9a03fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:30.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.497+0000 7f9a03fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T03:42:30.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.497+0000 7f9a037fe640 1 --2- 172.21.15.42:0/3136102632 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f99ec004810 0x7f99ec004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:30.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.497+0000 7f9a08e32640 10 monclient(hunting): handle_auth_done global_id 4323 payload 293 2024-09-16T03:42:30.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.497+0000 7f9a08e32640 1 -- 172.21.15.42:0/3136102632 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f99ec004810 msgr2=0x7f99ec004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:42:30.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.497+0000 7f9a08e32640 1 --2- 172.21.15.42:0/3136102632 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f99ec004810 0x7f99ec004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:30.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.497+0000 7f9a08e32640 1 -- 172.21.15.42:0/3136102632 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f99ec0039f0 msgr2=0x7f99ec003df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:42:30.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.497+0000 7f9a08e32640 1 --2- 172.21.15.42:0/3136102632 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f99ec0039f0 0x7f99ec003df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:30.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.497+0000 7f9a08e32640 10 monclient: _finish_hunting 0 2024-09-16T03:42:30.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.497+0000 7f9a08e32640 1 monclient: found mon.c 2024-09-16T03:42:30.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.497+0000 7f9a08e32640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:42:30.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.497+0000 7f9a08e32640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:42:30.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.497+0000 7f9a08e32640 1 -- 172.21.15.42:0/3136102632 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99ec003530 con 0x7f9a040a3d90 2024-09-16T03:42:30.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.497+0000 7f9a08e32640 10 monclient: _finish_auth 0 2024-09-16T03:42:30.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.497+0000 7f9a08e32640 10 monclient: _check_auth_tickets 2024-09-16T03:42:30.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.497+0000 7f9a08e32640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:42:30.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.497+0000 7f9a037fe640 10 monclient: get_auth_request con 0x7f99ec004810 auth_method 0 2024-09-16T03:42:30.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.497+0000 7f9a037fe640 1 --2- 172.21.15.42:0/3136102632 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f99ec004810 0x7f99ec004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T03:42:30.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.497+0000 7f9a0a895640 5 monclient: authenticate success, global_id 4323 2024-09-16T03:42:30.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.497+0000 7f9a0a895640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:42:30.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.501+0000 7f9a08e32640 1 --2- 172.21.15.42:0/3136102632 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9a040a3d90 0x7f99ec00b9d0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f9a040a56e0 tx=0x7f99fc01d2a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a3fdf8c95dbb7a81 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:30.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.501+0000 7f9a02ffd640 1 -- 172.21.15.42:0/3136102632 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99fc00a3f0 con 0x7f9a040a3d90 2024-09-16T03:42:30.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.501+0000 7f9a02ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:30.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.501+0000 7f9a02ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:42:30.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.501+0000 7f9a02ffd640 10 monclient: dump: 2024-09-16T03:42:30.520 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:30.520 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:30.520 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:30.520 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:30.521 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:30.521 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:30.521 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:30.521 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:30.521 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:30.521 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:30.521 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.501+0000 7f9a02ffd640 1 -- 172.21.15.42:0/3136102632 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f99fc0ad650 con 0x7f9a040a3d90 2024-09-16T03:42:30.521 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.501+0000 7f9a0a895640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:42:30.521 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.501+0000 7f9a02ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:42:30.521 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.501+0000 7f9a02ffd640 1 -- 172.21.15.42:0/3136102632 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99fc0ada40 con 0x7f9a040a3d90 2024-09-16T03:42:30.522 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.501+0000 7f9a02ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:30.522 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.501+0000 7f9a02ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:42:30.522 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.501+0000 7f9a02ffd640 10 monclient: dump: 2024-09-16T03:42:30.522 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:30.522 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:30.522 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:30.522 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:30.522 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:30.522 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:30.522 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:30.522 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:30.523 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:30.523 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:30.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.501+0000 7f9a0a895640 10 monclient: get_monmap_and_config success 2024-09-16T03:42:30.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.501+0000 7f9a0a895640 4 set_mon_vals no callback set 2024-09-16T03:42:30.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.501+0000 7f9a0a895640 10 monclient: shutdown 2024-09-16T03:42:30.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.501+0000 7f9a0a895640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:42:30.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.501+0000 7f9a0a895640 1 -- 172.21.15.42:0/3136102632 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9a040a3d90 msgr2=0x7f99ec00b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:30.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.501+0000 7f9a0a895640 1 --2- 172.21.15.42:0/3136102632 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9a040a3d90 0x7f99ec00b9d0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f9a040a56e0 tx=0x7f99fc01d2a0 comp rx=0 tx=0).stop 2024-09-16T03:42:30.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.501+0000 7f9a08e32640 1 -- 172.21.15.42:0/3136102632 reap_dead start 2024-09-16T03:42:30.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.501+0000 7f9a0a895640 1 -- 172.21.15.42:0/3136102632 shutdown_connections 2024-09-16T03:42:30.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.501+0000 7f9a0a895640 1 -- 172.21.15.42:0/3136102632 >> 172.21.15.42:0/3136102632 conn(0x7f9a040a9fb0 msgr2=0x7f9a040aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:42:30.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.501+0000 7f9a0a895640 1 -- 172.21.15.42:0/3136102632 shutdown_connections 2024-09-16T03:42:30.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.501+0000 7f9a0a895640 1 -- 172.21.15.42:0/3136102632 wait complete. 2024-09-16T03:42:30.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.501+0000 7f9a0a895640 10 monclient: build_initial_monmap 2024-09-16T03:42:30.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.501+0000 7f9a0a895640 10 monclient: monmap: 2024-09-16T03:42:30.524 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:42:30.524 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:42:30.524 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:42:30.525 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:42:30.525 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:42:30.525 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:30.525 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:42:30.525 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:42:30.525 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:42:30.525 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:30.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.501+0000 7f9a0a895640 1 librados: starting msgr at 2024-09-16T03:42:30.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.501+0000 7f9a0a895640 1 librados: starting objecter 2024-09-16T03:42:30.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.501+0000 7f9a0a895640 1 Processor -- start 2024-09-16T03:42:30.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.501+0000 7f9a0a895640 1 -- start start 2024-09-16T03:42:30.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.501+0000 7f9a0a895640 1 librados: setting wanted keys 2024-09-16T03:42:30.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.501+0000 7f9a0a895640 1 librados: calling monclient init 2024-09-16T03:42:30.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.501+0000 7f9a0a895640 10 monclient: init 2024-09-16T03:42:30.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.501+0000 7f9a0a895640 10 monclient: _reopen_session rank -1 2024-09-16T03:42:30.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.505+0000 7f9a0a895640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:42:30.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.505+0000 7f9a0a895640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9a040a3d90 0x7f9a0409c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:30.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.505+0000 7f9a0a895640 10 monclient(hunting): picked mon.noname-c con 0x7f9a040a3d90 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:42:30.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.505+0000 7f9a0a895640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9a0409ce80 0x7f9a0409d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:30.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.505+0000 7f9a0a895640 10 monclient(hunting): picked mon.noname-b con 0x7f9a0409ce80 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:42:30.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.505+0000 7f9a0a895640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9a04130390 0x7f9a04130820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:30.527 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.505+0000 7f9a03fff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9a040a3d90 0x7f9a0409c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:30.527 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.505+0000 7f9a03fff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9a040a3d90 0x7f9a0409c940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:41466/0 (socket says 172.21.15.42:41466) 2024-09-16T03:42:30.527 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.505+0000 7f9a03fff640 1 -- 172.21.15.42:0/1918499733 learned_addr learned my addr 172.21.15.42:0/1918499733 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:42:30.527 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.505+0000 7f9a08e32640 1 --2- 172.21.15.42:0/1918499733 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9a04130390 0x7f9a04130820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:30.527 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.505+0000 7f9a037fe640 1 --2- 172.21.15.42:0/1918499733 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9a0409ce80 0x7f9a0409d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:30.527 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.505+0000 7f9a0a895640 10 monclient(hunting): picked mon.noname-a con 0x7f9a04130390 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:42:30.527 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.505+0000 7f9a0a895640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:30.527 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.505+0000 7f9a0a895640 1 -- 172.21.15.42:0/1918499733 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f9a04140f70 con 0x7f9a04130390 2024-09-16T03:42:30.527 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.505+0000 7f9a0a895640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:30.527 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.505+0000 7f9a0a895640 1 -- 172.21.15.42:0/1918499733 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f9a04140df0 con 0x7f9a0409ce80 2024-09-16T03:42:30.527 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.505+0000 7f9a0a895640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:30.528 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.505+0000 7f9a0a895640 1 -- 172.21.15.42:0/1918499733 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f9a041410f0 con 0x7f9a040a3d90 2024-09-16T03:42:30.528 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.505+0000 7f9a0a895640 10 monclient(hunting): _renew_subs 2024-09-16T03:42:30.528 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.505+0000 7f9a0a895640 10 monclient(hunting): authenticate will time out at 997.476648s 2024-09-16T03:42:30.528 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.505+0000 7f9a03fff640 10 monclient(hunting): get_auth_request con 0x7f9a040a3d90 auth_method 0 2024-09-16T03:42:30.528 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.505+0000 7f9a03fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:30.529 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.505+0000 7f9a03fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:30.529 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.505+0000 7f9a03fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:30.529 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.505+0000 7f9a08e32640 10 monclient(hunting): get_auth_request con 0x7f9a04130390 auth_method 0 2024-09-16T03:42:30.529 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.505+0000 7f9a08e32640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:30.529 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.505+0000 7f9a08e32640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:30.529 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.505+0000 7f9a08e32640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:30.529 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.505+0000 7f9a03fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:30.530 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.505+0000 7f9a03fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:30.530 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.505+0000 7f9a03fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:30.530 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.505+0000 7f9a037fe640 10 monclient(hunting): get_auth_request con 0x7f9a0409ce80 auth_method 0 2024-09-16T03:42:30.530 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.505+0000 7f9a037fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:30.530 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.505+0000 7f9a037fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:30.530 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.505+0000 7f9a037fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:30.530 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.505+0000 7f9a08e32640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:30.530 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.505+0000 7f9a08e32640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:30.531 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.505+0000 7f9a08e32640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:30.531 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.505+0000 7f9a03fff640 10 monclient(hunting): handle_auth_done global_id 4280 payload 824 2024-09-16T03:42:30.531 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.505+0000 7f9a03fff640 1 -- 172.21.15.42:0/1918499733 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9a0409ce80 msgr2=0x7f9a0409d2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:30.531 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.505+0000 7f9a03fff640 1 --2- 172.21.15.42:0/1918499733 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9a0409ce80 0x7f9a0409d2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:30.531 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.505+0000 7f9a03fff640 1 -- 172.21.15.42:0/1918499733 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9a04130390 msgr2=0x7f9a04130820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:30.531 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.505+0000 7f9a03fff640 1 --2- 172.21.15.42:0/1918499733 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9a04130390 0x7f9a04130820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:30.531 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.505+0000 7f9a03fff640 10 monclient: _finish_hunting 0 2024-09-16T03:42:30.531 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.505+0000 7f9a03fff640 1 monclient: found mon.noname-c 2024-09-16T03:42:30.531 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.505+0000 7f9a03fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:42:30.531 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.505+0000 7f9a03fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:42:30.531 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.505+0000 7f9a03fff640 1 -- 172.21.15.42:0/1918499733 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a0409f260 con 0x7f9a040a3d90 2024-09-16T03:42:30.531 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.505+0000 7f9a03fff640 1 --2- 172.21.15.42:0/1918499733 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9a040a3d90 0x7f9a0409c940 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f99f400f640 tx=0x7f99f400fb00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:30.532 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.505+0000 7f9a00ff9640 10 client.?.objecter ms_handle_connect 0x7f9a040a3d90 2024-09-16T03:42:30.532 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.505+0000 7f9a00ff9640 10 client.?.objecter resend_mon_ops 2024-09-16T03:42:30.532 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.505+0000 7f9a00ff9640 1 -- 172.21.15.42:0/1918499733 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99f40052e0 con 0x7f9a040a3d90 2024-09-16T03:42:30.532 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.505+0000 7f9a00ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:30.532 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.505+0000 7f9a00ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:42:30.532 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.505+0000 7f9a00ff9640 10 monclient: dump: 2024-09-16T03:42:30.532 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:30.532 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:30.532 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:30.532 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:30.532 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:30.532 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:30.533 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:30.533 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:30.533 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:30.533 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:30.533 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.505+0000 7f9a00ff9640 10 monclient: _finish_auth 0 2024-09-16T03:42:30.533 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.505+0000 7f9a00ff9640 10 monclient: _check_auth_tickets 2024-09-16T03:42:30.533 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.505+0000 7f9a00ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:42:30.533 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.505+0000 7f9a0a895640 5 monclient: authenticate success, global_id 4280 2024-09-16T03:42:30.533 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.505+0000 7f9a00ff9640 1 -- 172.21.15.42:0/1918499733 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f99f4018070 con 0x7f9a040a3d90 2024-09-16T03:42:30.533 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.505+0000 7f9a0a895640 10 monclient: _renew_subs 2024-09-16T03:42:30.533 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.505+0000 7f9a0a895640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:42:30.534 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.505+0000 7f9a0a895640 1 -- 172.21.15.42:0/1918499733 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a0413fcd0 con 0x7f9a040a3d90 2024-09-16T03:42:30.534 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.505+0000 7f9a00ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T03:42:30.534 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.505+0000 7f9a00ff9640 1 -- 172.21.15.42:0/1918499733 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99f4003840 con 0x7f9a040a3d90 2024-09-16T03:42:30.534 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.505+0000 7f9a00ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:30.534 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.505+0000 7f9a00ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:42:30.534 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.505+0000 7f9a00ff9640 10 monclient: dump: 2024-09-16T03:42:30.534 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:30.534 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:30.534 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:30.534 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:30.534 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:30.534 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:30.535 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:30.535 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:30.535 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:30.535 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:30.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.505+0000 7f9a027fc640 4 set_mon_vals no callback set 2024-09-16T03:42:30.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.505+0000 7f9a00ff9640 1 -- 172.21.15.42:0/1918499733 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f99f40039e0 con 0x7f9a040a3d90 2024-09-16T03:42:30.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.505+0000 7f9a0a895640 10 client.4280.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:42:30.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.505+0000 7f9a0a895640 10 monclient: _renew_subs 2024-09-16T03:42:30.536 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.505+0000 7f9a0a895640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:42:30.536 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.505+0000 7f9a0a895640 1 -- 172.21.15.42:0/1918499733 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a0409ef80 con 0x7f9a040a3d90 2024-09-16T03:42:30.536 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.505+0000 7f9a0a895640 1 librados: init done 2024-09-16T03:42:30.536 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.505+0000 7f9a00ff9640 1 --2- 172.21.15.42:0/1918499733 >> v2:172.21.15.154:6812/254845342 conn(0x7f99f0037810 0x7f99f0039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:30.536 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.505+0000 7f9a00ff9640 1 -- 172.21.15.42:0/1918499733 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(18..18 src has 1..18) ==== 3562+0+0 (secure 0 0 0) 0x7f99f404d060 con 0x7f9a040a3d90 2024-09-16T03:42:30.536 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.505+0000 7f9a00ff9640 10 client.4280.objecter ms_dispatch 0x7f9a04000ba0 osd_map(18..18 src has 1..18) 2024-09-16T03:42:30.536 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.505+0000 7f9a00ff9640 3 client.4280.objecter handle_osd_map got epochs [18,18] > 0 2024-09-16T03:42:30.537 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.505+0000 7f9a00ff9640 3 client.4280.objecter handle_osd_map decoding full epoch 18 2024-09-16T03:42:30.537 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.505+0000 7f9a00ff9640 20 client.4280.objecter dump_active .. 0 homeless 2024-09-16T03:42:30.537 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.505+0000 7f9a037fe640 1 --2- 172.21.15.42:0/1918499733 >> v2:172.21.15.154:6812/254845342 conn(0x7f99f0037810 0x7f99f0039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:30.537 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.505+0000 7f9a037fe640 10 monclient: get_auth_request con 0x7f99f0037810 auth_method 0 2024-09-16T03:42:30.537 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.509+0000 7f9a0a895640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:42:30.537 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.509+0000 7f9a0a895640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:42:30.537 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.509+0000 7f9a0a895640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:42:30.537 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.509+0000 7f9a0a895640 1 -- 172.21.15.42:0/1918499733 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a040a5750 con 0x7f9a040a3d90 2024-09-16T03:42:30.537 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.509+0000 7f9a037fe640 1 --2- 172.21.15.42:0/1918499733 >> v2:172.21.15.154:6812/254845342 conn(0x7f99f0037810 0x7f99f0039cd0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f9a041370e0 tx=0x7f99f8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:30.537 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.509+0000 7f9a00ff9640 10 client.4280.objecter ms_handle_connect 0x7f99f0037810 2024-09-16T03:42:30.538 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.517+0000 7f9a00ff9640 1 -- 172.21.15.42:0/1918499733 <== mon.2 v2:172.21.15.42:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f99f4013c70 con 0x7f9a040a3d90 2024-09-16T03:42:30.538 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.517+0000 7f9a00ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:42:30.538 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.517+0000 7f9a00ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:42:30.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.553+0000 7f13eabf4640 10 monclient: get_monmap_and_config 2024-09-16T03:42:30.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.553+0000 7f13eabf4640 10 monclient: build_initial_monmap 2024-09-16T03:42:30.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.553+0000 7f13eabf4640 10 monclient: monmap: 2024-09-16T03:42:30.559 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:42:30.559 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:30.559 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:42:30.556381+0000 2024-09-16T03:42:30.559 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:42:30.556381+0000 2024-09-16T03:42:30.559 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:42:30.559 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:30.559 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:42:30.560 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:42:30.560 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:42:30.560 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:42:30.560 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:42:30.560 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:30.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.553+0000 7f13eabf4640 1 Processor -- start 2024-09-16T03:42:30.562 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.553+0000 7f13eabf4640 1 -- start start 2024-09-16T03:42:30.562 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.553+0000 7f13eabf4640 10 monclient: init 2024-09-16T03:42:30.562 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.553+0000 7f13eabf4640 10 monclient: _reopen_session rank -1 2024-09-16T03:42:30.562 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.553+0000 7f13eabf4640 10 monclient: _add_conns ranks=[2,1,0,3,4] 2024-09-16T03:42:30.562 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.553+0000 7f13eabf4640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f13e40a39b0 0x7f13e4133b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:30.562 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.553+0000 7f13eabf4640 10 monclient(hunting): picked mon.noname-b con 0x7f13e40a39b0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:42:30.562 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.553+0000 7f13eabf4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f13e40a3d90 addr v1:172.21.15.154:6789/0 2024-09-16T03:42:30.562 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.553+0000 7f13eabf4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f13e4137710 addr v1:172.21.15.42:6789/0 2024-09-16T03:42:30.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.553+0000 7f13eabf4640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f13e409c140 con 0x7f13e4137710 2024-09-16T03:42:30.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.553+0000 7f13e9191640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f13e4137710 0x7f13e413f3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:46816/0 (socket says 172.21.15.42:46816) 2024-09-16T03:42:30.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.553+0000 7f13e9191640 1 -- 172.21.15.42:0/2057823069 learned_addr learned my addr 172.21.15.42:0/2057823069 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:42:30.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.553+0000 7f13e8990640 1 --2- 172.21.15.42:0/2057823069 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f13e40a39b0 0x7f13e4133b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:30.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.553+0000 7f13eabf4640 1 -- 172.21.15.42:0/2057823069 --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f13e409c420 con 0x7f13e40a3d90 2024-09-16T03:42:30.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.553+0000 7f13eabf4640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:30.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.553+0000 7f13eabf4640 1 -- 172.21.15.42:0/2057823069 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f13e406dd60 con 0x7f13e40a39b0 2024-09-16T03:42:30.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.553+0000 7f13eabf4640 10 monclient(hunting): _renew_subs 2024-09-16T03:42:30.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.553+0000 7f13eabf4640 10 monclient(hunting): authenticate will time out at 997.526015s 2024-09-16T03:42:30.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.553+0000 7f13e8990640 10 monclient(hunting): get_auth_request con 0x7f13e40a39b0 auth_method 0 2024-09-16T03:42:30.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.553+0000 7f13e8990640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:30.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.553+0000 7f13e8990640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:30.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.553+0000 7f13e8990640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:30.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.553+0000 7f13e8990640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:30.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.553+0000 7f13e8990640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:30.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.553+0000 7f13e8990640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:30.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.553+0000 7f13e8990640 10 monclient(hunting): handle_auth_done global_id 4283 payload 274 2024-09-16T03:42:30.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.553+0000 7f13e8990640 1 -- 172.21.15.42:0/2057823069 >> v1:172.21.15.154:6789/0 conn(0x7f13e40a3d90 legacy=0x7f13e4137000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:30.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.553+0000 7f13e8990640 1 -- 172.21.15.42:0/2057823069 >> v1:172.21.15.42:6789/0 conn(0x7f13e4137710 legacy=0x7f13e413f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:30.565 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.553+0000 7f13e8990640 10 monclient: _finish_hunting 0 2024-09-16T03:42:30.565 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.553+0000 7f13e8990640 1 monclient: found mon.noname-b 2024-09-16T03:42:30.565 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.553+0000 7f13e8990640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:42:30.565 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.553+0000 7f13e8990640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T03:42:30.565 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.553+0000 7f13e8990640 1 -- 172.21.15.42:0/2057823069 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13e40a5e20 con 0x7f13e40a39b0 2024-09-16T03:42:30.569 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.557+0000 7f13e8990640 1 --2- 172.21.15.42:0/2057823069 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f13e40a39b0 0x7f13e4133b80 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f13d80097f0 tx=0x7f13d801bee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=21618b39907150a9 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:30.569 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.557+0000 7f13e37fe640 1 -- 172.21.15.42:0/2057823069 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13d80aa640 con 0x7f13e40a39b0 2024-09-16T03:42:30.569 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.557+0000 7f13e37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:30.569 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.557+0000 7f13e37fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:42:30.570 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.557+0000 7f13e37fe640 10 monclient: dump: 2024-09-16T03:42:30.570 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:30.570 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:30.570 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:30.570 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:30.570 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:30.570 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:30.570 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:30.570 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:30.570 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:30.570 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:30.570 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.557+0000 7f13e37fe640 10 monclient: _finish_auth 0 2024-09-16T03:42:30.570 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.557+0000 7f13e37fe640 10 monclient: _check_auth_tickets 2024-09-16T03:42:30.570 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.557+0000 7f13e37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:42:30.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.557+0000 7f13e37fe640 1 -- 172.21.15.42:0/2057823069 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f13d80af070 con 0x7f13e40a39b0 2024-09-16T03:42:30.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.557+0000 7f13e37fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:42:30.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.557+0000 7f13e37fe640 1 -- 172.21.15.42:0/2057823069 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13d80aaa00 con 0x7f13e40a39b0 2024-09-16T03:42:30.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.557+0000 7f13e37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:30.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.557+0000 7f13e37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:42:30.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.557+0000 7f13e37fe640 10 monclient: dump: 2024-09-16T03:42:30.571 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:30.571 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:30.571 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:30.571 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:30.571 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:30.571 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:30.571 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:30.571 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:30.572 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:30.572 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:30.572 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.557+0000 7f13eabf4640 5 monclient: authenticate success, global_id 4283 2024-09-16T03:42:30.572 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.557+0000 7f13eabf4640 10 monclient: get_monmap_and_config success 2024-09-16T03:42:30.572 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.557+0000 7f13eabf4640 4 set_mon_vals no callback set 2024-09-16T03:42:30.572 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.557+0000 7f13eabf4640 10 monclient: shutdown 2024-09-16T03:42:30.572 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.557+0000 7f13eabf4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:42:30.572 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.557+0000 7f13eabf4640 1 -- 172.21.15.42:0/2057823069 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f13e40a39b0 msgr2=0x7f13e4133b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:30.572 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.557+0000 7f13eabf4640 1 --2- 172.21.15.42:0/2057823069 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f13e40a39b0 0x7f13e4133b80 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f13d80097f0 tx=0x7f13d801bee0 comp rx=0 tx=0).stop 2024-09-16T03:42:30.572 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.557+0000 7f13eabf4640 1 -- 172.21.15.42:0/2057823069 shutdown_connections 2024-09-16T03:42:30.572 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.557+0000 7f13eabf4640 1 --2- 172.21.15.42:0/2057823069 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f13e40a39b0 0x7f13e4133b80 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:30.572 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.557+0000 7f13eabf4640 1 -- 172.21.15.42:0/2057823069 >> 172.21.15.42:0/2057823069 conn(0x7f13e40a9fb0 msgr2=0x7f13e40aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:42:30.572 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.557+0000 7f13eabf4640 1 -- 172.21.15.42:0/2057823069 shutdown_connections 2024-09-16T03:42:30.572 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.557+0000 7f13eabf4640 1 -- 172.21.15.42:0/2057823069 wait complete. 2024-09-16T03:42:30.573 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.561+0000 7f13eabf4640 10 monclient: build_initial_monmap 2024-09-16T03:42:30.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.561+0000 7f13eabf4640 10 monclient: monmap: 2024-09-16T03:42:30.574 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:42:30.574 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:42:30.574 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:42:30.574 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:42:30.574 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:42:30.574 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:30.574 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:42:30.574 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:42:30.574 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:42:30.574 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:30.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.561+0000 7f13eabf4640 1 librados: starting msgr at 2024-09-16T03:42:30.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.561+0000 7f13eabf4640 1 librados: starting objecter 2024-09-16T03:42:30.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.561+0000 7f13eabf4640 1 Processor -- start 2024-09-16T03:42:30.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.561+0000 7f13eabf4640 1 -- start start 2024-09-16T03:42:30.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.561+0000 7f13eabf4640 1 librados: setting wanted keys 2024-09-16T03:42:30.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.561+0000 7f13eabf4640 1 librados: calling monclient init 2024-09-16T03:42:30.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.561+0000 7f13eabf4640 10 monclient: init 2024-09-16T03:42:30.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.561+0000 7f13eabf4640 10 monclient: _reopen_session rank -1 2024-09-16T03:42:30.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.561+0000 7f13eabf4640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:42:30.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.561+0000 7f13eabf4640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f13e40a39b0 0x7f13e40a19d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:30.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.561+0000 7f13eabf4640 10 monclient(hunting): picked mon.noname-c con 0x7f13e40a39b0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:42:30.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.561+0000 7f13eabf4640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f13e40a3d90 0x7f13e40a1f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:30.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.561+0000 7f13eabf4640 10 monclient(hunting): picked mon.noname-b con 0x7f13e40a3d90 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:42:30.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.561+0000 7f13eabf4640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f13e4137710 0x7f13e40a2450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:30.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.561+0000 7f13e8990640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f13e40a39b0 0x7f13e40a19d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:30.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.561+0000 7f13e8990640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f13e40a39b0 0x7f13e40a19d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:41488/0 (socket says 172.21.15.42:41488) 2024-09-16T03:42:30.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.561+0000 7f13eabf4640 10 monclient(hunting): picked mon.noname-a con 0x7f13e4137710 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:42:30.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.561+0000 7f13eabf4640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:30.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.561+0000 7f13e8990640 1 -- 172.21.15.42:0/3303871727 learned_addr learned my addr 172.21.15.42:0/3303871727 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:42:30.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.561+0000 7f13eabf4640 1 -- 172.21.15.42:0/3303871727 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f13e406dee0 con 0x7f13e4137710 2024-09-16T03:42:30.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.561+0000 7f13eabf4640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:30.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.561+0000 7f13eabf4640 1 -- 172.21.15.42:0/3303871727 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f13e4143900 con 0x7f13e40a3d90 2024-09-16T03:42:30.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.561+0000 7f13eabf4640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:30.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.561+0000 7f13eabf4640 1 -- 172.21.15.42:0/3303871727 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f13e4143a80 con 0x7f13e40a39b0 2024-09-16T03:42:30.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.561+0000 7f13eabf4640 10 monclient(hunting): _renew_subs 2024-09-16T03:42:30.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.561+0000 7f13eabf4640 10 monclient(hunting): authenticate will time out at 997.533753s 2024-09-16T03:42:30.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.561+0000 7f13e8990640 10 monclient(hunting): get_auth_request con 0x7f13e40a39b0 auth_method 0 2024-09-16T03:42:30.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.561+0000 7f13e8990640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:30.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.561+0000 7f13e8990640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:30.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.561+0000 7f13e8990640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:30.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.561+0000 7f13e9191640 1 --2- 172.21.15.42:0/3303871727 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f13e4137710 0x7f13e40a2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:30.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.561+0000 7f13e3fff640 1 --2- 172.21.15.42:0/3303871727 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f13e40a3d90 0x7f13e40a1f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:30.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.561+0000 7f13e9191640 10 monclient(hunting): get_auth_request con 0x7f13e4137710 auth_method 0 2024-09-16T03:42:30.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.561+0000 7f13e9191640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:30.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.561+0000 7f13e9191640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:30.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.561+0000 7f13e9191640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:30.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.561+0000 7f13e3fff640 10 monclient(hunting): get_auth_request con 0x7f13e40a3d90 auth_method 0 2024-09-16T03:42:30.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.561+0000 7f13e3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:30.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.561+0000 7f13e3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:30.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.561+0000 7f13e3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:30.578 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.561+0000 7f13e8990640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:30.578 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.561+0000 7f13e8990640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:30.578 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.561+0000 7f13e8990640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:30.578 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.561+0000 7f13e9191640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:30.578 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.561+0000 7f13e9191640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:30.578 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.561+0000 7f13e9191640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:30.578 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.561+0000 7f13e3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:30.578 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.561+0000 7f13e3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:30.578 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.561+0000 7f13e3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:30.578 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.561+0000 7f13e8990640 10 monclient(hunting): handle_auth_done global_id 4286 payload 824 2024-09-16T03:42:30.578 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.561+0000 7f13e8990640 1 -- 172.21.15.42:0/3303871727 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f13e40a3d90 msgr2=0x7f13e40a1f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:30.578 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.561+0000 7f13e8990640 1 --2- 172.21.15.42:0/3303871727 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f13e40a3d90 0x7f13e40a1f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:30.578 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.561+0000 7f13e8990640 1 -- 172.21.15.42:0/3303871727 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f13e4137710 msgr2=0x7f13e40a2450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:30.578 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.561+0000 7f13e8990640 1 --2- 172.21.15.42:0/3303871727 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f13e4137710 0x7f13e40a2450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:30.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.561+0000 7f13e8990640 10 monclient: _finish_hunting 0 2024-09-16T03:42:30.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.561+0000 7f13e8990640 1 monclient: found mon.noname-c 2024-09-16T03:42:30.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.561+0000 7f13e8990640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:42:30.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.561+0000 7f13e8990640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:42:30.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.561+0000 7f13e8990640 1 -- 172.21.15.42:0/3303871727 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13e4132fc0 con 0x7f13e40a39b0 2024-09-16T03:42:30.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.561+0000 7f13e9191640 1 --2- 172.21.15.42:0/3303871727 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f13e4137710 0x7f13e40a2450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:42:30.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.561+0000 7f13e8990640 1 --2- 172.21.15.42:0/3303871727 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f13e40a39b0 0x7f13e40a19d0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f13d801a590 tx=0x7f13d801a720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:30.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.561+0000 7f13e17fa640 10 client.?.objecter ms_handle_connect 0x7f13e40a39b0 2024-09-16T03:42:30.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.561+0000 7f13e17fa640 10 client.?.objecter resend_mon_ops 2024-09-16T03:42:30.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.561+0000 7f13e17fa640 1 -- 172.21.15.42:0/3303871727 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13d80b4810 con 0x7f13e40a39b0 2024-09-16T03:42:30.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.561+0000 7f13e17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:30.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.561+0000 7f13e17fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:42:30.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.561+0000 7f13e17fa640 10 monclient: dump: 2024-09-16T03:42:30.580 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:30.580 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:30.580 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:30.580 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:30.580 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:30.580 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:30.580 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:30.580 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:30.580 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:30.580 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:30.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.561+0000 7f13e17fa640 10 monclient: _finish_auth 0 2024-09-16T03:42:30.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.561+0000 7f13e17fa640 10 monclient: _check_auth_tickets 2024-09-16T03:42:30.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.561+0000 7f13e17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:42:30.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.561+0000 7f13e17fa640 1 -- 172.21.15.42:0/3303871727 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f13d80b49b0 con 0x7f13e40a39b0 2024-09-16T03:42:30.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.561+0000 7f13eabf4640 5 monclient: authenticate success, global_id 4286 2024-09-16T03:42:30.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.561+0000 7f13eabf4640 10 monclient: _renew_subs 2024-09-16T03:42:30.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.561+0000 7f13eabf4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:42:30.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.561+0000 7f13eabf4640 1 -- 172.21.15.42:0/3303871727 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13e413ff10 con 0x7f13e40a39b0 2024-09-16T03:42:30.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.561+0000 7f13e17fa640 10 monclient: handle_config config(0 keys) 2024-09-16T03:42:30.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.561+0000 7f13e17fa640 1 -- 172.21.15.42:0/3303871727 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13d80b4da0 con 0x7f13e40a39b0 2024-09-16T03:42:30.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.561+0000 7f13e17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:30.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.561+0000 7f13e17fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:42:30.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.561+0000 7f13e17fa640 10 monclient: dump: 2024-09-16T03:42:30.581 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:30.581 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:30.581 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:30.581 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:30.581 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:30.582 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:30.582 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:30.582 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:30.582 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:30.582 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:30.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.561+0000 7f13eabf4640 10 client.4286.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:42:30.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.561+0000 7f13eabf4640 10 monclient: _renew_subs 2024-09-16T03:42:30.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.561+0000 7f13eabf4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:42:30.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.561+0000 7f13e2ffd640 4 set_mon_vals no callback set 2024-09-16T03:42:30.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.561+0000 7f13eabf4640 1 -- 172.21.15.42:0/3303871727 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f13e4133510 con 0x7f13e40a39b0 2024-09-16T03:42:30.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.561+0000 7f13eabf4640 1 librados: init done 2024-09-16T03:42:30.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.565+0000 7f13eabf4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:42:30.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.565+0000 7f13eabf4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:42:30.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.565+0000 7f13eabf4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:42:30.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.565+0000 7f13e17fa640 1 -- 172.21.15.42:0/3303871727 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f13d80bc020 con 0x7f13e40a39b0 2024-09-16T03:42:30.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.565+0000 7f13eabf4640 1 -- 172.21.15.42:0/3303871727 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f13e4139900 con 0x7f13e40a39b0 2024-09-16T03:42:30.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.565+0000 7f13e17fa640 1 --2- 172.21.15.42:0/3303871727 >> v2:172.21.15.154:6812/254845342 conn(0x7f13c8037840 0x7f13c8039d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:30.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.565+0000 7f13e17fa640 1 -- 172.21.15.42:0/3303871727 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(18..18 src has 1..18) ==== 3562+0+0 (secure 0 0 0) 0x7f13d80e2610 con 0x7f13e40a39b0 2024-09-16T03:42:30.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.565+0000 7f13e17fa640 10 client.4286.objecter ms_dispatch 0x7f13e4000ba0 osd_map(18..18 src has 1..18) 2024-09-16T03:42:30.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.565+0000 7f13e17fa640 3 client.4286.objecter handle_osd_map got epochs [18,18] > 0 2024-09-16T03:42:30.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.565+0000 7f13e17fa640 3 client.4286.objecter handle_osd_map decoding full epoch 18 2024-09-16T03:42:30.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.565+0000 7f13e17fa640 20 client.4286.objecter dump_active .. 0 homeless 2024-09-16T03:42:30.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.565+0000 7f13e3fff640 1 --2- 172.21.15.42:0/3303871727 >> v2:172.21.15.154:6812/254845342 conn(0x7f13c8037840 0x7f13c8039d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:30.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.565+0000 7f13e3fff640 10 monclient: get_auth_request con 0x7f13c8037840 auth_method 0 2024-09-16T03:42:30.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.565+0000 7f13e17fa640 10 client.4286.objecter ms_handle_connect 0x7f13c8037840 2024-09-16T03:42:30.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.565+0000 7f13e3fff640 1 --2- 172.21.15.42:0/3303871727 >> v2:172.21.15.154:6812/254845342 conn(0x7f13c8037840 0x7f13c8039d00 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f13d0000c10 tx=0x7f13d0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:30.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.569+0000 7f13e17fa640 1 -- 172.21.15.42:0/3303871727 <== mon.2 v2:172.21.15.42:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f13e4139900 con 0x7f13e40a39b0 2024-09-16T03:42:30.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.569+0000 7f13e17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:42:30.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.569+0000 7f13e17fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:42:30.588 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.577+0000 7f4268900640 10 monclient: get_monmap_and_config 2024-09-16T03:42:30.588 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.577+0000 7f4268900640 10 monclient: build_initial_monmap 2024-09-16T03:42:30.588 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.577+0000 7f4268900640 10 monclient: monmap: 2024-09-16T03:42:30.588 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:42:30.588 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:30.589 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:42:30.580454+0000 2024-09-16T03:42:30.589 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:42:30.580454+0000 2024-09-16T03:42:30.589 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:42:30.589 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:30.589 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:42:30.589 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:42:30.589 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:42:30.589 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:42:30.589 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:42:30.589 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:30.589 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.577+0000 7f4268900640 1 Processor -- start 2024-09-16T03:42:30.589 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.577+0000 7f4268900640 1 -- start start 2024-09-16T03:42:30.589 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.577+0000 7f4268900640 10 monclient: init 2024-09-16T03:42:30.589 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.577+0000 7f4268900640 10 monclient: _reopen_session rank -1 2024-09-16T03:42:30.590 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.577+0000 7f4268900640 10 monclient: _add_conns ranks=[0,4,1,2,3] 2024-09-16T03:42:30.590 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.577+0000 7f4268900640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f42640a39b0 addr v1:172.21.15.42:6789/0 2024-09-16T03:42:30.590 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.577+0000 7f4268900640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f42640a3d90 0x7f426413c380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:30.590 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.577+0000 7f4268900640 10 monclient(hunting): picked mon.noname-c con 0x7f42640a3d90 addr v2:172.21.15.154:3300/0 2024-09-16T03:42:30.590 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.577+0000 7f4268900640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f426413c8c0 addr v1:172.21.15.154:6789/0 2024-09-16T03:42:30.590 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.577+0000 7f4268900640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4264134290 con 0x7f42640a39b0 2024-09-16T03:42:30.590 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.577+0000 7f4268900640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f426409c140 con 0x7f426413c8c0 2024-09-16T03:42:30.590 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.577+0000 7f4268900640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:30.590 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.577+0000 7f4268900640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f426409c420 con 0x7f42640a3d90 2024-09-16T03:42:30.590 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.577+0000 7f4268900640 10 monclient(hunting): _renew_subs 2024-09-16T03:42:30.590 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.577+0000 7f4268900640 10 monclient(hunting): authenticate will time out at 997.549780s 2024-09-16T03:42:30.590 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.577+0000 7f426259c640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f42640a39b0 0x7f4264133b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:46822/0 (socket says 172.21.15.42:46822) 2024-09-16T03:42:30.590 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.577+0000 7f426259c640 1 -- 172.21.15.42:0/1220672880 learned_addr learned my addr 172.21.15.42:0/1220672880 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:42:30.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.577+0000 7f4261d9b640 1 --2- 172.21.15.42:0/1220672880 >> v2:172.21.15.154:3300/0 conn(0x7f42640a3d90 0x7f426413c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:30.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.577+0000 7f4261d9b640 10 monclient(hunting): get_auth_request con 0x7f42640a3d90 auth_method 0 2024-09-16T03:42:30.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.577+0000 7f4261d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:30.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.577+0000 7f4261d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:30.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.577+0000 7f4261d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:30.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.577+0000 7f426159a640 1 -- 172.21.15.42:0/1220672880 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1205577489 0 0) 0x7f4264134290 con 0x7f42640a39b0 2024-09-16T03:42:30.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.577+0000 7f426159a640 10 monclient(hunting): _negotiate 2024-09-16T03:42:30.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.577+0000 7f426159a640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:30.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.577+0000 7f426159a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:30.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.577+0000 7f426159a640 10 monclient(hunting): my global_id is 4335 2024-09-16T03:42:30.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.577+0000 7f426159a640 1 -- 172.21.15.42:0/1220672880 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f424c003530 con 0x7f42640a39b0 2024-09-16T03:42:30.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.577+0000 7f4261d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:30.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.577+0000 7f4261d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:30.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.577+0000 7f4261d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:30.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.577+0000 7f426159a640 1 -- 172.21.15.42:0/1220672880 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 545856425 0 0) 0x7f424c003530 con 0x7f42640a39b0 2024-09-16T03:42:30.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.577+0000 7f426159a640 1 -- 172.21.15.42:0/1220672880 >> v2:172.21.15.154:3300/0 conn(0x7f42640a3d90 msgr2=0x7f426413c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:42:30.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.577+0000 7f426159a640 1 --2- 172.21.15.42:0/1220672880 >> v2:172.21.15.154:3300/0 conn(0x7f42640a3d90 0x7f426413c380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:30.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.577+0000 7f426159a640 1 -- 172.21.15.42:0/1220672880 >> v1:172.21.15.154:6789/0 conn(0x7f426413c8c0 legacy=0x7f426413ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:30.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.577+0000 7f426159a640 10 monclient: _finish_hunting 0 2024-09-16T03:42:30.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.577+0000 7f426159a640 1 monclient: found mon.noname-a-legacy 2024-09-16T03:42:30.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.577+0000 7f426159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:42:30.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.577+0000 7f426159a640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.42:6789/0 2024-09-16T03:42:30.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.577+0000 7f426159a640 1 -- 172.21.15.42:0/1220672880 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f426406dd60 con 0x7f42640a39b0 2024-09-16T03:42:30.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.577+0000 7f426159a640 10 monclient: _finish_auth 0 2024-09-16T03:42:30.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.577+0000 7f426159a640 10 monclient: _check_auth_tickets 2024-09-16T03:42:30.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.577+0000 7f426159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:42:30.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.577+0000 7f426159a640 1 -- 172.21.15.42:0/1220672880 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2177413734 0 0) 0x7f4254001680 con 0x7f42640a39b0 2024-09-16T03:42:30.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.577+0000 7f426159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:30.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.577+0000 7f426159a640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T03:42:30.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.577+0000 7f426159a640 10 monclient: dump: 2024-09-16T03:42:30.593 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:30.593 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:30.593 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:30.593 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:30.593 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:30.593 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:30.593 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:30.593 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:30.593 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:30.593 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:30.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.577+0000 7f426159a640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] but i'm connected to v1:172.21.15.42:6789/0, reconnecting 2024-09-16T03:42:30.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.577+0000 7f426159a640 10 monclient: _reopen_session rank -1 2024-09-16T03:42:30.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.577+0000 7f426159a640 1 -- 172.21.15.42:0/1220672880 >> v1:172.21.15.42:6789/0 conn(0x7f42640a39b0 legacy=0x7f4264133b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:30.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.577+0000 7f426159a640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:42:30.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.577+0000 7f426159a640 1 --2- 172.21.15.42:0/1220672880 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f424c003a10 0x7f424c003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:30.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.577+0000 7f426159a640 10 monclient(hunting): picked mon.c con 0x7f424c003a10 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:42:30.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.577+0000 7f426159a640 1 --2- 172.21.15.42:0/1220672880 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f424c004810 0x7f424c004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:30.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.577+0000 7f426159a640 10 monclient(hunting): picked mon.a con 0x7f424c004810 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:42:30.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.577+0000 7f426159a640 1 --2- 172.21.15.42:0/1220672880 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f424c0052f0 0x7f424c00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:30.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.577+0000 7f426159a640 10 monclient(hunting): picked mon.b con 0x7f424c0052f0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:42:30.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.577+0000 7f426159a640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:30.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.577+0000 7f426159a640 1 -- 172.21.15.42:0/1220672880 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f426409c420 con 0x7f424c004810 2024-09-16T03:42:30.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.577+0000 7f426159a640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:30.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.577+0000 7f426159a640 1 -- 172.21.15.42:0/1220672880 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f424c00c1f0 con 0x7f424c0052f0 2024-09-16T03:42:30.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.577+0000 7f426159a640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:30.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.577+0000 7f426159a640 1 -- 172.21.15.42:0/1220672880 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f424c00c3f0 con 0x7f424c003a10 2024-09-16T03:42:30.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.577+0000 7f426159a640 10 monclient(hunting): _renew_subs 2024-09-16T03:42:30.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.577+0000 7f426159a640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:42:30.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.577+0000 7f426259c640 1 --2- 172.21.15.42:0/1220672880 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f424c003a10 0x7f424c003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:30.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.577+0000 7f426259c640 10 monclient(hunting): get_auth_request con 0x7f424c003a10 auth_method 0 2024-09-16T03:42:30.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.577+0000 7f426259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:30.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.577+0000 7f426259c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:30.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.577+0000 7f426259c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:42:30.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.577+0000 7f426259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:30.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.577+0000 7f426259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:30.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.577+0000 7f426259c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T03:42:30.596 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.577+0000 7f4261d9b640 1 --2- 172.21.15.42:0/1220672880 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f424c004810 0x7f424c004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:30.596 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.577+0000 7f4262d9d640 1 --2- 172.21.15.42:0/1220672880 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f424c0052f0 0x7f424c00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:30.596 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.577+0000 7f4261d9b640 10 monclient(hunting): get_auth_request con 0x7f424c004810 auth_method 0 2024-09-16T03:42:30.596 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.577+0000 7f4261d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:30.596 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.577+0000 7f4261d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:30.596 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.577+0000 7f4261d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:42:30.596 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.577+0000 7f4262d9d640 10 monclient(hunting): get_auth_request con 0x7f424c0052f0 auth_method 0 2024-09-16T03:42:30.596 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.577+0000 7f4262d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:30.596 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.577+0000 7f4262d9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:30.596 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.577+0000 7f4262d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:42:30.596 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.577+0000 7f426259c640 10 monclient(hunting): handle_auth_done global_id 4335 payload 293 2024-09-16T03:42:30.596 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.577+0000 7f426259c640 1 -- 172.21.15.42:0/1220672880 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f424c0052f0 msgr2=0x7f424c00b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:42:30.596 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.577+0000 7f426259c640 1 --2- 172.21.15.42:0/1220672880 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f424c0052f0 0x7f424c00b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:30.596 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.577+0000 7f426259c640 1 -- 172.21.15.42:0/1220672880 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f424c004810 msgr2=0x7f424c004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:42:30.597 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.577+0000 7f426259c640 1 --2- 172.21.15.42:0/1220672880 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f424c004810 0x7f424c004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:30.597 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.577+0000 7f426259c640 10 monclient: _finish_hunting 0 2024-09-16T03:42:30.597 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.577+0000 7f426259c640 1 monclient: found mon.c 2024-09-16T03:42:30.597 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.577+0000 7f426259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:42:30.597 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.577+0000 7f426259c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:42:30.597 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.577+0000 7f426259c640 1 -- 172.21.15.42:0/1220672880 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f424c003530 con 0x7f424c003a10 2024-09-16T03:42:30.597 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.577+0000 7f426259c640 10 monclient: _finish_auth 0 2024-09-16T03:42:30.597 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.577+0000 7f426259c640 10 monclient: _check_auth_tickets 2024-09-16T03:42:30.597 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.577+0000 7f426259c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:42:30.597 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.577+0000 7f4262d9d640 1 -- 172.21.15.42:0/1220672880 reap_dead start 2024-09-16T03:42:30.597 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.577+0000 7f4261d9b640 1 --2- 172.21.15.42:0/1220672880 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f424c004810 0x7f424c004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T03:42:30.597 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.577+0000 7f4268900640 5 monclient: authenticate success, global_id 4335 2024-09-16T03:42:30.597 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.577+0000 7f4268900640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:42:30.598 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.581+0000 7f426259c640 1 --2- 172.21.15.42:0/1220672880 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f424c003a10 0x7f424c003e30 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f425400c190 tx=0x7f425401cbc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5b5bf6f677f6eb10 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:30.603 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.581+0000 7f426159a640 1 -- 172.21.15.42:0/1220672880 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4254007c70 con 0x7f424c003a10 2024-09-16T03:42:30.603 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.581+0000 7f426159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:30.603 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.581+0000 7f426159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:42:30.603 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.581+0000 7f426159a640 10 monclient: dump: 2024-09-16T03:42:30.603 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:30.603 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:30.603 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:30.603 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:30.604 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:30.604 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:30.604 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:30.604 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:30.604 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:30.604 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:30.604 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.581+0000 7f426159a640 1 -- 172.21.15.42:0/1220672880 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f42540a9070 con 0x7f424c003a10 2024-09-16T03:42:30.604 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.581+0000 7f426159a640 10 monclient: handle_config config(0 keys) 2024-09-16T03:42:30.604 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.581+0000 7f426159a640 1 -- 172.21.15.42:0/1220672880 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4254005dc0 con 0x7f424c003a10 2024-09-16T03:42:30.604 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.581+0000 7f426159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:30.604 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.581+0000 7f426159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:42:30.604 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.581+0000 7f426159a640 10 monclient: dump: 2024-09-16T03:42:30.604 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:30.605 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:30.605 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:30.605 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:30.605 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:30.605 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:30.605 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:30.605 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:30.605 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:30.605 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:30.605 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.581+0000 7f4268900640 10 monclient: get_monmap_and_config success 2024-09-16T03:42:30.605 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.581+0000 7f4268900640 4 set_mon_vals no callback set 2024-09-16T03:42:30.605 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.581+0000 7f4268900640 10 monclient: shutdown 2024-09-16T03:42:30.605 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.581+0000 7f4268900640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:42:30.606 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.581+0000 7f4268900640 1 -- 172.21.15.42:0/1220672880 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f424c003a10 msgr2=0x7f424c003e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:30.606 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.581+0000 7f4268900640 1 --2- 172.21.15.42:0/1220672880 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f424c003a10 0x7f424c003e30 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f425400c190 tx=0x7f425401cbc0 comp rx=0 tx=0).stop 2024-09-16T03:42:30.606 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.581+0000 7f4268900640 1 -- 172.21.15.42:0/1220672880 shutdown_connections 2024-09-16T03:42:30.606 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.581+0000 7f4268900640 1 --2- 172.21.15.42:0/1220672880 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f424c003a10 0x7f424c003e30 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:30.606 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.581+0000 7f4268900640 1 -- 172.21.15.42:0/1220672880 >> 172.21.15.42:0/1220672880 conn(0x7f42640a9fb0 msgr2=0x7f42640aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:42:30.606 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.581+0000 7f4268900640 1 -- 172.21.15.42:0/1220672880 shutdown_connections 2024-09-16T03:42:30.606 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.581+0000 7f4268900640 1 -- 172.21.15.42:0/1220672880 wait complete. 2024-09-16T03:42:30.606 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.585+0000 7f4268900640 10 monclient: build_initial_monmap 2024-09-16T03:42:30.606 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.585+0000 7f4268900640 10 monclient: monmap: 2024-09-16T03:42:30.606 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:42:30.606 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:42:30.606 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:42:30.606 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:42:30.606 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:42:30.607 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:30.607 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:42:30.607 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:42:30.607 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:42:30.607 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:30.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.585+0000 7f4268900640 1 librados: starting msgr at 2024-09-16T03:42:30.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.585+0000 7f4268900640 1 librados: starting objecter 2024-09-16T03:42:30.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.585+0000 7f4268900640 1 Processor -- start 2024-09-16T03:42:30.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.585+0000 7f4268900640 1 -- start start 2024-09-16T03:42:30.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.585+0000 7f4268900640 1 librados: setting wanted keys 2024-09-16T03:42:30.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.585+0000 7f4268900640 1 librados: calling monclient init 2024-09-16T03:42:30.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.585+0000 7f4268900640 10 monclient: init 2024-09-16T03:42:30.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.585+0000 7f4268900640 10 monclient: _reopen_session rank -1 2024-09-16T03:42:30.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.585+0000 7f4268900640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T03:42:30.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.585+0000 7f4268900640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f424c003a10 0x7f42640a01d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:30.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.585+0000 7f4268900640 10 monclient(hunting): picked mon.noname-a con 0x7f424c003a10 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:42:30.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.585+0000 7f4268900640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f42640a0710 0x7f42640a0b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:30.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.585+0000 7f4268900640 10 monclient(hunting): picked mon.noname-b con 0x7f42640a0710 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:42:30.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.585+0000 7f4268900640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f42640a1090 0x7f42641d5ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:30.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.585+0000 7f4268900640 10 monclient(hunting): picked mon.noname-c con 0x7f42640a1090 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:42:30.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.585+0000 7f4268900640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:30.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.585+0000 7f4268900640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f4264137ba0 con 0x7f424c003a10 2024-09-16T03:42:30.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.585+0000 7f4268900640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:30.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.585+0000 7f4268900640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f4264137a20 con 0x7f42640a0710 2024-09-16T03:42:30.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.585+0000 7f4268900640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:30.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.585+0000 7f4268900640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f4264137d20 con 0x7f42640a1090 2024-09-16T03:42:30.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.585+0000 7f4268900640 10 monclient(hunting): _renew_subs 2024-09-16T03:42:30.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.585+0000 7f4268900640 10 monclient(hunting): authenticate will time out at 997.557462s 2024-09-16T03:42:30.609 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.585+0000 7f426259c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f424c003a10 0x7f42640a01d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:30.609 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.585+0000 7f426259c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f424c003a10 0x7f42640a01d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:37788/0 (socket says 172.21.15.42:37788) 2024-09-16T03:42:30.609 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.585+0000 7f426259c640 1 -- 172.21.15.42:0/2167052642 learned_addr learned my addr 172.21.15.42:0/2167052642 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:42:30.609 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.585+0000 7f426259c640 10 monclient(hunting): get_auth_request con 0x7f424c003a10 auth_method 0 2024-09-16T03:42:30.609 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.585+0000 7f426259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:30.609 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.585+0000 7f426259c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:30.609 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.585+0000 7f426259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:30.609 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.585+0000 7f4262d9d640 1 --2- 172.21.15.42:0/2167052642 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f42640a1090 0x7f42641d5ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:30.609 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.585+0000 7f4262d9d640 10 monclient(hunting): get_auth_request con 0x7f42640a1090 auth_method 0 2024-09-16T03:42:30.609 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.585+0000 7f4262d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:30.609 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.585+0000 7f4262d9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:30.609 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.585+0000 7f4262d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:30.609 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.585+0000 7f426259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:30.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.585+0000 7f426259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:30.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.585+0000 7f426259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:30.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.585+0000 7f4262d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:30.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.585+0000 7f4262d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:30.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.585+0000 7f4262d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:30.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.585+0000 7f4262d9d640 10 monclient(hunting): handle_auth_done global_id 4289 payload 824 2024-09-16T03:42:30.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.585+0000 7f4262d9d640 1 -- 172.21.15.42:0/2167052642 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f42640a0710 msgr2=0x7f42640a0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:30.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.585+0000 7f4262d9d640 1 --2- 172.21.15.42:0/2167052642 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f42640a0710 0x7f42640a0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:30.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.585+0000 7f4262d9d640 1 -- 172.21.15.42:0/2167052642 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f424c003a10 msgr2=0x7f42640a01d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:30.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.585+0000 7f4262d9d640 1 --2- 172.21.15.42:0/2167052642 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f424c003a10 0x7f42640a01d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:30.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.585+0000 7f4262d9d640 10 monclient: _finish_hunting 0 2024-09-16T03:42:30.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.585+0000 7f4262d9d640 1 monclient: found mon.noname-c 2024-09-16T03:42:30.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.585+0000 7f4262d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:42:30.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.585+0000 7f4262d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:42:30.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.585+0000 7f4262d9d640 1 -- 172.21.15.42:0/2167052642 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42641204a0 con 0x7f42640a1090 2024-09-16T03:42:30.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.585+0000 7f426259c640 1 --2- 172.21.15.42:0/2167052642 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f424c003a10 0x7f42640a01d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:42:30.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.585+0000 7f4262d9d640 1 --2- 172.21.15.42:0/2167052642 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f42640a1090 0x7f42641d5ac0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f42640a7a80 tx=0x7f425c00ec10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:30.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.585+0000 7f4252ffd640 10 client.?.objecter ms_handle_connect 0x7f42640a1090 2024-09-16T03:42:30.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.585+0000 7f4252ffd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:42:30.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.585+0000 7f4252ffd640 1 -- 172.21.15.42:0/2167052642 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f425c003980 con 0x7f42640a1090 2024-09-16T03:42:30.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.585+0000 7f4252ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:30.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.585+0000 7f4252ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:42:30.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.585+0000 7f4252ffd640 10 monclient: dump: 2024-09-16T03:42:30.611 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:30.611 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:30.611 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:30.611 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:30.612 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:30.612 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:30.612 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:30.612 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:30.612 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:30.612 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:30.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.585+0000 7f4252ffd640 10 monclient: _finish_auth 0 2024-09-16T03:42:30.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.585+0000 7f4252ffd640 10 monclient: _check_auth_tickets 2024-09-16T03:42:30.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.585+0000 7f4252ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:42:30.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.585+0000 7f4268900640 5 monclient: authenticate success, global_id 4289 2024-09-16T03:42:30.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.585+0000 7f4252ffd640 1 -- 172.21.15.42:0/2167052642 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f425c018070 con 0x7f42640a1090 2024-09-16T03:42:30.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.585+0000 7f4252ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:42:30.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.585+0000 7f4252ffd640 1 -- 172.21.15.42:0/2167052642 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f425c003dc0 con 0x7f42640a1090 2024-09-16T03:42:30.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.585+0000 7f4252ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:30.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.585+0000 7f4252ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:42:30.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.585+0000 7f4252ffd640 10 monclient: dump: 2024-09-16T03:42:30.613 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:30.613 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:30.613 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:30.613 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:30.613 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:30.613 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:30.613 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:30.613 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:30.613 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:30.613 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:30.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.585+0000 7f4260d99640 4 set_mon_vals no callback set 2024-09-16T03:42:30.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.585+0000 7f4268900640 10 monclient: _renew_subs 2024-09-16T03:42:30.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.585+0000 7f4268900640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:42:30.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.585+0000 7f4268900640 1 -- 172.21.15.42:0/2167052642 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42640a20b0 con 0x7f42640a1090 2024-09-16T03:42:30.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.585+0000 7f4268900640 10 client.4289.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:42:30.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.585+0000 7f4268900640 10 monclient: _renew_subs 2024-09-16T03:42:30.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.585+0000 7f4268900640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:42:30.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.585+0000 7f4268900640 1 -- 172.21.15.42:0/2167052642 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f4264120780 con 0x7f42640a1090 2024-09-16T03:42:30.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.585+0000 7f4268900640 1 librados: init done 2024-09-16T03:42:30.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.589+0000 7f4268900640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:42:30.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.589+0000 7f4268900640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:42:30.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.589+0000 7f4268900640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:42:30.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.589+0000 7f4268900640 1 -- 172.21.15.42:0/2167052642 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f42641333c0 con 0x7f42640a1090 2024-09-16T03:42:30.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.589+0000 7f4252ffd640 1 -- 172.21.15.42:0/2167052642 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f425c0050d0 con 0x7f42640a1090 2024-09-16T03:42:30.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.589+0000 7f4252ffd640 1 --2- 172.21.15.42:0/2167052642 >> v2:172.21.15.154:6812/254845342 conn(0x7f4244037860 0x7f4244039d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:30.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.589+0000 7f4252ffd640 1 -- 172.21.15.42:0/2167052642 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(18..18 src has 1..18) ==== 3562+0+0 (secure 0 0 0) 0x7f425c0482f0 con 0x7f42640a1090 2024-09-16T03:42:30.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.589+0000 7f4252ffd640 10 client.4289.objecter ms_dispatch 0x7f4264000ba0 osd_map(18..18 src has 1..18) 2024-09-16T03:42:30.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.589+0000 7f4252ffd640 3 client.4289.objecter handle_osd_map got epochs [18,18] > 0 2024-09-16T03:42:30.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.589+0000 7f4252ffd640 3 client.4289.objecter handle_osd_map decoding full epoch 18 2024-09-16T03:42:30.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.589+0000 7f4252ffd640 20 client.4289.objecter dump_active .. 0 homeless 2024-09-16T03:42:30.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.589+0000 7f426259c640 1 --2- 172.21.15.42:0/2167052642 >> v2:172.21.15.154:6812/254845342 conn(0x7f4244037860 0x7f4244039d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:30.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.589+0000 7f426259c640 10 monclient: get_auth_request con 0x7f4244037860 auth_method 0 2024-09-16T03:42:30.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.589+0000 7f426259c640 1 --2- 172.21.15.42:0/2167052642 >> v2:172.21.15.154:6812/254845342 conn(0x7f4244037860 0x7f4244039d20 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f425401cc00 tx=0x7f42540a7070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:30.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.589+0000 7f4252ffd640 10 client.4289.objecter ms_handle_connect 0x7f4244037860 2024-09-16T03:42:30.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.593+0000 7f4252ffd640 1 -- 172.21.15.42:0/2167052642 <== mon.2 v2:172.21.15.42:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f425c005790 con 0x7f42640a1090 2024-09-16T03:42:30.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.593+0000 7f4252ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:42:30.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.593+0000 7f4252ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:42:30.657 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.649+0000 7f0286c7a640 10 monclient: get_monmap_and_config 2024-09-16T03:42:30.657 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.649+0000 7f0286c7a640 10 monclient: build_initial_monmap 2024-09-16T03:42:30.657 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.649+0000 7f0286c7a640 10 monclient: monmap: 2024-09-16T03:42:30.658 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:42:30.658 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:30.658 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:42:30.655309+0000 2024-09-16T03:42:30.658 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:42:30.655309+0000 2024-09-16T03:42:30.658 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:42:30.658 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:30.658 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:42:30.658 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:42:30.658 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:42:30.658 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:42:30.658 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:42:30.658 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:30.660 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.653+0000 7f0286c7a640 1 Processor -- start 2024-09-16T03:42:30.660 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.653+0000 7f0286c7a640 1 -- start start 2024-09-16T03:42:30.660 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.653+0000 7f0286c7a640 10 monclient: init 2024-09-16T03:42:30.660 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.653+0000 7f0286c7a640 10 monclient: _reopen_session rank -1 2024-09-16T03:42:30.660 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.653+0000 7f0286c7a640 10 monclient: _add_conns ranks=[2,4,3,1,0] 2024-09-16T03:42:30.660 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.653+0000 7f0286c7a640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f02800a39b0 0x7f0280133b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:30.660 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.653+0000 7f0286c7a640 10 monclient(hunting): picked mon.noname-b con 0x7f02800a39b0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:42:30.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.653+0000 7f0286c7a640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f02800a3d90 0x7f0280137000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:30.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.653+0000 7f0286c7a640 10 monclient(hunting): picked mon.noname-c con 0x7f02800a3d90 addr v2:172.21.15.154:3300/0 2024-09-16T03:42:30.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.653+0000 7f0286c7a640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f0280137540 0x7f0280137950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:30.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.653+0000 7f0286c7a640 10 monclient(hunting): picked mon.noname-a con 0x7f0280137540 addr v2:172.21.15.42:3300/0 2024-09-16T03:42:30.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.653+0000 7f0284a16640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f02800a39b0 0x7f0280133b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:30.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.653+0000 7f0286c7a640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:30.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.653+0000 7f0284a16640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f02800a39b0 0x7f0280133b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:41512/0 (socket says 172.21.15.42:41512) 2024-09-16T03:42:30.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.653+0000 7f0286c7a640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f0280137e90 con 0x7f0280137540 2024-09-16T03:42:30.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.653+0000 7f0286c7a640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:30.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.653+0000 7f0284a16640 1 -- 172.21.15.42:0/3494075177 learned_addr learned my addr 172.21.15.42:0/3494075177 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:42:30.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.653+0000 7f0286c7a640 1 -- 172.21.15.42:0/3494075177 --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f028009c140 con 0x7f02800a3d90 2024-09-16T03:42:30.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.653+0000 7f0286c7a640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:30.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.653+0000 7f0286c7a640 1 -- 172.21.15.42:0/3494075177 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f028009c420 con 0x7f02800a39b0 2024-09-16T03:42:30.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.653+0000 7f0286c7a640 10 monclient(hunting): _renew_subs 2024-09-16T03:42:30.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.653+0000 7f0286c7a640 10 monclient(hunting): authenticate will time out at 997.625385s 2024-09-16T03:42:30.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.653+0000 7f0285217640 1 --2- 172.21.15.42:0/3494075177 >> v2:172.21.15.42:3300/0 conn(0x7f0280137540 0x7f0280137950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:30.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.653+0000 7f027ffff640 1 --2- 172.21.15.42:0/3494075177 >> v2:172.21.15.154:3300/0 conn(0x7f02800a3d90 0x7f0280137000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:30.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.653+0000 7f0285217640 10 monclient(hunting): get_auth_request con 0x7f0280137540 auth_method 0 2024-09-16T03:42:30.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.653+0000 7f0285217640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:30.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.653+0000 7f0285217640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:30.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.653+0000 7f0285217640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:30.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.653+0000 7f027ffff640 10 monclient(hunting): get_auth_request con 0x7f02800a3d90 auth_method 0 2024-09-16T03:42:30.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.653+0000 7f027ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:30.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.653+0000 7f027ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:30.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.653+0000 7f027ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:30.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.653+0000 7f0284a16640 10 monclient(hunting): get_auth_request con 0x7f02800a39b0 auth_method 0 2024-09-16T03:42:30.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.653+0000 7f0284a16640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:30.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.653+0000 7f0284a16640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:30.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.653+0000 7f0284a16640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:30.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.653+0000 7f0285217640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:30.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.653+0000 7f0285217640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:30.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.653+0000 7f0285217640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:30.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.653+0000 7f027ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:30.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.653+0000 7f027ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:30.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.653+0000 7f027ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:30.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.653+0000 7f0284a16640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:30.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.653+0000 7f0284a16640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:30.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.653+0000 7f0284a16640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:30.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.653+0000 7f0285217640 10 monclient(hunting): handle_auth_done global_id 4341 payload 274 2024-09-16T03:42:30.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.653+0000 7f0285217640 1 -- 172.21.15.42:0/3494075177 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f02800a39b0 msgr2=0x7f0280133b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:42:30.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.653+0000 7f0285217640 1 --2- 172.21.15.42:0/3494075177 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f02800a39b0 0x7f0280133b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:30.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.653+0000 7f0285217640 1 -- 172.21.15.42:0/3494075177 >> v2:172.21.15.154:3300/0 conn(0x7f02800a3d90 msgr2=0x7f0280137000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:42:30.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.653+0000 7f0285217640 1 --2- 172.21.15.42:0/3494075177 >> v2:172.21.15.154:3300/0 conn(0x7f02800a3d90 0x7f0280137000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:30.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.653+0000 7f0285217640 10 monclient: _finish_hunting 0 2024-09-16T03:42:30.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.653+0000 7f0285217640 1 monclient: found mon.noname-a 2024-09-16T03:42:30.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.653+0000 7f0285217640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:42:30.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.653+0000 7f0285217640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:42:30.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.653+0000 7f0285217640 1 -- 172.21.15.42:0/3494075177 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f028006dd60 con 0x7f0280137540 2024-09-16T03:42:30.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.657+0000 7f0285217640 1 --2- 172.21.15.42:0/3494075177 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0280137540 0x7f0280137950 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f028006e3c0 tx=0x7f027801c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0e42741220e093 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:30.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.657+0000 7f027f7fe640 1 -- 172.21.15.42:0/3494075177 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02780ac070 con 0x7f0280137540 2024-09-16T03:42:30.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.657+0000 7f027f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:30.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.657+0000 7f027f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:42:30.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.657+0000 7f027f7fe640 10 monclient: dump: 2024-09-16T03:42:30.670 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:30.670 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:30.670 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:30.670 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:30.670 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:30.670 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:30.670 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:30.670 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:30.670 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:30.670 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:30.670 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.657+0000 7f027f7fe640 10 monclient: _finish_auth 0 2024-09-16T03:42:30.670 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.657+0000 7f027f7fe640 10 monclient: _check_auth_tickets 2024-09-16T03:42:30.670 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.657+0000 7f027f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:42:30.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.657+0000 7f027f7fe640 1 -- 172.21.15.42:0/3494075177 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f02780a66a0 con 0x7f0280137540 2024-09-16T03:42:30.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.657+0000 7f0286c7a640 5 monclient: authenticate success, global_id 4341 2024-09-16T03:42:30.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.657+0000 7f0286c7a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:42:30.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.657+0000 7f027f7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:42:30.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.657+0000 7f0286c7a640 10 monclient: get_monmap_and_config success 2024-09-16T03:42:30.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.657+0000 7f0286c7a640 4 set_mon_vals no callback set 2024-09-16T03:42:30.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.657+0000 7f027f7fe640 1 -- 172.21.15.42:0/3494075177 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02780a6ab0 con 0x7f0280137540 2024-09-16T03:42:30.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.657+0000 7f027f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:30.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.657+0000 7f027f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:42:30.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.657+0000 7f027f7fe640 10 monclient: dump: 2024-09-16T03:42:30.671 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:30.671 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:30.671 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:30.672 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:30.672 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:30.672 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:30.672 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:30.672 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:30.672 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:30.672 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:30.672 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.657+0000 7f0286c7a640 10 monclient: shutdown 2024-09-16T03:42:30.672 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.657+0000 7f0286c7a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:42:30.672 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.657+0000 7f0286c7a640 1 -- 172.21.15.42:0/3494075177 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0280137540 msgr2=0x7f0280137950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:30.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.657+0000 7f0286c7a640 1 --2- 172.21.15.42:0/3494075177 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0280137540 0x7f0280137950 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f028006e3c0 tx=0x7f027801c810 comp rx=0 tx=0).stop 2024-09-16T03:42:30.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.657+0000 7f0286c7a640 1 -- 172.21.15.42:0/3494075177 shutdown_connections 2024-09-16T03:42:30.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.657+0000 7f0286c7a640 1 --2- 172.21.15.42:0/3494075177 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0280137540 0x7f0280137950 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:30.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.657+0000 7f0286c7a640 1 --2- 172.21.15.42:0/3494075177 >> v2:172.21.15.154:3300/0 conn(0x7f02800a3d90 0x7f0280137000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:30.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.657+0000 7f0286c7a640 1 --2- 172.21.15.42:0/3494075177 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f02800a39b0 0x7f0280133b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:30.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.657+0000 7f0286c7a640 1 -- 172.21.15.42:0/3494075177 >> 172.21.15.42:0/3494075177 conn(0x7f02800a9fb0 msgr2=0x7f02800aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:42:30.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.657+0000 7f0286c7a640 1 -- 172.21.15.42:0/3494075177 shutdown_connections 2024-09-16T03:42:30.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.657+0000 7f0286c7a640 1 -- 172.21.15.42:0/3494075177 wait complete. 2024-09-16T03:42:30.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.657+0000 7f0286c7a640 10 monclient: build_initial_monmap 2024-09-16T03:42:30.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.657+0000 7f0286c7a640 10 monclient: monmap: 2024-09-16T03:42:30.673 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:42:30.673 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:42:30.674 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:42:30.674 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:42:30.674 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:42:30.674 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:30.674 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:42:30.674 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:42:30.674 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:42:30.674 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:30.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.661+0000 7f0286c7a640 1 librados: starting msgr at 2024-09-16T03:42:30.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.661+0000 7f0286c7a640 1 librados: starting objecter 2024-09-16T03:42:30.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.661+0000 7f0286c7a640 1 Processor -- start 2024-09-16T03:42:30.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.661+0000 7f0286c7a640 1 -- start start 2024-09-16T03:42:30.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.661+0000 7f0286c7a640 1 librados: setting wanted keys 2024-09-16T03:42:30.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.661+0000 7f0286c7a640 1 librados: calling monclient init 2024-09-16T03:42:30.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.661+0000 7f0286c7a640 10 monclient: init 2024-09-16T03:42:30.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.661+0000 7f0286c7a640 10 monclient: _reopen_session rank -1 2024-09-16T03:42:30.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.661+0000 7f0286c7a640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:42:30.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.661+0000 7f0286c7a640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f02800a39b0 0x7f028013b090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:30.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.661+0000 7f0286c7a640 10 monclient(hunting): picked mon.noname-b con 0x7f02800a39b0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:42:30.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.661+0000 7f0286c7a640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f02800a3d90 0x7f028013b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:30.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.661+0000 7f0286c7a640 10 monclient(hunting): picked mon.noname-a con 0x7f02800a3d90 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:42:30.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.661+0000 7f027ffff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f02800a3d90 0x7f028013b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:30.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.661+0000 7f027ffff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f02800a3d90 0x7f028013b5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:37810/0 (socket says 172.21.15.42:37810) 2024-09-16T03:42:30.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.661+0000 7f0286c7a640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0280137540 0x7f028013bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:30.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.661+0000 7f027ffff640 1 -- 172.21.15.42:0/2914394776 learned_addr learned my addr 172.21.15.42:0/2914394776 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:42:30.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.661+0000 7f0286c7a640 10 monclient(hunting): picked mon.noname-c con 0x7f0280137540 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:42:30.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.661+0000 7f0286c7a640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:30.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.661+0000 7f0286c7a640 1 -- 172.21.15.42:0/2914394776 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f0280140dc0 con 0x7f02800a3d90 2024-09-16T03:42:30.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.661+0000 7f0286c7a640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:30.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.661+0000 7f0286c7a640 1 -- 172.21.15.42:0/2914394776 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f0280140f40 con 0x7f02800a39b0 2024-09-16T03:42:30.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.661+0000 7f0286c7a640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:30.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.661+0000 7f0286c7a640 1 -- 172.21.15.42:0/2914394776 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f02801410c0 con 0x7f0280137540 2024-09-16T03:42:30.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.661+0000 7f0286c7a640 10 monclient(hunting): _renew_subs 2024-09-16T03:42:30.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.661+0000 7f0286c7a640 10 monclient(hunting): authenticate will time out at 997.633639s 2024-09-16T03:42:30.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.661+0000 7f027ffff640 10 monclient(hunting): get_auth_request con 0x7f02800a3d90 auth_method 0 2024-09-16T03:42:30.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.661+0000 7f027ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:30.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.661+0000 7f027ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:30.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.661+0000 7f027ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:30.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.661+0000 7f0284a16640 1 --2- 172.21.15.42:0/2914394776 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f02800a39b0 0x7f028013b090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:30.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.661+0000 7f0284a16640 10 monclient(hunting): get_auth_request con 0x7f02800a39b0 auth_method 0 2024-09-16T03:42:30.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.661+0000 7f0284a16640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:30.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.661+0000 7f0284a16640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:30.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.661+0000 7f0284a16640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:30.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.661+0000 7f027ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:30.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.661+0000 7f027ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:30.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.661+0000 7f027ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:30.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.661+0000 7f0285217640 1 --2- 172.21.15.42:0/2914394776 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0280137540 0x7f028013bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:30.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.661+0000 7f0285217640 10 monclient(hunting): get_auth_request con 0x7f0280137540 auth_method 0 2024-09-16T03:42:30.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.661+0000 7f0285217640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:30.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.661+0000 7f0285217640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:30.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.661+0000 7f0285217640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:30.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.661+0000 7f0284a16640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:30.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.661+0000 7f0284a16640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:30.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.661+0000 7f0284a16640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:30.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.661+0000 7f0285217640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:30.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.661+0000 7f0285217640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:30.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.661+0000 7f0285217640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:30.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.661+0000 7f027ffff640 10 monclient(hunting): handle_auth_done global_id 4344 payload 824 2024-09-16T03:42:30.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.661+0000 7f027ffff640 1 -- 172.21.15.42:0/2914394776 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0280137540 msgr2=0x7f028013bb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:30.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.661+0000 7f027ffff640 1 --2- 172.21.15.42:0/2914394776 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0280137540 0x7f028013bb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:30.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.661+0000 7f027ffff640 1 -- 172.21.15.42:0/2914394776 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f02800a39b0 msgr2=0x7f028013b090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:30.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.661+0000 7f027ffff640 1 --2- 172.21.15.42:0/2914394776 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f02800a39b0 0x7f028013b090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:30.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.661+0000 7f027ffff640 10 monclient: _finish_hunting 0 2024-09-16T03:42:30.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.661+0000 7f027ffff640 1 monclient: found mon.noname-a 2024-09-16T03:42:30.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.661+0000 7f027ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:42:30.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.661+0000 7f027ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:42:30.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.661+0000 7f027ffff640 1 -- 172.21.15.42:0/2914394776 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02800a4c90 con 0x7f02800a3d90 2024-09-16T03:42:30.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.661+0000 7f027ffff640 1 --2- 172.21.15.42:0/2914394776 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f02800a3d90 0x7f028013b5d0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f026c00e930 tx=0x7f026c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:30.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.661+0000 7f027d7fa640 10 client.?.objecter ms_handle_connect 0x7f02800a3d90 2024-09-16T03:42:30.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.661+0000 7f027d7fa640 10 client.?.objecter resend_mon_ops 2024-09-16T03:42:30.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.661+0000 7f027d7fa640 1 -- 172.21.15.42:0/2914394776 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f026c017070 con 0x7f02800a3d90 2024-09-16T03:42:30.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.661+0000 7f027d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:30.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.661+0000 7f027d7fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:42:30.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.661+0000 7f027d7fa640 10 monclient: dump: 2024-09-16T03:42:30.679 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:30.679 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:30.679 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:30.679 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:30.679 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:30.679 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:30.679 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:30.680 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:30.680 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:30.680 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:30.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.661+0000 7f027d7fa640 10 monclient: _finish_auth 0 2024-09-16T03:42:30.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.661+0000 7f027d7fa640 10 monclient: _check_auth_tickets 2024-09-16T03:42:30.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.661+0000 7f027d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:42:30.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.661+0000 7f027d7fa640 1 -- 172.21.15.42:0/2914394776 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f026c00ca00 con 0x7f02800a3d90 2024-09-16T03:42:30.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.661+0000 7f0286c7a640 5 monclient: authenticate success, global_id 4344 2024-09-16T03:42:30.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.661+0000 7f0286c7a640 10 monclient: _renew_subs 2024-09-16T03:42:30.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.661+0000 7f0286c7a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:42:30.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.661+0000 7f0286c7a640 1 -- 172.21.15.42:0/2914394776 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f028009cd20 con 0x7f02800a3d90 2024-09-16T03:42:30.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.661+0000 7f027d7fa640 10 monclient: handle_config config(0 keys) 2024-09-16T03:42:30.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.661+0000 7f027effd640 4 set_mon_vals no callback set 2024-09-16T03:42:30.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.661+0000 7f0286c7a640 10 client.4344.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:42:30.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.661+0000 7f027d7fa640 1 -- 172.21.15.42:0/2914394776 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f026c00cdb0 con 0x7f02800a3d90 2024-09-16T03:42:30.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.661+0000 7f027d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:30.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.661+0000 7f027d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:42:30.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.661+0000 7f027d7fa640 10 monclient: dump: 2024-09-16T03:42:30.683 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:30.683 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:30.683 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:30.683 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:30.683 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:30.683 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:30.683 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:30.683 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:30.683 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:30.683 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:30.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.665+0000 7f0286c7a640 10 monclient: _renew_subs 2024-09-16T03:42:30.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.665+0000 7f0286c7a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:42:30.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.665+0000 7f0286c7a640 1 -- 172.21.15.42:0/2914394776 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f028009d5c0 con 0x7f02800a3d90 2024-09-16T03:42:30.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.665+0000 7f0286c7a640 1 librados: init done 2024-09-16T03:42:30.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.665+0000 7f027d7fa640 1 -- 172.21.15.42:0/2914394776 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f026c01a070 con 0x7f02800a3d90 2024-09-16T03:42:30.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.665+0000 7f027d7fa640 1 --2- 172.21.15.42:0/2914394776 >> v2:172.21.15.154:6812/254845342 conn(0x7f0264037810 0x7f0264039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:30.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.665+0000 7f027d7fa640 1 -- 172.21.15.42:0/2914394776 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 3534+0+0 (secure 0 0 0) 0x7f026c04b030 con 0x7f02800a3d90 2024-09-16T03:42:30.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.665+0000 7f027d7fa640 10 client.4344.objecter ms_dispatch 0x7f0280000ba0 osd_map(19..19 src has 1..19) 2024-09-16T03:42:30.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.665+0000 7f027d7fa640 3 client.4344.objecter handle_osd_map got epochs [19,19] > 0 2024-09-16T03:42:30.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.665+0000 7f027d7fa640 3 client.4344.objecter handle_osd_map decoding full epoch 19 2024-09-16T03:42:30.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.665+0000 7f027d7fa640 20 client.4344.objecter dump_active .. 0 homeless 2024-09-16T03:42:30.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.665+0000 7f0284a16640 1 --2- 172.21.15.42:0/2914394776 >> v2:172.21.15.154:6812/254845342 conn(0x7f0264037810 0x7f0264039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:30.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.665+0000 7f0284a16640 10 monclient: get_auth_request con 0x7f0264037810 auth_method 0 2024-09-16T03:42:30.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.665+0000 7f0286c7a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:42:30.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.665+0000 7f0286c7a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:42:30.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.665+0000 7f0286c7a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:42:30.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.665+0000 7f0286c7a640 1 -- 172.21.15.42:0/2914394776 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0280132a20 con 0x7f02800a3d90 2024-09-16T03:42:30.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.665+0000 7f0284a16640 1 --2- 172.21.15.42:0/2914394776 >> v2:172.21.15.154:6812/254845342 conn(0x7f0264037810 0x7f0264039cd0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f0274009b50 tx=0x7f0274009240 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:30.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.665+0000 7f027d7fa640 10 client.4344.objecter ms_handle_connect 0x7f0264037810 2024-09-16T03:42:30.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.669+0000 7f0c83fff640 10 monclient: get_monmap_and_config 2024-09-16T03:42:30.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.669+0000 7f0c83fff640 10 monclient: build_initial_monmap 2024-09-16T03:42:30.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.669+0000 7f0c83fff640 10 monclient: monmap: 2024-09-16T03:42:30.687 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:42:30.687 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:30.687 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:42:30.675929+0000 2024-09-16T03:42:30.687 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:42:30.675929+0000 2024-09-16T03:42:30.687 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:42:30.687 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:30.688 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:42:30.688 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:42:30.688 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:42:30.688 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:42:30.688 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:42:30.688 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:30.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.673+0000 7f0c83fff640 1 Processor -- start 2024-09-16T03:42:30.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.673+0000 7f0c83fff640 1 -- start start 2024-09-16T03:42:30.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.673+0000 7f0c83fff640 10 monclient: init 2024-09-16T03:42:30.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.673+0000 7f0c83fff640 10 monclient: _reopen_session rank -1 2024-09-16T03:42:30.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.673+0000 7f0c83fff640 10 monclient: _add_conns ranks=[3,2,1,0,4] 2024-09-16T03:42:30.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.673+0000 7f0c83fff640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f0c7c024f20 0x7f0c7c025300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:30.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.673+0000 7f0c83fff640 10 monclient(hunting): picked mon.noname-a con 0x7f0c7c024f20 addr v2:172.21.15.42:3300/0 2024-09-16T03:42:30.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.673+0000 7f0c83fff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0c7c0247b0 0x7f0c7c025840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:30.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.673+0000 7f0c83fff640 10 monclient(hunting): picked mon.noname-b con 0x7f0c7c0247b0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:42:30.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.673+0000 7f0c82ffd640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f0c7c024f20 0x7f0c7c025300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:30.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.673+0000 7f0c82ffd640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f0c7c024f20 0x7f0c7c025300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:37812/0 (socket says 172.21.15.42:37812) 2024-09-16T03:42:30.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.673+0000 7f0c82ffd640 1 -- 172.21.15.42:0/3390083573 learned_addr learned my addr 172.21.15.42:0/3390083573 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:42:30.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.673+0000 7f0c83fff640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0c7c0b82b0 addr v1:172.21.15.154:6789/0 2024-09-16T03:42:30.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.673+0000 7f0c83fff640 1 -- 172.21.15.42:0/3390083573 --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c7c026c10 con 0x7f0c7c0b82b0 2024-09-16T03:42:30.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.673+0000 7f0c83fff640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:30.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.673+0000 7f0c83fff640 1 -- 172.21.15.42:0/3390083573 --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f0c7c027c20 con 0x7f0c7c024f20 2024-09-16T03:42:30.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.673+0000 7f0c83fff640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:30.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.673+0000 7f0c83fff640 1 -- 172.21.15.42:0/3390083573 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f0c7c025d80 con 0x7f0c7c0247b0 2024-09-16T03:42:30.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.673+0000 7f0c83fff640 10 monclient(hunting): _renew_subs 2024-09-16T03:42:30.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.673+0000 7f027d7fa640 1 -- 172.21.15.42:0/2914394776 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f0280132a20 con 0x7f02800a3d90 2024-09-16T03:42:30.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.673+0000 7f027d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:42:30.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.673+0000 7f027d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:42:30.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.673+0000 7f0c827fc640 1 --2- 172.21.15.42:0/3390083573 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0c7c0247b0 0x7f0c7c025840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:30.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.673+0000 7f0c83fff640 10 monclient(hunting): authenticate will time out at 997.647202s 2024-09-16T03:42:30.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.673+0000 7f0c82ffd640 10 monclient(hunting): get_auth_request con 0x7f0c7c024f20 auth_method 0 2024-09-16T03:42:30.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.673+0000 7f0c82ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:30.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.673+0000 7f0c82ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:30.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.673+0000 7f0c82ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:30.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.673+0000 7f0c827fc640 10 monclient(hunting): get_auth_request con 0x7f0c7c0247b0 auth_method 0 2024-09-16T03:42:30.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.673+0000 7f0c827fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:30.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.673+0000 7f0c827fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:30.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.673+0000 7f0c827fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:30.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.673+0000 7f0c82ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:30.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.673+0000 7f0c82ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:30.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.673+0000 7f0c82ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:30.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.673+0000 7f0c827fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:30.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.673+0000 7f0c827fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:30.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.673+0000 7f0c827fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:30.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.673+0000 7f0c82ffd640 10 monclient(hunting): handle_auth_done global_id 4347 payload 274 2024-09-16T03:42:30.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.673+0000 7f0c82ffd640 1 -- 172.21.15.42:0/3390083573 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0c7c0247b0 msgr2=0x7f0c7c025840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:42:30.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.673+0000 7f0c82ffd640 1 --2- 172.21.15.42:0/3390083573 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0c7c0247b0 0x7f0c7c025840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:30.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.673+0000 7f0c82ffd640 1 -- 172.21.15.42:0/3390083573 >> v1:172.21.15.154:6789/0 conn(0x7f0c7c0b82b0 legacy=0x7f0c7c0b8700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:30.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.673+0000 7f0c82ffd640 10 monclient: _finish_hunting 0 2024-09-16T03:42:30.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.673+0000 7f0c82ffd640 1 monclient: found mon.noname-a 2024-09-16T03:42:30.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.673+0000 7f0c82ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:42:30.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.673+0000 7f0c82ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:42:30.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.673+0000 7f0c82ffd640 1 -- 172.21.15.42:0/3390083573 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c7c027660 con 0x7f0c7c024f20 2024-09-16T03:42:30.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.677+0000 7f0c82ffd640 1 --2- 172.21.15.42:0/3390083573 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0c7c024f20 0x7f0c7c025300 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f0c74009870 tx=0x7f0c7401c3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ab77e37dea1e3b7 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:30.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.677+0000 7f0c81ffb640 1 -- 172.21.15.42:0/3390083573 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c740b0070 con 0x7f0c7c024f20 2024-09-16T03:42:30.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.677+0000 7f0c81ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:30.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.677+0000 7f0c81ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:42:30.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.677+0000 7f0c81ffb640 10 monclient: dump: 2024-09-16T03:42:30.698 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:30.698 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:30.698 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:30.698 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:30.698 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:30.698 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:30.699 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:30.699 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:30.699 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:30.699 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:30.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.677+0000 7f0c81ffb640 10 monclient: _finish_auth 0 2024-09-16T03:42:30.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.677+0000 7f0c81ffb640 10 monclient: _check_auth_tickets 2024-09-16T03:42:30.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.677+0000 7f0c81ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:42:30.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.677+0000 7f0c83fff640 5 monclient: authenticate success, global_id 4347 2024-09-16T03:42:30.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.677+0000 7f0c83fff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:42:30.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.677+0000 7f0c81ffb640 1 -- 172.21.15.42:0/3390083573 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c740ab600 con 0x7f0c7c024f20 2024-09-16T03:42:30.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.677+0000 7f0c81ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T03:42:30.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.677+0000 7f0c81ffb640 1 -- 172.21.15.42:0/3390083573 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c740ab9d0 con 0x7f0c7c024f20 2024-09-16T03:42:30.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.677+0000 7f0c81ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:30.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.677+0000 7f0c81ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:42:30.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.677+0000 7f0c81ffb640 10 monclient: dump: 2024-09-16T03:42:30.700 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:30.700 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:30.700 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:30.700 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:30.700 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:30.700 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:30.700 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:30.700 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:30.700 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:30.700 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:30.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.677+0000 7f0c83fff640 10 monclient: get_monmap_and_config success 2024-09-16T03:42:30.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.677+0000 7f0c83fff640 4 set_mon_vals no callback set 2024-09-16T03:42:30.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.681+0000 7f0c83fff640 10 monclient: shutdown 2024-09-16T03:42:30.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.681+0000 7f0c83fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:42:30.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.681+0000 7f0c83fff640 1 -- 172.21.15.42:0/3390083573 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0c7c024f20 msgr2=0x7f0c7c025300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:30.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.681+0000 7f0c83fff640 1 --2- 172.21.15.42:0/3390083573 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0c7c024f20 0x7f0c7c025300 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f0c74009870 tx=0x7f0c7401c3d0 comp rx=0 tx=0).stop 2024-09-16T03:42:30.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.681+0000 7f0c83fff640 1 -- 172.21.15.42:0/3390083573 shutdown_connections 2024-09-16T03:42:30.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.681+0000 7f0c83fff640 1 --2- 172.21.15.42:0/3390083573 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0c7c0247b0 0x7f0c7c025840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:30.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.681+0000 7f0c83fff640 1 --2- 172.21.15.42:0/3390083573 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0c7c024f20 0x7f0c7c025300 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:30.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.681+0000 7f0c83fff640 1 -- 172.21.15.42:0/3390083573 >> 172.21.15.42:0/3390083573 conn(0x7f0c7c006370 msgr2=0x7f0c7c006780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:42:30.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.681+0000 7f0c83fff640 1 -- 172.21.15.42:0/3390083573 shutdown_connections 2024-09-16T03:42:30.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.681+0000 7f0c83fff640 1 -- 172.21.15.42:0/3390083573 wait complete. 2024-09-16T03:42:30.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.681+0000 7f0c83fff640 10 monclient: build_initial_monmap 2024-09-16T03:42:30.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.681+0000 7f0c83fff640 10 monclient: monmap: 2024-09-16T03:42:30.701 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:42:30.701 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:42:30.702 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:42:30.702 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:42:30.702 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:42:30.702 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:30.702 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:42:30.702 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:42:30.702 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:42:30.702 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:30.702 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.681+0000 7f0c83fff640 1 librados: starting msgr at 2024-09-16T03:42:30.702 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.681+0000 7f0c83fff640 1 librados: starting objecter 2024-09-16T03:42:30.702 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.681+0000 7f0c83fff640 1 Processor -- start 2024-09-16T03:42:30.702 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.681+0000 7f0c83fff640 1 -- start start 2024-09-16T03:42:30.702 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.681+0000 7f0c83fff640 1 librados: setting wanted keys 2024-09-16T03:42:30.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.681+0000 7f0c83fff640 1 librados: calling monclient init 2024-09-16T03:42:30.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.681+0000 7f0c83fff640 10 monclient: init 2024-09-16T03:42:30.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.685+0000 7f0c83fff640 10 monclient: _reopen_session rank -1 2024-09-16T03:42:30.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.685+0000 7f0c83fff640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T03:42:30.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.685+0000 7f0c83fff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0c7c0247b0 0x7f0c7c1539b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:30.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.685+0000 7f0c83fff640 10 monclient(hunting): picked mon.noname-a con 0x7f0c7c0247b0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:42:30.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.685+0000 7f0c83fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0c7c024f20 0x7f0c7c0c2d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:30.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.685+0000 7f0c83fff640 10 monclient(hunting): picked mon.noname-b con 0x7f0c7c024f20 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:42:30.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.685+0000 7f0c83fff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0c7c0b82b0 0x7f0c7c0c3270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:30.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.685+0000 7f0c82ffd640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0c7c0247b0 0x7f0c7c1539b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:30.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.685+0000 7f0c82ffd640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0c7c0247b0 0x7f0c7c1539b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:37816/0 (socket says 172.21.15.42:37816) 2024-09-16T03:42:30.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.685+0000 7f0c837fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0c7c0b82b0 0x7f0c7c0c3270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:30.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.685+0000 7f0c837fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0c7c0b82b0 0x7f0c7c0c3270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:41534/0 (socket says 172.21.15.42:41534) 2024-09-16T03:42:30.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.685+0000 7f0c827fc640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0c7c024f20 0x7f0c7c0c2d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:30.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.685+0000 7f0c827fc640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0c7c024f20 0x7f0c7c0c2d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.42:42150/0 (socket says 172.21.15.42:42150) 2024-09-16T03:42:30.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.685+0000 7f0c83fff640 10 monclient(hunting): picked mon.noname-c con 0x7f0c7c0b82b0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:42:30.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.685+0000 7f0c82ffd640 1 -- 172.21.15.42:0/3267284723 learned_addr learned my addr 172.21.15.42:0/3267284723 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:42:30.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.685+0000 7f0c83fff640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:30.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.685+0000 7f0c83fff640 1 -- 172.21.15.42:0/3267284723 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f0c7c0c6650 con 0x7f0c7c0247b0 2024-09-16T03:42:30.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.685+0000 7f0c83fff640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:30.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.685+0000 7f0c83fff640 1 -- 172.21.15.42:0/3267284723 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f0c7c027940 con 0x7f0c7c024f20 2024-09-16T03:42:30.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.685+0000 7f0c83fff640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:30.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.685+0000 7f0c83fff640 1 -- 172.21.15.42:0/3267284723 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f0c7c0c67d0 con 0x7f0c7c0b82b0 2024-09-16T03:42:30.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.685+0000 7f0c83fff640 10 monclient(hunting): _renew_subs 2024-09-16T03:42:30.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.685+0000 7f0c83fff640 10 monclient(hunting): authenticate will time out at 997.657289s 2024-09-16T03:42:30.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.685+0000 7f0c82ffd640 10 monclient(hunting): get_auth_request con 0x7f0c7c0247b0 auth_method 0 2024-09-16T03:42:30.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.685+0000 7f0c82ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:30.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.685+0000 7f0c82ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:30.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.685+0000 7f0c82ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:30.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.685+0000 7f0c837fe640 10 monclient(hunting): get_auth_request con 0x7f0c7c0b82b0 auth_method 0 2024-09-16T03:42:30.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.685+0000 7f0c837fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:30.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.685+0000 7f0c837fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:30.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.685+0000 7f0c837fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:30.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.685+0000 7f0c827fc640 10 monclient(hunting): get_auth_request con 0x7f0c7c024f20 auth_method 0 2024-09-16T03:42:30.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.685+0000 7f0c827fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:30.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.685+0000 7f0c827fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:30.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.685+0000 7f0c827fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:30.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.685+0000 7f0c82ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:30.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.685+0000 7f0c82ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:30.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.685+0000 7f0c82ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:30.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.685+0000 7f0c837fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:30.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.685+0000 7f0c837fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:30.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.685+0000 7f0c837fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:30.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.685+0000 7f0c827fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:30.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.685+0000 7f0c827fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:30.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.685+0000 7f0c827fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:30.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.685+0000 7f0c82ffd640 10 monclient(hunting): handle_auth_done global_id 4350 payload 824 2024-09-16T03:42:30.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.685+0000 7f0c82ffd640 1 -- 172.21.15.42:0/3267284723 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0c7c0b82b0 msgr2=0x7f0c7c0c3270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:30.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.685+0000 7f0c82ffd640 1 --2- 172.21.15.42:0/3267284723 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0c7c0b82b0 0x7f0c7c0c3270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:30.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.685+0000 7f0c82ffd640 1 -- 172.21.15.42:0/3267284723 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0c7c024f20 msgr2=0x7f0c7c0c2d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:30.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.685+0000 7f0c82ffd640 1 --2- 172.21.15.42:0/3267284723 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0c7c024f20 0x7f0c7c0c2d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:30.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.685+0000 7f0c82ffd640 10 monclient: _finish_hunting 0 2024-09-16T03:42:30.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.685+0000 7f0c82ffd640 1 monclient: found mon.noname-a 2024-09-16T03:42:30.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.685+0000 7f0c82ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:42:30.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.685+0000 7f0c82ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:42:30.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.685+0000 7f0c82ffd640 1 -- 172.21.15.42:0/3267284723 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c7c0c1800 con 0x7f0c7c0247b0 2024-09-16T03:42:30.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.685+0000 7f0c82ffd640 1 --2- 172.21.15.42:0/3267284723 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0c7c0247b0 0x7f0c7c1539b0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f0c7401a000 tx=0x7f0c74019170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:30.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.685+0000 7f0c6bfff640 10 client.?.objecter ms_handle_connect 0x7f0c7c0247b0 2024-09-16T03:42:30.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.685+0000 7f0c6bfff640 10 client.?.objecter resend_mon_ops 2024-09-16T03:42:30.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.685+0000 7f0c6bfff640 1 -- 172.21.15.42:0/3267284723 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c740b0070 con 0x7f0c7c0247b0 2024-09-16T03:42:30.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.685+0000 7f0c6bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:30.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.685+0000 7f0c6bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:42:30.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.685+0000 7f0c6bfff640 10 monclient: dump: 2024-09-16T03:42:30.707 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:30.707 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:30.707 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:30.708 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:30.708 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:30.708 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:30.708 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:30.708 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:30.708 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:30.708 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:30.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.685+0000 7f0c6bfff640 10 monclient: _finish_auth 0 2024-09-16T03:42:30.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.685+0000 7f0c6bfff640 10 monclient: _check_auth_tickets 2024-09-16T03:42:30.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.685+0000 7f0c6bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:42:30.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.685+0000 7f0c6bfff640 1 -- 172.21.15.42:0/3267284723 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c740b1040 con 0x7f0c7c0247b0 2024-09-16T03:42:30.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.685+0000 7f0c6bfff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:42:30.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.685+0000 7f0c6bfff640 1 -- 172.21.15.42:0/3267284723 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c740aa810 con 0x7f0c7c0247b0 2024-09-16T03:42:30.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.685+0000 7f0c6bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:30.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.685+0000 7f0c6bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:42:30.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.685+0000 7f0c6bfff640 10 monclient: dump: 2024-09-16T03:42:30.709 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:30.709 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:30.709 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:30.709 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:30.709 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:30.709 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:30.709 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:30.709 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:30.709 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:30.709 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:30.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.685+0000 7f0c817fa640 4 set_mon_vals no callback set 2024-09-16T03:42:30.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.685+0000 7f0c83fff640 5 monclient: authenticate success, global_id 4350 2024-09-16T03:42:30.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.685+0000 7f0c83fff640 10 monclient: _renew_subs 2024-09-16T03:42:30.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.685+0000 7f0c83fff640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:42:30.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.685+0000 7f0c83fff640 1 -- 172.21.15.42:0/3267284723 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c7c02d3b0 con 0x7f0c7c0247b0 2024-09-16T03:42:30.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.685+0000 7f0c83fff640 10 client.4350.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:42:30.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.685+0000 7f0c83fff640 10 monclient: _renew_subs 2024-09-16T03:42:30.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.685+0000 7f0c83fff640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:42:30.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.685+0000 7f0c83fff640 1 -- 172.21.15.42:0/3267284723 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c7c024c10 con 0x7f0c7c0247b0 2024-09-16T03:42:30.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.685+0000 7f0c83fff640 1 librados: init done 2024-09-16T03:42:30.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.689+0000 7f0c6bfff640 1 -- 172.21.15.42:0/3267284723 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f0c740aa9b0 con 0x7f0c7c0247b0 2024-09-16T03:42:30.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.689+0000 7f0c6bfff640 1 --2- 172.21.15.42:0/3267284723 >> v2:172.21.15.154:6812/254845342 conn(0x7f0c70037810 0x7f0c70039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:30.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.689+0000 7f0c6bfff640 1 -- 172.21.15.42:0/3267284723 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 3534+0+0 (secure 0 0 0) 0x7f0c740e6170 con 0x7f0c7c0247b0 2024-09-16T03:42:30.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.689+0000 7f0c6bfff640 10 client.4350.objecter ms_dispatch 0x7f0c84000ba0 osd_map(19..19 src has 1..19) 2024-09-16T03:42:30.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.689+0000 7f0c6bfff640 3 client.4350.objecter handle_osd_map got epochs [19,19] > 0 2024-09-16T03:42:30.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.689+0000 7f0c6bfff640 3 client.4350.objecter handle_osd_map decoding full epoch 19 2024-09-16T03:42:30.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.689+0000 7f0c6bfff640 20 client.4350.objecter dump_active .. 0 homeless 2024-09-16T03:42:30.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.689+0000 7f0c83fff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:42:30.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.689+0000 7f0c83fff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:42:30.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.689+0000 7f0c83fff640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:42:30.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.689+0000 7f0c83fff640 1 -- 172.21.15.42:0/3267284723 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0c7c0b8700 con 0x7f0c7c0247b0 2024-09-16T03:42:30.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.689+0000 7f0c827fc640 1 --2- 172.21.15.42:0/3267284723 >> v2:172.21.15.154:6812/254845342 conn(0x7f0c70037810 0x7f0c70039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:30.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.689+0000 7f0c827fc640 10 monclient: get_auth_request con 0x7f0c70037810 auth_method 0 2024-09-16T03:42:30.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.689+0000 7f0c827fc640 1 --2- 172.21.15.42:0/3267284723 >> v2:172.21.15.154:6812/254845342 conn(0x7f0c70037810 0x7f0c70039cd0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f0c7c0c1a10 tx=0x7f0c78009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:30.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.689+0000 7f0c6bfff640 10 client.4350.objecter ms_handle_connect 0x7f0c70037810 2024-09-16T03:42:30.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.697+0000 7f0c6bfff640 1 -- 172.21.15.42:0/3267284723 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f0c74016050 con 0x7f0c7c0247b0 2024-09-16T03:42:30.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.697+0000 7f0c6bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:42:30.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.697+0000 7f0c6bfff640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:42:30.765 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.757+0000 7f13eabf4640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-16T03:42:30.765 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.757+0000 7f13eabf4640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-16T03:42:30.766 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.757+0000 7f13eabf4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:42:30.766 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.757+0000 7f13eabf4640 1 -- 172.21.15.42:0/3303871727 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f13e406f680 con 0x7f13e40a39b0 2024-09-16T03:42:30.766 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.761+0000 7f13e17fa640 1 -- 172.21.15.42:0/3303871727 <== mon.2 v2:172.21.15.42:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f13d8015fd0 con 0x7f13e40a39b0 2024-09-16T03:42:30.766 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.761+0000 7f13e17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-16T03:42:30.766 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.761+0000 7f13e17fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:42:30.766 INFO:teuthology.orchestra.run.smithi042.stdout:51539607558 2024-09-16T03:42:30.769 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.761+0000 7f13eabf4640 10 librados: watch_flush enter 2024-09-16T03:42:30.769 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.761+0000 7f13eabf4640 10 librados: watch_flush exit 2024-09-16T03:42:30.769 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.761+0000 7f13eabf4640 20 client.4286.objecter shutdown clearing up homeless session... 2024-09-16T03:42:30.770 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.761+0000 7f13eabf4640 10 client.4286.objecter successfully canceled tick 2024-09-16T03:42:30.770 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.761+0000 7f13eabf4640 1 -- 172.21.15.42:0/3303871727 >> v2:172.21.15.154:6812/254845342 conn(0x7f13c8037840 msgr2=0x7f13c8039d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:30.770 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.761+0000 7f13eabf4640 1 --2- 172.21.15.42:0/3303871727 >> v2:172.21.15.154:6812/254845342 conn(0x7f13c8037840 0x7f13c8039d00 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f13d0000c10 tx=0x7f13d0009040 comp rx=0 tx=0).stop 2024-09-16T03:42:30.770 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.761+0000 7f13eabf4640 10 monclient: shutdown 2024-09-16T03:42:30.770 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.761+0000 7f13eabf4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:42:30.770 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.761+0000 7f13eabf4640 1 -- 172.21.15.42:0/3303871727 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f13e40a39b0 msgr2=0x7f13e40a19d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:30.770 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.761+0000 7f13eabf4640 1 --2- 172.21.15.42:0/3303871727 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f13e40a39b0 0x7f13e40a19d0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f13d801a590 tx=0x7f13d801a720 comp rx=0 tx=0).stop 2024-09-16T03:42:30.770 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.761+0000 7f13eabf4640 1 -- 172.21.15.42:0/3303871727 shutdown_connections 2024-09-16T03:42:30.770 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.761+0000 7f13eabf4640 1 --2- 172.21.15.42:0/3303871727 >> v2:172.21.15.154:6812/254845342 conn(0x7f13c8037840 0x7f13c8039d00 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:30.770 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.761+0000 7f13eabf4640 1 --2- 172.21.15.42:0/3303871727 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f13e4137710 0x7f13e40a2450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:30.770 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.761+0000 7f13eabf4640 1 --2- 172.21.15.42:0/3303871727 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f13e40a3d90 0x7f13e40a1f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:30.770 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.761+0000 7f13eabf4640 1 --2- 172.21.15.42:0/3303871727 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f13e40a39b0 0x7f13e40a19d0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:30.770 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.761+0000 7f13eabf4640 1 -- 172.21.15.42:0/3303871727 >> 172.21.15.42:0/3303871727 conn(0x7f13e40a9fb0 msgr2=0x7f13e4136ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:42:30.770 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.761+0000 7f13eabf4640 1 -- 172.21.15.42:0/3303871727 shutdown_connections 2024-09-16T03:42:30.771 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.761+0000 7f13eabf4640 1 -- 172.21.15.42:0/3303871727 wait complete. 2024-09-16T03:42:30.771 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.761+0000 7f13eabf4640 1 librados: shutdown 2024-09-16T03:42:30.785 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.777+0000 7fd86fc78640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-16T03:42:30.785 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.777+0000 7fd86fc78640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-16T03:42:30.785 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.777+0000 7fd86fc78640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:42:30.785 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.777+0000 7fd86fc78640 1 -- 172.21.15.42:0/1589220242 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7fd868068e40 con 0x7fd8680a6bc0 2024-09-16T03:42:30.786 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.777+0000 7fd85e7fc640 1 -- 172.21.15.42:0/1589220242 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fd85801ad20 con 0x7fd8680a6bc0 2024-09-16T03:42:30.786 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.777+0000 7fd85e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-16T03:42:30.786 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.777+0000 7fd85e7fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:42:30.786 INFO:teuthology.orchestra.run.smithi042.stdout:47244640260 2024-09-16T03:42:30.789 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.781+0000 7fd86fc78640 10 librados: watch_flush enter 2024-09-16T03:42:30.789 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.781+0000 7fd86fc78640 10 librados: watch_flush exit 2024-09-16T03:42:30.789 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.781+0000 7fd86fc78640 20 client.4320.objecter shutdown clearing up homeless session... 2024-09-16T03:42:30.789 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.781+0000 7fd86fc78640 10 client.4320.objecter successfully canceled tick 2024-09-16T03:42:30.789 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.781+0000 7f4268900640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-16T03:42:30.789 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.781+0000 7f4268900640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-16T03:42:30.789 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.781+0000 7f4268900640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:42:30.790 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.781+0000 7f4268900640 1 -- 172.21.15.42:0/2167052642 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f42640a8c50 con 0x7f42640a1090 2024-09-16T03:42:30.790 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.781+0000 7fd86fc78640 1 -- 172.21.15.42:0/1589220242 >> v2:172.21.15.154:6812/254845342 conn(0x7fd84c037810 msgr2=0x7fd84c039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:30.790 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.781+0000 7fd86fc78640 1 --2- 172.21.15.42:0/1589220242 >> v2:172.21.15.154:6812/254845342 conn(0x7fd84c037810 0x7fd84c039cd0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fd8640ad000 tx=0x7fd8640b2040 comp rx=0 tx=0).stop 2024-09-16T03:42:30.790 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.781+0000 7fd86fc78640 10 monclient: shutdown 2024-09-16T03:42:30.790 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.781+0000 7fd86fc78640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:42:30.790 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.781+0000 7fd86fc78640 1 -- 172.21.15.42:0/1589220242 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd8680a6bc0 msgr2=0x7fd86813b4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:30.790 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.781+0000 7fd86fc78640 1 --2- 172.21.15.42:0/1589220242 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd8680a6bc0 0x7fd86813b4f0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fd85800c780 tx=0x7fd85800cc40 comp rx=0 tx=0).stop 2024-09-16T03:42:30.790 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.781+0000 7fd86fc78640 1 -- 172.21.15.42:0/1589220242 shutdown_connections 2024-09-16T03:42:30.791 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.781+0000 7fd86fc78640 1 --2- 172.21.15.42:0/1589220242 >> v2:172.21.15.154:6812/254845342 conn(0x7fd84c037810 0x7fd84c039cd0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:30.791 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.781+0000 7fd86fc78640 1 --2- 172.21.15.42:0/1589220242 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd86813c2f0 0x7fd868120ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:30.791 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.781+0000 7fd86fc78640 1 --2- 172.21.15.42:0/1589220242 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd8680a6bc0 0x7fd86813b4f0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:30.791 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.781+0000 7fd86fc78640 1 --2- 172.21.15.42:0/1589220242 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd8680a67e0 0x7fd8681cf760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:30.791 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.781+0000 7fd86fc78640 1 -- 172.21.15.42:0/1589220242 >> 172.21.15.42:0/1589220242 conn(0x7fd8680a7ed0 msgr2=0x7fd86813e6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:42:30.791 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.781+0000 7f4252ffd640 1 -- 172.21.15.42:0/2167052642 <== mon.2 v2:172.21.15.42:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f425c01c580 con 0x7f42640a1090 2024-09-16T03:42:30.791 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.781+0000 7f4252ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-16T03:42:30.791 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.781+0000 7f4252ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:42:30.791 INFO:teuthology.orchestra.run.smithi042.stdout:47244640260 2024-09-16T03:42:30.792 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.785+0000 7fd86fc78640 1 -- 172.21.15.42:0/1589220242 shutdown_connections 2024-09-16T03:42:30.792 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.785+0000 7fd86fc78640 1 -- 172.21.15.42:0/1589220242 wait complete. 2024-09-16T03:42:30.792 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.785+0000 7fd86fc78640 1 librados: shutdown 2024-09-16T03:42:30.793 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607558 got 51539607558 for osd.5 2024-09-16T03:42:30.793 DEBUG:teuthology.parallel:result is None 2024-09-16T03:42:30.794 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.785+0000 7f4268900640 10 librados: watch_flush enter 2024-09-16T03:42:30.794 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.785+0000 7f4268900640 10 librados: watch_flush exit 2024-09-16T03:42:30.795 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.785+0000 7f4268900640 20 client.4289.objecter shutdown clearing up homeless session... 2024-09-16T03:42:30.795 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.785+0000 7f4268900640 10 client.4289.objecter successfully canceled tick 2024-09-16T03:42:30.795 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.785+0000 7f4268900640 1 -- 172.21.15.42:0/2167052642 >> v2:172.21.15.154:6812/254845342 conn(0x7f4244037860 msgr2=0x7f4244039d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:30.795 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.785+0000 7f4268900640 1 --2- 172.21.15.42:0/2167052642 >> v2:172.21.15.154:6812/254845342 conn(0x7f4244037860 0x7f4244039d20 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f425401cc00 tx=0x7f42540a7070 comp rx=0 tx=0).stop 2024-09-16T03:42:30.795 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.785+0000 7f4268900640 10 monclient: shutdown 2024-09-16T03:42:30.795 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.785+0000 7f4268900640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:42:30.795 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.785+0000 7f4268900640 1 -- 172.21.15.42:0/2167052642 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f42640a1090 msgr2=0x7f42641d5ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:30.795 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.785+0000 7f4268900640 1 --2- 172.21.15.42:0/2167052642 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f42640a1090 0x7f42641d5ac0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f42640a7a80 tx=0x7f425c00ec10 comp rx=0 tx=0).stop 2024-09-16T03:42:30.795 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.785+0000 7f4268900640 1 -- 172.21.15.42:0/2167052642 shutdown_connections 2024-09-16T03:42:30.795 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.785+0000 7f4268900640 1 --2- 172.21.15.42:0/2167052642 >> v2:172.21.15.154:6812/254845342 conn(0x7f4244037860 0x7f4244039d20 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:30.795 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.785+0000 7f4268900640 1 --2- 172.21.15.42:0/2167052642 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f42640a1090 0x7f42641d5ac0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:30.795 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.785+0000 7f4268900640 1 --2- 172.21.15.42:0/2167052642 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f42640a0710 0x7f42640a0b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:30.795 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.785+0000 7f4268900640 1 --2- 172.21.15.42:0/2167052642 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f424c003a10 0x7f42640a01d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:30.795 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.785+0000 7f4268900640 1 -- 172.21.15.42:0/2167052642 >> 172.21.15.42:0/2167052642 conn(0x7f42640a9fb0 msgr2=0x7f42640a9370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:42:30.796 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.785+0000 7f4268900640 1 -- 172.21.15.42:0/2167052642 shutdown_connections 2024-09-16T03:42:30.796 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.785+0000 7f4268900640 1 -- 172.21.15.42:0/2167052642 wait complete. 2024-09-16T03:42:30.796 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.785+0000 7f4268900640 1 librados: shutdown 2024-09-16T03:42:30.809 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640260 got 47244640260 for osd.3 2024-09-16T03:42:30.809 DEBUG:teuthology.parallel:result is None 2024-09-16T03:42:30.816 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640260 got 47244640260 for osd.2 2024-09-16T03:42:30.817 DEBUG:teuthology.parallel:result is None 2024-09-16T03:42:30.832 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.825+0000 7f9a0a895640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-16T03:42:30.832 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.825+0000 7f9a0a895640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-16T03:42:30.832 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.825+0000 7f9a0a895640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:42:30.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.825+0000 7f9a0a895640 1 -- 172.21.15.42:0/1918499733 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f9a0409db30 con 0x7f9a040a3d90 2024-09-16T03:42:30.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.825+0000 7f9a00ff9640 1 -- 172.21.15.42:0/1918499733 <== mon.2 v2:172.21.15.42:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f99f4003cd0 con 0x7f9a040a3d90 2024-09-16T03:42:30.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.825+0000 7f9a00ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-16T03:42:30.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.825+0000 7f9a00ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:42:30.833 INFO:teuthology.orchestra.run.smithi042.stdout:47244640260 2024-09-16T03:42:30.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.829+0000 7f9a0a895640 10 librados: watch_flush enter 2024-09-16T03:42:30.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.829+0000 7f9a0a895640 10 librados: watch_flush exit 2024-09-16T03:42:30.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.829+0000 7f9a0a895640 20 client.4280.objecter shutdown clearing up homeless session... 2024-09-16T03:42:30.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.829+0000 7f9a0a895640 10 client.4280.objecter successfully canceled tick 2024-09-16T03:42:30.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.829+0000 7f9a0a895640 1 -- 172.21.15.42:0/1918499733 >> v2:172.21.15.154:6812/254845342 conn(0x7f99f0037810 msgr2=0x7f99f0039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:30.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.829+0000 7f9a0a895640 1 --2- 172.21.15.42:0/1918499733 >> v2:172.21.15.154:6812/254845342 conn(0x7f99f0037810 0x7f99f0039cd0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f9a041370e0 tx=0x7f99f8008040 comp rx=0 tx=0).stop 2024-09-16T03:42:30.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.829+0000 7f9a0a895640 10 monclient: shutdown 2024-09-16T03:42:30.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.829+0000 7f9a0a895640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:42:30.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.829+0000 7f9a0a895640 1 -- 172.21.15.42:0/1918499733 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9a040a3d90 msgr2=0x7f9a0409c940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:30.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.829+0000 7f9a0a895640 1 --2- 172.21.15.42:0/1918499733 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9a040a3d90 0x7f9a0409c940 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f99f400f640 tx=0x7f99f400fb00 comp rx=0 tx=0).stop 2024-09-16T03:42:30.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.829+0000 7f9a0a895640 1 -- 172.21.15.42:0/1918499733 shutdown_connections 2024-09-16T03:42:30.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.829+0000 7f9a0a895640 1 --2- 172.21.15.42:0/1918499733 >> v2:172.21.15.154:6812/254845342 conn(0x7f99f0037810 0x7f99f0039cd0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:30.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.829+0000 7f9a0a895640 1 --2- 172.21.15.42:0/1918499733 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9a04130390 0x7f9a04130820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:30.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.829+0000 7f9a0a895640 1 --2- 172.21.15.42:0/1918499733 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9a0409ce80 0x7f9a0409d2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:30.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.829+0000 7f9a0a895640 1 --2- 172.21.15.42:0/1918499733 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9a040a3d90 0x7f9a0409c940 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:30.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.829+0000 7f9a0a895640 1 -- 172.21.15.42:0/1918499733 >> 172.21.15.42:0/1918499733 conn(0x7f9a040a9fb0 msgr2=0x7f9a04132b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:42:30.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.829+0000 7f9a0a895640 1 -- 172.21.15.42:0/1918499733 shutdown_connections 2024-09-16T03:42:30.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.829+0000 7f9a0a895640 1 -- 172.21.15.42:0/1918499733 wait complete. 2024-09-16T03:42:30.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.829+0000 7f9a0a895640 1 librados: shutdown 2024-09-16T03:42:30.855 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640260 got 47244640260 for osd.0 2024-09-16T03:42:30.856 DEBUG:teuthology.parallel:result is None 2024-09-16T03:42:30.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.901+0000 7f0286c7a640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-16T03:42:30.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.901+0000 7f0286c7a640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-16T03:42:30.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.901+0000 7f0286c7a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:42:30.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.901+0000 7f0286c7a640 1 -- 172.21.15.42:0/2914394776 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f0280137000 con 0x7f02800a3d90 2024-09-16T03:42:30.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.901+0000 7f027d7fa640 1 -- 172.21.15.42:0/2914394776 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f028009cd20 con 0x7f02800a3d90 2024-09-16T03:42:30.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.901+0000 7f027d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-16T03:42:30.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.901+0000 7f027d7fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:42:30.909 INFO:teuthology.orchestra.run.smithi042.stdout:47244640262 2024-09-16T03:42:30.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.901+0000 7f0286c7a640 10 librados: watch_flush enter 2024-09-16T03:42:30.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.905+0000 7f0286c7a640 10 librados: watch_flush exit 2024-09-16T03:42:30.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.905+0000 7f0286c7a640 20 client.4344.objecter shutdown clearing up homeless session... 2024-09-16T03:42:30.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.905+0000 7f0286c7a640 10 client.4344.objecter successfully canceled tick 2024-09-16T03:42:30.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.905+0000 7f0286c7a640 1 -- 172.21.15.42:0/2914394776 >> v2:172.21.15.154:6812/254845342 conn(0x7f0264037810 msgr2=0x7f0264039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:30.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.905+0000 7f0286c7a640 1 --2- 172.21.15.42:0/2914394776 >> v2:172.21.15.154:6812/254845342 conn(0x7f0264037810 0x7f0264039cd0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f0274009b50 tx=0x7f0274009240 comp rx=0 tx=0).stop 2024-09-16T03:42:30.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.905+0000 7f0286c7a640 10 monclient: shutdown 2024-09-16T03:42:30.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.905+0000 7f0286c7a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:42:30.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.905+0000 7f0286c7a640 1 -- 172.21.15.42:0/2914394776 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f02800a3d90 msgr2=0x7f028013b5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:30.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.905+0000 7f0286c7a640 1 --2- 172.21.15.42:0/2914394776 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f02800a3d90 0x7f028013b5d0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f026c00e930 tx=0x7f026c00edf0 comp rx=0 tx=0).stop 2024-09-16T03:42:30.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.905+0000 7f0286c7a640 1 -- 172.21.15.42:0/2914394776 shutdown_connections 2024-09-16T03:42:30.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.905+0000 7f0286c7a640 1 --2- 172.21.15.42:0/2914394776 >> v2:172.21.15.154:6812/254845342 conn(0x7f0264037810 0x7f0264039cd0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:30.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.905+0000 7f0286c7a640 1 --2- 172.21.15.42:0/2914394776 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0280137540 0x7f028013bb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:30.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.905+0000 7f0286c7a640 1 --2- 172.21.15.42:0/2914394776 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f02800a3d90 0x7f028013b5d0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:30.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.905+0000 7f0286c7a640 1 --2- 172.21.15.42:0/2914394776 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f02800a39b0 0x7f028013b090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:30.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.905+0000 7f0286c7a640 1 -- 172.21.15.42:0/2914394776 >> 172.21.15.42:0/2914394776 conn(0x7f02800a9fb0 msgr2=0x7f0280131980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:42:30.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.905+0000 7f0286c7a640 1 -- 172.21.15.42:0/2914394776 shutdown_connections 2024-09-16T03:42:30.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.905+0000 7f0286c7a640 1 -- 172.21.15.42:0/2914394776 wait complete. 2024-09-16T03:42:30.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.905+0000 7f0286c7a640 1 librados: shutdown 2024-09-16T03:42:30.925 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640262 got 47244640262 for osd.1 2024-09-16T03:42:30.925 DEBUG:teuthology.parallel:result is None 2024-09-16T03:42:30.940 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.933+0000 7f0c83fff640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-16T03:42:30.940 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.933+0000 7f0c83fff640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-16T03:42:30.940 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.933+0000 7f0c83fff640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:42:30.940 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.933+0000 7f0c83fff640 1 -- 172.21.15.42:0/3267284723 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f0c7c0bf510 con 0x7f0c7c0247b0 2024-09-16T03:42:30.940 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.933+0000 7f0c6bfff640 1 -- 172.21.15.42:0/3267284723 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f0c740c3030 con 0x7f0c7c0247b0 2024-09-16T03:42:30.941 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.933+0000 7f0c6bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-16T03:42:30.941 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.933+0000 7f0c6bfff640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:42:30.941 INFO:teuthology.orchestra.run.smithi042.stdout:47244640260 2024-09-16T03:42:30.943 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.937+0000 7f0c83fff640 10 librados: watch_flush enter 2024-09-16T03:42:30.943 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.937+0000 7f0c83fff640 10 librados: watch_flush exit 2024-09-16T03:42:30.943 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.937+0000 7f0c83fff640 20 client.4350.objecter shutdown clearing up homeless session... 2024-09-16T03:42:30.944 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.937+0000 7f0c83fff640 10 client.4350.objecter successfully canceled tick 2024-09-16T03:42:30.944 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.937+0000 7f0c83fff640 1 -- 172.21.15.42:0/3267284723 >> v2:172.21.15.154:6812/254845342 conn(0x7f0c70037810 msgr2=0x7f0c70039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:30.944 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.937+0000 7f0c83fff640 1 --2- 172.21.15.42:0/3267284723 >> v2:172.21.15.154:6812/254845342 conn(0x7f0c70037810 0x7f0c70039cd0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f0c7c0c1a10 tx=0x7f0c78009190 comp rx=0 tx=0).stop 2024-09-16T03:42:30.944 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.937+0000 7f0c83fff640 10 monclient: shutdown 2024-09-16T03:42:30.944 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.937+0000 7f0c83fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:42:30.944 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.937+0000 7f0c83fff640 1 -- 172.21.15.42:0/3267284723 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0c7c0247b0 msgr2=0x7f0c7c1539b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:30.944 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.937+0000 7f0c83fff640 1 --2- 172.21.15.42:0/3267284723 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0c7c0247b0 0x7f0c7c1539b0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f0c7401a000 tx=0x7f0c74019170 comp rx=0 tx=0).stop 2024-09-16T03:42:30.944 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.937+0000 7f0c83fff640 1 -- 172.21.15.42:0/3267284723 shutdown_connections 2024-09-16T03:42:30.944 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.937+0000 7f0c83fff640 1 --2- 172.21.15.42:0/3267284723 >> v2:172.21.15.154:6812/254845342 conn(0x7f0c70037810 0x7f0c70039cd0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:30.944 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.937+0000 7f0c83fff640 1 --2- 172.21.15.42:0/3267284723 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0c7c0b82b0 0x7f0c7c0c3270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:30.944 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.937+0000 7f0c83fff640 1 --2- 172.21.15.42:0/3267284723 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0c7c024f20 0x7f0c7c0c2d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:30.944 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.937+0000 7f0c83fff640 1 --2- 172.21.15.42:0/3267284723 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0c7c0247b0 0x7f0c7c1539b0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:30.944 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.937+0000 7f0c83fff640 1 -- 172.21.15.42:0/3267284723 >> 172.21.15.42:0/3267284723 conn(0x7f0c7c006370 msgr2=0x7f0c7c0bb360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:42:30.944 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.937+0000 7f0c83fff640 1 -- 172.21.15.42:0/3267284723 shutdown_connections 2024-09-16T03:42:30.945 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.937+0000 7f0c83fff640 1 -- 172.21.15.42:0/3267284723 wait complete. 2024-09-16T03:42:30.945 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:30.937+0000 7f0c83fff640 1 librados: shutdown 2024-09-16T03:42:30.958 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640260 got 47244640260 for osd.4 2024-09-16T03:42:30.958 DEBUG:teuthology.parallel:result is None 2024-09-16T03:42:30.958 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-16T03:42:30.959 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T03:42:31.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.113+0000 7f23378ad640 10 monclient: get_monmap_and_config 2024-09-16T03:42:31.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.113+0000 7f23378ad640 10 monclient: build_initial_monmap 2024-09-16T03:42:31.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.113+0000 7f23378ad640 10 monclient: monmap: 2024-09-16T03:42:31.121 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:42:31.121 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:31.121 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:42:31.119089+0000 2024-09-16T03:42:31.121 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:42:31.119089+0000 2024-09-16T03:42:31.121 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:42:31.122 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:31.122 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:42:31.122 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:42:31.122 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:42:31.122 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:42:31.122 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:42:31.122 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:31.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.113+0000 7f23378ad640 1 Processor -- start 2024-09-16T03:42:31.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.113+0000 7f23378ad640 1 -- start start 2024-09-16T03:42:31.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.113+0000 7f23378ad640 10 monclient: init 2024-09-16T03:42:31.125 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.117+0000 7f23378ad640 10 monclient: _reopen_session rank -1 2024-09-16T03:42:31.125 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.117+0000 7f23378ad640 10 monclient: _add_conns ranks=[3,2,4,0,1] 2024-09-16T03:42:31.125 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.117+0000 7f23378ad640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f23300a1330 0x7f23300a1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:31.125 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.117+0000 7f23378ad640 10 monclient(hunting): picked mon.noname-a con 0x7f23300a1330 addr v2:172.21.15.42:3300/0 2024-09-16T03:42:31.125 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.117+0000 7f23378ad640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f233006dac0 0x7f23300a1c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:31.125 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.117+0000 7f23378ad640 10 monclient(hunting): picked mon.noname-b con 0x7f233006dac0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:42:31.125 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.117+0000 7f23378ad640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f2330135100 0x7f2330135550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:31.125 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.117+0000 7f23378ad640 10 monclient(hunting): picked mon.noname-c con 0x7f2330135100 addr v2:172.21.15.154:3300/0 2024-09-16T03:42:31.125 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.117+0000 7f23378ad640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:31.125 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.117+0000 7f23378ad640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f233009c7d0 con 0x7f23300a1330 2024-09-16T03:42:31.125 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.117+0000 7f23378ad640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:31.125 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.117+0000 7f23378ad640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f233009c090 con 0x7f2330135100 2024-09-16T03:42:31.125 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.117+0000 7f23378ad640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:31.126 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.117+0000 7f23378ad640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f233009f610 con 0x7f233006dac0 2024-09-16T03:42:31.126 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.117+0000 7f23378ad640 10 monclient(hunting): _renew_subs 2024-09-16T03:42:31.126 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.117+0000 7f23378ad640 10 monclient(hunting): authenticate will time out at 998.088445s 2024-09-16T03:42:31.126 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.117+0000 7f2335649640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f23300a1330 0x7f23300a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:31.126 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.117+0000 7f2334e48640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f233006dac0 0x7f23300a1c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:31.126 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.117+0000 7f2335649640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f23300a1330 0x7f23300a1710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:37830/0 (socket says 172.21.15.42:37830) 2024-09-16T03:42:31.126 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.117+0000 7f2334e48640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f233006dac0 0x7f23300a1c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:41536/0 (socket says 172.21.15.42:41536) 2024-09-16T03:42:31.126 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.117+0000 7f2335649640 1 -- 172.21.15.42:0/3574767581 learned_addr learned my addr 172.21.15.42:0/3574767581 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:42:31.126 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.117+0000 7f2335649640 10 monclient(hunting): get_auth_request con 0x7f23300a1330 auth_method 0 2024-09-16T03:42:31.126 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.117+0000 7f2335649640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:31.126 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.117+0000 7f2335649640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:31.126 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.117+0000 7f2335649640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:31.126 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.117+0000 7f2334e48640 10 monclient(hunting): get_auth_request con 0x7f233006dac0 auth_method 0 2024-09-16T03:42:31.126 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.117+0000 7f2334e48640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:31.126 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.117+0000 7f2334e48640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:31.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.117+0000 7f2334e48640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:31.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.117+0000 7f2335649640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:31.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.117+0000 7f2335649640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:31.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.117+0000 7f2335649640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:31.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.117+0000 7f2334e48640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:31.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.117+0000 7f2334e48640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:31.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.117+0000 7f2334e48640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:31.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.117+0000 7f2335e4a640 1 --2- 172.21.15.42:0/3574767581 >> v2:172.21.15.154:3300/0 conn(0x7f2330135100 0x7f2330135550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:31.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.117+0000 7f2335e4a640 10 monclient(hunting): get_auth_request con 0x7f2330135100 auth_method 0 2024-09-16T03:42:31.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.117+0000 7f2335e4a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:31.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.117+0000 7f2335e4a640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:31.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.117+0000 7f2335e4a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:31.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.117+0000 7f2335649640 10 monclient(hunting): handle_auth_done global_id 4353 payload 274 2024-09-16T03:42:31.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.117+0000 7f2335649640 1 -- 172.21.15.42:0/3574767581 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f233006dac0 msgr2=0x7f23300a1c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:42:31.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.117+0000 7f2335649640 1 --2- 172.21.15.42:0/3574767581 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f233006dac0 0x7f23300a1c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:31.128 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.117+0000 7f2335649640 1 -- 172.21.15.42:0/3574767581 >> v2:172.21.15.154:3300/0 conn(0x7f2330135100 msgr2=0x7f2330135550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:42:31.128 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.117+0000 7f2335649640 1 --2- 172.21.15.42:0/3574767581 >> v2:172.21.15.154:3300/0 conn(0x7f2330135100 0x7f2330135550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:31.128 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.117+0000 7f2335649640 10 monclient: _finish_hunting 0 2024-09-16T03:42:31.128 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.117+0000 7f2335649640 1 monclient: found mon.noname-a 2024-09-16T03:42:31.128 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.117+0000 7f2335649640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:42:31.128 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.117+0000 7f2335649640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:42:31.128 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.117+0000 7f2335649640 1 -- 172.21.15.42:0/3574767581 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f233009f8f0 con 0x7f23300a1330 2024-09-16T03:42:31.128 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.117+0000 7f2334e48640 1 --2- 172.21.15.42:0/3574767581 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f233006dac0 0x7f23300a1c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:42:31.133 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.121+0000 7f2335649640 1 --2- 172.21.15.42:0/3574767581 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f23300a1330 0x7f23300a1710 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f233009bb00 tx=0x7f232c01c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e64c4fe11ef2f867 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:31.133 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.121+0000 7f2327fff640 1 -- 172.21.15.42:0/3574767581 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f232c0b0070 con 0x7f23300a1330 2024-09-16T03:42:31.133 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.121+0000 7f2327fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:31.133 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.121+0000 7f2327fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:42:31.133 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.121+0000 7f2327fff640 10 monclient: dump: 2024-09-16T03:42:31.133 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:31.133 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:31.133 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:31.133 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:31.133 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:31.133 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:31.133 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:31.133 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:31.134 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:31.134 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:31.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.121+0000 7f2327fff640 10 monclient: _finish_auth 0 2024-09-16T03:42:31.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.121+0000 7f2327fff640 10 monclient: _check_auth_tickets 2024-09-16T03:42:31.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.121+0000 7f2327fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:42:31.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.121+0000 7f23378ad640 5 monclient: authenticate success, global_id 4353 2024-09-16T03:42:31.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.121+0000 7f23378ad640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:42:31.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.121+0000 7f2327fff640 1 -- 172.21.15.42:0/3574767581 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f232c0ab600 con 0x7f23300a1330 2024-09-16T03:42:31.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.121+0000 7f2327fff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:42:31.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.121+0000 7f23378ad640 10 monclient: get_monmap_and_config success 2024-09-16T03:42:31.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.121+0000 7f23378ad640 4 set_mon_vals no callback set 2024-09-16T03:42:31.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.121+0000 7f2327fff640 1 -- 172.21.15.42:0/3574767581 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f232c0ab9d0 con 0x7f23300a1330 2024-09-16T03:42:31.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.121+0000 7f2327fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:31.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.121+0000 7f2327fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:42:31.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.121+0000 7f2327fff640 10 monclient: dump: 2024-09-16T03:42:31.135 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:31.135 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:31.135 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:31.135 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:31.135 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:31.135 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:31.135 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:31.135 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:31.135 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:31.135 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:31.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.121+0000 7f23378ad640 10 monclient: shutdown 2024-09-16T03:42:31.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.121+0000 7f23378ad640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:42:31.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.121+0000 7f23378ad640 1 -- 172.21.15.42:0/3574767581 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f23300a1330 msgr2=0x7f23300a1710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:31.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.121+0000 7f23378ad640 1 --2- 172.21.15.42:0/3574767581 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f23300a1330 0x7f23300a1710 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f233009bb00 tx=0x7f232c01c500 comp rx=0 tx=0).stop 2024-09-16T03:42:31.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.121+0000 7f23378ad640 1 -- 172.21.15.42:0/3574767581 shutdown_connections 2024-09-16T03:42:31.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.121+0000 7f23378ad640 1 --2- 172.21.15.42:0/3574767581 >> v2:172.21.15.154:3300/0 conn(0x7f2330135100 0x7f2330135550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:31.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.121+0000 7f23378ad640 1 --2- 172.21.15.42:0/3574767581 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f233006dac0 0x7f23300a1c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:31.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.121+0000 7f23378ad640 1 --2- 172.21.15.42:0/3574767581 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f23300a1330 0x7f23300a1710 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:31.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.121+0000 7f23378ad640 1 -- 172.21.15.42:0/3574767581 >> 172.21.15.42:0/3574767581 conn(0x7f23300a2420 msgr2=0x7f23300a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:42:31.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.121+0000 7f23378ad640 1 -- 172.21.15.42:0/3574767581 shutdown_connections 2024-09-16T03:42:31.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.121+0000 7f23378ad640 1 -- 172.21.15.42:0/3574767581 wait complete. 2024-09-16T03:42:31.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.121+0000 7f23378ad640 10 monclient: build_initial_monmap 2024-09-16T03:42:31.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.121+0000 7f23378ad640 10 monclient: monmap: 2024-09-16T03:42:31.136 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:42:31.136 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:42:31.136 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:42:31.136 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:42:31.136 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:42:31.136 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:31.137 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:42:31.137 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:42:31.137 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:42:31.137 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:31.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.121+0000 7f23378ad640 1 librados: starting msgr at 2024-09-16T03:42:31.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.121+0000 7f23378ad640 1 librados: starting objecter 2024-09-16T03:42:31.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.121+0000 7f23378ad640 1 Processor -- start 2024-09-16T03:42:31.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.121+0000 7f23378ad640 1 -- start start 2024-09-16T03:42:31.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.121+0000 7f23378ad640 1 librados: setting wanted keys 2024-09-16T03:42:31.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.121+0000 7f23378ad640 1 librados: calling monclient init 2024-09-16T03:42:31.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.121+0000 7f23378ad640 10 monclient: init 2024-09-16T03:42:31.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.121+0000 7f23378ad640 10 monclient: _reopen_session rank -1 2024-09-16T03:42:31.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.121+0000 7f23378ad640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:42:31.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.121+0000 7f23378ad640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2330135100 0x7f23301ccf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:31.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.121+0000 7f23378ad640 10 monclient(hunting): picked mon.noname-b con 0x7f2330135100 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:42:31.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.121+0000 7f23378ad640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f23301324d0 0x7f2330132950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:31.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.121+0000 7f23378ad640 10 monclient(hunting): picked mon.noname-a con 0x7f23301324d0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:42:31.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.121+0000 7f23378ad640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f23301394f0 0x7f2330139950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:31.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.121+0000 7f23378ad640 10 monclient(hunting): picked mon.noname-c con 0x7f23301394f0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:42:31.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.121+0000 7f23378ad640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:31.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.121+0000 7f23378ad640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f233009eef0 con 0x7f23301324d0 2024-09-16T03:42:31.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.121+0000 7f23378ad640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:31.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.121+0000 7f23378ad640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f233009ed70 con 0x7f2330135100 2024-09-16T03:42:31.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.121+0000 7f23378ad640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:31.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.121+0000 7f23378ad640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f233013eb70 con 0x7f23301394f0 2024-09-16T03:42:31.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.121+0000 7f23378ad640 10 monclient(hunting): _renew_subs 2024-09-16T03:42:31.140 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.121+0000 7f23378ad640 10 monclient(hunting): authenticate will time out at 998.095906s 2024-09-16T03:42:31.140 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.125+0000 7f2334e48640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f23301324d0 0x7f2330132950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:31.140 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.125+0000 7f2335e4a640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f23301394f0 0x7f2330139950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:31.140 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.125+0000 7f2334e48640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f23301324d0 0x7f2330132950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:37838/0 (socket says 172.21.15.42:37838) 2024-09-16T03:42:31.140 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.125+0000 7f2334e48640 1 -- 172.21.15.42:0/3847095089 learned_addr learned my addr 172.21.15.42:0/3847095089 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:42:31.140 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.125+0000 7f2334e48640 10 monclient(hunting): get_auth_request con 0x7f23301324d0 auth_method 0 2024-09-16T03:42:31.140 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.125+0000 7f2334e48640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:31.140 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.125+0000 7f2334e48640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:31.140 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.125+0000 7f2334e48640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:31.140 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.125+0000 7f2335e4a640 10 monclient(hunting): get_auth_request con 0x7f23301394f0 auth_method 0 2024-09-16T03:42:31.140 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.125+0000 7f2335e4a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:31.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.125+0000 7f2335e4a640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:31.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.125+0000 7f2335e4a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:31.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.125+0000 7f2334e48640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:31.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.125+0000 7f2334e48640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:31.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.125+0000 7f2334e48640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:31.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.125+0000 7f2335e4a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:31.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.125+0000 7f2335e4a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:31.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.125+0000 7f2335e4a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:31.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.125+0000 7f2334e48640 10 monclient(hunting): handle_auth_done global_id 4356 payload 824 2024-09-16T03:42:31.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.125+0000 7f2334e48640 1 -- 172.21.15.42:0/3847095089 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f23301394f0 msgr2=0x7f2330139950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:31.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.125+0000 7f2335649640 1 --2- 172.21.15.42:0/3847095089 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2330135100 0x7f23301ccf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:31.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.125+0000 7f2334e48640 1 --2- 172.21.15.42:0/3847095089 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f23301394f0 0x7f2330139950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:31.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.125+0000 7f2334e48640 1 -- 172.21.15.42:0/3847095089 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2330135100 msgr2=0x7f23301ccf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:31.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.125+0000 7f2334e48640 1 --2- 172.21.15.42:0/3847095089 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2330135100 0x7f23301ccf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:31.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.125+0000 7f2334e48640 10 monclient: _finish_hunting 0 2024-09-16T03:42:31.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.125+0000 7f2334e48640 1 monclient: found mon.noname-a 2024-09-16T03:42:31.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.125+0000 7f2334e48640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:42:31.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.125+0000 7f2334e48640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:42:31.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.125+0000 7f2334e48640 1 -- 172.21.15.42:0/3847095089 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f233011d8b0 con 0x7f23301324d0 2024-09-16T03:42:31.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.125+0000 7f2335e4a640 1 --2- 172.21.15.42:0/3847095089 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f23301394f0 0x7f2330139950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:42:31.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.125+0000 7f2334e48640 1 --2- 172.21.15.42:0/3847095089 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f23301324d0 0x7f2330132950 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f232000ec40 tx=0x7f232000c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:31.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.125+0000 7f2325ffb640 10 client.?.objecter ms_handle_connect 0x7f23301324d0 2024-09-16T03:42:31.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.125+0000 7f2325ffb640 10 client.?.objecter resend_mon_ops 2024-09-16T03:42:31.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.125+0000 7f2325ffb640 1 -- 172.21.15.42:0/3847095089 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2320017070 con 0x7f23301324d0 2024-09-16T03:42:31.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.125+0000 7f2325ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:31.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.125+0000 7f2325ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:42:31.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.125+0000 7f2325ffb640 10 monclient: dump: 2024-09-16T03:42:31.142 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:31.143 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:31.143 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:31.143 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:31.143 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:31.143 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:31.143 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:31.143 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:31.143 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:31.143 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:31.143 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.125+0000 7f2325ffb640 10 monclient: _finish_auth 0 2024-09-16T03:42:31.143 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.125+0000 7f2325ffb640 10 monclient: _check_auth_tickets 2024-09-16T03:42:31.143 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.125+0000 7f2325ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:42:31.143 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.125+0000 7f2325ffb640 1 -- 172.21.15.42:0/3847095089 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f232000cc30 con 0x7f23301324d0 2024-09-16T03:42:31.143 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.125+0000 7f2325ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T03:42:31.143 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.125+0000 7f2325ffb640 1 -- 172.21.15.42:0/3847095089 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2320013470 con 0x7f23301324d0 2024-09-16T03:42:31.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.125+0000 7f23378ad640 5 monclient: authenticate success, global_id 4356 2024-09-16T03:42:31.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.125+0000 7f23378ad640 10 monclient: _renew_subs 2024-09-16T03:42:31.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.125+0000 7f23378ad640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:42:31.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.125+0000 7f23378ad640 1 -- 172.21.15.42:0/3847095089 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f233011c6e0 con 0x7f23301324d0 2024-09-16T03:42:31.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.125+0000 7f2325ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:31.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.125+0000 7f2325ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:42:31.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.125+0000 7f2325ffb640 10 monclient: dump: 2024-09-16T03:42:31.144 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:31.144 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:31.144 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:31.144 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:31.144 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:31.144 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:31.144 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:31.144 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:31.145 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:31.145 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:31.145 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.125+0000 7f23277fe640 4 set_mon_vals no callback set 2024-09-16T03:42:31.145 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.125+0000 7f23378ad640 10 client.4356.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:42:31.145 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.125+0000 7f23378ad640 10 monclient: _renew_subs 2024-09-16T03:42:31.145 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.125+0000 7f23378ad640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:42:31.145 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.125+0000 7f23378ad640 1 -- 172.21.15.42:0/3847095089 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f233011de70 con 0x7f23301324d0 2024-09-16T03:42:31.145 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.125+0000 7f23378ad640 1 librados: init done 2024-09-16T03:42:31.145 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.125+0000 7f23378ad640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:42:31.145 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.125+0000 7f23378ad640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:42:31.145 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.125+0000 7f23378ad640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:42:31.145 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.125+0000 7f2325ffb640 1 -- 172.21.15.42:0/3847095089 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f232001a070 con 0x7f23301324d0 2024-09-16T03:42:31.145 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.125+0000 7f23378ad640 1 -- 172.21.15.42:0/3847095089 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2330135550 con 0x7f23301324d0 2024-09-16T03:42:31.145 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.125+0000 7f2325ffb640 1 --2- 172.21.15.42:0/3847095089 >> v2:172.21.15.154:6812/254845342 conn(0x7f2314037810 0x7f2314039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:31.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.125+0000 7f2325ffb640 1 -- 172.21.15.42:0/3847095089 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 3534+0+0 (secure 0 0 0) 0x7f23200486f0 con 0x7f23301324d0 2024-09-16T03:42:31.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.125+0000 7f2325ffb640 10 client.4356.objecter ms_dispatch 0x7f2330000ba0 osd_map(19..19 src has 1..19) 2024-09-16T03:42:31.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.125+0000 7f2325ffb640 3 client.4356.objecter handle_osd_map got epochs [19,19] > 0 2024-09-16T03:42:31.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.125+0000 7f2325ffb640 3 client.4356.objecter handle_osd_map decoding full epoch 19 2024-09-16T03:42:31.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.125+0000 7f2325ffb640 20 client.4356.objecter dump_active .. 0 homeless 2024-09-16T03:42:31.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.125+0000 7f2335649640 1 --2- 172.21.15.42:0/3847095089 >> v2:172.21.15.154:6812/254845342 conn(0x7f2314037810 0x7f2314039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:31.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.125+0000 7f2335649640 10 monclient: get_auth_request con 0x7f2314037810 auth_method 0 2024-09-16T03:42:31.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.125+0000 7f2335649640 1 --2- 172.21.15.42:0/3847095089 >> v2:172.21.15.154:6812/254845342 conn(0x7f2314037810 0x7f2314039cd0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f232c0ad000 tx=0x7f232c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:31.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.125+0000 7f2325ffb640 10 client.4356.objecter ms_handle_connect 0x7f2314037810 2024-09-16T03:42:31.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.129+0000 7f2325ffb640 1 -- 172.21.15.42:0/3847095089 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f2320013610 con 0x7f23301324d0 2024-09-16T03:42:31.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.129+0000 7f2325ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:42:31.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.129+0000 7f2325ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:42:31.294 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.285+0000 7f23378ad640 1 -- 172.21.15.42:0/3847095089 --> v2:172.21.15.154:6812/254845342 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f233011dc50 con 0x7f2314037810 2024-09-16T03:42:31.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.289+0000 7f2325ffb640 1 -- 172.21.15.42:0/3847095089 <== mgr.4102 v2:172.21.15.154:6812/254845342 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+34247 (secure 0 0 0) 0x7f233011dc50 con 0x7f2314037810 2024-09-16T03:42:31.298 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-09-16T03:42:31.298 INFO:teuthology.orchestra.run.smithi042.stderr:dumped all 2024-09-16T03:42:31.300 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.293+0000 7f23378ad640 10 librados: watch_flush enter 2024-09-16T03:42:31.300 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.293+0000 7f23378ad640 10 librados: watch_flush exit 2024-09-16T03:42:31.300 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.293+0000 7f23378ad640 20 client.4356.objecter shutdown clearing up homeless session... 2024-09-16T03:42:31.300 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.293+0000 7f23378ad640 10 client.4356.objecter successfully canceled tick 2024-09-16T03:42:31.300 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.293+0000 7f23378ad640 1 -- 172.21.15.42:0/3847095089 >> v2:172.21.15.154:6812/254845342 conn(0x7f2314037810 msgr2=0x7f2314039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:31.300 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.293+0000 7f23378ad640 1 --2- 172.21.15.42:0/3847095089 >> v2:172.21.15.154:6812/254845342 conn(0x7f2314037810 0x7f2314039cd0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f232c0ad000 tx=0x7f232c008040 comp rx=0 tx=0).stop 2024-09-16T03:42:31.300 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.293+0000 7f23378ad640 10 monclient: shutdown 2024-09-16T03:42:31.301 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.293+0000 7f23378ad640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:42:31.301 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.293+0000 7f23378ad640 1 -- 172.21.15.42:0/3847095089 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f23301324d0 msgr2=0x7f2330132950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:31.301 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.293+0000 7f23378ad640 1 --2- 172.21.15.42:0/3847095089 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f23301324d0 0x7f2330132950 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f232000ec40 tx=0x7f232000c520 comp rx=0 tx=0).stop 2024-09-16T03:42:31.301 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.293+0000 7f23378ad640 1 -- 172.21.15.42:0/3847095089 shutdown_connections 2024-09-16T03:42:31.301 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.293+0000 7f23378ad640 1 --2- 172.21.15.42:0/3847095089 >> v2:172.21.15.154:6812/254845342 conn(0x7f2314037810 0x7f2314039cd0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:31.301 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.293+0000 7f23378ad640 1 --2- 172.21.15.42:0/3847095089 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f23301394f0 0x7f2330139950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:31.301 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.293+0000 7f23378ad640 1 --2- 172.21.15.42:0/3847095089 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f23301324d0 0x7f2330132950 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:31.301 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.293+0000 7f23378ad640 1 --2- 172.21.15.42:0/3847095089 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2330135100 0x7f23301ccf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:31.301 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.293+0000 7f23378ad640 1 -- 172.21.15.42:0/3847095089 >> 172.21.15.42:0/3847095089 conn(0x7f23300a2420 msgr2=0x7f2330070ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:42:31.301 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.293+0000 7f23378ad640 1 -- 172.21.15.42:0/3847095089 shutdown_connections 2024-09-16T03:42:31.301 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.293+0000 7f23378ad640 1 -- 172.21.15.42:0/3847095089 wait complete. 2024-09-16T03:42:31.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.293+0000 7f23378ad640 1 librados: shutdown 2024-09-16T03:42:31.315 INFO:teuthology.orchestra.run.smithi042.stdout:{"pg_ready":true,"pg_map":{"version":20,"stamp":"2024-09-16T03:42:30.646698+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":161,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120,"ondisk_log_size":120,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":12,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":167368,"kb_used_data":1016,"kb_used_omap":9,"kb_used_meta":166198,"kb_avail":566063672,"statfs":{"total":579820584960,"available":579649200128,"internally_reserved":0,"allocated":1040384,"data_stored":1353088,"data_compressed":18838,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":9567,"internal_metadata":170187425},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.176308"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":19,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T03:42:27.624215+0000","last_change":"2024-09-16T03:42:27.624922+0000","last_active":"2024-09-16T03:42:27.624215+0000","last_peered":"2024-09-16T03:42:27.624215+0000","last_clean":"2024-09-16T03:42:27.624215+0000","last_became_active":"2024-09-16T03:42:25.615293+0000","last_became_peered":"2024-09-16T03:42:25.615293+0000","last_unstale":"2024-09-16T03:42:27.624215+0000","last_undegraded":"2024-09-16T03:42:27.624215+0000","last_fullsized":"2024-09-16T03:42:27.624215+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:42:24.583922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:42:24.583922+0000","last_clean_scrub_stamp":"2024-09-16T03:42:24.583922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:46:11.677251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010681690000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":19,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T03:42:27.630687+0000","last_change":"2024-09-16T03:42:27.631227+0000","last_active":"2024-09-16T03:42:27.630687+0000","last_peered":"2024-09-16T03:42:27.630687+0000","last_clean":"2024-09-16T03:42:27.630687+0000","last_became_active":"2024-09-16T03:42:25.608453+0000","last_became_peered":"2024-09-16T03:42:25.608453+0000","last_unstale":"2024-09-16T03:42:27.630687+0000","last_undegraded":"2024-09-16T03:42:27.630687+0000","last_fullsized":"2024-09-16T03:42:27.630687+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:42:24.583922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:42:24.583922+0000","last_clean_scrub_stamp":"2024-09-16T03:42:24.583922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:00:11.921493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001163533,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":19,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T03:42:27.624630+0000","last_change":"2024-09-16T03:42:27.625091+0000","last_active":"2024-09-16T03:42:27.624630+0000","last_peered":"2024-09-16T03:42:27.624630+0000","last_clean":"2024-09-16T03:42:27.624630+0000","last_became_active":"2024-09-16T03:42:25.614458+0000","last_became_peered":"2024-09-16T03:42:25.614458+0000","last_unstale":"2024-09-16T03:42:27.624630+0000","last_undegraded":"2024-09-16T03:42:27.624630+0000","last_fullsized":"2024-09-16T03:42:27.624630+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:42:24.583922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:42:24.583922+0000","last_clean_scrub_stamp":"2024-09-16T03:42:24.583922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:26:14.681436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012701769999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T03:42:27.630545+0000","last_change":"2024-09-16T03:42:27.631260+0000","last_active":"2024-09-16T03:42:27.630545+0000","last_peered":"2024-09-16T03:42:27.630545+0000","last_clean":"2024-09-16T03:42:27.630545+0000","last_became_active":"2024-09-16T03:42:25.609631+0000","last_became_peered":"2024-09-16T03:42:25.609631+0000","last_unstale":"2024-09-16T03:42:27.630545+0000","last_undegraded":"2024-09-16T03:42:27.630545+0000","last_fullsized":"2024-09-16T03:42:27.630545+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:42:24.583922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:42:24.583922+0000","last_clean_scrub_stamp":"2024-09-16T03:42:24.583922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:58:49.091085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001288897,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"18'2","reported_seq":21,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T03:42:27.633516+0000","last_change":"2024-09-16T03:42:27.632050+0000","last_active":"2024-09-16T03:42:27.633516+0000","last_peered":"2024-09-16T03:42:27.633516+0000","last_clean":"2024-09-16T03:42:27.633516+0000","last_became_active":"2024-09-16T03:42:25.617896+0000","last_became_peered":"2024-09-16T03:42:25.617896+0000","last_unstale":"2024-09-16T03:42:27.633516+0000","last_undegraded":"2024-09-16T03:42:27.633516+0000","last_fullsized":"2024-09-16T03:42:27.633516+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:42:24.583922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:42:24.583922+0000","last_clean_scrub_stamp":"2024-09-16T03:42:24.583922+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:41:08.754274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021072529999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":19,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T03:42:27.618683+0000","last_change":"2024-09-16T03:42:27.619252+0000","last_active":"2024-09-16T03:42:27.618683+0000","last_peered":"2024-09-16T03:42:27.618683+0000","last_clean":"2024-09-16T03:42:27.618683+0000","last_became_active":"2024-09-16T03:42:25.607533+0000","last_became_peered":"2024-09-16T03:42:25.607533+0000","last_unstale":"2024-09-16T03:42:27.618683+0000","last_undegraded":"2024-09-16T03:42:27.618683+0000","last_fullsized":"2024-09-16T03:42:27.618683+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:42:24.583922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:42:24.583922+0000","last_clean_scrub_stamp":"2024-09-16T03:42:24.583922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:25:58.536144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089083299999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":19,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T03:42:27.624050+0000","last_change":"2024-09-16T03:42:27.624504+0000","last_active":"2024-09-16T03:42:27.624050+0000","last_peered":"2024-09-16T03:42:27.624050+0000","last_clean":"2024-09-16T03:42:27.624050+0000","last_became_active":"2024-09-16T03:42:25.612386+0000","last_became_peered":"2024-09-16T03:42:25.612386+0000","last_unstale":"2024-09-16T03:42:27.624050+0000","last_undegraded":"2024-09-16T03:42:27.624050+0000","last_fullsized":"2024-09-16T03:42:27.624050+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:42:24.583922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:42:24.583922+0000","last_clean_scrub_stamp":"2024-09-16T03:42:24.583922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:00:49.446880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081399800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"16'1","reported_seq":20,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T03:42:27.630965+0000","last_change":"2024-09-16T03:42:27.630965+0000","last_active":"2024-09-16T03:42:27.630965+0000","last_peered":"2024-09-16T03:42:27.630965+0000","last_clean":"2024-09-16T03:42:27.630965+0000","last_became_active":"2024-09-16T03:42:25.615930+0000","last_became_peered":"2024-09-16T03:42:25.615930+0000","last_unstale":"2024-09-16T03:42:27.630965+0000","last_undegraded":"2024-09-16T03:42:27.630965+0000","last_fullsized":"2024-09-16T03:42:27.630965+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:42:24.583922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:42:24.583922+0000","last_clean_scrub_stamp":"2024-09-16T03:42:24.583922+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:59:59.653118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00099470900000000009,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"14'117","reported_seq":170,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T03:42:27.623504+0000","last_change":"2024-09-16T03:42:18.799294+0000","last_active":"2024-09-16T03:42:27.623504+0000","last_peered":"2024-09-16T03:42:27.623504+0000","last_clean":"2024-09-16T03:42:27.623504+0000","last_became_active":"2024-09-16T03:42:18.797239+0000","last_became_peered":"2024-09-16T03:42:18.797239+0000","last_unstale":"2024-09-16T03:42:27.623504+0000","last_undegraded":"2024-09-16T03:42:27.623504+0000","last_fullsized":"2024-09-16T03:42:27.623504+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:42:17.544237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:42:17.544237+0000","last_clean_scrub_stamp":"2024-09-16T03:42:17.544237+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:01:21.017593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":18838,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":12,"seq":51539607558,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27188,"kb_used_data":156,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344652,"statfs":{"total":96636764160,"available":96608923648,"internally_reserved":0,"allocated":159744,"data_stored":49906,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":11,"seq":47244640260,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27172,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344668,"statfs":{"total":96636764160,"available":96608940032,"internally_reserved":0,"allocated":147456,"data_stored":45759,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640260,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":208896,"data_stored":578815,"data_compressed":9419,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640260,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":155648,"data_stored":49887,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640262,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31352,"kb_used_data":156,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340488,"statfs":{"total":96636764160,"available":96604659712,"internally_reserved":0,"allocated":159744,"data_stored":49906,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640260,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":208896,"data_stored":578815,"data_compressed":9419,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":9419,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":9419,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T03:42:31.316 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T03:42:31.437 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.429+0000 7f3d74772640 10 monclient: get_monmap_and_config 2024-09-16T03:42:31.437 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.429+0000 7f3d74772640 10 monclient: build_initial_monmap 2024-09-16T03:42:31.437 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.429+0000 7f3d74772640 10 monclient: monmap: 2024-09-16T03:42:31.437 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:42:31.437 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:31.437 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:42:31.434965+0000 2024-09-16T03:42:31.437 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:42:31.434965+0000 2024-09-16T03:42:31.437 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:42:31.438 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:31.438 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:42:31.438 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:42:31.438 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:42:31.438 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:42:31.438 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:42:31.438 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:31.440 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.433+0000 7f3d74772640 1 Processor -- start 2024-09-16T03:42:31.440 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.433+0000 7f3d74772640 1 -- start start 2024-09-16T03:42:31.440 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.433+0000 7f3d74772640 10 monclient: init 2024-09-16T03:42:31.440 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.433+0000 7f3d74772640 10 monclient: _reopen_session rank -1 2024-09-16T03:42:31.440 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.433+0000 7f3d74772640 10 monclient: _add_conns ranks=[0,1,3,2,4] 2024-09-16T03:42:31.440 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.433+0000 7f3d74772640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3d6c0a39b0 addr v1:172.21.15.42:6789/0 2024-09-16T03:42:31.440 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.433+0000 7f3d74772640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3d6c0a3d90 addr v1:172.21.15.154:6789/0 2024-09-16T03:42:31.440 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.433+0000 7f3d74772640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f3d6c13daa0 0x7f3d6c13fe90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:31.440 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.433+0000 7f3d74772640 10 monclient(hunting): picked mon.noname-a con 0x7f3d6c13daa0 addr v2:172.21.15.42:3300/0 2024-09-16T03:42:31.440 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.433+0000 7f3d74772640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d6c134290 con 0x7f3d6c0a39b0 2024-09-16T03:42:31.440 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.433+0000 7f3d74772640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d6c09c140 con 0x7f3d6c0a3d90 2024-09-16T03:42:31.441 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.433+0000 7f3d74772640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:31.441 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.433+0000 7f3d74772640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f3d6c09c420 con 0x7f3d6c13daa0 2024-09-16T03:42:31.441 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.433+0000 7f3d74772640 10 monclient(hunting): _renew_subs 2024-09-16T03:42:31.441 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.433+0000 7f3d74772640 10 monclient(hunting): authenticate will time out at 998.405026s 2024-09-16T03:42:31.441 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.433+0000 7f3d7250e640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f3d6c0a39b0 0x7f3d6c133b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:46826/0 (socket says 172.21.15.42:46826) 2024-09-16T03:42:31.441 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.433+0000 7f3d7250e640 1 -- 172.21.15.42:0/2809010711 learned_addr learned my addr 172.21.15.42:0/2809010711 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:42:31.441 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.433+0000 7f3d72d0f640 1 --2- 172.21.15.42:0/2809010711 >> v2:172.21.15.42:3300/0 conn(0x7f3d6c13daa0 0x7f3d6c13fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:31.441 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.433+0000 7f3d72d0f640 10 monclient(hunting): get_auth_request con 0x7f3d6c13daa0 auth_method 0 2024-09-16T03:42:31.441 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.433+0000 7f3d72d0f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:31.441 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.433+0000 7f3d72d0f640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:31.441 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.433+0000 7f3d72d0f640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:31.441 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.433+0000 7f3d72d0f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:31.441 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.433+0000 7f3d72d0f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:31.442 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.433+0000 7f3d72d0f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:31.442 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.433+0000 7f3d7150c640 1 -- 172.21.15.42:0/2809010711 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4035436465 0 0) 0x7f3d6c134290 con 0x7f3d6c0a39b0 2024-09-16T03:42:31.442 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.433+0000 7f3d7150c640 10 monclient(hunting): _negotiate 2024-09-16T03:42:31.442 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.433+0000 7f3d7150c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:31.442 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.433+0000 7f3d7150c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:31.442 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.433+0000 7f3d7150c640 10 monclient(hunting): my global_id is 4362 2024-09-16T03:42:31.442 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.433+0000 7f3d7150c640 1 -- 172.21.15.42:0/2809010711 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3d58003530 con 0x7f3d6c0a39b0 2024-09-16T03:42:31.442 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.433+0000 7f3d72d0f640 10 monclient(hunting): handle_auth_done global_id 4359 payload 274 2024-09-16T03:42:31.442 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.433+0000 7f3d72d0f640 1 -- 172.21.15.42:0/2809010711 >> v1:172.21.15.154:6789/0 conn(0x7f3d6c0a3d90 legacy=0x7f3d6c13c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:42:31.442 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.433+0000 7f3d72d0f640 1 -- 172.21.15.42:0/2809010711 >> v1:172.21.15.42:6789/0 conn(0x7f3d6c0a39b0 legacy=0x7f3d6c133b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:31.442 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.433+0000 7f3d72d0f640 10 monclient: _finish_hunting 0 2024-09-16T03:42:31.442 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.433+0000 7f3d72d0f640 1 monclient: found mon.noname-a 2024-09-16T03:42:31.442 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.433+0000 7f3d72d0f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:42:31.442 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.433+0000 7f3d72d0f640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:42:31.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.433+0000 7f3d72d0f640 1 -- 172.21.15.42:0/2809010711 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d6c06dd60 con 0x7f3d6c13daa0 2024-09-16T03:42:31.448 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.433+0000 7f3d72d0f640 1 --2- 172.21.15.42:0/2809010711 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3d6c13daa0 0x7f3d6c13fe90 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f3d64009d90 tx=0x7f3d6401c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef43ad22cb2cbcae server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:31.448 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.433+0000 7f3d7150c640 1 -- 172.21.15.42:0/2809010711 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d640a66c0 con 0x7f3d6c13daa0 2024-09-16T03:42:31.448 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.433+0000 7f3d7150c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:31.448 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.433+0000 7f3d7150c640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:42:31.448 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.433+0000 7f3d7150c640 10 monclient: dump: 2024-09-16T03:42:31.448 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:31.448 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:31.448 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:31.448 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:31.448 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:31.448 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:31.449 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:31.449 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:31.449 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:31.449 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:31.449 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.433+0000 7f3d7150c640 10 monclient: _finish_auth 0 2024-09-16T03:42:31.449 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.433+0000 7f3d7150c640 10 monclient: _check_auth_tickets 2024-09-16T03:42:31.449 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.433+0000 7f3d7150c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:42:31.449 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.433+0000 7f3d7150c640 1 -- 172.21.15.42:0/2809010711 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d640a6860 con 0x7f3d6c13daa0 2024-09-16T03:42:31.449 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.433+0000 7f3d7150c640 10 monclient: handle_config config(0 keys) 2024-09-16T03:42:31.449 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.433+0000 7f3d7150c640 1 -- 172.21.15.42:0/2809010711 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d640a6c30 con 0x7f3d6c13daa0 2024-09-16T03:42:31.449 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.433+0000 7f3d74772640 5 monclient: authenticate success, global_id 4359 2024-09-16T03:42:31.449 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.433+0000 7f3d74772640 10 monclient: get_monmap_and_config success 2024-09-16T03:42:31.450 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.433+0000 7f3d74772640 4 set_mon_vals no callback set 2024-09-16T03:42:31.450 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.433+0000 7f3d7150c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:31.450 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.433+0000 7f3d7150c640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:42:31.450 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.433+0000 7f3d7150c640 10 monclient: dump: 2024-09-16T03:42:31.450 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:31.450 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:31.450 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:31.450 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:31.450 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:31.450 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:31.450 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:31.450 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:31.450 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:31.450 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:31.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.437+0000 7f3d74772640 10 monclient: shutdown 2024-09-16T03:42:31.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.437+0000 7f3d74772640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:42:31.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.437+0000 7f3d74772640 1 -- 172.21.15.42:0/2809010711 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3d6c13daa0 msgr2=0x7f3d6c13fe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:31.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.437+0000 7f3d74772640 1 --2- 172.21.15.42:0/2809010711 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3d6c13daa0 0x7f3d6c13fe90 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f3d64009d90 tx=0x7f3d6401c7f0 comp rx=0 tx=0).stop 2024-09-16T03:42:31.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.437+0000 7f3d74772640 1 -- 172.21.15.42:0/2809010711 shutdown_connections 2024-09-16T03:42:31.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.437+0000 7f3d74772640 1 --2- 172.21.15.42:0/2809010711 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3d6c13daa0 0x7f3d6c13fe90 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:31.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.437+0000 7f3d74772640 1 -- 172.21.15.42:0/2809010711 >> 172.21.15.42:0/2809010711 conn(0x7f3d6c0a9fb0 msgr2=0x7f3d6c0aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:42:31.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.437+0000 7f3d74772640 1 -- 172.21.15.42:0/2809010711 shutdown_connections 2024-09-16T03:42:31.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.437+0000 7f3d74772640 1 -- 172.21.15.42:0/2809010711 wait complete. 2024-09-16T03:42:31.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.437+0000 7f3d74772640 10 monclient: build_initial_monmap 2024-09-16T03:42:31.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.437+0000 7f3d74772640 10 monclient: monmap: 2024-09-16T03:42:31.451 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:42:31.451 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:42:31.451 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:42:31.452 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:42:31.452 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:42:31.452 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:31.452 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:42:31.452 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:42:31.452 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:42:31.452 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:31.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.437+0000 7f3d74772640 1 librados: starting msgr at 2024-09-16T03:42:31.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.437+0000 7f3d74772640 1 librados: starting objecter 2024-09-16T03:42:31.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.437+0000 7f3d74772640 1 Processor -- start 2024-09-16T03:42:31.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.437+0000 7f3d74772640 1 -- start start 2024-09-16T03:42:31.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.437+0000 7f3d74772640 1 librados: setting wanted keys 2024-09-16T03:42:31.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.437+0000 7f3d74772640 1 librados: calling monclient init 2024-09-16T03:42:31.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.437+0000 7f3d74772640 10 monclient: init 2024-09-16T03:42:31.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.437+0000 7f3d74772640 10 monclient: _reopen_session rank -1 2024-09-16T03:42:31.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.437+0000 7f3d74772640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:42:31.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.437+0000 7f3d74772640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3d6c0a39b0 0x7f3d6c12c520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:31.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.437+0000 7f3d74772640 10 monclient(hunting): picked mon.noname-a con 0x7f3d6c0a39b0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:42:31.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.437+0000 7f3d74772640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3d6c0a3d90 0x7f3d6c12ca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:31.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.437+0000 7f3d74772640 10 monclient(hunting): picked mon.noname-c con 0x7f3d6c0a3d90 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:42:31.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.437+0000 7f3d74772640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d6c13daa0 0x7f3d6c12f470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:31.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.437+0000 7f3d74772640 10 monclient(hunting): picked mon.noname-b con 0x7f3d6c13daa0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:42:31.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.437+0000 7f3d74772640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:31.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.437+0000 7f3d74772640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f3d6c142f80 con 0x7f3d6c0a39b0 2024-09-16T03:42:31.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.437+0000 7f3d74772640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:31.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.437+0000 7f3d74772640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f3d6c142e00 con 0x7f3d6c13daa0 2024-09-16T03:42:31.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.437+0000 7f3d74772640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:31.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.437+0000 7f3d74772640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f3d6c143100 con 0x7f3d6c0a3d90 2024-09-16T03:42:31.454 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.437+0000 7f3d74772640 10 monclient(hunting): _renew_subs 2024-09-16T03:42:31.454 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.437+0000 7f3d74772640 10 monclient(hunting): authenticate will time out at 998.409597s 2024-09-16T03:42:31.454 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.437+0000 7f3d7250e640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3d6c0a39b0 0x7f3d6c12c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:31.454 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.437+0000 7f3d71d0d640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3d6c0a3d90 0x7f3d6c12ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:31.454 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.437+0000 7f3d7250e640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3d6c0a39b0 0x7f3d6c12c520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:37856/0 (socket says 172.21.15.42:37856) 2024-09-16T03:42:31.454 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.437+0000 7f3d7250e640 1 -- 172.21.15.42:0/3784642611 learned_addr learned my addr 172.21.15.42:0/3784642611 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:42:31.454 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.437+0000 7f3d7250e640 10 monclient(hunting): get_auth_request con 0x7f3d6c0a39b0 auth_method 0 2024-09-16T03:42:31.454 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.437+0000 7f3d7250e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:31.454 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.437+0000 7f3d7250e640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:31.454 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.437+0000 7f3d7250e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:31.454 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.437+0000 7f3d71d0d640 10 monclient(hunting): get_auth_request con 0x7f3d6c0a3d90 auth_method 0 2024-09-16T03:42:31.454 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.437+0000 7f3d71d0d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:31.454 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.437+0000 7f3d71d0d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:31.455 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.437+0000 7f3d71d0d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:31.455 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.437+0000 7f3d7250e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:31.455 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.437+0000 7f3d7250e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:31.455 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.437+0000 7f3d7250e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:31.455 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.437+0000 7f3d71d0d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:31.455 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.437+0000 7f3d71d0d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:31.455 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.437+0000 7f3d71d0d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:31.455 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.437+0000 7f3d7250e640 10 monclient(hunting): handle_auth_done global_id 4365 payload 824 2024-09-16T03:42:31.455 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.437+0000 7f3d7250e640 1 -- 172.21.15.42:0/3784642611 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3d6c0a3d90 msgr2=0x7f3d6c12ca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:31.455 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.437+0000 7f3d7250e640 1 --2- 172.21.15.42:0/3784642611 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3d6c0a3d90 0x7f3d6c12ca60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:31.455 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.437+0000 7f3d72d0f640 1 --2- 172.21.15.42:0/3784642611 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d6c13daa0 0x7f3d6c12f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:31.455 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.437+0000 7f3d7250e640 1 -- 172.21.15.42:0/3784642611 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d6c13daa0 msgr2=0x7f3d6c12f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:31.455 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.437+0000 7f3d7250e640 1 --2- 172.21.15.42:0/3784642611 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d6c13daa0 0x7f3d6c12f470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:31.456 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.437+0000 7f3d7250e640 10 monclient: _finish_hunting 0 2024-09-16T03:42:31.456 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.437+0000 7f3d7250e640 1 monclient: found mon.noname-a 2024-09-16T03:42:31.456 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.437+0000 7f3d7250e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:42:31.456 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.437+0000 7f3d7250e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:42:31.456 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.437+0000 7f3d7250e640 1 -- 172.21.15.42:0/3784642611 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d6c1400a0 con 0x7f3d6c0a39b0 2024-09-16T03:42:31.456 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.437+0000 7f3d71d0d640 1 --2- 172.21.15.42:0/3784642611 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3d6c0a3d90 0x7f3d6c12ca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:42:31.456 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.437+0000 7f3d7250e640 1 --2- 172.21.15.42:0/3784642611 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3d6c0a39b0 0x7f3d6c12c520 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f3d6800c3c0 tx=0x7f3d6800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:31.456 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.437+0000 7f3d62ffd640 10 client.?.objecter ms_handle_connect 0x7f3d6c0a39b0 2024-09-16T03:42:31.456 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.437+0000 7f3d62ffd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:42:31.456 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.437+0000 7f3d62ffd640 1 -- 172.21.15.42:0/3784642611 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d68017070 con 0x7f3d6c0a39b0 2024-09-16T03:42:31.456 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.437+0000 7f3d62ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:31.456 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.437+0000 7f3d62ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:42:31.456 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.437+0000 7f3d62ffd640 10 monclient: dump: 2024-09-16T03:42:31.456 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:31.457 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:31.457 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:31.457 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:31.457 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:31.457 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:31.457 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:31.457 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:31.457 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:31.457 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:31.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.437+0000 7f3d62ffd640 10 monclient: _finish_auth 0 2024-09-16T03:42:31.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.437+0000 7f3d62ffd640 10 monclient: _check_auth_tickets 2024-09-16T03:42:31.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.437+0000 7f3d62ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:42:31.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.437+0000 7f3d62ffd640 1 -- 172.21.15.42:0/3784642611 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d68005cf0 con 0x7f3d6c0a39b0 2024-09-16T03:42:31.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.437+0000 7f3d74772640 5 monclient: authenticate success, global_id 4365 2024-09-16T03:42:31.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.437+0000 7f3d74772640 10 monclient: _renew_subs 2024-09-16T03:42:31.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.437+0000 7f3d74772640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:42:31.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.437+0000 7f3d74772640 1 -- 172.21.15.42:0/3784642611 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d6c06f680 con 0x7f3d6c0a39b0 2024-09-16T03:42:31.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.437+0000 7f3d62ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:42:31.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.437+0000 7f3d62ffd640 1 -- 172.21.15.42:0/3784642611 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d680072f0 con 0x7f3d6c0a39b0 2024-09-16T03:42:31.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.437+0000 7f3d62ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:31.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.437+0000 7f3d62ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:42:31.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.437+0000 7f3d62ffd640 10 monclient: dump: 2024-09-16T03:42:31.458 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:31.458 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:31.458 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:31.458 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:31.458 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:31.458 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:31.459 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:31.459 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:31.459 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:31.459 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:31.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.437+0000 7f3d70d0b640 4 set_mon_vals no callback set 2024-09-16T03:42:31.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.437+0000 7f3d74772640 10 client.4365.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:42:31.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.437+0000 7f3d74772640 10 monclient: _renew_subs 2024-09-16T03:42:31.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.437+0000 7f3d74772640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:42:31.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.437+0000 7f3d74772640 1 -- 172.21.15.42:0/3784642611 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d6c139b40 con 0x7f3d6c0a39b0 2024-09-16T03:42:31.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.437+0000 7f3d74772640 1 librados: init done 2024-09-16T03:42:31.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.437+0000 7f3d74772640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:42:31.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.437+0000 7f3d74772640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:42:31.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.437+0000 7f3d74772640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:42:31.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.437+0000 7f3d74772640 1 -- 172.21.15.42:0/3784642611 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3d6c133b80 con 0x7f3d6c0a39b0 2024-09-16T03:42:31.460 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.437+0000 7f3d62ffd640 1 -- 172.21.15.42:0/3784642611 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f3d6801a070 con 0x7f3d6c0a39b0 2024-09-16T03:42:31.460 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.437+0000 7f3d62ffd640 1 --2- 172.21.15.42:0/3784642611 >> v2:172.21.15.154:6812/254845342 conn(0x7f3d50037810 0x7f3d50039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:31.460 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.437+0000 7f3d62ffd640 1 -- 172.21.15.42:0/3784642611 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 3534+0+0 (secure 0 0 0) 0x7f3d68047a80 con 0x7f3d6c0a39b0 2024-09-16T03:42:31.460 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.437+0000 7f3d62ffd640 10 client.4365.objecter ms_dispatch 0x7f3d6c000ba0 osd_map(19..19 src has 1..19) 2024-09-16T03:42:31.460 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.437+0000 7f3d62ffd640 3 client.4365.objecter handle_osd_map got epochs [19,19] > 0 2024-09-16T03:42:31.460 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.437+0000 7f3d62ffd640 3 client.4365.objecter handle_osd_map decoding full epoch 19 2024-09-16T03:42:31.460 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.437+0000 7f3d62ffd640 20 client.4365.objecter dump_active .. 0 homeless 2024-09-16T03:42:31.460 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.441+0000 7f3d71d0d640 1 --2- 172.21.15.42:0/3784642611 >> v2:172.21.15.154:6812/254845342 conn(0x7f3d50037810 0x7f3d50039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:31.460 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.441+0000 7f3d71d0d640 10 monclient: get_auth_request con 0x7f3d50037810 auth_method 0 2024-09-16T03:42:31.460 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.441+0000 7f3d71d0d640 1 --2- 172.21.15.42:0/3784642611 >> v2:172.21.15.154:6812/254845342 conn(0x7f3d50037810 0x7f3d50039cd0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f3d6c130bf0 tx=0x7f3d5c007e40 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:31.460 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.441+0000 7f3d62ffd640 10 client.4365.objecter ms_handle_connect 0x7f3d50037810 2024-09-16T03:42:31.461 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.441+0000 7f3d62ffd640 1 -- 172.21.15.42:0/3784642611 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f3d68019e10 con 0x7f3d6c0a39b0 2024-09-16T03:42:31.461 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.445+0000 7f3d62ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:42:31.461 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.445+0000 7f3d62ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:42:31.603 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.597+0000 7f3d74772640 1 -- 172.21.15.42:0/3784642611 --> v2:172.21.15.154:6812/254845342 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f3d6c0a5750 con 0x7f3d50037810 2024-09-16T03:42:31.606 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.601+0000 7f3d62ffd640 1 -- 172.21.15.42:0/3784642611 <== mgr.4102 v2:172.21.15.154:6812/254845342 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+34247 (secure 0 0 0) 0x7f3d6c0a5750 con 0x7f3d50037810 2024-09-16T03:42:31.607 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-09-16T03:42:31.607 INFO:teuthology.orchestra.run.smithi042.stderr:dumped all 2024-09-16T03:42:31.609 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.601+0000 7f3d74772640 10 librados: watch_flush enter 2024-09-16T03:42:31.609 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.601+0000 7f3d74772640 10 librados: watch_flush exit 2024-09-16T03:42:31.609 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.601+0000 7f3d74772640 20 client.4365.objecter shutdown clearing up homeless session... 2024-09-16T03:42:31.609 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.601+0000 7f3d74772640 10 client.4365.objecter successfully canceled tick 2024-09-16T03:42:31.609 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.601+0000 7f3d74772640 1 -- 172.21.15.42:0/3784642611 >> v2:172.21.15.154:6812/254845342 conn(0x7f3d50037810 msgr2=0x7f3d50039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:31.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.601+0000 7f3d74772640 1 --2- 172.21.15.42:0/3784642611 >> v2:172.21.15.154:6812/254845342 conn(0x7f3d50037810 0x7f3d50039cd0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f3d6c130bf0 tx=0x7f3d5c007e40 comp rx=0 tx=0).stop 2024-09-16T03:42:31.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.601+0000 7f3d74772640 10 monclient: shutdown 2024-09-16T03:42:31.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.601+0000 7f3d74772640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:42:31.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.601+0000 7f3d74772640 1 -- 172.21.15.42:0/3784642611 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3d6c0a39b0 msgr2=0x7f3d6c12c520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:31.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.601+0000 7f3d74772640 1 --2- 172.21.15.42:0/3784642611 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3d6c0a39b0 0x7f3d6c12c520 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f3d6800c3c0 tx=0x7f3d6800c880 comp rx=0 tx=0).stop 2024-09-16T03:42:31.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.601+0000 7f3d74772640 1 -- 172.21.15.42:0/3784642611 shutdown_connections 2024-09-16T03:42:31.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.601+0000 7f3d74772640 1 --2- 172.21.15.42:0/3784642611 >> v2:172.21.15.154:6812/254845342 conn(0x7f3d50037810 0x7f3d50039cd0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:31.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.601+0000 7f3d74772640 1 --2- 172.21.15.42:0/3784642611 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d6c13daa0 0x7f3d6c12f470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:31.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.601+0000 7f3d74772640 1 --2- 172.21.15.42:0/3784642611 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3d6c0a3d90 0x7f3d6c12ca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:31.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.601+0000 7f3d74772640 1 --2- 172.21.15.42:0/3784642611 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3d6c0a39b0 0x7f3d6c12c520 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:31.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.601+0000 7f3d74772640 1 -- 172.21.15.42:0/3784642611 >> 172.21.15.42:0/3784642611 conn(0x7f3d6c0a9fb0 msgr2=0x7f3d6c131980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:42:31.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.601+0000 7f3d74772640 1 -- 172.21.15.42:0/3784642611 shutdown_connections 2024-09-16T03:42:31.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.601+0000 7f3d74772640 1 -- 172.21.15.42:0/3784642611 wait complete. 2024-09-16T03:42:31.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.601+0000 7f3d74772640 1 librados: shutdown 2024-09-16T03:42:31.624 INFO:teuthology.orchestra.run.smithi042.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-09-16T03:42:31.472067+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":161,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120,"ondisk_log_size":120,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":12,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":167368,"kb_used_data":1016,"kb_used_omap":9,"kb_used_meta":166198,"kb_avail":566063672,"statfs":{"total":579820584960,"available":579649200128,"internally_reserved":0,"allocated":1040384,"data_stored":1353088,"data_compressed":18838,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":9567,"internal_metadata":170187425},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.867729"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":19,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T03:42:27.624215+0000","last_change":"2024-09-16T03:42:27.624922+0000","last_active":"2024-09-16T03:42:27.624215+0000","last_peered":"2024-09-16T03:42:27.624215+0000","last_clean":"2024-09-16T03:42:27.624215+0000","last_became_active":"2024-09-16T03:42:25.615293+0000","last_became_peered":"2024-09-16T03:42:25.615293+0000","last_unstale":"2024-09-16T03:42:27.624215+0000","last_undegraded":"2024-09-16T03:42:27.624215+0000","last_fullsized":"2024-09-16T03:42:27.624215+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:42:24.583922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:42:24.583922+0000","last_clean_scrub_stamp":"2024-09-16T03:42:24.583922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:46:11.677251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010681690000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":19,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T03:42:27.630687+0000","last_change":"2024-09-16T03:42:27.631227+0000","last_active":"2024-09-16T03:42:27.630687+0000","last_peered":"2024-09-16T03:42:27.630687+0000","last_clean":"2024-09-16T03:42:27.630687+0000","last_became_active":"2024-09-16T03:42:25.608453+0000","last_became_peered":"2024-09-16T03:42:25.608453+0000","last_unstale":"2024-09-16T03:42:27.630687+0000","last_undegraded":"2024-09-16T03:42:27.630687+0000","last_fullsized":"2024-09-16T03:42:27.630687+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:42:24.583922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:42:24.583922+0000","last_clean_scrub_stamp":"2024-09-16T03:42:24.583922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:00:11.921493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001163533,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":19,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T03:42:27.624630+0000","last_change":"2024-09-16T03:42:27.625091+0000","last_active":"2024-09-16T03:42:27.624630+0000","last_peered":"2024-09-16T03:42:27.624630+0000","last_clean":"2024-09-16T03:42:27.624630+0000","last_became_active":"2024-09-16T03:42:25.614458+0000","last_became_peered":"2024-09-16T03:42:25.614458+0000","last_unstale":"2024-09-16T03:42:27.624630+0000","last_undegraded":"2024-09-16T03:42:27.624630+0000","last_fullsized":"2024-09-16T03:42:27.624630+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:42:24.583922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:42:24.583922+0000","last_clean_scrub_stamp":"2024-09-16T03:42:24.583922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:26:14.681436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012701769999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T03:42:27.630545+0000","last_change":"2024-09-16T03:42:27.631260+0000","last_active":"2024-09-16T03:42:27.630545+0000","last_peered":"2024-09-16T03:42:27.630545+0000","last_clean":"2024-09-16T03:42:27.630545+0000","last_became_active":"2024-09-16T03:42:25.609631+0000","last_became_peered":"2024-09-16T03:42:25.609631+0000","last_unstale":"2024-09-16T03:42:27.630545+0000","last_undegraded":"2024-09-16T03:42:27.630545+0000","last_fullsized":"2024-09-16T03:42:27.630545+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:42:24.583922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:42:24.583922+0000","last_clean_scrub_stamp":"2024-09-16T03:42:24.583922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:58:49.091085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001288897,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"18'2","reported_seq":21,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T03:42:27.633516+0000","last_change":"2024-09-16T03:42:27.632050+0000","last_active":"2024-09-16T03:42:27.633516+0000","last_peered":"2024-09-16T03:42:27.633516+0000","last_clean":"2024-09-16T03:42:27.633516+0000","last_became_active":"2024-09-16T03:42:25.617896+0000","last_became_peered":"2024-09-16T03:42:25.617896+0000","last_unstale":"2024-09-16T03:42:27.633516+0000","last_undegraded":"2024-09-16T03:42:27.633516+0000","last_fullsized":"2024-09-16T03:42:27.633516+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:42:24.583922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:42:24.583922+0000","last_clean_scrub_stamp":"2024-09-16T03:42:24.583922+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:41:08.754274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021072529999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":19,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T03:42:27.618683+0000","last_change":"2024-09-16T03:42:27.619252+0000","last_active":"2024-09-16T03:42:27.618683+0000","last_peered":"2024-09-16T03:42:27.618683+0000","last_clean":"2024-09-16T03:42:27.618683+0000","last_became_active":"2024-09-16T03:42:25.607533+0000","last_became_peered":"2024-09-16T03:42:25.607533+0000","last_unstale":"2024-09-16T03:42:27.618683+0000","last_undegraded":"2024-09-16T03:42:27.618683+0000","last_fullsized":"2024-09-16T03:42:27.618683+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:42:24.583922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:42:24.583922+0000","last_clean_scrub_stamp":"2024-09-16T03:42:24.583922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:25:58.536144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089083299999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":19,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T03:42:27.624050+0000","last_change":"2024-09-16T03:42:27.624504+0000","last_active":"2024-09-16T03:42:27.624050+0000","last_peered":"2024-09-16T03:42:27.624050+0000","last_clean":"2024-09-16T03:42:27.624050+0000","last_became_active":"2024-09-16T03:42:25.612386+0000","last_became_peered":"2024-09-16T03:42:25.612386+0000","last_unstale":"2024-09-16T03:42:27.624050+0000","last_undegraded":"2024-09-16T03:42:27.624050+0000","last_fullsized":"2024-09-16T03:42:27.624050+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:42:24.583922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:42:24.583922+0000","last_clean_scrub_stamp":"2024-09-16T03:42:24.583922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:00:49.446880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081399800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"16'1","reported_seq":20,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T03:42:27.630965+0000","last_change":"2024-09-16T03:42:27.630965+0000","last_active":"2024-09-16T03:42:27.630965+0000","last_peered":"2024-09-16T03:42:27.630965+0000","last_clean":"2024-09-16T03:42:27.630965+0000","last_became_active":"2024-09-16T03:42:25.615930+0000","last_became_peered":"2024-09-16T03:42:25.615930+0000","last_unstale":"2024-09-16T03:42:27.630965+0000","last_undegraded":"2024-09-16T03:42:27.630965+0000","last_fullsized":"2024-09-16T03:42:27.630965+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:42:24.583922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:42:24.583922+0000","last_clean_scrub_stamp":"2024-09-16T03:42:24.583922+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:59:59.653118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00099470900000000009,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"14'117","reported_seq":170,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T03:42:27.623504+0000","last_change":"2024-09-16T03:42:18.799294+0000","last_active":"2024-09-16T03:42:27.623504+0000","last_peered":"2024-09-16T03:42:27.623504+0000","last_clean":"2024-09-16T03:42:27.623504+0000","last_became_active":"2024-09-16T03:42:18.797239+0000","last_became_peered":"2024-09-16T03:42:18.797239+0000","last_unstale":"2024-09-16T03:42:27.623504+0000","last_undegraded":"2024-09-16T03:42:27.623504+0000","last_fullsized":"2024-09-16T03:42:27.623504+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:42:17.544237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:42:17.544237+0000","last_clean_scrub_stamp":"2024-09-16T03:42:17.544237+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:01:21.017593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":18838,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":12,"seq":51539607558,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27188,"kb_used_data":156,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344652,"statfs":{"total":96636764160,"available":96608923648,"internally_reserved":0,"allocated":159744,"data_stored":49906,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":11,"seq":47244640260,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27172,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344668,"statfs":{"total":96636764160,"available":96608940032,"internally_reserved":0,"allocated":147456,"data_stored":45759,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640260,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":208896,"data_stored":578815,"data_compressed":9419,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640260,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":155648,"data_stored":49887,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640262,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31352,"kb_used_data":156,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340488,"statfs":{"total":96636764160,"available":96604659712,"internally_reserved":0,"allocated":159744,"data_stored":49906,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640260,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":208896,"data_stored":578815,"data_compressed":9419,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":9419,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":9419,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T03:42:31.625 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-16T03:42:31.625 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-16T03:42:31.625 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-09-16T03:42:31.626 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-09-16T03:42:31.736 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.729+0000 7f2a339f8640 10 monclient: get_monmap_and_config 2024-09-16T03:42:31.736 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.729+0000 7f2a339f8640 10 monclient: build_initial_monmap 2024-09-16T03:42:31.736 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.729+0000 7f2a339f8640 10 monclient: monmap: 2024-09-16T03:42:31.736 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:42:31.736 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:31.736 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:42:31.733992+0000 2024-09-16T03:42:31.736 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:42:31.733992+0000 2024-09-16T03:42:31.736 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:42:31.736 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:31.736 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:42:31.737 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:42:31.737 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:42:31.737 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:42:31.737 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:42:31.737 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:31.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.729+0000 7f2a339f8640 1 Processor -- start 2024-09-16T03:42:31.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.729+0000 7f2a339f8640 1 -- start start 2024-09-16T03:42:31.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.729+0000 7f2a339f8640 10 monclient: init 2024-09-16T03:42:31.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.729+0000 7f2a339f8640 10 monclient: _reopen_session rank -1 2024-09-16T03:42:31.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.729+0000 7f2a339f8640 10 monclient: _add_conns ranks=[4,1,2,0,3] 2024-09-16T03:42:31.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.729+0000 7f2a339f8640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f2a2c09fb00 0x7f2a2c09ba50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:31.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.729+0000 7f2a339f8640 10 monclient(hunting): picked mon.noname-c con 0x7f2a2c09fb00 addr v2:172.21.15.154:3300/0 2024-09-16T03:42:31.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.729+0000 7f2a339f8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2a2c09bf90 addr v1:172.21.15.154:6789/0 2024-09-16T03:42:31.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.729+0000 7f2a339f8640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2a2c09c400 0x7f2a2c13b840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:31.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.729+0000 7f2a339f8640 10 monclient(hunting): picked mon.noname-b con 0x7f2a2c09c400 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:42:31.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.729+0000 7f2a339f8640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2a2c09c870 con 0x7f2a2c09bf90 2024-09-16T03:42:31.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.729+0000 7f2a339f8640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:31.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.729+0000 7f2a339f8640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f2a2c09e250 con 0x7f2a2c09fb00 2024-09-16T03:42:31.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.729+0000 7f2a339f8640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:31.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.729+0000 7f2a339f8640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f2a2c0a1e20 con 0x7f2a2c09c400 2024-09-16T03:42:31.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.729+0000 7f2a339f8640 10 monclient(hunting): _renew_subs 2024-09-16T03:42:31.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.729+0000 7f2a339f8640 10 monclient(hunting): authenticate will time out at 998.703793s 2024-09-16T03:42:31.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.733+0000 7f2a31f95640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2a2c09c400 0x7f2a2c13b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:31.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.733+0000 7f2a31794640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f2a2c09fb00 0x7f2a2c09ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:31.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.733+0000 7f2a31794640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f2a2c09fb00 0x7f2a2c09ba50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.42:42200/0 (socket says 172.21.15.42:42200) 2024-09-16T03:42:31.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.733+0000 7f2a31f95640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2a2c09c400 0x7f2a2c13b840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:41558/0 (socket says 172.21.15.42:41558) 2024-09-16T03:42:31.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.733+0000 7f2a31794640 1 -- 172.21.15.42:0/2685150645 learned_addr learned my addr 172.21.15.42:0/2685150645 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:42:31.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.733+0000 7f2a31794640 10 monclient(hunting): get_auth_request con 0x7f2a2c09fb00 auth_method 0 2024-09-16T03:42:31.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.733+0000 7f2a31794640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:31.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.733+0000 7f2a31794640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:31.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.733+0000 7f2a31794640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:31.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.733+0000 7f2a31f95640 10 monclient(hunting): get_auth_request con 0x7f2a2c09c400 auth_method 0 2024-09-16T03:42:31.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.733+0000 7f2a31f95640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:31.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.733+0000 7f2a31f95640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:31.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.733+0000 7f2a31f95640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:31.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.733+0000 7f2a31f95640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:31.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.733+0000 7f2a31f95640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:31.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.733+0000 7f2a31f95640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:31.742 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.733+0000 7f2a31794640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:31.742 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.733+0000 7f2a31794640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:31.742 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.733+0000 7f2a31794640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:31.742 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.733+0000 7f2a31f95640 10 monclient(hunting): handle_auth_done global_id 4313 payload 274 2024-09-16T03:42:31.742 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.733+0000 7f2a31f95640 1 -- 172.21.15.42:0/2685150645 >> v2:172.21.15.154:3300/0 conn(0x7f2a2c09fb00 msgr2=0x7f2a2c09ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:42:31.742 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.733+0000 7f2a31f95640 1 --2- 172.21.15.42:0/2685150645 >> v2:172.21.15.154:3300/0 conn(0x7f2a2c09fb00 0x7f2a2c09ba50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:31.742 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.733+0000 7f2a31f95640 1 -- 172.21.15.42:0/2685150645 >> v1:172.21.15.154:6789/0 conn(0x7f2a2c09bf90 legacy=0x7f2a2c138110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:31.742 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.733+0000 7f2a31f95640 10 monclient: _finish_hunting 0 2024-09-16T03:42:31.742 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.733+0000 7f2a31f95640 1 monclient: found mon.noname-b 2024-09-16T03:42:31.742 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.733+0000 7f2a31f95640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:42:31.742 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.733+0000 7f2a31f95640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T03:42:31.742 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.733+0000 7f2a31f95640 1 -- 172.21.15.42:0/2685150645 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a2c09ea30 con 0x7f2a2c09c400 2024-09-16T03:42:31.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.733+0000 7f2a31f95640 1 --2- 172.21.15.42:0/2685150645 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2a2c09c400 0x7f2a2c13b840 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f2a24009d90 tx=0x7f2a2401c830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c933e88eb9058390 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:31.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.733+0000 7f2a23fff640 1 -- 172.21.15.42:0/2685150645 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a240b2070 con 0x7f2a2c09c400 2024-09-16T03:42:31.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.733+0000 7f2a23fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:31.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.733+0000 7f2a23fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:42:31.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.733+0000 7f2a23fff640 10 monclient: dump: 2024-09-16T03:42:31.748 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:31.748 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:31.748 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:31.748 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:31.748 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:31.748 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:31.748 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:31.748 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:31.749 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:31.749 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:31.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.733+0000 7f2a23fff640 10 monclient: _finish_auth 0 2024-09-16T03:42:31.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.733+0000 7f2a23fff640 10 monclient: _check_auth_tickets 2024-09-16T03:42:31.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.733+0000 7f2a23fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:42:31.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.733+0000 7f2a23fff640 1 -- 172.21.15.42:0/2685150645 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a240a5870 con 0x7f2a2c09c400 2024-09-16T03:42:31.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.733+0000 7f2a23fff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:42:31.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.733+0000 7f2a23fff640 1 -- 172.21.15.42:0/2685150645 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a240a5c40 con 0x7f2a2c09c400 2024-09-16T03:42:31.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.733+0000 7f2a23fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:31.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.733+0000 7f2a23fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:42:31.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.733+0000 7f2a23fff640 10 monclient: dump: 2024-09-16T03:42:31.752 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:31.752 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:31.752 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:31.752 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:31.752 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:31.753 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:31.753 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:31.753 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:31.753 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:31.753 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:31.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.733+0000 7f2a339f8640 5 monclient: authenticate success, global_id 4313 2024-09-16T03:42:31.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.733+0000 7f2a339f8640 10 monclient: get_monmap_and_config success 2024-09-16T03:42:31.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.733+0000 7f2a339f8640 4 set_mon_vals no callback set 2024-09-16T03:42:31.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.733+0000 7f2a339f8640 10 monclient: shutdown 2024-09-16T03:42:31.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.733+0000 7f2a339f8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:42:31.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.733+0000 7f2a339f8640 1 -- 172.21.15.42:0/2685150645 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2a2c09c400 msgr2=0x7f2a2c13b840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:31.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.733+0000 7f2a339f8640 1 --2- 172.21.15.42:0/2685150645 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2a2c09c400 0x7f2a2c13b840 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f2a24009d90 tx=0x7f2a2401c830 comp rx=0 tx=0).stop 2024-09-16T03:42:31.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.733+0000 7f2a339f8640 1 -- 172.21.15.42:0/2685150645 shutdown_connections 2024-09-16T03:42:31.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.733+0000 7f2a339f8640 1 --2- 172.21.15.42:0/2685150645 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2a2c09c400 0x7f2a2c13b840 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:31.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.733+0000 7f2a339f8640 1 --2- 172.21.15.42:0/2685150645 >> v2:172.21.15.154:3300/0 conn(0x7f2a2c09fb00 0x7f2a2c09ba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:31.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.733+0000 7f2a339f8640 1 -- 172.21.15.42:0/2685150645 >> 172.21.15.42:0/2685150645 conn(0x7f2a2c0a2420 msgr2=0x7f2a2c0a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:42:31.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.733+0000 7f2a339f8640 1 -- 172.21.15.42:0/2685150645 shutdown_connections 2024-09-16T03:42:31.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.733+0000 7f2a339f8640 1 -- 172.21.15.42:0/2685150645 wait complete. 2024-09-16T03:42:31.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.733+0000 7f2a339f8640 10 monclient: build_initial_monmap 2024-09-16T03:42:31.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.733+0000 7f2a339f8640 10 monclient: monmap: 2024-09-16T03:42:31.754 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:42:31.754 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:42:31.754 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:42:31.754 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:42:31.754 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:42:31.754 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:31.755 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:42:31.755 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:42:31.755 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:42:31.755 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:31.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.737+0000 7f2a339f8640 1 librados: starting msgr at 2024-09-16T03:42:31.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.737+0000 7f2a339f8640 1 librados: starting objecter 2024-09-16T03:42:31.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.737+0000 7f2a339f8640 1 Processor -- start 2024-09-16T03:42:31.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.737+0000 7f2a339f8640 1 -- start start 2024-09-16T03:42:31.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.737+0000 7f2a339f8640 1 librados: setting wanted keys 2024-09-16T03:42:31.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.737+0000 7f2a339f8640 1 librados: calling monclient init 2024-09-16T03:42:31.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.737+0000 7f2a339f8640 10 monclient: init 2024-09-16T03:42:31.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.737+0000 7f2a339f8640 10 monclient: _reopen_session rank -1 2024-09-16T03:42:31.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.737+0000 7f2a339f8640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T03:42:31.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.737+0000 7f2a339f8640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2a2c09bf90 0x7f2a2c1cccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:31.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.737+0000 7f2a339f8640 10 monclient(hunting): picked mon.noname-a con 0x7f2a2c09bf90 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:42:31.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.737+0000 7f2a339f8640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2a2c09c400 0x7f2a2c136620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:31.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.737+0000 7f2a339f8640 10 monclient(hunting): picked mon.noname-b con 0x7f2a2c09c400 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:42:31.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.737+0000 7f2a339f8640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2a2c09fb00 0x7f2a2c136b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:31.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.737+0000 7f2a339f8640 10 monclient(hunting): picked mon.noname-c con 0x7f2a2c09fb00 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:42:31.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.737+0000 7f2a339f8640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:31.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.737+0000 7f2a339f8640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f2a2c0a3170 con 0x7f2a2c09bf90 2024-09-16T03:42:31.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.737+0000 7f2a339f8640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:31.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.737+0000 7f2a339f8640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f2a2c0a32f0 con 0x7f2a2c09c400 2024-09-16T03:42:31.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.737+0000 7f2a339f8640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:31.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.737+0000 7f2a339f8640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f2a2c09d380 con 0x7f2a2c09fb00 2024-09-16T03:42:31.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.737+0000 7f2a339f8640 10 monclient(hunting): _renew_subs 2024-09-16T03:42:31.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.737+0000 7f2a339f8640 10 monclient(hunting): authenticate will time out at 998.708683s 2024-09-16T03:42:31.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.737+0000 7f2a31794640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2a2c09bf90 0x7f2a2c1cccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:31.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.737+0000 7f2a31f95640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2a2c09fb00 0x7f2a2c136b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:31.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.737+0000 7f2a31794640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2a2c09bf90 0x7f2a2c1cccd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:37870/0 (socket says 172.21.15.42:37870) 2024-09-16T03:42:31.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.737+0000 7f2a31794640 1 -- 172.21.15.42:0/2376448011 learned_addr learned my addr 172.21.15.42:0/2376448011 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:42:31.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.737+0000 7f2a31f95640 10 monclient(hunting): get_auth_request con 0x7f2a2c09fb00 auth_method 0 2024-09-16T03:42:31.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.737+0000 7f2a31f95640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:31.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.737+0000 7f2a31f95640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:31.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.737+0000 7f2a31f95640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:31.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.737+0000 7f2a31794640 10 monclient(hunting): get_auth_request con 0x7f2a2c09bf90 auth_method 0 2024-09-16T03:42:31.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.737+0000 7f2a31794640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:31.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.737+0000 7f2a31794640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:31.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.737+0000 7f2a31794640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:31.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.737+0000 7f2a31f95640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:31.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.737+0000 7f2a31f95640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:31.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.737+0000 7f2a31f95640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:31.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.737+0000 7f2a31794640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:31.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.737+0000 7f2a31794640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:31.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.737+0000 7f2a31794640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:31.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.737+0000 7f2a31f95640 10 monclient(hunting): handle_auth_done global_id 4316 payload 824 2024-09-16T03:42:31.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.737+0000 7f2a30f93640 1 --2- 172.21.15.42:0/2376448011 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2a2c09c400 0x7f2a2c136620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:31.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.737+0000 7f2a31f95640 1 -- 172.21.15.42:0/2376448011 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2a2c09c400 msgr2=0x7f2a2c136620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:31.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.737+0000 7f2a31f95640 1 --2- 172.21.15.42:0/2376448011 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2a2c09c400 0x7f2a2c136620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:31.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.737+0000 7f2a31f95640 1 -- 172.21.15.42:0/2376448011 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2a2c09bf90 msgr2=0x7f2a2c1cccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:31.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.737+0000 7f2a31f95640 1 --2- 172.21.15.42:0/2376448011 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2a2c09bf90 0x7f2a2c1cccd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:31.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.737+0000 7f2a31f95640 10 monclient: _finish_hunting 0 2024-09-16T03:42:31.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.737+0000 7f2a31f95640 1 monclient: found mon.noname-c 2024-09-16T03:42:31.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.737+0000 7f2a31f95640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:42:31.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.737+0000 7f2a31f95640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:42:31.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.737+0000 7f2a31f95640 1 -- 172.21.15.42:0/2376448011 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a2c073a30 con 0x7f2a2c09fb00 2024-09-16T03:42:31.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.737+0000 7f2a31794640 1 --2- 172.21.15.42:0/2376448011 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2a2c09bf90 0x7f2a2c1cccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:42:31.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.737+0000 7f2a31f95640 1 --2- 172.21.15.42:0/2376448011 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2a2c09fb00 0x7f2a2c136b60 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f2a2c09e3d0 tx=0x7f2a240195d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:31.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.737+0000 7f2a21ffb640 10 client.?.objecter ms_handle_connect 0x7f2a2c09fb00 2024-09-16T03:42:31.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.737+0000 7f2a21ffb640 10 client.?.objecter resend_mon_ops 2024-09-16T03:42:31.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.737+0000 7f2a21ffb640 1 -- 172.21.15.42:0/2376448011 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a240b2050 con 0x7f2a2c09fb00 2024-09-16T03:42:31.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.737+0000 7f2a21ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:31.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.737+0000 7f2a21ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:42:31.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.737+0000 7f2a21ffb640 10 monclient: dump: 2024-09-16T03:42:31.759 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:31.759 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:31.759 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:31.760 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:31.760 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:31.760 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:31.760 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:31.760 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:31.760 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:31.760 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:31.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.737+0000 7f2a21ffb640 10 monclient: _finish_auth 0 2024-09-16T03:42:31.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.737+0000 7f2a21ffb640 10 monclient: _check_auth_tickets 2024-09-16T03:42:31.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.737+0000 7f2a21ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:42:31.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.737+0000 7f2a339f8640 5 monclient: authenticate success, global_id 4316 2024-09-16T03:42:31.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.737+0000 7f2a21ffb640 1 -- 172.21.15.42:0/2376448011 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a240b3040 con 0x7f2a2c09fb00 2024-09-16T03:42:31.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.737+0000 7f2a21ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T03:42:31.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.737+0000 7f2a21ffb640 1 -- 172.21.15.42:0/2376448011 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a240ad650 con 0x7f2a2c09fb00 2024-09-16T03:42:31.761 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.737+0000 7f2a21ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:31.761 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.737+0000 7f2a21ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:42:31.761 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.737+0000 7f2a21ffb640 10 monclient: dump: 2024-09-16T03:42:31.761 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:31.761 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:31.761 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:31.761 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:31.761 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:31.761 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:31.761 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:31.761 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:31.761 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:31.761 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:31.761 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.737+0000 7f2a237fe640 4 set_mon_vals no callback set 2024-09-16T03:42:31.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.737+0000 7f2a339f8640 10 monclient: _renew_subs 2024-09-16T03:42:31.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.737+0000 7f2a339f8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:42:31.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.737+0000 7f2a339f8640 1 -- 172.21.15.42:0/2376448011 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a2c122fd0 con 0x7f2a2c09fb00 2024-09-16T03:42:31.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.737+0000 7f2a339f8640 10 client.4316.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:42:31.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.737+0000 7f2a339f8640 10 monclient: _renew_subs 2024-09-16T03:42:31.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.737+0000 7f2a339f8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:42:31.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.737+0000 7f2a339f8640 1 -- 172.21.15.42:0/2376448011 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a2c120c70 con 0x7f2a2c09fb00 2024-09-16T03:42:31.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.737+0000 7f2a339f8640 1 librados: init done 2024-09-16T03:42:31.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.737+0000 7f2a339f8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:42:31.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.737+0000 7f2a339f8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:42:31.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.737+0000 7f2a339f8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:42:31.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.737+0000 7f2a21ffb640 1 -- 172.21.15.42:0/2376448011 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f2a240c0020 con 0x7f2a2c09fb00 2024-09-16T03:42:31.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.737+0000 7f2a339f8640 1 -- 172.21.15.42:0/2376448011 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2a2c09ba50 con 0x7f2a2c09fb00 2024-09-16T03:42:31.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.737+0000 7f2a21ffb640 1 --2- 172.21.15.42:0/2376448011 >> v2:172.21.15.154:6812/254845342 conn(0x7f2a10037810 0x7f2a10039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:31.763 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.737+0000 7f2a21ffb640 1 -- 172.21.15.42:0/2376448011 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(19..19 src has 1..19) ==== 3534+0+0 (secure 0 0 0) 0x7f2a240e2d20 con 0x7f2a2c09fb00 2024-09-16T03:42:31.763 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.737+0000 7f2a21ffb640 10 client.4316.objecter ms_dispatch 0x7f2a2c000ba0 osd_map(19..19 src has 1..19) 2024-09-16T03:42:31.763 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.737+0000 7f2a21ffb640 3 client.4316.objecter handle_osd_map got epochs [19,19] > 0 2024-09-16T03:42:31.763 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.737+0000 7f2a21ffb640 3 client.4316.objecter handle_osd_map decoding full epoch 19 2024-09-16T03:42:31.763 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.737+0000 7f2a21ffb640 20 client.4316.objecter dump_active .. 0 homeless 2024-09-16T03:42:31.763 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.737+0000 7f2a31794640 1 --2- 172.21.15.42:0/2376448011 >> v2:172.21.15.154:6812/254845342 conn(0x7f2a10037810 0x7f2a10039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:31.763 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.737+0000 7f2a31794640 10 monclient: get_auth_request con 0x7f2a10037810 auth_method 0 2024-09-16T03:42:31.763 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.741+0000 7f2a31794640 1 --2- 172.21.15.42:0/2376448011 >> v2:172.21.15.154:6812/254845342 conn(0x7f2a10037810 0x7f2a10039cd0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f2a2800a640 tx=0x7f2a2800a350 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:31.763 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.741+0000 7f2a21ffb640 10 client.4316.objecter ms_handle_connect 0x7f2a10037810 2024-09-16T03:42:31.764 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.741+0000 7f2a21ffb640 1 -- 172.21.15.42:0/2376448011 <== mon.2 v2:172.21.15.42:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f2a240e7070 con 0x7f2a2c09fb00 2024-09-16T03:42:31.764 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.741+0000 7f2a21ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:42:31.764 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.741+0000 7f2a21ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:42:31.943 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.937+0000 7f2a339f8640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2024-09-16T03:42:31.943 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.937+0000 7f2a339f8640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2024-09-16T03:42:31.943 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.937+0000 7f2a339f8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:42:31.943 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.937+0000 7f2a339f8640 1 -- 172.21.15.42:0/2376448011 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f2a2c069040 con 0x7f2a2c09fb00 2024-09-16T03:42:31.944 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.937+0000 7f2a21ffb640 1 -- 172.21.15.42:0/2376448011 <== mon.2 v2:172.21.15.42:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7f2a2c073a30 con 0x7f2a2c09fb00 2024-09-16T03:42:31.944 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.937+0000 7f2a21ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2024-09-16T03:42:31.944 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.937+0000 7f2a21ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:42:31.944 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-09-16T03:42:31.944 INFO:teuthology.orchestra.run.smithi042.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-16T03:42:31.947 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.941+0000 7f2a339f8640 10 librados: watch_flush enter 2024-09-16T03:42:31.947 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.941+0000 7f2a339f8640 10 librados: watch_flush exit 2024-09-16T03:42:31.947 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.941+0000 7f2a339f8640 20 client.4316.objecter shutdown clearing up homeless session... 2024-09-16T03:42:31.947 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.941+0000 7f2a339f8640 10 client.4316.objecter successfully canceled tick 2024-09-16T03:42:31.947 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.941+0000 7f2a339f8640 1 -- 172.21.15.42:0/2376448011 >> v2:172.21.15.154:6812/254845342 conn(0x7f2a10037810 msgr2=0x7f2a10039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:31.947 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.941+0000 7f2a339f8640 1 --2- 172.21.15.42:0/2376448011 >> v2:172.21.15.154:6812/254845342 conn(0x7f2a10037810 0x7f2a10039cd0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f2a2800a640 tx=0x7f2a2800a350 comp rx=0 tx=0).stop 2024-09-16T03:42:31.947 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.941+0000 7f2a339f8640 10 monclient: shutdown 2024-09-16T03:42:31.947 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.941+0000 7f2a339f8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:42:31.948 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.941+0000 7f2a339f8640 1 -- 172.21.15.42:0/2376448011 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2a2c09fb00 msgr2=0x7f2a2c136b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:31.948 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.941+0000 7f2a339f8640 1 --2- 172.21.15.42:0/2376448011 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2a2c09fb00 0x7f2a2c136b60 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f2a2c09e3d0 tx=0x7f2a240195d0 comp rx=0 tx=0).stop 2024-09-16T03:42:31.948 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.941+0000 7f2a339f8640 1 -- 172.21.15.42:0/2376448011 shutdown_connections 2024-09-16T03:42:31.948 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.941+0000 7f2a339f8640 1 --2- 172.21.15.42:0/2376448011 >> v2:172.21.15.154:6812/254845342 conn(0x7f2a10037810 0x7f2a10039cd0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:31.948 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.941+0000 7f2a339f8640 1 --2- 172.21.15.42:0/2376448011 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2a2c09fb00 0x7f2a2c136b60 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:31.948 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.941+0000 7f2a339f8640 1 --2- 172.21.15.42:0/2376448011 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2a2c09c400 0x7f2a2c136620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:31.948 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.941+0000 7f2a339f8640 1 --2- 172.21.15.42:0/2376448011 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2a2c09bf90 0x7f2a2c1cccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:31.948 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.941+0000 7f2a339f8640 1 -- 172.21.15.42:0/2376448011 >> 172.21.15.42:0/2376448011 conn(0x7f2a2c0a2420 msgr2=0x7f2a2c070650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:42:31.948 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.941+0000 7f2a339f8640 1 -- 172.21.15.42:0/2376448011 shutdown_connections 2024-09-16T03:42:31.948 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.941+0000 7f2a339f8640 1 -- 172.21.15.42:0/2376448011 wait complete. 2024-09-16T03:42:31.948 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:31.941+0000 7f2a339f8640 1 librados: shutdown 2024-09-16T03:42:31.962 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-09-16T03:42:31.962 INFO:teuthology.run_tasks:Running task mon_thrash... 2024-09-16T03:42:31.992 INFO:tasks.mon_thrash:Beginning mon_thrash... 2024-09-16T03:42:31.993 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T03:42:32.110 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.101+0000 7ff3c5a7b640 10 monclient: get_monmap_and_config 2024-09-16T03:42:32.110 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.101+0000 7ff3c5a7b640 10 monclient: build_initial_monmap 2024-09-16T03:42:32.110 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.101+0000 7ff3c5a7b640 10 monclient: monmap: 2024-09-16T03:42:32.110 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:42:32.110 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:32.110 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:42:32.107947+0000 2024-09-16T03:42:32.110 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:42:32.107947+0000 2024-09-16T03:42:32.110 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:42:32.110 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:32.110 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:42:32.111 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:42:32.111 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:42:32.111 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:42:32.111 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:42:32.111 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:32.112 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.105+0000 7ff3c5a7b640 1 Processor -- start 2024-09-16T03:42:32.113 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.105+0000 7ff3c5a7b640 1 -- start start 2024-09-16T03:42:32.113 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.105+0000 7ff3c5a7b640 10 monclient: init 2024-09-16T03:42:32.113 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.105+0000 7ff3c5a7b640 10 monclient: _reopen_session rank -1 2024-09-16T03:42:32.113 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.105+0000 7ff3c5a7b640 10 monclient: _add_conns ranks=[1,3,4,2,0] 2024-09-16T03:42:32.113 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.105+0000 7ff3c5a7b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff3c00a1330 addr v1:172.21.15.154:6789/0 2024-09-16T03:42:32.113 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.105+0000 7ff3c5a7b640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff3c006dac0 0x7ff3c013a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:32.113 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.105+0000 7ff3c5a7b640 10 monclient(hunting): picked mon.noname-a con 0x7ff3c006dac0 addr v2:172.21.15.42:3300/0 2024-09-16T03:42:32.113 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.105+0000 7ff3c5a7b640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7ff3c00a1e20 0x7ff3c013c890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:32.113 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.105+0000 7ff3c5a7b640 10 monclient(hunting): picked mon.noname-c con 0x7ff3c00a1e20 addr v2:172.21.15.154:3300/0 2024-09-16T03:42:32.113 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.105+0000 7ff3c5a7b640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff3c009c090 con 0x7ff3c00a1330 2024-09-16T03:42:32.113 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.105+0000 7ff3c5a7b640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:32.113 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.105+0000 7ff3c5a7b640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7ff3c009f610 con 0x7ff3c006dac0 2024-09-16T03:42:32.114 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.105+0000 7ff3c5a7b640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:32.114 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.105+0000 7ff3c5a7b640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7ff3c009f8f0 con 0x7ff3c00a1e20 2024-09-16T03:42:32.114 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.105+0000 7ff3c5a7b640 10 monclient(hunting): _renew_subs 2024-09-16T03:42:32.114 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.105+0000 7ff3c5a7b640 10 monclient(hunting): authenticate will time out at 999.077281s 2024-09-16T03:42:32.114 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.105+0000 7ff3beffd640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff3c006dac0 0x7ff3c013a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:32.114 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.105+0000 7ff3beffd640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff3c006dac0 0x7ff3c013a340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:37878/0 (socket says 172.21.15.42:37878) 2024-09-16T03:42:32.114 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.105+0000 7ff3beffd640 1 -- 172.21.15.42:0/607163405 learned_addr learned my addr 172.21.15.42:0/607163405 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:42:32.114 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.105+0000 7ff3beffd640 10 monclient(hunting): get_auth_request con 0x7ff3c006dac0 auth_method 0 2024-09-16T03:42:32.114 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.105+0000 7ff3beffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:32.114 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.105+0000 7ff3beffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:32.114 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.105+0000 7ff3beffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:32.114 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.105+0000 7ff3beffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:32.114 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.105+0000 7ff3beffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:32.114 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.105+0000 7ff3beffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:32.115 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.105+0000 7ff3bffff640 1 --2- 172.21.15.42:0/607163405 >> v2:172.21.15.154:3300/0 conn(0x7ff3c00a1e20 0x7ff3c013c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:32.115 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.105+0000 7ff3beffd640 10 monclient(hunting): handle_auth_done global_id 4371 payload 274 2024-09-16T03:42:32.115 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.105+0000 7ff3beffd640 1 -- 172.21.15.42:0/607163405 >> v2:172.21.15.154:3300/0 conn(0x7ff3c00a1e20 msgr2=0x7ff3c013c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:42:32.115 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.105+0000 7ff3beffd640 1 --2- 172.21.15.42:0/607163405 >> v2:172.21.15.154:3300/0 conn(0x7ff3c00a1e20 0x7ff3c013c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:32.115 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.105+0000 7ff3beffd640 1 -- 172.21.15.42:0/607163405 >> v1:172.21.15.154:6789/0 conn(0x7ff3c00a1330 legacy=0x7ff3c00a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:42:32.115 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.105+0000 7ff3beffd640 10 monclient: _finish_hunting 0 2024-09-16T03:42:32.115 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.105+0000 7ff3beffd640 1 monclient: found mon.noname-a 2024-09-16T03:42:32.115 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.105+0000 7ff3beffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:42:32.115 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.105+0000 7ff3beffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:42:32.115 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.105+0000 7ff3beffd640 1 -- 172.21.15.42:0/607163405 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3c009fbd0 con 0x7ff3c006dac0 2024-09-16T03:42:32.115 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.105+0000 7ff3bffff640 10 monclient: get_auth_request con 0x7ff3c00a1e20 auth_method 0 2024-09-16T03:42:32.115 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.105+0000 7ff3bffff640 1 --2- 172.21.15.42:0/607163405 >> v2:172.21.15.154:3300/0 conn(0x7ff3c00a1e20 0x7ff3c013c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T03:42:32.120 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.105+0000 7ff3beffd640 1 --2- 172.21.15.42:0/607163405 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff3c006dac0 0x7ff3c013a340 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7ff3c009c800 tx=0x7ff3b001c430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e53cd91566b03c58 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:32.120 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.105+0000 7ff3be7fc640 1 -- 172.21.15.42:0/607163405 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3b00b0070 con 0x7ff3c006dac0 2024-09-16T03:42:32.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.105+0000 7ff3be7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:32.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.105+0000 7ff3be7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:42:32.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.105+0000 7ff3be7fc640 10 monclient: dump: 2024-09-16T03:42:32.121 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:32.121 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:32.121 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:32.121 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:32.121 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:32.121 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:32.121 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:32.121 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:32.121 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:32.121 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:32.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.105+0000 7ff3be7fc640 10 monclient: _finish_auth 0 2024-09-16T03:42:32.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.105+0000 7ff3be7fc640 10 monclient: _check_auth_tickets 2024-09-16T03:42:32.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.105+0000 7ff3be7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:42:32.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.105+0000 7ff3be7fc640 1 -- 172.21.15.42:0/607163405 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff3b00ab600 con 0x7ff3c006dac0 2024-09-16T03:42:32.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.105+0000 7ff3c5a7b640 5 monclient: authenticate success, global_id 4371 2024-09-16T03:42:32.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.105+0000 7ff3c5a7b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:42:32.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.109+0000 7ff3be7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:42:32.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.109+0000 7ff3c5a7b640 10 monclient: get_monmap_and_config success 2024-09-16T03:42:32.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.109+0000 7ff3be7fc640 1 -- 172.21.15.42:0/607163405 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3b00ab9d0 con 0x7ff3c006dac0 2024-09-16T03:42:32.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.109+0000 7ff3c5a7b640 4 set_mon_vals no callback set 2024-09-16T03:42:32.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.109+0000 7ff3be7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:32.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.109+0000 7ff3be7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:42:32.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.109+0000 7ff3be7fc640 10 monclient: dump: 2024-09-16T03:42:32.122 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:32.123 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:32.123 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:32.123 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:32.123 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:32.123 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:32.123 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:32.123 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:32.123 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:32.123 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:32.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.109+0000 7ff3c5a7b640 10 monclient: shutdown 2024-09-16T03:42:32.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.109+0000 7ff3c5a7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:42:32.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.109+0000 7ff3c5a7b640 1 -- 172.21.15.42:0/607163405 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff3c006dac0 msgr2=0x7ff3c013a340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:32.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.109+0000 7ff3c5a7b640 1 --2- 172.21.15.42:0/607163405 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff3c006dac0 0x7ff3c013a340 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7ff3c009c800 tx=0x7ff3b001c430 comp rx=0 tx=0).stop 2024-09-16T03:42:32.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.109+0000 7ff3c5a7b640 1 -- 172.21.15.42:0/607163405 shutdown_connections 2024-09-16T03:42:32.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.109+0000 7ff3c5a7b640 1 --2- 172.21.15.42:0/607163405 >> v2:172.21.15.154:3300/0 conn(0x7ff3c00a1e20 0x7ff3c013c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:32.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.109+0000 7ff3c5a7b640 1 --2- 172.21.15.42:0/607163405 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff3c006dac0 0x7ff3c013a340 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:32.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.109+0000 7ff3c5a7b640 1 -- 172.21.15.42:0/607163405 >> 172.21.15.42:0/607163405 conn(0x7ff3c00a2420 msgr2=0x7ff3c00a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:42:32.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.109+0000 7ff3c5a7b640 1 -- 172.21.15.42:0/607163405 shutdown_connections 2024-09-16T03:42:32.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.109+0000 7ff3c5a7b640 1 -- 172.21.15.42:0/607163405 wait complete. 2024-09-16T03:42:32.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.109+0000 7ff3c5a7b640 10 monclient: build_initial_monmap 2024-09-16T03:42:32.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.109+0000 7ff3c5a7b640 10 monclient: monmap: 2024-09-16T03:42:32.124 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:42:32.124 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:42:32.124 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:42:32.124 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:42:32.124 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:42:32.124 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:32.124 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:42:32.125 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:42:32.125 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:42:32.125 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:32.125 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.109+0000 7ff3c5a7b640 1 librados: starting msgr at 2024-09-16T03:42:32.125 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.109+0000 7ff3c5a7b640 1 librados: starting objecter 2024-09-16T03:42:32.125 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.109+0000 7ff3c5a7b640 1 Processor -- start 2024-09-16T03:42:32.125 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.109+0000 7ff3c5a7b640 1 -- start start 2024-09-16T03:42:32.125 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.109+0000 7ff3c5a7b640 1 librados: setting wanted keys 2024-09-16T03:42:32.125 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.109+0000 7ff3c5a7b640 1 librados: calling monclient init 2024-09-16T03:42:32.125 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.109+0000 7ff3c5a7b640 10 monclient: init 2024-09-16T03:42:32.125 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.109+0000 7ff3c5a7b640 10 monclient: _reopen_session rank -1 2024-09-16T03:42:32.125 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.109+0000 7ff3c5a7b640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T03:42:32.125 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.109+0000 7ff3c5a7b640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff3c00a1330 0x7ff3c01cee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:32.125 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.109+0000 7ff3c5a7b640 10 monclient(hunting): picked mon.noname-a con 0x7ff3c00a1330 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:42:32.126 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.109+0000 7ff3c5a7b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff3c00a1e20 0x7ff3c0135530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:32.126 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.109+0000 7ff3c5a7b640 10 monclient(hunting): picked mon.noname-b con 0x7ff3c00a1e20 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:42:32.126 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.109+0000 7ff3c5a7b640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff3c0135a70 0x7ff3c009cb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:32.126 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.109+0000 7ff3c5a7b640 10 monclient(hunting): picked mon.noname-c con 0x7ff3c0135a70 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:42:32.126 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.109+0000 7ff3c5a7b640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:32.126 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.109+0000 7ff3c5a7b640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7ff3c013f800 con 0x7ff3c00a1330 2024-09-16T03:42:32.126 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.109+0000 7ff3c5a7b640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:32.126 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.109+0000 7ff3c5a7b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff3c009ef80 con 0x7ff3c00a1e20 2024-09-16T03:42:32.126 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.109+0000 7ff3c5a7b640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:32.126 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.109+0000 7ff3c5a7b640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7ff3c013f980 con 0x7ff3c0135a70 2024-09-16T03:42:32.126 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.109+0000 7ff3c5a7b640 10 monclient(hunting): _renew_subs 2024-09-16T03:42:32.126 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.109+0000 7ff3c5a7b640 10 monclient(hunting): authenticate will time out at 999.082365s 2024-09-16T03:42:32.126 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.109+0000 7ff3bf7fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff3c00a1330 0x7ff3c01cee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:32.126 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.109+0000 7ff3bf7fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff3c00a1330 0x7ff3c01cee40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:37890/0 (socket says 172.21.15.42:37890) 2024-09-16T03:42:32.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.109+0000 7ff3bf7fe640 1 -- 172.21.15.42:0/1083304853 learned_addr learned my addr 172.21.15.42:0/1083304853 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:42:32.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.109+0000 7ff3bf7fe640 10 monclient(hunting): get_auth_request con 0x7ff3c00a1330 auth_method 0 2024-09-16T03:42:32.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.109+0000 7ff3bf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:32.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.109+0000 7ff3bf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:32.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.109+0000 7ff3bf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:32.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.109+0000 7ff3bffff640 1 --2- 172.21.15.42:0/1083304853 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff3c0135a70 0x7ff3c009cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:32.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.109+0000 7ff3bf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:32.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.109+0000 7ff3bf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:32.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.109+0000 7ff3bf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:32.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.109+0000 7ff3bffff640 10 monclient(hunting): get_auth_request con 0x7ff3c0135a70 auth_method 0 2024-09-16T03:42:32.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.109+0000 7ff3bffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:32.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.109+0000 7ff3bffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:32.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.109+0000 7ff3bffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:32.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.109+0000 7ff3bffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:32.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.109+0000 7ff3bffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:32.128 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.109+0000 7ff3bffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:32.128 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.109+0000 7ff3beffd640 1 --2- 172.21.15.42:0/1083304853 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff3c00a1e20 0x7ff3c0135530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:32.128 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.109+0000 7ff3bf7fe640 10 monclient(hunting): handle_auth_done global_id 4374 payload 824 2024-09-16T03:42:32.128 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.109+0000 7ff3bf7fe640 1 -- 172.21.15.42:0/1083304853 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff3c0135a70 msgr2=0x7ff3c009cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:32.128 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.109+0000 7ff3bf7fe640 1 --2- 172.21.15.42:0/1083304853 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff3c0135a70 0x7ff3c009cb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:32.128 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.109+0000 7ff3bf7fe640 1 -- 172.21.15.42:0/1083304853 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff3c00a1e20 msgr2=0x7ff3c0135530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:32.128 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.109+0000 7ff3bf7fe640 1 --2- 172.21.15.42:0/1083304853 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff3c00a1e20 0x7ff3c0135530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:32.128 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.109+0000 7ff3bf7fe640 10 monclient: _finish_hunting 0 2024-09-16T03:42:32.128 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.109+0000 7ff3bf7fe640 1 monclient: found mon.noname-a 2024-09-16T03:42:32.128 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.109+0000 7ff3bf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:42:32.128 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.109+0000 7ff3bf7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:42:32.128 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.109+0000 7ff3bf7fe640 1 -- 172.21.15.42:0/1083304853 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3c0073a30 con 0x7ff3c00a1330 2024-09-16T03:42:32.128 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.109+0000 7ff3beffd640 10 monclient: get_auth_request con 0x7ff3c00a1e20 auth_method 0 2024-09-16T03:42:32.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.109+0000 7ff3beffd640 1 --2- 172.21.15.42:0/1083304853 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff3c00a1e20 0x7ff3c0135530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T03:42:32.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.109+0000 7ff3bf7fe640 1 --2- 172.21.15.42:0/1083304853 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff3c00a1330 0x7ff3c01cee40 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7ff3a800c4c0 tx=0x7ff3a8001e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:32.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.109+0000 7ff3b7fff640 10 client.?.objecter ms_handle_connect 0x7ff3c00a1330 2024-09-16T03:42:32.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.109+0000 7ff3b7fff640 10 client.?.objecter resend_mon_ops 2024-09-16T03:42:32.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.109+0000 7ff3b7fff640 1 -- 172.21.15.42:0/1083304853 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3a8017070 con 0x7ff3c00a1330 2024-09-16T03:42:32.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.109+0000 7ff3b7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:32.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.109+0000 7ff3b7fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:42:32.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.109+0000 7ff3b7fff640 10 monclient: dump: 2024-09-16T03:42:32.129 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:32.129 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:32.129 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:32.129 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:32.129 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:32.129 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:32.129 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:32.130 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:32.130 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:32.130 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:32.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.109+0000 7ff3b7fff640 10 monclient: _finish_auth 0 2024-09-16T03:42:32.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.109+0000 7ff3b7fff640 10 monclient: _check_auth_tickets 2024-09-16T03:42:32.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.109+0000 7ff3b7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:42:32.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.109+0000 7ff3b7fff640 1 -- 172.21.15.42:0/1083304853 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff3a8003600 con 0x7ff3c00a1330 2024-09-16T03:42:32.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.109+0000 7ff3b7fff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:42:32.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.109+0000 7ff3b7fff640 1 -- 172.21.15.42:0/1083304853 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3a80039f0 con 0x7ff3c00a1330 2024-09-16T03:42:32.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.109+0000 7ff3b7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:32.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.109+0000 7ff3b7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:42:32.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.109+0000 7ff3b7fff640 10 monclient: dump: 2024-09-16T03:42:32.130 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:32.130 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:32.131 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:32.131 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:32.131 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:32.131 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:32.131 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:32.131 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:32.131 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:32.131 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:32.131 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.109+0000 7ff3bdffb640 4 set_mon_vals no callback set 2024-09-16T03:42:32.131 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.109+0000 7ff3c5a7b640 5 monclient: authenticate success, global_id 4374 2024-09-16T03:42:32.131 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.109+0000 7ff3c5a7b640 10 monclient: _renew_subs 2024-09-16T03:42:32.131 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.109+0000 7ff3c5a7b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:42:32.131 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.109+0000 7ff3c5a7b640 1 -- 172.21.15.42:0/1083304853 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3c013c990 con 0x7ff3c00a1330 2024-09-16T03:42:32.131 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.109+0000 7ff3c5a7b640 10 client.4374.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:42:32.132 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.109+0000 7ff3c5a7b640 10 monclient: _renew_subs 2024-09-16T03:42:32.132 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.109+0000 7ff3c5a7b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:42:32.132 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.109+0000 7ff3c5a7b640 1 -- 172.21.15.42:0/1083304853 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3c0121c00 con 0x7ff3c00a1330 2024-09-16T03:42:32.132 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.109+0000 7ff3c5a7b640 1 librados: init done 2024-09-16T03:42:32.132 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.109+0000 7ff3c5a7b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:42:32.132 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.109+0000 7ff3c5a7b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:42:32.132 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.109+0000 7ff3c5a7b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:42:32.132 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.109+0000 7ff3c5a7b640 1 -- 172.21.15.42:0/1083304853 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff3c00a1710 con 0x7ff3c00a1330 2024-09-16T03:42:32.132 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.113+0000 7ff3b7fff640 1 -- 172.21.15.42:0/1083304853 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7ff3a8003b90 con 0x7ff3c00a1330 2024-09-16T03:42:32.132 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.113+0000 7ff3b7fff640 1 --2- 172.21.15.42:0/1083304853 >> v2:172.21.15.154:6812/254845342 conn(0x7ff3a0037840 0x7ff3a0039d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:32.132 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.113+0000 7ff3b7fff640 1 -- 172.21.15.42:0/1083304853 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 3534+0+0 (secure 0 0 0) 0x7ff3a8047930 con 0x7ff3c00a1330 2024-09-16T03:42:32.132 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.113+0000 7ff3b7fff640 10 client.4374.objecter ms_dispatch 0x7ff3c0000ba0 osd_map(19..19 src has 1..19) 2024-09-16T03:42:32.132 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.113+0000 7ff3b7fff640 3 client.4374.objecter handle_osd_map got epochs [19,19] > 0 2024-09-16T03:42:32.132 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.113+0000 7ff3b7fff640 3 client.4374.objecter handle_osd_map decoding full epoch 19 2024-09-16T03:42:32.133 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.113+0000 7ff3b7fff640 20 client.4374.objecter dump_active .. 0 homeless 2024-09-16T03:42:32.133 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.113+0000 7ff3beffd640 1 --2- 172.21.15.42:0/1083304853 >> v2:172.21.15.154:6812/254845342 conn(0x7ff3a0037840 0x7ff3a0039d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:32.133 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.113+0000 7ff3beffd640 10 monclient: get_auth_request con 0x7ff3a0037840 auth_method 0 2024-09-16T03:42:32.133 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.113+0000 7ff3b7fff640 10 client.4374.objecter ms_handle_connect 0x7ff3a0037840 2024-09-16T03:42:32.133 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.113+0000 7ff3beffd640 1 --2- 172.21.15.42:0/1083304853 >> v2:172.21.15.154:6812/254845342 conn(0x7ff3a0037840 0x7ff3a0039d00 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7ff3b00ad000 tx=0x7ff3b0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:32.133 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.117+0000 7ff3b7fff640 1 -- 172.21.15.42:0/1083304853 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7ff3a8011520 con 0x7ff3c00a1330 2024-09-16T03:42:32.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.117+0000 7ff3b7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:42:32.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.117+0000 7ff3b7fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:42:32.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.273+0000 7ff3c5a7b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-16T03:42:32.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.273+0000 7ff3c5a7b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T03:42:32.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.273+0000 7ff3c5a7b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:42:32.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.273+0000 7ff3c5a7b640 1 -- 172.21.15.42:0/1083304853 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff3c0069040 con 0x7ff3c00a1330 2024-09-16T03:42:32.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.273+0000 7ff3b7fff640 1 -- 172.21.15.42:0/1083304853 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v19) ==== 74+0+10910 (secure 0 0 0) 0x7ff3a801a540 con 0x7ff3c00a1330 2024-09-16T03:42:32.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.273+0000 7ff3b7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T03:42:32.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.273+0000 7ff3b7fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:42:32.282 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-09-16T03:42:32.282 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":19,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","created":"2024-09-16T03:42:05.977277+0000","modified":"2024-09-16T03:42:30.638747+0000","last_up_change":"2024-09-16T03:42:17.544237+0000","last_in_change":"2024-09-16T03:42:08.437271+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T03:42:17.542498+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-16T03:42:23.677980+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":18,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6b9fc058-2cff-46af-b95c-b78c72e1dc2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6808","nonce":1401549752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6809","nonce":1401549752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6811","nonce":1401549752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6810","nonce":1401549752}]},"public_addr":"172.21.15.42:6808/1401549752","cluster_addr":"172.21.15.42:6809/1401549752","heartbeat_back_addr":"172.21.15.42:6811/1401549752","heartbeat_front_addr":"172.21.15.42:6810/1401549752","state":["exists","up"]},{"osd":1,"uuid":"ee31f807-cb16-4c33-8e23-2573e1a6b5c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6800","nonce":4217863575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6801","nonce":4217863575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6803","nonce":4217863575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6802","nonce":4217863575}]},"public_addr":"172.21.15.42:6800/4217863575","cluster_addr":"172.21.15.42:6801/4217863575","heartbeat_back_addr":"172.21.15.42:6803/4217863575","heartbeat_front_addr":"172.21.15.42:6802/4217863575","state":["exists","up"]},{"osd":2,"uuid":"a26f3917-6702-43db-9355-e96da86a3365","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6804","nonce":263835041}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6805","nonce":263835041}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6807","nonce":263835041}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6806","nonce":263835041}]},"public_addr":"172.21.15.42:6804/263835041","cluster_addr":"172.21.15.42:6805/263835041","heartbeat_back_addr":"172.21.15.42:6807/263835041","heartbeat_front_addr":"172.21.15.42:6806/263835041","state":["exists","up"]},{"osd":3,"uuid":"4e809f2a-2ffd-4791-9f2d-18f1bfe7d2ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":4043102363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6801","nonce":4043102363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6803","nonce":4043102363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":4043102363}]},"public_addr":"172.21.15.154:6800/4043102363","cluster_addr":"172.21.15.154:6801/4043102363","heartbeat_back_addr":"172.21.15.154:6803/4043102363","heartbeat_front_addr":"172.21.15.154:6802/4043102363","state":["exists","up"]},{"osd":4,"uuid":"dcf637ab-5f72-4ed2-a99b-69083f0b6b15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":1038452759}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6805","nonce":1038452759}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6807","nonce":1038452759}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":1038452759}]},"public_addr":"172.21.15.154:6804/1038452759","cluster_addr":"172.21.15.154:6805/1038452759","heartbeat_back_addr":"172.21.15.154:6807/1038452759","heartbeat_front_addr":"172.21.15.154:6806/1038452759","state":["exists","up"]},{"osd":5,"uuid":"83c05e44-861b-4bb2-9657-19196e5ac1f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":3026127822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6809","nonce":3026127822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6811","nonce":3026127822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":3026127822}]},"public_addr":"172.21.15.154:6808/3026127822","cluster_addr":"172.21.15.154:6809/3026127822","heartbeat_back_addr":"172.21.15.154:6811/3026127822","heartbeat_front_addr":"172.21.15.154:6810/3026127822","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:42:15.086682+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:42:14.665142+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:42:14.860865+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:42:14.620446+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:42:15.163090+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:42:15.856205+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T03:42:32.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.277+0000 7ff3c5a7b640 10 librados: watch_flush enter 2024-09-16T03:42:32.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.277+0000 7ff3c5a7b640 10 librados: watch_flush exit 2024-09-16T03:42:32.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.277+0000 7ff3c5a7b640 20 client.4374.objecter shutdown clearing up homeless session... 2024-09-16T03:42:32.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.277+0000 7ff3c5a7b640 10 client.4374.objecter successfully canceled tick 2024-09-16T03:42:32.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.277+0000 7ff3c5a7b640 1 -- 172.21.15.42:0/1083304853 >> v2:172.21.15.154:6812/254845342 conn(0x7ff3a0037840 msgr2=0x7ff3a0039d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:32.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.277+0000 7ff3c5a7b640 1 --2- 172.21.15.42:0/1083304853 >> v2:172.21.15.154:6812/254845342 conn(0x7ff3a0037840 0x7ff3a0039d00 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7ff3b00ad000 tx=0x7ff3b0008040 comp rx=0 tx=0).stop 2024-09-16T03:42:32.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.277+0000 7ff3c5a7b640 10 monclient: shutdown 2024-09-16T03:42:32.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.277+0000 7ff3c5a7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:42:32.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.277+0000 7ff3c5a7b640 1 -- 172.21.15.42:0/1083304853 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff3c00a1330 msgr2=0x7ff3c01cee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:32.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.277+0000 7ff3c5a7b640 1 --2- 172.21.15.42:0/1083304853 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff3c00a1330 0x7ff3c01cee40 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7ff3a800c4c0 tx=0x7ff3a8001e80 comp rx=0 tx=0).stop 2024-09-16T03:42:32.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.277+0000 7ff3c5a7b640 1 -- 172.21.15.42:0/1083304853 shutdown_connections 2024-09-16T03:42:32.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.277+0000 7ff3c5a7b640 1 --2- 172.21.15.42:0/1083304853 >> v2:172.21.15.154:6812/254845342 conn(0x7ff3a0037840 0x7ff3a0039d00 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:32.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.277+0000 7ff3c5a7b640 1 --2- 172.21.15.42:0/1083304853 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff3c0135a70 0x7ff3c009cb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:32.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.277+0000 7ff3c5a7b640 1 --2- 172.21.15.42:0/1083304853 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff3c00a1e20 0x7ff3c0135530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:32.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.277+0000 7ff3c5a7b640 1 --2- 172.21.15.42:0/1083304853 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff3c00a1330 0x7ff3c01cee40 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:32.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.277+0000 7ff3c5a7b640 1 -- 172.21.15.42:0/1083304853 >> 172.21.15.42:0/1083304853 conn(0x7ff3c00a2420 msgr2=0x7ff3c00a45b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:42:32.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.277+0000 7ff3c5a7b640 1 -- 172.21.15.42:0/1083304853 shutdown_connections 2024-09-16T03:42:32.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.277+0000 7ff3c5a7b640 1 -- 172.21.15.42:0/1083304853 wait complete. 2024-09-16T03:42:32.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.277+0000 7ff3c5a7b640 1 librados: shutdown 2024-09-16T03:42:32.300 INFO:tasks.mon_thrash.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-16T03:42:17.542498+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '14', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 6.059999942779541, 'score_stable': 6.059999942779541, 'optimal_score': 0.33000001311302185, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-09-16T03:42:23.677980+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 18, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-16T03:42:32.300 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-16T03:42:32.421 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.413+0000 7f44340a3640 10 monclient: get_monmap_and_config 2024-09-16T03:42:32.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.413+0000 7f44340a3640 10 monclient: build_initial_monmap 2024-09-16T03:42:32.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.413+0000 7f44340a3640 10 monclient: monmap: 2024-09-16T03:42:32.422 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:42:32.422 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:32.422 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:42:32.419722+0000 2024-09-16T03:42:32.422 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:42:32.419722+0000 2024-09-16T03:42:32.422 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:42:32.422 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:32.422 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:42:32.422 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:42:32.422 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:42:32.422 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:42:32.423 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:42:32.423 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:32.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.417+0000 7f44340a3640 1 Processor -- start 2024-09-16T03:42:32.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.417+0000 7f44340a3640 1 -- start start 2024-09-16T03:42:32.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.417+0000 7f44340a3640 10 monclient: init 2024-09-16T03:42:32.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.417+0000 7f44340a3640 10 monclient: _reopen_session rank -1 2024-09-16T03:42:32.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.417+0000 7f44340a3640 10 monclient: _add_conns ranks=[2,3,0,4,1] 2024-09-16T03:42:32.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.417+0000 7f44340a3640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f442c0a4720 0x7f442c0a2500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:32.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.417+0000 7f44340a3640 10 monclient(hunting): picked mon.noname-b con 0x7f442c0a4720 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:42:32.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.417+0000 7f44340a3640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f442c0a4b00 0x7f442c0a4f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:32.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.417+0000 7f44340a3640 10 monclient(hunting): picked mon.noname-a con 0x7f442c0a4b00 addr v2:172.21.15.42:3300/0 2024-09-16T03:42:32.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.417+0000 7f44340a3640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f442c1374e0 addr v1:172.21.15.42:6789/0 2024-09-16T03:42:32.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.417+0000 7f44340a3640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f442c0a5770 con 0x7f442c1374e0 2024-09-16T03:42:32.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.417+0000 7f44340a3640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:32.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.417+0000 7f44340a3640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f442c0a12c0 con 0x7f442c0a4b00 2024-09-16T03:42:32.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.417+0000 7f44340a3640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:32.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.417+0000 7f44340a3640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f442c06ec40 con 0x7f442c0a4720 2024-09-16T03:42:32.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.417+0000 7f44340a3640 10 monclient(hunting): _renew_subs 2024-09-16T03:42:32.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.417+0000 7f44340a3640 10 monclient(hunting): authenticate will time out at 999.389359s 2024-09-16T03:42:32.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.417+0000 7f4431e3f640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f442c0a4720 0x7f442c0a2500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:32.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.417+0000 7f443163e640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f442c0a4b00 0x7f442c0a4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:32.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.417+0000 7f4432640640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f442c1374e0 0x7f442c137990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:46838/0 (socket says 172.21.15.42:46838) 2024-09-16T03:42:32.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.417+0000 7f4431e3f640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f442c0a4720 0x7f442c0a2500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:41586/0 (socket says 172.21.15.42:41586) 2024-09-16T03:42:32.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.417+0000 7f4432640640 1 -- 172.21.15.42:0/2671517293 learned_addr learned my addr 172.21.15.42:0/2671517293 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:42:32.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.417+0000 7f4431e3f640 10 monclient(hunting): get_auth_request con 0x7f442c0a4720 auth_method 0 2024-09-16T03:42:32.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.417+0000 7f4431e3f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:32.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.417+0000 7f4431e3f640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:32.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.417+0000 7f4431e3f640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:32.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.417+0000 7f443163e640 10 monclient(hunting): get_auth_request con 0x7f442c0a4b00 auth_method 0 2024-09-16T03:42:32.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.417+0000 7f443163e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:32.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.417+0000 7f443163e640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:32.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.417+0000 7f443163e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:32.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.417+0000 7f4431e3f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:32.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.417+0000 7f4431e3f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:32.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.417+0000 7f4431e3f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:32.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.417+0000 7f443163e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:32.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.417+0000 7f443163e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:32.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.417+0000 7f443163e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:32.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.417+0000 7f4430e3d640 1 -- 172.21.15.42:0/2671517293 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 823158949 0 0) 0x7f442c0a5770 con 0x7f442c1374e0 2024-09-16T03:42:32.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.417+0000 7f4430e3d640 10 monclient(hunting): _negotiate 2024-09-16T03:42:32.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.417+0000 7f4430e3d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:32.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.417+0000 7f4430e3d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:32.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.417+0000 7f4430e3d640 10 monclient(hunting): my global_id is 4380 2024-09-16T03:42:32.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.417+0000 7f4430e3d640 1 -- 172.21.15.42:0/2671517293 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4418003530 con 0x7f442c1374e0 2024-09-16T03:42:32.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.417+0000 7f4431e3f640 10 monclient(hunting): handle_auth_done global_id 4322 payload 274 2024-09-16T03:42:32.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.417+0000 7f4431e3f640 1 -- 172.21.15.42:0/2671517293 >> v2:172.21.15.42:3300/0 conn(0x7f442c0a4b00 msgr2=0x7f442c0a4f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:42:32.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.417+0000 7f4431e3f640 1 --2- 172.21.15.42:0/2671517293 >> v2:172.21.15.42:3300/0 conn(0x7f442c0a4b00 0x7f442c0a4f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:32.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.417+0000 7f4431e3f640 1 -- 172.21.15.42:0/2671517293 >> v1:172.21.15.42:6789/0 conn(0x7f442c1374e0 legacy=0x7f442c137990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:32.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.417+0000 7f4431e3f640 10 monclient: _finish_hunting 0 2024-09-16T03:42:32.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.417+0000 7f4431e3f640 1 monclient: found mon.noname-b 2024-09-16T03:42:32.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.417+0000 7f4431e3f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:42:32.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.417+0000 7f4431e3f640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T03:42:32.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.417+0000 7f4431e3f640 1 -- 172.21.15.42:0/2671517293 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f442c06f080 con 0x7f442c0a4720 2024-09-16T03:42:32.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.417+0000 7f443163e640 1 --2- 172.21.15.42:0/2671517293 >> v2:172.21.15.42:3300/0 conn(0x7f442c0a4b00 0x7f442c0a4f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:42:32.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.417+0000 7f4431e3f640 1 --2- 172.21.15.42:0/2671517293 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f442c0a4720 0x7f442c0a2500 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f44280097c0 tx=0x7f442801c360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f57f8f733d4b5fd8 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:32.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.421+0000 7f4430e3d640 1 -- 172.21.15.42:0/2671517293 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44280b0070 con 0x7f442c0a4720 2024-09-16T03:42:32.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.421+0000 7f4430e3d640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:32.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.421+0000 7f4430e3d640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:42:32.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.421+0000 7f4430e3d640 10 monclient: dump: 2024-09-16T03:42:32.434 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:32.434 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:32.434 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:32.434 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:32.434 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:32.434 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:32.435 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:32.435 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:32.435 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:32.435 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:32.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.421+0000 7f4430e3d640 10 monclient: _finish_auth 0 2024-09-16T03:42:32.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.421+0000 7f4430e3d640 10 monclient: _check_auth_tickets 2024-09-16T03:42:32.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.421+0000 7f4430e3d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:42:32.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.421+0000 7f44340a3640 5 monclient: authenticate success, global_id 4322 2024-09-16T03:42:32.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.421+0000 7f44340a3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:42:32.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.421+0000 7f4430e3d640 1 -- 172.21.15.42:0/2671517293 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f44280ab600 con 0x7f442c0a4720 2024-09-16T03:42:32.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.421+0000 7f4430e3d640 10 monclient: handle_config config(0 keys) 2024-09-16T03:42:32.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.421+0000 7f44340a3640 10 monclient: get_monmap_and_config success 2024-09-16T03:42:32.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.421+0000 7f4430e3d640 1 -- 172.21.15.42:0/2671517293 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44280ab9b0 con 0x7f442c0a4720 2024-09-16T03:42:32.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.421+0000 7f4430e3d640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:32.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.421+0000 7f44340a3640 4 set_mon_vals no callback set 2024-09-16T03:42:32.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.421+0000 7f4430e3d640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:42:32.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.421+0000 7f4430e3d640 10 monclient: dump: 2024-09-16T03:42:32.436 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:32.436 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:32.436 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:32.437 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:32.437 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:32.437 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:32.437 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:32.437 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:32.437 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:32.437 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:32.437 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.421+0000 7f44340a3640 10 monclient: shutdown 2024-09-16T03:42:32.437 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.421+0000 7f44340a3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:42:32.437 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.421+0000 7f44340a3640 1 -- 172.21.15.42:0/2671517293 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f442c0a4720 msgr2=0x7f442c0a2500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:32.437 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.421+0000 7f44340a3640 1 --2- 172.21.15.42:0/2671517293 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f442c0a4720 0x7f442c0a2500 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f44280097c0 tx=0x7f442801c360 comp rx=0 tx=0).stop 2024-09-16T03:42:32.437 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.421+0000 7f44340a3640 1 -- 172.21.15.42:0/2671517293 shutdown_connections 2024-09-16T03:42:32.438 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.421+0000 7f44340a3640 1 --2- 172.21.15.42:0/2671517293 >> v2:172.21.15.42:3300/0 conn(0x7f442c0a4b00 0x7f442c0a4f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:32.438 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.421+0000 7f44340a3640 1 --2- 172.21.15.42:0/2671517293 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f442c0a4720 0x7f442c0a2500 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:32.438 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.421+0000 7f44340a3640 1 -- 172.21.15.42:0/2671517293 >> 172.21.15.42:0/2671517293 conn(0x7f442c0aa320 msgr2=0x7f442c0a6820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:42:32.438 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.421+0000 7f44340a3640 1 -- 172.21.15.42:0/2671517293 shutdown_connections 2024-09-16T03:42:32.438 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.421+0000 7f44340a3640 1 -- 172.21.15.42:0/2671517293 wait complete. 2024-09-16T03:42:32.438 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.421+0000 7f44340a3640 10 monclient: build_initial_monmap 2024-09-16T03:42:32.438 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.421+0000 7f44340a3640 10 monclient: monmap: 2024-09-16T03:42:32.438 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:42:32.438 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:42:32.438 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:42:32.438 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:42:32.438 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:42:32.438 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:32.438 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:42:32.439 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:42:32.439 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:42:32.439 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:32.439 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.421+0000 7f44340a3640 1 librados: starting msgr at 2024-09-16T03:42:32.439 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.421+0000 7f44340a3640 1 librados: starting objecter 2024-09-16T03:42:32.439 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.421+0000 7f44340a3640 1 Processor -- start 2024-09-16T03:42:32.439 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.421+0000 7f44340a3640 1 -- start start 2024-09-16T03:42:32.439 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.421+0000 7f44340a3640 1 librados: setting wanted keys 2024-09-16T03:42:32.439 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.421+0000 7f44340a3640 1 librados: calling monclient init 2024-09-16T03:42:32.439 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.421+0000 7f44340a3640 10 monclient: init 2024-09-16T03:42:32.439 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.421+0000 7f44340a3640 10 monclient: _reopen_session rank -1 2024-09-16T03:42:32.439 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.421+0000 7f44340a3640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T03:42:32.439 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.421+0000 7f44340a3640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f442c0a4720 0x7f442c135d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:32.440 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.421+0000 7f44340a3640 10 monclient(hunting): picked mon.noname-a con 0x7f442c0a4720 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:42:32.440 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.421+0000 7f44340a3640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f442c0a4b00 0x7f442c136290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:32.440 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.421+0000 7f44340a3640 10 monclient(hunting): picked mon.noname-b con 0x7f442c0a4b00 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:42:32.440 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.421+0000 7f44340a3640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f442c1374e0 0x7f442c13c090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:32.440 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.421+0000 7f44340a3640 10 monclient(hunting): picked mon.noname-c con 0x7f442c1374e0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:42:32.440 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.421+0000 7f44340a3640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:32.440 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.421+0000 7f44340a3640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f442c1420e0 con 0x7f442c0a4720 2024-09-16T03:42:32.440 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.421+0000 7f44340a3640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:32.440 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.421+0000 7f44340a3640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f442c141f60 con 0x7f442c0a4b00 2024-09-16T03:42:32.440 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.421+0000 7f44340a3640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:32.440 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.421+0000 7f44340a3640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f442c142260 con 0x7f442c1374e0 2024-09-16T03:42:32.440 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.421+0000 7f44340a3640 10 monclient(hunting): _renew_subs 2024-09-16T03:42:32.440 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.421+0000 7f44340a3640 10 monclient(hunting): authenticate will time out at 999.394613s 2024-09-16T03:42:32.440 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.421+0000 7f4432640640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f442c1374e0 0x7f442c13c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:32.441 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.421+0000 7f4431e3f640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f442c0a4720 0x7f442c135d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:32.441 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.421+0000 7f4431e3f640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f442c0a4720 0x7f442c135d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:37902/0 (socket says 172.21.15.42:37902) 2024-09-16T03:42:32.441 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.421+0000 7f4431e3f640 1 -- 172.21.15.42:0/844468000 learned_addr learned my addr 172.21.15.42:0/844468000 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:42:32.441 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.421+0000 7f4432640640 10 monclient(hunting): get_auth_request con 0x7f442c1374e0 auth_method 0 2024-09-16T03:42:32.441 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.421+0000 7f4432640640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:32.441 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.421+0000 7f4432640640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:32.441 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.421+0000 7f4432640640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:32.441 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.421+0000 7f4431e3f640 10 monclient(hunting): get_auth_request con 0x7f442c0a4720 auth_method 0 2024-09-16T03:42:32.441 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.421+0000 7f4431e3f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:32.441 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.421+0000 7f4431e3f640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:32.441 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.421+0000 7f4431e3f640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:32.441 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.421+0000 7f4432640640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:32.441 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.421+0000 7f4432640640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:32.442 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.421+0000 7f4432640640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:32.442 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.421+0000 7f4431e3f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:32.442 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.421+0000 7f4431e3f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:32.442 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.421+0000 7f4431e3f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:32.442 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.421+0000 7f4432640640 10 monclient(hunting): handle_auth_done global_id 4325 payload 824 2024-09-16T03:42:32.442 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.421+0000 7f443163e640 1 --2- 172.21.15.42:0/844468000 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f442c0a4b00 0x7f442c136290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:32.442 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.421+0000 7f4432640640 1 -- 172.21.15.42:0/844468000 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f442c0a4b00 msgr2=0x7f442c136290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:32.442 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.421+0000 7f4432640640 1 --2- 172.21.15.42:0/844468000 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f442c0a4b00 0x7f442c136290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:32.442 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.421+0000 7f4432640640 1 -- 172.21.15.42:0/844468000 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f442c0a4720 msgr2=0x7f442c135d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:32.442 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.421+0000 7f4432640640 1 --2- 172.21.15.42:0/844468000 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f442c0a4720 0x7f442c135d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:32.442 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.421+0000 7f4432640640 10 monclient: _finish_hunting 0 2024-09-16T03:42:32.442 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.421+0000 7f4432640640 1 monclient: found mon.noname-c 2024-09-16T03:42:32.442 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.421+0000 7f4432640640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:42:32.442 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.421+0000 7f4432640640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:42:32.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.421+0000 7f4432640640 1 -- 172.21.15.42:0/844468000 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f442c1209d0 con 0x7f442c1374e0 2024-09-16T03:42:32.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.421+0000 7f4431e3f640 1 --2- 172.21.15.42:0/844468000 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f442c0a4720 0x7f442c135d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:42:32.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.421+0000 7f4432640640 1 --2- 172.21.15.42:0/844468000 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f442c1374e0 0x7f442c13c090 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f442400caa0 tx=0x7f442400cf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:32.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.421+0000 7f44227fc640 10 client.?.objecter ms_handle_connect 0x7f442c1374e0 2024-09-16T03:42:32.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.421+0000 7f44227fc640 10 client.?.objecter resend_mon_ops 2024-09-16T03:42:32.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.421+0000 7f44227fc640 1 -- 172.21.15.42:0/844468000 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4424017070 con 0x7f442c1374e0 2024-09-16T03:42:32.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.421+0000 7f44227fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:32.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.421+0000 7f44227fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:42:32.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.421+0000 7f44227fc640 10 monclient: dump: 2024-09-16T03:42:32.443 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:32.443 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:32.443 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:32.443 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:32.443 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:32.444 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:32.444 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:32.444 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:32.444 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:32.444 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:32.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.421+0000 7f44227fc640 10 monclient: _finish_auth 0 2024-09-16T03:42:32.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.421+0000 7f44227fc640 10 monclient: _check_auth_tickets 2024-09-16T03:42:32.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.421+0000 7f44227fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:42:32.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.421+0000 7f44340a3640 5 monclient: authenticate success, global_id 4325 2024-09-16T03:42:32.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.421+0000 7f44340a3640 10 monclient: _renew_subs 2024-09-16T03:42:32.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.421+0000 7f44227fc640 1 -- 172.21.15.42:0/844468000 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f442400ac00 con 0x7f442c1374e0 2024-09-16T03:42:32.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.421+0000 7f44340a3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:42:32.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.421+0000 7f44340a3640 1 -- 172.21.15.42:0/844468000 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f442c06e1e0 con 0x7f442c1374e0 2024-09-16T03:42:32.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.421+0000 7f44227fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:42:32.445 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.421+0000 7f44227fc640 1 -- 172.21.15.42:0/844468000 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4424012400 con 0x7f442c1374e0 2024-09-16T03:42:32.445 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.421+0000 7f44227fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:32.445 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.421+0000 7f44227fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:42:32.445 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.421+0000 7f4423fff640 4 set_mon_vals no callback set 2024-09-16T03:42:32.445 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.421+0000 7f44340a3640 10 client.4325.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:42:32.445 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.421+0000 7f44227fc640 10 monclient: dump: 2024-09-16T03:42:32.445 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:32.445 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:32.445 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:32.446 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:32.446 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:32.446 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:32.446 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:32.446 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:32.446 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:32.446 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:32.446 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.421+0000 7f44340a3640 10 monclient: _renew_subs 2024-09-16T03:42:32.446 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.421+0000 7f44340a3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:42:32.446 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.421+0000 7f44340a3640 1 -- 172.21.15.42:0/844468000 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f442c1206f0 con 0x7f442c1374e0 2024-09-16T03:42:32.446 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.421+0000 7f44340a3640 1 librados: init done 2024-09-16T03:42:32.446 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.425+0000 7f44227fc640 1 -- 172.21.15.42:0/844468000 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f442401a070 con 0x7f442c1374e0 2024-09-16T03:42:32.446 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.425+0000 7f44340a3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:42:32.446 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.425+0000 7f44340a3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:42:32.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.425+0000 7f44340a3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:42:32.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.425+0000 7f44340a3640 1 -- 172.21.15.42:0/844468000 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f442c0a2500 con 0x7f442c1374e0 2024-09-16T03:42:32.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.425+0000 7f44227fc640 1 --2- 172.21.15.42:0/844468000 >> v2:172.21.15.154:6812/254845342 conn(0x7f4410037810 0x7f4410039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:32.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.425+0000 7f44227fc640 1 -- 172.21.15.42:0/844468000 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(19..19 src has 1..19) ==== 3534+0+0 (secure 0 0 0) 0x7f4424048410 con 0x7f442c1374e0 2024-09-16T03:42:32.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.425+0000 7f44227fc640 10 client.4325.objecter ms_dispatch 0x7f442c000ba0 osd_map(19..19 src has 1..19) 2024-09-16T03:42:32.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.425+0000 7f44227fc640 3 client.4325.objecter handle_osd_map got epochs [19,19] > 0 2024-09-16T03:42:32.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.425+0000 7f44227fc640 3 client.4325.objecter handle_osd_map decoding full epoch 19 2024-09-16T03:42:32.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.425+0000 7f44227fc640 20 client.4325.objecter dump_active .. 0 homeless 2024-09-16T03:42:32.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.425+0000 7f4431e3f640 1 --2- 172.21.15.42:0/844468000 >> v2:172.21.15.154:6812/254845342 conn(0x7f4410037810 0x7f4410039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:32.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.425+0000 7f4431e3f640 10 monclient: get_auth_request con 0x7f4410037810 auth_method 0 2024-09-16T03:42:32.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.425+0000 7f4431e3f640 1 --2- 172.21.15.42:0/844468000 >> v2:172.21.15.154:6812/254845342 conn(0x7f4410037810 0x7f4410039cd0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f44280097c0 tx=0x7f4428008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:32.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.425+0000 7f44227fc640 10 client.4325.objecter ms_handle_connect 0x7f4410037810 2024-09-16T03:42:32.448 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.429+0000 7f44227fc640 1 -- 172.21.15.42:0/844468000 <== mon.2 v2:172.21.15.42:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f44240125a0 con 0x7f442c1374e0 2024-09-16T03:42:32.448 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.429+0000 7f44227fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:42:32.448 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.429+0000 7f44227fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:42:32.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.589+0000 7f44340a3640 10 start_mon_command cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}] 2024-09-16T03:42:32.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.589+0000 7f44340a3640 10 monclient: _send_command 2 [{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}] 2024-09-16T03:42:32.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.589+0000 7f44340a3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:42:32.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.589+0000 7f44340a3640 1 -- 172.21.15.42:0/844468000 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f442c13e610 con 0x7f442c1374e0 2024-09-16T03:42:32.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.589+0000 7f44227fc640 1 -- 172.21.15.42:0/844468000 <== mon.2 v2:172.21.15.42:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v19) ==== 93+0+10 (secure 0 0 0) 0x7f4424047230 con 0x7f442c1374e0 2024-09-16T03:42:32.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.589+0000 7f44227fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}] 2024-09-16T03:42:32.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.589+0000 7f44227fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:42:32.595 INFO:teuthology.orchestra.run.smithi042.stdout:pg_num: 1 2024-09-16T03:42:32.598 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.589+0000 7f44340a3640 10 librados: watch_flush enter 2024-09-16T03:42:32.598 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.589+0000 7f44340a3640 10 librados: watch_flush exit 2024-09-16T03:42:32.598 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.589+0000 7f44340a3640 20 client.4325.objecter shutdown clearing up homeless session... 2024-09-16T03:42:32.598 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.589+0000 7f44340a3640 10 client.4325.objecter successfully canceled tick 2024-09-16T03:42:32.598 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.589+0000 7f44340a3640 1 -- 172.21.15.42:0/844468000 >> v2:172.21.15.154:6812/254845342 conn(0x7f4410037810 msgr2=0x7f4410039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:32.598 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.589+0000 7f44340a3640 1 --2- 172.21.15.42:0/844468000 >> v2:172.21.15.154:6812/254845342 conn(0x7f4410037810 0x7f4410039cd0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f44280097c0 tx=0x7f4428008040 comp rx=0 tx=0).stop 2024-09-16T03:42:32.598 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.589+0000 7f44340a3640 10 monclient: shutdown 2024-09-16T03:42:32.598 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.589+0000 7f44340a3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:42:32.599 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.589+0000 7f44340a3640 1 -- 172.21.15.42:0/844468000 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f442c1374e0 msgr2=0x7f442c13c090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:32.599 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.589+0000 7f44340a3640 1 --2- 172.21.15.42:0/844468000 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f442c1374e0 0x7f442c13c090 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f442400caa0 tx=0x7f442400cf60 comp rx=0 tx=0).stop 2024-09-16T03:42:32.599 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.589+0000 7f44340a3640 1 -- 172.21.15.42:0/844468000 shutdown_connections 2024-09-16T03:42:32.599 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.589+0000 7f44340a3640 1 --2- 172.21.15.42:0/844468000 >> v2:172.21.15.154:6812/254845342 conn(0x7f4410037810 0x7f4410039cd0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:32.599 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.589+0000 7f44340a3640 1 --2- 172.21.15.42:0/844468000 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f442c1374e0 0x7f442c13c090 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:32.599 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.589+0000 7f44340a3640 1 --2- 172.21.15.42:0/844468000 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f442c0a4b00 0x7f442c136290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:32.599 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.589+0000 7f44340a3640 1 --2- 172.21.15.42:0/844468000 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f442c0a4720 0x7f442c135d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:32.599 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.589+0000 7f44340a3640 1 -- 172.21.15.42:0/844468000 >> 172.21.15.42:0/844468000 conn(0x7f442c0aa320 msgr2=0x7f442c0a04a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:42:32.599 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.589+0000 7f44340a3640 1 -- 172.21.15.42:0/844468000 shutdown_connections 2024-09-16T03:42:32.599 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.589+0000 7f44340a3640 1 -- 172.21.15.42:0/844468000 wait complete. 2024-09-16T03:42:32.599 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.589+0000 7f44340a3640 1 librados: shutdown 2024-09-16T03:42:32.613 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2024-09-16T03:42:32.729 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.721+0000 7f1b473c0640 10 monclient: get_monmap_and_config 2024-09-16T03:42:32.729 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.721+0000 7f1b473c0640 10 monclient: build_initial_monmap 2024-09-16T03:42:32.729 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.721+0000 7f1b473c0640 10 monclient: monmap: 2024-09-16T03:42:32.729 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:42:32.729 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:32.729 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:42:32.726981+0000 2024-09-16T03:42:32.729 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:42:32.726981+0000 2024-09-16T03:42:32.729 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:42:32.729 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:32.730 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:42:32.730 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:42:32.730 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:42:32.730 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:42:32.730 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:42:32.730 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:32.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.725+0000 7f1b473c0640 1 Processor -- start 2024-09-16T03:42:32.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.725+0000 7f1b473c0640 1 -- start start 2024-09-16T03:42:32.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.725+0000 7f1b473c0640 10 monclient: init 2024-09-16T03:42:32.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.725+0000 7f1b473c0640 10 monclient: _reopen_session rank -1 2024-09-16T03:42:32.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.725+0000 7f1b473c0640 10 monclient: _add_conns ranks=[2,3,4,1,0] 2024-09-16T03:42:32.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.725+0000 7f1b473c0640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1b400a4720 0x7f1b400a2500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:32.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.725+0000 7f1b473c0640 10 monclient(hunting): picked mon.noname-b con 0x7f1b400a4720 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:42:32.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.725+0000 7f1b473c0640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f1b400a4b00 0x7f1b400a4f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:32.733 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.725+0000 7f1b473c0640 10 monclient(hunting): picked mon.noname-a con 0x7f1b400a4b00 addr v2:172.21.15.42:3300/0 2024-09-16T03:42:32.733 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.725+0000 7f1b473c0640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f1b401374e0 0x7f1b40137990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:32.733 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.725+0000 7f1b473c0640 10 monclient(hunting): picked mon.noname-c con 0x7f1b401374e0 addr v2:172.21.15.154:3300/0 2024-09-16T03:42:32.733 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.725+0000 7f1b473c0640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:32.733 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.725+0000 7f1b473c0640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f1b400a5770 con 0x7f1b400a4b00 2024-09-16T03:42:32.733 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.725+0000 7f1b473c0640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:32.733 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.725+0000 7f1b473c0640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f1b400a12c0 con 0x7f1b401374e0 2024-09-16T03:42:32.733 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.725+0000 7f1b473c0640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:32.733 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.725+0000 7f1b473c0640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f1b4006ec40 con 0x7f1b400a4720 2024-09-16T03:42:32.733 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.725+0000 7f1b473c0640 10 monclient(hunting): _renew_subs 2024-09-16T03:42:32.733 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.725+0000 7f1b473c0640 10 monclient(hunting): authenticate will time out at 999.697076s 2024-09-16T03:42:32.733 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.725+0000 7f1b4495b640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f1b400a4b00 0x7f1b400a4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:32.733 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.725+0000 7f1b4515c640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1b400a4720 0x7f1b400a2500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:32.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.725+0000 7f1b4495b640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f1b400a4b00 0x7f1b400a4f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:37904/0 (socket says 172.21.15.42:37904) 2024-09-16T03:42:32.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.725+0000 7f1b4495b640 1 -- 172.21.15.42:0/2265101270 learned_addr learned my addr 172.21.15.42:0/2265101270 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:42:32.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.725+0000 7f1b4495b640 10 monclient(hunting): get_auth_request con 0x7f1b400a4b00 auth_method 0 2024-09-16T03:42:32.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.725+0000 7f1b4495b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:32.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.725+0000 7f1b4495b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:32.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.725+0000 7f1b4495b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:32.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.725+0000 7f1b4515c640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1b400a4720 0x7f1b400a2500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:41610/0 (socket says 172.21.15.42:41610) 2024-09-16T03:42:32.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.725+0000 7f1b4515c640 10 monclient(hunting): get_auth_request con 0x7f1b400a4720 auth_method 0 2024-09-16T03:42:32.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.725+0000 7f1b4515c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:32.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.725+0000 7f1b4515c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:32.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.725+0000 7f1b4515c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:32.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.725+0000 7f1b4495b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:32.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.725+0000 7f1b4495b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:32.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.725+0000 7f1b4495b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:32.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.725+0000 7f1b4515c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:32.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.725+0000 7f1b4515c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:32.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.725+0000 7f1b4595d640 1 --2- 172.21.15.42:0/2265101270 >> v2:172.21.15.154:3300/0 conn(0x7f1b401374e0 0x7f1b40137990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:32.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.725+0000 7f1b4515c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:32.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.725+0000 7f1b4595d640 10 monclient(hunting): get_auth_request con 0x7f1b401374e0 auth_method 0 2024-09-16T03:42:32.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.725+0000 7f1b4595d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:32.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.725+0000 7f1b4595d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:32.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.725+0000 7f1b4595d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:32.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.725+0000 7f1b4495b640 10 monclient(hunting): handle_auth_done global_id 4386 payload 274 2024-09-16T03:42:32.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.725+0000 7f1b4495b640 1 -- 172.21.15.42:0/2265101270 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1b400a4720 msgr2=0x7f1b400a2500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:42:32.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.725+0000 7f1b4495b640 1 --2- 172.21.15.42:0/2265101270 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1b400a4720 0x7f1b400a2500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:32.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.725+0000 7f1b4495b640 1 -- 172.21.15.42:0/2265101270 >> v2:172.21.15.154:3300/0 conn(0x7f1b401374e0 msgr2=0x7f1b40137990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:42:32.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.725+0000 7f1b4495b640 1 --2- 172.21.15.42:0/2265101270 >> v2:172.21.15.154:3300/0 conn(0x7f1b401374e0 0x7f1b40137990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:32.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.725+0000 7f1b4495b640 10 monclient: _finish_hunting 0 2024-09-16T03:42:32.736 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.725+0000 7f1b4495b640 1 monclient: found mon.noname-a 2024-09-16T03:42:32.736 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.725+0000 7f1b4495b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:42:32.736 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.725+0000 7f1b4495b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:42:32.736 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.725+0000 7f1b4495b640 1 -- 172.21.15.42:0/2265101270 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b4006f080 con 0x7f1b400a4b00 2024-09-16T03:42:32.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.725+0000 7f1b4495b640 1 --2- 172.21.15.42:0/2265101270 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1b400a4b00 0x7f1b400a4f00 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f1b400a66a0 tx=0x7f1b3001c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca8f4cfce1d51866 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:32.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.725+0000 7f1b37fff640 1 -- 172.21.15.42:0/2265101270 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b300b0070 con 0x7f1b400a4b00 2024-09-16T03:42:32.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.725+0000 7f1b37fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:32.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.725+0000 7f1b37fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:42:32.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.725+0000 7f1b37fff640 10 monclient: dump: 2024-09-16T03:42:32.741 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:32.742 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:32.742 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:32.742 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:32.742 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:32.742 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:32.742 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:32.742 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:32.742 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:32.742 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:32.742 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.725+0000 7f1b37fff640 10 monclient: _finish_auth 0 2024-09-16T03:42:32.742 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.725+0000 7f1b37fff640 10 monclient: _check_auth_tickets 2024-09-16T03:42:32.742 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.725+0000 7f1b37fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:42:32.742 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.725+0000 7f1b37fff640 1 -- 172.21.15.42:0/2265101270 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1b300ab600 con 0x7f1b400a4b00 2024-09-16T03:42:32.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.725+0000 7f1b473c0640 5 monclient: authenticate success, global_id 4386 2024-09-16T03:42:32.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.725+0000 7f1b473c0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:42:32.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.725+0000 7f1b37fff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:42:32.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.725+0000 7f1b473c0640 10 monclient: get_monmap_and_config success 2024-09-16T03:42:32.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.725+0000 7f1b37fff640 1 -- 172.21.15.42:0/2265101270 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b300ab9d0 con 0x7f1b400a4b00 2024-09-16T03:42:32.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.725+0000 7f1b473c0640 4 set_mon_vals no callback set 2024-09-16T03:42:32.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.725+0000 7f1b37fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:32.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.725+0000 7f1b37fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:42:32.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.725+0000 7f1b37fff640 10 monclient: dump: 2024-09-16T03:42:32.743 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:32.743 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:32.743 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:32.743 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:32.744 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:32.744 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:32.744 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:32.744 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:32.744 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:32.744 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:32.744 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.729+0000 7f1b473c0640 10 monclient: shutdown 2024-09-16T03:42:32.744 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.729+0000 7f1b473c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:42:32.744 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.729+0000 7f1b473c0640 1 -- 172.21.15.42:0/2265101270 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1b400a4b00 msgr2=0x7f1b400a4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:32.744 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.729+0000 7f1b473c0640 1 --2- 172.21.15.42:0/2265101270 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1b400a4b00 0x7f1b400a4f00 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f1b400a66a0 tx=0x7f1b3001c3e0 comp rx=0 tx=0).stop 2024-09-16T03:42:32.744 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.729+0000 7f1b473c0640 1 -- 172.21.15.42:0/2265101270 shutdown_connections 2024-09-16T03:42:32.744 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.729+0000 7f1b473c0640 1 --2- 172.21.15.42:0/2265101270 >> v2:172.21.15.154:3300/0 conn(0x7f1b401374e0 0x7f1b40137990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:32.744 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.729+0000 7f1b473c0640 1 --2- 172.21.15.42:0/2265101270 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1b400a4b00 0x7f1b400a4f00 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:32.744 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.729+0000 7f1b473c0640 1 --2- 172.21.15.42:0/2265101270 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1b400a4720 0x7f1b400a2500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:32.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.729+0000 7f1b473c0640 1 -- 172.21.15.42:0/2265101270 >> 172.21.15.42:0/2265101270 conn(0x7f1b400aa320 msgr2=0x7f1b400a6820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:42:32.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.729+0000 7f1b473c0640 1 -- 172.21.15.42:0/2265101270 shutdown_connections 2024-09-16T03:42:32.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.729+0000 7f1b473c0640 1 -- 172.21.15.42:0/2265101270 wait complete. 2024-09-16T03:42:32.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.729+0000 7f1b473c0640 10 monclient: build_initial_monmap 2024-09-16T03:42:32.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.729+0000 7f1b473c0640 10 monclient: monmap: 2024-09-16T03:42:32.745 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:42:32.745 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:42:32.745 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:42:32.745 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:42:32.745 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:42:32.745 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:32.745 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:42:32.745 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:42:32.745 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:42:32.745 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:32.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.729+0000 7f1b473c0640 1 librados: starting msgr at 2024-09-16T03:42:32.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.729+0000 7f1b473c0640 1 librados: starting objecter 2024-09-16T03:42:32.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.729+0000 7f1b473c0640 1 Processor -- start 2024-09-16T03:42:32.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.729+0000 7f1b473c0640 1 -- start start 2024-09-16T03:42:32.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.729+0000 7f1b473c0640 1 librados: setting wanted keys 2024-09-16T03:42:32.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.729+0000 7f1b473c0640 1 librados: calling monclient init 2024-09-16T03:42:32.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.729+0000 7f1b473c0640 10 monclient: init 2024-09-16T03:42:32.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.729+0000 7f1b473c0640 10 monclient: _reopen_session rank -1 2024-09-16T03:42:32.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.729+0000 7f1b473c0640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:42:32.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.729+0000 7f1b473c0640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1b400a4720 0x7f1b40120910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:32.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.729+0000 7f1b473c0640 10 monclient(hunting): picked mon.noname-c con 0x7f1b400a4720 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:42:32.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.729+0000 7f1b473c0640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1b400a4b00 0x7f1b4013bb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:32.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.729+0000 7f1b473c0640 10 monclient(hunting): picked mon.noname-a con 0x7f1b400a4b00 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:42:32.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.729+0000 7f1b473c0640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1b401374e0 0x7f1b40136700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:32.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.729+0000 7f1b473c0640 10 monclient(hunting): picked mon.noname-b con 0x7f1b401374e0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:42:32.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.729+0000 7f1b473c0640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:32.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.729+0000 7f1b473c0640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f1b400a7310 con 0x7f1b400a4b00 2024-09-16T03:42:32.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.729+0000 7f1b473c0640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:32.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.729+0000 7f1b473c0640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f1b400a7190 con 0x7f1b401374e0 2024-09-16T03:42:32.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.729+0000 7f1b473c0640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:32.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.729+0000 7f1b473c0640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f1b40140f50 con 0x7f1b400a4720 2024-09-16T03:42:32.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.729+0000 7f1b473c0640 10 monclient(hunting): _renew_subs 2024-09-16T03:42:32.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.729+0000 7f1b473c0640 10 monclient(hunting): authenticate will time out at 999.702109s 2024-09-16T03:42:32.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.729+0000 7f1b4515c640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1b400a4720 0x7f1b40120910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:32.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.729+0000 7f1b4495b640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1b400a4b00 0x7f1b4013bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:32.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.729+0000 7f1b4495b640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1b400a4b00 0x7f1b4013bb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:37910/0 (socket says 172.21.15.42:37910) 2024-09-16T03:42:32.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.729+0000 7f1b4495b640 1 -- 172.21.15.42:0/3459747363 learned_addr learned my addr 172.21.15.42:0/3459747363 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:42:32.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.729+0000 7f1b4515c640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1b400a4720 0x7f1b40120910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:41614/0 (socket says 172.21.15.42:41614) 2024-09-16T03:42:32.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.729+0000 7f1b4495b640 10 monclient(hunting): get_auth_request con 0x7f1b400a4b00 auth_method 0 2024-09-16T03:42:32.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.729+0000 7f1b4495b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:32.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.729+0000 7f1b4495b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:32.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.729+0000 7f1b4495b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:32.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.729+0000 7f1b4515c640 10 monclient(hunting): get_auth_request con 0x7f1b400a4720 auth_method 0 2024-09-16T03:42:32.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.729+0000 7f1b4515c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:32.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.729+0000 7f1b4515c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:32.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.729+0000 7f1b4515c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:32.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.729+0000 7f1b4495b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:32.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.729+0000 7f1b4495b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:32.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.729+0000 7f1b4495b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:32.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.729+0000 7f1b4515c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:32.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.729+0000 7f1b4515c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:32.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.729+0000 7f1b4515c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:32.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.729+0000 7f1b4515c640 10 monclient(hunting): handle_auth_done global_id 4331 payload 824 2024-09-16T03:42:32.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.729+0000 7f1b4515c640 1 -- 172.21.15.42:0/3459747363 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1b401374e0 msgr2=0x7f1b40136700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:32.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.729+0000 7f1b4515c640 1 --2- 172.21.15.42:0/3459747363 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1b401374e0 0x7f1b40136700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:32.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.729+0000 7f1b4515c640 1 -- 172.21.15.42:0/3459747363 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1b400a4b00 msgr2=0x7f1b4013bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:32.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.729+0000 7f1b4515c640 1 --2- 172.21.15.42:0/3459747363 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1b400a4b00 0x7f1b4013bb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:32.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.729+0000 7f1b4515c640 10 monclient: _finish_hunting 0 2024-09-16T03:42:32.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.729+0000 7f1b4515c640 1 monclient: found mon.noname-c 2024-09-16T03:42:32.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.729+0000 7f1b4515c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:42:32.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.729+0000 7f1b4515c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:42:32.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.729+0000 7f1b4515c640 1 -- 172.21.15.42:0/3459747363 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b401220a0 con 0x7f1b400a4720 2024-09-16T03:42:32.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.729+0000 7f1b4495b640 1 --2- 172.21.15.42:0/3459747363 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1b400a4b00 0x7f1b4013bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:42:32.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.729+0000 7f1b4515c640 1 --2- 172.21.15.42:0/3459747363 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1b400a4720 0x7f1b40120910 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f1b3c00e8a0 tx=0x7f1b3c00ed60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:32.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.729+0000 7f1b35ffb640 10 client.?.objecter ms_handle_connect 0x7f1b400a4720 2024-09-16T03:42:32.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.729+0000 7f1b35ffb640 10 client.?.objecter resend_mon_ops 2024-09-16T03:42:32.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.729+0000 7f1b35ffb640 1 -- 172.21.15.42:0/3459747363 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b3c017070 con 0x7f1b400a4720 2024-09-16T03:42:32.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.729+0000 7f1b35ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:32.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.729+0000 7f1b35ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:42:32.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.729+0000 7f1b35ffb640 10 monclient: dump: 2024-09-16T03:42:32.750 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:32.750 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:32.750 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:32.750 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:32.750 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:32.750 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:32.750 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:32.750 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:32.750 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:32.750 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:32.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.729+0000 7f1b35ffb640 10 monclient: _finish_auth 0 2024-09-16T03:42:32.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.729+0000 7f1b35ffb640 10 monclient: _check_auth_tickets 2024-09-16T03:42:32.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.729+0000 7f1b35ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:42:32.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.729+0000 7f1b473c0640 5 monclient: authenticate success, global_id 4331 2024-09-16T03:42:32.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.729+0000 7f1b473c0640 10 monclient: _renew_subs 2024-09-16T03:42:32.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.729+0000 7f1b473c0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:42:32.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.729+0000 7f1b473c0640 1 -- 172.21.15.42:0/3459747363 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b4013ccd0 con 0x7f1b400a4720 2024-09-16T03:42:32.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.729+0000 7f1b35ffb640 1 -- 172.21.15.42:0/3459747363 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1b3c00c940 con 0x7f1b400a4720 2024-09-16T03:42:32.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.729+0000 7f1b35ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T03:42:32.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.729+0000 7f1b35ffb640 1 -- 172.21.15.42:0/3459747363 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b3c00ccf0 con 0x7f1b400a4720 2024-09-16T03:42:32.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.729+0000 7f1b35ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:32.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.729+0000 7f1b35ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:42:32.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.729+0000 7f1b35ffb640 10 monclient: dump: 2024-09-16T03:42:32.751 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:32.751 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:32.751 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:32.752 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:32.752 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:32.752 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:32.752 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:32.752 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:32.752 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:32.752 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:32.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.729+0000 7f1b377fe640 4 set_mon_vals no callback set 2024-09-16T03:42:32.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.729+0000 7f1b473c0640 10 client.4331.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:42:32.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.729+0000 7f1b473c0640 10 monclient: _renew_subs 2024-09-16T03:42:32.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.729+0000 7f1b473c0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:42:32.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.729+0000 7f1b473c0640 1 -- 172.21.15.42:0/3459747363 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b40121800 con 0x7f1b400a4720 2024-09-16T03:42:32.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.729+0000 7f1b473c0640 1 librados: init done 2024-09-16T03:42:32.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.729+0000 7f1b473c0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:42:32.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.729+0000 7f1b473c0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:42:32.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.729+0000 7f1b473c0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:42:32.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.729+0000 7f1b35ffb640 1 -- 172.21.15.42:0/3459747363 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f1b3c01a070 con 0x7f1b400a4720 2024-09-16T03:42:32.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.729+0000 7f1b473c0640 1 -- 172.21.15.42:0/3459747363 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b400a4f00 con 0x7f1b400a4720 2024-09-16T03:42:32.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.729+0000 7f1b35ffb640 1 --2- 172.21.15.42:0/3459747363 >> v2:172.21.15.154:6812/254845342 conn(0x7f1b24037810 0x7f1b24039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:32.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.729+0000 7f1b35ffb640 1 -- 172.21.15.42:0/3459747363 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(19..19 src has 1..19) ==== 3534+0+0 (secure 0 0 0) 0x7f1b3c0487c0 con 0x7f1b400a4720 2024-09-16T03:42:32.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.729+0000 7f1b35ffb640 10 client.4331.objecter ms_dispatch 0x7f1b40000ba0 osd_map(19..19 src has 1..19) 2024-09-16T03:42:32.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.733+0000 7f1b35ffb640 3 client.4331.objecter handle_osd_map got epochs [19,19] > 0 2024-09-16T03:42:32.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.733+0000 7f1b35ffb640 3 client.4331.objecter handle_osd_map decoding full epoch 19 2024-09-16T03:42:32.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.733+0000 7f1b35ffb640 20 client.4331.objecter dump_active .. 0 homeless 2024-09-16T03:42:32.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.733+0000 7f1b4495b640 1 --2- 172.21.15.42:0/3459747363 >> v2:172.21.15.154:6812/254845342 conn(0x7f1b24037810 0x7f1b24039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:32.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.733+0000 7f1b4495b640 10 monclient: get_auth_request con 0x7f1b24037810 auth_method 0 2024-09-16T03:42:32.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.733+0000 7f1b35ffb640 10 client.4331.objecter ms_handle_connect 0x7f1b24037810 2024-09-16T03:42:32.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.733+0000 7f1b4495b640 1 --2- 172.21.15.42:0/3459747363 >> v2:172.21.15.154:6812/254845342 conn(0x7f1b24037810 0x7f1b24039cd0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f1b3001c420 tx=0x7f1b30008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:32.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.737+0000 7f1b35ffb640 1 -- 172.21.15.42:0/3459747363 <== mon.2 v2:172.21.15.42:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f1b3c01a360 con 0x7f1b400a4720 2024-09-16T03:42:32.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.737+0000 7f1b35ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:42:32.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.737+0000 7f1b35ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:42:32.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.893+0000 7f1b473c0640 10 start_mon_command cmd=[{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}] 2024-09-16T03:42:32.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.893+0000 7f1b473c0640 10 monclient: _send_command 2 [{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}] 2024-09-16T03:42:32.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.893+0000 7f1b473c0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:42:32.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.893+0000 7f1b473c0640 1 -- 172.21.15.42:0/3459747363 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) -- 0x7f1b40068e40 con 0x7f1b400a4720 2024-09-16T03:42:32.900 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.893+0000 7f1b35ffb640 1 -- 172.21.15.42:0/3459747363 <== mon.2 v2:172.21.15.42:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v19) ==== 92+0+10 (secure 0 0 0) 0x7f1b3c011e20 con 0x7f1b400a4720 2024-09-16T03:42:32.900 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.893+0000 7f1b35ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}] 2024-09-16T03:42:32.900 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.893+0000 7f1b35ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:42:32.900 INFO:teuthology.orchestra.run.smithi042.stdout:pg_num: 8 2024-09-16T03:42:32.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.893+0000 7f1b473c0640 10 librados: watch_flush enter 2024-09-16T03:42:32.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.893+0000 7f1b473c0640 10 librados: watch_flush exit 2024-09-16T03:42:32.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.893+0000 7f1b473c0640 20 client.4331.objecter shutdown clearing up homeless session... 2024-09-16T03:42:32.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.893+0000 7f1b473c0640 10 client.4331.objecter successfully canceled tick 2024-09-16T03:42:32.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.893+0000 7f1b473c0640 1 -- 172.21.15.42:0/3459747363 >> v2:172.21.15.154:6812/254845342 conn(0x7f1b24037810 msgr2=0x7f1b24039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:32.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.893+0000 7f1b473c0640 1 --2- 172.21.15.42:0/3459747363 >> v2:172.21.15.154:6812/254845342 conn(0x7f1b24037810 0x7f1b24039cd0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f1b3001c420 tx=0x7f1b30008040 comp rx=0 tx=0).stop 2024-09-16T03:42:32.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.893+0000 7f1b473c0640 10 monclient: shutdown 2024-09-16T03:42:32.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.893+0000 7f1b473c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:42:32.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.893+0000 7f1b473c0640 1 -- 172.21.15.42:0/3459747363 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1b400a4720 msgr2=0x7f1b40120910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:32.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.893+0000 7f1b473c0640 1 --2- 172.21.15.42:0/3459747363 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1b400a4720 0x7f1b40120910 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f1b3c00e8a0 tx=0x7f1b3c00ed60 comp rx=0 tx=0).stop 2024-09-16T03:42:32.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.897+0000 7f1b473c0640 1 -- 172.21.15.42:0/3459747363 shutdown_connections 2024-09-16T03:42:32.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.897+0000 7f1b473c0640 1 --2- 172.21.15.42:0/3459747363 >> v2:172.21.15.154:6812/254845342 conn(0x7f1b24037810 0x7f1b24039cd0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:32.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.897+0000 7f1b473c0640 1 --2- 172.21.15.42:0/3459747363 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1b401374e0 0x7f1b40136700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:32.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.897+0000 7f1b473c0640 1 --2- 172.21.15.42:0/3459747363 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1b400a4b00 0x7f1b4013bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:32.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.897+0000 7f1b473c0640 1 --2- 172.21.15.42:0/3459747363 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1b400a4720 0x7f1b40120910 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:32.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.897+0000 7f1b473c0640 1 -- 172.21.15.42:0/3459747363 >> 172.21.15.42:0/3459747363 conn(0x7f1b400aa320 msgr2=0x7f1b4013d960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:42:32.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.897+0000 7f1b473c0640 1 -- 172.21.15.42:0/3459747363 shutdown_connections 2024-09-16T03:42:32.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.897+0000 7f1b473c0640 1 -- 172.21.15.42:0/3459747363 wait complete. 2024-09-16T03:42:32.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:32.897+0000 7f1b473c0640 1 librados: shutdown 2024-09-16T03:42:32.917 INFO:tasks.mon_thrash.ceph_manager:waiting for clean 2024-09-16T03:42:32.918 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T03:42:33.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.029+0000 7f3321236640 10 monclient: get_monmap_and_config 2024-09-16T03:42:33.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.029+0000 7f3321236640 10 monclient: build_initial_monmap 2024-09-16T03:42:33.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.029+0000 7f3321236640 10 monclient: monmap: 2024-09-16T03:42:33.037 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:42:33.037 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:33.037 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:42:33.034671+0000 2024-09-16T03:42:33.037 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:42:33.034671+0000 2024-09-16T03:42:33.037 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:42:33.037 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:33.037 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:42:33.038 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:42:33.038 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:42:33.038 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:42:33.038 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:42:33.038 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:33.040 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.029+0000 7f3321236640 1 Processor -- start 2024-09-16T03:42:33.040 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.029+0000 7f3321236640 1 -- start start 2024-09-16T03:42:33.040 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.029+0000 7f3321236640 10 monclient: init 2024-09-16T03:42:33.040 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.033+0000 7f3321236640 10 monclient: _reopen_session rank -1 2024-09-16T03:42:33.040 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.033+0000 7f3321236640 10 monclient: _add_conns ranks=[0,3,4,1,2] 2024-09-16T03:42:33.040 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.033+0000 7f3321236640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f331c0a1330 addr v1:172.21.15.42:6789/0 2024-09-16T03:42:33.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.033+0000 7f3321236640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f331c06dac0 0x7f331c13a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:33.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.033+0000 7f3321236640 10 monclient(hunting): picked mon.noname-a con 0x7f331c06dac0 addr v2:172.21.15.42:3300/0 2024-09-16T03:42:33.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.033+0000 7f3321236640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f331c0a1e20 0x7f331c13c890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:33.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.033+0000 7f3321236640 10 monclient(hunting): picked mon.noname-c con 0x7f331c0a1e20 addr v2:172.21.15.154:3300/0 2024-09-16T03:42:33.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.033+0000 7f3321236640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f331c09c090 con 0x7f331c0a1330 2024-09-16T03:42:33.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.033+0000 7f3321236640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:33.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.033+0000 7f3321236640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f331c09f610 con 0x7f331c06dac0 2024-09-16T03:42:33.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.033+0000 7f3321236640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:33.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.033+0000 7f3321236640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f331c09f8f0 con 0x7f331c0a1e20 2024-09-16T03:42:33.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.033+0000 7f3321236640 10 monclient(hunting): _renew_subs 2024-09-16T03:42:33.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.033+0000 7f3321236640 10 monclient(hunting): authenticate will time out at 1000.004377s 2024-09-16T03:42:33.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.033+0000 7f331ad9d640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f331c0a1330 0x7f331c0a1710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:46854/0 (socket says 172.21.15.42:46854) 2024-09-16T03:42:33.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.033+0000 7f331ad9d640 1 -- 172.21.15.42:0/2744861286 learned_addr learned my addr 172.21.15.42:0/2744861286 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:42:33.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.033+0000 7f331a59c640 1 --2- 172.21.15.42:0/2744861286 >> v2:172.21.15.42:3300/0 conn(0x7f331c06dac0 0x7f331c13a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:33.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.033+0000 7f331a59c640 10 monclient(hunting): get_auth_request con 0x7f331c06dac0 auth_method 0 2024-09-16T03:42:33.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.033+0000 7f331a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:33.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.033+0000 7f331a59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:33.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.033+0000 7f331a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:33.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.033+0000 7f331a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:33.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.033+0000 7f331a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:33.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.033+0000 7f331a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:33.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.033+0000 7f331b59e640 1 --2- 172.21.15.42:0/2744861286 >> v2:172.21.15.154:3300/0 conn(0x7f331c0a1e20 0x7f331c13c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:33.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.033+0000 7f3319d9b640 1 -- 172.21.15.42:0/2744861286 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1998136107 0 0) 0x7f331c09c090 con 0x7f331c0a1330 2024-09-16T03:42:33.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.033+0000 7f3319d9b640 10 monclient(hunting): _negotiate 2024-09-16T03:42:33.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.033+0000 7f3319d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:33.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.033+0000 7f3319d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:33.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.033+0000 7f3319d9b640 10 monclient(hunting): my global_id is 4392 2024-09-16T03:42:33.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.033+0000 7f3319d9b640 1 -- 172.21.15.42:0/2744861286 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3304003530 con 0x7f331c0a1330 2024-09-16T03:42:33.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.033+0000 7f331b59e640 10 monclient(hunting): get_auth_request con 0x7f331c0a1e20 auth_method 0 2024-09-16T03:42:33.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.033+0000 7f331b59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:33.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.033+0000 7f331b59e640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:33.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.033+0000 7f331b59e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:33.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.033+0000 7f331a59c640 10 monclient(hunting): handle_auth_done global_id 4395 payload 274 2024-09-16T03:42:33.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.033+0000 7f331a59c640 1 -- 172.21.15.42:0/2744861286 >> v2:172.21.15.154:3300/0 conn(0x7f331c0a1e20 msgr2=0x7f331c13c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:42:33.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.033+0000 7f331a59c640 1 --2- 172.21.15.42:0/2744861286 >> v2:172.21.15.154:3300/0 conn(0x7f331c0a1e20 0x7f331c13c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:33.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.033+0000 7f331a59c640 1 -- 172.21.15.42:0/2744861286 >> v1:172.21.15.42:6789/0 conn(0x7f331c0a1330 legacy=0x7f331c0a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:33.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.033+0000 7f331a59c640 10 monclient: _finish_hunting 0 2024-09-16T03:42:33.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.033+0000 7f331a59c640 1 monclient: found mon.noname-a 2024-09-16T03:42:33.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.033+0000 7f331a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:42:33.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.033+0000 7f331a59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:42:33.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.033+0000 7f331a59c640 1 -- 172.21.15.42:0/2744861286 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f331c09fbd0 con 0x7f331c06dac0 2024-09-16T03:42:33.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.033+0000 7f331a59c640 1 --2- 172.21.15.42:0/2744861286 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f331c06dac0 0x7f331c13a340 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f331c09c800 tx=0x7f331001c3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b575b25bae63fb0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:33.050 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.033+0000 7f3319d9b640 1 -- 172.21.15.42:0/2744861286 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33100b0070 con 0x7f331c06dac0 2024-09-16T03:42:33.050 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.033+0000 7f3319d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:33.050 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.033+0000 7f3319d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:42:33.050 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.033+0000 7f3319d9b640 10 monclient: dump: 2024-09-16T03:42:33.050 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:33.050 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:33.050 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:33.050 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:33.050 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:33.050 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:33.050 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:33.050 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:33.050 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:33.051 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:33.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.033+0000 7f3319d9b640 10 monclient: _finish_auth 0 2024-09-16T03:42:33.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.033+0000 7f3319d9b640 10 monclient: _check_auth_tickets 2024-09-16T03:42:33.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.033+0000 7f3319d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:42:33.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.033+0000 7f3319d9b640 1 -- 172.21.15.42:0/2744861286 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f33100ab600 con 0x7f331c06dac0 2024-09-16T03:42:33.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.033+0000 7f3319d9b640 10 monclient: handle_config config(0 keys) 2024-09-16T03:42:33.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.033+0000 7f3319d9b640 1 -- 172.21.15.42:0/2744861286 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33100a4e60 con 0x7f331c06dac0 2024-09-16T03:42:33.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.033+0000 7f3321236640 5 monclient: authenticate success, global_id 4395 2024-09-16T03:42:33.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.033+0000 7f3321236640 10 monclient: get_monmap_and_config success 2024-09-16T03:42:33.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.033+0000 7f3321236640 4 set_mon_vals no callback set 2024-09-16T03:42:33.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.033+0000 7f3319d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:33.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.033+0000 7f3319d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:42:33.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.033+0000 7f3319d9b640 10 monclient: dump: 2024-09-16T03:42:33.051 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:33.052 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:33.052 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:33.052 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:33.052 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:33.052 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:33.052 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:33.052 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:33.052 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:33.052 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:33.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.033+0000 7f3321236640 10 monclient: shutdown 2024-09-16T03:42:33.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.033+0000 7f3321236640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:42:33.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.033+0000 7f3321236640 1 -- 172.21.15.42:0/2744861286 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f331c06dac0 msgr2=0x7f331c13a340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:33.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.033+0000 7f3321236640 1 --2- 172.21.15.42:0/2744861286 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f331c06dac0 0x7f331c13a340 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f331c09c800 tx=0x7f331001c3d0 comp rx=0 tx=0).stop 2024-09-16T03:42:33.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.033+0000 7f3321236640 1 -- 172.21.15.42:0/2744861286 shutdown_connections 2024-09-16T03:42:33.053 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.033+0000 7f3321236640 1 --2- 172.21.15.42:0/2744861286 >> v2:172.21.15.154:3300/0 conn(0x7f331c0a1e20 0x7f331c13c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:33.053 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.033+0000 7f3321236640 1 --2- 172.21.15.42:0/2744861286 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f331c06dac0 0x7f331c13a340 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:33.053 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.033+0000 7f3321236640 1 -- 172.21.15.42:0/2744861286 >> 172.21.15.42:0/2744861286 conn(0x7f331c0a2420 msgr2=0x7f331c0a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:42:33.053 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.037+0000 7f3321236640 1 -- 172.21.15.42:0/2744861286 shutdown_connections 2024-09-16T03:42:33.053 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.037+0000 7f3321236640 1 -- 172.21.15.42:0/2744861286 wait complete. 2024-09-16T03:42:33.053 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.037+0000 7f3321236640 10 monclient: build_initial_monmap 2024-09-16T03:42:33.053 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.037+0000 7f3321236640 10 monclient: monmap: 2024-09-16T03:42:33.053 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:42:33.053 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:42:33.053 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:42:33.053 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:42:33.053 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:42:33.053 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:33.054 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:42:33.054 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:42:33.054 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:42:33.054 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:33.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.037+0000 7f3321236640 1 librados: starting msgr at 2024-09-16T03:42:33.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.037+0000 7f3321236640 1 librados: starting objecter 2024-09-16T03:42:33.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.037+0000 7f3321236640 1 Processor -- start 2024-09-16T03:42:33.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.037+0000 7f3321236640 1 -- start start 2024-09-16T03:42:33.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.037+0000 7f3321236640 1 librados: setting wanted keys 2024-09-16T03:42:33.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.037+0000 7f3321236640 1 librados: calling monclient init 2024-09-16T03:42:33.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.037+0000 7f3321236640 10 monclient: init 2024-09-16T03:42:33.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.037+0000 7f3321236640 10 monclient: _reopen_session rank -1 2024-09-16T03:42:33.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.037+0000 7f3321236640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:42:33.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.037+0000 7f3321236640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f331c0a1330 0x7f331c1ceb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:33.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.037+0000 7f3321236640 10 monclient(hunting): picked mon.noname-c con 0x7f331c0a1330 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:42:33.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.037+0000 7f3321236640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f331c0a1e20 0x7f331c1351a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:33.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.037+0000 7f3321236640 10 monclient(hunting): picked mon.noname-a con 0x7f331c0a1e20 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:42:33.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.037+0000 7f3321236640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f331c1356e0 0x7f331c09cb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:33.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.037+0000 7f3321236640 10 monclient(hunting): picked mon.noname-b con 0x7f331c1356e0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:42:33.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.037+0000 7f3321236640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:33.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.037+0000 7f3321236640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f331c13f800 con 0x7f331c0a1e20 2024-09-16T03:42:33.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.037+0000 7f3321236640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:33.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.037+0000 7f3321236640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f331c09ef80 con 0x7f331c1356e0 2024-09-16T03:42:33.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.037+0000 7f3321236640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:33.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.037+0000 7f3321236640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f331c13f980 con 0x7f331c0a1330 2024-09-16T03:42:33.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.037+0000 7f3321236640 10 monclient(hunting): _renew_subs 2024-09-16T03:42:33.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.037+0000 7f3321236640 10 monclient(hunting): authenticate will time out at 1000.009564s 2024-09-16T03:42:33.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.037+0000 7f331a59c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f331c0a1e20 0x7f331c1351a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:33.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.037+0000 7f331ad9d640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f331c0a1330 0x7f331c1ceb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:33.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.037+0000 7f331a59c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f331c0a1e20 0x7f331c1351a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:37924/0 (socket says 172.21.15.42:37924) 2024-09-16T03:42:33.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.037+0000 7f331a59c640 1 -- 172.21.15.42:0/1884401016 learned_addr learned my addr 172.21.15.42:0/1884401016 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:42:33.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.037+0000 7f331a59c640 10 monclient(hunting): get_auth_request con 0x7f331c0a1e20 auth_method 0 2024-09-16T03:42:33.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.037+0000 7f331a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:33.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.037+0000 7f331a59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:33.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.037+0000 7f331a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:33.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.037+0000 7f331ad9d640 10 monclient(hunting): get_auth_request con 0x7f331c0a1330 auth_method 0 2024-09-16T03:42:33.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.037+0000 7f331ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:33.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.037+0000 7f331ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:33.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.037+0000 7f331ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:33.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.037+0000 7f331a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:33.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.037+0000 7f331a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:33.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.037+0000 7f331a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:33.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.037+0000 7f331ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:33.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.037+0000 7f331ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:33.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.037+0000 7f331ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:33.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.037+0000 7f331a59c640 10 monclient(hunting): handle_auth_done global_id 4398 payload 824 2024-09-16T03:42:33.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.037+0000 7f331a59c640 1 -- 172.21.15.42:0/1884401016 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f331c0a1330 msgr2=0x7f331c1ceb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:33.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.037+0000 7f331a59c640 1 --2- 172.21.15.42:0/1884401016 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f331c0a1330 0x7f331c1ceb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:33.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.037+0000 7f331b59e640 1 --2- 172.21.15.42:0/1884401016 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f331c1356e0 0x7f331c09cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:33.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.037+0000 7f331a59c640 1 -- 172.21.15.42:0/1884401016 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f331c1356e0 msgr2=0x7f331c09cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:33.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.037+0000 7f331a59c640 1 --2- 172.21.15.42:0/1884401016 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f331c1356e0 0x7f331c09cb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:33.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.037+0000 7f331a59c640 10 monclient: _finish_hunting 0 2024-09-16T03:42:33.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.037+0000 7f331a59c640 1 monclient: found mon.noname-a 2024-09-16T03:42:33.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.037+0000 7f331a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:42:33.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.037+0000 7f331a59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:42:33.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.037+0000 7f331a59c640 1 -- 172.21.15.42:0/1884401016 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f331c1212b0 con 0x7f331c0a1e20 2024-09-16T03:42:33.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.037+0000 7f331ad9d640 1 --2- 172.21.15.42:0/1884401016 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f331c0a1330 0x7f331c1ceb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:42:33.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.037+0000 7f331a59c640 1 --2- 172.21.15.42:0/1884401016 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f331c0a1e20 0x7f331c1351a0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f33100ae040 tx=0x7f3310019170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:33.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.037+0000 7f33037fe640 10 client.?.objecter ms_handle_connect 0x7f331c0a1e20 2024-09-16T03:42:33.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.037+0000 7f33037fe640 10 client.?.objecter resend_mon_ops 2024-09-16T03:42:33.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.037+0000 7f33037fe640 1 -- 172.21.15.42:0/1884401016 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33100b8020 con 0x7f331c0a1e20 2024-09-16T03:42:33.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.037+0000 7f33037fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:33.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.037+0000 7f33037fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:42:33.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.037+0000 7f33037fe640 10 monclient: dump: 2024-09-16T03:42:33.059 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:33.059 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:33.059 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:33.059 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:33.059 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:33.059 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:33.059 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:33.059 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:33.059 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:33.059 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:33.059 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.037+0000 7f33037fe640 10 monclient: _finish_auth 0 2024-09-16T03:42:33.059 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.037+0000 7f33037fe640 10 monclient: _check_auth_tickets 2024-09-16T03:42:33.059 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.037+0000 7f33037fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:42:33.059 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.037+0000 7f33037fe640 1 -- 172.21.15.42:0/1884401016 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f33100a4e60 con 0x7f331c0a1e20 2024-09-16T03:42:33.060 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.037+0000 7f33037fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:42:33.060 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.037+0000 7f33037fe640 1 -- 172.21.15.42:0/1884401016 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33100b0040 con 0x7f331c0a1e20 2024-09-16T03:42:33.060 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.037+0000 7f33037fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:33.060 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.037+0000 7f33037fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:42:33.060 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.037+0000 7f33037fe640 10 monclient: dump: 2024-09-16T03:42:33.060 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:33.060 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:33.060 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:33.060 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:33.060 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:33.060 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:33.060 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:33.060 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:33.061 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:33.061 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:33.061 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.037+0000 7f331959a640 4 set_mon_vals no callback set 2024-09-16T03:42:33.061 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.037+0000 7f3321236640 5 monclient: authenticate success, global_id 4398 2024-09-16T03:42:33.061 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.037+0000 7f3321236640 10 monclient: _renew_subs 2024-09-16T03:42:33.061 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.037+0000 7f3321236640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:42:33.061 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.037+0000 7f3321236640 1 -- 172.21.15.42:0/1884401016 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f331c121590 con 0x7f331c0a1e20 2024-09-16T03:42:33.061 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.037+0000 7f3321236640 10 client.4398.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:42:33.061 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.037+0000 7f3321236640 10 monclient: _renew_subs 2024-09-16T03:42:33.061 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.037+0000 7f3321236640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:42:33.061 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.037+0000 7f3321236640 1 -- 172.21.15.42:0/1884401016 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f331c13c480 con 0x7f331c0a1e20 2024-09-16T03:42:33.061 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.037+0000 7f3321236640 1 librados: init done 2024-09-16T03:42:33.061 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.037+0000 7f33037fe640 1 -- 172.21.15.42:0/1884401016 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f33100bb020 con 0x7f331c0a1e20 2024-09-16T03:42:33.061 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.037+0000 7f3321236640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:42:33.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.037+0000 7f3321236640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:42:33.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.037+0000 7f3321236640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:42:33.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.037+0000 7f3321236640 1 -- 172.21.15.42:0/1884401016 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f331c09e250 con 0x7f331c0a1e20 2024-09-16T03:42:33.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.037+0000 7f33037fe640 1 --2- 172.21.15.42:0/1884401016 >> v2:172.21.15.154:6812/254845342 conn(0x7f3308037840 0x7f3308039d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:33.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.037+0000 7f33037fe640 1 -- 172.21.15.42:0/1884401016 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 3534+0+0 (secure 0 0 0) 0x7f33100e2060 con 0x7f331c0a1e20 2024-09-16T03:42:33.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.037+0000 7f33037fe640 10 client.4398.objecter ms_dispatch 0x7f331c000ba0 osd_map(19..19 src has 1..19) 2024-09-16T03:42:33.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.037+0000 7f33037fe640 3 client.4398.objecter handle_osd_map got epochs [19,19] > 0 2024-09-16T03:42:33.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.037+0000 7f33037fe640 3 client.4398.objecter handle_osd_map decoding full epoch 19 2024-09-16T03:42:33.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.041+0000 7f33037fe640 20 client.4398.objecter dump_active .. 0 homeless 2024-09-16T03:42:33.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.041+0000 7f331ad9d640 1 --2- 172.21.15.42:0/1884401016 >> v2:172.21.15.154:6812/254845342 conn(0x7f3308037840 0x7f3308039d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:33.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.041+0000 7f331ad9d640 10 monclient: get_auth_request con 0x7f3308037840 auth_method 0 2024-09-16T03:42:33.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.041+0000 7f331ad9d640 1 --2- 172.21.15.42:0/1884401016 >> v2:172.21.15.154:6812/254845342 conn(0x7f3308037840 0x7f3308039d00 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f330c005890 tx=0x7f330c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:33.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.041+0000 7f33037fe640 10 client.4398.objecter ms_handle_connect 0x7f3308037840 2024-09-16T03:42:33.063 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.045+0000 7f33037fe640 1 -- 172.21.15.42:0/1884401016 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f33100c2030 con 0x7f331c0a1e20 2024-09-16T03:42:33.063 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.045+0000 7f33037fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:42:33.063 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.045+0000 7f33037fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:42:33.205 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.197+0000 7f3321236640 1 -- 172.21.15.42:0/1884401016 --> v2:172.21.15.154:6812/254845342 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f331c073a30 con 0x7f3308037840 2024-09-16T03:42:33.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.201+0000 7f33037fe640 1 -- 172.21.15.42:0/1884401016 <== mgr.4102 v2:172.21.15.154:6812/254845342 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+34247 (secure 0 0 0) 0x7f331c073a30 con 0x7f3308037840 2024-09-16T03:42:33.209 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-09-16T03:42:33.209 INFO:teuthology.orchestra.run.smithi042.stderr:dumped all 2024-09-16T03:42:33.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.205+0000 7f3321236640 10 librados: watch_flush enter 2024-09-16T03:42:33.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.205+0000 7f3321236640 10 librados: watch_flush exit 2024-09-16T03:42:33.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.205+0000 7f3321236640 20 client.4398.objecter shutdown clearing up homeless session... 2024-09-16T03:42:33.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.205+0000 7f3321236640 10 client.4398.objecter successfully canceled tick 2024-09-16T03:42:33.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.205+0000 7f3321236640 1 -- 172.21.15.42:0/1884401016 >> v2:172.21.15.154:6812/254845342 conn(0x7f3308037840 msgr2=0x7f3308039d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:33.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.205+0000 7f3321236640 1 --2- 172.21.15.42:0/1884401016 >> v2:172.21.15.154:6812/254845342 conn(0x7f3308037840 0x7f3308039d00 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f330c005890 tx=0x7f330c009040 comp rx=0 tx=0).stop 2024-09-16T03:42:33.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.205+0000 7f3321236640 10 monclient: shutdown 2024-09-16T03:42:33.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.205+0000 7f3321236640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:42:33.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.205+0000 7f3321236640 1 -- 172.21.15.42:0/1884401016 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f331c0a1e20 msgr2=0x7f331c1351a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:33.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.205+0000 7f3321236640 1 --2- 172.21.15.42:0/1884401016 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f331c0a1e20 0x7f331c1351a0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f33100ae040 tx=0x7f3310019170 comp rx=0 tx=0).stop 2024-09-16T03:42:33.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.205+0000 7f3321236640 1 -- 172.21.15.42:0/1884401016 shutdown_connections 2024-09-16T03:42:33.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.205+0000 7f3321236640 1 --2- 172.21.15.42:0/1884401016 >> v2:172.21.15.154:6812/254845342 conn(0x7f3308037840 0x7f3308039d00 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:33.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.205+0000 7f3321236640 1 --2- 172.21.15.42:0/1884401016 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f331c1356e0 0x7f331c09cb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:33.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.205+0000 7f3321236640 1 --2- 172.21.15.42:0/1884401016 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f331c0a1e20 0x7f331c1351a0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:33.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.205+0000 7f3321236640 1 --2- 172.21.15.42:0/1884401016 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f331c0a1330 0x7f331c1ceb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:33.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.205+0000 7f3321236640 1 -- 172.21.15.42:0/1884401016 >> 172.21.15.42:0/1884401016 conn(0x7f331c0a2420 msgr2=0x7f331c136320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:42:33.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.205+0000 7f3321236640 1 -- 172.21.15.42:0/1884401016 shutdown_connections 2024-09-16T03:42:33.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.205+0000 7f3321236640 1 -- 172.21.15.42:0/1884401016 wait complete. 2024-09-16T03:42:33.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.205+0000 7f3321236640 1 librados: shutdown 2024-09-16T03:42:33.226 INFO:teuthology.orchestra.run.smithi042.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-09-16T03:42:31.472067+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":161,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120,"ondisk_log_size":120,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":12,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":167368,"kb_used_data":1016,"kb_used_omap":9,"kb_used_meta":166198,"kb_avail":566063672,"statfs":{"total":579820584960,"available":579649200128,"internally_reserved":0,"allocated":1040384,"data_stored":1353088,"data_compressed":18838,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":9567,"internal_metadata":170187425},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.867729"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":19,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T03:42:27.624215+0000","last_change":"2024-09-16T03:42:27.624922+0000","last_active":"2024-09-16T03:42:27.624215+0000","last_peered":"2024-09-16T03:42:27.624215+0000","last_clean":"2024-09-16T03:42:27.624215+0000","last_became_active":"2024-09-16T03:42:25.615293+0000","last_became_peered":"2024-09-16T03:42:25.615293+0000","last_unstale":"2024-09-16T03:42:27.624215+0000","last_undegraded":"2024-09-16T03:42:27.624215+0000","last_fullsized":"2024-09-16T03:42:27.624215+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:42:24.583922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:42:24.583922+0000","last_clean_scrub_stamp":"2024-09-16T03:42:24.583922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:46:11.677251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010681690000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":19,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T03:42:27.630687+0000","last_change":"2024-09-16T03:42:27.631227+0000","last_active":"2024-09-16T03:42:27.630687+0000","last_peered":"2024-09-16T03:42:27.630687+0000","last_clean":"2024-09-16T03:42:27.630687+0000","last_became_active":"2024-09-16T03:42:25.608453+0000","last_became_peered":"2024-09-16T03:42:25.608453+0000","last_unstale":"2024-09-16T03:42:27.630687+0000","last_undegraded":"2024-09-16T03:42:27.630687+0000","last_fullsized":"2024-09-16T03:42:27.630687+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:42:24.583922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:42:24.583922+0000","last_clean_scrub_stamp":"2024-09-16T03:42:24.583922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:00:11.921493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001163533,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":19,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T03:42:27.624630+0000","last_change":"2024-09-16T03:42:27.625091+0000","last_active":"2024-09-16T03:42:27.624630+0000","last_peered":"2024-09-16T03:42:27.624630+0000","last_clean":"2024-09-16T03:42:27.624630+0000","last_became_active":"2024-09-16T03:42:25.614458+0000","last_became_peered":"2024-09-16T03:42:25.614458+0000","last_unstale":"2024-09-16T03:42:27.624630+0000","last_undegraded":"2024-09-16T03:42:27.624630+0000","last_fullsized":"2024-09-16T03:42:27.624630+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:42:24.583922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:42:24.583922+0000","last_clean_scrub_stamp":"2024-09-16T03:42:24.583922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:26:14.681436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012701769999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T03:42:27.630545+0000","last_change":"2024-09-16T03:42:27.631260+0000","last_active":"2024-09-16T03:42:27.630545+0000","last_peered":"2024-09-16T03:42:27.630545+0000","last_clean":"2024-09-16T03:42:27.630545+0000","last_became_active":"2024-09-16T03:42:25.609631+0000","last_became_peered":"2024-09-16T03:42:25.609631+0000","last_unstale":"2024-09-16T03:42:27.630545+0000","last_undegraded":"2024-09-16T03:42:27.630545+0000","last_fullsized":"2024-09-16T03:42:27.630545+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:42:24.583922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:42:24.583922+0000","last_clean_scrub_stamp":"2024-09-16T03:42:24.583922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:58:49.091085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001288897,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"18'2","reported_seq":21,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T03:42:27.633516+0000","last_change":"2024-09-16T03:42:27.632050+0000","last_active":"2024-09-16T03:42:27.633516+0000","last_peered":"2024-09-16T03:42:27.633516+0000","last_clean":"2024-09-16T03:42:27.633516+0000","last_became_active":"2024-09-16T03:42:25.617896+0000","last_became_peered":"2024-09-16T03:42:25.617896+0000","last_unstale":"2024-09-16T03:42:27.633516+0000","last_undegraded":"2024-09-16T03:42:27.633516+0000","last_fullsized":"2024-09-16T03:42:27.633516+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:42:24.583922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:42:24.583922+0000","last_clean_scrub_stamp":"2024-09-16T03:42:24.583922+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:41:08.754274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021072529999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":19,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T03:42:27.618683+0000","last_change":"2024-09-16T03:42:27.619252+0000","last_active":"2024-09-16T03:42:27.618683+0000","last_peered":"2024-09-16T03:42:27.618683+0000","last_clean":"2024-09-16T03:42:27.618683+0000","last_became_active":"2024-09-16T03:42:25.607533+0000","last_became_peered":"2024-09-16T03:42:25.607533+0000","last_unstale":"2024-09-16T03:42:27.618683+0000","last_undegraded":"2024-09-16T03:42:27.618683+0000","last_fullsized":"2024-09-16T03:42:27.618683+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:42:24.583922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:42:24.583922+0000","last_clean_scrub_stamp":"2024-09-16T03:42:24.583922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:25:58.536144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089083299999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":19,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T03:42:27.624050+0000","last_change":"2024-09-16T03:42:27.624504+0000","last_active":"2024-09-16T03:42:27.624050+0000","last_peered":"2024-09-16T03:42:27.624050+0000","last_clean":"2024-09-16T03:42:27.624050+0000","last_became_active":"2024-09-16T03:42:25.612386+0000","last_became_peered":"2024-09-16T03:42:25.612386+0000","last_unstale":"2024-09-16T03:42:27.624050+0000","last_undegraded":"2024-09-16T03:42:27.624050+0000","last_fullsized":"2024-09-16T03:42:27.624050+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:42:24.583922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:42:24.583922+0000","last_clean_scrub_stamp":"2024-09-16T03:42:24.583922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:00:49.446880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081399800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"16'1","reported_seq":20,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T03:42:27.630965+0000","last_change":"2024-09-16T03:42:27.630965+0000","last_active":"2024-09-16T03:42:27.630965+0000","last_peered":"2024-09-16T03:42:27.630965+0000","last_clean":"2024-09-16T03:42:27.630965+0000","last_became_active":"2024-09-16T03:42:25.615930+0000","last_became_peered":"2024-09-16T03:42:25.615930+0000","last_unstale":"2024-09-16T03:42:27.630965+0000","last_undegraded":"2024-09-16T03:42:27.630965+0000","last_fullsized":"2024-09-16T03:42:27.630965+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:42:24.583922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:42:24.583922+0000","last_clean_scrub_stamp":"2024-09-16T03:42:24.583922+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:59:59.653118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00099470900000000009,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"14'117","reported_seq":170,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T03:42:27.623504+0000","last_change":"2024-09-16T03:42:18.799294+0000","last_active":"2024-09-16T03:42:27.623504+0000","last_peered":"2024-09-16T03:42:27.623504+0000","last_clean":"2024-09-16T03:42:27.623504+0000","last_became_active":"2024-09-16T03:42:18.797239+0000","last_became_peered":"2024-09-16T03:42:18.797239+0000","last_unstale":"2024-09-16T03:42:27.623504+0000","last_undegraded":"2024-09-16T03:42:27.623504+0000","last_fullsized":"2024-09-16T03:42:27.623504+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:42:17.544237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:42:17.544237+0000","last_clean_scrub_stamp":"2024-09-16T03:42:17.544237+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:01:21.017593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":18838,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":12,"seq":51539607558,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27188,"kb_used_data":156,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344652,"statfs":{"total":96636764160,"available":96608923648,"internally_reserved":0,"allocated":159744,"data_stored":49906,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":11,"seq":47244640260,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27172,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344668,"statfs":{"total":96636764160,"available":96608940032,"internally_reserved":0,"allocated":147456,"data_stored":45759,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640260,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":208896,"data_stored":578815,"data_compressed":9419,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640260,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":155648,"data_stored":49887,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640262,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31352,"kb_used_data":156,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340488,"statfs":{"total":96636764160,"available":96604659712,"internally_reserved":0,"allocated":159744,"data_stored":49906,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640260,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":208896,"data_stored":578815,"data_compressed":9419,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":9419,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":9419,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T03:42:33.227 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T03:42:33.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.337+0000 7fc0acd66640 10 monclient: get_monmap_and_config 2024-09-16T03:42:33.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.337+0000 7fc0acd66640 10 monclient: build_initial_monmap 2024-09-16T03:42:33.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.337+0000 7fc0acd66640 10 monclient: monmap: 2024-09-16T03:42:33.343 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:42:33.343 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:33.343 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:42:33.341286+0000 2024-09-16T03:42:33.344 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:42:33.341286+0000 2024-09-16T03:42:33.344 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:42:33.344 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:33.344 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:42:33.344 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:42:33.344 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:42:33.344 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:42:33.344 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:42:33.344 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:33.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.337+0000 7fc0acd66640 1 Processor -- start 2024-09-16T03:42:33.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.337+0000 7fc0acd66640 1 -- start start 2024-09-16T03:42:33.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.337+0000 7fc0acd66640 10 monclient: init 2024-09-16T03:42:33.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.337+0000 7fc0acd66640 10 monclient: _reopen_session rank -1 2024-09-16T03:42:33.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.337+0000 7fc0acd66640 10 monclient: _add_conns ranks=[4,3,2,0,1] 2024-09-16T03:42:33.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.337+0000 7fc0acd66640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fc0a80a39b0 0x7fc0a8133b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:33.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.337+0000 7fc0acd66640 10 monclient(hunting): picked mon.noname-c con 0x7fc0a80a39b0 addr v2:172.21.15.154:3300/0 2024-09-16T03:42:33.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.337+0000 7fc0acd66640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fc0a80a3d90 0x7fc0a8137000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:33.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.337+0000 7fc0acd66640 10 monclient(hunting): picked mon.noname-a con 0x7fc0a80a3d90 addr v2:172.21.15.42:3300/0 2024-09-16T03:42:33.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.337+0000 7fc0acd66640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc0a8137540 0x7fc0a8137990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:33.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.337+0000 7fc0acd66640 10 monclient(hunting): picked mon.noname-b con 0x7fc0a8137540 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:42:33.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.337+0000 7fc0acd66640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:33.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.337+0000 7fc0acd66640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7fc0a809c140 con 0x7fc0a80a3d90 2024-09-16T03:42:33.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.337+0000 7fc0acd66640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:33.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.337+0000 7fc0acd66640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fc0a809c420 con 0x7fc0a80a39b0 2024-09-16T03:42:33.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.337+0000 7fc0acd66640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:33.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.337+0000 7fc0acd66640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fc0a806dd60 con 0x7fc0a8137540 2024-09-16T03:42:33.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.337+0000 7fc0acd66640 10 monclient(hunting): _renew_subs 2024-09-16T03:42:33.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.337+0000 7fc0acd66640 10 monclient(hunting): authenticate will time out at 1000.310695s 2024-09-16T03:42:33.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.337+0000 7fc0a5d9b640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fc0a80a3d90 0x7fc0a8137000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:33.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.337+0000 7fc0a6d9d640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc0a8137540 0x7fc0a8137990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:33.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.337+0000 7fc0a6d9d640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc0a8137540 0x7fc0a8137990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:41638/0 (socket says 172.21.15.42:41638) 2024-09-16T03:42:33.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.337+0000 7fc0a6d9d640 1 -- 172.21.15.42:0/1153166936 learned_addr learned my addr 172.21.15.42:0/1153166936 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:42:33.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.337+0000 7fc0a5d9b640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fc0a80a3d90 0x7fc0a8137000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:37930/0 (socket says 172.21.15.42:37930) 2024-09-16T03:42:33.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.337+0000 7fc0a6d9d640 10 monclient(hunting): get_auth_request con 0x7fc0a8137540 auth_method 0 2024-09-16T03:42:33.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.337+0000 7fc0a6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:33.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.337+0000 7fc0a6d9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:33.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.337+0000 7fc0a6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:33.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.337+0000 7fc0a5d9b640 10 monclient(hunting): get_auth_request con 0x7fc0a80a3d90 auth_method 0 2024-09-16T03:42:33.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.337+0000 7fc0a5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:33.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.337+0000 7fc0a5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:33.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.337+0000 7fc0a5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:33.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.337+0000 7fc0a5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:33.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.337+0000 7fc0a5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:33.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.337+0000 7fc0a5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:33.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.337+0000 7fc0a6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:33.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.337+0000 7fc0a6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:33.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.337+0000 7fc0a6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:33.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.337+0000 7fc0a659c640 1 --2- 172.21.15.42:0/1153166936 >> v2:172.21.15.154:3300/0 conn(0x7fc0a80a39b0 0x7fc0a8133b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:33.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.337+0000 7fc0a5d9b640 10 monclient(hunting): handle_auth_done global_id 4401 payload 274 2024-09-16T03:42:33.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.337+0000 7fc0a5d9b640 1 -- 172.21.15.42:0/1153166936 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc0a8137540 msgr2=0x7fc0a8137990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:42:33.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.337+0000 7fc0a5d9b640 1 --2- 172.21.15.42:0/1153166936 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc0a8137540 0x7fc0a8137990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:33.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.337+0000 7fc0a5d9b640 1 -- 172.21.15.42:0/1153166936 >> v2:172.21.15.154:3300/0 conn(0x7fc0a80a39b0 msgr2=0x7fc0a8133b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:42:33.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.337+0000 7fc0a5d9b640 1 --2- 172.21.15.42:0/1153166936 >> v2:172.21.15.154:3300/0 conn(0x7fc0a80a39b0 0x7fc0a8133b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:33.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.337+0000 7fc0a5d9b640 10 monclient: _finish_hunting 0 2024-09-16T03:42:33.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.337+0000 7fc0a5d9b640 1 monclient: found mon.noname-a 2024-09-16T03:42:33.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.337+0000 7fc0a5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:42:33.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.337+0000 7fc0a5d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:42:33.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.337+0000 7fc0a5d9b640 1 -- 172.21.15.42:0/1153166936 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0a80a5e20 con 0x7fc0a80a3d90 2024-09-16T03:42:33.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.337+0000 7fc0a659c640 10 monclient: get_auth_request con 0x7fc0a80a39b0 auth_method 0 2024-09-16T03:42:33.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.337+0000 7fc0a659c640 1 --2- 172.21.15.42:0/1153166936 >> v2:172.21.15.154:3300/0 conn(0x7fc0a80a39b0 0x7fc0a8133b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T03:42:33.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.337+0000 7fc0a6d9d640 1 --2- 172.21.15.42:0/1153166936 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc0a8137540 0x7fc0a8137990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:42:33.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.341+0000 7fc0a5d9b640 1 --2- 172.21.15.42:0/1153166936 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc0a80a3d90 0x7fc0a8137000 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fc0a80a7750 tx=0x7fc09001c340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7636ce30375c2d06 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:33.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.341+0000 7fc0a559a640 1 -- 172.21.15.42:0/1153166936 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0900b0070 con 0x7fc0a80a3d90 2024-09-16T03:42:33.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.341+0000 7fc0a559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:33.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.341+0000 7fc0a559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:42:33.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.341+0000 7fc0a559a640 10 monclient: dump: 2024-09-16T03:42:33.356 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:33.356 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:33.356 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:33.356 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:33.356 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:33.356 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:33.356 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:33.357 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:33.357 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:33.357 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:33.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.341+0000 7fc0a559a640 10 monclient: _finish_auth 0 2024-09-16T03:42:33.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.341+0000 7fc0a559a640 10 monclient: _check_auth_tickets 2024-09-16T03:42:33.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.341+0000 7fc0a559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:42:33.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.341+0000 7fc0a559a640 1 -- 172.21.15.42:0/1153166936 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc0900ab600 con 0x7fc0a80a3d90 2024-09-16T03:42:33.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.341+0000 7fc0a559a640 10 monclient: handle_config config(0 keys) 2024-09-16T03:42:33.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.341+0000 7fc0acd66640 5 monclient: authenticate success, global_id 4401 2024-09-16T03:42:33.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.341+0000 7fc0a559a640 1 -- 172.21.15.42:0/1153166936 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0900ab9d0 con 0x7fc0a80a3d90 2024-09-16T03:42:33.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.341+0000 7fc0acd66640 10 monclient: get_monmap_and_config success 2024-09-16T03:42:33.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.341+0000 7fc0acd66640 4 set_mon_vals no callback set 2024-09-16T03:42:33.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.341+0000 7fc0a559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:33.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.341+0000 7fc0a559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:42:33.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.341+0000 7fc0a559a640 10 monclient: dump: 2024-09-16T03:42:33.358 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:33.358 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:33.358 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:33.358 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:33.358 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:33.358 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:33.358 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:33.358 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:33.358 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:33.358 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:33.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.341+0000 7fc0acd66640 10 monclient: shutdown 2024-09-16T03:42:33.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.341+0000 7fc0acd66640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:42:33.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.341+0000 7fc0acd66640 1 -- 172.21.15.42:0/1153166936 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc0a80a3d90 msgr2=0x7fc0a8137000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:33.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.341+0000 7fc0acd66640 1 --2- 172.21.15.42:0/1153166936 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc0a80a3d90 0x7fc0a8137000 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fc0a80a7750 tx=0x7fc09001c340 comp rx=0 tx=0).stop 2024-09-16T03:42:33.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.341+0000 7fc0acd66640 1 -- 172.21.15.42:0/1153166936 shutdown_connections 2024-09-16T03:42:33.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.341+0000 7fc0acd66640 1 --2- 172.21.15.42:0/1153166936 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc0a8137540 0x7fc0a8137990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:33.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.341+0000 7fc0acd66640 1 --2- 172.21.15.42:0/1153166936 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc0a80a3d90 0x7fc0a8137000 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:33.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.341+0000 7fc0acd66640 1 --2- 172.21.15.42:0/1153166936 >> v2:172.21.15.154:3300/0 conn(0x7fc0a80a39b0 0x7fc0a8133b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:33.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.341+0000 7fc0acd66640 1 -- 172.21.15.42:0/1153166936 >> 172.21.15.42:0/1153166936 conn(0x7fc0a80a9fb0 msgr2=0x7fc0a80aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:42:33.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.341+0000 7fc0acd66640 1 -- 172.21.15.42:0/1153166936 shutdown_connections 2024-09-16T03:42:33.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.341+0000 7fc0acd66640 1 -- 172.21.15.42:0/1153166936 wait complete. 2024-09-16T03:42:33.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.341+0000 7fc0acd66640 10 monclient: build_initial_monmap 2024-09-16T03:42:33.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.341+0000 7fc0acd66640 10 monclient: monmap: 2024-09-16T03:42:33.359 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:42:33.360 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:42:33.360 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:42:33.360 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:42:33.360 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:42:33.360 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:33.360 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:42:33.360 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:42:33.360 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:42:33.360 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:33.360 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.341+0000 7fc0acd66640 1 librados: starting msgr at 2024-09-16T03:42:33.360 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.341+0000 7fc0acd66640 1 librados: starting objecter 2024-09-16T03:42:33.360 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.341+0000 7fc0acd66640 1 Processor -- start 2024-09-16T03:42:33.360 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.341+0000 7fc0acd66640 1 -- start start 2024-09-16T03:42:33.360 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.341+0000 7fc0acd66640 1 librados: setting wanted keys 2024-09-16T03:42:33.361 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.341+0000 7fc0acd66640 1 librados: calling monclient init 2024-09-16T03:42:33.361 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.341+0000 7fc0acd66640 10 monclient: init 2024-09-16T03:42:33.361 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.341+0000 7fc0acd66640 10 monclient: _reopen_session rank -1 2024-09-16T03:42:33.361 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.341+0000 7fc0acd66640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T03:42:33.361 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.341+0000 7fc0acd66640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc0a80a39b0 0x7fc0a81cf4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:33.361 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.341+0000 7fc0acd66640 10 monclient(hunting): picked mon.noname-a con 0x7fc0a80a39b0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:42:33.361 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.341+0000 7fc0acd66640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc0a80a3d90 0x7fc0a80a2f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:33.361 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.341+0000 7fc0acd66640 10 monclient(hunting): picked mon.noname-b con 0x7fc0a80a3d90 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:42:33.361 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.341+0000 7fc0acd66640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc0a8137540 0x7fc0a81cbfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:33.361 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.341+0000 7fc0acd66640 10 monclient(hunting): picked mon.noname-c con 0x7fc0a8137540 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:42:33.361 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.341+0000 7fc0acd66640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:33.361 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.341+0000 7fc0acd66640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fc0a8140dc0 con 0x7fc0a80a39b0 2024-09-16T03:42:33.361 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.341+0000 7fc0acd66640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:33.361 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.341+0000 7fc0acd66640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc0a8140f40 con 0x7fc0a80a3d90 2024-09-16T03:42:33.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.341+0000 7fc0acd66640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:33.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.341+0000 7fc0acd66640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fc0a813d770 con 0x7fc0a8137540 2024-09-16T03:42:33.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.341+0000 7fc0acd66640 10 monclient(hunting): _renew_subs 2024-09-16T03:42:33.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.341+0000 7fc0acd66640 10 monclient(hunting): authenticate will time out at 1000.315580s 2024-09-16T03:42:33.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.341+0000 7fc0a659c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc0a80a39b0 0x7fc0a81cf4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:33.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.341+0000 7fc0a6d9d640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc0a8137540 0x7fc0a81cbfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:33.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.341+0000 7fc0a659c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc0a80a39b0 0x7fc0a81cf4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:37938/0 (socket says 172.21.15.42:37938) 2024-09-16T03:42:33.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.341+0000 7fc0a659c640 1 -- 172.21.15.42:0/3671614257 learned_addr learned my addr 172.21.15.42:0/3671614257 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:42:33.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.341+0000 7fc0a6d9d640 10 monclient(hunting): get_auth_request con 0x7fc0a8137540 auth_method 0 2024-09-16T03:42:33.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.341+0000 7fc0a6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:33.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.341+0000 7fc0a6d9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:33.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.341+0000 7fc0a6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:33.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.341+0000 7fc0a659c640 10 monclient(hunting): get_auth_request con 0x7fc0a80a39b0 auth_method 0 2024-09-16T03:42:33.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.341+0000 7fc0a659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:33.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.341+0000 7fc0a659c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:33.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.341+0000 7fc0a659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:33.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.341+0000 7fc0a6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:33.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.341+0000 7fc0a6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:33.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.341+0000 7fc0a6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:33.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.341+0000 7fc0a659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:33.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.341+0000 7fc0a659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:33.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.341+0000 7fc0a659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:33.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.341+0000 7fc0a5d9b640 1 --2- 172.21.15.42:0/3671614257 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc0a80a3d90 0x7fc0a80a2f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:33.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.345+0000 7fc0a5d9b640 10 monclient(hunting): get_auth_request con 0x7fc0a80a3d90 auth_method 0 2024-09-16T03:42:33.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.345+0000 7fc0a5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:33.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.345+0000 7fc0a5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:33.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.345+0000 7fc0a5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:33.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.345+0000 7fc0a659c640 10 monclient(hunting): handle_auth_done global_id 4404 payload 824 2024-09-16T03:42:33.364 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.345+0000 7fc0a659c640 1 -- 172.21.15.42:0/3671614257 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc0a8137540 msgr2=0x7fc0a81cbfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:33.364 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.345+0000 7fc0a659c640 1 --2- 172.21.15.42:0/3671614257 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc0a8137540 0x7fc0a81cbfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:33.364 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.345+0000 7fc0a659c640 1 -- 172.21.15.42:0/3671614257 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc0a80a3d90 msgr2=0x7fc0a80a2f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:33.364 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.345+0000 7fc0a659c640 1 --2- 172.21.15.42:0/3671614257 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc0a80a3d90 0x7fc0a80a2f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:33.364 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.345+0000 7fc0a659c640 10 monclient: _finish_hunting 0 2024-09-16T03:42:33.364 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.345+0000 7fc0a659c640 1 monclient: found mon.noname-a 2024-09-16T03:42:33.364 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.345+0000 7fc0a659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:42:33.364 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.345+0000 7fc0a659c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:42:33.364 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.345+0000 7fc0a659c640 1 -- 172.21.15.42:0/3671614257 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0a81cff20 con 0x7fc0a80a39b0 2024-09-16T03:42:33.364 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.345+0000 7fc0a6d9d640 1 --2- 172.21.15.42:0/3671614257 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc0a8137540 0x7fc0a81cbfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:42:33.364 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.345+0000 7fc0a659c640 1 --2- 172.21.15.42:0/3671614257 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc0a80a39b0 0x7fc0a81cf4c0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fc0a000cf60 tx=0x7fc0a000cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:33.364 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.345+0000 7fc09effd640 10 client.?.objecter ms_handle_connect 0x7fc0a80a39b0 2024-09-16T03:42:33.364 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.345+0000 7fc09effd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:42:33.364 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.345+0000 7fc09effd640 1 -- 172.21.15.42:0/3671614257 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0a001a070 con 0x7fc0a80a39b0 2024-09-16T03:42:33.365 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.345+0000 7fc09effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:33.365 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.345+0000 7fc09effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:42:33.365 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.345+0000 7fc09effd640 10 monclient: dump: 2024-09-16T03:42:33.365 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:33.365 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:33.365 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:33.365 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:33.365 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:33.365 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:33.365 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:33.365 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:33.365 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:33.365 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:33.365 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.345+0000 7fc09effd640 10 monclient: _finish_auth 0 2024-09-16T03:42:33.366 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.345+0000 7fc09effd640 10 monclient: _check_auth_tickets 2024-09-16T03:42:33.366 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.345+0000 7fc09effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:42:33.366 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.345+0000 7fc09effd640 1 -- 172.21.15.42:0/3671614257 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc0a000d840 con 0x7fc0a80a39b0 2024-09-16T03:42:33.366 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.345+0000 7fc09effd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:42:33.366 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.345+0000 7fc09effd640 1 -- 172.21.15.42:0/3671614257 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0a000dc10 con 0x7fc0a80a39b0 2024-09-16T03:42:33.366 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.345+0000 7fc09effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:33.366 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.345+0000 7fc09effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:42:33.366 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.345+0000 7fc09effd640 10 monclient: dump: 2024-09-16T03:42:33.366 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:33.366 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:33.366 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:33.366 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:33.366 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:33.366 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:33.367 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:33.367 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:33.367 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:33.367 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:33.367 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.345+0000 7fc0acd66640 5 monclient: authenticate success, global_id 4404 2024-09-16T03:42:33.367 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.345+0000 7fc0a4d99640 4 set_mon_vals no callback set 2024-09-16T03:42:33.367 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.345+0000 7fc0acd66640 10 monclient: _renew_subs 2024-09-16T03:42:33.367 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.345+0000 7fc0acd66640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:42:33.367 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.345+0000 7fc0acd66640 1 -- 172.21.15.42:0/3671614257 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0a8132f20 con 0x7fc0a80a39b0 2024-09-16T03:42:33.367 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.345+0000 7fc0acd66640 10 client.4404.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:42:33.367 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.345+0000 7fc0acd66640 10 monclient: _renew_subs 2024-09-16T03:42:33.367 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.345+0000 7fc0acd66640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:42:33.367 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.345+0000 7fc0acd66640 1 -- 172.21.15.42:0/3671614257 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0a809d410 con 0x7fc0a80a39b0 2024-09-16T03:42:33.367 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.345+0000 7fc0acd66640 1 librados: init done 2024-09-16T03:42:33.368 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.345+0000 7fc09effd640 1 -- 172.21.15.42:0/3671614257 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fc0a001d070 con 0x7fc0a80a39b0 2024-09-16T03:42:33.368 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.345+0000 7fc0acd66640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:42:33.368 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.345+0000 7fc0acd66640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:42:33.368 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.345+0000 7fc0acd66640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:42:33.368 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.345+0000 7fc0acd66640 1 -- 172.21.15.42:0/3671614257 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc0a8133b80 con 0x7fc0a80a39b0 2024-09-16T03:42:33.368 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.345+0000 7fc09effd640 1 --2- 172.21.15.42:0/3671614257 >> v2:172.21.15.154:6812/254845342 conn(0x7fc088037810 0x7fc088039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:33.368 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.345+0000 7fc09effd640 1 -- 172.21.15.42:0/3671614257 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 3534+0+0 (secure 0 0 0) 0x7fc0a0048fe0 con 0x7fc0a80a39b0 2024-09-16T03:42:33.368 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.345+0000 7fc09effd640 10 client.4404.objecter ms_dispatch 0x7fc0a8000ba0 osd_map(19..19 src has 1..19) 2024-09-16T03:42:33.368 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.345+0000 7fc09effd640 3 client.4404.objecter handle_osd_map got epochs [19,19] > 0 2024-09-16T03:42:33.368 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.345+0000 7fc09effd640 3 client.4404.objecter handle_osd_map decoding full epoch 19 2024-09-16T03:42:33.368 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.345+0000 7fc09effd640 20 client.4404.objecter dump_active .. 0 homeless 2024-09-16T03:42:33.368 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.345+0000 7fc0a5d9b640 1 --2- 172.21.15.42:0/3671614257 >> v2:172.21.15.154:6812/254845342 conn(0x7fc088037810 0x7fc088039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:33.368 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.345+0000 7fc0a5d9b640 10 monclient: get_auth_request con 0x7fc088037810 auth_method 0 2024-09-16T03:42:33.368 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.345+0000 7fc0a5d9b640 1 --2- 172.21.15.42:0/3671614257 >> v2:172.21.15.154:6812/254845342 conn(0x7fc088037810 0x7fc088039cd0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fc0900ad000 tx=0x7fc090008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:33.369 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.345+0000 7fc09effd640 10 client.4404.objecter ms_handle_connect 0x7fc088037810 2024-09-16T03:42:33.369 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.349+0000 7fc09effd640 1 -- 172.21.15.42:0/3671614257 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7fc0a81cff20 con 0x7fc0a80a39b0 2024-09-16T03:42:33.369 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.349+0000 7fc09effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:42:33.369 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.349+0000 7fc09effd640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:42:33.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.505+0000 7fc0acd66640 1 -- 172.21.15.42:0/3671614257 --> v2:172.21.15.154:6812/254845342 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fc0a80a5750 con 0x7fc088037810 2024-09-16T03:42:33.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.509+0000 7fc09effd640 1 -- 172.21.15.42:0/3671614257 <== mgr.4102 v2:172.21.15.154:6812/254845342 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+34079 (secure 0 0 0) 0x7fc0a80a5750 con 0x7fc088037810 2024-09-16T03:42:33.516 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-09-16T03:42:33.516 INFO:teuthology.orchestra.run.smithi042.stderr:dumped all 2024-09-16T03:42:33.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.509+0000 7fc0acd66640 10 librados: watch_flush enter 2024-09-16T03:42:33.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.509+0000 7fc0acd66640 10 librados: watch_flush exit 2024-09-16T03:42:33.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.509+0000 7fc0acd66640 20 client.4404.objecter shutdown clearing up homeless session... 2024-09-16T03:42:33.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.509+0000 7fc0acd66640 10 client.4404.objecter successfully canceled tick 2024-09-16T03:42:33.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.509+0000 7fc0acd66640 1 -- 172.21.15.42:0/3671614257 >> v2:172.21.15.154:6812/254845342 conn(0x7fc088037810 msgr2=0x7fc088039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:33.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.509+0000 7fc0acd66640 1 --2- 172.21.15.42:0/3671614257 >> v2:172.21.15.154:6812/254845342 conn(0x7fc088037810 0x7fc088039cd0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fc0900ad000 tx=0x7fc090008040 comp rx=0 tx=0).stop 2024-09-16T03:42:33.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.509+0000 7fc0acd66640 10 monclient: shutdown 2024-09-16T03:42:33.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.509+0000 7fc0acd66640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:42:33.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.509+0000 7fc0acd66640 1 -- 172.21.15.42:0/3671614257 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc0a80a39b0 msgr2=0x7fc0a81cf4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:33.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.509+0000 7fc0acd66640 1 --2- 172.21.15.42:0/3671614257 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc0a80a39b0 0x7fc0a81cf4c0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fc0a000cf60 tx=0x7fc0a000cf90 comp rx=0 tx=0).stop 2024-09-16T03:42:33.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.509+0000 7fc0acd66640 1 -- 172.21.15.42:0/3671614257 shutdown_connections 2024-09-16T03:42:33.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.509+0000 7fc0acd66640 1 --2- 172.21.15.42:0/3671614257 >> v2:172.21.15.154:6812/254845342 conn(0x7fc088037810 0x7fc088039cd0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:33.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.509+0000 7fc0acd66640 1 --2- 172.21.15.42:0/3671614257 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc0a8137540 0x7fc0a81cbfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:33.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.509+0000 7fc0acd66640 1 --2- 172.21.15.42:0/3671614257 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc0a80a3d90 0x7fc0a80a2f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:33.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.509+0000 7fc0acd66640 1 --2- 172.21.15.42:0/3671614257 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc0a80a39b0 0x7fc0a81cf4c0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:33.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.509+0000 7fc0acd66640 1 -- 172.21.15.42:0/3671614257 >> 172.21.15.42:0/3671614257 conn(0x7fc0a80a9fb0 msgr2=0x7fc0a80a8490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:42:33.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.509+0000 7fc0acd66640 1 -- 172.21.15.42:0/3671614257 shutdown_connections 2024-09-16T03:42:33.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.509+0000 7fc0acd66640 1 -- 172.21.15.42:0/3671614257 wait complete. 2024-09-16T03:42:33.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:33.509+0000 7fc0acd66640 1 librados: shutdown 2024-09-16T03:42:33.533 INFO:teuthology.orchestra.run.smithi042.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2024-09-16T03:42:33.472618+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":161,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120,"ondisk_log_size":120,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":167392,"kb_used_data":1064,"kb_used_omap":9,"kb_used_meta":166198,"kb_avail":566063648,"statfs":{"total":579820584960,"available":579649175552,"internally_reserved":0,"allocated":1089536,"data_stored":1376026,"data_compressed":18838,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":9567,"internal_metadata":170187425},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.865877"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":21,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T03:42:30.649077+0000","last_change":"2024-09-16T03:42:27.624922+0000","last_active":"2024-09-16T03:42:30.649077+0000","last_peered":"2024-09-16T03:42:30.649077+0000","last_clean":"2024-09-16T03:42:30.649077+0000","last_became_active":"2024-09-16T03:42:25.615293+0000","last_became_peered":"2024-09-16T03:42:25.615293+0000","last_unstale":"2024-09-16T03:42:30.649077+0000","last_undegraded":"2024-09-16T03:42:30.649077+0000","last_fullsized":"2024-09-16T03:42:30.649077+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:42:24.583922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:42:24.583922+0000","last_clean_scrub_stamp":"2024-09-16T03:42:24.583922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:46:11.677251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010681690000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":21,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T03:42:30.645363+0000","last_change":"2024-09-16T03:42:27.631227+0000","last_active":"2024-09-16T03:42:30.645363+0000","last_peered":"2024-09-16T03:42:30.645363+0000","last_clean":"2024-09-16T03:42:30.645363+0000","last_became_active":"2024-09-16T03:42:25.608453+0000","last_became_peered":"2024-09-16T03:42:25.608453+0000","last_unstale":"2024-09-16T03:42:30.645363+0000","last_undegraded":"2024-09-16T03:42:30.645363+0000","last_fullsized":"2024-09-16T03:42:30.645363+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:42:24.583922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:42:24.583922+0000","last_clean_scrub_stamp":"2024-09-16T03:42:24.583922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:00:11.921493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001163533,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":21,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T03:42:30.649062+0000","last_change":"2024-09-16T03:42:27.625091+0000","last_active":"2024-09-16T03:42:30.649062+0000","last_peered":"2024-09-16T03:42:30.649062+0000","last_clean":"2024-09-16T03:42:30.649062+0000","last_became_active":"2024-09-16T03:42:25.614458+0000","last_became_peered":"2024-09-16T03:42:25.614458+0000","last_unstale":"2024-09-16T03:42:30.649062+0000","last_undegraded":"2024-09-16T03:42:30.649062+0000","last_fullsized":"2024-09-16T03:42:30.649062+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:42:24.583922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:42:24.583922+0000","last_clean_scrub_stamp":"2024-09-16T03:42:24.583922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:26:14.681436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012701769999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":21,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T03:42:30.645288+0000","last_change":"2024-09-16T03:42:27.631260+0000","last_active":"2024-09-16T03:42:30.645288+0000","last_peered":"2024-09-16T03:42:30.645288+0000","last_clean":"2024-09-16T03:42:30.645288+0000","last_became_active":"2024-09-16T03:42:25.609631+0000","last_became_peered":"2024-09-16T03:42:25.609631+0000","last_unstale":"2024-09-16T03:42:30.645288+0000","last_undegraded":"2024-09-16T03:42:30.645288+0000","last_fullsized":"2024-09-16T03:42:30.645288+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:42:24.583922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:42:24.583922+0000","last_clean_scrub_stamp":"2024-09-16T03:42:24.583922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:58:49.091085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001288897,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":21,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T03:42:27.633516+0000","last_change":"2024-09-16T03:42:27.632050+0000","last_active":"2024-09-16T03:42:27.633516+0000","last_peered":"2024-09-16T03:42:27.633516+0000","last_clean":"2024-09-16T03:42:27.633516+0000","last_became_active":"2024-09-16T03:42:25.617896+0000","last_became_peered":"2024-09-16T03:42:25.617896+0000","last_unstale":"2024-09-16T03:42:27.633516+0000","last_undegraded":"2024-09-16T03:42:27.633516+0000","last_fullsized":"2024-09-16T03:42:27.633516+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:42:24.583922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:42:24.583922+0000","last_clean_scrub_stamp":"2024-09-16T03:42:24.583922+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:41:08.754274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021072529999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":21,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T03:42:30.903523+0000","last_change":"2024-09-16T03:42:27.619252+0000","last_active":"2024-09-16T03:42:30.903523+0000","last_peered":"2024-09-16T03:42:30.903523+0000","last_clean":"2024-09-16T03:42:30.903523+0000","last_became_active":"2024-09-16T03:42:25.607533+0000","last_became_peered":"2024-09-16T03:42:25.607533+0000","last_unstale":"2024-09-16T03:42:30.903523+0000","last_undegraded":"2024-09-16T03:42:30.903523+0000","last_fullsized":"2024-09-16T03:42:30.903523+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:42:24.583922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:42:24.583922+0000","last_clean_scrub_stamp":"2024-09-16T03:42:24.583922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:25:58.536144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089083299999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":21,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T03:42:30.649015+0000","last_change":"2024-09-16T03:42:27.624504+0000","last_active":"2024-09-16T03:42:30.649015+0000","last_peered":"2024-09-16T03:42:30.649015+0000","last_clean":"2024-09-16T03:42:30.649015+0000","last_became_active":"2024-09-16T03:42:25.612386+0000","last_became_peered":"2024-09-16T03:42:25.612386+0000","last_unstale":"2024-09-16T03:42:30.649015+0000","last_undegraded":"2024-09-16T03:42:30.649015+0000","last_fullsized":"2024-09-16T03:42:30.649015+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:42:24.583922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:42:24.583922+0000","last_clean_scrub_stamp":"2024-09-16T03:42:24.583922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:00:49.446880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081399800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":20,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-16T03:42:27.630965+0000","last_change":"2024-09-16T03:42:27.630965+0000","last_active":"2024-09-16T03:42:27.630965+0000","last_peered":"2024-09-16T03:42:27.630965+0000","last_clean":"2024-09-16T03:42:27.630965+0000","last_became_active":"2024-09-16T03:42:25.615930+0000","last_became_peered":"2024-09-16T03:42:25.615930+0000","last_unstale":"2024-09-16T03:42:27.630965+0000","last_undegraded":"2024-09-16T03:42:27.630965+0000","last_fullsized":"2024-09-16T03:42:27.630965+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:42:24.583922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:42:24.583922+0000","last_clean_scrub_stamp":"2024-09-16T03:42:24.583922+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:59:59.653118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00099470900000000009,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"14'117","reported_seq":172,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-16T03:42:30.649173+0000","last_change":"2024-09-16T03:42:18.799294+0000","last_active":"2024-09-16T03:42:30.649173+0000","last_peered":"2024-09-16T03:42:30.649173+0000","last_clean":"2024-09-16T03:42:30.649173+0000","last_became_active":"2024-09-16T03:42:18.797239+0000","last_became_peered":"2024-09-16T03:42:18.797239+0000","last_unstale":"2024-09-16T03:42:30.649173+0000","last_undegraded":"2024-09-16T03:42:30.649173+0000","last_fullsized":"2024-09-16T03:42:30.649173+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:42:17.544237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:42:17.544237+0000","last_clean_scrub_stamp":"2024-09-16T03:42:17.544237+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:01:21.017593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":18838,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":12,"seq":51539607558,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27188,"kb_used_data":156,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344652,"statfs":{"total":96636764160,"available":96608923648,"internally_reserved":0,"allocated":159744,"data_stored":49906,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":11,"seq":47244640261,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27188,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344652,"statfs":{"total":96636764160,"available":96608923648,"internally_reserved":0,"allocated":163840,"data_stored":53649,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640261,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":217088,"data_stored":582577,"data_compressed":9419,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640261,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27188,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344652,"statfs":{"total":96636764160,"available":96608923648,"internally_reserved":0,"allocated":163840,"data_stored":53649,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640263,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31348,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340492,"statfs":{"total":96636764160,"available":96604663808,"internally_reserved":0,"allocated":167936,"data_stored":53668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640261,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":217088,"data_stored":582577,"data_compressed":9419,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":9419,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":9419,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T03:42:33.534 INFO:tasks.mon_thrash.ceph_manager:clean! 2024-09-16T03:42:33.534 DEBUG:tasks.mon_thrash:Yielding 2024-09-16T03:42:33.534 INFO:teuthology.run_tasks:Running task workunit... 2024-09-16T03:42:34.437 INFO:tasks.workunit:Pulling workunits from ref ad35db06e8924912f01e32b35576297c0f1352b4 2024-09-16T03:42:34.438 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-09-16T03:42:34.438 DEBUG:teuthology.orchestra.run.smithi154:> stat -- /home/ubuntu/cephtest/mnt.0 2024-09-16T03:42:34.438 INFO:tasks.mon_thrash.mon_thrasher:start thrashing 2024-09-16T03:42:34.439 INFO:tasks.mon_thrash.mon_thrasher:seed: 1726458153, revive delay: 20.0, thrash delay: 1.0 thrash many: False, maintain quorum: True store thrash: False, probability: 50 freeze mon: prob 10.0 duration 15.0 2024-09-16T03:42:34.439 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T03:42:34.439 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:42:34.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T03:42:34.445 INFO:teuthology.orchestra.run.smithi154.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-09-16T03:42:34.445 DEBUG:teuthology.orchestra.run.smithi154:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-09-16T03:42:34.491 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-09-16T03:42:34.491 DEBUG:teuthology.orchestra.run.smithi154:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-09-16T03:42:34.541 INFO:tasks.workunit:timeout=3h 2024-09-16T03:42:34.541 INFO:tasks.workunit:cleanup=True 2024-09-16T03:42:34.541 DEBUG:teuthology.orchestra.run.smithi154:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout ad35db06e8924912f01e32b35576297c0f1352b4 2024-09-16T03:42:34.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.564+0000 7fd19dc8a640 10 monclient: get_monmap_and_config 2024-09-16T03:42:34.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.564+0000 7fd19dc8a640 10 monclient: build_initial_monmap 2024-09-16T03:42:34.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.564+0000 7fd19dc8a640 10 monclient: monmap: 2024-09-16T03:42:34.571 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:42:34.571 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:34.572 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:42:34.569138+0000 2024-09-16T03:42:34.572 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:42:34.569138+0000 2024-09-16T03:42:34.572 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:42:34.572 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:34.572 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:42:34.572 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:42:34.572 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:42:34.572 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:42:34.572 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:42:34.573 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:34.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.564+0000 7fd19dc8a640 1 Processor -- start 2024-09-16T03:42:34.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.564+0000 7fd19dc8a640 1 -- start start 2024-09-16T03:42:34.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.564+0000 7fd19dc8a640 10 monclient: init 2024-09-16T03:42:34.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.564+0000 7fd19dc8a640 10 monclient: _reopen_session rank -1 2024-09-16T03:42:34.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.564+0000 7fd19dc8a640 10 monclient: _add_conns ranks=[4,2,0,1,3] 2024-09-16T03:42:34.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.564+0000 7fd19dc8a640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fd1980a1330 0x7fd19809f180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:34.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.564+0000 7fd19dc8a640 10 monclient(hunting): picked mon.noname-c con 0x7fd1980a1330 addr v2:172.21.15.154:3300/0 2024-09-16T03:42:34.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.564+0000 7fd19dc8a640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd1980a1710 0x7fd1980a1b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:34.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.564+0000 7fd19dc8a640 10 monclient(hunting): picked mon.noname-b con 0x7fd1980a1710 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:42:34.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.564+0000 7fd19dc8a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd1981340f0 addr v1:172.21.15.42:6789/0 2024-09-16T03:42:34.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.564+0000 7fd19dc8a640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd19809e4c0 con 0x7fd1981340f0 2024-09-16T03:42:34.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.564+0000 7fd19dc8a640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:34.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.564+0000 7fd19dc8a640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fd19809edf0 con 0x7fd1980a1330 2024-09-16T03:42:34.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.564+0000 7fd19dc8a640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:34.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.564+0000 7fd19dc8a640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fd19809cd60 con 0x7fd1980a1710 2024-09-16T03:42:34.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.564+0000 7fd19dc8a640 10 monclient(hunting): _renew_subs 2024-09-16T03:42:34.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.564+0000 7fd19dc8a640 10 monclient(hunting): authenticate will time out at 1001.538512s 2024-09-16T03:42:34.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.564+0000 7fd197fff640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fd1981340f0 0x7fd1981345a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:55838/0 (socket says 172.21.15.42:55838) 2024-09-16T03:42:34.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.564+0000 7fd197fff640 1 -- 172.21.15.42:0/3308084653 learned_addr learned my addr 172.21.15.42:0/3308084653 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:42:34.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.564+0000 7fd196ffd640 1 --2- 172.21.15.42:0/3308084653 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd1980a1710 0x7fd1980a1b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:34.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.564+0000 7fd196ffd640 10 monclient(hunting): get_auth_request con 0x7fd1980a1710 auth_method 0 2024-09-16T03:42:34.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.564+0000 7fd196ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:34.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.564+0000 7fd196ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:34.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.564+0000 7fd196ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:34.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.564+0000 7fd196ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:34.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.564+0000 7fd196ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:34.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.564+0000 7fd196ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:34.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.564+0000 7fd1977fe640 1 --2- 172.21.15.42:0/3308084653 >> v2:172.21.15.154:3300/0 conn(0x7fd1980a1330 0x7fd19809f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:34.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.564+0000 7fd196ffd640 10 monclient(hunting): handle_auth_done global_id 4343 payload 274 2024-09-16T03:42:34.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.564+0000 7fd196ffd640 1 -- 172.21.15.42:0/3308084653 >> v2:172.21.15.154:3300/0 conn(0x7fd1980a1330 msgr2=0x7fd19809f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:42:34.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.564+0000 7fd196ffd640 1 --2- 172.21.15.42:0/3308084653 >> v2:172.21.15.154:3300/0 conn(0x7fd1980a1330 0x7fd19809f180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:34.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.564+0000 7fd196ffd640 1 -- 172.21.15.42:0/3308084653 >> v1:172.21.15.42:6789/0 conn(0x7fd1981340f0 legacy=0x7fd1981345a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:34.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.564+0000 7fd196ffd640 10 monclient: _finish_hunting 0 2024-09-16T03:42:34.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.564+0000 7fd196ffd640 1 monclient: found mon.noname-b 2024-09-16T03:42:34.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.564+0000 7fd1967fc640 1 -- 172.21.15.42:0/3308084653 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 393968871 0 0) 0x7fd19809e4c0 con 0x7fd1981340f0 2024-09-16T03:42:34.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.564+0000 7fd196ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:42:34.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.564+0000 7fd196ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T03:42:34.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.564+0000 7fd196ffd640 1 -- 172.21.15.42:0/3308084653 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1980a2050 con 0x7fd1980a1710 2024-09-16T03:42:34.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.564+0000 7fd1967fc640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T03:42:34.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.568+0000 7fd196ffd640 1 --2- 172.21.15.42:0/3308084653 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd1980a1710 0x7fd1980a1b10 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fd18c0097f0 tx=0x7fd18c01c360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bcb7b3928e00f043 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:34.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.568+0000 7fd1967fc640 1 -- 172.21.15.42:0/3308084653 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd18c0b0070 con 0x7fd1980a1710 2024-09-16T03:42:34.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.568+0000 7fd1967fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:34.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.568+0000 7fd1967fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:42:34.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.568+0000 7fd1967fc640 10 monclient: dump: 2024-09-16T03:42:34.583 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:34.583 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:34.583 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:34.583 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:34.583 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:34.583 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:34.583 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:34.583 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:34.583 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:34.583 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:34.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.568+0000 7fd1967fc640 10 monclient: _finish_auth 0 2024-09-16T03:42:34.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.568+0000 7fd1967fc640 10 monclient: _check_auth_tickets 2024-09-16T03:42:34.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.568+0000 7fd1967fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:42:34.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.568+0000 7fd1967fc640 1 -- 172.21.15.42:0/3308084653 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd18c0ab600 con 0x7fd1980a1710 2024-09-16T03:42:34.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.568+0000 7fd19dc8a640 5 monclient: authenticate success, global_id 4343 2024-09-16T03:42:34.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.568+0000 7fd19dc8a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:42:34.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.568+0000 7fd1967fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:42:34.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.568+0000 7fd19dc8a640 10 monclient: get_monmap_and_config success 2024-09-16T03:42:34.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.568+0000 7fd19dc8a640 4 set_mon_vals no callback set 2024-09-16T03:42:34.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.568+0000 7fd1967fc640 1 -- 172.21.15.42:0/3308084653 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd18c0ab9d0 con 0x7fd1980a1710 2024-09-16T03:42:34.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.568+0000 7fd1967fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:34.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.568+0000 7fd1967fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:42:34.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.568+0000 7fd1967fc640 10 monclient: dump: 2024-09-16T03:42:34.585 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:34.585 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:34.585 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:34.585 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:34.585 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:34.585 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:34.585 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:34.585 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:34.585 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:34.585 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:34.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.568+0000 7fd19dc8a640 10 monclient: shutdown 2024-09-16T03:42:34.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.568+0000 7fd19dc8a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:42:34.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.568+0000 7fd19dc8a640 1 -- 172.21.15.42:0/3308084653 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd1980a1710 msgr2=0x7fd1980a1b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:34.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.568+0000 7fd19dc8a640 1 --2- 172.21.15.42:0/3308084653 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd1980a1710 0x7fd1980a1b10 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fd18c0097f0 tx=0x7fd18c01c360 comp rx=0 tx=0).stop 2024-09-16T03:42:34.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.568+0000 7fd19dc8a640 1 -- 172.21.15.42:0/3308084653 shutdown_connections 2024-09-16T03:42:34.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.568+0000 7fd19dc8a640 1 --2- 172.21.15.42:0/3308084653 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd1980a1710 0x7fd1980a1b10 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:34.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.568+0000 7fd19dc8a640 1 --2- 172.21.15.42:0/3308084653 >> v2:172.21.15.154:3300/0 conn(0x7fd1980a1330 0x7fd19809f180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:34.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.568+0000 7fd19dc8a640 1 -- 172.21.15.42:0/3308084653 >> 172.21.15.42:0/3308084653 conn(0x7fd1980a2420 msgr2=0x7fd1980a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:42:34.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.568+0000 7fd19dc8a640 1 -- 172.21.15.42:0/3308084653 shutdown_connections 2024-09-16T03:42:34.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.568+0000 7fd19dc8a640 1 -- 172.21.15.42:0/3308084653 wait complete. 2024-09-16T03:42:34.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.568+0000 7fd19dc8a640 10 monclient: build_initial_monmap 2024-09-16T03:42:34.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.568+0000 7fd19dc8a640 10 monclient: monmap: 2024-09-16T03:42:34.586 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:42:34.586 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:42:34.586 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:42:34.587 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:42:34.587 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:42:34.587 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:34.587 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:42:34.587 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:42:34.587 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:42:34.587 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:34.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.568+0000 7fd19dc8a640 1 librados: starting msgr at 2024-09-16T03:42:34.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.568+0000 7fd19dc8a640 1 librados: starting objecter 2024-09-16T03:42:34.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.568+0000 7fd19dc8a640 1 Processor -- start 2024-09-16T03:42:34.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.568+0000 7fd19dc8a640 1 -- start start 2024-09-16T03:42:34.588 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.568+0000 7fd19dc8a640 1 librados: setting wanted keys 2024-09-16T03:42:34.588 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.568+0000 7fd19dc8a640 1 librados: calling monclient init 2024-09-16T03:42:34.588 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.568+0000 7fd19dc8a640 10 monclient: init 2024-09-16T03:42:34.588 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.568+0000 7fd19dc8a640 10 monclient: _reopen_session rank -1 2024-09-16T03:42:34.588 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.568+0000 7fd19dc8a640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:42:34.588 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.568+0000 7fd19dc8a640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd1980a1330 0x7fd198138630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:34.588 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.568+0000 7fd19dc8a640 10 monclient(hunting): picked mon.noname-c con 0x7fd1980a1330 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:42:34.588 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.568+0000 7fd19dc8a640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd1980a1710 0x7fd198138b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:34.588 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.568+0000 7fd19dc8a640 10 monclient(hunting): picked mon.noname-b con 0x7fd1980a1710 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:42:34.588 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.568+0000 7fd19dc8a640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd1981340f0 0x7fd1981390b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:34.588 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.568+0000 7fd19dc8a640 10 monclient(hunting): picked mon.noname-a con 0x7fd1981340f0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:42:34.588 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.568+0000 7fd19dc8a640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:34.589 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.568+0000 7fd19dc8a640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fd19809bbd0 con 0x7fd1981340f0 2024-09-16T03:42:34.589 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.568+0000 7fd19dc8a640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:34.589 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.568+0000 7fd19dc8a640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd19809ba50 con 0x7fd1980a1710 2024-09-16T03:42:34.589 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.568+0000 7fd19dc8a640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:34.589 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.568+0000 7fd19dc8a640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fd19809bd50 con 0x7fd1980a1330 2024-09-16T03:42:34.589 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.568+0000 7fd19dc8a640 10 monclient(hunting): _renew_subs 2024-09-16T03:42:34.589 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.568+0000 7fd19dc8a640 10 monclient(hunting): authenticate will time out at 1001.543735s 2024-09-16T03:42:34.589 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.568+0000 7fd1977fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd1980a1330 0x7fd198138630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:34.589 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.568+0000 7fd1977fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd1980a1330 0x7fd198138630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:54018/0 (socket says 172.21.15.42:54018) 2024-09-16T03:42:34.589 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.568+0000 7fd1977fe640 1 -- 172.21.15.42:0/3842433300 learned_addr learned my addr 172.21.15.42:0/3842433300 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:42:34.589 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.568+0000 7fd1977fe640 10 monclient(hunting): get_auth_request con 0x7fd1980a1330 auth_method 0 2024-09-16T03:42:34.589 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.568+0000 7fd1977fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:34.590 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.568+0000 7fd1977fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:34.590 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.568+0000 7fd1977fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:34.590 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.568+0000 7fd197fff640 1 --2- 172.21.15.42:0/3842433300 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd1981340f0 0x7fd1981390b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:34.590 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.572+0000 7fd197fff640 10 monclient(hunting): get_auth_request con 0x7fd1981340f0 auth_method 0 2024-09-16T03:42:34.590 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.572+0000 7fd197fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:34.590 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.572+0000 7fd197fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:34.590 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.572+0000 7fd196ffd640 1 --2- 172.21.15.42:0/3842433300 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd1980a1710 0x7fd198138b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:34.590 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.572+0000 7fd197fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:34.590 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.572+0000 7fd1977fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:34.590 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.572+0000 7fd1977fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:34.590 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.572+0000 7fd1977fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:34.590 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.572+0000 7fd196ffd640 10 monclient(hunting): get_auth_request con 0x7fd1980a1710 auth_method 0 2024-09-16T03:42:34.590 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.572+0000 7fd196ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:34.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.572+0000 7fd196ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:34.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.572+0000 7fd196ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:34.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.572+0000 7fd197fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:34.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.572+0000 7fd197fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:34.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.572+0000 7fd197fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:34.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.572+0000 7fd1977fe640 10 monclient(hunting): handle_auth_done global_id 4346 payload 824 2024-09-16T03:42:34.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.572+0000 7fd1977fe640 1 -- 172.21.15.42:0/3842433300 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd1980a1710 msgr2=0x7fd198138b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:34.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.572+0000 7fd1977fe640 1 --2- 172.21.15.42:0/3842433300 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd1980a1710 0x7fd198138b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:34.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.572+0000 7fd1977fe640 1 -- 172.21.15.42:0/3842433300 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd1981340f0 msgr2=0x7fd1981390b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:34.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.572+0000 7fd1977fe640 1 --2- 172.21.15.42:0/3842433300 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd1981340f0 0x7fd1981390b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:34.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.572+0000 7fd1977fe640 10 monclient: _finish_hunting 0 2024-09-16T03:42:34.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.572+0000 7fd1977fe640 1 monclient: found mon.noname-c 2024-09-16T03:42:34.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.572+0000 7fd1977fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:42:34.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.572+0000 7fd1977fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:42:34.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.572+0000 7fd1977fe640 1 -- 172.21.15.42:0/3842433300 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd198121310 con 0x7fd1980a1330 2024-09-16T03:42:34.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.572+0000 7fd1977fe640 1 --2- 172.21.15.42:0/3842433300 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd1980a1330 0x7fd198138630 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fd18800c820 tx=0x7fd18800cce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:34.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.572+0000 7fd17ffff640 10 client.?.objecter ms_handle_connect 0x7fd1980a1330 2024-09-16T03:42:34.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.572+0000 7fd17ffff640 10 client.?.objecter resend_mon_ops 2024-09-16T03:42:34.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.572+0000 7fd17ffff640 1 -- 172.21.15.42:0/3842433300 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd188015070 con 0x7fd1980a1330 2024-09-16T03:42:34.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.572+0000 7fd17ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:34.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.572+0000 7fd17ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:42:34.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.572+0000 7fd17ffff640 10 monclient: dump: 2024-09-16T03:42:34.592 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:34.593 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:34.593 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:34.593 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:34.593 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:34.593 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:34.593 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:34.593 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:34.593 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:34.593 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:34.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.572+0000 7fd17ffff640 10 monclient: _finish_auth 0 2024-09-16T03:42:34.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.572+0000 7fd17ffff640 10 monclient: _check_auth_tickets 2024-09-16T03:42:34.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.572+0000 7fd17ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:42:34.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.572+0000 7fd17ffff640 1 -- 172.21.15.42:0/3842433300 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd188004240 con 0x7fd1980a1330 2024-09-16T03:42:34.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.572+0000 7fd19dc8a640 5 monclient: authenticate success, global_id 4346 2024-09-16T03:42:34.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.572+0000 7fd19dc8a640 10 monclient: _renew_subs 2024-09-16T03:42:34.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.572+0000 7fd19dc8a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:42:34.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.572+0000 7fd19dc8a640 1 -- 172.21.15.42:0/3842433300 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd198072220 con 0x7fd1980a1330 2024-09-16T03:42:34.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.572+0000 7fd17ffff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:42:34.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.572+0000 7fd17ffff640 1 -- 172.21.15.42:0/3842433300 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd18800ce10 con 0x7fd1980a1330 2024-09-16T03:42:34.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.572+0000 7fd17ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:34.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.572+0000 7fd17ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:42:34.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.572+0000 7fd17ffff640 10 monclient: dump: 2024-09-16T03:42:34.594 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:34.594 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:34.595 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:34.595 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:34.595 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:34.595 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:34.595 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:34.595 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:34.595 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:34.595 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:34.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.572+0000 7fd19dc8a640 10 client.4346.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:42:34.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.572+0000 7fd1957fa640 4 set_mon_vals no callback set 2024-09-16T03:42:34.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.572+0000 7fd19dc8a640 10 monclient: _renew_subs 2024-09-16T03:42:34.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.572+0000 7fd19dc8a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:42:34.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.572+0000 7fd19dc8a640 1 -- 172.21.15.42:0/3842433300 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd198120d50 con 0x7fd1980a1330 2024-09-16T03:42:34.596 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.572+0000 7fd19dc8a640 1 librados: init done 2024-09-16T03:42:34.596 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.572+0000 7fd19dc8a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:42:34.596 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.572+0000 7fd19dc8a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:42:34.596 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.572+0000 7fd19dc8a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:42:34.596 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.572+0000 7fd17ffff640 1 -- 172.21.15.42:0/3842433300 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fd188007830 con 0x7fd1980a1330 2024-09-16T03:42:34.596 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.572+0000 7fd19dc8a640 1 -- 172.21.15.42:0/3842433300 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd19809f180 con 0x7fd1980a1330 2024-09-16T03:42:34.596 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.572+0000 7fd17ffff640 1 --2- 172.21.15.42:0/3842433300 >> v2:172.21.15.154:6812/254845342 conn(0x7fd184037810 0x7fd184039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:34.596 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.572+0000 7fd17ffff640 1 -- 172.21.15.42:0/3842433300 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(19..19 src has 1..19) ==== 3534+0+0 (secure 0 0 0) 0x7fd1880484d0 con 0x7fd1980a1330 2024-09-16T03:42:34.596 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.572+0000 7fd17ffff640 10 client.4346.objecter ms_dispatch 0x7fd198000ba0 osd_map(19..19 src has 1..19) 2024-09-16T03:42:34.596 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.572+0000 7fd17ffff640 3 client.4346.objecter handle_osd_map got epochs [19,19] > 0 2024-09-16T03:42:34.596 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.572+0000 7fd17ffff640 3 client.4346.objecter handle_osd_map decoding full epoch 19 2024-09-16T03:42:34.596 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.572+0000 7fd17ffff640 20 client.4346.objecter dump_active .. 0 homeless 2024-09-16T03:42:34.597 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.572+0000 7fd196ffd640 1 --2- 172.21.15.42:0/3842433300 >> v2:172.21.15.154:6812/254845342 conn(0x7fd184037810 0x7fd184039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:34.597 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.572+0000 7fd196ffd640 10 monclient: get_auth_request con 0x7fd184037810 auth_method 0 2024-09-16T03:42:34.597 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.572+0000 7fd17ffff640 10 client.4346.objecter ms_handle_connect 0x7fd184037810 2024-09-16T03:42:34.597 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.572+0000 7fd196ffd640 1 --2- 172.21.15.42:0/3842433300 >> v2:172.21.15.154:6812/254845342 conn(0x7fd184037810 0x7fd184039cd0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fd18c0ad000 tx=0x7fd18c0a6000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:34.598 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.576+0000 7fd17ffff640 1 -- 172.21.15.42:0/3842433300 <== mon.2 v2:172.21.15.42:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7fd188011de0 con 0x7fd1980a1330 2024-09-16T03:42:34.598 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.576+0000 7fd17ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:42:34.598 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.576+0000 7fd17ffff640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:42:34.605 INFO:tasks.workunit.client.0.smithi154.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-09-16T03:42:34.801 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.792+0000 7fd19dc8a640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:42:34.801 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.792+0000 7fd19dc8a640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:42:34.801 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.792+0000 7fd19dc8a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:42:34.801 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.792+0000 7fd19dc8a640 1 -- 172.21.15.42:0/3842433300 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fd198071920 con 0x7fd1980a1330 2024-09-16T03:42:34.802 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":28,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:42:34.802 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.792+0000 7fd17ffff640 1 -- 172.21.15.42:0/3842433300 <== mon.2 v2:172.21.15.42:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7fd188007b20 con 0x7fd1980a1330 2024-09-16T03:42:34.802 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.792+0000 7fd17ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:42:34.802 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.792+0000 7fd17ffff640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:42:34.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.796+0000 7fd19dc8a640 10 librados: watch_flush enter 2024-09-16T03:42:34.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.796+0000 7fd19dc8a640 10 librados: watch_flush exit 2024-09-16T03:42:34.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.796+0000 7fd19dc8a640 20 client.4346.objecter shutdown clearing up homeless session... 2024-09-16T03:42:34.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.796+0000 7fd19dc8a640 10 client.4346.objecter successfully canceled tick 2024-09-16T03:42:34.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.796+0000 7fd19dc8a640 1 -- 172.21.15.42:0/3842433300 >> v2:172.21.15.154:6812/254845342 conn(0x7fd184037810 msgr2=0x7fd184039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:34.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.796+0000 7fd19dc8a640 1 --2- 172.21.15.42:0/3842433300 >> v2:172.21.15.154:6812/254845342 conn(0x7fd184037810 0x7fd184039cd0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fd18c0ad000 tx=0x7fd18c0a6000 comp rx=0 tx=0).stop 2024-09-16T03:42:34.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.796+0000 7fd19dc8a640 10 monclient: shutdown 2024-09-16T03:42:34.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.796+0000 7fd19dc8a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:42:34.806 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.796+0000 7fd19dc8a640 1 -- 172.21.15.42:0/3842433300 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd1980a1330 msgr2=0x7fd198138630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:34.806 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.796+0000 7fd19dc8a640 1 --2- 172.21.15.42:0/3842433300 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd1980a1330 0x7fd198138630 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fd18800c820 tx=0x7fd18800cce0 comp rx=0 tx=0).stop 2024-09-16T03:42:34.806 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.796+0000 7fd19dc8a640 1 -- 172.21.15.42:0/3842433300 shutdown_connections 2024-09-16T03:42:34.806 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.796+0000 7fd19dc8a640 1 --2- 172.21.15.42:0/3842433300 >> v2:172.21.15.154:6812/254845342 conn(0x7fd184037810 0x7fd184039cd0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:34.806 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.796+0000 7fd19dc8a640 1 --2- 172.21.15.42:0/3842433300 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd1981340f0 0x7fd1981390b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:34.806 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.796+0000 7fd19dc8a640 1 --2- 172.21.15.42:0/3842433300 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd1980a1710 0x7fd198138b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:34.806 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.796+0000 7fd19dc8a640 1 --2- 172.21.15.42:0/3842433300 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd1980a1330 0x7fd198138630 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:34.806 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.796+0000 7fd19dc8a640 1 -- 172.21.15.42:0/3842433300 >> 172.21.15.42:0/3842433300 conn(0x7fd1980a2420 msgr2=0x7fd19813b5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:42:34.806 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.796+0000 7fd19dc8a640 1 -- 172.21.15.42:0/3842433300 shutdown_connections 2024-09-16T03:42:34.806 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.796+0000 7fd19dc8a640 1 -- 172.21.15.42:0/3842433300 wait complete. 2024-09-16T03:42:34.806 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.796+0000 7fd19dc8a640 1 librados: shutdown 2024-09-16T03:42:34.819 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T03:42:34.819 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T03:42:34.819 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T03:42:34.820 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T03:42:34.943 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.936+0000 7faf12641640 10 monclient: get_monmap_and_config 2024-09-16T03:42:34.944 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.936+0000 7faf12641640 10 monclient: build_initial_monmap 2024-09-16T03:42:34.944 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.936+0000 7faf12641640 10 monclient: monmap: 2024-09-16T03:42:34.944 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:42:34.944 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:34.944 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:42:34.941194+0000 2024-09-16T03:42:34.944 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:42:34.941194+0000 2024-09-16T03:42:34.944 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:42:34.944 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:34.944 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:42:34.944 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:42:34.944 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:42:34.944 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:42:34.944 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:42:34.945 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:34.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.936+0000 7faf12641640 1 Processor -- start 2024-09-16T03:42:34.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.936+0000 7faf12641640 1 -- start start 2024-09-16T03:42:34.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.936+0000 7faf12641640 10 monclient: init 2024-09-16T03:42:34.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.936+0000 7faf12641640 10 monclient: _reopen_session rank -1 2024-09-16T03:42:34.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.936+0000 7faf12641640 10 monclient: _add_conns ranks=[4,2,1,3,0] 2024-09-16T03:42:34.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.936+0000 7faf12641640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7faf0c0a39b0 0x7faf0c133b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:34.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.936+0000 7faf12641640 10 monclient(hunting): picked mon.noname-c con 0x7faf0c0a39b0 addr v2:172.21.15.154:3300/0 2024-09-16T03:42:34.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.936+0000 7faf12641640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7faf0c0a3d90 0x7faf0c137000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:34.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.936+0000 7faf12641640 10 monclient(hunting): picked mon.noname-b con 0x7faf0c0a3d90 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:42:34.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.936+0000 7faf12641640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7faf0c137540 addr v1:172.21.15.154:6789/0 2024-09-16T03:42:34.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.936+0000 7faf12641640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faf0c09c140 con 0x7faf0c137540 2024-09-16T03:42:34.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.936+0000 7faf12641640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:34.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.936+0000 7faf12641640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7faf0c09c420 con 0x7faf0c0a39b0 2024-09-16T03:42:34.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.936+0000 7faf12641640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:34.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.936+0000 7faf12641640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7faf0c06dd60 con 0x7faf0c0a3d90 2024-09-16T03:42:34.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.936+0000 7faf12641640 10 monclient(hunting): _renew_subs 2024-09-16T03:42:34.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.936+0000 7faf12641640 10 monclient(hunting): authenticate will time out at 1001.911308s 2024-09-16T03:42:34.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.936+0000 7faf0b7fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7faf0c0a3d90 0x7faf0c137000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:34.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.936+0000 7faf0b7fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7faf0c0a3d90 0x7faf0c137000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:54032/0 (socket says 172.21.15.42:54032) 2024-09-16T03:42:34.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.936+0000 7faf0b7fe640 1 -- 172.21.15.42:0/119347701 learned_addr learned my addr 172.21.15.42:0/119347701 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:42:34.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.936+0000 7faf0b7fe640 10 monclient(hunting): get_auth_request con 0x7faf0c0a3d90 auth_method 0 2024-09-16T03:42:34.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.936+0000 7faf0b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:34.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.936+0000 7faf0b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:34.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.936+0000 7faf0b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:34.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.936+0000 7faf0b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:34.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.936+0000 7faf0b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:34.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.936+0000 7faf0b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:34.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.936+0000 7faf0bfff640 1 --2- 172.21.15.42:0/119347701 >> v2:172.21.15.154:3300/0 conn(0x7faf0c0a39b0 0x7faf0c133b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:34.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.936+0000 7faf0bfff640 10 monclient(hunting): get_auth_request con 0x7faf0c0a39b0 auth_method 0 2024-09-16T03:42:34.952 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.936+0000 7faf0bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:34.952 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.936+0000 7faf0bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:34.952 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.936+0000 7faf0bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:34.952 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.936+0000 7faf0b7fe640 10 monclient(hunting): handle_auth_done global_id 4349 payload 274 2024-09-16T03:42:34.952 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.936+0000 7faf0b7fe640 1 -- 172.21.15.42:0/119347701 >> v2:172.21.15.154:3300/0 conn(0x7faf0c0a39b0 msgr2=0x7faf0c133b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:42:34.952 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.936+0000 7faf0b7fe640 1 --2- 172.21.15.42:0/119347701 >> v2:172.21.15.154:3300/0 conn(0x7faf0c0a39b0 0x7faf0c133b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:34.952 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.936+0000 7faf0b7fe640 1 -- 172.21.15.42:0/119347701 >> v1:172.21.15.154:6789/0 conn(0x7faf0c137540 legacy=0x7faf0c13e390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:42:34.952 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.936+0000 7faf0b7fe640 10 monclient: _finish_hunting 0 2024-09-16T03:42:34.952 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.936+0000 7faf0b7fe640 1 monclient: found mon.noname-b 2024-09-16T03:42:34.952 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.936+0000 7faf0b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:42:34.953 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.936+0000 7faf0b7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T03:42:34.953 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.936+0000 7faf0b7fe640 1 -- 172.21.15.42:0/119347701 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf0c0a5e20 con 0x7faf0c0a3d90 2024-09-16T03:42:34.953 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.940+0000 7faf0b7fe640 1 --2- 172.21.15.42:0/119347701 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7faf0c0a3d90 0x7faf0c137000 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7faf00009870 tx=0x7faf0001c3e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=37b31275bc36748b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:34.953 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.940+0000 7faf0affd640 1 -- 172.21.15.42:0/119347701 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf000b0070 con 0x7faf0c0a3d90 2024-09-16T03:42:34.953 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.940+0000 7faf0affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:34.953 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.940+0000 7faf0affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:42:34.953 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.940+0000 7faf0affd640 10 monclient: dump: 2024-09-16T03:42:34.953 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:34.954 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:34.954 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:34.954 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:34.954 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:34.954 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:34.954 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:34.954 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:34.954 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:34.954 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:34.954 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.940+0000 7faf0affd640 10 monclient: _finish_auth 0 2024-09-16T03:42:34.955 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.940+0000 7faf0affd640 10 monclient: _check_auth_tickets 2024-09-16T03:42:34.955 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.940+0000 7faf0affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:42:34.955 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.940+0000 7faf0affd640 1 -- 172.21.15.42:0/119347701 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faf000ab600 con 0x7faf0c0a3d90 2024-09-16T03:42:34.955 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.940+0000 7faf12641640 5 monclient: authenticate success, global_id 4349 2024-09-16T03:42:34.955 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.940+0000 7faf12641640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:42:34.955 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.940+0000 7faf0affd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:42:34.955 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.940+0000 7faf12641640 10 monclient: get_monmap_and_config success 2024-09-16T03:42:34.956 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.940+0000 7faf0affd640 1 -- 172.21.15.42:0/119347701 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf000ab9d0 con 0x7faf0c0a3d90 2024-09-16T03:42:34.956 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.940+0000 7faf12641640 4 set_mon_vals no callback set 2024-09-16T03:42:34.956 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.940+0000 7faf0affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:34.956 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.940+0000 7faf0affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:42:34.956 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.940+0000 7faf0affd640 10 monclient: dump: 2024-09-16T03:42:34.956 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:34.956 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:34.956 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:34.956 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:34.956 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:34.956 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:34.956 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:34.956 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:34.956 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:34.957 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:34.957 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.940+0000 7faf12641640 10 monclient: shutdown 2024-09-16T03:42:34.957 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.940+0000 7faf12641640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:42:34.957 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.940+0000 7faf12641640 1 -- 172.21.15.42:0/119347701 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7faf0c0a3d90 msgr2=0x7faf0c137000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:34.957 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.940+0000 7faf12641640 1 --2- 172.21.15.42:0/119347701 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7faf0c0a3d90 0x7faf0c137000 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7faf00009870 tx=0x7faf0001c3e0 comp rx=0 tx=0).stop 2024-09-16T03:42:34.957 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.940+0000 7faf12641640 1 -- 172.21.15.42:0/119347701 shutdown_connections 2024-09-16T03:42:34.957 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.940+0000 7faf12641640 1 --2- 172.21.15.42:0/119347701 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7faf0c0a3d90 0x7faf0c137000 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:34.957 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.940+0000 7faf12641640 1 --2- 172.21.15.42:0/119347701 >> v2:172.21.15.154:3300/0 conn(0x7faf0c0a39b0 0x7faf0c133b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:34.957 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.940+0000 7faf12641640 1 -- 172.21.15.42:0/119347701 >> 172.21.15.42:0/119347701 conn(0x7faf0c0a9fb0 msgr2=0x7faf0c0aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:42:34.958 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.940+0000 7faf12641640 1 -- 172.21.15.42:0/119347701 shutdown_connections 2024-09-16T03:42:34.958 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.940+0000 7faf12641640 1 -- 172.21.15.42:0/119347701 wait complete. 2024-09-16T03:42:34.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.940+0000 7faf12641640 10 monclient: build_initial_monmap 2024-09-16T03:42:34.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.940+0000 7faf12641640 10 monclient: monmap: 2024-09-16T03:42:34.965 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:42:34.965 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:42:34.965 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:42:34.965 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:42:34.966 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:42:34.966 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:34.966 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:42:34.966 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:42:34.966 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:42:34.966 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:34.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.940+0000 7faf12641640 1 librados: starting msgr at 2024-09-16T03:42:34.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.940+0000 7faf12641640 1 librados: starting objecter 2024-09-16T03:42:34.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.940+0000 7faf12641640 1 Processor -- start 2024-09-16T03:42:34.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.940+0000 7faf12641640 1 -- start start 2024-09-16T03:42:34.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.940+0000 7faf12641640 1 librados: setting wanted keys 2024-09-16T03:42:34.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.940+0000 7faf12641640 1 librados: calling monclient init 2024-09-16T03:42:34.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.940+0000 7faf12641640 10 monclient: init 2024-09-16T03:42:34.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.944+0000 7faf12641640 10 monclient: _reopen_session rank -1 2024-09-16T03:42:34.967 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.944+0000 7faf12641640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:42:34.967 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.944+0000 7faf12641640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf0c0a39b0 0x7faf0c0a19d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:34.967 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.944+0000 7faf12641640 10 monclient(hunting): picked mon.noname-a con 0x7faf0c0a39b0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:42:34.967 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.944+0000 7faf12641640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7faf0c0a3d90 0x7faf0c0a1f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:34.967 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.944+0000 7faf12641640 10 monclient(hunting): picked mon.noname-c con 0x7faf0c0a3d90 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:42:34.967 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.944+0000 7faf12641640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faf0c137540 0x7faf0c0a2450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:34.967 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.944+0000 7faf12641640 10 monclient(hunting): picked mon.noname-b con 0x7faf0c137540 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:42:34.967 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.944+0000 7faf12641640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:34.967 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.944+0000 7faf12641640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7faf0c1424e0 con 0x7faf0c0a39b0 2024-09-16T03:42:34.967 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.944+0000 7faf12641640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:34.967 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.944+0000 7faf12641640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7faf0c142660 con 0x7faf0c137540 2024-09-16T03:42:34.967 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.944+0000 7faf12641640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:34.967 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.944+0000 7faf12641640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7faf0c1427e0 con 0x7faf0c0a3d90 2024-09-16T03:42:34.968 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.944+0000 7faf12641640 10 monclient(hunting): _renew_subs 2024-09-16T03:42:34.968 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.944+0000 7faf12641640 10 monclient(hunting): authenticate will time out at 1001.916324s 2024-09-16T03:42:34.968 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.944+0000 7faf0b7fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7faf0c0a3d90 0x7faf0c0a1f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:34.968 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.944+0000 7faf0bfff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf0c0a39b0 0x7faf0c0a19d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:34.968 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.944+0000 7faf0b7fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7faf0c0a3d90 0x7faf0c0a1f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:54044/0 (socket says 172.21.15.42:54044) 2024-09-16T03:42:34.968 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.944+0000 7faf0bfff640 10 monclient(hunting): get_auth_request con 0x7faf0c0a39b0 auth_method 0 2024-09-16T03:42:34.968 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.944+0000 7faf0b7fe640 1 -- 172.21.15.42:0/669448326 learned_addr learned my addr 172.21.15.42:0/669448326 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:42:34.968 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.944+0000 7faf0bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:34.968 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.944+0000 7faf0bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:34.968 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.944+0000 7faf0bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:34.968 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.944+0000 7faf0b7fe640 10 monclient(hunting): get_auth_request con 0x7faf0c0a3d90 auth_method 0 2024-09-16T03:42:34.968 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.944+0000 7faf0b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:34.969 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.944+0000 7faf0b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:34.969 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.944+0000 7faf0b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:34.969 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.944+0000 7faf0b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:34.969 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.944+0000 7faf0b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:34.969 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.944+0000 7faf0b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:34.969 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.944+0000 7faf0bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:34.969 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.944+0000 7faf0bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:34.969 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.944+0000 7faf0bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:34.969 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.944+0000 7faf10bde640 1 --2- 172.21.15.42:0/669448326 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faf0c137540 0x7faf0c0a2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:34.969 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.944+0000 7faf10bde640 10 monclient(hunting): get_auth_request con 0x7faf0c137540 auth_method 0 2024-09-16T03:42:34.969 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.944+0000 7faf10bde640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:34.969 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.944+0000 7faf10bde640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:34.969 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.944+0000 7faf10bde640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:34.969 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.944+0000 7faf0b7fe640 10 monclient(hunting): handle_auth_done global_id 4352 payload 824 2024-09-16T03:42:34.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.944+0000 7faf0b7fe640 1 -- 172.21.15.42:0/669448326 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faf0c137540 msgr2=0x7faf0c0a2450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:34.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.944+0000 7faf0b7fe640 1 --2- 172.21.15.42:0/669448326 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faf0c137540 0x7faf0c0a2450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:34.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.944+0000 7faf0b7fe640 1 -- 172.21.15.42:0/669448326 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf0c0a39b0 msgr2=0x7faf0c0a19d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:34.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.944+0000 7faf0b7fe640 1 --2- 172.21.15.42:0/669448326 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf0c0a39b0 0x7faf0c0a19d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:34.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.944+0000 7faf0b7fe640 10 monclient: _finish_hunting 0 2024-09-16T03:42:34.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.944+0000 7faf0b7fe640 1 monclient: found mon.noname-c 2024-09-16T03:42:34.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.944+0000 7faf0b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:42:34.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.944+0000 7faf0b7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:42:34.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.944+0000 7faf0b7fe640 1 -- 172.21.15.42:0/669448326 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf0c139e30 con 0x7faf0c0a3d90 2024-09-16T03:42:34.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.944+0000 7faf0bfff640 1 --2- 172.21.15.42:0/669448326 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf0c0a39b0 0x7faf0c0a19d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:42:34.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.944+0000 7faf0b7fe640 1 --2- 172.21.15.42:0/669448326 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7faf0c0a3d90 0x7faf0c0a1f10 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7faf000ae040 tx=0x7faf00019180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:34.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.944+0000 7faf08ff9640 10 client.?.objecter ms_handle_connect 0x7faf0c0a3d90 2024-09-16T03:42:34.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.944+0000 7faf08ff9640 10 client.?.objecter resend_mon_ops 2024-09-16T03:42:34.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.944+0000 7faf08ff9640 1 -- 172.21.15.42:0/669448326 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf000b0040 con 0x7faf0c0a3d90 2024-09-16T03:42:34.971 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.944+0000 7faf08ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:34.971 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.944+0000 7faf08ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:42:34.971 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.944+0000 7faf08ff9640 10 monclient: dump: 2024-09-16T03:42:34.971 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:34.971 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:34.971 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:34.971 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:34.971 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:34.971 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:34.971 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:34.971 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:34.971 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:34.971 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:34.971 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.944+0000 7faf08ff9640 10 monclient: _finish_auth 0 2024-09-16T03:42:34.972 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.944+0000 7faf08ff9640 10 monclient: _check_auth_tickets 2024-09-16T03:42:34.972 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.944+0000 7faf08ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:42:34.972 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.944+0000 7faf08ff9640 1 -- 172.21.15.42:0/669448326 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faf000b9070 con 0x7faf0c0a3d90 2024-09-16T03:42:34.972 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.944+0000 7faf08ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T03:42:34.972 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.944+0000 7faf08ff9640 1 -- 172.21.15.42:0/669448326 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf000b4680 con 0x7faf0c0a3d90 2024-09-16T03:42:34.972 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.944+0000 7faf08ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:34.972 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.944+0000 7faf08ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:42:34.972 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.944+0000 7faf08ff9640 10 monclient: dump: 2024-09-16T03:42:34.972 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:34.972 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:34.972 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:34.972 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:34.972 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:34.973 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:34.973 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:34.973 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:34.973 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:34.973 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:34.973 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.944+0000 7faf12641640 5 monclient: authenticate success, global_id 4352 2024-09-16T03:42:34.973 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.944+0000 7faf12641640 10 monclient: _renew_subs 2024-09-16T03:42:34.973 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.944+0000 7faf0a7fc640 4 set_mon_vals no callback set 2024-09-16T03:42:34.973 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.944+0000 7faf12641640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:42:34.973 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.944+0000 7faf12641640 1 -- 172.21.15.42:0/669448326 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf0c1332c0 con 0x7faf0c0a3d90 2024-09-16T03:42:34.973 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.944+0000 7faf12641640 10 client.4352.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:42:34.973 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.944+0000 7faf12641640 10 monclient: _renew_subs 2024-09-16T03:42:34.973 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.944+0000 7faf12641640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:42:34.974 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.944+0000 7faf12641640 1 -- 172.21.15.42:0/669448326 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7faf0c06f680 con 0x7faf0c0a3d90 2024-09-16T03:42:34.974 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.944+0000 7faf12641640 1 librados: init done 2024-09-16T03:42:34.974 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.944+0000 7faf12641640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:42:34.974 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.944+0000 7faf12641640 1 --2- 172.21.15.42:0/669448326 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf0c139870 0x7faf0c1d0140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:34.974 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.944+0000 7faf12641640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:34.974 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.944+0000 7faf12641640 1 -- 172.21.15.42:0/669448326 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7faf0c1342d0 con 0x7faf0c139870 2024-09-16T03:42:34.974 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.944+0000 7faf0bfff640 1 --2- 172.21.15.42:0/669448326 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf0c139870 0x7faf0c1d0140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:34.974 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.944+0000 7faf0bfff640 10 monclient: get_auth_request con 0x7faf0c139870 auth_method 0 2024-09-16T03:42:34.974 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.944+0000 7faf0bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:34.974 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.944+0000 7faf0bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:34.974 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.944+0000 7faf0bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:34.974 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.944+0000 7faf08ff9640 1 -- 172.21.15.42:0/669448326 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7faf000b4820 con 0x7faf0c0a3d90 2024-09-16T03:42:34.974 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.944+0000 7faf0bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:34.975 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.944+0000 7faf0bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:34.975 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.944+0000 7faf0bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:34.975 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.944+0000 7faf0bfff640 10 monclient(hunting): handle_auth_done global_id 4416 payload 824 2024-09-16T03:42:34.975 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.944+0000 7faf08ff9640 1 --2- 172.21.15.42:0/669448326 >> v2:172.21.15.154:6812/254845342 conn(0x7faef8037810 0x7faef8039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:34.975 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.944+0000 7faf0bfff640 1 -- 172.21.15.42:0/669448326 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7faf0c139c50 con 0x7faf0c139870 2024-09-16T03:42:34.975 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.944+0000 7faf08ff9640 1 -- 172.21.15.42:0/669448326 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(19..19 src has 1..19) ==== 3534+0+0 (secure 0 0 0) 0x7faf000e23a0 con 0x7faf0c0a3d90 2024-09-16T03:42:34.975 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.944+0000 7faf08ff9640 10 client.4352.objecter ms_dispatch 0x7faf0c000ba0 osd_map(19..19 src has 1..19) 2024-09-16T03:42:34.975 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.944+0000 7faf08ff9640 3 client.4352.objecter handle_osd_map got epochs [19,19] > 0 2024-09-16T03:42:34.975 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.944+0000 7faf08ff9640 3 client.4352.objecter handle_osd_map decoding full epoch 19 2024-09-16T03:42:34.975 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.944+0000 7faf08ff9640 20 client.4352.objecter dump_active .. 0 homeless 2024-09-16T03:42:34.975 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.944+0000 7faf08ff9640 10 client.4352.objecter ms_handle_connect 0x7faf0c139870 2024-09-16T03:42:34.975 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.944+0000 7faf08ff9640 10 client.4352.objecter resend_mon_ops 2024-09-16T03:42:34.975 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.944+0000 7faf0bfff640 1 --2- 172.21.15.42:0/669448326 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf0c139870 0x7faf0c1d0140 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7faefc007590 tx=0x7faefc007a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:34.975 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.944+0000 7faf08ff9640 1 -- 172.21.15.42:0/669448326 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faefc014590 con 0x7faf0c139870 2024-09-16T03:42:34.976 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.944+0000 7faf08ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:34.976 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.944+0000 7faf08ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:42:34.976 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.944+0000 7faf08ff9640 10 monclient: dump: 2024-09-16T03:42:34.976 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:34.976 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:34.976 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:34.976 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:34.976 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:34.976 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:34.976 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:34.976 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:34.976 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:34.976 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:34.976 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.944+0000 7faf10bde640 1 --2- 172.21.15.42:0/669448326 >> v2:172.21.15.154:6812/254845342 conn(0x7faef8037810 0x7faef8039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:34.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.944+0000 7faf10bde640 10 monclient: get_auth_request con 0x7faef8037810 auth_method 0 2024-09-16T03:42:34.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.944+0000 7faf08ff9640 1 -- 172.21.15.42:0/669448326 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7faefc01a070 con 0x7faf0c139870 2024-09-16T03:42:34.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.944+0000 7faf08ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:42:34.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.944+0000 7faf08ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:42:34.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.944+0000 7faf08ff9640 1 -- 172.21.15.42:0/669448326 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf0c139870 msgr2=0x7faf0c1d0140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:34.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.944+0000 7faf08ff9640 1 --2- 172.21.15.42:0/669448326 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf0c139870 0x7faf0c1d0140 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7faefc007590 tx=0x7faefc007a50 comp rx=0 tx=0).stop 2024-09-16T03:42:34.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.944+0000 7faf08ff9640 1 -- 172.21.15.42:0/669448326 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf0c139870 msgr2=0x7faf0c1d0140 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:42:34.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.944+0000 7faf08ff9640 1 --2- 172.21.15.42:0/669448326 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf0c139870 0x7faf0c1d0140 secure :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7faefc007590 tx=0x7faefc007a50 comp rx=0 tx=0).stop 2024-09-16T03:42:34.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.944+0000 7faf08ff9640 10 client.4352.objecter ms_handle_connect 0x7faef8037810 2024-09-16T03:42:34.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.944+0000 7faf10bde640 1 --2- 172.21.15.42:0/669448326 >> v2:172.21.15.154:6812/254845342 conn(0x7faef8037810 0x7faef8039cd0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7faf0c131950 tx=0x7faf04009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:34.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.952+0000 7faf12641640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:42:34.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.952+0000 7faf12641640 1 --2- 172.21.15.42:0/669448326 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf0c1d0680 0x7faf0c02c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:34.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.952+0000 7faf12641640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:34.978 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.952+0000 7faf12641640 1 -- 172.21.15.42:0/669448326 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7faf0c133b80 con 0x7faf0c1d0680 2024-09-16T03:42:34.978 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.952+0000 7faf0bfff640 1 --2- 172.21.15.42:0/669448326 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf0c1d0680 0x7faf0c02c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:34.978 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.952+0000 7faf0bfff640 10 monclient: get_auth_request con 0x7faf0c1d0680 auth_method 0 2024-09-16T03:42:34.978 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.952+0000 7faf0bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:34.978 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.952+0000 7faf0bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:34.978 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.952+0000 7faf0bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:34.978 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.952+0000 7faf0bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:34.978 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.952+0000 7faf0bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:34.978 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.952+0000 7faf0bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:34.978 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.952+0000 7faf0bfff640 10 monclient(hunting): handle_auth_done global_id 4419 payload 824 2024-09-16T03:42:34.979 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.952+0000 7faf0bfff640 1 -- 172.21.15.42:0/669448326 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7faf0c0a5750 con 0x7faf0c1d0680 2024-09-16T03:42:34.979 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.952+0000 7faf0bfff640 1 --2- 172.21.15.42:0/669448326 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf0c1d0680 0x7faf0c02c650 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7faefc007fd0 tx=0x7faefc004a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:34.979 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.952+0000 7faf08ff9640 10 client.4352.objecter ms_handle_connect 0x7faf0c1d0680 2024-09-16T03:42:34.979 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.952+0000 7faf08ff9640 10 client.4352.objecter resend_mon_ops 2024-09-16T03:42:34.979 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.952+0000 7faf08ff9640 1 -- 172.21.15.42:0/669448326 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faefc014960 con 0x7faf0c1d0680 2024-09-16T03:42:34.979 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.952+0000 7faf08ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:34.979 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.952+0000 7faf08ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:42:34.979 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.952+0000 7faf08ff9640 10 monclient: dump: 2024-09-16T03:42:34.979 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:34.979 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:34.979 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:34.979 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:34.979 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:34.980 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:34.980 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:34.980 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:34.980 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:34.980 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:34.980 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.952+0000 7faf08ff9640 1 -- 172.21.15.42:0/669448326 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4713 (secure 0 0 0) 0x7faefc014960 con 0x7faf0c1d0680 2024-09-16T03:42:34.980 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.952+0000 7faf08ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:42:34.980 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.952+0000 7faf08ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:42:34.980 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.952+0000 7faf08ff9640 1 -- 172.21.15.42:0/669448326 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf0c1d0680 msgr2=0x7faf0c02c650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:34.980 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.952+0000 7faf08ff9640 1 --2- 172.21.15.42:0/669448326 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf0c1d0680 0x7faf0c02c650 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7faefc007fd0 tx=0x7faefc004a50 comp rx=0 tx=0).stop 2024-09-16T03:42:34.980 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.952+0000 7faf08ff9640 1 -- 172.21.15.42:0/669448326 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf0c1d0680 msgr2=0x7faf0c02c650 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:42:34.980 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.952+0000 7faf08ff9640 1 --2- 172.21.15.42:0/669448326 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf0c1d0680 0x7faf0c02c650 secure :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7faefc007fd0 tx=0x7faefc004a50 comp rx=0 tx=0).stop 2024-09-16T03:42:34.981 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:42:34.981 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:42:34.981 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:42:34.981 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "leader", 2024-09-16T03:42:34.981 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 8, 2024-09-16T03:42:34.981 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 34323, 2024-09-16T03:42:34.981 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:42:34.981 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:42:34.981 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T03:42:34.981 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T03:42:34.981 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:42:34.981 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 29, 2024-09-16T03:42:34.982 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:42:34.982 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:42:34.982 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:42:34.982 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:42:34.982 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:42:34.982 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:42:34.982 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:42:34.982 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:42:34.982 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:42:34.982 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:42:34.982 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:42:34.982 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:42:34.982 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:42:34.982 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:42:34.983 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:42:34.983 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:42:34.983 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:42:34.983 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:42:34.983 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:42:34.983 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:42:34.983 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:42:34.983 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:42:34.983 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:42:34.983 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:42:34.983 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:42:34.983 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:42:34.983 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:42:34.983 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:42:34.984 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:42:34.984 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:42:34.984 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:42:34.984 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:42:34.984 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:42:34.984 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:42:34.984 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:42:34.984 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:42:34.984 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:42:34.984 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:42:34.985 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:42:34.985 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:42:34.985 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:42:34.985 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:42:34.985 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:42:34.985 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:42:34.985 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:42:34.985 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:42:34.985 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:42:34.985 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:42:34.985 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:42:34.985 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:42:34.986 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:42:34.986 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:42:34.986 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:42:34.986 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:42:34.986 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:42:34.986 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:42:34.986 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:42:34.986 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:42:34.986 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:42:34.986 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:42:34.986 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:42:34.986 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:42:34.986 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:42:34.987 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:42:34.987 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:42:34.987 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:42:34.987 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:42:34.987 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:42:34.987 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:42:34.987 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:42:34.987 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:42:34.987 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:42:34.987 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:42:34.988 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:42:34.988 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:42:34.988 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:42:34.988 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:42:34.988 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:42:34.988 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:42:34.988 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:42:34.988 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:42:34.988 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:42:34.988 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:42:34.988 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:42:34.988 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:42:34.988 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:42:34.989 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:42:34.989 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:42:34.989 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:42:34.989 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:42:34.989 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:42:34.989 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:42:34.989 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:42:34.989 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:42:34.989 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:42:34.989 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:42:34.989 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:42:34.990 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:42:34.990 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:42:34.990 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:42:34.990 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:42:34.990 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:42:34.990 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:42:34.990 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:42:34.990 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:42:34.990 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:42:34.990 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:42:34.990 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:42:34.990 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:42:34.991 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:42:34.991 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:42:34.991 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:42:34.991 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:42:34.991 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:42:34.991 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:42:34.991 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:42:34.991 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:42:34.991 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:42:34.991 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:42:34.991 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:42:34.992 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:42:34.992 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:42:34.992 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:42:34.992 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:42:34.992 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:42:34.992 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:42:34.992 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:42:34.992 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:42:34.992 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:42:34.992 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:42:34.993 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:42:34.993 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:42:34.993 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:42:34.993 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:42:34.993 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:42:34.993 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:42:34.993 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:42:34.993 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:42:34.994 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:42:34.994 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:42:34.994 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:42:34.994 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T03:42:34.994 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:42:34.994 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:42:34.994 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:42:34.994 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 2 2024-09-16T03:42:34.994 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:42:34.995 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:42:34.995 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:42:34.995 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:42:34.995 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:42:34.995 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:42:34.995 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:42:34.995 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:42:34.995 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:42:34.995 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:42:34.995 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:42:34.996 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:42:34.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.952+0000 7faf12641640 10 librados: watch_flush enter 2024-09-16T03:42:34.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.952+0000 7faf12641640 10 librados: watch_flush exit 2024-09-16T03:42:34.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.952+0000 7faf12641640 20 client.4352.objecter shutdown clearing up homeless session... 2024-09-16T03:42:34.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.952+0000 7faf12641640 10 client.4352.objecter successfully canceled tick 2024-09-16T03:42:34.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.952+0000 7faf12641640 1 -- 172.21.15.42:0/669448326 >> v2:172.21.15.154:6812/254845342 conn(0x7faef8037810 msgr2=0x7faef8039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:34.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.952+0000 7faf12641640 1 --2- 172.21.15.42:0/669448326 >> v2:172.21.15.154:6812/254845342 conn(0x7faef8037810 0x7faef8039cd0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7faf0c131950 tx=0x7faf04009040 comp rx=0 tx=0).stop 2024-09-16T03:42:34.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.952+0000 7faf12641640 10 monclient: shutdown 2024-09-16T03:42:34.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.952+0000 7faf12641640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:42:34.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.952+0000 7faf12641640 1 -- 172.21.15.42:0/669448326 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7faf0c0a3d90 msgr2=0x7faf0c0a1f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:34.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.952+0000 7faf12641640 1 --2- 172.21.15.42:0/669448326 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7faf0c0a3d90 0x7faf0c0a1f10 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7faf000ae040 tx=0x7faf00019180 comp rx=0 tx=0).stop 2024-09-16T03:42:34.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.952+0000 7faf10bde640 1 -- 172.21.15.42:0/669448326 reap_dead start 2024-09-16T03:42:34.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.952+0000 7faf10bde640 1 -- 172.21.15.42:0/669448326 reap_dead start 2024-09-16T03:42:34.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.952+0000 7faf12641640 1 -- 172.21.15.42:0/669448326 shutdown_connections 2024-09-16T03:42:34.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.952+0000 7faf12641640 1 -- 172.21.15.42:0/669448326 >> 172.21.15.42:0/669448326 conn(0x7faf0c0a9fb0 msgr2=0x7faf0c1321a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:42:34.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.952+0000 7faf12641640 1 -- 172.21.15.42:0/669448326 shutdown_connections 2024-09-16T03:42:34.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.952+0000 7faf12641640 1 -- 172.21.15.42:0/669448326 wait complete. 2024-09-16T03:42:34.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:34.952+0000 7faf12641640 1 librados: shutdown 2024-09-16T03:42:34.999 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:42:34.999 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T03:42:34.999 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T03:42:35.111 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.104+0000 7f4d70ecd640 10 monclient: get_monmap_and_config 2024-09-16T03:42:35.111 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.104+0000 7f4d70ecd640 10 monclient: build_initial_monmap 2024-09-16T03:42:35.111 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.104+0000 7f4d70ecd640 10 monclient: monmap: 2024-09-16T03:42:35.111 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:42:35.111 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:35.111 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:42:35.108987+0000 2024-09-16T03:42:35.111 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:42:35.108987+0000 2024-09-16T03:42:35.111 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:42:35.112 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:35.112 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:42:35.112 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:42:35.112 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:42:35.112 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:42:35.112 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:42:35.112 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:35.115 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.104+0000 7f4d70ecd640 1 Processor -- start 2024-09-16T03:42:35.115 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.104+0000 7f4d70ecd640 1 -- start start 2024-09-16T03:42:35.115 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.104+0000 7f4d70ecd640 10 monclient: init 2024-09-16T03:42:35.115 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.104+0000 7f4d70ecd640 10 monclient: _reopen_session rank -1 2024-09-16T03:42:35.115 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.104+0000 7f4d70ecd640 10 monclient: _add_conns ranks=[4,2,1,3,0] 2024-09-16T03:42:35.115 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.104+0000 7f4d70ecd640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f4d6c0a39b0 0x7f4d6c133b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:35.115 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.104+0000 7f4d70ecd640 10 monclient(hunting): picked mon.noname-c con 0x7f4d6c0a39b0 addr v2:172.21.15.154:3300/0 2024-09-16T03:42:35.115 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.104+0000 7f4d70ecd640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4d6c0a3d90 0x7f4d6c137000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:35.115 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.104+0000 7f4d70ecd640 10 monclient(hunting): picked mon.noname-b con 0x7f4d6c0a3d90 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:42:35.116 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.104+0000 7f4d70ecd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4d6c137540 addr v1:172.21.15.154:6789/0 2024-09-16T03:42:35.116 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.104+0000 7f4d70ecd640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4d6c09c140 con 0x7f4d6c137540 2024-09-16T03:42:35.116 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.104+0000 7f4d70ecd640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:35.116 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.104+0000 7f4d70ecd640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f4d6c09c420 con 0x7f4d6c0a39b0 2024-09-16T03:42:35.116 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.104+0000 7f4d70ecd640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:35.116 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.104+0000 7f4d70ecd640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f4d6c06dd60 con 0x7f4d6c0a3d90 2024-09-16T03:42:35.116 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.104+0000 7f4d70ecd640 10 monclient(hunting): _renew_subs 2024-09-16T03:42:35.116 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.104+0000 7f4d70ecd640 10 monclient(hunting): authenticate will time out at 1002.079067s 2024-09-16T03:42:35.116 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.104+0000 7f4d61d9b640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4d6c0a3d90 0x7f4d6c137000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:35.116 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.104+0000 7f4d61d9b640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4d6c0a3d90 0x7f4d6c137000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:54050/0 (socket says 172.21.15.42:54050) 2024-09-16T03:42:35.116 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.104+0000 7f4d61d9b640 1 -- 172.21.15.42:0/1489808803 learned_addr learned my addr 172.21.15.42:0/1489808803 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:42:35.116 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.104+0000 7f4d61d9b640 10 monclient(hunting): get_auth_request con 0x7f4d6c0a3d90 auth_method 0 2024-09-16T03:42:35.117 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.104+0000 7f4d61d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:35.117 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.104+0000 7f4d61d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:35.117 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.104+0000 7f4d61d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:35.117 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.104+0000 7f4d61d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:35.117 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.104+0000 7f4d61d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:35.117 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.104+0000 7f4d61d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:35.117 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.104+0000 7f4d61d9b640 10 monclient(hunting): handle_auth_done global_id 4355 payload 274 2024-09-16T03:42:35.117 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.104+0000 7f4d61d9b640 1 -- 172.21.15.42:0/1489808803 >> v2:172.21.15.154:3300/0 conn(0x7f4d6c0a39b0 msgr2=0x7f4d6c133b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:42:35.117 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.104+0000 7f4d61d9b640 1 --2- 172.21.15.42:0/1489808803 >> v2:172.21.15.154:3300/0 conn(0x7f4d6c0a39b0 0x7f4d6c133b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:35.117 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.104+0000 7f4d61d9b640 1 -- 172.21.15.42:0/1489808803 >> v1:172.21.15.154:6789/0 conn(0x7f4d6c137540 legacy=0x7f4d6c13e390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:42:35.117 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.104+0000 7f4d61d9b640 10 monclient: _finish_hunting 0 2024-09-16T03:42:35.117 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.104+0000 7f4d61d9b640 1 monclient: found mon.noname-b 2024-09-16T03:42:35.117 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.104+0000 7f4d61d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:42:35.118 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.104+0000 7f4d61d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T03:42:35.118 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.104+0000 7f4d61d9b640 1 -- 172.21.15.42:0/1489808803 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d6c0a5e20 con 0x7f4d6c0a3d90 2024-09-16T03:42:35.125 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.108+0000 7f4d61d9b640 1 --2- 172.21.15.42:0/1489808803 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4d6c0a3d90 0x7f4d6c137000 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f4d54009870 tx=0x7f4d5401c3e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=396b280f6e7a412d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:35.125 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.108+0000 7f4d69d9b640 1 -- 172.21.15.42:0/1489808803 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d540b0070 con 0x7f4d6c0a3d90 2024-09-16T03:42:35.125 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.108+0000 7f4d69d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:35.126 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.108+0000 7f4d69d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:42:35.126 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.108+0000 7f4d69d9b640 10 monclient: dump: 2024-09-16T03:42:35.126 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:35.126 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:35.126 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:35.126 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:35.126 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:35.127 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:35.127 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:35.127 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:35.127 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:35.127 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:35.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.108+0000 7f4d69d9b640 10 monclient: _finish_auth 0 2024-09-16T03:42:35.128 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.108+0000 7f4d69d9b640 10 monclient: _check_auth_tickets 2024-09-16T03:42:35.128 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.108+0000 7f4d69d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:42:35.128 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.108+0000 7f4d69d9b640 1 -- 172.21.15.42:0/1489808803 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d540ab600 con 0x7f4d6c0a3d90 2024-09-16T03:42:35.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.108+0000 7f4d70ecd640 5 monclient: authenticate success, global_id 4355 2024-09-16T03:42:35.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.108+0000 7f4d70ecd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:42:35.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.108+0000 7f4d69d9b640 10 monclient: handle_config config(0 keys) 2024-09-16T03:42:35.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.108+0000 7f4d70ecd640 10 monclient: get_monmap_and_config success 2024-09-16T03:42:35.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.108+0000 7f4d69d9b640 1 -- 172.21.15.42:0/1489808803 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d540ab9d0 con 0x7f4d6c0a3d90 2024-09-16T03:42:35.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.108+0000 7f4d69d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:35.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.108+0000 7f4d70ecd640 4 set_mon_vals no callback set 2024-09-16T03:42:35.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.108+0000 7f4d69d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:42:35.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.108+0000 7f4d69d9b640 10 monclient: dump: 2024-09-16T03:42:35.130 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:35.131 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:35.131 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:35.131 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:35.131 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:35.131 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:35.132 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:35.132 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:35.132 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:35.132 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:35.132 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.108+0000 7f4d70ecd640 10 monclient: shutdown 2024-09-16T03:42:35.132 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.108+0000 7f4d70ecd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:42:35.133 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.108+0000 7f4d70ecd640 1 -- 172.21.15.42:0/1489808803 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4d6c0a3d90 msgr2=0x7f4d6c137000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:35.133 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.108+0000 7f4d70ecd640 1 --2- 172.21.15.42:0/1489808803 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4d6c0a3d90 0x7f4d6c137000 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f4d54009870 tx=0x7f4d5401c3e0 comp rx=0 tx=0).stop 2024-09-16T03:42:35.133 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.108+0000 7f4d70ecd640 1 -- 172.21.15.42:0/1489808803 shutdown_connections 2024-09-16T03:42:35.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.108+0000 7f4d70ecd640 1 --2- 172.21.15.42:0/1489808803 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4d6c0a3d90 0x7f4d6c137000 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:35.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.108+0000 7f4d70ecd640 1 --2- 172.21.15.42:0/1489808803 >> v2:172.21.15.154:3300/0 conn(0x7f4d6c0a39b0 0x7f4d6c133b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:35.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.108+0000 7f4d70ecd640 1 -- 172.21.15.42:0/1489808803 >> 172.21.15.42:0/1489808803 conn(0x7f4d6c0a9fb0 msgr2=0x7f4d6c0aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:42:35.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.108+0000 7f4d70ecd640 1 -- 172.21.15.42:0/1489808803 shutdown_connections 2024-09-16T03:42:35.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.108+0000 7f4d70ecd640 1 -- 172.21.15.42:0/1489808803 wait complete. 2024-09-16T03:42:35.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.108+0000 7f4d70ecd640 10 monclient: build_initial_monmap 2024-09-16T03:42:35.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.108+0000 7f4d70ecd640 10 monclient: monmap: 2024-09-16T03:42:35.135 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:42:35.135 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:42:35.135 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:42:35.135 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:42:35.136 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:42:35.136 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:35.136 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:42:35.136 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:42:35.136 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:42:35.137 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:35.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.108+0000 7f4d70ecd640 1 librados: starting msgr at 2024-09-16T03:42:35.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.108+0000 7f4d70ecd640 1 librados: starting objecter 2024-09-16T03:42:35.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.108+0000 7f4d70ecd640 1 Processor -- start 2024-09-16T03:42:35.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.108+0000 7f4d70ecd640 1 -- start start 2024-09-16T03:42:35.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.108+0000 7f4d70ecd640 1 librados: setting wanted keys 2024-09-16T03:42:35.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.108+0000 7f4d70ecd640 1 librados: calling monclient init 2024-09-16T03:42:35.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.108+0000 7f4d70ecd640 10 monclient: init 2024-09-16T03:42:35.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.108+0000 7f4d70ecd640 10 monclient: _reopen_session rank -1 2024-09-16T03:42:35.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.108+0000 7f4d70ecd640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:42:35.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.108+0000 7f4d70ecd640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4d6c0a39b0 0x7f4d6c1308b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:35.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.108+0000 7f4d70ecd640 10 monclient(hunting): picked mon.noname-c con 0x7f4d6c0a39b0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:42:35.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.108+0000 7f4d70ecd640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4d6c0a3d90 0x7f4d6c0a18e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:35.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.108+0000 7f4d70ecd640 10 monclient(hunting): picked mon.noname-a con 0x7f4d6c0a3d90 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:42:35.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.108+0000 7f4d70ecd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4d6c137540 0x7f4d6c0a1e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:35.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.108+0000 7f4d70ecd640 10 monclient(hunting): picked mon.noname-b con 0x7f4d6c137540 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:42:35.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.108+0000 7f4d70ecd640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:35.140 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.108+0000 7f4d70ecd640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f4d6c1424e0 con 0x7f4d6c0a3d90 2024-09-16T03:42:35.140 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.108+0000 7f4d70ecd640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:35.140 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.108+0000 7f4d70ecd640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f4d6c142660 con 0x7f4d6c137540 2024-09-16T03:42:35.140 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.108+0000 7f4d70ecd640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:35.140 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.108+0000 7f4d70ecd640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f4d6c1427e0 con 0x7f4d6c0a39b0 2024-09-16T03:42:35.140 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.108+0000 7f4d70ecd640 10 monclient(hunting): _renew_subs 2024-09-16T03:42:35.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.108+0000 7f4d70ecd640 10 monclient(hunting): authenticate will time out at 1002.083776s 2024-09-16T03:42:35.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.108+0000 7f4d61d9b640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4d6c0a3d90 0x7f4d6c0a18e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:35.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.108+0000 7f4d6a59c640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4d6c0a39b0 0x7f4d6c1308b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:35.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.108+0000 7f4d61d9b640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4d6c0a3d90 0x7f4d6c0a18e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:42280/0 (socket says 172.21.15.42:42280) 2024-09-16T03:42:35.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.108+0000 7f4d61d9b640 1 -- 172.21.15.42:0/142120150 learned_addr learned my addr 172.21.15.42:0/142120150 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:42:35.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.108+0000 7f4d61d9b640 10 monclient(hunting): get_auth_request con 0x7f4d6c0a3d90 auth_method 0 2024-09-16T03:42:35.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.108+0000 7f4d61d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:35.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.108+0000 7f4d61d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:35.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.108+0000 7f4d61d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:35.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.108+0000 7f4d6a59c640 10 monclient(hunting): get_auth_request con 0x7f4d6c0a39b0 auth_method 0 2024-09-16T03:42:35.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.108+0000 7f4d6a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:35.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.108+0000 7f4d6a59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:35.143 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.108+0000 7f4d6a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:35.143 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.112+0000 7f4d61d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:35.143 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.112+0000 7f4d61d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:35.143 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.112+0000 7f4d61d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:35.143 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.112+0000 7f4d6a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:35.143 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.112+0000 7f4d6a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:35.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.112+0000 7f4d6a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:35.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.112+0000 7f4d6ad9d640 1 --2- 172.21.15.42:0/142120150 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4d6c137540 0x7f4d6c0a1e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:35.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.112+0000 7f4d61d9b640 10 monclient(hunting): handle_auth_done global_id 4422 payload 824 2024-09-16T03:42:35.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.112+0000 7f4d61d9b640 1 -- 172.21.15.42:0/142120150 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4d6c0a39b0 msgr2=0x7f4d6c1308b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:35.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.112+0000 7f4d61d9b640 1 --2- 172.21.15.42:0/142120150 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4d6c0a39b0 0x7f4d6c1308b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:35.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.112+0000 7f4d61d9b640 1 -- 172.21.15.42:0/142120150 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4d6c137540 msgr2=0x7f4d6c0a1e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:35.145 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.112+0000 7f4d61d9b640 1 --2- 172.21.15.42:0/142120150 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4d6c137540 0x7f4d6c0a1e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:35.145 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.112+0000 7f4d61d9b640 10 monclient: _finish_hunting 0 2024-09-16T03:42:35.145 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.112+0000 7f4d61d9b640 1 monclient: found mon.noname-a 2024-09-16T03:42:35.145 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.112+0000 7f4d61d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:42:35.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.112+0000 7f4d61d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:42:35.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.112+0000 7f4d61d9b640 1 -- 172.21.15.42:0/142120150 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d6c13a9d0 con 0x7f4d6c0a3d90 2024-09-16T03:42:35.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.112+0000 7f4d6ad9d640 10 monclient: get_auth_request con 0x7f4d6c137540 auth_method 0 2024-09-16T03:42:35.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.112+0000 7f4d6ad9d640 1 --2- 172.21.15.42:0/142120150 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4d6c137540 0x7f4d6c0a1e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T03:42:35.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.112+0000 7f4d61d9b640 1 --2- 172.21.15.42:0/142120150 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4d6c0a3d90 0x7f4d6c0a18e0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f4d540ae040 tx=0x7f4d5401ac20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:35.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.112+0000 7f4d637fe640 10 client.?.objecter ms_handle_connect 0x7f4d6c0a3d90 2024-09-16T03:42:35.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.112+0000 7f4d637fe640 10 client.?.objecter resend_mon_ops 2024-09-16T03:42:35.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.112+0000 7f4d637fe640 1 -- 172.21.15.42:0/142120150 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d540ba020 con 0x7f4d6c0a3d90 2024-09-16T03:42:35.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.112+0000 7f4d637fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:35.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.112+0000 7f4d637fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:42:35.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.112+0000 7f4d637fe640 10 monclient: dump: 2024-09-16T03:42:35.148 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:35.148 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:35.148 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:35.148 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:35.148 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:35.148 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:35.149 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:35.149 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:35.149 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:35.149 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:35.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.112+0000 7f4d637fe640 10 monclient: _finish_auth 0 2024-09-16T03:42:35.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.112+0000 7f4d637fe640 10 monclient: _check_auth_tickets 2024-09-16T03:42:35.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.112+0000 7f4d637fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:42:35.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.112+0000 7f4d637fe640 1 -- 172.21.15.42:0/142120150 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d540b6640 con 0x7f4d6c0a3d90 2024-09-16T03:42:35.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.112+0000 7f4d637fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:42:35.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.112+0000 7f4d637fe640 1 -- 172.21.15.42:0/142120150 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d540b0040 con 0x7f4d6c0a3d90 2024-09-16T03:42:35.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.112+0000 7f4d70ecd640 5 monclient: authenticate success, global_id 4422 2024-09-16T03:42:35.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.112+0000 7f4d70ecd640 10 monclient: _renew_subs 2024-09-16T03:42:35.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.112+0000 7f4d70ecd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:42:35.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.112+0000 7f4d70ecd640 1 -- 172.21.15.42:0/142120150 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d6c139d10 con 0x7f4d6c0a3d90 2024-09-16T03:42:35.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.112+0000 7f4d6959a640 4 set_mon_vals no callback set 2024-09-16T03:42:35.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.112+0000 7f4d637fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:35.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.112+0000 7f4d637fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:42:35.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.112+0000 7f4d637fe640 10 monclient: dump: 2024-09-16T03:42:35.151 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:35.151 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:35.151 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:35.151 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:35.151 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:35.151 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:35.151 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:35.152 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:35.152 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:35.152 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:35.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.112+0000 7f4d70ecd640 10 client.4422.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:42:35.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.112+0000 7f4d70ecd640 10 monclient: _renew_subs 2024-09-16T03:42:35.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.112+0000 7f4d70ecd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:42:35.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.112+0000 7f4d70ecd640 1 -- 172.21.15.42:0/142120150 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d6c132ae0 con 0x7f4d6c0a3d90 2024-09-16T03:42:35.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.112+0000 7f4d70ecd640 1 librados: init done 2024-09-16T03:42:35.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.112+0000 7f4d70ecd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:42:35.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.112+0000 7f4d70ecd640 1 --2- 172.21.15.42:0/142120150 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4d6c130df0 0x7f4d6c0a2360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:35.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.112+0000 7f4d70ecd640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:35.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.112+0000 7f4d70ecd640 1 -- 172.21.15.42:0/142120150 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f4d6c1342d0 con 0x7f4d6c130df0 2024-09-16T03:42:35.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.112+0000 7f4d6a59c640 1 --2- 172.21.15.42:0/142120150 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4d6c130df0 0x7f4d6c0a2360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:35.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.112+0000 7f4d6a59c640 10 monclient: get_auth_request con 0x7f4d6c130df0 auth_method 0 2024-09-16T03:42:35.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.112+0000 7f4d6a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:35.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.112+0000 7f4d6a59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:35.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.112+0000 7f4d6a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:35.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.112+0000 7f4d637fe640 1 -- 172.21.15.42:0/142120150 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f4d540bd020 con 0x7f4d6c0a3d90 2024-09-16T03:42:35.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.112+0000 7f4d6a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:35.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.112+0000 7f4d6a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:35.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.112+0000 7f4d6a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:35.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.112+0000 7f4d6a59c640 10 monclient(hunting): handle_auth_done global_id 4361 payload 824 2024-09-16T03:42:35.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.112+0000 7f4d637fe640 1 --2- 172.21.15.42:0/142120150 >> v2:172.21.15.154:6812/254845342 conn(0x7f4d4c037810 0x7f4d4c039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:35.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.112+0000 7f4d6a59c640 1 -- 172.21.15.42:0/142120150 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4d6c133b80 con 0x7f4d6c130df0 2024-09-16T03:42:35.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.112+0000 7f4d637fe640 1 -- 172.21.15.42:0/142120150 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 3534+0+0 (secure 0 0 0) 0x7f4d540e2990 con 0x7f4d6c0a3d90 2024-09-16T03:42:35.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.112+0000 7f4d637fe640 10 client.4422.objecter ms_dispatch 0x7f4d6c000ba0 osd_map(19..19 src has 1..19) 2024-09-16T03:42:35.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.112+0000 7f4d637fe640 3 client.4422.objecter handle_osd_map got epochs [19,19] > 0 2024-09-16T03:42:35.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.112+0000 7f4d637fe640 3 client.4422.objecter handle_osd_map decoding full epoch 19 2024-09-16T03:42:35.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.112+0000 7f4d637fe640 20 client.4422.objecter dump_active .. 0 homeless 2024-09-16T03:42:35.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.112+0000 7f4d637fe640 10 client.4422.objecter ms_handle_connect 0x7f4d6c130df0 2024-09-16T03:42:35.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.112+0000 7f4d637fe640 10 client.4422.objecter resend_mon_ops 2024-09-16T03:42:35.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.112+0000 7f4d6a59c640 1 --2- 172.21.15.42:0/142120150 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4d6c130df0 0x7f4d6c0a2360 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f4d5c00e7a0 tx=0x7f4d5c00ec60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:35.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.112+0000 7f4d637fe640 1 -- 172.21.15.42:0/142120150 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d5c017070 con 0x7f4d6c130df0 2024-09-16T03:42:35.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.112+0000 7f4d637fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:35.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.112+0000 7f4d637fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:42:35.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.112+0000 7f4d637fe640 10 monclient: dump: 2024-09-16T03:42:35.155 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:35.155 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:35.156 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:35.156 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:35.156 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:35.156 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:35.156 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:35.156 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:35.156 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:35.156 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:35.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.112+0000 7f4d6ad9d640 1 --2- 172.21.15.42:0/142120150 >> v2:172.21.15.154:6812/254845342 conn(0x7f4d4c037810 0x7f4d4c039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:35.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.112+0000 7f4d6ad9d640 10 monclient: get_auth_request con 0x7f4d4c037810 auth_method 0 2024-09-16T03:42:35.157 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.112+0000 7f4d637fe640 1 -- 172.21.15.42:0/142120150 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f4d5c00c7a0 con 0x7f4d6c130df0 2024-09-16T03:42:35.157 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.112+0000 7f4d637fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:42:35.157 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.112+0000 7f4d637fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:42:35.157 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.112+0000 7f4d637fe640 1 -- 172.21.15.42:0/142120150 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4d6c130df0 msgr2=0x7f4d6c0a2360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:35.157 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.112+0000 7f4d637fe640 1 --2- 172.21.15.42:0/142120150 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4d6c130df0 0x7f4d6c0a2360 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f4d5c00e7a0 tx=0x7f4d5c00ec60 comp rx=0 tx=0).stop 2024-09-16T03:42:35.157 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.112+0000 7f4d637fe640 1 -- 172.21.15.42:0/142120150 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4d6c130df0 msgr2=0x7f4d6c0a2360 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:42:35.157 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.112+0000 7f4d637fe640 1 --2- 172.21.15.42:0/142120150 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4d6c130df0 0x7f4d6c0a2360 secure :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f4d5c00e7a0 tx=0x7f4d5c00ec60 comp rx=0 tx=0).stop 2024-09-16T03:42:35.157 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.112+0000 7f4d637fe640 10 client.4422.objecter ms_handle_connect 0x7f4d4c037810 2024-09-16T03:42:35.157 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.112+0000 7f4d6ad9d640 1 --2- 172.21.15.42:0/142120150 >> v2:172.21.15.154:6812/254845342 conn(0x7f4d4c037810 0x7f4d4c039cd0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f4d64007fd0 tx=0x7f4d64009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:35.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.120+0000 7f4d70ecd640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:42:35.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.120+0000 7f4d70ecd640 1 --2- 172.21.15.42:0/142120150 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4d6c0a28a0 0x7f4d6c09d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:35.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.120+0000 7f4d70ecd640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:35.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.120+0000 7f4d70ecd640 1 -- 172.21.15.42:0/142120150 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f4d6c0a5750 con 0x7f4d6c0a28a0 2024-09-16T03:42:35.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.120+0000 7f4d6a59c640 1 --2- 172.21.15.42:0/142120150 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4d6c0a28a0 0x7f4d6c09d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:35.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.120+0000 7f4d6a59c640 10 monclient: get_auth_request con 0x7f4d6c0a28a0 auth_method 0 2024-09-16T03:42:35.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.120+0000 7f4d6a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:35.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.120+0000 7f4d6a59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:35.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.120+0000 7f4d6a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:35.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.120+0000 7f4d6a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:35.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.120+0000 7f4d6a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:35.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.120+0000 7f4d6a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:35.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.120+0000 7f4d6a59c640 10 monclient(hunting): handle_auth_done global_id 4364 payload 824 2024-09-16T03:42:35.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.120+0000 7f4d6a59c640 1 -- 172.21.15.42:0/142120150 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f4d6c1385e0 con 0x7f4d6c0a28a0 2024-09-16T03:42:35.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.120+0000 7f4d6a59c640 1 --2- 172.21.15.42:0/142120150 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4d6c0a28a0 0x7f4d6c09d8f0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f4d5c00e7a0 tx=0x7f4d5c0047e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:35.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.120+0000 7f4d637fe640 10 client.4422.objecter ms_handle_connect 0x7f4d6c0a28a0 2024-09-16T03:42:35.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.120+0000 7f4d637fe640 10 client.4422.objecter resend_mon_ops 2024-09-16T03:42:35.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.120+0000 7f4d637fe640 1 -- 172.21.15.42:0/142120150 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d5c017070 con 0x7f4d6c0a28a0 2024-09-16T03:42:35.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.120+0000 7f4d637fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:35.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.120+0000 7f4d637fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:42:35.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.120+0000 7f4d637fe640 10 monclient: dump: 2024-09-16T03:42:35.163 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:35.163 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:35.163 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:35.164 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:35.164 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:35.164 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:35.164 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:35.164 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:35.165 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:35.165 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:35.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.120+0000 7f4d637fe640 1 -- 172.21.15.42:0/142120150 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4711 (secure 0 0 0) 0x7f4d5c006430 con 0x7f4d6c0a28a0 2024-09-16T03:42:35.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.120+0000 7f4d637fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:42:35.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.120+0000 7f4d637fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:42:35.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.120+0000 7f4d637fe640 1 -- 172.21.15.42:0/142120150 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4d6c0a28a0 msgr2=0x7f4d6c09d8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:35.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.120+0000 7f4d637fe640 1 --2- 172.21.15.42:0/142120150 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4d6c0a28a0 0x7f4d6c09d8f0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f4d5c00e7a0 tx=0x7f4d5c0047e0 comp rx=0 tx=0).stop 2024-09-16T03:42:35.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.120+0000 7f4d637fe640 1 -- 172.21.15.42:0/142120150 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4d6c0a28a0 msgr2=0x7f4d6c09d8f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:42:35.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.120+0000 7f4d637fe640 1 --2- 172.21.15.42:0/142120150 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4d6c0a28a0 0x7f4d6c09d8f0 secure :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f4d5c00e7a0 tx=0x7f4d5c0047e0 comp rx=0 tx=0).stop 2024-09-16T03:42:35.167 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:42:35.167 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:42:35.167 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:42:35.167 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T03:42:35.167 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 8, 2024-09-16T03:42:35.167 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 34491, 2024-09-16T03:42:35.168 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:42:35.168 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:42:35.168 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T03:42:35.168 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T03:42:35.168 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:42:35.169 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 29, 2024-09-16T03:42:35.169 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:42:35.169 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:42:35.169 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:42:35.169 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:42:35.169 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:42:35.170 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:42:35.170 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:42:35.170 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:42:35.170 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:42:35.170 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:42:35.170 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:42:35.170 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:42:35.170 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:42:35.170 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:42:35.171 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:42:35.171 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:42:35.171 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:42:35.171 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:42:35.171 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:42:35.171 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:42:35.171 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:42:35.172 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:42:35.172 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:42:35.172 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:42:35.172 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:42:35.172 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:42:35.172 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:42:35.172 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:42:35.172 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:42:35.173 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:42:35.173 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:42:35.173 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:42:35.173 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:42:35.173 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:42:35.173 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:42:35.173 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:42:35.173 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:42:35.174 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:42:35.174 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:42:35.174 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:42:35.174 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:42:35.174 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:42:35.174 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:42:35.174 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:42:35.174 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:42:35.175 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:42:35.175 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:42:35.175 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:42:35.175 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:42:35.175 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:42:35.175 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:42:35.175 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:42:35.175 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:42:35.175 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:42:35.175 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:42:35.176 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:42:35.176 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:42:35.176 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:42:35.176 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:42:35.176 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:42:35.176 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:42:35.176 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:42:35.176 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:42:35.176 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:42:35.176 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:42:35.176 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:42:35.176 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:42:35.176 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:42:35.176 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:42:35.177 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:42:35.177 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:42:35.177 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:42:35.177 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:42:35.177 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:42:35.177 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:42:35.177 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:42:35.177 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:42:35.177 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:42:35.177 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:42:35.177 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:42:35.177 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:42:35.177 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:42:35.177 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:42:35.178 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:42:35.178 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:42:35.178 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:42:35.178 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:42:35.178 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:42:35.178 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:42:35.178 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:42:35.178 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:42:35.178 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:42:35.178 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:42:35.178 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:42:35.178 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:42:35.178 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:42:35.178 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:42:35.178 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:42:35.179 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:42:35.179 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:42:35.179 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:42:35.179 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:42:35.179 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:42:35.179 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:42:35.179 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:42:35.179 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:42:35.179 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:42:35.179 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:42:35.179 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:42:35.179 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:42:35.179 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:42:35.180 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:42:35.180 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:42:35.180 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:42:35.180 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:42:35.180 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:42:35.180 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:42:35.180 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:42:35.180 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:42:35.180 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:42:35.180 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:42:35.180 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:42:35.180 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:42:35.180 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:42:35.181 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:42:35.181 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:42:35.181 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:42:35.181 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:42:35.181 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:42:35.181 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:42:35.181 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:42:35.181 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:42:35.181 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:42:35.181 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:42:35.181 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:42:35.181 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:42:35.181 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:42:35.181 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:42:35.182 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:42:35.182 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:42:35.182 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:42:35.182 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T03:42:35.182 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:42:35.182 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:42:35.182 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:42:35.182 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 2 2024-09-16T03:42:35.182 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:42:35.182 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:42:35.182 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:42:35.182 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:42:35.182 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:42:35.182 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:42:35.182 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:42:35.183 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:42:35.183 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:42:35.183 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:42:35.183 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:42:35.183 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:42:35.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.120+0000 7f4d70ecd640 10 librados: watch_flush enter 2024-09-16T03:42:35.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.120+0000 7f4d70ecd640 10 librados: watch_flush exit 2024-09-16T03:42:35.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.120+0000 7f4d70ecd640 20 client.4422.objecter shutdown clearing up homeless session... 2024-09-16T03:42:35.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.120+0000 7f4d70ecd640 10 client.4422.objecter successfully canceled tick 2024-09-16T03:42:35.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.120+0000 7f4d70ecd640 1 -- 172.21.15.42:0/142120150 >> v2:172.21.15.154:6812/254845342 conn(0x7f4d4c037810 msgr2=0x7f4d4c039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:35.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.120+0000 7f4d70ecd640 1 --2- 172.21.15.42:0/142120150 >> v2:172.21.15.154:6812/254845342 conn(0x7f4d4c037810 0x7f4d4c039cd0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f4d64007fd0 tx=0x7f4d64009040 comp rx=0 tx=0).stop 2024-09-16T03:42:35.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.120+0000 7f4d70ecd640 10 monclient: shutdown 2024-09-16T03:42:35.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.120+0000 7f4d70ecd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:42:35.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.120+0000 7f4d70ecd640 1 -- 172.21.15.42:0/142120150 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4d6c0a3d90 msgr2=0x7f4d6c0a18e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:35.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.120+0000 7f4d70ecd640 1 --2- 172.21.15.42:0/142120150 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4d6c0a3d90 0x7f4d6c0a18e0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f4d540ae040 tx=0x7f4d5401ac20 comp rx=0 tx=0).stop 2024-09-16T03:42:35.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.120+0000 7f4d6ad9d640 1 -- 172.21.15.42:0/142120150 reap_dead start 2024-09-16T03:42:35.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.120+0000 7f4d70ecd640 1 -- 172.21.15.42:0/142120150 shutdown_connections 2024-09-16T03:42:35.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.120+0000 7f4d70ecd640 1 --2- 172.21.15.42:0/142120150 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4d6c0a3d90 0x7f4d6c0a18e0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:35.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.120+0000 7f4d70ecd640 1 -- 172.21.15.42:0/142120150 >> 172.21.15.42:0/142120150 conn(0x7f4d6c0a9fb0 msgr2=0x7f4d6c131980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:42:35.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.120+0000 7f4d70ecd640 1 -- 172.21.15.42:0/142120150 shutdown_connections 2024-09-16T03:42:35.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.120+0000 7f4d70ecd640 1 -- 172.21.15.42:0/142120150 wait complete. 2024-09-16T03:42:35.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.120+0000 7f4d70ecd640 1 librados: shutdown 2024-09-16T03:42:35.186 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:42:35.186 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T03:42:35.186 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T03:42:35.309 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.300+0000 7f913cdae640 10 monclient: get_monmap_and_config 2024-09-16T03:42:35.309 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.300+0000 7f913cdae640 10 monclient: build_initial_monmap 2024-09-16T03:42:35.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.300+0000 7f913cdae640 10 monclient: monmap: 2024-09-16T03:42:35.310 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:42:35.310 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:35.310 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:42:35.307284+0000 2024-09-16T03:42:35.311 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:42:35.307284+0000 2024-09-16T03:42:35.311 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:42:35.311 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:35.311 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:42:35.311 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:42:35.311 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:42:35.311 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:42:35.311 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:42:35.311 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:35.314 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.304+0000 7f913cdae640 1 Processor -- start 2024-09-16T03:42:35.314 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.304+0000 7f913cdae640 1 -- start start 2024-09-16T03:42:35.314 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.304+0000 7f913cdae640 10 monclient: init 2024-09-16T03:42:35.314 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.304+0000 7f913cdae640 10 monclient: _reopen_session rank -1 2024-09-16T03:42:35.314 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.304+0000 7f913cdae640 10 monclient: _add_conns ranks=[1,3,4,0,2] 2024-09-16T03:42:35.314 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.304+0000 7f913cdae640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f91380a39b0 addr v1:172.21.15.154:6789/0 2024-09-16T03:42:35.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.304+0000 7f913cdae640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f91380a3d90 0x7f913813c380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:35.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.304+0000 7f913cdae640 10 monclient(hunting): picked mon.noname-a con 0x7f91380a3d90 addr v2:172.21.15.42:3300/0 2024-09-16T03:42:35.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.304+0000 7f913cdae640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f913813c8c0 0x7f913813ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:35.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.304+0000 7f913cdae640 10 monclient(hunting): picked mon.noname-c con 0x7f913813c8c0 addr v2:172.21.15.154:3300/0 2024-09-16T03:42:35.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.304+0000 7f913cdae640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9138134290 con 0x7f91380a39b0 2024-09-16T03:42:35.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.304+0000 7f913cdae640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:35.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.304+0000 7f913cdae640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f913809c140 con 0x7f91380a3d90 2024-09-16T03:42:35.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.304+0000 7f913cdae640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:35.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.304+0000 7f913cdae640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f913809c420 con 0x7f913813c8c0 2024-09-16T03:42:35.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.304+0000 7f913cdae640 10 monclient(hunting): _renew_subs 2024-09-16T03:42:35.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.304+0000 7f913cdae640 10 monclient(hunting): authenticate will time out at 1002.277241s 2024-09-16T03:42:35.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.304+0000 7f912dd9b640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f91380a3d90 0x7f913813c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:35.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.304+0000 7f912dd9b640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f91380a3d90 0x7f913813c380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:42286/0 (socket says 172.21.15.42:42286) 2024-09-16T03:42:35.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.304+0000 7f912dd9b640 1 -- 172.21.15.42:0/3672620197 learned_addr learned my addr 172.21.15.42:0/3672620197 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:42:35.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.304+0000 7f912dd9b640 10 monclient(hunting): get_auth_request con 0x7f91380a3d90 auth_method 0 2024-09-16T03:42:35.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.304+0000 7f912dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:35.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.304+0000 7f912dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:35.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.304+0000 7f912dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:35.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.304+0000 7f912dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:35.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.304+0000 7f912dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:35.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.304+0000 7f912dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:35.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.304+0000 7f9136d9d640 1 --2- 172.21.15.42:0/3672620197 >> v2:172.21.15.154:3300/0 conn(0x7f913813c8c0 0x7f913813ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:35.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.304+0000 7f9136d9d640 10 monclient(hunting): get_auth_request con 0x7f913813c8c0 auth_method 0 2024-09-16T03:42:35.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.304+0000 7f9136d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:35.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.304+0000 7f9136d9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:35.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.304+0000 7f9136d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:35.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.304+0000 7f912dd9b640 10 monclient(hunting): handle_auth_done global_id 4425 payload 274 2024-09-16T03:42:35.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.304+0000 7f912dd9b640 1 -- 172.21.15.42:0/3672620197 >> v2:172.21.15.154:3300/0 conn(0x7f913813c8c0 msgr2=0x7f913813ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:42:35.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.304+0000 7f912dd9b640 1 --2- 172.21.15.42:0/3672620197 >> v2:172.21.15.154:3300/0 conn(0x7f913813c8c0 0x7f913813ecb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:35.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.304+0000 7f912dd9b640 1 -- 172.21.15.42:0/3672620197 >> v1:172.21.15.154:6789/0 conn(0x7f91380a39b0 legacy=0x7f9138133b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:42:35.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.304+0000 7f912dd9b640 10 monclient: _finish_hunting 0 2024-09-16T03:42:35.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.304+0000 7f912dd9b640 1 monclient: found mon.noname-a 2024-09-16T03:42:35.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.304+0000 7f912dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:42:35.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.304+0000 7f912dd9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:42:35.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.304+0000 7f912dd9b640 1 -- 172.21.15.42:0/3672620197 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f913806dd60 con 0x7f91380a3d90 2024-09-16T03:42:35.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.304+0000 7f912dd9b640 1 --2- 172.21.15.42:0/3672620197 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f91380a3d90 0x7f913813c380 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f913806b540 tx=0x7f912001c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80ce6e3d5e17644c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:35.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.308+0000 7f9135d9b640 1 -- 172.21.15.42:0/3672620197 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91200b0070 con 0x7f91380a3d90 2024-09-16T03:42:35.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.308+0000 7f9135d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:35.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.308+0000 7f9135d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:42:35.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.308+0000 7f9135d9b640 10 monclient: dump: 2024-09-16T03:42:35.318 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:35.318 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:35.318 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:35.318 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:35.318 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:35.318 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:35.319 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:35.319 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:35.319 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:35.319 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:35.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.308+0000 7f9135d9b640 10 monclient: _finish_auth 0 2024-09-16T03:42:35.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.308+0000 7f9135d9b640 10 monclient: _check_auth_tickets 2024-09-16T03:42:35.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.308+0000 7f9135d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:42:35.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.308+0000 7f9135d9b640 1 -- 172.21.15.42:0/3672620197 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f91200ab600 con 0x7f91380a3d90 2024-09-16T03:42:35.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.308+0000 7f9135d9b640 10 monclient: handle_config config(0 keys) 2024-09-16T03:42:35.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.308+0000 7f9135d9b640 1 -- 172.21.15.42:0/3672620197 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91200ab9d0 con 0x7f91380a3d90 2024-09-16T03:42:35.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.308+0000 7f913cdae640 5 monclient: authenticate success, global_id 4425 2024-09-16T03:42:35.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.308+0000 7f913cdae640 10 monclient: get_monmap_and_config success 2024-09-16T03:42:35.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.308+0000 7f913cdae640 4 set_mon_vals no callback set 2024-09-16T03:42:35.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.308+0000 7f9135d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:35.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.308+0000 7f9135d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:42:35.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.308+0000 7f9135d9b640 10 monclient: dump: 2024-09-16T03:42:35.320 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:35.320 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:35.320 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:35.320 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:35.320 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:35.320 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:35.320 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:35.320 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:35.320 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:35.320 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:35.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.308+0000 7f913cdae640 10 monclient: shutdown 2024-09-16T03:42:35.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.308+0000 7f913cdae640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:42:35.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.308+0000 7f913cdae640 1 -- 172.21.15.42:0/3672620197 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f91380a3d90 msgr2=0x7f913813c380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:35.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.308+0000 7f913cdae640 1 --2- 172.21.15.42:0/3672620197 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f91380a3d90 0x7f913813c380 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f913806b540 tx=0x7f912001c3a0 comp rx=0 tx=0).stop 2024-09-16T03:42:35.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.308+0000 7f913cdae640 1 -- 172.21.15.42:0/3672620197 shutdown_connections 2024-09-16T03:42:35.329 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.308+0000 7f913cdae640 1 --2- 172.21.15.42:0/3672620197 >> v2:172.21.15.154:3300/0 conn(0x7f913813c8c0 0x7f913813ecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:35.329 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.308+0000 7f913cdae640 1 --2- 172.21.15.42:0/3672620197 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f91380a3d90 0x7f913813c380 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:35.329 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.308+0000 7f913cdae640 1 -- 172.21.15.42:0/3672620197 >> 172.21.15.42:0/3672620197 conn(0x7f91380a9fb0 msgr2=0x7f91380aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:42:35.329 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.308+0000 7f913cdae640 1 -- 172.21.15.42:0/3672620197 shutdown_connections 2024-09-16T03:42:35.329 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.308+0000 7f913cdae640 1 -- 172.21.15.42:0/3672620197 wait complete. 2024-09-16T03:42:35.329 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.308+0000 7f913cdae640 10 monclient: build_initial_monmap 2024-09-16T03:42:35.329 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.308+0000 7f913cdae640 10 monclient: monmap: 2024-09-16T03:42:35.329 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:42:35.329 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:42:35.329 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:42:35.329 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:42:35.329 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:42:35.329 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:35.329 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:42:35.330 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:42:35.330 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:42:35.330 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:35.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.308+0000 7f913cdae640 1 librados: starting msgr at 2024-09-16T03:42:35.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.308+0000 7f913cdae640 1 librados: starting objecter 2024-09-16T03:42:35.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.308+0000 7f913cdae640 1 Processor -- start 2024-09-16T03:42:35.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.308+0000 7f913cdae640 1 -- start start 2024-09-16T03:42:35.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.308+0000 7f913cdae640 1 librados: setting wanted keys 2024-09-16T03:42:35.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.308+0000 7f913cdae640 1 librados: calling monclient init 2024-09-16T03:42:35.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.308+0000 7f913cdae640 10 monclient: init 2024-09-16T03:42:35.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.308+0000 7f913cdae640 10 monclient: _reopen_session rank -1 2024-09-16T03:42:35.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.308+0000 7f913cdae640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T03:42:35.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.308+0000 7f913cdae640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f91380a39b0 0x7f91381cf0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:35.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.308+0000 7f913cdae640 10 monclient(hunting): picked mon.noname-b con 0x7f91380a39b0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:42:35.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.308+0000 7f913cdae640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f91380a3d90 0x7f91381cf600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:35.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.308+0000 7f913cdae640 10 monclient(hunting): picked mon.noname-c con 0x7f91380a3d90 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:42:35.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.308+0000 7f913cdae640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f913813c8c0 0x7f91381cfb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:35.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.308+0000 7f913cdae640 10 monclient(hunting): picked mon.noname-a con 0x7f913813c8c0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:42:35.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.308+0000 7f913cdae640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:35.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.308+0000 7f913cdae640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f9138141da0 con 0x7f913813c8c0 2024-09-16T03:42:35.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.308+0000 7f913cdae640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:35.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.308+0000 7f913cdae640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f9138141c20 con 0x7f91380a39b0 2024-09-16T03:42:35.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.308+0000 7f913cdae640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:35.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.308+0000 7f913cdae640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f9138141f20 con 0x7f91380a3d90 2024-09-16T03:42:35.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.308+0000 7f913cdae640 10 monclient(hunting): _renew_subs 2024-09-16T03:42:35.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.308+0000 7f913cdae640 10 monclient(hunting): authenticate will time out at 1002.282323s 2024-09-16T03:42:35.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.308+0000 7f912dd9b640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f91380a3d90 0x7f91381cf600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:35.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.308+0000 7f9136d9d640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f913813c8c0 0x7f91381cfb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:35.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.308+0000 7f912dd9b640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f91380a3d90 0x7f91381cf600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:54084/0 (socket says 172.21.15.42:54084) 2024-09-16T03:42:35.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.308+0000 7f912dd9b640 1 -- 172.21.15.42:0/353231530 learned_addr learned my addr 172.21.15.42:0/353231530 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:42:35.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.308+0000 7f912dd9b640 10 monclient(hunting): get_auth_request con 0x7f91380a3d90 auth_method 0 2024-09-16T03:42:35.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.308+0000 7f912dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:35.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.308+0000 7f912dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:35.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.308+0000 7f912dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:35.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.308+0000 7f9136d9d640 10 monclient(hunting): get_auth_request con 0x7f913813c8c0 auth_method 0 2024-09-16T03:42:35.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.308+0000 7f9136d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:35.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.308+0000 7f9136d9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:35.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.308+0000 7f9136d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:35.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.308+0000 7f912dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:35.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.308+0000 7f912dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:35.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.308+0000 7f912dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:35.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.308+0000 7f9136d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:35.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.308+0000 7f9136d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:35.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.308+0000 7f9136d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:35.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.308+0000 7f913659c640 1 --2- 172.21.15.42:0/353231530 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f91380a39b0 0x7f91381cf0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:35.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.308+0000 7f913659c640 10 monclient(hunting): get_auth_request con 0x7f91380a39b0 auth_method 0 2024-09-16T03:42:35.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.308+0000 7f913659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:35.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.308+0000 7f913659c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:35.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.308+0000 7f913659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:35.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.308+0000 7f912dd9b640 10 monclient(hunting): handle_auth_done global_id 4367 payload 824 2024-09-16T03:42:35.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.308+0000 7f912dd9b640 1 -- 172.21.15.42:0/353231530 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f91380a39b0 msgr2=0x7f91381cf0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:35.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.308+0000 7f912dd9b640 1 --2- 172.21.15.42:0/353231530 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f91380a39b0 0x7f91381cf0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:35.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.308+0000 7f912dd9b640 1 -- 172.21.15.42:0/353231530 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f913813c8c0 msgr2=0x7f91381cfb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:35.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.308+0000 7f912dd9b640 1 --2- 172.21.15.42:0/353231530 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f913813c8c0 0x7f91381cfb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:35.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.308+0000 7f912dd9b640 10 monclient: _finish_hunting 0 2024-09-16T03:42:35.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.308+0000 7f912dd9b640 1 monclient: found mon.noname-c 2024-09-16T03:42:35.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.308+0000 7f912dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:42:35.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.308+0000 7f912dd9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:42:35.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.308+0000 7f912dd9b640 1 -- 172.21.15.42:0/353231530 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9138139340 con 0x7f91380a3d90 2024-09-16T03:42:35.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.308+0000 7f9136d9d640 1 --2- 172.21.15.42:0/353231530 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f913813c8c0 0x7f91381cfb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:42:35.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.308+0000 7f912dd9b640 1 --2- 172.21.15.42:0/353231530 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f91380a3d90 0x7f91381cf600 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f912001aa50 tx=0x7f9120019140 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:35.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.308+0000 7f912f7fe640 10 client.?.objecter ms_handle_connect 0x7f91380a3d90 2024-09-16T03:42:35.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.308+0000 7f912f7fe640 10 client.?.objecter resend_mon_ops 2024-09-16T03:42:35.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.308+0000 7f912f7fe640 1 -- 172.21.15.42:0/353231530 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91200b0040 con 0x7f91380a3d90 2024-09-16T03:42:35.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.308+0000 7f912f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:35.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.308+0000 7f912f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:42:35.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.308+0000 7f912f7fe640 10 monclient: dump: 2024-09-16T03:42:35.334 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:35.335 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:35.335 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:35.335 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:35.335 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:35.335 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:35.335 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:35.335 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:35.335 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:35.335 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:35.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.308+0000 7f912f7fe640 10 monclient: _finish_auth 0 2024-09-16T03:42:35.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.308+0000 7f912f7fe640 10 monclient: _check_auth_tickets 2024-09-16T03:42:35.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.308+0000 7f912f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:42:35.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.308+0000 7f912f7fe640 1 -- 172.21.15.42:0/353231530 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f91200b9070 con 0x7f91380a3d90 2024-09-16T03:42:35.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.308+0000 7f913cdae640 5 monclient: authenticate success, global_id 4367 2024-09-16T03:42:35.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.308+0000 7f913cdae640 10 monclient: _renew_subs 2024-09-16T03:42:35.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.308+0000 7f913cdae640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:42:35.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.308+0000 7f913cdae640 1 -- 172.21.15.42:0/353231530 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f913813b8c0 con 0x7f91380a3d90 2024-09-16T03:42:35.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.308+0000 7f912f7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:42:35.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.308+0000 7f912f7fe640 1 -- 172.21.15.42:0/353231530 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91200b4860 con 0x7f91380a3d90 2024-09-16T03:42:35.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.308+0000 7f912f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:35.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.308+0000 7f912f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:42:35.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.308+0000 7f912f7fe640 10 monclient: dump: 2024-09-16T03:42:35.336 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:35.337 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:35.337 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:35.337 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:35.337 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:35.337 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:35.337 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:35.337 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:35.337 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:35.337 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:35.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.308+0000 7f913559a640 4 set_mon_vals no callback set 2024-09-16T03:42:35.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.308+0000 7f913cdae640 10 client.4367.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:42:35.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.308+0000 7f913cdae640 10 monclient: _renew_subs 2024-09-16T03:42:35.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.308+0000 7f913cdae640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:42:35.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.308+0000 7f913cdae640 1 -- 172.21.15.42:0/353231530 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f913813a920 con 0x7f91380a3d90 2024-09-16T03:42:35.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.308+0000 7f913cdae640 1 librados: init done 2024-09-16T03:42:35.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.312+0000 7f913cdae640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:42:35.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.312+0000 7f913cdae640 1 --2- 172.21.15.42:0/353231530 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f913812d2d0 0x7f913812d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:35.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.312+0000 7f913cdae640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:35.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.312+0000 7f913cdae640 1 -- 172.21.15.42:0/353231530 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f913813c200 con 0x7f913812d2d0 2024-09-16T03:42:35.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.312+0000 7f912f7fe640 1 -- 172.21.15.42:0/353231530 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f91200b4a00 con 0x7f91380a3d90 2024-09-16T03:42:35.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.312+0000 7f912f7fe640 1 --2- 172.21.15.42:0/353231530 >> v2:172.21.15.154:6812/254845342 conn(0x7f9118037810 0x7f9118039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:35.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.312+0000 7f912f7fe640 1 -- 172.21.15.42:0/353231530 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(19..19 src has 1..19) ==== 3534+0+0 (secure 0 0 0) 0x7f91200e25c0 con 0x7f91380a3d90 2024-09-16T03:42:35.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.312+0000 7f912f7fe640 10 client.4367.objecter ms_dispatch 0x7f9138000ba0 osd_map(19..19 src has 1..19) 2024-09-16T03:42:35.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.312+0000 7f912f7fe640 3 client.4367.objecter handle_osd_map got epochs [19,19] > 0 2024-09-16T03:42:35.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.312+0000 7f912f7fe640 3 client.4367.objecter handle_osd_map decoding full epoch 19 2024-09-16T03:42:35.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.312+0000 7f913659c640 1 --2- 172.21.15.42:0/353231530 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f913812d2d0 0x7f913812d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:35.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.312+0000 7f913659c640 10 monclient: get_auth_request con 0x7f913812d2d0 auth_method 0 2024-09-16T03:42:35.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.312+0000 7f913659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:35.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.312+0000 7f913659c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:35.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.312+0000 7f913659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:35.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.312+0000 7f912f7fe640 20 client.4367.objecter dump_active .. 0 homeless 2024-09-16T03:42:35.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.312+0000 7f913659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:35.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.312+0000 7f913659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:35.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.312+0000 7f913659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:35.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.312+0000 7f9136d9d640 1 --2- 172.21.15.42:0/353231530 >> v2:172.21.15.154:6812/254845342 conn(0x7f9118037810 0x7f9118039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:35.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.312+0000 7f9136d9d640 10 monclient: get_auth_request con 0x7f9118037810 auth_method 0 2024-09-16T03:42:35.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.312+0000 7f913659c640 10 monclient(hunting): handle_auth_done global_id 4360 payload 824 2024-09-16T03:42:35.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.312+0000 7f913659c640 1 -- 172.21.15.42:0/353231530 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9138133b80 con 0x7f913812d2d0 2024-09-16T03:42:35.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.312+0000 7f912f7fe640 10 client.4367.objecter ms_handle_connect 0x7f913812d2d0 2024-09-16T03:42:35.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.312+0000 7f912f7fe640 10 client.4367.objecter resend_mon_ops 2024-09-16T03:42:35.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.312+0000 7f913659c640 1 --2- 172.21.15.42:0/353231530 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f913812d2d0 0x7f913812d6b0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f912800e450 tx=0x7f912800e910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:35.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.312+0000 7f912f7fe640 10 client.4367.objecter ms_handle_connect 0x7f9118037810 2024-09-16T03:42:35.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.312+0000 7f9136d9d640 1 --2- 172.21.15.42:0/353231530 >> v2:172.21.15.154:6812/254845342 conn(0x7f9118037810 0x7f9118039cd0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f913813b0f0 tx=0x7f9130009820 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:35.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.312+0000 7f912f7fe640 1 -- 172.21.15.42:0/353231530 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9128007240 con 0x7f913812d2d0 2024-09-16T03:42:35.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.312+0000 7f912f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:35.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.312+0000 7f912f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:42:35.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.312+0000 7f912f7fe640 10 monclient: dump: 2024-09-16T03:42:35.340 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:35.340 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:35.340 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:35.340 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:35.340 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:35.340 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:35.340 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:35.341 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:35.341 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:35.341 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:35.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.312+0000 7f912f7fe640 1 -- 172.21.15.42:0/353231530 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f9128007240 con 0x7f913812d2d0 2024-09-16T03:42:35.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.312+0000 7f912f7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:42:35.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.312+0000 7f912f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:42:35.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.312+0000 7f912f7fe640 1 -- 172.21.15.42:0/353231530 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f913812d2d0 msgr2=0x7f913812d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:35.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.312+0000 7f912f7fe640 1 --2- 172.21.15.42:0/353231530 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f913812d2d0 0x7f913812d6b0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f912800e450 tx=0x7f912800e910 comp rx=0 tx=0).stop 2024-09-16T03:42:35.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.312+0000 7f912f7fe640 1 -- 172.21.15.42:0/353231530 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f913812d2d0 msgr2=0x7f913812d6b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:42:35.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.312+0000 7f912f7fe640 1 --2- 172.21.15.42:0/353231530 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f913812d2d0 0x7f913812d6b0 secure :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f912800e450 tx=0x7f912800e910 comp rx=0 tx=0).stop 2024-09-16T03:42:35.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.320+0000 7f913cdae640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:42:35.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.320+0000 7f913cdae640 1 --2- 172.21.15.42:0/353231530 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f913812dbf0 0x7f9138135be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:35.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.320+0000 7f913cdae640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:35.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.320+0000 7f913cdae640 1 -- 172.21.15.42:0/353231530 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f91380008e0 con 0x7f913812dbf0 2024-09-16T03:42:35.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.320+0000 7f913659c640 1 --2- 172.21.15.42:0/353231530 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f913812dbf0 0x7f9138135be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:35.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.320+0000 7f913659c640 10 monclient: get_auth_request con 0x7f913812dbf0 auth_method 0 2024-09-16T03:42:35.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.320+0000 7f913659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:35.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.320+0000 7f913659c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:35.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.320+0000 7f913659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:35.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.320+0000 7f913659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:35.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.320+0000 7f913659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:35.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.320+0000 7f913659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:35.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.320+0000 7f913659c640 10 monclient(hunting): handle_auth_done global_id 4363 payload 824 2024-09-16T03:42:35.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.320+0000 7f913659c640 1 -- 172.21.15.42:0/353231530 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f913813ab60 con 0x7f913812dbf0 2024-09-16T03:42:35.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.320+0000 7f913659c640 1 --2- 172.21.15.42:0/353231530 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f913812dbf0 0x7f9138135be0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f912800e450 tx=0x7f9128003770 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:35.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.320+0000 7f912f7fe640 10 client.4367.objecter ms_handle_connect 0x7f913812dbf0 2024-09-16T03:42:35.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.320+0000 7f912f7fe640 10 client.4367.objecter resend_mon_ops 2024-09-16T03:42:35.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.320+0000 7f912f7fe640 1 -- 172.21.15.42:0/353231530 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f912801e760 con 0x7f913812dbf0 2024-09-16T03:42:35.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.320+0000 7f912f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:35.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.320+0000 7f912f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:42:35.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.320+0000 7f912f7fe640 10 monclient: dump: 2024-09-16T03:42:35.344 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:35.344 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:35.344 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:35.344 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:35.344 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:35.345 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:35.345 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:35.345 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:35.345 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:35.345 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:35.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.320+0000 7f912f7fe640 1 -- 172.21.15.42:0/353231530 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4878 (secure 0 0 0) 0x7f9128029460 con 0x7f913812dbf0 2024-09-16T03:42:35.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.320+0000 7f912f7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:42:35.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.320+0000 7f912f7fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:42:35.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.320+0000 7f912f7fe640 1 -- 172.21.15.42:0/353231530 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f913812dbf0 msgr2=0x7f9138135be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:35.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.320+0000 7f912f7fe640 1 --2- 172.21.15.42:0/353231530 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f913812dbf0 0x7f9138135be0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f912800e450 tx=0x7f9128003770 comp rx=0 tx=0).stop 2024-09-16T03:42:35.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.320+0000 7f912f7fe640 1 -- 172.21.15.42:0/353231530 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f913812dbf0 msgr2=0x7f9138135be0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:42:35.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.320+0000 7f912f7fe640 1 --2- 172.21.15.42:0/353231530 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f913812dbf0 0x7f9138135be0 secure :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f912800e450 tx=0x7f9128003770 comp rx=0 tx=0).stop 2024-09-16T03:42:35.347 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:42:35.347 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:42:35.347 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:42:35.347 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T03:42:35.348 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 8, 2024-09-16T03:42:35.348 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 34655, 2024-09-16T03:42:35.348 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:42:35.348 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:42:35.348 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T03:42:35.348 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T03:42:35.348 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:42:35.348 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 29, 2024-09-16T03:42:35.348 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:42:35.348 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:42:35.348 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:42:35.348 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:42:35.348 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:42:35.348 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:42:35.349 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:42:35.349 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:42:35.349 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:42:35.349 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:42:35.349 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:42:35.349 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:42:35.349 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:42:35.349 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:42:35.349 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:42:35.349 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:42:35.349 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:42:35.349 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:42:35.349 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:42:35.349 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:42:35.350 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:42:35.350 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:42:35.350 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:42:35.350 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:42:35.350 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:42:35.350 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:42:35.350 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:42:35.350 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:42:35.350 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:42:35.350 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:42:35.350 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:42:35.350 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:42:35.350 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:42:35.351 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:42:35.351 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:42:35.351 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:42:35.351 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:42:35.351 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:42:35.351 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:42:35.351 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:42:35.351 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:42:35.351 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:42:35.351 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:42:35.351 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:42:35.351 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:42:35.351 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:42:35.351 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:42:35.352 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:42:35.352 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:42:35.352 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:42:35.352 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:42:35.352 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:42:35.352 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:42:35.352 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:42:35.352 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:42:35.352 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:42:35.352 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:42:35.352 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:42:35.352 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:42:35.352 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:42:35.352 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:42:35.353 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:42:35.353 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:42:35.353 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:42:35.353 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:42:35.353 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:42:35.353 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:42:35.353 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:42:35.353 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:42:35.353 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:42:35.353 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:42:35.353 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:42:35.353 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:42:35.353 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:42:35.353 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:42:35.354 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:42:35.354 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:42:35.354 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:42:35.354 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:42:35.354 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:42:35.354 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:42:35.354 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:42:35.354 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:42:35.354 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:42:35.354 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:42:35.354 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:42:35.354 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:42:35.354 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:42:35.354 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:42:35.355 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:42:35.355 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:42:35.355 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:42:35.355 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:42:35.355 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:42:35.355 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:42:35.355 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:42:35.355 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:42:35.355 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:42:35.355 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:42:35.355 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:42:35.355 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:42:35.355 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:42:35.355 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:42:35.355 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:42:35.356 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:42:35.356 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:42:35.356 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:42:35.356 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:42:35.356 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:42:35.356 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:42:35.356 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:42:35.356 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:42:35.356 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:42:35.356 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:42:35.356 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:42:35.356 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:42:35.356 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:42:35.356 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:42:35.357 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:42:35.357 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:42:35.357 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:42:35.357 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:42:35.357 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:42:35.357 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:42:35.357 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:42:35.357 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:42:35.357 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:42:35.357 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:42:35.357 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:42:35.357 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:42:35.357 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:42:35.358 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:42:35.358 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:42:35.358 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:42:35.358 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:42:35.358 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:42:35.358 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:42:35.358 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:42:35.358 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:42:35.358 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:42:35.358 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:42:35.358 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T03:42:35.358 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:42:35.358 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:42:35.358 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:42:35.358 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 2 2024-09-16T03:42:35.359 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:42:35.359 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:42:35.359 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:42:35.359 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:42:35.359 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:42:35.359 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:42:35.359 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 5 2024-09-16T03:42:35.359 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:42:35.359 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:42:35.359 INFO:teuthology.orchestra.run.smithi042.stdout: "mgr": [ 2024-09-16T03:42:35.359 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:42:35.359 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:42:35.359 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:42:35.359 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:42:35.360 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:42:35.360 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:42:35.360 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:42:35.360 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:42:35.360 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:42:35.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.320+0000 7f913cdae640 10 librados: watch_flush enter 2024-09-16T03:42:35.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.320+0000 7f913cdae640 10 librados: watch_flush exit 2024-09-16T03:42:35.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.320+0000 7f913cdae640 20 client.4367.objecter shutdown clearing up homeless session... 2024-09-16T03:42:35.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.320+0000 7f913cdae640 10 client.4367.objecter successfully canceled tick 2024-09-16T03:42:35.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.320+0000 7f913cdae640 1 -- 172.21.15.42:0/353231530 >> v2:172.21.15.154:6812/254845342 conn(0x7f9118037810 msgr2=0x7f9118039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:35.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.320+0000 7f913cdae640 1 --2- 172.21.15.42:0/353231530 >> v2:172.21.15.154:6812/254845342 conn(0x7f9118037810 0x7f9118039cd0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f913813b0f0 tx=0x7f9130009820 comp rx=0 tx=0).stop 2024-09-16T03:42:35.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.320+0000 7f913cdae640 10 monclient: shutdown 2024-09-16T03:42:35.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.320+0000 7f913cdae640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:42:35.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.320+0000 7f913cdae640 1 -- 172.21.15.42:0/353231530 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f91380a3d90 msgr2=0x7f91381cf600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:35.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.320+0000 7f913cdae640 1 --2- 172.21.15.42:0/353231530 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f91380a3d90 0x7f91381cf600 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f912001aa50 tx=0x7f9120019140 comp rx=0 tx=0).stop 2024-09-16T03:42:35.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.320+0000 7f9136d9d640 1 -- 172.21.15.42:0/353231530 reap_dead start 2024-09-16T03:42:35.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.320+0000 7f9136d9d640 1 -- 172.21.15.42:0/353231530 reap_dead start 2024-09-16T03:42:35.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.320+0000 7f913cdae640 1 -- 172.21.15.42:0/353231530 shutdown_connections 2024-09-16T03:42:35.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.320+0000 7f913cdae640 1 -- 172.21.15.42:0/353231530 >> 172.21.15.42:0/353231530 conn(0x7f91380a9fb0 msgr2=0x7f9138131980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:42:35.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.320+0000 7f913cdae640 1 -- 172.21.15.42:0/353231530 shutdown_connections 2024-09-16T03:42:35.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.320+0000 7f913cdae640 1 -- 172.21.15.42:0/353231530 wait complete. 2024-09-16T03:42:35.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:35.320+0000 7f913cdae640 1 librados: shutdown 2024-09-16T03:42:35.363 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:42:35.363 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-09-16T03:42:35.363 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['a', 'c'] 2024-09-16T03:42:35.364 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-09-16T03:42:35.364 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-16T03:42:35.364 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-16T03:42:35.364 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T03:42:35.390 INFO:tasks.ceph.mon.b:Stopped 2024-09-16T03:42:35.390 INFO:tasks.mon_thrash:Sending STOP to mon a 2024-09-16T03:42:35.391 INFO:tasks.ceph.mon.a:Sent signal 19 2024-09-16T03:42:35.391 INFO:tasks.mon_thrash:Sending STOP to mon c 2024-09-16T03:42:35.391 INFO:tasks.ceph.mon.c:Sent signal 19 2024-09-16T03:42:35.392 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-16T03:42:50.393 INFO:tasks.mon_thrash:Sending CONT to mon a 2024-09-16T03:42:50.393 INFO:tasks.ceph.mon.a:Sent signal 18 2024-09-16T03:42:50.394 INFO:tasks.mon_thrash:Sending CONT to mon c 2024-09-16T03:42:50.394 INFO:tasks.ceph.mon.c:Sent signal 18 2024-09-16T03:42:50.394 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T03:42:50.394 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:42:50.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:50.516+0000 7fac37f9e640 10 monclient: get_monmap_and_config 2024-09-16T03:42:50.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:50.516+0000 7fac37f9e640 10 monclient: build_initial_monmap 2024-09-16T03:42:50.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:50.516+0000 7fac37f9e640 10 monclient: monmap: 2024-09-16T03:42:50.526 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:42:50.526 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:50.526 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:42:50.523812+0000 2024-09-16T03:42:50.526 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:42:50.523812+0000 2024-09-16T03:42:50.526 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:42:50.526 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:50.526 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:42:50.526 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:42:50.527 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:42:50.527 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:42:50.527 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:42:50.527 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:50.528 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:50.520+0000 7fac37f9e640 1 Processor -- start 2024-09-16T03:42:50.529 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:50.520+0000 7fac37f9e640 1 -- start start 2024-09-16T03:42:50.529 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:50.520+0000 7fac37f9e640 10 monclient: init 2024-09-16T03:42:50.529 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:50.520+0000 7fac37f9e640 10 monclient: _reopen_session rank -1 2024-09-16T03:42:50.529 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:50.520+0000 7fac37f9e640 10 monclient: _add_conns ranks=[3,1,2,0,4] 2024-09-16T03:42:50.529 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:50.520+0000 7fac37f9e640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fac300a1330 0x7fac300a1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:50.529 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:50.520+0000 7fac37f9e640 10 monclient(hunting): picked mon.noname-a con 0x7fac300a1330 addr v2:172.21.15.42:3300/0 2024-09-16T03:42:50.529 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:50.520+0000 7fac37f9e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fac3006dac0 addr v1:172.21.15.154:6789/0 2024-09-16T03:42:50.529 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:50.520+0000 7fac37f9e640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fac300a1c50 0x7fac30135810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:50.529 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:50.520+0000 7fac37f9e640 10 monclient(hunting): picked mon.noname-b con 0x7fac300a1c50 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:42:50.529 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:50.520+0000 7fac37f9e640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fac3009c090 con 0x7fac3006dac0 2024-09-16T03:42:50.529 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:50.520+0000 7fac37f9e640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:50.529 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:50.520+0000 7fac37f9e640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7fac3009f610 con 0x7fac300a1330 2024-09-16T03:42:50.529 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:50.520+0000 7fac37f9e640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:50.530 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:50.520+0000 7fac37f9e640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fac3009f8f0 con 0x7fac300a1c50 2024-09-16T03:42:50.530 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:50.520+0000 7fac37f9e640 10 monclient(hunting): _renew_subs 2024-09-16T03:42:50.530 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:50.520+0000 7fac37f9e640 10 monclient(hunting): authenticate will time out at 1017.493264s 2024-09-16T03:42:50.530 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:50.520+0000 7fac35539640 1 -- >> v1:172.21.15.154:6789/0 conn(0x7fac3006dac0 legacy=0x7fac30135100 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.154:6789/0 2024-09-16T03:42:50.530 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:50.520+0000 7fac35d3a640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fac300a1330 0x7fac300a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:50.530 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:50.520+0000 7fac3653b640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fac300a1c50 0x7fac30135810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:50.530 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:50.520+0000 7fac35d3a640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fac300a1330 0x7fac300a1710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:40160/0 (socket says 172.21.15.42:40160) 2024-09-16T03:42:50.530 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:50.520+0000 7fac35d3a640 1 -- 172.21.15.42:0/1415442644 learned_addr learned my addr 172.21.15.42:0/1415442644 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:42:50.530 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:50.520+0000 7fac35d3a640 10 monclient(hunting): get_auth_request con 0x7fac300a1330 auth_method 0 2024-09-16T03:42:50.530 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:50.520+0000 7fac35d3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:50.530 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:50.520+0000 7fac35d3a640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:50.530 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:50.520+0000 7fac35d3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:50.530 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:50.520+0000 7fac3653b640 10 monclient(hunting): get_auth_request con 0x7fac300a1c50 auth_method 0 2024-09-16T03:42:50.531 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:50.520+0000 7fac3653b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:50.531 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:50.520+0000 7fac3653b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:50.531 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:50.520+0000 7fac3653b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:50.531 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:50.520+0000 7fac3653b640 1 -- 172.21.15.42:0/1415442644 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fac300a1c50 msgr2=0x7fac30135810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T03:42:50.531 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:50.520+0000 7fac3653b640 1 -- 172.21.15.42:0/1415442644 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fac300a1c50 msgr2=0x7fac30135810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:42:50.531 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:50.520+0000 7fac3653b640 1 --2- 172.21.15.42:0/1415442644 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fac300a1c50 0x7fac30135810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:42:50.531 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:50.520+0000 7fac35d3a640 1 -- 172.21.15.42:0/1415442644 >> v2:172.21.15.42:3300/0 conn(0x7fac300a1330 msgr2=0x7fac300a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T03:42:50.531 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:50.520+0000 7fac35d3a640 1 -- 172.21.15.42:0/1415442644 >> v2:172.21.15.42:3300/0 conn(0x7fac300a1330 msgr2=0x7fac300a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:42:50.531 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:50.520+0000 7fac35d3a640 1 --2- 172.21.15.42:0/1415442644 >> v2:172.21.15.42:3300/0 conn(0x7fac300a1330 0x7fac300a1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:42:50.531 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:50.520+0000 7fac35d3a640 1 --2- 172.21.15.42:0/1415442644 >> v2:172.21.15.42:3300/0 conn(0x7fac300a1330 0x7fac300a1710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:42:50.531 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:50.520+0000 7fac3653b640 1 --2- 172.21.15.42:0/1415442644 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fac300a1c50 0x7fac30135810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:42:50.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:50.724+0000 7fac3653b640 1 --2- 172.21.15.42:0/1415442644 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fac300a1c50 0x7fac30135810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:50.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:50.724+0000 7fac35539640 1 -- 172.21.15.42:0/1415442644 >> v1:172.21.15.154:6789/0 conn(0x7fac3006dac0 legacy=0x7fac30135100 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.154:6789/0 2024-09-16T03:42:50.733 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:50.724+0000 7fac3653b640 10 monclient(hunting): get_auth_request con 0x7fac300a1c50 auth_method 0 2024-09-16T03:42:50.733 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:50.724+0000 7fac3653b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:50.733 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:50.724+0000 7fac3653b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:50.733 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:50.724+0000 7fac3653b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:42:50.733 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:50.724+0000 7fac35d3a640 1 --2- 172.21.15.42:0/1415442644 >> v2:172.21.15.42:3300/0 conn(0x7fac300a1330 0x7fac300a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:50.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:50.724+0000 7fac3653b640 1 -- 172.21.15.42:0/1415442644 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fac300a1c50 msgr2=0x7fac30135810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T03:42:50.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:50.724+0000 7fac3653b640 1 -- 172.21.15.42:0/1415442644 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fac300a1c50 msgr2=0x7fac30135810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:42:50.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:50.724+0000 7fac3653b640 1 --2- 172.21.15.42:0/1415442644 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fac300a1c50 0x7fac30135810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:42:50.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:50.724+0000 7fac35d3a640 10 monclient(hunting): get_auth_request con 0x7fac300a1330 auth_method 0 2024-09-16T03:42:50.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:50.724+0000 7fac35d3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:50.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:50.724+0000 7fac35d3a640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:50.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:50.724+0000 7fac35d3a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:42:50.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:50.724+0000 7fac3653b640 1 --2- 172.21.15.42:0/1415442644 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fac300a1c50 0x7fac30135810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T03:42:50.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:50.724+0000 7fac35d3a640 1 -- 172.21.15.42:0/1415442644 >> v2:172.21.15.42:3300/0 conn(0x7fac300a1330 msgr2=0x7fac300a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T03:42:50.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:50.724+0000 7fac35d3a640 1 -- 172.21.15.42:0/1415442644 >> v2:172.21.15.42:3300/0 conn(0x7fac300a1330 msgr2=0x7fac300a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:42:50.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:50.724+0000 7fac35d3a640 1 --2- 172.21.15.42:0/1415442644 >> v2:172.21.15.42:3300/0 conn(0x7fac300a1330 0x7fac300a1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:42:50.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:50.724+0000 7fac35d3a640 1 --2- 172.21.15.42:0/1415442644 >> v2:172.21.15.42:3300/0 conn(0x7fac300a1330 0x7fac300a1710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T03:42:51.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:51.128+0000 7fac35539640 1 -- 172.21.15.42:0/1415442644 >> v1:172.21.15.154:6789/0 conn(0x7fac3006dac0 legacy=0x7fac30135100 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.154:6789/0 2024-09-16T03:42:51.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:51.128+0000 7fac3653b640 1 --2- 172.21.15.42:0/1415442644 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fac300a1c50 0x7fac30135810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:51.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:51.128+0000 7fac35d3a640 1 --2- 172.21.15.42:0/1415442644 >> v2:172.21.15.42:3300/0 conn(0x7fac300a1330 0x7fac300a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:51.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:51.128+0000 7fac3653b640 10 monclient(hunting): get_auth_request con 0x7fac300a1c50 auth_method 0 2024-09-16T03:42:51.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:51.128+0000 7fac3653b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:51.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:51.128+0000 7fac3653b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:51.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:51.128+0000 7fac3653b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:42:51.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:51.128+0000 7fac35d3a640 10 monclient(hunting): get_auth_request con 0x7fac300a1330 auth_method 0 2024-09-16T03:42:51.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:51.128+0000 7fac35d3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:51.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:51.128+0000 7fac35d3a640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:51.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:51.128+0000 7fac35d3a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:42:51.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:51.128+0000 7fac3653b640 1 -- 172.21.15.42:0/1415442644 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fac300a1c50 msgr2=0x7fac30135810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T03:42:51.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:51.128+0000 7fac35d3a640 1 -- 172.21.15.42:0/1415442644 >> v2:172.21.15.42:3300/0 conn(0x7fac300a1330 msgr2=0x7fac300a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T03:42:51.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:51.128+0000 7fac3653b640 1 -- 172.21.15.42:0/1415442644 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fac300a1c50 msgr2=0x7fac30135810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:42:51.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:51.128+0000 7fac35d3a640 1 -- 172.21.15.42:0/1415442644 >> v2:172.21.15.42:3300/0 conn(0x7fac300a1330 msgr2=0x7fac300a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:42:51.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:51.128+0000 7fac3653b640 1 --2- 172.21.15.42:0/1415442644 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fac300a1c50 0x7fac30135810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:42:51.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:51.128+0000 7fac35d3a640 1 --2- 172.21.15.42:0/1415442644 >> v2:172.21.15.42:3300/0 conn(0x7fac300a1330 0x7fac300a1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:42:51.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:51.128+0000 7fac35d3a640 1 --2- 172.21.15.42:0/1415442644 >> v2:172.21.15.42:3300/0 conn(0x7fac300a1330 0x7fac300a1710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T03:42:51.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:51.128+0000 7fac3653b640 1 --2- 172.21.15.42:0/1415442644 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fac300a1c50 0x7fac30135810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T03:42:51.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:51.932+0000 7fac35539640 1 -- 172.21.15.42:0/1415442644 >> v1:172.21.15.154:6789/0 conn(0x7fac3006dac0 legacy=0x7fac30135100 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.154:6789/0 2024-09-16T03:42:51.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:51.932+0000 7fac35d3a640 1 --2- 172.21.15.42:0/1415442644 >> v2:172.21.15.42:3300/0 conn(0x7fac300a1330 0x7fac300a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:51.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:51.932+0000 7fac35d3a640 10 monclient(hunting): get_auth_request con 0x7fac300a1330 auth_method 0 2024-09-16T03:42:51.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:51.932+0000 7fac35d3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:51.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:51.932+0000 7fac35d3a640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:51.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:51.932+0000 7fac35d3a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:42:51.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:51.932+0000 7fac35d3a640 1 -- 172.21.15.42:0/1415442644 >> v2:172.21.15.42:3300/0 conn(0x7fac300a1330 msgr2=0x7fac300a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T03:42:51.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:51.932+0000 7fac35d3a640 1 -- 172.21.15.42:0/1415442644 >> v2:172.21.15.42:3300/0 conn(0x7fac300a1330 msgr2=0x7fac300a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:42:51.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:51.932+0000 7fac35d3a640 1 --2- 172.21.15.42:0/1415442644 >> v2:172.21.15.42:3300/0 conn(0x7fac300a1330 0x7fac300a1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:42:51.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:51.932+0000 7fac3653b640 1 --2- 172.21.15.42:0/1415442644 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fac300a1c50 0x7fac30135810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:51.938 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:51.932+0000 7fac35d3a640 1 --2- 172.21.15.42:0/1415442644 >> v2:172.21.15.42:3300/0 conn(0x7fac300a1330 0x7fac300a1710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T03:42:51.938 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:51.932+0000 7fac3653b640 10 monclient(hunting): get_auth_request con 0x7fac300a1c50 auth_method 0 2024-09-16T03:42:51.938 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:51.932+0000 7fac3653b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:51.938 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:51.932+0000 7fac3653b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:51.938 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:51.932+0000 7fac3653b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:42:51.938 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:51.932+0000 7fac3653b640 1 -- 172.21.15.42:0/1415442644 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fac300a1c50 msgr2=0x7fac30135810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T03:42:51.938 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:51.932+0000 7fac3653b640 1 -- 172.21.15.42:0/1415442644 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fac300a1c50 msgr2=0x7fac30135810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:42:51.938 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:51.932+0000 7fac3653b640 1 --2- 172.21.15.42:0/1415442644 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fac300a1c50 0x7fac30135810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:42:51.938 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:51.932+0000 7fac3653b640 1 --2- 172.21.15.42:0/1415442644 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fac300a1c50 0x7fac30135810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T03:42:53.528 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:53.520+0000 7fac237fe640 10 monclient(hunting): tick 2024-09-16T03:42:53.529 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:53.520+0000 7fac237fe640 10 monclient(hunting): _check_auth_tickets 2024-09-16T03:42:53.529 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:53.520+0000 7fac237fe640 1 monclient(hunting): continuing hunt 2024-09-16T03:42:53.529 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:53.520+0000 7fac237fe640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T03:42:53.529 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:53.520+0000 7fac237fe640 1 -- 172.21.15.42:0/1415442644 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fac300a1c50 msgr2=0x7fac30135810 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:42:53.529 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:53.520+0000 7fac237fe640 1 --2- 172.21.15.42:0/1415442644 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fac300a1c50 0x7fac30135810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:53.530 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:53.520+0000 7fac237fe640 1 -- 172.21.15.42:0/1415442644 >> v2:172.21.15.42:3300/0 conn(0x7fac300a1330 msgr2=0x7fac300a1710 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:42:53.530 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:53.520+0000 7fac237fe640 1 --2- 172.21.15.42:0/1415442644 >> v2:172.21.15.42:3300/0 conn(0x7fac300a1330 0x7fac300a1710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:53.530 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:53.520+0000 7fac237fe640 1 -- 172.21.15.42:0/1415442644 >> v1:172.21.15.154:6789/0 conn(0x7fac3006dac0 legacy=0x7fac30135100 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:42:53.530 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:53.520+0000 7fac237fe640 10 monclient: _add_conns ranks=[0,4] 2024-09-16T03:42:53.530 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:53.520+0000 7fac237fe640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fac14007ae0 addr v1:172.21.15.42:6789/0 2024-09-16T03:42:53.531 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:53.520+0000 7fac237fe640 1 --2- 172.21.15.42:0/1415442644 >> v2:172.21.15.154:3300/0 conn(0x7fac1400b790 0x7fac1400dc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:53.531 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:53.520+0000 7fac237fe640 10 monclient(hunting): picked mon.noname-c con 0x7fac1400b790 addr v2:172.21.15.154:3300/0 2024-09-16T03:42:53.531 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:53.520+0000 7fac237fe640 1 -- 172.21.15.42:0/1415442644 --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fac3009fbd0 con 0x7fac14007ae0 2024-09-16T03:42:53.531 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:53.520+0000 7fac237fe640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:53.532 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:53.520+0000 7fac237fe640 1 -- 172.21.15.42:0/1415442644 --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fac3009f610 con 0x7fac1400b790 2024-09-16T03:42:53.532 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:53.520+0000 7fac237fe640 10 monclient(hunting): _renew_subs 2024-09-16T03:42:53.532 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:53.520+0000 7fac35539640 1 -- 172.21.15.42:0/1415442644 >> v2:172.21.15.154:3300/0 conn(0x7fac1400b790 msgr2=0x7fac1400dc10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T03:42:53.532 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:53.520+0000 7fac35539640 1 --2- 172.21.15.42:0/1415442644 >> v2:172.21.15.154:3300/0 conn(0x7fac1400b790 0x7fac1400dc10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:42:53.728 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:53.724+0000 7fac35539640 1 -- 172.21.15.42:0/1415442644 >> v2:172.21.15.154:3300/0 conn(0x7fac1400b790 msgr2=0x7fac1400dc10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T03:42:53.728 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:53.724+0000 7fac35539640 1 --2- 172.21.15.42:0/1415442644 >> v2:172.21.15.154:3300/0 conn(0x7fac1400b790 0x7fac1400dc10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T03:42:54.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:54.128+0000 7fac35539640 1 -- 172.21.15.42:0/1415442644 >> v2:172.21.15.154:3300/0 conn(0x7fac1400b790 msgr2=0x7fac1400dc10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T03:42:54.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:54.128+0000 7fac35539640 1 --2- 172.21.15.42:0/1415442644 >> v2:172.21.15.154:3300/0 conn(0x7fac1400b790 0x7fac1400dc10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T03:42:54.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:54.932+0000 7fac35539640 1 -- 172.21.15.42:0/1415442644 >> v2:172.21.15.154:3300/0 conn(0x7fac1400b790 msgr2=0x7fac1400dc10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T03:42:54.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:54.932+0000 7fac35539640 1 --2- 172.21.15.42:0/1415442644 >> v2:172.21.15.154:3300/0 conn(0x7fac1400b790 0x7fac1400dc10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T03:42:55.448 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.440+0000 7fac34d38640 1 -- 172.21.15.42:0/1415442644 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1430387747 0 0) 0x7fac3009fbd0 con 0x7fac14007ae0 2024-09-16T03:42:55.448 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.440+0000 7fac34d38640 10 monclient(hunting): _negotiate 2024-09-16T03:42:55.449 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.440+0000 7fac34d38640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:55.449 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.440+0000 7fac34d38640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:55.449 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.440+0000 7fac34d38640 10 monclient(hunting): my global_id is 4431 2024-09-16T03:42:55.449 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.440+0000 7fac34d38640 1 -- 172.21.15.42:0/1415442644 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fac1c003530 con 0x7fac14007ae0 2024-09-16T03:42:55.454 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.444+0000 7fac34d38640 1 -- 172.21.15.42:0/1415442644 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2641394692 0 0) 0x7fac1c003530 con 0x7fac14007ae0 2024-09-16T03:42:55.454 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.444+0000 7fac34d38640 1 -- 172.21.15.42:0/1415442644 >> v2:172.21.15.154:3300/0 conn(0x7fac1400b790 msgr2=0x7fac1400dc10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:42:55.454 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.444+0000 7fac34d38640 1 --2- 172.21.15.42:0/1415442644 >> v2:172.21.15.154:3300/0 conn(0x7fac1400b790 0x7fac1400dc10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:55.454 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.444+0000 7fac34d38640 10 monclient: _finish_hunting 0 2024-09-16T03:42:55.454 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.444+0000 7fac34d38640 1 monclient: found mon.noname-a-legacy 2024-09-16T03:42:55.454 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.444+0000 7fac34d38640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:42:55.454 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.444+0000 7fac34d38640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.42:6789/0 2024-09-16T03:42:55.454 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.444+0000 7fac34d38640 1 -- 172.21.15.42:0/1415442644 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac3009c090 con 0x7fac14007ae0 2024-09-16T03:42:55.454 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.444+0000 7fac34d38640 10 monclient: _finish_auth 0 2024-09-16T03:42:55.454 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.444+0000 7fac34d38640 10 monclient: _check_auth_tickets 2024-09-16T03:42:55.454 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.444+0000 7fac34d38640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:42:55.454 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.444+0000 7fac34d38640 1 -- 172.21.15.42:0/1415442644 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2177413734 0 0) 0x7fac2400f040 con 0x7fac14007ae0 2024-09-16T03:42:55.455 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.444+0000 7fac34d38640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:55.455 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.444+0000 7fac34d38640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T03:42:55.455 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.444+0000 7fac34d38640 10 monclient: dump: 2024-09-16T03:42:55.455 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:55.455 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:55.455 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:55.455 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:55.455 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:55.455 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:55.455 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:55.455 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:55.455 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:55.455 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:55.456 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.444+0000 7fac34d38640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] but i'm connected to v1:172.21.15.42:6789/0, reconnecting 2024-09-16T03:42:55.456 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.444+0000 7fac34d38640 10 monclient: _reopen_session rank -1 2024-09-16T03:42:55.456 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.444+0000 7fac34d38640 1 -- 172.21.15.42:0/1415442644 >> v1:172.21.15.42:6789/0 conn(0x7fac14007ae0 legacy=0x7fac14009f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:55.456 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.444+0000 7fac3653b640 1 -- 172.21.15.42:0/1415442644 reap_dead start 2024-09-16T03:42:55.456 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.444+0000 7fac34d38640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:42:55.456 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.444+0000 7fac34d38640 1 --2- 172.21.15.42:0/1415442644 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fac1c0039c0 0x7fac1c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:55.456 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.444+0000 7fac34d38640 10 monclient(hunting): picked mon.b con 0x7fac1c0039c0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:42:55.456 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.444+0000 7fac34d38640 1 --2- 172.21.15.42:0/1415442644 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fac1c004810 0x7fac1c004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:55.456 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.444+0000 7fac34d38640 10 monclient(hunting): picked mon.a con 0x7fac1c004810 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:42:55.456 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.444+0000 7fac34d38640 1 --2- 172.21.15.42:0/1415442644 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fac1c0052f0 0x7fac1c00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:55.456 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.444+0000 7fac34d38640 10 monclient(hunting): picked mon.c con 0x7fac1c0052f0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:42:55.456 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.444+0000 7fac34d38640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:55.456 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.444+0000 7fac34d38640 1 -- 172.21.15.42:0/1415442644 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fac3009f610 con 0x7fac1c004810 2024-09-16T03:42:55.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.444+0000 7fac34d38640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:55.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.444+0000 7fac34d38640 1 -- 172.21.15.42:0/1415442644 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fac1c00c240 con 0x7fac1c0039c0 2024-09-16T03:42:55.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.444+0000 7fac34d38640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:55.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.444+0000 7fac34d38640 1 -- 172.21.15.42:0/1415442644 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fac1c00c440 con 0x7fac1c0052f0 2024-09-16T03:42:55.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.444+0000 7fac34d38640 10 monclient(hunting): _renew_subs 2024-09-16T03:42:55.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.444+0000 7fac34d38640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:42:55.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.444+0000 7fac35d3a640 1 -- 172.21.15.42:0/1415442644 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fac1c0039c0 msgr2=0x7fac1c003dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T03:42:55.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.444+0000 7fac35539640 1 --2- 172.21.15.42:0/1415442644 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fac1c004810 0x7fac1c004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:55.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.444+0000 7fac3653b640 1 --2- 172.21.15.42:0/1415442644 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fac1c0052f0 0x7fac1c00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:55.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.444+0000 7fac35d3a640 1 --2- 172.21.15.42:0/1415442644 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fac1c0039c0 0x7fac1c003dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:42:55.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.444+0000 7fac3653b640 10 monclient(hunting): get_auth_request con 0x7fac1c0052f0 auth_method 0 2024-09-16T03:42:55.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.444+0000 7fac3653b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:55.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.444+0000 7fac3653b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:55.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.444+0000 7fac3653b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:42:55.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.444+0000 7fac35539640 10 monclient(hunting): get_auth_request con 0x7fac1c004810 auth_method 0 2024-09-16T03:42:55.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.448+0000 7fac35539640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:55.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.448+0000 7fac35539640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:55.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.448+0000 7fac35539640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:42:55.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.448+0000 7fac3653b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:55.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.448+0000 7fac3653b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:55.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.448+0000 7fac3653b640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T03:42:55.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.448+0000 7fac35539640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:55.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.448+0000 7fac35539640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:55.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.448+0000 7fac35539640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T03:42:55.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.448+0000 7fac35539640 10 monclient(hunting): handle_auth_done global_id 4431 payload 293 2024-09-16T03:42:55.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.448+0000 7fac35539640 1 -- 172.21.15.42:0/1415442644 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fac1c0052f0 msgr2=0x7fac1c00b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:42:55.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.448+0000 7fac35539640 1 --2- 172.21.15.42:0/1415442644 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fac1c0052f0 0x7fac1c00b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:55.460 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.448+0000 7fac35539640 1 -- 172.21.15.42:0/1415442644 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fac1c0039c0 msgr2=0x7fac1c003dc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:42:55.460 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.448+0000 7fac35539640 1 --2- 172.21.15.42:0/1415442644 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fac1c0039c0 0x7fac1c003dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:55.460 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.448+0000 7fac35539640 10 monclient: _finish_hunting 0 2024-09-16T03:42:55.460 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.448+0000 7fac35539640 1 monclient: found mon.a 2024-09-16T03:42:55.460 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.448+0000 7fac35539640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:42:55.460 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.448+0000 7fac35539640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:42:55.460 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.448+0000 7fac35539640 1 -- 172.21.15.42:0/1415442644 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac1c003530 con 0x7fac1c004810 2024-09-16T03:42:55.460 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.448+0000 7fac35539640 10 monclient: _finish_auth 0 2024-09-16T03:42:55.460 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.448+0000 7fac35539640 10 monclient: _check_auth_tickets 2024-09-16T03:42:55.460 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.448+0000 7fac35539640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:42:55.460 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.448+0000 7fac37f9e640 5 monclient: authenticate success, global_id 4431 2024-09-16T03:42:55.460 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.448+0000 7fac37f9e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:42:55.460 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.448+0000 7fac3653b640 1 --2- 172.21.15.42:0/1415442644 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fac1c0052f0 0x7fac1c00b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:42:55.461 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.452+0000 7fac35539640 1 --2- 172.21.15.42:0/1415442644 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fac1c004810 0x7fac1c004c90 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fac2c003cc0 tx=0x7fac2c01ce90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed0b6c13dd9f3f11 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:55.461 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.452+0000 7fac34d38640 1 -- 172.21.15.42:0/1415442644 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac2c007d40 con 0x7fac1c004810 2024-09-16T03:42:55.462 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.452+0000 7fac34d38640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:55.462 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.452+0000 7fac34d38640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:42:55.462 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.452+0000 7fac34d38640 10 monclient: dump: 2024-09-16T03:42:55.462 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:55.462 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:55.462 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:55.462 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:55.462 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:55.462 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:55.462 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:55.462 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:55.462 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:55.463 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:55.463 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.452+0000 7fac34d38640 1 -- 172.21.15.42:0/1415442644 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac2c0a9070 con 0x7fac1c004810 2024-09-16T03:42:55.463 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.452+0000 7fac37f9e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:42:55.463 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.452+0000 7fac34d38640 10 monclient: handle_config config(0 keys) 2024-09-16T03:42:55.463 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.452+0000 7fac37f9e640 10 monclient: get_monmap_and_config success 2024-09-16T03:42:55.463 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.452+0000 7fac37f9e640 4 set_mon_vals no callback set 2024-09-16T03:42:55.463 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.452+0000 7fac34d38640 1 -- 172.21.15.42:0/1415442644 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac2c0a5cc0 con 0x7fac1c004810 2024-09-16T03:42:55.463 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.452+0000 7fac34d38640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:55.463 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.452+0000 7fac34d38640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:42:55.463 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.452+0000 7fac34d38640 10 monclient: dump: 2024-09-16T03:42:55.463 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:55.463 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:55.463 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:55.463 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:55.464 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:55.464 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:55.464 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:55.464 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:55.464 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:55.464 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:55.465 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.456+0000 7fac37f9e640 10 monclient: shutdown 2024-09-16T03:42:55.465 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.456+0000 7fac37f9e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:42:55.465 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.456+0000 7fac37f9e640 1 -- 172.21.15.42:0/1415442644 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fac1c004810 msgr2=0x7fac1c004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:55.466 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.456+0000 7fac37f9e640 1 --2- 172.21.15.42:0/1415442644 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fac1c004810 0x7fac1c004c90 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fac2c003cc0 tx=0x7fac2c01ce90 comp rx=0 tx=0).stop 2024-09-16T03:42:55.466 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.456+0000 7fac37f9e640 1 -- 172.21.15.42:0/1415442644 shutdown_connections 2024-09-16T03:42:55.466 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.456+0000 7fac37f9e640 1 --2- 172.21.15.42:0/1415442644 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fac1c0052f0 0x7fac1c00b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:55.466 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.456+0000 7fac37f9e640 1 --2- 172.21.15.42:0/1415442644 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fac1c004810 0x7fac1c004c90 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:55.466 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.456+0000 7fac37f9e640 1 --2- 172.21.15.42:0/1415442644 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fac1c0039c0 0x7fac1c003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:55.466 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.456+0000 7fac37f9e640 1 -- 172.21.15.42:0/1415442644 >> 172.21.15.42:0/1415442644 conn(0x7fac300a2420 msgr2=0x7fac300a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:42:55.466 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.456+0000 7fac37f9e640 1 -- 172.21.15.42:0/1415442644 shutdown_connections 2024-09-16T03:42:55.466 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.456+0000 7fac37f9e640 1 -- 172.21.15.42:0/1415442644 wait complete. 2024-09-16T03:42:55.466 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.456+0000 7fac37f9e640 10 monclient: build_initial_monmap 2024-09-16T03:42:55.466 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.456+0000 7fac37f9e640 10 monclient: monmap: 2024-09-16T03:42:55.466 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:42:55.466 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:42:55.466 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:42:55.467 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:42:55.467 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:42:55.467 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:55.467 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:42:55.467 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:42:55.467 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:42:55.467 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:55.467 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.456+0000 7fac37f9e640 1 librados: starting msgr at 2024-09-16T03:42:55.467 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.456+0000 7fac37f9e640 1 librados: starting objecter 2024-09-16T03:42:55.467 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.456+0000 7fac37f9e640 1 Processor -- start 2024-09-16T03:42:55.467 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.456+0000 7fac37f9e640 1 -- start start 2024-09-16T03:42:55.467 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.456+0000 7fac37f9e640 1 librados: setting wanted keys 2024-09-16T03:42:55.467 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.456+0000 7fac37f9e640 1 librados: calling monclient init 2024-09-16T03:42:55.468 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.456+0000 7fac37f9e640 10 monclient: init 2024-09-16T03:42:55.471 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.460+0000 7fac37f9e640 10 monclient: _reopen_session rank -1 2024-09-16T03:42:55.471 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.460+0000 7fac37f9e640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:42:55.471 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.460+0000 7fac37f9e640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fac1c0039c0 0x7fac301cab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:55.471 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.460+0000 7fac37f9e640 10 monclient(hunting): picked mon.noname-a con 0x7fac1c0039c0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:42:55.471 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.460+0000 7fac37f9e640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fac1c004810 0x7fac301346e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:55.471 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.460+0000 7fac37f9e640 10 monclient(hunting): picked mon.noname-c con 0x7fac1c004810 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:42:55.471 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.460+0000 7fac37f9e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fac1c0052f0 0x7fac3011f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:55.471 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.460+0000 7fac37f9e640 10 monclient(hunting): picked mon.noname-b con 0x7fac1c0052f0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:42:55.471 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.460+0000 7fac37f9e640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:55.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.460+0000 7fac37f9e640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fac30135ca0 con 0x7fac1c0039c0 2024-09-16T03:42:55.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.460+0000 7fac37f9e640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:55.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.460+0000 7fac37f9e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fac30135b20 con 0x7fac1c0052f0 2024-09-16T03:42:55.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.460+0000 7fac37f9e640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:55.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.460+0000 7fac37f9e640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fac30135e20 con 0x7fac1c004810 2024-09-16T03:42:55.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.460+0000 7fac37f9e640 10 monclient(hunting): _renew_subs 2024-09-16T03:42:55.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.460+0000 7fac37f9e640 10 monclient(hunting): authenticate will time out at 1022.432270s 2024-09-16T03:42:55.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.460+0000 7fac3653b640 1 -- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fac1c0052f0 msgr2=0x7fac3011f2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T03:42:55.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.460+0000 7fac35d3a640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fac1c0039c0 0x7fac301cab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:55.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.460+0000 7fac3653b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fac1c0052f0 0x7fac3011f2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:42:55.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.460+0000 7fac35d3a640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fac1c0039c0 0x7fac301cab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:53074/0 (socket says 172.21.15.42:53074) 2024-09-16T03:42:55.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.460+0000 7fac35539640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fac1c004810 0x7fac301346e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:55.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.460+0000 7fac35d3a640 1 -- 172.21.15.42:0/3868463619 learned_addr learned my addr 172.21.15.42:0/3868463619 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:42:55.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.460+0000 7fac35d3a640 10 monclient(hunting): get_auth_request con 0x7fac1c0039c0 auth_method 0 2024-09-16T03:42:55.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.460+0000 7fac35d3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:55.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.460+0000 7fac35d3a640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:55.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.460+0000 7fac35d3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:55.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.460+0000 7fac35539640 10 monclient(hunting): get_auth_request con 0x7fac1c004810 auth_method 0 2024-09-16T03:42:55.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.460+0000 7fac35539640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:55.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.460+0000 7fac35539640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:55.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.460+0000 7fac35539640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:55.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.460+0000 7fac35d3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:55.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.460+0000 7fac35d3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:55.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.460+0000 7fac35d3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:55.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.460+0000 7fac35539640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:55.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.460+0000 7fac35539640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:55.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.460+0000 7fac35539640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:55.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.460+0000 7fac35539640 10 monclient(hunting): handle_auth_done global_id 4370 payload 824 2024-09-16T03:42:55.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.460+0000 7fac35539640 1 -- 172.21.15.42:0/3868463619 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fac1c0052f0 msgr2=0x7fac3011f2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:42:55.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.460+0000 7fac35539640 1 --2- 172.21.15.42:0/3868463619 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fac1c0052f0 0x7fac3011f2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:55.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.460+0000 7fac35539640 1 -- 172.21.15.42:0/3868463619 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fac1c0039c0 msgr2=0x7fac301cab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:55.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.460+0000 7fac35539640 1 --2- 172.21.15.42:0/3868463619 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fac1c0039c0 0x7fac301cab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:55.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.460+0000 7fac35539640 10 monclient: _finish_hunting 0 2024-09-16T03:42:55.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.460+0000 7fac35539640 1 monclient: found mon.noname-c 2024-09-16T03:42:55.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.460+0000 7fac35539640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:42:55.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.460+0000 7fac35539640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:42:55.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.460+0000 7fac35539640 1 -- 172.21.15.42:0/3868463619 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac30122df0 con 0x7fac1c004810 2024-09-16T03:42:55.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.460+0000 7fac35539640 1 --2- 172.21.15.42:0/3868463619 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fac1c004810 0x7fac301346e0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fac2c0aa000 tx=0x7fac1c005840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:55.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.460+0000 7fac227fc640 10 client.?.objecter ms_handle_connect 0x7fac1c004810 2024-09-16T03:42:55.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.460+0000 7fac227fc640 10 client.?.objecter resend_mon_ops 2024-09-16T03:42:55.475 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.460+0000 7fac227fc640 1 -- 172.21.15.42:0/3868463619 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac2c00a0d0 con 0x7fac1c004810 2024-09-16T03:42:55.475 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.460+0000 7fac227fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:55.475 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.460+0000 7fac227fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:42:55.475 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.460+0000 7fac227fc640 10 monclient: dump: 2024-09-16T03:42:55.475 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:55.475 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:55.475 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:55.475 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:55.475 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:55.475 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:55.475 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:55.475 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:55.475 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:55.476 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:55.476 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.460+0000 7fac227fc640 10 monclient: _finish_auth 0 2024-09-16T03:42:55.476 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.460+0000 7fac227fc640 10 monclient: _check_auth_tickets 2024-09-16T03:42:55.476 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.460+0000 7fac227fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:42:55.476 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.460+0000 7fac227fc640 1 -- 172.21.15.42:0/3868463619 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac2c0ae470 con 0x7fac1c004810 2024-09-16T03:42:55.476 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.460+0000 7fac227fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:42:55.476 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.460+0000 7fac227fc640 1 -- 172.21.15.42:0/3868463619 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac2c0ae860 con 0x7fac1c004810 2024-09-16T03:42:55.476 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.460+0000 7fac37f9e640 5 monclient: authenticate success, global_id 4370 2024-09-16T03:42:55.476 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.460+0000 7fac37f9e640 10 monclient: _renew_subs 2024-09-16T03:42:55.476 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.460+0000 7fac37f9e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:42:55.476 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.460+0000 7fac23fff640 4 set_mon_vals no callback set 2024-09-16T03:42:55.476 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.460+0000 7fac37f9e640 1 -- 172.21.15.42:0/3868463619 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac301bb240 con 0x7fac1c004810 2024-09-16T03:42:55.476 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.460+0000 7fac227fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:55.476 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.460+0000 7fac227fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:42:55.477 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.460+0000 7fac227fc640 10 monclient: dump: 2024-09-16T03:42:55.477 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:55.477 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:55.477 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:55.477 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:55.477 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:55.477 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:55.477 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:55.477 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:55.477 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:55.477 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:55.477 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.460+0000 7fac37f9e640 10 client.4370.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:42:55.477 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.460+0000 7fac37f9e640 10 monclient: _renew_subs 2024-09-16T03:42:55.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.460+0000 7fac37f9e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:42:55.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.460+0000 7fac37f9e640 1 -- 172.21.15.42:0/3868463619 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fac30122b10 con 0x7fac1c004810 2024-09-16T03:42:55.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.460+0000 7fac37f9e640 1 librados: init done 2024-09-16T03:42:55.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.460+0000 7fac227fc640 1 -- 172.21.15.42:0/3868463619 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fac2c0aea00 con 0x7fac1c004810 2024-09-16T03:42:55.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.460+0000 7fac227fc640 1 --2- 172.21.15.42:0/3868463619 >> v2:172.21.15.154:6812/254845342 conn(0x7fac1403c2f0 0x7fac1403e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:55.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.460+0000 7fac227fc640 1 -- 172.21.15.42:0/3868463619 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(19..19 src has 1..19) ==== 3534+0+0 (secure 0 0 0) 0x7fac2c0e2790 con 0x7fac1c004810 2024-09-16T03:42:55.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.460+0000 7fac227fc640 10 client.4370.objecter ms_dispatch 0x7fac30000ba0 osd_map(19..19 src has 1..19) 2024-09-16T03:42:55.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.460+0000 7fac227fc640 3 client.4370.objecter handle_osd_map got epochs [19,19] > 0 2024-09-16T03:42:55.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.460+0000 7fac227fc640 3 client.4370.objecter handle_osd_map decoding full epoch 19 2024-09-16T03:42:55.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.460+0000 7fac227fc640 20 client.4370.objecter dump_active .. 0 homeless 2024-09-16T03:42:55.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.460+0000 7fac37f9e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:42:55.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.460+0000 7fac37f9e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:42:55.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.460+0000 7fac37f9e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:42:55.479 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.460+0000 7fac37f9e640 1 -- 172.21.15.42:0/3868463619 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fac30136320 con 0x7fac1c004810 2024-09-16T03:42:55.479 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.460+0000 7fac35d3a640 1 --2- 172.21.15.42:0/3868463619 >> v2:172.21.15.154:6812/254845342 conn(0x7fac1403c2f0 0x7fac1403e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:55.479 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.460+0000 7fac35d3a640 10 monclient: get_auth_request con 0x7fac1403c2f0 auth_method 0 2024-09-16T03:42:55.479 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.460+0000 7fac35d3a640 1 --2- 172.21.15.42:0/3868463619 >> v2:172.21.15.154:6812/254845342 conn(0x7fac1403c2f0 0x7fac1403e7b0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fac2400c5c0 tx=0x7fac24008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:55.479 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.460+0000 7fac227fc640 10 client.4370.objecter ms_handle_connect 0x7fac1403c2f0 2024-09-16T03:42:55.479 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.468+0000 7fac227fc640 1 -- 172.21.15.42:0/3868463619 <== mon.2 v2:172.21.15.42:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7fac2c016ba0 con 0x7fac1c004810 2024-09-16T03:42:55.479 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.468+0000 7fac227fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:42:55.479 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.468+0000 7fac227fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:42:55.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.708+0000 7fac37f9e640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:42:55.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.708+0000 7fac37f9e640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:42:55.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.708+0000 7fac37f9e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:42:55.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.708+0000 7fac37f9e640 1 -- 172.21.15.42:0/3868463619 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fac30123030 con 0x7fac1c004810 2024-09-16T03:42:55.716 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.708+0000 7fac227fc640 1 -- 172.21.15.42:0/3868463619 <== mon.2 v2:172.21.15.42:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1554 (secure 0 0 0) 0x7fac2c0c7030 con 0x7fac1c004810 2024-09-16T03:42:55.716 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.708+0000 7fac227fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:42:55.716 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.708+0000 7fac227fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:42:55.717 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":12,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:42:55.719 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.712+0000 7fac37f9e640 10 librados: watch_flush enter 2024-09-16T03:42:55.719 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.712+0000 7fac37f9e640 10 librados: watch_flush exit 2024-09-16T03:42:55.719 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.712+0000 7fac37f9e640 20 client.4370.objecter shutdown clearing up homeless session... 2024-09-16T03:42:55.719 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.712+0000 7fac37f9e640 10 client.4370.objecter successfully canceled tick 2024-09-16T03:42:55.719 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.712+0000 7fac37f9e640 1 -- 172.21.15.42:0/3868463619 >> v2:172.21.15.154:6812/254845342 conn(0x7fac1403c2f0 msgr2=0x7fac1403e7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:55.719 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.712+0000 7fac37f9e640 1 --2- 172.21.15.42:0/3868463619 >> v2:172.21.15.154:6812/254845342 conn(0x7fac1403c2f0 0x7fac1403e7b0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fac2400c5c0 tx=0x7fac24008040 comp rx=0 tx=0).stop 2024-09-16T03:42:55.719 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.712+0000 7fac37f9e640 10 monclient: shutdown 2024-09-16T03:42:55.719 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.712+0000 7fac37f9e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:42:55.720 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.712+0000 7fac37f9e640 1 -- 172.21.15.42:0/3868463619 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fac1c004810 msgr2=0x7fac301346e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:55.720 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.712+0000 7fac37f9e640 1 --2- 172.21.15.42:0/3868463619 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fac1c004810 0x7fac301346e0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fac2c0aa000 tx=0x7fac1c005840 comp rx=0 tx=0).stop 2024-09-16T03:42:55.720 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.712+0000 7fac37f9e640 1 -- 172.21.15.42:0/3868463619 shutdown_connections 2024-09-16T03:42:55.720 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.712+0000 7fac37f9e640 1 --2- 172.21.15.42:0/3868463619 >> v2:172.21.15.154:6812/254845342 conn(0x7fac1403c2f0 0x7fac1403e7b0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:55.720 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.712+0000 7fac37f9e640 1 --2- 172.21.15.42:0/3868463619 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fac1c0052f0 0x7fac3011f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:55.720 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.712+0000 7fac37f9e640 1 --2- 172.21.15.42:0/3868463619 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fac1c004810 0x7fac301346e0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:55.720 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.712+0000 7fac37f9e640 1 --2- 172.21.15.42:0/3868463619 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fac1c0039c0 0x7fac301cab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:55.720 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.712+0000 7fac37f9e640 1 -- 172.21.15.42:0/3868463619 >> 172.21.15.42:0/3868463619 conn(0x7fac300a2420 msgr2=0x7fac300a45b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:42:55.720 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.712+0000 7fac37f9e640 1 -- 172.21.15.42:0/3868463619 shutdown_connections 2024-09-16T03:42:55.720 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.712+0000 7fac37f9e640 1 -- 172.21.15.42:0/3868463619 wait complete. 2024-09-16T03:42:55.720 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.712+0000 7fac37f9e640 1 librados: shutdown 2024-09-16T03:42:55.736 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T03:42:55.736 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-16T03:42:55.736 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T03:42:55.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.848+0000 7faf45073640 10 monclient: get_monmap_and_config 2024-09-16T03:42:55.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.848+0000 7faf45073640 10 monclient: build_initial_monmap 2024-09-16T03:42:55.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.848+0000 7faf45073640 10 monclient: monmap: 2024-09-16T03:42:55.855 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:42:55.855 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:55.855 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:42:55.852524+0000 2024-09-16T03:42:55.855 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:42:55.852524+0000 2024-09-16T03:42:55.855 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:42:55.855 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:55.855 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:42:55.855 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:42:55.855 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:42:55.855 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:42:55.856 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:42:55.856 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:55.860 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.848+0000 7faf45073640 1 Processor -- start 2024-09-16T03:42:55.860 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.848+0000 7faf45073640 1 -- start start 2024-09-16T03:42:55.860 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.848+0000 7faf45073640 10 monclient: init 2024-09-16T03:42:55.860 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.848+0000 7faf45073640 10 monclient: _reopen_session rank -1 2024-09-16T03:42:55.860 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.848+0000 7faf45073640 10 monclient: _add_conns ranks=[4,3,1,0,2] 2024-09-16T03:42:55.860 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.848+0000 7faf45073640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7faf400a1330 0x7faf400a1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:55.860 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.848+0000 7faf45073640 10 monclient(hunting): picked mon.noname-c con 0x7faf400a1330 addr v2:172.21.15.154:3300/0 2024-09-16T03:42:55.860 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.848+0000 7faf45073640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7faf4006dac0 0x7faf400a1c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:55.860 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.848+0000 7faf45073640 10 monclient(hunting): picked mon.noname-a con 0x7faf4006dac0 addr v2:172.21.15.42:3300/0 2024-09-16T03:42:55.860 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.848+0000 7faf45073640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7faf40135100 addr v1:172.21.15.154:6789/0 2024-09-16T03:42:55.861 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.848+0000 7faf45073640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faf4009c7d0 con 0x7faf40135100 2024-09-16T03:42:55.861 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.848+0000 7faf45073640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:55.861 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.848+0000 7faf45073640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7faf4009c090 con 0x7faf4006dac0 2024-09-16T03:42:55.861 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.848+0000 7faf45073640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:55.861 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.848+0000 7faf45073640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7faf4009f610 con 0x7faf400a1330 2024-09-16T03:42:55.861 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.848+0000 7faf45073640 10 monclient(hunting): _renew_subs 2024-09-16T03:42:55.861 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.848+0000 7faf45073640 10 monclient(hunting): authenticate will time out at 1022.821985s 2024-09-16T03:42:55.861 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.848+0000 7faf3ed9d640 1 -- >> v2:172.21.15.154:3300/0 conn(0x7faf400a1330 msgr2=0x7faf400a1710 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T03:42:55.861 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.848+0000 7faf3ed9d640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7faf400a1330 0x7faf400a1710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:42:55.861 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.848+0000 7faf3f59e640 1 -- >> v1:172.21.15.154:6789/0 conn(0x7faf40135100 legacy=0x7faf40135550 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.154:6789/0 2024-09-16T03:42:55.862 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.848+0000 7faf3e59c640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7faf4006dac0 0x7faf400a1c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:55.862 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.848+0000 7faf3e59c640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7faf4006dac0 0x7faf400a1c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:53078/0 (socket says 172.21.15.42:53078) 2024-09-16T03:42:55.862 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.848+0000 7faf3e59c640 1 -- 172.21.15.42:0/2402219405 learned_addr learned my addr 172.21.15.42:0/2402219405 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:42:55.862 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.848+0000 7faf3e59c640 10 monclient(hunting): get_auth_request con 0x7faf4006dac0 auth_method 0 2024-09-16T03:42:55.862 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.848+0000 7faf3e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:55.862 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.848+0000 7faf3e59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:55.862 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.848+0000 7faf3e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:55.862 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.848+0000 7faf3e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:55.862 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.848+0000 7faf3e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:55.862 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.848+0000 7faf3e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:55.862 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.848+0000 7faf3e59c640 10 monclient(hunting): handle_auth_done global_id 4437 payload 274 2024-09-16T03:42:55.862 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.848+0000 7faf3e59c640 1 -- 172.21.15.42:0/2402219405 >> v2:172.21.15.154:3300/0 conn(0x7faf400a1330 msgr2=0x7faf400a1710 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:42:55.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.848+0000 7faf3e59c640 1 --2- 172.21.15.42:0/2402219405 >> v2:172.21.15.154:3300/0 conn(0x7faf400a1330 0x7faf400a1710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:55.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.848+0000 7faf3e59c640 1 -- 172.21.15.42:0/2402219405 >> v1:172.21.15.154:6789/0 conn(0x7faf40135100 legacy=0x7faf40135550 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:42:55.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.848+0000 7faf3e59c640 10 monclient: _finish_hunting 0 2024-09-16T03:42:55.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.848+0000 7faf3e59c640 1 monclient: found mon.noname-a 2024-09-16T03:42:55.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.848+0000 7faf3e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:42:55.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.848+0000 7faf3e59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:42:55.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.848+0000 7faf3e59c640 1 -- 172.21.15.42:0/2402219405 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf4009f8f0 con 0x7faf4006dac0 2024-09-16T03:42:55.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.852+0000 7faf3e59c640 1 --2- 172.21.15.42:0/2402219405 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf4006dac0 0x7faf400a1c50 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7faf4006add0 tx=0x7faf3401c980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=957e2fa9c8e02a07 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:55.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.852+0000 7faf3dd9b640 1 -- 172.21.15.42:0/2402219405 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf340a5860 con 0x7faf4006dac0 2024-09-16T03:42:55.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.852+0000 7faf3dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:55.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.852+0000 7faf3dd9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:42:55.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.852+0000 7faf3dd9b640 10 monclient: dump: 2024-09-16T03:42:55.863 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:55.863 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:55.864 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:55.864 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:55.864 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:55.864 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:55.864 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:55.864 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:55.864 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:55.864 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:55.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.852+0000 7faf3dd9b640 10 monclient: _finish_auth 0 2024-09-16T03:42:55.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.852+0000 7faf3dd9b640 10 monclient: _check_auth_tickets 2024-09-16T03:42:55.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.852+0000 7faf3dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:42:55.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.852+0000 7faf3dd9b640 1 -- 172.21.15.42:0/2402219405 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faf340a5a00 con 0x7faf4006dac0 2024-09-16T03:42:55.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.852+0000 7faf3dd9b640 10 monclient: handle_config config(0 keys) 2024-09-16T03:42:55.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.852+0000 7faf3dd9b640 1 -- 172.21.15.42:0/2402219405 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf340a5dd0 con 0x7faf4006dac0 2024-09-16T03:42:55.865 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.852+0000 7faf3dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:55.865 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.852+0000 7faf3dd9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:42:55.865 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.852+0000 7faf3dd9b640 10 monclient: dump: 2024-09-16T03:42:55.865 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:55.865 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:55.865 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:55.865 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:55.865 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:55.865 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:55.865 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:55.865 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:55.865 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:55.865 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:55.865 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.852+0000 7faf45073640 5 monclient: authenticate success, global_id 4437 2024-09-16T03:42:55.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.852+0000 7faf45073640 10 monclient: get_monmap_and_config success 2024-09-16T03:42:55.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.852+0000 7faf45073640 4 set_mon_vals no callback set 2024-09-16T03:42:55.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.852+0000 7faf45073640 10 monclient: shutdown 2024-09-16T03:42:55.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.852+0000 7faf45073640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:42:55.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.852+0000 7faf45073640 1 -- 172.21.15.42:0/2402219405 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf4006dac0 msgr2=0x7faf400a1c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:55.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.852+0000 7faf45073640 1 --2- 172.21.15.42:0/2402219405 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf4006dac0 0x7faf400a1c50 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7faf4006add0 tx=0x7faf3401c980 comp rx=0 tx=0).stop 2024-09-16T03:42:55.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.852+0000 7faf45073640 1 -- 172.21.15.42:0/2402219405 shutdown_connections 2024-09-16T03:42:55.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.852+0000 7faf45073640 1 --2- 172.21.15.42:0/2402219405 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf4006dac0 0x7faf400a1c50 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:55.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.852+0000 7faf45073640 1 --2- 172.21.15.42:0/2402219405 >> v2:172.21.15.154:3300/0 conn(0x7faf400a1330 0x7faf400a1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:55.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.852+0000 7faf45073640 1 -- 172.21.15.42:0/2402219405 >> 172.21.15.42:0/2402219405 conn(0x7faf400a2420 msgr2=0x7faf400a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:42:55.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.852+0000 7faf45073640 1 -- 172.21.15.42:0/2402219405 shutdown_connections 2024-09-16T03:42:55.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.852+0000 7faf45073640 1 -- 172.21.15.42:0/2402219405 wait complete. 2024-09-16T03:42:55.873 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.852+0000 7faf45073640 10 monclient: build_initial_monmap 2024-09-16T03:42:55.874 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.852+0000 7faf45073640 10 monclient: monmap: 2024-09-16T03:42:55.874 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:42:55.874 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:42:55.874 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:42:55.874 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:42:55.874 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:42:55.874 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:55.874 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:42:55.874 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:42:55.874 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:42:55.874 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:55.874 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.852+0000 7faf45073640 1 librados: starting msgr at 2024-09-16T03:42:55.874 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.852+0000 7faf45073640 1 librados: starting objecter 2024-09-16T03:42:55.875 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.852+0000 7faf45073640 1 Processor -- start 2024-09-16T03:42:55.875 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.852+0000 7faf45073640 1 -- start start 2024-09-16T03:42:55.875 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.852+0000 7faf45073640 1 librados: setting wanted keys 2024-09-16T03:42:55.875 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.852+0000 7faf45073640 1 librados: calling monclient init 2024-09-16T03:42:55.875 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.852+0000 7faf45073640 10 monclient: init 2024-09-16T03:42:55.875 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.852+0000 7faf45073640 10 monclient: _reopen_session rank -1 2024-09-16T03:42:55.875 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.852+0000 7faf45073640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T03:42:55.875 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.852+0000 7faf45073640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf400a1330 0x7faf4011d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:55.875 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.852+0000 7faf45073640 10 monclient(hunting): picked mon.noname-a con 0x7faf400a1330 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:42:55.875 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.852+0000 7faf45073640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faf40135100 0x7faf40134300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:55.875 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.852+0000 7faf45073640 10 monclient(hunting): picked mon.noname-b con 0x7faf40135100 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:42:55.875 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.852+0000 7faf45073640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7faf40134840 0x7faf4011bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:55.875 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.852+0000 7faf45073640 10 monclient(hunting): picked mon.noname-c con 0x7faf40134840 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:42:55.875 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.852+0000 7faf45073640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:55.876 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.852+0000 7faf45073640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7faf4013fb80 con 0x7faf400a1330 2024-09-16T03:42:55.876 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.852+0000 7faf45073640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:55.876 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.852+0000 7faf45073640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7faf4009ef80 con 0x7faf40135100 2024-09-16T03:42:55.876 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.852+0000 7faf45073640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:55.876 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.852+0000 7faf45073640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7faf4013fd00 con 0x7faf40134840 2024-09-16T03:42:55.876 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.852+0000 7faf45073640 10 monclient(hunting): _renew_subs 2024-09-16T03:42:55.876 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.856+0000 7faf45073640 10 monclient(hunting): authenticate will time out at 1022.828011s 2024-09-16T03:42:55.876 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.856+0000 7faf3e59c640 1 -- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faf40135100 msgr2=0x7faf40134300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T03:42:55.876 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.856+0000 7faf3ed9d640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf400a1330 0x7faf4011d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:55.876 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.856+0000 7faf3e59c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faf40135100 0x7faf40134300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:42:55.876 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.856+0000 7faf3ed9d640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf400a1330 0x7faf4011d500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:53088/0 (socket says 172.21.15.42:53088) 2024-09-16T03:42:55.876 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.856+0000 7faf3ed9d640 1 -- 172.21.15.42:0/3138446801 learned_addr learned my addr 172.21.15.42:0/3138446801 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:42:55.876 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.856+0000 7faf3ed9d640 10 monclient(hunting): get_auth_request con 0x7faf400a1330 auth_method 0 2024-09-16T03:42:55.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.856+0000 7faf3ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:55.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.856+0000 7faf3ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:55.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.856+0000 7faf3ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:55.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.856+0000 7faf3f59e640 1 --2- 172.21.15.42:0/3138446801 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7faf40134840 0x7faf4011bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:55.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.856+0000 7faf3ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:55.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.856+0000 7faf3ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:55.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.856+0000 7faf3ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:55.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.856+0000 7faf3f59e640 10 monclient(hunting): get_auth_request con 0x7faf40134840 auth_method 0 2024-09-16T03:42:55.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.856+0000 7faf3f59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:55.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.856+0000 7faf3f59e640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:55.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.856+0000 7faf3f59e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:55.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.856+0000 7faf3f59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:55.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.856+0000 7faf3f59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:55.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.856+0000 7faf3f59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:55.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.856+0000 7faf3ed9d640 10 monclient(hunting): handle_auth_done global_id 4440 payload 824 2024-09-16T03:42:55.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.856+0000 7faf3ed9d640 1 -- 172.21.15.42:0/3138446801 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7faf40134840 msgr2=0x7faf4011bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:55.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.856+0000 7faf3ed9d640 1 --2- 172.21.15.42:0/3138446801 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7faf40134840 0x7faf4011bf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:55.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.856+0000 7faf3ed9d640 1 -- 172.21.15.42:0/3138446801 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faf40135100 msgr2=0x7faf40134300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:42:55.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.856+0000 7faf3ed9d640 1 --2- 172.21.15.42:0/3138446801 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faf40135100 0x7faf40134300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:55.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.856+0000 7faf3ed9d640 10 monclient: _finish_hunting 0 2024-09-16T03:42:55.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.856+0000 7faf3ed9d640 1 monclient: found mon.noname-a 2024-09-16T03:42:55.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.856+0000 7faf3ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:42:55.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.856+0000 7faf3ed9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:42:55.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.856+0000 7faf3ed9d640 1 -- 172.21.15.42:0/3138446801 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf4013cd80 con 0x7faf400a1330 2024-09-16T03:42:55.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.856+0000 7faf3ed9d640 1 --2- 172.21.15.42:0/3138446801 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf400a1330 0x7faf4011d500 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7faf3000b530 tx=0x7faf3000b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:55.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.856+0000 7faf277fe640 10 client.?.objecter ms_handle_connect 0x7faf400a1330 2024-09-16T03:42:55.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.856+0000 7faf277fe640 10 client.?.objecter resend_mon_ops 2024-09-16T03:42:55.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.856+0000 7faf277fe640 1 -- 172.21.15.42:0/3138446801 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf30013020 con 0x7faf400a1330 2024-09-16T03:42:55.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.856+0000 7faf277fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:55.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.856+0000 7faf277fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:42:55.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.856+0000 7faf277fe640 10 monclient: dump: 2024-09-16T03:42:55.879 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:55.879 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:55.879 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:55.879 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:55.879 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:55.879 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:55.879 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:55.879 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:55.879 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:55.880 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:55.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.856+0000 7faf277fe640 10 monclient: _finish_auth 0 2024-09-16T03:42:55.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.856+0000 7faf277fe640 10 monclient: _check_auth_tickets 2024-09-16T03:42:55.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.856+0000 7faf277fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:42:55.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.856+0000 7faf45073640 5 monclient: authenticate success, global_id 4440 2024-09-16T03:42:55.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.856+0000 7faf277fe640 1 -- 172.21.15.42:0/3138446801 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faf30003df0 con 0x7faf400a1330 2024-09-16T03:42:55.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.856+0000 7faf277fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:42:55.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.856+0000 7faf277fe640 1 -- 172.21.15.42:0/3138446801 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf300052b0 con 0x7faf400a1330 2024-09-16T03:42:55.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.856+0000 7faf45073640 10 monclient: _renew_subs 2024-09-16T03:42:55.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.856+0000 7faf45073640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:42:55.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.856+0000 7faf45073640 1 -- 172.21.15.42:0/3138446801 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf4011e3f0 con 0x7faf400a1330 2024-09-16T03:42:55.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.856+0000 7faf277fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:55.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.856+0000 7faf3d59a640 4 set_mon_vals no callback set 2024-09-16T03:42:55.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.856+0000 7faf277fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:42:55.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.856+0000 7faf277fe640 10 monclient: dump: 2024-09-16T03:42:55.881 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:55.881 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:55.881 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:55.881 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:55.881 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:55.881 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:55.881 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:55.881 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:55.881 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:55.881 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:55.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.856+0000 7faf45073640 10 client.4440.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:42:55.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.856+0000 7faf45073640 10 monclient: _renew_subs 2024-09-16T03:42:55.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.856+0000 7faf45073640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:42:55.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.856+0000 7faf45073640 1 -- 172.21.15.42:0/3138446801 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf401cd000 con 0x7faf400a1330 2024-09-16T03:42:55.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.856+0000 7faf45073640 1 librados: init done 2024-09-16T03:42:55.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.856+0000 7faf45073640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:42:55.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.856+0000 7faf45073640 1 --2- 172.21.15.42:0/3138446801 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf4006dac0 0x7faf4002c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:55.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.856+0000 7faf45073640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:55.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.856+0000 7faf45073640 1 -- 172.21.15.42:0/3138446801 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7faf40073a30 con 0x7faf4006dac0 2024-09-16T03:42:55.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.856+0000 7faf277fe640 1 -- 172.21.15.42:0/3138446801 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7faf30016020 con 0x7faf400a1330 2024-09-16T03:42:55.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.856+0000 7faf3e59c640 1 --2- 172.21.15.42:0/3138446801 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf4006dac0 0x7faf4002c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:55.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.856+0000 7faf3e59c640 10 monclient: get_auth_request con 0x7faf4006dac0 auth_method 0 2024-09-16T03:42:55.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.856+0000 7faf3e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:55.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.856+0000 7faf3e59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:55.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.856+0000 7faf3e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:55.883 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.856+0000 7faf3e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:55.883 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.856+0000 7faf3e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:55.883 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.856+0000 7faf3e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:55.883 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.856+0000 7faf277fe640 1 --2- 172.21.15.42:0/3138446801 >> v2:172.21.15.154:6812/254845342 conn(0x7faf2c0377c0 0x7faf2c039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:55.883 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.856+0000 7faf3e59c640 10 monclient(hunting): handle_auth_done global_id 4443 payload 824 2024-09-16T03:42:55.883 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.856+0000 7faf277fe640 1 -- 172.21.15.42:0/3138446801 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 3534+0+0 (secure 0 0 0) 0x7faf300481e0 con 0x7faf400a1330 2024-09-16T03:42:55.883 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.856+0000 7faf3e59c640 1 -- 172.21.15.42:0/3138446801 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7faf40071c10 con 0x7faf4006dac0 2024-09-16T03:42:55.883 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.856+0000 7faf277fe640 10 client.4440.objecter ms_dispatch 0x7faf40000ba0 osd_map(19..19 src has 1..19) 2024-09-16T03:42:55.883 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.856+0000 7faf277fe640 3 client.4440.objecter handle_osd_map got epochs [19,19] > 0 2024-09-16T03:42:55.883 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.856+0000 7faf277fe640 3 client.4440.objecter handle_osd_map decoding full epoch 19 2024-09-16T03:42:55.883 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.856+0000 7faf277fe640 20 client.4440.objecter dump_active .. 0 homeless 2024-09-16T03:42:55.883 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.856+0000 7faf277fe640 10 client.4440.objecter ms_handle_connect 0x7faf4006dac0 2024-09-16T03:42:55.883 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.856+0000 7faf277fe640 10 client.4440.objecter resend_mon_ops 2024-09-16T03:42:55.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.856+0000 7faf3e59c640 1 --2- 172.21.15.42:0/3138446801 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf4006dac0 0x7faf4002c650 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7faf400a21c0 tx=0x7faf34019720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:55.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.856+0000 7faf3f59e640 1 --2- 172.21.15.42:0/3138446801 >> v2:172.21.15.154:6812/254845342 conn(0x7faf2c0377c0 0x7faf2c039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:55.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.856+0000 7faf3f59e640 10 monclient: get_auth_request con 0x7faf2c0377c0 auth_method 0 2024-09-16T03:42:55.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.856+0000 7faf277fe640 1 -- 172.21.15.42:0/3138446801 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf340bd030 con 0x7faf4006dac0 2024-09-16T03:42:55.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.856+0000 7faf277fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:55.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.856+0000 7faf277fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:42:55.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.856+0000 7faf277fe640 10 monclient: dump: 2024-09-16T03:42:55.884 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:55.884 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:55.884 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:55.885 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:55.885 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:55.885 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:55.885 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:55.885 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:55.885 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:55.885 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:55.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.856+0000 7faf277fe640 1 -- 172.21.15.42:0/3138446801 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7faf340b9470 con 0x7faf4006dac0 2024-09-16T03:42:55.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.856+0000 7faf277fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:42:55.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.856+0000 7faf277fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:42:55.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.856+0000 7faf277fe640 1 -- 172.21.15.42:0/3138446801 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf4006dac0 msgr2=0x7faf4002c650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:55.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.856+0000 7faf277fe640 1 --2- 172.21.15.42:0/3138446801 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf4006dac0 0x7faf4002c650 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7faf400a21c0 tx=0x7faf34019720 comp rx=0 tx=0).stop 2024-09-16T03:42:55.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.856+0000 7faf277fe640 1 -- 172.21.15.42:0/3138446801 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf4006dac0 msgr2=0x7faf4002c650 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:42:55.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.856+0000 7faf277fe640 1 --2- 172.21.15.42:0/3138446801 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf4006dac0 0x7faf4002c650 secure :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7faf400a21c0 tx=0x7faf34019720 comp rx=0 tx=0).stop 2024-09-16T03:42:55.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.856+0000 7faf277fe640 10 client.4440.objecter ms_handle_connect 0x7faf2c0377c0 2024-09-16T03:42:55.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.856+0000 7faf3f59e640 1 --2- 172.21.15.42:0/3138446801 >> v2:172.21.15.154:6812/254845342 conn(0x7faf2c0377c0 0x7faf2c039c80 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7faf40134c40 tx=0x7faf38009060 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:55.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.864+0000 7faf45073640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:42:55.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.864+0000 7faf45073640 1 --2- 172.21.15.42:0/3138446801 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf4002cb90 0x7faf4002cf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:55.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.864+0000 7faf45073640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:55.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.864+0000 7faf45073640 1 -- 172.21.15.42:0/3138446801 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7faf4011eda0 con 0x7faf4002cb90 2024-09-16T03:42:55.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.864+0000 7faf3e59c640 1 --2- 172.21.15.42:0/3138446801 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf4002cb90 0x7faf4002cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:55.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.864+0000 7faf3e59c640 10 monclient: get_auth_request con 0x7faf4002cb90 auth_method 0 2024-09-16T03:42:55.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.864+0000 7faf3e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:55.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.864+0000 7faf3e59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:55.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.864+0000 7faf3e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:55.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.864+0000 7faf3e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:55.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.864+0000 7faf3e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:55.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.864+0000 7faf3e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:55.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.864+0000 7faf3e59c640 10 monclient(hunting): handle_auth_done global_id 4446 payload 824 2024-09-16T03:42:55.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.864+0000 7faf3e59c640 1 -- 172.21.15.42:0/3138446801 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7faf40135550 con 0x7faf4002cb90 2024-09-16T03:42:55.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.864+0000 7faf277fe640 10 client.4440.objecter ms_handle_connect 0x7faf4002cb90 2024-09-16T03:42:55.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.864+0000 7faf3e59c640 1 --2- 172.21.15.42:0/3138446801 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf4002cb90 0x7faf4002cf70 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7faf3401a5b0 tx=0x7faf34009810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:55.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.864+0000 7faf277fe640 10 client.4440.objecter resend_mon_ops 2024-09-16T03:42:55.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.864+0000 7faf277fe640 1 -- 172.21.15.42:0/3138446801 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf340bd070 con 0x7faf4002cb90 2024-09-16T03:42:55.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.864+0000 7faf277fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:55.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.864+0000 7faf277fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:42:55.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.864+0000 7faf277fe640 10 monclient: dump: 2024-09-16T03:42:55.887 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:55.888 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:55.888 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:55.888 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:55.888 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:55.888 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:55.888 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:55.888 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:55.888 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:55.888 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:55.888 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.864+0000 7faf277fe640 1 -- 172.21.15.42:0/3138446801 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4702 (secure 0 0 0) 0x7faf340b76a0 con 0x7faf4002cb90 2024-09-16T03:42:55.888 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.864+0000 7faf277fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:42:55.888 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.864+0000 7faf277fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:42:55.888 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.864+0000 7faf277fe640 1 -- 172.21.15.42:0/3138446801 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf4002cb90 msgr2=0x7faf4002cf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:55.889 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.864+0000 7faf277fe640 1 --2- 172.21.15.42:0/3138446801 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf4002cb90 0x7faf4002cf70 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7faf3401a5b0 tx=0x7faf34009810 comp rx=0 tx=0).stop 2024-09-16T03:42:55.889 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.864+0000 7faf277fe640 1 -- 172.21.15.42:0/3138446801 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf4002cb90 msgr2=0x7faf4002cf70 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:42:55.889 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.864+0000 7faf277fe640 1 --2- 172.21.15.42:0/3138446801 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf4002cb90 0x7faf4002cf70 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:55.889 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:42:55.889 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:42:55.889 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:42:55.889 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "leader", 2024-09-16T03:42:55.889 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 12, 2024-09-16T03:42:55.889 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 55235, 2024-09-16T03:42:55.890 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:42:55.890 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:42:55.890 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T03:42:55.890 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:42:55.890 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 0, 2024-09-16T03:42:55.890 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:42:55.890 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:42:55.890 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:42:55.890 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:42:55.890 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:42:55.890 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:42:55.890 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:42:55.890 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:42:55.890 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:42:55.891 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:42:55.891 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:42:55.891 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:42:55.891 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:42:55.891 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:42:55.891 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:42:55.891 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:42:55.891 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:42:55.891 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:42:55.891 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:42:55.891 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:42:55.891 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:42:55.891 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:42:55.891 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:42:55.892 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:42:55.892 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:42:55.892 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:42:55.892 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:42:55.892 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:42:55.892 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:42:55.892 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:42:55.892 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:42:55.892 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:42:55.892 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:42:55.893 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:42:55.893 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:42:55.893 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:42:55.893 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:42:55.893 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:42:55.893 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:42:55.893 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:42:55.893 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:42:55.893 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:42:55.893 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:42:55.894 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:42:55.894 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:42:55.894 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:42:55.894 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:42:55.894 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:42:55.894 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:42:55.894 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:42:55.894 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:42:55.895 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:42:55.895 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:42:55.895 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:42:55.895 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:42:55.895 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:42:55.895 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:42:55.895 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:42:55.895 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:42:55.895 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:42:55.895 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:42:55.895 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:42:55.895 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:42:55.895 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:42:55.895 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:42:55.896 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:42:55.896 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:42:55.896 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:42:55.896 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:42:55.896 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:42:55.896 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:42:55.896 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:42:55.896 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:42:55.896 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:42:55.896 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:42:55.896 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:42:55.896 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:42:55.896 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:42:55.897 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:42:55.897 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:42:55.897 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:42:55.897 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:42:55.897 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:42:55.897 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:42:55.897 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:42:55.897 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:42:55.897 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:42:55.897 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:42:55.897 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:42:55.897 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:42:55.897 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:42:55.898 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:42:55.898 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:42:55.898 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:42:55.898 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:42:55.898 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:42:55.898 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:42:55.898 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:42:55.898 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:42:55.898 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:42:55.898 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:42:55.898 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:42:55.898 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:42:55.898 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:42:55.898 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:42:55.899 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:42:55.899 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:42:55.899 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:42:55.899 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:42:55.899 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:42:55.899 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:42:55.899 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:42:55.899 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:42:55.899 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:42:55.899 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:42:55.899 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:42:55.899 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:42:55.899 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:42:55.899 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:42:55.900 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:42:55.900 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:42:55.900 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:42:55.900 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:42:55.900 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:42:55.900 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:42:55.900 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:42:55.900 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:42:55.900 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:42:55.900 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:42:55.900 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:42:55.900 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:42:55.900 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:42:55.901 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:42:55.901 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:42:55.901 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:42:55.901 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:42:55.901 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:42:55.901 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:42:55.901 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:42:55.901 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:42:55.901 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:42:55.901 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T03:42:55.901 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:42:55.901 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:42:55.901 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:42:55.901 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 4 2024-09-16T03:42:55.902 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:42:55.902 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:42:55.902 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:42:55.902 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:42:55.902 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:42:55.902 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:42:55.902 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 4 2024-09-16T03:42:55.902 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:42:55.902 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:42:55.902 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:42:55.902 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:42:55.902 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:42:55.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.864+0000 7faf45073640 10 librados: watch_flush enter 2024-09-16T03:42:55.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.864+0000 7faf45073640 10 librados: watch_flush exit 2024-09-16T03:42:55.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.864+0000 7faf45073640 20 client.4440.objecter shutdown clearing up homeless session... 2024-09-16T03:42:55.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.864+0000 7faf45073640 10 client.4440.objecter successfully canceled tick 2024-09-16T03:42:55.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.864+0000 7faf45073640 1 -- 172.21.15.42:0/3138446801 >> v2:172.21.15.154:6812/254845342 conn(0x7faf2c0377c0 msgr2=0x7faf2c039c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:55.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.864+0000 7faf45073640 1 --2- 172.21.15.42:0/3138446801 >> v2:172.21.15.154:6812/254845342 conn(0x7faf2c0377c0 0x7faf2c039c80 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7faf40134c40 tx=0x7faf38009060 comp rx=0 tx=0).stop 2024-09-16T03:42:55.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.864+0000 7faf45073640 10 monclient: shutdown 2024-09-16T03:42:55.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.864+0000 7faf45073640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:42:55.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.864+0000 7faf45073640 1 -- 172.21.15.42:0/3138446801 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf400a1330 msgr2=0x7faf4011d500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:55.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.864+0000 7faf45073640 1 --2- 172.21.15.42:0/3138446801 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf400a1330 0x7faf4011d500 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7faf3000b530 tx=0x7faf3000b9f0 comp rx=0 tx=0).stop 2024-09-16T03:42:55.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.864+0000 7faf3f59e640 1 -- 172.21.15.42:0/3138446801 reap_dead start 2024-09-16T03:42:55.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.864+0000 7faf45073640 1 -- 172.21.15.42:0/3138446801 shutdown_connections 2024-09-16T03:42:55.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.864+0000 7faf45073640 1 --2- 172.21.15.42:0/3138446801 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faf400a1330 0x7faf4011d500 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:55.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.864+0000 7faf45073640 1 -- 172.21.15.42:0/3138446801 >> 172.21.15.42:0/3138446801 conn(0x7faf400a2420 msgr2=0x7faf400a44c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:42:55.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.864+0000 7faf45073640 1 -- 172.21.15.42:0/3138446801 shutdown_connections 2024-09-16T03:42:55.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.864+0000 7faf45073640 1 -- 172.21.15.42:0/3138446801 wait complete. 2024-09-16T03:42:55.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:55.864+0000 7faf45073640 1 librados: shutdown 2024-09-16T03:42:55.906 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T03:42:55.906 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-16T03:42:55.906 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T03:42:56.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.024+0000 7fd0a1a4d640 10 monclient: get_monmap_and_config 2024-09-16T03:42:56.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.024+0000 7fd0a1a4d640 10 monclient: build_initial_monmap 2024-09-16T03:42:56.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.024+0000 7fd0a1a4d640 10 monclient: monmap: 2024-09-16T03:42:56.031 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:42:56.031 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:56.031 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:42:56.028805+0000 2024-09-16T03:42:56.031 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:42:56.028805+0000 2024-09-16T03:42:56.031 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:42:56.031 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:56.031 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:42:56.032 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:42:56.032 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:42:56.032 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:42:56.032 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:42:56.032 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:56.045 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.024+0000 7fd0a1a4d640 1 Processor -- start 2024-09-16T03:42:56.045 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.024+0000 7fd0a1a4d640 1 -- start start 2024-09-16T03:42:56.045 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.024+0000 7fd0a1a4d640 10 monclient: init 2024-09-16T03:42:56.045 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.024+0000 7fd0a1a4d640 10 monclient: _reopen_session rank -1 2024-09-16T03:42:56.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.024+0000 7fd0a1a4d640 10 monclient: _add_conns ranks=[1,2,3,4,0] 2024-09-16T03:42:56.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.024+0000 7fd0a1a4d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd09c06db90 addr v1:172.21.15.154:6789/0 2024-09-16T03:42:56.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.024+0000 7fd0a1a4d640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd09c1340c0 0x7fd09c13c190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:56.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.024+0000 7fd0a1a4d640 10 monclient(hunting): picked mon.noname-b con 0x7fd09c1340c0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:42:56.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.024+0000 7fd0a1a4d640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd09c13c6d0 0x7fd09c13eac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:56.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.024+0000 7fd0a1a4d640 10 monclient(hunting): picked mon.noname-a con 0x7fd09c13c6d0 addr v2:172.21.15.42:3300/0 2024-09-16T03:42:56.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.024+0000 7fd0a1a4d640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd09c06ece0 con 0x7fd09c06db90 2024-09-16T03:42:56.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.024+0000 7fd0a1a4d640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:56.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.024+0000 7fd0a1a4d640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7fd09c06df70 con 0x7fd09c13c6d0 2024-09-16T03:42:56.048 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.024+0000 7fd0a1a4d640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:56.048 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.024+0000 7fd0a1a4d640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fd09c09bf80 con 0x7fd09c1340c0 2024-09-16T03:42:56.048 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.024+0000 7fd0a1a4d640 10 monclient(hunting): _renew_subs 2024-09-16T03:42:56.048 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.024+0000 7fd0a1a4d640 10 monclient(hunting): authenticate will time out at 1022.998297s 2024-09-16T03:42:56.048 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.024+0000 7fd09affd640 1 -- >> v1:172.21.15.154:6789/0 conn(0x7fd09c06db90 legacy=0x7fd09c0a37f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.154:6789/0 2024-09-16T03:42:56.048 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.024+0000 7fd09b7fe640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd09c13c6d0 0x7fd09c13eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:56.048 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.024+0000 7fd09a7fc640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd09c1340c0 0x7fd09c13c190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:56.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.024+0000 7fd09b7fe640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd09c13c6d0 0x7fd09c13eac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:53112/0 (socket says 172.21.15.42:53112) 2024-09-16T03:42:56.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.024+0000 7fd09a7fc640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd09c1340c0 0x7fd09c13c190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:45234/0 (socket says 172.21.15.42:45234) 2024-09-16T03:42:56.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.024+0000 7fd09b7fe640 1 -- 172.21.15.42:0/1944535788 learned_addr learned my addr 172.21.15.42:0/1944535788 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:42:56.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.024+0000 7fd09b7fe640 10 monclient(hunting): get_auth_request con 0x7fd09c13c6d0 auth_method 0 2024-09-16T03:42:56.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.024+0000 7fd09b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:56.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.024+0000 7fd09b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:56.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.024+0000 7fd09b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:56.050 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.024+0000 7fd09a7fc640 10 monclient(hunting): get_auth_request con 0x7fd09c1340c0 auth_method 0 2024-09-16T03:42:56.050 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.024+0000 7fd09a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:56.050 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.024+0000 7fd09a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:56.050 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.024+0000 7fd09a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:56.050 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.024+0000 7fd09b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:56.050 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.024+0000 7fd09b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:56.050 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.024+0000 7fd09b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:56.050 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.024+0000 7fd09a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:56.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.024+0000 7fd09a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:56.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.024+0000 7fd09a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:56.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.024+0000 7fd09b7fe640 10 monclient(hunting): handle_auth_done global_id 4449 payload 274 2024-09-16T03:42:56.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.024+0000 7fd09b7fe640 1 -- 172.21.15.42:0/1944535788 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd09c1340c0 msgr2=0x7fd09c13c190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:42:56.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.024+0000 7fd09b7fe640 1 --2- 172.21.15.42:0/1944535788 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd09c1340c0 0x7fd09c13c190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:56.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.024+0000 7fd09b7fe640 1 -- 172.21.15.42:0/1944535788 >> v1:172.21.15.154:6789/0 conn(0x7fd09c06db90 legacy=0x7fd09c0a37f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:42:56.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.024+0000 7fd09b7fe640 10 monclient: _finish_hunting 0 2024-09-16T03:42:56.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.024+0000 7fd09b7fe640 1 monclient: found mon.noname-a 2024-09-16T03:42:56.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.024+0000 7fd09b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:42:56.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.024+0000 7fd09b7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:42:56.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.024+0000 7fd09b7fe640 1 -- 172.21.15.42:0/1944535788 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd09c1344d0 con 0x7fd09c13c6d0 2024-09-16T03:42:56.053 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.024+0000 7fd09a7fc640 1 --2- 172.21.15.42:0/1944535788 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd09c1340c0 0x7fd09c13c190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:42:56.053 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.028+0000 7fd09b7fe640 1 --2- 172.21.15.42:0/1944535788 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd09c13c6d0 0x7fd09c13eac0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fd094009870 tx=0x7fd09401c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f14acf9f07e3f63 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:56.053 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.028+0000 7fd099ffb640 1 -- 172.21.15.42:0/1944535788 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0940b1070 con 0x7fd09c13c6d0 2024-09-16T03:42:56.053 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.028+0000 7fd099ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:56.053 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.028+0000 7fd099ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:42:56.053 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.028+0000 7fd099ffb640 10 monclient: dump: 2024-09-16T03:42:56.054 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:56.054 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:56.054 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:56.054 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:56.054 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:56.054 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:56.054 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:56.055 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:56.055 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:56.055 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:56.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.028+0000 7fd099ffb640 10 monclient: _finish_auth 0 2024-09-16T03:42:56.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.028+0000 7fd099ffb640 10 monclient: _check_auth_tickets 2024-09-16T03:42:56.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.028+0000 7fd099ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:42:56.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.028+0000 7fd099ffb640 1 -- 172.21.15.42:0/1944535788 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd0940ac600 con 0x7fd09c13c6d0 2024-09-16T03:42:56.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.028+0000 7fd0a1a4d640 5 monclient: authenticate success, global_id 4449 2024-09-16T03:42:56.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.028+0000 7fd0a1a4d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:42:56.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.028+0000 7fd099ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T03:42:56.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.028+0000 7fd0a1a4d640 10 monclient: get_monmap_and_config success 2024-09-16T03:42:56.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.028+0000 7fd099ffb640 1 -- 172.21.15.42:0/1944535788 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0940ac9d0 con 0x7fd09c13c6d0 2024-09-16T03:42:56.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.028+0000 7fd099ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:56.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.028+0000 7fd0a1a4d640 4 set_mon_vals no callback set 2024-09-16T03:42:56.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.028+0000 7fd099ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:42:56.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.028+0000 7fd099ffb640 10 monclient: dump: 2024-09-16T03:42:56.056 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:56.056 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:56.057 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:56.057 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:56.057 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:56.057 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:56.057 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:56.057 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:56.057 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:56.057 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:56.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.028+0000 7fd0a1a4d640 10 monclient: shutdown 2024-09-16T03:42:56.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.028+0000 7fd0a1a4d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:42:56.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.028+0000 7fd0a1a4d640 1 -- 172.21.15.42:0/1944535788 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd09c13c6d0 msgr2=0x7fd09c13eac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:56.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.028+0000 7fd0a1a4d640 1 --2- 172.21.15.42:0/1944535788 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd09c13c6d0 0x7fd09c13eac0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fd094009870 tx=0x7fd09401c3e0 comp rx=0 tx=0).stop 2024-09-16T03:42:56.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.028+0000 7fd0a1a4d640 1 -- 172.21.15.42:0/1944535788 shutdown_connections 2024-09-16T03:42:56.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.028+0000 7fd0a1a4d640 1 --2- 172.21.15.42:0/1944535788 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd09c13c6d0 0x7fd09c13eac0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:56.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.028+0000 7fd0a1a4d640 1 --2- 172.21.15.42:0/1944535788 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd09c1340c0 0x7fd09c13c190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:56.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.028+0000 7fd0a1a4d640 1 -- 172.21.15.42:0/1944535788 >> 172.21.15.42:0/1944535788 conn(0x7fd09c0a9fb0 msgr2=0x7fd09c0aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:42:56.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.028+0000 7fd0a1a4d640 1 -- 172.21.15.42:0/1944535788 shutdown_connections 2024-09-16T03:42:56.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.028+0000 7fd0a1a4d640 1 -- 172.21.15.42:0/1944535788 wait complete. 2024-09-16T03:42:56.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.028+0000 7fd0a1a4d640 10 monclient: build_initial_monmap 2024-09-16T03:42:56.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.028+0000 7fd0a1a4d640 10 monclient: monmap: 2024-09-16T03:42:56.058 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:42:56.058 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:42:56.058 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:42:56.058 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:42:56.058 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:42:56.058 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:56.059 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:42:56.059 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:42:56.059 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:42:56.059 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:56.059 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.028+0000 7fd0a1a4d640 1 librados: starting msgr at 2024-09-16T03:42:56.059 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.028+0000 7fd0a1a4d640 1 librados: starting objecter 2024-09-16T03:42:56.059 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.028+0000 7fd0a1a4d640 1 Processor -- start 2024-09-16T03:42:56.059 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.028+0000 7fd0a1a4d640 1 -- start start 2024-09-16T03:42:56.059 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.028+0000 7fd0a1a4d640 1 librados: setting wanted keys 2024-09-16T03:42:56.059 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.028+0000 7fd0a1a4d640 1 librados: calling monclient init 2024-09-16T03:42:56.059 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.028+0000 7fd0a1a4d640 10 monclient: init 2024-09-16T03:42:56.059 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.028+0000 7fd0a1a4d640 10 monclient: _reopen_session rank -1 2024-09-16T03:42:56.059 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.028+0000 7fd0a1a4d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:42:56.060 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.028+0000 7fd0a1a4d640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd09c06db90 0x7fd09c139d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:56.060 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.028+0000 7fd0a1a4d640 10 monclient(hunting): picked mon.noname-c con 0x7fd09c06db90 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:42:56.060 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.028+0000 7fd0a1a4d640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd09c1340c0 0x7fd09c13a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:56.060 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.028+0000 7fd0a1a4d640 10 monclient(hunting): picked mon.noname-a con 0x7fd09c1340c0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:42:56.060 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.028+0000 7fd0a1a4d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd09c13c6d0 0x7fd09c13a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:56.060 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.028+0000 7fd0a1a4d640 10 monclient(hunting): picked mon.noname-b con 0x7fd09c13c6d0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:42:56.060 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.028+0000 7fd0a1a4d640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:56.060 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.028+0000 7fd0a1a4d640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fd09c141820 con 0x7fd09c1340c0 2024-09-16T03:42:56.061 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.028+0000 7fd0a1a4d640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:56.061 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.028+0000 7fd0a1a4d640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd09c1419a0 con 0x7fd09c13c6d0 2024-09-16T03:42:56.061 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.028+0000 7fd0a1a4d640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:56.061 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.028+0000 7fd0a1a4d640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fd09c141b20 con 0x7fd09c06db90 2024-09-16T03:42:56.061 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.028+0000 7fd0a1a4d640 10 monclient(hunting): _renew_subs 2024-09-16T03:42:56.061 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.028+0000 7fd0a1a4d640 10 monclient(hunting): authenticate will time out at 1023.003322s 2024-09-16T03:42:56.061 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.028+0000 7fd09a7fc640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd09c1340c0 0x7fd09c13a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:56.061 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.028+0000 7fd09b7fe640 1 -- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd09c13c6d0 msgr2=0x7fd09c13a790 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T03:42:56.061 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.028+0000 7fd09a7fc640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd09c1340c0 0x7fd09c13a250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:53118/0 (socket says 172.21.15.42:53118) 2024-09-16T03:42:56.061 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.028+0000 7fd09a7fc640 1 -- 172.21.15.42:0/1648632058 learned_addr learned my addr 172.21.15.42:0/1648632058 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:42:56.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.028+0000 7fd09a7fc640 10 monclient(hunting): get_auth_request con 0x7fd09c1340c0 auth_method 0 2024-09-16T03:42:56.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.028+0000 7fd09a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:56.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.028+0000 7fd09a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:56.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.028+0000 7fd09a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:56.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.028+0000 7fd09b7fe640 1 --2- 172.21.15.42:0/1648632058 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd09c13c6d0 0x7fd09c13a790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:42:56.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.028+0000 7fd09a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:56.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.028+0000 7fd09a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:56.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.028+0000 7fd09a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:56.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.028+0000 7fd09affd640 1 --2- 172.21.15.42:0/1648632058 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd09c06db90 0x7fd09c139d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:56.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.028+0000 7fd09affd640 10 monclient(hunting): get_auth_request con 0x7fd09c06db90 auth_method 0 2024-09-16T03:42:56.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.028+0000 7fd09affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:56.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.028+0000 7fd09affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:56.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.028+0000 7fd09affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:56.063 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.028+0000 7fd09affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:56.063 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.028+0000 7fd09affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:56.063 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.028+0000 7fd09affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:56.063 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.028+0000 7fd09a7fc640 10 monclient(hunting): handle_auth_done global_id 4452 payload 824 2024-09-16T03:42:56.063 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.028+0000 7fd09a7fc640 1 -- 172.21.15.42:0/1648632058 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd09c06db90 msgr2=0x7fd09c139d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:56.063 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.028+0000 7fd09a7fc640 1 --2- 172.21.15.42:0/1648632058 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd09c06db90 0x7fd09c139d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:56.063 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.028+0000 7fd09a7fc640 1 -- 172.21.15.42:0/1648632058 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd09c13c6d0 msgr2=0x7fd09c13a790 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:42:56.064 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.028+0000 7fd09a7fc640 1 --2- 172.21.15.42:0/1648632058 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd09c13c6d0 0x7fd09c13a790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:56.064 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.028+0000 7fd09a7fc640 10 monclient: _finish_hunting 0 2024-09-16T03:42:56.064 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.028+0000 7fd09a7fc640 1 monclient: found mon.noname-a 2024-09-16T03:42:56.064 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.028+0000 7fd09a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:42:56.064 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.028+0000 7fd09a7fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:42:56.064 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.028+0000 7fd09a7fc640 1 -- 172.21.15.42:0/1648632058 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd09c09e1d0 con 0x7fd09c1340c0 2024-09-16T03:42:56.064 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.032+0000 7fd09a7fc640 1 --2- 172.21.15.42:0/1648632058 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd09c1340c0 0x7fd09c13a250 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fd08400ebe0 tx=0x7fd08400c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:56.064 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.032+0000 7fd093fff640 10 client.?.objecter ms_handle_connect 0x7fd09c1340c0 2024-09-16T03:42:56.064 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.032+0000 7fd093fff640 10 client.?.objecter resend_mon_ops 2024-09-16T03:42:56.065 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.032+0000 7fd093fff640 1 -- 172.21.15.42:0/1648632058 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd084017070 con 0x7fd09c1340c0 2024-09-16T03:42:56.065 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.032+0000 7fd093fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:56.065 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.032+0000 7fd093fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:42:56.065 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.032+0000 7fd093fff640 10 monclient: dump: 2024-09-16T03:42:56.065 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:56.065 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:56.065 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:56.065 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:56.066 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:56.066 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:56.066 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:56.066 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:56.066 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:56.067 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:56.067 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.032+0000 7fd093fff640 10 monclient: _finish_auth 0 2024-09-16T03:42:56.067 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.032+0000 7fd093fff640 10 monclient: _check_auth_tickets 2024-09-16T03:42:56.067 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.032+0000 7fd093fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:42:56.067 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.032+0000 7fd093fff640 1 -- 172.21.15.42:0/1648632058 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd08400cc30 con 0x7fd09c1340c0 2024-09-16T03:42:56.067 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.032+0000 7fd0a1a4d640 5 monclient: authenticate success, global_id 4452 2024-09-16T03:42:56.067 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.032+0000 7fd0a1a4d640 10 monclient: _renew_subs 2024-09-16T03:42:56.068 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.032+0000 7fd0a1a4d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:42:56.068 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.032+0000 7fd0a1a4d640 1 -- 172.21.15.42:0/1648632058 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd09c09e4b0 con 0x7fd09c1340c0 2024-09-16T03:42:56.068 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.032+0000 7fd093fff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:42:56.068 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.032+0000 7fd093fff640 1 -- 172.21.15.42:0/1648632058 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd084013470 con 0x7fd09c1340c0 2024-09-16T03:42:56.068 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.032+0000 7fd093fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:56.068 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.032+0000 7fd093fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:42:56.069 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.032+0000 7fd093fff640 10 monclient: dump: 2024-09-16T03:42:56.069 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:56.069 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:56.069 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:56.069 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:56.069 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:56.069 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:56.070 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:56.070 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:56.070 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:56.070 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:56.070 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.032+0000 7fd0997fa640 4 set_mon_vals no callback set 2024-09-16T03:42:56.070 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.032+0000 7fd0a1a4d640 10 client.4452.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:42:56.070 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.032+0000 7fd0a1a4d640 10 monclient: _renew_subs 2024-09-16T03:42:56.070 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.032+0000 7fd0a1a4d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:42:56.071 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.032+0000 7fd0a1a4d640 1 -- 172.21.15.42:0/1648632058 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd09c09def0 con 0x7fd09c1340c0 2024-09-16T03:42:56.071 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.032+0000 7fd0a1a4d640 1 librados: init done 2024-09-16T03:42:56.071 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.032+0000 7fd0a1a4d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:42:56.071 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.032+0000 7fd0a1a4d640 1 --2- 172.21.15.42:0/1648632058 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd09c1cedf0 0x7fd09c1cf1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:56.071 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.032+0000 7fd0a1a4d640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:56.071 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.032+0000 7fd0a1a4d640 1 -- 172.21.15.42:0/1648632058 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fd09c13c210 con 0x7fd09c1cedf0 2024-09-16T03:42:56.072 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.032+0000 7fd09affd640 1 --2- 172.21.15.42:0/1648632058 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd09c1cedf0 0x7fd09c1cf1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:56.072 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.032+0000 7fd09affd640 10 monclient: get_auth_request con 0x7fd09c1cedf0 auth_method 0 2024-09-16T03:42:56.072 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.032+0000 7fd09affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:56.072 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.032+0000 7fd09affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:56.072 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.032+0000 7fd09affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:56.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.032+0000 7fd09affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:56.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.032+0000 7fd09affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:56.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.032+0000 7fd09affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:56.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.032+0000 7fd09affd640 10 monclient(hunting): handle_auth_done global_id 4382 payload 824 2024-09-16T03:42:56.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.032+0000 7fd09affd640 1 -- 172.21.15.42:0/1648632058 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd09c0a37f0 con 0x7fd09c1cedf0 2024-09-16T03:42:56.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.032+0000 7fd093fff640 10 client.4452.objecter ms_handle_connect 0x7fd09c1cedf0 2024-09-16T03:42:56.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.032+0000 7fd093fff640 10 client.4452.objecter resend_mon_ops 2024-09-16T03:42:56.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.032+0000 7fd09affd640 1 --2- 172.21.15.42:0/1648632058 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd09c1cedf0 0x7fd09c1cf1d0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fd08c009ed0 tx=0x7fd08c007660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:56.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.032+0000 7fd093fff640 1 -- 172.21.15.42:0/1648632058 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fd08401a070 con 0x7fd09c1340c0 2024-09-16T03:42:56.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.032+0000 7fd093fff640 1 --2- 172.21.15.42:0/1648632058 >> v2:172.21.15.154:6812/254845342 conn(0x7fd07c0377c0 0x7fd07c039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:56.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.032+0000 7fd093fff640 1 -- 172.21.15.42:0/1648632058 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 3534+0+0 (secure 0 0 0) 0x7fd0840486e0 con 0x7fd09c1340c0 2024-09-16T03:42:56.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.032+0000 7fd093fff640 10 client.4452.objecter ms_dispatch 0x7fd09c000ba0 osd_map(19..19 src has 1..19) 2024-09-16T03:42:56.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.032+0000 7fd093fff640 3 client.4452.objecter handle_osd_map got epochs [19,19] > 0 2024-09-16T03:42:56.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.032+0000 7fd093fff640 3 client.4452.objecter handle_osd_map decoding full epoch 19 2024-09-16T03:42:56.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.032+0000 7fd093fff640 20 client.4452.objecter dump_active .. 0 homeless 2024-09-16T03:42:56.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.032+0000 7fd093fff640 1 -- 172.21.15.42:0/1648632058 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd08c01a070 con 0x7fd09c1cedf0 2024-09-16T03:42:56.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.032+0000 7fd093fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:56.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.032+0000 7fd093fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:42:56.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.032+0000 7fd093fff640 10 monclient: dump: 2024-09-16T03:42:56.076 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:56.076 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:56.076 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:56.076 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:56.076 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:56.076 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:56.076 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:56.076 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:56.076 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:56.076 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:56.077 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.032+0000 7fd093fff640 1 -- 172.21.15.42:0/1648632058 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fd08c015680 con 0x7fd09c1cedf0 2024-09-16T03:42:56.077 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.032+0000 7fd093fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:42:56.077 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.032+0000 7fd093fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:42:56.077 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.032+0000 7fd093fff640 1 -- 172.21.15.42:0/1648632058 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd09c1cedf0 msgr2=0x7fd09c1cf1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:56.077 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.032+0000 7fd093fff640 1 --2- 172.21.15.42:0/1648632058 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd09c1cedf0 0x7fd09c1cf1d0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fd08c009ed0 tx=0x7fd08c007660 comp rx=0 tx=0).stop 2024-09-16T03:42:56.077 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.032+0000 7fd093fff640 1 -- 172.21.15.42:0/1648632058 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd09c1cedf0 msgr2=0x7fd09c1cf1d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:42:56.077 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.032+0000 7fd093fff640 1 --2- 172.21.15.42:0/1648632058 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd09c1cedf0 0x7fd09c1cf1d0 secure :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fd08c009ed0 tx=0x7fd08c007660 comp rx=0 tx=0).stop 2024-09-16T03:42:56.077 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.032+0000 7fd09b7fe640 1 --2- 172.21.15.42:0/1648632058 >> v2:172.21.15.154:6812/254845342 conn(0x7fd07c0377c0 0x7fd07c039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:56.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.032+0000 7fd09b7fe640 10 monclient: get_auth_request con 0x7fd07c0377c0 auth_method 0 2024-09-16T03:42:56.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.032+0000 7fd093fff640 10 client.4452.objecter ms_handle_connect 0x7fd07c0377c0 2024-09-16T03:42:56.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.032+0000 7fd09b7fe640 1 --2- 172.21.15.42:0/1648632058 >> v2:172.21.15.154:6812/254845342 conn(0x7fd07c0377c0 0x7fd07c039c80 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fd0940ae000 tx=0x7fd094008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:56.081 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.040+0000 7fd0a1a4d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:42:56.081 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.040+0000 7fd0a1a4d640 1 --2- 172.21.15.42:0/1648632058 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd09c1cf710 0x7fd09c1349e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:42:56.081 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.040+0000 7fd0a1a4d640 10 monclient(hunting): start opening mon connection 2024-09-16T03:42:56.081 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.040+0000 7fd0a1a4d640 1 -- 172.21.15.42:0/1648632058 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fd09c09caa0 con 0x7fd09c1cf710 2024-09-16T03:42:56.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.040+0000 7fd09affd640 1 --2- 172.21.15.42:0/1648632058 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd09c1cf710 0x7fd09c1349e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:42:56.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.040+0000 7fd09affd640 10 monclient: get_auth_request con 0x7fd09c1cf710 auth_method 0 2024-09-16T03:42:56.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.040+0000 7fd09affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:42:56.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.040+0000 7fd09affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:42:56.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.040+0000 7fd09affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:42:56.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.040+0000 7fd09affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:42:56.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.040+0000 7fd09affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:42:56.083 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.040+0000 7fd09affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:42:56.083 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.040+0000 7fd09affd640 10 monclient(hunting): handle_auth_done global_id 4385 payload 824 2024-09-16T03:42:56.083 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.040+0000 7fd09affd640 1 -- 172.21.15.42:0/1648632058 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fd09c068e40 con 0x7fd09c1cf710 2024-09-16T03:42:56.083 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.040+0000 7fd093fff640 10 client.4452.objecter ms_handle_connect 0x7fd09c1cf710 2024-09-16T03:42:56.083 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.040+0000 7fd093fff640 10 client.4452.objecter resend_mon_ops 2024-09-16T03:42:56.083 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.040+0000 7fd09affd640 1 --2- 172.21.15.42:0/1648632058 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd09c1cf710 0x7fd09c1349e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fd08c009ed0 tx=0x7fd08c005d10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:42:56.083 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.040+0000 7fd093fff640 1 -- 172.21.15.42:0/1648632058 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd08c01a050 con 0x7fd09c1cf710 2024-09-16T03:42:56.083 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.040+0000 7fd093fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:42:56.083 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.040+0000 7fd093fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:42:56.083 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.040+0000 7fd093fff640 10 monclient: dump: 2024-09-16T03:42:56.083 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:42:56.083 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:42:56.083 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:56.084 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:42:56.084 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:42:56.084 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:42:56.084 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:42:56.084 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:42:56.084 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:42:56.084 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:42:56.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.040+0000 7fd093fff640 1 -- 172.21.15.42:0/1648632058 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4867 (secure 0 0 0) 0x7fd08c01b040 con 0x7fd09c1cf710 2024-09-16T03:42:56.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.040+0000 7fd093fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:42:56.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.040+0000 7fd093fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:42:56.085 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.040+0000 7fd093fff640 1 -- 172.21.15.42:0/1648632058 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd09c1cf710 msgr2=0x7fd09c1349e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:56.085 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.040+0000 7fd093fff640 1 --2- 172.21.15.42:0/1648632058 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd09c1cf710 0x7fd09c1349e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fd08c009ed0 tx=0x7fd08c005d10 comp rx=0 tx=0).stop 2024-09-16T03:42:56.085 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.040+0000 7fd093fff640 1 -- 172.21.15.42:0/1648632058 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd09c1cf710 msgr2=0x7fd09c1349e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:42:56.085 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.040+0000 7fd093fff640 1 --2- 172.21.15.42:0/1648632058 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd09c1cf710 0x7fd09c1349e0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:56.085 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:42:56.086 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:42:56.086 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:42:56.086 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T03:42:56.086 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 12, 2024-09-16T03:42:56.086 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 55411, 2024-09-16T03:42:56.086 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:42:56.086 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:42:56.086 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T03:42:56.086 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:42:56.086 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 0, 2024-09-16T03:42:56.086 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:42:56.087 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:42:56.087 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:42:56.087 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:42:56.087 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:42:56.087 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:42:56.087 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:42:56.087 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:42:56.087 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:42:56.087 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:42:56.087 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:42:56.087 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:42:56.087 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:42:56.087 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:42:56.087 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:42:56.088 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:42:56.088 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:42:56.088 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:42:56.088 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:42:56.088 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:42:56.088 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:42:56.088 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:42:56.088 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:42:56.088 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:42:56.088 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:42:56.088 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:42:56.088 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:42:56.088 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:42:56.089 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:42:56.089 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:42:56.089 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:42:56.089 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:42:56.089 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:42:56.089 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:42:56.089 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:42:56.089 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:42:56.089 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:42:56.089 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:42:56.089 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:42:56.090 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:42:56.090 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:42:56.090 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:42:56.090 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:42:56.090 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:42:56.090 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:42:56.090 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:42:56.090 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:42:56.090 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:42:56.090 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:42:56.090 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:42:56.090 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:42:56.091 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:42:56.091 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:42:56.091 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:42:56.091 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:42:56.091 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:42:56.091 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:42:56.091 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:42:56.091 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:42:56.091 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:42:56.091 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:42:56.091 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:42:56.091 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:42:56.091 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:42:56.091 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:42:56.092 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:42:56.092 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:42:56.092 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:42:56.092 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:42:56.092 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:42:56.092 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:42:56.092 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:42:56.092 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:42:56.092 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:42:56.092 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:42:56.092 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:42:56.092 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:42:56.092 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:42:56.093 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:42:56.093 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:42:56.093 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:42:56.093 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:42:56.093 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:42:56.093 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:42:56.093 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:42:56.093 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:42:56.093 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:42:56.093 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:42:56.093 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:42:56.093 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:42:56.093 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:42:56.093 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:42:56.094 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:42:56.094 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:42:56.094 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:42:56.094 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:42:56.094 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:42:56.094 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:42:56.094 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:42:56.094 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:42:56.094 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:42:56.094 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:42:56.094 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:42:56.094 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:42:56.094 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:42:56.094 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:42:56.095 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:42:56.095 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:42:56.095 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:42:56.095 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:42:56.095 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:42:56.095 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:42:56.095 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:42:56.095 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:42:56.095 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:42:56.095 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:42:56.095 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:42:56.095 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:42:56.095 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:42:56.096 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:42:56.096 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:42:56.096 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:42:56.096 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:42:56.096 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:42:56.096 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:42:56.096 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:42:56.096 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:42:56.096 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:42:56.096 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:42:56.096 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:42:56.096 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:42:56.096 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:42:56.097 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:42:56.097 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:42:56.097 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:42:56.097 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:42:56.097 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:42:56.097 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:42:56.097 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:42:56.097 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:42:56.097 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:42:56.097 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T03:42:56.097 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:42:56.097 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:42:56.097 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:42:56.098 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 2 2024-09-16T03:42:56.098 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:42:56.098 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:42:56.098 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:42:56.098 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:42:56.098 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:42:56.098 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:42:56.098 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 3 2024-09-16T03:42:56.098 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:42:56.098 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:42:56.098 INFO:teuthology.orchestra.run.smithi042.stdout: "mgr": [ 2024-09-16T03:42:56.098 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:42:56.098 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:42:56.098 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:42:56.099 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:42:56.099 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:42:56.099 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:42:56.099 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:42:56.099 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:42:56.099 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:42:56.101 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.040+0000 7fd0a1a4d640 10 librados: watch_flush enter 2024-09-16T03:42:56.101 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.040+0000 7fd0a1a4d640 10 librados: watch_flush exit 2024-09-16T03:42:56.101 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.040+0000 7fd0a1a4d640 20 client.4452.objecter shutdown clearing up homeless session... 2024-09-16T03:42:56.101 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.040+0000 7fd0a1a4d640 10 client.4452.objecter successfully canceled tick 2024-09-16T03:42:56.101 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.040+0000 7fd0a1a4d640 1 -- 172.21.15.42:0/1648632058 >> v2:172.21.15.154:6812/254845342 conn(0x7fd07c0377c0 msgr2=0x7fd07c039c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:56.101 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.040+0000 7fd0a1a4d640 1 --2- 172.21.15.42:0/1648632058 >> v2:172.21.15.154:6812/254845342 conn(0x7fd07c0377c0 0x7fd07c039c80 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fd0940ae000 tx=0x7fd094008040 comp rx=0 tx=0).stop 2024-09-16T03:42:56.101 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.040+0000 7fd0a1a4d640 10 monclient: shutdown 2024-09-16T03:42:56.101 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.040+0000 7fd0a1a4d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:42:56.101 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.040+0000 7fd0a1a4d640 1 -- 172.21.15.42:0/1648632058 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd09c1340c0 msgr2=0x7fd09c13a250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:42:56.101 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.040+0000 7fd0a1a4d640 1 --2- 172.21.15.42:0/1648632058 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd09c1340c0 0x7fd09c13a250 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fd08400ebe0 tx=0x7fd08400c520 comp rx=0 tx=0).stop 2024-09-16T03:42:56.101 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.040+0000 7fd09b7fe640 1 -- 172.21.15.42:0/1648632058 reap_dead start 2024-09-16T03:42:56.101 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.040+0000 7fd0a1a4d640 1 -- 172.21.15.42:0/1648632058 shutdown_connections 2024-09-16T03:42:56.101 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.040+0000 7fd0a1a4d640 1 --2- 172.21.15.42:0/1648632058 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd09c1340c0 0x7fd09c13a250 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:42:56.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.040+0000 7fd0a1a4d640 1 -- 172.21.15.42:0/1648632058 >> 172.21.15.42:0/1648632058 conn(0x7fd09c0a9fb0 msgr2=0x7fd09c0a9130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:42:56.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.040+0000 7fd0a1a4d640 1 -- 172.21.15.42:0/1648632058 shutdown_connections 2024-09-16T03:42:56.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.040+0000 7fd0a1a4d640 1 -- 172.21.15.42:0/1648632058 wait complete. 2024-09-16T03:42:56.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:42:56.040+0000 7fd0a1a4d640 1 librados: shutdown 2024-09-16T03:42:56.102 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T03:42:56.102 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-16T03:43:16.103 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-16T03:43:16.104 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-09-16T03:43:16.104 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-16T03:43:16.104 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-16T03:43:16.107 INFO:tasks.ceph.mon.b:Started 2024-09-16T03:43:16.107 INFO:tasks.mon_thrash:Sending STOP to mon a 2024-09-16T03:43:16.108 INFO:tasks.ceph.mon.a:Sent signal 19 2024-09-16T03:43:16.108 INFO:tasks.mon_thrash:Sending STOP to mon c 2024-09-16T03:43:16.108 INFO:tasks.ceph.mon.c:Sent signal 19 2024-09-16T03:43:16.108 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-16T03:43:28.407 INFO:tasks.workunit.client.0.smithi154.stderr:Note: switching to 'ad35db06e8924912f01e32b35576297c0f1352b4'. 2024-09-16T03:43:28.407 INFO:tasks.workunit.client.0.smithi154.stderr: 2024-09-16T03:43:28.407 INFO:tasks.workunit.client.0.smithi154.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-09-16T03:43:28.407 INFO:tasks.workunit.client.0.smithi154.stderr:changes and commit them, and you can discard any commits you make in this 2024-09-16T03:43:28.407 INFO:tasks.workunit.client.0.smithi154.stderr:state without impacting any branches by switching back to a branch. 2024-09-16T03:43:28.408 INFO:tasks.workunit.client.0.smithi154.stderr: 2024-09-16T03:43:28.408 INFO:tasks.workunit.client.0.smithi154.stderr:If you want to create a new branch to retain commits you create, you may 2024-09-16T03:43:28.408 INFO:tasks.workunit.client.0.smithi154.stderr:do so (now or later) by using -c with the switch command. Example: 2024-09-16T03:43:28.408 INFO:tasks.workunit.client.0.smithi154.stderr: 2024-09-16T03:43:28.408 INFO:tasks.workunit.client.0.smithi154.stderr: git switch -c 2024-09-16T03:43:28.408 INFO:tasks.workunit.client.0.smithi154.stderr: 2024-09-16T03:43:28.408 INFO:tasks.workunit.client.0.smithi154.stderr:Or undo this operation with: 2024-09-16T03:43:28.408 INFO:tasks.workunit.client.0.smithi154.stderr: 2024-09-16T03:43:28.408 INFO:tasks.workunit.client.0.smithi154.stderr: git switch - 2024-09-16T03:43:28.408 INFO:tasks.workunit.client.0.smithi154.stderr: 2024-09-16T03:43:28.408 INFO:tasks.workunit.client.0.smithi154.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-09-16T03:43:28.408 INFO:tasks.workunit.client.0.smithi154.stderr: 2024-09-16T03:43:28.408 INFO:tasks.workunit.client.0.smithi154.stderr:HEAD is now at ad35db06e89 Merge pull request #59794 from zdover23/wip-doc-2024-09-15-backport-59613-to-squid 2024-09-16T03:43:28.413 DEBUG:teuthology.orchestra.run.smithi154:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-09-16T03:43:28.459 INFO:tasks.workunit.client.0.smithi154.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-09-16T03:43:28.463 INFO:tasks.workunit.client.0.smithi154.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-16T03:43:28.463 INFO:tasks.workunit.client.0.smithi154.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-09-16T03:43:29.251 INFO:tasks.workunit.client.0.smithi154.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-09-16T03:43:29.325 INFO:tasks.workunit.client.0.smithi154.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-09-16T03:43:29.375 INFO:tasks.workunit.client.0.smithi154.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-16T03:43:29.376 INFO:tasks.workunit.client.0.smithi154.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-16T03:43:29.376 INFO:tasks.workunit.client.0.smithi154.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-09-16T03:43:29.425 INFO:tasks.workunit.client.0.smithi154.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-16T03:43:29.429 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-09-16T03:43:29.429 DEBUG:teuthology.orchestra.run.smithi154:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-09-16T03:43:29.475 INFO:tasks.workunit:Running workunits matching rados/test.sh on client.0... 2024-09-16T03:43:29.477 INFO:tasks.workunit:Running workunit rados/test.sh... 2024-09-16T03:43:29.477 DEBUG:teuthology.orchestra.run.smithi154:workunit test rados/test.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ad35db06e8924912f01e32b35576297c0f1352b4 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh 2024-09-16T03:43:29.528 INFO:tasks.workunit.client.0.smithi154.stderr:+ parallel=1 2024-09-16T03:43:29.528 INFO:tasks.workunit.client.0.smithi154.stderr:+ '[' '' = --serial ']' 2024-09-16T03:43:29.528 INFO:tasks.workunit.client.0.smithi154.stderr:+ crimson=0 2024-09-16T03:43:29.528 INFO:tasks.workunit.client.0.smithi154.stderr:+ '[' '' = --crimson ']' 2024-09-16T03:43:29.528 INFO:tasks.workunit.client.0.smithi154.stderr:+ color= 2024-09-16T03:43:29.528 INFO:tasks.workunit.client.0.smithi154.stderr:+ '[' -t 1 ']' 2024-09-16T03:43:29.528 INFO:tasks.workunit.client.0.smithi154.stderr:+ trap cleanup EXIT ERR HUP INT QUIT 2024-09-16T03:43:29.528 INFO:tasks.workunit.client.0.smithi154.stderr:+ GTEST_OUTPUT_DIR=/home/ubuntu/cephtest/archive/unit_test_xml_report 2024-09-16T03:43:29.528 INFO:tasks.workunit.client.0.smithi154.stderr:+ mkdir -p /home/ubuntu/cephtest/archive/unit_test_xml_report 2024-09-16T03:43:29.529 INFO:tasks.workunit.client.0.smithi154.stderr:+ declare -A pids 2024-09-16T03:43:29.529 INFO:tasks.workunit.client.0.smithi154.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T03:43:29.529 INFO:tasks.workunit.client.0.smithi154.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T03:43:29.529 INFO:tasks.workunit.client.0.smithi154.stderr:++ printf %25s api_aio 2024-09-16T03:43:29.530 INFO:tasks.workunit.client.0.smithi154.stderr:+ r=' api_aio' 2024-09-16T03:43:29.530 INFO:tasks.workunit.client.0.smithi154.stderr:++ echo api_aio 2024-09-16T03:43:29.530 INFO:tasks.workunit.client.0.smithi154.stderr:++ awk '{print $1}' 2024-09-16T03:43:29.532 INFO:tasks.workunit.client.0.smithi154.stderr:+ ff=api_aio 2024-09-16T03:43:29.533 INFO:tasks.workunit.client.0.smithi154.stderr:+ pid=28078 2024-09-16T03:43:29.533 INFO:tasks.workunit.client.0.smithi154.stderr:+ echo 'test api_aio on pid 28078' 2024-09-16T03:43:29.533 INFO:tasks.workunit.client.0.smithi154.stdout:test api_aio on pid 28078 2024-09-16T03:43:29.534 INFO:tasks.workunit.client.0.smithi154.stderr:+ pids[$f]=28078 2024-09-16T03:43:29.534 INFO:tasks.workunit.client.0.smithi154.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T03:43:29.534 INFO:tasks.workunit.client.0.smithi154.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T03:43:29.535 INFO:tasks.workunit.client.0.smithi154.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_aio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio.xml 2>&1 | tee ceph_test_rados_api_aio.log | sed "s/^/ api_aio: /"' 2024-09-16T03:43:29.535 INFO:tasks.workunit.client.0.smithi154.stderr:++ printf %25s api_aio_pp 2024-09-16T03:43:29.535 INFO:tasks.workunit.client.0.smithi154.stderr:+ r=' api_aio_pp' 2024-09-16T03:43:29.535 INFO:tasks.workunit.client.0.smithi154.stderr:+ '[' -z '' ']' 2024-09-16T03:43:29.535 INFO:tasks.workunit.client.0.smithi154.stderr:+ return 2024-09-16T03:43:29.535 INFO:tasks.workunit.client.0.smithi154.stderr:+ case $- in 2024-09-16T03:43:29.535 INFO:tasks.workunit.client.0.smithi154.stderr:+ return 2024-09-16T03:43:29.535 INFO:tasks.workunit.client.0.smithi154.stderr:++ echo api_aio_pp 2024-09-16T03:43:29.535 INFO:tasks.workunit.client.0.smithi154.stderr:++ awk '{print $1}' 2024-09-16T03:43:29.535 INFO:tasks.workunit.client.0.smithi154.stderr:+ ceph_test_rados_api_aio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio.xml 2024-09-16T03:43:29.535 INFO:tasks.workunit.client.0.smithi154.stderr:+ sed 's/^/ api_aio: /' 2024-09-16T03:43:29.535 INFO:tasks.workunit.client.0.smithi154.stderr:+ tee ceph_test_rados_api_aio.log 2024-09-16T03:43:29.536 INFO:tasks.workunit.client.0.smithi154.stderr:+ ff=api_aio_pp 2024-09-16T03:43:29.536 INFO:tasks.workunit.client.0.smithi154.stderr:+ pid=28086 2024-09-16T03:43:29.536 INFO:tasks.workunit.client.0.smithi154.stderr:+ echo 'test api_aio_pp on pid 28086' 2024-09-16T03:43:29.536 INFO:tasks.workunit.client.0.smithi154.stdout:test api_aio_pp on pid 28086 2024-09-16T03:43:29.537 INFO:tasks.workunit.client.0.smithi154.stderr:+ pids[$f]=28086 2024-09-16T03:43:29.538 INFO:tasks.workunit.client.0.smithi154.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T03:43:29.538 INFO:tasks.workunit.client.0.smithi154.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T03:43:29.538 INFO:tasks.workunit.client.0.smithi154.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_aio_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio_pp.xml 2>&1 | tee ceph_test_rados_api_aio_pp.log | sed "s/^/ api_aio_pp: /"' 2024-09-16T03:43:29.538 INFO:tasks.workunit.client.0.smithi154.stderr:++ printf %25s api_io 2024-09-16T03:43:29.538 INFO:tasks.workunit.client.0.smithi154.stderr:+ r=' api_io' 2024-09-16T03:43:29.538 INFO:tasks.workunit.client.0.smithi154.stderr:+ '[' -z '' ']' 2024-09-16T03:43:29.538 INFO:tasks.workunit.client.0.smithi154.stderr:+ return 2024-09-16T03:43:29.538 INFO:tasks.workunit.client.0.smithi154.stderr:+ case $- in 2024-09-16T03:43:29.538 INFO:tasks.workunit.client.0.smithi154.stderr:+ return 2024-09-16T03:43:29.538 INFO:tasks.workunit.client.0.smithi154.stderr:++ echo api_io 2024-09-16T03:43:29.538 INFO:tasks.workunit.client.0.smithi154.stderr:++ awk '{print $1}' 2024-09-16T03:43:29.538 INFO:tasks.workunit.client.0.smithi154.stderr:+ ceph_test_rados_api_aio_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio_pp.xml 2024-09-16T03:43:29.538 INFO:tasks.workunit.client.0.smithi154.stderr:+ tee ceph_test_rados_api_aio_pp.log 2024-09-16T03:43:29.539 INFO:tasks.workunit.client.0.smithi154.stderr:+ sed 's/^/ api_aio_pp: /' 2024-09-16T03:43:29.539 INFO:tasks.workunit.client.0.smithi154.stderr:+ ff=api_io 2024-09-16T03:43:29.539 INFO:tasks.workunit.client.0.smithi154.stderr:+ pid=28094 2024-09-16T03:43:29.540 INFO:tasks.workunit.client.0.smithi154.stdout:test api_io on pid 28094 2024-09-16T03:43:29.541 INFO:tasks.workunit.client.0.smithi154.stderr:+ echo 'test api_io on pid 28094' 2024-09-16T03:43:29.541 INFO:tasks.workunit.client.0.smithi154.stderr:+ pids[$f]=28094 2024-09-16T03:43:29.541 INFO:tasks.workunit.client.0.smithi154.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T03:43:29.541 INFO:tasks.workunit.client.0.smithi154.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_io --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io.xml 2>&1 | tee ceph_test_rados_api_io.log | sed "s/^/ api_io: /"' 2024-09-16T03:43:29.541 INFO:tasks.workunit.client.0.smithi154.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T03:43:29.541 INFO:tasks.workunit.client.0.smithi154.stderr:++ printf %25s api_io_pp 2024-09-16T03:43:29.541 INFO:tasks.workunit.client.0.smithi154.stderr:+ r=' api_io_pp' 2024-09-16T03:43:29.542 INFO:tasks.workunit.client.0.smithi154.stderr:++ echo api_io_pp 2024-09-16T03:43:29.542 INFO:tasks.workunit.client.0.smithi154.stderr:+ '[' -z '' ']' 2024-09-16T03:43:29.542 INFO:tasks.workunit.client.0.smithi154.stderr:+ return 2024-09-16T03:43:29.542 INFO:tasks.workunit.client.0.smithi154.stderr:+ case $- in 2024-09-16T03:43:29.542 INFO:tasks.workunit.client.0.smithi154.stderr:+ return 2024-09-16T03:43:29.542 INFO:tasks.workunit.client.0.smithi154.stderr:++ awk '{print $1}' 2024-09-16T03:43:29.542 INFO:tasks.workunit.client.0.smithi154.stderr:+ ceph_test_rados_api_io --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io.xml 2024-09-16T03:43:29.542 INFO:tasks.workunit.client.0.smithi154.stderr:+ tee ceph_test_rados_api_io.log 2024-09-16T03:43:29.542 INFO:tasks.workunit.client.0.smithi154.stderr:+ sed 's/^/ api_io: /' 2024-09-16T03:43:29.543 INFO:tasks.workunit.client.0.smithi154.stderr:+ ff=api_io_pp 2024-09-16T03:43:29.543 INFO:tasks.workunit.client.0.smithi154.stderr:+ pid=28102 2024-09-16T03:43:29.543 INFO:tasks.workunit.client.0.smithi154.stdout:test api_io_pp on pid 28102 2024-09-16T03:43:29.544 INFO:tasks.workunit.client.0.smithi154.stderr:+ echo 'test api_io_pp on pid 28102' 2024-09-16T03:43:29.544 INFO:tasks.workunit.client.0.smithi154.stderr:+ pids[$f]=28102 2024-09-16T03:43:29.544 INFO:tasks.workunit.client.0.smithi154.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T03:43:29.545 INFO:tasks.workunit.client.0.smithi154.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_io_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io_pp.xml 2>&1 | tee ceph_test_rados_api_io_pp.log | sed "s/^/ api_io_pp: /"' 2024-09-16T03:43:29.545 INFO:tasks.workunit.client.0.smithi154.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T03:43:29.545 INFO:tasks.workunit.client.0.smithi154.stderr:++ printf %25s api_asio 2024-09-16T03:43:29.545 INFO:tasks.workunit.client.0.smithi154.stderr:+ r=' api_asio' 2024-09-16T03:43:29.545 INFO:tasks.workunit.client.0.smithi154.stderr:+ '[' -z '' ']' 2024-09-16T03:43:29.545 INFO:tasks.workunit.client.0.smithi154.stderr:+ return 2024-09-16T03:43:29.545 INFO:tasks.workunit.client.0.smithi154.stderr:+ case $- in 2024-09-16T03:43:29.545 INFO:tasks.workunit.client.0.smithi154.stderr:+ return 2024-09-16T03:43:29.545 INFO:tasks.workunit.client.0.smithi154.stderr:++ echo api_asio 2024-09-16T03:43:29.545 INFO:tasks.workunit.client.0.smithi154.stderr:++ awk '{print $1}' 2024-09-16T03:43:29.546 INFO:tasks.workunit.client.0.smithi154.stderr:+ ceph_test_rados_api_io_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io_pp.xml 2024-09-16T03:43:29.546 INFO:tasks.workunit.client.0.smithi154.stderr:+ tee ceph_test_rados_api_io_pp.log 2024-09-16T03:43:29.546 INFO:tasks.workunit.client.0.smithi154.stderr:+ sed 's/^/ api_io_pp: /' 2024-09-16T03:43:29.546 INFO:tasks.workunit.client.0.smithi154.stderr:+ ff=api_asio 2024-09-16T03:43:29.546 INFO:tasks.workunit.client.0.smithi154.stderr:+ pid=28110 2024-09-16T03:43:29.547 INFO:tasks.workunit.client.0.smithi154.stdout:test api_asio on pid 28110 2024-09-16T03:43:29.548 INFO:tasks.workunit.client.0.smithi154.stderr:+ echo 'test api_asio on pid 28110' 2024-09-16T03:43:29.548 INFO:tasks.workunit.client.0.smithi154.stderr:+ pids[$f]=28110 2024-09-16T03:43:29.548 INFO:tasks.workunit.client.0.smithi154.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T03:43:29.548 INFO:tasks.workunit.client.0.smithi154.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_asio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_asio.xml 2>&1 | tee ceph_test_rados_api_asio.log | sed "s/^/ api_asio: /"' 2024-09-16T03:43:29.548 INFO:tasks.workunit.client.0.smithi154.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T03:43:29.548 INFO:tasks.workunit.client.0.smithi154.stderr:++ printf %25s api_list 2024-09-16T03:43:29.548 INFO:tasks.workunit.client.0.smithi154.stderr:+ r=' api_list' 2024-09-16T03:43:29.548 INFO:tasks.workunit.client.0.smithi154.stderr:+ '[' -z '' ']' 2024-09-16T03:43:29.548 INFO:tasks.workunit.client.0.smithi154.stderr:+ return 2024-09-16T03:43:29.549 INFO:tasks.workunit.client.0.smithi154.stderr:+ case $- in 2024-09-16T03:43:29.549 INFO:tasks.workunit.client.0.smithi154.stderr:+ return 2024-09-16T03:43:29.549 INFO:tasks.workunit.client.0.smithi154.stderr:++ echo api_list 2024-09-16T03:43:29.549 INFO:tasks.workunit.client.0.smithi154.stderr:++ awk '{print $1}' 2024-09-16T03:43:29.549 INFO:tasks.workunit.client.0.smithi154.stderr:+ ceph_test_rados_api_asio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_asio.xml 2024-09-16T03:43:29.549 INFO:tasks.workunit.client.0.smithi154.stderr:+ tee ceph_test_rados_api_asio.log 2024-09-16T03:43:29.549 INFO:tasks.workunit.client.0.smithi154.stderr:+ sed 's/^/ api_asio: /' 2024-09-16T03:43:29.550 INFO:tasks.workunit.client.0.smithi154.stderr:+ ff=api_list 2024-09-16T03:43:29.551 INFO:tasks.workunit.client.0.smithi154.stderr:+ pid=28118 2024-09-16T03:43:29.551 INFO:tasks.workunit.client.0.smithi154.stderr:+ echo 'test api_list on pid 28118' 2024-09-16T03:43:29.551 INFO:tasks.workunit.client.0.smithi154.stdout:test api_list on pid 28118 2024-09-16T03:43:29.552 INFO:tasks.workunit.client.0.smithi154.stderr:+ pids[$f]=28118 2024-09-16T03:43:29.552 INFO:tasks.workunit.client.0.smithi154.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T03:43:29.552 INFO:tasks.workunit.client.0.smithi154.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_list --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_list.xml 2>&1 | tee ceph_test_rados_api_list.log | sed "s/^/ api_list: /"' 2024-09-16T03:43:29.552 INFO:tasks.workunit.client.0.smithi154.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T03:43:29.552 INFO:tasks.workunit.client.0.smithi154.stderr:++ printf %25s api_lock 2024-09-16T03:43:29.552 INFO:tasks.workunit.client.0.smithi154.stderr:+ r=' api_lock' 2024-09-16T03:43:29.552 INFO:tasks.workunit.client.0.smithi154.stderr:++ echo api_lock 2024-09-16T03:43:29.553 INFO:tasks.workunit.client.0.smithi154.stderr:+ '[' -z '' ']' 2024-09-16T03:43:29.553 INFO:tasks.workunit.client.0.smithi154.stderr:+ return 2024-09-16T03:43:29.553 INFO:tasks.workunit.client.0.smithi154.stderr:+ case $- in 2024-09-16T03:43:29.553 INFO:tasks.workunit.client.0.smithi154.stderr:+ return 2024-09-16T03:43:29.553 INFO:tasks.workunit.client.0.smithi154.stderr:+ ceph_test_rados_api_list --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_list.xml 2024-09-16T03:43:29.553 INFO:tasks.workunit.client.0.smithi154.stderr:+ tee ceph_test_rados_api_list.log 2024-09-16T03:43:29.555 INFO:tasks.workunit.client.0.smithi154.stderr:++ awk '{print $1}' 2024-09-16T03:43:29.558 INFO:tasks.workunit.client.0.smithi154.stdout:test api_lock on pid 28126 2024-09-16T03:43:29.558 INFO:tasks.workunit.client.0.smithi154.stderr:+ ff=api_lock 2024-09-16T03:43:29.558 INFO:tasks.workunit.client.0.smithi154.stderr:+ pid=28126 2024-09-16T03:43:29.558 INFO:tasks.workunit.client.0.smithi154.stderr:+ echo 'test api_lock on pid 28126' 2024-09-16T03:43:29.558 INFO:tasks.workunit.client.0.smithi154.stderr:+ pids[$f]=28126 2024-09-16T03:43:29.558 INFO:tasks.workunit.client.0.smithi154.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T03:43:29.558 INFO:tasks.workunit.client.0.smithi154.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T03:43:29.558 INFO:tasks.workunit.client.0.smithi154.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_lock --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock.xml 2>&1 | tee ceph_test_rados_api_lock.log | sed "s/^/ api_lock: /"' 2024-09-16T03:43:29.559 INFO:tasks.workunit.client.0.smithi154.stderr:+ '[' -z '' ']' 2024-09-16T03:43:29.559 INFO:tasks.workunit.client.0.smithi154.stderr:+ return 2024-09-16T03:43:29.559 INFO:tasks.workunit.client.0.smithi154.stderr:+ case $- in 2024-09-16T03:43:29.559 INFO:tasks.workunit.client.0.smithi154.stderr:+ return 2024-09-16T03:43:29.559 INFO:tasks.workunit.client.0.smithi154.stderr:+ ceph_test_rados_api_lock --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock.xml 2024-09-16T03:43:29.567 INFO:tasks.workunit.client.0.smithi154.stderr:++ printf %25s api_lock_pp 2024-09-16T03:43:29.567 INFO:tasks.workunit.client.0.smithi154.stderr:+ r=' api_lock_pp' 2024-09-16T03:43:29.568 INFO:tasks.workunit.client.0.smithi154.stderr:++ awk '{print $1}' 2024-09-16T03:43:29.571 INFO:tasks.workunit.client.0.smithi154.stderr:+ sed 's/^/ api_list: /' 2024-09-16T03:43:29.571 INFO:tasks.workunit.client.0.smithi154.stderr:+ tee ceph_test_rados_api_lock.log 2024-09-16T03:43:29.571 INFO:tasks.workunit.client.0.smithi154.stderr:+ sed 's/^/ api_lock: /' 2024-09-16T03:43:29.577 INFO:tasks.workunit.client.0.smithi154.stderr:++ echo api_lock_pp 2024-09-16T03:43:29.579 INFO:tasks.workunit.client.0.smithi154.stdout:test api_lock_pp on pid 28151 2024-09-16T03:43:29.579 INFO:tasks.workunit.client.0.smithi154.stderr:+ ff=api_lock_pp 2024-09-16T03:43:29.580 INFO:tasks.workunit.client.0.smithi154.stderr:+ pid=28151 2024-09-16T03:43:29.580 INFO:tasks.workunit.client.0.smithi154.stderr:+ echo 'test api_lock_pp on pid 28151' 2024-09-16T03:43:29.580 INFO:tasks.workunit.client.0.smithi154.stderr:+ pids[$f]=28151 2024-09-16T03:43:29.580 INFO:tasks.workunit.client.0.smithi154.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T03:43:29.580 INFO:tasks.workunit.client.0.smithi154.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T03:43:29.580 INFO:tasks.workunit.client.0.smithi154.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_lock_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock_pp.xml 2>&1 | tee ceph_test_rados_api_lock_pp.log | sed "s/^/ api_lock_pp: /"' 2024-09-16T03:43:29.581 INFO:tasks.workunit.client.0.smithi154.stderr:+ '[' -z '' ']' 2024-09-16T03:43:29.581 INFO:tasks.workunit.client.0.smithi154.stderr:+ return 2024-09-16T03:43:29.581 INFO:tasks.workunit.client.0.smithi154.stderr:+ case $- in 2024-09-16T03:43:29.581 INFO:tasks.workunit.client.0.smithi154.stderr:+ return 2024-09-16T03:43:29.581 INFO:tasks.workunit.client.0.smithi154.stderr:+ ceph_test_rados_api_lock_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock_pp.xml 2024-09-16T03:43:29.581 INFO:tasks.workunit.client.0.smithi154.stderr:+ tee ceph_test_rados_api_lock_pp.log 2024-09-16T03:43:29.585 INFO:tasks.workunit.client.0.smithi154.stderr:++ printf %25s api_misc 2024-09-16T03:43:29.585 INFO:tasks.workunit.client.0.smithi154.stderr:+ r=' api_misc' 2024-09-16T03:43:29.585 INFO:tasks.workunit.client.0.smithi154.stderr:+ sed 's/^/ api_lock_pp: /' 2024-09-16T03:43:29.586 INFO:tasks.workunit.client.0.smithi154.stderr:++ echo api_misc 2024-09-16T03:43:29.586 INFO:tasks.workunit.client.0.smithi154.stderr:++ awk '{print $1}' 2024-09-16T03:43:29.588 INFO:tasks.workunit.client.0.smithi154.stdout:test api_misc on pid 28181 2024-09-16T03:43:29.589 INFO:tasks.workunit.client.0.smithi154.stderr:+ ff=api_misc 2024-09-16T03:43:29.589 INFO:tasks.workunit.client.0.smithi154.stderr:+ pid=28181 2024-09-16T03:43:29.589 INFO:tasks.workunit.client.0.smithi154.stderr:+ echo 'test api_misc on pid 28181' 2024-09-16T03:43:29.589 INFO:tasks.workunit.client.0.smithi154.stderr:+ pids[$f]=28181 2024-09-16T03:43:29.589 INFO:tasks.workunit.client.0.smithi154.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T03:43:29.589 INFO:tasks.workunit.client.0.smithi154.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T03:43:29.589 INFO:tasks.workunit.client.0.smithi154.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_misc --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc.xml 2>&1 | tee ceph_test_rados_api_misc.log | sed "s/^/ api_misc: /"' 2024-09-16T03:43:29.589 INFO:tasks.workunit.client.0.smithi154.stderr:++ printf %25s api_misc_pp 2024-09-16T03:43:29.590 INFO:tasks.workunit.client.0.smithi154.stderr:+ r=' api_misc_pp' 2024-09-16T03:43:29.591 INFO:tasks.workunit.client.0.smithi154.stderr:++ echo api_misc_pp 2024-09-16T03:43:29.591 INFO:tasks.workunit.client.0.smithi154.stderr:++ awk '{print $1}' 2024-09-16T03:43:29.591 INFO:tasks.workunit.client.0.smithi154.stderr:+ '[' -z '' ']' 2024-09-16T03:43:29.591 INFO:tasks.workunit.client.0.smithi154.stderr:+ return 2024-09-16T03:43:29.591 INFO:tasks.workunit.client.0.smithi154.stderr:+ case $- in 2024-09-16T03:43:29.591 INFO:tasks.workunit.client.0.smithi154.stderr:+ return 2024-09-16T03:43:29.591 INFO:tasks.workunit.client.0.smithi154.stderr:+ ceph_test_rados_api_misc --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc.xml 2024-09-16T03:43:29.591 INFO:tasks.workunit.client.0.smithi154.stderr:+ tee ceph_test_rados_api_misc.log 2024-09-16T03:43:29.591 INFO:tasks.workunit.client.0.smithi154.stderr:+ sed 's/^/ api_misc: /' 2024-09-16T03:43:29.592 INFO:tasks.workunit.client.0.smithi154.stderr:+ ff=api_misc_pp 2024-09-16T03:43:29.592 INFO:tasks.workunit.client.0.smithi154.stderr:+ pid=28196 2024-09-16T03:43:29.592 INFO:tasks.workunit.client.0.smithi154.stderr:+ echo 'test api_misc_pp on pid 28196' 2024-09-16T03:43:29.592 INFO:tasks.workunit.client.0.smithi154.stdout:test api_misc_pp on pid 28196 2024-09-16T03:43:29.593 INFO:tasks.workunit.client.0.smithi154.stderr:+ pids[$f]=28196 2024-09-16T03:43:29.593 INFO:tasks.workunit.client.0.smithi154.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T03:43:29.593 INFO:tasks.workunit.client.0.smithi154.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T03:43:29.593 INFO:tasks.workunit.client.0.smithi154.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_misc_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc_pp.xml 2>&1 | tee ceph_test_rados_api_misc_pp.log | sed "s/^/ api_misc_pp: /"' 2024-09-16T03:43:29.594 INFO:tasks.workunit.client.0.smithi154.stderr:+ '[' -z '' ']' 2024-09-16T03:43:29.594 INFO:tasks.workunit.client.0.smithi154.stderr:+ return 2024-09-16T03:43:29.594 INFO:tasks.workunit.client.0.smithi154.stderr:+ case $- in 2024-09-16T03:43:29.594 INFO:tasks.workunit.client.0.smithi154.stderr:+ return 2024-09-16T03:43:29.594 INFO:tasks.workunit.client.0.smithi154.stderr:+ ceph_test_rados_api_misc_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc_pp.xml 2024-09-16T03:43:29.594 INFO:tasks.workunit.client.0.smithi154.stderr:+ tee ceph_test_rados_api_misc_pp.log 2024-09-16T03:43:29.594 INFO:tasks.workunit.client.0.smithi154.stderr:+ sed 's/^/ api_misc_pp: /' 2024-09-16T03:43:29.595 INFO:tasks.workunit.client.0.smithi154.stderr:++ printf %25s api_tier_pp 2024-09-16T03:43:29.595 INFO:tasks.workunit.client.0.smithi154.stderr:+ r=' api_tier_pp' 2024-09-16T03:43:29.595 INFO:tasks.workunit.client.0.smithi154.stderr:++ awk '{print $1}' 2024-09-16T03:43:29.595 INFO:tasks.workunit.client.0.smithi154.stderr:++ echo api_tier_pp 2024-09-16T03:43:29.597 INFO:tasks.workunit.client.0.smithi154.stderr:+ ff=api_tier_pp 2024-09-16T03:43:29.598 INFO:tasks.workunit.client.0.smithi154.stderr:+ pid=28211 2024-09-16T03:43:29.598 INFO:tasks.workunit.client.0.smithi154.stdout:test api_tier_pp on pid 28211 2024-09-16T03:43:29.599 INFO:tasks.workunit.client.0.smithi154.stderr:+ echo 'test api_tier_pp on pid 28211' 2024-09-16T03:43:29.599 INFO:tasks.workunit.client.0.smithi154.stderr:+ pids[$f]=28211 2024-09-16T03:43:29.599 INFO:tasks.workunit.client.0.smithi154.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T03:43:29.599 INFO:tasks.workunit.client.0.smithi154.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_tier_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_tier_pp.xml 2>&1 | tee ceph_test_rados_api_tier_pp.log | sed "s/^/ api_tier_pp: /"' 2024-09-16T03:43:29.600 INFO:tasks.workunit.client.0.smithi154.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T03:43:29.600 INFO:tasks.workunit.client.0.smithi154.stderr:++ printf %25s api_pool 2024-09-16T03:43:29.600 INFO:tasks.workunit.client.0.smithi154.stderr:+ r=' api_pool' 2024-09-16T03:43:29.600 INFO:tasks.workunit.client.0.smithi154.stderr:+ '[' -z '' ']' 2024-09-16T03:43:29.600 INFO:tasks.workunit.client.0.smithi154.stderr:++ echo api_pool 2024-09-16T03:43:29.600 INFO:tasks.workunit.client.0.smithi154.stderr:+ return 2024-09-16T03:43:29.600 INFO:tasks.workunit.client.0.smithi154.stderr:+ case $- in 2024-09-16T03:43:29.600 INFO:tasks.workunit.client.0.smithi154.stderr:+ return 2024-09-16T03:43:29.600 INFO:tasks.workunit.client.0.smithi154.stderr:++ awk '{print $1}' 2024-09-16T03:43:29.600 INFO:tasks.workunit.client.0.smithi154.stderr:+ ceph_test_rados_api_tier_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_tier_pp.xml 2024-09-16T03:43:29.600 INFO:tasks.workunit.client.0.smithi154.stderr:+ tee ceph_test_rados_api_tier_pp.log 2024-09-16T03:43:29.600 INFO:tasks.workunit.client.0.smithi154.stderr:+ sed 's/^/ api_tier_pp: /' 2024-09-16T03:43:29.601 INFO:tasks.workunit.client.0.smithi154.stderr:+ ff=api_pool 2024-09-16T03:43:29.601 INFO:tasks.workunit.client.0.smithi154.stderr:+ pid=28219 2024-09-16T03:43:29.602 INFO:tasks.workunit.client.0.smithi154.stdout:test api_pool on pid 28219 2024-09-16T03:43:29.603 INFO:tasks.workunit.client.0.smithi154.stderr:+ echo 'test api_pool on pid 28219' 2024-09-16T03:43:29.603 INFO:tasks.workunit.client.0.smithi154.stderr:+ pids[$f]=28219 2024-09-16T03:43:29.603 INFO:tasks.workunit.client.0.smithi154.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T03:43:29.603 INFO:tasks.workunit.client.0.smithi154.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T03:43:29.603 INFO:tasks.workunit.client.0.smithi154.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_pool --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_pool.xml 2>&1 | tee ceph_test_rados_api_pool.log | sed "s/^/ api_pool: /"' 2024-09-16T03:43:29.603 INFO:tasks.workunit.client.0.smithi154.stderr:++ printf %25s api_snapshots 2024-09-16T03:43:29.603 INFO:tasks.workunit.client.0.smithi154.stderr:+ r=' api_snapshots' 2024-09-16T03:43:29.603 INFO:tasks.workunit.client.0.smithi154.stderr:++ echo api_snapshots 2024-09-16T03:43:29.603 INFO:tasks.workunit.client.0.smithi154.stderr:+ '[' -z '' ']' 2024-09-16T03:43:29.603 INFO:tasks.workunit.client.0.smithi154.stderr:+ return 2024-09-16T03:43:29.603 INFO:tasks.workunit.client.0.smithi154.stderr:++ awk '{print $1}' 2024-09-16T03:43:29.604 INFO:tasks.workunit.client.0.smithi154.stderr:+ case $- in 2024-09-16T03:43:29.604 INFO:tasks.workunit.client.0.smithi154.stderr:+ return 2024-09-16T03:43:29.604 INFO:tasks.workunit.client.0.smithi154.stderr:+ ceph_test_rados_api_pool --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_pool.xml 2024-09-16T03:43:29.604 INFO:tasks.workunit.client.0.smithi154.stderr:+ tee ceph_test_rados_api_pool.log 2024-09-16T03:43:29.605 INFO:tasks.workunit.client.0.smithi154.stderr:+ ff=api_snapshots 2024-09-16T03:43:29.605 INFO:tasks.workunit.client.0.smithi154.stdout:test api_snapshots on pid 28227 2024-09-16T03:43:29.606 INFO:tasks.workunit.client.0.smithi154.stderr:+ pid=28227 2024-09-16T03:43:29.606 INFO:tasks.workunit.client.0.smithi154.stderr:+ echo 'test api_snapshots on pid 28227' 2024-09-16T03:43:29.606 INFO:tasks.workunit.client.0.smithi154.stderr:+ pids[$f]=28227 2024-09-16T03:43:29.606 INFO:tasks.workunit.client.0.smithi154.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T03:43:29.606 INFO:tasks.workunit.client.0.smithi154.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T03:43:29.606 INFO:tasks.workunit.client.0.smithi154.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_snapshots --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots.xml 2>&1 | tee ceph_test_rados_api_snapshots.log | sed "s/^/ api_snapshots: /"' 2024-09-16T03:43:29.606 INFO:tasks.workunit.client.0.smithi154.stderr:++ printf %25s api_snapshots_pp 2024-09-16T03:43:29.606 INFO:tasks.workunit.client.0.smithi154.stderr:+ r=' api_snapshots_pp' 2024-09-16T03:43:29.607 INFO:tasks.workunit.client.0.smithi154.stderr:+ '[' -z '' ']' 2024-09-16T03:43:29.607 INFO:tasks.workunit.client.0.smithi154.stderr:++ echo api_snapshots_pp 2024-09-16T03:43:29.607 INFO:tasks.workunit.client.0.smithi154.stderr:+ return 2024-09-16T03:43:29.607 INFO:tasks.workunit.client.0.smithi154.stderr:+ case $- in 2024-09-16T03:43:29.607 INFO:tasks.workunit.client.0.smithi154.stderr:+ sed 's/^/ api_pool: /' 2024-09-16T03:43:29.608 INFO:tasks.workunit.client.0.smithi154.stderr:+ return 2024-09-16T03:43:29.608 INFO:tasks.workunit.client.0.smithi154.stderr:++ awk '{print $1}' 2024-09-16T03:43:29.608 INFO:tasks.workunit.client.0.smithi154.stderr:+ tee ceph_test_rados_api_snapshots.log 2024-09-16T03:43:29.608 INFO:tasks.workunit.client.0.smithi154.stderr:+ sed 's/^/ api_snapshots: /' 2024-09-16T03:43:29.609 INFO:tasks.workunit.client.0.smithi154.stderr:+ ff=api_snapshots_pp 2024-09-16T03:43:29.609 INFO:tasks.workunit.client.0.smithi154.stdout:test api_snapshots_pp on pid 28235 2024-09-16T03:43:29.610 INFO:tasks.workunit.client.0.smithi154.stderr:+ pid=28235 2024-09-16T03:43:29.610 INFO:tasks.workunit.client.0.smithi154.stderr:+ echo 'test api_snapshots_pp on pid 28235' 2024-09-16T03:43:29.610 INFO:tasks.workunit.client.0.smithi154.stderr:+ pids[$f]=28235 2024-09-16T03:43:29.610 INFO:tasks.workunit.client.0.smithi154.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T03:43:29.610 INFO:tasks.workunit.client.0.smithi154.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T03:43:29.610 INFO:tasks.workunit.client.0.smithi154.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_snapshots_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots_pp.xml 2>&1 | tee ceph_test_rados_api_snapshots_pp.log | sed "s/^/ api_snapshots_pp: /"' 2024-09-16T03:43:29.611 INFO:tasks.workunit.client.0.smithi154.stderr:++ printf %25s api_stat 2024-09-16T03:43:29.611 INFO:tasks.workunit.client.0.smithi154.stderr:+ ceph_test_rados_api_snapshots --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots.xml 2024-09-16T03:43:29.611 INFO:tasks.workunit.client.0.smithi154.stderr:+ r=' api_stat' 2024-09-16T03:43:29.612 INFO:tasks.workunit.client.0.smithi154.stderr:+ '[' -z '' ']' 2024-09-16T03:43:29.612 INFO:tasks.workunit.client.0.smithi154.stderr:++ awk '{print $1}' 2024-09-16T03:43:29.612 INFO:tasks.workunit.client.0.smithi154.stderr:+ return 2024-09-16T03:43:29.612 INFO:tasks.workunit.client.0.smithi154.stderr:+ case $- in 2024-09-16T03:43:29.612 INFO:tasks.workunit.client.0.smithi154.stderr:+ return 2024-09-16T03:43:29.612 INFO:tasks.workunit.client.0.smithi154.stderr:++ echo api_stat 2024-09-16T03:43:29.612 INFO:tasks.workunit.client.0.smithi154.stderr:+ sed 's/^/ api_snapshots_pp: /' 2024-09-16T03:43:29.612 INFO:tasks.workunit.client.0.smithi154.stderr:+ tee ceph_test_rados_api_snapshots_pp.log 2024-09-16T03:43:29.613 INFO:tasks.workunit.client.0.smithi154.stdout:test api_stat on pid 28243 2024-09-16T03:43:29.613 INFO:tasks.workunit.client.0.smithi154.stderr:+ ff=api_stat 2024-09-16T03:43:29.614 INFO:tasks.workunit.client.0.smithi154.stderr:+ pid=28243 2024-09-16T03:43:29.614 INFO:tasks.workunit.client.0.smithi154.stderr:+ echo 'test api_stat on pid 28243' 2024-09-16T03:43:29.614 INFO:tasks.workunit.client.0.smithi154.stderr:+ pids[$f]=28243 2024-09-16T03:43:29.614 INFO:tasks.workunit.client.0.smithi154.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T03:43:29.614 INFO:tasks.workunit.client.0.smithi154.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T03:43:29.614 INFO:tasks.workunit.client.0.smithi154.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_stat --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat.xml 2>&1 | tee ceph_test_rados_api_stat.log | sed "s/^/ api_stat: /"' 2024-09-16T03:43:29.614 INFO:tasks.workunit.client.0.smithi154.stderr:+ ceph_test_rados_api_snapshots_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots_pp.xml 2024-09-16T03:43:29.615 INFO:tasks.workunit.client.0.smithi154.stderr:+ '[' -z '' ']' 2024-09-16T03:43:29.615 INFO:tasks.workunit.client.0.smithi154.stderr:+ return 2024-09-16T03:43:29.615 INFO:tasks.workunit.client.0.smithi154.stderr:+ case $- in 2024-09-16T03:43:29.615 INFO:tasks.workunit.client.0.smithi154.stderr:+ return 2024-09-16T03:43:29.616 INFO:tasks.workunit.client.0.smithi154.stderr:+ tee ceph_test_rados_api_stat.log 2024-09-16T03:43:29.616 INFO:tasks.workunit.client.0.smithi154.stderr:+ ceph_test_rados_api_stat --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat.xml 2024-09-16T03:43:29.616 INFO:tasks.workunit.client.0.smithi154.stderr:+ sed 's/^/ api_stat: /' 2024-09-16T03:43:29.621 INFO:tasks.workunit.client.0.smithi154.stderr:++ printf %25s api_stat_pp 2024-09-16T03:43:29.621 INFO:tasks.workunit.client.0.smithi154.stderr:+ r=' api_stat_pp' 2024-09-16T03:43:29.622 INFO:tasks.workunit.client.0.smithi154.stderr:++ awk '{print $1}' 2024-09-16T03:43:29.625 INFO:tasks.workunit.client.0.smithi154.stderr:++ echo api_stat_pp 2024-09-16T03:43:29.626 INFO:tasks.workunit.client.0.smithi154.stdout:test api_stat_pp on pid 28260 2024-09-16T03:43:29.626 INFO:tasks.workunit.client.0.smithi154.stderr:+ ff=api_stat_pp 2024-09-16T03:43:29.626 INFO:tasks.workunit.client.0.smithi154.stderr:+ pid=28260 2024-09-16T03:43:29.626 INFO:tasks.workunit.client.0.smithi154.stderr:+ echo 'test api_stat_pp on pid 28260' 2024-09-16T03:43:29.626 INFO:tasks.workunit.client.0.smithi154.stderr:+ pids[$f]=28260 2024-09-16T03:43:29.626 INFO:tasks.workunit.client.0.smithi154.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T03:43:29.626 INFO:tasks.workunit.client.0.smithi154.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T03:43:29.627 INFO:tasks.workunit.client.0.smithi154.stderr:++ printf %25s api_watch_notify 2024-09-16T03:43:29.627 INFO:tasks.workunit.client.0.smithi154.stderr:+ r=' api_watch_notify' 2024-09-16T03:43:29.628 INFO:tasks.workunit.client.0.smithi154.stderr:++ echo api_watch_notify 2024-09-16T03:43:29.628 INFO:tasks.workunit.client.0.smithi154.stderr:++ awk '{print $1}' 2024-09-16T03:43:29.631 INFO:tasks.workunit.client.0.smithi154.stdout:test api_watch_notify on pid 28269 2024-09-16T03:43:29.631 INFO:tasks.workunit.client.0.smithi154.stderr:+ ff=api_watch_notify 2024-09-16T03:43:29.631 INFO:tasks.workunit.client.0.smithi154.stderr:+ pid=28269 2024-09-16T03:43:29.631 INFO:tasks.workunit.client.0.smithi154.stderr:+ echo 'test api_watch_notify on pid 28269' 2024-09-16T03:43:29.631 INFO:tasks.workunit.client.0.smithi154.stderr:+ pids[$f]=28269 2024-09-16T03:43:29.632 INFO:tasks.workunit.client.0.smithi154.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T03:43:29.632 INFO:tasks.workunit.client.0.smithi154.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T03:43:29.632 INFO:tasks.workunit.client.0.smithi154.stderr:++ printf %25s api_watch_notify_pp 2024-09-16T03:43:29.632 INFO:tasks.workunit.client.0.smithi154.stderr:+ r=' api_watch_notify_pp' 2024-09-16T03:43:29.632 INFO:tasks.workunit.client.0.smithi154.stderr:++ echo api_watch_notify_pp 2024-09-16T03:43:29.632 INFO:tasks.workunit.client.0.smithi154.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_watch_notify --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify.xml 2>&1 | tee ceph_test_rados_api_watch_notify.log | sed "s/^/ api_watch_notify: /"' 2024-09-16T03:43:29.634 INFO:tasks.workunit.client.0.smithi154.stderr:+ '[' -z '' ']' 2024-09-16T03:43:29.634 INFO:tasks.workunit.client.0.smithi154.stderr:+ return 2024-09-16T03:43:29.634 INFO:tasks.workunit.client.0.smithi154.stderr:+ case $- in 2024-09-16T03:43:29.634 INFO:tasks.workunit.client.0.smithi154.stderr:+ return 2024-09-16T03:43:29.634 INFO:tasks.workunit.client.0.smithi154.stderr:++ awk '{print $1}' 2024-09-16T03:43:29.637 INFO:tasks.workunit.client.0.smithi154.stdout:test api_watch_notify_pp on pid 28280 2024-09-16T03:43:29.637 INFO:tasks.workunit.client.0.smithi154.stderr:+ ff=api_watch_notify_pp 2024-09-16T03:43:29.637 INFO:tasks.workunit.client.0.smithi154.stderr:+ pid=28280 2024-09-16T03:43:29.637 INFO:tasks.workunit.client.0.smithi154.stderr:+ echo 'test api_watch_notify_pp on pid 28280' 2024-09-16T03:43:29.637 INFO:tasks.workunit.client.0.smithi154.stderr:+ pids[$f]=28280 2024-09-16T03:43:29.637 INFO:tasks.workunit.client.0.smithi154.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T03:43:29.637 INFO:tasks.workunit.client.0.smithi154.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T03:43:29.637 INFO:tasks.workunit.client.0.smithi154.stderr:+ ceph_test_rados_api_watch_notify --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify.xml 2024-09-16T03:43:29.643 INFO:tasks.workunit.client.0.smithi154.stderr:+ sed 's/^/ api_watch_notify: /' 2024-09-16T03:43:29.643 INFO:tasks.workunit.client.0.smithi154.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_stat_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat_pp.xml 2>&1 | tee ceph_test_rados_api_stat_pp.log | sed "s/^/ api_stat_pp: /"' 2024-09-16T03:43:29.643 INFO:tasks.workunit.client.0.smithi154.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_watch_notify_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify_pp.xml 2>&1 | tee ceph_test_rados_api_watch_notify_pp.log | sed "s/^/ api_watch_notify_pp: /"' 2024-09-16T03:43:29.643 INFO:tasks.workunit.client.0.smithi154.stderr:+ tee ceph_test_rados_api_watch_notify.log 2024-09-16T03:43:29.644 INFO:tasks.workunit.client.0.smithi154.stderr:+ '[' -z '' ']' 2024-09-16T03:43:29.644 INFO:tasks.workunit.client.0.smithi154.stderr:+ return 2024-09-16T03:43:29.644 INFO:tasks.workunit.client.0.smithi154.stderr:+ case $- in 2024-09-16T03:43:29.644 INFO:tasks.workunit.client.0.smithi154.stderr:+ return 2024-09-16T03:43:29.645 INFO:tasks.workunit.client.0.smithi154.stderr:+ ceph_test_rados_api_stat_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat_pp.xml 2024-09-16T03:43:29.650 INFO:tasks.workunit.client.0.smithi154.stderr:++ printf %25s api_cmd 2024-09-16T03:43:29.651 INFO:tasks.workunit.client.0.smithi154.stderr:+ '[' -z '' ']' 2024-09-16T03:43:29.652 INFO:tasks.workunit.client.0.smithi154.stderr:+ r=' api_cmd' 2024-09-16T03:43:29.652 INFO:tasks.workunit.client.0.smithi154.stderr:+ return 2024-09-16T03:43:29.652 INFO:tasks.workunit.client.0.smithi154.stderr:+ case $- in 2024-09-16T03:43:29.652 INFO:tasks.workunit.client.0.smithi154.stderr:+ return 2024-09-16T03:43:29.653 INFO:tasks.workunit.client.0.smithi154.stderr:+ sed 's/^/ api_watch_notify_pp: /' 2024-09-16T03:43:29.653 INFO:tasks.workunit.client.0.smithi154.stderr:+ tee ceph_test_rados_api_watch_notify_pp.log 2024-09-16T03:43:29.653 INFO:tasks.workunit.client.0.smithi154.stderr:++ awk '{print $1}' 2024-09-16T03:43:29.654 INFO:tasks.workunit.client.0.smithi154.stderr:+ ceph_test_rados_api_watch_notify_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify_pp.xml 2024-09-16T03:43:29.654 INFO:tasks.workunit.client.0.smithi154.stderr:+ sed 's/^/ api_stat_pp: /' 2024-09-16T03:43:29.656 INFO:tasks.workunit.client.0.smithi154.stderr:++ echo api_cmd 2024-09-16T03:43:29.656 INFO:tasks.workunit.client.0.smithi154.stderr:+ tee ceph_test_rados_api_stat_pp.log 2024-09-16T03:43:29.657 INFO:tasks.workunit.client.0.smithi154.stderr:+ ff=api_cmd 2024-09-16T03:43:29.657 INFO:tasks.workunit.client.0.smithi154.stderr:+ pid=28322 2024-09-16T03:43:29.657 INFO:tasks.workunit.client.0.smithi154.stderr:+ echo 'test api_cmd on pid 28322' 2024-09-16T03:43:29.657 INFO:tasks.workunit.client.0.smithi154.stdout:test api_cmd on pid 28322 2024-09-16T03:43:29.658 INFO:tasks.workunit.client.0.smithi154.stderr:+ pids[$f]=28322 2024-09-16T03:43:29.658 INFO:tasks.workunit.client.0.smithi154.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T03:43:29.658 INFO:tasks.workunit.client.0.smithi154.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T03:43:29.658 INFO:tasks.workunit.client.0.smithi154.stderr:++ printf %25s api_cmd_pp 2024-09-16T03:43:29.658 INFO:tasks.workunit.client.0.smithi154.stderr:+ r=' api_cmd_pp' 2024-09-16T03:43:29.659 INFO:tasks.workunit.client.0.smithi154.stderr:++ echo api_cmd_pp 2024-09-16T03:43:29.659 INFO:tasks.workunit.client.0.smithi154.stderr:++ awk '{print $1}' 2024-09-16T03:43:29.659 INFO:tasks.workunit.client.0.smithi154.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cmd --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd.xml 2>&1 | tee ceph_test_rados_api_cmd.log | sed "s/^/ api_cmd: /"' 2024-09-16T03:43:29.661 INFO:tasks.workunit.client.0.smithi154.stderr:+ '[' -z '' ']' 2024-09-16T03:43:29.661 INFO:tasks.workunit.client.0.smithi154.stderr:+ return 2024-09-16T03:43:29.661 INFO:tasks.workunit.client.0.smithi154.stderr:+ case $- in 2024-09-16T03:43:29.661 INFO:tasks.workunit.client.0.smithi154.stderr:+ return 2024-09-16T03:43:29.661 INFO:tasks.workunit.client.0.smithi154.stderr:+ ff=api_cmd_pp 2024-09-16T03:43:29.662 INFO:tasks.workunit.client.0.smithi154.stdout:test api_cmd_pp on pid 28346 2024-09-16T03:43:29.663 INFO:tasks.workunit.client.0.smithi154.stderr:+ pid=28346 2024-09-16T03:43:29.663 INFO:tasks.workunit.client.0.smithi154.stderr:+ echo 'test api_cmd_pp on pid 28346' 2024-09-16T03:43:29.663 INFO:tasks.workunit.client.0.smithi154.stderr:+ pids[$f]=28346 2024-09-16T03:43:29.663 INFO:tasks.workunit.client.0.smithi154.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T03:43:29.663 INFO:tasks.workunit.client.0.smithi154.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T03:43:29.663 INFO:tasks.workunit.client.0.smithi154.stderr:+ tee ceph_test_rados_api_cmd.log 2024-09-16T03:43:29.663 INFO:tasks.workunit.client.0.smithi154.stderr:+ sed 's/^/ api_cmd: /' 2024-09-16T03:43:29.663 INFO:tasks.workunit.client.0.smithi154.stderr:++ printf %25s api_service 2024-09-16T03:43:29.663 INFO:tasks.workunit.client.0.smithi154.stderr:+ r=' api_service' 2024-09-16T03:43:29.664 INFO:tasks.workunit.client.0.smithi154.stderr:+ ceph_test_rados_api_cmd --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd.xml 2024-09-16T03:43:29.664 INFO:tasks.workunit.client.0.smithi154.stderr:++ echo api_service 2024-09-16T03:43:29.664 INFO:tasks.workunit.client.0.smithi154.stderr:++ awk '{print $1}' 2024-09-16T03:43:29.664 INFO:tasks.workunit.client.0.smithi154.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cmd_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd_pp.xml 2>&1 | tee ceph_test_rados_api_cmd_pp.log | sed "s/^/ api_cmd_pp: /"' 2024-09-16T03:43:29.665 INFO:tasks.workunit.client.0.smithi154.stderr:+ '[' -z '' ']' 2024-09-16T03:43:29.665 INFO:tasks.workunit.client.0.smithi154.stderr:+ return 2024-09-16T03:43:29.665 INFO:tasks.workunit.client.0.smithi154.stderr:+ case $- in 2024-09-16T03:43:29.665 INFO:tasks.workunit.client.0.smithi154.stderr:+ return 2024-09-16T03:43:29.665 INFO:tasks.workunit.client.0.smithi154.stderr:+ ceph_test_rados_api_cmd_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd_pp.xml 2024-09-16T03:43:29.665 INFO:tasks.workunit.client.0.smithi154.stderr:+ tee ceph_test_rados_api_cmd_pp.log 2024-09-16T03:43:29.665 INFO:tasks.workunit.client.0.smithi154.stdout:test api_service on pid 28356 2024-09-16T03:43:29.666 INFO:tasks.workunit.client.0.smithi154.stderr:+ ff=api_service 2024-09-16T03:43:29.667 INFO:tasks.workunit.client.0.smithi154.stderr:+ sed 's/^/ api_cmd_pp: /' 2024-09-16T03:43:29.667 INFO:tasks.workunit.client.0.smithi154.stderr:+ pid=28356 2024-09-16T03:43:29.667 INFO:tasks.workunit.client.0.smithi154.stderr:+ echo 'test api_service on pid 28356' 2024-09-16T03:43:29.667 INFO:tasks.workunit.client.0.smithi154.stderr:+ pids[$f]=28356 2024-09-16T03:43:29.667 INFO:tasks.workunit.client.0.smithi154.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T03:43:29.667 INFO:tasks.workunit.client.0.smithi154.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T03:43:29.667 INFO:tasks.workunit.client.0.smithi154.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_service --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service.xml 2>&1 | tee ceph_test_rados_api_service.log | sed "s/^/ api_service: /"' 2024-09-16T03:43:29.667 INFO:tasks.workunit.client.0.smithi154.stderr:++ printf %25s api_service_pp 2024-09-16T03:43:29.667 INFO:tasks.workunit.client.0.smithi154.stderr:+ r=' api_service_pp' 2024-09-16T03:43:29.667 INFO:tasks.workunit.client.0.smithi154.stderr:+ '[' -z '' ']' 2024-09-16T03:43:29.667 INFO:tasks.workunit.client.0.smithi154.stderr:+ return 2024-09-16T03:43:29.667 INFO:tasks.workunit.client.0.smithi154.stderr:+ case $- in 2024-09-16T03:43:29.667 INFO:tasks.workunit.client.0.smithi154.stderr:+ return 2024-09-16T03:43:29.668 INFO:tasks.workunit.client.0.smithi154.stderr:+ ceph_test_rados_api_service --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service.xml 2024-09-16T03:43:29.668 INFO:tasks.workunit.client.0.smithi154.stderr:+ sed 's/^/ api_service: /' 2024-09-16T03:43:29.668 INFO:tasks.workunit.client.0.smithi154.stderr:++ echo api_service_pp 2024-09-16T03:43:29.668 INFO:tasks.workunit.client.0.smithi154.stderr:++ awk '{print $1}' 2024-09-16T03:43:29.669 INFO:tasks.workunit.client.0.smithi154.stderr:+ tee ceph_test_rados_api_service.log 2024-09-16T03:43:29.670 INFO:tasks.workunit.client.0.smithi154.stdout:test api_service_pp on pid 28364 2024-09-16T03:43:29.670 INFO:tasks.workunit.client.0.smithi154.stderr:+ ff=api_service_pp 2024-09-16T03:43:29.671 INFO:tasks.workunit.client.0.smithi154.stderr:+ pid=28364 2024-09-16T03:43:29.671 INFO:tasks.workunit.client.0.smithi154.stderr:+ echo 'test api_service_pp on pid 28364' 2024-09-16T03:43:29.671 INFO:tasks.workunit.client.0.smithi154.stderr:+ pids[$f]=28364 2024-09-16T03:43:29.671 INFO:tasks.workunit.client.0.smithi154.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T03:43:29.671 INFO:tasks.workunit.client.0.smithi154.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T03:43:29.671 INFO:tasks.workunit.client.0.smithi154.stderr:++ printf %25s api_c_write_operations 2024-09-16T03:43:29.671 INFO:tasks.workunit.client.0.smithi154.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_service_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service_pp.xml 2>&1 | tee ceph_test_rados_api_service_pp.log | sed "s/^/ api_service_pp: /"' 2024-09-16T03:43:29.671 INFO:tasks.workunit.client.0.smithi154.stderr:+ r=' api_c_write_operations' 2024-09-16T03:43:29.672 INFO:tasks.workunit.client.0.smithi154.stderr:++ awk '{print $1}' 2024-09-16T03:43:29.672 INFO:tasks.workunit.client.0.smithi154.stderr:+ '[' -z '' ']' 2024-09-16T03:43:29.672 INFO:tasks.workunit.client.0.smithi154.stderr:+ return 2024-09-16T03:43:29.672 INFO:tasks.workunit.client.0.smithi154.stderr:+ case $- in 2024-09-16T03:43:29.672 INFO:tasks.workunit.client.0.smithi154.stderr:+ return 2024-09-16T03:43:29.672 INFO:tasks.workunit.client.0.smithi154.stderr:+ sed 's/^/ api_service_pp: /' 2024-09-16T03:43:29.673 INFO:tasks.workunit.client.0.smithi154.stderr:+ tee ceph_test_rados_api_service_pp.log 2024-09-16T03:43:29.673 INFO:tasks.workunit.client.0.smithi154.stderr:+ ceph_test_rados_api_service_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service_pp.xml 2024-09-16T03:43:29.675 INFO:tasks.workunit.client.0.smithi154.stderr:++ echo api_c_write_operations 2024-09-16T03:43:29.675 INFO:tasks.workunit.client.0.smithi154.stdout:test api_c_write_operations on pid 28372 2024-09-16T03:43:29.676 INFO:tasks.workunit.client.0.smithi154.stderr:+ ff=api_c_write_operations 2024-09-16T03:43:29.676 INFO:tasks.workunit.client.0.smithi154.stderr:+ pid=28372 2024-09-16T03:43:29.676 INFO:tasks.workunit.client.0.smithi154.stderr:+ echo 'test api_c_write_operations on pid 28372' 2024-09-16T03:43:29.676 INFO:tasks.workunit.client.0.smithi154.stderr:+ pids[$f]=28372 2024-09-16T03:43:29.676 INFO:tasks.workunit.client.0.smithi154.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T03:43:29.676 INFO:tasks.workunit.client.0.smithi154.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T03:43:29.677 INFO:tasks.workunit.client.0.smithi154.stderr:++ printf %25s api_c_read_operations 2024-09-16T03:43:29.677 INFO:tasks.workunit.client.0.smithi154.stderr:+ r=' api_c_read_operations' 2024-09-16T03:43:29.677 INFO:tasks.workunit.client.0.smithi154.stderr:++ echo api_c_read_operations 2024-09-16T03:43:29.677 INFO:tasks.workunit.client.0.smithi154.stderr:++ awk '{print $1}' 2024-09-16T03:43:29.678 INFO:tasks.workunit.client.0.smithi154.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_c_write_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_write_operations.xml 2>&1 | tee ceph_test_rados_api_c_write_operations.log | sed "s/^/ api_c_write_operations: /"' 2024-09-16T03:43:29.679 INFO:tasks.workunit.client.0.smithi154.stderr:+ '[' -z '' ']' 2024-09-16T03:43:29.679 INFO:tasks.workunit.client.0.smithi154.stderr:+ return 2024-09-16T03:43:29.679 INFO:tasks.workunit.client.0.smithi154.stderr:+ case $- in 2024-09-16T03:43:29.679 INFO:tasks.workunit.client.0.smithi154.stderr:+ return 2024-09-16T03:43:29.680 INFO:tasks.workunit.client.0.smithi154.stdout:test api_c_read_operations on pid 28382 2024-09-16T03:43:29.680 INFO:tasks.workunit.client.0.smithi154.stderr:+ ff=api_c_read_operations 2024-09-16T03:43:29.680 INFO:tasks.workunit.client.0.smithi154.stderr:+ pid=28382 2024-09-16T03:43:29.681 INFO:tasks.workunit.client.0.smithi154.stderr:+ echo 'test api_c_read_operations on pid 28382' 2024-09-16T03:43:29.681 INFO:tasks.workunit.client.0.smithi154.stderr:+ pids[$f]=28382 2024-09-16T03:43:29.681 INFO:tasks.workunit.client.0.smithi154.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T03:43:29.681 INFO:tasks.workunit.client.0.smithi154.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T03:43:29.681 INFO:tasks.workunit.client.0.smithi154.stderr:+ sed 's/^/ api_c_write_operations: /' 2024-09-16T03:43:29.681 INFO:tasks.workunit.client.0.smithi154.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_c_read_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_read_operations.xml 2>&1 | tee ceph_test_rados_api_c_read_operations.log | sed "s/^/ api_c_read_operations: /"' 2024-09-16T03:43:29.681 INFO:tasks.workunit.client.0.smithi154.stderr:++ printf %25s api_cls_remote_reads 2024-09-16T03:43:29.682 INFO:tasks.workunit.client.0.smithi154.stderr:+ r=' api_cls_remote_reads' 2024-09-16T03:43:29.682 INFO:tasks.workunit.client.0.smithi154.stderr:+ ceph_test_rados_api_c_write_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_write_operations.xml 2024-09-16T03:43:29.682 INFO:tasks.workunit.client.0.smithi154.stderr:+ tee ceph_test_rados_api_c_write_operations.log 2024-09-16T03:43:29.684 INFO:tasks.workunit.client.0.smithi154.stderr:+ '[' -z '' ']' 2024-09-16T03:43:29.684 INFO:tasks.workunit.client.0.smithi154.stderr:+ return 2024-09-16T03:43:29.684 INFO:tasks.workunit.client.0.smithi154.stderr:+ case $- in 2024-09-16T03:43:29.684 INFO:tasks.workunit.client.0.smithi154.stderr:+ return 2024-09-16T03:43:29.684 INFO:tasks.workunit.client.0.smithi154.stderr:++ echo api_cls_remote_reads 2024-09-16T03:43:29.685 INFO:tasks.workunit.client.0.smithi154.stderr:++ awk '{print $1}' 2024-09-16T03:43:29.685 INFO:tasks.workunit.client.0.smithi154.stderr:+ tee ceph_test_rados_api_c_read_operations.log 2024-09-16T03:43:29.685 INFO:tasks.workunit.client.0.smithi154.stderr:+ ceph_test_rados_api_c_read_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_read_operations.xml 2024-09-16T03:43:29.686 INFO:tasks.workunit.client.0.smithi154.stderr:+ sed 's/^/ api_c_read_operations: /' 2024-09-16T03:43:29.688 INFO:tasks.workunit.client.0.smithi154.stdout:test api_cls_remote_reads on pid 28399 2024-09-16T03:43:29.688 INFO:tasks.workunit.client.0.smithi154.stderr:+ ff=api_cls_remote_reads 2024-09-16T03:43:29.689 INFO:tasks.workunit.client.0.smithi154.stderr:+ pid=28399 2024-09-16T03:43:29.689 INFO:tasks.workunit.client.0.smithi154.stderr:+ echo 'test api_cls_remote_reads on pid 28399' 2024-09-16T03:43:29.689 INFO:tasks.workunit.client.0.smithi154.stderr:+ pids[$f]=28399 2024-09-16T03:43:29.689 INFO:tasks.workunit.client.0.smithi154.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T03:43:29.689 INFO:tasks.workunit.client.0.smithi154.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T03:43:29.689 INFO:tasks.workunit.client.0.smithi154.stderr:++ printf %25s list_parallel 2024-09-16T03:43:29.689 INFO:tasks.workunit.client.0.smithi154.stderr:+ r=' list_parallel' 2024-09-16T03:43:29.690 INFO:tasks.workunit.client.0.smithi154.stderr:++ echo list_parallel 2024-09-16T03:43:29.690 INFO:tasks.workunit.client.0.smithi154.stderr:++ awk '{print $1}' 2024-09-16T03:43:29.692 INFO:tasks.workunit.client.0.smithi154.stdout:test list_parallel on pid 28404 2024-09-16T03:43:29.693 INFO:tasks.workunit.client.0.smithi154.stderr:+ ff=list_parallel 2024-09-16T03:43:29.693 INFO:tasks.workunit.client.0.smithi154.stderr:+ pid=28404 2024-09-16T03:43:29.693 INFO:tasks.workunit.client.0.smithi154.stderr:+ echo 'test list_parallel on pid 28404' 2024-09-16T03:43:29.693 INFO:tasks.workunit.client.0.smithi154.stderr:+ pids[$f]=28404 2024-09-16T03:43:29.693 INFO:tasks.workunit.client.0.smithi154.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T03:43:29.693 INFO:tasks.workunit.client.0.smithi154.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T03:43:29.693 INFO:tasks.workunit.client.0.smithi154.stderr:++ printf %25s open_pools_parallel 2024-09-16T03:43:29.693 INFO:tasks.workunit.client.0.smithi154.stderr:+ r=' open_pools_parallel' 2024-09-16T03:43:29.694 INFO:tasks.workunit.client.0.smithi154.stderr:++ echo open_pools_parallel 2024-09-16T03:43:29.694 INFO:tasks.workunit.client.0.smithi154.stderr:++ awk '{print $1}' 2024-09-16T03:43:29.696 INFO:tasks.workunit.client.0.smithi154.stdout:test open_pools_parallel on pid 28415 2024-09-16T03:43:29.696 INFO:tasks.workunit.client.0.smithi154.stderr:+ ff=open_pools_parallel 2024-09-16T03:43:29.696 INFO:tasks.workunit.client.0.smithi154.stderr:+ pid=28415 2024-09-16T03:43:29.696 INFO:tasks.workunit.client.0.smithi154.stderr:+ echo 'test open_pools_parallel on pid 28415' 2024-09-16T03:43:29.696 INFO:tasks.workunit.client.0.smithi154.stderr:+ pids[$f]=28415 2024-09-16T03:43:29.696 INFO:tasks.workunit.client.0.smithi154.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-16T03:43:29.696 INFO:tasks.workunit.client.0.smithi154.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T03:43:29.696 INFO:tasks.workunit.client.0.smithi154.stderr:++ printf %25s delete_pools_parallel 2024-09-16T03:43:29.697 INFO:tasks.workunit.client.0.smithi154.stderr:+ r=' delete_pools_parallel' 2024-09-16T03:43:29.697 INFO:tasks.workunit.client.0.smithi154.stderr:++ awk '{print $1}' 2024-09-16T03:43:29.698 INFO:tasks.workunit.client.0.smithi154.stderr:++ echo delete_pools_parallel 2024-09-16T03:43:29.698 INFO:tasks.workunit.client.0.smithi154.stderr:+ bash -o pipefail -exc 'ceph_test_rados_list_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/list_parallel.xml 2>&1 | tee ceph_test_rados_list_parallel.log | sed "s/^/ list_parallel: /"' 2024-09-16T03:43:29.699 INFO:tasks.workunit.client.0.smithi154.stdout:test delete_pools_parallel on pid 28423 2024-09-16T03:43:29.700 INFO:tasks.workunit.client.0.smithi154.stderr:+ ff=delete_pools_parallel 2024-09-16T03:43:29.700 INFO:tasks.workunit.client.0.smithi154.stderr:+ pid=28423 2024-09-16T03:43:29.700 INFO:tasks.workunit.client.0.smithi154.stderr:+ echo 'test delete_pools_parallel on pid 28423' 2024-09-16T03:43:29.700 INFO:tasks.workunit.client.0.smithi154.stderr:+ pids[$f]=28423 2024-09-16T03:43:29.701 INFO:tasks.workunit.client.0.smithi154.stderr:+ '[' -z '' ']' 2024-09-16T03:43:29.701 INFO:tasks.workunit.client.0.smithi154.stderr:+ return 2024-09-16T03:43:29.701 INFO:tasks.workunit.client.0.smithi154.stderr:+ case $- in 2024-09-16T03:43:29.701 INFO:tasks.workunit.client.0.smithi154.stderr:+ return 2024-09-16T03:43:29.701 INFO:tasks.workunit.client.0.smithi154.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-09-16T03:43:29.701 INFO:tasks.workunit.client.0.smithi154.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T03:43:29.701 INFO:tasks.workunit.client.0.smithi154.stderr:+ tee ceph_test_rados_list_parallel.log 2024-09-16T03:43:29.701 INFO:tasks.workunit.client.0.smithi154.stderr:+ sed 's/^/ list_parallel: /' 2024-09-16T03:43:29.702 INFO:tasks.workunit.client.0.smithi154.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cls_remote_reads --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cls_remote_reads.xml 2>&1 | tee ceph_test_rados_api_cls_remote_reads.log | sed "s/^/ api_cls_remote_reads: /"' 2024-09-16T03:43:29.702 INFO:tasks.workunit.client.0.smithi154.stderr:++ printf %25s cls 2024-09-16T03:43:29.702 INFO:tasks.workunit.client.0.smithi154.stderr:+ r=' cls' 2024-09-16T03:43:29.702 INFO:tasks.workunit.client.0.smithi154.stderr:+ ceph_test_rados_list_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/list_parallel.xml 2024-09-16T03:43:29.703 INFO:tasks.workunit.client.0.smithi154.stderr:+ bash -o pipefail -exc 'ceph_test_rados_delete_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/delete_pools_parallel.xml 2>&1 | tee ceph_test_rados_delete_pools_parallel.log | sed "s/^/ delete_pools_parallel: /"' 2024-09-16T03:43:29.703 INFO:tasks.workunit.client.0.smithi154.stderr:+ '[' -z '' ']' 2024-09-16T03:43:29.703 INFO:tasks.workunit.client.0.smithi154.stderr:+ return 2024-09-16T03:43:29.703 INFO:tasks.workunit.client.0.smithi154.stderr:+ case $- in 2024-09-16T03:43:29.703 INFO:tasks.workunit.client.0.smithi154.stderr:+ return 2024-09-16T03:43:29.704 INFO:tasks.workunit.client.0.smithi154.stderr:+ '[' -z '' ']' 2024-09-16T03:43:29.704 INFO:tasks.workunit.client.0.smithi154.stderr:+ return 2024-09-16T03:43:29.704 INFO:tasks.workunit.client.0.smithi154.stderr:+ case $- in 2024-09-16T03:43:29.704 INFO:tasks.workunit.client.0.smithi154.stderr:+ return 2024-09-16T03:43:29.704 INFO:tasks.workunit.client.0.smithi154.stderr:++ echo cls 2024-09-16T03:43:29.705 INFO:tasks.workunit.client.0.smithi154.stderr:+ ceph_test_rados_delete_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/delete_pools_parallel.xml 2024-09-16T03:43:29.705 INFO:tasks.workunit.client.0.smithi154.stderr:+ ceph_test_rados_api_cls_remote_reads --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cls_remote_reads.xml 2024-09-16T03:43:29.705 INFO:tasks.workunit.client.0.smithi154.stderr:+ sed 's/^/ api_cls_remote_reads: /' 2024-09-16T03:43:29.707 INFO:tasks.workunit.client.0.smithi154.stderr:+ bash -o pipefail -exc 'ceph_test_rados_open_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/open_pools_parallel.xml 2>&1 | tee ceph_test_rados_open_pools_parallel.log | sed "s/^/ open_pools_parallel: /"' 2024-09-16T03:43:29.707 INFO:tasks.workunit.client.0.smithi154.stderr:+ tee ceph_test_rados_delete_pools_parallel.log 2024-09-16T03:43:29.707 INFO:tasks.workunit.client.0.smithi154.stderr:++ awk '{print $1}' 2024-09-16T03:43:29.708 INFO:tasks.workunit.client.0.smithi154.stderr:+ '[' -z '' ']' 2024-09-16T03:43:29.709 INFO:tasks.workunit.client.0.smithi154.stderr:+ return 2024-09-16T03:43:29.709 INFO:tasks.workunit.client.0.smithi154.stderr:+ case $- in 2024-09-16T03:43:29.709 INFO:tasks.workunit.client.0.smithi154.stderr:+ return 2024-09-16T03:43:29.710 INFO:tasks.workunit.client.0.smithi154.stderr:+ ff=cls 2024-09-16T03:43:29.710 INFO:tasks.workunit.client.0.smithi154.stderr:+ pid=28461 2024-09-16T03:43:29.710 INFO:tasks.workunit.client.0.smithi154.stdout:test cls on pid 28461 2024-09-16T03:43:29.711 INFO:tasks.workunit.client.0.smithi154.stderr:+ echo 'test cls on pid 28461' 2024-09-16T03:43:29.711 INFO:tasks.workunit.client.0.smithi154.stderr:+ pids[$f]=28461 2024-09-16T03:43:29.711 INFO:tasks.workunit.client.0.smithi154.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-09-16T03:43:29.711 INFO:tasks.workunit.client.0.smithi154.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T03:43:29.711 INFO:tasks.workunit.client.0.smithi154.stderr:+ tee ceph_test_rados_open_pools_parallel.log 2024-09-16T03:43:29.711 INFO:tasks.workunit.client.0.smithi154.stderr:++ printf %25s cmd 2024-09-16T03:43:29.711 INFO:tasks.workunit.client.0.smithi154.stderr:+ sed 's/^/ delete_pools_parallel: /' 2024-09-16T03:43:29.712 INFO:tasks.workunit.client.0.smithi154.stderr:+ r=' cmd' 2024-09-16T03:43:29.712 INFO:tasks.workunit.client.0.smithi154.stderr:++ echo cmd 2024-09-16T03:43:29.715 INFO:tasks.workunit.client.0.smithi154.stderr:+ sed 's/^/ open_pools_parallel: /' 2024-09-16T03:43:29.719 INFO:tasks.workunit.client.0.smithi154.stderr:+ tee ceph_test_rados_api_cls_remote_reads.log 2024-09-16T03:43:29.720 INFO:tasks.workunit.client.0.smithi154.stderr:+ ceph_test_rados_open_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/open_pools_parallel.xml 2024-09-16T03:43:29.720 INFO:tasks.workunit.client.0.smithi154.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_cls 2>&1 | tee ceph_test_neorados_cls.log | sed "s/^/ cls: /"' 2024-09-16T03:43:29.720 INFO:tasks.workunit.client.0.smithi154.stderr:++ awk '{print $1}' 2024-09-16T03:43:29.720 INFO:tasks.workunit.client.0.smithi154.stderr:+ '[' -z '' ']' 2024-09-16T03:43:29.727 INFO:tasks.workunit.client.0.smithi154.stderr:+ return 2024-09-16T03:43:29.727 INFO:tasks.workunit.client.0.smithi154.stderr:+ case $- in 2024-09-16T03:43:29.727 INFO:tasks.workunit.client.0.smithi154.stderr:+ return 2024-09-16T03:43:29.727 INFO:tasks.workunit.client.0.smithi154.stderr:+ sed 's/^/ cls: /' 2024-09-16T03:43:29.733 INFO:tasks.workunit.client.0.smithi154.stderr:+ ceph_test_neorados_cls 2024-09-16T03:43:29.733 INFO:tasks.workunit.client.0.smithi154.stderr:+ tee ceph_test_neorados_cls.log 2024-09-16T03:43:29.738 INFO:tasks.workunit.client.0.smithi154.stdout:test cmd on pid 28496 2024-09-16T03:43:29.738 INFO:tasks.workunit.client.0.smithi154.stderr:+ ff=cmd 2024-09-16T03:43:29.738 INFO:tasks.workunit.client.0.smithi154.stderr:+ pid=28496 2024-09-16T03:43:29.738 INFO:tasks.workunit.client.0.smithi154.stderr:+ echo 'test cmd on pid 28496' 2024-09-16T03:43:29.738 INFO:tasks.workunit.client.0.smithi154.stderr:+ pids[$f]=28496 2024-09-16T03:43:29.738 INFO:tasks.workunit.client.0.smithi154.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-09-16T03:43:29.739 INFO:tasks.workunit.client.0.smithi154.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T03:43:29.739 INFO:tasks.workunit.client.0.smithi154.stderr:++ printf %25s handler_error 2024-09-16T03:43:29.739 INFO:tasks.workunit.client.0.smithi154.stderr:+ r=' handler_error' 2024-09-16T03:43:29.739 INFO:tasks.workunit.client.0.smithi154.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_cmd 2>&1 | tee ceph_test_neorados_cmd.log | sed "s/^/ cmd: /"' 2024-09-16T03:43:29.740 INFO:tasks.workunit.client.0.smithi154.stderr:+ '[' -z '' ']' 2024-09-16T03:43:29.740 INFO:tasks.workunit.client.0.smithi154.stderr:+ return 2024-09-16T03:43:29.740 INFO:tasks.workunit.client.0.smithi154.stderr:+ case $- in 2024-09-16T03:43:29.740 INFO:tasks.workunit.client.0.smithi154.stderr:+ return 2024-09-16T03:43:29.741 INFO:tasks.workunit.client.0.smithi154.stderr:+ sed 's/^/ cmd: /' 2024-09-16T03:43:29.742 INFO:tasks.workunit.client.0.smithi154.stderr:+ tee ceph_test_neorados_cmd.log 2024-09-16T03:43:29.743 INFO:tasks.workunit.client.0.smithi154.stderr:++ echo handler_error 2024-09-16T03:43:29.745 INFO:tasks.workunit.client.0.smithi154.stderr:+ ceph_test_neorados_cmd 2024-09-16T03:43:29.745 INFO:tasks.workunit.client.0.smithi154.stderr:++ awk '{print $1}' 2024-09-16T03:43:29.748 INFO:tasks.workunit.client.0.smithi154.stdout:test handler_error on pid 28521 2024-09-16T03:43:29.749 INFO:tasks.workunit.client.0.smithi154.stderr:+ ff=handler_error 2024-09-16T03:43:29.749 INFO:tasks.workunit.client.0.smithi154.stderr:+ pid=28521 2024-09-16T03:43:29.749 INFO:tasks.workunit.client.0.smithi154.stderr:+ echo 'test handler_error on pid 28521' 2024-09-16T03:43:29.749 INFO:tasks.workunit.client.0.smithi154.stderr:+ pids[$f]=28521 2024-09-16T03:43:29.749 INFO:tasks.workunit.client.0.smithi154.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-09-16T03:43:29.749 INFO:tasks.workunit.client.0.smithi154.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T03:43:29.749 INFO:tasks.workunit.client.0.smithi154.stderr:++ printf %25s io 2024-09-16T03:43:29.749 INFO:tasks.workunit.client.0.smithi154.stderr:+ r=' io' 2024-09-16T03:43:29.749 INFO:tasks.workunit.client.0.smithi154.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_handler_error 2>&1 | tee ceph_test_neorados_handler_error.log | sed "s/^/ handler_error: /"' 2024-09-16T03:43:29.750 INFO:tasks.workunit.client.0.smithi154.stderr:+ '[' -z '' ']' 2024-09-16T03:43:29.750 INFO:tasks.workunit.client.0.smithi154.stderr:+ return 2024-09-16T03:43:29.751 INFO:tasks.workunit.client.0.smithi154.stderr:+ case $- in 2024-09-16T03:43:29.751 INFO:tasks.workunit.client.0.smithi154.stderr:+ return 2024-09-16T03:43:29.751 INFO:tasks.workunit.client.0.smithi154.stderr:+ sed 's/^/ handler_error: /' 2024-09-16T03:43:29.752 INFO:tasks.workunit.client.0.smithi154.stderr:++ echo io 2024-09-16T03:43:29.752 INFO:tasks.workunit.client.0.smithi154.stderr:++ awk '{print $1}' 2024-09-16T03:43:29.753 INFO:tasks.workunit.client.0.smithi154.stderr:+ tee ceph_test_neorados_handler_error.log 2024-09-16T03:43:29.754 INFO:tasks.workunit.client.0.smithi154.stderr:+ ceph_test_neorados_handler_error 2024-09-16T03:43:29.755 INFO:tasks.workunit.client.0.smithi154.stdout:test io on pid 28543 2024-09-16T03:43:29.756 INFO:tasks.workunit.client.0.smithi154.stderr:+ ff=io 2024-09-16T03:43:29.756 INFO:tasks.workunit.client.0.smithi154.stderr:+ pid=28543 2024-09-16T03:43:29.756 INFO:tasks.workunit.client.0.smithi154.stderr:+ echo 'test io on pid 28543' 2024-09-16T03:43:29.756 INFO:tasks.workunit.client.0.smithi154.stderr:+ pids[$f]=28543 2024-09-16T03:43:29.756 INFO:tasks.workunit.client.0.smithi154.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-09-16T03:43:29.756 INFO:tasks.workunit.client.0.smithi154.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T03:43:29.756 INFO:tasks.workunit.client.0.smithi154.stderr:++ printf %25s ec_io 2024-09-16T03:43:29.756 INFO:tasks.workunit.client.0.smithi154.stderr:+ r=' ec_io' 2024-09-16T03:43:29.757 INFO:tasks.workunit.client.0.smithi154.stderr:++ awk '{print $1}' 2024-09-16T03:43:29.758 INFO:tasks.workunit.client.0.smithi154.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_io 2>&1 | tee ceph_test_neorados_io.log | sed "s/^/ io: /"' 2024-09-16T03:43:29.758 INFO:tasks.workunit.client.0.smithi154.stderr:++ echo ec_io 2024-09-16T03:43:29.759 INFO:tasks.workunit.client.0.smithi154.stdout:test ec_io on pid 28552 2024-09-16T03:43:29.761 INFO:tasks.workunit.client.0.smithi154.stderr:+ ff=ec_io 2024-09-16T03:43:29.761 INFO:tasks.workunit.client.0.smithi154.stderr:+ '[' -z '' ']' 2024-09-16T03:43:29.761 INFO:tasks.workunit.client.0.smithi154.stderr:+ return 2024-09-16T03:43:29.761 INFO:tasks.workunit.client.0.smithi154.stderr:+ case $- in 2024-09-16T03:43:29.761 INFO:tasks.workunit.client.0.smithi154.stderr:+ return 2024-09-16T03:43:29.761 INFO:tasks.workunit.client.0.smithi154.stderr:+ pid=28552 2024-09-16T03:43:29.761 INFO:tasks.workunit.client.0.smithi154.stderr:+ echo 'test ec_io on pid 28552' 2024-09-16T03:43:29.762 INFO:tasks.workunit.client.0.smithi154.stderr:+ pids[$f]=28552 2024-09-16T03:43:29.762 INFO:tasks.workunit.client.0.smithi154.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-09-16T03:43:29.762 INFO:tasks.workunit.client.0.smithi154.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T03:43:29.762 INFO:tasks.workunit.client.0.smithi154.stderr:+ tee ceph_test_neorados_io.log 2024-09-16T03:43:29.762 INFO:tasks.workunit.client.0.smithi154.stderr:+ ceph_test_neorados_io 2024-09-16T03:43:29.762 INFO:tasks.workunit.client.0.smithi154.stderr:+ sed 's/^/ io: /' 2024-09-16T03:43:29.762 INFO:tasks.workunit.client.0.smithi154.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_ec_io 2>&1 | tee ceph_test_neorados_ec_io.log | sed "s/^/ ec_io: /"' 2024-09-16T03:43:29.763 INFO:tasks.workunit.client.0.smithi154.stderr:+ '[' -z '' ']' 2024-09-16T03:43:29.763 INFO:tasks.workunit.client.0.smithi154.stderr:+ return 2024-09-16T03:43:29.763 INFO:tasks.workunit.client.0.smithi154.stderr:+ case $- in 2024-09-16T03:43:29.763 INFO:tasks.workunit.client.0.smithi154.stderr:+ return 2024-09-16T03:43:29.763 INFO:tasks.workunit.client.0.smithi154.stderr:+ ceph_test_neorados_ec_io 2024-09-16T03:43:29.764 INFO:tasks.workunit.client.0.smithi154.stderr:+ tee ceph_test_neorados_ec_io.log 2024-09-16T03:43:29.765 INFO:tasks.workunit.client.0.smithi154.stderr:+ sed 's/^/ ec_io: /' 2024-09-16T03:43:29.767 INFO:tasks.workunit.client.0.smithi154.stderr:++ printf %25s list 2024-09-16T03:43:29.767 INFO:tasks.workunit.client.0.smithi154.stderr:+ r=' list' 2024-09-16T03:43:29.769 INFO:tasks.workunit.client.0.smithi154.stderr:++ echo list 2024-09-16T03:43:29.769 INFO:tasks.workunit.client.0.smithi154.stderr:++ awk '{print $1}' 2024-09-16T03:43:29.770 INFO:tasks.workunit.client.0.smithi154.stderr:+ ff=list 2024-09-16T03:43:29.770 INFO:tasks.workunit.client.0.smithi154.stdout:test list on pid 28586 2024-09-16T03:43:29.771 INFO:tasks.workunit.client.0.smithi154.stderr:+ pid=28586 2024-09-16T03:43:29.771 INFO:tasks.workunit.client.0.smithi154.stderr:+ echo 'test list on pid 28586' 2024-09-16T03:43:29.771 INFO:tasks.workunit.client.0.smithi154.stderr:+ pids[$f]=28586 2024-09-16T03:43:29.771 INFO:tasks.workunit.client.0.smithi154.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-09-16T03:43:29.771 INFO:tasks.workunit.client.0.smithi154.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T03:43:29.771 INFO:tasks.workunit.client.0.smithi154.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_list 2>&1 | tee ceph_test_neorados_list.log | sed "s/^/ list: /"' 2024-09-16T03:43:29.771 INFO:tasks.workunit.client.0.smithi154.stderr:++ printf %25s ec_list 2024-09-16T03:43:29.771 INFO:tasks.workunit.client.0.smithi154.stderr:+ r=' ec_list' 2024-09-16T03:43:29.772 INFO:tasks.workunit.client.0.smithi154.stderr:++ echo ec_list 2024-09-16T03:43:29.772 INFO:tasks.workunit.client.0.smithi154.stderr:++ awk '{print $1}' 2024-09-16T03:43:29.772 INFO:tasks.workunit.client.0.smithi154.stderr:+ '[' -z '' ']' 2024-09-16T03:43:29.773 INFO:tasks.workunit.client.0.smithi154.stderr:+ return 2024-09-16T03:43:29.773 INFO:tasks.workunit.client.0.smithi154.stderr:+ case $- in 2024-09-16T03:43:29.773 INFO:tasks.workunit.client.0.smithi154.stderr:+ return 2024-09-16T03:43:29.773 INFO:tasks.workunit.client.0.smithi154.stderr:+ ceph_test_neorados_list 2024-09-16T03:43:29.773 INFO:tasks.workunit.client.0.smithi154.stderr:+ sed 's/^/ list: /' 2024-09-16T03:43:29.774 INFO:tasks.workunit.client.0.smithi154.stderr:+ ff=ec_list 2024-09-16T03:43:29.774 INFO:tasks.workunit.client.0.smithi154.stderr:+ pid=28597 2024-09-16T03:43:29.774 INFO:tasks.workunit.client.0.smithi154.stdout:test ec_list on pid 28597 2024-09-16T03:43:29.775 INFO:tasks.workunit.client.0.smithi154.stderr:+ echo 'test ec_list on pid 28597' 2024-09-16T03:43:29.775 INFO:tasks.workunit.client.0.smithi154.stderr:+ pids[$f]=28597 2024-09-16T03:43:29.775 INFO:tasks.workunit.client.0.smithi154.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-09-16T03:43:29.775 INFO:tasks.workunit.client.0.smithi154.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T03:43:29.776 INFO:tasks.workunit.client.0.smithi154.stderr:++ printf %25s misc 2024-09-16T03:43:29.776 INFO:tasks.workunit.client.0.smithi154.stderr:+ r=' misc' 2024-09-16T03:43:29.776 INFO:tasks.workunit.client.0.smithi154.stderr:++ echo misc 2024-09-16T03:43:29.776 INFO:tasks.workunit.client.0.smithi154.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_ec_list 2>&1 | tee ceph_test_neorados_ec_list.log | sed "s/^/ ec_list: /"' 2024-09-16T03:43:29.776 INFO:tasks.workunit.client.0.smithi154.stderr:++ awk '{print $1}' 2024-09-16T03:43:29.776 INFO:tasks.workunit.client.0.smithi154.stderr:+ tee ceph_test_neorados_list.log 2024-09-16T03:43:29.777 INFO:tasks.workunit.client.0.smithi154.stderr:+ '[' -z '' ']' 2024-09-16T03:43:29.777 INFO:tasks.workunit.client.0.smithi154.stderr:+ return 2024-09-16T03:43:29.778 INFO:tasks.workunit.client.0.smithi154.stderr:+ case $- in 2024-09-16T03:43:29.778 INFO:tasks.workunit.client.0.smithi154.stderr:+ return 2024-09-16T03:43:29.778 INFO:tasks.workunit.client.0.smithi154.stderr:+ tee ceph_test_neorados_ec_list.log 2024-09-16T03:43:29.778 INFO:tasks.workunit.client.0.smithi154.stderr:+ ceph_test_neorados_ec_list 2024-09-16T03:43:29.778 INFO:tasks.workunit.client.0.smithi154.stdout:test misc on pid 28615 2024-09-16T03:43:29.779 INFO:tasks.workunit.client.0.smithi154.stderr:+ ff=misc 2024-09-16T03:43:29.779 INFO:tasks.workunit.client.0.smithi154.stderr:+ pid=28615 2024-09-16T03:43:29.779 INFO:tasks.workunit.client.0.smithi154.stderr:+ echo 'test misc on pid 28615' 2024-09-16T03:43:29.779 INFO:tasks.workunit.client.0.smithi154.stderr:+ pids[$f]=28615 2024-09-16T03:43:29.779 INFO:tasks.workunit.client.0.smithi154.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-09-16T03:43:29.779 INFO:tasks.workunit.client.0.smithi154.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T03:43:29.779 INFO:tasks.workunit.client.0.smithi154.stderr:+ sed 's/^/ ec_list: /' 2024-09-16T03:43:29.779 INFO:tasks.workunit.client.0.smithi154.stderr:++ printf %25s pool 2024-09-16T03:43:29.779 INFO:tasks.workunit.client.0.smithi154.stderr:+ r=' pool' 2024-09-16T03:43:29.779 INFO:tasks.workunit.client.0.smithi154.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_misc 2>&1 | tee ceph_test_neorados_misc.log | sed "s/^/ misc: /"' 2024-09-16T03:43:29.780 INFO:tasks.workunit.client.0.smithi154.stderr:++ echo pool 2024-09-16T03:43:29.780 INFO:tasks.workunit.client.0.smithi154.stderr:++ awk '{print $1}' 2024-09-16T03:43:29.780 INFO:tasks.workunit.client.0.smithi154.stderr:+ '[' -z '' ']' 2024-09-16T03:43:29.781 INFO:tasks.workunit.client.0.smithi154.stderr:+ return 2024-09-16T03:43:29.781 INFO:tasks.workunit.client.0.smithi154.stderr:+ case $- in 2024-09-16T03:43:29.781 INFO:tasks.workunit.client.0.smithi154.stderr:+ return 2024-09-16T03:43:29.781 INFO:tasks.workunit.client.0.smithi154.stderr:+ ceph_test_neorados_misc 2024-09-16T03:43:29.781 INFO:tasks.workunit.client.0.smithi154.stderr:+ sed 's/^/ misc: /' 2024-09-16T03:43:29.782 INFO:tasks.workunit.client.0.smithi154.stdout:test pool on pid 28628 2024-09-16T03:43:29.782 INFO:tasks.workunit.client.0.smithi154.stderr:+ ff=pool 2024-09-16T03:43:29.782 INFO:tasks.workunit.client.0.smithi154.stderr:+ pid=28628 2024-09-16T03:43:29.782 INFO:tasks.workunit.client.0.smithi154.stderr:+ echo 'test pool on pid 28628' 2024-09-16T03:43:29.782 INFO:tasks.workunit.client.0.smithi154.stderr:+ pids[$f]=28628 2024-09-16T03:43:29.782 INFO:tasks.workunit.client.0.smithi154.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-09-16T03:43:29.782 INFO:tasks.workunit.client.0.smithi154.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T03:43:29.782 INFO:tasks.workunit.client.0.smithi154.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_pool 2>&1 | tee ceph_test_neorados_pool.log | sed "s/^/ pool: /"' 2024-09-16T03:43:29.783 INFO:tasks.workunit.client.0.smithi154.stderr:++ printf %25s read_operations 2024-09-16T03:43:29.783 INFO:tasks.workunit.client.0.smithi154.stderr:+ '[' -z '' ']' 2024-09-16T03:43:29.783 INFO:tasks.workunit.client.0.smithi154.stderr:+ return 2024-09-16T03:43:29.784 INFO:tasks.workunit.client.0.smithi154.stderr:+ r=' read_operations' 2024-09-16T03:43:29.784 INFO:tasks.workunit.client.0.smithi154.stderr:+ case $- in 2024-09-16T03:43:29.784 INFO:tasks.workunit.client.0.smithi154.stderr:+ return 2024-09-16T03:43:29.784 INFO:tasks.workunit.client.0.smithi154.stderr:+ tee ceph_test_neorados_pool.log 2024-09-16T03:43:29.784 INFO:tasks.workunit.client.0.smithi154.stderr:+ ceph_test_neorados_pool 2024-09-16T03:43:29.784 INFO:tasks.workunit.client.0.smithi154.stderr:++ echo read_operations 2024-09-16T03:43:29.785 INFO:tasks.workunit.client.0.smithi154.stderr:+ sed 's/^/ pool: /' 2024-09-16T03:43:29.785 INFO:tasks.workunit.client.0.smithi154.stderr:++ awk '{print $1}' 2024-09-16T03:43:29.787 INFO:tasks.workunit.client.0.smithi154.stdout:test read_operations on pid 28637 2024-09-16T03:43:29.787 INFO:tasks.workunit.client.0.smithi154.stderr:+ ff=read_operations 2024-09-16T03:43:29.787 INFO:tasks.workunit.client.0.smithi154.stderr:+ tee ceph_test_neorados_misc.log 2024-09-16T03:43:29.787 INFO:tasks.workunit.client.0.smithi154.stderr:+ pid=28637 2024-09-16T03:43:29.787 INFO:tasks.workunit.client.0.smithi154.stderr:+ echo 'test read_operations on pid 28637' 2024-09-16T03:43:29.787 INFO:tasks.workunit.client.0.smithi154.stderr:+ pids[$f]=28637 2024-09-16T03:43:29.787 INFO:tasks.workunit.client.0.smithi154.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-09-16T03:43:29.787 INFO:tasks.workunit.client.0.smithi154.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T03:43:29.788 INFO:tasks.workunit.client.0.smithi154.stderr:++ printf %25s snapshots 2024-09-16T03:43:29.788 INFO:tasks.workunit.client.0.smithi154.stderr:+ r=' snapshots' 2024-09-16T03:43:29.788 INFO:tasks.workunit.client.0.smithi154.stderr:++ echo snapshots 2024-09-16T03:43:29.788 INFO:tasks.workunit.client.0.smithi154.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_read_operations 2>&1 | tee ceph_test_neorados_read_operations.log | sed "s/^/ read_operations: /"' 2024-09-16T03:43:29.789 INFO:tasks.workunit.client.0.smithi154.stderr:++ awk '{print $1}' 2024-09-16T03:43:29.791 INFO:tasks.workunit.client.0.smithi154.stderr:+ '[' -z '' ']' 2024-09-16T03:43:29.791 INFO:tasks.workunit.client.0.smithi154.stderr:+ return 2024-09-16T03:43:29.791 INFO:tasks.workunit.client.0.smithi154.stderr:+ case $- in 2024-09-16T03:43:29.791 INFO:tasks.workunit.client.0.smithi154.stdout:test snapshots on pid 28648 2024-09-16T03:43:29.792 INFO:tasks.workunit.client.0.smithi154.stderr:+ ff=snapshots 2024-09-16T03:43:29.792 INFO:tasks.workunit.client.0.smithi154.stderr:+ return 2024-09-16T03:43:29.792 INFO:tasks.workunit.client.0.smithi154.stderr:+ pid=28648 2024-09-16T03:43:29.792 INFO:tasks.workunit.client.0.smithi154.stderr:+ echo 'test snapshots on pid 28648' 2024-09-16T03:43:29.792 INFO:tasks.workunit.client.0.smithi154.stderr:+ pids[$f]=28648 2024-09-16T03:43:29.793 INFO:tasks.workunit.client.0.smithi154.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-09-16T03:43:29.793 INFO:tasks.workunit.client.0.smithi154.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T03:43:29.793 INFO:tasks.workunit.client.0.smithi154.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_snapshots 2>&1 | tee ceph_test_neorados_snapshots.log | sed "s/^/ snapshots: /"' 2024-09-16T03:43:29.793 INFO:tasks.workunit.client.0.smithi154.stderr:+ ceph_test_neorados_read_operations 2024-09-16T03:43:29.793 INFO:tasks.workunit.client.0.smithi154.stderr:+ '[' -z '' ']' 2024-09-16T03:43:29.793 INFO:tasks.workunit.client.0.smithi154.stderr:+ return 2024-09-16T03:43:29.793 INFO:tasks.workunit.client.0.smithi154.stderr:+ case $- in 2024-09-16T03:43:29.793 INFO:tasks.workunit.client.0.smithi154.stderr:+ return 2024-09-16T03:43:29.793 INFO:tasks.workunit.client.0.smithi154.stderr:+ ceph_test_neorados_snapshots 2024-09-16T03:43:29.794 INFO:tasks.workunit.client.0.smithi154.stderr:+ tee ceph_test_neorados_read_operations.log 2024-09-16T03:43:29.794 INFO:tasks.workunit.client.0.smithi154.stderr:+ sed 's/^/ snapshots: /' 2024-09-16T03:43:29.795 INFO:tasks.workunit.client.0.smithi154.stderr:+ sed 's/^/ read_operations: /' 2024-09-16T03:43:29.795 INFO:tasks.workunit.client.0.smithi154.stderr:++ printf %25s watch_notify 2024-09-16T03:43:29.796 INFO:tasks.workunit.client.0.smithi154.stderr:+ r=' watch_notify' 2024-09-16T03:43:29.799 INFO:tasks.workunit.client.0.smithi154.stderr:++ echo watch_notify 2024-09-16T03:43:29.799 INFO:tasks.workunit.client.0.smithi154.stderr:++ awk '{print $1}' 2024-09-16T03:43:29.802 INFO:tasks.workunit.client.0.smithi154.stdout:test watch_notify on pid 28669 2024-09-16T03:43:29.803 INFO:tasks.workunit.client.0.smithi154.stderr:+ ff=watch_notify 2024-09-16T03:43:29.803 INFO:tasks.workunit.client.0.smithi154.stderr:+ tee ceph_test_neorados_snapshots.log 2024-09-16T03:43:29.803 INFO:tasks.workunit.client.0.smithi154.stderr:+ pid=28669 2024-09-16T03:43:29.803 INFO:tasks.workunit.client.0.smithi154.stderr:+ echo 'test watch_notify on pid 28669' 2024-09-16T03:43:29.803 INFO:tasks.workunit.client.0.smithi154.stderr:+ pids[$f]=28669 2024-09-16T03:43:29.803 INFO:tasks.workunit.client.0.smithi154.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-09-16T03:43:29.803 INFO:tasks.workunit.client.0.smithi154.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T03:43:29.803 INFO:tasks.workunit.client.0.smithi154.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_watch_notify 2>&1 | tee ceph_test_neorados_watch_notify.log | sed "s/^/ watch_notify: /"' 2024-09-16T03:43:29.804 INFO:tasks.workunit.client.0.smithi154.stderr:+ '[' -z '' ']' 2024-09-16T03:43:29.805 INFO:tasks.workunit.client.0.smithi154.stderr:+ return 2024-09-16T03:43:29.805 INFO:tasks.workunit.client.0.smithi154.stderr:+ case $- in 2024-09-16T03:43:29.805 INFO:tasks.workunit.client.0.smithi154.stderr:+ return 2024-09-16T03:43:29.805 INFO:tasks.workunit.client.0.smithi154.stderr:+ ceph_test_neorados_watch_notify 2024-09-16T03:43:29.805 INFO:tasks.workunit.client.0.smithi154.stderr:+ tee ceph_test_neorados_watch_notify.log 2024-09-16T03:43:29.805 INFO:tasks.workunit.client.0.smithi154.stderr:+ sed 's/^/ watch_notify: /' 2024-09-16T03:43:29.806 INFO:tasks.workunit.client.0.smithi154.stderr:++ printf %25s write_operations 2024-09-16T03:43:29.806 INFO:tasks.workunit.client.0.smithi154.stderr:+ r=' write_operations' 2024-09-16T03:43:29.807 INFO:tasks.workunit.client.0.smithi154.stderr:++ echo write_operations 2024-09-16T03:43:29.807 INFO:tasks.workunit.client.0.smithi154.stderr:++ awk '{print $1}' 2024-09-16T03:43:29.809 INFO:tasks.workunit.client.0.smithi154.stderr:+ ff=write_operations 2024-09-16T03:43:29.809 INFO:tasks.workunit.client.0.smithi154.stdout:test write_operations on pid 28691 2024-09-16T03:43:29.809 INFO:tasks.workunit.client.0.smithi154.stderr:+ pid=28691 2024-09-16T03:43:29.809 INFO:tasks.workunit.client.0.smithi154.stderr:+ echo 'test write_operations on pid 28691' 2024-09-16T03:43:29.809 INFO:tasks.workunit.client.0.smithi154.stderr:+ pids[$f]=28691 2024-09-16T03:43:29.810 INFO:tasks.workunit.client.0.smithi154.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_write_operations 2>&1 | tee ceph_test_neorados_write_operations.log | sed "s/^/ write_operations: /"' 2024-09-16T03:43:29.810 INFO:tasks.workunit.client.0.smithi154.stderr:+ ret=0 2024-09-16T03:43:29.810 INFO:tasks.workunit.client.0.smithi154.stderr:+ '[' 1 -eq 1 ']' 2024-09-16T03:43:29.810 INFO:tasks.workunit.client.0.smithi154.stderr:+ for t in "${!pids[@]}" 2024-09-16T03:43:29.810 INFO:tasks.workunit.client.0.smithi154.stderr:+ pid=28211 2024-09-16T03:43:29.810 INFO:tasks.workunit.client.0.smithi154.stderr:+ wait 28211 2024-09-16T03:43:29.811 INFO:tasks.workunit.client.0.smithi154.stderr:+ '[' -z '' ']' 2024-09-16T03:43:29.811 INFO:tasks.workunit.client.0.smithi154.stderr:+ return 2024-09-16T03:43:29.811 INFO:tasks.workunit.client.0.smithi154.stderr:+ case $- in 2024-09-16T03:43:29.811 INFO:tasks.workunit.client.0.smithi154.stderr:+ return 2024-09-16T03:43:29.811 INFO:tasks.workunit.client.0.smithi154.stderr:+ ceph_test_neorados_write_operations 2024-09-16T03:43:29.811 INFO:tasks.workunit.client.0.smithi154.stderr:+ tee ceph_test_neorados_write_operations.log 2024-09-16T03:43:29.811 INFO:tasks.workunit.client.0.smithi154.stderr:+ sed 's/^/ write_operations: /' 2024-09-16T03:43:31.109 INFO:tasks.mon_thrash:Sending CONT to mon a 2024-09-16T03:43:31.110 INFO:tasks.ceph.mon.a:Sent signal 18 2024-09-16T03:43:31.110 INFO:tasks.mon_thrash:Sending CONT to mon c 2024-09-16T03:43:31.110 INFO:tasks.ceph.mon.c:Sent signal 18 2024-09-16T03:43:31.110 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T03:43:31.111 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:43:31.173 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: [==========] Running 12 tests from 1 test suite. 2024-09-16T03:43:31.173 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: [----------] Global test environment set-up. 2024-09-16T03:43:31.173 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: [----------] 12 tests from AsioRados 2024-09-16T03:43:31.173 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.167+0000 7f9102191880 10 monclient: get_monmap_and_config 2024-09-16T03:43:31.173 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.167+0000 7f9102191880 10 monclient: build_initial_monmap 2024-09-16T03:43:31.173 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.167+0000 7f9102191880 10 monclient: monmap: 2024-09-16T03:43:31.173 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: epoch 0 2024-09-16T03:43:31.174 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:43:31.174 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: last_changed 0.000000 2024-09-16T03:43:31.174 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: created 0.000000 2024-09-16T03:43:31.174 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: min_mon_release 0 (unknown) 2024-09-16T03:43:31.174 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: election_strategy: 1 2024-09-16T03:43:31.174 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:43:31.174 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:43:31.174 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:43:31.174 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.174 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.171+0000 7f9102191880 1 Processor -- start 2024-09-16T03:43:31.174 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.171+0000 7f9102191880 1 -- start start 2024-09-16T03:43:31.175 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.171+0000 7f9102191880 10 monclient: init 2024-09-16T03:43:31.175 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.171+0000 7f9102191880 10 monclient: _reopen_session rank -1 2024-09-16T03:43:31.175 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.171+0000 7f9102191880 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:43:31.175 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.171+0000 7f9102191880 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5607e05bda50 0x5607e06551e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:43:31.175 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.171+0000 7f9102191880 10 monclient(hunting): picked mon.noname-b con 0x5607e05bda50 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:43:31.175 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.171+0000 7f9102191880 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5607e05bde30 0x5607e05bae70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:43:31.175 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.171+0000 7f9102191880 10 monclient(hunting): picked mon.noname-a con 0x5607e05bde30 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:43:31.175 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.171+0000 7f9102191880 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x5607e05be240 0x5607e05bb3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:43:31.175 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.171+0000 7f9102191880 10 monclient(hunting): picked mon.noname-c con 0x5607e05be240 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:43:31.175 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.171+0000 7f9102191880 10 monclient(hunting): start opening mon connection 2024-09-16T03:43:31.175 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.171+0000 7f9102191880 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x5607e05be650 con 0x5607e05bde30 2024-09-16T03:43:31.176 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.171+0000 7f9102191880 10 monclient(hunting): start opening mon connection 2024-09-16T03:43:31.176 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.171+0000 7f9102191880 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5607e04e91a0 con 0x5607e05bda50 2024-09-16T03:43:31.176 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.171+0000 7f9102191880 10 monclient(hunting): start opening mon connection 2024-09-16T03:43:31.176 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.171+0000 7f9102191880 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x5607e059de90 con 0x5607e05be240 2024-09-16T03:43:31.176 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.171+0000 7f9102191880 10 monclient(hunting): _renew_subs 2024-09-16T03:43:31.176 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.171+0000 7f9102191880 10 monclient(hunting): authenticate will time out at 1050.001482s 2024-09-16T03:43:31.176 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.171+0000 7f91004a6640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x5607e05be240 0x5607e05bb3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:43:31.177 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.171+0000 7f90ff4a4640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5607e05bde30 0x5607e05bae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:43:31.177 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.171+0000 7f91004a6640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x5607e05be240 0x5607e05bb3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.154:42590/0 (socket says 172.21.15.154:42590) 2024-09-16T03:43:31.177 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.171+0000 7f91004a6640 1 -- 172.21.15.154:0/3435713051 learned_addr learned my addr 172.21.15.154:0/3435713051 (peer_addr_for_me v2:172.21.15.154:0/0) 2024-09-16T03:43:31.177 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.171+0000 7f91004a6640 10 monclient(hunting): get_auth_request con 0x5607e05be240 auth_method 0 2024-09-16T03:43:31.177 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.171+0000 7f91004a6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:43:31.177 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.171+0000 7f91004a6640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:43:31.177 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.171+0000 7f91004a6640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:43:31.177 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.171+0000 7f91004a6640 1 -- 172.21.15.154:0/3435713051 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x5607e05be240 msgr2=0x5607e05bb3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 17 2024-09-16T03:43:31.177 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.171+0000 7f91004a6640 1 -- 172.21.15.154:0/3435713051 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x5607e05be240 msgr2=0x5607e05bb3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:43:31.178 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.171+0000 7f91004a6640 1 --2- 172.21.15.154:0/3435713051 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x5607e05be240 0x5607e05bb3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:43:31.178 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.171+0000 7f91004a6640 1 --2- 172.21.15.154:0/3435713051 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x5607e05be240 0x5607e05bb3b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:43:31.178 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.171+0000 7f90ff4a4640 10 monclient(hunting): get_auth_request con 0x5607e05bde30 auth_method 0 2024-09-16T03:43:31.178 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.171+0000 7f90ff4a4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:43:31.178 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.171+0000 7f90ff4a4640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:43:31.178 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.171+0000 7f90ff4a4640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:43:31.178 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.171+0000 7f90ff4a4640 1 -- 172.21.15.154:0/3435713051 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5607e05bde30 msgr2=0x5607e05bae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 16 2024-09-16T03:43:31.178 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.171+0000 7f90ff4a4640 1 -- 172.21.15.154:0/3435713051 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5607e05bde30 msgr2=0x5607e05bae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:43:31.261 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.253+0000 7f600d323640 10 monclient: get_monmap_and_config 2024-09-16T03:43:31.261 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.253+0000 7f600d323640 10 monclient: build_initial_monmap 2024-09-16T03:43:31.261 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.253+0000 7f600d323640 10 monclient: monmap: 2024-09-16T03:43:31.261 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:43:31.261 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:43:31.261 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:43:31.259765+0000 2024-09-16T03:43:31.262 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:43:31.259765+0000 2024-09-16T03:43:31.262 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:43:31.262 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:43:31.262 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:43:31.262 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:43:31.262 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:43:31.262 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:43:31.262 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:43:31.262 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:43:31.264 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.257+0000 7f600d323640 1 Processor -- start 2024-09-16T03:43:31.264 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.257+0000 7f600d323640 1 -- start start 2024-09-16T03:43:31.264 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.257+0000 7f600d323640 10 monclient: init 2024-09-16T03:43:31.264 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.257+0000 7f600d323640 10 monclient: _reopen_session rank -1 2024-09-16T03:43:31.264 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.257+0000 7f600d323640 10 monclient: _add_conns ranks=[4,3,1,0,2] 2024-09-16T03:43:31.264 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.257+0000 7f600d323640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f60080a39b0 0x7f6008133b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:43:31.264 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.257+0000 7f600d323640 10 monclient(hunting): picked mon.noname-c con 0x7f60080a39b0 addr v2:172.21.15.154:3300/0 2024-09-16T03:43:31.264 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.257+0000 7f600d323640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f60080a3d90 0x7f6008137000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:43:31.264 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.257+0000 7f600d323640 10 monclient(hunting): picked mon.noname-a con 0x7f60080a3d90 addr v2:172.21.15.42:3300/0 2024-09-16T03:43:31.264 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.257+0000 7f600d323640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6008137540 addr v1:172.21.15.154:6789/0 2024-09-16T03:43:31.265 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.257+0000 7f600d323640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f600809c140 con 0x7f6008137540 2024-09-16T03:43:31.265 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.257+0000 7f600d323640 10 monclient(hunting): start opening mon connection 2024-09-16T03:43:31.265 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.257+0000 7f600d323640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f600809c420 con 0x7f60080a3d90 2024-09-16T03:43:31.265 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.257+0000 7f600d323640 10 monclient(hunting): start opening mon connection 2024-09-16T03:43:31.265 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.257+0000 7f600d323640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f600806dd60 con 0x7f60080a39b0 2024-09-16T03:43:31.265 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.257+0000 7f600d323640 10 monclient(hunting): _renew_subs 2024-09-16T03:43:31.265 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.257+0000 7f600d323640 10 monclient(hunting): authenticate will time out at 1058.228993s 2024-09-16T03:43:31.265 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.257+0000 7f6006ffd640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f60080a39b0 0x7f6008133b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:43:31.265 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.257+0000 7f60077fe640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f6008137540 0x7f600813e390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.42:54002/0 (socket says 172.21.15.42:54002) 2024-09-16T03:43:31.266 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.257+0000 7f60077fe640 1 -- 172.21.15.42:0/3464863790 learned_addr learned my addr 172.21.15.42:0/3464863790 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:43:31.266 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.257+0000 7f6006ffd640 10 monclient(hunting): get_auth_request con 0x7f60080a39b0 auth_method 0 2024-09-16T03:43:31.266 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.257+0000 7f6006ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:43:31.266 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.257+0000 7f6006ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:43:31.266 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.257+0000 7f6006ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:43:31.266 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.257+0000 7f5ffe7fc640 1 --2- 172.21.15.42:0/3464863790 >> v2:172.21.15.42:3300/0 conn(0x7f60080a3d90 0x7f6008137000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:43:31.266 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.257+0000 7f6006ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:43:31.266 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.257+0000 7f6006ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:43:31.266 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.257+0000 7f6006ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:43:31.266 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.257+0000 7f5ffe7fc640 10 monclient(hunting): get_auth_request con 0x7f60080a3d90 auth_method 0 2024-09-16T03:43:31.266 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.257+0000 7f5ffe7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:43:31.266 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.257+0000 7f5ffe7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:43:31.267 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.257+0000 7f5ffe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:43:31.267 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.257+0000 7f6006ffd640 10 monclient(hunting): handle_auth_done global_id 14122 payload 274 2024-09-16T03:43:31.267 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.257+0000 7f6006ffd640 1 -- 172.21.15.42:0/3464863790 >> v2:172.21.15.42:3300/0 conn(0x7f60080a3d90 msgr2=0x7f6008137000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:43:31.267 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.257+0000 7f6006ffd640 1 --2- 172.21.15.42:0/3464863790 >> v2:172.21.15.42:3300/0 conn(0x7f60080a3d90 0x7f6008137000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:43:31.267 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.257+0000 7f6006ffd640 1 -- 172.21.15.42:0/3464863790 >> v1:172.21.15.154:6789/0 conn(0x7f6008137540 legacy=0x7f600813e390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:43:31.267 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.257+0000 7f6006ffd640 10 monclient: _finish_hunting 0 2024-09-16T03:43:31.267 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.257+0000 7f6006ffd640 1 monclient: found mon.noname-c 2024-09-16T03:43:31.267 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.257+0000 7f6006ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:43:31.268 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.257+0000 7f6006ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.154:3300/0 2024-09-16T03:43:31.268 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.257+0000 7f6006ffd640 1 -- 172.21.15.42:0/3464863790 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60080a5e20 con 0x7f60080a39b0 2024-09-16T03:43:31.275 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.257+0000 7f6006ffd640 1 --2- 172.21.15.42:0/3464863790 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f60080a39b0 0x7f6008133b80 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f60080a7750 tx=0x7f5ff801c340 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7c2deac16468a80e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:43:31.275 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.261+0000 7f60067fc640 1 -- 172.21.15.42:0/3464863790 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ff80b0070 con 0x7f60080a39b0 2024-09-16T03:43:31.275 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.261+0000 7f60067fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:43:31.276 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.261+0000 7f60067fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:43:31.276 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.261+0000 7f60067fc640 10 monclient: dump: 2024-09-16T03:43:31.276 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:43:31.276 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:43:31.276 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:43:31.276 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:43:31.276 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:43:31.276 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:43:31.276 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:43:31.276 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:43:31.277 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:43:31.277 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:43:31.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.261+0000 7f60067fc640 10 monclient: _finish_auth 0 2024-09-16T03:43:31.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.261+0000 7f60067fc640 10 monclient: _check_auth_tickets 2024-09-16T03:43:31.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.261+0000 7f60067fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:43:31.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.261+0000 7f600d323640 5 monclient: authenticate success, global_id 14122 2024-09-16T03:43:31.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.261+0000 7f600d323640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:43:31.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.261+0000 7f60067fc640 1 -- 172.21.15.42:0/3464863790 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ff80ab600 con 0x7f60080a39b0 2024-09-16T03:43:31.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.261+0000 7f60067fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:43:31.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.261+0000 7f600d323640 10 monclient: get_monmap_and_config success 2024-09-16T03:43:31.278 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.261+0000 7f600d323640 4 set_mon_vals no callback set 2024-09-16T03:43:31.278 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.261+0000 7f60067fc640 1 -- 172.21.15.42:0/3464863790 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ff80ab9d0 con 0x7f60080a39b0 2024-09-16T03:43:31.278 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.261+0000 7f60067fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:43:31.278 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.261+0000 7f60067fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:43:31.278 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.261+0000 7f60067fc640 10 monclient: dump: 2024-09-16T03:43:31.278 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:43:31.278 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:43:31.278 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:43:31.278 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:43:31.278 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:43:31.278 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:43:31.278 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:43:31.279 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:43:31.279 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:43:31.279 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:43:31.279 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.261+0000 7f600d323640 10 monclient: shutdown 2024-09-16T03:43:31.279 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.261+0000 7f600d323640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:43:31.279 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.261+0000 7f600d323640 1 -- 172.21.15.42:0/3464863790 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f60080a39b0 msgr2=0x7f6008133b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:43:31.279 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.261+0000 7f600d323640 1 --2- 172.21.15.42:0/3464863790 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f60080a39b0 0x7f6008133b80 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f60080a7750 tx=0x7f5ff801c340 comp rx=0 tx=0).stop 2024-09-16T03:43:31.279 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.261+0000 7f600d323640 1 -- 172.21.15.42:0/3464863790 shutdown_connections 2024-09-16T03:43:31.279 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.261+0000 7f600d323640 1 --2- 172.21.15.42:0/3464863790 >> v2:172.21.15.42:3300/0 conn(0x7f60080a3d90 0x7f6008137000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:43:31.279 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.261+0000 7f600d323640 1 --2- 172.21.15.42:0/3464863790 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f60080a39b0 0x7f6008133b80 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:43:31.279 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.261+0000 7f600d323640 1 -- 172.21.15.42:0/3464863790 >> 172.21.15.42:0/3464863790 conn(0x7f60080a9fb0 msgr2=0x7f60080aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:43:31.279 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.261+0000 7f600d323640 1 -- 172.21.15.42:0/3464863790 shutdown_connections 2024-09-16T03:43:31.279 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.261+0000 7f600d323640 1 -- 172.21.15.42:0/3464863790 wait complete. 2024-09-16T03:43:31.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.261+0000 7f600d323640 10 monclient: build_initial_monmap 2024-09-16T03:43:31.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.261+0000 7f600d323640 10 monclient: monmap: 2024-09-16T03:43:31.280 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:43:31.280 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:43:31.280 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:43:31.280 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:43:31.280 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:43:31.280 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:43:31.280 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:43:31.280 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:43:31.280 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:43:31.280 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:43:31.281 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.261+0000 7f600d323640 1 librados: starting msgr at 2024-09-16T03:43:31.281 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.261+0000 7f600d323640 1 librados: starting objecter 2024-09-16T03:43:31.281 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.261+0000 7f600d323640 1 Processor -- start 2024-09-16T03:43:31.281 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.261+0000 7f600d323640 1 -- start start 2024-09-16T03:43:31.281 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.261+0000 7f600d323640 1 librados: setting wanted keys 2024-09-16T03:43:31.281 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.261+0000 7f600d323640 1 librados: calling monclient init 2024-09-16T03:43:31.281 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.261+0000 7f600d323640 10 monclient: init 2024-09-16T03:43:31.281 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.261+0000 7f600d323640 10 monclient: _reopen_session rank -1 2024-09-16T03:43:31.281 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.261+0000 7f600d323640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:43:31.281 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.261+0000 7f600d323640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f60080a39b0 0x7f600813d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:43:31.281 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.261+0000 7f600d323640 10 monclient(hunting): picked mon.noname-c con 0x7f60080a39b0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:43:31.281 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.261+0000 7f600d323640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f60080a3d90 0x7f600813d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:43:31.281 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.261+0000 7f600d323640 10 monclient(hunting): picked mon.noname-a con 0x7f60080a3d90 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:43:31.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.261+0000 7f600d323640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6008137540 0x7f600813ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:43:31.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.261+0000 7f600d323640 10 monclient(hunting): picked mon.noname-b con 0x7f6008137540 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:43:31.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.261+0000 7f600d323640 10 monclient(hunting): start opening mon connection 2024-09-16T03:43:31.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.261+0000 7f600d323640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f60081424e0 con 0x7f60080a3d90 2024-09-16T03:43:31.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.261+0000 7f600d323640 10 monclient(hunting): start opening mon connection 2024-09-16T03:43:31.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.261+0000 7f600d323640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f6008142660 con 0x7f6008137540 2024-09-16T03:43:31.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.261+0000 7f600d323640 10 monclient(hunting): start opening mon connection 2024-09-16T03:43:31.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.261+0000 7f600d323640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f60081427e0 con 0x7f60080a39b0 2024-09-16T03:43:31.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.261+0000 7f600d323640 10 monclient(hunting): _renew_subs 2024-09-16T03:43:31.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.261+0000 7f600d323640 10 monclient(hunting): authenticate will time out at 1058.234231s 2024-09-16T03:43:31.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.261+0000 7f6006ffd640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f60080a39b0 0x7f600813d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:43:31.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.261+0000 7f5ffe7fc640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f60080a3d90 0x7f600813d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:43:31.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.261+0000 7f6006ffd640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f60080a39b0 0x7f600813d450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:37652/0 (socket says 172.21.15.42:37652) 2024-09-16T03:43:31.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.261+0000 7f6006ffd640 1 -- 172.21.15.42:0/296434318 learned_addr learned my addr 172.21.15.42:0/296434318 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:43:31.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.261+0000 7f60077fe640 1 --2- 172.21.15.42:0/296434318 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6008137540 0x7f600813ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:43:31.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.261+0000 7f6006ffd640 10 monclient(hunting): get_auth_request con 0x7f60080a39b0 auth_method 0 2024-09-16T03:43:31.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.261+0000 7f6006ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:43:31.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.261+0000 7f6006ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:43:31.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.261+0000 7f6006ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:43:31.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.261+0000 7f5ffe7fc640 10 monclient(hunting): get_auth_request con 0x7f60080a3d90 auth_method 0 2024-09-16T03:43:31.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.261+0000 7f5ffe7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:43:31.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.261+0000 7f5ffe7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:43:31.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.261+0000 7f5ffe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:43:31.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.261+0000 7f60077fe640 10 monclient(hunting): get_auth_request con 0x7f6008137540 auth_method 0 2024-09-16T03:43:31.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.261+0000 7f60077fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:43:31.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.261+0000 7f60077fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:43:31.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.261+0000 7f60077fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:43:31.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.261+0000 7f6006ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:43:31.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.261+0000 7f6006ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:43:31.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.261+0000 7f6006ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:43:31.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.261+0000 7f60077fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:43:31.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.261+0000 7f60077fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:43:31.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.261+0000 7f60077fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:43:31.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.261+0000 7f5ffe7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:43:31.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.261+0000 7f5ffe7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:43:31.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.261+0000 7f5ffe7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:43:31.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.261+0000 7f60077fe640 10 monclient(hunting): handle_auth_done global_id 14128 payload 824 2024-09-16T03:43:31.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.261+0000 7f60077fe640 1 -- 172.21.15.42:0/296434318 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f60080a39b0 msgr2=0x7f600813d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:43:31.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.261+0000 7f60077fe640 1 --2- 172.21.15.42:0/296434318 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f60080a39b0 0x7f600813d450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:43:31.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.261+0000 7f60077fe640 1 -- 172.21.15.42:0/296434318 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f60080a3d90 msgr2=0x7f600813d990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:43:31.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.261+0000 7f60077fe640 1 --2- 172.21.15.42:0/296434318 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f60080a3d90 0x7f600813d990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:43:31.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.261+0000 7f60077fe640 10 monclient: _finish_hunting 0 2024-09-16T03:43:31.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.261+0000 7f60077fe640 1 monclient: found mon.noname-b 2024-09-16T03:43:31.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.261+0000 7f60077fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:43:31.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.261+0000 7f60077fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.154:3300/0 2024-09-16T03:43:31.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.261+0000 7f60077fe640 1 -- 172.21.15.42:0/296434318 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f600809e530 con 0x7f6008137540 2024-09-16T03:43:31.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.261+0000 7f60077fe640 1 --2- 172.21.15.42:0/296434318 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6008137540 0x7f600813ded0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f600000caa0 tx=0x7f600000cf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:43:31.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.261+0000 7f5ffffff640 10 client.?.objecter ms_handle_connect 0x7f6008137540 2024-09-16T03:43:31.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.261+0000 7f5ffffff640 10 client.?.objecter resend_mon_ops 2024-09-16T03:43:31.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.261+0000 7f5ffffff640 1 -- 172.21.15.42:0/296434318 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6000017070 con 0x7f6008137540 2024-09-16T03:43:31.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.261+0000 7f5ffffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:43:31.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.261+0000 7f5ffffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:43:31.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.261+0000 7f5ffffff640 10 monclient: dump: 2024-09-16T03:43:31.287 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:43:31.287 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:43:31.288 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:43:31.288 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:43:31.288 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:43:31.288 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:43:31.288 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:43:31.288 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:43:31.288 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:43:31.288 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:43:31.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.261+0000 7f5ffffff640 10 monclient: _finish_auth 0 2024-09-16T03:43:31.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.261+0000 7f5ffffff640 10 monclient: _check_auth_tickets 2024-09-16T03:43:31.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.261+0000 7f5ffffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:43:31.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.261+0000 7f600d323640 5 monclient: authenticate success, global_id 14128 2024-09-16T03:43:31.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.261+0000 7f5ffffff640 1 -- 172.21.15.42:0/296434318 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f600000aba0 con 0x7f6008137540 2024-09-16T03:43:31.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.261+0000 7f600d323640 10 monclient: _renew_subs 2024-09-16T03:43:31.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.261+0000 7f600d323640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2024-09-16T03:43:31.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.261+0000 7f600d323640 1 -- 172.21.15.42:0/296434318 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f600809e810 con 0x7f6008137540 2024-09-16T03:43:31.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.261+0000 7f5ffffff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:43:31.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.261+0000 7f5ffffff640 1 -- 172.21.15.42:0/296434318 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6000012400 con 0x7f6008137540 2024-09-16T03:43:31.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.261+0000 7f5ffffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:43:31.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.261+0000 7f5ffffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:43:31.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.261+0000 7f5ffffff640 10 monclient: dump: 2024-09-16T03:43:31.290 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:43:31.290 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:43:31.290 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:43:31.290 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:43:31.290 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:43:31.290 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:43:31.290 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:43:31.290 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:43:31.290 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:43:31.290 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:43:31.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.261+0000 7f600d323640 10 client.14128.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:43:31.291 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.261+0000 7f600d323640 10 monclient: _renew_subs 2024-09-16T03:43:31.291 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.261+0000 7f600d323640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2024-09-16T03:43:31.291 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.261+0000 7f600d323640 1 -- 172.21.15.42:0/296434318 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f600809d0e0 con 0x7f6008137540 2024-09-16T03:43:31.291 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.261+0000 7f600d323640 1 librados: init done 2024-09-16T03:43:31.291 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.261+0000 7f6005ffb640 4 set_mon_vals no callback set 2024-09-16T03:43:31.291 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.265+0000 7f5ffffff640 1 -- 172.21.15.42:0/296434318 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f600001a070 con 0x7f6008137540 2024-09-16T03:43:31.291 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.265+0000 7f600d323640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:43:31.291 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.265+0000 7f5ffffff640 1 --2- 172.21.15.42:0/296434318 >> v2:172.21.15.154:6812/254845342 conn(0x7f5fe8037840 0x7f5fe8039d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:43:31.291 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.265+0000 7f5ffffff640 1 -- 172.21.15.42:0/296434318 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 3534+0+0 (secure 0 0 0) 0x7f6000048330 con 0x7f6008137540 2024-09-16T03:43:31.291 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.265+0000 7f5ffffff640 10 client.14128.objecter ms_dispatch 0x7f6008000ba0 osd_map(19..19 src has 1..19) 2024-09-16T03:43:31.291 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.265+0000 7f5ffffff640 3 client.14128.objecter handle_osd_map got epochs [19,19] > 0 2024-09-16T03:43:31.291 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.265+0000 7f5ffffff640 3 client.14128.objecter handle_osd_map decoding full epoch 19 2024-09-16T03:43:31.291 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.265+0000 7f5ffffff640 20 client.14128.objecter dump_active .. 0 homeless 2024-09-16T03:43:31.292 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.265+0000 7f6006ffd640 1 --2- 172.21.15.42:0/296434318 >> v2:172.21.15.154:6812/254845342 conn(0x7f5fe8037840 0x7f5fe8039d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:43:31.292 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.265+0000 7f600d323640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:43:31.292 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.265+0000 7f600d323640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2024-09-16T03:43:31.292 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.265+0000 7f600d323640 1 -- 172.21.15.42:0/296434318 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6008133b80 con 0x7f6008137540 2024-09-16T03:43:31.292 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.265+0000 7f6006ffd640 10 monclient: get_auth_request con 0x7f5fe8037840 auth_method 0 2024-09-16T03:43:31.292 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.265+0000 7f5ffffff640 10 client.14128.objecter ms_handle_connect 0x7f5fe8037840 2024-09-16T03:43:31.292 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.265+0000 7f6006ffd640 1 --2- 172.21.15.42:0/296434318 >> v2:172.21.15.154:6812/254845342 conn(0x7f5fe8037840 0x7f5fe8039d00 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f5ff801c380 tx=0x7f5ff8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:43:31.292 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.269+0000 7f5ffffff640 1 -- 172.21.15.42:0/296434318 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f6000011df0 con 0x7f6008137540 2024-09-16T03:43:31.292 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.269+0000 7f5ffffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:43:31.292 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.269+0000 7f5ffffff640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:43:31.378 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.171+0000 7f90ff4a4640 1 --2- 172.21.15.154:0/3435713051 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5607e05bde30 0x5607e05bae70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:43:31.378 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.171+0000 7f90ff4a4640 1 --2- 172.21.15.154:0/3435713051 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5607e05bde30 0x5607e05bae70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:43:31.379 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.171+0000 7f90ffca5640 1 --2- 172.21.15.154:0/3435713051 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5607e05bda50 0x5607e06551e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:43:31.379 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.171+0000 7f90ffca5640 10 monclient(hunting): get_auth_request con 0x5607e05bda50 auth_method 0 2024-09-16T03:43:31.379 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.171+0000 7f90ffca5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:43:31.379 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.171+0000 7f90ffca5640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:43:31.379 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.171+0000 7f90ffca5640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:43:31.379 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.175+0000 7f90ffca5640 1 -- 172.21.15.154:0/3435713051 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5607e05bda50 msgr2=0x5607e06551e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 18 2024-09-16T03:43:31.379 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.175+0000 7f90ffca5640 1 -- 172.21.15.154:0/3435713051 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5607e05bda50 msgr2=0x5607e06551e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:43:31.379 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.175+0000 7f90ffca5640 1 --2- 172.21.15.154:0/3435713051 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5607e05bda50 0x5607e06551e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:43:31.379 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.175+0000 7f90ffca5640 1 --2- 172.21.15.154:0/3435713051 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5607e05bda50 0x5607e06551e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:43:31.379 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.375+0000 7f90ff4a4640 1 --2- 172.21.15.154:0/3435713051 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5607e05bde30 0x5607e05bae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:43:31.379 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.375+0000 7f90ff4a4640 10 monclient(hunting): get_auth_request con 0x5607e05bde30 auth_method 0 2024-09-16T03:43:31.380 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.375+0000 7f90ff4a4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:43:31.380 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.375+0000 7f90ff4a4640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:43:31.380 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.375+0000 7f90ff4a4640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:43:31.380 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.375+0000 7f90ff4a4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:43:31.380 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.375+0000 7f90ff4a4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:43:31.380 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.375+0000 7f90ff4a4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:43:31.380 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.375+0000 7f91004a6640 1 --2- 172.21.15.154:0/3435713051 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x5607e05be240 0x5607e05bb3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:43:31.380 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.375+0000 7f91004a6640 10 monclient(hunting): get_auth_request con 0x5607e05be240 auth_method 0 2024-09-16T03:43:31.380 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.375+0000 7f91004a6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:43:31.381 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.375+0000 7f91004a6640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:43:31.381 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.375+0000 7f91004a6640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:43:31.381 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.375+0000 7f91004a6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:43:31.381 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.375+0000 7f91004a6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:43:31.381 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.375+0000 7f91004a6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:43:31.381 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.375+0000 7f91004a6640 10 monclient(hunting): handle_auth_done global_id 4514 payload 274 2024-09-16T03:43:31.381 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.375+0000 7f91004a6640 1 -- 172.21.15.154:0/3435713051 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5607e05bda50 msgr2=0x5607e06551e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:43:31.381 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.375+0000 7f91004a6640 1 --2- 172.21.15.154:0/3435713051 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5607e05bda50 0x5607e06551e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:43:31.381 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.375+0000 7f91004a6640 1 -- 172.21.15.154:0/3435713051 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5607e05bde30 msgr2=0x5607e05bae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:43:31.381 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.375+0000 7f91004a6640 1 --2- 172.21.15.154:0/3435713051 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5607e05bde30 0x5607e05bae70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:43:31.381 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.375+0000 7f91004a6640 10 monclient: _finish_hunting 0 2024-09-16T03:43:31.381 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.375+0000 7f91004a6640 1 monclient: found mon.noname-c 2024-09-16T03:43:31.381 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.375+0000 7f91004a6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:43:31.382 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.375+0000 7f91004a6640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:43:31.382 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.375+0000 7f91004a6640 1 -- 172.21.15.154:0/3435713051 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5607e059c7b0 con 0x5607e05be240 2024-09-16T03:43:31.382 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.375+0000 7f91004a6640 1 --2- 172.21.15.154:0/3435713051 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x5607e05be240 0x5607e05bb3b0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f90f8005c20 tx=0x7f90f800e980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b4e652eea706cb30 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:43:31.382 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.379+0000 7f90fe4a2640 1 -- 172.21.15.154:0/3435713051 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90f8017020 con 0x5607e05be240 2024-09-16T03:43:31.382 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.379+0000 7f90fe4a2640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:43:31.382 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.379+0000 7f90fe4a2640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:43:31.382 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.379+0000 7f90fe4a2640 10 monclient: dump: 2024-09-16T03:43:31.382 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: epoch 1 2024-09-16T03:43:31.382 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:43:31.382 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:43:31.383 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:43:31.383 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: min_mon_release 19 (squid) 2024-09-16T03:43:31.383 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: election_strategy: 1 2024-09-16T03:43:31.383 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:43:31.383 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:43:31.383 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:43:31.384 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.384 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.379+0000 7f90fe4a2640 10 monclient: _finish_auth 0 2024-09-16T03:43:31.384 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.379+0000 7f90fe4a2640 10 monclient: _check_auth_tickets 2024-09-16T03:43:31.384 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.379+0000 7f90fe4a2640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:43:31.385 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.379+0000 7f90fe4a2640 1 -- 172.21.15.154:0/3435713051 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f90f8013520 con 0x5607e05be240 2024-09-16T03:43:31.385 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.379+0000 7f90fe4a2640 10 monclient: handle_config config(0 keys) 2024-09-16T03:43:31.385 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.379+0000 7f90fe4a2640 1 -- 172.21.15.154:0/3435713051 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90f8013910 con 0x5607e05be240 2024-09-16T03:43:31.385 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.379+0000 7f90fe4a2640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:43:31.385 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.379+0000 7f90fe4a2640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:43:31.385 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.379+0000 7f90fe4a2640 10 monclient: dump: 2024-09-16T03:43:31.385 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: epoch 1 2024-09-16T03:43:31.385 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:43:31.385 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:43:31.385 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:43:31.385 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: min_mon_release 19 (squid) 2024-09-16T03:43:31.385 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: election_strategy: 1 2024-09-16T03:43:31.385 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:43:31.385 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:43:31.386 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:43:31.386 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.386 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.379+0000 7f9102191880 5 monclient: authenticate success, global_id 4514 2024-09-16T03:43:31.386 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.379+0000 7f9102191880 10 monclient: get_monmap_and_config success 2024-09-16T03:43:31.386 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.379+0000 7f9102191880 4 set_mon_vals no callback set 2024-09-16T03:43:31.386 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.379+0000 7f9102191880 10 monclient: shutdown 2024-09-16T03:43:31.386 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.379+0000 7f9102191880 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:43:31.386 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.379+0000 7f9102191880 1 -- 172.21.15.154:0/3435713051 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x5607e05be240 msgr2=0x5607e05bb3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:43:31.386 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.379+0000 7f9102191880 1 --2- 172.21.15.154:0/3435713051 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x5607e05be240 0x5607e05bb3b0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f90f8005c20 tx=0x7f90f800e980 comp rx=0 tx=0).stop 2024-09-16T03:43:31.386 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.379+0000 7f9102191880 1 -- 172.21.15.154:0/3435713051 shutdown_connections 2024-09-16T03:43:31.386 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.379+0000 7f9102191880 1 --2- 172.21.15.154:0/3435713051 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x5607e05be240 0x5607e05bb3b0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:43:31.386 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.379+0000 7f9102191880 1 --2- 172.21.15.154:0/3435713051 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5607e05bde30 0x5607e05bae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:43:31.387 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.379+0000 7f9102191880 1 --2- 172.21.15.154:0/3435713051 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5607e05bda50 0x5607e06551e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:43:31.387 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.379+0000 7f9102191880 1 -- 172.21.15.154:0/3435713051 >> 172.21.15.154:0/3435713051 conn(0x5607e05988f0 msgr2=0x5607e05ba3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:43:31.387 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.379+0000 7f9102191880 1 -- 172.21.15.154:0/3435713051 shutdown_connections 2024-09-16T03:43:31.387 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.379+0000 7f9102191880 1 -- 172.21.15.154:0/3435713051 wait complete. 2024-09-16T03:43:31.387 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.383+0000 7f9102191880 10 monclient: build_initial_monmap 2024-09-16T03:43:31.387 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.383+0000 7f9102191880 10 monclient: monmap: 2024-09-16T03:43:31.388 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: epoch 0 2024-09-16T03:43:31.388 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:43:31.388 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: last_changed 0.000000 2024-09-16T03:43:31.388 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: created 0.000000 2024-09-16T03:43:31.388 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: min_mon_release 0 (unknown) 2024-09-16T03:43:31.388 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: election_strategy: 1 2024-09-16T03:43:31.388 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:43:31.388 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:43:31.388 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:43:31.388 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.388 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.383+0000 7f9102191880 1 librados: starting msgr at 2024-09-16T03:43:31.388 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.383+0000 7f9102191880 1 librados: starting objecter 2024-09-16T03:43:31.388 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.383+0000 7f9102191880 1 Processor -- start 2024-09-16T03:43:31.389 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.383+0000 7f9102191880 1 -- start start 2024-09-16T03:43:31.389 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.383+0000 7f9102191880 1 librados: setting wanted keys 2024-09-16T03:43:31.389 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.383+0000 7f9102191880 1 librados: calling monclient init 2024-09-16T03:43:31.389 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.383+0000 7f9102191880 10 monclient: init 2024-09-16T03:43:31.389 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.383+0000 7f9102191880 10 monclient: _reopen_session rank -1 2024-09-16T03:43:31.389 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.383+0000 7f9102191880 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T03:43:31.389 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.383+0000 7f9102191880 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5607e05bda50 0x5607e0590bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:43:31.389 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.383+0000 7f9102191880 10 monclient(hunting): picked mon.noname-b con 0x5607e05bda50 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:43:31.389 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.383+0000 7f9102191880 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x5607e05bde30 0x5607e0591a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:43:31.389 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.383+0000 7f9102191880 10 monclient(hunting): picked mon.noname-c con 0x5607e05bde30 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:43:31.389 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.383+0000 7f9102191880 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5607e05be240 0x5607e06e4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:43:31.389 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.383+0000 7f9102191880 10 monclient(hunting): picked mon.noname-a con 0x5607e05be240 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:43:31.390 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.383+0000 7f9102191880 10 monclient(hunting): start opening mon connection 2024-09-16T03:43:31.390 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.383+0000 7f9102191880 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x5607e05bd450 con 0x5607e05be240 2024-09-16T03:43:31.390 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.383+0000 7f9102191880 10 monclient(hunting): start opening mon connection 2024-09-16T03:43:31.390 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.383+0000 7f9102191880 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x5607e059d590 con 0x5607e05bda50 2024-09-16T03:43:31.390 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.383+0000 7f9102191880 10 monclient(hunting): start opening mon connection 2024-09-16T03:43:31.390 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.383+0000 7f9102191880 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x5607e05bcc30 con 0x5607e05bde30 2024-09-16T03:43:31.390 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.383+0000 7f9102191880 10 monclient(hunting): _renew_subs 2024-09-16T03:43:31.390 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.383+0000 7f9102191880 10 monclient(hunting): authenticate will time out at 1050.213273s 2024-09-16T03:43:31.390 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.383+0000 7f90ffca5640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5607e05bda50 0x5607e0590bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:43:31.390 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.383+0000 7f90ffca5640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5607e05bda50 0x5607e0590bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.154:39744/0 (socket says 172.21.15.154:39744) 2024-09-16T03:43:31.390 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.383+0000 7f91004a6640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5607e05be240 0x5607e06e4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:43:31.391 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.383+0000 7f90ffca5640 1 -- 172.21.15.154:0/332177798 learned_addr learned my addr 172.21.15.154:0/332177798 (peer_addr_for_me v2:172.21.15.154:0/0) 2024-09-16T03:43:31.391 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.383+0000 7f90ffca5640 10 monclient(hunting): get_auth_request con 0x5607e05bda50 auth_method 0 2024-09-16T03:43:31.391 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.383+0000 7f90ffca5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:43:31.391 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.383+0000 7f90ffca5640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:43:31.391 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.383+0000 7f90ffca5640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:43:31.391 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.383+0000 7f91004a6640 10 monclient(hunting): get_auth_request con 0x5607e05be240 auth_method 0 2024-09-16T03:43:31.391 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.383+0000 7f91004a6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:43:31.391 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.383+0000 7f91004a6640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:43:31.391 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.383+0000 7f91004a6640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:43:31.391 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.383+0000 7f90ff4a4640 1 --2- 172.21.15.154:0/332177798 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x5607e05bde30 0x5607e0591a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:43:31.391 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.383+0000 7f90ff4a4640 10 monclient(hunting): get_auth_request con 0x5607e05bde30 auth_method 0 2024-09-16T03:43:31.391 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.383+0000 7f90ff4a4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:43:31.391 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.383+0000 7f90ff4a4640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:43:31.391 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.383+0000 7f90ff4a4640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:43:31.392 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.383+0000 7f90ffca5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:43:31.392 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.383+0000 7f90ffca5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:43:31.392 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.383+0000 7f90ffca5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:43:31.392 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.383+0000 7f91004a6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:43:31.392 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.383+0000 7f91004a6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:43:31.392 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.383+0000 7f91004a6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:43:31.392 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.383+0000 7f90ff4a4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:43:31.392 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.383+0000 7f90ff4a4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:43:31.392 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.383+0000 7f90ff4a4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:43:31.393 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.383+0000 7f90ffca5640 10 monclient(hunting): handle_auth_done global_id 14182 payload 824 2024-09-16T03:43:31.393 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.383+0000 7f90ffca5640 1 -- 172.21.15.154:0/332177798 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x5607e05bde30 msgr2=0x5607e0591a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:43:31.393 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.383+0000 7f90ffca5640 1 --2- 172.21.15.154:0/332177798 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x5607e05bde30 0x5607e0591a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:43:31.393 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.383+0000 7f90ffca5640 1 -- 172.21.15.154:0/332177798 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5607e05be240 msgr2=0x5607e06e4d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:43:31.393 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.383+0000 7f90ffca5640 1 --2- 172.21.15.154:0/332177798 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x5607e05be240 0x5607e06e4d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:43:31.393 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.383+0000 7f90ffca5640 10 monclient: _finish_hunting 0 2024-09-16T03:43:31.393 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.383+0000 7f90ffca5640 1 monclient: found mon.noname-b 2024-09-16T03:43:31.393 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.383+0000 7f90ffca5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:43:31.393 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.383+0000 7f90ffca5640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.154:3300/0 2024-09-16T03:43:31.393 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.383+0000 7f90ffca5640 1 -- 172.21.15.154:0/332177798 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5607e058ac30 con 0x5607e05bda50 2024-09-16T03:43:31.393 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.383+0000 7f90fd4a0640 10 client.?.objecter ms_handle_connect 0x5607e05bda50 2024-09-16T03:43:31.393 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.383+0000 7f90fd4a0640 10 client.?.objecter resend_mon_ops 2024-09-16T03:43:31.393 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.383+0000 7f90ffca5640 1 --2- 172.21.15.154:0/332177798 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5607e05bda50 0x5607e0590bf0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f90f000a500 tx=0x7f90f0018050 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:43:31.393 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.383+0000 7f90fd4a0640 1 -- 172.21.15.154:0/332177798 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90f00ba070 con 0x5607e05bda50 2024-09-16T03:43:31.394 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.383+0000 7f90fd4a0640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:43:31.394 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.383+0000 7f90fd4a0640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:43:31.394 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.383+0000 7f90fd4a0640 10 monclient: dump: 2024-09-16T03:43:31.395 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: epoch 1 2024-09-16T03:43:31.395 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:43:31.395 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:43:31.395 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:43:31.395 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: min_mon_release 19 (squid) 2024-09-16T03:43:31.395 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: election_strategy: 1 2024-09-16T03:43:31.395 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:43:31.395 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:43:31.395 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:43:31.395 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.395 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.383+0000 7f90fd4a0640 10 monclient: _finish_auth 0 2024-09-16T03:43:31.396 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.383+0000 7f90fd4a0640 10 monclient: _check_auth_tickets 2024-09-16T03:43:31.396 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.383+0000 7f90fd4a0640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:43:31.396 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.383+0000 7f90fd4a0640 1 -- 172.21.15.154:0/332177798 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f90f000a230 con 0x5607e05bda50 2024-09-16T03:43:31.396 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.383+0000 7f9102191880 5 monclient: authenticate success, global_id 14182 2024-09-16T03:43:31.396 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.383+0000 7f9102191880 10 monclient: _renew_subs 2024-09-16T03:43:31.396 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.383+0000 7f9102191880 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2024-09-16T03:43:31.396 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.383+0000 7f9102191880 1 -- 172.21.15.154:0/332177798 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5607e058aed0 con 0x5607e05bda50 2024-09-16T03:43:31.396 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.383+0000 7f90fd4a0640 10 monclient: handle_config config(0 keys) 2024-09-16T03:43:31.396 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.383+0000 7f90fe4a2640 4 set_mon_vals no callback set 2024-09-16T03:43:31.396 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.383+0000 7f9102191880 10 client.14182.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:43:31.396 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.383+0000 7f90fd4a0640 1 -- 172.21.15.154:0/332177798 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90f00af040 con 0x5607e05bda50 2024-09-16T03:43:31.396 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.383+0000 7f90fd4a0640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:43:31.396 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.383+0000 7f90fd4a0640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:43:31.397 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.383+0000 7f90fd4a0640 10 monclient: dump: 2024-09-16T03:43:31.397 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: epoch 1 2024-09-16T03:43:31.397 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:43:31.397 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:43:31.397 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:43:31.397 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: min_mon_release 19 (squid) 2024-09-16T03:43:31.397 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: election_strategy: 1 2024-09-16T03:43:31.397 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:43:31.397 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:43:31.397 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:43:31.397 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.397 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.383+0000 7f90fd4a0640 1 -- 172.21.15.154:0/332177798 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f90f00b5660 con 0x5607e05bda50 2024-09-16T03:43:31.397 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.383+0000 7f9102191880 10 monclient: _renew_subs 2024-09-16T03:43:31.397 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.383+0000 7f9102191880 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2024-09-16T03:43:31.397 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.383+0000 7f9102191880 1 -- 172.21.15.154:0/332177798 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5607e058a910 con 0x5607e05bda50 2024-09-16T03:43:31.398 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.383+0000 7f9102191880 1 librados: init done 2024-09-16T03:43:31.398 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.383+0000 7f9102191880 10 librados: wait_for_osdmap waiting 2024-09-16T03:43:31.398 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.383+0000 7f90fd4a0640 1 --2- 172.21.15.154:0/332177798 >> v2:172.21.15.154:6812/254845342 conn(0x7f90ec035a30 0x7f90ec037ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:43:31.516 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.509+0000 7f600d323640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:43:31.516 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.509+0000 7f600d323640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:43:31.516 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.509+0000 7f600d323640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2024-09-16T03:43:31.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.509+0000 7f600d323640 1 -- 172.21.15.42:0/296434318 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f60080a5750 con 0x7f6008137540 2024-09-16T03:43:31.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.509+0000 7f5ffffff640 1 -- 172.21.15.42:0/296434318 <== mon.1 v2:172.21.15.154:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7f6000007070 con 0x7f6008137540 2024-09-16T03:43:31.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.509+0000 7f5ffffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:43:31.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.509+0000 7f5ffffff640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:43:31.519 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":16,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:43:31.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.513+0000 7f600d323640 10 librados: watch_flush enter 2024-09-16T03:43:31.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.513+0000 7f600d323640 10 librados: watch_flush exit 2024-09-16T03:43:31.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.513+0000 7f600d323640 20 client.14128.objecter shutdown clearing up homeless session... 2024-09-16T03:43:31.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.513+0000 7f600d323640 10 client.14128.objecter successfully canceled tick 2024-09-16T03:43:31.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.513+0000 7f600d323640 1 -- 172.21.15.42:0/296434318 >> v2:172.21.15.154:6812/254845342 conn(0x7f5fe8037840 msgr2=0x7f5fe8039d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:43:31.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.513+0000 7f600d323640 1 --2- 172.21.15.42:0/296434318 >> v2:172.21.15.154:6812/254845342 conn(0x7f5fe8037840 0x7f5fe8039d00 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f5ff801c380 tx=0x7f5ff8008040 comp rx=0 tx=0).stop 2024-09-16T03:43:31.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.513+0000 7f600d323640 10 monclient: shutdown 2024-09-16T03:43:31.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.513+0000 7f600d323640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:43:31.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.513+0000 7f600d323640 1 -- 172.21.15.42:0/296434318 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6008137540 msgr2=0x7f600813ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:43:31.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.513+0000 7f600d323640 1 --2- 172.21.15.42:0/296434318 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6008137540 0x7f600813ded0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f600000caa0 tx=0x7f600000cf60 comp rx=0 tx=0).stop 2024-09-16T03:43:31.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.513+0000 7f600d323640 1 -- 172.21.15.42:0/296434318 shutdown_connections 2024-09-16T03:43:31.521 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.513+0000 7f600d323640 1 --2- 172.21.15.42:0/296434318 >> v2:172.21.15.154:6812/254845342 conn(0x7f5fe8037840 0x7f5fe8039d00 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:43:31.521 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.513+0000 7f600d323640 1 --2- 172.21.15.42:0/296434318 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6008137540 0x7f600813ded0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:43:31.521 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.513+0000 7f600d323640 1 --2- 172.21.15.42:0/296434318 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f60080a3d90 0x7f600813d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:43:31.521 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.513+0000 7f600d323640 1 --2- 172.21.15.42:0/296434318 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f60080a39b0 0x7f600813d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:43:31.521 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.513+0000 7f600d323640 1 -- 172.21.15.42:0/296434318 >> 172.21.15.42:0/296434318 conn(0x7f60080a9fb0 msgr2=0x7f6008131db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:43:31.521 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.513+0000 7f600d323640 1 -- 172.21.15.42:0/296434318 shutdown_connections 2024-09-16T03:43:31.521 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.513+0000 7f600d323640 1 -- 172.21.15.42:0/296434318 wait complete. 2024-09-16T03:43:31.521 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.513+0000 7f600d323640 1 librados: shutdown 2024-09-16T03:43:31.537 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T03:43:31.537 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T03:43:31.537 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T03:43:31.649 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.645+0000 7f7ba2b69640 10 monclient: get_monmap_and_config 2024-09-16T03:43:31.650 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.645+0000 7f7ba2b69640 10 monclient: build_initial_monmap 2024-09-16T03:43:31.650 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.645+0000 7f7ba2b69640 10 monclient: monmap: 2024-09-16T03:43:31.650 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:43:31.650 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:43:31.650 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:43:31.648338+0000 2024-09-16T03:43:31.650 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:43:31.648338+0000 2024-09-16T03:43:31.650 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:43:31.650 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:43:31.650 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:43:31.650 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:43:31.650 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:43:31.650 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:43:31.650 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:43:31.651 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:43:31.653 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.645+0000 7f7ba2b69640 1 Processor -- start 2024-09-16T03:43:31.653 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.645+0000 7f7ba2b69640 1 -- start start 2024-09-16T03:43:31.653 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.645+0000 7f7ba2b69640 10 monclient: init 2024-09-16T03:43:31.653 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.645+0000 7f7ba2b69640 10 monclient: _reopen_session rank -1 2024-09-16T03:43:31.653 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.645+0000 7f7ba2b69640 10 monclient: _add_conns ranks=[1,0,2,3,4] 2024-09-16T03:43:31.653 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.645+0000 7f7ba2b69640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7b9c0a2230 addr v1:172.21.15.154:6789/0 2024-09-16T03:43:31.653 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.645+0000 7f7ba2b69640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7b9c0a2610 addr v1:172.21.15.42:6789/0 2024-09-16T03:43:31.653 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.645+0000 7f7ba2b69640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7b9c13b450 0x7f7b9c13d840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:43:31.653 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.645+0000 7f7ba2b69640 10 monclient(hunting): picked mon.noname-b con 0x7f7b9c13b450 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:43:31.653 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.645+0000 7f7ba2b69640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7b9c06c570 con 0x7f7b9c0a2610 2024-09-16T03:43:31.653 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.645+0000 7f7ba2b69640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7b9c0715f0 con 0x7f7b9c0a2230 2024-09-16T03:43:31.653 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.645+0000 7f7ba2b69640 10 monclient(hunting): start opening mon connection 2024-09-16T03:43:31.654 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.645+0000 7f7ba2b69640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f7b9c0a0ef0 con 0x7f7b9c13b450 2024-09-16T03:43:31.654 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.645+0000 7f7ba2b69640 10 monclient(hunting): _renew_subs 2024-09-16T03:43:31.654 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.645+0000 7f7ba2b69640 10 monclient(hunting): authenticate will time out at 1058.617767s 2024-09-16T03:43:31.654 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.645+0000 7f7ba1106640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7b9c13b450 0x7f7b9c13d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:43:31.654 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.645+0000 7f7ba1106640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7b9c13b450 0x7f7b9c13d840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:37664/0 (socket says 172.21.15.42:37664) 2024-09-16T03:43:31.654 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.645+0000 7f7ba1106640 1 -- 172.21.15.42:0/4019736978 learned_addr learned my addr 172.21.15.42:0/4019736978 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:43:31.654 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.645+0000 7f7ba1106640 10 monclient(hunting): get_auth_request con 0x7f7b9c13b450 auth_method 0 2024-09-16T03:43:31.654 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.645+0000 7f7ba1106640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:43:31.654 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.645+0000 7f7ba1106640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:43:31.654 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.645+0000 7f7ba1106640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:43:31.654 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.645+0000 7f7ba1106640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:43:31.654 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.645+0000 7f7ba1106640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:43:31.654 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.645+0000 7f7ba1106640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:43:31.654 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.645+0000 7f7ba1106640 10 monclient(hunting): handle_auth_done global_id 4538 payload 274 2024-09-16T03:43:31.655 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.645+0000 7f7ba1106640 1 -- 172.21.15.42:0/4019736978 >> v1:172.21.15.154:6789/0 conn(0x7f7b9c0a2230 legacy=0x7f7b9c0722b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:43:31.655 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.645+0000 7f7ba1106640 1 -- 172.21.15.42:0/4019736978 >> v1:172.21.15.42:6789/0 conn(0x7f7b9c0a2610 legacy=0x7f7b9c0a2a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:43:31.655 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.645+0000 7f7ba1106640 10 monclient: _finish_hunting 0 2024-09-16T03:43:31.655 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.645+0000 7f7ba1106640 1 monclient: found mon.noname-b 2024-09-16T03:43:31.655 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.645+0000 7f7ba1106640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:43:31.655 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.645+0000 7f7ba1106640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T03:43:31.655 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.645+0000 7f7ba1106640 1 -- 172.21.15.42:0/4019736978 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b9c0a3c90 con 0x7f7b9c13b450 2024-09-16T03:43:31.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.649+0000 7f7ba1106640 1 --2- 172.21.15.42:0/4019736978 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7b9c13b450 0x7f7b9c13d840 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f7b94009eb0 tx=0x7f7b9401c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aeacc3ebe6352695 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:43:31.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.649+0000 7f7b9b7fe640 1 -- 172.21.15.42:0/4019736978 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b940a64b0 con 0x7f7b9c13b450 2024-09-16T03:43:31.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.649+0000 7f7b9b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:43:31.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.649+0000 7f7b9b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:43:31.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.649+0000 7f7b9b7fe640 10 monclient: dump: 2024-09-16T03:43:31.661 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:43:31.662 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:43:31.662 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:43:31.662 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:43:31.662 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:43:31.662 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:43:31.662 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:43:31.662 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:43:31.662 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:43:31.662 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:43:31.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.649+0000 7f7b9b7fe640 10 monclient: _finish_auth 0 2024-09-16T03:43:31.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.649+0000 7f7b9b7fe640 10 monclient: _check_auth_tickets 2024-09-16T03:43:31.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.649+0000 7f7b9b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:43:31.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.649+0000 7f7b9b7fe640 1 -- 172.21.15.42:0/4019736978 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7b940a6650 con 0x7f7b9c13b450 2024-09-16T03:43:31.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.649+0000 7f7b9b7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:43:31.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.649+0000 7f7b9b7fe640 1 -- 172.21.15.42:0/4019736978 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b940a6a20 con 0x7f7b9c13b450 2024-09-16T03:43:31.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.649+0000 7f7ba2b69640 5 monclient: authenticate success, global_id 4538 2024-09-16T03:43:31.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.649+0000 7f7ba2b69640 10 monclient: get_monmap_and_config success 2024-09-16T03:43:31.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.649+0000 7f7ba2b69640 4 set_mon_vals no callback set 2024-09-16T03:43:31.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.649+0000 7f7b9b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:43:31.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.649+0000 7f7b9b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:43:31.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.649+0000 7f7b9b7fe640 10 monclient: dump: 2024-09-16T03:43:31.663 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:43:31.663 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:43:31.663 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:43:31.663 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:43:31.663 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:43:31.663 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:43:31.663 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:43:31.664 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:43:31.664 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:43:31.664 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:43:31.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.649+0000 7f7ba2b69640 10 monclient: shutdown 2024-09-16T03:43:31.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.649+0000 7f7ba2b69640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:43:31.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.649+0000 7f7ba2b69640 1 -- 172.21.15.42:0/4019736978 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7b9c13b450 msgr2=0x7f7b9c13d840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:43:31.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.649+0000 7f7ba2b69640 1 --2- 172.21.15.42:0/4019736978 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7b9c13b450 0x7f7b9c13d840 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f7b94009eb0 tx=0x7f7b9401c8f0 comp rx=0 tx=0).stop 2024-09-16T03:43:31.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.649+0000 7f7ba2b69640 1 -- 172.21.15.42:0/4019736978 shutdown_connections 2024-09-16T03:43:31.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.649+0000 7f7ba2b69640 1 --2- 172.21.15.42:0/4019736978 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7b9c13b450 0x7f7b9c13d840 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:43:31.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.649+0000 7f7ba2b69640 1 -- 172.21.15.42:0/4019736978 >> 172.21.15.42:0/4019736978 conn(0x7f7b9c0a6e80 msgr2=0x7f7b9c0a3320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:43:31.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.649+0000 7f7ba2b69640 1 -- 172.21.15.42:0/4019736978 shutdown_connections 2024-09-16T03:43:31.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.649+0000 7f7ba2b69640 1 -- 172.21.15.42:0/4019736978 wait complete. 2024-09-16T03:43:31.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.649+0000 7f7ba2b69640 10 monclient: build_initial_monmap 2024-09-16T03:43:31.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.649+0000 7f7ba2b69640 10 monclient: monmap: 2024-09-16T03:43:31.665 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:43:31.665 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:43:31.665 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:43:31.665 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:43:31.665 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:43:31.665 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:43:31.665 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:43:31.665 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:43:31.665 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:43:31.665 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:43:31.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.649+0000 7f7ba2b69640 1 librados: starting msgr at 2024-09-16T03:43:31.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.649+0000 7f7ba2b69640 1 librados: starting objecter 2024-09-16T03:43:31.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.649+0000 7f7ba2b69640 1 Processor -- start 2024-09-16T03:43:31.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.649+0000 7f7ba2b69640 1 -- start start 2024-09-16T03:43:31.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.649+0000 7f7ba2b69640 1 librados: setting wanted keys 2024-09-16T03:43:31.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.649+0000 7f7ba2b69640 1 librados: calling monclient init 2024-09-16T03:43:31.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.649+0000 7f7ba2b69640 10 monclient: init 2024-09-16T03:43:31.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.649+0000 7f7ba2b69640 10 monclient: _reopen_session rank -1 2024-09-16T03:43:31.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.649+0000 7f7ba2b69640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:43:31.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.649+0000 7f7ba2b69640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7b9c0a2230 0x7f7b9c1cd820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:43:31.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.649+0000 7f7ba2b69640 10 monclient(hunting): picked mon.noname-c con 0x7f7b9c0a2230 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:43:31.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.649+0000 7f7ba2b69640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7b9c0a2610 0x7f7b9c09ea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:43:31.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.649+0000 7f7ba2b69640 10 monclient(hunting): picked mon.noname-a con 0x7f7b9c0a2610 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:43:31.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.649+0000 7f7ba2b69640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7b9c13b450 0x7f7b9c09efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:43:31.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.649+0000 7f7ba2b69640 10 monclient(hunting): picked mon.noname-b con 0x7f7b9c13b450 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:43:31.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.649+0000 7f7ba2b69640 10 monclient(hunting): start opening mon connection 2024-09-16T03:43:31.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.649+0000 7f7ba2b69640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f7b9c140dd0 con 0x7f7b9c0a2610 2024-09-16T03:43:31.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.649+0000 7f7ba2b69640 10 monclient(hunting): start opening mon connection 2024-09-16T03:43:31.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.649+0000 7f7ba2b69640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f7b9c072a00 con 0x7f7b9c13b450 2024-09-16T03:43:31.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.649+0000 7f7ba2b69640 10 monclient(hunting): start opening mon connection 2024-09-16T03:43:31.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.649+0000 7f7ba2b69640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f7b9c140f50 con 0x7f7b9c0a2230 2024-09-16T03:43:31.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.649+0000 7f7ba2b69640 10 monclient(hunting): _renew_subs 2024-09-16T03:43:31.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.649+0000 7f7ba2b69640 10 monclient(hunting): authenticate will time out at 1058.622929s 2024-09-16T03:43:31.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.649+0000 7f7ba0905640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7b9c0a2230 0x7f7b9c1cd820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:43:31.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.649+0000 7f7b9bfff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7b9c0a2610 0x7f7b9c09ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:43:31.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.649+0000 7f7ba0905640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7b9c0a2230 0x7f7b9c1cd820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:37672/0 (socket says 172.21.15.42:37672) 2024-09-16T03:43:31.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.649+0000 7f7ba0905640 1 -- 172.21.15.42:0/2912281203 learned_addr learned my addr 172.21.15.42:0/2912281203 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:43:31.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.649+0000 7f7ba0905640 10 monclient(hunting): get_auth_request con 0x7f7b9c0a2230 auth_method 0 2024-09-16T03:43:31.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.649+0000 7f7ba0905640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:43:31.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.649+0000 7f7ba0905640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:43:31.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.649+0000 7f7ba0905640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:43:31.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.649+0000 7f7b9bfff640 10 monclient(hunting): get_auth_request con 0x7f7b9c0a2610 auth_method 0 2024-09-16T03:43:31.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.649+0000 7f7b9bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:43:31.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.649+0000 7f7b9bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:43:31.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.649+0000 7f7b9bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:43:31.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.649+0000 7f7ba1106640 1 --2- 172.21.15.42:0/2912281203 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7b9c13b450 0x7f7b9c09efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:43:31.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.649+0000 7f7b9bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:43:31.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.649+0000 7f7b9bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:43:31.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.649+0000 7f7b9bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:43:31.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.649+0000 7f7ba0905640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:43:31.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.649+0000 7f7ba0905640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:43:31.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.649+0000 7f7ba0905640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:43:31.670 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.649+0000 7f7ba1106640 10 monclient(hunting): get_auth_request con 0x7f7b9c13b450 auth_method 0 2024-09-16T03:43:31.670 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.649+0000 7f7ba1106640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:43:31.670 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.649+0000 7f7ba1106640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:43:31.670 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.649+0000 7f7ba1106640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:43:31.670 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.649+0000 7f7ba1106640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:43:31.670 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.649+0000 7f7ba1106640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:43:31.670 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.649+0000 7f7ba1106640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:43:31.670 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.649+0000 7f7ba0905640 10 monclient(hunting): handle_auth_done global_id 4541 payload 824 2024-09-16T03:43:31.670 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.649+0000 7f7ba0905640 1 -- 172.21.15.42:0/2912281203 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7b9c13b450 msgr2=0x7f7b9c09efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:43:31.670 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.649+0000 7f7ba0905640 1 --2- 172.21.15.42:0/2912281203 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7b9c13b450 0x7f7b9c09efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:43:31.670 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.649+0000 7f7ba0905640 1 -- 172.21.15.42:0/2912281203 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7b9c0a2610 msgr2=0x7f7b9c09ea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:43:31.670 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.649+0000 7f7ba0905640 1 --2- 172.21.15.42:0/2912281203 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7b9c0a2610 0x7f7b9c09ea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:43:31.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.649+0000 7f7ba0905640 10 monclient: _finish_hunting 0 2024-09-16T03:43:31.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.649+0000 7f7ba0905640 1 monclient: found mon.noname-c 2024-09-16T03:43:31.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.649+0000 7f7ba0905640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:43:31.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.649+0000 7f7ba0905640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:43:31.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.649+0000 7f7ba0905640 1 -- 172.21.15.42:0/2912281203 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b9c11e8c0 con 0x7f7b9c0a2230 2024-09-16T03:43:31.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.649+0000 7f7ba0905640 1 --2- 172.21.15.42:0/2912281203 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7b9c0a2230 0x7f7b9c1cd820 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f7b8c00c500 tx=0x7f7b8c00c9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:43:31.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.649+0000 7f7b997fa640 10 client.?.objecter ms_handle_connect 0x7f7b9c0a2230 2024-09-16T03:43:31.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.649+0000 7f7b997fa640 10 client.?.objecter resend_mon_ops 2024-09-16T03:43:31.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.649+0000 7f7b997fa640 1 -- 172.21.15.42:0/2912281203 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b8c005220 con 0x7f7b9c0a2230 2024-09-16T03:43:31.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.649+0000 7f7b997fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:43:31.672 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.649+0000 7f7b997fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:43:31.672 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.649+0000 7f7b997fa640 10 monclient: dump: 2024-09-16T03:43:31.672 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:43:31.672 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:43:31.672 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:43:31.672 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:43:31.672 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:43:31.672 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:43:31.672 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:43:31.672 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:43:31.672 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:43:31.672 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:43:31.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.649+0000 7f7b997fa640 10 monclient: _finish_auth 0 2024-09-16T03:43:31.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.649+0000 7f7b997fa640 10 monclient: _check_auth_tickets 2024-09-16T03:43:31.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.649+0000 7f7b997fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:43:31.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.653+0000 7f7ba2b69640 5 monclient: authenticate success, global_id 4541 2024-09-16T03:43:31.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.653+0000 7f7b997fa640 1 -- 172.21.15.42:0/2912281203 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7b8c0053c0 con 0x7f7b9c0a2230 2024-09-16T03:43:31.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.653+0000 7f7b997fa640 10 monclient: handle_config config(0 keys) 2024-09-16T03:43:31.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.653+0000 7f7b997fa640 1 -- 172.21.15.42:0/2912281203 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b8c007270 con 0x7f7b9c0a2230 2024-09-16T03:43:31.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.653+0000 7f7b997fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:43:31.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.653+0000 7f7b997fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:43:31.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.653+0000 7f7b997fa640 10 monclient: dump: 2024-09-16T03:43:31.674 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:43:31.674 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:43:31.674 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:43:31.675 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:43:31.675 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:43:31.675 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:43:31.675 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:43:31.675 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:43:31.675 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:43:31.675 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:43:31.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.653+0000 7f7b9affd640 4 set_mon_vals no callback set 2024-09-16T03:43:31.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.653+0000 7f7ba2b69640 10 monclient: _renew_subs 2024-09-16T03:43:31.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.653+0000 7f7ba2b69640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:43:31.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.653+0000 7f7ba2b69640 1 -- 172.21.15.42:0/2912281203 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b9c13d580 con 0x7f7b9c0a2230 2024-09-16T03:43:31.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.653+0000 7f7ba2b69640 10 client.4541.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:43:31.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.653+0000 7f7ba2b69640 10 monclient: _renew_subs 2024-09-16T03:43:31.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.653+0000 7f7ba2b69640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:43:31.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.653+0000 7f7ba2b69640 1 -- 172.21.15.42:0/2912281203 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b9c135840 con 0x7f7b9c0a2230 2024-09-16T03:43:31.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.653+0000 7f7ba2b69640 1 librados: init done 2024-09-16T03:43:31.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.653+0000 7f7ba2b69640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:43:31.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.653+0000 7f7ba2b69640 1 --2- 172.21.15.42:0/2912281203 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7b9c1cdd60 0x7f7b9c09f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:43:31.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.653+0000 7f7ba2b69640 10 monclient(hunting): start opening mon connection 2024-09-16T03:43:31.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.653+0000 7f7ba2b69640 1 -- 172.21.15.42:0/2912281203 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f7b9c0722b0 con 0x7f7b9c1cdd60 2024-09-16T03:43:31.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.653+0000 7f7b997fa640 1 -- 172.21.15.42:0/2912281203 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f7b8c019020 con 0x7f7b9c0a2230 2024-09-16T03:43:31.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.653+0000 7f7b9bfff640 1 --2- 172.21.15.42:0/2912281203 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7b9c1cdd60 0x7f7b9c09f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:43:31.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.653+0000 7f7b9bfff640 10 monclient: get_auth_request con 0x7f7b9c1cdd60 auth_method 0 2024-09-16T03:43:31.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.653+0000 7f7b9bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:43:31.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.653+0000 7f7b9bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:43:31.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.653+0000 7f7b9bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:43:31.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.653+0000 7f7b997fa640 1 --2- 172.21.15.42:0/2912281203 >> v2:172.21.15.154:6812/254845342 conn(0x7f7b88037810 0x7f7b88039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:43:31.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.653+0000 7f7b9bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:43:31.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.653+0000 7f7b9bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:43:31.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.653+0000 7f7b9bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:43:31.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.653+0000 7f7b997fa640 1 -- 172.21.15.42:0/2912281203 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(19..19 src has 1..19) ==== 3534+0+0 (secure 0 0 0) 0x7f7b8c047a10 con 0x7f7b9c0a2230 2024-09-16T03:43:31.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.653+0000 7f7b997fa640 10 client.4541.objecter ms_dispatch 0x7f7b9c000ba0 osd_map(19..19 src has 1..19) 2024-09-16T03:43:31.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.653+0000 7f7b997fa640 3 client.4541.objecter handle_osd_map got epochs [19,19] > 0 2024-09-16T03:43:31.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.653+0000 7f7b997fa640 3 client.4541.objecter handle_osd_map decoding full epoch 19 2024-09-16T03:43:31.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.653+0000 7f7b997fa640 20 client.4541.objecter dump_active .. 0 homeless 2024-09-16T03:43:31.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.653+0000 7f7b9bfff640 10 monclient(hunting): handle_auth_done global_id 4692 payload 824 2024-09-16T03:43:31.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.653+0000 7f7b9bfff640 1 -- 172.21.15.42:0/2912281203 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7b9c13ba30 con 0x7f7b9c1cdd60 2024-09-16T03:43:31.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.653+0000 7f7ba1106640 1 --2- 172.21.15.42:0/2912281203 >> v2:172.21.15.154:6812/254845342 conn(0x7f7b88037810 0x7f7b88039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:43:31.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.653+0000 7f7ba1106640 10 monclient: get_auth_request con 0x7f7b88037810 auth_method 0 2024-09-16T03:43:31.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.653+0000 7f7b9bfff640 1 --2- 172.21.15.42:0/2912281203 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7b9c1cdd60 0x7f7b9c09f4f0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f7b9000f540 tx=0x7f7b9000fa00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:43:31.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.653+0000 7f7b997fa640 10 client.4541.objecter ms_handle_connect 0x7f7b9c1cdd60 2024-09-16T03:43:31.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.653+0000 7f7b997fa640 10 client.4541.objecter resend_mon_ops 2024-09-16T03:43:31.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.653+0000 7f7b997fa640 10 client.4541.objecter ms_handle_connect 0x7f7b88037810 2024-09-16T03:43:31.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.653+0000 7f7ba1106640 1 --2- 172.21.15.42:0/2912281203 >> v2:172.21.15.154:6812/254845342 conn(0x7f7b88037810 0x7f7b88039cd0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f7b940ae000 tx=0x7f7b940b1040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:43:31.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.665+0000 7f7b997fa640 1 -- 172.21.15.42:0/2912281203 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b90019070 con 0x7f7b9c1cdd60 2024-09-16T03:43:31.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.665+0000 7f7b997fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:43:31.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.665+0000 7f7b997fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:43:31.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.665+0000 7f7b997fa640 10 monclient: dump: 2024-09-16T03:43:31.678 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:43:31.679 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:43:31.679 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:43:31.679 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:43:31.679 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:43:31.679 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:43:31.679 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:43:31.679 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:43:31.679 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:43:31.679 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:43:31.892 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.885+0000 7f7b997fa640 1 -- 172.21.15.42:0/2912281203 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f7b90005320 con 0x7f7b9c1cdd60 2024-09-16T03:43:31.892 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.885+0000 7f7b997fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:43:31.892 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.885+0000 7f7b997fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:43:31.892 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.885+0000 7f7b997fa640 1 -- 172.21.15.42:0/2912281203 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7b9c1cdd60 msgr2=0x7f7b9c09f4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:43:31.892 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.885+0000 7f7b997fa640 1 --2- 172.21.15.42:0/2912281203 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7b9c1cdd60 0x7f7b9c09f4f0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f7b9000f540 tx=0x7f7b9000fa00 comp rx=0 tx=0).stop 2024-09-16T03:43:31.892 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.885+0000 7f7b997fa640 1 -- 172.21.15.42:0/2912281203 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7b9c1cdd60 msgr2=0x7f7b9c09f4f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:43:31.892 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.885+0000 7f7b997fa640 1 --2- 172.21.15.42:0/2912281203 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7b9c1cdd60 0x7f7b9c09f4f0 secure :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f7b9000f540 tx=0x7f7b9000fa00 comp rx=0 tx=0).stop 2024-09-16T03:43:31.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.893+0000 7f7ba2b69640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:43:31.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.893+0000 7f7ba2b69640 1 --2- 172.21.15.42:0/2912281203 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7b9c1c60c0 0x7f7b9c09fa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:43:31.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.893+0000 7f7ba2b69640 10 monclient(hunting): start opening mon connection 2024-09-16T03:43:31.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.893+0000 7f7ba2b69640 1 -- 172.21.15.42:0/2912281203 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f7b9c06b2e0 con 0x7f7b9c1c60c0 2024-09-16T03:43:31.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.893+0000 7f7b9bfff640 1 --2- 172.21.15.42:0/2912281203 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7b9c1c60c0 0x7f7b9c09fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:43:31.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.893+0000 7f7b9bfff640 10 monclient: get_auth_request con 0x7f7b9c1c60c0 auth_method 0 2024-09-16T03:43:31.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.893+0000 7f7b9bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:43:31.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.893+0000 7f7b9bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:43:31.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.893+0000 7f7b9bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:43:31.900 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.893+0000 7f7b9bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:43:31.900 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.893+0000 7f7b9bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:43:31.900 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.893+0000 7f7b9bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:43:31.900 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.893+0000 7f7b9bfff640 10 monclient(hunting): handle_auth_done global_id 4695 payload 824 2024-09-16T03:43:31.900 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.893+0000 7f7b9bfff640 1 -- 172.21.15.42:0/2912281203 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f7b9c1d8480 con 0x7f7b9c1c60c0 2024-09-16T03:43:31.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.893+0000 7f7b9bfff640 1 --2- 172.21.15.42:0/2912281203 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7b9c1c60c0 0x7f7b9c09fa30 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f7b9000f540 tx=0x7f7b90007f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:43:31.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.893+0000 7f7b997fa640 10 client.4541.objecter ms_handle_connect 0x7f7b9c1c60c0 2024-09-16T03:43:31.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.893+0000 7f7b997fa640 10 client.4541.objecter resend_mon_ops 2024-09-16T03:43:31.964 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.957+0000 7f7b997fa640 1 -- 172.21.15.42:0/2912281203 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b90019070 con 0x7f7b9c1c60c0 2024-09-16T03:43:31.964 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.957+0000 7f7b997fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:43:31.964 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.957+0000 7f7b997fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:43:31.964 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:31.957+0000 7f7b997fa640 10 monclient: dump: 2024-09-16T03:43:31.964 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:43:31.964 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:43:31.965 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:43:31.965 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:43:31.965 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:43:31.965 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:43:31.965 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:43:31.965 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:43:31.965 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:43:31.965 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:43:32.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.033+0000 7f7b997fa640 1 -- 172.21.15.42:0/2912281203 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4714 (secure 0 0 0) 0x7f7b90007510 con 0x7f7b9c1c60c0 2024-09-16T03:43:32.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.033+0000 7f7b997fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:43:32.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.033+0000 7f7b997fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:43:32.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.033+0000 7f7b997fa640 1 -- 172.21.15.42:0/2912281203 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7b9c1c60c0 msgr2=0x7f7b9c09fa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:43:32.039 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.033+0000 7f7b997fa640 1 --2- 172.21.15.42:0/2912281203 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7b9c1c60c0 0x7f7b9c09fa30 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f7b9000f540 tx=0x7f7b90007f90 comp rx=0 tx=0).stop 2024-09-16T03:43:32.039 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.033+0000 7f7b997fa640 1 -- 172.21.15.42:0/2912281203 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7b9c1c60c0 msgr2=0x7f7b9c09fa30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:43:32.039 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.033+0000 7f7b997fa640 1 --2- 172.21.15.42:0/2912281203 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7b9c1c60c0 0x7f7b9c09fa30 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:43:32.039 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:43:32.039 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:43:32.039 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:43:32.039 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "leader", 2024-09-16T03:43:32.039 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 16, 2024-09-16T03:43:32.040 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 91399, 2024-09-16T03:43:32.040 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:43:32.040 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:43:32.040 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T03:43:32.040 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T03:43:32.040 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:43:32.040 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 0, 2024-09-16T03:43:32.040 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:43:32.040 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:43:32.040 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:43:32.040 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:43:32.040 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:43:32.041 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:43:32.041 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:43:32.041 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:43:32.041 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:43:32.041 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:43:32.041 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:43:32.041 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:43:32.041 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:43:32.041 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:43:32.041 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:43:32.041 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:43:32.041 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:43:32.041 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:43:32.042 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:43:32.042 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:43:32.042 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:43:32.042 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:43:32.042 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:43:32.042 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:43:32.042 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:43:32.042 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:43:32.042 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:43:32.042 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:43:32.042 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:43:32.042 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:43:32.042 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:43:32.042 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:43:32.043 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:43:32.043 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:43:32.043 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:43:32.043 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:43:32.043 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:43:32.043 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:43:32.043 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:43:32.043 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:43:32.043 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:43:32.043 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:43:32.043 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:43:32.043 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:43:32.043 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:43:32.043 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:43:32.044 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:43:32.044 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:43:32.044 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:43:32.044 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:43:32.044 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:43:32.044 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:43:32.044 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:43:32.044 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:43:32.044 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:43:32.044 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:43:32.044 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:43:32.044 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:43:32.044 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:43:32.045 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:43:32.045 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:43:32.045 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:43:32.045 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:43:32.045 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:43:32.045 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:43:32.045 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:43:32.045 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:43:32.045 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:43:32.045 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:43:32.045 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:43:32.045 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:43:32.045 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:43:32.045 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:43:32.046 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:43:32.046 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:43:32.046 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:43:32.046 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:43:32.046 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:43:32.046 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:43:32.046 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:43:32.046 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:43:32.046 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:43:32.046 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:43:32.046 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:43:32.046 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:43:32.046 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:43:32.046 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:43:32.047 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:43:32.047 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:43:32.047 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:43:32.047 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:43:32.047 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:43:32.047 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:43:32.047 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:43:32.047 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:43:32.047 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:43:32.047 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:43:32.047 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:43:32.047 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:43:32.047 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:43:32.047 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:43:32.048 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:43:32.048 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:43:32.048 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:43:32.048 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:43:32.048 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:43:32.048 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:43:32.048 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:43:32.048 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:43:32.048 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:43:32.048 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:43:32.048 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:43:32.048 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:43:32.048 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:43:32.048 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:43:32.049 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:43:32.049 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:43:32.049 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:43:32.049 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:43:32.049 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:43:32.049 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:43:32.049 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:43:32.049 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:43:32.050 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:43:32.050 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:43:32.050 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:43:32.050 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:43:32.050 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:43:32.050 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:43:32.050 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:43:32.050 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:43:32.050 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:43:32.050 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:43:32.050 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:43:32.050 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:43:32.050 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:43:32.050 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:43:32.051 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:43:32.051 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:43:32.051 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:43:32.051 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:43:32.051 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T03:43:32.051 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:43:32.051 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:43:32.051 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:43:32.051 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 4 2024-09-16T03:43:32.051 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:43:32.051 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:43:32.051 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:43:32.051 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:43:32.051 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:43:32.052 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:43:32.052 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 12 2024-09-16T03:43:32.052 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:43:32.052 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:43:32.052 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:43:32.052 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:43:32.052 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:43:32.053 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.033+0000 7f7ba2b69640 10 librados: watch_flush enter 2024-09-16T03:43:32.053 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.033+0000 7f7ba2b69640 10 librados: watch_flush exit 2024-09-16T03:43:32.053 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.033+0000 7f7ba2b69640 20 client.4541.objecter shutdown clearing up homeless session... 2024-09-16T03:43:32.053 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.033+0000 7f7ba2b69640 10 client.4541.objecter successfully canceled tick 2024-09-16T03:43:32.053 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.033+0000 7f7ba2b69640 1 -- 172.21.15.42:0/2912281203 >> v2:172.21.15.154:6812/254845342 conn(0x7f7b88037810 msgr2=0x7f7b88039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:43:32.053 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.033+0000 7f7ba2b69640 1 --2- 172.21.15.42:0/2912281203 >> v2:172.21.15.154:6812/254845342 conn(0x7f7b88037810 0x7f7b88039cd0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f7b940ae000 tx=0x7f7b940b1040 comp rx=0 tx=0).stop 2024-09-16T03:43:32.053 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.033+0000 7f7ba2b69640 10 monclient: shutdown 2024-09-16T03:43:32.053 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.033+0000 7f7ba2b69640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:43:32.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.033+0000 7f7ba2b69640 1 -- 172.21.15.42:0/2912281203 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7b9c0a2230 msgr2=0x7f7b9c1cd820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:43:32.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.033+0000 7f7ba2b69640 1 --2- 172.21.15.42:0/2912281203 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7b9c0a2230 0x7f7b9c1cd820 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f7b8c00c500 tx=0x7f7b8c00c9c0 comp rx=0 tx=0).stop 2024-09-16T03:43:32.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.033+0000 7f7ba1106640 1 -- 172.21.15.42:0/2912281203 reap_dead start 2024-09-16T03:43:32.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.033+0000 7f7ba1106640 1 -- 172.21.15.42:0/2912281203 reap_dead start 2024-09-16T03:43:32.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.033+0000 7f7ba2b69640 1 -- 172.21.15.42:0/2912281203 shutdown_connections 2024-09-16T03:43:32.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.033+0000 7f7ba2b69640 1 -- 172.21.15.42:0/2912281203 >> 172.21.15.42:0/2912281203 conn(0x7f7b9c0a6e80 msgr2=0x7f7b9c0a54b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:43:32.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.033+0000 7f7ba2b69640 1 -- 172.21.15.42:0/2912281203 shutdown_connections 2024-09-16T03:43:32.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.033+0000 7f7ba2b69640 1 -- 172.21.15.42:0/2912281203 wait complete. 2024-09-16T03:43:32.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.033+0000 7f7ba2b69640 1 librados: shutdown 2024-09-16T03:43:32.061 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:43:32.061 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T03:43:32.061 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T03:43:32.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.181+0000 7f2d69cc8640 10 monclient: get_monmap_and_config 2024-09-16T03:43:32.187 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.181+0000 7f2d69cc8640 10 monclient: build_initial_monmap 2024-09-16T03:43:32.187 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.181+0000 7f2d69cc8640 10 monclient: monmap: 2024-09-16T03:43:32.187 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:43:32.187 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:43:32.187 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:43:32.184618+0000 2024-09-16T03:43:32.188 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:43:32.184618+0000 2024-09-16T03:43:32.188 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:43:32.188 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:43:32.188 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:43:32.188 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:43:32.188 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:43:32.188 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:43:32.188 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:43:32.188 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:43:32.197 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.181+0000 7f2d69cc8640 1 Processor -- start 2024-09-16T03:43:32.198 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.181+0000 7f2d69cc8640 1 -- start start 2024-09-16T03:43:32.198 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.181+0000 7f2d69cc8640 10 monclient: init 2024-09-16T03:43:32.198 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.181+0000 7f2d69cc8640 10 monclient: _reopen_session rank -1 2024-09-16T03:43:32.198 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.181+0000 7f2d69cc8640 10 monclient: _add_conns ranks=[0,2,4,3,1] 2024-09-16T03:43:32.198 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.181+0000 7f2d69cc8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2d640a1330 addr v1:172.21.15.42:6789/0 2024-09-16T03:43:32.198 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.181+0000 7f2d69cc8640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2d6406dac0 0x7f2d6413a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:43:32.198 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.181+0000 7f2d69cc8640 10 monclient(hunting): picked mon.noname-b con 0x7f2d6406dac0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:43:32.198 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.181+0000 7f2d69cc8640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f2d640a1e20 0x7f2d6413c890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:43:32.198 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.181+0000 7f2d69cc8640 10 monclient(hunting): picked mon.noname-c con 0x7f2d640a1e20 addr v2:172.21.15.154:3300/0 2024-09-16T03:43:32.198 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.181+0000 7f2d69cc8640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2d6409c7d0 con 0x7f2d640a1330 2024-09-16T03:43:32.198 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.181+0000 7f2d69cc8640 10 monclient(hunting): start opening mon connection 2024-09-16T03:43:32.198 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.181+0000 7f2d69cc8640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f2d6409c090 con 0x7f2d640a1e20 2024-09-16T03:43:32.198 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.181+0000 7f2d69cc8640 10 monclient(hunting): start opening mon connection 2024-09-16T03:43:32.198 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.181+0000 7f2d69cc8640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f2d6409f610 con 0x7f2d6406dac0 2024-09-16T03:43:32.199 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.181+0000 7f2d69cc8640 10 monclient(hunting): _renew_subs 2024-09-16T03:43:32.199 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.181+0000 7f2d69cc8640 10 monclient(hunting): authenticate will time out at 1059.153809s 2024-09-16T03:43:32.199 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.181+0000 7f2d62ffd640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2d6406dac0 0x7f2d6413a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:43:32.199 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.181+0000 7f2d62ffd640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2d6406dac0 0x7f2d6413a340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:37682/0 (socket says 172.21.15.42:37682) 2024-09-16T03:43:32.199 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.181+0000 7f2d62ffd640 1 -- 172.21.15.42:0/1957812452 learned_addr learned my addr 172.21.15.42:0/1957812452 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:43:32.199 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.181+0000 7f2d62ffd640 10 monclient(hunting): get_auth_request con 0x7f2d6406dac0 auth_method 0 2024-09-16T03:43:32.199 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.181+0000 7f2d62ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:43:32.199 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.181+0000 7f2d62ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:43:32.199 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.181+0000 7f2d62ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:43:32.199 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.181+0000 7f2d62ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:43:32.199 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.181+0000 7f2d62ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:43:32.199 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.181+0000 7f2d62ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:43:32.199 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.181+0000 7f2d63fff640 1 --2- 172.21.15.42:0/1957812452 >> v2:172.21.15.154:3300/0 conn(0x7f2d640a1e20 0x7f2d6413c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:43:32.200 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.181+0000 7f2d63fff640 10 monclient(hunting): get_auth_request con 0x7f2d640a1e20 auth_method 0 2024-09-16T03:43:32.200 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.181+0000 7f2d63fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:43:32.200 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.181+0000 7f2d63fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:43:32.200 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.181+0000 7f2d63fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:43:32.200 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.181+0000 7f2d62ffd640 10 monclient(hunting): handle_auth_done global_id 4544 payload 274 2024-09-16T03:43:32.200 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.181+0000 7f2d62ffd640 1 -- 172.21.15.42:0/1957812452 >> v2:172.21.15.154:3300/0 conn(0x7f2d640a1e20 msgr2=0x7f2d6413c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:43:32.200 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.181+0000 7f2d62ffd640 1 --2- 172.21.15.42:0/1957812452 >> v2:172.21.15.154:3300/0 conn(0x7f2d640a1e20 0x7f2d6413c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:43:32.200 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.181+0000 7f2d62ffd640 1 -- 172.21.15.42:0/1957812452 >> v1:172.21.15.42:6789/0 conn(0x7f2d640a1330 legacy=0x7f2d640a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:43:32.200 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.181+0000 7f2d62ffd640 10 monclient: _finish_hunting 0 2024-09-16T03:43:32.200 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.181+0000 7f2d62ffd640 1 monclient: found mon.noname-b 2024-09-16T03:43:32.200 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.181+0000 7f2d62ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:43:32.200 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.181+0000 7f2d62ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T03:43:32.200 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.181+0000 7f2d62ffd640 1 -- 172.21.15.42:0/1957812452 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d6409f8f0 con 0x7f2d6406dac0 2024-09-16T03:43:32.201 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.185+0000 7f2d62ffd640 1 --2- 172.21.15.42:0/1957812452 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2d6406dac0 0x7f2d6413a340 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f2d5c009e50 tx=0x7f2d5c01c9e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b5ca98d038c5a56f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:43:32.201 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.185+0000 7f2d627fc640 1 -- 172.21.15.42:0/1957812452 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d5c0a58d0 con 0x7f2d6406dac0 2024-09-16T03:43:32.201 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.185+0000 7f2d627fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:43:32.201 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.185+0000 7f2d627fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:43:32.201 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.185+0000 7f2d627fc640 10 monclient: dump: 2024-09-16T03:43:32.201 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:43:32.201 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:43:32.201 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:43:32.201 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:43:32.201 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:43:32.201 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:43:32.201 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:43:32.201 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:43:32.202 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:43:32.202 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:43:32.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.185+0000 7f2d627fc640 10 monclient: _finish_auth 0 2024-09-16T03:43:32.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.185+0000 7f2d627fc640 10 monclient: _check_auth_tickets 2024-09-16T03:43:32.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.185+0000 7f2d627fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:43:32.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.185+0000 7f2d69cc8640 5 monclient: authenticate success, global_id 4544 2024-09-16T03:43:32.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.185+0000 7f2d69cc8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:43:32.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.185+0000 7f2d627fc640 1 -- 172.21.15.42:0/1957812452 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d5c0a5a70 con 0x7f2d6406dac0 2024-09-16T03:43:32.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.185+0000 7f2d627fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:43:32.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.185+0000 7f2d69cc8640 10 monclient: get_monmap_and_config success 2024-09-16T03:43:32.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.185+0000 7f2d627fc640 1 -- 172.21.15.42:0/1957812452 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d5c0a5e40 con 0x7f2d6406dac0 2024-09-16T03:43:32.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.185+0000 7f2d69cc8640 4 set_mon_vals no callback set 2024-09-16T03:43:32.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.185+0000 7f2d627fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:43:32.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.185+0000 7f2d627fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:43:32.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.185+0000 7f2d627fc640 10 monclient: dump: 2024-09-16T03:43:32.203 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:43:32.203 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:43:32.203 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:43:32.203 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:43:32.203 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:43:32.203 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:43:32.203 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:43:32.203 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:43:32.203 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:43:32.203 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:43:32.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.185+0000 7f2d69cc8640 10 monclient: shutdown 2024-09-16T03:43:32.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.185+0000 7f2d69cc8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:43:32.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.185+0000 7f2d69cc8640 1 -- 172.21.15.42:0/1957812452 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2d6406dac0 msgr2=0x7f2d6413a340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:43:32.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.185+0000 7f2d69cc8640 1 --2- 172.21.15.42:0/1957812452 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2d6406dac0 0x7f2d6413a340 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f2d5c009e50 tx=0x7f2d5c01c9e0 comp rx=0 tx=0).stop 2024-09-16T03:43:32.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.185+0000 7f2d69cc8640 1 -- 172.21.15.42:0/1957812452 shutdown_connections 2024-09-16T03:43:32.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.185+0000 7f2d69cc8640 1 --2- 172.21.15.42:0/1957812452 >> v2:172.21.15.154:3300/0 conn(0x7f2d640a1e20 0x7f2d6413c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:43:32.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.185+0000 7f2d69cc8640 1 --2- 172.21.15.42:0/1957812452 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2d6406dac0 0x7f2d6413a340 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:43:32.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.185+0000 7f2d69cc8640 1 -- 172.21.15.42:0/1957812452 >> 172.21.15.42:0/1957812452 conn(0x7f2d640a2420 msgr2=0x7f2d640a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:43:32.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.185+0000 7f2d69cc8640 1 -- 172.21.15.42:0/1957812452 shutdown_connections 2024-09-16T03:43:32.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.185+0000 7f2d69cc8640 1 -- 172.21.15.42:0/1957812452 wait complete. 2024-09-16T03:43:32.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.185+0000 7f2d69cc8640 10 monclient: build_initial_monmap 2024-09-16T03:43:32.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.185+0000 7f2d69cc8640 10 monclient: monmap: 2024-09-16T03:43:32.204 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:43:32.204 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:43:32.204 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:43:32.205 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:43:32.205 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:43:32.205 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:43:32.205 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:43:32.205 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:43:32.205 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:43:32.205 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:43:32.205 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.185+0000 7f2d69cc8640 1 librados: starting msgr at 2024-09-16T03:43:32.205 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.185+0000 7f2d69cc8640 1 librados: starting objecter 2024-09-16T03:43:32.205 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.185+0000 7f2d69cc8640 1 Processor -- start 2024-09-16T03:43:32.205 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.185+0000 7f2d69cc8640 1 -- start start 2024-09-16T03:43:32.205 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.185+0000 7f2d69cc8640 1 librados: setting wanted keys 2024-09-16T03:43:32.205 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.185+0000 7f2d69cc8640 1 librados: calling monclient init 2024-09-16T03:43:32.205 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.185+0000 7f2d69cc8640 10 monclient: init 2024-09-16T03:43:32.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.185+0000 7f2d69cc8640 10 monclient: _reopen_session rank -1 2024-09-16T03:43:32.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.185+0000 7f2d69cc8640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T03:43:32.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.185+0000 7f2d69cc8640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2d640a1330 0x7f2d64071100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:43:32.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.185+0000 7f2d69cc8640 10 monclient(hunting): picked mon.noname-a con 0x7f2d640a1330 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:43:32.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.185+0000 7f2d69cc8640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d640a1e20 0x7f2d641325b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:43:32.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.185+0000 7f2d69cc8640 10 monclient(hunting): picked mon.noname-b con 0x7f2d640a1e20 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:43:32.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.185+0000 7f2d69cc8640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2d64132af0 0x7f2d641cedf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:43:32.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.185+0000 7f2d69cc8640 10 monclient(hunting): picked mon.noname-c con 0x7f2d64132af0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:43:32.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.185+0000 7f2d69cc8640 10 monclient(hunting): start opening mon connection 2024-09-16T03:43:32.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.185+0000 7f2d69cc8640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f2d6413f770 con 0x7f2d640a1330 2024-09-16T03:43:32.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.185+0000 7f2d69cc8640 10 monclient(hunting): start opening mon connection 2024-09-16T03:43:32.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.185+0000 7f2d69cc8640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f2d6413f5f0 con 0x7f2d640a1e20 2024-09-16T03:43:32.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.185+0000 7f2d69cc8640 10 monclient(hunting): start opening mon connection 2024-09-16T03:43:32.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.185+0000 7f2d69cc8640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f2d6413f8f0 con 0x7f2d64132af0 2024-09-16T03:43:32.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.185+0000 7f2d69cc8640 10 monclient(hunting): _renew_subs 2024-09-16T03:43:32.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.185+0000 7f2d69cc8640 10 monclient(hunting): authenticate will time out at 1059.158811s 2024-09-16T03:43:32.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.185+0000 7f2d63fff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2d64132af0 0x7f2d641cedf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:43:32.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.185+0000 7f2d637fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2d640a1330 0x7f2d64071100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:43:32.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.185+0000 7f2d63fff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2d64132af0 0x7f2d641cedf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:37692/0 (socket says 172.21.15.42:37692) 2024-09-16T03:43:32.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.185+0000 7f2d63fff640 1 -- 172.21.15.42:0/551534357 learned_addr learned my addr 172.21.15.42:0/551534357 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:43:32.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.185+0000 7f2d63fff640 10 monclient(hunting): get_auth_request con 0x7f2d64132af0 auth_method 0 2024-09-16T03:43:32.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.185+0000 7f2d63fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:43:32.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.185+0000 7f2d63fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:43:32.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.185+0000 7f2d63fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:43:32.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.185+0000 7f2d637fe640 10 monclient(hunting): get_auth_request con 0x7f2d640a1330 auth_method 0 2024-09-16T03:43:32.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.185+0000 7f2d637fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:43:32.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.185+0000 7f2d637fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:43:32.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.185+0000 7f2d637fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:43:32.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.185+0000 7f2d63fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:43:32.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.185+0000 7f2d63fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:43:32.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.185+0000 7f2d63fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:43:32.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.185+0000 7f2d62ffd640 1 --2- 172.21.15.42:0/551534357 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d640a1e20 0x7f2d641325b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:43:32.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.185+0000 7f2d62ffd640 10 monclient(hunting): get_auth_request con 0x7f2d640a1e20 auth_method 0 2024-09-16T03:43:32.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.185+0000 7f2d62ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:43:32.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.185+0000 7f2d62ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:43:32.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.185+0000 7f2d62ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:43:32.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.185+0000 7f2d63fff640 10 monclient(hunting): handle_auth_done global_id 4547 payload 824 2024-09-16T03:43:32.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.185+0000 7f2d63fff640 1 -- 172.21.15.42:0/551534357 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d640a1e20 msgr2=0x7f2d641325b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:43:32.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.185+0000 7f2d63fff640 1 --2- 172.21.15.42:0/551534357 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d640a1e20 0x7f2d641325b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:43:32.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.185+0000 7f2d63fff640 1 -- 172.21.15.42:0/551534357 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2d640a1330 msgr2=0x7f2d64071100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:43:32.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.185+0000 7f2d63fff640 1 --2- 172.21.15.42:0/551534357 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2d640a1330 0x7f2d64071100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:43:32.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.185+0000 7f2d63fff640 10 monclient: _finish_hunting 0 2024-09-16T03:43:32.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.185+0000 7f2d63fff640 1 monclient: found mon.noname-c 2024-09-16T03:43:32.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.185+0000 7f2d63fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:43:32.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.185+0000 7f2d63fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:43:32.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.185+0000 7f2d63fff640 1 -- 172.21.15.42:0/551534357 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d64073a30 con 0x7f2d64132af0 2024-09-16T03:43:32.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.185+0000 7f2d637fe640 1 --2- 172.21.15.42:0/551534357 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2d640a1330 0x7f2d64071100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T03:43:32.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.185+0000 7f2d63fff640 1 --2- 172.21.15.42:0/551534357 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2d64132af0 0x7f2d641cedf0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f2d5400d800 tx=0x7f2d5400dcc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:43:32.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.185+0000 7f2d4bfff640 10 client.?.objecter ms_handle_connect 0x7f2d64132af0 2024-09-16T03:43:32.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.185+0000 7f2d4bfff640 10 client.?.objecter resend_mon_ops 2024-09-16T03:43:32.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.185+0000 7f2d4bfff640 1 -- 172.21.15.42:0/551534357 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d5400b6c0 con 0x7f2d64132af0 2024-09-16T03:43:32.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.185+0000 7f2d4bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:43:32.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.185+0000 7f2d4bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:43:32.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.185+0000 7f2d4bfff640 10 monclient: dump: 2024-09-16T03:43:32.210 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:43:32.210 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:43:32.210 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:43:32.210 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:43:32.210 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:43:32.210 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:43:32.210 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:43:32.210 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:43:32.210 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:43:32.210 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:43:32.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.185+0000 7f2d4bfff640 10 monclient: _finish_auth 0 2024-09-16T03:43:32.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.185+0000 7f2d4bfff640 10 monclient: _check_auth_tickets 2024-09-16T03:43:32.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.185+0000 7f2d4bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:43:32.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.185+0000 7f2d69cc8640 5 monclient: authenticate success, global_id 4547 2024-09-16T03:43:32.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.185+0000 7f2d4bfff640 1 -- 172.21.15.42:0/551534357 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d54019070 con 0x7f2d64132af0 2024-09-16T03:43:32.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.185+0000 7f2d4bfff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:43:32.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.185+0000 7f2d4bfff640 1 -- 172.21.15.42:0/551534357 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d5400baa0 con 0x7f2d64132af0 2024-09-16T03:43:32.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.185+0000 7f2d4bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:43:32.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.185+0000 7f2d4bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:43:32.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.185+0000 7f2d4bfff640 10 monclient: dump: 2024-09-16T03:43:32.211 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:43:32.211 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:43:32.211 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:43:32.211 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:43:32.211 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:43:32.211 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:43:32.211 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:43:32.211 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:43:32.212 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:43:32.212 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:43:32.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.185+0000 7f2d61ffb640 4 set_mon_vals no callback set 2024-09-16T03:43:32.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.185+0000 7f2d69cc8640 10 monclient: _renew_subs 2024-09-16T03:43:32.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.185+0000 7f2d69cc8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:43:32.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.185+0000 7f2d69cc8640 1 -- 172.21.15.42:0/551534357 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d6411ec80 con 0x7f2d64132af0 2024-09-16T03:43:32.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.185+0000 7f2d69cc8640 10 client.4547.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:43:32.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.185+0000 7f2d69cc8640 10 monclient: _renew_subs 2024-09-16T03:43:32.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.185+0000 7f2d69cc8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:43:32.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.185+0000 7f2d69cc8640 1 -- 172.21.15.42:0/551534357 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d6411e9a0 con 0x7f2d64132af0 2024-09-16T03:43:32.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.185+0000 7f2d69cc8640 1 librados: init done 2024-09-16T03:43:32.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.189+0000 7f2d69cc8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:43:32.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.189+0000 7f2d69cc8640 1 --2- 172.21.15.42:0/551534357 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2d6406dac0 0x7f2d6402c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:43:32.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.189+0000 7f2d69cc8640 10 monclient(hunting): start opening mon connection 2024-09-16T03:43:32.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.189+0000 7f2d69cc8640 1 -- 172.21.15.42:0/551534357 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f2d6411d600 con 0x7f2d6406dac0 2024-09-16T03:43:32.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.189+0000 7f2d637fe640 1 --2- 172.21.15.42:0/551534357 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2d6406dac0 0x7f2d6402c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:43:32.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.189+0000 7f2d637fe640 10 monclient: get_auth_request con 0x7f2d6406dac0 auth_method 0 2024-09-16T03:43:32.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.189+0000 7f2d637fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:43:32.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.189+0000 7f2d637fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:43:32.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.189+0000 7f2d637fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:43:32.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.189+0000 7f2d4bfff640 1 -- 172.21.15.42:0/551534357 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f2d5400bc40 con 0x7f2d64132af0 2024-09-16T03:43:32.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.189+0000 7f2d637fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:43:32.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.189+0000 7f2d637fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:43:32.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.189+0000 7f2d637fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:43:32.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.189+0000 7f2d4bfff640 1 --2- 172.21.15.42:0/551534357 >> v2:172.21.15.154:6812/254845342 conn(0x7f2d50037810 0x7f2d50039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:43:32.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.189+0000 7f2d637fe640 10 monclient(hunting): handle_auth_done global_id 4550 payload 824 2024-09-16T03:43:32.214 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.189+0000 7f2d637fe640 1 -- 172.21.15.42:0/551534357 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2d6409ccf0 con 0x7f2d6406dac0 2024-09-16T03:43:32.214 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.189+0000 7f2d4bfff640 1 -- 172.21.15.42:0/551534357 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(19..19 src has 1..19) ==== 3534+0+0 (secure 0 0 0) 0x7f2d54048be0 con 0x7f2d64132af0 2024-09-16T03:43:32.214 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.189+0000 7f2d4bfff640 10 client.4547.objecter ms_dispatch 0x7f2d64000ba0 osd_map(19..19 src has 1..19) 2024-09-16T03:43:32.214 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.189+0000 7f2d4bfff640 3 client.4547.objecter handle_osd_map got epochs [19,19] > 0 2024-09-16T03:43:32.214 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.189+0000 7f2d4bfff640 3 client.4547.objecter handle_osd_map decoding full epoch 19 2024-09-16T03:43:32.214 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.189+0000 7f2d4bfff640 20 client.4547.objecter dump_active .. 0 homeless 2024-09-16T03:43:32.214 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.189+0000 7f2d4bfff640 10 client.4547.objecter ms_handle_connect 0x7f2d6406dac0 2024-09-16T03:43:32.214 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.189+0000 7f2d637fe640 1 --2- 172.21.15.42:0/551534357 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2d6406dac0 0x7f2d6402c650 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f2d5800e4e0 tx=0x7f2d5800e9a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:43:32.214 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.189+0000 7f2d4bfff640 10 client.4547.objecter resend_mon_ops 2024-09-16T03:43:32.214 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.189+0000 7f2d4bfff640 1 -- 172.21.15.42:0/551534357 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d580052e0 con 0x7f2d6406dac0 2024-09-16T03:43:32.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.189+0000 7f2d4bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:43:32.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.189+0000 7f2d4bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:43:32.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.189+0000 7f2d4bfff640 10 monclient: dump: 2024-09-16T03:43:32.215 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:43:32.215 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:43:32.215 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:43:32.215 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:43:32.215 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:43:32.215 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:43:32.215 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:43:32.215 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:43:32.215 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:43:32.215 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:43:32.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.189+0000 7f2d62ffd640 1 --2- 172.21.15.42:0/551534357 >> v2:172.21.15.154:6812/254845342 conn(0x7f2d50037810 0x7f2d50039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:43:32.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.189+0000 7f2d62ffd640 10 monclient: get_auth_request con 0x7f2d50037810 auth_method 0 2024-09-16T03:43:32.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.189+0000 7f2d4bfff640 1 -- 172.21.15.42:0/551534357 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f2d580052e0 con 0x7f2d6406dac0 2024-09-16T03:43:32.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.189+0000 7f2d4bfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:43:32.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.189+0000 7f2d4bfff640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:43:32.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.189+0000 7f2d4bfff640 1 -- 172.21.15.42:0/551534357 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2d6406dac0 msgr2=0x7f2d6402c650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:43:32.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.189+0000 7f2d4bfff640 1 --2- 172.21.15.42:0/551534357 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2d6406dac0 0x7f2d6402c650 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f2d5800e4e0 tx=0x7f2d5800e9a0 comp rx=0 tx=0).stop 2024-09-16T03:43:32.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.189+0000 7f2d4bfff640 1 -- 172.21.15.42:0/551534357 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2d6406dac0 msgr2=0x7f2d6402c650 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:43:32.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.189+0000 7f2d4bfff640 1 --2- 172.21.15.42:0/551534357 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2d6406dac0 0x7f2d6402c650 secure :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f2d5800e4e0 tx=0x7f2d5800e9a0 comp rx=0 tx=0).stop 2024-09-16T03:43:32.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.189+0000 7f2d4bfff640 10 client.4547.objecter ms_handle_connect 0x7f2d50037810 2024-09-16T03:43:32.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.189+0000 7f2d62ffd640 1 --2- 172.21.15.42:0/551534357 >> v2:172.21.15.154:6812/254845342 conn(0x7f2d50037810 0x7f2d50039cd0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f2d5c0ae000 tx=0x7f2d5c0a7070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:43:32.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.201+0000 7f2d69cc8640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:43:32.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.201+0000 7f2d69cc8640 1 --2- 172.21.15.42:0/551534357 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2d6402cb90 0x7f2d6402cf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:43:32.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.201+0000 7f2d69cc8640 10 monclient(hunting): start opening mon connection 2024-09-16T03:43:32.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.201+0000 7f2d69cc8640 1 -- 172.21.15.42:0/551534357 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f2d640a1710 con 0x7f2d6402cb90 2024-09-16T03:43:32.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.201+0000 7f2d637fe640 1 --2- 172.21.15.42:0/551534357 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2d6402cb90 0x7f2d6402cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:43:32.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.201+0000 7f2d637fe640 10 monclient: get_auth_request con 0x7f2d6402cb90 auth_method 0 2024-09-16T03:43:32.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.201+0000 7f2d637fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:43:32.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.201+0000 7f2d637fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:43:32.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.201+0000 7f2d637fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:43:32.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.201+0000 7f2d637fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:43:32.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.201+0000 7f2d637fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:43:32.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.201+0000 7f2d637fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:43:32.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.205+0000 7f2d637fe640 10 monclient(hunting): handle_auth_done global_id 4553 payload 824 2024-09-16T03:43:32.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.205+0000 7f2d637fe640 1 -- 172.21.15.42:0/551534357 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f2d64135890 con 0x7f2d6402cb90 2024-09-16T03:43:32.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.205+0000 7f2d637fe640 1 --2- 172.21.15.42:0/551534357 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2d6402cb90 0x7f2d6402cf70 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f2d5800e4e0 tx=0x7f2d58000ee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:43:32.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.205+0000 7f2d4bfff640 10 client.4547.objecter ms_handle_connect 0x7f2d6402cb90 2024-09-16T03:43:32.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.205+0000 7f2d4bfff640 10 client.4547.objecter resend_mon_ops 2024-09-16T03:43:32.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.205+0000 7f2d4bfff640 1 -- 172.21.15.42:0/551534357 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d58007230 con 0x7f2d6402cb90 2024-09-16T03:43:32.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.205+0000 7f2d4bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:43:32.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.205+0000 7f2d4bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:43:32.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.205+0000 7f2d4bfff640 10 monclient: dump: 2024-09-16T03:43:32.219 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:43:32.219 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:43:32.219 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:43:32.219 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:43:32.220 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:43:32.220 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:43:32.220 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:43:32.220 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:43:32.220 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:43:32.220 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:43:32.220 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.205+0000 7f2d4bfff640 1 -- 172.21.15.42:0/551534357 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4879 (secure 0 0 0) 0x7f2d58007230 con 0x7f2d6402cb90 2024-09-16T03:43:32.220 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.205+0000 7f2d4bfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:43:32.220 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.205+0000 7f2d4bfff640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:43:32.220 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.205+0000 7f2d4bfff640 1 -- 172.21.15.42:0/551534357 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2d6402cb90 msgr2=0x7f2d6402cf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:43:32.220 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.205+0000 7f2d4bfff640 1 --2- 172.21.15.42:0/551534357 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2d6402cb90 0x7f2d6402cf70 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f2d5800e4e0 tx=0x7f2d58000ee0 comp rx=0 tx=0).stop 2024-09-16T03:43:32.220 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.205+0000 7f2d4bfff640 1 -- 172.21.15.42:0/551534357 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2d6402cb90 msgr2=0x7f2d6402cf70 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:43:32.220 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.205+0000 7f2d4bfff640 1 --2- 172.21.15.42:0/551534357 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2d6402cb90 0x7f2d6402cf70 secure :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f2d5800e4e0 tx=0x7f2d58000ee0 comp rx=0 tx=0).stop 2024-09-16T03:43:32.221 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:43:32.221 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:43:32.221 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:43:32.221 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T03:43:32.221 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 16, 2024-09-16T03:43:32.221 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 91575, 2024-09-16T03:43:32.221 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:43:32.221 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:43:32.221 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T03:43:32.221 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T03:43:32.221 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:43:32.222 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 1, 2024-09-16T03:43:32.222 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:43:32.222 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:43:32.222 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:43:32.222 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:43:32.222 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:43:32.222 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:43:32.222 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:43:32.222 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:43:32.222 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:43:32.222 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:43:32.222 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:43:32.222 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:43:32.222 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:43:32.223 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:43:32.223 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:43:32.223 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:43:32.223 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:43:32.223 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:43:32.223 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:43:32.223 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:43:32.223 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:43:32.223 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:43:32.223 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:43:32.223 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:43:32.223 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:43:32.223 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:43:32.223 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:43:32.224 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:43:32.224 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:43:32.224 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:43:32.224 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:43:32.224 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:43:32.224 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:43:32.224 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:43:32.224 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:43:32.224 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:43:32.224 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:43:32.224 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:43:32.224 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:43:32.225 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:43:32.225 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:43:32.225 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:43:32.225 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:43:32.225 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:43:32.225 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:43:32.225 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:43:32.225 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:43:32.225 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:43:32.225 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:43:32.225 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:43:32.225 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:43:32.225 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:43:32.225 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:43:32.226 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:43:32.226 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:43:32.226 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:43:32.226 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:43:32.226 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:43:32.226 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:43:32.226 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:43:32.226 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:43:32.226 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:43:32.226 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:43:32.226 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:43:32.226 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:43:32.226 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:43:32.226 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:43:32.227 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:43:32.227 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:43:32.227 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:43:32.227 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:43:32.227 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:43:32.227 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:43:32.227 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:43:32.227 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:43:32.227 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:43:32.227 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:43:32.227 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:43:32.227 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:43:32.227 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:43:32.228 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:43:32.228 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:43:32.228 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:43:32.228 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:43:32.228 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:43:32.228 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:43:32.228 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:43:32.228 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:43:32.228 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:43:32.228 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:43:32.228 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:43:32.228 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:43:32.228 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:43:32.228 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:43:32.229 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:43:32.229 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:43:32.229 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:43:32.229 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:43:32.229 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:43:32.229 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:43:32.229 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:43:32.229 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:43:32.229 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:43:32.229 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:43:32.229 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:43:32.229 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:43:32.229 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:43:32.229 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:43:32.230 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:43:32.230 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:43:32.230 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:43:32.230 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:43:32.230 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:43:32.230 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:43:32.230 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:43:32.230 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:43:32.230 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:43:32.230 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:43:32.230 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:43:32.230 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:43:32.230 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:43:32.230 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:43:32.231 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:43:32.231 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:43:32.231 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:43:32.231 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:43:32.231 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:43:32.231 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:43:32.231 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:43:32.231 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:43:32.231 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:43:32.231 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:43:32.231 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:43:32.231 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:43:32.231 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:43:32.231 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:43:32.232 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:43:32.232 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:43:32.232 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:43:32.232 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:43:32.232 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:43:32.232 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T03:43:32.232 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:43:32.232 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:43:32.232 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:43:32.232 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 2 2024-09-16T03:43:32.232 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:43:32.232 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:43:32.232 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:43:32.232 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:43:32.233 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:43:32.233 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:43:32.233 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 26 2024-09-16T03:43:32.233 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:43:32.233 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:43:32.233 INFO:teuthology.orchestra.run.smithi042.stdout: "mgr": [ 2024-09-16T03:43:32.233 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:43:32.233 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:43:32.233 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:43:32.233 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:43:32.233 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:43:32.233 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:43:32.233 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:43:32.234 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:43:32.234 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:43:32.235 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.205+0000 7f2d69cc8640 10 librados: watch_flush enter 2024-09-16T03:43:32.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.205+0000 7f2d69cc8640 10 librados: watch_flush exit 2024-09-16T03:43:32.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.205+0000 7f2d69cc8640 20 client.4547.objecter shutdown clearing up homeless session... 2024-09-16T03:43:32.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.205+0000 7f2d69cc8640 10 client.4547.objecter successfully canceled tick 2024-09-16T03:43:32.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.205+0000 7f2d69cc8640 1 -- 172.21.15.42:0/551534357 >> v2:172.21.15.154:6812/254845342 conn(0x7f2d50037810 msgr2=0x7f2d50039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:43:32.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.205+0000 7f2d69cc8640 1 --2- 172.21.15.42:0/551534357 >> v2:172.21.15.154:6812/254845342 conn(0x7f2d50037810 0x7f2d50039cd0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f2d5c0ae000 tx=0x7f2d5c0a7070 comp rx=0 tx=0).stop 2024-09-16T03:43:32.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.205+0000 7f2d69cc8640 10 monclient: shutdown 2024-09-16T03:43:32.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.205+0000 7f2d69cc8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:43:32.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.205+0000 7f2d69cc8640 1 -- 172.21.15.42:0/551534357 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2d64132af0 msgr2=0x7f2d641cedf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:43:32.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.205+0000 7f2d69cc8640 1 --2- 172.21.15.42:0/551534357 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2d64132af0 0x7f2d641cedf0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f2d5400d800 tx=0x7f2d5400dcc0 comp rx=0 tx=0).stop 2024-09-16T03:43:32.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.205+0000 7f2d63fff640 1 -- 172.21.15.42:0/551534357 reap_dead start 2024-09-16T03:43:32.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.205+0000 7f2d63fff640 1 -- 172.21.15.42:0/551534357 reap_dead start 2024-09-16T03:43:32.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.205+0000 7f2d69cc8640 1 -- 172.21.15.42:0/551534357 shutdown_connections 2024-09-16T03:43:32.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.205+0000 7f2d69cc8640 1 -- 172.21.15.42:0/551534357 >> 172.21.15.42:0/551534357 conn(0x7f2d640a2420 msgr2=0x7f2d641332f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:43:32.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.205+0000 7f2d69cc8640 1 -- 172.21.15.42:0/551534357 shutdown_connections 2024-09-16T03:43:32.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.205+0000 7f2d69cc8640 1 -- 172.21.15.42:0/551534357 wait complete. 2024-09-16T03:43:32.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.205+0000 7f2d69cc8640 1 librados: shutdown 2024-09-16T03:43:32.237 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:43:32.237 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T03:43:32.237 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T03:43:32.380 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.373+0000 7ff250866640 10 monclient: get_monmap_and_config 2024-09-16T03:43:32.380 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.373+0000 7ff250866640 10 monclient: build_initial_monmap 2024-09-16T03:43:32.380 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.373+0000 7ff250866640 10 monclient: monmap: 2024-09-16T03:43:32.380 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:43:32.380 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:43:32.380 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:43:32.378663+0000 2024-09-16T03:43:32.380 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:43:32.378663+0000 2024-09-16T03:43:32.380 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:43:32.380 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:43:32.381 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:43:32.381 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:43:32.381 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:43:32.381 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:43:32.381 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:43:32.381 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:43:32.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.373+0000 7ff250866640 1 Processor -- start 2024-09-16T03:43:32.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.373+0000 7ff250866640 1 -- start start 2024-09-16T03:43:32.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.373+0000 7ff250866640 10 monclient: init 2024-09-16T03:43:32.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.373+0000 7ff250866640 10 monclient: _reopen_session rank -1 2024-09-16T03:43:32.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.373+0000 7ff250866640 10 monclient: _add_conns ranks=[1,0,4,2,3] 2024-09-16T03:43:32.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.373+0000 7ff250866640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff24c0a39b0 addr v1:172.21.15.154:6789/0 2024-09-16T03:43:32.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.373+0000 7ff250866640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff24c0a3d90 addr v1:172.21.15.42:6789/0 2024-09-16T03:43:32.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.373+0000 7ff250866640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7ff24c13daa0 0x7ff24c13fe90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:43:32.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.373+0000 7ff250866640 10 monclient(hunting): picked mon.noname-c con 0x7ff24c13daa0 addr v2:172.21.15.154:3300/0 2024-09-16T03:43:32.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.373+0000 7ff250866640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff24c134290 con 0x7ff24c0a3d90 2024-09-16T03:43:32.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.373+0000 7ff250866640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff24c09c140 con 0x7ff24c0a39b0 2024-09-16T03:43:32.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.373+0000 7ff250866640 10 monclient(hunting): start opening mon connection 2024-09-16T03:43:32.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.373+0000 7ff250866640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7ff24c09c420 con 0x7ff24c13daa0 2024-09-16T03:43:32.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.373+0000 7ff250866640 10 monclient(hunting): _renew_subs 2024-09-16T03:43:32.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.373+0000 7ff250866640 10 monclient(hunting): authenticate will time out at 1059.347934s 2024-09-16T03:43:32.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.377+0000 7ff249d9b640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7ff24c0a3d90 0x7ff24c13c380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:33038/0 (socket says 172.21.15.42:33038) 2024-09-16T03:43:32.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.377+0000 7ff249d9b640 1 -- 172.21.15.42:0/3647060086 learned_addr learned my addr 172.21.15.42:0/3647060086 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:43:32.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.377+0000 7ff24ad9d640 1 --2- 172.21.15.42:0/3647060086 >> v2:172.21.15.154:3300/0 conn(0x7ff24c13daa0 0x7ff24c13fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:43:32.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.377+0000 7ff24ad9d640 10 monclient(hunting): get_auth_request con 0x7ff24c13daa0 auth_method 0 2024-09-16T03:43:32.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.377+0000 7ff24ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:43:32.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.377+0000 7ff24ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:43:32.385 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.377+0000 7ff24ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:43:32.385 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.377+0000 7ff24ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:43:32.389 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.377+0000 7ff24ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:43:32.389 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.377+0000 7ff24ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:43:32.389 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.377+0000 7ff24ad9d640 10 monclient(hunting): handle_auth_done global_id 14206 payload 274 2024-09-16T03:43:32.389 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.377+0000 7ff24ad9d640 1 -- 172.21.15.42:0/3647060086 >> v1:172.21.15.154:6789/0 conn(0x7ff24c0a39b0 legacy=0x7ff24c133b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:43:32.389 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.377+0000 7ff24ad9d640 1 -- 172.21.15.42:0/3647060086 >> v1:172.21.15.42:6789/0 conn(0x7ff24c0a3d90 legacy=0x7ff24c13c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:43:32.389 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.377+0000 7ff24ad9d640 10 monclient: _finish_hunting 0 2024-09-16T03:43:32.389 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.377+0000 7ff24ad9d640 1 monclient: found mon.noname-c 2024-09-16T03:43:32.389 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.377+0000 7ff24ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:43:32.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.377+0000 7ff24ad9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.154:3300/0 2024-09-16T03:43:32.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.377+0000 7ff24ad9d640 1 -- 172.21.15.42:0/3647060086 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff24c06dd60 con 0x7ff24c13daa0 2024-09-16T03:43:32.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.381+0000 7ff24ad9d640 1 --2- 172.21.15.42:0/3647060086 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff24c13daa0 0x7ff24c13fe90 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7ff244009d90 tx=0x7ff24401c830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7ed924d1c89de510 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:43:32.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.381+0000 7ff24959a640 1 -- 172.21.15.42:0/3647060086 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2440a6700 con 0x7ff24c13daa0 2024-09-16T03:43:32.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.381+0000 7ff24959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:43:32.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.381+0000 7ff24959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:43:32.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.381+0000 7ff24959a640 10 monclient: dump: 2024-09-16T03:43:32.397 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:43:32.397 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:43:32.397 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:43:32.397 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:43:32.397 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:43:32.397 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:43:32.397 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:43:32.397 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:43:32.397 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:43:32.397 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:43:32.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.381+0000 7ff24959a640 10 monclient: _finish_auth 0 2024-09-16T03:43:32.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.381+0000 7ff24959a640 10 monclient: _check_auth_tickets 2024-09-16T03:43:32.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.381+0000 7ff24959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:43:32.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.381+0000 7ff24959a640 1 -- 172.21.15.42:0/3647060086 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff2440a68a0 con 0x7ff24c13daa0 2024-09-16T03:43:32.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.381+0000 7ff24959a640 10 monclient: handle_config config(0 keys) 2024-09-16T03:43:32.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.381+0000 7ff24959a640 1 -- 172.21.15.42:0/3647060086 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2440a6c70 con 0x7ff24c13daa0 2024-09-16T03:43:32.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.381+0000 7ff24959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:43:32.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.381+0000 7ff24959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:43:32.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.381+0000 7ff24959a640 10 monclient: dump: 2024-09-16T03:43:32.398 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:43:32.398 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:43:32.398 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:43:32.398 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:43:32.398 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:43:32.399 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:43:32.399 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:43:32.399 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:43:32.399 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:43:32.399 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:43:32.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.381+0000 7ff250866640 5 monclient: authenticate success, global_id 14206 2024-09-16T03:43:32.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.381+0000 7ff250866640 10 monclient: get_monmap_and_config success 2024-09-16T03:43:32.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.381+0000 7ff250866640 4 set_mon_vals no callback set 2024-09-16T03:43:32.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.381+0000 7ff250866640 10 monclient: shutdown 2024-09-16T03:43:32.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.381+0000 7ff250866640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:43:32.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.381+0000 7ff250866640 1 -- 172.21.15.42:0/3647060086 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff24c13daa0 msgr2=0x7ff24c13fe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:43:32.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.381+0000 7ff250866640 1 --2- 172.21.15.42:0/3647060086 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff24c13daa0 0x7ff24c13fe90 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7ff244009d90 tx=0x7ff24401c830 comp rx=0 tx=0).stop 2024-09-16T03:43:32.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.381+0000 7ff250866640 1 -- 172.21.15.42:0/3647060086 shutdown_connections 2024-09-16T03:43:32.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.381+0000 7ff250866640 1 --2- 172.21.15.42:0/3647060086 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff24c13daa0 0x7ff24c13fe90 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:43:32.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.381+0000 7ff250866640 1 -- 172.21.15.42:0/3647060086 >> 172.21.15.42:0/3647060086 conn(0x7ff24c0a9fb0 msgr2=0x7ff24c0aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:43:32.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.381+0000 7ff250866640 1 -- 172.21.15.42:0/3647060086 shutdown_connections 2024-09-16T03:43:32.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.381+0000 7ff250866640 1 -- 172.21.15.42:0/3647060086 wait complete. 2024-09-16T03:43:32.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.381+0000 7ff250866640 10 monclient: build_initial_monmap 2024-09-16T03:43:32.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.381+0000 7ff250866640 10 monclient: monmap: 2024-09-16T03:43:32.400 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:43:32.400 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:43:32.400 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:43:32.400 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:43:32.400 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:43:32.400 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:43:32.400 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:43:32.400 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:43:32.401 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:43:32.401 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:43:32.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.385+0000 7ff250866640 1 librados: starting msgr at 2024-09-16T03:43:32.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.385+0000 7ff250866640 1 librados: starting objecter 2024-09-16T03:43:32.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.385+0000 7ff250866640 1 Processor -- start 2024-09-16T03:43:32.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.385+0000 7ff250866640 1 -- start start 2024-09-16T03:43:32.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.385+0000 7ff250866640 1 librados: setting wanted keys 2024-09-16T03:43:32.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.385+0000 7ff250866640 1 librados: calling monclient init 2024-09-16T03:43:32.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.385+0000 7ff250866640 10 monclient: init 2024-09-16T03:43:32.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.385+0000 7ff250866640 10 monclient: _reopen_session rank -1 2024-09-16T03:43:32.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.385+0000 7ff250866640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T03:43:32.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.385+0000 7ff250866640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff24c0a39b0 0x7ff24c139070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:43:32.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.385+0000 7ff250866640 10 monclient(hunting): picked mon.noname-b con 0x7ff24c0a39b0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:43:32.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.385+0000 7ff250866640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff24c0a3d90 0x7ff24c12e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:43:32.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.385+0000 7ff250866640 10 monclient(hunting): picked mon.noname-c con 0x7ff24c0a3d90 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:43:32.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.385+0000 7ff250866640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff24c13daa0 0x7ff24c12ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:43:32.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.385+0000 7ff250866640 10 monclient(hunting): picked mon.noname-a con 0x7ff24c13daa0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:43:32.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.385+0000 7ff250866640 10 monclient(hunting): start opening mon connection 2024-09-16T03:43:32.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.385+0000 7ff250866640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7ff24c142f80 con 0x7ff24c13daa0 2024-09-16T03:43:32.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.385+0000 7ff250866640 10 monclient(hunting): start opening mon connection 2024-09-16T03:43:32.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.385+0000 7ff250866640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff24c142e00 con 0x7ff24c0a39b0 2024-09-16T03:43:32.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.385+0000 7ff250866640 10 monclient(hunting): start opening mon connection 2024-09-16T03:43:32.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.385+0000 7ff250866640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7ff24c143100 con 0x7ff24c0a3d90 2024-09-16T03:43:32.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.385+0000 7ff250866640 10 monclient(hunting): _renew_subs 2024-09-16T03:43:32.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.385+0000 7ff250866640 10 monclient(hunting): authenticate will time out at 1059.356892s 2024-09-16T03:43:32.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.385+0000 7ff24ad9d640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff24c13daa0 0x7ff24c12ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:43:32.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.385+0000 7ff24ad9d640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff24c13daa0 0x7ff24c12ebe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:33622/0 (socket says 172.21.15.42:33622) 2024-09-16T03:43:32.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.385+0000 7ff24ad9d640 1 -- 172.21.15.42:0/1697848610 learned_addr learned my addr 172.21.15.42:0/1697848610 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:43:32.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.385+0000 7ff24ad9d640 10 monclient(hunting): get_auth_request con 0x7ff24c13daa0 auth_method 0 2024-09-16T03:43:32.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.385+0000 7ff24ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:43:32.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.385+0000 7ff24ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:43:32.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.385+0000 7ff24ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:43:32.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.385+0000 7ff249d9b640 1 --2- 172.21.15.42:0/1697848610 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff24c0a3d90 0x7ff24c12e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:43:32.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.385+0000 7ff249d9b640 10 monclient(hunting): get_auth_request con 0x7ff24c0a3d90 auth_method 0 2024-09-16T03:43:32.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.385+0000 7ff249d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:43:32.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.385+0000 7ff249d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:43:32.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.385+0000 7ff249d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:43:32.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.385+0000 7ff24ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:43:32.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.385+0000 7ff24ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:43:32.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.385+0000 7ff24ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:43:32.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.385+0000 7ff24a59c640 1 --2- 172.21.15.42:0/1697848610 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff24c0a39b0 0x7ff24c139070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:43:32.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.385+0000 7ff249d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:43:32.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.385+0000 7ff249d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:43:32.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.385+0000 7ff249d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:43:32.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.385+0000 7ff24a59c640 10 monclient(hunting): get_auth_request con 0x7ff24c0a39b0 auth_method 0 2024-09-16T03:43:32.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.385+0000 7ff24a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:43:32.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.385+0000 7ff24a59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:43:32.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.385+0000 7ff24a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:43:32.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.385+0000 7ff24a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:43:32.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.385+0000 7ff24a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:43:32.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.385+0000 7ff24a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:43:32.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.385+0000 7ff249d9b640 10 monclient(hunting): handle_auth_done global_id 4556 payload 824 2024-09-16T03:43:32.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.385+0000 7ff249d9b640 1 -- 172.21.15.42:0/1697848610 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff24c0a39b0 msgr2=0x7ff24c139070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:43:32.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.385+0000 7ff249d9b640 1 --2- 172.21.15.42:0/1697848610 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff24c0a39b0 0x7ff24c139070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:43:32.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.385+0000 7ff249d9b640 1 -- 172.21.15.42:0/1697848610 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff24c13daa0 msgr2=0x7ff24c12ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:43:32.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.385+0000 7ff249d9b640 1 --2- 172.21.15.42:0/1697848610 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff24c13daa0 0x7ff24c12ebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:43:32.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.385+0000 7ff249d9b640 10 monclient: _finish_hunting 0 2024-09-16T03:43:32.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.385+0000 7ff249d9b640 1 monclient: found mon.noname-c 2024-09-16T03:43:32.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.385+0000 7ff249d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:43:32.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.385+0000 7ff249d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:43:32.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.385+0000 7ff249d9b640 1 -- 172.21.15.42:0/1697848610 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff24c06f680 con 0x7ff24c0a3d90 2024-09-16T03:43:32.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.385+0000 7ff249d9b640 1 --2- 172.21.15.42:0/1697848610 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff24c0a3d90 0x7ff24c12e6a0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7ff24000c430 tx=0x7ff24000c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:43:32.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.385+0000 7ff23affd640 10 client.?.objecter ms_handle_connect 0x7ff24c0a3d90 2024-09-16T03:43:32.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.385+0000 7ff23affd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:43:32.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.385+0000 7ff23affd640 1 -- 172.21.15.42:0/1697848610 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff240017070 con 0x7ff24c0a3d90 2024-09-16T03:43:32.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.385+0000 7ff23affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:43:32.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.385+0000 7ff23affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:43:32.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.385+0000 7ff23affd640 10 monclient: dump: 2024-09-16T03:43:32.406 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:43:32.406 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:43:32.406 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:43:32.406 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:43:32.406 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:43:32.406 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:43:32.406 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:43:32.406 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:43:32.406 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:43:32.406 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:43:32.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.385+0000 7ff23affd640 10 monclient: _finish_auth 0 2024-09-16T03:43:32.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.385+0000 7ff23affd640 10 monclient: _check_auth_tickets 2024-09-16T03:43:32.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.385+0000 7ff23affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:43:32.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.385+0000 7ff23affd640 1 -- 172.21.15.42:0/1697848610 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff240005cf0 con 0x7ff24c0a3d90 2024-09-16T03:43:32.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.385+0000 7ff23affd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:43:32.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.385+0000 7ff23affd640 1 -- 172.21.15.42:0/1697848610 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2400072d0 con 0x7ff24c0a3d90 2024-09-16T03:43:32.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.385+0000 7ff23affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:43:32.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.385+0000 7ff23affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:43:32.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.385+0000 7ff23affd640 10 monclient: dump: 2024-09-16T03:43:32.407 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:43:32.407 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:43:32.407 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:43:32.407 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:43:32.407 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:43:32.407 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:43:32.407 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:43:32.408 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:43:32.408 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:43:32.408 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:43:32.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.385+0000 7ff248d99640 4 set_mon_vals no callback set 2024-09-16T03:43:32.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.385+0000 7ff250866640 5 monclient: authenticate success, global_id 4556 2024-09-16T03:43:32.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.385+0000 7ff250866640 10 monclient: _renew_subs 2024-09-16T03:43:32.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.385+0000 7ff250866640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:43:32.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.385+0000 7ff250866640 1 -- 172.21.15.42:0/1697848610 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff24c134bd0 con 0x7ff24c0a3d90 2024-09-16T03:43:32.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.385+0000 7ff250866640 10 client.4556.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:43:32.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.385+0000 7ff250866640 10 monclient: _renew_subs 2024-09-16T03:43:32.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.385+0000 7ff250866640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:43:32.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.385+0000 7ff250866640 1 -- 172.21.15.42:0/1697848610 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff24c130390 con 0x7ff24c0a3d90 2024-09-16T03:43:32.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.385+0000 7ff250866640 1 librados: init done 2024-09-16T03:43:32.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.385+0000 7ff250866640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:43:32.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.385+0000 7ff250866640 1 --2- 172.21.15.42:0/1697848610 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff24c12f120 0x7ff24c12b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:43:32.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.385+0000 7ff250866640 10 monclient(hunting): start opening mon connection 2024-09-16T03:43:32.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.385+0000 7ff250866640 1 -- 172.21.15.42:0/1697848610 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff24c12f500 con 0x7ff24c12f120 2024-09-16T03:43:32.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.385+0000 7ff24a59c640 1 --2- 172.21.15.42:0/1697848610 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff24c12f120 0x7ff24c12b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:43:32.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.389+0000 7ff24a59c640 10 monclient: get_auth_request con 0x7ff24c12f120 auth_method 0 2024-09-16T03:43:32.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.389+0000 7ff24a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:43:32.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.389+0000 7ff24a59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:43:32.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.389+0000 7ff24a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:43:32.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.389+0000 7ff23affd640 1 -- 172.21.15.42:0/1697848610 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7ff24001a070 con 0x7ff24c0a3d90 2024-09-16T03:43:32.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.389+0000 7ff24a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:43:32.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.389+0000 7ff24a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:43:32.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.389+0000 7ff24a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:43:32.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.389+0000 7ff23affd640 1 --2- 172.21.15.42:0/1697848610 >> v2:172.21.15.154:6812/254845342 conn(0x7ff22c037810 0x7ff22c039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:43:32.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.389+0000 7ff23affd640 1 -- 172.21.15.42:0/1697848610 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(20..20 src has 1..20) ==== 13897+0+0 (secure 0 0 0) 0x7ff24004a2f0 con 0x7ff24c0a3d90 2024-09-16T03:43:32.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.389+0000 7ff23affd640 10 client.4556.objecter ms_dispatch 0x7ff24c000ba0 osd_map(20..20 src has 1..20) 2024-09-16T03:43:32.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.389+0000 7ff23affd640 3 client.4556.objecter handle_osd_map got epochs [20,20] > 0 2024-09-16T03:43:32.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.389+0000 7ff23affd640 3 client.4556.objecter handle_osd_map decoding full epoch 20 2024-09-16T03:43:32.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.389+0000 7ff23affd640 20 client.4556.objecter dump_active .. 0 homeless 2024-09-16T03:43:32.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.389+0000 7ff24a59c640 10 monclient(hunting): handle_auth_done global_id 14215 payload 824 2024-09-16T03:43:32.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.389+0000 7ff24a59c640 1 -- 172.21.15.42:0/1697848610 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff24c133b80 con 0x7ff24c12f120 2024-09-16T03:43:32.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.389+0000 7ff24ad9d640 1 --2- 172.21.15.42:0/1697848610 >> v2:172.21.15.154:6812/254845342 conn(0x7ff22c037810 0x7ff22c039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:43:32.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.389+0000 7ff24ad9d640 10 monclient: get_auth_request con 0x7ff22c037810 auth_method 0 2024-09-16T03:43:32.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.389+0000 7ff23affd640 10 client.4556.objecter ms_handle_connect 0x7ff24c12f120 2024-09-16T03:43:32.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.389+0000 7ff24a59c640 1 --2- 172.21.15.42:0/1697848610 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff24c12f120 0x7ff24c12b2b0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7ff23c00f4a0 tx=0x7ff23c00f960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:43:32.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.389+0000 7ff23affd640 10 client.4556.objecter resend_mon_ops 2024-09-16T03:43:32.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.389+0000 7ff23affd640 1 -- 172.21.15.42:0/1697848610 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff23c019070 con 0x7ff24c12f120 2024-09-16T03:43:32.411 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.389+0000 7ff23affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:43:32.411 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.389+0000 7ff23affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:43:32.411 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.389+0000 7ff23affd640 10 monclient: dump: 2024-09-16T03:43:32.411 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:43:32.411 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:43:32.411 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:43:32.411 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:43:32.411 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:43:32.411 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:43:32.411 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:43:32.411 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:43:32.411 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:43:32.411 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:43:32.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.389+0000 7ff23affd640 10 client.4556.objecter ms_handle_connect 0x7ff22c037810 2024-09-16T03:43:32.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.389+0000 7ff24ad9d640 1 --2- 172.21.15.42:0/1697848610 >> v2:172.21.15.154:6812/254845342 conn(0x7ff22c037810 0x7ff22c039cd0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7ff244019440 tx=0x7ff2440a7000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:43:32.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.389+0000 7ff23affd640 1 -- 172.21.15.42:0/1697848610 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7ff23c005320 con 0x7ff24c12f120 2024-09-16T03:43:32.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.389+0000 7ff23affd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:43:32.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.389+0000 7ff23affd640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:43:32.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.389+0000 7ff23affd640 1 -- 172.21.15.42:0/1697848610 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff24c12f120 msgr2=0x7ff24c12b2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:43:32.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.389+0000 7ff23affd640 1 --2- 172.21.15.42:0/1697848610 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff24c12f120 0x7ff24c12b2b0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7ff23c00f4a0 tx=0x7ff23c00f960 comp rx=0 tx=0).stop 2024-09-16T03:43:32.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.389+0000 7ff23affd640 1 -- 172.21.15.42:0/1697848610 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff24c12f120 msgr2=0x7ff24c12b2b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:43:32.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.389+0000 7ff23affd640 1 --2- 172.21.15.42:0/1697848610 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff24c12f120 0x7ff24c12b2b0 secure :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7ff23c00f4a0 tx=0x7ff23c00f960 comp rx=0 tx=0).stop 2024-09-16T03:43:32.414 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.397+0000 7ff250866640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:43:32.414 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.397+0000 7ff250866640 1 --2- 172.21.15.42:0/1697848610 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff24c12b7f0 0x7ff24c12bc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:43:32.414 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.397+0000 7ff250866640 10 monclient(hunting): start opening mon connection 2024-09-16T03:43:32.414 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.397+0000 7ff250866640 1 -- 172.21.15.42:0/1697848610 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff24c12c140 con 0x7ff24c12b7f0 2024-09-16T03:43:32.414 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.397+0000 7ff24a59c640 1 --2- 172.21.15.42:0/1697848610 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff24c12b7f0 0x7ff24c12bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:43:32.414 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.397+0000 7ff24a59c640 10 monclient: get_auth_request con 0x7ff24c12b7f0 auth_method 0 2024-09-16T03:43:32.414 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.397+0000 7ff24a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:43:32.414 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.397+0000 7ff24a59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:43:32.414 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.397+0000 7ff24a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:43:32.415 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.397+0000 7ff24a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:43:32.415 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.397+0000 7ff24a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:43:32.415 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.397+0000 7ff24a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:43:32.415 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.397+0000 7ff24a59c640 10 monclient(hunting): handle_auth_done global_id 14218 payload 824 2024-09-16T03:43:32.415 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.397+0000 7ff24a59c640 1 -- 172.21.15.42:0/1697848610 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ff24c13d880 con 0x7ff24c12b7f0 2024-09-16T03:43:32.415 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.397+0000 7ff23affd640 10 client.4556.objecter ms_handle_connect 0x7ff24c12b7f0 2024-09-16T03:43:32.415 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.397+0000 7ff23affd640 10 client.4556.objecter resend_mon_ops 2024-09-16T03:43:32.415 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.397+0000 7ff24a59c640 1 --2- 172.21.15.42:0/1697848610 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff24c12b7f0 0x7ff24c12bc00 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7ff23c00f4a0 tx=0x7ff23c007f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:43:32.415 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.397+0000 7ff23affd640 1 -- 172.21.15.42:0/1697848610 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff23c019070 con 0x7ff24c12b7f0 2024-09-16T03:43:32.415 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.397+0000 7ff23affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:43:32.415 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.397+0000 7ff23affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:43:32.415 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.397+0000 7ff23affd640 10 monclient: dump: 2024-09-16T03:43:32.415 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:43:32.415 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:43:32.416 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:43:32.416 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:43:32.416 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:43:32.416 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:43:32.416 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:43:32.416 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:43:32.416 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:43:32.416 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:43:32.416 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.397+0000 7ff23affd640 1 -- 172.21.15.42:0/1697848610 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4545 (secure 0 0 0) 0x7ff23c0073a0 con 0x7ff24c12b7f0 2024-09-16T03:43:32.416 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.397+0000 7ff23affd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:43:32.416 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.397+0000 7ff23affd640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:43:32.416 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.397+0000 7ff23affd640 1 -- 172.21.15.42:0/1697848610 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff24c12b7f0 msgr2=0x7ff24c12bc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:43:32.416 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.397+0000 7ff23affd640 1 --2- 172.21.15.42:0/1697848610 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff24c12b7f0 0x7ff24c12bc00 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7ff23c00f4a0 tx=0x7ff23c007f90 comp rx=0 tx=0).stop 2024-09-16T03:43:32.417 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.397+0000 7ff23affd640 1 -- 172.21.15.42:0/1697848610 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff24c12b7f0 msgr2=0x7ff24c12bc00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:43:32.417 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.397+0000 7ff23affd640 1 --2- 172.21.15.42:0/1697848610 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff24c12b7f0 0x7ff24c12bc00 secure :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7ff23c00f4a0 tx=0x7ff23c007f90 comp rx=0 tx=0).stop 2024-09-16T03:43:32.417 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:43:32.417 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:43:32.417 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:43:32.417 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T03:43:32.417 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 16, 2024-09-16T03:43:32.417 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 16202, 2024-09-16T03:43:32.417 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:43:32.418 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:43:32.418 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T03:43:32.418 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T03:43:32.418 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:43:32.418 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 1, 2024-09-16T03:43:32.418 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:43:32.418 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:43:32.418 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:43:32.418 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:43:32.418 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:43:32.418 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:43:32.418 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:43:32.418 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:43:32.418 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:43:32.419 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:43:32.419 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:43:32.419 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:43:32.419 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:43:32.419 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:43:32.419 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:43:32.419 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:43:32.419 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:43:32.419 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:43:32.419 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:43:32.419 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:43:32.419 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:43:32.419 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:43:32.420 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:43:32.420 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:43:32.420 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:43:32.420 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:43:32.420 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:43:32.420 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:43:32.420 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:43:32.420 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:43:32.420 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:43:32.420 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:43:32.420 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:43:32.420 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:43:32.420 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:43:32.420 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:43:32.421 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:43:32.421 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:43:32.421 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:43:32.421 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:43:32.421 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:43:32.421 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:43:32.421 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:43:32.421 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:43:32.421 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:43:32.421 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:43:32.421 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:43:32.421 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:43:32.421 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:43:32.422 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:43:32.422 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:43:32.422 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:43:32.422 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:43:32.422 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:43:32.422 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:43:32.422 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:43:32.422 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:43:32.422 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:43:32.422 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:43:32.422 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:43:32.422 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:43:32.422 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:43:32.422 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:43:32.423 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:43:32.423 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:43:32.423 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:43:32.423 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:43:32.423 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:43:32.423 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:43:32.423 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:43:32.423 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:43:32.423 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:43:32.423 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:43:32.423 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:43:32.423 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:43:32.423 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:43:32.423 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:43:32.424 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:43:32.424 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:43:32.424 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:43:32.424 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:43:32.424 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:43:32.424 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:43:32.424 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:43:32.424 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:43:32.424 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:43:32.424 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:43:32.424 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:43:32.424 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:43:32.424 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:43:32.425 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:43:32.425 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:43:32.425 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:43:32.425 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:43:32.426 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:43:32.426 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:43:32.426 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:43:32.426 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:43:32.426 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:43:32.426 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:43:32.426 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:43:32.426 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:43:32.426 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:43:32.426 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:43:32.427 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:43:32.427 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:43:32.427 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:43:32.427 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:43:32.427 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:43:32.427 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:43:32.427 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:43:32.427 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:43:32.427 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:43:32.427 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:43:32.427 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:43:32.427 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:43:32.427 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:43:32.428 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:43:32.428 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:43:32.428 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:43:32.428 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:43:32.428 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:43:32.428 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:43:32.428 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:43:32.428 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:43:32.428 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:43:32.428 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:43:32.428 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:43:32.428 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:43:32.428 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:43:32.428 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:43:32.429 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:43:32.429 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:43:32.429 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:43:32.429 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:43:32.429 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:43:32.429 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:43:32.429 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:43:32.429 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:43:32.429 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:43:32.429 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:43:32.429 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:43:32.429 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:43:32.429 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:43:32.430 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:43:32.430 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 13 2024-09-16T03:43:32.430 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:43:32.430 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:43:32.430 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:43:32.430 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:43:32.430 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:43:32.432 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.397+0000 7ff250866640 10 librados: watch_flush enter 2024-09-16T03:43:32.432 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.397+0000 7ff250866640 10 librados: watch_flush exit 2024-09-16T03:43:32.432 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.397+0000 7ff250866640 20 client.4556.objecter shutdown clearing up homeless session... 2024-09-16T03:43:32.432 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.397+0000 7ff250866640 10 client.4556.objecter successfully canceled tick 2024-09-16T03:43:32.432 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.397+0000 7ff250866640 1 -- 172.21.15.42:0/1697848610 >> v2:172.21.15.154:6812/254845342 conn(0x7ff22c037810 msgr2=0x7ff22c039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:43:32.432 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.397+0000 7ff250866640 1 --2- 172.21.15.42:0/1697848610 >> v2:172.21.15.154:6812/254845342 conn(0x7ff22c037810 0x7ff22c039cd0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7ff244019440 tx=0x7ff2440a7000 comp rx=0 tx=0).stop 2024-09-16T03:43:32.432 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.397+0000 7ff250866640 10 monclient: shutdown 2024-09-16T03:43:32.432 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.397+0000 7ff250866640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:43:32.432 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.397+0000 7ff250866640 1 -- 172.21.15.42:0/1697848610 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff24c0a3d90 msgr2=0x7ff24c12e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:43:32.432 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.397+0000 7ff250866640 1 --2- 172.21.15.42:0/1697848610 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff24c0a3d90 0x7ff24c12e6a0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7ff24000c430 tx=0x7ff24000c8f0 comp rx=0 tx=0).stop 2024-09-16T03:43:32.432 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.397+0000 7ff24ad9d640 1 -- 172.21.15.42:0/1697848610 reap_dead start 2024-09-16T03:43:32.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.397+0000 7ff24ad9d640 1 -- 172.21.15.42:0/1697848610 reap_dead start 2024-09-16T03:43:32.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.397+0000 7ff250866640 1 -- 172.21.15.42:0/1697848610 shutdown_connections 2024-09-16T03:43:32.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.397+0000 7ff250866640 1 -- 172.21.15.42:0/1697848610 >> 172.21.15.42:0/1697848610 conn(0x7ff24c0a9fb0 msgr2=0x7ff24c131980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:43:32.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.397+0000 7ff250866640 1 -- 172.21.15.42:0/1697848610 shutdown_connections 2024-09-16T03:43:32.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.397+0000 7ff250866640 1 -- 172.21.15.42:0/1697848610 wait complete. 2024-09-16T03:43:32.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.397+0000 7ff250866640 1 librados: shutdown 2024-09-16T03:43:32.433 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:43:32.433 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T03:43:32.433 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T03:43:32.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.561+0000 7f4453c4d640 10 monclient: get_monmap_and_config 2024-09-16T03:43:32.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.561+0000 7f4453c4d640 10 monclient: build_initial_monmap 2024-09-16T03:43:32.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.561+0000 7f4453c4d640 10 monclient: monmap: 2024-09-16T03:43:32.567 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:43:32.568 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:43:32.568 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:43:32.565950+0000 2024-09-16T03:43:32.568 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:43:32.565950+0000 2024-09-16T03:43:32.568 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:43:32.568 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:43:32.568 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:43:32.568 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:43:32.568 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:43:32.568 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:43:32.568 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:43:32.568 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:43:32.570 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.561+0000 7f4453c4d640 1 Processor -- start 2024-09-16T03:43:32.570 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.561+0000 7f4453c4d640 1 -- start start 2024-09-16T03:43:32.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.561+0000 7f4453c4d640 10 monclient: init 2024-09-16T03:43:32.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.561+0000 7f4453c4d640 10 monclient: _reopen_session rank -1 2024-09-16T03:43:32.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.561+0000 7f4453c4d640 10 monclient: _add_conns ranks=[2,0,3,1,4] 2024-09-16T03:43:32.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.561+0000 7f4453c4d640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f444c0a67e0 0x7f444c0a6bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:43:32.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.561+0000 7f4453c4d640 10 monclient(hunting): picked mon.noname-b con 0x7f444c0a67e0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:43:32.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.561+0000 7f4453c4d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f444c072560 addr v1:172.21.15.42:6789/0 2024-09-16T03:43:32.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.561+0000 7f4453c4d640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f444c0a7100 0x7f444c1357a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:43:32.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.561+0000 7f4453c4d640 10 monclient(hunting): picked mon.noname-a con 0x7f444c0a7100 addr v2:172.21.15.42:3300/0 2024-09-16T03:43:32.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.561+0000 7f4453c4d640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f444c071b80 con 0x7f444c072560 2024-09-16T03:43:32.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.561+0000 7f4453c4d640 10 monclient(hunting): start opening mon connection 2024-09-16T03:43:32.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.561+0000 7f4453c4d640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f444c072ed0 con 0x7f444c0a7100 2024-09-16T03:43:32.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.561+0000 7f4453c4d640 10 monclient(hunting): start opening mon connection 2024-09-16T03:43:32.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.561+0000 7f4453c4d640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f444c070030 con 0x7f444c0a67e0 2024-09-16T03:43:32.572 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.561+0000 7f4453c4d640 10 monclient(hunting): _renew_subs 2024-09-16T03:43:32.572 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.561+0000 7f4453c4d640 10 monclient(hunting): authenticate will time out at 1059.535298s 2024-09-16T03:43:32.572 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.561+0000 7f44519e9640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f444c0a67e0 0x7f444c0a6bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:43:32.572 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.561+0000 7f44511e8640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f444c072560 0x7f444c135090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:33052/0 (socket says 172.21.15.42:33052) 2024-09-16T03:43:32.572 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.561+0000 7f44521ea640 1 --2- 172.21.15.42:0/872165802 >> v2:172.21.15.42:3300/0 conn(0x7f444c0a7100 0x7f444c1357a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:43:32.572 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.561+0000 7f44511e8640 1 -- 172.21.15.42:0/872165802 learned_addr learned my addr 172.21.15.42:0/872165802 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:43:32.572 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.561+0000 7f44519e9640 10 monclient(hunting): get_auth_request con 0x7f444c0a67e0 auth_method 0 2024-09-16T03:43:32.572 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.561+0000 7f44519e9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:43:32.572 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.561+0000 7f44519e9640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:43:32.572 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.561+0000 7f44519e9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:43:32.572 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.561+0000 7f44521ea640 10 monclient(hunting): get_auth_request con 0x7f444c0a7100 auth_method 0 2024-09-16T03:43:32.572 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.561+0000 7f44521ea640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:43:32.572 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.561+0000 7f44521ea640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:43:32.573 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.561+0000 7f44521ea640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:43:32.573 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.561+0000 7f44521ea640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:43:32.573 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.561+0000 7f44521ea640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:43:32.573 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.561+0000 7f44521ea640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:43:32.573 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.561+0000 7f44519e9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:43:32.573 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.561+0000 7f44519e9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:43:32.573 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.561+0000 7f44519e9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:43:32.573 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.561+0000 7f44509e7640 1 -- 172.21.15.42:0/872165802 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3283359589 0 0) 0x7f444c071b80 con 0x7f444c072560 2024-09-16T03:43:32.573 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.561+0000 7f44509e7640 10 monclient(hunting): _negotiate 2024-09-16T03:43:32.573 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.561+0000 7f44509e7640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:43:32.573 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.561+0000 7f44509e7640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:43:32.573 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.561+0000 7f44509e7640 10 monclient(hunting): my global_id is 4707 2024-09-16T03:43:32.573 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.561+0000 7f44509e7640 1 -- 172.21.15.42:0/872165802 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4438003530 con 0x7f444c072560 2024-09-16T03:43:32.573 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.561+0000 7f44521ea640 10 monclient(hunting): handle_auth_done global_id 4704 payload 274 2024-09-16T03:43:32.573 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.561+0000 7f44521ea640 1 -- 172.21.15.42:0/872165802 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f444c0a67e0 msgr2=0x7f444c0a6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:43:32.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.561+0000 7f44521ea640 1 --2- 172.21.15.42:0/872165802 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f444c0a67e0 0x7f444c0a6bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:43:32.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.561+0000 7f44521ea640 1 -- 172.21.15.42:0/872165802 >> v1:172.21.15.42:6789/0 conn(0x7f444c072560 legacy=0x7f444c135090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:43:32.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.561+0000 7f44521ea640 10 monclient: _finish_hunting 0 2024-09-16T03:43:32.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.561+0000 7f44521ea640 1 monclient: found mon.noname-a 2024-09-16T03:43:32.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.561+0000 7f44521ea640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:43:32.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.561+0000 7f44521ea640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:43:32.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.561+0000 7f44521ea640 1 -- 172.21.15.42:0/872165802 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f444c0712e0 con 0x7f444c0a7100 2024-09-16T03:43:32.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.565+0000 7f44521ea640 1 --2- 172.21.15.42:0/872165802 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f444c0a7100 0x7f444c1357a0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f4444009d90 tx=0x7f444401c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86dd726a4d91a685 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:43:32.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.565+0000 7f44509e7640 1 -- 172.21.15.42:0/872165802 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44440b2070 con 0x7f444c0a7100 2024-09-16T03:43:32.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.565+0000 7f44509e7640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:43:32.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.565+0000 7f44509e7640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:43:32.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.565+0000 7f44509e7640 10 monclient: dump: 2024-09-16T03:43:32.580 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:43:32.580 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:43:32.580 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:43:32.580 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:43:32.580 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:43:32.580 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:43:32.580 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:43:32.580 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:43:32.580 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:43:32.580 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:43:32.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.565+0000 7f44509e7640 10 monclient: _finish_auth 0 2024-09-16T03:43:32.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.565+0000 7f44509e7640 10 monclient: _check_auth_tickets 2024-09-16T03:43:32.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.565+0000 7f44509e7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:43:32.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.565+0000 7f44509e7640 1 -- 172.21.15.42:0/872165802 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f44440a58e0 con 0x7f444c0a7100 2024-09-16T03:43:32.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.565+0000 7f4453c4d640 5 monclient: authenticate success, global_id 4704 2024-09-16T03:43:32.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.565+0000 7f4453c4d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:43:32.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.565+0000 7f44509e7640 10 monclient: handle_config config(0 keys) 2024-09-16T03:43:32.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.565+0000 7f44509e7640 1 -- 172.21.15.42:0/872165802 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44440a5cb0 con 0x7f444c0a7100 2024-09-16T03:43:32.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.565+0000 7f4453c4d640 10 monclient: get_monmap_and_config success 2024-09-16T03:43:32.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.565+0000 7f4453c4d640 4 set_mon_vals no callback set 2024-09-16T03:43:32.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.565+0000 7f44509e7640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:43:32.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.565+0000 7f44509e7640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:43:32.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.565+0000 7f44509e7640 10 monclient: dump: 2024-09-16T03:43:32.581 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:43:32.581 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:43:32.582 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:43:32.582 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:43:32.582 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:43:32.582 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:43:32.582 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:43:32.582 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:43:32.582 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:43:32.582 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:43:32.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.569+0000 7f4453c4d640 10 monclient: shutdown 2024-09-16T03:43:32.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.569+0000 7f4453c4d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:43:32.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.569+0000 7f4453c4d640 1 -- 172.21.15.42:0/872165802 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f444c0a7100 msgr2=0x7f444c1357a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:43:32.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.569+0000 7f4453c4d640 1 --2- 172.21.15.42:0/872165802 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f444c0a7100 0x7f444c1357a0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f4444009d90 tx=0x7f444401c800 comp rx=0 tx=0).stop 2024-09-16T03:43:32.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.569+0000 7f4453c4d640 1 -- 172.21.15.42:0/872165802 shutdown_connections 2024-09-16T03:43:32.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.569+0000 7f4453c4d640 1 --2- 172.21.15.42:0/872165802 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f444c0a7100 0x7f444c1357a0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:43:32.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.569+0000 7f4453c4d640 1 --2- 172.21.15.42:0/872165802 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f444c0a67e0 0x7f444c0a6bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:43:32.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.569+0000 7f4453c4d640 1 -- 172.21.15.42:0/872165802 >> 172.21.15.42:0/872165802 conn(0x7f444c0a7ed0 msgr2=0x7f444c0a43d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:43:32.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.569+0000 7f4453c4d640 1 -- 172.21.15.42:0/872165802 shutdown_connections 2024-09-16T03:43:32.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.569+0000 7f4453c4d640 1 -- 172.21.15.42:0/872165802 wait complete. 2024-09-16T03:43:32.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.569+0000 7f4453c4d640 10 monclient: build_initial_monmap 2024-09-16T03:43:32.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.569+0000 7f4453c4d640 10 monclient: monmap: 2024-09-16T03:43:32.584 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:43:32.584 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:43:32.584 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:43:32.584 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:43:32.584 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:43:32.584 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:43:32.584 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:43:32.584 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:43:32.584 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:43:32.584 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:43:32.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.569+0000 7f4453c4d640 1 librados: starting msgr at 2024-09-16T03:43:32.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.569+0000 7f4453c4d640 1 librados: starting objecter 2024-09-16T03:43:32.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.569+0000 7f4453c4d640 1 Processor -- start 2024-09-16T03:43:32.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.569+0000 7f4453c4d640 1 -- start start 2024-09-16T03:43:32.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.569+0000 7f4453c4d640 1 librados: setting wanted keys 2024-09-16T03:43:32.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.569+0000 7f4453c4d640 1 librados: calling monclient init 2024-09-16T03:43:32.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.569+0000 7f4453c4d640 10 monclient: init 2024-09-16T03:43:32.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.569+0000 7f4453c4d640 10 monclient: _reopen_session rank -1 2024-09-16T03:43:32.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.569+0000 7f4453c4d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:43:32.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.569+0000 7f4453c4d640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f444c072560 0x7f444c134740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:43:32.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.569+0000 7f4453c4d640 10 monclient(hunting): picked mon.noname-c con 0x7f444c072560 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:43:32.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.569+0000 7f4453c4d640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f444c0a67e0 0x7f444c134c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:43:32.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.569+0000 7f4453c4d640 10 monclient(hunting): picked mon.noname-a con 0x7f444c0a67e0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:43:32.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.569+0000 7f4453c4d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f444c0a7100 0x7f444c13aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:43:32.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.569+0000 7f4453c4d640 10 monclient(hunting): picked mon.noname-b con 0x7f444c0a7100 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:43:32.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.569+0000 7f4453c4d640 10 monclient(hunting): start opening mon connection 2024-09-16T03:43:32.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.569+0000 7f4453c4d640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f444c0a4ec0 con 0x7f444c0a67e0 2024-09-16T03:43:32.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.569+0000 7f4453c4d640 10 monclient(hunting): start opening mon connection 2024-09-16T03:43:32.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.569+0000 7f4453c4d640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f444c0a4d40 con 0x7f444c0a7100 2024-09-16T03:43:32.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.569+0000 7f4453c4d640 10 monclient(hunting): start opening mon connection 2024-09-16T03:43:32.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.569+0000 7f4453c4d640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f444c135ce0 con 0x7f444c072560 2024-09-16T03:43:32.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.569+0000 7f4453c4d640 10 monclient(hunting): _renew_subs 2024-09-16T03:43:32.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.569+0000 7f4453c4d640 10 monclient(hunting): authenticate will time out at 1059.542189s 2024-09-16T03:43:32.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.569+0000 7f44519e9640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f444c072560 0x7f444c134740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:43:32.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.569+0000 7f44511e8640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f444c0a67e0 0x7f444c134c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:43:32.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.569+0000 7f44519e9640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f444c072560 0x7f444c134740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:37720/0 (socket says 172.21.15.42:37720) 2024-09-16T03:43:32.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.569+0000 7f44519e9640 1 -- 172.21.15.42:0/106671836 learned_addr learned my addr 172.21.15.42:0/106671836 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:43:32.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.569+0000 7f44519e9640 10 monclient(hunting): get_auth_request con 0x7f444c072560 auth_method 0 2024-09-16T03:43:32.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.569+0000 7f44519e9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:43:32.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.569+0000 7f44519e9640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:43:32.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.569+0000 7f44519e9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:43:32.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.569+0000 7f44511e8640 10 monclient(hunting): get_auth_request con 0x7f444c0a67e0 auth_method 0 2024-09-16T03:43:32.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.569+0000 7f44511e8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:43:32.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.569+0000 7f44511e8640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:43:32.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.569+0000 7f44511e8640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:43:32.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.569+0000 7f44511e8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:43:32.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.569+0000 7f44511e8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:43:32.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.569+0000 7f44511e8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:43:32.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.569+0000 7f44519e9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:43:32.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.569+0000 7f44519e9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:43:32.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.569+0000 7f44519e9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:43:32.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.569+0000 7f44511e8640 10 monclient(hunting): handle_auth_done global_id 4710 payload 824 2024-09-16T03:43:32.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.569+0000 7f44511e8640 1 -- 172.21.15.42:0/106671836 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f444c072560 msgr2=0x7f444c134740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:43:32.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.569+0000 7f44511e8640 1 --2- 172.21.15.42:0/106671836 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f444c072560 0x7f444c134740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:43:32.588 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.569+0000 7f44511e8640 1 -- 172.21.15.42:0/106671836 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f444c0a7100 msgr2=0x7f444c13aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:43:32.588 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.569+0000 7f44511e8640 1 --2- 172.21.15.42:0/106671836 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f444c0a7100 0x7f444c13aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:43:32.588 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.569+0000 7f44511e8640 10 monclient: _finish_hunting 0 2024-09-16T03:43:32.588 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.569+0000 7f44511e8640 1 monclient: found mon.noname-a 2024-09-16T03:43:32.588 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.569+0000 7f44511e8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:43:32.588 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.569+0000 7f44511e8640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:43:32.588 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.569+0000 7f44511e8640 1 -- 172.21.15.42:0/106671836 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f444c11f040 con 0x7f444c0a67e0 2024-09-16T03:43:32.588 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.569+0000 7f44511e8640 1 --2- 172.21.15.42:0/106671836 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f444c0a67e0 0x7f444c134c80 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f443c00efd0 tx=0x7f443c001a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:43:32.588 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.569+0000 7f44427fc640 10 client.?.objecter ms_handle_connect 0x7f444c0a67e0 2024-09-16T03:43:32.588 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.569+0000 7f44427fc640 10 client.?.objecter resend_mon_ops 2024-09-16T03:43:32.588 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.569+0000 7f44427fc640 1 -- 172.21.15.42:0/106671836 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f443c00e070 con 0x7f444c0a67e0 2024-09-16T03:43:32.588 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.569+0000 7f44427fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:43:32.588 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.569+0000 7f44427fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:43:32.589 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.569+0000 7f44427fc640 10 monclient: dump: 2024-09-16T03:43:32.589 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:43:32.589 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:43:32.589 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:43:32.589 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:43:32.589 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:43:32.589 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:43:32.589 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:43:32.589 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:43:32.589 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:43:32.589 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:43:32.589 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.569+0000 7f44427fc640 10 monclient: _finish_auth 0 2024-09-16T03:43:32.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.569+0000 7f44427fc640 10 monclient: _check_auth_tickets 2024-09-16T03:43:32.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.569+0000 7f44427fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:43:32.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.569+0000 7f4453c4d640 5 monclient: authenticate success, global_id 4710 2024-09-16T03:43:32.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.569+0000 7f44427fc640 1 -- 172.21.15.42:0/106671836 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f443c00e210 con 0x7f444c0a67e0 2024-09-16T03:43:32.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.569+0000 7f4453c4d640 10 monclient: _renew_subs 2024-09-16T03:43:32.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.569+0000 7f4453c4d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:43:32.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.569+0000 7f4453c4d640 1 -- 172.21.15.42:0/106671836 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f444c0a0c10 con 0x7f444c0a67e0 2024-09-16T03:43:32.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.569+0000 7f44427fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:43:32.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.569+0000 7f44427fc640 1 -- 172.21.15.42:0/106671836 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f443c005b00 con 0x7f444c0a67e0 2024-09-16T03:43:32.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.569+0000 7f44427fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:43:32.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.569+0000 7f44427fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:43:32.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.569+0000 7f44427fc640 10 monclient: dump: 2024-09-16T03:43:32.593 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:43:32.593 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:43:32.594 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:43:32.594 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:43:32.594 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:43:32.594 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:43:32.594 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:43:32.594 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:43:32.594 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:43:32.594 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:43:32.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.569+0000 7f4453c4d640 10 client.4710.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:43:32.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.569+0000 7f4453c4d640 10 monclient: _renew_subs 2024-09-16T03:43:32.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.569+0000 7f4453c4d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:43:32.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.569+0000 7f4453c4d640 1 -- 172.21.15.42:0/106671836 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f444c13cb30 con 0x7f444c0a67e0 2024-09-16T03:43:32.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.569+0000 7f4453c4d640 1 librados: init done 2024-09-16T03:43:32.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.569+0000 7f4443fff640 4 set_mon_vals no callback set 2024-09-16T03:43:32.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.569+0000 7f44427fc640 1 -- 172.21.15.42:0/106671836 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f443c005ca0 con 0x7f444c0a67e0 2024-09-16T03:43:32.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.573+0000 7f4453c4d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:43:32.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.573+0000 7f4453c4d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:43:32.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.573+0000 7f4453c4d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:43:32.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.573+0000 7f4453c4d640 1 -- 172.21.15.42:0/106671836 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4434005090 con 0x7f444c0a67e0 2024-09-16T03:43:32.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.573+0000 7f44427fc640 1 --2- 172.21.15.42:0/106671836 >> v2:172.21.15.154:6812/254845342 conn(0x7f4430037840 0x7f4430039d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:43:32.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.573+0000 7f44427fc640 1 -- 172.21.15.42:0/106671836 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 13897+0+0 (secure 0 0 0) 0x7f443c04e990 con 0x7f444c0a67e0 2024-09-16T03:43:32.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.573+0000 7f44427fc640 10 client.4710.objecter ms_dispatch 0x7f444c000ba0 osd_map(20..20 src has 1..20) 2024-09-16T03:43:32.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.573+0000 7f44427fc640 3 client.4710.objecter handle_osd_map got epochs [20,20] > 0 2024-09-16T03:43:32.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.573+0000 7f44427fc640 3 client.4710.objecter handle_osd_map decoding full epoch 20 2024-09-16T03:43:32.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.573+0000 7f44427fc640 20 client.4710.objecter dump_active .. 0 homeless 2024-09-16T03:43:32.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.573+0000 7f44519e9640 1 --2- 172.21.15.42:0/106671836 >> v2:172.21.15.154:6812/254845342 conn(0x7f4430037840 0x7f4430039d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:43:32.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.573+0000 7f44519e9640 10 monclient: get_auth_request con 0x7f4430037840 auth_method 0 2024-09-16T03:43:32.596 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.573+0000 7f44519e9640 1 --2- 172.21.15.42:0/106671836 >> v2:172.21.15.154:6812/254845342 conn(0x7f4430037840 0x7f4430039d00 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f4448009c20 tx=0x7f4448009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:43:32.596 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.573+0000 7f44427fc640 10 client.4710.objecter ms_handle_connect 0x7f4430037840 2024-09-16T03:43:32.596 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.577+0000 7f44427fc640 1 -- 172.21.15.42:0/106671836 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f443c04b000 con 0x7f444c0a67e0 2024-09-16T03:43:32.596 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.577+0000 7f44427fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:43:32.596 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.577+0000 7f44427fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:43:32.813 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.805+0000 7f4453c4d640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T03:43:32.813 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.805+0000 7f4453c4d640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T03:43:32.813 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.805+0000 7f4453c4d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:43:32.813 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.805+0000 7f4453c4d640 1 -- 172.21.15.42:0/106671836 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f4434005650 con 0x7f444c0a67e0 2024-09-16T03:43:32.813 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.809+0000 7f44427fc640 1 -- 172.21.15.42:0/106671836 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f443c021a60 con 0x7f444c0a67e0 2024-09-16T03:43:32.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.809+0000 7f44427fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T03:43:32.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.809+0000 7f44427fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:43:32.815 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:43:32.809+0000 7f89824fe640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:43:32.815 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:43:32.809+0000 7f89824fe640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=32,config=2,health=11,kv=10,logm=45} crc {auth=3840062311,config=692619927,health=632208798,kv=871254744,logm=1453121405}) 2024-09-16T03:43:32.815 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:43:32.809+0000 7f89824fe640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=33,config=1,health=11,kv=10,logm=45} crc {auth=2219382330,config=1228700967,health=492630217,kv=3446188498,logm=365078855}) 2024-09-16T03:43:32.815 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:43:32.809+0000 7f89824fe640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:43:32.815 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:43:32.809+0000 7f89824fe640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=32,config=2,health=11,kv=10,logm=45} crc {auth=3840062311,config=692619927,health=632208798,kv=871254744,logm=1453121405}) 2024-09-16T03:43:32.816 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:43:32.809+0000 7f89824fe640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=33,config=2,health=11,kv=9,logm=45} crc {auth=2219382330,config=692619927,health=305525474,kv=3446188498,logm=3663628831}) 2024-09-16T03:43:32.816 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.809+0000 7f4453c4d640 10 librados: watch_flush enter 2024-09-16T03:43:32.817 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.809+0000 7f4453c4d640 10 librados: watch_flush exit 2024-09-16T03:43:32.817 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.809+0000 7f4453c4d640 20 client.4710.objecter shutdown clearing up homeless session... 2024-09-16T03:43:32.817 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.809+0000 7f4453c4d640 10 client.4710.objecter successfully canceled tick 2024-09-16T03:43:32.817 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.809+0000 7f4453c4d640 1 -- 172.21.15.42:0/106671836 >> v2:172.21.15.154:6812/254845342 conn(0x7f4430037840 msgr2=0x7f4430039d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:43:32.817 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.809+0000 7f4453c4d640 1 --2- 172.21.15.42:0/106671836 >> v2:172.21.15.154:6812/254845342 conn(0x7f4430037840 0x7f4430039d00 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f4448009c20 tx=0x7f4448009190 comp rx=0 tx=0).stop 2024-09-16T03:43:32.817 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.809+0000 7f4453c4d640 10 monclient: shutdown 2024-09-16T03:43:32.817 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.809+0000 7f4453c4d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:43:32.817 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.809+0000 7f4453c4d640 1 -- 172.21.15.42:0/106671836 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f444c0a67e0 msgr2=0x7f444c134c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:43:32.817 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.809+0000 7f4453c4d640 1 --2- 172.21.15.42:0/106671836 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f444c0a67e0 0x7f444c134c80 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f443c00efd0 tx=0x7f443c001a40 comp rx=0 tx=0).stop 2024-09-16T03:43:32.817 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.809+0000 7f4453c4d640 1 -- 172.21.15.42:0/106671836 shutdown_connections 2024-09-16T03:43:32.817 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.809+0000 7f4453c4d640 1 --2- 172.21.15.42:0/106671836 >> v2:172.21.15.154:6812/254845342 conn(0x7f4430037840 0x7f4430039d00 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:43:32.817 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.809+0000 7f4453c4d640 1 --2- 172.21.15.42:0/106671836 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f444c0a7100 0x7f444c13aa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:43:32.818 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.809+0000 7f4453c4d640 1 --2- 172.21.15.42:0/106671836 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f444c0a67e0 0x7f444c134c80 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:43:32.818 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.809+0000 7f4453c4d640 1 --2- 172.21.15.42:0/106671836 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f444c072560 0x7f444c134740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:43:32.818 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.809+0000 7f4453c4d640 1 -- 172.21.15.42:0/106671836 >> 172.21.15.42:0/106671836 conn(0x7f444c0a7ed0 msgr2=0x7f444c0a04f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:43:32.818 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.809+0000 7f4453c4d640 1 -- 172.21.15.42:0/106671836 shutdown_connections 2024-09-16T03:43:32.818 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.809+0000 7f4453c4d640 1 -- 172.21.15.42:0/106671836 wait complete. 2024-09-16T03:43:32.818 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:32.809+0000 7f4453c4d640 1 librados: shutdown 2024-09-16T03:43:32.818 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:43:32.813+0000 7f89824fe640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:43:32.818 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:43:32.813+0000 7f89824fe640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3624483910}) 2024-09-16T03:43:32.818 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:43:32.813+0000 7f89824fe640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2065865964}) 2024-09-16T03:43:32.819 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:43:32.813+0000 7f89824fe640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:43:32.819 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:43:32.813+0000 7f89824fe640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3624483910}) 2024-09-16T03:43:32.819 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:43:32.813+0000 7f89824fe640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4242638446}) 2024-09-16T03:43:32.819 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:43:32.813+0000 7f89824fe640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:43:32.819 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:43:32.813+0000 7f89824fe640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=386764388}) 2024-09-16T03:43:32.819 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:43:32.813+0000 7f89824fe640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1667336772}) 2024-09-16T03:43:32.819 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:43:32.813+0000 7f89824fe640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:43:32.820 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:43:32.813+0000 7f89824fe640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=386764388}) 2024-09-16T03:43:32.820 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:43:32.813+0000 7f89824fe640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1516688375}) 2024-09-16T03:43:32.821 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:43:32.813+0000 7f89824fe640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:43:32.821 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:43:32.813+0000 7f89824fe640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1025372322}) 2024-09-16T03:43:32.821 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:43:32.813+0000 7f89824fe640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2586899173}) 2024-09-16T03:43:32.821 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:43:32.813+0000 7f89824fe640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:43:32.821 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:43:32.813+0000 7f89824fe640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1025372322}) 2024-09-16T03:43:32.821 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:43:32.813+0000 7f89824fe640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2757716554}) 2024-09-16T03:43:32.824 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:43:32.817+0000 7f89824fe640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:43:32.824 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:43:32.817+0000 7f89824fe640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=5,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=9,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=41} crc {logm=436729673,mdsmap=4088524775,mgr=911650295,mgr_command_descs=2107673600,mgr_metadata=1367272202,mgrstat=584021557,mon_config_key=1020304259,monmap=3937513981,osd_metadata=4182959540,osd_pg_creating=2579136188,osd_snap=841190635,osdmap=3498781957}) 2024-09-16T03:43:32.825 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:43:32.817+0000 7f89824fe640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=5,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=41} crc {logm=436729673,mdsmap=4088524775,mgr=1669371261,mgr_command_descs=2107673600,mgr_metadata=1367272202,mgrstat=584021557,mon_config_key=85057112,monmap=3937513981,osd_metadata=4182959540,osd_pg_creating=2579136188,osd_snap=841190635,osdmap=4167433846}) 2024-09-16T03:43:32.825 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:43:32.817+0000 7f89824fe640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:43:32.825 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:43:32.817+0000 7f89824fe640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=5,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=9,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=41} crc {logm=436729673,mdsmap=4088524775,mgr=911650295,mgr_command_descs=2107673600,mgr_metadata=1367272202,mgrstat=584021557,mon_config_key=1020304259,monmap=3937513981,osd_metadata=4182959540,osd_pg_creating=2579136188,osd_snap=841190635,osdmap=3498781957}) 2024-09-16T03:43:32.825 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:43:32.817+0000 7f89824fe640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=5,mdsmap=3,mgr=8,mgr_metadata=1,mgrstat=12,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=2,osdmap=40} crc {logm=436729673,mdsmap=4088524775,mgr=460109776,mgr_metadata=1367272202,mgrstat=584021557,mon_config_key=85057112,monmap=3937513981,osd_metadata=4182959540,osd_pg_creating=2579136188,osd_snap=841190635,osdmap=2401131182}) 2024-09-16T03:43:32.835 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T03:43:33.263 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.383+0000 7f90fd4a0640 1 -- 172.21.15.154:0/332177798 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 3534+0+0 (secure 0 0 0) 0x7f90f00e5120 con 0x5607e05bda50 2024-09-16T03:43:33.264 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.383+0000 7f90fd4a0640 10 client.14182.objecter ms_dispatch 0x5607e05160c0 osd_map(19..19 src has 1..19) 2024-09-16T03:43:33.264 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.383+0000 7f90fd4a0640 3 client.14182.objecter handle_osd_map got epochs [19,19] > 0 2024-09-16T03:43:33.264 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.383+0000 7f90fd4a0640 3 client.14182.objecter handle_osd_map decoding full epoch 19 2024-09-16T03:43:33.264 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.383+0000 7f90ff4a4640 1 --2- 172.21.15.154:0/332177798 >> v2:172.21.15.154:6812/254845342 conn(0x7f90ec035a30 0x7f90ec037ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:43:33.264 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.383+0000 7f90fd4a0640 20 client.14182.objecter dump_active .. 0 homeless 2024-09-16T03:43:33.264 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.383+0000 7f90ff4a4640 10 monclient: get_auth_request con 0x7f90ec035a30 auth_method 0 2024-09-16T03:43:33.264 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.383+0000 7f9102191880 10 librados: wait_for_osdmap done waiting 2024-09-16T03:43:33.264 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.383+0000 7f9102191880 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2024-09-16T03:43:33.264 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.383+0000 7f9102191880 1 -- 172.21.15.154:0/332177798 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f90f8013520 con 0x5607e05bda50 2024-09-16T03:43:33.264 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.387+0000 7f90ff4a4640 1 --2- 172.21.15.154:0/332177798 >> v2:172.21.15.154:6812/254845342 conn(0x7f90ec035a30 0x7f90ec037ef0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f90f4000f70 tx=0x7f90f4010040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:43:33.264 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.387+0000 7f90fd4a0640 10 client.14182.objecter ms_handle_connect 0x7f90ec035a30 2024-09-16T03:43:33.264 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.395+0000 7f90fd4a0640 1 -- 172.21.15.154:0/332177798 <== mon.1 v2:172.21.15.154:3300/0 6 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f90f00e7060 con 0x5607e05bda50 2024-09-16T03:43:33.265 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.395+0000 7f90fd4a0640 10 monclient: handle_get_version_reply finishing 1 version 19 2024-09-16T03:43:33.265 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.395+0000 7f90fdca1640 10 client.14182.objecter _get_latest_version latest 19, have it 2024-09-16T03:43:33.265 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.395+0000 7f9102191880 10 client.14182.objecter create_pool name=ceph_test_rados_api_asio 2024-09-16T03:43:33.265 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.395+0000 7f9102191880 10 client.14182.objecter pool_op_submit 1 2024-09-16T03:43:33.265 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.395+0000 7f9102191880 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2024-09-16T03:43:33.265 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:31.395+0000 7f9102191880 1 -- 172.21.15.154:0/332177798 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- pool_op(create pool 0 tid 1 name ceph_test_rados_api_asio v0) -- 0x5607e05450f0 con 0x5607e05bda50 2024-09-16T03:43:33.265 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:33.259+0000 7f90fd4a0640 1 -- 172.21.15.154:0/332177798 <== mon.1 v2:172.21.15.154:3300/0 7 ==== pool_op_reply(tid 1 (0) Success v21) ==== 43+0+0 (secure 0 0 0) 0x7f90f0014fc0 con 0x5607e05bda50 2024-09-16T03:43:33.265 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:33.263+0000 7f90fd4a0640 10 client.14182.objecter ms_dispatch 0x5607e05160c0 pool_op_reply(tid 1 (0) Success v21) 2024-09-16T03:43:33.265 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:33.263+0000 7f90fd4a0640 10 client.14182.objecter handle_pool_op_reply pool_op_reply(tid 1 (0) Success v21) 2024-09-16T03:43:33.265 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:33.263+0000 7f90fd4a0640 10 client.14182.objecter have request 1 at 0x5607e064d3e0 Op: create 2024-09-16T03:43:33.265 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:33.263+0000 7f90fd4a0640 20 client.14182.objecter waiting for client to reach epoch 21 before calling back 2024-09-16T03:43:33.266 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:33.263+0000 7f90fd4a0640 10 client.14182.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:43:33.266 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:33.263+0000 7f90fd4a0640 10 monclient: _renew_subs 2024-09-16T03:43:33.267 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:33.263+0000 7f90fd4a0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2024-09-16T03:43:33.267 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:33.263+0000 7f90fd4a0640 1 -- 172.21.15.154:0/332177798 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=20}) -- 0x7f90ec03aec0 con 0x5607e05bda50 2024-09-16T03:43:33.267 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:33.263+0000 7f90fd4a0640 10 client.14182.objecter done 2024-09-16T03:43:33.267 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:33.263+0000 7f90fd4a0640 1 -- 172.21.15.154:0/332177798 <== mon.1 v2:172.21.15.154:3300/0 8 ==== osd_map(20..21 src has 1..21) ==== 20656+0+0 (secure 0 0 0) 0x7f90f00b47e0 con 0x5607e05bda50 2024-09-16T03:43:33.267 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:33.263+0000 7f90fd4a0640 10 client.14182.objecter ms_dispatch 0x5607e05160c0 osd_map(20..21 src has 1..21) 2024-09-16T03:43:33.267 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:33.263+0000 7f90fd4a0640 3 client.14182.objecter handle_osd_map got epochs [20,21] > 19 2024-09-16T03:43:33.267 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:33.263+0000 7f90fd4a0640 3 client.14182.objecter handle_osd_map decoding incremental epoch 20 2024-09-16T03:43:33.267 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:33.263+0000 7f90fd4a0640 3 client.14182.objecter handle_osd_map decoding incremental epoch 21 2024-09-16T03:43:33.267 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:33.263+0000 7f90fd4a0640 20 client.14182.objecter dump_active .. 0 homeless 2024-09-16T03:43:33.267 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:33.263+0000 7f9102191880 10 librados: set snap read head -> 1 2024-09-16T03:43:33.267 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:33.263+0000 7f9102191880 10 librados: writefull oid=exist nspace= 2024-09-16T03:43:33.267 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:33.263+0000 7f9102191880 10 client.14182.objecter _op_submit op 0x5607e06e5280 2024-09-16T03:43:33.267 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:33.263+0000 7f9102191880 20 client.14182.objecter _calc_target epoch 21 base exist @31 precalc_pgid 0 pgid 0.0 is_write 2024-09-16T03:43:33.267 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:33.263+0000 7f9102191880 20 client.14182.objecter _calc_target target exist @31 -> pgid 31.d3a59201 2024-09-16T03:43:33.268 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:33.263+0000 7f9102191880 10 client.14182.objecter _calc_target raw pgid 31.d3a59201 -> actual 31.1 acting [1,4] primary 1 2024-09-16T03:43:33.268 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:33.263+0000 7f9102191880 1 --2- 172.21.15.154:0/332177798 >> v2:172.21.15.42:6800/4217863575 conn(0x5607e06f6d30 0x5607e06f9120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:43:33.268 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:33.263+0000 7f9102191880 20 client.14182.objecter _get_session s=0x5607e0653f50 osd=1 3 2024-09-16T03:43:33.268 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:33.263+0000 7f9102191880 10 client.14182.objecter _op_submit oid exist '@31' '@31' [writefull 0~5 in=5b] tid 2 osd.1 2024-09-16T03:43:33.268 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:33.263+0000 7f9102191880 20 client.14182.objecter get_session s=0x5607e0653f50 osd=1 3 2024-09-16T03:43:33.268 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:33.263+0000 7f9102191880 15 client.14182.objecter _session_op_assign 1 2 2024-09-16T03:43:33.268 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:33.263+0000 7f9102191880 15 client.14182.objecter _send_op 2 to 31.1 on osd.1 2024-09-16T03:43:33.268 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:33.263+0000 7f9102191880 1 -- 172.21.15.154:0/332177798 --> v2:172.21.15.42:6800/4217863575 -- osd_op(unknown.0.0:2 31.1 31:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21) -- 0x5607e06f9660 con 0x5607e06f6d30 2024-09-16T03:43:33.268 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:33.263+0000 7f9102191880 20 client.14182.objecter put_session s=0x5607e0653f50 osd=1 4 2024-09-16T03:43:33.268 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:33.263+0000 7f9102191880 5 client.14182.objecter 1 in flight 2024-09-16T03:43:33.836 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T03:43:33.836 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:43:33.974 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.969+0000 7f50cc20b640 10 monclient: get_monmap_and_config 2024-09-16T03:43:33.975 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.969+0000 7f50cc20b640 10 monclient: build_initial_monmap 2024-09-16T03:43:33.975 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.969+0000 7f50cc20b640 10 monclient: monmap: 2024-09-16T03:43:33.975 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:43:33.975 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:43:33.975 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:43:33.973331+0000 2024-09-16T03:43:33.975 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:43:33.973331+0000 2024-09-16T03:43:33.975 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:43:33.975 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:43:33.975 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:43:33.975 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:43:33.975 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:43:33.975 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:43:33.975 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:43:33.976 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:43:33.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.969+0000 7f50cc20b640 1 Processor -- start 2024-09-16T03:43:33.978 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.969+0000 7f50cc20b640 1 -- start start 2024-09-16T03:43:33.978 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.969+0000 7f50cc20b640 10 monclient: init 2024-09-16T03:43:33.978 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.969+0000 7f50cc20b640 10 monclient: _reopen_session rank -1 2024-09-16T03:43:33.978 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.969+0000 7f50cc20b640 10 monclient: _add_conns ranks=[1,2,3,0,4] 2024-09-16T03:43:33.978 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.969+0000 7f50cc20b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f50c40a39b0 addr v1:172.21.15.154:6789/0 2024-09-16T03:43:33.978 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.969+0000 7f50cc20b640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f50c40a3d90 0x7f50c413c380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:43:33.978 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.969+0000 7f50cc20b640 10 monclient(hunting): picked mon.noname-b con 0x7f50c40a3d90 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:43:33.978 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.969+0000 7f50cc20b640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f50c413c8c0 0x7f50c413ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:43:33.978 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.969+0000 7f50cc20b640 10 monclient(hunting): picked mon.noname-a con 0x7f50c413c8c0 addr v2:172.21.15.42:3300/0 2024-09-16T03:43:33.978 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.969+0000 7f50cc20b640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f50c4134290 con 0x7f50c40a39b0 2024-09-16T03:43:33.978 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.969+0000 7f50cc20b640 10 monclient(hunting): start opening mon connection 2024-09-16T03:43:33.978 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.969+0000 7f50cc20b640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f50c409c140 con 0x7f50c413c8c0 2024-09-16T03:43:33.978 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.969+0000 7f50cc20b640 10 monclient(hunting): start opening mon connection 2024-09-16T03:43:33.979 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.969+0000 7f50cc20b640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f50c409c420 con 0x7f50c40a3d90 2024-09-16T03:43:33.979 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.969+0000 7f50cc20b640 10 monclient(hunting): _renew_subs 2024-09-16T03:43:33.979 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.969+0000 7f50c9fa7640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f50c40a39b0 0x7f50c4133b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.42:54032/0 (socket says 172.21.15.42:54032) 2024-09-16T03:43:33.979 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.969+0000 7f50cc20b640 10 monclient(hunting): authenticate will time out at 1060.943078s 2024-09-16T03:43:33.979 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.969+0000 7f50c9fa7640 1 -- 172.21.15.42:0/190392604 learned_addr learned my addr 172.21.15.42:0/190392604 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:43:33.979 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.969+0000 7f50c97a6640 1 --2- 172.21.15.42:0/190392604 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f50c40a3d90 0x7f50c413c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:43:33.979 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.969+0000 7f50c97a6640 10 monclient(hunting): get_auth_request con 0x7f50c40a3d90 auth_method 0 2024-09-16T03:43:33.979 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.969+0000 7f50c97a6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:43:33.979 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.969+0000 7f50c97a6640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:43:33.979 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.969+0000 7f50c97a6640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:43:33.979 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.969+0000 7f50c97a6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:43:33.979 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.969+0000 7f50c97a6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:43:33.979 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.969+0000 7f50c97a6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:43:33.980 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.969+0000 7f50ca7a8640 1 --2- 172.21.15.42:0/190392604 >> v2:172.21.15.42:3300/0 conn(0x7f50c413c8c0 0x7f50c413ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:43:33.980 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.969+0000 7f50ca7a8640 10 monclient(hunting): get_auth_request con 0x7f50c413c8c0 auth_method 0 2024-09-16T03:43:33.980 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.969+0000 7f50ca7a8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:43:33.980 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.969+0000 7f50ca7a8640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:43:33.980 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.969+0000 7f50ca7a8640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:43:33.980 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.969+0000 7f50ca7a8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:43:33.980 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.969+0000 7f50ca7a8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:43:33.980 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.969+0000 7f50c8fa5640 1 -- 172.21.15.42:0/190392604 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3963372879 0 0) 0x7f50c4134290 con 0x7f50c40a39b0 2024-09-16T03:43:33.980 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.969+0000 7f50ca7a8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:43:33.980 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.969+0000 7f50c8fa5640 10 monclient(hunting): _negotiate 2024-09-16T03:43:33.980 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.969+0000 7f50c8fa5640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:43:33.980 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.969+0000 7f50c8fa5640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:43:33.980 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.969+0000 7f50c8fa5640 10 monclient(hunting): my global_id is 14272 2024-09-16T03:43:33.981 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.969+0000 7f50c8fa5640 1 -- 172.21.15.42:0/190392604 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f50b0003530 con 0x7f50c40a39b0 2024-09-16T03:43:33.981 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.969+0000 7f50c97a6640 10 monclient(hunting): handle_auth_done global_id 4586 payload 274 2024-09-16T03:43:33.981 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.969+0000 7f50c97a6640 1 -- 172.21.15.42:0/190392604 >> v2:172.21.15.42:3300/0 conn(0x7f50c413c8c0 msgr2=0x7f50c413ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:43:33.981 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.969+0000 7f50c97a6640 1 --2- 172.21.15.42:0/190392604 >> v2:172.21.15.42:3300/0 conn(0x7f50c413c8c0 0x7f50c413ecb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:43:33.981 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.969+0000 7f50c97a6640 1 -- 172.21.15.42:0/190392604 >> v1:172.21.15.154:6789/0 conn(0x7f50c40a39b0 legacy=0x7f50c4133b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:43:33.981 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.969+0000 7f50c97a6640 10 monclient: _finish_hunting 0 2024-09-16T03:43:33.981 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.969+0000 7f50c97a6640 1 monclient: found mon.noname-b 2024-09-16T03:43:33.981 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.969+0000 7f50c97a6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:43:33.981 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.969+0000 7f50c97a6640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T03:43:33.981 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.969+0000 7f50c97a6640 1 -- 172.21.15.42:0/190392604 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50c406dd60 con 0x7f50c40a3d90 2024-09-16T03:43:33.984 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.973+0000 7f50c97a6640 1 --2- 172.21.15.42:0/190392604 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f50c40a3d90 0x7f50c413c380 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f50b40097c0 tx=0x7f50b401c360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fac2ba02217cdfa server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:43:33.984 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.973+0000 7f50c8fa5640 1 -- 172.21.15.42:0/190392604 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50b40b0070 con 0x7f50c40a3d90 2024-09-16T03:43:33.984 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.973+0000 7f50c8fa5640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:43:33.984 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.973+0000 7f50c8fa5640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:43:33.985 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.973+0000 7f50c8fa5640 10 monclient: dump: 2024-09-16T03:43:33.985 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:43:33.985 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:43:33.985 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:43:33.985 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:43:33.985 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:43:33.985 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:43:33.985 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:43:33.985 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:43:33.985 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:43:33.985 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:43:33.985 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.973+0000 7f50c8fa5640 10 monclient: _finish_auth 0 2024-09-16T03:43:33.985 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.973+0000 7f50c8fa5640 10 monclient: _check_auth_tickets 2024-09-16T03:43:33.986 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.973+0000 7f50c8fa5640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:43:33.986 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.973+0000 7f50cc20b640 5 monclient: authenticate success, global_id 4586 2024-09-16T03:43:33.986 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.973+0000 7f50cc20b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:43:33.986 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.973+0000 7f50c8fa5640 1 -- 172.21.15.42:0/190392604 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f50b40ab600 con 0x7f50c40a3d90 2024-09-16T03:43:33.986 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.973+0000 7f50c8fa5640 10 monclient: handle_config config(0 keys) 2024-09-16T03:43:33.986 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.973+0000 7f50c8fa5640 1 -- 172.21.15.42:0/190392604 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50b40ab9d0 con 0x7f50c40a3d90 2024-09-16T03:43:33.986 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.973+0000 7f50c8fa5640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:43:33.986 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.973+0000 7f50c8fa5640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:43:33.986 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.973+0000 7f50c8fa5640 10 monclient: dump: 2024-09-16T03:43:33.986 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:43:33.986 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:43:33.986 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:43:33.986 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:43:33.987 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:43:33.987 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:43:33.987 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:43:33.987 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:43:33.987 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:43:33.987 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:43:33.987 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.973+0000 7f50cc20b640 10 monclient: get_monmap_and_config success 2024-09-16T03:43:33.987 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.973+0000 7f50cc20b640 4 set_mon_vals no callback set 2024-09-16T03:43:33.987 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.973+0000 7f50cc20b640 10 monclient: shutdown 2024-09-16T03:43:33.987 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.973+0000 7f50cc20b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:43:33.987 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.973+0000 7f50cc20b640 1 -- 172.21.15.42:0/190392604 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f50c40a3d90 msgr2=0x7f50c413c380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:43:33.987 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.973+0000 7f50cc20b640 1 --2- 172.21.15.42:0/190392604 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f50c40a3d90 0x7f50c413c380 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f50b40097c0 tx=0x7f50b401c360 comp rx=0 tx=0).stop 2024-09-16T03:43:33.987 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.973+0000 7f50cc20b640 1 -- 172.21.15.42:0/190392604 shutdown_connections 2024-09-16T03:43:33.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.973+0000 7f50cc20b640 1 --2- 172.21.15.42:0/190392604 >> v2:172.21.15.42:3300/0 conn(0x7f50c413c8c0 0x7f50c413ecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:43:33.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.973+0000 7f50cc20b640 1 --2- 172.21.15.42:0/190392604 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f50c40a3d90 0x7f50c413c380 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:43:33.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.973+0000 7f50cc20b640 1 -- 172.21.15.42:0/190392604 >> 172.21.15.42:0/190392604 conn(0x7f50c40a9fb0 msgr2=0x7f50c40aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:43:33.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.973+0000 7f50cc20b640 1 -- 172.21.15.42:0/190392604 shutdown_connections 2024-09-16T03:43:33.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.973+0000 7f50cc20b640 1 -- 172.21.15.42:0/190392604 wait complete. 2024-09-16T03:43:33.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.977+0000 7f50cc20b640 10 monclient: build_initial_monmap 2024-09-16T03:43:33.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.977+0000 7f50cc20b640 10 monclient: monmap: 2024-09-16T03:43:33.988 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:43:33.988 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:43:33.988 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:43:33.988 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:43:33.988 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:43:33.989 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:43:33.989 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:43:33.989 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:43:33.989 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:43:33.989 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:43:33.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.977+0000 7f50cc20b640 1 librados: starting msgr at 2024-09-16T03:43:33.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.977+0000 7f50cc20b640 1 librados: starting objecter 2024-09-16T03:43:33.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.977+0000 7f50cc20b640 1 Processor -- start 2024-09-16T03:43:33.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.977+0000 7f50cc20b640 1 -- start start 2024-09-16T03:43:33.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.977+0000 7f50cc20b640 1 librados: setting wanted keys 2024-09-16T03:43:33.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.977+0000 7f50cc20b640 1 librados: calling monclient init 2024-09-16T03:43:33.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.977+0000 7f50cc20b640 10 monclient: init 2024-09-16T03:43:33.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.977+0000 7f50cc20b640 10 monclient: _reopen_session rank -1 2024-09-16T03:43:33.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.977+0000 7f50cc20b640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T03:43:33.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.977+0000 7f50cc20b640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f50c40a39b0 0x7f50c41cf2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:43:33.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.977+0000 7f50cc20b640 10 monclient(hunting): picked mon.noname-a con 0x7f50c40a39b0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:43:33.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.977+0000 7f50cc20b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f50c40a3d90 0x7f50c41cf7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:43:33.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.977+0000 7f50cc20b640 10 monclient(hunting): picked mon.noname-b con 0x7f50c40a3d90 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:43:33.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.977+0000 7f50cc20b640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f50c413c8c0 0x7f50c41cfd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:43:33.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.977+0000 7f50cc20b640 10 monclient(hunting): picked mon.noname-c con 0x7f50c413c8c0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:43:33.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.977+0000 7f50cc20b640 10 monclient(hunting): start opening mon connection 2024-09-16T03:43:33.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.977+0000 7f50cc20b640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f50c4141da0 con 0x7f50c40a39b0 2024-09-16T03:43:33.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.977+0000 7f50cc20b640 10 monclient(hunting): start opening mon connection 2024-09-16T03:43:33.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.977+0000 7f50cc20b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f50c4141c20 con 0x7f50c40a3d90 2024-09-16T03:43:33.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.977+0000 7f50cc20b640 10 monclient(hunting): start opening mon connection 2024-09-16T03:43:33.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.977+0000 7f50cc20b640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f50c4141f20 con 0x7f50c413c8c0 2024-09-16T03:43:33.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.977+0000 7f50cc20b640 10 monclient(hunting): _renew_subs 2024-09-16T03:43:33.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.977+0000 7f50cc20b640 10 monclient(hunting): authenticate will time out at 1060.949432s 2024-09-16T03:43:33.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.977+0000 7f50ca7a8640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f50c413c8c0 0x7f50c41cfd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:43:33.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.977+0000 7f50c9fa7640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f50c40a39b0 0x7f50c41cf2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:43:33.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.977+0000 7f50ca7a8640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f50c413c8c0 0x7f50c41cfd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:37740/0 (socket says 172.21.15.42:37740) 2024-09-16T03:43:33.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.977+0000 7f50ca7a8640 1 -- 172.21.15.42:0/2337586359 learned_addr learned my addr 172.21.15.42:0/2337586359 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:43:33.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.977+0000 7f50c9fa7640 10 monclient(hunting): get_auth_request con 0x7f50c40a39b0 auth_method 0 2024-09-16T03:43:33.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.977+0000 7f50c9fa7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:43:33.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.977+0000 7f50c97a6640 1 --2- 172.21.15.42:0/2337586359 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f50c40a3d90 0x7f50c41cf7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:43:33.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.977+0000 7f50c9fa7640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:43:33.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.977+0000 7f50c9fa7640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:43:33.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.977+0000 7f50ca7a8640 10 monclient(hunting): get_auth_request con 0x7f50c413c8c0 auth_method 0 2024-09-16T03:43:33.992 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.977+0000 7f50ca7a8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:43:33.992 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.977+0000 7f50ca7a8640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:43:33.992 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.977+0000 7f50ca7a8640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:43:33.992 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.977+0000 7f50c97a6640 10 monclient(hunting): get_auth_request con 0x7f50c40a3d90 auth_method 0 2024-09-16T03:43:33.992 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.977+0000 7f50c97a6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:43:33.992 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.977+0000 7f50c97a6640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:43:33.992 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.977+0000 7f50c97a6640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:43:33.992 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.977+0000 7f50c97a6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:43:33.992 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.977+0000 7f50c97a6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:43:33.992 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.977+0000 7f50c97a6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:43:33.992 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.977+0000 7f50ca7a8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:43:33.992 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.977+0000 7f50ca7a8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:43:33.992 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.977+0000 7f50ca7a8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:43:33.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.977+0000 7f50c9fa7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:43:33.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.977+0000 7f50c9fa7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:43:33.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.977+0000 7f50c9fa7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:43:33.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.977+0000 7f50ca7a8640 10 monclient(hunting): handle_auth_done global_id 4589 payload 824 2024-09-16T03:43:33.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.977+0000 7f50ca7a8640 1 -- 172.21.15.42:0/2337586359 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f50c40a3d90 msgr2=0x7f50c41cf7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:43:33.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.977+0000 7f50ca7a8640 1 --2- 172.21.15.42:0/2337586359 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f50c40a3d90 0x7f50c41cf7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:43:33.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.977+0000 7f50ca7a8640 1 -- 172.21.15.42:0/2337586359 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f50c40a39b0 msgr2=0x7f50c41cf2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:43:33.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.977+0000 7f50ca7a8640 1 --2- 172.21.15.42:0/2337586359 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f50c40a39b0 0x7f50c41cf2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:43:33.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.977+0000 7f50ca7a8640 10 monclient: _finish_hunting 0 2024-09-16T03:43:33.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.977+0000 7f50ca7a8640 1 monclient: found mon.noname-c 2024-09-16T03:43:33.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.977+0000 7f50ca7a8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:43:33.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.977+0000 7f50ca7a8640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:43:33.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.977+0000 7f50ca7a8640 1 -- 172.21.15.42:0/2337586359 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50c413ac90 con 0x7f50c413c8c0 2024-09-16T03:43:33.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.977+0000 7f50c9fa7640 1 --2- 172.21.15.42:0/2337586359 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f50c40a39b0 0x7f50c41cf2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:43:33.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.977+0000 7f50ca7a8640 1 --2- 172.21.15.42:0/2337586359 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f50c413c8c0 0x7f50c41cfd30 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f50bc00efc0 tx=0x7f50bc00c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:43:33.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.977+0000 7f50ba7fc640 10 client.?.objecter ms_handle_connect 0x7f50c413c8c0 2024-09-16T03:43:33.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.977+0000 7f50ba7fc640 10 client.?.objecter resend_mon_ops 2024-09-16T03:43:33.994 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:33.263+0000 7f91004a6640 1 --2- 172.21.15.154:0/332177798 >> v2:172.21.15.42:6800/4217863575 conn(0x5607e06f6d30 0x5607e06f9120 unknown :-1 s=BANNER_CONNECTIN api_list: [==========] Running 11 tests from 3 test suites. 2024-09-16T03:43:33.994 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: [----------] Global test environment set-up. 2024-09-16T03:43:33.994 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: [----------] 7 tests from LibRadosList 2024-09-16T03:43:33.995 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: [ RUN ] LibRadosList.ListObjects 2024-09-16T03:43:33.995 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: [ OK ] LibRadosList.ListObjects (331 ms) 2024-09-16T03:43:33.995 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: [ RUN ] LibRadosList.ListObjectsZeroInName 2024-09-16T03:43:33.995 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: [ OK ] LibRadosList.ListObjectsZeroInName (38 ms) 2024-09-16T03:43:33.995 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: [ RUN ] LibRadosList.ListObjectsNS 2024-09-16T03:43:33.995 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: myset foo1,foo2,foo3 2024-09-16T03:43:33.995 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: foo1 2024-09-16T03:43:33.995 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: foo2 2024-09-16T03:43:33.995 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: foo3 2024-09-16T03:43:33.995 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: myset foo1,foo4,foo5 2024-09-16T03:43:33.995 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: foo4 2024-09-16T03:43:33.995 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: foo5 2024-09-16T03:43:33.995 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: foo1 2024-09-16T03:43:33.996 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: myset foo6,foo7 2024-09-16T03:43:33.996 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: foo7 2024-09-16T03:43:33.996 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: foo6 2024-09-16T03:43:33.996 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: myset :foo1,:foo2,:foo3,ns1:foo1,ns1:foo4,ns1:foo5,ns2:foo6,ns2:foo7 2024-09-16T03:43:33.996 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: ns1:foo4 2024-09-16T03:43:33.996 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: ns1:foo5 2024-09-16T03:43:33.996 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: ns2:foo7 2024-09-16T03:43:33.996 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: ns2:foo6 2024-09-16T03:43:33.996 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: ns1:foo1 2024-09-16T03:43:33.996 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: :foo1 2024-09-16T03:43:33.996 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: :foo2 2024-09-16T03:43:33.996 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: :foo3 2024-09-16T03:43:33.996 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: [ OK ] LibRadosList.ListObjectsNS (94 ms) 2024-09-16T03:43:33.996 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: [ RUN ] LibRadosList.ListObjectsStart 2024-09-16T03:43:33.997 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: 1 0 2024-09-16T03:43:33.997 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: 10 0 2024-09-16T03:43:33.997 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: 13 0 2024-09-16T03:43:33.997 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: 7 0 2024-09-16T03:43:33.997 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: 14 0 2024-09-16T03:43:33.997 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: 0 0 2024-09-16T03:43:33.997 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: 15 0 2024-09-16T03:43:33.997 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: 11 0 2024-09-16T03:43:33.997 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: 5 0 2024-09-16T03:43:33.997 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: 8 0 2024-09-16T03:43:33.997 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: 6 0 2024-09-16T03:43:33.997 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: 3 0 2024-09-16T03:43:33.997 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: 4 0 2024-09-16T03:43:33.997 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: 12 0 2024-09-16T03:43:33.998 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: 9 0 2024-09-16T03:43:33.998 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: 2 0 2024-09-16T03:43:33.998 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: have 1 expect one of 0,1,10,11,12,13,14,15,2,3,4,5,6,7,8,9 2024-09-16T03:43:33.998 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: [ OK ] LibRadosList.ListObjectsStart (95 ms) 2024-09-16T03:43:33.998 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: [ RUN ] LibRadosList.ListObjectsCursor 2024-09-16T03:43:33.998 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: x cursor=MIN 2024-09-16T03:43:33.998 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: > oid=1 cursor=12:02547ec2:::1:head 2024-09-16T03:43:33.998 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: > oid=10 cursor=12:52ea6a34:::10:head 2024-09-16T03:43:33.998 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: > oid=13 cursor=12:566253c9:::13:head 2024-09-16T03:43:33.998 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: > oid=7 cursor=12:5c6b0b28:::7:head 2024-09-16T03:43:33.998 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: > oid=14 cursor=12:62a1935d:::14:head 2024-09-16T03:43:33.998 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: > oid=0 cursor=12:6cac518f:::0:head 2024-09-16T03:43:33.998 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: > oid=15 cursor=12:863748b0:::15:head 2024-09-16T03:43:33.999 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: > oid=11 cursor=12:89d3ae78:::11:head 2024-09-16T03:43:33.999 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: > oid=5 cursor=12:b29083e3:::5:head 2024-09-16T03:43:33.999 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: > oid=8 cursor=12:bd63b0f1:::8:head 2024-09-16T03:43:33.999 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: > oid=6 cursor=12:c4fdafeb:::6:head 2024-09-16T03:43:33.999 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: > oid=3 cursor=12:cfc208b3:::3:head 2024-09-16T03:43:33.999 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: > oid=4 cursor=12:d83876eb:::4:head 2024-09-16T03:43:33.999 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: > oid=12 cursor=12:de5d7c5f:::12:head 2024-09-16T03:43:33.999 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: > oid=9 cursor=12:e960b815:::9:head 2024-09-16T03:43:33.999 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: > oid=2 cursor=12:f905c69b:::2:head 2024-09-16T03:43:33.999 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: FIRST> seek to MIN oid=1 2024-09-16T03:43:33.999 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: : oid=1 cursor=12:02547ec2:::1:head 2024-09-16T03:43:33.999 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: : seek to 12:02547ec2:::1:head 2024-09-16T03:43:33.999 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: > 12:02547ec2:::1:head -> 1 2024-09-16T03:43:33.999 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: : oid=10 cursor=12:52ea6a34:::10:head 2024-09-16T03:43:34.000 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: : seek to 12:52ea6a34:::10:head 2024-09-16T03:43:34.000 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: > 12:52ea6a34:::10:head -> 10 2024-09-16T03:43:34.000 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: : oid=13 cursor=12:566253c9:::13:head 2024-09-16T03:43:34.000 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: : seek to 12:566253c9:::13:head 2024-09-16T03:43:34.000 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: > 12:566253c9:::13:head -> 13 2024-09-16T03:43:34.000 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: : oid=7 cursor=12:5c6b0b28:::7:head 2024-09-16T03:43:34.000 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: : seek to 12:5c6b0b28:::7:head 2024-09-16T03:43:34.000 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: > 12:5c6b0b28:::7:head -> 7 2024-09-16T03:43:34.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.977+0000 7f50ba7fc640 1 -- 172.21.15.42:0/2337586359 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50bc018020 con 0x7f50c413c8c0 2024-09-16T03:43:34.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.977+0000 7f50ba7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:43:34.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.977+0000 7f50ba7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:43:34.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.977+0000 7f50ba7fc640 10 monclient: dump: 2024-09-16T03:43:34.002 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:43:34.002 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:43:34.002 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:43:34.002 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:43:34.002 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:43:34.002 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:43:34.002 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:43:34.003 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:43:34.003 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:43:34.003 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:43:34.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.977+0000 7f50ba7fc640 10 monclient: _finish_auth 0 2024-09-16T03:43:34.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.977+0000 7f50ba7fc640 10 monclient: _check_auth_tickets 2024-09-16T03:43:34.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.977+0000 7f50ba7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:43:34.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.977+0000 7f50ba7fc640 1 -- 172.21.15.42:0/2337586359 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f50bc014480 con 0x7f50c413c8c0 2024-09-16T03:43:34.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.977+0000 7f50ba7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:43:34.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.977+0000 7f50ba7fc640 1 -- 172.21.15.42:0/2337586359 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50bc014870 con 0x7f50c413c8c0 2024-09-16T03:43:34.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.977+0000 7f50ba7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:43:34.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.977+0000 7f50ba7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:43:34.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.977+0000 7f50ba7fc640 10 monclient: dump: 2024-09-16T03:43:34.003 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:43:34.003 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:43:34.004 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:43:34.004 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:43:34.004 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:43:34.004 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:43:34.004 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:43:34.004 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:43:34.004 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:43:34.004 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:43:34.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.977+0000 7f50bbfff640 4 set_mon_vals no callback set 2024-09-16T03:43:34.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.981+0000 7f50cc20b640 5 monclient: authenticate success, global_id 4589 2024-09-16T03:43:34.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.981+0000 7f50cc20b640 10 monclient: _renew_subs 2024-09-16T03:43:34.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.981+0000 7f50cc20b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:43:34.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.981+0000 7f50cc20b640 1 -- 172.21.15.42:0/2337586359 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50c413ee30 con 0x7f50c413c8c0 2024-09-16T03:43:34.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.981+0000 7f50cc20b640 10 client.4589.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:43:34.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.981+0000 7f50cc20b640 10 monclient: _renew_subs 2024-09-16T03:43:34.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.981+0000 7f50cc20b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:43:34.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.981+0000 7f50cc20b640 1 -- 172.21.15.42:0/2337586359 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f50c413c2e0 con 0x7f50c413c8c0 2024-09-16T03:43:34.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.981+0000 7f50cc20b640 1 librados: init done 2024-09-16T03:43:34.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.981+0000 7f50ba7fc640 1 -- 172.21.15.42:0/2337586359 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f50bc01b020 con 0x7f50c413c8c0 2024-09-16T03:43:34.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.981+0000 7f50cc20b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:43:34.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.981+0000 7f50cc20b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:43:34.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.981+0000 7f50ba7fc640 1 --2- 172.21.15.42:0/2337586359 >> v2:172.21.15.154:6812/254845342 conn(0x7f50a8037860 0x7f50a8039d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:43:34.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.981+0000 7f50cc20b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:43:34.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.981+0000 7f50ba7fc640 1 -- 172.21.15.42:0/2337586359 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 16949+0+0 (secure 0 0 0) 0x7f50bc04c980 con 0x7f50c413c8c0 2024-09-16T03:43:34.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.981+0000 7f50cc20b640 1 -- 172.21.15.42:0/2337586359 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f50c4133b80 con 0x7f50c413c8c0 2024-09-16T03:43:34.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.981+0000 7f50ba7fc640 10 client.4589.objecter ms_dispatch 0x7f50c4000ba0 osd_map(21..21 src has 1..21) 2024-09-16T03:43:34.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.981+0000 7f50ba7fc640 3 client.4589.objecter handle_osd_map got epochs [21,21] > 0 2024-09-16T03:43:34.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.981+0000 7f50ba7fc640 3 client.4589.objecter handle_osd_map decoding full epoch 21 2024-09-16T03:43:34.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.981+0000 7f50c9fa7640 1 --2- 172.21.15.42:0/2337586359 >> v2:172.21.15.154:6812/254845342 conn(0x7f50a8037860 0x7f50a8039d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:43:34.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.981+0000 7f50c9fa7640 10 monclient: get_auth_request con 0x7f50a8037860 auth_method 0 2024-09-16T03:43:34.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.981+0000 7f50ba7fc640 20 client.4589.objecter dump_active .. 0 homeless 2024-09-16T03:43:34.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.981+0000 7f50c9fa7640 1 --2- 172.21.15.42:0/2337586359 >> v2:172.21.15.154:6812/254845342 conn(0x7f50a8037860 0x7f50a8039d20 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f50c00011f0 tx=0x7f50c0007d40 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:43:34.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.981+0000 7f50ba7fc640 10 client.4589.objecter ms_handle_connect 0x7f50a8037860 2024-09-16T03:43:34.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.989+0000 7f50ba7fc640 1 -- 172.21.15.42:0/2337586359 <== mon.2 v2:172.21.15.42:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f50bc022030 con 0x7f50c413c8c0 2024-09-16T03:43:34.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.989+0000 7f50ba7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:43:34.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:33.989+0000 7f50ba7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:43:34.080 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: : oid=14 cursor=12:62a1935d:::14:head 2024-09-16T03:43:34.081 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: : seek to 12:62a1935d:::14:head 2024-09-16T03:43:34.081 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: > 12:62a1935d:::14:head -> 14 2024-09-16T03:43:34.081 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: : oid=0 cursor=12:6cac518f:::0:head 2024-09-16T03:43:34.081 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: : seek to 12:6cac518f:::0:head 2024-09-16T03:43:34.081 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: > 12:6cac518f:::0:head -> 0 2024-09-16T03:43:34.081 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: : oid=15 cursor=12:863748b0:::15:head 2024-09-16T03:43:34.081 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: : seek to 12:863748b0:::15:head 2024-09-16T03:43:34.081 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: > 12:863748b0:::15:head -> 15 2024-09-16T03:43:34.081 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: : oid=11 cursor=12:89d3ae78:::11:head 2024-09-16T03:43:34.081 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: : seek to 12:89d3ae78:::11:head 2024-09-16T03:43:34.081 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: > 12:89d3ae78:::11:head -> 11 2024-09-16T03:43:34.081 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: : oid=5 cursor=12:b29083e3:::5:head 2024-09-16T03:43:34.081 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: : seek to 12:b29083e3:::5:head 2024-09-16T03:43:34.082 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: > 12:b29083e3:::5:head -> 5 2024-09-16T03:43:34.082 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: : oid=8 cursor=12:bd63b0f1:::8:head 2024-09-16T03:43:34.082 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: : seek to 12:bd63b0f1:::8:head 2024-09-16T03:43:34.082 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: > 12:bd63b0f1:::8:head -> 8 2024-09-16T03:43:34.082 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: : oid=6 cursor=12:c4fdafeb:::6:head 2024-09-16T03:43:34.082 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: : seek to 12:c4fdafeb:::6:head 2024-09-16T03:43:34.082 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: > 12:c4fdafeb:::6:head -> 6 2024-09-16T03:43:34.082 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: : oid=3 cursor=12:cfc208b3:::3:head 2024-09-16T03:43:34.082 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: : seek to 12:cfc208b3:::3:head 2024-09-16T03:43:34.082 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: > 12:cfc208b3:::3:head -> 3 2024-09-16T03:43:34.082 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: : oid=4 cursor=12:d83876eb:::4:head 2024-09-16T03:43:34.082 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: : seek to 12:d83876eb:::4:head 2024-09-16T03:43:34.082 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: > 12:d83876eb:::4:head -> 4 2024-09-16T03:43:34.083 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: : oid=12 cursor=12:de5d7c5f:::12:head 2024-09-16T03:43:34.083 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: : seek to 12:de5d7c5f:::12:head 2024-09-16T03:43:34.083 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: > 12:de5d7c5f:::12:head -> 12 2024-09-16T03:43:34.083 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: : oid=9 cursor=12:e960b815:::9:head 2024-09-16T03:43:34.083 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: : seek to 12:e960b815:::9:head 2024-09-16T03:43:34.083 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: > 12:e960b815:::9:head -> 9 2024-09-16T03:43:34.083 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: : oid=2 cursor=12:f905c69b:::2:head 2024-09-16T03:43:34.083 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: : seek to 12:f905c69b:::2:head 2024-09-16T03:43:34.083 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: > 12:f905c69b:::2:head -> 2 2024-09-16T03:43:34.083 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: : seek to 12:02547ec2:::1:head 2024-09-16T03:43:34.083 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: : cursor()=12:02547ec2:::1:head expected=12:02547ec2:::1:head 2024-09-16T03:43:34.083 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: > 12:02547ec2:::1:head -> 1 2024-09-16T03:43:34.083 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: : entry=1 expected=1 2024-09-16T03:43:34.084 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: : seek to 12:52ea6a34:::10:head 2024-09-16T03:43:34.084 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: : cursor()=12:52ea6a34:::10:head expected=12:52ea6a34:::10:head 2024-09-16T03:43:34.084 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: > 12:52ea6a34:::10:head -> 10 2024-09-16T03:43:34.084 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: : entry=10 expected=10 2024-09-16T03:43:34.084 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: : seek to 12:f905c69b:::2:head 2024-09-16T03:43:34.084 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: : cursor()=12:f905c69b:::2:head expected=12:f905c69b:::2:head 2024-09-16T03:43:34.084 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: > 12:f905c69b:::2:head -> 2 2024-09-16T03:43:34.084 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: : entry=2 expected=2 2024-09-16T03:43:34.084 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: : seek to 12:cfc208b3:::3:head 2024-09-16T03:43:34.084 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: : cursor()=12:cfc208b3:::3:head expected=12:cfc208b3:::3:head 2024-09-16T03:43:34.084 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: > 12:cfc208b3:::3:head -> 3 2024-09-16T03:43:34.084 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: : entry=3 expected=3 2024-09-16T03:43:34.084 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: : seek to 12:bd63b0f1:::8:head 2024-09-16T03:43:34.084 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: : cursor()=12:bd63b0f1:::8:head expected=12:bd63b0f1:::8:head 2024-09-16T03:43:34.085 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: > 12:bd63b0f1:::8:head -> 8 2024-09-16T03:43:34.085 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: : entry=8 expected=8 2024-09-16T03:43:34.085 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: : seek to 12:b29083e3:::5:head 2024-09-16T03:43:34.085 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: : cursor()=12:b29083e3:::5:head expected=12:b29083e3:::5:head 2024-09-16T03:43:34.085 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: > 12:b29083e3:::5:head -> 5 2024-09-16T03:43:34.085 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: : entry=5 expected=5 2024-09-16T03:43:34.085 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: : seek to 12:89d3ae78:::11:head 2024-09-16T03:43:34.085 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: : cursor()=12:89d3ae78:::11:head expected=12:89d3ae78:::11:head 2024-09-16T03:43:34.085 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: > 12:89d3ae78:::11:head -> 11 2024-09-16T03:43:34.085 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: : entry=11 expected=11 2024-09-16T03:43:34.085 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: : seek to 12:863748b0:::15:head 2024-09-16T03:43:34.085 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: : cursor()=12:863748b0:::15:head expected=12:863748b0:::15:head 2024-09-16T03:43:34.085 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: > 12:863748b0:::15:head -> 15 2024-09-16T03:43:34.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.297+0000 7f50cc20b640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:43:34.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.297+0000 7f50cc20b640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:43:34.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.297+0000 7f50cc20b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:43:34.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.297+0000 7f50cc20b640 1 -- 172.21.15.42:0/2337586359 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f50c40a5750 con 0x7f50c413c8c0 2024-09-16T03:43:34.305 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":16,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:43:34.306 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.301+0000 7f50ba7fc640 1 -- 172.21.15.42:0/2337586359 <== mon.2 v2:172.21.15.42:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7f50bc010040 con 0x7f50c413c8c0 2024-09-16T03:43:34.306 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.301+0000 7f50ba7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:43:34.306 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.301+0000 7f50ba7fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:43:34.309 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.301+0000 7f50cc20b640 10 librados: watch_flush enter 2024-09-16T03:43:34.309 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.301+0000 7f50cc20b640 10 librados: watch_flush exit 2024-09-16T03:43:34.309 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.301+0000 7f50cc20b640 20 client.4589.objecter shutdown clearing up homeless session... 2024-09-16T03:43:34.309 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.301+0000 7f50cc20b640 10 client.4589.objecter successfully canceled tick 2024-09-16T03:43:34.309 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.305+0000 7f50cc20b640 1 -- 172.21.15.42:0/2337586359 >> v2:172.21.15.154:6812/254845342 conn(0x7f50a8037860 msgr2=0x7f50a8039d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:43:34.309 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.305+0000 7f50cc20b640 1 --2- 172.21.15.42:0/2337586359 >> v2:172.21.15.154:6812/254845342 conn(0x7f50a8037860 0x7f50a8039d20 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f50c00011f0 tx=0x7f50c0007d40 comp rx=0 tx=0).stop 2024-09-16T03:43:34.309 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.305+0000 7f50cc20b640 10 monclient: shutdown 2024-09-16T03:43:34.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.305+0000 7f50cc20b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:43:34.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.305+0000 7f50cc20b640 1 -- 172.21.15.42:0/2337586359 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f50c413c8c0 msgr2=0x7f50c41cfd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:43:34.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.305+0000 7f50cc20b640 1 --2- 172.21.15.42:0/2337586359 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f50c413c8c0 0x7f50c41cfd30 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f50bc00efc0 tx=0x7f50bc00c490 comp rx=0 tx=0).stop 2024-09-16T03:43:34.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.305+0000 7f50cc20b640 1 -- 172.21.15.42:0/2337586359 shutdown_connections 2024-09-16T03:43:34.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.305+0000 7f50cc20b640 1 --2- 172.21.15.42:0/2337586359 >> v2:172.21.15.154:6812/254845342 conn(0x7f50a8037860 0x7f50a8039d20 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:43:34.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.305+0000 7f50cc20b640 1 --2- 172.21.15.42:0/2337586359 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f50c413c8c0 0x7f50c41cfd30 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:43:34.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.305+0000 7f50cc20b640 1 --2- 172.21.15.42:0/2337586359 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f50c40a3d90 0x7f50c41cf7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:43:34.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.305+0000 7f50cc20b640 1 --2- 172.21.15.42:0/2337586359 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f50c40a39b0 0x7f50c41cf2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:43:34.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.305+0000 7f50cc20b640 1 -- 172.21.15.42:0/2337586359 >> 172.21.15.42:0/2337586359 conn(0x7f50c40a9fb0 msgr2=0x7f50c4131980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:43:34.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.305+0000 7f50cc20b640 1 -- 172.21.15.42:0/2337586359 shutdown_connections 2024-09-16T03:43:34.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.305+0000 7f50cc20b640 1 -- 172.21.15.42:0/2337586359 wait complete. 2024-09-16T03:43:34.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.305+0000 7f50cc20b640 1 librados: shutdown 2024-09-16T03:43:34.343 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T03:43:34.343 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T03:43:34.344 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T03:43:34.344 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T03:43:34.428 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: : entG pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:43:34.428 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:33.263+0000 7f91004a6640 10 monclient: get_auth_request con 0x5607e06f6d30 auth_method 0 2024-09-16T03:43:34.428 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:33.267+0000 7f90fd4a0640 10 client.14182.objecter ms_handle_connect 0x5607e06f6d30 2024-09-16T03:43:34.428 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:33.267+0000 7f91004a6640 1 --2- 172.21.15.154:0/332177798 >> v2:172.21.15.42:6800/4217863575 conn(0x5607e06f6d30 0x5607e06f9120 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:43:34.428 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.383+0000 7f90e7fff640 10 monclient: tick 2024-09-16T03:43:34.429 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.383+0000 7f90e7fff640 10 monclient: _check_auth_tickets 2024-09-16T03:43:34.429 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.383+0000 7f90e7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:43:34.429 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.383+0000 7f90e7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:43:34.429 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.399+0000 7f90fd4a0640 1 -- 172.21.15.154:0/332177798 <== osd.1 v2:172.21.15.42:6800/4217863575 1 ==== osd_map(22..22 src has 1..22) ==== 6583+0+0 (crc 0 0 0) 0x7f90f80136d0 con 0x5607e06f6d30 2024-09-16T03:43:34.429 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.399+0000 7f90fd4a0640 10 client.14182.objecter ms_dispatch 0x5607e05160c0 osd_map(22..22 src has 1..22) 2024-09-16T03:43:34.429 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.399+0000 7f90fd4a0640 3 client.14182.objecter handle_osd_map got epochs [22,22] > 21 2024-09-16T03:43:34.429 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.399+0000 7f90fd4a0640 3 client.14182.objecter handle_osd_map decoding incremental epoch 22 2024-09-16T03:43:34.429 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.399+0000 7f90fd4a0640 10 client.14182.objecter checking op 2 2024-09-16T03:43:34.429 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.399+0000 7f90fd4a0640 20 client.14182.objecter _calc_target epoch 22 base exist @31 precalc_pgid 0 pgid 0.0 is_write 2024-09-16T03:43:34.429 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.399+0000 7f90fd4a0640 20 client.14182.objecter _calc_target target exist @31 -> pgid 31.d3a59201 2024-09-16T03:43:34.429 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.399+0000 7f90fd4a0640 20 client.14182.objecter dump_active .. 0 homeless 2024-09-16T03:43:34.429 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.399+0000 7f90fd4a0640 20 client.14182.objecter 2 31.d3a59201 osd.1 exist [writefull 0~5 in=5b] 2024-09-16T03:43:34.429 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.427+0000 7f91004a6640 1 -- 172.21.15.154:0/332177798 <== osd.1 v2:172.21.15.42:6800/4217863575 2 ==== osd_op_reply(2 exist [writefull 0~5] v22'1 uv1 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f90f80138d0 con 0x5607e06f6d30 2024-09-16T03:43:34.430 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.427+0000 7f91004a6640 10 client.14182.objecter ms_dispatch 0x5607e05160c0 osd_op_reply(2 exist [writefull 0~5] v22'1 uv1 ondisk = 0) 2024-09-16T03:43:34.430 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.427+0000 7f91004a6640 10 client.14182.objecter in handle_osd_op_reply 2024-09-16T03:43:34.430 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.427+0000 7f91004a6640 7 client.14182.objecter handle_osd_op_reply 2 ondisk uv 1 in 31.1 attempt 0 2024-09-16T03:43:34.430 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.427+0000 7f91004a6640 10 client.14182.objecter op 0 rval 0 len 0 2024-09-16T03:43:34.430 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.427+0000 7f91004a6640 15 client.14182.objecter handle_osd_op_reply completed tid 2 2024-09-16T03:43:34.430 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.427+0000 7f91004a6640 15 client.14182.objecter _finish_op 2 2024-09-16T03:43:34.430 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.427+0000 7f91004a6640 20 client.14182.objecter put_session s=0x5607e0653f50 osd=1 4 2024-09-16T03:43:34.430 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.427+0000 7f91004a6640 15 client.14182.objecter _session_op_remove 1 2 2024-09-16T03:43:34.431 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.427+0000 7f91004a6640 5 client.14182.objecter 0 in flight 2024-09-16T03:43:34.431 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: [ RUN ] AsioRados.AsyncReadCallback 2024-09-16T03:43:34.431 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.427+0000 7f9102191880 10 librados: Objecter returned from writefull r=0 2024-09-16T03:43:34.435 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.427+0000 7f9102191880 10 client.14182.objecter _op_submit op 0x5607e06e5280 2024-09-16T03:43:34.435 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.427+0000 7f9102191880 20 client.14182.objecter _calc_target epoch 22 base exist @31 precalc_pgid 0 pgid 0.0 is_read 2024-09-16T03:43:34.435 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.427+0000 7f9102191880 20 client.14182.objecter _calc_target target exist @31 -> pgid 31.d3a59201 2024-09-16T03:43:34.435 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.427+0000 7f9102191880 10 client.14182.objecter _calc_target raw pgid 31.d3a59201 -> actual 31.1 acting [1,4] primary 1 2024-09-16T03:43:34.435 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.427+0000 7f9102191880 20 client.14182.objecter _get_session s=0x5607e0653f50 osd=1 3 2024-09-16T03:43:34.435 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.427+0000 7f9102191880 10 client.14182.objecter _op_submit oid exist '@31' '@31' [read 0~256] tid 3 osd.1 2024-09-16T03:43:34.435 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.427+0000 7f9102191880 20 client.14182.objecter get_session s=0x5607e0653f50 osd=1 3 2024-09-16T03:43:34.436 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.427+0000 7f9102191880 15 client.14182.objecter _session_op_assign 1 3 2024-09-16T03:43:34.436 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.427+0000 7f9102191880 15 client.14182.objecter _send_op 3 to 31.1 on osd.1 2024-09-16T03:43:34.436 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.427+0000 7f9102191880 1 -- 172.21.15.154:0/332177798 --> v2:172.21.15.42:6800/4217863575 -- osd_op(unknown.0.0:3 31.1 31:8049a5cb:::exist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22) -- 0x5607e0651e40 con 0x5607e06f6d30 2024-09-16T03:43:34.436 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.427+0000 7f9102191880 20 client.14182.objecter put_session s=0x5607e0653f50 osd=1 4 2024-09-16T03:43:34.436 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.427+0000 7f9102191880 5 client.14182.objecter 1 in flight 2024-09-16T03:43:34.436 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.427+0000 7f9102191880 10 client.14182.objecter _op_submit op 0x5607e0652230 2024-09-16T03:43:34.436 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.427+0000 7f9102191880 20 client.14182.objecter _calc_target epoch 22 base noexist @31 precalc_pgid 0 pgid 0.0 is_read 2024-09-16T03:43:34.436 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.427+0000 7f9102191880 20 client.14182.objecter _calc_target target noexist @31 -> pgid 31.a7bbaa1 2024-09-16T03:43:34.436 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.427+0000 7f9102191880 10 client.14182.objecter _calc_target raw pgid 31.a7bbaa1 -> actual 31.1 acting [1,4] primary 1 2024-09-16T03:43:34.436 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.427+0000 7f9102191880 20 client.14182.objecter _get_session s=0x5607e0653f50 osd=1 4 2024-09-16T03:43:34.436 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.427+0000 7f9102191880 10 client.14182.objecter _op_submit oid noexist '@31' '@31' [read 0~256] tid 4 osd.1 2024-09-16T03:43:34.436 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.427+0000 7f9102191880 20 client.14182.objecter get_session s=0x5607e0653f50 osd=1 4 2024-09-16T03:43:34.436 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.427+0000 7f9102191880 15 client.14182.objecter _session_op_assign 1 4 2024-09-16T03:43:34.437 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.427+0000 7f9102191880 15 client.14182.objecter _send_op 4 to 31.1 on osd.1 2024-09-16T03:43:34.437 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.427+0000 7f9102191880 1 -- 172.21.15.154:0/332177798 --> v2:172.21.15.42:6800/4217863575 -- osd_op(unknown.0.0:4 31.1 31:855dde50:::noexist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22) -- 0x5607e06526e0 con 0x5607e06f6d30 2024-09-16T03:43:34.437 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.427+0000 7f9102191880 20 client.14182.objecter put_session s=0x5607e0653f50 osd=1 5 2024-09-16T03:43:34.437 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.427+0000 7f9102191880 5 client.14182.objecter 2 in flight 2024-09-16T03:43:34.437 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.435+0000 7f91004a6640 1 -- 172.21.15.154:0/332177798 <== osd.1 v2:172.21.15.42:6800/4217863575 3 ==== osd_op_reply(3 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) ==== 149+0+5 (crc 0 0 0) 0x7f90f80138d0 con 0x5607e06f6d30 2024-09-16T03:43:34.437 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.435+0000 7f91004a6640 10 client.14182.objecter ms_dispatch 0x5607e05160c0 osd_op_reply(3 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) 2024-09-16T03:43:34.437 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.435+0000 7f91004a6640 10 client.14182.objecter in handle_osd_op_reply 2024-09-16T03:43:34.438 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.435+0000 7f91004a6640 7 client.14182.objecter handle_osd_op_reply 3 ondisk uv 1 in 31.1 attempt 0 2024-09-16T03:43:34.438 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.435+0000 7f91004a6640 10 client.14182.objecter op 0 rval 0 len 5 2024-09-16T03:43:34.438 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.435+0000 7f91004a6640 15 client.14182.objecter handle_osd_op_reply completed tid 3 2024-09-16T03:43:34.438 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.435+0000 7f91004a6640 15 client.14182.objecter _finish_op 3 2024-09-16T03:43:34.438 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.435+0000 7f91004a6640 20 client.14182.objecter put_session s=0x5607e0653f50 osd=1 5 2024-09-16T03:43:34.438 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.435+0000 7f91004a6640 15 client.14182.objecter _session_op_remove 1 3 2024-09-16T03:43:34.438 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.435+0000 7f91004a6640 5 client.14182.objecter 1 in flight 2024-09-16T03:43:34.438 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.435+0000 7f91004a6640 1 -- 172.21.15.154:0/332177798 <== osd.1 v2:172.21.15.42:6800/4217863575 4 ==== osd_op_reply(4 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 151+0+0 (crc 0 0 0) 0x7f90f80138d0 con 0x5607e06f6d30 2024-09-16T03:43:34.438 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.435+0000 7f91004a6640 10 client.14182.objecter ms_dispatch 0x5607e05160c0 osd_op_reply(4 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) 2024-09-16T03:43:34.438 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.435+0000 7f91004a6640 10 client.14182.objecter in handle_osd_op_reply 2024-09-16T03:43:34.439 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.435+0000 7f91004a6640 7 client.14182.objecter handle_osd_op_reply 4 ondisk uv 0 in 31.1 attempt 0 2024-09-16T03:43:34.439 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.435+0000 7f91004a6640 10 client.14182.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-09-16T03:43:34.439 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.435+0000 7f91004a6640 10 client.14182.objecter op 0 rval 0 len 0 2024-09-16T03:43:34.440 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.435+0000 7f91004a6640 15 client.14182.objecter handle_osd_op_reply completed tid 4 2024-09-16T03:43:34.440 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.435+0000 7f91004a6640 15 client.14182.objecter _finish_op 4 2024-09-16T03:43:34.440 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.435+0000 7f91004a6640 20 client.14182.objecter put_session s=0x5607e0653f50 osd=1 4 2024-09-16T03:43:34.440 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.435+0000 7f91004a6640 15 client.14182.objecter _session_op_remove 1 4 2024-09-16T03:43:34.440 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.435+0000 7f91004a6640 5 client.14182.objecter 0 in flight 2024-09-16T03:43:34.440 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: [ OK ] AsioRados.AsyncReadCallback (8 ms) 2024-09-16T03:43:34.440 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: [ RUN ] AsioRados.AsyncReadFuture 2024-09-16T03:43:34.440 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.435+0000 7f9102191880 10 client.14182.objecter _op_submit op 0x5607e0652230 2024-09-16T03:43:34.440 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.435+0000 7f9102191880 20 client.14182.objecter _calc_target epoch 22 base exist @31 precalc_pgid 0 pgid 0.0 is_read 2024-09-16T03:43:34.440 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.435+0000 7f9102191880 20 client.14182.objecter _calc_target target exist @31 -> pgid 31.d3a59201 2024-09-16T03:43:34.440 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.435+0000 7f9102191880 10 client.14182.objecter _calc_target raw pgid 31.d3a59201 -> actual 31.1 acting [1,4] primary 1 2024-09-16T03:43:34.440 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.435+0000 7f9102191880 20 client.14182.objecter _get_session s=0x5607e0653f50 osd=1 3 2024-09-16T03:43:34.441 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.435+0000 7f9102191880 10 client.14182.objecter _op_submit oid exist '@31' '@31' [read 0~256] tid 5 osd.1 2024-09-16T03:43:34.441 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.435+0000 7f9102191880 20 client.14182.objecter get_session s=0x5607e0653f50 osd=1 3 2024-09-16T03:43:34.441 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.435+0000 7f9102191880 15 client.14182.objecter _session_op_assign 1 5 2024-09-16T03:43:34.441 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.435+0000 7f9102191880 15 client.14182.objecter _send_op 5 to 31.1 on osd.1 2024-09-16T03:43:34.441 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.435+0000 7f9102191880 1 -- 172.21.15.154:0/332177798 --> v2:172.21.15.42:6800/4217863575 -- osd_op(unknown.0.0 handler_error: Running main() from gmock_main.cc 2024-09-16T03:43:34.441 INFO:tasks.workunit.client.0.smithi154.stdout: handler_error: [==========] Running 1 test from 1 test suite. 2024-09-16T03:43:34.441 INFO:tasks.workunit.client.0.smithi154.stdout: handler_error: [----------] Global test environment set-up. 2024-09-16T03:43:34.441 INFO:tasks.workunit.client.0.smithi154.stdout: handler_error: [----------] 1 test from neocls_handler_error 2024-09-16T03:43:34.441 INFO:tasks.workunit.client.0.smithi154.stdout: handler_error: [ RUN ] neocls_handler_error.test_handler_error 2024-09-16T03:43:34.441 INFO:tasks.workunit.client.0.smithi154.stdout: handler_error: [ OK ] neocls_handler_error.test_handler_error (4669 ms) 2024-09-16T03:43:34.442 INFO:tasks.workunit.client.0.smithi154.stdout: handler_error: [----------] 1 test from neocls_handler_error (4669 ms total) 2024-09-16T03:43:34.442 INFO:tasks.workunit.client.0.smithi154.stdout: handler_error: 2024-09-16T03:43:34.442 INFO:tasks.workunit.client.0.smithi154.stdout: handler_error: [----------] Global test environment tear-down 2024-09-16T03:43:34.442 INFO:tasks.workunit.client.0.smithi154.stdout: handler_error: [==========] 1 test from 1 test suite ran. (4669 ms total) 2024-09-16T03:43:34.442 INFO:tasks.workunit.client.0.smithi154.stdout: handler_error: [ PASSED ] 1 test. 2024-09-16T03:43:34.444 INFO:tasks.workunit.client.0.smithi154.stdout::5 31.1 31:8049a5cb:::exist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22) -- 0x5607e06e5280 con 0x5607e06f6d30 2024-09-16T03:43:34.444 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.435+0000 7f9102191880 20 client.14182.objecter put_session s=0x5607e0653f50 osd=1 4 2024-09-16T03:43:34.444 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.435+0000 7f9102191880 5 client.14182.objecter 1 in flight 2024-09-16T03:43:34.444 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.435+0000 7f9102191880 10 client.14182.objecter _op_submit op 0x5607e06f9a00 2024-09-16T03:43:34.444 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.435+0000 7f9102191880 20 client.14182.objecter _calc_target epoch 22 base noexist @31 precalc_pgid 0 pgid 0.0 is_read 2024-09-16T03:43:34.444 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.435+0000 7f9102191880 20 client.14182.objecter _calc_target target noexist @31 -> pgid 31.a7bbaa1 2024-09-16T03:43:34.444 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.435+0000 7f9102191880 10 client.14182.objecter _calc_target raw pgid 31.a7bbaa1 -> actual 31.1 acting [1,4] primary 1 2024-09-16T03:43:34.444 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.435+0000 7f9102191880 20 client.14182.objecter _get_session s=0x5607e0653f50 osd=1 4 2024-09-16T03:43:34.444 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.435+0000 7f9102191880 10 client.14182.objecter _op_submit oid noexist '@31' '@31' [read 0~256] tid 6 osd.1 2024-09-16T03:43:34.444 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.435+0000 7f9102191880 20 client.14182.objecter get_session s=0x5607e0653f50 osd=1 4 2024-09-16T03:43:34.444 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.435+0000 7f9102191880 15 client.14182.objecter _session_op_assign 1 6 2024-09-16T03:43:34.444 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.435+0000 7f9102191880 15 client.14182.objecter _send_op 6 to 31.1 on osd.1 2024-09-16T03:43:34.445 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.435+0000 7f9102191880 1 -- 172.21.15.154:0/332177798 --> v2:172.21.15.42:6800/4217863575 -- osd_op(unknown.0.0:6 31.1 31:855dde50:::noexist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22) -- 0x5607e06f9eb0 con 0x5607e06f6d30 2024-09-16T03:43:34.445 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.435+0000 7f9102191880 20 client.14182.objecter put_session s=0x5607e0653f50 osd=1 5 2024-09-16T03:43:34.445 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.435+0000 7f9102191880 5 client.14182.objecter 2 in flight 2024-09-16T03:43:34.445 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: [ OK ] AsioRados.AsyncReadFuture (8 ms) 2024-09-16T03:43:34.445 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: [ RUN ] AsioRados.AsyncReadYield 2024-09-16T03:43:34.445 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.443+0000 7f91004a6640 1 -- 172.21.15.154:0/332177798 <== osd.1 v2:172.21.15.42:6800/4217863575 5 ==== osd_op_reply(5 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) ==== 149+0+5 (crc 0 0 0) 0x7f90f80138d0 con 0x5607e06f6d30 2024-09-16T03:43:34.445 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.443+0000 7f91004a6640 10 client.14182.objecter ms_dispatch 0x5607e05160c0 osd_op_reply(5 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) 2024-09-16T03:43:34.445 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.443+0000 7f91004a6640 10 client.14182.objecter in handle_osd_op_reply 2024-09-16T03:43:34.445 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.443+0000 7f91004a6640 7 client.14182.objecter handle_osd_op_reply 5 ondisk uv 1 in 31.1 attempt 0 2024-09-16T03:43:34.445 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.443+0000 7f91004a6640 10 client.14182.objecter op 0 rval 0 len 5 2024-09-16T03:43:34.445 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.443+0000 7f91004a6640 15 client.14182.objecter handle_osd_op_reply completed tid 5 2024-09-16T03:43:34.445 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.443+0000 7f91004a6640 15 client.14182.objecter _finish_op 5 2024-09-16T03:43:34.445 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.443+0000 7f91004a6640 20 client.14182.objecter put_session s=0x5607e0653f50 osd=1 5 2024-09-16T03:43:34.446 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.443+0000 7f91004a6640 15 client.14182.objecter _session_op_remove 1 5 2024-09-16T03:43:34.446 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.443+0000 7f91004a6640 5 client.14182.objecter 1 in flight 2024-09-16T03:43:34.446 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.443+0000 7f91004a6640 1 -- 172.21.15.154:0/332177798 <== osd.1 v2:172.21.15.42:6800/4217863575 6 ==== osd_op_reply(6 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 151+0+0 (crc 0 0 0) 0x7f90f80138d0 con 0x5607e06f6d30 2024-09-16T03:43:34.446 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.443+0000 7f91004a6640 10 client.14182.objecter ms_dispatch 0x5607e05160c0 osd_op_reply(6 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) 2024-09-16T03:43:34.446 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.443+0000 7f91004a6640 10 client.14182.objecter in handle_osd_op_reply 2024-09-16T03:43:34.446 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.443+0000 7f91004a6640 7 client.14182.objecter handle_osd_op_reply 6 ondisk uv 0 in 31.1 attempt 0 2024-09-16T03:43:34.446 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.443+0000 7f91004a6640 10 client.14182.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-09-16T03:43:34.446 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.443+0000 7f91004a6640 10 client.14182.objecter op 0 rval 0 len 0 2024-09-16T03:43:34.446 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.443+0000 7f91004a6640 15 client.14182.objecter handle_osd_op_reply completed tid 6 2024-09-16T03:43:34.446 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.443+0000 7f91004a6640 15 client.14182.objecter _finish_op 6 2024-09-16T03:43:34.446 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.443+0000 7f91004a6640 20 client.14182.objecter put_session s=0x5607e0653f50 osd=1 4 2024-09-16T03:43:34.446 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.443+0000 7f91004a6640 15 client.14182.objecter _session_op_remove 1 6 2024-09-16T03:43:34.446 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.443+0000 7f91004a6640 5 client.14182.objecter 0 in flight 2024-09-16T03:43:34.446 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.443+0000 7f9102191880 10 client.14182.objecter _op_submit op 0x5607e06f9a00 2024-09-16T03:43:34.447 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.443+0000 7f9102191880 20 client.14182.objecter _calc_target epoch 22 base exist @31 precalc_pgid 0 pgid 0.0 is_read 2024-09-16T03:43:34.447 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.443+0000 7f9102191880 20 client.14182.objecter _calc_target target exist @31 -> pgid 31.d3a59201 2024-09-16T03:43:34.447 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.443+0000 7f9102191880 10 client.14182.objecter _calc_target raw pgid 31.d3a59201 -> actual 31.1 acting [1,4] primary 1 2024-09-16T03:43:34.447 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.443+0000 7f9102191880 20 client.14182.objecter _get_session s=0x5607e0653f50 osd=1 3 2024-09-16T03:43:34.447 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.443+0000 7f9102191880 10 client.14182.objecter _op_submit oid exist '@31' '@31' [read 0~256] tid 7 osd.1 2024-09-16T03:43:34.447 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.443+0000 7f9102191880 20 client.14182.objecter get_session s=0x5607e0653f50 osd=1 3 2024-09-16T03:43:34.447 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.443+0000 7f9102191880 15 client.14182.objecter _session_op_assign 1 7 2024-09-16T03:43:34.447 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.443+0000 7f9102191880 15 client.14182.objecter _send_op 7 to 31.1 on osd.1 2024-09-16T03:43:34.447 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.443+0000 7f9102191880 1 -- 172.21.15.154:0/332177798 --> v2:172.21.15.42:6800/4217863575 -- osd_op(unknown.0.0:7 31.1 31:8049a5cb:::exist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22) -- 0x5607e0652230 con 0x5607e06f6d30 2024-09-16T03:43:34.447 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.443+0000 7f9102191880 20 client.14182.objecter put_session s=0x5607e0653f50 osd=1 4 2024-09-16T03:43:34.447 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.443+0000 7f9102191880 5 client.14182.objecter 1 in flight 2024-09-16T03:43:34.447 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.443+0000 7f9102191880 10 client.14182.objecter _op_submit op 0x5607e073a8e0 2024-09-16T03:43:34.447 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.443+0000 7f9102191880 20 client.14182.objecter _calc_target epoch 22 base noexist @31 precalc_pgid 0 pgid 0.0 is_read 2024-09-16T03:43:34.447 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.443+0000 7f9102191880 20 client.14182.objecter _calc_target target noexist @31 -> pgid 31.a7bbaa1 2024-09-16T03:43:34.448 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.443+0000 7f9102191880 10 client.14182.objecter _calc_target raw pgid 31.a7bbaa1 -> actual 31.1 acting [1,4] primary 1 2024-09-16T03:43:34.448 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.443+0000 7f9102191880 20 client.14182.objecter _get_session s=0x5607e0653f50 osd=1 4 2024-09-16T03:43:34.448 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.443+0000 7f9102191880 10 client.14182.objecter _op_submit oid noexist '@31' '@31' [read 0~256] tid 8 osd.1 2024-09-16T03:43:34.448 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.443+0000 7f9102191880 20 client.14182.objecter get_session s=0x5607e0653f50 osd=1 4 2024-09-16T03:43:34.448 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.443+0000 7f9102191880 15 client.14182.objecter _session_op_assign 1 8 2024-09-16T03:43:34.448 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.443+0000 7f9102191880 15 client.14182.objecter _send_op 8 to 31.1 on osd.1 2024-09-16T03:43:34.449 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.443+0000 7f9102191880 1 -- 172.21.15.154:0/332177798 --> v2:172.21.15.42:6800/4217863575 -- osd_op(unknown.0.0:8 31.1 31:855dde50:::noexist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22) -- 0x5607e073ad90 con 0x5607e06f6d30 2024-09-16T03:43:34.449 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.443+0000 7f9102191880 20 client.14182.objecter put_session s=0x5607e0653f50 osd=1 5 2024-09-16T03:43:34.449 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.443+0000 7f9102191880 5 client.14182.objecter 2 in flight 2024-09-16T03:43:34.449 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.443+0000 7f91004a6640 1 -- 172.21.15.154:0/332177798 <== osd.1 v2:172.21.15.42:6800/4217863575 7 ==== osd_op_reply(7 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) ==== 149+0+5 (crc 0 0 0) 0x7f90f80138d0 con 0x5607e06f6d30 2024-09-16T03:43:34.449 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.443+0000 7f91004a6640 10 client.14182.objecter ms_dispatch 0x5607e05160c0 osd_op_reply(7 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) 2024-09-16T03:43:34.449 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.443+0000 7f91004a6640 10 client.14182.objecter in handle_osd_op_reply 2024-09-16T03:43:34.449 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.443+0000 7f91004a6640 7 client.14182.objecter handle_osd_op_reply 7 ondisk uv 1 in 31.1 attempt 0 2024-09-16T03:43:34.449 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.443+0000 7f91004a6640 10 client.14182.objecter op 0 rval 0 len 5 2024-09-16T03:43:34.449 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.443+0000 7f91004a6640 15 client.14182.objecter handle_osd_op_reply completed tid 7 2024-09-16T03:43:34.449 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.443+0000 7f91004a6640 15 client.14182.objecter _finish_op 7 2024-09-16T03:43:34.449 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.443+0000 7f91004a6640 20 client.14182.objecter put_session s=0x5607e0653f50 osd=1 5 2024-09-16T03:43:34.449 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.443+0000 7f91004a6640 15 client.14182.objecter _session_op_remove 1 7 2024-09-16T03:43:34.449 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.443+0000 7f91004a6640 5 client.14182.objecter 1 in flight 2024-09-16T03:43:34.449 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: [ OK ] AsioRados.AsyncReadYield (1 ms) 2024-09-16T03:43:34.450 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteCallback 2024-09-16T03:43:34.450 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.443+0000 7f91004a6640 1 -- 172.21.15.154:0/332177798 <== osd.1 v2:172.21.15.42:6800/4217863575 8 ==== osd_op_reply(8 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 151+0+0 (crc 0 0 0) 0x7f90f80138d0 con 0x5607e06f6d30 2024-09-16T03:43:34.450 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.443+0000 7f91004a6640 10 client.14182.objecter ms_dispatch 0x5607e05160c0 osd_op_reply(8 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) 2024-09-16T03:43:34.450 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.443+0000 7f91004a6640 10 client.14182.objecter in handle_osd_op_reply 2024-09-16T03:43:34.450 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.443+0000 7f91004a6640 7 client.14182.objecter handle_osd_op_reply 8 ondisk uv 0 in 31.1 attempt 0 2024-09-16T03:43:34.450 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.443+0000 7f91004a6640 10 client.14182.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-09-16T03:43:34.450 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.443+0000 7f91004a6640 10 client.14182.objecter op 0 rval 0 len 0 2024-09-16T03:43:34.450 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.443+0000 7f91004a6640 15 client.14182.objecter handle_osd_op_reply completed tid 8 2024-09-16T03:43:34.450 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.443+0000 7f91004a6640 15 client.14182.objecter _finish_op 8 2024-09-16T03:43:34.450 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.443+0000 7f91004a6640 20 client.14182.objecter put_session s=0x5607e0653f50 osd=1 4 2024-09-16T03:43:34.450 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.443+0000 7f91004a6640 15 client.14182.objecter _session_op_remove 1 8 2024-09-16T03:43:34.450 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.443+0000 7f91004a6640 5 client.14182.objecter 0 in flight 2024-09-16T03:43:34.450 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.443+0000 7f9102191880 20 librados: aio_write api_cmd_pp: Running main() from gmock_main.cc 2024-09-16T03:43:34.450 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd_pp: [==========] Running 3 tests from 1 test suite. 2024-09-16T03:43:34.451 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd_pp: [----------] Global test environment set-up. 2024-09-16T03:43:34.451 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd_pp: [----------] 3 tests from LibRadosCmd 2024-09-16T03:43:34.451 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.MonDescribePP 2024-09-16T03:43:34.451 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd_pp: [ OK ] LibRadosCmd.MonDescribePP (1547 ms) 2024-09-16T03:43:34.451 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.OSDCmdPP 2024-09-16T03:43:34.451 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd_pp: [ OK ] LibRadosCmd.OSDCmdPP (23 ms) 2024-09-16T03:43:34.451 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.PGCmdPP 2024-09-16T03:43:34.451 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd_pp: [ OK ] LibRadosCmd.PGCmdPP (3197 ms) 2024-09-16T03:43:34.451 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd_pp: [----------] 3 tests from LibRadosCmd (4767 ms total) 2024-09-16T03:43:34.451 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd_pp: 2024-09-16T03:43:34.451 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd_pp: [----------] Global test environment tear-down 2024-09-16T03:43:34.451 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd_pp: [==========] 3 tests from 1 test suite ran. (4767 ms total) 2024-09-16T03:43:34.451 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd_pp: [ PASSED ] 3 tests. 2024-09-16T03:43:34.455 INFO:tasks.workunit.client.0.smithi154.stdout: api_cls_remote_reads: [==========] Running 1 test from 1 test suite. 2024-09-16T03:43:34.455 INFO:tasks.workunit.client.0.smithi154.stdout: api_cls_remote_reads: [----------] Global test environment set-up. 2024-09-16T03:43:34.455 INFO:tasks.workunit.client.0.smithi154.stdout: api_cls_remote_reads: [----------] 1 test from ClsTestRemoteReads 2024-09-16T03:43:34.455 INFO:tasks.workunit.client.0.smithi154.stdout: api_cls_remote_reads: [ RUN ] ClsTestRemoteReads.TestGather 2024-09-16T03:43:34.455 INFO:tasks.workunit.client.0.smithi154.stdout: api_cls_remote_reads: [ OK ] ClsTestRemoteReads.TestGather (4706 ms) 2024-09-16T03:43:34.455 INFO:tasks.workunit.client.0.smithi154.stdout: api_cls_remote_reads: [----------] 1 test from ClsTestRemoteReads (4706 ms total) 2024-09-16T03:43:34.455 INFO:tasks.workunit.client.0.smithi154.stdout: api_cls_remote_reads: 2024-09-16T03:43:34.455 INFO:tasks.workunit.client.0.smithi154.stdout: api_cls_remote_reads: [----------] Global test environment tear-down 2024-09-16T03:43:34.455 INFO:tasks.workunit.client.0.smithi154.stdout: api_cls_remote_reads: [==========] 1 test from 1 test suite ran. (4706 ms total) 2024-09-16T03:43:34.455 INFO:tasks.workunit.client.0.smithi154.stdout: api_cls_remote_reads: [ PASSED ] 1 test. 2024-09-16T03:43:34.472 INFO:tasks.workunit.client.0.smithi154.stdout:exist 0~5 snapc=0=[] snap_seq=head 2024-09-16T03:43:34.473 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.443+0000 7f9102191880 20 librados: queue_aio_write 0x5607e050b7a0 completion 0x5607e04d7450 write_seq 1 2024-09-16T03:43:34.473 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.443+0000 7f9102191880 10 client.14182.objecter _op_submit op 0x5607e06f9a00 2024-09-16T03:43:34.473 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.443+0000 7f9102191880 20 client.14182.objecter _calc_target epoch 22 base exist @31 precalc_pgid 0 pgid 0.0 is_write 2024-09-16T03:43:34.473 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.443+0000 7f9102191880 20 client.14182.objecter _calc_target target exist @31 -> pgid 31.d3a59201 2024-09-16T03:43:34.473 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.443+0000 7f9102191880 10 client.14182.objecter _calc_target raw pgid 31.d3a59201 -> actual 31.1 acting [1,4] primary 1 2024-09-16T03:43:34.473 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.443+0000 7f9102191880 20 client.14182.objecter _get_session s=0x5607e0653f50 osd=1 3 2024-09-16T03:43:34.473 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.443+0000 7f9102191880 10 client.14182.objecter _op_submit oid exist '@31' '@31' [write 0~5 in=5b] tid 9 osd.1 2024-09-16T03:43:34.473 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.443+0000 7f9102191880 20 client.14182.objecter get_session s=0x5607e0653f50 osd=1 3 2024-09-16T03:43:34.473 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.443+0000 7f9102191880 15 client.14182.objecter _session_op_assign 1 9 2024-09-16T03:43:34.473 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.443+0000 7f9102191880 15 client.14182.objecter _send_op 9 to 31.1 on osd.1 2024-09-16T03:43:34.473 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.443+0000 7f9102191880 1 -- 172.21.15.154:0/332177798 --> v2:172.21.15.42:6800/4217863575 -- osd_op(unknown.0.0:9 31.1 31:8049a5cb:::exist:head [write 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22) -- 0x5607e06fb260 con 0x5607e06f6d30 2024-09-16T03:43:34.473 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.443+0000 7f9102191880 20 client.14182.objecter put_session s=0x5607e0653f50 osd=1 4 2024-09-16T03:43:34.473 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.443+0000 7f9102191880 5 client.14182.objecter 1 in flight 2024-09-16T03:43:34.473 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.443+0000 7f9102191880 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=1 2024-09-16T03:43:34.474 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.471+0000 7f91004a6640 1 -- 172.21.15.154:0/332177798 <== osd.1 v2:172.21.15.42:6800/4217863575 9 ==== osd_op_reply(9 exist [write 0~5] v22'2 uv2 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f90f80138d0 con 0x5607e06f6d30 2024-09-16T03:43:34.474 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: [ OK ] AsioRados.AsyncWriteCallback (28 ms) 2024-09-16T03:43:34.474 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteFuture 2024-09-16T03:43:34.474 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.471+0000 7f91004a6640 10 client.14182.objecter ms_dispatch 0x5607e05160c0 osd_op_reply(9 exist [write 0~5] v22'2 uv2 ondisk = 0) 2024-09-16T03:43:34.474 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.471+0000 7f91004a6640 10 client.14182.objecter in handle_osd_op_reply 2024-09-16T03:43:34.474 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.471+0000 7f91004a6640 7 client.14182.objecter handle_osd_op_reply 9 ondisk uv 2 in 31.1 attempt 0 2024-09-16T03:43:34.474 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.471+0000 7f91004a6640 10 client.14182.objecter op 0 rval 0 len 0 2024-09-16T03:43:34.474 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.471+0000 7f91004a6640 15 client.14182.objecter handle_osd_op_reply completed tid 9 2024-09-16T03:43:34.474 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.471+0000 7f91004a6640 15 client.14182.objecter _finish_op 9 2024-09-16T03:43:34.474 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.471+0000 7f91004a6640 20 client.14182.objecter put_session s=0x5607e0653f50 osd=1 4 2024-09-16T03:43:34.474 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.471+0000 7f91004a6640 15 client.14182.objecter _session_op_remove 1 9 2024-09-16T03:43:34.474 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.471+0000 7f91004a6640 5 client.14182.objecter 0 in flight 2024-09-16T03:43:34.474 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.471+0000 7f91004a6640 20 librados: complete_aio_write 0x5607e04d7450 2024-09-16T03:43:34.475 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.471+0000 7f9102191880 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=head 2024-09-16T03:43:34.475 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.471+0000 7f9102191880 20 librados: queue_aio_write 0x5607e050b7a0 completion 0x5607e04e4520 write_seq 2 2024-09-16T03:43:34.508 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.471+0000 7f9102191880 10 client.14182.objecter _op_submit op 0x5607e06f9a00 2024-09-16T03:43:34.508 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.471+0000 7f9102191880 20 client.14182.objecter _calc_target epoch 22 base exist @31 precalc_pgid 0 pgid 0.0 is_write 2024-09-16T03:43:34.508 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.471+0000 7f9102191880 20 client.14182.objecter _calc_target target exist @31 -> pgid 31.d3a59201 2024-09-16T03:43:34.508 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.471+0000 7f9102191880 10 client.14182.objecter _calc_target raw pgid 31.d3a59201 -> actual 31.1 acting [1,4] primary 1 2024-09-16T03:43:34.509 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.471+0000 7f9102191880 20 client.14182.objecter _get_session s=0x5607e0653f50 osd=1 3 2024-09-16T03:43:34.509 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.471+0000 7f9102191880 10 client.14182.objecter _op_submit oid exist '@31' '@31' [write 0~5 in=5b] tid 10 osd.1 2024-09-16T03:43:34.509 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.471+0000 7f9102191880 20 client.14182.objecter get_session s=0x5607e0653f50 osd=1 3 2024-09-16T03:43:34.509 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.471+0000 7f9102191880 15 client.14182.objecter _session_op_assign 1 10 2024-09-16T03:43:34.509 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.471+0000 7f9102191880 15 client.14182.objecter _send_op 10 to 31.1 on osd.1 2024-09-16T03:43:34.509 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.471+0000 7f9102191880 1 -- 172.21.15.154:0/332177798 --> v2:172.21.15.42:6800/4217863575 -- osd_op(unknown.0.0:10 31.1 31:8049a5cb:::exist:head [write 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22) -- 0x5607e06fb260 con 0x5607e06f6d30 2024-09-16T03:43:34.509 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.471+0000 7f9102191880 20 client.14182.objecter put_session s=0x5607e0653f50 osd=1 4 2024-09-16T03:43:34.509 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.471+0000 7f9102191880 5 client.14182.objecter 1 in flight 2024-09-16T03:43:34.509 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.471+0000 7f9102191880 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=1 2024-09-16T03:43:34.509 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.507+0000 7f91004a6640 1 -- 172.21.15.154:0/332177798 <== osd.1 v2:172.21.15.42:6800/4217863575 10 ==== osd_op_reply(10 exist [write 0~5] v22'3 uv3 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f90f80138d0 con 0x5607e06f6d30 2024-09-16T03:43:34.509 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.507+0000 7f91004a6640 10 client.14182.objecter ms_dispatch 0x5607e05160c0 osd_op_reply(10 exist [write 0~5] v22'3 uv3 ondisk = 0) 2024-09-16T03:43:34.509 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.507+0000 7f91004a6640 10 client.14182.objecter in handle_osd_op_reply 2024-09-16T03:43:34.509 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.507+0000 7f91004a6640 7 client.14182.objecter handle_osd_op_reply 10 ondisk uv 3 in 31.1 attempt 0 2024-09-16T03:43:34.510 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.507+0000 7f91004a6640 10 client.14182.objecter op 0 rval 0 len 0 2024-09-16T03:43:34.510 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.507+0000 7f91004a6640 15 client.14182.objecter handle_osd_op_reply completed tid 10 2024-09-16T03:43:34.510 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.507+0000 7f91004a6640 15 client.14182.objecter _finish_op 10 2024-09-16T03:43:34.510 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.507+0000 7f91004a6640 20 client.14182.objecter put_session s=0x5607e0653f50 osd=1 4 2024-09-16T03:43:34.510 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.507+0000 7f91004a6640 15 client.14182.objecter _session_op_remove 1 10 2024-09-16T03:43:34.510 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.507+0000 7f91004a6640 5 client.14182.objecter 0 in flight 2024-09-16T03:43:34.510 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.507+0000 7f91004a6640 20 librados: complete_aio_write 0x5607e04e4520 2024-09-16T03:43:34.510 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: [ OK ] AsioRados.AsyncWriteFuture (35 ms) 2024-09-16T03:43:34.510 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteYield 2024-09-16T03:43:34.510 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.507+0000 7f9102191880 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=head 2024-09-16T03:43:34.510 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.507+0000 7f9102191880 20 librados: queue_aio_write 0x5607e050b7a0 completion 0x5607e06f9a00 write_seq 3 2024-09-16T03:43:34.510 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.507+0000 7f9102191880 10 client.14182.objecter _op_submit op 0x5607e071d920 2024-09-16T03:43:34.528 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.507+0000 7f9102191880 20 client.14182.objecter _calc_target epoch 22 base exist @31 precalc_pgid 0 pgid 0.0 is_write 2024-09-16T03:43:34.529 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.507+0000 7f9102191880 20 client.14182.objecter _calc_target target exist @31 -> pgid 31.d3a59201 2024-09-16T03:43:34.529 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.507+0000 7f9102191880 10 client.14182.objecter _calc_target raw pgid 31.d3a59201 -> actual 31.1 acting [1,4] primary 1 2024-09-16T03:43:34.529 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.507+0000 7f9102191880 20 client.14182.objecter _get_session s=0x5607e0653f50 osd=1 3 2024-09-16T03:43:34.529 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.507+0000 7f9102191880 10 client.14182.objecter _op_submit oid exist '@31' '@31' [write 0~5 in=5b] tid 11 osd.1 2024-09-16T03:43:34.529 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.507+0000 7f9102191880 20 client.14182.objecter get_session s=0x5607e0653f50 osd=1 3 2024-09-16T03:43:34.529 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.507+0000 7f9102191880 15 client.14182.objecter _session_op_assign 1 11 2024-09-16T03:43:34.529 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.507+0000 7f9102191880 15 client.14182.objecter _send_op 11 to 31.1 on osd.1 2024-09-16T03:43:34.529 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.507+0000 7f9102191880 1 -- 172.21.15.154:0/332177798 --> v2:172.21.15.42:6800/4217863575 -- osd_op(unknown.0.0:11 31.1 31:8049a5cb:::exist:head [write 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22) -- 0x5607e071ddd0 con 0x5607e06f6d30 2024-09-16T03:43:34.529 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.507+0000 7f9102191880 20 client.14182.objecter put_session s=0x5607e0653f50 osd=1 4 2024-09-16T03:43:34.529 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.507+0000 7f9102191880 5 client.14182.objecter 1 in flight 2024-09-16T03:43:34.529 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.507+0000 7f9102191880 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=1 2024-09-16T03:43:34.530 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.523+0000 7f91004a6640 1 -- 172.21.15.154:0/332177798 <== osd.1 v2:172.21.15.42:6800/4217863575 11 ==== osd_op_reply(11 exist [write 0~5] v22'4 uv4 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f90f80138d0 con 0x5607e06f6d30 2024-09-16T03:43:34.530 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.523+0000 7f91004a6640 10 client.14182.objecter ms_dispatch 0x5607e05160c0 osd_op_reply(11 exist [write 0~5] v22'4 uv4 ondisk = 0) 2024-09-16T03:43:34.530 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.523+0000 7f91004a6640 10 client.14182.objecter in handle_osd_op_reply 2024-09-16T03:43:34.530 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.523+0000 7f91004a6640 7 client.14182.objecter handle_osd_op_reply 11 ondisk uv 4 in 31.1 attempt 0 2024-09-16T03:43:34.530 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.523+0000 7f91004a6640 10 client.14182.objecter op 0 rval 0 len 0 2024-09-16T03:43:34.530 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.523+0000 7f91004a6640 15 client.14182.objecter handle_osd_op_reply completed tid 11 2024-09-16T03:43:34.530 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.523+0000 7f91004a6640 15 client.14182.objecter _finish_op 11 2024-09-16T03:43:34.530 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.523+0000 7f91004a6640 20 client.14182.objecter put_session s=0x5607e0653f50 osd=1 4 2024-09-16T03:43:34.530 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.523+0000 7f91004a6640 15 client.14182.objecter _session_op_remove 1 11 2024-09-16T03:43:34.530 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.523+0000 7f91004a6640 5 client.14182.objecter 0 in flight 2024-09-16T03:43:34.530 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.523+0000 7f91004a6640 20 librados: complete_aio_write 0x5607e06f9a00 2024-09-16T03:43:34.530 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: [ OK ] AsioRados.AsyncWriteYield (19 ms) 2024-09-16T03:43:34.530 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationCallback 2024-09-16T03:43:34.531 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.523+0000 7f9102191880 10 client.14182.objecter _op_submit op 0x5607e06fa250 2024-09-16T03:43:34.531 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.523+0000 7f9102191880 20 client.14182.objecter _calc_target epoch 22 base exist @31 precalc_pgid 0 pgid 0.0 is_read 2024-09-16T03:43:34.531 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.523+0000 7f9102191880 20 client.14182.objecter _calc_target target exist @31 -> pgid 31.d3a59201 2024-09-16T03:43:34.531 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.523+0000 7f9102191880 10 client.14182.objecter _calc_target raw pgid 31.d3a59201 -> actual 31.1 acting [1,4] primary 1 2024-09-16T03:43:34.532 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.523+0000 7f9102191880 20 client.14182.objecter _get_session s=0x5607e0653f50 osd=1 3 2024-09-16T03:43:34.532 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.523+0000 7f9102191880 10 client.14182.objecter _op_submit oid exist '@31' '@31' [read 0~0] tid 12 osd.1 2024-09-16T03:43:34.532 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.523+0000 7f9102191880 20 client.14182.objecter get_session s=0x5607e0653f50 osd=1 3 2024-09-16T03:43:34.532 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.523+0000 7f9102191880 15 client.14182.objecter _session_op_assign 1 12 2024-09-16T03:43:34.532 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.523+0000 7f9102191880 15 client.14182.objecter _send_op 12 to 31.1 on osd.1 2024-09-16T03:43:34.532 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.523+0000 7f9102191880 1 -- 172.21.15.154:0/332177798 --> v2:172.21.15.42:6800/4217863575 -- osd_op(unknown.0.0:12 31.1 31:8049a5cb:::exist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22) -- 0x5607e06fa700 con 0x5607e06f6d30 2024-09-16T03:43:34.532 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.523+0000 7f9102191880 20 client.14182.objecter put_session s=0x5607e0653f50 osd=1 4 2024-09-16T03:43:34.532 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.523+0000 7f9102191880 5 client.14182.objecter 1 in flight 2024-09-16T03:43:34.533 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.523+0000 7f9102191880 10 client.14182.objecter _op_submit op 0x5607e06faaa0 2024-09-16T03:43:34.533 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.523+0000 7f9102191880 20 client.14182.objecter _calc_target epoch 22 base noexist @31 precalc_pgid 0 pgid 0.0 is_read 2024-09-16T03:43:34.533 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.523+0000 7f9102191880 20 client.14182.objecter _calc_target target noexist @31 -> pgid 31.a7bbaa1 2024-09-16T03:43:34.533 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.523+0000 7f9102191880 10 client.14182.objecter _calc_target raw pgid 31.a7bbaa1 -> actual 31.1 acting [1,4] primary 1 2024-09-16T03:43:34.533 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.523+0000 7f9102191880 20 client.14182.objecter _get_session s=0x5607e0653f50 osd=1 4 2024-09-16T03:43:34.533 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.523+0000 7f9102191880 10 client.14182.objecter _op_submit oid noexist '@31' '@31' [read 0~0] tid 13 osd.1 2024-09-16T03:43:34.533 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.523+0000 7f9102191880 20 client.14182.objecter get_session s=0x5607e0653f50 osd=1 4 2024-09-16T03:43:34.533 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.523+0000 7f9102191880 15 client.14182.objecter _session_op_assign 1 13 2024-09-16T03:43:34.533 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.523+0000 7f9102191880 15 client.14182.objecter _send_op 13 to 31.1 on osd.1 2024-09-16T03:43:34.533 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.523+0000 7f9102191880 1 -- 172.21.15.154:0/332177798 --> v2:172.21.15.42:6800/4217863575 -- osd_op(unknown.0.0:13 31.1 31:855dde50:::noexist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22) -- 0x5607e071e250 con 0x5607e06f6d30 2024-09-16T03:43:34.533 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.523+0000 7f9102191880 20 client.14182.objecter put_session s=0x5607e0653f50 osd=1 5 2024-09-16T03:43:34.533 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.523+0000 7f9102191880 5 client.14182.objecter 2 in flight 2024-09-16T03:43:34.534 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.527+0000 7f91004a6640 1 -- 172.21.15.154:0/332177798 <== osd.1 v2:172.21.15.42:6800/4217863575 12 ==== osd_op_reply(12 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) ==== 149+0+5 (crc 0 0 0) 0x7f90f80138d0 con 0x5607e06f6d30 2024-09-16T03:43:34.534 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.527+0000 7f91004a6640 10 client.14182.objecter ms_dispatch 0x5607e05160c0 osd_op_reply(12 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) 2024-09-16T03:43:34.534 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.527+0000 7f91004a6640 10 client.14182.objecter in handle_osd_op_reply 2024-09-16T03:43:34.534 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.527+0000 7f91004a6640 7 client.14182.objecter handle_osd_op_reply 12 ondisk uv 4 in 31.1 attempt 0 2024-09-16T03:43:34.534 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.527+0000 7f91004a6640 10 client.14182.objecter op 0 rval 0 len 5 2024-09-16T03:43:34.534 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.527+0000 7f91004a6640 15 client.14182.objecter handle_osd_op_reply completed tid 12 2024-09-16T03:43:34.534 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.527+0000 7f91004a6640 15 client.14182.objecter _finish_op 12 2024-09-16T03:43:34.534 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.527+0000 7f91004a6640 20 client.14182.objecter put_session s=0x5607e0653f50 osd=1 5 2024-09-16T03:43:34.534 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.527+0000 7f91004a6640 15 client.14182.objecter _session_op_remove 1 12 2024-09-16T03:43:34.535 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.527+0000 7f91004a6640 5 client.14182.objecter 1 in flight 2024-09-16T03:43:34.535 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.527+0000 7f91004a6640 1 -- 172.21.15.154:0/332177798 <== osd.1 v2:172.21.15.42:6800/4217863575 13 ==== osd_op_reply(13 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 151+0+0 (crc 0 0 0) 0x7f90f80138d0 con 0x5607e06f6d30 2024-09-16T03:43:34.535 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.527+0000 7f91004a6640 10 client.14182.objecter ms_dispatch 0x5607e05160c0 osd_op_reply(13 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) 2024-09-16T03:43:34.535 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.527+0000 7f91004a6640 10 client.14182.objecter in handle_osd_op_reply 2024-09-16T03:43:34.535 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.527+0000 7f91004a6640 7 client.14182.objecter handle_osd_op_reply 13 ondisk uv 0 in 31.1 attempt 0 2024-09-16T03:43:34.535 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.527+0000 7f91004a6640 10 client.14182.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-09-16T03:43:34.535 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.527+0000 7f91004a6640 10 client.14182.objecter op 0 rval 0 len 0 2024-09-16T03:43:34.535 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.527+0000 7f91004a6640 15 client.14182.objecter handle_osd_op_reply completed tid 13 2024-09-16T03:43:34.535 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.527+0000 7f91004a6640 15 client.14182.objecter _finish_op 13 2024-09-16T03:43:34.535 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.527+0000 7f91004a6640 20 client.14182.objecter put_session s=0x5607e0653f50 osd=1 4 2024-09-16T03:43:34.535 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.527+0000 7f91004a6640 15 client.14182.objecter _session_op_remove 1 13 2024-09-16T03:43:34.535 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.527+0000 7f91004a6640 5 client.14182.objecter 0 in flight 2024-09-16T03:43:34.535 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationCallback (1 ms) 2024-09-16T03:43:34.536 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationFuture 2024-09-16T03:43:34.536 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.527+0000 7f9102191880 10 client.14182.objecter _op_submit op 0x5607e06fa360 2024-09-16T03:43:34.536 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.527+0000 7f9102191880 20 client.14182.objecter _calc_target epoch 22 base exist @31 precalc_pgid 0 pgid 0.0 is_read 2024-09-16T03:43:34.536 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.527+0000 7f9102191880 20 client.14182.objecter _calc_target target exist @31 -> pgid 31.d3a59201 2024-09-16T03:43:34.536 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.527+0000 7f9102191880 10 client.14182.objecter _calc_target raw pgid 31.d3a59201 -> actual 31.1 acting [1,4] primary 1 2024-09-16T03:43:34.536 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.527+0000 7f9102191880 20 client.14182.objecter _get_session s=0x5607e0653f50 osd=1 3 2024-09-16T03:43:34.536 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.527+0000 7f9102191880 10 client.14182.objecter _op_submit oid exist '@31' '@31' [read 0~0] tid 14 osd.1 2024-09-16T03:43:34.536 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.527+0000 7f9102191880 20 client.14182.objecter get_session s=0x5607e0653f50 osd=1 3 2024-09-16T03:43:34.536 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.527+0000 7f9102191880 15 client.14182.objecter _session_op_assign 1 14 2024-09-16T03:43:34.536 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.527+0000 7f9102191880 15 client.14182.objecter _send_op 14 to 31.1 on osd.1 2024-09-16T03:43:34.536 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.527+0000 7f9102191880 1 -- 172.21.15.154:0/332177798 --> v2:172.21.15.42:6800/4217863575 -- osd_op(unknown.0.0:14 31.1 31:8049a5cb:::exist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22) -- 0x5607e06fa810 con 0x5607e06f6d30 2024-09-16T03:43:34.536 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.527+0000 7f9102191880 20 client.14182.objecter put_session s=0x5607e0653f50 osd=1 4 2024-09-16T03:43:34.536 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.527+0000 7f9102191880 5 client.14182.objecter 1 in flight 2024-09-16T03:43:34.537 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.527+0000 7f9102191880 10 client.14182.objecter _op_submit op 0x5607e06fad70 2024-09-16T03:43:34.537 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.527+0000 7f9102191880 20 client.14182.objecter _calc_target epoch 22 base noexist @31 precalc_pgid 0 pgid 0.0 is_read 2024-09-16T03:43:34.537 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.527+0000 7f9102191880 20 client.14182.objecter _calc_target target noexist @31 -> pgid 31.a7bbaa1 2024-09-16T03:43:34.537 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.527+0000 7f9102191880 10 client.14182.objecter _calc_target raw pgid 31.a7bbaa1 -> actual 31.1 acting [1,4] primary 1 2024-09-16T03:43:34.537 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.527+0000 7f9102191880 20 client.14182.objecter _get_session s=0x5607e0653f50 osd=1 4 2024-09-16T03:43:34.537 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.527+0000 7f9102191880 10 client.14182.objecter _op_submit oid noexist '@31' '@31' [read 0~0] tid 15 osd.1 2024-09-16T03:43:34.537 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.527+0000 7f9102191880 20 client.14182.objecter get_session s=0x5607e0653f50 osd=1 4 2024-09-16T03:43:34.537 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.527+0000 7f9102191880 15 client.14182.objecter _session_op_assign 1 15 2024-09-16T03:43:34.537 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.527+0000 7f9102191880 15 client.14182.objecter _send_op 15 to 31.1 on osd.1 2024-09-16T03:43:34.537 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.527+0000 7f9102191880 1 -- 172.21.15.154:0/332177798 --> v2:172.21.15.42:6800/4217863575 -- osd_op(unknown.0.0:15 31.1 31:855dde50:::noexist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22) -- 0x5607e071e250 con 0x5607e06f6d30 2024-09-16T03:43:34.537 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.527+0000 7f9102191880 20 client.14182.objecter put_session s=0x5607e0653f50 osd=1 5 2024-09-16T03:43:34.537 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.527+0000 7f9102191880 5 client.14182.objecter 2 in flight 2024-09-16T03:43:34.537 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.527+0000 7f91004a6640 1 -- 172.21.15.154:0/332177798 <== osd.1 v2:172.21.15.42:6800/4217863575 14 ==== osd_op_reply(14 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) ==== 149+0+5 (crc 0 0 0) 0x7f90f80138d0 con 0x5607e06f6d30 2024-09-16T03:43:34.538 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.527+0000 7f91004a6640 10 client.14182.objecter ms_dispatch 0x5607e05160c0 osd_op_reply(14 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) 2024-09-16T03:43:34.538 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.527+0000 7f91004a6640 10 client.14182.objecter in handle_osd_op_reply 2024-09-16T03:43:34.538 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.527+0000 7f91004a6640 7 client.14182.objecter handle_osd_op_reply 14 ondisk uv 4 in 31.1 attempt 0 2024-09-16T03:43:34.538 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.527+0000 7f91004a6640 10 client.14182.objecter op 0 rval 0 len 5 2024-09-16T03:43:34.538 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.527+0000 7f91004a6640 15 client.14182.objecter handle_osd_op_reply completed tid 14 2024-09-16T03:43:34.538 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.527+0000 7f91004a6640 15 client.14182.objecter _finish_op 14 2024-09-16T03:43:34.538 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.527+0000 7f91004a6640 20 client.14182.objecter put_session s=0x5607e0653f50 osd=1 5 2024-09-16T03:43:34.538 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.527+0000 7f91004a6640 15 client.14182.objecter _session_op_remove 1 14 2024-09-16T03:43:34.538 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.527+0000 7f91004a6640 5 client.14182.objecter 1 in flight 2024-09-16T03:43:34.538 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.527+0000 7f91004a6640 1 -- 172.21.15.154:0/332177798 <== osd.1 v2:172.21.15.42:6800/4217863575 15 ==== osd_op_reply(15 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 151+0+0 (crc 0 0 0) 0x7f90f80138d0 con 0x5607e06f6d30 2024-09-16T03:43:34.538 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.527+0000 7f91004a6640 10 client.14182.objecter ms_dispatch 0x5607e05160c0 osd_op_reply(15 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) 2024-09-16T03:43:34.538 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.527+0000 7f91004a6640 10 client.14182.objecter in handle_osd_op_reply 2024-09-16T03:43:34.538 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.527+0000 7f91004a6640 7 client.14182.objecter handle_osd_op_reply 15 ondisk uv 0 in 31.1 attempt 0 2024-09-16T03:43:34.539 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.527+0000 7f91004a6640 10 client.14182.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-09-16T03:43:34.539 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.527+0000 7f91004a6640 10 client.14182.objecter op 0 rval 0 len 0 2024-09-16T03:43:34.539 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.527+0000 7f91004a6640 15 client.14182.objecter handle_osd_op_reply completed tid 15 2024-09-16T03:43:34.539 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.527+0000 7f91004a6640 15 client.14182.objecter _finish_op 15 2024-09-16T03:43:34.539 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.527+0000 7f91004a6640 20 client.14182.objecter put_session s=0x5607e0653f50 osd=1 4 2024-09-16T03:43:34.539 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.527+0000 7f91004a6640 15 client.14182.objecter _session_op_remove 1 15 2024-09-16T03:43:34.539 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.527+0000 7f91004a6640 5 client.14182.objecter 0 in flight 2024-09-16T03:43:34.539 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationFuture (1 ms) 2024-09-16T03:43:34.539 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationYield 2024-09-16T03:43:34.539 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.527+0000 7f9102191880 10 client.14182.objecter _op_submit op 0x5607e06fad70 2024-09-16T03:43:34.539 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.527+0000 7f9102191880 20 client.14182.objecter _calc_target epoch 22 base exist @31 precalc_pgid 0 pgid 0.0 is_read 2024-09-16T03:43:34.539 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.527+0000 7f9102191880 20 client.14182.objecter _calc_target target exist @31 -> pgid 31.d3a59201 2024-09-16T03:43:34.539 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.527+0000 7f9102191880 10 client.14182.objecter _calc_target raw pgid 31.d3a59201 -> actual 31.1 acting [1,4] primary 1 2024-09-16T03:43:34.539 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.527+0000 7f9102191880 20 client.14182.objecter _get_session s=0x5607e0653f50 osd=1 3 2024-09-16T03:43:34.540 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.527+0000 7f9102191880 10 client.14182.objecter _op_submit oid exist '@31' '@31' [read 0~0] tid 16 osd.1 2024-09-16T03:43:34.540 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.527+0000 7f9102191880 20 client.14182.objecter get_session s=0x5607e0653f50 osd=1 3 2024-09-16T03:43:34.540 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.527+0000 7f9102191880 15 client.14182.objecter _session_op_assign 1 16 2024-09-16T03:43:34.540 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.527+0000 7f9102191880 15 client.14182.objecter _send_op 16 to 31.1 on osd.1 2024-09-16T03:43:34.540 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.527+0000 7f9102191880 1 -- 172.21.15.154:0/332177798 --> v2:172.21.15.42:6800/4217863575 -- osd_op(unknown.0.0:16 31.1 31:8049a5cb:::exist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22) -- 0x5607e071d2b0 con 0x5607e06f6d30 2024-09-16T03:43:34.540 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.527+0000 7f9102191880 20 client.14182.objecter put_session s=0x5607e0653f50 osd=1 4 2024-09-16T03:43:34.540 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.527+0000 7f9102191880 5 client.14182.objecter 1 in flight 2024-09-16T03:43:34.540 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.527+0000 7f9102191880 10 client.14182.objecter _op_submit op 0x5607e071d840 2024-09-16T03:43:34.540 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.527+0000 7f9102191880 20 client.14182.objecter _calc_target epoch 22 base noexist @31 precalc_pgid 0 pgid 0.0 is_read 2024-09-16T03:43:34.540 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.527+0000 7f9102191880 20 client.14182.objecter _calc_target target noexist @31 -> pgid 31.a7bbaa1 2024-09-16T03:43:34.540 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.527+0000 7f9102191880 10 client.14182.objecter _calc_target raw pgid 31.a7bbaa1 -> actual 31.1 acting [1,4] primary 1 2024-09-16T03:43:34.540 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.527+0000 7f9102191880 20 client.14182.objecter _get_session s=0x5607e0653f50 osd=1 4 2024-09-16T03:43:34.540 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.527+0000 7f9102191880 10 client.14182.objecter _op_submit oid noexist '@31' '@31' [read 0~0] tid 17 osd.1 2024-09-16T03:43:34.541 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.527+0000 7f9102191880 20 client.14182.objecter get_session s=0x5607e0653f50 osd=1 4 2024-09-16T03:43:34.541 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.527+0000 7f9102191880 15 client.14182.objecter _session_op_assign 1 17 2024-09-16T03:43:34.541 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.527+0000 7f9102191880 15 client.14182.objecter _send_op 17 to 31.1 on osd.1 2024-09-16T03:43:34.541 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.527+0000 7f9102191880 1 -- 172.21.15.154:0/332177798 --> v2:172.21.15.42:6800/4217863575 -- osd_op(unknown.0.0:17 31.1 31:855dde50:::noexist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22) -- 0x5607e071dcf0 con 0x5607e06f6d30 2024-09-16T03:43:34.541 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.527+0000 7f9102191880 20 client.14182.objecter put_session s=0x5607e0653f50 osd=1 5 2024-09-16T03:43:34.541 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.527+0000 7f9102191880 5 client.14182.objecter 2 in flight 2024-09-16T03:43:34.541 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.527+0000 7f91004a6640 1 -- 172.21.15.154:0/332177798 <== osd.1 v2:172.21.15.42:6800/4217863575 16 ==== osd_op_reply(16 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) ==== 149+0+5 (crc 0 0 0) 0x7f90f80138d0 con 0x5607e06f6d30 2024-09-16T03:43:34.541 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.527+0000 7f91004a6640 10 client.14182.objecter ms_dispatch 0x5607e05160c0 osd_op_reply(16 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) 2024-09-16T03:43:34.541 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.527+0000 7f91004a6640 10 client.14182.objecter in handle_osd_op_reply 2024-09-16T03:43:34.541 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.527+0000 7f91004a6640 7 client.14182.objecter handle_osd_op_reply 16 ondisk uv 4 in 31.1 attempt 0 2024-09-16T03:43:34.541 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.527+0000 7f91004a6640 10 client.14182.objecter op 0 rval 0 len 5 2024-09-16T03:43:34.541 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.527+0000 7f91004a6640 15 client.14182.objecter handle_osd_op_reply completed tid 16 2024-09-16T03:43:34.541 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.527+0000 7f91004a6640 15 client.14182.objecter _finish_op 16 2024-09-16T03:43:34.541 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.527+0000 7f91004a6640 20 client.14182.objecter put_session s=0x5607e0653f50 osd=1 5 2024-09-16T03:43:34.542 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.527+0000 7f91004a6640 15 client.14182.objecter _session_op_remove 1 16 2024-09-16T03:43:34.542 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.527+0000 7f91004a6640 5 client.14182.objecter 1 in flight 2024-09-16T03:43:34.542 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationYield (1 ms) 2024-09-16T03:43:34.542 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationCallback 2024-09-16T03:43:34.542 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.527+0000 7f91004a6640 1 -- 172.21.15.154:0/332177798 <== osd.1 v2:172.21.15.42:6800/4217863575 17 ==== osd_op_reply(17 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 151+0+0 (crc 0 0 0) 0x7f90f80138d0 con 0x5607e06f6d30 2024-09-16T03:43:34.542 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.527+0000 7f91004a6640 10 client.14182.objecter ms_dispatch 0x5607e05160c0 osd_op_reply(17 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) 2024-09-16T03:43:34.542 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.527+0000 7f91004a6640 10 client.14182.objecter in handle_osd_op_reply 2024-09-16T03:43:34.542 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.527+0000 7f91004a6640 7 client.14182.objecter handle_osd_op_reply 17 ondisk uv 0 in 31.1 attempt 0 2024-09-16T03:43:34.542 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.527+0000 7f91004a6640 10 client.14182.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-09-16T03:43:34.542 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.527+0000 7f91004a6640 10 client.14182.objecter op 0 rval 0 len 0 2024-09-16T03:43:34.542 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.527+0000 7f91004a6640 15 client.14182.objecter handle_osd_op_reply completed tid 17 2024-09-16T03:43:34.542 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.527+0000 7f91004a6640 15 client.14182.objecter _finish_op 17 2024-09-16T03:43:34.542 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.527+0000 7f91004a6640 20 client.14182.objecter put_session s=0x5607e0653f50 osd=1 4 2024-09-16T03:43:34.542 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.527+0000 7f91004a6640 15 client.14182.objecter _session_op_remove 1 17 2024-09-16T03:43:34.543 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.527+0000 7f91004a6640 5 client.14182.objecter 0 in flight 2024-09-16T03:43:34.543 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.527+0000 7f9102191880 20 librados: queue_aio_write 0x5607e050b7a0 completion 0x5607e06fac60 write_seq 4 2024-09-16T03:43:34.543 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.527+0000 7f9102191880 10 client.14182.objecter _op_submit op 0x5607e06fad70 2024-09-16T03:43:34.543 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.527+0000 7f9102191880 20 client.14182.objecter _calc_target epoch 22 base exist @31 precalc_pgid 0 pgid 0.0 is_write 2024-09-16T03:43:34.543 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.527+0000 7f9102191880 20 client.14182.objecter _calc_target target exist @31 -> pgid 31.d3a59201 2024-09-16T03:43:34.551 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.527+0000 7f9102191880 10 client.14182.objecter _calc_target raw pgid 31.d3a59201 -> actual 31.1 acting [1,4] primary 1 2024-09-16T03:43:34.551 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.527+0000 7f9102191880 20 client.14182.objecter _get_session s=0x5607e0653f50 osd=1 3 2024-09-16T03:43:34.551 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.527+0000 7f9102191880 10 client.14182.objecter _op_submit oid exist '@31' '@31' [writefull 0~5 in=5b] tid 18 osd.1 2024-09-16T03:43:34.551 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.527+0000 7f9102191880 20 client.14182.objecter get_session s=0x5607e0653f50 osd=1 3 2024-09-16T03:43:34.551 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.527+0000 7f9102191880 15 client.14182.objecter _session_op_assign 1 18 2024-09-16T03:43:34.551 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.527+0000 7f9102191880 15 client.14182.objecter _send_op 18 to 31.1 on osd.1 2024-09-16T03:43:34.551 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.527+0000 7f9102191880 1 -- 172.21.15.154:0/332177798 --> v2:172.21.15.42:6800/4217863575 -- osd_op(unknown.0.0:18 31.1 31:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22) -- 0x5607e06fa360 con 0x5607e06f6d30 2024-09-16T03:43:34.551 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.527+0000 7f9102191880 20 client.14182.objecter put_session s=0x5607e0653f50 osd=1 4 2024-09-16T03:43:34.551 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.527+0000 7f9102191880 5 client.14182.objecter 1 in flight 2024-09-16T03:43:34.551 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.547+0000 7f91004a6640 1 -- 172.21.15.154:0/332177798 <== osd.1 v2:172.21.15.42:6800/4217863575 18 ==== osd_op_reply(18 exist [writefull 0~5] v22'5 uv5 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f90f80138d0 con 0x5607e06f6d30 2024-09-16T03:43:34.551 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.547+0000 7f91004a6640 10 client.14182.objecter ms_dispatch 0x5607e05160c0 osd_op_reply(18 exist [writefull 0~5] v22'5 uv5 ondisk = 0) 2024-09-16T03:43:34.552 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.547+0000 7f91004a6640 10 client.14182.objecter in handle_osd_op_reply 2024-09-16T03:43:34.552 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.547+0000 7f91004a6640 7 client.14182.objecter handle_osd_op_reply 18 ondisk uv 5 in 31.1 attempt 0 2024-09-16T03:43:34.552 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.547+0000 7f91004a6640 10 client.14182.objecter op 0 rval 0 len 0 2024-09-16T03:43:34.552 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.547+0000 7f91004a6640 15 client.14182.objecter handle_osd_op_reply completed tid 18 2024-09-16T03:43:34.552 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.547+0000 7f91004a6640 15 client.14182.objecter _finish_op 18 2024-09-16T03:43:34.552 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.547+0000 7f91004a6640 20 client.14182.objecter put_session s=0x5607e0653f50 osd=1 4 2024-09-16T03:43:34.552 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.547+0000 7f91004a6640 15 client.14182.objecter _session_op_remove 1 18 2024-09-16T03:43:34.552 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.547+0000 7f91004a6640 5 client.14182.objecter 0 in flight 2024-09-16T03:43:34.552 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.547+0000 7f91004a6640 20 librados: complete_aio_write 0x5607e06fac60 2024-09-16T03:43:34.552 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationCallback (21 ms) 2024-09-16T03:43:34.552 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationFuture 2024-09-16T03:43:34.552 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.547+0000 7f9102191880 20 librados: queue_aio_write 0x5607e050b7a0 completion 0x5607e06f9c60 write_seq 5 2024-09-16T03:43:34.552 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.547+0000 7f9102191880 10 client.14182.objecter _op_submit op 0x5607e06fad70 2024-09-16T03:43:34.552 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.547+0000 7f9102191880 20 client.14182.objecter _calc_target epoch 22 base exist @31 precalc_pgid 0 pgid 0.0 is_write 2024-09-16T03:43:34.553 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.547+0000 7f9102191880 20 client.14182.objecter _calc_target target exist @31 -> pgid 31.d3a59201 2024-09-16T03:43:34.553 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.547+0000 7f9102191880 10 client.14182.objecter _calc_target raw pgid 31.d3a59201 -> actual 31.1 acting [1,4] primary 1 2024-09-16T03:43:34.553 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.547+0000 7f9102191880 20 client.14182.objecter _get_session s=0x5607e0653f50 osd=1 3 2024-09-16T03:43:34.553 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.547+0000 7f9102191880 10 client.14182.objecter _op_submit oid exist '@31' '@31' [writefull 0~5 in=5b] tid 19 osd.1 2024-09-16T03:43:34.560 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.553+0000 7fbce4dc5640 10 monclient: get_monmap_and_config 2024-09-16T03:43:34.560 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.553+0000 7fbce4dc5640 10 monclient: build_initial_monmap 2024-09-16T03:43:34.560 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.553+0000 7fbce4dc5640 10 monclient: monmap: 2024-09-16T03:43:34.560 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:43:34.560 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:43:34.560 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:43:34.558955+0000 2024-09-16T03:43:34.560 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:43:34.558955+0000 2024-09-16T03:43:34.560 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:43:34.561 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:43:34.561 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:43:34.561 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:43:34.561 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:43:34.561 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:43:34.561 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:43:34.561 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:43:34.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.553+0000 7fbce4dc5640 1 Processor -- start 2024-09-16T03:43:34.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.553+0000 7fbce4dc5640 1 -- start start 2024-09-16T03:43:34.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.553+0000 7fbce4dc5640 10 monclient: init 2024-09-16T03:43:34.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.557+0000 7fbce4dc5640 10 monclient: _reopen_session rank -1 2024-09-16T03:43:34.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.557+0000 7fbce4dc5640 10 monclient: _add_conns ranks=[4,3,1,0,2] 2024-09-16T03:43:34.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.557+0000 7fbce4dc5640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fbce00a2230 0x7fbce00a2610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:43:34.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.557+0000 7fbce4dc5640 10 monclient(hunting): picked mon.noname-c con 0x7fbce00a2230 addr v2:172.21.15.154:3300/0 2024-09-16T03:43:34.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.557+0000 7fbce4dc5640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fbce0072740 0x7fbce00a2b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:43:34.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.557+0000 7fbce4dc5640 10 monclient(hunting): picked mon.noname-a con 0x7fbce0072740 addr v2:172.21.15.42:3300/0 2024-09-16T03:43:34.565 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.557+0000 7fbce4dc5640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbce0134ff0 addr v1:172.21.15.154:6789/0 2024-09-16T03:43:34.565 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.557+0000 7fbce4dc5640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbce006c050 con 0x7fbce0134ff0 2024-09-16T03:43:34.565 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.557+0000 7fbce4dc5640 10 monclient(hunting): start opening mon connection 2024-09-16T03:43:34.565 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.557+0000 7fbce4dc5640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7fbce0071b10 con 0x7fbce0072740 2024-09-16T03:43:34.565 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.557+0000 7fbce4dc5640 10 monclient(hunting): start opening mon connection 2024-09-16T03:43:34.565 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.557+0000 7fbce4dc5640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fbce006c750 con 0x7fbce00a2230 2024-09-16T03:43:34.565 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.557+0000 7fbce4dc5640 10 monclient(hunting): _renew_subs 2024-09-16T03:43:34.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.557+0000 7fbce4dc5640 10 monclient(hunting): authenticate will time out at 1061.528324s 2024-09-16T03:43:34.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.557+0000 7fbcd5d9b640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fbce0072740 0x7fbce00a2b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:43:34.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.557+0000 7fbcded9d640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7fbce0134ff0 0x7fbce0135460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.42:46736/0 (socket says 172.21.15.42:46736) 2024-09-16T03:43:34.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.557+0000 7fbcded9d640 1 -- 172.21.15.42:0/1148426661 learned_addr learned my addr 172.21.15.42:0/1148426661 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:43:34.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.557+0000 7fbcd5d9b640 1 --2- 172.21.15.42:0/1148426661 >> v2:172.21.15.42:3300/0 conn(0x7fbce0072740 0x7fbce00a2b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:51998/0 (socket says 172.21.15.42:51998) 2024-09-16T03:43:34.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.557+0000 7fbcd5d9b640 10 monclient(hunting): get_auth_request con 0x7fbce0072740 auth_method 0 2024-09-16T03:43:34.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.557+0000 7fbcd5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:43:34.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.557+0000 7fbcd5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:43:34.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.557+0000 7fbcd5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:43:34.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.557+0000 7fbcde59c640 1 --2- 172.21.15.42:0/1148426661 >> v2:172.21.15.154:3300/0 conn(0x7fbce00a2230 0x7fbce00a2610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:43:34.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.557+0000 7fbcde59c640 10 monclient(hunting): get_auth_request con 0x7fbce00a2230 auth_method 0 2024-09-16T03:43:34.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.557+0000 7fbcde59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:43:34.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.557+0000 7fbcde59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:43:34.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.557+0000 7fbcde59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:43:34.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.557+0000 7fbcde59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:43:34.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.557+0000 7fbcde59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:43:34.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.557+0000 7fbcde59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:43:34.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.557+0000 7fbcd5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:43:34.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.557+0000 7fbcd5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:43:34.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.557+0000 7fbcd5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:43:34.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.557+0000 7fbcde59c640 10 monclient(hunting): handle_auth_done global_id 14404 payload 274 2024-09-16T03:43:34.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.557+0000 7fbcde59c640 1 -- 172.21.15.42:0/1148426661 >> v2:172.21.15.42:3300/0 conn(0x7fbce0072740 msgr2=0x7fbce00a2b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:43:34.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.557+0000 7fbcde59c640 1 --2- 172.21.15.42:0/1148426661 >> v2:172.21.15.42:3300/0 conn(0x7fbce0072740 0x7fbce00a2b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:43:34.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.557+0000 7fbcde59c640 1 -- 172.21.15.42:0/1148426661 >> v1:172.21.15.154:6789/0 conn(0x7fbce0134ff0 legacy=0x7fbce0135460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:43:34.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.557+0000 7fbcde59c640 10 monclient: _finish_hunting 0 2024-09-16T03:43:34.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.557+0000 7fbcde59c640 1 monclient: found mon.noname-c 2024-09-16T03:43:34.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.557+0000 7fbcde59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:43:34.568 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.557+0000 7fbcde59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.154:3300/0 2024-09-16T03:43:34.568 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.557+0000 7fbcde59c640 1 -- 172.21.15.42:0/1148426661 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbce006c3b0 con 0x7fbce00a2230 2024-09-16T03:43:34.568 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.547+0000 7f9102191880 20 client.14182.objecter get_session s=0x5607e0653f50 osd=1 3 2024-09-16T03:43:34.568 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.547+0000 7f9102191880 15 client.14182.objecter _session_op_assign 1 19 2024-09-16T03:43:34.568 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.547+0000 7f9102191880 15 client.14182.objecter _send_op 19 to 31.1 on osd.1 2024-09-16T03:43:34.568 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.547+0000 7f9102191880 1 -- 172.21.15.154:0/332177798 --> v2:172.21.15.42:6800/4217863575 -- osd_op(unknown.0.0:19 31.1 31:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22) -- 0x5607e06fa360 con 0x5607e06f6d30 2024-09-16T03:43:34.569 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.547+0000 7f9102191880 20 client.14182.objecter put_session s=0x5607e0653f50 osd=1 4 2024-09-16T03:43:34.569 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.547+0000 7f9102191880 5 client.14182.objecter 1 in flight 2024-09-16T03:43:34.569 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.559+0000 7f91004a6640 1 -- 172.21.15.154:0/332177798 <== osd.1 v2:172.21.15.42:6800/4217863575 19 ==== osd_op_reply(19 exist [writefull 0~5] v22'6 uv6 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f90f80138d0 con 0x5607e06f6d30 2024-09-16T03:43:34.569 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.559+0000 7f91004a6640 10 client.14182.objecter ms_dispatch 0x5607e05160c0 osd_op_reply(19 exist [writefull 0~5] v22'6 uv6 ondisk = 0) 2024-09-16T03:43:34.569 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.559+0000 7f91004a6640 10 client.14182.objecter in handle_osd_op_reply 2024-09-16T03:43:34.569 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.559+0000 7f91004a6640 7 client.14182.objecter handle_osd_op_reply 19 ondisk uv 6 in 31.1 attempt 0 2024-09-16T03:43:34.569 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.559+0000 7f91004a6640 10 client.14182.objecter op 0 rval 0 len 0 2024-09-16T03:43:34.569 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.559+0000 7f91004a6640 15 client.14182.objecter handle_osd_op_reply completed tid 19 2024-09-16T03:43:34.569 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.559+0000 7f91004a6640 15 client.14182.objecter _finish_op 19 2024-09-16T03:43:34.569 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.559+0000 7f91004a6640 20 client.14182.objecter put_session s=0x5607e0653f50 osd=1 4 2024-09-16T03:43:34.569 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.559+0000 7f91004a6640 15 client.14182.objecter _session_op_remove 1 19 2024-09-16T03:43:34.569 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.559+0000 7f91004a6640 5 client.14182.objecter 0 in flight 2024-09-16T03:43:34.569 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.559+0000 7f91004a6640 20 librados: complete_aio_write 0x5607e06f9c60 2024-09-16T03:43:34.569 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationFuture (12 ms) 2024-09-16T03:43:34.570 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationYield 2024-09-16T03:43:34.570 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.559+0000 7f9102191880 20 librados: queue_aio_write 0x5607e050b7a0 completion 0x5607e06f9c60 write_seq 6 2024-09-16T03:43:34.570 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.559+0000 7f9102191880 10 client.14182.objecter _op_submit op 0x5607e06fad70 2024-09-16T03:43:34.570 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.559+0000 7f9102191880 20 client.14182.objecter _calc_target epoch 22 base exist @31 precalc_pgid 0 pgid 0.0 is_write 2024-09-16T03:43:34.570 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.559+0000 7f9102191880 20 client.14182.objecter _calc_target target exist @31 -> pgid 31.d3a59201 2024-09-16T03:43:34.570 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.559+0000 7f9102191880 10 client.14182.objecter _calc_target raw pgid 31.d3a59201 -> actual 31.1 acting [1,4] primary 1 2024-09-16T03:43:34.570 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.559+0000 7f9102191880 20 client.14182.objecter _get_session s=0x5607e0653f50 osd=1 3 2024-09-16T03:43:34.570 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.559+0000 7f9102191880 10 client.14182.objecter _op_submit oid exist '@31' '@31' [writefull 0~5 in=5b] tid 20 osd.1 2024-09-16T03:43:34.570 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.559+0000 7f9102191880 20 client.14182.objecter get_session s=0x5607e0653f50 osd=1 3 2024-09-16T03:43:34.570 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.559+0000 7f9102191880 15 client.14182.objecter _session_op_assign 1 20 2024-09-16T03:43:34.570 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.559+0000 7f9102191880 15 client.14182.objecter _send_op 20 to 31.1 on osd.1 2024-09-16T03:43:34.575 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.559+0000 7f9102191880 1 -- 172.21.15.154:0/332177798 --> v2:172.21.15.42:6800/4217863575 -- osd_op(unknown.0.0:20 31.1 31:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22) -- 0x5607e06fa360 con 0x5607e06f6d30 2024-09-16T03:43:34.576 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.559+0000 7f9102191880 20 client.14182.objecter put_session s=0x5607e0653f50 osd=1 4 2024-09-16T03:43:34.576 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.559+0000 7f9102191880 5 client.14182.objecter 1 in flight 2024-09-16T03:43:34.576 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.571+0000 7f91004a6640 1 -- 172.21.15.154:0/332177798 <== osd.1 v2:172.21.15.42:6800/4217863575 20 ==== osd_op_reply(20 exist [writefull 0~5] v22'7 uv7 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f90f80138d0 con 0x5607e06f6d30 2024-09-16T03:43:34.576 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.571+0000 7f91004a6640 10 client.14182.objecter ms_dispatch 0x5607e05160c0 osd_op_reply(20 exist [writefull 0~5] v22'7 uv7 ondisk = 0) 2024-09-16T03:43:34.576 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.571+0000 7f91004a6640 10 client.14182.objecter in handle_osd_op_reply 2024-09-16T03:43:34.576 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.571+0000 7f91004a6640 7 client.14182.objecter handle_osd_op_reply 20 ondisk uv 7 in 31.1 attempt 0 2024-09-16T03:43:34.576 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.571+0000 7f91004a6640 10 client.14182.objecter op 0 rval 0 len 0 2024-09-16T03:43:34.576 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.571+0000 7f91004a6640 15 client.14182.objecter handle_osd_op_reply completed tid 20 2024-09-16T03:43:34.576 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.571+0000 7f91004a6640 15 client.14182.objecter _finish_op 20 2024-09-16T03:43:34.576 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.571+0000 7f91004a6640 20 client.14182.objecter put_session s=0x5607e0653f50 osd=1 4 2024-09-16T03:43:34.576 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.571+0000 7f91004a6640 15 client.14182.objecter _session_op_remove 1 20 2024-09-16T03:43:34.576 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.571+0000 7f91004a6640 5 client.14182.objecter 0 in flight 2024-09-16T03:43:34.576 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.571+0000 7f91004a6640 20 librados: complete_aio_write 0x5607e06f9c60 2024-09-16T03:43:34.577 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationYield (9 ms) 2024-09-16T03:43:34.577 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.571+0000 7f9102191880 10 librados: watch_flush enter 2024-09-16T03:43:34.577 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.571+0000 7f9102191880 10 librados: watch_flush exit 2024-09-16T03:43:34.577 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.571+0000 7f9102191880 10 client.14182.objecter close_session for osd.1 2024-09-16T03:43:34.577 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.571+0000 7f9102191880 1 -- 172.21.15.154:0/332177798 >> v2:172.21.15.42:6800/4217863575 conn(0x5607e06f6d30 msgr2=0x5607e06f9120 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:43:34.577 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.571+0000 7f9102191880 1 --2- 172.21.15.154:0/332177798 >> v2:172.21.15.42:6800/4217863575 conn(0x5607e06f6d30 0x5607e06f9120 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:43:34.577 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.571+0000 7f9102191880 20 client.14182.objecter put_session s=0x5607e0653f50 osd=1 1 2024-09-16T03:43:34.577 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.571+0000 7f9102191880 20 client.14182.objecter shutdown clearing up homeless session... 2024-09-16T03:43:34.577 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.571+0000 7f9102191880 10 client.14182.objecter successfully canceled tick 2024-09-16T03:43:34.577 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.571+0000 7f9102191880 1 -- 172.21.15.154:0/332177798 >> v2:172.21.15.154:6812/254845342 conn(0x7f90ec035a30 msgr2=0x7f90ec037ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:43:34.577 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.571+0000 7f9102191880 1 --2- 172.21.15.154:0/332177798 >> v2:172.21.15.154:6812/254845342 conn(0x7f90ec035a30 0x7f90ec037ef0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f90f4000f70 tx=0x7f90f4010040 comp rx=0 tx=0).stop 2024-09-16T03:43:34.577 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.571+0000 7f9102191880 10 monclient: shutdown 2024-09-16T03:43:34.577 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.571+0000 7f9102191880 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:43:34.578 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.561+0000 7fbcde59c640 1 --2- 172.21.15.42:0/1148426661 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbce00a2230 0x7fbce00a2610 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fbcd00097c0 tx=0x7fbcd001c320 comp rx=0 tx=0).ready entity=mon.1 client_cookie=52c3e57b2b0fa20c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:43:34.578 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.561+0000 7fbcddd9b640 1 -- 172.21.15.42:0/1148426661 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcd00b0070 con 0x7fbce00a2230 2024-09-16T03:43:34.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.561+0000 7fbcddd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:43:34.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.561+0000 7fbcddd9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:43:34.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.561+0000 7fbcddd9b640 10 monclient: dump: 2024-09-16T03:43:34.579 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:43:34.579 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:43:34.579 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:43:34.579 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:43:34.579 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:43:34.579 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:43:34.579 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:43:34.579 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:43:34.579 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:43:34.579 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:43:34.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.561+0000 7fbcddd9b640 10 monclient: _finish_auth 0 2024-09-16T03:43:34.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.561+0000 7fbcddd9b640 10 monclient: _check_auth_tickets 2024-09-16T03:43:34.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.561+0000 7fbcddd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:43:34.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.561+0000 7fbcddd9b640 1 -- 172.21.15.42:0/1148426661 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbcd00ab600 con 0x7fbce00a2230 2024-09-16T03:43:34.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.561+0000 7fbce4dc5640 5 monclient: authenticate success, global_id 14404 2024-09-16T03:43:34.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.561+0000 7fbce4dc5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:43:34.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.561+0000 7fbcddd9b640 10 monclient: handle_config config(0 keys) 2024-09-16T03:43:34.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.561+0000 7fbce4dc5640 10 monclient: get_monmap_and_config success 2024-09-16T03:43:34.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.561+0000 7fbcddd9b640 1 -- 172.21.15.42:0/1148426661 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcd00ab9d0 con 0x7fbce00a2230 2024-09-16T03:43:34.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.561+0000 7fbcddd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:43:34.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.561+0000 7fbce4dc5640 4 set_mon_vals no callback set 2024-09-16T03:43:34.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.561+0000 7fbcddd9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:43:34.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.561+0000 7fbcddd9b640 10 monclient: dump: 2024-09-16T03:43:34.581 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:43:34.581 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:43:34.581 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:43:34.581 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:43:34.581 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:43:34.581 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:43:34.581 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:43:34.581 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:43:34.581 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:43:34.581 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:43:34.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.561+0000 7fbce4dc5640 10 monclient: shutdown 2024-09-16T03:43:34.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.561+0000 7fbce4dc5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:43:34.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.561+0000 7fbce4dc5640 1 -- 172.21.15.42:0/1148426661 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbce00a2230 msgr2=0x7fbce00a2610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:43:34.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.561+0000 7fbce4dc5640 1 --2- 172.21.15.42:0/1148426661 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbce00a2230 0x7fbce00a2610 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fbcd00097c0 tx=0x7fbcd001c320 comp rx=0 tx=0).stop 2024-09-16T03:43:34.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.561+0000 7fbce4dc5640 1 -- 172.21.15.42:0/1148426661 shutdown_connections 2024-09-16T03:43:34.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.561+0000 7fbce4dc5640 1 --2- 172.21.15.42:0/1148426661 >> v2:172.21.15.42:3300/0 conn(0x7fbce0072740 0x7fbce00a2b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:43:34.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.561+0000 7fbce4dc5640 1 --2- 172.21.15.42:0/1148426661 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbce00a2230 0x7fbce00a2610 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:43:34.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.561+0000 7fbce4dc5640 1 -- 172.21.15.42:0/1148426661 >> 172.21.15.42:0/1148426661 conn(0x7fbce00a6e80 msgr2=0x7fbce00a3320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:43:34.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.561+0000 7fbce4dc5640 1 -- 172.21.15.42:0/1148426661 shutdown_connections 2024-09-16T03:43:34.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.561+0000 7fbce4dc5640 1 -- 172.21.15.42:0/1148426661 wait complete. 2024-09-16T03:43:34.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.561+0000 7fbce4dc5640 10 monclient: build_initial_monmap 2024-09-16T03:43:34.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.561+0000 7fbce4dc5640 10 monclient: monmap: 2024-09-16T03:43:34.582 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:43:34.582 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:43:34.582 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:43:34.582 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:43:34.582 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:43:34.583 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:43:34.583 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:43:34.583 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:43:34.583 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:43:34.583 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:43:34.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.561+0000 7fbce4dc5640 1 librados: starting msgr at 2024-09-16T03:43:34.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.561+0000 7fbce4dc5640 1 librados: starting objecter 2024-09-16T03:43:34.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.561+0000 7fbce4dc5640 1 Processor -- start 2024-09-16T03:43:34.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.561+0000 7fbce4dc5640 1 -- start start 2024-09-16T03:43:34.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.561+0000 7fbce4dc5640 1 librados: setting wanted keys 2024-09-16T03:43:34.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.561+0000 7fbce4dc5640 1 librados: calling monclient init 2024-09-16T03:43:34.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.561+0000 7fbce4dc5640 10 monclient: init 2024-09-16T03:43:34.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.561+0000 7fbce4dc5640 10 monclient: _reopen_session rank -1 2024-09-16T03:43:34.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.561+0000 7fbce4dc5640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T03:43:34.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.561+0000 7fbce4dc5640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbce00a2230 0x7fbce01cf2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:43:34.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.561+0000 7fbce4dc5640 10 monclient(hunting): picked mon.noname-b con 0x7fbce00a2230 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:43:34.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.561+0000 7fbce4dc5640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbce0134ff0 0x7fbce0134340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:43:34.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.561+0000 7fbce4dc5640 10 monclient(hunting): picked mon.noname-c con 0x7fbce0134ff0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:43:34.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.561+0000 7fbce4dc5640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbce0134880 0x7fbce011ce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:43:34.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.561+0000 7fbce4dc5640 10 monclient(hunting): picked mon.noname-a con 0x7fbce0134880 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:43:34.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.561+0000 7fbce4dc5640 10 monclient(hunting): start opening mon connection 2024-09-16T03:43:34.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.561+0000 7fbce4dc5640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fbce006cba0 con 0x7fbce0134880 2024-09-16T03:43:34.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.561+0000 7fbce4dc5640 10 monclient(hunting): start opening mon connection 2024-09-16T03:43:34.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.561+0000 7fbce4dc5640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fbce00a3090 con 0x7fbce00a2230 2024-09-16T03:43:34.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.561+0000 7fbce4dc5640 10 monclient(hunting): start opening mon connection 2024-09-16T03:43:34.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.561+0000 7fbce4dc5640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fbce013ff00 con 0x7fbce0134ff0 2024-09-16T03:43:34.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.561+0000 7fbce4dc5640 10 monclient(hunting): _renew_subs 2024-09-16T03:43:34.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.561+0000 7fbce4dc5640 10 monclient(hunting): authenticate will time out at 1061.535697s 2024-09-16T03:43:34.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.561+0000 7fbcded9d640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbce0134880 0x7fbce011ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:43:34.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.561+0000 7fbcd5d9b640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbce0134ff0 0x7fbce0134340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:43:34.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.561+0000 7fbcd5d9b640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbce0134ff0 0x7fbce0134340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:55552/0 (socket says 172.21.15.42:55552) 2024-09-16T03:43:34.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.561+0000 7fbcd5d9b640 1 -- 172.21.15.42:0/2909514699 learned_addr learned my addr 172.21.15.42:0/2909514699 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:43:34.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.561+0000 7fbcd5d9b640 10 monclient(hunting): get_auth_request con 0x7fbce0134ff0 auth_method 0 2024-09-16T03:43:34.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.561+0000 7fbcd5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:43:34.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.561+0000 7fbcd5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:43:34.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.561+0000 7fbcd5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:43:34.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.561+0000 7fbcded9d640 10 monclient(hunting): get_auth_request con 0x7fbce0134880 auth_method 0 2024-09-16T03:43:34.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.561+0000 7fbcded9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:43:34.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.561+0000 7fbcded9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:43:34.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.561+0000 7fbcded9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:43:34.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.561+0000 7fbcd5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:43:34.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.561+0000 7fbcd5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:43:34.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.561+0000 7fbcd5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:43:34.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.561+0000 7fbcded9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:43:34.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.561+0000 7fbcded9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:43:34.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.565+0000 7fbcded9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:43:34.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.565+0000 7fbcd5d9b640 10 monclient(hunting): handle_auth_done global_id 4691 payload 824 2024-09-16T03:43:34.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.565+0000 7fbcd5d9b640 1 -- 172.21.15.42:0/2909514699 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbce00a2230 msgr2=0x7fbce01cf2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:43:34.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.565+0000 7fbcd5d9b640 1 --2- 172.21.15.42:0/2909514699 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbce00a2230 0x7fbce01cf2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:43:34.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.565+0000 7fbcd5d9b640 1 -- 172.21.15.42:0/2909514699 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbce0134880 msgr2=0x7fbce011ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:43:34.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.565+0000 7fbcd5d9b640 1 --2- 172.21.15.42:0/2909514699 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbce0134880 0x7fbce011ce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:43:34.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.565+0000 7fbcd5d9b640 10 monclient: _finish_hunting 0 2024-09-16T03:43:34.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.565+0000 7fbcd5d9b640 1 monclient: found mon.noname-c 2024-09-16T03:43:34.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.565+0000 7fbcd5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:43:34.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.565+0000 7fbcd5d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:43:34.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.565+0000 7fbcd5d9b640 1 -- 172.21.15.42:0/2909514699 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbce00a6300 con 0x7fbce0134ff0 2024-09-16T03:43:34.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.565+0000 7fbcded9d640 1 --2- 172.21.15.42:0/2909514699 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbce0134880 0x7fbce011ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:43:34.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.565+0000 7fbcd5d9b640 1 --2- 172.21.15.42:0/2909514699 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbce0134ff0 0x7fbce0134340 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fbcc800e930 tx=0x7fbcc800edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:43:34.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.565+0000 7fbcd77fe640 10 client.?.objecter ms_handle_connect 0x7fbce0134ff0 2024-09-16T03:43:34.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.565+0000 7fbcd77fe640 10 client.?.objecter resend_mon_ops 2024-09-16T03:43:34.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.565+0000 7fbcd77fe640 1 -- 172.21.15.42:0/2909514699 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcc8017070 con 0x7fbce0134ff0 2024-09-16T03:43:34.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.565+0000 7fbcd77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:43:34.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.565+0000 7fbcd77fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:43:34.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.565+0000 7fbcd77fe640 10 monclient: dump: 2024-09-16T03:43:34.588 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:43:34.588 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:43:34.588 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:43:34.588 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:43:34.588 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:43:34.588 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:43:34.588 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:43:34.588 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:43:34.588 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:43:34.588 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:43:34.588 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.565+0000 7fbcd77fe640 10 monclient: _finish_auth 0 2024-09-16T03:43:34.588 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.565+0000 7fbcd77fe640 10 monclient: _check_auth_tickets 2024-09-16T03:43:34.588 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.565+0000 7fbcd77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:43:34.589 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.565+0000 7fbcd77fe640 1 -- 172.21.15.42:0/2909514699 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbcc800ca00 con 0x7fbce0134ff0 2024-09-16T03:43:34.589 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.565+0000 7fbce4dc5640 5 monclient: authenticate success, global_id 4691 2024-09-16T03:43:34.589 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.565+0000 7fbce4dc5640 10 monclient: _renew_subs 2024-09-16T03:43:34.589 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.565+0000 7fbce4dc5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:43:34.589 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.565+0000 7fbce4dc5640 1 -- 172.21.15.42:0/2909514699 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbce0120cf0 con 0x7fbce0134ff0 2024-09-16T03:43:34.589 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.565+0000 7fbcd77fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:43:34.589 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.565+0000 7fbcd77fe640 1 -- 172.21.15.42:0/2909514699 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcc800cdb0 con 0x7fbce0134ff0 2024-09-16T03:43:34.589 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.565+0000 7fbcd77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:43:34.589 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.565+0000 7fbcd77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:43:34.589 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.565+0000 7fbce4dc5640 10 client.4691.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:43:34.589 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.565+0000 7fbcdd59a640 4 set_mon_vals no callback set 2024-09-16T03:43:34.589 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.565+0000 7fbcd77fe640 10 monclient: dump: 2024-09-16T03:43:34.589 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:43:34.590 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:43:34.590 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:43:34.590 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:43:34.590 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:43:34.590 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:43:34.590 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:43:34.590 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:43:34.590 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:43:34.590 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:43:34.590 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.565+0000 7fbce4dc5640 10 monclient: _renew_subs 2024-09-16T03:43:34.590 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.565+0000 7fbce4dc5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:43:34.590 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.565+0000 7fbce4dc5640 1 -- 172.21.15.42:0/2909514699 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fbce0120a10 con 0x7fbce0134ff0 2024-09-16T03:43:34.590 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.565+0000 7fbce4dc5640 1 librados: init done 2024-09-16T03:43:34.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.565+0000 7fbcd77fe640 1 -- 172.21.15.42:0/2909514699 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fbcc801a070 con 0x7fbce0134ff0 2024-09-16T03:43:34.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.565+0000 7fbcd77fe640 1 --2- 172.21.15.42:0/2909514699 >> v2:172.21.15.154:6812/254845342 conn(0x7fbcc0037810 0x7fbcc0039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:43:34.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.565+0000 7fbcd77fe640 1 -- 172.21.15.42:0/2909514699 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(22..22 src has 1..22) ==== 11679+0+0 (secure 0 0 0) 0x7fbcc804ab50 con 0x7fbce0134ff0 2024-09-16T03:43:34.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.565+0000 7fbcd77fe640 10 client.4691.objecter ms_dispatch 0x7fbce0000ba0 osd_map(22..22 src has 1..22) 2024-09-16T03:43:34.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.565+0000 7fbcd77fe640 3 client.4691.objecter handle_osd_map got epochs [22,22] > 0 2024-09-16T03:43:34.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.565+0000 7fbcd77fe640 3 client.4691.objecter handle_osd_map decoding full epoch 22 2024-09-16T03:43:34.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.565+0000 7fbcd77fe640 20 client.4691.objecter dump_active .. 0 homeless 2024-09-16T03:43:34.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.565+0000 7fbcde59c640 1 --2- 172.21.15.42:0/2909514699 >> v2:172.21.15.154:6812/254845342 conn(0x7fbcc0037810 0x7fbcc0039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:43:34.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.565+0000 7fbcde59c640 10 monclient: get_auth_request con 0x7fbcc0037810 auth_method 0 2024-09-16T03:43:34.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.565+0000 7fbce4dc5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:43:34.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.565+0000 7fbce4dc5640 1 --2- 172.21.15.42:0/2909514699 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbce0072740 0x7fbce002c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:43:34.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.565+0000 7fbce4dc5640 10 monclient(hunting): start opening mon connection 2024-09-16T03:43:34.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.565+0000 7fbce4dc5640 1 -- 172.21.15.42:0/2909514699 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fbce011f5d0 con 0x7fbce0072740 2024-09-16T03:43:34.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.565+0000 7fbcded9d640 1 --2- 172.21.15.42:0/2909514699 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbce0072740 0x7fbce002c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:43:34.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.565+0000 7fbcded9d640 10 monclient: get_auth_request con 0x7fbce0072740 auth_method 0 2024-09-16T03:43:34.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.565+0000 7fbcded9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:43:34.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.565+0000 7fbcded9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:43:34.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.565+0000 7fbcded9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:43:34.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.565+0000 7fbcded9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:43:34.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.565+0000 7fbcded9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:43:34.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.565+0000 7fbcded9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:43:34.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.565+0000 7fbcded9d640 10 monclient(hunting): handle_auth_done global_id 4857 payload 824 2024-09-16T03:43:34.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.565+0000 7fbcded9d640 1 -- 172.21.15.42:0/2909514699 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbce00738a0 con 0x7fbce0072740 2024-09-16T03:43:34.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.565+0000 7fbcd77fe640 10 client.4691.objecter ms_handle_connect 0x7fbce0072740 2024-09-16T03:43:34.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.565+0000 7fbcd77fe640 10 client.4691.objecter resend_mon_ops 2024-09-16T03:43:34.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.565+0000 7fbcded9d640 1 --2- 172.21.15.42:0/2909514699 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbce0072740 0x7fbce002c650 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fbcd80012f0 tx=0x7fbcd800d810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:43:34.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.569+0000 7fbcde59c640 1 --2- 172.21.15.42:0/2909514699 >> v2:172.21.15.154:6812/254845342 conn(0x7fbcc0037810 0x7fbcc0039cd0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fbcd00ad000 tx=0x7fbcd0004740 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:43:34.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.569+0000 7fbcd77fe640 10 client.4691.objecter ms_handle_connect 0x7fbcc0037810 2024-09-16T03:43:34.593 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.571+0000 7f9102191880 1 -- 172.21.15.154:0/332177798 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5607e05bda50 msgr2=0x5607e0590bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:43:34.594 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.571+0000 7f9102191880 1 --2- 172.21.15.154:0/332177798 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x5607e05bda50 0x5607e0590bf0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f90f000a500 tx=0x7f90f0018050 comp rx=0 tx=0).stop 2024-09-16T03:43:34.594 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.571+0000 7f91004a6640 1 -- 172.21.15.154:0/332177798 reap_dead start 2024-09-16T03:43:34.594 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.571+0000 7f9102191880 1 -- 172.21.15.154:0/332177798 shutdown_connections 2024-09-16T03:43:34.594 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.571+0000 7f9102191880 1 -- 172.21.15.154:0/332177798 >> 172.21.15.154:0/332177798 conn(0x5607e05988f0 msgr2=0x5607e05bc4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:43:34.594 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.571+0000 7f9102191880 1 -- 172.21.15.154:0/332177798 shutdown_connections 2024-09-16T03:43:34.594 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.571+0000 7f9102191880 1 -- 172.21.15.154:0/332177798 wait complete. 2024-09-16T03:43:34.594 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.571+0000 7f9102191880 1 librados: shutdown 2024-09-16T03:43:34.594 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: [----------] 12 tests from AsioRados (144 ms total) 2024-09-16T03:43:34.594 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: 2024-09-16T03:43:34.594 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: [----------] Global test environment tear-down 2024-09-16T03:43:34.594 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: [==========] 12 tests from 1 test suite ran. (3403 ms total) 2024-09-16T03:43:34.594 INFO:tasks.workunit.client.0.smithi154.stdout: api_asio: [ PASSED ] 12 tests. 2024-09-16T03:43:34.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.589+0000 7fbcd77fe640 1 -- 172.21.15.42:0/2909514699 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcd8019070 con 0x7fbce0072740 2024-09-16T03:43:34.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.589+0000 7fbcd77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:43:34.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.589+0000 7fbcd77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:43:34.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.589+0000 7fbcd77fe640 10 monclient: dump: 2024-09-16T03:43:34.595 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:43:34.596 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:43:34.596 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:43:34.596 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:43:34.596 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:43:34.596 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:43:34.596 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:43:34.596 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:43:34.596 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:43:34.596 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:43:34.630 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.625+0000 7fbcd77fe640 1 -- 172.21.15.42:0/2909514699 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fbcd8007320 con 0x7fbce0072740 2024-09-16T03:43:34.631 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.625+0000 7fbcd77fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:43:34.631 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.625+0000 7fbcd77fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:43:34.631 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.625+0000 7fbcd77fe640 1 -- 172.21.15.42:0/2909514699 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbce0072740 msgr2=0x7fbce002c650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:43:34.631 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.625+0000 7fbcd77fe640 1 --2- 172.21.15.42:0/2909514699 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbce0072740 0x7fbce002c650 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fbcd80012f0 tx=0x7fbcd800d810 comp rx=0 tx=0).stop 2024-09-16T03:43:34.631 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.625+0000 7fbcd77fe640 1 -- 172.21.15.42:0/2909514699 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbce0072740 msgr2=0x7fbce002c650 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:43:34.632 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.625+0000 7fbcd77fe640 1 --2- 172.21.15.42:0/2909514699 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbce0072740 0x7fbce002c650 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:43:34.640 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.633+0000 7fbce4dc5640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:43:34.640 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.633+0000 7fbce4dc5640 1 --2- 172.21.15.42:0/2909514699 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbce002cb90 0x7fbce002cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:43:34.640 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.633+0000 7fbce4dc5640 10 monclient(hunting): start opening mon connection 2024-09-16T03:43:34.640 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.633+0000 7fbce4dc5640 1 -- 172.21.15.42:0/2909514699 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fbce0135460 con 0x7fbce002cb90 2024-09-16T03:43:34.640 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.633+0000 7fbcded9d640 1 --2- 172.21.15.42:0/2909514699 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbce002cb90 0x7fbce002cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:43:34.640 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.633+0000 7fbcded9d640 10 monclient: get_auth_request con 0x7fbce002cb90 auth_method 0 2024-09-16T03:43:34.640 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.633+0000 7fbcded9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:43:34.640 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.633+0000 7fbcded9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:43:34.640 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.633+0000 7fbcded9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:43:34.640 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.633+0000 7fbcded9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:43:34.640 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.633+0000 7fbcded9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:43:34.640 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.633+0000 7fbcded9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:43:34.641 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.633+0000 7fbcded9d640 10 monclient(hunting): handle_auth_done global_id 4866 payload 824 2024-09-16T03:43:34.641 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.633+0000 7fbcded9d640 1 -- 172.21.15.42:0/2909514699 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fbce01207f0 con 0x7fbce002cb90 2024-09-16T03:43:34.641 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.633+0000 7fbcded9d640 1 --2- 172.21.15.42:0/2909514699 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbce002cb90 0x7fbce002cfa0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fbcd80012f0 tx=0x7fbcd8005b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:43:34.641 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.633+0000 7fbcd77fe640 10 client.4691.objecter ms_handle_connect 0x7fbce002cb90 2024-09-16T03:43:34.641 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.633+0000 7fbcd77fe640 10 client.4691.objecter resend_mon_ops 2024-09-16T03:43:34.651 INFO:tasks.workunit.client.0.smithi154.stdout: list_parallel: process_1_[28466]: starting. 2024-09-16T03:43:34.651 INFO:tasks.workunit.client.0.smithi154.stdout: list_parallel: process_1_[28466]: creating pool ceph_test_rados_list_parallel.smithi154-28425 2024-09-16T03:43:34.651 INFO:tasks.workunit.client.0.smithi154.stdout: list_parallel: process_1_[28466]: created object 0... 2024-09-16T03:43:34.651 INFO:tasks.workunit.client.0.smithi154.stdout: list_parallel: process_1_[28466]: created object 25... 2024-09-16T03:43:34.651 INFO:tasks.workunit.client.0.smithi154.stdout: list_parallel: process_1_[28466]: created object 49... 2024-09-16T03:43:34.651 INFO:tasks.workunit.client.0.smithi154.stdout: list_parallel: process_1_[28466]: finishing. 2024-09-16T03:43:34.651 INFO:tasks.workunit.client.0.smithi154.stdout: list_parallel: process_1_[28466]: shutting down. 2024-09-16T03:43:34.651 INFO:tasks.workunit.client.0.smithi154.stdout: list_parallel: process_2_[28467]: starting. 2024-09-16T03:43:34.651 INFO:tasks.workunit.client.0.smithi154.stdout: list_parallel: process_2_[28467]: listing objects. 2024-09-16T03:43:34.651 INFO:tasks.workunit.client.0.smithi154.stdout: list_parallel: process_2_[28467]: listed object 0... 2024-09-16T03:43:34.651 INFO:tasks.workunit.client.0.smithi154.stdout: list_parallel: process_2_[28467]: listed object 25... 2024-09-16T03:43:34.652 INFO:tasks.workunit.client.0.smithi154.stdout: list_parallel: process_2_[28467]: saw 50 objects 2024-09-16T03:43:34.652 INFO:tasks.workunit.client.0.smithi154.stdout: list_parallel: process_2_[28467]: shutting down. 2024-09-16T03:43:34.652 INFO:tasks.workunit.client.0.smithi154.stdout: list_parallel: ******************************* 2024-09-16T03:43:34.652 INFO:tasks.workunit.client.0.smithi154.stdout: list_parallel: process_3_[29233]: starting. 2024-09-16T03:43:34.652 INFO:tasks.workunit.client.0.smithi154.stdout: list_parallel: process_3_[29233]: creating pool ceph_test_rados_list_parallel.smithi154-28425 2024-09-16T03:43:34.652 INFO:tasks.workunit.client.0.smithi154.stdout: list_parallel: process_3_[29233]: created object 0... 2024-09-16T03:43:34.652 INFO:tasks.workunit.client.0.smithi154.stdout: list_parallel: process_3_[29233]: created object 25... 2024-09-16T03:43:34.652 INFO:tasks.workunit.client.0.smithi154.stdout: list_parallel: process_3_[29233]: created object 49... 2024-09-16T03:43:34.652 INFO:tasks.workunit.client.0.smithi154.stdout: list_parallel: process_3_[29233]: finishing. 2024-09-16T03:43:34.652 INFO:tasks.workunit.client.0.smithi154.stdout: list_parallel: process_3_[29233]: shutting down. 2024-09-16T03:43:34.652 INFO:tasks.workunit.client.0.smithi154.stdout: list_parallel: ******************************* 2024-09-16T03:43:34.652 INFO:tasks.workunit.client.0.smithi154.stdout: list_parallel: process_4_[29234]: starting. 2024-09-16T03:43:34.652 INFO:tasks.workunit.client.0.smithi154.stdout: list_parallel: process_4_[29234]: listing objects. 2024-09-16T03:43:34.652 INFO:tasks.workunit.client.0.smithi154.stdout: list_parallel: process_4_[29234]: listed object 0... 2024-09-16T03:43:34.653 INFO:tasks.workunit.client.0.smithi154.stdout: list_parallel: process_4_[29234]: listed object 25... 2024-09-16T03:43:34.653 INFO:tasks.workunit.client.0.smithi154.stdout: list_parallel: process_4_[29234]: saw 46 objects 2024-09-16T03:43:34.653 INFO:tasks.workunit.client.0.smithi154.stdout: list_parallel: process_4_[29234]: shutting down. 2024-09-16T03:43:34.653 INFO:tasks.workunit.client.0.smithi154.stdout: list_parallel: ******************************* 2024-09-16T03:43:34.653 INFO:tasks.workunit.client.0.smithi154.stdout: list_parallel: process_5_[29235]: starting. 2024-09-16T03:43:34.653 INFO:tasks.workunit.client.0.smithi154.stdout: list_parallel: process_5_[29235]: removed 25 objects... 2024-09-16T03:43:34.653 INFO:tasks.workunit.client.0.smithi154.stdout: list_parallel: process_5_[29235]: removed half of the objects 2024-09-16T03:43:34.653 INFO:tasks.workunit.client.0.smithi154.stdout: list_parallel: process_5_[29235]: removed 50 objects... 2024-09-16T03:43:34.653 INFO:tasks.workunit.client.0.smithi154.stdout: list_parallel: process_5_[29235]: removed 50 objects 2024-09-16T03:43:34.653 INFO:tasks.workunit.client.0.smithi154.stdout: list_parallel: process_5_[29235]: shutting down. 2024-09-16T03:43:34.653 INFO:tasks.workunit.client.0.smithi154.stdout: list_parallel: ******************************* 2024-09-16T03:43:34.653 INFO:tasks.workunit.client.0.smithi154.stdout: list_parallel: ******************************* 2024-09-16T03:43:34.653 INFO:tasks.workunit.client.0.smithi154.stdout: list_parallel: process_6_[29284]: starting. 2024-09-16T03:43:34.653 INFO:tasks.workunit.client.0.smithi154.stdout: list_parallel: process_6_[29284]: creating pool ceph_test_rados_list_parallel.smithi154-28425 2024-09-16T03:43:34.654 INFO:tasks.workunit.client.0.smithi154.stdout: list_parallel: process_6_[29284]: created object 0... 2024-09-16T03:43:34.654 INFO:tasks.workunit.client.0.smithi154.stdout: list_parallel: process_6_[29284]: created object 25... 2024-09-16T03:43:34.654 INFO:tasks.workunit.client.0.smithi154.stdout: list_parallel: process_6_[29284]: created object 49... 2024-09-16T03:43:34.654 INFO:tasks.workunit.client.0.smithi154.stdout: list_parallel: process_6_[29284]: finishing. 2024-09-16T03:43:34.654 INFO:tasks.workunit.client.0.smithi154.stdout: list_parallel: process_6_[29284]: shutting down. 2024-09-16T03:43:34.654 INFO:tasks.workunit.client.0.smithi154.stdout: list_parallel: ******************************* 2024-09-16T03:43:34.654 INFO:tasks.workunit.client.0.smithi154.stdout: list_parallel: ******************************* 2024-09-16T03:43:34.654 INFO:tasks.workunit.client.0.smithi154.stdout: list_parallel: process_7_[29285]: starting. 2024-09-16T03:43:34.654 INFO:tasks.workunit.client.0.smithi154.stdout: list_parallel: process_7_[29285]: listing objects. 2024-09-16T03:43:34.654 INFO:tasks.workunit.client.0.smithi154.stdout: list_parallel: process_7_[29285]: listed object 0... 2024-09-16T03:43:34.654 INFO:tasks.workunit.client.0.smithi154.stdout: list_parallel: process_7_[29285]: listed object 25... 2024-09-16T03:43:34.654 INFO:tasks.workunit.client.0.smithi154.stdout: list_parallel: process_7_[29285]: listed object 50... 2024-09-16T03:43:34.654 INFO:tasks.workunit.client.0.smithi154.stdout: list_parallel: process_7_[29285]: saw 51 objects 2024-09-16T03:43:34.654 INFO:tasks.workunit.client.0.smithi154.stdout: list_parallel: process_7_[29285]: shutting down. 2024-09-16T03:43:34.655 INFO:tasks.workunit.client.0.smithi154.stdout: list_parallel: ******************************* 2024-09-16T03:43:34.655 INFO:tasks.workunit.client.0.smithi154.stdout: list_parallel: ******************************* 2024-09-16T03:43:34.655 INFO:tasks.workunit.client.0.smithi154.stdout: list_parallel: process_8_[29286]: starting. 2024-09-16T03:43:34.655 INFO:tasks.workunit.client.0.smithi154.stdout: list_parallel: process_8_[29286]: added 25 objects... 2024-09-16T03:43:34.655 INFO:tasks.workunit.client.0.smithi154.stdout: list_parallel: process_8_[29286]: added half of the objects 2024-09-16T03:43:34.655 INFO:tasks.workunit.client.0.smithi154.stdout: list_parallel: process_8_[29286]: added 50 objects... 2024-09-16T03:43:34.655 INFO:tasks.workunit.client.0.smithi154.stdout: list_parallel: process_8_[29286]: added 50 objects 2024-09-16T03:43:34.655 INFO:tasks.workunit.client.0.smithi154.stdout: list_parallel: process_8_[29286]: shutting down. 2024-09-16T03:43:34.655 INFO:tasks.workunit.client.0.smithi154.stdout: list_parallel: ******************************* 2024-09-16T03:43:34.655 INFO:tasks.workunit.client.0.smithi154.stdout: list_parallel: ******************************* 2024-09-16T03:43:34.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.661+0000 7fbcd77fe640 1 -- 172.21.15.42:0/2909514699 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcd8019070 con 0x7fbce002cb90 2024-09-16T03:43:34.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.661+0000 7fbcd77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:43:34.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.661+0000 7fbcd77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:43:34.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.661+0000 7fbcd77fe640 10 monclient: dump: 2024-09-16T03:43:34.667 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:43:34.667 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:43:34.667 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:43:34.667 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:43:34.668 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:43:34.668 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:43:34.668 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:43:34.668 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:43:34.668 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:43:34.668 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:43:34.793 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.785+0000 7fbcd77fe640 1 -- 172.21.15.42:0/2909514699 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4714 (secure 0 0 0) 0x7fbcd801b2d0 con 0x7fbce002cb90 2024-09-16T03:43:34.793 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.785+0000 7fbcd77fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:43:34.793 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.785+0000 7fbcd77fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:43:34.794 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.785+0000 7fbcd77fe640 1 -- 172.21.15.42:0/2909514699 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbce002cb90 msgr2=0x7fbce002cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:43:34.794 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.785+0000 7fbcd77fe640 1 --2- 172.21.15.42:0/2909514699 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbce002cb90 0x7fbce002cfa0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fbcd80012f0 tx=0x7fbcd8005b80 comp rx=0 tx=0).stop 2024-09-16T03:43:34.794 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.785+0000 7fbcd77fe640 1 -- 172.21.15.42:0/2909514699 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbce002cb90 msgr2=0x7fbce002cfa0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:43:34.794 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.785+0000 7fbcd77fe640 1 --2- 172.21.15.42:0/2909514699 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbce002cb90 0x7fbce002cfa0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:43:34.794 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:43:34.795 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:43:34.795 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:43:34.795 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "leader", 2024-09-16T03:43:34.795 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 16, 2024-09-16T03:43:34.795 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 94155, 2024-09-16T03:43:34.795 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:43:34.795 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:43:34.795 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T03:43:34.795 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T03:43:34.795 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:43:34.795 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 3, 2024-09-16T03:43:34.795 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:43:34.795 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:43:34.796 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:43:34.796 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:43:34.796 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:43:34.796 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:43:34.796 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:43:34.796 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:43:34.796 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:43:34.796 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:43:34.796 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:43:34.796 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:43:34.796 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:43:34.796 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:43:34.796 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:43:34.797 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:43:34.797 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:43:34.797 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:43:34.797 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:43:34.797 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:43:34.797 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:43:34.797 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:43:34.797 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:43:34.797 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:43:34.797 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:43:34.797 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:43:34.797 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:43:34.797 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:43:34.797 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:43:34.798 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:43:34.798 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:43:34.798 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:43:34.798 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:43:34.798 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:43:34.798 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:43:34.798 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:43:34.798 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:43:34.798 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:43:34.798 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:43:34.798 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:43:34.798 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:43:34.798 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:43:34.798 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:43:34.799 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:43:34.799 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:43:34.799 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:43:34.799 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:43:34.799 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:43:34.799 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:43:34.799 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:43:34.799 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:43:34.799 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:43:34.799 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:43:34.799 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:43:34.799 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:43:34.799 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:43:34.800 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:43:34.800 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:43:34.800 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:43:34.800 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:43:34.800 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:43:34.800 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:43:34.800 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:43:34.800 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:43:34.800 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:43:34.800 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:43:34.800 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:43:34.800 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:43:34.800 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:43:34.800 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:43:34.801 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:43:34.801 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:43:34.801 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:43:34.801 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:43:34.801 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:43:34.801 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:43:34.801 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:43:34.801 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:43:34.801 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:43:34.801 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:43:34.801 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:43:34.801 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:43:34.801 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:43:34.801 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:43:34.802 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:43:34.802 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:43:34.802 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:43:34.802 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:43:34.802 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:43:34.802 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:43:34.802 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:43:34.802 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:43:34.802 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:43:34.802 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:43:34.802 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:43:34.802 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:43:34.802 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:43:34.802 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:43:34.803 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:43:34.803 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:43:34.803 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:43:34.803 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:43:34.803 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:43:34.803 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:43:34.803 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:43:34.803 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:43:34.803 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:43:34.803 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:43:34.803 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:43:34.803 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:43:34.803 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:43:34.803 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:43:34.804 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:43:34.804 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:43:34.804 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:43:34.804 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:43:34.804 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:43:34.804 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:43:34.804 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:43:34.804 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:43:34.804 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:43:34.804 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:43:34.804 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:43:34.804 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:43:34.804 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:43:34.805 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:43:34.805 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:43:34.805 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:43:34.805 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:43:34.805 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:43:34.805 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:43:34.805 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:43:34.805 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:43:34.805 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:43:34.805 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:43:34.805 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:43:34.805 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:43:34.805 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:43:34.805 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:43:34.806 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:43:34.806 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:43:34.806 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T03:43:34.806 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:43:34.806 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:43:34.806 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:43:34.806 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 3 2024-09-16T03:43:34.806 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:43:34.806 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:43:34.806 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:43:34.806 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:43:34.806 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:43:34.806 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:43:34.806 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 14 2024-09-16T03:43:34.807 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:43:34.807 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:43:34.807 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:43:34.807 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:43:34.807 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:43:34.808 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.789+0000 7fbce4dc5640 10 librados: watch_flush enter 2024-09-16T03:43:34.808 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.789+0000 7fbce4dc5640 10 librados: watch_flush exit 2024-09-16T03:43:34.808 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.789+0000 7fbce4dc5640 20 client.4691.objecter shutdown clearing up homeless session... 2024-09-16T03:43:34.808 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.789+0000 7fbce4dc5640 10 client.4691.objecter successfully canceled tick 2024-09-16T03:43:34.808 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.789+0000 7fbce4dc5640 1 -- 172.21.15.42:0/2909514699 >> v2:172.21.15.154:6812/254845342 conn(0x7fbcc0037810 msgr2=0x7fbcc0039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:43:34.808 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.789+0000 7fbce4dc5640 1 --2- 172.21.15.42:0/2909514699 >> v2:172.21.15.154:6812/254845342 conn(0x7fbcc0037810 0x7fbcc0039cd0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fbcd00ad000 tx=0x7fbcd0004740 comp rx=0 tx=0).stop 2024-09-16T03:43:34.808 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.789+0000 7fbce4dc5640 10 monclient: shutdown 2024-09-16T03:43:34.808 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.789+0000 7fbce4dc5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:43:34.808 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.789+0000 7fbce4dc5640 1 -- 172.21.15.42:0/2909514699 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbce0134ff0 msgr2=0x7fbce0134340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:43:34.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.789+0000 7fbcded9d640 1 -- 172.21.15.42:0/2909514699 reap_dead start 2024-09-16T03:43:34.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.789+0000 7fbce4dc5640 1 --2- 172.21.15.42:0/2909514699 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbce0134ff0 0x7fbce0134340 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fbcc800e930 tx=0x7fbcc800edf0 comp rx=0 tx=0).stop 2024-09-16T03:43:34.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.789+0000 7fbce4dc5640 1 -- 172.21.15.42:0/2909514699 shutdown_connections 2024-09-16T03:43:34.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.789+0000 7fbce4dc5640 1 --2- 172.21.15.42:0/2909514699 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbce0134ff0 0x7fbce0134340 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:43:34.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.789+0000 7fbce4dc5640 1 -- 172.21.15.42:0/2909514699 >> 172.21.15.42:0/2909514699 conn(0x7fbce00a6e80 msgr2=0x7fbce013c8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:43:34.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.789+0000 7fbce4dc5640 1 -- 172.21.15.42:0/2909514699 shutdown_connections 2024-09-16T03:43:34.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.789+0000 7fbce4dc5640 1 -- 172.21.15.42:0/2909514699 wait complete. 2024-09-16T03:43:34.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.789+0000 7fbce4dc5640 1 librados: shutdown 2024-09-16T03:43:34.825 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:43:34.825 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T03:43:34.826 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T03:43:34.945 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.937+0000 7f5828d1c640 10 monclient: get_monmap_and_config 2024-09-16T03:43:34.945 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.937+0000 7f5828d1c640 10 monclient: build_initial_monmap 2024-09-16T03:43:34.945 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.937+0000 7f5828d1c640 10 monclient: monmap: 2024-09-16T03:43:34.945 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:43:34.945 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:43:34.945 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:43:34.943642+0000 2024-09-16T03:43:34.945 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:43:34.943642+0000 2024-09-16T03:43:34.945 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:43:34.945 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:43:34.945 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:43:34.946 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:43:34.946 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:43:34.946 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:43:34.946 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:43:34.946 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:43:34.947 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.941+0000 7f5828d1c640 1 Processor -- start 2024-09-16T03:43:34.948 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.941+0000 7f5828d1c640 1 -- start start 2024-09-16T03:43:34.948 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.941+0000 7f5828d1c640 10 monclient: init 2024-09-16T03:43:34.948 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.941+0000 7f5828d1c640 10 monclient: _reopen_session rank -1 2024-09-16T03:43:34.948 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.941+0000 7f5828d1c640 10 monclient: _add_conns ranks=[3,0,4,1,2] 2024-09-16T03:43:34.948 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.941+0000 7f5828d1c640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f582406db90 0x7f58241340c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:43:34.948 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.941+0000 7f5828d1c640 10 monclient(hunting): picked mon.noname-a con 0x7f582406db90 addr v2:172.21.15.42:3300/0 2024-09-16T03:43:34.948 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.941+0000 7f5828d1c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f58240a37f0 addr v1:172.21.15.42:6789/0 2024-09-16T03:43:34.948 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.941+0000 7f5828d1c640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f58240a3c00 0x7f5824137520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:43:34.948 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.941+0000 7f5828d1c640 10 monclient(hunting): picked mon.noname-c con 0x7f58240a3c00 addr v2:172.21.15.154:3300/0 2024-09-16T03:43:34.948 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.941+0000 7f5828d1c640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f582406ece0 con 0x7f58240a37f0 2024-09-16T03:43:34.948 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.941+0000 7f5828d1c640 10 monclient(hunting): start opening mon connection 2024-09-16T03:43:34.948 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.941+0000 7f5828d1c640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f582406df70 con 0x7f582406db90 2024-09-16T03:43:34.948 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.941+0000 7f5828d1c640 10 monclient(hunting): start opening mon connection 2024-09-16T03:43:34.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.941+0000 7f5828d1c640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f582409bf80 con 0x7f58240a3c00 2024-09-16T03:43:34.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.941+0000 7f5828d1c640 10 monclient(hunting): _renew_subs 2024-09-16T03:43:34.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.941+0000 7f5828d1c640 10 monclient(hunting): authenticate will time out at 1061.913200s 2024-09-16T03:43:34.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.941+0000 7f582259c640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f582406db90 0x7f58241340c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:43:34.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.941+0000 7f582259c640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f582406db90 0x7f58241340c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:52020/0 (socket says 172.21.15.42:52020) 2024-09-16T03:43:34.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.941+0000 7f582259c640 1 -- 172.21.15.42:0/2938218479 learned_addr learned my addr 172.21.15.42:0/2938218479 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:43:34.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.941+0000 7f582259c640 10 monclient(hunting): get_auth_request con 0x7f582406db90 auth_method 0 2024-09-16T03:43:34.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.941+0000 7f582259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:43:34.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.941+0000 7f582259c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:43:34.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.941+0000 7f582259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:43:34.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.941+0000 7f5822d9d640 1 --2- 172.21.15.42:0/2938218479 >> v2:172.21.15.154:3300/0 conn(0x7f58240a3c00 0x7f5824137520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:43:34.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.941+0000 7f5822d9d640 10 monclient(hunting): get_auth_request con 0x7f58240a3c00 auth_method 0 2024-09-16T03:43:34.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.941+0000 7f5822d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:43:34.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.941+0000 7f5822d9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:43:34.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.941+0000 7f5822d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:43:34.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.941+0000 7f582259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:43:34.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.941+0000 7f582259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:43:34.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.941+0000 7f582259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:43:34.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.941+0000 7f5822d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:43:34.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.941+0000 7f5822d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:43:34.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.941+0000 7f5822d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:43:34.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.941+0000 7f5822d9d640 10 monclient(hunting): handle_auth_done global_id 14452 payload 274 2024-09-16T03:43:34.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.941+0000 7f5822d9d640 1 -- 172.21.15.42:0/2938218479 >> v2:172.21.15.42:3300/0 conn(0x7f582406db90 msgr2=0x7f58241340c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:43:34.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.941+0000 7f5822d9d640 1 --2- 172.21.15.42:0/2938218479 >> v2:172.21.15.42:3300/0 conn(0x7f582406db90 0x7f58241340c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:43:34.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.941+0000 7f5822d9d640 1 -- 172.21.15.42:0/2938218479 >> v1:172.21.15.42:6789/0 conn(0x7f58240a37f0 legacy=0x7f5824136e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:43:34.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.941+0000 7f5822d9d640 10 monclient: _finish_hunting 0 2024-09-16T03:43:34.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.941+0000 7f5822d9d640 1 monclient: found mon.noname-c 2024-09-16T03:43:34.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.941+0000 7f5822d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:43:34.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.941+0000 7f5822d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.154:3300/0 2024-09-16T03:43:34.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.941+0000 7f5822d9d640 1 -- 172.21.15.42:0/2938218479 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5824134780 con 0x7f58240a3c00 2024-09-16T03:43:34.956 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.945+0000 7f5822d9d640 1 --2- 172.21.15.42:0/2938218479 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f58240a3c00 0x7f5824137520 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f582409bbd0 tx=0x7f581c01d220 comp rx=0 tx=0).ready entity=mon.1 client_cookie=447664ecdfdf7eaf server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:43:34.956 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.945+0000 7f582159a640 1 -- 172.21.15.42:0/2938218479 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f581c0b2070 con 0x7f58240a3c00 2024-09-16T03:43:34.956 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.945+0000 7f582159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:43:34.956 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.945+0000 7f582159a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:43:34.957 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.945+0000 7f582159a640 10 monclient: dump: 2024-09-16T03:43:34.957 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:43:34.957 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:43:34.957 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:43:34.957 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:43:34.957 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:43:34.957 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:43:34.957 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:43:34.957 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:43:34.957 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:43:34.957 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:43:34.957 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.945+0000 7f582159a640 10 monclient: _finish_auth 0 2024-09-16T03:43:34.957 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.945+0000 7f582159a640 10 monclient: _check_auth_tickets 2024-09-16T03:43:34.958 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.945+0000 7f582159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:43:34.958 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.945+0000 7f582159a640 1 -- 172.21.15.42:0/2938218479 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f581c0a58a0 con 0x7f58240a3c00 2024-09-16T03:43:34.958 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.945+0000 7f582159a640 10 monclient: handle_config config(0 keys) 2024-09-16T03:43:34.958 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.945+0000 7f582159a640 1 -- 172.21.15.42:0/2938218479 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f581c0a5c70 con 0x7f58240a3c00 2024-09-16T03:43:34.958 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.945+0000 7f5828d1c640 5 monclient: authenticate success, global_id 14452 2024-09-16T03:43:34.958 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.945+0000 7f5828d1c640 10 monclient: get_monmap_and_config success 2024-09-16T03:43:34.958 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.945+0000 7f5828d1c640 4 set_mon_vals no callback set 2024-09-16T03:43:34.958 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.945+0000 7f582159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:43:34.958 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.945+0000 7f582159a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:43:34.958 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.945+0000 7f582159a640 10 monclient: dump: 2024-09-16T03:43:34.958 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:43:34.958 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:43:34.958 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:43:34.958 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:43:34.959 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:43:34.959 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:43:34.959 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:43:34.959 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:43:34.959 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:43:34.959 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:43:34.959 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.945+0000 7f5828d1c640 10 monclient: shutdown 2024-09-16T03:43:34.959 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.945+0000 7f5828d1c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:43:34.959 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.945+0000 7f5828d1c640 1 -- 172.21.15.42:0/2938218479 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f58240a3c00 msgr2=0x7f5824137520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:43:34.959 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.945+0000 7f5828d1c640 1 --2- 172.21.15.42:0/2938218479 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f58240a3c00 0x7f5824137520 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f582409bbd0 tx=0x7f581c01d220 comp rx=0 tx=0).stop 2024-09-16T03:43:34.959 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.945+0000 7f5828d1c640 1 -- 172.21.15.42:0/2938218479 shutdown_connections 2024-09-16T03:43:34.959 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.945+0000 7f5828d1c640 1 --2- 172.21.15.42:0/2938218479 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f58240a3c00 0x7f5824137520 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:43:34.959 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.945+0000 7f5828d1c640 1 --2- 172.21.15.42:0/2938218479 >> v2:172.21.15.42:3300/0 conn(0x7f582406db90 0x7f58241340c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:43:34.959 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.945+0000 7f5828d1c640 1 -- 172.21.15.42:0/2938218479 >> 172.21.15.42:0/2938218479 conn(0x7f58240a9fb0 msgr2=0x7f58240aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:43:34.960 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.945+0000 7f5828d1c640 1 -- 172.21.15.42:0/2938218479 shutdown_connections 2024-09-16T03:43:34.960 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.945+0000 7f5828d1c640 1 -- 172.21.15.42:0/2938218479 wait complete. 2024-09-16T03:43:34.960 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.945+0000 7f5828d1c640 10 monclient: build_initial_monmap 2024-09-16T03:43:34.960 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.945+0000 7f5828d1c640 10 monclient: monmap: 2024-09-16T03:43:34.960 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:43:34.960 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:43:34.960 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:43:34.960 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:43:34.960 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:43:34.960 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:43:34.960 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:43:34.960 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:43:34.960 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:43:34.961 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:43:34.961 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.945+0000 7f5828d1c640 1 librados: starting msgr at 2024-09-16T03:43:34.961 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.945+0000 7f5828d1c640 1 librados: starting objecter 2024-09-16T03:43:34.961 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.945+0000 7f5828d1c640 1 Processor -- start 2024-09-16T03:43:34.961 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.945+0000 7f5828d1c640 1 -- start start 2024-09-16T03:43:34.961 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.945+0000 7f5828d1c640 1 librados: setting wanted keys 2024-09-16T03:43:34.961 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.945+0000 7f5828d1c640 1 librados: calling monclient init 2024-09-16T03:43:34.961 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.945+0000 7f5828d1c640 10 monclient: init 2024-09-16T03:43:34.961 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.945+0000 7f5828d1c640 10 monclient: _reopen_session rank -1 2024-09-16T03:43:34.961 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.945+0000 7f5828d1c640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:43:34.961 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.945+0000 7f5828d1c640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f582406db90 0x7f58240708b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:43:34.961 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.945+0000 7f5828d1c640 10 monclient(hunting): picked mon.noname-c con 0x7f582406db90 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:43:34.961 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.945+0000 7f5828d1c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f58240a37f0 0x7f58240a17b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:43:34.962 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.945+0000 7f5828d1c640 10 monclient(hunting): picked mon.noname-a con 0x7f58240a37f0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:43:34.962 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.945+0000 7f5828d1c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f58240a3c00 0x7f58240a1cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:43:34.962 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.945+0000 7f5828d1c640 10 monclient(hunting): picked mon.noname-b con 0x7f58240a3c00 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:43:34.962 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.945+0000 7f5828d1c640 10 monclient(hunting): start opening mon connection 2024-09-16T03:43:34.962 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.945+0000 7f5828d1c640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f58240a40d0 con 0x7f58240a37f0 2024-09-16T03:43:34.962 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.945+0000 7f5828d1c640 10 monclient(hunting): start opening mon connection 2024-09-16T03:43:34.962 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.945+0000 7f5828d1c640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f5824137c70 con 0x7f58240a3c00 2024-09-16T03:43:34.962 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.945+0000 7f5828d1c640 10 monclient(hunting): start opening mon connection 2024-09-16T03:43:34.962 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.945+0000 7f5828d1c640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f58241419d0 con 0x7f582406db90 2024-09-16T03:43:34.962 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.945+0000 7f5828d1c640 10 monclient(hunting): _renew_subs 2024-09-16T03:43:34.962 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.945+0000 7f5828d1c640 10 monclient(hunting): authenticate will time out at 1061.919314s 2024-09-16T03:43:34.962 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.945+0000 7f582259c640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f582406db90 0x7f58240708b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:43:34.962 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.945+0000 7f5821d9b640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f58240a37f0 0x7f58240a17b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:43:34.962 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.945+0000 7f582259c640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f582406db90 0x7f58240708b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:55554/0 (socket says 172.21.15.42:55554) 2024-09-16T03:43:34.963 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.945+0000 7f582259c640 1 -- 172.21.15.42:0/2609102902 learned_addr learned my addr 172.21.15.42:0/2609102902 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:43:34.963 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.945+0000 7f5821d9b640 10 monclient(hunting): get_auth_request con 0x7f58240a37f0 auth_method 0 2024-09-16T03:43:34.963 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.945+0000 7f5821d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:43:34.963 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.945+0000 7f5821d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:43:34.963 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.945+0000 7f5821d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:43:34.963 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.945+0000 7f582259c640 10 monclient(hunting): get_auth_request con 0x7f582406db90 auth_method 0 2024-09-16T03:43:34.963 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.945+0000 7f582259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:43:34.963 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.945+0000 7f582259c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:43:34.963 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.945+0000 7f582259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:43:34.963 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.945+0000 7f5822d9d640 1 --2- 172.21.15.42:0/2609102902 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f58240a3c00 0x7f58240a1cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:43:34.963 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.945+0000 7f5822d9d640 10 monclient(hunting): get_auth_request con 0x7f58240a3c00 auth_method 0 2024-09-16T03:43:34.963 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.945+0000 7f5822d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:43:34.963 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.945+0000 7f5822d9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:43:34.964 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.945+0000 7f5822d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:43:34.964 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.945+0000 7f5821d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:43:34.964 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.945+0000 7f5821d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:43:34.964 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.945+0000 7f5821d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:43:34.964 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.945+0000 7f5822d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:43:34.964 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.945+0000 7f5822d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:43:34.964 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.945+0000 7f5822d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:43:34.964 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.945+0000 7f582259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:43:34.964 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.945+0000 7f582259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:43:34.964 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.945+0000 7f582259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:43:34.964 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.945+0000 7f5821d9b640 10 monclient(hunting): handle_auth_done global_id 4893 payload 824 2024-09-16T03:43:34.964 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.945+0000 7f5821d9b640 1 -- 172.21.15.42:0/2609102902 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f582406db90 msgr2=0x7f58240708b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:43:34.964 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.945+0000 7f5821d9b640 1 --2- 172.21.15.42:0/2609102902 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f582406db90 0x7f58240708b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:43:34.964 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.945+0000 7f5821d9b640 1 -- 172.21.15.42:0/2609102902 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f58240a3c00 msgr2=0x7f58240a1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:43:34.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.945+0000 7f5821d9b640 1 --2- 172.21.15.42:0/2609102902 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f58240a3c00 0x7f58240a1cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:43:34.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.945+0000 7f5821d9b640 10 monclient: _finish_hunting 0 2024-09-16T03:43:34.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.945+0000 7f5821d9b640 1 monclient: found mon.noname-a 2024-09-16T03:43:34.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.945+0000 7f5821d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:43:34.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.945+0000 7f5821d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:43:34.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.945+0000 7f5821d9b640 1 -- 172.21.15.42:0/2609102902 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58240a3450 con 0x7f58240a37f0 2024-09-16T03:43:34.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.945+0000 7f582259c640 1 --2- 172.21.15.42:0/2609102902 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f582406db90 0x7f58240708b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:43:34.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.949+0000 7f5821d9b640 1 --2- 172.21.15.42:0/2609102902 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f58240a37f0 0x7f58240a17b0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f581800c3c0 tx=0x7f581800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:43:34.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.949+0000 7f5812ffd640 10 client.?.objecter ms_handle_connect 0x7f58240a37f0 2024-09-16T03:43:34.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.949+0000 7f5812ffd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:43:34.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.949+0000 7f5812ffd640 1 -- 172.21.15.42:0/2609102902 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58180052d0 con 0x7f58240a37f0 2024-09-16T03:43:34.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.949+0000 7f5812ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:43:34.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.949+0000 7f5812ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:43:34.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.949+0000 7f5812ffd640 10 monclient: dump: 2024-09-16T03:43:34.966 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:43:34.966 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:43:34.966 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:43:34.966 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:43:34.966 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:43:34.966 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:43:34.966 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:43:34.966 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:43:34.966 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:43:34.966 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:43:34.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.949+0000 7f5812ffd640 10 monclient: _finish_auth 0 2024-09-16T03:43:34.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.949+0000 7f5812ffd640 10 monclient: _check_auth_tickets 2024-09-16T03:43:34.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.949+0000 7f5812ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:43:34.967 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.949+0000 7f5812ffd640 1 -- 172.21.15.42:0/2609102902 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5818005470 con 0x7f58240a37f0 2024-09-16T03:43:34.967 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.949+0000 7f5812ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:43:34.967 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.949+0000 7f5812ffd640 1 -- 172.21.15.42:0/2609102902 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5818007260 con 0x7f58240a37f0 2024-09-16T03:43:34.967 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.949+0000 7f5812ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:43:34.967 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.949+0000 7f5812ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:43:34.967 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.949+0000 7f5812ffd640 10 monclient: dump: 2024-09-16T03:43:34.967 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:43:34.967 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:43:34.967 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:43:34.967 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:43:34.967 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:43:34.967 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:43:34.967 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:43:34.967 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:43:34.968 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:43:34.968 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:43:34.968 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.949+0000 7f5828d1c640 5 monclient: authenticate success, global_id 4893 2024-09-16T03:43:34.968 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.949+0000 7f5828d1c640 10 monclient: _renew_subs 2024-09-16T03:43:34.968 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.949+0000 7f5828d1c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:43:34.968 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.949+0000 7f5828d1c640 1 -- 172.21.15.42:0/2609102902 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5824071130 con 0x7f58240a37f0 2024-09-16T03:43:34.968 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.949+0000 7f5828d1c640 10 client.4893.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:43:34.968 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.949+0000 7f5828d1c640 10 monclient: _renew_subs 2024-09-16T03:43:34.968 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.949+0000 7f5828d1c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:43:34.968 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.949+0000 7f5828d1c640 1 -- 172.21.15.42:0/2609102902 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5824070240 con 0x7f58240a37f0 2024-09-16T03:43:34.968 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.949+0000 7f5828d1c640 1 librados: init done 2024-09-16T03:43:34.968 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.949+0000 7f5820d99640 4 set_mon_vals no callback set 2024-09-16T03:43:34.968 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.949+0000 7f5828d1c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:43:34.968 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.949+0000 7f5812ffd640 1 -- 172.21.15.42:0/2609102902 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f5818007400 con 0x7f58240a37f0 2024-09-16T03:43:34.969 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.949+0000 7f5828d1c640 1 --2- 172.21.15.42:0/2609102902 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f582402d7d0 0x7f58240a2230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:43:34.969 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.949+0000 7f5828d1c640 10 monclient(hunting): start opening mon connection 2024-09-16T03:43:34.969 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.949+0000 7f5828d1c640 1 -- 172.21.15.42:0/2609102902 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f582402dbb0 con 0x7f582402d7d0 2024-09-16T03:43:34.969 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.949+0000 7f582259c640 1 --2- 172.21.15.42:0/2609102902 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f582402d7d0 0x7f58240a2230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:43:34.969 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.949+0000 7f582259c640 10 monclient: get_auth_request con 0x7f582402d7d0 auth_method 0 2024-09-16T03:43:34.969 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.949+0000 7f582259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:43:34.969 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.949+0000 7f582259c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:43:34.969 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.949+0000 7f582259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:43:34.969 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.949+0000 7f5812ffd640 1 --2- 172.21.15.42:0/2609102902 >> v2:172.21.15.154:6812/254845342 conn(0x7f5804037810 0x7f5804039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:43:34.969 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.949+0000 7f582259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:43:34.969 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.949+0000 7f582259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:43:34.969 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.949+0000 7f582259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:43:34.969 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.949+0000 7f5812ffd640 1 -- 172.21.15.42:0/2609102902 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 11679+0+0 (secure 0 0 0) 0x7f5818049b60 con 0x7f58240a37f0 2024-09-16T03:43:34.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.949+0000 7f5812ffd640 10 client.4893.objecter ms_dispatch 0x7f5824000ba0 osd_map(22..22 src has 1..22) 2024-09-16T03:43:34.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.949+0000 7f5812ffd640 3 client.4893.objecter handle_osd_map got epochs [22,22] > 0 2024-09-16T03:43:34.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.949+0000 7f5812ffd640 3 client.4893.objecter handle_osd_map decoding full epoch 22 2024-09-16T03:43:34.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.949+0000 7f5822d9d640 1 --2- 172.21.15.42:0/2609102902 >> v2:172.21.15.154:6812/254845342 conn(0x7f5804037810 0x7f5804039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:43:34.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.949+0000 7f5812ffd640 20 client.4893.objecter dump_active .. 0 homeless 2024-09-16T03:43:34.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.949+0000 7f5822d9d640 10 monclient: get_auth_request con 0x7f5804037810 auth_method 0 2024-09-16T03:43:34.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.949+0000 7f582259c640 10 monclient(hunting): handle_auth_done global_id 4715 payload 824 2024-09-16T03:43:34.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.949+0000 7f582259c640 1 -- 172.21.15.42:0/2609102902 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f58241340c0 con 0x7f582402d7d0 2024-09-16T03:43:34.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.949+0000 7f582259c640 1 --2- 172.21.15.42:0/2609102902 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f582402d7d0 0x7f58240a2230 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f5814011d60 tx=0x7f581400f610 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:43:34.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.949+0000 7f5812ffd640 10 client.4893.objecter ms_handle_connect 0x7f582402d7d0 2024-09-16T03:43:34.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.949+0000 7f5812ffd640 10 client.4893.objecter resend_mon_ops 2024-09-16T03:43:34.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.949+0000 7f5812ffd640 1 -- 172.21.15.42:0/2609102902 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5814004020 con 0x7f582402d7d0 2024-09-16T03:43:34.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.949+0000 7f5812ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:43:34.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.949+0000 7f5812ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:43:34.971 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.949+0000 7f5812ffd640 10 monclient: dump: 2024-09-16T03:43:34.971 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:43:34.971 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:43:34.971 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:43:34.971 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:43:34.971 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:43:34.971 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:43:34.971 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:43:34.971 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:43:34.971 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:43:34.971 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:43:34.971 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.949+0000 7f5822d9d640 1 --2- 172.21.15.42:0/2609102902 >> v2:172.21.15.154:6812/254845342 conn(0x7f5804037810 0x7f5804039cd0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f581c0af000 tx=0x7f581c004ac0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:43:34.971 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.949+0000 7f5812ffd640 10 client.4893.objecter ms_handle_connect 0x7f5804037810 2024-09-16T03:43:34.971 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.949+0000 7f5812ffd640 1 -- 172.21.15.42:0/2609102902 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f5814004020 con 0x7f582402d7d0 2024-09-16T03:43:34.972 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.949+0000 7f5812ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:43:34.972 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.949+0000 7f5812ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:43:34.972 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.949+0000 7f5812ffd640 1 -- 172.21.15.42:0/2609102902 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f582402d7d0 msgr2=0x7f58240a2230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:43:34.972 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.949+0000 7f5812ffd640 1 --2- 172.21.15.42:0/2609102902 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f582402d7d0 0x7f58240a2230 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f5814011d60 tx=0x7f581400f610 comp rx=0 tx=0).stop 2024-09-16T03:43:34.972 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.949+0000 7f5812ffd640 1 -- 172.21.15.42:0/2609102902 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f582402d7d0 msgr2=0x7f58240a2230 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:43:34.972 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.949+0000 7f5812ffd640 1 --2- 172.21.15.42:0/2609102902 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f582402d7d0 0x7f58240a2230 secure :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f5814011d60 tx=0x7f581400f610 comp rx=0 tx=0).stop 2024-09-16T03:43:34.974 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.961+0000 7f5828d1c640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:43:34.974 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.961+0000 7f5828d1c640 1 --2- 172.21.15.42:0/2609102902 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5824062ea0 0x7f58240632b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:43:34.974 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.961+0000 7f5828d1c640 10 monclient(hunting): start opening mon connection 2024-09-16T03:43:34.974 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.961+0000 7f5828d1c640 1 -- 172.21.15.42:0/2609102902 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f582406f680 con 0x7f5824062ea0 2024-09-16T03:43:34.975 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.961+0000 7f582259c640 1 --2- 172.21.15.42:0/2609102902 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5824062ea0 0x7f58240632b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:43:34.975 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.961+0000 7f582259c640 10 monclient: get_auth_request con 0x7f5824062ea0 auth_method 0 2024-09-16T03:43:34.975 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.961+0000 7f582259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:43:34.975 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.961+0000 7f582259c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:43:34.975 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.961+0000 7f582259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:43:34.975 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.961+0000 7f582259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:43:34.975 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.961+0000 7f582259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:43:34.975 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.961+0000 7f582259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:43:34.975 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.961+0000 7f582259c640 10 monclient(hunting): handle_auth_done global_id 4718 payload 824 2024-09-16T03:43:34.975 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.961+0000 7f582259c640 1 -- 172.21.15.42:0/2609102902 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f582409de20 con 0x7f5824062ea0 2024-09-16T03:43:34.975 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.961+0000 7f5812ffd640 10 client.4893.objecter ms_handle_connect 0x7f5824062ea0 2024-09-16T03:43:34.975 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.961+0000 7f5812ffd640 10 client.4893.objecter resend_mon_ops 2024-09-16T03:43:34.975 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.961+0000 7f582259c640 1 --2- 172.21.15.42:0/2609102902 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5824062ea0 0x7f58240632b0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f5814011d60 tx=0x7f58140046d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:43:34.976 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.961+0000 7f5812ffd640 1 -- 172.21.15.42:0/2609102902 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5814014850 con 0x7f5824062ea0 2024-09-16T03:43:34.976 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.961+0000 7f5812ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:43:34.976 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.961+0000 7f5812ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:43:34.976 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.961+0000 7f5812ffd640 10 monclient: dump: 2024-09-16T03:43:34.976 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:43:34.976 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:43:34.976 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:43:34.976 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:43:34.976 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:43:34.976 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:43:34.976 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:43:34.976 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:43:34.976 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:43:34.976 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:43:34.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.961+0000 7f5812ffd640 1 -- 172.21.15.42:0/2609102902 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4879 (secure 0 0 0) 0x7f58140133a0 con 0x7f5824062ea0 2024-09-16T03:43:34.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.961+0000 7f5812ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:43:34.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.961+0000 7f5812ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:43:34.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.961+0000 7f5812ffd640 1 -- 172.21.15.42:0/2609102902 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5824062ea0 msgr2=0x7f58240632b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:43:34.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.961+0000 7f5812ffd640 1 --2- 172.21.15.42:0/2609102902 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5824062ea0 0x7f58240632b0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f5814011d60 tx=0x7f58140046d0 comp rx=0 tx=0).stop 2024-09-16T03:43:34.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.961+0000 7f5812ffd640 1 -- 172.21.15.42:0/2609102902 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5824062ea0 msgr2=0x7f58240632b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:43:34.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.961+0000 7f5812ffd640 1 --2- 172.21.15.42:0/2609102902 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5824062ea0 0x7f58240632b0 secure :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f5814011d60 tx=0x7f58140046d0 comp rx=0 tx=0).stop 2024-09-16T03:43:34.977 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:43:34.977 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:43:34.977 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:43:34.978 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T03:43:34.978 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 16, 2024-09-16T03:43:34.978 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 94331, 2024-09-16T03:43:34.978 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:43:34.978 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:43:34.978 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T03:43:34.978 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T03:43:34.978 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:43:34.978 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 3, 2024-09-16T03:43:34.978 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:43:34.978 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:43:34.978 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:43:34.978 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:43:34.979 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:43:34.979 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:43:34.979 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:43:34.979 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:43:34.979 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:43:34.979 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:43:34.979 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:43:34.979 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:43:34.979 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:43:34.979 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:43:34.979 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:43:34.979 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:43:34.979 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:43:34.980 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:43:34.980 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:43:34.980 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:43:34.980 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:43:34.980 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:43:34.980 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:43:34.980 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:43:34.980 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:43:34.980 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:43:34.980 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:43:34.980 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:43:34.980 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:43:34.980 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:43:34.980 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:43:34.981 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:43:34.981 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:43:34.981 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:43:34.981 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:43:34.981 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:43:34.981 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:43:34.981 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:43:34.981 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:43:34.981 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:43:34.981 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:43:34.981 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:43:34.981 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:43:34.981 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:43:34.981 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:43:34.982 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:43:34.982 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:43:34.982 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:43:34.982 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:43:34.982 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:43:34.982 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:43:34.982 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:43:34.982 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:43:34.982 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:43:34.982 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:43:34.982 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:43:34.982 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:43:34.982 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:43:34.982 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:43:34.983 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:43:34.983 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:43:34.983 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:43:34.983 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:43:34.983 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:43:34.983 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:43:34.983 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:43:34.983 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:43:34.983 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:43:34.983 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:43:34.983 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:43:34.983 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:43:34.983 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:43:34.983 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:43:34.984 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:43:34.984 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:43:34.984 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:43:34.984 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:43:34.984 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:43:34.984 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:43:34.984 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:43:34.984 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:43:34.984 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:43:34.984 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:43:34.984 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:43:34.984 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:43:34.984 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:43:34.984 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:43:34.985 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:43:34.985 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:43:34.985 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:43:34.985 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:43:34.985 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:43:34.985 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:43:34.985 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:43:34.985 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:43:34.985 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:43:34.985 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:43:34.985 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:43:34.985 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:43:34.985 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:43:34.985 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:43:34.986 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:43:34.986 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:43:34.986 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:43:34.986 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:43:34.986 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:43:34.986 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:43:34.986 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:43:34.986 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:43:34.986 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:43:34.986 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:43:34.986 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:43:34.986 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:43:34.986 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:43:34.987 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:43:34.987 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:43:34.987 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:43:34.987 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:43:34.987 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:43:34.987 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:43:34.987 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:43:34.987 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:43:34.987 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:43:34.987 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:43:34.987 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:43:34.987 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:43:34.987 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:43:34.987 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:43:34.988 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:43:34.988 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:43:34.988 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:43:34.988 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:43:34.988 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:43:34.988 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:43:34.988 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:43:34.988 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:43:34.988 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:43:34.988 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:43:34.988 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:43:34.988 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:43:34.988 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:43:34.988 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T03:43:34.989 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:43:34.989 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:43:34.989 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:43:34.989 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 2 2024-09-16T03:43:34.989 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:43:34.989 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:43:34.989 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:43:34.989 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:43:34.989 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:43:34.989 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:43:34.989 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 16 2024-09-16T03:43:34.989 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:43:34.989 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:43:34.989 INFO:teuthology.orchestra.run.smithi042.stdout: "mgr": [ 2024-09-16T03:43:34.990 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:43:34.990 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:43:34.990 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:43:34.990 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:43:34.990 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:43:34.990 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:43:34.990 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:43:34.990 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:43:34.990 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:43:34.992 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.961+0000 7f5828d1c640 10 librados: watch_flush enter 2024-09-16T03:43:34.992 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.961+0000 7f5828d1c640 10 librados: watch_flush exit 2024-09-16T03:43:34.992 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.961+0000 7f5828d1c640 20 client.4893.objecter shutdown clearing up homeless session... 2024-09-16T03:43:34.992 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.961+0000 7f5828d1c640 10 client.4893.objecter successfully canceled tick 2024-09-16T03:43:34.992 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.961+0000 7f5828d1c640 1 -- 172.21.15.42:0/2609102902 >> v2:172.21.15.154:6812/254845342 conn(0x7f5804037810 msgr2=0x7f5804039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:43:34.992 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.961+0000 7f5828d1c640 1 --2- 172.21.15.42:0/2609102902 >> v2:172.21.15.154:6812/254845342 conn(0x7f5804037810 0x7f5804039cd0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f581c0af000 tx=0x7f581c004ac0 comp rx=0 tx=0).stop 2024-09-16T03:43:34.992 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.961+0000 7f5828d1c640 10 monclient: shutdown 2024-09-16T03:43:34.992 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.961+0000 7f5828d1c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:43:34.992 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.961+0000 7f5828d1c640 1 -- 172.21.15.42:0/2609102902 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f58240a37f0 msgr2=0x7f58240a17b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:43:34.992 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.961+0000 7f5828d1c640 1 --2- 172.21.15.42:0/2609102902 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f58240a37f0 0x7f58240a17b0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f581800c3c0 tx=0x7f581800c880 comp rx=0 tx=0).stop 2024-09-16T03:43:34.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.961+0000 7f5822d9d640 1 -- 172.21.15.42:0/2609102902 reap_dead start 2024-09-16T03:43:34.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.961+0000 7f5822d9d640 1 -- 172.21.15.42:0/2609102902 reap_dead start 2024-09-16T03:43:34.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.961+0000 7f5828d1c640 1 -- 172.21.15.42:0/2609102902 shutdown_connections 2024-09-16T03:43:34.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.961+0000 7f5828d1c640 1 -- 172.21.15.42:0/2609102902 >> 172.21.15.42:0/2609102902 conn(0x7f58240a9fb0 msgr2=0x7f5824136ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:43:34.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.961+0000 7f5828d1c640 1 -- 172.21.15.42:0/2609102902 shutdown_connections 2024-09-16T03:43:34.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.961+0000 7f5828d1c640 1 -- 172.21.15.42:0/2609102902 wait complete. 2024-09-16T03:43:34.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:34.961+0000 7f5828d1c640 1 librados: shutdown 2024-09-16T03:43:34.993 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:43:34.994 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T03:43:34.994 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T03:43:35.112 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.105+0000 7f6917aae640 10 monclient: get_monmap_and_config 2024-09-16T03:43:35.112 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.105+0000 7f6917aae640 10 monclient: build_initial_monmap 2024-09-16T03:43:35.112 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.105+0000 7f6917aae640 10 monclient: monmap: 2024-09-16T03:43:35.112 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:43:35.112 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:43:35.112 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:43:35.110564+0000 2024-09-16T03:43:35.113 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:43:35.110564+0000 2024-09-16T03:43:35.113 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:43:35.113 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:43:35.113 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:43:35.113 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:43:35.113 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:43:35.113 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:43:35.113 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:43:35.113 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:43:35.113 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.105+0000 7f6917aae640 1 Processor -- start 2024-09-16T03:43:35.113 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.105+0000 7f6917aae640 1 -- start start 2024-09-16T03:43:35.113 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.105+0000 7f6917aae640 10 monclient: init 2024-09-16T03:43:35.115 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.105+0000 7f6917aae640 10 monclient: _reopen_session rank -1 2024-09-16T03:43:35.115 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.105+0000 7f6917aae640 10 monclient: _add_conns ranks=[1,3,2,0,4] 2024-09-16T03:43:35.115 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.105+0000 7f6917aae640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f69100a39b0 addr v1:172.21.15.154:6789/0 2024-09-16T03:43:35.115 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.105+0000 7f6917aae640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f69100a3d90 0x7f691013c380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:43:35.115 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.105+0000 7f6917aae640 10 monclient(hunting): picked mon.noname-a con 0x7f69100a3d90 addr v2:172.21.15.42:3300/0 2024-09-16T03:43:35.115 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.105+0000 7f6917aae640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f691013c8c0 0x7f691013ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:43:35.115 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.105+0000 7f6917aae640 10 monclient(hunting): picked mon.noname-b con 0x7f691013c8c0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:43:35.115 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.105+0000 7f6917aae640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6910134290 con 0x7f69100a39b0 2024-09-16T03:43:35.116 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.105+0000 7f6917aae640 10 monclient(hunting): start opening mon connection 2024-09-16T03:43:35.116 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.105+0000 7f6917aae640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f691009c140 con 0x7f69100a3d90 2024-09-16T03:43:35.116 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.105+0000 7f6917aae640 10 monclient(hunting): start opening mon connection 2024-09-16T03:43:35.116 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.105+0000 7f6917aae640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f691009c420 con 0x7f691013c8c0 2024-09-16T03:43:35.116 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.105+0000 7f6917aae640 10 monclient(hunting): _renew_subs 2024-09-16T03:43:35.116 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.105+0000 7f6917aae640 10 monclient(hunting): authenticate will time out at 1062.079628s 2024-09-16T03:43:35.116 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.105+0000 7f6915049640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f69100a3d90 0x7f691013c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:43:35.116 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.105+0000 7f691604b640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f691013c8c0 0x7f691013ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:43:35.116 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.105+0000 7f691604b640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f691013c8c0 0x7f691013ecb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:55584/0 (socket says 172.21.15.42:55584) 2024-09-16T03:43:35.116 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.105+0000 7f6915049640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f69100a3d90 0x7f691013c380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:52044/0 (socket says 172.21.15.42:52044) 2024-09-16T03:43:35.116 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.105+0000 7f691604b640 1 -- 172.21.15.42:0/864534753 learned_addr learned my addr 172.21.15.42:0/864534753 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:43:35.116 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.105+0000 7f691604b640 10 monclient(hunting): get_auth_request con 0x7f691013c8c0 auth_method 0 2024-09-16T03:43:35.116 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.105+0000 7f691604b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:43:35.117 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.105+0000 7f691604b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:43:35.117 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.105+0000 7f691604b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:43:35.117 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.105+0000 7f6915049640 10 monclient(hunting): get_auth_request con 0x7f69100a3d90 auth_method 0 2024-09-16T03:43:35.117 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.105+0000 7f6915049640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:43:35.117 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.105+0000 7f6915049640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:43:35.117 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.105+0000 7f6915049640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:43:35.117 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.105+0000 7f6915049640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:43:35.117 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.105+0000 7f6915049640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:43:35.117 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.105+0000 7f6915049640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:43:35.117 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.105+0000 7f691604b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:43:35.117 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.105+0000 7f691604b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:43:35.117 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.109+0000 7f691604b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:43:35.117 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.109+0000 7f6915049640 10 monclient(hunting): handle_auth_done global_id 4896 payload 274 2024-09-16T03:43:35.117 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.109+0000 7f6915049640 1 -- 172.21.15.42:0/864534753 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f691013c8c0 msgr2=0x7f691013ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:43:35.118 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.109+0000 7f6915049640 1 --2- 172.21.15.42:0/864534753 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f691013c8c0 0x7f691013ecb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:43:35.118 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.109+0000 7f6915049640 1 -- 172.21.15.42:0/864534753 >> v1:172.21.15.154:6789/0 conn(0x7f69100a39b0 legacy=0x7f6910133b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:43:35.118 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.109+0000 7f6915049640 10 monclient: _finish_hunting 0 2024-09-16T03:43:35.118 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.109+0000 7f6915049640 1 monclient: found mon.noname-a 2024-09-16T03:43:35.118 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.109+0000 7f6915049640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:43:35.118 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.109+0000 7f6915049640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:43:35.118 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.109+0000 7f6915049640 1 -- 172.21.15.42:0/864534753 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f691006dd60 con 0x7f69100a3d90 2024-09-16T03:43:35.118 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.109+0000 7f691604b640 1 --2- 172.21.15.42:0/864534753 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f691013c8c0 0x7f691013ecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:43:35.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.109+0000 7f6915049640 1 --2- 172.21.15.42:0/864534753 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f69100a3d90 0x7f691013c380 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f69040097c0 tx=0x7f690401c360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ca9beca85b41b45 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:43:35.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.109+0000 7f6914848640 1 -- 172.21.15.42:0/864534753 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69040b0070 con 0x7f69100a3d90 2024-09-16T03:43:35.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.109+0000 7f6914848640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:43:35.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.109+0000 7f6914848640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:43:35.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.109+0000 7f6914848640 10 monclient: dump: 2024-09-16T03:43:35.124 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:43:35.125 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:43:35.125 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:43:35.125 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:43:35.125 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:43:35.125 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:43:35.125 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:43:35.125 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:43:35.125 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:43:35.125 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:43:35.125 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.109+0000 7f6914848640 10 monclient: _finish_auth 0 2024-09-16T03:43:35.125 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.109+0000 7f6914848640 10 monclient: _check_auth_tickets 2024-09-16T03:43:35.125 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.109+0000 7f6914848640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:43:35.125 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.109+0000 7f6917aae640 5 monclient: authenticate success, global_id 4896 2024-09-16T03:43:35.125 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.109+0000 7f6914848640 1 -- 172.21.15.42:0/864534753 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f69040ab600 con 0x7f69100a3d90 2024-09-16T03:43:35.126 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.109+0000 7f6917aae640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:43:35.126 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.109+0000 7f6914848640 10 monclient: handle_config config(0 keys) 2024-09-16T03:43:35.126 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.109+0000 7f6917aae640 10 monclient: get_monmap_and_config success 2024-09-16T03:43:35.126 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.109+0000 7f6917aae640 4 set_mon_vals no callback set 2024-09-16T03:43:35.126 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.109+0000 7f6914848640 1 -- 172.21.15.42:0/864534753 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69040ab9d0 con 0x7f69100a3d90 2024-09-16T03:43:35.126 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.109+0000 7f6914848640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:43:35.126 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.109+0000 7f6914848640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:43:35.126 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.109+0000 7f6914848640 10 monclient: dump: 2024-09-16T03:43:35.126 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:43:35.126 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:43:35.126 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:43:35.126 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:43:35.126 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:43:35.126 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:43:35.127 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:43:35.127 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:43:35.127 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:43:35.127 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:43:35.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.109+0000 7f6917aae640 10 monclient: shutdown 2024-09-16T03:43:35.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.109+0000 7f6917aae640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:43:35.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.109+0000 7f6917aae640 1 -- 172.21.15.42:0/864534753 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f69100a3d90 msgr2=0x7f691013c380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:43:35.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.109+0000 7f6917aae640 1 --2- 172.21.15.42:0/864534753 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f69100a3d90 0x7f691013c380 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f69040097c0 tx=0x7f690401c360 comp rx=0 tx=0).stop 2024-09-16T03:43:35.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.109+0000 7f6917aae640 1 -- 172.21.15.42:0/864534753 shutdown_connections 2024-09-16T03:43:35.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.109+0000 7f6917aae640 1 --2- 172.21.15.42:0/864534753 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f691013c8c0 0x7f691013ecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:43:35.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.109+0000 7f6917aae640 1 --2- 172.21.15.42:0/864534753 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f69100a3d90 0x7f691013c380 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:43:35.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.109+0000 7f6917aae640 1 -- 172.21.15.42:0/864534753 >> 172.21.15.42:0/864534753 conn(0x7f69100a9fb0 msgr2=0x7f69100aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:43:35.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.109+0000 7f6917aae640 1 -- 172.21.15.42:0/864534753 shutdown_connections 2024-09-16T03:43:35.128 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.109+0000 7f6917aae640 1 -- 172.21.15.42:0/864534753 wait complete. 2024-09-16T03:43:35.128 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.109+0000 7f6917aae640 10 monclient: build_initial_monmap 2024-09-16T03:43:35.128 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.109+0000 7f6917aae640 10 monclient: monmap: 2024-09-16T03:43:35.128 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:43:35.128 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:43:35.128 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:43:35.128 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:43:35.128 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:43:35.128 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:43:35.128 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:43:35.128 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:43:35.128 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:43:35.128 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:43:35.128 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.113+0000 7f6917aae640 1 librados: starting msgr at 2024-09-16T03:43:35.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.113+0000 7f6917aae640 1 librados: starting objecter 2024-09-16T03:43:35.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.113+0000 7f6917aae640 1 Processor -- start 2024-09-16T03:43:35.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.113+0000 7f6917aae640 1 -- start start 2024-09-16T03:43:35.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.113+0000 7f6917aae640 1 librados: setting wanted keys 2024-09-16T03:43:35.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.113+0000 7f6917aae640 1 librados: calling monclient init 2024-09-16T03:43:35.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.113+0000 7f6917aae640 10 monclient: init 2024-09-16T03:43:35.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.113+0000 7f6917aae640 10 monclient: _reopen_session rank -1 2024-09-16T03:43:35.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.113+0000 7f6917aae640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T03:43:35.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.113+0000 7f6917aae640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f69100a39b0 0x7f691012d720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:43:35.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.113+0000 7f6917aae640 10 monclient(hunting): picked mon.noname-b con 0x7f69100a39b0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:43:35.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.113+0000 7f6917aae640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f69100a3d90 0x7f691012dc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:43:35.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.113+0000 7f6917aae640 10 monclient(hunting): picked mon.noname-c con 0x7f69100a3d90 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:43:35.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.113+0000 7f6917aae640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f691013c8c0 0x7f6910138130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:43:35.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.113+0000 7f6917aae640 10 monclient(hunting): picked mon.noname-a con 0x7f691013c8c0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:43:35.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.113+0000 7f6917aae640 10 monclient(hunting): start opening mon connection 2024-09-16T03:43:35.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.113+0000 7f6917aae640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f6910141da0 con 0x7f691013c8c0 2024-09-16T03:43:35.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.113+0000 7f6917aae640 10 monclient(hunting): start opening mon connection 2024-09-16T03:43:35.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.113+0000 7f6917aae640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f6910141c20 con 0x7f69100a39b0 2024-09-16T03:43:35.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.113+0000 7f6917aae640 10 monclient(hunting): start opening mon connection 2024-09-16T03:43:35.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.113+0000 7f6917aae640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f6910141f20 con 0x7f69100a3d90 2024-09-16T03:43:35.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.113+0000 7f6917aae640 10 monclient(hunting): _renew_subs 2024-09-16T03:43:35.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.113+0000 7f6917aae640 10 monclient(hunting): authenticate will time out at 1062.084951s 2024-09-16T03:43:35.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.113+0000 7f6915049640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f69100a3d90 0x7f691012dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:43:35.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.113+0000 7f691604b640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f691013c8c0 0x7f6910138130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:43:35.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.113+0000 7f6915049640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f69100a3d90 0x7f691012dc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:55596/0 (socket says 172.21.15.42:55596) 2024-09-16T03:43:35.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.113+0000 7f6915049640 1 -- 172.21.15.42:0/23394599 learned_addr learned my addr 172.21.15.42:0/23394599 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:43:35.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.113+0000 7f6915049640 10 monclient(hunting): get_auth_request con 0x7f69100a3d90 auth_method 0 2024-09-16T03:43:35.131 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.113+0000 7f6915049640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:43:35.131 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.113+0000 7f6915049640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:43:35.131 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.113+0000 7f6915049640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:43:35.131 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.113+0000 7f691604b640 10 monclient(hunting): get_auth_request con 0x7f691013c8c0 auth_method 0 2024-09-16T03:43:35.131 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.113+0000 7f691604b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:43:35.131 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.113+0000 7f691604b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:43:35.131 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.113+0000 7f691604b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:43:35.131 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.113+0000 7f6915049640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:43:35.131 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.113+0000 7f6915049640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:43:35.131 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.113+0000 7f6915049640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:43:35.131 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.113+0000 7f691604b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:43:35.131 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.113+0000 7f691604b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:43:35.131 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.113+0000 7f691604b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:43:35.132 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.113+0000 7f691584a640 1 --2- 172.21.15.42:0/23394599 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f69100a39b0 0x7f691012d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:43:35.132 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.113+0000 7f691584a640 10 monclient(hunting): get_auth_request con 0x7f69100a39b0 auth_method 0 2024-09-16T03:43:35.132 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.113+0000 7f691584a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:43:35.132 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.113+0000 7f691584a640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:43:35.132 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.113+0000 7f691584a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:43:35.132 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.113+0000 7f6915049640 10 monclient(hunting): handle_auth_done global_id 4724 payload 824 2024-09-16T03:43:35.132 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.113+0000 7f6915049640 1 -- 172.21.15.42:0/23394599 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f69100a39b0 msgr2=0x7f691012d720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:43:35.132 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.113+0000 7f6915049640 1 --2- 172.21.15.42:0/23394599 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f69100a39b0 0x7f691012d720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:43:35.132 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.113+0000 7f6915049640 1 -- 172.21.15.42:0/23394599 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f691013c8c0 msgr2=0x7f6910138130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:43:35.132 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.113+0000 7f6915049640 1 --2- 172.21.15.42:0/23394599 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f691013c8c0 0x7f6910138130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:43:35.132 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.113+0000 7f6915049640 10 monclient: _finish_hunting 0 2024-09-16T03:43:35.132 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.113+0000 7f6915049640 1 monclient: found mon.noname-c 2024-09-16T03:43:35.132 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.113+0000 7f6915049640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:43:35.133 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.113+0000 7f6915049640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:43:35.133 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.113+0000 7f6915049640 1 -- 172.21.15.42:0/23394599 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6910131000 con 0x7f69100a3d90 2024-09-16T03:43:35.133 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.113+0000 7f691604b640 1 --2- 172.21.15.42:0/23394599 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f691013c8c0 0x7f6910138130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:43:35.133 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.113+0000 7f6915049640 1 --2- 172.21.15.42:0/23394599 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f69100a3d90 0x7f691012dc60 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f69040ae040 tx=0x7f6904019100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:43:35.133 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.113+0000 7f690e7fc640 10 client.?.objecter ms_handle_connect 0x7f69100a3d90 2024-09-16T03:43:35.133 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.113+0000 7f690e7fc640 10 client.?.objecter resend_mon_ops 2024-09-16T03:43:35.133 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.113+0000 7f690e7fc640 1 -- 172.21.15.42:0/23394599 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69040b0040 con 0x7f69100a3d90 2024-09-16T03:43:35.133 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.113+0000 7f690e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:43:35.133 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.113+0000 7f690e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:43:35.133 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.113+0000 7f690e7fc640 10 monclient: dump: 2024-09-16T03:43:35.133 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:43:35.133 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:43:35.133 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:43:35.134 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:43:35.134 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:43:35.134 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:43:35.134 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:43:35.134 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:43:35.134 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:43:35.134 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:43:35.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.113+0000 7f690e7fc640 10 monclient: _finish_auth 0 2024-09-16T03:43:35.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.113+0000 7f690e7fc640 10 monclient: _check_auth_tickets 2024-09-16T03:43:35.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.113+0000 7f690e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:43:35.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.113+0000 7f690e7fc640 1 -- 172.21.15.42:0/23394599 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f69040b43f0 con 0x7f69100a3d90 2024-09-16T03:43:35.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.113+0000 7f6917aae640 5 monclient: authenticate success, global_id 4724 2024-09-16T03:43:35.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.113+0000 7f6917aae640 10 monclient: _renew_subs 2024-09-16T03:43:35.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.113+0000 7f6917aae640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:43:35.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.113+0000 7f6917aae640 1 -- 172.21.15.42:0/23394599 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f691013b830 con 0x7f69100a3d90 2024-09-16T03:43:35.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.113+0000 7f690e7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:43:35.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.113+0000 7f690e7fc640 1 -- 172.21.15.42:0/23394599 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69040b47c0 con 0x7f69100a3d90 2024-09-16T03:43:35.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.113+0000 7f690e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:43:35.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.113+0000 7f690e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:43:35.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.113+0000 7f690e7fc640 10 monclient: dump: 2024-09-16T03:43:35.135 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:43:35.135 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:43:35.135 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:43:35.135 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:43:35.135 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:43:35.135 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:43:35.135 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:43:35.135 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:43:35.136 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:43:35.136 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:43:35.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.113+0000 7f690ffff640 4 set_mon_vals no callback set 2024-09-16T03:43:35.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.113+0000 7f6917aae640 10 client.4724.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:43:35.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.113+0000 7f6917aae640 10 monclient: _renew_subs 2024-09-16T03:43:35.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.113+0000 7f6917aae640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:43:35.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.113+0000 7f6917aae640 1 -- 172.21.15.42:0/23394599 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f691013a890 con 0x7f69100a3d90 2024-09-16T03:43:35.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.113+0000 7f6917aae640 1 librados: init done 2024-09-16T03:43:35.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.113+0000 7f6917aae640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:43:35.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.113+0000 7f6917aae640 1 --2- 172.21.15.42:0/23394599 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6910138670 0x7f6910138a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:43:35.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.113+0000 7f6917aae640 10 monclient(hunting): start opening mon connection 2024-09-16T03:43:35.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.113+0000 7f6917aae640 1 -- 172.21.15.42:0/23394599 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f691013aee0 con 0x7f6910138670 2024-09-16T03:43:35.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.113+0000 7f690e7fc640 1 -- 172.21.15.42:0/23394599 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f69040b4960 con 0x7f69100a3d90 2024-09-16T03:43:35.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.113+0000 7f691584a640 1 --2- 172.21.15.42:0/23394599 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6910138670 0x7f6910138a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:43:35.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.113+0000 7f691584a640 10 monclient: get_auth_request con 0x7f6910138670 auth_method 0 2024-09-16T03:43:35.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.113+0000 7f691584a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:43:35.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.113+0000 7f691584a640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:43:35.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.113+0000 7f691584a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:43:35.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.113+0000 7f690e7fc640 1 --2- 172.21.15.42:0/23394599 >> v2:172.21.15.154:6812/254845342 conn(0x7f68fc037840 0x7f68fc039d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:43:35.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.113+0000 7f691584a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:43:35.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.113+0000 7f691584a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:43:35.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.113+0000 7f691584a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:43:35.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.113+0000 7f690e7fc640 1 -- 172.21.15.42:0/23394599 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(22..22 src has 1..22) ==== 11679+0+0 (secure 0 0 0) 0x7f69040e4280 con 0x7f69100a3d90 2024-09-16T03:43:35.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.113+0000 7f690e7fc640 10 client.4724.objecter ms_dispatch 0x7f6910000ba0 osd_map(22..22 src has 1..22) 2024-09-16T03:43:35.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.113+0000 7f690e7fc640 3 client.4724.objecter handle_osd_map got epochs [22,22] > 0 2024-09-16T03:43:35.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.113+0000 7f690e7fc640 3 client.4724.objecter handle_osd_map decoding full epoch 22 2024-09-16T03:43:35.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.113+0000 7f691604b640 1 --2- 172.21.15.42:0/23394599 >> v2:172.21.15.154:6812/254845342 conn(0x7f68fc037840 0x7f68fc039d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:43:35.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.113+0000 7f691604b640 10 monclient: get_auth_request con 0x7f68fc037840 auth_method 0 2024-09-16T03:43:35.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.113+0000 7f690e7fc640 20 client.4724.objecter dump_active .. 0 homeless 2024-09-16T03:43:35.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.113+0000 7f691584a640 10 monclient(hunting): handle_auth_done global_id 14464 payload 824 2024-09-16T03:43:35.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.113+0000 7f691584a640 1 -- 172.21.15.42:0/23394599 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6910138f90 con 0x7f6910138670 2024-09-16T03:43:35.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.113+0000 7f690e7fc640 10 client.4724.objecter ms_handle_connect 0x7f68fc037840 2024-09-16T03:43:35.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.113+0000 7f691604b640 1 --2- 172.21.15.42:0/23394599 >> v2:172.21.15.154:6812/254845342 conn(0x7f68fc037840 0x7f68fc039d00 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f6908009630 tx=0x7f6908009240 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:43:35.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.113+0000 7f690e7fc640 10 client.4724.objecter ms_handle_connect 0x7f6910138670 2024-09-16T03:43:35.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.113+0000 7f691584a640 1 --2- 172.21.15.42:0/23394599 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6910138670 0x7f6910138a50 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f690000e7a0 tx=0x7f690000ec60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:43:35.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.113+0000 7f690e7fc640 10 client.4724.objecter resend_mon_ops 2024-09-16T03:43:35.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.117+0000 7f690e7fc640 1 -- 172.21.15.42:0/23394599 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6900003b90 con 0x7f6910138670 2024-09-16T03:43:35.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.117+0000 7f690e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:43:35.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.117+0000 7f690e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:43:35.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.117+0000 7f690e7fc640 10 monclient: dump: 2024-09-16T03:43:35.138 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:43:35.138 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:43:35.139 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:43:35.139 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:43:35.139 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:43:35.139 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:43:35.139 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:43:35.139 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:43:35.139 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:43:35.139 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:43:35.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.117+0000 7f690e7fc640 1 -- 172.21.15.42:0/23394599 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f6900018070 con 0x7f6910138670 2024-09-16T03:43:35.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.117+0000 7f690e7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:43:35.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.117+0000 7f690e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:43:35.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.117+0000 7f690e7fc640 1 -- 172.21.15.42:0/23394599 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6910138670 msgr2=0x7f6910138a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:43:35.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.117+0000 7f690e7fc640 1 --2- 172.21.15.42:0/23394599 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6910138670 0x7f6910138a50 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f690000e7a0 tx=0x7f690000ec60 comp rx=0 tx=0).stop 2024-09-16T03:43:35.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.117+0000 7f690e7fc640 1 -- 172.21.15.42:0/23394599 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6910138670 msgr2=0x7f6910138a50 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:43:35.140 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.117+0000 7f690e7fc640 1 --2- 172.21.15.42:0/23394599 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6910138670 0x7f6910138a50 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:43:35.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.121+0000 7f6917aae640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:43:35.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.121+0000 7f6917aae640 1 --2- 172.21.15.42:0/23394599 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6910135bc0 0x7f6910135fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:43:35.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.121+0000 7f6917aae640 10 monclient(hunting): start opening mon connection 2024-09-16T03:43:35.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.121+0000 7f6917aae640 1 -- 172.21.15.42:0/23394599 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f691006f680 con 0x7f6910135bc0 2024-09-16T03:43:35.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.125+0000 7f691584a640 1 --2- 172.21.15.42:0/23394599 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6910135bc0 0x7f6910135fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:43:35.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.125+0000 7f691584a640 10 monclient: get_auth_request con 0x7f6910135bc0 auth_method 0 2024-09-16T03:43:35.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.125+0000 7f691584a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:43:35.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.125+0000 7f691584a640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:43:35.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.125+0000 7f691584a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:43:35.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.125+0000 7f691584a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:43:35.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.125+0000 7f691584a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:43:35.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.125+0000 7f691584a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:43:35.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.125+0000 7f691584a640 10 monclient(hunting): handle_auth_done global_id 14467 payload 824 2024-09-16T03:43:35.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.125+0000 7f691584a640 1 -- 172.21.15.42:0/23394599 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f6910133b80 con 0x7f6910135bc0 2024-09-16T03:43:35.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.125+0000 7f690e7fc640 10 client.4724.objecter ms_handle_connect 0x7f6910135bc0 2024-09-16T03:43:35.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.125+0000 7f690e7fc640 10 client.4724.objecter resend_mon_ops 2024-09-16T03:43:35.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.125+0000 7f691584a640 1 --2- 172.21.15.42:0/23394599 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6910135bc0 0x7f6910135fd0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f690000e7a0 tx=0x7f6900001260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:43:35.143 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.125+0000 7f690e7fc640 1 -- 172.21.15.42:0/23394599 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6900006d70 con 0x7f6910135bc0 2024-09-16T03:43:35.143 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.125+0000 7f690e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:43:35.143 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.125+0000 7f690e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:43:35.143 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.125+0000 7f690e7fc640 10 monclient: dump: 2024-09-16T03:43:35.143 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:43:35.143 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:43:35.143 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:43:35.143 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:43:35.143 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:43:35.143 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:43:35.143 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:43:35.143 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:43:35.143 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:43:35.144 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:43:35.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.125+0000 7f690e7fc640 1 -- 172.21.15.42:0/23394599 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4712 (secure 0 0 0) 0x7f6900006d70 con 0x7f6910135bc0 2024-09-16T03:43:35.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.125+0000 7f690e7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:43:35.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.125+0000 7f690e7fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:43:35.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.125+0000 7f690e7fc640 1 -- 172.21.15.42:0/23394599 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6910135bc0 msgr2=0x7f6910135fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:43:35.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.125+0000 7f690e7fc640 1 --2- 172.21.15.42:0/23394599 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6910135bc0 0x7f6910135fd0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f690000e7a0 tx=0x7f6900001260 comp rx=0 tx=0).stop 2024-09-16T03:43:35.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.125+0000 7f690e7fc640 1 -- 172.21.15.42:0/23394599 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6910135bc0 msgr2=0x7f6910135fd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:43:35.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.125+0000 7f690e7fc640 1 --2- 172.21.15.42:0/23394599 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6910135bc0 0x7f6910135fd0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:43:35.144 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:43:35.144 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:43:35.145 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:43:35.145 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T03:43:35.145 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 16, 2024-09-16T03:43:35.145 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 18926, 2024-09-16T03:43:35.145 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:43:35.145 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:43:35.145 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T03:43:35.145 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T03:43:35.145 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:43:35.145 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 3, 2024-09-16T03:43:35.145 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:43:35.145 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:43:35.145 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:43:35.146 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:43:35.146 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:43:35.146 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:43:35.146 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:43:35.146 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:43:35.146 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:43:35.146 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:43:35.146 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:43:35.146 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:43:35.146 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:43:35.146 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:43:35.146 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:43:35.146 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:43:35.146 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:43:35.147 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:43:35.147 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:43:35.147 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:43:35.147 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:43:35.147 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:43:35.147 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:43:35.147 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:43:35.147 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:43:35.147 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:43:35.147 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:43:35.147 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:43:35.147 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:43:35.147 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:43:35.147 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:43:35.148 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:43:35.148 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:43:35.148 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:43:35.148 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:43:35.148 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:43:35.148 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:43:35.148 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:43:35.148 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:43:35.148 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:43:35.148 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:43:35.148 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:43:35.148 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:43:35.148 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:43:35.149 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:43:35.149 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:43:35.149 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:43:35.149 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:43:35.149 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:43:35.149 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:43:35.149 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:43:35.149 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:43:35.149 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:43:35.149 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:43:35.149 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:43:35.149 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:43:35.149 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:43:35.149 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:43:35.150 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:43:35.150 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:43:35.150 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:43:35.150 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:43:35.150 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:43:35.150 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:43:35.150 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:43:35.150 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:43:35.150 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:43:35.150 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:43:35.150 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:43:35.150 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:43:35.150 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:43:35.151 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:43:35.151 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:43:35.151 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:43:35.151 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:43:35.151 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:43:35.151 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:43:35.151 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:43:35.151 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:43:35.151 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:43:35.151 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:43:35.151 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:43:35.151 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:43:35.151 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:43:35.151 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:43:35.151 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:43:35.152 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:43:35.152 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:43:35.152 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:43:35.152 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:43:35.152 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:43:35.152 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:43:35.152 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:43:35.152 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:43:35.152 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:43:35.152 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:43:35.152 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:43:35.152 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:43:35.152 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:43:35.153 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:43:35.153 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:43:35.153 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:43:35.153 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:43:35.153 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:43:35.153 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:43:35.153 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:43:35.153 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:43:35.153 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:43:35.153 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:43:35.153 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:43:35.153 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:43:35.153 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:43:35.153 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:43:35.154 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:43:35.154 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:43:35.154 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:43:35.154 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:43:35.154 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:43:35.154 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:43:35.154 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:43:35.154 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:43:35.154 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:43:35.154 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:43:35.154 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:43:35.154 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:43:35.154 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:43:35.154 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:43:35.154 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:43:35.155 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:43:35.155 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:43:35.155 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:43:35.155 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:43:35.155 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:43:35.155 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:43:35.155 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:43:35.155 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:43:35.155 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:43:35.155 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:43:35.155 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:43:35.155 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:43:35.155 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:43:35.155 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T03:43:35.156 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:43:35.156 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:43:35.156 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:43:35.156 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:43:35.156 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:43:35.156 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:43:35.156 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:43:35.156 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:43:35.156 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:43:35.156 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:43:35.156 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 18 2024-09-16T03:43:35.156 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:43:35.156 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:43:35.156 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:43:35.157 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:43:35.157 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:43:35.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.125+0000 7f6917aae640 10 librados: watch_flush enter 2024-09-16T03:43:35.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.125+0000 7f6917aae640 10 librados: watch_flush exit 2024-09-16T03:43:35.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.125+0000 7f6917aae640 20 client.4724.objecter shutdown clearing up homeless session... 2024-09-16T03:43:35.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.125+0000 7f6917aae640 10 client.4724.objecter successfully canceled tick 2024-09-16T03:43:35.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.125+0000 7f6917aae640 1 -- 172.21.15.42:0/23394599 >> v2:172.21.15.154:6812/254845342 conn(0x7f68fc037840 msgr2=0x7f68fc039d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:43:35.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.125+0000 7f6917aae640 1 --2- 172.21.15.42:0/23394599 >> v2:172.21.15.154:6812/254845342 conn(0x7f68fc037840 0x7f68fc039d00 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f6908009630 tx=0x7f6908009240 comp rx=0 tx=0).stop 2024-09-16T03:43:35.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.125+0000 7f6917aae640 10 monclient: shutdown 2024-09-16T03:43:35.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.125+0000 7f6917aae640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:43:35.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.125+0000 7f6917aae640 1 -- 172.21.15.42:0/23394599 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f69100a3d90 msgr2=0x7f691012dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:43:35.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.125+0000 7f6917aae640 1 --2- 172.21.15.42:0/23394599 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f69100a3d90 0x7f691012dc60 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f69040ae040 tx=0x7f6904019100 comp rx=0 tx=0).stop 2024-09-16T03:43:35.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.125+0000 7f691604b640 1 -- 172.21.15.42:0/23394599 reap_dead start 2024-09-16T03:43:35.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.125+0000 7f691604b640 1 -- 172.21.15.42:0/23394599 reap_dead start 2024-09-16T03:43:35.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.125+0000 7f6917aae640 1 -- 172.21.15.42:0/23394599 shutdown_connections 2024-09-16T03:43:35.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.125+0000 7f6917aae640 1 -- 172.21.15.42:0/23394599 >> 172.21.15.42:0/23394599 conn(0x7f69100a9fb0 msgr2=0x7f6910131980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:43:35.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.125+0000 7f6917aae640 1 -- 172.21.15.42:0/23394599 shutdown_connections 2024-09-16T03:43:35.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.125+0000 7f6917aae640 1 -- 172.21.15.42:0/23394599 wait complete. 2024-09-16T03:43:35.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:35.125+0000 7f6917aae640 1 librados: shutdown 2024-09-16T03:43:35.160 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:43:35.160 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-09-16T03:43:35.160 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['c'] 2024-09-16T03:43:35.160 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-09-16T03:43:35.160 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-16T03:43:35.160 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-16T03:43:35.160 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T03:43:35.179 INFO:tasks.ceph.mon.b:Stopped 2024-09-16T03:43:35.179 INFO:tasks.mon_thrash:Sending STOP to mon c 2024-09-16T03:43:35.180 INFO:tasks.ceph.mon.c:Sent signal 19 2024-09-16T03:43:35.180 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-16T03:43:35.255 INFO:tasks.workunit.client.0.smithi154.stdout: list_parallel: ******************************* 2024-09-16T03:43:35.255 INFO:tasks.workunit.client.0.smithi154.stdout: list_parallel: process_9_[29335]: starting. 2024-09-16T03:43:35.255 INFO:tasks.workunit.client.0.smithi154.stdout: list_parallel: process_9_[29335]: creating pool ceph_test_rados_list_parallel.smithi154-28425 2024-09-16T03:43:35.255 INFO:tasks.workunit.client.0.smithi154.stdout: list_parallel: process_9_[29335]: created object 0... 2024-09-16T03:43:35.255 INFO:tasks.workunit.client.0.smithi154.stdout: list_parallel: process_9_[29335]: created object 25... 2024-09-16T03:43:35.255 INFO:tasks.workunit.client.0.smithi154.stdout: list_parallel: process_9_[29335]: created object 49... 2024-09-16T03:43:35.256 INFO:tasks.workunit.client.0.smithi154.stdout: list_parallel: process_9_[29335]: finishing. 2024-09-16T03:43:35.256 INFO:tasks.workunit.client.0.smithi154.stdout: list_parallel: process_9_[29335]: shutting down. 2024-09-16T03:43:35.256 INFO:tasks.workunit.client.0.smithi154.stdout: list_parallel: ******************************* 2024-09-16T03:43:35.256 INFO:tasks.workunit.client.0.smithi154.stdout: list_parallel: ******************************* 2024-09-16T03:43:35.256 INFO:tasks.workunit.client.0.smithi154.stdout: list_parallel: ******************************* 2024-09-16T03:43:35.256 INFO:tasks.workunit.client.0.smithi154.stdout: list_parallel: process_13_[29339]: starting. 2024-09-16T03:43:35.256 INFO:tasks.workunit.client.0.smithi154.stdout: list_parallel: process_13_[29339]: removed 25 objects... 2024-09-16T03:43:35.256 INFO:tasks.workunit.client.0.smithi154.stdout: list_parallel: process_13_[29339]: removed half of the objects 2024-09-16T03:43:35.256 INFO:tasks.workunit.client.0.smithi154.stdout: list_parallel: process_13_[29339]: removed 50 objects... 2024-09-16T03:43:35.256 INFO:tasks.workunit.client.0.smithi154.stdout: list_parallel: process_13_[29339]: removed 50 objects 2024-09-16T03:43:35.256 INFO:tasks.workunit.client.0.smithi154.stdout: list_parallel: process_13_[29339]: shutting down. 2024-09-16T03:43:35.257 INFO:tasks.workunit.client.0.smithi154.stdout: list_parallel: ******************************* 2024-09-16T03:43:35.257 INFO:tasks.workunit.client.0.smithi154.stdout: list_parallel: ******************************* 2024-09-16T03:43:35.257 INFO:tasks.workunit.client.0.smithi154.stdout: list_parallel: ******************************* 2024-09-16T03:43:35.257 INFO:tasks.workunit.client.0.smithi154.stdout: list_parallel: process_12_[29338]: starting. 2024-09-16T03:43:35.257 INFO:tasks.workunit.client.0.smithi154.stdout: list_parallel: process_12_[29338]: added 25 objects... 2024-09-16T03:43:35.257 INFO:tasks.workunit.client.0.smithi154.stdout: list_parallel: process_12_[29338]: added half of the objects 2024-09-16T03:43:35.257 INFO:tasks.workunit.client.0.smithi154.stdout: list_parallel: process_12_[29338]: added 50 objects... 2024-09-16T03:43:35.257 INFO:tasks.workunit.client.0.smithi154.stdout: list_parallel: process_12_[29338]: added 50 objects 2024-09-16T03:43:35.257 INFO:tasks.workunit.client.0.smithi154.stdout: list_parallel: process_12_[29338]: shutting down. 2024-09-16T03:43:35.257 INFO:tasks.workunit.client.0.smithi154.stdout: list_parallel: ******************************* 2024-09-16T03:43:35.257 INFO:tasks.workunit.client.0.smithi154.stdout: list_parallel: ******************************* 2024-09-16T03:43:35.257 INFO:tasks.workunit.client.0.smithi154.stdout: list_parallel: ******************************* 2024-09-16T03:43:35.257 INFO:tasks.workunit.client.0.smithi154.stdout: list_parallel: process_11_[29337]: starting. 2024-09-16T03:43:35.258 INFO:tasks.workunit.client.0.smithi154.stdout: list_parallel: process_11_[29337]: added 25 objects... 2024-09-16T03:43:35.258 INFO:tasks.workunit.client.0.smithi154.stdout: list_parallel: process_11_[29337]: added half of the objects 2024-09-16T03:43:35.258 INFO:tasks.workunit.client.0.smithi154.stdout: list_parallel: process_11_[29337]: added 50 objects... 2024-09-16T03:43:35.258 INFO:tasks.workunit.client.0.smithi154.stdout: list_parallel: process_11_[29337]: added 50 objects 2024-09-16T03:43:35.258 INFO:tasks.workunit.client.0.smithi154.stdout: list_parallel: process_11_[29337]: shutting down. 2024-09-16T03:43:35.258 INFO:tasks.workunit.client.0.smithi154.stdout: list_parallel: ******************************* 2024-09-16T03:43:35.258 INFO:tasks.workunit.client.0.smithi154.stdout: list_parallel: ******************************* 2024-09-16T03:43:35.258 INFO:tasks.workunit.client.0.smithi154.stdout: list_parallel: ******************************* 2024-09-16T03:43:35.258 INFO:tasks.workunit.client.0.smithi154.stdout: list_parallel: process_10_[29336]: starting. 2024-09-16T03:43:35.258 INFO:tasks.workunit.client.0.smithi154.stdout: list_parallel: process_10_[29336]: listing objects. 2024-09-16T03:43:35.258 INFO:tasks.workunit.client.0.smithi154.stdout: list_parallel: process_10_[29336]: listed object 0... 2024-09-16T03:43:35.258 INFO:tasks.workunit.client.0.smithi154.stdout: list_parallel: process_10_[29336]: listed object 25... 2024-09-16T03:43:35.258 INFO:tasks.workunit.client.0.smithi154.stdout: list_parallel: process_10_[29336]: listed object 50... 2024-09-16T03:43:35.259 INFO:tasks.workunit.client.0.smithi154.stdout: list_parallel: process_10_[29336]: listed object 75... 2024-09-16T03:43:35.259 INFO:tasks.workunit.client.0.smithi154.stdout: list_parallel: process_10_[29336]: saw 99 objects 2024-09-16T03:43:35.259 INFO:tasks.workunit.client.0.smithi154.stdout: list_parallel: process_10_[29336]: shutting down. 2024-09-16T03:43:35.259 INFO:tasks.workunit.client.0.smithi154.stdout: list_parallel: ******************************* 2024-09-16T03:43:35.259 INFO:tasks.workunit.client.0.smithi154.stdout: list_parallel: ******************************* 2024-09-16T03:43:35.259 INFO:tasks.workunit.client.0.smithi154.stdout: list_parallel: ******************************* 2024-09-16T03:43:35.259 INFO:tasks.workunit.client.0.smithi154.stdout: list_parallel: ******************************* 2024-09-16T03:43:35.259 INFO:tasks.workunit.client.0.smithi154.stdout: list_parallel: process_14_[29714]: starting. 2024-09-16T03:43:35.259 INFO:tasks.workunit.client.0.smithi154.stdout: list_parallel: process_14_[29714]: creating pool ceph_test_rados_list_parallel.smithi154-28425 2024-09-16T03:43:35.259 INFO:tasks.workunit.client.0.smithi154.stdout: list_parallel: process_14_[29714]: created object 0... 2024-09-16T03:43:35.259 INFO:tasks.workunit.client.0.smithi154.stdout: list_parallel: process_14_[29714]: created object 25... 2024-09-16T03:43:35.260 INFO:tasks.workunit.client.0.smithi154.stdout: list_parallel: process_14_[29714]: created object 49... 2024-09-16T03:43:35.260 INFO:tasks.workunit.client.0.smithi154.stdout: list_parallel: process_14_[29714]: finishing. 2024-09-16T03:43:35.260 INFO:tasks.workunit.client.0.smithi154.stdout: list_parallel: process_14_[29714]: shutting down. 2024-09-16T03:43:35.260 INFO:tasks.workunit.client.0.smithi154.stdout: list_parallel: ******************************* 2024-09-16T03:43:35.260 INFO:tasks.workunit.client.0.smithi154.stdout: list_parallel: ******************************* 2024-09-16T03:43:35.260 INFO:tasks.workunit.client.0.smithi154.stdout: list_parallel: ******************************* 2024-09-16T03:43:35.260 INFO:tasks.workunit.client.0.smithi154.stdout: list_parallel: ******************************* 2024-09-16T03:43:35.260 INFO:tasks.workunit.client.0.smithi154.stdout: list_parallel: process_15_[29715]: starting. 2024-09-16T03:43:35.260 INFO:tasks.workunit.client.0.smithi154.stdout: list_parallel: process_15_[29715]: listing objects. 2024-09-16T03:43:35.260 INFO:tasks.workunit.client.0.smithi154.stdout: list_parallel: process_15_[29715]: listed object 0... 2024-09-16T03:43:43.749 INFO:tasks.workunit.client.0.smithi154.stdout: list_parallel: process_15_[29715]: listed object 2 api_service: [==========] Running 4 tests from 1 test suite. 2024-09-16T03:43:43.749 INFO:tasks.workunit.client.0.smithi154.stdout: api_service: [----------] Global test environment set-up. 2024-09-16T03:43:43.749 INFO:tasks.workunit.client.0.smithi154.stdout: api_service: [----------] 4 tests from LibRadosService 2024-09-16T03:43:43.749 INFO:tasks.workunit.client.0.smithi154.stdout: api_service: [ RUN ] LibRadosService.RegisterEarly 2024-09-16T03:43:43.749 INFO:tasks.workunit.client.0.smithi154.stdout: api_service: [ OK ] LibRadosService.RegisterEarly (6631 ms) 2024-09-16T03:43:43.749 INFO:tasks.workunit.client.0.smithi154.stdout: api_service: [ RUN ] LibRadosService.RegisterLate 2024-09-16T03:43:43.749 INFO:tasks.workunit.client.0.smithi154.stdout: api_service: [ OK ] LibRadosService.RegisterLate (44 ms) 2024-09-16T03:43:43.749 INFO:tasks.workunit.client.0.smithi154.stdout: api_service: [ RUN ] LibRadosService.StatusFormat 2024-09-16T03:43:43.749 INFO:tasks.workunit.client.0.smithi154.stdout: api_service: cluster: 2024-09-16T03:43:43.750 INFO:tasks.workunit.client.0.smithi154.stdout: api_service: id: 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:43:43.750 INFO:tasks.workunit.client.0.smithi154.stdout: api_service: health: HEALTH_WARN 2024-09-16T03:43:43.750 INFO:tasks.workunit.client.0.smithi154.stdout: api_service: 15 pool(s) do not have an application enabled 2024-09-16T03:43:43.750 INFO:tasks.workunit.client.0.smithi154.stdout: api_service: 2024-09-16T03:43:43.750 INFO:tasks.workunit.client.0.smithi154.stdout: api_service: services: 2024-09-16T03:43:43.750 INFO:tasks.workunit.client.0.smithi154.stdout: api_service: mon: 3 daemons, quorum a,b,c (age 8s) 2024-09-16T03:43:43.750 INFO:tasks.workunit.client.0.smithi154.stdout: api_service: mgr: x(active, since 88s) 2024-09-16T03:43:43.750 INFO:tasks.workunit.client.0.smithi154.stdout: api_service: osd: 6 osds: 6 up (since 82s), 6 in (since 91s) 2024-09-16T03:43:43.750 INFO:tasks.workunit.client.0.smithi154.stdout: api_service: laundry: 2 daemons active (1 hosts) 2024-09-16T03:43:43.750 INFO:tasks.workunit.client.0.smithi154.stdout: api_service: 2024-09-16T03:43:43.750 INFO:tasks.workunit.client.0.smithi154.stdout: api_service: data: 2024-09-16T03:43:43.750 INFO:tasks.workunit.client.0.smithi154.stdout: api_service: pools: 35 pools, 1065 pgs 2024-09-16T03:43:43.750 INFO:tasks.workunit.client.0.smithi154.stdout: api_service: objects: 4 objects, 577 KiB 2024-09-16T03:43:43.750 INFO:tasks.workunit.client.0.smithi154.stdout: api_service: usage: 164 MiB used, 540 GiB / 540 GiB avail 2024-09-16T03:43:43.751 INFO:tasks.workunit.client.0.smithi154.stdout: api_service: pgs: 39.812% pgs unknown 2024-09-16T03:43:43.751 INFO:tasks.workunit.client.0.smithi154.stdout: api_service: 59.343% pgs not active 2024-09-16T03:43:43.751 INFO:tasks.workunit.client.0.smithi154.stdout: api_service: 632 creating+peering 2024-09-16T03:43:43.751 INFO:tasks.workunit.client.0.smithi154.stdout: api_service: 424 unknown 2024-09-16T03:43:43.751 INFO:tasks.workunit.client.0.smithi154.stdout: api_service: 9 active+clean 2024-09-16T03:43:43.751 INFO:tasks.workunit.client.0.smithi154.stdout: api_service: 2024-09-16T03:43:43.751 INFO:tasks.workunit.client.0.smithi154.stdout: api_service: 2024-09-16T03:43:43.751 INFO:tasks.workunit.client.0.smithi154.stdout: api_service: cluster: 2024-09-16T03:43:43.751 INFO:tasks.workunit.client.0.smithi154.stdout: api_service: id: 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:43:43.751 INFO:tasks.workunit.client.0.smithi154.stdout: api_service: health: HEALTH_WARN 2024-09-16T03:43:43.751 INFO:tasks.workunit.client.0.smithi154.stdout: api_service: 15 pool(s) do not have an application enabled 2024-09-16T03:43:43.751 INFO:tasks.workunit.client.0.smithi154.stdout: api_service: 2024-09-16T03:43:43.751 INFO:tasks.workunit.client.0.smithi154.stdout: api_service: services: 2024-09-16T03:43:43.751 INFO:tasks.workunit.client.0.smithi154.stdout: api_service: mon: 3 daemons, quorum a,b,c (age 10s) 2024-09-16T03:43:43.752 INFO:tasks.workunit.client.0.smithi154.stdout: api_service: mgr: x(active, since 90s) 2024-09-16T03:43:43.752 INFO:tasks.workunit.client.0.smithi154.stdout: api_service: osd: 6 osds: 6 up (since 84s), 6 in (since 93s) 2024-09-16T03:43:43.752 INFO:tasks.workunit.client.0.smithi154.stdout: api_service: laundry: 2 daemons active (1 hosts) 2024-09-16T03:43:43.752 INFO:tasks.workunit.client.0.smithi154.stdout: api_service: 2024-09-16T03:43:43.752 INFO:tasks.workunit.client.0.smithi154.stdout: api_service: data: 2024-09-16T03:43:43.752 INFO:tasks.workunit.client.0.smithi154.stdout: api_service: pools: 35 pools, 1065 pgs 2024-09-16T03:43:43.752 INFO:tasks.workunit.client.0.smithi154.stdout: api_service: objects: 4 objects, 577 KiB 2024-09-16T03:43:43.752 INFO:tasks.workunit.client.0.smithi154.stdout: api_service: usage: 164 MiB used, 540 GiB / 540 GiB avail 2024-09-16T03:43:43.752 INFO:tasks.workunit.client.0.smithi154.stdout: api_service: pgs: 39.812% pgs unknown 2024-09-16T03:43:43.752 INFO:tasks.workunit.client.0.smithi154.stdout: api_service: 59.343% pgs not active 2024-09-16T03:43:43.752 INFO:tasks.workunit.client.0.smithi154.stdout: api_service: 632 creating+peering 2024-09-16T03:43:43.752 INFO:tasks.workunit.client.0.smithi154.stdout: api_service: 424 unknown 2024-09-16T03:43:43.752 INFO:tasks.workunit.client.0.smithi154.stdout: api_service: 9 active+clean 2024-09-16T03:43:43.753 INFO:tasks.workunit.client.0.smithi154.stdout: api_service: 2024-09-16T03:43:43.753 INFO:tasks.workunit.client.0.smithi154.stdout: api_service: 2024-09-16T03:43:43.753 INFO:tasks.workunit.client.0.smithi154.stdout: api_service: cluster: 2024-09-16T03:43:43.753 INFO:tasks.workunit.client.0.smithi154.stdout: api_service: id: 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:43:43.753 INFO:tasks.workunit.client.0.smithi154.stdout: api_service: health: HEALTH_WARN 2024-09-16T03:43:43.753 INFO:tasks.workunit.client.0.smithi154.stdout: api_service: 15 pool(s) do not have an application enabled 2024-09-16T03:43:43.753 INFO:tasks.workunit.client.0.smithi154.stdout: api_service: 2024-09-16T03:43:43.753 INFO:tasks.workunit.client.0.smithi154.stdout: api_service: services: 2024-09-16T03:43:43.753 INFO:tasks.workunit.client.0.smithi154.stdout: api_service: mon: 3 daemons, quorum a,b,c (age 12s) 2024-09-16T03:43:43.753 INFO:tasks.workunit.client.0.smithi154.stdout: api_service: mgr: x(active, since 92s) 2024-09-16T03:43:43.753 INFO:tasks.workunit.client.0.smithi154.stdout: api_service: osd: 6 osds: 6 up (since 86s), 6 in (since 95s) 2024-09-16T03:43:43.753 INFO:tasks.workunit.client.0.smithi154.stdout: api_service: laundry: 2 daemons active (1 hosts) 2024-09-16T03:43:43.754 INFO:tasks.workunit.client.0.smithi154.stdout: api_service: 2024-09-16T03:43:43.754 INFO:tasks.workunit.client.0.smithi154.stdout: api_service: data: 2024-09-16T03:43:43.754 INFO:tasks.workunit.client.0.smithi154.stdout: api_service: pools: 35 pools, 1065 pgs 2024-09-16T03:43:43.754 INFO:tasks.workunit.client.0.smithi154.stdout: api_service: objects: 4 objects, 577 KiB 2024-09-16T03:43:43.754 INFO:tasks.workunit.client.0.smithi154.stdout: api_service: usage: 164 MiB used, 540 GiB / 540 GiB avail 2024-09-16T03:43:43.754 INFO:tasks.workunit.client.0.smithi154.stdout: api_service: pgs: 39.812% pgs unknown 2024-09-16T03:43:43.754 INFO:tasks.workunit.client.0.smithi154.stdout: api_service: 59.343% pgs not active 2024-09-16T03:43:43.754 INFO:tasks.workunit.client.0.smithi154.stdout: api_service: 632 creating+peering 2024-09-16T03:43:50.181 INFO:tasks.mon_thrash:Sending CONT to mon c 2024-09-16T03:43:50.181 INFO:tasks.ceph.mon.c:Sent signal 18 2024-09-16T03:43:50.181 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T03:43:50.182 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:43:50.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:50.309+0000 7f42ff4c9640 10 monclient: get_monmap_and_config 2024-09-16T03:43:50.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:50.309+0000 7f42ff4c9640 10 monclient: build_initial_monmap 2024-09-16T03:43:50.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:50.309+0000 7f42ff4c9640 10 monclient: monmap: 2024-09-16T03:43:50.317 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:43:50.317 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:43:50.317 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:43:50.315672+0000 2024-09-16T03:43:50.317 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:43:50.315672+0000 2024-09-16T03:43:50.317 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:43:50.318 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:43:50.318 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:43:50.318 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:43:50.318 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:43:50.318 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:43:50.318 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:43:50.318 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:43:50.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:50.313+0000 7f42ff4c9640 1 Processor -- start 2024-09-16T03:43:50.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:50.313+0000 7f42ff4c9640 1 -- start start 2024-09-16T03:43:50.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:50.313+0000 7f42ff4c9640 10 monclient: init 2024-09-16T03:43:50.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:50.313+0000 7f42ff4c9640 10 monclient: _reopen_session rank -1 2024-09-16T03:43:50.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:50.313+0000 7f42ff4c9640 10 monclient: _add_conns ranks=[4,0,3,2,1] 2024-09-16T03:43:50.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:50.313+0000 7f42ff4c9640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f42f809fb00 0x7f42f80a1330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:43:50.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:50.313+0000 7f42ff4c9640 10 monclient(hunting): picked mon.noname-c con 0x7f42f809fb00 addr v2:172.21.15.154:3300/0 2024-09-16T03:43:50.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:50.313+0000 7f42ff4c9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f42f809ba50 addr v1:172.21.15.42:6789/0 2024-09-16T03:43:50.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:50.313+0000 7f42ff4c9640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f42f809c5d0 0x7f42f813c240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:43:50.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:50.313+0000 7f42ff4c9640 10 monclient(hunting): picked mon.noname-a con 0x7f42f809c5d0 addr v2:172.21.15.42:3300/0 2024-09-16T03:43:50.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:50.313+0000 7f42ff4c9640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f42f80a1e20 con 0x7f42f809ba50 2024-09-16T03:43:50.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:50.313+0000 7f42ff4c9640 10 monclient(hunting): start opening mon connection 2024-09-16T03:43:50.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:50.313+0000 7f42ff4c9640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f42f809e250 con 0x7f42f809c5d0 2024-09-16T03:43:50.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:50.313+0000 7f42ff4c9640 10 monclient(hunting): start opening mon connection 2024-09-16T03:43:50.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:50.313+0000 7f42ff4c9640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f42f809ea30 con 0x7f42f809fb00 2024-09-16T03:43:50.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:50.313+0000 7f42ff4c9640 10 monclient(hunting): _renew_subs 2024-09-16T03:43:50.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:50.313+0000 7f42ff4c9640 10 monclient(hunting): authenticate will time out at 1077.285113s 2024-09-16T03:43:50.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:50.313+0000 7f42fda66640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f42f809c5d0 0x7f42f813c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:43:50.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:50.313+0000 7f42fca64640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f42f809ba50 0x7f42f809bec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:51238/0 (socket says 172.21.15.42:51238) 2024-09-16T03:43:50.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:50.313+0000 7f42fca64640 1 -- 172.21.15.42:0/3680561365 learned_addr learned my addr 172.21.15.42:0/3680561365 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:43:50.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:50.313+0000 7f42fd265640 1 -- 172.21.15.42:0/3680561365 >> v2:172.21.15.154:3300/0 conn(0x7f42f809fb00 msgr2=0x7f42f80a1330 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T03:43:50.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:50.313+0000 7f42fd265640 1 --2- 172.21.15.42:0/3680561365 >> v2:172.21.15.154:3300/0 conn(0x7f42f809fb00 0x7f42f80a1330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:43:50.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:50.313+0000 7f42fda66640 10 monclient(hunting): get_auth_request con 0x7f42f809c5d0 auth_method 0 2024-09-16T03:43:50.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:50.313+0000 7f42fda66640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:43:50.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:50.313+0000 7f42fda66640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:43:50.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:50.313+0000 7f42fda66640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:43:50.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:50.313+0000 7f42fda66640 1 -- 172.21.15.42:0/3680561365 >> v2:172.21.15.42:3300/0 conn(0x7f42f809c5d0 msgr2=0x7f42f813c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T03:43:50.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:50.313+0000 7f42fda66640 1 -- 172.21.15.42:0/3680561365 >> v2:172.21.15.42:3300/0 conn(0x7f42f809c5d0 msgr2=0x7f42f813c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:43:50.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:50.313+0000 7f42fda66640 1 --2- 172.21.15.42:0/3680561365 >> v2:172.21.15.42:3300/0 conn(0x7f42f809c5d0 0x7f42f813c240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:43:50.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:50.313+0000 7f42fda66640 1 --2- 172.21.15.42:0/3680561365 >> v2:172.21.15.42:3300/0 conn(0x7f42f809c5d0 0x7f42f813c240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:43:50.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:50.513+0000 7f42fd265640 1 -- 172.21.15.42:0/3680561365 >> v2:172.21.15.154:3300/0 conn(0x7f42f809fb00 msgr2=0x7f42f80a1330 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T03:43:50.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:50.513+0000 7f42fd265640 1 --2- 172.21.15.42:0/3680561365 >> v2:172.21.15.154:3300/0 conn(0x7f42f809fb00 0x7f42f80a1330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T03:43:50.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:50.513+0000 7f42fda66640 1 --2- 172.21.15.42:0/3680561365 >> v2:172.21.15.42:3300/0 conn(0x7f42f809c5d0 0x7f42f813c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:43:50.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:50.513+0000 7f42fda66640 10 monclient(hunting): get_auth_request con 0x7f42f809c5d0 auth_method 0 2024-09-16T03:43:50.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:50.513+0000 7f42fda66640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:43:50.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:50.513+0000 7f42fda66640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:43:50.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:50.513+0000 7f42fda66640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:43:50.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:50.513+0000 7f42fda66640 1 -- 172.21.15.42:0/3680561365 >> v2:172.21.15.42:3300/0 conn(0x7f42f809c5d0 msgr2=0x7f42f813c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T03:43:50.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:50.513+0000 7f42fda66640 1 -- 172.21.15.42:0/3680561365 >> v2:172.21.15.42:3300/0 conn(0x7f42f809c5d0 msgr2=0x7f42f813c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:43:50.521 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:50.513+0000 7f42fda66640 1 --2- 172.21.15.42:0/3680561365 >> v2:172.21.15.42:3300/0 conn(0x7f42f809c5d0 0x7f42f813c240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:43:50.521 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:50.513+0000 7f42fda66640 1 --2- 172.21.15.42:0/3680561365 >> v2:172.21.15.42:3300/0 conn(0x7f42f809c5d0 0x7f42f813c240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T03:43:50.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:50.913+0000 7f42fd265640 1 -- 172.21.15.42:0/3680561365 >> v2:172.21.15.154:3300/0 conn(0x7f42f809fb00 msgr2=0x7f42f80a1330 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T03:43:50.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:50.913+0000 7f42fd265640 1 --2- 172.21.15.42:0/3680561365 >> v2:172.21.15.154:3300/0 conn(0x7f42f809fb00 0x7f42f80a1330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T03:43:50.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:50.913+0000 7f42fda66640 1 --2- 172.21.15.42:0/3680561365 >> v2:172.21.15.42:3300/0 conn(0x7f42f809c5d0 0x7f42f813c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:43:50.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:50.913+0000 7f42fda66640 10 monclient(hunting): get_auth_request con 0x7f42f809c5d0 auth_method 0 2024-09-16T03:43:50.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:50.913+0000 7f42fda66640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:43:50.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:50.913+0000 7f42fda66640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:43:50.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:50.913+0000 7f42fda66640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:43:50.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:50.913+0000 7f42fda66640 1 -- 172.21.15.42:0/3680561365 >> v2:172.21.15.42:3300/0 conn(0x7f42f809c5d0 msgr2=0x7f42f813c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T03:43:50.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:50.913+0000 7f42fda66640 1 -- 172.21.15.42:0/3680561365 >> v2:172.21.15.42:3300/0 conn(0x7f42f809c5d0 msgr2=0x7f42f813c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:43:50.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:50.913+0000 7f42fda66640 1 --2- 172.21.15.42:0/3680561365 >> v2:172.21.15.42:3300/0 conn(0x7f42f809c5d0 0x7f42f813c240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:43:50.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:50.913+0000 7f42fda66640 1 --2- 172.21.15.42:0/3680561365 >> v2:172.21.15.42:3300/0 conn(0x7f42f809c5d0 0x7f42f813c240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T03:43:51.721 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:51.717+0000 7f42fd265640 1 -- 172.21.15.42:0/3680561365 >> v2:172.21.15.154:3300/0 conn(0x7f42f809fb00 msgr2=0x7f42f80a1330 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T03:43:51.721 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:51.717+0000 7f42fd265640 1 --2- 172.21.15.42:0/3680561365 >> v2:172.21.15.154:3300/0 conn(0x7f42f809fb00 0x7f42f80a1330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T03:43:51.722 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:51.717+0000 7f42fda66640 1 --2- 172.21.15.42:0/3680561365 >> v2:172.21.15.42:3300/0 conn(0x7f42f809c5d0 0x7f42f813c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:43:51.722 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:51.717+0000 7f42fda66640 10 monclient(hunting): get_auth_request con 0x7f42f809c5d0 auth_method 0 2024-09-16T03:43:51.722 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:51.717+0000 7f42fda66640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:43:51.722 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:51.717+0000 7f42fda66640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:43:51.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:51.717+0000 7f42fda66640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:43:51.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:51.717+0000 7f42fda66640 1 -- 172.21.15.42:0/3680561365 >> v2:172.21.15.42:3300/0 conn(0x7f42f809c5d0 msgr2=0x7f42f813c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T03:43:51.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:51.717+0000 7f42fda66640 1 -- 172.21.15.42:0/3680561365 >> v2:172.21.15.42:3300/0 conn(0x7f42f809c5d0 msgr2=0x7f42f813c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:43:51.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:51.717+0000 7f42fda66640 1 --2- 172.21.15.42:0/3680561365 >> v2:172.21.15.42:3300/0 conn(0x7f42f809c5d0 0x7f42f813c240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:43:51.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:51.717+0000 7f42fda66640 1 --2- 172.21.15.42:0/3680561365 >> v2:172.21.15.42:3300/0 conn(0x7f42f809c5d0 0x7f42f813c240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T03:43:53.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:53.313+0000 7f42eeffd640 10 monclient(hunting): tick 2024-09-16T03:43:53.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:53.313+0000 7f42eeffd640 10 monclient(hunting): _check_auth_tickets 2024-09-16T03:43:53.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:53.313+0000 7f42eeffd640 1 monclient(hunting): continuing hunt 2024-09-16T03:43:53.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:53.313+0000 7f42eeffd640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T03:43:53.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:53.313+0000 7f42eeffd640 1 -- 172.21.15.42:0/3680561365 >> v2:172.21.15.154:3300/0 conn(0x7f42f809fb00 msgr2=0x7f42f80a1330 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:43:53.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:53.313+0000 7f42eeffd640 1 --2- 172.21.15.42:0/3680561365 >> v2:172.21.15.154:3300/0 conn(0x7f42f809fb00 0x7f42f80a1330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:43:53.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:53.313+0000 7f42eeffd640 1 -- 172.21.15.42:0/3680561365 >> v2:172.21.15.42:3300/0 conn(0x7f42f809c5d0 msgr2=0x7f42f813c240 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:43:53.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:53.313+0000 7f42eeffd640 1 --2- 172.21.15.42:0/3680561365 >> v2:172.21.15.42:3300/0 conn(0x7f42f809c5d0 0x7f42f813c240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:43:53.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:53.313+0000 7f42eeffd640 1 -- 172.21.15.42:0/3680561365 >> v1:172.21.15.42:6789/0 conn(0x7f42f809ba50 legacy=0x7f42f809bec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:43:53.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:53.313+0000 7f42eeffd640 10 monclient: _add_conns ranks=[2,1] 2024-09-16T03:43:53.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:53.313+0000 7f42eeffd640 1 --2- 172.21.15.42:0/3680561365 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f42dc0036a0 0x7f42dc005af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:43:53.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:53.313+0000 7f42eeffd640 10 monclient(hunting): picked mon.noname-b con 0x7f42dc0036a0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:43:53.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:53.313+0000 7f42eeffd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f42dc006220 addr v1:172.21.15.154:6789/0 2024-09-16T03:43:53.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:53.313+0000 7f42eeffd640 1 -- 172.21.15.42:0/3680561365 --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f42f809fee0 con 0x7f42dc006220 2024-09-16T03:43:53.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:53.313+0000 7f42eeffd640 10 monclient(hunting): start opening mon connection 2024-09-16T03:43:53.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:53.313+0000 7f42eeffd640 1 -- 172.21.15.42:0/3680561365 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f42f809e250 con 0x7f42dc0036a0 2024-09-16T03:43:53.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:53.313+0000 7f42eeffd640 10 monclient(hunting): _renew_subs 2024-09-16T03:43:53.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:53.313+0000 7f42fca64640 1 -- 172.21.15.42:0/3680561365 >> v1:172.21.15.154:6789/0 conn(0x7f42dc006220 legacy=0x7f42dc0086c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.154:6789/0 2024-09-16T03:43:53.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:53.313+0000 7f42fd265640 1 --2- 172.21.15.42:0/3680561365 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f42dc0036a0 0x7f42dc005af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:43:53.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:53.313+0000 7f42fd265640 10 monclient(hunting): get_auth_request con 0x7f42dc0036a0 auth_method 0 2024-09-16T03:43:53.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:53.313+0000 7f42fd265640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:43:53.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:53.313+0000 7f42fd265640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:43:53.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:53.313+0000 7f42fd265640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:43:53.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:53.313+0000 7f42fd265640 1 -- 172.21.15.42:0/3680561365 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f42dc0036a0 msgr2=0x7f42dc005af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T03:43:53.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:53.313+0000 7f42fd265640 1 -- 172.21.15.42:0/3680561365 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f42dc0036a0 msgr2=0x7f42dc005af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:43:53.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:53.313+0000 7f42fd265640 1 --2- 172.21.15.42:0/3680561365 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f42dc0036a0 0x7f42dc005af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:43:53.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:53.313+0000 7f42fd265640 1 --2- 172.21.15.42:0/3680561365 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f42dc0036a0 0x7f42dc005af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:43:53.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:53.513+0000 7f42fca64640 1 -- 172.21.15.42:0/3680561365 >> v1:172.21.15.154:6789/0 conn(0x7f42dc006220 legacy=0x7f42dc0086c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.154:6789/0 2024-09-16T03:43:53.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:53.513+0000 7f42fd265640 1 --2- 172.21.15.42:0/3680561365 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f42dc0036a0 0x7f42dc005af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:43:53.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:53.513+0000 7f42fd265640 10 monclient(hunting): get_auth_request con 0x7f42dc0036a0 auth_method 0 2024-09-16T03:43:53.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:53.513+0000 7f42fd265640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:43:53.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:53.513+0000 7f42fd265640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:43:53.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:53.513+0000 7f42fd265640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:43:53.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:53.513+0000 7f42fd265640 1 -- 172.21.15.42:0/3680561365 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f42dc0036a0 msgr2=0x7f42dc005af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T03:43:53.521 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:53.513+0000 7f42fd265640 1 -- 172.21.15.42:0/3680561365 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f42dc0036a0 msgr2=0x7f42dc005af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:43:53.521 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:53.513+0000 7f42fd265640 1 --2- 172.21.15.42:0/3680561365 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f42dc0036a0 0x7f42dc005af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:43:53.521 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:53.513+0000 7f42fd265640 1 --2- 172.21.15.42:0/3680561365 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f42dc0036a0 0x7f42dc005af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T03:43:53.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:53.913+0000 7f42fca64640 1 -- 172.21.15.42:0/3680561365 >> v1:172.21.15.154:6789/0 conn(0x7f42dc006220 legacy=0x7f42dc0086c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.154:6789/0 2024-09-16T03:43:53.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:53.913+0000 7f42fd265640 1 --2- 172.21.15.42:0/3680561365 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f42dc0036a0 0x7f42dc005af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:43:53.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:53.913+0000 7f42fd265640 10 monclient(hunting): get_auth_request con 0x7f42dc0036a0 auth_method 0 2024-09-16T03:43:53.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:53.913+0000 7f42fd265640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:43:53.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:53.913+0000 7f42fd265640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:43:53.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:53.913+0000 7f42fd265640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:43:53.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:53.917+0000 7f42fd265640 1 -- 172.21.15.42:0/3680561365 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f42dc0036a0 msgr2=0x7f42dc005af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T03:43:53.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:53.917+0000 7f42fd265640 1 -- 172.21.15.42:0/3680561365 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f42dc0036a0 msgr2=0x7f42dc005af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:43:53.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:53.917+0000 7f42fd265640 1 --2- 172.21.15.42:0/3680561365 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f42dc0036a0 0x7f42dc005af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:43:53.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:53.917+0000 7f42fd265640 1 --2- 172.21.15.42:0/3680561365 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f42dc0036a0 0x7f42dc005af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T03:43:54.721 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:54.717+0000 7f42fca64640 1 -- 172.21.15.42:0/3680561365 >> v1:172.21.15.154:6789/0 conn(0x7f42dc006220 legacy=0x7f42dc0086c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.154:6789/0 2024-09-16T03:43:54.725 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:54.721+0000 7f42fd265640 1 --2- 172.21.15.42:0/3680561365 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f42dc0036a0 0x7f42dc005af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:43:54.725 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:54.721+0000 7f42fd265640 10 monclient(hunting): get_auth_request con 0x7f42dc0036a0 auth_method 0 2024-09-16T03:43:54.726 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:54.721+0000 7f42fd265640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:43:54.726 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:54.721+0000 7f42fd265640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:43:54.726 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:54.721+0000 7f42fd265640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:43:54.726 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:54.721+0000 7f42fd265640 1 -- 172.21.15.42:0/3680561365 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f42dc0036a0 msgr2=0x7f42dc005af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T03:43:54.726 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:54.721+0000 7f42fd265640 1 -- 172.21.15.42:0/3680561365 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f42dc0036a0 msgr2=0x7f42dc005af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:43:54.726 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:54.721+0000 7f42fd265640 1 --2- 172.21.15.42:0/3680561365 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f42dc0036a0 0x7f42dc005af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:43:54.726 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:54.721+0000 7f42fd265640 1 --2- 172.21.15.42:0/3680561365 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f42dc0036a0 0x7f42dc005af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T03:43:56.271 INFO:tasks.workunit.client.0.smithi154.stdout: api_service: 4245... 2024-09-16T03:43:56.271 INFO:tasks.workunit.client.0.smithi154.stdout: list_parallel: process_15_[29715]: listed object 50... 2024-09-16T03:43:56.271 INFO:tasks.workunit.client.0.smithi154.stdout: list_parallel: process_15_[29715]: listed object 75... 2024-09-16T03:43:56.271 INFO:tasks.workunit.client.0.smithi154.stdout: list_parallel: process_15_[29715]: listed object 100... 2024-09-16T03:43:56.272 INFO:tasks.workunit.client.0.smithi154.stdout: list_parallel: process_15_[29715]: listed object 125... 2024-09-16T03:43:56.272 INFO:tasks.workunit.client.0.smithi154.stdout: list_parallel: process_15_[29715]: saw 150 objects 2024-09-16T03:43:56.272 INFO:tasks.workunit.client.0.smithi154.stdout: list_parallel: process_15_[29715]: shutting down. 2024-09-16T03:43:56.272 INFO:tasks.workunit.client.0.smithi154.stdout: list_parallel: ******************************* 2024-09-16T03:43:56.272 INFO:tasks.workunit.client.0.smithi154.stdout: list_parallel: ******************************* 2024-09-16T03:43:56.272 INFO:tasks.workunit.client.0.smithi154.stdout: list_parallel: ******************************* 2024-09-16T03:43:56.272 INFO:tasks.workunit.client.0.smithi154.stdout: list_parallel: ******************************* 2024-09-16T03:43:56.272 INFO:tasks.workunit.client.0.smithi154.stdout: list_parallel: process_16_[29716]: starting. 2024-09-16T03:43:56.272 INFO:tasks.workunit.client.0.smithi154.stdout: list_parallel: process_16_[29716]: added 25 objects... 2024-09-16T03:43:56.272 INFO:tasks.workunit.client.0.smithi154.stdout: list_parallel: process_16_[29716]: added half of the objects 2024-09-16T03:43:56.272 INFO:tasks.workunit.client.0.smithi154.stdout: list_parallel: process_16_[29716]: added 50 objects... 2024-09-16T03:43:56.272 INFO:tasks.workunit.client.0.smithi154.stdout: list_parallel: process_16_[29716]: added 50 objects 2024-09-16T03:43:56.272 INFO:tasks.workunit.client.0.smithi154.stdout: list_parallel: process_16_[29716]: shutting down. 2024-09-16T03:43:56.272 INFO:tasks.workunit.client.0.smithi154.stdout: list_parallel: ******************************* 2024-09-16T03:43:56.273 INFO:tasks.workunit.client.0.smithi154.stdout: list_parallel: ******************************* 2024-09-16T03:43:56.273 INFO:tasks.workunit.client.0.smithi154.stdout: list_parallel: ******************************* 2024-09-16T03:43:56.273 INFO:tasks.workunit.client.0.smithi154.stdout: list_parallel: ******************************* 2024-09-16T03:43:56.273 INFO:tasks.workunit.client.0.smithi154.stdout: list_parallel: ******************************* 2024-09-16T03:43:56.273 INFO:tasks.workunit.client.0.smithi154.stdout: list_parallel: ******* SUCCESS ********** 2024-09-16T03:43:56.288 INFO:tasks.workunit.client.0.smithi154.stdout: api_c_read_operations: Running main() from gmock_main.cc 2024-09-16T03:43:56.288 INFO:tasks.workunit.client.0.smithi154.stdout: api_c_read_operations: [==========] Running 17 tests from 1 test suite. 2024-09-16T03:43:56.288 INFO:tasks.workunit.client.0.smithi154.stdout: api_c_read_operations: [----------] Global test environment set-up. 2024-09-16T03:43:56.288 INFO:tasks.workunit.client.0.smithi154.stdout: api_c_read_operations: [----------] 17 tests from CReadOpsTest 2024-09-16T03:43:56.288 INFO:tasks.workunit.client.0.smithi154.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.NewDelete 2024-09-16T03:43:56.288 INFO:tasks.workunit.client.0.smithi154.stdout: api_c_read_operations: [ OK ] CReadOpsTest.NewDelete (0 ms) 2024-09-16T03:43:56.288 INFO:tasks.workunit.client.0.smithi154.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.SetOpFlags 2024-09-16T03:43:56.288 INFO:tasks.workunit.client.0.smithi154.stdout: api_c_read_operations: [ OK ] CReadOpsTest.SetOpFlags (207 ms) 2024-09-16T03:43:56.288 INFO:tasks.workunit.client.0.smithi154.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.AssertExists 2024-09-16T03:43:56.289 INFO:tasks.workunit.client.0.smithi154.stdout: api_c_read_operations: [ OK ] CReadOpsTest.AssertExists (32 ms) 2024-09-16T03:43:56.289 INFO:tasks.workunit.client.0.smithi154.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.AssertVersion 2024-09-16T03:43:56.289 INFO:tasks.workunit.client.0.smithi154.stdout: api_c_read_operations: [ OK ] CReadOpsTest.AssertVersion (23 ms) 2024-09-16T03:43:56.289 INFO:tasks.workunit.client.0.smithi154.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.CmpXattr 2024-09-16T03:43:56.289 INFO:tasks.workunit.client.0.smithi154.stdout: api_c_read_operations: [ OK ] CReadOpsTest.CmpXattr (43 ms) 2024-09-16T03:43:56.289 INFO:tasks.workunit.client.0.smithi154.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Read 2024-09-16T03:43:56.289 INFO:tasks.workunit.client.0.smithi154.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Read (16 ms) 2024-09-16T03:43:56.289 INFO:tasks.workunit.client.0.smithi154.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Checksum 2024-09-16T03:43:56.289 INFO:tasks.workunit.client.0.smithi154.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Checksum (5 ms) 2024-09-16T03:43:56.289 INFO:tasks.workunit.client.0.smithi154.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.RWOrderedRead 2024-09-16T03:43:56.289 INFO:tasks.workunit.client.0.smithi154.stdout: api_c_read_operations: [ OK ] CReadOpsTest.RWOrderedRead (5 ms) 2024-09-16T03:43:56.289 INFO:tasks.workunit.client.0.smithi154.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.ShortRead 2024-09-16T03:43:56.289 INFO:tasks.workunit.client.0.smithi154.stdout: api_c_read_operations: [ OK ] CReadOpsTest.ShortRead (8 ms) 2024-09-16T03:43:56.289 INFO:tasks.workunit.client.0.smithi154.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Exec 2024-09-16T03:43:56.290 INFO:tasks.workunit.client.0.smithi154.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Exec (4 ms) 2024-09-16T03:43:56.290 INFO:tasks.workunit.client.0.smithi154.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.ExecUserBuf 2024-09-16T03:43:56.290 INFO:tasks.workunit.client.0.smithi154.stdout: api_c_read_operations: [ OK ] CReadOpsTest.ExecUserBuf (3 ms) 2024-09-16T03:43:56.290 INFO:tasks.workunit.client.0.smithi154.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Stat 2024-09-16T03:43:56.290 INFO:tasks.workunit.client.0.smithi154.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Stat (6 ms) 2024-09-16T03:43:56.290 INFO:tasks.workunit.client.0.smithi154.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Stat2 2024-09-16T03:43:56.290 INFO:tasks.workunit.client.0.smithi154.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Stat2 (5 ms) 2024-09-16T03:43:56.290 INFO:tasks.workunit.client.0.smithi154.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Omap 2024-09-16T03:43:56.290 INFO:tasks.workunit.client.0.smithi154.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Omap (24 ms) 2024-09-16T03:43:56.290 INFO:tasks.workunit.client.0.smithi154.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.OmapNuls 2024-09-16T03:43:56.290 INFO:tasks.workunit.client.0.smithi154.stdout: api_c_read_operations: [ OK ] CReadOpsTest.OmapNuls (13 ms) 2024-09-16T03:43:56.290 INFO:tasks.workunit.client.0.smithi154.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.GetXattrs 2024-09-16T03:43:56.290 INFO:tasks.workunit.client.0.smithi154.stdout: api_c_read_operations: [ OK ] CReadOpsTest.GetXattrs (12 ms) 2024-09-16T03:43:56.290 INFO:tasks.workunit.client.0.smithi154.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.CmpExt 2024-09-16T03:43:56.291 INFO:tasks.workunit.client.0.smithi154.stdout: api_c_read_operations: [ OK ] CReadOpsTest.CmpExt (7 ms) 2024-09-16T03:43:56.291 INFO:tasks.workunit.client.0.smithi154.stdout: api_c_read_operations: [----------] 17 tests from CReadOpsTest (413 ms total) 2024-09-16T03:43:56.291 INFO:tasks.workunit.client.0.smithi154.stdout: api_c_read_operations: 2024-09-16T03:43:56.291 INFO:tasks.workunit.client.0.smithi154.stdout: api_c_read_operations: [----------] Global test environment tear-down 2024-09-16T03:43:56.291 INFO:tasks.workunit.client.0.smithi154.stdout: api_c_read_operations: [==========] 17 tests from 1 test suite ran. (26583 ms total) 2024-09-16T03:43:56.291 INFO:tasks.workunit.client.0.smithi154.stdout: api_c_read_operations: [ PASSED ] 17 tests. 2024-09-16T03:43:56.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.313+0000 7f42eeffd640 10 monclient(hunting): tick 2024-09-16T03:43:56.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.313+0000 7f42eeffd640 10 monclient(hunting): _check_auth_tickets 2024-09-16T03:43:56.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.313+0000 7f42eeffd640 1 monclient(hunting): continuing hunt 2024-09-16T03:43:56.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.313+0000 7f42eeffd640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T03:43:56.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.313+0000 7f42eeffd640 1 -- 172.21.15.42:0/3680561365 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f42dc0036a0 msgr2=0x7f42dc005af0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:43:56.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.313+0000 7f42eeffd640 1 --2- 172.21.15.42:0/3680561365 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f42dc0036a0 0x7f42dc005af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:43:56.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.313+0000 7f42eeffd640 1 -- 172.21.15.42:0/3680561365 >> v1:172.21.15.154:6789/0 conn(0x7f42dc006220 legacy=0x7f42dc0086c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:43:56.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.313+0000 7f42eeffd640 10 monclient: _add_conns ranks=[3,4,1,2,0] 2024-09-16T03:43:56.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.313+0000 7f42eeffd640 1 --2- 172.21.15.42:0/3680561365 >> v2:172.21.15.42:3300/0 conn(0x7f42f809c5d0 0x7f42dc009f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:43:56.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.313+0000 7f42eeffd640 10 monclient(hunting): picked mon.noname-a con 0x7f42f809c5d0 addr v2:172.21.15.42:3300/0 2024-09-16T03:43:56.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.313+0000 7f42eeffd640 1 --2- 172.21.15.42:0/3680561365 >> v2:172.21.15.154:3300/0 conn(0x7f42f809fb00 0x7f42dc00a500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:43:56.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.313+0000 7f42eeffd640 10 monclient(hunting): picked mon.noname-c con 0x7f42f809fb00 addr v2:172.21.15.154:3300/0 2024-09-16T03:43:56.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.313+0000 7f42eeffd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f42dc00aa90 addr v1:172.21.15.154:6789/0 2024-09-16T03:43:56.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.313+0000 7f42eeffd640 1 -- 172.21.15.42:0/3680561365 --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f42dc00b060 con 0x7f42dc00aa90 2024-09-16T03:43:56.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.313+0000 7f42eeffd640 10 monclient(hunting): start opening mon connection 2024-09-16T03:43:56.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.313+0000 7f42eeffd640 1 -- 172.21.15.42:0/3680561365 --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f42f809e250 con 0x7f42f809c5d0 2024-09-16T03:43:56.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.313+0000 7f42eeffd640 10 monclient(hunting): start opening mon connection 2024-09-16T03:43:56.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.313+0000 7f42eeffd640 1 -- 172.21.15.42:0/3680561365 --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f42f809ea30 con 0x7f42f809fb00 2024-09-16T03:43:56.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.313+0000 7f42eeffd640 10 monclient(hunting): _renew_subs 2024-09-16T03:43:56.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.313+0000 7f42fda66640 1 -- 172.21.15.42:0/3680561365 reap_dead start 2024-09-16T03:43:56.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.313+0000 7f42fca64640 1 -- 172.21.15.42:0/3680561365 >> v2:172.21.15.154:3300/0 conn(0x7f42f809fb00 msgr2=0x7f42dc00a500 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T03:43:56.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.313+0000 7f42fca64640 1 --2- 172.21.15.42:0/3680561365 >> v2:172.21.15.154:3300/0 conn(0x7f42f809fb00 0x7f42dc00a500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:43:56.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.313+0000 7f42fd265640 1 --2- 172.21.15.42:0/3680561365 >> v2:172.21.15.42:3300/0 conn(0x7f42f809c5d0 0x7f42dc009f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:43:56.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.313+0000 7f42fd265640 10 monclient(hunting): get_auth_request con 0x7f42f809c5d0 auth_method 0 2024-09-16T03:43:56.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.313+0000 7f42fd265640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:43:56.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.313+0000 7f42fd265640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:43:56.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.313+0000 7f42fd265640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:43:56.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.313+0000 7f42fd265640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:43:56.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.313+0000 7f42fd265640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:43:56.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.313+0000 7f42fd265640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:43:56.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.313+0000 7f42fda66640 1 -- 172.21.15.42:0/3680561365 >> v1:172.21.15.154:6789/0 conn(0x7f42dc00aa90 legacy=0x7f42dc0114c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.154:6789/0 2024-09-16T03:43:56.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.317+0000 7f42fd265640 10 monclient(hunting): handle_auth_done global_id 5067 payload 274 2024-09-16T03:43:56.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.317+0000 7f42fd265640 1 -- 172.21.15.42:0/3680561365 >> v2:172.21.15.154:3300/0 conn(0x7f42f809fb00 msgr2=0x7f42dc00a500 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:43:56.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.317+0000 7f42fd265640 1 --2- 172.21.15.42:0/3680561365 >> v2:172.21.15.154:3300/0 conn(0x7f42f809fb00 0x7f42dc00a500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:43:56.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.317+0000 7f42fd265640 1 -- 172.21.15.42:0/3680561365 >> v1:172.21.15.154:6789/0 conn(0x7f42dc00aa90 legacy=0x7f42dc0114c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:43:56.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.317+0000 7f42fd265640 10 monclient: _finish_hunting 0 2024-09-16T03:43:56.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.317+0000 7f42fd265640 1 monclient: found mon.noname-a 2024-09-16T03:43:56.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.317+0000 7f42fd265640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:43:56.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.317+0000 7f42fd265640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:43:56.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.317+0000 7f42fd265640 1 -- 172.21.15.42:0/3680561365 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42f809fee0 con 0x7f42f809c5d0 2024-09-16T03:43:56.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.321+0000 7f42fd265640 1 --2- 172.21.15.42:0/3680561365 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f42f809c5d0 0x7f42dc009f60 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f42f80a2230 tx=0x7f42f401d410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb0dee673309b413 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:43:56.389 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.385+0000 7f42effff640 1 -- 172.21.15.42:0/3680561365 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42f40ac070 con 0x7f42f809c5d0 2024-09-16T03:43:56.389 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.385+0000 7f42effff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:43:56.389 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.385+0000 7f42effff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:43:56.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.385+0000 7f42effff640 10 monclient: dump: 2024-09-16T03:43:56.390 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:43:56.390 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:43:56.390 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:43:56.390 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:43:56.390 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:43:56.390 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:43:56.390 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:43:56.390 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:43:56.390 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:43:56.390 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:43:56.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.385+0000 7f42effff640 10 monclient: _finish_auth 0 2024-09-16T03:43:56.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.385+0000 7f42effff640 10 monclient: _check_auth_tickets 2024-09-16T03:43:56.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.385+0000 7f42effff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:43:56.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.385+0000 7f42effff640 1 -- 172.21.15.42:0/3680561365 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f42f40b0400 con 0x7f42f809c5d0 2024-09-16T03:43:56.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.385+0000 7f42ff4c9640 5 monclient: authenticate success, global_id 5067 2024-09-16T03:43:56.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.385+0000 7f42ff4c9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:43:56.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.385+0000 7f42effff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:43:56.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.385+0000 7f42effff640 1 -- 172.21.15.42:0/3680561365 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42f40b07f0 con 0x7f42f809c5d0 2024-09-16T03:43:56.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.385+0000 7f42effff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:43:56.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.385+0000 7f42effff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:43:56.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.385+0000 7f42effff640 10 monclient: dump: 2024-09-16T03:43:56.391 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:43:56.391 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:43:56.391 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:43:56.391 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:43:56.391 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:43:56.391 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:43:56.392 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:43:56.392 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:43:56.392 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:43:56.392 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:43:56.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.385+0000 7f42ff4c9640 10 monclient: get_monmap_and_config success 2024-09-16T03:43:56.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.385+0000 7f42ff4c9640 4 set_mon_vals no callback set 2024-09-16T03:43:56.393 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd: Running main() from gmock_main.cc 2024-09-16T03:43:56.393 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd: [==========] Running 4 tests from 1 test suite. 2024-09-16T03:43:56.393 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd: [----------] Global test environment set-up. 2024-09-16T03:43:56.393 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd: [----------] 4 tests from LibRadosCmd 2024-09-16T03:43:56.393 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd: [ RUN ] LibRadosCmd.MonDescribe 2024-09-16T03:43:56.393 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd: [ OK ] LibRadosCmd.MonDescribe (1710 ms) 2024-09-16T03:43:56.393 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd: [ RUN ] LibRadosCmd.OSDCmd 2024-09-16T03:43:56.393 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd: [ OK ] LibRadosCmd.OSDCmd (20 ms) 2024-09-16T03:43:56.393 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd: [ RUN ] LibRadosCmd.PGCmd 2024-09-16T03:43:56.393 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd: [ OK ] LibRadosCmd.PGCmd (24866 ms) 2024-09-16T03:43:56.394 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd: [ RUN ] LibRadosCmd.WatchLog 2024-09-16T03:43:56.394 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd: got: 2024-09-16T03:43:35.233585+0000 mon.a [INF] from='client.14137 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi154-28301-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi154-28301-1"}]: dispatch 2024-09-16T03:43:56.394 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd: got: 2024-09-16T03:43:45.266941+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi154-28301-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi154-28301-1"}]: dispatch 2024-09-16T03:43:56.394 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd: got: 2024-09-16T03:43:45.267132+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritesmithi154-28563-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi154-28563-1"}]: dispatch 2024-09-16T03:43:56.394 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd: got: 2024-09-16T03:43:45.267298+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectssmithi154-28611-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi154-28611-1"}]: dispatch 2024-09-16T03:43:56.394 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd: got: 2024-09-16T03:43:45.267450+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi154-28216-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-16T03:43:56.394 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd: got: 2024-09-16T03:43:45.267603+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi154-28249-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-16T03:43:56.394 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd: got: 2024-09-16T03:43:45.267754+0000 mon.a [INF] from='client.4673 172.21.15.154:0/2745019661' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi154-28099-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-16T03:43:56.394 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd: got: 2024-09-16T03:43:45.267904+0000 mon.a [INF] from='client.? 172.21.15.154:0/281442000' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi154-28107-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-16T03:43:56.394 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd: got: 2024-09-16T03:43:45.268042+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi154-28286-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-16T03:43:56.394 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd: got: 2024-09-16T03:43:45.268214+0000 mon.a [INF] from='client.14242 172.21.15.154:0/3494558292' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi154-28216-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-16T03:43:56.394 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd: got: 2024-09-16T03:43:45.268368+0000 mon.a [INF] from='client.14137 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi154-28301-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi154-28301-1"}]: dispatch 2024-09-16T03:43:56.394 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd: got: 2024-09-16T03:43:50.185892+0000 mon.c [INF] from='client.? 172.21.15.154:0/2163961442' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "6.0"}]: dispatch 2024-09-16T03:43:56.395 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd: got: 2024-09-16T03:43:50.186623+0000 mon.a [INF] mon.a calling monitor election 2024-09-16T03:43:56.395 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd: got: 2024-09-16T03:43:50.187502+0000 mon.c [INF] mon.c calling monitor election 2024-09-16T03:43:56.395 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd: got: 2024-09-16T03:43:50.210340+0000 mon.c [INF] from='client.? 172.21.15.154:0/2163961442' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "6.1"}]: dispatch 2024-09-16T03:43:56.395 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd: got: 2024-09-16T03:43:50.211765+0000 mon.c [INF] from='client.? 172.21.15.154:0/2163961442' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "6.2"}]: dispatch 2024-09-16T03:43:56.395 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd: got: 2024-09-16T03:43:50.213225+0000 mon.c [INF] from='client.? 172.21.15.154:0/2163961442' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "6.3"}]: dispatch 2024-09-16T03:43:56.395 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd: got: 2024-09-16T03:43:50.214820+0000 mon.c [INF] from='client.? 172.21.15.154:0/2163961442' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "6.4"}]: dispatch 2024-09-16T03:43:56.395 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd: got: 2024-09-16T03:43:50.216222+0000 mon.c [INF] from='client.? 172.21.15.154:0/2163961442' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "6.5"}]: dispatch 2024-09-16T03:43:56.395 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd: got: 2024-09-16T03:43:50.217526+0000 mon.c [INF] from='client.? 172.21.15.154:0/2163961442' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "6.6"}]: dispatch 2024-09-16T03:43:56.395 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd: got: 2024-09-16T03:43:50.218818+0000 mon.c [INF] from='client.? 172.21.15.154:0/2163961442' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "6.7"}]: dispatch 2024-09-16T03:43:56.395 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd: got: 2024-09-16T03:43:50.220198+0000 mon.c [INF] from='client.? 172.21.15.154:0/2163961442' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "6.8"}]: dispatch 2024-09-16T03:43:56.395 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd: got: 2024-09-16T03:43:50.221667+0000 mon.c [INF] from='client.? 172.21.15.154:0/2163961442' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "6.9"}]: dispatch 2024-09-16T03:43:56.395 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd: got: 2024-09-16T03:43:55.209492+0000 mon.a [INF] mon.a is new leader, mons a,c in quorum (ranks 0,2) 2024-09-16T03:43:56.395 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd: got: 2024-09-16T03:43:55.211342+0000 mon.a [INF] from='client.? ' entity='' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritesmithi154-28563-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi154-28563-1"}]: dispatch 2024-09-16T03:43:56.395 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd: got: 2024-09-16T03:43:55.211944+0000 mon.a [INF] from='client.? ' entity='' cmd=[{"prefix": "osd pool create", "pool": "ListObjectssmithi154-28611-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi154-28611-1"}]: dispatch 2024-09-16T03:43:56.396 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd: got: 2024-09-16T03:43:55.212660+0000 mon.a [INF] from='client.? ' entity='' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi154-28249-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-16T03:43:56.396 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd: got: 2024-09-16T03:43:55.213978+0000 mon.a [INF] from='client.14419 ' entity='' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi154-28286-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-16T03:43:56.396 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd: got: 2024-09-16T03:43:55.214348+0000 mon.a [INF] from='client.14137 ' entity='' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi154-28301-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi154-28301-1"}]: dispatch 2024-09-16T03:43:56.396 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd: got: 2024-09-16T03:43:55.242141+0000 mon.a [WRN] Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-09-16T03:43:56.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.385+0000 7f42ff4c9640 10 monclient: shutdown 2024-09-16T03:43:56.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.385+0000 7f42ff4c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:43:56.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.385+0000 7f42ff4c9640 1 -- 172.21.15.42:0/3680561365 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f42f809c5d0 msgr2=0x7f42dc009f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:43:56.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.385+0000 7f42ff4c9640 1 --2- 172.21.15.42:0/3680561365 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f42f809c5d0 0x7f42dc009f60 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f42f80a2230 tx=0x7f42f401d410 comp rx=0 tx=0).stop 2024-09-16T03:43:56.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.385+0000 7f42ff4c9640 1 -- 172.21.15.42:0/3680561365 shutdown_connections 2024-09-16T03:43:56.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.385+0000 7f42ff4c9640 1 --2- 172.21.15.42:0/3680561365 >> v2:172.21.15.154:3300/0 conn(0x7f42f809fb00 0x7f42dc00a500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:43:56.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.385+0000 7f42ff4c9640 1 --2- 172.21.15.42:0/3680561365 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f42f809c5d0 0x7f42dc009f60 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:43:56.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.385+0000 7f42ff4c9640 1 -- 172.21.15.42:0/3680561365 >> 172.21.15.42:0/3680561365 conn(0x7f42f80a2420 msgr2=0x7f42f80a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:43:56.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.385+0000 7f42ff4c9640 1 -- 172.21.15.42:0/3680561365 shutdown_connections 2024-09-16T03:43:56.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.385+0000 7f42ff4c9640 1 -- 172.21.15.42:0/3680561365 wait complete. 2024-09-16T03:43:56.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.385+0000 7f42ff4c9640 10 monclient: build_initial_monmap 2024-09-16T03:43:56.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.385+0000 7f42ff4c9640 10 monclient: monmap: 2024-09-16T03:43:56.398 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:43:56.398 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:43:56.398 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:43:56.399 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:43:56.399 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:43:56.399 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:43:56.399 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:43:56.399 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:43:56.399 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:43:56.399 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:43:56.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.385+0000 7f42ff4c9640 1 librados: starting msgr at 2024-09-16T03:43:56.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.385+0000 7f42ff4c9640 1 librados: starting objecter 2024-09-16T03:43:56.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.385+0000 7f42ff4c9640 1 Processor -- start 2024-09-16T03:43:56.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.389+0000 7f42ff4c9640 1 -- start start 2024-09-16T03:43:56.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.389+0000 7f42ff4c9640 1 librados: setting wanted keys 2024-09-16T03:43:56.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.389+0000 7f42ff4c9640 1 librados: calling monclient init 2024-09-16T03:43:56.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.389+0000 7f42ff4c9640 10 monclient: init 2024-09-16T03:43:56.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.389+0000 7f42ff4c9640 10 monclient: _reopen_session rank -1 2024-09-16T03:43:56.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.389+0000 7f42ff4c9640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:43:56.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.389+0000 7f42ff4c9640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f42dc00aa90 0x7f42f813b030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:43:56.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.389+0000 7f42ff4c9640 10 monclient(hunting): picked mon.noname-b con 0x7f42dc00aa90 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:43:56.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.389+0000 7f42ff4c9640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f42f809c5d0 0x7f42f813b570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:43:56.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.389+0000 7f42ff4c9640 10 monclient(hunting): picked mon.noname-a con 0x7f42f809c5d0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:43:56.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.389+0000 7f42ff4c9640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f42f809fb00 0x7f42f813bab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:43:56.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.389+0000 7f42ff4c9640 10 monclient(hunting): picked mon.noname-c con 0x7f42f809fb00 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:43:56.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.389+0000 7f42ff4c9640 10 monclient(hunting): start opening mon connection 2024-09-16T03:43:56.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.389+0000 7f42ff4c9640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f42f809c270 con 0x7f42f809c5d0 2024-09-16T03:43:56.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.389+0000 7f42ff4c9640 10 monclient(hunting): start opening mon connection 2024-09-16T03:43:56.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.389+0000 7f42ff4c9640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f42f809d140 con 0x7f42dc00aa90 2024-09-16T03:43:56.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.389+0000 7f42ff4c9640 10 monclient(hunting): start opening mon connection 2024-09-16T03:43:56.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.389+0000 7f42ff4c9640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f42f809c3f0 con 0x7f42f809fb00 2024-09-16T03:43:56.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.389+0000 7f42ff4c9640 10 monclient(hunting): _renew_subs 2024-09-16T03:43:56.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.389+0000 7f42ff4c9640 10 monclient(hunting): authenticate will time out at 1083.361051s 2024-09-16T03:43:56.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.389+0000 7f42fca64640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f42f809c5d0 0x7f42f813b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:43:56.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.389+0000 7f42fca64640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f42f809c5d0 0x7f42f813b570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:50460/0 (socket says 172.21.15.42:50460) 2024-09-16T03:43:56.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.389+0000 7f42fca64640 1 -- 172.21.15.42:0/1177262664 learned_addr learned my addr 172.21.15.42:0/1177262664 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:43:56.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.389+0000 7f42fca64640 10 monclient(hunting): get_auth_request con 0x7f42f809c5d0 auth_method 0 2024-09-16T03:43:56.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.389+0000 7f42fca64640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:43:56.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.389+0000 7f42fca64640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:43:56.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.389+0000 7f42fca64640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:43:56.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.389+0000 7f42fd265640 1 -- 172.21.15.42:0/1177262664 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f42dc00aa90 msgr2=0x7f42f813b030 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T03:43:56.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.389+0000 7f42fd265640 1 --2- 172.21.15.42:0/1177262664 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f42dc00aa90 0x7f42f813b030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:43:56.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.389+0000 7f42fca64640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:43:56.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.389+0000 7f42fca64640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:43:56.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.389+0000 7f42fda66640 1 --2- 172.21.15.42:0/1177262664 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f42f809fb00 0x7f42f813bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:43:56.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.389+0000 7f42fca64640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:43:56.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.389+0000 7f42fda66640 10 monclient(hunting): get_auth_request con 0x7f42f809fb00 auth_method 0 2024-09-16T03:43:56.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.389+0000 7f42fda66640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:43:56.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.389+0000 7f42fda66640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:43:56.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.389+0000 7f42fda66640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:43:56.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.389+0000 7f42fda66640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:43:56.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.389+0000 7f42fda66640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:43:56.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.389+0000 7f42fda66640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:43:56.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.389+0000 7f42fca64640 10 monclient(hunting): handle_auth_done global_id 5094 payload 824 2024-09-16T03:43:56.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.389+0000 7f42fca64640 1 -- 172.21.15.42:0/1177262664 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f42f809fb00 msgr2=0x7f42f813bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:43:56.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.389+0000 7f42fca64640 1 --2- 172.21.15.42:0/1177262664 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f42f809fb00 0x7f42f813bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:43:56.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.389+0000 7f42fca64640 1 -- 172.21.15.42:0/1177262664 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f42dc00aa90 msgr2=0x7f42f813b030 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:43:56.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.389+0000 7f42fca64640 1 --2- 172.21.15.42:0/1177262664 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f42dc00aa90 0x7f42f813b030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:43:56.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.389+0000 7f42fca64640 10 monclient: _finish_hunting 0 2024-09-16T03:43:56.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.389+0000 7f42fca64640 1 monclient: found mon.noname-a 2024-09-16T03:43:56.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.389+0000 7f42fca64640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:43:56.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.389+0000 7f42fca64640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:43:56.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.389+0000 7f42fca64640 1 -- 172.21.15.42:0/1177262664 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42f813d350 con 0x7f42f809c5d0 2024-09-16T03:43:56.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.389+0000 7f42fca64640 1 --2- 172.21.15.42:0/1177262664 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f42f809c5d0 0x7f42f813b570 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f42e8005890 tx=0x7f42e800c980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:43:56.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.389+0000 7f42edffb640 10 client.?.objecter ms_handle_connect 0x7f42f809c5d0 2024-09-16T03:43:56.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.393+0000 7f42edffb640 10 client.?.objecter resend_mon_ops 2024-09-16T03:43:56.442 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.437+0000 7f42edffb640 1 -- 172.21.15.42:0/1177262664 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42e8017070 con 0x7f42f809c5d0 2024-09-16T03:43:56.442 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.437+0000 7f42edffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:43:56.442 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.437+0000 7f42edffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:43:56.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.437+0000 7f42edffb640 10 monclient: dump: 2024-09-16T03:43:56.443 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:43:56.443 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:43:56.443 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:43:56.443 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:43:56.443 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:43:56.443 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:43:56.443 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:43:56.443 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:43:56.443 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:43:56.443 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:43:56.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.437+0000 7f42edffb640 10 monclient: _finish_auth 0 2024-09-16T03:43:56.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.437+0000 7f42edffb640 10 monclient: _check_auth_tickets 2024-09-16T03:43:56.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.437+0000 7f42edffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:43:56.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.437+0000 7f42edffb640 1 -- 172.21.15.42:0/1177262664 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f42e8005cf0 con 0x7f42f809c5d0 2024-09-16T03:43:56.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.437+0000 7f42edffb640 10 monclient: handle_config config(0 keys) 2024-09-16T03:43:56.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.437+0000 7f42edffb640 1 -- 172.21.15.42:0/1177262664 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42e8007270 con 0x7f42f809c5d0 2024-09-16T03:43:56.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.437+0000 7f42ff4c9640 5 monclient: authenticate success, global_id 5094 2024-09-16T03:43:56.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.437+0000 7f42ff4c9640 10 monclient: _renew_subs 2024-09-16T03:43:56.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.437+0000 7f42ff4c9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:43:56.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.437+0000 7f42ff4c9640 1 -- 172.21.15.42:0/1177262664 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42f8072400 con 0x7f42f809c5d0 2024-09-16T03:43:56.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.437+0000 7f42edffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:43:56.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.437+0000 7f42edffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:43:56.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.437+0000 7f42edffb640 10 monclient: dump: 2024-09-16T03:43:56.444 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:43:56.444 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:43:56.444 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:43:56.445 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:43:56.445 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:43:56.445 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:43:56.445 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:43:56.445 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:43:56.445 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:43:56.445 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:43:56.445 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.437+0000 7f42ff4c9640 10 client.5094.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:43:56.445 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.437+0000 7f42ff4c9640 10 monclient: _renew_subs 2024-09-16T03:43:56.445 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.437+0000 7f42ff4c9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:43:56.445 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.437+0000 7f42ff4c9640 1 -- 172.21.15.42:0/1177262664 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42f813ca70 con 0x7f42f809c5d0 2024-09-16T03:43:56.445 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.437+0000 7f42ff4c9640 1 librados: init done 2024-09-16T03:43:56.446 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.437+0000 7f42eeffd640 4 set_mon_vals no callback set 2024-09-16T03:43:56.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.441+0000 7f42edffb640 1 -- 172.21.15.42:0/1177262664 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f42e801a070 con 0x7f42f809c5d0 2024-09-16T03:43:56.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.441+0000 7f42ff4c9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:43:56.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.441+0000 7f42edffb640 1 --2- 172.21.15.42:0/1177262664 >> v2:172.21.15.154:6812/254845342 conn(0x7f42e40378b0 0x7f42e4039d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:43:56.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.441+0000 7f42ff4c9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:43:56.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.441+0000 7f42ff4c9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:43:56.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.441+0000 7f42ff4c9640 1 -- 172.21.15.42:0/1177262664 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f42d8005090 con 0x7f42f809c5d0 2024-09-16T03:43:56.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.441+0000 7f42edffb640 1 -- 172.21.15.42:0/1177262664 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 15393+0+0 (secure 0 0 0) 0x7f42e804a8a0 con 0x7f42f809c5d0 2024-09-16T03:43:56.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.441+0000 7f42edffb640 10 client.5094.objecter ms_dispatch 0x7f42f8000ba0 osd_map(23..23 src has 1..23) 2024-09-16T03:43:56.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.441+0000 7f42edffb640 3 client.5094.objecter handle_osd_map got epochs [23,23] > 0 2024-09-16T03:43:56.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.441+0000 7f42edffb640 3 client.5094.objecter handle_osd_map decoding full epoch 23 2024-09-16T03:43:56.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.441+0000 7f42fd265640 1 --2- 172.21.15.42:0/1177262664 >> v2:172.21.15.154:6812/254845342 conn(0x7f42e40378b0 0x7f42e4039d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:43:56.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.441+0000 7f42edffb640 20 client.5094.objecter dump_active .. 0 homeless 2024-09-16T03:43:56.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.441+0000 7f42fd265640 10 monclient: get_auth_request con 0x7f42e40378b0 auth_method 0 2024-09-16T03:43:56.448 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.441+0000 7f42edffb640 10 client.5094.objecter ms_handle_connect 0x7f42e40378b0 2024-09-16T03:43:56.448 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.441+0000 7f42fd265640 1 --2- 172.21.15.42:0/1177262664 >> v2:172.21.15.154:6812/254845342 conn(0x7f42e40378b0 0x7f42e4039d70 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f42f40b1000 tx=0x7f42f4005dc0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:43:56.482 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.477+0000 7f42edffb640 1 -- 172.21.15.42:0/1177262664 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f42e801a730 con 0x7f42f809c5d0 2024-09-16T03:43:56.483 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.477+0000 7f42edffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:43:56.483 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.477+0000 7f42edffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:43:56.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.701+0000 7f42ff4c9640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:43:56.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.701+0000 7f42ff4c9640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:43:56.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.701+0000 7f42ff4c9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:43:56.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.701+0000 7f42ff4c9640 1 -- 172.21.15.42:0/1177262664 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f42d8005380 con 0x7f42f809c5d0 2024-09-16T03:43:56.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.705+0000 7f42edffb640 1 -- 172.21.15.42:0/1177262664 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1554 (secure 0 0 0) 0x7f42e801a990 con 0x7f42f809c5d0 2024-09-16T03:43:56.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.705+0000 7f42edffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:43:56.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.705+0000 7f42edffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:43:56.710 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":18,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:43:56.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.705+0000 7f42ff4c9640 10 librados: watch_flush enter 2024-09-16T03:43:56.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.705+0000 7f42ff4c9640 10 librados: watch_flush exit 2024-09-16T03:43:56.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.705+0000 7f42ff4c9640 20 client.5094.objecter shutdown clearing up homeless session... 2024-09-16T03:43:56.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.705+0000 7f42ff4c9640 10 client.5094.objecter successfully canceled tick 2024-09-16T03:43:56.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.705+0000 7f42ff4c9640 1 -- 172.21.15.42:0/1177262664 >> v2:172.21.15.154:6812/254845342 conn(0x7f42e40378b0 msgr2=0x7f42e4039d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:43:56.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.705+0000 7f42ff4c9640 1 --2- 172.21.15.42:0/1177262664 >> v2:172.21.15.154:6812/254845342 conn(0x7f42e40378b0 0x7f42e4039d70 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f42f40b1000 tx=0x7f42f4005dc0 comp rx=0 tx=0).stop 2024-09-16T03:43:56.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.705+0000 7f42ff4c9640 10 monclient: shutdown 2024-09-16T03:43:56.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.705+0000 7f42ff4c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:43:56.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.705+0000 7f42ff4c9640 1 -- 172.21.15.42:0/1177262664 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f42f809c5d0 msgr2=0x7f42f813b570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:43:56.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.705+0000 7f42ff4c9640 1 --2- 172.21.15.42:0/1177262664 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f42f809c5d0 0x7f42f813b570 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f42e8005890 tx=0x7f42e800c980 comp rx=0 tx=0).stop 2024-09-16T03:43:56.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.705+0000 7f42ff4c9640 1 -- 172.21.15.42:0/1177262664 shutdown_connections 2024-09-16T03:43:56.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.705+0000 7f42ff4c9640 1 --2- 172.21.15.42:0/1177262664 >> v2:172.21.15.154:6812/254845342 conn(0x7f42e40378b0 0x7f42e4039d70 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:43:56.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.705+0000 7f42ff4c9640 1 --2- 172.21.15.42:0/1177262664 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f42f809fb00 0x7f42f813bab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:43:56.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.705+0000 7f42ff4c9640 1 --2- 172.21.15.42:0/1177262664 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f42f809c5d0 0x7f42f813b570 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:43:56.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.705+0000 7f42ff4c9640 1 --2- 172.21.15.42:0/1177262664 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f42dc00aa90 0x7f42f813b030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:43:56.714 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.705+0000 7f42ff4c9640 1 -- 172.21.15.42:0/1177262664 >> 172.21.15.42:0/1177262664 conn(0x7f42f80a2420 msgr2=0x7f42f806f910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:43:56.714 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.705+0000 7f42ff4c9640 1 -- 172.21.15.42:0/1177262664 shutdown_connections 2024-09-16T03:43:56.714 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.705+0000 7f42ff4c9640 1 -- 172.21.15.42:0/1177262664 wait complete. 2024-09-16T03:43:56.714 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.705+0000 7f42ff4c9640 1 librados: shutdown 2024-09-16T03:43:56.729 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T03:43:56.730 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-16T03:43:56.730 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T03:43:56.861 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.853+0000 7f1644a7d640 10 monclient: get_monmap_and_config 2024-09-16T03:43:56.861 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.853+0000 7f1644a7d640 10 monclient: build_initial_monmap 2024-09-16T03:43:56.861 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.853+0000 7f1644a7d640 10 monclient: monmap: 2024-09-16T03:43:56.861 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:43:56.861 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:43:56.861 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:43:56.859932+0000 2024-09-16T03:43:56.862 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:43:56.859932+0000 2024-09-16T03:43:56.862 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:43:56.862 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:43:56.862 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:43:56.862 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:43:56.862 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:43:56.862 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:43:56.862 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:43:56.862 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:43:56.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.857+0000 7f1644a7d640 1 Processor -- start 2024-09-16T03:43:56.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.857+0000 7f1644a7d640 1 -- start start 2024-09-16T03:43:56.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.857+0000 7f1644a7d640 10 monclient: init 2024-09-16T03:43:56.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.857+0000 7f1644a7d640 10 monclient: _reopen_session rank -1 2024-09-16T03:43:56.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.857+0000 7f1644a7d640 10 monclient: _add_conns ranks=[1,3,2,0,4] 2024-09-16T03:43:56.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.857+0000 7f1644a7d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f164009fb00 addr v1:172.21.15.154:6789/0 2024-09-16T03:43:56.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.857+0000 7f1644a7d640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f164009c160 0x7f1640139120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:43:56.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.857+0000 7f1644a7d640 10 monclient(hunting): picked mon.noname-a con 0x7f164009c160 addr v2:172.21.15.42:3300/0 2024-09-16T03:43:56.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.857+0000 7f1644a7d640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f164009c5d0 0x7f164013b670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:43:56.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.857+0000 7f1644a7d640 10 monclient(hunting): picked mon.noname-b con 0x7f164009c5d0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:43:56.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.857+0000 7f1644a7d640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f16400a1e20 con 0x7f164009fb00 2024-09-16T03:43:56.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.857+0000 7f1644a7d640 10 monclient(hunting): start opening mon connection 2024-09-16T03:43:56.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.857+0000 7f1644a7d640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f164009e250 con 0x7f164009c160 2024-09-16T03:43:56.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.857+0000 7f1644a7d640 10 monclient(hunting): start opening mon connection 2024-09-16T03:43:56.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.857+0000 7f1644a7d640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f164009ea30 con 0x7f164009c5d0 2024-09-16T03:43:56.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.857+0000 7f1644a7d640 10 monclient(hunting): _renew_subs 2024-09-16T03:43:56.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.857+0000 7f1644a7d640 10 monclient(hunting): authenticate will time out at 1083.829498s 2024-09-16T03:43:56.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.857+0000 7f163ed9d640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f164009c5d0 0x7f164013b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:43:56.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.857+0000 7f163e59c640 1 -- >> v1:172.21.15.154:6789/0 conn(0x7f164009fb00 legacy=0x7f164009ba50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.154:6789/0 2024-09-16T03:43:56.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.857+0000 7f163dd9b640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f164009c160 0x7f1640139120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:43:56.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.857+0000 7f163ed9d640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f164009c5d0 0x7f164013b670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:37646/0 (socket says 172.21.15.42:37646) 2024-09-16T03:43:56.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.857+0000 7f163dd9b640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f164009c160 0x7f1640139120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:50464/0 (socket says 172.21.15.42:50464) 2024-09-16T03:43:56.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.857+0000 7f163ed9d640 1 -- 172.21.15.42:0/438603655 learned_addr learned my addr 172.21.15.42:0/438603655 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:43:56.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.857+0000 7f163ed9d640 10 monclient(hunting): get_auth_request con 0x7f164009c5d0 auth_method 0 2024-09-16T03:43:56.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.857+0000 7f163ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:43:56.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.857+0000 7f163ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:43:56.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.857+0000 7f163ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:43:56.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.857+0000 7f163dd9b640 10 monclient(hunting): get_auth_request con 0x7f164009c160 auth_method 0 2024-09-16T03:43:56.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.857+0000 7f163dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:43:56.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.857+0000 7f163dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:43:56.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.857+0000 7f163dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:43:56.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.857+0000 7f163ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:43:56.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.857+0000 7f163ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:43:56.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.857+0000 7f163ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:43:56.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.857+0000 7f163dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:43:56.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.857+0000 7f163dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:43:56.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.857+0000 7f163dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:43:56.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.857+0000 7f163ed9d640 10 monclient(hunting): handle_auth_done global_id 4778 payload 274 2024-09-16T03:43:56.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.857+0000 7f163ed9d640 1 -- 172.21.15.42:0/438603655 >> v2:172.21.15.42:3300/0 conn(0x7f164009c160 msgr2=0x7f1640139120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:43:56.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.857+0000 7f163ed9d640 1 --2- 172.21.15.42:0/438603655 >> v2:172.21.15.42:3300/0 conn(0x7f164009c160 0x7f1640139120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:43:56.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.857+0000 7f163ed9d640 1 -- 172.21.15.42:0/438603655 >> v1:172.21.15.154:6789/0 conn(0x7f164009fb00 legacy=0x7f164009ba50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:43:56.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.857+0000 7f163ed9d640 10 monclient: _finish_hunting 0 2024-09-16T03:43:56.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.857+0000 7f163ed9d640 1 monclient: found mon.noname-b 2024-09-16T03:43:56.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.857+0000 7f163ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:43:56.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.857+0000 7f163ed9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T03:43:56.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.857+0000 7f163ed9d640 1 -- 172.21.15.42:0/438603655 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f164009fee0 con 0x7f164009c5d0 2024-09-16T03:43:56.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.861+0000 7f163ed9d640 1 --2- 172.21.15.42:0/438603655 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f164009c5d0 0x7f164013b670 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f1638006fd0 tx=0x7f163801e870 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a2e1d54ca57081e1 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:43:56.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.861+0000 7f163d59a640 1 -- 172.21.15.42:0/438603655 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16380ad070 con 0x7f164009c5d0 2024-09-16T03:43:56.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.861+0000 7f163d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:43:56.871 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.861+0000 7f163d59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:43:56.871 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.861+0000 7f163d59a640 10 monclient: dump: 2024-09-16T03:43:56.871 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:43:56.871 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:43:56.871 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:43:56.871 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:43:56.871 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:43:56.871 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:43:56.871 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:43:56.872 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:43:56.872 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:43:56.872 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:43:56.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.861+0000 7f163d59a640 10 monclient: _finish_auth 0 2024-09-16T03:43:56.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.861+0000 7f163d59a640 10 monclient: _check_auth_tickets 2024-09-16T03:43:56.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.861+0000 7f163d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:43:56.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.861+0000 7f1644a7d640 5 monclient: authenticate success, global_id 4778 2024-09-16T03:43:56.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.861+0000 7f163d59a640 1 -- 172.21.15.42:0/438603655 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1638004230 con 0x7f164009c5d0 2024-09-16T03:43:56.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.861+0000 7f1644a7d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:43:56.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.861+0000 7f163d59a640 10 monclient: handle_config config(0 keys) 2024-09-16T03:43:56.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.861+0000 7f1644a7d640 10 monclient: get_monmap_and_config success 2024-09-16T03:43:56.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.861+0000 7f1644a7d640 4 set_mon_vals no callback set 2024-09-16T03:43:56.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.861+0000 7f163d59a640 1 -- 172.21.15.42:0/438603655 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16380a7680 con 0x7f164009c5d0 2024-09-16T03:43:56.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.861+0000 7f163d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:43:56.873 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.861+0000 7f163d59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:43:56.873 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.861+0000 7f163d59a640 10 monclient: dump: 2024-09-16T03:43:56.873 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:43:56.873 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:43:56.873 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:43:56.873 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:43:56.873 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:43:56.873 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:43:56.873 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:43:56.873 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:43:56.873 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:43:56.873 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:43:56.873 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.861+0000 7f1644a7d640 10 monclient: shutdown 2024-09-16T03:43:56.874 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.861+0000 7f1644a7d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:43:56.874 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.861+0000 7f1644a7d640 1 -- 172.21.15.42:0/438603655 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f164009c5d0 msgr2=0x7f164013b670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:43:56.874 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.861+0000 7f1644a7d640 1 --2- 172.21.15.42:0/438603655 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f164009c5d0 0x7f164013b670 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f1638006fd0 tx=0x7f163801e870 comp rx=0 tx=0).stop 2024-09-16T03:43:56.874 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.861+0000 7f1644a7d640 1 -- 172.21.15.42:0/438603655 shutdown_connections 2024-09-16T03:43:56.875 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.861+0000 7f1644a7d640 1 --2- 172.21.15.42:0/438603655 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f164009c5d0 0x7f164013b670 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:43:56.875 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.861+0000 7f1644a7d640 1 --2- 172.21.15.42:0/438603655 >> v2:172.21.15.42:3300/0 conn(0x7f164009c160 0x7f1640139120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:43:56.875 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.861+0000 7f1644a7d640 1 -- 172.21.15.42:0/438603655 >> 172.21.15.42:0/438603655 conn(0x7f16400a2420 msgr2=0x7f16400a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:43:56.875 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.861+0000 7f1644a7d640 1 -- 172.21.15.42:0/438603655 shutdown_connections 2024-09-16T03:43:56.875 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.861+0000 7f1644a7d640 1 -- 172.21.15.42:0/438603655 wait complete. 2024-09-16T03:43:56.875 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.861+0000 7f1644a7d640 10 monclient: build_initial_monmap 2024-09-16T03:43:56.875 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.861+0000 7f1644a7d640 10 monclient: monmap: 2024-09-16T03:43:56.875 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:43:56.875 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:43:56.875 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:43:56.875 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:43:56.876 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:43:56.876 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:43:56.876 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:43:56.876 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:43:56.876 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:43:56.876 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:43:56.876 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.861+0000 7f1644a7d640 1 librados: starting msgr at 2024-09-16T03:43:56.876 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.861+0000 7f1644a7d640 1 librados: starting objecter 2024-09-16T03:43:56.876 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.861+0000 7f1644a7d640 1 Processor -- start 2024-09-16T03:43:56.876 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.861+0000 7f1644a7d640 1 -- start start 2024-09-16T03:43:56.876 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.861+0000 7f1644a7d640 1 librados: setting wanted keys 2024-09-16T03:43:56.876 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.861+0000 7f1644a7d640 1 librados: calling monclient init 2024-09-16T03:43:56.876 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.861+0000 7f1644a7d640 10 monclient: init 2024-09-16T03:43:56.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.861+0000 7f1644a7d640 10 monclient: _reopen_session rank -1 2024-09-16T03:43:56.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.861+0000 7f1644a7d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T03:43:56.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.861+0000 7f1644a7d640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f164009c160 0x7f1640132c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:43:56.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.861+0000 7f1644a7d640 10 monclient(hunting): picked mon.noname-a con 0x7f164009c160 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:43:56.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.861+0000 7f1644a7d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f164009c5d0 0x7f1640133150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:43:56.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.861+0000 7f1644a7d640 10 monclient(hunting): picked mon.noname-b con 0x7f164009c5d0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:43:56.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.861+0000 7f1644a7d640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f164009fb00 0x7f16401394e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:43:56.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.861+0000 7f1644a7d640 10 monclient(hunting): picked mon.noname-c con 0x7f164009fb00 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:43:56.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.861+0000 7f1644a7d640 10 monclient(hunting): start opening mon connection 2024-09-16T03:43:56.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.861+0000 7f1644a7d640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f16400a32f0 con 0x7f164009c160 2024-09-16T03:43:56.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.861+0000 7f1644a7d640 10 monclient(hunting): start opening mon connection 2024-09-16T03:43:56.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.861+0000 7f1644a7d640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f16400a3170 con 0x7f164009c5d0 2024-09-16T03:43:56.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.861+0000 7f1644a7d640 10 monclient(hunting): start opening mon connection 2024-09-16T03:43:56.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.861+0000 7f1644a7d640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f164009d380 con 0x7f164009fb00 2024-09-16T03:43:56.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.861+0000 7f1644a7d640 10 monclient(hunting): _renew_subs 2024-09-16T03:43:56.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.861+0000 7f1644a7d640 10 monclient(hunting): authenticate will time out at 1083.834630s 2024-09-16T03:43:56.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.861+0000 7f163e59c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f164009c160 0x7f1640132c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:43:56.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.861+0000 7f163e59c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f164009c160 0x7f1640132c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:50466/0 (socket says 172.21.15.42:50466) 2024-09-16T03:43:56.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.861+0000 7f163e59c640 1 -- 172.21.15.42:0/588770419 learned_addr learned my addr 172.21.15.42:0/588770419 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:43:56.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.861+0000 7f163e59c640 10 monclient(hunting): get_auth_request con 0x7f164009c160 auth_method 0 2024-09-16T03:43:56.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.861+0000 7f163e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:43:56.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.861+0000 7f163e59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:43:56.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.861+0000 7f163e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:43:56.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.861+0000 7f163ed9d640 1 --2- 172.21.15.42:0/588770419 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f164009fb00 0x7f16401394e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:43:56.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.861+0000 7f163ed9d640 10 monclient(hunting): get_auth_request con 0x7f164009fb00 auth_method 0 2024-09-16T03:43:56.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.861+0000 7f163ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:43:56.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.861+0000 7f163ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:43:56.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.861+0000 7f163ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:43:56.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.861+0000 7f163dd9b640 1 -- 172.21.15.42:0/588770419 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f164009c5d0 msgr2=0x7f1640133150 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T03:43:56.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.861+0000 7f163dd9b640 1 --2- 172.21.15.42:0/588770419 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f164009c5d0 0x7f1640133150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:43:56.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.861+0000 7f163e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:43:56.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.861+0000 7f163e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:43:56.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.861+0000 7f163e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:43:56.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.861+0000 7f163ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:43:56.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.861+0000 7f163ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:43:56.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.861+0000 7f163ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:43:56.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.861+0000 7f163e59c640 10 monclient(hunting): handle_auth_done global_id 5112 payload 824 2024-09-16T03:43:56.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.861+0000 7f163e59c640 1 -- 172.21.15.42:0/588770419 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f164009fb00 msgr2=0x7f16401394e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:43:56.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.861+0000 7f163e59c640 1 --2- 172.21.15.42:0/588770419 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f164009fb00 0x7f16401394e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:43:56.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.861+0000 7f163e59c640 1 -- 172.21.15.42:0/588770419 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f164009c5d0 msgr2=0x7f1640133150 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:43:56.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.861+0000 7f163e59c640 1 --2- 172.21.15.42:0/588770419 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f164009c5d0 0x7f1640133150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:43:56.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.861+0000 7f163e59c640 10 monclient: _finish_hunting 0 2024-09-16T03:43:56.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.861+0000 7f163e59c640 1 monclient: found mon.noname-a 2024-09-16T03:43:56.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.861+0000 7f163e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:43:56.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.861+0000 7f163e59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:43:56.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.861+0000 7f163e59c640 1 -- 172.21.15.42:0/588770419 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f164006de70 con 0x7f164009c160 2024-09-16T03:43:56.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.861+0000 7f163ed9d640 1 --2- 172.21.15.42:0/588770419 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f164009fb00 0x7f16401394e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:43:56.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.861+0000 7f163e59c640 1 --2- 172.21.15.42:0/588770419 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f164009c160 0x7f1640132c10 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f163000b530 tx=0x7f163000b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:43:56.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.861+0000 7f162effd640 10 client.?.objecter ms_handle_connect 0x7f164009c160 2024-09-16T03:43:56.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.861+0000 7f162effd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:43:56.888 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.861+0000 7f162effd640 1 -- 172.21.15.42:0/588770419 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1630013020 con 0x7f164009c160 2024-09-16T03:43:56.888 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.861+0000 7f162effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:43:56.888 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.861+0000 7f162effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:43:56.888 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.861+0000 7f162effd640 10 monclient: dump: 2024-09-16T03:43:56.888 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:43:56.888 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:43:56.888 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:43:56.888 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:43:56.888 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:43:56.888 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:43:56.888 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:43:56.888 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:43:56.888 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:43:56.888 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:43:56.889 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.861+0000 7f162effd640 10 monclient: _finish_auth 0 2024-09-16T03:43:56.889 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.861+0000 7f162effd640 10 monclient: _check_auth_tickets 2024-09-16T03:43:56.889 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.861+0000 7f162effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:43:56.889 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.861+0000 7f162effd640 1 -- 172.21.15.42:0/588770419 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1630003df0 con 0x7f164009c160 2024-09-16T03:43:56.889 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.861+0000 7f1644a7d640 5 monclient: authenticate success, global_id 5112 2024-09-16T03:43:56.889 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.861+0000 7f1644a7d640 10 monclient: _renew_subs 2024-09-16T03:43:56.889 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.861+0000 7f1644a7d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:43:56.889 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.861+0000 7f1644a7d640 1 -- 172.21.15.42:0/588770419 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1640073a30 con 0x7f164009c160 2024-09-16T03:43:56.889 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.861+0000 7f162effd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:43:56.889 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.861+0000 7f162effd640 1 -- 172.21.15.42:0/588770419 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16300052b0 con 0x7f164009c160 2024-09-16T03:43:56.889 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.861+0000 7f162effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:43:56.889 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.861+0000 7f162effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:43:56.889 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.861+0000 7f162effd640 10 monclient: dump: 2024-09-16T03:43:56.889 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:43:56.890 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:43:56.890 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:43:56.890 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:43:56.890 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:43:56.890 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:43:56.890 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:43:56.890 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:43:56.890 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:43:56.890 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:43:56.890 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.861+0000 7f163cd99640 4 set_mon_vals no callback set 2024-09-16T03:43:56.890 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.861+0000 7f1644a7d640 10 client.5112.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:43:56.890 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.861+0000 7f1644a7d640 10 monclient: _renew_subs 2024-09-16T03:43:56.890 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.861+0000 7f1644a7d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:43:56.890 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.861+0000 7f1644a7d640 1 -- 172.21.15.42:0/588770419 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f164011d760 con 0x7f164009c160 2024-09-16T03:43:56.891 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.861+0000 7f1644a7d640 1 librados: init done 2024-09-16T03:43:56.891 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.865+0000 7f1644a7d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:43:56.891 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.865+0000 7f1644a7d640 1 --2- 172.21.15.42:0/588770419 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1640139a20 0x7f1640139e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:43:56.891 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.865+0000 7f1644a7d640 10 monclient(hunting): start opening mon connection 2024-09-16T03:43:56.891 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.865+0000 7f1644a7d640 1 -- 172.21.15.42:0/588770419 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f16401341f0 con 0x7f1640139a20 2024-09-16T03:43:56.891 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.865+0000 7f163dd9b640 1 --2- 172.21.15.42:0/588770419 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1640139a20 0x7f1640139e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:43:56.891 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.865+0000 7f163dd9b640 10 monclient: get_auth_request con 0x7f1640139a20 auth_method 0 2024-09-16T03:43:56.891 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.865+0000 7f163dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:43:56.891 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.865+0000 7f163dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:43:56.891 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.865+0000 7f163dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:43:56.891 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.865+0000 7f162effd640 1 -- 172.21.15.42:0/588770419 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f1630016020 con 0x7f164009c160 2024-09-16T03:43:56.891 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.865+0000 7f163dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:43:56.892 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.865+0000 7f163dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:43:56.892 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.865+0000 7f163dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:43:56.892 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.865+0000 7f162effd640 1 --2- 172.21.15.42:0/588770419 >> v2:172.21.15.154:6812/254845342 conn(0x7f16200377c0 0x7f1620039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:43:56.892 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.865+0000 7f163dd9b640 10 monclient(hunting): handle_auth_done global_id 5115 payload 824 2024-09-16T03:43:56.892 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.865+0000 7f163dd9b640 1 -- 172.21.15.42:0/588770419 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f164011d3e0 con 0x7f1640139a20 2024-09-16T03:43:56.892 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.865+0000 7f162effd640 1 -- 172.21.15.42:0/588770419 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 15393+0+0 (secure 0 0 0) 0x7f163004b130 con 0x7f164009c160 2024-09-16T03:43:56.892 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.865+0000 7f162effd640 10 client.5112.objecter ms_dispatch 0x7f1640000ba0 osd_map(23..23 src has 1..23) 2024-09-16T03:43:56.892 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.865+0000 7f162effd640 3 client.5112.objecter handle_osd_map got epochs [23,23] > 0 2024-09-16T03:43:56.892 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.865+0000 7f162effd640 3 client.5112.objecter handle_osd_map decoding full epoch 23 2024-09-16T03:43:56.892 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.865+0000 7f163dd9b640 1 --2- 172.21.15.42:0/588770419 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1640139a20 0x7f1640139e00 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f164013ac60 tx=0x7f163400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:43:56.892 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.865+0000 7f162effd640 20 client.5112.objecter dump_active .. 0 homeless 2024-09-16T03:43:56.892 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.865+0000 7f162effd640 10 client.5112.objecter ms_handle_connect 0x7f1640139a20 2024-09-16T03:43:56.893 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.865+0000 7f162effd640 10 client.5112.objecter resend_mon_ops 2024-09-16T03:43:56.893 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.865+0000 7f162effd640 1 -- 172.21.15.42:0/588770419 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1634018020 con 0x7f1640139a20 2024-09-16T03:43:56.893 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.865+0000 7f162effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:43:56.893 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.865+0000 7f162effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:43:56.893 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.865+0000 7f162effd640 10 monclient: dump: 2024-09-16T03:43:56.893 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:43:56.893 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:43:56.893 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:43:56.893 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:43:56.893 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:43:56.893 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:43:56.893 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:43:56.893 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:43:56.893 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:43:56.894 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:43:56.894 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.865+0000 7f163ed9d640 1 --2- 172.21.15.42:0/588770419 >> v2:172.21.15.154:6812/254845342 conn(0x7f16200377c0 0x7f1620039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:43:56.894 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.865+0000 7f163ed9d640 10 monclient: get_auth_request con 0x7f16200377c0 auth_method 0 2024-09-16T03:43:56.894 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.865+0000 7f162effd640 1 -- 172.21.15.42:0/588770419 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f1634014480 con 0x7f1640139a20 2024-09-16T03:43:56.894 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.865+0000 7f162effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:43:56.894 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.865+0000 7f162effd640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:43:56.894 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.865+0000 7f162effd640 1 -- 172.21.15.42:0/588770419 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1640139a20 msgr2=0x7f1640139e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:43:56.894 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.865+0000 7f162effd640 1 --2- 172.21.15.42:0/588770419 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1640139a20 0x7f1640139e00 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f164013ac60 tx=0x7f163400c490 comp rx=0 tx=0).stop 2024-09-16T03:43:56.894 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.865+0000 7f162effd640 1 -- 172.21.15.42:0/588770419 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1640139a20 msgr2=0x7f1640139e00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:43:56.894 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.865+0000 7f162effd640 1 --2- 172.21.15.42:0/588770419 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1640139a20 0x7f1640139e00 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:43:56.894 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.865+0000 7f162effd640 10 client.5112.objecter ms_handle_connect 0x7f16200377c0 2024-09-16T03:43:56.894 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.865+0000 7f163ed9d640 1 --2- 172.21.15.42:0/588770419 >> v2:172.21.15.154:6812/254845342 conn(0x7f16200377c0 0x7f1620039c80 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f16380b1000 tx=0x7f16380a9070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:43:56.894 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.869+0000 7f1644a7d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:43:56.894 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.869+0000 7f1644a7d640 1 --2- 172.21.15.42:0/588770419 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f164013a340 0x7f16401353c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:43:56.895 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.873+0000 7f1644a7d640 10 monclient(hunting): start opening mon connection 2024-09-16T03:43:56.895 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.873+0000 7f1644a7d640 1 -- 172.21.15.42:0/588770419 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f164009ba50 con 0x7f164013a340 2024-09-16T03:43:56.895 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.873+0000 7f163dd9b640 1 --2- 172.21.15.42:0/588770419 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f164013a340 0x7f16401353c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:43:56.895 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.873+0000 7f163dd9b640 10 monclient: get_auth_request con 0x7f164013a340 auth_method 0 2024-09-16T03:43:56.895 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.873+0000 7f163dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:43:56.895 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.873+0000 7f163dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:43:56.895 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.873+0000 7f163dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:43:56.895 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.873+0000 7f163dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:43:56.895 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.873+0000 7f163dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:43:56.895 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.873+0000 7f163dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:43:56.895 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.873+0000 7f163dd9b640 10 monclient(hunting): handle_auth_done global_id 5118 payload 824 2024-09-16T03:43:56.895 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.873+0000 7f163dd9b640 1 -- 172.21.15.42:0/588770419 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f1640069040 con 0x7f164013a340 2024-09-16T03:43:56.896 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.873+0000 7f162effd640 10 client.5112.objecter ms_handle_connect 0x7f164013a340 2024-09-16T03:43:56.896 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.873+0000 7f163dd9b640 1 --2- 172.21.15.42:0/588770419 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f164013a340 0x7f16401353c0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f164013ac60 tx=0x7f1634004ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:43:56.896 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.873+0000 7f162effd640 10 client.5112.objecter resend_mon_ops 2024-09-16T03:43:56.896 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.873+0000 7f162effd640 1 -- 172.21.15.42:0/588770419 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1634018070 con 0x7f164013a340 2024-09-16T03:43:56.896 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.873+0000 7f162effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:43:56.896 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.873+0000 7f162effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:43:56.896 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.873+0000 7f162effd640 10 monclient: dump: 2024-09-16T03:43:56.896 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:43:56.897 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:43:56.897 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:43:56.897 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:43:56.897 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:43:56.897 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:43:56.897 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:43:56.897 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:43:56.897 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:43:56.897 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:43:56.897 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.873+0000 7f162effd640 1 -- 172.21.15.42:0/588770419 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4704 (secure 0 0 0) 0x7f163401d640 con 0x7f164013a340 2024-09-16T03:43:56.897 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.873+0000 7f162effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:43:56.897 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.873+0000 7f162effd640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:43:56.897 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.873+0000 7f162effd640 1 -- 172.21.15.42:0/588770419 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f164013a340 msgr2=0x7f16401353c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:43:56.897 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.873+0000 7f162effd640 1 --2- 172.21.15.42:0/588770419 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f164013a340 0x7f16401353c0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f164013ac60 tx=0x7f1634004ac0 comp rx=0 tx=0).stop 2024-09-16T03:43:56.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.873+0000 7f162effd640 1 -- 172.21.15.42:0/588770419 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f164013a340 msgr2=0x7f16401353c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:43:56.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.873+0000 7f162effd640 1 --2- 172.21.15.42:0/588770419 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f164013a340 0x7f16401353c0 secure :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f164013ac60 tx=0x7f1634004ac0 comp rx=0 tx=0).stop 2024-09-16T03:43:56.898 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:43:56.898 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:43:56.898 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:43:56.898 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "leader", 2024-09-16T03:43:56.898 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 18, 2024-09-16T03:43:56.898 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 116243, 2024-09-16T03:43:56.898 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:43:56.898 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:43:56.899 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T03:43:56.899 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:43:56.899 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 1, 2024-09-16T03:43:56.899 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:43:56.899 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:43:56.899 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:43:56.899 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:43:56.899 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:43:56.899 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:43:56.899 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:43:56.899 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:43:56.899 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:43:56.899 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:43:56.899 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:43:56.900 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:43:56.900 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:43:56.900 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:43:56.900 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:43:56.900 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:43:56.900 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:43:56.900 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:43:56.900 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:43:56.900 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:43:56.900 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:43:56.900 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:43:56.900 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:43:56.901 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:43:56.901 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:43:56.901 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:43:56.901 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:43:56.901 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:43:56.901 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:43:56.901 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:43:56.901 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:43:56.901 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:43:56.901 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:43:56.901 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:43:56.901 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:43:56.901 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:43:56.901 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:43:56.902 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:43:56.902 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:43:56.902 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:43:56.902 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:43:56.902 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:43:56.902 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:43:56.902 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:43:56.902 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:43:56.902 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:43:56.902 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:43:56.902 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:43:56.902 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:43:56.902 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:43:56.902 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:43:56.903 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:43:56.903 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:43:56.903 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:43:56.903 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:43:56.903 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:43:56.903 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:43:56.903 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:43:56.903 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:43:56.903 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:43:56.903 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:43:56.903 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:43:56.903 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:43:56.903 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:43:56.903 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:43:56.904 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:43:56.904 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:43:56.904 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:43:56.904 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:43:56.904 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:43:56.904 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:43:56.904 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:43:56.904 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:43:56.904 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:43:56.904 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:43:56.904 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:43:56.904 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:43:56.904 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:43:56.905 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:43:56.905 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:43:56.905 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:43:56.905 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:43:56.905 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:43:56.905 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:43:56.905 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:43:56.905 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:43:56.905 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:43:56.905 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:43:56.905 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:43:56.905 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:43:56.905 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:43:56.905 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:43:56.906 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:43:56.906 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:43:56.906 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:43:56.906 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:43:56.906 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:43:56.906 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:43:56.906 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:43:56.906 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:43:56.906 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:43:56.906 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:43:56.906 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:43:56.906 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:43:56.906 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:43:56.906 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:43:56.907 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:43:56.907 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:43:56.907 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:43:56.907 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:43:56.907 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:43:56.907 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:43:56.907 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:43:56.907 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:43:56.907 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:43:56.907 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:43:56.907 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:43:56.907 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:43:56.908 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:43:56.908 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:43:56.908 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:43:56.908 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:43:56.908 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:43:56.908 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:43:56.908 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:43:56.908 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:43:56.908 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:43:56.908 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:43:56.908 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:43:56.908 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:43:56.908 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:43:56.909 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:43:56.909 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:43:56.909 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:43:56.909 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:43:56.909 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:43:56.909 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:43:56.909 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:43:56.909 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:43:56.909 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:43:56.909 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:43:56.909 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T03:43:56.909 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:43:56.910 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:43:56.910 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:43:56.910 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 4 2024-09-16T03:43:56.910 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:43:56.910 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:43:56.910 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:43:56.910 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:43:56.910 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:43:56.910 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:43:56.910 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 42 2024-09-16T03:43:56.910 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:43:56.911 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:43:56.911 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:43:56.911 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:43:56.911 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:43:56.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.873+0000 7f1644a7d640 10 librados: watch_flush enter 2024-09-16T03:43:56.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.873+0000 7f1644a7d640 10 librados: watch_flush exit 2024-09-16T03:43:56.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.873+0000 7f1644a7d640 20 client.5112.objecter shutdown clearing up homeless session... 2024-09-16T03:43:56.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.873+0000 7f1644a7d640 10 client.5112.objecter successfully canceled tick 2024-09-16T03:43:56.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.873+0000 7f1644a7d640 1 -- 172.21.15.42:0/588770419 >> v2:172.21.15.154:6812/254845342 conn(0x7f16200377c0 msgr2=0x7f1620039c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:43:56.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.873+0000 7f1644a7d640 1 --2- 172.21.15.42:0/588770419 >> v2:172.21.15.154:6812/254845342 conn(0x7f16200377c0 0x7f1620039c80 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f16380b1000 tx=0x7f16380a9070 comp rx=0 tx=0).stop 2024-09-16T03:43:56.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.873+0000 7f1644a7d640 10 monclient: shutdown 2024-09-16T03:43:56.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.873+0000 7f1644a7d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:43:56.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.873+0000 7f1644a7d640 1 -- 172.21.15.42:0/588770419 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f164009c160 msgr2=0x7f1640132c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:43:56.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.873+0000 7f1644a7d640 1 --2- 172.21.15.42:0/588770419 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f164009c160 0x7f1640132c10 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f163000b530 tx=0x7f163000b9f0 comp rx=0 tx=0).stop 2024-09-16T03:43:56.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.873+0000 7f163ed9d640 1 -- 172.21.15.42:0/588770419 reap_dead start 2024-09-16T03:43:56.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.873+0000 7f163ed9d640 1 -- 172.21.15.42:0/588770419 reap_dead start 2024-09-16T03:43:56.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.873+0000 7f1644a7d640 1 -- 172.21.15.42:0/588770419 shutdown_connections 2024-09-16T03:43:56.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.873+0000 7f1644a7d640 1 -- 172.21.15.42:0/588770419 >> 172.21.15.42:0/588770419 conn(0x7f16400a2420 msgr2=0x7f1640070fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:43:56.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.873+0000 7f1644a7d640 1 -- 172.21.15.42:0/588770419 shutdown_connections 2024-09-16T03:43:56.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.873+0000 7f1644a7d640 1 -- 172.21.15.42:0/588770419 wait complete. 2024-09-16T03:43:56.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:56.873+0000 7f1644a7d640 1 librados: shutdown 2024-09-16T03:43:56.915 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T03:43:56.915 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-16T03:43:56.915 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T03:43:57.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.025+0000 7f1447dc6640 10 monclient: get_monmap_and_config 2024-09-16T03:43:57.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.025+0000 7f1447dc6640 10 monclient: build_initial_monmap 2024-09-16T03:43:57.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.025+0000 7f1447dc6640 10 monclient: monmap: 2024-09-16T03:43:57.032 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:43:57.032 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:43:57.032 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:43:57.030895+0000 2024-09-16T03:43:57.032 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:43:57.030895+0000 2024-09-16T03:43:57.032 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:43:57.032 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:43:57.033 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:43:57.033 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:43:57.033 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:43:57.033 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:43:57.033 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:43:57.033 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:43:57.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.025+0000 7f1447dc6640 1 Processor -- start 2024-09-16T03:43:57.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.025+0000 7f1447dc6640 1 -- start start 2024-09-16T03:43:57.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.025+0000 7f1447dc6640 10 monclient: init 2024-09-16T03:43:57.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.029+0000 7f1447dc6640 10 monclient: _reopen_session rank -1 2024-09-16T03:43:57.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.029+0000 7f1447dc6640 10 monclient: _add_conns ranks=[0,3,4,1,2] 2024-09-16T03:43:57.036 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.029+0000 7f1447dc6640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f14400a39b0 addr v1:172.21.15.42:6789/0 2024-09-16T03:43:57.036 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.029+0000 7f1447dc6640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f14400a3d90 0x7f144013c380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:43:57.036 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.029+0000 7f1447dc6640 10 monclient(hunting): picked mon.noname-a con 0x7f14400a3d90 addr v2:172.21.15.42:3300/0 2024-09-16T03:43:57.036 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.029+0000 7f1447dc6640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f144013c8c0 0x7f144013ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:43:57.036 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.029+0000 7f1447dc6640 10 monclient(hunting): picked mon.noname-c con 0x7f144013c8c0 addr v2:172.21.15.154:3300/0 2024-09-16T03:43:57.036 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.029+0000 7f1447dc6640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1440134290 con 0x7f14400a39b0 2024-09-16T03:43:57.036 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.029+0000 7f1447dc6640 10 monclient(hunting): start opening mon connection 2024-09-16T03:43:57.036 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.029+0000 7f1447dc6640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f144009c140 con 0x7f14400a3d90 2024-09-16T03:43:57.036 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.029+0000 7f1447dc6640 10 monclient(hunting): start opening mon connection 2024-09-16T03:43:57.036 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.029+0000 7f1447dc6640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f144009c420 con 0x7f144013c8c0 2024-09-16T03:43:57.036 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.029+0000 7f1447dc6640 10 monclient(hunting): _renew_subs 2024-09-16T03:43:57.036 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.029+0000 7f1447dc6640 10 monclient(hunting): authenticate will time out at 1084.000303s 2024-09-16T03:43:57.036 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.029+0000 7f1445b62640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f14400a39b0 0x7f1440133b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:54268/0 (socket says 172.21.15.42:54268) 2024-09-16T03:43:57.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.029+0000 7f1445b62640 1 -- 172.21.15.42:0/3794049036 learned_addr learned my addr 172.21.15.42:0/3794049036 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:43:57.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.029+0000 7f1446363640 1 -- 172.21.15.42:0/3794049036 >> v2:172.21.15.154:3300/0 conn(0x7f144013c8c0 msgr2=0x7f144013ecb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T03:43:57.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.029+0000 7f1446363640 1 --2- 172.21.15.42:0/3794049036 >> v2:172.21.15.154:3300/0 conn(0x7f144013c8c0 0x7f144013ecb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:43:57.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.029+0000 7f1445361640 1 --2- 172.21.15.42:0/3794049036 >> v2:172.21.15.42:3300/0 conn(0x7f14400a3d90 0x7f144013c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:43:57.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.029+0000 7f1445361640 10 monclient(hunting): get_auth_request con 0x7f14400a3d90 auth_method 0 2024-09-16T03:43:57.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.029+0000 7f1445361640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:43:57.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.029+0000 7f1445361640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:43:57.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.029+0000 7f1445361640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:43:57.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.029+0000 7f1445361640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:43:57.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.029+0000 7f1445361640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:43:57.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.029+0000 7f1445361640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:43:57.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.029+0000 7f1445361640 10 monclient(hunting): handle_auth_done global_id 5121 payload 274 2024-09-16T03:43:57.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.029+0000 7f1445361640 1 -- 172.21.15.42:0/3794049036 >> v2:172.21.15.154:3300/0 conn(0x7f144013c8c0 msgr2=0x7f144013ecb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:43:57.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.029+0000 7f1445361640 1 --2- 172.21.15.42:0/3794049036 >> v2:172.21.15.154:3300/0 conn(0x7f144013c8c0 0x7f144013ecb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:43:57.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.029+0000 7f1445361640 1 -- 172.21.15.42:0/3794049036 >> v1:172.21.15.42:6789/0 conn(0x7f14400a39b0 legacy=0x7f1440133b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:43:57.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.029+0000 7f1445361640 10 monclient: _finish_hunting 0 2024-09-16T03:43:57.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.029+0000 7f1445361640 1 monclient: found mon.noname-a 2024-09-16T03:43:57.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.029+0000 7f1445361640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:43:57.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.029+0000 7f1445361640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:43:57.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.029+0000 7f1444b60640 1 -- 172.21.15.42:0/3794049036 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4123496853 0 0) 0x7f1440134290 con 0x7f14400a39b0 2024-09-16T03:43:57.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.029+0000 7f1445361640 1 -- 172.21.15.42:0/3794049036 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f144006dd60 con 0x7f14400a3d90 2024-09-16T03:43:57.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.029+0000 7f1444b60640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T03:43:57.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.029+0000 7f1445361640 1 --2- 172.21.15.42:0/3794049036 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f14400a3d90 0x7f144013c380 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f144006b540 tx=0x7f143c01c360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83d9b21d0314a62c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:43:57.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.029+0000 7f1444b60640 1 -- 172.21.15.42:0/3794049036 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f143c0b0070 con 0x7f14400a3d90 2024-09-16T03:43:57.045 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.029+0000 7f1444b60640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:43:57.045 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.029+0000 7f1444b60640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:43:57.045 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.029+0000 7f1444b60640 10 monclient: dump: 2024-09-16T03:43:57.045 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:43:57.045 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:43:57.045 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:43:57.045 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:43:57.045 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:43:57.045 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:43:57.045 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:43:57.045 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:43:57.045 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:43:57.045 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:43:57.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.029+0000 7f1444b60640 10 monclient: _finish_auth 0 2024-09-16T03:43:57.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.029+0000 7f1444b60640 10 monclient: _check_auth_tickets 2024-09-16T03:43:57.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.029+0000 7f1444b60640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:43:57.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.029+0000 7f1444b60640 1 -- 172.21.15.42:0/3794049036 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f143c0ab600 con 0x7f14400a3d90 2024-09-16T03:43:57.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.029+0000 7f1444b60640 10 monclient: handle_config config(0 keys) 2024-09-16T03:43:57.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.029+0000 7f1444b60640 1 -- 172.21.15.42:0/3794049036 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f143c0ab9d0 con 0x7f14400a3d90 2024-09-16T03:43:57.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.029+0000 7f1447dc6640 5 monclient: authenticate success, global_id 5121 2024-09-16T03:43:57.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.029+0000 7f1447dc6640 10 monclient: get_monmap_and_config success 2024-09-16T03:43:57.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.029+0000 7f1447dc6640 4 set_mon_vals no callback set 2024-09-16T03:43:57.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.029+0000 7f1444b60640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:43:57.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.029+0000 7f1444b60640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:43:57.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.029+0000 7f1444b60640 10 monclient: dump: 2024-09-16T03:43:57.047 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:43:57.047 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:43:57.047 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:43:57.047 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:43:57.047 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:43:57.047 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:43:57.047 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:43:57.047 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:43:57.047 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:43:57.047 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:43:57.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.029+0000 7f1447dc6640 10 monclient: shutdown 2024-09-16T03:43:57.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.029+0000 7f1447dc6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:43:57.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.029+0000 7f1447dc6640 1 -- 172.21.15.42:0/3794049036 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f14400a3d90 msgr2=0x7f144013c380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:43:57.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.029+0000 7f1447dc6640 1 --2- 172.21.15.42:0/3794049036 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f14400a3d90 0x7f144013c380 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f144006b540 tx=0x7f143c01c360 comp rx=0 tx=0).stop 2024-09-16T03:43:57.048 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.029+0000 7f1447dc6640 1 -- 172.21.15.42:0/3794049036 shutdown_connections 2024-09-16T03:43:57.048 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.029+0000 7f1447dc6640 1 --2- 172.21.15.42:0/3794049036 >> v2:172.21.15.154:3300/0 conn(0x7f144013c8c0 0x7f144013ecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:43:57.048 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.029+0000 7f1447dc6640 1 --2- 172.21.15.42:0/3794049036 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f14400a3d90 0x7f144013c380 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:43:57.048 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.029+0000 7f1447dc6640 1 -- 172.21.15.42:0/3794049036 >> 172.21.15.42:0/3794049036 conn(0x7f14400a9fb0 msgr2=0x7f14400aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:43:57.048 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.029+0000 7f1447dc6640 1 -- 172.21.15.42:0/3794049036 shutdown_connections 2024-09-16T03:43:57.048 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.033+0000 7f1447dc6640 1 -- 172.21.15.42:0/3794049036 wait complete. 2024-09-16T03:43:57.048 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.033+0000 7f1447dc6640 10 monclient: build_initial_monmap 2024-09-16T03:43:57.048 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.033+0000 7f1447dc6640 10 monclient: monmap: 2024-09-16T03:43:57.048 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:43:57.048 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:43:57.048 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:43:57.048 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:43:57.049 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:43:57.049 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:43:57.049 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:43:57.049 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:43:57.049 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:43:57.049 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:43:57.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.033+0000 7f1447dc6640 1 librados: starting msgr at 2024-09-16T03:43:57.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.033+0000 7f1447dc6640 1 librados: starting objecter 2024-09-16T03:43:57.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.033+0000 7f1447dc6640 1 Processor -- start 2024-09-16T03:43:57.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.033+0000 7f1447dc6640 1 -- start start 2024-09-16T03:43:57.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.033+0000 7f1447dc6640 1 librados: setting wanted keys 2024-09-16T03:43:57.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.033+0000 7f1447dc6640 1 librados: calling monclient init 2024-09-16T03:43:57.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.033+0000 7f1447dc6640 10 monclient: init 2024-09-16T03:43:57.050 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.033+0000 7f1447dc6640 10 monclient: _reopen_session rank -1 2024-09-16T03:43:57.050 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.033+0000 7f1447dc6640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:43:57.050 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.033+0000 7f1447dc6640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f14400a39b0 0x7f1440138130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:43:57.050 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.033+0000 7f1447dc6640 10 monclient(hunting): picked mon.noname-c con 0x7f14400a39b0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:43:57.050 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.033+0000 7f1447dc6640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f14400a3d90 0x7f1440138670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:43:57.050 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.033+0000 7f1447dc6640 10 monclient(hunting): picked mon.noname-a con 0x7f14400a3d90 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:43:57.050 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.033+0000 7f1447dc6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f144013c8c0 0x7f1440138bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:43:57.050 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.033+0000 7f1447dc6640 10 monclient(hunting): picked mon.noname-b con 0x7f144013c8c0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:43:57.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.033+0000 7f1447dc6640 10 monclient(hunting): start opening mon connection 2024-09-16T03:43:57.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.033+0000 7f1447dc6640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f1440141da0 con 0x7f14400a3d90 2024-09-16T03:43:57.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.033+0000 7f1447dc6640 10 monclient(hunting): start opening mon connection 2024-09-16T03:43:57.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.033+0000 7f1447dc6640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f1440141c20 con 0x7f144013c8c0 2024-09-16T03:43:57.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.033+0000 7f1447dc6640 10 monclient(hunting): start opening mon connection 2024-09-16T03:43:57.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.033+0000 7f1447dc6640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f1440141f20 con 0x7f14400a39b0 2024-09-16T03:43:57.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.033+0000 7f1447dc6640 10 monclient(hunting): _renew_subs 2024-09-16T03:43:57.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.033+0000 7f1447dc6640 10 monclient(hunting): authenticate will time out at 1084.005582s 2024-09-16T03:43:57.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.033+0000 7f1445361640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f14400a3d90 0x7f1440138670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:43:57.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.033+0000 7f1445b62640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f14400a39b0 0x7f1440138130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:43:57.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.033+0000 7f1445361640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f14400a3d90 0x7f1440138670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:50518/0 (socket says 172.21.15.42:50518) 2024-09-16T03:43:57.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.033+0000 7f1445361640 1 -- 172.21.15.42:0/2974167823 learned_addr learned my addr 172.21.15.42:0/2974167823 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:43:57.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.033+0000 7f1445361640 10 monclient(hunting): get_auth_request con 0x7f14400a3d90 auth_method 0 2024-09-16T03:43:57.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.033+0000 7f1445361640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:43:57.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.033+0000 7f1445361640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:43:57.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.033+0000 7f1445361640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:43:57.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.033+0000 7f1445b62640 10 monclient(hunting): get_auth_request con 0x7f14400a39b0 auth_method 0 2024-09-16T03:43:57.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.033+0000 7f1445b62640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:43:57.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.033+0000 7f1445b62640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:43:57.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.033+0000 7f1445b62640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:43:57.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.033+0000 7f1446363640 1 -- 172.21.15.42:0/2974167823 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f144013c8c0 msgr2=0x7f1440138bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T03:43:57.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.033+0000 7f1446363640 1 --2- 172.21.15.42:0/2974167823 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f144013c8c0 0x7f1440138bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:43:57.053 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.033+0000 7f1445361640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:43:57.053 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.033+0000 7f1445361640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:43:57.053 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.033+0000 7f1445361640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:43:57.053 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.033+0000 7f1445b62640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:43:57.053 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.033+0000 7f1445b62640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:43:57.053 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.033+0000 7f1445b62640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:43:57.053 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.033+0000 7f1445b62640 10 monclient(hunting): handle_auth_done global_id 4784 payload 824 2024-09-16T03:43:57.053 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.033+0000 7f1445b62640 1 -- 172.21.15.42:0/2974167823 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f144013c8c0 msgr2=0x7f1440138bb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:43:57.053 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.033+0000 7f1445b62640 1 --2- 172.21.15.42:0/2974167823 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f144013c8c0 0x7f1440138bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:43:57.053 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.033+0000 7f1445b62640 1 -- 172.21.15.42:0/2974167823 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f14400a3d90 msgr2=0x7f1440138670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:43:57.053 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.033+0000 7f1445b62640 1 --2- 172.21.15.42:0/2974167823 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f14400a3d90 0x7f1440138670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:43:57.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.033+0000 7f1445b62640 10 monclient: _finish_hunting 0 2024-09-16T03:43:57.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.033+0000 7f1445b62640 1 monclient: found mon.noname-c 2024-09-16T03:43:57.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.033+0000 7f1445b62640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:43:57.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.033+0000 7f1445b62640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:43:57.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.033+0000 7f1445b62640 1 -- 172.21.15.42:0/2974167823 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1440139260 con 0x7f14400a39b0 2024-09-16T03:43:57.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.033+0000 7f1445361640 1 --2- 172.21.15.42:0/2974167823 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f14400a3d90 0x7f1440138670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:43:57.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.033+0000 7f1445b62640 1 --2- 172.21.15.42:0/2974167823 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f14400a39b0 0x7f1440138130 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f143400c430 tx=0x7f143400c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:43:57.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.033+0000 7f14327fc640 10 client.?.objecter ms_handle_connect 0x7f14400a39b0 2024-09-16T03:43:57.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.033+0000 7f14327fc640 10 client.?.objecter resend_mon_ops 2024-09-16T03:43:57.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.033+0000 7f14327fc640 1 -- 172.21.15.42:0/2974167823 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14340074e0 con 0x7f14400a39b0 2024-09-16T03:43:57.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.033+0000 7f14327fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:43:57.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.033+0000 7f14327fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:43:57.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.033+0000 7f14327fc640 10 monclient: dump: 2024-09-16T03:43:57.055 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:43:57.055 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:43:57.055 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:43:57.055 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:43:57.055 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:43:57.055 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:43:57.055 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:43:57.055 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:43:57.055 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:43:57.055 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:43:57.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.033+0000 7f14327fc640 10 monclient: _finish_auth 0 2024-09-16T03:43:57.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.033+0000 7f14327fc640 10 monclient: _check_auth_tickets 2024-09-16T03:43:57.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.033+0000 7f14327fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:43:57.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.033+0000 7f14327fc640 1 -- 172.21.15.42:0/2974167823 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f14340047f0 con 0x7f14400a39b0 2024-09-16T03:43:57.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.033+0000 7f14327fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:43:57.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.033+0000 7f14327fc640 1 -- 172.21.15.42:0/2974167823 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1434004be0 con 0x7f14400a39b0 2024-09-16T03:43:57.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.033+0000 7f1447dc6640 5 monclient: authenticate success, global_id 4784 2024-09-16T03:43:57.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.033+0000 7f1447dc6640 10 monclient: _renew_subs 2024-09-16T03:43:57.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.033+0000 7f1447dc6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:43:57.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.033+0000 7f1447dc6640 1 -- 172.21.15.42:0/2974167823 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f144013c320 con 0x7f14400a39b0 2024-09-16T03:43:57.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.033+0000 7f1433fff640 4 set_mon_vals no callback set 2024-09-16T03:43:57.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.033+0000 7f14327fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:43:57.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.033+0000 7f14327fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:43:57.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.033+0000 7f14327fc640 10 monclient: dump: 2024-09-16T03:43:57.056 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:43:57.056 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:43:57.057 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:43:57.057 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:43:57.057 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:43:57.057 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:43:57.057 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:43:57.057 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:43:57.057 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:43:57.057 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:43:57.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.033+0000 7f1447dc6640 10 client.4784.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:43:57.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.033+0000 7f1447dc6640 10 monclient: _renew_subs 2024-09-16T03:43:57.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.033+0000 7f1447dc6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:43:57.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.033+0000 7f1447dc6640 1 -- 172.21.15.42:0/2974167823 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f144013a8b0 con 0x7f14400a39b0 2024-09-16T03:43:57.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.033+0000 7f1447dc6640 1 librados: init done 2024-09-16T03:43:57.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.033+0000 7f1447dc6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:43:57.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.033+0000 7f1447dc6640 1 --2- 172.21.15.42:0/2974167823 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f144012d2d0 0x7f144012d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:43:57.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.033+0000 7f1447dc6640 10 monclient(hunting): start opening mon connection 2024-09-16T03:43:57.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.033+0000 7f1447dc6640 1 -- 172.21.15.42:0/2974167823 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f1440133b80 con 0x7f144012d2d0 2024-09-16T03:43:57.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.033+0000 7f1445361640 1 --2- 172.21.15.42:0/2974167823 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f144012d2d0 0x7f144012d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:43:57.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.033+0000 7f1445361640 10 monclient: get_auth_request con 0x7f144012d2d0 auth_method 0 2024-09-16T03:43:57.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.033+0000 7f1445361640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:43:57.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.033+0000 7f1445361640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:43:57.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.033+0000 7f1445361640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:43:57.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.033+0000 7f14327fc640 1 -- 172.21.15.42:0/2974167823 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f1434011070 con 0x7f14400a39b0 2024-09-16T03:43:57.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.033+0000 7f1445361640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:43:57.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.033+0000 7f1445361640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:43:57.059 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.033+0000 7f1445361640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:43:57.059 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.033+0000 7f1445361640 10 monclient(hunting): handle_auth_done global_id 4787 payload 824 2024-09-16T03:43:57.059 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.033+0000 7f14327fc640 1 --2- 172.21.15.42:0/2974167823 >> v2:172.21.15.154:6812/254845342 conn(0x7f14240377c0 0x7f1424039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:43:57.059 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.033+0000 7f1445361640 1 -- 172.21.15.42:0/2974167823 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f14400a5750 con 0x7f144012d2d0 2024-09-16T03:43:57.059 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.033+0000 7f14327fc640 1 -- 172.21.15.42:0/2974167823 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(23..23 src has 1..23) ==== 15393+0+0 (secure 0 0 0) 0x7f143404b1a0 con 0x7f14400a39b0 2024-09-16T03:43:57.059 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.033+0000 7f14327fc640 10 client.4784.objecter ms_dispatch 0x7f1440000ba0 osd_map(23..23 src has 1..23) 2024-09-16T03:43:57.059 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.033+0000 7f14327fc640 3 client.4784.objecter handle_osd_map got epochs [23,23] > 0 2024-09-16T03:43:57.059 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.033+0000 7f14327fc640 3 client.4784.objecter handle_osd_map decoding full epoch 23 2024-09-16T03:43:57.059 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.037+0000 7f1445361640 1 --2- 172.21.15.42:0/2974167823 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f144012d2d0 0x7f144012d6b0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f143c019f90 tx=0x7f143c019100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:43:57.059 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.037+0000 7f14327fc640 20 client.4784.objecter dump_active .. 0 homeless 2024-09-16T03:43:57.059 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.037+0000 7f14327fc640 10 client.4784.objecter ms_handle_connect 0x7f144012d2d0 2024-09-16T03:43:57.059 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.037+0000 7f14327fc640 10 client.4784.objecter resend_mon_ops 2024-09-16T03:43:57.059 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.037+0000 7f14327fc640 1 -- 172.21.15.42:0/2974167823 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f143c0bc070 con 0x7f144012d2d0 2024-09-16T03:43:57.060 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.037+0000 7f1446363640 1 --2- 172.21.15.42:0/2974167823 >> v2:172.21.15.154:6812/254845342 conn(0x7f14240377c0 0x7f1424039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:43:57.060 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.037+0000 7f14327fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:43:57.060 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.037+0000 7f14327fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:43:57.060 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.037+0000 7f14327fc640 10 monclient: dump: 2024-09-16T03:43:57.060 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:43:57.060 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:43:57.060 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:43:57.060 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:43:57.060 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:43:57.060 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:43:57.060 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:43:57.060 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:43:57.060 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:43:57.060 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:43:57.060 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.037+0000 7f1446363640 10 monclient: get_auth_request con 0x7f14240377c0 auth_method 0 2024-09-16T03:43:57.061 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.037+0000 7f14327fc640 1 -- 172.21.15.42:0/2974167823 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f143c0b1040 con 0x7f144012d2d0 2024-09-16T03:43:57.061 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.037+0000 7f14327fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:43:57.061 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.037+0000 7f14327fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:43:57.061 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.037+0000 7f14327fc640 1 -- 172.21.15.42:0/2974167823 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f144012d2d0 msgr2=0x7f144012d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:43:57.061 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.037+0000 7f14327fc640 1 --2- 172.21.15.42:0/2974167823 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f144012d2d0 0x7f144012d6b0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f143c019f90 tx=0x7f143c019100 comp rx=0 tx=0).stop 2024-09-16T03:43:57.061 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.037+0000 7f14327fc640 1 -- 172.21.15.42:0/2974167823 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f144012d2d0 msgr2=0x7f144012d6b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:43:57.061 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.037+0000 7f14327fc640 1 --2- 172.21.15.42:0/2974167823 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f144012d2d0 0x7f144012d6b0 secure :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f143c019f90 tx=0x7f143c019100 comp rx=0 tx=0).stop 2024-09-16T03:43:57.061 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.037+0000 7f14327fc640 10 client.4784.objecter ms_handle_connect 0x7f14240377c0 2024-09-16T03:43:57.061 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.037+0000 7f1446363640 1 --2- 172.21.15.42:0/2974167823 >> v2:172.21.15.154:6812/254845342 conn(0x7f14240377c0 0x7f1424039c80 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f1438003cc0 tx=0x7f1438009270 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:43:57.063 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.041+0000 7f1447dc6640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:43:57.063 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.041+0000 7f1447dc6640 1 --2- 172.21.15.42:0/2974167823 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f144012dbf0 0x7f1440135bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:43:57.063 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.041+0000 7f1447dc6640 10 monclient(hunting): start opening mon connection 2024-09-16T03:43:57.063 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.041+0000 7f1447dc6640 1 -- 172.21.15.42:0/2974167823 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f1440139670 con 0x7f144012dbf0 2024-09-16T03:43:57.063 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.041+0000 7f1445361640 1 --2- 172.21.15.42:0/2974167823 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f144012dbf0 0x7f1440135bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:43:57.064 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.041+0000 7f1445361640 10 monclient: get_auth_request con 0x7f144012dbf0 auth_method 0 2024-09-16T03:43:57.064 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.041+0000 7f1445361640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:43:57.064 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.041+0000 7f1445361640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:43:57.064 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.041+0000 7f1445361640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:43:57.064 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.041+0000 7f1445361640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:43:57.064 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.041+0000 7f1445361640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:43:57.064 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.041+0000 7f1445361640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:43:57.064 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.041+0000 7f1445361640 10 monclient(hunting): handle_auth_done global_id 4790 payload 824 2024-09-16T03:43:57.064 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.041+0000 7f1445361640 1 -- 172.21.15.42:0/2974167823 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f144012e000 con 0x7f144012dbf0 2024-09-16T03:43:57.064 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.041+0000 7f1445361640 1 --2- 172.21.15.42:0/2974167823 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f144012dbf0 0x7f1440135bc0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f143c000c10 tx=0x7f143c008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:43:57.064 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.041+0000 7f14327fc640 10 client.4784.objecter ms_handle_connect 0x7f144012dbf0 2024-09-16T03:43:57.064 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.041+0000 7f14327fc640 10 client.4784.objecter resend_mon_ops 2024-09-16T03:43:57.064 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.041+0000 7f14327fc640 1 -- 172.21.15.42:0/2974167823 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f143c0b0040 con 0x7f144012dbf0 2024-09-16T03:43:57.065 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.041+0000 7f14327fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:43:57.065 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.041+0000 7f14327fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:43:57.065 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.041+0000 7f14327fc640 10 monclient: dump: 2024-09-16T03:43:57.065 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:43:57.065 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:43:57.065 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:43:57.065 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:43:57.065 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:43:57.065 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:43:57.065 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:43:57.065 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:43:57.065 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:43:57.065 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:43:57.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.041+0000 7f14327fc640 1 -- 172.21.15.42:0/2974167823 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4869 (secure 0 0 0) 0x7f143c0aa3a0 con 0x7f144012dbf0 2024-09-16T03:43:57.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.041+0000 7f14327fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:43:57.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.041+0000 7f14327fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:43:57.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.041+0000 7f14327fc640 1 -- 172.21.15.42:0/2974167823 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f144012dbf0 msgr2=0x7f1440135bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:43:57.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.041+0000 7f14327fc640 1 --2- 172.21.15.42:0/2974167823 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f144012dbf0 0x7f1440135bc0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f143c000c10 tx=0x7f143c008040 comp rx=0 tx=0).stop 2024-09-16T03:43:57.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.041+0000 7f14327fc640 1 -- 172.21.15.42:0/2974167823 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f144012dbf0 msgr2=0x7f1440135bc0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:43:57.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.041+0000 7f14327fc640 1 --2- 172.21.15.42:0/2974167823 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f144012dbf0 0x7f1440135bc0 secure :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f143c000c10 tx=0x7f143c008040 comp rx=0 tx=0).stop 2024-09-16T03:43:57.066 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:43:57.066 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:43:57.067 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:43:57.067 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T03:43:57.067 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 18, 2024-09-16T03:43:57.067 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 116411, 2024-09-16T03:43:57.067 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:43:57.067 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:43:57.067 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T03:43:57.067 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:43:57.067 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 1, 2024-09-16T03:43:57.067 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:43:57.067 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:43:57.067 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:43:57.067 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:43:57.067 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:43:57.067 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:43:57.068 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:43:57.068 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:43:57.068 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:43:57.068 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:43:57.068 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:43:57.068 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:43:57.068 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:43:57.068 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:43:57.068 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:43:57.068 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:43:57.068 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:43:57.068 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:43:57.068 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:43:57.069 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:43:57.069 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:43:57.069 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:43:57.069 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:43:57.069 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:43:57.069 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:43:57.069 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:43:57.069 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:43:57.069 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:43:57.069 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:43:57.069 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:43:57.069 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:43:57.069 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:43:57.070 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:43:57.070 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:43:57.070 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:43:57.070 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:43:57.070 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:43:57.070 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:43:57.070 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:43:57.070 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:43:57.070 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:43:57.070 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:43:57.070 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:43:57.070 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:43:57.070 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:43:57.070 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:43:57.071 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:43:57.071 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:43:57.071 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:43:57.071 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:43:57.071 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:43:57.071 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:43:57.071 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:43:57.071 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:43:57.071 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:43:57.071 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:43:57.071 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:43:57.071 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:43:57.071 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:43:57.071 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:43:57.072 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:43:57.072 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:43:57.072 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:43:57.072 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:43:57.072 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:43:57.072 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:43:57.072 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:43:57.072 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:43:57.072 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:43:57.072 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:43:57.072 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:43:57.072 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:43:57.072 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:43:57.073 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:43:57.073 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:43:57.073 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:43:57.073 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:43:57.073 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:43:57.073 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:43:57.073 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:43:57.073 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:43:57.073 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:43:57.073 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:43:57.073 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:43:57.073 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:43:57.073 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:43:57.074 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:43:57.074 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:43:57.074 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:43:57.074 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:43:57.074 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:43:57.074 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:43:57.074 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:43:57.074 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:43:57.074 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:43:57.074 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:43:57.074 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:43:57.074 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:43:57.074 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:43:57.074 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:43:57.075 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:43:57.075 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:43:57.075 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:43:57.075 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:43:57.075 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:43:57.075 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:43:57.075 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:43:57.075 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:43:57.075 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:43:57.075 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:43:57.075 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:43:57.075 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:43:57.075 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:43:57.075 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:43:57.076 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:43:57.076 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:43:57.076 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:43:57.076 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:43:57.076 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:43:57.076 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:43:57.076 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:43:57.076 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:43:57.077 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:43:57.077 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:43:57.077 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:43:57.077 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:43:57.077 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:43:57.077 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:43:57.077 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:43:57.077 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:43:57.077 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:43:57.077 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:43:57.077 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:43:57.077 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:43:57.077 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:43:57.077 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:43:57.078 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:43:57.078 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:43:57.078 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:43:57.078 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:43:57.078 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:43:57.078 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T03:43:57.078 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:43:57.078 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:43:57.078 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:43:57.078 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 2 2024-09-16T03:43:57.078 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:43:57.078 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:43:57.079 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:43:57.079 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:43:57.079 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:43:57.079 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:43:57.079 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 22 2024-09-16T03:43:57.079 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:43:57.079 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:43:57.079 INFO:teuthology.orchestra.run.smithi042.stdout: "mgr": [ 2024-09-16T03:43:57.079 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:43:57.079 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:43:57.079 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:43:57.079 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:43:57.079 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:43:57.079 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:43:57.080 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:43:57.080 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:43:57.080 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:43:57.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.041+0000 7f1447dc6640 10 librados: watch_flush enter 2024-09-16T03:43:57.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.041+0000 7f1447dc6640 10 librados: watch_flush exit 2024-09-16T03:43:57.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.041+0000 7f1447dc6640 20 client.4784.objecter shutdown clearing up homeless session... 2024-09-16T03:43:57.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.041+0000 7f1447dc6640 10 client.4784.objecter successfully canceled tick 2024-09-16T03:43:57.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.045+0000 7f1447dc6640 1 -- 172.21.15.42:0/2974167823 >> v2:172.21.15.154:6812/254845342 conn(0x7f14240377c0 msgr2=0x7f1424039c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:43:57.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.045+0000 7f1447dc6640 1 --2- 172.21.15.42:0/2974167823 >> v2:172.21.15.154:6812/254845342 conn(0x7f14240377c0 0x7f1424039c80 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f1438003cc0 tx=0x7f1438009270 comp rx=0 tx=0).stop 2024-09-16T03:43:57.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.045+0000 7f1447dc6640 10 monclient: shutdown 2024-09-16T03:43:57.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.045+0000 7f1447dc6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:43:57.083 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.045+0000 7f1447dc6640 1 -- 172.21.15.42:0/2974167823 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f14400a39b0 msgr2=0x7f1440138130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:43:57.083 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.045+0000 7f1447dc6640 1 --2- 172.21.15.42:0/2974167823 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f14400a39b0 0x7f1440138130 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f143400c430 tx=0x7f143400c8f0 comp rx=0 tx=0).stop 2024-09-16T03:43:57.083 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.045+0000 7f1446363640 1 -- 172.21.15.42:0/2974167823 reap_dead start 2024-09-16T03:43:57.083 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.045+0000 7f1447dc6640 1 -- 172.21.15.42:0/2974167823 shutdown_connections 2024-09-16T03:43:57.083 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.045+0000 7f1447dc6640 1 --2- 172.21.15.42:0/2974167823 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f14400a39b0 0x7f1440138130 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:43:57.083 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.045+0000 7f1447dc6640 1 -- 172.21.15.42:0/2974167823 >> 172.21.15.42:0/2974167823 conn(0x7f14400a9fb0 msgr2=0x7f1440131980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:43:57.083 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.045+0000 7f1447dc6640 1 -- 172.21.15.42:0/2974167823 shutdown_connections 2024-09-16T03:43:57.083 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.045+0000 7f1447dc6640 1 -- 172.21.15.42:0/2974167823 wait complete. 2024-09-16T03:43:57.083 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:43:57.045+0000 7f1447dc6640 1 librados: shutdown 2024-09-16T03:43:57.083 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T03:43:57.084 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-16T03:43:57.476 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd: got: 2024-09-16T03:43:55.823875+0000 mon.a [WRN] overall HEALTH_WARN 1/3 mons down, quorum a,c; 15 pool(s) do not have an application enabled 2024-09-16T03:43:57.476 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd: got: 2024-09-16T03:43:56.239452+0000 mon.a [INF] from='client.4673 172.21.15.154:0/2745019661' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi154-28099-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T03:43:57.476 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd: got: 2024-09-16T03:43:56.239691+0000 mon.a [INF] from='client.? 172.21.15.154:0/281442000' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi154-28107-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T03:43:57.476 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd: got: 2024-09-16T03:43:56.239841+0000 mon.a [INF] from='client.14242 172.21.15.154:0/3494558292' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi154-28216-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T03:43:57.476 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd: got: 2024-09-16T03:43:56.240009+0000 mon.a [INF] from='client.? ' entity='' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritesmithi154-28563-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi154-28563-1"}]': finished 2024-09-16T03:43:57.476 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd: got: 2024-09-16T03:43:56.240192+0000 mon.a [INF] from='client.? ' entity='' cmd='[{"prefix": "osd pool create", "pool": "ListObjectssmithi154-28611-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi154-28611-1"}]': finished 2024-09-16T03:43:57.476 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd: got: 2024-09-16T03:43:56.240393+0000 mon.a [INF] from='client.? ' entity='' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi154-28249-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T03:43:57.477 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd: got: 2024-09-16T03:43:56.240642+0000 mon.a [INF] from='client.14419 ' entity='' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi154-28286-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T03:43:57.477 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd: got: 2024-09-16T03:43:56.240856+0000 mon.a [INF] from='client.14137 ' entity='' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi154-28301-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi154-28301-1"}]': finished 2024-09-16T03:43:57.477 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd: got: 2024-09-16T03:43:56.252269+0000 mon.a [INF] from='client.4673 172.21.15.154:0/2745019661' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi154-28099-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi154-28099-16"}]: dispatch 2024-09-16T03:43:57.477 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd: got: 2024-09-16T03:43:56.255251+0000 mon.a [INF] from='client.? 172.21.15.154:0/281442000' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi154-28107-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi154-28107-22"}]: dispatch 2024-09-16T03:43:57.477 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd: got: 2024-09-16T03:43:56.259179+0000 mon.c [INF] from='client.? 172.21.15.154:0/499491003' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi154-28249-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi154-28249-7"}]: dispatch 2024-09-16T03:43:57.477 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd: got: 2024-09-16T03:43:56.260597+0000 mon.a [INF] from='client.? 172.21.15.154:0/3360117001' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi154-28091-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-16T03:43:57.477 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd: got: 2024-09-16T03:43:56.260786+0000 mon.c [INF] from='client.14419 172.21.15.154:0/2947288884' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi154-28286-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi154-28286-7"}]: dispatch 2024-09-16T03:43:57.477 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd: got: 2024-09-16T03:43:56.262279+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi154-28249-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi154-28249-7"}]: dispatch 2024-09-16T03:43:57.477 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd: got: 2024-09-16T03:43:56.264364+0000 mon.a [INF] from='client.14419 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi154-28286-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi154-28286-7"}]: dispatch 2024-09-16T03:43:57.478 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd: got: 2024-09-16T03:43:56.273397+0000 mon.c [INF] from='client.14413 172.21.15.154:0/1034107445' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi154-28224-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-16T03:43:57.478 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd: got: 2024-09-16T03:43:56.317531+0000 mon.a [INF] from='client.? 172.21.15.154:0/2819479515' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi154-28083-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-16T03:43:57.478 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd: got: 2024-09-16T03:43:56.317774+0000 mon.a [INF] from='client.? 172.21.15.154:0/90038503' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi154-28381-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-16T03:43:57.478 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd: got: 2024-09-16T03:43:56.318024+0000 mon.a [INF] from='client.14413 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi154-28224-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-16T03:43:57.478 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd: got: 2024-09-16T03:43:56.352326+0000 mon.c [INF] from='client.? 172.21.15.154:0/3244559533' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi154-28160-10"}]: dispatch 2024-09-16T03:43:57.478 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd: got: 2024-09-16T03:43:56.384461+0000 mon.c [INF] from='client.? 172.21.15.154:0/1956241571' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-09-16T03:43:57.478 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd: got: 2024-09-16T03:43:56.387703+0000 client.admin [INF] onexx 2024-09-16T03:43:57.478 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd: got: 2024-09-16T03:43:56.441376+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi154-28160-10"}]: dispatch 2024-09-16T03:43:57.478 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd: got: 2024-09-16T03:43:56.441785+0000 mon.a [INF] from='client.? 172.21.15.154:0/2547152935' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi154-28277-12"}]: dispatch 2024-09-16T03:43:57.478 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd: got: 2024-09-16T03:43:56.441981+0000 mon.a [INF] from='client.? 172.21.15.154:0/212121185' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi154-28128-10"}]: dispatch 2024-09-16T03:43:57.478 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd: got: 2024-09-16T03:43:56.443287+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-09-16T03:43:57.478 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd: got: 2024-09-16T03:43:56.443430+0000 mon.c [INF] from='client.? 172.21.15.154:0/3244559533' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi154-28160-10"}]: dispatch 2024-09-16T03:43:57.478 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd: got: 2024-09-16T03:43:56.444623+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi154-28160-10"}]: dispatch 2024-09-16T03:43:57.479 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd: got: 2024-09-16T03:43:56.444809+0000 mon.a [INF] from='client.? 172.21.15.154:0/212121185' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi154-28128-10"}]: dispatch 2024-09-16T03:43:58.461 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd: got: 2024-09-16T03:43:56.445215+0000 mon. delete_pools_parallel: process_1_[28468]: starting. 2024-09-16T03:43:58.461 INFO:tasks.workunit.client.0.smithi154.stdout: delete_pools_parallel: process_1_[28468]: creating pool ceph_test_rados_delete_pools_parallel.smithi154-28449 2024-09-16T03:43:58.461 INFO:tasks.workunit.client.0.smithi154.stdout: delete_pools_parallel: process_1_[28468]: created object 0... 2024-09-16T03:43:58.461 INFO:tasks.workunit.client.0.smithi154.stdout: delete_pools_parallel: process_1_[28468]: created object 25... 2024-09-16T03:43:58.462 INFO:tasks.workunit.client.0.smithi154.stdout: delete_pools_parallel: process_1_[28468]: created object 49... 2024-09-16T03:43:58.462 INFO:tasks.workunit.client.0.smithi154.stdout: delete_pools_parallel: process_1_[28468]: finishing. 2024-09-16T03:43:58.462 INFO:tasks.workunit.client.0.smithi154.stdout: delete_pools_parallel: process_1_[28468]: shutting down. 2024-09-16T03:43:58.462 INFO:tasks.workunit.client.0.smithi154.stdout: delete_pools_parallel: process_2_[28469]: starting. 2024-09-16T03:43:58.462 INFO:tasks.workunit.client.0.smithi154.stdout: delete_pools_parallel: process_2_[28469]: deleting pool ceph_test_rados_delete_pools_parallel.smithi154-28449 2024-09-16T03:43:58.462 INFO:tasks.workunit.client.0.smithi154.stdout: delete_pools_parallel: process_2_[28469]: shutting down. 2024-09-16T03:43:58.462 INFO:tasks.workunit.client.0.smithi154.stdout: delete_pools_parallel: ******************************* 2024-09-16T03:43:58.462 INFO:tasks.workunit.client.0.smithi154.stdout: delete_pools_parallel: process_3_[29410]: starting. 2024-09-16T03:43:58.462 INFO:tasks.workunit.client.0.smithi154.stdout: delete_pools_parallel: process_3_[29410]: creating pool ceph_test_rados_delete_pools_parallel.smithi154-28449 2024-09-16T03:43:58.462 INFO:tasks.workunit.client.0.smithi154.stdout: delete_pools_parallel: process_3_[29410]: created object 0... 2024-09-16T03:43:58.462 INFO:tasks.workunit.client.0.smithi154.stdout: delete_pools_parallel: process_3_[29410]: created object 25... 2024-09-16T03:43:58.462 INFO:tasks.workunit.client.0.smithi154.stdout: delete_pools_parallel: process_3_[29410]: created object 49... 2024-09-16T03:43:58.462 INFO:tasks.workunit.client.0.smithi154.stdout: delete_pools_parallel: process_3_[29410]: finishing. 2024-09-16T03:43:58.462 INFO:tasks.workunit.client.0.smithi154.stdout: delete_pools_parallel: process_3_[29410]: shutting down. 2024-09-16T03:43:58.463 INFO:tasks.workunit.client.0.smithi154.stdout: delete_pools_parallel: ******************************* 2024-09-16T03:43:58.463 INFO:tasks.workunit.client.0.smithi154.stdout: delete_pools_parallel: process_5_[29412]: starting. 2024-09-16T03:43:58.463 INFO:tasks.workunit.client.0.smithi154.stdout: delete_pools_parallel: process_5_[29412]: listing objects. 2024-09-16T03:43:58.463 INFO:tasks.workunit.client.0.smithi154.stdout: delete_pools_parallel: process_5_[29412]: listed object 0... 2024-09-16T03:43:58.463 INFO:tasks.workunit.client.0.smithi154.stdout: delete_pools_parallel: process_5_[29412]: listed object 25... 2024-09-16T03:43:58.463 INFO:tasks.workunit.client.0.smithi154.stdout: delete_pools_parallel: process_5_[29412]: saw 50 objects 2024-09-16T03:43:58.463 INFO:tasks.workunit.client.0.smithi154.stdout: delete_pools_parallel: process_5_[29412]: shutting down. 2024-09-16T03:43:58.463 INFO:tasks.workunit.client.0.smithi154.stdout: delete_pools_parallel: ******************************* 2024-09-16T03:43:58.463 INFO:tasks.workunit.client.0.smithi154.stdout: delete_pools_parallel: process_4_[29411]: starting. 2024-09-16T03:43:58.463 INFO:tasks.workunit.client.0.smithi154.stdout: delete_pools_parallel: process_4_[29411]: deleting pool ceph_test_rados_delete_pools_parallel.smithi154-28449 2024-09-16T03:43:58.463 INFO:tasks.workunit.client.0.smithi154.stdout: delete_pools_parallel: process_4_[29411]: shutting down. 2024-09-16T03:43:58.463 INFO:tasks.workunit.client.0.smithi154.stdout: delete_pools_parallel: ******************************* 2024-09-16T03:43:58.463 INFO:tasks.workunit.client.0.smithi154.stdout: delete_pools_parallel: ******************************* 2024-09-16T03:43:58.463 INFO:tasks.workunit.client.0.smithi154.stdout: delete_pools_parallel: ******* SUCCESS ********** 2024-09-16T03:43:58.464 INFO:tasks.workunit.client.0.smithi154.stdout: open_pools_parallel: process_1_[28487]: starting. 2024-09-16T03:43:58.464 INFO:tasks.workunit.client.0.smithi154.stdout: open_pools_parallel: process_1_[28487]: creating pool ceph_test_rados_open_pools_parallel.smithi154-28458 2024-09-16T03:43:58.464 INFO:tasks.workunit.client.0.smithi154.stdout: open_pools_parallel: process_1_[28487]: created object 0... 2024-09-16T03:43:58.464 INFO:tasks.workunit.client.0.smithi154.stdout: open_pools_parallel: process_1_[28487]: created object 25... 2024-09-16T03:43:58.464 INFO:tasks.workunit.client.0.smithi154.stdout: open_pools_parallel: process_1_[28487]: created object 49... 2024-09-16T03:43:58.464 INFO:tasks.workunit.client.0.smithi154.stdout: open_pools_parallel: process_1_[28487]: finishing. 2024-09-16T03:43:58.464 INFO:tasks.workunit.client.0.smithi154.stdout: open_pools_parallel: process_1_[28487]: shutting down. 2024-09-16T03:43:58.464 INFO:tasks.workunit.client.0.smithi154.stdout: open_pools_parallel: process_2_[28489]: starting. 2024-09-16T03:43:58.464 INFO:tasks.workunit.client.0.smithi154.stdout: open_pools_parallel: process_2_[28489]: rados_pool_create. 2024-09-16T03:43:58.465 INFO:tasks.workunit.client.0.smithi154.stdout: open_pools_parallel: process_2_[28489]: rados_ioctx_create. 2024-09-16T03:43:58.465 INFO:tasks.workunit.client.0.smithi154.stdout: open_pools_parallel: process_2_[28489]: shutting down. 2024-09-16T03:43:58.465 INFO:tasks.workunit.client.0.smithi154.stdout: open_pools_parallel: ******************************* 2024-09-16T03:43:58.465 INFO:tasks.workunit.client.0.smithi154.stdout: open_pools_parallel: process_3_[29416]: starting. 2024-09-16T03:43:58.465 INFO:tasks.workunit.client.0.smithi154.stdout: open_pools_parallel: process_3_[29416]: creating pool ceph_test_rados_open_pools_parallel.smithi154-28458 2024-09-16T03:43:58.465 INFO:tasks.workunit.client.0.smithi154.stdout: open_pools_parallel: process_3_[29416]: created object 0... 2024-09-16T03:43:58.465 INFO:tasks.workunit.client.0.smithi154.stdout: open_pools_parallel: process_3_[29416]: created object 25... 2024-09-16T03:43:58.465 INFO:tasks.workunit.client.0.smithi154.stdout: open_pools_parallel: process_3_[29416]: created object 49... 2024-09-16T03:43:58.465 INFO:tasks.workunit.client.0.smithi154.stdout: open_pools_parallel: process_3_[29416]: finishing. 2024-09-16T03:43:58.465 INFO:tasks.workunit.client.0.smithi154.stdout: open_pools_parallel: process_3_[29416]: shutting down. 2024-09-16T03:43:58.465 INFO:tasks.workunit.client.0.smithi154.stdout: open_pools_parallel: ******************************* 2024-09-16T03:43:58.465 INFO:tasks.workunit.client.0.smithi154.stdout: open_pools_parallel: process_4_[29417]: starting. 2024-09-16T03:43:58.465 INFO:tasks.workunit.client.0.smithi154.stdout: open_pools_parallel: process_4_[29417]: rados_pool_create. 2024-09-16T03:43:58.466 INFO:tasks.workunit.client.0.smithi154.stdout: open_pools_parallel: process_4_[29417]: rados_ioctx_create. 2024-09-16T03:43:58.466 INFO:tasks.workunit.client.0.smithi154.stdout: open_pools_parallel: process_4_[29417]: shutting down. 2024-09-16T03:43:58.466 INFO:tasks.workunit.client.0.smithi154.stdout: open_pools_parallel: ******************************* 2024-09-16T03:43:58.466 INFO:tasks.workunit.client.0.smithi154.stdout: open_pools_parallel: ******************************* 2024-09-16T03:43:58.466 INFO:tasks.workunit.client.0.smithi154.stdout: open_pools_parallel: ******* SUCCESS ********** 2024-09-16T03:43:59.297 INFO:tasks.workunit.client.0.smithi154.stdout: cmd: Running main() from gmock_main.cc 2024-09-16T03:43:59.298 INFO:tasks.workunit.client.0.smithi154.stdout: cmd: [==========] Running 3 tests from 1 test suite. 2024-09-16T03:43:59.298 INFO:tasks.workunit.client.0.smithi154.stdout: cmd: [----------] Global test environment set-up. 2024-09-16T03:43:59.298 INFO:tasks.workunit.client.0.smithi154.stdout: cmd: [----------] 3 tests from NeoRadosCmd 2024-09-16T03:43:59.298 INFO:tasks.workunit.client.0.smithi154.stdout: cmd: [ RUN ] NeoRadosCmd.MonDescribe 2024-09-16T03:43:59.298 INFO:tasks.workunit.client.0.smithi154.stdout: cmd: [ OK ] NeoRadosCmd.MonDescribe (3540 ms) 2024-09-16T03:43:59.298 INFO:tasks.workunit.client.0.smithi154.stdout: cmd: [ RUN ] NeoRadosCmd.OSDCmd 2024-09-16T03:43:59.298 INFO:tasks.workunit.client.0.smithi154.stdout: cmd: [ OK ] NeoRadosCmd.OSDCmd (22970 ms) 2024-09-16T03:43:59.298 INFO:tasks.workunit.client.0.smithi154.stdout: cmd: [ RUN ] NeoRadosCmd.PGCmd 2024-09-16T03:43:59.298 INFO:tasks.workunit.client.0.smithi154.stdout: cmd: [ OK ] NeoRadosCmd.PGCmd (3021 ms) 2024-09-16T03:43:59.298 INFO:tasks.workunit.client.0.smithi154.stdout: cmd: [----------] 3 tests from NeoRadosCmd (29531 ms total) 2024-09-16T03:43:59.299 INFO:tasks.workunit.client.0.smithi154.stdout: cmd: 2024-09-16T03:43:59.299 INFO:tasks.workunit.client.0.smithi154.stdout: cmd: [----------] Global test environment tear-down 2024-09-16T03:43:59.299 INFO:tasks.workunit.client.0.smithi154.stdout: cmd: [==========] 3 tests from 1 test suite ran. (29531 ms total) 2024-09-16T03:43:59.299 INFO:tasks.workunit.client.0.smithi154.stdout: cmd: [ PASSED ] 3 tests. 2024-09-16T03:43:59.460 INFO:tasks.workunit.client.0.smithi154.stdout: api_io_pp: Running main() from gmock_main.cc 2024-09-16T03:43:59.460 INFO:tasks.workunit.client.0.smithi154.stdout: api_io_pp: [==========] Running 37 tests from 2 test suites. 2024-09-16T03:43:59.460 INFO:tasks.workunit.client.0.smithi154.stdout: api_io_pp: [----------] Global test environment set-up. 2024-09-16T03:43:59.460 INFO:tasks.workunit.client.0.smithi154.stdout: api_io_pp: [----------] 20 tests from LibRadosIoPP 2024-09-16T03:43:59.460 INFO:tasks.workunit.client.0.smithi154.stdout: api_io_pp: seed 28107 2024-09-16T03:43:59.460 INFO:tasks.workunit.client.0.smithi154.stdout: api_io_pp: [ RUN ] LibRadosIoPP.TooBigPP 2024-09-16T03:43:59.461 INFO:tasks.workunit.client.0.smithi154.stdout: api_io_pp: [ OK ] LibRadosIoPP.TooBigPP (0 ms) 2024-09-16T03:43:59.461 INFO:tasks.workunit.client.0.smithi154.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SimpleWritePP 2024-09-16T03:43:59.461 INFO:tasks.workunit.client.0.smithi154.stdout: api_io_pp: [ OK ] LibRadosIoPP.SimpleWritePP (299 ms) 2024-09-16T03:43:59.461 INFO:tasks.workunit.client.0.smithi154.stdout: api_io_pp: [ RUN ] LibRadosIoPP.ReadOpPP 2024-09-16T03:43:59.461 INFO:tasks.workunit.client.0.smithi154.stdout: api_io_pp: [ OK ] LibRadosIoPP.ReadOpPP (14 ms) 2024-09-16T03:43:59.461 INFO:tasks.workunit.client.0.smithi154.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SparseReadOpPP 2024-09-16T03:43:59.461 INFO:tasks.workunit.client.0.smithi154.stdout: api_io_pp: [ OK ] LibRadosIoPP.SparseReadOpPP (17 ms) 2024-09-16T03:43:59.461 INFO:tasks.workunit.client.0.smithi154.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RoundTripPP 2024-09-16T03:43:59.461 INFO:tasks.workunit.client.0.smithi154.stdout: api_io_pp: [ OK ] LibRadosIoPP.RoundTripPP (4 ms) 2024-09-16T03:43:59.461 INFO:tasks.workunit.client.0.smithi154.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RoundTripPP2 2024-09-16T03:43:59.461 INFO:tasks.workunit.client.0.smithi154.stdout: api_io_pp: [ OK ] LibRadosIoPP.RoundTripPP2 (3 ms) 2024-09-16T03:43:59.461 INFO:tasks.workunit.client.0.smithi154.stdout: api_io_pp: [ RUN ] LibRadosIoPP.Checksum 2024-09-16T03:43:59.461 INFO:tasks.workunit.client.0.smithi154.stdout: api_io_pp: [ OK ] LibRadosIoPP.Checksum (3 ms) 2024-09-16T03:43:59.462 INFO:tasks.workunit.client.0.smithi154.stdout: api_io_pp: [ RUN ] LibRadosIoPP.ReadIntoBufferlist 2024-09-16T03:43:59.462 INFO:tasks.workunit.client.0.smithi154.stdout: api_io_pp: [ OK ] LibRadosIoPP.ReadIntoBufferlist (3 ms) 2024-09-16T03:43:59.462 INFO:tasks.workunit.client.0.smithi154.stdout: api_io_pp: [ RUN ] LibRadosIoPP.OverlappingWriteRoundTripPP 2024-09-16T03:43:59.462 INFO:tasks.workunit.client.0.smithi154.stdout: api_io_pp: [ OK ] LibRadosIoPP.OverlappingWriteRoundTripPP (5 ms) 2024-09-16T03:43:59.462 INFO:tasks.workunit.client.0.smithi154.stdout: api_io_pp: [ RUN ] LibRadosIoPP.WriteFullRoundTripPP 2024-09-16T03:43:59.462 INFO:tasks.workunit.client.0.smithi154.stdout: api_io_pp: [ OK ] LibRadosIoPP.WriteFullRoundTripPP (5 ms) 2024-09-16T03:43:59.462 INFO:tasks.workunit.client.0.smithi154.stdout: api_io_pp: [ RUN ] LibRadosIoPP.WriteFullRoundTripPP2 2024-09-16T03:43:59.462 INFO:tasks.workunit.client.0.smithi154.stdout: api_io_pp: [ OK ] LibRadosIoPP.WriteFullRoundTripPP2 (3 ms) 2024-09-16T03:43:59.462 INFO:tasks.workunit.client.0.smithi154.stdout: api_io_pp: [ RUN ] LibRadosIoPP.AppendRoundTripPP 2024-09-16T03:43:59.462 INFO:tasks.workunit.client.0.smithi154.stdout: api_io_pp: [ OK ] LibRadosIoPP.AppendRoundTripPP (7 ms) 2024-09-16T03:43:59.462 INFO:tasks.workunit.client.0.smithi154.stdout: api_io_pp: [ RUN ] LibRadosIoPP.TruncTestPP 2024-09-16T03:43:59.462 INFO:tasks.workunit.client.0.smithi154.stdout: api_io_pp: [ OK ] LibRadosIoPP.TruncTestPP (4 ms) 2024-09-16T03:43:59.462 INFO:tasks.workunit.client.0.smithi154.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RemoveTestPP 2024-09-16T03:43:59.462 INFO:tasks.workunit.client.0.smithi154.stdout: api_io_pp: [ OK ] LibRadosIoPP.RemoveTestPP (6 ms) 2024-09-16T03:43:59.463 INFO:tasks.workunit.client.0.smithi154.stdout: api_io_pp: [ RUN ] LibRadosIoPP.XattrsRoundTripPP 2024-09-16T03:43:59.463 INFO:tasks.workunit.client.0.smithi154.stdout: api_io_pp: [ OK ] LibRadosIoPP.XattrsRoundTripPP (6 ms) 2024-09-16T03:43:59.463 INFO:tasks.workunit.client.0.smithi154.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RmXattrPP 2024-09-16T03:43:59.463 INFO:tasks.workunit.client.0.smithi154.stdout: api_io_pp: [ OK ] LibRadosIoPP.RmXattrPP (24 ms) 2024-09-16T03:43:59.463 INFO:tasks.workunit.client.0.smithi154.stdout: api_io_pp: [ RUN ] LibRadosIoPP.XattrListPP 2024-09-16T03:43:59.463 INFO:tasks.workunit.client.0.smithi154.stdout: api_io_pp: [ OK ] LibRadosIoPP.XattrListPP (10 ms) 2024-09-16T03:43:59.463 INFO:tasks.workunit.client.0.smithi154.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtPP 2024-09-16T03:43:59.463 INFO:tasks.workunit.client.0.smithi154.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtPP (6 ms) 2024-09-16T03:43:59.463 INFO:tasks.workunit.client.0.smithi154.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtDNEPP 2024-09-16T03:43:59.463 INFO:tasks.workunit.client.0.smithi154.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtDNEPP (4 ms) 2024-09-16T03:43:59.463 INFO:tasks.workunit.client.0.smithi154.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtMismatchPP 2024-09-16T03:43:59.463 INFO:tasks.workunit.client.0.smithi154.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtMismatchPP (9 ms) 2024-09-16T03:43:59.463 INFO:tasks.workunit.client.0.smithi154.stdout: api_io_pp: [----------] 20 tests from LibRadosIoPP (432 ms total) 2024-09-16T03:43:59.463 INFO:tasks.workunit.client.0.smithi154.stdout: api_io_pp: 2024-09-16T03:43:59.464 INFO:tasks.workunit.client.0.smithi154.stdout: api_io_pp: [----------] 17 tests from LibRadosIoECPP 2024-09-16T03:43:59.464 INFO:tasks.workunit.client.0.smithi154.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.SimpleWritePP 2024-09-16T03:43:59.464 INFO:tasks.workunit.client.0.smithi154.stdout: api_io_pp: [ OK ] LibRadosIoECPP.SimpleWritePP (973 ms) 2024-09-16T03:43:59.464 INFO:tasks.workunit.client.0.smithi154.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.ReadOpPP 2024-09-16T03:43:59.464 INFO:tasks.workunit.client.0.smithi154.stdout: api_io_pp: [ OK ] LibRadosIoECPP.ReadOpPP (21 ms) 2024-09-16T03:43:59.464 INFO:tasks.workunit.client.0.smithi154.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.SparseReadOpPP 2024-09-16T03:43:59.464 INFO:tasks.workunit.client.0.smithi154.stdout: api_io_pp: [ OK ] LibRadosIoECPP.SparseReadOpPP (4 ms) 2024-09-16T03:43:59.464 INFO:tasks.workunit.client.0.smithi154.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RoundTripPP 2024-09-16T03:43:59.464 INFO:tasks.workunit.client.0.smithi154.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RoundTripPP (4 ms) 2024-09-16T03:43:59.464 INFO:tasks.workunit.client.0.smithi154.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RoundTripPP2 2024-09-16T03:43:59.464 INFO:tasks.workunit.client.0.smithi154.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RoundTripPP2 (3 ms) 2024-09-16T03:44:00.482 INFO:tasks.workunit.client.0.smithi154.stdout: api_io_pp: [ RUN c [INF] from='client.? 172.21.15.154:0/3244559533' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi154-28160-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-16T03:44:00.482 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd: got: 2024-09-16T03:43:56.476409+0000 mon.a [INF] from='client.? 172.21.15.154:0/2547152935' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi154-28277-12"}]: dispatch 2024-09-16T03:44:00.482 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd: got: 2024-09-16T03:43:56.476588+0000 mon.a [INF] from='client.? 172.21.15.154:0/212121185' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi154-28128-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-16T03:44:00.482 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd: got: 2024-09-16T03:43:56.481505+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi154-28160-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-16T03:44:00.482 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd: got: 2024-09-16T03:43:56.512171+0000 mon.a [INF] from='client.? 172.21.15.154:0/2547152935' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi154-28277-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-16T03:44:00.482 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd: got: 2024-09-16T03:43:56.822092+0000 mon.a [WRN] Health check update: 8 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-16T03:44:00.482 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd: got: 2024-09-16T03:43:58.472787+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["twoxx"]}]': finished 2024-09-16T03:44:00.483 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd: got: 2024-09-16T03:43:59.271930+0000 mon.a [INF] from='client.? 172.21.15.154:0/2547152935' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi154-28277-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi154-28277-12"}]': finished 2024-09-16T03:44:00.483 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd: got: 2024-09-16T03:43:59.272221+0000 mon.a [INF] from='client.? 172.21.15.154:0/212121185' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi154-28128-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi154-28128-10"}]': finished 2024-09-16T03:44:00.483 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd: got: 2024-09-16T03:43:59.272442+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi154-28160-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi154-28160-10"}]': finished 2024-09-16T03:44:00.483 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd: got: 2024-09-16T03:43:59.272697+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi154-28563-1"}]': finished 2024-09-16T03:44:00.483 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd: got: 2024-09-16T03:43:59.272904+0000 mon.a [INF] from='client.14242 172.21.15.154:0/3494558292' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi154-28216-4", "tierpool": "test-rados-api-smithi154-28216-4-cache"}]': finished 2024-09-16T03:44:00.483 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd: got: 2024-09-16T03:43:59.273173+0000 mon.a [INF] from='client.? 172.21.15.154:0/3360117001' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi154-28091-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T03:44:00.483 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd: got: 2024-09-16T03:43:59.273336+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi154-28611-1"}]': finished 2024-09-16T03:44:00.483 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd: got: 2024-09-16T03:43:59.273507+0000 mon.a [INF] from='client.14137 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi154-28301-1"}]': finished 2024-09-16T03:44:00.497 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd: got: 2024-09-16T03:43:59.282273+0000 mon.c [INF] from='client.? 172.21.15.154:0/2016191396' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi154-28611-1"}]: dispatch 2024-09-16T03:44:00.497 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd: got: 2024-09-16T03:43:59.282621+0000 mon.c [INF] from='client.14137 172.21.15.154:0/531147286' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi154-28301-1"}]: dispatch 2024-09-16T03:44:00.497 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd: got: 2024-09-16T03:43:59.288176+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi154-28611-1"}]: dispatch 2024-09-16T03:44:00.497 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd: got: 2024-09-16T03:43:59.290080+0000 mon.a [INF] from='client.14242 172.21.15.154:0/3494558292' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi154-28216-4", "overlaypool": "test-rados-api-smithi154-28216-4-cache"}]: dispatch 2024-09-16T03:44:00.497 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd: got: 2024-09-16T03:43:59.290832+0000 mon.a [INF] from='client.14137 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi154-28301-1"}]: dispatch 2024-09-16T03:44:00.498 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd: got: 2024-09-16T03:43:59.298944+0000 mon.c [INF] from='client.? 172.21.15.154:0/4166986944' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi154-28083-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-16T03:44:00.498 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd: got: 2024-09-16T03:43:59.300450+0000 mon.a [INF] from='client.? 172.21.15.154:0/3360117001' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi154-28091-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-09-16T03:44:00.498 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd: got: 2024-09-16T03:43:59.301878+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi154-28083-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-16T03:44:00.498 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd: got: 2024-09-16T03:43:59.302934+0000 mon.c [INF] from='client.? 172.21.15.154:0/45313984' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi154-28381-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-16T03:44:00.498 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd: got: 2024-09-16T03:43:59.303312+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi154-28381-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-16T03:44:00.498 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd: got: 2024-09-16T03:43:59.305494+0000 mon.c [INF] from='client.? 172.21.15.154:0/163515610' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi154-28224-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-16T03:44:00.498 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd: got: 2024-09-16T03:43:59.305853+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi154-28224-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-16T03:44:00.498 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd: got: 2024-09-16T03:43:59.345378+0000 mon.c [INF] from='client.? 172.21.15.154:0/4136475447' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi154-28563-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-16T03:44:00.498 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd: got: 2024-09-16T03:43:59.346008+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi154-28563-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-16T03:44:00.498 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd: got: 2024-09-16T03:44:00.279456+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi154-28611-1"}]': finished 2024-09-16T03:44:00.498 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd: got: 2024-09-16T03:44:00.279632+0000 mon.a [INF] from='client.14242 172.21.15.154:0/3494558292' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi154-28216-4", "overlaypool": "test-rados-api-smithi154-28216-4-cache"}]': finished 2024-09-16T03:44:00.499 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd: got: 2024-09-16T03:44:00.279810+0000 mon.a [INF] from='client.14137 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi154-28301-1"}]': finished 2024-09-16T03:44:00.499 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd: got: 2024-09-16T03:44:00.280020+0000 mon.a [INF] from='client.? 172.21.15.154:0/3360117001' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi154-28091-3", "field": "max_bytes", "val": "4096"}]': finished 2024-09-16T03:44:00.499 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd: got: 2024-09-16T03:44:00.280219+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi154-28083-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T03:44:00.499 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd: got: 2024-09-16T03:44:00.280338+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi154-28381-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T03:44:00.499 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd: got: 2024-09-16T03:44:00.280521+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi154-28224-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-16T03:44:00.499 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd: got: 2024-09-16T03:44:00.280743+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi154-28563-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T03:44:00.499 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd: got: 2024-09-16T03:44:00.290258+0000 mon.a [INF] from='client.14242 172.21.15.154:0/3494558292' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi154-28216-4-cache", "mode": "writeback"}]: dispatch 2024-09-16T03:44:00.499 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd: got: 2024-09-16T03:44:00.290863+0000 mon.a [INF] from='client.4673 172.21.15.154:0/2745019661' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi154-28099-16"}]: dispatch 2024-09-16T03:44:00.499 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd: got: 2024-09-16T03:44:00.293623+0000 mon.c [INF] from='client.? 172.21.15.154:0/4136475447' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReadOpsmithi154-28563-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi154-28563-2"}]: dispatch 2024-09-16T03:44:00.499 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd: got: 2024-09-16T03:44:00.296449+0000 mon.c [INF] from='client.4658 172.21.15.154:0/499491003' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi154-28249-7"}]: dispatch 2024-09-16T03:44:00.499 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd: got: 2024-09-16T03:44:00.296782+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReadOpsmithi154-28563-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi154-28563-2"}]: dispatch 2024-09-16T03:44:00.500 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd: got: 2024-09-16T03:44:00.298309+0000 mon.a [INF] from='client.4658 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi154-28249-7"}]: dispatch 2024-09-16T03:44:00.500 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd: got: 2024-09-16T03:44:00.301639+0000 mon.c [INF] from='client.14419 172.21.15.154:0/2947288884' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi154-28286-7"}]: dispatch 2024-09-16T03:44:00.500 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd: got: 2024-09-16T03:44:00.347150+0000 mon.a [INF] from='client.? 172.21.15.154:0/281442000' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi154-28107-22"}]: dispatch 2024-09-16T03:44:00.500 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd: got: 2024-09-16T03:44:00.347615+0000 mon.a [INF] from='client.14419 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi154-28286-7"}]: dispatch 2024-09-16T03:44:01.499 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd: got: 2024-09-16T03:44:00.424206+0000 mon.a [INF] from='client.? 172.21.15.154:0/2921146271' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi154-28611-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-16T03:44:01.499 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd: got: 2024-09-16T03:44:00.480587+0000 mon.c [INF] from='client.? 172.21.15.154:0/1956241571' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-09-16T03:44:01.499 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd: got: 2024-09-16T03:44:00.480942+0000 client.admin [INF] threexx 2024-09-16T03:44:01.500 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd: got: 2024-09-16T03:44:00.499928+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-09-16T03:44:01.500 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd: got: 2024-09-16T03:44:01.149037+0000 mon.a [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-16T03:44:01.500 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd: got: 2024-09-16T03:44:01.151839+0000 mon.a [INF] from='client.14242 172.21.15.154:0/3494558292' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi154-28216-4-cache", "mode": "writeback"}]': finished 2024-09-16T03:44:01.500 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd: got: 2024-09-16T03:44:01.151992+0000 mon.a [INF] from='client.4673 172.21.15.154:0/2745019661' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi154-28099-16"}]': finished 2024-09-16T03:44:01.500 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd: got: 2024-09-16T03:44:01.152141+0000 mon.a [INF] from='client.4658 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi154-28249-7"}]': finished 2024-09-16T03:44:01.500 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd: got: 2024-09-16T03:44:01.152395+0000 mon.a [INF] from='client.? 172.21.15.154:0/281442000' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi154-28107-22"}]': finished 2024-09-16T03:44:01.500 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd: got: 2024-09-16T03:44:01.152663+0000 mon.a [INF] from='client.14419 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi154-28286-7"}]': finished 2024-09-16T03:44:01.500 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd: got: 2024-09-16T03:44:01.152888+0000 mon.a [INF] from='client.? 172.21.15.154:0/2921146271' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi154-28611-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-16T03:44:01.500 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd: got: 2024-09-16T03:44:01.160783+0000 mon.c [INF] from='client.4658 172.21.15.154:0/499491003' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi154-28249-7"}]: dispatch 2024-09-16T03:44:01.500 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd: got: 2024-09-16T03:44:01.161288+0000 mon.c [INF] from='client.14419 172.21.15.154:0/2947288884' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi154-28286-7"}]: dispatch 2024-09-16T03:44:01.500 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd: got: 2024-09-16T03:44:01.214436+0000 mon.a [INF] from='client.? 172.21.15.154:0/2921146271' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi154-28611-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi154-28611-2"}]: dispatch 2024-09-16T03:44:01.500 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd: got: 2024-09-16T03:44:01.215009+0000 mon.a [INF] from='client.4658 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi154-28249-7"}]: dispatch 2024-09-16T03:44:01.500 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd: got: 2024-09-16T03:44:01.215604+0000 mon.a [INF] from='client.? 172.21.15.154:0/281442000' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi154-28107-22"}]: dispatch 2024-09-16T03:44:01.500 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd: got: 2024-09-16T03:44:01.215836+0000 mon.a [INF] from='client.4673 172.21.15.154:0/2745019661' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi154-28099-16"}]: dispatch 2024-09-16T03:44:01.501 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd: got: 2024-09-16T03:44:01.217465+0000 mon.a [INF] from='client.14419 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi154-28286-7"}]: dispatch 2024-09-16T03:44:01.532 INFO:tasks.workunit.client.0.smithi154.stdout: api_service_pp: [==========] Running 4 tests from 1 test suite. 2024-09-16T03:44:01.533 INFO:tasks.workunit.client.0.smithi154.stdout: api_service_pp: [----------] Global test environment set-up. 2024-09-16T03:44:01.533 INFO:tasks.workunit.client.0.smithi154.stdout: api_service_pp: [----------] 4 tests from LibRadosServicePP 2024-09-16T03:44:01.533 INFO:tasks.workunit.client.0.smithi154.stdout: api_service_pp: [ RUN ] LibRadosServicePP.RegisterEarly 2024-09-16T03:44:01.534 INFO:tasks.workunit.client.0.smithi154.stdout: api_service_pp: [ OK ] LibRadosServicePP.RegisterEarly (6658 ms) 2024-09-16T03:44:01.534 INFO:tasks.workunit.client.0.smithi154.stdout: api_service_pp: [ RUN ] LibRadosServicePP.RegisterLate 2024-09-16T03:44:01.534 INFO:tasks.workunit.client.0.smithi154.stdout: api_service_pp: [ OK ] LibRadosServicePP.RegisterLate (56 ms) 2024-09-16T03:44:01.534 INFO:tasks.workunit.client.0.smithi154.stdout: api_service_pp: [ RUN ] LibRadosServicePP.Status 2024-09-16T03:44:01.534 INFO:tasks.workunit.client.0.smithi154.stdout: api_service_pp: [ OK ] LibRadosServicePP.Status (20042 ms) 2024-09-16T03:44:01.534 INFO:tasks.workunit.client.0.smithi154.stdout: api_service_pp: [ RUN ] LibRadosServicePP.Close 2024-09-16T03:44:01.535 INFO:tasks.workunit.client.0.smithi154.stdout: api_service_pp: attempt 0 of 20 2024-09-16T03:44:01.535 INFO:tasks.workunit.client.0.smithi154.stdout: api_service_pp: [ OK ] LibRadosServicePP.Close (5066 ms) 2024-09-16T03:44:01.535 INFO:tasks.workunit.client.0.smithi154.stdout: api_service_pp: [----------] 4 tests from LibRadosServicePP (31822 ms total) 2024-09-16T03:44:01.535 INFO:tasks.workunit.client.0.smithi154.stdout: api_service_pp: 2024-09-16T03:44:01.535 INFO:tasks.workunit.client.0.smithi154.stdout: api_service_pp: [----------] Global test environment tear-down 2024-09-16T03:44:01.535 INFO:tasks.workunit.client.0.smithi154.stdout: api_service_pp: [==========] 4 tests from 1 test suite ran. (31822 ms total) 2024-09-16T03:44:01.535 INFO:tasks.workunit.client.0.smithi154.stdout: api_service_pp: [ PASSED ] 4 tests. 2024-09-16T03:44:02.188 INFO:tasks.workunit.client.0.smithi154.stdout: api_stat_pp: Running main() from gmock_main.cc 2024-09-16T03:44:02.189 INFO:tasks.workunit.client.0.smithi154.stdout: api_stat_pp: [==========] Running 9 tests from 2 test suites. 2024-09-16T03:44:02.189 INFO:tasks.workunit.client.0.smithi154.stdout: api_stat_pp: [----------] Global test environment set-up. 2024-09-16T03:44:02.189 INFO:tasks.workunit.client.0.smithi154.stdout: api_stat_pp: [----------] 5 tests from LibRadosStatPP 2024-09-16T03:44:02.189 INFO:tasks.workunit.client.0.smithi154.stdout: api_stat_pp: seed 28286 2024-09-16T03:44:02.189 INFO:tasks.workunit.client.0.smithi154.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.StatPP 2024-09-16T03:44:02.189 INFO:tasks.workunit.client.0.smithi154.stdout: api_stat_pp: [ OK ] LibRadosStatPP.StatPP (340 ms) 2024-09-16T03:44:02.189 INFO:tasks.workunit.client.0.smithi154.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.Stat2Mtime2PP 2024-09-16T03:44:02.189 INFO:tasks.workunit.client.0.smithi154.stdout: api_stat_pp: [ OK ] LibRadosStatPP.Stat2Mtime2PP (5 ms) 2024-09-16T03:44:02.189 INFO:tasks.workunit.client.0.smithi154.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.ClusterStatPP 2024-09-16T03:44:02.189 INFO:tasks.workunit.client.0.smithi154.stdout: api_stat_pp: [ OK ] LibRadosStatPP.ClusterStatPP (0 ms) 2024-09-16T03:44:02.189 INFO:tasks.workunit.client.0.smithi154.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.PoolStatPP 2024-09-16T03:44:02.189 INFO:tasks.workunit.client.0.smithi154.stdout: api_stat_pp: [ OK ] LibRadosStatPP.PoolStatPP (4 ms) 2024-09-16T03:44:02.189 INFO:tasks.workunit.client.0.smithi154.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.StatPPNS 2024-09-16T03:44:02.190 INFO:tasks.workunit.client.0.smithi154.stdout: api_stat_pp: [ OK ] LibRadosStatPP.StatPPNS (20 ms) 2024-09-16T03:44:02.190 INFO:tasks.workunit.client.0.smithi154.stdout: api_stat_pp: [----------] 5 tests from LibRadosStatPP (369 ms total) 2024-09-16T03:44:02.190 INFO:tasks.workunit.client.0.smithi154.stdout: api_stat_pp: 2024-09-16T03:44:02.190 INFO:tasks.workunit.client.0.smithi154.stdout: api_stat_pp: [----------] 4 tests from LibRadosStatECPP 2024-09-16T03:44:02.190 INFO:tasks.workunit.client.0.smithi154.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.StatPP 2024-09-16T03:44:02.190 INFO:tasks.workunit.client.0.smithi154.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.StatPP (1145 ms) 2024-09-16T03:44:02.190 INFO:tasks.workunit.client.0.smithi154.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.ClusterStatPP 2024-09-16T03:44:02.190 INFO:tasks.workunit.client.0.smithi154.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.ClusterStatPP (0 ms) 2024-09-16T03:44:02.190 INFO:tasks.workunit.client.0.smithi154.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.PoolStatPP 2024-09-16T03:44:02.190 INFO:tasks.workunit.client.0.smithi154.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.PoolStatPP (9 ms) 2024-09-16T03:44:02.190 INFO:tasks.workunit.client.0.smithi154.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.StatPPNS 2024-09-16T03:44:02.190 INFO:tasks.workunit.client.0.smithi154.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.StatPPNS (16 ms) 2024-09-16T03:44:02.190 INFO:tasks.workunit.client.0.smithi154.stdout: api_stat_pp: [----------] 4 tests from LibRadosStatECPP (1170 ms total) 2024-09-16T03:44:02.190 INFO:tasks.workunit.client.0.smithi154.stdout: api_stat_pp: 2024-09-16T03:44:02.190 INFO:tasks.workunit.client.0.smithi154.stdout: api_stat_pp: [----------] Global test environment tear-down 2024-09-16T03:44:02.191 INFO:tasks.workunit.client.0.smithi154.stdout: api_stat_pp: [==========] 9 tests from 2 test suites ran. (32521 ms total) 2024-09-16T03:44:02.191 INFO:tasks.workunit.client.0.smithi154.stdout: api_stat_pp: [ PASSED ] 9 tests. 2024-09-16T03:44:02.191 INFO:tasks.workunit.client.0.smithi154.stdout: api_stat: Running main() from gmock_main.cc 2024-09-16T03:44:02.191 INFO:tasks.workunit.client.0.smithi154.stdout: api_stat: [==========] Running 9 tests from 2 test suites. 2024-09-16T03:44:02.191 INFO:tasks.workunit.client.0.smithi154.stdout: api_stat: [----------] Global test environment set-up. 2024-09-16T03:44:02.191 INFO:tasks.workunit.client.0.smithi154.stdout: api_stat: [----------] 5 tests from LibRadosStat 2024-09-16T03:44:02.191 INFO:tasks.workunit.client.0.smithi154.stdout: api_stat: [ RUN ] LibRadosStat.Stat 2024-09-16T03:44:02.191 INFO:tasks.workunit.client.0.smithi154.stdout: api_stat: [ OK ] LibRadosStat.Stat (207 ms) 2024-09-16T03:44:02.191 INFO:tasks.workunit.client.0.smithi154.stdout: api_stat: [ RUN ] LibRadosStat.Stat2 2024-09-16T03:44:02.191 INFO:tasks.workunit.client.0.smithi154.stdout: api_stat: [ OK ] LibRadosStat.Stat2 (96 ms) 2024-09-16T03:44:02.192 INFO:tasks.workunit.client.0.smithi154.stdout: api_stat: [ RUN ] LibRadosStat.StatNS 2024-09-16T03:44:02.192 INFO:tasks.workunit.client.0.smithi154.stdout: api_stat: [ OK ] LibRadosStat.StatNS (27 ms) 2024-09-16T03:44:02.192 INFO:tasks.workunit.client.0.smithi154.stdout: api_stat: [ RUN ] LibRadosStat.ClusterStat 2024-09-16T03:44:02.192 INFO:tasks.workunit.client.0.smithi154.stdout: api_stat: [ OK ] LibRadosStat.ClusterStat (1 ms) 2024-09-16T03:44:02.193 INFO:tasks.workunit.client.0.smithi154.stdout: api_stat: [ RUN ] LibRadosStat.PoolStat 2024-09-16T03:44:02.193 INFO:tasks.workunit.client.0.smithi154.stdout: api_stat: [ OK ] LibRadosStat.PoolStat (4 ms) 2024-09-16T03:44:02.193 INFO:tasks.workunit.client.0.smithi154.stdout: api_stat: [----------] 5 tests from LibRadosStat (335 ms total) 2024-09-16T03:44:02.193 INFO:tasks.workunit.client.0.smithi154.stdout: api_stat: 2024-09-16T03:44:02.193 INFO:tasks.workunit.client.0.smithi154.stdout: api_stat: [----------] 4 tests from LibRadosStatEC 2024-09-16T03:44:02.193 INFO:tasks.workunit.client.0.smithi154.stdout: api_stat: [ RUN ] LibRadosStatEC.Stat 2024-09-16T03:44:02.193 INFO:tasks.workunit.client.0.smithi154.stdout: api_stat: [ OK ] LibRadosStatEC.Stat (1146 ms) 2024-09-16T03:44:02.193 INFO:tasks.workunit.client.0.smithi154.stdout: api_stat: [ RUN ] LibRadosStatEC.StatNS 2024-09-16T03:44:02.193 INFO:tasks.workunit.client.0.smithi154.stdout: api_stat: [ OK ] LibRadosStatEC.StatNS (18 ms) 2024-09-16T03:44:02.193 INFO:tasks.workunit.client.0.smithi154.stdout: api_stat: [ RUN ] LibRadosStatEC.ClusterStat 2024-09-16T03:44:02.193 INFO:tasks.workunit.client.0.smithi154.stdout: api_stat: [ OK ] LibRadosStatEC.ClusterStat (1 ms) 2024-09-16T03:44:02.193 INFO:tasks.workunit.client.0.smithi154.stdout: api_stat: [ RUN ] LibRadosStatEC.PoolStat 2024-09-16T03:44:02.194 INFO:tasks.workunit.client.0.smithi154.stdout: api_stat: [ OK ] LibRadosStatEC.PoolStat (3 ms) 2024-09-16T03:44:02.194 INFO:tasks.workunit.client.0.smithi154.stdout: api_stat: [----------] 4 tests from LibRadosStatEC (1168 ms total) 2024-09-16T03:44:02.194 INFO:tasks.workunit.client.0.smithi154.stdout: api_stat: 2024-09-16T03:44:02.194 INFO:tasks.workunit.client.0.smithi154.stdout: api_stat: [----------] Global test environment tear-down 2024-09-16T03:44:02.194 INFO:tasks.workunit.client.0.smithi154.stdout: api_stat: [==========] 9 tests from 2 test suites ran. (32551 ms total) 2024-09-16T03:44:02.194 INFO:tasks.workunit.client.0.smithi154.stdout: api_stat: [ PASSED ] 9 tests. 2024-09-16T03:44:02.233 INFO:tasks.workunit.client.0.smithi154.stdout: api_io: Running main() from gmock_main.cc 2024-09-16T03:44:02.233 INFO:tasks.workunit.client.0.smithi154.stdout: api_io: [==========] Running 24 tests from 2 test suites. 2024-09-16T03:44:02.233 INFO:tasks.workunit.client.0.smithi154.stdout: api_io: [----------] Global test environment set-up. 2024-09-16T03:44:02.233 INFO:tasks.workunit.client.0.smithi154.stdout: api_io: [----------] 14 tests from LibRadosIo 2024-09-16T03:44:02.233 INFO:tasks.workunit.client.0.smithi154.stdout: api_io: [ RUN ] LibRadosIo.SimpleWrite 2024-09-16T03:44:02.233 INFO:tasks.workunit.client.0.smithi154.stdout: api_io: [ OK ] LibRadosIo.SimpleWrite (317 ms) 2024-09-16T03:44:02.234 INFO:tasks.workunit.client.0.smithi154.stdout: api_io: [ RUN ] LibRadosIo.TooBig 2024-09-16T03:44:02.234 INFO:tasks.workunit.client.0.smithi154.stdout: api_io: [ OK ] LibRadosIo.TooBig (0 ms) 2024-09-16T03:44:02.234 INFO:tasks.workunit.client.0.smithi154.stdout: api_io: [ RUN ] LibRadosIo.ReadTimeout 2024-09-16T03:44:02.234 INFO:tasks.workunit.client.0.smithi154.stdout: api_io: no timeout :/ 2024-09-16T03:44:02.234 INFO:tasks.workunit.client.0.smithi154.stdout: api_io: no timeout :/ 2024-09-16T03:44:02.234 INFO:tasks.workunit.client.0.smithi154.stdout: api_io: no timeout :/ 2024-09-16T03:44:02.234 INFO:tasks.workunit.client.0.smithi154.stdout: api_io: no timeout :/ 2024-09-16T03:44:02.234 INFO:tasks.workunit.client.0.smithi154.stdout: api_io: no timeout :/ 2024-09-16T03:44:02.234 INFO:tasks.workunit.client.0.smithi154.stdout: api_io: [ OK ] LibRadosIo.ReadTimeout (45 ms) 2024-09-16T03:44:02.234 INFO:tasks.workunit.client.0.smithi154.stdout: api_io: [ RUN ] LibRadosIo.RoundTrip 2024-09-16T03:44:02.234 INFO:tasks.workunit.client.0.smithi154.stdout: api_io: [ OK ] LibRadosIo.RoundTrip (7 ms) 2024-09-16T03:44:02.234 INFO:tasks.workunit.client.0.smithi154.stdout: api_io: [ RUN ] LibRadosIo.Checksum 2024-09-16T03:44:02.234 INFO:tasks.workunit.client.0.smithi154.stdout: api_io: [ OK ] LibRadosIo.Checksum (2 ms) 2024-09-16T03:44:02.234 INFO:tasks.workunit.client.0.smithi154.stdout: api_io: [ RUN ] LibRadosIo.OverlappingWriteRoundTrip 2024-09-16T03:44:02.234 INFO:tasks.workunit.client.0.smithi154.stdout: api_io: [ OK ] LibRadosIo.OverlappingWriteRoundTrip (6 ms) 2024-09-16T03:44:02.235 INFO:tasks.workunit.client.0.smithi154.stdout: api_io: [ RUN ] LibRadosIo.WriteFullRoundTrip 2024-09-16T03:44:02.235 INFO:tasks.workunit.client.0.smithi154.stdout: api_io: [ OK ] LibRadosIo.WriteFullRoundTrip (8 ms) 2024-09-16T03:44:02.235 INFO:tasks.workunit.client.0.smithi154.stdout: api_io: [ RUN ] LibRadosIo.AppendRoundTrip 2024-09-16T03:44:02.235 INFO:tasks.workunit.client.0.smithi154.stdout: api_io: [ OK ] LibRadosIo.AppendRoundTrip (4 ms) 2024-09-16T03:44:02.235 INFO:tasks.workunit.client.0.smithi154.stdout: api_io: [ RUN ] LibRadosIo.ZeroLenZero 2024-09-16T03:44:02.235 INFO:tasks.workunit.client.0.smithi154.stdout: api_io: [ OK ] LibRadosIo.ZeroLenZero (2 ms) 2024-09-16T03:44:02.235 INFO:tasks.workunit.client.0.smithi154.stdout: api_io: [ RUN ] LibRadosIo.TruncTest 2024-09-16T03:44:02.235 INFO:tasks.workunit.client.0.smithi154.stdout: api_io: [ OK ] LibRadosIo.TruncTest (11 ms) 2024-09-16T03:44:02.235 INFO:tasks.workunit.client.0.smithi154.stdout: api_io: [ RUN ] LibRadosIo.RemoveTest 2024-09-16T03:44:02.235 INFO:tasks.workunit.client.0.smithi154.stdout: api_io: [ OK ] LibRadosIo.RemoveTest (3 ms) 2024-09-16T03:44:02.235 INFO:tasks.workunit.client.0.smithi154.stdout: api_io: [ RUN ] LibRadosIo.XattrsRoundTrip 2024-09-16T03:44:02.235 INFO:tasks.workunit.client.0.smithi154.stdout: api_io: [ OK ] LibRadosIo.XattrsRoundTrip (8 ms) 2024-09-16T03:44:02.235 INFO:tasks.workunit.client.0.smithi154.stdout: api_io: [ RUN ] LibRadosIo.RmXattr 2024-09-16T03:44:02.235 INFO:tasks.workunit.client.0.smithi154.stdout: api_io: [ OK ] LibRadosIo.RmXattr (25 ms) 2024-09-16T03:44:02.235 INFO:tasks.workunit.client.0.smithi154.stdout: api_io: [ RUN ] LibRadosIo.XattrIter 2024-09-16T03:44:02.236 INFO:tasks.workunit.client.0.smithi154.stdout: api_io: [ OK ] LibRadosIo.XattrIter (9 ms) 2024-09-16T03:44:02.236 INFO:tasks.workunit.client.0.smithi154.stdout: api_io: [----------] 14 tests from LibRadosIo (447 ms total) 2024-09-16T03:44:02.236 INFO:tasks.workunit.client.0.smithi154.stdout: api_io: 2024-09-16T03:44:02.236 INFO:tasks.workunit.client.0.smithi154.stdout: api_io: [----------] 10 tests from LibRadosIoEC 2024-09-16T03:44:02.236 INFO:tasks.workunit.client.0.smithi154.stdout: api_io: [ RUN ] LibRadosIoEC.SimpleWrite 2024-09-16T03:44:02.236 INFO:tasks.workunit.client.0.smithi154.stdout: api_io: [ OK ] LibRadosIoEC.SimpleWrite (977 ms) 2024-09-16T03:44:02.236 INFO:tasks.workunit.client.0.smithi154.stdout: api_io: [ RUN ] LibRadosIoEC.RoundTrip 2024-09-16T03:44:02.236 INFO:tasks.workunit.client.0.smithi154.stdout: api_io: [ OK ] LibRadosIoEC.RoundTrip (8 ms) 2024-09-16T03:44:02.236 INFO:tasks.workunit.client.0.smithi154.stdout: api_io: [ RUN ] LibRadosIoEC.OverlappingWriteRoundTrip 2024-09-16T03:44:02.236 INFO:tasks.workunit.client.0.smithi154.stdout: api_io: [ OK ] LibRadosIoEC.OverlappingWriteRoundTrip (8 ms) 2024-09-16T03:44:02.236 INFO:tasks.workunit.client.0.smithi154.stdout: api_io: [ RUN ] LibRadosIoEC.WriteFullRoundTrip 2024-09-16T03:44:02.236 INFO:tasks.workunit.client.0.smithi154.stdout: api_io: [ OK ] LibRadosIoEC.WriteFullRoundTrip (5 ms) 2024-09-16T03:44:02.237 INFO:tasks.workunit.client.0.smithi154.stdout: api_io: [ RUN ] LibRadosIoEC.AppendRoundTrip 2024-09-16T03:44:02.237 INFO:tasks.workunit.client.0.smithi154.stdout: api_io: [ OK ] LibRadosIoEC.AppendRoundTrip (12 ms) 2024-09-16T03:44:02.237 INFO:tasks.workunit.client.0.smithi154.stdout: api_io: [ RUN ] LibRadosIoEC.TruncTest 2024-09-16T03:44:02.237 INFO:tasks.workunit.client.0.smithi154.stdout: api_io: [ OK ] LibRadosIoEC.TruncTest (4 ms) 2024-09-16T03:44:02.237 INFO:tasks.workunit.client.0.smithi154.stdout: api_io: [ RUN ] LibRadosIoEC.RemoveTest 2024-09-16T03:44:02.237 INFO:tasks.workunit.client.0.smithi154.stdout: api_io: [ OK ] LibRadosIoEC.RemoveTest (4 ms) 2024-09-16T03:44:02.237 INFO:tasks.workunit.client.0.smithi154.stdout: api_io: [ RUN ] LibRadosIoEC.XattrsRoundTrip 2024-09-16T03:44:02.237 INFO:tasks.workunit.client.0.smithi154.stdout: api_io: [ OK ] LibRadosIoEC.XattrsRoundTrip (6 ms) 2024-09-16T03:44:02.237 INFO:tasks.workunit.client.0.smithi154.stdout: api_io: [ RUN ] LibRadosIoEC.RmXattr 2024-09-16T03:44:02.237 INFO:tasks.workunit.client.0.smithi154.stdout: api_io: [ OK ] LibRadosIoEC.RmXattr (15 ms) 2024-09-16T03:44:02.237 INFO:tasks.workunit.client.0.smithi154.stdout: api_io: [ RUN ] LibRadosIoEC.XattrIter 2024-09-16T03:44:02.238 INFO:tasks.workunit.client.0.smithi154.stdout: api_io: [ OK ] LibRadosIoEC.XattrIter (5 ms) 2024-09-16T03:44:02.238 INFO:tasks.workunit.client.0.smithi154.stdout: api_io: [----------] 10 tests from LibRadosIoEC (1045 ms total) 2024-09-16T03:44:02.238 INFO:tasks.workunit.client.0.smithi154.stdout: api_io: 2024-09-16T03:44:02.240 INFO:tasks.workunit.client.0.smithi154.stdout: api_io: [----------] Global test environment tear-down 2024-09-16T03:44:02.240 INFO:tasks.workunit.client.0.smithi154.stdout: api_io: [==========] 24 tests from 2 test suites ran. (32676 ms total) 2024-09-16T03:44:02.241 INFO:tasks.workunit.client.0.smithi154.stdout: api_io: [ PASSED ] 24 tests. 2024-09-16T03:44:02.241 INFO:tasks.workunit.client.0.smithi154.stdout:] LibRadosIoECPP.OverlappingWriteRoundTripPP 2024-09-16T03:44:02.241 INFO:tasks.workunit.client.0.smithi154.stdout: api_io_pp: [ OK ] LibRadosIoECPP.OverlappingWriteRoundTripPP (5 ms) 2024-09-16T03:44:02.241 INFO:tasks.workunit.client.0.smithi154.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.WriteFullRoundTripPP 2024-09-16T03:44:02.241 INFO:tasks.workunit.client.0.smithi154.stdout: api_io_pp: [ OK ] LibRadosIoECPP.WriteFullRoundTripPP (217 ms) 2024-09-16T03:44:02.241 INFO:tasks.workunit.client.0.smithi154.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.WriteFullRoundTripPP2 2024-09-16T03:44:02.241 INFO:tasks.workunit.client.0.smithi154.stdout: api_io_pp: [ OK ] LibRadosIoECPP.WriteFullRoundTripPP2 (9 ms) 2024-09-16T03:44:02.241 INFO:tasks.workunit.client.0.smithi154.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.AppendRoundTripPP 2024-09-16T03:44:02.241 INFO:tasks.workunit.client.0.smithi154.stdout: api_io_pp: [ OK ] LibRadosIoECPP.AppendRoundTripPP (10 ms) 2024-09-16T03:44:02.242 INFO:tasks.workunit.client.0.smithi154.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.TruncTestPP 2024-09-16T03:44:02.242 INFO:tasks.workunit.client.0.smithi154.stdout: api_io_pp: [ OK ] LibRadosIoECPP.TruncTestPP (10 ms) 2024-09-16T03:44:02.242 INFO:tasks.workunit.client.0.smithi154.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RemoveTestPP 2024-09-16T03:44:02.242 INFO:tasks.workunit.client.0.smithi154.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RemoveTestPP (10 ms) 2024-09-16T03:44:02.242 INFO:tasks.workunit.client.0.smithi154.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.XattrsRoundTripPP 2024-09-16T03:44:02.242 INFO:tasks.workunit.client.0.smithi154.stdout: api_io_pp: [ OK ] LibRadosIoECPP.XattrsRoundTripPP (9 ms) 2024-09-16T03:44:02.242 INFO:tasks.workunit.client.0.smithi154.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RmXattrPP 2024-09-16T03:44:02.242 INFO:tasks.workunit.client.0.smithi154.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RmXattrPP (26 ms) 2024-09-16T03:44:02.242 INFO:tasks.workunit.client.0.smithi154.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.XattrListPP 2024-09-16T03:44:02.242 INFO:tasks.workunit.client.0.smithi154.stdout: api_io_pp: [ OK ] LibRadosIoECPP.XattrListPP (16 ms) 2024-09-16T03:44:02.242 INFO:tasks.workunit.client.0.smithi154.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtPP 2024-09-16T03:44:02.242 INFO:tasks.workunit.client.0.smithi154.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtPP (15 ms) 2024-09-16T03:44:02.242 INFO:tasks.workunit.client.0.smithi154.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtDNEPP 2024-09-16T03:44:02.243 INFO:tasks.workunit.client.0.smithi154.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtDNEPP (6 ms) 2024-09-16T03:44:02.243 INFO:tasks.workunit.client.0.smithi154.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtMismatchPP 2024-09-16T03:44:02.243 INFO:tasks.workunit.client.0.smithi154.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtMismatchPP (11 ms) 2024-09-16T03:44:02.243 INFO:tasks.workunit.client.0.smithi154.stdout: api_io_pp: [----------] 17 tests from LibRadosIoECPP (1349 ms total) 2024-09-16T03:44:02.243 INFO:tasks.workunit.client.0.smithi154.stdout: api_io_pp: 2024-09-16T03:44:02.243 INFO:tasks.workunit.client.0.smithi154.stdout: api_io_pp: [----------] Global test environment tear-down 2024-09-16T03:44:02.243 INFO:tasks.workunit.client.0.smithi154.stdout: api_io_pp: [==========] 37 tests from 2 test suites ran. (32672 ms total) 2024-09-16T03:44:02.243 INFO:tasks.workunit.client.0.smithi154.stdout: api_io_pp: [ PASSED ] 37 tests. 2024-09-16T03:44:02.249 INFO:tasks.workunit.client.0.smithi154.stdout: list: Running main() from gmock_main.cc 2024-09-16T03:44:02.249 INFO:tasks.workunit.client.0.smithi154.stdout: list: [==========] Running 3 tests from 1 test suite. 2024-09-16T03:44:02.249 INFO:tasks.workunit.client.0.smithi154.stdout: list: [----------] Global test environment set-up. 2024-09-16T03:44:02.249 INFO:tasks.workunit.client.0.smithi154.stdout: list: [----------] 3 tests from NeoradosList 2024-09-16T03:44:02.250 INFO:tasks.workunit.client.0.smithi154.stdout: list: [ RUN ] NeoradosList.ListObjects 2024-09-16T03:44:02.250 INFO:tasks.workunit.client.0.smithi154.stdout: list: [ OK ] NeoradosList.ListObjects (4659 ms) 2024-09-16T03:44:02.250 INFO:tasks.workunit.client.0.smithi154.stdout: list: [ RUN ] NeoradosList.ListObjectsNS 2024-09-16T03:44:02.250 INFO:tasks.workunit.client.0.smithi154.stdout: list: [ OK ] NeoradosList.ListObjectsNS (23997 ms) 2024-09-16T03:44:02.250 INFO:tasks.workunit.client.0.smithi154.stdout: list: [ RUN ] NeoradosList.ListObjectsMany 2024-09-16T03:44:02.250 INFO:tasks.workunit.client.0.smithi154.stdout: list: [ OK ] NeoradosList.ListObjectsMany (3798 ms) 2024-09-16T03:44:02.250 INFO:tasks.workunit.client.0.smithi154.stdout: list: [----------] 3 tests from NeoradosList (32454 ms total) 2024-09-16T03:44:02.250 INFO:tasks.workunit.client.0.smithi154.stdout: list: 2024-09-16T03:44:02.250 INFO:tasks.workunit.client.0.smithi154.stdout: list: [----------] Global test environment tear-down 2024-09-16T03:44:02.250 INFO:tasks.workunit.client.0.smithi154.stdout: list: [==========] 3 tests from 1 test suite ran. (32454 ms total) 2024-09-16T03:44:02.250 INFO:tasks.workunit.client.0.smithi154.stdout: list: [ PASSED ] 3 tests. 2024-09-16T03:44:03.202 INFO:tasks.workunit.client.0.smithi154.stdout: watch_notify: Running main() from gmock_main.cc 2024-09-16T03:44:03.202 INFO:tasks.workunit.client.0.smithi154.stdout: watch_notify: [==========] Running 2 tests from 1 test suite. 2024-09-16T03:44:03.203 INFO:tasks.workunit.client.0.smithi154.stdout: watch_notify: [----------] Global test environment set-up. 2024-09-16T03:44:03.203 INFO:tasks.workunit.client.0.smithi154.stdout: watch_notify: [----------] 2 tests from NeoRadosWatchNotify 2024-09-16T03:44:03.203 INFO:tasks.workunit.client.0.smithi154.stdout: watch_notify: [ RUN ] NeoRadosWatchNotify.WatchNotify 2024-09-16T03:44:03.203 INFO:tasks.workunit.client.0.smithi154.stdout: watch_notify: handle_notify cookie 94018919100448 notify_id 94489280513 notifier_gid 4511 2024-09-16T03:44:03.203 INFO:tasks.workunit.client.0.smithi154.stdout: watch_notify: [ OK ] NeoRadosWatchNotify.WatchNotify (26452 ms) 2024-09-16T03:44:03.203 INFO:tasks.workunit.client.0.smithi154.stdout: watch_notify: [ RUN ] NeoRadosWatchNotify.WatchNotifyTimeout 2024-09-16T03:44:03.203 INFO:tasks.workunit.client.0.smithi154.stdout: watch_notify: Trying... 2024-09-16T03:44:03.203 INFO:tasks.workunit.client.0.smithi154.stdout: watch_notify: handle_notify cookie 94018931572720 notify_id 107374182402 notifier_gid 4748 2024-09-16T03:44:03.203 INFO:tasks.workunit.client.0.smithi154.stdout: watch_notify: Waiting for 3.000000000s 2024-09-16T03:44:03.203 INFO:tasks.workunit.client.0.smithi154.stdout: watch_notify: Timed out. 2024-09-16T03:44:03.203 INFO:tasks.workunit.client.0.smithi154.stdout: watch_notify: Flushing... 2024-09-16T03:44:03.203 INFO:tasks.workunit.client.0.smithi154.stdout: watch_notify: Flushed... 2024-09-16T03:44:03.203 INFO:tasks.workunit.client.0.smithi154.stdout: watch_notify: [ OK ] NeoRadosWatchNotify.WatchNotifyTimeout (6921 ms) 2024-09-16T03:44:03.204 INFO:tasks.workunit.client.0.smithi154.stdout: watch_notify: [----------] 2 tests from NeoRadosWatchNotify (33373 ms total) 2024-09-16T03:44:03.204 INFO:tasks.workunit.client.0.smithi154.stdout: watch_notify: 2024-09-16T03:44:03.204 INFO:tasks.workunit.client.0.smithi154.stdout: watch_notify: [----------] Global test environment tear-down 2024-09-16T03:44:03.204 INFO:tasks.workunit.client.0.smithi154.stdout: watch_notify: [==========] 2 tests from 1 test suite ran. (33373 ms total) 2024-09-16T03:44:03.204 INFO:tasks.workunit.client.0.smithi154.stdout: watch_notify: [ PASSED ] 2 tests. 2024-09-16T03:44:03.205 INFO:tasks.workunit.client.0.smithi154.stdout: api_watch_notify: Running main() from gmock_main.cc 2024-09-16T03:44:03.205 INFO:tasks.workunit.client.0.smithi154.stdout: api_watch_notify: [==========] Running 11 tests from 2 test suites. 2024-09-16T03:44:03.206 INFO:tasks.workunit.client.0.smithi154.stdout: api_watch_notify: [----------] Global test environment set-up. 2024-09-16T03:44:03.206 INFO:tasks.workunit.client.0.smithi154.stdout: api_watch_notify: [----------] 10 tests from LibRadosWatchNotify 2024-09-16T03:44:03.206 INFO:tasks.workunit.client.0.smithi154.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify 2024-09-16T03:44:03.206 INFO:tasks.workunit.client.0.smithi154.stdout: api_watch_notify: watch_notify_test_cb 2024-09-16T03:44:03.206 INFO:tasks.workunit.client.0.smithi154.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify (238 ms) 2024-09-16T03:44:03.206 INFO:tasks.workunit.client.0.smithi154.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.Watch2Delete 2024-09-16T03:44:03.206 INFO:tasks.workunit.client.0.smithi154.stdout: api_watch_notify: waiting up to 300 for disconnect notification ... 2024-09-16T03:44:03.206 INFO:tasks.workunit.client.0.smithi154.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94629872840512 err -107 2024-09-16T03:44:03.206 INFO:tasks.workunit.client.0.smithi154.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.Watch2Delete (1091 ms) 2024-09-16T03:44:03.206 INFO:tasks.workunit.client.0.smithi154.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchDelete 2024-09-16T03:44:03.207 INFO:tasks.workunit.client.0.smithi154.stdout: api_watch_notify: waiting up to 300 for disconnect notification ... 2024-09-16T03:44:03.207 INFO:tasks.workunit.client.0.smithi154.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94629872842704 err -107 2024-09-16T03:44:03.207 INFO:tasks.workunit.client.0.smithi154.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchDelete (1014 ms) 2024-09-16T03:44:03.207 INFO:tasks.workunit.client.0.smithi154.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2 2024-09-16T03:44:03.207 INFO:tasks.workunit.client.0.smithi154.stdout: api_watch_notify: watch_notify2_test_cb from 4388 notify_id 94489280512 cookie 94629872859456 2024-09-16T03:44:03.207 INFO:tasks.workunit.client.0.smithi154.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2 (16 ms) 2024-09-16T03:44:03.207 INFO:tasks.workunit.client.0.smithi154.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchNotify2 2024-09-16T03:44:03.207 INFO:tasks.workunit.client.0.smithi154.stdout: api_watch_notify: watch_notify2_test_cb from 4388 notify_id 94489280512 cookie 94629872859456 2024-09-16T03:44:03.207 INFO:tasks.workunit.client.0.smithi154.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchNotify2 (15 ms) 2024-09-16T03:44:03.207 INFO:tasks.workunit.client.0.smithi154.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioNotify 2024-09-16T03:44:03.208 INFO:tasks.workunit.client.0.smithi154.stdout: api_watch_notify: watch_notify2_test_cb from 4388 notify_id 94489280513 cookie 94629872859456 2024-09-16T03:44:03.208 INFO:tasks.workunit.client.0.smithi154.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioNotify (13 ms) 2024-09-16T03:44:03.208 INFO:tasks.workunit.client.0.smithi154.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2Multi 2024-09-16T03:44:03.208 INFO:tasks.workunit.client.0.smithi154.stdout: api_watch_notify: watch_notify2_test_cb from 4388 notify_id 94489280512 cookie 94629872865696 2024-09-16T03:44:03.208 INFO:tasks.workunit.client.0.smithi154.stdout: api_watch_notify: watch_notify2_test_cb from 4388 notify_id 94489280512 cookie 94629872870160 2024-09-16T03:44:03.208 INFO:tasks.workunit.client.0.smithi154.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2Multi (23 ms) 2024-09-16T03:44:03.208 INFO:tasks.workunit.client.0.smithi154.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2Timeout 2024-09-16T03:44:03.208 INFO:tasks.workunit.client.0.smithi154.stdout: api_watch_notify: watch_notify2_test_cb from 4388 notify_id 94489280512 cookie 94629872870096 2024-09-16T03:44:03.208 INFO:tasks.workunit.client.0.smithi154.stdout: api_watch_notify: watch_notify2_test_cb from 4388 notify_id 94489280513 cookie 94629872870096 2024-09-16T03:44:03.208 INFO:tasks.workunit.client.0.smithi154.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2Timeout (3013 ms) 2024-09-16T03:44:03.209 INFO:tasks.workunit.client.0.smithi154.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.Watch3Timeout 2024-09-16T03:44:03.209 INFO:tasks.workunit.client.0.smithi154.stdout: api_watch_notify: waiting up to 1024 for osd to time us out ... 2024-09-16T03:44:03.209 INFO:tasks.workunit.client.0.smithi154.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94629872877760 err -107 2024-09-16T03:44:03.209 INFO:tasks.workunit.client.0.smithi154.stdout: api_watch_notify: watch_notify2_test_cb from 4388 notify_id 94489280513 cookie 94629872877760 2024-09-16T03:44:03.209 INFO:tasks.workunit.client.0.smithi154.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.Watch3Timeout (5021 ms) 2024-09-16T03:44:03.209 INFO:tasks.workunit.client.0.smithi154.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchDelete2 2024-09-16T03:44:03.209 INFO:tasks.workunit.client.0.smithi154.stdout: api_watch_notify: waiting up to 30 for disconnect notification ... 2024-09-16T03:44:03.209 INFO:tasks.workunit.client.0.smithi154.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94629872877760 err -107 2024-09-16T03:44:03.209 INFO:tasks.workunit.client.0.smithi154.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchDelete2 (1014 ms) 2024-09-16T03:44:03.209 INFO:tasks.workunit.client.0.smithi154.stdout: api_watch_notify: [----------] 10 tests from LibRadosWatchNotify (11458 ms total) 2024-09-16T03:44:03.209 INFO:tasks.workunit.client.0.smithi154.stdout: api_watch_notify: 2024-09-16T03:44:03.209 INFO:tasks.workunit.client.0.smithi154.stdout: api_watch_notify: [----------] 1 test from LibRadosWatchNotifyEC 2024-09-16T03:44:03.209 INFO:tasks.workunit.client.0.smithi154.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotifyEC.WatchNotify 2024-09-16T03:44:03.210 INFO:tasks.workunit.client.0.smithi154.stdout: api_watch_notify: watch_notify_test_cb 2024-09-16T03:44:03.210 INFO:tasks.workunit.client.0.smithi154.stdout: api_watch_notify: [ OK ] LibRadosWatchNotifyEC.WatchNotify (1189 ms) 2024-09-16T03:44:03.210 INFO:tasks.workunit.client.0.smithi154.stdout: api_watch_notify: [----------] 1 test from LibRadosWatchNotifyEC (1189 ms total) 2024-09-16T03:44:03.210 INFO:tasks.workunit.client.0.smithi154.stdout: api_watch_notify: 2024-09-16T03:44:03.210 INFO:tasks.workunit.client.0.smithi154.stdout: api_watch_notify: [----------] Global test environment tear-down 2024-09-16T03:44:03.210 INFO:tasks.workunit.client.0.smithi154.stdout: api_watch_notify: [==========] 11 tests from 2 test suites ran. (33553 ms total) 2024-09-16T03:44:03.211 INFO:tasks.workunit.client.0.smithi154.stdout: api_watch_notify: [ PASSED ] 11 tests. 2024-09-16T03:44:04.529 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd: got: 2024-09-16T03:44:01.222196+0000 mon.a [INF] from='client.? 172.21.15.154:0/2547152935' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi154-28277-12"}]: dispatch 2024-09-16T03:44:04.529 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd: got: 2024-09-16T03:44:01.224266+0000 mon.c [INF] from='client.? 172.21.15.154:0/1219849304' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi154-28301-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-16T03:44:04.529 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd: got: 2024-09-16T03:44:01.224854+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi154-28301-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-16T03:44:04.529 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd: got: 2024-09-16T03:44:01.351939+0000 mon.a [INF] from='client.14242 172.21.15.154:0/3494558292' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi154-28216-4"}]: dispatch 2024-09-16T03:44:04.529 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd: [ OK ] LibRadosCmd.WatchLog (8248 ms) 2024-09-16T03:44:04.529 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd: [----------] 4 tests from LibRadosCmd (34844 ms total) 2024-09-16T03:44:04.529 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd: 2024-09-16T03:44:04.529 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd: [----------] Global test environment tear-down 2024-09-16T03:44:04.529 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd: [==========] 4 tests from 1 test suite ran. (34844 ms total) 2024-09-16T03:44:04.529 INFO:tasks.workunit.client.0.smithi154.stdout: api_cmd: [ PASSED ] 4 tests. 2024-09-16T03:44:05.220 INFO:tasks.workunit.client.0.smithi154.stdout: api_lock_pp: Running main() from gmock_main.cc 2024-09-16T03:44:05.221 INFO:tasks.workunit.client.0.smithi154.stdout: api_lock_pp: [==========] Running 16 tests from 2 test suites. 2024-09-16T03:44:05.221 INFO:tasks.workunit.client.0.smithi154.stdout: api_lock_pp: [----------] Global test environment set-up. 2024-09-16T03:44:05.221 INFO:tasks.workunit.client.0.smithi154.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockPP 2024-09-16T03:44:05.221 INFO:tasks.workunit.client.0.smithi154.stdout: api_lock_pp: seed 28160 2024-09-16T03:44:05.221 INFO:tasks.workunit.client.0.smithi154.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockExclusivePP 2024-09-16T03:44:05.221 INFO:tasks.workunit.client.0.smithi154.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockExclusivePP (219 ms) 2024-09-16T03:44:05.221 INFO:tasks.workunit.client.0.smithi154.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockSharedPP 2024-09-16T03:44:05.221 INFO:tasks.workunit.client.0.smithi154.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockSharedPP (83 ms) 2024-09-16T03:44:05.221 INFO:tasks.workunit.client.0.smithi154.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockExclusiveDurPP 2024-09-16T03:44:05.221 INFO:tasks.workunit.client.0.smithi154.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockExclusiveDurPP (1052 ms) 2024-09-16T03:44:05.221 INFO:tasks.workunit.client.0.smithi154.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockSharedDurPP 2024-09-16T03:44:05.221 INFO:tasks.workunit.client.0.smithi154.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockSharedDurPP (1014 ms) 2024-09-16T03:44:05.221 INFO:tasks.workunit.client.0.smithi154.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockMayRenewPP 2024-09-16T03:44:05.222 INFO:tasks.workunit.client.0.smithi154.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockMayRenewPP (14 ms) 2024-09-16T03:44:05.222 INFO:tasks.workunit.client.0.smithi154.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.UnlockPP 2024-09-16T03:44:05.222 INFO:tasks.workunit.client.0.smithi154.stdout: api_lock_pp: [ OK ] LibRadosLockPP.UnlockPP (13 ms) 2024-09-16T03:44:05.222 INFO:tasks.workunit.client.0.smithi154.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.ListLockersPP 2024-09-16T03:44:05.222 INFO:tasks.workunit.client.0.smithi154.stdout: api_lock_pp: [ OK ] LibRadosLockPP.ListLockersPP (15 ms) 2024-09-16T03:44:05.222 INFO:tasks.workunit.client.0.smithi154.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.BreakLockPP 2024-09-16T03:44:05.222 INFO:tasks.workunit.client.0.smithi154.stdout: api_lock_pp: [ OK ] LibRadosLockPP.BreakLockPP (8 ms) 2024-09-16T03:44:05.222 INFO:tasks.workunit.client.0.smithi154.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockPP (2418 ms total) 2024-09-16T03:44:05.222 INFO:tasks.workunit.client.0.smithi154.stdout: api_lock_pp: 2024-09-16T03:44:05.222 INFO:tasks.workunit.client.0.smithi154.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockECPP 2024-09-16T03:44:05.222 INFO:tasks.workunit.client.0.smithi154.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockExclusivePP 2024-09-16T03:44:05.222 INFO:tasks.workunit.client.0.smithi154.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockExclusivePP (1193 ms) 2024-09-16T03:44:05.222 INFO:tasks.workunit.client.0.smithi154.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockSharedPP 2024-09-16T03:44:05.222 INFO:tasks.workunit.client.0.smithi154.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockSharedPP (12 ms) 2024-09-16T03:44:05.223 INFO:tasks.workunit.client.0.smithi154.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockExclusiveDurPP 2024-09-16T03:44:05.223 INFO:tasks.workunit.client.0.smithi154.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockExclusiveDurPP (1005 ms) 2024-09-16T03:44:05.223 INFO:tasks.workunit.client.0.smithi154.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockSharedDurPP 2024-09-16T03:44:05.223 INFO:tasks.workunit.client.0.smithi154.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockSharedDurPP (1013 ms) 2024-09-16T03:44:05.223 INFO:tasks.workunit.client.0.smithi154.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockMayRenewPP 2024-09-16T03:44:05.223 INFO:tasks.workunit.client.0.smithi154.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockMayRenewPP (13 ms) 2024-09-16T03:44:05.223 INFO:tasks.workunit.client.0.smithi154.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.UnlockPP 2024-09-16T03:44:05.223 INFO:tasks.workunit.client.0.smithi154.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.UnlockPP (14 ms) 2024-09-16T03:44:05.223 INFO:tasks.workunit.client.0.smithi154.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.ListLockersPP 2024-09-16T03:44:05.223 INFO:tasks.workunit.client.0.smithi154.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.ListLockersPP (17 ms) 2024-09-16T03:44:05.223 INFO:tasks.workunit.client.0.smithi154.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.BreakLockPP 2024-09-16T03:44:05.223 INFO:tasks.workunit.client.0.smithi154.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.BreakLockPP (9 ms) 2024-09-16T03:44:05.223 INFO:tasks.workunit.client.0.smithi154.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockECPP (3277 ms total) 2024-09-16T03:44:05.223 INFO:tasks.workunit.client.0.smithi154.stdout: api_lock_pp: 2024-09-16T03:44:05.224 INFO:tasks.workunit.client.0.smithi154.stdout: api_lock_pp: [----------] Global test environment tear-down 2024-09-16T03:44:05.224 INFO:tasks.workunit.client.0.smithi154.stdout: api_lock_pp: [==========] 16 tests from 2 test suites ran. (35618 ms total) 2024-09-16T03:44:05.224 INFO:tasks.workunit.client.0.smithi154.stdout: api_lock_pp: [ PASSED ] 16 tests. 2024-09-16T03:44:05.227 INFO:tasks.workunit.client.0.smithi154.stdout: api_lock: Running main() from gmock_main.cc 2024-09-16T03:44:05.227 INFO:tasks.workunit.client.0.smithi154.stdout: api_lock: [==========] Running 16 tests from 2 test suites. 2024-09-16T03:44:05.227 INFO:tasks.workunit.client.0.smithi154.stdout: api_lock: [----------] Global test environment set-up. 2024-09-16T03:44:05.227 INFO:tasks.workunit.client.0.smithi154.stdout: api_lock: [----------] 8 tests from LibRadosLock 2024-09-16T03:44:05.227 INFO:tasks.workunit.client.0.smithi154.stdout: api_lock: [ RUN ] LibRadosLock.LockExclusive 2024-09-16T03:44:05.227 INFO:tasks.workunit.client.0.smithi154.stdout: api_lock: [ OK ] LibRadosLock.LockExclusive (294 ms) 2024-09-16T03:44:05.227 INFO:tasks.workunit.client.0.smithi154.stdout: api_lock: [ RUN ] LibRadosLock.LockShared 2024-09-16T03:44:05.227 INFO:tasks.workunit.client.0.smithi154.stdout: api_lock: [ OK ] LibRadosLock.LockShared (3 ms) 2024-09-16T03:44:05.228 INFO:tasks.workunit.client.0.smithi154.stdout: api_lock: [ RUN ] LibRadosLock.LockExclusiveDur 2024-09-16T03:44:05.228 INFO:tasks.workunit.client.0.smithi154.stdout: api_lock: [ OK ] LibRadosLock.LockExclusiveDur (1032 ms) 2024-09-16T03:44:05.228 INFO:tasks.workunit.client.0.smithi154.stdout: api_lock: [ RUN ] LibRadosLock.LockSharedDur 2024-09-16T03:44:05.228 INFO:tasks.workunit.client.0.smithi154.stdout: api_lock: [ OK ] LibRadosLock.LockSharedDur (1009 ms) 2024-09-16T03:44:05.228 INFO:tasks.workunit.client.0.smithi154.stdout: api_lock: [ RUN ] LibRadosLock.LockMayRenew 2024-09-16T03:44:05.228 INFO:tasks.workunit.client.0.smithi154.stdout: api_lock: [ OK ] LibRadosLock.LockMayRenew (15 ms) 2024-09-16T03:44:05.228 INFO:tasks.workunit.client.0.smithi154.stdout: api_lock: [ RUN ] LibRadosLock.Unlock 2024-09-16T03:44:05.228 INFO:tasks.workunit.client.0.smithi154.stdout: api_lock: [ OK ] LibRadosLock.Unlock (16 ms) 2024-09-16T03:44:05.228 INFO:tasks.workunit.client.0.smithi154.stdout: api_lock: [ RUN ] LibRadosLock.ListLockers 2024-09-16T03:44:05.228 INFO:tasks.workunit.client.0.smithi154.stdout: api_lock: [ OK ] LibRadosLock.ListLockers (15 ms) 2024-09-16T03:44:05.228 INFO:tasks.workunit.client.0.smithi154.stdout: api_lock: [ RUN ] LibRadosLock.BreakLock 2024-09-16T03:44:05.228 INFO:tasks.workunit.client.0.smithi154.stdout: api_lock: [ OK ] LibRadosLock.BreakLock (9 ms) 2024-09-16T03:44:05.228 INFO:tasks.workunit.client.0.smithi154.stdout: api_lock: [----------] 8 tests from LibRadosLock (2393 ms total) 2024-09-16T03:44:05.228 INFO:tasks.workunit.client.0.smithi154.stdout: api_lock: 2024-09-16T03:44:05.229 INFO:tasks.workunit.client.0.smithi154.stdout: api_lock: [----------] 8 tests from LibRadosLockEC 2024-09-16T03:44:05.229 INFO:tasks.workunit.client.0.smithi154.stdout: api_lock: [ RUN ] LibRadosLockEC.LockExclusive 2024-09-16T03:44:05.229 INFO:tasks.workunit.client.0.smithi154.stdout: api_lock: [ OK ] LibRadosLockEC.LockExclusive (1180 ms) 2024-09-16T03:44:05.229 INFO:tasks.workunit.client.0.smithi154.stdout: api_lock: [ RUN ] LibRadosLockEC.LockShared 2024-09-16T03:44:05.229 INFO:tasks.workunit.client.0.smithi154.stdout: api_lock: [ OK ] LibRadosLockEC.LockShared (9 ms) 2024-09-16T03:44:05.229 INFO:tasks.workunit.client.0.smithi154.stdout: api_lock: [ RUN ] LibRadosLockEC.LockExclusiveDur 2024-09-16T03:44:05.229 INFO:tasks.workunit.client.0.smithi154.stdout: api_lock: [ OK ] LibRadosLockEC.LockExclusiveDur (1006 ms) 2024-09-16T03:44:05.229 INFO:tasks.workunit.client.0.smithi154.stdout: api_lock: [ RUN ] LibRadosLockEC.LockSharedDur 2024-09-16T03:44:05.229 INFO:tasks.workunit.client.0.smithi154.stdout: api_lock: [ OK ] LibRadosLockEC.LockSharedDur (1013 ms) 2024-09-16T03:44:05.229 INFO:tasks.workunit.client.0.smithi154.stdout: api_lock: [ RUN ] LibRadosLockEC.LockMayRenew 2024-09-16T03:44:05.229 INFO:tasks.workunit.client.0.smithi154.stdout: api_lock: [ OK ] LibRadosLockEC.LockMayRenew (19 ms) 2024-09-16T03:44:05.229 INFO:tasks.workunit.client.0.smithi154.stdout: api_lock: [ RUN ] LibRadosLockEC.Unlock 2024-09-16T03:44:05.229 INFO:tasks.workunit.client.0.smithi154.stdout: api_lock: [ OK ] LibRadosLockEC.Unlock (11 ms) 2024-09-16T03:44:05.229 INFO:tasks.workunit.client.0.smithi154.stdout: api_lock: [ RUN ] LibRadosLockEC.ListLockers 2024-09-16T03:44:05.229 INFO:tasks.workunit.client.0.smithi154.stdout: api_lock: [ OK ] LibRadosLockEC.ListLockers (15 ms) 2024-09-16T03:44:05.230 INFO:tasks.workunit.client.0.smithi154.stdout: api_lock: [ RUN ] LibRadosLockEC.BreakLock 2024-09-16T03:44:05.230 INFO:tasks.workunit.client.0.smithi154.stdout: api_lock: [ OK ] LibRadosLockEC.BreakLock (10 ms) 2024-09-16T03:44:05.230 INFO:tasks.workunit.client.0.smithi154.stdout: api_lock: [----------] 8 tests from LibRadosLockEC (3263 ms total) 2024-09-16T03:44:05.230 INFO:tasks.workunit.client.0.smithi154.stdout: api_lock: 2024-09-16T03:44:05.230 INFO:tasks.workunit.client.0.smithi154.stdout: api_lock: [----------] Global test environment tear-down 2024-09-16T03:44:05.230 INFO:tasks.workunit.client.0.smithi154.stdout: api_lock: [==========] 16 tests from 2 test suites ran. (35644 ms total) 2024-09-16T03:44:05.230 INFO:tasks.workunit.client.0.smithi154.stdout: api_lock: [ PASSED ] 16 tests. 2024-09-16T03:44:06.183 INFO:tasks.workunit.client.0.smithi154.stdout: ec_io: Running main() from gmock_main.cc 2024-09-16T03:44:06.183 INFO:tasks.workunit.client.0.smithi154.stdout: ec_io: [==========] Running 2 tests from 1 test suite. 2024-09-16T03:44:06.183 INFO:tasks.workunit.client.0.smithi154.stdout: ec_io: [----------] Global test environment set-up. 2024-09-16T03:44:06.184 INFO:tasks.workunit.client.0.smithi154.stdout: ec_io: [----------] 2 tests from NeoRadosECIo 2024-09-16T03:44:06.184 INFO:tasks.workunit.client.0.smithi154.stdout: ec_io: [ RUN ] NeoRadosECIo.SimpleWrite 2024-09-16T03:44:06.184 INFO:tasks.workunit.client.0.smithi154.stdout: ec_io: [ OK ] NeoRadosECIo.SimpleWrite (29504 ms) 2024-09-16T03:44:06.184 INFO:tasks.workunit.client.0.smithi154.stdout: ec_io: [ RUN ] NeoRadosECIo.ReadOp 2024-09-16T03:44:06.184 INFO:tasks.workunit.client.0.smithi154.stdout: ec_io: [ OK ] NeoRadosECIo.ReadOp (6890 ms) 2024-09-16T03:44:06.184 INFO:tasks.workunit.client.0.smithi154.stdout: ec_io: [----------] 2 tests from NeoRadosECIo (36395 ms total) 2024-09-16T03:44:06.184 INFO:tasks.workunit.client.0.smithi154.stdout: ec_io: 2024-09-16T03:44:06.184 INFO:tasks.workunit.client.0.smithi154.stdout: ec_io: [----------] Global test environment tear-down 2024-09-16T03:44:06.184 INFO:tasks.workunit.client.0.smithi154.stdout: ec_io: [==========] 2 tests from 1 test suite ran. (36395 ms total) 2024-09-16T03:44:06.184 INFO:tasks.workunit.client.0.smithi154.stdout: ec_io: [ PASSED ] 2 tests. 2024-09-16T03:44:06.186 INFO:tasks.workunit.client.0.smithi154.stdout: pool: Running main() from gmock_main.cc 2024-09-16T03:44:06.186 INFO:tasks.workunit.client.0.smithi154.stdout: pool: [==========] Running 6 tests from 1 test suite. 2024-09-16T03:44:06.186 INFO:tasks.workunit.client.0.smithi154.stdout: pool: [----------] Global test environment set-up. 2024-09-16T03:44:06.186 INFO:tasks.workunit.client.0.smithi154.stdout: pool: [----------] 6 tests from NeoRadosPools 2024-09-16T03:44:06.186 INFO:tasks.workunit.client.0.smithi154.stdout: pool: [ RUN ] NeoRadosPools.PoolList 2024-09-16T03:44:06.186 INFO:tasks.workunit.client.0.smithi154.stdout: pool: [ OK ] NeoRadosPools.PoolList (3498 ms) 2024-09-16T03:44:06.186 INFO:tasks.workunit.client.0.smithi154.stdout: pool: [ RUN ] NeoRadosPools.PoolLookup 2024-09-16T03:44:06.186 INFO:tasks.workunit.client.0.smithi154.stdout: pool: [ OK ] NeoRadosPools.PoolLookup (22976 ms) 2024-09-16T03:44:06.187 INFO:tasks.workunit.client.0.smithi154.stdout: pool: [ RUN ] NeoRadosPools.PoolLookupOtherInstance 2024-09-16T03:44:06.187 INFO:tasks.workunit.client.0.smithi154.stdout: pool: [ OK ] NeoRadosPools.PoolLookupOtherInstance (2177 ms) 2024-09-16T03:44:06.187 INFO:tasks.workunit.client.0.smithi154.stdout: pool: [ RUN ] NeoRadosPools.PoolDelete 2024-09-16T03:44:06.187 INFO:tasks.workunit.client.0.smithi154.stdout: pool: [ OK ] NeoRadosPools.PoolDelete (3787 ms) 2024-09-16T03:44:06.187 INFO:tasks.workunit.client.0.smithi154.stdout: pool: [ RUN ] NeoRadosPools.PoolCreateDelete 2024-09-16T03:44:06.187 INFO:tasks.workunit.client.0.smithi154.stdout: pool: [ OK ] NeoRadosPools.PoolCreateDelete (1968 ms) 2024-09-16T03:44:06.187 INFO:tasks.workunit.client.0.smithi154.stdout: pool: [ RUN ] NeoRadosPools.PoolCreateWithCrushRule 2024-09-16T03:44:06.187 INFO:tasks.workunit.client.0.smithi154.stdout: pool: [ OK ] NeoRadosPools.PoolCreateWithCrushRule (1968 ms) 2024-09-16T03:44:06.187 INFO:tasks.workunit.client.0.smithi154.stdout: pool: [----------] 6 tests from NeoRadosPools (36374 ms total) 2024-09-16T03:44:06.187 INFO:tasks.workunit.client.0.smithi154.stdout: pool: 2024-09-16T03:44:06.187 INFO:tasks.workunit.client.0.smithi154.stdout: pool: [----------] Global test environment tear-down 2024-09-16T03:44:06.187 INFO:tasks.workunit.client.0.smithi154.stdout: pool: [==========] 6 tests from 1 test suite ran. (36374 ms total) 2024-09-16T03:44:06.187 INFO:tasks.workunit.client.0.smithi154.stdout: pool: [ PASSED ] 6 tests. 2024-09-16T03:44:09.391 INFO:tasks.workunit.client.0.smithi154.stdout: api_watch_notify_pp: Running main() from gmock_main.cc 2024-09-16T03:44:09.391 INFO:tasks.workunit.client.0.smithi154.stdout: api_watch_notify_pp: [==========] Running 16 tests from 2 test suites. 2024-09-16T03:44:09.391 INFO:tasks.workunit.client.0.smithi154.stdout: api_watch_notify_pp: [----------] Global test environment set-up. 2024-09-16T03:44:09.391 INFO:tasks.workunit.client.0.smithi154.stdout: api_watch_notify_pp: [----------] 2 tests from LibRadosWatchNotifyECPP 2024-09-16T03:44:09.391 INFO:tasks.workunit.client.0.smithi154.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyECPP.WatchNotify 2024-09-16T03:44:09.392 INFO:tasks.workunit.client.0.smithi154.stdout: api_watch_notify_pp: notify 2024-09-16T03:44:09.392 INFO:tasks.workunit.client.0.smithi154.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyECPP.WatchNotify (1204 ms) 2024-09-16T03:44:09.392 INFO:tasks.workunit.client.0.smithi154.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyECPP.WatchNotifyTimeout 2024-09-16T03:44:09.392 INFO:tasks.workunit.client.0.smithi154.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyECPP.WatchNotifyTimeout (20 ms) 2024-09-16T03:44:09.392 INFO:tasks.workunit.client.0.smithi154.stdout: api_watch_notify_pp: [----------] 2 tests from LibRadosWatchNotifyECPP (1224 ms total) 2024-09-16T03:44:09.392 INFO:tasks.workunit.client.0.smithi154.stdout: api_watch_notify_pp: 2024-09-16T03:44:09.392 INFO:tasks.workunit.client.0.smithi154.stdout: api_watch_notify_pp: [----------] 14 tests from LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP 2024-09-16T03:44:09.392 INFO:tasks.workunit.client.0.smithi154.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/0 2024-09-16T03:44:09.392 INFO:tasks.workunit.client.0.smithi154.stdout: api_watch_notify_pp: notify 2024-09-16T03:44:09.392 INFO:tasks.workunit.client.0.smithi154.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/0 (117 ms) 2024-09-16T03:44:09.392 INFO:tasks.workunit.client.0.smithi154.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/1 2024-09-16T03:44:09.392 INFO:tasks.workunit.client.0.smithi154.stdout: api_watch_notify_pp: notify 2024-09-16T03:44:09.392 INFO:tasks.workunit.client.0.smithi154.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/1 (4041 ms) 2024-09-16T03:44:09.393 INFO:tasks.workunit.client.0.smithi154.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/0 2024-09-16T03:44:09.393 INFO:tasks.workunit.client.0.smithi154.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/0 (5 ms) 2024-09-16T03:44:09.393 INFO:tasks.workunit.client.0.smithi154.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/1 2024-09-16T03:44:09.393 INFO:tasks.workunit.client.0.smithi154.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/1 (4 ms) 2024-09-16T03:44:09.393 INFO:tasks.workunit.client.0.smithi154.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/0 2024-09-16T03:44:09.393 INFO:tasks.workunit.client.0.smithi154.stdout: api_watch_notify_pp: handle_notify cookie 94745268853024 notify_id 141733920771 notifier_gid 4865 2024-09-16T03:44:09.393 INFO:tasks.workunit.client.0.smithi154.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/0 (5 ms) 2024-09-16T03:44:09.393 INFO:tasks.workunit.client.0.smithi154.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/1 2024-09-16T03:44:09.393 INFO:tasks.workunit.client.0.smithi154.stdout: api_watch_notify_pp: handle_notify cookie 94745268853024 notify_id 141733920772 notifier_gid 4865 2024-09-16T03:44:09.393 INFO:tasks.workunit.client.0.smithi154.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/1 (7 ms) 2024-09-16T03:44:09.393 INFO:tasks.workunit.client.0.smithi154.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/0 2024-09-16T03:44:09.393 INFO:tasks.workunit.client.0.smithi154.stdout: api_watch_notify_pp: handle_notify cookie 94745269675424 notify_id 141733920771 notifier_gid 4865 2024-09-16T03:44:09.393 INFO:tasks.workunit.client.0.smithi154.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/0 (7 ms) 2024-09-16T03:44:09.393 INFO:tasks.workunit.client.0.smithi154.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/1 2024-09-16T03:44:09.394 INFO:tasks.workunit.client.0.smithi154.stdout: api_watch_notify_pp: handle_notify cookie 94745269675424 notify_id 141733920773 notifier_gid 4865 2024-09-16T03:44:09.394 INFO:tasks.workunit.client.0.smithi154.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/1 (6 ms) 2024-09-16T03:44:09.394 INFO:tasks.workunit.client.0.smithi154.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/0 2024-09-16T03:44:09.394 INFO:tasks.workunit.client.0.smithi154.stdout: api_watch_notify_pp: handle_notify cookie 94745269675424 notify_id 141733920774 notifier_gid 4865 2024-09-16T03:44:09.394 INFO:tasks.workunit.client.0.smithi154.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/0 (7 ms) 2024-09-16T03:44:09.394 INFO:tasks.workunit.client.0.smithi154.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/1 2024-09-16T03:44:09.394 INFO:tasks.workunit.client.0.smithi154.stdout: api_watch_notify_pp: handle_notify cookie 94745269682944 notify_id 141733920772 notifier_gid 4865 2024-09-16T03:44:09.394 INFO:tasks.workunit.client.0.smithi154.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/1 (5 ms) 2024-09-16T03:44:09.394 INFO:tasks.workunit.client.0.smithi154.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/0 2024-09-16T03:44:09.394 INFO:tasks.workunit.client.0.smithi154.stdout: api_watch_notify_pp: trying... 2024-09-16T03:44:09.394 INFO:tasks.workunit.client.0.smithi154.stdout: api_watch_notify_pp: handle_notify cookie 94745269682944 notify_id 141733920775 notifier_gid 4865 2024-09-16T03:44:09.394 INFO:tasks.workunit.client.0.smithi154.stdout: api_watch_notify_pp: timed out 2024-09-16T03:44:09.394 INFO:tasks.workunit.client.0.smithi154.stdout: api_watch_notify_pp: flushing 2024-09-16T03:44:17.084 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-16T03:44:17.084 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-09-16T03:44:17.085 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-16T03:44:17.085 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-16T03:44:17.087 INFO:tasks.ceph.mon.b:Started 2024-09-16T03:44:17.088 INFO:tasks.mon_thrash:Sending STOP to mon c 2024-09-16T03:44:17.088 INFO:tasks.ceph.mon.c:Sent signal 19 2024-09-16T03:44:17.088 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-16T03:44:17.976 INFO:tasks.workunit.client.0.smithi154.stdout: api_watch_noti unknown 2024-09-16T03:44:17.976 INFO:tasks.workunit.client.0.smithi154.stdout: api_service: 9 active+clean 2024-09-16T03:44:17.976 INFO:tasks.workunit.client.0.smithi154.stdout: api_service: 2024-09-16T03:44:17.976 INFO:tasks.workunit.client.0.smithi154.stdout: api_service: 2024-09-16T03:44:17.976 INFO:tasks.workunit.client.0.smithi154.stdout: api_service: cluster: 2024-09-16T03:44:17.976 INFO:tasks.workunit.client.0.smithi154.stdout: api_service: id: 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:44:17.976 INFO:tasks.workunit.client.0.smithi154.stdout: api_service: health: HEALTH_WARN 2024-09-16T03:44:17.976 INFO:tasks.workunit.client.0.smithi154.stdout: api_service: 1/3 mons down, quorum a,c 2024-09-16T03:44:17.976 INFO:tasks.workunit.client.0.smithi154.stdout: api_service: 15 pool(s) do not have an application enabled 2024-09-16T03:44:17.976 INFO:tasks.workunit.client.0.smithi154.stdout: api_service: 2024-09-16T03:44:17.976 INFO:tasks.workunit.client.0.smithi154.stdout: api_service: services: 2024-09-16T03:44:17.976 INFO:tasks.workunit.client.0.smithi154.stdout: api_service: mon: 3 daemons, quorum a,c (age 0.623812s), out of quorum: b 2024-09-16T03:44:17.977 INFO:tasks.workunit.client.0.smithi154.stdout: api_service: mgr: x(active, since 104s) 2024-09-16T03:44:17.977 INFO:tasks.workunit.client.0.smithi154.stdout: api_service: osd: 6 osds: 6 up (since 98s), 6 in (since 107s) 2024-09-16T03:44:17.977 INFO:tasks.workunit.client.0.smithi154.stdout: api_service: laundry: 2 daemons active (1 hosts) 2024-09-16T03:44:17.977 INFO:tasks.workunit.client.0.smithi154.stdout: api_service: 2024-09-16T03:44:17.977 INFO:tasks.workunit.client.0.smithi154.stdout: api_service: data: 2024-09-16T03:44:17.977 INFO:tasks.workunit.client.0.smithi154.stdout: api_service: pools: 35 pools, 1065 pgs 2024-09-16T03:44:17.977 INFO:tasks.workunit.client.0.smithi154.stdout: api_service: objects: 4 objects, 577 KiB 2024-09-16T03:44:17.977 INFO:tasks.workunit.client.0.smithi154.stdout: api_service: usage: 164 MiB used, 540 GiB / 540 GiB avail 2024-09-16T03:44:17.977 INFO:tasks.workunit.client.0.smithi154.stdout: api_service: pgs: 39.812% pgs unknown 2024-09-16T03:44:17.977 INFO:tasks.workunit.client.0.smithi154.stdout: api_service: 59.343% pgs not active 2024-09-16T03:44:17.977 INFO:tasks.workunit.client.0.smithi154.stdout: api_service: 632 creating+peering 2024-09-16T03:44:17.977 INFO:tasks.workunit.client.0.smithi154.stdout: api_service: 424 unknown 2024-09-16T03:44:17.977 INFO:tasks.workunit.client.0.smithi154.stdout: api_service: 9 active+clean 2024-09-16T03:44:17.977 INFO:tasks.workunit.client.0.smithi154.stdout: api_service: 2024-09-16T03:44:17.978 INFO:tasks.workunit.client.0.smithi154.stdout: api_service: 2024-09-16T03:44:17.978 INFO:tasks.workunit.client.0.smithi154.stdout: api_service: cluster: 2024-09-16T03:44:17.978 INFO:tasks.workunit.client.0.smithi154.stdout: api_service: id: 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:44:17.978 INFO:tasks.workunit.client.0.smithi154.stdout: api_service: health: HEALTH_WARN 2024-09-16T03:44:17.978 INFO:tasks.workunit.client.0.smithi154.stdout: api_service: 1/3 mons down, quorum a,c 2024-09-16T03:44:17.978 INFO:tasks.workunit.client.0.smithi154.stdout: api_service: 8 pool(s) do not have an application enabled 2024-09-16T03:44:17.978 INFO:tasks.workunit.client.0.smithi154.stdout: api_service: 2024-09-16T03:44:17.978 INFO:tasks.workunit.client.0.smithi154.stdout: api_service: services: 2024-09-16T03:44:17.978 INFO:tasks.workunit.client.0.smithi154.stdout: api_service: mon: 3 daemons, quorum a,c (age 2s), out of quorum: b 2024-09-16T03:44:17.978 INFO:tasks.workunit.client.0.smithi154.stdout: api_service: mgr: x(active, since 106s) 2024-09-16T03:44:17.978 INFO:tasks.workunit.client.0.smithi154.stdout: api_service: osd: 6 osds: 6 up (since 100s), 6 in (since 109s) 2024-09-16T03:44:17.978 INFO:tasks.workunit.client.0.smithi154.stdout: api_service: foo: 16 portals active (1 hosts, 3 zones) 2024-09-16T03:44:17.978 INFO:tasks.workunit.client.0.smithi154.stdout: api_service: laundry: 1 daemon active (1 hosts) 2024-09-16T03:44:17.978 INFO:tasks.workunit.client.0.smithi154.stdout: api_service: 2024-09-16T03:44:17.979 INFO:tasks.workunit.client.0.smithi154.stdout: api_service: data: 2024-09-16T03:44:17.979 INFO:tasks.workunit.client.0.smithi154.stdout: api_service: pools: 20 pools, 585 pgs 2024-09-16T03:44:17.979 INFO:tasks.workunit.client.0.smithi154.stdout: api_service: objects: 228 objects, 145 MiB 2024-09-16T03:44:17.979 INFO:tasks.workunit.client.0.smithi154.stdout: api_service: usage: 186 MiB used, 540 GiB / 540 GiB avail 2024-09-16T03:44:17.979 INFO:tasks.workunit.client.0.smithi154.stdout: api_service: pgs: 21.880% pgs not active 2024-09-16T03:44:17.979 INFO:tasks.workunit.client.0.smithi154.stdout: api_service: 457 active+clean 2024-09-16T03:44:17.979 INFO:tasks.workunit.client.0.smithi154.stdout: api_service: 128 creating+peering 2024-09-16T03:44:17.979 INFO:tasks.workunit.client.0.smithi154.stdout: api_service: 2024-09-16T03:44:17.979 INFO:tasks.workunit.client.0.smithi154.stdout: api_service: io: 2024-09-16T03:44:17.979 INFO:tasks.workunit.client.0.smithi154.stdout: api_service: client: 85 B/s wr, 0 op/s rd, 0 op/s wr 2024-09-16T03:44:17.979 INFO:tasks.workunit.client.0.smithi154.stdout: api_service: 2024-09-16T03:44:17.979 INFO:tasks.workunit.client.0.smithi154.stdout: api_service: 2024-09-16T03:44:17.979 INFO:tasks.workunit.client.0.smithi154.stdout: api_service: [ OK ] LibRadosService.StatusFormat (21531 ms) 2024-09-16T03:44:17.979 INFO:tasks.workunit.client.0.smithi154.stdout: api_service: [ RUN ] LibRadosService.Status 2024-09-16T03:44:17.980 INFO:tasks.workunit.client.0.smithi154.stdout: api_service: [ OK ] LibRadosService.Status (20051 ms) 2024-09-16T03:44:17.980 INFO:tasks.workunit.client.0.smithi154.stdout: api_service: [----------] 4 tests from LibRadosService (48257 ms total) 2024-09-16T03:44:17.980 INFO:tasks.workunit.client.0.smithi154.stdout: api_service: 2024-09-16T03:44:17.980 INFO:tasks.workunit.client.0.smithi154.stdout: api_service: [----------] Global test environment tear-down 2024-09-16T03:44:17.980 INFO:tasks.workunit.client.0.smithi154.stdout: api_service: [==========] 4 tests from 1 test suite ran. (48257 ms total) 2024-09-16T03:44:17.980 INFO:tasks.workunit.client.0.smithi154.stdout: api_service: [ PASSED ] 4 tests. 2024-09-16T03:44:23.141 INFO:tasks.workunit.client.0.smithi154.stdout: ec_list: Running main() from gmock_main.cc 2024-09-16T03:44:23.141 INFO:tasks.workunit.client.0.smithi154.stdout: ec_list: [==========] Running 3 tests from 1 test suite. 2024-09-16T03:44:23.142 INFO:tasks.workunit.client.0.smithi154.stdout: ec_list: [----------] Global test environment set-up. 2024-09-16T03:44:23.142 INFO:tasks.workunit.client.0.smithi154.stdout: ec_list: [----------] 3 tests from NeoradosECList 2024-09-16T03:44:23.142 INFO:tasks.workunit.client.0.smithi154.stdout: ec_list: [ RUN ] NeoradosECList.ListObjects 2024-09-16T03:44:23.142 INFO:tasks.workunit.client.0.smithi154.stdout: ec_list: [ OK ] NeoradosECList.ListObjects (30500 ms) 2024-09-16T03:44:23.142 INFO:tasks.workunit.client.0.smithi154.stdout: ec_list: [ RUN ] NeoradosECList.ListObjectsNS 2024-09-16T03:44:23.142 INFO:tasks.workunit.client.0.smithi154.stdout: ec_list: [ OK ] NeoradosECList.ListObjectsNS (6872 ms) 2024-09-16T03:44:23.142 INFO:tasks.workunit.client.0.smithi154.stdout: ec_list: [ RUN ] NeoradosECList.ListObjectsMany 2024-09-16T03:44:23.142 INFO:tasks.workunit.client.0.smithi154.stdout: ec_list: [ OK ] NeoradosECList.ListObjectsMany (15963 ms) 2024-09-16T03:44:23.142 INFO:tasks.workunit.client.0.smithi154.stdout: ec_list: [----------] 3 tests from NeoradosECList (53335 ms total) 2024-09-16T03:44:23.142 INFO:tasks.workunit.client.0.smithi154.stdout: ec_list: 2024-09-16T03:44:23.142 INFO:tasks.workunit.client.0.smithi154.stdout: ec_list: [----------] Global test environment tear-down 2024-09-16T03:44:23.142 INFO:tasks.workunit.client.0.smithi154.stdout: ec_list: [==========] 3 tests from 1 test suite ran. (53335 ms total) 2024-09-16T03:44:23.143 INFO:tasks.workunit.client.0.smithi154.stdout: ec_list: [ PASSED ] 3 tests. 2024-09-16T03:44:32.089 INFO:tasks.mon_thrash:Sending CONT to mon c 2024-09-16T03:44:32.090 INFO:tasks.ceph.mon.c:Sent signal 18 2024-09-16T03:44:32.090 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T03:44:32.090 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:44:32.249 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.245+0000 7ff4d9f2c640 10 monclient: get_monmap_and_config 2024-09-16T03:44:32.249 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.245+0000 7ff4d9f2c640 10 monclient: build_initial_monmap 2024-09-16T03:44:32.249 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.245+0000 7ff4d9f2c640 10 monclient: monmap: 2024-09-16T03:44:32.250 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:44:32.250 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:44:32.250 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:44:32.248568+0000 2024-09-16T03:44:32.250 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:44:32.248568+0000 2024-09-16T03:44:32.250 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:44:32.250 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:44:32.250 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:44:32.250 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:44:32.250 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:44:32.250 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:44:32.250 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:44:32.250 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:44:32.251 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.245+0000 7ff4d9f2c640 1 Processor -- start 2024-09-16T03:44:32.251 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.245+0000 7ff4d9f2c640 1 -- start start 2024-09-16T03:44:32.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.245+0000 7ff4d9f2c640 10 monclient: init 2024-09-16T03:44:32.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.245+0000 7ff4d9f2c640 10 monclient: _reopen_session rank -1 2024-09-16T03:44:32.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.245+0000 7ff4d9f2c640 10 monclient: _add_conns ranks=[0,4,1,3,2] 2024-09-16T03:44:32.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.245+0000 7ff4d9f2c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff4d40a39b0 addr v1:172.21.15.42:6789/0 2024-09-16T03:44:32.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.245+0000 7ff4d9f2c640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7ff4d40a3d90 0x7ff4d413c380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:44:32.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.245+0000 7ff4d9f2c640 10 monclient(hunting): picked mon.noname-c con 0x7ff4d40a3d90 addr v2:172.21.15.154:3300/0 2024-09-16T03:44:32.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.245+0000 7ff4d9f2c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff4d413c8c0 addr v1:172.21.15.154:6789/0 2024-09-16T03:44:32.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.245+0000 7ff4d9f2c640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff4d4134290 con 0x7ff4d40a39b0 2024-09-16T03:44:32.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.245+0000 7ff4d9f2c640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff4d409c140 con 0x7ff4d413c8c0 2024-09-16T03:44:32.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.245+0000 7ff4d9f2c640 10 monclient(hunting): start opening mon connection 2024-09-16T03:44:32.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.245+0000 7ff4d9f2c640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7ff4d409c420 con 0x7ff4d40a3d90 2024-09-16T03:44:32.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.245+0000 7ff4d9f2c640 10 monclient(hunting): _renew_subs 2024-09-16T03:44:32.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.245+0000 7ff4d9f2c640 10 monclient(hunting): authenticate will time out at 1119.217822s 2024-09-16T03:44:32.253 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.245+0000 7ff4d37fe640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7ff4d40a39b0 0x7ff4d4133b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:49538/0 (socket says 172.21.15.42:49538) 2024-09-16T03:44:32.253 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.245+0000 7ff4d37fe640 1 -- 172.21.15.42:0/4076912177 learned_addr learned my addr 172.21.15.42:0/4076912177 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:44:32.276 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.269+0000 7ff4d27fc640 1 -- 172.21.15.42:0/4076912177 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2447926700 0 0) 0x7ff4d4134290 con 0x7ff4d40a39b0 2024-09-16T03:44:32.276 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.269+0000 7ff4d27fc640 10 monclient(hunting): _negotiate 2024-09-16T03:44:32.276 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.269+0000 7ff4d27fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:44:32.276 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.269+0000 7ff4d27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:44:32.276 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.269+0000 7ff4d27fc640 10 monclient(hunting): my global_id is 24576 2024-09-16T03:44:32.276 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.269+0000 7ff4d27fc640 1 -- 172.21.15.42:0/4076912177 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff4bc003530 con 0x7ff4d40a39b0 2024-09-16T03:44:32.279 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.273+0000 7ff4d27fc640 1 -- 172.21.15.42:0/4076912177 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 57313446 0 0) 0x7ff4bc003530 con 0x7ff4d40a39b0 2024-09-16T03:44:32.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.273+0000 7ff4d27fc640 1 -- 172.21.15.42:0/4076912177 >> v2:172.21.15.154:3300/0 conn(0x7ff4d40a3d90 msgr2=0x7ff4d413c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:44:32.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.273+0000 7ff4d27fc640 1 --2- 172.21.15.42:0/4076912177 >> v2:172.21.15.154:3300/0 conn(0x7ff4d40a3d90 0x7ff4d413c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:44:32.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.273+0000 7ff4d27fc640 1 -- 172.21.15.42:0/4076912177 >> v1:172.21.15.154:6789/0 conn(0x7ff4d413c8c0 legacy=0x7ff4d413ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:44:32.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.273+0000 7ff4d27fc640 10 monclient: _finish_hunting 0 2024-09-16T03:44:32.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.273+0000 7ff4d27fc640 1 monclient: found mon.noname-a-legacy 2024-09-16T03:44:32.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.273+0000 7ff4d27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:44:32.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.273+0000 7ff4d27fc640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.42:6789/0 2024-09-16T03:44:32.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.273+0000 7ff4d27fc640 1 -- 172.21.15.42:0/4076912177 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4d406dd60 con 0x7ff4d40a39b0 2024-09-16T03:44:32.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.273+0000 7ff4d27fc640 10 monclient: _finish_auth 0 2024-09-16T03:44:32.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.273+0000 7ff4d27fc640 10 monclient: _check_auth_tickets 2024-09-16T03:44:32.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.273+0000 7ff4d27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:44:32.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.273+0000 7ff4d27fc640 1 -- 172.21.15.42:0/4076912177 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2177413734 0 0) 0x7ff4c4001680 con 0x7ff4d40a39b0 2024-09-16T03:44:32.281 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.273+0000 7ff4d27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:44:32.281 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.273+0000 7ff4d27fc640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T03:44:32.281 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.273+0000 7ff4d27fc640 10 monclient: dump: 2024-09-16T03:44:32.281 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:44:32.281 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:44:32.281 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:32.281 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:32.281 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:44:32.281 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:44:32.281 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:44:32.281 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:44:32.281 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:44:32.281 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:44:32.281 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.273+0000 7ff4d27fc640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] but i'm connected to v1:172.21.15.42:6789/0, reconnecting 2024-09-16T03:44:32.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.273+0000 7ff4d27fc640 10 monclient: _reopen_session rank -1 2024-09-16T03:44:32.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.273+0000 7ff4d27fc640 1 -- 172.21.15.42:0/4076912177 >> v1:172.21.15.42:6789/0 conn(0x7ff4d40a39b0 legacy=0x7ff4d4133b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:44:32.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.273+0000 7ff4d27fc640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:44:32.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.273+0000 7ff4d27fc640 1 --2- 172.21.15.42:0/4076912177 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff4bc0039c0 0x7ff4bc003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:44:32.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.273+0000 7ff4d27fc640 10 monclient(hunting): picked mon.a con 0x7ff4bc0039c0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:44:32.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.273+0000 7ff4d27fc640 1 --2- 172.21.15.42:0/4076912177 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff4bc004810 0x7ff4bc004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:44:32.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.273+0000 7ff4d27fc640 10 monclient(hunting): picked mon.c con 0x7ff4bc004810 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:44:32.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.273+0000 7ff4d27fc640 1 --2- 172.21.15.42:0/4076912177 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff4bc0053d0 0x7ff4bc00ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:44:32.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.273+0000 7ff4d27fc640 10 monclient(hunting): picked mon.b con 0x7ff4bc0053d0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:44:32.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.273+0000 7ff4d27fc640 10 monclient(hunting): start opening mon connection 2024-09-16T03:44:32.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.273+0000 7ff4d27fc640 1 -- 172.21.15.42:0/4076912177 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7ff4d409c420 con 0x7ff4bc0039c0 2024-09-16T03:44:32.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.273+0000 7ff4d27fc640 10 monclient(hunting): start opening mon connection 2024-09-16T03:44:32.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.273+0000 7ff4d27fc640 1 -- 172.21.15.42:0/4076912177 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff4bc00c310 con 0x7ff4bc0053d0 2024-09-16T03:44:32.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.273+0000 7ff4d27fc640 10 monclient(hunting): start opening mon connection 2024-09-16T03:44:32.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.273+0000 7ff4d27fc640 1 -- 172.21.15.42:0/4076912177 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7ff4bc00c510 con 0x7ff4bc004810 2024-09-16T03:44:32.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.273+0000 7ff4d27fc640 10 monclient(hunting): _renew_subs 2024-09-16T03:44:32.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.273+0000 7ff4d27fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:44:32.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.273+0000 7ff4d2ffd640 1 --2- 172.21.15.42:0/4076912177 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff4bc004810 0x7ff4bc004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:44:32.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.273+0000 7ff4d2ffd640 10 monclient(hunting): get_auth_request con 0x7ff4bc004810 auth_method 0 2024-09-16T03:44:32.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.273+0000 7ff4d2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:44:32.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.273+0000 7ff4d2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:44:32.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.273+0000 7ff4d2ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:44:32.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.273+0000 7ff4d2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:44:32.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.273+0000 7ff4d2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:44:32.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.273+0000 7ff4d2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T03:44:32.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.273+0000 7ff4d2ffd640 10 monclient(hunting): handle_auth_done global_id 24576 payload 293 2024-09-16T03:44:32.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.273+0000 7ff4d2ffd640 1 -- 172.21.15.42:0/4076912177 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff4bc0053d0 msgr2=0x7ff4bc00ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:44:32.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.273+0000 7ff4d2ffd640 1 --2- 172.21.15.42:0/4076912177 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff4bc0053d0 0x7ff4bc00ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:44:32.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.273+0000 7ff4d2ffd640 1 -- 172.21.15.42:0/4076912177 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff4bc0039c0 msgr2=0x7ff4bc003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:44:32.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.273+0000 7ff4d2ffd640 1 --2- 172.21.15.42:0/4076912177 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff4bc0039c0 0x7ff4bc003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:44:32.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.273+0000 7ff4d2ffd640 10 monclient: _finish_hunting 0 2024-09-16T03:44:32.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.273+0000 7ff4d2ffd640 1 monclient: found mon.c 2024-09-16T03:44:32.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.273+0000 7ff4d2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:44:32.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.273+0000 7ff4d2ffd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:44:32.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.273+0000 7ff4d2ffd640 1 -- 172.21.15.42:0/4076912177 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4bc003530 con 0x7ff4bc004810 2024-09-16T03:44:32.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.273+0000 7ff4d2ffd640 10 monclient: _finish_auth 0 2024-09-16T03:44:32.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.273+0000 7ff4d2ffd640 10 monclient: _check_auth_tickets 2024-09-16T03:44:32.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.273+0000 7ff4d2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:44:32.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.273+0000 7ff4d3fff640 1 -- 172.21.15.42:0/4076912177 reap_dead start 2024-09-16T03:44:32.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.273+0000 7ff4d9f2c640 5 monclient: authenticate success, global_id 24576 2024-09-16T03:44:32.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.273+0000 7ff4d9f2c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:44:32.285 INFO:tasks.workunit.client.0.smithi154.stdout: api_c_write_operations: Running main() from gmock_main.cc 2024-09-16T03:44:32.285 INFO:tasks.workunit.client.0.smithi154.stdout: api_c_write_operations: [==========] Running 8 tests from 2 test suites. 2024-09-16T03:44:32.285 INFO:tasks.workunit.client.0.smithi154.stdout: api_c_write_operations: [----------] Global test environment set-up. 2024-09-16T03:44:32.285 INFO:tasks.workunit.client.0.smithi154.stdout: api_c_write_operations: [----------] 1 test from LibradosCWriteOps 2024-09-16T03:44:32.286 INFO:tasks.workunit.client.0.smithi154.stdout: api_c_write_operations: [ RUN ] LibradosCWriteOps.NewDelete 2024-09-16T03:44:32.286 INFO:tasks.workunit.client.0.smithi154.stdout: api_c_write_operations: [ OK ] LibradosCWriteOps.NewDelete (0 ms) 2024-09-16T03:44:32.286 INFO:tasks.workunit.client.0.smithi154.stdout: api_c_write_operations: [----------] 1 test from LibradosCWriteOps (0 ms total) 2024-09-16T03:44:32.286 INFO:tasks.workunit.client.0.smithi154.stdout: api_c_write_operations: 2024-09-16T03:44:32.286 INFO:tasks.workunit.client.0.smithi154.stdout: api_c_write_operations: [----------] 7 tests from LibRadosCWriteOps 2024-09-16T03:44:32.286 INFO:tasks.workunit.client.0.smithi154.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.assertExists 2024-09-16T03:44:32.286 INFO:tasks.workunit.client.0.smithi154.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.assertExists (4739 ms) 2024-09-16T03:44:32.286 INFO:tasks.workunit.client.0.smithi154.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.WriteOpAssertVersion 2024-09-16T03:44:32.286 INFO:tasks.workunit.client.0.smithi154.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.WriteOpAssertVersion (24016 ms) 2024-09-16T03:44:32.286 INFO:tasks.workunit.client.0.smithi154.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Xattrs 2024-09-16T03:44:32.286 INFO:tasks.workunit.client.0.smithi154.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Xattrs (2783 ms) 2024-09-16T03:44:32.286 INFO:tasks.workunit.client.0.smithi154.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Write 2024-09-16T03:44:32.286 INFO:tasks.workunit.client.0.smithi154.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Write (2978 ms) 2024-09-16T03:44:32.287 INFO:tasks.workunit.client.0.smithi154.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Exec 2024-09-16T03:44:32.287 INFO:tasks.workunit.client.0.smithi154.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Exec (2981 ms) 2024-09-16T03:44:32.287 INFO:tasks.workunit.client.0.smithi154.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.WriteSame 2024-09-16T03:44:32.287 INFO:tasks.workunit.client.0.smithi154.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.WriteSame (3026 ms) 2024-09-16T03:44:32.287 INFO:tasks.workunit.client.0.smithi154.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.CmpExt 2024-09-16T03:44:32.287 INFO:tasks.workunit.client.0.smithi154.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.CmpExt (22055 ms) 2024-09-16T03:44:32.287 INFO:tasks.workunit.client.0.smithi154.stdout: api_c_write_operations: [----------] 7 tests from LibRadosCWriteOps (62579 ms total) 2024-09-16T03:44:32.287 INFO:tasks.workunit.client.0.smithi154.stdout: api_c_write_operations: 2024-09-16T03:44:32.287 INFO:tasks.workunit.client.0.smithi154.stdout: api_c_write_operations: [----------] Global test environment tear-down 2024-09-16T03:44:32.287 INFO:tasks.workunit.client.0.smithi154.stdout: api_c_write_operations: [==========] 8 tests from 2 test suites ran. (62579 ms total) 2024-09-16T03:44:32.287 INFO:tasks.workunit.client.0.smithi154.stdout: api_c_write_operations: [ PASSED ] 8 tests. 2024-09-16T03:44:32.291 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.277+0000 7ff4d2ffd640 1 --2- 172.21.15.42:0/4076912177 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff4bc004810 0x7ff4bc004c90 secure :-1 s=READY pgs=1792 cs=0 l=1 rev1=1 crypto rx=0x7ff4c8009890 tx=0x7ff4c801c380 comp rx=0 tx=0).ready entity=mon.2 client_cookie=486f0b58c2e8e1fa server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:44:32.292 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.277+0000 7ff4d27fc640 1 -- 172.21.15.42:0/4076912177 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4c80b0070 con 0x7ff4bc004810 2024-09-16T03:44:32.292 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.277+0000 7ff4d27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:44:32.292 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.277+0000 7ff4d27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:44:32.292 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.277+0000 7ff4d27fc640 10 monclient: dump: 2024-09-16T03:44:32.292 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:44:32.292 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:44:32.292 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:32.292 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:32.292 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:44:32.292 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:44:32.292 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:44:32.292 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:44:32.292 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:44:32.293 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:44:32.293 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.277+0000 7ff4d9f2c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:44:32.293 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.277+0000 7ff4d27fc640 1 -- 172.21.15.42:0/4076912177 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4c80ab620 con 0x7ff4bc004810 2024-09-16T03:44:32.293 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.277+0000 7ff4d27fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:44:32.293 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.277+0000 7ff4d27fc640 1 -- 172.21.15.42:0/4076912177 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4c8009460 con 0x7ff4bc004810 2024-09-16T03:44:32.293 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.277+0000 7ff4d27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:44:32.293 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.277+0000 7ff4d27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:44:32.293 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.277+0000 7ff4d27fc640 10 monclient: dump: 2024-09-16T03:44:32.293 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:44:32.293 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:44:32.293 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:32.293 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:32.293 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:44:32.294 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:44:32.294 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:44:32.294 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:44:32.294 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:44:32.294 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:44:32.294 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.277+0000 7ff4d9f2c640 10 monclient: get_monmap_and_config success 2024-09-16T03:44:32.294 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.277+0000 7ff4d9f2c640 4 set_mon_vals no callback set 2024-09-16T03:44:32.294 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.277+0000 7ff4d9f2c640 10 monclient: shutdown 2024-09-16T03:44:32.294 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.277+0000 7ff4d9f2c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:44:32.294 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.277+0000 7ff4d9f2c640 1 -- 172.21.15.42:0/4076912177 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff4bc004810 msgr2=0x7ff4bc004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:44:32.294 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.277+0000 7ff4d9f2c640 1 --2- 172.21.15.42:0/4076912177 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff4bc004810 0x7ff4bc004c90 secure :-1 s=READY pgs=1792 cs=0 l=1 rev1=1 crypto rx=0x7ff4c8009890 tx=0x7ff4c801c380 comp rx=0 tx=0).stop 2024-09-16T03:44:32.294 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.281+0000 7ff4d9f2c640 1 -- 172.21.15.42:0/4076912177 shutdown_connections 2024-09-16T03:44:32.294 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.281+0000 7ff4d9f2c640 1 --2- 172.21.15.42:0/4076912177 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff4bc004810 0x7ff4bc004c90 unknown :-1 s=CLOSED pgs=1792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:44:32.294 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.281+0000 7ff4d9f2c640 1 -- 172.21.15.42:0/4076912177 >> 172.21.15.42:0/4076912177 conn(0x7ff4d40a9fb0 msgr2=0x7ff4d40aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:44:32.295 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.281+0000 7ff4d9f2c640 1 -- 172.21.15.42:0/4076912177 shutdown_connections 2024-09-16T03:44:32.295 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.281+0000 7ff4d9f2c640 1 -- 172.21.15.42:0/4076912177 wait complete. 2024-09-16T03:44:32.295 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.281+0000 7ff4d9f2c640 10 monclient: build_initial_monmap 2024-09-16T03:44:32.295 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.281+0000 7ff4d9f2c640 10 monclient: monmap: 2024-09-16T03:44:32.295 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:44:32.295 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:44:32.295 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:44:32.295 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:44:32.295 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:44:32.295 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:44:32.295 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:44:32.295 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:44:32.295 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:44:32.296 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:44:32.296 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.281+0000 7ff4d9f2c640 1 librados: starting msgr at 2024-09-16T03:44:32.296 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.281+0000 7ff4d9f2c640 1 librados: starting objecter 2024-09-16T03:44:32.296 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.281+0000 7ff4d9f2c640 1 Processor -- start 2024-09-16T03:44:32.296 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.281+0000 7ff4d9f2c640 1 -- start start 2024-09-16T03:44:32.296 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.281+0000 7ff4d9f2c640 1 librados: setting wanted keys 2024-09-16T03:44:32.296 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.281+0000 7ff4d9f2c640 1 librados: calling monclient init 2024-09-16T03:44:32.296 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.281+0000 7ff4d9f2c640 10 monclient: init 2024-09-16T03:44:32.296 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.285+0000 7ff4d9f2c640 10 monclient: _reopen_session rank -1 2024-09-16T03:44:32.296 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.285+0000 7ff4d9f2c640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:44:32.296 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.285+0000 7ff4d9f2c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff4bc004810 0x7ff4d4073110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:44:32.296 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.285+0000 7ff4d9f2c640 10 monclient(hunting): picked mon.noname-a con 0x7ff4bc004810 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:44:32.296 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.285+0000 7ff4d9f2c640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff4d4072620 0x7ff4d40a0030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:44:32.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.285+0000 7ff4d9f2c640 10 monclient(hunting): picked mon.noname-c con 0x7ff4d4072620 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:44:32.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.285+0000 7ff4d9f2c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff4d4072a60 0x7ff4d40a0570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:44:32.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.285+0000 7ff4d37fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff4bc004810 0x7ff4d4073110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:44:32.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.285+0000 7ff4d37fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff4bc004810 0x7ff4d4073110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:35082/0 (socket says 172.21.15.42:35082) 2024-09-16T03:44:32.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.285+0000 7ff4d9f2c640 10 monclient(hunting): picked mon.noname-b con 0x7ff4d4072a60 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:44:32.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.285+0000 7ff4d9f2c640 10 monclient(hunting): start opening mon connection 2024-09-16T03:44:32.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.285+0000 7ff4d37fe640 1 -- 172.21.15.42:0/4237531094 learned_addr learned my addr 172.21.15.42:0/4237531094 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:44:32.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.285+0000 7ff4d9f2c640 1 -- 172.21.15.42:0/4237531094 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7ff4d4137ba0 con 0x7ff4bc004810 2024-09-16T03:44:32.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.285+0000 7ff4d9f2c640 10 monclient(hunting): start opening mon connection 2024-09-16T03:44:32.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.285+0000 7ff4d9f2c640 1 -- 172.21.15.42:0/4237531094 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff4d4137a20 con 0x7ff4d4072a60 2024-09-16T03:44:32.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.285+0000 7ff4d9f2c640 10 monclient(hunting): start opening mon connection 2024-09-16T03:44:32.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.285+0000 7ff4d9f2c640 1 -- 172.21.15.42:0/4237531094 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7ff4d4137d20 con 0x7ff4d4072620 2024-09-16T03:44:32.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.285+0000 7ff4d9f2c640 10 monclient(hunting): _renew_subs 2024-09-16T03:44:32.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.285+0000 7ff4d9f2c640 10 monclient(hunting): authenticate will time out at 1119.256466s 2024-09-16T03:44:32.298 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.285+0000 7ff4d2ffd640 1 --2- 172.21.15.42:0/4237531094 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff4d4072620 0x7ff4d40a0030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:44:32.298 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.285+0000 7ff4d37fe640 10 monclient(hunting): get_auth_request con 0x7ff4bc004810 auth_method 0 2024-09-16T03:44:32.298 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.285+0000 7ff4d37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:44:32.298 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.285+0000 7ff4d37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:44:32.298 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.285+0000 7ff4d37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:44:32.298 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.285+0000 7ff4d2ffd640 10 monclient(hunting): get_auth_request con 0x7ff4d4072620 auth_method 0 2024-09-16T03:44:32.298 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.285+0000 7ff4d2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:44:32.298 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.285+0000 7ff4d2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:44:32.298 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.285+0000 7ff4d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:44:32.298 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.285+0000 7ff4d2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:44:32.298 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.285+0000 7ff4d2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:44:32.298 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.285+0000 7ff4d2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:44:32.298 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.285+0000 7ff4d37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:44:32.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.285+0000 7ff4d37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:44:32.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.285+0000 7ff4d37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:44:32.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.285+0000 7ff4d2ffd640 10 monclient(hunting): handle_auth_done global_id 10934 payload 824 2024-09-16T03:44:32.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.285+0000 7ff4d2ffd640 1 -- 172.21.15.42:0/4237531094 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff4d4072a60 msgr2=0x7ff4d40a0570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:44:32.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.285+0000 7ff4d2ffd640 1 --2- 172.21.15.42:0/4237531094 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff4d4072a60 0x7ff4d40a0570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:44:32.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.285+0000 7ff4d2ffd640 1 -- 172.21.15.42:0/4237531094 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff4bc004810 msgr2=0x7ff4d4073110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:44:32.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.285+0000 7ff4d2ffd640 1 --2- 172.21.15.42:0/4237531094 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff4bc004810 0x7ff4d4073110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:44:32.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.285+0000 7ff4d2ffd640 10 monclient: _finish_hunting 0 2024-09-16T03:44:32.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.285+0000 7ff4d2ffd640 1 monclient: found mon.noname-c 2024-09-16T03:44:32.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.285+0000 7ff4d2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:44:32.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.285+0000 7ff4d2ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:44:32.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.285+0000 7ff4d2ffd640 1 -- 172.21.15.42:0/4237531094 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4d4133250 con 0x7ff4d4072620 2024-09-16T03:44:32.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.285+0000 7ff4d37fe640 1 --2- 172.21.15.42:0/4237531094 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff4bc004810 0x7ff4d4073110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:44:32.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.285+0000 7ff4d2ffd640 1 --2- 172.21.15.42:0/4237531094 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff4d4072620 0x7ff4d40a0030 secure :-1 s=READY pgs=1793 cs=0 l=1 rev1=1 crypto rx=0x7ff4bc00c6e0 tx=0x7ff4c8019120 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:44:32.300 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.285+0000 7ff4bbfff640 10 client.?.objecter ms_handle_connect 0x7ff4d4072620 2024-09-16T03:44:32.300 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.285+0000 7ff4bbfff640 10 client.?.objecter resend_mon_ops 2024-09-16T03:44:32.300 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.285+0000 7ff4bbfff640 1 -- 172.21.15.42:0/4237531094 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4c80b0050 con 0x7ff4d4072620 2024-09-16T03:44:32.300 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.285+0000 7ff4bbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:44:32.300 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.285+0000 7ff4bbfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:44:32.300 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.285+0000 7ff4bbfff640 10 monclient: dump: 2024-09-16T03:44:32.300 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:44:32.300 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:44:32.300 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:32.300 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:32.300 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:44:32.300 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:44:32.300 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:44:32.301 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:44:32.301 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:44:32.301 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:44:32.301 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.285+0000 7ff4bbfff640 10 monclient: _finish_auth 0 2024-09-16T03:44:32.301 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.285+0000 7ff4bbfff640 10 monclient: _check_auth_tickets 2024-09-16T03:44:32.301 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.285+0000 7ff4bbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:44:32.301 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.285+0000 7ff4d9f2c640 5 monclient: authenticate success, global_id 10934 2024-09-16T03:44:32.301 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.285+0000 7ff4bbfff640 1 -- 172.21.15.42:0/4237531094 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4c80b1040 con 0x7ff4d4072620 2024-09-16T03:44:32.301 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.285+0000 7ff4d9f2c640 10 monclient: _renew_subs 2024-09-16T03:44:32.301 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.285+0000 7ff4d9f2c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:44:32.301 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.285+0000 7ff4d9f2c640 1 -- 172.21.15.42:0/4237531094 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4d40a2830 con 0x7ff4d4072620 2024-09-16T03:44:32.301 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.285+0000 7ff4bbfff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:44:32.301 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.285+0000 7ff4bbfff640 1 -- 172.21.15.42:0/4237531094 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4c8006d70 con 0x7ff4d4072620 2024-09-16T03:44:32.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.285+0000 7ff4bbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:44:32.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.285+0000 7ff4bbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:44:32.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.285+0000 7ff4bbfff640 10 monclient: dump: 2024-09-16T03:44:32.302 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:44:32.302 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:44:32.302 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:32.302 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:32.302 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:44:32.302 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:44:32.302 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:44:32.302 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:44:32.302 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:44:32.302 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:44:32.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.285+0000 7ff4d9f2c640 10 client.10934.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:44:32.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.285+0000 7ff4d9f2c640 10 monclient: _renew_subs 2024-09-16T03:44:32.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.285+0000 7ff4d9f2c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:44:32.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.285+0000 7ff4d9f2c640 1 -- 172.21.15.42:0/4237531094 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4d40a8ac0 con 0x7ff4d4072620 2024-09-16T03:44:32.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.285+0000 7ff4d9f2c640 1 librados: init done 2024-09-16T03:44:32.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.285+0000 7ff4bbfff640 1 -- 172.21.15.42:0/4237531094 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7ff4c80be030 con 0x7ff4d4072620 2024-09-16T03:44:32.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.285+0000 7ff4bbfff640 1 --2- 172.21.15.42:0/4237531094 >> v2:172.21.15.154:6812/254845342 conn(0x7ff4c0037840 0x7ff4c0039d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:44:32.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.285+0000 7ff4bbfff640 1 -- 172.21.15.42:0/4237531094 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(54..54 src has 1..54) ==== 10680+0+0 (secure 0 0 0) 0x7ff4c80e3d30 con 0x7ff4d4072620 2024-09-16T03:44:32.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.285+0000 7ff4bbfff640 10 client.10934.objecter ms_dispatch 0x7ff4d4000ba0 osd_map(54..54 src has 1..54) 2024-09-16T03:44:32.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.285+0000 7ff4bbfff640 3 client.10934.objecter handle_osd_map got epochs [54,54] > 0 2024-09-16T03:44:32.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.285+0000 7ff4bbfff640 3 client.10934.objecter handle_osd_map decoding full epoch 54 2024-09-16T03:44:32.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.285+0000 7ff4d9f2c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:44:32.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.285+0000 7ff4d9f2c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:44:32.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.285+0000 7ff4d9f2c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:44:32.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.285+0000 7ff4bbfff640 20 client.10934.objecter dump_active .. 0 homeless 2024-09-16T03:44:32.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.285+0000 7ff4d9f2c640 1 -- 172.21.15.42:0/4237531094 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff4d4134230 con 0x7ff4d4072620 2024-09-16T03:44:32.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.289+0000 7ff4d1ffb640 4 set_mon_vals no callback set 2024-09-16T03:44:32.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.289+0000 7ff4d37fe640 1 --2- 172.21.15.42:0/4237531094 >> v2:172.21.15.154:6812/254845342 conn(0x7ff4c0037840 0x7ff4c0039d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:44:32.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.289+0000 7ff4d37fe640 10 monclient: get_auth_request con 0x7ff4c0037840 auth_method 0 2024-09-16T03:44:32.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.293+0000 7ff4bbfff640 1 -- 172.21.15.42:0/4237531094 <== mon.2 v2:172.21.15.42:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7ff4c80c5060 con 0x7ff4d4072620 2024-09-16T03:44:32.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.293+0000 7ff4bbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:44:32.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.293+0000 7ff4bbfff640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:44:32.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.293+0000 7ff4d37fe640 1 --2- 172.21.15.42:0/4237531094 >> v2:172.21.15.154:6812/254845342 conn(0x7ff4c0037840 0x7ff4c0039d00 secure :-1 s=READY pgs=2878 cs=0 l=1 rev1=1 crypto rx=0x7ff4c4000f70 tx=0x7ff4c4001ad0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:44:32.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.293+0000 7ff4bbfff640 10 client.10934.objecter ms_handle_connect 0x7ff4c0037840 2024-09-16T03:44:32.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.541+0000 7ff4d9f2c640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:44:32.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.541+0000 7ff4d9f2c640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:44:32.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.541+0000 7ff4d9f2c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:44:32.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.541+0000 7ff4d9f2c640 1 -- 172.21.15.42:0/4237531094 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7ff4d40a5750 con 0x7ff4d4072620 2024-09-16T03:44:32.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.541+0000 7ff4bbfff640 1 -- 172.21.15.42:0/4237531094 <== mon.2 v2:172.21.15.42:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7ff4c8016090 con 0x7ff4d4072620 2024-09-16T03:44:32.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.541+0000 7ff4bbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:44:32.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.541+0000 7ff4bbfff640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:44:32.548 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":22,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:44:32.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.545+0000 7ff4d9f2c640 10 librados: watch_flush enter 2024-09-16T03:44:32.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.545+0000 7ff4d9f2c640 10 librados: watch_flush exit 2024-09-16T03:44:32.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.545+0000 7ff4d9f2c640 20 client.10934.objecter shutdown clearing up homeless session... 2024-09-16T03:44:32.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.545+0000 7ff4d9f2c640 10 client.10934.objecter successfully canceled tick 2024-09-16T03:44:32.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.545+0000 7ff4d9f2c640 1 -- 172.21.15.42:0/4237531094 >> v2:172.21.15.154:6812/254845342 conn(0x7ff4c0037840 msgr2=0x7ff4c0039d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:44:32.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.545+0000 7ff4d9f2c640 1 --2- 172.21.15.42:0/4237531094 >> v2:172.21.15.154:6812/254845342 conn(0x7ff4c0037840 0x7ff4c0039d00 secure :-1 s=READY pgs=2878 cs=0 l=1 rev1=1 crypto rx=0x7ff4c4000f70 tx=0x7ff4c4001ad0 comp rx=0 tx=0).stop 2024-09-16T03:44:32.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.545+0000 7ff4d9f2c640 10 monclient: shutdown 2024-09-16T03:44:32.551 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.545+0000 7ff4d9f2c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:44:32.551 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.545+0000 7ff4d9f2c640 1 -- 172.21.15.42:0/4237531094 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff4d4072620 msgr2=0x7ff4d40a0030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:44:32.551 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.545+0000 7ff4d9f2c640 1 --2- 172.21.15.42:0/4237531094 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff4d4072620 0x7ff4d40a0030 secure :-1 s=READY pgs=1793 cs=0 l=1 rev1=1 crypto rx=0x7ff4bc00c6e0 tx=0x7ff4c8019120 comp rx=0 tx=0).stop 2024-09-16T03:44:32.551 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.545+0000 7ff4d9f2c640 1 -- 172.21.15.42:0/4237531094 shutdown_connections 2024-09-16T03:44:32.551 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.545+0000 7ff4d9f2c640 1 --2- 172.21.15.42:0/4237531094 >> v2:172.21.15.154:6812/254845342 conn(0x7ff4c0037840 0x7ff4c0039d00 unknown :-1 s=CLOSED pgs=2878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:44:32.551 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.545+0000 7ff4d9f2c640 1 --2- 172.21.15.42:0/4237531094 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff4d4072a60 0x7ff4d40a0570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:44:32.551 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.545+0000 7ff4d9f2c640 1 --2- 172.21.15.42:0/4237531094 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff4d4072620 0x7ff4d40a0030 unknown :-1 s=CLOSED pgs=1793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:44:32.551 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.545+0000 7ff4d9f2c640 1 --2- 172.21.15.42:0/4237531094 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff4bc004810 0x7ff4d4073110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:44:32.551 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.545+0000 7ff4d9f2c640 1 -- 172.21.15.42:0/4237531094 >> 172.21.15.42:0/4237531094 conn(0x7ff4d40a9fb0 msgr2=0x7ff4d40a90b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:44:32.551 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.545+0000 7ff4d9f2c640 1 -- 172.21.15.42:0/4237531094 shutdown_connections 2024-09-16T03:44:32.551 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.545+0000 7ff4d9f2c640 1 -- 172.21.15.42:0/4237531094 wait complete. 2024-09-16T03:44:32.551 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.545+0000 7ff4d9f2c640 1 librados: shutdown 2024-09-16T03:44:32.569 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T03:44:32.569 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T03:44:32.569 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T03:44:32.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.689+0000 7f63ac212640 10 monclient: get_monmap_and_config 2024-09-16T03:44:32.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.689+0000 7f63ac212640 10 monclient: build_initial_monmap 2024-09-16T03:44:32.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.689+0000 7f63ac212640 10 monclient: monmap: 2024-09-16T03:44:32.694 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:44:32.694 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:44:32.694 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:44:32.693109+0000 2024-09-16T03:44:32.695 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:44:32.693109+0000 2024-09-16T03:44:32.695 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:44:32.695 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:44:32.695 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:44:32.695 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:44:32.695 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:44:32.695 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:44:32.695 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:44:32.695 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:44:32.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.689+0000 7f63ac212640 1 Processor -- start 2024-09-16T03:44:32.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.689+0000 7f63ac212640 1 -- start start 2024-09-16T03:44:32.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.689+0000 7f63ac212640 10 monclient: init 2024-09-16T03:44:32.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.689+0000 7f63ac212640 10 monclient: _reopen_session rank -1 2024-09-16T03:44:32.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.689+0000 7f63ac212640 10 monclient: _add_conns ranks=[4,1,2,0,3] 2024-09-16T03:44:32.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.689+0000 7f63ac212640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f63a406db90 0x7f63a41340c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:44:32.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.689+0000 7f63ac212640 10 monclient(hunting): picked mon.noname-c con 0x7f63a406db90 addr v2:172.21.15.154:3300/0 2024-09-16T03:44:32.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.689+0000 7f63ac212640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f63a40a37f0 addr v1:172.21.15.154:6789/0 2024-09-16T03:44:32.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.689+0000 7f63ac212640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f63a40a3c00 0x7f63a4137520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:44:32.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.689+0000 7f63ac212640 10 monclient(hunting): picked mon.noname-b con 0x7f63a40a3c00 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:44:32.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.689+0000 7f63ac212640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f63a406ece0 con 0x7f63a40a37f0 2024-09-16T03:44:32.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.689+0000 7f63ac212640 10 monclient(hunting): start opening mon connection 2024-09-16T03:44:32.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.689+0000 7f63ac212640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f63a406df70 con 0x7f63a406db90 2024-09-16T03:44:32.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.689+0000 7f63ac212640 10 monclient(hunting): start opening mon connection 2024-09-16T03:44:32.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.689+0000 7f63ac212640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f63a409bf80 con 0x7f63a40a3c00 2024-09-16T03:44:32.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.689+0000 7f63ac212640 10 monclient(hunting): _renew_subs 2024-09-16T03:44:32.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.689+0000 7f63ac212640 10 monclient(hunting): authenticate will time out at 1119.662862s 2024-09-16T03:44:32.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.689+0000 7f63aa7af640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f63a40a3c00 0x7f63a4137520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:44:32.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.689+0000 7f63aa7af640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f63a40a3c00 0x7f63a4137520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:41164/0 (socket says 172.21.15.42:41164) 2024-09-16T03:44:32.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.689+0000 7f63aa7af640 1 -- 172.21.15.42:0/622628787 learned_addr learned my addr 172.21.15.42:0/622628787 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:44:32.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.689+0000 7f63aa7af640 10 monclient(hunting): get_auth_request con 0x7f63a40a3c00 auth_method 0 2024-09-16T03:44:32.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.689+0000 7f63aa7af640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:44:32.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.689+0000 7f63aa7af640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:44:32.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.689+0000 7f63aa7af640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:44:32.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.689+0000 7f63a9fae640 1 --2- 172.21.15.42:0/622628787 >> v2:172.21.15.154:3300/0 conn(0x7f63a406db90 0x7f63a41340c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:44:32.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.689+0000 7f63a9fae640 10 monclient(hunting): get_auth_request con 0x7f63a406db90 auth_method 0 2024-09-16T03:44:32.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.689+0000 7f63a9fae640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:44:32.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.689+0000 7f63a9fae640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:44:32.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.689+0000 7f63a9fae640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:44:32.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.689+0000 7f63aa7af640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:44:32.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.689+0000 7f63aa7af640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:44:32.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.689+0000 7f63aa7af640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:44:32.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.689+0000 7f63aa7af640 10 monclient(hunting): handle_auth_done global_id 11168 payload 274 2024-09-16T03:44:32.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.689+0000 7f63aa7af640 1 -- 172.21.15.42:0/622628787 >> v2:172.21.15.154:3300/0 conn(0x7f63a406db90 msgr2=0x7f63a41340c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:44:32.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.689+0000 7f63aa7af640 1 --2- 172.21.15.42:0/622628787 >> v2:172.21.15.154:3300/0 conn(0x7f63a406db90 0x7f63a41340c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:44:32.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.689+0000 7f63aa7af640 1 -- 172.21.15.42:0/622628787 >> v1:172.21.15.154:6789/0 conn(0x7f63a40a37f0 legacy=0x7f63a4136e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:44:32.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.689+0000 7f63aa7af640 10 monclient: _finish_hunting 0 2024-09-16T03:44:32.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.689+0000 7f63aa7af640 1 monclient: found mon.noname-b 2024-09-16T03:44:32.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.689+0000 7f63aa7af640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:44:32.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.689+0000 7f63aa7af640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T03:44:32.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.689+0000 7f63aa7af640 1 -- 172.21.15.42:0/622628787 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63a4134780 con 0x7f63a40a3c00 2024-09-16T03:44:32.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.693+0000 7f63aa7af640 1 --2- 172.21.15.42:0/622628787 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f63a40a3c00 0x7f63a4137520 secure :-1 s=READY pgs=1853 cs=0 l=1 rev1=1 crypto rx=0x7f639c009e50 tx=0x7f639c01d310 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d4e3f0722df18376 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:44:32.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.693+0000 7f63a8fac640 1 -- 172.21.15.42:0/622628787 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f639c0a5640 con 0x7f63a40a3c00 2024-09-16T03:44:32.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.693+0000 7f63a8fac640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:44:32.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.693+0000 7f63a8fac640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:44:32.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.693+0000 7f63a8fac640 10 monclient: dump: 2024-09-16T03:44:32.709 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:44:32.709 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:44:32.710 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:32.710 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:32.710 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:44:32.710 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:44:32.710 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:44:32.710 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:44:32.710 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:44:32.710 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:44:32.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.693+0000 7f63a8fac640 10 monclient: _finish_auth 0 2024-09-16T03:44:32.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.693+0000 7f63a8fac640 10 monclient: _check_auth_tickets 2024-09-16T03:44:32.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.693+0000 7f63a8fac640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:44:32.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.693+0000 7f63ac212640 5 monclient: authenticate success, global_id 11168 2024-09-16T03:44:32.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.693+0000 7f63a8fac640 1 -- 172.21.15.42:0/622628787 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f639c0a57e0 con 0x7f63a40a3c00 2024-09-16T03:44:32.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.693+0000 7f63ac212640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:44:32.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.693+0000 7f63a8fac640 10 monclient: handle_config config(0 keys) 2024-09-16T03:44:32.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.693+0000 7f63ac212640 10 monclient: get_monmap_and_config success 2024-09-16T03:44:32.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.693+0000 7f63ac212640 4 set_mon_vals no callback set 2024-09-16T03:44:32.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.693+0000 7f63a8fac640 1 -- 172.21.15.42:0/622628787 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f639c0a5bb0 con 0x7f63a40a3c00 2024-09-16T03:44:32.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.693+0000 7f63a8fac640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:44:32.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.693+0000 7f63a8fac640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:44:32.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.693+0000 7f63a8fac640 10 monclient: dump: 2024-09-16T03:44:32.712 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:44:32.712 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:44:32.712 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:32.712 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:32.712 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:44:32.712 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:44:32.712 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:44:32.713 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:44:32.713 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:44:32.713 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:44:32.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.693+0000 7f63ac212640 10 monclient: shutdown 2024-09-16T03:44:32.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.693+0000 7f63ac212640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:44:32.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.693+0000 7f63ac212640 1 -- 172.21.15.42:0/622628787 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f63a40a3c00 msgr2=0x7f63a4137520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:44:32.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.693+0000 7f63ac212640 1 --2- 172.21.15.42:0/622628787 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f63a40a3c00 0x7f63a4137520 secure :-1 s=READY pgs=1853 cs=0 l=1 rev1=1 crypto rx=0x7f639c009e50 tx=0x7f639c01d310 comp rx=0 tx=0).stop 2024-09-16T03:44:32.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.693+0000 7f63ac212640 1 -- 172.21.15.42:0/622628787 shutdown_connections 2024-09-16T03:44:32.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.693+0000 7f63ac212640 1 --2- 172.21.15.42:0/622628787 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f63a40a3c00 0x7f63a4137520 unknown :-1 s=CLOSED pgs=1853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:44:32.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.693+0000 7f63ac212640 1 --2- 172.21.15.42:0/622628787 >> v2:172.21.15.154:3300/0 conn(0x7f63a406db90 0x7f63a41340c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:44:32.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.693+0000 7f63ac212640 1 -- 172.21.15.42:0/622628787 >> 172.21.15.42:0/622628787 conn(0x7f63a40a9fb0 msgr2=0x7f63a40aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:44:32.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.693+0000 7f63ac212640 1 -- 172.21.15.42:0/622628787 shutdown_connections 2024-09-16T03:44:32.714 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.693+0000 7f63ac212640 1 -- 172.21.15.42:0/622628787 wait complete. 2024-09-16T03:44:32.714 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.693+0000 7f63ac212640 10 monclient: build_initial_monmap 2024-09-16T03:44:32.714 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.693+0000 7f63ac212640 10 monclient: monmap: 2024-09-16T03:44:32.714 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:44:32.714 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:44:32.714 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:44:32.714 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:44:32.714 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:44:32.714 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:44:32.715 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:44:32.715 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:44:32.715 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:44:32.715 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:44:32.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.693+0000 7f63ac212640 1 librados: starting msgr at 2024-09-16T03:44:32.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.693+0000 7f63ac212640 1 librados: starting objecter 2024-09-16T03:44:32.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.693+0000 7f63ac212640 1 Processor -- start 2024-09-16T03:44:32.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.693+0000 7f63ac212640 1 -- start start 2024-09-16T03:44:32.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.693+0000 7f63ac212640 1 librados: setting wanted keys 2024-09-16T03:44:32.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.693+0000 7f63ac212640 1 librados: calling monclient init 2024-09-16T03:44:32.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.693+0000 7f63ac212640 10 monclient: init 2024-09-16T03:44:32.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.697+0000 7f63ac212640 10 monclient: _reopen_session rank -1 2024-09-16T03:44:32.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.697+0000 7f63ac212640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:44:32.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.697+0000 7f63ac212640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f63a406db90 0x7f63a41cee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:44:32.716 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.697+0000 7f63ac212640 10 monclient(hunting): picked mon.noname-b con 0x7f63a406db90 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:44:32.716 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.697+0000 7f63ac212640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f63a40a37f0 0x7f63a413e460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:44:32.716 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.697+0000 7f63ac212640 10 monclient(hunting): picked mon.noname-a con 0x7f63a40a37f0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:44:32.716 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.697+0000 7f63ac212640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f63a40a3c00 0x7f63a413e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:44:32.716 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.697+0000 7f63ac212640 10 monclient(hunting): picked mon.noname-c con 0x7f63a40a3c00 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:44:32.716 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.697+0000 7f63ac212640 10 monclient(hunting): start opening mon connection 2024-09-16T03:44:32.716 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.697+0000 7f63ac212640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f63a41419d0 con 0x7f63a40a37f0 2024-09-16T03:44:32.716 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.697+0000 7f63ac212640 10 monclient(hunting): start opening mon connection 2024-09-16T03:44:32.716 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.697+0000 7f63ac212640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f63a4137c70 con 0x7f63a406db90 2024-09-16T03:44:32.716 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.697+0000 7f63ac212640 10 monclient(hunting): start opening mon connection 2024-09-16T03:44:32.716 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.697+0000 7f63ac212640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f63a4141b50 con 0x7f63a40a3c00 2024-09-16T03:44:32.716 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.697+0000 7f63ac212640 10 monclient(hunting): _renew_subs 2024-09-16T03:44:32.716 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.697+0000 7f63ac212640 10 monclient(hunting): authenticate will time out at 1119.668180s 2024-09-16T03:44:32.717 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.697+0000 7f63a97ad640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f63a40a37f0 0x7f63a413e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:44:32.717 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.697+0000 7f63aa7af640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f63a40a3c00 0x7f63a413e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:44:32.717 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.697+0000 7f63a97ad640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f63a40a37f0 0x7f63a413e460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:35092/0 (socket says 172.21.15.42:35092) 2024-09-16T03:44:32.717 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.697+0000 7f63a97ad640 1 -- 172.21.15.42:0/1402273532 learned_addr learned my addr 172.21.15.42:0/1402273532 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:44:32.717 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.697+0000 7f63a97ad640 10 monclient(hunting): get_auth_request con 0x7f63a40a37f0 auth_method 0 2024-09-16T03:44:32.717 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.697+0000 7f63a97ad640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:44:32.717 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.697+0000 7f63a97ad640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:44:32.717 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.697+0000 7f63a97ad640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:44:32.717 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.697+0000 7f63aa7af640 10 monclient(hunting): get_auth_request con 0x7f63a40a3c00 auth_method 0 2024-09-16T03:44:32.717 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.697+0000 7f63aa7af640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:44:32.717 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.697+0000 7f63aa7af640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:44:32.717 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.697+0000 7f63aa7af640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:44:32.718 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.697+0000 7f63aa7af640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:44:32.718 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.697+0000 7f63aa7af640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:44:32.718 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.697+0000 7f63aa7af640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:44:32.718 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.697+0000 7f63a97ad640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:44:32.718 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.697+0000 7f63a97ad640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:44:32.718 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.697+0000 7f63a97ad640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:44:32.718 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.697+0000 7f63aa7af640 10 monclient(hunting): handle_auth_done global_id 11180 payload 824 2024-09-16T03:44:32.718 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.697+0000 7f63aa7af640 1 -- 172.21.15.42:0/1402273532 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f63a406db90 msgr2=0x7f63a41cee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:44:32.718 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.697+0000 7f63aa7af640 1 --2- 172.21.15.42:0/1402273532 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f63a406db90 0x7f63a41cee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:44:32.718 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.697+0000 7f63aa7af640 1 -- 172.21.15.42:0/1402273532 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f63a40a37f0 msgr2=0x7f63a413e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:44:32.719 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.697+0000 7f63aa7af640 1 --2- 172.21.15.42:0/1402273532 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f63a40a37f0 0x7f63a413e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:44:32.719 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.697+0000 7f63aa7af640 10 monclient: _finish_hunting 0 2024-09-16T03:44:32.719 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.697+0000 7f63aa7af640 1 monclient: found mon.noname-c 2024-09-16T03:44:32.719 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.697+0000 7f63aa7af640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:44:32.719 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.697+0000 7f63aa7af640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:44:32.719 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.697+0000 7f63aa7af640 1 -- 172.21.15.42:0/1402273532 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63a4139940 con 0x7f63a40a3c00 2024-09-16T03:44:32.719 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.697+0000 7f63a97ad640 1 --2- 172.21.15.42:0/1402273532 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f63a40a37f0 0x7f63a413e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:44:32.719 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.697+0000 7f63aa7af640 1 --2- 172.21.15.42:0/1402273532 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f63a40a3c00 0x7f63a413e9a0 secure :-1 s=READY pgs=1855 cs=0 l=1 rev1=1 crypto rx=0x7f639c0ab000 tx=0x7f639c019650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:44:32.719 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.697+0000 7f639a7fc640 10 client.?.objecter ms_handle_connect 0x7f63a40a3c00 2024-09-16T03:44:32.719 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.697+0000 7f639a7fc640 10 client.?.objecter resend_mon_ops 2024-09-16T03:44:32.719 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.697+0000 7f639a7fc640 1 -- 172.21.15.42:0/1402273532 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f639c0bc030 con 0x7f63a40a3c00 2024-09-16T03:44:32.720 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.697+0000 7f639a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:44:32.720 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.697+0000 7f639a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:44:32.720 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.697+0000 7f639a7fc640 10 monclient: dump: 2024-09-16T03:44:32.720 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:44:32.720 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:44:32.720 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:32.720 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:32.720 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:44:32.720 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:44:32.720 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:44:32.720 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:44:32.721 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:44:32.721 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:44:32.721 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.697+0000 7f639a7fc640 10 monclient: _finish_auth 0 2024-09-16T03:44:32.721 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.697+0000 7f639a7fc640 10 monclient: _check_auth_tickets 2024-09-16T03:44:32.721 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.697+0000 7f639a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:44:32.721 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.697+0000 7f63ac212640 5 monclient: authenticate success, global_id 11180 2024-09-16T03:44:32.721 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.697+0000 7f639a7fc640 1 -- 172.21.15.42:0/1402273532 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f639c0b8470 con 0x7f63a40a3c00 2024-09-16T03:44:32.721 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.697+0000 7f639a7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:44:32.721 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.697+0000 7f639a7fc640 1 -- 172.21.15.42:0/1402273532 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f639c009510 con 0x7f63a40a3c00 2024-09-16T03:44:32.721 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.697+0000 7f639a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:44:32.721 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.697+0000 7f639a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:44:32.722 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.697+0000 7f639a7fc640 10 monclient: dump: 2024-09-16T03:44:32.722 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:44:32.722 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:44:32.722 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:32.722 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:32.722 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:44:32.722 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:44:32.722 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:44:32.722 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:44:32.722 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:44:32.722 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:44:32.722 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.697+0000 7f639bfff640 4 set_mon_vals no callback set 2024-09-16T03:44:32.722 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.697+0000 7f63ac212640 10 monclient: _renew_subs 2024-09-16T03:44:32.722 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.697+0000 7f63ac212640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:44:32.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.697+0000 7f63ac212640 1 -- 172.21.15.42:0/1402273532 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63a41cf340 con 0x7f63a40a3c00 2024-09-16T03:44:32.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.697+0000 7f63ac212640 10 client.11180.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:44:32.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.697+0000 7f63ac212640 10 monclient: _renew_subs 2024-09-16T03:44:32.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.697+0000 7f63ac212640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:44:32.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.697+0000 7f63ac212640 1 -- 172.21.15.42:0/1402273532 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f63a4139c20 con 0x7f63a40a3c00 2024-09-16T03:44:32.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.697+0000 7f63ac212640 1 librados: init done 2024-09-16T03:44:32.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.697+0000 7f63ac212640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:44:32.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.697+0000 7f63ac212640 1 --2- 172.21.15.42:0/1402273532 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f63a40a1fe0 0x7f63a402c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:44:32.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.697+0000 7f63ac212640 10 monclient(hunting): start opening mon connection 2024-09-16T03:44:32.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.697+0000 7f63ac212640 1 -- 172.21.15.42:0/1402273532 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f63a4133680 con 0x7f63a40a1fe0 2024-09-16T03:44:32.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.697+0000 7f639a7fc640 1 -- 172.21.15.42:0/1402273532 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f639c0bf070 con 0x7f63a40a3c00 2024-09-16T03:44:32.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.697+0000 7f63a9fae640 1 --2- 172.21.15.42:0/1402273532 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f63a40a1fe0 0x7f63a402c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:44:32.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.697+0000 7f63a9fae640 10 monclient: get_auth_request con 0x7f63a40a1fe0 auth_method 0 2024-09-16T03:44:32.724 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.697+0000 7f63a9fae640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:44:32.724 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.697+0000 7f63a9fae640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:44:32.724 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.697+0000 7f63a9fae640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:44:32.724 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.697+0000 7f63a9fae640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:44:32.724 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.697+0000 7f63a9fae640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:44:32.724 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.697+0000 7f63a9fae640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:44:32.724 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.697+0000 7f639a7fc640 1 --2- 172.21.15.42:0/1402273532 >> v2:172.21.15.154:6812/254845342 conn(0x7f6388037810 0x7f6388039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:44:32.724 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.697+0000 7f639a7fc640 1 -- 172.21.15.42:0/1402273532 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(54..54 src has 1..54) ==== 10680+0+0 (secure 0 0 0) 0x7f639c0e4ea0 con 0x7f63a40a3c00 2024-09-16T03:44:32.724 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.697+0000 7f639a7fc640 10 client.11180.objecter ms_dispatch 0x7f63a4000ba0 osd_map(54..54 src has 1..54) 2024-09-16T03:44:32.724 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.697+0000 7f639a7fc640 3 client.11180.objecter handle_osd_map got epochs [54,54] > 0 2024-09-16T03:44:32.724 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.697+0000 7f639a7fc640 3 client.11180.objecter handle_osd_map decoding full epoch 54 2024-09-16T03:44:32.724 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.697+0000 7f63a9fae640 10 monclient(hunting): handle_auth_done global_id 24972 payload 824 2024-09-16T03:44:32.725 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.697+0000 7f63a9fae640 1 -- 172.21.15.42:0/1402273532 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f63a41340c0 con 0x7f63a40a1fe0 2024-09-16T03:44:32.725 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.697+0000 7f639a7fc640 20 client.11180.objecter dump_active .. 0 homeless 2024-09-16T03:44:32.725 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.697+0000 7f63a9fae640 1 --2- 172.21.15.42:0/1402273532 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f63a40a1fe0 0x7f63a402c650 secure :-1 s=READY pgs=6768 cs=0 l=1 rev1=1 crypto rx=0x7f639400d4d0 tx=0x7f639400d990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:44:32.725 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.697+0000 7f639a7fc640 10 client.11180.objecter ms_handle_connect 0x7f63a40a1fe0 2024-09-16T03:44:32.725 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.697+0000 7f639a7fc640 10 client.11180.objecter resend_mon_ops 2024-09-16T03:44:32.725 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.697+0000 7f639a7fc640 1 -- 172.21.15.42:0/1402273532 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63940125f0 con 0x7f63a40a1fe0 2024-09-16T03:44:32.725 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.697+0000 7f639a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:44:32.725 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.697+0000 7f63a97ad640 1 --2- 172.21.15.42:0/1402273532 >> v2:172.21.15.154:6812/254845342 conn(0x7f6388037810 0x7f6388039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:44:32.725 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.697+0000 7f639a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:44:32.725 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.697+0000 7f639a7fc640 10 monclient: dump: 2024-09-16T03:44:32.725 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:44:32.725 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:44:32.725 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:32.726 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:32.726 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:44:32.726 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:44:32.726 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:44:32.726 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:44:32.726 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:44:32.726 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:44:32.726 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.697+0000 7f63a97ad640 10 monclient: get_auth_request con 0x7f6388037810 auth_method 0 2024-09-16T03:44:32.726 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.697+0000 7f639a7fc640 1 -- 172.21.15.42:0/1402273532 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f6394019070 con 0x7f63a40a1fe0 2024-09-16T03:44:32.726 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.697+0000 7f639a7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:44:32.726 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.697+0000 7f639a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:44:32.726 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.697+0000 7f639a7fc640 1 -- 172.21.15.42:0/1402273532 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f63a40a1fe0 msgr2=0x7f63a402c650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:44:32.726 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.697+0000 7f639a7fc640 1 --2- 172.21.15.42:0/1402273532 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f63a40a1fe0 0x7f63a402c650 secure :-1 s=READY pgs=6768 cs=0 l=1 rev1=1 crypto rx=0x7f639400d4d0 tx=0x7f639400d990 comp rx=0 tx=0).stop 2024-09-16T03:44:32.727 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.697+0000 7f639a7fc640 1 -- 172.21.15.42:0/1402273532 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f63a40a1fe0 msgr2=0x7f63a402c650 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:44:32.727 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.697+0000 7f639a7fc640 1 --2- 172.21.15.42:0/1402273532 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f63a40a1fe0 0x7f63a402c650 unknown :-1 s=CLOSED pgs=6768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:44:32.727 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.701+0000 7f63a97ad640 1 --2- 172.21.15.42:0/1402273532 >> v2:172.21.15.154:6812/254845342 conn(0x7f6388037810 0x7f6388039cd0 secure :-1 s=READY pgs=2918 cs=0 l=1 rev1=1 crypto rx=0x7f63a4139b50 tx=0x7f63a0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:44:32.727 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.701+0000 7f639a7fc640 10 client.11180.objecter ms_handle_connect 0x7f6388037810 2024-09-16T03:44:32.728 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.705+0000 7f63ac212640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:44:32.728 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.705+0000 7f63ac212640 1 --2- 172.21.15.42:0/1402273532 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f63a40a23c0 0x7f63a402cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:44:32.728 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.705+0000 7f63ac212640 10 monclient(hunting): start opening mon connection 2024-09-16T03:44:32.729 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.705+0000 7f63ac212640 1 -- 172.21.15.42:0/1402273532 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f63a409de20 con 0x7f63a40a23c0 2024-09-16T03:44:32.729 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.705+0000 7f63a9fae640 1 --2- 172.21.15.42:0/1402273532 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f63a40a23c0 0x7f63a402cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:44:32.729 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.705+0000 7f63a9fae640 10 monclient: get_auth_request con 0x7f63a40a23c0 auth_method 0 2024-09-16T03:44:32.729 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.705+0000 7f63a9fae640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:44:32.729 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.705+0000 7f63a9fae640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:44:32.729 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.705+0000 7f63a9fae640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:44:32.729 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.705+0000 7f63a9fae640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:44:32.729 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.705+0000 7f63a9fae640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:44:32.729 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.705+0000 7f63a9fae640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:44:32.729 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.705+0000 7f63a9fae640 10 monclient(hunting): handle_auth_done global_id 24984 payload 824 2024-09-16T03:44:32.729 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.705+0000 7f63a9fae640 1 -- 172.21.15.42:0/1402273532 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f63a4068e40 con 0x7f63a40a23c0 2024-09-16T03:44:32.729 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.705+0000 7f639a7fc640 10 client.11180.objecter ms_handle_connect 0x7f63a40a23c0 2024-09-16T03:44:32.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.705+0000 7f639a7fc640 10 client.11180.objecter resend_mon_ops 2024-09-16T03:44:32.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.705+0000 7f63a9fae640 1 --2- 172.21.15.42:0/1402273532 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f63a40a23c0 0x7f63a402cb90 secure :-1 s=READY pgs=6771 cs=0 l=1 rev1=1 crypto rx=0x7f6394018000 tx=0x7f63940046d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:44:32.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.705+0000 7f639a7fc640 1 -- 172.21.15.42:0/1402273532 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6394012bf0 con 0x7f63a40a23c0 2024-09-16T03:44:32.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.705+0000 7f639a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:44:32.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.705+0000 7f639a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:44:32.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.705+0000 7f639a7fc640 10 monclient: dump: 2024-09-16T03:44:32.730 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:44:32.730 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:44:32.730 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:32.730 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:32.730 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:44:32.730 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:44:32.730 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:44:32.731 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:44:32.731 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:44:32.731 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:44:32.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.705+0000 7f639a7fc640 1 -- 172.21.15.42:0/1402273532 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4715 (secure 0 0 0) 0x7f6394012bf0 con 0x7f63a40a23c0 2024-09-16T03:44:32.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.705+0000 7f639a7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:44:32.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.705+0000 7f639a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:44:32.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.705+0000 7f639a7fc640 1 -- 172.21.15.42:0/1402273532 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f63a40a23c0 msgr2=0x7f63a402cb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:44:32.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.705+0000 7f639a7fc640 1 --2- 172.21.15.42:0/1402273532 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f63a40a23c0 0x7f63a402cb90 secure :-1 s=READY pgs=6771 cs=0 l=1 rev1=1 crypto rx=0x7f6394018000 tx=0x7f63940046d0 comp rx=0 tx=0).stop 2024-09-16T03:44:32.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.705+0000 7f639a7fc640 1 -- 172.21.15.42:0/1402273532 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f63a40a23c0 msgr2=0x7f63a402cb90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:44:32.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.705+0000 7f639a7fc640 1 --2- 172.21.15.42:0/1402273532 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f63a40a23c0 0x7f63a402cb90 secure :-1 s=CLOSED pgs=6771 cs=0 l=1 rev1=1 crypto rx=0x7f6394018000 tx=0x7f63940046d0 comp rx=0 tx=0).stop 2024-09-16T03:44:32.732 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:44:32.732 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:44:32.732 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:44:32.732 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "leader", 2024-09-16T03:44:32.732 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 22, 2024-09-16T03:44:32.732 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 152075, 2024-09-16T03:44:32.732 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:44:32.732 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:44:32.732 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T03:44:32.733 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T03:44:32.733 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:44:32.733 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 0, 2024-09-16T03:44:32.733 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:44:32.733 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:44:32.733 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:44:32.733 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:44:32.733 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:44:32.733 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:44:32.733 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:44:32.733 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:44:32.733 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:44:32.734 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:44:32.734 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:44:32.734 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:44:32.734 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:44:32.734 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:44:32.734 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:44:32.734 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:44:32.734 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:44:32.734 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:44:32.734 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:44:32.734 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:44:32.734 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:44:32.734 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:44:32.734 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:44:32.735 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:44:32.735 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:44:32.735 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:44:32.735 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:44:32.735 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:44:32.735 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:44:32.735 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:44:32.735 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:44:32.735 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:44:32.735 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:44:32.735 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:44:32.735 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:44:32.735 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:44:32.735 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:44:32.736 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:44:32.736 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:44:32.736 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:44:32.736 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:44:32.736 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:44:32.736 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:44:32.736 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:44:32.736 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:44:32.736 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:44:32.736 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:44:32.736 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:44:32.736 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:44:32.736 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:44:32.737 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:44:32.737 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:44:32.737 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:44:32.737 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:44:32.737 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:44:32.737 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:44:32.737 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:44:32.737 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:44:32.737 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:44:32.737 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:44:32.737 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:44:32.737 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:44:32.737 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:44:32.738 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:44:32.738 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:44:32.738 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:44:32.738 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:44:32.738 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:44:32.738 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:44:32.738 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:44:32.738 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:44:32.738 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:44:32.738 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:44:32.738 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:44:32.738 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:44:32.738 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:44:32.739 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:44:32.739 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:44:32.739 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:44:32.739 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:44:32.739 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:44:32.739 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:44:32.739 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:44:32.739 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:44:32.739 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:44:32.739 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:44:32.739 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:44:32.739 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:44:32.739 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:44:32.740 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:44:32.740 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:44:32.740 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:44:32.740 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:44:32.740 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:44:32.740 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:44:32.740 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:44:32.740 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:44:32.740 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:44:32.740 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:44:32.740 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:44:32.740 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:44:32.740 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:44:32.741 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:44:32.741 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:44:32.741 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:44:32.741 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:44:32.741 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:44:32.741 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:44:32.741 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:44:32.741 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:44:32.741 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:44:32.741 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:44:32.742 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:44:32.742 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:44:32.742 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:44:32.742 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:44:32.742 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:44:32.742 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:44:32.742 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:44:32.742 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:44:32.742 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:44:32.742 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:44:32.742 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:44:32.742 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:44:32.742 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:44:32.743 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:44:32.743 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:44:32.743 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:44:32.743 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:44:32.743 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:44:32.743 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:44:32.743 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:44:32.743 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:44:32.743 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:44:32.743 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:44:32.743 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:44:32.743 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:44:32.744 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:44:32.744 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:44:32.744 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:44:32.744 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:44:32.744 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T03:44:32.744 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:44:32.744 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:44:32.744 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:44:32.744 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 4 2024-09-16T03:44:32.744 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:44:32.744 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:44:32.744 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:44:32.744 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:44:32.745 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:44:32.745 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:44:32.745 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 14 2024-09-16T03:44:32.745 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:44:32.745 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:44:32.745 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:44:32.745 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:44:32.745 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:44:32.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.705+0000 7f63ac212640 10 librados: watch_flush enter 2024-09-16T03:44:32.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.705+0000 7f63ac212640 10 librados: watch_flush exit 2024-09-16T03:44:32.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.705+0000 7f63ac212640 20 client.11180.objecter shutdown clearing up homeless session... 2024-09-16T03:44:32.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.705+0000 7f63ac212640 10 client.11180.objecter successfully canceled tick 2024-09-16T03:44:32.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.705+0000 7f63ac212640 1 -- 172.21.15.42:0/1402273532 >> v2:172.21.15.154:6812/254845342 conn(0x7f6388037810 msgr2=0x7f6388039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:44:32.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.705+0000 7f63ac212640 1 --2- 172.21.15.42:0/1402273532 >> v2:172.21.15.154:6812/254845342 conn(0x7f6388037810 0x7f6388039cd0 secure :-1 s=READY pgs=2918 cs=0 l=1 rev1=1 crypto rx=0x7f63a4139b50 tx=0x7f63a0009040 comp rx=0 tx=0).stop 2024-09-16T03:44:32.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.705+0000 7f63ac212640 10 monclient: shutdown 2024-09-16T03:44:32.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.705+0000 7f63ac212640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:44:32.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.705+0000 7f63ac212640 1 -- 172.21.15.42:0/1402273532 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f63a40a3c00 msgr2=0x7f63a413e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:44:32.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.705+0000 7f63ac212640 1 --2- 172.21.15.42:0/1402273532 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f63a40a3c00 0x7f63a413e9a0 secure :-1 s=READY pgs=1855 cs=0 l=1 rev1=1 crypto rx=0x7f639c0ab000 tx=0x7f639c019650 comp rx=0 tx=0).stop 2024-09-16T03:44:32.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.705+0000 7f63aa7af640 1 -- 172.21.15.42:0/1402273532 reap_dead start 2024-09-16T03:44:32.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.705+0000 7f63aa7af640 1 -- 172.21.15.42:0/1402273532 reap_dead start 2024-09-16T03:44:32.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.705+0000 7f63ac212640 1 -- 172.21.15.42:0/1402273532 shutdown_connections 2024-09-16T03:44:32.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.705+0000 7f63ac212640 1 -- 172.21.15.42:0/1402273532 >> 172.21.15.42:0/1402273532 conn(0x7f63a40a9fb0 msgr2=0x7f63a41369f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:44:32.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.705+0000 7f63ac212640 1 -- 172.21.15.42:0/1402273532 shutdown_connections 2024-09-16T03:44:32.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.709+0000 7f63ac212640 1 -- 172.21.15.42:0/1402273532 wait complete. 2024-09-16T03:44:32.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.709+0000 7f63ac212640 1 librados: shutdown 2024-09-16T03:44:32.748 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:44:32.748 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T03:44:32.749 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T03:44:32.875 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.869+0000 7f317a4df640 10 monclient: get_monmap_and_config 2024-09-16T03:44:32.876 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.869+0000 7f317a4df640 10 monclient: build_initial_monmap 2024-09-16T03:44:32.876 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.869+0000 7f317a4df640 10 monclient: monmap: 2024-09-16T03:44:32.876 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:44:32.876 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:44:32.876 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:44:32.874931+0000 2024-09-16T03:44:32.876 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:44:32.874931+0000 2024-09-16T03:44:32.876 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:44:32.876 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:44:32.876 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:44:32.876 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:44:32.877 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:44:32.877 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:44:32.877 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:44:32.877 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:44:32.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.869+0000 7f317a4df640 1 Processor -- start 2024-09-16T03:44:32.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.869+0000 7f317a4df640 1 -- start start 2024-09-16T03:44:32.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.869+0000 7f317a4df640 10 monclient: init 2024-09-16T03:44:32.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.873+0000 7f317a4df640 10 monclient: _reopen_session rank -1 2024-09-16T03:44:32.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.873+0000 7f317a4df640 10 monclient: _add_conns ranks=[2,0,4,1,3] 2024-09-16T03:44:32.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.873+0000 7f317a4df640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f31740a39b0 0x7f3174133b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:44:32.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.873+0000 7f317a4df640 10 monclient(hunting): picked mon.noname-b con 0x7f31740a39b0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:44:32.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.873+0000 7f317a4df640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f31740a3d90 addr v1:172.21.15.42:6789/0 2024-09-16T03:44:32.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.873+0000 7f317a4df640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f3174137710 0x7f317413f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:44:32.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.873+0000 7f317a4df640 10 monclient(hunting): picked mon.noname-c con 0x7f3174137710 addr v2:172.21.15.154:3300/0 2024-09-16T03:44:32.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.873+0000 7f317a4df640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f317409c140 con 0x7f31740a3d90 2024-09-16T03:44:32.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.873+0000 7f317a4df640 10 monclient(hunting): start opening mon connection 2024-09-16T03:44:32.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.873+0000 7f317a4df640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f317409c420 con 0x7f3174137710 2024-09-16T03:44:32.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.873+0000 7f317a4df640 10 monclient(hunting): start opening mon connection 2024-09-16T03:44:32.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.873+0000 7f317a4df640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f317406dd60 con 0x7f31740a39b0 2024-09-16T03:44:32.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.873+0000 7f317a4df640 10 monclient(hunting): _renew_subs 2024-09-16T03:44:32.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.873+0000 7f317a4df640 10 monclient(hunting): authenticate will time out at 1119.844172s 2024-09-16T03:44:32.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.873+0000 7f3173fff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f31740a39b0 0x7f3174133b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:44:32.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.873+0000 7f316bfff640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f31740a3d90 0x7f3174137000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:49546/0 (socket says 172.21.15.42:49546) 2024-09-16T03:44:32.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.873+0000 7f316bfff640 1 -- 172.21.15.42:0/3279987791 learned_addr learned my addr 172.21.15.42:0/3279987791 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:44:32.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.873+0000 7f3173fff640 10 monclient(hunting): get_auth_request con 0x7f31740a39b0 auth_method 0 2024-09-16T03:44:32.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.873+0000 7f3173fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:44:32.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.873+0000 7f3173fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:44:32.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.873+0000 7f3173fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:44:32.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.873+0000 7f3173fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:44:32.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.873+0000 7f3173fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:44:32.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.873+0000 7f3173fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:44:32.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.873+0000 7f31737fe640 1 -- 172.21.15.42:0/3279987791 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3986378468 0 0) 0x7f317409c140 con 0x7f31740a3d90 2024-09-16T03:44:32.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.873+0000 7f31737fe640 10 monclient(hunting): _negotiate 2024-09-16T03:44:32.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.873+0000 7f31737fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:44:32.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.873+0000 7f31737fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:44:32.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.873+0000 7f31737fe640 10 monclient(hunting): my global_id is 25203 2024-09-16T03:44:32.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.873+0000 7f31737fe640 1 -- 172.21.15.42:0/3279987791 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3160003530 con 0x7f31740a3d90 2024-09-16T03:44:32.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.873+0000 7f3173fff640 10 monclient(hunting): handle_auth_done global_id 11330 payload 274 2024-09-16T03:44:32.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.873+0000 7f3173fff640 1 -- 172.21.15.42:0/3279987791 >> v2:172.21.15.154:3300/0 conn(0x7f3174137710 msgr2=0x7f317413f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:44:32.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.873+0000 7f3173fff640 1 --2- 172.21.15.42:0/3279987791 >> v2:172.21.15.154:3300/0 conn(0x7f3174137710 0x7f317413f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:44:32.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.873+0000 7f3173fff640 1 -- 172.21.15.42:0/3279987791 >> v1:172.21.15.42:6789/0 conn(0x7f31740a3d90 legacy=0x7f3174137000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:44:32.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.873+0000 7f3173fff640 10 monclient: _finish_hunting 0 2024-09-16T03:44:32.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.873+0000 7f3173fff640 1 monclient: found mon.noname-b 2024-09-16T03:44:32.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.873+0000 7f3173fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:44:32.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.873+0000 7f3173fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T03:44:32.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.873+0000 7f3173fff640 1 -- 172.21.15.42:0/3279987791 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31740a5e20 con 0x7f31740a39b0 2024-09-16T03:44:32.888 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.873+0000 7f3173fff640 1 --2- 172.21.15.42:0/3279987791 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f31740a39b0 0x7f3174133b80 secure :-1 s=READY pgs=1894 cs=0 l=1 rev1=1 crypto rx=0x7f31640097c0 tx=0x7f316401c380 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e0714e12dbb7c134 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:44:32.888 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.873+0000 7f31737fe640 1 -- 172.21.15.42:0/3279987791 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31640b0070 con 0x7f31740a39b0 2024-09-16T03:44:32.888 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.873+0000 7f31737fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:44:32.888 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.873+0000 7f31737fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:44:32.888 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.873+0000 7f31737fe640 10 monclient: dump: 2024-09-16T03:44:32.889 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:44:32.889 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:44:32.889 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:32.889 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:32.889 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:44:32.889 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:44:32.889 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:44:32.889 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:44:32.889 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:44:32.889 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:44:32.889 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.873+0000 7f31737fe640 10 monclient: _finish_auth 0 2024-09-16T03:44:32.890 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.873+0000 7f31737fe640 10 monclient: _check_auth_tickets 2024-09-16T03:44:32.890 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.873+0000 7f31737fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:44:32.890 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.873+0000 7f31737fe640 1 -- 172.21.15.42:0/3279987791 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f31640ab600 con 0x7f31740a39b0 2024-09-16T03:44:32.890 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.873+0000 7f317a4df640 5 monclient: authenticate success, global_id 11330 2024-09-16T03:44:32.890 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.873+0000 7f31737fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:44:32.890 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.873+0000 7f317a4df640 10 monclient: get_monmap_and_config success 2024-09-16T03:44:32.890 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.873+0000 7f317a4df640 4 set_mon_vals no callback set 2024-09-16T03:44:32.890 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.873+0000 7f31737fe640 1 -- 172.21.15.42:0/3279987791 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31640aba10 con 0x7f31740a39b0 2024-09-16T03:44:32.890 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.873+0000 7f31737fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:44:32.890 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.873+0000 7f31737fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:44:32.890 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.873+0000 7f31737fe640 10 monclient: dump: 2024-09-16T03:44:32.890 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:44:32.890 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:44:32.891 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:32.891 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:32.891 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:44:32.891 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:44:32.891 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:44:32.891 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:44:32.891 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:44:32.891 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:44:32.891 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.877+0000 7f317a4df640 10 monclient: shutdown 2024-09-16T03:44:32.891 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.877+0000 7f317a4df640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:44:32.891 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.877+0000 7f317a4df640 1 -- 172.21.15.42:0/3279987791 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f31740a39b0 msgr2=0x7f3174133b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:44:32.891 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.877+0000 7f317a4df640 1 --2- 172.21.15.42:0/3279987791 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f31740a39b0 0x7f3174133b80 secure :-1 s=READY pgs=1894 cs=0 l=1 rev1=1 crypto rx=0x7f31640097c0 tx=0x7f316401c380 comp rx=0 tx=0).stop 2024-09-16T03:44:32.891 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.877+0000 7f317a4df640 1 -- 172.21.15.42:0/3279987791 shutdown_connections 2024-09-16T03:44:32.892 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.877+0000 7f317a4df640 1 --2- 172.21.15.42:0/3279987791 >> v2:172.21.15.154:3300/0 conn(0x7f3174137710 0x7f317413f3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:44:32.892 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.877+0000 7f317a4df640 1 --2- 172.21.15.42:0/3279987791 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f31740a39b0 0x7f3174133b80 unknown :-1 s=CLOSED pgs=1894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:44:32.892 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.877+0000 7f317a4df640 1 -- 172.21.15.42:0/3279987791 >> 172.21.15.42:0/3279987791 conn(0x7f31740a9fb0 msgr2=0x7f31740aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:44:32.892 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.877+0000 7f317a4df640 1 -- 172.21.15.42:0/3279987791 shutdown_connections 2024-09-16T03:44:32.892 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.877+0000 7f317a4df640 1 -- 172.21.15.42:0/3279987791 wait complete. 2024-09-16T03:44:32.892 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.877+0000 7f317a4df640 10 monclient: build_initial_monmap 2024-09-16T03:44:32.892 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.877+0000 7f317a4df640 10 monclient: monmap: 2024-09-16T03:44:32.892 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:44:32.892 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:44:32.892 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:44:32.892 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:44:32.892 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:44:32.893 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:44:32.893 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:44:32.893 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:44:32.893 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:44:32.893 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:44:32.893 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.877+0000 7f317a4df640 1 librados: starting msgr at 2024-09-16T03:44:32.893 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.877+0000 7f317a4df640 1 librados: starting objecter 2024-09-16T03:44:32.893 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.877+0000 7f317a4df640 1 Processor -- start 2024-09-16T03:44:32.893 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.877+0000 7f317a4df640 1 -- start start 2024-09-16T03:44:32.893 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.877+0000 7f317a4df640 1 librados: setting wanted keys 2024-09-16T03:44:32.893 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.877+0000 7f317a4df640 1 librados: calling monclient init 2024-09-16T03:44:32.894 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.877+0000 7f317a4df640 10 monclient: init 2024-09-16T03:44:32.894 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.877+0000 7f317a4df640 10 monclient: _reopen_session rank -1 2024-09-16T03:44:32.894 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.877+0000 7f317a4df640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:44:32.894 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.877+0000 7f317a4df640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f31740a39b0 0x7f3174130560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:44:32.894 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.877+0000 7f317a4df640 10 monclient(hunting): picked mon.noname-b con 0x7f31740a39b0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:44:32.894 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.877+0000 7f317a4df640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f31740a3d90 0x7f3174130aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:44:32.894 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.877+0000 7f317a4df640 10 monclient(hunting): picked mon.noname-a con 0x7f31740a3d90 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:44:32.894 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.877+0000 7f317a4df640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3174137710 0x7f31740a1810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:44:32.894 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.877+0000 7f317a4df640 10 monclient(hunting): picked mon.noname-c con 0x7f3174137710 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:44:32.894 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.877+0000 7f317a4df640 10 monclient(hunting): start opening mon connection 2024-09-16T03:44:32.894 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.877+0000 7f317a4df640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f3174142310 con 0x7f31740a3d90 2024-09-16T03:44:32.894 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.877+0000 7f317a4df640 10 monclient(hunting): start opening mon connection 2024-09-16T03:44:32.895 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.877+0000 7f317a4df640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f3174142490 con 0x7f31740a39b0 2024-09-16T03:44:32.895 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.877+0000 7f317a4df640 10 monclient(hunting): start opening mon connection 2024-09-16T03:44:32.895 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.877+0000 7f317a4df640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f317406f680 con 0x7f3174137710 2024-09-16T03:44:32.895 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.877+0000 7f317a4df640 10 monclient(hunting): _renew_subs 2024-09-16T03:44:32.895 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.877+0000 7f317a4df640 10 monclient(hunting): authenticate will time out at 1119.850357s 2024-09-16T03:44:32.895 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.877+0000 7f316bfff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f31740a3d90 0x7f3174130aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:44:32.895 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.877+0000 7f316bfff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f31740a3d90 0x7f3174130aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:35110/0 (socket says 172.21.15.42:35110) 2024-09-16T03:44:32.895 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.877+0000 7f316bfff640 1 -- 172.21.15.42:0/2368931436 learned_addr learned my addr 172.21.15.42:0/2368931436 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:44:32.895 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.877+0000 7f316bfff640 10 monclient(hunting): get_auth_request con 0x7f31740a3d90 auth_method 0 2024-09-16T03:44:32.895 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.877+0000 7f316bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:44:32.895 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.877+0000 7f316bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:44:32.895 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.877+0000 7f316bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:44:32.896 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.877+0000 7f316bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:44:32.896 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.877+0000 7f316bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:44:32.896 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.877+0000 7f316bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:44:32.896 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.877+0000 7f3178a7c640 1 --2- 172.21.15.42:0/2368931436 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3174137710 0x7f31740a1810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:44:32.896 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.877+0000 7f3178a7c640 10 monclient(hunting): get_auth_request con 0x7f3174137710 auth_method 0 2024-09-16T03:44:32.896 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.877+0000 7f3178a7c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:44:32.896 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.877+0000 7f3178a7c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:44:32.896 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.877+0000 7f3178a7c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:44:32.897 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.877+0000 7f316bfff640 10 monclient(hunting): handle_auth_done global_id 25233 payload 824 2024-09-16T03:44:32.897 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.877+0000 7f316bfff640 1 -- 172.21.15.42:0/2368931436 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3174137710 msgr2=0x7f31740a1810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:44:32.897 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.877+0000 7f316bfff640 1 --2- 172.21.15.42:0/2368931436 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3174137710 0x7f31740a1810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:44:32.897 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.877+0000 7f316bfff640 1 -- 172.21.15.42:0/2368931436 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f31740a39b0 msgr2=0x7f3174130560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:44:32.897 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.877+0000 7f316bfff640 1 --2- 172.21.15.42:0/2368931436 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f31740a39b0 0x7f3174130560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:44:32.897 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.877+0000 7f316bfff640 10 monclient: _finish_hunting 0 2024-09-16T03:44:32.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.877+0000 7f316bfff640 1 monclient: found mon.noname-a 2024-09-16T03:44:32.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.877+0000 7f316bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:44:32.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.877+0000 7f316bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:44:32.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.877+0000 7f316bfff640 1 -- 172.21.15.42:0/2368931436 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31740a3410 con 0x7f31740a3d90 2024-09-16T03:44:32.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.877+0000 7f316bfff640 1 --2- 172.21.15.42:0/2368931436 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f31740a3d90 0x7f3174130aa0 secure :-1 s=READY pgs=6838 cs=0 l=1 rev1=1 crypto rx=0x7f315c00c450 tx=0x7f315c00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:44:32.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.877+0000 7f31717fa640 10 client.?.objecter ms_handle_connect 0x7f31740a3d90 2024-09-16T03:44:32.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.877+0000 7f31717fa640 10 client.?.objecter resend_mon_ops 2024-09-16T03:44:32.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.877+0000 7f31717fa640 1 -- 172.21.15.42:0/2368931436 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f315c017070 con 0x7f31740a3d90 2024-09-16T03:44:32.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.877+0000 7f31717fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:44:32.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.877+0000 7f31717fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:44:32.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.877+0000 7f31717fa640 10 monclient: dump: 2024-09-16T03:44:32.899 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:44:32.900 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:44:32.900 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:32.900 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:32.900 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:44:32.900 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:44:32.900 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:44:32.901 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:44:32.901 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:44:32.901 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:44:32.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.877+0000 7f31717fa640 10 monclient: _finish_auth 0 2024-09-16T03:44:32.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.877+0000 7f31717fa640 10 monclient: _check_auth_tickets 2024-09-16T03:44:32.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.877+0000 7f31717fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:44:32.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.877+0000 7f31717fa640 1 -- 172.21.15.42:0/2368931436 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f315c005db0 con 0x7f31740a3d90 2024-09-16T03:44:32.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.877+0000 7f31717fa640 10 monclient: handle_config config(0 keys) 2024-09-16T03:44:32.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.877+0000 7f31717fa640 1 -- 172.21.15.42:0/2368931436 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f315c007290 con 0x7f31740a3d90 2024-09-16T03:44:32.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.877+0000 7f31717fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:44:32.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.877+0000 7f31717fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:44:32.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.877+0000 7f31717fa640 10 monclient: dump: 2024-09-16T03:44:32.902 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:44:32.902 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:44:32.902 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:32.902 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:32.902 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:44:32.902 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:44:32.902 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:44:32.902 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:44:32.902 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:44:32.902 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:44:32.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.877+0000 7f317a4df640 5 monclient: authenticate success, global_id 25233 2024-09-16T03:44:32.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.877+0000 7f317a4df640 10 monclient: _renew_subs 2024-09-16T03:44:32.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.877+0000 7f317a4df640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:44:32.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.877+0000 7f317a4df640 1 -- 172.21.15.42:0/2368931436 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f317413f5c0 con 0x7f31740a3d90 2024-09-16T03:44:32.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.877+0000 7f3172ffd640 4 set_mon_vals no callback set 2024-09-16T03:44:32.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.877+0000 7f317a4df640 10 client.25233.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:44:32.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.877+0000 7f317a4df640 10 monclient: _renew_subs 2024-09-16T03:44:32.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.877+0000 7f317a4df640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:44:32.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.877+0000 7f317a4df640 1 -- 172.21.15.42:0/2368931436 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f317413efa0 con 0x7f31740a3d90 2024-09-16T03:44:32.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.877+0000 7f317a4df640 1 librados: init done 2024-09-16T03:44:32.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.881+0000 7f317a4df640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:44:32.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.881+0000 7f317a4df640 1 --2- 172.21.15.42:0/2368931436 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3174062ea0 0x7f3174063280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:44:32.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.881+0000 7f317a4df640 10 monclient(hunting): start opening mon connection 2024-09-16T03:44:32.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.881+0000 7f317a4df640 1 -- 172.21.15.42:0/2368931436 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f31741342d0 con 0x7f3174062ea0 2024-09-16T03:44:32.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.881+0000 7f31717fa640 1 -- 172.21.15.42:0/2368931436 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f315c01a070 con 0x7f31740a3d90 2024-09-16T03:44:32.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.881+0000 7f3173fff640 1 --2- 172.21.15.42:0/2368931436 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3174062ea0 0x7f3174063280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:44:32.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.881+0000 7f3173fff640 10 monclient: get_auth_request con 0x7f3174062ea0 auth_method 0 2024-09-16T03:44:32.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.881+0000 7f3173fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:44:32.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.881+0000 7f3173fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:44:32.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.881+0000 7f3173fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:44:32.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.881+0000 7f3173fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:44:32.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.881+0000 7f3173fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:44:32.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.881+0000 7f3173fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:44:32.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.881+0000 7f31717fa640 1 --2- 172.21.15.42:0/2368931436 >> v2:172.21.15.154:6812/254845342 conn(0x7f3154037810 0x7f3154039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:44:32.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.881+0000 7f3173fff640 10 monclient(hunting): handle_auth_done global_id 11363 payload 824 2024-09-16T03:44:32.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.881+0000 7f3173fff640 1 -- 172.21.15.42:0/2368931436 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3174133b80 con 0x7f3174062ea0 2024-09-16T03:44:32.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.881+0000 7f31717fa640 1 -- 172.21.15.42:0/2368931436 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 10680+0+0 (secure 0 0 0) 0x7f315c049680 con 0x7f31740a3d90 2024-09-16T03:44:32.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.881+0000 7f31717fa640 10 client.25233.objecter ms_dispatch 0x7f3174000ba0 osd_map(54..54 src has 1..54) 2024-09-16T03:44:32.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.881+0000 7f31717fa640 3 client.25233.objecter handle_osd_map got epochs [54,54] > 0 2024-09-16T03:44:32.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.881+0000 7f31717fa640 3 client.25233.objecter handle_osd_map decoding full epoch 54 2024-09-16T03:44:32.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.881+0000 7f31717fa640 20 client.25233.objecter dump_active .. 0 homeless 2024-09-16T03:44:32.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.881+0000 7f31717fa640 10 client.25233.objecter ms_handle_connect 0x7f3174062ea0 2024-09-16T03:44:32.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.881+0000 7f31717fa640 10 client.25233.objecter resend_mon_ops 2024-09-16T03:44:32.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.881+0000 7f3173fff640 1 --2- 172.21.15.42:0/2368931436 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3174062ea0 0x7f3174063280 secure :-1 s=READY pgs=1901 cs=0 l=1 rev1=1 crypto rx=0x7f317406e3c0 tx=0x7f3164019120 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:44:32.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.881+0000 7f31717fa640 1 -- 172.21.15.42:0/2368931436 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31640bb020 con 0x7f3174062ea0 2024-09-16T03:44:32.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.881+0000 7f31717fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:44:32.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.881+0000 7f31717fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:44:32.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.881+0000 7f31717fa640 10 monclient: dump: 2024-09-16T03:44:32.905 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:44:32.906 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:44:32.906 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:32.906 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:32.906 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:44:32.906 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:44:32.906 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:44:32.906 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:44:32.906 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:44:32.906 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:44:32.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.881+0000 7f3178a7c640 1 --2- 172.21.15.42:0/2368931436 >> v2:172.21.15.154:6812/254845342 conn(0x7f3154037810 0x7f3154039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:44:32.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.881+0000 7f3178a7c640 10 monclient: get_auth_request con 0x7f3154037810 auth_method 0 2024-09-16T03:44:32.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.881+0000 7f31717fa640 1 -- 172.21.15.42:0/2368931436 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f31640b1040 con 0x7f3174062ea0 2024-09-16T03:44:32.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.881+0000 7f31717fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:44:32.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.881+0000 7f31717fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:44:32.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.881+0000 7f31717fa640 1 -- 172.21.15.42:0/2368931436 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3174062ea0 msgr2=0x7f3174063280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:44:32.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.881+0000 7f31717fa640 1 --2- 172.21.15.42:0/2368931436 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3174062ea0 0x7f3174063280 secure :-1 s=READY pgs=1901 cs=0 l=1 rev1=1 crypto rx=0x7f317406e3c0 tx=0x7f3164019120 comp rx=0 tx=0).stop 2024-09-16T03:44:32.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.881+0000 7f31717fa640 1 -- 172.21.15.42:0/2368931436 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3174062ea0 msgr2=0x7f3174063280 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:44:32.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.881+0000 7f31717fa640 1 --2- 172.21.15.42:0/2368931436 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3174062ea0 0x7f3174063280 secure :-1 s=CLOSED pgs=1901 cs=0 l=1 rev1=1 crypto rx=0x7f317406e3c0 tx=0x7f3164019120 comp rx=0 tx=0).stop 2024-09-16T03:44:32.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.881+0000 7f3178a7c640 1 --2- 172.21.15.42:0/2368931436 >> v2:172.21.15.154:6812/254845342 conn(0x7f3154037810 0x7f3154039cd0 secure :-1 s=READY pgs=2954 cs=0 l=1 rev1=1 crypto rx=0x7f316c009ea0 tx=0x7f316c009410 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:44:32.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.881+0000 7f31717fa640 10 client.25233.objecter ms_handle_connect 0x7f3154037810 2024-09-16T03:44:32.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.889+0000 7f317a4df640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:44:32.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.889+0000 7f317a4df640 1 --2- 172.21.15.42:0/2368931436 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f317402c650 0x7f317402ca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:44:32.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.889+0000 7f317a4df640 10 monclient(hunting): start opening mon connection 2024-09-16T03:44:32.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.889+0000 7f317a4df640 1 -- 172.21.15.42:0/2368931436 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f3174137090 con 0x7f317402c650 2024-09-16T03:44:32.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.889+0000 7f3173fff640 1 --2- 172.21.15.42:0/2368931436 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f317402c650 0x7f317402ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:44:32.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.889+0000 7f3173fff640 10 monclient: get_auth_request con 0x7f317402c650 auth_method 0 2024-09-16T03:44:32.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.889+0000 7f3173fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:44:32.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.889+0000 7f3173fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:44:32.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.889+0000 7f3173fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:44:32.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.889+0000 7f3173fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:44:32.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.889+0000 7f3173fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:44:32.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.889+0000 7f3173fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:44:32.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.889+0000 7f3173fff640 10 monclient(hunting): handle_auth_done global_id 11366 payload 824 2024-09-16T03:44:32.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.889+0000 7f3173fff640 1 -- 172.21.15.42:0/2368931436 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f31740a5750 con 0x7f317402c650 2024-09-16T03:44:32.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.889+0000 7f31717fa640 10 client.25233.objecter ms_handle_connect 0x7f317402c650 2024-09-16T03:44:32.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.889+0000 7f31717fa640 10 client.25233.objecter resend_mon_ops 2024-09-16T03:44:32.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.889+0000 7f3173fff640 1 --2- 172.21.15.42:0/2368931436 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f317402c650 0x7f317402ca60 secure :-1 s=READY pgs=1903 cs=0 l=1 rev1=1 crypto rx=0x7f3164000c10 tx=0x7f31640a8000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:44:32.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.889+0000 7f31717fa640 1 -- 172.21.15.42:0/2368931436 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31640bb070 con 0x7f317402c650 2024-09-16T03:44:32.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.889+0000 7f31717fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:44:32.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.889+0000 7f31717fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:44:32.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.889+0000 7f31717fa640 10 monclient: dump: 2024-09-16T03:44:32.911 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:44:32.911 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:44:32.911 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:32.912 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:32.912 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:44:32.912 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:44:32.912 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:44:32.912 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:44:32.912 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:44:32.912 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:44:32.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.889+0000 7f31717fa640 1 -- 172.21.15.42:0/2368931436 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4880 (secure 0 0 0) 0x7f31640c8480 con 0x7f317402c650 2024-09-16T03:44:32.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.889+0000 7f31717fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:44:32.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.889+0000 7f31717fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:44:32.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.889+0000 7f31717fa640 1 -- 172.21.15.42:0/2368931436 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f317402c650 msgr2=0x7f317402ca60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:44:32.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.889+0000 7f31717fa640 1 --2- 172.21.15.42:0/2368931436 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f317402c650 0x7f317402ca60 secure :-1 s=READY pgs=1903 cs=0 l=1 rev1=1 crypto rx=0x7f3164000c10 tx=0x7f31640a8000 comp rx=0 tx=0).stop 2024-09-16T03:44:32.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.889+0000 7f31717fa640 1 -- 172.21.15.42:0/2368931436 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f317402c650 msgr2=0x7f317402ca60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:44:32.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.889+0000 7f31717fa640 1 --2- 172.21.15.42:0/2368931436 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f317402c650 0x7f317402ca60 secure :-1 s=CLOSED pgs=1903 cs=0 l=1 rev1=1 crypto rx=0x7f3164000c10 tx=0x7f31640a8000 comp rx=0 tx=0).stop 2024-09-16T03:44:32.913 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:44:32.913 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:44:32.913 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:44:32.913 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T03:44:32.913 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 22, 2024-09-16T03:44:32.913 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 152259, 2024-09-16T03:44:32.913 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:44:32.913 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:44:32.914 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T03:44:32.914 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T03:44:32.914 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:44:32.914 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 0, 2024-09-16T03:44:32.914 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:44:32.914 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:44:32.914 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:44:32.914 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:44:32.914 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:44:32.914 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:44:32.914 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:44:32.914 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:44:32.914 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:44:32.914 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:44:32.915 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:44:32.915 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:44:32.915 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:44:32.915 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:44:32.915 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:44:32.915 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:44:32.915 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:44:32.915 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:44:32.915 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:44:32.915 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:44:32.916 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:44:32.916 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:44:32.916 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:44:32.916 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:44:32.916 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:44:32.916 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:44:32.916 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:44:32.916 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:44:32.916 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:44:32.916 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:44:32.916 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:44:32.916 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:44:32.917 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:44:32.917 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:44:32.917 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:44:32.917 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:44:32.917 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:44:32.917 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:44:32.917 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:44:32.917 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:44:32.917 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:44:32.917 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:44:32.917 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:44:32.917 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:44:32.917 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:44:32.918 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:44:32.918 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:44:32.918 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:44:32.918 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:44:32.918 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:44:32.918 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:44:32.918 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:44:32.918 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:44:32.918 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:44:32.918 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:44:32.918 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:44:32.918 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:44:32.918 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:44:32.918 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:44:32.919 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:44:32.919 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:44:32.919 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:44:32.919 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:44:32.919 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:44:32.919 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:44:32.919 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:44:32.919 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:44:32.919 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:44:32.919 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:44:32.919 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:44:32.919 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:44:32.919 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:44:32.919 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:44:32.920 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:44:32.920 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:44:32.920 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:44:32.920 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:44:32.920 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:44:32.920 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:44:32.920 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:44:32.920 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:44:32.920 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:44:32.920 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:44:32.920 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:44:32.920 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:44:32.920 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:44:32.921 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:44:32.921 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:44:32.921 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:44:32.921 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:44:32.921 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:44:32.921 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:44:32.921 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:44:32.921 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:44:32.921 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:44:32.921 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:44:32.921 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:44:32.921 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:44:32.921 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:44:32.921 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:44:32.922 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:44:32.922 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:44:32.922 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:44:32.922 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:44:32.922 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:44:32.922 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:44:32.922 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:44:32.922 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:44:32.922 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:44:32.922 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:44:32.922 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:44:32.922 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:44:32.922 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:44:32.922 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:44:32.923 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:44:32.923 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:44:32.923 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:44:32.923 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:44:32.923 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:44:32.923 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:44:32.923 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:44:32.923 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:44:32.923 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:44:32.923 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:44:32.923 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:44:32.923 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:44:32.923 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:44:32.924 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:44:32.924 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:44:32.924 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:44:32.924 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:44:32.924 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:44:32.924 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:44:32.924 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:44:32.924 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:44:32.924 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:44:32.925 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:44:32.925 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:44:32.925 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:44:32.925 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:44:32.925 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:44:32.925 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T03:44:32.925 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:44:32.925 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:44:32.926 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:44:32.926 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 2 2024-09-16T03:44:32.926 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:44:32.926 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:44:32.926 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:44:32.926 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:44:32.926 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:44:32.926 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:44:32.926 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 25 2024-09-16T03:44:32.926 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:44:32.926 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:44:32.926 INFO:teuthology.orchestra.run.smithi042.stdout: "mgr": [ 2024-09-16T03:44:32.926 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:44:32.926 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:44:32.927 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:44:32.927 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:44:32.927 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:44:32.927 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:44:32.927 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:44:32.927 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:44:32.927 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:44:32.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.889+0000 7f317a4df640 10 librados: watch_flush enter 2024-09-16T03:44:32.929 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.889+0000 7f317a4df640 10 librados: watch_flush exit 2024-09-16T03:44:32.929 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.889+0000 7f317a4df640 20 client.25233.objecter shutdown clearing up homeless session... 2024-09-16T03:44:32.929 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.889+0000 7f317a4df640 10 client.25233.objecter successfully canceled tick 2024-09-16T03:44:32.929 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.889+0000 7f317a4df640 1 -- 172.21.15.42:0/2368931436 >> v2:172.21.15.154:6812/254845342 conn(0x7f3154037810 msgr2=0x7f3154039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:44:32.929 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.889+0000 7f317a4df640 1 --2- 172.21.15.42:0/2368931436 >> v2:172.21.15.154:6812/254845342 conn(0x7f3154037810 0x7f3154039cd0 secure :-1 s=READY pgs=2954 cs=0 l=1 rev1=1 crypto rx=0x7f316c009ea0 tx=0x7f316c009410 comp rx=0 tx=0).stop 2024-09-16T03:44:32.929 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.889+0000 7f317a4df640 10 monclient: shutdown 2024-09-16T03:44:32.929 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.889+0000 7f317a4df640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:44:32.929 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.889+0000 7f317a4df640 1 -- 172.21.15.42:0/2368931436 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f31740a3d90 msgr2=0x7f3174130aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:44:32.929 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.889+0000 7f317a4df640 1 --2- 172.21.15.42:0/2368931436 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f31740a3d90 0x7f3174130aa0 secure :-1 s=READY pgs=6838 cs=0 l=1 rev1=1 crypto rx=0x7f315c00c450 tx=0x7f315c00c910 comp rx=0 tx=0).stop 2024-09-16T03:44:32.929 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.889+0000 7f3178a7c640 1 -- 172.21.15.42:0/2368931436 reap_dead start 2024-09-16T03:44:32.929 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.889+0000 7f3178a7c640 1 -- 172.21.15.42:0/2368931436 reap_dead start 2024-09-16T03:44:32.929 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.889+0000 7f317a4df640 1 -- 172.21.15.42:0/2368931436 shutdown_connections 2024-09-16T03:44:32.930 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.889+0000 7f317a4df640 1 -- 172.21.15.42:0/2368931436 >> 172.21.15.42:0/2368931436 conn(0x7f31740a9fb0 msgr2=0x7f31741321d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:44:32.930 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.889+0000 7f317a4df640 1 -- 172.21.15.42:0/2368931436 shutdown_connections 2024-09-16T03:44:32.930 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.889+0000 7f317a4df640 1 -- 172.21.15.42:0/2368931436 wait complete. 2024-09-16T03:44:32.930 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:32.889+0000 7f317a4df640 1 librados: shutdown 2024-09-16T03:44:32.930 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:44:32.930 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T03:44:32.930 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T03:44:33.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.049+0000 7f086a9ce640 10 monclient: get_monmap_and_config 2024-09-16T03:44:33.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.049+0000 7f086a9ce640 10 monclient: build_initial_monmap 2024-09-16T03:44:33.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.049+0000 7f086a9ce640 10 monclient: monmap: 2024-09-16T03:44:33.055 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:44:33.055 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:44:33.055 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:44:33.054352+0000 2024-09-16T03:44:33.055 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:44:33.054352+0000 2024-09-16T03:44:33.056 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:44:33.056 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:44:33.056 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:44:33.056 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:44:33.056 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:44:33.056 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:44:33.056 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:44:33.056 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:44:33.059 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.049+0000 7f086a9ce640 1 Processor -- start 2024-09-16T03:44:33.059 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.049+0000 7f086a9ce640 1 -- start start 2024-09-16T03:44:33.059 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.049+0000 7f086a9ce640 10 monclient: init 2024-09-16T03:44:33.059 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.049+0000 7f086a9ce640 10 monclient: _reopen_session rank -1 2024-09-16T03:44:33.059 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.049+0000 7f086a9ce640 10 monclient: _add_conns ranks=[3,0,2,4,1] 2024-09-16T03:44:33.059 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.049+0000 7f086a9ce640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f086406db90 0x7f08641340c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:44:33.059 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.049+0000 7f086a9ce640 10 monclient(hunting): picked mon.noname-a con 0x7f086406db90 addr v2:172.21.15.42:3300/0 2024-09-16T03:44:33.059 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.049+0000 7f086a9ce640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f08640a37f0 addr v1:172.21.15.42:6789/0 2024-09-16T03:44:33.059 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.049+0000 7f086a9ce640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f08640a3c00 0x7f0864137520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:44:33.059 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.049+0000 7f086a9ce640 10 monclient(hunting): picked mon.noname-b con 0x7f08640a3c00 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:44:33.059 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.049+0000 7f086a9ce640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f086406ece0 con 0x7f08640a37f0 2024-09-16T03:44:33.060 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.049+0000 7f086a9ce640 10 monclient(hunting): start opening mon connection 2024-09-16T03:44:33.060 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.049+0000 7f086a9ce640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f086406df70 con 0x7f086406db90 2024-09-16T03:44:33.060 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.049+0000 7f086a9ce640 10 monclient(hunting): start opening mon connection 2024-09-16T03:44:33.060 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.049+0000 7f086a9ce640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f086409bf80 con 0x7f08640a3c00 2024-09-16T03:44:33.060 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.049+0000 7f086a9ce640 10 monclient(hunting): _renew_subs 2024-09-16T03:44:33.060 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.049+0000 7f086a9ce640 10 monclient(hunting): authenticate will time out at 1120.023399s 2024-09-16T03:44:33.060 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.049+0000 7f0863fff640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f086406db90 0x7f08641340c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:44:33.060 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.049+0000 7f0868f6b640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f08640a3c00 0x7f0864137520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:44:33.060 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.049+0000 7f08637fe640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f08640a37f0 0x7f0864136e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:49552/0 (socket says 172.21.15.42:49552) 2024-09-16T03:44:33.060 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.049+0000 7f08637fe640 1 -- 172.21.15.42:0/2648821470 learned_addr learned my addr 172.21.15.42:0/2648821470 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:44:33.060 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.049+0000 7f0868f6b640 10 monclient(hunting): get_auth_request con 0x7f08640a3c00 auth_method 0 2024-09-16T03:44:33.060 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.049+0000 7f0868f6b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:44:33.060 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.049+0000 7f0868f6b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:44:33.061 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.049+0000 7f0868f6b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:44:33.061 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.049+0000 7f0863fff640 10 monclient(hunting): get_auth_request con 0x7f086406db90 auth_method 0 2024-09-16T03:44:33.061 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.049+0000 7f0863fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:44:33.061 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.049+0000 7f0863fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:44:33.061 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.049+0000 7f0863fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:44:33.061 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.049+0000 7f0868f6b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:44:33.061 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.049+0000 7f0868f6b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:44:33.061 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.049+0000 7f0868f6b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:44:33.061 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.049+0000 7f0863fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:44:33.061 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.049+0000 7f0863fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:44:33.061 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.049+0000 7f0863fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:44:33.061 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.049+0000 7f0868f6b640 10 monclient(hunting): handle_auth_done global_id 11540 payload 274 2024-09-16T03:44:33.061 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.049+0000 7f0868f6b640 1 -- 172.21.15.42:0/2648821470 >> v2:172.21.15.42:3300/0 conn(0x7f086406db90 msgr2=0x7f08641340c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:44:33.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.049+0000 7f0868f6b640 1 --2- 172.21.15.42:0/2648821470 >> v2:172.21.15.42:3300/0 conn(0x7f086406db90 0x7f08641340c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:44:33.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.049+0000 7f0868f6b640 1 -- 172.21.15.42:0/2648821470 >> v1:172.21.15.42:6789/0 conn(0x7f08640a37f0 legacy=0x7f0864136e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:44:33.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.049+0000 7f0868f6b640 10 monclient: _finish_hunting 0 2024-09-16T03:44:33.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.049+0000 7f0868f6b640 1 monclient: found mon.noname-b 2024-09-16T03:44:33.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.049+0000 7f0868f6b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:44:33.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.049+0000 7f0868f6b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T03:44:33.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.049+0000 7f0868f6b640 1 -- 172.21.15.42:0/2648821470 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0864134780 con 0x7f08640a3c00 2024-09-16T03:44:33.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.049+0000 7f0863fff640 1 --2- 172.21.15.42:0/2648821470 >> v2:172.21.15.42:3300/0 conn(0x7f086406db90 0x7f08641340c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:44:33.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.053+0000 7f0868f6b640 1 --2- 172.21.15.42:0/2648821470 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f08640a3c00 0x7f0864137520 secure :-1 s=READY pgs=1942 cs=0 l=1 rev1=1 crypto rx=0x7f085c0097c0 tx=0x7f085c01cdc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=82a910f8c89062e6 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:44:33.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.053+0000 7f0862ffd640 1 -- 172.21.15.42:0/2648821470 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f085c0b0070 con 0x7f08640a3c00 2024-09-16T03:44:33.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.053+0000 7f0862ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:44:33.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.053+0000 7f0862ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:44:33.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.053+0000 7f0862ffd640 10 monclient: dump: 2024-09-16T03:44:33.062 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:44:33.063 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:44:33.063 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:33.063 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:33.063 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:44:33.063 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:44:33.063 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:44:33.063 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:44:33.063 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:44:33.063 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:44:33.063 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.053+0000 7f0862ffd640 10 monclient: _finish_auth 0 2024-09-16T03:44:33.063 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.053+0000 7f0862ffd640 10 monclient: _check_auth_tickets 2024-09-16T03:44:33.063 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.053+0000 7f0862ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:44:33.063 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.053+0000 7f086a9ce640 5 monclient: authenticate success, global_id 11540 2024-09-16T03:44:33.063 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.053+0000 7f086a9ce640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:44:33.064 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.053+0000 7f0862ffd640 1 -- 172.21.15.42:0/2648821470 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f085c0ab600 con 0x7f08640a3c00 2024-09-16T03:44:33.064 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.053+0000 7f0862ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:44:33.064 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.053+0000 7f086a9ce640 10 monclient: get_monmap_and_config success 2024-09-16T03:44:33.064 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.053+0000 7f086a9ce640 4 set_mon_vals no callback set 2024-09-16T03:44:33.064 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.053+0000 7f0862ffd640 1 -- 172.21.15.42:0/2648821470 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f085c0ab9d0 con 0x7f08640a3c00 2024-09-16T03:44:33.064 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.053+0000 7f0862ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:44:33.064 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.053+0000 7f0862ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:44:33.064 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.053+0000 7f0862ffd640 10 monclient: dump: 2024-09-16T03:44:33.064 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:44:33.064 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:44:33.064 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:33.064 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:33.064 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:44:33.065 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:44:33.065 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:44:33.065 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:44:33.065 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:44:33.065 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:44:33.071 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.053+0000 7f086a9ce640 10 monclient: shutdown 2024-09-16T03:44:33.071 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.053+0000 7f086a9ce640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:44:33.071 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.053+0000 7f086a9ce640 1 -- 172.21.15.42:0/2648821470 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f08640a3c00 msgr2=0x7f0864137520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:44:33.071 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.053+0000 7f086a9ce640 1 --2- 172.21.15.42:0/2648821470 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f08640a3c00 0x7f0864137520 secure :-1 s=READY pgs=1942 cs=0 l=1 rev1=1 crypto rx=0x7f085c0097c0 tx=0x7f085c01cdc0 comp rx=0 tx=0).stop 2024-09-16T03:44:33.071 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.053+0000 7f086a9ce640 1 -- 172.21.15.42:0/2648821470 shutdown_connections 2024-09-16T03:44:33.071 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.053+0000 7f086a9ce640 1 --2- 172.21.15.42:0/2648821470 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f08640a3c00 0x7f0864137520 unknown :-1 s=CLOSED pgs=1942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:44:33.071 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.053+0000 7f086a9ce640 1 --2- 172.21.15.42:0/2648821470 >> v2:172.21.15.42:3300/0 conn(0x7f086406db90 0x7f08641340c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:44:33.071 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.053+0000 7f086a9ce640 1 -- 172.21.15.42:0/2648821470 >> 172.21.15.42:0/2648821470 conn(0x7f08640a9fb0 msgr2=0x7f08640aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:44:33.071 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.053+0000 7f086a9ce640 1 -- 172.21.15.42:0/2648821470 shutdown_connections 2024-09-16T03:44:33.072 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.053+0000 7f086a9ce640 1 -- 172.21.15.42:0/2648821470 wait complete. 2024-09-16T03:44:33.072 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.053+0000 7f086a9ce640 10 monclient: build_initial_monmap 2024-09-16T03:44:33.072 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.053+0000 7f086a9ce640 10 monclient: monmap: 2024-09-16T03:44:33.072 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:44:33.072 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:44:33.072 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:44:33.072 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:44:33.072 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:44:33.072 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:44:33.073 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:44:33.073 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:44:33.073 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:44:33.073 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:44:33.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.053+0000 7f086a9ce640 1 librados: starting msgr at 2024-09-16T03:44:33.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.053+0000 7f086a9ce640 1 librados: starting objecter 2024-09-16T03:44:33.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.053+0000 7f086a9ce640 1 Processor -- start 2024-09-16T03:44:33.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.053+0000 7f086a9ce640 1 -- start start 2024-09-16T03:44:33.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.053+0000 7f086a9ce640 1 librados: setting wanted keys 2024-09-16T03:44:33.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.053+0000 7f086a9ce640 1 librados: calling monclient init 2024-09-16T03:44:33.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.053+0000 7f086a9ce640 10 monclient: init 2024-09-16T03:44:33.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.057+0000 7f086a9ce640 10 monclient: _reopen_session rank -1 2024-09-16T03:44:33.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.057+0000 7f086a9ce640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T03:44:33.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.057+0000 7f086a9ce640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f086406db90 0x7f08641ceec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:44:33.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.057+0000 7f086a9ce640 10 monclient(hunting): picked mon.noname-a con 0x7f086406db90 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:44:33.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.057+0000 7f086a9ce640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f08640a37f0 0x7f08641cf400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:44:33.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.057+0000 7f086a9ce640 10 monclient(hunting): picked mon.noname-b con 0x7f08640a37f0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:44:33.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.057+0000 7f086a9ce640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f08640a3c00 0x7f08641cf940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:44:33.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.057+0000 7f086a9ce640 10 monclient(hunting): picked mon.noname-c con 0x7f08640a3c00 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:44:33.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.057+0000 7f086a9ce640 10 monclient(hunting): start opening mon connection 2024-09-16T03:44:33.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.057+0000 7f086a9ce640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f08641419d0 con 0x7f086406db90 2024-09-16T03:44:33.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.057+0000 7f086a9ce640 10 monclient(hunting): start opening mon connection 2024-09-16T03:44:33.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.057+0000 7f086a9ce640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f0864137c70 con 0x7f08640a37f0 2024-09-16T03:44:33.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.057+0000 7f086a9ce640 10 monclient(hunting): start opening mon connection 2024-09-16T03:44:33.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.057+0000 7f086a9ce640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f0864141b50 con 0x7f08640a3c00 2024-09-16T03:44:33.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.057+0000 7f086a9ce640 10 monclient(hunting): _renew_subs 2024-09-16T03:44:33.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.057+0000 7f086a9ce640 10 monclient(hunting): authenticate will time out at 1120.028260s 2024-09-16T03:44:33.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.057+0000 7f0868f6b640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f08640a3c00 0x7f08641cf940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:44:33.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.057+0000 7f0868f6b640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f08640a3c00 0x7f08641cf940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:41232/0 (socket says 172.21.15.42:41232) 2024-09-16T03:44:33.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.057+0000 7f0868f6b640 1 -- 172.21.15.42:0/3694570215 learned_addr learned my addr 172.21.15.42:0/3694570215 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:44:33.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.057+0000 7f0868f6b640 10 monclient(hunting): get_auth_request con 0x7f08640a3c00 auth_method 0 2024-09-16T03:44:33.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.057+0000 7f0868f6b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:44:33.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.057+0000 7f0868f6b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:44:33.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.057+0000 7f0868f6b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:44:33.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.057+0000 7f0863fff640 1 --2- 172.21.15.42:0/3694570215 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f086406db90 0x7f08641ceec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:44:33.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.057+0000 7f0863fff640 10 monclient(hunting): get_auth_request con 0x7f086406db90 auth_method 0 2024-09-16T03:44:33.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.057+0000 7f0863fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:44:33.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.057+0000 7f0863fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:44:33.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.057+0000 7f0863fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:44:33.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.057+0000 7f0868f6b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:44:33.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.057+0000 7f0868f6b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:44:33.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.057+0000 7f0868f6b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:44:33.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.057+0000 7f0863fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:44:33.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.057+0000 7f0863fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:44:33.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.057+0000 7f0863fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:44:33.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.057+0000 7f0868f6b640 10 monclient(hunting): handle_auth_done global_id 11543 payload 824 2024-09-16T03:44:33.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.057+0000 7f0868f6b640 1 -- 172.21.15.42:0/3694570215 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f08640a37f0 msgr2=0x7f08641cf400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:44:33.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.057+0000 7f0868f6b640 1 --2- 172.21.15.42:0/3694570215 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f08640a37f0 0x7f08641cf400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:44:33.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.057+0000 7f0868f6b640 1 -- 172.21.15.42:0/3694570215 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f086406db90 msgr2=0x7f08641ceec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:44:33.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.057+0000 7f0868f6b640 1 --2- 172.21.15.42:0/3694570215 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f086406db90 0x7f08641ceec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:44:33.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.057+0000 7f0868f6b640 10 monclient: _finish_hunting 0 2024-09-16T03:44:33.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.057+0000 7f0868f6b640 1 monclient: found mon.noname-c 2024-09-16T03:44:33.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.057+0000 7f0868f6b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:44:33.077 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.057+0000 7f0868f6b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:44:33.077 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.057+0000 7f0868f6b640 1 -- 172.21.15.42:0/3694570215 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08640a2800 con 0x7f08640a3c00 2024-09-16T03:44:33.077 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.057+0000 7f0868f6b640 1 --2- 172.21.15.42:0/3694570215 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f08640a3c00 0x7f08641cf940 secure :-1 s=READY pgs=1944 cs=0 l=1 rev1=1 crypto rx=0x7f085c019f90 tx=0x7f085c019100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:44:33.077 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.057+0000 7f0860ff9640 10 client.?.objecter ms_handle_connect 0x7f08640a3c00 2024-09-16T03:44:33.077 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.057+0000 7f0860ff9640 10 client.?.objecter resend_mon_ops 2024-09-16T03:44:33.077 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.057+0000 7f0860ff9640 1 -- 172.21.15.42:0/3694570215 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f085c0abe60 con 0x7f08640a3c00 2024-09-16T03:44:33.077 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.057+0000 7f0860ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:44:33.077 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.057+0000 7f0860ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:44:33.077 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.057+0000 7f0860ff9640 10 monclient: dump: 2024-09-16T03:44:33.077 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:44:33.077 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:44:33.077 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:33.077 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:33.077 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:44:33.078 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:44:33.078 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:44:33.078 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:44:33.078 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:44:33.078 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:44:33.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.057+0000 7f0860ff9640 10 monclient: _finish_auth 0 2024-09-16T03:44:33.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.057+0000 7f0860ff9640 10 monclient: _check_auth_tickets 2024-09-16T03:44:33.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.057+0000 7f0860ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:44:33.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.057+0000 7f086a9ce640 5 monclient: authenticate success, global_id 11543 2024-09-16T03:44:33.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.057+0000 7f0860ff9640 1 -- 172.21.15.42:0/3694570215 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f085c0b9070 con 0x7f08640a3c00 2024-09-16T03:44:33.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.057+0000 7f086a9ce640 10 monclient: _renew_subs 2024-09-16T03:44:33.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.057+0000 7f086a9ce640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:44:33.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.057+0000 7f086a9ce640 1 -- 172.21.15.42:0/3694570215 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08640a1f60 con 0x7f08640a3c00 2024-09-16T03:44:33.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.057+0000 7f0860ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T03:44:33.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.057+0000 7f0860ff9640 1 -- 172.21.15.42:0/3694570215 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f085c0b0040 con 0x7f08640a3c00 2024-09-16T03:44:33.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.057+0000 7f0860ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:44:33.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.057+0000 7f0860ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:44:33.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.057+0000 7f0860ff9640 10 monclient: dump: 2024-09-16T03:44:33.079 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:44:33.079 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:44:33.079 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:33.079 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:33.079 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:44:33.079 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:44:33.079 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:44:33.079 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:44:33.079 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:44:33.080 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:44:33.080 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.057+0000 7f086a9ce640 10 client.11543.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:44:33.080 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.057+0000 7f086a9ce640 10 monclient: _renew_subs 2024-09-16T03:44:33.080 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.057+0000 7f086a9ce640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:44:33.080 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.057+0000 7f08627fc640 4 set_mon_vals no callback set 2024-09-16T03:44:33.080 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.057+0000 7f086a9ce640 1 -- 172.21.15.42:0/3694570215 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f08640a2f60 con 0x7f08640a3c00 2024-09-16T03:44:33.080 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.057+0000 7f086a9ce640 1 librados: init done 2024-09-16T03:44:33.080 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.057+0000 7f086a9ce640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:44:33.080 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.057+0000 7f086a9ce640 1 --2- 172.21.15.42:0/3694570215 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f086412d2d0 0x7f086412d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:44:33.080 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.057+0000 7f086a9ce640 10 monclient(hunting): start opening mon connection 2024-09-16T03:44:33.080 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.057+0000 7f086a9ce640 1 -- 172.21.15.42:0/3694570215 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f08640a2130 con 0x7f086412d2d0 2024-09-16T03:44:33.080 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.057+0000 7f0860ff9640 1 -- 172.21.15.42:0/3694570215 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f085c0b4650 con 0x7f08640a3c00 2024-09-16T03:44:33.080 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.057+0000 7f0860ff9640 1 --2- 172.21.15.42:0/3694570215 >> v2:172.21.15.154:6812/254845342 conn(0x7f0850037810 0x7f0850039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:44:33.080 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.057+0000 7f0860ff9640 1 -- 172.21.15.42:0/3694570215 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(54..54 src has 1..54) ==== 10680+0+0 (secure 0 0 0) 0x7f085c0e3e30 con 0x7f08640a3c00 2024-09-16T03:44:33.081 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.057+0000 7f0860ff9640 10 client.11543.objecter ms_dispatch 0x7f0864000ba0 osd_map(54..54 src has 1..54) 2024-09-16T03:44:33.081 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.057+0000 7f0860ff9640 3 client.11543.objecter handle_osd_map got epochs [54,54] > 0 2024-09-16T03:44:33.081 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.057+0000 7f0860ff9640 3 client.11543.objecter handle_osd_map decoding full epoch 54 2024-09-16T03:44:33.081 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.057+0000 7f0860ff9640 20 client.11543.objecter dump_active .. 0 homeless 2024-09-16T03:44:33.081 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.057+0000 7f08637fe640 1 --2- 172.21.15.42:0/3694570215 >> v2:172.21.15.154:6812/254845342 conn(0x7f0850037810 0x7f0850039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:44:33.081 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.057+0000 7f08637fe640 10 monclient: get_auth_request con 0x7f0850037810 auth_method 0 2024-09-16T03:44:33.081 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.061+0000 7f0863fff640 1 --2- 172.21.15.42:0/3694570215 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f086412d2d0 0x7f086412d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:44:33.081 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.061+0000 7f0863fff640 10 monclient: get_auth_request con 0x7f086412d2d0 auth_method 0 2024-09-16T03:44:33.081 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.061+0000 7f0863fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:44:33.081 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.061+0000 7f0863fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:44:33.081 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.061+0000 7f0863fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:44:33.081 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.061+0000 7f0863fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:44:33.081 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.061+0000 7f0863fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:44:33.081 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.061+0000 7f0863fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:44:33.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.061+0000 7f08637fe640 1 --2- 172.21.15.42:0/3694570215 >> v2:172.21.15.154:6812/254845342 conn(0x7f0850037810 0x7f0850039cd0 secure :-1 s=READY pgs=2988 cs=0 l=1 rev1=1 crypto rx=0x7f0858000f70 tx=0x7f0858007a90 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:44:33.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.061+0000 7f0860ff9640 10 client.11543.objecter ms_handle_connect 0x7f0850037810 2024-09-16T03:44:33.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.061+0000 7f0863fff640 10 monclient(hunting): handle_auth_done global_id 271150 payload 824 2024-09-16T03:44:33.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.061+0000 7f0863fff640 1 -- 172.21.15.42:0/3694570215 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f086406f6a0 con 0x7f086412d2d0 2024-09-16T03:44:33.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.061+0000 7f0860ff9640 10 client.11543.objecter ms_handle_connect 0x7f086412d2d0 2024-09-16T03:44:33.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.061+0000 7f0860ff9640 10 client.11543.objecter resend_mon_ops 2024-09-16T03:44:33.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.061+0000 7f0863fff640 1 --2- 172.21.15.42:0/3694570215 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f086412d2d0 0x7f086412d6b0 secure :-1 s=READY pgs=2391 cs=0 l=1 rev1=1 crypto rx=0x7f0854009fc0 tx=0x7f085400f580 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:44:33.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.061+0000 7f0860ff9640 1 -- 172.21.15.42:0/3694570215 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0854019070 con 0x7f086412d2d0 2024-09-16T03:44:33.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.061+0000 7f0860ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:44:33.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.061+0000 7f0860ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:44:33.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.061+0000 7f0860ff9640 10 monclient: dump: 2024-09-16T03:44:33.082 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:44:33.082 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:44:33.083 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:33.083 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:33.083 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:44:33.083 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:44:33.083 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:44:33.083 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:44:33.083 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:44:33.083 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:44:33.083 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.061+0000 7f0860ff9640 1 -- 172.21.15.42:0/3694570215 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f0854003ba0 con 0x7f086412d2d0 2024-09-16T03:44:33.083 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.061+0000 7f0860ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:44:33.083 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.061+0000 7f0860ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:44:33.083 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.061+0000 7f0860ff9640 1 -- 172.21.15.42:0/3694570215 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f086412d2d0 msgr2=0x7f086412d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:44:33.083 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.061+0000 7f0860ff9640 1 --2- 172.21.15.42:0/3694570215 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f086412d2d0 0x7f086412d6b0 secure :-1 s=READY pgs=2391 cs=0 l=1 rev1=1 crypto rx=0x7f0854009fc0 tx=0x7f085400f580 comp rx=0 tx=0).stop 2024-09-16T03:44:33.083 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.061+0000 7f0860ff9640 1 -- 172.21.15.42:0/3694570215 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f086412d2d0 msgr2=0x7f086412d6b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:44:33.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.061+0000 7f0860ff9640 1 --2- 172.21.15.42:0/3694570215 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f086412d2d0 0x7f086412d6b0 secure :-1 s=CLOSED pgs=2391 cs=0 l=1 rev1=1 crypto rx=0x7f0854009fc0 tx=0x7f085400f580 comp rx=0 tx=0).stop 2024-09-16T03:44:33.085 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.065+0000 7f086a9ce640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:44:33.085 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.065+0000 7f086a9ce640 1 --2- 172.21.15.42:0/3694570215 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f086412dbf0 0x7f086412f500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:44:33.085 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.065+0000 7f086a9ce640 10 monclient(hunting): start opening mon connection 2024-09-16T03:44:33.085 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.065+0000 7f086a9ce640 1 -- 172.21.15.42:0/3694570215 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f08641340c0 con 0x7f086412dbf0 2024-09-16T03:44:33.085 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.065+0000 7f0863fff640 1 --2- 172.21.15.42:0/3694570215 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f086412dbf0 0x7f086412f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:44:33.085 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.069+0000 7f0863fff640 10 monclient: get_auth_request con 0x7f086412dbf0 auth_method 0 2024-09-16T03:44:33.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.069+0000 7f0863fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:44:33.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.069+0000 7f0863fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:44:33.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.069+0000 7f0863fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:44:33.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.069+0000 7f0863fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:44:33.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.069+0000 7f0863fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:44:33.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.069+0000 7f0863fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:44:33.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.069+0000 7f0863fff640 10 monclient(hunting): handle_auth_done global_id 271153 payload 824 2024-09-16T03:44:33.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.069+0000 7f0863fff640 1 -- 172.21.15.42:0/3694570215 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f0864068e40 con 0x7f086412dbf0 2024-09-16T03:44:33.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.069+0000 7f0863fff640 1 --2- 172.21.15.42:0/3694570215 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f086412dbf0 0x7f086412f500 secure :-1 s=READY pgs=2392 cs=0 l=1 rev1=1 crypto rx=0x7f085401b000 tx=0x7f0854004a50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:44:33.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.069+0000 7f0860ff9640 10 client.11543.objecter ms_handle_connect 0x7f086412dbf0 2024-09-16T03:44:33.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.069+0000 7f0860ff9640 10 client.11543.objecter resend_mon_ops 2024-09-16T03:44:33.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.069+0000 7f0860ff9640 1 -- 172.21.15.42:0/3694570215 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f085401e720 con 0x7f086412dbf0 2024-09-16T03:44:33.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.069+0000 7f0860ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:44:33.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.069+0000 7f0860ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:44:33.087 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.069+0000 7f0860ff9640 10 monclient: dump: 2024-09-16T03:44:33.087 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:44:33.087 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:44:33.087 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:33.087 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:33.087 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:44:33.087 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:44:33.087 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:44:33.087 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:44:33.087 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:44:33.087 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:44:33.087 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.069+0000 7f0860ff9640 1 -- 172.21.15.42:0/3694570215 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4544 (secure 0 0 0) 0x7f0854015430 con 0x7f086412dbf0 2024-09-16T03:44:33.087 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.069+0000 7f0860ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:44:33.087 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.069+0000 7f0860ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:44:33.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.069+0000 7f0860ff9640 1 -- 172.21.15.42:0/3694570215 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f086412dbf0 msgr2=0x7f086412f500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:44:33.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.069+0000 7f0860ff9640 1 --2- 172.21.15.42:0/3694570215 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f086412dbf0 0x7f086412f500 secure :-1 s=READY pgs=2392 cs=0 l=1 rev1=1 crypto rx=0x7f085401b000 tx=0x7f0854004a50 comp rx=0 tx=0).stop 2024-09-16T03:44:33.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.069+0000 7f0860ff9640 1 -- 172.21.15.42:0/3694570215 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f086412dbf0 msgr2=0x7f086412f500 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:44:33.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.069+0000 7f0860ff9640 1 --2- 172.21.15.42:0/3694570215 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f086412dbf0 0x7f086412f500 secure :-1 s=CLOSED pgs=2392 cs=0 l=1 rev1=1 crypto rx=0x7f085401b000 tx=0x7f0854004a50 comp rx=0 tx=0).stop 2024-09-16T03:44:33.088 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:44:33.088 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:44:33.088 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:44:33.088 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T03:44:33.088 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 22, 2024-09-16T03:44:33.089 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 15798, 2024-09-16T03:44:33.089 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:44:33.089 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:44:33.089 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T03:44:33.089 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T03:44:33.089 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:44:33.089 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 0, 2024-09-16T03:44:33.089 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:44:33.089 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:44:33.089 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:44:33.089 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:44:33.089 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:44:33.089 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:44:33.090 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:44:33.090 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:44:33.090 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:44:33.090 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:44:33.090 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:44:33.090 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:44:33.090 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:44:33.090 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:44:33.090 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:44:33.090 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:44:33.090 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:44:33.090 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:44:33.090 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:44:33.091 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:44:33.091 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:44:33.091 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:44:33.091 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:44:33.091 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:44:33.091 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:44:33.091 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:44:33.091 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:44:33.091 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:44:33.091 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:44:33.091 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:44:33.091 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:44:33.091 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:44:33.091 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:44:33.092 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:44:33.092 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:44:33.092 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:44:33.092 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:44:33.092 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:44:33.092 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:44:33.092 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:44:33.092 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:44:33.092 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:44:33.092 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:44:33.092 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:44:33.092 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:44:33.092 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:44:33.092 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:44:33.093 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:44:33.093 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:44:33.093 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:44:33.093 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:44:33.093 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:44:33.093 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:44:33.093 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:44:33.093 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:44:33.093 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:44:33.093 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:44:33.093 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:44:33.093 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:44:33.093 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:44:33.093 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:44:33.094 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:44:33.094 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:44:33.094 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:44:33.094 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:44:33.094 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:44:33.094 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:44:33.094 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:44:33.094 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:44:33.094 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:44:33.094 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:44:33.094 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:44:33.094 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:44:33.094 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:44:33.094 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:44:33.095 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:44:33.095 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:44:33.095 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:44:33.095 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:44:33.095 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:44:33.095 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:44:33.095 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:44:33.095 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:44:33.095 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:44:33.095 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:44:33.095 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:44:33.095 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:44:33.095 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:44:33.095 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:44:33.096 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:44:33.096 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:44:33.096 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:44:33.096 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:44:33.096 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:44:33.096 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:44:33.096 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:44:33.096 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:44:33.096 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:44:33.096 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:44:33.096 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:44:33.096 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:44:33.096 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:44:33.096 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:44:33.097 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:44:33.097 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:44:33.097 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:44:33.097 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:44:33.097 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:44:33.097 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:44:33.097 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:44:33.097 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:44:33.097 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:44:33.097 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:44:33.097 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:44:33.097 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:44:33.097 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:44:33.098 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:44:33.098 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:44:33.098 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:44:33.098 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:44:33.098 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:44:33.098 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:44:33.098 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:44:33.098 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:44:33.098 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:44:33.098 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:44:33.098 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:44:33.098 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:44:33.098 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:44:33.098 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:44:33.099 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:44:33.099 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:44:33.099 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:44:33.099 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:44:33.099 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:44:33.099 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:44:33.099 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:44:33.099 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:44:33.099 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:44:33.099 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:44:33.099 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:44:33.099 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:44:33.099 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:44:33.099 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:44:33.100 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:44:33.100 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 4 2024-09-16T03:44:33.100 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:44:33.100 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:44:33.100 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:44:33.100 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:44:33.100 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:44:33.101 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.069+0000 7f086a9ce640 10 librados: watch_flush enter 2024-09-16T03:44:33.101 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.069+0000 7f086a9ce640 10 librados: watch_flush exit 2024-09-16T03:44:33.101 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.069+0000 7f086a9ce640 20 client.11543.objecter shutdown clearing up homeless session... 2024-09-16T03:44:33.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.069+0000 7f086a9ce640 10 client.11543.objecter successfully canceled tick 2024-09-16T03:44:33.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.069+0000 7f086a9ce640 1 -- 172.21.15.42:0/3694570215 >> v2:172.21.15.154:6812/254845342 conn(0x7f0850037810 msgr2=0x7f0850039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:44:33.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.069+0000 7f086a9ce640 1 --2- 172.21.15.42:0/3694570215 >> v2:172.21.15.154:6812/254845342 conn(0x7f0850037810 0x7f0850039cd0 secure :-1 s=READY pgs=2988 cs=0 l=1 rev1=1 crypto rx=0x7f0858000f70 tx=0x7f0858007a90 comp rx=0 tx=0).stop 2024-09-16T03:44:33.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.069+0000 7f086a9ce640 10 monclient: shutdown 2024-09-16T03:44:33.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.069+0000 7f086a9ce640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:44:33.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.069+0000 7f086a9ce640 1 -- 172.21.15.42:0/3694570215 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f08640a3c00 msgr2=0x7f08641cf940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:44:33.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.069+0000 7f086a9ce640 1 --2- 172.21.15.42:0/3694570215 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f08640a3c00 0x7f08641cf940 secure :-1 s=READY pgs=1944 cs=0 l=1 rev1=1 crypto rx=0x7f085c019f90 tx=0x7f085c019100 comp rx=0 tx=0).stop 2024-09-16T03:44:33.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.069+0000 7f0868f6b640 1 -- 172.21.15.42:0/3694570215 reap_dead start 2024-09-16T03:44:33.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.069+0000 7f0868f6b640 1 -- 172.21.15.42:0/3694570215 reap_dead start 2024-09-16T03:44:33.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.069+0000 7f086a9ce640 1 -- 172.21.15.42:0/3694570215 shutdown_connections 2024-09-16T03:44:33.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.069+0000 7f086a9ce640 1 -- 172.21.15.42:0/3694570215 >> 172.21.15.42:0/3694570215 conn(0x7f08640a9fb0 msgr2=0x7f08640a7a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:44:33.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.069+0000 7f086a9ce640 1 -- 172.21.15.42:0/3694570215 shutdown_connections 2024-09-16T03:44:33.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.069+0000 7f086a9ce640 1 -- 172.21.15.42:0/3694570215 wait complete. 2024-09-16T03:44:33.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.069+0000 7f086a9ce640 1 librados: shutdown 2024-09-16T03:44:33.103 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:44:33.103 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T03:44:33.103 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T03:44:33.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.217+0000 7fa401f2e640 10 monclient: get_monmap_and_config 2024-09-16T03:44:33.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.217+0000 7fa401f2e640 10 monclient: build_initial_monmap 2024-09-16T03:44:33.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.217+0000 7fa401f2e640 10 monclient: monmap: 2024-09-16T03:44:33.239 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:44:33.239 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:44:33.239 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:44:33.222980+0000 2024-09-16T03:44:33.239 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:44:33.222980+0000 2024-09-16T03:44:33.239 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:44:33.240 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:44:33.240 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:44:33.240 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:44:33.240 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:44:33.240 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:44:33.241 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:44:33.241 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:44:33.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.217+0000 7fa401f2e640 1 Processor -- start 2024-09-16T03:44:33.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.217+0000 7fa401f2e640 1 -- start start 2024-09-16T03:44:33.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.217+0000 7fa401f2e640 10 monclient: init 2024-09-16T03:44:33.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.217+0000 7fa401f2e640 10 monclient: _reopen_session rank -1 2024-09-16T03:44:33.242 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.217+0000 7fa401f2e640 10 monclient: _add_conns ranks=[3,0,1,4,2] 2024-09-16T03:44:33.242 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.217+0000 7fa401f2e640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fa3fc0a39b0 0x7fa3fc133b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:44:33.242 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.217+0000 7fa401f2e640 10 monclient(hunting): picked mon.noname-a con 0x7fa3fc0a39b0 addr v2:172.21.15.42:3300/0 2024-09-16T03:44:33.242 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.217+0000 7fa401f2e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa3fc0a3d90 addr v1:172.21.15.42:6789/0 2024-09-16T03:44:33.242 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.217+0000 7fa401f2e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa3fc137710 addr v1:172.21.15.154:6789/0 2024-09-16T03:44:33.243 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.217+0000 7fa401f2e640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa3fc09c140 con 0x7fa3fc0a3d90 2024-09-16T03:44:33.243 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.217+0000 7fa401f2e640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa3fc09c420 con 0x7fa3fc137710 2024-09-16T03:44:33.243 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.221+0000 7fa401f2e640 10 monclient(hunting): start opening mon connection 2024-09-16T03:44:33.243 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.221+0000 7fa401f2e640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7fa3fc06dd60 con 0x7fa3fc0a39b0 2024-09-16T03:44:33.243 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.221+0000 7fa401f2e640 10 monclient(hunting): _renew_subs 2024-09-16T03:44:33.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.221+0000 7fa401f2e640 10 monclient(hunting): authenticate will time out at 1120.192016s 2024-09-16T03:44:33.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.221+0000 7fa3fb7fe640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fa3fc0a39b0 0x7fa3fc133b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:44:33.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.221+0000 7fa3faffd640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fa3fc0a3d90 0x7fa3fc137000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:49566/0 (socket says 172.21.15.42:49566) 2024-09-16T03:44:33.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.221+0000 7fa3faffd640 1 -- 172.21.15.42:0/2905867124 learned_addr learned my addr 172.21.15.42:0/2905867124 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:44:33.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.221+0000 7fa3fb7fe640 10 monclient(hunting): get_auth_request con 0x7fa3fc0a39b0 auth_method 0 2024-09-16T03:44:33.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.221+0000 7fa3fb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:44:33.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.221+0000 7fa3fb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:44:33.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.221+0000 7fa3fb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:44:33.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.221+0000 7fa3fb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:44:33.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.221+0000 7fa3fb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:44:33.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.221+0000 7fa3fb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:44:33.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.221+0000 7fa3fb7fe640 10 monclient(hunting): handle_auth_done global_id 25638 payload 274 2024-09-16T03:44:33.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.221+0000 7fa3fb7fe640 1 -- 172.21.15.42:0/2905867124 >> v1:172.21.15.154:6789/0 conn(0x7fa3fc137710 legacy=0x7fa3fc13f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:44:33.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.221+0000 7fa3fb7fe640 1 -- 172.21.15.42:0/2905867124 >> v1:172.21.15.42:6789/0 conn(0x7fa3fc0a3d90 legacy=0x7fa3fc137000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:44:33.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.221+0000 7fa3fb7fe640 10 monclient: _finish_hunting 0 2024-09-16T03:44:33.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.221+0000 7fa3fb7fe640 1 monclient: found mon.noname-a 2024-09-16T03:44:33.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.221+0000 7fa3fb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:44:33.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.221+0000 7fa3fa7fc640 1 -- 172.21.15.42:0/2905867124 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 420000036 0 0) 0x7fa3fc09c140 con 0x7fa3fc0a3d90 2024-09-16T03:44:33.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.221+0000 7fa3fb7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:44:33.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.221+0000 7fa3fb7fe640 1 -- 172.21.15.42:0/2905867124 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3fc0a5e20 con 0x7fa3fc0a39b0 2024-09-16T03:44:33.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.221+0000 7fa3fa7fc640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T03:44:33.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.221+0000 7fa3fb7fe640 1 --2- 172.21.15.42:0/2905867124 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa3fc0a39b0 0x7fa3fc133b80 secure :-1 s=READY pgs=6936 cs=0 l=1 rev1=1 crypto rx=0x7fa3ec0097c0 tx=0x7fa3ec01c320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20581310eb3798be server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:44:33.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.221+0000 7fa3fa7fc640 1 -- 172.21.15.42:0/2905867124 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3ec0b0070 con 0x7fa3fc0a39b0 2024-09-16T03:44:33.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.221+0000 7fa3fa7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:44:33.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.221+0000 7fa3fa7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:44:33.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.221+0000 7fa3fa7fc640 10 monclient: dump: 2024-09-16T03:44:33.249 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:44:33.249 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:44:33.249 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:33.249 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:33.249 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:44:33.250 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:44:33.250 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:44:33.250 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:44:33.250 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:44:33.250 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:44:33.251 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.221+0000 7fa3fa7fc640 10 monclient: _finish_auth 0 2024-09-16T03:44:33.251 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.221+0000 7fa3fa7fc640 10 monclient: _check_auth_tickets 2024-09-16T03:44:33.251 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.221+0000 7fa3fa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:44:33.251 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.221+0000 7fa3fa7fc640 1 -- 172.21.15.42:0/2905867124 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa3ec0ab600 con 0x7fa3fc0a39b0 2024-09-16T03:44:33.251 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.221+0000 7fa401f2e640 5 monclient: authenticate success, global_id 25638 2024-09-16T03:44:33.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.221+0000 7fa401f2e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:44:33.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.221+0000 7fa3fa7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:44:33.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.221+0000 7fa401f2e640 10 monclient: get_monmap_and_config success 2024-09-16T03:44:33.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.221+0000 7fa3fa7fc640 1 -- 172.21.15.42:0/2905867124 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3ec0ab9d0 con 0x7fa3fc0a39b0 2024-09-16T03:44:33.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.221+0000 7fa3fa7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:44:33.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.221+0000 7fa401f2e640 4 set_mon_vals no callback set 2024-09-16T03:44:33.253 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.221+0000 7fa3fa7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:44:33.253 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.221+0000 7fa3fa7fc640 10 monclient: dump: 2024-09-16T03:44:33.253 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:44:33.253 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:44:33.253 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:33.254 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:33.254 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:44:33.254 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:44:33.254 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:44:33.254 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:44:33.254 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:44:33.255 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:44:33.255 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.221+0000 7fa401f2e640 10 monclient: shutdown 2024-09-16T03:44:33.255 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.221+0000 7fa401f2e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:44:33.255 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.221+0000 7fa401f2e640 1 -- 172.21.15.42:0/2905867124 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa3fc0a39b0 msgr2=0x7fa3fc133b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:44:33.255 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.221+0000 7fa401f2e640 1 --2- 172.21.15.42:0/2905867124 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa3fc0a39b0 0x7fa3fc133b80 secure :-1 s=READY pgs=6936 cs=0 l=1 rev1=1 crypto rx=0x7fa3ec0097c0 tx=0x7fa3ec01c320 comp rx=0 tx=0).stop 2024-09-16T03:44:33.256 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.221+0000 7fa401f2e640 1 -- 172.21.15.42:0/2905867124 shutdown_connections 2024-09-16T03:44:33.256 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.221+0000 7fa401f2e640 1 --2- 172.21.15.42:0/2905867124 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa3fc0a39b0 0x7fa3fc133b80 unknown :-1 s=CLOSED pgs=6936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:44:33.256 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.221+0000 7fa401f2e640 1 -- 172.21.15.42:0/2905867124 >> 172.21.15.42:0/2905867124 conn(0x7fa3fc0a9fb0 msgr2=0x7fa3fc0aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:44:33.256 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.221+0000 7fa401f2e640 1 -- 172.21.15.42:0/2905867124 shutdown_connections 2024-09-16T03:44:33.257 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.221+0000 7fa401f2e640 1 -- 172.21.15.42:0/2905867124 wait complete. 2024-09-16T03:44:33.257 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.221+0000 7fa401f2e640 10 monclient: build_initial_monmap 2024-09-16T03:44:33.257 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.221+0000 7fa401f2e640 10 monclient: monmap: 2024-09-16T03:44:33.257 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:44:33.257 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:44:33.258 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:44:33.258 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:44:33.258 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:44:33.258 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:44:33.258 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:44:33.258 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:44:33.259 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:44:33.259 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:44:33.259 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.225+0000 7fa401f2e640 1 librados: starting msgr at 2024-09-16T03:44:33.259 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.225+0000 7fa401f2e640 1 librados: starting objecter 2024-09-16T03:44:33.259 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.225+0000 7fa401f2e640 1 Processor -- start 2024-09-16T03:44:33.260 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.225+0000 7fa401f2e640 1 -- start start 2024-09-16T03:44:33.260 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.225+0000 7fa401f2e640 1 librados: setting wanted keys 2024-09-16T03:44:33.260 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.225+0000 7fa401f2e640 1 librados: calling monclient init 2024-09-16T03:44:33.260 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.225+0000 7fa401f2e640 10 monclient: init 2024-09-16T03:44:33.260 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.225+0000 7fa401f2e640 10 monclient: _reopen_session rank -1 2024-09-16T03:44:33.261 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.225+0000 7fa401f2e640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:44:33.261 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.225+0000 7fa401f2e640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa3fc0a39b0 0x7fa3fc130560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:44:33.261 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.225+0000 7fa401f2e640 10 monclient(hunting): picked mon.noname-c con 0x7fa3fc0a39b0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:44:33.261 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.225+0000 7fa401f2e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa3fc0a3d90 0x7fa3fc130aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:44:33.261 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.225+0000 7fa401f2e640 10 monclient(hunting): picked mon.noname-b con 0x7fa3fc0a3d90 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:44:33.262 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.225+0000 7fa401f2e640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa3fc137710 0x7fa3fc0a16a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:44:33.262 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.225+0000 7fa401f2e640 10 monclient(hunting): picked mon.noname-a con 0x7fa3fc137710 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:44:33.262 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.225+0000 7fa401f2e640 10 monclient(hunting): start opening mon connection 2024-09-16T03:44:33.262 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.225+0000 7fa401f2e640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fa3fc143900 con 0x7fa3fc137710 2024-09-16T03:44:33.262 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.225+0000 7fa401f2e640 10 monclient(hunting): start opening mon connection 2024-09-16T03:44:33.263 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.225+0000 7fa401f2e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa3fc06dee0 con 0x7fa3fc0a3d90 2024-09-16T03:44:33.263 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.225+0000 7fa401f2e640 10 monclient(hunting): start opening mon connection 2024-09-16T03:44:33.263 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.225+0000 7fa401f2e640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fa3fc143a80 con 0x7fa3fc0a39b0 2024-09-16T03:44:33.263 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.225+0000 7fa401f2e640 10 monclient(hunting): _renew_subs 2024-09-16T03:44:33.263 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.225+0000 7fa401f2e640 10 monclient(hunting): authenticate will time out at 1120.196877s 2024-09-16T03:44:33.263 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.225+0000 7fa3fb7fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa3fc0a39b0 0x7fa3fc130560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:44:33.264 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.225+0000 7fa3fbfff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa3fc137710 0x7fa3fc0a16a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:44:33.264 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.225+0000 7fa3fb7fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa3fc0a39b0 0x7fa3fc130560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:41248/0 (socket says 172.21.15.42:41248) 2024-09-16T03:44:33.264 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.225+0000 7fa3fb7fe640 1 -- 172.21.15.42:0/2037568604 learned_addr learned my addr 172.21.15.42:0/2037568604 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:44:33.264 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.225+0000 7fa3fb7fe640 10 monclient(hunting): get_auth_request con 0x7fa3fc0a39b0 auth_method 0 2024-09-16T03:44:33.264 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.225+0000 7fa3fb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:44:33.265 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.225+0000 7fa3fb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:44:33.265 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.225+0000 7fa3fb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:44:33.265 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.225+0000 7fa3fbfff640 10 monclient(hunting): get_auth_request con 0x7fa3fc137710 auth_method 0 2024-09-16T03:44:33.265 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.225+0000 7fa3fbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:44:33.265 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.225+0000 7fa3fbfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:44:33.266 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.225+0000 7fa3fbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:44:33.266 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.225+0000 7fa3fb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:44:33.266 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.225+0000 7fa3fb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:44:33.266 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.225+0000 7fa3fb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:44:33.266 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.225+0000 7fa3fbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:44:33.267 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.225+0000 7fa3fbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:44:33.267 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.225+0000 7fa3fbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:44:33.267 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.225+0000 7fa3fb7fe640 10 monclient(hunting): handle_auth_done global_id 11663 payload 824 2024-09-16T03:44:33.267 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.225+0000 7fa3fb7fe640 1 -- 172.21.15.42:0/2037568604 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa3fc0a3d90 msgr2=0x7fa3fc130aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:44:33.267 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.225+0000 7fa3fb7fe640 1 --2- 172.21.15.42:0/2037568604 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa3fc0a3d90 0x7fa3fc130aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:44:33.267 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.225+0000 7fa3fb7fe640 1 -- 172.21.15.42:0/2037568604 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa3fc137710 msgr2=0x7fa3fc0a16a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:44:33.268 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.225+0000 7fa3fb7fe640 1 --2- 172.21.15.42:0/2037568604 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa3fc137710 0x7fa3fc0a16a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:44:33.268 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.225+0000 7fa3fb7fe640 10 monclient: _finish_hunting 0 2024-09-16T03:44:33.268 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.225+0000 7fa3fb7fe640 1 monclient: found mon.noname-c 2024-09-16T03:44:33.268 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.225+0000 7fa3fb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:44:33.268 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.225+0000 7fa3fb7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:44:33.269 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.225+0000 7fa3fb7fe640 1 -- 172.21.15.42:0/2037568604 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3fc06f680 con 0x7fa3fc0a39b0 2024-09-16T03:44:33.269 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.225+0000 7fa3fbfff640 1 --2- 172.21.15.42:0/2037568604 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa3fc137710 0x7fa3fc0a16a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:44:33.269 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.225+0000 7fa3fb7fe640 1 --2- 172.21.15.42:0/2037568604 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa3fc0a39b0 0x7fa3fc130560 secure :-1 s=READY pgs=1967 cs=0 l=1 rev1=1 crypto rx=0x7fa3ec0ae040 tx=0x7fa3ec01ab60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:44:33.269 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.225+0000 7fa3e3fff640 10 client.?.objecter ms_handle_connect 0x7fa3fc0a39b0 2024-09-16T03:44:33.269 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.225+0000 7fa3e3fff640 10 client.?.objecter resend_mon_ops 2024-09-16T03:44:33.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.225+0000 7fa3e3fff640 1 -- 172.21.15.42:0/2037568604 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3ec0b0050 con 0x7fa3fc0a39b0 2024-09-16T03:44:33.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.225+0000 7fa3e3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:44:33.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.225+0000 7fa3e3fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:44:33.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.225+0000 7fa3e3fff640 10 monclient: dump: 2024-09-16T03:44:33.270 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:44:33.271 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:44:33.271 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:33.271 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:33.271 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:44:33.271 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:44:33.272 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:44:33.272 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:44:33.272 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:44:33.272 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:44:33.273 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.225+0000 7fa3e3fff640 10 monclient: _finish_auth 0 2024-09-16T03:44:33.273 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.225+0000 7fa3e3fff640 10 monclient: _check_auth_tickets 2024-09-16T03:44:33.273 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.225+0000 7fa3e3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:44:33.273 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.225+0000 7fa401f2e640 5 monclient: authenticate success, global_id 11663 2024-09-16T03:44:33.273 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.225+0000 7fa3e3fff640 1 -- 172.21.15.42:0/2037568604 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa3ec0bc070 con 0x7fa3fc0a39b0 2024-09-16T03:44:33.274 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.225+0000 7fa401f2e640 10 monclient: _renew_subs 2024-09-16T03:44:33.274 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.225+0000 7fa401f2e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:44:33.274 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.225+0000 7fa401f2e640 1 -- 172.21.15.42:0/2037568604 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3fc137100 con 0x7fa3fc0a39b0 2024-09-16T03:44:33.274 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.225+0000 7fa3e3fff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:44:33.274 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.225+0000 7fa3e3fff640 1 -- 172.21.15.42:0/2037568604 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3ec0aac40 con 0x7fa3fc0a39b0 2024-09-16T03:44:33.275 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.225+0000 7fa3e3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:44:33.275 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.225+0000 7fa3e3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:44:33.275 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.225+0000 7fa3e3fff640 10 monclient: dump: 2024-09-16T03:44:33.275 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:44:33.275 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:44:33.276 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:33.276 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:33.276 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:44:33.276 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:44:33.276 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:44:33.277 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:44:33.277 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:44:33.277 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:44:33.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.225+0000 7fa3f9ffb640 4 set_mon_vals no callback set 2024-09-16T03:44:33.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.225+0000 7fa401f2e640 10 client.11663.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:44:33.278 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.225+0000 7fa401f2e640 10 monclient: _renew_subs 2024-09-16T03:44:33.278 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.225+0000 7fa401f2e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:44:33.278 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.225+0000 7fa401f2e640 1 -- 172.21.15.42:0/2037568604 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3fc139ad0 con 0x7fa3fc0a39b0 2024-09-16T03:44:33.278 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.225+0000 7fa401f2e640 1 librados: init done 2024-09-16T03:44:33.278 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.225+0000 7fa401f2e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:44:33.279 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.225+0000 7fa401f2e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:44:33.279 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.225+0000 7fa401f2e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:44:33.279 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.225+0000 7fa3e3fff640 1 -- 172.21.15.42:0/2037568604 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fa3ec0c1460 con 0x7fa3fc0a39b0 2024-09-16T03:44:33.279 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.225+0000 7fa401f2e640 1 -- 172.21.15.42:0/2037568604 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa3fc1381f0 con 0x7fa3fc0a39b0 2024-09-16T03:44:33.279 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.225+0000 7fa3e3fff640 1 --2- 172.21.15.42:0/2037568604 >> v2:172.21.15.154:6812/254845342 conn(0x7fa3e8037840 0x7fa3e8039d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:44:33.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.225+0000 7fa3e3fff640 1 -- 172.21.15.42:0/2037568604 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(54..54 src has 1..54) ==== 10680+0+0 (secure 0 0 0) 0x7fa3ec0e4f00 con 0x7fa3fc0a39b0 2024-09-16T03:44:33.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.225+0000 7fa3e3fff640 10 client.11663.objecter ms_dispatch 0x7fa3fc000ba0 osd_map(54..54 src has 1..54) 2024-09-16T03:44:33.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.225+0000 7fa3e3fff640 3 client.11663.objecter handle_osd_map got epochs [54,54] > 0 2024-09-16T03:44:33.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.225+0000 7fa3e3fff640 3 client.11663.objecter handle_osd_map decoding full epoch 54 2024-09-16T03:44:33.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.225+0000 7fa3e3fff640 20 client.11663.objecter dump_active .. 0 homeless 2024-09-16T03:44:33.281 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.229+0000 7fa3faffd640 1 --2- 172.21.15.42:0/2037568604 >> v2:172.21.15.154:6812/254845342 conn(0x7fa3e8037840 0x7fa3e8039d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:44:33.281 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.229+0000 7fa3faffd640 10 monclient: get_auth_request con 0x7fa3e8037840 auth_method 0 2024-09-16T03:44:33.281 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.229+0000 7fa3faffd640 1 --2- 172.21.15.42:0/2037568604 >> v2:172.21.15.154:6812/254845342 conn(0x7fa3e8037840 0x7fa3e8039d00 secure :-1 s=READY pgs=3019 cs=0 l=1 rev1=1 crypto rx=0x7fa3f00011f0 tx=0x7fa3f0007930 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:44:33.281 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.229+0000 7fa3e3fff640 10 client.11663.objecter ms_handle_connect 0x7fa3e8037840 2024-09-16T03:44:33.281 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.229+0000 7fa3e3fff640 1 -- 172.21.15.42:0/2037568604 <== mon.2 v2:172.21.15.42:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7fa3fc06f680 con 0x7fa3fc0a39b0 2024-09-16T03:44:33.281 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.229+0000 7fa3e3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:44:33.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.229+0000 7fa3e3fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:44:33.298 INFO:tasks.workunit.client.0.smithi154.stdout: write_operations: Running main() from gmock_main.cc 2024-09-16T03:44:33.298 INFO:tasks.workunit.client.0.smithi154.stdout: write_operations: [==========] Running 7 tests from 1 test suite. 2024-09-16T03:44:33.298 INFO:tasks.workunit.client.0.smithi154.stdout: write_operations: [----------] Global test environment set-up. 2024-09-16T03:44:33.299 INFO:tasks.workunit.client.0.smithi154.stdout: write_operations: [----------] 7 tests from NeoRadosWriteOps 2024-09-16T03:44:33.299 INFO:tasks.workunit.client.0.smithi154.stdout: write_operations: [ RUN ] NeoRadosWriteOps.AssertExists 2024-09-16T03:44:33.299 INFO:tasks.workunit.client.0.smithi154.stdout: write_operations: [ OK ] NeoRadosWriteOps.AssertExists (4615 ms) 2024-09-16T03:44:33.299 INFO:tasks.workunit.client.0.smithi154.stdout: write_operations: [ RUN ] NeoRadosWriteOps.AssertVersion 2024-09-16T03:44:33.299 INFO:tasks.workunit.client.0.smithi154.stdout: write_operations: [ OK ] NeoRadosWriteOps.AssertVersion (24010 ms) 2024-09-16T03:44:33.299 INFO:tasks.workunit.client.0.smithi154.stdout: write_operations: [ RUN ] NeoRadosWriteOps.Xattrs 2024-09-16T03:44:33.299 INFO:tasks.workunit.client.0.smithi154.stdout: write_operations: [ OK ] NeoRadosWriteOps.Xattrs (2774 ms) 2024-09-16T03:44:33.299 INFO:tasks.workunit.client.0.smithi154.stdout: write_operations: [ RUN ] NeoRadosWriteOps.Write 2024-09-16T03:44:33.299 INFO:tasks.workunit.client.0.smithi154.stdout: write_operations: [ OK ] NeoRadosWriteOps.Write (2967 ms) 2024-09-16T03:44:33.299 INFO:tasks.workunit.client.0.smithi154.stdout: write_operations: [ RUN ] NeoRadosWriteOps.Exec 2024-09-16T03:44:33.299 INFO:tasks.workunit.client.0.smithi154.stdout: write_operations: [ OK ] NeoRadosWriteOps.Exec (2992 ms) 2024-09-16T03:44:33.299 INFO:tasks.workunit.client.0.smithi154.stdout: write_operations: [ RUN ] NeoRadosWriteOps.WriteSame 2024-09-16T03:44:33.300 INFO:tasks.workunit.client.0.smithi154.stdout: write_operations: [ OK ] NeoRadosWriteOps.WriteSame (3041 ms) 2024-09-16T03:44:33.300 INFO:tasks.workunit.client.0.smithi154.stdout: write_operations: [ RUN ] NeoRadosWriteOps.CmpExt 2024-09-16T03:44:33.300 INFO:tasks.workunit.client.0.smithi154.stdout: write_operations: [ OK ] NeoRadosWriteOps.CmpExt (23068 ms) 2024-09-16T03:44:33.300 INFO:tasks.workunit.client.0.smithi154.stdout: write_operations: [----------] 7 tests from NeoRadosWriteOps (63467 ms total) 2024-09-16T03:44:33.300 INFO:tasks.workunit.client.0.smithi154.stdout: write_operations: 2024-09-16T03:44:33.300 INFO:tasks.workunit.client.0.smithi154.stdout: write_operations: [----------] Global test environment tear-down 2024-09-16T03:44:33.300 INFO:tasks.workunit.client.0.smithi154.stdout: write_operations: [==========] 7 tests from 1 test suite ran. (63467 ms total) 2024-09-16T03:44:33.300 INFO:tasks.workunit.client.0.smithi154.stdout: write_operations: [ PASSED ] 7 tests. 2024-09-16T03:44:33.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.513+0000 7fa401f2e640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T03:44:33.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.513+0000 7fa401f2e640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T03:44:33.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.513+0000 7fa401f2e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:44:33.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.513+0000 7fa401f2e640 1 -- 172.21.15.42:0/2037568604 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7fa3fc133b80 con 0x7fa3fc0a39b0 2024-09-16T03:44:33.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.517+0000 7fa3e3fff640 1 -- 172.21.15.42:0/2037568604 <== mon.2 v2:172.21.15.42:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7fa3ec016030 con 0x7fa3fc0a39b0 2024-09-16T03:44:33.521 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.517+0000 7fa3e3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T03:44:33.521 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.517+0000 7fa3e3fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:44:33.522 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:44:33.517+0000 7f89824fe640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:44:33.522 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:44:33.517+0000 7f89824fe640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=25,config=2,health=12,kv=10,logm=51} crc {auth=1389949703,config=692619927,health=2723269353,kv=3733109450,logm=3504213558}) 2024-09-16T03:44:33.522 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:44:33.517+0000 7f89824fe640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=24,config=2,health=11,kv=10,logm=53} crc {auth=957206052,config=692619927,health=2515522790,kv=3446188498,logm=2538049792}) 2024-09-16T03:44:33.522 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:44:33.517+0000 7f89824fe640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:44:33.522 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:44:33.517+0000 7f89824fe640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=25,config=2,health=12,kv=10,logm=51} crc {auth=1389949703,config=692619927,health=2723269353,kv=3733109450,logm=3504213558}) 2024-09-16T03:44:33.522 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:44:33.517+0000 7f89824fe640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=24,config=2,health=12,kv=10,logm=52} crc {auth=3355007410,config=692619927,health=2723269353,kv=1270464448,logm=2627731146}) 2024-09-16T03:44:33.523 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:44:33.517+0000 7f89824fe640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:44:33.523 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:44:33.517+0000 7f89824fe640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1261301433}) 2024-09-16T03:44:33.523 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:44:33.517+0000 7f89824fe640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=987112443}) 2024-09-16T03:44:33.523 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:44:33.517+0000 7f89824fe640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:44:33.523 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:44:33.517+0000 7f89824fe640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1261301433}) 2024-09-16T03:44:33.523 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:44:33.517+0000 7f89824fe640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2268240231}) 2024-09-16T03:44:33.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.517+0000 7fa401f2e640 10 librados: watch_flush enter 2024-09-16T03:44:33.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.517+0000 7fa401f2e640 10 librados: watch_flush exit 2024-09-16T03:44:33.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.517+0000 7fa401f2e640 20 client.11663.objecter shutdown clearing up homeless session... 2024-09-16T03:44:33.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.517+0000 7fa401f2e640 10 client.11663.objecter successfully canceled tick 2024-09-16T03:44:33.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.517+0000 7fa401f2e640 1 -- 172.21.15.42:0/2037568604 >> v2:172.21.15.154:6812/254845342 conn(0x7fa3e8037840 msgr2=0x7fa3e8039d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:44:33.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.517+0000 7fa401f2e640 1 --2- 172.21.15.42:0/2037568604 >> v2:172.21.15.154:6812/254845342 conn(0x7fa3e8037840 0x7fa3e8039d00 secure :-1 s=READY pgs=3019 cs=0 l=1 rev1=1 crypto rx=0x7fa3f00011f0 tx=0x7fa3f0007930 comp rx=0 tx=0).stop 2024-09-16T03:44:33.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.517+0000 7fa401f2e640 10 monclient: shutdown 2024-09-16T03:44:33.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.517+0000 7fa401f2e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:44:33.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.517+0000 7fa401f2e640 1 -- 172.21.15.42:0/2037568604 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa3fc0a39b0 msgr2=0x7fa3fc130560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:44:33.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.517+0000 7fa401f2e640 1 --2- 172.21.15.42:0/2037568604 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa3fc0a39b0 0x7fa3fc130560 secure :-1 s=READY pgs=1967 cs=0 l=1 rev1=1 crypto rx=0x7fa3ec0ae040 tx=0x7fa3ec01ab60 comp rx=0 tx=0).stop 2024-09-16T03:44:33.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.517+0000 7fa401f2e640 1 -- 172.21.15.42:0/2037568604 shutdown_connections 2024-09-16T03:44:33.527 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.517+0000 7fa401f2e640 1 --2- 172.21.15.42:0/2037568604 >> v2:172.21.15.154:6812/254845342 conn(0x7fa3e8037840 0x7fa3e8039d00 unknown :-1 s=CLOSED pgs=3019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:44:33.527 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.517+0000 7fa401f2e640 1 --2- 172.21.15.42:0/2037568604 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa3fc137710 0x7fa3fc0a16a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:44:33.527 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.517+0000 7fa401f2e640 1 --2- 172.21.15.42:0/2037568604 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa3fc0a3d90 0x7fa3fc130aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:44:33.527 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.517+0000 7fa401f2e640 1 --2- 172.21.15.42:0/2037568604 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa3fc0a39b0 0x7fa3fc130560 unknown :-1 s=CLOSED pgs=1967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:44:33.527 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.517+0000 7fa401f2e640 1 -- 172.21.15.42:0/2037568604 >> 172.21.15.42:0/2037568604 conn(0x7fa3fc0a9fb0 msgr2=0x7fa3fc0a8fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:44:33.527 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.517+0000 7fa401f2e640 1 -- 172.21.15.42:0/2037568604 shutdown_connections 2024-09-16T03:44:33.527 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.517+0000 7fa401f2e640 1 -- 172.21.15.42:0/2037568604 wait complete. 2024-09-16T03:44:33.527 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:33.517+0000 7fa401f2e640 1 librados: shutdown 2024-09-16T03:44:33.531 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:44:33.517+0000 7f89824fe640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:44:33.531 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:44:33.517+0000 7f89824fe640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=898228171}) 2024-09-16T03:44:33.531 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:44:33.517+0000 7f89824fe640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1952359222}) 2024-09-16T03:44:33.531 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:44:33.517+0000 7f89824fe640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:44:33.531 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:44:33.517+0000 7f89824fe640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=898228171}) 2024-09-16T03:44:33.531 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:44:33.517+0000 7f89824fe640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3029048173}) 2024-09-16T03:44:33.531 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:44:33.521+0000 7f89824fe640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:44:33.531 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:44:33.521+0000 7f89824fe640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2859634154}) 2024-09-16T03:44:33.531 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:44:33.521+0000 7f89824fe640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=158538413}) 2024-09-16T03:44:33.531 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:44:33.521+0000 7f89824fe640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:44:33.531 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:44:33.521+0000 7f89824fe640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2859634154}) 2024-09-16T03:44:33.531 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:44:33.521+0000 7f89824fe640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=124037212}) 2024-09-16T03:44:33.532 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:44:33.521+0000 7f89824fe640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:44:33.532 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:44:33.521+0000 7f89824fe640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2415937070}) 2024-09-16T03:44:33.532 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:44:33.521+0000 7f89824fe640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1545335485}) 2024-09-16T03:44:33.532 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:44:33.521+0000 7f89824fe640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:44:33.532 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:44:33.521+0000 7f89824fe640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2415937070}) 2024-09-16T03:44:33.532 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:44:33.521+0000 7f89824fe640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4044358087}) 2024-09-16T03:44:33.532 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:44:33.521+0000 7f89824fe640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:44:33.532 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:44:33.521+0000 7f89824fe640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2823017202}) 2024-09-16T03:44:33.532 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:44:33.521+0000 7f89824fe640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1248309524}) 2024-09-16T03:44:33.532 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:44:33.521+0000 7f89824fe640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:44:33.532 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:44:33.521+0000 7f89824fe640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2823017202}) 2024-09-16T03:44:33.532 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:44:33.521+0000 7f89824fe640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1021838112}) 2024-09-16T03:44:33.532 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:44:33.525+0000 7f89824fe640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:44:33.532 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:44:33.525+0000 7f89824fe640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2239219104}) 2024-09-16T03:44:33.533 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:44:33.525+0000 7f89824fe640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4166723975}) 2024-09-16T03:44:33.533 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:44:33.525+0000 7f89824fe640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:44:33.533 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:44:33.525+0000 7f89824fe640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2239219104}) 2024-09-16T03:44:33.533 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:44:33.525+0000 7f89824fe640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=679242426}) 2024-09-16T03:44:33.534 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:44:33.525+0000 7f89824fe640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:44:33.534 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:44:33.525+0000 7f89824fe640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1698535792}) 2024-09-16T03:44:33.535 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:44:33.525+0000 7f89824fe640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1736230170}) 2024-09-16T03:44:33.535 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:44:33.525+0000 7f89824fe640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:44:33.535 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:44:33.525+0000 7f89824fe640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1698535792}) 2024-09-16T03:44:33.535 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:44:33.525+0000 7f89824fe640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3968180719}) 2024-09-16T03:44:33.535 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:44:33.529+0000 7f89824fe640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:44:33.535 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:44:33.529+0000 7f89824fe640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2663445102}) 2024-09-16T03:44:33.535 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:44:33.529+0000 7f89824fe640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=383084513}) 2024-09-16T03:44:33.535 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:44:33.529+0000 7f89824fe640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:44:33.535 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:44:33.529+0000 7f89824fe640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2663445102}) 2024-09-16T03:44:33.535 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:44:33.529+0000 7f89824fe640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3727610644}) 2024-09-16T03:44:33.535 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:44:33.529+0000 7f89824fe640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:44:33.535 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:44:33.529+0000 7f89824fe640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=931340964}) 2024-09-16T03:44:33.535 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:44:33.529+0000 7f89824fe640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=461329552}) 2024-09-16T03:44:33.536 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:44:33.529+0000 7f89824fe640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:44:33.536 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:44:33.529+0000 7f89824fe640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=931340964}) 2024-09-16T03:44:33.536 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:44:33.529+0000 7f89824fe640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1082995936}) 2024-09-16T03:44:33.537 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:44:33.529+0000 7f89824fe640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:44:33.537 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:44:33.529+0000 7f89824fe640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=878118825}) 2024-09-16T03:44:33.537 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:44:33.529+0000 7f89824fe640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=275475614}) 2024-09-16T03:44:33.537 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:44:33.529+0000 7f89824fe640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:44:33.537 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:44:33.529+0000 7f89824fe640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=878118825}) 2024-09-16T03:44:33.537 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:44:33.529+0000 7f89824fe640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2613208360}) 2024-09-16T03:44:33.543 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T03:44:33.556 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:44:33.549+0000 7f89824fe640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:44:33.556 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:44:33.549+0000 7f89824fe640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=15,mdsmap=2,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=11,osdmap=32} crc {logm=3987886056,mdsmap=210808725,mgr=1483503728,mgr_command_descs=2107673600,mgr_metadata=1367272202,mgrstat=3658095249,mon_config_key=85057112,monmap=3937513981,osd_metadata=4182959540,osd_pg_creating=1466602723,osd_snap=4069505003,osdmap=2389119971}) 2024-09-16T03:44:33.556 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:44:33.549+0000 7f89824fe640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=17,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=11,monmap=2,osd_metadata=6,osd_pg_creating=1,osd_snap=11,osdmap=29} crc {logm=377712463,mdsmap=4088524775,mgr=460109776,mgr_command_descs=2107673600,mgr_metadata=1367272202,mgrstat=3658095249,mon_config_key=85057112,monmap=268092033,osd_metadata=4182959540,osd_pg_creating=1466602723,osd_snap=4069505003,osdmap=1843484545}) 2024-09-16T03:44:33.556 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:44:33.549+0000 7f89824fe640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:44:33.556 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:44:33.549+0000 7f89824fe640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=15,mdsmap=2,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=11,osdmap=32} crc {logm=3987886056,mdsmap=210808725,mgr=1483503728,mgr_command_descs=2107673600,mgr_metadata=1367272202,mgrstat=3658095249,mon_config_key=85057112,monmap=3937513981,osd_metadata=4182959540,osd_pg_creating=1466602723,osd_snap=4069505003,osdmap=2389119971}) 2024-09-16T03:44:33.557 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:44:33.549+0000 7f89824fe640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=17,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=10,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=11,osdmap=31} crc {logm=2970154416,mdsmap=4088524775,mgr=390803354,mgr_command_descs=2107673600,mgr_metadata=1367272202,mgrstat=315475654,mon_config_key=541432806,monmap=3937513981,osd_metadata=4182959540,osd_pg_creating=1466602723,osd_snap=2206166155,osdmap=3336730715}) 2024-09-16T03:44:33.561 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:44:33.557+0000 7f89824fe640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:44:33.561 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:44:33.557+0000 7f89824fe640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=64} crc {osdmap=4041221486}) 2024-09-16T03:44:33.562 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:44:33.557+0000 7f89824fe640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=62} crc {osdmap=4210638544}) 2024-09-16T03:44:33.562 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:44:33.557+0000 7f89824fe640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:44:33.562 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:44:33.557+0000 7f89824fe640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=64} crc {osdmap=4041221486}) 2024-09-16T03:44:33.562 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:44:33.557+0000 7f89824fe640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=64} crc {osdmap=1656204719}) 2024-09-16T03:44:34.543 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T03:44:34.544 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:44:34.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.669+0000 7ff8967ad640 10 monclient: get_monmap_and_config 2024-09-16T03:44:34.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.669+0000 7ff8967ad640 10 monclient: build_initial_monmap 2024-09-16T03:44:34.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.669+0000 7ff8967ad640 10 monclient: monmap: 2024-09-16T03:44:34.674 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:44:34.674 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:44:34.674 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:44:34.673179+0000 2024-09-16T03:44:34.674 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:44:34.673179+0000 2024-09-16T03:44:34.674 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:44:34.674 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:44:34.674 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:44:34.675 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:44:34.675 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:44:34.675 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:44:34.675 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:44:34.675 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:44:34.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.669+0000 7ff8967ad640 1 Processor -- start 2024-09-16T03:44:34.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.669+0000 7ff8967ad640 1 -- start start 2024-09-16T03:44:34.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.669+0000 7ff8967ad640 10 monclient: init 2024-09-16T03:44:34.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.669+0000 7ff8967ad640 10 monclient: _reopen_session rank -1 2024-09-16T03:44:34.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.669+0000 7ff8967ad640 10 monclient: _add_conns ranks=[0,2,1,3,4] 2024-09-16T03:44:34.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.669+0000 7ff8967ad640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff8900a0de0 addr v1:172.21.15.42:6789/0 2024-09-16T03:44:34.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.669+0000 7ff8967ad640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff8900a25b0 0x7ff89013a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:44:34.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.669+0000 7ff8967ad640 10 monclient(hunting): picked mon.noname-b con 0x7ff8900a25b0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:44:34.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.669+0000 7ff8967ad640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff89013a5f0 addr v1:172.21.15.154:6789/0 2024-09-16T03:44:34.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.669+0000 7ff8967ad640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff8900a1b80 con 0x7ff8900a0de0 2024-09-16T03:44:34.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.669+0000 7ff8967ad640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff8900a18d0 con 0x7ff89013a5f0 2024-09-16T03:44:34.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.669+0000 7ff8967ad640 10 monclient(hunting): start opening mon connection 2024-09-16T03:44:34.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.669+0000 7ff8967ad640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7ff8900756e0 con 0x7ff8900a25b0 2024-09-16T03:44:34.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.669+0000 7ff8967ad640 10 monclient(hunting): _renew_subs 2024-09-16T03:44:34.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.669+0000 7ff8967ad640 10 monclient(hunting): authenticate will time out at 1121.643077s 2024-09-16T03:44:34.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.669+0000 7ff88ffff640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7ff8900a0de0 0x7ff8900a11c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:47894/0 (socket says 172.21.15.42:47894) 2024-09-16T03:44:34.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.669+0000 7ff88ffff640 1 -- 172.21.15.42:0/1999710388 learned_addr learned my addr 172.21.15.42:0/1999710388 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:44:34.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.669+0000 7ff88f7fe640 1 --2- 172.21.15.42:0/1999710388 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff8900a25b0 0x7ff89013a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:44:34.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.669+0000 7ff88f7fe640 10 monclient(hunting): get_auth_request con 0x7ff8900a25b0 auth_method 0 2024-09-16T03:44:34.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.669+0000 7ff88f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:44:34.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.669+0000 7ff88f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:44:34.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.669+0000 7ff88f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:44:34.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.669+0000 7ff88f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:44:34.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.669+0000 7ff88f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:44:34.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.669+0000 7ff88f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:44:34.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.669+0000 7ff88effd640 1 -- 172.21.15.42:0/1999710388 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 827336901 0 0) 0x7ff8900a1b80 con 0x7ff8900a0de0 2024-09-16T03:44:34.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.669+0000 7ff88effd640 10 monclient(hunting): _negotiate 2024-09-16T03:44:34.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.669+0000 7ff88effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:44:34.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.669+0000 7ff88effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:44:34.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.669+0000 7ff88effd640 10 monclient(hunting): my global_id is 27000 2024-09-16T03:44:34.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.669+0000 7ff88effd640 1 -- 172.21.15.42:0/1999710388 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff878003530 con 0x7ff8900a0de0 2024-09-16T03:44:34.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.669+0000 7ff88f7fe640 10 monclient(hunting): handle_auth_done global_id 12686 payload 274 2024-09-16T03:44:34.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.669+0000 7ff88f7fe640 1 -- 172.21.15.42:0/1999710388 >> v1:172.21.15.154:6789/0 conn(0x7ff89013a5f0 legacy=0x7ff89013c9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:44:34.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.669+0000 7ff88f7fe640 1 -- 172.21.15.42:0/1999710388 >> v1:172.21.15.42:6789/0 conn(0x7ff8900a0de0 legacy=0x7ff8900a11c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:44:34.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.669+0000 7ff88effd640 1 -- 172.21.15.42:0/1999710388 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 291296986 0 0) 0x7ff878003530 con 0x7ff8900a0de0 2024-09-16T03:44:34.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.669+0000 7ff88f7fe640 10 monclient: _finish_hunting 0 2024-09-16T03:44:34.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.669+0000 7ff88f7fe640 1 monclient: found mon.noname-b 2024-09-16T03:44:34.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.669+0000 7ff88f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:44:34.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.669+0000 7ff88f7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T03:44:34.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.669+0000 7ff88f7fe640 1 -- 172.21.15.42:0/1999710388 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8900a0260 con 0x7ff8900a25b0 2024-09-16T03:44:34.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.669+0000 7ff88effd640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T03:44:34.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.673+0000 7ff88f7fe640 1 --2- 172.21.15.42:0/1999710388 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff8900a25b0 0x7ff89013a0b0 secure :-1 s=READY pgs=2194 cs=0 l=1 rev1=1 crypto rx=0x7ff8840097f0 tx=0x7ff88401c310 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fbb09f7622ab9010 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:44:34.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.673+0000 7ff88effd640 1 -- 172.21.15.42:0/1999710388 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8840b0070 con 0x7ff8900a25b0 2024-09-16T03:44:34.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.673+0000 7ff88effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:44:34.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.673+0000 7ff88effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:44:34.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.673+0000 7ff88effd640 10 monclient: dump: 2024-09-16T03:44:34.688 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:44:34.688 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:44:34.688 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:34.688 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:34.688 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:44:34.688 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:44:34.688 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:44:34.688 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:44:34.689 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:44:34.689 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:44:34.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.673+0000 7ff88effd640 10 monclient: _finish_auth 0 2024-09-16T03:44:34.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.673+0000 7ff88effd640 10 monclient: _check_auth_tickets 2024-09-16T03:44:34.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.673+0000 7ff88effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:44:34.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.673+0000 7ff88effd640 1 -- 172.21.15.42:0/1999710388 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff8840ab400 con 0x7ff8900a25b0 2024-09-16T03:44:34.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.673+0000 7ff8967ad640 5 monclient: authenticate success, global_id 12686 2024-09-16T03:44:34.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.673+0000 7ff8967ad640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:44:34.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.673+0000 7ff88effd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:44:34.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.673+0000 7ff8967ad640 10 monclient: get_monmap_and_config success 2024-09-16T03:44:34.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.673+0000 7ff8967ad640 4 set_mon_vals no callback set 2024-09-16T03:44:34.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.673+0000 7ff88effd640 1 -- 172.21.15.42:0/1999710388 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff880001680 con 0x7ff8900a25b0 2024-09-16T03:44:34.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.673+0000 7ff88effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:44:34.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.673+0000 7ff88effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:44:34.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.673+0000 7ff88effd640 10 monclient: dump: 2024-09-16T03:44:34.690 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:44:34.690 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:44:34.690 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:34.690 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:34.690 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:44:34.690 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:44:34.690 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:44:34.690 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:44:34.690 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:44:34.691 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:44:34.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.673+0000 7ff8967ad640 10 monclient: shutdown 2024-09-16T03:44:34.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.673+0000 7ff8967ad640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:44:34.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.673+0000 7ff8967ad640 1 -- 172.21.15.42:0/1999710388 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff8900a25b0 msgr2=0x7ff89013a0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:44:34.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.673+0000 7ff8967ad640 1 --2- 172.21.15.42:0/1999710388 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff8900a25b0 0x7ff89013a0b0 secure :-1 s=READY pgs=2194 cs=0 l=1 rev1=1 crypto rx=0x7ff8840097f0 tx=0x7ff88401c310 comp rx=0 tx=0).stop 2024-09-16T03:44:34.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.673+0000 7ff8967ad640 1 -- 172.21.15.42:0/1999710388 shutdown_connections 2024-09-16T03:44:34.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.673+0000 7ff8967ad640 1 --2- 172.21.15.42:0/1999710388 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff8900a25b0 0x7ff89013a0b0 unknown :-1 s=CLOSED pgs=2194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:44:34.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.673+0000 7ff8967ad640 1 -- 172.21.15.42:0/1999710388 >> 172.21.15.42:0/1999710388 conn(0x7ff8900a2b40 msgr2=0x7ff8900a2f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:44:34.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.673+0000 7ff8967ad640 1 -- 172.21.15.42:0/1999710388 shutdown_connections 2024-09-16T03:44:34.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.673+0000 7ff8967ad640 1 -- 172.21.15.42:0/1999710388 wait complete. 2024-09-16T03:44:34.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.673+0000 7ff8967ad640 10 monclient: build_initial_monmap 2024-09-16T03:44:34.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.673+0000 7ff8967ad640 10 monclient: monmap: 2024-09-16T03:44:34.691 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:44:34.691 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:44:34.692 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:44:34.692 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:44:34.692 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:44:34.692 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:44:34.692 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:44:34.692 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:44:34.692 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:44:34.692 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:44:34.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.673+0000 7ff8967ad640 1 librados: starting msgr at 2024-09-16T03:44:34.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.673+0000 7ff8967ad640 1 librados: starting objecter 2024-09-16T03:44:34.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.673+0000 7ff8967ad640 1 Processor -- start 2024-09-16T03:44:34.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.673+0000 7ff8967ad640 1 -- start start 2024-09-16T03:44:34.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.673+0000 7ff8967ad640 1 librados: setting wanted keys 2024-09-16T03:44:34.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.673+0000 7ff8967ad640 1 librados: calling monclient init 2024-09-16T03:44:34.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.673+0000 7ff8967ad640 10 monclient: init 2024-09-16T03:44:34.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.677+0000 7ff8967ad640 10 monclient: _reopen_session rank -1 2024-09-16T03:44:34.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.677+0000 7ff8967ad640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:44:34.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.677+0000 7ff8967ad640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff8900a0de0 0x7ff8901cee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:44:34.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.677+0000 7ff8967ad640 10 monclient(hunting): picked mon.noname-c con 0x7ff8900a0de0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:44:34.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.677+0000 7ff8967ad640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff8900a25b0 0x7ff8901cf6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:44:34.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.677+0000 7ff8967ad640 10 monclient(hunting): picked mon.noname-b con 0x7ff8900a25b0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:44:34.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.677+0000 7ff8967ad640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff89013a5f0 0x7ff89009d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:44:34.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.677+0000 7ff8967ad640 10 monclient(hunting): picked mon.noname-a con 0x7ff89013a5f0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:44:34.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.677+0000 7ff8967ad640 10 monclient(hunting): start opening mon connection 2024-09-16T03:44:34.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.677+0000 7ff8967ad640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7ff890140aa0 con 0x7ff89013a5f0 2024-09-16T03:44:34.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.677+0000 7ff8967ad640 10 monclient(hunting): start opening mon connection 2024-09-16T03:44:34.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.677+0000 7ff8967ad640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff890140920 con 0x7ff8900a25b0 2024-09-16T03:44:34.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.677+0000 7ff8967ad640 10 monclient(hunting): start opening mon connection 2024-09-16T03:44:34.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.677+0000 7ff8967ad640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7ff890140c20 con 0x7ff8900a0de0 2024-09-16T03:44:34.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.677+0000 7ff8967ad640 10 monclient(hunting): _renew_subs 2024-09-16T03:44:34.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.677+0000 7ff8967ad640 10 monclient(hunting): authenticate will time out at 1121.648376s 2024-09-16T03:44:34.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.677+0000 7ff88ffff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff8900a0de0 0x7ff8901cee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:44:34.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.677+0000 7ff88ffff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff8900a0de0 0x7ff8901cee60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:39124/0 (socket says 172.21.15.42:39124) 2024-09-16T03:44:34.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.677+0000 7ff88ffff640 1 -- 172.21.15.42:0/1572511341 learned_addr learned my addr 172.21.15.42:0/1572511341 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:44:34.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.677+0000 7ff88ffff640 10 monclient(hunting): get_auth_request con 0x7ff8900a0de0 auth_method 0 2024-09-16T03:44:34.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.677+0000 7ff88ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:44:34.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.677+0000 7ff88ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:44:34.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.677+0000 7ff88ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:44:34.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.677+0000 7ff894d4a640 1 --2- 172.21.15.42:0/1572511341 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff89013a5f0 0x7ff89009d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:44:34.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.677+0000 7ff894d4a640 10 monclient(hunting): get_auth_request con 0x7ff89013a5f0 auth_method 0 2024-09-16T03:44:34.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.677+0000 7ff894d4a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:44:34.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.677+0000 7ff894d4a640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:44:34.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.677+0000 7ff894d4a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:44:34.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.677+0000 7ff88ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:44:34.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.677+0000 7ff88ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:44:34.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.677+0000 7ff88ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:44:34.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.677+0000 7ff894d4a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:44:34.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.677+0000 7ff894d4a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:44:34.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.677+0000 7ff894d4a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:44:34.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.677+0000 7ff88ffff640 10 monclient(hunting): handle_auth_done global_id 12689 payload 824 2024-09-16T03:44:34.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.677+0000 7ff88ffff640 1 -- 172.21.15.42:0/1572511341 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff8900a25b0 msgr2=0x7ff8901cf6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:44:34.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.677+0000 7ff88ffff640 1 --2- 172.21.15.42:0/1572511341 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff8900a25b0 0x7ff8901cf6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:44:34.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.677+0000 7ff88ffff640 1 -- 172.21.15.42:0/1572511341 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff89013a5f0 msgr2=0x7ff89009d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:44:34.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.677+0000 7ff88ffff640 1 --2- 172.21.15.42:0/1572511341 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff89013a5f0 0x7ff89009d570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:44:34.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.677+0000 7ff88ffff640 10 monclient: _finish_hunting 0 2024-09-16T03:44:34.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.677+0000 7ff88ffff640 1 monclient: found mon.noname-c 2024-09-16T03:44:34.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.677+0000 7ff88ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:44:34.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.677+0000 7ff88ffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:44:34.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.677+0000 7ff88ffff640 1 -- 172.21.15.42:0/1572511341 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff89009f210 con 0x7ff8900a0de0 2024-09-16T03:44:34.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.677+0000 7ff88ffff640 1 --2- 172.21.15.42:0/1572511341 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff8900a0de0 0x7ff8901cee60 secure :-1 s=READY pgs=2195 cs=0 l=1 rev1=1 crypto rx=0x7ff88000c760 tx=0x7ff88000cc20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:44:34.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.677+0000 7ff88cff9640 10 client.?.objecter ms_handle_connect 0x7ff8900a0de0 2024-09-16T03:44:34.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.677+0000 7ff88cff9640 10 client.?.objecter resend_mon_ops 2024-09-16T03:44:34.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.677+0000 7ff88cff9640 1 -- 172.21.15.42:0/1572511341 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff880017070 con 0x7ff8900a0de0 2024-09-16T03:44:34.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.677+0000 7ff88cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:44:34.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.677+0000 7ff88cff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:44:34.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.677+0000 7ff88cff9640 10 monclient: dump: 2024-09-16T03:44:34.698 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:44:34.698 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:44:34.699 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:34.699 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:34.699 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:44:34.699 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:44:34.699 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:44:34.699 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:44:34.699 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:44:34.699 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:44:34.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.677+0000 7ff88cff9640 10 monclient: _finish_auth 0 2024-09-16T03:44:34.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.677+0000 7ff88cff9640 10 monclient: _check_auth_tickets 2024-09-16T03:44:34.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.677+0000 7ff88cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:44:34.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.677+0000 7ff88cff9640 1 -- 172.21.15.42:0/1572511341 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff880005ae0 con 0x7ff8900a0de0 2024-09-16T03:44:34.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.677+0000 7ff88cff9640 10 monclient: handle_config config(0 keys) 2024-09-16T03:44:34.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.677+0000 7ff88cff9640 1 -- 172.21.15.42:0/1572511341 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff880007070 con 0x7ff8900a0de0 2024-09-16T03:44:34.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.677+0000 7ff88cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:44:34.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.677+0000 7ff88cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:44:34.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.677+0000 7ff88cff9640 10 monclient: dump: 2024-09-16T03:44:34.700 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:44:34.700 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:44:34.700 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:34.700 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:34.700 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:44:34.700 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:44:34.700 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:44:34.700 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:44:34.701 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:44:34.701 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:44:34.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.677+0000 7ff88e7fc640 4 set_mon_vals no callback set 2024-09-16T03:44:34.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.677+0000 7ff8967ad640 5 monclient: authenticate success, global_id 12689 2024-09-16T03:44:34.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.677+0000 7ff8967ad640 10 monclient: _renew_subs 2024-09-16T03:44:34.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.677+0000 7ff8967ad640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:44:34.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.677+0000 7ff8967ad640 1 -- 172.21.15.42:0/1572511341 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff89013b0b0 con 0x7ff8900a0de0 2024-09-16T03:44:34.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.677+0000 7ff8967ad640 10 client.12689.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:44:34.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.677+0000 7ff8967ad640 10 monclient: _renew_subs 2024-09-16T03:44:34.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.677+0000 7ff8967ad640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:44:34.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.677+0000 7ff8967ad640 1 -- 172.21.15.42:0/1572511341 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff89009d230 con 0x7ff8900a0de0 2024-09-16T03:44:34.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.677+0000 7ff8967ad640 1 librados: init done 2024-09-16T03:44:34.702 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.677+0000 7ff88cff9640 1 -- 172.21.15.42:0/1572511341 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7ff88001a070 con 0x7ff8900a0de0 2024-09-16T03:44:34.702 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.677+0000 7ff8967ad640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:44:34.702 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.677+0000 7ff8967ad640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:44:34.702 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.677+0000 7ff8967ad640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:44:34.702 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.677+0000 7ff8967ad640 1 -- 172.21.15.42:0/1572511341 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff8900a11c0 con 0x7ff8900a0de0 2024-09-16T03:44:34.702 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.677+0000 7ff88cff9640 1 --2- 172.21.15.42:0/1572511341 >> v2:172.21.15.154:6812/254845342 conn(0x7ff87c037810 0x7ff87c039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:44:34.702 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.677+0000 7ff88cff9640 1 -- 172.21.15.42:0/1572511341 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(56..56 src has 1..56) ==== 9613+0+0 (secure 0 0 0) 0x7ff880049410 con 0x7ff8900a0de0 2024-09-16T03:44:34.702 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.677+0000 7ff88cff9640 10 client.12689.objecter ms_dispatch 0x7ff890000ba0 osd_map(56..56 src has 1..56) 2024-09-16T03:44:34.702 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.677+0000 7ff88cff9640 3 client.12689.objecter handle_osd_map got epochs [56,56] > 0 2024-09-16T03:44:34.702 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.677+0000 7ff88cff9640 3 client.12689.objecter handle_osd_map decoding full epoch 56 2024-09-16T03:44:34.702 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.677+0000 7ff88cff9640 20 client.12689.objecter dump_active .. 0 homeless 2024-09-16T03:44:34.702 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.681+0000 7ff88f7fe640 1 --2- 172.21.15.42:0/1572511341 >> v2:172.21.15.154:6812/254845342 conn(0x7ff87c037810 0x7ff87c039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:44:34.702 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.681+0000 7ff88f7fe640 10 monclient: get_auth_request con 0x7ff87c037810 auth_method 0 2024-09-16T03:44:34.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.685+0000 7ff88cff9640 1 -- 172.21.15.42:0/1572511341 <== mon.2 v2:172.21.15.42:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7ff88001a360 con 0x7ff8900a0de0 2024-09-16T03:44:34.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.685+0000 7ff88cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:44:34.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.685+0000 7ff88cff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:44:34.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.693+0000 7ff88f7fe640 1 --2- 172.21.15.42:0/1572511341 >> v2:172.21.15.154:6812/254845342 conn(0x7ff87c037810 0x7ff87c039cd0 secure :-1 s=READY pgs=3233 cs=0 l=1 rev1=1 crypto rx=0x7ff8840ad000 tx=0x7ff884008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:44:34.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.693+0000 7ff88cff9640 10 client.12689.objecter ms_handle_connect 0x7ff87c037810 2024-09-16T03:44:34.938 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.933+0000 7ff8967ad640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:44:34.938 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.933+0000 7ff8967ad640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:44:34.938 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.933+0000 7ff8967ad640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:44:34.938 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.933+0000 7ff8967ad640 1 -- 172.21.15.42:0/1572511341 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7ff890072fa0 con 0x7ff8900a0de0 2024-09-16T03:44:34.938 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.933+0000 7ff88cff9640 1 -- 172.21.15.42:0/1572511341 <== mon.2 v2:172.21.15.42:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7ff88001bdd0 con 0x7ff8900a0de0 2024-09-16T03:44:34.938 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.933+0000 7ff88cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:44:34.938 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.933+0000 7ff88cff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:44:34.939 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":22,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:44:34.940 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.933+0000 7ff8967ad640 10 librados: watch_flush enter 2024-09-16T03:44:34.941 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.933+0000 7ff8967ad640 10 librados: watch_flush exit 2024-09-16T03:44:34.941 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.933+0000 7ff8967ad640 20 client.12689.objecter shutdown clearing up homeless session... 2024-09-16T03:44:34.941 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.933+0000 7ff8967ad640 10 client.12689.objecter successfully canceled tick 2024-09-16T03:44:34.941 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.933+0000 7ff8967ad640 1 -- 172.21.15.42:0/1572511341 >> v2:172.21.15.154:6812/254845342 conn(0x7ff87c037810 msgr2=0x7ff87c039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:44:34.941 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.933+0000 7ff8967ad640 1 --2- 172.21.15.42:0/1572511341 >> v2:172.21.15.154:6812/254845342 conn(0x7ff87c037810 0x7ff87c039cd0 secure :-1 s=READY pgs=3233 cs=0 l=1 rev1=1 crypto rx=0x7ff8840ad000 tx=0x7ff884008040 comp rx=0 tx=0).stop 2024-09-16T03:44:34.941 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.933+0000 7ff8967ad640 10 monclient: shutdown 2024-09-16T03:44:34.941 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.933+0000 7ff8967ad640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:44:34.941 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.933+0000 7ff8967ad640 1 -- 172.21.15.42:0/1572511341 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff8900a0de0 msgr2=0x7ff8901cee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:44:34.941 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.933+0000 7ff8967ad640 1 --2- 172.21.15.42:0/1572511341 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff8900a0de0 0x7ff8901cee60 secure :-1 s=READY pgs=2195 cs=0 l=1 rev1=1 crypto rx=0x7ff88000c760 tx=0x7ff88000cc20 comp rx=0 tx=0).stop 2024-09-16T03:44:34.941 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.933+0000 7ff8967ad640 1 -- 172.21.15.42:0/1572511341 shutdown_connections 2024-09-16T03:44:34.941 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.933+0000 7ff8967ad640 1 --2- 172.21.15.42:0/1572511341 >> v2:172.21.15.154:6812/254845342 conn(0x7ff87c037810 0x7ff87c039cd0 unknown :-1 s=CLOSED pgs=3233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:44:34.941 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.933+0000 7ff8967ad640 1 --2- 172.21.15.42:0/1572511341 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff89013a5f0 0x7ff89009d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:44:34.941 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.933+0000 7ff8967ad640 1 --2- 172.21.15.42:0/1572511341 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff8900a25b0 0x7ff8901cf6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:44:34.942 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.933+0000 7ff8967ad640 1 --2- 172.21.15.42:0/1572511341 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff8900a0de0 0x7ff8901cee60 unknown :-1 s=CLOSED pgs=2195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:44:34.942 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.933+0000 7ff8967ad640 1 -- 172.21.15.42:0/1572511341 >> 172.21.15.42:0/1572511341 conn(0x7ff8900a2b40 msgr2=0x7ff89013d190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:44:34.942 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.933+0000 7ff8967ad640 1 -- 172.21.15.42:0/1572511341 shutdown_connections 2024-09-16T03:44:34.942 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.933+0000 7ff8967ad640 1 -- 172.21.15.42:0/1572511341 wait complete. 2024-09-16T03:44:34.942 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:34.937+0000 7ff8967ad640 1 librados: shutdown 2024-09-16T03:44:34.958 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T03:44:34.958 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T03:44:34.958 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T03:44:34.958 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T03:44:35.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.077+0000 7fb7d16cd640 10 monclient: get_monmap_and_config 2024-09-16T03:44:35.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.077+0000 7fb7d16cd640 10 monclient: build_initial_monmap 2024-09-16T03:44:35.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.077+0000 7fb7d16cd640 10 monclient: monmap: 2024-09-16T03:44:35.082 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:44:35.082 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:44:35.082 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:44:35.081471+0000 2024-09-16T03:44:35.083 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:44:35.081471+0000 2024-09-16T03:44:35.083 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:44:35.083 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:44:35.083 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:44:35.083 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:44:35.083 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:44:35.083 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:44:35.083 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:44:35.083 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:44:35.093 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.077+0000 7fb7d16cd640 1 Processor -- start 2024-09-16T03:44:35.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.077+0000 7fb7d16cd640 1 -- start start 2024-09-16T03:44:35.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.077+0000 7fb7d16cd640 10 monclient: init 2024-09-16T03:44:35.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.077+0000 7fb7d16cd640 10 monclient: _reopen_session rank -1 2024-09-16T03:44:35.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.077+0000 7fb7d16cd640 10 monclient: _add_conns ranks=[0,4,1,2,3] 2024-09-16T03:44:35.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.077+0000 7fb7d16cd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb7cc0a39b0 addr v1:172.21.15.42:6789/0 2024-09-16T03:44:35.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.077+0000 7fb7d16cd640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fb7cc0a3d90 0x7fb7cc13c380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:44:35.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.077+0000 7fb7d16cd640 10 monclient(hunting): picked mon.noname-c con 0x7fb7cc0a3d90 addr v2:172.21.15.154:3300/0 2024-09-16T03:44:35.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.077+0000 7fb7d16cd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb7cc13c8c0 addr v1:172.21.15.154:6789/0 2024-09-16T03:44:35.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.077+0000 7fb7d16cd640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb7cc134290 con 0x7fb7cc0a39b0 2024-09-16T03:44:35.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.077+0000 7fb7d16cd640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb7cc09c140 con 0x7fb7cc13c8c0 2024-09-16T03:44:35.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.077+0000 7fb7d16cd640 10 monclient(hunting): start opening mon connection 2024-09-16T03:44:35.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.077+0000 7fb7d16cd640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fb7cc09c420 con 0x7fb7cc0a3d90 2024-09-16T03:44:35.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.077+0000 7fb7d16cd640 10 monclient(hunting): _renew_subs 2024-09-16T03:44:35.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.077+0000 7fb7d16cd640 10 monclient(hunting): authenticate will time out at 1122.050736s 2024-09-16T03:44:35.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.077+0000 7fb7caffd640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fb7cc0a39b0 0x7fb7cc133b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:47910/0 (socket says 172.21.15.42:47910) 2024-09-16T03:44:35.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.077+0000 7fb7caffd640 1 -- 172.21.15.42:0/168968180 learned_addr learned my addr 172.21.15.42:0/168968180 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:44:35.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.077+0000 7fb7ca7fc640 1 --2- 172.21.15.42:0/168968180 >> v2:172.21.15.154:3300/0 conn(0x7fb7cc0a3d90 0x7fb7cc13c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:44:35.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.077+0000 7fb7ca7fc640 10 monclient(hunting): get_auth_request con 0x7fb7cc0a3d90 auth_method 0 2024-09-16T03:44:35.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.077+0000 7fb7ca7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:44:35.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.077+0000 7fb7ca7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:44:35.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.077+0000 7fb7ca7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:44:35.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.077+0000 7fb7ca7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:44:35.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.077+0000 7fb7ca7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:44:35.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.077+0000 7fb7ca7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:44:35.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.077+0000 7fb7c9ffb640 1 -- 172.21.15.42:0/168968180 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 608607822 0 0) 0x7fb7cc134290 con 0x7fb7cc0a39b0 2024-09-16T03:44:35.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.077+0000 7fb7c9ffb640 10 monclient(hunting): _negotiate 2024-09-16T03:44:35.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.077+0000 7fb7c9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:44:35.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.077+0000 7fb7c9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:44:35.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.077+0000 7fb7c9ffb640 10 monclient(hunting): my global_id is 27471 2024-09-16T03:44:35.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.077+0000 7fb7c9ffb640 1 -- 172.21.15.42:0/168968180 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb7b8003530 con 0x7fb7cc0a39b0 2024-09-16T03:44:35.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.077+0000 7fb7c9ffb640 1 -- 172.21.15.42:0/168968180 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2131578807 0 0) 0x7fb7b8003530 con 0x7fb7cc0a39b0 2024-09-16T03:44:35.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.077+0000 7fb7c9ffb640 1 -- 172.21.15.42:0/168968180 >> v2:172.21.15.154:3300/0 conn(0x7fb7cc0a3d90 msgr2=0x7fb7cc13c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:44:35.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.077+0000 7fb7c9ffb640 1 --2- 172.21.15.42:0/168968180 >> v2:172.21.15.154:3300/0 conn(0x7fb7cc0a3d90 0x7fb7cc13c380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:44:35.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.077+0000 7fb7c9ffb640 1 -- 172.21.15.42:0/168968180 >> v1:172.21.15.154:6789/0 conn(0x7fb7cc13c8c0 legacy=0x7fb7cc13ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:44:35.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.077+0000 7fb7c9ffb640 10 monclient: _finish_hunting 0 2024-09-16T03:44:35.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.077+0000 7fb7c9ffb640 1 monclient: found mon.noname-a-legacy 2024-09-16T03:44:35.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.077+0000 7fb7c9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:44:35.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.077+0000 7fb7c9ffb640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.42:6789/0 2024-09-16T03:44:35.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.077+0000 7fb7c9ffb640 1 -- 172.21.15.42:0/168968180 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7cc06dd60 con 0x7fb7cc0a39b0 2024-09-16T03:44:35.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.077+0000 7fb7c9ffb640 10 monclient: _finish_auth 0 2024-09-16T03:44:35.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.077+0000 7fb7c9ffb640 10 monclient: _check_auth_tickets 2024-09-16T03:44:35.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.077+0000 7fb7c9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:44:35.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.077+0000 7fb7c9ffb640 1 -- 172.21.15.42:0/168968180 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2177413734 0 0) 0x7fb7bc001680 con 0x7fb7cc0a39b0 2024-09-16T03:44:35.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.077+0000 7fb7c9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:44:35.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.077+0000 7fb7c9ffb640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T03:44:35.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.077+0000 7fb7c9ffb640 10 monclient: dump: 2024-09-16T03:44:35.097 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:44:35.097 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:44:35.097 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:35.097 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:35.098 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:44:35.098 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:44:35.098 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:44:35.098 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:44:35.098 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:44:35.098 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:44:35.098 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.077+0000 7fb7c9ffb640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] but i'm connected to v1:172.21.15.42:6789/0, reconnecting 2024-09-16T03:44:35.098 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.077+0000 7fb7c9ffb640 10 monclient: _reopen_session rank -1 2024-09-16T03:44:35.098 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.077+0000 7fb7c9ffb640 1 -- 172.21.15.42:0/168968180 >> v1:172.21.15.42:6789/0 conn(0x7fb7cc0a39b0 legacy=0x7fb7cc133b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:44:35.098 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.077+0000 7fb7c9ffb640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:44:35.098 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.077+0000 7fb7c9ffb640 1 --2- 172.21.15.42:0/168968180 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb7b80039c0 0x7fb7b8003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:44:35.098 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.077+0000 7fb7c9ffb640 10 monclient(hunting): picked mon.b con 0x7fb7b80039c0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:44:35.098 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.077+0000 7fb7c9ffb640 1 --2- 172.21.15.42:0/168968180 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb7b8004810 0x7fb7b8004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:44:35.099 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.077+0000 7fb7c9ffb640 10 monclient(hunting): picked mon.a con 0x7fb7b8004810 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:44:35.099 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.077+0000 7fb7c9ffb640 1 --2- 172.21.15.42:0/168968180 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb7b8005220 0x7fb7b800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:44:35.099 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.077+0000 7fb7c9ffb640 10 monclient(hunting): picked mon.c con 0x7fb7b8005220 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:44:35.099 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.077+0000 7fb7c9ffb640 10 monclient(hunting): start opening mon connection 2024-09-16T03:44:35.099 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.077+0000 7fb7c9ffb640 1 -- 172.21.15.42:0/168968180 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fb7cc09c420 con 0x7fb7b8004810 2024-09-16T03:44:35.099 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.077+0000 7fb7c9ffb640 10 monclient(hunting): start opening mon connection 2024-09-16T03:44:35.099 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.077+0000 7fb7c9ffb640 1 -- 172.21.15.42:0/168968180 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fb7b800c170 con 0x7fb7b80039c0 2024-09-16T03:44:35.099 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.077+0000 7fb7c9ffb640 10 monclient(hunting): start opening mon connection 2024-09-16T03:44:35.099 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.077+0000 7fb7c9ffb640 1 -- 172.21.15.42:0/168968180 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fb7b800c370 con 0x7fb7b8005220 2024-09-16T03:44:35.099 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.077+0000 7fb7c9ffb640 10 monclient(hunting): _renew_subs 2024-09-16T03:44:35.100 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.077+0000 7fb7c9ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:44:35.100 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.077+0000 7fb7ca7fc640 1 --2- 172.21.15.42:0/168968180 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb7b8004810 0x7fb7b8004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:44:35.100 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.077+0000 7fb7ca7fc640 10 monclient(hunting): get_auth_request con 0x7fb7b8004810 auth_method 0 2024-09-16T03:44:35.100 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.077+0000 7fb7ca7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:44:35.100 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.077+0000 7fb7ca7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:44:35.100 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.077+0000 7fb7ca7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:44:35.100 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.077+0000 7fb7cb7fe640 1 --2- 172.21.15.42:0/168968180 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb7b8005220 0x7fb7b800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:44:35.100 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.077+0000 7fb7caffd640 1 --2- 172.21.15.42:0/168968180 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb7b80039c0 0x7fb7b8003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:44:35.100 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.077+0000 7fb7cb7fe640 10 monclient(hunting): get_auth_request con 0x7fb7b8005220 auth_method 0 2024-09-16T03:44:35.100 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.077+0000 7fb7cb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:44:35.100 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.077+0000 7fb7cb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:44:35.100 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.077+0000 7fb7cb7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:44:35.100 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.077+0000 7fb7caffd640 10 monclient(hunting): get_auth_request con 0x7fb7b80039c0 auth_method 0 2024-09-16T03:44:35.101 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.077+0000 7fb7caffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:44:35.101 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.077+0000 7fb7caffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:44:35.101 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.077+0000 7fb7caffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:44:35.101 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.081+0000 7fb7ca7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:44:35.101 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.081+0000 7fb7ca7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:44:35.101 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.081+0000 7fb7ca7fc640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T03:44:35.101 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.081+0000 7fb7cb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:44:35.101 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.081+0000 7fb7cb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:44:35.101 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.081+0000 7fb7cb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T03:44:35.101 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.081+0000 7fb7ca7fc640 10 monclient(hunting): handle_auth_done global_id 27471 payload 293 2024-09-16T03:44:35.101 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.081+0000 7fb7ca7fc640 1 -- 172.21.15.42:0/168968180 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb7b8005220 msgr2=0x7fb7b800b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:44:35.101 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.081+0000 7fb7ca7fc640 1 --2- 172.21.15.42:0/168968180 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb7b8005220 0x7fb7b800b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:44:35.101 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.081+0000 7fb7ca7fc640 1 -- 172.21.15.42:0/168968180 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb7b80039c0 msgr2=0x7fb7b8003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:44:35.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.081+0000 7fb7ca7fc640 1 --2- 172.21.15.42:0/168968180 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb7b80039c0 0x7fb7b8003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:44:35.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.081+0000 7fb7ca7fc640 10 monclient: _finish_hunting 0 2024-09-16T03:44:35.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.081+0000 7fb7ca7fc640 1 monclient: found mon.a 2024-09-16T03:44:35.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.081+0000 7fb7ca7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:44:35.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.081+0000 7fb7ca7fc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:44:35.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.081+0000 7fb7ca7fc640 1 -- 172.21.15.42:0/168968180 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7b8003530 con 0x7fb7b8004810 2024-09-16T03:44:35.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.081+0000 7fb7ca7fc640 10 monclient: _finish_auth 0 2024-09-16T03:44:35.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.081+0000 7fb7ca7fc640 10 monclient: _check_auth_tickets 2024-09-16T03:44:35.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.081+0000 7fb7ca7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:44:35.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.081+0000 7fb7cb7fe640 1 -- 172.21.15.42:0/168968180 reap_dead start 2024-09-16T03:44:35.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.081+0000 7fb7d16cd640 5 monclient: authenticate success, global_id 27471 2024-09-16T03:44:35.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.081+0000 7fb7d16cd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:44:35.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.081+0000 7fb7ca7fc640 1 --2- 172.21.15.42:0/168968180 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb7b8004810 0x7fb7b8004c90 secure :-1 s=READY pgs=7434 cs=0 l=1 rev1=1 crypto rx=0x7fb7b4009ae0 tx=0x7fb7b401ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c091b41fb94d859f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:44:35.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.081+0000 7fb7c9ffb640 1 -- 172.21.15.42:0/168968180 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7b40096b0 con 0x7fb7b8004810 2024-09-16T03:44:35.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.081+0000 7fb7c9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:44:35.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.081+0000 7fb7c9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:44:35.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.081+0000 7fb7c9ffb640 10 monclient: dump: 2024-09-16T03:44:35.103 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:44:35.103 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:44:35.103 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:35.103 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:35.103 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:44:35.103 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:44:35.103 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:44:35.103 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:44:35.103 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:44:35.103 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:44:35.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.081+0000 7fb7d16cd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:44:35.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.081+0000 7fb7c9ffb640 1 -- 172.21.15.42:0/168968180 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb7b40a6430 con 0x7fb7b8004810 2024-09-16T03:44:35.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.081+0000 7fb7c9ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T03:44:35.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.081+0000 7fb7c9ffb640 1 -- 172.21.15.42:0/168968180 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7b40a6820 con 0x7fb7b8004810 2024-09-16T03:44:35.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.081+0000 7fb7d16cd640 10 monclient: get_monmap_and_config success 2024-09-16T03:44:35.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.081+0000 7fb7d16cd640 4 set_mon_vals no callback set 2024-09-16T03:44:35.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.081+0000 7fb7c9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:44:35.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.081+0000 7fb7c9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:44:35.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.081+0000 7fb7c9ffb640 10 monclient: dump: 2024-09-16T03:44:35.104 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:44:35.104 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:44:35.104 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:35.104 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:35.104 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:44:35.104 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:44:35.105 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:44:35.105 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:44:35.105 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:44:35.105 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:44:35.105 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.081+0000 7fb7d16cd640 10 monclient: shutdown 2024-09-16T03:44:35.105 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.081+0000 7fb7d16cd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:44:35.105 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.081+0000 7fb7d16cd640 1 -- 172.21.15.42:0/168968180 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb7b8004810 msgr2=0x7fb7b8004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:44:35.105 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.081+0000 7fb7d16cd640 1 --2- 172.21.15.42:0/168968180 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb7b8004810 0x7fb7b8004c90 secure :-1 s=READY pgs=7434 cs=0 l=1 rev1=1 crypto rx=0x7fb7b4009ae0 tx=0x7fb7b401ca50 comp rx=0 tx=0).stop 2024-09-16T03:44:35.105 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.081+0000 7fb7d16cd640 1 -- 172.21.15.42:0/168968180 shutdown_connections 2024-09-16T03:44:35.105 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.081+0000 7fb7d16cd640 1 --2- 172.21.15.42:0/168968180 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb7b8004810 0x7fb7b8004c90 unknown :-1 s=CLOSED pgs=7434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:44:35.105 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.081+0000 7fb7d16cd640 1 -- 172.21.15.42:0/168968180 >> 172.21.15.42:0/168968180 conn(0x7fb7cc0a9fb0 msgr2=0x7fb7cc0aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:44:35.105 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.081+0000 7fb7d16cd640 1 -- 172.21.15.42:0/168968180 shutdown_connections 2024-09-16T03:44:35.105 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.081+0000 7fb7d16cd640 1 -- 172.21.15.42:0/168968180 wait complete. 2024-09-16T03:44:35.106 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.081+0000 7fb7d16cd640 10 monclient: build_initial_monmap 2024-09-16T03:44:35.106 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.081+0000 7fb7d16cd640 10 monclient: monmap: 2024-09-16T03:44:35.106 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:44:35.106 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:44:35.106 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:44:35.106 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:44:35.106 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:44:35.106 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:44:35.106 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:44:35.106 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:44:35.106 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:44:35.106 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:44:35.106 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.085+0000 7fb7d16cd640 1 librados: starting msgr at 2024-09-16T03:44:35.106 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.085+0000 7fb7d16cd640 1 librados: starting objecter 2024-09-16T03:44:35.107 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.085+0000 7fb7d16cd640 1 Processor -- start 2024-09-16T03:44:35.107 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.085+0000 7fb7d16cd640 1 -- start start 2024-09-16T03:44:35.107 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.085+0000 7fb7d16cd640 1 librados: setting wanted keys 2024-09-16T03:44:35.107 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.085+0000 7fb7d16cd640 1 librados: calling monclient init 2024-09-16T03:44:35.107 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.085+0000 7fb7d16cd640 10 monclient: init 2024-09-16T03:44:35.107 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.085+0000 7fb7d16cd640 10 monclient: _reopen_session rank -1 2024-09-16T03:44:35.107 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.085+0000 7fb7d16cd640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:44:35.107 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.085+0000 7fb7d16cd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb7b8004810 0x7fb7cc09fd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:44:35.107 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.085+0000 7fb7d16cd640 10 monclient(hunting): picked mon.noname-b con 0x7fb7b8004810 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:44:35.107 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.085+0000 7fb7d16cd640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb7cc0a0280 0x7fb7cc0a06c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:44:35.107 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.085+0000 7fb7d16cd640 10 monclient(hunting): picked mon.noname-a con 0x7fb7cc0a0280 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:44:35.107 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.085+0000 7fb7d16cd640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb7cc0a0c00 0x7fb7cc1d5ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:44:35.107 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.085+0000 7fb7d16cd640 10 monclient(hunting): picked mon.noname-c con 0x7fb7cc0a0c00 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:44:35.107 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.085+0000 7fb7d16cd640 10 monclient(hunting): start opening mon connection 2024-09-16T03:44:35.108 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.085+0000 7fb7d16cd640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fb7cc137ba0 con 0x7fb7cc0a0280 2024-09-16T03:44:35.108 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.085+0000 7fb7d16cd640 10 monclient(hunting): start opening mon connection 2024-09-16T03:44:35.108 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.085+0000 7fb7d16cd640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fb7cc137a20 con 0x7fb7b8004810 2024-09-16T03:44:35.108 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.085+0000 7fb7d16cd640 10 monclient(hunting): start opening mon connection 2024-09-16T03:44:35.108 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.085+0000 7fb7d16cd640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fb7cc137d20 con 0x7fb7cc0a0c00 2024-09-16T03:44:35.108 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.085+0000 7fb7d16cd640 10 monclient(hunting): _renew_subs 2024-09-16T03:44:35.108 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.085+0000 7fb7d16cd640 10 monclient(hunting): authenticate will time out at 1122.057033s 2024-09-16T03:44:35.108 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.085+0000 7fb7ca7fc640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb7cc0a0280 0x7fb7cc0a06c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:44:35.108 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.085+0000 7fb7ca7fc640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb7cc0a0280 0x7fb7cc0a06c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:48260/0 (socket says 172.21.15.42:48260) 2024-09-16T03:44:35.108 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.085+0000 7fb7ca7fc640 1 -- 172.21.15.42:0/669116732 learned_addr learned my addr 172.21.15.42:0/669116732 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:44:35.108 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.085+0000 7fb7ca7fc640 10 monclient(hunting): get_auth_request con 0x7fb7cc0a0280 auth_method 0 2024-09-16T03:44:35.108 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.085+0000 7fb7ca7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:44:35.108 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.085+0000 7fb7ca7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:44:35.109 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.085+0000 7fb7ca7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:44:35.109 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.085+0000 7fb7cb7fe640 1 --2- 172.21.15.42:0/669116732 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb7cc0a0c00 0x7fb7cc1d5ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:44:35.109 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.085+0000 7fb7cb7fe640 10 monclient(hunting): get_auth_request con 0x7fb7cc0a0c00 auth_method 0 2024-09-16T03:44:35.109 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.085+0000 7fb7cb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:44:35.109 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.085+0000 7fb7cb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:44:35.109 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.085+0000 7fb7cb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:44:35.109 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.085+0000 7fb7ca7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:44:35.109 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.085+0000 7fb7ca7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:44:35.109 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.085+0000 7fb7ca7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:44:35.109 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.085+0000 7fb7cb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:44:35.109 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.085+0000 7fb7cb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:44:35.109 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.085+0000 7fb7cb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:44:35.109 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.085+0000 7fb7ca7fc640 10 monclient(hunting): handle_auth_done global_id 27474 payload 824 2024-09-16T03:44:35.110 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.085+0000 7fb7ca7fc640 1 -- 172.21.15.42:0/669116732 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb7cc0a0c00 msgr2=0x7fb7cc1d5ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:44:35.110 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.085+0000 7fb7ca7fc640 1 --2- 172.21.15.42:0/669116732 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb7cc0a0c00 0x7fb7cc1d5ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:44:35.110 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.085+0000 7fb7ca7fc640 1 -- 172.21.15.42:0/669116732 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb7b8004810 msgr2=0x7fb7cc09fd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:44:35.110 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.085+0000 7fb7ca7fc640 1 --2- 172.21.15.42:0/669116732 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb7b8004810 0x7fb7cc09fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:44:35.110 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.085+0000 7fb7ca7fc640 10 monclient: _finish_hunting 0 2024-09-16T03:44:35.110 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.085+0000 7fb7ca7fc640 1 monclient: found mon.noname-a 2024-09-16T03:44:35.110 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.085+0000 7fb7ca7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:44:35.110 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.085+0000 7fb7ca7fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:44:35.110 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.085+0000 7fb7ca7fc640 1 -- 172.21.15.42:0/669116732 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7cc1333c0 con 0x7fb7cc0a0280 2024-09-16T03:44:35.110 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.085+0000 7fb7ca7fc640 1 --2- 172.21.15.42:0/669116732 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb7cc0a0280 0x7fb7cc0a06c0 secure :-1 s=READY pgs=7435 cs=0 l=1 rev1=1 crypto rx=0x7fb7b401c880 tx=0x7fb7b40197f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:44:35.110 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.085+0000 7fb7c37fe640 10 client.?.objecter ms_handle_connect 0x7fb7cc0a0280 2024-09-16T03:44:35.110 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.085+0000 7fb7c37fe640 10 client.?.objecter resend_mon_ops 2024-09-16T03:44:35.111 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.085+0000 7fb7c37fe640 1 -- 172.21.15.42:0/669116732 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7b40be070 con 0x7fb7cc0a0280 2024-09-16T03:44:35.111 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.085+0000 7fb7c37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:44:35.111 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.085+0000 7fb7c37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:44:35.111 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.085+0000 7fb7c37fe640 10 monclient: dump: 2024-09-16T03:44:35.111 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:44:35.111 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:44:35.111 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:35.111 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:35.111 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:44:35.111 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:44:35.111 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:44:35.111 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:44:35.111 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:44:35.111 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:44:35.112 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.085+0000 7fb7c37fe640 10 monclient: _finish_auth 0 2024-09-16T03:44:35.112 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.085+0000 7fb7c37fe640 10 monclient: _check_auth_tickets 2024-09-16T03:44:35.112 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.085+0000 7fb7c37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:44:35.112 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.085+0000 7fb7c37fe640 1 -- 172.21.15.42:0/669116732 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb7b400f000 con 0x7fb7cc0a0280 2024-09-16T03:44:35.112 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.085+0000 7fb7c37fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:44:35.112 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.085+0000 7fb7c37fe640 1 -- 172.21.15.42:0/669116732 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7b40b9410 con 0x7fb7cc0a0280 2024-09-16T03:44:35.112 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.085+0000 7fb7d16cd640 5 monclient: authenticate success, global_id 27474 2024-09-16T03:44:35.112 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.085+0000 7fb7c97fa640 4 set_mon_vals no callback set 2024-09-16T03:44:35.112 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.085+0000 7fb7d16cd640 10 monclient: _renew_subs 2024-09-16T03:44:35.112 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.085+0000 7fb7d16cd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:44:35.112 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.085+0000 7fb7d16cd640 1 -- 172.21.15.42:0/669116732 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7cc132f70 con 0x7fb7cc0a0280 2024-09-16T03:44:35.112 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.085+0000 7fb7c37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:44:35.112 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.085+0000 7fb7c37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:44:35.113 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.085+0000 7fb7c37fe640 10 monclient: dump: 2024-09-16T03:44:35.113 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:44:35.113 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:44:35.113 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:35.113 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:35.113 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:44:35.113 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:44:35.113 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:44:35.113 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:44:35.113 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:44:35.113 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:44:35.113 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.085+0000 7fb7d16cd640 10 client.27474.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:44:35.113 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.085+0000 7fb7d16cd640 10 monclient: _renew_subs 2024-09-16T03:44:35.114 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.085+0000 7fb7d16cd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:44:35.114 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.085+0000 7fb7d16cd640 1 -- 172.21.15.42:0/669116732 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7cc1202a0 con 0x7fb7cc0a0280 2024-09-16T03:44:35.114 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.085+0000 7fb7d16cd640 1 librados: init done 2024-09-16T03:44:35.114 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.085+0000 7fb7d16cd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:44:35.114 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.085+0000 7fb7d16cd640 1 --2- 172.21.15.42:0/669116732 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb7cc0a7ab0 0x7fb7cc0a7e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:44:35.114 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.085+0000 7fb7d16cd640 10 monclient(hunting): start opening mon connection 2024-09-16T03:44:35.114 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.085+0000 7fb7d16cd640 1 -- 172.21.15.42:0/669116732 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fb7cc137ba0 con 0x7fb7cc0a7ab0 2024-09-16T03:44:35.114 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.085+0000 7fb7c37fe640 1 -- 172.21.15.42:0/669116732 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fb7b40c1070 con 0x7fb7cc0a0280 2024-09-16T03:44:35.114 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.085+0000 7fb7caffd640 1 --2- 172.21.15.42:0/669116732 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb7cc0a7ab0 0x7fb7cc0a7e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:44:35.114 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.085+0000 7fb7caffd640 10 monclient: get_auth_request con 0x7fb7cc0a7ab0 auth_method 0 2024-09-16T03:44:35.114 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.085+0000 7fb7caffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:44:35.114 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.085+0000 7fb7caffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:44:35.114 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.085+0000 7fb7caffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:44:35.114 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.085+0000 7fb7caffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:44:35.115 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.085+0000 7fb7caffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:44:35.115 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.085+0000 7fb7caffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:44:35.115 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.085+0000 7fb7c37fe640 1 --2- 172.21.15.42:0/669116732 >> v2:172.21.15.154:6812/254845342 conn(0x7fb7ac037840 0x7fb7ac039d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:44:35.115 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.085+0000 7fb7c37fe640 1 -- 172.21.15.42:0/669116732 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 9613+0+0 (secure 0 0 0) 0x7fb7b40e38d0 con 0x7fb7cc0a0280 2024-09-16T03:44:35.115 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.085+0000 7fb7c37fe640 10 client.27474.objecter ms_dispatch 0x7fb7cc000ba0 osd_map(56..56 src has 1..56) 2024-09-16T03:44:35.115 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.085+0000 7fb7c37fe640 3 client.27474.objecter handle_osd_map got epochs [56,56] > 0 2024-09-16T03:44:35.115 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.085+0000 7fb7c37fe640 3 client.27474.objecter handle_osd_map decoding full epoch 56 2024-09-16T03:44:35.115 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.085+0000 7fb7caffd640 10 monclient(hunting): handle_auth_done global_id 27483 payload 824 2024-09-16T03:44:35.115 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.085+0000 7fb7caffd640 1 -- 172.21.15.42:0/669116732 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb7cc134230 con 0x7fb7cc0a7ab0 2024-09-16T03:44:35.115 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.085+0000 7fb7cb7fe640 1 --2- 172.21.15.42:0/669116732 >> v2:172.21.15.154:6812/254845342 conn(0x7fb7ac037840 0x7fb7ac039d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:44:35.115 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.085+0000 7fb7c37fe640 20 client.27474.objecter dump_active .. 0 homeless 2024-09-16T03:44:35.115 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.085+0000 7fb7cb7fe640 10 monclient: get_auth_request con 0x7fb7ac037840 auth_method 0 2024-09-16T03:44:35.115 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.085+0000 7fb7caffd640 1 --2- 172.21.15.42:0/669116732 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb7cc0a7ab0 0x7fb7cc0a7e90 secure :-1 s=READY pgs=7437 cs=0 l=1 rev1=1 crypto rx=0x7fb7bc00e400 tx=0x7fb7bc00e8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:44:35.115 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.085+0000 7fb7c37fe640 10 client.27474.objecter ms_handle_connect 0x7fb7cc0a7ab0 2024-09-16T03:44:35.116 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.085+0000 7fb7c37fe640 10 client.27474.objecter resend_mon_ops 2024-09-16T03:44:35.116 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.089+0000 7fb7c37fe640 1 -- 172.21.15.42:0/669116732 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7bc019070 con 0x7fb7cc0a7ab0 2024-09-16T03:44:35.116 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.089+0000 7fb7c37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:44:35.116 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.089+0000 7fb7c37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:44:35.116 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.089+0000 7fb7c37fe640 10 monclient: dump: 2024-09-16T03:44:35.116 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:44:35.116 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:44:35.116 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:35.116 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:35.116 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:44:35.116 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:44:35.116 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:44:35.116 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:44:35.117 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:44:35.117 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:44:35.117 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.089+0000 7fb7cb7fe640 1 --2- 172.21.15.42:0/669116732 >> v2:172.21.15.154:6812/254845342 conn(0x7fb7ac037840 0x7fb7ac039d00 secure :-1 s=READY pgs=3303 cs=0 l=1 rev1=1 crypto rx=0x7fb7cc0a4170 tx=0x7fb7c4009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:44:35.117 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.089+0000 7fb7c37fe640 10 client.27474.objecter ms_handle_connect 0x7fb7ac037840 2024-09-16T03:44:35.117 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.089+0000 7fb7c37fe640 1 -- 172.21.15.42:0/669116732 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fb7bc010510 con 0x7fb7cc0a7ab0 2024-09-16T03:44:35.117 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.089+0000 7fb7c37fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:44:35.117 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.089+0000 7fb7c37fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:44:35.117 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.089+0000 7fb7c37fe640 1 -- 172.21.15.42:0/669116732 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb7cc0a7ab0 msgr2=0x7fb7cc0a7e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:44:35.117 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.089+0000 7fb7c37fe640 1 --2- 172.21.15.42:0/669116732 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb7cc0a7ab0 0x7fb7cc0a7e90 secure :-1 s=READY pgs=7437 cs=0 l=1 rev1=1 crypto rx=0x7fb7bc00e400 tx=0x7fb7bc00e8c0 comp rx=0 tx=0).stop 2024-09-16T03:44:35.117 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.089+0000 7fb7c37fe640 1 -- 172.21.15.42:0/669116732 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb7cc0a7ab0 msgr2=0x7fb7cc0a7e90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:44:35.117 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.089+0000 7fb7c37fe640 1 --2- 172.21.15.42:0/669116732 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb7cc0a7ab0 0x7fb7cc0a7e90 secure :-1 s=CLOSED pgs=7437 cs=0 l=1 rev1=1 crypto rx=0x7fb7bc00e400 tx=0x7fb7bc00e8c0 comp rx=0 tx=0).stop 2024-09-16T03:44:35.119 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.097+0000 7fb7d16cd640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:44:35.119 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.097+0000 7fb7d16cd640 1 --2- 172.21.15.42:0/669116732 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb7cc0a83d0 0x7fb7cc1da020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:44:35.119 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.097+0000 7fb7d16cd640 10 monclient(hunting): start opening mon connection 2024-09-16T03:44:35.119 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.097+0000 7fb7d16cd640 1 -- 172.21.15.42:0/669116732 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fb7cc0a5750 con 0x7fb7cc0a83d0 2024-09-16T03:44:35.119 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.097+0000 7fb7caffd640 1 --2- 172.21.15.42:0/669116732 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb7cc0a83d0 0x7fb7cc1da020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:44:35.119 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.097+0000 7fb7caffd640 10 monclient: get_auth_request con 0x7fb7cc0a83d0 auth_method 0 2024-09-16T03:44:35.120 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.097+0000 7fb7caffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:44:35.120 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.097+0000 7fb7caffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:44:35.120 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.097+0000 7fb7caffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:44:35.120 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.097+0000 7fb7caffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:44:35.120 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.097+0000 7fb7caffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:44:35.120 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.097+0000 7fb7caffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:44:35.120 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.097+0000 7fb7caffd640 10 monclient(hunting): handle_auth_done global_id 27507 payload 824 2024-09-16T03:44:35.120 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.097+0000 7fb7caffd640 1 -- 172.21.15.42:0/669116732 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fb7cc0a1e40 con 0x7fb7cc0a83d0 2024-09-16T03:44:35.120 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.097+0000 7fb7caffd640 1 --2- 172.21.15.42:0/669116732 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb7cc0a83d0 0x7fb7cc1da020 secure :-1 s=READY pgs=7443 cs=0 l=1 rev1=1 crypto rx=0x7fb7bc00e400 tx=0x7fb7bc000bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:44:35.120 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.097+0000 7fb7c37fe640 10 client.27474.objecter ms_handle_connect 0x7fb7cc0a83d0 2024-09-16T03:44:35.120 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.097+0000 7fb7c37fe640 10 client.27474.objecter resend_mon_ops 2024-09-16T03:44:35.120 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.097+0000 7fb7c37fe640 1 -- 172.21.15.42:0/669116732 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7bc019030 con 0x7fb7cc0a83d0 2024-09-16T03:44:35.120 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.097+0000 7fb7c37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:44:35.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.097+0000 7fb7c37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:44:35.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.097+0000 7fb7c37fe640 10 monclient: dump: 2024-09-16T03:44:35.121 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:44:35.121 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:44:35.121 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:35.121 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:35.121 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:44:35.121 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:44:35.121 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:44:35.121 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:44:35.121 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:44:35.121 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:44:35.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.097+0000 7fb7c37fe640 1 -- 172.21.15.42:0/669116732 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4715 (secure 0 0 0) 0x7fb7bc01b830 con 0x7fb7cc0a83d0 2024-09-16T03:44:35.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.097+0000 7fb7c37fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:44:35.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.097+0000 7fb7c37fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:44:35.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.097+0000 7fb7c37fe640 1 -- 172.21.15.42:0/669116732 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb7cc0a83d0 msgr2=0x7fb7cc1da020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:44:35.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.097+0000 7fb7c37fe640 1 --2- 172.21.15.42:0/669116732 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb7cc0a83d0 0x7fb7cc1da020 secure :-1 s=READY pgs=7443 cs=0 l=1 rev1=1 crypto rx=0x7fb7bc00e400 tx=0x7fb7bc000bd0 comp rx=0 tx=0).stop 2024-09-16T03:44:35.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.097+0000 7fb7c37fe640 1 -- 172.21.15.42:0/669116732 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb7cc0a83d0 msgr2=0x7fb7cc1da020 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:44:35.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.097+0000 7fb7c37fe640 1 --2- 172.21.15.42:0/669116732 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb7cc0a83d0 0x7fb7cc1da020 secure :-1 s=CLOSED pgs=7443 cs=0 l=1 rev1=1 crypto rx=0x7fb7bc00e400 tx=0x7fb7bc000bd0 comp rx=0 tx=0).stop 2024-09-16T03:44:35.122 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:44:35.122 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:44:35.122 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:44:35.122 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "leader", 2024-09-16T03:44:35.123 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 22, 2024-09-16T03:44:35.123 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 154467, 2024-09-16T03:44:35.123 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:44:35.123 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:44:35.123 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T03:44:35.123 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T03:44:35.123 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:44:35.123 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 2, 2024-09-16T03:44:35.123 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:44:35.123 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:44:35.123 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:44:35.123 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:44:35.123 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:44:35.124 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:44:35.124 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:44:35.124 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:44:35.124 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:44:35.124 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:44:35.124 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:44:35.124 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:44:35.124 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:44:35.124 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:44:35.124 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:44:35.124 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:44:35.124 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:44:35.124 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:44:35.125 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:44:35.125 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:44:35.125 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:44:35.125 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:44:35.125 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:44:35.125 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:44:35.125 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:44:35.125 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:44:35.125 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:44:35.125 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:44:35.125 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:44:35.125 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:44:35.125 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:44:35.126 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:44:35.126 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:44:35.126 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:44:35.126 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:44:35.126 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:44:35.126 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:44:35.126 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:44:35.126 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:44:35.126 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:44:35.126 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:44:35.126 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:44:35.126 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:44:35.126 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:44:35.127 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:44:35.127 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:44:35.127 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:44:35.127 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:44:35.127 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:44:35.127 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:44:35.127 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:44:35.127 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:44:35.127 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:44:35.127 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:44:35.127 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:44:35.127 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:44:35.127 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:44:35.127 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:44:35.128 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:44:35.128 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:44:35.128 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:44:35.128 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:44:35.128 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:44:35.128 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:44:35.128 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:44:35.128 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:44:35.128 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:44:35.128 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:44:35.128 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:44:35.128 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:44:35.128 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:44:35.129 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:44:35.129 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:44:35.129 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:44:35.129 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:44:35.129 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:44:35.129 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:44:35.129 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:44:35.129 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:44:35.129 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:44:35.129 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:44:35.129 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:44:35.129 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:44:35.129 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:44:35.130 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:44:35.130 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:44:35.130 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:44:35.130 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:44:35.130 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:44:35.130 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:44:35.130 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:44:35.130 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:44:35.130 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:44:35.130 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:44:35.130 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:44:35.130 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:44:35.130 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:44:35.130 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:44:35.131 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:44:35.131 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:44:35.131 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:44:35.131 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:44:35.131 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:44:35.131 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:44:35.131 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:44:35.131 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:44:35.131 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:44:35.131 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:44:35.131 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:44:35.131 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:44:35.131 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:44:35.132 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:44:35.132 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:44:35.132 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:44:35.132 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:44:35.132 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:44:35.132 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:44:35.132 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:44:35.132 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:44:35.132 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:44:35.132 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:44:35.132 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:44:35.132 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:44:35.133 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:44:35.133 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:44:35.133 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:44:35.133 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:44:35.133 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:44:35.133 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:44:35.133 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:44:35.133 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:44:35.133 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:44:35.133 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:44:35.133 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:44:35.133 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:44:35.133 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:44:35.134 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:44:35.134 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:44:35.134 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:44:35.134 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:44:35.134 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:44:35.134 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T03:44:35.134 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:44:35.134 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:44:35.134 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:44:35.134 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 4 2024-09-16T03:44:35.134 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:44:35.134 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:44:35.134 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:44:35.135 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:44:35.135 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:44:35.135 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:44:35.135 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 11 2024-09-16T03:44:35.135 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:44:35.135 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:44:35.135 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:44:35.135 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:44:35.135 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:44:35.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.097+0000 7fb7d16cd640 10 librados: watch_flush enter 2024-09-16T03:44:35.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.097+0000 7fb7d16cd640 10 librados: watch_flush exit 2024-09-16T03:44:35.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.097+0000 7fb7d16cd640 20 client.27474.objecter shutdown clearing up homeless session... 2024-09-16T03:44:35.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.097+0000 7fb7d16cd640 10 client.27474.objecter successfully canceled tick 2024-09-16T03:44:35.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.097+0000 7fb7d16cd640 1 -- 172.21.15.42:0/669116732 >> v2:172.21.15.154:6812/254845342 conn(0x7fb7ac037840 msgr2=0x7fb7ac039d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:44:35.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.097+0000 7fb7d16cd640 1 --2- 172.21.15.42:0/669116732 >> v2:172.21.15.154:6812/254845342 conn(0x7fb7ac037840 0x7fb7ac039d00 secure :-1 s=READY pgs=3303 cs=0 l=1 rev1=1 crypto rx=0x7fb7cc0a4170 tx=0x7fb7c4009040 comp rx=0 tx=0).stop 2024-09-16T03:44:35.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.097+0000 7fb7d16cd640 10 monclient: shutdown 2024-09-16T03:44:35.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.097+0000 7fb7d16cd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:44:35.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.097+0000 7fb7d16cd640 1 -- 172.21.15.42:0/669116732 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb7cc0a0280 msgr2=0x7fb7cc0a06c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:44:35.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.097+0000 7fb7d16cd640 1 --2- 172.21.15.42:0/669116732 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb7cc0a0280 0x7fb7cc0a06c0 secure :-1 s=READY pgs=7435 cs=0 l=1 rev1=1 crypto rx=0x7fb7b401c880 tx=0x7fb7b40197f0 comp rx=0 tx=0).stop 2024-09-16T03:44:35.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.097+0000 7fb7cb7fe640 1 -- 172.21.15.42:0/669116732 reap_dead start 2024-09-16T03:44:35.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.097+0000 7fb7cb7fe640 1 -- 172.21.15.42:0/669116732 reap_dead start 2024-09-16T03:44:35.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.097+0000 7fb7d16cd640 1 -- 172.21.15.42:0/669116732 shutdown_connections 2024-09-16T03:44:35.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.097+0000 7fb7d16cd640 1 -- 172.21.15.42:0/669116732 >> 172.21.15.42:0/669116732 conn(0x7fb7cc0a9fb0 msgr2=0x7fb7cc0a93a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:44:35.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.097+0000 7fb7d16cd640 1 -- 172.21.15.42:0/669116732 shutdown_connections 2024-09-16T03:44:35.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.097+0000 7fb7d16cd640 1 -- 172.21.15.42:0/669116732 wait complete. 2024-09-16T03:44:35.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.097+0000 7fb7d16cd640 1 librados: shutdown 2024-09-16T03:44:35.138 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:44:35.138 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T03:44:35.139 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T03:44:35.253 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.249+0000 7fd50aac4640 10 monclient: get_monmap_and_config 2024-09-16T03:44:35.253 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.249+0000 7fd50aac4640 10 monclient: build_initial_monmap 2024-09-16T03:44:35.253 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.249+0000 7fd50aac4640 10 monclient: monmap: 2024-09-16T03:44:35.253 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:44:35.253 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:44:35.253 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:44:35.252197+0000 2024-09-16T03:44:35.253 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:44:35.252197+0000 2024-09-16T03:44:35.253 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:44:35.253 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:44:35.253 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:44:35.253 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:44:35.254 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:44:35.254 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:44:35.254 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:44:35.254 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:44:35.256 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.249+0000 7fd50aac4640 1 Processor -- start 2024-09-16T03:44:35.256 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.249+0000 7fd50aac4640 1 -- start start 2024-09-16T03:44:35.256 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.249+0000 7fd50aac4640 10 monclient: init 2024-09-16T03:44:35.256 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.249+0000 7fd50aac4640 10 monclient: _reopen_session rank -1 2024-09-16T03:44:35.256 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.249+0000 7fd50aac4640 10 monclient: _add_conns ranks=[2,0,3,4,1] 2024-09-16T03:44:35.256 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.249+0000 7fd50aac4640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd5040a39b0 0x7fd504133b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:44:35.257 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.249+0000 7fd50aac4640 10 monclient(hunting): picked mon.noname-b con 0x7fd5040a39b0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:44:35.257 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.249+0000 7fd50aac4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd5040a3d90 addr v1:172.21.15.42:6789/0 2024-09-16T03:44:35.257 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.249+0000 7fd50aac4640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd504137710 0x7fd50413f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:44:35.257 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.249+0000 7fd50aac4640 10 monclient(hunting): picked mon.noname-a con 0x7fd504137710 addr v2:172.21.15.42:3300/0 2024-09-16T03:44:35.257 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.249+0000 7fd50aac4640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd50409c140 con 0x7fd5040a3d90 2024-09-16T03:44:35.257 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.249+0000 7fd50aac4640 10 monclient(hunting): start opening mon connection 2024-09-16T03:44:35.257 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.249+0000 7fd50aac4640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7fd50409c420 con 0x7fd504137710 2024-09-16T03:44:35.257 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.249+0000 7fd50aac4640 10 monclient(hunting): start opening mon connection 2024-09-16T03:44:35.257 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.249+0000 7fd50aac4640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fd50406dd60 con 0x7fd5040a39b0 2024-09-16T03:44:35.257 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.249+0000 7fd50aac4640 10 monclient(hunting): _renew_subs 2024-09-16T03:44:35.257 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.249+0000 7fd50aac4640 10 monclient(hunting): authenticate will time out at 1122.221836s 2024-09-16T03:44:35.258 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.249+0000 7fd503fff640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fd5040a3d90 0x7fd504137000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:47916/0 (socket says 172.21.15.42:47916) 2024-09-16T03:44:35.258 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.249+0000 7fd503fff640 1 -- 172.21.15.42:0/674447635 learned_addr learned my addr 172.21.15.42:0/674447635 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:44:35.258 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.249+0000 7fd508860640 1 --2- 172.21.15.42:0/674447635 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd5040a39b0 0x7fd504133b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:44:35.258 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.249+0000 7fd508860640 10 monclient(hunting): get_auth_request con 0x7fd5040a39b0 auth_method 0 2024-09-16T03:44:35.258 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.249+0000 7fd508860640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:44:35.258 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.249+0000 7fd508860640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:44:35.258 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.249+0000 7fd508860640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:44:35.258 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.249+0000 7fd509061640 1 --2- 172.21.15.42:0/674447635 >> v2:172.21.15.42:3300/0 conn(0x7fd504137710 0x7fd50413f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:44:35.258 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.249+0000 7fd509061640 10 monclient(hunting): get_auth_request con 0x7fd504137710 auth_method 0 2024-09-16T03:44:35.258 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.249+0000 7fd509061640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:44:35.258 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.249+0000 7fd509061640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:44:35.258 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.249+0000 7fd509061640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:44:35.258 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.249+0000 7fd508860640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:44:35.259 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.249+0000 7fd508860640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:44:35.259 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.249+0000 7fd508860640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:44:35.259 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.249+0000 7fd509061640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:44:35.259 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.249+0000 7fd509061640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:44:35.259 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.249+0000 7fd509061640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:44:35.259 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.249+0000 7fd5037fe640 1 -- 172.21.15.42:0/674447635 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3040214355 0 0) 0x7fd50409c140 con 0x7fd5040a3d90 2024-09-16T03:44:35.259 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.249+0000 7fd5037fe640 10 monclient(hunting): _negotiate 2024-09-16T03:44:35.259 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.249+0000 7fd5037fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:44:35.259 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.249+0000 7fd5037fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:44:35.259 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.249+0000 7fd5037fe640 10 monclient(hunting): my global_id is 27528 2024-09-16T03:44:35.259 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.249+0000 7fd5037fe640 1 -- 172.21.15.42:0/674447635 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd4e4003530 con 0x7fd5040a3d90 2024-09-16T03:44:35.259 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.249+0000 7fd508860640 10 monclient(hunting): handle_auth_done global_id 13091 payload 274 2024-09-16T03:44:35.259 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.249+0000 7fd508860640 1 -- 172.21.15.42:0/674447635 >> v2:172.21.15.42:3300/0 conn(0x7fd504137710 msgr2=0x7fd50413f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:44:35.259 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.249+0000 7fd508860640 1 --2- 172.21.15.42:0/674447635 >> v2:172.21.15.42:3300/0 conn(0x7fd504137710 0x7fd50413f3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:44:35.259 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.249+0000 7fd508860640 1 -- 172.21.15.42:0/674447635 >> v1:172.21.15.42:6789/0 conn(0x7fd5040a3d90 legacy=0x7fd504137000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:44:35.260 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.249+0000 7fd508860640 10 monclient: _finish_hunting 0 2024-09-16T03:44:35.260 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.249+0000 7fd508860640 1 monclient: found mon.noname-b 2024-09-16T03:44:35.260 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.249+0000 7fd508860640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:44:35.260 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.249+0000 7fd508860640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T03:44:35.260 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.249+0000 7fd508860640 1 -- 172.21.15.42:0/674447635 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5040a5e20 con 0x7fd5040a39b0 2024-09-16T03:44:35.267 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.253+0000 7fd508860640 1 --2- 172.21.15.42:0/674447635 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd5040a39b0 0x7fd504133b80 secure :-1 s=READY pgs=2271 cs=0 l=1 rev1=1 crypto rx=0x7fd4fc009d90 tx=0x7fd4fc01c830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f10b2ffee6cf2619 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:44:35.267 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.253+0000 7fd5037fe640 1 -- 172.21.15.42:0/674447635 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4fc0a6700 con 0x7fd5040a39b0 2024-09-16T03:44:35.267 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.253+0000 7fd5037fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:44:35.267 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.253+0000 7fd5037fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:44:35.267 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.253+0000 7fd5037fe640 10 monclient: dump: 2024-09-16T03:44:35.268 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:44:35.268 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:44:35.268 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:35.268 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:35.268 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:44:35.268 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:44:35.268 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:44:35.268 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:44:35.268 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:44:35.268 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:44:35.268 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.253+0000 7fd5037fe640 10 monclient: _finish_auth 0 2024-09-16T03:44:35.268 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.253+0000 7fd5037fe640 10 monclient: _check_auth_tickets 2024-09-16T03:44:35.268 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.253+0000 7fd5037fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:44:35.269 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.253+0000 7fd50aac4640 5 monclient: authenticate success, global_id 13091 2024-09-16T03:44:35.269 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.253+0000 7fd50aac4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:44:35.269 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.253+0000 7fd5037fe640 1 -- 172.21.15.42:0/674447635 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd4fc0a68a0 con 0x7fd5040a39b0 2024-09-16T03:44:35.269 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.253+0000 7fd5037fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:44:35.269 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.253+0000 7fd5037fe640 1 -- 172.21.15.42:0/674447635 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4fc0a6c50 con 0x7fd5040a39b0 2024-09-16T03:44:35.269 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.253+0000 7fd5037fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:44:35.269 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.253+0000 7fd5037fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:44:35.269 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.253+0000 7fd5037fe640 10 monclient: dump: 2024-09-16T03:44:35.269 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:44:35.269 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:44:35.269 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:35.269 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:35.269 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:44:35.269 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:44:35.270 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:44:35.270 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:44:35.270 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:44:35.270 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:44:35.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.253+0000 7fd50aac4640 10 monclient: get_monmap_and_config success 2024-09-16T03:44:35.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.253+0000 7fd50aac4640 4 set_mon_vals no callback set 2024-09-16T03:44:35.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.253+0000 7fd50aac4640 10 monclient: shutdown 2024-09-16T03:44:35.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.253+0000 7fd50aac4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:44:35.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.253+0000 7fd50aac4640 1 -- 172.21.15.42:0/674447635 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd5040a39b0 msgr2=0x7fd504133b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:44:35.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.253+0000 7fd50aac4640 1 --2- 172.21.15.42:0/674447635 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd5040a39b0 0x7fd504133b80 secure :-1 s=READY pgs=2271 cs=0 l=1 rev1=1 crypto rx=0x7fd4fc009d90 tx=0x7fd4fc01c830 comp rx=0 tx=0).stop 2024-09-16T03:44:35.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.253+0000 7fd50aac4640 1 -- 172.21.15.42:0/674447635 shutdown_connections 2024-09-16T03:44:35.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.253+0000 7fd50aac4640 1 --2- 172.21.15.42:0/674447635 >> v2:172.21.15.42:3300/0 conn(0x7fd504137710 0x7fd50413f3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:44:35.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.253+0000 7fd50aac4640 1 --2- 172.21.15.42:0/674447635 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd5040a39b0 0x7fd504133b80 unknown :-1 s=CLOSED pgs=2271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:44:35.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.253+0000 7fd50aac4640 1 -- 172.21.15.42:0/674447635 >> 172.21.15.42:0/674447635 conn(0x7fd5040a9fb0 msgr2=0x7fd5040aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:44:35.271 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.253+0000 7fd50aac4640 1 -- 172.21.15.42:0/674447635 shutdown_connections 2024-09-16T03:44:35.271 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.253+0000 7fd50aac4640 1 -- 172.21.15.42:0/674447635 wait complete. 2024-09-16T03:44:35.271 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.253+0000 7fd50aac4640 10 monclient: build_initial_monmap 2024-09-16T03:44:35.271 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.253+0000 7fd50aac4640 10 monclient: monmap: 2024-09-16T03:44:35.271 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:44:35.271 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:44:35.271 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:44:35.271 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:44:35.271 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:44:35.271 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:44:35.271 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:44:35.271 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:44:35.271 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:44:35.271 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:44:35.272 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.253+0000 7fd50aac4640 1 librados: starting msgr at 2024-09-16T03:44:35.272 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.253+0000 7fd50aac4640 1 librados: starting objecter 2024-09-16T03:44:35.272 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.253+0000 7fd50aac4640 1 Processor -- start 2024-09-16T03:44:35.272 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.253+0000 7fd50aac4640 1 -- start start 2024-09-16T03:44:35.272 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.253+0000 7fd50aac4640 1 librados: setting wanted keys 2024-09-16T03:44:35.272 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.253+0000 7fd50aac4640 1 librados: calling monclient init 2024-09-16T03:44:35.272 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.253+0000 7fd50aac4640 10 monclient: init 2024-09-16T03:44:35.272 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.253+0000 7fd50aac4640 10 monclient: _reopen_session rank -1 2024-09-16T03:44:35.272 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.253+0000 7fd50aac4640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:44:35.272 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.253+0000 7fd50aac4640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd5040a39b0 0x7fd504130560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:44:35.272 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.253+0000 7fd50aac4640 10 monclient(hunting): picked mon.noname-a con 0x7fd5040a39b0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:44:35.272 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.253+0000 7fd50aac4640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd5040a3d90 0x7fd504130aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:44:35.272 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.253+0000 7fd50aac4640 10 monclient(hunting): picked mon.noname-c con 0x7fd5040a3d90 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:44:35.272 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.253+0000 7fd50aac4640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd504137710 0x7fd5040a17f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:44:35.272 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.253+0000 7fd50aac4640 10 monclient(hunting): picked mon.noname-b con 0x7fd504137710 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:44:35.273 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.253+0000 7fd50aac4640 10 monclient(hunting): start opening mon connection 2024-09-16T03:44:35.273 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.253+0000 7fd50aac4640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fd504142310 con 0x7fd5040a39b0 2024-09-16T03:44:35.273 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.253+0000 7fd50aac4640 10 monclient(hunting): start opening mon connection 2024-09-16T03:44:35.273 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.253+0000 7fd50aac4640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd504142490 con 0x7fd504137710 2024-09-16T03:44:35.273 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.253+0000 7fd50aac4640 10 monclient(hunting): start opening mon connection 2024-09-16T03:44:35.273 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.253+0000 7fd50aac4640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fd504139ed0 con 0x7fd5040a3d90 2024-09-16T03:44:35.273 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.253+0000 7fd50aac4640 10 monclient(hunting): _renew_subs 2024-09-16T03:44:35.273 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.253+0000 7fd50aac4640 10 monclient(hunting): authenticate will time out at 1122.226816s 2024-09-16T03:44:35.273 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.253+0000 7fd508860640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd5040a39b0 0x7fd504130560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:44:35.273 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.253+0000 7fd503fff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd5040a3d90 0x7fd504130aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:44:35.273 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.253+0000 7fd508860640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd5040a39b0 0x7fd504130560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:48302/0 (socket says 172.21.15.42:48302) 2024-09-16T03:44:35.273 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.253+0000 7fd508860640 1 -- 172.21.15.42:0/3704374897 learned_addr learned my addr 172.21.15.42:0/3704374897 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:44:35.273 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.253+0000 7fd508860640 10 monclient(hunting): get_auth_request con 0x7fd5040a39b0 auth_method 0 2024-09-16T03:44:35.273 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.253+0000 7fd508860640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:44:35.274 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.253+0000 7fd508860640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:44:35.274 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.253+0000 7fd508860640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:44:35.274 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.253+0000 7fd503fff640 10 monclient(hunting): get_auth_request con 0x7fd5040a3d90 auth_method 0 2024-09-16T03:44:35.274 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.253+0000 7fd503fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:44:35.274 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.253+0000 7fd503fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:44:35.274 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.253+0000 7fd503fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:44:35.274 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.253+0000 7fd503fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:44:35.274 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.253+0000 7fd503fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:44:35.274 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.253+0000 7fd503fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:44:35.274 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.253+0000 7fd508860640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:44:35.274 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.253+0000 7fd508860640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:44:35.274 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.253+0000 7fd508860640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:44:35.274 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.253+0000 7fd509061640 1 --2- 172.21.15.42:0/3704374897 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd504137710 0x7fd5040a17f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:44:35.274 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.253+0000 7fd503fff640 10 monclient(hunting): handle_auth_done global_id 13094 payload 824 2024-09-16T03:44:35.275 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.253+0000 7fd503fff640 1 -- 172.21.15.42:0/3704374897 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd504137710 msgr2=0x7fd5040a17f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:44:35.275 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.253+0000 7fd503fff640 1 --2- 172.21.15.42:0/3704374897 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd504137710 0x7fd5040a17f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:44:35.275 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.253+0000 7fd503fff640 1 -- 172.21.15.42:0/3704374897 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd5040a39b0 msgr2=0x7fd504130560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:44:35.275 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.253+0000 7fd503fff640 1 --2- 172.21.15.42:0/3704374897 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd5040a39b0 0x7fd504130560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:44:35.275 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.253+0000 7fd503fff640 10 monclient: _finish_hunting 0 2024-09-16T03:44:35.275 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.253+0000 7fd503fff640 1 monclient: found mon.noname-c 2024-09-16T03:44:35.275 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.253+0000 7fd503fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:44:35.275 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.253+0000 7fd503fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:44:35.275 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.253+0000 7fd503fff640 1 -- 172.21.15.42:0/3704374897 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd504139910 con 0x7fd5040a3d90 2024-09-16T03:44:35.275 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.253+0000 7fd509061640 10 monclient: get_auth_request con 0x7fd504137710 auth_method 0 2024-09-16T03:44:35.275 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.253+0000 7fd509061640 1 --2- 172.21.15.42:0/3704374897 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd504137710 0x7fd5040a17f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T03:44:35.275 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.253+0000 7fd503fff640 1 --2- 172.21.15.42:0/3704374897 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd5040a3d90 0x7fd504130aa0 secure :-1 s=READY pgs=2272 cs=0 l=1 rev1=1 crypto rx=0x7fd4ec007fd0 tx=0x7fd4ec00c890 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:44:35.275 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.253+0000 7fd5017fa640 10 client.?.objecter ms_handle_connect 0x7fd5040a3d90 2024-09-16T03:44:35.275 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.253+0000 7fd5017fa640 10 client.?.objecter resend_mon_ops 2024-09-16T03:44:35.275 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.253+0000 7fd5017fa640 1 -- 172.21.15.42:0/3704374897 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4ec017070 con 0x7fd5040a3d90 2024-09-16T03:44:35.276 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.253+0000 7fd5017fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:44:35.276 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.253+0000 7fd5017fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:44:35.276 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.253+0000 7fd5017fa640 10 monclient: dump: 2024-09-16T03:44:35.276 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:44:35.276 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:44:35.276 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:35.276 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:35.276 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:44:35.276 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:44:35.276 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:44:35.276 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:44:35.276 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:44:35.276 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:44:35.276 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.253+0000 7fd5017fa640 10 monclient: _finish_auth 0 2024-09-16T03:44:35.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.253+0000 7fd5017fa640 10 monclient: _check_auth_tickets 2024-09-16T03:44:35.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.253+0000 7fd5017fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:44:35.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.253+0000 7fd50aac4640 5 monclient: authenticate success, global_id 13094 2024-09-16T03:44:35.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.253+0000 7fd50aac4640 10 monclient: _renew_subs 2024-09-16T03:44:35.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.253+0000 7fd5017fa640 1 -- 172.21.15.42:0/3704374897 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd4ec005cf0 con 0x7fd5040a3d90 2024-09-16T03:44:35.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.253+0000 7fd50aac4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:44:35.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.253+0000 7fd50aac4640 1 -- 172.21.15.42:0/3704374897 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd504137100 con 0x7fd5040a3d90 2024-09-16T03:44:35.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.253+0000 7fd5017fa640 10 monclient: handle_config config(0 keys) 2024-09-16T03:44:35.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.253+0000 7fd5017fa640 1 -- 172.21.15.42:0/3704374897 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4ec0072f0 con 0x7fd5040a3d90 2024-09-16T03:44:35.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.253+0000 7fd5017fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:44:35.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.253+0000 7fd5017fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:44:35.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.253+0000 7fd5017fa640 10 monclient: dump: 2024-09-16T03:44:35.277 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:44:35.277 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:44:35.278 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:35.278 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:35.278 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:44:35.278 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:44:35.278 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:44:35.278 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:44:35.278 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:44:35.278 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:44:35.278 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.253+0000 7fd502ffd640 4 set_mon_vals no callback set 2024-09-16T03:44:35.278 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.253+0000 7fd50aac4640 10 client.13094.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:44:35.278 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.253+0000 7fd50aac4640 10 monclient: _renew_subs 2024-09-16T03:44:35.278 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.253+0000 7fd50aac4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:44:35.278 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.253+0000 7fd50aac4640 1 -- 172.21.15.42:0/3704374897 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd504139bf0 con 0x7fd5040a3d90 2024-09-16T03:44:35.278 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.253+0000 7fd50aac4640 1 librados: init done 2024-09-16T03:44:35.278 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.257+0000 7fd50aac4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:44:35.279 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.257+0000 7fd50aac4640 1 --2- 172.21.15.42:0/3704374897 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd5040a1d30 0x7fd5040a2110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:44:35.279 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.257+0000 7fd50aac4640 10 monclient(hunting): start opening mon connection 2024-09-16T03:44:35.279 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.257+0000 7fd50aac4640 1 -- 172.21.15.42:0/3704374897 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fd5041342d0 con 0x7fd5040a1d30 2024-09-16T03:44:35.279 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.257+0000 7fd5017fa640 1 -- 172.21.15.42:0/3704374897 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fd4ec01a070 con 0x7fd5040a3d90 2024-09-16T03:44:35.279 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.257+0000 7fd508860640 1 --2- 172.21.15.42:0/3704374897 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd5040a1d30 0x7fd5040a2110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:44:35.279 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.257+0000 7fd508860640 10 monclient: get_auth_request con 0x7fd5040a1d30 auth_method 0 2024-09-16T03:44:35.279 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.257+0000 7fd508860640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:44:35.279 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.257+0000 7fd508860640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:44:35.279 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.257+0000 7fd508860640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:44:35.279 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.257+0000 7fd508860640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:44:35.279 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.257+0000 7fd508860640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:44:35.279 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.257+0000 7fd508860640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:44:35.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.257+0000 7fd5017fa640 1 --2- 172.21.15.42:0/3704374897 >> v2:172.21.15.154:6812/254845342 conn(0x7fd4e8037810 0x7fd4e8039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:44:35.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.257+0000 7fd5017fa640 1 -- 172.21.15.42:0/3704374897 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(56..56 src has 1..56) ==== 9613+0+0 (secure 0 0 0) 0x7fd4ec0492a0 con 0x7fd5040a3d90 2024-09-16T03:44:35.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.257+0000 7fd5017fa640 10 client.13094.objecter ms_dispatch 0x7fd504000ba0 osd_map(56..56 src has 1..56) 2024-09-16T03:44:35.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.257+0000 7fd5017fa640 3 client.13094.objecter handle_osd_map got epochs [56,56] > 0 2024-09-16T03:44:35.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.257+0000 7fd5017fa640 3 client.13094.objecter handle_osd_map decoding full epoch 56 2024-09-16T03:44:35.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.257+0000 7fd508860640 10 monclient(hunting): handle_auth_done global_id 13097 payload 824 2024-09-16T03:44:35.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.257+0000 7fd508860640 1 -- 172.21.15.42:0/3704374897 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd504139750 con 0x7fd5040a1d30 2024-09-16T03:44:35.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.257+0000 7fd5017fa640 20 client.13094.objecter dump_active .. 0 homeless 2024-09-16T03:44:35.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.257+0000 7fd509061640 1 --2- 172.21.15.42:0/3704374897 >> v2:172.21.15.154:6812/254845342 conn(0x7fd4e8037810 0x7fd4e8039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:44:35.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.257+0000 7fd508860640 1 --2- 172.21.15.42:0/3704374897 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd5040a1d30 0x7fd5040a2110 secure :-1 s=READY pgs=2273 cs=0 l=1 rev1=1 crypto rx=0x7fd4fc01aee0 tx=0x7fd4fc0195d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:44:35.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.257+0000 7fd5017fa640 10 client.13094.objecter ms_handle_connect 0x7fd5040a1d30 2024-09-16T03:44:35.281 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.257+0000 7fd5017fa640 10 client.13094.objecter resend_mon_ops 2024-09-16T03:44:35.281 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.257+0000 7fd509061640 10 monclient: get_auth_request con 0x7fd4e8037810 auth_method 0 2024-09-16T03:44:35.281 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.257+0000 7fd5017fa640 1 -- 172.21.15.42:0/3704374897 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4fc0be070 con 0x7fd5040a1d30 2024-09-16T03:44:35.281 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.257+0000 7fd5017fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:44:35.281 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.257+0000 7fd5017fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:44:35.281 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.257+0000 7fd5017fa640 10 monclient: dump: 2024-09-16T03:44:35.281 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:44:35.281 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:44:35.281 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:35.281 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:35.281 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:44:35.281 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:44:35.281 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:44:35.281 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:44:35.282 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:44:35.282 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:44:35.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.257+0000 7fd5017fa640 1 -- 172.21.15.42:0/3704374897 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fd4fc004580 con 0x7fd5040a1d30 2024-09-16T03:44:35.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.257+0000 7fd5017fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:44:35.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.257+0000 7fd5017fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:44:35.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.257+0000 7fd5017fa640 1 -- 172.21.15.42:0/3704374897 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd5040a1d30 msgr2=0x7fd5040a2110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:44:35.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.257+0000 7fd5017fa640 1 --2- 172.21.15.42:0/3704374897 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd5040a1d30 0x7fd5040a2110 secure :-1 s=READY pgs=2273 cs=0 l=1 rev1=1 crypto rx=0x7fd4fc01aee0 tx=0x7fd4fc0195d0 comp rx=0 tx=0).stop 2024-09-16T03:44:35.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.257+0000 7fd5017fa640 1 -- 172.21.15.42:0/3704374897 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd5040a1d30 msgr2=0x7fd5040a2110 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:44:35.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.257+0000 7fd5017fa640 1 --2- 172.21.15.42:0/3704374897 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd5040a1d30 0x7fd5040a2110 secure :-1 s=CLOSED pgs=2273 cs=0 l=1 rev1=1 crypto rx=0x7fd4fc01aee0 tx=0x7fd4fc0195d0 comp rx=0 tx=0).stop 2024-09-16T03:44:35.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.257+0000 7fd5017fa640 10 client.13094.objecter ms_handle_connect 0x7fd4e8037810 2024-09-16T03:44:35.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.257+0000 7fd509061640 1 --2- 172.21.15.42:0/3704374897 >> v2:172.21.15.154:6812/254845342 conn(0x7fd4e8037810 0x7fd4e8039cd0 secure :-1 s=READY pgs=3304 cs=0 l=1 rev1=1 crypto rx=0x7fd5041394d0 tx=0x7fd4f4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:44:35.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.265+0000 7fd50aac4640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:44:35.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.265+0000 7fd50aac4640 1 --2- 172.21.15.42:0/3704374897 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd5040a2650 0x7fd50409c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:44:35.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.265+0000 7fd50aac4640 10 monclient(hunting): start opening mon connection 2024-09-16T03:44:35.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.265+0000 7fd50aac4640 1 -- 172.21.15.42:0/3704374897 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fd5040a3450 con 0x7fd5040a2650 2024-09-16T03:44:35.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.265+0000 7fd508860640 1 --2- 172.21.15.42:0/3704374897 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd5040a2650 0x7fd50409c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:44:35.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.265+0000 7fd508860640 10 monclient: get_auth_request con 0x7fd5040a2650 auth_method 0 2024-09-16T03:44:35.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.265+0000 7fd508860640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:44:35.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.265+0000 7fd508860640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:44:35.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.265+0000 7fd508860640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:44:35.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.265+0000 7fd508860640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:44:35.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.265+0000 7fd508860640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:44:35.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.265+0000 7fd508860640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:44:35.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.265+0000 7fd508860640 10 monclient(hunting): handle_auth_done global_id 13100 payload 824 2024-09-16T03:44:35.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.265+0000 7fd508860640 1 -- 172.21.15.42:0/3704374897 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fd504133b80 con 0x7fd5040a2650 2024-09-16T03:44:35.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.265+0000 7fd508860640 1 --2- 172.21.15.42:0/3704374897 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd5040a2650 0x7fd50409c920 secure :-1 s=READY pgs=2274 cs=0 l=1 rev1=1 crypto rx=0x7fd4fc01d0d0 tx=0x7fd4fc0a9000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:44:35.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.265+0000 7fd5017fa640 10 client.13094.objecter ms_handle_connect 0x7fd5040a2650 2024-09-16T03:44:35.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.265+0000 7fd5017fa640 10 client.13094.objecter resend_mon_ops 2024-09-16T03:44:35.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.265+0000 7fd5017fa640 1 -- 172.21.15.42:0/3704374897 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4fc0a6700 con 0x7fd5040a2650 2024-09-16T03:44:35.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.265+0000 7fd5017fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:44:35.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.265+0000 7fd5017fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:44:35.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.265+0000 7fd5017fa640 10 monclient: dump: 2024-09-16T03:44:35.286 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:44:35.286 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:44:35.286 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:35.286 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:35.286 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:44:35.287 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:44:35.287 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:44:35.287 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:44:35.287 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:44:35.287 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:44:35.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.265+0000 7fd5017fa640 1 -- 172.21.15.42:0/3704374897 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4879 (secure 0 0 0) 0x7fd4fc0a6700 con 0x7fd5040a2650 2024-09-16T03:44:35.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.265+0000 7fd5017fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:44:35.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.265+0000 7fd5017fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:44:35.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.265+0000 7fd5017fa640 1 -- 172.21.15.42:0/3704374897 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd5040a2650 msgr2=0x7fd50409c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:44:35.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.265+0000 7fd5017fa640 1 --2- 172.21.15.42:0/3704374897 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd5040a2650 0x7fd50409c920 secure :-1 s=READY pgs=2274 cs=0 l=1 rev1=1 crypto rx=0x7fd4fc01d0d0 tx=0x7fd4fc0a9000 comp rx=0 tx=0).stop 2024-09-16T03:44:35.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.265+0000 7fd5017fa640 1 -- 172.21.15.42:0/3704374897 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd5040a2650 msgr2=0x7fd50409c920 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:44:35.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.265+0000 7fd5017fa640 1 --2- 172.21.15.42:0/3704374897 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd5040a2650 0x7fd50409c920 secure :-1 s=CLOSED pgs=2274 cs=0 l=1 rev1=1 crypto rx=0x7fd4fc01d0d0 tx=0x7fd4fc0a9000 comp rx=0 tx=0).stop 2024-09-16T03:44:35.288 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:44:35.288 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:44:35.288 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:44:35.288 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T03:44:35.288 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 22, 2024-09-16T03:44:35.288 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 154635, 2024-09-16T03:44:35.288 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:44:35.288 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:44:35.288 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T03:44:35.288 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T03:44:35.288 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:44:35.289 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 3, 2024-09-16T03:44:35.289 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:44:35.289 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:44:35.289 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:44:35.289 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:44:35.289 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:44:35.289 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:44:35.289 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:44:35.289 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:44:35.289 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:44:35.289 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:44:35.289 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:44:35.289 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:44:35.289 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:44:35.290 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:44:35.290 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:44:35.290 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:44:35.290 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:44:35.290 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:44:35.290 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:44:35.290 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:44:35.290 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:44:35.290 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:44:35.290 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:44:35.290 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:44:35.290 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:44:35.290 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:44:35.290 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:44:35.291 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:44:35.291 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:44:35.291 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:44:35.291 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:44:35.291 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:44:35.291 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:44:35.291 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:44:35.291 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:44:35.291 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:44:35.291 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:44:35.291 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:44:35.291 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:44:35.291 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:44:35.291 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:44:35.292 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:44:35.292 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:44:35.292 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:44:35.292 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:44:35.292 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:44:35.292 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:44:35.292 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:44:35.292 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:44:35.292 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:44:35.292 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:44:35.292 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:44:35.292 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:44:35.292 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:44:35.292 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:44:35.293 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:44:35.293 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:44:35.293 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:44:35.293 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:44:35.293 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:44:35.293 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:44:35.293 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:44:35.293 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:44:35.293 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:44:35.293 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:44:35.293 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:44:35.293 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:44:35.293 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:44:35.294 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:44:35.294 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:44:35.294 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:44:35.294 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:44:35.294 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:44:35.294 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:44:35.294 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:44:35.294 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:44:35.294 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:44:35.294 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:44:35.294 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:44:35.295 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:44:35.295 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:44:35.295 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:44:35.295 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:44:35.295 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:44:35.295 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:44:35.295 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:44:35.295 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:44:35.295 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:44:35.295 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:44:35.295 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:44:35.295 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:44:35.296 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:44:35.296 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:44:35.296 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:44:35.296 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:44:35.296 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:44:35.296 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:44:35.296 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:44:35.296 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:44:35.296 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:44:35.296 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:44:35.296 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:44:35.296 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:44:35.296 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:44:35.296 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:44:35.297 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:44:35.297 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:44:35.297 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:44:35.297 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:44:35.297 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:44:35.297 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:44:35.297 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:44:35.297 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:44:35.297 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:44:35.297 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:44:35.297 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:44:35.297 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:44:35.297 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:44:35.297 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:44:35.298 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:44:35.298 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:44:35.298 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:44:35.298 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:44:35.298 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:44:35.298 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:44:35.298 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:44:35.298 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:44:35.298 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:44:35.298 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:44:35.298 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:44:35.298 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:44:35.298 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:44:35.298 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:44:35.299 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:44:35.299 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:44:35.299 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:44:35.299 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:44:35.299 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:44:35.299 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:44:35.299 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:44:35.299 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:44:35.299 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T03:44:35.299 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:44:35.299 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:44:35.299 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:44:35.299 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 2 2024-09-16T03:44:35.299 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:44:35.299 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:44:35.300 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:44:35.300 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:44:35.300 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:44:35.300 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:44:35.300 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 9 2024-09-16T03:44:35.300 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:44:35.300 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:44:35.300 INFO:teuthology.orchestra.run.smithi042.stdout: "mgr": [ 2024-09-16T03:44:35.300 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:44:35.300 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:44:35.300 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:44:35.300 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:44:35.300 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:44:35.300 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:44:35.301 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:44:35.301 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:44:35.301 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:44:35.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.265+0000 7fd50aac4640 10 librados: watch_flush enter 2024-09-16T03:44:35.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.265+0000 7fd50aac4640 10 librados: watch_flush exit 2024-09-16T03:44:35.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.265+0000 7fd50aac4640 20 client.13094.objecter shutdown clearing up homeless session... 2024-09-16T03:44:35.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.265+0000 7fd50aac4640 10 client.13094.objecter successfully canceled tick 2024-09-16T03:44:35.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.265+0000 7fd50aac4640 1 -- 172.21.15.42:0/3704374897 >> v2:172.21.15.154:6812/254845342 conn(0x7fd4e8037810 msgr2=0x7fd4e8039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:44:35.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.265+0000 7fd50aac4640 1 --2- 172.21.15.42:0/3704374897 >> v2:172.21.15.154:6812/254845342 conn(0x7fd4e8037810 0x7fd4e8039cd0 secure :-1 s=READY pgs=3304 cs=0 l=1 rev1=1 crypto rx=0x7fd5041394d0 tx=0x7fd4f4008040 comp rx=0 tx=0).stop 2024-09-16T03:44:35.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.265+0000 7fd50aac4640 10 monclient: shutdown 2024-09-16T03:44:35.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.265+0000 7fd50aac4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:44:35.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.265+0000 7fd50aac4640 1 -- 172.21.15.42:0/3704374897 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd5040a3d90 msgr2=0x7fd504130aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:44:35.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.265+0000 7fd50aac4640 1 --2- 172.21.15.42:0/3704374897 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd5040a3d90 0x7fd504130aa0 secure :-1 s=READY pgs=2272 cs=0 l=1 rev1=1 crypto rx=0x7fd4ec007fd0 tx=0x7fd4ec00c890 comp rx=0 tx=0).stop 2024-09-16T03:44:35.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.265+0000 7fd509061640 1 -- 172.21.15.42:0/3704374897 reap_dead start 2024-09-16T03:44:35.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.265+0000 7fd50aac4640 1 -- 172.21.15.42:0/3704374897 shutdown_connections 2024-09-16T03:44:35.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.265+0000 7fd50aac4640 1 --2- 172.21.15.42:0/3704374897 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd5040a3d90 0x7fd504130aa0 unknown :-1 s=CLOSED pgs=2272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:44:35.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.265+0000 7fd50aac4640 1 -- 172.21.15.42:0/3704374897 >> 172.21.15.42:0/3704374897 conn(0x7fd5040a9fb0 msgr2=0x7fd5040a7a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:44:35.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.265+0000 7fd50aac4640 1 -- 172.21.15.42:0/3704374897 shutdown_connections 2024-09-16T03:44:35.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.265+0000 7fd50aac4640 1 -- 172.21.15.42:0/3704374897 wait complete. 2024-09-16T03:44:35.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.265+0000 7fd50aac4640 1 librados: shutdown 2024-09-16T03:44:35.304 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:44:35.304 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T03:44:35.304 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T03:44:35.343 INFO:tasks.workunit.client.0.smithi154.stdout: api_misc: [==========] Running 12 tests from 4 test suites. 2024-09-16T03:44:35.343 INFO:tasks.workunit.client.0.smithi154.stdout: api_misc: [----------] Global test environment set-up. 2024-09-16T03:44:35.343 INFO:tasks.workunit.client.0.smithi154.stdout: api_misc: [----------] 1 test from LibRadosMiscVersion 2024-09-16T03:44:35.343 INFO:tasks.workunit.client.0.smithi154.stdout: api_misc: [ RUN ] LibRadosMiscVersion.Version 2024-09-16T03:44:35.344 INFO:tasks.workunit.client.0.smithi154.stdout: api_misc: [ OK ] LibRadosMiscVersion.Version (0 ms) 2024-09-16T03:44:35.344 INFO:tasks.workunit.client.0.smithi154.stdout: api_misc: [----------] 1 test from LibRadosMiscVersion (0 ms total) 2024-09-16T03:44:35.344 INFO:tasks.workunit.client.0.smithi154.stdout: api_misc: 2024-09-16T03:44:35.344 INFO:tasks.workunit.client.0.smithi154.stdout: api_misc: [----------] 2 tests from LibRadosMiscConnectFailure 2024-09-16T03:44:35.344 INFO:tasks.workunit.client.0.smithi154.stdout: api_misc: [ RUN ] LibRadosMiscConnectFailure.ConnectFailure 2024-09-16T03:44:35.344 INFO:tasks.workunit.client.0.smithi154.stdout: api_misc: unable to get monitor info from DNS SRV with service name: 2024-09-16T03:43:31.231+0000 7ff95397c880 -1 failed for service _ceph-mon._tcp 2024-09-16T03:44:35.344 INFO:tasks.workunit.client.0.smithi154.stdout: api_misc: ceph-mon 2024-09-16T03:44:35.344 INFO:tasks.workunit.client.0.smithi154.stdout: api_misc: 2024-09-16T03:43:31.231+0000 7ff95397c880 -1 monclient: get_monmap_and_config cannot identify monitors to contact 2024-09-16T03:44:35.344 INFO:tasks.workunit.client.0.smithi154.stdout: api_misc: [ OK ] LibRadosMiscConnectFailure.ConnectFailure (1608 ms) 2024-09-16T03:44:35.344 INFO:tasks.workunit.client.0.smithi154.stdout: api_misc: [ RUN ] LibRadosMiscConnectFailure.ConnectTimeout 2024-09-16T03:44:35.344 INFO:tasks.workunit.client.0.smithi154.stdout: api_misc: [ OK ] LibRadosMiscConnectFailure.ConnectTimeout (5006 ms) 2024-09-16T03:44:35.344 INFO:tasks.workunit.client.0.smithi154.stdout: api_misc: [----------] 2 tests from LibRadosMiscConnectFailure (6614 ms total) 2024-09-16T03:44:35.345 INFO:tasks.workunit.client.0.smithi154.stdout: api_misc: 2024-09-16T03:44:35.345 INFO:tasks.workunit.client.0.smithi154.stdout: api_misc: [----------] 1 test from LibRadosMiscPool 2024-09-16T03:44:35.345 INFO:tasks.workunit.client.0.smithi154.stdout: api_misc: [ RUN ] LibRadosMiscPool.PoolCreationRace 2024-09-16T03:44:35.345 INFO:tasks.workunit.client.0.smithi154.stdout: api_misc: started 0x55e7dc8ce4f0 2024-09-16T03:44:35.345 INFO:tasks.workunit.client.0.smithi154.stdout: api_misc: started 0x55e7dd5126a0 2024-09-16T03:44:35.345 INFO:tasks.workunit.client.0.smithi154.stdout: api_misc: started 2 aios 2024-09-16T03:44:35.345 INFO:tasks.workunit.client.0.smithi154.stdout: api_misc: waiting 0x55e7dc8ce4f0 2024-09-16T03:44:35.345 INFO:tasks.workunit.client.0.smithi154.stdout: api_misc: waiting 0x55e7dd5126a0 2024-09-16T03:44:35.345 INFO:tasks.workunit.client.0.smithi154.stdout: api_misc: done. 2024-09-16T03:44:35.345 INFO:tasks.workunit.client.0.smithi154.stdout: api_misc: [ OK ] LibRadosMiscPool.PoolCreationRace (24998 ms) 2024-09-16T03:44:35.345 INFO:tasks.workunit.client.0.smithi154.stdout: api_misc: [----------] 1 test from LibRadosMiscPool (24999 ms total) 2024-09-16T03:44:35.345 INFO:tasks.workunit.client.0.smithi154.stdout: api_misc: 2024-09-16T03:44:35.345 INFO:tasks.workunit.client.0.smithi154.stdout: api_misc: [----------] 8 tests from LibRadosMisc 2024-09-16T03:44:35.345 INFO:tasks.workunit.client.0.smithi154.stdout: api_misc: [ RUN ] LibRadosMisc.ClusterFSID 2024-09-16T03:44:35.346 INFO:tasks.workunit.client.0.smithi154.stdout: api_misc: [ OK ] LibRadosMisc.ClusterFSID (0 ms) 2024-09-16T03:44:35.346 INFO:tasks.workunit.client.0.smithi154.stdout: api_misc: [ RUN ] LibRadosMisc.Exec 2024-09-16T03:44:35.346 INFO:tasks.workunit.client.0.smithi154.stdout: api_misc: [ OK ] LibRadosMisc.Exec (171 ms) 2024-09-16T03:44:35.346 INFO:tasks.workunit.client.0.smithi154.stdout: api_misc: [ RUN ] LibRadosMisc.WriteSame 2024-09-16T03:44:35.346 INFO:tasks.workunit.client.0.smithi154.stdout: api_misc: [ OK ] LibRadosMisc.WriteSame (22 ms) 2024-09-16T03:44:35.346 INFO:tasks.workunit.client.0.smithi154.stdout: api_misc: [ RUN ] LibRadosMisc.CmpExt 2024-09-16T03:44:35.346 INFO:tasks.workunit.client.0.smithi154.stdout: api_misc: [ OK ] LibRadosMisc.CmpExt (3 ms) 2024-09-16T03:44:35.346 INFO:tasks.workunit.client.0.smithi154.stdout: api_misc: [ RUN ] LibRadosMisc.Applications 2024-09-16T03:44:35.347 INFO:tasks.workunit.client.0.smithi154.stdout: api_misc: [ OK ] LibRadosMisc.Applications (4805 ms) 2024-09-16T03:44:35.347 INFO:tasks.workunit.client.0.smithi154.stdout: api_misc: [ RUN ] LibRadosMisc.MinCompatOSD 2024-09-16T03:44:35.347 INFO:tasks.workunit.client.0.smithi154.stdout: api_misc: [ OK ] LibRadosMisc.MinCompatOSD (0 ms) 2024-09-16T03:44:35.347 INFO:tasks.workunit.client.0.smithi154.stdout: api_misc: [ RUN ] LibRadosMisc.MinCompatClient 2024-09-16T03:44:35.347 INFO:tasks.workunit.client.0.smithi154.stdout: api_misc: [ OK ] LibRadosMisc.MinCompatClient (0 ms) 2024-09-16T03:44:35.347 INFO:tasks.workunit.client.0.smithi154.stdout: api_misc: [ RUN ] LibRadosMisc.ShutdownRace 2024-09-16T03:44:35.347 INFO:tasks.workunit.client.0.smithi154.stdout: api_misc: [ OK ] LibRadosMisc.ShutdownRace (26929 ms) 2024-09-16T03:44:35.347 INFO:tasks.workunit.client.0.smithi154.stdout: api_misc: [----------] 8 tests from LibRadosMisc (31930 ms total) 2024-09-16T03:44:35.347 INFO:tasks.workunit.client.0.smithi154.stdout: api_misc: 2024-09-16T03:44:35.348 INFO:tasks.workunit.client.0.smithi154.stdout: api_misc: [----------] Global test environment tear-down 2024-09-16T03:44:35.348 INFO:tasks.workunit.client.0.smithi154.stdout: api_misc: [==========] 12 tests from 4 test suites ran. (65677 ms total) 2024-09-16T03:44:35.348 INFO:tasks.workunit.client.0.smithi154.stdout: api_misc: [ PASSED ] 12 tests. 2024-09-16T03:44:35.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.453+0000 7fbb5b7c8640 10 monclient: get_monmap_and_config 2024-09-16T03:44:35.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.453+0000 7fbb5b7c8640 10 monclient: build_initial_monmap 2024-09-16T03:44:35.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.453+0000 7fbb5b7c8640 10 monclient: monmap: 2024-09-16T03:44:35.457 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:44:35.458 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:44:35.458 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:44:35.456340+0000 2024-09-16T03:44:35.458 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:44:35.456340+0000 2024-09-16T03:44:35.458 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:44:35.458 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:44:35.458 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:44:35.458 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:44:35.458 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:44:35.459 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:44:35.459 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:44:35.459 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:44:35.461 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.453+0000 7fbb5b7c8640 1 Processor -- start 2024-09-16T03:44:35.461 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.453+0000 7fbb5b7c8640 1 -- start start 2024-09-16T03:44:35.461 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.453+0000 7fbb5b7c8640 10 monclient: init 2024-09-16T03:44:35.461 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.453+0000 7fbb5b7c8640 10 monclient: _reopen_session rank -1 2024-09-16T03:44:35.461 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.453+0000 7fbb5b7c8640 10 monclient: _add_conns ranks=[2,4,0,1,3] 2024-09-16T03:44:35.461 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.453+0000 7fbb5b7c8640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbb540a39b0 0x7fbb54133b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:44:35.462 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.453+0000 7fbb5b7c8640 10 monclient(hunting): picked mon.noname-b con 0x7fbb540a39b0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:44:35.462 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.453+0000 7fbb5b7c8640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fbb540a3d90 0x7fbb54137000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:44:35.462 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.453+0000 7fbb5b7c8640 10 monclient(hunting): picked mon.noname-c con 0x7fbb540a3d90 addr v2:172.21.15.154:3300/0 2024-09-16T03:44:35.462 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.453+0000 7fbb5b7c8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbb54137540 addr v1:172.21.15.42:6789/0 2024-09-16T03:44:35.462 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.453+0000 7fbb5b7c8640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbb5409c140 con 0x7fbb54137540 2024-09-16T03:44:35.462 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.453+0000 7fbb5b7c8640 10 monclient(hunting): start opening mon connection 2024-09-16T03:44:35.462 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.453+0000 7fbb5b7c8640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fbb5409c420 con 0x7fbb540a3d90 2024-09-16T03:44:35.462 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.453+0000 7fbb5b7c8640 10 monclient(hunting): start opening mon connection 2024-09-16T03:44:35.462 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.453+0000 7fbb5b7c8640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fbb5406dd60 con 0x7fbb540a39b0 2024-09-16T03:44:35.462 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.453+0000 7fbb5b7c8640 10 monclient(hunting): _renew_subs 2024-09-16T03:44:35.462 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.453+0000 7fbb5b7c8640 10 monclient(hunting): authenticate will time out at 1122.425567s 2024-09-16T03:44:35.462 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.453+0000 7fbb59d65640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fbb54137540 0x7fbb5413e390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:47918/0 (socket says 172.21.15.42:47918) 2024-09-16T03:44:35.462 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.453+0000 7fbb59d65640 1 -- 172.21.15.42:0/909896527 learned_addr learned my addr 172.21.15.42:0/909896527 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:44:35.462 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.453+0000 7fbb59564640 1 --2- 172.21.15.42:0/909896527 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbb540a39b0 0x7fbb54133b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:44:35.463 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.453+0000 7fbb59564640 10 monclient(hunting): get_auth_request con 0x7fbb540a39b0 auth_method 0 2024-09-16T03:44:35.463 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.453+0000 7fbb59564640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:44:35.463 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.453+0000 7fbb59564640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:44:35.463 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.453+0000 7fbb59564640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:44:35.463 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.453+0000 7fbb59564640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:44:35.463 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.453+0000 7fbb59564640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:44:35.463 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.453+0000 7fbb59564640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:44:35.463 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.453+0000 7fbb4bfff640 1 -- 172.21.15.42:0/909896527 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2262355810 0 0) 0x7fbb5409c140 con 0x7fbb54137540 2024-09-16T03:44:35.463 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.453+0000 7fbb4bfff640 10 monclient(hunting): _negotiate 2024-09-16T03:44:35.463 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.453+0000 7fbb4bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:44:35.463 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.453+0000 7fbb4bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:44:35.463 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.453+0000 7fbb4bfff640 10 monclient(hunting): my global_id is 27534 2024-09-16T03:44:35.463 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.453+0000 7fbb4bfff640 1 -- 172.21.15.42:0/909896527 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbb40003530 con 0x7fbb54137540 2024-09-16T03:44:35.463 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.453+0000 7fbb58d63640 1 --2- 172.21.15.42:0/909896527 >> v2:172.21.15.154:3300/0 conn(0x7fbb540a3d90 0x7fbb54137000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:44:35.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.453+0000 7fbb59564640 10 monclient(hunting): handle_auth_done global_id 13103 payload 274 2024-09-16T03:44:35.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.453+0000 7fbb59564640 1 -- 172.21.15.42:0/909896527 >> v2:172.21.15.154:3300/0 conn(0x7fbb540a3d90 msgr2=0x7fbb54137000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:44:35.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.453+0000 7fbb59564640 1 --2- 172.21.15.42:0/909896527 >> v2:172.21.15.154:3300/0 conn(0x7fbb540a3d90 0x7fbb54137000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:44:35.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.453+0000 7fbb59564640 1 -- 172.21.15.42:0/909896527 >> v1:172.21.15.42:6789/0 conn(0x7fbb54137540 legacy=0x7fbb5413e390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:44:35.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.453+0000 7fbb59564640 10 monclient: _finish_hunting 0 2024-09-16T03:44:35.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.453+0000 7fbb59564640 1 monclient: found mon.noname-b 2024-09-16T03:44:35.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.453+0000 7fbb59564640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:44:35.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.453+0000 7fbb59564640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T03:44:35.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.453+0000 7fbb59564640 1 -- 172.21.15.42:0/909896527 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb540a5e20 con 0x7fbb540a39b0 2024-09-16T03:44:35.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.453+0000 7fbb58d63640 10 monclient: get_auth_request con 0x7fbb540a3d90 auth_method 0 2024-09-16T03:44:35.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.453+0000 7fbb58d63640 1 --2- 172.21.15.42:0/909896527 >> v2:172.21.15.154:3300/0 conn(0x7fbb540a3d90 0x7fbb54137000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T03:44:35.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.457+0000 7fbb59564640 1 --2- 172.21.15.42:0/909896527 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbb540a39b0 0x7fbb54133b80 secure :-1 s=READY pgs=2275 cs=0 l=1 rev1=1 crypto rx=0x7fbb500097c0 tx=0x7fbb5001c360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4e0adb141d3fe66e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:44:35.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.457+0000 7fbb4bfff640 1 -- 172.21.15.42:0/909896527 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb500b0070 con 0x7fbb540a39b0 2024-09-16T03:44:35.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.457+0000 7fbb4bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:44:35.465 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.457+0000 7fbb4bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:44:35.465 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.457+0000 7fbb4bfff640 10 monclient: dump: 2024-09-16T03:44:35.465 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:44:35.465 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:44:35.465 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:35.465 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:35.465 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:44:35.465 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:44:35.465 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:44:35.465 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:44:35.465 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:44:35.465 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:44:35.465 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.457+0000 7fbb4bfff640 10 monclient: _finish_auth 0 2024-09-16T03:44:35.465 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.457+0000 7fbb4bfff640 10 monclient: _check_auth_tickets 2024-09-16T03:44:35.466 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.457+0000 7fbb4bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:44:35.466 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.457+0000 7fbb4bfff640 1 -- 172.21.15.42:0/909896527 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb500ab600 con 0x7fbb540a39b0 2024-09-16T03:44:35.466 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.457+0000 7fbb4bfff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:44:35.466 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.457+0000 7fbb4bfff640 1 -- 172.21.15.42:0/909896527 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb500ab9d0 con 0x7fbb540a39b0 2024-09-16T03:44:35.466 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.457+0000 7fbb5b7c8640 5 monclient: authenticate success, global_id 13103 2024-09-16T03:44:35.466 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.457+0000 7fbb5b7c8640 10 monclient: get_monmap_and_config success 2024-09-16T03:44:35.466 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.457+0000 7fbb5b7c8640 4 set_mon_vals no callback set 2024-09-16T03:44:35.466 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.457+0000 7fbb4bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:44:35.466 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.457+0000 7fbb4bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:44:35.467 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.457+0000 7fbb4bfff640 10 monclient: dump: 2024-09-16T03:44:35.467 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:44:35.467 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:44:35.467 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:35.467 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:35.467 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:44:35.467 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:44:35.467 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:44:35.467 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:44:35.467 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:44:35.467 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:44:35.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.457+0000 7fbb5b7c8640 10 monclient: shutdown 2024-09-16T03:44:35.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.457+0000 7fbb5b7c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:44:35.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.457+0000 7fbb5b7c8640 1 -- 172.21.15.42:0/909896527 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbb540a39b0 msgr2=0x7fbb54133b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:44:35.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.457+0000 7fbb5b7c8640 1 --2- 172.21.15.42:0/909896527 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbb540a39b0 0x7fbb54133b80 secure :-1 s=READY pgs=2275 cs=0 l=1 rev1=1 crypto rx=0x7fbb500097c0 tx=0x7fbb5001c360 comp rx=0 tx=0).stop 2024-09-16T03:44:35.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.457+0000 7fbb5b7c8640 1 -- 172.21.15.42:0/909896527 shutdown_connections 2024-09-16T03:44:35.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.457+0000 7fbb5b7c8640 1 --2- 172.21.15.42:0/909896527 >> v2:172.21.15.154:3300/0 conn(0x7fbb540a3d90 0x7fbb54137000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:44:35.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.457+0000 7fbb5b7c8640 1 --2- 172.21.15.42:0/909896527 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbb540a39b0 0x7fbb54133b80 unknown :-1 s=CLOSED pgs=2275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:44:35.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.457+0000 7fbb5b7c8640 1 -- 172.21.15.42:0/909896527 >> 172.21.15.42:0/909896527 conn(0x7fbb540a9fb0 msgr2=0x7fbb540aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:44:35.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.457+0000 7fbb5b7c8640 1 -- 172.21.15.42:0/909896527 shutdown_connections 2024-09-16T03:44:35.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.457+0000 7fbb5b7c8640 1 -- 172.21.15.42:0/909896527 wait complete. 2024-09-16T03:44:35.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.457+0000 7fbb5b7c8640 10 monclient: build_initial_monmap 2024-09-16T03:44:35.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.457+0000 7fbb5b7c8640 10 monclient: monmap: 2024-09-16T03:44:35.474 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:44:35.474 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:44:35.474 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:44:35.474 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:44:35.474 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:44:35.475 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:44:35.475 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:44:35.475 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:44:35.475 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:44:35.475 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:44:35.475 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.457+0000 7fbb5b7c8640 1 librados: starting msgr at 2024-09-16T03:44:35.475 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.457+0000 7fbb5b7c8640 1 librados: starting objecter 2024-09-16T03:44:35.475 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.457+0000 7fbb5b7c8640 1 Processor -- start 2024-09-16T03:44:35.475 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.457+0000 7fbb5b7c8640 1 -- start start 2024-09-16T03:44:35.475 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.457+0000 7fbb5b7c8640 1 librados: setting wanted keys 2024-09-16T03:44:35.475 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.457+0000 7fbb5b7c8640 1 librados: calling monclient init 2024-09-16T03:44:35.475 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.457+0000 7fbb5b7c8640 10 monclient: init 2024-09-16T03:44:35.475 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.457+0000 7fbb5b7c8640 10 monclient: _reopen_session rank -1 2024-09-16T03:44:35.476 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.457+0000 7fbb5b7c8640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:44:35.476 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.457+0000 7fbb5b7c8640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbb540a39b0 0x7fbb54130300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:44:35.476 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.457+0000 7fbb5b7c8640 10 monclient(hunting): picked mon.noname-a con 0x7fbb540a39b0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:44:35.476 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.457+0000 7fbb5b7c8640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbb540a3d90 0x7fbb54130840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:44:35.476 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.457+0000 7fbb5b7c8640 10 monclient(hunting): picked mon.noname-c con 0x7fbb540a3d90 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:44:35.476 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.457+0000 7fbb5b7c8640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbb54137540 0x7fbb54130d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:44:35.476 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.457+0000 7fbb5b7c8640 10 monclient(hunting): picked mon.noname-b con 0x7fbb54137540 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:44:35.476 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.457+0000 7fbb5b7c8640 10 monclient(hunting): start opening mon connection 2024-09-16T03:44:35.476 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.457+0000 7fbb5b7c8640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fbb541424e0 con 0x7fbb540a39b0 2024-09-16T03:44:35.476 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.457+0000 7fbb5b7c8640 10 monclient(hunting): start opening mon connection 2024-09-16T03:44:35.476 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.457+0000 7fbb5b7c8640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fbb54142660 con 0x7fbb54137540 2024-09-16T03:44:35.476 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.457+0000 7fbb5b7c8640 10 monclient(hunting): start opening mon connection 2024-09-16T03:44:35.476 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.457+0000 7fbb5b7c8640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fbb5409e500 con 0x7fbb540a3d90 2024-09-16T03:44:35.476 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.457+0000 7fbb5b7c8640 10 monclient(hunting): _renew_subs 2024-09-16T03:44:35.477 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.457+0000 7fbb5b7c8640 10 monclient(hunting): authenticate will time out at 1122.430884s 2024-09-16T03:44:35.477 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.457+0000 7fbb59564640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbb540a39b0 0x7fbb54130300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:44:35.477 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.457+0000 7fbb58d63640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbb540a3d90 0x7fbb54130840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:44:35.477 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.457+0000 7fbb59564640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbb540a39b0 0x7fbb54130300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:48316/0 (socket says 172.21.15.42:48316) 2024-09-16T03:44:35.477 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.457+0000 7fbb59564640 1 -- 172.21.15.42:0/1915391206 learned_addr learned my addr 172.21.15.42:0/1915391206 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:44:35.477 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.457+0000 7fbb59564640 10 monclient(hunting): get_auth_request con 0x7fbb540a39b0 auth_method 0 2024-09-16T03:44:35.477 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.457+0000 7fbb59564640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:44:35.477 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.457+0000 7fbb59564640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:44:35.477 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.457+0000 7fbb59564640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:44:35.477 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.457+0000 7fbb58d63640 10 monclient(hunting): get_auth_request con 0x7fbb540a3d90 auth_method 0 2024-09-16T03:44:35.477 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.457+0000 7fbb58d63640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:44:35.477 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.457+0000 7fbb58d63640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:44:35.477 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.457+0000 7fbb58d63640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:44:35.477 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.457+0000 7fbb59564640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:44:35.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.457+0000 7fbb59564640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:44:35.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.457+0000 7fbb59564640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:44:35.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.457+0000 7fbb58d63640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:44:35.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.457+0000 7fbb58d63640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:44:35.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.457+0000 7fbb58d63640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:44:35.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.457+0000 7fbb59d65640 1 --2- 172.21.15.42:0/1915391206 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbb54137540 0x7fbb54130d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:44:35.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.457+0000 7fbb59d65640 10 monclient(hunting): get_auth_request con 0x7fbb54137540 auth_method 0 2024-09-16T03:44:35.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.457+0000 7fbb59d65640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:44:35.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.457+0000 7fbb59d65640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:44:35.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.457+0000 7fbb59d65640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:44:35.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.457+0000 7fbb59564640 10 monclient(hunting): handle_auth_done global_id 27537 payload 824 2024-09-16T03:44:35.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.457+0000 7fbb59564640 1 -- 172.21.15.42:0/1915391206 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbb540a3d90 msgr2=0x7fbb54130840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:44:35.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.457+0000 7fbb59564640 1 --2- 172.21.15.42:0/1915391206 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbb540a3d90 0x7fbb54130840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:44:35.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.457+0000 7fbb59564640 1 -- 172.21.15.42:0/1915391206 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbb54137540 msgr2=0x7fbb54130d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:44:35.479 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.457+0000 7fbb59564640 1 --2- 172.21.15.42:0/1915391206 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbb54137540 0x7fbb54130d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:44:35.479 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.457+0000 7fbb59564640 10 monclient: _finish_hunting 0 2024-09-16T03:44:35.479 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.457+0000 7fbb59564640 1 monclient: found mon.noname-a 2024-09-16T03:44:35.479 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.457+0000 7fbb59564640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:44:35.479 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.457+0000 7fbb59564640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:44:35.479 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.457+0000 7fbb59564640 1 -- 172.21.15.42:0/1915391206 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb5409df40 con 0x7fbb540a39b0 2024-09-16T03:44:35.479 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.457+0000 7fbb59564640 1 --2- 172.21.15.42:0/1915391206 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbb540a39b0 0x7fbb54130300 secure :-1 s=READY pgs=7447 cs=0 l=1 rev1=1 crypto rx=0x7fbb500ae040 tx=0x7fbb50019100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:44:35.479 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.457+0000 7fbb49ffb640 10 client.?.objecter ms_handle_connect 0x7fbb540a39b0 2024-09-16T03:44:35.479 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.457+0000 7fbb49ffb640 10 client.?.objecter resend_mon_ops 2024-09-16T03:44:35.479 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.457+0000 7fbb49ffb640 1 -- 172.21.15.42:0/1915391206 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb500b0040 con 0x7fbb540a39b0 2024-09-16T03:44:35.479 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.457+0000 7fbb49ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:44:35.479 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.457+0000 7fbb49ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:44:35.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.457+0000 7fbb49ffb640 10 monclient: dump: 2024-09-16T03:44:35.480 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:44:35.480 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:44:35.480 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:35.480 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:35.480 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:44:35.480 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:44:35.480 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:44:35.480 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:44:35.481 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:44:35.481 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:44:35.481 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.457+0000 7fbb49ffb640 10 monclient: _finish_auth 0 2024-09-16T03:44:35.481 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.457+0000 7fbb49ffb640 10 monclient: _check_auth_tickets 2024-09-16T03:44:35.481 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.457+0000 7fbb49ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:44:35.481 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.457+0000 7fbb5b7c8640 5 monclient: authenticate success, global_id 27537 2024-09-16T03:44:35.481 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.457+0000 7fbb49ffb640 1 -- 172.21.15.42:0/1915391206 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb500b43f0 con 0x7fbb540a39b0 2024-09-16T03:44:35.481 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.457+0000 7fbb49ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T03:44:35.481 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.457+0000 7fbb49ffb640 1 -- 172.21.15.42:0/1915391206 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb500b4800 con 0x7fbb540a39b0 2024-09-16T03:44:35.481 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.457+0000 7fbb49ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:44:35.481 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.457+0000 7fbb49ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:44:35.481 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.457+0000 7fbb4b7fe640 4 set_mon_vals no callback set 2024-09-16T03:44:35.481 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.457+0000 7fbb49ffb640 10 monclient: dump: 2024-09-16T03:44:35.482 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:44:35.482 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:44:35.482 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:35.482 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:35.482 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:44:35.482 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:44:35.482 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:44:35.482 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:44:35.482 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:44:35.482 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:44:35.482 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.461+0000 7fbb5b7c8640 10 monclient: _renew_subs 2024-09-16T03:44:35.482 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.461+0000 7fbb5b7c8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:44:35.482 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.461+0000 7fbb5b7c8640 1 -- 172.21.15.42:0/1915391206 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb5413ef10 con 0x7fbb540a39b0 2024-09-16T03:44:35.482 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.461+0000 7fbb5b7c8640 10 client.27537.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:44:35.483 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.461+0000 7fbb5b7c8640 10 monclient: _renew_subs 2024-09-16T03:44:35.483 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.461+0000 7fbb5b7c8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:44:35.483 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.461+0000 7fbb5b7c8640 1 -- 172.21.15.42:0/1915391206 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb541372d0 con 0x7fbb540a39b0 2024-09-16T03:44:35.483 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.461+0000 7fbb5b7c8640 1 librados: init done 2024-09-16T03:44:35.483 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.461+0000 7fbb5b7c8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:44:35.483 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.461+0000 7fbb5b7c8640 1 --2- 172.21.15.42:0/1915391206 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbb54138130 0x7fbb54138510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:44:35.483 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.461+0000 7fbb5b7c8640 10 monclient(hunting): start opening mon connection 2024-09-16T03:44:35.483 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.461+0000 7fbb5b7c8640 1 -- 172.21.15.42:0/1915391206 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fbb541342d0 con 0x7fbb54138130 2024-09-16T03:44:35.483 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.461+0000 7fbb49ffb640 1 -- 172.21.15.42:0/1915391206 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fbb500b49a0 con 0x7fbb540a39b0 2024-09-16T03:44:35.483 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.461+0000 7fbb58d63640 1 --2- 172.21.15.42:0/1915391206 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbb54138130 0x7fbb54138510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:44:35.483 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.461+0000 7fbb58d63640 10 monclient: get_auth_request con 0x7fbb54138130 auth_method 0 2024-09-16T03:44:35.483 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.461+0000 7fbb58d63640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:44:35.483 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.461+0000 7fbb58d63640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:44:35.483 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.461+0000 7fbb58d63640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:44:35.483 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.461+0000 7fbb49ffb640 1 --2- 172.21.15.42:0/1915391206 >> v2:172.21.15.154:6812/254845342 conn(0x7fbb38037810 0x7fbb38039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:44:35.484 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.461+0000 7fbb49ffb640 1 -- 172.21.15.42:0/1915391206 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 10244+0+0 (secure 0 0 0) 0x7fbb500e3d90 con 0x7fbb540a39b0 2024-09-16T03:44:35.484 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.461+0000 7fbb49ffb640 10 client.27537.objecter ms_dispatch 0x7fbb54000ba0 osd_map(57..57 src has 1..57) 2024-09-16T03:44:35.484 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.461+0000 7fbb49ffb640 3 client.27537.objecter handle_osd_map got epochs [57,57] > 0 2024-09-16T03:44:35.484 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.461+0000 7fbb49ffb640 3 client.27537.objecter handle_osd_map decoding full epoch 57 2024-09-16T03:44:35.484 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.461+0000 7fbb58d63640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:44:35.484 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.461+0000 7fbb58d63640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:44:35.484 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.461+0000 7fbb58d63640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:44:35.484 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.461+0000 7fbb49ffb640 20 client.27537.objecter dump_active .. 0 homeless 2024-09-16T03:44:35.484 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.461+0000 7fbb59d65640 1 --2- 172.21.15.42:0/1915391206 >> v2:172.21.15.154:6812/254845342 conn(0x7fbb38037810 0x7fbb38039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:44:35.484 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.461+0000 7fbb59d65640 10 monclient: get_auth_request con 0x7fbb38037810 auth_method 0 2024-09-16T03:44:35.484 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.461+0000 7fbb58d63640 10 monclient(hunting): handle_auth_done global_id 272380 payload 824 2024-09-16T03:44:35.484 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.461+0000 7fbb58d63640 1 -- 172.21.15.42:0/1915391206 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbb54133b80 con 0x7fbb54138130 2024-09-16T03:44:35.484 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.461+0000 7fbb49ffb640 10 client.27537.objecter ms_handle_connect 0x7fbb54138130 2024-09-16T03:44:35.485 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.461+0000 7fbb58d63640 1 --2- 172.21.15.42:0/1915391206 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbb54138130 0x7fbb54138510 secure :-1 s=READY pgs=2720 cs=0 l=1 rev1=1 crypto rx=0x7fbb5409e150 tx=0x7fbb4400fce0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:44:35.485 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.461+0000 7fbb49ffb640 10 client.27537.objecter resend_mon_ops 2024-09-16T03:44:35.485 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.461+0000 7fbb49ffb640 10 client.27537.objecter ms_handle_connect 0x7fbb38037810 2024-09-16T03:44:35.485 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.461+0000 7fbb59d65640 1 --2- 172.21.15.42:0/1915391206 >> v2:172.21.15.154:6812/254845342 conn(0x7fbb38037810 0x7fbb38039cd0 secure :-1 s=READY pgs=3305 cs=0 l=1 rev1=1 crypto rx=0x7fbb4c001570 tx=0x7fbb4c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:44:35.485 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.461+0000 7fbb49ffb640 1 -- 172.21.15.42:0/1915391206 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb44004220 con 0x7fbb54138130 2024-09-16T03:44:35.485 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.461+0000 7fbb49ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:44:35.485 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.461+0000 7fbb49ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:44:35.485 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.461+0000 7fbb49ffb640 10 monclient: dump: 2024-09-16T03:44:35.485 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:44:35.485 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:44:35.485 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:35.485 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:35.485 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:44:35.485 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:44:35.485 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:44:35.486 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:44:35.486 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:44:35.486 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:44:35.486 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.461+0000 7fbb49ffb640 1 -- 172.21.15.42:0/1915391206 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fbb44004220 con 0x7fbb54138130 2024-09-16T03:44:35.486 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.461+0000 7fbb49ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:44:35.486 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.461+0000 7fbb49ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:44:35.486 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.461+0000 7fbb49ffb640 1 -- 172.21.15.42:0/1915391206 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbb54138130 msgr2=0x7fbb54138510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:44:35.486 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.461+0000 7fbb49ffb640 1 --2- 172.21.15.42:0/1915391206 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbb54138130 0x7fbb54138510 secure :-1 s=READY pgs=2720 cs=0 l=1 rev1=1 crypto rx=0x7fbb5409e150 tx=0x7fbb4400fce0 comp rx=0 tx=0).stop 2024-09-16T03:44:35.486 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.461+0000 7fbb49ffb640 1 -- 172.21.15.42:0/1915391206 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbb54138130 msgr2=0x7fbb54138510 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:44:35.486 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.461+0000 7fbb49ffb640 1 --2- 172.21.15.42:0/1915391206 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbb54138130 0x7fbb54138510 secure :-1 s=CLOSED pgs=2720 cs=0 l=1 rev1=1 crypto rx=0x7fbb5409e150 tx=0x7fbb4400fce0 comp rx=0 tx=0).stop 2024-09-16T03:44:35.488 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.469+0000 7fbb5b7c8640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:44:35.488 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.469+0000 7fbb5b7c8640 1 --2- 172.21.15.42:0/1915391206 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbb54138a50 0x7fbb5413cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:44:35.488 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.469+0000 7fbb5b7c8640 10 monclient(hunting): start opening mon connection 2024-09-16T03:44:35.488 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.469+0000 7fbb5b7c8640 1 -- 172.21.15.42:0/1915391206 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fbb5406df10 con 0x7fbb54138a50 2024-09-16T03:44:35.488 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.469+0000 7fbb58d63640 1 --2- 172.21.15.42:0/1915391206 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbb54138a50 0x7fbb5413cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:44:35.488 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.469+0000 7fbb58d63640 10 monclient: get_auth_request con 0x7fbb54138a50 auth_method 0 2024-09-16T03:44:35.488 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.469+0000 7fbb58d63640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:44:35.488 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.469+0000 7fbb58d63640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:44:35.488 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.469+0000 7fbb58d63640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:44:35.488 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.469+0000 7fbb58d63640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:44:35.488 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.469+0000 7fbb58d63640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:44:35.489 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.469+0000 7fbb58d63640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:44:35.489 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.469+0000 7fbb58d63640 10 monclient(hunting): handle_auth_done global_id 272383 payload 824 2024-09-16T03:44:35.489 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.469+0000 7fbb58d63640 1 -- 172.21.15.42:0/1915391206 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fbb540a5750 con 0x7fbb54138a50 2024-09-16T03:44:35.489 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.469+0000 7fbb58d63640 1 --2- 172.21.15.42:0/1915391206 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbb54138a50 0x7fbb5413cdd0 secure :-1 s=READY pgs=2721 cs=0 l=1 rev1=1 crypto rx=0x7fbb5409e150 tx=0x7fbb440046d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:44:35.489 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.469+0000 7fbb49ffb640 10 client.27537.objecter ms_handle_connect 0x7fbb54138a50 2024-09-16T03:44:35.489 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.469+0000 7fbb49ffb640 10 client.27537.objecter resend_mon_ops 2024-09-16T03:44:35.489 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.469+0000 7fbb49ffb640 1 -- 172.21.15.42:0/1915391206 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb44003980 con 0x7fbb54138a50 2024-09-16T03:44:35.489 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.469+0000 7fbb49ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:44:35.489 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.469+0000 7fbb49ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:44:35.489 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.469+0000 7fbb49ffb640 10 monclient: dump: 2024-09-16T03:44:35.489 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:44:35.489 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:44:35.489 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:35.489 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:35.490 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:44:35.490 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:44:35.490 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:44:35.490 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:44:35.490 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:44:35.490 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:44:35.490 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.469+0000 7fbb49ffb640 1 -- 172.21.15.42:0/1915391206 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4544 (secure 0 0 0) 0x7fbb44003980 con 0x7fbb54138a50 2024-09-16T03:44:35.490 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.469+0000 7fbb49ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:44:35.490 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.469+0000 7fbb49ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:44:35.490 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.469+0000 7fbb49ffb640 1 -- 172.21.15.42:0/1915391206 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbb54138a50 msgr2=0x7fbb5413cdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:44:35.490 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.469+0000 7fbb49ffb640 1 --2- 172.21.15.42:0/1915391206 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbb54138a50 0x7fbb5413cdd0 secure :-1 s=READY pgs=2721 cs=0 l=1 rev1=1 crypto rx=0x7fbb5409e150 tx=0x7fbb440046d0 comp rx=0 tx=0).stop 2024-09-16T03:44:35.490 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.469+0000 7fbb49ffb640 1 -- 172.21.15.42:0/1915391206 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbb54138a50 msgr2=0x7fbb5413cdd0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:44:35.490 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.469+0000 7fbb49ffb640 1 --2- 172.21.15.42:0/1915391206 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbb54138a50 0x7fbb5413cdd0 secure :-1 s=CLOSED pgs=2721 cs=0 l=1 rev1=1 crypto rx=0x7fbb5409e150 tx=0x7fbb440046d0 comp rx=0 tx=0).stop 2024-09-16T03:44:35.491 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:44:35.491 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:44:35.491 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:44:35.491 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T03:44:35.491 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 22, 2024-09-16T03:44:35.491 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 18198, 2024-09-16T03:44:35.491 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:44:35.491 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:44:35.491 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T03:44:35.491 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T03:44:35.491 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:44:35.491 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 3, 2024-09-16T03:44:35.492 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:44:35.492 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:44:35.492 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:44:35.492 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:44:35.492 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:44:35.492 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:44:35.492 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:44:35.492 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:44:35.492 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:44:35.492 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:44:35.492 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:44:35.492 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:44:35.492 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:44:35.492 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:44:35.493 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:44:35.493 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:44:35.493 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:44:35.493 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:44:35.493 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:44:35.493 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:44:35.493 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:44:35.493 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:44:35.493 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:44:35.493 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:44:35.493 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:44:35.493 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:44:35.493 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:44:35.493 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:44:35.493 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:44:35.494 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:44:35.494 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:44:35.494 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:44:35.494 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:44:35.494 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:44:35.494 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:44:35.494 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:44:35.494 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:44:35.494 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:44:35.494 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:44:35.494 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:44:35.494 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:44:35.494 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:44:35.494 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:44:35.495 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:44:35.495 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:44:35.495 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:44:35.495 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:44:35.495 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:44:35.495 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:44:35.495 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:44:35.495 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:44:35.495 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:44:35.495 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:44:35.495 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:44:35.495 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:44:35.495 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:44:35.495 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:44:35.495 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:44:35.496 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:44:35.496 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:44:35.496 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:44:35.496 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:44:35.496 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:44:35.496 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:44:35.496 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:44:35.496 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:44:35.496 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:44:35.496 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:44:35.496 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:44:35.496 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:44:35.496 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:44:35.496 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:44:35.497 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:44:35.497 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:44:35.497 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:44:35.497 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:44:35.497 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:44:35.497 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:44:35.497 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:44:35.497 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:44:35.497 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:44:35.497 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:44:35.497 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:44:35.497 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:44:35.497 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:44:35.497 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:44:35.497 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:44:35.498 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:44:35.498 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:44:35.498 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:44:35.498 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:44:35.498 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:44:35.498 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:44:35.498 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:44:35.498 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:44:35.498 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:44:35.498 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:44:35.498 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:44:35.498 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:44:35.498 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:44:35.498 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:44:35.498 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:44:35.499 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:44:35.499 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:44:35.499 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:44:35.499 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:44:35.499 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:44:35.499 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:44:35.499 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:44:35.499 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:44:35.499 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:44:35.499 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:44:35.499 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:44:35.499 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:44:35.499 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:44:35.499 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:44:35.499 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:44:35.500 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:44:35.500 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:44:35.500 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:44:35.500 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:44:35.500 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:44:35.500 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:44:35.500 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:44:35.500 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:44:35.500 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:44:35.500 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:44:35.500 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:44:35.500 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:44:35.500 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:44:35.500 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:44:35.501 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:44:35.501 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:44:35.501 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:44:35.501 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:44:35.501 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:44:35.501 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:44:35.501 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:44:35.501 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:44:35.501 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:44:35.501 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:44:35.501 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:44:35.501 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:44:35.501 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:44:35.501 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:44:35.501 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 4 2024-09-16T03:44:35.502 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:44:35.502 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:44:35.502 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:44:35.502 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:44:35.502 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:44:35.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.473+0000 7fbb5b7c8640 10 librados: watch_flush enter 2024-09-16T03:44:35.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.473+0000 7fbb5b7c8640 10 librados: watch_flush exit 2024-09-16T03:44:35.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.473+0000 7fbb5b7c8640 20 client.27537.objecter shutdown clearing up homeless session... 2024-09-16T03:44:35.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.473+0000 7fbb5b7c8640 10 client.27537.objecter successfully canceled tick 2024-09-16T03:44:35.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.473+0000 7fbb5b7c8640 1 -- 172.21.15.42:0/1915391206 >> v2:172.21.15.154:6812/254845342 conn(0x7fbb38037810 msgr2=0x7fbb38039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:44:35.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.473+0000 7fbb5b7c8640 1 --2- 172.21.15.42:0/1915391206 >> v2:172.21.15.154:6812/254845342 conn(0x7fbb38037810 0x7fbb38039cd0 secure :-1 s=READY pgs=3305 cs=0 l=1 rev1=1 crypto rx=0x7fbb4c001570 tx=0x7fbb4c009040 comp rx=0 tx=0).stop 2024-09-16T03:44:35.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.473+0000 7fbb5b7c8640 10 monclient: shutdown 2024-09-16T03:44:35.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.473+0000 7fbb5b7c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:44:35.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.473+0000 7fbb5b7c8640 1 -- 172.21.15.42:0/1915391206 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbb540a39b0 msgr2=0x7fbb54130300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:44:35.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.473+0000 7fbb5b7c8640 1 --2- 172.21.15.42:0/1915391206 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbb540a39b0 0x7fbb54130300 secure :-1 s=READY pgs=7447 cs=0 l=1 rev1=1 crypto rx=0x7fbb500ae040 tx=0x7fbb50019100 comp rx=0 tx=0).stop 2024-09-16T03:44:35.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.473+0000 7fbb59d65640 1 -- 172.21.15.42:0/1915391206 reap_dead start 2024-09-16T03:44:35.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.473+0000 7fbb59d65640 1 -- 172.21.15.42:0/1915391206 reap_dead start 2024-09-16T03:44:35.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.473+0000 7fbb5b7c8640 1 -- 172.21.15.42:0/1915391206 shutdown_connections 2024-09-16T03:44:35.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.473+0000 7fbb5b7c8640 1 -- 172.21.15.42:0/1915391206 >> 172.21.15.42:0/1915391206 conn(0x7fbb540a9fb0 msgr2=0x7fbb540a4cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:44:35.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.473+0000 7fbb5b7c8640 1 -- 172.21.15.42:0/1915391206 shutdown_connections 2024-09-16T03:44:35.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.473+0000 7fbb5b7c8640 1 -- 172.21.15.42:0/1915391206 wait complete. 2024-09-16T03:44:35.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.473+0000 7fbb5b7c8640 1 librados: shutdown 2024-09-16T03:44:35.505 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:44:35.505 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-09-16T03:44:35.505 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-16T03:44:35.505 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-09-16T03:44:35.505 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-16T03:44:35.505 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-16T03:44:35.506 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T03:44:35.522 INFO:tasks.ceph.mon.a:Stopped 2024-09-16T03:44:35.522 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T03:44:35.522 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:44:35.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.681+0000 7f7296306640 10 monclient: get_monmap_and_config 2024-09-16T03:44:35.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.681+0000 7f7296306640 10 monclient: build_initial_monmap 2024-09-16T03:44:35.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.681+0000 7f7296306640 10 monclient: monmap: 2024-09-16T03:44:35.687 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:44:35.687 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:44:35.687 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:44:35.686249+0000 2024-09-16T03:44:35.687 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:44:35.686249+0000 2024-09-16T03:44:35.687 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:44:35.687 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:44:35.688 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:44:35.688 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:44:35.688 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:44:35.688 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:44:35.688 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:44:35.688 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:44:35.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.681+0000 7f7296306640 1 Processor -- start 2024-09-16T03:44:35.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.681+0000 7f7296306640 1 -- start start 2024-09-16T03:44:35.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.681+0000 7f7296306640 10 monclient: init 2024-09-16T03:44:35.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.681+0000 7f7296306640 10 monclient: _reopen_session rank -1 2024-09-16T03:44:35.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.681+0000 7f7296306640 10 monclient: _add_conns ranks=[0,1,3,2,4] 2024-09-16T03:44:35.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.681+0000 7f7296306640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f72900a39b0 addr v1:172.21.15.42:6789/0 2024-09-16T03:44:35.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.681+0000 7f7296306640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f72900a3d90 addr v1:172.21.15.154:6789/0 2024-09-16T03:44:35.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.681+0000 7f7296306640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f729013daa0 0x7f729013fe90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:44:35.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.681+0000 7f7296306640 10 monclient(hunting): picked mon.noname-a con 0x7f729013daa0 addr v2:172.21.15.42:3300/0 2024-09-16T03:44:35.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.681+0000 7f7296306640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7290134290 con 0x7f72900a39b0 2024-09-16T03:44:35.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.681+0000 7f7296306640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f729009c140 con 0x7f72900a3d90 2024-09-16T03:44:35.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.681+0000 7f7296306640 10 monclient(hunting): start opening mon connection 2024-09-16T03:44:35.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.681+0000 7f7296306640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f729009c420 con 0x7f729013daa0 2024-09-16T03:44:35.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.681+0000 7f7296306640 10 monclient(hunting): _renew_subs 2024-09-16T03:44:35.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.681+0000 7f7296306640 10 monclient(hunting): authenticate will time out at 1122.655693s 2024-09-16T03:44:35.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.681+0000 7f72948a3640 1 -- >> v2:172.21.15.42:3300/0 conn(0x7f729013daa0 msgr2=0x7f729013fe90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3300/0 2024-09-16T03:44:35.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.681+0000 7f72948a3640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f729013daa0 0x7f729013fe90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:44:35.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.681+0000 7f728ffff640 1 -- >> v1:172.21.15.42:6789/0 conn(0x7f72900a39b0 legacy=0x7f7290133b80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.42:6789/0 2024-09-16T03:44:35.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.685+0000 7f728f7fe640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f72900a3d90 0x7f729013c380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.42:47464/0 (socket says 172.21.15.42:47464) 2024-09-16T03:44:35.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.685+0000 7f728f7fe640 1 -- 172.21.15.42:0/2440843393 learned_addr learned my addr 172.21.15.42:0/2440843393 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:44:35.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.685+0000 7f728effd640 1 -- 172.21.15.42:0/2440843393 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1097520763 0 0) 0x7f729009c140 con 0x7f72900a3d90 2024-09-16T03:44:35.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.685+0000 7f728effd640 10 monclient(hunting): _negotiate 2024-09-16T03:44:35.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.685+0000 7f728effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:44:35.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.685+0000 7f728effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:44:35.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.685+0000 7f728effd640 10 monclient(hunting): my global_id is 272386 2024-09-16T03:44:35.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.685+0000 7f728effd640 1 -- 172.21.15.42:0/2440843393 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7278003530 con 0x7f72900a3d90 2024-09-16T03:44:35.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.685+0000 7f728effd640 1 -- 172.21.15.42:0/2440843393 <== mon.1 v1:172.21.15.154:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4236798565 0 0) 0x7f7278003530 con 0x7f72900a3d90 2024-09-16T03:44:35.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.685+0000 7f728effd640 1 -- 172.21.15.42:0/2440843393 >> v2:172.21.15.42:3300/0 conn(0x7f729013daa0 msgr2=0x7f729013fe90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:44:35.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.685+0000 7f728effd640 1 --2- 172.21.15.42:0/2440843393 >> v2:172.21.15.42:3300/0 conn(0x7f729013daa0 0x7f729013fe90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:44:35.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.685+0000 7f728effd640 1 -- 172.21.15.42:0/2440843393 >> v1:172.21.15.42:6789/0 conn(0x7f72900a39b0 legacy=0x7f7290133b80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:44:35.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.685+0000 7f728effd640 10 monclient: _finish_hunting 0 2024-09-16T03:44:35.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.685+0000 7f728effd640 1 monclient: found mon.noname-c-legacy 2024-09-16T03:44:35.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.685+0000 7f728effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:44:35.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.685+0000 7f728effd640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.154:6789/0 2024-09-16T03:44:35.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.685+0000 7f728effd640 1 -- 172.21.15.42:0/2440843393 --> v1:172.21.15.154:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f729006dd60 con 0x7f72900a3d90 2024-09-16T03:44:35.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.685+0000 7f728effd640 10 monclient: _finish_auth 0 2024-09-16T03:44:35.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.685+0000 7f728effd640 10 monclient: _check_auth_tickets 2024-09-16T03:44:35.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.685+0000 7f728effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:44:35.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.685+0000 7f728effd640 1 -- 172.21.15.42:0/2440843393 <== mon.1 v1:172.21.15.154:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2177413734 0 0) 0x7f7284001680 con 0x7f72900a3d90 2024-09-16T03:44:35.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.685+0000 7f728effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:44:35.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.685+0000 7f728effd640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-16T03:44:35.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.685+0000 7f728effd640 10 monclient: dump: 2024-09-16T03:44:35.695 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:44:35.695 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:44:35.695 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:35.696 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:35.696 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:44:35.696 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:44:35.696 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:44:35.696 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:44:35.696 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:44:35.696 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:44:35.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.685+0000 7f728effd640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] but i'm connected to v1:172.21.15.154:6789/0, reconnecting 2024-09-16T03:44:35.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.685+0000 7f728effd640 10 monclient: _reopen_session rank -1 2024-09-16T03:44:35.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.685+0000 7f728effd640 1 -- 172.21.15.42:0/2440843393 >> v1:172.21.15.154:6789/0 conn(0x7f72900a3d90 legacy=0x7f729013c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:44:35.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.685+0000 7f728effd640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:44:35.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.685+0000 7f728effd640 1 --2- 172.21.15.42:0/2440843393 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f72780039c0 0x7f7278003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:44:35.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.685+0000 7f728effd640 10 monclient(hunting): picked mon.b con 0x7f72780039c0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:44:35.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.685+0000 7f728effd640 1 --2- 172.21.15.42:0/2440843393 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7278004810 0x7f7278004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:44:35.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.685+0000 7f728effd640 10 monclient(hunting): picked mon.a con 0x7f7278004810 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:44:35.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.685+0000 7f728effd640 1 --2- 172.21.15.42:0/2440843393 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7278005380 0x7f727800b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:44:35.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.685+0000 7f728effd640 10 monclient(hunting): picked mon.c con 0x7f7278005380 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:44:35.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.685+0000 7f728effd640 10 monclient(hunting): start opening mon connection 2024-09-16T03:44:35.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.685+0000 7f728effd640 1 -- 172.21.15.42:0/2440843393 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f729009c420 con 0x7f7278004810 2024-09-16T03:44:35.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.685+0000 7f728effd640 10 monclient(hunting): start opening mon connection 2024-09-16T03:44:35.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.685+0000 7f728effd640 1 -- 172.21.15.42:0/2440843393 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f727800c2d0 con 0x7f72780039c0 2024-09-16T03:44:35.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.685+0000 7f728effd640 10 monclient(hunting): start opening mon connection 2024-09-16T03:44:35.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.685+0000 7f728effd640 1 -- 172.21.15.42:0/2440843393 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f727800c4d0 con 0x7f7278005380 2024-09-16T03:44:35.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.685+0000 7f728effd640 10 monclient(hunting): _renew_subs 2024-09-16T03:44:35.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.685+0000 7f728effd640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:44:35.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.685+0000 7f728f7fe640 1 -- 172.21.15.42:0/2440843393 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7278004810 msgr2=0x7f7278004c90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3300/0 2024-09-16T03:44:35.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.685+0000 7f728f7fe640 1 --2- 172.21.15.42:0/2440843393 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7278004810 0x7f7278004c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:44:35.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.685+0000 7f72948a3640 1 --2- 172.21.15.42:0/2440843393 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7278005380 0x7f727800b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:44:35.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.685+0000 7f72948a3640 10 monclient(hunting): get_auth_request con 0x7f7278005380 auth_method 0 2024-09-16T03:44:35.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.685+0000 7f72948a3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:44:35.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.685+0000 7f72948a3640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:44:35.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.685+0000 7f72948a3640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:44:35.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.685+0000 7f72948a3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:44:35.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.685+0000 7f72948a3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:44:35.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.685+0000 7f72948a3640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T03:44:35.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.685+0000 7f728ffff640 1 --2- 172.21.15.42:0/2440843393 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f72780039c0 0x7f7278003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:44:35.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.685+0000 7f728ffff640 10 monclient(hunting): get_auth_request con 0x7f72780039c0 auth_method 0 2024-09-16T03:44:35.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.685+0000 7f728ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:44:35.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.685+0000 7f728ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:44:35.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.685+0000 7f728ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:44:35.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.685+0000 7f72948a3640 10 monclient(hunting): handle_auth_done global_id 272386 payload 293 2024-09-16T03:44:35.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.685+0000 7f72948a3640 1 -- 172.21.15.42:0/2440843393 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f72780039c0 msgr2=0x7f7278003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:44:35.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.685+0000 7f72948a3640 1 --2- 172.21.15.42:0/2440843393 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f72780039c0 0x7f7278003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:44:35.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.685+0000 7f72948a3640 1 -- 172.21.15.42:0/2440843393 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7278004810 msgr2=0x7f7278004c90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:44:35.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.685+0000 7f72948a3640 1 --2- 172.21.15.42:0/2440843393 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7278004810 0x7f7278004c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:44:35.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.685+0000 7f72948a3640 10 monclient: _finish_hunting 0 2024-09-16T03:44:35.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.685+0000 7f72948a3640 1 monclient: found mon.c 2024-09-16T03:44:35.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.685+0000 7f72948a3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:44:35.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.685+0000 7f72948a3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:44:35.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.685+0000 7f72948a3640 1 -- 172.21.15.42:0/2440843393 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7278003530 con 0x7f7278005380 2024-09-16T03:44:35.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.685+0000 7f72948a3640 10 monclient: _finish_auth 0 2024-09-16T03:44:35.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.685+0000 7f72948a3640 10 monclient: _check_auth_tickets 2024-09-16T03:44:35.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.685+0000 7f72948a3640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:44:35.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.685+0000 7f7296306640 5 monclient: authenticate success, global_id 272386 2024-09-16T03:44:35.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.685+0000 7f7296306640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:44:35.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.685+0000 7f72948a3640 1 -- 172.21.15.42:0/2440843393 reap_dead start 2024-09-16T03:44:35.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.685+0000 7f72948a3640 1 --2- 172.21.15.42:0/2440843393 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7278005380 0x7f727800b9f0 secure :-1 s=READY pgs=2288 cs=0 l=1 rev1=1 crypto rx=0x7f7280003990 tx=0x7f728001c420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eaa85c23fbd4d13e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:44:35.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.685+0000 7f728effd640 1 -- 172.21.15.42:0/2440843393 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7280009520 con 0x7f7278005380 2024-09-16T03:44:35.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.685+0000 7f728effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:44:35.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.689+0000 7f728effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:44:35.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.689+0000 7f728effd640 10 monclient: dump: 2024-09-16T03:44:35.705 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:44:35.705 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:44:35.705 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:35.705 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:35.705 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:44:35.705 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:44:35.706 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:44:35.706 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:44:35.706 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:44:35.706 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:44:35.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.689+0000 7f7296306640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:44:35.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.689+0000 7f728effd640 1 -- 172.21.15.42:0/2440843393 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7280003bc0 con 0x7f7278005380 2024-09-16T03:44:35.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.689+0000 7f728effd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:44:35.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.689+0000 7f7296306640 10 monclient: get_monmap_and_config success 2024-09-16T03:44:35.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.689+0000 7f7296306640 4 set_mon_vals no callback set 2024-09-16T03:44:35.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.689+0000 7f728effd640 1 -- 172.21.15.42:0/2440843393 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7280005070 con 0x7f7278005380 2024-09-16T03:44:35.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.689+0000 7f728effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:44:35.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.689+0000 7f728effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:44:35.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.689+0000 7f728effd640 10 monclient: dump: 2024-09-16T03:44:35.707 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:44:35.707 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:44:35.707 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:35.707 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:35.707 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:44:35.707 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:44:35.707 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:44:35.707 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:44:35.707 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:44:35.707 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:44:35.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.689+0000 7f7296306640 10 monclient: shutdown 2024-09-16T03:44:35.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.689+0000 7f7296306640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:44:35.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.689+0000 7f7296306640 1 -- 172.21.15.42:0/2440843393 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7278005380 msgr2=0x7f727800b9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:44:35.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.689+0000 7f7296306640 1 --2- 172.21.15.42:0/2440843393 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7278005380 0x7f727800b9f0 secure :-1 s=READY pgs=2288 cs=0 l=1 rev1=1 crypto rx=0x7f7280003990 tx=0x7f728001c420 comp rx=0 tx=0).stop 2024-09-16T03:44:35.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.689+0000 7f7296306640 1 -- 172.21.15.42:0/2440843393 shutdown_connections 2024-09-16T03:44:35.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.689+0000 7f7296306640 1 --2- 172.21.15.42:0/2440843393 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7278005380 0x7f727800b9f0 unknown :-1 s=CLOSED pgs=2288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:44:35.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.689+0000 7f7296306640 1 -- 172.21.15.42:0/2440843393 >> 172.21.15.42:0/2440843393 conn(0x7f72900a9fb0 msgr2=0x7f72900aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:44:35.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.689+0000 7f7296306640 1 -- 172.21.15.42:0/2440843393 shutdown_connections 2024-09-16T03:44:35.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.689+0000 7f7296306640 1 -- 172.21.15.42:0/2440843393 wait complete. 2024-09-16T03:44:35.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.689+0000 7f7296306640 10 monclient: build_initial_monmap 2024-09-16T03:44:35.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.689+0000 7f7296306640 10 monclient: monmap: 2024-09-16T03:44:35.708 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:44:35.708 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:44:35.708 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:44:35.708 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:44:35.708 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:44:35.708 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:44:35.709 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:44:35.709 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:44:35.709 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:44:35.709 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:44:35.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.689+0000 7f7296306640 1 librados: starting msgr at 2024-09-16T03:44:35.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.689+0000 7f7296306640 1 librados: starting objecter 2024-09-16T03:44:35.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.689+0000 7f7296306640 1 Processor -- start 2024-09-16T03:44:35.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.689+0000 7f7296306640 1 -- start start 2024-09-16T03:44:35.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.689+0000 7f7296306640 1 librados: setting wanted keys 2024-09-16T03:44:35.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.689+0000 7f7296306640 1 librados: calling monclient init 2024-09-16T03:44:35.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.689+0000 7f7296306640 10 monclient: init 2024-09-16T03:44:35.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.689+0000 7f7296306640 10 monclient: _reopen_session rank -1 2024-09-16T03:44:35.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.689+0000 7f7296306640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:44:35.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.689+0000 7f7296306640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7278005380 0x7f72900a2540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:44:35.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.689+0000 7f7296306640 10 monclient(hunting): picked mon.noname-b con 0x7f7278005380 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:44:35.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.689+0000 7f7296306640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f729013b070 0x7f729013b4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:44:35.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.689+0000 7f7296306640 10 monclient(hunting): picked mon.noname-a con 0x7f729013b070 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:44:35.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.689+0000 7f7296306640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f729013b9f0 0x7f72901d59b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:44:35.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.689+0000 7f7296306640 10 monclient(hunting): picked mon.noname-c con 0x7f729013b9f0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:44:35.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.689+0000 7f7296306640 10 monclient(hunting): start opening mon connection 2024-09-16T03:44:35.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.689+0000 7f7296306640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f7290133f10 con 0x7f729013b070 2024-09-16T03:44:35.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.689+0000 7f7296306640 10 monclient(hunting): start opening mon connection 2024-09-16T03:44:35.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.689+0000 7f7296306640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f7290133d90 con 0x7f7278005380 2024-09-16T03:44:35.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.689+0000 7f7296306640 10 monclient(hunting): start opening mon connection 2024-09-16T03:44:35.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.689+0000 7f7296306640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f7290134090 con 0x7f729013b9f0 2024-09-16T03:44:35.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.689+0000 7f7296306640 10 monclient(hunting): _renew_subs 2024-09-16T03:44:35.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.689+0000 7f7296306640 10 monclient(hunting): authenticate will time out at 1122.662248s 2024-09-16T03:44:35.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.689+0000 7f728f7fe640 1 -- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f729013b070 msgr2=0x7f729013b4b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.42:3300/0 2024-09-16T03:44:35.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.689+0000 7f728f7fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f729013b070 0x7f729013b4b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:44:35.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.689+0000 7f72948a3640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f729013b9f0 0x7f72901d59b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:44:35.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.689+0000 7f72948a3640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f729013b9f0 0x7f72901d59b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:39220/0 (socket says 172.21.15.42:39220) 2024-09-16T03:44:35.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.689+0000 7f72948a3640 1 -- 172.21.15.42:0/3901781520 learned_addr learned my addr 172.21.15.42:0/3901781520 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:44:35.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.689+0000 7f72948a3640 10 monclient(hunting): get_auth_request con 0x7f729013b9f0 auth_method 0 2024-09-16T03:44:35.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.689+0000 7f72948a3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:44:35.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.689+0000 7f72948a3640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:44:35.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.689+0000 7f72948a3640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:44:35.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.689+0000 7f72948a3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:44:35.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.689+0000 7f72948a3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:44:35.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.689+0000 7f72948a3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:44:35.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.689+0000 7f728ffff640 1 --2- 172.21.15.42:0/3901781520 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7278005380 0x7f72900a2540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:44:35.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.689+0000 7f728ffff640 10 monclient(hunting): get_auth_request con 0x7f7278005380 auth_method 0 2024-09-16T03:44:35.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.689+0000 7f728ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:44:35.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.689+0000 7f728ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:44:35.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.689+0000 7f728ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:44:35.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.689+0000 7f72948a3640 10 monclient(hunting): handle_auth_done global_id 13109 payload 824 2024-09-16T03:44:35.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.689+0000 7f72948a3640 1 -- 172.21.15.42:0/3901781520 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7278005380 msgr2=0x7f72900a2540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:44:35.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.689+0000 7f72948a3640 1 --2- 172.21.15.42:0/3901781520 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7278005380 0x7f72900a2540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:44:35.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.689+0000 7f72948a3640 1 -- 172.21.15.42:0/3901781520 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f729013b070 msgr2=0x7f729013b4b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:44:35.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.689+0000 7f72948a3640 1 --2- 172.21.15.42:0/3901781520 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f729013b070 0x7f729013b4b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:44:35.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.689+0000 7f72948a3640 10 monclient: _finish_hunting 0 2024-09-16T03:44:35.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.689+0000 7f72948a3640 1 monclient: found mon.noname-c 2024-09-16T03:44:35.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.689+0000 7f72948a3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:44:35.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.689+0000 7f72948a3640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:44:35.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.689+0000 7f72948a3640 1 -- 172.21.15.42:0/3901781520 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f729009ddb0 con 0x7f729013b9f0 2024-09-16T03:44:35.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.689+0000 7f72948a3640 1 --2- 172.21.15.42:0/3901781520 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f729013b9f0 0x7f72901d59b0 secure :-1 s=READY pgs=2289 cs=0 l=1 rev1=1 crypto rx=0x7f7280006fd0 tx=0x7f7278005840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:44:35.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.689+0000 7f728cff9640 10 client.?.objecter ms_handle_connect 0x7f729013b9f0 2024-09-16T03:44:35.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.689+0000 7f728cff9640 10 client.?.objecter resend_mon_ops 2024-09-16T03:44:35.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.689+0000 7f728cff9640 1 -- 172.21.15.42:0/3901781520 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7280009500 con 0x7f729013b9f0 2024-09-16T03:44:35.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.689+0000 7f728cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:44:35.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.689+0000 7f728cff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:44:35.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.689+0000 7f728cff9640 10 monclient: dump: 2024-09-16T03:44:35.713 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:44:35.713 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:44:35.713 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:35.713 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:35.713 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:44:35.713 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:44:35.714 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:44:35.714 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:44:35.714 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:44:35.714 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:44:35.714 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.689+0000 7f728cff9640 10 monclient: _finish_auth 0 2024-09-16T03:44:35.714 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.689+0000 7f728cff9640 10 monclient: _check_auth_tickets 2024-09-16T03:44:35.714 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.689+0000 7f728cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:44:35.714 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.689+0000 7f728cff9640 1 -- 172.21.15.42:0/3901781520 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7280003d70 con 0x7f729013b9f0 2024-09-16T03:44:35.714 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.689+0000 7f728cff9640 10 monclient: handle_config config(0 keys) 2024-09-16T03:44:35.714 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.689+0000 7f728cff9640 1 -- 172.21.15.42:0/3901781520 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72800ac460 con 0x7f729013b9f0 2024-09-16T03:44:35.714 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.689+0000 7f728cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:44:35.714 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.689+0000 7f728cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:44:35.714 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.689+0000 7f728cff9640 10 monclient: dump: 2024-09-16T03:44:35.714 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:44:35.715 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:44:35.715 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:35.715 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:35.715 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:44:35.715 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:44:35.715 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:44:35.715 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:44:35.715 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:44:35.715 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:44:35.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.689+0000 7f7296306640 5 monclient: authenticate success, global_id 13109 2024-09-16T03:44:35.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.689+0000 7f7296306640 10 monclient: _renew_subs 2024-09-16T03:44:35.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.689+0000 7f728e7fc640 4 set_mon_vals no callback set 2024-09-16T03:44:35.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.689+0000 7f7296306640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:44:35.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.689+0000 7f7296306640 1 -- 172.21.15.42:0/3901781520 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72901334c0 con 0x7f729013b9f0 2024-09-16T03:44:35.716 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.689+0000 7f7296306640 10 client.13109.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:44:35.716 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.689+0000 7f7296306640 10 monclient: _renew_subs 2024-09-16T03:44:35.716 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.689+0000 7f7296306640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:44:35.716 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.689+0000 7f7296306640 1 -- 172.21.15.42:0/3901781520 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f72900a94f0 con 0x7f729013b9f0 2024-09-16T03:44:35.716 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.689+0000 7f7296306640 1 librados: init done 2024-09-16T03:44:35.716 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.693+0000 7f7296306640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:44:35.716 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.693+0000 7f7296306640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:44:35.716 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.693+0000 7f7296306640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:44:35.716 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.693+0000 7f7296306640 1 -- 172.21.15.42:0/3901781520 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7290134210 con 0x7f729013b9f0 2024-09-16T03:44:35.716 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.693+0000 7f728cff9640 1 -- 172.21.15.42:0/3901781520 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f72800ac600 con 0x7f729013b9f0 2024-09-16T03:44:35.716 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.693+0000 7f728cff9640 1 --2- 172.21.15.42:0/3901781520 >> v2:172.21.15.154:6812/254845342 conn(0x7f727c0377c0 0x7f727c039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:44:35.716 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.693+0000 7f728cff9640 1 -- 172.21.15.42:0/3901781520 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(57..57 src has 1..57) ==== 10244+0+0 (secure 0 0 0) 0x7f72800e35a0 con 0x7f729013b9f0 2024-09-16T03:44:35.717 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.693+0000 7f728cff9640 10 client.13109.objecter ms_dispatch 0x7f7290000ba0 osd_map(57..57 src has 1..57) 2024-09-16T03:44:35.717 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.693+0000 7f728cff9640 3 client.13109.objecter handle_osd_map got epochs [57,57] > 0 2024-09-16T03:44:35.717 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.693+0000 7f728cff9640 3 client.13109.objecter handle_osd_map decoding full epoch 57 2024-09-16T03:44:35.717 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.693+0000 7f728cff9640 20 client.13109.objecter dump_active .. 0 homeless 2024-09-16T03:44:35.717 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.693+0000 7f728ffff640 1 --2- 172.21.15.42:0/3901781520 >> v2:172.21.15.154:6812/254845342 conn(0x7f727c0377c0 0x7f727c039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:44:35.717 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.693+0000 7f728ffff640 10 monclient: get_auth_request con 0x7f727c0377c0 auth_method 0 2024-09-16T03:44:35.717 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.693+0000 7f728ffff640 1 --2- 172.21.15.42:0/3901781520 >> v2:172.21.15.154:6812/254845342 conn(0x7f727c0377c0 0x7f727c039c80 secure :-1 s=READY pgs=3306 cs=0 l=1 rev1=1 crypto rx=0x7f7288003d10 tx=0x7f728800b370 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:44:35.717 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.693+0000 7f728cff9640 10 client.13109.objecter ms_handle_connect 0x7f727c0377c0 2024-09-16T03:44:35.717 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.697+0000 7f728cff9640 1 -- 172.21.15.42:0/3901781520 <== mon.2 v2:172.21.15.42:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f72800bc030 con 0x7f729013b9f0 2024-09-16T03:44:35.717 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.697+0000 7f728cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:44:35.717 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.697+0000 7f728cff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:44:35.933 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.929+0000 7f7296306640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:44:35.933 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.929+0000 7f7296306640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:44:35.933 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.929+0000 7f7296306640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:44:35.933 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.929+0000 7f7296306640 1 -- 172.21.15.42:0/3901781520 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f72900a5750 con 0x7f729013b9f0 2024-09-16T03:44:35.934 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.929+0000 7f728cff9640 1 -- 172.21.15.42:0/3901781520 <== mon.2 v2:172.21.15.42:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7f7280016130 con 0x7f729013b9f0 2024-09-16T03:44:35.934 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.929+0000 7f728cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:44:35.934 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.929+0000 7f728cff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:44:35.934 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":22,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:44:35.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.929+0000 7f7296306640 10 librados: watch_flush enter 2024-09-16T03:44:35.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.929+0000 7f7296306640 10 librados: watch_flush exit 2024-09-16T03:44:35.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.929+0000 7f7296306640 20 client.13109.objecter shutdown clearing up homeless session... 2024-09-16T03:44:35.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.929+0000 7f7296306640 10 client.13109.objecter successfully canceled tick 2024-09-16T03:44:35.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.929+0000 7f7296306640 1 -- 172.21.15.42:0/3901781520 >> v2:172.21.15.154:6812/254845342 conn(0x7f727c0377c0 msgr2=0x7f727c039c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:44:35.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.929+0000 7f7296306640 1 --2- 172.21.15.42:0/3901781520 >> v2:172.21.15.154:6812/254845342 conn(0x7f727c0377c0 0x7f727c039c80 secure :-1 s=READY pgs=3306 cs=0 l=1 rev1=1 crypto rx=0x7f7288003d10 tx=0x7f728800b370 comp rx=0 tx=0).stop 2024-09-16T03:44:35.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.929+0000 7f7296306640 10 monclient: shutdown 2024-09-16T03:44:35.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.929+0000 7f7296306640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:44:35.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.929+0000 7f7296306640 1 -- 172.21.15.42:0/3901781520 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f729013b9f0 msgr2=0x7f72901d59b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:44:35.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.929+0000 7f7296306640 1 --2- 172.21.15.42:0/3901781520 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f729013b9f0 0x7f72901d59b0 secure :-1 s=READY pgs=2289 cs=0 l=1 rev1=1 crypto rx=0x7f7280006fd0 tx=0x7f7278005840 comp rx=0 tx=0).stop 2024-09-16T03:44:35.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.929+0000 7f7296306640 1 -- 172.21.15.42:0/3901781520 shutdown_connections 2024-09-16T03:44:35.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.929+0000 7f7296306640 1 --2- 172.21.15.42:0/3901781520 >> v2:172.21.15.154:6812/254845342 conn(0x7f727c0377c0 0x7f727c039c80 unknown :-1 s=CLOSED pgs=3306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:44:35.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.929+0000 7f7296306640 1 --2- 172.21.15.42:0/3901781520 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f729013b9f0 0x7f72901d59b0 unknown :-1 s=CLOSED pgs=2289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:44:35.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.929+0000 7f7296306640 1 --2- 172.21.15.42:0/3901781520 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f729013b070 0x7f729013b4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:44:35.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.929+0000 7f7296306640 1 --2- 172.21.15.42:0/3901781520 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7278005380 0x7f72900a2540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:44:35.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.929+0000 7f7296306640 1 -- 172.21.15.42:0/3901781520 >> 172.21.15.42:0/3901781520 conn(0x7f72900a9fb0 msgr2=0x7f72900a7ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:44:35.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.929+0000 7f7296306640 1 -- 172.21.15.42:0/3901781520 shutdown_connections 2024-09-16T03:44:35.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.929+0000 7f7296306640 1 -- 172.21.15.42:0/3901781520 wait complete. 2024-09-16T03:44:35.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:35.929+0000 7f7296306640 1 librados: shutdown 2024-09-16T03:44:38.954 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:44:39.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.081+0000 7f35ec433640 10 monclient: get_monmap_and_config 2024-09-16T03:44:39.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.081+0000 7f35ec433640 10 monclient: build_initial_monmap 2024-09-16T03:44:39.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.081+0000 7f35ec433640 10 monclient: monmap: 2024-09-16T03:44:39.087 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:44:39.087 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:44:39.087 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:44:39.085906+0000 2024-09-16T03:44:39.087 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:44:39.085906+0000 2024-09-16T03:44:39.087 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:44:39.087 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:44:39.087 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:44:39.087 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:44:39.087 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:44:39.087 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:44:39.087 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:44:39.087 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:44:39.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.081+0000 7f35ec433640 1 Processor -- start 2024-09-16T03:44:39.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.081+0000 7f35ec433640 1 -- start start 2024-09-16T03:44:39.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.081+0000 7f35ec433640 10 monclient: init 2024-09-16T03:44:39.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.081+0000 7f35ec433640 10 monclient: _reopen_session rank -1 2024-09-16T03:44:39.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.081+0000 7f35ec433640 10 monclient: _add_conns ranks=[0,1,4,3,2] 2024-09-16T03:44:39.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.081+0000 7f35ec433640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f35e40a2230 addr v1:172.21.15.42:6789/0 2024-09-16T03:44:39.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.081+0000 7f35ec433640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f35e4072740 addr v1:172.21.15.154:6789/0 2024-09-16T03:44:39.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.081+0000 7f35ec433640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f35e40a2d20 0x7f35e413dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:44:39.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.081+0000 7f35ec433640 10 monclient(hunting): picked mon.noname-c con 0x7f35e40a2d20 addr v2:172.21.15.154:3300/0 2024-09-16T03:44:39.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.081+0000 7f35ec433640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f35e406c050 con 0x7f35e40a2230 2024-09-16T03:44:39.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.081+0000 7f35ec433640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f35e406c3b0 con 0x7f35e4072740 2024-09-16T03:44:39.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.081+0000 7f35ec433640 10 monclient(hunting): start opening mon connection 2024-09-16T03:44:39.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.081+0000 7f35ec433640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f35e4071b10 con 0x7f35e40a2d20 2024-09-16T03:44:39.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.081+0000 7f35ec433640 10 monclient(hunting): _renew_subs 2024-09-16T03:44:39.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.081+0000 7f35ea1cf640 1 -- >> v1:172.21.15.42:6789/0 conn(0x7f35e40a2230 legacy=0x7f35e40a2610 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.42:6789/0 2024-09-16T03:44:39.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.081+0000 7f35ec433640 10 monclient(hunting): authenticate will time out at 1126.055510s 2024-09-16T03:44:39.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.081+0000 7f35ea9d0640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f35e40a2d20 0x7f35e413dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:44:39.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.081+0000 7f35e99ce640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f35e4072740 0x7f35e413a4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.42:47480/0 (socket says 172.21.15.42:47480) 2024-09-16T03:44:39.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.081+0000 7f35e99ce640 1 -- 172.21.15.42:0/2379632424 learned_addr learned my addr 172.21.15.42:0/2379632424 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:44:39.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.081+0000 7f35ea9d0640 10 monclient(hunting): get_auth_request con 0x7f35e40a2d20 auth_method 0 2024-09-16T03:44:39.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.085+0000 7f35ea9d0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:44:39.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.085+0000 7f35ea9d0640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:44:39.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.085+0000 7f35ea9d0640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:44:39.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.085+0000 7f35ea9d0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:44:39.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.085+0000 7f35ea9d0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:44:39.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.085+0000 7f35ea9d0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:44:39.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.085+0000 7f35ea9d0640 10 monclient(hunting): handle_auth_done global_id 272392 payload 274 2024-09-16T03:44:39.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.085+0000 7f35ea9d0640 1 -- 172.21.15.42:0/2379632424 >> v1:172.21.15.154:6789/0 conn(0x7f35e4072740 legacy=0x7f35e413a4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:44:39.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.085+0000 7f35ea9d0640 1 -- 172.21.15.42:0/2379632424 >> v1:172.21.15.42:6789/0 conn(0x7f35e40a2230 legacy=0x7f35e40a2610 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:44:39.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.085+0000 7f35ea9d0640 10 monclient: _finish_hunting 0 2024-09-16T03:44:39.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.085+0000 7f35ea9d0640 1 monclient: found mon.noname-c 2024-09-16T03:44:39.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.085+0000 7f35ea9d0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:44:39.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.085+0000 7f35ea9d0640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.154:3300/0 2024-09-16T03:44:39.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.085+0000 7f35ea9d0640 1 -- 172.21.15.42:0/2379632424 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35e40a3c90 con 0x7f35e40a2d20 2024-09-16T03:44:39.093 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.085+0000 7f35ea9d0640 1 --2- 172.21.15.42:0/2379632424 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f35e40a2d20 0x7f35e413dbe0 secure :-1 s=READY pgs=2730 cs=0 l=1 rev1=1 crypto rx=0x7f35dc009dc0 tx=0x7f35dc01c7d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cf58781a350b6ab0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:44:39.093 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.085+0000 7f35e91cd640 1 -- 172.21.15.42:0/2379632424 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35dc0b2070 con 0x7f35e40a2d20 2024-09-16T03:44:39.093 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.085+0000 7f35e91cd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:44:39.093 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.085+0000 7f35e91cd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:44:39.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.085+0000 7f35e91cd640 10 monclient: dump: 2024-09-16T03:44:39.094 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:44:39.094 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:44:39.094 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:39.094 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:39.094 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:44:39.094 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:44:39.094 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:44:39.094 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:44:39.094 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:44:39.094 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:44:39.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.085+0000 7f35e91cd640 10 monclient: _finish_auth 0 2024-09-16T03:44:39.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.085+0000 7f35e91cd640 10 monclient: _check_auth_tickets 2024-09-16T03:44:39.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.085+0000 7f35e91cd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:44:39.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.085+0000 7f35e91cd640 1 -- 172.21.15.42:0/2379632424 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f35dc0a57a0 con 0x7f35e40a2d20 2024-09-16T03:44:39.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.085+0000 7f35e91cd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:44:39.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.085+0000 7f35e91cd640 1 -- 172.21.15.42:0/2379632424 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35dc0a5b70 con 0x7f35e40a2d20 2024-09-16T03:44:39.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.085+0000 7f35e91cd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:44:39.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.085+0000 7f35e91cd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:44:39.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.085+0000 7f35e91cd640 10 monclient: dump: 2024-09-16T03:44:39.095 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:44:39.095 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:44:39.095 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:39.095 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:39.095 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:44:39.095 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:44:39.095 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:44:39.095 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:44:39.096 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:44:39.096 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:44:39.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.085+0000 7f35ec433640 5 monclient: authenticate success, global_id 272392 2024-09-16T03:44:39.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.085+0000 7f35ec433640 10 monclient: get_monmap_and_config success 2024-09-16T03:44:39.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.085+0000 7f35ec433640 4 set_mon_vals no callback set 2024-09-16T03:44:39.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.089+0000 7f35ec433640 10 monclient: shutdown 2024-09-16T03:44:39.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.089+0000 7f35ec433640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:44:39.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.089+0000 7f35ec433640 1 -- 172.21.15.42:0/2379632424 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f35e40a2d20 msgr2=0x7f35e413dbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:44:39.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.089+0000 7f35ec433640 1 --2- 172.21.15.42:0/2379632424 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f35e40a2d20 0x7f35e413dbe0 secure :-1 s=READY pgs=2730 cs=0 l=1 rev1=1 crypto rx=0x7f35dc009dc0 tx=0x7f35dc01c7d0 comp rx=0 tx=0).stop 2024-09-16T03:44:39.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.089+0000 7f35ec433640 1 -- 172.21.15.42:0/2379632424 shutdown_connections 2024-09-16T03:44:39.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.089+0000 7f35ec433640 1 --2- 172.21.15.42:0/2379632424 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f35e40a2d20 0x7f35e413dbe0 unknown :-1 s=CLOSED pgs=2730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:44:39.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.089+0000 7f35ec433640 1 -- 172.21.15.42:0/2379632424 >> 172.21.15.42:0/2379632424 conn(0x7f35e40a6e80 msgr2=0x7f35e40a3320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:44:39.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.089+0000 7f35ec433640 1 -- 172.21.15.42:0/2379632424 shutdown_connections 2024-09-16T03:44:39.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.089+0000 7f35ec433640 1 -- 172.21.15.42:0/2379632424 wait complete. 2024-09-16T03:44:39.100 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.089+0000 7f35ec433640 10 monclient: build_initial_monmap 2024-09-16T03:44:39.101 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.089+0000 7f35ec433640 10 monclient: monmap: 2024-09-16T03:44:39.101 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:44:39.101 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:44:39.101 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:44:39.101 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:44:39.101 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:44:39.101 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:44:39.101 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:44:39.101 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:44:39.101 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:44:39.101 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:44:39.101 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.089+0000 7f35ec433640 1 librados: starting msgr at 2024-09-16T03:44:39.101 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.089+0000 7f35ec433640 1 librados: starting objecter 2024-09-16T03:44:39.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.089+0000 7f35ec433640 1 Processor -- start 2024-09-16T03:44:39.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.089+0000 7f35ec433640 1 -- start start 2024-09-16T03:44:39.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.089+0000 7f35ec433640 1 librados: setting wanted keys 2024-09-16T03:44:39.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.089+0000 7f35ec433640 1 librados: calling monclient init 2024-09-16T03:44:39.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.089+0000 7f35ec433640 10 monclient: init 2024-09-16T03:44:39.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.089+0000 7f35ec433640 10 monclient: _reopen_session rank -1 2024-09-16T03:44:39.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.089+0000 7f35ec433640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T03:44:39.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.089+0000 7f35ec433640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f35e40a2230 0x7f35e41ceda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:44:39.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.089+0000 7f35ec433640 10 monclient(hunting): picked mon.noname-a con 0x7f35e40a2230 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:44:39.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.089+0000 7f35ec433640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f35e40a2d20 0x7f35e41cf4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:44:39.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.089+0000 7f35ec433640 10 monclient(hunting): picked mon.noname-b con 0x7f35e40a2d20 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:44:39.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.089+0000 7f35ec433640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f35e41cfa10 0x7f35e413a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:44:39.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.089+0000 7f35ec433640 10 monclient(hunting): picked mon.noname-c con 0x7f35e41cfa10 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:44:39.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.089+0000 7f35ec433640 10 monclient(hunting): start opening mon connection 2024-09-16T03:44:39.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.089+0000 7f35ec433640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f35e406ebe0 con 0x7f35e40a2230 2024-09-16T03:44:39.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.089+0000 7f35ec433640 10 monclient(hunting): start opening mon connection 2024-09-16T03:44:39.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.089+0000 7f35ec433640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f35e406ea60 con 0x7f35e40a2d20 2024-09-16T03:44:39.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.089+0000 7f35ec433640 10 monclient(hunting): start opening mon connection 2024-09-16T03:44:39.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.089+0000 7f35ec433640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f35e406ed60 con 0x7f35e41cfa10 2024-09-16T03:44:39.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.089+0000 7f35ec433640 10 monclient(hunting): _renew_subs 2024-09-16T03:44:39.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.089+0000 7f35ea1cf640 1 -- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f35e40a2230 msgr2=0x7f35e41ceda0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.42:3300/0 2024-09-16T03:44:39.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.089+0000 7f35ec433640 10 monclient(hunting): authenticate will time out at 1126.062237s 2024-09-16T03:44:39.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.089+0000 7f35ea1cf640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f35e40a2230 0x7f35e41ceda0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:44:39.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.089+0000 7f35ea9d0640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f35e41cfa10 0x7f35e413a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:44:39.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.089+0000 7f35ea9d0640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f35e41cfa10 0x7f35e413a250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:39234/0 (socket says 172.21.15.42:39234) 2024-09-16T03:44:39.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.089+0000 7f35ea9d0640 1 -- 172.21.15.42:0/4002465881 learned_addr learned my addr 172.21.15.42:0/4002465881 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:44:39.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.089+0000 7f35ea9d0640 10 monclient(hunting): get_auth_request con 0x7f35e41cfa10 auth_method 0 2024-09-16T03:44:39.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.089+0000 7f35ea9d0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:44:39.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.089+0000 7f35ea9d0640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:44:39.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.089+0000 7f35ea9d0640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:44:39.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.089+0000 7f35ea9d0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:44:39.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.089+0000 7f35ea9d0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:44:39.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.089+0000 7f35ea9d0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:44:39.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.089+0000 7f35ea9d0640 10 monclient(hunting): handle_auth_done global_id 13112 payload 824 2024-09-16T03:44:39.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.089+0000 7f35ea9d0640 1 -- 172.21.15.42:0/4002465881 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f35e40a2d20 msgr2=0x7f35e41cf4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:44:39.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.089+0000 7f35ea9d0640 1 --2- 172.21.15.42:0/4002465881 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f35e40a2d20 0x7f35e41cf4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:44:39.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.089+0000 7f35ea9d0640 1 -- 172.21.15.42:0/4002465881 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f35e40a2230 msgr2=0x7f35e41ceda0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:44:39.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.089+0000 7f35ea9d0640 1 --2- 172.21.15.42:0/4002465881 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f35e40a2230 0x7f35e41ceda0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:44:39.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.089+0000 7f35ea9d0640 10 monclient: _finish_hunting 0 2024-09-16T03:44:39.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.089+0000 7f35ea9d0640 1 monclient: found mon.noname-c 2024-09-16T03:44:39.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.089+0000 7f35ea9d0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:44:39.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.089+0000 7f35ea9d0640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:44:39.105 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.089+0000 7f35ea9d0640 1 -- 172.21.15.42:0/4002465881 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35e413dc20 con 0x7f35e41cfa10 2024-09-16T03:44:39.105 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.089+0000 7f35ea9d0640 1 --2- 172.21.15.42:0/4002465881 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f35e41cfa10 0x7f35e413a250 secure :-1 s=READY pgs=2294 cs=0 l=1 rev1=1 crypto rx=0x7f35dc01a400 tx=0x7f35dc019570 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:44:39.105 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.089+0000 7f35daffd640 10 client.?.objecter ms_handle_connect 0x7f35e41cfa10 2024-09-16T03:44:39.105 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.089+0000 7f35daffd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:44:39.105 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.089+0000 7f35daffd640 1 -- 172.21.15.42:0/4002465881 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35dc0bd020 con 0x7f35e41cfa10 2024-09-16T03:44:39.105 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.089+0000 7f35daffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:44:39.105 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.089+0000 7f35daffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:44:39.105 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.089+0000 7f35daffd640 10 monclient: dump: 2024-09-16T03:44:39.105 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:44:39.105 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:44:39.105 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:39.105 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:39.105 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:44:39.106 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:44:39.106 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:44:39.106 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:44:39.106 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:44:39.106 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:44:39.106 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.089+0000 7f35daffd640 10 monclient: _finish_auth 0 2024-09-16T03:44:39.106 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.089+0000 7f35daffd640 10 monclient: _check_auth_tickets 2024-09-16T03:44:39.106 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.089+0000 7f35daffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:44:39.106 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.089+0000 7f35ec433640 5 monclient: authenticate success, global_id 13112 2024-09-16T03:44:39.106 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.089+0000 7f35ec433640 10 monclient: _renew_subs 2024-09-16T03:44:39.106 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.089+0000 7f35daffd640 1 -- 172.21.15.42:0/4002465881 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f35dc0ad7f0 con 0x7f35e41cfa10 2024-09-16T03:44:39.106 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.089+0000 7f35ec433640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:44:39.106 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.089+0000 7f35ec433640 1 -- 172.21.15.42:0/4002465881 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35e411cf10 con 0x7f35e41cfa10 2024-09-16T03:44:39.106 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.089+0000 7f35daffd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:44:39.107 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.089+0000 7f35daffd640 1 -- 172.21.15.42:0/4002465881 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35dc0b2040 con 0x7f35e41cfa10 2024-09-16T03:44:39.107 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.089+0000 7f35daffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:44:39.107 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.089+0000 7f35daffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:44:39.107 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.089+0000 7f35e89cc640 4 set_mon_vals no callback set 2024-09-16T03:44:39.107 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.089+0000 7f35daffd640 10 monclient: dump: 2024-09-16T03:44:39.107 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:44:39.107 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:44:39.107 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:39.107 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:39.107 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:44:39.107 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:44:39.107 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:44:39.107 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:44:39.107 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:44:39.108 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:44:39.108 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.089+0000 7f35ec433640 10 client.13112.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:44:39.108 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.089+0000 7f35ec433640 10 monclient: _renew_subs 2024-09-16T03:44:39.108 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.089+0000 7f35ec433640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:44:39.108 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.089+0000 7f35ec433640 1 -- 172.21.15.42:0/4002465881 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f35e413cc70 con 0x7f35e41cfa10 2024-09-16T03:44:39.108 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.089+0000 7f35ec433640 1 librados: init done 2024-09-16T03:44:39.108 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.093+0000 7f35daffd640 1 -- 172.21.15.42:0/4002465881 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f35dc0c0020 con 0x7f35e41cfa10 2024-09-16T03:44:39.108 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.093+0000 7f35daffd640 1 --2- 172.21.15.42:0/4002465881 >> v2:172.21.15.154:6812/254845342 conn(0x7f35c80377c0 0x7f35c8039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:44:39.108 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.093+0000 7f35daffd640 1 -- 172.21.15.42:0/4002465881 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(57..57 src has 1..57) ==== 10244+0+0 (secure 0 0 0) 0x7f35dc0e61c0 con 0x7f35e41cfa10 2024-09-16T03:44:39.108 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.093+0000 7f35daffd640 10 client.13112.objecter ms_dispatch 0x7f35e4000ba0 osd_map(57..57 src has 1..57) 2024-09-16T03:44:39.108 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.093+0000 7f35daffd640 3 client.13112.objecter handle_osd_map got epochs [57,57] > 0 2024-09-16T03:44:39.108 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.093+0000 7f35daffd640 3 client.13112.objecter handle_osd_map decoding full epoch 57 2024-09-16T03:44:39.108 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.093+0000 7f35ec433640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:44:39.108 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.093+0000 7f35ec433640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:44:39.109 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.093+0000 7f35ec433640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:44:39.109 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.093+0000 7f35ec433640 1 -- 172.21.15.42:0/4002465881 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f35e40a2610 con 0x7f35e41cfa10 2024-09-16T03:44:39.109 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.093+0000 7f35daffd640 20 client.13112.objecter dump_active .. 0 homeless 2024-09-16T03:44:39.109 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.093+0000 7f35ea1cf640 1 --2- 172.21.15.42:0/4002465881 >> v2:172.21.15.154:6812/254845342 conn(0x7f35c80377c0 0x7f35c8039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:44:39.109 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.093+0000 7f35ea1cf640 10 monclient: get_auth_request con 0x7f35c80377c0 auth_method 0 2024-09-16T03:44:39.109 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.093+0000 7f35daffd640 10 client.13112.objecter ms_handle_connect 0x7f35c80377c0 2024-09-16T03:44:39.109 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.093+0000 7f35ea1cf640 1 --2- 172.21.15.42:0/4002465881 >> v2:172.21.15.154:6812/254845342 conn(0x7f35c80377c0 0x7f35c8039c80 secure :-1 s=READY pgs=3307 cs=0 l=1 rev1=1 crypto rx=0x7f35e0003990 tx=0x7f35e0006d90 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:44:39.109 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.097+0000 7f35daffd640 1 -- 172.21.15.42:0/4002465881 <== mon.2 v2:172.21.15.42:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f35dc0164e0 con 0x7f35e41cfa10 2024-09-16T03:44:39.109 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.097+0000 7f35daffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:44:39.109 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.097+0000 7f35daffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:44:39.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.321+0000 7f35ec433640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:44:39.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.321+0000 7f35ec433640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:44:39.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.321+0000 7f35ec433640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:44:39.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.321+0000 7f35ec433640 1 -- 172.21.15.42:0/4002465881 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f35e413c7d0 con 0x7f35e41cfa10 2024-09-16T03:44:39.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.321+0000 7f35daffd640 1 -- 172.21.15.42:0/4002465881 <== mon.2 v2:172.21.15.42:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7f35dc0b6c20 con 0x7f35e41cfa10 2024-09-16T03:44:39.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.321+0000 7f35daffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:44:39.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.321+0000 7f35daffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:44:39.328 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":22,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:44:39.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.325+0000 7f35ec433640 10 librados: watch_flush enter 2024-09-16T03:44:39.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.325+0000 7f35ec433640 10 librados: watch_flush exit 2024-09-16T03:44:39.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.325+0000 7f35ec433640 20 client.13112.objecter shutdown clearing up homeless session... 2024-09-16T03:44:39.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.325+0000 7f35ec433640 10 client.13112.objecter successfully canceled tick 2024-09-16T03:44:39.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.325+0000 7f35ec433640 1 -- 172.21.15.42:0/4002465881 >> v2:172.21.15.154:6812/254845342 conn(0x7f35c80377c0 msgr2=0x7f35c8039c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:44:39.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.325+0000 7f35ec433640 1 --2- 172.21.15.42:0/4002465881 >> v2:172.21.15.154:6812/254845342 conn(0x7f35c80377c0 0x7f35c8039c80 secure :-1 s=READY pgs=3307 cs=0 l=1 rev1=1 crypto rx=0x7f35e0003990 tx=0x7f35e0006d90 comp rx=0 tx=0).stop 2024-09-16T03:44:39.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.325+0000 7f35ec433640 10 monclient: shutdown 2024-09-16T03:44:39.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.325+0000 7f35ec433640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:44:39.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.325+0000 7f35ec433640 1 -- 172.21.15.42:0/4002465881 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f35e41cfa10 msgr2=0x7f35e413a250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:44:39.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.325+0000 7f35ec433640 1 --2- 172.21.15.42:0/4002465881 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f35e41cfa10 0x7f35e413a250 secure :-1 s=READY pgs=2294 cs=0 l=1 rev1=1 crypto rx=0x7f35dc01a400 tx=0x7f35dc019570 comp rx=0 tx=0).stop 2024-09-16T03:44:39.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.325+0000 7f35ec433640 1 -- 172.21.15.42:0/4002465881 shutdown_connections 2024-09-16T03:44:39.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.325+0000 7f35ec433640 1 --2- 172.21.15.42:0/4002465881 >> v2:172.21.15.154:6812/254845342 conn(0x7f35c80377c0 0x7f35c8039c80 unknown :-1 s=CLOSED pgs=3307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:44:39.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.325+0000 7f35ec433640 1 --2- 172.21.15.42:0/4002465881 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f35e41cfa10 0x7f35e413a250 unknown :-1 s=CLOSED pgs=2294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:44:39.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.325+0000 7f35ec433640 1 --2- 172.21.15.42:0/4002465881 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f35e40a2d20 0x7f35e41cf4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:44:39.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.325+0000 7f35ec433640 1 --2- 172.21.15.42:0/4002465881 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f35e40a2230 0x7f35e41ceda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:44:39.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.325+0000 7f35ec433640 1 -- 172.21.15.42:0/4002465881 >> 172.21.15.42:0/4002465881 conn(0x7f35e40a6e80 msgr2=0x7f35e40a56b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:44:39.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.325+0000 7f35ec433640 1 -- 172.21.15.42:0/4002465881 shutdown_connections 2024-09-16T03:44:39.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.325+0000 7f35ec433640 1 -- 172.21.15.42:0/4002465881 wait complete. 2024-09-16T03:44:39.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:39.325+0000 7f35ec433640 1 librados: shutdown 2024-09-16T03:44:42.347 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:44:42.482 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.477+0000 7f60aaca1640 10 monclient: get_monmap_and_config 2024-09-16T03:44:42.483 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.477+0000 7f60aaca1640 10 monclient: build_initial_monmap 2024-09-16T03:44:42.483 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.477+0000 7f60aaca1640 10 monclient: monmap: 2024-09-16T03:44:42.483 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:44:42.483 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:44:42.483 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:44:42.481777+0000 2024-09-16T03:44:42.483 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:44:42.481777+0000 2024-09-16T03:44:42.483 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:44:42.483 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:44:42.483 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:44:42.483 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:44:42.483 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:44:42.483 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:44:42.484 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:44:42.484 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:44:42.486 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.477+0000 7f60aaca1640 1 Processor -- start 2024-09-16T03:44:42.486 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.477+0000 7f60aaca1640 1 -- start start 2024-09-16T03:44:42.486 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.477+0000 7f60aaca1640 10 monclient: init 2024-09-16T03:44:42.486 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.477+0000 7f60aaca1640 10 monclient: _reopen_session rank -1 2024-09-16T03:44:42.486 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.477+0000 7f60aaca1640 10 monclient: _add_conns ranks=[4,2,3,1,0] 2024-09-16T03:44:42.486 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.477+0000 7f60aaca1640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f60a406db90 0x7f60a41340c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:44:42.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.477+0000 7f60aaca1640 10 monclient(hunting): picked mon.noname-c con 0x7f60a406db90 addr v2:172.21.15.154:3300/0 2024-09-16T03:44:42.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.477+0000 7f60aaca1640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f60a40a37f0 0x7f60a40a3c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:44:42.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.477+0000 7f60aaca1640 10 monclient(hunting): picked mon.noname-b con 0x7f60a40a37f0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:44:42.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.477+0000 7f60aaca1640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f60a4136e10 0x7f60a4137240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:44:42.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.477+0000 7f60aaca1640 10 monclient(hunting): picked mon.noname-a con 0x7f60a4136e10 addr v2:172.21.15.42:3300/0 2024-09-16T03:44:42.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.477+0000 7f60aaca1640 10 monclient(hunting): start opening mon connection 2024-09-16T03:44:42.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.481+0000 7f60aaca1640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f60a406df70 con 0x7f60a4136e10 2024-09-16T03:44:42.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.481+0000 7f60aaca1640 10 monclient(hunting): start opening mon connection 2024-09-16T03:44:42.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.481+0000 7f60aaca1640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f60a406ece0 con 0x7f60a406db90 2024-09-16T03:44:42.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.481+0000 7f60aaca1640 10 monclient(hunting): start opening mon connection 2024-09-16T03:44:42.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.481+0000 7f60aaca1640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f60a409bf80 con 0x7f60a40a37f0 2024-09-16T03:44:42.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.481+0000 7f60aaca1640 10 monclient(hunting): _renew_subs 2024-09-16T03:44:42.488 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.481+0000 7f60aaca1640 10 monclient(hunting): authenticate will time out at 1129.452037s 2024-09-16T03:44:42.488 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.481+0000 7f60a923e640 1 -- >> v2:172.21.15.42:3300/0 conn(0x7f60a4136e10 msgr2=0x7f60a4137240 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3300/0 2024-09-16T03:44:42.488 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.481+0000 7f60a923e640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f60a4136e10 0x7f60a4137240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:44:42.488 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.481+0000 7f60a3fff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f60a40a37f0 0x7f60a40a3c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:44:42.488 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.481+0000 7f60a3fff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f60a40a37f0 0x7f60a40a3c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:39238/0 (socket says 172.21.15.42:39238) 2024-09-16T03:44:42.488 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.481+0000 7f60a3fff640 1 -- 172.21.15.42:0/2389582003 learned_addr learned my addr 172.21.15.42:0/2389582003 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:44:42.488 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.481+0000 7f60a3fff640 10 monclient(hunting): get_auth_request con 0x7f60a40a37f0 auth_method 0 2024-09-16T03:44:42.488 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.481+0000 7f60a3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:44:42.488 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.481+0000 7f60a3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:44:42.488 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.481+0000 7f60a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:44:42.489 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.481+0000 7f60a3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:44:42.489 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.481+0000 7f60a3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:44:42.489 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.481+0000 7f60a3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:44:42.489 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.481+0000 7f60a3fff640 10 monclient(hunting): handle_auth_done global_id 13115 payload 274 2024-09-16T03:44:42.489 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.481+0000 7f60a3fff640 1 -- 172.21.15.42:0/2389582003 >> v2:172.21.15.154:3300/0 conn(0x7f60a406db90 msgr2=0x7f60a41340c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:44:42.489 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.481+0000 7f60a3fff640 1 --2- 172.21.15.42:0/2389582003 >> v2:172.21.15.154:3300/0 conn(0x7f60a406db90 0x7f60a41340c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:44:42.489 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.481+0000 7f60a3fff640 1 -- 172.21.15.42:0/2389582003 >> v2:172.21.15.42:3300/0 conn(0x7f60a4136e10 msgr2=0x7f60a4137240 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:44:42.489 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.481+0000 7f60a3fff640 1 --2- 172.21.15.42:0/2389582003 >> v2:172.21.15.42:3300/0 conn(0x7f60a4136e10 0x7f60a4137240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:44:42.489 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.481+0000 7f60a3fff640 10 monclient: _finish_hunting 0 2024-09-16T03:44:42.489 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.481+0000 7f60a3fff640 1 monclient: found mon.noname-b 2024-09-16T03:44:42.490 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.481+0000 7f60a3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:44:42.490 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.481+0000 7f60a3fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T03:44:42.490 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.481+0000 7f60a3fff640 1 -- 172.21.15.42:0/2389582003 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60a4134780 con 0x7f60a40a37f0 2024-09-16T03:44:42.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.481+0000 7f60a3fff640 1 --2- 172.21.15.42:0/2389582003 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f60a40a37f0 0x7f60a40a3c00 secure :-1 s=READY pgs=2296 cs=0 l=1 rev1=1 crypto rx=0x7f60a40a76f0 tx=0x7f608c01c400 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c1c7503b01aeb950 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:44:42.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.481+0000 7f60a37fe640 1 -- 172.21.15.42:0/2389582003 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f608c0b0070 con 0x7f60a40a37f0 2024-09-16T03:44:42.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.481+0000 7f60a37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:44:42.498 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.481+0000 7f60a37fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:44:42.498 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.481+0000 7f60a37fe640 10 monclient: dump: 2024-09-16T03:44:42.498 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:44:42.498 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:44:42.498 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:42.498 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:42.498 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:44:42.498 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:44:42.498 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:44:42.498 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:44:42.499 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:44:42.499 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:44:42.499 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.481+0000 7f60a37fe640 10 monclient: _finish_auth 0 2024-09-16T03:44:42.499 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.481+0000 7f60a37fe640 10 monclient: _check_auth_tickets 2024-09-16T03:44:42.499 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.481+0000 7f60a37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:44:42.499 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.481+0000 7f60a37fe640 1 -- 172.21.15.42:0/2389582003 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f608c0a4e50 con 0x7f60a40a37f0 2024-09-16T03:44:42.499 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.481+0000 7f60aaca1640 5 monclient: authenticate success, global_id 13115 2024-09-16T03:44:42.499 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.481+0000 7f60aaca1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:44:42.499 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.481+0000 7f60a37fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:44:42.499 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.481+0000 7f60aaca1640 10 monclient: get_monmap_and_config success 2024-09-16T03:44:42.499 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.481+0000 7f60aaca1640 4 set_mon_vals no callback set 2024-09-16T03:44:42.499 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.481+0000 7f60a37fe640 1 -- 172.21.15.42:0/2389582003 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f608c0ab820 con 0x7f60a40a37f0 2024-09-16T03:44:42.499 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.481+0000 7f60a37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:44:42.500 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.481+0000 7f60a37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:44:42.500 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.481+0000 7f60a37fe640 10 monclient: dump: 2024-09-16T03:44:42.500 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:44:42.500 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:44:42.500 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:42.500 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:42.500 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:44:42.500 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:44:42.500 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:44:42.500 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:44:42.500 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:44:42.500 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:44:42.501 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.481+0000 7f60aaca1640 10 monclient: shutdown 2024-09-16T03:44:42.501 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.481+0000 7f60aaca1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:44:42.501 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.481+0000 7f60aaca1640 1 -- 172.21.15.42:0/2389582003 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f60a40a37f0 msgr2=0x7f60a40a3c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:44:42.501 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.481+0000 7f60aaca1640 1 --2- 172.21.15.42:0/2389582003 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f60a40a37f0 0x7f60a40a3c00 secure :-1 s=READY pgs=2296 cs=0 l=1 rev1=1 crypto rx=0x7f60a40a76f0 tx=0x7f608c01c400 comp rx=0 tx=0).stop 2024-09-16T03:44:42.501 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.481+0000 7f60aaca1640 1 -- 172.21.15.42:0/2389582003 shutdown_connections 2024-09-16T03:44:42.501 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.481+0000 7f60aaca1640 1 --2- 172.21.15.42:0/2389582003 >> v2:172.21.15.42:3300/0 conn(0x7f60a4136e10 0x7f60a4137240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:44:42.501 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.481+0000 7f60aaca1640 1 --2- 172.21.15.42:0/2389582003 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f60a40a37f0 0x7f60a40a3c00 unknown :-1 s=CLOSED pgs=2296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:44:42.501 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.481+0000 7f60aaca1640 1 --2- 172.21.15.42:0/2389582003 >> v2:172.21.15.154:3300/0 conn(0x7f60a406db90 0x7f60a41340c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:44:42.501 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.481+0000 7f60aaca1640 1 -- 172.21.15.42:0/2389582003 >> 172.21.15.42:0/2389582003 conn(0x7f60a40a9fb0 msgr2=0x7f60a40aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:44:42.501 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.481+0000 7f60aaca1640 1 -- 172.21.15.42:0/2389582003 shutdown_connections 2024-09-16T03:44:42.501 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.481+0000 7f60aaca1640 1 -- 172.21.15.42:0/2389582003 wait complete. 2024-09-16T03:44:42.501 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.485+0000 7f60aaca1640 10 monclient: build_initial_monmap 2024-09-16T03:44:42.501 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.485+0000 7f60aaca1640 10 monclient: monmap: 2024-09-16T03:44:42.502 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:44:42.502 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:44:42.502 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:44:42.502 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:44:42.502 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:44:42.502 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:44:42.502 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:44:42.502 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:44:42.502 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:44:42.502 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:44:42.502 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.485+0000 7f60aaca1640 1 librados: starting msgr at 2024-09-16T03:44:42.502 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.485+0000 7f60aaca1640 1 librados: starting objecter 2024-09-16T03:44:42.502 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.485+0000 7f60aaca1640 1 Processor -- start 2024-09-16T03:44:42.502 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.485+0000 7f60aaca1640 1 -- start start 2024-09-16T03:44:42.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.485+0000 7f60aaca1640 1 librados: setting wanted keys 2024-09-16T03:44:42.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.485+0000 7f60aaca1640 1 librados: calling monclient init 2024-09-16T03:44:42.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.485+0000 7f60aaca1640 10 monclient: init 2024-09-16T03:44:42.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.485+0000 7f60aaca1640 10 monclient: _reopen_session rank -1 2024-09-16T03:44:42.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.485+0000 7f60aaca1640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:44:42.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.485+0000 7f60aaca1640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f60a406db90 0x7f60a413a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:44:42.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.485+0000 7f60aaca1640 10 monclient(hunting): picked mon.noname-b con 0x7f60a406db90 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:44:42.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.485+0000 7f60aaca1640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f60a40a37f0 0x7f60a413ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:44:42.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.485+0000 7f60aaca1640 10 monclient(hunting): picked mon.noname-a con 0x7f60a40a37f0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:44:42.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.485+0000 7f60aaca1640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f60a4136e10 0x7f60a413b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:44:42.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.485+0000 7f60aaca1640 10 monclient(hunting): picked mon.noname-c con 0x7f60a4136e10 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:44:42.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.485+0000 7f60aaca1640 10 monclient(hunting): start opening mon connection 2024-09-16T03:44:42.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.485+0000 7f60aaca1640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f60a4140bd0 con 0x7f60a40a37f0 2024-09-16T03:44:42.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.485+0000 7f60aaca1640 10 monclient(hunting): start opening mon connection 2024-09-16T03:44:42.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.485+0000 7f60aaca1640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f60a4137c80 con 0x7f60a406db90 2024-09-16T03:44:42.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.485+0000 7f60aaca1640 10 monclient(hunting): start opening mon connection 2024-09-16T03:44:42.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.485+0000 7f60aaca1640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f60a4140d50 con 0x7f60a4136e10 2024-09-16T03:44:42.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.485+0000 7f60aaca1640 10 monclient(hunting): _renew_subs 2024-09-16T03:44:42.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.485+0000 7f60aaca1640 10 monclient(hunting): authenticate will time out at 1129.457350s 2024-09-16T03:44:42.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.485+0000 7f60a3fff640 1 -- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f60a40a37f0 msgr2=0x7f60a413ac70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.42:3300/0 2024-09-16T03:44:42.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.485+0000 7f60a3fff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f60a40a37f0 0x7f60a413ac70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:44:42.505 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.485+0000 7f60a923e640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f60a4136e10 0x7f60a413b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:44:42.505 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.485+0000 7f60a923e640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f60a4136e10 0x7f60a413b1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:39254/0 (socket says 172.21.15.42:39254) 2024-09-16T03:44:42.505 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.485+0000 7f60a923e640 1 -- 172.21.15.42:0/3708032650 learned_addr learned my addr 172.21.15.42:0/3708032650 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:44:42.505 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.485+0000 7f60a923e640 10 monclient(hunting): get_auth_request con 0x7f60a4136e10 auth_method 0 2024-09-16T03:44:42.505 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.485+0000 7f60a923e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:44:42.505 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.485+0000 7f60a923e640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:44:42.505 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.485+0000 7f60a923e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:44:42.505 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.485+0000 7f60a923e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:44:42.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.485+0000 7f60a923e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:44:42.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.485+0000 7f60a923e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:44:42.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.485+0000 7f60a8a3d640 1 --2- 172.21.15.42:0/3708032650 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f60a406db90 0x7f60a413a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:44:42.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.485+0000 7f60a8a3d640 10 monclient(hunting): get_auth_request con 0x7f60a406db90 auth_method 0 2024-09-16T03:44:42.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.485+0000 7f60a8a3d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:44:42.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.485+0000 7f60a8a3d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:44:42.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.485+0000 7f60a8a3d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:44:42.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.485+0000 7f60a923e640 10 monclient(hunting): handle_auth_done global_id 13118 payload 824 2024-09-16T03:44:42.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.485+0000 7f60a923e640 1 -- 172.21.15.42:0/3708032650 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f60a406db90 msgr2=0x7f60a413a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:44:42.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.485+0000 7f60a923e640 1 --2- 172.21.15.42:0/3708032650 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f60a406db90 0x7f60a413a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:44:42.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.485+0000 7f60a923e640 1 -- 172.21.15.42:0/3708032650 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f60a40a37f0 msgr2=0x7f60a413ac70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:44:42.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.485+0000 7f60a923e640 1 --2- 172.21.15.42:0/3708032650 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f60a40a37f0 0x7f60a413ac70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:44:42.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.485+0000 7f60a923e640 10 monclient: _finish_hunting 0 2024-09-16T03:44:42.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.485+0000 7f60a923e640 1 monclient: found mon.noname-c 2024-09-16T03:44:42.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.485+0000 7f60a923e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:44:42.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.485+0000 7f60a923e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:44:42.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.485+0000 7f60a923e640 1 -- 172.21.15.42:0/3708032650 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60a413c340 con 0x7f60a4136e10 2024-09-16T03:44:42.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.485+0000 7f60a923e640 1 --2- 172.21.15.42:0/3708032650 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f60a4136e10 0x7f60a413b1b0 secure :-1 s=READY pgs=2297 cs=0 l=1 rev1=1 crypto rx=0x7f609c00be60 tx=0x7f609c00bf50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:44:42.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.485+0000 7f60a17fa640 10 client.?.objecter ms_handle_connect 0x7f60a4136e10 2024-09-16T03:44:42.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.485+0000 7f60a17fa640 10 client.?.objecter resend_mon_ops 2024-09-16T03:44:42.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.485+0000 7f60a17fa640 1 -- 172.21.15.42:0/3708032650 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f609c016430 con 0x7f60a4136e10 2024-09-16T03:44:42.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.485+0000 7f60a17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:44:42.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.485+0000 7f60a17fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:44:42.508 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.485+0000 7f60a17fa640 10 monclient: dump: 2024-09-16T03:44:42.508 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:44:42.508 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:44:42.508 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:42.508 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:42.508 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:44:42.509 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:44:42.509 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:44:42.509 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:44:42.509 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:44:42.509 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:44:42.509 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.485+0000 7f60a17fa640 10 monclient: _finish_auth 0 2024-09-16T03:44:42.509 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.485+0000 7f60a17fa640 10 monclient: _check_auth_tickets 2024-09-16T03:44:42.509 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.485+0000 7f60a17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:44:42.509 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.485+0000 7f60aaca1640 5 monclient: authenticate success, global_id 13118 2024-09-16T03:44:42.509 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.485+0000 7f60a17fa640 1 -- 172.21.15.42:0/3708032650 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f609c010030 con 0x7f60a4136e10 2024-09-16T03:44:42.509 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.485+0000 7f60aaca1640 10 monclient: _renew_subs 2024-09-16T03:44:42.509 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.485+0000 7f60aaca1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:44:42.509 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.485+0000 7f60aaca1640 1 -- 172.21.15.42:0/3708032650 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60a406f6a0 con 0x7f60a4136e10 2024-09-16T03:44:42.509 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.485+0000 7f60a17fa640 10 monclient: handle_config config(0 keys) 2024-09-16T03:44:42.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.485+0000 7f60a17fa640 1 -- 172.21.15.42:0/3708032650 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f609c016850 con 0x7f60a4136e10 2024-09-16T03:44:42.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.485+0000 7f60a17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:44:42.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.485+0000 7f60a17fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:44:42.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.485+0000 7f60a17fa640 10 monclient: dump: 2024-09-16T03:44:42.510 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:44:42.510 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:44:42.510 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:42.510 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:42.510 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:44:42.510 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:44:42.510 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:44:42.510 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:44:42.510 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:44:42.510 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:44:42.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.485+0000 7f60a2ffd640 4 set_mon_vals no callback set 2024-09-16T03:44:42.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.485+0000 7f60aaca1640 10 client.13118.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:44:42.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.485+0000 7f60aaca1640 10 monclient: _renew_subs 2024-09-16T03:44:42.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.485+0000 7f60aaca1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:44:42.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.485+0000 7f60aaca1640 1 -- 172.21.15.42:0/3708032650 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f60a40a8a00 con 0x7f60a4136e10 2024-09-16T03:44:42.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.485+0000 7f60aaca1640 1 librados: init done 2024-09-16T03:44:42.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.485+0000 7f60a17fa640 1 -- 172.21.15.42:0/3708032650 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f609c0153e0 con 0x7f60a4136e10 2024-09-16T03:44:42.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.485+0000 7f60aaca1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:44:42.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.485+0000 7f60aaca1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:44:42.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.485+0000 7f60aaca1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:44:42.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.485+0000 7f60aaca1640 1 -- 172.21.15.42:0/3708032650 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f60a40a3c00 con 0x7f60a4136e10 2024-09-16T03:44:42.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.485+0000 7f60a17fa640 1 --2- 172.21.15.42:0/3708032650 >> v2:172.21.15.154:6812/254845342 conn(0x7f6084037810 0x7f6084039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:44:42.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.485+0000 7f60a17fa640 1 -- 172.21.15.42:0/3708032650 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(57..57 src has 1..57) ==== 10244+0+0 (secure 0 0 0) 0x7f609c04a5f0 con 0x7f60a4136e10 2024-09-16T03:44:42.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.485+0000 7f60a17fa640 10 client.13118.objecter ms_dispatch 0x7f60a4000ba0 osd_map(57..57 src has 1..57) 2024-09-16T03:44:42.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.485+0000 7f60a17fa640 3 client.13118.objecter handle_osd_map got epochs [57,57] > 0 2024-09-16T03:44:42.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.485+0000 7f60a17fa640 3 client.13118.objecter handle_osd_map decoding full epoch 57 2024-09-16T03:44:42.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.485+0000 7f60a17fa640 20 client.13118.objecter dump_active .. 0 homeless 2024-09-16T03:44:42.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.485+0000 7f60a8a3d640 1 --2- 172.21.15.42:0/3708032650 >> v2:172.21.15.154:6812/254845342 conn(0x7f6084037810 0x7f6084039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:44:42.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.485+0000 7f60a8a3d640 10 monclient: get_auth_request con 0x7f6084037810 auth_method 0 2024-09-16T03:44:42.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.489+0000 7f60a8a3d640 1 --2- 172.21.15.42:0/3708032650 >> v2:172.21.15.154:6812/254845342 conn(0x7f6084037810 0x7f6084039cd0 secure :-1 s=READY pgs=3308 cs=0 l=1 rev1=1 crypto rx=0x7f6094006fd0 tx=0x7f6094008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:44:42.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.489+0000 7f60a17fa640 10 client.13118.objecter ms_handle_connect 0x7f6084037810 2024-09-16T03:44:42.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.493+0000 7f60a17fa640 1 -- 172.21.15.42:0/3708032650 <== mon.2 v2:172.21.15.42:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f60a413c340 con 0x7f60a4136e10 2024-09-16T03:44:42.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.493+0000 7f60a17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:44:42.513 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.493+0000 7f60a17fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:44:42.718 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.713+0000 7f60aaca1640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:44:42.719 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.713+0000 7f60aaca1640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:44:42.719 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.713+0000 7f60aaca1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:44:42.719 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.713+0000 7f60aaca1640 1 -- 172.21.15.42:0/3708032650 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f60a409cd00 con 0x7f60a4136e10 2024-09-16T03:44:42.719 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.713+0000 7f60a17fa640 1 -- 172.21.15.42:0/3708032650 <== mon.2 v2:172.21.15.42:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7f609c047aa0 con 0x7f60a4136e10 2024-09-16T03:44:42.719 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.713+0000 7f60a17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:44:42.719 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.713+0000 7f60a17fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:44:42.720 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":22,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:44:42.722 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.717+0000 7f60aaca1640 10 librados: watch_flush enter 2024-09-16T03:44:42.722 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.717+0000 7f60aaca1640 10 librados: watch_flush exit 2024-09-16T03:44:42.722 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.717+0000 7f60aaca1640 20 client.13118.objecter shutdown clearing up homeless session... 2024-09-16T03:44:42.722 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.717+0000 7f60aaca1640 10 client.13118.objecter successfully canceled tick 2024-09-16T03:44:42.722 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.717+0000 7f60aaca1640 1 -- 172.21.15.42:0/3708032650 >> v2:172.21.15.154:6812/254845342 conn(0x7f6084037810 msgr2=0x7f6084039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:44:42.722 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.717+0000 7f60aaca1640 1 --2- 172.21.15.42:0/3708032650 >> v2:172.21.15.154:6812/254845342 conn(0x7f6084037810 0x7f6084039cd0 secure :-1 s=READY pgs=3308 cs=0 l=1 rev1=1 crypto rx=0x7f6094006fd0 tx=0x7f6094008040 comp rx=0 tx=0).stop 2024-09-16T03:44:42.722 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.717+0000 7f60aaca1640 10 monclient: shutdown 2024-09-16T03:44:42.722 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.717+0000 7f60aaca1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:44:42.722 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.717+0000 7f60aaca1640 1 -- 172.21.15.42:0/3708032650 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f60a4136e10 msgr2=0x7f60a413b1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:44:42.722 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.717+0000 7f60aaca1640 1 --2- 172.21.15.42:0/3708032650 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f60a4136e10 0x7f60a413b1b0 secure :-1 s=READY pgs=2297 cs=0 l=1 rev1=1 crypto rx=0x7f609c00be60 tx=0x7f609c00bf50 comp rx=0 tx=0).stop 2024-09-16T03:44:42.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.717+0000 7f60aaca1640 1 -- 172.21.15.42:0/3708032650 shutdown_connections 2024-09-16T03:44:42.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.717+0000 7f60aaca1640 1 --2- 172.21.15.42:0/3708032650 >> v2:172.21.15.154:6812/254845342 conn(0x7f6084037810 0x7f6084039cd0 unknown :-1 s=CLOSED pgs=3308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:44:42.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.717+0000 7f60aaca1640 1 --2- 172.21.15.42:0/3708032650 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f60a4136e10 0x7f60a413b1b0 unknown :-1 s=CLOSED pgs=2297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:44:42.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.717+0000 7f60aaca1640 1 --2- 172.21.15.42:0/3708032650 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f60a40a37f0 0x7f60a413ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:44:42.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.717+0000 7f60aaca1640 1 --2- 172.21.15.42:0/3708032650 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f60a406db90 0x7f60a413a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:44:42.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.717+0000 7f60aaca1640 1 -- 172.21.15.42:0/3708032650 >> 172.21.15.42:0/3708032650 conn(0x7f60a40a9fb0 msgr2=0x7f60a41308b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:44:42.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.717+0000 7f60aaca1640 1 -- 172.21.15.42:0/3708032650 shutdown_connections 2024-09-16T03:44:42.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.717+0000 7f60aaca1640 1 -- 172.21.15.42:0/3708032650 wait complete. 2024-09-16T03:44:42.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:42.717+0000 7f60aaca1640 1 librados: shutdown 2024-09-16T03:44:45.739 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:44:45.874 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:45.869+0000 7fbe25705640 10 monclient: get_monmap_and_config 2024-09-16T03:44:45.874 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:45.869+0000 7fbe25705640 10 monclient: build_initial_monmap 2024-09-16T03:44:45.874 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:45.869+0000 7fbe25705640 10 monclient: monmap: 2024-09-16T03:44:45.874 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:44:45.874 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:44:45.874 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:44:45.873179+0000 2024-09-16T03:44:45.874 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:44:45.873179+0000 2024-09-16T03:44:45.874 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:44:45.874 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:44:45.874 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:44:45.874 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:44:45.874 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:44:45.875 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:44:45.875 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:44:45.875 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:44:45.876 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:45.869+0000 7fbe25705640 1 Processor -- start 2024-09-16T03:44:45.876 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:45.869+0000 7fbe25705640 1 -- start start 2024-09-16T03:44:45.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:45.869+0000 7fbe25705640 10 monclient: init 2024-09-16T03:44:45.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:45.869+0000 7fbe25705640 10 monclient: _reopen_session rank -1 2024-09-16T03:44:45.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:45.869+0000 7fbe25705640 10 monclient: _add_conns ranks=[4,0,1,2,3] 2024-09-16T03:44:45.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:45.869+0000 7fbe25705640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fbe2006db90 0x7fbe201340c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:44:45.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:45.869+0000 7fbe25705640 10 monclient(hunting): picked mon.noname-c con 0x7fbe2006db90 addr v2:172.21.15.154:3300/0 2024-09-16T03:44:45.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:45.869+0000 7fbe25705640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbe200a37f0 addr v1:172.21.15.42:6789/0 2024-09-16T03:44:45.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:45.869+0000 7fbe25705640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbe200a3c00 addr v1:172.21.15.154:6789/0 2024-09-16T03:44:45.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:45.869+0000 7fbe25705640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbe2006ece0 con 0x7fbe200a37f0 2024-09-16T03:44:45.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:45.869+0000 7fbe25705640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbe2009bf80 con 0x7fbe200a3c00 2024-09-16T03:44:45.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:45.869+0000 7fbe25705640 10 monclient(hunting): start opening mon connection 2024-09-16T03:44:45.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:45.869+0000 7fbe25705640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fbe2006df70 con 0x7fbe2006db90 2024-09-16T03:44:45.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:45.869+0000 7fbe25705640 10 monclient(hunting): _renew_subs 2024-09-16T03:44:45.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:45.869+0000 7fbe25705640 10 monclient(hunting): authenticate will time out at 1132.842572s 2024-09-16T03:44:45.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:45.869+0000 7fbe1e7fc640 1 -- >> v1:172.21.15.42:6789/0 conn(0x7fbe200a37f0 legacy=0x7fbe20136e10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.42:6789/0 2024-09-16T03:44:45.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:45.869+0000 7fbe1effd640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fbe2006db90 0x7fbe201340c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:44:45.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:45.869+0000 7fbe1effd640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fbe2006db90 0x7fbe201340c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.42:58074/0 (socket says 172.21.15.42:58074) 2024-09-16T03:44:45.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:45.869+0000 7fbe1f7fe640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7fbe200a3c00 0x7fbe20137520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.42:35330/0 (socket says 172.21.15.42:35330) 2024-09-16T03:44:45.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:45.869+0000 7fbe1effd640 1 -- 172.21.15.42:0/1426970928 learned_addr learned my addr 172.21.15.42:0/1426970928 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:44:45.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:45.869+0000 7fbe1effd640 10 monclient(hunting): get_auth_request con 0x7fbe2006db90 auth_method 0 2024-09-16T03:44:45.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:45.869+0000 7fbe1effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:44:45.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:45.869+0000 7fbe1effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:44:45.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:45.869+0000 7fbe1effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:44:45.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:45.869+0000 7fbe1effd640 1 -- 172.21.15.42:0/1426970928 >> v2:172.21.15.154:3300/0 conn(0x7fbe2006db90 msgr2=0x7fbe201340c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T03:44:45.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:45.869+0000 7fbe1effd640 1 -- 172.21.15.42:0/1426970928 >> v2:172.21.15.154:3300/0 conn(0x7fbe2006db90 msgr2=0x7fbe201340c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:44:45.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:45.869+0000 7fbe1effd640 1 --2- 172.21.15.42:0/1426970928 >> v2:172.21.15.154:3300/0 conn(0x7fbe2006db90 0x7fbe201340c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:44:45.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:45.869+0000 7fbe1effd640 1 --2- 172.21.15.42:0/1426970928 >> v2:172.21.15.154:3300/0 conn(0x7fbe2006db90 0x7fbe201340c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:44:46.077 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:46.073+0000 7fbe1e7fc640 1 -- 172.21.15.42:0/1426970928 >> v1:172.21.15.42:6789/0 conn(0x7fbe200a37f0 legacy=0x7fbe20136e10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.42:6789/0 2024-09-16T03:44:46.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:46.073+0000 7fbe1effd640 1 --2- 172.21.15.42:0/1426970928 >> v2:172.21.15.154:3300/0 conn(0x7fbe2006db90 0x7fbe201340c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:44:46.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:46.073+0000 7fbe1effd640 10 monclient(hunting): get_auth_request con 0x7fbe2006db90 auth_method 0 2024-09-16T03:44:46.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:46.073+0000 7fbe1effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:44:46.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:46.073+0000 7fbe1effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:44:46.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:46.073+0000 7fbe1effd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:44:46.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:46.073+0000 7fbe1effd640 1 -- 172.21.15.42:0/1426970928 >> v2:172.21.15.154:3300/0 conn(0x7fbe2006db90 msgr2=0x7fbe201340c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T03:44:46.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:46.073+0000 7fbe1effd640 1 -- 172.21.15.42:0/1426970928 >> v2:172.21.15.154:3300/0 conn(0x7fbe2006db90 msgr2=0x7fbe201340c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:44:46.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:46.073+0000 7fbe1effd640 1 --2- 172.21.15.42:0/1426970928 >> v2:172.21.15.154:3300/0 conn(0x7fbe2006db90 0x7fbe201340c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:44:46.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:46.073+0000 7fbe1effd640 1 --2- 172.21.15.42:0/1426970928 >> v2:172.21.15.154:3300/0 conn(0x7fbe2006db90 0x7fbe201340c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T03:44:46.481 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:46.477+0000 7fbe1e7fc640 1 -- 172.21.15.42:0/1426970928 >> v1:172.21.15.42:6789/0 conn(0x7fbe200a37f0 legacy=0x7fbe20136e10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.42:6789/0 2024-09-16T03:44:46.481 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:46.477+0000 7fbe1effd640 1 --2- 172.21.15.42:0/1426970928 >> v2:172.21.15.154:3300/0 conn(0x7fbe2006db90 0x7fbe201340c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:44:46.482 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:46.477+0000 7fbe1effd640 10 monclient(hunting): get_auth_request con 0x7fbe2006db90 auth_method 0 2024-09-16T03:44:46.482 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:46.477+0000 7fbe1effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:44:46.482 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:46.477+0000 7fbe1effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:44:46.482 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:46.477+0000 7fbe1effd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:44:46.482 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:46.477+0000 7fbe1effd640 1 -- 172.21.15.42:0/1426970928 >> v2:172.21.15.154:3300/0 conn(0x7fbe2006db90 msgr2=0x7fbe201340c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T03:44:46.482 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:46.477+0000 7fbe1effd640 1 -- 172.21.15.42:0/1426970928 >> v2:172.21.15.154:3300/0 conn(0x7fbe2006db90 msgr2=0x7fbe201340c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:44:46.482 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:46.477+0000 7fbe1effd640 1 --2- 172.21.15.42:0/1426970928 >> v2:172.21.15.154:3300/0 conn(0x7fbe2006db90 0x7fbe201340c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:44:46.482 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:46.477+0000 7fbe1effd640 1 --2- 172.21.15.42:0/1426970928 >> v2:172.21.15.154:3300/0 conn(0x7fbe2006db90 0x7fbe201340c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T03:44:47.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:47.281+0000 7fbe1e7fc640 1 -- 172.21.15.42:0/1426970928 >> v1:172.21.15.42:6789/0 conn(0x7fbe200a37f0 legacy=0x7fbe20136e10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.42:6789/0 2024-09-16T03:44:47.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:47.281+0000 7fbe1effd640 1 --2- 172.21.15.42:0/1426970928 >> v2:172.21.15.154:3300/0 conn(0x7fbe2006db90 0x7fbe201340c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:44:47.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:47.281+0000 7fbe1effd640 10 monclient(hunting): get_auth_request con 0x7fbe2006db90 auth_method 0 2024-09-16T03:44:47.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:47.281+0000 7fbe1effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:44:47.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:47.281+0000 7fbe1effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:44:47.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:47.281+0000 7fbe1effd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:44:47.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:47.281+0000 7fbe1effd640 1 -- 172.21.15.42:0/1426970928 >> v2:172.21.15.154:3300/0 conn(0x7fbe2006db90 msgr2=0x7fbe201340c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T03:44:47.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:47.281+0000 7fbe1effd640 1 -- 172.21.15.42:0/1426970928 >> v2:172.21.15.154:3300/0 conn(0x7fbe2006db90 msgr2=0x7fbe201340c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:44:47.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:47.281+0000 7fbe1effd640 1 --2- 172.21.15.42:0/1426970928 >> v2:172.21.15.154:3300/0 conn(0x7fbe2006db90 0x7fbe201340c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:44:47.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:47.281+0000 7fbe1effd640 1 --2- 172.21.15.42:0/1426970928 >> v2:172.21.15.154:3300/0 conn(0x7fbe2006db90 0x7fbe201340c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T03:44:48.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:48.873+0000 7fbe1cff9640 10 monclient(hunting): tick 2024-09-16T03:44:48.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:48.873+0000 7fbe1cff9640 10 monclient(hunting): _check_auth_tickets 2024-09-16T03:44:48.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:48.873+0000 7fbe1cff9640 1 monclient(hunting): continuing hunt 2024-09-16T03:44:48.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:48.873+0000 7fbe1cff9640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T03:44:48.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:48.873+0000 7fbe1cff9640 1 -- 172.21.15.42:0/1426970928 >> v2:172.21.15.154:3300/0 conn(0x7fbe2006db90 msgr2=0x7fbe201340c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:44:48.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:48.873+0000 7fbe1cff9640 1 --2- 172.21.15.42:0/1426970928 >> v2:172.21.15.154:3300/0 conn(0x7fbe2006db90 0x7fbe201340c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:44:48.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:48.873+0000 7fbe1cff9640 1 -- 172.21.15.42:0/1426970928 >> v1:172.21.15.154:6789/0 conn(0x7fbe200a3c00 legacy=0x7fbe20137520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:44:48.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:48.873+0000 7fbe1cff9640 1 -- 172.21.15.42:0/1426970928 >> v1:172.21.15.42:6789/0 conn(0x7fbe200a37f0 legacy=0x7fbe20136e10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:44:48.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:48.873+0000 7fbe1cff9640 10 monclient: _add_conns ranks=[3,2] 2024-09-16T03:44:48.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:48.873+0000 7fbe1cff9640 1 --2- 172.21.15.42:0/1426970928 >> v2:172.21.15.42:3300/0 conn(0x7fbe0c0036d0 0x7fbe0c005b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:44:48.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:48.873+0000 7fbe1cff9640 10 monclient(hunting): picked mon.noname-a con 0x7fbe0c0036d0 addr v2:172.21.15.42:3300/0 2024-09-16T03:44:48.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:48.873+0000 7fbe1cff9640 1 --2- 172.21.15.42:0/1426970928 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbe0c0061c0 0x7fbe0c008660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:44:48.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:48.873+0000 7fbe1cff9640 10 monclient(hunting): picked mon.noname-b con 0x7fbe0c0061c0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:44:48.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:48.873+0000 7fbe1cff9640 10 monclient(hunting): start opening mon connection 2024-09-16T03:44:48.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:48.873+0000 7fbe1cff9640 1 -- 172.21.15.42:0/1426970928 --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7fbe2006df70 con 0x7fbe0c0036d0 2024-09-16T03:44:48.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:48.873+0000 7fbe1cff9640 10 monclient(hunting): start opening mon connection 2024-09-16T03:44:48.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:48.873+0000 7fbe1cff9640 1 -- 172.21.15.42:0/1426970928 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fbe0c008d90 con 0x7fbe0c0061c0 2024-09-16T03:44:48.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:48.873+0000 7fbe1cff9640 10 monclient(hunting): _renew_subs 2024-09-16T03:44:48.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:48.873+0000 7fbe1effd640 1 -- 172.21.15.42:0/1426970928 >> v2:172.21.15.42:3300/0 conn(0x7fbe0c0036d0 msgr2=0x7fbe0c005b30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3300/0 2024-09-16T03:44:48.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:48.873+0000 7fbe1effd640 1 --2- 172.21.15.42:0/1426970928 >> v2:172.21.15.42:3300/0 conn(0x7fbe0c0036d0 0x7fbe0c005b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:44:48.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:48.873+0000 7fbe1e7fc640 1 --2- 172.21.15.42:0/1426970928 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbe0c0061c0 0x7fbe0c008660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:44:48.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:48.873+0000 7fbe1e7fc640 10 monclient(hunting): get_auth_request con 0x7fbe0c0061c0 auth_method 0 2024-09-16T03:44:48.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:48.873+0000 7fbe1e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:44:48.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:48.873+0000 7fbe1e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:44:48.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:48.873+0000 7fbe1e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:44:48.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:48.873+0000 7fbe1e7fc640 1 -- 172.21.15.42:0/1426970928 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbe0c0061c0 msgr2=0x7fbe0c008660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T03:44:48.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:48.873+0000 7fbe1e7fc640 1 -- 172.21.15.42:0/1426970928 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbe0c0061c0 msgr2=0x7fbe0c008660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:44:48.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:48.873+0000 7fbe1e7fc640 1 --2- 172.21.15.42:0/1426970928 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbe0c0061c0 0x7fbe0c008660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:44:48.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:48.873+0000 7fbe1e7fc640 1 --2- 172.21.15.42:0/1426970928 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbe0c0061c0 0x7fbe0c008660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:44:49.081 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:49.077+0000 7fbe1effd640 1 -- 172.21.15.42:0/1426970928 >> v2:172.21.15.42:3300/0 conn(0x7fbe0c0036d0 msgr2=0x7fbe0c005b30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3300/0 2024-09-16T03:44:49.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:49.077+0000 7fbe1effd640 1 --2- 172.21.15.42:0/1426970928 >> v2:172.21.15.42:3300/0 conn(0x7fbe0c0036d0 0x7fbe0c005b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T03:44:49.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:49.077+0000 7fbe1e7fc640 1 --2- 172.21.15.42:0/1426970928 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbe0c0061c0 0x7fbe0c008660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:44:49.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:49.077+0000 7fbe1e7fc640 10 monclient(hunting): get_auth_request con 0x7fbe0c0061c0 auth_method 0 2024-09-16T03:44:49.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:49.077+0000 7fbe1e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:44:49.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:49.077+0000 7fbe1e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:44:49.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:49.077+0000 7fbe1e7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:44:49.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:49.077+0000 7fbe1e7fc640 1 -- 172.21.15.42:0/1426970928 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbe0c0061c0 msgr2=0x7fbe0c008660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T03:44:49.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:49.077+0000 7fbe1e7fc640 1 -- 172.21.15.42:0/1426970928 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbe0c0061c0 msgr2=0x7fbe0c008660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:44:49.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:49.077+0000 7fbe1e7fc640 1 --2- 172.21.15.42:0/1426970928 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbe0c0061c0 0x7fbe0c008660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:44:49.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:49.077+0000 7fbe1e7fc640 1 --2- 172.21.15.42:0/1426970928 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbe0c0061c0 0x7fbe0c008660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T03:44:49.485 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:49.481+0000 7fbe1effd640 1 -- 172.21.15.42:0/1426970928 >> v2:172.21.15.42:3300/0 conn(0x7fbe0c0036d0 msgr2=0x7fbe0c005b30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3300/0 2024-09-16T03:44:49.485 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:49.481+0000 7fbe1effd640 1 --2- 172.21.15.42:0/1426970928 >> v2:172.21.15.42:3300/0 conn(0x7fbe0c0036d0 0x7fbe0c005b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T03:44:49.485 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:49.481+0000 7fbe1e7fc640 1 --2- 172.21.15.42:0/1426970928 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbe0c0061c0 0x7fbe0c008660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:44:49.486 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:49.481+0000 7fbe1e7fc640 10 monclient(hunting): get_auth_request con 0x7fbe0c0061c0 auth_method 0 2024-09-16T03:44:49.486 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:49.481+0000 7fbe1e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:44:49.486 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:49.481+0000 7fbe1e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:44:49.486 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:49.481+0000 7fbe1e7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:44:49.486 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:49.481+0000 7fbe1e7fc640 1 -- 172.21.15.42:0/1426970928 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbe0c0061c0 msgr2=0x7fbe0c008660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T03:44:49.486 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:49.481+0000 7fbe1e7fc640 1 -- 172.21.15.42:0/1426970928 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbe0c0061c0 msgr2=0x7fbe0c008660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:44:49.486 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:49.481+0000 7fbe1e7fc640 1 --2- 172.21.15.42:0/1426970928 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbe0c0061c0 0x7fbe0c008660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:44:49.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:49.481+0000 7fbe1e7fc640 1 --2- 172.21.15.42:0/1426970928 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbe0c0061c0 0x7fbe0c008660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T03:44:50.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:50.285+0000 7fbe1effd640 1 -- 172.21.15.42:0/1426970928 >> v2:172.21.15.42:3300/0 conn(0x7fbe0c0036d0 msgr2=0x7fbe0c005b30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3300/0 2024-09-16T03:44:50.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:50.285+0000 7fbe1effd640 1 --2- 172.21.15.42:0/1426970928 >> v2:172.21.15.42:3300/0 conn(0x7fbe0c0036d0 0x7fbe0c005b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T03:44:50.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:50.285+0000 7fbe1e7fc640 1 --2- 172.21.15.42:0/1426970928 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbe0c0061c0 0x7fbe0c008660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:44:50.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:50.285+0000 7fbe1e7fc640 10 monclient(hunting): get_auth_request con 0x7fbe0c0061c0 auth_method 0 2024-09-16T03:44:50.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:50.285+0000 7fbe1e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:44:50.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:50.285+0000 7fbe1e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:44:50.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:50.285+0000 7fbe1e7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:44:50.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:50.285+0000 7fbe1e7fc640 1 -- 172.21.15.42:0/1426970928 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbe0c0061c0 msgr2=0x7fbe0c008660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T03:44:50.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:50.285+0000 7fbe1e7fc640 1 -- 172.21.15.42:0/1426970928 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbe0c0061c0 msgr2=0x7fbe0c008660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:44:50.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:50.285+0000 7fbe1e7fc640 1 --2- 172.21.15.42:0/1426970928 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbe0c0061c0 0x7fbe0c008660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:44:50.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:50.285+0000 7fbe1e7fc640 1 --2- 172.21.15.42:0/1426970928 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbe0c0061c0 0x7fbe0c008660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T03:44:50.604 INFO:tasks.workunit.client.0.smithi154.stdout:fy_pp: flushed 2024-09-16T03:44:50.604 INFO:tasks.workunit.client.0.smithi154.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/0 (3010 ms) 2024-09-16T03:44:50.604 INFO:tasks.workunit.client.0.smithi154.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/1 2024-09-16T03:44:50.605 INFO:tasks.workunit.client.0.smithi154.stdout: api_watch_notify_pp: trying... 2024-09-16T03:44:50.605 INFO:tasks.workunit.client.0.smithi154.stdout: api_watch_notify_pp: handle_notify cookie 94745269682944 notify_id 154618822661 notifier_gid 4865 2024-09-16T03:44:50.605 INFO:tasks.workunit.client.0.smithi154.stdout: api_watch_notify_pp: timed out 2024-09-16T03:44:50.605 INFO:tasks.workunit.client.0.smithi154.stdout: api_watch_notify_pp: flushing 2024-09-16T03:44:50.605 INFO:tasks.workunit.client.0.smithi154.stdout: api_watch_notify_pp: flushed 2024-09-16T03:44:50.605 INFO:tasks.workunit.client.0.smithi154.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/1 (3008 ms) 2024-09-16T03:44:50.605 INFO:tasks.workunit.client.0.smithi154.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/0 2024-09-16T03:44:50.605 INFO:tasks.workunit.client.0.smithi154.stdout: api_watch_notify_pp: List watches 2024-09-16T03:44:50.605 INFO:tasks.workunit.client.0.smithi154.stdout: api_watch_notify_pp: notify2 2024-09-16T03:44:50.605 INFO:tasks.workunit.client.0.smithi154.stdout: api_watch_notify_pp: handle_notify cookie 94745269682944 notify_id 167503724552 notifier_gid 4865 2024-09-16T03:44:50.605 INFO:tasks.workunit.client.0.smithi154.stdout: api_watch_notify_pp: notify2 done 2024-09-16T03:44:50.605 INFO:tasks.workunit.client.0.smithi154.stdout: api_watch_notify_pp: watch_check 2024-09-16T03:44:50.605 INFO:tasks.workunit.client.0.smithi154.stdout: api_watch_notify_pp: unwatch2 2024-09-16T03:44:50.606 INFO:tasks.workunit.client.0.smithi154.stdout: api_watch_notify_pp: flushing 2024-09-16T03:44:50.606 INFO:tasks.workunit.client.0.smithi154.stdout: api_watch_notify_pp: done 2024-09-16T03:44:50.606 INFO:tasks.workunit.client.0.smithi154.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/0 (3025 ms) 2024-09-16T03:44:50.606 INFO:tasks.workunit.client.0.smithi154.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/1 2024-09-16T03:44:50.606 INFO:tasks.workunit.client.0.smithi154.stdout: api_watch_notify_pp: List watches 2024-09-16T03:44:50.606 INFO:tasks.workunit.client.0.smithi154.stdout: api_watch_notify_pp: notify2 2024-09-16T03:44:50.606 INFO:tasks.workunit.client.0.smithi154.stdout: api_watch_notify_pp: handle_notify cookie 94745269675424 notify_id 180388626435 notifier_gid 4865 2024-09-16T03:44:50.606 INFO:tasks.workunit.client.0.smithi154.stdout: api_watch_notify_pp: notify2 done 2024-09-16T03:44:50.606 INFO:tasks.workunit.client.0.smithi154.stdout: api_watch_notify_pp: watch_check 2024-09-16T03:44:50.606 INFO:tasks.workunit.client.0.smithi154.stdout: api_watch_notify_pp: unwatch2 2024-09-16T03:44:50.606 INFO:tasks.workunit.client.0.smithi154.stdout: api_watch_notify_pp: flushing 2024-09-16T03:44:50.606 INFO:tasks.workunit.client.0.smithi154.stdout: api_watch_notify_pp: done 2024-09-16T03:44:50.606 INFO:tasks.workunit.client.0.smithi154.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/1 (3012 ms) 2024-09-16T03:44:50.607 INFO:tasks.workunit.client.0.smithi154.stdout: api_watch_notify_pp: [----------] 14 tests from LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP (16259 ms total) 2024-09-16T03:44:50.607 INFO:tasks.workunit.client.0.smithi154.stdout: api_watch_notify_pp: 2024-09-16T03:44:50.607 INFO:tasks.workunit.client.0.smithi154.stdout: api_watch_notify_pp: [----------] Global test environment tear-down 2024-09-16T03:44:50.607 INFO:tasks.workunit.client.0.smithi154.stdout: api_watch_notify_pp: [==========] 16 tests from 2 test suites ran. (80927 ms total) 2024-09-16T03:44:50.607 INFO:tasks.workunit.client.0.smithi154.stdout: api_watch_notify_pp: [ PASSED ] 16 tests. 2024-09-16T03:44:51.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.873+0000 7fbe1cff9640 10 monclient(hunting): tick 2024-09-16T03:44:51.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.873+0000 7fbe1cff9640 10 monclient(hunting): _check_auth_tickets 2024-09-16T03:44:51.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.873+0000 7fbe1cff9640 1 monclient(hunting): continuing hunt 2024-09-16T03:44:51.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.873+0000 7fbe1cff9640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T03:44:51.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.873+0000 7fbe1cff9640 1 -- 172.21.15.42:0/1426970928 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbe0c0061c0 msgr2=0x7fbe0c008660 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:44:51.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.873+0000 7fbe1cff9640 1 --2- 172.21.15.42:0/1426970928 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbe0c0061c0 0x7fbe0c008660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:44:51.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.873+0000 7fbe1cff9640 1 -- 172.21.15.42:0/1426970928 >> v2:172.21.15.42:3300/0 conn(0x7fbe0c0036d0 msgr2=0x7fbe0c005b30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:44:51.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.873+0000 7fbe1cff9640 1 --2- 172.21.15.42:0/1426970928 >> v2:172.21.15.42:3300/0 conn(0x7fbe0c0036d0 0x7fbe0c005b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:44:51.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.873+0000 7fbe1cff9640 10 monclient: _add_conns ranks=[4,3,0,1,2] 2024-09-16T03:44:51.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.873+0000 7fbe1f7fe640 1 -- 172.21.15.42:0/1426970928 reap_dead start 2024-09-16T03:44:51.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.873+0000 7fbe1cff9640 1 --2- 172.21.15.42:0/1426970928 >> v2:172.21.15.154:3300/0 conn(0x7fbe2006db90 0x7fbe0c00b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:44:51.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.873+0000 7fbe1cff9640 10 monclient(hunting): picked mon.noname-c con 0x7fbe2006db90 addr v2:172.21.15.154:3300/0 2024-09-16T03:44:51.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.873+0000 7fbe1cff9640 1 --2- 172.21.15.42:0/1426970928 >> v2:172.21.15.42:3300/0 conn(0x7fbe0c0036d0 0x7fbe0c005af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:44:51.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.873+0000 7fbe1cff9640 10 monclient(hunting): picked mon.noname-a con 0x7fbe0c0036d0 addr v2:172.21.15.42:3300/0 2024-09-16T03:44:51.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.873+0000 7fbe1cff9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbe0c006650 addr v1:172.21.15.42:6789/0 2024-09-16T03:44:51.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.873+0000 7fbe1cff9640 1 -- 172.21.15.42:0/1426970928 --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbe20137c30 con 0x7fbe0c006650 2024-09-16T03:44:51.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.873+0000 7fbe1cff9640 10 monclient(hunting): start opening mon connection 2024-09-16T03:44:51.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.873+0000 7fbe1cff9640 1 -- 172.21.15.42:0/1426970928 --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7fbe2006df70 con 0x7fbe0c0036d0 2024-09-16T03:44:51.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.873+0000 7fbe1cff9640 10 monclient(hunting): start opening mon connection 2024-09-16T03:44:51.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.873+0000 7fbe1cff9640 1 -- 172.21.15.42:0/1426970928 --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fbe0c008d90 con 0x7fbe2006db90 2024-09-16T03:44:51.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.873+0000 7fbe1cff9640 10 monclient(hunting): _renew_subs 2024-09-16T03:44:51.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.873+0000 7fbe1e7fc640 1 -- 172.21.15.42:0/1426970928 >> v2:172.21.15.42:3300/0 conn(0x7fbe0c0036d0 msgr2=0x7fbe0c005af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3300/0 2024-09-16T03:44:51.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.873+0000 7fbe1f7fe640 1 -- 172.21.15.42:0/1426970928 >> v1:172.21.15.42:6789/0 conn(0x7fbe0c006650 legacy=0x7fbe0c00b700 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.42:6789/0 2024-09-16T03:44:51.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.873+0000 7fbe1e7fc640 1 --2- 172.21.15.42:0/1426970928 >> v2:172.21.15.42:3300/0 conn(0x7fbe0c0036d0 0x7fbe0c005af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:44:51.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.881+0000 7fbe1effd640 1 --2- 172.21.15.42:0/1426970928 >> v2:172.21.15.154:3300/0 conn(0x7fbe2006db90 0x7fbe0c00b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:44:51.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.881+0000 7fbe1effd640 10 monclient(hunting): get_auth_request con 0x7fbe2006db90 auth_method 0 2024-09-16T03:44:51.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.881+0000 7fbe1effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:44:51.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.881+0000 7fbe1effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:44:51.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.881+0000 7fbe1effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:44:51.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.881+0000 7fbe1effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:44:51.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.881+0000 7fbe1effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:44:51.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.881+0000 7fbe1effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:44:51.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.881+0000 7fbe1effd640 10 monclient(hunting): handle_auth_done global_id 272446 payload 274 2024-09-16T03:44:51.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.881+0000 7fbe1effd640 1 -- 172.21.15.42:0/1426970928 >> v2:172.21.15.42:3300/0 conn(0x7fbe0c0036d0 msgr2=0x7fbe0c005af0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:44:51.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.881+0000 7fbe1effd640 1 --2- 172.21.15.42:0/1426970928 >> v2:172.21.15.42:3300/0 conn(0x7fbe0c0036d0 0x7fbe0c005af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:44:51.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.881+0000 7fbe1effd640 1 -- 172.21.15.42:0/1426970928 >> v1:172.21.15.42:6789/0 conn(0x7fbe0c006650 legacy=0x7fbe0c00b700 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:44:51.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.881+0000 7fbe1effd640 10 monclient: _finish_hunting 0 2024-09-16T03:44:51.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.881+0000 7fbe1effd640 1 monclient: found mon.noname-c 2024-09-16T03:44:51.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.881+0000 7fbe1effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:44:51.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.881+0000 7fbe1effd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.154:3300/0 2024-09-16T03:44:51.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.881+0000 7fbe1effd640 1 -- 172.21.15.42:0/1426970928 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe2006ece0 con 0x7fbe2006db90 2024-09-16T03:44:51.892 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.885+0000 7fbe1effd640 1 --2- 172.21.15.42:0/1426970928 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbe2006db90 0x7fbe0c00b1c0 secure :-1 s=READY pgs=2751 cs=0 l=1 rev1=1 crypto rx=0x7fbe2009bba0 tx=0x7fbe1001d370 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e6e88d7cbfa35a7a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:44:51.893 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.885+0000 7fbe1dffb640 1 -- 172.21.15.42:0/1426970928 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe100b3070 con 0x7fbe2006db90 2024-09-16T03:44:51.893 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.889+0000 7fbe1dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:44:51.893 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.889+0000 7fbe1dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:44:51.893 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.889+0000 7fbe1dffb640 10 monclient: dump: 2024-09-16T03:44:51.893 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:44:51.893 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:44:51.893 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:51.893 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:51.893 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:44:51.894 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:44:51.894 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:44:51.894 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:44:51.894 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:44:51.894 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:44:51.894 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.889+0000 7fbe1dffb640 10 monclient: _finish_auth 0 2024-09-16T03:44:51.894 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.889+0000 7fbe1dffb640 10 monclient: _check_auth_tickets 2024-09-16T03:44:51.894 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.889+0000 7fbe1dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:44:51.894 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.889+0000 7fbe1dffb640 1 -- 172.21.15.42:0/1426970928 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbe100ae410 con 0x7fbe2006db90 2024-09-16T03:44:51.894 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.889+0000 7fbe1dffb640 10 monclient: handle_config config(0 keys) 2024-09-16T03:44:51.894 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.889+0000 7fbe1dffb640 1 -- 172.21.15.42:0/1426970928 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe1000f040 con 0x7fbe2006db90 2024-09-16T03:44:51.894 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.889+0000 7fbe25705640 5 monclient: authenticate success, global_id 272446 2024-09-16T03:44:51.895 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.889+0000 7fbe25705640 10 monclient: get_monmap_and_config success 2024-09-16T03:44:51.895 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.889+0000 7fbe25705640 4 set_mon_vals no callback set 2024-09-16T03:44:51.895 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.889+0000 7fbe1dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:44:51.895 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.889+0000 7fbe1dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:44:51.895 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.889+0000 7fbe1dffb640 10 monclient: dump: 2024-09-16T03:44:51.895 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:44:51.895 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:44:51.895 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:51.895 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:51.895 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:44:51.895 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:44:51.895 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:44:51.896 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:44:51.896 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:44:51.896 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:44:51.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.889+0000 7fbe25705640 10 monclient: shutdown 2024-09-16T03:44:51.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.889+0000 7fbe25705640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:44:51.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.889+0000 7fbe25705640 1 -- 172.21.15.42:0/1426970928 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbe2006db90 msgr2=0x7fbe0c00b1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:44:51.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.889+0000 7fbe25705640 1 --2- 172.21.15.42:0/1426970928 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbe2006db90 0x7fbe0c00b1c0 secure :-1 s=READY pgs=2751 cs=0 l=1 rev1=1 crypto rx=0x7fbe2009bba0 tx=0x7fbe1001d370 comp rx=0 tx=0).stop 2024-09-16T03:44:51.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.889+0000 7fbe25705640 1 -- 172.21.15.42:0/1426970928 shutdown_connections 2024-09-16T03:44:51.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.889+0000 7fbe25705640 1 --2- 172.21.15.42:0/1426970928 >> v2:172.21.15.42:3300/0 conn(0x7fbe0c0036d0 0x7fbe0c005af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:44:51.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.889+0000 7fbe25705640 1 --2- 172.21.15.42:0/1426970928 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbe2006db90 0x7fbe0c00b1c0 unknown :-1 s=CLOSED pgs=2751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:44:51.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.889+0000 7fbe25705640 1 -- 172.21.15.42:0/1426970928 >> 172.21.15.42:0/1426970928 conn(0x7fbe200a9fb0 msgr2=0x7fbe200aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:44:51.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.889+0000 7fbe25705640 1 -- 172.21.15.42:0/1426970928 shutdown_connections 2024-09-16T03:44:51.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.889+0000 7fbe25705640 1 -- 172.21.15.42:0/1426970928 wait complete. 2024-09-16T03:44:51.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.889+0000 7fbe25705640 10 monclient: build_initial_monmap 2024-09-16T03:44:51.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.889+0000 7fbe25705640 10 monclient: monmap: 2024-09-16T03:44:51.902 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:44:51.902 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:44:51.902 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:44:51.902 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:44:51.902 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:44:51.902 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:44:51.902 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:44:51.902 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:44:51.902 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:44:51.903 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:44:51.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.889+0000 7fbe25705640 1 librados: starting msgr at 2024-09-16T03:44:51.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.889+0000 7fbe25705640 1 librados: starting objecter 2024-09-16T03:44:51.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.889+0000 7fbe25705640 1 Processor -- start 2024-09-16T03:44:51.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.889+0000 7fbe25705640 1 -- start start 2024-09-16T03:44:51.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.889+0000 7fbe25705640 1 librados: setting wanted keys 2024-09-16T03:44:51.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.889+0000 7fbe25705640 1 librados: calling monclient init 2024-09-16T03:44:51.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.889+0000 7fbe25705640 10 monclient: init 2024-09-16T03:44:51.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.889+0000 7fbe25705640 10 monclient: _reopen_session rank -1 2024-09-16T03:44:51.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.893+0000 7fbe25705640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T03:44:51.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.893+0000 7fbe25705640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbe0c0036d0 0x7fbe200a24c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:44:51.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.893+0000 7fbe25705640 10 monclient(hunting): picked mon.noname-b con 0x7fbe0c0036d0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:44:51.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.893+0000 7fbe25705640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbe0c006650 0x7fbe20130450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:44:51.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.893+0000 7fbe25705640 10 monclient(hunting): picked mon.noname-c con 0x7fbe0c006650 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:44:51.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.893+0000 7fbe25705640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbe2006db90 0x7fbe20130990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:44:51.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.893+0000 7fbe25705640 10 monclient(hunting): picked mon.noname-a con 0x7fbe2006db90 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:44:51.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.893+0000 7fbe25705640 10 monclient(hunting): start opening mon connection 2024-09-16T03:44:51.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.893+0000 7fbe25705640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fbe20137620 con 0x7fbe2006db90 2024-09-16T03:44:51.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.893+0000 7fbe25705640 10 monclient(hunting): start opening mon connection 2024-09-16T03:44:51.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.893+0000 7fbe25705640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fbe201377a0 con 0x7fbe0c0036d0 2024-09-16T03:44:51.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.893+0000 7fbe25705640 10 monclient(hunting): start opening mon connection 2024-09-16T03:44:51.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.893+0000 7fbe25705640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fbe20137920 con 0x7fbe0c006650 2024-09-16T03:44:51.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.893+0000 7fbe25705640 10 monclient(hunting): _renew_subs 2024-09-16T03:44:51.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.893+0000 7fbe25705640 10 monclient(hunting): authenticate will time out at 1138.864212s 2024-09-16T03:44:51.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.893+0000 7fbe1f7fe640 1 -- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbe2006db90 msgr2=0x7fbe20130990 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.42:3300/0 2024-09-16T03:44:51.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.893+0000 7fbe1f7fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbe2006db90 0x7fbe20130990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:44:51.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.893+0000 7fbe1e7fc640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbe0c006650 0x7fbe20130450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:44:51.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.893+0000 7fbe1e7fc640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbe0c006650 0x7fbe20130450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:35884/0 (socket says 172.21.15.42:35884) 2024-09-16T03:44:51.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.893+0000 7fbe1e7fc640 1 -- 172.21.15.42:0/3251726720 learned_addr learned my addr 172.21.15.42:0/3251726720 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:44:51.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.893+0000 7fbe1e7fc640 10 monclient(hunting): get_auth_request con 0x7fbe0c006650 auth_method 0 2024-09-16T03:44:51.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.893+0000 7fbe1e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:44:51.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.893+0000 7fbe1e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:44:51.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.893+0000 7fbe1e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:44:51.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.893+0000 7fbe1e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:44:51.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.893+0000 7fbe1e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:44:51.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.893+0000 7fbe1e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:44:51.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.893+0000 7fbe1effd640 1 --2- 172.21.15.42:0/3251726720 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbe0c0036d0 0x7fbe200a24c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:44:51.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.893+0000 7fbe1effd640 10 monclient(hunting): get_auth_request con 0x7fbe0c0036d0 auth_method 0 2024-09-16T03:44:51.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.893+0000 7fbe1effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:44:51.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.893+0000 7fbe1effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:44:51.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.893+0000 7fbe1effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:44:51.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.893+0000 7fbe1e7fc640 10 monclient(hunting): handle_auth_done global_id 13136 payload 824 2024-09-16T03:44:51.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.893+0000 7fbe1e7fc640 1 -- 172.21.15.42:0/3251726720 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbe0c0036d0 msgr2=0x7fbe200a24c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:44:51.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.893+0000 7fbe1e7fc640 1 --2- 172.21.15.42:0/3251726720 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbe0c0036d0 0x7fbe200a24c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:44:51.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.893+0000 7fbe1e7fc640 1 -- 172.21.15.42:0/3251726720 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbe2006db90 msgr2=0x7fbe20130990 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:44:51.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.893+0000 7fbe1e7fc640 1 --2- 172.21.15.42:0/3251726720 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbe2006db90 0x7fbe20130990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:44:51.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.893+0000 7fbe1e7fc640 10 monclient: _finish_hunting 0 2024-09-16T03:44:51.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.893+0000 7fbe1e7fc640 1 monclient: found mon.noname-c 2024-09-16T03:44:51.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.893+0000 7fbe1e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:44:51.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.893+0000 7fbe1e7fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:44:51.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.893+0000 7fbe1e7fc640 1 -- 172.21.15.42:0/3251726720 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe2006f6a0 con 0x7fbe0c006650 2024-09-16T03:44:51.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.893+0000 7fbe1e7fc640 1 --2- 172.21.15.42:0/3251726720 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbe0c006650 0x7fbe20130450 secure :-1 s=READY pgs=2301 cs=0 l=1 rev1=1 crypto rx=0x7fbe1400bad0 tx=0x7fbe1400bf90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:44:51.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.893+0000 7fbe077fe640 10 client.?.objecter ms_handle_connect 0x7fbe0c006650 2024-09-16T03:44:51.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.893+0000 7fbe077fe640 10 client.?.objecter resend_mon_ops 2024-09-16T03:44:51.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.893+0000 7fbe077fe640 1 -- 172.21.15.42:0/3251726720 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe1401b020 con 0x7fbe0c006650 2024-09-16T03:44:51.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.893+0000 7fbe077fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:44:51.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.893+0000 7fbe077fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:44:51.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.893+0000 7fbe077fe640 10 monclient: dump: 2024-09-16T03:44:51.907 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:44:51.907 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:44:51.907 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:51.907 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:51.908 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:44:51.908 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:44:51.908 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:44:51.908 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:44:51.908 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:44:51.908 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:44:51.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.893+0000 7fbe077fe640 10 monclient: _finish_auth 0 2024-09-16T03:44:51.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.893+0000 7fbe077fe640 10 monclient: _check_auth_tickets 2024-09-16T03:44:51.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.893+0000 7fbe077fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:44:51.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.893+0000 7fbe077fe640 1 -- 172.21.15.42:0/3251726720 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbe14005a80 con 0x7fbe0c006650 2024-09-16T03:44:51.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.893+0000 7fbe077fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:44:51.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.893+0000 7fbe077fe640 1 -- 172.21.15.42:0/3251726720 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe14017630 con 0x7fbe0c006650 2024-09-16T03:44:51.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.893+0000 7fbe077fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:44:51.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.893+0000 7fbe077fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:44:51.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.893+0000 7fbe077fe640 10 monclient: dump: 2024-09-16T03:44:51.909 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:44:51.909 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:44:51.909 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:51.909 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:51.909 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:44:51.909 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:44:51.909 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:44:51.909 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:44:51.909 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:44:51.909 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:44:51.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.893+0000 7fbe25705640 5 monclient: authenticate success, global_id 13136 2024-09-16T03:44:51.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.893+0000 7fbe25705640 10 monclient: _renew_subs 2024-09-16T03:44:51.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.893+0000 7fbe1d7fa640 4 set_mon_vals no callback set 2024-09-16T03:44:51.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.893+0000 7fbe25705640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:44:51.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.893+0000 7fbe25705640 1 -- 172.21.15.42:0/3251726720 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe20122e60 con 0x7fbe0c006650 2024-09-16T03:44:51.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.893+0000 7fbe25705640 10 client.13136.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:44:51.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.893+0000 7fbe25705640 10 monclient: _renew_subs 2024-09-16T03:44:51.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.893+0000 7fbe25705640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:44:51.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.893+0000 7fbe25705640 1 -- 172.21.15.42:0/3251726720 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe200a3150 con 0x7fbe0c006650 2024-09-16T03:44:51.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.893+0000 7fbe25705640 1 librados: init done 2024-09-16T03:44:51.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.893+0000 7fbe077fe640 1 -- 172.21.15.42:0/3251726720 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fbe1401e020 con 0x7fbe0c006650 2024-09-16T03:44:51.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.893+0000 7fbe25705640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:44:51.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.893+0000 7fbe25705640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:44:51.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.893+0000 7fbe25705640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:44:51.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.893+0000 7fbe25705640 1 -- 172.21.15.42:0/3251726720 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe2013a2a0 con 0x7fbe0c006650 2024-09-16T03:44:51.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.893+0000 7fbe077fe640 1 --2- 172.21.15.42:0/3251726720 >> v2:172.21.15.154:6812/254845342 conn(0x7fbe0c038be0 0x7fbe0c03b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:44:51.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.893+0000 7fbe077fe640 1 -- 172.21.15.42:0/3251726720 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(59..59 src has 1..59) ==== 8888+0+0 (secure 0 0 0) 0x7fbe1404a9b0 con 0x7fbe0c006650 2024-09-16T03:44:51.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.893+0000 7fbe077fe640 10 client.13136.objecter ms_dispatch 0x7fbe20000ba0 osd_map(59..59 src has 1..59) 2024-09-16T03:44:51.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.893+0000 7fbe077fe640 3 client.13136.objecter handle_osd_map got epochs [59,59] > 0 2024-09-16T03:44:51.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.893+0000 7fbe077fe640 3 client.13136.objecter handle_osd_map decoding full epoch 59 2024-09-16T03:44:51.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.893+0000 7fbe077fe640 20 client.13136.objecter dump_active .. 0 homeless 2024-09-16T03:44:51.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.893+0000 7fbe1effd640 1 --2- 172.21.15.42:0/3251726720 >> v2:172.21.15.154:6812/254845342 conn(0x7fbe0c038be0 0x7fbe0c03b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:44:51.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.893+0000 7fbe1effd640 10 monclient: get_auth_request con 0x7fbe0c038be0 auth_method 0 2024-09-16T03:44:51.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.897+0000 7fbe1effd640 1 --2- 172.21.15.42:0/3251726720 >> v2:172.21.15.154:6812/254845342 conn(0x7fbe0c038be0 0x7fbe0c03b0a0 secure :-1 s=READY pgs=3316 cs=0 l=1 rev1=1 crypto rx=0x7fbe100b0000 tx=0x7fbe100a7000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:44:51.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.897+0000 7fbe077fe640 10 client.13136.objecter ms_handle_connect 0x7fbe0c038be0 2024-09-16T03:44:51.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.901+0000 7fbe077fe640 1 -- 172.21.15.42:0/3251726720 <== mon.2 v2:172.21.15.42:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7fbe14025020 con 0x7fbe0c006650 2024-09-16T03:44:51.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.901+0000 7fbe077fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:44:51.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:51.901+0000 7fbe077fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:44:52.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.129+0000 7fbe25705640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:44:52.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.129+0000 7fbe25705640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:44:52.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.129+0000 7fbe25705640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:44:52.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.129+0000 7fbe25705640 1 -- 172.21.15.42:0/3251726720 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fbe201cf780 con 0x7fbe0c006650 2024-09-16T03:44:52.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.129+0000 7fbe1e7fc640 1 -- 172.21.15.42:0/3251726720 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbe0c006650 msgr2=0x7fbe20130450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-16T03:44:52.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.129+0000 7fbe1e7fc640 1 -- 172.21.15.42:0/3251726720 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbe0c006650 msgr2=0x7fbe20130450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T03:44:52.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.129+0000 7fbe1e7fc640 1 --2- 172.21.15.42:0/3251726720 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbe0c006650 0x7fbe20130450 secure :-1 s=READY pgs=2301 cs=0 l=1 rev1=1 crypto rx=0x7fbe1400bad0 tx=0x7fbe1400bf90 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:44:52.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.129+0000 7fbe1e7fc640 1 --2- 172.21.15.42:0/3251726720 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbe0c006650 0x7fbe20130450 secure :-1 s=READY pgs=2301 cs=0 l=1 rev1=1 crypto rx=0x7fbe1400bad0 tx=0x7fbe1400bf90 comp rx=0 tx=0).stop 2024-09-16T03:44:52.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.129+0000 7fbe077fe640 10 monclient: ms_handle_reset current mon [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:44:52.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.129+0000 7fbe077fe640 10 monclient: _reopen_session rank -1 2024-09-16T03:44:52.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.129+0000 7fbe077fe640 1 -- 172.21.15.42:0/3251726720 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbe0c006650 msgr2=0x7fbe20130450 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:44:52.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.129+0000 7fbe077fe640 1 --2- 172.21.15.42:0/3251726720 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbe0c006650 0x7fbe20130450 unknown :-1 s=CLOSED pgs=2301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:44:52.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.129+0000 7fbe077fe640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:44:52.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.129+0000 7fbe077fe640 1 --2- 172.21.15.42:0/3251726720 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbe2006db90 0x7fbe0c04a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:44:52.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.129+0000 7fbe077fe640 10 monclient(hunting): picked mon.a con 0x7fbe2006db90 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:44:52.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.129+0000 7fbe077fe640 1 --2- 172.21.15.42:0/3251726720 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbe0c04b0e0 0x7fbe0c04d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:44:52.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.129+0000 7fbe077fe640 10 monclient(hunting): picked mon.c con 0x7fbe0c04b0e0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:44:52.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.129+0000 7fbe077fe640 1 --2- 172.21.15.42:0/3251726720 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbe0c0036d0 0x7fbe0c0500e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:44:52.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.129+0000 7fbe077fe640 10 monclient(hunting): picked mon.b con 0x7fbe0c0036d0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:44:52.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.129+0000 7fbe077fe640 10 monclient(hunting): start opening mon connection 2024-09-16T03:44:52.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.129+0000 7fbe077fe640 1 -- 172.21.15.42:0/3251726720 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fbe0c050af0 con 0x7fbe2006db90 2024-09-16T03:44:52.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.129+0000 7fbe077fe640 10 monclient(hunting): start opening mon connection 2024-09-16T03:44:52.140 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.129+0000 7fbe1e7fc640 1 -- 172.21.15.42:0/3251726720 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbe2006db90 msgr2=0x7fbe0c04a8f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.42:3300/0 2024-09-16T03:44:52.140 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.129+0000 7fbe077fe640 1 -- 172.21.15.42:0/3251726720 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fbe0c050c70 con 0x7fbe0c0036d0 2024-09-16T03:44:52.140 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.129+0000 7fbe077fe640 10 monclient(hunting): start opening mon connection 2024-09-16T03:44:52.140 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.129+0000 7fbe077fe640 1 -- 172.21.15.42:0/3251726720 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fbe0c050e40 con 0x7fbe0c04b0e0 2024-09-16T03:44:52.140 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.129+0000 7fbe077fe640 10 monclient(hunting): _renew_subs 2024-09-16T03:44:52.140 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.129+0000 7fbe1e7fc640 1 --2- 172.21.15.42:0/3251726720 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbe2006db90 0x7fbe0c04a8f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:44:52.140 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.129+0000 7fbe1f7fe640 1 --2- 172.21.15.42:0/3251726720 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbe0c04b0e0 0x7fbe0c04d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:44:52.140 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.129+0000 7fbe1f7fe640 10 monclient(hunting): get_auth_request con 0x7fbe0c04b0e0 auth_method 0 2024-09-16T03:44:52.140 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.129+0000 7fbe1f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:44:52.140 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.129+0000 7fbe1f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:44:52.140 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.129+0000 7fbe1f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:44:52.140 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.129+0000 7fbe1f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:44:52.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.129+0000 7fbe1f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:44:52.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.129+0000 7fbe1f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T03:44:52.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.129+0000 7fbe1effd640 1 --2- 172.21.15.42:0/3251726720 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbe0c0036d0 0x7fbe0c0500e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:44:52.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.129+0000 7fbe1effd640 10 monclient(hunting): get_auth_request con 0x7fbe0c0036d0 auth_method 0 2024-09-16T03:44:52.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.129+0000 7fbe1effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:44:52.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.129+0000 7fbe1effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:44:52.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.129+0000 7fbe1effd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:44:52.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.129+0000 7fbe1effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:44:52.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.129+0000 7fbe1effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:44:52.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.129+0000 7fbe1effd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T03:44:52.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.129+0000 7fbe1f7fe640 10 monclient(hunting): handle_auth_done global_id 13136 payload 293 2024-09-16T03:44:52.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.129+0000 7fbe1f7fe640 1 -- 172.21.15.42:0/3251726720 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbe0c0036d0 msgr2=0x7fbe0c0500e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:44:52.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.129+0000 7fbe1f7fe640 1 --2- 172.21.15.42:0/3251726720 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbe0c0036d0 0x7fbe0c0500e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:44:52.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.129+0000 7fbe1f7fe640 1 -- 172.21.15.42:0/3251726720 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbe2006db90 msgr2=0x7fbe0c04a8f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:44:52.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.129+0000 7fbe1f7fe640 1 --2- 172.21.15.42:0/3251726720 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbe2006db90 0x7fbe0c04a8f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:44:52.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.129+0000 7fbe1f7fe640 10 monclient: _finish_hunting 0 2024-09-16T03:44:52.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.129+0000 7fbe1f7fe640 1 monclient: found mon.c 2024-09-16T03:44:52.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.129+0000 7fbe1f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:44:52.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.129+0000 7fbe1f7fe640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:44:52.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.129+0000 7fbe1f7fe640 1 -- 172.21.15.42:0/3251726720 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) -- 0x7fbe0c051040 con 0x7fbe0c04b0e0 2024-09-16T03:44:52.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.129+0000 7fbe1f7fe640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:44:52.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.129+0000 7fbe1f7fe640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:44:52.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.129+0000 7fbe1f7fe640 1 -- 172.21.15.42:0/3251726720 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fbe18001ca0 con 0x7fbe0c04b0e0 2024-09-16T03:44:52.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.133+0000 7fbe1f7fe640 10 monclient: _finish_auth 0 2024-09-16T03:44:52.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.133+0000 7fbe1f7fe640 10 monclient: _check_auth_tickets 2024-09-16T03:44:52.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.133+0000 7fbe1f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:44:52.143 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.133+0000 7fbe1f7fe640 1 --2- 172.21.15.42:0/3251726720 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbe0c04b0e0 0x7fbe0c04d570 secure :-1 s=READY pgs=2302 cs=0 l=1 rev1=1 crypto rx=0x7fbe2006ef40 tx=0x7fbe1400d040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:44:52.143 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.133+0000 7fbe077fe640 10 client.13136.objecter ms_handle_connect 0x7fbe0c04b0e0 2024-09-16T03:44:52.143 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.133+0000 7fbe077fe640 10 client.13136.objecter resend_mon_ops 2024-09-16T03:44:52.143 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.133+0000 7fbe077fe640 1 -- 172.21.15.42:0/3251726720 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe18001870 con 0x7fbe0c04b0e0 2024-09-16T03:44:52.143 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.133+0000 7fbe077fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:44:52.143 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.133+0000 7fbe077fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:44:52.143 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.133+0000 7fbe077fe640 10 monclient: dump: 2024-09-16T03:44:52.143 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:44:52.143 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:44:52.143 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:52.143 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:52.143 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:44:52.143 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:44:52.143 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:44:52.144 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:44:52.144 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:44:52.144 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:44:52.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.133+0000 7fbe077fe640 1 -- 172.21.15.42:0/3251726720 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbe180016b0 con 0x7fbe0c04b0e0 2024-09-16T03:44:52.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.133+0000 7fbe077fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:44:52.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.133+0000 7fbe1d7fa640 4 set_mon_vals no callback set 2024-09-16T03:44:52.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.133+0000 7fbe077fe640 1 -- 172.21.15.42:0/3251726720 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fbe1800cc00 con 0x7fbe0c04b0e0 2024-09-16T03:44:52.145 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.133+0000 7fbe077fe640 1 -- 172.21.15.42:0/3251726720 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1554 (secure 0 0 0) 0x7fbe2009bf80 con 0x7fbe0c04b0e0 2024-09-16T03:44:52.145 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.133+0000 7fbe077fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:44:52.145 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.133+0000 7fbe077fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:44:52.145 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":24,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":1,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:44:52.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.137+0000 7fbe25705640 10 librados: watch_flush enter 2024-09-16T03:44:52.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.137+0000 7fbe25705640 10 librados: watch_flush exit 2024-09-16T03:44:52.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.137+0000 7fbe25705640 20 client.13136.objecter shutdown clearing up homeless session... 2024-09-16T03:44:52.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.137+0000 7fbe25705640 10 client.13136.objecter successfully canceled tick 2024-09-16T03:44:52.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.137+0000 7fbe25705640 1 -- 172.21.15.42:0/3251726720 >> v2:172.21.15.154:6812/254845342 conn(0x7fbe0c038be0 msgr2=0x7fbe0c03b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:44:52.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.137+0000 7fbe25705640 1 --2- 172.21.15.42:0/3251726720 >> v2:172.21.15.154:6812/254845342 conn(0x7fbe0c038be0 0x7fbe0c03b0a0 secure :-1 s=READY pgs=3316 cs=0 l=1 rev1=1 crypto rx=0x7fbe100b0000 tx=0x7fbe100a7000 comp rx=0 tx=0).stop 2024-09-16T03:44:52.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.137+0000 7fbe25705640 10 monclient: shutdown 2024-09-16T03:44:52.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.137+0000 7fbe25705640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:44:52.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.137+0000 7fbe25705640 1 -- 172.21.15.42:0/3251726720 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbe0c04b0e0 msgr2=0x7fbe0c04d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:44:52.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.137+0000 7fbe25705640 1 --2- 172.21.15.42:0/3251726720 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbe0c04b0e0 0x7fbe0c04d570 secure :-1 s=READY pgs=2302 cs=0 l=1 rev1=1 crypto rx=0x7fbe2006ef40 tx=0x7fbe1400d040 comp rx=0 tx=0).stop 2024-09-16T03:44:52.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.137+0000 7fbe25705640 1 -- 172.21.15.42:0/3251726720 shutdown_connections 2024-09-16T03:44:52.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.137+0000 7fbe25705640 1 --2- 172.21.15.42:0/3251726720 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbe0c0036d0 0x7fbe0c0500e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:44:52.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.137+0000 7fbe25705640 1 --2- 172.21.15.42:0/3251726720 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbe0c04b0e0 0x7fbe0c04d570 unknown :-1 s=CLOSED pgs=2302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:44:52.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.137+0000 7fbe25705640 1 --2- 172.21.15.42:0/3251726720 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbe2006db90 0x7fbe0c04a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:44:52.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.137+0000 7fbe25705640 1 --2- 172.21.15.42:0/3251726720 >> v2:172.21.15.154:6812/254845342 conn(0x7fbe0c038be0 0x7fbe0c03b0a0 unknown :-1 s=CLOSED pgs=3316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:44:52.148 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.137+0000 7fbe25705640 1 -- 172.21.15.42:0/3251726720 >> 172.21.15.42:0/3251726720 conn(0x7fbe200a9fb0 msgr2=0x7fbe200a7a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:44:52.148 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.137+0000 7fbe25705640 1 -- 172.21.15.42:0/3251726720 shutdown_connections 2024-09-16T03:44:52.148 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.137+0000 7fbe25705640 1 -- 172.21.15.42:0/3251726720 wait complete. 2024-09-16T03:44:52.148 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.137+0000 7fbe25705640 1 librados: shutdown 2024-09-16T03:44:52.158 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T03:44:52.159 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-16T03:44:52.159 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T03:44:52.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.277+0000 7f2e60a75640 10 monclient: get_monmap_and_config 2024-09-16T03:44:52.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.277+0000 7f2e60a75640 10 monclient: build_initial_monmap 2024-09-16T03:44:52.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.277+0000 7f2e60a75640 10 monclient: monmap: 2024-09-16T03:44:52.282 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:44:52.282 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:44:52.282 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:44:52.281345+0000 2024-09-16T03:44:52.282 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:44:52.281345+0000 2024-09-16T03:44:52.283 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:44:52.283 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:44:52.283 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:44:52.283 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:44:52.283 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:44:52.283 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:44:52.283 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:44:52.283 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:44:52.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.277+0000 7f2e60a75640 1 Processor -- start 2024-09-16T03:44:52.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.277+0000 7f2e60a75640 1 -- start start 2024-09-16T03:44:52.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.277+0000 7f2e60a75640 10 monclient: init 2024-09-16T03:44:52.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.277+0000 7f2e60a75640 10 monclient: _reopen_session rank -1 2024-09-16T03:44:52.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.277+0000 7f2e60a75640 10 monclient: _add_conns ranks=[0,1,2,3,4] 2024-09-16T03:44:52.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.277+0000 7f2e60a75640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2e5c0a1330 addr v1:172.21.15.42:6789/0 2024-09-16T03:44:52.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.277+0000 7f2e60a75640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2e5c0a1710 addr v1:172.21.15.154:6789/0 2024-09-16T03:44:52.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.277+0000 7f2e60a75640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2e5c13a3b0 0x7f2e5c13c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:44:52.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.277+0000 7f2e60a75640 10 monclient(hunting): picked mon.noname-b con 0x7f2e5c13a3b0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:44:52.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.277+0000 7f2e60a75640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2e5c09e4c0 con 0x7f2e5c0a1330 2024-09-16T03:44:52.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.277+0000 7f2e60a75640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2e5c0a3170 con 0x7f2e5c0a1710 2024-09-16T03:44:52.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.277+0000 7f2e60a75640 10 monclient(hunting): start opening mon connection 2024-09-16T03:44:52.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.277+0000 7f2e60a75640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f2e5c09edf0 con 0x7f2e5c13a3b0 2024-09-16T03:44:52.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.277+0000 7f2e60a75640 10 monclient(hunting): _renew_subs 2024-09-16T03:44:52.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.277+0000 7f2e60a75640 10 monclient(hunting): authenticate will time out at 1139.250724s 2024-09-16T03:44:52.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.277+0000 7f2e5a59c640 1 -- >> v1:172.21.15.42:6789/0 conn(0x7f2e5c0a1330 legacy=0x7f2e5c09f180 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.42:6789/0 2024-09-16T03:44:52.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.277+0000 7f2e5ad9d640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2e5c13a3b0 0x7f2e5c13c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:44:52.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.277+0000 7f2e5ad9d640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2e5c13a3b0 0x7f2e5c13c7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:35910/0 (socket says 172.21.15.42:35910) 2024-09-16T03:44:52.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.277+0000 7f2e5ad9d640 1 -- 172.21.15.42:0/2451081554 learned_addr learned my addr 172.21.15.42:0/2451081554 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:44:52.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.277+0000 7f2e5ad9d640 10 monclient(hunting): get_auth_request con 0x7f2e5c13a3b0 auth_method 0 2024-09-16T03:44:52.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.277+0000 7f2e5ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:44:52.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.277+0000 7f2e5ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:44:52.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.277+0000 7f2e5ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:44:52.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.277+0000 7f2e5ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:44:52.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.277+0000 7f2e5ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:44:52.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.277+0000 7f2e5ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:44:52.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.277+0000 7f2e5ad9d640 10 monclient(hunting): handle_auth_done global_id 13139 payload 274 2024-09-16T03:44:52.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.277+0000 7f2e5ad9d640 1 -- 172.21.15.42:0/2451081554 >> v1:172.21.15.154:6789/0 conn(0x7f2e5c0a1710 legacy=0x7f2e5c0a1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:44:52.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.277+0000 7f2e5ad9d640 1 -- 172.21.15.42:0/2451081554 >> v1:172.21.15.42:6789/0 conn(0x7f2e5c0a1330 legacy=0x7f2e5c09f180 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:44:52.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.277+0000 7f2e5ad9d640 10 monclient: _finish_hunting 0 2024-09-16T03:44:52.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.277+0000 7f2e5ad9d640 1 monclient: found mon.noname-b 2024-09-16T03:44:52.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.277+0000 7f2e5ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:44:52.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.277+0000 7f2e5ad9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T03:44:52.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.277+0000 7f2e5ad9d640 1 -- 172.21.15.42:0/2451081554 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e5c09d030 con 0x7f2e5c13a3b0 2024-09-16T03:44:52.295 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.281+0000 7f2e5ad9d640 1 --2- 172.21.15.42:0/2451081554 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2e5c13a3b0 0x7f2e5c13c7a0 secure :-1 s=READY pgs=2303 cs=0 l=1 rev1=1 crypto rx=0x7f2e5c073490 tx=0x7f2e4c01c3c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dc5b3e92e368c203 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:44:52.295 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.281+0000 7f2e5959a640 1 -- 172.21.15.42:0/2451081554 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e4c0b0070 con 0x7f2e5c13a3b0 2024-09-16T03:44:52.296 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.281+0000 7f2e5959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:44:52.296 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.281+0000 7f2e5959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:44:52.296 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.281+0000 7f2e5959a640 10 monclient: dump: 2024-09-16T03:44:52.296 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:44:52.296 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:44:52.296 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:52.296 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:52.296 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:44:52.296 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:44:52.296 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:44:52.296 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:44:52.296 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:44:52.296 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:44:52.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.281+0000 7f2e5959a640 10 monclient: _finish_auth 0 2024-09-16T03:44:52.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.281+0000 7f2e5959a640 10 monclient: _check_auth_tickets 2024-09-16T03:44:52.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.281+0000 7f2e5959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:44:52.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.281+0000 7f2e5959a640 1 -- 172.21.15.42:0/2451081554 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e4c0ab600 con 0x7f2e5c13a3b0 2024-09-16T03:44:52.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.281+0000 7f2e60a75640 5 monclient: authenticate success, global_id 13139 2024-09-16T03:44:52.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.281+0000 7f2e60a75640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:44:52.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.281+0000 7f2e5959a640 10 monclient: handle_config config(0 keys) 2024-09-16T03:44:52.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.281+0000 7f2e60a75640 10 monclient: get_monmap_and_config success 2024-09-16T03:44:52.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.281+0000 7f2e5959a640 1 -- 172.21.15.42:0/2451081554 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e4c0ab9d0 con 0x7f2e5c13a3b0 2024-09-16T03:44:52.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.281+0000 7f2e5959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:44:52.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.281+0000 7f2e60a75640 4 set_mon_vals no callback set 2024-09-16T03:44:52.298 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.281+0000 7f2e5959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:44:52.298 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.281+0000 7f2e5959a640 10 monclient: dump: 2024-09-16T03:44:52.298 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:44:52.298 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:44:52.298 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:52.298 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:52.299 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:44:52.299 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:44:52.299 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:44:52.299 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:44:52.299 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:44:52.299 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:44:52.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.281+0000 7f2e60a75640 10 monclient: shutdown 2024-09-16T03:44:52.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.281+0000 7f2e60a75640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:44:52.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.281+0000 7f2e60a75640 1 -- 172.21.15.42:0/2451081554 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2e5c13a3b0 msgr2=0x7f2e5c13c7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:44:52.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.281+0000 7f2e60a75640 1 --2- 172.21.15.42:0/2451081554 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2e5c13a3b0 0x7f2e5c13c7a0 secure :-1 s=READY pgs=2303 cs=0 l=1 rev1=1 crypto rx=0x7f2e5c073490 tx=0x7f2e4c01c3c0 comp rx=0 tx=0).stop 2024-09-16T03:44:52.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.281+0000 7f2e60a75640 1 -- 172.21.15.42:0/2451081554 shutdown_connections 2024-09-16T03:44:52.300 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.281+0000 7f2e60a75640 1 --2- 172.21.15.42:0/2451081554 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2e5c13a3b0 0x7f2e5c13c7a0 unknown :-1 s=CLOSED pgs=2303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:44:52.300 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.281+0000 7f2e60a75640 1 -- 172.21.15.42:0/2451081554 >> 172.21.15.42:0/2451081554 conn(0x7f2e5c0a2420 msgr2=0x7f2e5c0a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:44:52.300 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.281+0000 7f2e60a75640 1 -- 172.21.15.42:0/2451081554 shutdown_connections 2024-09-16T03:44:52.300 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.281+0000 7f2e60a75640 1 -- 172.21.15.42:0/2451081554 wait complete. 2024-09-16T03:44:52.300 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.281+0000 7f2e60a75640 10 monclient: build_initial_monmap 2024-09-16T03:44:52.300 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.281+0000 7f2e60a75640 10 monclient: monmap: 2024-09-16T03:44:52.300 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:44:52.300 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:44:52.300 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:44:52.300 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:44:52.300 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:44:52.300 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:44:52.301 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:44:52.301 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:44:52.301 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:44:52.301 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:44:52.301 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.281+0000 7f2e60a75640 1 librados: starting msgr at 2024-09-16T03:44:52.301 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.281+0000 7f2e60a75640 1 librados: starting objecter 2024-09-16T03:44:52.301 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.281+0000 7f2e60a75640 1 Processor -- start 2024-09-16T03:44:52.301 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.281+0000 7f2e60a75640 1 -- start start 2024-09-16T03:44:52.301 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.281+0000 7f2e60a75640 1 librados: setting wanted keys 2024-09-16T03:44:52.301 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.281+0000 7f2e60a75640 1 librados: calling monclient init 2024-09-16T03:44:52.301 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.281+0000 7f2e60a75640 10 monclient: init 2024-09-16T03:44:52.301 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.281+0000 7f2e60a75640 10 monclient: _reopen_session rank -1 2024-09-16T03:44:52.301 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.281+0000 7f2e60a75640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:44:52.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.281+0000 7f2e60a75640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2e5c0a1330 0x7f2e5c133fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:44:52.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.281+0000 7f2e60a75640 10 monclient(hunting): picked mon.noname-c con 0x7f2e5c0a1330 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:44:52.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.281+0000 7f2e60a75640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2e5c0a1710 0x7f2e5c134500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:44:52.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.281+0000 7f2e60a75640 10 monclient(hunting): picked mon.noname-b con 0x7f2e5c0a1710 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:44:52.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.281+0000 7f2e60a75640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2e5c13a3b0 0x7f2e5c11bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:44:52.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.281+0000 7f2e60a75640 10 monclient(hunting): picked mon.noname-a con 0x7f2e5c13a3b0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:44:52.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.281+0000 7f2e60a75640 10 monclient(hunting): start opening mon connection 2024-09-16T03:44:52.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.281+0000 7f2e60a75640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f2e5c13fd30 con 0x7f2e5c13a3b0 2024-09-16T03:44:52.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.281+0000 7f2e60a75640 10 monclient(hunting): start opening mon connection 2024-09-16T03:44:52.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.281+0000 7f2e60a75640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f2e5c09cd60 con 0x7f2e5c0a1710 2024-09-16T03:44:52.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.281+0000 7f2e60a75640 10 monclient(hunting): start opening mon connection 2024-09-16T03:44:52.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.281+0000 7f2e60a75640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f2e5c13feb0 con 0x7f2e5c0a1330 2024-09-16T03:44:52.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.281+0000 7f2e60a75640 10 monclient(hunting): _renew_subs 2024-09-16T03:44:52.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.281+0000 7f2e60a75640 10 monclient(hunting): authenticate will time out at 1139.255740s 2024-09-16T03:44:52.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.281+0000 7f2e5ad9d640 1 -- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2e5c13a3b0 msgr2=0x7f2e5c11bf70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.42:3300/0 2024-09-16T03:44:52.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.281+0000 7f2e5ad9d640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2e5c13a3b0 0x7f2e5c11bf70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:44:52.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.281+0000 7f2e5a59c640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2e5c0a1330 0x7f2e5c133fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:44:52.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.281+0000 7f2e5a59c640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2e5c0a1330 0x7f2e5c133fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:35920/0 (socket says 172.21.15.42:35920) 2024-09-16T03:44:52.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.281+0000 7f2e5a59c640 1 -- 172.21.15.42:0/1675938153 learned_addr learned my addr 172.21.15.42:0/1675938153 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:44:52.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.281+0000 7f2e5a59c640 10 monclient(hunting): get_auth_request con 0x7f2e5c0a1330 auth_method 0 2024-09-16T03:44:52.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.281+0000 7f2e5a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:44:52.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.281+0000 7f2e5a59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:44:52.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.281+0000 7f2e5a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:44:52.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.285+0000 7f2e5a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:44:52.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.285+0000 7f2e5a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:44:52.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.285+0000 7f2e5a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:44:52.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.285+0000 7f2e5a59c640 10 monclient(hunting): handle_auth_done global_id 13142 payload 824 2024-09-16T03:44:52.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.285+0000 7f2e59d9b640 1 --2- 172.21.15.42:0/1675938153 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2e5c0a1710 0x7f2e5c134500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:44:52.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.285+0000 7f2e5a59c640 1 -- 172.21.15.42:0/1675938153 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2e5c0a1710 msgr2=0x7f2e5c134500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:44:52.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.285+0000 7f2e5a59c640 1 --2- 172.21.15.42:0/1675938153 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2e5c0a1710 0x7f2e5c134500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:44:52.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.285+0000 7f2e5a59c640 1 -- 172.21.15.42:0/1675938153 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2e5c13a3b0 msgr2=0x7f2e5c11bf70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:44:52.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.285+0000 7f2e5a59c640 1 --2- 172.21.15.42:0/1675938153 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2e5c13a3b0 0x7f2e5c11bf70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:44:52.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.285+0000 7f2e5a59c640 10 monclient: _finish_hunting 0 2024-09-16T03:44:52.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.285+0000 7f2e5a59c640 1 monclient: found mon.noname-c 2024-09-16T03:44:52.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.285+0000 7f2e5a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:44:52.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.285+0000 7f2e5a59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:44:52.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.285+0000 7f2e5a59c640 1 -- 172.21.15.42:0/1675938153 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e5c0a0c00 con 0x7f2e5c0a1330 2024-09-16T03:44:52.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.285+0000 7f2e59d9b640 10 monclient: get_auth_request con 0x7f2e5c0a1710 auth_method 0 2024-09-16T03:44:52.306 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.285+0000 7f2e59d9b640 1 --2- 172.21.15.42:0/1675938153 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2e5c0a1710 0x7f2e5c134500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T03:44:52.306 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.285+0000 7f2e5a59c640 1 --2- 172.21.15.42:0/1675938153 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2e5c0a1330 0x7f2e5c133fc0 secure :-1 s=READY pgs=2304 cs=0 l=1 rev1=1 crypto rx=0x7f2e5400b9e0 tx=0x7f2e5400bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:44:52.306 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.285+0000 7f2e4affd640 10 client.?.objecter ms_handle_connect 0x7f2e5c0a1330 2024-09-16T03:44:52.306 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.285+0000 7f2e4affd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:44:52.306 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.285+0000 7f2e4affd640 1 -- 172.21.15.42:0/1675938153 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e5400cba0 con 0x7f2e5c0a1330 2024-09-16T03:44:52.306 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.285+0000 7f2e4affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:44:52.306 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.285+0000 7f2e4affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:44:52.306 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.285+0000 7f2e4affd640 10 monclient: dump: 2024-09-16T03:44:52.306 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:44:52.306 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:44:52.306 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:52.306 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:52.307 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:44:52.307 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:44:52.307 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:44:52.307 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:44:52.307 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:44:52.307 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:44:52.307 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.285+0000 7f2e4affd640 10 monclient: _finish_auth 0 2024-09-16T03:44:52.307 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.285+0000 7f2e4affd640 10 monclient: _check_auth_tickets 2024-09-16T03:44:52.307 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.285+0000 7f2e4affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:44:52.307 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.285+0000 7f2e60a75640 5 monclient: authenticate success, global_id 13142 2024-09-16T03:44:52.307 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.285+0000 7f2e60a75640 10 monclient: _renew_subs 2024-09-16T03:44:52.307 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.285+0000 7f2e4affd640 1 -- 172.21.15.42:0/1675938153 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e54010070 con 0x7f2e5c0a1330 2024-09-16T03:44:52.307 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.285+0000 7f2e60a75640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:44:52.307 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.285+0000 7f2e60a75640 1 -- 172.21.15.42:0/1675938153 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e5c073a30 con 0x7f2e5c0a1330 2024-09-16T03:44:52.308 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.285+0000 7f2e4affd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:44:52.308 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.285+0000 7f2e4affd640 1 -- 172.21.15.42:0/1675938153 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e54015470 con 0x7f2e5c0a1330 2024-09-16T03:44:52.308 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.285+0000 7f2e4affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:44:52.308 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.285+0000 7f2e4affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:44:52.308 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.285+0000 7f2e4affd640 10 monclient: dump: 2024-09-16T03:44:52.308 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:44:52.308 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:44:52.308 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:52.308 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:52.308 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:44:52.308 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:44:52.308 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:44:52.308 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:44:52.309 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:44:52.309 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:44:52.309 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.285+0000 7f2e60a75640 10 client.13142.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:44:52.309 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.285+0000 7f2e60a75640 10 monclient: _renew_subs 2024-09-16T03:44:52.309 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.285+0000 7f2e60a75640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:44:52.309 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.285+0000 7f2e60a75640 1 -- 172.21.15.42:0/1675938153 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e5c11ed20 con 0x7f2e5c0a1330 2024-09-16T03:44:52.309 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.285+0000 7f2e60a75640 1 librados: init done 2024-09-16T03:44:52.309 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.285+0000 7f2e58d99640 4 set_mon_vals no callback set 2024-09-16T03:44:52.309 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.285+0000 7f2e60a75640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:44:52.309 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.285+0000 7f2e60a75640 1 --2- 172.21.15.42:0/1675938153 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2e5c11c4b0 0x7f2e5c11c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:44:52.309 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.285+0000 7f2e60a75640 10 monclient(hunting): start opening mon connection 2024-09-16T03:44:52.309 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.285+0000 7f2e60a75640 1 -- 172.21.15.42:0/1675938153 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f2e5c11cdd0 con 0x7f2e5c11c4b0 2024-09-16T03:44:52.309 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.285+0000 7f2e4affd640 1 -- 172.21.15.42:0/1675938153 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f2e5401e3f0 con 0x7f2e5c0a1330 2024-09-16T03:44:52.309 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.285+0000 7f2e59d9b640 1 --2- 172.21.15.42:0/1675938153 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2e5c11c4b0 0x7f2e5c11c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:44:52.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.285+0000 7f2e59d9b640 10 monclient: get_auth_request con 0x7f2e5c11c4b0 auth_method 0 2024-09-16T03:44:52.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.285+0000 7f2e59d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:44:52.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.285+0000 7f2e59d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:44:52.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.285+0000 7f2e59d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:44:52.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.285+0000 7f2e59d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:44:52.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.285+0000 7f2e59d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:44:52.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.285+0000 7f2e59d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:44:52.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.285+0000 7f2e4affd640 1 --2- 172.21.15.42:0/1675938153 >> v2:172.21.15.154:6812/254845342 conn(0x7f2e3c0377c0 0x7f2e3c039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:44:52.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.285+0000 7f2e4affd640 1 -- 172.21.15.42:0/1675938153 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(59..59 src has 1..59) ==== 8888+0+0 (secure 0 0 0) 0x7f2e54049e60 con 0x7f2e5c0a1330 2024-09-16T03:44:52.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.285+0000 7f2e4affd640 10 client.13142.objecter ms_dispatch 0x7f2e5c000ba0 osd_map(59..59 src has 1..59) 2024-09-16T03:44:52.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.285+0000 7f2e4affd640 3 client.13142.objecter handle_osd_map got epochs [59,59] > 0 2024-09-16T03:44:52.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.285+0000 7f2e4affd640 3 client.13142.objecter handle_osd_map decoding full epoch 59 2024-09-16T03:44:52.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.285+0000 7f2e59d9b640 10 monclient(hunting): handle_auth_done global_id 13145 payload 824 2024-09-16T03:44:52.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.285+0000 7f2e59d9b640 1 -- 172.21.15.42:0/1675938153 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2e5c09f180 con 0x7f2e5c11c4b0 2024-09-16T03:44:52.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.285+0000 7f2e4affd640 20 client.13142.objecter dump_active .. 0 homeless 2024-09-16T03:44:52.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.285+0000 7f2e4affd640 10 client.13142.objecter ms_handle_connect 0x7f2e5c11c4b0 2024-09-16T03:44:52.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.285+0000 7f2e4affd640 10 client.13142.objecter resend_mon_ops 2024-09-16T03:44:52.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.285+0000 7f2e59d9b640 1 --2- 172.21.15.42:0/1675938153 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2e5c11c4b0 0x7f2e5c11c890 secure :-1 s=READY pgs=2305 cs=0 l=1 rev1=1 crypto rx=0x7f2e5c139f00 tx=0x7f2e5000d950 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:44:52.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.285+0000 7f2e5ad9d640 1 --2- 172.21.15.42:0/1675938153 >> v2:172.21.15.154:6812/254845342 conn(0x7f2e3c0377c0 0x7f2e3c039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:44:52.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.285+0000 7f2e5ad9d640 10 monclient: get_auth_request con 0x7f2e3c0377c0 auth_method 0 2024-09-16T03:44:52.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.285+0000 7f2e4affd640 1 -- 172.21.15.42:0/1675938153 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e50016070 con 0x7f2e5c11c4b0 2024-09-16T03:44:52.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.285+0000 7f2e4affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:44:52.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.285+0000 7f2e4affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:44:52.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.285+0000 7f2e4affd640 10 monclient: dump: 2024-09-16T03:44:52.311 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:44:52.311 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:44:52.311 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:52.312 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:52.312 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:44:52.312 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:44:52.312 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:44:52.312 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:44:52.312 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:44:52.312 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:44:52.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.285+0000 7f2e4affd640 1 -- 172.21.15.42:0/1675938153 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f2e50012800 con 0x7f2e5c11c4b0 2024-09-16T03:44:52.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.285+0000 7f2e4affd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:44:52.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.285+0000 7f2e4affd640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:44:52.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.285+0000 7f2e4affd640 1 -- 172.21.15.42:0/1675938153 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2e5c11c4b0 msgr2=0x7f2e5c11c890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:44:52.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.285+0000 7f2e4affd640 1 --2- 172.21.15.42:0/1675938153 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2e5c11c4b0 0x7f2e5c11c890 secure :-1 s=READY pgs=2305 cs=0 l=1 rev1=1 crypto rx=0x7f2e5c139f00 tx=0x7f2e5000d950 comp rx=0 tx=0).stop 2024-09-16T03:44:52.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.285+0000 7f2e4affd640 1 -- 172.21.15.42:0/1675938153 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2e5c11c4b0 msgr2=0x7f2e5c11c890 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:44:52.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.285+0000 7f2e4affd640 1 --2- 172.21.15.42:0/1675938153 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2e5c11c4b0 0x7f2e5c11c890 secure :-1 s=CLOSED pgs=2305 cs=0 l=1 rev1=1 crypto rx=0x7f2e5c139f00 tx=0x7f2e5000d950 comp rx=0 tx=0).stop 2024-09-16T03:44:52.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.285+0000 7f2e4affd640 10 client.13142.objecter ms_handle_connect 0x7f2e3c0377c0 2024-09-16T03:44:52.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.285+0000 7f2e5ad9d640 1 --2- 172.21.15.42:0/1675938153 >> v2:172.21.15.154:6812/254845342 conn(0x7f2e3c0377c0 0x7f2e3c039c80 secure :-1 s=READY pgs=3317 cs=0 l=1 rev1=1 crypto rx=0x7f2e4c0ad000 tx=0x7f2e4c0a6000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:44:52.314 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.293+0000 7f2e60a75640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:44:52.314 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.293+0000 7f2e60a75640 1 --2- 172.21.15.42:0/1675938153 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2e5c06d8e0 0x7f2e5c06dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:44:52.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.293+0000 7f2e60a75640 10 monclient(hunting): start opening mon connection 2024-09-16T03:44:52.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.293+0000 7f2e60a75640 1 -- 172.21.15.42:0/1675938153 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f2e5c11d740 con 0x7f2e5c06d8e0 2024-09-16T03:44:52.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.293+0000 7f2e59d9b640 1 --2- 172.21.15.42:0/1675938153 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2e5c06d8e0 0x7f2e5c06dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:44:52.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.293+0000 7f2e59d9b640 10 monclient: get_auth_request con 0x7f2e5c06d8e0 auth_method 0 2024-09-16T03:44:52.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.293+0000 7f2e59d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:44:52.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.293+0000 7f2e59d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:44:52.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.293+0000 7f2e59d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:44:52.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.293+0000 7f2e59d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:44:52.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.293+0000 7f2e59d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:44:52.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.293+0000 7f2e59d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:44:52.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.293+0000 7f2e59d9b640 10 monclient(hunting): handle_auth_done global_id 13148 payload 824 2024-09-16T03:44:52.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.293+0000 7f2e59d9b640 1 -- 172.21.15.42:0/1675938153 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f2e5c069040 con 0x7f2e5c06d8e0 2024-09-16T03:44:52.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.293+0000 7f2e59d9b640 1 --2- 172.21.15.42:0/1675938153 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2e5c06d8e0 0x7f2e5c06dcf0 secure :-1 s=READY pgs=2306 cs=0 l=1 rev1=1 crypto rx=0x7f2e50018000 tx=0x7f2e50003770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:44:52.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.293+0000 7f2e4affd640 10 client.13142.objecter ms_handle_connect 0x7f2e5c06d8e0 2024-09-16T03:44:52.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.293+0000 7f2e4affd640 10 client.13142.objecter resend_mon_ops 2024-09-16T03:44:52.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.293+0000 7f2e4affd640 1 -- 172.21.15.42:0/1675938153 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e50029020 con 0x7f2e5c06d8e0 2024-09-16T03:44:52.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.293+0000 7f2e4affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:44:52.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.293+0000 7f2e4affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:44:52.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.293+0000 7f2e4affd640 10 monclient: dump: 2024-09-16T03:44:52.316 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:44:52.316 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:44:52.316 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:52.316 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:52.316 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:44:52.316 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:44:52.317 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:44:52.317 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:44:52.317 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:44:52.317 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:44:52.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.293+0000 7f2e4affd640 1 -- 172.21.15.42:0/1675938153 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4869 (secure 0 0 0) 0x7f2e50024350 con 0x7f2e5c06d8e0 2024-09-16T03:44:52.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.293+0000 7f2e4affd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:44:52.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.293+0000 7f2e4affd640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:44:52.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.293+0000 7f2e4affd640 1 -- 172.21.15.42:0/1675938153 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2e5c06d8e0 msgr2=0x7f2e5c06dcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:44:52.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.293+0000 7f2e4affd640 1 --2- 172.21.15.42:0/1675938153 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2e5c06d8e0 0x7f2e5c06dcf0 secure :-1 s=READY pgs=2306 cs=0 l=1 rev1=1 crypto rx=0x7f2e50018000 tx=0x7f2e50003770 comp rx=0 tx=0).stop 2024-09-16T03:44:52.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.293+0000 7f2e4affd640 1 -- 172.21.15.42:0/1675938153 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2e5c06d8e0 msgr2=0x7f2e5c06dcf0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:44:52.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.293+0000 7f2e4affd640 1 --2- 172.21.15.42:0/1675938153 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2e5c06d8e0 0x7f2e5c06dcf0 secure :-1 s=CLOSED pgs=2306 cs=0 l=1 rev1=1 crypto rx=0x7f2e50018000 tx=0x7f2e50003770 comp rx=0 tx=0).stop 2024-09-16T03:44:52.318 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:44:52.318 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:44:52.318 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:44:52.318 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T03:44:52.318 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 24, 2024-09-16T03:44:52.318 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 171663, 2024-09-16T03:44:52.318 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:44:52.318 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T03:44:52.318 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T03:44:52.318 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:44:52.318 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 1, 2024-09-16T03:44:52.318 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:44:52.319 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:44:52.319 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:44:52.319 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:44:52.319 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:44:52.319 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:44:52.319 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:44:52.319 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:44:52.319 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:44:52.319 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:44:52.319 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:44:52.319 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:44:52.319 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:44:52.319 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:44:52.320 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:44:52.320 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:44:52.320 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:44:52.320 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:44:52.320 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:44:52.320 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:44:52.320 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:44:52.320 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:44:52.320 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:44:52.320 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:44:52.321 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:44:52.321 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:44:52.321 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:44:52.321 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:44:52.321 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:44:52.321 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:44:52.321 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:44:52.321 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:44:52.321 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:44:52.321 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:44:52.321 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:44:52.321 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:44:52.321 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:44:52.321 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:44:52.322 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:44:52.322 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:44:52.322 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:44:52.322 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:44:52.322 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:44:52.322 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:44:52.322 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:44:52.322 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:44:52.322 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:44:52.322 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:44:52.322 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:44:52.322 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:44:52.322 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:44:52.322 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:44:52.323 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:44:52.323 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:44:52.323 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:44:52.323 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:44:52.323 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:44:52.323 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:44:52.323 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:44:52.323 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:44:52.323 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:44:52.323 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:44:52.323 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:44:52.323 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:44:52.323 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:44:52.323 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:44:52.324 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:44:52.324 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:44:52.324 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:44:52.324 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:44:52.324 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:44:52.324 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:44:52.324 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:44:52.324 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:44:52.324 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:44:52.324 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:44:52.324 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:44:52.324 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:44:52.324 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:44:52.325 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:44:52.325 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:44:52.325 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:44:52.325 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:44:52.325 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:44:52.325 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:44:52.325 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:44:52.325 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:44:52.325 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:44:52.325 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:44:52.325 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:44:52.325 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:44:52.325 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:44:52.326 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:44:52.326 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:44:52.326 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:44:52.326 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:44:52.326 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:44:52.326 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:44:52.326 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:44:52.326 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:44:52.326 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:44:52.326 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:44:52.326 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:44:52.326 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:44:52.326 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:44:52.327 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:44:52.327 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:44:52.327 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:44:52.327 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:44:52.327 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:44:52.327 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:44:52.327 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:44:52.327 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:44:52.327 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:44:52.327 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:44:52.327 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:44:52.327 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:44:52.327 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:44:52.328 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:44:52.328 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:44:52.328 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:44:52.328 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:44:52.328 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:44:52.328 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:44:52.328 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:44:52.328 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:44:52.328 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:44:52.328 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:44:52.328 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:44:52.328 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:44:52.328 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:44:52.329 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:44:52.329 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:44:52.329 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:44:52.329 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:44:52.329 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:44:52.329 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:44:52.329 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:44:52.329 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:44:52.329 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:44:52.329 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:44:52.329 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T03:44:52.329 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:44:52.330 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:44:52.330 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:44:52.330 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 6 2024-09-16T03:44:52.330 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:44:52.330 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:44:52.330 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:44:52.330 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:44:52.330 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:44:52.330 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:44:52.330 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 11 2024-09-16T03:44:52.330 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:44:52.330 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:44:52.330 INFO:teuthology.orchestra.run.smithi042.stdout: "mgr": [ 2024-09-16T03:44:52.330 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:44:52.330 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:44:52.331 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:44:52.331 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:44:52.331 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:44:52.331 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:44:52.331 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:44:52.331 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:44:52.331 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:44:52.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.293+0000 7f2e60a75640 10 librados: watch_flush enter 2024-09-16T03:44:52.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.293+0000 7f2e60a75640 10 librados: watch_flush exit 2024-09-16T03:44:52.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.293+0000 7f2e60a75640 20 client.13142.objecter shutdown clearing up homeless session... 2024-09-16T03:44:52.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.293+0000 7f2e60a75640 10 client.13142.objecter successfully canceled tick 2024-09-16T03:44:52.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.293+0000 7f2e60a75640 1 -- 172.21.15.42:0/1675938153 >> v2:172.21.15.154:6812/254845342 conn(0x7f2e3c0377c0 msgr2=0x7f2e3c039c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:44:52.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.293+0000 7f2e60a75640 1 --2- 172.21.15.42:0/1675938153 >> v2:172.21.15.154:6812/254845342 conn(0x7f2e3c0377c0 0x7f2e3c039c80 secure :-1 s=READY pgs=3317 cs=0 l=1 rev1=1 crypto rx=0x7f2e4c0ad000 tx=0x7f2e4c0a6000 comp rx=0 tx=0).stop 2024-09-16T03:44:52.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.293+0000 7f2e60a75640 10 monclient: shutdown 2024-09-16T03:44:52.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.293+0000 7f2e60a75640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:44:52.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.293+0000 7f2e60a75640 1 -- 172.21.15.42:0/1675938153 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2e5c0a1330 msgr2=0x7f2e5c133fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:44:52.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.293+0000 7f2e60a75640 1 --2- 172.21.15.42:0/1675938153 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2e5c0a1330 0x7f2e5c133fc0 secure :-1 s=READY pgs=2304 cs=0 l=1 rev1=1 crypto rx=0x7f2e5400b9e0 tx=0x7f2e5400bea0 comp rx=0 tx=0).stop 2024-09-16T03:44:52.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.293+0000 7f2e5ad9d640 1 -- 172.21.15.42:0/1675938153 reap_dead start 2024-09-16T03:44:52.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.293+0000 7f2e5ad9d640 1 -- 172.21.15.42:0/1675938153 reap_dead start 2024-09-16T03:44:52.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.293+0000 7f2e60a75640 1 -- 172.21.15.42:0/1675938153 shutdown_connections 2024-09-16T03:44:52.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.293+0000 7f2e60a75640 1 -- 172.21.15.42:0/1675938153 >> 172.21.15.42:0/1675938153 conn(0x7f2e5c0a2420 msgr2=0x7f2e5c09c270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:44:52.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.293+0000 7f2e60a75640 1 -- 172.21.15.42:0/1675938153 shutdown_connections 2024-09-16T03:44:52.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.293+0000 7f2e60a75640 1 -- 172.21.15.42:0/1675938153 wait complete. 2024-09-16T03:44:52.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.293+0000 7f2e60a75640 1 librados: shutdown 2024-09-16T03:44:52.334 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T03:44:52.334 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-16T03:44:52.334 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T03:44:52.469 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.465+0000 7fe60eba8640 10 monclient: get_monmap_and_config 2024-09-16T03:44:52.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.465+0000 7fe60eba8640 10 monclient: build_initial_monmap 2024-09-16T03:44:52.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.465+0000 7fe60eba8640 10 monclient: monmap: 2024-09-16T03:44:52.470 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:44:52.470 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:44:52.470 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:44:52.468943+0000 2024-09-16T03:44:52.470 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:44:52.468943+0000 2024-09-16T03:44:52.470 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:44:52.470 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:44:52.470 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:44:52.470 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:44:52.471 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:44:52.471 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:44:52.471 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:44:52.471 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:44:52.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.465+0000 7fe60eba8640 1 Processor -- start 2024-09-16T03:44:52.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.465+0000 7fe60eba8640 1 -- start start 2024-09-16T03:44:52.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.465+0000 7fe60eba8640 10 monclient: init 2024-09-16T03:44:52.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.465+0000 7fe60eba8640 10 monclient: _reopen_session rank -1 2024-09-16T03:44:52.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.465+0000 7fe60eba8640 10 monclient: _add_conns ranks=[4,3,0,1,2] 2024-09-16T03:44:52.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.465+0000 7fe60eba8640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fe6080a1330 0x7fe60809f180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:44:52.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.465+0000 7fe60eba8640 10 monclient(hunting): picked mon.noname-c con 0x7fe6080a1330 addr v2:172.21.15.154:3300/0 2024-09-16T03:44:52.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.465+0000 7fe60eba8640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe6080a1710 0x7fe6080a1b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:44:52.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.465+0000 7fe60eba8640 10 monclient(hunting): picked mon.noname-a con 0x7fe6080a1710 addr v2:172.21.15.42:3300/0 2024-09-16T03:44:52.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.465+0000 7fe60eba8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe6081340f0 addr v1:172.21.15.42:6789/0 2024-09-16T03:44:52.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.465+0000 7fe60eba8640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe60809e4c0 con 0x7fe6081340f0 2024-09-16T03:44:52.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.465+0000 7fe60eba8640 10 monclient(hunting): start opening mon connection 2024-09-16T03:44:52.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.465+0000 7fe60eba8640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7fe60809edf0 con 0x7fe6080a1710 2024-09-16T03:44:52.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.465+0000 7fe60eba8640 10 monclient(hunting): start opening mon connection 2024-09-16T03:44:52.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.465+0000 7fe60eba8640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fe60809cd60 con 0x7fe6080a1330 2024-09-16T03:44:52.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.465+0000 7fe60eba8640 10 monclient(hunting): _renew_subs 2024-09-16T03:44:52.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.465+0000 7fe60eba8640 10 monclient(hunting): authenticate will time out at 1139.438103s 2024-09-16T03:44:52.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.465+0000 7fe60d145640 1 -- >> v1:172.21.15.42:6789/0 conn(0x7fe6081340f0 legacy=0x7fe608134540 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.42:6789/0 2024-09-16T03:44:52.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.465+0000 7fe607fff640 1 -- >> v2:172.21.15.42:3300/0 conn(0x7fe6080a1710 msgr2=0x7fe6080a1b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3300/0 2024-09-16T03:44:52.475 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.465+0000 7fe607fff640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe6080a1710 0x7fe6080a1b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:44:52.475 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.465+0000 7fe60c944640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fe6080a1330 0x7fe60809f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:44:52.475 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.465+0000 7fe60c944640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fe6080a1330 0x7fe60809f180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.42:58160/0 (socket says 172.21.15.42:58160) 2024-09-16T03:44:52.475 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.465+0000 7fe60c944640 1 -- 172.21.15.42:0/258965522 learned_addr learned my addr 172.21.15.42:0/258965522 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:44:52.475 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.465+0000 7fe60c944640 10 monclient(hunting): get_auth_request con 0x7fe6080a1330 auth_method 0 2024-09-16T03:44:52.475 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.465+0000 7fe60c944640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:44:52.475 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.465+0000 7fe60c944640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:44:52.475 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.465+0000 7fe60c944640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:44:52.475 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.465+0000 7fe60c944640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:44:52.475 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.465+0000 7fe60c944640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:44:52.476 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.465+0000 7fe60c944640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:44:52.476 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.465+0000 7fe60c944640 10 monclient(hunting): handle_auth_done global_id 272458 payload 274 2024-09-16T03:44:52.476 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.465+0000 7fe60c944640 1 -- 172.21.15.42:0/258965522 >> v2:172.21.15.42:3300/0 conn(0x7fe6080a1710 msgr2=0x7fe6080a1b40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:44:52.476 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.465+0000 7fe60c944640 1 --2- 172.21.15.42:0/258965522 >> v2:172.21.15.42:3300/0 conn(0x7fe6080a1710 0x7fe6080a1b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:44:52.476 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.465+0000 7fe60c944640 1 -- 172.21.15.42:0/258965522 >> v1:172.21.15.42:6789/0 conn(0x7fe6081340f0 legacy=0x7fe608134540 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:44:52.476 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.465+0000 7fe60c944640 10 monclient: _finish_hunting 0 2024-09-16T03:44:52.476 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.465+0000 7fe60c944640 1 monclient: found mon.noname-c 2024-09-16T03:44:52.476 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.465+0000 7fe60c944640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:44:52.476 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.465+0000 7fe60c944640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.154:3300/0 2024-09-16T03:44:52.476 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.465+0000 7fe60c944640 1 -- 172.21.15.42:0/258965522 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6080a2080 con 0x7fe6080a1330 2024-09-16T03:44:52.483 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.469+0000 7fe60c944640 1 --2- 172.21.15.42:0/258965522 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe6080a1330 0x7fe60809f180 secure :-1 s=READY pgs=2756 cs=0 l=1 rev1=1 crypto rx=0x7fe60806add0 tx=0x7fe5f801eb20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=755f9e97dbcd6094 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:44:52.484 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.469+0000 7fe6077fe640 1 -- 172.21.15.42:0/258965522 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5f80b4070 con 0x7fe6080a1330 2024-09-16T03:44:52.484 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.469+0000 7fe6077fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:44:52.484 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.469+0000 7fe6077fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:44:52.484 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.469+0000 7fe6077fe640 10 monclient: dump: 2024-09-16T03:44:52.484 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:44:52.484 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:44:52.484 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:52.484 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:52.484 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:44:52.484 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:44:52.484 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:44:52.484 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:44:52.485 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:44:52.485 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:44:52.485 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.469+0000 7fe6077fe640 10 monclient: _finish_auth 0 2024-09-16T03:44:52.485 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.469+0000 7fe6077fe640 10 monclient: _check_auth_tickets 2024-09-16T03:44:52.485 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.469+0000 7fe6077fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:44:52.485 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.469+0000 7fe6077fe640 1 -- 172.21.15.42:0/258965522 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe5f80a7a00 con 0x7fe6080a1330 2024-09-16T03:44:52.485 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.469+0000 7fe6077fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:44:52.485 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.469+0000 7fe6077fe640 1 -- 172.21.15.42:0/258965522 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5f80a7dd0 con 0x7fe6080a1330 2024-09-16T03:44:52.485 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.469+0000 7fe60eba8640 5 monclient: authenticate success, global_id 272458 2024-09-16T03:44:52.485 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.469+0000 7fe60eba8640 10 monclient: get_monmap_and_config success 2024-09-16T03:44:52.485 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.469+0000 7fe60eba8640 4 set_mon_vals no callback set 2024-09-16T03:44:52.485 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.469+0000 7fe6077fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:44:52.485 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.469+0000 7fe6077fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:44:52.485 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.469+0000 7fe6077fe640 10 monclient: dump: 2024-09-16T03:44:52.486 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:44:52.486 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:44:52.486 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:52.486 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:52.486 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:44:52.486 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:44:52.486 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:44:52.486 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:44:52.486 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:44:52.486 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:44:52.486 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.469+0000 7fe60eba8640 10 monclient: shutdown 2024-09-16T03:44:52.486 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.469+0000 7fe60eba8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:44:52.486 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.469+0000 7fe60eba8640 1 -- 172.21.15.42:0/258965522 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe6080a1330 msgr2=0x7fe60809f180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:44:52.486 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.469+0000 7fe60eba8640 1 --2- 172.21.15.42:0/258965522 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe6080a1330 0x7fe60809f180 secure :-1 s=READY pgs=2756 cs=0 l=1 rev1=1 crypto rx=0x7fe60806add0 tx=0x7fe5f801eb20 comp rx=0 tx=0).stop 2024-09-16T03:44:52.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.469+0000 7fe60eba8640 1 -- 172.21.15.42:0/258965522 shutdown_connections 2024-09-16T03:44:52.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.469+0000 7fe60eba8640 1 --2- 172.21.15.42:0/258965522 >> v2:172.21.15.42:3300/0 conn(0x7fe6080a1710 0x7fe6080a1b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:44:52.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.469+0000 7fe60eba8640 1 --2- 172.21.15.42:0/258965522 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe6080a1330 0x7fe60809f180 unknown :-1 s=CLOSED pgs=2756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:44:52.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.469+0000 7fe60eba8640 1 -- 172.21.15.42:0/258965522 >> 172.21.15.42:0/258965522 conn(0x7fe6080a2420 msgr2=0x7fe6080a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:44:52.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.469+0000 7fe60eba8640 1 -- 172.21.15.42:0/258965522 shutdown_connections 2024-09-16T03:44:52.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.469+0000 7fe60eba8640 1 -- 172.21.15.42:0/258965522 wait complete. 2024-09-16T03:44:52.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.469+0000 7fe60eba8640 10 monclient: build_initial_monmap 2024-09-16T03:44:52.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.469+0000 7fe60eba8640 10 monclient: monmap: 2024-09-16T03:44:52.487 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:44:52.487 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:44:52.487 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:44:52.487 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:44:52.487 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:44:52.487 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:44:52.488 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:44:52.488 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:44:52.488 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:44:52.488 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:44:52.488 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.469+0000 7fe60eba8640 1 librados: starting msgr at 2024-09-16T03:44:52.488 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.469+0000 7fe60eba8640 1 librados: starting objecter 2024-09-16T03:44:52.488 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.469+0000 7fe60eba8640 1 Processor -- start 2024-09-16T03:44:52.488 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.469+0000 7fe60eba8640 1 -- start start 2024-09-16T03:44:52.488 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.469+0000 7fe60eba8640 1 librados: setting wanted keys 2024-09-16T03:44:52.488 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.469+0000 7fe60eba8640 1 librados: calling monclient init 2024-09-16T03:44:52.488 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.469+0000 7fe60eba8640 10 monclient: init 2024-09-16T03:44:52.488 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.473+0000 7fe60eba8640 10 monclient: _reopen_session rank -1 2024-09-16T03:44:52.488 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.473+0000 7fe60eba8640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T03:44:52.489 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.473+0000 7fe60eba8640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe6080a1330 0x7fe6081cf6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:44:52.489 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.473+0000 7fe60eba8640 10 monclient(hunting): picked mon.noname-a con 0x7fe6080a1330 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:44:52.489 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.473+0000 7fe60eba8640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe6080a1710 0x7fe608133430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:44:52.489 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.473+0000 7fe60eba8640 10 monclient(hunting): picked mon.noname-b con 0x7fe6080a1710 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:44:52.489 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.473+0000 7fe60eba8640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe6081340f0 0x7fe608133970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:44:52.489 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.473+0000 7fe60eba8640 10 monclient(hunting): picked mon.noname-c con 0x7fe6081340f0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:44:52.489 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.473+0000 7fe60eba8640 10 monclient(hunting): start opening mon connection 2024-09-16T03:44:52.489 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.473+0000 7fe60eba8640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fe60809c3e0 con 0x7fe6080a1330 2024-09-16T03:44:52.489 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.473+0000 7fe60eba8640 10 monclient(hunting): start opening mon connection 2024-09-16T03:44:52.489 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.473+0000 7fe60eba8640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe60809c260 con 0x7fe6080a1710 2024-09-16T03:44:52.489 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.473+0000 7fe60eba8640 10 monclient(hunting): start opening mon connection 2024-09-16T03:44:52.489 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.473+0000 7fe60c944640 1 -- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe6080a1330 msgr2=0x7fe6081cf6f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.42:3300/0 2024-09-16T03:44:52.489 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.473+0000 7fe60eba8640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fe60809c560 con 0x7fe6081340f0 2024-09-16T03:44:52.489 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.473+0000 7fe60eba8640 10 monclient(hunting): _renew_subs 2024-09-16T03:44:52.490 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.473+0000 7fe60eba8640 10 monclient(hunting): authenticate will time out at 1139.444418s 2024-09-16T03:44:52.490 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.473+0000 7fe60c944640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe6080a1330 0x7fe6081cf6f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:44:52.490 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.473+0000 7fe60d145640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe6081340f0 0x7fe608133970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:44:52.490 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.473+0000 7fe60d145640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe6081340f0 0x7fe608133970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:35972/0 (socket says 172.21.15.42:35972) 2024-09-16T03:44:52.490 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.473+0000 7fe60d145640 1 -- 172.21.15.42:0/2383149494 learned_addr learned my addr 172.21.15.42:0/2383149494 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:44:52.490 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.473+0000 7fe60d145640 10 monclient(hunting): get_auth_request con 0x7fe6081340f0 auth_method 0 2024-09-16T03:44:52.490 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.473+0000 7fe60d145640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:44:52.490 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.473+0000 7fe60d145640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:44:52.490 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.473+0000 7fe60d145640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:44:52.490 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.473+0000 7fe607fff640 1 --2- 172.21.15.42:0/2383149494 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe6080a1710 0x7fe608133430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:44:52.490 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.473+0000 7fe607fff640 10 monclient(hunting): get_auth_request con 0x7fe6080a1710 auth_method 0 2024-09-16T03:44:52.491 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.473+0000 7fe607fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:44:52.491 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.473+0000 7fe607fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:44:52.491 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.473+0000 7fe607fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:44:52.491 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.473+0000 7fe60d145640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:44:52.491 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.473+0000 7fe60d145640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:44:52.491 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.473+0000 7fe60d145640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:44:52.491 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.473+0000 7fe607fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:44:52.491 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.473+0000 7fe607fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:44:52.491 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.473+0000 7fe607fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:44:52.491 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.473+0000 7fe60d145640 10 monclient(hunting): handle_auth_done global_id 13154 payload 824 2024-09-16T03:44:52.491 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.473+0000 7fe60d145640 1 -- 172.21.15.42:0/2383149494 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe6080a1710 msgr2=0x7fe608133430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:44:52.491 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.473+0000 7fe60d145640 1 --2- 172.21.15.42:0/2383149494 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe6080a1710 0x7fe608133430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:44:52.491 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.473+0000 7fe60d145640 1 -- 172.21.15.42:0/2383149494 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe6080a1330 msgr2=0x7fe6081cf6f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:44:52.491 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.473+0000 7fe60d145640 1 --2- 172.21.15.42:0/2383149494 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe6080a1330 0x7fe6081cf6f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:44:52.492 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.473+0000 7fe60d145640 10 monclient: _finish_hunting 0 2024-09-16T03:44:52.492 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.473+0000 7fe60d145640 1 monclient: found mon.noname-c 2024-09-16T03:44:52.492 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.473+0000 7fe60d145640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:44:52.493 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.473+0000 7fe60d145640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:44:52.493 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.473+0000 7fe60d145640 1 -- 172.21.15.42:0/2383149494 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe608073a30 con 0x7fe6081340f0 2024-09-16T03:44:52.493 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.473+0000 7fe60d145640 1 --2- 172.21.15.42:0/2383149494 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe6081340f0 0x7fe608133970 secure :-1 s=READY pgs=2308 cs=0 l=1 rev1=1 crypto rx=0x7fe60000be60 tx=0x7fe60000bf50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:44:52.493 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.473+0000 7fe6057fa640 10 client.?.objecter ms_handle_connect 0x7fe6081340f0 2024-09-16T03:44:52.493 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.473+0000 7fe6057fa640 10 client.?.objecter resend_mon_ops 2024-09-16T03:44:52.493 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.473+0000 7fe6057fa640 1 -- 172.21.15.42:0/2383149494 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe600016430 con 0x7fe6081340f0 2024-09-16T03:44:52.493 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.473+0000 7fe6057fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:44:52.493 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.473+0000 7fe6057fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:44:52.493 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.473+0000 7fe6057fa640 10 monclient: dump: 2024-09-16T03:44:52.493 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:44:52.493 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:44:52.493 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:52.493 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:52.494 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:44:52.494 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:44:52.494 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:44:52.494 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:44:52.494 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:44:52.494 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:44:52.494 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.473+0000 7fe6057fa640 10 monclient: _finish_auth 0 2024-09-16T03:44:52.494 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.473+0000 7fe6057fa640 10 monclient: _check_auth_tickets 2024-09-16T03:44:52.494 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.473+0000 7fe6057fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:44:52.494 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.473+0000 7fe6057fa640 1 -- 172.21.15.42:0/2383149494 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe600010030 con 0x7fe6081340f0 2024-09-16T03:44:52.494 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.473+0000 7fe60eba8640 5 monclient: authenticate success, global_id 13154 2024-09-16T03:44:52.494 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.473+0000 7fe60eba8640 10 monclient: _renew_subs 2024-09-16T03:44:52.494 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.473+0000 7fe60eba8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:44:52.495 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.473+0000 7fe60eba8640 1 -- 172.21.15.42:0/2383149494 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe608121840 con 0x7fe6081340f0 2024-09-16T03:44:52.495 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.473+0000 7fe6057fa640 10 monclient: handle_config config(0 keys) 2024-09-16T03:44:52.495 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.473+0000 7fe6057fa640 1 -- 172.21.15.42:0/2383149494 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe600016850 con 0x7fe6081340f0 2024-09-16T03:44:52.495 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.473+0000 7fe6057fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:44:52.495 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.473+0000 7fe6057fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:44:52.495 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.473+0000 7fe6057fa640 10 monclient: dump: 2024-09-16T03:44:52.495 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:44:52.495 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:44:52.495 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:52.495 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:52.495 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:44:52.495 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:44:52.495 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:44:52.496 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:44:52.496 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:44:52.496 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:44:52.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.473+0000 7fe60eba8640 10 client.13154.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:44:52.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.473+0000 7fe60eba8640 10 monclient: _renew_subs 2024-09-16T03:44:52.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.473+0000 7fe606ffd640 4 set_mon_vals no callback set 2024-09-16T03:44:52.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.473+0000 7fe60eba8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:44:52.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.473+0000 7fe60eba8640 1 -- 172.21.15.42:0/2383149494 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6080a0e40 con 0x7fe6081340f0 2024-09-16T03:44:52.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.473+0000 7fe60eba8640 1 librados: init done 2024-09-16T03:44:52.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.473+0000 7fe60eba8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:44:52.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.473+0000 7fe60eba8640 1 --2- 172.21.15.42:0/2383149494 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe608121a80 0x7fe608139ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:44:52.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.473+0000 7fe60eba8640 10 monclient(hunting): start opening mon connection 2024-09-16T03:44:52.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.473+0000 7fe60eba8640 1 -- 172.21.15.42:0/2383149494 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe60809f180 con 0x7fe608121a80 2024-09-16T03:44:52.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.473+0000 7fe6057fa640 1 -- 172.21.15.42:0/2383149494 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fe6000153e0 con 0x7fe6081340f0 2024-09-16T03:44:52.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.473+0000 7fe60c944640 1 --2- 172.21.15.42:0/2383149494 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe608121a80 0x7fe608139ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:44:52.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.473+0000 7fe60c944640 10 monclient: get_auth_request con 0x7fe608121a80 auth_method 0 2024-09-16T03:44:52.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.473+0000 7fe60c944640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:44:52.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.473+0000 7fe60c944640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:44:52.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.473+0000 7fe60c944640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:44:52.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.473+0000 7fe60c944640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:44:52.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.473+0000 7fe60c944640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:44:52.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.473+0000 7fe60c944640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:44:52.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.473+0000 7fe6057fa640 1 --2- 172.21.15.42:0/2383149494 >> v2:172.21.15.154:6812/254845342 conn(0x7fe5f40377c0 0x7fe5f4039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:44:52.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.473+0000 7fe6057fa640 1 -- 172.21.15.42:0/2383149494 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(60..60 src has 1..60) ==== 9270+0+0 (secure 0 0 0) 0x7fe60004a2a0 con 0x7fe6081340f0 2024-09-16T03:44:52.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.473+0000 7fe60c944640 10 monclient(hunting): handle_auth_done global_id 272464 payload 824 2024-09-16T03:44:52.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.473+0000 7fe6057fa640 10 client.13154.objecter ms_dispatch 0x7fe608000ba0 osd_map(60..60 src has 1..60) 2024-09-16T03:44:52.498 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.473+0000 7fe6057fa640 3 client.13154.objecter handle_osd_map got epochs [60,60] > 0 2024-09-16T03:44:52.498 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.473+0000 7fe6057fa640 3 client.13154.objecter handle_osd_map decoding full epoch 60 2024-09-16T03:44:52.498 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.473+0000 7fe60c944640 1 -- 172.21.15.42:0/2383149494 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe6081203f0 con 0x7fe608121a80 2024-09-16T03:44:52.498 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.473+0000 7fe6057fa640 20 client.13154.objecter dump_active .. 0 homeless 2024-09-16T03:44:52.498 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.473+0000 7fe607fff640 1 --2- 172.21.15.42:0/2383149494 >> v2:172.21.15.154:6812/254845342 conn(0x7fe5f40377c0 0x7fe5f4039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:44:52.498 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.473+0000 7fe607fff640 10 monclient: get_auth_request con 0x7fe5f40377c0 auth_method 0 2024-09-16T03:44:52.498 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.473+0000 7fe6057fa640 10 client.13154.objecter ms_handle_connect 0x7fe608121a80 2024-09-16T03:44:52.498 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.473+0000 7fe6057fa640 10 client.13154.objecter resend_mon_ops 2024-09-16T03:44:52.498 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.473+0000 7fe60c944640 1 --2- 172.21.15.42:0/2383149494 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe608121a80 0x7fe608139ea0 secure :-1 s=READY pgs=2757 cs=0 l=1 rev1=1 crypto rx=0x7fe60809df50 tx=0x7fe5f801d360 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:44:52.498 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.473+0000 7fe6057fa640 1 -- 172.21.15.42:0/2383149494 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5f80b4040 con 0x7fe608121a80 2024-09-16T03:44:52.498 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.473+0000 7fe6057fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:44:52.498 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.473+0000 7fe6057fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:44:52.498 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.473+0000 7fe6057fa640 10 monclient: dump: 2024-09-16T03:44:52.499 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:44:52.499 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:44:52.499 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:52.499 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:52.499 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:44:52.499 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:44:52.499 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:44:52.499 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:44:52.499 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:44:52.499 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:44:52.499 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.473+0000 7fe6057fa640 1 -- 172.21.15.42:0/2383149494 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fe5f8004220 con 0x7fe608121a80 2024-09-16T03:44:52.499 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.473+0000 7fe6057fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:44:52.499 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.473+0000 7fe6057fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:44:52.499 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.473+0000 7fe6057fa640 1 -- 172.21.15.42:0/2383149494 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe608121a80 msgr2=0x7fe608139ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:44:52.500 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.473+0000 7fe6057fa640 1 --2- 172.21.15.42:0/2383149494 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe608121a80 0x7fe608139ea0 secure :-1 s=READY pgs=2757 cs=0 l=1 rev1=1 crypto rx=0x7fe60809df50 tx=0x7fe5f801d360 comp rx=0 tx=0).stop 2024-09-16T03:44:52.500 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.473+0000 7fe607fff640 1 --2- 172.21.15.42:0/2383149494 >> v2:172.21.15.154:6812/254845342 conn(0x7fe5f40377c0 0x7fe5f4039c80 secure :-1 s=READY pgs=3319 cs=0 l=1 rev1=1 crypto rx=0x7fe60813b740 tx=0x7fe5fc009110 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:44:52.500 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.473+0000 7fe6057fa640 1 -- 172.21.15.42:0/2383149494 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe608121a80 msgr2=0x7fe608139ea0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:44:52.500 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.473+0000 7fe6057fa640 1 --2- 172.21.15.42:0/2383149494 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe608121a80 0x7fe608139ea0 secure :-1 s=CLOSED pgs=2757 cs=0 l=1 rev1=1 crypto rx=0x7fe60809df50 tx=0x7fe5f801d360 comp rx=0 tx=0).stop 2024-09-16T03:44:52.500 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.473+0000 7fe6057fa640 10 client.13154.objecter ms_handle_connect 0x7fe5f40377c0 2024-09-16T03:44:52.501 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.481+0000 7fe60eba8640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:44:52.502 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.481+0000 7fe60eba8640 1 --2- 172.21.15.42:0/2383149494 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe60813a3e0 0x7fe60813a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:44:52.502 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.481+0000 7fe60eba8640 10 monclient(hunting): start opening mon connection 2024-09-16T03:44:52.502 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.481+0000 7fe60eba8640 1 -- 172.21.15.42:0/2383149494 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe60813ad30 con 0x7fe60813a3e0 2024-09-16T03:44:52.502 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.481+0000 7fe60c944640 1 --2- 172.21.15.42:0/2383149494 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe60813a3e0 0x7fe60813a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:44:52.502 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.481+0000 7fe60c944640 10 monclient: get_auth_request con 0x7fe60813a3e0 auth_method 0 2024-09-16T03:44:52.502 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.481+0000 7fe60c944640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:44:52.502 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.481+0000 7fe60c944640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:44:52.502 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.481+0000 7fe60c944640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:44:52.502 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.481+0000 7fe60c944640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:44:52.502 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.481+0000 7fe60c944640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:44:52.502 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.481+0000 7fe60c944640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:44:52.502 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.481+0000 7fe60c944640 10 monclient(hunting): handle_auth_done global_id 272467 payload 824 2024-09-16T03:44:52.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.481+0000 7fe60c944640 1 -- 172.21.15.42:0/2383149494 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fe608121e60 con 0x7fe60813a3e0 2024-09-16T03:44:52.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.481+0000 7fe6057fa640 10 client.13154.objecter ms_handle_connect 0x7fe60813a3e0 2024-09-16T03:44:52.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.481+0000 7fe60c944640 1 --2- 172.21.15.42:0/2383149494 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe60813a3e0 0x7fe60813a7f0 secure :-1 s=READY pgs=2758 cs=0 l=1 rev1=1 crypto rx=0x7fe5f801c750 tx=0x7fe5f801b8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:44:52.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.481+0000 7fe6057fa640 10 client.13154.objecter resend_mon_ops 2024-09-16T03:44:52.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.481+0000 7fe6057fa640 1 -- 172.21.15.42:0/2383149494 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5f80b4040 con 0x7fe60813a3e0 2024-09-16T03:44:52.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.481+0000 7fe6057fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:44:52.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.481+0000 7fe6057fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:44:52.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.481+0000 7fe6057fa640 10 monclient: dump: 2024-09-16T03:44:52.503 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:44:52.503 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:44:52.503 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:52.503 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:44:52.503 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:44:52.504 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:44:52.504 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:44:52.504 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:44:52.504 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:44:52.504 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:44:52.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.481+0000 7fe6057fa640 1 -- 172.21.15.42:0/2383149494 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4535 (secure 0 0 0) 0x7fe5f8003e60 con 0x7fe60813a3e0 2024-09-16T03:44:52.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.481+0000 7fe6057fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:44:52.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.481+0000 7fe6057fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:44:52.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.481+0000 7fe6057fa640 1 -- 172.21.15.42:0/2383149494 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe60813a3e0 msgr2=0x7fe60813a7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:44:52.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.481+0000 7fe6057fa640 1 --2- 172.21.15.42:0/2383149494 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe60813a3e0 0x7fe60813a7f0 secure :-1 s=READY pgs=2758 cs=0 l=1 rev1=1 crypto rx=0x7fe5f801c750 tx=0x7fe5f801b8c0 comp rx=0 tx=0).stop 2024-09-16T03:44:52.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.481+0000 7fe6057fa640 1 -- 172.21.15.42:0/2383149494 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe60813a3e0 msgr2=0x7fe60813a7f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:44:52.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.481+0000 7fe6057fa640 1 --2- 172.21.15.42:0/2383149494 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe60813a3e0 0x7fe60813a7f0 secure :-1 s=CLOSED pgs=2758 cs=0 l=1 rev1=1 crypto rx=0x7fe5f801c750 tx=0x7fe5f801b8c0 comp rx=0 tx=0).stop 2024-09-16T03:44:52.505 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:44:52.505 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:44:52.505 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:44:52.505 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "leader", 2024-09-16T03:44:52.505 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 24, 2024-09-16T03:44:52.505 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 35209, 2024-09-16T03:44:52.505 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:44:52.505 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T03:44:52.505 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T03:44:52.505 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:44:52.505 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 2, 2024-09-16T03:44:52.505 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:44:52.505 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:44:52.506 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:44:52.506 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:44:52.506 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:44:52.506 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:44:52.506 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:44:52.506 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:44:52.506 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:44:52.506 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:44:52.506 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:44:52.506 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:44:52.506 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:44:52.506 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:44:52.507 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:44:52.507 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:44:52.507 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:44:52.507 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:44:52.507 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:44:52.507 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:44:52.507 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:44:52.507 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:44:52.507 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:44:52.507 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:44:52.507 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:44:52.507 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:44:52.507 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:44:52.508 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:44:52.508 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:44:52.508 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:44:52.508 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:44:52.508 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:44:52.508 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:44:52.508 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:44:52.508 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:44:52.508 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:44:52.508 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:44:52.508 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:44:52.508 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:44:52.508 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:44:52.508 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:44:52.509 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:44:52.509 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:44:52.509 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:44:52.509 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:44:52.509 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:44:52.509 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:44:52.509 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:44:52.509 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:44:52.509 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:44:52.509 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:44:52.509 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:44:52.509 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:44:52.509 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:44:52.509 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:44:52.510 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:44:52.510 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:44:52.510 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:44:52.510 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:44:52.510 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:44:52.510 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:44:52.510 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:44:52.510 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:44:52.510 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:44:52.510 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:44:52.510 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:44:52.510 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:44:52.510 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:44:52.510 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:44:52.511 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:44:52.511 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:44:52.511 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:44:52.511 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:44:52.511 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:44:52.511 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:44:52.511 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:44:52.511 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:44:52.511 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:44:52.511 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:44:52.511 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:44:52.511 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:44:52.511 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:44:52.511 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:44:52.512 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:44:52.512 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:44:52.512 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:44:52.512 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:44:52.512 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:44:52.512 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:44:52.512 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:44:52.512 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:44:52.512 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:44:52.512 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:44:52.512 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:44:52.512 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:44:52.512 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:44:52.513 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:44:52.513 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:44:52.513 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:44:52.513 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:44:52.513 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:44:52.513 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:44:52.513 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:44:52.513 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:44:52.513 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:44:52.513 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:44:52.513 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:44:52.513 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:44:52.513 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:44:52.513 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:44:52.514 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:44:52.514 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:44:52.514 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:44:52.514 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:44:52.514 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:44:52.514 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:44:52.514 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:44:52.514 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:44:52.514 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:44:52.514 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:44:52.514 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:44:52.514 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:44:52.514 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:44:52.515 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:44:52.515 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:44:52.515 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:44:52.515 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:44:52.515 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:44:52.515 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:44:52.515 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:44:52.515 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:44:52.515 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:44:52.515 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:44:52.515 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:44:52.515 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:44:52.515 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:44:52.515 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:44:52.516 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:44:52.516 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:44:52.516 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:44:52.516 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:44:52.516 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:44:52.516 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:44:52.516 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:44:52.516 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:44:52.516 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 9 2024-09-16T03:44:52.516 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:44:52.516 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:44:52.516 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:44:52.516 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:44:52.516 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:44:52.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.481+0000 7fe60eba8640 10 librados: watch_flush enter 2024-09-16T03:44:52.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.481+0000 7fe60eba8640 10 librados: watch_flush exit 2024-09-16T03:44:52.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.481+0000 7fe60eba8640 20 client.13154.objecter shutdown clearing up homeless session... 2024-09-16T03:44:52.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.481+0000 7fe60eba8640 10 client.13154.objecter successfully canceled tick 2024-09-16T03:44:52.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.481+0000 7fe60eba8640 1 -- 172.21.15.42:0/2383149494 >> v2:172.21.15.154:6812/254845342 conn(0x7fe5f40377c0 msgr2=0x7fe5f4039c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:44:52.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.481+0000 7fe60eba8640 1 --2- 172.21.15.42:0/2383149494 >> v2:172.21.15.154:6812/254845342 conn(0x7fe5f40377c0 0x7fe5f4039c80 secure :-1 s=READY pgs=3319 cs=0 l=1 rev1=1 crypto rx=0x7fe60813b740 tx=0x7fe5fc009110 comp rx=0 tx=0).stop 2024-09-16T03:44:52.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.481+0000 7fe60eba8640 10 monclient: shutdown 2024-09-16T03:44:52.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.481+0000 7fe60eba8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:44:52.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.481+0000 7fe60eba8640 1 -- 172.21.15.42:0/2383149494 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe6081340f0 msgr2=0x7fe608133970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:44:52.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.481+0000 7fe60eba8640 1 --2- 172.21.15.42:0/2383149494 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe6081340f0 0x7fe608133970 secure :-1 s=READY pgs=2308 cs=0 l=1 rev1=1 crypto rx=0x7fe60000be60 tx=0x7fe60000bf50 comp rx=0 tx=0).stop 2024-09-16T03:44:52.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.481+0000 7fe60d145640 1 -- 172.21.15.42:0/2383149494 reap_dead start 2024-09-16T03:44:52.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.481+0000 7fe60d145640 1 -- 172.21.15.42:0/2383149494 reap_dead start 2024-09-16T03:44:52.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.481+0000 7fe60eba8640 1 -- 172.21.15.42:0/2383149494 shutdown_connections 2024-09-16T03:44:52.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.481+0000 7fe60eba8640 1 -- 172.21.15.42:0/2383149494 >> 172.21.15.42:0/2383149494 conn(0x7fe6080a2420 msgr2=0x7fe608070590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:44:52.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.481+0000 7fe60eba8640 1 -- 172.21.15.42:0/2383149494 shutdown_connections 2024-09-16T03:44:52.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.481+0000 7fe60eba8640 1 -- 172.21.15.42:0/2383149494 wait complete. 2024-09-16T03:44:52.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:44:52.481+0000 7fe60eba8640 1 librados: shutdown 2024-09-16T03:44:52.520 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T03:44:52.520 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-16T03:44:55.346 INFO:tasks.workunit.client.0.smithi154.stdout: misc: Running main() from gmock_main.cc 2024-09-16T03:44:55.346 INFO:tasks.workunit.client.0.smithi154.stdout: misc: [==========] Running 12 tests from 1 test suite. 2024-09-16T03:44:55.346 INFO:tasks.workunit.client.0.smithi154.stdout: misc: [----------] Global test environment set-up. 2024-09-16T03:44:55.346 INFO:tasks.workunit.client.0.smithi154.stdout: misc: [----------] 12 tests from NeoRadosMisc 2024-09-16T03:44:55.346 INFO:tasks.workunit.client.0.smithi154.stdout: misc: [ RUN ] NeoRadosMisc.Version 2024-09-16T03:44:55.346 INFO:tasks.workunit.client.0.smithi154.stdout: misc: [ OK ] NeoRadosMisc.Version (4637 ms) 2024-09-16T03:44:55.346 INFO:tasks.workunit.client.0.smithi154.stdout: misc: [ RUN ] NeoRadosMisc.WaitOSDMap 2024-09-16T03:44:55.346 INFO:tasks.workunit.client.0.smithi154.stdout: misc: [ OK ] NeoRadosMisc.WaitOSDMap (23039 ms) 2024-09-16T03:44:55.346 INFO:tasks.workunit.client.0.smithi154.stdout: misc: [ RUN ] NeoRadosMisc.LongName 2024-09-16T03:44:55.346 INFO:tasks.workunit.client.0.smithi154.stdout: misc: [ OK ] NeoRadosMisc.LongName (2833 ms) 2024-09-16T03:44:55.346 INFO:tasks.workunit.client.0.smithi154.stdout: misc: [ RUN ] NeoRadosMisc.LongLocator 2024-09-16T03:44:55.347 INFO:tasks.workunit.client.0.smithi154.stdout: misc: [ OK ] NeoRadosMisc.LongLocator (2879 ms) 2024-09-16T03:44:55.347 INFO:tasks.workunit.client.0.smithi154.stdout: misc: [ RUN ] NeoRadosMisc.LongNamespace 2024-09-16T03:44:55.347 INFO:tasks.workunit.client.0.smithi154.stdout: misc: [ OK ] NeoRadosMisc.LongNamespace (2991 ms) 2024-09-16T03:44:55.347 INFO:tasks.workunit.client.0.smithi154.stdout: misc: [ RUN ] NeoRadosMisc.LongAttrName 2024-09-16T03:44:55.347 INFO:tasks.workunit.client.0.smithi154.stdout: misc: [ OK ] NeoRadosMisc.LongAttrName (3076 ms) 2024-09-16T03:44:55.347 INFO:tasks.workunit.client.0.smithi154.stdout: misc: [ RUN ] NeoRadosMisc.Exec 2024-09-16T03:44:55.347 INFO:tasks.workunit.client.0.smithi154.stdout: misc: [ OK ] NeoRadosMisc.Exec (2967 ms) 2024-09-16T03:44:55.347 INFO:tasks.workunit.client.0.smithi154.stdout: misc: [ RUN ] NeoRadosMisc.Operate1 2024-09-16T03:44:55.347 INFO:tasks.workunit.client.0.smithi154.stdout: misc: [ OK ] NeoRadosMisc.Operate1 (3017 ms) 2024-09-16T03:44:55.347 INFO:tasks.workunit.client.0.smithi154.stdout: misc: [ RUN ] NeoRadosMisc.Operate2 2024-09-16T03:44:55.347 INFO:tasks.workunit.client.0.smithi154.stdout: misc: [ OK ] NeoRadosMisc.Operate2 (18055 ms) 2024-09-16T03:44:55.347 INFO:tasks.workunit.client.0.smithi154.stdout: misc: [ RUN ] NeoRadosMisc.BigObject 2024-09-16T03:44:55.347 INFO:tasks.workunit.client.0.smithi154.stdout: misc: [ OK ] NeoRadosMisc.BigObject (17307 ms) 2024-09-16T03:44:55.348 INFO:tasks.workunit.client.0.smithi154.stdout: misc: [ RUN ] NeoRadosMisc.BigAttr 2024-09-16T03:44:55.348 INFO:tasks.workunit.client.0.smithi154.stdout: misc: [ OK ] NeoRadosMisc.BigAttr (1721 ms) 2024-09-16T03:44:55.348 INFO:tasks.workunit.client.0.smithi154.stdout: misc: [ RUN ] NeoRadosMisc.WriteSame 2024-09-16T03:44:55.348 INFO:tasks.workunit.client.0.smithi154.stdout: misc: [ OK ] NeoRadosMisc.WriteSame (3014 ms) 2024-09-16T03:44:55.348 INFO:tasks.workunit.client.0.smithi154.stdout: misc: [----------] 12 tests from NeoRadosMisc (85536 ms total) 2024-09-16T03:44:55.348 INFO:tasks.workunit.client.0.smithi154.stdout: misc: 2024-09-16T03:44:55.348 INFO:tasks.workunit.client.0.smithi154.stdout: misc: [----------] Global test environment tear-down 2024-09-16T03:44:55.348 INFO:tasks.workunit.client.0.smithi154.stdout: misc: [==========] 12 tests from 1 test suite ran. (85536 ms total) 2024-09-16T03:44:55.348 INFO:tasks.workunit.client.0.smithi154.stdout: misc: [ PASSED ] 12 tests. 2024-09-16T03:44:59.334 INFO:tasks.workunit.client.0.smithi154.stdout: api_pool: Running main() from gmock_main.cc 2024-09-16T03:44:59.334 INFO:tasks.workunit.client.0.smithi154.stdout: api_pool: [==========] Running 9 tests from 1 test suite. 2024-09-16T03:44:59.334 INFO:tasks.workunit.client.0.smithi154.stdout: api_pool: [----------] Global test environment set-up. 2024-09-16T03:44:59.334 INFO:tasks.workunit.client.0.smithi154.stdout: api_pool: [----------] 9 tests from LibRadosPools 2024-09-16T03:44:59.334 INFO:tasks.workunit.client.0.smithi154.stdout: api_pool: [ RUN ] LibRadosPools.PoolList 2024-09-16T03:44:59.334 INFO:tasks.workunit.client.0.smithi154.stdout: api_pool: [ OK ] LibRadosPools.PoolList (4833 ms) 2024-09-16T03:44:59.335 INFO:tasks.workunit.client.0.smithi154.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookup 2024-09-16T03:44:59.335 INFO:tasks.workunit.client.0.smithi154.stdout: api_pool: [ OK ] LibRadosPools.PoolLookup (23998 ms) 2024-09-16T03:44:59.335 INFO:tasks.workunit.client.0.smithi154.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookup2 2024-09-16T03:44:59.335 INFO:tasks.workunit.client.0.smithi154.stdout: api_pool: [ OK ] LibRadosPools.PoolLookup2 (2781 ms) 2024-09-16T03:44:59.335 INFO:tasks.workunit.client.0.smithi154.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookupOtherInstance 2024-09-16T03:44:59.335 INFO:tasks.workunit.client.0.smithi154.stdout: api_pool: [ OK ] LibRadosPools.PoolLookupOtherInstance (2986 ms) 2024-09-16T03:44:59.335 INFO:tasks.workunit.client.0.smithi154.stdout: api_pool: [ RUN ] LibRadosPools.PoolReverseLookupOtherInstance 2024-09-16T03:44:59.335 INFO:tasks.workunit.client.0.smithi154.stdout: api_pool: [ OK ] LibRadosPools.PoolReverseLookupOtherInstance (2985 ms) 2024-09-16T03:44:59.335 INFO:tasks.workunit.client.0.smithi154.stdout: api_pool: [ RUN ] LibRadosPools.PoolDelete 2024-09-16T03:44:59.335 INFO:tasks.workunit.client.0.smithi154.stdout: api_pool: [ OK ] LibRadosPools.PoolDelete (5022 ms) 2024-09-16T03:44:59.335 INFO:tasks.workunit.client.0.smithi154.stdout: api_pool: [ RUN ] LibRadosPools.PoolCreateDelete 2024-09-16T03:44:59.335 INFO:tasks.workunit.client.0.smithi154.stdout: api_pool: [ OK ] LibRadosPools.PoolCreateDelete (20029 ms) 2024-09-16T03:44:59.335 INFO:tasks.workunit.client.0.smithi154.stdout: api_pool: [ RUN ] LibRadosPools.PoolCreateWithCrushRule 2024-09-16T03:44:59.335 INFO:tasks.workunit.client.0.smithi154.stdout: api_pool: [ OK ] LibRadosPools.PoolCreateWithCrushRule (19294 ms) 2024-09-16T03:44:59.336 INFO:tasks.workunit.client.0.smithi154.stdout: api_pool: [ RUN ] LibRadosPools.PoolGetBaseTier 2024-09-16T03:44:59.336 INFO:tasks.workunit.client.0.smithi154.stdout: api_pool: [ OK ] LibRadosPools.PoolGetBaseTier (7774 ms) 2024-09-16T03:44:59.336 INFO:tasks.workunit.client.0.smithi154.stdout: api_pool: [----------] 9 tests from LibRadosPools (89704 ms total) 2024-09-16T03:44:59.336 INFO:tasks.workunit.client.0.smithi154.stdout: api_pool: 2024-09-16T03:44:59.336 INFO:tasks.workunit.client.0.smithi154.stdout: api_pool: [----------] Global test environment tear-down 2024-09-16T03:44:59.336 INFO:tasks.workunit.client.0.smithi154.stdout: api_pool: [==========] 9 tests from 1 test suite ran. (89704 ms total) 2024-09-16T03:44:59.336 INFO:tasks.workunit.client.0.smithi154.stdout: api_pool: [ PASSED ] 9 tests. 2024-09-16T03:45:02.335 INFO:tasks.workunit.client.0.smithi154.stdout: io: Running main() from gmock_main.cc 2024-09-16T03:45:02.335 INFO:tasks.workunit.client.0.smithi154.stdout: io: [==========] Running 14 tests from 1 test suite. 2024-09-16T03:45:02.335 INFO:tasks.workunit.client.0.smithi154.stdout: io: [----------] Global test environment set-up. 2024-09-16T03:45:02.335 INFO:tasks.workunit.client.0.smithi154.stdout: io: [----------] 14 tests from NeoRadosIo 2024-09-16T03:45:02.335 INFO:tasks.workunit.client.0.smithi154.stdout: io: [ RUN ] NeoRadosIo.Limits 2024-09-16T03:45:02.335 INFO:tasks.workunit.client.0.smithi154.stdout: io: [ OK ] NeoRadosIo.Limits (4657 ms) 2024-09-16T03:45:02.336 INFO:tasks.workunit.client.0.smithi154.stdout: io: [ RUN ] NeoRadosIo.SimpleWrite 2024-09-16T03:45:02.336 INFO:tasks.workunit.client.0.smithi154.stdout: io: [ OK ] NeoRadosIo.SimpleWrite (24017 ms) 2024-09-16T03:45:02.336 INFO:tasks.workunit.client.0.smithi154.stdout: io: [ RUN ] NeoRadosIo.ReadOp 2024-09-16T03:45:02.336 INFO:tasks.workunit.client.0.smithi154.stdout: io: [ OK ] NeoRadosIo.ReadOp (2775 ms) 2024-09-16T03:45:02.336 INFO:tasks.workunit.client.0.smithi154.stdout: io: [ RUN ] NeoRadosIo.SparseRead 2024-09-16T03:45:02.336 INFO:tasks.workunit.client.0.smithi154.stdout: io: [ OK ] NeoRadosIo.SparseRead (2987 ms) 2024-09-16T03:45:02.336 INFO:tasks.workunit.client.0.smithi154.stdout: io: [ RUN ] NeoRadosIo.RoundTrip 2024-09-16T03:45:02.336 INFO:tasks.workunit.client.0.smithi154.stdout: io: [ OK ] NeoRadosIo.RoundTrip (2975 ms) 2024-09-16T03:45:02.336 INFO:tasks.workunit.client.0.smithi154.stdout: io: [ RUN ] NeoRadosIo.ReadIntoBuufferlist 2024-09-16T03:45:02.336 INFO:tasks.workunit.client.0.smithi154.stdout: io: [ OK ] NeoRadosIo.ReadIntoBuufferlist (3030 ms) 2024-09-16T03:45:02.336 INFO:tasks.workunit.client.0.smithi154.stdout: io: [ RUN ] NeoRadosIo.OverlappingWriteRoundTrip 2024-09-16T03:45:02.336 INFO:tasks.workunit.client.0.smithi154.stdout: io: [ OK ] NeoRadosIo.OverlappingWriteRoundTrip (3008 ms) 2024-09-16T03:45:02.336 INFO:tasks.workunit.client.0.smithi154.stdout: io: [ RUN ] NeoRadosIo.WriteFullRoundTrip 2024-09-16T03:45:02.337 INFO:tasks.workunit.client.0.smithi154.stdout: io: [ OK ] NeoRadosIo.WriteFullRoundTrip (2954 ms) 2024-09-16T03:45:02.337 INFO:tasks.workunit.client.0.smithi154.stdout: io: [ RUN ] NeoRadosIo.AppendRoundTrip 2024-09-16T03:45:02.337 INFO:tasks.workunit.client.0.smithi154.stdout: io: [ OK ] NeoRadosIo.AppendRoundTrip (17104 ms) 2024-09-16T03:45:02.337 INFO:tasks.workunit.client.0.smithi154.stdout: io: [ RUN ] NeoRadosIo.Trunc 2024-09-16T03:45:02.337 INFO:tasks.workunit.client.0.smithi154.stdout: io: [ OK ] NeoRadosIo.Trunc (17301 ms) 2024-09-16T03:45:02.337 INFO:tasks.workunit.client.0.smithi154.stdout: io: [ RUN ] NeoRadosIo.Remove 2024-09-16T03:45:02.337 INFO:tasks.workunit.client.0.smithi154.stdout: io: [ OK ] NeoRadosIo.Remove (2737 ms) 2024-09-16T03:45:02.337 INFO:tasks.workunit.client.0.smithi154.stdout: io: [ RUN ] NeoRadosIo.XattrsRoundTrip 2024-09-16T03:45:02.337 INFO:tasks.workunit.client.0.smithi154.stdout: io: [ OK ] NeoRadosIo.XattrsRoundTrip (3023 ms) 2024-09-16T03:45:02.337 INFO:tasks.workunit.client.0.smithi154.stdout: io: [ RUN ] NeoRadosIo.RmXattr 2024-09-16T03:45:02.337 INFO:tasks.workunit.client.0.smithi154.stdout: io: [ OK ] NeoRadosIo.RmXattr (2990 ms) 2024-09-16T03:45:02.337 INFO:tasks.workunit.client.0.smithi154.stdout: io: [ RUN ] NeoRadosIo.GetXattrs 2024-09-16T03:45:02.337 INFO:tasks.workunit.client.0.smithi154.stdout: io: [ OK ] NeoRadosIo.GetXattrs (2984 ms) 2024-09-16T03:45:02.337 INFO:tasks.workunit.client.0.smithi154.stdout: io: [----------] 14 tests from NeoRadosIo (92543 ms total) 2024-09-16T03:45:02.338 INFO:tasks.workunit.client.0.smithi154.stdout: io: 2024-09-16T03:45:02.338 INFO:tasks.workunit.client.0.smithi154.stdout: io: [----------] Global test environment tear-down 2024-09-16T03:45:02.338 INFO:tasks.workunit.client.0.smithi154.stdout: io: [==========] 14 tests from 1 test suite ran. (92543 ms total) 2024-09-16T03:45:02.338 INFO:tasks.workunit.client.0.smithi154.stdout: io: [ PASSED ] 14 tests. 2024-09-16T03:45:02.338 INFO:tasks.workunit.client.0.smithi154.stdout: read_operations: Running main() from gmock_main.cc 2024-09-16T03:45:02.338 INFO:tasks.workunit.client.0.smithi154.stdout: read_operations: [==========] Running 14 tests from 1 test suite. 2024-09-16T03:45:02.338 INFO:tasks.workunit.client.0.smithi154.stdout: read_operations: [----------] Global test environment set-up. 2024-09-16T03:45:02.338 INFO:tasks.workunit.client.0.smithi154.stdout: read_operations: [----------] 14 tests from NeoRadosReadOps 2024-09-16T03:45:02.338 INFO:tasks.workunit.client.0.smithi154.stdout: read_operations: [ RUN ] NeoRadosReadOps.SetOpFlags 2024-09-16T03:45:02.338 INFO:tasks.workunit.client.0.smithi154.stdout: read_operations: [ OK ] NeoRadosReadOps.SetOpFlags (4626 ms) 2024-09-16T03:45:02.338 INFO:tasks.workunit.client.0.smithi154.stdout: read_operations: [ RUN ] NeoRadosReadOps.AssertExists 2024-09-16T03:45:02.338 INFO:tasks.workunit.client.0.smithi154.stdout: read_operations: [ OK ] NeoRadosReadOps.AssertExists (24025 ms) 2024-09-16T03:45:02.338 INFO:tasks.workunit.client.0.smithi154.stdout: read_operations: [ RUN ] NeoRadosReadOps.AssertVersion 2024-09-16T03:45:02.338 INFO:tasks.workunit.client.0.smithi154.stdout: read_operations: [ OK ] NeoRadosReadOps.AssertVersion (2766 ms) 2024-09-16T03:45:02.339 INFO:tasks.workunit.client.0.smithi154.stdout: read_operations: [ RUN ] NeoRadosReadOps.CmpXattr 2024-09-16T03:45:02.339 INFO:tasks.workunit.client.0.smithi154.stdout: read_operations: [ OK ] NeoRadosReadOps.CmpXattr (2988 ms) 2024-09-16T03:45:02.339 INFO:tasks.workunit.client.0.smithi154.stdout: read_operations: [ RUN ] NeoRadosReadOps.Read 2024-09-16T03:45:02.339 INFO:tasks.workunit.client.0.smithi154.stdout: read_operations: [ OK ] NeoRadosReadOps.Read (2974 ms) 2024-09-16T03:45:02.339 INFO:tasks.workunit.client.0.smithi154.stdout: read_operations: [ RUN ] NeoRadosReadOps.Checksum 2024-09-16T03:45:02.339 INFO:tasks.workunit.client.0.smithi154.stdout: read_operations: [ OK ] NeoRadosReadOps.Checksum (3056 ms) 2024-09-16T03:45:02.339 INFO:tasks.workunit.client.0.smithi154.stdout: read_operations: [ RUN ] NeoRadosReadOps.RWOrderedRead 2024-09-16T03:45:02.339 INFO:tasks.workunit.client.0.smithi154.stdout: read_operations: [ OK ] NeoRadosReadOps.RWOrderedRead (2978 ms) 2024-09-16T03:45:02.339 INFO:tasks.workunit.client.0.smithi154.stdout: read_operations: [ RUN ] NeoRadosReadOps.ShortRead 2024-09-16T03:45:02.339 INFO:tasks.workunit.client.0.smithi154.stdout: read_operations: [ OK ] NeoRadosReadOps.ShortRead (2951 ms) 2024-09-16T03:45:02.339 INFO:tasks.workunit.client.0.smithi154.stdout: read_operations: [ RUN ] NeoRadosReadOps.Exec 2024-09-16T03:45:02.339 INFO:tasks.workunit.client.0.smithi154.stdout: read_operations: [ OK ] NeoRadosReadOps.Exec (17105 ms) 2024-09-16T03:45:02.339 INFO:tasks.workunit.client.0.smithi154.stdout: read_operations: [ RUN ] NeoRadosReadOps.Stat 2024-09-16T03:45:02.339 INFO:tasks.workunit.client.0.smithi154.stdout: read_operations: [ OK ] NeoRadosReadOps.Stat (17316 ms) 2024-09-16T03:45:02.340 INFO:tasks.workunit.client.0.smithi154.stdout: read_operations: [ RUN ] NeoRadosReadOps.Omap 2024-09-16T03:45:02.340 INFO:tasks.workunit.client.0.smithi154.stdout: read_operations: [ OK ] NeoRadosReadOps.Omap (2734 ms) 2024-09-16T03:45:02.340 INFO:tasks.workunit.client.0.smithi154.stdout: read_operations: [ RUN ] NeoRadosReadOps.OmapNuls 2024-09-16T03:45:02.340 INFO:tasks.workunit.client.0.smithi154.stdout: read_operations: [ OK ] NeoRadosReadOps.OmapNuls (3025 ms) 2024-09-16T03:45:02.340 INFO:tasks.workunit.client.0.smithi154.stdout: read_operations: [ RUN ] NeoRadosReadOps.GetXattrs 2024-09-16T03:45:02.340 INFO:tasks.workunit.client.0.smithi154.stdout: read_operations: [ OK ] NeoRadosReadOps.GetXattrs (2973 ms) 2024-09-16T03:45:02.340 INFO:tasks.workunit.client.0.smithi154.stdout: read_operations: [ RUN ] NeoRadosReadOps.CmpExt 2024-09-16T03:45:02.340 INFO:tasks.workunit.client.0.smithi154.stdout: read_operations: [ OK ] NeoRadosReadOps.CmpExt (2995 ms) 2024-09-16T03:45:02.340 INFO:tasks.workunit.client.0.smithi154.stdout: read_operations: [----------] 14 tests from NeoRadosReadOps (92512 ms total) 2024-09-16T03:45:02.340 INFO:tasks.workunit.client.0.smithi154.stdout: read_operations: 2024-09-16T03:45:02.340 INFO:tasks.workunit.client.0.smithi154.stdout: read_operations: [----------] Global test environment tear-down 2024-09-16T03:45:02.340 INFO:tasks.workunit.client.0.smithi154.stdout: read_operations: [==========] 14 tests from 1 test suite ran. (92513 ms total) 2024-09-16T03:45:02.340 INFO:tasks.workunit.client.0.smithi154.stdout: read_operations: [ PASSED ] 14 tests. 2024-09-16T03:45:07.337 INFO:tasks.workunit.client.0.smithi154.stdout: api_misc_pp: [==========] Running 31 tests from 7 test suites. 2024-09-16T03:45:07.338 INFO:tasks.workunit.client.0.smithi154.stdout: api_misc_pp: [----------] Global test environment set-up. 2024-09-16T03:45:07.338 INFO:tasks.workunit.client.0.smithi154.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscVersion 2024-09-16T03:45:07.338 INFO:tasks.workunit.client.0.smithi154.stdout: api_misc_pp: [ RUN ] LibRadosMiscVersion.VersionPP 2024-09-16T03:45:07.338 INFO:tasks.workunit.client.0.smithi154.stdout: api_misc_pp: [ OK ] LibRadosMiscVersion.VersionPP (0 ms) 2024-09-16T03:45:07.338 INFO:tasks.workunit.client.0.smithi154.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscVersion (0 ms total) 2024-09-16T03:45:07.338 INFO:tasks.workunit.client.0.smithi154.stdout: api_misc_pp: 2024-09-16T03:45:07.338 INFO:tasks.workunit.client.0.smithi154.stdout: api_misc_pp: [----------] 22 tests from LibRadosMiscPP 2024-09-16T03:45:07.338 INFO:tasks.workunit.client.0.smithi154.stdout: api_misc_pp: seed 28198 2024-09-16T03:45:07.338 INFO:tasks.workunit.client.0.smithi154.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.WaitOSDMapPP 2024-09-16T03:45:07.338 INFO:tasks.workunit.client.0.smithi154.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.WaitOSDMapPP (0 ms) 2024-09-16T03:45:07.338 INFO:tasks.workunit.client.0.smithi154.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongNamePP 2024-09-16T03:45:07.338 INFO:tasks.workunit.client.0.smithi154.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongNamePP (221 ms) 2024-09-16T03:45:07.339 INFO:tasks.workunit.client.0.smithi154.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongLocatorPP 2024-09-16T03:45:07.339 INFO:tasks.workunit.client.0.smithi154.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongLocatorPP (115 ms) 2024-09-16T03:45:07.339 INFO:tasks.workunit.client.0.smithi154.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongNSpacePP 2024-09-16T03:45:07.339 INFO:tasks.workunit.client.0.smithi154.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongNSpacePP (19 ms) 2024-09-16T03:45:07.339 INFO:tasks.workunit.client.0.smithi154.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongAttrNamePP 2024-09-16T03:45:07.339 INFO:tasks.workunit.client.0.smithi154.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongAttrNamePP (9 ms) 2024-09-16T03:45:07.339 INFO:tasks.workunit.client.0.smithi154.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.ExecPP 2024-09-16T03:45:07.339 INFO:tasks.workunit.client.0.smithi154.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.ExecPP (3 ms) 2024-09-16T03:45:07.339 INFO:tasks.workunit.client.0.smithi154.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BadFlagsPP 2024-09-16T03:45:07.339 INFO:tasks.workunit.client.0.smithi154.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BadFlagsPP (3 ms) 2024-09-16T03:45:07.339 INFO:tasks.workunit.client.0.smithi154.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Operate1PP 2024-09-16T03:45:07.339 INFO:tasks.workunit.client.0.smithi154.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Operate1PP (5 ms) 2024-09-16T03:45:07.339 INFO:tasks.workunit.client.0.smithi154.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Operate2PP 2024-09-16T03:45:07.339 INFO:tasks.workunit.client.0.smithi154.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Operate2PP (3 ms) 2024-09-16T03:45:07.340 INFO:tasks.workunit.client.0.smithi154.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BigObjectPP 2024-09-16T03:45:07.340 INFO:tasks.workunit.client.0.smithi154.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BigObjectPP (11 ms) 2024-09-16T03:45:07.340 INFO:tasks.workunit.client.0.smithi154.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AioOperatePP 2024-09-16T03:45:07.340 INFO:tasks.workunit.client.0.smithi154.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AioOperatePP (3 ms) 2024-09-16T03:45:07.340 INFO:tasks.workunit.client.0.smithi154.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AssertExistsPP 2024-09-16T03:45:07.340 INFO:tasks.workunit.client.0.smithi154.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AssertExistsPP (14 ms) 2024-09-16T03:45:07.340 INFO:tasks.workunit.client.0.smithi154.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AssertVersionPP 2024-09-16T03:45:07.340 INFO:tasks.workunit.client.0.smithi154.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AssertVersionPP (18 ms) 2024-09-16T03:45:07.340 INFO:tasks.workunit.client.0.smithi154.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BigAttrPP 2024-09-16T03:45:07.340 INFO:tasks.workunit.client.0.smithi154.stdout: api_misc_pp: osd_max_attr_size = 0 2024-09-16T03:45:07.340 INFO:tasks.workunit.client.0.smithi154.stdout: api_misc_pp: osd_max_attr_size == 0; skipping test 2024-09-16T03:45:07.340 INFO:tasks.workunit.client.0.smithi154.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BigAttrPP (3405 ms) 2024-09-16T03:45:07.340 INFO:tasks.workunit.client.0.smithi154.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CopyPP 2024-09-16T03:45:07.340 INFO:tasks.workunit.client.0.smithi154.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CopyPP (382 ms) 2024-09-16T03:45:07.341 INFO:tasks.workunit.client.0.smithi154.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CopyScrubPP 2024-09-16T03:45:07.341 INFO:tasks.workunit.client.0.smithi154.stdout: api_misc_pp: waiting for initial deep scrubs... 2024-09-16T03:45:07.341 INFO:tasks.workunit.client.0.smithi154.stdout: api_misc_pp: done waiting, doing copies 2024-09-16T03:45:07.341 INFO:tasks.workunit.client.0.smithi154.stdout: api_misc_pp: waiting for final deep scrubs... 2024-09-16T03:45:07.341 INFO:tasks.workunit.client.0.smithi154.stdout: api_misc_pp: done waiting 2024-09-16T03:45:07.341 INFO:tasks.workunit.client.0.smithi154.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CopyScrubPP (84727 ms) 2024-09-16T03:45:07.341 INFO:tasks.workunit.client.0.smithi154.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.WriteSamePP 2024-09-16T03:45:07.341 INFO:tasks.workunit.client.0.smithi154.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.WriteSamePP (15 ms) 2024-09-16T03:45:07.341 INFO:tasks.workunit.client.0.smithi154.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CmpExtPP 2024-09-16T03:45:07.341 INFO:tasks.workunit.client.0.smithi154.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CmpExtPP (8 ms) 2024-09-16T03:45:07.341 INFO:tasks.workunit.client.0.smithi154.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Applications 2024-09-16T03:45:07.341 INFO:tasks.workunit.client.0.smithi154.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Applications (4112 ms) 2024-09-16T03:45:07.341 INFO:tasks.workunit.client.0.smithi154.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.MinCompatOSD 2024-09-16T03:45:07.341 INFO:tasks.workunit.client.0.smithi154.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.MinCompatOSD (0 ms) 2024-09-16T03:45:07.342 INFO:tasks.workunit.client.0.smithi154.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.MinCompatClient 2024-09-16T03:45:07.342 INFO:tasks.workunit.client.0.smithi154.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.MinCompatClient (0 ms) 2024-09-16T03:45:07.342 INFO:tasks.workunit.client.0.smithi154.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Conf 2024-09-16T03:45:07.342 INFO:tasks.workunit.client.0.smithi154.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Conf (0 ms) 2024-09-16T03:45:12.521 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-16T03:45:12.522 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-09-16T03:45:12.522 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-16T03:45:12.522 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-16T03:45:12.525 INFO:tasks.ceph.mon.a:Started 2024-09-16T03:45:12.525 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T03:45:12.525 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:45:12.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.669+0000 7f2d3a629640 10 monclient: get_monmap_and_config 2024-09-16T03:45:12.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.669+0000 7f2d3a629640 10 monclient: build_initial_monmap 2024-09-16T03:45:12.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.669+0000 7f2d3a629640 10 monclient: monmap: 2024-09-16T03:45:12.676 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:45:12.676 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:45:12.676 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:45:12.674987+0000 2024-09-16T03:45:12.676 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:45:12.674987+0000 2024-09-16T03:45:12.676 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:45:12.676 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:45:12.676 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:45:12.676 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:45:12.676 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:45:12.676 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:45:12.676 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:45:12.676 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:45:12.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.669+0000 7f2d3a629640 1 Processor -- start 2024-09-16T03:45:12.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.669+0000 7f2d3a629640 1 -- start start 2024-09-16T03:45:12.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.669+0000 7f2d3a629640 10 monclient: init 2024-09-16T03:45:12.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.669+0000 7f2d3a629640 10 monclient: _reopen_session rank -1 2024-09-16T03:45:12.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.669+0000 7f2d3a629640 10 monclient: _add_conns ranks=[2,4,1,0,3] 2024-09-16T03:45:12.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.673+0000 7f2d3a629640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2d340a39b0 0x7f2d34133b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:45:12.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.673+0000 7f2d3a629640 10 monclient(hunting): picked mon.noname-b con 0x7f2d340a39b0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:45:12.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.673+0000 7f2d3a629640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f2d340a3d90 0x7f2d34137000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:45:12.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.673+0000 7f2d3a629640 10 monclient(hunting): picked mon.noname-c con 0x7f2d340a3d90 addr v2:172.21.15.154:3300/0 2024-09-16T03:45:12.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.673+0000 7f2d3a629640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2d34137540 addr v1:172.21.15.154:6789/0 2024-09-16T03:45:12.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.673+0000 7f2d3a629640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2d3409c140 con 0x7f2d34137540 2024-09-16T03:45:12.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.673+0000 7f2d3a629640 10 monclient(hunting): start opening mon connection 2024-09-16T03:45:12.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.673+0000 7f2d3a629640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f2d3409c420 con 0x7f2d340a3d90 2024-09-16T03:45:12.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.673+0000 7f2d3a629640 10 monclient(hunting): start opening mon connection 2024-09-16T03:45:12.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.673+0000 7f2d3a629640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f2d3406dd60 con 0x7f2d340a39b0 2024-09-16T03:45:12.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.673+0000 7f2d3a629640 10 monclient(hunting): _renew_subs 2024-09-16T03:45:12.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.673+0000 7f2d3a629640 10 monclient(hunting): authenticate will time out at 1159.644087s 2024-09-16T03:45:12.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.673+0000 7f2d33fff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2d340a39b0 0x7f2d34133b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:45:12.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.673+0000 7f2d33fff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2d340a39b0 0x7f2d34133b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:55702/0 (socket says 172.21.15.42:55702) 2024-09-16T03:45:12.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.673+0000 7f2d33fff640 1 -- 172.21.15.42:0/2824809850 learned_addr learned my addr 172.21.15.42:0/2824809850 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:45:12.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.673+0000 7f2d33fff640 10 monclient(hunting): get_auth_request con 0x7f2d340a39b0 auth_method 0 2024-09-16T03:45:12.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.673+0000 7f2d33fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:45:12.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.673+0000 7f2d33fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:45:12.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.673+0000 7f2d33fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:45:12.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.673+0000 7f2d33fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:45:12.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.673+0000 7f2d33fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:45:12.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.673+0000 7f2d33fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:45:12.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.673+0000 7f2d33fff640 10 monclient(hunting): handle_auth_done global_id 13256 payload 274 2024-09-16T03:45:12.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.673+0000 7f2d33fff640 1 -- 172.21.15.42:0/2824809850 >> v2:172.21.15.154:3300/0 conn(0x7f2d340a3d90 msgr2=0x7f2d34137000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:45:12.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.673+0000 7f2d33fff640 1 --2- 172.21.15.42:0/2824809850 >> v2:172.21.15.154:3300/0 conn(0x7f2d340a3d90 0x7f2d34137000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:45:12.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.673+0000 7f2d33fff640 1 -- 172.21.15.42:0/2824809850 >> v1:172.21.15.154:6789/0 conn(0x7f2d34137540 legacy=0x7f2d3413e390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:45:12.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.673+0000 7f2d33fff640 10 monclient: _finish_hunting 0 2024-09-16T03:45:12.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.673+0000 7f2d33fff640 1 monclient: found mon.noname-b 2024-09-16T03:45:12.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.673+0000 7f2d33fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:45:12.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.673+0000 7f2d33fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T03:45:12.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.673+0000 7f2d33fff640 1 -- 172.21.15.42:0/2824809850 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d340a5e20 con 0x7f2d340a39b0 2024-09-16T03:45:12.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.673+0000 7f2d33fff640 1 --2- 172.21.15.42:0/2824809850 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2d340a39b0 0x7f2d34133b80 secure :-1 s=READY pgs=2331 cs=0 l=1 rev1=1 crypto rx=0x7f2d24009870 tx=0x7f2d2401c3e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a4145c5ad927b727 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:45:12.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.673+0000 7f2d32ffd640 1 -- 172.21.15.42:0/2824809850 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d240b0070 con 0x7f2d340a39b0 2024-09-16T03:45:12.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.673+0000 7f2d32ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:45:12.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.673+0000 7f2d32ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:45:12.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.673+0000 7f2d32ffd640 10 monclient: dump: 2024-09-16T03:45:12.687 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:45:12.687 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:45:12.687 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:12.687 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:12.687 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:45:12.687 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:45:12.688 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:45:12.688 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:45:12.688 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:45:12.688 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:45:12.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.673+0000 7f2d32ffd640 10 monclient: _finish_auth 0 2024-09-16T03:45:12.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.673+0000 7f2d32ffd640 10 monclient: _check_auth_tickets 2024-09-16T03:45:12.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.673+0000 7f2d32ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:45:12.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.673+0000 7f2d32ffd640 1 -- 172.21.15.42:0/2824809850 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d240ab600 con 0x7f2d340a39b0 2024-09-16T03:45:12.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.673+0000 7f2d3a629640 5 monclient: authenticate success, global_id 13256 2024-09-16T03:45:12.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.673+0000 7f2d3a629640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:45:12.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.673+0000 7f2d32ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:45:12.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.673+0000 7f2d3a629640 10 monclient: get_monmap_and_config success 2024-09-16T03:45:12.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.673+0000 7f2d32ffd640 1 -- 172.21.15.42:0/2824809850 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d240ab9d0 con 0x7f2d340a39b0 2024-09-16T03:45:12.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.673+0000 7f2d32ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:45:12.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.673+0000 7f2d3a629640 4 set_mon_vals no callback set 2024-09-16T03:45:12.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.673+0000 7f2d32ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:45:12.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.673+0000 7f2d32ffd640 10 monclient: dump: 2024-09-16T03:45:12.689 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:45:12.689 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:45:12.689 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:12.689 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:12.689 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:45:12.689 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:45:12.689 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:45:12.689 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:45:12.689 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:45:12.690 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:45:12.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.673+0000 7f2d3a629640 10 monclient: shutdown 2024-09-16T03:45:12.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.673+0000 7f2d3a629640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:45:12.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.673+0000 7f2d3a629640 1 -- 172.21.15.42:0/2824809850 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2d340a39b0 msgr2=0x7f2d34133b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:45:12.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.673+0000 7f2d3a629640 1 --2- 172.21.15.42:0/2824809850 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2d340a39b0 0x7f2d34133b80 secure :-1 s=READY pgs=2331 cs=0 l=1 rev1=1 crypto rx=0x7f2d24009870 tx=0x7f2d2401c3e0 comp rx=0 tx=0).stop 2024-09-16T03:45:12.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.673+0000 7f2d3a629640 1 -- 172.21.15.42:0/2824809850 shutdown_connections 2024-09-16T03:45:12.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.673+0000 7f2d3a629640 1 --2- 172.21.15.42:0/2824809850 >> v2:172.21.15.154:3300/0 conn(0x7f2d340a3d90 0x7f2d34137000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:45:12.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.673+0000 7f2d3a629640 1 --2- 172.21.15.42:0/2824809850 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2d340a39b0 0x7f2d34133b80 unknown :-1 s=CLOSED pgs=2331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:45:12.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.673+0000 7f2d3a629640 1 -- 172.21.15.42:0/2824809850 >> 172.21.15.42:0/2824809850 conn(0x7f2d340a9fb0 msgr2=0x7f2d340aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:45:12.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.673+0000 7f2d3a629640 1 -- 172.21.15.42:0/2824809850 shutdown_connections 2024-09-16T03:45:12.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.673+0000 7f2d3a629640 1 -- 172.21.15.42:0/2824809850 wait complete. 2024-09-16T03:45:12.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.677+0000 7f2d3a629640 10 monclient: build_initial_monmap 2024-09-16T03:45:12.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.677+0000 7f2d3a629640 10 monclient: monmap: 2024-09-16T03:45:12.691 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:45:12.691 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:45:12.691 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:45:12.691 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:45:12.691 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:45:12.691 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:45:12.691 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:45:12.691 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:45:12.691 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:45:12.691 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:45:12.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.677+0000 7f2d3a629640 1 librados: starting msgr at 2024-09-16T03:45:12.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.677+0000 7f2d3a629640 1 librados: starting objecter 2024-09-16T03:45:12.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.677+0000 7f2d3a629640 1 Processor -- start 2024-09-16T03:45:12.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.677+0000 7f2d3a629640 1 -- start start 2024-09-16T03:45:12.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.677+0000 7f2d3a629640 1 librados: setting wanted keys 2024-09-16T03:45:12.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.677+0000 7f2d3a629640 1 librados: calling monclient init 2024-09-16T03:45:12.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.677+0000 7f2d3a629640 10 monclient: init 2024-09-16T03:45:12.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.677+0000 7f2d3a629640 10 monclient: _reopen_session rank -1 2024-09-16T03:45:12.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.677+0000 7f2d3a629640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T03:45:12.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.677+0000 7f2d3a629640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d340a39b0 0x7f2d34130560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:45:12.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.677+0000 7f2d3a629640 10 monclient(hunting): picked mon.noname-b con 0x7f2d340a39b0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:45:12.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.677+0000 7f2d3a629640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2d340a3d90 0x7f2d34130aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:45:12.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.677+0000 7f2d3a629640 10 monclient(hunting): picked mon.noname-c con 0x7f2d340a3d90 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:45:12.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.677+0000 7f2d3a629640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2d34137540 0x7f2d3413d090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:45:12.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.677+0000 7f2d3a629640 10 monclient(hunting): picked mon.noname-a con 0x7f2d34137540 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:45:12.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.677+0000 7f2d3a629640 10 monclient(hunting): start opening mon connection 2024-09-16T03:45:12.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.677+0000 7f2d3a629640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f2d341424e0 con 0x7f2d34137540 2024-09-16T03:45:12.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.677+0000 7f2d3a629640 10 monclient(hunting): start opening mon connection 2024-09-16T03:45:12.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.677+0000 7f2d3a629640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f2d34142660 con 0x7f2d340a39b0 2024-09-16T03:45:12.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.677+0000 7f2d3a629640 10 monclient(hunting): start opening mon connection 2024-09-16T03:45:12.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.677+0000 7f2d3a629640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f2d3409d040 con 0x7f2d340a3d90 2024-09-16T03:45:12.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.677+0000 7f2d3a629640 10 monclient(hunting): _renew_subs 2024-09-16T03:45:12.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.677+0000 7f2d3a629640 10 monclient(hunting): authenticate will time out at 1159.649093s 2024-09-16T03:45:12.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.677+0000 7f2d337fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2d340a3d90 0x7f2d34130aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:45:12.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.677+0000 7f2d33fff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d340a39b0 0x7f2d34130560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:45:12.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.677+0000 7f2d38bc6640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2d34137540 0x7f2d3413d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:45:12.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.677+0000 7f2d337fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2d340a3d90 0x7f2d34130aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:55712/0 (socket says 172.21.15.42:55712) 2024-09-16T03:45:12.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.677+0000 7f2d337fe640 1 -- 172.21.15.42:0/2872780876 learned_addr learned my addr 172.21.15.42:0/2872780876 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:45:12.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.677+0000 7f2d337fe640 10 monclient(hunting): get_auth_request con 0x7f2d340a3d90 auth_method 0 2024-09-16T03:45:12.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.677+0000 7f2d337fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:45:12.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.677+0000 7f2d337fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:45:12.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.677+0000 7f2d337fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:45:12.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.677+0000 7f2d33fff640 10 monclient(hunting): get_auth_request con 0x7f2d340a39b0 auth_method 0 2024-09-16T03:45:12.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.677+0000 7f2d33fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:45:12.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.677+0000 7f2d33fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:45:12.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.677+0000 7f2d33fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:45:12.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.677+0000 7f2d38bc6640 10 monclient(hunting): get_auth_request con 0x7f2d34137540 auth_method 0 2024-09-16T03:45:12.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.677+0000 7f2d38bc6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:45:12.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.677+0000 7f2d38bc6640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:45:12.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.677+0000 7f2d38bc6640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:45:12.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.677+0000 7f2d337fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:45:12.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.677+0000 7f2d337fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:45:12.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.677+0000 7f2d337fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:45:12.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.677+0000 7f2d38bc6640 1 -- 172.21.15.42:0/2872780876 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2d34137540 msgr2=0x7f2d3413d090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-16T03:45:12.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.677+0000 7f2d38bc6640 1 -- 172.21.15.42:0/2872780876 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2d34137540 msgr2=0x7f2d3413d090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T03:45:12.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.677+0000 7f2d38bc6640 1 --2- 172.21.15.42:0/2872780876 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2d34137540 0x7f2d3413d090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:45:12.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.677+0000 7f2d33fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:45:12.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.677+0000 7f2d33fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:45:12.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.677+0000 7f2d38bc6640 1 --2- 172.21.15.42:0/2872780876 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2d34137540 0x7f2d3413d090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:45:12.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.677+0000 7f2d33fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:45:12.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.677+0000 7f2d337fe640 10 monclient(hunting): handle_auth_done global_id 13259 payload 824 2024-09-16T03:45:12.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.677+0000 7f2d337fe640 1 -- 172.21.15.42:0/2872780876 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d340a39b0 msgr2=0x7f2d34130560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:45:12.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.677+0000 7f2d337fe640 1 --2- 172.21.15.42:0/2872780876 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d340a39b0 0x7f2d34130560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:45:12.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.677+0000 7f2d337fe640 1 -- 172.21.15.42:0/2872780876 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2d34137540 msgr2=0x7f2d3413d090 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:45:12.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.677+0000 7f2d337fe640 1 --2- 172.21.15.42:0/2872780876 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2d34137540 0x7f2d3413d090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:45:12.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.677+0000 7f2d337fe640 10 monclient: _finish_hunting 0 2024-09-16T03:45:12.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.677+0000 7f2d337fe640 1 monclient: found mon.noname-c 2024-09-16T03:45:12.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.677+0000 7f2d337fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:45:12.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.677+0000 7f2d337fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:45:12.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.677+0000 7f2d337fe640 1 -- 172.21.15.42:0/2872780876 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d3409e4f0 con 0x7f2d340a3d90 2024-09-16T03:45:12.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.677+0000 7f2d337fe640 1 --2- 172.21.15.42:0/2872780876 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2d340a3d90 0x7f2d34130aa0 secure :-1 s=READY pgs=2332 cs=0 l=1 rev1=1 crypto rx=0x7f2d2800b580 tx=0x7f2d2800ba40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:45:12.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.677+0000 7f2d30ff9640 10 client.?.objecter ms_handle_connect 0x7f2d340a3d90 2024-09-16T03:45:12.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.677+0000 7f2d30ff9640 10 client.?.objecter resend_mon_ops 2024-09-16T03:45:12.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.677+0000 7f2d30ff9640 1 -- 172.21.15.42:0/2872780876 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d28014070 con 0x7f2d340a3d90 2024-09-16T03:45:12.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.677+0000 7f2d30ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:45:12.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.677+0000 7f2d30ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:45:12.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.677+0000 7f2d30ff9640 10 monclient: dump: 2024-09-16T03:45:12.696 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:45:12.697 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:45:12.697 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:12.697 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:12.697 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:45:12.697 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:45:12.697 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:45:12.697 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:45:12.697 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:45:12.697 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:45:12.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.677+0000 7f2d30ff9640 10 monclient: _finish_auth 0 2024-09-16T03:45:12.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.677+0000 7f2d30ff9640 10 monclient: _check_auth_tickets 2024-09-16T03:45:12.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.677+0000 7f2d30ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:45:12.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.677+0000 7f2d30ff9640 1 -- 172.21.15.42:0/2872780876 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d28010890 con 0x7f2d340a3d90 2024-09-16T03:45:12.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.677+0000 7f2d3a629640 5 monclient: authenticate success, global_id 13259 2024-09-16T03:45:12.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.677+0000 7f2d3a629640 10 monclient: _renew_subs 2024-09-16T03:45:12.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.677+0000 7f2d3a629640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:45:12.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.677+0000 7f2d3a629640 1 -- 172.21.15.42:0/2872780876 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d3409e7d0 con 0x7f2d340a3d90 2024-09-16T03:45:12.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.677+0000 7f2d30ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T03:45:12.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.677+0000 7f2d30ff9640 1 -- 172.21.15.42:0/2872780876 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d28010c80 con 0x7f2d340a3d90 2024-09-16T03:45:12.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.677+0000 7f2d30ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:45:12.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.677+0000 7f2d30ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:45:12.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.677+0000 7f2d3a629640 10 client.13259.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:45:12.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.677+0000 7f2d30ff9640 10 monclient: dump: 2024-09-16T03:45:12.698 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:45:12.698 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:45:12.698 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:12.698 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:12.699 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:45:12.699 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:45:12.699 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:45:12.699 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:45:12.699 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:45:12.699 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:45:12.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.677+0000 7f2d3a629640 10 monclient: _renew_subs 2024-09-16T03:45:12.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.677+0000 7f2d327fc640 4 set_mon_vals no callback set 2024-09-16T03:45:12.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.677+0000 7f2d3a629640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:45:12.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.677+0000 7f2d3a629640 1 -- 172.21.15.42:0/2872780876 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d3406f680 con 0x7f2d340a3d90 2024-09-16T03:45:12.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.677+0000 7f2d3a629640 1 librados: init done 2024-09-16T03:45:12.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.677+0000 7f2d3a629640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:45:12.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.677+0000 7f2d3a629640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:45:12.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.677+0000 7f2d3a629640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:45:12.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.677+0000 7f2d30ff9640 1 -- 172.21.15.42:0/2872780876 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f2d28017070 con 0x7f2d340a3d90 2024-09-16T03:45:12.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.677+0000 7f2d3a629640 1 -- 172.21.15.42:0/2872780876 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2d34133b80 con 0x7f2d340a3d90 2024-09-16T03:45:12.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.677+0000 7f2d30ff9640 1 --2- 172.21.15.42:0/2872780876 >> v2:172.21.15.154:6812/254845342 conn(0x7f2d20037810 0x7f2d20039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:45:12.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.677+0000 7f2d30ff9640 1 -- 172.21.15.42:0/2872780876 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(80..80 src has 1..80) ==== 8822+0+0 (secure 0 0 0) 0x7f2d28049990 con 0x7f2d340a3d90 2024-09-16T03:45:12.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.677+0000 7f2d30ff9640 10 client.13259.objecter ms_dispatch 0x7f2d34000ba0 osd_map(80..80 src has 1..80) 2024-09-16T03:45:12.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.677+0000 7f2d30ff9640 3 client.13259.objecter handle_osd_map got epochs [80,80] > 0 2024-09-16T03:45:12.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.677+0000 7f2d30ff9640 3 client.13259.objecter handle_osd_map decoding full epoch 80 2024-09-16T03:45:12.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.677+0000 7f2d30ff9640 20 client.13259.objecter dump_active .. 0 homeless 2024-09-16T03:45:12.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.681+0000 7f2d33fff640 1 --2- 172.21.15.42:0/2872780876 >> v2:172.21.15.154:6812/254845342 conn(0x7f2d20037810 0x7f2d20039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:45:12.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.681+0000 7f2d33fff640 10 monclient: get_auth_request con 0x7f2d20037810 auth_method 0 2024-09-16T03:45:12.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.681+0000 7f2d30ff9640 10 client.13259.objecter ms_handle_connect 0x7f2d20037810 2024-09-16T03:45:12.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.681+0000 7f2d33fff640 1 --2- 172.21.15.42:0/2872780876 >> v2:172.21.15.154:6812/254845342 conn(0x7f2d20037810 0x7f2d20039cd0 secure :-1 s=READY pgs=3346 cs=0 l=1 rev1=1 crypto rx=0x7f2d240ad000 tx=0x7f2d24008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:45:12.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.685+0000 7f2d30ff9640 1 -- 172.21.15.42:0/2872780876 <== mon.2 v2:172.21.15.42:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f2d28010e20 con 0x7f2d340a3d90 2024-09-16T03:45:12.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.685+0000 7f2d30ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:45:12.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.685+0000 7f2d30ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:45:12.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.913+0000 7f2d3a629640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:45:12.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.913+0000 7f2d3a629640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:45:12.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.913+0000 7f2d3a629640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:45:12.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.913+0000 7f2d3a629640 1 -- 172.21.15.42:0/2872780876 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f2d340a5750 con 0x7f2d340a3d90 2024-09-16T03:45:12.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.913+0000 7f2d30ff9640 1 -- 172.21.15.42:0/2872780876 <== mon.2 v2:172.21.15.42:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1555 (secure 0 0 0) 0x7f2d28018cb0 con 0x7f2d340a3d90 2024-09-16T03:45:12.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.913+0000 7f2d30ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:45:12.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.913+0000 7f2d30ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:45:12.922 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":24,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":22,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:45:12.923 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.913+0000 7f2d3a629640 10 librados: watch_flush enter 2024-09-16T03:45:12.923 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.917+0000 7f2d3a629640 10 librados: watch_flush exit 2024-09-16T03:45:12.923 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.917+0000 7f2d3a629640 20 client.13259.objecter shutdown clearing up homeless session... 2024-09-16T03:45:12.923 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.917+0000 7f2d3a629640 10 client.13259.objecter successfully canceled tick 2024-09-16T03:45:12.923 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.917+0000 7f2d3a629640 1 -- 172.21.15.42:0/2872780876 >> v2:172.21.15.154:6812/254845342 conn(0x7f2d20037810 msgr2=0x7f2d20039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:45:12.923 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.917+0000 7f2d3a629640 1 --2- 172.21.15.42:0/2872780876 >> v2:172.21.15.154:6812/254845342 conn(0x7f2d20037810 0x7f2d20039cd0 secure :-1 s=READY pgs=3346 cs=0 l=1 rev1=1 crypto rx=0x7f2d240ad000 tx=0x7f2d24008040 comp rx=0 tx=0).stop 2024-09-16T03:45:12.923 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.917+0000 7f2d3a629640 10 monclient: shutdown 2024-09-16T03:45:12.924 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.917+0000 7f2d3a629640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:45:12.924 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.917+0000 7f2d3a629640 1 -- 172.21.15.42:0/2872780876 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2d340a3d90 msgr2=0x7f2d34130aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:45:12.924 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.917+0000 7f2d3a629640 1 --2- 172.21.15.42:0/2872780876 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2d340a3d90 0x7f2d34130aa0 secure :-1 s=READY pgs=2332 cs=0 l=1 rev1=1 crypto rx=0x7f2d2800b580 tx=0x7f2d2800ba40 comp rx=0 tx=0).stop 2024-09-16T03:45:12.924 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.917+0000 7f2d3a629640 1 -- 172.21.15.42:0/2872780876 shutdown_connections 2024-09-16T03:45:12.924 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.917+0000 7f2d3a629640 1 --2- 172.21.15.42:0/2872780876 >> v2:172.21.15.154:6812/254845342 conn(0x7f2d20037810 0x7f2d20039cd0 unknown :-1 s=CLOSED pgs=3346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:45:12.924 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.917+0000 7f2d3a629640 1 --2- 172.21.15.42:0/2872780876 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2d34137540 0x7f2d3413d090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:45:12.924 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.917+0000 7f2d3a629640 1 --2- 172.21.15.42:0/2872780876 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2d340a3d90 0x7f2d34130aa0 unknown :-1 s=CLOSED pgs=2332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:45:12.924 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.917+0000 7f2d3a629640 1 --2- 172.21.15.42:0/2872780876 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d340a39b0 0x7f2d34130560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:45:12.924 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.917+0000 7f2d3a629640 1 -- 172.21.15.42:0/2872780876 >> 172.21.15.42:0/2872780876 conn(0x7f2d340a9fb0 msgr2=0x7f2d34131980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:45:12.924 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.917+0000 7f2d3a629640 1 -- 172.21.15.42:0/2872780876 shutdown_connections 2024-09-16T03:45:12.924 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.917+0000 7f2d3a629640 1 -- 172.21.15.42:0/2872780876 wait complete. 2024-09-16T03:45:12.924 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:12.917+0000 7f2d3a629640 1 librados: shutdown 2024-09-16T03:45:15.939 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:45:16.071 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.065+0000 7f6cd2659640 10 monclient: get_monmap_and_config 2024-09-16T03:45:16.071 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.065+0000 7f6cd2659640 10 monclient: build_initial_monmap 2024-09-16T03:45:16.071 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.065+0000 7f6cd2659640 10 monclient: monmap: 2024-09-16T03:45:16.071 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:45:16.071 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:45:16.071 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:45:16.070153+0000 2024-09-16T03:45:16.071 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:45:16.070153+0000 2024-09-16T03:45:16.071 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:45:16.071 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:45:16.071 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:45:16.071 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:45:16.072 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:45:16.072 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:45:16.072 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:45:16.072 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:45:16.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.065+0000 7f6cd2659640 1 Processor -- start 2024-09-16T03:45:16.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.065+0000 7f6cd2659640 1 -- start start 2024-09-16T03:45:16.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.065+0000 7f6cd2659640 10 monclient: init 2024-09-16T03:45:16.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.065+0000 7f6cd2659640 10 monclient: _reopen_session rank -1 2024-09-16T03:45:16.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.065+0000 7f6cd2659640 10 monclient: _add_conns ranks=[2,1,4,3,0] 2024-09-16T03:45:16.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.065+0000 7f6cd2659640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f6ccc0a1330 0x7f6ccc09f180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:45:16.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.065+0000 7f6cd2659640 10 monclient(hunting): picked mon.noname-b con 0x7f6ccc0a1330 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:45:16.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.065+0000 7f6cd2659640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6ccc0a1710 addr v1:172.21.15.154:6789/0 2024-09-16T03:45:16.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.065+0000 7f6cd2659640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f6ccc1393a0 0x7f6ccc13b790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:45:16.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.065+0000 7f6cd2659640 10 monclient(hunting): picked mon.noname-c con 0x7f6ccc1393a0 addr v2:172.21.15.154:3300/0 2024-09-16T03:45:16.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.065+0000 7f6cd2659640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6ccc09e4c0 con 0x7f6ccc0a1710 2024-09-16T03:45:16.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.065+0000 7f6cd2659640 10 monclient(hunting): start opening mon connection 2024-09-16T03:45:16.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.065+0000 7f6cd2659640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f6ccc09edf0 con 0x7f6ccc1393a0 2024-09-16T03:45:16.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.065+0000 7f6cd2659640 10 monclient(hunting): start opening mon connection 2024-09-16T03:45:16.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.065+0000 7f6cd2659640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f6ccc09cd60 con 0x7f6ccc0a1330 2024-09-16T03:45:16.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.065+0000 7f6cd2659640 10 monclient(hunting): _renew_subs 2024-09-16T03:45:16.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.065+0000 7f6cd2659640 10 monclient(hunting): authenticate will time out at 1163.039768s 2024-09-16T03:45:16.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.065+0000 7f6ccbfff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f6ccc0a1330 0x7f6ccc09f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:45:16.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.069+0000 7f6ccbfff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f6ccc0a1330 0x7f6ccc09f180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:38506/0 (socket says 172.21.15.42:38506) 2024-09-16T03:45:16.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.069+0000 7f6ccbfff640 1 -- 172.21.15.42:0/1456618132 learned_addr learned my addr 172.21.15.42:0/1456618132 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:45:16.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.069+0000 7f6ccbfff640 10 monclient(hunting): get_auth_request con 0x7f6ccc0a1330 auth_method 0 2024-09-16T03:45:16.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.069+0000 7f6ccbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:45:16.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.069+0000 7f6ccbfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:45:16.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.069+0000 7f6ccbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:45:16.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.069+0000 7f6ccbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:45:16.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.069+0000 7f6ccbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:45:16.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.069+0000 7f6ccbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:45:16.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.069+0000 7f6cd0bf6640 1 --2- 172.21.15.42:0/1456618132 >> v2:172.21.15.154:3300/0 conn(0x7f6ccc1393a0 0x7f6ccc13b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:45:16.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.069+0000 7f6cd0bf6640 10 monclient(hunting): get_auth_request con 0x7f6ccc1393a0 auth_method 0 2024-09-16T03:45:16.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.069+0000 7f6cd0bf6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:45:16.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.069+0000 7f6cd0bf6640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:45:16.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.069+0000 7f6cd0bf6640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:45:16.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.069+0000 7f6ccbfff640 10 monclient(hunting): handle_auth_done global_id 13271 payload 274 2024-09-16T03:45:16.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.069+0000 7f6ccbfff640 1 -- 172.21.15.42:0/1456618132 >> v2:172.21.15.154:3300/0 conn(0x7f6ccc1393a0 msgr2=0x7f6ccc13b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:45:16.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.069+0000 7f6ccbfff640 1 --2- 172.21.15.42:0/1456618132 >> v2:172.21.15.154:3300/0 conn(0x7f6ccc1393a0 0x7f6ccc13b790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:45:16.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.069+0000 7f6ccbfff640 1 -- 172.21.15.42:0/1456618132 >> v1:172.21.15.154:6789/0 conn(0x7f6ccc0a1710 legacy=0x7f6ccc0a1b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:45:16.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.069+0000 7f6ccbfff640 10 monclient: _finish_hunting 0 2024-09-16T03:45:16.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.069+0000 7f6ccbfff640 1 monclient: found mon.noname-b 2024-09-16T03:45:16.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.069+0000 7f6ccbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:45:16.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.069+0000 7f6ccbfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T03:45:16.077 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.069+0000 7f6ccbfff640 1 -- 172.21.15.42:0/1456618132 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ccc09f6c0 con 0x7f6ccc0a1330 2024-09-16T03:45:16.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.069+0000 7f6ccbfff640 1 --2- 172.21.15.42:0/1456618132 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f6ccc0a1330 0x7f6ccc09f180 secure :-1 s=READY pgs=2340 cs=0 l=1 rev1=1 crypto rx=0x7f6cbc009e50 tx=0x7f6cbc01c8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3c90bcd292774f6b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:45:16.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.069+0000 7f6ccaffd640 1 -- 172.21.15.42:0/1456618132 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cbc0a5640 con 0x7f6ccc0a1330 2024-09-16T03:45:16.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.069+0000 7f6ccaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:45:16.083 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.069+0000 7f6ccaffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:45:16.083 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.069+0000 7f6ccaffd640 10 monclient: dump: 2024-09-16T03:45:16.083 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:45:16.083 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:45:16.083 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:16.083 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:16.083 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:45:16.083 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:45:16.083 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:45:16.084 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:45:16.084 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:45:16.084 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:45:16.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.069+0000 7f6ccaffd640 10 monclient: _finish_auth 0 2024-09-16T03:45:16.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.069+0000 7f6ccaffd640 10 monclient: _check_auth_tickets 2024-09-16T03:45:16.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.069+0000 7f6ccaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:45:16.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.069+0000 7f6ccaffd640 1 -- 172.21.15.42:0/1456618132 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6cbc0a57e0 con 0x7f6ccc0a1330 2024-09-16T03:45:16.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.069+0000 7f6ccaffd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:45:16.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.069+0000 7f6ccaffd640 1 -- 172.21.15.42:0/1456618132 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cbc0a5bb0 con 0x7f6ccc0a1330 2024-09-16T03:45:16.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.069+0000 7f6cd2659640 5 monclient: authenticate success, global_id 13271 2024-09-16T03:45:16.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.069+0000 7f6cd2659640 10 monclient: get_monmap_and_config success 2024-09-16T03:45:16.085 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.069+0000 7f6cd2659640 4 set_mon_vals no callback set 2024-09-16T03:45:16.085 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.069+0000 7f6ccaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:45:16.085 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.069+0000 7f6ccaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:45:16.085 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.069+0000 7f6ccaffd640 10 monclient: dump: 2024-09-16T03:45:16.085 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:45:16.085 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:45:16.085 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:16.085 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:16.085 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:45:16.085 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:45:16.085 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:45:16.085 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:45:16.086 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:45:16.086 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:45:16.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.069+0000 7f6cd2659640 10 monclient: shutdown 2024-09-16T03:45:16.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.069+0000 7f6cd2659640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:45:16.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.069+0000 7f6cd2659640 1 -- 172.21.15.42:0/1456618132 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f6ccc0a1330 msgr2=0x7f6ccc09f180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:45:16.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.069+0000 7f6cd2659640 1 --2- 172.21.15.42:0/1456618132 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f6ccc0a1330 0x7f6ccc09f180 secure :-1 s=READY pgs=2340 cs=0 l=1 rev1=1 crypto rx=0x7f6cbc009e50 tx=0x7f6cbc01c8b0 comp rx=0 tx=0).stop 2024-09-16T03:45:16.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.069+0000 7f6cd2659640 1 -- 172.21.15.42:0/1456618132 shutdown_connections 2024-09-16T03:45:16.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.069+0000 7f6cd2659640 1 --2- 172.21.15.42:0/1456618132 >> v2:172.21.15.154:3300/0 conn(0x7f6ccc1393a0 0x7f6ccc13b790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:45:16.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.069+0000 7f6cd2659640 1 --2- 172.21.15.42:0/1456618132 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f6ccc0a1330 0x7f6ccc09f180 unknown :-1 s=CLOSED pgs=2340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:45:16.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.069+0000 7f6cd2659640 1 -- 172.21.15.42:0/1456618132 >> 172.21.15.42:0/1456618132 conn(0x7f6ccc0a2420 msgr2=0x7f6ccc0a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:45:16.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.069+0000 7f6cd2659640 1 -- 172.21.15.42:0/1456618132 shutdown_connections 2024-09-16T03:45:16.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.069+0000 7f6cd2659640 1 -- 172.21.15.42:0/1456618132 wait complete. 2024-09-16T03:45:16.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.073+0000 7f6cd2659640 10 monclient: build_initial_monmap 2024-09-16T03:45:16.087 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.073+0000 7f6cd2659640 10 monclient: monmap: 2024-09-16T03:45:16.087 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:45:16.087 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:45:16.087 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:45:16.087 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:45:16.087 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:45:16.087 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:45:16.087 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:45:16.087 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:45:16.087 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:45:16.087 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:45:16.087 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.073+0000 7f6cd2659640 1 librados: starting msgr at 2024-09-16T03:45:16.087 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.073+0000 7f6cd2659640 1 librados: starting objecter 2024-09-16T03:45:16.087 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.073+0000 7f6cd2659640 1 Processor -- start 2024-09-16T03:45:16.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.073+0000 7f6cd2659640 1 -- start start 2024-09-16T03:45:16.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.073+0000 7f6cd2659640 1 librados: setting wanted keys 2024-09-16T03:45:16.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.073+0000 7f6cd2659640 1 librados: calling monclient init 2024-09-16T03:45:16.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.073+0000 7f6cd2659640 10 monclient: init 2024-09-16T03:45:16.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.073+0000 7f6cd2659640 10 monclient: _reopen_session rank -1 2024-09-16T03:45:16.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.073+0000 7f6cd2659640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:45:16.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.073+0000 7f6cd2659640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6ccc0a1330 0x7f6ccc1ccbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:45:16.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.073+0000 7f6cd2659640 10 monclient(hunting): picked mon.noname-a con 0x7f6ccc0a1330 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:45:16.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.073+0000 7f6cd2659640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f6ccc0a1710 0x7f6ccc134340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:45:16.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.073+0000 7f6cd2659640 10 monclient(hunting): picked mon.noname-c con 0x7f6ccc0a1710 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:45:16.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.073+0000 7f6cd2659640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6ccc1393a0 0x7f6ccc11bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:45:16.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.073+0000 7f6cd2659640 10 monclient(hunting): picked mon.noname-b con 0x7f6ccc1393a0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:45:16.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.073+0000 7f6cd2659640 10 monclient(hunting): start opening mon connection 2024-09-16T03:45:16.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.073+0000 7f6cd2659640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f6ccc13e880 con 0x7f6ccc0a1330 2024-09-16T03:45:16.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.073+0000 7f6cd2659640 10 monclient(hunting): start opening mon connection 2024-09-16T03:45:16.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.073+0000 7f6cd2659640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f6ccc13e700 con 0x7f6ccc1393a0 2024-09-16T03:45:16.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.073+0000 7f6cd2659640 10 monclient(hunting): start opening mon connection 2024-09-16T03:45:16.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.073+0000 7f6cd2659640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f6ccc13ea00 con 0x7f6ccc0a1710 2024-09-16T03:45:16.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.073+0000 7f6cd2659640 10 monclient(hunting): _renew_subs 2024-09-16T03:45:16.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.073+0000 7f6cd2659640 10 monclient(hunting): authenticate will time out at 1163.045348s 2024-09-16T03:45:16.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.073+0000 7f6ccbfff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6ccc0a1330 0x7f6ccc1ccbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:45:16.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.073+0000 7f6ccb7fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f6ccc0a1710 0x7f6ccc134340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:45:16.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.073+0000 7f6ccbfff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6ccc0a1330 0x7f6ccc1ccbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:33280/0 (socket says 172.21.15.42:33280) 2024-09-16T03:45:16.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.073+0000 7f6ccbfff640 1 -- 172.21.15.42:0/835229877 learned_addr learned my addr 172.21.15.42:0/835229877 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:45:16.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.073+0000 7f6ccbfff640 10 monclient(hunting): get_auth_request con 0x7f6ccc0a1330 auth_method 0 2024-09-16T03:45:16.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.073+0000 7f6ccbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:45:16.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.073+0000 7f6ccbfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:45:16.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.073+0000 7f6ccbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:45:16.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.073+0000 7f6ccb7fe640 10 monclient(hunting): get_auth_request con 0x7f6ccc0a1710 auth_method 0 2024-09-16T03:45:16.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.073+0000 7f6ccb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:45:16.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.073+0000 7f6ccb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:45:16.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.073+0000 7f6ccb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:45:16.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.073+0000 7f6ccb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:45:16.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.073+0000 7f6ccb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:45:16.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.073+0000 7f6ccbfff640 1 -- 172.21.15.42:0/835229877 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6ccc0a1330 msgr2=0x7f6ccc1ccbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-16T03:45:16.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.073+0000 7f6ccbfff640 1 -- 172.21.15.42:0/835229877 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6ccc0a1330 msgr2=0x7f6ccc1ccbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T03:45:16.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.073+0000 7f6ccbfff640 1 --2- 172.21.15.42:0/835229877 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6ccc0a1330 0x7f6ccc1ccbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:45:16.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.073+0000 7f6ccb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:45:16.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.073+0000 7f6cd0bf6640 1 --2- 172.21.15.42:0/835229877 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6ccc1393a0 0x7f6ccc11bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:45:16.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.073+0000 7f6cd0bf6640 10 monclient(hunting): get_auth_request con 0x7f6ccc1393a0 auth_method 0 2024-09-16T03:45:16.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.073+0000 7f6cd0bf6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:45:16.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.073+0000 7f6ccbfff640 1 --2- 172.21.15.42:0/835229877 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6ccc0a1330 0x7f6ccc1ccbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:45:16.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.073+0000 7f6cd0bf6640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:45:16.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.073+0000 7f6cd0bf6640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:45:16.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.073+0000 7f6cd0bf6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:45:16.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.073+0000 7f6cd0bf6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:45:16.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.073+0000 7f6cd0bf6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:45:16.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.073+0000 7f6ccb7fe640 10 monclient(hunting): handle_auth_done global_id 13274 payload 824 2024-09-16T03:45:16.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.073+0000 7f6ccb7fe640 1 -- 172.21.15.42:0/835229877 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6ccc1393a0 msgr2=0x7f6ccc11bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:45:16.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.073+0000 7f6ccb7fe640 1 --2- 172.21.15.42:0/835229877 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6ccc1393a0 0x7f6ccc11bf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:45:16.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.073+0000 7f6ccb7fe640 1 -- 172.21.15.42:0/835229877 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6ccc0a1330 msgr2=0x7f6ccc1ccbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:45:16.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.073+0000 7f6ccb7fe640 1 --2- 172.21.15.42:0/835229877 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6ccc0a1330 0x7f6ccc1ccbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:45:16.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.073+0000 7f6ccb7fe640 10 monclient: _finish_hunting 0 2024-09-16T03:45:16.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.073+0000 7f6ccb7fe640 1 monclient: found mon.noname-c 2024-09-16T03:45:16.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.073+0000 7f6ccb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:45:16.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.073+0000 7f6ccb7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:45:16.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.073+0000 7f6ccb7fe640 1 -- 172.21.15.42:0/835229877 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ccc11fce0 con 0x7f6ccc0a1710 2024-09-16T03:45:16.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.073+0000 7f6ccb7fe640 1 --2- 172.21.15.42:0/835229877 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f6ccc0a1710 0x7f6ccc134340 secure :-1 s=READY pgs=2341 cs=0 l=1 rev1=1 crypto rx=0x7f6ccc073490 tx=0x7f6cb400c870 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:45:16.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.073+0000 7f6cc8ff9640 10 client.?.objecter ms_handle_connect 0x7f6ccc0a1710 2024-09-16T03:45:16.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.073+0000 7f6cc8ff9640 10 client.?.objecter resend_mon_ops 2024-09-16T03:45:16.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.073+0000 7f6cc8ff9640 1 -- 172.21.15.42:0/835229877 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cb4011790 con 0x7f6ccc0a1710 2024-09-16T03:45:16.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.073+0000 7f6cc8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:45:16.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.073+0000 7f6cc8ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:45:16.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.073+0000 7f6cc8ff9640 10 monclient: dump: 2024-09-16T03:45:16.093 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:45:16.093 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:45:16.093 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:16.093 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:16.093 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:45:16.093 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:45:16.093 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:45:16.093 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:45:16.093 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:45:16.093 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:45:16.093 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.073+0000 7f6cc8ff9640 10 monclient: _finish_auth 0 2024-09-16T03:45:16.093 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.073+0000 7f6cc8ff9640 10 monclient: _check_auth_tickets 2024-09-16T03:45:16.093 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.073+0000 7f6cc8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:45:16.093 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.073+0000 7f6cd2659640 5 monclient: authenticate success, global_id 13274 2024-09-16T03:45:16.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.073+0000 7f6cc8ff9640 1 -- 172.21.15.42:0/835229877 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6cb4017070 con 0x7f6ccc0a1710 2024-09-16T03:45:16.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.073+0000 7f6cc8ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T03:45:16.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.073+0000 7f6cd2659640 10 monclient: _renew_subs 2024-09-16T03:45:16.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.073+0000 7f6cc8ff9640 1 -- 172.21.15.42:0/835229877 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cb4011bb0 con 0x7f6ccc0a1710 2024-09-16T03:45:16.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.073+0000 7f6cd2659640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:45:16.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.073+0000 7f6cd2659640 1 -- 172.21.15.42:0/835229877 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ccc0a0de0 con 0x7f6ccc0a1710 2024-09-16T03:45:16.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.073+0000 7f6cca7fc640 4 set_mon_vals no callback set 2024-09-16T03:45:16.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.073+0000 7f6cd2659640 10 client.13274.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:45:16.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.073+0000 7f6cd2659640 10 monclient: _renew_subs 2024-09-16T03:45:16.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.073+0000 7f6cd2659640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:45:16.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.073+0000 7f6cd2659640 1 -- 172.21.15.42:0/835229877 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ccc11e700 con 0x7f6ccc0a1710 2024-09-16T03:45:16.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.073+0000 7f6cd2659640 1 librados: init done 2024-09-16T03:45:16.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.073+0000 7f6cc8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:45:16.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.073+0000 7f6cc8ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:45:16.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.073+0000 7f6cc8ff9640 10 monclient: dump: 2024-09-16T03:45:16.095 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:45:16.095 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:45:16.095 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:16.095 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:16.095 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:45:16.095 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:45:16.095 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:45:16.095 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:45:16.095 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:45:16.095 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:45:16.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.073+0000 7f6cd2659640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:45:16.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.073+0000 7f6cd2659640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:45:16.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.073+0000 7f6cd2659640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:45:16.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.073+0000 7f6cd2659640 1 -- 172.21.15.42:0/835229877 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ccc09f180 con 0x7f6ccc0a1710 2024-09-16T03:45:16.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.073+0000 7f6cc8ff9640 1 -- 172.21.15.42:0/835229877 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f6cb401c460 con 0x7f6ccc0a1710 2024-09-16T03:45:16.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.073+0000 7f6cc8ff9640 1 --2- 172.21.15.42:0/835229877 >> v2:172.21.15.154:6812/254845342 conn(0x7f6cac037840 0x7f6cac039d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:45:16.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.073+0000 7f6cc8ff9640 1 -- 172.21.15.42:0/835229877 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(83..83 src has 1..83) ==== 7543+0+0 (secure 0 0 0) 0x7f6cb40494b0 con 0x7f6ccc0a1710 2024-09-16T03:45:16.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.073+0000 7f6cc8ff9640 10 client.13274.objecter ms_dispatch 0x7f6ccc000ba0 osd_map(83..83 src has 1..83) 2024-09-16T03:45:16.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.073+0000 7f6cc8ff9640 3 client.13274.objecter handle_osd_map got epochs [83,83] > 0 2024-09-16T03:45:16.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.073+0000 7f6cc8ff9640 3 client.13274.objecter handle_osd_map decoding full epoch 83 2024-09-16T03:45:16.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.073+0000 7f6cc8ff9640 20 client.13274.objecter dump_active .. 0 homeless 2024-09-16T03:45:16.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.077+0000 7f6ccbfff640 1 --2- 172.21.15.42:0/835229877 >> v2:172.21.15.154:6812/254845342 conn(0x7f6cac037840 0x7f6cac039d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:45:16.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.077+0000 7f6ccbfff640 10 monclient: get_auth_request con 0x7f6cac037840 auth_method 0 2024-09-16T03:45:16.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.077+0000 7f6ccbfff640 1 --2- 172.21.15.42:0/835229877 >> v2:172.21.15.154:6812/254845342 conn(0x7f6cac037840 0x7f6cac039d00 secure :-1 s=READY pgs=3349 cs=0 l=1 rev1=1 crypto rx=0x7f6cbc009e50 tx=0x7f6cbc0a7070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:45:16.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.077+0000 7f6cc8ff9640 10 client.13274.objecter ms_handle_connect 0x7f6cac037840 2024-09-16T03:45:16.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.081+0000 7f6cc8ff9640 1 -- 172.21.15.42:0/835229877 <== mon.2 v2:172.21.15.42:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f6cb401a760 con 0x7f6ccc0a1710 2024-09-16T03:45:16.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.081+0000 7f6cc8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:45:16.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.081+0000 7f6cc8ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:45:16.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.305+0000 7f6cd2659640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:45:16.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.305+0000 7f6cd2659640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:45:16.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.305+0000 7f6cd2659640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:45:16.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.305+0000 7f6cd2659640 1 -- 172.21.15.42:0/835229877 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f6ccc11fa90 con 0x7f6ccc0a1710 2024-09-16T03:45:16.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.305+0000 7f6cc8ff9640 1 -- 172.21.15.42:0/835229877 <== mon.2 v2:172.21.15.42:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1555 (secure 0 0 0) 0x7f6cb4012420 con 0x7f6ccc0a1710 2024-09-16T03:45:16.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.305+0000 7f6cc8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:45:16.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.305+0000 7f6cc8ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:45:16.312 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":24,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":25,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:45:16.314 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.309+0000 7f6cd2659640 10 librados: watch_flush enter 2024-09-16T03:45:16.314 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.309+0000 7f6cd2659640 10 librados: watch_flush exit 2024-09-16T03:45:16.314 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.309+0000 7f6cd2659640 20 client.13274.objecter shutdown clearing up homeless session... 2024-09-16T03:45:16.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.309+0000 7f6cd2659640 10 client.13274.objecter successfully canceled tick 2024-09-16T03:45:16.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.309+0000 7f6cd2659640 1 -- 172.21.15.42:0/835229877 >> v2:172.21.15.154:6812/254845342 conn(0x7f6cac037840 msgr2=0x7f6cac039d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:45:16.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.309+0000 7f6cd2659640 1 --2- 172.21.15.42:0/835229877 >> v2:172.21.15.154:6812/254845342 conn(0x7f6cac037840 0x7f6cac039d00 secure :-1 s=READY pgs=3349 cs=0 l=1 rev1=1 crypto rx=0x7f6cbc009e50 tx=0x7f6cbc0a7070 comp rx=0 tx=0).stop 2024-09-16T03:45:16.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.309+0000 7f6cd2659640 10 monclient: shutdown 2024-09-16T03:45:16.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.309+0000 7f6cd2659640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:45:16.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.309+0000 7f6cd2659640 1 -- 172.21.15.42:0/835229877 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f6ccc0a1710 msgr2=0x7f6ccc134340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:45:16.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.309+0000 7f6cd2659640 1 --2- 172.21.15.42:0/835229877 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f6ccc0a1710 0x7f6ccc134340 secure :-1 s=READY pgs=2341 cs=0 l=1 rev1=1 crypto rx=0x7f6ccc073490 tx=0x7f6cb400c870 comp rx=0 tx=0).stop 2024-09-16T03:45:16.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.309+0000 7f6cd2659640 1 -- 172.21.15.42:0/835229877 shutdown_connections 2024-09-16T03:45:16.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.309+0000 7f6cd2659640 1 --2- 172.21.15.42:0/835229877 >> v2:172.21.15.154:6812/254845342 conn(0x7f6cac037840 0x7f6cac039d00 unknown :-1 s=CLOSED pgs=3349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:45:16.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.309+0000 7f6cd2659640 1 --2- 172.21.15.42:0/835229877 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6ccc1393a0 0x7f6ccc11bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:45:16.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.309+0000 7f6cd2659640 1 --2- 172.21.15.42:0/835229877 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f6ccc0a1710 0x7f6ccc134340 unknown :-1 s=CLOSED pgs=2341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:45:16.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.309+0000 7f6cd2659640 1 --2- 172.21.15.42:0/835229877 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6ccc0a1330 0x7f6ccc1ccbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:45:16.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.309+0000 7f6cd2659640 1 -- 172.21.15.42:0/835229877 >> 172.21.15.42:0/835229877 conn(0x7f6ccc0a2420 msgr2=0x7f6ccc09c270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:45:16.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.309+0000 7f6cd2659640 1 -- 172.21.15.42:0/835229877 shutdown_connections 2024-09-16T03:45:16.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.309+0000 7f6cd2659640 1 -- 172.21.15.42:0/835229877 wait complete. 2024-09-16T03:45:16.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:16.309+0000 7f6cd2659640 1 librados: shutdown 2024-09-16T03:45:17.342 INFO:tasks.workunit.client.0.smithi154.stdout: api_misc_pp: [----------] 22 tests from LibRadosMis api_snapshots: Running main() from gmock_main.cc 2024-09-16T03:45:17.342 INFO:tasks.workunit.client.0.smithi154.stdout: api_snapshots: [==========] Running 13 tests from 4 test suites. 2024-09-16T03:45:17.342 INFO:tasks.workunit.client.0.smithi154.stdout: api_snapshots: [----------] Global test environment set-up. 2024-09-16T03:45:17.342 INFO:tasks.workunit.client.0.smithi154.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshots 2024-09-16T03:45:17.343 INFO:tasks.workunit.client.0.smithi154.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapList 2024-09-16T03:45:17.343 INFO:tasks.workunit.client.0.smithi154.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapList (22963 ms) 2024-09-16T03:45:17.343 INFO:tasks.workunit.client.0.smithi154.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapRemove 2024-09-16T03:45:17.343 INFO:tasks.workunit.client.0.smithi154.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapRemove (2187 ms) 2024-09-16T03:45:17.343 INFO:tasks.workunit.client.0.smithi154.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.Rollback 2024-09-16T03:45:17.343 INFO:tasks.workunit.client.0.smithi154.stdout: api_snapshots: [ OK ] LibRadosSnapshots.Rollback (1850 ms) 2024-09-16T03:45:17.343 INFO:tasks.workunit.client.0.smithi154.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapGetName 2024-09-16T03:45:17.343 INFO:tasks.workunit.client.0.smithi154.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapGetName (1926 ms) 2024-09-16T03:45:17.343 INFO:tasks.workunit.client.0.smithi154.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshots (28926 ms total) 2024-09-16T03:45:17.343 INFO:tasks.workunit.client.0.smithi154.stdout: api_snapshots: 2024-09-16T03:45:17.343 INFO:tasks.workunit.client.0.smithi154.stdout: api_snapshots: [----------] 3 tests from LibRadosSnapshotsSelfManaged 2024-09-16T03:45:17.343 INFO:tasks.workunit.client.0.smithi154.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.Snap 2024-09-16T03:45:17.343 INFO:tasks.workunit.client.0.smithi154.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.Snap (4111 ms) 2024-09-16T03:45:17.343 INFO:tasks.workunit.client.0.smithi154.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.Rollback 2024-09-16T03:45:17.344 INFO:tasks.workunit.client.0.smithi154.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.Rollback (4029 ms) 2024-09-16T03:45:17.344 INFO:tasks.workunit.client.0.smithi154.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.FutureSnapRollback 2024-09-16T03:45:17.344 INFO:tasks.workunit.client.0.smithi154.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.FutureSnapRollback (20112 ms) 2024-09-16T03:45:17.344 INFO:tasks.workunit.client.0.smithi154.stdout: api_snapshots: [----------] 3 tests from LibRadosSnapshotsSelfManaged (28252 ms total) 2024-09-16T03:45:17.344 INFO:tasks.workunit.client.0.smithi154.stdout: api_snapshots: 2024-09-16T03:45:17.344 INFO:tasks.workunit.client.0.smithi154.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshotsEC 2024-09-16T03:45:17.344 INFO:tasks.workunit.client.0.smithi154.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapList 2024-09-16T03:45:17.344 INFO:tasks.workunit.client.0.smithi154.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapList (2780 ms) 2024-09-16T03:45:17.344 INFO:tasks.workunit.client.0.smithi154.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapRemove 2024-09-16T03:45:17.344 INFO:tasks.workunit.client.0.smithi154.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapRemove (2014 ms) 2024-09-16T03:45:17.344 INFO:tasks.workunit.client.0.smithi154.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.Rollback 2024-09-16T03:45:17.344 INFO:tasks.workunit.client.0.smithi154.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.Rollback (1977 ms) 2024-09-16T03:45:17.344 INFO:tasks.workunit.client.0.smithi154.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapGetName 2024-09-16T03:45:17.344 INFO:tasks.workunit.client.0.smithi154.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapGetName (2013 ms) 2024-09-16T03:45:17.345 INFO:tasks.workunit.client.0.smithi154.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshotsEC (8784 ms total) 2024-09-16T03:45:17.345 INFO:tasks.workunit.client.0.smithi154.stdout: api_snapshots: 2024-09-16T03:45:17.345 INFO:tasks.workunit.client.0.smithi154.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManagedEC 2024-09-16T03:45:17.345 INFO:tasks.workunit.client.0.smithi154.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManagedEC.Snap 2024-09-16T03:45:17.345 INFO:tasks.workunit.client.0.smithi154.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManagedEC.Snap (4048 ms) 2024-09-16T03:45:17.345 INFO:tasks.workunit.client.0.smithi154.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManagedEC.Rollback 2024-09-16T03:45:17.345 INFO:tasks.workunit.client.0.smithi154.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManagedEC.Rollback (4253 ms) 2024-09-16T03:45:17.345 INFO:tasks.workunit.client.0.smithi154.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManagedEC (8301 ms total) 2024-09-16T03:45:17.345 INFO:tasks.workunit.client.0.smithi154.stdout: api_snapshots: 2024-09-16T03:45:17.345 INFO:tasks.workunit.client.0.smithi154.stdout: api_snapshots: [----------] Global test environment tear-down 2024-09-16T03:45:17.345 INFO:tasks.workunit.client.0.smithi154.stdout: api_snapshots: [==========] 13 tests from 4 test suites ran. (107709 ms total) 2024-09-16T03:45:17.345 INFO:tasks.workunit.client.0.smithi154.stdout: api_snapshots: [ PASSED ] 13 tests. 2024-09-16T03:45:19.331 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:45:19.466 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.457+0000 7f26fcbec640 10 monclient: get_monmap_and_config 2024-09-16T03:45:19.466 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.457+0000 7f26fcbec640 10 monclient: build_initial_monmap 2024-09-16T03:45:19.466 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.457+0000 7f26fcbec640 10 monclient: monmap: 2024-09-16T03:45:19.466 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:45:19.466 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:45:19.466 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:45:19.463303+0000 2024-09-16T03:45:19.467 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:45:19.463303+0000 2024-09-16T03:45:19.467 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:45:19.467 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:45:19.467 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:45:19.467 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:45:19.467 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:45:19.467 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:45:19.467 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:45:19.467 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:45:19.467 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.457+0000 7f26fcbec640 1 Processor -- start 2024-09-16T03:45:19.467 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.461+0000 7f26fcbec640 1 -- start start 2024-09-16T03:45:19.467 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.461+0000 7f26fcbec640 10 monclient: init 2024-09-16T03:45:19.467 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.461+0000 7f26fcbec640 10 monclient: _reopen_session rank -1 2024-09-16T03:45:19.468 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.461+0000 7f26fcbec640 10 monclient: _add_conns ranks=[3,4,2,1,0] 2024-09-16T03:45:19.468 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.461+0000 7f26fcbec640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f26f80a1330 0x7f26f809f180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:45:19.468 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.461+0000 7f26fcbec640 10 monclient(hunting): picked mon.noname-a con 0x7f26f80a1330 addr v2:172.21.15.42:3300/0 2024-09-16T03:45:19.468 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.461+0000 7f26fcbec640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f26f80a1710 0x7f26f80a1b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:45:19.468 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.461+0000 7f26fcbec640 10 monclient(hunting): picked mon.noname-c con 0x7f26f80a1710 addr v2:172.21.15.154:3300/0 2024-09-16T03:45:19.468 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.461+0000 7f26fcbec640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f26f81340f0 0x7f26f8134580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:45:19.468 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.461+0000 7f26fcbec640 10 monclient(hunting): picked mon.noname-b con 0x7f26f81340f0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:45:19.468 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.461+0000 7f26fcbec640 10 monclient(hunting): start opening mon connection 2024-09-16T03:45:19.468 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.461+0000 7f26fcbec640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f26f809edf0 con 0x7f26f80a1330 2024-09-16T03:45:19.468 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.461+0000 7f26fcbec640 10 monclient(hunting): start opening mon connection 2024-09-16T03:45:19.468 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.461+0000 7f26fcbec640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f26f809cd60 con 0x7f26f80a1710 2024-09-16T03:45:19.468 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.461+0000 7f26fcbec640 10 monclient(hunting): start opening mon connection 2024-09-16T03:45:19.468 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.461+0000 7f26fcbec640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f26f809e4c0 con 0x7f26f81340f0 2024-09-16T03:45:19.468 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.461+0000 7f26fcbec640 10 monclient(hunting): _renew_subs 2024-09-16T03:45:19.469 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.461+0000 7f26fcbec640 10 monclient(hunting): authenticate will time out at 1166.432793s 2024-09-16T03:45:19.469 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.461+0000 7f26f6d9d640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f26f81340f0 0x7f26f8134580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:45:19.469 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.461+0000 7f26f6d9d640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f26f81340f0 0x7f26f8134580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:38554/0 (socket says 172.21.15.42:38554) 2024-09-16T03:45:19.469 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.461+0000 7f26f6d9d640 1 -- 172.21.15.42:0/910444886 learned_addr learned my addr 172.21.15.42:0/910444886 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:45:19.469 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.461+0000 7f26f6d9d640 10 monclient(hunting): get_auth_request con 0x7f26f81340f0 auth_method 0 2024-09-16T03:45:19.469 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.461+0000 7f26f6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:45:19.469 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.461+0000 7f26f6d9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:45:19.469 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.461+0000 7f26f6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:45:19.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.461+0000 7f26f659c640 1 --2- 172.21.15.42:0/910444886 >> v2:172.21.15.42:3300/0 conn(0x7f26f80a1330 0x7f26f809f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:45:19.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.461+0000 7f26f6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:45:19.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.461+0000 7f26f6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:45:19.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.461+0000 7f26f6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:45:19.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.461+0000 7f26f659c640 10 monclient(hunting): get_auth_request con 0x7f26f80a1330 auth_method 0 2024-09-16T03:45:19.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.461+0000 7f26f659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:45:19.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.461+0000 7f26f659c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:45:19.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.461+0000 7f26f659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:45:19.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.461+0000 7f26f659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:45:19.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.461+0000 7f26f659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:45:19.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.461+0000 7f26f5d9b640 1 --2- 172.21.15.42:0/910444886 >> v2:172.21.15.154:3300/0 conn(0x7f26f80a1710 0x7f26f80a1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:45:19.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.461+0000 7f26f659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:45:19.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.461+0000 7f26f6d9d640 10 monclient(hunting): handle_auth_done global_id 13295 payload 274 2024-09-16T03:45:19.471 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.461+0000 7f26f6d9d640 1 -- 172.21.15.42:0/910444886 >> v2:172.21.15.154:3300/0 conn(0x7f26f80a1710 msgr2=0x7f26f80a1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:45:19.471 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.461+0000 7f26f6d9d640 1 --2- 172.21.15.42:0/910444886 >> v2:172.21.15.154:3300/0 conn(0x7f26f80a1710 0x7f26f80a1b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:45:19.471 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.461+0000 7f26f6d9d640 1 -- 172.21.15.42:0/910444886 >> v2:172.21.15.42:3300/0 conn(0x7f26f80a1330 msgr2=0x7f26f809f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:45:19.471 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.461+0000 7f26f6d9d640 1 --2- 172.21.15.42:0/910444886 >> v2:172.21.15.42:3300/0 conn(0x7f26f80a1330 0x7f26f809f180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:45:19.471 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.461+0000 7f26f6d9d640 10 monclient: _finish_hunting 0 2024-09-16T03:45:19.471 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.461+0000 7f26f6d9d640 1 monclient: found mon.noname-b 2024-09-16T03:45:19.471 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.461+0000 7f26f6d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:45:19.471 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.461+0000 7f26f6d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T03:45:19.471 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.461+0000 7f26f6d9d640 1 -- 172.21.15.42:0/910444886 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26f80a2080 con 0x7f26f81340f0 2024-09-16T03:45:19.476 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.461+0000 7f26f6d9d640 1 --2- 172.21.15.42:0/910444886 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f26f81340f0 0x7f26f8134580 secure :-1 s=READY pgs=2349 cs=0 l=1 rev1=1 crypto rx=0x7f26f8073490 tx=0x7f26f001e5f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=91ce059f12705e10 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:45:19.477 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.461+0000 7f26f559a640 1 -- 172.21.15.42:0/910444886 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26f00ac430 con 0x7f26f81340f0 2024-09-16T03:45:19.477 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.461+0000 7f26f559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:45:19.477 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.461+0000 7f26f559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:45:19.477 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.461+0000 7f26f559a640 10 monclient: dump: 2024-09-16T03:45:19.477 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:45:19.477 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:45:19.477 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:19.477 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:19.477 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:45:19.477 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:45:19.477 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:45:19.477 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:45:19.478 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:45:19.478 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:45:19.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.461+0000 7f26f559a640 10 monclient: _finish_auth 0 2024-09-16T03:45:19.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.461+0000 7f26f559a640 10 monclient: _check_auth_tickets 2024-09-16T03:45:19.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.461+0000 7f26f559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:45:19.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.461+0000 7f26fcbec640 5 monclient: authenticate success, global_id 13295 2024-09-16T03:45:19.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.461+0000 7f26f559a640 1 -- 172.21.15.42:0/910444886 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f26f00b1070 con 0x7f26f81340f0 2024-09-16T03:45:19.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.461+0000 7f26fcbec640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:45:19.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.461+0000 7f26f559a640 10 monclient: handle_config config(0 keys) 2024-09-16T03:45:19.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.461+0000 7f26fcbec640 10 monclient: get_monmap_and_config success 2024-09-16T03:45:19.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.461+0000 7f26fcbec640 4 set_mon_vals no callback set 2024-09-16T03:45:19.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.461+0000 7f26f559a640 1 -- 172.21.15.42:0/910444886 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26f00ac7f0 con 0x7f26f81340f0 2024-09-16T03:45:19.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.461+0000 7f26f559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:45:19.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.461+0000 7f26f559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:45:19.479 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.461+0000 7f26f559a640 10 monclient: dump: 2024-09-16T03:45:19.479 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:45:19.479 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:45:19.479 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:19.479 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:19.479 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:45:19.479 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:45:19.479 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:45:19.479 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:45:19.479 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:45:19.480 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:45:19.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.465+0000 7f26fcbec640 10 monclient: shutdown 2024-09-16T03:45:19.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.465+0000 7f26fcbec640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:45:19.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.465+0000 7f26fcbec640 1 -- 172.21.15.42:0/910444886 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f26f81340f0 msgr2=0x7f26f8134580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:45:19.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.465+0000 7f26fcbec640 1 --2- 172.21.15.42:0/910444886 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f26f81340f0 0x7f26f8134580 secure :-1 s=READY pgs=2349 cs=0 l=1 rev1=1 crypto rx=0x7f26f8073490 tx=0x7f26f001e5f0 comp rx=0 tx=0).stop 2024-09-16T03:45:19.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.465+0000 7f26fcbec640 1 -- 172.21.15.42:0/910444886 shutdown_connections 2024-09-16T03:45:19.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.465+0000 7f26fcbec640 1 --2- 172.21.15.42:0/910444886 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f26f81340f0 0x7f26f8134580 unknown :-1 s=CLOSED pgs=2349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:45:19.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.465+0000 7f26fcbec640 1 --2- 172.21.15.42:0/910444886 >> v2:172.21.15.154:3300/0 conn(0x7f26f80a1710 0x7f26f80a1b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:45:19.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.465+0000 7f26fcbec640 1 --2- 172.21.15.42:0/910444886 >> v2:172.21.15.42:3300/0 conn(0x7f26f80a1330 0x7f26f809f180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:45:19.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.465+0000 7f26fcbec640 1 -- 172.21.15.42:0/910444886 >> 172.21.15.42:0/910444886 conn(0x7f26f80a2420 msgr2=0x7f26f80a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:45:19.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.465+0000 7f26fcbec640 1 -- 172.21.15.42:0/910444886 shutdown_connections 2024-09-16T03:45:19.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.465+0000 7f26fcbec640 1 -- 172.21.15.42:0/910444886 wait complete. 2024-09-16T03:45:19.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.465+0000 7f26fcbec640 10 monclient: build_initial_monmap 2024-09-16T03:45:19.481 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.465+0000 7f26fcbec640 10 monclient: monmap: 2024-09-16T03:45:19.481 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:45:19.481 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:45:19.481 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:45:19.481 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:45:19.481 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:45:19.481 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:45:19.481 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:45:19.481 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:45:19.481 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:45:19.481 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:45:19.481 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.465+0000 7f26fcbec640 1 librados: starting msgr at 2024-09-16T03:45:19.481 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.465+0000 7f26fcbec640 1 librados: starting objecter 2024-09-16T03:45:19.481 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.465+0000 7f26fcbec640 1 Processor -- start 2024-09-16T03:45:19.482 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.465+0000 7f26fcbec640 1 -- start start 2024-09-16T03:45:19.482 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.465+0000 7f26fcbec640 1 librados: setting wanted keys 2024-09-16T03:45:19.482 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.465+0000 7f26fcbec640 1 librados: calling monclient init 2024-09-16T03:45:19.482 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.465+0000 7f26fcbec640 10 monclient: init 2024-09-16T03:45:19.482 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.465+0000 7f26fcbec640 10 monclient: _reopen_session rank -1 2024-09-16T03:45:19.482 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.465+0000 7f26fcbec640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T03:45:19.482 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.465+0000 7f26fcbec640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f26f80a1330 0x7f26f81cf280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:45:19.482 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.465+0000 7f26fcbec640 10 monclient(hunting): picked mon.noname-a con 0x7f26f80a1330 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:45:19.482 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.465+0000 7f26fcbec640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f26f80a1710 0x7f26f8133240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:45:19.482 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.465+0000 7f26fcbec640 10 monclient(hunting): picked mon.noname-b con 0x7f26f80a1710 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:45:19.482 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.465+0000 7f26fcbec640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f26f81340f0 0x7f26f8133780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:45:19.482 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.465+0000 7f26fcbec640 10 monclient(hunting): picked mon.noname-c con 0x7f26f81340f0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:45:19.482 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.465+0000 7f26fcbec640 10 monclient(hunting): start opening mon connection 2024-09-16T03:45:19.482 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.465+0000 7f26fcbec640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f26f809d1b0 con 0x7f26f80a1330 2024-09-16T03:45:19.483 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.465+0000 7f26fcbec640 10 monclient(hunting): start opening mon connection 2024-09-16T03:45:19.483 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.465+0000 7f26fcbec640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f26f809d030 con 0x7f26f80a1710 2024-09-16T03:45:19.483 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.465+0000 7f26fcbec640 10 monclient(hunting): start opening mon connection 2024-09-16T03:45:19.483 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.465+0000 7f26fcbec640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f26f809c260 con 0x7f26f81340f0 2024-09-16T03:45:19.483 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.465+0000 7f26fcbec640 10 monclient(hunting): _renew_subs 2024-09-16T03:45:19.483 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.465+0000 7f26fcbec640 10 monclient(hunting): authenticate will time out at 1166.437915s 2024-09-16T03:45:19.483 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.465+0000 7f26f6d9d640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f26f81340f0 0x7f26f8133780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:45:19.483 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.465+0000 7f26f6d9d640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f26f81340f0 0x7f26f8133780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:38568/0 (socket says 172.21.15.42:38568) 2024-09-16T03:45:19.483 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.465+0000 7f26f6d9d640 1 -- 172.21.15.42:0/1870022491 learned_addr learned my addr 172.21.15.42:0/1870022491 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:45:19.483 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.465+0000 7f26f6d9d640 10 monclient(hunting): get_auth_request con 0x7f26f81340f0 auth_method 0 2024-09-16T03:45:19.483 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.465+0000 7f26f6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:45:19.483 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.465+0000 7f26f6d9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:45:19.483 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.465+0000 7f26f6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:45:19.483 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.465+0000 7f26f659c640 1 --2- 172.21.15.42:0/1870022491 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f26f80a1330 0x7f26f81cf280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:45:19.484 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.465+0000 7f26f6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:45:19.484 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.465+0000 7f26f6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:45:19.484 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.465+0000 7f26f6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:45:19.484 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.465+0000 7f26f659c640 10 monclient(hunting): get_auth_request con 0x7f26f80a1330 auth_method 0 2024-09-16T03:45:19.484 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.465+0000 7f26f659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:45:19.484 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.465+0000 7f26f659c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:45:19.484 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.465+0000 7f26f659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:45:19.484 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.465+0000 7f26f659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:45:19.484 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.465+0000 7f26f659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:45:19.484 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.465+0000 7f26f659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:45:19.484 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.465+0000 7f26f6d9d640 10 monclient(hunting): handle_auth_done global_id 13298 payload 824 2024-09-16T03:45:19.484 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.465+0000 7f26f5d9b640 1 --2- 172.21.15.42:0/1870022491 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f26f80a1710 0x7f26f8133240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:45:19.484 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.465+0000 7f26f6d9d640 1 -- 172.21.15.42:0/1870022491 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f26f80a1710 msgr2=0x7f26f8133240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:45:19.485 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.465+0000 7f26f6d9d640 1 --2- 172.21.15.42:0/1870022491 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f26f80a1710 0x7f26f8133240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:45:19.485 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.465+0000 7f26f6d9d640 1 -- 172.21.15.42:0/1870022491 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f26f80a1330 msgr2=0x7f26f81cf280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:45:19.485 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.465+0000 7f26f6d9d640 1 --2- 172.21.15.42:0/1870022491 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f26f80a1330 0x7f26f81cf280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:45:19.485 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.465+0000 7f26f6d9d640 10 monclient: _finish_hunting 0 2024-09-16T03:45:19.485 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.465+0000 7f26f6d9d640 1 monclient: found mon.noname-c 2024-09-16T03:45:19.485 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.465+0000 7f26f6d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:45:19.485 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.465+0000 7f26f6d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:45:19.485 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.465+0000 7f26f6d9d640 1 -- 172.21.15.42:0/1870022491 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26f8073a30 con 0x7f26f81340f0 2024-09-16T03:45:19.485 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.465+0000 7f26f6d9d640 1 --2- 172.21.15.42:0/1870022491 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f26f81340f0 0x7f26f8133780 secure :-1 s=READY pgs=2350 cs=0 l=1 rev1=1 crypto rx=0x7f26f00ae040 tx=0x7f26f001b390 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:45:19.485 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.465+0000 7f26eeffd640 10 client.?.objecter ms_handle_connect 0x7f26f81340f0 2024-09-16T03:45:19.485 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.465+0000 7f26eeffd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:45:19.485 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.465+0000 7f26eeffd640 1 -- 172.21.15.42:0/1870022491 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26f00bb070 con 0x7f26f81340f0 2024-09-16T03:45:19.485 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.465+0000 7f26eeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:45:19.486 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.465+0000 7f26eeffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:45:19.486 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.465+0000 7f26eeffd640 10 monclient: dump: 2024-09-16T03:45:19.486 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:45:19.486 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:45:19.486 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:19.486 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:19.486 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:45:19.486 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:45:19.486 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:45:19.486 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:45:19.486 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:45:19.486 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:45:19.486 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.465+0000 7f26eeffd640 10 monclient: _finish_auth 0 2024-09-16T03:45:19.486 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.465+0000 7f26eeffd640 10 monclient: _check_auth_tickets 2024-09-16T03:45:19.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.465+0000 7f26eeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:45:19.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.465+0000 7f26eeffd640 1 -- 172.21.15.42:0/1870022491 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f26f00b5620 con 0x7f26f81340f0 2024-09-16T03:45:19.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.465+0000 7f26eeffd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:45:19.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.465+0000 7f26eeffd640 1 -- 172.21.15.42:0/1870022491 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26f00b5a10 con 0x7f26f81340f0 2024-09-16T03:45:19.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.465+0000 7f26fcbec640 5 monclient: authenticate success, global_id 13298 2024-09-16T03:45:19.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.465+0000 7f26fcbec640 10 monclient: _renew_subs 2024-09-16T03:45:19.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.465+0000 7f26fcbec640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:45:19.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.465+0000 7f26fcbec640 1 -- 172.21.15.42:0/1870022491 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26f8122c20 con 0x7f26f81340f0 2024-09-16T03:45:19.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.465+0000 7f26eeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:45:19.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.465+0000 7f26eeffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:45:19.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.465+0000 7f26eeffd640 10 monclient: dump: 2024-09-16T03:45:19.487 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:45:19.487 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:45:19.488 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:19.488 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:19.488 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:45:19.488 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:45:19.488 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:45:19.488 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:45:19.488 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:45:19.488 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:45:19.488 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.465+0000 7f26f4d99640 4 set_mon_vals no callback set 2024-09-16T03:45:19.488 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.465+0000 7f26fcbec640 10 client.13298.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:45:19.488 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.465+0000 7f26fcbec640 10 monclient: _renew_subs 2024-09-16T03:45:19.488 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.465+0000 7f26fcbec640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:45:19.488 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.465+0000 7f26fcbec640 1 -- 172.21.15.42:0/1870022491 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f26f8120930 con 0x7f26f81340f0 2024-09-16T03:45:19.488 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.465+0000 7f26fcbec640 1 librados: init done 2024-09-16T03:45:19.489 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.465+0000 7f26fcbec640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:45:19.489 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.465+0000 7f26fcbec640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:45:19.489 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.465+0000 7f26fcbec640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:45:19.489 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.465+0000 7f26fcbec640 1 -- 172.21.15.42:0/1870022491 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f26f809c610 con 0x7f26f81340f0 2024-09-16T03:45:19.489 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.465+0000 7f26eeffd640 1 -- 172.21.15.42:0/1870022491 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f26f00be070 con 0x7f26f81340f0 2024-09-16T03:45:19.489 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.465+0000 7f26eeffd640 1 --2- 172.21.15.42:0/1870022491 >> v2:172.21.15.154:6812/254845342 conn(0x7f26d8037840 0x7f26d8039d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:45:19.489 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.465+0000 7f26eeffd640 1 -- 172.21.15.42:0/1870022491 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(87..87 src has 1..87) ==== 7347+0+0 (secure 0 0 0) 0x7f26f00e35a0 con 0x7f26f81340f0 2024-09-16T03:45:19.489 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.469+0000 7f26eeffd640 10 client.13298.objecter ms_dispatch 0x7f26f8000ba0 osd_map(87..87 src has 1..87) 2024-09-16T03:45:19.489 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.469+0000 7f26eeffd640 3 client.13298.objecter handle_osd_map got epochs [87,87] > 0 2024-09-16T03:45:19.489 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.469+0000 7f26eeffd640 3 client.13298.objecter handle_osd_map decoding full epoch 87 2024-09-16T03:45:19.489 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.469+0000 7f26eeffd640 20 client.13298.objecter dump_active .. 0 homeless 2024-09-16T03:45:19.489 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.469+0000 7f26f659c640 1 --2- 172.21.15.42:0/1870022491 >> v2:172.21.15.154:6812/254845342 conn(0x7f26d8037840 0x7f26d8039d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:45:19.489 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.469+0000 7f26f659c640 10 monclient: get_auth_request con 0x7f26d8037840 auth_method 0 2024-09-16T03:45:19.490 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.469+0000 7f26f659c640 1 --2- 172.21.15.42:0/1870022491 >> v2:172.21.15.154:6812/254845342 conn(0x7f26d8037840 0x7f26d8039d00 secure :-1 s=READY pgs=3354 cs=0 l=1 rev1=1 crypto rx=0x7f26e0009ae0 tx=0x7f26e0009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:45:19.490 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.469+0000 7f26eeffd640 10 client.13298.objecter ms_handle_connect 0x7f26d8037840 2024-09-16T03:45:19.490 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.473+0000 7f26eeffd640 1 -- 172.21.15.42:0/1870022491 <== mon.2 v2:172.21.15.42:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f26f8073a30 con 0x7f26f81340f0 2024-09-16T03:45:19.490 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.473+0000 7f26eeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:45:19.490 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.473+0000 7f26eeffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:45:19.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.701+0000 7f26fcbec640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:45:19.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.701+0000 7f26fcbec640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:45:19.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.701+0000 7f26fcbec640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:45:19.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.701+0000 7f26fcbec640 1 -- 172.21.15.42:0/1870022491 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f26f81cf7c0 con 0x7f26f81340f0 2024-09-16T03:45:19.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.701+0000 7f26eeffd640 1 -- 172.21.15.42:0/1870022491 <== mon.2 v2:172.21.15.42:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7f26f0018300 con 0x7f26f81340f0 2024-09-16T03:45:19.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.701+0000 7f26eeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:45:19.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.701+0000 7f26eeffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:45:19.706 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":28,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:45:19.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.701+0000 7f26fcbec640 10 librados: watch_flush enter 2024-09-16T03:45:19.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.701+0000 7f26fcbec640 10 librados: watch_flush exit 2024-09-16T03:45:19.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.701+0000 7f26fcbec640 20 client.13298.objecter shutdown clearing up homeless session... 2024-09-16T03:45:19.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.701+0000 7f26fcbec640 10 client.13298.objecter successfully canceled tick 2024-09-16T03:45:19.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.701+0000 7f26fcbec640 1 -- 172.21.15.42:0/1870022491 >> v2:172.21.15.154:6812/254845342 conn(0x7f26d8037840 msgr2=0x7f26d8039d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:45:19.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.701+0000 7f26fcbec640 1 --2- 172.21.15.42:0/1870022491 >> v2:172.21.15.154:6812/254845342 conn(0x7f26d8037840 0x7f26d8039d00 secure :-1 s=READY pgs=3354 cs=0 l=1 rev1=1 crypto rx=0x7f26e0009ae0 tx=0x7f26e0009190 comp rx=0 tx=0).stop 2024-09-16T03:45:19.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.701+0000 7f26fcbec640 10 monclient: shutdown 2024-09-16T03:45:19.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.701+0000 7f26fcbec640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:45:19.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.701+0000 7f26fcbec640 1 -- 172.21.15.42:0/1870022491 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f26f81340f0 msgr2=0x7f26f8133780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:45:19.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.701+0000 7f26fcbec640 1 --2- 172.21.15.42:0/1870022491 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f26f81340f0 0x7f26f8133780 secure :-1 s=READY pgs=2350 cs=0 l=1 rev1=1 crypto rx=0x7f26f00ae040 tx=0x7f26f001b390 comp rx=0 tx=0).stop 2024-09-16T03:45:19.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.701+0000 7f26fcbec640 1 -- 172.21.15.42:0/1870022491 shutdown_connections 2024-09-16T03:45:19.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.701+0000 7f26fcbec640 1 --2- 172.21.15.42:0/1870022491 >> v2:172.21.15.154:6812/254845342 conn(0x7f26d8037840 0x7f26d8039d00 unknown :-1 s=CLOSED pgs=3354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:45:19.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.701+0000 7f26fcbec640 1 --2- 172.21.15.42:0/1870022491 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f26f81340f0 0x7f26f8133780 unknown :-1 s=CLOSED pgs=2350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:45:19.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.701+0000 7f26fcbec640 1 --2- 172.21.15.42:0/1870022491 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f26f80a1710 0x7f26f8133240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:45:19.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.701+0000 7f26fcbec640 1 --2- 172.21.15.42:0/1870022491 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f26f80a1330 0x7f26f81cf280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:45:19.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.701+0000 7f26fcbec640 1 -- 172.21.15.42:0/1870022491 >> 172.21.15.42:0/1870022491 conn(0x7f26f80a2420 msgr2=0x7f26f80a4dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:45:19.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.705+0000 7f26fcbec640 1 -- 172.21.15.42:0/1870022491 shutdown_connections 2024-09-16T03:45:19.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.705+0000 7f26fcbec640 1 -- 172.21.15.42:0/1870022491 wait complete. 2024-09-16T03:45:19.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.705+0000 7f26fcbec640 1 librados: shutdown 2024-09-16T03:45:19.725 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T03:45:19.726 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T03:45:19.726 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T03:45:19.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.849+0000 7ff01ec03640 10 monclient: get_monmap_and_config 2024-09-16T03:45:19.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.849+0000 7ff01ec03640 10 monclient: build_initial_monmap 2024-09-16T03:45:19.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.849+0000 7ff01ec03640 10 monclient: monmap: 2024-09-16T03:45:19.853 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:45:19.853 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:45:19.854 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:45:19.852748+0000 2024-09-16T03:45:19.854 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:45:19.852748+0000 2024-09-16T03:45:19.854 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:45:19.854 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:45:19.854 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:45:19.854 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:45:19.854 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:45:19.854 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:45:19.854 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:45:19.854 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:45:19.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.849+0000 7ff01ec03640 1 Processor -- start 2024-09-16T03:45:19.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.849+0000 7ff01ec03640 1 -- start start 2024-09-16T03:45:19.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.849+0000 7ff01ec03640 10 monclient: init 2024-09-16T03:45:19.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.849+0000 7ff01ec03640 10 monclient: _reopen_session rank -1 2024-09-16T03:45:19.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.849+0000 7ff01ec03640 10 monclient: _add_conns ranks=[0,3,1,4,2] 2024-09-16T03:45:19.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.849+0000 7ff01ec03640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff0180a39b0 addr v1:172.21.15.42:6789/0 2024-09-16T03:45:19.857 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.849+0000 7ff01ec03640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff0180a3d90 0x7ff01813c380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:45:19.857 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.849+0000 7ff01ec03640 10 monclient(hunting): picked mon.noname-a con 0x7ff0180a3d90 addr v2:172.21.15.42:3300/0 2024-09-16T03:45:19.857 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.849+0000 7ff01ec03640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff01813c8c0 addr v1:172.21.15.154:6789/0 2024-09-16T03:45:19.857 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.849+0000 7ff01ec03640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff018134290 con 0x7ff0180a39b0 2024-09-16T03:45:19.857 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.849+0000 7ff01ec03640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff01809c140 con 0x7ff01813c8c0 2024-09-16T03:45:19.857 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.849+0000 7ff01ec03640 10 monclient(hunting): start opening mon connection 2024-09-16T03:45:19.857 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.849+0000 7ff01ec03640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7ff01809c420 con 0x7ff0180a3d90 2024-09-16T03:45:19.857 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.849+0000 7ff01ec03640 10 monclient(hunting): _renew_subs 2024-09-16T03:45:19.857 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.849+0000 7ff01ec03640 10 monclient(hunting): authenticate will time out at 1166.822916s 2024-09-16T03:45:19.857 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.849+0000 7ff00ffff640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff0180a3d90 0x7ff01813c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:45:19.858 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.849+0000 7ff00ffff640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff0180a3d90 0x7ff01813c380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:33336/0 (socket says 172.21.15.42:33336) 2024-09-16T03:45:19.858 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.849+0000 7ff00ffff640 1 -- 172.21.15.42:0/1781644058 learned_addr learned my addr 172.21.15.42:0/1781644058 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:45:19.858 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.849+0000 7ff00ffff640 10 monclient(hunting): get_auth_request con 0x7ff0180a3d90 auth_method 0 2024-09-16T03:45:19.858 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.849+0000 7ff00ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:45:19.858 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.849+0000 7ff00ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:45:19.858 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.849+0000 7ff00ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:45:19.858 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.849+0000 7ff00ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:45:19.858 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.849+0000 7ff00ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:45:19.858 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.849+0000 7ff00ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:45:19.858 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.849+0000 7ff00ffff640 10 monclient(hunting): handle_auth_done global_id 744114 payload 274 2024-09-16T03:45:19.858 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.849+0000 7ff00ffff640 1 -- 172.21.15.42:0/1781644058 >> v1:172.21.15.154:6789/0 conn(0x7ff01813c8c0 legacy=0x7ff01813ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:45:19.858 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.849+0000 7ff00ffff640 1 -- 172.21.15.42:0/1781644058 >> v1:172.21.15.42:6789/0 conn(0x7ff0180a39b0 legacy=0x7ff018133b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:45:19.858 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.849+0000 7ff00ffff640 10 monclient: _finish_hunting 0 2024-09-16T03:45:19.858 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.849+0000 7ff00ffff640 1 monclient: found mon.noname-a 2024-09-16T03:45:19.859 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.849+0000 7ff00ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:45:19.859 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.849+0000 7ff00ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:45:19.859 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.849+0000 7ff00ffff640 1 -- 172.21.15.42:0/1781644058 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff01806dd60 con 0x7ff0180a3d90 2024-09-16T03:45:19.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.853+0000 7ff00ffff640 1 --2- 172.21.15.42:0/1781644058 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff0180a3d90 0x7ff01813c380 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7ff008009870 tx=0x7ff00801c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2272a33f9ead557d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:45:19.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.853+0000 7ff00f7fe640 1 -- 172.21.15.42:0/1781644058 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0080b0070 con 0x7ff0180a3d90 2024-09-16T03:45:19.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.853+0000 7ff00f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:45:19.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.853+0000 7ff00f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:45:19.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.853+0000 7ff00f7fe640 10 monclient: dump: 2024-09-16T03:45:19.866 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:45:19.866 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:45:19.866 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:19.867 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:19.867 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:45:19.867 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:45:19.867 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:45:19.867 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:45:19.867 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:45:19.867 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:45:19.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.853+0000 7ff00f7fe640 10 monclient: _finish_auth 0 2024-09-16T03:45:19.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.853+0000 7ff00f7fe640 10 monclient: _check_auth_tickets 2024-09-16T03:45:19.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.853+0000 7ff00f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:45:19.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.853+0000 7ff00f7fe640 1 -- 172.21.15.42:0/1781644058 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff0080ab600 con 0x7ff0180a3d90 2024-09-16T03:45:19.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.853+0000 7ff00f7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:45:19.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.853+0000 7ff00f7fe640 1 -- 172.21.15.42:0/1781644058 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0080ab9d0 con 0x7ff0180a3d90 2024-09-16T03:45:19.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.853+0000 7ff01ec03640 5 monclient: authenticate success, global_id 744114 2024-09-16T03:45:19.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.853+0000 7ff01ec03640 10 monclient: get_monmap_and_config success 2024-09-16T03:45:19.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.853+0000 7ff01ec03640 4 set_mon_vals no callback set 2024-09-16T03:45:19.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.853+0000 7ff00f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:45:19.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.853+0000 7ff00f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:45:19.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.853+0000 7ff00f7fe640 10 monclient: dump: 2024-09-16T03:45:19.868 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:45:19.868 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:45:19.868 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:19.868 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:19.869 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:45:19.869 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:45:19.869 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:45:19.869 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:45:19.869 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:45:19.869 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:45:19.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.853+0000 7ff01ec03640 10 monclient: shutdown 2024-09-16T03:45:19.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.853+0000 7ff01ec03640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:45:19.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.853+0000 7ff01ec03640 1 -- 172.21.15.42:0/1781644058 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff0180a3d90 msgr2=0x7ff01813c380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:45:19.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.853+0000 7ff01ec03640 1 --2- 172.21.15.42:0/1781644058 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff0180a3d90 0x7ff01813c380 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7ff008009870 tx=0x7ff00801c3e0 comp rx=0 tx=0).stop 2024-09-16T03:45:19.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.853+0000 7ff01ec03640 1 -- 172.21.15.42:0/1781644058 shutdown_connections 2024-09-16T03:45:19.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.853+0000 7ff01ec03640 1 --2- 172.21.15.42:0/1781644058 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff0180a3d90 0x7ff01813c380 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:45:19.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.853+0000 7ff01ec03640 1 -- 172.21.15.42:0/1781644058 >> 172.21.15.42:0/1781644058 conn(0x7ff0180a9fb0 msgr2=0x7ff0180aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:45:19.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.853+0000 7ff01ec03640 1 -- 172.21.15.42:0/1781644058 shutdown_connections 2024-09-16T03:45:19.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.853+0000 7ff01ec03640 1 -- 172.21.15.42:0/1781644058 wait complete. 2024-09-16T03:45:19.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.853+0000 7ff01ec03640 10 monclient: build_initial_monmap 2024-09-16T03:45:19.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.853+0000 7ff01ec03640 10 monclient: monmap: 2024-09-16T03:45:19.870 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:45:19.870 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:45:19.870 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:45:19.870 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:45:19.870 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:45:19.870 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:45:19.870 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:45:19.870 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:45:19.870 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:45:19.871 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:45:19.871 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.857+0000 7ff01ec03640 1 librados: starting msgr at 2024-09-16T03:45:19.871 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.857+0000 7ff01ec03640 1 librados: starting objecter 2024-09-16T03:45:19.871 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.857+0000 7ff01ec03640 1 Processor -- start 2024-09-16T03:45:19.871 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.857+0000 7ff01ec03640 1 -- start start 2024-09-16T03:45:19.871 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.857+0000 7ff01ec03640 1 librados: setting wanted keys 2024-09-16T03:45:19.871 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.857+0000 7ff01ec03640 1 librados: calling monclient init 2024-09-16T03:45:19.871 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.857+0000 7ff01ec03640 10 monclient: init 2024-09-16T03:45:19.871 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.857+0000 7ff01ec03640 10 monclient: _reopen_session rank -1 2024-09-16T03:45:19.871 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.857+0000 7ff01ec03640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:45:19.871 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.857+0000 7ff01ec03640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff0180a39b0 0x7ff01812c520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:45:19.871 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.857+0000 7ff01ec03640 10 monclient(hunting): picked mon.noname-c con 0x7ff0180a39b0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:45:19.871 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.857+0000 7ff01ec03640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff0180a3d90 0x7ff01812ca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:45:19.871 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.857+0000 7ff01ec03640 10 monclient(hunting): picked mon.noname-b con 0x7ff0180a3d90 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:45:19.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.857+0000 7ff01ec03640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff01813c8c0 0x7ff018139180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:45:19.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.857+0000 7ff01ec03640 10 monclient(hunting): picked mon.noname-a con 0x7ff01813c8c0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:45:19.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.857+0000 7ff01ec03640 10 monclient(hunting): start opening mon connection 2024-09-16T03:45:19.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.857+0000 7ff01ec03640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7ff018142f80 con 0x7ff01813c8c0 2024-09-16T03:45:19.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.857+0000 7ff01ec03640 10 monclient(hunting): start opening mon connection 2024-09-16T03:45:19.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.857+0000 7ff01ec03640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff018142e00 con 0x7ff0180a3d90 2024-09-16T03:45:19.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.857+0000 7ff01ec03640 10 monclient(hunting): start opening mon connection 2024-09-16T03:45:19.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.857+0000 7ff01ec03640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7ff018143100 con 0x7ff0180a39b0 2024-09-16T03:45:19.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.857+0000 7ff01ec03640 10 monclient(hunting): _renew_subs 2024-09-16T03:45:19.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.857+0000 7ff01ec03640 10 monclient(hunting): authenticate will time out at 1166.828754s 2024-09-16T03:45:19.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.857+0000 7ff01d1a0640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff01813c8c0 0x7ff018139180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:45:19.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.857+0000 7ff01c99f640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff0180a39b0 0x7ff01812c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:45:19.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.857+0000 7ff01d1a0640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff01813c8c0 0x7ff018139180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:33344/0 (socket says 172.21.15.42:33344) 2024-09-16T03:45:19.873 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.857+0000 7ff01d1a0640 1 -- 172.21.15.42:0/3794457826 learned_addr learned my addr 172.21.15.42:0/3794457826 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:45:19.873 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.857+0000 7ff01c99f640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff0180a39b0 0x7ff01812c520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:38572/0 (socket says 172.21.15.42:38572) 2024-09-16T03:45:19.873 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.857+0000 7ff01d1a0640 10 monclient(hunting): get_auth_request con 0x7ff01813c8c0 auth_method 0 2024-09-16T03:45:19.873 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.857+0000 7ff01d1a0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:45:19.873 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.857+0000 7ff01d1a0640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:45:19.873 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.857+0000 7ff01d1a0640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:45:19.873 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.857+0000 7ff01c99f640 10 monclient(hunting): get_auth_request con 0x7ff0180a39b0 auth_method 0 2024-09-16T03:45:19.873 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.857+0000 7ff01c99f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:45:19.873 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.857+0000 7ff01c99f640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:45:19.873 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.857+0000 7ff01c99f640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:45:19.873 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.857+0000 7ff01d1a0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:45:19.873 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.857+0000 7ff01d1a0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:45:19.873 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.857+0000 7ff01d1a0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:45:19.874 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.857+0000 7ff01c99f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:45:19.874 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.857+0000 7ff01c99f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:45:19.874 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.857+0000 7ff01c99f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:45:19.874 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.857+0000 7ff00ffff640 1 --2- 172.21.15.42:0/3794457826 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff0180a3d90 0x7ff01812ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:45:19.874 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.857+0000 7ff00ffff640 10 monclient(hunting): get_auth_request con 0x7ff0180a3d90 auth_method 0 2024-09-16T03:45:19.874 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.857+0000 7ff00ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:45:19.874 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.857+0000 7ff00ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:45:19.874 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.857+0000 7ff00ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:45:19.874 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.857+0000 7ff01c99f640 10 monclient(hunting): handle_auth_done global_id 13301 payload 824 2024-09-16T03:45:19.874 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.857+0000 7ff01c99f640 1 -- 172.21.15.42:0/3794457826 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff0180a3d90 msgr2=0x7ff01812ca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:45:19.874 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.857+0000 7ff01c99f640 1 --2- 172.21.15.42:0/3794457826 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff0180a3d90 0x7ff01812ca60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:45:19.874 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.857+0000 7ff01c99f640 1 -- 172.21.15.42:0/3794457826 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff01813c8c0 msgr2=0x7ff018139180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:45:19.874 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.857+0000 7ff01c99f640 1 --2- 172.21.15.42:0/3794457826 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff01813c8c0 0x7ff018139180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:45:19.875 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.857+0000 7ff01c99f640 10 monclient: _finish_hunting 0 2024-09-16T03:45:19.875 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.857+0000 7ff01c99f640 1 monclient: found mon.noname-c 2024-09-16T03:45:19.875 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.857+0000 7ff01c99f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:45:19.875 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.857+0000 7ff01c99f640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:45:19.875 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.857+0000 7ff01c99f640 1 -- 172.21.15.42:0/3794457826 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff01812e140 con 0x7ff0180a39b0 2024-09-16T03:45:19.875 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.857+0000 7ff01d1a0640 1 --2- 172.21.15.42:0/3794457826 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff01813c8c0 0x7ff018139180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:45:19.875 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.857+0000 7ff01c99f640 1 --2- 172.21.15.42:0/3794457826 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff0180a39b0 0x7ff01812c520 secure :-1 s=READY pgs=2351 cs=0 l=1 rev1=1 crypto rx=0x7ff01400c3c0 tx=0x7ff01400c880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:45:19.875 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.857+0000 7ff00d7fa640 10 client.?.objecter ms_handle_connect 0x7ff0180a39b0 2024-09-16T03:45:19.875 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.857+0000 7ff00d7fa640 10 client.?.objecter resend_mon_ops 2024-09-16T03:45:19.875 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.857+0000 7ff00d7fa640 1 -- 172.21.15.42:0/3794457826 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0140052d0 con 0x7ff0180a39b0 2024-09-16T03:45:19.875 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.857+0000 7ff00d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:45:19.875 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.857+0000 7ff00d7fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:45:19.875 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.857+0000 7ff00d7fa640 10 monclient: dump: 2024-09-16T03:45:19.875 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:45:19.876 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:45:19.876 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:19.876 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:19.876 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:45:19.876 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:45:19.876 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:45:19.876 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:45:19.876 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:45:19.876 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:45:19.876 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.857+0000 7ff00d7fa640 10 monclient: _finish_auth 0 2024-09-16T03:45:19.876 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.857+0000 7ff00d7fa640 10 monclient: _check_auth_tickets 2024-09-16T03:45:19.876 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.857+0000 7ff00d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:45:19.876 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.857+0000 7ff01ec03640 5 monclient: authenticate success, global_id 13301 2024-09-16T03:45:19.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.857+0000 7ff00d7fa640 1 -- 172.21.15.42:0/3794457826 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff014005470 con 0x7ff0180a39b0 2024-09-16T03:45:19.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.857+0000 7ff01ec03640 10 monclient: _renew_subs 2024-09-16T03:45:19.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.857+0000 7ff01ec03640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:45:19.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.857+0000 7ff01ec03640 1 -- 172.21.15.42:0/3794457826 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff01813ab00 con 0x7ff0180a39b0 2024-09-16T03:45:19.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.857+0000 7ff00d7fa640 10 monclient: handle_config config(0 keys) 2024-09-16T03:45:19.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.857+0000 7ff00d7fa640 1 -- 172.21.15.42:0/3794457826 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff014007260 con 0x7ff0180a39b0 2024-09-16T03:45:19.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.857+0000 7ff00d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:45:19.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.857+0000 7ff00d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:45:19.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.857+0000 7ff01ec03640 10 client.13301.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:45:19.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.857+0000 7ff00effd640 4 set_mon_vals no callback set 2024-09-16T03:45:19.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.857+0000 7ff00d7fa640 10 monclient: dump: 2024-09-16T03:45:19.877 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:45:19.877 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:45:19.878 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:19.878 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:19.878 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:45:19.878 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:45:19.878 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:45:19.878 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:45:19.879 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:45:19.879 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:45:19.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.857+0000 7ff01ec03640 10 monclient: _renew_subs 2024-09-16T03:45:19.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.857+0000 7ff01ec03640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:45:19.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.857+0000 7ff01ec03640 1 -- 172.21.15.42:0/3794457826 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff01813c120 con 0x7ff0180a39b0 2024-09-16T03:45:19.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.857+0000 7ff01ec03640 1 librados: init done 2024-09-16T03:45:19.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.857+0000 7ff01ec03640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:45:19.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.857+0000 7ff01ec03640 1 --2- 172.21.15.42:0/3794457826 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff0181396c0 0x7ff018139aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:45:19.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.857+0000 7ff01ec03640 10 monclient(hunting): start opening mon connection 2024-09-16T03:45:19.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.857+0000 7ff01ec03640 1 -- 172.21.15.42:0/3794457826 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7ff01813fe60 con 0x7ff0181396c0 2024-09-16T03:45:19.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.857+0000 7ff00ffff640 1 --2- 172.21.15.42:0/3794457826 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff0181396c0 0x7ff018139aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:45:19.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.857+0000 7ff00ffff640 10 monclient: get_auth_request con 0x7ff0181396c0 auth_method 0 2024-09-16T03:45:19.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.857+0000 7ff00ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:45:19.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.857+0000 7ff00ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:45:19.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.857+0000 7ff00ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:45:19.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.857+0000 7ff00ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:45:19.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.857+0000 7ff00ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:45:19.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.857+0000 7ff00ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:45:19.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.857+0000 7ff00ffff640 10 monclient(hunting): handle_auth_done global_id 744120 payload 824 2024-09-16T03:45:19.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.857+0000 7ff00ffff640 1 -- 172.21.15.42:0/3794457826 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff01813afd0 con 0x7ff0181396c0 2024-09-16T03:45:19.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.857+0000 7ff00ffff640 1 --2- 172.21.15.42:0/3794457826 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff0181396c0 0x7ff018139aa0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7ff0180a7750 tx=0x7ff00801ac20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:45:19.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.857+0000 7ff00d7fa640 10 client.13301.objecter ms_handle_connect 0x7ff0181396c0 2024-09-16T03:45:19.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.857+0000 7ff00d7fa640 10 client.13301.objecter resend_mon_ops 2024-09-16T03:45:19.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.857+0000 7ff00d7fa640 1 -- 172.21.15.42:0/3794457826 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7ff014007400 con 0x7ff0180a39b0 2024-09-16T03:45:19.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.857+0000 7ff00d7fa640 1 --2- 172.21.15.42:0/3794457826 >> v2:172.21.15.154:6812/254845342 conn(0x7ff004037810 0x7ff004039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:45:19.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.857+0000 7ff00d7fa640 1 -- 172.21.15.42:0/3794457826 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(87..87 src has 1..87) ==== 7347+0+0 (secure 0 0 0) 0x7ff014048900 con 0x7ff0180a39b0 2024-09-16T03:45:19.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.857+0000 7ff00d7fa640 10 client.13301.objecter ms_dispatch 0x7ff018000ba0 osd_map(87..87 src has 1..87) 2024-09-16T03:45:19.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.857+0000 7ff00d7fa640 3 client.13301.objecter handle_osd_map got epochs [87,87] > 0 2024-09-16T03:45:19.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.857+0000 7ff00d7fa640 3 client.13301.objecter handle_osd_map decoding full epoch 87 2024-09-16T03:45:19.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.857+0000 7ff00d7fa640 20 client.13301.objecter dump_active .. 0 homeless 2024-09-16T03:45:19.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.857+0000 7ff00d7fa640 1 -- 172.21.15.42:0/3794457826 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0080b0040 con 0x7ff0181396c0 2024-09-16T03:45:19.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.857+0000 7ff00d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:45:19.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.857+0000 7ff00d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:45:19.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.857+0000 7ff00d7fa640 10 monclient: dump: 2024-09-16T03:45:19.881 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:45:19.881 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:45:19.881 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:19.881 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:19.881 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:45:19.881 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:45:19.882 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:45:19.882 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:45:19.882 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:45:19.882 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:45:19.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.857+0000 7ff01d1a0640 1 --2- 172.21.15.42:0/3794457826 >> v2:172.21.15.154:6812/254845342 conn(0x7ff004037810 0x7ff004039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:45:19.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.857+0000 7ff01d1a0640 10 monclient: get_auth_request con 0x7ff004037810 auth_method 0 2024-09-16T03:45:19.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.861+0000 7ff00d7fa640 1 -- 172.21.15.42:0/3794457826 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7ff0080b5660 con 0x7ff0181396c0 2024-09-16T03:45:19.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.861+0000 7ff00d7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:45:19.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.861+0000 7ff00d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:45:19.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.861+0000 7ff00d7fa640 1 -- 172.21.15.42:0/3794457826 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff0181396c0 msgr2=0x7ff018139aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:45:19.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.861+0000 7ff00d7fa640 1 --2- 172.21.15.42:0/3794457826 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff0181396c0 0x7ff018139aa0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7ff0180a7750 tx=0x7ff00801ac20 comp rx=0 tx=0).stop 2024-09-16T03:45:19.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.861+0000 7ff00d7fa640 1 -- 172.21.15.42:0/3794457826 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff0181396c0 msgr2=0x7ff018139aa0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:45:19.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.861+0000 7ff00d7fa640 1 --2- 172.21.15.42:0/3794457826 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff0181396c0 0x7ff018139aa0 secure :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7ff0180a7750 tx=0x7ff00801ac20 comp rx=0 tx=0).stop 2024-09-16T03:45:19.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.861+0000 7ff00d7fa640 10 client.13301.objecter ms_handle_connect 0x7ff004037810 2024-09-16T03:45:19.883 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.861+0000 7ff01d1a0640 1 --2- 172.21.15.42:0/3794457826 >> v2:172.21.15.154:6812/254845342 conn(0x7ff004037810 0x7ff004039cd0 secure :-1 s=READY pgs=3355 cs=0 l=1 rev1=1 crypto rx=0x7ff018132de0 tx=0x7ff010003df0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:45:19.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.865+0000 7ff01ec03640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:45:19.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.865+0000 7ff01ec03640 1 --2- 172.21.15.42:0/3794457826 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff018139fe0 0x7ff01813cca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:45:19.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.865+0000 7ff01ec03640 10 monclient(hunting): start opening mon connection 2024-09-16T03:45:19.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.865+0000 7ff01ec03640 1 -- 172.21.15.42:0/3794457826 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7ff018133b80 con 0x7ff018139fe0 2024-09-16T03:45:19.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.865+0000 7ff00ffff640 1 --2- 172.21.15.42:0/3794457826 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff018139fe0 0x7ff01813cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:45:19.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.865+0000 7ff00ffff640 10 monclient: get_auth_request con 0x7ff018139fe0 auth_method 0 2024-09-16T03:45:19.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.865+0000 7ff00ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:45:19.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.865+0000 7ff00ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:45:19.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.865+0000 7ff00ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:45:19.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.865+0000 7ff00ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:45:19.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.865+0000 7ff00ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:45:19.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.865+0000 7ff00ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:45:19.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.865+0000 7ff00ffff640 10 monclient(hunting): handle_auth_done global_id 744123 payload 824 2024-09-16T03:45:19.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.865+0000 7ff00ffff640 1 -- 172.21.15.42:0/3794457826 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ff0180a5750 con 0x7ff018139fe0 2024-09-16T03:45:19.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.865+0000 7ff00ffff640 1 --2- 172.21.15.42:0/3794457826 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff018139fe0 0x7ff01813cca0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7ff008000c10 tx=0x7ff008019180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:45:19.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.865+0000 7ff00d7fa640 10 client.13301.objecter ms_handle_connect 0x7ff018139fe0 2024-09-16T03:45:19.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.865+0000 7ff00d7fa640 10 client.13301.objecter resend_mon_ops 2024-09-16T03:45:19.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.865+0000 7ff00d7fa640 1 -- 172.21.15.42:0/3794457826 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0080c3070 con 0x7ff018139fe0 2024-09-16T03:45:19.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.865+0000 7ff00d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:45:19.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.865+0000 7ff00d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:45:19.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.865+0000 7ff00d7fa640 10 monclient: dump: 2024-09-16T03:45:19.886 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:45:19.886 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:45:19.886 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:19.886 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:19.886 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:45:19.886 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:45:19.886 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:45:19.886 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:45:19.886 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:45:19.886 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:45:19.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.865+0000 7ff00d7fa640 1 -- 172.21.15.42:0/3794457826 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4712 (secure 0 0 0) 0x7ff0080bfc00 con 0x7ff018139fe0 2024-09-16T03:45:19.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.865+0000 7ff00d7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:45:19.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.865+0000 7ff00d7fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:45:19.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.865+0000 7ff00d7fa640 1 -- 172.21.15.42:0/3794457826 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff018139fe0 msgr2=0x7ff01813cca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:45:19.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.865+0000 7ff00d7fa640 1 --2- 172.21.15.42:0/3794457826 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff018139fe0 0x7ff01813cca0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7ff008000c10 tx=0x7ff008019180 comp rx=0 tx=0).stop 2024-09-16T03:45:19.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.865+0000 7ff00d7fa640 1 -- 172.21.15.42:0/3794457826 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff018139fe0 msgr2=0x7ff01813cca0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:45:19.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.865+0000 7ff00d7fa640 1 --2- 172.21.15.42:0/3794457826 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff018139fe0 0x7ff01813cca0 secure :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7ff008000c10 tx=0x7ff008019180 comp rx=0 tx=0).stop 2024-09-16T03:45:19.887 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:45:19.887 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:45:19.887 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:45:19.887 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "leader", 2024-09-16T03:45:19.887 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 28, 2024-09-16T03:45:19.888 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 7235, 2024-09-16T03:45:19.888 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:45:19.888 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:45:19.888 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T03:45:19.888 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T03:45:19.888 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:45:19.888 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 0, 2024-09-16T03:45:19.888 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:45:19.888 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:45:19.889 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:45:19.889 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:45:19.889 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:45:19.889 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:45:19.889 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:45:19.889 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:45:19.889 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:45:19.889 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:45:19.889 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:45:19.889 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:45:19.889 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:45:19.889 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:45:19.889 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:45:19.890 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:45:19.890 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:45:19.890 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:45:19.890 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:45:19.890 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:45:19.890 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:45:19.890 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:45:19.890 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:45:19.890 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:45:19.890 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:45:19.890 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:45:19.890 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:45:19.890 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:45:19.890 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:45:19.891 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:45:19.891 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:45:19.891 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:45:19.891 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:45:19.891 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:45:19.891 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:45:19.891 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:45:19.891 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:45:19.891 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:45:19.891 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:45:19.891 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:45:19.891 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:45:19.891 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:45:19.891 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:45:19.892 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:45:19.892 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:45:19.892 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:45:19.892 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:45:19.892 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:45:19.892 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:45:19.892 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:45:19.892 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:45:19.892 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:45:19.892 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:45:19.892 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:45:19.892 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:45:19.892 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:45:19.892 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:45:19.893 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:45:19.893 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:45:19.893 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:45:19.893 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:45:19.893 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:45:19.893 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:45:19.893 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:45:19.893 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:45:19.893 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:45:19.893 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:45:19.893 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:45:19.893 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:45:19.893 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:45:19.893 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:45:19.894 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:45:19.894 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:45:19.894 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:45:19.894 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:45:19.894 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:45:19.894 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:45:19.894 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:45:19.894 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:45:19.894 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:45:19.894 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:45:19.894 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:45:19.894 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:45:19.894 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:45:19.894 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:45:19.895 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:45:19.895 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:45:19.895 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:45:19.895 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:45:19.895 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:45:19.895 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:45:19.895 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:45:19.895 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:45:19.895 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:45:19.895 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:45:19.895 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:45:19.895 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:45:19.895 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:45:19.895 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:45:19.896 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:45:19.896 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:45:19.896 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:45:19.896 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:45:19.896 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:45:19.896 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:45:19.896 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:45:19.896 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:45:19.896 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:45:19.896 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:45:19.896 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:45:19.896 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:45:19.896 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:45:19.896 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:45:19.897 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:45:19.897 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:45:19.897 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:45:19.897 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:45:19.897 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:45:19.897 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:45:19.897 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:45:19.897 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:45:19.897 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:45:19.897 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:45:19.897 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:45:19.897 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:45:19.897 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:45:19.897 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:45:19.898 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:45:19.898 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:45:19.898 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:45:19.898 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:45:19.898 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:45:19.898 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:45:19.898 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:45:19.898 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:45:19.898 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:45:19.898 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:45:19.898 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:45:19.898 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:45:19.898 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:45:19.898 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:45:19.899 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T03:45:19.899 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:45:19.899 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:45:19.899 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:45:19.899 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:45:19.899 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:45:19.899 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:45:19.899 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:45:19.899 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:45:19.899 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:45:19.899 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:45:19.899 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:45:19.900 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:45:19.900 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:45:19.900 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:45:19.900 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:45:19.900 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:45:19.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.865+0000 7ff01ec03640 10 librados: watch_flush enter 2024-09-16T03:45:19.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.865+0000 7ff01ec03640 10 librados: watch_flush exit 2024-09-16T03:45:19.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.865+0000 7ff01ec03640 20 client.13301.objecter shutdown clearing up homeless session... 2024-09-16T03:45:19.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.865+0000 7ff01ec03640 10 client.13301.objecter successfully canceled tick 2024-09-16T03:45:19.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.865+0000 7ff01ec03640 1 -- 172.21.15.42:0/3794457826 >> v2:172.21.15.154:6812/254845342 conn(0x7ff004037810 msgr2=0x7ff004039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:45:19.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.865+0000 7ff01ec03640 1 --2- 172.21.15.42:0/3794457826 >> v2:172.21.15.154:6812/254845342 conn(0x7ff004037810 0x7ff004039cd0 secure :-1 s=READY pgs=3355 cs=0 l=1 rev1=1 crypto rx=0x7ff018132de0 tx=0x7ff010003df0 comp rx=0 tx=0).stop 2024-09-16T03:45:19.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.865+0000 7ff01ec03640 10 monclient: shutdown 2024-09-16T03:45:19.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.865+0000 7ff01ec03640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:45:19.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.865+0000 7ff01ec03640 1 -- 172.21.15.42:0/3794457826 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff0180a39b0 msgr2=0x7ff01812c520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:45:19.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.865+0000 7ff01ec03640 1 --2- 172.21.15.42:0/3794457826 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff0180a39b0 0x7ff01812c520 secure :-1 s=READY pgs=2351 cs=0 l=1 rev1=1 crypto rx=0x7ff01400c3c0 tx=0x7ff01400c880 comp rx=0 tx=0).stop 2024-09-16T03:45:19.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.865+0000 7ff01d1a0640 1 -- 172.21.15.42:0/3794457826 reap_dead start 2024-09-16T03:45:19.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.865+0000 7ff01d1a0640 1 -- 172.21.15.42:0/3794457826 reap_dead start 2024-09-16T03:45:19.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.865+0000 7ff01ec03640 1 -- 172.21.15.42:0/3794457826 shutdown_connections 2024-09-16T03:45:19.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.865+0000 7ff01ec03640 1 -- 172.21.15.42:0/3794457826 >> 172.21.15.42:0/3794457826 conn(0x7ff0180a9fb0 msgr2=0x7ff018136bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:45:19.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.865+0000 7ff01ec03640 1 -- 172.21.15.42:0/3794457826 shutdown_connections 2024-09-16T03:45:19.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.865+0000 7ff01ec03640 1 -- 172.21.15.42:0/3794457826 wait complete. 2024-09-16T03:45:19.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:19.865+0000 7ff01ec03640 1 librados: shutdown 2024-09-16T03:45:19.903 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:45:19.903 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T03:45:19.903 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T03:45:20.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.013+0000 7f181fb1d640 10 monclient: get_monmap_and_config 2024-09-16T03:45:20.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.013+0000 7f181fb1d640 10 monclient: build_initial_monmap 2024-09-16T03:45:20.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.013+0000 7f181fb1d640 10 monclient: monmap: 2024-09-16T03:45:20.017 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:45:20.017 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:45:20.017 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:45:20.016277+0000 2024-09-16T03:45:20.017 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:45:20.016277+0000 2024-09-16T03:45:20.017 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:45:20.017 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:45:20.017 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:45:20.017 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:45:20.018 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:45:20.018 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:45:20.018 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:45:20.018 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:45:20.019 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.013+0000 7f181fb1d640 1 Processor -- start 2024-09-16T03:45:20.019 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.013+0000 7f181fb1d640 1 -- start start 2024-09-16T03:45:20.019 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.013+0000 7f181fb1d640 10 monclient: init 2024-09-16T03:45:20.020 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.013+0000 7f181fb1d640 10 monclient: _reopen_session rank -1 2024-09-16T03:45:20.020 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.013+0000 7f181fb1d640 10 monclient: _add_conns ranks=[1,3,2,0,4] 2024-09-16T03:45:20.020 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.013+0000 7f181fb1d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f18180a39b0 addr v1:172.21.15.154:6789/0 2024-09-16T03:45:20.020 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.013+0000 7f181fb1d640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f18180a3d90 0x7f181813c380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:45:20.020 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.013+0000 7f181fb1d640 10 monclient(hunting): picked mon.noname-a con 0x7f18180a3d90 addr v2:172.21.15.42:3300/0 2024-09-16T03:45:20.020 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.013+0000 7f181fb1d640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f181813c8c0 0x7f181813ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:45:20.020 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.013+0000 7f181fb1d640 10 monclient(hunting): picked mon.noname-b con 0x7f181813c8c0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:45:20.020 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.013+0000 7f181fb1d640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1818134290 con 0x7f18180a39b0 2024-09-16T03:45:20.020 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.013+0000 7f181fb1d640 10 monclient(hunting): start opening mon connection 2024-09-16T03:45:20.020 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.013+0000 7f181fb1d640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f181809c140 con 0x7f18180a3d90 2024-09-16T03:45:20.020 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.013+0000 7f181fb1d640 10 monclient(hunting): start opening mon connection 2024-09-16T03:45:20.020 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.013+0000 7f181fb1d640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f181809c420 con 0x7f181813c8c0 2024-09-16T03:45:20.020 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.013+0000 7f181fb1d640 10 monclient(hunting): _renew_subs 2024-09-16T03:45:20.020 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.013+0000 7f181fb1d640 10 monclient(hunting): authenticate will time out at 1166.985561s 2024-09-16T03:45:20.021 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.013+0000 7f181d0b8640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f18180a3d90 0x7f181813c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:45:20.021 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.013+0000 7f181e0ba640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f181813c8c0 0x7f181813ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:45:20.021 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.013+0000 7f181d0b8640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f18180a3d90 0x7f181813c380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:33364/0 (socket says 172.21.15.42:33364) 2024-09-16T03:45:20.021 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.013+0000 7f181d0b8640 1 -- 172.21.15.42:0/2627477758 learned_addr learned my addr 172.21.15.42:0/2627477758 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:45:20.021 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.013+0000 7f181d0b8640 10 monclient(hunting): get_auth_request con 0x7f18180a3d90 auth_method 0 2024-09-16T03:45:20.021 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.013+0000 7f181d0b8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:45:20.021 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.013+0000 7f181d0b8640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:45:20.021 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.013+0000 7f181d0b8640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:45:20.021 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.013+0000 7f181e0ba640 10 monclient(hunting): get_auth_request con 0x7f181813c8c0 auth_method 0 2024-09-16T03:45:20.021 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.013+0000 7f181e0ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:45:20.021 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.013+0000 7f181e0ba640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:45:20.021 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.013+0000 7f181e0ba640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:45:20.021 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.013+0000 7f181d0b8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:45:20.021 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.013+0000 7f181d0b8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:45:20.022 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.013+0000 7f181d0b8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:45:20.022 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.013+0000 7f181e0ba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:45:20.022 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.013+0000 7f181e0ba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:45:20.022 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.013+0000 7f181e0ba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:45:20.022 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.013+0000 7f181d0b8640 10 monclient(hunting): handle_auth_done global_id 744126 payload 274 2024-09-16T03:45:20.022 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.013+0000 7f181d0b8640 1 -- 172.21.15.42:0/2627477758 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f181813c8c0 msgr2=0x7f181813ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:45:20.022 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.013+0000 7f181d0b8640 1 --2- 172.21.15.42:0/2627477758 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f181813c8c0 0x7f181813ecb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:45:20.022 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.013+0000 7f181d0b8640 1 -- 172.21.15.42:0/2627477758 >> v1:172.21.15.154:6789/0 conn(0x7f18180a39b0 legacy=0x7f1818133b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:45:20.022 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.013+0000 7f181d0b8640 10 monclient: _finish_hunting 0 2024-09-16T03:45:20.023 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.013+0000 7f181d0b8640 1 monclient: found mon.noname-a 2024-09-16T03:45:20.023 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.013+0000 7f181d0b8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:45:20.023 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.013+0000 7f181d0b8640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:45:20.023 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.013+0000 7f181d0b8640 1 -- 172.21.15.42:0/2627477758 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f181806dd60 con 0x7f18180a3d90 2024-09-16T03:45:20.029 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.017+0000 7f181d0b8640 1 --2- 172.21.15.42:0/2627477758 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f18180a3d90 0x7f181813c380 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f1808009870 tx=0x7f180801c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d859cb567d08ce6 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:45:20.029 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.017+0000 7f181c8b7640 1 -- 172.21.15.42:0/2627477758 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18080b0070 con 0x7f18180a3d90 2024-09-16T03:45:20.029 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.017+0000 7f181c8b7640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:45:20.029 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.017+0000 7f181c8b7640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:45:20.029 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.017+0000 7f181c8b7640 10 monclient: dump: 2024-09-16T03:45:20.029 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:45:20.030 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:45:20.030 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:20.030 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:20.030 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:45:20.030 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:45:20.030 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:45:20.030 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:45:20.030 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:45:20.030 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:45:20.030 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.017+0000 7f181c8b7640 10 monclient: _finish_auth 0 2024-09-16T03:45:20.030 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.017+0000 7f181c8b7640 10 monclient: _check_auth_tickets 2024-09-16T03:45:20.030 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.017+0000 7f181c8b7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:45:20.030 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.017+0000 7f181fb1d640 5 monclient: authenticate success, global_id 744126 2024-09-16T03:45:20.030 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.017+0000 7f181c8b7640 1 -- 172.21.15.42:0/2627477758 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f18080ab600 con 0x7f18180a3d90 2024-09-16T03:45:20.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.017+0000 7f181c8b7640 10 monclient: handle_config config(0 keys) 2024-09-16T03:45:20.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.017+0000 7f181fb1d640 10 monclient: get_monmap_and_config success 2024-09-16T03:45:20.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.017+0000 7f181fb1d640 4 set_mon_vals no callback set 2024-09-16T03:45:20.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.017+0000 7f181c8b7640 1 -- 172.21.15.42:0/2627477758 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18080ab9d0 con 0x7f18180a3d90 2024-09-16T03:45:20.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.017+0000 7f181c8b7640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:45:20.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.017+0000 7f181c8b7640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:45:20.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.017+0000 7f181c8b7640 10 monclient: dump: 2024-09-16T03:45:20.031 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:45:20.031 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:45:20.031 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:20.031 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:20.031 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:45:20.031 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:45:20.031 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:45:20.032 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:45:20.032 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:45:20.032 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:45:20.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.017+0000 7f181fb1d640 10 monclient: shutdown 2024-09-16T03:45:20.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.017+0000 7f181fb1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:45:20.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.017+0000 7f181fb1d640 1 -- 172.21.15.42:0/2627477758 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f18180a3d90 msgr2=0x7f181813c380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:45:20.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.017+0000 7f181fb1d640 1 --2- 172.21.15.42:0/2627477758 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f18180a3d90 0x7f181813c380 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f1808009870 tx=0x7f180801c3e0 comp rx=0 tx=0).stop 2024-09-16T03:45:20.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.017+0000 7f181fb1d640 1 -- 172.21.15.42:0/2627477758 shutdown_connections 2024-09-16T03:45:20.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.017+0000 7f181fb1d640 1 --2- 172.21.15.42:0/2627477758 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f181813c8c0 0x7f181813ecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:45:20.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.017+0000 7f181fb1d640 1 --2- 172.21.15.42:0/2627477758 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f18180a3d90 0x7f181813c380 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:45:20.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.017+0000 7f181fb1d640 1 -- 172.21.15.42:0/2627477758 >> 172.21.15.42:0/2627477758 conn(0x7f18180a9fb0 msgr2=0x7f18180aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:45:20.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.017+0000 7f181fb1d640 1 -- 172.21.15.42:0/2627477758 shutdown_connections 2024-09-16T03:45:20.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.017+0000 7f181fb1d640 1 -- 172.21.15.42:0/2627477758 wait complete. 2024-09-16T03:45:20.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.017+0000 7f181fb1d640 10 monclient: build_initial_monmap 2024-09-16T03:45:20.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.017+0000 7f181fb1d640 10 monclient: monmap: 2024-09-16T03:45:20.033 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:45:20.033 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:45:20.033 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:45:20.033 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:45:20.033 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:45:20.033 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:45:20.033 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:45:20.033 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:45:20.034 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:45:20.034 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:45:20.034 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.017+0000 7f181fb1d640 1 librados: starting msgr at 2024-09-16T03:45:20.034 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.017+0000 7f181fb1d640 1 librados: starting objecter 2024-09-16T03:45:20.034 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.017+0000 7f181fb1d640 1 Processor -- start 2024-09-16T03:45:20.034 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.017+0000 7f181fb1d640 1 -- start start 2024-09-16T03:45:20.034 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.017+0000 7f181fb1d640 1 librados: setting wanted keys 2024-09-16T03:45:20.034 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.017+0000 7f181fb1d640 1 librados: calling monclient init 2024-09-16T03:45:20.034 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.017+0000 7f181fb1d640 10 monclient: init 2024-09-16T03:45:20.034 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.017+0000 7f181fb1d640 10 monclient: _reopen_session rank -1 2024-09-16T03:45:20.034 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.017+0000 7f181fb1d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:45:20.034 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.017+0000 7f181fb1d640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f18180a39b0 0x7f1818135c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:45:20.034 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.017+0000 7f181fb1d640 10 monclient(hunting): picked mon.noname-a con 0x7f18180a39b0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:45:20.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.017+0000 7f181fb1d640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f18180a3d90 0x7f18181361d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:45:20.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.017+0000 7f181fb1d640 10 monclient(hunting): picked mon.noname-c con 0x7f18180a3d90 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:45:20.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.017+0000 7f181fb1d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f181813c8c0 0x7f1818136710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:45:20.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.017+0000 7f181fb1d640 10 monclient(hunting): picked mon.noname-b con 0x7f181813c8c0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:45:20.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.017+0000 7f181fb1d640 10 monclient(hunting): start opening mon connection 2024-09-16T03:45:20.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.017+0000 7f181fb1d640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f1818141da0 con 0x7f18180a39b0 2024-09-16T03:45:20.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.017+0000 7f181fb1d640 10 monclient(hunting): start opening mon connection 2024-09-16T03:45:20.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.017+0000 7f181fb1d640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f1818141c20 con 0x7f181813c8c0 2024-09-16T03:45:20.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.017+0000 7f181fb1d640 10 monclient(hunting): start opening mon connection 2024-09-16T03:45:20.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.017+0000 7f181fb1d640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f1818141f20 con 0x7f18180a3d90 2024-09-16T03:45:20.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.017+0000 7f181fb1d640 10 monclient(hunting): _renew_subs 2024-09-16T03:45:20.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.017+0000 7f181fb1d640 10 monclient(hunting): authenticate will time out at 1166.990780s 2024-09-16T03:45:20.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.017+0000 7f181d8b9640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f18180a39b0 0x7f1818135c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:45:20.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.017+0000 7f181d0b8640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f18180a3d90 0x7f18181361d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:45:20.036 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.017+0000 7f181d0b8640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f18180a3d90 0x7f18181361d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:38586/0 (socket says 172.21.15.42:38586) 2024-09-16T03:45:20.036 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.017+0000 7f181d8b9640 10 monclient(hunting): get_auth_request con 0x7f18180a39b0 auth_method 0 2024-09-16T03:45:20.036 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.017+0000 7f181d0b8640 1 -- 172.21.15.42:0/4224753831 learned_addr learned my addr 172.21.15.42:0/4224753831 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:45:20.036 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.017+0000 7f181d8b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:45:20.036 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.017+0000 7f181d8b9640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:45:20.036 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.017+0000 7f181d8b9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:45:20.036 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.017+0000 7f181d0b8640 10 monclient(hunting): get_auth_request con 0x7f18180a3d90 auth_method 0 2024-09-16T03:45:20.036 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.017+0000 7f181d0b8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:45:20.036 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.017+0000 7f181d0b8640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:45:20.036 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.017+0000 7f181d0b8640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:45:20.036 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.017+0000 7f181d8b9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:45:20.036 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.017+0000 7f181d8b9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:45:20.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.017+0000 7f181d8b9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:45:20.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.017+0000 7f181d0b8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:45:20.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.017+0000 7f181d0b8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:45:20.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.017+0000 7f181d0b8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:45:20.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.017+0000 7f181e0ba640 1 --2- 172.21.15.42:0/4224753831 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f181813c8c0 0x7f1818136710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:45:20.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.017+0000 7f181e0ba640 10 monclient(hunting): get_auth_request con 0x7f181813c8c0 auth_method 0 2024-09-16T03:45:20.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.017+0000 7f181e0ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:45:20.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.017+0000 7f181e0ba640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:45:20.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.017+0000 7f181e0ba640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:45:20.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.017+0000 7f181d0b8640 10 monclient(hunting): handle_auth_done global_id 13307 payload 824 2024-09-16T03:45:20.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.017+0000 7f181d0b8640 1 -- 172.21.15.42:0/4224753831 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f181813c8c0 msgr2=0x7f1818136710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:45:20.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.017+0000 7f181d0b8640 1 --2- 172.21.15.42:0/4224753831 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f181813c8c0 0x7f1818136710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:45:20.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.017+0000 7f181d0b8640 1 -- 172.21.15.42:0/4224753831 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f18180a39b0 msgr2=0x7f1818135c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:45:20.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.017+0000 7f181d0b8640 1 --2- 172.21.15.42:0/4224753831 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f18180a39b0 0x7f1818135c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:45:20.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.017+0000 7f181d0b8640 10 monclient: _finish_hunting 0 2024-09-16T03:45:20.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.017+0000 7f181d0b8640 1 monclient: found mon.noname-c 2024-09-16T03:45:20.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.017+0000 7f181d0b8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:45:20.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.017+0000 7f181d0b8640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:45:20.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.017+0000 7f181d0b8640 1 -- 172.21.15.42:0/4224753831 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f181813bca0 con 0x7f18180a3d90 2024-09-16T03:45:20.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.017+0000 7f181d8b9640 1 --2- 172.21.15.42:0/4224753831 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f18180a39b0 0x7f1818135c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:45:20.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.017+0000 7f181d0b8640 1 --2- 172.21.15.42:0/4224753831 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f18180a3d90 0x7f18181361d0 secure :-1 s=READY pgs=2352 cs=0 l=1 rev1=1 crypto rx=0x7f18080ae040 tx=0x7f1808019180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:45:20.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.017+0000 7f180e7fc640 10 client.?.objecter ms_handle_connect 0x7f18180a3d90 2024-09-16T03:45:20.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.017+0000 7f180e7fc640 10 client.?.objecter resend_mon_ops 2024-09-16T03:45:20.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.017+0000 7f180e7fc640 1 -- 172.21.15.42:0/4224753831 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18080b0040 con 0x7f18180a3d90 2024-09-16T03:45:20.039 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.017+0000 7f180e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:45:20.039 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.017+0000 7f180e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:45:20.039 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.017+0000 7f180e7fc640 10 monclient: dump: 2024-09-16T03:45:20.039 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:45:20.039 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:45:20.039 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:20.039 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:20.039 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:45:20.039 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:45:20.039 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:45:20.039 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:45:20.039 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:45:20.039 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:45:20.039 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.017+0000 7f180e7fc640 10 monclient: _finish_auth 0 2024-09-16T03:45:20.040 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.017+0000 7f180e7fc640 10 monclient: _check_auth_tickets 2024-09-16T03:45:20.040 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.017+0000 7f180e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:45:20.040 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.017+0000 7f180e7fc640 1 -- 172.21.15.42:0/4224753831 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f18080b4440 con 0x7f18180a3d90 2024-09-16T03:45:20.040 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.017+0000 7f181fb1d640 5 monclient: authenticate success, global_id 13307 2024-09-16T03:45:20.040 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.017+0000 7f181fb1d640 10 monclient: _renew_subs 2024-09-16T03:45:20.040 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.017+0000 7f181fb1d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:45:20.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.017+0000 7f181fb1d640 1 -- 172.21.15.42:0/4224753831 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18181378f0 con 0x7f18180a3d90 2024-09-16T03:45:20.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.017+0000 7f180e7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:45:20.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.017+0000 7f180e7fc640 1 -- 172.21.15.42:0/4224753831 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18080b4830 con 0x7f18180a3d90 2024-09-16T03:45:20.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.017+0000 7f180e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:45:20.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.017+0000 7f180e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:45:20.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.017+0000 7f180e7fc640 10 monclient: dump: 2024-09-16T03:45:20.041 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:45:20.041 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:45:20.041 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:20.041 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:20.041 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:45:20.041 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:45:20.041 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:45:20.041 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:45:20.042 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:45:20.042 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:45:20.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.017+0000 7f180f7fe640 4 set_mon_vals no callback set 2024-09-16T03:45:20.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.017+0000 7f181fb1d640 10 client.13307.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:45:20.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.017+0000 7f181fb1d640 10 monclient: _renew_subs 2024-09-16T03:45:20.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.017+0000 7f181fb1d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:45:20.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.017+0000 7f181fb1d640 1 -- 172.21.15.42:0/4224753831 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f1818139c20 con 0x7f18180a3d90 2024-09-16T03:45:20.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.017+0000 7f181fb1d640 1 librados: init done 2024-09-16T03:45:20.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.021+0000 7f181fb1d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:45:20.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.021+0000 7f181fb1d640 1 --2- 172.21.15.42:0/4224753831 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f18181cf1d0 0x7f18181cf5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:45:20.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.021+0000 7f181fb1d640 10 monclient(hunting): start opening mon connection 2024-09-16T03:45:20.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.021+0000 7f181fb1d640 1 -- 172.21.15.42:0/4224753831 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f1818133b80 con 0x7f18181cf1d0 2024-09-16T03:45:20.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.021+0000 7f181d8b9640 1 --2- 172.21.15.42:0/4224753831 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f18181cf1d0 0x7f18181cf5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:45:20.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.021+0000 7f181d8b9640 10 monclient: get_auth_request con 0x7f18181cf1d0 auth_method 0 2024-09-16T03:45:20.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.021+0000 7f181d8b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:45:20.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.021+0000 7f181d8b9640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:45:20.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.021+0000 7f181d8b9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:45:20.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.021+0000 7f181d8b9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:45:20.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.021+0000 7f181d8b9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:45:20.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.021+0000 7f181d8b9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:45:20.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.021+0000 7f181d8b9640 10 monclient(hunting): handle_auth_done global_id 13310 payload 824 2024-09-16T03:45:20.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.021+0000 7f180e7fc640 1 -- 172.21.15.42:0/4224753831 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f18080bb020 con 0x7f18180a3d90 2024-09-16T03:45:20.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.021+0000 7f181d8b9640 1 -- 172.21.15.42:0/4224753831 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f18180a5750 con 0x7f18181cf1d0 2024-09-16T03:45:20.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.021+0000 7f181d8b9640 1 --2- 172.21.15.42:0/4224753831 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f18181cf1d0 0x7f18181cf5b0 secure :-1 s=READY pgs=2353 cs=0 l=1 rev1=1 crypto rx=0x7f181400fae0 tx=0x7f1814004510 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:45:20.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.021+0000 7f180e7fc640 1 --2- 172.21.15.42:0/4224753831 >> v2:172.21.15.154:6812/254845342 conn(0x7f17fc037840 0x7f17fc039d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:45:20.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.021+0000 7f180e7fc640 10 client.13307.objecter ms_handle_connect 0x7f18181cf1d0 2024-09-16T03:45:20.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.021+0000 7f180e7fc640 10 client.13307.objecter resend_mon_ops 2024-09-16T03:45:20.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.021+0000 7f180e7fc640 1 -- 172.21.15.42:0/4224753831 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(87..87 src has 1..87) ==== 7347+0+0 (secure 0 0 0) 0x7f18080e2ad0 con 0x7f18180a3d90 2024-09-16T03:45:20.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.021+0000 7f180e7fc640 10 client.13307.objecter ms_dispatch 0x7f1818000ba0 osd_map(87..87 src has 1..87) 2024-09-16T03:45:20.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.021+0000 7f180e7fc640 3 client.13307.objecter handle_osd_map got epochs [87,87] > 0 2024-09-16T03:45:20.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.021+0000 7f180e7fc640 3 client.13307.objecter handle_osd_map decoding full epoch 87 2024-09-16T03:45:20.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.021+0000 7f180e7fc640 20 client.13307.objecter dump_active .. 0 homeless 2024-09-16T03:45:20.045 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.021+0000 7f180e7fc640 1 -- 172.21.15.42:0/4224753831 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f181400dad0 con 0x7f18181cf1d0 2024-09-16T03:45:20.045 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.021+0000 7f180e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:45:20.045 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.021+0000 7f180e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:45:20.045 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.021+0000 7f180e7fc640 10 monclient: dump: 2024-09-16T03:45:20.045 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:45:20.045 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:45:20.045 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:20.045 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:20.045 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:45:20.045 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:45:20.046 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:45:20.046 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:45:20.046 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:45:20.046 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:45:20.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.021+0000 7f181e0ba640 1 --2- 172.21.15.42:0/4224753831 >> v2:172.21.15.154:6812/254845342 conn(0x7f17fc037840 0x7f17fc039d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:45:20.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.021+0000 7f181e0ba640 10 monclient: get_auth_request con 0x7f17fc037840 auth_method 0 2024-09-16T03:45:20.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.021+0000 7f180e7fc640 1 -- 172.21.15.42:0/4224753831 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f181400dad0 con 0x7f18181cf1d0 2024-09-16T03:45:20.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.021+0000 7f180e7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:45:20.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.021+0000 7f180e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:45:20.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.021+0000 7f180e7fc640 1 -- 172.21.15.42:0/4224753831 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f18181cf1d0 msgr2=0x7f18181cf5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:45:20.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.021+0000 7f180e7fc640 1 --2- 172.21.15.42:0/4224753831 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f18181cf1d0 0x7f18181cf5b0 secure :-1 s=READY pgs=2353 cs=0 l=1 rev1=1 crypto rx=0x7f181400fae0 tx=0x7f1814004510 comp rx=0 tx=0).stop 2024-09-16T03:45:20.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.021+0000 7f180e7fc640 1 -- 172.21.15.42:0/4224753831 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f18181cf1d0 msgr2=0x7f18181cf5b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:45:20.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.021+0000 7f180e7fc640 1 --2- 172.21.15.42:0/4224753831 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f18181cf1d0 0x7f18181cf5b0 secure :-1 s=CLOSED pgs=2353 cs=0 l=1 rev1=1 crypto rx=0x7f181400fae0 tx=0x7f1814004510 comp rx=0 tx=0).stop 2024-09-16T03:45:20.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.021+0000 7f180e7fc640 10 client.13307.objecter ms_handle_connect 0x7f17fc037840 2024-09-16T03:45:20.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.021+0000 7f181e0ba640 1 --2- 172.21.15.42:0/4224753831 >> v2:172.21.15.154:6812/254845342 conn(0x7f17fc037840 0x7f17fc039d00 secure :-1 s=READY pgs=3356 cs=0 l=1 rev1=1 crypto rx=0x7f18100074a0 tx=0x7f181000b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:45:20.048 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.029+0000 7f181fb1d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:45:20.048 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.029+0000 7f181fb1d640 1 --2- 172.21.15.42:0/4224753831 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f18181cfaf0 0x7f181812e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:45:20.048 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.029+0000 7f181fb1d640 10 monclient(hunting): start opening mon connection 2024-09-16T03:45:20.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.029+0000 7f181fb1d640 1 -- 172.21.15.42:0/4224753831 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f1818068e40 con 0x7f18181cfaf0 2024-09-16T03:45:20.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.029+0000 7f181d8b9640 1 --2- 172.21.15.42:0/4224753831 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f18181cfaf0 0x7f181812e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:45:20.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.029+0000 7f181d8b9640 10 monclient: get_auth_request con 0x7f18181cfaf0 auth_method 0 2024-09-16T03:45:20.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.029+0000 7f181d8b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:45:20.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.029+0000 7f181d8b9640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:45:20.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.029+0000 7f181d8b9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:45:20.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.029+0000 7f181d8b9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:45:20.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.029+0000 7f181d8b9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:45:20.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.029+0000 7f181d8b9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:45:20.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.029+0000 7f181d8b9640 10 monclient(hunting): handle_auth_done global_id 13313 payload 824 2024-09-16T03:45:20.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.029+0000 7f181d8b9640 1 -- 172.21.15.42:0/4224753831 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f1818138770 con 0x7f18181cfaf0 2024-09-16T03:45:20.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.029+0000 7f180e7fc640 10 client.13307.objecter ms_handle_connect 0x7f18181cfaf0 2024-09-16T03:45:20.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.029+0000 7f181d8b9640 1 --2- 172.21.15.42:0/4224753831 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f18181cfaf0 0x7f181812e680 secure :-1 s=READY pgs=2354 cs=0 l=1 rev1=1 crypto rx=0x7f181400fae0 tx=0x7f1814003770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:45:20.050 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.029+0000 7f180e7fc640 10 client.13307.objecter resend_mon_ops 2024-09-16T03:45:20.050 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.029+0000 7f180e7fc640 1 -- 172.21.15.42:0/4224753831 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f181401ee30 con 0x7f18181cfaf0 2024-09-16T03:45:20.050 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.029+0000 7f180e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:45:20.050 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.029+0000 7f180e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:45:20.050 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.029+0000 7f180e7fc640 10 monclient: dump: 2024-09-16T03:45:20.050 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:45:20.050 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:45:20.050 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:20.050 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:20.050 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:45:20.050 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:45:20.050 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:45:20.050 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:45:20.050 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:45:20.051 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:45:20.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.029+0000 7f180e7fc640 1 -- 172.21.15.42:0/4224753831 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4879 (secure 0 0 0) 0x7f18140045e0 con 0x7f18181cfaf0 2024-09-16T03:45:20.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.029+0000 7f180e7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:45:20.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.029+0000 7f180e7fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:45:20.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.029+0000 7f180e7fc640 1 -- 172.21.15.42:0/4224753831 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f18181cfaf0 msgr2=0x7f181812e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:45:20.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.029+0000 7f180e7fc640 1 --2- 172.21.15.42:0/4224753831 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f18181cfaf0 0x7f181812e680 secure :-1 s=READY pgs=2354 cs=0 l=1 rev1=1 crypto rx=0x7f181400fae0 tx=0x7f1814003770 comp rx=0 tx=0).stop 2024-09-16T03:45:20.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.029+0000 7f180e7fc640 1 -- 172.21.15.42:0/4224753831 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f18181cfaf0 msgr2=0x7f181812e680 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:45:20.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.029+0000 7f180e7fc640 1 --2- 172.21.15.42:0/4224753831 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f18181cfaf0 0x7f181812e680 secure :-1 s=CLOSED pgs=2354 cs=0 l=1 rev1=1 crypto rx=0x7f181400fae0 tx=0x7f1814003770 comp rx=0 tx=0).stop 2024-09-16T03:45:20.051 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:45:20.051 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:45:20.051 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:45:20.052 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T03:45:20.052 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 28, 2024-09-16T03:45:20.052 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 199399, 2024-09-16T03:45:20.052 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:45:20.052 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:45:20.052 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T03:45:20.052 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T03:45:20.052 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:45:20.052 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 1, 2024-09-16T03:45:20.052 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:45:20.052 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:45:20.052 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:45:20.052 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:45:20.052 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:45:20.053 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:45:20.053 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:45:20.053 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:45:20.053 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:45:20.053 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:45:20.053 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:45:20.053 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:45:20.053 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:45:20.053 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:45:20.054 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:45:20.054 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:45:20.054 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:45:20.054 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:45:20.054 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:45:20.054 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:45:20.054 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:45:20.054 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:45:20.054 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:45:20.054 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:45:20.054 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:45:20.054 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:45:20.054 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:45:20.054 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:45:20.055 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:45:20.055 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:45:20.055 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:45:20.055 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:45:20.055 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:45:20.055 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:45:20.055 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:45:20.055 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:45:20.055 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:45:20.055 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:45:20.055 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:45:20.055 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:45:20.055 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:45:20.055 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:45:20.056 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:45:20.056 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:45:20.056 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:45:20.056 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:45:20.056 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:45:20.056 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:45:20.056 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:45:20.056 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:45:20.056 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:45:20.056 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:45:20.056 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:45:20.056 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:45:20.056 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:45:20.056 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:45:20.057 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:45:20.057 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:45:20.057 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:45:20.057 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:45:20.057 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:45:20.057 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:45:20.057 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:45:20.057 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:45:20.057 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:45:20.057 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:45:20.057 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:45:20.057 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:45:20.057 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:45:20.057 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:45:20.058 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:45:20.058 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:45:20.058 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:45:20.058 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:45:20.058 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:45:20.058 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:45:20.058 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:45:20.058 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:45:20.058 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:45:20.058 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:45:20.058 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:45:20.058 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:45:20.058 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:45:20.058 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:45:20.059 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:45:20.059 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:45:20.059 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:45:20.059 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:45:20.059 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:45:20.059 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:45:20.059 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:45:20.059 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:45:20.059 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:45:20.059 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:45:20.059 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:45:20.059 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:45:20.059 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:45:20.059 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:45:20.059 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:45:20.060 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:45:20.060 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:45:20.060 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:45:20.060 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:45:20.060 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:45:20.060 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:45:20.060 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:45:20.060 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:45:20.060 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:45:20.060 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:45:20.060 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:45:20.060 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:45:20.060 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:45:20.060 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:45:20.061 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:45:20.061 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:45:20.061 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:45:20.061 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:45:20.061 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:45:20.061 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:45:20.061 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:45:20.061 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:45:20.061 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:45:20.061 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:45:20.061 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:45:20.061 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:45:20.061 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:45:20.061 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:45:20.062 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:45:20.062 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:45:20.062 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:45:20.062 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:45:20.062 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:45:20.062 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:45:20.062 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:45:20.062 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:45:20.062 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:45:20.062 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:45:20.062 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:45:20.062 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:45:20.062 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:45:20.062 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:45:20.063 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T03:45:20.063 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:45:20.063 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:45:20.063 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:45:20.063 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 4 2024-09-16T03:45:20.063 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:45:20.063 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:45:20.063 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:45:20.063 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:45:20.063 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:45:20.064 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:45:20.064 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 8 2024-09-16T03:45:20.064 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:45:20.064 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:45:20.064 INFO:teuthology.orchestra.run.smithi042.stdout: "mgr": [ 2024-09-16T03:45:20.064 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:45:20.064 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:45:20.064 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:45:20.064 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:45:20.064 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:45:20.064 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:45:20.064 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:45:20.064 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:45:20.064 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:45:20.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.029+0000 7f181fb1d640 10 librados: watch_flush enter 2024-09-16T03:45:20.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.029+0000 7f181fb1d640 10 librados: watch_flush exit 2024-09-16T03:45:20.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.029+0000 7f181fb1d640 20 client.13307.objecter shutdown clearing up homeless session... 2024-09-16T03:45:20.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.029+0000 7f181fb1d640 10 client.13307.objecter successfully canceled tick 2024-09-16T03:45:20.067 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.029+0000 7f181fb1d640 1 -- 172.21.15.42:0/4224753831 >> v2:172.21.15.154:6812/254845342 conn(0x7f17fc037840 msgr2=0x7f17fc039d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:45:20.067 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.029+0000 7f181fb1d640 1 --2- 172.21.15.42:0/4224753831 >> v2:172.21.15.154:6812/254845342 conn(0x7f17fc037840 0x7f17fc039d00 secure :-1 s=READY pgs=3356 cs=0 l=1 rev1=1 crypto rx=0x7f18100074a0 tx=0x7f181000b040 comp rx=0 tx=0).stop 2024-09-16T03:45:20.067 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.029+0000 7f181fb1d640 10 monclient: shutdown 2024-09-16T03:45:20.067 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.029+0000 7f181fb1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:45:20.067 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.029+0000 7f181fb1d640 1 -- 172.21.15.42:0/4224753831 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f18180a3d90 msgr2=0x7f18181361d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:45:20.067 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.029+0000 7f181fb1d640 1 --2- 172.21.15.42:0/4224753831 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f18180a3d90 0x7f18181361d0 secure :-1 s=READY pgs=2352 cs=0 l=1 rev1=1 crypto rx=0x7f18080ae040 tx=0x7f1808019180 comp rx=0 tx=0).stop 2024-09-16T03:45:20.067 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.029+0000 7f181e0ba640 1 -- 172.21.15.42:0/4224753831 reap_dead start 2024-09-16T03:45:20.067 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.029+0000 7f181e0ba640 1 -- 172.21.15.42:0/4224753831 reap_dead start 2024-09-16T03:45:20.067 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.029+0000 7f181fb1d640 1 -- 172.21.15.42:0/4224753831 shutdown_connections 2024-09-16T03:45:20.067 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.029+0000 7f181fb1d640 1 -- 172.21.15.42:0/4224753831 >> 172.21.15.42:0/4224753831 conn(0x7f18180a9fb0 msgr2=0x7f18180a8a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:45:20.067 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.029+0000 7f181fb1d640 1 -- 172.21.15.42:0/4224753831 shutdown_connections 2024-09-16T03:45:20.068 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.029+0000 7f181fb1d640 1 -- 172.21.15.42:0/4224753831 wait complete. 2024-09-16T03:45:20.068 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.029+0000 7f181fb1d640 1 librados: shutdown 2024-09-16T03:45:20.068 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:45:20.068 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T03:45:20.068 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T03:45:20.188 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.181+0000 7f776111f640 10 monclient: get_monmap_and_config 2024-09-16T03:45:20.189 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.181+0000 7f776111f640 10 monclient: build_initial_monmap 2024-09-16T03:45:20.189 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.181+0000 7f776111f640 10 monclient: monmap: 2024-09-16T03:45:20.189 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:45:20.189 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:45:20.189 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:45:20.188030+0000 2024-09-16T03:45:20.189 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:45:20.188030+0000 2024-09-16T03:45:20.189 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:45:20.189 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:45:20.189 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:45:20.189 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:45:20.189 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:45:20.189 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:45:20.189 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:45:20.190 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:45:20.191 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.185+0000 7f776111f640 1 Processor -- start 2024-09-16T03:45:20.191 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.185+0000 7f776111f640 1 -- start start 2024-09-16T03:45:20.191 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.185+0000 7f776111f640 10 monclient: init 2024-09-16T03:45:20.192 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.185+0000 7f776111f640 10 monclient: _reopen_session rank -1 2024-09-16T03:45:20.192 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.185+0000 7f776111f640 10 monclient: _add_conns ranks=[3,2,1,0,4] 2024-09-16T03:45:20.192 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.185+0000 7f776111f640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f775c0a1330 0x7f775c0a1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:45:20.192 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.185+0000 7f776111f640 10 monclient(hunting): picked mon.noname-a con 0x7f775c0a1330 addr v2:172.21.15.42:3300/0 2024-09-16T03:45:20.192 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.185+0000 7f776111f640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f775c06dac0 0x7f775c0a1c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:45:20.192 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.185+0000 7f776111f640 10 monclient(hunting): picked mon.noname-b con 0x7f775c06dac0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:45:20.192 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.185+0000 7f776111f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f775c135100 addr v1:172.21.15.154:6789/0 2024-09-16T03:45:20.192 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.185+0000 7f776111f640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f775c09c7d0 con 0x7f775c135100 2024-09-16T03:45:20.192 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.185+0000 7f776111f640 10 monclient(hunting): start opening mon connection 2024-09-16T03:45:20.192 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.185+0000 7f776111f640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f775c09c090 con 0x7f775c0a1330 2024-09-16T03:45:20.192 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.185+0000 7f776111f640 10 monclient(hunting): start opening mon connection 2024-09-16T03:45:20.192 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.185+0000 7f776111f640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f775c09f610 con 0x7f775c06dac0 2024-09-16T03:45:20.192 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.185+0000 7f776111f640 10 monclient(hunting): _renew_subs 2024-09-16T03:45:20.193 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.185+0000 7f776111f640 10 monclient(hunting): authenticate will time out at 1167.157512s 2024-09-16T03:45:20.193 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.185+0000 7f775ad9d640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f775c0a1330 0x7f775c0a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:45:20.193 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.185+0000 7f775a59c640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f775c06dac0 0x7f775c0a1c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:45:20.193 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.185+0000 7f775ad9d640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f775c0a1330 0x7f775c0a1710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:33376/0 (socket says 172.21.15.42:33376) 2024-09-16T03:45:20.193 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.185+0000 7f775ad9d640 1 -- 172.21.15.42:0/4175972916 learned_addr learned my addr 172.21.15.42:0/4175972916 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:45:20.193 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.185+0000 7f775ad9d640 10 monclient(hunting): get_auth_request con 0x7f775c0a1330 auth_method 0 2024-09-16T03:45:20.193 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.185+0000 7f775ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:45:20.193 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.185+0000 7f775ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:45:20.193 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.185+0000 7f775ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:45:20.193 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.185+0000 7f775a59c640 10 monclient(hunting): get_auth_request con 0x7f775c06dac0 auth_method 0 2024-09-16T03:45:20.193 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.185+0000 7f775a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:45:20.193 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.185+0000 7f775a59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:45:20.193 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.185+0000 7f775a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:45:20.193 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.185+0000 7f775ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:45:20.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.185+0000 7f775ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:45:20.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.185+0000 7f775ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:45:20.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.185+0000 7f775a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:45:20.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.185+0000 7f775a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:45:20.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.185+0000 7f775a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:45:20.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.185+0000 7f775ad9d640 10 monclient(hunting): handle_auth_done global_id 744138 payload 274 2024-09-16T03:45:20.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.185+0000 7f775ad9d640 1 -- 172.21.15.42:0/4175972916 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f775c06dac0 msgr2=0x7f775c0a1c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:45:20.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.185+0000 7f775ad9d640 1 --2- 172.21.15.42:0/4175972916 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f775c06dac0 0x7f775c0a1c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:45:20.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.185+0000 7f775ad9d640 1 -- 172.21.15.42:0/4175972916 >> v1:172.21.15.154:6789/0 conn(0x7f775c135100 legacy=0x7f775c135550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:45:20.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.185+0000 7f775ad9d640 10 monclient: _finish_hunting 0 2024-09-16T03:45:20.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.185+0000 7f775ad9d640 1 monclient: found mon.noname-a 2024-09-16T03:45:20.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.185+0000 7f775ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:45:20.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.185+0000 7f775ad9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:45:20.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.185+0000 7f775ad9d640 1 -- 172.21.15.42:0/4175972916 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f775c09f8f0 con 0x7f775c0a1330 2024-09-16T03:45:20.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.185+0000 7f775a59c640 1 --2- 172.21.15.42:0/4175972916 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f775c06dac0 0x7f775c0a1c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:45:20.201 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.189+0000 7f775ad9d640 1 --2- 172.21.15.42:0/4175972916 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f775c0a1330 0x7f775c0a1710 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f774c009870 tx=0x7f774c01c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8cf5e8d09564684 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:45:20.201 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.189+0000 7f7759d9b640 1 -- 172.21.15.42:0/4175972916 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f774c0b1070 con 0x7f775c0a1330 2024-09-16T03:45:20.201 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.189+0000 7f7759d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:45:20.201 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.189+0000 7f7759d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:45:20.201 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.189+0000 7f7759d9b640 10 monclient: dump: 2024-09-16T03:45:20.202 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:45:20.202 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:45:20.202 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:20.202 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:20.202 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:45:20.202 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:45:20.202 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:45:20.202 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:45:20.202 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:45:20.202 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:45:20.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.189+0000 7f7759d9b640 10 monclient: _finish_auth 0 2024-09-16T03:45:20.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.189+0000 7f7759d9b640 10 monclient: _check_auth_tickets 2024-09-16T03:45:20.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.189+0000 7f7759d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:45:20.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.189+0000 7f776111f640 5 monclient: authenticate success, global_id 744138 2024-09-16T03:45:20.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.189+0000 7f7759d9b640 1 -- 172.21.15.42:0/4175972916 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f774c0ac630 con 0x7f775c0a1330 2024-09-16T03:45:20.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.189+0000 7f776111f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:45:20.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.189+0000 7f7759d9b640 10 monclient: handle_config config(0 keys) 2024-09-16T03:45:20.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.189+0000 7f776111f640 10 monclient: get_monmap_and_config success 2024-09-16T03:45:20.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.189+0000 7f776111f640 4 set_mon_vals no callback set 2024-09-16T03:45:20.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.189+0000 7f7759d9b640 1 -- 172.21.15.42:0/4175972916 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f774c0a8050 con 0x7f775c0a1330 2024-09-16T03:45:20.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.189+0000 7f7759d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:45:20.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.189+0000 7f7759d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:45:20.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.189+0000 7f7759d9b640 10 monclient: dump: 2024-09-16T03:45:20.203 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:45:20.203 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:45:20.203 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:20.203 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:20.203 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:45:20.204 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:45:20.204 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:45:20.204 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:45:20.204 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:45:20.204 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:45:20.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.189+0000 7f776111f640 10 monclient: shutdown 2024-09-16T03:45:20.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.189+0000 7f776111f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:45:20.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.189+0000 7f776111f640 1 -- 172.21.15.42:0/4175972916 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f775c0a1330 msgr2=0x7f775c0a1710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:45:20.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.189+0000 7f776111f640 1 --2- 172.21.15.42:0/4175972916 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f775c0a1330 0x7f775c0a1710 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f774c009870 tx=0x7f774c01c500 comp rx=0 tx=0).stop 2024-09-16T03:45:20.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.189+0000 7f776111f640 1 -- 172.21.15.42:0/4175972916 shutdown_connections 2024-09-16T03:45:20.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.189+0000 7f776111f640 1 --2- 172.21.15.42:0/4175972916 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f775c06dac0 0x7f775c0a1c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:45:20.205 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.189+0000 7f776111f640 1 --2- 172.21.15.42:0/4175972916 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f775c0a1330 0x7f775c0a1710 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:45:20.205 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.189+0000 7f776111f640 1 -- 172.21.15.42:0/4175972916 >> 172.21.15.42:0/4175972916 conn(0x7f775c0a2420 msgr2=0x7f775c0a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:45:20.205 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.189+0000 7f776111f640 1 -- 172.21.15.42:0/4175972916 shutdown_connections 2024-09-16T03:45:20.205 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.189+0000 7f776111f640 1 -- 172.21.15.42:0/4175972916 wait complete. 2024-09-16T03:45:20.205 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.189+0000 7f776111f640 10 monclient: build_initial_monmap 2024-09-16T03:45:20.205 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.189+0000 7f776111f640 10 monclient: monmap: 2024-09-16T03:45:20.205 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:45:20.205 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:45:20.205 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:45:20.205 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:45:20.205 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:45:20.205 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:45:20.205 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:45:20.205 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:45:20.206 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:45:20.206 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:45:20.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.189+0000 7f776111f640 1 librados: starting msgr at 2024-09-16T03:45:20.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.189+0000 7f776111f640 1 librados: starting objecter 2024-09-16T03:45:20.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.189+0000 7f776111f640 1 Processor -- start 2024-09-16T03:45:20.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.189+0000 7f776111f640 1 -- start start 2024-09-16T03:45:20.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.189+0000 7f776111f640 1 librados: setting wanted keys 2024-09-16T03:45:20.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.189+0000 7f776111f640 1 librados: calling monclient init 2024-09-16T03:45:20.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.189+0000 7f776111f640 10 monclient: init 2024-09-16T03:45:20.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.189+0000 7f776111f640 10 monclient: _reopen_session rank -1 2024-09-16T03:45:20.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.189+0000 7f776111f640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:45:20.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.189+0000 7f776111f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f775c0a1330 0x7f775c13b470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:45:20.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.189+0000 7f776111f640 10 monclient(hunting): picked mon.noname-b con 0x7f775c0a1330 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:45:20.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.189+0000 7f776111f640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f775c135100 0x7f775c134400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:45:20.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.189+0000 7f776111f640 10 monclient(hunting): picked mon.noname-a con 0x7f775c135100 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:45:20.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.189+0000 7f776111f640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f775c134940 0x7f775c1cb3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:45:20.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.189+0000 7f776111f640 10 monclient(hunting): picked mon.noname-c con 0x7f775c134940 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:45:20.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.189+0000 7f776111f640 10 monclient(hunting): start opening mon connection 2024-09-16T03:45:20.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.189+0000 7f776111f640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f775c13dd80 con 0x7f775c135100 2024-09-16T03:45:20.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.189+0000 7f776111f640 10 monclient(hunting): start opening mon connection 2024-09-16T03:45:20.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.189+0000 7f776111f640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f775c09ef80 con 0x7f775c0a1330 2024-09-16T03:45:20.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.189+0000 7f776111f640 10 monclient(hunting): start opening mon connection 2024-09-16T03:45:20.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.189+0000 7f776111f640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f775c13df00 con 0x7f775c134940 2024-09-16T03:45:20.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.189+0000 7f776111f640 10 monclient(hunting): _renew_subs 2024-09-16T03:45:20.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.189+0000 7f776111f640 10 monclient(hunting): authenticate will time out at 1167.163313s 2024-09-16T03:45:20.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.189+0000 7f775a59c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f775c135100 0x7f775c134400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:45:20.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.189+0000 7f775b59e640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f775c134940 0x7f775c1cb3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:45:20.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.189+0000 7f775a59c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f775c135100 0x7f775c134400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:33378/0 (socket says 172.21.15.42:33378) 2024-09-16T03:45:20.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.189+0000 7f775a59c640 1 -- 172.21.15.42:0/725489980 learned_addr learned my addr 172.21.15.42:0/725489980 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:45:20.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.189+0000 7f775a59c640 10 monclient(hunting): get_auth_request con 0x7f775c135100 auth_method 0 2024-09-16T03:45:20.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.189+0000 7f775a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:45:20.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.189+0000 7f775a59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:45:20.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.189+0000 7f775a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:45:20.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.189+0000 7f775b59e640 10 monclient(hunting): get_auth_request con 0x7f775c134940 auth_method 0 2024-09-16T03:45:20.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.189+0000 7f775b59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:45:20.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.189+0000 7f775b59e640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:45:20.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.189+0000 7f775b59e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:45:20.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.189+0000 7f775b59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:45:20.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.189+0000 7f775b59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:45:20.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.189+0000 7f775b59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:45:20.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.189+0000 7f775a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:45:20.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.189+0000 7f775a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:45:20.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.189+0000 7f775a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:45:20.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.189+0000 7f775ad9d640 1 --2- 172.21.15.42:0/725489980 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f775c0a1330 0x7f775c13b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:45:20.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.189+0000 7f775ad9d640 10 monclient(hunting): get_auth_request con 0x7f775c0a1330 auth_method 0 2024-09-16T03:45:20.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.189+0000 7f775ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:45:20.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.189+0000 7f775ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:45:20.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.189+0000 7f775ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:45:20.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.189+0000 7f775b59e640 10 monclient(hunting): handle_auth_done global_id 13325 payload 824 2024-09-16T03:45:20.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.189+0000 7f775b59e640 1 -- 172.21.15.42:0/725489980 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f775c0a1330 msgr2=0x7f775c13b470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:45:20.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.189+0000 7f775b59e640 1 --2- 172.21.15.42:0/725489980 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f775c0a1330 0x7f775c13b470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:45:20.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.189+0000 7f775b59e640 1 -- 172.21.15.42:0/725489980 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f775c135100 msgr2=0x7f775c134400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:45:20.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.189+0000 7f775b59e640 1 --2- 172.21.15.42:0/725489980 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f775c135100 0x7f775c134400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:45:20.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.189+0000 7f775b59e640 10 monclient: _finish_hunting 0 2024-09-16T03:45:20.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.189+0000 7f775b59e640 1 monclient: found mon.noname-c 2024-09-16T03:45:20.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.189+0000 7f775b59e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:45:20.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.189+0000 7f775b59e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:45:20.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.189+0000 7f775b59e640 1 -- 172.21.15.42:0/725489980 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f775c13c080 con 0x7f775c134940 2024-09-16T03:45:20.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.189+0000 7f775a59c640 1 --2- 172.21.15.42:0/725489980 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f775c135100 0x7f775c134400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:45:20.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.193+0000 7f775b59e640 1 --2- 172.21.15.42:0/725489980 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f775c134940 0x7f775c1cb3b0 secure :-1 s=READY pgs=2356 cs=0 l=1 rev1=1 crypto rx=0x7f775400c780 tx=0x7f775400cc40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:45:20.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.193+0000 7f77437fe640 10 client.?.objecter ms_handle_connect 0x7f775c134940 2024-09-16T03:45:20.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.193+0000 7f77437fe640 10 client.?.objecter resend_mon_ops 2024-09-16T03:45:20.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.193+0000 7f77437fe640 1 -- 172.21.15.42:0/725489980 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7754017070 con 0x7f775c134940 2024-09-16T03:45:20.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.193+0000 7f77437fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:45:20.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.193+0000 7f77437fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:45:20.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.193+0000 7f77437fe640 10 monclient: dump: 2024-09-16T03:45:20.211 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:45:20.211 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:45:20.211 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:20.211 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:20.211 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:45:20.211 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:45:20.211 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:45:20.211 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:45:20.211 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:45:20.211 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:45:20.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.193+0000 7f77437fe640 10 monclient: _finish_auth 0 2024-09-16T03:45:20.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.193+0000 7f77437fe640 10 monclient: _check_auth_tickets 2024-09-16T03:45:20.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.193+0000 7f77437fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:45:20.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.193+0000 7f776111f640 5 monclient: authenticate success, global_id 13325 2024-09-16T03:45:20.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.193+0000 7f77437fe640 1 -- 172.21.15.42:0/725489980 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7754003db0 con 0x7f775c134940 2024-09-16T03:45:20.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.193+0000 7f77437fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:45:20.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.193+0000 7f776111f640 10 monclient: _renew_subs 2024-09-16T03:45:20.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.193+0000 7f77437fe640 1 -- 172.21.15.42:0/725489980 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77540052b0 con 0x7f775c134940 2024-09-16T03:45:20.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.193+0000 7f776111f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:45:20.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.193+0000 7f776111f640 1 -- 172.21.15.42:0/725489980 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f775c11d060 con 0x7f775c134940 2024-09-16T03:45:20.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.193+0000 7f77437fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:45:20.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.193+0000 7f77437fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:45:20.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.193+0000 7f77437fe640 10 monclient: dump: 2024-09-16T03:45:20.212 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:45:20.212 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:45:20.212 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:20.212 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:20.212 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:45:20.213 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:45:20.213 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:45:20.213 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:45:20.213 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:45:20.213 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:45:20.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.193+0000 7f7758d99640 4 set_mon_vals no callback set 2024-09-16T03:45:20.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.193+0000 7f776111f640 10 client.13325.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:45:20.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.193+0000 7f776111f640 10 monclient: _renew_subs 2024-09-16T03:45:20.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.193+0000 7f776111f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:45:20.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.193+0000 7f776111f640 1 -- 172.21.15.42:0/725489980 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f775c11e4f0 con 0x7f775c134940 2024-09-16T03:45:20.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.193+0000 7f776111f640 1 librados: init done 2024-09-16T03:45:20.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.193+0000 7f776111f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:45:20.214 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.193+0000 7f776111f640 1 --2- 172.21.15.42:0/725489980 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f775c06dac0 0x7f775c1cb8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:45:20.214 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.193+0000 7f776111f640 10 monclient(hunting): start opening mon connection 2024-09-16T03:45:20.214 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.193+0000 7f776111f640 1 -- 172.21.15.42:0/725489980 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f775c11e730 con 0x7f775c06dac0 2024-09-16T03:45:20.214 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.193+0000 7f77437fe640 1 -- 172.21.15.42:0/725489980 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f775401a070 con 0x7f775c134940 2024-09-16T03:45:20.214 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.193+0000 7f775ad9d640 1 --2- 172.21.15.42:0/725489980 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f775c06dac0 0x7f775c1cb8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:45:20.214 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.193+0000 7f775ad9d640 10 monclient: get_auth_request con 0x7f775c06dac0 auth_method 0 2024-09-16T03:45:20.214 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.193+0000 7f775ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:45:20.214 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.193+0000 7f775ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:45:20.214 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.193+0000 7f775ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:45:20.214 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.193+0000 7f77437fe640 1 --2- 172.21.15.42:0/725489980 >> v2:172.21.15.154:6812/254845342 conn(0x7f7748037810 0x7f7748039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:45:20.214 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.193+0000 7f775ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:45:20.214 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.193+0000 7f775ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:45:20.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.193+0000 7f77437fe640 1 -- 172.21.15.42:0/725489980 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(88..88 src has 1..88) ==== 7799+0+0 (secure 0 0 0) 0x7f7754048d00 con 0x7f775c134940 2024-09-16T03:45:20.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.193+0000 7f775ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:45:20.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.193+0000 7f77437fe640 10 client.13325.objecter ms_dispatch 0x7f775c000ba0 osd_map(88..88 src has 1..88) 2024-09-16T03:45:20.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.193+0000 7f77437fe640 3 client.13325.objecter handle_osd_map got epochs [88,88] > 0 2024-09-16T03:45:20.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.193+0000 7f77437fe640 3 client.13325.objecter handle_osd_map decoding full epoch 88 2024-09-16T03:45:20.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.193+0000 7f77437fe640 20 client.13325.objecter dump_active .. 0 homeless 2024-09-16T03:45:20.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.193+0000 7f775a59c640 1 --2- 172.21.15.42:0/725489980 >> v2:172.21.15.154:6812/254845342 conn(0x7f7748037810 0x7f7748039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:45:20.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.193+0000 7f775a59c640 10 monclient: get_auth_request con 0x7f7748037810 auth_method 0 2024-09-16T03:45:20.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.193+0000 7f775ad9d640 10 monclient(hunting): handle_auth_done global_id 272695 payload 824 2024-09-16T03:45:20.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.193+0000 7f775ad9d640 1 -- 172.21.15.42:0/725489980 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f775c135550 con 0x7f775c06dac0 2024-09-16T03:45:20.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.193+0000 7f775ad9d640 1 --2- 172.21.15.42:0/725489980 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f775c06dac0 0x7f775c1cb8f0 secure :-1 s=READY pgs=2843 cs=0 l=1 rev1=1 crypto rx=0x7f775c09bb00 tx=0x7f774c0192a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:45:20.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.193+0000 7f77437fe640 10 client.13325.objecter ms_handle_connect 0x7f775c06dac0 2024-09-16T03:45:20.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.193+0000 7f77437fe640 10 client.13325.objecter resend_mon_ops 2024-09-16T03:45:20.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.193+0000 7f77437fe640 1 -- 172.21.15.42:0/725489980 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f774c0bb030 con 0x7f775c06dac0 2024-09-16T03:45:20.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.193+0000 7f77437fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:45:20.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.193+0000 7f77437fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:45:20.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.193+0000 7f77437fe640 10 monclient: dump: 2024-09-16T03:45:20.216 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:45:20.216 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:45:20.216 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:20.216 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:20.216 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:45:20.216 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:45:20.216 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:45:20.216 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:45:20.216 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:45:20.216 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:45:20.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.193+0000 7f77437fe640 10 client.13325.objecter ms_handle_connect 0x7f7748037810 2024-09-16T03:45:20.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.193+0000 7f775a59c640 1 --2- 172.21.15.42:0/725489980 >> v2:172.21.15.154:6812/254845342 conn(0x7f7748037810 0x7f7748039cd0 secure :-1 s=READY pgs=3357 cs=0 l=1 rev1=1 crypto rx=0x7f775c0a53f0 tx=0x7f775000f040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:45:20.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.193+0000 7f77437fe640 1 -- 172.21.15.42:0/725489980 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f774c0b7420 con 0x7f775c06dac0 2024-09-16T03:45:20.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.193+0000 7f77437fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:45:20.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.193+0000 7f77437fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:45:20.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.193+0000 7f77437fe640 1 -- 172.21.15.42:0/725489980 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f775c06dac0 msgr2=0x7f775c1cb8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:45:20.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.193+0000 7f77437fe640 1 --2- 172.21.15.42:0/725489980 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f775c06dac0 0x7f775c1cb8f0 secure :-1 s=READY pgs=2843 cs=0 l=1 rev1=1 crypto rx=0x7f775c09bb00 tx=0x7f774c0192a0 comp rx=0 tx=0).stop 2024-09-16T03:45:20.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.193+0000 7f77437fe640 1 -- 172.21.15.42:0/725489980 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f775c06dac0 msgr2=0x7f775c1cb8f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:45:20.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.193+0000 7f77437fe640 1 --2- 172.21.15.42:0/725489980 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f775c06dac0 0x7f775c1cb8f0 secure :-1 s=CLOSED pgs=2843 cs=0 l=1 rev1=1 crypto rx=0x7f775c09bb00 tx=0x7f774c0192a0 comp rx=0 tx=0).stop 2024-09-16T03:45:20.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.201+0000 7f776111f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:45:20.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.201+0000 7f776111f640 1 --2- 172.21.15.42:0/725489980 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f775c1cbe30 0x7f775c137330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:45:20.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.201+0000 7f776111f640 10 monclient(hunting): start opening mon connection 2024-09-16T03:45:20.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.201+0000 7f776111f640 1 -- 172.21.15.42:0/725489980 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f775c1cc240 con 0x7f775c1cbe30 2024-09-16T03:45:20.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.201+0000 7f775ad9d640 1 --2- 172.21.15.42:0/725489980 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f775c1cbe30 0x7f775c137330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:45:20.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.201+0000 7f775ad9d640 10 monclient: get_auth_request con 0x7f775c1cbe30 auth_method 0 2024-09-16T03:45:20.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.201+0000 7f775ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:45:20.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.201+0000 7f775ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:45:20.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.201+0000 7f775ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:45:20.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.201+0000 7f775ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:45:20.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.201+0000 7f775ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:45:20.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.201+0000 7f775ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:45:20.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.201+0000 7f775ad9d640 10 monclient(hunting): handle_auth_done global_id 272698 payload 824 2024-09-16T03:45:20.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.201+0000 7f775ad9d640 1 -- 172.21.15.42:0/725489980 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f775c11ea10 con 0x7f775c1cbe30 2024-09-16T03:45:20.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.201+0000 7f775ad9d640 1 --2- 172.21.15.42:0/725489980 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f775c1cbe30 0x7f775c137330 secure :-1 s=READY pgs=2844 cs=0 l=1 rev1=1 crypto rx=0x7f774c009870 tx=0x7f775c0a22d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:45:20.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.201+0000 7f77437fe640 10 client.13325.objecter ms_handle_connect 0x7f775c1cbe30 2024-09-16T03:45:20.220 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.201+0000 7f77437fe640 10 client.13325.objecter resend_mon_ops 2024-09-16T03:45:20.220 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.205+0000 7f77437fe640 1 -- 172.21.15.42:0/725489980 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f774c0bb040 con 0x7f775c1cbe30 2024-09-16T03:45:20.220 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.205+0000 7f77437fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:45:20.220 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.205+0000 7f77437fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:45:20.220 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.205+0000 7f77437fe640 10 monclient: dump: 2024-09-16T03:45:20.220 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:45:20.220 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:45:20.220 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:20.220 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:20.220 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:45:20.220 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:45:20.220 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:45:20.220 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:45:20.221 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:45:20.221 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:45:20.221 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.205+0000 7f77437fe640 1 -- 172.21.15.42:0/725489980 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4711 (secure 0 0 0) 0x7f774c0c1a20 con 0x7f775c1cbe30 2024-09-16T03:45:20.221 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.205+0000 7f77437fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:45:20.221 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.205+0000 7f77437fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:45:20.221 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.205+0000 7f77437fe640 1 -- 172.21.15.42:0/725489980 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f775c1cbe30 msgr2=0x7f775c137330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:45:20.221 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.205+0000 7f77437fe640 1 --2- 172.21.15.42:0/725489980 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f775c1cbe30 0x7f775c137330 secure :-1 s=READY pgs=2844 cs=0 l=1 rev1=1 crypto rx=0x7f774c009870 tx=0x7f775c0a22d0 comp rx=0 tx=0).stop 2024-09-16T03:45:20.221 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.205+0000 7f77437fe640 1 -- 172.21.15.42:0/725489980 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f775c1cbe30 msgr2=0x7f775c137330 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:45:20.221 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.205+0000 7f77437fe640 1 --2- 172.21.15.42:0/725489980 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f775c1cbe30 0x7f775c137330 secure :-1 s=CLOSED pgs=2844 cs=0 l=1 rev1=1 crypto rx=0x7f774c009870 tx=0x7f775c0a22d0 comp rx=0 tx=0).stop 2024-09-16T03:45:20.221 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:45:20.221 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:45:20.222 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:45:20.222 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T03:45:20.222 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 28, 2024-09-16T03:45:20.222 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 62929, 2024-09-16T03:45:20.222 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:45:20.222 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:45:20.222 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T03:45:20.222 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T03:45:20.222 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:45:20.222 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 1, 2024-09-16T03:45:20.223 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:45:20.223 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:45:20.223 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:45:20.223 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:45:20.223 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:45:20.223 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:45:20.223 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:45:20.223 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:45:20.223 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:45:20.223 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:45:20.223 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:45:20.223 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:45:20.223 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:45:20.224 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:45:20.224 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:45:20.224 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:45:20.224 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:45:20.224 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:45:20.224 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:45:20.224 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:45:20.224 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:45:20.224 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:45:20.224 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:45:20.224 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:45:20.224 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:45:20.224 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:45:20.224 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:45:20.225 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:45:20.225 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:45:20.225 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:45:20.225 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:45:20.225 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:45:20.225 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:45:20.225 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:45:20.225 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:45:20.225 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:45:20.225 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:45:20.225 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:45:20.225 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:45:20.225 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:45:20.225 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:45:20.226 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:45:20.226 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:45:20.226 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:45:20.226 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:45:20.226 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:45:20.226 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:45:20.226 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:45:20.226 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:45:20.226 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:45:20.226 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:45:20.226 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:45:20.226 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:45:20.226 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:45:20.226 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:45:20.227 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:45:20.227 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:45:20.227 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:45:20.227 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:45:20.227 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:45:20.227 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:45:20.227 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:45:20.227 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:45:20.227 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:45:20.227 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:45:20.227 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:45:20.227 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:45:20.227 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:45:20.227 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:45:20.228 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:45:20.228 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:45:20.228 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:45:20.228 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:45:20.228 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:45:20.228 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:45:20.228 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:45:20.228 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:45:20.228 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:45:20.228 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:45:20.228 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:45:20.228 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:45:20.228 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:45:20.228 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:45:20.229 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:45:20.229 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:45:20.229 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:45:20.229 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:45:20.229 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:45:20.229 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:45:20.229 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:45:20.229 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:45:20.229 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:45:20.229 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:45:20.229 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:45:20.229 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:45:20.229 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:45:20.229 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:45:20.230 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:45:20.230 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:45:20.230 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:45:20.230 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:45:20.230 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:45:20.230 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:45:20.230 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:45:20.230 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:45:20.230 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:45:20.230 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:45:20.230 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:45:20.230 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:45:20.230 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:45:20.230 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:45:20.231 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:45:20.231 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:45:20.231 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:45:20.231 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:45:20.231 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:45:20.231 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:45:20.231 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:45:20.231 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:45:20.231 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:45:20.231 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:45:20.231 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:45:20.231 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:45:20.231 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:45:20.231 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:45:20.232 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:45:20.232 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:45:20.232 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:45:20.232 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:45:20.232 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:45:20.232 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:45:20.232 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:45:20.232 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:45:20.232 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:45:20.232 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:45:20.232 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:45:20.232 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:45:20.232 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:45:20.232 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:45:20.233 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:45:20.233 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:45:20.233 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T03:45:20.233 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:45:20.233 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:45:20.233 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:45:20.233 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:45:20.233 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:45:20.233 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:45:20.233 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:45:20.233 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:45:20.233 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:45:20.233 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:45:20.233 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 7 2024-09-16T03:45:20.234 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:45:20.234 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:45:20.234 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:45:20.234 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:45:20.234 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:45:20.235 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.205+0000 7f776111f640 10 librados: watch_flush enter 2024-09-16T03:45:20.235 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.205+0000 7f776111f640 10 librados: watch_flush exit 2024-09-16T03:45:20.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.205+0000 7f776111f640 20 client.13325.objecter shutdown clearing up homeless session... 2024-09-16T03:45:20.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.205+0000 7f776111f640 10 client.13325.objecter successfully canceled tick 2024-09-16T03:45:20.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.205+0000 7f776111f640 1 -- 172.21.15.42:0/725489980 >> v2:172.21.15.154:6812/254845342 conn(0x7f7748037810 msgr2=0x7f7748039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:45:20.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.205+0000 7f776111f640 1 --2- 172.21.15.42:0/725489980 >> v2:172.21.15.154:6812/254845342 conn(0x7f7748037810 0x7f7748039cd0 secure :-1 s=READY pgs=3357 cs=0 l=1 rev1=1 crypto rx=0x7f775c0a53f0 tx=0x7f775000f040 comp rx=0 tx=0).stop 2024-09-16T03:45:20.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.205+0000 7f776111f640 10 monclient: shutdown 2024-09-16T03:45:20.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.205+0000 7f776111f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:45:20.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.205+0000 7f776111f640 1 -- 172.21.15.42:0/725489980 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f775c134940 msgr2=0x7f775c1cb3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:45:20.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.205+0000 7f776111f640 1 --2- 172.21.15.42:0/725489980 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f775c134940 0x7f775c1cb3b0 secure :-1 s=READY pgs=2356 cs=0 l=1 rev1=1 crypto rx=0x7f775400c780 tx=0x7f775400cc40 comp rx=0 tx=0).stop 2024-09-16T03:45:20.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.205+0000 7f775b59e640 1 -- 172.21.15.42:0/725489980 reap_dead start 2024-09-16T03:45:20.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.205+0000 7f775b59e640 1 -- 172.21.15.42:0/725489980 reap_dead start 2024-09-16T03:45:20.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.205+0000 7f776111f640 1 -- 172.21.15.42:0/725489980 shutdown_connections 2024-09-16T03:45:20.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.205+0000 7f776111f640 1 -- 172.21.15.42:0/725489980 >> 172.21.15.42:0/725489980 conn(0x7f775c0a2420 msgr2=0x7f775c0a3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:45:20.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.205+0000 7f776111f640 1 -- 172.21.15.42:0/725489980 shutdown_connections 2024-09-16T03:45:20.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.205+0000 7f776111f640 1 -- 172.21.15.42:0/725489980 wait complete. 2024-09-16T03:45:20.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.205+0000 7f776111f640 1 librados: shutdown 2024-09-16T03:45:20.237 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:45:20.237 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T03:45:20.237 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T03:45:20.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.345+0000 7fac2da8c640 10 monclient: get_monmap_and_config 2024-09-16T03:45:20.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.345+0000 7fac2da8c640 10 monclient: build_initial_monmap 2024-09-16T03:45:20.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.345+0000 7fac2da8c640 10 monclient: monmap: 2024-09-16T03:45:20.352 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:45:20.352 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:45:20.354 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:45:20.351391+0000 2024-09-16T03:45:20.354 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:45:20.351391+0000 2024-09-16T03:45:20.354 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:45:20.354 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:45:20.354 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:45:20.354 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:45:20.354 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:45:20.354 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:45:20.355 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:45:20.355 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:45:20.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.349+0000 7fac2da8c640 1 Processor -- start 2024-09-16T03:45:20.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.349+0000 7fac2da8c640 1 -- start start 2024-09-16T03:45:20.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.349+0000 7fac2da8c640 10 monclient: init 2024-09-16T03:45:20.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.349+0000 7fac2da8c640 10 monclient: _reopen_session rank -1 2024-09-16T03:45:20.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.349+0000 7fac2da8c640 10 monclient: _add_conns ranks=[4,0,2,3,1] 2024-09-16T03:45:20.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.349+0000 7fac2da8c640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fac280a1330 0x7fac280a1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:45:20.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.349+0000 7fac2da8c640 10 monclient(hunting): picked mon.noname-c con 0x7fac280a1330 addr v2:172.21.15.154:3300/0 2024-09-16T03:45:20.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.349+0000 7fac2da8c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fac2809c920 addr v1:172.21.15.42:6789/0 2024-09-16T03:45:20.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.349+0000 7fac2da8c640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fac28072d40 0x7fac280a1c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:45:20.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.349+0000 7fac2da8c640 10 monclient(hunting): picked mon.noname-b con 0x7fac28072d40 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:45:20.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.349+0000 7fac2da8c640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fac28072350 con 0x7fac2809c920 2024-09-16T03:45:20.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.349+0000 7fac2da8c640 10 monclient(hunting): start opening mon connection 2024-09-16T03:45:20.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.349+0000 7fac2da8c640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fac28072b50 con 0x7fac280a1330 2024-09-16T03:45:20.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.349+0000 7fac2da8c640 10 monclient(hunting): start opening mon connection 2024-09-16T03:45:20.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.349+0000 7fac2da8c640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fac280a2190 con 0x7fac28072d40 2024-09-16T03:45:20.364 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.349+0000 7fac2da8c640 10 monclient(hunting): _renew_subs 2024-09-16T03:45:20.364 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.349+0000 7fac2da8c640 10 monclient(hunting): authenticate will time out at 1167.320924s 2024-09-16T03:45:20.364 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.349+0000 7fac27fff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fac28072d40 0x7fac280a1c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:45:20.364 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.349+0000 7fac26ffd640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fac2809c920 0x7fac28137120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:55270/0 (socket says 172.21.15.42:55270) 2024-09-16T03:45:20.364 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.349+0000 7fac26ffd640 1 -- 172.21.15.42:0/406437351 learned_addr learned my addr 172.21.15.42:0/406437351 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:45:20.364 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.349+0000 7fac27fff640 10 monclient(hunting): get_auth_request con 0x7fac28072d40 auth_method 0 2024-09-16T03:45:20.364 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.349+0000 7fac27fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:45:20.364 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.349+0000 7fac27fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:45:20.364 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.349+0000 7fac27fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:45:20.364 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.349+0000 7fac27fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:45:20.364 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.349+0000 7fac27fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:45:20.364 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.349+0000 7fac27fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:45:20.365 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.349+0000 7fac277fe640 1 --2- 172.21.15.42:0/406437351 >> v2:172.21.15.154:3300/0 conn(0x7fac280a1330 0x7fac280a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:45:20.365 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.349+0000 7fac277fe640 10 monclient(hunting): get_auth_request con 0x7fac280a1330 auth_method 0 2024-09-16T03:45:20.365 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.349+0000 7fac277fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:45:20.365 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.349+0000 7fac277fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:45:20.365 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.349+0000 7fac277fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:45:20.365 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.349+0000 7fac27fff640 10 monclient(hunting): handle_auth_done global_id 13328 payload 274 2024-09-16T03:45:20.365 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.349+0000 7fac27fff640 1 -- 172.21.15.42:0/406437351 >> v2:172.21.15.154:3300/0 conn(0x7fac280a1330 msgr2=0x7fac280a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:45:20.365 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.349+0000 7fac27fff640 1 --2- 172.21.15.42:0/406437351 >> v2:172.21.15.154:3300/0 conn(0x7fac280a1330 0x7fac280a1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:45:20.365 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.349+0000 7fac27fff640 1 -- 172.21.15.42:0/406437351 >> v1:172.21.15.42:6789/0 conn(0x7fac2809c920 legacy=0x7fac28137120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:45:20.365 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.349+0000 7fac27fff640 10 monclient: _finish_hunting 0 2024-09-16T03:45:20.366 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.349+0000 7fac27fff640 1 monclient: found mon.noname-b 2024-09-16T03:45:20.366 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.349+0000 7fac27fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:45:20.366 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.349+0000 7fac27fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T03:45:20.366 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.349+0000 7fac27fff640 1 -- 172.21.15.42:0/406437351 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac2809bc60 con 0x7fac28072d40 2024-09-16T03:45:20.366 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.349+0000 7fac27fff640 1 --2- 172.21.15.42:0/406437351 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fac28072d40 0x7fac280a1c50 secure :-1 s=READY pgs=2357 cs=0 l=1 rev1=1 crypto rx=0x7fac200097c0 tx=0x7fac2001c330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aafda20f374141eb server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:45:20.366 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.349+0000 7fac267fc640 1 -- 172.21.15.42:0/406437351 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac200b0070 con 0x7fac28072d40 2024-09-16T03:45:20.366 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.349+0000 7fac267fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:45:20.366 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.349+0000 7fac267fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:45:20.366 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.349+0000 7fac267fc640 10 monclient: dump: 2024-09-16T03:45:20.366 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:45:20.366 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:45:20.366 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:20.366 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:20.367 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:45:20.367 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:45:20.367 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:45:20.367 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:45:20.367 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:45:20.367 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:45:20.367 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.349+0000 7fac267fc640 10 monclient: _finish_auth 0 2024-09-16T03:45:20.367 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.349+0000 7fac267fc640 10 monclient: _check_auth_tickets 2024-09-16T03:45:20.367 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.349+0000 7fac267fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:45:20.367 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.349+0000 7fac267fc640 1 -- 172.21.15.42:0/406437351 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac200ab600 con 0x7fac28072d40 2024-09-16T03:45:20.367 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.349+0000 7fac2da8c640 5 monclient: authenticate success, global_id 13328 2024-09-16T03:45:20.367 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.349+0000 7fac2da8c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:45:20.367 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.349+0000 7fac267fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:45:20.368 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.349+0000 7fac2da8c640 10 monclient: get_monmap_and_config success 2024-09-16T03:45:20.368 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.349+0000 7fac267fc640 1 -- 172.21.15.42:0/406437351 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac200ab9d0 con 0x7fac28072d40 2024-09-16T03:45:20.368 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.349+0000 7fac267fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:45:20.368 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.349+0000 7fac2da8c640 4 set_mon_vals no callback set 2024-09-16T03:45:20.368 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.349+0000 7fac267fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:45:20.368 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.349+0000 7fac267fc640 10 monclient: dump: 2024-09-16T03:45:20.368 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:45:20.368 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:45:20.368 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:20.368 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:20.368 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:45:20.368 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:45:20.368 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:45:20.368 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:45:20.369 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:45:20.369 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:45:20.369 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.353+0000 7fac2da8c640 10 monclient: shutdown 2024-09-16T03:45:20.369 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.353+0000 7fac2da8c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:45:20.369 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.353+0000 7fac2da8c640 1 -- 172.21.15.42:0/406437351 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fac28072d40 msgr2=0x7fac280a1c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:45:20.369 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.353+0000 7fac2da8c640 1 --2- 172.21.15.42:0/406437351 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fac28072d40 0x7fac280a1c50 secure :-1 s=READY pgs=2357 cs=0 l=1 rev1=1 crypto rx=0x7fac200097c0 tx=0x7fac2001c330 comp rx=0 tx=0).stop 2024-09-16T03:45:20.369 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.353+0000 7fac2da8c640 1 -- 172.21.15.42:0/406437351 shutdown_connections 2024-09-16T03:45:20.369 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.353+0000 7fac2da8c640 1 --2- 172.21.15.42:0/406437351 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fac28072d40 0x7fac280a1c50 unknown :-1 s=CLOSED pgs=2357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:45:20.369 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.353+0000 7fac2da8c640 1 --2- 172.21.15.42:0/406437351 >> v2:172.21.15.154:3300/0 conn(0x7fac280a1330 0x7fac280a1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:45:20.369 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.353+0000 7fac2da8c640 1 -- 172.21.15.42:0/406437351 >> 172.21.15.42:0/406437351 conn(0x7fac280a2420 msgr2=0x7fac280a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:45:20.369 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.353+0000 7fac2da8c640 1 -- 172.21.15.42:0/406437351 shutdown_connections 2024-09-16T03:45:20.369 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.353+0000 7fac2da8c640 1 -- 172.21.15.42:0/406437351 wait complete. 2024-09-16T03:45:20.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.353+0000 7fac2da8c640 10 monclient: build_initial_monmap 2024-09-16T03:45:20.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.353+0000 7fac2da8c640 10 monclient: monmap: 2024-09-16T03:45:20.370 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:45:20.370 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:45:20.370 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:45:20.370 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:45:20.370 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:45:20.370 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:45:20.370 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:45:20.370 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:45:20.370 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:45:20.370 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:45:20.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.353+0000 7fac2da8c640 1 librados: starting msgr at 2024-09-16T03:45:20.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.353+0000 7fac2da8c640 1 librados: starting objecter 2024-09-16T03:45:20.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.353+0000 7fac2da8c640 1 Processor -- start 2024-09-16T03:45:20.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.353+0000 7fac2da8c640 1 -- start start 2024-09-16T03:45:20.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.353+0000 7fac2da8c640 1 librados: setting wanted keys 2024-09-16T03:45:20.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.353+0000 7fac2da8c640 1 librados: calling monclient init 2024-09-16T03:45:20.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.353+0000 7fac2da8c640 10 monclient: init 2024-09-16T03:45:20.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.353+0000 7fac2da8c640 10 monclient: _reopen_session rank -1 2024-09-16T03:45:20.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.353+0000 7fac2da8c640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:45:20.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.353+0000 7fac2da8c640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fac28072d40 0x7fac28135bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:45:20.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.353+0000 7fac2da8c640 10 monclient(hunting): picked mon.noname-c con 0x7fac28072d40 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:45:20.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.353+0000 7fac2da8c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fac2809c920 0x7fac28136130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:45:20.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.353+0000 7fac2da8c640 10 monclient(hunting): picked mon.noname-b con 0x7fac2809c920 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:45:20.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.353+0000 7fac2da8c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fac280a1330 0x7fac2811cba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:45:20.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.353+0000 7fac2da8c640 10 monclient(hunting): picked mon.noname-a con 0x7fac280a1330 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:45:20.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.353+0000 7fac2da8c640 10 monclient(hunting): start opening mon connection 2024-09-16T03:45:20.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.353+0000 7fac2da8c640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fac28141a70 con 0x7fac280a1330 2024-09-16T03:45:20.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.353+0000 7fac2da8c640 10 monclient(hunting): start opening mon connection 2024-09-16T03:45:20.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.353+0000 7fac2da8c640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fac281418f0 con 0x7fac2809c920 2024-09-16T03:45:20.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.353+0000 7fac2da8c640 10 monclient(hunting): start opening mon connection 2024-09-16T03:45:20.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.353+0000 7fac2da8c640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fac28141bf0 con 0x7fac28072d40 2024-09-16T03:45:20.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.353+0000 7fac2da8c640 10 monclient(hunting): _renew_subs 2024-09-16T03:45:20.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.353+0000 7fac2da8c640 10 monclient(hunting): authenticate will time out at 1167.326266s 2024-09-16T03:45:20.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.353+0000 7fac27fff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fac280a1330 0x7fac2811cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:45:20.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.353+0000 7fac277fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fac28072d40 0x7fac28135bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:45:20.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.353+0000 7fac277fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fac28072d40 0x7fac28135bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:38668/0 (socket says 172.21.15.42:38668) 2024-09-16T03:45:20.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.353+0000 7fac27fff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fac280a1330 0x7fac2811cba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:33384/0 (socket says 172.21.15.42:33384) 2024-09-16T03:45:20.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.353+0000 7fac27fff640 1 -- 172.21.15.42:0/2240341206 learned_addr learned my addr 172.21.15.42:0/2240341206 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:45:20.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.353+0000 7fac27fff640 10 monclient(hunting): get_auth_request con 0x7fac280a1330 auth_method 0 2024-09-16T03:45:20.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.353+0000 7fac27fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:45:20.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.353+0000 7fac27fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:45:20.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.353+0000 7fac27fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:45:20.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.353+0000 7fac277fe640 10 monclient(hunting): get_auth_request con 0x7fac28072d40 auth_method 0 2024-09-16T03:45:20.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.353+0000 7fac277fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:45:20.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.353+0000 7fac277fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:45:20.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.353+0000 7fac277fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:45:20.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.353+0000 7fac26ffd640 1 --2- 172.21.15.42:0/2240341206 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fac2809c920 0x7fac28136130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:45:20.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.353+0000 7fac27fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:45:20.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.353+0000 7fac27fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:45:20.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.353+0000 7fac27fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:45:20.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.353+0000 7fac26ffd640 10 monclient(hunting): get_auth_request con 0x7fac2809c920 auth_method 0 2024-09-16T03:45:20.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.353+0000 7fac26ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:45:20.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.353+0000 7fac26ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:45:20.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.353+0000 7fac26ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:45:20.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.353+0000 7fac277fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:45:20.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.353+0000 7fac277fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:45:20.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.353+0000 7fac277fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:45:20.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.353+0000 7fac27fff640 10 monclient(hunting): handle_auth_done global_id 744147 payload 824 2024-09-16T03:45:20.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.353+0000 7fac27fff640 1 -- 172.21.15.42:0/2240341206 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fac28072d40 msgr2=0x7fac28135bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:45:20.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.353+0000 7fac27fff640 1 --2- 172.21.15.42:0/2240341206 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fac28072d40 0x7fac28135bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:45:20.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.353+0000 7fac27fff640 1 -- 172.21.15.42:0/2240341206 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fac2809c920 msgr2=0x7fac28136130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:45:20.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.353+0000 7fac27fff640 1 --2- 172.21.15.42:0/2240341206 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fac2809c920 0x7fac28136130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:45:20.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.353+0000 7fac27fff640 10 monclient: _finish_hunting 0 2024-09-16T03:45:20.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.353+0000 7fac27fff640 1 monclient: found mon.noname-a 2024-09-16T03:45:20.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.353+0000 7fac27fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:45:20.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.353+0000 7fac27fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:45:20.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.353+0000 7fac27fff640 1 -- 172.21.15.42:0/2240341206 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac281204d0 con 0x7fac280a1330 2024-09-16T03:45:20.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.353+0000 7fac26ffd640 1 --2- 172.21.15.42:0/2240341206 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fac2809c920 0x7fac28136130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T03:45:20.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.353+0000 7fac27fff640 1 --2- 172.21.15.42:0/2240341206 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fac280a1330 0x7fac2811cba0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fac20019f60 tx=0x7fac200190d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:45:20.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.353+0000 7fac0ffff640 10 client.?.objecter ms_handle_connect 0x7fac280a1330 2024-09-16T03:45:20.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.353+0000 7fac0ffff640 10 client.?.objecter resend_mon_ops 2024-09-16T03:45:20.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.353+0000 7fac0ffff640 1 -- 172.21.15.42:0/2240341206 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac200b0040 con 0x7fac280a1330 2024-09-16T03:45:20.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.353+0000 7fac0ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:45:20.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.353+0000 7fac0ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:45:20.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.353+0000 7fac0ffff640 10 monclient: dump: 2024-09-16T03:45:20.375 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:45:20.375 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:45:20.375 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:20.375 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:20.376 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:45:20.376 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:45:20.376 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:45:20.376 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:45:20.376 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:45:20.376 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:45:20.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.353+0000 7fac0ffff640 10 monclient: _finish_auth 0 2024-09-16T03:45:20.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.353+0000 7fac0ffff640 10 monclient: _check_auth_tickets 2024-09-16T03:45:20.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.353+0000 7fac0ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:45:20.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.353+0000 7fac2da8c640 5 monclient: authenticate success, global_id 744147 2024-09-16T03:45:20.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.353+0000 7fac0ffff640 1 -- 172.21.15.42:0/2240341206 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac200b9070 con 0x7fac280a1330 2024-09-16T03:45:20.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.353+0000 7fac2da8c640 10 monclient: _renew_subs 2024-09-16T03:45:20.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.353+0000 7fac2da8c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:45:20.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.353+0000 7fac2da8c640 1 -- 172.21.15.42:0/2240341206 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac2811f300 con 0x7fac280a1330 2024-09-16T03:45:20.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.353+0000 7fac0ffff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:45:20.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.353+0000 7fac0ffff640 1 -- 172.21.15.42:0/2240341206 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac200b4650 con 0x7fac280a1330 2024-09-16T03:45:20.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.353+0000 7fac0ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:45:20.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.353+0000 7fac0ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:45:20.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.353+0000 7fac0ffff640 10 monclient: dump: 2024-09-16T03:45:20.377 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:45:20.377 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:45:20.377 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:20.377 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:20.377 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:45:20.377 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:45:20.377 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:45:20.378 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:45:20.378 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:45:20.378 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:45:20.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.353+0000 7fac2da8c640 10 client.744147.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:45:20.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.353+0000 7fac2da8c640 10 monclient: _renew_subs 2024-09-16T03:45:20.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.353+0000 7fac2da8c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:45:20.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.353+0000 7fac2da8c640 1 -- 172.21.15.42:0/2240341206 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac2809e610 con 0x7fac280a1330 2024-09-16T03:45:20.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.353+0000 7fac25ffb640 4 set_mon_vals no callback set 2024-09-16T03:45:20.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.353+0000 7fac2da8c640 1 librados: init done 2024-09-16T03:45:20.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.357+0000 7fac2da8c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:45:20.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.357+0000 7fac2da8c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:45:20.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.357+0000 7fac2da8c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:45:20.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.357+0000 7fac2da8c640 1 -- 172.21.15.42:0/2240341206 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fac28137120 con 0x7fac280a1330 2024-09-16T03:45:20.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.357+0000 7fac0ffff640 1 -- 172.21.15.42:0/2240341206 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fac200b47f0 con 0x7fac280a1330 2024-09-16T03:45:20.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.357+0000 7fac0ffff640 1 --2- 172.21.15.42:0/2240341206 >> v2:172.21.15.154:6812/254845342 conn(0x7fac14037810 0x7fac14039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:45:20.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.357+0000 7fac0ffff640 1 -- 172.21.15.42:0/2240341206 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 7799+0+0 (secure 0 0 0) 0x7fac200e3530 con 0x7fac280a1330 2024-09-16T03:45:20.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.357+0000 7fac0ffff640 10 client.744147.objecter ms_dispatch 0x7fac28000ba0 osd_map(88..88 src has 1..88) 2024-09-16T03:45:20.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.357+0000 7fac0ffff640 3 client.744147.objecter handle_osd_map got epochs [88,88] > 0 2024-09-16T03:45:20.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.357+0000 7fac0ffff640 3 client.744147.objecter handle_osd_map decoding full epoch 88 2024-09-16T03:45:20.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.357+0000 7fac0ffff640 20 client.744147.objecter dump_active .. 0 homeless 2024-09-16T03:45:20.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.357+0000 7fac277fe640 1 --2- 172.21.15.42:0/2240341206 >> v2:172.21.15.154:6812/254845342 conn(0x7fac14037810 0x7fac14039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:45:20.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.357+0000 7fac277fe640 10 monclient: get_auth_request con 0x7fac14037810 auth_method 0 2024-09-16T03:45:20.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.357+0000 7fac277fe640 1 --2- 172.21.15.42:0/2240341206 >> v2:172.21.15.154:6812/254845342 conn(0x7fac14037810 0x7fac14039cd0 secure :-1 s=READY pgs=3358 cs=0 l=1 rev1=1 crypto rx=0x7fac18007a60 tx=0x7fac18008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:45:20.380 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.357+0000 7fac0ffff640 10 client.744147.objecter ms_handle_connect 0x7fac14037810 2024-09-16T03:45:20.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.361+0000 7fac0ffff640 1 -- 172.21.15.42:0/2240341206 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7fac20016070 con 0x7fac280a1330 2024-09-16T03:45:20.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.361+0000 7fac0ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:45:20.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.361+0000 7fac0ffff640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:45:20.602 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.597+0000 7fac2da8c640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T03:45:20.602 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.597+0000 7fac2da8c640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T03:45:20.602 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.597+0000 7fac2da8c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:45:20.603 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.597+0000 7fac2da8c640 1 -- 172.21.15.42:0/2240341206 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7fac2811f080 con 0x7fac280a1330 2024-09-16T03:45:20.604 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.597+0000 7fac0ffff640 1 -- 172.21.15.42:0/2240341206 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7fac200be7e0 con 0x7fac280a1330 2024-09-16T03:45:20.604 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.597+0000 7fac0ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T03:45:20.604 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.597+0000 7fac0ffff640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:45:20.605 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:45:20.601+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:45:20.605 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:45:20.601+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=26,config=2,health=12,kv=10,logm=50} crc {auth=4152348210,config=692619927,health=507275256,kv=3446188498,logm=2661287144}) 2024-09-16T03:45:20.605 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:45:20.601+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=26,config=2,health=11,kv=9,logm=52} crc {auth=1628693116,config=692619928,health=1213107286,kv=3915008329,logm=2342918078}) 2024-09-16T03:45:20.605 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:45:20.601+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:45:20.605 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:45:20.601+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=26,config=2,health=12,kv=10,logm=50} crc {auth=4152348210,config=692619927,health=507275256,kv=3446188498,logm=2661287144}) 2024-09-16T03:45:20.605 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:45:20.601+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=27,config=2,health=11,kv=10,logm=50} crc {auth=2457519432,config=692619927,health=479172043,kv=3446188498,logm=3483445214}) 2024-09-16T03:45:20.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.601+0000 7fac2da8c640 10 librados: watch_flush enter 2024-09-16T03:45:20.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.601+0000 7fac2da8c640 10 librados: watch_flush exit 2024-09-16T03:45:20.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.601+0000 7fac2da8c640 20 client.744147.objecter shutdown clearing up homeless session... 2024-09-16T03:45:20.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.601+0000 7fac2da8c640 10 client.744147.objecter successfully canceled tick 2024-09-16T03:45:20.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.601+0000 7fac2da8c640 1 -- 172.21.15.42:0/2240341206 >> v2:172.21.15.154:6812/254845342 conn(0x7fac14037810 msgr2=0x7fac14039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:45:20.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.601+0000 7fac2da8c640 1 --2- 172.21.15.42:0/2240341206 >> v2:172.21.15.154:6812/254845342 conn(0x7fac14037810 0x7fac14039cd0 secure :-1 s=READY pgs=3358 cs=0 l=1 rev1=1 crypto rx=0x7fac18007a60 tx=0x7fac18008040 comp rx=0 tx=0).stop 2024-09-16T03:45:20.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.601+0000 7fac2da8c640 10 monclient: shutdown 2024-09-16T03:45:20.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.601+0000 7fac2da8c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:45:20.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.601+0000 7fac2da8c640 1 -- 172.21.15.42:0/2240341206 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fac280a1330 msgr2=0x7fac2811cba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:45:20.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.601+0000 7fac2da8c640 1 --2- 172.21.15.42:0/2240341206 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fac280a1330 0x7fac2811cba0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fac20019f60 tx=0x7fac200190d0 comp rx=0 tx=0).stop 2024-09-16T03:45:20.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.601+0000 7fac2da8c640 1 -- 172.21.15.42:0/2240341206 shutdown_connections 2024-09-16T03:45:20.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.601+0000 7fac2da8c640 1 --2- 172.21.15.42:0/2240341206 >> v2:172.21.15.154:6812/254845342 conn(0x7fac14037810 0x7fac14039cd0 unknown :-1 s=CLOSED pgs=3358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:45:20.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.601+0000 7fac2da8c640 1 --2- 172.21.15.42:0/2240341206 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fac280a1330 0x7fac2811cba0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:45:20.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.601+0000 7fac2da8c640 1 --2- 172.21.15.42:0/2240341206 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fac2809c920 0x7fac28136130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:45:20.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.601+0000 7fac2da8c640 1 --2- 172.21.15.42:0/2240341206 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fac28072d40 0x7fac28135bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:45:20.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.601+0000 7fac2da8c640 1 -- 172.21.15.42:0/2240341206 >> 172.21.15.42:0/2240341206 conn(0x7fac280a2420 msgr2=0x7fac280a3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:45:20.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.601+0000 7fac2da8c640 1 -- 172.21.15.42:0/2240341206 shutdown_connections 2024-09-16T03:45:20.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.601+0000 7fac2da8c640 1 -- 172.21.15.42:0/2240341206 wait complete. 2024-09-16T03:45:20.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:20.601+0000 7fac2da8c640 1 librados: shutdown 2024-09-16T03:45:20.609 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:45:20.601+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:45:20.609 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:45:20.601+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2129358068}) 2024-09-16T03:45:20.609 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:45:20.601+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2866515628}) 2024-09-16T03:45:20.609 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:45:20.601+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:45:20.610 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:45:20.601+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2129358068}) 2024-09-16T03:45:20.610 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:45:20.601+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2434999130}) 2024-09-16T03:45:20.611 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:45:20.605+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:45:20.611 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:45:20.605+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1139046159}) 2024-09-16T03:45:20.611 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:45:20.605+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1486109203}) 2024-09-16T03:45:20.611 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:45:20.605+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:45:20.611 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:45:20.605+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1139046159}) 2024-09-16T03:45:20.611 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:45:20.605+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1441479567}) 2024-09-16T03:45:20.613 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:45:20.605+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:45:20.613 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:45:20.609+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2036554771}) 2024-09-16T03:45:20.613 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:45:20.609+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1647412098}) 2024-09-16T03:45:20.613 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:45:20.609+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:45:20.613 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:45:20.609+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2036554771}) 2024-09-16T03:45:20.613 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:45:20.609+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4283455770}) 2024-09-16T03:45:20.615 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:45:20.609+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:45:20.615 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:45:20.609+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2166107751}) 2024-09-16T03:45:20.615 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:45:20.609+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=585969022}) 2024-09-16T03:45:20.615 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:45:20.609+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:45:20.615 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:45:20.609+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2166107751}) 2024-09-16T03:45:20.615 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:45:20.609+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3536595851}) 2024-09-16T03:45:20.617 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:45:20.613+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:45:20.617 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:45:20.613+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1967725480}) 2024-09-16T03:45:20.617 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:45:20.613+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3775460998}) 2024-09-16T03:45:20.617 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:45:20.613+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:45:20.617 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:45:20.613+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1967725480}) 2024-09-16T03:45:20.618 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:45:20.613+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4094190944}) 2024-09-16T03:45:20.619 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:45:20.613+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:45:20.619 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:45:20.613+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1450883097}) 2024-09-16T03:45:20.619 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:45:20.613+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3510173940}) 2024-09-16T03:45:20.619 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:45:20.613+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:45:20.620 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:45:20.613+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1450883097}) 2024-09-16T03:45:20.620 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:45:20.613+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1705481215}) 2024-09-16T03:45:20.621 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:45:20.617+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:45:20.621 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:45:20.617+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=362404259}) 2024-09-16T03:45:20.621 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:45:20.617+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3682568000}) 2024-09-16T03:45:20.622 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:45:20.617+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:45:20.622 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:45:20.617+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=362404259}) 2024-09-16T03:45:20.622 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:45:20.617+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2858389584}) 2024-09-16T03:45:20.624 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:45:20.617+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:45:20.624 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:45:20.617+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3153604607}) 2024-09-16T03:45:20.624 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:45:20.617+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1723630348}) 2024-09-16T03:45:20.624 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:45:20.617+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:45:20.624 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:45:20.617+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3153604607}) 2024-09-16T03:45:20.624 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:45:20.617+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3287522532}) 2024-09-16T03:45:20.625 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T03:45:20.626 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:45:20.621+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:45:20.626 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:45:20.621+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=381187988}) 2024-09-16T03:45:20.626 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:45:20.621+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2032557645}) 2024-09-16T03:45:20.626 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:45:20.621+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:45:20.626 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:45:20.621+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=381187988}) 2024-09-16T03:45:20.626 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:45:20.621+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3790610962}) 2024-09-16T03:45:20.632 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:45:20.621+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:45:20.632 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:45:20.621+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2613186444}) 2024-09-16T03:45:20.632 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:45:20.621+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1439216028}) 2024-09-16T03:45:20.632 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:45:20.621+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:45:20.632 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:45:20.621+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2613186444}) 2024-09-16T03:45:20.632 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:45:20.621+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=428766347}) 2024-09-16T03:45:20.632 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:45:20.625+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:45:20.632 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:45:20.625+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3275464917}) 2024-09-16T03:45:20.632 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:45:20.625+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2759854891}) 2024-09-16T03:45:20.632 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:45:20.625+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:45:20.633 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:45:20.625+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3275464917}) 2024-09-16T03:45:20.633 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:45:20.625+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2584874859}) 2024-09-16T03:45:20.633 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:45:20.625+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:45:20.633 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:45:20.625+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3060552950}) 2024-09-16T03:45:20.633 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:45:20.625+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4113879202}) 2024-09-16T03:45:20.633 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:45:20.625+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:45:20.633 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:45:20.625+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3060552950}) 2024-09-16T03:45:20.633 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:45:20.625+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2648472261}) 2024-09-16T03:45:20.634 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:45:20.629+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:45:20.634 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:45:20.629+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1449321835}) 2024-09-16T03:45:20.634 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:45:20.629+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3497970083}) 2024-09-16T03:45:20.634 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:45:20.629+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:45:20.634 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:45:20.629+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1449321835}) 2024-09-16T03:45:20.634 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:45:20.629+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=767065606}) 2024-09-16T03:45:20.636 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:45:20.629+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:45:20.636 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:45:20.629+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3616084404}) 2024-09-16T03:45:20.636 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:45:20.629+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1589947254}) 2024-09-16T03:45:20.636 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:45:20.629+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:45:20.636 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:45:20.629+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3616084404}) 2024-09-16T03:45:20.636 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:45:20.629+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2212229957}) 2024-09-16T03:45:20.638 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:45:20.633+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:45:20.638 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:45:20.633+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1143200585}) 2024-09-16T03:45:20.638 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:45:20.633+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3241567713}) 2024-09-16T03:45:20.638 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:45:20.633+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:45:20.638 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:45:20.633+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1143200585}) 2024-09-16T03:45:20.638 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:45:20.633+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3066300336}) 2024-09-16T03:45:20.640 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:45:20.633+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:45:20.640 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:45:20.633+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=35,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=19,osdmap=2} crc {logm=1797432600,mdsmap=4088524775,mgr=460109776,mgr_command_descs=2107673600,mgr_metadata=1367272202,mgrstat=3411412513,mon_config_key=85057112,monmap=3937513981,osd_metadata=4182959540,osd_pg_creating=1978664336,osd_snap=153263511,osdmap=2758291570}) 2024-09-16T03:45:20.640 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:45:20.633+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=33,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=10,monmap=2,osd_metadata=6,osd_pg_creating=1,osd_snap=19,osdmap=8} crc {logm=2540602315,mdsmap=4088524775,mgr=1669371261,mgr_command_descs=2107673600,mgr_metadata=1367272202,mgrstat=1807769407,mon_config_key=1386403448,monmap=3140514009,osd_metadata=4182959540,osd_pg_creating=1978664336,osd_snap=153263511,osdmap=615050958}) 2024-09-16T03:45:20.641 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:45:20.633+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:45:20.641 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:45:20.633+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=35,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=19,osdmap=2} crc {logm=1797432600,mdsmap=4088524775,mgr=460109776,mgr_command_descs=2107673600,mgr_metadata=1367272202,mgrstat=3411412513,mon_config_key=85057112,monmap=3937513981,osd_metadata=4182959540,osd_pg_creating=1978664336,osd_snap=153263511,osdmap=2758291570}) 2024-09-16T03:45:20.641 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:45:20.633+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=34,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=10,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=18,osdmap=6} crc {logm=3815505952,mdsmap=4088524775,mgr=460109776,mgr_command_descs=2107673600,mgr_metadata=1367272202,mgrstat=3069235641,mon_config_key=1386403448,monmap=3937513981,osd_metadata=2316228721,osd_pg_creating=1978664336,osd_snap=1399930182,osdmap=3754357431}) 2024-09-16T03:45:20.645 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:45:20.641+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:45:20.645 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:45:20.641+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=378175388}) 2024-09-16T03:45:20.645 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:45:20.641+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=562052510}) 2024-09-16T03:45:20.645 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:45:20.641+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:45:20.645 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:45:20.641+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=378175388}) 2024-09-16T03:45:20.645 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:45:20.641+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=525990625}) 2024-09-16T03:45:20.656 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:45:20.649+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:45:20.656 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:45:20.649+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=40} crc {osdmap=2889163882}) 2024-09-16T03:45:20.657 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:45:20.649+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=40} crc {osdmap=216523730}) 2024-09-16T03:45:20.657 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:45:20.649+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:45:20.657 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:45:20.649+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=40} crc {osdmap=2889163882}) 2024-09-16T03:45:20.657 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:45:20.649+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=38} crc {osdmap=4047836034}) 2024-09-16T03:45:21.625 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T03:45:21.626 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:45:21.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.753+0000 7f88a168a640 10 monclient: get_monmap_and_config 2024-09-16T03:45:21.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.753+0000 7f88a168a640 10 monclient: build_initial_monmap 2024-09-16T03:45:21.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.753+0000 7f88a168a640 10 monclient: monmap: 2024-09-16T03:45:21.758 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:45:21.758 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:45:21.759 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:45:21.757676+0000 2024-09-16T03:45:21.759 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:45:21.757676+0000 2024-09-16T03:45:21.759 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:45:21.759 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:45:21.759 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:45:21.759 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:45:21.759 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:45:21.759 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:45:21.759 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:45:21.759 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:45:21.761 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.753+0000 7f88a168a640 1 Processor -- start 2024-09-16T03:45:21.761 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.753+0000 7f88a168a640 1 -- start start 2024-09-16T03:45:21.761 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.753+0000 7f88a168a640 10 monclient: init 2024-09-16T03:45:21.761 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.753+0000 7f88a168a640 10 monclient: _reopen_session rank -1 2024-09-16T03:45:21.761 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.753+0000 7f88a168a640 10 monclient: _add_conns ranks=[2,3,1,4,0] 2024-09-16T03:45:21.761 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.753+0000 7f88a168a640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f889c0a1330 0x7f889c0a1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:45:21.761 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.753+0000 7f88a168a640 10 monclient(hunting): picked mon.noname-b con 0x7f889c0a1330 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:45:21.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.753+0000 7f88a168a640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f889c06dac0 0x7f889c0a1c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:45:21.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.753+0000 7f88a168a640 10 monclient(hunting): picked mon.noname-a con 0x7f889c06dac0 addr v2:172.21.15.42:3300/0 2024-09-16T03:45:21.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.753+0000 7f88a168a640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f889c135100 addr v1:172.21.15.154:6789/0 2024-09-16T03:45:21.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.753+0000 7f88a168a640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f889c09c7d0 con 0x7f889c135100 2024-09-16T03:45:21.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.753+0000 7f88a168a640 10 monclient(hunting): start opening mon connection 2024-09-16T03:45:21.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.753+0000 7f88a168a640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f889c09c090 con 0x7f889c06dac0 2024-09-16T03:45:21.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.753+0000 7f88a168a640 10 monclient(hunting): start opening mon connection 2024-09-16T03:45:21.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.753+0000 7f88a168a640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f889c09f610 con 0x7f889c0a1330 2024-09-16T03:45:21.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.753+0000 7f88a168a640 10 monclient(hunting): _renew_subs 2024-09-16T03:45:21.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.753+0000 7f88a168a640 10 monclient(hunting): authenticate will time out at 1168.727144s 2024-09-16T03:45:21.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.753+0000 7f889a7fc640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f889c06dac0 0x7f889c0a1c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:45:21.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.753+0000 7f889affd640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f889c0a1330 0x7f889c0a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:45:21.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.753+0000 7f889a7fc640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f889c06dac0 0x7f889c0a1c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:33410/0 (socket says 172.21.15.42:33410) 2024-09-16T03:45:21.763 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.753+0000 7f889a7fc640 1 -- 172.21.15.42:0/2452554444 learned_addr learned my addr 172.21.15.42:0/2452554444 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:45:21.763 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.753+0000 7f889a7fc640 10 monclient(hunting): get_auth_request con 0x7f889c06dac0 auth_method 0 2024-09-16T03:45:21.763 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.753+0000 7f889a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:45:21.763 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.753+0000 7f889a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:45:21.763 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.753+0000 7f889a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:45:21.763 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.753+0000 7f889affd640 10 monclient(hunting): get_auth_request con 0x7f889c0a1330 auth_method 0 2024-09-16T03:45:21.763 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.753+0000 7f889affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:45:21.763 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.753+0000 7f889affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:45:21.763 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.753+0000 7f889affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:45:21.763 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.753+0000 7f889a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:45:21.764 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.753+0000 7f889a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:45:21.764 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.753+0000 7f889a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:45:21.764 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.753+0000 7f889affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:45:21.764 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.753+0000 7f889affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:45:21.764 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.753+0000 7f889affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:45:21.764 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.753+0000 7f889a7fc640 10 monclient(hunting): handle_auth_done global_id 744162 payload 274 2024-09-16T03:45:21.764 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.753+0000 7f889a7fc640 1 -- 172.21.15.42:0/2452554444 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f889c0a1330 msgr2=0x7f889c0a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:45:21.764 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.753+0000 7f889a7fc640 1 --2- 172.21.15.42:0/2452554444 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f889c0a1330 0x7f889c0a1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:45:21.764 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.753+0000 7f889a7fc640 1 -- 172.21.15.42:0/2452554444 >> v1:172.21.15.154:6789/0 conn(0x7f889c135100 legacy=0x7f889c135550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:45:21.764 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.753+0000 7f889a7fc640 10 monclient: _finish_hunting 0 2024-09-16T03:45:21.764 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.753+0000 7f889a7fc640 1 monclient: found mon.noname-a 2024-09-16T03:45:21.764 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.753+0000 7f889a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:45:21.764 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.753+0000 7f889a7fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:45:21.764 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.753+0000 7f889a7fc640 1 -- 172.21.15.42:0/2452554444 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f889c09f8f0 con 0x7f889c06dac0 2024-09-16T03:45:21.770 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.757+0000 7f889a7fc640 1 --2- 172.21.15.42:0/2452554444 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f889c06dac0 0x7f889c0a1c50 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f888c009870 tx=0x7f888c01c540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc601d9da2f43eaf server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:45:21.770 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.757+0000 7f8899ffb640 1 -- 172.21.15.42:0/2452554444 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f888c0ab650 con 0x7f889c06dac0 2024-09-16T03:45:21.771 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.757+0000 7f8899ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:45:21.771 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.757+0000 7f8899ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:45:21.771 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.757+0000 7f8899ffb640 10 monclient: dump: 2024-09-16T03:45:21.771 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:45:21.771 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:45:21.771 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:21.771 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:21.771 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:45:21.771 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:45:21.771 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:45:21.771 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:45:21.771 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:45:21.771 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:45:21.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.757+0000 7f8899ffb640 10 monclient: _finish_auth 0 2024-09-16T03:45:21.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.757+0000 7f8899ffb640 10 monclient: _check_auth_tickets 2024-09-16T03:45:21.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.757+0000 7f8899ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:45:21.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.757+0000 7f8899ffb640 1 -- 172.21.15.42:0/2452554444 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f888c0b0070 con 0x7f889c06dac0 2024-09-16T03:45:21.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.757+0000 7f88a168a640 5 monclient: authenticate success, global_id 744162 2024-09-16T03:45:21.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.757+0000 7f88a168a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:45:21.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.757+0000 7f8899ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T03:45:21.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.757+0000 7f88a168a640 10 monclient: get_monmap_and_config success 2024-09-16T03:45:21.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.757+0000 7f8899ffb640 1 -- 172.21.15.42:0/2452554444 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f888c0aba10 con 0x7f889c06dac0 2024-09-16T03:45:21.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.757+0000 7f88a168a640 4 set_mon_vals no callback set 2024-09-16T03:45:21.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.757+0000 7f8899ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:45:21.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.757+0000 7f8899ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:45:21.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.757+0000 7f8899ffb640 10 monclient: dump: 2024-09-16T03:45:21.773 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:45:21.773 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:45:21.773 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:21.773 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:21.773 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:45:21.773 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:45:21.773 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:45:21.773 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:45:21.773 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:45:21.773 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:45:21.774 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.757+0000 7f88a168a640 10 monclient: shutdown 2024-09-16T03:45:21.774 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.757+0000 7f88a168a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:45:21.774 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.757+0000 7f88a168a640 1 -- 172.21.15.42:0/2452554444 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f889c06dac0 msgr2=0x7f889c0a1c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:45:21.774 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.757+0000 7f88a168a640 1 --2- 172.21.15.42:0/2452554444 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f889c06dac0 0x7f889c0a1c50 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f888c009870 tx=0x7f888c01c540 comp rx=0 tx=0).stop 2024-09-16T03:45:21.774 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.757+0000 7f88a168a640 1 -- 172.21.15.42:0/2452554444 shutdown_connections 2024-09-16T03:45:21.774 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.757+0000 7f88a168a640 1 --2- 172.21.15.42:0/2452554444 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f889c06dac0 0x7f889c0a1c50 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:45:21.774 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.757+0000 7f88a168a640 1 --2- 172.21.15.42:0/2452554444 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f889c0a1330 0x7f889c0a1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:45:21.774 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.757+0000 7f88a168a640 1 -- 172.21.15.42:0/2452554444 >> 172.21.15.42:0/2452554444 conn(0x7f889c0a2420 msgr2=0x7f889c0a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:45:21.774 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.757+0000 7f88a168a640 1 -- 172.21.15.42:0/2452554444 shutdown_connections 2024-09-16T03:45:21.774 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.757+0000 7f88a168a640 1 -- 172.21.15.42:0/2452554444 wait complete. 2024-09-16T03:45:21.774 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.757+0000 7f88a168a640 10 monclient: build_initial_monmap 2024-09-16T03:45:21.774 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.757+0000 7f88a168a640 10 monclient: monmap: 2024-09-16T03:45:21.774 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:45:21.775 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:45:21.775 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:45:21.775 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:45:21.775 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:45:21.775 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:45:21.775 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:45:21.775 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:45:21.775 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:45:21.775 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:45:21.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.757+0000 7f88a168a640 1 librados: starting msgr at 2024-09-16T03:45:21.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.757+0000 7f88a168a640 1 librados: starting objecter 2024-09-16T03:45:21.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.757+0000 7f88a168a640 1 Processor -- start 2024-09-16T03:45:21.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.757+0000 7f88a168a640 1 -- start start 2024-09-16T03:45:21.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.757+0000 7f88a168a640 1 librados: setting wanted keys 2024-09-16T03:45:21.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.757+0000 7f88a168a640 1 librados: calling monclient init 2024-09-16T03:45:21.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.757+0000 7f88a168a640 10 monclient: init 2024-09-16T03:45:21.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.761+0000 7f88a168a640 10 monclient: _reopen_session rank -1 2024-09-16T03:45:21.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.761+0000 7f88a168a640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T03:45:21.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.761+0000 7f88a168a640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f889c0a1330 0x7f889c11bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:45:21.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.761+0000 7f88a168a640 10 monclient(hunting): picked mon.noname-b con 0x7f889c0a1330 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:45:21.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.761+0000 7f88a168a640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f889c135100 0x7f889c11c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:45:21.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.761+0000 7f88a168a640 10 monclient(hunting): picked mon.noname-c con 0x7f889c135100 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:45:21.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.761+0000 7f88a168a640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f889c11c9f0 0x7f889c133d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:45:21.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.761+0000 7f88a168a640 10 monclient(hunting): picked mon.noname-a con 0x7f889c11c9f0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:45:21.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.761+0000 7f88a168a640 10 monclient(hunting): start opening mon connection 2024-09-16T03:45:21.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.761+0000 7f88a168a640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f889c13fd00 con 0x7f889c11c9f0 2024-09-16T03:45:21.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.761+0000 7f88a168a640 10 monclient(hunting): start opening mon connection 2024-09-16T03:45:21.777 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.761+0000 7f88a168a640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f889c13fb80 con 0x7f889c0a1330 2024-09-16T03:45:21.777 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.761+0000 7f88a168a640 10 monclient(hunting): start opening mon connection 2024-09-16T03:45:21.777 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.761+0000 7f88a168a640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f889c13fe80 con 0x7f889c135100 2024-09-16T03:45:21.777 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.761+0000 7f88a168a640 10 monclient(hunting): _renew_subs 2024-09-16T03:45:21.777 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.761+0000 7f88a168a640 10 monclient(hunting): authenticate will time out at 1168.732340s 2024-09-16T03:45:21.777 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.761+0000 7f889a7fc640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f889c135100 0x7f889c11c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:45:21.777 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.761+0000 7f889a7fc640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f889c135100 0x7f889c11c4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:38706/0 (socket says 172.21.15.42:38706) 2024-09-16T03:45:21.777 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.761+0000 7f889a7fc640 1 -- 172.21.15.42:0/2750939584 learned_addr learned my addr 172.21.15.42:0/2750939584 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:45:21.777 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.761+0000 7f889a7fc640 10 monclient(hunting): get_auth_request con 0x7f889c135100 auth_method 0 2024-09-16T03:45:21.777 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.761+0000 7f889b7fe640 1 --2- 172.21.15.42:0/2750939584 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f889c11c9f0 0x7f889c133d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:45:21.777 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.761+0000 7f889a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:45:21.777 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.761+0000 7f889a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:45:21.777 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.761+0000 7f889a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:45:21.778 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.761+0000 7f889b7fe640 10 monclient(hunting): get_auth_request con 0x7f889c11c9f0 auth_method 0 2024-09-16T03:45:21.778 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.761+0000 7f889b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:45:21.778 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.761+0000 7f889b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:45:21.778 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.761+0000 7f889b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:45:21.778 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.761+0000 7f889a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:45:21.778 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.761+0000 7f889a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:45:21.778 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.761+0000 7f889a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:45:21.778 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.761+0000 7f889b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:45:21.778 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.761+0000 7f889b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:45:21.778 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.761+0000 7f889b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:45:21.778 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.761+0000 7f889affd640 1 --2- 172.21.15.42:0/2750939584 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f889c0a1330 0x7f889c11bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:45:21.778 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.761+0000 7f889affd640 10 monclient(hunting): get_auth_request con 0x7f889c0a1330 auth_method 0 2024-09-16T03:45:21.778 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.761+0000 7f889affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:45:21.778 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.761+0000 7f889affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:45:21.779 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.761+0000 7f889affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:45:21.779 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.761+0000 7f889a7fc640 10 monclient(hunting): handle_auth_done global_id 13349 payload 824 2024-09-16T03:45:21.779 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.761+0000 7f889a7fc640 1 -- 172.21.15.42:0/2750939584 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f889c0a1330 msgr2=0x7f889c11bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:45:21.779 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.761+0000 7f889a7fc640 1 --2- 172.21.15.42:0/2750939584 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f889c0a1330 0x7f889c11bf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:45:21.779 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.761+0000 7f889a7fc640 1 -- 172.21.15.42:0/2750939584 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f889c11c9f0 msgr2=0x7f889c133d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:45:21.779 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.761+0000 7f889a7fc640 1 --2- 172.21.15.42:0/2750939584 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f889c11c9f0 0x7f889c133d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:45:21.779 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.761+0000 7f889a7fc640 10 monclient: _finish_hunting 0 2024-09-16T03:45:21.779 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.761+0000 7f889a7fc640 1 monclient: found mon.noname-c 2024-09-16T03:45:21.779 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.761+0000 7f889a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:45:21.779 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.761+0000 7f889a7fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:45:21.779 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.761+0000 7f889a7fc640 1 -- 172.21.15.42:0/2750939584 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f889c11ee10 con 0x7f889c135100 2024-09-16T03:45:21.779 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.761+0000 7f889a7fc640 1 --2- 172.21.15.42:0/2750939584 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f889c135100 0x7f889c11c4b0 secure :-1 s=READY pgs=2359 cs=0 l=1 rev1=1 crypto rx=0x7f888c01abf0 tx=0x7f888c0192e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:45:21.779 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.761+0000 7f88837fe640 10 client.?.objecter ms_handle_connect 0x7f889c135100 2024-09-16T03:45:21.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.761+0000 7f88837fe640 10 client.?.objecter resend_mon_ops 2024-09-16T03:45:21.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.761+0000 7f88837fe640 1 -- 172.21.15.42:0/2750939584 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f888c0be5f0 con 0x7f889c135100 2024-09-16T03:45:21.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.761+0000 7f88837fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:45:21.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.761+0000 7f88837fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:45:21.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.761+0000 7f88837fe640 10 monclient: dump: 2024-09-16T03:45:21.780 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:45:21.780 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:45:21.780 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:21.780 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:21.780 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:45:21.780 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:45:21.780 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:45:21.780 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:45:21.781 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:45:21.781 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:45:21.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.761+0000 7f88837fe640 10 monclient: _finish_auth 0 2024-09-16T03:45:21.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.761+0000 7f88837fe640 10 monclient: _check_auth_tickets 2024-09-16T03:45:21.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.761+0000 7f88837fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:45:21.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.761+0000 7f88837fe640 1 -- 172.21.15.42:0/2750939584 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f888c0be790 con 0x7f889c135100 2024-09-16T03:45:21.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.761+0000 7f88a168a640 5 monclient: authenticate success, global_id 13349 2024-09-16T03:45:21.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.761+0000 7f88a168a640 10 monclient: _renew_subs 2024-09-16T03:45:21.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.761+0000 7f88a168a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:45:21.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.761+0000 7f88a168a640 1 -- 172.21.15.42:0/2750939584 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f889c071bb0 con 0x7f889c135100 2024-09-16T03:45:21.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.761+0000 7f88837fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:45:21.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.761+0000 7f88837fe640 1 -- 172.21.15.42:0/2750939584 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f888c0beb80 con 0x7f889c135100 2024-09-16T03:45:21.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.761+0000 7f88837fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:45:21.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.761+0000 7f88837fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:45:21.782 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.761+0000 7f88837fe640 10 monclient: dump: 2024-09-16T03:45:21.782 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:45:21.782 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:45:21.782 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:21.782 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:21.782 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:45:21.782 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:45:21.782 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:45:21.782 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:45:21.782 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:45:21.782 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:45:21.782 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.761+0000 7f88a168a640 10 client.13349.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:45:21.782 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.761+0000 7f88a168a640 10 monclient: _renew_subs 2024-09-16T03:45:21.783 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.761+0000 7f88997fa640 4 set_mon_vals no callback set 2024-09-16T03:45:21.783 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.761+0000 7f88a168a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:45:21.783 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.761+0000 7f88a168a640 1 -- 172.21.15.42:0/2750939584 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f889c073a30 con 0x7f889c135100 2024-09-16T03:45:21.783 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.761+0000 7f88a168a640 1 librados: init done 2024-09-16T03:45:21.783 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.761+0000 7f88a168a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:45:21.783 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.761+0000 7f88a168a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:45:21.783 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.761+0000 7f88a168a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:45:21.783 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.761+0000 7f88837fe640 1 -- 172.21.15.42:0/2750939584 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f888c0b43f0 con 0x7f889c135100 2024-09-16T03:45:21.783 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.761+0000 7f88a168a640 1 -- 172.21.15.42:0/2750939584 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f889c135510 con 0x7f889c135100 2024-09-16T03:45:21.783 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.761+0000 7f88837fe640 1 --2- 172.21.15.42:0/2750939584 >> v2:172.21.15.154:6812/254845342 conn(0x7f8888037810 0x7f8888039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:45:21.783 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.761+0000 7f88837fe640 1 -- 172.21.15.42:0/2750939584 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(89..89 src has 1..89) ==== 6618+0+0 (secure 0 0 0) 0x7f888c0e4010 con 0x7f889c135100 2024-09-16T03:45:21.784 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.761+0000 7f88837fe640 10 client.13349.objecter ms_dispatch 0x7f889c000ba0 osd_map(89..89 src has 1..89) 2024-09-16T03:45:21.784 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.761+0000 7f88837fe640 3 client.13349.objecter handle_osd_map got epochs [89,89] > 0 2024-09-16T03:45:21.784 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.761+0000 7f88837fe640 3 client.13349.objecter handle_osd_map decoding full epoch 89 2024-09-16T03:45:21.784 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.761+0000 7f88837fe640 20 client.13349.objecter dump_active .. 0 homeless 2024-09-16T03:45:21.784 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.761+0000 7f889affd640 1 --2- 172.21.15.42:0/2750939584 >> v2:172.21.15.154:6812/254845342 conn(0x7f8888037810 0x7f8888039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:45:21.784 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.761+0000 7f889affd640 10 monclient: get_auth_request con 0x7f8888037810 auth_method 0 2024-09-16T03:45:21.784 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.761+0000 7f88837fe640 10 client.13349.objecter ms_handle_connect 0x7f8888037810 2024-09-16T03:45:21.784 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.761+0000 7f889affd640 1 --2- 172.21.15.42:0/2750939584 >> v2:172.21.15.154:6812/254845342 conn(0x7f8888037810 0x7f8888039cd0 secure :-1 s=READY pgs=3362 cs=0 l=1 rev1=1 crypto rx=0x7f8890009af0 tx=0x7f8890008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:45:21.785 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.765+0000 7f88837fe640 1 -- 172.21.15.42:0/2750939584 <== mon.2 v2:172.21.15.42:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f888c016130 con 0x7f889c135100 2024-09-16T03:45:21.785 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.765+0000 7f88837fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:45:21.785 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.765+0000 7f88837fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:45:21.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.993+0000 7f88a168a640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:45:21.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.993+0000 7f88a168a640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:45:21.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.993+0000 7f88a168a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:45:21.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.993+0000 7f88a168a640 1 -- 172.21.15.42:0/2750939584 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f889c11d3a0 con 0x7f889c135100 2024-09-16T03:45:21.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.993+0000 7f88837fe640 1 -- 172.21.15.42:0/2750939584 <== mon.2 v2:172.21.15.42:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7f888c0b3380 con 0x7f889c135100 2024-09-16T03:45:21.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.993+0000 7f88837fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:45:21.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.993+0000 7f88837fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:45:21.998 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":28,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:45:22.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.993+0000 7f88a168a640 10 librados: watch_flush enter 2024-09-16T03:45:22.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.993+0000 7f88a168a640 10 librados: watch_flush exit 2024-09-16T03:45:22.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.993+0000 7f88a168a640 20 client.13349.objecter shutdown clearing up homeless session... 2024-09-16T03:45:22.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.993+0000 7f88a168a640 10 client.13349.objecter successfully canceled tick 2024-09-16T03:45:22.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.993+0000 7f88a168a640 1 -- 172.21.15.42:0/2750939584 >> v2:172.21.15.154:6812/254845342 conn(0x7f8888037810 msgr2=0x7f8888039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:45:22.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.993+0000 7f88a168a640 1 --2- 172.21.15.42:0/2750939584 >> v2:172.21.15.154:6812/254845342 conn(0x7f8888037810 0x7f8888039cd0 secure :-1 s=READY pgs=3362 cs=0 l=1 rev1=1 crypto rx=0x7f8890009af0 tx=0x7f8890008040 comp rx=0 tx=0).stop 2024-09-16T03:45:22.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.993+0000 7f88a168a640 10 monclient: shutdown 2024-09-16T03:45:22.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.993+0000 7f88a168a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:45:22.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.993+0000 7f88a168a640 1 -- 172.21.15.42:0/2750939584 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f889c135100 msgr2=0x7f889c11c4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:45:22.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.993+0000 7f88a168a640 1 --2- 172.21.15.42:0/2750939584 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f889c135100 0x7f889c11c4b0 secure :-1 s=READY pgs=2359 cs=0 l=1 rev1=1 crypto rx=0x7f888c01abf0 tx=0x7f888c0192e0 comp rx=0 tx=0).stop 2024-09-16T03:45:22.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.993+0000 7f88a168a640 1 -- 172.21.15.42:0/2750939584 shutdown_connections 2024-09-16T03:45:22.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.993+0000 7f88a168a640 1 --2- 172.21.15.42:0/2750939584 >> v2:172.21.15.154:6812/254845342 conn(0x7f8888037810 0x7f8888039cd0 unknown :-1 s=CLOSED pgs=3362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:45:22.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.993+0000 7f88a168a640 1 --2- 172.21.15.42:0/2750939584 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f889c11c9f0 0x7f889c133d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:45:22.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.993+0000 7f88a168a640 1 --2- 172.21.15.42:0/2750939584 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f889c135100 0x7f889c11c4b0 unknown :-1 s=CLOSED pgs=2359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:45:22.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.993+0000 7f88a168a640 1 --2- 172.21.15.42:0/2750939584 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f889c0a1330 0x7f889c11bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:45:22.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.993+0000 7f88a168a640 1 -- 172.21.15.42:0/2750939584 >> 172.21.15.42:0/2750939584 conn(0x7f889c0a2420 msgr2=0x7f889c13c590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:45:22.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.993+0000 7f88a168a640 1 -- 172.21.15.42:0/2750939584 shutdown_connections 2024-09-16T03:45:22.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.993+0000 7f88a168a640 1 -- 172.21.15.42:0/2750939584 wait complete. 2024-09-16T03:45:22.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:21.993+0000 7f88a168a640 1 librados: shutdown 2024-09-16T03:45:22.017 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T03:45:22.017 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T03:45:22.017 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T03:45:22.017 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T03:45:22.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.141+0000 7fe26d8ab640 10 monclient: get_monmap_and_config 2024-09-16T03:45:22.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.141+0000 7fe26d8ab640 10 monclient: build_initial_monmap 2024-09-16T03:45:22.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.141+0000 7fe26d8ab640 10 monclient: monmap: 2024-09-16T03:45:22.147 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:45:22.147 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:45:22.147 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:45:22.146232+0000 2024-09-16T03:45:22.147 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:45:22.146232+0000 2024-09-16T03:45:22.147 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:45:22.147 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:45:22.147 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:45:22.147 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:45:22.147 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:45:22.148 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:45:22.148 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:45:22.148 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:45:22.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.141+0000 7fe26d8ab640 1 Processor -- start 2024-09-16T03:45:22.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.141+0000 7fe26d8ab640 1 -- start start 2024-09-16T03:45:22.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.141+0000 7fe26d8ab640 10 monclient: init 2024-09-16T03:45:22.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.141+0000 7fe26d8ab640 10 monclient: _reopen_session rank -1 2024-09-16T03:45:22.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.141+0000 7fe26d8ab640 10 monclient: _add_conns ranks=[1,0,3,4,2] 2024-09-16T03:45:22.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.145+0000 7fe26d8ab640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe2680a39b0 addr v1:172.21.15.154:6789/0 2024-09-16T03:45:22.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.145+0000 7fe26d8ab640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe2680a3d90 addr v1:172.21.15.42:6789/0 2024-09-16T03:45:22.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.145+0000 7fe26d8ab640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe26813daa0 0x7fe26813fe90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:45:22.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.145+0000 7fe26d8ab640 10 monclient(hunting): picked mon.noname-a con 0x7fe26813daa0 addr v2:172.21.15.42:3300/0 2024-09-16T03:45:22.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.145+0000 7fe26d8ab640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe268134290 con 0x7fe2680a3d90 2024-09-16T03:45:22.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.145+0000 7fe26d8ab640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe26809c140 con 0x7fe2680a39b0 2024-09-16T03:45:22.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.145+0000 7fe26d8ab640 10 monclient(hunting): start opening mon connection 2024-09-16T03:45:22.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.145+0000 7fe26d8ab640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7fe26809c420 con 0x7fe26813daa0 2024-09-16T03:45:22.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.145+0000 7fe26d8ab640 10 monclient(hunting): _renew_subs 2024-09-16T03:45:22.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.145+0000 7fe26d8ab640 10 monclient(hunting): authenticate will time out at 1169.116115s 2024-09-16T03:45:22.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.145+0000 7fe266ffd640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7fe2680a39b0 0x7fe268133b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.42:53176/0 (socket says 172.21.15.42:53176) 2024-09-16T03:45:22.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.145+0000 7fe2677fe640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe26813daa0 0x7fe26813fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:45:22.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.145+0000 7fe266ffd640 1 -- 172.21.15.42:0/3999190668 learned_addr learned my addr 172.21.15.42:0/3999190668 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:45:22.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.145+0000 7fe2677fe640 10 monclient(hunting): get_auth_request con 0x7fe26813daa0 auth_method 0 2024-09-16T03:45:22.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.145+0000 7fe2677fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:45:22.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.145+0000 7fe2677fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:45:22.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.145+0000 7fe2677fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:45:22.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.145+0000 7fe265ffb640 1 -- 172.21.15.42:0/3999190668 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3087671033 0 0) 0x7fe26809c140 con 0x7fe2680a39b0 2024-09-16T03:45:22.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.145+0000 7fe265ffb640 10 monclient(hunting): _negotiate 2024-09-16T03:45:22.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.145+0000 7fe265ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:45:22.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.145+0000 7fe265ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:45:22.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.145+0000 7fe265ffb640 10 monclient(hunting): my global_id is 272743 2024-09-16T03:45:22.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.145+0000 7fe265ffb640 1 -- 172.21.15.42:0/3999190668 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe250003530 con 0x7fe2680a39b0 2024-09-16T03:45:22.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.145+0000 7fe2677fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:45:22.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.145+0000 7fe2677fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:45:22.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.145+0000 7fe2677fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:45:22.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.145+0000 7fe265ffb640 1 -- 172.21.15.42:0/3999190668 <== mon.1 v1:172.21.15.154:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3193164223 0 0) 0x7fe250003530 con 0x7fe2680a39b0 2024-09-16T03:45:22.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.145+0000 7fe265ffb640 1 -- 172.21.15.42:0/3999190668 >> v2:172.21.15.42:3300/0 conn(0x7fe26813daa0 msgr2=0x7fe26813fe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:45:22.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.145+0000 7fe265ffb640 1 --2- 172.21.15.42:0/3999190668 >> v2:172.21.15.42:3300/0 conn(0x7fe26813daa0 0x7fe26813fe90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:45:22.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.145+0000 7fe265ffb640 1 -- 172.21.15.42:0/3999190668 >> v1:172.21.15.42:6789/0 conn(0x7fe2680a3d90 legacy=0x7fe26813c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:45:22.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.145+0000 7fe265ffb640 10 monclient: _finish_hunting 0 2024-09-16T03:45:22.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.145+0000 7fe265ffb640 1 monclient: found mon.noname-c-legacy 2024-09-16T03:45:22.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.145+0000 7fe265ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:45:22.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.145+0000 7fe265ffb640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.154:6789/0 2024-09-16T03:45:22.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.145+0000 7fe265ffb640 1 -- 172.21.15.42:0/3999190668 --> v1:172.21.15.154:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe26806dd60 con 0x7fe2680a39b0 2024-09-16T03:45:22.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.145+0000 7fe265ffb640 10 monclient: _finish_auth 0 2024-09-16T03:45:22.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.145+0000 7fe265ffb640 10 monclient: _check_auth_tickets 2024-09-16T03:45:22.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.145+0000 7fe265ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:45:22.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.145+0000 7fe265ffb640 1 -- 172.21.15.42:0/3999190668 <== mon.1 v1:172.21.15.154:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2177413734 0 0) 0x7fe258001680 con 0x7fe2680a39b0 2024-09-16T03:45:22.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.145+0000 7fe265ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:45:22.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.145+0000 7fe265ffb640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-16T03:45:22.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.145+0000 7fe265ffb640 10 monclient: dump: 2024-09-16T03:45:22.172 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:45:22.172 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:45:22.172 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:22.172 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:22.172 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:45:22.172 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:45:22.172 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:45:22.172 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:45:22.172 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:45:22.172 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:45:22.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.145+0000 7fe265ffb640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] but i'm connected to v1:172.21.15.154:6789/0, reconnecting 2024-09-16T03:45:22.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.145+0000 7fe265ffb640 10 monclient: _reopen_session rank -1 2024-09-16T03:45:22.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.145+0000 7fe265ffb640 1 -- 172.21.15.42:0/3999190668 >> v1:172.21.15.154:6789/0 conn(0x7fe2680a39b0 legacy=0x7fe268133b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:45:22.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.145+0000 7fe265ffb640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T03:45:22.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.145+0000 7fe265ffb640 1 --2- 172.21.15.42:0/3999190668 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe2500039c0 0x7fe250003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:45:22.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.145+0000 7fe265ffb640 10 monclient(hunting): picked mon.b con 0x7fe2500039c0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:45:22.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.145+0000 7fe265ffb640 1 --2- 172.21.15.42:0/3999190668 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe250004810 0x7fe250004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:45:22.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.145+0000 7fe265ffb640 10 monclient(hunting): picked mon.c con 0x7fe250004810 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:45:22.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.145+0000 7fe265ffb640 1 --2- 172.21.15.42:0/3999190668 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe2500052f0 0x7fe25000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:45:22.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.145+0000 7fe265ffb640 10 monclient(hunting): picked mon.a con 0x7fe2500052f0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:45:22.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.145+0000 7fe265ffb640 10 monclient(hunting): start opening mon connection 2024-09-16T03:45:22.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.145+0000 7fe265ffb640 1 -- 172.21.15.42:0/3999190668 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fe26809c420 con 0x7fe2500052f0 2024-09-16T03:45:22.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.145+0000 7fe265ffb640 10 monclient(hunting): start opening mon connection 2024-09-16T03:45:22.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.145+0000 7fe265ffb640 1 -- 172.21.15.42:0/3999190668 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe25000c1c0 con 0x7fe2500039c0 2024-09-16T03:45:22.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.145+0000 7fe265ffb640 10 monclient(hunting): start opening mon connection 2024-09-16T03:45:22.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.145+0000 7fe265ffb640 1 -- 172.21.15.42:0/3999190668 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fe25000c3c0 con 0x7fe250004810 2024-09-16T03:45:22.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.145+0000 7fe265ffb640 10 monclient(hunting): _renew_subs 2024-09-16T03:45:22.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.145+0000 7fe265ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:45:22.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.145+0000 7fe2667fc640 1 --2- 172.21.15.42:0/3999190668 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe250004810 0x7fe250004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:45:22.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.145+0000 7fe2667fc640 10 monclient(hunting): get_auth_request con 0x7fe250004810 auth_method 0 2024-09-16T03:45:22.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.145+0000 7fe2667fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:45:22.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.145+0000 7fe2667fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:45:22.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.145+0000 7fe2667fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:45:22.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.145+0000 7fe266ffd640 1 --2- 172.21.15.42:0/3999190668 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe2500039c0 0x7fe250003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:45:22.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.145+0000 7fe266ffd640 10 monclient(hunting): get_auth_request con 0x7fe2500039c0 auth_method 0 2024-09-16T03:45:22.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.145+0000 7fe266ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:45:22.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.145+0000 7fe266ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:45:22.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.145+0000 7fe266ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:45:22.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.145+0000 7fe2667fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:45:22.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.145+0000 7fe2667fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:45:22.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.145+0000 7fe2677fe640 1 --2- 172.21.15.42:0/3999190668 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe2500052f0 0x7fe25000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:45:22.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.145+0000 7fe2667fc640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T03:45:22.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.145+0000 7fe266ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:45:22.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.145+0000 7fe266ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:45:22.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.145+0000 7fe266ffd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T03:45:22.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.145+0000 7fe2677fe640 10 monclient(hunting): get_auth_request con 0x7fe2500052f0 auth_method 0 2024-09-16T03:45:22.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.145+0000 7fe2677fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:45:22.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.145+0000 7fe2677fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:45:22.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.145+0000 7fe2677fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:45:22.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.145+0000 7fe2667fc640 10 monclient(hunting): handle_auth_done global_id 272743 payload 293 2024-09-16T03:45:22.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.145+0000 7fe2667fc640 1 -- 172.21.15.42:0/3999190668 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe2500039c0 msgr2=0x7fe250003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:45:22.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.145+0000 7fe2667fc640 1 --2- 172.21.15.42:0/3999190668 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe2500039c0 0x7fe250003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:45:22.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.145+0000 7fe2667fc640 1 -- 172.21.15.42:0/3999190668 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe2500052f0 msgr2=0x7fe25000b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:45:22.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.145+0000 7fe2667fc640 1 --2- 172.21.15.42:0/3999190668 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe2500052f0 0x7fe25000b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:45:22.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.145+0000 7fe2667fc640 10 monclient: _finish_hunting 0 2024-09-16T03:45:22.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.145+0000 7fe2667fc640 1 monclient: found mon.c 2024-09-16T03:45:22.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.145+0000 7fe2667fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:45:22.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.145+0000 7fe2667fc640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:45:22.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.145+0000 7fe2667fc640 1 -- 172.21.15.42:0/3999190668 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe250003530 con 0x7fe250004810 2024-09-16T03:45:22.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.145+0000 7fe2667fc640 10 monclient: _finish_auth 0 2024-09-16T03:45:22.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.145+0000 7fe2667fc640 10 monclient: _check_auth_tickets 2024-09-16T03:45:22.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.145+0000 7fe2667fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:45:22.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.145+0000 7fe26d8ab640 5 monclient: authenticate success, global_id 272743 2024-09-16T03:45:22.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.145+0000 7fe26d8ab640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:45:22.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.145+0000 7fe2677fe640 1 -- 172.21.15.42:0/3999190668 reap_dead start 2024-09-16T03:45:22.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.145+0000 7fe2667fc640 1 --2- 172.21.15.42:0/3999190668 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe250004810 0x7fe250004c90 secure :-1 s=READY pgs=2360 cs=0 l=1 rev1=1 crypto rx=0x7fe25c001750 tx=0x7fe25c01c3c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=82adf2d2498bf84a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:45:22.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.149+0000 7fe265ffb640 1 -- 172.21.15.42:0/3999190668 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe25c0013a0 con 0x7fe250004810 2024-09-16T03:45:22.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.149+0000 7fe265ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:45:22.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.149+0000 7fe265ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:45:22.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.149+0000 7fe265ffb640 10 monclient: dump: 2024-09-16T03:45:22.177 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:45:22.177 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:45:22.177 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:22.177 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:22.177 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:45:22.177 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:45:22.177 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:45:22.178 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:45:22.178 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:45:22.178 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:45:22.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.149+0000 7fe26d8ab640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:45:22.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.149+0000 7fe265ffb640 1 -- 172.21.15.42:0/3999190668 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe25c0052f0 con 0x7fe250004810 2024-09-16T03:45:22.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.149+0000 7fe265ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T03:45:22.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.149+0000 7fe26d8ab640 10 monclient: get_monmap_and_config success 2024-09-16T03:45:22.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.149+0000 7fe265ffb640 1 -- 172.21.15.42:0/3999190668 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe25c003840 con 0x7fe250004810 2024-09-16T03:45:22.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.149+0000 7fe26d8ab640 4 set_mon_vals no callback set 2024-09-16T03:45:22.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.149+0000 7fe265ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:45:22.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.149+0000 7fe265ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:45:22.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.149+0000 7fe265ffb640 10 monclient: dump: 2024-09-16T03:45:22.179 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:45:22.179 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:45:22.179 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:22.179 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:22.179 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:45:22.179 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:45:22.179 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:45:22.179 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:45:22.179 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:45:22.179 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:45:22.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.149+0000 7fe26d8ab640 10 monclient: shutdown 2024-09-16T03:45:22.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.149+0000 7fe26d8ab640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:45:22.180 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.149+0000 7fe26d8ab640 1 -- 172.21.15.42:0/3999190668 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe250004810 msgr2=0x7fe250004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:45:22.180 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.149+0000 7fe26d8ab640 1 --2- 172.21.15.42:0/3999190668 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe250004810 0x7fe250004c90 secure :-1 s=READY pgs=2360 cs=0 l=1 rev1=1 crypto rx=0x7fe25c001750 tx=0x7fe25c01c3c0 comp rx=0 tx=0).stop 2024-09-16T03:45:22.180 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.149+0000 7fe26d8ab640 1 -- 172.21.15.42:0/3999190668 shutdown_connections 2024-09-16T03:45:22.180 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.149+0000 7fe26d8ab640 1 --2- 172.21.15.42:0/3999190668 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe250004810 0x7fe250004c90 unknown :-1 s=CLOSED pgs=2360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:45:22.180 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.149+0000 7fe26d8ab640 1 -- 172.21.15.42:0/3999190668 >> 172.21.15.42:0/3999190668 conn(0x7fe2680a9fb0 msgr2=0x7fe2680aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:45:22.180 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.149+0000 7fe26d8ab640 1 -- 172.21.15.42:0/3999190668 shutdown_connections 2024-09-16T03:45:22.180 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.149+0000 7fe26d8ab640 1 -- 172.21.15.42:0/3999190668 wait complete. 2024-09-16T03:45:22.180 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.149+0000 7fe26d8ab640 10 monclient: build_initial_monmap 2024-09-16T03:45:22.180 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.149+0000 7fe26d8ab640 10 monclient: monmap: 2024-09-16T03:45:22.180 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:45:22.180 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:45:22.180 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:45:22.180 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:45:22.181 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:45:22.181 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:45:22.181 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:45:22.181 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:45:22.181 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:45:22.181 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:45:22.181 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.149+0000 7fe26d8ab640 1 librados: starting msgr at 2024-09-16T03:45:22.181 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.149+0000 7fe26d8ab640 1 librados: starting objecter 2024-09-16T03:45:22.181 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.149+0000 7fe26d8ab640 1 Processor -- start 2024-09-16T03:45:22.181 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.149+0000 7fe26d8ab640 1 -- start start 2024-09-16T03:45:22.181 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.149+0000 7fe26d8ab640 1 librados: setting wanted keys 2024-09-16T03:45:22.181 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.149+0000 7fe26d8ab640 1 librados: calling monclient init 2024-09-16T03:45:22.181 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.149+0000 7fe26d8ab640 10 monclient: init 2024-09-16T03:45:22.181 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.149+0000 7fe26d8ab640 10 monclient: _reopen_session rank -1 2024-09-16T03:45:22.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.149+0000 7fe26d8ab640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:45:22.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.149+0000 7fe26d8ab640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe250004810 0x7fe2680a33e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:45:22.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.149+0000 7fe26d8ab640 10 monclient(hunting): picked mon.noname-b con 0x7fe250004810 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:45:22.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.149+0000 7fe26d8ab640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe26813afe0 0x7fe26813b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:45:22.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.149+0000 7fe26d8ab640 10 monclient(hunting): picked mon.noname-a con 0x7fe26813afe0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:45:22.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.149+0000 7fe26d8ab640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe26813b960 0x7fe2681d59b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:45:22.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.149+0000 7fe26d8ab640 10 monclient(hunting): picked mon.noname-c con 0x7fe26813b960 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:45:22.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.149+0000 7fe26d8ab640 10 monclient(hunting): start opening mon connection 2024-09-16T03:45:22.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.149+0000 7fe26d8ab640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fe268133f10 con 0x7fe26813afe0 2024-09-16T03:45:22.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.149+0000 7fe26d8ab640 10 monclient(hunting): start opening mon connection 2024-09-16T03:45:22.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.149+0000 7fe26d8ab640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe268133d90 con 0x7fe250004810 2024-09-16T03:45:22.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.149+0000 7fe26d8ab640 10 monclient(hunting): start opening mon connection 2024-09-16T03:45:22.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.149+0000 7fe26d8ab640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fe268134090 con 0x7fe26813b960 2024-09-16T03:45:22.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.149+0000 7fe26d8ab640 10 monclient(hunting): _renew_subs 2024-09-16T03:45:22.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.149+0000 7fe26d8ab640 10 monclient(hunting): authenticate will time out at 1169.122303s 2024-09-16T03:45:22.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.149+0000 7fe2677fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe26813b960 0x7fe2681d59b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:45:22.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.149+0000 7fe2677fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe26813b960 0x7fe2681d59b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:38714/0 (socket says 172.21.15.42:38714) 2024-09-16T03:45:22.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.149+0000 7fe2677fe640 1 -- 172.21.15.42:0/410360227 learned_addr learned my addr 172.21.15.42:0/410360227 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:45:22.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.149+0000 7fe2677fe640 10 monclient(hunting): get_auth_request con 0x7fe26813b960 auth_method 0 2024-09-16T03:45:22.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.149+0000 7fe2677fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:45:22.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.149+0000 7fe2677fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:45:22.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.149+0000 7fe2677fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:45:22.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.149+0000 7fe2667fc640 1 --2- 172.21.15.42:0/410360227 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe26813afe0 0x7fe26813b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:45:22.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.149+0000 7fe2667fc640 10 monclient(hunting): get_auth_request con 0x7fe26813afe0 auth_method 0 2024-09-16T03:45:22.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.149+0000 7fe2667fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:45:22.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.149+0000 7fe2667fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:45:22.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.149+0000 7fe2667fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:45:22.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.149+0000 7fe2677fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:45:22.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.149+0000 7fe2677fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:45:22.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.149+0000 7fe2677fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:45:22.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.149+0000 7fe2667fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:45:22.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.149+0000 7fe2667fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:45:22.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.149+0000 7fe266ffd640 1 --2- 172.21.15.42:0/410360227 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe250004810 0x7fe2680a33e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:45:22.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.149+0000 7fe2667fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:45:22.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.149+0000 7fe266ffd640 10 monclient(hunting): get_auth_request con 0x7fe250004810 auth_method 0 2024-09-16T03:45:22.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.149+0000 7fe266ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:45:22.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.149+0000 7fe266ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:45:22.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.149+0000 7fe266ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:45:22.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.149+0000 7fe2677fe640 10 monclient(hunting): handle_auth_done global_id 13355 payload 824 2024-09-16T03:45:22.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.149+0000 7fe2677fe640 1 -- 172.21.15.42:0/410360227 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe250004810 msgr2=0x7fe2680a33e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:45:22.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.149+0000 7fe2677fe640 1 --2- 172.21.15.42:0/410360227 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe250004810 0x7fe2680a33e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:45:22.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.149+0000 7fe2677fe640 1 -- 172.21.15.42:0/410360227 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe26813afe0 msgr2=0x7fe26813b420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:45:22.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.149+0000 7fe2677fe640 1 --2- 172.21.15.42:0/410360227 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe26813afe0 0x7fe26813b420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:45:22.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.149+0000 7fe2677fe640 10 monclient: _finish_hunting 0 2024-09-16T03:45:22.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.149+0000 7fe2677fe640 1 monclient: found mon.noname-c 2024-09-16T03:45:22.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.149+0000 7fe2677fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:45:22.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.149+0000 7fe2677fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:45:22.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.149+0000 7fe2677fe640 1 -- 172.21.15.42:0/410360227 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2680a9580 con 0x7fe26813b960 2024-09-16T03:45:22.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.149+0000 7fe266ffd640 1 --2- 172.21.15.42:0/410360227 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe250004810 0x7fe2680a33e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T03:45:22.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.149+0000 7fe2677fe640 1 --2- 172.21.15.42:0/410360227 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe26813b960 0x7fe2681d59b0 secure :-1 s=READY pgs=2361 cs=0 l=1 rev1=1 crypto rx=0x7fe26806e7a0 tx=0x7fe2600115a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:45:22.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.149+0000 7fe24ffff640 10 client.?.objecter ms_handle_connect 0x7fe26813b960 2024-09-16T03:45:22.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.149+0000 7fe24ffff640 10 client.?.objecter resend_mon_ops 2024-09-16T03:45:22.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.149+0000 7fe24ffff640 1 -- 172.21.15.42:0/410360227 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe260011e10 con 0x7fe26813b960 2024-09-16T03:45:22.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.149+0000 7fe24ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:45:22.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.149+0000 7fe24ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:45:22.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.149+0000 7fe24ffff640 10 monclient: dump: 2024-09-16T03:45:22.186 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:45:22.186 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:45:22.186 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:22.186 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:22.186 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:45:22.186 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:45:22.186 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:45:22.186 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:45:22.186 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:45:22.186 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:45:22.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.149+0000 7fe24ffff640 10 monclient: _finish_auth 0 2024-09-16T03:45:22.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.149+0000 7fe24ffff640 10 monclient: _check_auth_tickets 2024-09-16T03:45:22.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.149+0000 7fe24ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:45:22.187 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.149+0000 7fe24ffff640 1 -- 172.21.15.42:0/410360227 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe26001b070 con 0x7fe26813b960 2024-09-16T03:45:22.187 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.149+0000 7fe26d8ab640 5 monclient: authenticate success, global_id 13355 2024-09-16T03:45:22.187 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.149+0000 7fe26d8ab640 10 monclient: _renew_subs 2024-09-16T03:45:22.187 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.149+0000 7fe26d8ab640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:45:22.187 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.149+0000 7fe26d8ab640 1 -- 172.21.15.42:0/410360227 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe268133550 con 0x7fe26813b960 2024-09-16T03:45:22.187 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.149+0000 7fe24ffff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:45:22.187 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.149+0000 7fe24ffff640 1 -- 172.21.15.42:0/410360227 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2600039a0 con 0x7fe26813b960 2024-09-16T03:45:22.187 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.149+0000 7fe24ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:45:22.187 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.149+0000 7fe24ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:45:22.187 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.149+0000 7fe24ffff640 10 monclient: dump: 2024-09-16T03:45:22.187 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:45:22.187 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:45:22.187 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:22.188 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:22.188 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:45:22.188 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:45:22.188 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:45:22.188 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:45:22.188 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:45:22.188 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:45:22.188 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.149+0000 7fe26d8ab640 10 client.13355.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:45:22.188 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.149+0000 7fe26d8ab640 10 monclient: _renew_subs 2024-09-16T03:45:22.188 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.149+0000 7fe26d8ab640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:45:22.188 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.149+0000 7fe2657fa640 4 set_mon_vals no callback set 2024-09-16T03:45:22.188 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.149+0000 7fe26d8ab640 1 -- 172.21.15.42:0/410360227 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fe26813bde0 con 0x7fe26813b960 2024-09-16T03:45:22.188 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.149+0000 7fe26d8ab640 1 librados: init done 2024-09-16T03:45:22.189 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.153+0000 7fe26d8ab640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:45:22.189 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.153+0000 7fe26d8ab640 1 --2- 172.21.15.42:0/410360227 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe26809d5f0 0x7fe268137120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:45:22.189 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.153+0000 7fe26d8ab640 10 monclient(hunting): start opening mon connection 2024-09-16T03:45:22.189 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.153+0000 7fe26d8ab640 1 -- 172.21.15.42:0/410360227 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fe26809c8d0 con 0x7fe26809d5f0 2024-09-16T03:45:22.189 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.153+0000 7fe266ffd640 1 --2- 172.21.15.42:0/410360227 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe26809d5f0 0x7fe268137120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:45:22.189 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.153+0000 7fe24ffff640 1 -- 172.21.15.42:0/410360227 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fe260003b40 con 0x7fe26813b960 2024-09-16T03:45:22.189 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.153+0000 7fe266ffd640 10 monclient: get_auth_request con 0x7fe26809d5f0 auth_method 0 2024-09-16T03:45:22.189 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.153+0000 7fe266ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:45:22.189 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.153+0000 7fe266ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:45:22.189 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.153+0000 7fe266ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:45:22.189 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.153+0000 7fe266ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:45:22.189 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.153+0000 7fe266ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:45:22.189 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.153+0000 7fe266ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:45:22.189 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.153+0000 7fe24ffff640 1 --2- 172.21.15.42:0/410360227 >> v2:172.21.15.154:6812/254845342 conn(0x7fe254037840 0x7fe254039d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:45:22.190 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.153+0000 7fe24ffff640 1 -- 172.21.15.42:0/410360227 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(90..90 src has 1..90) ==== 7319+0+0 (secure 0 0 0) 0x7fe260049c20 con 0x7fe26813b960 2024-09-16T03:45:22.190 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.153+0000 7fe24ffff640 10 client.13355.objecter ms_dispatch 0x7fe268000ba0 osd_map(90..90 src has 1..90) 2024-09-16T03:45:22.190 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.153+0000 7fe24ffff640 3 client.13355.objecter handle_osd_map got epochs [90,90] > 0 2024-09-16T03:45:22.190 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.153+0000 7fe24ffff640 3 client.13355.objecter handle_osd_map decoding full epoch 90 2024-09-16T03:45:22.190 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.153+0000 7fe24ffff640 20 client.13355.objecter dump_active .. 0 homeless 2024-09-16T03:45:22.190 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.153+0000 7fe266ffd640 10 monclient(hunting): handle_auth_done global_id 744180 payload 824 2024-09-16T03:45:22.190 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.153+0000 7fe2667fc640 1 --2- 172.21.15.42:0/410360227 >> v2:172.21.15.154:6812/254845342 conn(0x7fe254037840 0x7fe254039d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:45:22.190 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.153+0000 7fe266ffd640 1 -- 172.21.15.42:0/410360227 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe2680a5750 con 0x7fe26809d5f0 2024-09-16T03:45:22.190 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.153+0000 7fe2667fc640 10 monclient: get_auth_request con 0x7fe254037840 auth_method 0 2024-09-16T03:45:22.190 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.153+0000 7fe24ffff640 10 client.13355.objecter ms_handle_connect 0x7fe26809d5f0 2024-09-16T03:45:22.191 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.153+0000 7fe24ffff640 10 client.13355.objecter resend_mon_ops 2024-09-16T03:45:22.191 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.153+0000 7fe266ffd640 1 --2- 172.21.15.42:0/410360227 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe26809d5f0 0x7fe268137120 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fe258001ec0 tx=0x7fe258007720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:45:22.191 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.153+0000 7fe24ffff640 10 client.13355.objecter ms_handle_connect 0x7fe254037840 2024-09-16T03:45:22.191 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.153+0000 7fe2667fc640 1 --2- 172.21.15.42:0/410360227 >> v2:172.21.15.154:6812/254845342 conn(0x7fe254037840 0x7fe254039d00 secure :-1 s=READY pgs=3364 cs=0 l=1 rev1=1 crypto rx=0x7fe2500057f0 tx=0x7fe25c0011a0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:45:22.193 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.173+0000 7fe24ffff640 1 -- 172.21.15.42:0/410360227 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe258001650 con 0x7fe26809d5f0 2024-09-16T03:45:22.193 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.173+0000 7fe24ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:45:22.193 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.173+0000 7fe24ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:45:22.193 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.173+0000 7fe24ffff640 10 monclient: dump: 2024-09-16T03:45:22.193 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:45:22.193 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:45:22.193 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:22.193 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:22.193 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:45:22.193 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:45:22.193 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:45:22.193 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:45:22.193 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:45:22.194 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:45:22.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.173+0000 7fe24ffff640 1 -- 172.21.15.42:0/410360227 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fe25801a070 con 0x7fe26809d5f0 2024-09-16T03:45:22.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.173+0000 7fe24ffff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:45:22.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.173+0000 7fe24ffff640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:45:22.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.173+0000 7fe24ffff640 1 -- 172.21.15.42:0/410360227 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe26809d5f0 msgr2=0x7fe268137120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:45:22.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.173+0000 7fe24ffff640 1 --2- 172.21.15.42:0/410360227 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe26809d5f0 0x7fe268137120 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fe258001ec0 tx=0x7fe258007720 comp rx=0 tx=0).stop 2024-09-16T03:45:22.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.173+0000 7fe24ffff640 1 -- 172.21.15.42:0/410360227 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe26809d5f0 msgr2=0x7fe268137120 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:45:22.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.173+0000 7fe24ffff640 1 --2- 172.21.15.42:0/410360227 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe26809d5f0 0x7fe268137120 secure :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fe258001ec0 tx=0x7fe258007720 comp rx=0 tx=0).stop 2024-09-16T03:45:22.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.181+0000 7fe26d8ab640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:45:22.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.181+0000 7fe26d8ab640 1 --2- 172.21.15.42:0/410360227 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe26809d9d0 0x7fe268137660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:45:22.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.181+0000 7fe26d8ab640 10 monclient(hunting): start opening mon connection 2024-09-16T03:45:22.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.181+0000 7fe26d8ab640 1 -- 172.21.15.42:0/410360227 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fe268068e40 con 0x7fe26809d9d0 2024-09-16T03:45:22.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.181+0000 7fe266ffd640 1 --2- 172.21.15.42:0/410360227 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe26809d9d0 0x7fe268137660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:45:22.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.181+0000 7fe266ffd640 10 monclient: get_auth_request con 0x7fe26809d9d0 auth_method 0 2024-09-16T03:45:22.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.181+0000 7fe266ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:45:22.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.181+0000 7fe266ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:45:22.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.181+0000 7fe266ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:45:22.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.181+0000 7fe266ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:45:22.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.181+0000 7fe266ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:45:22.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.181+0000 7fe266ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:45:22.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.181+0000 7fe266ffd640 10 monclient(hunting): handle_auth_done global_id 744183 payload 824 2024-09-16T03:45:22.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.181+0000 7fe266ffd640 1 -- 172.21.15.42:0/410360227 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fe2681db090 con 0x7fe26809d9d0 2024-09-16T03:45:22.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.181+0000 7fe266ffd640 1 --2- 172.21.15.42:0/410360227 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe26809d9d0 0x7fe268137660 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fe258019000 tx=0x7fe258007900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:45:22.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.181+0000 7fe24ffff640 10 client.13355.objecter ms_handle_connect 0x7fe26809d9d0 2024-09-16T03:45:22.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.181+0000 7fe24ffff640 10 client.13355.objecter resend_mon_ops 2024-09-16T03:45:22.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.181+0000 7fe24ffff640 1 -- 172.21.15.42:0/410360227 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe25801fe10 con 0x7fe26809d9d0 2024-09-16T03:45:22.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.181+0000 7fe24ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:45:22.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.181+0000 7fe24ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:45:22.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.181+0000 7fe24ffff640 10 monclient: dump: 2024-09-16T03:45:22.196 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:45:22.196 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:45:22.196 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:22.196 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:22.196 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:45:22.196 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:45:22.196 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:45:22.196 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:45:22.196 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:45:22.196 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:45:22.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.181+0000 7fe24ffff640 1 -- 172.21.15.42:0/410360227 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4712 (secure 0 0 0) 0x7fe258015680 con 0x7fe26809d9d0 2024-09-16T03:45:22.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.181+0000 7fe24ffff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:45:22.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.181+0000 7fe24ffff640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:45:22.197 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.181+0000 7fe24ffff640 1 -- 172.21.15.42:0/410360227 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe26809d9d0 msgr2=0x7fe268137660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:45:22.197 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.181+0000 7fe24ffff640 1 --2- 172.21.15.42:0/410360227 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe26809d9d0 0x7fe268137660 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fe258019000 tx=0x7fe258007900 comp rx=0 tx=0).stop 2024-09-16T03:45:22.197 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.181+0000 7fe24ffff640 1 -- 172.21.15.42:0/410360227 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe26809d9d0 msgr2=0x7fe268137660 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:45:22.197 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.181+0000 7fe24ffff640 1 --2- 172.21.15.42:0/410360227 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe26809d9d0 0x7fe268137660 secure :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fe258019000 tx=0x7fe258007900 comp rx=0 tx=0).stop 2024-09-16T03:45:22.197 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:45:22.197 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:45:22.197 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:45:22.197 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "leader", 2024-09-16T03:45:22.197 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 28, 2024-09-16T03:45:22.197 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 9551, 2024-09-16T03:45:22.197 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:45:22.198 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:45:22.198 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T03:45:22.198 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T03:45:22.198 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:45:22.198 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 3, 2024-09-16T03:45:22.198 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:45:22.198 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:45:22.198 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:45:22.198 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:45:22.198 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:45:22.198 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:45:22.198 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:45:22.198 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:45:22.198 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:45:22.199 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:45:22.199 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:45:22.199 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:45:22.199 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:45:22.199 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:45:22.199 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:45:22.199 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:45:22.199 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:45:22.199 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:45:22.199 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:45:22.199 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:45:22.199 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:45:22.199 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:45:22.199 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:45:22.199 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:45:22.200 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:45:22.200 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:45:22.200 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:45:22.200 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:45:22.200 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:45:22.200 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:45:22.200 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:45:22.200 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:45:22.200 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:45:22.200 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:45:22.200 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:45:22.200 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:45:22.200 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:45:22.200 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:45:22.201 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:45:22.201 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:45:22.201 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:45:22.201 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:45:22.201 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:45:22.201 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:45:22.201 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:45:22.201 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:45:22.201 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:45:22.201 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:45:22.201 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:45:22.201 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:45:22.201 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:45:22.201 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:45:22.202 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:45:22.202 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:45:22.202 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:45:22.202 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:45:22.202 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:45:22.202 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:45:22.202 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:45:22.202 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:45:22.202 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:45:22.202 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:45:22.202 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:45:22.202 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:45:22.202 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:45:22.202 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:45:22.202 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:45:22.203 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:45:22.203 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:45:22.203 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:45:22.203 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:45:22.203 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:45:22.203 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:45:22.203 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:45:22.203 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:45:22.203 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:45:22.203 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:45:22.204 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:45:22.204 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:45:22.204 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:45:22.204 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:45:22.204 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:45:22.204 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:45:22.204 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:45:22.204 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:45:22.204 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:45:22.204 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:45:22.204 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:45:22.204 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:45:22.204 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:45:22.204 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:45:22.205 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:45:22.205 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:45:22.205 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:45:22.205 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:45:22.205 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:45:22.205 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:45:22.205 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:45:22.205 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:45:22.205 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:45:22.205 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:45:22.205 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:45:22.205 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:45:22.205 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:45:22.205 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:45:22.205 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:45:22.206 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:45:22.206 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:45:22.206 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:45:22.206 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:45:22.206 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:45:22.206 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:45:22.206 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:45:22.206 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:45:22.206 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:45:22.206 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:45:22.206 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:45:22.206 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:45:22.206 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:45:22.206 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:45:22.207 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:45:22.207 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:45:22.207 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:45:22.207 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:45:22.207 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:45:22.207 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:45:22.207 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:45:22.207 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:45:22.207 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:45:22.207 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:45:22.207 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:45:22.207 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:45:22.207 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:45:22.207 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:45:22.207 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:45:22.208 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:45:22.208 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:45:22.208 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:45:22.208 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:45:22.208 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:45:22.208 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:45:22.208 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T03:45:22.208 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:45:22.208 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:45:22.208 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:45:22.208 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:45:22.208 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:45:22.208 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:45:22.208 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:45:22.209 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:45:22.209 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:45:22.209 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:45:22.209 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:45:22.209 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:45:22.209 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:45:22.209 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:45:22.209 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:45:22.209 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:45:22.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.181+0000 7fe26d8ab640 10 librados: watch_flush enter 2024-09-16T03:45:22.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.181+0000 7fe26d8ab640 10 librados: watch_flush exit 2024-09-16T03:45:22.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.181+0000 7fe26d8ab640 20 client.13355.objecter shutdown clearing up homeless session... 2024-09-16T03:45:22.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.181+0000 7fe26d8ab640 10 client.13355.objecter successfully canceled tick 2024-09-16T03:45:22.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.181+0000 7fe26d8ab640 1 -- 172.21.15.42:0/410360227 >> v2:172.21.15.154:6812/254845342 conn(0x7fe254037840 msgr2=0x7fe254039d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:45:22.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.181+0000 7fe26d8ab640 1 --2- 172.21.15.42:0/410360227 >> v2:172.21.15.154:6812/254845342 conn(0x7fe254037840 0x7fe254039d00 secure :-1 s=READY pgs=3364 cs=0 l=1 rev1=1 crypto rx=0x7fe2500057f0 tx=0x7fe25c0011a0 comp rx=0 tx=0).stop 2024-09-16T03:45:22.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.181+0000 7fe26d8ab640 10 monclient: shutdown 2024-09-16T03:45:22.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.181+0000 7fe26d8ab640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:45:22.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.181+0000 7fe26d8ab640 1 -- 172.21.15.42:0/410360227 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe26813b960 msgr2=0x7fe2681d59b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:45:22.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.181+0000 7fe26d8ab640 1 --2- 172.21.15.42:0/410360227 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe26813b960 0x7fe2681d59b0 secure :-1 s=READY pgs=2361 cs=0 l=1 rev1=1 crypto rx=0x7fe26806e7a0 tx=0x7fe2600115a0 comp rx=0 tx=0).stop 2024-09-16T03:45:22.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.181+0000 7fe2677fe640 1 -- 172.21.15.42:0/410360227 reap_dead start 2024-09-16T03:45:22.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.181+0000 7fe2677fe640 1 -- 172.21.15.42:0/410360227 reap_dead start 2024-09-16T03:45:22.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.181+0000 7fe26d8ab640 1 -- 172.21.15.42:0/410360227 shutdown_connections 2024-09-16T03:45:22.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.181+0000 7fe26d8ab640 1 -- 172.21.15.42:0/410360227 >> 172.21.15.42:0/410360227 conn(0x7fe2680a9fb0 msgr2=0x7fe26813f3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:45:22.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.181+0000 7fe26d8ab640 1 -- 172.21.15.42:0/410360227 shutdown_connections 2024-09-16T03:45:22.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.181+0000 7fe26d8ab640 1 -- 172.21.15.42:0/410360227 wait complete. 2024-09-16T03:45:22.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.181+0000 7fe26d8ab640 1 librados: shutdown 2024-09-16T03:45:22.212 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:45:22.212 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T03:45:22.212 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T03:45:22.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.321+0000 7f66595c6640 10 monclient: get_monmap_and_config 2024-09-16T03:45:22.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.321+0000 7f66595c6640 10 monclient: build_initial_monmap 2024-09-16T03:45:22.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.321+0000 7f66595c6640 10 monclient: monmap: 2024-09-16T03:45:22.326 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:45:22.326 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:45:22.326 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:45:22.325186+0000 2024-09-16T03:45:22.326 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:45:22.325186+0000 2024-09-16T03:45:22.326 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:45:22.326 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:45:22.326 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:45:22.326 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:45:22.326 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:45:22.327 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:45:22.327 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:45:22.327 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:45:22.329 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.321+0000 7f66595c6640 1 Processor -- start 2024-09-16T03:45:22.329 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.321+0000 7f66595c6640 1 -- start start 2024-09-16T03:45:22.329 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.321+0000 7f66595c6640 10 monclient: init 2024-09-16T03:45:22.329 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.321+0000 7f66595c6640 10 monclient: _reopen_session rank -1 2024-09-16T03:45:22.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.321+0000 7f66595c6640 10 monclient: _add_conns ranks=[0,3,4,2,1] 2024-09-16T03:45:22.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.321+0000 7f66595c6640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f66540a1330 addr v1:172.21.15.42:6789/0 2024-09-16T03:45:22.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.321+0000 7f66595c6640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f665406dac0 0x7f665413a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:45:22.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.321+0000 7f66595c6640 10 monclient(hunting): picked mon.noname-a con 0x7f665406dac0 addr v2:172.21.15.42:3300/0 2024-09-16T03:45:22.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.321+0000 7f66595c6640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f66540a1e20 0x7f665413c890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:45:22.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.321+0000 7f66595c6640 10 monclient(hunting): picked mon.noname-c con 0x7f66540a1e20 addr v2:172.21.15.154:3300/0 2024-09-16T03:45:22.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.321+0000 7f66595c6640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f665409c090 con 0x7f66540a1330 2024-09-16T03:45:22.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.321+0000 7f66595c6640 10 monclient(hunting): start opening mon connection 2024-09-16T03:45:22.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.321+0000 7f66595c6640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f665409f610 con 0x7f665406dac0 2024-09-16T03:45:22.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.321+0000 7f66595c6640 10 monclient(hunting): start opening mon connection 2024-09-16T03:45:22.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.321+0000 7f66595c6640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f665409f8f0 con 0x7f66540a1e20 2024-09-16T03:45:22.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.321+0000 7f66595c6640 10 monclient(hunting): _renew_subs 2024-09-16T03:45:22.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.321+0000 7f66595c6640 10 monclient(hunting): authenticate will time out at 1169.294674s 2024-09-16T03:45:22.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.321+0000 7f66527fc640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f665406dac0 0x7f665413a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:45:22.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.321+0000 7f66527fc640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f665406dac0 0x7f665413a340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:33494/0 (socket says 172.21.15.42:33494) 2024-09-16T03:45:22.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.321+0000 7f66527fc640 1 -- 172.21.15.42:0/2229575927 learned_addr learned my addr 172.21.15.42:0/2229575927 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:45:22.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.321+0000 7f66527fc640 10 monclient(hunting): get_auth_request con 0x7f665406dac0 auth_method 0 2024-09-16T03:45:22.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.321+0000 7f66527fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:45:22.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.321+0000 7f6652ffd640 1 --1- 172.21.15.42:0/2229575927 >> v1:172.21.15.42:6789/0 conn(0x7f66540a1330 0x7f66540a1710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:55288/0 (socket says 172.21.15.42:55288) 2024-09-16T03:45:22.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.321+0000 7f66527fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:45:22.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.321+0000 7f66527fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:45:22.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.321+0000 7f66527fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:45:22.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.321+0000 7f66527fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:45:22.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.321+0000 7f66537fe640 1 --2- 172.21.15.42:0/2229575927 >> v2:172.21.15.154:3300/0 conn(0x7f66540a1e20 0x7f665413c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:45:22.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.321+0000 7f66527fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:45:22.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.321+0000 7f66537fe640 10 monclient(hunting): get_auth_request con 0x7f66540a1e20 auth_method 0 2024-09-16T03:45:22.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.321+0000 7f66537fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:45:22.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.321+0000 7f66537fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:45:22.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.321+0000 7f66537fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:45:22.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.321+0000 7f66527fc640 10 monclient(hunting): handle_auth_done global_id 744186 payload 274 2024-09-16T03:45:22.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.321+0000 7f66527fc640 1 -- 172.21.15.42:0/2229575927 >> v2:172.21.15.154:3300/0 conn(0x7f66540a1e20 msgr2=0x7f665413c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:45:22.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.321+0000 7f66527fc640 1 --2- 172.21.15.42:0/2229575927 >> v2:172.21.15.154:3300/0 conn(0x7f66540a1e20 0x7f665413c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:45:22.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.321+0000 7f66527fc640 1 -- 172.21.15.42:0/2229575927 >> v1:172.21.15.42:6789/0 conn(0x7f66540a1330 legacy=0x7f66540a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:45:22.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.321+0000 7f66527fc640 10 monclient: _finish_hunting 0 2024-09-16T03:45:22.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.321+0000 7f66527fc640 1 monclient: found mon.noname-a 2024-09-16T03:45:22.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.321+0000 7f66527fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:45:22.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.321+0000 7f66527fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:45:22.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.321+0000 7f66527fc640 1 -- 172.21.15.42:0/2229575927 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f665409fbd0 con 0x7f665406dac0 2024-09-16T03:45:22.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.321+0000 7f66537fe640 1 --2- 172.21.15.42:0/2229575927 >> v2:172.21.15.154:3300/0 conn(0x7f66540a1e20 0x7f665413c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T03:45:22.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.325+0000 7f66527fc640 1 --2- 172.21.15.42:0/2229575927 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f665406dac0 0x7f665413a340 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f665409c800 tx=0x7f663c01c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0a29aa8a428a6c3 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:45:22.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.325+0000 7f6651ffb640 1 -- 172.21.15.42:0/2229575927 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f663c0b0070 con 0x7f665406dac0 2024-09-16T03:45:22.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.325+0000 7f6651ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:45:22.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.325+0000 7f6651ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:45:22.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.325+0000 7f6651ffb640 10 monclient: dump: 2024-09-16T03:45:22.340 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:45:22.340 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:45:22.340 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:22.340 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:22.340 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:45:22.340 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:45:22.340 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:45:22.341 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:45:22.341 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:45:22.341 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:45:22.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.325+0000 7f6651ffb640 10 monclient: _finish_auth 0 2024-09-16T03:45:22.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.325+0000 7f6651ffb640 10 monclient: _check_auth_tickets 2024-09-16T03:45:22.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.325+0000 7f6651ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:45:22.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.325+0000 7f6651ffb640 1 -- 172.21.15.42:0/2229575927 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f663c0ab600 con 0x7f665406dac0 2024-09-16T03:45:22.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.325+0000 7f6651ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T03:45:22.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.325+0000 7f6651ffb640 1 -- 172.21.15.42:0/2229575927 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f663c0ab9d0 con 0x7f665406dac0 2024-09-16T03:45:22.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.325+0000 7f66595c6640 5 monclient: authenticate success, global_id 744186 2024-09-16T03:45:22.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.325+0000 7f66595c6640 10 monclient: get_monmap_and_config success 2024-09-16T03:45:22.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.325+0000 7f66595c6640 4 set_mon_vals no callback set 2024-09-16T03:45:22.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.325+0000 7f6651ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:45:22.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.325+0000 7f6651ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:45:22.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.325+0000 7f6651ffb640 10 monclient: dump: 2024-09-16T03:45:22.342 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:45:22.342 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:45:22.342 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:22.342 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:22.342 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:45:22.342 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:45:22.342 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:45:22.342 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:45:22.342 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:45:22.343 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:45:22.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.325+0000 7f66595c6640 10 monclient: shutdown 2024-09-16T03:45:22.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.325+0000 7f66595c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:45:22.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.325+0000 7f66595c6640 1 -- 172.21.15.42:0/2229575927 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f665406dac0 msgr2=0x7f665413a340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:45:22.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.325+0000 7f66595c6640 1 --2- 172.21.15.42:0/2229575927 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f665406dac0 0x7f665413a340 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f665409c800 tx=0x7f663c01c470 comp rx=0 tx=0).stop 2024-09-16T03:45:22.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.325+0000 7f66595c6640 1 -- 172.21.15.42:0/2229575927 shutdown_connections 2024-09-16T03:45:22.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.325+0000 7f66595c6640 1 --2- 172.21.15.42:0/2229575927 >> v2:172.21.15.154:3300/0 conn(0x7f66540a1e20 0x7f665413c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:45:22.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.325+0000 7f66595c6640 1 --2- 172.21.15.42:0/2229575927 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f665406dac0 0x7f665413a340 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:45:22.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.325+0000 7f66595c6640 1 -- 172.21.15.42:0/2229575927 >> 172.21.15.42:0/2229575927 conn(0x7f66540a2420 msgr2=0x7f66540a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:45:22.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.325+0000 7f66595c6640 1 -- 172.21.15.42:0/2229575927 shutdown_connections 2024-09-16T03:45:22.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.325+0000 7f66595c6640 1 -- 172.21.15.42:0/2229575927 wait complete. 2024-09-16T03:45:22.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.325+0000 7f66595c6640 10 monclient: build_initial_monmap 2024-09-16T03:45:22.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.325+0000 7f66595c6640 10 monclient: monmap: 2024-09-16T03:45:22.344 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:45:22.344 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:45:22.344 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:45:22.344 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:45:22.344 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:45:22.344 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:45:22.344 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:45:22.344 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:45:22.344 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:45:22.344 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:45:22.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.325+0000 7f66595c6640 1 librados: starting msgr at 2024-09-16T03:45:22.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.325+0000 7f66595c6640 1 librados: starting objecter 2024-09-16T03:45:22.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.325+0000 7f66595c6640 1 Processor -- start 2024-09-16T03:45:22.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.325+0000 7f66595c6640 1 -- start start 2024-09-16T03:45:22.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.325+0000 7f66595c6640 1 librados: setting wanted keys 2024-09-16T03:45:22.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.325+0000 7f66595c6640 1 librados: calling monclient init 2024-09-16T03:45:22.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.325+0000 7f66595c6640 10 monclient: init 2024-09-16T03:45:22.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.325+0000 7f66595c6640 10 monclient: _reopen_session rank -1 2024-09-16T03:45:22.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.325+0000 7f66595c6640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T03:45:22.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.325+0000 7f66595c6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f66540a1330 0x7f6654071090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:45:22.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.325+0000 7f66595c6640 10 monclient(hunting): picked mon.noname-b con 0x7f66540a1330 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:45:22.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.325+0000 7f66595c6640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f66540a1e20 0x7f66541324d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:45:22.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.325+0000 7f66595c6640 10 monclient(hunting): picked mon.noname-c con 0x7f66540a1e20 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:45:22.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.325+0000 7f66595c6640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6654132a10 0x7f665411bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:45:22.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.325+0000 7f66595c6640 10 monclient(hunting): picked mon.noname-a con 0x7f6654132a10 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:45:22.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.325+0000 7f66595c6640 10 monclient(hunting): start opening mon connection 2024-09-16T03:45:22.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.325+0000 7f66595c6640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f665413f800 con 0x7f6654132a10 2024-09-16T03:45:22.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.325+0000 7f66595c6640 10 monclient(hunting): start opening mon connection 2024-09-16T03:45:22.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.325+0000 7f66595c6640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f665409ef80 con 0x7f66540a1330 2024-09-16T03:45:22.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.325+0000 7f66595c6640 10 monclient(hunting): start opening mon connection 2024-09-16T03:45:22.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.325+0000 7f66595c6640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f665413f980 con 0x7f66540a1e20 2024-09-16T03:45:22.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.325+0000 7f66595c6640 10 monclient(hunting): _renew_subs 2024-09-16T03:45:22.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.325+0000 7f66595c6640 10 monclient(hunting): authenticate will time out at 1169.299963s 2024-09-16T03:45:22.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.329+0000 7f66527fc640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f66540a1e20 0x7f66541324d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:45:22.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.329+0000 7f66537fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6654132a10 0x7f665411bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:45:22.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.329+0000 7f66527fc640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f66540a1e20 0x7f66541324d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:38720/0 (socket says 172.21.15.42:38720) 2024-09-16T03:45:22.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.329+0000 7f66527fc640 1 -- 172.21.15.42:0/885426664 learned_addr learned my addr 172.21.15.42:0/885426664 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:45:22.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.329+0000 7f66537fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6654132a10 0x7f665411bf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:33506/0 (socket says 172.21.15.42:33506) 2024-09-16T03:45:22.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.329+0000 7f66527fc640 10 monclient(hunting): get_auth_request con 0x7f66540a1e20 auth_method 0 2024-09-16T03:45:22.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.329+0000 7f66527fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:45:22.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.329+0000 7f66527fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:45:22.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.329+0000 7f66527fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:45:22.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.329+0000 7f66537fe640 10 monclient(hunting): get_auth_request con 0x7f6654132a10 auth_method 0 2024-09-16T03:45:22.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.329+0000 7f66537fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:45:22.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.329+0000 7f66537fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:45:22.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.329+0000 7f66537fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:45:22.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.329+0000 7f66527fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:45:22.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.329+0000 7f66527fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:45:22.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.329+0000 7f66527fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:45:22.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.329+0000 7f66537fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:45:22.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.329+0000 7f66537fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:45:22.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.329+0000 7f66537fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:45:22.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.329+0000 7f6652ffd640 1 --2- 172.21.15.42:0/885426664 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f66540a1330 0x7f6654071090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:45:22.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.329+0000 7f6652ffd640 10 monclient(hunting): get_auth_request con 0x7f66540a1330 auth_method 0 2024-09-16T03:45:22.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.329+0000 7f6652ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:45:22.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.329+0000 7f6652ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:45:22.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.329+0000 7f6652ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:45:22.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.329+0000 7f66527fc640 10 monclient(hunting): handle_auth_done global_id 13358 payload 824 2024-09-16T03:45:22.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.329+0000 7f66527fc640 1 -- 172.21.15.42:0/885426664 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f66540a1330 msgr2=0x7f6654071090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:45:22.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.329+0000 7f66527fc640 1 --2- 172.21.15.42:0/885426664 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f66540a1330 0x7f6654071090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:45:22.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.329+0000 7f66527fc640 1 -- 172.21.15.42:0/885426664 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6654132a10 msgr2=0x7f665411bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:45:22.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.329+0000 7f66527fc640 1 --2- 172.21.15.42:0/885426664 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6654132a10 0x7f665411bf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:45:22.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.329+0000 7f66527fc640 10 monclient: _finish_hunting 0 2024-09-16T03:45:22.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.329+0000 7f66527fc640 1 monclient: found mon.noname-c 2024-09-16T03:45:22.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.329+0000 7f66527fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:45:22.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.329+0000 7f66527fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:45:22.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.329+0000 7f66527fc640 1 -- 172.21.15.42:0/885426664 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f665409cc80 con 0x7f66540a1e20 2024-09-16T03:45:22.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.329+0000 7f66537fe640 1 --2- 172.21.15.42:0/885426664 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6654132a10 0x7f665411bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:45:22.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.329+0000 7f6652ffd640 1 --2- 172.21.15.42:0/885426664 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f66540a1330 0x7f6654071090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T03:45:22.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.329+0000 7f66527fc640 1 --2- 172.21.15.42:0/885426664 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f66540a1e20 0x7f66541324d0 secure :-1 s=READY pgs=2362 cs=0 l=1 rev1=1 crypto rx=0x7f663c019080 tx=0x7f663c019210 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:45:22.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.329+0000 7f664b7fe640 10 client.?.objecter ms_handle_connect 0x7f66540a1e20 2024-09-16T03:45:22.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.329+0000 7f664b7fe640 10 client.?.objecter resend_mon_ops 2024-09-16T03:45:22.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.329+0000 7f664b7fe640 1 -- 172.21.15.42:0/885426664 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f663c0b9070 con 0x7f66540a1e20 2024-09-16T03:45:22.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.329+0000 7f664b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:45:22.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.329+0000 7f664b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:45:22.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.329+0000 7f664b7fe640 10 monclient: dump: 2024-09-16T03:45:22.349 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:45:22.349 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:45:22.349 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:22.349 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:22.350 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:45:22.350 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:45:22.350 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:45:22.350 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:45:22.350 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:45:22.350 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:45:22.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.329+0000 7f664b7fe640 10 monclient: _finish_auth 0 2024-09-16T03:45:22.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.329+0000 7f664b7fe640 10 monclient: _check_auth_tickets 2024-09-16T03:45:22.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.329+0000 7f664b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:45:22.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.329+0000 7f664b7fe640 1 -- 172.21.15.42:0/885426664 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f663c0b4400 con 0x7f66540a1e20 2024-09-16T03:45:22.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.329+0000 7f66595c6640 5 monclient: authenticate success, global_id 13358 2024-09-16T03:45:22.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.329+0000 7f66595c6640 10 monclient: _renew_subs 2024-09-16T03:45:22.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.329+0000 7f66595c6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:45:22.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.329+0000 7f66595c6640 1 -- 172.21.15.42:0/885426664 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6654073a30 con 0x7f66540a1e20 2024-09-16T03:45:22.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.329+0000 7f664b7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:45:22.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.329+0000 7f664b7fe640 1 -- 172.21.15.42:0/885426664 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f663c0b0040 con 0x7f66540a1e20 2024-09-16T03:45:22.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.329+0000 7f664b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:45:22.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.329+0000 7f664b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:45:22.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.329+0000 7f664b7fe640 10 monclient: dump: 2024-09-16T03:45:22.351 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:45:22.351 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:45:22.351 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:22.351 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:22.351 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:45:22.351 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:45:22.351 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:45:22.351 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:45:22.351 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:45:22.352 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:45:22.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.329+0000 7f66595c6640 10 client.13358.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:45:22.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.329+0000 7f66595c6640 10 monclient: _renew_subs 2024-09-16T03:45:22.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.329+0000 7f66517fa640 4 set_mon_vals no callback set 2024-09-16T03:45:22.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.329+0000 7f66595c6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:45:22.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.329+0000 7f66595c6640 1 -- 172.21.15.42:0/885426664 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f665411e8c0 con 0x7f66540a1e20 2024-09-16T03:45:22.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.329+0000 7f66595c6640 1 librados: init done 2024-09-16T03:45:22.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.329+0000 7f66595c6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:45:22.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.329+0000 7f66595c6640 1 --2- 172.21.15.42:0/885426664 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f665406dac0 0x7f665411c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:45:22.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.329+0000 7f66595c6640 10 monclient(hunting): start opening mon connection 2024-09-16T03:45:22.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.329+0000 7f66595c6640 1 -- 172.21.15.42:0/885426664 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f6654132e20 con 0x7f665406dac0 2024-09-16T03:45:22.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.329+0000 7f6652ffd640 1 --2- 172.21.15.42:0/885426664 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f665406dac0 0x7f665411c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:45:22.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.329+0000 7f6652ffd640 10 monclient: get_auth_request con 0x7f665406dac0 auth_method 0 2024-09-16T03:45:22.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.329+0000 7f6652ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:45:22.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.329+0000 7f6652ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:45:22.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.329+0000 7f6652ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:45:22.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.329+0000 7f664b7fe640 1 -- 172.21.15.42:0/885426664 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f663c0bc070 con 0x7f66540a1e20 2024-09-16T03:45:22.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.329+0000 7f6652ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:45:22.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.329+0000 7f6652ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:45:22.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.329+0000 7f6652ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:45:22.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.329+0000 7f6652ffd640 10 monclient(hunting): handle_auth_done global_id 13361 payload 824 2024-09-16T03:45:22.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.329+0000 7f664b7fe640 1 --2- 172.21.15.42:0/885426664 >> v2:172.21.15.154:6812/254845342 conn(0x7f6634037810 0x7f6634039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:45:22.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.329+0000 7f6652ffd640 1 -- 172.21.15.42:0/885426664 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f66540a1710 con 0x7f665406dac0 2024-09-16T03:45:22.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.329+0000 7f664b7fe640 1 -- 172.21.15.42:0/885426664 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(90..90 src has 1..90) ==== 7319+0+0 (secure 0 0 0) 0x7f663c0e2800 con 0x7f66540a1e20 2024-09-16T03:45:22.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.329+0000 7f664b7fe640 10 client.13358.objecter ms_dispatch 0x7f6654000ba0 osd_map(90..90 src has 1..90) 2024-09-16T03:45:22.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.329+0000 7f664b7fe640 3 client.13358.objecter handle_osd_map got epochs [90,90] > 0 2024-09-16T03:45:22.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.329+0000 7f664b7fe640 3 client.13358.objecter handle_osd_map decoding full epoch 90 2024-09-16T03:45:22.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.329+0000 7f664b7fe640 20 client.13358.objecter dump_active .. 0 homeless 2024-09-16T03:45:22.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.329+0000 7f664b7fe640 10 client.13358.objecter ms_handle_connect 0x7f665406dac0 2024-09-16T03:45:22.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.329+0000 7f664b7fe640 10 client.13358.objecter resend_mon_ops 2024-09-16T03:45:22.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.329+0000 7f6652ffd640 1 --2- 172.21.15.42:0/885426664 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f665406dac0 0x7f665411c4b0 secure :-1 s=READY pgs=2363 cs=0 l=1 rev1=1 crypto rx=0x7f664c007fd0 tx=0x7f664c001fe0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:45:22.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.329+0000 7f664b7fe640 1 -- 172.21.15.42:0/885426664 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f664c00e940 con 0x7f665406dac0 2024-09-16T03:45:22.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.329+0000 7f664b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:45:22.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.329+0000 7f664b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:45:22.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.329+0000 7f664b7fe640 10 monclient: dump: 2024-09-16T03:45:22.354 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:45:22.354 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:45:22.354 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:22.354 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:22.355 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:45:22.355 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:45:22.355 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:45:22.355 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:45:22.355 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:45:22.355 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:45:22.355 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.329+0000 7f66537fe640 1 --2- 172.21.15.42:0/885426664 >> v2:172.21.15.154:6812/254845342 conn(0x7f6634037810 0x7f6634039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:45:22.355 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.329+0000 7f66537fe640 10 monclient: get_auth_request con 0x7f6634037810 auth_method 0 2024-09-16T03:45:22.355 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.329+0000 7f664b7fe640 1 -- 172.21.15.42:0/885426664 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f664c018070 con 0x7f665406dac0 2024-09-16T03:45:22.355 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.329+0000 7f664b7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:45:22.355 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.329+0000 7f664b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:45:22.355 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.329+0000 7f664b7fe640 1 -- 172.21.15.42:0/885426664 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f665406dac0 msgr2=0x7f665411c4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:45:22.355 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.329+0000 7f664b7fe640 1 --2- 172.21.15.42:0/885426664 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f665406dac0 0x7f665411c4b0 secure :-1 s=READY pgs=2363 cs=0 l=1 rev1=1 crypto rx=0x7f664c007fd0 tx=0x7f664c001fe0 comp rx=0 tx=0).stop 2024-09-16T03:45:22.355 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.329+0000 7f664b7fe640 1 -- 172.21.15.42:0/885426664 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f665406dac0 msgr2=0x7f665411c4b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:45:22.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.329+0000 7f664b7fe640 1 --2- 172.21.15.42:0/885426664 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f665406dac0 0x7f665411c4b0 secure :-1 s=CLOSED pgs=2363 cs=0 l=1 rev1=1 crypto rx=0x7f664c007fd0 tx=0x7f664c001fe0 comp rx=0 tx=0).stop 2024-09-16T03:45:22.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.329+0000 7f664b7fe640 10 client.13358.objecter ms_handle_connect 0x7f6634037810 2024-09-16T03:45:22.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.329+0000 7f66537fe640 1 --2- 172.21.15.42:0/885426664 >> v2:172.21.15.154:6812/254845342 conn(0x7f6634037810 0x7f6634039cd0 secure :-1 s=READY pgs=3365 cs=0 l=1 rev1=1 crypto rx=0x7f6644009d30 tx=0x7f6644004740 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:45:22.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.337+0000 7f66595c6640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:45:22.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.337+0000 7f66595c6640 1 --2- 172.21.15.42:0/885426664 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f665411c9f0 0x7f66541347d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:45:22.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.337+0000 7f66595c6640 10 monclient(hunting): start opening mon connection 2024-09-16T03:45:22.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.337+0000 7f66595c6640 1 -- 172.21.15.42:0/885426664 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f665411ce00 con 0x7f665411c9f0 2024-09-16T03:45:22.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.337+0000 7f6652ffd640 1 --2- 172.21.15.42:0/885426664 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f665411c9f0 0x7f66541347d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:45:22.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.337+0000 7f6652ffd640 10 monclient: get_auth_request con 0x7f665411c9f0 auth_method 0 2024-09-16T03:45:22.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.337+0000 7f6652ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:45:22.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.337+0000 7f6652ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:45:22.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.337+0000 7f6652ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:45:22.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.337+0000 7f6652ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:45:22.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.337+0000 7f6652ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:45:22.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.337+0000 7f6652ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:45:22.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.337+0000 7f6652ffd640 10 monclient(hunting): handle_auth_done global_id 13364 payload 824 2024-09-16T03:45:22.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.337+0000 7f6652ffd640 1 -- 172.21.15.42:0/885426664 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f6654069040 con 0x7f665411c9f0 2024-09-16T03:45:22.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.337+0000 7f664b7fe640 10 client.13358.objecter ms_handle_connect 0x7f665411c9f0 2024-09-16T03:45:22.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.337+0000 7f664b7fe640 10 client.13358.objecter resend_mon_ops 2024-09-16T03:45:22.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.337+0000 7f6652ffd640 1 --2- 172.21.15.42:0/885426664 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f665411c9f0 0x7f66541347d0 secure :-1 s=READY pgs=2364 cs=0 l=1 rev1=1 crypto rx=0x7f664c007fd0 tx=0x7f664c001260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:45:22.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.337+0000 7f664b7fe640 1 -- 172.21.15.42:0/885426664 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f664c028640 con 0x7f665411c9f0 2024-09-16T03:45:22.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.337+0000 7f664b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:45:22.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.337+0000 7f664b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:45:22.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.337+0000 7f664b7fe640 10 monclient: dump: 2024-09-16T03:45:22.359 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:45:22.359 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:45:22.359 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:22.359 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:22.359 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:45:22.359 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:45:22.359 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:45:22.359 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:45:22.359 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:45:22.359 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:45:22.360 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.337+0000 7f664b7fe640 1 -- 172.21.15.42:0/885426664 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4879 (secure 0 0 0) 0x7f664c013420 con 0x7f665411c9f0 2024-09-16T03:45:22.360 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.337+0000 7f664b7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:45:22.360 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.337+0000 7f664b7fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:45:22.360 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.337+0000 7f664b7fe640 1 -- 172.21.15.42:0/885426664 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f665411c9f0 msgr2=0x7f66541347d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:45:22.360 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.337+0000 7f664b7fe640 1 --2- 172.21.15.42:0/885426664 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f665411c9f0 0x7f66541347d0 secure :-1 s=READY pgs=2364 cs=0 l=1 rev1=1 crypto rx=0x7f664c007fd0 tx=0x7f664c001260 comp rx=0 tx=0).stop 2024-09-16T03:45:22.360 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.337+0000 7f664b7fe640 1 -- 172.21.15.42:0/885426664 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f665411c9f0 msgr2=0x7f66541347d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:45:22.360 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.337+0000 7f664b7fe640 1 --2- 172.21.15.42:0/885426664 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f665411c9f0 0x7f66541347d0 secure :-1 s=CLOSED pgs=2364 cs=0 l=1 rev1=1 crypto rx=0x7f664c007fd0 tx=0x7f664c001260 comp rx=0 tx=0).stop 2024-09-16T03:45:22.360 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:45:22.360 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:45:22.360 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:45:22.361 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T03:45:22.361 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 28, 2024-09-16T03:45:22.361 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 201707, 2024-09-16T03:45:22.361 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:45:22.361 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:45:22.361 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T03:45:22.361 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T03:45:22.361 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:45:22.361 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 3, 2024-09-16T03:45:22.361 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:45:22.361 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:45:22.361 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:45:22.361 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:45:22.362 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:45:22.362 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:45:22.362 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:45:22.362 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:45:22.362 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:45:22.362 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:45:22.362 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:45:22.362 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:45:22.362 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:45:22.362 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:45:22.362 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:45:22.362 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:45:22.362 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:45:22.362 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:45:22.363 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:45:22.363 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:45:22.363 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:45:22.363 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:45:22.363 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:45:22.363 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:45:22.363 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:45:22.363 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:45:22.363 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:45:22.363 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:45:22.363 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:45:22.363 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:45:22.363 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:45:22.363 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:45:22.364 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:45:22.364 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:45:22.364 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:45:22.364 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:45:22.364 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:45:22.364 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:45:22.364 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:45:22.364 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:45:22.364 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:45:22.364 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:45:22.364 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:45:22.364 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:45:22.364 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:45:22.364 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:45:22.365 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:45:22.365 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:45:22.365 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:45:22.365 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:45:22.365 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:45:22.365 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:45:22.365 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:45:22.365 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:45:22.365 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:45:22.365 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:45:22.365 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:45:22.365 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:45:22.365 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:45:22.365 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:45:22.366 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:45:22.366 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:45:22.366 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:45:22.366 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:45:22.366 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:45:22.366 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:45:22.366 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:45:22.366 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:45:22.366 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:45:22.366 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:45:22.366 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:45:22.366 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:45:22.366 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:45:22.366 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:45:22.367 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:45:22.367 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:45:22.367 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:45:22.367 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:45:22.367 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:45:22.367 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:45:22.367 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:45:22.367 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:45:22.367 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:45:22.367 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:45:22.367 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:45:22.367 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:45:22.367 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:45:22.367 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:45:22.368 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:45:22.368 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:45:22.368 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:45:22.368 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:45:22.368 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:45:22.368 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:45:22.368 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:45:22.368 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:45:22.368 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:45:22.368 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:45:22.369 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:45:22.369 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:45:22.369 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:45:22.369 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:45:22.369 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:45:22.369 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:45:22.369 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:45:22.369 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:45:22.369 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:45:22.369 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:45:22.369 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:45:22.369 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:45:22.370 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:45:22.370 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:45:22.370 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:45:22.370 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:45:22.370 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:45:22.370 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:45:22.370 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:45:22.370 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:45:22.370 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:45:22.370 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:45:22.370 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:45:22.370 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:45:22.371 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:45:22.371 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:45:22.371 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:45:22.371 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:45:22.371 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:45:22.371 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:45:22.371 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:45:22.371 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:45:22.371 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:45:22.371 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:45:22.371 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:45:22.371 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:45:22.371 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:45:22.371 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:45:22.372 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:45:22.372 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:45:22.372 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:45:22.372 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:45:22.372 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:45:22.372 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T03:45:22.372 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:45:22.372 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:45:22.372 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:45:22.372 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 4 2024-09-16T03:45:22.372 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:45:22.372 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:45:22.372 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:45:22.372 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:45:22.373 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:45:22.373 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:45:22.373 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 8 2024-09-16T03:45:22.373 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:45:22.373 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:45:22.373 INFO:teuthology.orchestra.run.smithi042.stdout: "mgr": [ 2024-09-16T03:45:22.373 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:45:22.373 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:45:22.373 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:45:22.373 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:45:22.373 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:45:22.373 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:45:22.373 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:45:22.373 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:45:22.373 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:45:22.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.337+0000 7f66595c6640 10 librados: watch_flush enter 2024-09-16T03:45:22.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.337+0000 7f66595c6640 10 librados: watch_flush exit 2024-09-16T03:45:22.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.337+0000 7f66595c6640 20 client.13358.objecter shutdown clearing up homeless session... 2024-09-16T03:45:22.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.337+0000 7f66595c6640 10 client.13358.objecter successfully canceled tick 2024-09-16T03:45:22.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.337+0000 7f66595c6640 1 -- 172.21.15.42:0/885426664 >> v2:172.21.15.154:6812/254845342 conn(0x7f6634037810 msgr2=0x7f6634039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:45:22.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.337+0000 7f66595c6640 1 --2- 172.21.15.42:0/885426664 >> v2:172.21.15.154:6812/254845342 conn(0x7f6634037810 0x7f6634039cd0 secure :-1 s=READY pgs=3365 cs=0 l=1 rev1=1 crypto rx=0x7f6644009d30 tx=0x7f6644004740 comp rx=0 tx=0).stop 2024-09-16T03:45:22.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.337+0000 7f66595c6640 10 monclient: shutdown 2024-09-16T03:45:22.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.337+0000 7f66595c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:45:22.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.337+0000 7f66595c6640 1 -- 172.21.15.42:0/885426664 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f66540a1e20 msgr2=0x7f66541324d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:45:22.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.337+0000 7f66595c6640 1 --2- 172.21.15.42:0/885426664 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f66540a1e20 0x7f66541324d0 secure :-1 s=READY pgs=2362 cs=0 l=1 rev1=1 crypto rx=0x7f663c019080 tx=0x7f663c019210 comp rx=0 tx=0).stop 2024-09-16T03:45:22.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.337+0000 7f66537fe640 1 -- 172.21.15.42:0/885426664 reap_dead start 2024-09-16T03:45:22.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.337+0000 7f66537fe640 1 -- 172.21.15.42:0/885426664 reap_dead start 2024-09-16T03:45:22.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.337+0000 7f66595c6640 1 -- 172.21.15.42:0/885426664 shutdown_connections 2024-09-16T03:45:22.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.337+0000 7f66595c6640 1 -- 172.21.15.42:0/885426664 >> 172.21.15.42:0/885426664 conn(0x7f66540a2420 msgr2=0x7f66540a45b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:45:22.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.337+0000 7f66595c6640 1 -- 172.21.15.42:0/885426664 shutdown_connections 2024-09-16T03:45:22.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.337+0000 7f66595c6640 1 -- 172.21.15.42:0/885426664 wait complete. 2024-09-16T03:45:22.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.337+0000 7f66595c6640 1 librados: shutdown 2024-09-16T03:45:22.377 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:45:22.377 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T03:45:22.377 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T03:45:22.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.489+0000 7f197a9cf640 10 monclient: get_monmap_and_config 2024-09-16T03:45:22.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.493+0000 7f197a9cf640 10 monclient: build_initial_monmap 2024-09-16T03:45:22.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.493+0000 7f197a9cf640 10 monclient: monmap: 2024-09-16T03:45:22.497 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:45:22.497 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:45:22.497 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:45:22.496195+0000 2024-09-16T03:45:22.497 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:45:22.496195+0000 2024-09-16T03:45:22.497 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:45:22.497 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:45:22.497 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:45:22.497 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:45:22.497 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:45:22.497 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:45:22.498 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:45:22.498 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:45:22.499 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.493+0000 7f197a9cf640 1 Processor -- start 2024-09-16T03:45:22.499 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.493+0000 7f197a9cf640 1 -- start start 2024-09-16T03:45:22.500 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.493+0000 7f197a9cf640 10 monclient: init 2024-09-16T03:45:22.500 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.493+0000 7f197a9cf640 10 monclient: _reopen_session rank -1 2024-09-16T03:45:22.500 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.493+0000 7f197a9cf640 10 monclient: _add_conns ranks=[0,2,1,4,3] 2024-09-16T03:45:22.500 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.493+0000 7f197a9cf640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f19741353a0 addr v1:172.21.15.42:6789/0 2024-09-16T03:45:22.500 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.493+0000 7f197a9cf640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1974138160 0x7f197413a550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:45:22.500 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.493+0000 7f197a9cf640 10 monclient(hunting): picked mon.noname-b con 0x7f1974138160 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:45:22.500 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.493+0000 7f197a9cf640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f197413aa90 addr v1:172.21.15.154:6789/0 2024-09-16T03:45:22.500 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.493+0000 7f197a9cf640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f19740a36b0 con 0x7f19741353a0 2024-09-16T03:45:22.500 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.493+0000 7f197a9cf640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f19740a0260 con 0x7f197413aa90 2024-09-16T03:45:22.500 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.493+0000 7f197a9cf640 10 monclient(hunting): start opening mon connection 2024-09-16T03:45:22.500 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.493+0000 7f197a9cf640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f1974134e50 con 0x7f1974138160 2024-09-16T03:45:22.500 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.493+0000 7f197a9cf640 10 monclient(hunting): _renew_subs 2024-09-16T03:45:22.500 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.493+0000 7f197a9cf640 10 monclient(hunting): authenticate will time out at 1169.465543s 2024-09-16T03:45:22.501 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.493+0000 7f19737fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1974138160 0x7f197413a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:45:22.501 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.493+0000 7f1973fff640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f19741353a0 0x7f1974135780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:55290/0 (socket says 172.21.15.42:55290) 2024-09-16T03:45:22.501 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.493+0000 7f19737fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1974138160 0x7f197413a550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:38750/0 (socket says 172.21.15.42:38750) 2024-09-16T03:45:22.501 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.493+0000 7f1973fff640 1 -- 172.21.15.42:0/1792406696 learned_addr learned my addr 172.21.15.42:0/1792406696 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:45:22.501 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.493+0000 7f19737fe640 10 monclient(hunting): get_auth_request con 0x7f1974138160 auth_method 0 2024-09-16T03:45:22.502 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.493+0000 7f19737fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:45:22.502 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.493+0000 7f19737fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:45:22.502 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.493+0000 7f19737fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:45:22.502 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.493+0000 7f19737fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:45:22.502 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.493+0000 7f19737fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:45:22.502 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.493+0000 7f19737fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:45:22.502 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.493+0000 7f19737fe640 10 monclient(hunting): handle_auth_done global_id 13367 payload 274 2024-09-16T03:45:22.502 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.493+0000 7f19737fe640 1 -- 172.21.15.42:0/1792406696 >> v1:172.21.15.154:6789/0 conn(0x7f197413aa90 legacy=0x7f197413ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:45:22.502 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.493+0000 7f19737fe640 1 -- 172.21.15.42:0/1792406696 >> v1:172.21.15.42:6789/0 conn(0x7f19741353a0 legacy=0x7f1974135780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:45:22.502 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.493+0000 7f19737fe640 10 monclient: _finish_hunting 0 2024-09-16T03:45:22.502 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.493+0000 7f19737fe640 1 monclient: found mon.noname-b 2024-09-16T03:45:22.502 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.493+0000 7f19737fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:45:22.502 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.493+0000 7f19737fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T03:45:22.502 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.493+0000 7f1972ffd640 1 -- 172.21.15.42:0/1792406696 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2594888225 0 0) 0x7f19740a36b0 con 0x7f19741353a0 2024-09-16T03:45:22.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.493+0000 7f19737fe640 1 -- 172.21.15.42:0/1792406696 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19740a1e20 con 0x7f1974138160 2024-09-16T03:45:22.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.493+0000 7f1972ffd640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T03:45:22.509 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.497+0000 7f19737fe640 1 --2- 172.21.15.42:0/1792406696 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1974138160 0x7f197413a550 secure :-1 s=READY pgs=2365 cs=0 l=1 rev1=1 crypto rx=0x7f19680097f0 tx=0x7f196801c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5d4c595d89ce13d3 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:45:22.509 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.497+0000 7f1972ffd640 1 -- 172.21.15.42:0/1792406696 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19680b0070 con 0x7f1974138160 2024-09-16T03:45:22.509 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.497+0000 7f1972ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:45:22.509 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.497+0000 7f1972ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:45:22.509 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.497+0000 7f1972ffd640 10 monclient: dump: 2024-09-16T03:45:22.509 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:45:22.509 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:45:22.509 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:22.510 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:22.510 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:45:22.510 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:45:22.510 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:45:22.510 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:45:22.510 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:45:22.510 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:45:22.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.497+0000 7f1972ffd640 10 monclient: _finish_auth 0 2024-09-16T03:45:22.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.497+0000 7f1972ffd640 10 monclient: _check_auth_tickets 2024-09-16T03:45:22.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.497+0000 7f1972ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:45:22.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.497+0000 7f197a9cf640 5 monclient: authenticate success, global_id 13367 2024-09-16T03:45:22.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.497+0000 7f1972ffd640 1 -- 172.21.15.42:0/1792406696 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f19680ab600 con 0x7f1974138160 2024-09-16T03:45:22.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.497+0000 7f197a9cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:45:22.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.497+0000 7f1972ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:45:22.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.497+0000 7f197a9cf640 10 monclient: get_monmap_and_config success 2024-09-16T03:45:22.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.497+0000 7f197a9cf640 4 set_mon_vals no callback set 2024-09-16T03:45:22.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.497+0000 7f1972ffd640 1 -- 172.21.15.42:0/1792406696 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19680ab9d0 con 0x7f1974138160 2024-09-16T03:45:22.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.497+0000 7f1972ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:45:22.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.497+0000 7f1972ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:45:22.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.497+0000 7f1972ffd640 10 monclient: dump: 2024-09-16T03:45:22.511 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:45:22.511 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:45:22.511 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:22.511 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:22.511 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:45:22.511 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:45:22.511 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:45:22.512 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:45:22.512 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:45:22.512 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:45:22.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.497+0000 7f197a9cf640 10 monclient: shutdown 2024-09-16T03:45:22.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.497+0000 7f197a9cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:45:22.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.497+0000 7f197a9cf640 1 -- 172.21.15.42:0/1792406696 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1974138160 msgr2=0x7f197413a550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:45:22.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.497+0000 7f197a9cf640 1 --2- 172.21.15.42:0/1792406696 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1974138160 0x7f197413a550 secure :-1 s=READY pgs=2365 cs=0 l=1 rev1=1 crypto rx=0x7f19680097f0 tx=0x7f196801c3a0 comp rx=0 tx=0).stop 2024-09-16T03:45:22.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.497+0000 7f197a9cf640 1 -- 172.21.15.42:0/1792406696 shutdown_connections 2024-09-16T03:45:22.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.497+0000 7f197a9cf640 1 --2- 172.21.15.42:0/1792406696 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1974138160 0x7f197413a550 unknown :-1 s=CLOSED pgs=2365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:45:22.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.497+0000 7f197a9cf640 1 -- 172.21.15.42:0/1792406696 >> 172.21.15.42:0/1792406696 conn(0x7f19740a6640 msgr2=0x7f19740a2b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:45:22.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.497+0000 7f197a9cf640 1 -- 172.21.15.42:0/1792406696 shutdown_connections 2024-09-16T03:45:22.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.497+0000 7f197a9cf640 1 -- 172.21.15.42:0/1792406696 wait complete. 2024-09-16T03:45:22.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.497+0000 7f197a9cf640 10 monclient: build_initial_monmap 2024-09-16T03:45:22.513 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.497+0000 7f197a9cf640 10 monclient: monmap: 2024-09-16T03:45:22.513 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:45:22.513 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:45:22.513 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:45:22.513 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:45:22.513 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:45:22.513 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:45:22.513 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:45:22.513 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:45:22.513 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:45:22.513 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:45:22.513 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.497+0000 7f197a9cf640 1 librados: starting msgr at 2024-09-16T03:45:22.513 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.497+0000 7f197a9cf640 1 librados: starting objecter 2024-09-16T03:45:22.513 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.497+0000 7f197a9cf640 1 Processor -- start 2024-09-16T03:45:22.514 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.497+0000 7f197a9cf640 1 -- start start 2024-09-16T03:45:22.514 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.497+0000 7f197a9cf640 1 librados: setting wanted keys 2024-09-16T03:45:22.514 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.497+0000 7f197a9cf640 1 librados: calling monclient init 2024-09-16T03:45:22.514 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.497+0000 7f197a9cf640 10 monclient: init 2024-09-16T03:45:22.514 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.497+0000 7f197a9cf640 10 monclient: _reopen_session rank -1 2024-09-16T03:45:22.514 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.497+0000 7f197a9cf640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:45:22.514 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.497+0000 7f197a9cf640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f19741353a0 0x7f19740728d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:45:22.514 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.497+0000 7f197a9cf640 10 monclient(hunting): picked mon.noname-c con 0x7f19741353a0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:45:22.514 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.497+0000 7f197a9cf640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1974138160 0x7f1974072e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:45:22.514 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.497+0000 7f197a9cf640 10 monclient(hunting): picked mon.noname-a con 0x7f1974138160 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:45:22.514 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.497+0000 7f197a9cf640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f197413aa90 0x7f197406f860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:45:22.514 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.497+0000 7f197a9cf640 10 monclient(hunting): picked mon.noname-b con 0x7f197413aa90 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:45:22.514 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.497+0000 7f197a9cf640 10 monclient(hunting): start opening mon connection 2024-09-16T03:45:22.514 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.497+0000 7f197a9cf640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f1974140f40 con 0x7f1974138160 2024-09-16T03:45:22.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.497+0000 7f197a9cf640 10 monclient(hunting): start opening mon connection 2024-09-16T03:45:22.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.497+0000 7f197a9cf640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f1974140dc0 con 0x7f197413aa90 2024-09-16T03:45:22.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.497+0000 7f197a9cf640 10 monclient(hunting): start opening mon connection 2024-09-16T03:45:22.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.497+0000 7f197a9cf640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f19741410c0 con 0x7f19741353a0 2024-09-16T03:45:22.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.497+0000 7f197a9cf640 10 monclient(hunting): _renew_subs 2024-09-16T03:45:22.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.497+0000 7f197a9cf640 10 monclient(hunting): authenticate will time out at 1169.470554s 2024-09-16T03:45:22.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.497+0000 7f1973fff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f19741353a0 0x7f19740728d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:45:22.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.497+0000 7f19737fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1974138160 0x7f1974072e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:45:22.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.497+0000 7f1973fff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f19741353a0 0x7f19740728d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:38752/0 (socket says 172.21.15.42:38752) 2024-09-16T03:45:22.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.497+0000 7f1973fff640 1 -- 172.21.15.42:0/3329438034 learned_addr learned my addr 172.21.15.42:0/3329438034 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:45:22.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.497+0000 7f1973fff640 10 monclient(hunting): get_auth_request con 0x7f19741353a0 auth_method 0 2024-09-16T03:45:22.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.497+0000 7f1973fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:45:22.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.497+0000 7f1973fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:45:22.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.497+0000 7f1973fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:45:22.516 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.497+0000 7f19737fe640 10 monclient(hunting): get_auth_request con 0x7f1974138160 auth_method 0 2024-09-16T03:45:22.516 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.497+0000 7f19737fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:45:22.516 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.497+0000 7f19737fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:45:22.516 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.497+0000 7f19737fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:45:22.516 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.497+0000 7f19737fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:45:22.516 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.497+0000 7f19737fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:45:22.516 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.497+0000 7f19737fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:45:22.516 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.497+0000 7f1973fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:45:22.516 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.497+0000 7f1973fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:45:22.516 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.497+0000 7f1973fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:45:22.516 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.497+0000 7f1978f6c640 1 --2- 172.21.15.42:0/3329438034 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f197413aa90 0x7f197406f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:45:22.516 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.497+0000 7f19737fe640 10 monclient(hunting): handle_auth_done global_id 744195 payload 824 2024-09-16T03:45:22.516 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.497+0000 7f19737fe640 1 -- 172.21.15.42:0/3329438034 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f19741353a0 msgr2=0x7f19740728d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:45:22.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.497+0000 7f19737fe640 1 --2- 172.21.15.42:0/3329438034 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f19741353a0 0x7f19740728d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:45:22.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.497+0000 7f19737fe640 1 -- 172.21.15.42:0/3329438034 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f197413aa90 msgr2=0x7f197406f860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:45:22.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.497+0000 7f19737fe640 1 --2- 172.21.15.42:0/3329438034 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f197413aa90 0x7f197406f860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:45:22.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.497+0000 7f19737fe640 10 monclient: _finish_hunting 0 2024-09-16T03:45:22.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.497+0000 7f19737fe640 1 monclient: found mon.noname-a 2024-09-16T03:45:22.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.497+0000 7f19737fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:45:22.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.497+0000 7f19737fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:45:22.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.497+0000 7f19737fe640 1 -- 172.21.15.42:0/3329438034 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f197409d230 con 0x7f1974138160 2024-09-16T03:45:22.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.497+0000 7f1978f6c640 10 monclient: get_auth_request con 0x7f197413aa90 auth_method 0 2024-09-16T03:45:22.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.497+0000 7f1978f6c640 1 --2- 172.21.15.42:0/3329438034 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f197413aa90 0x7f197406f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T03:45:22.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.497+0000 7f1973fff640 1 --2- 172.21.15.42:0/3329438034 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f19741353a0 0x7f19740728d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:45:22.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.497+0000 7f19737fe640 1 --2- 172.21.15.42:0/3329438034 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1974138160 0x7f1974072e10 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f19680ab000 tx=0x7f1968019140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:45:22.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.497+0000 7f1970ff9640 10 client.?.objecter ms_handle_connect 0x7f1974138160 2024-09-16T03:45:22.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.497+0000 7f1970ff9640 10 client.?.objecter resend_mon_ops 2024-09-16T03:45:22.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.497+0000 7f1970ff9640 1 -- 172.21.15.42:0/3329438034 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19680b8020 con 0x7f1974138160 2024-09-16T03:45:22.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.497+0000 7f1970ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:45:22.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.497+0000 7f1970ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:45:22.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.497+0000 7f1970ff9640 10 monclient: dump: 2024-09-16T03:45:22.518 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:45:22.518 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:45:22.518 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:22.518 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:22.518 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:45:22.518 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:45:22.518 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:45:22.518 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:45:22.518 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:45:22.518 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:45:22.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.497+0000 7f1970ff9640 10 monclient: _finish_auth 0 2024-09-16T03:45:22.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.497+0000 7f1970ff9640 10 monclient: _check_auth_tickets 2024-09-16T03:45:22.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.497+0000 7f1970ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:45:22.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.497+0000 7f1970ff9640 1 -- 172.21.15.42:0/3329438034 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f19680b4610 con 0x7f1974138160 2024-09-16T03:45:22.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.497+0000 7f197a9cf640 5 monclient: authenticate success, global_id 744195 2024-09-16T03:45:22.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.497+0000 7f197a9cf640 10 monclient: _renew_subs 2024-09-16T03:45:22.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.497+0000 7f197a9cf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:45:22.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.497+0000 7f197a9cf640 1 -- 172.21.15.42:0/3329438034 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f197413a880 con 0x7f1974138160 2024-09-16T03:45:22.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.497+0000 7f1970ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T03:45:22.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.497+0000 7f1970ff9640 1 -- 172.21.15.42:0/3329438034 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19680b0040 con 0x7f1974138160 2024-09-16T03:45:22.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.497+0000 7f1970ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:45:22.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.497+0000 7f1970ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:45:22.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.497+0000 7f1970ff9640 10 monclient: dump: 2024-09-16T03:45:22.519 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:45:22.520 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:45:22.520 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:22.520 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:22.520 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:45:22.520 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:45:22.520 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:45:22.520 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:45:22.520 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:45:22.520 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:45:22.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.497+0000 7f1971ffb640 4 set_mon_vals no callback set 2024-09-16T03:45:22.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.497+0000 7f197a9cf640 10 client.744195.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:45:22.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.497+0000 7f197a9cf640 10 monclient: _renew_subs 2024-09-16T03:45:22.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.497+0000 7f197a9cf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:45:22.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.497+0000 7f197a9cf640 1 -- 172.21.15.42:0/3329438034 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f197406eba0 con 0x7f1974138160 2024-09-16T03:45:22.521 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.497+0000 7f197a9cf640 1 librados: init done 2024-09-16T03:45:22.521 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.501+0000 7f1970ff9640 1 -- 172.21.15.42:0/3329438034 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f19680bb020 con 0x7f1974138160 2024-09-16T03:45:22.521 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.501+0000 7f197a9cf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:45:22.521 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.501+0000 7f197a9cf640 1 --2- 172.21.15.42:0/3329438034 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1974070560 0x7f1974070940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:45:22.521 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.501+0000 7f197a9cf640 10 monclient(hunting): start opening mon connection 2024-09-16T03:45:22.521 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.501+0000 7f197a9cf640 1 -- 172.21.15.42:0/3329438034 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f1974135780 con 0x7f1974070560 2024-09-16T03:45:22.521 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.501+0000 7f1970ff9640 1 --2- 172.21.15.42:0/3329438034 >> v2:172.21.15.154:6812/254845342 conn(0x7f1960037810 0x7f1960039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:45:22.521 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.501+0000 7f1970ff9640 1 -- 172.21.15.42:0/3329438034 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 7319+0+0 (secure 0 0 0) 0x7f19680e32d0 con 0x7f1974138160 2024-09-16T03:45:22.521 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.501+0000 7f1970ff9640 10 client.744195.objecter ms_dispatch 0x7f1974000ba0 osd_map(90..90 src has 1..90) 2024-09-16T03:45:22.521 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.501+0000 7f1970ff9640 3 client.744195.objecter handle_osd_map got epochs [90,90] > 0 2024-09-16T03:45:22.521 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.501+0000 7f1970ff9640 3 client.744195.objecter handle_osd_map decoding full epoch 90 2024-09-16T03:45:22.521 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.501+0000 7f1970ff9640 20 client.744195.objecter dump_active .. 0 homeless 2024-09-16T03:45:22.521 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.501+0000 7f1973fff640 1 --2- 172.21.15.42:0/3329438034 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1974070560 0x7f1974070940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:45:22.522 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.501+0000 7f1973fff640 10 monclient: get_auth_request con 0x7f1974070560 auth_method 0 2024-09-16T03:45:22.522 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.501+0000 7f1973fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:45:22.522 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.501+0000 7f1973fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:45:22.522 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.501+0000 7f1973fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:45:22.522 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.501+0000 7f1978f6c640 1 --2- 172.21.15.42:0/3329438034 >> v2:172.21.15.154:6812/254845342 conn(0x7f1960037810 0x7f1960039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:45:22.522 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.501+0000 7f1978f6c640 10 monclient: get_auth_request con 0x7f1960037810 auth_method 0 2024-09-16T03:45:22.522 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.501+0000 7f1973fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:45:22.522 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.501+0000 7f1973fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:45:22.522 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.501+0000 7f1973fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:45:22.522 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.501+0000 7f1973fff640 10 monclient(hunting): handle_auth_done global_id 272755 payload 824 2024-09-16T03:45:22.522 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.501+0000 7f1973fff640 1 -- 172.21.15.42:0/3329438034 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f19740a3450 con 0x7f1974070560 2024-09-16T03:45:22.522 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.501+0000 7f1978f6c640 1 --2- 172.21.15.42:0/3329438034 >> v2:172.21.15.154:6812/254845342 conn(0x7f1960037810 0x7f1960039cd0 secure :-1 s=READY pgs=3366 cs=0 l=1 rev1=1 crypto rx=0x7f196c007fd0 tx=0x7f196c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:45:22.522 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.501+0000 7f1970ff9640 10 client.744195.objecter ms_handle_connect 0x7f1960037810 2024-09-16T03:45:22.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.501+0000 7f1970ff9640 10 client.744195.objecter ms_handle_connect 0x7f1974070560 2024-09-16T03:45:22.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.501+0000 7f1970ff9640 10 client.744195.objecter resend_mon_ops 2024-09-16T03:45:22.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.501+0000 7f1973fff640 1 --2- 172.21.15.42:0/3329438034 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1974070560 0x7f1974070940 secure :-1 s=READY pgs=2862 cs=0 l=1 rev1=1 crypto rx=0x7f1964004a20 tx=0x7f1964004ee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:45:22.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.501+0000 7f1970ff9640 1 -- 172.21.15.42:0/3329438034 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f196400fb30 con 0x7f1974070560 2024-09-16T03:45:22.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.501+0000 7f1970ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:45:22.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.501+0000 7f1970ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:45:22.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.501+0000 7f1970ff9640 10 monclient: dump: 2024-09-16T03:45:22.523 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:45:22.523 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:45:22.523 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:22.523 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:22.523 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:45:22.523 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:45:22.523 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:45:22.524 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:45:22.524 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:45:22.524 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:45:22.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.501+0000 7f1970ff9640 1 -- 172.21.15.42:0/3329438034 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f1964018070 con 0x7f1974070560 2024-09-16T03:45:22.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.501+0000 7f1970ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:45:22.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.501+0000 7f1970ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:45:22.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.501+0000 7f1970ff9640 1 -- 172.21.15.42:0/3329438034 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1974070560 msgr2=0x7f1974070940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:45:22.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.501+0000 7f1970ff9640 1 --2- 172.21.15.42:0/3329438034 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1974070560 0x7f1974070940 secure :-1 s=READY pgs=2862 cs=0 l=1 rev1=1 crypto rx=0x7f1964004a20 tx=0x7f1964004ee0 comp rx=0 tx=0).stop 2024-09-16T03:45:22.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.501+0000 7f1970ff9640 1 -- 172.21.15.42:0/3329438034 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1974070560 msgr2=0x7f1974070940 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:45:22.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.501+0000 7f1970ff9640 1 --2- 172.21.15.42:0/3329438034 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1974070560 0x7f1974070940 secure :-1 s=CLOSED pgs=2862 cs=0 l=1 rev1=1 crypto rx=0x7f1964004a20 tx=0x7f1964004ee0 comp rx=0 tx=0).stop 2024-09-16T03:45:22.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.509+0000 7f197a9cf640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:45:22.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.509+0000 7f197a9cf640 1 --2- 172.21.15.42:0/3329438034 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f197402c650 0x7f197402ca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:45:22.527 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.509+0000 7f197a9cf640 10 monclient(hunting): start opening mon connection 2024-09-16T03:45:22.528 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.509+0000 7f197a9cf640 1 -- 172.21.15.42:0/3329438034 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f1974137630 con 0x7f197402c650 2024-09-16T03:45:22.528 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.509+0000 7f1973fff640 1 --2- 172.21.15.42:0/3329438034 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f197402c650 0x7f197402ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:45:22.528 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.509+0000 7f1973fff640 10 monclient: get_auth_request con 0x7f197402c650 auth_method 0 2024-09-16T03:45:22.528 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.509+0000 7f1973fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:45:22.528 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.509+0000 7f1973fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:45:22.528 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.509+0000 7f1973fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:45:22.528 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.509+0000 7f1973fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:45:22.528 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.509+0000 7f1973fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:45:22.528 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.509+0000 7f1973fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:45:22.528 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.509+0000 7f1973fff640 10 monclient(hunting): handle_auth_done global_id 272758 payload 824 2024-09-16T03:45:22.528 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.509+0000 7f1973fff640 1 -- 172.21.15.42:0/3329438034 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f197406e870 con 0x7f197402c650 2024-09-16T03:45:22.528 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.509+0000 7f1973fff640 1 --2- 172.21.15.42:0/3329438034 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f197402c650 0x7f197402ca30 secure :-1 s=READY pgs=2863 cs=0 l=1 rev1=1 crypto rx=0x7f1964004a20 tx=0x7f1964007db0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:45:22.528 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.509+0000 7f1970ff9640 10 client.744195.objecter ms_handle_connect 0x7f197402c650 2024-09-16T03:45:22.529 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.509+0000 7f1970ff9640 10 client.744195.objecter resend_mon_ops 2024-09-16T03:45:22.529 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.509+0000 7f1970ff9640 1 -- 172.21.15.42:0/3329438034 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19640136c0 con 0x7f197402c650 2024-09-16T03:45:22.529 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.509+0000 7f1970ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:45:22.529 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.509+0000 7f1970ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:45:22.529 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.509+0000 7f1970ff9640 10 monclient: dump: 2024-09-16T03:45:22.529 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:45:22.529 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:45:22.529 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:22.529 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:22.529 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:45:22.530 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:45:22.530 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:45:22.530 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:45:22.530 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:45:22.530 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:45:22.530 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.509+0000 7f1970ff9640 1 -- 172.21.15.42:0/3329438034 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4711 (secure 0 0 0) 0x7f19640136c0 con 0x7f197402c650 2024-09-16T03:45:22.530 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.509+0000 7f1970ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:45:22.530 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.509+0000 7f1970ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:45:22.530 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.509+0000 7f1970ff9640 1 -- 172.21.15.42:0/3329438034 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f197402c650 msgr2=0x7f197402ca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:45:22.530 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.509+0000 7f1970ff9640 1 --2- 172.21.15.42:0/3329438034 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f197402c650 0x7f197402ca30 secure :-1 s=READY pgs=2863 cs=0 l=1 rev1=1 crypto rx=0x7f1964004a20 tx=0x7f1964007db0 comp rx=0 tx=0).stop 2024-09-16T03:45:22.530 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.509+0000 7f1970ff9640 1 -- 172.21.15.42:0/3329438034 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f197402c650 msgr2=0x7f197402ca30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:45:22.530 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.509+0000 7f1970ff9640 1 --2- 172.21.15.42:0/3329438034 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f197402c650 0x7f197402ca30 unknown :-1 s=CLOSED pgs=2863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:45:22.531 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:45:22.531 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:45:22.531 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:45:22.531 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T03:45:22.531 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 28, 2024-09-16T03:45:22.531 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 65237, 2024-09-16T03:45:22.531 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:45:22.531 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:45:22.531 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T03:45:22.531 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T03:45:22.531 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:45:22.531 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 3, 2024-09-16T03:45:22.532 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:45:22.532 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:45:22.532 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:45:22.532 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:45:22.532 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:45:22.532 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:45:22.532 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:45:22.532 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:45:22.532 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:45:22.532 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:45:22.532 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:45:22.532 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:45:22.532 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:45:22.532 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:45:22.533 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:45:22.533 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:45:22.533 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:45:22.533 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:45:22.533 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:45:22.533 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:45:22.533 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:45:22.533 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:45:22.533 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:45:22.533 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:45:22.533 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:45:22.533 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:45:22.533 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:45:22.533 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:45:22.534 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:45:22.534 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:45:22.534 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:45:22.534 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:45:22.534 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:45:22.534 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:45:22.534 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:45:22.534 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:45:22.534 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:45:22.534 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:45:22.534 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:45:22.534 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:45:22.534 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:45:22.534 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:45:22.535 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:45:22.535 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:45:22.535 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:45:22.535 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:45:22.535 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:45:22.535 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:45:22.535 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:45:22.535 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:45:22.535 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:45:22.535 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:45:22.535 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:45:22.535 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:45:22.535 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:45:22.535 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:45:22.536 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:45:22.536 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:45:22.536 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:45:22.536 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:45:22.536 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:45:22.536 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:45:22.536 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:45:22.536 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:45:22.536 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:45:22.536 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:45:22.536 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:45:22.536 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:45:22.536 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:45:22.536 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:45:22.537 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:45:22.537 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:45:22.537 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:45:22.537 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:45:22.537 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:45:22.537 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:45:22.537 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:45:22.537 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:45:22.537 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:45:22.537 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:45:22.537 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:45:22.537 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:45:22.537 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:45:22.538 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:45:22.538 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:45:22.538 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:45:22.538 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:45:22.538 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:45:22.538 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:45:22.538 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:45:22.538 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:45:22.538 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:45:22.538 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:45:22.538 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:45:22.538 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:45:22.538 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:45:22.538 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:45:22.538 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:45:22.539 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:45:22.539 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:45:22.539 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:45:22.539 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:45:22.539 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:45:22.539 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:45:22.539 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:45:22.539 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:45:22.539 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:45:22.539 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:45:22.539 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:45:22.539 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:45:22.539 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:45:22.539 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:45:22.540 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:45:22.540 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:45:22.540 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:45:22.540 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:45:22.540 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:45:22.540 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:45:22.540 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:45:22.540 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:45:22.540 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:45:22.540 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:45:22.540 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:45:22.540 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:45:22.540 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:45:22.540 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:45:22.541 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:45:22.541 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:45:22.541 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:45:22.541 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:45:22.541 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:45:22.541 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:45:22.541 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:45:22.541 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:45:22.541 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:45:22.541 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:45:22.541 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:45:22.541 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:45:22.541 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:45:22.542 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:45:22.542 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:45:22.542 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T03:45:22.542 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:45:22.542 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:45:22.542 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:45:22.542 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:45:22.542 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:45:22.542 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:45:22.542 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:45:22.542 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:45:22.542 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:45:22.542 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:45:22.542 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 7 2024-09-16T03:45:22.542 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:45:22.543 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:45:22.543 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:45:22.543 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:45:22.543 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:45:22.545 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.509+0000 7f197a9cf640 10 librados: watch_flush enter 2024-09-16T03:45:22.545 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.509+0000 7f197a9cf640 10 librados: watch_flush exit 2024-09-16T03:45:22.545 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.509+0000 7f197a9cf640 20 client.744195.objecter shutdown clearing up homeless session... 2024-09-16T03:45:22.545 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.509+0000 7f197a9cf640 10 client.744195.objecter successfully canceled tick 2024-09-16T03:45:22.545 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.509+0000 7f197a9cf640 1 -- 172.21.15.42:0/3329438034 >> v2:172.21.15.154:6812/254845342 conn(0x7f1960037810 msgr2=0x7f1960039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:45:22.545 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.509+0000 7f197a9cf640 1 --2- 172.21.15.42:0/3329438034 >> v2:172.21.15.154:6812/254845342 conn(0x7f1960037810 0x7f1960039cd0 secure :-1 s=READY pgs=3366 cs=0 l=1 rev1=1 crypto rx=0x7f196c007fd0 tx=0x7f196c009040 comp rx=0 tx=0).stop 2024-09-16T03:45:22.545 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.509+0000 7f197a9cf640 10 monclient: shutdown 2024-09-16T03:45:22.545 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.509+0000 7f197a9cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:45:22.545 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.509+0000 7f197a9cf640 1 -- 172.21.15.42:0/3329438034 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1974138160 msgr2=0x7f1974072e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:45:22.545 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.509+0000 7f197a9cf640 1 --2- 172.21.15.42:0/3329438034 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1974138160 0x7f1974072e10 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f19680ab000 tx=0x7f1968019140 comp rx=0 tx=0).stop 2024-09-16T03:45:22.545 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.509+0000 7f1978f6c640 1 -- 172.21.15.42:0/3329438034 reap_dead start 2024-09-16T03:45:22.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.513+0000 7f1978f6c640 1 -- 172.21.15.42:0/3329438034 reap_dead start 2024-09-16T03:45:22.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.513+0000 7f197a9cf640 1 -- 172.21.15.42:0/3329438034 shutdown_connections 2024-09-16T03:45:22.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.513+0000 7f197a9cf640 1 -- 172.21.15.42:0/3329438034 >> 172.21.15.42:0/3329438034 conn(0x7f19740a6640 msgr2=0x7f19741346f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:45:22.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.513+0000 7f197a9cf640 1 -- 172.21.15.42:0/3329438034 shutdown_connections 2024-09-16T03:45:22.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.513+0000 7f197a9cf640 1 -- 172.21.15.42:0/3329438034 wait complete. 2024-09-16T03:45:22.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.513+0000 7f197a9cf640 1 librados: shutdown 2024-09-16T03:45:22.546 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:45:22.546 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-09-16T03:45:22.546 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-16T03:45:22.547 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-09-16T03:45:22.547 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-16T03:45:22.547 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-16T03:45:22.547 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T03:45:22.567 INFO:tasks.ceph.mon.b:Stopped 2024-09-16T03:45:22.567 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T03:45:22.567 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:45:22.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.677+0000 7f45d8c4b640 10 monclient: get_monmap_and_config 2024-09-16T03:45:22.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.677+0000 7f45d8c4b640 10 monclient: build_initial_monmap 2024-09-16T03:45:22.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.677+0000 7f45d8c4b640 10 monclient: monmap: 2024-09-16T03:45:22.681 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:45:22.681 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:45:22.681 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:45:22.680607+0000 2024-09-16T03:45:22.682 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:45:22.680607+0000 2024-09-16T03:45:22.682 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:45:22.682 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:45:22.682 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:45:22.682 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:45:22.682 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:45:22.682 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:45:22.682 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:45:22.682 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:45:22.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.677+0000 7f45d8c4b640 1 Processor -- start 2024-09-16T03:45:22.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.677+0000 7f45d8c4b640 1 -- start start 2024-09-16T03:45:22.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.677+0000 7f45d8c4b640 10 monclient: init 2024-09-16T03:45:22.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.677+0000 7f45d8c4b640 10 monclient: _reopen_session rank -1 2024-09-16T03:45:22.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.677+0000 7f45d8c4b640 10 monclient: _add_conns ranks=[2,4,0,1,3] 2024-09-16T03:45:22.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.677+0000 7f45d8c4b640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f45d40a1330 0x7f45d40a1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:45:22.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.677+0000 7f45d8c4b640 10 monclient(hunting): picked mon.noname-b con 0x7f45d40a1330 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:45:22.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.677+0000 7f45d8c4b640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f45d406dac0 0x7f45d40a1c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:45:22.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.677+0000 7f45d8c4b640 10 monclient(hunting): picked mon.noname-c con 0x7f45d406dac0 addr v2:172.21.15.154:3300/0 2024-09-16T03:45:22.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.677+0000 7f45d8c4b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f45d4135100 addr v1:172.21.15.42:6789/0 2024-09-16T03:45:22.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.677+0000 7f45d8c4b640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f45d409c7d0 con 0x7f45d4135100 2024-09-16T03:45:22.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.677+0000 7f45d8c4b640 10 monclient(hunting): start opening mon connection 2024-09-16T03:45:22.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.677+0000 7f45d8c4b640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f45d409c090 con 0x7f45d406dac0 2024-09-16T03:45:22.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.677+0000 7f45d8c4b640 10 monclient(hunting): start opening mon connection 2024-09-16T03:45:22.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.677+0000 7f45d8c4b640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f45d409f610 con 0x7f45d40a1330 2024-09-16T03:45:22.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.677+0000 7f45d8c4b640 10 monclient(hunting): _renew_subs 2024-09-16T03:45:22.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.677+0000 7f45d8c4b640 10 monclient(hunting): authenticate will time out at 1169.650033s 2024-09-16T03:45:22.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.677+0000 7f45d259c640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f45d40a1330 0x7f45d40a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:45:22.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.677+0000 7f45d2d9d640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f45d4135100 0x7f45d4135550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:55302/0 (socket says 172.21.15.42:55302) 2024-09-16T03:45:22.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.677+0000 7f45d2d9d640 1 -- 172.21.15.42:0/2579195779 learned_addr learned my addr 172.21.15.42:0/2579195779 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:45:22.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.677+0000 7f45d259c640 10 monclient(hunting): get_auth_request con 0x7f45d40a1330 auth_method 0 2024-09-16T03:45:22.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.677+0000 7f45d259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:45:22.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.677+0000 7f45d259c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:45:22.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.677+0000 7f45d259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:45:22.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.677+0000 7f45d1d9b640 1 -- 172.21.15.42:0/2579195779 >> v2:172.21.15.154:3300/0 conn(0x7f45d406dac0 msgr2=0x7f45d40a1c50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T03:45:22.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.677+0000 7f45d1d9b640 1 --2- 172.21.15.42:0/2579195779 >> v2:172.21.15.154:3300/0 conn(0x7f45d406dac0 0x7f45d40a1c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:45:22.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.677+0000 7f45d259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:45:22.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.677+0000 7f45d259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:45:22.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.677+0000 7f45d259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:45:22.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.677+0000 7f45d259c640 10 monclient(hunting): handle_auth_done global_id 13373 payload 274 2024-09-16T03:45:22.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.677+0000 7f45d159a640 1 -- 172.21.15.42:0/2579195779 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 643202409 0 0) 0x7f45d409c7d0 con 0x7f45d4135100 2024-09-16T03:45:22.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.677+0000 7f45d259c640 1 -- 172.21.15.42:0/2579195779 >> v2:172.21.15.154:3300/0 conn(0x7f45d406dac0 msgr2=0x7f45d40a1c50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:45:22.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.677+0000 7f45d259c640 1 --2- 172.21.15.42:0/2579195779 >> v2:172.21.15.154:3300/0 conn(0x7f45d406dac0 0x7f45d40a1c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:45:22.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.677+0000 7f45d259c640 1 -- 172.21.15.42:0/2579195779 >> v1:172.21.15.42:6789/0 conn(0x7f45d4135100 legacy=0x7f45d4135550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:45:22.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.677+0000 7f45d259c640 10 monclient: _finish_hunting 0 2024-09-16T03:45:22.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.677+0000 7f45d259c640 1 monclient: found mon.noname-b 2024-09-16T03:45:22.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.677+0000 7f45d259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:45:22.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.677+0000 7f45d259c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T03:45:22.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.677+0000 7f45d259c640 1 -- 172.21.15.42:0/2579195779 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45d409f8f0 con 0x7f45d40a1330 2024-09-16T03:45:22.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.677+0000 7f45d159a640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T03:45:22.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.681+0000 7f45d259c640 1 --2- 172.21.15.42:0/2579195779 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f45d40a1330 0x7f45d40a1710 secure :-1 s=READY pgs=2372 cs=0 l=1 rev1=1 crypto rx=0x7f45c40097c0 tx=0x7f45c401c4a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=11e2ba6bb0ef3992 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:45:22.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.681+0000 7f45d159a640 1 -- 172.21.15.42:0/2579195779 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45c40b1070 con 0x7f45d40a1330 2024-09-16T03:45:22.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.681+0000 7f45d159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:45:22.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.681+0000 7f45d159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:45:22.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.681+0000 7f45d159a640 10 monclient: dump: 2024-09-16T03:45:22.692 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:45:22.692 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:45:22.692 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:22.693 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:22.693 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:45:22.693 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:45:22.693 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:45:22.693 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:45:22.693 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:45:22.693 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:45:22.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.681+0000 7f45d159a640 10 monclient: _finish_auth 0 2024-09-16T03:45:22.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.681+0000 7f45d159a640 10 monclient: _check_auth_tickets 2024-09-16T03:45:22.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.681+0000 7f45d159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:45:22.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.681+0000 7f45d8c4b640 5 monclient: authenticate success, global_id 13373 2024-09-16T03:45:22.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.681+0000 7f45d8c4b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:45:22.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.681+0000 7f45d159a640 1 -- 172.21.15.42:0/2579195779 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f45c40ac410 con 0x7f45d40a1330 2024-09-16T03:45:22.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.681+0000 7f45d159a640 10 monclient: handle_config config(0 keys) 2024-09-16T03:45:22.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.681+0000 7f45d159a640 1 -- 172.21.15.42:0/2579195779 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45c40ac7e0 con 0x7f45d40a1330 2024-09-16T03:45:22.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.681+0000 7f45d159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:45:22.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.681+0000 7f45d159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:45:22.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.681+0000 7f45d159a640 10 monclient: dump: 2024-09-16T03:45:22.694 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:45:22.694 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:45:22.694 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:22.694 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:22.694 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:45:22.694 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:45:22.694 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:45:22.694 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:45:22.694 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:45:22.694 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:45:22.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.681+0000 7f45d8c4b640 10 monclient: get_monmap_and_config success 2024-09-16T03:45:22.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.681+0000 7f45d8c4b640 4 set_mon_vals no callback set 2024-09-16T03:45:22.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.681+0000 7f45d8c4b640 10 monclient: shutdown 2024-09-16T03:45:22.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.681+0000 7f45d8c4b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:45:22.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.681+0000 7f45d8c4b640 1 -- 172.21.15.42:0/2579195779 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f45d40a1330 msgr2=0x7f45d40a1710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:45:22.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.681+0000 7f45d8c4b640 1 --2- 172.21.15.42:0/2579195779 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f45d40a1330 0x7f45d40a1710 secure :-1 s=READY pgs=2372 cs=0 l=1 rev1=1 crypto rx=0x7f45c40097c0 tx=0x7f45c401c4a0 comp rx=0 tx=0).stop 2024-09-16T03:45:22.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.681+0000 7f45d8c4b640 1 -- 172.21.15.42:0/2579195779 shutdown_connections 2024-09-16T03:45:22.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.681+0000 7f45d8c4b640 1 --2- 172.21.15.42:0/2579195779 >> v2:172.21.15.154:3300/0 conn(0x7f45d406dac0 0x7f45d40a1c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:45:22.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.681+0000 7f45d8c4b640 1 --2- 172.21.15.42:0/2579195779 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f45d40a1330 0x7f45d40a1710 unknown :-1 s=CLOSED pgs=2372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:45:22.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.681+0000 7f45d8c4b640 1 -- 172.21.15.42:0/2579195779 >> 172.21.15.42:0/2579195779 conn(0x7f45d40a2420 msgr2=0x7f45d40a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:45:22.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.681+0000 7f45d8c4b640 1 -- 172.21.15.42:0/2579195779 shutdown_connections 2024-09-16T03:45:22.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.681+0000 7f45d8c4b640 1 -- 172.21.15.42:0/2579195779 wait complete. 2024-09-16T03:45:22.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.681+0000 7f45d8c4b640 10 monclient: build_initial_monmap 2024-09-16T03:45:22.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.681+0000 7f45d8c4b640 10 monclient: monmap: 2024-09-16T03:45:22.696 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:45:22.696 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:45:22.696 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:45:22.696 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:45:22.696 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:45:22.696 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:45:22.696 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:45:22.696 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:45:22.696 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:45:22.696 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:45:22.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.681+0000 7f45d8c4b640 1 librados: starting msgr at 2024-09-16T03:45:22.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.681+0000 7f45d8c4b640 1 librados: starting objecter 2024-09-16T03:45:22.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.681+0000 7f45d8c4b640 1 Processor -- start 2024-09-16T03:45:22.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.681+0000 7f45d8c4b640 1 -- start start 2024-09-16T03:45:22.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.681+0000 7f45d8c4b640 1 librados: setting wanted keys 2024-09-16T03:45:22.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.681+0000 7f45d8c4b640 1 librados: calling monclient init 2024-09-16T03:45:22.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.681+0000 7f45d8c4b640 10 monclient: init 2024-09-16T03:45:22.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.681+0000 7f45d8c4b640 10 monclient: _reopen_session rank -1 2024-09-16T03:45:22.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.681+0000 7f45d8c4b640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:45:22.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.681+0000 7f45d8c4b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f45d40a1330 0x7f45d413bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:45:22.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.681+0000 7f45d8c4b640 10 monclient(hunting): picked mon.noname-b con 0x7f45d40a1330 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:45:22.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.681+0000 7f45d8c4b640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f45d4135100 0x7f45d41341c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:45:22.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.681+0000 7f45d8c4b640 10 monclient(hunting): picked mon.noname-a con 0x7f45d4135100 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:45:22.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.681+0000 7f45d8c4b640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f45d4134700 0x7f45d41cce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:45:22.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.681+0000 7f45d8c4b640 10 monclient(hunting): picked mon.noname-c con 0x7f45d4134700 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:45:22.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.681+0000 7f45d8c4b640 10 monclient(hunting): start opening mon connection 2024-09-16T03:45:22.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.681+0000 7f45d8c4b640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f45d413fd00 con 0x7f45d4135100 2024-09-16T03:45:22.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.681+0000 7f45d8c4b640 10 monclient(hunting): start opening mon connection 2024-09-16T03:45:22.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.681+0000 7f45d8c4b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f45d413fb80 con 0x7f45d40a1330 2024-09-16T03:45:22.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.681+0000 7f45d8c4b640 10 monclient(hunting): start opening mon connection 2024-09-16T03:45:22.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.681+0000 7f45d8c4b640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f45d413fe80 con 0x7f45d4134700 2024-09-16T03:45:22.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.681+0000 7f45d8c4b640 10 monclient(hunting): _renew_subs 2024-09-16T03:45:22.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.681+0000 7f45d8c4b640 10 monclient(hunting): authenticate will time out at 1169.655197s 2024-09-16T03:45:22.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.681+0000 7f45d1d9b640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f45d4135100 0x7f45d41341c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:45:22.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.681+0000 7f45d259c640 1 -- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f45d40a1330 msgr2=0x7f45d413bfb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T03:45:22.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.681+0000 7f45d2d9d640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f45d4134700 0x7f45d41cce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:45:22.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.681+0000 7f45d259c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f45d40a1330 0x7f45d413bfb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:45:22.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.681+0000 7f45d1d9b640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f45d4135100 0x7f45d41341c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:33532/0 (socket says 172.21.15.42:33532) 2024-09-16T03:45:22.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.681+0000 7f45d1d9b640 1 -- 172.21.15.42:0/2708588659 learned_addr learned my addr 172.21.15.42:0/2708588659 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:45:22.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.681+0000 7f45d2d9d640 10 monclient(hunting): get_auth_request con 0x7f45d4134700 auth_method 0 2024-09-16T03:45:22.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.681+0000 7f45d2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:45:22.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.681+0000 7f45d2d9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:45:22.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.681+0000 7f45d2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:45:22.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.681+0000 7f45d1d9b640 10 monclient(hunting): get_auth_request con 0x7f45d4135100 auth_method 0 2024-09-16T03:45:22.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.681+0000 7f45d1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:45:22.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.681+0000 7f45d1d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:45:22.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.681+0000 7f45d1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:45:22.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.681+0000 7f45d2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:45:22.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.681+0000 7f45d2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:45:22.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.681+0000 7f45d2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:45:22.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.681+0000 7f45d1d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:45:22.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.681+0000 7f45d1d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:45:22.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.681+0000 7f45d1d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:45:22.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.681+0000 7f45d2d9d640 10 monclient(hunting): handle_auth_done global_id 13376 payload 824 2024-09-16T03:45:22.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.681+0000 7f45d2d9d640 1 -- 172.21.15.42:0/2708588659 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f45d40a1330 msgr2=0x7f45d413bfb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:45:22.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.681+0000 7f45d2d9d640 1 --2- 172.21.15.42:0/2708588659 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f45d40a1330 0x7f45d413bfb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:45:22.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.681+0000 7f45d2d9d640 1 -- 172.21.15.42:0/2708588659 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f45d4135100 msgr2=0x7f45d41341c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:45:22.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.681+0000 7f45d2d9d640 1 --2- 172.21.15.42:0/2708588659 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f45d4135100 0x7f45d41341c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:45:22.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.681+0000 7f45d2d9d640 10 monclient: _finish_hunting 0 2024-09-16T03:45:22.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.681+0000 7f45d2d9d640 1 monclient: found mon.noname-c 2024-09-16T03:45:22.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.681+0000 7f45d2d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:45:22.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.681+0000 7f45d2d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:45:22.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.681+0000 7f45d2d9d640 1 -- 172.21.15.42:0/2708588659 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45d413d100 con 0x7f45d4134700 2024-09-16T03:45:22.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.681+0000 7f45d2d9d640 1 --2- 172.21.15.42:0/2708588659 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f45d4134700 0x7f45d41cce90 secure :-1 s=READY pgs=2373 cs=0 l=1 rev1=1 crypto rx=0x7f45cc00c710 tx=0x7f45cc00cbd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:45:22.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.681+0000 7f45c2ffd640 10 client.?.objecter ms_handle_connect 0x7f45d4134700 2024-09-16T03:45:22.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.685+0000 7f45c2ffd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:45:22.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.685+0000 7f45c2ffd640 1 -- 172.21.15.42:0/2708588659 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45cc018070 con 0x7f45d4134700 2024-09-16T03:45:22.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.685+0000 7f45c2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:45:22.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.685+0000 7f45c2ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:45:22.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.685+0000 7f45c2ffd640 10 monclient: dump: 2024-09-16T03:45:22.701 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:45:22.701 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:45:22.702 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:22.702 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:22.702 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:45:22.702 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:45:22.702 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:45:22.702 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:45:22.702 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:45:22.702 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:45:22.702 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.685+0000 7f45c2ffd640 10 monclient: _finish_auth 0 2024-09-16T03:45:22.702 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.685+0000 7f45c2ffd640 10 monclient: _check_auth_tickets 2024-09-16T03:45:22.702 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.685+0000 7f45c2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:45:22.702 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.685+0000 7f45d8c4b640 5 monclient: authenticate success, global_id 13376 2024-09-16T03:45:22.702 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.685+0000 7f45c2ffd640 1 -- 172.21.15.42:0/2708588659 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f45cc00ce20 con 0x7f45d4134700 2024-09-16T03:45:22.702 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.685+0000 7f45d8c4b640 10 monclient: _renew_subs 2024-09-16T03:45:22.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.685+0000 7f45d8c4b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:45:22.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.685+0000 7f45d8c4b640 1 -- 172.21.15.42:0/2708588659 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45d40a4df0 con 0x7f45d4134700 2024-09-16T03:45:22.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.685+0000 7f45c2ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:45:22.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.685+0000 7f45c2ffd640 1 -- 172.21.15.42:0/2708588659 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45cc00a820 con 0x7f45d4134700 2024-09-16T03:45:22.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.685+0000 7f45c2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:45:22.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.685+0000 7f45c2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:45:22.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.685+0000 7f45c2ffd640 10 monclient: dump: 2024-09-16T03:45:22.703 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:45:22.703 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:45:22.703 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:22.703 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:22.703 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:45:22.703 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:45:22.703 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:45:22.704 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:45:22.704 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:45:22.704 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:45:22.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.685+0000 7f45d0d99640 4 set_mon_vals no callback set 2024-09-16T03:45:22.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.685+0000 7f45d8c4b640 10 client.13376.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:45:22.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.685+0000 7f45d8c4b640 10 monclient: _renew_subs 2024-09-16T03:45:22.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.685+0000 7f45d8c4b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:45:22.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.685+0000 7f45d8c4b640 1 -- 172.21.15.42:0/2708588659 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f45d411e590 con 0x7f45d4134700 2024-09-16T03:45:22.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.685+0000 7f45d8c4b640 1 librados: init done 2024-09-16T03:45:22.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.685+0000 7f45c2ffd640 1 -- 172.21.15.42:0/2708588659 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f45cc01b070 con 0x7f45d4134700 2024-09-16T03:45:22.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.685+0000 7f45d8c4b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:45:22.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.685+0000 7f45d8c4b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:45:22.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.685+0000 7f45d8c4b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:45:22.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.685+0000 7f45d8c4b640 1 -- 172.21.15.42:0/2708588659 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f45d4135510 con 0x7f45d4134700 2024-09-16T03:45:22.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.685+0000 7f45c2ffd640 1 --2- 172.21.15.42:0/2708588659 >> v2:172.21.15.154:6812/254845342 conn(0x7f45b40377c0 0x7f45b4039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:45:22.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.685+0000 7f45c2ffd640 1 -- 172.21.15.42:0/2708588659 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(90..90 src has 1..90) ==== 7319+0+0 (secure 0 0 0) 0x7f45cc0489f0 con 0x7f45d4134700 2024-09-16T03:45:22.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.685+0000 7f45c2ffd640 10 client.13376.objecter ms_dispatch 0x7f45d4000ba0 osd_map(90..90 src has 1..90) 2024-09-16T03:45:22.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.685+0000 7f45c2ffd640 3 client.13376.objecter handle_osd_map got epochs [90,90] > 0 2024-09-16T03:45:22.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.685+0000 7f45c2ffd640 3 client.13376.objecter handle_osd_map decoding full epoch 90 2024-09-16T03:45:22.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.685+0000 7f45c2ffd640 20 client.13376.objecter dump_active .. 0 homeless 2024-09-16T03:45:22.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.685+0000 7f45d259c640 1 --2- 172.21.15.42:0/2708588659 >> v2:172.21.15.154:6812/254845342 conn(0x7f45b40377c0 0x7f45b4039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:45:22.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.685+0000 7f45d259c640 10 monclient: get_auth_request con 0x7f45b40377c0 auth_method 0 2024-09-16T03:45:22.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.685+0000 7f45c2ffd640 10 client.13376.objecter ms_handle_connect 0x7f45b40377c0 2024-09-16T03:45:22.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.685+0000 7f45d259c640 1 --2- 172.21.15.42:0/2708588659 >> v2:172.21.15.154:6812/254845342 conn(0x7f45b40377c0 0x7f45b4039c80 secure :-1 s=READY pgs=3367 cs=0 l=1 rev1=1 crypto rx=0x7f45c40ae000 tx=0x7f45c4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:45:22.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.689+0000 7f45c2ffd640 1 -- 172.21.15.42:0/2708588659 <== mon.2 v2:172.21.15.42:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f45cc04ae20 con 0x7f45d4134700 2024-09-16T03:45:22.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.689+0000 7f45c2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:45:22.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.689+0000 7f45c2ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:45:22.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.917+0000 7f45d8c4b640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:45:22.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.917+0000 7f45d8c4b640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:45:22.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.917+0000 7f45d8c4b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:45:22.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.917+0000 7f45d8c4b640 1 -- 172.21.15.42:0/2708588659 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f45d4069040 con 0x7f45d4134700 2024-09-16T03:45:22.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.917+0000 7f45c2ffd640 1 -- 172.21.15.42:0/2708588659 <== mon.2 v2:172.21.15.42:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7f45d413d100 con 0x7f45d4134700 2024-09-16T03:45:22.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.917+0000 7f45c2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:45:22.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.917+0000 7f45c2ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:45:22.923 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":28,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:45:22.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.917+0000 7f45d8c4b640 10 librados: watch_flush enter 2024-09-16T03:45:22.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.917+0000 7f45d8c4b640 10 librados: watch_flush exit 2024-09-16T03:45:22.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.917+0000 7f45d8c4b640 20 client.13376.objecter shutdown clearing up homeless session... 2024-09-16T03:45:22.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.917+0000 7f45d8c4b640 10 client.13376.objecter successfully canceled tick 2024-09-16T03:45:22.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.921+0000 7f45d8c4b640 1 -- 172.21.15.42:0/2708588659 >> v2:172.21.15.154:6812/254845342 conn(0x7f45b40377c0 msgr2=0x7f45b4039c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:45:22.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.921+0000 7f45d8c4b640 1 --2- 172.21.15.42:0/2708588659 >> v2:172.21.15.154:6812/254845342 conn(0x7f45b40377c0 0x7f45b4039c80 secure :-1 s=READY pgs=3367 cs=0 l=1 rev1=1 crypto rx=0x7f45c40ae000 tx=0x7f45c4008040 comp rx=0 tx=0).stop 2024-09-16T03:45:22.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.921+0000 7f45d8c4b640 10 monclient: shutdown 2024-09-16T03:45:22.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.921+0000 7f45d8c4b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:45:22.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.921+0000 7f45d8c4b640 1 -- 172.21.15.42:0/2708588659 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f45d4134700 msgr2=0x7f45d41cce90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:45:22.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.921+0000 7f45d8c4b640 1 --2- 172.21.15.42:0/2708588659 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f45d4134700 0x7f45d41cce90 secure :-1 s=READY pgs=2373 cs=0 l=1 rev1=1 crypto rx=0x7f45cc00c710 tx=0x7f45cc00cbd0 comp rx=0 tx=0).stop 2024-09-16T03:45:22.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.921+0000 7f45d8c4b640 1 -- 172.21.15.42:0/2708588659 shutdown_connections 2024-09-16T03:45:22.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.921+0000 7f45d8c4b640 1 --2- 172.21.15.42:0/2708588659 >> v2:172.21.15.154:6812/254845342 conn(0x7f45b40377c0 0x7f45b4039c80 unknown :-1 s=CLOSED pgs=3367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:45:22.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.921+0000 7f45d8c4b640 1 --2- 172.21.15.42:0/2708588659 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f45d4134700 0x7f45d41cce90 unknown :-1 s=CLOSED pgs=2373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:45:22.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.921+0000 7f45d8c4b640 1 --2- 172.21.15.42:0/2708588659 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f45d4135100 0x7f45d41341c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:45:22.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.921+0000 7f45d8c4b640 1 --2- 172.21.15.42:0/2708588659 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f45d40a1330 0x7f45d413bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:45:22.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.921+0000 7f45d8c4b640 1 -- 172.21.15.42:0/2708588659 >> 172.21.15.42:0/2708588659 conn(0x7f45d40a2420 msgr2=0x7f45d40a4550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:45:22.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.921+0000 7f45d8c4b640 1 -- 172.21.15.42:0/2708588659 shutdown_connections 2024-09-16T03:45:22.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.921+0000 7f45d8c4b640 1 -- 172.21.15.42:0/2708588659 wait complete. 2024-09-16T03:45:22.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:22.921+0000 7f45d8c4b640 1 librados: shutdown 2024-09-16T03:45:25.942 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:45:26.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.069+0000 7faefe733640 10 monclient: get_monmap_and_config 2024-09-16T03:45:26.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.069+0000 7faefe733640 10 monclient: build_initial_monmap 2024-09-16T03:45:26.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.069+0000 7faefe733640 10 monclient: monmap: 2024-09-16T03:45:26.075 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:45:26.075 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:45:26.075 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:45:26.073926+0000 2024-09-16T03:45:26.075 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:45:26.073926+0000 2024-09-16T03:45:26.075 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:45:26.075 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:45:26.075 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:45:26.075 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:45:26.075 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:45:26.076 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:45:26.076 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:45:26.076 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:45:26.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.069+0000 7faefe733640 1 Processor -- start 2024-09-16T03:45:26.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.069+0000 7faefe733640 1 -- start start 2024-09-16T03:45:26.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.069+0000 7faefe733640 10 monclient: init 2024-09-16T03:45:26.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.069+0000 7faefe733640 10 monclient: _reopen_session rank -1 2024-09-16T03:45:26.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.069+0000 7faefe733640 10 monclient: _add_conns ranks=[0,3,1,4,2] 2024-09-16T03:45:26.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.069+0000 7faefe733640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7faef80a1330 addr v1:172.21.15.42:6789/0 2024-09-16T03:45:26.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.069+0000 7faefe733640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7faef806dac0 0x7faef813a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:45:26.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.069+0000 7faefe733640 10 monclient(hunting): picked mon.noname-a con 0x7faef806dac0 addr v2:172.21.15.42:3300/0 2024-09-16T03:45:26.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.069+0000 7faefe733640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7faef80a1e20 addr v1:172.21.15.154:6789/0 2024-09-16T03:45:26.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.069+0000 7faefe733640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faef809c090 con 0x7faef80a1330 2024-09-16T03:45:26.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.069+0000 7faefe733640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faef809f610 con 0x7faef80a1e20 2024-09-16T03:45:26.080 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.069+0000 7faefe733640 10 monclient(hunting): start opening mon connection 2024-09-16T03:45:26.080 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.069+0000 7faefe733640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7faef809f8f0 con 0x7faef806dac0 2024-09-16T03:45:26.080 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.069+0000 7faefe733640 10 monclient(hunting): _renew_subs 2024-09-16T03:45:26.080 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.069+0000 7faefe733640 10 monclient(hunting): authenticate will time out at 1173.043752s 2024-09-16T03:45:26.080 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.069+0000 7faef7fff640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7faef80a1330 0x7faef80a1710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:54242/0 (socket says 172.21.15.42:54242) 2024-09-16T03:45:26.080 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.069+0000 7faef7fff640 1 -- 172.21.15.42:0/4182061073 learned_addr learned my addr 172.21.15.42:0/4182061073 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:45:26.080 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.069+0000 7faefccd0640 1 -- 172.21.15.42:0/4182061073 >> v1:172.21.15.154:6789/0 conn(0x7faef80a1e20 legacy=0x7faef813c890 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.154:6789/0 2024-09-16T03:45:26.080 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.069+0000 7faef77fe640 1 --2- 172.21.15.42:0/4182061073 >> v2:172.21.15.42:3300/0 conn(0x7faef806dac0 0x7faef813a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:45:26.080 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.069+0000 7faef77fe640 10 monclient(hunting): get_auth_request con 0x7faef806dac0 auth_method 0 2024-09-16T03:45:26.080 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.069+0000 7faef77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:45:26.080 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.069+0000 7faef77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:45:26.080 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.069+0000 7faef77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:45:26.081 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.069+0000 7faef77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:45:26.081 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.069+0000 7faef77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:45:26.081 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.073+0000 7faef77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:45:26.081 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.073+0000 7faef77fe640 10 monclient(hunting): handle_auth_done global_id 744204 payload 274 2024-09-16T03:45:26.081 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.073+0000 7faef77fe640 1 -- 172.21.15.42:0/4182061073 >> v1:172.21.15.154:6789/0 conn(0x7faef80a1e20 legacy=0x7faef813c890 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:45:26.081 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.073+0000 7faef77fe640 1 -- 172.21.15.42:0/4182061073 >> v1:172.21.15.42:6789/0 conn(0x7faef80a1330 legacy=0x7faef80a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:45:26.081 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.073+0000 7faef77fe640 10 monclient: _finish_hunting 0 2024-09-16T03:45:26.081 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.073+0000 7faef77fe640 1 monclient: found mon.noname-a 2024-09-16T03:45:26.081 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.073+0000 7faef77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:45:26.081 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.073+0000 7faef77fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:45:26.081 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.073+0000 7faef77fe640 1 -- 172.21.15.42:0/4182061073 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faef809fbd0 con 0x7faef806dac0 2024-09-16T03:45:26.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.073+0000 7faef77fe640 1 --2- 172.21.15.42:0/4182061073 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faef806dac0 0x7faef813a340 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7faeec0097c0 tx=0x7faeec01c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d316966e93857074 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:45:26.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.073+0000 7faef6ffd640 1 -- 172.21.15.42:0/4182061073 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faeec0b0070 con 0x7faef806dac0 2024-09-16T03:45:26.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.073+0000 7faef6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:45:26.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.073+0000 7faef6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:45:26.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.073+0000 7faef6ffd640 10 monclient: dump: 2024-09-16T03:45:26.089 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:45:26.089 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:45:26.089 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:26.089 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:26.089 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:45:26.089 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:45:26.090 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:45:26.090 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:45:26.090 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:45:26.090 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:45:26.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.073+0000 7faef6ffd640 10 monclient: _finish_auth 0 2024-09-16T03:45:26.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.073+0000 7faef6ffd640 10 monclient: _check_auth_tickets 2024-09-16T03:45:26.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.073+0000 7faef6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:45:26.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.073+0000 7faefe733640 5 monclient: authenticate success, global_id 744204 2024-09-16T03:45:26.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.073+0000 7faef6ffd640 1 -- 172.21.15.42:0/4182061073 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faeec0ab600 con 0x7faef806dac0 2024-09-16T03:45:26.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.073+0000 7faef6ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:45:26.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.073+0000 7faef6ffd640 1 -- 172.21.15.42:0/4182061073 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faeec0ab9d0 con 0x7faef806dac0 2024-09-16T03:45:26.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.073+0000 7faef6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:45:26.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.073+0000 7faef6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:45:26.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.073+0000 7faef6ffd640 10 monclient: dump: 2024-09-16T03:45:26.091 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:45:26.091 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:45:26.091 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:26.091 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:26.091 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:45:26.091 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:45:26.091 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:45:26.091 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:45:26.091 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:45:26.091 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:45:26.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.073+0000 7faefe733640 10 monclient: get_monmap_and_config success 2024-09-16T03:45:26.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.073+0000 7faefe733640 4 set_mon_vals no callback set 2024-09-16T03:45:26.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.073+0000 7faefe733640 10 monclient: shutdown 2024-09-16T03:45:26.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.073+0000 7faefe733640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:45:26.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.073+0000 7faefe733640 1 -- 172.21.15.42:0/4182061073 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faef806dac0 msgr2=0x7faef813a340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:45:26.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.073+0000 7faefe733640 1 --2- 172.21.15.42:0/4182061073 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faef806dac0 0x7faef813a340 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7faeec0097c0 tx=0x7faeec01c440 comp rx=0 tx=0).stop 2024-09-16T03:45:26.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.073+0000 7faefe733640 1 -- 172.21.15.42:0/4182061073 shutdown_connections 2024-09-16T03:45:26.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.073+0000 7faefe733640 1 --2- 172.21.15.42:0/4182061073 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faef806dac0 0x7faef813a340 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:45:26.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.073+0000 7faefe733640 1 -- 172.21.15.42:0/4182061073 >> 172.21.15.42:0/4182061073 conn(0x7faef80a2420 msgr2=0x7faef80a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:45:26.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.073+0000 7faefe733640 1 -- 172.21.15.42:0/4182061073 shutdown_connections 2024-09-16T03:45:26.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.073+0000 7faefe733640 1 -- 172.21.15.42:0/4182061073 wait complete. 2024-09-16T03:45:26.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.073+0000 7faefe733640 10 monclient: build_initial_monmap 2024-09-16T03:45:26.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.077+0000 7faefe733640 10 monclient: monmap: 2024-09-16T03:45:26.092 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:45:26.092 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:45:26.092 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:45:26.093 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:45:26.093 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:45:26.093 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:45:26.093 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:45:26.093 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:45:26.093 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:45:26.093 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:45:26.093 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.077+0000 7faefe733640 1 librados: starting msgr at 2024-09-16T03:45:26.093 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.077+0000 7faefe733640 1 librados: starting objecter 2024-09-16T03:45:26.093 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.077+0000 7faefe733640 1 Processor -- start 2024-09-16T03:45:26.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.077+0000 7faefe733640 1 -- start start 2024-09-16T03:45:26.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.077+0000 7faefe733640 1 librados: setting wanted keys 2024-09-16T03:45:26.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.077+0000 7faefe733640 1 librados: calling monclient init 2024-09-16T03:45:26.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.077+0000 7faefe733640 10 monclient: init 2024-09-16T03:45:26.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.077+0000 7faefe733640 10 monclient: _reopen_session rank -1 2024-09-16T03:45:26.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.077+0000 7faefe733640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T03:45:26.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.077+0000 7faefe733640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faef80a1330 0x7faef813c120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:45:26.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.077+0000 7faefe733640 10 monclient(hunting): picked mon.noname-a con 0x7faef80a1330 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:45:26.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.077+0000 7faefe733640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faef80a1e20 0x7faef8135280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:45:26.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.077+0000 7faefe733640 10 monclient(hunting): picked mon.noname-b con 0x7faef80a1e20 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:45:26.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.077+0000 7faefe733640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7faef81ccf00 0x7faef81cd390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:45:26.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.077+0000 7faefe733640 10 monclient(hunting): picked mon.noname-c con 0x7faef81ccf00 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:45:26.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.077+0000 7faefe733640 10 monclient(hunting): start opening mon connection 2024-09-16T03:45:26.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.077+0000 7faefe733640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7faef8140b60 con 0x7faef80a1330 2024-09-16T03:45:26.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.077+0000 7faefe733640 10 monclient(hunting): start opening mon connection 2024-09-16T03:45:26.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.077+0000 7faefe733640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7faef81409e0 con 0x7faef80a1e20 2024-09-16T03:45:26.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.077+0000 7faefe733640 10 monclient(hunting): start opening mon connection 2024-09-16T03:45:26.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.077+0000 7faefe733640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7faef8140ce0 con 0x7faef81ccf00 2024-09-16T03:45:26.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.077+0000 7faefe733640 10 monclient(hunting): _renew_subs 2024-09-16T03:45:26.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.077+0000 7faefe733640 10 monclient(hunting): authenticate will time out at 1173.048919s 2024-09-16T03:45:26.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.077+0000 7faef77fe640 1 -- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faef80a1e20 msgr2=0x7faef8135280 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T03:45:26.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.077+0000 7faef77fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faef80a1e20 0x7faef8135280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:45:26.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.077+0000 7faef7fff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faef80a1330 0x7faef813c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:45:26.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.077+0000 7faefccd0640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7faef81ccf00 0x7faef81cd390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:45:26.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.077+0000 7faef7fff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faef80a1330 0x7faef813c120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:39006/0 (socket says 172.21.15.42:39006) 2024-09-16T03:45:26.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.077+0000 7faefccd0640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7faef81ccf00 0x7faef81cd390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:39128/0 (socket says 172.21.15.42:39128) 2024-09-16T03:45:26.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.077+0000 7faef7fff640 1 -- 172.21.15.42:0/2655697277 learned_addr learned my addr 172.21.15.42:0/2655697277 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:45:26.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.077+0000 7faef7fff640 10 monclient(hunting): get_auth_request con 0x7faef80a1330 auth_method 0 2024-09-16T03:45:26.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.077+0000 7faef7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:45:26.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.077+0000 7faef7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:45:26.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.077+0000 7faef7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:45:26.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.077+0000 7faefccd0640 10 monclient(hunting): get_auth_request con 0x7faef81ccf00 auth_method 0 2024-09-16T03:45:26.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.077+0000 7faefccd0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:45:26.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.077+0000 7faefccd0640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:45:26.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.077+0000 7faefccd0640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:45:26.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.077+0000 7faef7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:45:26.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.077+0000 7faef7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:45:26.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.077+0000 7faef7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:45:26.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.077+0000 7faefccd0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:45:26.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.077+0000 7faefccd0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:45:26.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.077+0000 7faefccd0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:45:26.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.077+0000 7faef7fff640 10 monclient(hunting): handle_auth_done global_id 744210 payload 824 2024-09-16T03:45:26.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.077+0000 7faef7fff640 1 -- 172.21.15.42:0/2655697277 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7faef81ccf00 msgr2=0x7faef81cd390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:45:26.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.077+0000 7faef7fff640 1 --2- 172.21.15.42:0/2655697277 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7faef81ccf00 0x7faef81cd390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:45:26.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.077+0000 7faef7fff640 1 -- 172.21.15.42:0/2655697277 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faef80a1e20 msgr2=0x7faef8135280 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:45:26.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.077+0000 7faef7fff640 1 --2- 172.21.15.42:0/2655697277 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faef80a1e20 0x7faef8135280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:45:26.098 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.077+0000 7faef7fff640 10 monclient: _finish_hunting 0 2024-09-16T03:45:26.098 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.077+0000 7faef7fff640 1 monclient: found mon.noname-a 2024-09-16T03:45:26.098 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.077+0000 7faef7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:45:26.098 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.077+0000 7faef7fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:45:26.098 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.077+0000 7faef7fff640 1 -- 172.21.15.42:0/2655697277 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faef811ec00 con 0x7faef80a1330 2024-09-16T03:45:26.098 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.077+0000 7faef7fff640 1 --2- 172.21.15.42:0/2655697277 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faef80a1330 0x7faef813c120 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7faee800c420 tx=0x7faee800c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:45:26.098 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.077+0000 7faef4ff9640 10 client.?.objecter ms_handle_connect 0x7faef80a1330 2024-09-16T03:45:26.098 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.077+0000 7faef4ff9640 10 client.?.objecter resend_mon_ops 2024-09-16T03:45:26.099 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.077+0000 7faef4ff9640 1 -- 172.21.15.42:0/2655697277 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faee8017070 con 0x7faef80a1330 2024-09-16T03:45:26.099 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.077+0000 7faef4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:45:26.099 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.077+0000 7faef4ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:45:26.099 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.077+0000 7faef4ff9640 10 monclient: dump: 2024-09-16T03:45:26.099 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:45:26.099 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:45:26.099 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:26.099 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:26.099 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:45:26.099 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:45:26.100 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:45:26.100 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:45:26.100 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:45:26.100 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:45:26.100 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.077+0000 7faef4ff9640 10 monclient: _finish_auth 0 2024-09-16T03:45:26.100 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.077+0000 7faef4ff9640 10 monclient: _check_auth_tickets 2024-09-16T03:45:26.100 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.077+0000 7faef4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:45:26.100 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.077+0000 7faefe733640 5 monclient: authenticate success, global_id 744210 2024-09-16T03:45:26.100 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.077+0000 7faef4ff9640 1 -- 172.21.15.42:0/2655697277 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faee8005cf0 con 0x7faef80a1330 2024-09-16T03:45:26.100 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.077+0000 7faefe733640 10 monclient: _renew_subs 2024-09-16T03:45:26.101 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.077+0000 7faefe733640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:45:26.101 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.077+0000 7faefe733640 1 -- 172.21.15.42:0/2655697277 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faef806e5f0 con 0x7faef80a1330 2024-09-16T03:45:26.101 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.077+0000 7faef4ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T03:45:26.101 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.077+0000 7faef4ff9640 1 -- 172.21.15.42:0/2655697277 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faee80072f0 con 0x7faef80a1330 2024-09-16T03:45:26.101 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.077+0000 7faef4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:45:26.101 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.077+0000 7faef4ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:45:26.101 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.077+0000 7faef4ff9640 10 monclient: dump: 2024-09-16T03:45:26.101 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:45:26.101 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:45:26.101 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:26.101 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:26.101 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:45:26.102 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:45:26.102 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:45:26.102 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:45:26.102 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:45:26.102 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:45:26.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.077+0000 7faef67fc640 4 set_mon_vals no callback set 2024-09-16T03:45:26.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.077+0000 7faefe733640 10 client.744210.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:45:26.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.077+0000 7faefe733640 10 monclient: _renew_subs 2024-09-16T03:45:26.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.077+0000 7faefe733640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:45:26.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.077+0000 7faefe733640 1 -- 172.21.15.42:0/2655697277 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faef813d420 con 0x7faef80a1330 2024-09-16T03:45:26.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.077+0000 7faefe733640 1 librados: init done 2024-09-16T03:45:26.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.077+0000 7faef4ff9640 1 -- 172.21.15.42:0/2655697277 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7faee801a070 con 0x7faef80a1330 2024-09-16T03:45:26.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.077+0000 7faefe733640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:45:26.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.077+0000 7faefe733640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:45:26.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.077+0000 7faefe733640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:45:26.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.077+0000 7faefe733640 1 -- 172.21.15.42:0/2655697277 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faef80a1710 con 0x7faef80a1330 2024-09-16T03:45:26.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.077+0000 7faef4ff9640 1 --2- 172.21.15.42:0/2655697277 >> v2:172.21.15.154:6812/254845342 conn(0x7faee40377c0 0x7faee4039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:45:26.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.077+0000 7faef4ff9640 1 -- 172.21.15.42:0/2655697277 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 7319+0+0 (secure 0 0 0) 0x7faee8048940 con 0x7faef80a1330 2024-09-16T03:45:26.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.077+0000 7faef4ff9640 10 client.744210.objecter ms_dispatch 0x7faef8000ba0 osd_map(90..90 src has 1..90) 2024-09-16T03:45:26.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.077+0000 7faef4ff9640 3 client.744210.objecter handle_osd_map got epochs [90,90] > 0 2024-09-16T03:45:26.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.077+0000 7faef4ff9640 3 client.744210.objecter handle_osd_map decoding full epoch 90 2024-09-16T03:45:26.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.077+0000 7faef4ff9640 20 client.744210.objecter dump_active .. 0 homeless 2024-09-16T03:45:26.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.077+0000 7faef77fe640 1 --2- 172.21.15.42:0/2655697277 >> v2:172.21.15.154:6812/254845342 conn(0x7faee40377c0 0x7faee4039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:45:26.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.077+0000 7faef77fe640 10 monclient: get_auth_request con 0x7faee40377c0 auth_method 0 2024-09-16T03:45:26.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.081+0000 7faef77fe640 1 --2- 172.21.15.42:0/2655697277 >> v2:172.21.15.154:6812/254845342 conn(0x7faee40377c0 0x7faee4039c80 secure :-1 s=READY pgs=3368 cs=0 l=1 rev1=1 crypto rx=0x7faeec0ad000 tx=0x7faeec008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:45:26.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.081+0000 7faef4ff9640 10 client.744210.objecter ms_handle_connect 0x7faee40377c0 2024-09-16T03:45:26.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.085+0000 7faef4ff9640 1 -- 172.21.15.42:0/2655697277 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7faee8019e10 con 0x7faef80a1330 2024-09-16T03:45:26.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.085+0000 7faef4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:45:26.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.085+0000 7faef4ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:45:26.308 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.301+0000 7faefe733640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:45:26.308 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.301+0000 7faefe733640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:45:26.308 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.301+0000 7faefe733640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:45:26.308 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.301+0000 7faefe733640 1 -- 172.21.15.42:0/2655697277 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7faef8069040 con 0x7faef80a1330 2024-09-16T03:45:26.309 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.305+0000 7faef4ff9640 1 -- 172.21.15.42:0/2655697277 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7faee8007670 con 0x7faef80a1330 2024-09-16T03:45:26.309 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.305+0000 7faef4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:45:26.309 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.305+0000 7faef4ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:45:26.309 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":28,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:45:26.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.305+0000 7faefe733640 10 librados: watch_flush enter 2024-09-16T03:45:26.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.305+0000 7faefe733640 10 librados: watch_flush exit 2024-09-16T03:45:26.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.305+0000 7faefe733640 20 client.744210.objecter shutdown clearing up homeless session... 2024-09-16T03:45:26.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.305+0000 7faefe733640 10 client.744210.objecter successfully canceled tick 2024-09-16T03:45:26.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.305+0000 7faefe733640 1 -- 172.21.15.42:0/2655697277 >> v2:172.21.15.154:6812/254845342 conn(0x7faee40377c0 msgr2=0x7faee4039c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:45:26.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.305+0000 7faefe733640 1 --2- 172.21.15.42:0/2655697277 >> v2:172.21.15.154:6812/254845342 conn(0x7faee40377c0 0x7faee4039c80 secure :-1 s=READY pgs=3368 cs=0 l=1 rev1=1 crypto rx=0x7faeec0ad000 tx=0x7faeec008040 comp rx=0 tx=0).stop 2024-09-16T03:45:26.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.305+0000 7faefe733640 10 monclient: shutdown 2024-09-16T03:45:26.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.305+0000 7faefe733640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:45:26.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.305+0000 7faefe733640 1 -- 172.21.15.42:0/2655697277 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faef80a1330 msgr2=0x7faef813c120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:45:26.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.305+0000 7faefe733640 1 --2- 172.21.15.42:0/2655697277 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faef80a1330 0x7faef813c120 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7faee800c420 tx=0x7faee800c8e0 comp rx=0 tx=0).stop 2024-09-16T03:45:26.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.305+0000 7faefe733640 1 -- 172.21.15.42:0/2655697277 shutdown_connections 2024-09-16T03:45:26.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.305+0000 7faefe733640 1 --2- 172.21.15.42:0/2655697277 >> v2:172.21.15.154:6812/254845342 conn(0x7faee40377c0 0x7faee4039c80 unknown :-1 s=CLOSED pgs=3368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:45:26.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.305+0000 7faefe733640 1 --2- 172.21.15.42:0/2655697277 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7faef81ccf00 0x7faef81cd390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:45:26.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.305+0000 7faefe733640 1 --2- 172.21.15.42:0/2655697277 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faef80a1e20 0x7faef8135280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:45:26.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.305+0000 7faefe733640 1 --2- 172.21.15.42:0/2655697277 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faef80a1330 0x7faef813c120 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:45:26.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.305+0000 7faefe733640 1 -- 172.21.15.42:0/2655697277 >> 172.21.15.42:0/2655697277 conn(0x7faef80a2420 msgr2=0x7faef80a4dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:45:26.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.305+0000 7faefe733640 1 -- 172.21.15.42:0/2655697277 shutdown_connections 2024-09-16T03:45:26.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.305+0000 7faefe733640 1 -- 172.21.15.42:0/2655697277 wait complete. 2024-09-16T03:45:26.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:26.305+0000 7faefe733640 1 librados: shutdown 2024-09-16T03:45:29.328 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:45:29.460 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.452+0000 7fc4e8f21640 10 monclient: get_monmap_and_config 2024-09-16T03:45:29.460 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.452+0000 7fc4e8f21640 10 monclient: build_initial_monmap 2024-09-16T03:45:29.460 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.452+0000 7fc4e8f21640 10 monclient: monmap: 2024-09-16T03:45:29.460 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:45:29.460 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:45:29.461 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:45:29.459608+0000 2024-09-16T03:45:29.461 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:45:29.459608+0000 2024-09-16T03:45:29.461 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:45:29.461 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:45:29.461 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:45:29.461 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:45:29.461 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:45:29.461 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:45:29.461 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:45:29.461 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:45:29.463 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.456+0000 7fc4e8f21640 1 Processor -- start 2024-09-16T03:45:29.463 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.456+0000 7fc4e8f21640 1 -- start start 2024-09-16T03:45:29.463 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.456+0000 7fc4e8f21640 10 monclient: init 2024-09-16T03:45:29.463 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.456+0000 7fc4e8f21640 10 monclient: _reopen_session rank -1 2024-09-16T03:45:29.463 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.456+0000 7fc4e8f21640 10 monclient: _add_conns ranks=[0,2,1,4,3] 2024-09-16T03:45:29.463 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.456+0000 7fc4e8f21640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc4e40a1330 addr v1:172.21.15.42:6789/0 2024-09-16T03:45:29.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.456+0000 7fc4e8f21640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc4e409c920 0x7fc4e413c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:45:29.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.456+0000 7fc4e8f21640 10 monclient(hunting): picked mon.noname-b con 0x7fc4e409c920 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:45:29.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.456+0000 7fc4e8f21640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc4e4072d40 addr v1:172.21.15.154:6789/0 2024-09-16T03:45:29.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.456+0000 7fc4e8f21640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc4e4072350 con 0x7fc4e40a1330 2024-09-16T03:45:29.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.456+0000 7fc4e8f21640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc4e409bc60 con 0x7fc4e4072d40 2024-09-16T03:45:29.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.456+0000 7fc4e8f21640 10 monclient(hunting): start opening mon connection 2024-09-16T03:45:29.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.456+0000 7fc4e8f21640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fc4e4072b50 con 0x7fc4e409c920 2024-09-16T03:45:29.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.456+0000 7fc4e8f21640 10 monclient(hunting): _renew_subs 2024-09-16T03:45:29.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.456+0000 7fc4e8f21640 10 monclient(hunting): authenticate will time out at 1176.429130s 2024-09-16T03:45:29.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.456+0000 7fc4e259c640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fc4e40a1330 0x7fc4e40a1710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:54252/0 (socket says 172.21.15.42:54252) 2024-09-16T03:45:29.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.456+0000 7fc4e259c640 1 -- 172.21.15.42:0/3284378301 learned_addr learned my addr 172.21.15.42:0/3284378301 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:45:29.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.456+0000 7fc4e2d9d640 1 -- >> v1:172.21.15.154:6789/0 conn(0x7fc4e4072d40 legacy=0x7fc4e413eac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.154:6789/0 2024-09-16T03:45:29.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.456+0000 7fc4e1d9b640 1 --2- 172.21.15.42:0/3284378301 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc4e409c920 0x7fc4e413c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:45:29.465 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.456+0000 7fc4e1d9b640 10 monclient(hunting): get_auth_request con 0x7fc4e409c920 auth_method 0 2024-09-16T03:45:29.465 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.456+0000 7fc4e1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:45:29.465 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.456+0000 7fc4e1d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:45:29.465 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.456+0000 7fc4e1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:45:29.465 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.456+0000 7fc4e1d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:45:29.465 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.456+0000 7fc4e1d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:45:29.465 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.456+0000 7fc4e1d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:45:29.465 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.456+0000 7fc4e1d9b640 10 monclient(hunting): handle_auth_done global_id 13382 payload 274 2024-09-16T03:45:29.465 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.456+0000 7fc4e1d9b640 1 -- 172.21.15.42:0/3284378301 >> v1:172.21.15.154:6789/0 conn(0x7fc4e4072d40 legacy=0x7fc4e413eac0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:45:29.465 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.456+0000 7fc4e1d9b640 1 -- 172.21.15.42:0/3284378301 >> v1:172.21.15.42:6789/0 conn(0x7fc4e40a1330 legacy=0x7fc4e40a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:45:29.465 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.456+0000 7fc4e1d9b640 10 monclient: _finish_hunting 0 2024-09-16T03:45:29.465 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.456+0000 7fc4e1d9b640 1 monclient: found mon.noname-b 2024-09-16T03:45:29.465 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.456+0000 7fc4e1d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:45:29.466 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.456+0000 7fc4e1d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T03:45:29.466 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.456+0000 7fc4e1d9b640 1 -- 172.21.15.42:0/3284378301 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4e409bf40 con 0x7fc4e409c920 2024-09-16T03:45:29.471 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.456+0000 7fc4e1d9b640 1 --2- 172.21.15.42:0/3284378301 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc4e409c920 0x7fc4e413c570 secure :-1 s=READY pgs=2374 cs=0 l=1 rev1=1 crypto rx=0x7fc4cc009dc0 tx=0x7fc4cc01c810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=58a9291730ee48d6 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:45:29.471 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.456+0000 7fc4e159a640 1 -- 172.21.15.42:0/3284378301 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4cc0b2070 con 0x7fc4e409c920 2024-09-16T03:45:29.471 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.456+0000 7fc4e159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:45:29.471 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.460+0000 7fc4e159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:45:29.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.460+0000 7fc4e159a640 10 monclient: dump: 2024-09-16T03:45:29.472 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:45:29.472 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:45:29.472 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:29.472 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:29.472 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:45:29.472 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:45:29.472 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:45:29.472 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:45:29.472 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:45:29.472 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:45:29.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.460+0000 7fc4e159a640 10 monclient: _finish_auth 0 2024-09-16T03:45:29.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.460+0000 7fc4e159a640 10 monclient: _check_auth_tickets 2024-09-16T03:45:29.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.460+0000 7fc4e159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:45:29.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.460+0000 7fc4e8f21640 5 monclient: authenticate success, global_id 13382 2024-09-16T03:45:29.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.460+0000 7fc4e8f21640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:45:29.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.460+0000 7fc4e159a640 1 -- 172.21.15.42:0/3284378301 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc4cc0a57e0 con 0x7fc4e409c920 2024-09-16T03:45:29.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.460+0000 7fc4e159a640 10 monclient: handle_config config(0 keys) 2024-09-16T03:45:29.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.460+0000 7fc4e8f21640 10 monclient: get_monmap_and_config success 2024-09-16T03:45:29.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.460+0000 7fc4e8f21640 4 set_mon_vals no callback set 2024-09-16T03:45:29.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.460+0000 7fc4e159a640 1 -- 172.21.15.42:0/3284378301 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4cc0a5bb0 con 0x7fc4e409c920 2024-09-16T03:45:29.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.460+0000 7fc4e159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:45:29.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.460+0000 7fc4e159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:45:29.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.460+0000 7fc4e159a640 10 monclient: dump: 2024-09-16T03:45:29.473 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:45:29.474 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:45:29.474 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:29.474 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:29.474 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:45:29.474 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:45:29.474 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:45:29.474 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:45:29.474 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:45:29.474 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:45:29.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.460+0000 7fc4e8f21640 10 monclient: shutdown 2024-09-16T03:45:29.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.460+0000 7fc4e8f21640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:45:29.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.460+0000 7fc4e8f21640 1 -- 172.21.15.42:0/3284378301 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc4e409c920 msgr2=0x7fc4e413c570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:45:29.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.460+0000 7fc4e8f21640 1 --2- 172.21.15.42:0/3284378301 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc4e409c920 0x7fc4e413c570 secure :-1 s=READY pgs=2374 cs=0 l=1 rev1=1 crypto rx=0x7fc4cc009dc0 tx=0x7fc4cc01c810 comp rx=0 tx=0).stop 2024-09-16T03:45:29.475 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.460+0000 7fc4e8f21640 1 -- 172.21.15.42:0/3284378301 shutdown_connections 2024-09-16T03:45:29.475 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.460+0000 7fc4e8f21640 1 --2- 172.21.15.42:0/3284378301 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc4e409c920 0x7fc4e413c570 unknown :-1 s=CLOSED pgs=2374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:45:29.475 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.460+0000 7fc4e8f21640 1 -- 172.21.15.42:0/3284378301 >> 172.21.15.42:0/3284378301 conn(0x7fc4e40a2420 msgr2=0x7fc4e40a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:45:29.475 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.460+0000 7fc4e8f21640 1 -- 172.21.15.42:0/3284378301 shutdown_connections 2024-09-16T03:45:29.475 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.460+0000 7fc4e8f21640 1 -- 172.21.15.42:0/3284378301 wait complete. 2024-09-16T03:45:29.475 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.460+0000 7fc4e8f21640 10 monclient: build_initial_monmap 2024-09-16T03:45:29.475 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.460+0000 7fc4e8f21640 10 monclient: monmap: 2024-09-16T03:45:29.475 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:45:29.475 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:45:29.475 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:45:29.475 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:45:29.475 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:45:29.475 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:45:29.476 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:45:29.476 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:45:29.476 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:45:29.476 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:45:29.476 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.460+0000 7fc4e8f21640 1 librados: starting msgr at 2024-09-16T03:45:29.476 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.460+0000 7fc4e8f21640 1 librados: starting objecter 2024-09-16T03:45:29.476 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.460+0000 7fc4e8f21640 1 Processor -- start 2024-09-16T03:45:29.476 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.460+0000 7fc4e8f21640 1 -- start start 2024-09-16T03:45:29.476 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.460+0000 7fc4e8f21640 1 librados: setting wanted keys 2024-09-16T03:45:29.476 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.460+0000 7fc4e8f21640 1 librados: calling monclient init 2024-09-16T03:45:29.476 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.460+0000 7fc4e8f21640 10 monclient: init 2024-09-16T03:45:29.476 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.460+0000 7fc4e8f21640 10 monclient: _reopen_session rank -1 2024-09-16T03:45:29.476 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.460+0000 7fc4e8f21640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:45:29.477 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.460+0000 7fc4e8f21640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc4e4072d40 0x7fc4e41cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:45:29.477 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.460+0000 7fc4e8f21640 10 monclient(hunting): picked mon.noname-a con 0x7fc4e4072d40 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:45:29.477 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.460+0000 7fc4e8f21640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc4e409c920 0x7fc4e41cfac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:45:29.477 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.460+0000 7fc4e8f21640 10 monclient(hunting): picked mon.noname-c con 0x7fc4e409c920 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:45:29.477 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.460+0000 7fc4e8f21640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc4e40a1330 0x7fc4e413ba00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:45:29.477 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.460+0000 7fc4e8f21640 10 monclient(hunting): picked mon.noname-b con 0x7fc4e40a1330 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:45:29.477 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.460+0000 7fc4e8f21640 10 monclient(hunting): start opening mon connection 2024-09-16T03:45:29.477 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.460+0000 7fc4e8f21640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fc4e4142f00 con 0x7fc4e4072d40 2024-09-16T03:45:29.477 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.460+0000 7fc4e8f21640 10 monclient(hunting): start opening mon connection 2024-09-16T03:45:29.477 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.460+0000 7fc4e8f21640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc4e4142d80 con 0x7fc4e40a1330 2024-09-16T03:45:29.477 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.460+0000 7fc4e8f21640 10 monclient(hunting): start opening mon connection 2024-09-16T03:45:29.477 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.460+0000 7fc4e8f21640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fc4e4143080 con 0x7fc4e409c920 2024-09-16T03:45:29.477 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.460+0000 7fc4e8f21640 10 monclient(hunting): _renew_subs 2024-09-16T03:45:29.477 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.460+0000 7fc4e8f21640 10 monclient(hunting): authenticate will time out at 1176.434274s 2024-09-16T03:45:29.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.460+0000 7fc4e1d9b640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc4e409c920 0x7fc4e41cfac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:45:29.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.460+0000 7fc4e1d9b640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc4e409c920 0x7fc4e41cfac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:39142/0 (socket says 172.21.15.42:39142) 2024-09-16T03:45:29.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.460+0000 7fc4e1d9b640 1 -- 172.21.15.42:0/2775919493 learned_addr learned my addr 172.21.15.42:0/2775919493 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:45:29.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.460+0000 7fc4e1d9b640 10 monclient(hunting): get_auth_request con 0x7fc4e409c920 auth_method 0 2024-09-16T03:45:29.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.460+0000 7fc4e1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:45:29.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.460+0000 7fc4e1d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:45:29.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.460+0000 7fc4e1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:45:29.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.460+0000 7fc4e259c640 1 --2- 172.21.15.42:0/2775919493 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc4e4072d40 0x7fc4e41cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:45:29.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.460+0000 7fc4e2d9d640 1 -- 172.21.15.42:0/2775919493 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc4e40a1330 msgr2=0x7fc4e413ba00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T03:45:29.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.460+0000 7fc4e1d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:45:29.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.460+0000 7fc4e1d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:45:29.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.460+0000 7fc4e1d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:45:29.479 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.460+0000 7fc4e259c640 10 monclient(hunting): get_auth_request con 0x7fc4e4072d40 auth_method 0 2024-09-16T03:45:29.479 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.460+0000 7fc4e2d9d640 1 --2- 172.21.15.42:0/2775919493 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc4e40a1330 0x7fc4e413ba00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:45:29.479 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.460+0000 7fc4e259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:45:29.479 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.460+0000 7fc4e259c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:45:29.479 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.460+0000 7fc4e259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:45:29.479 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.460+0000 7fc4e259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:45:29.479 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.460+0000 7fc4e259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:45:29.479 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.460+0000 7fc4e259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:45:29.479 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.460+0000 7fc4e1d9b640 10 monclient(hunting): handle_auth_done global_id 13385 payload 824 2024-09-16T03:45:29.479 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.460+0000 7fc4e1d9b640 1 -- 172.21.15.42:0/2775919493 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc4e40a1330 msgr2=0x7fc4e413ba00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:45:29.479 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.460+0000 7fc4e1d9b640 1 --2- 172.21.15.42:0/2775919493 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc4e40a1330 0x7fc4e413ba00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:45:29.479 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.460+0000 7fc4e1d9b640 1 -- 172.21.15.42:0/2775919493 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc4e4072d40 msgr2=0x7fc4e41cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:45:29.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.460+0000 7fc4e1d9b640 1 --2- 172.21.15.42:0/2775919493 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc4e4072d40 0x7fc4e41cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:45:29.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.460+0000 7fc4e1d9b640 10 monclient: _finish_hunting 0 2024-09-16T03:45:29.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.460+0000 7fc4e1d9b640 1 monclient: found mon.noname-c 2024-09-16T03:45:29.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.460+0000 7fc4e1d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:45:29.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.460+0000 7fc4e1d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:45:29.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.460+0000 7fc4e1d9b640 1 -- 172.21.15.42:0/2775919493 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4e413f2b0 con 0x7fc4e409c920 2024-09-16T03:45:29.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.460+0000 7fc4e1d9b640 1 --2- 172.21.15.42:0/2775919493 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc4e409c920 0x7fc4e41cfac0 secure :-1 s=READY pgs=2375 cs=0 l=1 rev1=1 crypto rx=0x7fc4cc0b0040 tx=0x7fc4cc0195b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:45:29.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.460+0000 7fc4daffd640 10 client.?.objecter ms_handle_connect 0x7fc4e409c920 2024-09-16T03:45:29.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.460+0000 7fc4daffd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:45:29.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.460+0000 7fc4daffd640 1 -- 172.21.15.42:0/2775919493 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4cc0bd020 con 0x7fc4e409c920 2024-09-16T03:45:29.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.460+0000 7fc4daffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:45:29.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.460+0000 7fc4daffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:45:29.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.460+0000 7fc4daffd640 10 monclient: dump: 2024-09-16T03:45:29.481 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:45:29.481 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:45:29.481 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:29.481 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:29.481 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:45:29.481 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:45:29.481 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:45:29.481 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:45:29.481 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:45:29.481 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:45:29.481 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.460+0000 7fc4daffd640 10 monclient: _finish_auth 0 2024-09-16T03:45:29.481 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.460+0000 7fc4daffd640 10 monclient: _check_auth_tickets 2024-09-16T03:45:29.481 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.460+0000 7fc4daffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:45:29.482 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.460+0000 7fc4daffd640 1 -- 172.21.15.42:0/2775919493 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc4cc0ad870 con 0x7fc4e409c920 2024-09-16T03:45:29.482 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.460+0000 7fc4e8f21640 5 monclient: authenticate success, global_id 13385 2024-09-16T03:45:29.482 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.460+0000 7fc4e8f21640 10 monclient: _renew_subs 2024-09-16T03:45:29.482 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.460+0000 7fc4e8f21640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:45:29.482 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.460+0000 7fc4e8f21640 1 -- 172.21.15.42:0/2775919493 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4e411bf70 con 0x7fc4e409c920 2024-09-16T03:45:29.482 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.460+0000 7fc4daffd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:45:29.482 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.460+0000 7fc4daffd640 1 -- 172.21.15.42:0/2775919493 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4cc0b2040 con 0x7fc4e409c920 2024-09-16T03:45:29.482 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.460+0000 7fc4daffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:45:29.482 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.460+0000 7fc4daffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:45:29.482 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.460+0000 7fc4daffd640 10 monclient: dump: 2024-09-16T03:45:29.482 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:45:29.482 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:45:29.483 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:29.483 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:29.483 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:45:29.483 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:45:29.483 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:45:29.483 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:45:29.483 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:45:29.483 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:45:29.483 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.460+0000 7fc4e0d99640 4 set_mon_vals no callback set 2024-09-16T03:45:29.483 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.460+0000 7fc4e8f21640 10 client.13385.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:45:29.483 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.460+0000 7fc4e8f21640 10 monclient: _renew_subs 2024-09-16T03:45:29.483 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.460+0000 7fc4e8f21640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:45:29.483 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.460+0000 7fc4e8f21640 1 -- 172.21.15.42:0/2775919493 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4e40a3c30 con 0x7fc4e409c920 2024-09-16T03:45:29.483 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.460+0000 7fc4e8f21640 1 librados: init done 2024-09-16T03:45:29.484 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.464+0000 7fc4e8f21640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:45:29.484 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.464+0000 7fc4e8f21640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:45:29.484 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.464+0000 7fc4e8f21640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:45:29.484 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.464+0000 7fc4e8f21640 1 -- 172.21.15.42:0/2775919493 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc4e40a1710 con 0x7fc4e409c920 2024-09-16T03:45:29.484 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.464+0000 7fc4daffd640 1 -- 172.21.15.42:0/2775919493 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fc4cc0c0020 con 0x7fc4e409c920 2024-09-16T03:45:29.484 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.464+0000 7fc4daffd640 1 --2- 172.21.15.42:0/2775919493 >> v2:172.21.15.154:6812/254845342 conn(0x7fc4c40377c0 0x7fc4c4039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:45:29.484 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.464+0000 7fc4daffd640 1 -- 172.21.15.42:0/2775919493 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(90..90 src has 1..90) ==== 7319+0+0 (secure 0 0 0) 0x7fc4cc0e4790 con 0x7fc4e409c920 2024-09-16T03:45:29.484 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.464+0000 7fc4daffd640 10 client.13385.objecter ms_dispatch 0x7fc4e4000ba0 osd_map(90..90 src has 1..90) 2024-09-16T03:45:29.484 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.464+0000 7fc4daffd640 3 client.13385.objecter handle_osd_map got epochs [90,90] > 0 2024-09-16T03:45:29.484 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.464+0000 7fc4daffd640 3 client.13385.objecter handle_osd_map decoding full epoch 90 2024-09-16T03:45:29.484 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.464+0000 7fc4daffd640 20 client.13385.objecter dump_active .. 0 homeless 2024-09-16T03:45:29.484 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.464+0000 7fc4e259c640 1 --2- 172.21.15.42:0/2775919493 >> v2:172.21.15.154:6812/254845342 conn(0x7fc4c40377c0 0x7fc4c4039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:45:29.484 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.464+0000 7fc4e259c640 10 monclient: get_auth_request con 0x7fc4c40377c0 auth_method 0 2024-09-16T03:45:29.485 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.464+0000 7fc4daffd640 10 client.13385.objecter ms_handle_connect 0x7fc4c40377c0 2024-09-16T03:45:29.485 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.464+0000 7fc4e259c640 1 --2- 172.21.15.42:0/2775919493 >> v2:172.21.15.154:6812/254845342 conn(0x7fc4c40377c0 0x7fc4c4039c80 secure :-1 s=READY pgs=3369 cs=0 l=1 rev1=1 crypto rx=0x7fc4d4000f70 tx=0x7fc4d4007e90 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:45:29.485 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.468+0000 7fc4daffd640 1 -- 172.21.15.42:0/2775919493 <== mon.2 v2:172.21.15.42:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7fc4cc016520 con 0x7fc4e409c920 2024-09-16T03:45:29.485 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.468+0000 7fc4daffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:45:29.485 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.468+0000 7fc4daffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:45:29.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.696+0000 7fc4e8f21640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:45:29.702 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.696+0000 7fc4e8f21640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:45:29.702 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.696+0000 7fc4e8f21640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:45:29.702 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.696+0000 7fc4e8f21640 1 -- 172.21.15.42:0/2775919493 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fc4e4069040 con 0x7fc4e409c920 2024-09-16T03:45:29.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.696+0000 7fc4daffd640 1 -- 172.21.15.42:0/2775919493 <== mon.2 v2:172.21.15.42:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7fc4cc0b73d0 con 0x7fc4e409c920 2024-09-16T03:45:29.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.696+0000 7fc4daffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:45:29.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.696+0000 7fc4daffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:45:29.703 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":28,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:45:29.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.696+0000 7fc4e8f21640 10 librados: watch_flush enter 2024-09-16T03:45:29.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.700+0000 7fc4e8f21640 10 librados: watch_flush exit 2024-09-16T03:45:29.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.700+0000 7fc4e8f21640 20 client.13385.objecter shutdown clearing up homeless session... 2024-09-16T03:45:29.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.700+0000 7fc4e8f21640 10 client.13385.objecter successfully canceled tick 2024-09-16T03:45:29.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.700+0000 7fc4e8f21640 1 -- 172.21.15.42:0/2775919493 >> v2:172.21.15.154:6812/254845342 conn(0x7fc4c40377c0 msgr2=0x7fc4c4039c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:45:29.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.700+0000 7fc4e8f21640 1 --2- 172.21.15.42:0/2775919493 >> v2:172.21.15.154:6812/254845342 conn(0x7fc4c40377c0 0x7fc4c4039c80 secure :-1 s=READY pgs=3369 cs=0 l=1 rev1=1 crypto rx=0x7fc4d4000f70 tx=0x7fc4d4007e90 comp rx=0 tx=0).stop 2024-09-16T03:45:29.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.700+0000 7fc4e8f21640 10 monclient: shutdown 2024-09-16T03:45:29.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.700+0000 7fc4e8f21640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:45:29.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.700+0000 7fc4e8f21640 1 -- 172.21.15.42:0/2775919493 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc4e409c920 msgr2=0x7fc4e41cfac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:45:29.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.700+0000 7fc4e8f21640 1 --2- 172.21.15.42:0/2775919493 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc4e409c920 0x7fc4e41cfac0 secure :-1 s=READY pgs=2375 cs=0 l=1 rev1=1 crypto rx=0x7fc4cc0b0040 tx=0x7fc4cc0195b0 comp rx=0 tx=0).stop 2024-09-16T03:45:29.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.700+0000 7fc4e8f21640 1 -- 172.21.15.42:0/2775919493 shutdown_connections 2024-09-16T03:45:29.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.700+0000 7fc4e8f21640 1 --2- 172.21.15.42:0/2775919493 >> v2:172.21.15.154:6812/254845342 conn(0x7fc4c40377c0 0x7fc4c4039c80 unknown :-1 s=CLOSED pgs=3369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:45:29.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.700+0000 7fc4e8f21640 1 --2- 172.21.15.42:0/2775919493 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc4e40a1330 0x7fc4e413ba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:45:29.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.700+0000 7fc4e8f21640 1 --2- 172.21.15.42:0/2775919493 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc4e409c920 0x7fc4e41cfac0 unknown :-1 s=CLOSED pgs=2375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:45:29.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.700+0000 7fc4e8f21640 1 --2- 172.21.15.42:0/2775919493 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc4e4072d40 0x7fc4e41cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:45:29.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.700+0000 7fc4e8f21640 1 -- 172.21.15.42:0/2775919493 >> 172.21.15.42:0/2775919493 conn(0x7fc4e40a2420 msgr2=0x7fc4e409db90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:45:29.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.700+0000 7fc4e8f21640 1 -- 172.21.15.42:0/2775919493 shutdown_connections 2024-09-16T03:45:29.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.700+0000 7fc4e8f21640 1 -- 172.21.15.42:0/2775919493 wait complete. 2024-09-16T03:45:29.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:29.700+0000 7fc4e8f21640 1 librados: shutdown 2024-09-16T03:45:32.723 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:45:32.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:32.848+0000 7fca0be95640 10 monclient: get_monmap_and_config 2024-09-16T03:45:32.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:32.848+0000 7fca0be95640 10 monclient: build_initial_monmap 2024-09-16T03:45:32.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:32.848+0000 7fca0be95640 10 monclient: monmap: 2024-09-16T03:45:32.855 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:45:32.856 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:45:32.856 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:45:32.854768+0000 2024-09-16T03:45:32.856 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:45:32.854768+0000 2024-09-16T03:45:32.856 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:45:32.856 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:45:32.856 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:45:32.856 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:45:32.856 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:45:32.856 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:45:32.856 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:45:32.857 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:45:32.858 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:32.848+0000 7fca0be95640 1 Processor -- start 2024-09-16T03:45:32.859 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:32.848+0000 7fca0be95640 1 -- start start 2024-09-16T03:45:32.859 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:32.848+0000 7fca0be95640 10 monclient: init 2024-09-16T03:45:32.859 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:32.852+0000 7fca0be95640 10 monclient: _reopen_session rank -1 2024-09-16T03:45:32.859 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:32.852+0000 7fca0be95640 10 monclient: _add_conns ranks=[2,0,4,3,1] 2024-09-16T03:45:32.859 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:32.852+0000 7fca0be95640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fca040a1560 0x7fca040a1940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:45:32.859 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:32.852+0000 7fca0be95640 10 monclient(hunting): picked mon.noname-b con 0x7fca040a1560 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:45:32.859 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:32.852+0000 7fca0be95640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fca040a0bc0 addr v1:172.21.15.42:6789/0 2024-09-16T03:45:32.859 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:32.852+0000 7fca0be95640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fca0413a340 0x7fca0413c730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:45:32.859 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:32.852+0000 7fca0be95640 10 monclient(hunting): picked mon.noname-c con 0x7fca0413a340 addr v2:172.21.15.154:3300/0 2024-09-16T03:45:32.859 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:32.852+0000 7fca0be95640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fca040a2ae0 con 0x7fca040a0bc0 2024-09-16T03:45:32.859 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:32.852+0000 7fca0be95640 10 monclient(hunting): start opening mon connection 2024-09-16T03:45:32.859 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:32.852+0000 7fca0be95640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fca04071220 con 0x7fca0413a340 2024-09-16T03:45:32.860 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:32.852+0000 7fca0be95640 10 monclient(hunting): start opening mon connection 2024-09-16T03:45:32.860 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:32.852+0000 7fca0be95640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fca04071500 con 0x7fca040a1560 2024-09-16T03:45:32.860 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:32.852+0000 7fca0be95640 10 monclient(hunting): _renew_subs 2024-09-16T03:45:32.860 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:32.852+0000 7fca0be95640 10 monclient(hunting): authenticate will time out at 1179.824503s 2024-09-16T03:45:32.860 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:32.852+0000 7fca0a432640 1 -- >> v2:172.21.15.154:3300/0 conn(0x7fca0413a340 msgr2=0x7fca0413c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T03:45:32.860 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:32.852+0000 7fca0a432640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fca0413a340 0x7fca0413c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:45:32.860 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:32.852+0000 7fca09430640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fca040a0bc0 0x7fca040a1e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:54264/0 (socket says 172.21.15.42:54264) 2024-09-16T03:45:32.860 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:32.852+0000 7fca09c31640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fca040a1560 0x7fca040a1940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:45:32.860 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:32.852+0000 7fca09430640 1 -- 172.21.15.42:0/1492721511 learned_addr learned my addr 172.21.15.42:0/1492721511 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:45:32.861 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:32.852+0000 7fca09c31640 10 monclient(hunting): get_auth_request con 0x7fca040a1560 auth_method 0 2024-09-16T03:45:32.861 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:32.852+0000 7fca09c31640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:45:32.861 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:32.852+0000 7fca09c31640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:45:32.861 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:32.852+0000 7fca09c31640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:45:32.861 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:32.852+0000 7fca09c31640 1 -- 172.21.15.42:0/1492721511 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fca040a1560 msgr2=0x7fca040a1940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T03:45:32.861 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:32.852+0000 7fca09c31640 1 -- 172.21.15.42:0/1492721511 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fca040a1560 msgr2=0x7fca040a1940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:45:32.862 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:32.852+0000 7fca09c31640 1 --2- 172.21.15.42:0/1492721511 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fca040a1560 0x7fca040a1940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:45:32.862 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:32.852+0000 7fca09c31640 1 --2- 172.21.15.42:0/1492721511 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fca040a1560 0x7fca040a1940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:45:33.061 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:33.056+0000 7fca0a432640 1 -- 172.21.15.42:0/1492721511 >> v2:172.21.15.154:3300/0 conn(0x7fca0413a340 msgr2=0x7fca0413c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T03:45:33.061 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:33.056+0000 7fca09c31640 1 --2- 172.21.15.42:0/1492721511 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fca040a1560 0x7fca040a1940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:45:33.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:33.056+0000 7fca0a432640 1 --2- 172.21.15.42:0/1492721511 >> v2:172.21.15.154:3300/0 conn(0x7fca0413a340 0x7fca0413c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T03:45:33.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:33.056+0000 7fca09c31640 10 monclient(hunting): get_auth_request con 0x7fca040a1560 auth_method 0 2024-09-16T03:45:33.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:33.056+0000 7fca09c31640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:45:33.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:33.056+0000 7fca09c31640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:45:33.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:33.056+0000 7fca09c31640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:45:33.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:33.056+0000 7fca09c31640 1 -- 172.21.15.42:0/1492721511 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fca040a1560 msgr2=0x7fca040a1940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T03:45:33.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:33.056+0000 7fca09c31640 1 -- 172.21.15.42:0/1492721511 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fca040a1560 msgr2=0x7fca040a1940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:45:33.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:33.056+0000 7fca09c31640 1 --2- 172.21.15.42:0/1492721511 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fca040a1560 0x7fca040a1940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:45:33.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:33.056+0000 7fca09c31640 1 --2- 172.21.15.42:0/1492721511 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fca040a1560 0x7fca040a1940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T03:45:33.465 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:33.460+0000 7fca0a432640 1 -- 172.21.15.42:0/1492721511 >> v2:172.21.15.154:3300/0 conn(0x7fca0413a340 msgr2=0x7fca0413c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T03:45:33.466 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:33.460+0000 7fca0a432640 1 --2- 172.21.15.42:0/1492721511 >> v2:172.21.15.154:3300/0 conn(0x7fca0413a340 0x7fca0413c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T03:45:33.466 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:33.460+0000 7fca09c31640 1 --2- 172.21.15.42:0/1492721511 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fca040a1560 0x7fca040a1940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:45:33.466 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:33.460+0000 7fca09c31640 10 monclient(hunting): get_auth_request con 0x7fca040a1560 auth_method 0 2024-09-16T03:45:33.466 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:33.460+0000 7fca09c31640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:45:33.466 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:33.460+0000 7fca09c31640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:45:33.466 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:33.460+0000 7fca09c31640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:45:33.466 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:33.460+0000 7fca09c31640 1 -- 172.21.15.42:0/1492721511 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fca040a1560 msgr2=0x7fca040a1940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T03:45:33.466 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:33.460+0000 7fca09c31640 1 -- 172.21.15.42:0/1492721511 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fca040a1560 msgr2=0x7fca040a1940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:45:33.466 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:33.460+0000 7fca09c31640 1 --2- 172.21.15.42:0/1492721511 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fca040a1560 0x7fca040a1940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:45:33.466 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:33.460+0000 7fca09c31640 1 --2- 172.21.15.42:0/1492721511 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fca040a1560 0x7fca040a1940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T03:45:34.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:34.264+0000 7fca09c31640 1 --2- 172.21.15.42:0/1492721511 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fca040a1560 0x7fca040a1940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:45:34.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:34.264+0000 7fca09c31640 10 monclient(hunting): get_auth_request con 0x7fca040a1560 auth_method 0 2024-09-16T03:45:34.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:34.264+0000 7fca09c31640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:45:34.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:34.264+0000 7fca09c31640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:45:34.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:34.264+0000 7fca09c31640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:45:34.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:34.264+0000 7fca0a432640 1 -- 172.21.15.42:0/1492721511 >> v2:172.21.15.154:3300/0 conn(0x7fca0413a340 msgr2=0x7fca0413c730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T03:45:34.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:34.264+0000 7fca0a432640 1 --2- 172.21.15.42:0/1492721511 >> v2:172.21.15.154:3300/0 conn(0x7fca0413a340 0x7fca0413c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T03:45:34.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:34.264+0000 7fca09c31640 1 -- 172.21.15.42:0/1492721511 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fca040a1560 msgr2=0x7fca040a1940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T03:45:34.271 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:34.264+0000 7fca09c31640 1 -- 172.21.15.42:0/1492721511 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fca040a1560 msgr2=0x7fca040a1940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:45:34.271 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:34.264+0000 7fca09c31640 1 --2- 172.21.15.42:0/1492721511 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fca040a1560 0x7fca040a1940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:45:34.271 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:34.264+0000 7fca09c31640 1 --2- 172.21.15.42:0/1492721511 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fca040a1560 0x7fca040a1940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T03:45:35.858 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:35.852+0000 7fca037fe640 10 monclient(hunting): tick 2024-09-16T03:45:35.858 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:35.852+0000 7fca037fe640 10 monclient(hunting): _check_auth_tickets 2024-09-16T03:45:35.858 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:35.852+0000 7fca037fe640 1 monclient(hunting): continuing hunt 2024-09-16T03:45:35.858 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:35.852+0000 7fca037fe640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T03:45:35.858 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:35.852+0000 7fca037fe640 1 -- 172.21.15.42:0/1492721511 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fca040a1560 msgr2=0x7fca040a1940 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:45:35.858 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:35.852+0000 7fca037fe640 1 --2- 172.21.15.42:0/1492721511 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fca040a1560 0x7fca040a1940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:45:35.858 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:35.852+0000 7fca037fe640 1 -- 172.21.15.42:0/1492721511 >> v2:172.21.15.154:3300/0 conn(0x7fca0413a340 msgr2=0x7fca0413c730 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:45:35.859 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:35.852+0000 7fca037fe640 1 --2- 172.21.15.42:0/1492721511 >> v2:172.21.15.154:3300/0 conn(0x7fca0413a340 0x7fca0413c730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:45:35.859 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:35.852+0000 7fca037fe640 1 -- 172.21.15.42:0/1492721511 >> v1:172.21.15.42:6789/0 conn(0x7fca040a0bc0 legacy=0x7fca040a1e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:45:35.859 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:35.852+0000 7fca037fe640 10 monclient: _add_conns ranks=[3,1] 2024-09-16T03:45:35.859 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:35.852+0000 7fca037fe640 1 --2- 172.21.15.42:0/1492721511 >> v2:172.21.15.42:3300/0 conn(0x7fc9f0003720 0x7fc9f0005b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:45:35.859 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:35.852+0000 7fca037fe640 10 monclient(hunting): picked mon.noname-a con 0x7fc9f0003720 addr v2:172.21.15.42:3300/0 2024-09-16T03:45:35.859 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:35.852+0000 7fca037fe640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc9f0006270 addr v1:172.21.15.154:6789/0 2024-09-16T03:45:35.859 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:35.852+0000 7fca037fe640 1 -- 172.21.15.42:0/1492721511 --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fca040717e0 con 0x7fc9f0006270 2024-09-16T03:45:35.859 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:35.852+0000 7fca037fe640 10 monclient(hunting): start opening mon connection 2024-09-16T03:45:35.859 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:35.852+0000 7fca037fe640 1 -- 172.21.15.42:0/1492721511 --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7fca04071220 con 0x7fc9f0003720 2024-09-16T03:45:35.859 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:35.852+0000 7fca037fe640 10 monclient(hunting): _renew_subs 2024-09-16T03:45:35.859 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:35.852+0000 7fca09430640 1 -- 172.21.15.42:0/1492721511 >> v1:172.21.15.154:6789/0 conn(0x7fc9f0006270 legacy=0x7fc9f0008730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.154:6789/0 2024-09-16T03:45:35.859 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:35.852+0000 7fca09c31640 1 --2- 172.21.15.42:0/1492721511 >> v2:172.21.15.42:3300/0 conn(0x7fc9f0003720 0x7fc9f0005b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:45:35.859 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:35.852+0000 7fca09c31640 10 monclient(hunting): get_auth_request con 0x7fc9f0003720 auth_method 0 2024-09-16T03:45:35.859 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:35.852+0000 7fca09c31640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:45:35.860 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:35.852+0000 7fca09c31640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:45:35.860 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:35.852+0000 7fca09c31640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:45:35.860 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:35.852+0000 7fca09c31640 1 -- 172.21.15.42:0/1492721511 >> v2:172.21.15.42:3300/0 conn(0x7fc9f0003720 msgr2=0x7fc9f0005b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T03:45:35.860 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:35.852+0000 7fca09c31640 1 -- 172.21.15.42:0/1492721511 >> v2:172.21.15.42:3300/0 conn(0x7fc9f0003720 msgr2=0x7fc9f0005b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:45:35.860 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:35.852+0000 7fca09c31640 1 --2- 172.21.15.42:0/1492721511 >> v2:172.21.15.42:3300/0 conn(0x7fc9f0003720 0x7fc9f0005b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:45:35.860 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:35.852+0000 7fca09c31640 1 --2- 172.21.15.42:0/1492721511 >> v2:172.21.15.42:3300/0 conn(0x7fc9f0003720 0x7fc9f0005b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:45:36.059 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:36.056+0000 7fca09430640 1 -- 172.21.15.42:0/1492721511 >> v1:172.21.15.154:6789/0 conn(0x7fc9f0006270 legacy=0x7fc9f0008730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.154:6789/0 2024-09-16T03:45:36.059 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:36.056+0000 7fca09c31640 1 --2- 172.21.15.42:0/1492721511 >> v2:172.21.15.42:3300/0 conn(0x7fc9f0003720 0x7fc9f0005b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:45:36.059 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:36.056+0000 7fca09c31640 10 monclient(hunting): get_auth_request con 0x7fc9f0003720 auth_method 0 2024-09-16T03:45:36.059 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:36.056+0000 7fca09c31640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:45:36.059 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:36.056+0000 7fca09c31640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:45:36.059 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:36.056+0000 7fca09c31640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:45:36.059 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:36.056+0000 7fca09c31640 1 -- 172.21.15.42:0/1492721511 >> v2:172.21.15.42:3300/0 conn(0x7fc9f0003720 msgr2=0x7fc9f0005b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T03:45:36.059 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:36.056+0000 7fca09c31640 1 -- 172.21.15.42:0/1492721511 >> v2:172.21.15.42:3300/0 conn(0x7fc9f0003720 msgr2=0x7fc9f0005b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:45:36.060 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:36.056+0000 7fca09c31640 1 --2- 172.21.15.42:0/1492721511 >> v2:172.21.15.42:3300/0 conn(0x7fc9f0003720 0x7fc9f0005b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:45:36.060 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:36.056+0000 7fca09c31640 1 --2- 172.21.15.42:0/1492721511 >> v2:172.21.15.42:3300/0 conn(0x7fc9f0003720 0x7fc9f0005b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T03:45:36.465 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:36.460+0000 7fca09430640 1 -- 172.21.15.42:0/1492721511 >> v1:172.21.15.154:6789/0 conn(0x7fc9f0006270 legacy=0x7fc9f0008730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.154:6789/0 2024-09-16T03:45:36.465 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:36.460+0000 7fca09c31640 1 --2- 172.21.15.42:0/1492721511 >> v2:172.21.15.42:3300/0 conn(0x7fc9f0003720 0x7fc9f0005b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:45:36.465 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:36.460+0000 7fca09c31640 10 monclient(hunting): get_auth_request con 0x7fc9f0003720 auth_method 0 2024-09-16T03:45:36.465 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:36.460+0000 7fca09c31640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:45:36.466 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:36.460+0000 7fca09c31640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:45:36.466 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:36.460+0000 7fca09c31640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:45:36.466 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:36.460+0000 7fca09c31640 1 -- 172.21.15.42:0/1492721511 >> v2:172.21.15.42:3300/0 conn(0x7fc9f0003720 msgr2=0x7fc9f0005b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T03:45:36.466 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:36.460+0000 7fca09c31640 1 -- 172.21.15.42:0/1492721511 >> v2:172.21.15.42:3300/0 conn(0x7fc9f0003720 msgr2=0x7fc9f0005b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:45:36.466 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:36.460+0000 7fca09c31640 1 --2- 172.21.15.42:0/1492721511 >> v2:172.21.15.42:3300/0 conn(0x7fc9f0003720 0x7fc9f0005b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:45:36.466 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:36.460+0000 7fca09c31640 1 --2- 172.21.15.42:0/1492721511 >> v2:172.21.15.42:3300/0 conn(0x7fc9f0003720 0x7fc9f0005b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T03:45:37.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.264+0000 7fca09430640 1 -- 172.21.15.42:0/1492721511 >> v1:172.21.15.154:6789/0 conn(0x7fc9f0006270 legacy=0x7fc9f0008730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.154:6789/0 2024-09-16T03:45:37.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.264+0000 7fca09c31640 1 --2- 172.21.15.42:0/1492721511 >> v2:172.21.15.42:3300/0 conn(0x7fc9f0003720 0x7fc9f0005b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:45:37.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.264+0000 7fca09c31640 10 monclient(hunting): get_auth_request con 0x7fc9f0003720 auth_method 0 2024-09-16T03:45:37.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.264+0000 7fca09c31640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:45:37.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.264+0000 7fca09c31640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:45:37.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.264+0000 7fca09c31640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:45:37.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.264+0000 7fca09c31640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:45:37.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.264+0000 7fca09c31640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:45:37.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.264+0000 7fca09c31640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:45:37.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.264+0000 7fca09c31640 10 monclient(hunting): handle_auth_done global_id 744216 payload 274 2024-09-16T03:45:37.272 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.264+0000 7fca09c31640 1 -- 172.21.15.42:0/1492721511 >> v1:172.21.15.154:6789/0 conn(0x7fc9f0006270 legacy=0x7fc9f0008730 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:45:37.272 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.264+0000 7fca09c31640 10 monclient: _finish_hunting 0 2024-09-16T03:45:37.272 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.264+0000 7fca09c31640 1 monclient: found mon.noname-a 2024-09-16T03:45:37.272 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.264+0000 7fca09c31640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:45:37.272 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.264+0000 7fca09c31640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:45:37.272 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.264+0000 7fca09c31640 1 -- 172.21.15.42:0/1492721511 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9f000afe0 con 0x7fc9f0003720 2024-09-16T03:45:37.275 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.268+0000 7fca09c31640 1 --2- 172.21.15.42:0/1492721511 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc9f0003720 0x7fc9f0005b80 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fc9f4000ee0 tx=0x7fc9f401de00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=769eec509cdf42b0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:45:37.276 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.268+0000 7fca08c2f640 1 -- 172.21.15.42:0/1492721511 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9f40ac070 con 0x7fc9f0003720 2024-09-16T03:45:37.276 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.268+0000 7fca08c2f640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:45:37.276 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.268+0000 7fca08c2f640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:45:37.276 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.268+0000 7fca08c2f640 10 monclient: dump: 2024-09-16T03:45:37.276 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:45:37.277 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:45:37.277 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:37.277 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:37.277 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:45:37.277 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:45:37.277 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:45:37.277 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:45:37.277 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:45:37.277 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:45:37.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.268+0000 7fca08c2f640 10 monclient: _finish_auth 0 2024-09-16T03:45:37.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.268+0000 7fca08c2f640 10 monclient: _check_auth_tickets 2024-09-16T03:45:37.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.268+0000 7fca08c2f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:45:37.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.268+0000 7fca08c2f640 1 -- 172.21.15.42:0/1492721511 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc9f40a6c40 con 0x7fc9f0003720 2024-09-16T03:45:37.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.268+0000 7fca0be95640 5 monclient: authenticate success, global_id 744216 2024-09-16T03:45:37.278 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.268+0000 7fca0be95640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:45:37.278 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.268+0000 7fca08c2f640 10 monclient: handle_config config(0 keys) 2024-09-16T03:45:37.278 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.268+0000 7fca0be95640 10 monclient: get_monmap_and_config success 2024-09-16T03:45:37.278 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.268+0000 7fca0be95640 4 set_mon_vals no callback set 2024-09-16T03:45:37.278 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.268+0000 7fca08c2f640 1 -- 172.21.15.42:0/1492721511 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9f400f040 con 0x7fc9f0003720 2024-09-16T03:45:37.278 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.268+0000 7fca08c2f640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:45:37.278 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.268+0000 7fca08c2f640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:45:37.278 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.268+0000 7fca08c2f640 10 monclient: dump: 2024-09-16T03:45:37.278 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:45:37.278 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:45:37.278 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:37.278 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:37.278 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:45:37.279 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:45:37.279 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:45:37.279 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:45:37.279 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:45:37.279 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:45:37.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.272+0000 7fca0be95640 10 monclient: shutdown 2024-09-16T03:45:37.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.272+0000 7fca0be95640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:45:37.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.272+0000 7fca0be95640 1 -- 172.21.15.42:0/1492721511 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc9f0003720 msgr2=0x7fc9f0005b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:45:37.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.272+0000 7fca0be95640 1 --2- 172.21.15.42:0/1492721511 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc9f0003720 0x7fc9f0005b80 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fc9f4000ee0 tx=0x7fc9f401de00 comp rx=0 tx=0).stop 2024-09-16T03:45:37.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.272+0000 7fca0a432640 1 -- 172.21.15.42:0/1492721511 reap_dead start 2024-09-16T03:45:37.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.272+0000 7fca0be95640 1 -- 172.21.15.42:0/1492721511 shutdown_connections 2024-09-16T03:45:37.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.272+0000 7fca0be95640 1 --2- 172.21.15.42:0/1492721511 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc9f0003720 0x7fc9f0005b80 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:45:37.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.272+0000 7fca0be95640 1 -- 172.21.15.42:0/1492721511 >> 172.21.15.42:0/1492721511 conn(0x7fca04075ae0 msgr2=0x7fca04075ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:45:37.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.272+0000 7fca0be95640 1 -- 172.21.15.42:0/1492721511 shutdown_connections 2024-09-16T03:45:37.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.272+0000 7fca0be95640 1 -- 172.21.15.42:0/1492721511 wait complete. 2024-09-16T03:45:37.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.272+0000 7fca0be95640 10 monclient: build_initial_monmap 2024-09-16T03:45:37.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.272+0000 7fca0be95640 10 monclient: monmap: 2024-09-16T03:45:37.285 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:45:37.285 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:45:37.285 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:45:37.285 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:45:37.285 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:45:37.285 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:45:37.285 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:45:37.285 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:45:37.285 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:45:37.286 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:45:37.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.272+0000 7fca0be95640 1 librados: starting msgr at 2024-09-16T03:45:37.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.272+0000 7fca0be95640 1 librados: starting objecter 2024-09-16T03:45:37.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.272+0000 7fca0be95640 1 Processor -- start 2024-09-16T03:45:37.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.272+0000 7fca0be95640 1 -- start start 2024-09-16T03:45:37.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.272+0000 7fca0be95640 1 librados: setting wanted keys 2024-09-16T03:45:37.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.272+0000 7fca0be95640 1 librados: calling monclient init 2024-09-16T03:45:37.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.272+0000 7fca0be95640 10 monclient: init 2024-09-16T03:45:37.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.272+0000 7fca0be95640 10 monclient: _reopen_session rank -1 2024-09-16T03:45:37.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.272+0000 7fca0be95640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T03:45:37.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.272+0000 7fca0be95640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc9f0003720 0x7fca0406eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:45:37.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.272+0000 7fca0be95640 10 monclient(hunting): picked mon.noname-b con 0x7fc9f0003720 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:45:37.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.272+0000 7fca0be95640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fca0406f0c0 0x7fca0406f520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:45:37.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.272+0000 7fca0be95640 10 monclient(hunting): picked mon.noname-c con 0x7fca0406f0c0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:45:37.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.272+0000 7fca0be95640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fca0406fa60 0x7fca041d9ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:45:37.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.272+0000 7fca0be95640 10 monclient(hunting): picked mon.noname-a con 0x7fca0406fa60 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:45:37.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.272+0000 7fca0be95640 10 monclient(hunting): start opening mon connection 2024-09-16T03:45:37.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.272+0000 7fca0be95640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fca0413f820 con 0x7fca0406fa60 2024-09-16T03:45:37.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.272+0000 7fca0be95640 10 monclient(hunting): start opening mon connection 2024-09-16T03:45:37.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.272+0000 7fca0be95640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fca0413f6a0 con 0x7fc9f0003720 2024-09-16T03:45:37.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.272+0000 7fca0be95640 10 monclient(hunting): start opening mon connection 2024-09-16T03:45:37.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.272+0000 7fca0be95640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fca0413f9a0 con 0x7fca0406f0c0 2024-09-16T03:45:37.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.272+0000 7fca0be95640 10 monclient(hunting): _renew_subs 2024-09-16T03:45:37.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.272+0000 7fca0be95640 10 monclient(hunting): authenticate will time out at 1184.247396s 2024-09-16T03:45:37.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.272+0000 7fca09c31640 1 -- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc9f0003720 msgr2=0x7fca0406eb80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T03:45:37.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.272+0000 7fca09c31640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc9f0003720 0x7fca0406eb80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:45:37.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.272+0000 7fca0a432640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fca0406fa60 0x7fca041d9ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:45:37.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.272+0000 7fca09430640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fca0406f0c0 0x7fca0406f520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:45:37.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.272+0000 7fca09430640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fca0406f0c0 0x7fca0406f520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:33970/0 (socket says 172.21.15.42:33970) 2024-09-16T03:45:37.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.272+0000 7fca09430640 1 -- 172.21.15.42:0/1676646491 learned_addr learned my addr 172.21.15.42:0/1676646491 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:45:37.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.272+0000 7fca09430640 10 monclient(hunting): get_auth_request con 0x7fca0406f0c0 auth_method 0 2024-09-16T03:45:37.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.272+0000 7fca09430640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:45:37.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.272+0000 7fca09430640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:45:37.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.272+0000 7fca09430640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:45:37.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.272+0000 7fca0a432640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fca0406fa60 0x7fca041d9ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:42708/0 (socket says 172.21.15.42:42708) 2024-09-16T03:45:37.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.272+0000 7fca0a432640 10 monclient(hunting): get_auth_request con 0x7fca0406fa60 auth_method 0 2024-09-16T03:45:37.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.272+0000 7fca0a432640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:45:37.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.272+0000 7fca0a432640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:45:37.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.272+0000 7fca0a432640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:45:37.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.272+0000 7fca09430640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:45:37.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.272+0000 7fca09430640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:45:37.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.272+0000 7fca09430640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:45:37.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.272+0000 7fca0a432640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:45:37.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.272+0000 7fca0a432640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:45:37.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.272+0000 7fca0a432640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:45:37.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.276+0000 7fca09430640 10 monclient(hunting): handle_auth_done global_id 13388 payload 824 2024-09-16T03:45:37.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.276+0000 7fca09430640 1 -- 172.21.15.42:0/1676646491 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc9f0003720 msgr2=0x7fca0406eb80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:45:37.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.276+0000 7fca09430640 1 --2- 172.21.15.42:0/1676646491 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc9f0003720 0x7fca0406eb80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:45:37.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.276+0000 7fca09430640 1 -- 172.21.15.42:0/1676646491 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fca0406fa60 msgr2=0x7fca041d9ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:45:37.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.276+0000 7fca09430640 1 --2- 172.21.15.42:0/1676646491 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fca0406fa60 0x7fca041d9ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:45:37.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.276+0000 7fca09430640 10 monclient: _finish_hunting 0 2024-09-16T03:45:37.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.276+0000 7fca09430640 1 monclient: found mon.noname-c 2024-09-16T03:45:37.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.276+0000 7fca09430640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:45:37.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.276+0000 7fca09430640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:45:37.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.276+0000 7fca09430640 1 -- 172.21.15.42:0/1676646491 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca04139a80 con 0x7fca0406f0c0 2024-09-16T03:45:37.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.276+0000 7fca0a432640 1 --2- 172.21.15.42:0/1676646491 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fca0406fa60 0x7fca041d9ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:45:37.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.276+0000 7fca09430640 1 --2- 172.21.15.42:0/1676646491 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fca0406f0c0 0x7fca0406f520 secure :-1 s=READY pgs=2376 cs=0 l=1 rev1=1 crypto rx=0x7fc9f800c500 tx=0x7fc9f800c9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:45:37.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.276+0000 7fca027fc640 10 client.?.objecter ms_handle_connect 0x7fca0406f0c0 2024-09-16T03:45:37.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.276+0000 7fca027fc640 10 client.?.objecter resend_mon_ops 2024-09-16T03:45:37.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.276+0000 7fca027fc640 1 -- 172.21.15.42:0/1676646491 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9f8017070 con 0x7fca0406f0c0 2024-09-16T03:45:37.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.276+0000 7fca027fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:45:37.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.276+0000 7fca027fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:45:37.291 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.276+0000 7fca027fc640 10 monclient: dump: 2024-09-16T03:45:37.291 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:45:37.291 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:45:37.291 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:37.291 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:37.291 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:45:37.291 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:45:37.291 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:45:37.291 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:45:37.291 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:45:37.291 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:45:37.291 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.276+0000 7fca027fc640 10 monclient: _finish_auth 0 2024-09-16T03:45:37.291 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.276+0000 7fca027fc640 10 monclient: _check_auth_tickets 2024-09-16T03:45:37.292 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.276+0000 7fca027fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:45:37.292 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.276+0000 7fca027fc640 1 -- 172.21.15.42:0/1676646491 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc9f8005cf0 con 0x7fca0406f0c0 2024-09-16T03:45:37.292 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.276+0000 7fca0be95640 5 monclient: authenticate success, global_id 13388 2024-09-16T03:45:37.292 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.276+0000 7fca0be95640 10 monclient: _renew_subs 2024-09-16T03:45:37.292 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.276+0000 7fca0be95640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:45:37.292 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.276+0000 7fca0be95640 1 -- 172.21.15.42:0/1676646491 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca041360b0 con 0x7fca0406f0c0 2024-09-16T03:45:37.292 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.276+0000 7fca027fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:45:37.292 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.276+0000 7fca027fc640 1 -- 172.21.15.42:0/1676646491 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9f80072f0 con 0x7fca0406f0c0 2024-09-16T03:45:37.292 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.276+0000 7fca027fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:45:37.293 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.276+0000 7fca027fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:45:37.293 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.276+0000 7fca027fc640 10 monclient: dump: 2024-09-16T03:45:37.293 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:45:37.293 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:45:37.293 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:37.293 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:37.293 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:45:37.293 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:45:37.293 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:45:37.293 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:45:37.293 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:45:37.293 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:45:37.293 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.276+0000 7fca037fe640 4 set_mon_vals no callback set 2024-09-16T03:45:37.294 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.276+0000 7fca0be95640 10 client.13388.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:45:37.294 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.276+0000 7fca0be95640 10 monclient: _renew_subs 2024-09-16T03:45:37.294 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.276+0000 7fca0be95640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:45:37.294 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.276+0000 7fca0be95640 1 -- 172.21.15.42:0/1676646491 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fca040705c0 con 0x7fca0406f0c0 2024-09-16T03:45:37.294 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.276+0000 7fca0be95640 1 librados: init done 2024-09-16T03:45:37.294 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.276+0000 7fca0be95640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:45:37.294 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.276+0000 7fca0be95640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:45:37.294 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.276+0000 7fca0be95640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:45:37.294 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.276+0000 7fca0be95640 1 -- 172.21.15.42:0/1676646491 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fca040a1940 con 0x7fca0406f0c0 2024-09-16T03:45:37.294 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.276+0000 7fca027fc640 1 -- 172.21.15.42:0/1676646491 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fc9f801a070 con 0x7fca0406f0c0 2024-09-16T03:45:37.294 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.276+0000 7fca027fc640 1 --2- 172.21.15.42:0/1676646491 >> v2:172.21.15.154:6812/254845342 conn(0x7fc9f0038620 0x7fc9f003aae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:45:37.294 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.276+0000 7fca027fc640 1 -- 172.21.15.42:0/1676646491 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(91..91 src has 1..91) ==== 7832+0+0 (secure 0 0 0) 0x7fc9f8048ab0 con 0x7fca0406f0c0 2024-09-16T03:45:37.294 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.276+0000 7fca027fc640 10 client.13388.objecter ms_dispatch 0x7fca04000ba0 osd_map(91..91 src has 1..91) 2024-09-16T03:45:37.295 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.276+0000 7fca027fc640 3 client.13388.objecter handle_osd_map got epochs [91,91] > 0 2024-09-16T03:45:37.295 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.276+0000 7fca027fc640 3 client.13388.objecter handle_osd_map decoding full epoch 91 2024-09-16T03:45:37.295 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.276+0000 7fca027fc640 20 client.13388.objecter dump_active .. 0 homeless 2024-09-16T03:45:37.295 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.276+0000 7fca09c31640 1 --2- 172.21.15.42:0/1676646491 >> v2:172.21.15.154:6812/254845342 conn(0x7fc9f0038620 0x7fc9f003aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:45:37.295 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.276+0000 7fca09c31640 10 monclient: get_auth_request con 0x7fc9f0038620 auth_method 0 2024-09-16T03:45:37.295 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.276+0000 7fca09c31640 1 --2- 172.21.15.42:0/1676646491 >> v2:172.21.15.154:6812/254845342 conn(0x7fc9f0038620 0x7fc9f003aae0 secure :-1 s=READY pgs=3370 cs=0 l=1 rev1=1 crypto rx=0x7fc9f40af000 tx=0x7fc9f40a8070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:45:37.295 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.276+0000 7fca027fc640 10 client.13388.objecter ms_handle_connect 0x7fc9f0038620 2024-09-16T03:45:37.295 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.284+0000 7fca027fc640 1 -- 172.21.15.42:0/1676646491 <== mon.2 v2:172.21.15.42:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7fc9f8019e10 con 0x7fca0406f0c0 2024-09-16T03:45:37.296 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.284+0000 7fca027fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:45:37.296 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.284+0000 7fca027fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:45:37.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.508+0000 7fca0be95640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:45:37.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.508+0000 7fca0be95640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:45:37.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.508+0000 7fca0be95640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:45:37.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.508+0000 7fca0be95640 1 -- 172.21.15.42:0/1676646491 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fca04132e40 con 0x7fca0406f0c0 2024-09-16T03:45:37.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.508+0000 7fca027fc640 1 -- 172.21.15.42:0/1676646491 <== mon.2 v2:172.21.15.42:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1554 (secure 0 0 0) 0x7fc9f8007490 con 0x7fca0406f0c0 2024-09-16T03:45:37.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.508+0000 7fca027fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:45:37.516 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.508+0000 7fca027fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:45:37.516 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":32,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:45:37.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.508+0000 7fca0be95640 10 librados: watch_flush enter 2024-09-16T03:45:37.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.508+0000 7fca0be95640 10 librados: watch_flush exit 2024-09-16T03:45:37.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.508+0000 7fca0be95640 20 client.13388.objecter shutdown clearing up homeless session... 2024-09-16T03:45:37.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.508+0000 7fca0be95640 10 client.13388.objecter successfully canceled tick 2024-09-16T03:45:37.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.508+0000 7fca0be95640 1 -- 172.21.15.42:0/1676646491 >> v2:172.21.15.154:6812/254845342 conn(0x7fc9f0038620 msgr2=0x7fc9f003aae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:45:37.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.508+0000 7fca0be95640 1 --2- 172.21.15.42:0/1676646491 >> v2:172.21.15.154:6812/254845342 conn(0x7fc9f0038620 0x7fc9f003aae0 secure :-1 s=READY pgs=3370 cs=0 l=1 rev1=1 crypto rx=0x7fc9f40af000 tx=0x7fc9f40a8070 comp rx=0 tx=0).stop 2024-09-16T03:45:37.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.508+0000 7fca0be95640 10 monclient: shutdown 2024-09-16T03:45:37.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.508+0000 7fca0be95640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:45:37.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.508+0000 7fca0be95640 1 -- 172.21.15.42:0/1676646491 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fca0406f0c0 msgr2=0x7fca0406f520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:45:37.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.508+0000 7fca0be95640 1 --2- 172.21.15.42:0/1676646491 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fca0406f0c0 0x7fca0406f520 secure :-1 s=READY pgs=2376 cs=0 l=1 rev1=1 crypto rx=0x7fc9f800c500 tx=0x7fc9f800c9c0 comp rx=0 tx=0).stop 2024-09-16T03:45:37.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.508+0000 7fca0be95640 1 -- 172.21.15.42:0/1676646491 shutdown_connections 2024-09-16T03:45:37.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.508+0000 7fca0be95640 1 --2- 172.21.15.42:0/1676646491 >> v2:172.21.15.154:6812/254845342 conn(0x7fc9f0038620 0x7fc9f003aae0 unknown :-1 s=CLOSED pgs=3370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:45:37.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.508+0000 7fca0be95640 1 --2- 172.21.15.42:0/1676646491 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fca0406fa60 0x7fca041d9ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:45:37.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.508+0000 7fca0be95640 1 --2- 172.21.15.42:0/1676646491 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fca0406f0c0 0x7fca0406f520 unknown :-1 s=CLOSED pgs=2376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:45:37.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.508+0000 7fca0be95640 1 --2- 172.21.15.42:0/1676646491 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc9f0003720 0x7fca0406eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:45:37.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.508+0000 7fca0be95640 1 -- 172.21.15.42:0/1676646491 >> 172.21.15.42:0/1676646491 conn(0x7fca04075ae0 msgr2=0x7fca040a4830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:45:37.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.512+0000 7fca0be95640 1 -- 172.21.15.42:0/1676646491 shutdown_connections 2024-09-16T03:45:37.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.512+0000 7fca0be95640 1 -- 172.21.15.42:0/1676646491 wait complete. 2024-09-16T03:45:37.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.512+0000 7fca0be95640 1 librados: shutdown 2024-09-16T03:45:37.534 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T03:45:37.534 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-16T03:45:37.534 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T03:45:37.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.656+0000 7ffa80fe5640 10 monclient: get_monmap_and_config 2024-09-16T03:45:37.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.656+0000 7ffa80fe5640 10 monclient: build_initial_monmap 2024-09-16T03:45:37.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.656+0000 7ffa80fe5640 10 monclient: monmap: 2024-09-16T03:45:37.662 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:45:37.662 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:45:37.662 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:45:37.661595+0000 2024-09-16T03:45:37.663 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:45:37.661595+0000 2024-09-16T03:45:37.663 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:45:37.663 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:45:37.663 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:45:37.663 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:45:37.663 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:45:37.663 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:45:37.663 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:45:37.663 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:45:37.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.656+0000 7ffa80fe5640 1 Processor -- start 2024-09-16T03:45:37.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.656+0000 7ffa80fe5640 1 -- start start 2024-09-16T03:45:37.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.656+0000 7ffa80fe5640 10 monclient: init 2024-09-16T03:45:37.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.656+0000 7ffa80fe5640 10 monclient: _reopen_session rank -1 2024-09-16T03:45:37.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.656+0000 7ffa80fe5640 10 monclient: _add_conns ranks=[1,4,0,2,3] 2024-09-16T03:45:37.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.656+0000 7ffa80fe5640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ffa7c0a39b0 addr v1:172.21.15.154:6789/0 2024-09-16T03:45:37.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.656+0000 7ffa80fe5640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7ffa7c0a3d90 0x7ffa7c13c380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:45:37.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.656+0000 7ffa80fe5640 10 monclient(hunting): picked mon.noname-c con 0x7ffa7c0a3d90 addr v2:172.21.15.154:3300/0 2024-09-16T03:45:37.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.656+0000 7ffa80fe5640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ffa7c13c8c0 addr v1:172.21.15.42:6789/0 2024-09-16T03:45:37.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.656+0000 7ffa80fe5640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffa7c134290 con 0x7ffa7c13c8c0 2024-09-16T03:45:37.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.656+0000 7ffa80fe5640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffa7c09c140 con 0x7ffa7c0a39b0 2024-09-16T03:45:37.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.656+0000 7ffa80fe5640 10 monclient(hunting): start opening mon connection 2024-09-16T03:45:37.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.656+0000 7ffa80fe5640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7ffa7c09c420 con 0x7ffa7c0a3d90 2024-09-16T03:45:37.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.656+0000 7ffa80fe5640 10 monclient(hunting): _renew_subs 2024-09-16T03:45:37.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.656+0000 7ffa80fe5640 10 monclient(hunting): authenticate will time out at 1184.631096s 2024-09-16T03:45:37.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.656+0000 7ffa7a59c640 1 -- >> v1:172.21.15.154:6789/0 conn(0x7ffa7c0a39b0 legacy=0x7ffa7c133b80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.154:6789/0 2024-09-16T03:45:37.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.656+0000 7ffa79d9b640 1 -- >> v2:172.21.15.154:3300/0 conn(0x7ffa7c0a3d90 msgr2=0x7ffa7c13c380 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T03:45:37.670 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.656+0000 7ffa79d9b640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7ffa7c0a3d90 0x7ffa7c13c380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:45:37.670 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.656+0000 7ffa7ad9d640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7ffa7c13c8c0 0x7ffa7c13ecb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:44972/0 (socket says 172.21.15.42:44972) 2024-09-16T03:45:37.670 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.656+0000 7ffa7ad9d640 1 -- 172.21.15.42:0/332238429 learned_addr learned my addr 172.21.15.42:0/332238429 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:45:37.670 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.656+0000 7ffa7959a640 1 -- 172.21.15.42:0/332238429 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1680965663 0 0) 0x7ffa7c134290 con 0x7ffa7c13c8c0 2024-09-16T03:45:37.670 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.656+0000 7ffa7959a640 10 monclient(hunting): _negotiate 2024-09-16T03:45:37.670 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.656+0000 7ffa7959a640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:45:37.670 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.656+0000 7ffa7959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:45:37.670 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.656+0000 7ffa7959a640 10 monclient(hunting): my global_id is 744222 2024-09-16T03:45:37.670 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.656+0000 7ffa7959a640 1 -- 172.21.15.42:0/332238429 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffa68003530 con 0x7ffa7c13c8c0 2024-09-16T03:45:37.670 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.656+0000 7ffa7959a640 1 -- 172.21.15.42:0/332238429 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 936205792 0 0) 0x7ffa68003530 con 0x7ffa7c13c8c0 2024-09-16T03:45:37.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.656+0000 7ffa7959a640 1 -- 172.21.15.42:0/332238429 >> v2:172.21.15.154:3300/0 conn(0x7ffa7c0a3d90 msgr2=0x7ffa7c13c380 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:45:37.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.656+0000 7ffa7959a640 1 --2- 172.21.15.42:0/332238429 >> v2:172.21.15.154:3300/0 conn(0x7ffa7c0a3d90 0x7ffa7c13c380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:45:37.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.660+0000 7ffa7959a640 1 -- 172.21.15.42:0/332238429 >> v1:172.21.15.154:6789/0 conn(0x7ffa7c0a39b0 legacy=0x7ffa7c133b80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:45:37.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.660+0000 7ffa7959a640 10 monclient: _finish_hunting 0 2024-09-16T03:45:37.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.660+0000 7ffa7959a640 1 monclient: found mon.noname-a-legacy 2024-09-16T03:45:37.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.660+0000 7ffa7959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:45:37.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.660+0000 7ffa7959a640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.42:6789/0 2024-09-16T03:45:37.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.660+0000 7ffa7959a640 1 -- 172.21.15.42:0/332238429 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa7c06dd60 con 0x7ffa7c13c8c0 2024-09-16T03:45:37.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.660+0000 7ffa7959a640 10 monclient: _finish_auth 0 2024-09-16T03:45:37.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.660+0000 7ffa7959a640 10 monclient: _check_auth_tickets 2024-09-16T03:45:37.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.660+0000 7ffa7959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:45:37.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.660+0000 7ffa7959a640 1 -- 172.21.15.42:0/332238429 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2177413734 0 0) 0x7ffa74001a00 con 0x7ffa7c13c8c0 2024-09-16T03:45:37.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.660+0000 7ffa80fe5640 5 monclient: authenticate success, global_id 744222 2024-09-16T03:45:37.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.660+0000 7ffa80fe5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:45:37.672 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.660+0000 7ffa7959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:45:37.672 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.660+0000 7ffa7959a640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T03:45:37.672 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.660+0000 7ffa7959a640 10 monclient: dump: 2024-09-16T03:45:37.672 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:45:37.672 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:45:37.672 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:37.672 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:37.672 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:45:37.672 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:45:37.672 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:45:37.672 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:45:37.672 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:45:37.672 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:45:37.672 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.660+0000 7ffa7959a640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] but i'm connected to v1:172.21.15.42:6789/0, reconnecting 2024-09-16T03:45:37.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.660+0000 7ffa7959a640 10 monclient: _reopen_session rank -1 2024-09-16T03:45:37.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.660+0000 7ffa7959a640 1 -- 172.21.15.42:0/332238429 >> v1:172.21.15.42:6789/0 conn(0x7ffa7c13c8c0 legacy=0x7ffa7c13ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:45:37.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.660+0000 7ffa7959a640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T03:45:37.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.660+0000 7ffa7959a640 1 --2- 172.21.15.42:0/332238429 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ffa680039c0 0x7ffa68003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:45:37.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.660+0000 7ffa7959a640 10 monclient(hunting): picked mon.a con 0x7ffa680039c0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:45:37.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.660+0000 7ffa7959a640 1 --2- 172.21.15.42:0/332238429 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffa68004810 0x7ffa68004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:45:37.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.660+0000 7ffa7959a640 10 monclient(hunting): picked mon.b con 0x7ffa68004810 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:45:37.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.660+0000 7ffa7959a640 1 --2- 172.21.15.42:0/332238429 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ffa68005380 0x7ffa6800b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:45:37.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.660+0000 7ffa7959a640 10 monclient(hunting): picked mon.c con 0x7ffa68005380 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:45:37.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.660+0000 7ffa7959a640 10 monclient(hunting): start opening mon connection 2024-09-16T03:45:37.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.660+0000 7ffa7959a640 1 -- 172.21.15.42:0/332238429 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7ffa7c09c420 con 0x7ffa680039c0 2024-09-16T03:45:37.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.660+0000 7ffa7959a640 10 monclient(hunting): start opening mon connection 2024-09-16T03:45:37.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.660+0000 7ffa7959a640 1 -- 172.21.15.42:0/332238429 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ffa6800c2d0 con 0x7ffa68004810 2024-09-16T03:45:37.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.660+0000 7ffa7959a640 10 monclient(hunting): start opening mon connection 2024-09-16T03:45:37.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.660+0000 7ffa7959a640 1 -- 172.21.15.42:0/332238429 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7ffa6800c4d0 con 0x7ffa68005380 2024-09-16T03:45:37.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.660+0000 7ffa7959a640 10 monclient(hunting): _renew_subs 2024-09-16T03:45:37.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.660+0000 7ffa7959a640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:45:37.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.660+0000 7ffa80fe5640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-16T03:45:37.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.660+0000 7ffa7a59c640 1 --2- 172.21.15.42:0/332238429 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ffa680039c0 0x7ffa68003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:45:37.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.660+0000 7ffa7a59c640 10 monclient(hunting): get_auth_request con 0x7ffa680039c0 auth_method 0 2024-09-16T03:45:37.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.660+0000 7ffa7a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:45:37.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.660+0000 7ffa7a59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:45:37.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.660+0000 7ffa7a59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:45:37.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.660+0000 7ffa79d9b640 1 -- 172.21.15.42:0/332238429 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffa68004810 msgr2=0x7ffa68004c90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T03:45:37.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.660+0000 7ffa79d9b640 1 --2- 172.21.15.42:0/332238429 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffa68004810 0x7ffa68004c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:45:37.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.660+0000 7ffa7ad9d640 1 --2- 172.21.15.42:0/332238429 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ffa68005380 0x7ffa6800b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:45:37.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.660+0000 7ffa7a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:45:37.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.660+0000 7ffa7a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:45:37.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.660+0000 7ffa7a59c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T03:45:37.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.660+0000 7ffa7ad9d640 10 monclient(hunting): get_auth_request con 0x7ffa68005380 auth_method 0 2024-09-16T03:45:37.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.660+0000 7ffa7ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:45:37.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.660+0000 7ffa7ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:45:37.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.660+0000 7ffa7ad9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:45:37.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.660+0000 7ffa7ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:45:37.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.660+0000 7ffa7ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:45:37.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.660+0000 7ffa7ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T03:45:37.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.660+0000 7ffa7a59c640 10 monclient(hunting): handle_auth_done global_id 744222 payload 293 2024-09-16T03:45:37.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.660+0000 7ffa7a59c640 1 -- 172.21.15.42:0/332238429 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ffa68005380 msgr2=0x7ffa6800b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:45:37.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.660+0000 7ffa7a59c640 1 --2- 172.21.15.42:0/332238429 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ffa68005380 0x7ffa6800b9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:45:37.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.660+0000 7ffa7a59c640 1 -- 172.21.15.42:0/332238429 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffa68004810 msgr2=0x7ffa68004c90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:45:37.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.660+0000 7ffa7a59c640 1 --2- 172.21.15.42:0/332238429 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffa68004810 0x7ffa68004c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:45:37.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.660+0000 7ffa7a59c640 10 monclient: _finish_hunting 0 2024-09-16T03:45:37.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.660+0000 7ffa7a59c640 1 monclient: found mon.a 2024-09-16T03:45:37.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.660+0000 7ffa7a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:45:37.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.660+0000 7ffa7a59c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:45:37.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.660+0000 7ffa7a59c640 1 -- 172.21.15.42:0/332238429 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa68003530 con 0x7ffa680039c0 2024-09-16T03:45:37.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.660+0000 7ffa7a59c640 10 monclient: _finish_auth 0 2024-09-16T03:45:37.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.660+0000 7ffa7a59c640 10 monclient: _check_auth_tickets 2024-09-16T03:45:37.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.660+0000 7ffa7a59c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:45:37.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.660+0000 7ffa7ad9d640 1 -- 172.21.15.42:0/332238429 reap_dead start 2024-09-16T03:45:37.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.660+0000 7ffa7a59c640 1 --2- 172.21.15.42:0/332238429 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ffa680039c0 0x7ffa68003dc0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7ffa6c003940 tx=0x7ffa6c01c420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9276eca4317ab52f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:45:37.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.660+0000 7ffa7959a640 1 -- 172.21.15.42:0/332238429 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa6c009520 con 0x7ffa680039c0 2024-09-16T03:45:37.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.660+0000 7ffa7959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:45:37.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.660+0000 7ffa7959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:45:37.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.660+0000 7ffa7959a640 10 monclient: dump: 2024-09-16T03:45:37.676 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:45:37.677 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:45:37.677 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:37.677 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:37.677 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:45:37.677 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:45:37.677 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:45:37.677 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:45:37.677 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:45:37.677 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:45:37.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.660+0000 7ffa7959a640 1 -- 172.21.15.42:0/332238429 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffa6c003bc0 con 0x7ffa680039c0 2024-09-16T03:45:37.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.660+0000 7ffa7959a640 10 monclient: handle_config config(0 keys) 2024-09-16T03:45:37.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.660+0000 7ffa7959a640 1 -- 172.21.15.42:0/332238429 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa6c005070 con 0x7ffa680039c0 2024-09-16T03:45:37.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.660+0000 7ffa80fe5640 10 monclient: get_monmap_and_config success 2024-09-16T03:45:37.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.660+0000 7ffa80fe5640 4 set_mon_vals no callback set 2024-09-16T03:45:37.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.660+0000 7ffa7959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:45:37.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.660+0000 7ffa7959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:45:37.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.660+0000 7ffa7959a640 10 monclient: dump: 2024-09-16T03:45:37.678 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:45:37.678 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:45:37.678 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:37.678 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:37.678 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:45:37.678 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:45:37.678 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:45:37.678 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:45:37.678 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:45:37.678 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:45:37.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.660+0000 7ffa80fe5640 10 monclient: shutdown 2024-09-16T03:45:37.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.660+0000 7ffa80fe5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:45:37.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.660+0000 7ffa80fe5640 1 -- 172.21.15.42:0/332238429 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ffa680039c0 msgr2=0x7ffa68003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:45:37.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.660+0000 7ffa80fe5640 1 --2- 172.21.15.42:0/332238429 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ffa680039c0 0x7ffa68003dc0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7ffa6c003940 tx=0x7ffa6c01c420 comp rx=0 tx=0).stop 2024-09-16T03:45:37.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.660+0000 7ffa80fe5640 1 -- 172.21.15.42:0/332238429 shutdown_connections 2024-09-16T03:45:37.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.660+0000 7ffa80fe5640 1 --2- 172.21.15.42:0/332238429 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ffa680039c0 0x7ffa68003dc0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:45:37.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.660+0000 7ffa80fe5640 1 -- 172.21.15.42:0/332238429 >> 172.21.15.42:0/332238429 conn(0x7ffa7c0a9fb0 msgr2=0x7ffa7c0aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:45:37.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.660+0000 7ffa80fe5640 1 -- 172.21.15.42:0/332238429 shutdown_connections 2024-09-16T03:45:37.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.660+0000 7ffa80fe5640 1 -- 172.21.15.42:0/332238429 wait complete. 2024-09-16T03:45:37.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.664+0000 7ffa80fe5640 10 monclient: build_initial_monmap 2024-09-16T03:45:37.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.664+0000 7ffa80fe5640 10 monclient: monmap: 2024-09-16T03:45:37.686 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:45:37.686 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:45:37.686 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:45:37.686 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:45:37.686 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:45:37.686 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:45:37.686 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:45:37.686 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:45:37.686 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:45:37.686 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:45:37.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.664+0000 7ffa80fe5640 1 librados: starting msgr at 2024-09-16T03:45:37.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.664+0000 7ffa80fe5640 1 librados: starting objecter 2024-09-16T03:45:37.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.664+0000 7ffa80fe5640 1 Processor -- start 2024-09-16T03:45:37.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.664+0000 7ffa80fe5640 1 -- start start 2024-09-16T03:45:37.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.664+0000 7ffa80fe5640 1 librados: setting wanted keys 2024-09-16T03:45:37.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.664+0000 7ffa80fe5640 1 librados: calling monclient init 2024-09-16T03:45:37.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.664+0000 7ffa80fe5640 10 monclient: init 2024-09-16T03:45:37.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.664+0000 7ffa80fe5640 10 monclient: _reopen_session rank -1 2024-09-16T03:45:37.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.664+0000 7ffa80fe5640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:45:37.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.664+0000 7ffa80fe5640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ffa680039c0 0x7ffa7c130df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:45:37.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.664+0000 7ffa80fe5640 10 monclient(hunting): picked mon.noname-c con 0x7ffa680039c0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:45:37.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.664+0000 7ffa80fe5640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffa7c09fd10 0x7ffa7c0a0130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:45:37.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.664+0000 7ffa80fe5640 10 monclient(hunting): picked mon.noname-b con 0x7ffa7c09fd10 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:45:37.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.664+0000 7ffa80fe5640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ffa7c0a0670 0x7ffa7c1d5ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:45:37.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.664+0000 7ffa80fe5640 10 monclient(hunting): picked mon.noname-a con 0x7ffa7c0a0670 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:45:37.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.664+0000 7ffa80fe5640 10 monclient(hunting): start opening mon connection 2024-09-16T03:45:37.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.664+0000 7ffa80fe5640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7ffa7c137ba0 con 0x7ffa7c0a0670 2024-09-16T03:45:37.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.664+0000 7ffa80fe5640 10 monclient(hunting): start opening mon connection 2024-09-16T03:45:37.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.664+0000 7ffa80fe5640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ffa7c137a20 con 0x7ffa7c09fd10 2024-09-16T03:45:37.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.664+0000 7ffa80fe5640 10 monclient(hunting): start opening mon connection 2024-09-16T03:45:37.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.664+0000 7ffa80fe5640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7ffa7c137d20 con 0x7ffa680039c0 2024-09-16T03:45:37.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.664+0000 7ffa80fe5640 10 monclient(hunting): _renew_subs 2024-09-16T03:45:37.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.664+0000 7ffa80fe5640 10 monclient(hunting): authenticate will time out at 1184.637355s 2024-09-16T03:45:37.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.664+0000 7ffa79d9b640 1 -- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffa7c09fd10 msgr2=0x7ffa7c0a0130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T03:45:37.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.664+0000 7ffa79d9b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffa7c09fd10 0x7ffa7c0a0130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:45:37.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.664+0000 7ffa7ad9d640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ffa7c0a0670 0x7ffa7c1d5ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:45:37.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.664+0000 7ffa7a59c640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ffa680039c0 0x7ffa7c130df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:45:37.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.664+0000 7ffa7ad9d640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ffa7c0a0670 0x7ffa7c1d5ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:42730/0 (socket says 172.21.15.42:42730) 2024-09-16T03:45:37.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.664+0000 7ffa7a59c640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ffa680039c0 0x7ffa7c130df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:33988/0 (socket says 172.21.15.42:33988) 2024-09-16T03:45:37.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.664+0000 7ffa7ad9d640 1 -- 172.21.15.42:0/4112333812 learned_addr learned my addr 172.21.15.42:0/4112333812 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:45:37.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.664+0000 7ffa7ad9d640 10 monclient(hunting): get_auth_request con 0x7ffa7c0a0670 auth_method 0 2024-09-16T03:45:37.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.664+0000 7ffa7ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:45:37.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.664+0000 7ffa7ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:45:37.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.664+0000 7ffa7ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:45:37.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.664+0000 7ffa7a59c640 10 monclient(hunting): get_auth_request con 0x7ffa680039c0 auth_method 0 2024-09-16T03:45:37.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.664+0000 7ffa7a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:45:37.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.664+0000 7ffa7a59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:45:37.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.664+0000 7ffa7a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:45:37.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.664+0000 7ffa7ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:45:37.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.664+0000 7ffa7ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:45:37.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.664+0000 7ffa7ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:45:37.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.664+0000 7ffa7a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:45:37.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.664+0000 7ffa7a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:45:37.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.664+0000 7ffa7a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:45:37.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.664+0000 7ffa7ad9d640 10 monclient(hunting): handle_auth_done global_id 744225 payload 824 2024-09-16T03:45:37.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.664+0000 7ffa7ad9d640 1 -- 172.21.15.42:0/4112333812 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ffa680039c0 msgr2=0x7ffa7c130df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:45:37.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.664+0000 7ffa7ad9d640 1 --2- 172.21.15.42:0/4112333812 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ffa680039c0 0x7ffa7c130df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:45:37.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.664+0000 7ffa7ad9d640 1 -- 172.21.15.42:0/4112333812 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffa7c09fd10 msgr2=0x7ffa7c0a0130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:45:37.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.664+0000 7ffa7ad9d640 1 --2- 172.21.15.42:0/4112333812 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffa7c09fd10 0x7ffa7c0a0130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:45:37.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.664+0000 7ffa7ad9d640 10 monclient: _finish_hunting 0 2024-09-16T03:45:37.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.664+0000 7ffa7ad9d640 1 monclient: found mon.noname-a 2024-09-16T03:45:37.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.664+0000 7ffa7ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:45:37.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.664+0000 7ffa7ad9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:45:37.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.664+0000 7ffa7ad9d640 1 -- 172.21.15.42:0/4112333812 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa7c13c600 con 0x7ffa7c0a0670 2024-09-16T03:45:37.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.664+0000 7ffa7a59c640 1 --2- 172.21.15.42:0/4112333812 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ffa680039c0 0x7ffa7c130df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:45:37.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.664+0000 7ffa7ad9d640 1 --2- 172.21.15.42:0/4112333812 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ffa7c0a0670 0x7ffa7c1d5ac0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7ffa74000f90 tx=0x7ffa74007b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:45:37.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.664+0000 7ffa72ffd640 10 client.?.objecter ms_handle_connect 0x7ffa7c0a0670 2024-09-16T03:45:37.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.664+0000 7ffa72ffd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:45:37.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.664+0000 7ffa72ffd640 1 -- 172.21.15.42:0/4112333812 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa74017020 con 0x7ffa7c0a0670 2024-09-16T03:45:37.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.664+0000 7ffa72ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:45:37.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.664+0000 7ffa72ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:45:37.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.664+0000 7ffa72ffd640 10 monclient: dump: 2024-09-16T03:45:37.691 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:45:37.691 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:45:37.691 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:37.691 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:37.691 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:45:37.691 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:45:37.692 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:45:37.692 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:45:37.692 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:45:37.692 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:45:37.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.664+0000 7ffa72ffd640 10 monclient: _finish_auth 0 2024-09-16T03:45:37.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.664+0000 7ffa72ffd640 10 monclient: _check_auth_tickets 2024-09-16T03:45:37.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.664+0000 7ffa72ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:45:37.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.664+0000 7ffa72ffd640 1 -- 172.21.15.42:0/4112333812 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffa7400ab60 con 0x7ffa7c0a0670 2024-09-16T03:45:37.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.664+0000 7ffa80fe5640 5 monclient: authenticate success, global_id 744225 2024-09-16T03:45:37.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.664+0000 7ffa80fe5640 10 monclient: _renew_subs 2024-09-16T03:45:37.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.664+0000 7ffa80fe5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:45:37.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.664+0000 7ffa80fe5640 1 -- 172.21.15.42:0/4112333812 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa7c0a0b20 con 0x7ffa7c0a0670 2024-09-16T03:45:37.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.664+0000 7ffa72ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:45:37.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.664+0000 7ffa72ffd640 1 -- 172.21.15.42:0/4112333812 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa740133f0 con 0x7ffa7c0a0670 2024-09-16T03:45:37.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.664+0000 7ffa72ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:45:37.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.664+0000 7ffa72ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:45:37.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.664+0000 7ffa72ffd640 10 monclient: dump: 2024-09-16T03:45:37.693 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:45:37.693 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:45:37.693 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:37.693 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:37.693 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:45:37.693 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:45:37.693 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:45:37.693 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:45:37.693 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:45:37.693 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:45:37.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.664+0000 7ffa80fe5640 10 client.744225.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:45:37.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.664+0000 7ffa80fe5640 10 monclient: _renew_subs 2024-09-16T03:45:37.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.664+0000 7ffa80fe5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:45:37.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.664+0000 7ffa80fe5640 1 -- 172.21.15.42:0/4112333812 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa7c0a2d10 con 0x7ffa7c0a0670 2024-09-16T03:45:37.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.664+0000 7ffa80fe5640 1 librados: init done 2024-09-16T03:45:37.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.664+0000 7ffa78d99640 4 set_mon_vals no callback set 2024-09-16T03:45:37.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.664+0000 7ffa80fe5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:45:37.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.664+0000 7ffa80fe5640 1 --2- 172.21.15.42:0/4112333812 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ffa7c09c940 0x7ffa7c09cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:45:37.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.664+0000 7ffa80fe5640 10 monclient(hunting): start opening mon connection 2024-09-16T03:45:37.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.664+0000 7ffa80fe5640 1 -- 172.21.15.42:0/4112333812 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7ffa7c134230 con 0x7ffa7c09c940 2024-09-16T03:45:37.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.664+0000 7ffa7a59c640 1 --2- 172.21.15.42:0/4112333812 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ffa7c09c940 0x7ffa7c09cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:45:37.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.664+0000 7ffa7a59c640 10 monclient: get_auth_request con 0x7ffa7c09c940 auth_method 0 2024-09-16T03:45:37.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.664+0000 7ffa7a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:45:37.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.664+0000 7ffa7a59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:45:37.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.664+0000 7ffa7a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:45:37.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.664+0000 7ffa7a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:45:37.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.664+0000 7ffa7a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:45:37.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.664+0000 7ffa7a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:45:37.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.664+0000 7ffa7a59c640 10 monclient(hunting): handle_auth_done global_id 744228 payload 824 2024-09-16T03:45:37.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.664+0000 7ffa7a59c640 1 -- 172.21.15.42:0/4112333812 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ffa7c0a5750 con 0x7ffa7c09c940 2024-09-16T03:45:37.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.664+0000 7ffa72ffd640 1 -- 172.21.15.42:0/4112333812 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7ffa7401a020 con 0x7ffa7c0a0670 2024-09-16T03:45:37.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.668+0000 7ffa7a59c640 1 --2- 172.21.15.42:0/4112333812 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ffa7c09c940 0x7ffa7c09cd20 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7ffa6c006fd0 tx=0x7ffa68005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:45:37.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.668+0000 7ffa72ffd640 1 --2- 172.21.15.42:0/4112333812 >> v2:172.21.15.154:6812/254845342 conn(0x7ffa5c0377f0 0x7ffa5c039cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:45:37.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.668+0000 7ffa72ffd640 10 client.744225.objecter ms_handle_connect 0x7ffa7c09c940 2024-09-16T03:45:37.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.668+0000 7ffa72ffd640 10 client.744225.objecter resend_mon_ops 2024-09-16T03:45:37.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.668+0000 7ffa72ffd640 1 -- 172.21.15.42:0/4112333812 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 7832+0+0 (secure 0 0 0) 0x7ffa74049f80 con 0x7ffa7c0a0670 2024-09-16T03:45:37.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.668+0000 7ffa72ffd640 10 client.744225.objecter ms_dispatch 0x7ffa7c000ba0 osd_map(91..91 src has 1..91) 2024-09-16T03:45:37.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.668+0000 7ffa72ffd640 3 client.744225.objecter handle_osd_map got epochs [91,91] > 0 2024-09-16T03:45:37.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.668+0000 7ffa72ffd640 3 client.744225.objecter handle_osd_map decoding full epoch 91 2024-09-16T03:45:37.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.668+0000 7ffa72ffd640 20 client.744225.objecter dump_active .. 0 homeless 2024-09-16T03:45:37.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.668+0000 7ffa72ffd640 1 -- 172.21.15.42:0/4112333812 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa6c0056c0 con 0x7ffa7c09c940 2024-09-16T03:45:37.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.668+0000 7ffa72ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:45:37.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.668+0000 7ffa72ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:45:37.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.668+0000 7ffa72ffd640 10 monclient: dump: 2024-09-16T03:45:37.696 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:45:37.696 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:45:37.696 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:37.696 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:37.696 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:45:37.696 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:45:37.696 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:45:37.696 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:45:37.696 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:45:37.697 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:45:37.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.668+0000 7ffa79d9b640 1 --2- 172.21.15.42:0/4112333812 >> v2:172.21.15.154:6812/254845342 conn(0x7ffa5c0377f0 0x7ffa5c039cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:45:37.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.668+0000 7ffa72ffd640 1 -- 172.21.15.42:0/4112333812 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7ffa6c0056c0 con 0x7ffa7c09c940 2024-09-16T03:45:37.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.668+0000 7ffa72ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:45:37.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.668+0000 7ffa72ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:45:37.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.668+0000 7ffa72ffd640 1 -- 172.21.15.42:0/4112333812 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ffa7c09c940 msgr2=0x7ffa7c09cd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:45:37.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.668+0000 7ffa72ffd640 1 --2- 172.21.15.42:0/4112333812 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ffa7c09c940 0x7ffa7c09cd20 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7ffa6c006fd0 tx=0x7ffa68005840 comp rx=0 tx=0).stop 2024-09-16T03:45:37.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.668+0000 7ffa72ffd640 1 -- 172.21.15.42:0/4112333812 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ffa7c09c940 msgr2=0x7ffa7c09cd20 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:45:37.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.668+0000 7ffa72ffd640 1 --2- 172.21.15.42:0/4112333812 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ffa7c09c940 0x7ffa7c09cd20 secure :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7ffa6c006fd0 tx=0x7ffa68005840 comp rx=0 tx=0).stop 2024-09-16T03:45:37.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.668+0000 7ffa79d9b640 10 monclient: get_auth_request con 0x7ffa5c0377f0 auth_method 0 2024-09-16T03:45:37.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.668+0000 7ffa79d9b640 1 --2- 172.21.15.42:0/4112333812 >> v2:172.21.15.154:6812/254845342 conn(0x7ffa5c0377f0 0x7ffa5c039cb0 secure :-1 s=READY pgs=3371 cs=0 l=1 rev1=1 crypto rx=0x7ffa64006fd0 tx=0x7ffa64006d40 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:45:37.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.668+0000 7ffa72ffd640 10 client.744225.objecter ms_handle_connect 0x7ffa5c0377f0 2024-09-16T03:45:37.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.672+0000 7ffa80fe5640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:45:37.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.672+0000 7ffa80fe5640 1 --2- 172.21.15.42:0/4112333812 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ffa7c09d260 0x7ffa7c1da020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:45:37.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.672+0000 7ffa80fe5640 10 monclient(hunting): start opening mon connection 2024-09-16T03:45:37.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.672+0000 7ffa80fe5640 1 -- 172.21.15.42:0/4112333812 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7ffa7c068e40 con 0x7ffa7c09d260 2024-09-16T03:45:37.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.672+0000 7ffa7a59c640 1 --2- 172.21.15.42:0/4112333812 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ffa7c09d260 0x7ffa7c1da020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:45:37.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.672+0000 7ffa7a59c640 10 monclient: get_auth_request con 0x7ffa7c09d260 auth_method 0 2024-09-16T03:45:37.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.672+0000 7ffa7a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:45:37.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.672+0000 7ffa7a59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:45:37.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.672+0000 7ffa7a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:45:37.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.672+0000 7ffa7a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:45:37.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.672+0000 7ffa7a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:45:37.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.672+0000 7ffa7a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:45:37.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.672+0000 7ffa7a59c640 10 monclient(hunting): handle_auth_done global_id 744231 payload 824 2024-09-16T03:45:37.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.672+0000 7ffa7a59c640 1 -- 172.21.15.42:0/4112333812 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ffa7c0a2510 con 0x7ffa7c09d260 2024-09-16T03:45:37.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.672+0000 7ffa7a59c640 1 --2- 172.21.15.42:0/4112333812 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ffa7c09d260 0x7ffa7c1da020 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7ffa6c003940 tx=0x7ffa6c0a6000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:45:37.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.672+0000 7ffa72ffd640 10 client.744225.objecter ms_handle_connect 0x7ffa7c09d260 2024-09-16T03:45:37.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.672+0000 7ffa72ffd640 10 client.744225.objecter resend_mon_ops 2024-09-16T03:45:37.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.672+0000 7ffa72ffd640 1 -- 172.21.15.42:0/4112333812 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa6c0bfa80 con 0x7ffa7c09d260 2024-09-16T03:45:37.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.672+0000 7ffa72ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:45:37.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.672+0000 7ffa72ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:45:37.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.672+0000 7ffa72ffd640 10 monclient: dump: 2024-09-16T03:45:37.699 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:45:37.699 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:45:37.699 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:37.699 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:37.699 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:45:37.699 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:45:37.699 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:45:37.699 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:45:37.699 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:45:37.699 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:45:37.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.672+0000 7ffa72ffd640 1 -- 172.21.15.42:0/4112333812 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4702 (secure 0 0 0) 0x7ffa6c0b8e30 con 0x7ffa7c09d260 2024-09-16T03:45:37.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.672+0000 7ffa72ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:45:37.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.672+0000 7ffa72ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:45:37.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.672+0000 7ffa72ffd640 1 -- 172.21.15.42:0/4112333812 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ffa7c09d260 msgr2=0x7ffa7c1da020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:45:37.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.672+0000 7ffa72ffd640 1 --2- 172.21.15.42:0/4112333812 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ffa7c09d260 0x7ffa7c1da020 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7ffa6c003940 tx=0x7ffa6c0a6000 comp rx=0 tx=0).stop 2024-09-16T03:45:37.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.672+0000 7ffa72ffd640 1 -- 172.21.15.42:0/4112333812 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ffa7c09d260 msgr2=0x7ffa7c1da020 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:45:37.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.672+0000 7ffa72ffd640 1 --2- 172.21.15.42:0/4112333812 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ffa7c09d260 0x7ffa7c1da020 secure :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7ffa6c003940 tx=0x7ffa6c0a6000 comp rx=0 tx=0).stop 2024-09-16T03:45:37.700 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:45:37.700 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:45:37.700 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:45:37.701 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "leader", 2024-09-16T03:45:37.701 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 32, 2024-09-16T03:45:37.701 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 25043, 2024-09-16T03:45:37.701 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:45:37.701 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:45:37.701 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T03:45:37.701 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:45:37.701 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 0, 2024-09-16T03:45:37.701 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:45:37.701 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:45:37.701 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:45:37.701 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:45:37.701 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:45:37.701 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:45:37.702 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:45:37.702 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:45:37.702 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:45:37.702 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:45:37.702 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:45:37.702 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:45:37.702 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:45:37.702 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:45:37.702 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:45:37.702 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:45:37.702 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:45:37.702 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:45:37.702 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:45:37.702 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:45:37.702 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:45:37.703 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:45:37.703 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:45:37.703 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:45:37.703 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:45:37.703 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:45:37.703 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:45:37.703 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:45:37.703 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:45:37.703 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:45:37.703 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:45:37.703 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:45:37.703 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:45:37.703 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:45:37.703 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:45:37.704 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:45:37.704 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:45:37.704 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:45:37.704 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:45:37.704 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:45:37.704 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:45:37.704 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:45:37.704 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:45:37.704 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:45:37.704 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:45:37.704 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:45:37.704 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:45:37.704 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:45:37.704 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:45:37.705 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:45:37.705 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:45:37.705 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:45:37.705 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:45:37.705 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:45:37.705 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:45:37.705 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:45:37.705 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:45:37.705 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:45:37.705 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:45:37.705 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:45:37.705 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:45:37.705 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:45:37.705 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:45:37.706 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:45:37.706 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:45:37.706 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:45:37.706 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:45:37.706 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:45:37.706 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:45:37.706 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:45:37.706 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:45:37.706 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:45:37.706 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:45:37.706 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:45:37.706 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:45:37.706 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:45:37.706 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:45:37.707 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:45:37.707 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:45:37.707 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:45:37.707 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:45:37.707 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:45:37.707 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:45:37.707 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:45:37.707 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:45:37.707 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:45:37.707 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:45:37.708 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:45:37.708 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:45:37.708 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:45:37.708 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:45:37.708 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:45:37.708 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:45:37.708 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:45:37.708 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:45:37.708 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:45:37.708 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:45:37.708 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:45:37.708 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:45:37.708 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:45:37.708 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:45:37.709 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:45:37.709 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:45:37.709 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:45:37.709 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:45:37.709 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:45:37.709 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:45:37.709 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:45:37.709 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:45:37.709 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:45:37.709 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:45:37.709 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:45:37.709 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:45:37.709 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:45:37.709 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:45:37.710 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:45:37.710 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:45:37.710 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:45:37.710 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:45:37.710 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:45:37.710 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:45:37.710 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:45:37.710 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:45:37.710 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:45:37.710 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:45:37.710 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:45:37.710 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:45:37.710 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:45:37.710 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:45:37.710 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:45:37.711 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:45:37.711 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:45:37.711 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:45:37.711 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:45:37.711 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:45:37.711 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:45:37.711 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:45:37.711 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:45:37.711 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:45:37.711 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:45:37.711 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:45:37.711 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T03:45:37.711 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:45:37.711 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:45:37.712 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:45:37.712 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:45:37.712 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:45:37.712 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:45:37.712 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:45:37.712 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:45:37.712 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:45:37.712 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:45:37.712 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 4 2024-09-16T03:45:37.712 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:45:37.712 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:45:37.712 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:45:37.712 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:45:37.712 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:45:37.714 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.676+0000 7ffa80fe5640 10 librados: watch_flush enter 2024-09-16T03:45:37.714 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.676+0000 7ffa80fe5640 10 librados: watch_flush exit 2024-09-16T03:45:37.714 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.676+0000 7ffa80fe5640 20 client.744225.objecter shutdown clearing up homeless session... 2024-09-16T03:45:37.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.676+0000 7ffa80fe5640 10 client.744225.objecter successfully canceled tick 2024-09-16T03:45:37.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.676+0000 7ffa80fe5640 1 -- 172.21.15.42:0/4112333812 >> v2:172.21.15.154:6812/254845342 conn(0x7ffa5c0377f0 msgr2=0x7ffa5c039cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:45:37.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.676+0000 7ffa80fe5640 1 --2- 172.21.15.42:0/4112333812 >> v2:172.21.15.154:6812/254845342 conn(0x7ffa5c0377f0 0x7ffa5c039cb0 secure :-1 s=READY pgs=3371 cs=0 l=1 rev1=1 crypto rx=0x7ffa64006fd0 tx=0x7ffa64006d40 comp rx=0 tx=0).stop 2024-09-16T03:45:37.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.676+0000 7ffa80fe5640 10 monclient: shutdown 2024-09-16T03:45:37.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.676+0000 7ffa80fe5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:45:37.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.676+0000 7ffa80fe5640 1 -- 172.21.15.42:0/4112333812 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ffa7c0a0670 msgr2=0x7ffa7c1d5ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:45:37.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.676+0000 7ffa80fe5640 1 --2- 172.21.15.42:0/4112333812 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ffa7c0a0670 0x7ffa7c1d5ac0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7ffa74000f90 tx=0x7ffa74007b20 comp rx=0 tx=0).stop 2024-09-16T03:45:37.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.676+0000 7ffa7ad9d640 1 -- 172.21.15.42:0/4112333812 reap_dead start 2024-09-16T03:45:37.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.676+0000 7ffa80fe5640 1 -- 172.21.15.42:0/4112333812 shutdown_connections 2024-09-16T03:45:37.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.676+0000 7ffa80fe5640 1 --2- 172.21.15.42:0/4112333812 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ffa7c0a0670 0x7ffa7c1d5ac0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:45:37.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.676+0000 7ffa80fe5640 1 -- 172.21.15.42:0/4112333812 >> 172.21.15.42:0/4112333812 conn(0x7ffa7c0a9fb0 msgr2=0x7ffa7c0a7ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:45:37.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.676+0000 7ffa80fe5640 1 -- 172.21.15.42:0/4112333812 shutdown_connections 2024-09-16T03:45:37.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.676+0000 7ffa80fe5640 1 -- 172.21.15.42:0/4112333812 wait complete. 2024-09-16T03:45:37.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.676+0000 7ffa80fe5640 1 librados: shutdown 2024-09-16T03:45:37.716 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T03:45:37.716 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-16T03:45:37.716 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T03:45:37.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.828+0000 7fa3d9a7d640 10 monclient: get_monmap_and_config 2024-09-16T03:45:37.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.828+0000 7fa3d9a7d640 10 monclient: build_initial_monmap 2024-09-16T03:45:37.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.828+0000 7fa3d9a7d640 10 monclient: monmap: 2024-09-16T03:45:37.836 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:45:37.836 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:45:37.836 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:45:37.835223+0000 2024-09-16T03:45:37.836 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:45:37.835223+0000 2024-09-16T03:45:37.836 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:45:37.836 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:45:37.836 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:45:37.836 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:45:37.837 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:45:37.837 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:45:37.837 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:45:37.837 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:45:37.848 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.828+0000 7fa3d9a7d640 1 Processor -- start 2024-09-16T03:45:37.849 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.832+0000 7fa3d9a7d640 1 -- start start 2024-09-16T03:45:37.849 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.832+0000 7fa3d9a7d640 10 monclient: init 2024-09-16T03:45:37.849 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.832+0000 7fa3d9a7d640 10 monclient: _reopen_session rank -1 2024-09-16T03:45:37.849 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.832+0000 7fa3d9a7d640 10 monclient: _add_conns ranks=[4,0,3,1,2] 2024-09-16T03:45:37.849 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.832+0000 7fa3d9a7d640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fa3d4137620 0x7fa3d4137a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:45:37.849 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.832+0000 7fa3d9a7d640 10 monclient(hunting): picked mon.noname-c con 0x7fa3d4137620 addr v2:172.21.15.154:3300/0 2024-09-16T03:45:37.849 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.832+0000 7fa3d9a7d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa3d41370f0 addr v1:172.21.15.42:6789/0 2024-09-16T03:45:37.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.832+0000 7fa3d9a7d640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fa3d40a2940 0x7fa3d413f430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:45:37.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.832+0000 7fa3d9a7d640 10 monclient(hunting): picked mon.noname-a con 0x7fa3d40a2940 addr v2:172.21.15.42:3300/0 2024-09-16T03:45:37.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.832+0000 7fa3d9a7d640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa3d40a0ba0 con 0x7fa3d41370f0 2024-09-16T03:45:37.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.832+0000 7fa3d9a7d640 10 monclient(hunting): start opening mon connection 2024-09-16T03:45:37.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.832+0000 7fa3d9a7d640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7fa3d406e8b0 con 0x7fa3d40a2940 2024-09-16T03:45:37.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.832+0000 7fa3d9a7d640 10 monclient(hunting): start opening mon connection 2024-09-16T03:45:37.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.832+0000 7fa3d9a7d640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fa3d40a0e80 con 0x7fa3d4137620 2024-09-16T03:45:37.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.832+0000 7fa3d9a7d640 10 monclient(hunting): _renew_subs 2024-09-16T03:45:37.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.832+0000 7fa3d9a7d640 10 monclient(hunting): authenticate will time out at 1184.804837s 2024-09-16T03:45:37.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.832+0000 7fa3d2ffd640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fa3d41370f0 0x7fa3d40a2230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:44980/0 (socket says 172.21.15.42:44980) 2024-09-16T03:45:37.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.832+0000 7fa3d37fe640 1 -- >> v2:172.21.15.154:3300/0 conn(0x7fa3d4137620 msgr2=0x7fa3d4137a00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T03:45:37.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.832+0000 7fa3d2ffd640 1 -- 172.21.15.42:0/459455835 learned_addr learned my addr 172.21.15.42:0/459455835 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:45:37.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.832+0000 7fa3d3fff640 1 --2- 172.21.15.42:0/459455835 >> v2:172.21.15.42:3300/0 conn(0x7fa3d40a2940 0x7fa3d413f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:45:37.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.832+0000 7fa3d37fe640 1 --2- 172.21.15.42:0/459455835 >> v2:172.21.15.154:3300/0 conn(0x7fa3d4137620 0x7fa3d4137a00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:45:37.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.832+0000 7fa3d3fff640 10 monclient(hunting): get_auth_request con 0x7fa3d40a2940 auth_method 0 2024-09-16T03:45:37.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.832+0000 7fa3d3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:45:37.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.832+0000 7fa3d3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:45:37.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.832+0000 7fa3d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:45:37.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.832+0000 7fa3d3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:45:37.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.832+0000 7fa3d3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:45:37.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.832+0000 7fa3d3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:45:37.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.832+0000 7fa3d3fff640 10 monclient(hunting): handle_auth_done global_id 744234 payload 274 2024-09-16T03:45:37.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.832+0000 7fa3d3fff640 1 -- 172.21.15.42:0/459455835 >> v2:172.21.15.154:3300/0 conn(0x7fa3d4137620 msgr2=0x7fa3d4137a00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:45:37.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.832+0000 7fa3d3fff640 1 --2- 172.21.15.42:0/459455835 >> v2:172.21.15.154:3300/0 conn(0x7fa3d4137620 0x7fa3d4137a00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:45:37.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.832+0000 7fa3d3fff640 1 -- 172.21.15.42:0/459455835 >> v1:172.21.15.42:6789/0 conn(0x7fa3d41370f0 legacy=0x7fa3d40a2230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:45:37.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.832+0000 7fa3d3fff640 10 monclient: _finish_hunting 0 2024-09-16T03:45:37.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.832+0000 7fa3d3fff640 1 monclient: found mon.noname-a 2024-09-16T03:45:37.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.832+0000 7fa3d3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:45:37.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.832+0000 7fa3d3fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:45:37.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.832+0000 7fa3d3fff640 1 -- 172.21.15.42:0/459455835 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3d4137f40 con 0x7fa3d40a2940 2024-09-16T03:45:37.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.832+0000 7fa3d3fff640 1 --2- 172.21.15.42:0/459455835 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa3d40a2940 0x7fa3d413f430 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fa3d407b760 tx=0x7fa3c401c360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db2fda7dd8a713be server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:45:37.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.832+0000 7fa3d27fc640 1 -- 172.21.15.42:0/459455835 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3c40b2070 con 0x7fa3d40a2940 2024-09-16T03:45:37.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.832+0000 7fa3d27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:45:37.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.832+0000 7fa3d27fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:45:37.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.832+0000 7fa3d27fc640 10 monclient: dump: 2024-09-16T03:45:37.853 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:45:37.853 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:45:37.853 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:37.853 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:37.853 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:45:37.853 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:45:37.853 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:45:37.853 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:45:37.853 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:45:37.853 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:45:37.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.832+0000 7fa3d27fc640 10 monclient: _finish_auth 0 2024-09-16T03:45:37.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.832+0000 7fa3d27fc640 10 monclient: _check_auth_tickets 2024-09-16T03:45:37.859 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.832+0000 7fa3d27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:45:37.859 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.832+0000 7fa3d27fc640 1 -- 172.21.15.42:0/459455835 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa3c40ad680 con 0x7fa3d40a2940 2024-09-16T03:45:37.859 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.832+0000 7fa3d9a7d640 5 monclient: authenticate success, global_id 744234 2024-09-16T03:45:37.859 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.832+0000 7fa3d9a7d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:45:37.859 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.832+0000 7fa3d27fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:45:37.859 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.832+0000 7fa3d27fc640 1 -- 172.21.15.42:0/459455835 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3c40ada50 con 0x7fa3d40a2940 2024-09-16T03:45:37.860 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.832+0000 7fa3d9a7d640 10 monclient: get_monmap_and_config success 2024-09-16T03:45:37.860 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.832+0000 7fa3d9a7d640 4 set_mon_vals no callback set 2024-09-16T03:45:37.860 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.832+0000 7fa3d27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:45:37.860 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.832+0000 7fa3d27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:45:37.860 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.832+0000 7fa3d27fc640 10 monclient: dump: 2024-09-16T03:45:37.860 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:45:37.860 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:45:37.860 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:37.860 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:37.860 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:45:37.861 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:45:37.861 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:45:37.861 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:45:37.861 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:45:37.861 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:45:37.861 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.836+0000 7fa3d9a7d640 10 monclient: shutdown 2024-09-16T03:45:37.861 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.836+0000 7fa3d9a7d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:45:37.861 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.836+0000 7fa3d9a7d640 1 -- 172.21.15.42:0/459455835 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa3d40a2940 msgr2=0x7fa3d413f430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:45:37.861 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.836+0000 7fa3d9a7d640 1 --2- 172.21.15.42:0/459455835 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa3d40a2940 0x7fa3d413f430 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fa3d407b760 tx=0x7fa3c401c360 comp rx=0 tx=0).stop 2024-09-16T03:45:37.861 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.836+0000 7fa3d9a7d640 1 -- 172.21.15.42:0/459455835 shutdown_connections 2024-09-16T03:45:37.862 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.836+0000 7fa3d9a7d640 1 --2- 172.21.15.42:0/459455835 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa3d40a2940 0x7fa3d413f430 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:45:37.862 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.836+0000 7fa3d9a7d640 1 --2- 172.21.15.42:0/459455835 >> v2:172.21.15.154:3300/0 conn(0x7fa3d4137620 0x7fa3d4137a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:45:37.862 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.836+0000 7fa3d9a7d640 1 -- 172.21.15.42:0/459455835 >> 172.21.15.42:0/459455835 conn(0x7fa3d40a6e80 msgr2=0x7fa3d40a3320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:45:37.862 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.836+0000 7fa3d9a7d640 1 -- 172.21.15.42:0/459455835 shutdown_connections 2024-09-16T03:45:37.862 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.836+0000 7fa3d9a7d640 1 -- 172.21.15.42:0/459455835 wait complete. 2024-09-16T03:45:37.862 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.836+0000 7fa3d9a7d640 10 monclient: build_initial_monmap 2024-09-16T03:45:37.862 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.836+0000 7fa3d9a7d640 10 monclient: monmap: 2024-09-16T03:45:37.862 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:45:37.862 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:45:37.862 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:45:37.862 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:45:37.862 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:45:37.862 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:45:37.862 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:45:37.863 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:45:37.863 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:45:37.863 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:45:37.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.836+0000 7fa3d9a7d640 1 librados: starting msgr at 2024-09-16T03:45:37.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.836+0000 7fa3d9a7d640 1 librados: starting objecter 2024-09-16T03:45:37.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.836+0000 7fa3d9a7d640 1 Processor -- start 2024-09-16T03:45:37.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.836+0000 7fa3d9a7d640 1 -- start start 2024-09-16T03:45:37.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.836+0000 7fa3d9a7d640 1 librados: setting wanted keys 2024-09-16T03:45:37.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.836+0000 7fa3d9a7d640 1 librados: calling monclient init 2024-09-16T03:45:37.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.836+0000 7fa3d9a7d640 10 monclient: init 2024-09-16T03:45:37.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.836+0000 7fa3d9a7d640 10 monclient: _reopen_session rank -1 2024-09-16T03:45:37.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.836+0000 7fa3d9a7d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T03:45:37.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.836+0000 7fa3d9a7d640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa3d40a2940 0x7fa3d4135fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:45:37.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.836+0000 7fa3d9a7d640 10 monclient(hunting): picked mon.noname-a con 0x7fa3d40a2940 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:45:37.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.836+0000 7fa3d9a7d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa3d41370f0 0x7fa3d41364f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:45:37.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.836+0000 7fa3d9a7d640 10 monclient(hunting): picked mon.noname-b con 0x7fa3d41370f0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:45:37.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.836+0000 7fa3d9a7d640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa3d4137620 0x7fa3d411ce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:45:37.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.836+0000 7fa3d9a7d640 10 monclient(hunting): picked mon.noname-c con 0x7fa3d4137620 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:45:37.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.836+0000 7fa3d9a7d640 10 monclient(hunting): start opening mon connection 2024-09-16T03:45:37.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.836+0000 7fa3d9a7d640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fa3d4142520 con 0x7fa3d40a2940 2024-09-16T03:45:37.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.836+0000 7fa3d9a7d640 10 monclient(hunting): start opening mon connection 2024-09-16T03:45:37.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.836+0000 7fa3d9a7d640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa3d41423a0 con 0x7fa3d41370f0 2024-09-16T03:45:37.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.836+0000 7fa3d9a7d640 10 monclient(hunting): start opening mon connection 2024-09-16T03:45:37.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.836+0000 7fa3d9a7d640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fa3d41426a0 con 0x7fa3d4137620 2024-09-16T03:45:37.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.836+0000 7fa3d9a7d640 10 monclient(hunting): _renew_subs 2024-09-16T03:45:37.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.836+0000 7fa3d9a7d640 10 monclient(hunting): authenticate will time out at 1184.810089s 2024-09-16T03:45:37.865 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.836+0000 7fa3d37fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa3d40a2940 0x7fa3d4135fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:45:37.865 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.836+0000 7fa3d37fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa3d40a2940 0x7fa3d4135fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:42774/0 (socket says 172.21.15.42:42774) 2024-09-16T03:45:37.865 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.836+0000 7fa3d37fe640 1 -- 172.21.15.42:0/1419676785 learned_addr learned my addr 172.21.15.42:0/1419676785 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:45:37.865 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.836+0000 7fa3d37fe640 10 monclient(hunting): get_auth_request con 0x7fa3d40a2940 auth_method 0 2024-09-16T03:45:37.865 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.836+0000 7fa3d37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:45:37.865 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.836+0000 7fa3d37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:45:37.865 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.836+0000 7fa3d37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:45:37.865 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.836+0000 7fa3d3fff640 1 --2- 172.21.15.42:0/1419676785 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa3d4137620 0x7fa3d411ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:45:37.865 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.836+0000 7fa3d2ffd640 1 -- 172.21.15.42:0/1419676785 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa3d41370f0 msgr2=0x7fa3d41364f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T03:45:37.865 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.836+0000 7fa3d2ffd640 1 --2- 172.21.15.42:0/1419676785 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa3d41370f0 0x7fa3d41364f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:45:37.865 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.836+0000 7fa3d3fff640 10 monclient(hunting): get_auth_request con 0x7fa3d4137620 auth_method 0 2024-09-16T03:45:37.865 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.836+0000 7fa3d3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:45:37.865 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.836+0000 7fa3d3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:45:37.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.836+0000 7fa3d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:45:37.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.836+0000 7fa3d37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:45:37.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.836+0000 7fa3d37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:45:37.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.836+0000 7fa3d37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:45:37.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.836+0000 7fa3d3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:45:37.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.836+0000 7fa3d3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:45:37.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.836+0000 7fa3d3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:45:37.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.836+0000 7fa3d3fff640 10 monclient(hunting): handle_auth_done global_id 13394 payload 824 2024-09-16T03:45:37.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.836+0000 7fa3d3fff640 1 -- 172.21.15.42:0/1419676785 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa3d41370f0 msgr2=0x7fa3d41364f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:45:37.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.836+0000 7fa3d3fff640 1 --2- 172.21.15.42:0/1419676785 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa3d41370f0 0x7fa3d41364f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:45:37.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.836+0000 7fa3d3fff640 1 -- 172.21.15.42:0/1419676785 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa3d40a2940 msgr2=0x7fa3d4135fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:45:37.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.836+0000 7fa3d3fff640 1 --2- 172.21.15.42:0/1419676785 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa3d40a2940 0x7fa3d4135fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:45:37.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.836+0000 7fa3d3fff640 10 monclient: _finish_hunting 0 2024-09-16T03:45:37.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.836+0000 7fa3d3fff640 1 monclient: found mon.noname-c 2024-09-16T03:45:37.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.836+0000 7fa3d3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:45:37.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.836+0000 7fa3d3fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:45:37.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.836+0000 7fa3d3fff640 1 -- 172.21.15.42:0/1419676785 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3d413f270 con 0x7fa3d4137620 2024-09-16T03:45:37.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.836+0000 7fa3d37fe640 1 --2- 172.21.15.42:0/1419676785 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa3d40a2940 0x7fa3d4135fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:45:37.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.836+0000 7fa3d3fff640 1 --2- 172.21.15.42:0/1419676785 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa3d4137620 0x7fa3d411ce70 secure :-1 s=READY pgs=2378 cs=0 l=1 rev1=1 crypto rx=0x7fa3c40ac000 tx=0x7fa3c401aba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:45:37.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.836+0000 7fa3cbfff640 10 client.?.objecter ms_handle_connect 0x7fa3d4137620 2024-09-16T03:45:37.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.836+0000 7fa3cbfff640 10 client.?.objecter resend_mon_ops 2024-09-16T03:45:37.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.836+0000 7fa3cbfff640 1 -- 172.21.15.42:0/1419676785 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3c40bc020 con 0x7fa3d4137620 2024-09-16T03:45:37.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.836+0000 7fa3cbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:45:37.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.836+0000 7fa3cbfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:45:37.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.836+0000 7fa3cbfff640 10 monclient: dump: 2024-09-16T03:45:37.867 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:45:37.868 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:45:37.868 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:37.868 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:37.868 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:45:37.868 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:45:37.868 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:45:37.868 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:45:37.868 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:45:37.868 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:45:37.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.836+0000 7fa3cbfff640 10 monclient: _finish_auth 0 2024-09-16T03:45:37.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.836+0000 7fa3cbfff640 10 monclient: _check_auth_tickets 2024-09-16T03:45:37.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.836+0000 7fa3cbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:45:37.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.836+0000 7fa3d9a7d640 5 monclient: authenticate success, global_id 13394 2024-09-16T03:45:37.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.836+0000 7fa3cbfff640 1 -- 172.21.15.42:0/1419676785 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa3c40adcb0 con 0x7fa3d4137620 2024-09-16T03:45:37.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.836+0000 7fa3cbfff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:45:37.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.836+0000 7fa3cbfff640 1 -- 172.21.15.42:0/1419676785 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3c40b2040 con 0x7fa3d4137620 2024-09-16T03:45:37.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.836+0000 7fa3d9a7d640 10 monclient: _renew_subs 2024-09-16T03:45:37.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.836+0000 7fa3d9a7d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:45:37.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.836+0000 7fa3d9a7d640 1 -- 172.21.15.42:0/1419676785 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3d40738a0 con 0x7fa3d4137620 2024-09-16T03:45:37.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.836+0000 7fa3cbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:45:37.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.836+0000 7fa3d1ffb640 4 set_mon_vals no callback set 2024-09-16T03:45:37.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.836+0000 7fa3cbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:45:37.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.836+0000 7fa3cbfff640 10 monclient: dump: 2024-09-16T03:45:37.869 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:45:37.869 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:45:37.869 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:37.869 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:37.869 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:45:37.870 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:45:37.870 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:45:37.870 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:45:37.870 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:45:37.870 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:45:37.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.836+0000 7fa3d9a7d640 10 client.13394.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:45:37.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.836+0000 7fa3d9a7d640 10 monclient: _renew_subs 2024-09-16T03:45:37.871 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.836+0000 7fa3d9a7d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:45:37.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.836+0000 7fa3d9a7d640 1 -- 172.21.15.42:0/1419676785 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3d41207b0 con 0x7fa3d4137620 2024-09-16T03:45:37.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.836+0000 7fa3d9a7d640 1 librados: init done 2024-09-16T03:45:37.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.836+0000 7fa3d9a7d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:45:37.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.836+0000 7fa3d9a7d640 1 --2- 172.21.15.42:0/1419676785 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa3d4120980 0x7fa3d411d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:45:37.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.836+0000 7fa3d9a7d640 10 monclient(hunting): start opening mon connection 2024-09-16T03:45:37.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.836+0000 7fa3d9a7d640 1 -- 172.21.15.42:0/1419676785 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fa3d4068250 con 0x7fa3d4120980 2024-09-16T03:45:37.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.836+0000 7fa3d37fe640 1 --2- 172.21.15.42:0/1419676785 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa3d4120980 0x7fa3d411d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:45:37.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.840+0000 7fa3d37fe640 10 monclient: get_auth_request con 0x7fa3d4120980 auth_method 0 2024-09-16T03:45:37.873 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.840+0000 7fa3d37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:45:37.873 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.840+0000 7fa3d37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:45:37.873 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.840+0000 7fa3d37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:45:37.873 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.840+0000 7fa3cbfff640 1 -- 172.21.15.42:0/1419676785 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fa3c40bf020 con 0x7fa3d4137620 2024-09-16T03:45:37.873 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.840+0000 7fa3d37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:45:37.873 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.840+0000 7fa3d37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:45:37.873 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.840+0000 7fa3d37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:45:37.873 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.840+0000 7fa3d37fe640 10 monclient(hunting): handle_auth_done global_id 13397 payload 824 2024-09-16T03:45:37.873 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.840+0000 7fa3d37fe640 1 -- 172.21.15.42:0/1419676785 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa3d4137a00 con 0x7fa3d4120980 2024-09-16T03:45:37.873 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.840+0000 7fa3cbfff640 1 --2- 172.21.15.42:0/1419676785 >> v2:172.21.15.154:6812/254845342 conn(0x7fa3b40377c0 0x7fa3b4039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:45:37.873 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.840+0000 7fa3cbfff640 1 -- 172.21.15.42:0/1419676785 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(91..91 src has 1..91) ==== 7832+0+0 (secure 0 0 0) 0x7fa3c40e3960 con 0x7fa3d4137620 2024-09-16T03:45:37.873 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.840+0000 7fa3cbfff640 10 client.13394.objecter ms_dispatch 0x7fa3d4000ba0 osd_map(91..91 src has 1..91) 2024-09-16T03:45:37.873 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.840+0000 7fa3cbfff640 3 client.13394.objecter handle_osd_map got epochs [91,91] > 0 2024-09-16T03:45:37.874 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.840+0000 7fa3cbfff640 3 client.13394.objecter handle_osd_map decoding full epoch 91 2024-09-16T03:45:37.874 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.840+0000 7fa3d37fe640 1 --2- 172.21.15.42:0/1419676785 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa3d4120980 0x7fa3d411d3b0 secure :-1 s=READY pgs=2379 cs=0 l=1 rev1=1 crypto rx=0x7fa3cc00e4e0 tx=0x7fa3cc00e9a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:45:37.874 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.840+0000 7fa3cbfff640 20 client.13394.objecter dump_active .. 0 homeless 2024-09-16T03:45:37.874 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.840+0000 7fa3cbfff640 10 client.13394.objecter ms_handle_connect 0x7fa3d4120980 2024-09-16T03:45:37.874 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.840+0000 7fa3cbfff640 10 client.13394.objecter resend_mon_ops 2024-09-16T03:45:37.874 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.840+0000 7fa3d2ffd640 1 --2- 172.21.15.42:0/1419676785 >> v2:172.21.15.154:6812/254845342 conn(0x7fa3b40377c0 0x7fa3b4039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:45:37.874 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.840+0000 7fa3cbfff640 1 -- 172.21.15.42:0/1419676785 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3cc0054e0 con 0x7fa3d4120980 2024-09-16T03:45:37.874 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.840+0000 7fa3cbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:45:37.874 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.840+0000 7fa3cbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:45:37.874 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.840+0000 7fa3cbfff640 10 monclient: dump: 2024-09-16T03:45:37.874 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:45:37.874 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:45:37.875 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:37.875 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:37.875 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:45:37.875 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:45:37.875 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:45:37.875 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:45:37.875 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:45:37.875 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:45:37.875 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.840+0000 7fa3d2ffd640 10 monclient: get_auth_request con 0x7fa3b40377c0 auth_method 0 2024-09-16T03:45:37.875 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.840+0000 7fa3cbfff640 1 -- 172.21.15.42:0/1419676785 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fa3cc018070 con 0x7fa3d4120980 2024-09-16T03:45:37.875 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.840+0000 7fa3cbfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:45:37.875 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.840+0000 7fa3cbfff640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:45:37.875 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.840+0000 7fa3cbfff640 1 -- 172.21.15.42:0/1419676785 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa3d4120980 msgr2=0x7fa3d411d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:45:37.876 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.840+0000 7fa3cbfff640 1 --2- 172.21.15.42:0/1419676785 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa3d4120980 0x7fa3d411d3b0 secure :-1 s=READY pgs=2379 cs=0 l=1 rev1=1 crypto rx=0x7fa3cc00e4e0 tx=0x7fa3cc00e9a0 comp rx=0 tx=0).stop 2024-09-16T03:45:37.876 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.840+0000 7fa3cbfff640 1 -- 172.21.15.42:0/1419676785 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa3d4120980 msgr2=0x7fa3d411d3b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:45:37.876 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.840+0000 7fa3cbfff640 1 --2- 172.21.15.42:0/1419676785 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa3d4120980 0x7fa3d411d3b0 unknown :-1 s=CLOSED pgs=2379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:45:37.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.840+0000 7fa3cbfff640 10 client.13394.objecter ms_handle_connect 0x7fa3b40377c0 2024-09-16T03:45:37.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.840+0000 7fa3d2ffd640 1 --2- 172.21.15.42:0/1419676785 >> v2:172.21.15.154:6812/254845342 conn(0x7fa3b40377c0 0x7fa3b4039c80 secure :-1 s=READY pgs=3372 cs=0 l=1 rev1=1 crypto rx=0x7fa3d411e490 tx=0x7fa3bc007b10 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:45:37.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.844+0000 7fa3d9a7d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:45:37.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.844+0000 7fa3d9a7d640 1 --2- 172.21.15.42:0/1419676785 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa3d411d8f0 0x7fa3d411dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:45:37.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.844+0000 7fa3d9a7d640 10 monclient(hunting): start opening mon connection 2024-09-16T03:45:37.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.844+0000 7fa3d9a7d640 1 -- 172.21.15.42:0/1419676785 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fa3d411e240 con 0x7fa3d411d8f0 2024-09-16T03:45:37.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.844+0000 7fa3d37fe640 1 --2- 172.21.15.42:0/1419676785 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa3d411d8f0 0x7fa3d411dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:45:37.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.844+0000 7fa3d37fe640 10 monclient: get_auth_request con 0x7fa3d411d8f0 auth_method 0 2024-09-16T03:45:37.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.844+0000 7fa3d37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:45:37.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.844+0000 7fa3d37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:45:37.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.844+0000 7fa3d37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:45:37.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.844+0000 7fa3d37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:45:37.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.844+0000 7fa3d37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:45:37.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.844+0000 7fa3d37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:45:37.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.844+0000 7fa3d37fe640 10 monclient(hunting): handle_auth_done global_id 13400 payload 824 2024-09-16T03:45:37.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.844+0000 7fa3d37fe640 1 -- 172.21.15.42:0/1419676785 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fa3d406eda0 con 0x7fa3d411d8f0 2024-09-16T03:45:37.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.844+0000 7fa3cbfff640 10 client.13394.objecter ms_handle_connect 0x7fa3d411d8f0 2024-09-16T03:45:37.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.844+0000 7fa3cbfff640 10 client.13394.objecter resend_mon_ops 2024-09-16T03:45:37.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.844+0000 7fa3d37fe640 1 --2- 172.21.15.42:0/1419676785 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa3d411d8f0 0x7fa3d411dd00 secure :-1 s=READY pgs=2380 cs=0 l=1 rev1=1 crypto rx=0x7fa3cc00e4e0 tx=0x7fa3cc005d60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:45:37.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.844+0000 7fa3cbfff640 1 -- 172.21.15.42:0/1419676785 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3cc01c620 con 0x7fa3d411d8f0 2024-09-16T03:45:37.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.844+0000 7fa3cbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:45:37.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.844+0000 7fa3cbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:45:37.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.844+0000 7fa3cbfff640 10 monclient: dump: 2024-09-16T03:45:37.880 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:45:37.880 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:45:37.880 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:37.880 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:37.881 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:45:37.882 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:45:37.882 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:45:37.882 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:45:37.882 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:45:37.882 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:45:37.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.848+0000 7fa3cbfff640 1 -- 172.21.15.42:0/1419676785 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4869 (secure 0 0 0) 0x7fa3cc01c620 con 0x7fa3d411d8f0 2024-09-16T03:45:37.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.848+0000 7fa3cbfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:45:37.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.848+0000 7fa3cbfff640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:45:37.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.848+0000 7fa3cbfff640 1 -- 172.21.15.42:0/1419676785 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa3d411d8f0 msgr2=0x7fa3d411dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:45:37.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.848+0000 7fa3cbfff640 1 --2- 172.21.15.42:0/1419676785 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa3d411d8f0 0x7fa3d411dd00 secure :-1 s=READY pgs=2380 cs=0 l=1 rev1=1 crypto rx=0x7fa3cc00e4e0 tx=0x7fa3cc005d60 comp rx=0 tx=0).stop 2024-09-16T03:45:37.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.848+0000 7fa3cbfff640 1 -- 172.21.15.42:0/1419676785 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa3d411d8f0 msgr2=0x7fa3d411dd00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:45:37.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.848+0000 7fa3cbfff640 1 --2- 172.21.15.42:0/1419676785 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa3d411d8f0 0x7fa3d411dd00 secure :-1 s=CLOSED pgs=2380 cs=0 l=1 rev1=1 crypto rx=0x7fa3cc00e4e0 tx=0x7fa3cc005d60 comp rx=0 tx=0).stop 2024-09-16T03:45:37.883 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:45:37.883 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:45:37.883 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:45:37.883 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T03:45:37.883 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 32, 2024-09-16T03:45:37.883 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 217215, 2024-09-16T03:45:37.883 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:45:37.883 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:45:37.883 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T03:45:37.883 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:45:37.884 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 0, 2024-09-16T03:45:37.884 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:45:37.884 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:45:37.884 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:45:37.884 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:45:37.884 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:45:37.884 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:45:37.884 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:45:37.884 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:45:37.884 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:45:37.884 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:45:37.884 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:45:37.884 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:45:37.885 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:45:37.885 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:45:37.885 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:45:37.885 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:45:37.885 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:45:37.885 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:45:37.885 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:45:37.885 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:45:37.885 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:45:37.885 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:45:37.885 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:45:37.885 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:45:37.885 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:45:37.885 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:45:37.886 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:45:37.886 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:45:37.886 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:45:37.886 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:45:37.886 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:45:37.886 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:45:37.886 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:45:37.886 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:45:37.886 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:45:37.886 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:45:37.886 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:45:37.886 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:45:37.886 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:45:37.886 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:45:37.887 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:45:37.887 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:45:37.887 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:45:37.887 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:45:37.887 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:45:37.887 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:45:37.887 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:45:37.887 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:45:37.887 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:45:37.887 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:45:37.887 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:45:37.887 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:45:37.887 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:45:37.887 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:45:37.888 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:45:37.888 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:45:37.888 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:45:37.888 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:45:37.888 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:45:37.888 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:45:37.888 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:45:37.888 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:45:37.888 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:45:37.889 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:45:37.889 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:45:37.889 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:45:37.889 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:45:37.889 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:45:37.889 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:45:37.889 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:45:37.889 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:45:37.889 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:45:37.889 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:45:37.889 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:45:37.889 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:45:37.889 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:45:37.889 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:45:37.890 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:45:37.890 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:45:37.890 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:45:37.890 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:45:37.890 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:45:37.890 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:45:37.890 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:45:37.890 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:45:37.890 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:45:37.890 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:45:37.890 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:45:37.890 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:45:37.890 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:45:37.890 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:45:37.891 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:45:37.891 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:45:37.891 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:45:37.891 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:45:37.891 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:45:37.891 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:45:37.891 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:45:37.891 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:45:37.891 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:45:37.891 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:45:37.891 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:45:37.891 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:45:37.891 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:45:37.891 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:45:37.891 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:45:37.892 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:45:37.892 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:45:37.892 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:45:37.892 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:45:37.892 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:45:37.892 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:45:37.892 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:45:37.892 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:45:37.892 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:45:37.893 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:45:37.893 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:45:37.893 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:45:37.893 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:45:37.893 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:45:37.893 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:45:37.893 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:45:37.893 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:45:37.893 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:45:37.893 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:45:37.893 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:45:37.893 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:45:37.893 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:45:37.893 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:45:37.894 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:45:37.894 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:45:37.894 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:45:37.894 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:45:37.894 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:45:37.894 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:45:37.894 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:45:37.894 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:45:37.894 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:45:37.894 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:45:37.894 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:45:37.894 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:45:37.894 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T03:45:37.894 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:45:37.895 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:45:37.895 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:45:37.895 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 5 2024-09-16T03:45:37.895 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:45:37.895 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:45:37.895 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:45:37.895 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:45:37.895 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:45:37.895 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:45:37.895 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 12 2024-09-16T03:45:37.895 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:45:37.895 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:45:37.895 INFO:teuthology.orchestra.run.smithi042.stdout: "mgr": [ 2024-09-16T03:45:37.895 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:45:37.895 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:45:37.896 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:45:37.896 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:45:37.896 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:45:37.896 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:45:37.896 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:45:37.896 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:45:37.896 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:45:37.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.848+0000 7fa3d9a7d640 10 librados: watch_flush enter 2024-09-16T03:45:37.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.848+0000 7fa3d9a7d640 10 librados: watch_flush exit 2024-09-16T03:45:37.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.848+0000 7fa3d9a7d640 20 client.13394.objecter shutdown clearing up homeless session... 2024-09-16T03:45:37.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.848+0000 7fa3d9a7d640 10 client.13394.objecter successfully canceled tick 2024-09-16T03:45:37.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.848+0000 7fa3d9a7d640 1 -- 172.21.15.42:0/1419676785 >> v2:172.21.15.154:6812/254845342 conn(0x7fa3b40377c0 msgr2=0x7fa3b4039c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:45:37.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.848+0000 7fa3d9a7d640 1 --2- 172.21.15.42:0/1419676785 >> v2:172.21.15.154:6812/254845342 conn(0x7fa3b40377c0 0x7fa3b4039c80 secure :-1 s=READY pgs=3372 cs=0 l=1 rev1=1 crypto rx=0x7fa3d411e490 tx=0x7fa3bc007b10 comp rx=0 tx=0).stop 2024-09-16T03:45:37.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.848+0000 7fa3d9a7d640 10 monclient: shutdown 2024-09-16T03:45:37.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.848+0000 7fa3d9a7d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:45:37.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.848+0000 7fa3d9a7d640 1 -- 172.21.15.42:0/1419676785 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa3d4137620 msgr2=0x7fa3d411ce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:45:37.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.848+0000 7fa3d9a7d640 1 --2- 172.21.15.42:0/1419676785 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa3d4137620 0x7fa3d411ce70 secure :-1 s=READY pgs=2378 cs=0 l=1 rev1=1 crypto rx=0x7fa3c40ac000 tx=0x7fa3c401aba0 comp rx=0 tx=0).stop 2024-09-16T03:45:37.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.848+0000 7fa3d3fff640 1 -- 172.21.15.42:0/1419676785 reap_dead start 2024-09-16T03:45:37.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.848+0000 7fa3d3fff640 1 -- 172.21.15.42:0/1419676785 reap_dead start 2024-09-16T03:45:37.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.848+0000 7fa3d9a7d640 1 -- 172.21.15.42:0/1419676785 shutdown_connections 2024-09-16T03:45:37.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.848+0000 7fa3d9a7d640 1 -- 172.21.15.42:0/1419676785 >> 172.21.15.42:0/1419676785 conn(0x7fa3d40a6e80 msgr2=0x7fa3d4070230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:45:37.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.848+0000 7fa3d9a7d640 1 -- 172.21.15.42:0/1419676785 shutdown_connections 2024-09-16T03:45:37.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.848+0000 7fa3d9a7d640 1 -- 172.21.15.42:0/1419676785 wait complete. 2024-09-16T03:45:37.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:37.848+0000 7fa3d9a7d640 1 librados: shutdown 2024-09-16T03:45:37.899 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T03:45:37.899 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-16T03:45:38.156 INFO:tasks.workunit.client.0.smithi154.stdout: snapshots: Running main() from gmock_main.cc 2024-09-16T03:45:38.156 INFO:tasks.workunit.client.0.smithi154.stdout: snapshots: [==========] Running 11 tests from 2 test suites. 2024-09-16T03:45:38.156 INFO:tasks.workunit.client.0.smithi154.stdout: snapshots: [----------] Global test environment set-up. 2024-09-16T03:45:38.156 INFO:tasks.workunit.client.0.smithi154.stdout: snapshots: [----------] 5 tests from NeoRadosSnapshots 2024-09-16T03:45:38.157 INFO:tasks.workunit.client.0.smithi154.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapList 2024-09-16T03:45:38.157 INFO:tasks.workunit.client.0.smithi154.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapList (28640 ms) 2024-09-16T03:45:38.157 INFO:tasks.workunit.client.0.smithi154.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapRemove 2024-09-16T03:45:38.157 INFO:tasks.workunit.client.0.smithi154.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapRemove (4741 ms) 2024-09-16T03:45:38.157 INFO:tasks.workunit.client.0.smithi154.stdout: snapshots: [ RUN ] NeoRadosSnapshots.Rollback 2024-09-16T03:45:38.157 INFO:tasks.workunit.client.0.smithi154.stdout: snapshots: [ OK ] NeoRadosSnapshots.Rollback (4000 ms) 2024-09-16T03:45:38.157 INFO:tasks.workunit.client.0.smithi154.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapGetName 2024-09-16T03:45:38.157 INFO:tasks.workunit.client.0.smithi154.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapGetName (5030 ms) 2024-09-16T03:45:38.157 INFO:tasks.workunit.client.0.smithi154.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapCreateRemove 2024-09-16T03:45:38.157 INFO:tasks.workunit.client.0.smithi154.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapCreateRemove (12900 ms) 2024-09-16T03:45:38.157 INFO:tasks.workunit.client.0.smithi154.stdout: snapshots: [----------] 5 tests from NeoRadosSnapshots (55311 ms total) 2024-09-16T03:45:38.157 INFO:tasks.workunit.client.0.smithi154.stdout: snapshots: 2024-09-16T03:45:38.157 INFO:tasks.workunit.client.0.smithi154.stdout: snapshots: [----------] 6 tests from NeoRadosSelfManagedSnaps 2024-09-16T03:45:38.157 INFO:tasks.workunit.client.0.smithi154.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.Snap 2024-09-16T03:45:38.158 INFO:tasks.workunit.client.0.smithi154.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.Snap (5209 ms) 2024-09-16T03:45:38.158 INFO:tasks.workunit.client.0.smithi154.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.Rollback 2024-09-16T03:45:38.158 INFO:tasks.workunit.client.0.smithi154.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.Rollback (20266 ms) 2024-09-16T03:45:38.158 INFO:tasks.workunit.client.0.smithi154.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.SnapOverlap 2024-09-16T03:45:38.158 INFO:tasks.workunit.client.0.smithi154.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.SnapOverlap (7727 ms) 2024-09-16T03:45:38.158 INFO:tasks.workunit.client.0.smithi154.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.Bug11677 2024-09-16T03:45:38.158 INFO:tasks.workunit.client.0.smithi154.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.Bug11677 (6014 ms) 2024-09-16T03:45:38.158 INFO:tasks.workunit.client.0.smithi154.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.OrderSnap 2024-09-16T03:45:38.158 INFO:tasks.workunit.client.0.smithi154.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.OrderSnap (3990 ms) 2024-09-16T03:45:38.158 INFO:tasks.workunit.client.0.smithi154.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.ReusePurgedSnap 2024-09-16T03:45:38.158 INFO:tasks.workunit.client.0.smithi154.stdout: snapshots: Deleting snap 3 in pool ReusePurgedSnapsmithi154-28653-11. 2024-09-16T03:45:38.158 INFO:tasks.workunit.client.0.smithi154.stdout: snapshots: Waiting for snaps to purge. 2024-09-16T03:45:38.158 INFO:tasks.workunit.client.0.smithi154.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.ReusePurgedSnap (29824 ms) 2024-09-16T03:45:38.158 INFO:tasks.workunit.client.0.smithi154.stdout: snapshots: [----------] 6 tests from NeoRadosSelfManagedSnaps (73030 ms total) 2024-09-16T03:45:38.158 INFO:tasks.workunit.client.0.smithi154.stdout: snapshots: 2024-09-16T03:45:38.159 INFO:tasks.workunit.client.0.smithi154.stdout: snapshots: [----------] Global test environment tear-down 2024-09-16T03:45:38.159 INFO:tasks.workunit.client.0.smithi154.stdout: snapshots: [==========] 11 tests from 2 test suites ran. (128341 ms total) 2024-09-16T03:45:38.159 INFO:tasks.workunit.client.0.smithi154.stdout: snapshots: [ PASSED ] 11 tests. 2024-09-16T03:45:44.667 INFO:tasks.workunit.client.0.smithi154.stdout: api_snapshots_pp: Running main() from gmock_main.cc 2024-09-16T03:45:44.668 INFO:tasks.workunit.client.0.smithi154.stdout: api_snapshots_pp: [==========] Running 21 tests from 5 test suites. 2024-09-16T03:45:44.668 INFO:tasks.workunit.client.0.smithi154.stdout: api_snapshots_pp: [----------] Global test environment set-up. 2024-09-16T03:45:44.668 INFO:tasks.workunit.client.0.smithi154.stdout: api_snapshots_pp: [----------] 5 tests from LibRadosSnapshotsPP 2024-09-16T03:45:44.668 INFO:tasks.workunit.client.0.smithi154.stdout: api_snapshots_pp: seed 28240 2024-09-16T03:45:44.668 INFO:tasks.workunit.client.0.smithi154.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapListPP 2024-09-16T03:45:44.668 INFO:tasks.workunit.client.0.smithi154.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapListPP (22957 ms) 2024-09-16T03:45:44.669 INFO:tasks.workunit.client.0.smithi154.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapRemovePP 2024-09-16T03:45:44.669 INFO:tasks.workunit.client.0.smithi154.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapRemovePP (2196 ms) 2024-09-16T03:45:44.669 INFO:tasks.workunit.client.0.smithi154.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.RollbackPP 2024-09-16T03:45:44.669 INFO:tasks.workunit.client.0.smithi154.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.RollbackPP (1837 ms) 2024-09-16T03:45:44.669 INFO:tasks.workunit.client.0.smithi154.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapGetNamePP 2024-09-16T03:45:44.670 INFO:tasks.workunit.client.0.smithi154.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapGetNamePP (1936 ms) 2024-09-16T03:45:44.670 INFO:tasks.workunit.client.0.smithi154.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapCreateRemovePP 2024-09-16T03:45:44.670 INFO:tasks.workunit.client.0.smithi154.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapCreateRemovePP (3942 ms) 2024-09-16T03:45:44.670 INFO:tasks.workunit.client.0.smithi154.stdout: api_snapshots_pp: [----------] 5 tests from LibRadosSnapshotsPP (32868 ms total) 2024-09-16T03:45:44.670 INFO:tasks.workunit.client.0.smithi154.stdout: api_snapshots_pp: 2024-09-16T03:45:44.670 INFO:tasks.workunit.client.0.smithi154.stdout: api_snapshots_pp: [----------] 7 tests from LibRadosSnapshotsSelfManagedPP 2024-09-16T03:45:44.671 INFO:tasks.workunit.client.0.smithi154.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.SnapPP 2024-09-16T03:45:44.671 INFO:tasks.workunit.client.0.smithi154.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.SnapPP (4052 ms) 2024-09-16T03:45:44.671 INFO:tasks.workunit.client.0.smithi154.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.RollbackPP 2024-09-16T03:45:44.671 INFO:tasks.workunit.client.0.smithi154.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.RollbackPP (18993 ms) 2024-09-16T03:45:44.671 INFO:tasks.workunit.client.0.smithi154.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.SnapOverlapPP 2024-09-16T03:45:44.671 INFO:tasks.workunit.client.0.smithi154.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.SnapOverlapPP (20056 ms) 2024-09-16T03:45:44.671 INFO:tasks.workunit.client.0.smithi154.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.Bug11677 2024-09-16T03:45:44.672 INFO:tasks.workunit.client.0.smithi154.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.Bug11677 (4031 ms) 2024-09-16T03:45:44.672 INFO:tasks.workunit.client.0.smithi154.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.OrderSnap 2024-09-16T03:45:44.672 INFO:tasks.workunit.client.0.smithi154.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.OrderSnap (2086 ms) 2024-09-16T03:45:44.672 INFO:tasks.workunit.client.0.smithi154.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.WriteRollback 2024-09-16T03:45:44.672 INFO:tasks.workunit.client.0.smithi154.stdout: api_snapshots_pp: ./src/test/librados/snapshots_cxx.cc:460: Skipped 2024-09-16T03:45:44.672 INFO:tasks.workunit.client.0.smithi154.stdout: api_snapshots_pp: 2024-09-16T03:45:44.673 INFO:tasks.workunit.client.0.smithi154.stdout: api_snapshots_pp: [ SKIPPED ] LibRadosSnapshotsSelfManagedPP.WriteRollback (0 ms) 2024-09-16T03:45:44.673 INFO:tasks.workunit.client.0.smithi154.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.ReusePurgedSnap 2024-09-16T03:45:44.673 INFO:tasks.workunit.client.0.smithi154.stdout: api_snapshots_pp: deleting snap 14 in pool LibRadosSnapshotsSelfManagedPP_smithi154-28240-7 2024-09-16T03:45:44.673 INFO:tasks.workunit.client.0.smithi154.stdout: api_snapshots_pp: waiting for snaps to purge 2024-09-16T03:45:44.673 INFO:tasks.workunit.client.0.smithi154.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.ReusePurgedSnap (17924 ms) 2024-09-16T03:45:44.673 INFO:tasks.workunit.client.0.smithi154.stdout: api_snapshots_pp: [----------] 7 tests from LibRadosSnapshotsSelfManagedPP (67142 ms total) 2024-09-16T03:45:44.673 INFO:tasks.workunit.client.0.smithi154.stdout: api_snapshots_pp: 2024-09-16T03:45:44.673 INFO:tasks.workunit.client.0.smithi154.stdout: api_snapshots_pp: [----------] 2 tests from LibRadosPoolIsInSelfmanagedSnapsMode 2024-09-16T03:45:44.674 INFO:tasks.workunit.client.0.smithi154.stdout: api_snapshots_pp: [ RUN ] LibRadosPoolIsInSelfmanagedSnapsMode.NotConnected 2024-09-16T03:45:44.674 INFO:tasks.workunit.client.0.smithi154.stdout: api_snapshots_pp: [ OK ] LibRadosPoolIsInSelfmanagedSnapsMode.NotConnected (45 ms) 2024-09-16T03:45:44.674 INFO:tasks.workunit.client.0.smithi154.stdout: api_snapshots_pp: [ RUN ] LibRadosPoolIsInSelfmanagedSnapsMode.FreshInstance 2024-09-16T03:45:44.674 INFO:tasks.workunit.client.0.smithi154.stdout: api_snapshots_pp: [ OK ] LibRadosPoolIsInSelfmanagedSnapsMode.FreshInstance (4504 ms) 2024-09-16T03:45:44.674 INFO:tasks.workunit.client.0.smithi154.stdout: api_snapshots_pp: [----------] 2 tests from LibRadosPoolIsInSelfmanagedSnapsMode (4552 ms total) 2024-09-16T03:45:44.674 INFO:tasks.workunit.client.0.smithi154.stdout: api_snapshots_pp: 2024-09-16T03:45:44.675 INFO:tasks.workunit.client.0.smithi154.stdout: api_snapshots_pp: [----------] 4 tests from LibRadosSnapshotsECPP 2024-09-16T03:45:44.675 INFO:tasks.workunit.client.0.smithi154.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapListPP 2024-09-16T03:45:44.675 INFO:tasks.workunit.client.0.smithi154.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapListPP (3017 ms) 2024-09-16T03:45:44.675 INFO:tasks.workunit.client.0.smithi154.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapRemovePP 2024-09-16T03:45:44.675 INFO:tasks.workunit.client.0.smithi154.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapRemovePP (1508 ms) 2024-09-16T03:45:44.675 INFO:tasks.workunit.client.0.smithi154.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.RollbackPP 2024-09-16T03:45:44.676 INFO:tasks.workunit.client.0.smithi154.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.RollbackPP (2005 ms) 2024-09-16T03:45:48.687 INFO:tasks.workunit.client.0.smithi154.stdout: api_snapshocPP (93074 ms total) 2024-09-16T03:45:48.687 INFO:tasks.workunit.client.0.smithi154.stdout: api_misc_pp: 2024-09-16T03:45:48.687 INFO:tasks.workunit.client.0.smithi154.stdout: api_misc_pp: [----------] 1 test from LibRadosTwoPoolsECPP 2024-09-16T03:45:48.687 INFO:tasks.workunit.client.0.smithi154.stdout: api_misc_pp: [ RUN ] LibRadosTwoPoolsECPP.CopyFrom 2024-09-16T03:45:48.687 INFO:tasks.workunit.client.0.smithi154.stdout: api_misc_pp: [ OK ] LibRadosTwoPoolsECPP.CopyFrom (573 ms) 2024-09-16T03:45:48.688 INFO:tasks.workunit.client.0.smithi154.stdout: api_misc_pp: [----------] 1 test from LibRadosTwoPoolsECPP (573 ms total) 2024-09-16T03:45:48.688 INFO:tasks.workunit.client.0.smithi154.stdout: api_misc_pp: 2024-09-16T03:45:48.688 INFO:tasks.workunit.client.0.smithi154.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/0, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)0, Checksummer::xxhash32, ceph_le > 2024-09-16T03:45:48.688 INFO:tasks.workunit.client.0.smithi154.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/0.Subset 2024-09-16T03:45:48.688 INFO:tasks.workunit.client.0.smithi154.stdout: api_misc_pp: [ OK ] LibRadosChecksum/0.Subset (31 ms) 2024-09-16T03:45:48.688 INFO:tasks.workunit.client.0.smithi154.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/0.Chunked 2024-09-16T03:45:48.688 INFO:tasks.workunit.client.0.smithi154.stdout: api_misc_pp: [ OK ] LibRadosChecksum/0.Chunked (6 ms) 2024-09-16T03:45:48.688 INFO:tasks.workunit.client.0.smithi154.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/0 (37 ms total) 2024-09-16T03:45:48.688 INFO:tasks.workunit.client.0.smithi154.stdout: api_misc_pp: 2024-09-16T03:45:48.688 INFO:tasks.workunit.client.0.smithi154.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/1, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)1, Checksummer::xxhash64, ceph_le > 2024-09-16T03:45:48.689 INFO:tasks.workunit.client.0.smithi154.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/1.Subset 2024-09-16T03:45:48.689 INFO:tasks.workunit.client.0.smithi154.stdout: api_misc_pp: [ OK ] LibRadosChecksum/1.Subset (45 ms) 2024-09-16T03:45:48.689 INFO:tasks.workunit.client.0.smithi154.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/1.Chunked 2024-09-16T03:45:48.689 INFO:tasks.workunit.client.0.smithi154.stdout: api_misc_pp: [ OK ] LibRadosChecksum/1.Chunked (10 ms) 2024-09-16T03:45:48.689 INFO:tasks.workunit.client.0.smithi154.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/1 (55 ms total) 2024-09-16T03:45:48.689 INFO:tasks.workunit.client.0.smithi154.stdout: api_misc_pp: 2024-09-16T03:45:48.689 INFO:tasks.workunit.client.0.smithi154.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/2, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)2, Checksummer::crc32c, ceph_le > 2024-09-16T03:45:48.689 INFO:tasks.workunit.client.0.smithi154.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/2.Subset 2024-09-16T03:45:48.689 INFO:tasks.workunit.client.0.smithi154.stdout: api_misc_pp: [ OK ] LibRadosChecksum/2.Subset (72 ms) 2024-09-16T03:45:48.690 INFO:tasks.workunit.client.0.smithi154.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/2.Chunked 2024-09-16T03:45:48.690 INFO:tasks.workunit.client.0.smithi154.stdout: api_misc_pp: [ OK ] LibRadosChecksum/2.Chunked (5 ms) 2024-09-16T03:45:48.690 INFO:tasks.workunit.client.0.smithi154.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/2 (77 ms total) 2024-09-16T03:45:48.690 INFO:tasks.workunit.client.0.smithi154.stdout: api_misc_pp: 2024-09-16T03:45:48.690 INFO:tasks.workunit.client.0.smithi154.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscECPP 2024-09-16T03:45:48.690 INFO:tasks.workunit.client.0.smithi154.stdout: api_misc_pp: [ RUN ] LibRadosMiscECPP.CompareExtentRange 2024-09-16T03:45:48.690 INFO:tasks.workunit.client.0.smithi154.stdout: api_misc_pp: [ OK ] LibRadosMiscECPP.CompareExtentRange (1062 ms) 2024-09-16T03:45:48.690 INFO:tasks.workunit.client.0.smithi154.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscECPP (1062 ms total) 2024-09-16T03:45:48.690 INFO:tasks.workunit.client.0.smithi154.stdout: api_misc_pp: 2024-09-16T03:45:48.690 INFO:tasks.workunit.client.0.smithi154.stdout: api_misc_pp: [----------] Global test environment tear-down 2024-09-16T03:45:48.691 INFO:tasks.workunit.client.0.smithi154.stdout: api_misc_pp: [==========] 31 tests from 7 test suites ran. (139043 ms total) 2024-09-16T03:45:48.691 INFO:tasks.workunit.client.0.smithi154.stdout: api_misc_pp: [ PASSED ] 31 tests. 2024-09-16T03:45:53.681 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio: Running main() from gmock_main.cc 2024-09-16T03:45:53.682 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio: [==========] Running 42 tests from 2 test suites. 2024-09-16T03:45:53.682 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio: [----------] Global test environment set-up. 2024-09-16T03:45:53.682 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio: [----------] 26 tests from LibRadosAio 2024-09-16T03:45:53.682 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio: [ RUN ] LibRadosAio.TooBig 2024-09-16T03:45:53.682 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio: [ OK ] LibRadosAio.TooBig (4883 ms) 2024-09-16T03:45:53.682 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio: [ RUN ] LibRadosAio.SimpleWrite 2024-09-16T03:45:53.682 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio: [ OK ] LibRadosAio.SimpleWrite (24027 ms) 2024-09-16T03:45:53.682 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio: [ RUN ] LibRadosAio.WaitForSafe 2024-09-16T03:45:53.682 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio: [ OK ] LibRadosAio.WaitForSafe (2772 ms) 2024-09-16T03:45:53.682 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip 2024-09-16T03:45:53.682 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip (2973 ms) 2024-09-16T03:45:53.682 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip2 2024-09-16T03:45:53.682 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip2 (2982 ms) 2024-09-16T03:45:53.683 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip3 2024-09-16T03:45:53.683 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip3 (3055 ms) 2024-09-16T03:45:53.683 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripAppend 2024-09-16T03:45:53.683 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio: [ OK ] LibRadosAio.RoundTripAppend (2977 ms) 2024-09-16T03:45:53.683 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio: [ RUN ] LibRadosAio.RemoveTest 2024-09-16T03:45:53.683 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio: [ OK ] LibRadosAio.RemoveTest (2968 ms) 2024-09-16T03:45:53.683 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio: [ RUN ] LibRadosAio.XattrsRoundTrip 2024-09-16T03:45:53.683 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio: [ OK ] LibRadosAio.XattrsRoundTrip (18123 ms) 2024-09-16T03:45:53.683 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio: [ RUN ] LibRadosAio.RmXattr 2024-09-16T03:45:53.683 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio: [ OK ] LibRadosAio.RmXattr (17237 ms) 2024-09-16T03:45:53.683 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio: [ RUN ] LibRadosAio.XattrIter 2024-09-16T03:45:53.683 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio: [ OK ] LibRadosAio.XattrIter (2781 ms) 2024-09-16T03:45:53.683 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio: [ RUN ] LibRadosAio.IsComplete 2024-09-16T03:45:53.683 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio: [ OK ] LibRadosAio.IsComplete (2997 ms) 2024-09-16T03:45:53.684 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio: [ RUN ] LibRadosAio.IsSafe 2024-09-16T03:45:53.684 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio: [ OK ] LibRadosAio.IsSafe (3006 ms) 2024-09-16T03:45:53.684 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio: [ RUN ] LibRadosAio.ReturnValue 2024-09-16T03:45:53.684 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio: [ OK ] LibRadosAio.ReturnValue (2992 ms) 2024-09-16T03:45:53.684 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio: [ RUN ] LibRadosAio.Flush 2024-09-16T03:45:53.684 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio: [ OK ] LibRadosAio.Flush (3017 ms) 2024-09-16T03:45:53.684 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio: [ RUN ] LibRadosAio.FlushAsync 2024-09-16T03:45:53.684 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio: [ OK ] LibRadosAio.FlushAsync (3042 ms) 2024-09-16T03:45:53.684 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripWriteFull 2024-09-16T03:45:53.684 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio: [ OK ] LibRadosAio.RoundTripWriteFull (3182 ms) 2024-09-16T03:45:53.684 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripWriteSame 2024-09-16T03:45:53.684 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio: [ OK ] LibRadosAio.RoundTripWriteSame (3001 ms) 2024-09-16T03:45:53.684 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio: [ RUN ] LibRadosAio.SimpleStat 2024-09-16T03:45:53.684 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio: [ OK ] LibRadosAio.SimpleStat (2779 ms) 2024-09-16T03:45:53.685 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio: [ RUN ] LibRadosAio.OperateMtime 2024-09-16T03:45:53.685 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio: [ OK ] LibRadosAio.OperateMtime (3761 ms) 2024-09-16T03:45:53.685 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio: [ RUN ] LibRadosAio.Operate2Mtime 2024-09-16T03:45:53.685 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio: [ OK ] LibRadosAio.Operate2Mtime (17048 ms) 2024-09-16T03:45:53.685 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio: [ RUN ] LibRadosAio.SimpleStatNS 2024-09-16T03:45:53.685 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio: [ OK ] LibRadosAio.SimpleStatNS (3027 ms) 2024-09-16T03:45:53.685 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio: [ RUN ] LibRadosAio.StatRemove 2024-09-16T03:45:53.685 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio: [ OK ] LibRadosAio.StatRemove (2497 ms) 2024-09-16T03:45:53.685 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio: [ RUN ] LibRadosAio.ExecuteClass 2024-09-16T03:45:53.685 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio: [ OK ] LibRadosAio.ExecuteClass (2984 ms) 2024-09-16T03:45:53.685 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio: [ RUN ] LibRadosAio.MultiWrite 2024-09-16T03:45:53.685 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio: [ OK ] LibRadosAio.MultiWrite (3014 ms) 2024-09-16T03:45:53.685 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio: [ RUN ] LibRadosAio.AioUnlock 2024-09-16T03:45:53.686 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio: [ OK ] LibRadosAio.AioUnlock (2993 ms) 2024-09-16T03:45:53.686 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio: [----------] 26 tests from LibRadosAio (144120 ms total) 2024-09-16T03:45:53.686 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio: 2024-09-16T03:45:53.686 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio: [----------] 16 tests from LibRadosAioEC 2024-09-16T03:45:57.900 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-16T03:45:57.901 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-09-16T03:45:57.901 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-16T03:45:57.901 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-16T03:45:57.904 INFO:tasks.ceph.mon.b:Started 2024-09-16T03:45:57.904 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T03:45:57.904 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:45:58.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.028+0000 7f81cec3c640 10 monclient: get_monmap_and_config 2024-09-16T03:45:58.036 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.028+0000 7f81cec3c640 10 monclient: build_initial_monmap 2024-09-16T03:45:58.036 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.028+0000 7f81cec3c640 10 monclient: monmap: 2024-09-16T03:45:58.036 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:45:58.036 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:45:58.036 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:45:58.034630+0000 2024-09-16T03:45:58.037 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:45:58.034630+0000 2024-09-16T03:45:58.037 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:45:58.037 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:45:58.037 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:45:58.037 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:45:58.038 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:45:58.038 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:45:58.038 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:45:58.038 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:45:58.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.028+0000 7f81cec3c640 1 Processor -- start 2024-09-16T03:45:58.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.028+0000 7f81cec3c640 1 -- start start 2024-09-16T03:45:58.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.028+0000 7f81cec3c640 10 monclient: init 2024-09-16T03:45:58.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.032+0000 7f81cec3c640 10 monclient: _reopen_session rank -1 2024-09-16T03:45:58.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.032+0000 7f81cec3c640 10 monclient: _add_conns ranks=[1,3,0,2,4] 2024-09-16T03:45:58.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.032+0000 7f81cec3c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f81c80a39b0 addr v1:172.21.15.154:6789/0 2024-09-16T03:45:58.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.032+0000 7f81cec3c640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f81c80a3d90 0x7f81c813c380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:45:58.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.032+0000 7f81cec3c640 10 monclient(hunting): picked mon.noname-a con 0x7f81c80a3d90 addr v2:172.21.15.42:3300/0 2024-09-16T03:45:58.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.032+0000 7f81cec3c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f81c813c8c0 addr v1:172.21.15.42:6789/0 2024-09-16T03:45:58.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.032+0000 7f81cec3c640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f81c8134290 con 0x7f81c813c8c0 2024-09-16T03:45:58.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.032+0000 7f81cec3c640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f81c809c140 con 0x7f81c80a39b0 2024-09-16T03:45:58.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.032+0000 7f81cec3c640 10 monclient(hunting): start opening mon connection 2024-09-16T03:45:58.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.032+0000 7f81cec3c640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f81c809c420 con 0x7f81c80a3d90 2024-09-16T03:45:58.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.032+0000 7f81cec3c640 10 monclient(hunting): _renew_subs 2024-09-16T03:45:58.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.032+0000 7f81cec3c640 10 monclient(hunting): authenticate will time out at 1205.004490s 2024-09-16T03:45:58.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.032+0000 7f81c7fff640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f81c80a3d90 0x7f81c813c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:45:58.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.032+0000 7f81c7fff640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f81c80a3d90 0x7f81c813c380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:41326/0 (socket says 172.21.15.42:41326) 2024-09-16T03:45:58.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.032+0000 7f81c7fff640 1 -- 172.21.15.42:0/975340013 learned_addr learned my addr 172.21.15.42:0/975340013 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:45:58.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.032+0000 7f81cd1d9640 1 --1- 172.21.15.42:0/975340013 >> v1:172.21.15.42:6789/0 conn(0x7f81c813c8c0 0x7f81c813ecb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:56626/0 (socket says 172.21.15.42:56626) 2024-09-16T03:45:58.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.032+0000 7f81c7fff640 10 monclient(hunting): get_auth_request con 0x7f81c80a3d90 auth_method 0 2024-09-16T03:45:58.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.032+0000 7f81c7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:45:58.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.032+0000 7f81c7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:45:58.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.032+0000 7f81c7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:45:58.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.032+0000 7f81c7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:45:58.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.032+0000 7f81c7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:45:58.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.032+0000 7f81c7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:45:58.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.032+0000 7f81c7fff640 10 monclient(hunting): handle_auth_done global_id 744321 payload 274 2024-09-16T03:45:58.059 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.032+0000 7f81c7fff640 1 -- 172.21.15.42:0/975340013 >> v1:172.21.15.154:6789/0 conn(0x7f81c80a39b0 legacy=0x7f81c8133b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:45:58.059 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.032+0000 7f81c7fff640 1 -- 172.21.15.42:0/975340013 >> v1:172.21.15.42:6789/0 conn(0x7f81c813c8c0 legacy=0x7f81c813ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:45:58.059 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.032+0000 7f81c7fff640 10 monclient: _finish_hunting 0 2024-09-16T03:45:58.059 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.032+0000 7f81c7fff640 1 monclient: found mon.noname-a 2024-09-16T03:45:58.059 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.032+0000 7f81c7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:45:58.059 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.032+0000 7f81c7fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:45:58.060 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.032+0000 7f81c7fff640 1 -- 172.21.15.42:0/975340013 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81c806dd60 con 0x7f81c80a3d90 2024-09-16T03:45:58.060 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.032+0000 7f81c7fff640 1 --2- 172.21.15.42:0/975340013 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f81c80a3d90 0x7f81c813c380 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f81bc009870 tx=0x7f81bc01c420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9cc85e0fda121be server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:45:58.060 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.036+0000 7f81c77fe640 1 -- 172.21.15.42:0/975340013 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81bc0b0070 con 0x7f81c80a3d90 2024-09-16T03:45:58.060 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.036+0000 7f81c77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:45:58.061 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.036+0000 7f81c77fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:45:58.061 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.036+0000 7f81c77fe640 10 monclient: dump: 2024-09-16T03:45:58.061 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:45:58.061 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:45:58.061 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:58.061 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:58.061 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:45:58.062 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:45:58.062 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:45:58.062 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:45:58.062 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:45:58.062 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:45:58.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.036+0000 7f81c77fe640 10 monclient: _finish_auth 0 2024-09-16T03:45:58.063 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.036+0000 7f81c77fe640 10 monclient: _check_auth_tickets 2024-09-16T03:45:58.063 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.036+0000 7f81c77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:45:58.063 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.036+0000 7f81cec3c640 5 monclient: authenticate success, global_id 744321 2024-09-16T03:45:58.063 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.036+0000 7f81c77fe640 1 -- 172.21.15.42:0/975340013 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f81bc0ab600 con 0x7f81c80a3d90 2024-09-16T03:45:58.063 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.036+0000 7f81c77fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:45:58.063 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.036+0000 7f81cec3c640 10 monclient: get_monmap_and_config success 2024-09-16T03:45:58.063 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.036+0000 7f81c77fe640 1 -- 172.21.15.42:0/975340013 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81bc0ab9d0 con 0x7f81c80a3d90 2024-09-16T03:45:58.064 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.036+0000 7f81c77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:45:58.064 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.036+0000 7f81cec3c640 4 set_mon_vals no callback set 2024-09-16T03:45:58.064 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.036+0000 7f81c77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:45:58.064 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.036+0000 7f81c77fe640 10 monclient: dump: 2024-09-16T03:45:58.064 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:45:58.064 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:45:58.065 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:58.065 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:58.065 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:45:58.065 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:45:58.065 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:45:58.065 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:45:58.066 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:45:58.066 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:45:58.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.036+0000 7f81cec3c640 10 monclient: shutdown 2024-09-16T03:45:58.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.036+0000 7f81cec3c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:45:58.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.036+0000 7f81cec3c640 1 -- 172.21.15.42:0/975340013 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f81c80a3d90 msgr2=0x7f81c813c380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:45:58.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.036+0000 7f81cec3c640 1 --2- 172.21.15.42:0/975340013 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f81c80a3d90 0x7f81c813c380 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f81bc009870 tx=0x7f81bc01c420 comp rx=0 tx=0).stop 2024-09-16T03:45:58.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.036+0000 7f81cec3c640 1 -- 172.21.15.42:0/975340013 shutdown_connections 2024-09-16T03:45:58.067 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.036+0000 7f81cec3c640 1 --2- 172.21.15.42:0/975340013 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f81c80a3d90 0x7f81c813c380 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:45:58.067 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.036+0000 7f81cec3c640 1 -- 172.21.15.42:0/975340013 >> 172.21.15.42:0/975340013 conn(0x7f81c80a9fb0 msgr2=0x7f81c80aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:45:58.067 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.036+0000 7f81cec3c640 1 -- 172.21.15.42:0/975340013 shutdown_connections 2024-09-16T03:45:58.067 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.036+0000 7f81cec3c640 1 -- 172.21.15.42:0/975340013 wait complete. 2024-09-16T03:45:58.067 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.036+0000 7f81cec3c640 10 monclient: build_initial_monmap 2024-09-16T03:45:58.067 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.036+0000 7f81cec3c640 10 monclient: monmap: 2024-09-16T03:45:58.067 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:45:58.068 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:45:58.068 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:45:58.068 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:45:58.068 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:45:58.068 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:45:58.068 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:45:58.068 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:45:58.068 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:45:58.068 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:45:58.068 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.036+0000 7f81cec3c640 1 librados: starting msgr at 2024-09-16T03:45:58.068 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.036+0000 7f81cec3c640 1 librados: starting objecter 2024-09-16T03:45:58.068 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.036+0000 7f81cec3c640 1 Processor -- start 2024-09-16T03:45:58.068 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.036+0000 7f81cec3c640 1 -- start start 2024-09-16T03:45:58.069 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.036+0000 7f81cec3c640 1 librados: setting wanted keys 2024-09-16T03:45:58.069 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.036+0000 7f81cec3c640 1 librados: calling monclient init 2024-09-16T03:45:58.069 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.036+0000 7f81cec3c640 10 monclient: init 2024-09-16T03:45:58.069 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.036+0000 7f81cec3c640 10 monclient: _reopen_session rank -1 2024-09-16T03:45:58.069 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.036+0000 7f81cec3c640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:45:58.069 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.036+0000 7f81cec3c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f81c80a39b0 0x7f81c8130ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:45:58.069 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.036+0000 7f81cec3c640 10 monclient(hunting): picked mon.noname-a con 0x7f81c80a39b0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:45:58.069 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.036+0000 7f81cec3c640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f81c80a3d90 0x7f81c8130ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:45:58.069 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.036+0000 7f81cec3c640 10 monclient(hunting): picked mon.noname-c con 0x7f81c80a3d90 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:45:58.069 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.036+0000 7f81cec3c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f81c813c8c0 0x7f81c8139620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:45:58.069 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.036+0000 7f81cec3c640 10 monclient(hunting): picked mon.noname-b con 0x7f81c813c8c0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:45:58.069 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.036+0000 7f81cec3c640 10 monclient(hunting): start opening mon connection 2024-09-16T03:45:58.069 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.036+0000 7f81cec3c640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f81c8142f80 con 0x7f81c80a39b0 2024-09-16T03:45:58.070 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.036+0000 7f81cec3c640 10 monclient(hunting): start opening mon connection 2024-09-16T03:45:58.070 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.036+0000 7f81cec3c640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f81c8142e00 con 0x7f81c813c8c0 2024-09-16T03:45:58.070 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.036+0000 7f81cec3c640 10 monclient(hunting): start opening mon connection 2024-09-16T03:45:58.070 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.036+0000 7f81cec3c640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f81c8143100 con 0x7f81c80a3d90 2024-09-16T03:45:58.070 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.036+0000 7f81cec3c640 10 monclient(hunting): _renew_subs 2024-09-16T03:45:58.070 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.036+0000 7f81cec3c640 10 monclient(hunting): authenticate will time out at 1205.010462s 2024-09-16T03:45:58.070 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.036+0000 7f81c7fff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f81c80a3d90 0x7f81c8130ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:45:58.070 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.036+0000 7f81cc9d8640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f81c80a39b0 0x7f81c8130ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:45:58.070 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.036+0000 7f81c7fff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f81c80a3d90 0x7f81c8130ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:53228/0 (socket says 172.21.15.42:53228) 2024-09-16T03:45:58.070 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.036+0000 7f81c7fff640 1 -- 172.21.15.42:0/2708754816 learned_addr learned my addr 172.21.15.42:0/2708754816 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:45:58.070 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.036+0000 7f81c7fff640 10 monclient(hunting): get_auth_request con 0x7f81c80a3d90 auth_method 0 2024-09-16T03:45:58.070 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.036+0000 7f81c7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:45:58.070 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.036+0000 7f81c7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:45:58.071 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.036+0000 7f81c7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:45:58.071 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.036+0000 7f81cc9d8640 10 monclient(hunting): get_auth_request con 0x7f81c80a39b0 auth_method 0 2024-09-16T03:45:58.071 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.036+0000 7f81cc9d8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:45:58.071 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.036+0000 7f81cc9d8640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:45:58.071 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.036+0000 7f81cc9d8640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:45:58.071 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.036+0000 7f81c7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:45:58.071 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.036+0000 7f81c7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:45:58.071 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.036+0000 7f81c7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:45:58.071 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.036+0000 7f81cc9d8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:45:58.071 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.036+0000 7f81cc9d8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:45:58.071 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.036+0000 7f81cc9d8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:45:58.071 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.036+0000 7f81cd1d9640 1 --2- 172.21.15.42:0/2708754816 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f81c813c8c0 0x7f81c8139620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:45:58.072 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.036+0000 7f81cd1d9640 10 monclient(hunting): get_auth_request con 0x7f81c813c8c0 auth_method 0 2024-09-16T03:45:58.072 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.036+0000 7f81cd1d9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:45:58.072 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.036+0000 7f81cd1d9640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:45:58.072 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.036+0000 7f81cd1d9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:45:58.072 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.036+0000 7f81c7fff640 10 monclient(hunting): handle_auth_done global_id 13487 payload 824 2024-09-16T03:45:58.072 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.036+0000 7f81c7fff640 1 -- 172.21.15.42:0/2708754816 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f81c813c8c0 msgr2=0x7f81c8139620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:45:58.072 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.036+0000 7f81c7fff640 1 --2- 172.21.15.42:0/2708754816 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f81c813c8c0 0x7f81c8139620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:45:58.072 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.036+0000 7f81c7fff640 1 -- 172.21.15.42:0/2708754816 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f81c80a39b0 msgr2=0x7f81c8130ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:45:58.072 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.036+0000 7f81c7fff640 1 --2- 172.21.15.42:0/2708754816 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f81c80a39b0 0x7f81c8130ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:45:58.072 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.036+0000 7f81c7fff640 10 monclient: _finish_hunting 0 2024-09-16T03:45:58.072 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.036+0000 7f81c7fff640 1 monclient: found mon.noname-c 2024-09-16T03:45:58.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.036+0000 7f81c7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:45:58.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.036+0000 7f81c7fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:45:58.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.036+0000 7f81c7fff640 1 -- 172.21.15.42:0/2708754816 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81c813fcd0 con 0x7f81c80a3d90 2024-09-16T03:45:58.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.036+0000 7f81c7fff640 1 --2- 172.21.15.42:0/2708754816 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f81c80a3d90 0x7f81c8130ff0 secure :-1 s=READY pgs=2406 cs=0 l=1 rev1=1 crypto rx=0x7f81bc0ae040 tx=0x7f81bc0191c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:45:58.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.036+0000 7f81c57fa640 10 client.?.objecter ms_handle_connect 0x7f81c80a3d90 2024-09-16T03:45:58.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.036+0000 7f81c57fa640 10 client.?.objecter resend_mon_ops 2024-09-16T03:45:58.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.036+0000 7f81c57fa640 1 -- 172.21.15.42:0/2708754816 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81bc0b0040 con 0x7f81c80a3d90 2024-09-16T03:45:58.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.036+0000 7f81c57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:45:58.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.036+0000 7f81c57fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:45:58.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.036+0000 7f81c57fa640 10 monclient: dump: 2024-09-16T03:45:58.074 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:45:58.075 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:45:58.075 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:58.075 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:58.075 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:45:58.075 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:45:58.075 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:45:58.075 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:45:58.075 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:45:58.075 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:45:58.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.036+0000 7f81c57fa640 10 monclient: _finish_auth 0 2024-09-16T03:45:58.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.036+0000 7f81c57fa640 10 monclient: _check_auth_tickets 2024-09-16T03:45:58.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.036+0000 7f81c57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:45:58.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.036+0000 7f81cec3c640 5 monclient: authenticate success, global_id 13487 2024-09-16T03:45:58.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.036+0000 7f81c57fa640 1 -- 172.21.15.42:0/2708754816 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f81bc0b4440 con 0x7f81c80a3d90 2024-09-16T03:45:58.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.036+0000 7f81cec3c640 10 monclient: _renew_subs 2024-09-16T03:45:58.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.036+0000 7f81cec3c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:45:58.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.036+0000 7f81cec3c640 1 -- 172.21.15.42:0/2708754816 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81c812e570 con 0x7f81c80a3d90 2024-09-16T03:45:58.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.036+0000 7f81c57fa640 10 monclient: handle_config config(0 keys) 2024-09-16T03:45:58.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.036+0000 7f81c57fa640 1 -- 172.21.15.42:0/2708754816 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81bc0b4850 con 0x7f81c80a3d90 2024-09-16T03:45:58.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.036+0000 7f81c57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:45:58.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.036+0000 7f81c57fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:45:58.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.036+0000 7f81c57fa640 10 monclient: dump: 2024-09-16T03:45:58.076 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:45:58.076 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:45:58.076 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:58.077 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:45:58.077 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:45:58.077 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:45:58.077 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:45:58.077 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:45:58.077 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:45:58.077 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:45:58.077 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.036+0000 7f81c67fc640 4 set_mon_vals no callback set 2024-09-16T03:45:58.077 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.036+0000 7f81cec3c640 10 client.13487.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:45:58.077 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.036+0000 7f81cec3c640 10 monclient: _renew_subs 2024-09-16T03:45:58.077 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.036+0000 7f81cec3c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:45:58.077 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.036+0000 7f81cec3c640 1 -- 172.21.15.42:0/2708754816 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f81c8140120 con 0x7f81c80a3d90 2024-09-16T03:45:58.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.036+0000 7f81cec3c640 1 librados: init done 2024-09-16T03:45:58.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.040+0000 7f81cec3c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:45:58.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.040+0000 7f81cec3c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:45:58.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.040+0000 7f81cec3c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:45:58.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.040+0000 7f81cec3c640 1 -- 172.21.15.42:0/2708754816 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f81c813b060 con 0x7f81c80a3d90 2024-09-16T03:45:58.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.040+0000 7f81c57fa640 1 -- 172.21.15.42:0/2708754816 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f81bc0bb020 con 0x7f81c80a3d90 2024-09-16T03:45:58.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.040+0000 7f81c57fa640 1 --2- 172.21.15.42:0/2708754816 >> v2:172.21.15.154:6812/254845342 conn(0x7f81b4037810 0x7f81b4039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:45:58.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.040+0000 7f81c57fa640 1 -- 172.21.15.42:0/2708754816 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(112..112 src has 1..112) ==== 8143+0+0 (secure 0 0 0) 0x7f81bc0e2fe0 con 0x7f81c80a3d90 2024-09-16T03:45:58.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.040+0000 7f81c57fa640 10 client.13487.objecter ms_dispatch 0x7f81c8000ba0 osd_map(112..112 src has 1..112) 2024-09-16T03:45:58.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.040+0000 7f81c57fa640 3 client.13487.objecter handle_osd_map got epochs [112,112] > 0 2024-09-16T03:45:58.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.040+0000 7f81c57fa640 3 client.13487.objecter handle_osd_map decoding full epoch 112 2024-09-16T03:45:58.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.040+0000 7f81c57fa640 20 client.13487.objecter dump_active .. 0 homeless 2024-09-16T03:45:58.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.040+0000 7f81cc9d8640 1 --2- 172.21.15.42:0/2708754816 >> v2:172.21.15.154:6812/254845342 conn(0x7f81b4037810 0x7f81b4039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:45:58.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.040+0000 7f81cc9d8640 10 monclient: get_auth_request con 0x7f81b4037810 auth_method 0 2024-09-16T03:45:58.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.040+0000 7f81cc9d8640 1 --2- 172.21.15.42:0/2708754816 >> v2:172.21.15.154:6812/254845342 conn(0x7f81b4037810 0x7f81b4039cd0 secure :-1 s=READY pgs=3390 cs=0 l=1 rev1=1 crypto rx=0x7f81b80012f0 tx=0x7f81b8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:45:58.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.040+0000 7f81c57fa640 10 client.13487.objecter ms_handle_connect 0x7f81b4037810 2024-09-16T03:45:58.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.044+0000 7f81c57fa640 1 -- 172.21.15.42:0/2708754816 <== mon.2 v2:172.21.15.42:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f81bc0c2020 con 0x7f81c80a3d90 2024-09-16T03:45:58.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.044+0000 7f81c57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:45:58.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.044+0000 7f81c57fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:45:58.276 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.268+0000 7f81cec3c640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:45:58.276 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.268+0000 7f81cec3c640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:45:58.276 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.268+0000 7f81cec3c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:45:58.276 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.268+0000 7f81cec3c640 1 -- 172.21.15.42:0/2708754816 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f81c8133b80 con 0x7f81c80a3d90 2024-09-16T03:45:58.277 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":32,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":21,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:45:58.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.272+0000 7f81c57fa640 1 -- 172.21.15.42:0/2708754816 <== mon.2 v2:172.21.15.42:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1555 (secure 0 0 0) 0x7f81c813fcd0 con 0x7f81c80a3d90 2024-09-16T03:45:58.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.272+0000 7f81c57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:45:58.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.272+0000 7f81c57fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:45:58.279 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.272+0000 7f81cec3c640 10 librados: watch_flush enter 2024-09-16T03:45:58.279 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.272+0000 7f81cec3c640 10 librados: watch_flush exit 2024-09-16T03:45:58.279 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.272+0000 7f81cec3c640 20 client.13487.objecter shutdown clearing up homeless session... 2024-09-16T03:45:58.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.272+0000 7f81cec3c640 10 client.13487.objecter successfully canceled tick 2024-09-16T03:45:58.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.272+0000 7f81cec3c640 1 -- 172.21.15.42:0/2708754816 >> v2:172.21.15.154:6812/254845342 conn(0x7f81b4037810 msgr2=0x7f81b4039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:45:58.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.272+0000 7f81cec3c640 1 --2- 172.21.15.42:0/2708754816 >> v2:172.21.15.154:6812/254845342 conn(0x7f81b4037810 0x7f81b4039cd0 secure :-1 s=READY pgs=3390 cs=0 l=1 rev1=1 crypto rx=0x7f81b80012f0 tx=0x7f81b8009040 comp rx=0 tx=0).stop 2024-09-16T03:45:58.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.272+0000 7f81cec3c640 10 monclient: shutdown 2024-09-16T03:45:58.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.272+0000 7f81cec3c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:45:58.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.272+0000 7f81cec3c640 1 -- 172.21.15.42:0/2708754816 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f81c80a3d90 msgr2=0x7f81c8130ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:45:58.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.272+0000 7f81cec3c640 1 --2- 172.21.15.42:0/2708754816 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f81c80a3d90 0x7f81c8130ff0 secure :-1 s=READY pgs=2406 cs=0 l=1 rev1=1 crypto rx=0x7f81bc0ae040 tx=0x7f81bc0191c0 comp rx=0 tx=0).stop 2024-09-16T03:45:58.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.272+0000 7f81cec3c640 1 -- 172.21.15.42:0/2708754816 shutdown_connections 2024-09-16T03:45:58.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.272+0000 7f81cec3c640 1 --2- 172.21.15.42:0/2708754816 >> v2:172.21.15.154:6812/254845342 conn(0x7f81b4037810 0x7f81b4039cd0 unknown :-1 s=CLOSED pgs=3390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:45:58.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.272+0000 7f81cec3c640 1 --2- 172.21.15.42:0/2708754816 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f81c813c8c0 0x7f81c8139620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:45:58.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.272+0000 7f81cec3c640 1 --2- 172.21.15.42:0/2708754816 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f81c80a3d90 0x7f81c8130ff0 unknown :-1 s=CLOSED pgs=2406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:45:58.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.272+0000 7f81cec3c640 1 --2- 172.21.15.42:0/2708754816 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f81c80a39b0 0x7f81c8130ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:45:58.281 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.272+0000 7f81cec3c640 1 -- 172.21.15.42:0/2708754816 >> 172.21.15.42:0/2708754816 conn(0x7f81c80a9fb0 msgr2=0x7f81c813ecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:45:58.281 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.272+0000 7f81cec3c640 1 -- 172.21.15.42:0/2708754816 shutdown_connections 2024-09-16T03:45:58.281 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.272+0000 7f81cec3c640 1 -- 172.21.15.42:0/2708754816 wait complete. 2024-09-16T03:45:58.281 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:45:58.272+0000 7f81cec3c640 1 librados: shutdown 2024-09-16T03:46:01.297 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:46:01.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.424+0000 7f1dbd379640 10 monclient: get_monmap_and_config 2024-09-16T03:46:01.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.424+0000 7f1dbd379640 10 monclient: build_initial_monmap 2024-09-16T03:46:01.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.424+0000 7f1dbd379640 10 monclient: monmap: 2024-09-16T03:46:01.430 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:46:01.431 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:46:01.431 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:46:01.429582+0000 2024-09-16T03:46:01.431 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:46:01.429582+0000 2024-09-16T03:46:01.431 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:46:01.431 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:46:01.431 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:46:01.431 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:46:01.431 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:46:01.431 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:46:01.431 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:46:01.431 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:46:01.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.424+0000 7f1dbd379640 1 Processor -- start 2024-09-16T03:46:01.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.424+0000 7f1dbd379640 1 -- start start 2024-09-16T03:46:01.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.424+0000 7f1dbd379640 10 monclient: init 2024-09-16T03:46:01.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.424+0000 7f1dbd379640 10 monclient: _reopen_session rank -1 2024-09-16T03:46:01.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.424+0000 7f1dbd379640 10 monclient: _add_conns ranks=[0,3,2,4,1] 2024-09-16T03:46:01.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.424+0000 7f1dbd379640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1db80a1330 addr v1:172.21.15.42:6789/0 2024-09-16T03:46:01.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.424+0000 7f1dbd379640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f1db80a1710 0x7f1db80a1b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:46:01.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.424+0000 7f1dbd379640 10 monclient(hunting): picked mon.noname-a con 0x7f1db80a1710 addr v2:172.21.15.42:3300/0 2024-09-16T03:46:01.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.424+0000 7f1dbd379640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1db81393a0 0x7f1db813b790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:46:01.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.424+0000 7f1dbd379640 10 monclient(hunting): picked mon.noname-b con 0x7f1db81393a0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:46:01.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.424+0000 7f1dbd379640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1db809e4c0 con 0x7f1db80a1330 2024-09-16T03:46:01.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.424+0000 7f1dbd379640 10 monclient(hunting): start opening mon connection 2024-09-16T03:46:01.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.424+0000 7f1dbd379640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f1db809edf0 con 0x7f1db80a1710 2024-09-16T03:46:01.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.424+0000 7f1dbd379640 10 monclient(hunting): start opening mon connection 2024-09-16T03:46:01.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.424+0000 7f1dbd379640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f1db809cd60 con 0x7f1db81393a0 2024-09-16T03:46:01.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.424+0000 7f1dbd379640 10 monclient(hunting): _renew_subs 2024-09-16T03:46:01.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.424+0000 7f1dbd379640 10 monclient(hunting): authenticate will time out at 1208.399605s 2024-09-16T03:46:01.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.424+0000 7f1db77fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1db81393a0 0x7f1db813b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:46:01.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.424+0000 7f1db77fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1db81393a0 0x7f1db813b790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:53232/0 (socket says 172.21.15.42:53232) 2024-09-16T03:46:01.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.424+0000 7f1db77fe640 1 -- 172.21.15.42:0/4207224108 learned_addr learned my addr 172.21.15.42:0/4207224108 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:46:01.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.424+0000 7f1db77fe640 10 monclient(hunting): get_auth_request con 0x7f1db81393a0 auth_method 0 2024-09-16T03:46:01.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.424+0000 7f1db77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:46:01.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.424+0000 7f1db77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:46:01.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.424+0000 7f1db77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:46:01.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.424+0000 7f1daffff640 1 --2- 172.21.15.42:0/4207224108 >> v2:172.21.15.42:3300/0 conn(0x7f1db80a1710 0x7f1db80a1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:46:01.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.424+0000 7f1daffff640 10 monclient(hunting): get_auth_request con 0x7f1db80a1710 auth_method 0 2024-09-16T03:46:01.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.424+0000 7f1daffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:46:01.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.424+0000 7f1daffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:46:01.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.424+0000 7f1daffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:46:01.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.424+0000 7f1db77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:46:01.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.424+0000 7f1db77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:46:01.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.428+0000 7f1db77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:46:01.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.428+0000 7f1daffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:46:01.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.428+0000 7f1daffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:46:01.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.428+0000 7f1daffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:46:01.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.428+0000 7f1db77fe640 10 monclient(hunting): handle_auth_done global_id 13496 payload 274 2024-09-16T03:46:01.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.428+0000 7f1db77fe640 1 -- 172.21.15.42:0/4207224108 >> v2:172.21.15.42:3300/0 conn(0x7f1db80a1710 msgr2=0x7f1db80a1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:46:01.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.428+0000 7f1db77fe640 1 --2- 172.21.15.42:0/4207224108 >> v2:172.21.15.42:3300/0 conn(0x7f1db80a1710 0x7f1db80a1b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:46:01.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.428+0000 7f1db77fe640 1 -- 172.21.15.42:0/4207224108 >> v1:172.21.15.42:6789/0 conn(0x7f1db80a1330 legacy=0x7f1db809f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:46:01.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.428+0000 7f1db77fe640 10 monclient: _finish_hunting 0 2024-09-16T03:46:01.437 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.428+0000 7f1db77fe640 1 monclient: found mon.noname-b 2024-09-16T03:46:01.437 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.428+0000 7f1db77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:46:01.437 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.428+0000 7f1db77fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T03:46:01.437 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.428+0000 7f1db77fe640 1 -- 172.21.15.42:0/4207224108 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1db80a2080 con 0x7f1db81393a0 2024-09-16T03:46:01.445 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.428+0000 7f1db77fe640 1 --2- 172.21.15.42:0/4207224108 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1db81393a0 0x7f1db813b790 secure :-1 s=READY pgs=2407 cs=0 l=1 rev1=1 crypto rx=0x7f1db000d150 tx=0x7f1db001eaf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=df74ae774d050abe server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:46:01.445 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.428+0000 7f1db67fc640 1 -- 172.21.15.42:0/4207224108 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1db000b2d0 con 0x7f1db81393a0 2024-09-16T03:46:01.445 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.428+0000 7f1db67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:46:01.445 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.428+0000 7f1db67fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:46:01.445 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.428+0000 7f1db67fc640 10 monclient: dump: 2024-09-16T03:46:01.445 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:46:01.445 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:46:01.446 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:01.446 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:01.446 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:46:01.446 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:46:01.446 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:46:01.446 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:46:01.446 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:46:01.446 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:46:01.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.428+0000 7f1db67fc640 10 monclient: _finish_auth 0 2024-09-16T03:46:01.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.428+0000 7f1db67fc640 10 monclient: _check_auth_tickets 2024-09-16T03:46:01.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.428+0000 7f1db67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:46:01.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.428+0000 7f1dbd379640 5 monclient: authenticate success, global_id 13496 2024-09-16T03:46:01.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.428+0000 7f1dbd379640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:46:01.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.428+0000 7f1db67fc640 1 -- 172.21.15.42:0/4207224108 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1db000b470 con 0x7f1db81393a0 2024-09-16T03:46:01.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.428+0000 7f1db67fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:46:01.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.428+0000 7f1dbd379640 10 monclient: get_monmap_and_config success 2024-09-16T03:46:01.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.428+0000 7f1dbd379640 4 set_mon_vals no callback set 2024-09-16T03:46:01.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.428+0000 7f1db67fc640 1 -- 172.21.15.42:0/4207224108 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1db00a79f0 con 0x7f1db81393a0 2024-09-16T03:46:01.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.428+0000 7f1db67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:46:01.448 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.428+0000 7f1db67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:46:01.448 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.428+0000 7f1db67fc640 10 monclient: dump: 2024-09-16T03:46:01.448 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:46:01.448 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:46:01.448 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:01.448 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:01.448 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:46:01.448 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:46:01.448 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:46:01.448 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:46:01.448 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:46:01.448 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:46:01.448 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.428+0000 7f1dbd379640 10 monclient: shutdown 2024-09-16T03:46:01.449 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.428+0000 7f1dbd379640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:46:01.449 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.428+0000 7f1dbd379640 1 -- 172.21.15.42:0/4207224108 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1db81393a0 msgr2=0x7f1db813b790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:46:01.449 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.428+0000 7f1dbd379640 1 --2- 172.21.15.42:0/4207224108 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1db81393a0 0x7f1db813b790 secure :-1 s=READY pgs=2407 cs=0 l=1 rev1=1 crypto rx=0x7f1db000d150 tx=0x7f1db001eaf0 comp rx=0 tx=0).stop 2024-09-16T03:46:01.449 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.428+0000 7f1dbd379640 1 -- 172.21.15.42:0/4207224108 shutdown_connections 2024-09-16T03:46:01.449 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.428+0000 7f1dbd379640 1 --2- 172.21.15.42:0/4207224108 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1db81393a0 0x7f1db813b790 unknown :-1 s=CLOSED pgs=2407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:46:01.449 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.428+0000 7f1dbd379640 1 --2- 172.21.15.42:0/4207224108 >> v2:172.21.15.42:3300/0 conn(0x7f1db80a1710 0x7f1db80a1b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:46:01.449 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.428+0000 7f1dbd379640 1 -- 172.21.15.42:0/4207224108 >> 172.21.15.42:0/4207224108 conn(0x7f1db80a2420 msgr2=0x7f1db80a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:46:01.449 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.428+0000 7f1dbd379640 1 -- 172.21.15.42:0/4207224108 shutdown_connections 2024-09-16T03:46:01.449 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.428+0000 7f1dbd379640 1 -- 172.21.15.42:0/4207224108 wait complete. 2024-09-16T03:46:01.449 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.428+0000 7f1dbd379640 10 monclient: build_initial_monmap 2024-09-16T03:46:01.449 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.428+0000 7f1dbd379640 10 monclient: monmap: 2024-09-16T03:46:01.449 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:46:01.450 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:46:01.450 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:46:01.450 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:46:01.450 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:46:01.450 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:46:01.450 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:46:01.450 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:46:01.450 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:46:01.450 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:46:01.450 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.432+0000 7f1dbd379640 1 librados: starting msgr at 2024-09-16T03:46:01.450 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.432+0000 7f1dbd379640 1 librados: starting objecter 2024-09-16T03:46:01.450 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.432+0000 7f1dbd379640 1 Processor -- start 2024-09-16T03:46:01.450 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.432+0000 7f1dbd379640 1 -- start start 2024-09-16T03:46:01.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.432+0000 7f1dbd379640 1 librados: setting wanted keys 2024-09-16T03:46:01.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.432+0000 7f1dbd379640 1 librados: calling monclient init 2024-09-16T03:46:01.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.432+0000 7f1dbd379640 10 monclient: init 2024-09-16T03:46:01.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.432+0000 7f1dbd379640 10 monclient: _reopen_session rank -1 2024-09-16T03:46:01.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.432+0000 7f1dbd379640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:46:01.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.432+0000 7f1dbd379640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1db80a1330 0x7f1db8071580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:46:01.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.432+0000 7f1dbd379640 10 monclient(hunting): picked mon.noname-a con 0x7f1db80a1330 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:46:01.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.432+0000 7f1dbd379640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1db80a1710 0x7f1db8134570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:46:01.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.432+0000 7f1dbd379640 10 monclient(hunting): picked mon.noname-c con 0x7f1db80a1710 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:46:01.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.432+0000 7f1dbd379640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1db81393a0 0x7f1db8134ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:46:01.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.432+0000 7f1dbd379640 10 monclient(hunting): picked mon.noname-b con 0x7f1db81393a0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:46:01.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.432+0000 7f1dbd379640 10 monclient(hunting): start opening mon connection 2024-09-16T03:46:01.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.432+0000 7f1dbd379640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f1db809c630 con 0x7f1db80a1330 2024-09-16T03:46:01.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.432+0000 7f1dbd379640 10 monclient(hunting): start opening mon connection 2024-09-16T03:46:01.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.432+0000 7f1dbd379640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f1db809c4b0 con 0x7f1db81393a0 2024-09-16T03:46:01.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.432+0000 7f1dbd379640 10 monclient(hunting): start opening mon connection 2024-09-16T03:46:01.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.432+0000 7f1dbd379640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f1db809c7b0 con 0x7f1db80a1710 2024-09-16T03:46:01.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.432+0000 7f1dbd379640 10 monclient(hunting): _renew_subs 2024-09-16T03:46:01.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.432+0000 7f1dbd379640 10 monclient(hunting): authenticate will time out at 1208.404802s 2024-09-16T03:46:01.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.432+0000 7f1daffff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1db80a1710 0x7f1db8134570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:46:01.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.432+0000 7f1db6ffd640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1db80a1330 0x7f1db8071580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:46:01.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.432+0000 7f1daffff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1db80a1710 0x7f1db8134570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:53244/0 (socket says 172.21.15.42:53244) 2024-09-16T03:46:01.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.432+0000 7f1daffff640 1 -- 172.21.15.42:0/1298771068 learned_addr learned my addr 172.21.15.42:0/1298771068 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:46:01.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.432+0000 7f1db6ffd640 10 monclient(hunting): get_auth_request con 0x7f1db80a1330 auth_method 0 2024-09-16T03:46:01.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.432+0000 7f1db6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:46:01.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.432+0000 7f1db6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:46:01.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.432+0000 7f1db6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:46:01.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.432+0000 7f1daffff640 10 monclient(hunting): get_auth_request con 0x7f1db80a1710 auth_method 0 2024-09-16T03:46:01.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.432+0000 7f1daffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:46:01.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.432+0000 7f1daffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:46:01.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.432+0000 7f1daffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:46:01.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.432+0000 7f1db6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:46:01.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.432+0000 7f1db6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:46:01.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.432+0000 7f1db6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:46:01.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.432+0000 7f1daffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:46:01.454 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.432+0000 7f1daffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:46:01.454 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.432+0000 7f1daffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:46:01.454 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.432+0000 7f1db6ffd640 10 monclient(hunting): handle_auth_done global_id 744351 payload 824 2024-09-16T03:46:01.454 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.432+0000 7f1db6ffd640 1 -- 172.21.15.42:0/1298771068 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1db80a1710 msgr2=0x7f1db8134570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:46:01.454 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.432+0000 7f1db6ffd640 1 --2- 172.21.15.42:0/1298771068 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1db80a1710 0x7f1db8134570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:46:01.454 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.432+0000 7f1db77fe640 1 --2- 172.21.15.42:0/1298771068 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1db81393a0 0x7f1db8134ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:46:01.454 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.432+0000 7f1db6ffd640 1 -- 172.21.15.42:0/1298771068 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1db81393a0 msgr2=0x7f1db8134ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:46:01.455 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.432+0000 7f1db6ffd640 1 --2- 172.21.15.42:0/1298771068 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1db81393a0 0x7f1db8134ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:46:01.455 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.432+0000 7f1db6ffd640 10 monclient: _finish_hunting 0 2024-09-16T03:46:01.455 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.432+0000 7f1db6ffd640 1 monclient: found mon.noname-a 2024-09-16T03:46:01.455 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.432+0000 7f1db6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:46:01.455 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.432+0000 7f1db6ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:46:01.455 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.432+0000 7f1db6ffd640 1 -- 172.21.15.42:0/1298771068 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1db8120840 con 0x7f1db80a1330 2024-09-16T03:46:01.455 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.432+0000 7f1db6ffd640 1 --2- 172.21.15.42:0/1298771068 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1db80a1330 0x7f1db8071580 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f1da800c3c0 tx=0x7f1da800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:46:01.455 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.432+0000 7f1daf7fe640 10 client.?.objecter ms_handle_connect 0x7f1db80a1330 2024-09-16T03:46:01.455 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.432+0000 7f1daf7fe640 10 client.?.objecter resend_mon_ops 2024-09-16T03:46:01.455 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.432+0000 7f1daf7fe640 1 -- 172.21.15.42:0/1298771068 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1da80052d0 con 0x7f1db80a1330 2024-09-16T03:46:01.455 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.432+0000 7f1daf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:46:01.455 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.432+0000 7f1daf7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:46:01.455 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.432+0000 7f1daf7fe640 10 monclient: dump: 2024-09-16T03:46:01.456 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:46:01.456 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:46:01.456 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:01.456 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:01.456 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:46:01.456 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:46:01.456 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:46:01.456 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:46:01.456 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:46:01.456 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:46:01.456 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.432+0000 7f1daf7fe640 10 monclient: _finish_auth 0 2024-09-16T03:46:01.456 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.432+0000 7f1daf7fe640 10 monclient: _check_auth_tickets 2024-09-16T03:46:01.456 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.432+0000 7f1daf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:46:01.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.432+0000 7f1daf7fe640 1 -- 172.21.15.42:0/1298771068 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1da8005470 con 0x7f1db80a1330 2024-09-16T03:46:01.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.432+0000 7f1dbd379640 5 monclient: authenticate success, global_id 744351 2024-09-16T03:46:01.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.432+0000 7f1dbd379640 10 monclient: _renew_subs 2024-09-16T03:46:01.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.432+0000 7f1dbd379640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:46:01.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.432+0000 7f1dbd379640 1 -- 172.21.15.42:0/1298771068 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1db8122f40 con 0x7f1db80a1330 2024-09-16T03:46:01.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.432+0000 7f1daf7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:46:01.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.432+0000 7f1daf7fe640 1 -- 172.21.15.42:0/1298771068 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1da8007260 con 0x7f1db80a1330 2024-09-16T03:46:01.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.432+0000 7f1daf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:46:01.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.432+0000 7f1daf7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:46:01.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.432+0000 7f1daf7fe640 10 monclient: dump: 2024-09-16T03:46:01.457 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:46:01.457 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:46:01.457 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:01.458 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:01.458 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:46:01.458 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:46:01.458 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:46:01.458 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:46:01.458 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:46:01.458 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:46:01.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.432+0000 7f1dbd379640 10 client.744351.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:46:01.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.432+0000 7f1dbd379640 10 monclient: _renew_subs 2024-09-16T03:46:01.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.432+0000 7f1db5ffb640 4 set_mon_vals no callback set 2024-09-16T03:46:01.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.432+0000 7f1dbd379640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:46:01.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.432+0000 7f1dbd379640 1 -- 172.21.15.42:0/1298771068 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1db8073a30 con 0x7f1db80a1330 2024-09-16T03:46:01.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.432+0000 7f1dbd379640 1 librados: init done 2024-09-16T03:46:01.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.432+0000 7f1dbd379640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:46:01.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.432+0000 7f1dbd379640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:46:01.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.432+0000 7f1dbd379640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:46:01.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.432+0000 7f1dbd379640 1 -- 172.21.15.42:0/1298771068 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1db80a1b40 con 0x7f1db80a1330 2024-09-16T03:46:01.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.432+0000 7f1daf7fe640 1 -- 172.21.15.42:0/1298771068 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f1da8007400 con 0x7f1db80a1330 2024-09-16T03:46:01.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.432+0000 7f1daf7fe640 1 --2- 172.21.15.42:0/1298771068 >> v2:172.21.15.154:6812/254845342 conn(0x7f1d98037810 0x7f1d98039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:46:01.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.432+0000 7f1daf7fe640 1 -- 172.21.15.42:0/1298771068 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(115..115 src has 1..115) ==== 7306+0+0 (secure 0 0 0) 0x7f1da8048890 con 0x7f1db80a1330 2024-09-16T03:46:01.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.432+0000 7f1daf7fe640 10 client.744351.objecter ms_dispatch 0x7f1db8000ba0 osd_map(115..115 src has 1..115) 2024-09-16T03:46:01.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.432+0000 7f1daf7fe640 3 client.744351.objecter handle_osd_map got epochs [115,115] > 0 2024-09-16T03:46:01.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.432+0000 7f1daf7fe640 3 client.744351.objecter handle_osd_map decoding full epoch 115 2024-09-16T03:46:01.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.432+0000 7f1daf7fe640 20 client.744351.objecter dump_active .. 0 homeless 2024-09-16T03:46:01.460 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.432+0000 7f1daffff640 1 --2- 172.21.15.42:0/1298771068 >> v2:172.21.15.154:6812/254845342 conn(0x7f1d98037810 0x7f1d98039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:46:01.460 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.432+0000 7f1daffff640 10 monclient: get_auth_request con 0x7f1d98037810 auth_method 0 2024-09-16T03:46:01.460 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.436+0000 7f1daf7fe640 10 client.744351.objecter ms_handle_connect 0x7f1d98037810 2024-09-16T03:46:01.460 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.436+0000 7f1daffff640 1 --2- 172.21.15.42:0/1298771068 >> v2:172.21.15.154:6812/254845342 conn(0x7f1d98037810 0x7f1d98039cd0 secure :-1 s=READY pgs=3393 cs=0 l=1 rev1=1 crypto rx=0x7f1db8071f00 tx=0x7f1da0009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:46:01.460 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.436+0000 7f1daf7fe640 1 -- 172.21.15.42:0/1298771068 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f1db8120840 con 0x7f1db80a1330 2024-09-16T03:46:01.460 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.436+0000 7f1daf7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:46:01.460 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.436+0000 7f1daf7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:46:01.670 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.664+0000 7f1dbd379640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:46:01.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.664+0000 7f1dbd379640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:46:01.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.664+0000 7f1dbd379640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:46:01.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.664+0000 7f1dbd379640 1 -- 172.21.15.42:0/1298771068 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f1db8069040 con 0x7f1db80a1330 2024-09-16T03:46:01.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.664+0000 7f1daf7fe640 1 -- 172.21.15.42:0/1298771068 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1555 (secure 0 0 0) 0x7f1da8012a30 con 0x7f1db80a1330 2024-09-16T03:46:01.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.664+0000 7f1daf7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:46:01.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.664+0000 7f1daf7fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:46:01.672 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":32,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":24,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:46:01.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.668+0000 7f1dbd379640 10 librados: watch_flush enter 2024-09-16T03:46:01.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.668+0000 7f1dbd379640 10 librados: watch_flush exit 2024-09-16T03:46:01.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.668+0000 7f1dbd379640 20 client.744351.objecter shutdown clearing up homeless session... 2024-09-16T03:46:01.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.668+0000 7f1dbd379640 10 client.744351.objecter successfully canceled tick 2024-09-16T03:46:01.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.668+0000 7f1dbd379640 1 -- 172.21.15.42:0/1298771068 >> v2:172.21.15.154:6812/254845342 conn(0x7f1d98037810 msgr2=0x7f1d98039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:46:01.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.668+0000 7f1dbd379640 1 --2- 172.21.15.42:0/1298771068 >> v2:172.21.15.154:6812/254845342 conn(0x7f1d98037810 0x7f1d98039cd0 secure :-1 s=READY pgs=3393 cs=0 l=1 rev1=1 crypto rx=0x7f1db8071f00 tx=0x7f1da0009190 comp rx=0 tx=0).stop 2024-09-16T03:46:01.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.668+0000 7f1dbd379640 10 monclient: shutdown 2024-09-16T03:46:01.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.668+0000 7f1dbd379640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:46:01.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.668+0000 7f1dbd379640 1 -- 172.21.15.42:0/1298771068 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1db80a1330 msgr2=0x7f1db8071580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:46:01.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.668+0000 7f1dbd379640 1 --2- 172.21.15.42:0/1298771068 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1db80a1330 0x7f1db8071580 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f1da800c3c0 tx=0x7f1da800c880 comp rx=0 tx=0).stop 2024-09-16T03:46:01.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.668+0000 7f1dbd379640 1 -- 172.21.15.42:0/1298771068 shutdown_connections 2024-09-16T03:46:01.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.668+0000 7f1dbd379640 1 --2- 172.21.15.42:0/1298771068 >> v2:172.21.15.154:6812/254845342 conn(0x7f1d98037810 0x7f1d98039cd0 unknown :-1 s=CLOSED pgs=3393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:46:01.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.668+0000 7f1dbd379640 1 --2- 172.21.15.42:0/1298771068 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1db81393a0 0x7f1db8134ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:46:01.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.668+0000 7f1dbd379640 1 --2- 172.21.15.42:0/1298771068 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1db80a1710 0x7f1db8134570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:46:01.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.668+0000 7f1dbd379640 1 --2- 172.21.15.42:0/1298771068 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1db80a1330 0x7f1db8071580 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:46:01.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.668+0000 7f1dbd379640 1 -- 172.21.15.42:0/1298771068 >> 172.21.15.42:0/1298771068 conn(0x7f1db80a2420 msgr2=0x7f1db80a3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:46:01.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.668+0000 7f1dbd379640 1 -- 172.21.15.42:0/1298771068 shutdown_connections 2024-09-16T03:46:01.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.668+0000 7f1dbd379640 1 -- 172.21.15.42:0/1298771068 wait complete. 2024-09-16T03:46:01.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:01.668+0000 7f1dbd379640 1 librados: shutdown 2024-09-16T03:46:04.691 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:46:04.848 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.840+0000 7f004d5cd640 10 monclient: get_monmap_and_config 2024-09-16T03:46:04.848 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.840+0000 7f004d5cd640 10 monclient: build_initial_monmap 2024-09-16T03:46:04.848 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.840+0000 7f004d5cd640 10 monclient: monmap: 2024-09-16T03:46:04.848 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:46:04.848 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:46:04.849 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:46:04.846873+0000 2024-09-16T03:46:04.849 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:46:04.846873+0000 2024-09-16T03:46:04.849 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:46:04.849 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:46:04.849 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:46:04.849 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:46:04.849 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:46:04.849 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:46:04.849 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:46:04.849 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:46:04.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.840+0000 7f004d5cd640 1 Processor -- start 2024-09-16T03:46:04.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.840+0000 7f004d5cd640 1 -- start start 2024-09-16T03:46:04.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.840+0000 7f004d5cd640 10 monclient: init 2024-09-16T03:46:04.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.840+0000 7f004d5cd640 10 monclient: _reopen_session rank -1 2024-09-16T03:46:04.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.840+0000 7f004d5cd640 10 monclient: _add_conns ranks=[4,3,2,1,0] 2024-09-16T03:46:04.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.840+0000 7f004d5cd640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f00480a39b0 0x7f0048133b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:46:04.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.840+0000 7f004d5cd640 10 monclient(hunting): picked mon.noname-c con 0x7f00480a39b0 addr v2:172.21.15.154:3300/0 2024-09-16T03:46:04.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.840+0000 7f004d5cd640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f00480a3d90 0x7f0048137000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:46:04.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.840+0000 7f004d5cd640 10 monclient(hunting): picked mon.noname-a con 0x7f00480a3d90 addr v2:172.21.15.42:3300/0 2024-09-16T03:46:04.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.840+0000 7f004d5cd640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0048137540 0x7f0048137990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:46:04.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.840+0000 7f004d5cd640 10 monclient(hunting): picked mon.noname-b con 0x7f0048137540 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:46:04.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.840+0000 7f004d5cd640 10 monclient(hunting): start opening mon connection 2024-09-16T03:46:04.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.840+0000 7f004d5cd640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f004809c140 con 0x7f00480a3d90 2024-09-16T03:46:04.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.840+0000 7f004d5cd640 10 monclient(hunting): start opening mon connection 2024-09-16T03:46:04.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.840+0000 7f004d5cd640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f004809c420 con 0x7f00480a39b0 2024-09-16T03:46:04.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.840+0000 7f004d5cd640 10 monclient(hunting): start opening mon connection 2024-09-16T03:46:04.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.840+0000 7f004d5cd640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f004806dd60 con 0x7f0048137540 2024-09-16T03:46:04.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.840+0000 7f004d5cd640 10 monclient(hunting): _renew_subs 2024-09-16T03:46:04.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.840+0000 7f004d5cd640 10 monclient(hunting): authenticate will time out at 1211.815959s 2024-09-16T03:46:04.861 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.844+0000 7f00477fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0048137540 0x7f0048137990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:46:04.861 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.844+0000 7f00467fc640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f00480a3d90 0x7f0048137000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:46:04.861 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.844+0000 7f00477fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0048137540 0x7f0048137990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:51594/0 (socket says 172.21.15.42:51594) 2024-09-16T03:46:04.861 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.844+0000 7f00477fe640 1 -- 172.21.15.42:0/743728868 learned_addr learned my addr 172.21.15.42:0/743728868 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:46:04.861 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.844+0000 7f00477fe640 10 monclient(hunting): get_auth_request con 0x7f0048137540 auth_method 0 2024-09-16T03:46:04.861 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.844+0000 7f00477fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:46:04.861 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.844+0000 7f00477fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:46:04.861 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.844+0000 7f00477fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:46:04.861 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.844+0000 7f00467fc640 10 monclient(hunting): get_auth_request con 0x7f00480a3d90 auth_method 0 2024-09-16T03:46:04.862 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.844+0000 7f00467fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:46:04.862 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.844+0000 7f00467fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:46:04.862 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.844+0000 7f00467fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:46:04.862 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.844+0000 7f00477fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:46:04.862 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.844+0000 7f00477fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:46:04.862 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.844+0000 7f0046ffd640 1 --2- 172.21.15.42:0/743728868 >> v2:172.21.15.154:3300/0 conn(0x7f00480a39b0 0x7f0048133b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:46:04.862 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.844+0000 7f00477fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:46:04.862 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.844+0000 7f00467fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:46:04.862 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.844+0000 7f00467fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:46:04.862 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.844+0000 7f00467fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:46:04.862 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.844+0000 7f0046ffd640 10 monclient(hunting): get_auth_request con 0x7f00480a39b0 auth_method 0 2024-09-16T03:46:04.862 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.844+0000 7f0046ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:46:04.862 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.844+0000 7f0046ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:46:04.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.844+0000 7f0046ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:46:04.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.844+0000 7f00477fe640 10 monclient(hunting): handle_auth_done global_id 13508 payload 274 2024-09-16T03:46:04.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.844+0000 7f00477fe640 1 -- 172.21.15.42:0/743728868 >> v2:172.21.15.154:3300/0 conn(0x7f00480a39b0 msgr2=0x7f0048133b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:46:04.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.844+0000 7f00477fe640 1 --2- 172.21.15.42:0/743728868 >> v2:172.21.15.154:3300/0 conn(0x7f00480a39b0 0x7f0048133b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:46:04.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.844+0000 7f00477fe640 1 -- 172.21.15.42:0/743728868 >> v2:172.21.15.42:3300/0 conn(0x7f00480a3d90 msgr2=0x7f0048137000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:46:04.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.844+0000 7f00477fe640 1 --2- 172.21.15.42:0/743728868 >> v2:172.21.15.42:3300/0 conn(0x7f00480a3d90 0x7f0048137000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:46:04.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.844+0000 7f00477fe640 10 monclient: _finish_hunting 0 2024-09-16T03:46:04.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.844+0000 7f00477fe640 1 monclient: found mon.noname-b 2024-09-16T03:46:04.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.844+0000 7f00477fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:46:04.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.844+0000 7f00477fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T03:46:04.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.844+0000 7f00477fe640 1 -- 172.21.15.42:0/743728868 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00480a5e20 con 0x7f0048137540 2024-09-16T03:46:04.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.844+0000 7f00477fe640 1 --2- 172.21.15.42:0/743728868 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0048137540 0x7f0048137990 secure :-1 s=READY pgs=2410 cs=0 l=1 rev1=1 crypto rx=0x7f00480a7750 tx=0x7f004001c8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ee5bc9d55ed45ed9 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:46:04.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.844+0000 7f0045ffb640 1 -- 172.21.15.42:0/743728868 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00400a5530 con 0x7f0048137540 2024-09-16T03:46:04.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.844+0000 7f0045ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:46:04.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.844+0000 7f0045ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:46:04.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.844+0000 7f0045ffb640 10 monclient: dump: 2024-09-16T03:46:04.864 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:46:04.864 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:46:04.864 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:04.864 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:04.864 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:46:04.864 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:46:04.864 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:46:04.864 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:46:04.864 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:46:04.865 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:46:04.865 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.844+0000 7f0045ffb640 10 monclient: _finish_auth 0 2024-09-16T03:46:04.865 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.844+0000 7f0045ffb640 10 monclient: _check_auth_tickets 2024-09-16T03:46:04.865 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.844+0000 7f0045ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:46:04.865 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.844+0000 7f0045ffb640 1 -- 172.21.15.42:0/743728868 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f00400a56d0 con 0x7f0048137540 2024-09-16T03:46:04.865 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.844+0000 7f004d5cd640 5 monclient: authenticate success, global_id 13508 2024-09-16T03:46:04.865 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.844+0000 7f004d5cd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:46:04.865 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.844+0000 7f0045ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T03:46:04.865 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.844+0000 7f004d5cd640 10 monclient: get_monmap_and_config success 2024-09-16T03:46:04.865 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.844+0000 7f0045ffb640 1 -- 172.21.15.42:0/743728868 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00400a5aa0 con 0x7f0048137540 2024-09-16T03:46:04.865 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.844+0000 7f0045ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:46:04.865 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.844+0000 7f004d5cd640 4 set_mon_vals no callback set 2024-09-16T03:46:04.865 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.844+0000 7f0045ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:46:04.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.844+0000 7f0045ffb640 10 monclient: dump: 2024-09-16T03:46:04.866 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:46:04.866 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:46:04.866 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:04.866 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:04.866 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:46:04.866 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:46:04.866 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:46:04.866 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:46:04.866 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:46:04.867 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:46:04.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.844+0000 7f004d5cd640 10 monclient: shutdown 2024-09-16T03:46:04.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.844+0000 7f004d5cd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:46:04.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.844+0000 7f004d5cd640 1 -- 172.21.15.42:0/743728868 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0048137540 msgr2=0x7f0048137990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:46:04.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.844+0000 7f004d5cd640 1 --2- 172.21.15.42:0/743728868 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0048137540 0x7f0048137990 secure :-1 s=READY pgs=2410 cs=0 l=1 rev1=1 crypto rx=0x7f00480a7750 tx=0x7f004001c8b0 comp rx=0 tx=0).stop 2024-09-16T03:46:04.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.844+0000 7f004d5cd640 1 -- 172.21.15.42:0/743728868 shutdown_connections 2024-09-16T03:46:04.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.844+0000 7f004d5cd640 1 --2- 172.21.15.42:0/743728868 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0048137540 0x7f0048137990 unknown :-1 s=CLOSED pgs=2410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:46:04.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.844+0000 7f004d5cd640 1 --2- 172.21.15.42:0/743728868 >> v2:172.21.15.42:3300/0 conn(0x7f00480a3d90 0x7f0048137000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:46:04.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.844+0000 7f004d5cd640 1 --2- 172.21.15.42:0/743728868 >> v2:172.21.15.154:3300/0 conn(0x7f00480a39b0 0x7f0048133b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:46:04.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.844+0000 7f004d5cd640 1 -- 172.21.15.42:0/743728868 >> 172.21.15.42:0/743728868 conn(0x7f00480a9fb0 msgr2=0x7f00480aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:46:04.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.844+0000 7f004d5cd640 1 -- 172.21.15.42:0/743728868 shutdown_connections 2024-09-16T03:46:04.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.844+0000 7f004d5cd640 1 -- 172.21.15.42:0/743728868 wait complete. 2024-09-16T03:46:04.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.848+0000 7f004d5cd640 10 monclient: build_initial_monmap 2024-09-16T03:46:04.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.848+0000 7f004d5cd640 10 monclient: monmap: 2024-09-16T03:46:04.868 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:46:04.868 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:46:04.868 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:46:04.868 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:46:04.868 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:46:04.868 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:46:04.868 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:46:04.868 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:46:04.868 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:46:04.868 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:46:04.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.848+0000 7f004d5cd640 1 librados: starting msgr at 2024-09-16T03:46:04.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.848+0000 7f004d5cd640 1 librados: starting objecter 2024-09-16T03:46:04.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.848+0000 7f004d5cd640 1 Processor -- start 2024-09-16T03:46:04.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.848+0000 7f004d5cd640 1 -- start start 2024-09-16T03:46:04.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.848+0000 7f004d5cd640 1 librados: setting wanted keys 2024-09-16T03:46:04.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.848+0000 7f004d5cd640 1 librados: calling monclient init 2024-09-16T03:46:04.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.848+0000 7f004d5cd640 10 monclient: init 2024-09-16T03:46:04.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.848+0000 7f004d5cd640 10 monclient: _reopen_session rank -1 2024-09-16T03:46:04.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.848+0000 7f004d5cd640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T03:46:04.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.848+0000 7f004d5cd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f00480a39b0 0x7f00481cf0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:46:04.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.848+0000 7f004d5cd640 10 monclient(hunting): picked mon.noname-b con 0x7f00480a39b0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:46:04.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.848+0000 7f004d5cd640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f00480a3d90 0x7f00480a2ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:46:04.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.848+0000 7f004d5cd640 10 monclient(hunting): picked mon.noname-c con 0x7f00480a3d90 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:46:04.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.848+0000 7f004d5cd640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0048137540 0x7f004813b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:46:04.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.848+0000 7f004d5cd640 10 monclient(hunting): picked mon.noname-a con 0x7f0048137540 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:46:04.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.848+0000 7f004d5cd640 10 monclient(hunting): start opening mon connection 2024-09-16T03:46:04.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.848+0000 7f004d5cd640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f0048140dc0 con 0x7f0048137540 2024-09-16T03:46:04.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.848+0000 7f004d5cd640 10 monclient(hunting): start opening mon connection 2024-09-16T03:46:04.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.848+0000 7f004d5cd640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f0048140f40 con 0x7f00480a39b0 2024-09-16T03:46:04.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.848+0000 7f004d5cd640 10 monclient(hunting): start opening mon connection 2024-09-16T03:46:04.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.848+0000 7f004d5cd640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f00481cffe0 con 0x7f00480a3d90 2024-09-16T03:46:04.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.848+0000 7f004d5cd640 10 monclient(hunting): _renew_subs 2024-09-16T03:46:04.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.848+0000 7f004d5cd640 10 monclient(hunting): authenticate will time out at 1211.821097s 2024-09-16T03:46:04.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.848+0000 7f00467fc640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f00480a3d90 0x7f00480a2ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:46:04.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.848+0000 7f00467fc640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f00480a3d90 0x7f00480a2ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:51610/0 (socket says 172.21.15.42:51610) 2024-09-16T03:46:04.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.848+0000 7f00467fc640 1 -- 172.21.15.42:0/853099491 learned_addr learned my addr 172.21.15.42:0/853099491 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:46:04.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.848+0000 7f00467fc640 10 monclient(hunting): get_auth_request con 0x7f00480a3d90 auth_method 0 2024-09-16T03:46:04.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.848+0000 7f00467fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:46:04.871 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.848+0000 7f00467fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:46:04.871 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.848+0000 7f00467fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:46:04.871 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.848+0000 7f00477fe640 1 --2- 172.21.15.42:0/853099491 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0048137540 0x7f004813b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:46:04.871 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.848+0000 7f00477fe640 10 monclient(hunting): get_auth_request con 0x7f0048137540 auth_method 0 2024-09-16T03:46:04.871 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.848+0000 7f00477fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:46:04.871 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.848+0000 7f00477fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:46:04.871 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.848+0000 7f00477fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:46:04.871 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.848+0000 7f00467fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:46:04.871 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.848+0000 7f00467fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:46:04.871 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.848+0000 7f00467fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:46:04.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.848+0000 7f0046ffd640 1 --2- 172.21.15.42:0/853099491 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f00480a39b0 0x7f00481cf0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:46:04.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.848+0000 7f00477fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:46:04.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.848+0000 7f00477fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:46:04.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.848+0000 7f00477fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:46:04.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.848+0000 7f0046ffd640 10 monclient(hunting): get_auth_request con 0x7f00480a39b0 auth_method 0 2024-09-16T03:46:04.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.848+0000 7f0046ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:46:04.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.848+0000 7f0046ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:46:04.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.848+0000 7f0046ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:46:04.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.848+0000 7f00467fc640 10 monclient(hunting): handle_auth_done global_id 13511 payload 824 2024-09-16T03:46:04.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.848+0000 7f00467fc640 1 -- 172.21.15.42:0/853099491 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f00480a39b0 msgr2=0x7f00481cf0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:46:04.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.848+0000 7f0046ffd640 1 -- 172.21.15.42:0/853099491 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f00480a39b0 msgr2=0x7f00481cf0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-16T03:46:04.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.848+0000 7f0046ffd640 1 -- 172.21.15.42:0/853099491 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f00480a39b0 msgr2=0x7f00481cf0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T03:46:04.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.848+0000 7f0046ffd640 1 --2- 172.21.15.42:0/853099491 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f00480a39b0 0x7f00481cf0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:46:04.873 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.848+0000 7f0046ffd640 1 --2- 172.21.15.42:0/853099491 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f00480a39b0 0x7f00481cf0d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:46:04.873 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.848+0000 7f00467fc640 1 --2- 172.21.15.42:0/853099491 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f00480a39b0 0x7f00481cf0d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:46:04.873 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.848+0000 7f00467fc640 1 -- 172.21.15.42:0/853099491 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0048137540 msgr2=0x7f004813b220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:46:04.873 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.848+0000 7f00467fc640 1 --2- 172.21.15.42:0/853099491 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0048137540 0x7f004813b220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:46:04.873 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.848+0000 7f00467fc640 10 monclient: _finish_hunting 0 2024-09-16T03:46:04.873 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.848+0000 7f00467fc640 1 monclient: found mon.noname-c 2024-09-16T03:46:04.873 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.848+0000 7f00467fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:46:04.873 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.848+0000 7f00467fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:46:04.873 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.848+0000 7f00467fc640 1 -- 172.21.15.42:0/853099491 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f004809d490 con 0x7f00480a3d90 2024-09-16T03:46:04.873 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.848+0000 7f00477fe640 1 --2- 172.21.15.42:0/853099491 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0048137540 0x7f004813b220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:46:04.873 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.848+0000 7f00467fc640 1 --2- 172.21.15.42:0/853099491 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f00480a3d90 0x7f00480a2ce0 secure :-1 s=READY pgs=2411 cs=0 l=1 rev1=1 crypto rx=0x7f003c00e930 tx=0x7f003c00edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:46:04.873 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.848+0000 7f002f7fe640 10 client.?.objecter ms_handle_connect 0x7f00480a3d90 2024-09-16T03:46:04.874 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.848+0000 7f002f7fe640 10 client.?.objecter resend_mon_ops 2024-09-16T03:46:04.874 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.848+0000 7f002f7fe640 1 -- 172.21.15.42:0/853099491 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f003c017070 con 0x7f00480a3d90 2024-09-16T03:46:04.874 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.848+0000 7f002f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:46:04.874 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.848+0000 7f002f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:46:04.874 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.848+0000 7f002f7fe640 10 monclient: dump: 2024-09-16T03:46:04.874 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:46:04.874 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:46:04.874 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:04.874 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:04.874 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:46:04.874 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:46:04.874 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:46:04.875 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:46:04.875 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:46:04.875 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:46:04.875 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.848+0000 7f002f7fe640 10 monclient: _finish_auth 0 2024-09-16T03:46:04.875 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.848+0000 7f002f7fe640 10 monclient: _check_auth_tickets 2024-09-16T03:46:04.875 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.848+0000 7f002f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:46:04.875 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.848+0000 7f002f7fe640 1 -- 172.21.15.42:0/853099491 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f003c00ca00 con 0x7f00480a3d90 2024-09-16T03:46:04.875 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.848+0000 7f004d5cd640 5 monclient: authenticate success, global_id 13511 2024-09-16T03:46:04.875 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.848+0000 7f002f7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:46:04.875 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.848+0000 7f002f7fe640 1 -- 172.21.15.42:0/853099491 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f003c00cdb0 con 0x7f00480a3d90 2024-09-16T03:46:04.875 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.848+0000 7f004d5cd640 10 monclient: _renew_subs 2024-09-16T03:46:04.875 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.848+0000 7f004d5cd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:46:04.875 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.848+0000 7f004d5cd640 1 -- 172.21.15.42:0/853099491 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00481cf610 con 0x7f00480a3d90 2024-09-16T03:46:04.875 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.848+0000 7f00457fa640 4 set_mon_vals no callback set 2024-09-16T03:46:04.876 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.848+0000 7f002f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:46:04.876 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.848+0000 7f002f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:46:04.876 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.848+0000 7f002f7fe640 10 monclient: dump: 2024-09-16T03:46:04.876 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:46:04.876 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:46:04.876 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:04.876 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:04.876 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:46:04.876 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:46:04.876 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:46:04.876 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:46:04.876 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:46:04.877 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:46:04.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.848+0000 7f004d5cd640 10 client.13511.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:46:04.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.848+0000 7f004d5cd640 10 monclient: _renew_subs 2024-09-16T03:46:04.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.848+0000 7f004d5cd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:46:04.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.848+0000 7f004d5cd640 1 -- 172.21.15.42:0/853099491 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f004809d1b0 con 0x7f00480a3d90 2024-09-16T03:46:04.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.848+0000 7f004d5cd640 1 librados: init done 2024-09-16T03:46:04.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.848+0000 7f004d5cd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:46:04.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.848+0000 7f004d5cd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:46:04.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.848+0000 7f004d5cd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:46:04.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.848+0000 7f004d5cd640 1 -- 172.21.15.42:0/853099491 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0048137000 con 0x7f00480a3d90 2024-09-16T03:46:04.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.848+0000 7f002f7fe640 1 -- 172.21.15.42:0/853099491 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f003c01a070 con 0x7f00480a3d90 2024-09-16T03:46:04.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.848+0000 7f002f7fe640 1 --2- 172.21.15.42:0/853099491 >> v2:172.21.15.154:6812/254845342 conn(0x7f0034037810 0x7f0034039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:46:04.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.848+0000 7f002f7fe640 1 -- 172.21.15.42:0/853099491 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(119..119 src has 1..119) ==== 8177+0+0 (secure 0 0 0) 0x7f003c049e30 con 0x7f00480a3d90 2024-09-16T03:46:04.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.848+0000 7f002f7fe640 10 client.13511.objecter ms_dispatch 0x7f0048000ba0 osd_map(119..119 src has 1..119) 2024-09-16T03:46:04.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.848+0000 7f002f7fe640 3 client.13511.objecter handle_osd_map got epochs [119,119] > 0 2024-09-16T03:46:04.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.848+0000 7f002f7fe640 3 client.13511.objecter handle_osd_map decoding full epoch 119 2024-09-16T03:46:04.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.848+0000 7f002f7fe640 20 client.13511.objecter dump_active .. 0 homeless 2024-09-16T03:46:04.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.848+0000 7f0046ffd640 1 --2- 172.21.15.42:0/853099491 >> v2:172.21.15.154:6812/254845342 conn(0x7f0034037810 0x7f0034039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:46:04.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.848+0000 7f0046ffd640 10 monclient: get_auth_request con 0x7f0034037810 auth_method 0 2024-09-16T03:46:04.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.852+0000 7f002f7fe640 10 client.13511.objecter ms_handle_connect 0x7f0034037810 2024-09-16T03:46:04.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.852+0000 7f0046ffd640 1 --2- 172.21.15.42:0/853099491 >> v2:172.21.15.154:6812/254845342 conn(0x7f0034037810 0x7f0034039cd0 secure :-1 s=READY pgs=3395 cs=0 l=1 rev1=1 crypto rx=0x7f003800b990 tx=0x7f0038008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:46:04.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.856+0000 7f002f7fe640 1 -- 172.21.15.42:0/853099491 <== mon.2 v2:172.21.15.42:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f003c011350 con 0x7f00480a3d90 2024-09-16T03:46:04.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.856+0000 7f002f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:46:04.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:04.856+0000 7f002f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:46:05.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.084+0000 7f004d5cd640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:46:05.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.084+0000 7f004d5cd640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:46:05.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.084+0000 7f004d5cd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:46:05.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.084+0000 7f004d5cd640 1 -- 172.21.15.42:0/853099491 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f00480a5750 con 0x7f00480a3d90 2024-09-16T03:46:05.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.084+0000 7f002f7fe640 1 -- 172.21.15.42:0/853099491 <== mon.2 v2:172.21.15.42:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7f003c047e20 con 0x7f00480a3d90 2024-09-16T03:46:05.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.084+0000 7f002f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:46:05.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.084+0000 7f002f7fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:46:05.090 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":34,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:46:05.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.084+0000 7f004d5cd640 10 librados: watch_flush enter 2024-09-16T03:46:05.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.084+0000 7f004d5cd640 10 librados: watch_flush exit 2024-09-16T03:46:05.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.084+0000 7f004d5cd640 20 client.13511.objecter shutdown clearing up homeless session... 2024-09-16T03:46:05.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.084+0000 7f004d5cd640 10 client.13511.objecter successfully canceled tick 2024-09-16T03:46:05.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.084+0000 7f004d5cd640 1 -- 172.21.15.42:0/853099491 >> v2:172.21.15.154:6812/254845342 conn(0x7f0034037810 msgr2=0x7f0034039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:46:05.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.084+0000 7f004d5cd640 1 --2- 172.21.15.42:0/853099491 >> v2:172.21.15.154:6812/254845342 conn(0x7f0034037810 0x7f0034039cd0 secure :-1 s=READY pgs=3395 cs=0 l=1 rev1=1 crypto rx=0x7f003800b990 tx=0x7f0038008040 comp rx=0 tx=0).stop 2024-09-16T03:46:05.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.084+0000 7f004d5cd640 10 monclient: shutdown 2024-09-16T03:46:05.093 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.084+0000 7f004d5cd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:46:05.093 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.084+0000 7f004d5cd640 1 -- 172.21.15.42:0/853099491 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f00480a3d90 msgr2=0x7f00480a2ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:46:05.093 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.084+0000 7f004d5cd640 1 --2- 172.21.15.42:0/853099491 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f00480a3d90 0x7f00480a2ce0 secure :-1 s=READY pgs=2411 cs=0 l=1 rev1=1 crypto rx=0x7f003c00e930 tx=0x7f003c00edf0 comp rx=0 tx=0).stop 2024-09-16T03:46:05.093 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.084+0000 7f004d5cd640 1 -- 172.21.15.42:0/853099491 shutdown_connections 2024-09-16T03:46:05.093 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.084+0000 7f004d5cd640 1 --2- 172.21.15.42:0/853099491 >> v2:172.21.15.154:6812/254845342 conn(0x7f0034037810 0x7f0034039cd0 unknown :-1 s=CLOSED pgs=3395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:46:05.093 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.084+0000 7f004d5cd640 1 --2- 172.21.15.42:0/853099491 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0048137540 0x7f004813b220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:46:05.093 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.084+0000 7f004d5cd640 1 --2- 172.21.15.42:0/853099491 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f00480a3d90 0x7f00480a2ce0 unknown :-1 s=CLOSED pgs=2411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:46:05.093 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.084+0000 7f004d5cd640 1 --2- 172.21.15.42:0/853099491 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f00480a39b0 0x7f00481cf0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:46:05.093 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.084+0000 7f004d5cd640 1 -- 172.21.15.42:0/853099491 >> 172.21.15.42:0/853099491 conn(0x7f00480a9fb0 msgr2=0x7f00480a7fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:46:05.093 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.084+0000 7f004d5cd640 1 -- 172.21.15.42:0/853099491 shutdown_connections 2024-09-16T03:46:05.093 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.084+0000 7f004d5cd640 1 -- 172.21.15.42:0/853099491 wait complete. 2024-09-16T03:46:05.093 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.084+0000 7f004d5cd640 1 librados: shutdown 2024-09-16T03:46:05.110 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T03:46:05.110 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T03:46:05.110 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T03:46:05.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.220+0000 7fae8df90640 10 monclient: get_monmap_and_config 2024-09-16T03:46:05.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.220+0000 7fae8df90640 10 monclient: build_initial_monmap 2024-09-16T03:46:05.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.220+0000 7fae8df90640 10 monclient: monmap: 2024-09-16T03:46:05.229 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:46:05.229 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:46:05.229 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:46:05.227646+0000 2024-09-16T03:46:05.229 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:46:05.227646+0000 2024-09-16T03:46:05.229 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:46:05.229 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:46:05.229 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:46:05.229 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:46:05.229 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:46:05.229 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:46:05.229 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:46:05.229 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:46:05.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.224+0000 7fae8df90640 1 Processor -- start 2024-09-16T03:46:05.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.224+0000 7fae8df90640 1 -- start start 2024-09-16T03:46:05.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.224+0000 7fae8df90640 10 monclient: init 2024-09-16T03:46:05.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.224+0000 7fae8df90640 10 monclient: _reopen_session rank -1 2024-09-16T03:46:05.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.224+0000 7fae8df90640 10 monclient: _add_conns ranks=[2,3,4,0,1] 2024-09-16T03:46:05.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.224+0000 7fae8df90640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fae880a1330 0x7fae880a1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:46:05.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.224+0000 7fae8df90640 10 monclient(hunting): picked mon.noname-b con 0x7fae880a1330 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:46:05.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.224+0000 7fae8df90640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fae8806dac0 0x7fae880a1c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:46:05.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.224+0000 7fae8df90640 10 monclient(hunting): picked mon.noname-a con 0x7fae8806dac0 addr v2:172.21.15.42:3300/0 2024-09-16T03:46:05.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.224+0000 7fae8df90640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fae88135100 0x7fae88135550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:46:05.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.224+0000 7fae8df90640 10 monclient(hunting): picked mon.noname-c con 0x7fae88135100 addr v2:172.21.15.154:3300/0 2024-09-16T03:46:05.233 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.224+0000 7fae8df90640 10 monclient(hunting): start opening mon connection 2024-09-16T03:46:05.233 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.224+0000 7fae8df90640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7fae8809c7d0 con 0x7fae8806dac0 2024-09-16T03:46:05.233 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.224+0000 7fae8df90640 10 monclient(hunting): start opening mon connection 2024-09-16T03:46:05.233 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.224+0000 7fae8df90640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fae8809c090 con 0x7fae88135100 2024-09-16T03:46:05.233 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.224+0000 7fae8df90640 10 monclient(hunting): start opening mon connection 2024-09-16T03:46:05.233 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.224+0000 7fae8df90640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fae8809f610 con 0x7fae880a1330 2024-09-16T03:46:05.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.224+0000 7fae8df90640 10 monclient(hunting): _renew_subs 2024-09-16T03:46:05.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.224+0000 7fae8df90640 10 monclient(hunting): authenticate will time out at 1212.197080s 2024-09-16T03:46:05.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.224+0000 7fae86ffd640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fae8806dac0 0x7fae880a1c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:46:05.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.224+0000 7fae877fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fae880a1330 0x7fae880a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:46:05.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.224+0000 7fae86ffd640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fae8806dac0 0x7fae880a1c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:44254/0 (socket says 172.21.15.42:44254) 2024-09-16T03:46:05.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.224+0000 7fae86ffd640 1 -- 172.21.15.42:0/3348975316 learned_addr learned my addr 172.21.15.42:0/3348975316 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:46:05.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.224+0000 7fae86ffd640 10 monclient(hunting): get_auth_request con 0x7fae8806dac0 auth_method 0 2024-09-16T03:46:05.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.224+0000 7fae86ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:46:05.235 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.224+0000 7fae86ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:46:05.235 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.224+0000 7fae86ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:46:05.235 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.224+0000 7fae877fe640 10 monclient(hunting): get_auth_request con 0x7fae880a1330 auth_method 0 2024-09-16T03:46:05.235 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.224+0000 7fae877fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:46:05.235 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.224+0000 7fae877fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:46:05.235 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.224+0000 7fae877fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:46:05.235 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.224+0000 7fae86ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:46:05.235 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.224+0000 7fae86ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:46:05.235 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.224+0000 7fae86ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:46:05.235 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.224+0000 7fae87fff640 1 --2- 172.21.15.42:0/3348975316 >> v2:172.21.15.154:3300/0 conn(0x7fae88135100 0x7fae88135550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:46:05.235 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.224+0000 7fae877fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:46:05.235 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.224+0000 7fae877fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:46:05.235 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.224+0000 7fae877fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:46:05.235 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.224+0000 7fae87fff640 10 monclient(hunting): get_auth_request con 0x7fae88135100 auth_method 0 2024-09-16T03:46:05.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.224+0000 7fae87fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:46:05.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.224+0000 7fae87fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:46:05.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.224+0000 7fae87fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:46:05.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.224+0000 7fae86ffd640 10 monclient(hunting): handle_auth_done global_id 744366 payload 274 2024-09-16T03:46:05.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.224+0000 7fae86ffd640 1 -- 172.21.15.42:0/3348975316 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fae880a1330 msgr2=0x7fae880a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:46:05.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.224+0000 7fae86ffd640 1 --2- 172.21.15.42:0/3348975316 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fae880a1330 0x7fae880a1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:46:05.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.224+0000 7fae86ffd640 1 -- 172.21.15.42:0/3348975316 >> v2:172.21.15.154:3300/0 conn(0x7fae88135100 msgr2=0x7fae88135550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:46:05.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.224+0000 7fae86ffd640 1 --2- 172.21.15.42:0/3348975316 >> v2:172.21.15.154:3300/0 conn(0x7fae88135100 0x7fae88135550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:46:05.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.224+0000 7fae86ffd640 10 monclient: _finish_hunting 0 2024-09-16T03:46:05.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.224+0000 7fae86ffd640 1 monclient: found mon.noname-a 2024-09-16T03:46:05.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.224+0000 7fae86ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:46:05.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.224+0000 7fae86ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:46:05.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.224+0000 7fae86ffd640 1 -- 172.21.15.42:0/3348975316 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae8809f8f0 con 0x7fae8806dac0 2024-09-16T03:46:05.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.224+0000 7fae86ffd640 1 --2- 172.21.15.42:0/3348975316 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fae8806dac0 0x7fae880a1c50 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fae8809bb00 tx=0x7fae7001c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21cc52dc53772d8f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:46:05.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.228+0000 7fae867fc640 1 -- 172.21.15.42:0/3348975316 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae700b1070 con 0x7fae8806dac0 2024-09-16T03:46:05.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.228+0000 7fae867fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:46:05.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.228+0000 7fae867fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:46:05.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.228+0000 7fae867fc640 10 monclient: dump: 2024-09-16T03:46:05.245 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:46:05.245 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:46:05.245 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:05.245 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:05.245 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:46:05.245 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:46:05.245 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:46:05.245 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:46:05.245 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:46:05.245 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:46:05.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.228+0000 7fae867fc640 10 monclient: _finish_auth 0 2024-09-16T03:46:05.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.228+0000 7fae867fc640 10 monclient: _check_auth_tickets 2024-09-16T03:46:05.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.228+0000 7fae867fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:46:05.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.228+0000 7fae867fc640 1 -- 172.21.15.42:0/3348975316 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fae700ac620 con 0x7fae8806dac0 2024-09-16T03:46:05.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.228+0000 7fae8df90640 5 monclient: authenticate success, global_id 744366 2024-09-16T03:46:05.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.228+0000 7fae8df90640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:46:05.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.228+0000 7fae867fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:46:05.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.228+0000 7fae8df90640 10 monclient: get_monmap_and_config success 2024-09-16T03:46:05.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.228+0000 7fae867fc640 1 -- 172.21.15.42:0/3348975316 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae700ac9f0 con 0x7fae8806dac0 2024-09-16T03:46:05.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.228+0000 7fae8df90640 4 set_mon_vals no callback set 2024-09-16T03:46:05.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.228+0000 7fae867fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:46:05.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.228+0000 7fae867fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:46:05.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.228+0000 7fae867fc640 10 monclient: dump: 2024-09-16T03:46:05.246 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:46:05.246 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:46:05.247 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:05.247 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:05.247 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:46:05.247 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:46:05.247 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:46:05.247 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:46:05.247 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:46:05.247 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:46:05.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.228+0000 7fae8df90640 10 monclient: shutdown 2024-09-16T03:46:05.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.228+0000 7fae8df90640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:46:05.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.228+0000 7fae8df90640 1 -- 172.21.15.42:0/3348975316 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fae8806dac0 msgr2=0x7fae880a1c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:46:05.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.228+0000 7fae8df90640 1 --2- 172.21.15.42:0/3348975316 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fae8806dac0 0x7fae880a1c50 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fae8809bb00 tx=0x7fae7001c440 comp rx=0 tx=0).stop 2024-09-16T03:46:05.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.228+0000 7fae8df90640 1 -- 172.21.15.42:0/3348975316 shutdown_connections 2024-09-16T03:46:05.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.228+0000 7fae8df90640 1 --2- 172.21.15.42:0/3348975316 >> v2:172.21.15.154:3300/0 conn(0x7fae88135100 0x7fae88135550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:46:05.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.228+0000 7fae8df90640 1 --2- 172.21.15.42:0/3348975316 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fae8806dac0 0x7fae880a1c50 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:46:05.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.228+0000 7fae8df90640 1 --2- 172.21.15.42:0/3348975316 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fae880a1330 0x7fae880a1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:46:05.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.228+0000 7fae8df90640 1 -- 172.21.15.42:0/3348975316 >> 172.21.15.42:0/3348975316 conn(0x7fae880a2420 msgr2=0x7fae880a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:46:05.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.228+0000 7fae8df90640 1 -- 172.21.15.42:0/3348975316 shutdown_connections 2024-09-16T03:46:05.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.228+0000 7fae8df90640 1 -- 172.21.15.42:0/3348975316 wait complete. 2024-09-16T03:46:05.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.228+0000 7fae8df90640 10 monclient: build_initial_monmap 2024-09-16T03:46:05.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.228+0000 7fae8df90640 10 monclient: monmap: 2024-09-16T03:46:05.248 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:46:05.249 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:46:05.249 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:46:05.249 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:46:05.249 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:46:05.249 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:46:05.249 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:46:05.249 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:46:05.249 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:46:05.249 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:46:05.249 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.228+0000 7fae8df90640 1 librados: starting msgr at 2024-09-16T03:46:05.249 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.228+0000 7fae8df90640 1 librados: starting objecter 2024-09-16T03:46:05.249 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.228+0000 7fae8df90640 1 Processor -- start 2024-09-16T03:46:05.249 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.228+0000 7fae8df90640 1 -- start start 2024-09-16T03:46:05.250 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.228+0000 7fae8df90640 1 librados: setting wanted keys 2024-09-16T03:46:05.250 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.228+0000 7fae8df90640 1 librados: calling monclient init 2024-09-16T03:46:05.250 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.228+0000 7fae8df90640 10 monclient: init 2024-09-16T03:46:05.250 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.228+0000 7fae8df90640 10 monclient: _reopen_session rank -1 2024-09-16T03:46:05.250 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.228+0000 7fae8df90640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:46:05.250 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.228+0000 7fae8df90640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fae880a1330 0x7fae881cefa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:46:05.250 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.228+0000 7fae8df90640 10 monclient(hunting): picked mon.noname-c con 0x7fae880a1330 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:46:05.250 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.228+0000 7fae8df90640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fae88135100 0x7fae881345a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:46:05.250 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.228+0000 7fae8df90640 10 monclient(hunting): picked mon.noname-a con 0x7fae88135100 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:46:05.250 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.228+0000 7fae8df90640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fae88134ae0 0x7fae8811dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:46:05.250 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.228+0000 7fae8df90640 10 monclient(hunting): picked mon.noname-b con 0x7fae88134ae0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:46:05.250 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.228+0000 7fae8df90640 10 monclient(hunting): start opening mon connection 2024-09-16T03:46:05.250 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.228+0000 7fae8df90640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fae8809eef0 con 0x7fae88135100 2024-09-16T03:46:05.250 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.228+0000 7fae8df90640 10 monclient(hunting): start opening mon connection 2024-09-16T03:46:05.251 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.228+0000 7fae8df90640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fae8809ed70 con 0x7fae88134ae0 2024-09-16T03:46:05.251 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.228+0000 7fae8df90640 10 monclient(hunting): start opening mon connection 2024-09-16T03:46:05.251 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.228+0000 7fae8df90640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fae8813eb70 con 0x7fae880a1330 2024-09-16T03:46:05.251 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.228+0000 7fae8df90640 10 monclient(hunting): _renew_subs 2024-09-16T03:46:05.251 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.228+0000 7fae8df90640 10 monclient(hunting): authenticate will time out at 1212.202350s 2024-09-16T03:46:05.251 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.228+0000 7fae86ffd640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fae88135100 0x7fae881345a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:46:05.251 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.228+0000 7fae877fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fae880a1330 0x7fae881cefa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:46:05.251 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.228+0000 7fae86ffd640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fae88135100 0x7fae881345a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:44262/0 (socket says 172.21.15.42:44262) 2024-09-16T03:46:05.251 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.228+0000 7fae877fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fae880a1330 0x7fae881cefa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:51626/0 (socket says 172.21.15.42:51626) 2024-09-16T03:46:05.251 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.228+0000 7fae86ffd640 1 -- 172.21.15.42:0/864528068 learned_addr learned my addr 172.21.15.42:0/864528068 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:46:05.251 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.228+0000 7fae86ffd640 10 monclient(hunting): get_auth_request con 0x7fae88135100 auth_method 0 2024-09-16T03:46:05.251 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.228+0000 7fae86ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:46:05.251 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.228+0000 7fae86ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:46:05.251 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.228+0000 7fae86ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:46:05.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.228+0000 7fae877fe640 10 monclient(hunting): get_auth_request con 0x7fae880a1330 auth_method 0 2024-09-16T03:46:05.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.228+0000 7fae877fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:46:05.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.228+0000 7fae877fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:46:05.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.228+0000 7fae877fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:46:05.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.228+0000 7fae86ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:46:05.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.228+0000 7fae86ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:46:05.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.228+0000 7fae86ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:46:05.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.228+0000 7fae877fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:46:05.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.228+0000 7fae877fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:46:05.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.228+0000 7fae877fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:46:05.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.228+0000 7fae87fff640 1 --2- 172.21.15.42:0/864528068 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fae88134ae0 0x7fae8811dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:46:05.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.228+0000 7fae87fff640 10 monclient(hunting): get_auth_request con 0x7fae88134ae0 auth_method 0 2024-09-16T03:46:05.253 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.228+0000 7fae87fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:46:05.253 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.228+0000 7fae87fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:46:05.253 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.228+0000 7fae87fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:46:05.253 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.228+0000 7fae86ffd640 10 monclient(hunting): handle_auth_done global_id 744369 payload 824 2024-09-16T03:46:05.253 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.228+0000 7fae86ffd640 1 -- 172.21.15.42:0/864528068 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fae880a1330 msgr2=0x7fae881cefa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:46:05.253 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.228+0000 7fae86ffd640 1 --2- 172.21.15.42:0/864528068 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fae880a1330 0x7fae881cefa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:46:05.253 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.228+0000 7fae86ffd640 1 -- 172.21.15.42:0/864528068 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fae88134ae0 msgr2=0x7fae8811dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:46:05.253 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.228+0000 7fae86ffd640 1 --2- 172.21.15.42:0/864528068 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fae88134ae0 0x7fae8811dff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:46:05.253 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.228+0000 7fae86ffd640 10 monclient: _finish_hunting 0 2024-09-16T03:46:05.253 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.228+0000 7fae86ffd640 1 monclient: found mon.noname-a 2024-09-16T03:46:05.253 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.228+0000 7fae86ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:46:05.253 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.228+0000 7fae86ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:46:05.253 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.228+0000 7fae86ffd640 1 -- 172.21.15.42:0/864528068 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae88073a30 con 0x7fae88135100 2024-09-16T03:46:05.253 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.228+0000 7fae877fe640 1 --2- 172.21.15.42:0/864528068 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fae880a1330 0x7fae881cefa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:46:05.254 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.228+0000 7fae86ffd640 1 --2- 172.21.15.42:0/864528068 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fae88135100 0x7fae881345a0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fae700af040 tx=0x7fae700191e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:46:05.254 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.228+0000 7fae7ffff640 10 client.?.objecter ms_handle_connect 0x7fae88135100 2024-09-16T03:46:05.254 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.228+0000 7fae7ffff640 10 client.?.objecter resend_mon_ops 2024-09-16T03:46:05.254 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.228+0000 7fae7ffff640 1 -- 172.21.15.42:0/864528068 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae700b1040 con 0x7fae88135100 2024-09-16T03:46:05.254 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.228+0000 7fae7ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:46:05.254 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.228+0000 7fae7ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:46:05.254 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.228+0000 7fae7ffff640 10 monclient: dump: 2024-09-16T03:46:05.254 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:46:05.254 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:46:05.254 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:05.254 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:05.254 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:46:05.254 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:46:05.254 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:46:05.255 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:46:05.255 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:46:05.255 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:46:05.255 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.228+0000 7fae7ffff640 10 monclient: _finish_auth 0 2024-09-16T03:46:05.255 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.228+0000 7fae7ffff640 10 monclient: _check_auth_tickets 2024-09-16T03:46:05.255 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.228+0000 7fae7ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:46:05.255 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.228+0000 7fae7ffff640 1 -- 172.21.15.42:0/864528068 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fae700b5420 con 0x7fae88135100 2024-09-16T03:46:05.255 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.228+0000 7fae7ffff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:46:05.255 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.228+0000 7fae7ffff640 1 -- 172.21.15.42:0/864528068 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae700b57f0 con 0x7fae88135100 2024-09-16T03:46:05.255 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.228+0000 7fae7ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:46:05.255 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.228+0000 7fae7ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:46:05.255 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.228+0000 7fae7ffff640 10 monclient: dump: 2024-09-16T03:46:05.255 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:46:05.256 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:46:05.256 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:05.256 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:05.256 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:46:05.256 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:46:05.256 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:46:05.256 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:46:05.256 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:46:05.256 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:46:05.256 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.228+0000 7fae8df90640 5 monclient: authenticate success, global_id 744369 2024-09-16T03:46:05.256 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.228+0000 7fae8df90640 10 monclient: _renew_subs 2024-09-16T03:46:05.256 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.228+0000 7fae85ffb640 4 set_mon_vals no callback set 2024-09-16T03:46:05.256 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.228+0000 7fae8df90640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:46:05.256 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.228+0000 7fae8df90640 1 -- 172.21.15.42:0/864528068 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae880a5380 con 0x7fae88135100 2024-09-16T03:46:05.257 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.228+0000 7fae8df90640 10 client.744369.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:46:05.257 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.228+0000 7fae8df90640 10 monclient: _renew_subs 2024-09-16T03:46:05.257 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.228+0000 7fae8df90640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:46:05.257 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.228+0000 7fae8df90640 1 -- 172.21.15.42:0/864528068 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae881219c0 con 0x7fae88135100 2024-09-16T03:46:05.257 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.228+0000 7fae8df90640 1 librados: init done 2024-09-16T03:46:05.257 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.232+0000 7fae8df90640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:46:05.257 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.232+0000 7fae8df90640 1 --2- 172.21.15.42:0/864528068 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fae8806dac0 0x7fae8811e530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:46:05.257 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.232+0000 7fae8df90640 10 monclient(hunting): start opening mon connection 2024-09-16T03:46:05.257 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.232+0000 7fae8df90640 1 -- 172.21.15.42:0/864528068 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fae88134ef0 con 0x7fae8806dac0 2024-09-16T03:46:05.257 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.232+0000 7fae7ffff640 1 -- 172.21.15.42:0/864528068 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fae700b5990 con 0x7fae88135100 2024-09-16T03:46:05.257 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.232+0000 7fae877fe640 1 --2- 172.21.15.42:0/864528068 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fae8806dac0 0x7fae8811e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:46:05.257 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.232+0000 7fae877fe640 10 monclient: get_auth_request con 0x7fae8806dac0 auth_method 0 2024-09-16T03:46:05.257 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.232+0000 7fae877fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:46:05.257 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.232+0000 7fae877fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:46:05.258 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.232+0000 7fae877fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:46:05.258 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.232+0000 7fae877fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:46:05.258 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.232+0000 7fae877fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:46:05.258 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.232+0000 7fae7ffff640 1 --2- 172.21.15.42:0/864528068 >> v2:172.21.15.154:6812/254845342 conn(0x7fae68037840 0x7fae68039d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:46:05.258 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.232+0000 7fae877fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:46:05.258 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.232+0000 7fae7ffff640 1 -- 172.21.15.42:0/864528068 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(119..119 src has 1..119) ==== 8177+0+0 (secure 0 0 0) 0x7fae700e3600 con 0x7fae88135100 2024-09-16T03:46:05.258 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.232+0000 7fae7ffff640 10 client.744369.objecter ms_dispatch 0x7fae88000ba0 osd_map(119..119 src has 1..119) 2024-09-16T03:46:05.258 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.232+0000 7fae7ffff640 3 client.744369.objecter handle_osd_map got epochs [119,119] > 0 2024-09-16T03:46:05.258 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.232+0000 7fae7ffff640 3 client.744369.objecter handle_osd_map decoding full epoch 119 2024-09-16T03:46:05.259 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.232+0000 7fae877fe640 10 monclient(hunting): handle_auth_done global_id 744372 payload 824 2024-09-16T03:46:05.259 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.232+0000 7fae877fe640 1 -- 172.21.15.42:0/864528068 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fae8811fc90 con 0x7fae8806dac0 2024-09-16T03:46:05.259 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.232+0000 7fae7ffff640 20 client.744369.objecter dump_active .. 0 homeless 2024-09-16T03:46:05.259 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.232+0000 7fae87fff640 1 --2- 172.21.15.42:0/864528068 >> v2:172.21.15.154:6812/254845342 conn(0x7fae68037840 0x7fae68039d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:46:05.259 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.232+0000 7fae87fff640 10 monclient: get_auth_request con 0x7fae68037840 auth_method 0 2024-09-16T03:46:05.259 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.232+0000 7fae7ffff640 10 client.744369.objecter ms_handle_connect 0x7fae8806dac0 2024-09-16T03:46:05.259 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.232+0000 7fae7ffff640 10 client.744369.objecter resend_mon_ops 2024-09-16T03:46:05.259 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.232+0000 7fae877fe640 1 --2- 172.21.15.42:0/864528068 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fae8806dac0 0x7fae8811e530 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fae80007560 tx=0x7fae80007a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:46:05.259 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.232+0000 7fae7ffff640 1 -- 172.21.15.42:0/864528068 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae8001b070 con 0x7fae8806dac0 2024-09-16T03:46:05.259 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.232+0000 7fae7ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:46:05.259 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.232+0000 7fae7ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:46:05.259 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.232+0000 7fae7ffff640 10 monclient: dump: 2024-09-16T03:46:05.259 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:46:05.259 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:46:05.260 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:05.260 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:05.260 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:46:05.260 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:46:05.260 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:46:05.260 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:46:05.260 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:46:05.260 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:46:05.260 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.232+0000 7fae7ffff640 1 -- 172.21.15.42:0/864528068 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fae80017910 con 0x7fae8806dac0 2024-09-16T03:46:05.260 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.232+0000 7fae7ffff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:46:05.261 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.232+0000 7fae7ffff640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:46:05.261 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.232+0000 7fae7ffff640 1 -- 172.21.15.42:0/864528068 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fae8806dac0 msgr2=0x7fae8811e530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:46:05.261 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.232+0000 7fae7ffff640 1 --2- 172.21.15.42:0/864528068 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fae8806dac0 0x7fae8811e530 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fae80007560 tx=0x7fae80007a20 comp rx=0 tx=0).stop 2024-09-16T03:46:05.261 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.232+0000 7fae7ffff640 1 -- 172.21.15.42:0/864528068 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fae8806dac0 msgr2=0x7fae8811e530 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:46:05.261 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.232+0000 7fae7ffff640 1 --2- 172.21.15.42:0/864528068 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fae8806dac0 0x7fae8811e530 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:46:05.261 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.232+0000 7fae7ffff640 10 client.744369.objecter ms_handle_connect 0x7fae68037840 2024-09-16T03:46:05.261 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.232+0000 7fae87fff640 1 --2- 172.21.15.42:0/864528068 >> v2:172.21.15.154:6812/254845342 conn(0x7fae68037840 0x7fae68039d00 secure :-1 s=READY pgs=3396 cs=0 l=1 rev1=1 crypto rx=0x7fae7800d8d0 tx=0x7fae78008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:46:05.261 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.236+0000 7fae8df90640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:46:05.261 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.236+0000 7fae8df90640 1 --2- 172.21.15.42:0/864528068 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fae88121e70 0x7fae8811ea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:46:05.261 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.236+0000 7fae8df90640 10 monclient(hunting): start opening mon connection 2024-09-16T03:46:05.261 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.236+0000 7fae8df90640 1 -- 172.21.15.42:0/864528068 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fae88135550 con 0x7fae88121e70 2024-09-16T03:46:05.261 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.236+0000 7fae877fe640 1 --2- 172.21.15.42:0/864528068 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fae88121e70 0x7fae8811ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:46:05.261 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.236+0000 7fae877fe640 10 monclient: get_auth_request con 0x7fae88121e70 auth_method 0 2024-09-16T03:46:05.262 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.236+0000 7fae877fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:46:05.262 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.236+0000 7fae877fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:46:05.262 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.236+0000 7fae877fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:46:05.262 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.236+0000 7fae877fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:46:05.262 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.236+0000 7fae877fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:46:05.262 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.236+0000 7fae877fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:46:05.262 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.240+0000 7fae877fe640 10 monclient(hunting): handle_auth_done global_id 744375 payload 824 2024-09-16T03:46:05.262 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.240+0000 7fae877fe640 1 -- 172.21.15.42:0/864528068 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fae88120570 con 0x7fae88121e70 2024-09-16T03:46:05.263 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.240+0000 7fae877fe640 1 --2- 172.21.15.42:0/864528068 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fae88121e70 0x7fae8811ea70 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fae8000efd0 tx=0x7fae80004ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:46:05.263 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.240+0000 7fae7ffff640 10 client.744369.objecter ms_handle_connect 0x7fae88121e70 2024-09-16T03:46:05.263 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.240+0000 7fae7ffff640 10 client.744369.objecter resend_mon_ops 2024-09-16T03:46:05.263 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.240+0000 7fae7ffff640 1 -- 172.21.15.42:0/864528068 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae8002e020 con 0x7fae88121e70 2024-09-16T03:46:05.264 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.240+0000 7fae7ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:46:05.264 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.240+0000 7fae7ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:46:05.264 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.240+0000 7fae7ffff640 10 monclient: dump: 2024-09-16T03:46:05.264 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:46:05.264 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:46:05.264 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:05.264 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:05.264 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:46:05.264 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:46:05.264 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:46:05.264 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:46:05.264 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:46:05.264 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:46:05.264 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.240+0000 7fae7ffff640 1 -- 172.21.15.42:0/864528068 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4713 (secure 0 0 0) 0x7fae800138f0 con 0x7fae88121e70 2024-09-16T03:46:05.265 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.240+0000 7fae7ffff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:46:05.265 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.240+0000 7fae7ffff640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:46:05.265 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.240+0000 7fae7ffff640 1 -- 172.21.15.42:0/864528068 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fae88121e70 msgr2=0x7fae8811ea70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:46:05.265 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.240+0000 7fae7ffff640 1 --2- 172.21.15.42:0/864528068 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fae88121e70 0x7fae8811ea70 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fae8000efd0 tx=0x7fae80004ac0 comp rx=0 tx=0).stop 2024-09-16T03:46:05.265 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.240+0000 7fae7ffff640 1 -- 172.21.15.42:0/864528068 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fae88121e70 msgr2=0x7fae8811ea70 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:46:05.265 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.240+0000 7fae7ffff640 1 --2- 172.21.15.42:0/864528068 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fae88121e70 0x7fae8811ea70 secure :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fae8000efd0 tx=0x7fae80004ac0 comp rx=0 tx=0).stop 2024-09-16T03:46:05.265 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:46:05.265 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:46:05.266 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:46:05.266 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "leader", 2024-09-16T03:46:05.266 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 34, 2024-09-16T03:46:05.267 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 52611, 2024-09-16T03:46:05.267 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:46:05.267 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:46:05.267 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T03:46:05.267 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T03:46:05.267 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:46:05.267 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 2, 2024-09-16T03:46:05.268 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:46:05.268 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:46:05.268 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:46:05.268 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:46:05.268 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:46:05.268 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:46:05.268 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:46:05.268 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:46:05.269 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:46:05.269 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:46:05.269 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:46:05.269 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:46:05.269 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:46:05.269 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:46:05.269 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:46:05.269 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:46:05.269 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:46:05.269 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:46:05.269 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:46:05.270 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:46:05.270 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:46:05.270 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:46:05.270 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:46:05.270 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:46:05.270 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:46:05.270 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:46:05.270 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:46:05.270 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:46:05.270 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:46:05.270 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:46:05.270 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:46:05.271 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:46:05.271 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:46:05.271 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:46:05.271 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:46:05.271 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:46:05.271 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:46:05.271 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:46:05.271 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:46:05.271 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:46:05.271 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:46:05.272 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:46:05.272 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:46:05.272 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:46:05.272 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:46:05.272 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:46:05.272 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:46:05.272 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:46:05.272 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:46:05.272 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:46:05.272 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:46:05.273 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:46:05.273 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:46:05.273 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:46:05.273 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:46:05.273 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:46:05.273 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:46:05.273 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:46:05.273 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:46:05.273 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:46:05.273 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:46:05.273 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:46:05.273 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:46:05.274 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:46:05.274 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:46:05.274 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:46:05.274 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:46:05.274 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:46:05.274 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:46:05.274 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:46:05.274 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:46:05.274 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:46:05.274 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:46:05.274 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:46:05.274 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:46:05.275 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:46:05.275 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:46:05.275 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:46:05.275 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:46:05.275 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:46:05.275 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:46:05.275 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:46:05.275 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:46:05.275 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:46:05.275 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:46:05.275 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:46:05.275 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:46:05.276 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:46:05.276 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:46:05.276 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:46:05.276 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:46:05.276 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:46:05.276 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:46:05.276 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:46:05.276 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:46:05.276 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:46:05.276 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:46:05.276 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:46:05.277 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:46:05.277 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:46:05.277 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:46:05.277 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:46:05.277 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:46:05.277 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:46:05.277 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:46:05.277 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:46:05.277 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:46:05.277 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:46:05.277 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:46:05.277 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:46:05.278 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:46:05.278 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:46:05.278 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:46:05.278 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:46:05.278 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:46:05.278 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:46:05.278 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:46:05.278 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:46:05.278 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:46:05.278 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:46:05.278 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:46:05.279 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:46:05.279 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:46:05.279 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:46:05.279 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:46:05.279 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:46:05.279 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:46:05.279 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:46:05.279 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:46:05.279 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:46:05.279 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:46:05.279 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:46:05.280 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:46:05.280 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:46:05.280 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:46:05.280 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:46:05.280 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:46:05.280 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:46:05.280 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:46:05.280 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:46:05.280 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:46:05.280 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T03:46:05.280 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:46:05.280 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:46:05.281 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:46:05.281 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:46:05.281 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:46:05.281 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:46:05.281 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:46:05.281 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:46:05.281 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:46:05.281 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:46:05.281 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 4 2024-09-16T03:46:05.281 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:46:05.281 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:46:05.281 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:46:05.282 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:46:05.282 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:46:05.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.240+0000 7fae8df90640 10 librados: watch_flush enter 2024-09-16T03:46:05.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.240+0000 7fae8df90640 10 librados: watch_flush exit 2024-09-16T03:46:05.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.240+0000 7fae8df90640 20 client.744369.objecter shutdown clearing up homeless session... 2024-09-16T03:46:05.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.240+0000 7fae8df90640 10 client.744369.objecter successfully canceled tick 2024-09-16T03:46:05.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.240+0000 7fae8df90640 1 -- 172.21.15.42:0/864528068 >> v2:172.21.15.154:6812/254845342 conn(0x7fae68037840 msgr2=0x7fae68039d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:46:05.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.240+0000 7fae8df90640 1 --2- 172.21.15.42:0/864528068 >> v2:172.21.15.154:6812/254845342 conn(0x7fae68037840 0x7fae68039d00 secure :-1 s=READY pgs=3396 cs=0 l=1 rev1=1 crypto rx=0x7fae7800d8d0 tx=0x7fae78008040 comp rx=0 tx=0).stop 2024-09-16T03:46:05.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.240+0000 7fae8df90640 10 monclient: shutdown 2024-09-16T03:46:05.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.240+0000 7fae8df90640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:46:05.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.240+0000 7fae8df90640 1 -- 172.21.15.42:0/864528068 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fae88135100 msgr2=0x7fae881345a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:46:05.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.240+0000 7fae8df90640 1 --2- 172.21.15.42:0/864528068 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fae88135100 0x7fae881345a0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fae700af040 tx=0x7fae700191e0 comp rx=0 tx=0).stop 2024-09-16T03:46:05.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.240+0000 7fae87fff640 1 -- 172.21.15.42:0/864528068 reap_dead start 2024-09-16T03:46:05.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.240+0000 7fae87fff640 1 -- 172.21.15.42:0/864528068 reap_dead start 2024-09-16T03:46:05.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.240+0000 7fae8df90640 1 -- 172.21.15.42:0/864528068 shutdown_connections 2024-09-16T03:46:05.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.240+0000 7fae8df90640 1 -- 172.21.15.42:0/864528068 >> 172.21.15.42:0/864528068 conn(0x7fae880a2420 msgr2=0x7fae880a3720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:46:05.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.240+0000 7fae8df90640 1 -- 172.21.15.42:0/864528068 shutdown_connections 2024-09-16T03:46:05.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.240+0000 7fae8df90640 1 -- 172.21.15.42:0/864528068 wait complete. 2024-09-16T03:46:05.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.240+0000 7fae8df90640 1 librados: shutdown 2024-09-16T03:46:05.285 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:46:05.285 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T03:46:05.286 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T03:46:05.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.396+0000 7fd1dce4b640 10 monclient: get_monmap_and_config 2024-09-16T03:46:05.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.396+0000 7fd1dce4b640 10 monclient: build_initial_monmap 2024-09-16T03:46:05.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.396+0000 7fd1dce4b640 10 monclient: monmap: 2024-09-16T03:46:05.403 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:46:05.403 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:46:05.403 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:46:05.402141+0000 2024-09-16T03:46:05.403 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:46:05.402141+0000 2024-09-16T03:46:05.403 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:46:05.403 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:46:05.403 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:46:05.403 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:46:05.404 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:46:05.404 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:46:05.404 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:46:05.404 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:46:05.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.396+0000 7fd1dce4b640 1 Processor -- start 2024-09-16T03:46:05.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.396+0000 7fd1dce4b640 1 -- start start 2024-09-16T03:46:05.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.396+0000 7fd1dce4b640 10 monclient: init 2024-09-16T03:46:05.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.396+0000 7fd1dce4b640 10 monclient: _reopen_session rank -1 2024-09-16T03:46:05.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.396+0000 7fd1dce4b640 10 monclient: _add_conns ranks=[2,4,0,1,3] 2024-09-16T03:46:05.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.396+0000 7fd1dce4b640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd1d80a1330 0x7fd1d80a1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:46:05.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.396+0000 7fd1dce4b640 10 monclient(hunting): picked mon.noname-b con 0x7fd1d80a1330 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:46:05.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.396+0000 7fd1dce4b640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fd1d809c920 0x7fd1d80a1c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:46:05.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.396+0000 7fd1dce4b640 10 monclient(hunting): picked mon.noname-c con 0x7fd1d809c920 addr v2:172.21.15.154:3300/0 2024-09-16T03:46:05.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.396+0000 7fd1dce4b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd1d8072d40 addr v1:172.21.15.42:6789/0 2024-09-16T03:46:05.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.396+0000 7fd1dce4b640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd1d8072350 con 0x7fd1d8072d40 2024-09-16T03:46:05.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.396+0000 7fd1dce4b640 10 monclient(hunting): start opening mon connection 2024-09-16T03:46:05.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.396+0000 7fd1dce4b640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fd1d8072b50 con 0x7fd1d809c920 2024-09-16T03:46:05.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.396+0000 7fd1dce4b640 10 monclient(hunting): start opening mon connection 2024-09-16T03:46:05.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.396+0000 7fd1dce4b640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fd1d80a2190 con 0x7fd1d80a1330 2024-09-16T03:46:05.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.396+0000 7fd1dce4b640 10 monclient(hunting): _renew_subs 2024-09-16T03:46:05.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.396+0000 7fd1dce4b640 10 monclient(hunting): authenticate will time out at 1212.371261s 2024-09-16T03:46:05.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.396+0000 7fd1d659c640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd1d80a1330 0x7fd1d80a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:46:05.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.396+0000 7fd1d6d9d640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fd1d8072d40 0x7fd1d8137120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:48156/0 (socket says 172.21.15.42:48156) 2024-09-16T03:46:05.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.396+0000 7fd1d6d9d640 1 -- 172.21.15.42:0/3984495840 learned_addr learned my addr 172.21.15.42:0/3984495840 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:46:05.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.396+0000 7fd1d659c640 10 monclient(hunting): get_auth_request con 0x7fd1d80a1330 auth_method 0 2024-09-16T03:46:05.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.396+0000 7fd1d659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:46:05.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.396+0000 7fd1d659c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:46:05.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.396+0000 7fd1d659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:46:05.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.396+0000 7fd1d659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:46:05.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.396+0000 7fd1d659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:46:05.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.396+0000 7fd1d659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:46:05.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.396+0000 7fd1d5d9b640 1 --2- 172.21.15.42:0/3984495840 >> v2:172.21.15.154:3300/0 conn(0x7fd1d809c920 0x7fd1d80a1c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:46:05.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.396+0000 7fd1d5d9b640 10 monclient(hunting): get_auth_request con 0x7fd1d809c920 auth_method 0 2024-09-16T03:46:05.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.396+0000 7fd1d5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:46:05.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.396+0000 7fd1d5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:46:05.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.396+0000 7fd1d5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:46:05.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.396+0000 7fd1d559a640 1 -- 172.21.15.42:0/3984495840 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2853748757 0 0) 0x7fd1d8072350 con 0x7fd1d8072d40 2024-09-16T03:46:05.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.396+0000 7fd1d559a640 10 monclient(hunting): _negotiate 2024-09-16T03:46:05.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.396+0000 7fd1d559a640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:46:05.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.396+0000 7fd1d559a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:46:05.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.396+0000 7fd1d559a640 10 monclient(hunting): my global_id is 744378 2024-09-16T03:46:05.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.396+0000 7fd1d559a640 1 -- 172.21.15.42:0/3984495840 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd1c4003530 con 0x7fd1d8072d40 2024-09-16T03:46:05.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.396+0000 7fd1d659c640 10 monclient(hunting): handle_auth_done global_id 13520 payload 274 2024-09-16T03:46:05.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.396+0000 7fd1d659c640 1 -- 172.21.15.42:0/3984495840 >> v2:172.21.15.154:3300/0 conn(0x7fd1d809c920 msgr2=0x7fd1d80a1c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:46:05.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.396+0000 7fd1d659c640 1 --2- 172.21.15.42:0/3984495840 >> v2:172.21.15.154:3300/0 conn(0x7fd1d809c920 0x7fd1d80a1c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:46:05.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.396+0000 7fd1d659c640 1 -- 172.21.15.42:0/3984495840 >> v1:172.21.15.42:6789/0 conn(0x7fd1d8072d40 legacy=0x7fd1d8137120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:46:05.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.396+0000 7fd1d659c640 10 monclient: _finish_hunting 0 2024-09-16T03:46:05.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.396+0000 7fd1d659c640 1 monclient: found mon.noname-b 2024-09-16T03:46:05.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.396+0000 7fd1d659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:46:05.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.396+0000 7fd1d659c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T03:46:05.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.396+0000 7fd1d659c640 1 -- 172.21.15.42:0/3984495840 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1d809bc60 con 0x7fd1d80a1330 2024-09-16T03:46:05.416 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.400+0000 7fd1d659c640 1 --2- 172.21.15.42:0/3984495840 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd1d80a1330 0x7fd1d80a1710 secure :-1 s=READY pgs=2412 cs=0 l=1 rev1=1 crypto rx=0x7fd1c80097c0 tx=0x7fd1c801c3f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=569c7eab9e0fbf83 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:46:05.416 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.400+0000 7fd1d559a640 1 -- 172.21.15.42:0/3984495840 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1c80b0070 con 0x7fd1d80a1330 2024-09-16T03:46:05.416 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.400+0000 7fd1d559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:46:05.416 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.400+0000 7fd1d559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:46:05.416 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.400+0000 7fd1d559a640 10 monclient: dump: 2024-09-16T03:46:05.416 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:46:05.417 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:46:05.417 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:05.417 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:05.417 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:46:05.417 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:46:05.417 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:46:05.417 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:46:05.417 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:46:05.417 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:46:05.417 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.400+0000 7fd1d559a640 10 monclient: _finish_auth 0 2024-09-16T03:46:05.417 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.400+0000 7fd1d559a640 10 monclient: _check_auth_tickets 2024-09-16T03:46:05.417 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.400+0000 7fd1d559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:46:05.418 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.400+0000 7fd1dce4b640 5 monclient: authenticate success, global_id 13520 2024-09-16T03:46:05.418 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.400+0000 7fd1dce4b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:46:05.418 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.400+0000 7fd1d559a640 1 -- 172.21.15.42:0/3984495840 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd1c80ab600 con 0x7fd1d80a1330 2024-09-16T03:46:05.418 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.400+0000 7fd1d559a640 10 monclient: handle_config config(0 keys) 2024-09-16T03:46:05.418 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.400+0000 7fd1d559a640 1 -- 172.21.15.42:0/3984495840 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1c80ab9b0 con 0x7fd1d80a1330 2024-09-16T03:46:05.418 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.400+0000 7fd1dce4b640 10 monclient: get_monmap_and_config success 2024-09-16T03:46:05.418 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.400+0000 7fd1dce4b640 4 set_mon_vals no callback set 2024-09-16T03:46:05.418 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.400+0000 7fd1d559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:46:05.418 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.400+0000 7fd1d559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:46:05.418 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.400+0000 7fd1d559a640 10 monclient: dump: 2024-09-16T03:46:05.418 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:46:05.419 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:46:05.419 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:05.419 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:05.419 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:46:05.419 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:46:05.419 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:46:05.419 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:46:05.419 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:46:05.419 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:46:05.419 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.400+0000 7fd1dce4b640 10 monclient: shutdown 2024-09-16T03:46:05.419 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.400+0000 7fd1dce4b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:46:05.419 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.400+0000 7fd1dce4b640 1 -- 172.21.15.42:0/3984495840 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd1d80a1330 msgr2=0x7fd1d80a1710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:46:05.419 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.400+0000 7fd1dce4b640 1 --2- 172.21.15.42:0/3984495840 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd1d80a1330 0x7fd1d80a1710 secure :-1 s=READY pgs=2412 cs=0 l=1 rev1=1 crypto rx=0x7fd1c80097c0 tx=0x7fd1c801c3f0 comp rx=0 tx=0).stop 2024-09-16T03:46:05.419 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.400+0000 7fd1dce4b640 1 -- 172.21.15.42:0/3984495840 shutdown_connections 2024-09-16T03:46:05.420 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.400+0000 7fd1dce4b640 1 --2- 172.21.15.42:0/3984495840 >> v2:172.21.15.154:3300/0 conn(0x7fd1d809c920 0x7fd1d80a1c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:46:05.420 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.400+0000 7fd1dce4b640 1 --2- 172.21.15.42:0/3984495840 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd1d80a1330 0x7fd1d80a1710 unknown :-1 s=CLOSED pgs=2412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:46:05.420 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.400+0000 7fd1dce4b640 1 -- 172.21.15.42:0/3984495840 >> 172.21.15.42:0/3984495840 conn(0x7fd1d80a2420 msgr2=0x7fd1d80a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:46:05.420 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.400+0000 7fd1dce4b640 1 -- 172.21.15.42:0/3984495840 shutdown_connections 2024-09-16T03:46:05.420 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.400+0000 7fd1dce4b640 1 -- 172.21.15.42:0/3984495840 wait complete. 2024-09-16T03:46:05.420 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.404+0000 7fd1dce4b640 10 monclient: build_initial_monmap 2024-09-16T03:46:05.420 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.404+0000 7fd1dce4b640 10 monclient: monmap: 2024-09-16T03:46:05.420 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:46:05.420 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:46:05.420 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:46:05.421 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:46:05.421 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:46:05.421 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:46:05.421 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:46:05.421 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:46:05.421 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:46:05.421 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:46:05.421 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.404+0000 7fd1dce4b640 1 librados: starting msgr at 2024-09-16T03:46:05.421 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.404+0000 7fd1dce4b640 1 librados: starting objecter 2024-09-16T03:46:05.421 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.404+0000 7fd1dce4b640 1 Processor -- start 2024-09-16T03:46:05.421 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.404+0000 7fd1dce4b640 1 -- start start 2024-09-16T03:46:05.421 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.404+0000 7fd1dce4b640 1 librados: setting wanted keys 2024-09-16T03:46:05.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.404+0000 7fd1dce4b640 1 librados: calling monclient init 2024-09-16T03:46:05.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.404+0000 7fd1dce4b640 10 monclient: init 2024-09-16T03:46:05.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.404+0000 7fd1dce4b640 10 monclient: _reopen_session rank -1 2024-09-16T03:46:05.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.404+0000 7fd1dce4b640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:46:05.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.404+0000 7fd1dce4b640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd1d8072d40 0x7fd1d8135e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:46:05.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.404+0000 7fd1dce4b640 10 monclient(hunting): picked mon.noname-c con 0x7fd1d8072d40 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:46:05.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.404+0000 7fd1dce4b640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd1d809c920 0x7fd1d8136380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:46:05.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.404+0000 7fd1dce4b640 10 monclient(hunting): picked mon.noname-a con 0x7fd1d809c920 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:46:05.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.404+0000 7fd1dce4b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd1d80a1330 0x7fd1d811c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:46:05.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.404+0000 7fd1dce4b640 10 monclient(hunting): picked mon.noname-b con 0x7fd1d80a1330 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:46:05.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.404+0000 7fd1dce4b640 10 monclient(hunting): start opening mon connection 2024-09-16T03:46:05.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.404+0000 7fd1dce4b640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fd1d8141a60 con 0x7fd1d809c920 2024-09-16T03:46:05.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.404+0000 7fd1dce4b640 10 monclient(hunting): start opening mon connection 2024-09-16T03:46:05.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.404+0000 7fd1dce4b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd1d81418e0 con 0x7fd1d80a1330 2024-09-16T03:46:05.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.404+0000 7fd1dce4b640 10 monclient(hunting): start opening mon connection 2024-09-16T03:46:05.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.404+0000 7fd1dce4b640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fd1d8141be0 con 0x7fd1d8072d40 2024-09-16T03:46:05.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.404+0000 7fd1dce4b640 10 monclient(hunting): _renew_subs 2024-09-16T03:46:05.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.404+0000 7fd1dce4b640 10 monclient(hunting): authenticate will time out at 1212.377364s 2024-09-16T03:46:05.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.404+0000 7fd1d659c640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd1d8072d40 0x7fd1d8135e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:46:05.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.404+0000 7fd1d659c640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd1d8072d40 0x7fd1d8135e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:51646/0 (socket says 172.21.15.42:51646) 2024-09-16T03:46:05.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.404+0000 7fd1d659c640 1 -- 172.21.15.42:0/1361267842 learned_addr learned my addr 172.21.15.42:0/1361267842 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:46:05.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.404+0000 7fd1d659c640 10 monclient(hunting): get_auth_request con 0x7fd1d8072d40 auth_method 0 2024-09-16T03:46:05.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.404+0000 7fd1d659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:46:05.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.404+0000 7fd1d659c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:46:05.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.404+0000 7fd1d659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:46:05.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.404+0000 7fd1d5d9b640 1 --2- 172.21.15.42:0/1361267842 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd1d809c920 0x7fd1d8136380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:46:05.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.404+0000 7fd1d5d9b640 10 monclient(hunting): get_auth_request con 0x7fd1d809c920 auth_method 0 2024-09-16T03:46:05.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.404+0000 7fd1d5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:46:05.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.404+0000 7fd1d5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:46:05.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.404+0000 7fd1d5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:46:05.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.404+0000 7fd1d659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:46:05.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.404+0000 7fd1d659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:46:05.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.404+0000 7fd1d659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:46:05.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.404+0000 7fd1d5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:46:05.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.404+0000 7fd1d5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:46:05.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.404+0000 7fd1d5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:46:05.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.404+0000 7fd1d6d9d640 1 --2- 172.21.15.42:0/1361267842 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd1d80a1330 0x7fd1d811c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:46:05.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.404+0000 7fd1d6d9d640 10 monclient(hunting): get_auth_request con 0x7fd1d80a1330 auth_method 0 2024-09-16T03:46:05.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.404+0000 7fd1d6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:46:05.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.404+0000 7fd1d6d9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:46:05.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.404+0000 7fd1d6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:46:05.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.404+0000 7fd1d659c640 10 monclient(hunting): handle_auth_done global_id 13523 payload 824 2024-09-16T03:46:05.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.404+0000 7fd1d659c640 1 -- 172.21.15.42:0/1361267842 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd1d80a1330 msgr2=0x7fd1d811c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:46:05.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.404+0000 7fd1d659c640 1 --2- 172.21.15.42:0/1361267842 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd1d80a1330 0x7fd1d811c9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:46:05.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.404+0000 7fd1d659c640 1 -- 172.21.15.42:0/1361267842 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd1d809c920 msgr2=0x7fd1d8136380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:46:05.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.404+0000 7fd1d659c640 1 --2- 172.21.15.42:0/1361267842 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd1d809c920 0x7fd1d8136380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:46:05.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.404+0000 7fd1d659c640 10 monclient: _finish_hunting 0 2024-09-16T03:46:05.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.404+0000 7fd1d659c640 1 monclient: found mon.noname-c 2024-09-16T03:46:05.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.404+0000 7fd1d659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:46:05.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.404+0000 7fd1d659c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:46:05.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.404+0000 7fd1d659c640 1 -- 172.21.15.42:0/1361267842 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1d811ef30 con 0x7fd1d8072d40 2024-09-16T03:46:05.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.404+0000 7fd1d5d9b640 1 --2- 172.21.15.42:0/1361267842 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd1d809c920 0x7fd1d8136380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:46:05.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.404+0000 7fd1d659c640 1 --2- 172.21.15.42:0/1361267842 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd1d8072d40 0x7fd1d8135e40 secure :-1 s=READY pgs=2413 cs=0 l=1 rev1=1 crypto rx=0x7fd1c80097c0 tx=0x7fd1c8019190 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:46:05.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.404+0000 7fd1ceffd640 10 client.?.objecter ms_handle_connect 0x7fd1d8072d40 2024-09-16T03:46:05.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.404+0000 7fd1ceffd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:46:05.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.404+0000 7fd1ceffd640 1 -- 172.21.15.42:0/1361267842 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1c80b0050 con 0x7fd1d8072d40 2024-09-16T03:46:05.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.404+0000 7fd1ceffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:46:05.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.404+0000 7fd1ceffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:46:05.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.404+0000 7fd1ceffd640 10 monclient: dump: 2024-09-16T03:46:05.426 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:46:05.426 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:46:05.426 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:05.426 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:05.426 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:46:05.426 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:46:05.426 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:46:05.427 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:46:05.427 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:46:05.427 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:46:05.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.404+0000 7fd1ceffd640 10 monclient: _finish_auth 0 2024-09-16T03:46:05.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.404+0000 7fd1ceffd640 10 monclient: _check_auth_tickets 2024-09-16T03:46:05.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.404+0000 7fd1ceffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:46:05.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.404+0000 7fd1ceffd640 1 -- 172.21.15.42:0/1361267842 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd1c80bc070 con 0x7fd1d8072d40 2024-09-16T03:46:05.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.404+0000 7fd1ceffd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:46:05.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.404+0000 7fd1ceffd640 1 -- 172.21.15.42:0/1361267842 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1c80abde0 con 0x7fd1d8072d40 2024-09-16T03:46:05.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.404+0000 7fd1ceffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:46:05.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.404+0000 7fd1ceffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:46:05.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.404+0000 7fd1ceffd640 10 monclient: dump: 2024-09-16T03:46:05.427 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:46:05.427 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:46:05.428 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:05.428 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:05.428 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:46:05.428 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:46:05.428 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:46:05.428 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:46:05.428 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:46:05.428 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:46:05.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.404+0000 7fd1dce4b640 5 monclient: authenticate success, global_id 13523 2024-09-16T03:46:05.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.404+0000 7fd1d4d99640 4 set_mon_vals no callback set 2024-09-16T03:46:05.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.404+0000 7fd1dce4b640 10 monclient: _renew_subs 2024-09-16T03:46:05.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.404+0000 7fd1dce4b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:46:05.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.404+0000 7fd1dce4b640 1 -- 172.21.15.42:0/1361267842 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1d8120940 con 0x7fd1d8072d40 2024-09-16T03:46:05.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.404+0000 7fd1dce4b640 10 client.13523.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:46:05.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.404+0000 7fd1dce4b640 10 monclient: _renew_subs 2024-09-16T03:46:05.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.404+0000 7fd1dce4b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:46:05.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.404+0000 7fd1dce4b640 1 -- 172.21.15.42:0/1361267842 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1d80a52e0 con 0x7fd1d8072d40 2024-09-16T03:46:05.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.404+0000 7fd1dce4b640 1 librados: init done 2024-09-16T03:46:05.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.404+0000 7fd1dce4b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:46:05.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.404+0000 7fd1ceffd640 1 -- 172.21.15.42:0/1361267842 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fd1c80ab4c0 con 0x7fd1d8072d40 2024-09-16T03:46:05.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.404+0000 7fd1dce4b640 1 --2- 172.21.15.42:0/1361267842 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd1d8120210 0x7fd1d811cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:46:05.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.404+0000 7fd1dce4b640 10 monclient(hunting): start opening mon connection 2024-09-16T03:46:05.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.404+0000 7fd1dce4b640 1 -- 172.21.15.42:0/1361267842 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fd1d811f3d0 con 0x7fd1d8120210 2024-09-16T03:46:05.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.404+0000 7fd1d5d9b640 1 --2- 172.21.15.42:0/1361267842 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd1d8120210 0x7fd1d811cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:46:05.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.404+0000 7fd1ceffd640 1 --2- 172.21.15.42:0/1361267842 >> v2:172.21.15.154:6812/254845342 conn(0x7fd1b8037840 0x7fd1b8039d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:46:05.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.404+0000 7fd1d5d9b640 10 monclient: get_auth_request con 0x7fd1d8120210 auth_method 0 2024-09-16T03:46:05.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.404+0000 7fd1d5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:46:05.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.404+0000 7fd1d5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:46:05.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.404+0000 7fd1d5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:46:05.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.404+0000 7fd1ceffd640 1 -- 172.21.15.42:0/1361267842 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(119..119 src has 1..119) ==== 8177+0+0 (secure 0 0 0) 0x7fd1c80e3f00 con 0x7fd1d8072d40 2024-09-16T03:46:05.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.404+0000 7fd1ceffd640 10 client.13523.objecter ms_dispatch 0x7fd1d8000ba0 osd_map(119..119 src has 1..119) 2024-09-16T03:46:05.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.404+0000 7fd1ceffd640 3 client.13523.objecter handle_osd_map got epochs [119,119] > 0 2024-09-16T03:46:05.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.404+0000 7fd1ceffd640 3 client.13523.objecter handle_osd_map decoding full epoch 119 2024-09-16T03:46:05.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.404+0000 7fd1d5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:46:05.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.404+0000 7fd1d5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:46:05.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.404+0000 7fd1d5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:46:05.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.404+0000 7fd1ceffd640 20 client.13523.objecter dump_active .. 0 homeless 2024-09-16T03:46:05.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.404+0000 7fd1d5d9b640 10 monclient(hunting): handle_auth_done global_id 13526 payload 824 2024-09-16T03:46:05.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.404+0000 7fd1d5d9b640 1 -- 172.21.15.42:0/1361267842 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd1d8073a30 con 0x7fd1d8120210 2024-09-16T03:46:05.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.404+0000 7fd1d6d9d640 1 --2- 172.21.15.42:0/1361267842 >> v2:172.21.15.154:6812/254845342 conn(0x7fd1b8037840 0x7fd1b8039d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:46:05.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.408+0000 7fd1d6d9d640 10 monclient: get_auth_request con 0x7fd1b8037840 auth_method 0 2024-09-16T03:46:05.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.408+0000 7fd1d5d9b640 1 --2- 172.21.15.42:0/1361267842 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd1d8120210 0x7fd1d811cef0 secure :-1 s=READY pgs=2414 cs=0 l=1 rev1=1 crypto rx=0x7fd1d811e250 tx=0x7fd1c000e1d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:46:05.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.408+0000 7fd1ceffd640 10 client.13523.objecter ms_handle_connect 0x7fd1d8120210 2024-09-16T03:46:05.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.408+0000 7fd1ceffd640 10 client.13523.objecter resend_mon_ops 2024-09-16T03:46:05.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.408+0000 7fd1ceffd640 1 -- 172.21.15.42:0/1361267842 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1c0018020 con 0x7fd1d8120210 2024-09-16T03:46:05.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.408+0000 7fd1ceffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:46:05.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.408+0000 7fd1ceffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:46:05.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.408+0000 7fd1ceffd640 10 monclient: dump: 2024-09-16T03:46:05.431 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:46:05.431 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:46:05.431 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:05.431 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:05.431 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:46:05.431 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:46:05.431 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:46:05.432 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:46:05.432 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:46:05.432 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:46:05.432 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.408+0000 7fd1ceffd640 1 -- 172.21.15.42:0/1361267842 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fd1c000ed10 con 0x7fd1d8120210 2024-09-16T03:46:05.432 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.408+0000 7fd1ceffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:46:05.432 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.408+0000 7fd1d6d9d640 1 --2- 172.21.15.42:0/1361267842 >> v2:172.21.15.154:6812/254845342 conn(0x7fd1b8037840 0x7fd1b8039d00 secure :-1 s=READY pgs=3397 cs=0 l=1 rev1=1 crypto rx=0x7fd1d0001570 tx=0x7fd1d0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:46:05.432 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.408+0000 7fd1ceffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:46:05.432 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.408+0000 7fd1ceffd640 1 -- 172.21.15.42:0/1361267842 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd1d8120210 msgr2=0x7fd1d811cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:46:05.432 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.408+0000 7fd1ceffd640 1 --2- 172.21.15.42:0/1361267842 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd1d8120210 0x7fd1d811cef0 secure :-1 s=READY pgs=2414 cs=0 l=1 rev1=1 crypto rx=0x7fd1d811e250 tx=0x7fd1c000e1d0 comp rx=0 tx=0).stop 2024-09-16T03:46:05.432 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.408+0000 7fd1ceffd640 1 -- 172.21.15.42:0/1361267842 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd1d8120210 msgr2=0x7fd1d811cef0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:46:05.432 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.408+0000 7fd1ceffd640 1 --2- 172.21.15.42:0/1361267842 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd1d8120210 0x7fd1d811cef0 unknown :-1 s=CLOSED pgs=2414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:46:05.432 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.408+0000 7fd1ceffd640 10 client.13523.objecter ms_handle_connect 0x7fd1b8037840 2024-09-16T03:46:05.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.412+0000 7fd1dce4b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:46:05.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.412+0000 7fd1dce4b640 1 --2- 172.21.15.42:0/1361267842 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd1d811d430 0x7fd1d811d840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:46:05.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.412+0000 7fd1dce4b640 10 monclient(hunting): start opening mon connection 2024-09-16T03:46:05.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.412+0000 7fd1dce4b640 1 -- 172.21.15.42:0/1361267842 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fd1d811dd80 con 0x7fd1d811d430 2024-09-16T03:46:05.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.412+0000 7fd1d5d9b640 1 --2- 172.21.15.42:0/1361267842 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd1d811d430 0x7fd1d811d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:46:05.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.412+0000 7fd1d5d9b640 10 monclient: get_auth_request con 0x7fd1d811d430 auth_method 0 2024-09-16T03:46:05.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.412+0000 7fd1d5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:46:05.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.412+0000 7fd1d5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:46:05.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.412+0000 7fd1d5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:46:05.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.412+0000 7fd1d5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:46:05.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.412+0000 7fd1d5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:46:05.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.412+0000 7fd1d5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:46:05.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.412+0000 7fd1d5d9b640 10 monclient(hunting): handle_auth_done global_id 13529 payload 824 2024-09-16T03:46:05.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.412+0000 7fd1d5d9b640 1 -- 172.21.15.42:0/1361267842 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fd1d8137120 con 0x7fd1d811d430 2024-09-16T03:46:05.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.412+0000 7fd1ceffd640 10 client.13523.objecter ms_handle_connect 0x7fd1d811d430 2024-09-16T03:46:05.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.412+0000 7fd1ceffd640 10 client.13523.objecter resend_mon_ops 2024-09-16T03:46:05.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.412+0000 7fd1d5d9b640 1 --2- 172.21.15.42:0/1361267842 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd1d811d430 0x7fd1d811d840 secure :-1 s=READY pgs=2415 cs=0 l=1 rev1=1 crypto rx=0x7fd1c0019000 tx=0x7fd1c00046d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:46:05.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.412+0000 7fd1ceffd640 1 -- 172.21.15.42:0/1361267842 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1c0018030 con 0x7fd1d811d430 2024-09-16T03:46:05.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.412+0000 7fd1ceffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:46:05.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.412+0000 7fd1ceffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:46:05.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.412+0000 7fd1ceffd640 10 monclient: dump: 2024-09-16T03:46:05.436 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:46:05.436 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:46:05.436 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:05.436 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:05.436 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:46:05.436 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:46:05.436 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:46:05.436 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:46:05.436 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:46:05.436 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:46:05.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.412+0000 7fd1ceffd640 1 -- 172.21.15.42:0/1361267842 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4880 (secure 0 0 0) 0x7fd1c001f0d0 con 0x7fd1d811d430 2024-09-16T03:46:05.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.412+0000 7fd1ceffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:46:05.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.412+0000 7fd1ceffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:46:05.437 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.412+0000 7fd1ceffd640 1 -- 172.21.15.42:0/1361267842 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd1d811d430 msgr2=0x7fd1d811d840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:46:05.437 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.412+0000 7fd1ceffd640 1 --2- 172.21.15.42:0/1361267842 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd1d811d430 0x7fd1d811d840 secure :-1 s=READY pgs=2415 cs=0 l=1 rev1=1 crypto rx=0x7fd1c0019000 tx=0x7fd1c00046d0 comp rx=0 tx=0).stop 2024-09-16T03:46:05.437 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.412+0000 7fd1ceffd640 1 -- 172.21.15.42:0/1361267842 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd1d811d430 msgr2=0x7fd1d811d840 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:46:05.437 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.412+0000 7fd1ceffd640 1 --2- 172.21.15.42:0/1361267842 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd1d811d430 0x7fd1d811d840 secure :-1 s=CLOSED pgs=2415 cs=0 l=1 rev1=1 crypto rx=0x7fd1c0019000 tx=0x7fd1c00046d0 comp rx=0 tx=0).stop 2024-09-16T03:46:05.437 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:46:05.437 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:46:05.437 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:46:05.437 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T03:46:05.437 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 34, 2024-09-16T03:46:05.437 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 244783, 2024-09-16T03:46:05.438 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:46:05.438 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:46:05.438 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T03:46:05.438 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T03:46:05.438 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:46:05.438 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 2, 2024-09-16T03:46:05.438 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:46:05.438 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:46:05.438 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:46:05.438 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:46:05.438 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:46:05.438 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:46:05.438 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:46:05.438 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:46:05.439 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:46:05.439 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:46:05.439 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:46:05.439 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:46:05.439 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:46:05.439 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:46:05.439 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:46:05.439 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:46:05.439 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:46:05.439 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:46:05.439 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:46:05.439 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:46:05.439 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:46:05.439 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:46:05.439 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:46:05.440 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:46:05.440 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:46:05.440 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:46:05.440 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:46:05.440 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:46:05.440 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:46:05.440 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:46:05.440 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:46:05.440 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:46:05.440 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:46:05.440 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:46:05.440 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:46:05.440 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:46:05.441 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:46:05.441 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:46:05.441 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:46:05.441 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:46:05.441 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:46:05.441 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:46:05.441 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:46:05.441 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:46:05.441 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:46:05.441 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:46:05.441 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:46:05.441 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:46:05.441 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:46:05.442 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:46:05.442 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:46:05.442 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:46:05.442 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:46:05.442 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:46:05.442 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:46:05.442 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:46:05.442 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:46:05.442 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:46:05.442 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:46:05.442 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:46:05.442 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:46:05.442 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:46:05.442 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:46:05.442 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:46:05.443 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:46:05.443 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:46:05.443 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:46:05.443 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:46:05.443 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:46:05.443 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:46:05.443 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:46:05.443 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:46:05.443 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:46:05.443 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:46:05.443 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:46:05.443 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:46:05.443 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:46:05.444 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:46:05.444 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:46:05.444 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:46:05.444 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:46:05.444 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:46:05.444 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:46:05.444 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:46:05.444 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:46:05.444 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:46:05.444 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:46:05.444 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:46:05.444 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:46:05.444 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:46:05.445 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:46:05.445 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:46:05.445 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:46:05.445 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:46:05.445 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:46:05.445 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:46:05.445 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:46:05.445 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:46:05.445 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:46:05.445 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:46:05.445 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:46:05.445 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:46:05.445 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:46:05.446 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:46:05.446 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:46:05.446 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:46:05.446 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:46:05.446 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:46:05.446 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:46:05.446 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:46:05.446 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:46:05.446 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:46:05.446 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:46:05.446 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:46:05.446 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:46:05.446 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:46:05.447 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:46:05.447 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:46:05.447 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:46:05.447 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:46:05.447 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:46:05.447 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:46:05.447 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:46:05.447 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:46:05.447 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:46:05.447 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:46:05.447 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:46:05.447 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:46:05.447 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:46:05.447 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:46:05.448 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:46:05.448 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:46:05.448 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:46:05.448 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:46:05.448 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:46:05.448 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:46:05.448 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:46:05.448 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:46:05.448 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:46:05.448 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:46:05.448 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:46:05.448 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T03:46:05.448 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:46:05.448 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:46:05.449 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:46:05.449 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 5 2024-09-16T03:46:05.449 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:46:05.449 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:46:05.449 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:46:05.449 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:46:05.449 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:46:05.449 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:46:05.449 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 10 2024-09-16T03:46:05.449 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:46:05.449 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:46:05.449 INFO:teuthology.orchestra.run.smithi042.stdout: "mgr": [ 2024-09-16T03:46:05.449 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:46:05.449 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:46:05.450 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:46:05.450 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:46:05.450 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:46:05.450 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:46:05.450 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:46:05.450 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:46:05.450 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:46:05.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.416+0000 7fd1dce4b640 10 librados: watch_flush enter 2024-09-16T03:46:05.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.416+0000 7fd1dce4b640 10 librados: watch_flush exit 2024-09-16T03:46:05.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.416+0000 7fd1dce4b640 20 client.13523.objecter shutdown clearing up homeless session... 2024-09-16T03:46:05.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.416+0000 7fd1dce4b640 10 client.13523.objecter successfully canceled tick 2024-09-16T03:46:05.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.416+0000 7fd1dce4b640 1 -- 172.21.15.42:0/1361267842 >> v2:172.21.15.154:6812/254845342 conn(0x7fd1b8037840 msgr2=0x7fd1b8039d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:46:05.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.416+0000 7fd1dce4b640 1 --2- 172.21.15.42:0/1361267842 >> v2:172.21.15.154:6812/254845342 conn(0x7fd1b8037840 0x7fd1b8039d00 secure :-1 s=READY pgs=3397 cs=0 l=1 rev1=1 crypto rx=0x7fd1d0001570 tx=0x7fd1d0009040 comp rx=0 tx=0).stop 2024-09-16T03:46:05.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.416+0000 7fd1dce4b640 10 monclient: shutdown 2024-09-16T03:46:05.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.416+0000 7fd1dce4b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:46:05.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.416+0000 7fd1dce4b640 1 -- 172.21.15.42:0/1361267842 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd1d8072d40 msgr2=0x7fd1d8135e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:46:05.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.416+0000 7fd1dce4b640 1 --2- 172.21.15.42:0/1361267842 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd1d8072d40 0x7fd1d8135e40 secure :-1 s=READY pgs=2413 cs=0 l=1 rev1=1 crypto rx=0x7fd1c80097c0 tx=0x7fd1c8019190 comp rx=0 tx=0).stop 2024-09-16T03:46:05.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.416+0000 7fd1d6d9d640 1 -- 172.21.15.42:0/1361267842 reap_dead start 2024-09-16T03:46:05.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.416+0000 7fd1d6d9d640 1 -- 172.21.15.42:0/1361267842 reap_dead start 2024-09-16T03:46:05.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.416+0000 7fd1dce4b640 1 -- 172.21.15.42:0/1361267842 shutdown_connections 2024-09-16T03:46:05.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.416+0000 7fd1dce4b640 1 -- 172.21.15.42:0/1361267842 >> 172.21.15.42:0/1361267842 conn(0x7fd1d80a2420 msgr2=0x7fd1d80a3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:46:05.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.416+0000 7fd1dce4b640 1 -- 172.21.15.42:0/1361267842 shutdown_connections 2024-09-16T03:46:05.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.416+0000 7fd1dce4b640 1 -- 172.21.15.42:0/1361267842 wait complete. 2024-09-16T03:46:05.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.416+0000 7fd1dce4b640 1 librados: shutdown 2024-09-16T03:46:05.453 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:46:05.453 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T03:46:05.453 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T03:46:05.570 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.564+0000 7f9fb9bf8640 10 monclient: get_monmap_and_config 2024-09-16T03:46:05.570 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.564+0000 7f9fb9bf8640 10 monclient: build_initial_monmap 2024-09-16T03:46:05.570 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.564+0000 7f9fb9bf8640 10 monclient: monmap: 2024-09-16T03:46:05.570 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:46:05.570 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:46:05.570 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:46:05.569239+0000 2024-09-16T03:46:05.570 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:46:05.569239+0000 2024-09-16T03:46:05.570 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:46:05.570 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:46:05.570 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:46:05.571 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:46:05.571 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:46:05.571 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:46:05.571 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:46:05.571 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:46:05.573 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.564+0000 7f9fb9bf8640 1 Processor -- start 2024-09-16T03:46:05.573 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.564+0000 7f9fb9bf8640 1 -- start start 2024-09-16T03:46:05.573 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.564+0000 7f9fb9bf8640 10 monclient: init 2024-09-16T03:46:05.573 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.564+0000 7f9fb9bf8640 10 monclient: _reopen_session rank -1 2024-09-16T03:46:05.573 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.564+0000 7f9fb9bf8640 10 monclient: _add_conns ranks=[0,2,1,3,4] 2024-09-16T03:46:05.573 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.564+0000 7f9fb9bf8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9fb40a39b0 addr v1:172.21.15.42:6789/0 2024-09-16T03:46:05.573 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.564+0000 7f9fb9bf8640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9fb40a3d90 0x7f9fb413c380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:46:05.573 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.564+0000 7f9fb9bf8640 10 monclient(hunting): picked mon.noname-b con 0x7f9fb40a3d90 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:46:05.573 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.564+0000 7f9fb9bf8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9fb413c8c0 addr v1:172.21.15.154:6789/0 2024-09-16T03:46:05.573 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.564+0000 7f9fb9bf8640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9fb4134290 con 0x7f9fb40a39b0 2024-09-16T03:46:05.573 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.564+0000 7f9fb9bf8640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9fb409c140 con 0x7f9fb413c8c0 2024-09-16T03:46:05.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.564+0000 7f9fb9bf8640 10 monclient(hunting): start opening mon connection 2024-09-16T03:46:05.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.564+0000 7f9fb9bf8640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f9fb409c420 con 0x7f9fb40a3d90 2024-09-16T03:46:05.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.564+0000 7f9fb9bf8640 10 monclient(hunting): _renew_subs 2024-09-16T03:46:05.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.564+0000 7f9fb9bf8640 10 monclient(hunting): authenticate will time out at 1212.539381s 2024-09-16T03:46:05.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.564+0000 7f9fb2ffd640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9fb40a3d90 0x7f9fb413c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:46:05.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.564+0000 7f9fb37fe640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f9fb40a39b0 0x7f9fb4133b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:48166/0 (socket says 172.21.15.42:48166) 2024-09-16T03:46:05.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.564+0000 7f9fb37fe640 1 -- 172.21.15.42:0/3725932537 learned_addr learned my addr 172.21.15.42:0/3725932537 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:46:05.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.564+0000 7f9fb2ffd640 10 monclient(hunting): get_auth_request con 0x7f9fb40a3d90 auth_method 0 2024-09-16T03:46:05.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.564+0000 7f9fb2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:46:05.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.564+0000 7f9fb2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:46:05.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.564+0000 7f9fb2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:46:05.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.564+0000 7f9fb2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:46:05.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.564+0000 7f9fb2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:46:05.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.564+0000 7f9fb2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:46:05.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.564+0000 7f9fb2ffd640 10 monclient(hunting): handle_auth_done global_id 13532 payload 274 2024-09-16T03:46:05.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.564+0000 7f9fb2ffd640 1 -- 172.21.15.42:0/3725932537 >> v1:172.21.15.154:6789/0 conn(0x7f9fb413c8c0 legacy=0x7f9fb413ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:46:05.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.564+0000 7f9fb2ffd640 1 -- 172.21.15.42:0/3725932537 >> v1:172.21.15.42:6789/0 conn(0x7f9fb40a39b0 legacy=0x7f9fb4133b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:46:05.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.564+0000 7f9fb2ffd640 10 monclient: _finish_hunting 0 2024-09-16T03:46:05.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.564+0000 7f9fb2ffd640 1 monclient: found mon.noname-b 2024-09-16T03:46:05.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.564+0000 7f9fb27fc640 1 -- 172.21.15.42:0/3725932537 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2334445287 0 0) 0x7f9fb4134290 con 0x7f9fb40a39b0 2024-09-16T03:46:05.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.564+0000 7f9fb2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:46:05.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.564+0000 7f9fb2ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T03:46:05.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.564+0000 7f9fb2ffd640 1 -- 172.21.15.42:0/3725932537 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9fb406dd60 con 0x7f9fb40a3d90 2024-09-16T03:46:05.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.564+0000 7f9fb27fc640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T03:46:05.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.568+0000 7f9fb2ffd640 1 --2- 172.21.15.42:0/3725932537 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9fb40a3d90 0x7f9fb413c380 secure :-1 s=READY pgs=2416 cs=0 l=1 rev1=1 crypto rx=0x7f9fa80097f0 tx=0x7f9fa801c360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b9c48f38481e1b93 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:46:05.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.568+0000 7f9fb27fc640 1 -- 172.21.15.42:0/3725932537 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fa80b0070 con 0x7f9fb40a3d90 2024-09-16T03:46:05.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.568+0000 7f9fb27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:46:05.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.568+0000 7f9fb27fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:46:05.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.568+0000 7f9fb27fc640 10 monclient: dump: 2024-09-16T03:46:05.583 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:46:05.583 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:46:05.583 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:05.583 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:05.584 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:46:05.584 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:46:05.584 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:46:05.584 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:46:05.584 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:46:05.584 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:46:05.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.568+0000 7f9fb27fc640 10 monclient: _finish_auth 0 2024-09-16T03:46:05.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.568+0000 7f9fb27fc640 10 monclient: _check_auth_tickets 2024-09-16T03:46:05.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.568+0000 7f9fb27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:46:05.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.568+0000 7f9fb9bf8640 5 monclient: authenticate success, global_id 13532 2024-09-16T03:46:05.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.568+0000 7f9fb27fc640 1 -- 172.21.15.42:0/3725932537 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9fa80ab600 con 0x7f9fb40a3d90 2024-09-16T03:46:05.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.568+0000 7f9fb27fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:46:05.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.568+0000 7f9fb9bf8640 10 monclient: get_monmap_and_config success 2024-09-16T03:46:05.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.568+0000 7f9fb9bf8640 4 set_mon_vals no callback set 2024-09-16T03:46:05.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.568+0000 7f9fb27fc640 1 -- 172.21.15.42:0/3725932537 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fa80ab9d0 con 0x7f9fb40a3d90 2024-09-16T03:46:05.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.568+0000 7f9fb27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:46:05.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.568+0000 7f9fb27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:46:05.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.568+0000 7f9fb27fc640 10 monclient: dump: 2024-09-16T03:46:05.585 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:46:05.585 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:46:05.585 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:05.585 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:05.585 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:46:05.585 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:46:05.586 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:46:05.586 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:46:05.586 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:46:05.586 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:46:05.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.568+0000 7f9fb9bf8640 10 monclient: shutdown 2024-09-16T03:46:05.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.568+0000 7f9fb9bf8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:46:05.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.568+0000 7f9fb9bf8640 1 -- 172.21.15.42:0/3725932537 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9fb40a3d90 msgr2=0x7f9fb413c380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:46:05.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.568+0000 7f9fb9bf8640 1 --2- 172.21.15.42:0/3725932537 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9fb40a3d90 0x7f9fb413c380 secure :-1 s=READY pgs=2416 cs=0 l=1 rev1=1 crypto rx=0x7f9fa80097f0 tx=0x7f9fa801c360 comp rx=0 tx=0).stop 2024-09-16T03:46:05.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.568+0000 7f9fb9bf8640 1 -- 172.21.15.42:0/3725932537 shutdown_connections 2024-09-16T03:46:05.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.568+0000 7f9fb9bf8640 1 --2- 172.21.15.42:0/3725932537 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9fb40a3d90 0x7f9fb413c380 unknown :-1 s=CLOSED pgs=2416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:46:05.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.568+0000 7f9fb9bf8640 1 -- 172.21.15.42:0/3725932537 >> 172.21.15.42:0/3725932537 conn(0x7f9fb40a9fb0 msgr2=0x7f9fb40aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:46:05.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.568+0000 7f9fb9bf8640 1 -- 172.21.15.42:0/3725932537 shutdown_connections 2024-09-16T03:46:05.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.568+0000 7f9fb9bf8640 1 -- 172.21.15.42:0/3725932537 wait complete. 2024-09-16T03:46:05.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.568+0000 7f9fb9bf8640 10 monclient: build_initial_monmap 2024-09-16T03:46:05.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.568+0000 7f9fb9bf8640 10 monclient: monmap: 2024-09-16T03:46:05.587 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:46:05.587 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:46:05.587 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:46:05.587 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:46:05.587 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:46:05.587 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:46:05.587 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:46:05.587 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:46:05.587 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:46:05.587 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:46:05.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.568+0000 7f9fb9bf8640 1 librados: starting msgr at 2024-09-16T03:46:05.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.568+0000 7f9fb9bf8640 1 librados: starting objecter 2024-09-16T03:46:05.588 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.568+0000 7f9fb9bf8640 1 Processor -- start 2024-09-16T03:46:05.588 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.568+0000 7f9fb9bf8640 1 -- start start 2024-09-16T03:46:05.588 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.568+0000 7f9fb9bf8640 1 librados: setting wanted keys 2024-09-16T03:46:05.588 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.568+0000 7f9fb9bf8640 1 librados: calling monclient init 2024-09-16T03:46:05.588 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.568+0000 7f9fb9bf8640 10 monclient: init 2024-09-16T03:46:05.588 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.572+0000 7f9fb9bf8640 10 monclient: _reopen_session rank -1 2024-09-16T03:46:05.588 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.572+0000 7f9fb9bf8640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T03:46:05.588 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.572+0000 7f9fb9bf8640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9fb40a39b0 0x7f9fb4130560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:46:05.588 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.572+0000 7f9fb9bf8640 10 monclient(hunting): picked mon.noname-b con 0x7f9fb40a39b0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:46:05.588 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.572+0000 7f9fb9bf8640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9fb40a3d90 0x7f9fb4130aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:46:05.588 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.572+0000 7f9fb9bf8640 10 monclient(hunting): picked mon.noname-c con 0x7f9fb40a3d90 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:46:05.588 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.572+0000 7f9fb9bf8640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9fb413c8c0 0x7f9fb4138ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:46:05.588 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.572+0000 7f9fb9bf8640 10 monclient(hunting): picked mon.noname-a con 0x7f9fb413c8c0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:46:05.588 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.572+0000 7f9fb9bf8640 10 monclient(hunting): start opening mon connection 2024-09-16T03:46:05.589 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.572+0000 7f9fb9bf8640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f9fb4142f80 con 0x7f9fb413c8c0 2024-09-16T03:46:05.589 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.572+0000 7f9fb9bf8640 10 monclient(hunting): start opening mon connection 2024-09-16T03:46:05.589 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.572+0000 7f9fb9bf8640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f9fb4142e00 con 0x7f9fb40a39b0 2024-09-16T03:46:05.589 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.572+0000 7f9fb9bf8640 10 monclient(hunting): start opening mon connection 2024-09-16T03:46:05.589 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.572+0000 7f9fb9bf8640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f9fb4143100 con 0x7f9fb40a3d90 2024-09-16T03:46:05.589 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.572+0000 7f9fb9bf8640 10 monclient(hunting): _renew_subs 2024-09-16T03:46:05.589 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.572+0000 7f9fb9bf8640 10 monclient(hunting): authenticate will time out at 1212.544415s 2024-09-16T03:46:05.589 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.572+0000 7f9fb2ffd640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9fb40a3d90 0x7f9fb4130aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:46:05.589 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.572+0000 7f9fb3fff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9fb413c8c0 0x7f9fb4138ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:46:05.589 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.572+0000 7f9fb2ffd640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9fb40a3d90 0x7f9fb4130aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:51692/0 (socket says 172.21.15.42:51692) 2024-09-16T03:46:05.589 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.572+0000 7f9fb2ffd640 1 -- 172.21.15.42:0/1734319193 learned_addr learned my addr 172.21.15.42:0/1734319193 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:46:05.589 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.572+0000 7f9fb2ffd640 10 monclient(hunting): get_auth_request con 0x7f9fb40a3d90 auth_method 0 2024-09-16T03:46:05.589 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.572+0000 7f9fb2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:46:05.590 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.572+0000 7f9fb2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:46:05.590 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.572+0000 7f9fb2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:46:05.590 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.572+0000 7f9fb3fff640 10 monclient(hunting): get_auth_request con 0x7f9fb413c8c0 auth_method 0 2024-09-16T03:46:05.590 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.572+0000 7f9fb3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:46:05.590 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.572+0000 7f9fb3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:46:05.590 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.572+0000 7f9fb3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:46:05.590 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.572+0000 7f9fb2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:46:05.590 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.572+0000 7f9fb2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:46:05.590 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.572+0000 7f9fb2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:46:05.590 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.572+0000 7f9fb3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:46:05.590 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.572+0000 7f9fb3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:46:05.590 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.572+0000 7f9fb3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:46:05.590 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.572+0000 7f9fb2ffd640 10 monclient(hunting): handle_auth_done global_id 13535 payload 824 2024-09-16T03:46:05.590 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.572+0000 7f9fb2ffd640 1 -- 172.21.15.42:0/1734319193 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9fb40a39b0 msgr2=0x7f9fb4130560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:46:05.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.572+0000 7f9fb2ffd640 1 --2- 172.21.15.42:0/1734319193 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9fb40a39b0 0x7f9fb4130560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:46:05.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.572+0000 7f9fb2ffd640 1 -- 172.21.15.42:0/1734319193 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9fb413c8c0 msgr2=0x7f9fb4138ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:46:05.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.572+0000 7f9fb2ffd640 1 --2- 172.21.15.42:0/1734319193 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9fb413c8c0 0x7f9fb4138ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:46:05.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.572+0000 7f9fb2ffd640 10 monclient: _finish_hunting 0 2024-09-16T03:46:05.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.572+0000 7f9fb2ffd640 1 monclient: found mon.noname-c 2024-09-16T03:46:05.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.572+0000 7f9fb2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:46:05.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.572+0000 7f9fb2ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:46:05.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.572+0000 7f9fb2ffd640 1 -- 172.21.15.42:0/1734319193 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9fb4132990 con 0x7f9fb40a3d90 2024-09-16T03:46:05.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.572+0000 7f9fb2ffd640 1 --2- 172.21.15.42:0/1734319193 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9fb40a3d90 0x7f9fb4130aa0 secure :-1 s=READY pgs=2417 cs=0 l=1 rev1=1 crypto rx=0x7f9fa80ae040 tx=0x7f9fa801aba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:46:05.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.572+0000 7f9f9bfff640 10 client.?.objecter ms_handle_connect 0x7f9fb40a3d90 2024-09-16T03:46:05.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.572+0000 7f9f9bfff640 10 client.?.objecter resend_mon_ops 2024-09-16T03:46:05.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.572+0000 7f9f9bfff640 1 -- 172.21.15.42:0/1734319193 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fa80b0040 con 0x7f9fb40a3d90 2024-09-16T03:46:05.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.572+0000 7f9f9bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:46:05.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.572+0000 7f9f9bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:46:05.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.572+0000 7f9f9bfff640 10 monclient: dump: 2024-09-16T03:46:05.592 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:46:05.592 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:46:05.592 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:05.592 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:05.592 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:46:05.592 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:46:05.592 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:46:05.592 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:46:05.592 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:46:05.592 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:46:05.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.572+0000 7f9f9bfff640 10 monclient: _finish_auth 0 2024-09-16T03:46:05.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.572+0000 7f9f9bfff640 10 monclient: _check_auth_tickets 2024-09-16T03:46:05.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.572+0000 7f9f9bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:46:05.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.572+0000 7f9f9bfff640 1 -- 172.21.15.42:0/1734319193 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9fa8004230 con 0x7f9fb40a3d90 2024-09-16T03:46:05.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.572+0000 7f9fb9bf8640 5 monclient: authenticate success, global_id 13535 2024-09-16T03:46:05.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.572+0000 7f9f9bfff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:46:05.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.572+0000 7f9fb9bf8640 10 monclient: _renew_subs 2024-09-16T03:46:05.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.572+0000 7f9f9bfff640 1 -- 172.21.15.42:0/1734319193 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fa8003620 con 0x7f9fb40a3d90 2024-09-16T03:46:05.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.572+0000 7f9fb9bf8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:46:05.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.572+0000 7f9fb9bf8640 1 -- 172.21.15.42:0/1734319193 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9fb409cbf0 con 0x7f9fb40a3d90 2024-09-16T03:46:05.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.572+0000 7f9f9bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:46:05.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.572+0000 7f9fb1ffb640 4 set_mon_vals no callback set 2024-09-16T03:46:05.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.572+0000 7f9f9bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:46:05.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.572+0000 7f9f9bfff640 10 monclient: dump: 2024-09-16T03:46:05.593 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:46:05.594 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:46:05.594 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:05.594 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:05.594 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:46:05.594 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:46:05.594 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:46:05.594 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:46:05.594 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:46:05.594 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:46:05.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.572+0000 7f9fb9bf8640 10 client.13535.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:46:05.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.572+0000 7f9fb9bf8640 10 monclient: _renew_subs 2024-09-16T03:46:05.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.572+0000 7f9fb9bf8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:46:05.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.572+0000 7f9fb9bf8640 1 -- 172.21.15.42:0/1734319193 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f9fb409c910 con 0x7f9fb40a3d90 2024-09-16T03:46:05.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.572+0000 7f9fb9bf8640 1 librados: init done 2024-09-16T03:46:05.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.572+0000 7f9fb9bf8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:46:05.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.572+0000 7f9fb9bf8640 1 --2- 172.21.15.42:0/1734319193 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9fb413b2a0 0x7f9fb4139410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:46:05.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.572+0000 7f9fb9bf8640 10 monclient(hunting): start opening mon connection 2024-09-16T03:46:05.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.572+0000 7f9fb9bf8640 1 -- 172.21.15.42:0/1734319193 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f9fb413c300 con 0x7f9fb413b2a0 2024-09-16T03:46:05.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.572+0000 7f9f9bfff640 1 -- 172.21.15.42:0/1734319193 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f9fa80ab4c0 con 0x7f9fb40a3d90 2024-09-16T03:46:05.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.572+0000 7f9fb37fe640 1 --2- 172.21.15.42:0/1734319193 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9fb413b2a0 0x7f9fb4139410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:46:05.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.572+0000 7f9fb37fe640 10 monclient: get_auth_request con 0x7f9fb413b2a0 auth_method 0 2024-09-16T03:46:05.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.572+0000 7f9fb37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:46:05.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.572+0000 7f9fb37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:46:05.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.572+0000 7f9fb37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:46:05.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.572+0000 7f9f9bfff640 1 --2- 172.21.15.42:0/1734319193 >> v2:172.21.15.154:6812/254845342 conn(0x7f9fa0037810 0x7f9fa0039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:46:05.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.572+0000 7f9f9bfff640 1 -- 172.21.15.42:0/1734319193 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(119..119 src has 1..119) ==== 8177+0+0 (secure 0 0 0) 0x7f9fa80e45e0 con 0x7f9fb40a3d90 2024-09-16T03:46:05.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.572+0000 7f9f9bfff640 10 client.13535.objecter ms_dispatch 0x7f9fb4000ba0 osd_map(119..119 src has 1..119) 2024-09-16T03:46:05.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.572+0000 7f9f9bfff640 3 client.13535.objecter handle_osd_map got epochs [119,119] > 0 2024-09-16T03:46:05.596 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.572+0000 7f9f9bfff640 3 client.13535.objecter handle_osd_map decoding full epoch 119 2024-09-16T03:46:05.596 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.572+0000 7f9fb37fe640 1 -- 172.21.15.42:0/1734319193 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9fb413b2a0 msgr2=0x7f9fb4139410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-16T03:46:05.596 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.572+0000 7f9fb37fe640 1 -- 172.21.15.42:0/1734319193 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9fb413b2a0 msgr2=0x7f9fb4139410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T03:46:05.596 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.572+0000 7f9fb37fe640 1 --2- 172.21.15.42:0/1734319193 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9fb413b2a0 0x7f9fb4139410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:46:05.596 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.572+0000 7f9f9bfff640 20 client.13535.objecter dump_active .. 0 homeless 2024-09-16T03:46:05.596 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.572+0000 7f9fb37fe640 1 --2- 172.21.15.42:0/1734319193 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9fb413b2a0 0x7f9fb4139410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:46:05.597 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.584+0000 7f9fb3fff640 1 --2- 172.21.15.42:0/1734319193 >> v2:172.21.15.154:6812/254845342 conn(0x7f9fa0037810 0x7f9fa0039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:46:05.597 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.584+0000 7f9fb3fff640 10 monclient: get_auth_request con 0x7f9fa0037810 auth_method 0 2024-09-16T03:46:05.597 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.584+0000 7f9fb3fff640 1 --2- 172.21.15.42:0/1734319193 >> v2:172.21.15.154:6812/254845342 conn(0x7f9fa0037810 0x7f9fa0039cd0 secure :-1 s=READY pgs=3398 cs=0 l=1 rev1=1 crypto rx=0x7f9fb41402e0 tx=0x7f9fac003d50 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:46:05.597 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.584+0000 7f9f9bfff640 10 client.13535.objecter ms_handle_connect 0x7f9fa0037810 2024-09-16T03:46:05.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.776+0000 7f9fb37fe640 1 --2- 172.21.15.42:0/1734319193 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9fb413b2a0 0x7f9fb4139410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:46:05.782 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.776+0000 7f9fb37fe640 10 monclient: get_auth_request con 0x7f9fb413b2a0 auth_method 0 2024-09-16T03:46:05.782 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.776+0000 7f9fb37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:46:05.782 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.776+0000 7f9fb37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:46:05.782 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.776+0000 7f9fb37fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:46:05.782 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.776+0000 7f9fb37fe640 1 -- 172.21.15.42:0/1734319193 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9fb413b2a0 msgr2=0x7f9fb4139410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-16T03:46:05.782 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.776+0000 7f9fb37fe640 1 -- 172.21.15.42:0/1734319193 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9fb413b2a0 msgr2=0x7f9fb4139410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T03:46:05.782 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.776+0000 7f9fb37fe640 1 --2- 172.21.15.42:0/1734319193 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9fb413b2a0 0x7f9fb4139410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:46:05.782 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:05.776+0000 7f9fb37fe640 1 --2- 172.21.15.42:0/1734319193 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9fb413b2a0 0x7f9fb4139410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T03:46:05.796 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleWr api_aio_pp: Running main() from gmock_main.cc 2024-09-16T03:46:05.797 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio_pp: [==========] Running 57 tests from 4 test suites. 2024-09-16T03:46:05.797 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio_pp: [----------] Global test environment set-up. 2024-09-16T03:46:05.797 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio_pp: [----------] 32 tests from LibRadosAio 2024-09-16T03:46:05.797 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio_pp: [ RUN ] LibRadosAio.TooBigPP 2024-09-16T03:46:05.797 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio_pp: [ OK ] LibRadosAio.TooBigPP (4881 ms) 2024-09-16T03:46:05.797 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio_pp: [ RUN ] LibRadosAio.PoolQuotaPP 2024-09-16T03:46:05.797 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio_pp: [ OK ] LibRadosAio.PoolQuotaPP (38779 ms) 2024-09-16T03:46:05.797 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleWritePP 2024-09-16T03:46:05.797 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleWritePP (11909 ms) 2024-09-16T03:46:05.797 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio_pp: [ RUN ] LibRadosAio.WaitForSafePP 2024-09-16T03:46:05.797 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio_pp: [ OK ] LibRadosAio.WaitForSafePP (3094 ms) 2024-09-16T03:46:05.797 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP 2024-09-16T03:46:05.797 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP (3015 ms) 2024-09-16T03:46:05.797 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP2 2024-09-16T03:46:05.798 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP2 (3068 ms) 2024-09-16T03:46:05.798 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP3 2024-09-16T03:46:05.798 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP3 (17247 ms) 2024-09-16T03:46:05.798 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripSparseReadPP 2024-09-16T03:46:05.798 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripSparseReadPP (2793 ms) 2024-09-16T03:46:05.798 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio_pp: [ RUN ] LibRadosAio.IsCompletePP 2024-09-16T03:46:05.798 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio_pp: [ OK ] LibRadosAio.IsCompletePP (2987 ms) 2024-09-16T03:46:05.798 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio_pp: [ RUN ] LibRadosAio.IsSafePP 2024-09-16T03:46:05.798 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio_pp: [ OK ] LibRadosAio.IsSafePP (3029 ms) 2024-09-16T03:46:05.798 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio_pp: [ RUN ] LibRadosAio.ReturnValuePP 2024-09-16T03:46:05.798 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio_pp: [ OK ] LibRadosAio.ReturnValuePP (2976 ms) 2024-09-16T03:46:05.798 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio_pp: [ RUN ] LibRadosAio.FlushPP 2024-09-16T03:46:05.798 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio_pp: [ OK ] LibRadosAio.FlushPP (3013 ms) 2024-09-16T03:46:05.799 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio_pp: [ RUN ] LibRadosAio.FlushAsyncPP 2024-09-16T03:46:05.799 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio_pp: [ OK ] LibRadosAio.FlushAsyncPP (3035 ms) 2024-09-16T03:46:05.799 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteFullPP 2024-09-16T03:46:05.799 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteFullPP (3187 ms) 2024-09-16T03:46:05.799 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteFullPP2 2024-09-16T03:46:05.799 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteFullPP2 (3001 ms) 2024-09-16T03:46:05.799 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteSamePP 2024-09-16T03:46:05.799 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteSamePP (2767 ms) 2024-09-16T03:46:05.799 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteSamePP2 2024-09-16T03:46:05.799 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteSamePP2 (2725 ms) 2024-09-16T03:46:05.799 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleStatPPNS 2024-09-16T03:46:05.799 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleStatPPNS (18089 ms) 2024-09-16T03:46:05.799 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleStatPP 2024-09-16T03:46:05.800 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleStatPP (3038 ms) 2024-09-16T03:46:05.800 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio_pp: [ RUN ] LibRadosAio.OperateMtime 2024-09-16T03:46:05.800 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio_pp: [ OK ] LibRadosAio.OperateMtime (2493 ms) 2024-09-16T03:46:05.800 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio_pp: [ RUN ] LibRadosAio.OperateMtime2 2024-09-16T03:46:05.800 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio_pp: [ OK ] LibRadosAio.OperateMtime2 (2982 ms) 2024-09-16T03:46:05.800 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio_pp: [ RUN ] LibRadosAio.StatRemovePP 2024-09-16T03:46:05.800 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio_pp: [ OK ] LibRadosAio.StatRemovePP (3013 ms) 2024-09-16T03:46:05.800 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio_pp: [ RUN ] LibRadosAio.ExecuteClassPP 2024-09-16T03:46:05.800 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio_pp: [ OK ] LibRadosAio.ExecuteClassPP (2995 ms) 2024-09-16T03:46:05.800 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio_pp: [ RUN ] LibRadosAio.OmapPP 2024-09-16T03:46:05.800 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio_pp: [ OK ] LibRadosAio.OmapPP (3052 ms) 2024-09-16T03:46:05.800 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio_pp: [ RUN ] LibRadosAio.MultiWritePP 2024-09-16T03:46:05.800 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio_pp: [ OK ] LibRadosAio.MultiWritePP (3037 ms) 2024-09-16T03:46:05.801 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio_pp: [ RUN ] LibRadosAio.AioUnlockPP 2024-09-16T03:46:05.801 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio_pp: [ OK ] LibRadosAio.AioUnlockPP (2953 ms) 2024-09-16T03:46:05.801 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripAppendPP 2024-09-16T03:46:06.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:06.180+0000 7f9fb37fe640 1 --2- 172.21.15.42:0/1734319193 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9fb413b2a0 0x7f9fb4139410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:46:06.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:06.180+0000 7f9fb37fe640 1 -- 172.21.15.42:0/1734319193 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9fb413b2a0 msgr2=0x7f9fb4139410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-16T03:46:06.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:06.180+0000 7f9fb37fe640 1 -- 172.21.15.42:0/1734319193 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9fb413b2a0 msgr2=0x7f9fb4139410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T03:46:06.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:06.180+0000 7f9fb37fe640 1 --2- 172.21.15.42:0/1734319193 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9fb413b2a0 0x7f9fb4139410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:46:06.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:06.180+0000 7f9fb37fe640 1 --2- 172.21.15.42:0/1734319193 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9fb413b2a0 0x7f9fb4139410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T03:46:06.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:06.984+0000 7f9fb37fe640 1 --2- 172.21.15.42:0/1734319193 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9fb413b2a0 0x7f9fb4139410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:46:06.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:06.984+0000 7f9fb37fe640 10 monclient: get_auth_request con 0x7f9fb413b2a0 auth_method 0 2024-09-16T03:46:06.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:06.984+0000 7f9fb37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:46:06.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:06.984+0000 7f9fb37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:46:06.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:06.984+0000 7f9fb37fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:46:06.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:06.984+0000 7f9fb37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:46:06.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:06.984+0000 7f9fb37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:46:06.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:06.984+0000 7f9fb37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:46:06.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:06.984+0000 7f9fb37fe640 10 monclient(hunting): handle_auth_done global_id 754102 payload 824 2024-09-16T03:46:06.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:06.984+0000 7f9fb37fe640 1 -- 172.21.15.42:0/1734319193 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9fb4133b80 con 0x7f9fb413b2a0 2024-09-16T03:46:06.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:06.984+0000 7f9fb37fe640 1 --2- 172.21.15.42:0/1734319193 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9fb413b2a0 0x7f9fb4139410 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f9fa40011f0 tx=0x7f9fa4001690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:46:06.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:06.984+0000 7f9f9bfff640 10 client.13535.objecter ms_handle_connect 0x7f9fb413b2a0 2024-09-16T03:46:06.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:06.984+0000 7f9f9bfff640 10 client.13535.objecter resend_mon_ops 2024-09-16T03:46:06.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:06.984+0000 7f9f9bfff640 1 -- 172.21.15.42:0/1734319193 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fa401b070 con 0x7f9fb413b2a0 2024-09-16T03:46:06.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:06.984+0000 7f9f9bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:46:06.992 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:06.984+0000 7f9f9bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:46:06.992 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:06.984+0000 7f9f9bfff640 10 monclient: dump: 2024-09-16T03:46:06.992 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:46:06.992 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:46:06.992 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:06.992 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:06.992 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:46:06.992 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:46:06.992 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:46:06.992 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:46:06.993 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:46:06.993 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:46:06.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:06.984+0000 7f9f9bfff640 1 -- 172.21.15.42:0/1734319193 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f9fa4016670 con 0x7f9fb413b2a0 2024-09-16T03:46:06.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:06.984+0000 7f9f9bfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:46:06.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:06.984+0000 7f9f9bfff640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:46:06.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:06.984+0000 7f9f9bfff640 1 -- 172.21.15.42:0/1734319193 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9fb413b2a0 msgr2=0x7f9fb4139410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:46:06.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:06.984+0000 7f9f9bfff640 1 --2- 172.21.15.42:0/1734319193 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9fb413b2a0 0x7f9fb4139410 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f9fa40011f0 tx=0x7f9fa4001690 comp rx=0 tx=0).stop 2024-09-16T03:46:06.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:06.984+0000 7f9f9bfff640 1 -- 172.21.15.42:0/1734319193 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9fb413b2a0 msgr2=0x7f9fb4139410 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:46:06.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:06.984+0000 7f9f9bfff640 1 --2- 172.21.15.42:0/1734319193 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9fb413b2a0 0x7f9fb4139410 secure :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f9fa40011f0 tx=0x7f9fa4001690 comp rx=0 tx=0).stop 2024-09-16T03:46:06.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:06.992+0000 7f9fb9bf8640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:46:06.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:06.992+0000 7f9fb9bf8640 1 --2- 172.21.15.42:0/1734319193 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9fb4139950 0x7f9fb4139d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:46:06.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:06.992+0000 7f9fb9bf8640 10 monclient(hunting): start opening mon connection 2024-09-16T03:46:06.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:06.992+0000 7f9fb9bf8640 1 -- 172.21.15.42:0/1734319193 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f9fb413a270 con 0x7f9fb4139950 2024-09-16T03:46:07.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:06.992+0000 7f9fb37fe640 1 --2- 172.21.15.42:0/1734319193 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9fb4139950 0x7f9fb4139d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:46:07.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:06.992+0000 7f9fb37fe640 10 monclient: get_auth_request con 0x7f9fb4139950 auth_method 0 2024-09-16T03:46:07.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:06.992+0000 7f9fb37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:46:07.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:06.992+0000 7f9fb37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:46:07.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:06.992+0000 7f9fb37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:46:07.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:06.992+0000 7f9fb37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:46:07.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:06.992+0000 7f9fb37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:46:07.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:06.992+0000 7f9fb37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:46:07.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:06.992+0000 7f9fb37fe640 10 monclient(hunting): handle_auth_done global_id 754105 payload 824 2024-09-16T03:46:07.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:06.992+0000 7f9fb37fe640 1 -- 172.21.15.42:0/1734319193 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f9fb40a5750 con 0x7f9fb4139950 2024-09-16T03:46:07.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:06.996+0000 7f9f9bfff640 10 client.13535.objecter ms_handle_connect 0x7f9fb4139950 2024-09-16T03:46:07.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:06.996+0000 7f9fb37fe640 1 --2- 172.21.15.42:0/1734319193 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9fb4139950 0x7f9fb4139d30 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f9fa4018000 tx=0x7f9fa4001100 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:46:07.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:06.996+0000 7f9f9bfff640 10 client.13535.objecter resend_mon_ops 2024-09-16T03:46:07.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:06.996+0000 7f9f9bfff640 1 -- 172.21.15.42:0/1734319193 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fa401b070 con 0x7f9fb4139950 2024-09-16T03:46:07.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:06.996+0000 7f9f9bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:46:07.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:06.996+0000 7f9f9bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:46:07.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:06.996+0000 7f9f9bfff640 10 monclient: dump: 2024-09-16T03:46:07.003 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:46:07.003 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:46:07.003 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:07.003 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:07.003 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:46:07.003 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:46:07.003 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:46:07.003 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:46:07.003 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:46:07.003 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:46:07.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:06.996+0000 7f9f9bfff640 1 -- 172.21.15.42:0/1734319193 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4543 (secure 0 0 0) 0x7f9fa401f320 con 0x7f9fb4139950 2024-09-16T03:46:07.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:06.996+0000 7f9f9bfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:46:07.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:06.996+0000 7f9f9bfff640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:46:07.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:06.996+0000 7f9f9bfff640 1 -- 172.21.15.42:0/1734319193 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9fb4139950 msgr2=0x7f9fb4139d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:46:07.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:06.996+0000 7f9f9bfff640 1 --2- 172.21.15.42:0/1734319193 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9fb4139950 0x7f9fb4139d30 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f9fa4018000 tx=0x7f9fa4001100 comp rx=0 tx=0).stop 2024-09-16T03:46:07.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:06.996+0000 7f9f9bfff640 1 -- 172.21.15.42:0/1734319193 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9fb4139950 msgr2=0x7f9fb4139d30 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:46:07.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:06.996+0000 7f9f9bfff640 1 --2- 172.21.15.42:0/1734319193 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9fb4139950 0x7f9fb4139d30 secure :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f9fa4018000 tx=0x7f9fa4001100 comp rx=0 tx=0).stop 2024-09-16T03:46:07.004 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:46:07.004 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:46:07.004 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:46:07.004 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T03:46:07.005 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 34, 2024-09-16T03:46:07.005 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 9015, 2024-09-16T03:46:07.005 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:46:07.006 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:46:07.006 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T03:46:07.006 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T03:46:07.006 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:46:07.006 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 3, 2024-09-16T03:46:07.006 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:46:07.006 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:46:07.006 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:46:07.006 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:46:07.007 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:46:07.007 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:46:07.007 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:46:07.007 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:46:07.007 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:46:07.007 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:46:07.007 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:46:07.007 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:46:07.007 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:46:07.007 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:46:07.007 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:46:07.007 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:46:07.008 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:46:07.008 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:46:07.008 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:46:07.008 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:46:07.008 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:46:07.008 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:46:07.008 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:46:07.008 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:46:07.008 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:46:07.008 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:46:07.008 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:46:07.008 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:46:07.009 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:46:07.009 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:46:07.009 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:46:07.009 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:46:07.009 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:46:07.009 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:46:07.009 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:46:07.009 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:46:07.009 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:46:07.010 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:46:07.010 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:46:07.010 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:46:07.010 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:46:07.010 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:46:07.010 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:46:07.010 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:46:07.010 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:46:07.010 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:46:07.010 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:46:07.010 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:46:07.010 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:46:07.011 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:46:07.011 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:46:07.011 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:46:07.011 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:46:07.011 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:46:07.011 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:46:07.011 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:46:07.011 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:46:07.011 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:46:07.011 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:46:07.011 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:46:07.011 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:46:07.012 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:46:07.012 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:46:07.012 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:46:07.012 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:46:07.012 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:46:07.012 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:46:07.012 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:46:07.012 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:46:07.012 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:46:07.012 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:46:07.012 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:46:07.013 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:46:07.013 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:46:07.013 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:46:07.013 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:46:07.013 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:46:07.013 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:46:07.013 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:46:07.013 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:46:07.013 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:46:07.013 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:46:07.013 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:46:07.013 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:46:07.013 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:46:07.014 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:46:07.014 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:46:07.014 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:46:07.014 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:46:07.014 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:46:07.014 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:46:07.014 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:46:07.014 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:46:07.014 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:46:07.014 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:46:07.014 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:46:07.014 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:46:07.015 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:46:07.015 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:46:07.015 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:46:07.015 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:46:07.015 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:46:07.015 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:46:07.015 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:46:07.015 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:46:07.015 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:46:07.015 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:46:07.016 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:46:07.016 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:46:07.016 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:46:07.016 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:46:07.016 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:46:07.016 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:46:07.016 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:46:07.016 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:46:07.016 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:46:07.017 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:46:07.017 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:46:07.017 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:46:07.017 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:46:07.017 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:46:07.017 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:46:07.017 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:46:07.017 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:46:07.017 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:46:07.017 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:46:07.017 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:46:07.017 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:46:07.018 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:46:07.018 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:46:07.018 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:46:07.018 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:46:07.018 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:46:07.018 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:46:07.018 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:46:07.018 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:46:07.018 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:46:07.018 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:46:07.018 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:46:07.018 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:46:07.019 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:46:07.019 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:46:07.019 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:46:07.019 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:46:07.019 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:46:07.019 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 2 2024-09-16T03:46:07.019 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:46:07.019 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:46:07.019 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:46:07.019 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:46:07.019 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:46:07.021 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:06.996+0000 7f9fb9bf8640 10 librados: watch_flush enter 2024-09-16T03:46:07.021 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:06.996+0000 7f9fb9bf8640 10 librados: watch_flush exit 2024-09-16T03:46:07.021 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:06.996+0000 7f9fb9bf8640 20 client.13535.objecter shutdown clearing up homeless session... 2024-09-16T03:46:07.021 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:06.996+0000 7f9fb9bf8640 10 client.13535.objecter successfully canceled tick 2024-09-16T03:46:07.021 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:06.996+0000 7f9fb9bf8640 1 -- 172.21.15.42:0/1734319193 >> v2:172.21.15.154:6812/254845342 conn(0x7f9fa0037810 msgr2=0x7f9fa0039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:46:07.021 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:06.996+0000 7f9fb9bf8640 1 --2- 172.21.15.42:0/1734319193 >> v2:172.21.15.154:6812/254845342 conn(0x7f9fa0037810 0x7f9fa0039cd0 secure :-1 s=READY pgs=3398 cs=0 l=1 rev1=1 crypto rx=0x7f9fb41402e0 tx=0x7f9fac003d50 comp rx=0 tx=0).stop 2024-09-16T03:46:07.021 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:06.996+0000 7f9fb9bf8640 10 monclient: shutdown 2024-09-16T03:46:07.022 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:06.996+0000 7f9fb9bf8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:46:07.022 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:06.996+0000 7f9fb9bf8640 1 -- 172.21.15.42:0/1734319193 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9fb40a3d90 msgr2=0x7f9fb4130aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:46:07.022 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:06.996+0000 7f9fb9bf8640 1 --2- 172.21.15.42:0/1734319193 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9fb40a3d90 0x7f9fb4130aa0 secure :-1 s=READY pgs=2417 cs=0 l=1 rev1=1 crypto rx=0x7f9fa80ae040 tx=0x7f9fa801aba0 comp rx=0 tx=0).stop 2024-09-16T03:46:07.022 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:06.996+0000 7f9fb3fff640 1 -- 172.21.15.42:0/1734319193 reap_dead start 2024-09-16T03:46:07.022 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:06.996+0000 7f9fb3fff640 1 -- 172.21.15.42:0/1734319193 reap_dead start 2024-09-16T03:46:07.022 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:06.996+0000 7f9fb9bf8640 1 -- 172.21.15.42:0/1734319193 shutdown_connections 2024-09-16T03:46:07.022 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:06.996+0000 7f9fb9bf8640 1 -- 172.21.15.42:0/1734319193 >> 172.21.15.42:0/1734319193 conn(0x7f9fb40a9fb0 msgr2=0x7f9fb40a8820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:46:07.022 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:06.996+0000 7f9fb9bf8640 1 -- 172.21.15.42:0/1734319193 shutdown_connections 2024-09-16T03:46:07.022 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:06.996+0000 7f9fb9bf8640 1 -- 172.21.15.42:0/1734319193 wait complete. 2024-09-16T03:46:07.022 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:06.996+0000 7f9fb9bf8640 1 librados: shutdown 2024-09-16T03:46:07.026 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:46:07.026 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T03:46:07.026 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T03:46:07.157 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.148+0000 7f4dd817c640 10 monclient: get_monmap_and_config 2024-09-16T03:46:07.157 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.148+0000 7f4dd817c640 10 monclient: build_initial_monmap 2024-09-16T03:46:07.157 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.148+0000 7f4dd817c640 10 monclient: monmap: 2024-09-16T03:46:07.157 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:46:07.157 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:46:07.157 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:46:07.156005+0000 2024-09-16T03:46:07.157 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:46:07.156005+0000 2024-09-16T03:46:07.157 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:46:07.157 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:46:07.157 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:46:07.157 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:46:07.158 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:46:07.158 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:46:07.158 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:46:07.158 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:46:07.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.152+0000 7f4dd817c640 1 Processor -- start 2024-09-16T03:46:07.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.152+0000 7f4dd817c640 1 -- start start 2024-09-16T03:46:07.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.152+0000 7f4dd817c640 10 monclient: init 2024-09-16T03:46:07.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.152+0000 7f4dd817c640 10 monclient: _reopen_session rank -1 2024-09-16T03:46:07.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.152+0000 7f4dd817c640 10 monclient: _add_conns ranks=[3,1,2,4,0] 2024-09-16T03:46:07.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.152+0000 7f4dd817c640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f4dd006db90 0x7f4dd01340c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:46:07.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.152+0000 7f4dd817c640 10 monclient(hunting): picked mon.noname-a con 0x7f4dd006db90 addr v2:172.21.15.42:3300/0 2024-09-16T03:46:07.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.152+0000 7f4dd817c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4dd00a37f0 addr v1:172.21.15.154:6789/0 2024-09-16T03:46:07.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.152+0000 7f4dd817c640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4dd00a3c00 0x7f4dd0137520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:46:07.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.152+0000 7f4dd817c640 10 monclient(hunting): picked mon.noname-b con 0x7f4dd00a3c00 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:46:07.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.152+0000 7f4dd817c640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4dd006ece0 con 0x7f4dd00a37f0 2024-09-16T03:46:07.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.152+0000 7f4dd817c640 10 monclient(hunting): start opening mon connection 2024-09-16T03:46:07.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.152+0000 7f4dd817c640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f4dd006df70 con 0x7f4dd006db90 2024-09-16T03:46:07.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.152+0000 7f4dd817c640 10 monclient(hunting): start opening mon connection 2024-09-16T03:46:07.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.152+0000 7f4dd817c640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f4dd009bf80 con 0x7f4dd00a3c00 2024-09-16T03:46:07.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.152+0000 7f4dd817c640 10 monclient(hunting): _renew_subs 2024-09-16T03:46:07.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.152+0000 7f4dd817c640 10 monclient(hunting): authenticate will time out at 1214.125583s 2024-09-16T03:46:07.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.152+0000 7f4dd6719640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4dd00a3c00 0x7f4dd0137520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:46:07.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.152+0000 7f4dd6719640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4dd00a3c00 0x7f4dd0137520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:51704/0 (socket says 172.21.15.42:51704) 2024-09-16T03:46:07.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.152+0000 7f4dd5f18640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f4dd006db90 0x7f4dd01340c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:46:07.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.152+0000 7f4dd6719640 1 -- 172.21.15.42:0/821873303 learned_addr learned my addr 172.21.15.42:0/821873303 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:46:07.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.152+0000 7f4dd6719640 10 monclient(hunting): get_auth_request con 0x7f4dd00a3c00 auth_method 0 2024-09-16T03:46:07.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.152+0000 7f4dd6719640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:46:07.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.152+0000 7f4dd6719640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:46:07.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.152+0000 7f4dd6719640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:46:07.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.152+0000 7f4dd5f18640 10 monclient(hunting): get_auth_request con 0x7f4dd006db90 auth_method 0 2024-09-16T03:46:07.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.152+0000 7f4dd5f18640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:46:07.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.152+0000 7f4dd5f18640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:46:07.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.152+0000 7f4dd5f18640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:46:07.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.152+0000 7f4dd6719640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:46:07.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.152+0000 7f4dd6719640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:46:07.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.152+0000 7f4dd6719640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:46:07.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.152+0000 7f4dd5f18640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:46:07.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.152+0000 7f4dd5f18640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:46:07.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.152+0000 7f4dd5f18640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:46:07.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.152+0000 7f4dd5f18640 10 monclient(hunting): handle_auth_done global_id 744396 payload 274 2024-09-16T03:46:07.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.152+0000 7f4dd5f18640 1 -- 172.21.15.42:0/821873303 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4dd00a3c00 msgr2=0x7f4dd0137520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:46:07.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.152+0000 7f4dd5f18640 1 --2- 172.21.15.42:0/821873303 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4dd00a3c00 0x7f4dd0137520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:46:07.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.152+0000 7f4dd5f18640 1 -- 172.21.15.42:0/821873303 >> v1:172.21.15.154:6789/0 conn(0x7f4dd00a37f0 legacy=0x7f4dd0136e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:46:07.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.152+0000 7f4dd5f18640 10 monclient: _finish_hunting 0 2024-09-16T03:46:07.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.152+0000 7f4dd5f18640 1 monclient: found mon.noname-a 2024-09-16T03:46:07.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.152+0000 7f4dd5f18640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:46:07.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.152+0000 7f4dd5f18640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:46:07.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.152+0000 7f4dd5f18640 1 -- 172.21.15.42:0/821873303 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4dd0134780 con 0x7f4dd006db90 2024-09-16T03:46:07.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.152+0000 7f4dd6719640 1 --2- 172.21.15.42:0/821873303 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4dd00a3c00 0x7f4dd0137520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:46:07.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.156+0000 7f4dd5f18640 1 --2- 172.21.15.42:0/821873303 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4dd006db90 0x7f4dd01340c0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f4dcc0097c0 tx=0x7f4dcc01c360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b8d222eef4bb590 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:46:07.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.156+0000 7f4dd4f16640 1 -- 172.21.15.42:0/821873303 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dcc0b0070 con 0x7f4dd006db90 2024-09-16T03:46:07.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.156+0000 7f4dd4f16640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:46:07.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.156+0000 7f4dd4f16640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:46:07.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.156+0000 7f4dd4f16640 10 monclient: dump: 2024-09-16T03:46:07.169 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:46:07.169 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:46:07.169 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:07.169 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:07.169 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:46:07.170 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:46:07.170 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:46:07.170 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:46:07.170 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:46:07.170 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:46:07.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.156+0000 7f4dd4f16640 10 monclient: _finish_auth 0 2024-09-16T03:46:07.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.156+0000 7f4dd4f16640 10 monclient: _check_auth_tickets 2024-09-16T03:46:07.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.156+0000 7f4dd4f16640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:46:07.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.156+0000 7f4dd4f16640 1 -- 172.21.15.42:0/821873303 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4dcc0ab600 con 0x7f4dd006db90 2024-09-16T03:46:07.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.156+0000 7f4dd817c640 5 monclient: authenticate success, global_id 744396 2024-09-16T03:46:07.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.156+0000 7f4dd817c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:46:07.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.156+0000 7f4dd4f16640 10 monclient: handle_config config(0 keys) 2024-09-16T03:46:07.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.156+0000 7f4dd817c640 10 monclient: get_monmap_and_config success 2024-09-16T03:46:07.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.156+0000 7f4dd4f16640 1 -- 172.21.15.42:0/821873303 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dcc0ab9d0 con 0x7f4dd006db90 2024-09-16T03:46:07.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.156+0000 7f4dd817c640 4 set_mon_vals no callback set 2024-09-16T03:46:07.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.156+0000 7f4dd4f16640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:46:07.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.156+0000 7f4dd4f16640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:46:07.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.156+0000 7f4dd4f16640 10 monclient: dump: 2024-09-16T03:46:07.171 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:46:07.171 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:46:07.171 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:07.171 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:07.171 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:46:07.171 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:46:07.171 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:46:07.171 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:46:07.171 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:46:07.171 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:46:07.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.156+0000 7f4dd817c640 10 monclient: shutdown 2024-09-16T03:46:07.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.156+0000 7f4dd817c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:46:07.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.156+0000 7f4dd817c640 1 -- 172.21.15.42:0/821873303 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4dd006db90 msgr2=0x7f4dd01340c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:46:07.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.156+0000 7f4dd817c640 1 --2- 172.21.15.42:0/821873303 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4dd006db90 0x7f4dd01340c0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f4dcc0097c0 tx=0x7f4dcc01c360 comp rx=0 tx=0).stop 2024-09-16T03:46:07.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.156+0000 7f4dd817c640 1 -- 172.21.15.42:0/821873303 shutdown_connections 2024-09-16T03:46:07.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.156+0000 7f4dd817c640 1 --2- 172.21.15.42:0/821873303 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4dd00a3c00 0x7f4dd0137520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:46:07.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.156+0000 7f4dd817c640 1 --2- 172.21.15.42:0/821873303 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4dd006db90 0x7f4dd01340c0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:46:07.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.156+0000 7f4dd817c640 1 -- 172.21.15.42:0/821873303 >> 172.21.15.42:0/821873303 conn(0x7f4dd00a9fb0 msgr2=0x7f4dd00aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:46:07.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.156+0000 7f4dd817c640 1 -- 172.21.15.42:0/821873303 shutdown_connections 2024-09-16T03:46:07.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.156+0000 7f4dd817c640 1 -- 172.21.15.42:0/821873303 wait complete. 2024-09-16T03:46:07.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.156+0000 7f4dd817c640 10 monclient: build_initial_monmap 2024-09-16T03:46:07.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.156+0000 7f4dd817c640 10 monclient: monmap: 2024-09-16T03:46:07.172 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:46:07.173 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:46:07.173 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:46:07.173 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:46:07.173 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:46:07.173 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:46:07.173 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:46:07.173 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:46:07.173 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:46:07.173 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:46:07.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.156+0000 7f4dd817c640 1 librados: starting msgr at 2024-09-16T03:46:07.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.156+0000 7f4dd817c640 1 librados: starting objecter 2024-09-16T03:46:07.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.156+0000 7f4dd817c640 1 Processor -- start 2024-09-16T03:46:07.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.156+0000 7f4dd817c640 1 -- start start 2024-09-16T03:46:07.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.156+0000 7f4dd817c640 1 librados: setting wanted keys 2024-09-16T03:46:07.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.156+0000 7f4dd817c640 1 librados: calling monclient init 2024-09-16T03:46:07.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.156+0000 7f4dd817c640 10 monclient: init 2024-09-16T03:46:07.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.156+0000 7f4dd817c640 10 monclient: _reopen_session rank -1 2024-09-16T03:46:07.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.156+0000 7f4dd817c640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:46:07.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.156+0000 7f4dd817c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4dd006db90 0x7f4dd00a2f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:46:07.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.156+0000 7f4dd817c640 10 monclient(hunting): picked mon.noname-b con 0x7f4dd006db90 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:46:07.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.156+0000 7f4dd817c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4dd00a37f0 0x7f4dd01cbfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:46:07.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.156+0000 7f4dd817c640 10 monclient(hunting): picked mon.noname-a con 0x7f4dd00a37f0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:46:07.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.156+0000 7f4dd817c640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4dd00a3c00 0x7f4dd01cc4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:46:07.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.156+0000 7f4dd817c640 10 monclient(hunting): picked mon.noname-c con 0x7f4dd00a3c00 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:46:07.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.156+0000 7f4dd817c640 10 monclient(hunting): start opening mon connection 2024-09-16T03:46:07.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.156+0000 7f4dd817c640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f4dd01419d0 con 0x7f4dd00a37f0 2024-09-16T03:46:07.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.156+0000 7f4dd817c640 10 monclient(hunting): start opening mon connection 2024-09-16T03:46:07.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.156+0000 7f4dd817c640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f4dd0137c70 con 0x7f4dd006db90 2024-09-16T03:46:07.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.156+0000 7f4dd817c640 10 monclient(hunting): start opening mon connection 2024-09-16T03:46:07.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.156+0000 7f4dd817c640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f4dd0141b50 con 0x7f4dd00a3c00 2024-09-16T03:46:07.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.156+0000 7f4dd817c640 10 monclient(hunting): _renew_subs 2024-09-16T03:46:07.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.156+0000 7f4dd817c640 10 monclient(hunting): authenticate will time out at 1214.130646s 2024-09-16T03:46:07.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.156+0000 7f4dd6719640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4dd00a3c00 0x7f4dd01cc4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:46:07.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.156+0000 7f4dd5717640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4dd00a37f0 0x7f4dd01cbfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:46:07.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.156+0000 7f4dd6719640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4dd00a3c00 0x7f4dd01cc4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:51720/0 (socket says 172.21.15.42:51720) 2024-09-16T03:46:07.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.156+0000 7f4dd5717640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4dd00a37f0 0x7f4dd01cbfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:44310/0 (socket says 172.21.15.42:44310) 2024-09-16T03:46:07.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.156+0000 7f4dd6719640 1 -- 172.21.15.42:0/2349222517 learned_addr learned my addr 172.21.15.42:0/2349222517 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:46:07.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.156+0000 7f4dd6719640 10 monclient(hunting): get_auth_request con 0x7f4dd00a3c00 auth_method 0 2024-09-16T03:46:07.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.156+0000 7f4dd6719640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:46:07.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.156+0000 7f4dd6719640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:46:07.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.156+0000 7f4dd6719640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:46:07.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.156+0000 7f4dd5717640 10 monclient(hunting): get_auth_request con 0x7f4dd00a37f0 auth_method 0 2024-09-16T03:46:07.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.156+0000 7f4dd5717640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:46:07.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.156+0000 7f4dd5717640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:46:07.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.156+0000 7f4dd5717640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:46:07.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.156+0000 7f4dd6719640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:46:07.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.156+0000 7f4dd6719640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:46:07.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.156+0000 7f4dd6719640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:46:07.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.156+0000 7f4dd5717640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:46:07.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.156+0000 7f4dd5717640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:46:07.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.156+0000 7f4dd5f18640 1 --2- 172.21.15.42:0/2349222517 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4dd006db90 0x7f4dd00a2f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:46:07.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.156+0000 7f4dd5717640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:46:07.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.156+0000 7f4dd5f18640 10 monclient(hunting): get_auth_request con 0x7f4dd006db90 auth_method 0 2024-09-16T03:46:07.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.156+0000 7f4dd5f18640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:46:07.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.156+0000 7f4dd5f18640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:46:07.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.156+0000 7f4dd5f18640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:46:07.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.156+0000 7f4dd6719640 10 monclient(hunting): handle_auth_done global_id 13544 payload 824 2024-09-16T03:46:07.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.156+0000 7f4dd6719640 1 -- 172.21.15.42:0/2349222517 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4dd006db90 msgr2=0x7f4dd00a2f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:46:07.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.156+0000 7f4dd6719640 1 --2- 172.21.15.42:0/2349222517 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4dd006db90 0x7f4dd00a2f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:46:07.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.156+0000 7f4dd6719640 1 -- 172.21.15.42:0/2349222517 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4dd00a37f0 msgr2=0x7f4dd01cbfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:46:07.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.156+0000 7f4dd6719640 1 --2- 172.21.15.42:0/2349222517 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4dd00a37f0 0x7f4dd01cbfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:46:07.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.156+0000 7f4dd6719640 10 monclient: _finish_hunting 0 2024-09-16T03:46:07.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.156+0000 7f4dd6719640 1 monclient: found mon.noname-c 2024-09-16T03:46:07.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.156+0000 7f4dd6719640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:46:07.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.156+0000 7f4dd6719640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:46:07.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.156+0000 7f4dd6719640 1 -- 172.21.15.42:0/2349222517 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4dd006f6a0 con 0x7f4dd00a3c00 2024-09-16T03:46:07.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.156+0000 7f4dd5f18640 1 --2- 172.21.15.42:0/2349222517 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4dd006db90 0x7f4dd00a2f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T03:46:07.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.156+0000 7f4dd6719640 1 --2- 172.21.15.42:0/2349222517 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4dd00a3c00 0x7f4dd01cc4e0 secure :-1 s=READY pgs=2418 cs=0 l=1 rev1=1 crypto rx=0x7f4dc800efc0 tx=0x7f4dc800c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:46:07.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.156+0000 7f4dc67fc640 10 client.?.objecter ms_handle_connect 0x7f4dd00a3c00 2024-09-16T03:46:07.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.156+0000 7f4dc67fc640 10 client.?.objecter resend_mon_ops 2024-09-16T03:46:07.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.156+0000 7f4dc67fc640 1 -- 172.21.15.42:0/2349222517 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dc800cbe0 con 0x7f4dd00a3c00 2024-09-16T03:46:07.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.156+0000 7f4dc67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:46:07.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.156+0000 7f4dc67fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:46:07.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.156+0000 7f4dc67fc640 10 monclient: dump: 2024-09-16T03:46:07.178 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:46:07.178 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:46:07.178 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:07.178 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:07.178 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:46:07.178 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:46:07.179 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:46:07.179 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:46:07.179 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:46:07.179 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:46:07.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.156+0000 7f4dc67fc640 10 monclient: _finish_auth 0 2024-09-16T03:46:07.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.156+0000 7f4dc67fc640 10 monclient: _check_auth_tickets 2024-09-16T03:46:07.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.156+0000 7f4dc67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:46:07.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.156+0000 7f4dc67fc640 1 -- 172.21.15.42:0/2349222517 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4dc800cd80 con 0x7f4dd00a3c00 2024-09-16T03:46:07.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.156+0000 7f4dc67fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:46:07.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.156+0000 7f4dc67fc640 1 -- 172.21.15.42:0/2349222517 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dc8003bc0 con 0x7f4dd00a3c00 2024-09-16T03:46:07.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.156+0000 7f4dd817c640 5 monclient: authenticate success, global_id 13544 2024-09-16T03:46:07.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.156+0000 7f4dd817c640 10 monclient: _renew_subs 2024-09-16T03:46:07.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.156+0000 7f4dd817c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:46:07.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.156+0000 7f4dd817c640 1 -- 172.21.15.42:0/2349222517 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4dd00a0780 con 0x7f4dd00a3c00 2024-09-16T03:46:07.180 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.156+0000 7f4dc67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:46:07.180 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.156+0000 7f4dc67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:46:07.180 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.156+0000 7f4dc77fe640 4 set_mon_vals no callback set 2024-09-16T03:46:07.180 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.156+0000 7f4dc67fc640 10 monclient: dump: 2024-09-16T03:46:07.180 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:46:07.180 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:46:07.180 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:07.180 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:07.180 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:46:07.180 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:46:07.181 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:46:07.181 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:46:07.181 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:46:07.181 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:46:07.181 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.156+0000 7f4dd817c640 10 client.13544.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:46:07.181 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.156+0000 7f4dd817c640 10 monclient: _renew_subs 2024-09-16T03:46:07.181 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.156+0000 7f4dd817c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:46:07.181 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.156+0000 7f4dd817c640 1 -- 172.21.15.42:0/2349222517 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f4dd01cca20 con 0x7f4dd00a3c00 2024-09-16T03:46:07.181 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.156+0000 7f4dd817c640 1 librados: init done 2024-09-16T03:46:07.181 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.160+0000 7f4dd817c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:46:07.181 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.160+0000 7f4dd817c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:46:07.181 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.160+0000 7f4dd817c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:46:07.181 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.160+0000 7f4dc67fc640 1 -- 172.21.15.42:0/2349222517 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f4dc801b030 con 0x7f4dd00a3c00 2024-09-16T03:46:07.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.160+0000 7f4dd817c640 1 -- 172.21.15.42:0/2349222517 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4dd01340c0 con 0x7f4dd00a3c00 2024-09-16T03:46:07.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.160+0000 7f4dc67fc640 1 --2- 172.21.15.42:0/2349222517 >> v2:172.21.15.154:6812/254845342 conn(0x7f4db4037810 0x7f4db4039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:46:07.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.160+0000 7f4dc67fc640 1 -- 172.21.15.42:0/2349222517 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(121..121 src has 1..121) ==== 6576+0+0 (secure 0 0 0) 0x7f4dc8049600 con 0x7f4dd00a3c00 2024-09-16T03:46:07.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.160+0000 7f4dc67fc640 10 client.13544.objecter ms_dispatch 0x7f4dd0000ba0 osd_map(121..121 src has 1..121) 2024-09-16T03:46:07.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.160+0000 7f4dc67fc640 3 client.13544.objecter handle_osd_map got epochs [121,121] > 0 2024-09-16T03:46:07.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.160+0000 7f4dc67fc640 3 client.13544.objecter handle_osd_map decoding full epoch 121 2024-09-16T03:46:07.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.160+0000 7f4dc67fc640 20 client.13544.objecter dump_active .. 0 homeless 2024-09-16T03:46:07.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.160+0000 7f4dd5f18640 1 --2- 172.21.15.42:0/2349222517 >> v2:172.21.15.154:6812/254845342 conn(0x7f4db4037810 0x7f4db4039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:46:07.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.160+0000 7f4dd5f18640 10 monclient: get_auth_request con 0x7f4db4037810 auth_method 0 2024-09-16T03:46:07.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.160+0000 7f4dc67fc640 10 client.13544.objecter ms_handle_connect 0x7f4db4037810 2024-09-16T03:46:07.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.160+0000 7f4dd5f18640 1 --2- 172.21.15.42:0/2349222517 >> v2:172.21.15.154:6812/254845342 conn(0x7f4db4037810 0x7f4db4039cd0 secure :-1 s=READY pgs=3400 cs=0 l=1 rev1=1 crypto rx=0x7f4dcc0ad000 tx=0x7f4dcc008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:46:07.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.164+0000 7f4dc67fc640 1 -- 172.21.15.42:0/2349222517 <== mon.2 v2:172.21.15.42:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f4dc8022020 con 0x7f4dd00a3c00 2024-09-16T03:46:07.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.164+0000 7f4dc67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:46:07.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.164+0000 7f4dc67fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:46:07.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.392+0000 7f4dd817c640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T03:46:07.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.392+0000 7f4dd817c640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T03:46:07.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.392+0000 7f4dd817c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:46:07.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.392+0000 7f4dd817c640 1 -- 172.21.15.42:0/2349222517 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f4dd00a2620 con 0x7f4dd00a3c00 2024-09-16T03:46:07.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.396+0000 7f4dc67fc640 1 -- 172.21.15.42:0/2349222517 <== mon.2 v2:172.21.15.42:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f4dc8010040 con 0x7f4dd00a3c00 2024-09-16T03:46:07.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.396+0000 7f4dc67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T03:46:07.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.396+0000 7f4dc67fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:46:07.403 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:46:07.396+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:46:07.403 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:46:07.396+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=28,config=2,health=11,kv=10,logm=49} crc {auth=4136992891,config=692619927,health=90558344,kv=3446188498,logm=1860824718}) 2024-09-16T03:46:07.403 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:46:07.396+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=26,config=2,health=11,kv=10,logm=51} crc {auth=3332500693,config=692619927,health=3538214632,kv=3446188498,logm=2313569201}) 2024-09-16T03:46:07.403 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:46:07.396+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:46:07.404 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:46:07.396+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=28,config=2,health=11,kv=10,logm=49} crc {auth=4136992891,config=692619927,health=90558344,kv=3446188498,logm=1860824718}) 2024-09-16T03:46:07.404 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:46:07.396+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=26,config=2,health=12,kv=10,logm=50} crc {auth=1734950772,config=692619927,health=117272779,kv=3446188498,logm=810027094}) 2024-09-16T03:46:07.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.400+0000 7f4dd817c640 10 librados: watch_flush enter 2024-09-16T03:46:07.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.400+0000 7f4dd817c640 10 librados: watch_flush exit 2024-09-16T03:46:07.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.400+0000 7f4dd817c640 20 client.13544.objecter shutdown clearing up homeless session... 2024-09-16T03:46:07.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.400+0000 7f4dd817c640 10 client.13544.objecter successfully canceled tick 2024-09-16T03:46:07.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.400+0000 7f4dd817c640 1 -- 172.21.15.42:0/2349222517 >> v2:172.21.15.154:6812/254845342 conn(0x7f4db4037810 msgr2=0x7f4db4039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:46:07.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.400+0000 7f4dd817c640 1 --2- 172.21.15.42:0/2349222517 >> v2:172.21.15.154:6812/254845342 conn(0x7f4db4037810 0x7f4db4039cd0 secure :-1 s=READY pgs=3400 cs=0 l=1 rev1=1 crypto rx=0x7f4dcc0ad000 tx=0x7f4dcc008040 comp rx=0 tx=0).stop 2024-09-16T03:46:07.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.400+0000 7f4dd817c640 10 monclient: shutdown 2024-09-16T03:46:07.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.400+0000 7f4dd817c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:46:07.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.400+0000 7f4dd817c640 1 -- 172.21.15.42:0/2349222517 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4dd00a3c00 msgr2=0x7f4dd01cc4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:46:07.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.400+0000 7f4dd817c640 1 --2- 172.21.15.42:0/2349222517 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4dd00a3c00 0x7f4dd01cc4e0 secure :-1 s=READY pgs=2418 cs=0 l=1 rev1=1 crypto rx=0x7f4dc800efc0 tx=0x7f4dc800c490 comp rx=0 tx=0).stop 2024-09-16T03:46:07.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.400+0000 7f4dd817c640 1 -- 172.21.15.42:0/2349222517 shutdown_connections 2024-09-16T03:46:07.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.400+0000 7f4dd817c640 1 --2- 172.21.15.42:0/2349222517 >> v2:172.21.15.154:6812/254845342 conn(0x7f4db4037810 0x7f4db4039cd0 unknown :-1 s=CLOSED pgs=3400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:46:07.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.400+0000 7f4dd817c640 1 --2- 172.21.15.42:0/2349222517 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4dd00a3c00 0x7f4dd01cc4e0 unknown :-1 s=CLOSED pgs=2418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:46:07.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.400+0000 7f4dd817c640 1 --2- 172.21.15.42:0/2349222517 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4dd00a37f0 0x7f4dd01cbfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:46:07.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.400+0000 7f4dd817c640 1 --2- 172.21.15.42:0/2349222517 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4dd006db90 0x7f4dd00a2f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:46:07.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.400+0000 7f4dd817c640 1 -- 172.21.15.42:0/2349222517 >> 172.21.15.42:0/2349222517 conn(0x7f4dd00a9fb0 msgr2=0x7f4dd00a7ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:46:07.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.400+0000 7f4dd817c640 1 -- 172.21.15.42:0/2349222517 shutdown_connections 2024-09-16T03:46:07.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.400+0000 7f4dd817c640 1 -- 172.21.15.42:0/2349222517 wait complete. 2024-09-16T03:46:07.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:07.400+0000 7f4dd817c640 1 librados: shutdown 2024-09-16T03:46:07.408 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:46:07.400+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:46:07.408 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:46:07.400+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2328052943}) 2024-09-16T03:46:07.408 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:46:07.400+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2034516508}) 2024-09-16T03:46:07.408 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:46:07.400+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:46:07.408 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:46:07.400+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2328052943}) 2024-09-16T03:46:07.408 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:46:07.400+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2455657015}) 2024-09-16T03:46:07.408 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:46:07.400+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:46:07.408 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:46:07.400+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=150834995}) 2024-09-16T03:46:07.408 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:46:07.400+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=793980529}) 2024-09-16T03:46:07.409 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:46:07.400+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:46:07.409 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:46:07.400+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=150834995}) 2024-09-16T03:46:07.409 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:46:07.400+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2074884409}) 2024-09-16T03:46:07.410 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:46:07.404+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:46:07.410 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:46:07.404+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1433280468}) 2024-09-16T03:46:07.410 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:46:07.404+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2726444452}) 2024-09-16T03:46:07.410 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:46:07.404+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:46:07.410 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:46:07.404+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1433280468}) 2024-09-16T03:46:07.410 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:46:07.404+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3384557132}) 2024-09-16T03:46:07.412 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:46:07.404+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:46:07.412 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:46:07.404+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3662224964}) 2024-09-16T03:46:07.412 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:46:07.404+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1729866943}) 2024-09-16T03:46:07.412 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:46:07.404+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:46:07.412 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:46:07.404+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3662224964}) 2024-09-16T03:46:07.412 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:46:07.404+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=278657429}) 2024-09-16T03:46:07.415 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:46:07.408+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:46:07.415 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:46:07.408+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3665782951}) 2024-09-16T03:46:07.415 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:46:07.408+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3693727771}) 2024-09-16T03:46:07.415 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:46:07.408+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:46:07.415 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:46:07.408+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3665782951}) 2024-09-16T03:46:07.415 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:46:07.408+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2126714860}) 2024-09-16T03:46:07.416 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:46:07.408+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:46:07.416 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:46:07.408+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4004608189}) 2024-09-16T03:46:07.416 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:46:07.408+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1159031412}) 2024-09-16T03:46:07.416 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:46:07.408+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:46:07.416 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:46:07.408+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4004608189}) 2024-09-16T03:46:07.417 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:46:07.408+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2971270300}) 2024-09-16T03:46:07.418 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:46:07.412+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:46:07.418 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:46:07.412+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2864458875}) 2024-09-16T03:46:07.418 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:46:07.412+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3443611347}) 2024-09-16T03:46:07.418 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:46:07.412+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:46:07.418 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:46:07.412+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2864458875}) 2024-09-16T03:46:07.419 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:46:07.412+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1333129716}) 2024-09-16T03:46:07.420 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:46:07.412+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:46:07.420 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:46:07.412+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=881336399}) 2024-09-16T03:46:07.420 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:46:07.412+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3751855772}) 2024-09-16T03:46:07.420 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:46:07.412+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:46:07.421 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:46:07.412+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=881336399}) 2024-09-16T03:46:07.421 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:46:07.412+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1603640021}) 2024-09-16T03:46:07.422 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:46:07.416+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:46:07.423 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:46:07.416+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3410167681}) 2024-09-16T03:46:07.423 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:46:07.416+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2908465478}) 2024-09-16T03:46:07.423 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:46:07.416+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:46:07.423 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:46:07.416+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3410167681}) 2024-09-16T03:46:07.423 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:46:07.416+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2748353053}) 2024-09-16T03:46:07.424 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T03:46:07.424 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:46:07.416+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:46:07.425 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:46:07.416+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2601521633}) 2024-09-16T03:46:07.425 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:46:07.416+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1631993926}) 2024-09-16T03:46:07.425 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:46:07.416+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:46:07.425 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:46:07.416+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2601521633}) 2024-09-16T03:46:07.425 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:46:07.416+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=760803250}) 2024-09-16T03:46:07.426 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:46:07.420+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:46:07.426 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:46:07.420+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3051575906}) 2024-09-16T03:46:07.426 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:46:07.420+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1826261025}) 2024-09-16T03:46:07.427 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:46:07.420+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:46:07.427 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:46:07.420+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3051575906}) 2024-09-16T03:46:07.427 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:46:07.420+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2684707492}) 2024-09-16T03:46:07.428 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:46:07.420+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:46:07.429 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:46:07.420+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1671796756}) 2024-09-16T03:46:07.429 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:46:07.420+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3844474680}) 2024-09-16T03:46:07.429 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:46:07.424+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:46:07.429 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:46:07.424+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1671796756}) 2024-09-16T03:46:07.429 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:46:07.424+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4161325522}) 2024-09-16T03:46:07.431 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:46:07.424+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:46:07.431 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:46:07.424+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=458742516}) 2024-09-16T03:46:07.431 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:46:07.424+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=26612018}) 2024-09-16T03:46:07.431 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:46:07.424+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:46:07.431 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:46:07.424+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=458742516}) 2024-09-16T03:46:07.432 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:46:07.424+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2273454375}) 2024-09-16T03:46:07.433 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:46:07.424+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:46:07.433 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:46:07.424+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1192580360}) 2024-09-16T03:46:07.433 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:46:07.424+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2197885753}) 2024-09-16T03:46:07.433 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:46:07.424+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:46:07.433 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:46:07.424+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1192580360}) 2024-09-16T03:46:07.433 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:46:07.424+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1081783167}) 2024-09-16T03:46:07.435 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:46:07.428+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:46:07.435 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:46:07.428+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1258017042}) 2024-09-16T03:46:07.435 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:46:07.428+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2863306570}) 2024-09-16T03:46:07.435 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:46:07.428+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:46:07.436 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:46:07.428+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1258017042}) 2024-09-16T03:46:07.436 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:46:07.428+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3313569168}) 2024-09-16T03:46:07.437 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:46:07.428+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:46:07.437 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:46:07.428+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3588329727}) 2024-09-16T03:46:07.437 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:46:07.428+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=482933055}) 2024-09-16T03:46:07.437 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:46:07.428+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:46:07.437 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:46:07.428+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3588329727}) 2024-09-16T03:46:07.437 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:46:07.428+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=441997579}) 2024-09-16T03:46:07.438 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:46:07.432+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:46:07.439 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:46:07.432+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1955225029}) 2024-09-16T03:46:07.439 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:46:07.432+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1773379972}) 2024-09-16T03:46:07.439 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:46:07.432+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:46:07.439 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:46:07.432+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1955225029}) 2024-09-16T03:46:07.439 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:46:07.432+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3648340674}) 2024-09-16T03:46:07.440 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:46:07.432+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:46:07.441 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:46:07.432+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1561789549}) 2024-09-16T03:46:07.441 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:46:07.432+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=486207272}) 2024-09-16T03:46:07.441 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:46:07.432+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:46:07.441 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:46:07.432+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1561789549}) 2024-09-16T03:46:07.441 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:46:07.432+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=438125565}) 2024-09-16T03:46:07.442 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:46:07.436+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:46:07.443 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:46:07.436+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=302630416}) 2024-09-16T03:46:07.443 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:46:07.436+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=935671445}) 2024-09-16T03:46:07.443 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:46:07.436+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:46:07.443 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:46:07.436+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=302630416}) 2024-09-16T03:46:07.443 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:46:07.436+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2960745472}) 2024-09-16T03:46:07.445 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:46:07.440+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:46:07.445 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:46:07.440+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=74,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=4} crc {logm=2311823898,mdsmap=4088524775,mgr=366503433,mgr_command_descs=2107673600,mgr_metadata=1367272202,mgrstat=649647926,mon_config_key=2532506080}) 2024-09-16T03:46:07.445 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:46:07.440+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=78,mdsmap=2,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=2} crc {logm=717083570,mdsmap=3140514009,mgr=1406439472,mgr_command_descs=2107673600,mgr_metadata=1367272202,mgrstat=2822735435,mon_config_key=1452145213}) 2024-09-16T03:46:07.445 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:46:07.440+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:46:07.446 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:46:07.440+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=74,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=4} crc {logm=2311823898,mdsmap=4088524775,mgr=366503433,mgr_command_descs=2107673600,mgr_metadata=1367272202,mgrstat=649647926,mon_config_key=2532506080}) 2024-09-16T03:46:07.446 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:46:07.440+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=76,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=2} crc {logm=2509178773,mdsmap=4088524775,mgr=2815798751,mgr_command_descs=2107673600,mgr_metadata=1367272202,mgrstat=649647926,mon_config_key=1452145213}) 2024-09-16T03:46:07.448 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:46:07.440+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:46:07.448 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:46:07.440+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mon_config_key=6,monmap=3,osd_metadata=5,osd_snap=21,osdmap=65} crc {mon_config_key=2877378336,monmap=3937513981,osd_metadata=4196947761,osd_snap=2700444072,osdmap=3801755076}) 2024-09-16T03:46:07.448 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:46:07.440+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mon_config_key=6,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=24,osdmap=60} crc {mon_config_key=1070296285,monmap=3937513981,osd_metadata=4182959540,osd_pg_creating=943183274,osd_snap=613327586,osdmap=899772304}) 2024-09-16T03:46:07.448 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:46:07.440+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:46:07.448 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:46:07.440+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mon_config_key=6,monmap=3,osd_metadata=5,osd_snap=21,osdmap=65} crc {mon_config_key=2877378336,monmap=3937513981,osd_metadata=4196947761,osd_snap=2700444072,osdmap=3801755076}) 2024-09-16T03:46:07.448 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:46:07.440+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mon_config_key=7,monmap=2,osd_metadata=6,osd_pg_creating=1,osd_snap=23,osdmap=61} crc {mon_config_key=160474886,monmap=268092033,osd_metadata=4182959540,osd_pg_creating=943183274,osd_snap=1850586524,osdmap=1328992835}) 2024-09-16T03:46:07.451 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:46:07.444+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:46:07.451 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:46:07.444+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=521231118}) 2024-09-16T03:46:07.451 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:46:07.444+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1715645591}) 2024-09-16T03:46:07.451 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:46:07.444+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:46:07.452 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:46:07.444+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=521231118}) 2024-09-16T03:46:07.452 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:46:07.444+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3139023976}) 2024-09-16T03:46:07.464 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:46:07.456+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:46:07.464 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:46:07.456+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=27} crc {osdmap=695112984}) 2024-09-16T03:46:07.465 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:46:07.456+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=28} crc {osdmap=3215262022}) 2024-09-16T03:46:07.465 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:46:07.456+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:46:07.465 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:46:07.456+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=27} crc {osdmap=695112984}) 2024-09-16T03:46:07.465 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:46:07.456+0000 7f8e11076640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=29} crc {osdmap=3740740198}) 2024-09-16T03:46:07.838 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio_pp: [ OK ] LibRadosts_pp: [ RUN ] LibRadosSnapshotsECPP.SnapGetNamePP 2024-09-16T03:46:07.838 INFO:tasks.workunit.client.0.smithi154.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapGetNamePP (2023 ms) 2024-09-16T03:46:07.838 INFO:tasks.workunit.client.0.smithi154.stdout: api_snapshots_pp: [----------] 4 tests from LibRadosSnapshotsECPP (8553 ms total) 2024-09-16T03:46:07.838 INFO:tasks.workunit.client.0.smithi154.stdout: api_snapshots_pp: 2024-09-16T03:46:07.838 INFO:tasks.workunit.client.0.smithi154.stdout: api_snapshots_pp: [----------] 3 tests from LibRadosSnapshotsSelfManagedECPP 2024-09-16T03:46:07.838 INFO:tasks.workunit.client.0.smithi154.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.SnapPP 2024-09-16T03:46:07.838 INFO:tasks.workunit.client.0.smithi154.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.SnapPP (4061 ms) 2024-09-16T03:46:07.838 INFO:tasks.workunit.client.0.smithi154.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.RollbackPP 2024-09-16T03:46:07.838 INFO:tasks.workunit.client.0.smithi154.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.RollbackPP (4005 ms) 2024-09-16T03:46:07.839 INFO:tasks.workunit.client.0.smithi154.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.Bug11677 2024-09-16T03:46:07.839 INFO:tasks.workunit.client.0.smithi154.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.Bug11677 (4031 ms) 2024-09-16T03:46:07.839 INFO:tasks.workunit.client.0.smithi154.stdout: api_snapshots_pp: [----------] 3 tests from LibRadosSnapshotsSelfManagedECPP (12097 ms total) 2024-09-16T03:46:07.839 INFO:tasks.workunit.client.0.smithi154.stdout: api_snapshots_pp: 2024-09-16T03:46:07.839 INFO:tasks.workunit.client.0.smithi154.stdout: api_snapshots_pp: [----------] Global test environment tear-down 2024-09-16T03:46:07.839 INFO:tasks.workunit.client.0.smithi154.stdout: api_snapshots_pp: [==========] 21 tests from 5 test suites ran. (158202 ms total) 2024-09-16T03:46:07.839 INFO:tasks.workunit.client.0.smithi154.stdout: api_snapshots_pp: [ PASSED ] 20 tests. 2024-09-16T03:46:07.839 INFO:tasks.workunit.client.0.smithi154.stdout: api_snapshots_pp: [ SKIPPED ] 1 test, listed below: 2024-09-16T03:46:07.839 INFO:tasks.workunit.client.0.smithi154.stdout: api_snapshots_pp: [ SKIPPED ] LibRadosSnapshotsSelfManagedPP.WriteRollback 2024-09-16T03:46:08.424 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T03:46:08.425 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:46:08.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.548+0000 7f39c17e9640 10 monclient: get_monmap_and_config 2024-09-16T03:46:08.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.548+0000 7f39c17e9640 10 monclient: build_initial_monmap 2024-09-16T03:46:08.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.548+0000 7f39c17e9640 10 monclient: monmap: 2024-09-16T03:46:08.555 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:46:08.555 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:46:08.555 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:46:08.554305+0000 2024-09-16T03:46:08.555 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:46:08.554305+0000 2024-09-16T03:46:08.555 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:46:08.556 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:46:08.556 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:46:08.556 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:46:08.556 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:46:08.556 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:46:08.556 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:46:08.556 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:46:08.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.548+0000 7f39c17e9640 1 Processor -- start 2024-09-16T03:46:08.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.548+0000 7f39c17e9640 1 -- start start 2024-09-16T03:46:08.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.548+0000 7f39c17e9640 10 monclient: init 2024-09-16T03:46:08.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.548+0000 7f39c17e9640 10 monclient: _reopen_session rank -1 2024-09-16T03:46:08.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.548+0000 7f39c17e9640 10 monclient: _add_conns ranks=[0,2,3,4,1] 2024-09-16T03:46:08.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.548+0000 7f39c17e9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f39bc0a1330 addr v1:172.21.15.42:6789/0 2024-09-16T03:46:08.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.548+0000 7f39c17e9640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f39bc06dac0 0x7f39bc13a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:46:08.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.548+0000 7f39c17e9640 10 monclient(hunting): picked mon.noname-b con 0x7f39bc06dac0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:46:08.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.548+0000 7f39c17e9640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f39bc0a1e20 0x7f39bc13c890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:46:08.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.548+0000 7f39c17e9640 10 monclient(hunting): picked mon.noname-a con 0x7f39bc0a1e20 addr v2:172.21.15.42:3300/0 2024-09-16T03:46:08.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.548+0000 7f39c17e9640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f39bc09c7d0 con 0x7f39bc0a1330 2024-09-16T03:46:08.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.548+0000 7f39c17e9640 10 monclient(hunting): start opening mon connection 2024-09-16T03:46:08.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.548+0000 7f39c17e9640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f39bc09c090 con 0x7f39bc0a1e20 2024-09-16T03:46:08.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.548+0000 7f39c17e9640 10 monclient(hunting): start opening mon connection 2024-09-16T03:46:08.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.548+0000 7f39c17e9640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f39bc09f610 con 0x7f39bc06dac0 2024-09-16T03:46:08.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.548+0000 7f39c17e9640 10 monclient(hunting): _renew_subs 2024-09-16T03:46:08.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.552+0000 7f39c17e9640 10 monclient(hunting): authenticate will time out at 1215.524010s 2024-09-16T03:46:08.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.552+0000 7f39baffd640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f39bc0a1330 0x7f39bc0a1710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:48168/0 (socket says 172.21.15.42:48168) 2024-09-16T03:46:08.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.552+0000 7f39baffd640 1 -- 172.21.15.42:0/1817205599 learned_addr learned my addr 172.21.15.42:0/1817205599 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:46:08.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.552+0000 7f39bb7fe640 1 --2- 172.21.15.42:0/1817205599 >> v2:172.21.15.42:3300/0 conn(0x7f39bc0a1e20 0x7f39bc13c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:46:08.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.552+0000 7f39bb7fe640 10 monclient(hunting): get_auth_request con 0x7f39bc0a1e20 auth_method 0 2024-09-16T03:46:08.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.552+0000 7f39bb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:46:08.560 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.552+0000 7f39bb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:46:08.560 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.552+0000 7f39bb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:46:08.560 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.552+0000 7f39ba7fc640 1 --2- 172.21.15.42:0/1817205599 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f39bc06dac0 0x7f39bc13a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:46:08.560 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.552+0000 7f39ba7fc640 10 monclient(hunting): get_auth_request con 0x7f39bc06dac0 auth_method 0 2024-09-16T03:46:08.560 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.552+0000 7f39ba7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:46:08.560 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.552+0000 7f39ba7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:46:08.560 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.552+0000 7f39ba7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:46:08.560 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.552+0000 7f39bb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:46:08.560 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.552+0000 7f39bb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:46:08.560 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.552+0000 7f39bb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:46:08.560 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.552+0000 7f39ba7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:46:08.560 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.552+0000 7f39ba7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:46:08.560 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.552+0000 7f39ba7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:46:08.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.552+0000 7f39ba7fc640 10 monclient(hunting): handle_auth_done global_id 13553 payload 274 2024-09-16T03:46:08.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.552+0000 7f39ba7fc640 1 -- 172.21.15.42:0/1817205599 >> v2:172.21.15.42:3300/0 conn(0x7f39bc0a1e20 msgr2=0x7f39bc13c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:46:08.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.552+0000 7f39ba7fc640 1 --2- 172.21.15.42:0/1817205599 >> v2:172.21.15.42:3300/0 conn(0x7f39bc0a1e20 0x7f39bc13c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:46:08.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.552+0000 7f39ba7fc640 1 -- 172.21.15.42:0/1817205599 >> v1:172.21.15.42:6789/0 conn(0x7f39bc0a1330 legacy=0x7f39bc0a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:46:08.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.552+0000 7f39ba7fc640 10 monclient: _finish_hunting 0 2024-09-16T03:46:08.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.552+0000 7f39ba7fc640 1 monclient: found mon.noname-b 2024-09-16T03:46:08.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.552+0000 7f39ba7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:46:08.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.552+0000 7f39ba7fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T03:46:08.565 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.552+0000 7f39ba7fc640 1 -- 172.21.15.42:0/1817205599 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39bc09f8f0 con 0x7f39bc06dac0 2024-09-16T03:46:08.565 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.552+0000 7f39bb7fe640 1 --2- 172.21.15.42:0/1817205599 >> v2:172.21.15.42:3300/0 conn(0x7f39bc0a1e20 0x7f39bc13c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:46:08.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.552+0000 7f39ba7fc640 1 --2- 172.21.15.42:0/1817205599 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f39bc06dac0 0x7f39bc13a340 secure :-1 s=READY pgs=2422 cs=0 l=1 rev1=1 crypto rx=0x7f39b0009d90 tx=0x7f39b001c920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fe7db9ef668124e9 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:46:08.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.552+0000 7f39b9ffb640 1 -- 172.21.15.42:0/1817205599 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39b00b2070 con 0x7f39bc06dac0 2024-09-16T03:46:08.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.552+0000 7f39b9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:46:08.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.552+0000 7f39b9ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:46:08.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.552+0000 7f39b9ffb640 10 monclient: dump: 2024-09-16T03:46:08.571 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:46:08.571 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:46:08.571 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:08.571 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:08.572 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:46:08.572 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:46:08.572 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:46:08.572 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:46:08.572 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:46:08.572 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:46:08.572 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.552+0000 7f39b9ffb640 10 monclient: _finish_auth 0 2024-09-16T03:46:08.572 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.552+0000 7f39b9ffb640 10 monclient: _check_auth_tickets 2024-09-16T03:46:08.572 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.552+0000 7f39b9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:46:08.573 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.552+0000 7f39b9ffb640 1 -- 172.21.15.42:0/1817205599 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f39b00a5a00 con 0x7f39bc06dac0 2024-09-16T03:46:08.573 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.552+0000 7f39b9ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T03:46:08.573 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.552+0000 7f39b9ffb640 1 -- 172.21.15.42:0/1817205599 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39b00a5dd0 con 0x7f39bc06dac0 2024-09-16T03:46:08.573 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.552+0000 7f39c17e9640 5 monclient: authenticate success, global_id 13553 2024-09-16T03:46:08.573 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.552+0000 7f39c17e9640 10 monclient: get_monmap_and_config success 2024-09-16T03:46:08.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.552+0000 7f39c17e9640 4 set_mon_vals no callback set 2024-09-16T03:46:08.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.552+0000 7f39b9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:46:08.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.552+0000 7f39b9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:46:08.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.552+0000 7f39b9ffb640 10 monclient: dump: 2024-09-16T03:46:08.574 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:46:08.574 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:46:08.575 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:08.575 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:08.575 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:46:08.575 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:46:08.575 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:46:08.575 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:46:08.575 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:46:08.575 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:46:08.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.552+0000 7f39c17e9640 10 monclient: shutdown 2024-09-16T03:46:08.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.552+0000 7f39c17e9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:46:08.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.552+0000 7f39c17e9640 1 -- 172.21.15.42:0/1817205599 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f39bc06dac0 msgr2=0x7f39bc13a340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:46:08.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.552+0000 7f39c17e9640 1 --2- 172.21.15.42:0/1817205599 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f39bc06dac0 0x7f39bc13a340 secure :-1 s=READY pgs=2422 cs=0 l=1 rev1=1 crypto rx=0x7f39b0009d90 tx=0x7f39b001c920 comp rx=0 tx=0).stop 2024-09-16T03:46:08.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.552+0000 7f39c17e9640 1 -- 172.21.15.42:0/1817205599 shutdown_connections 2024-09-16T03:46:08.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.552+0000 7f39c17e9640 1 --2- 172.21.15.42:0/1817205599 >> v2:172.21.15.42:3300/0 conn(0x7f39bc0a1e20 0x7f39bc13c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:46:08.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.552+0000 7f39c17e9640 1 --2- 172.21.15.42:0/1817205599 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f39bc06dac0 0x7f39bc13a340 unknown :-1 s=CLOSED pgs=2422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:46:08.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.552+0000 7f39c17e9640 1 -- 172.21.15.42:0/1817205599 >> 172.21.15.42:0/1817205599 conn(0x7f39bc0a2420 msgr2=0x7f39bc0a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:46:08.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.552+0000 7f39c17e9640 1 -- 172.21.15.42:0/1817205599 shutdown_connections 2024-09-16T03:46:08.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.552+0000 7f39c17e9640 1 -- 172.21.15.42:0/1817205599 wait complete. 2024-09-16T03:46:08.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.556+0000 7f39c17e9640 10 monclient: build_initial_monmap 2024-09-16T03:46:08.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.556+0000 7f39c17e9640 10 monclient: monmap: 2024-09-16T03:46:08.576 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:46:08.576 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:46:08.576 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:46:08.576 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:46:08.576 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:46:08.576 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:46:08.577 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:46:08.577 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:46:08.577 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:46:08.577 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:46:08.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.556+0000 7f39c17e9640 1 librados: starting msgr at 2024-09-16T03:46:08.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.556+0000 7f39c17e9640 1 librados: starting objecter 2024-09-16T03:46:08.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.556+0000 7f39c17e9640 1 Processor -- start 2024-09-16T03:46:08.578 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.556+0000 7f39c17e9640 1 -- start start 2024-09-16T03:46:08.578 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.556+0000 7f39c17e9640 1 librados: setting wanted keys 2024-09-16T03:46:08.578 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.556+0000 7f39c17e9640 1 librados: calling monclient init 2024-09-16T03:46:08.578 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.556+0000 7f39c17e9640 10 monclient: init 2024-09-16T03:46:08.578 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.556+0000 7f39c17e9640 10 monclient: _reopen_session rank -1 2024-09-16T03:46:08.578 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.556+0000 7f39c17e9640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:46:08.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.556+0000 7f39c17e9640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f39bc0a1330 0x7f39bc1ccbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:46:08.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.556+0000 7f39c17e9640 10 monclient(hunting): picked mon.noname-c con 0x7f39bc0a1330 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:46:08.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.556+0000 7f39c17e9640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f39bc0a1e20 0x7f39bc1323f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:46:08.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.556+0000 7f39c17e9640 10 monclient(hunting): picked mon.noname-b con 0x7f39bc0a1e20 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:46:08.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.556+0000 7f39c17e9640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f39bc139200 0x7f39bc1396b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:46:08.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.556+0000 7f39c17e9640 10 monclient(hunting): picked mon.noname-a con 0x7f39bc139200 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:46:08.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.556+0000 7f39c17e9640 10 monclient(hunting): start opening mon connection 2024-09-16T03:46:08.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.556+0000 7f39c17e9640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f39bc13f770 con 0x7f39bc139200 2024-09-16T03:46:08.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.556+0000 7f39c17e9640 10 monclient(hunting): start opening mon connection 2024-09-16T03:46:08.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.556+0000 7f39c17e9640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f39bc13f5f0 con 0x7f39bc0a1e20 2024-09-16T03:46:08.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.556+0000 7f39c17e9640 10 monclient(hunting): start opening mon connection 2024-09-16T03:46:08.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.556+0000 7f39c17e9640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f39bc13f8f0 con 0x7f39bc0a1330 2024-09-16T03:46:08.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.556+0000 7f39c17e9640 10 monclient(hunting): _renew_subs 2024-09-16T03:46:08.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.556+0000 7f39c17e9640 10 monclient(hunting): authenticate will time out at 1215.529267s 2024-09-16T03:46:08.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.556+0000 7f39baffd640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f39bc0a1330 0x7f39bc1ccbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:46:08.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.556+0000 7f39bb7fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f39bc139200 0x7f39bc1396b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:46:08.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.556+0000 7f39bb7fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f39bc139200 0x7f39bc1396b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:44328/0 (socket says 172.21.15.42:44328) 2024-09-16T03:46:08.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.556+0000 7f39bb7fe640 1 -- 172.21.15.42:0/1322426247 learned_addr learned my addr 172.21.15.42:0/1322426247 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:46:08.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.556+0000 7f39bb7fe640 10 monclient(hunting): get_auth_request con 0x7f39bc139200 auth_method 0 2024-09-16T03:46:08.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.556+0000 7f39bb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:46:08.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.556+0000 7f39bb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:46:08.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.556+0000 7f39baffd640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f39bc0a1330 0x7f39bc1ccbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:51744/0 (socket says 172.21.15.42:51744) 2024-09-16T03:46:08.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.556+0000 7f39bb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:46:08.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.556+0000 7f39baffd640 10 monclient(hunting): get_auth_request con 0x7f39bc0a1330 auth_method 0 2024-09-16T03:46:08.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.556+0000 7f39baffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:46:08.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.556+0000 7f39baffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:46:08.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.556+0000 7f39baffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:46:08.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.556+0000 7f39bb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:46:08.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.556+0000 7f39bb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:46:08.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.556+0000 7f39bb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:46:08.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.556+0000 7f39baffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:46:08.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.556+0000 7f39baffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:46:08.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.556+0000 7f39baffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:46:08.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.556+0000 7f39bb7fe640 10 monclient(hunting): handle_auth_done global_id 744411 payload 824 2024-09-16T03:46:08.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.556+0000 7f39bb7fe640 1 -- 172.21.15.42:0/1322426247 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f39bc0a1330 msgr2=0x7f39bc1ccbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:46:08.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.556+0000 7f39bb7fe640 1 --2- 172.21.15.42:0/1322426247 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f39bc0a1330 0x7f39bc1ccbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:46:08.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.556+0000 7f39bb7fe640 1 -- 172.21.15.42:0/1322426247 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f39bc0a1e20 msgr2=0x7f39bc1323f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:46:08.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.556+0000 7f39ba7fc640 1 --2- 172.21.15.42:0/1322426247 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f39bc0a1e20 0x7f39bc1323f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:46:08.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.556+0000 7f39bb7fe640 1 --2- 172.21.15.42:0/1322426247 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f39bc0a1e20 0x7f39bc1323f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:46:08.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.556+0000 7f39bb7fe640 10 monclient: _finish_hunting 0 2024-09-16T03:46:08.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.556+0000 7f39bb7fe640 1 monclient: found mon.noname-a 2024-09-16T03:46:08.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.556+0000 7f39bb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:46:08.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.556+0000 7f39bb7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:46:08.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.556+0000 7f39bb7fe640 1 -- 172.21.15.42:0/1322426247 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39bc11caa0 con 0x7f39bc139200 2024-09-16T03:46:08.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.556+0000 7f39baffd640 1 --2- 172.21.15.42:0/1322426247 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f39bc0a1330 0x7f39bc1ccbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:46:08.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.556+0000 7f39ba7fc640 10 monclient: get_auth_request con 0x7f39bc0a1e20 auth_method 0 2024-09-16T03:46:08.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.556+0000 7f39ba7fc640 1 --2- 172.21.15.42:0/1322426247 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f39bc0a1e20 0x7f39bc1323f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T03:46:08.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.556+0000 7f39bb7fe640 1 --2- 172.21.15.42:0/1322426247 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f39bc139200 0x7f39bc1396b0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f39b400e950 tx=0x7f39b400ee10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:46:08.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.556+0000 7f39a37fe640 10 client.?.objecter ms_handle_connect 0x7f39bc139200 2024-09-16T03:46:08.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.556+0000 7f39a37fe640 10 client.?.objecter resend_mon_ops 2024-09-16T03:46:08.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.556+0000 7f39a37fe640 1 -- 172.21.15.42:0/1322426247 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39b4017070 con 0x7f39bc139200 2024-09-16T03:46:08.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.556+0000 7f39a37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:46:08.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.556+0000 7f39a37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:46:08.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.556+0000 7f39a37fe640 10 monclient: dump: 2024-09-16T03:46:08.586 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:46:08.586 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:46:08.586 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:08.586 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:08.587 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:46:08.587 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:46:08.587 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:46:08.587 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:46:08.587 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:46:08.587 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:46:08.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.556+0000 7f39a37fe640 10 monclient: _finish_auth 0 2024-09-16T03:46:08.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.556+0000 7f39a37fe640 10 monclient: _check_auth_tickets 2024-09-16T03:46:08.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.556+0000 7f39a37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:46:08.588 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.556+0000 7f39a37fe640 1 -- 172.21.15.42:0/1322426247 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f39b400ca00 con 0x7f39bc139200 2024-09-16T03:46:08.588 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.556+0000 7f39c17e9640 5 monclient: authenticate success, global_id 744411 2024-09-16T03:46:08.588 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.556+0000 7f39c17e9640 10 monclient: _renew_subs 2024-09-16T03:46:08.588 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.556+0000 7f39c17e9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:46:08.588 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.556+0000 7f39c17e9640 1 -- 172.21.15.42:0/1322426247 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39bc11cd80 con 0x7f39bc139200 2024-09-16T03:46:08.589 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.556+0000 7f39a37fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:46:08.589 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.556+0000 7f39a37fe640 1 -- 172.21.15.42:0/1322426247 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39b400cdd0 con 0x7f39bc139200 2024-09-16T03:46:08.589 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.556+0000 7f39a37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:46:08.589 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.556+0000 7f39a37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:46:08.589 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.556+0000 7f39a37fe640 10 monclient: dump: 2024-09-16T03:46:08.590 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:46:08.590 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:46:08.590 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:08.590 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:08.590 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:46:08.590 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:46:08.590 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:46:08.590 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:46:08.590 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:46:08.591 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:46:08.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.556+0000 7f39b97fa640 4 set_mon_vals no callback set 2024-09-16T03:46:08.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.556+0000 7f39c17e9640 10 client.744411.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:46:08.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.556+0000 7f39c17e9640 10 monclient: _renew_subs 2024-09-16T03:46:08.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.556+0000 7f39c17e9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:46:08.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.556+0000 7f39c17e9640 1 -- 172.21.15.42:0/1322426247 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39bc06e190 con 0x7f39bc139200 2024-09-16T03:46:08.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.556+0000 7f39c17e9640 1 librados: init done 2024-09-16T03:46:08.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.556+0000 7f39c17e9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:46:08.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.556+0000 7f39c17e9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:46:08.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.556+0000 7f39c17e9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:46:08.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.556+0000 7f39a37fe640 1 -- 172.21.15.42:0/1322426247 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f39b4013470 con 0x7f39bc139200 2024-09-16T03:46:08.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.556+0000 7f39c17e9640 1 -- 172.21.15.42:0/1322426247 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f39bc0a1710 con 0x7f39bc139200 2024-09-16T03:46:08.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.556+0000 7f39a37fe640 1 --2- 172.21.15.42:0/1322426247 >> v2:172.21.15.154:6812/254845342 conn(0x7f39a8037810 0x7f39a8039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:46:08.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.556+0000 7f39a37fe640 1 -- 172.21.15.42:0/1322426247 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(122..122 src has 1..122) ==== 5967+0+0 (secure 0 0 0) 0x7f39b4048cb0 con 0x7f39bc139200 2024-09-16T03:46:08.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.556+0000 7f39a37fe640 10 client.744411.objecter ms_dispatch 0x7f39bc000ba0 osd_map(122..122 src has 1..122) 2024-09-16T03:46:08.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.556+0000 7f39a37fe640 3 client.744411.objecter handle_osd_map got epochs [122,122] > 0 2024-09-16T03:46:08.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.556+0000 7f39a37fe640 3 client.744411.objecter handle_osd_map decoding full epoch 122 2024-09-16T03:46:08.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.556+0000 7f39a37fe640 20 client.744411.objecter dump_active .. 0 homeless 2024-09-16T03:46:08.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.556+0000 7f39baffd640 1 --2- 172.21.15.42:0/1322426247 >> v2:172.21.15.154:6812/254845342 conn(0x7f39a8037810 0x7f39a8039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:46:08.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.560+0000 7f39baffd640 10 monclient: get_auth_request con 0x7f39a8037810 auth_method 0 2024-09-16T03:46:08.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.560+0000 7f39baffd640 1 --2- 172.21.15.42:0/1322426247 >> v2:172.21.15.154:6812/254845342 conn(0x7f39a8037810 0x7f39a8039cd0 secure :-1 s=READY pgs=3402 cs=0 l=1 rev1=1 crypto rx=0x7f39ac000f70 tx=0x7f39ac007e90 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:46:08.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.560+0000 7f39a37fe640 10 client.744411.objecter ms_handle_connect 0x7f39a8037810 2024-09-16T03:46:08.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.564+0000 7f39a37fe640 1 -- 172.21.15.42:0/1322426247 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f39b40194d0 con 0x7f39bc139200 2024-09-16T03:46:08.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.564+0000 7f39a37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:46:08.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.564+0000 7f39a37fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:46:08.793 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.788+0000 7f39c17e9640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:46:08.793 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.788+0000 7f39c17e9640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:46:08.793 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.788+0000 7f39c17e9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:46:08.793 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.788+0000 7f39c17e9640 1 -- 172.21.15.42:0/1322426247 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f39bc13aea0 con 0x7f39bc139200 2024-09-16T03:46:08.794 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.788+0000 7f39a37fe640 1 -- 172.21.15.42:0/1322426247 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7f39b4013760 con 0x7f39bc139200 2024-09-16T03:46:08.794 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.788+0000 7f39a37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:46:08.794 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.788+0000 7f39a37fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:46:08.794 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":34,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:46:08.798 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.792+0000 7f39c17e9640 10 librados: watch_flush enter 2024-09-16T03:46:08.798 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.792+0000 7f39c17e9640 10 librados: watch_flush exit 2024-09-16T03:46:08.798 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.792+0000 7f39c17e9640 20 client.744411.objecter shutdown clearing up homeless session... 2024-09-16T03:46:08.798 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.792+0000 7f39c17e9640 10 client.744411.objecter successfully canceled tick 2024-09-16T03:46:08.799 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.792+0000 7f39c17e9640 1 -- 172.21.15.42:0/1322426247 >> v2:172.21.15.154:6812/254845342 conn(0x7f39a8037810 msgr2=0x7f39a8039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:46:08.799 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.792+0000 7f39c17e9640 1 --2- 172.21.15.42:0/1322426247 >> v2:172.21.15.154:6812/254845342 conn(0x7f39a8037810 0x7f39a8039cd0 secure :-1 s=READY pgs=3402 cs=0 l=1 rev1=1 crypto rx=0x7f39ac000f70 tx=0x7f39ac007e90 comp rx=0 tx=0).stop 2024-09-16T03:46:08.799 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.792+0000 7f39c17e9640 10 monclient: shutdown 2024-09-16T03:46:08.800 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.792+0000 7f39c17e9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:46:08.800 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.792+0000 7f39c17e9640 1 -- 172.21.15.42:0/1322426247 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f39bc139200 msgr2=0x7f39bc1396b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:46:08.801 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.792+0000 7f39c17e9640 1 --2- 172.21.15.42:0/1322426247 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f39bc139200 0x7f39bc1396b0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f39b400e950 tx=0x7f39b400ee10 comp rx=0 tx=0).stop 2024-09-16T03:46:08.801 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.792+0000 7f39c17e9640 1 -- 172.21.15.42:0/1322426247 shutdown_connections 2024-09-16T03:46:08.801 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.792+0000 7f39c17e9640 1 --2- 172.21.15.42:0/1322426247 >> v2:172.21.15.154:6812/254845342 conn(0x7f39a8037810 0x7f39a8039cd0 unknown :-1 s=CLOSED pgs=3402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:46:08.802 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.792+0000 7f39c17e9640 1 --2- 172.21.15.42:0/1322426247 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f39bc139200 0x7f39bc1396b0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:46:08.802 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.792+0000 7f39c17e9640 1 --2- 172.21.15.42:0/1322426247 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f39bc0a1e20 0x7f39bc1323f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:46:08.802 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.792+0000 7f39c17e9640 1 --2- 172.21.15.42:0/1322426247 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f39bc0a1330 0x7f39bc1ccbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:46:08.802 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.792+0000 7f39c17e9640 1 -- 172.21.15.42:0/1322426247 >> 172.21.15.42:0/1322426247 conn(0x7f39bc0a2420 msgr2=0x7f39bc135300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:46:08.802 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.792+0000 7f39c17e9640 1 -- 172.21.15.42:0/1322426247 shutdown_connections 2024-09-16T03:46:08.802 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.792+0000 7f39c17e9640 1 -- 172.21.15.42:0/1322426247 wait complete. 2024-09-16T03:46:08.802 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.792+0000 7f39c17e9640 1 librados: shutdown 2024-09-16T03:46:08.815 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T03:46:08.815 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T03:46:08.815 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T03:46:08.815 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T03:46:08.945 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.940+0000 7f405f1cb640 10 monclient: get_monmap_and_config 2024-09-16T03:46:08.945 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.940+0000 7f405f1cb640 10 monclient: build_initial_monmap 2024-09-16T03:46:08.946 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.940+0000 7f405f1cb640 10 monclient: monmap: 2024-09-16T03:46:08.946 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:46:08.946 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:46:08.946 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:46:08.944815+0000 2024-09-16T03:46:08.946 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:46:08.944815+0000 2024-09-16T03:46:08.946 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:46:08.946 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:46:08.946 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:46:08.946 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:46:08.946 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:46:08.946 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:46:08.946 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:46:08.947 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:46:08.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.940+0000 7f405f1cb640 1 Processor -- start 2024-09-16T03:46:08.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.940+0000 7f405f1cb640 1 -- start start 2024-09-16T03:46:08.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.940+0000 7f405f1cb640 10 monclient: init 2024-09-16T03:46:08.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.940+0000 7f405f1cb640 10 monclient: _reopen_session rank -1 2024-09-16T03:46:08.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.940+0000 7f405f1cb640 10 monclient: _add_conns ranks=[4,1,2,3,0] 2024-09-16T03:46:08.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.940+0000 7f405f1cb640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f40580a39b0 0x7f4058133b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:46:08.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.940+0000 7f405f1cb640 10 monclient(hunting): picked mon.noname-c con 0x7f40580a39b0 addr v2:172.21.15.154:3300/0 2024-09-16T03:46:08.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.940+0000 7f405f1cb640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f40580a3d90 addr v1:172.21.15.154:6789/0 2024-09-16T03:46:08.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.940+0000 7f405f1cb640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4058137710 0x7f405813f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:46:08.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.940+0000 7f405f1cb640 10 monclient(hunting): picked mon.noname-b con 0x7f4058137710 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:46:08.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.940+0000 7f405f1cb640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f405809c140 con 0x7f40580a3d90 2024-09-16T03:46:08.952 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.940+0000 7f405f1cb640 10 monclient(hunting): start opening mon connection 2024-09-16T03:46:08.952 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.940+0000 7f405f1cb640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f405809c420 con 0x7f40580a39b0 2024-09-16T03:46:08.952 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.940+0000 7f405f1cb640 10 monclient(hunting): start opening mon connection 2024-09-16T03:46:08.952 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.940+0000 7f405f1cb640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f405806dd60 con 0x7f4058137710 2024-09-16T03:46:08.952 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.940+0000 7f405f1cb640 10 monclient(hunting): _renew_subs 2024-09-16T03:46:08.952 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.940+0000 7f405f1cb640 10 monclient(hunting): authenticate will time out at 1215.914272s 2024-09-16T03:46:08.952 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.940+0000 7f405d768640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4058137710 0x7f405813f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:46:08.952 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.940+0000 7f405d768640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4058137710 0x7f405813f3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:51758/0 (socket says 172.21.15.42:51758) 2024-09-16T03:46:08.952 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.940+0000 7f405d768640 1 -- 172.21.15.42:0/3273337504 learned_addr learned my addr 172.21.15.42:0/3273337504 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:46:08.952 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.940+0000 7f405d768640 10 monclient(hunting): get_auth_request con 0x7f4058137710 auth_method 0 2024-09-16T03:46:08.952 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.940+0000 7f405d768640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:46:08.952 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.940+0000 7f405d768640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:46:08.952 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.940+0000 7f405d768640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:46:08.953 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.940+0000 7f405d768640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:46:08.953 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.940+0000 7f405d768640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:46:08.953 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.940+0000 7f405d768640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:46:08.954 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.940+0000 7f405cf67640 1 --2- 172.21.15.42:0/3273337504 >> v2:172.21.15.154:3300/0 conn(0x7f40580a39b0 0x7f4058133b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:46:08.954 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.940+0000 7f405cf67640 10 monclient(hunting): get_auth_request con 0x7f40580a39b0 auth_method 0 2024-09-16T03:46:08.954 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.940+0000 7f405cf67640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:46:08.954 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.940+0000 7f405cf67640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:46:08.955 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.940+0000 7f405cf67640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:46:08.956 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.940+0000 7f405d768640 10 monclient(hunting): handle_auth_done global_id 13562 payload 274 2024-09-16T03:46:08.956 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.940+0000 7f405d768640 1 -- 172.21.15.42:0/3273337504 >> v2:172.21.15.154:3300/0 conn(0x7f40580a39b0 msgr2=0x7f4058133b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:46:08.956 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.940+0000 7f405d768640 1 --2- 172.21.15.42:0/3273337504 >> v2:172.21.15.154:3300/0 conn(0x7f40580a39b0 0x7f4058133b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:46:08.956 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.940+0000 7f405d768640 1 -- 172.21.15.42:0/3273337504 >> v1:172.21.15.154:6789/0 conn(0x7f40580a3d90 legacy=0x7f4058137000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:46:08.956 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.940+0000 7f405d768640 10 monclient: _finish_hunting 0 2024-09-16T03:46:08.957 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.940+0000 7f405d768640 1 monclient: found mon.noname-b 2024-09-16T03:46:08.957 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.940+0000 7f405d768640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:46:08.957 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.940+0000 7f405d768640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T03:46:08.957 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.940+0000 7f405d768640 1 -- 172.21.15.42:0/3273337504 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40580a5e20 con 0x7f4058137710 2024-09-16T03:46:08.957 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.940+0000 7f405cf67640 1 --2- 172.21.15.42:0/3273337504 >> v2:172.21.15.154:3300/0 conn(0x7f40580a39b0 0x7f4058133b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T03:46:08.957 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.944+0000 7f405d768640 1 --2- 172.21.15.42:0/3273337504 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4058137710 0x7f405813f3a0 secure :-1 s=READY pgs=2423 cs=0 l=1 rev1=1 crypto rx=0x7f4050009870 tx=0x7f405001c420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4fd98d14e815df93 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:46:08.957 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.944+0000 7f40577fe640 1 -- 172.21.15.42:0/3273337504 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40500b0070 con 0x7f4058137710 2024-09-16T03:46:08.957 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.944+0000 7f40577fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:46:08.958 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.944+0000 7f40577fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:46:08.958 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.944+0000 7f40577fe640 10 monclient: dump: 2024-09-16T03:46:08.958 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:46:08.958 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:46:08.958 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:08.958 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:08.958 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:46:08.958 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:46:08.958 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:46:08.959 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:46:08.959 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:46:08.959 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:46:08.959 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.944+0000 7f40577fe640 10 monclient: _finish_auth 0 2024-09-16T03:46:08.959 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.944+0000 7f40577fe640 10 monclient: _check_auth_tickets 2024-09-16T03:46:08.959 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.944+0000 7f40577fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:46:08.959 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.944+0000 7f40577fe640 1 -- 172.21.15.42:0/3273337504 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40500ab600 con 0x7f4058137710 2024-09-16T03:46:08.959 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.944+0000 7f405f1cb640 5 monclient: authenticate success, global_id 13562 2024-09-16T03:46:08.959 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.944+0000 7f405f1cb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:46:08.959 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.944+0000 7f40577fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:46:08.959 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.944+0000 7f405f1cb640 10 monclient: get_monmap_and_config success 2024-09-16T03:46:08.959 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.944+0000 7f40577fe640 1 -- 172.21.15.42:0/3273337504 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40500ab9d0 con 0x7f4058137710 2024-09-16T03:46:08.959 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.944+0000 7f405f1cb640 4 set_mon_vals no callback set 2024-09-16T03:46:08.959 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.944+0000 7f40577fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:46:08.960 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.944+0000 7f40577fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:46:08.960 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.944+0000 7f40577fe640 10 monclient: dump: 2024-09-16T03:46:08.960 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:46:08.960 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:46:08.960 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:08.960 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:08.960 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:46:08.960 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:46:08.960 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:46:08.960 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:46:08.960 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:46:08.960 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:46:08.967 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.944+0000 7f405f1cb640 10 monclient: shutdown 2024-09-16T03:46:08.967 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.944+0000 7f405f1cb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:46:08.968 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.944+0000 7f405f1cb640 1 -- 172.21.15.42:0/3273337504 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4058137710 msgr2=0x7f405813f3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:46:08.968 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.944+0000 7f405f1cb640 1 --2- 172.21.15.42:0/3273337504 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4058137710 0x7f405813f3a0 secure :-1 s=READY pgs=2423 cs=0 l=1 rev1=1 crypto rx=0x7f4050009870 tx=0x7f405001c420 comp rx=0 tx=0).stop 2024-09-16T03:46:08.968 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.944+0000 7f405f1cb640 1 -- 172.21.15.42:0/3273337504 shutdown_connections 2024-09-16T03:46:08.968 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.944+0000 7f405f1cb640 1 --2- 172.21.15.42:0/3273337504 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4058137710 0x7f405813f3a0 unknown :-1 s=CLOSED pgs=2423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:46:08.968 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.944+0000 7f405f1cb640 1 --2- 172.21.15.42:0/3273337504 >> v2:172.21.15.154:3300/0 conn(0x7f40580a39b0 0x7f4058133b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:46:08.968 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.944+0000 7f405f1cb640 1 -- 172.21.15.42:0/3273337504 >> 172.21.15.42:0/3273337504 conn(0x7f40580a9fb0 msgr2=0x7f40580aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:46:08.969 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.944+0000 7f405f1cb640 1 -- 172.21.15.42:0/3273337504 shutdown_connections 2024-09-16T03:46:08.969 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.944+0000 7f405f1cb640 1 -- 172.21.15.42:0/3273337504 wait complete. 2024-09-16T03:46:08.969 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.944+0000 7f405f1cb640 10 monclient: build_initial_monmap 2024-09-16T03:46:08.969 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.944+0000 7f405f1cb640 10 monclient: monmap: 2024-09-16T03:46:08.969 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:46:08.969 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:46:08.969 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:46:08.969 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:46:08.969 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:46:08.969 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:46:08.969 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:46:08.969 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:46:08.969 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:46:08.970 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:46:08.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.944+0000 7f405f1cb640 1 librados: starting msgr at 2024-09-16T03:46:08.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.944+0000 7f405f1cb640 1 librados: starting objecter 2024-09-16T03:46:08.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.944+0000 7f405f1cb640 1 Processor -- start 2024-09-16T03:46:08.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.944+0000 7f405f1cb640 1 -- start start 2024-09-16T03:46:08.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.944+0000 7f405f1cb640 1 librados: setting wanted keys 2024-09-16T03:46:08.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.944+0000 7f405f1cb640 1 librados: calling monclient init 2024-09-16T03:46:08.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.944+0000 7f405f1cb640 10 monclient: init 2024-09-16T03:46:08.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.948+0000 7f405f1cb640 10 monclient: _reopen_session rank -1 2024-09-16T03:46:08.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.948+0000 7f405f1cb640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:46:08.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.948+0000 7f405f1cb640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f40580a39b0 0x7f40581cf910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:46:08.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.948+0000 7f405f1cb640 10 monclient(hunting): picked mon.noname-c con 0x7f40580a39b0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:46:08.972 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.948+0000 7f405f1cb640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f40580a3d90 0x7f405813e790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:46:08.972 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.948+0000 7f405f1cb640 10 monclient(hunting): picked mon.noname-b con 0x7f40580a3d90 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:46:08.972 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.948+0000 7f405f1cb640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4058137710 0x7f40580a1aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:46:08.972 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.948+0000 7f405f1cb640 10 monclient(hunting): picked mon.noname-a con 0x7f4058137710 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:46:08.972 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.948+0000 7f405f1cb640 10 monclient(hunting): start opening mon connection 2024-09-16T03:46:08.972 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.948+0000 7f405f1cb640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f4058142310 con 0x7f4058137710 2024-09-16T03:46:08.972 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.948+0000 7f405f1cb640 10 monclient(hunting): start opening mon connection 2024-09-16T03:46:08.972 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.948+0000 7f405f1cb640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f4058142490 con 0x7f40580a3d90 2024-09-16T03:46:08.972 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.948+0000 7f405f1cb640 10 monclient(hunting): start opening mon connection 2024-09-16T03:46:08.972 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.948+0000 7f405f1cb640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f4058142610 con 0x7f40580a39b0 2024-09-16T03:46:08.972 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.948+0000 7f405f1cb640 10 monclient(hunting): _renew_subs 2024-09-16T03:46:08.973 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.948+0000 7f405f1cb640 10 monclient(hunting): authenticate will time out at 1215.920628s 2024-09-16T03:46:08.973 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.948+0000 7f405cf67640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f40580a39b0 0x7f40581cf910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:46:08.973 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.948+0000 7f405cf67640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f40580a39b0 0x7f40581cf910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:51762/0 (socket says 172.21.15.42:51762) 2024-09-16T03:46:08.973 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.948+0000 7f405cf67640 1 -- 172.21.15.42:0/3880401267 learned_addr learned my addr 172.21.15.42:0/3880401267 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:46:08.973 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.948+0000 7f405cf67640 10 monclient(hunting): get_auth_request con 0x7f40580a39b0 auth_method 0 2024-09-16T03:46:08.973 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.948+0000 7f405cf67640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:46:08.973 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.948+0000 7f405cf67640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:46:08.973 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.948+0000 7f405cf67640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:46:08.973 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.948+0000 7f405d768640 1 --2- 172.21.15.42:0/3880401267 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4058137710 0x7f40580a1aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:46:08.973 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.948+0000 7f405d768640 10 monclient(hunting): get_auth_request con 0x7f4058137710 auth_method 0 2024-09-16T03:46:08.973 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.948+0000 7f405d768640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:46:08.973 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.948+0000 7f405d768640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:46:08.973 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.948+0000 7f405d768640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:46:08.974 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.948+0000 7f405cf67640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:46:08.974 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.948+0000 7f405cf67640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:46:08.974 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.948+0000 7f405cf67640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:46:08.974 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.948+0000 7f405d768640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:46:08.974 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.948+0000 7f405d768640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:46:08.974 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.948+0000 7f405d768640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:46:08.974 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.948+0000 7f4057fff640 1 --2- 172.21.15.42:0/3880401267 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f40580a3d90 0x7f405813e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:46:08.974 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.948+0000 7f4057fff640 10 monclient(hunting): get_auth_request con 0x7f40580a3d90 auth_method 0 2024-09-16T03:46:08.974 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.948+0000 7f4057fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:46:08.974 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.948+0000 7f4057fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:46:08.974 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.948+0000 7f4057fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:46:08.974 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.948+0000 7f405cf67640 10 monclient(hunting): handle_auth_done global_id 13565 payload 824 2024-09-16T03:46:08.974 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.948+0000 7f405cf67640 1 -- 172.21.15.42:0/3880401267 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f40580a3d90 msgr2=0x7f405813e790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:46:08.975 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.948+0000 7f405cf67640 1 --2- 172.21.15.42:0/3880401267 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f40580a3d90 0x7f405813e790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:46:08.975 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.948+0000 7f405cf67640 1 -- 172.21.15.42:0/3880401267 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4058137710 msgr2=0x7f40580a1aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:46:08.975 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.948+0000 7f405cf67640 1 --2- 172.21.15.42:0/3880401267 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4058137710 0x7f40580a1aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:46:08.975 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.948+0000 7f405cf67640 10 monclient: _finish_hunting 0 2024-09-16T03:46:08.975 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.948+0000 7f405cf67640 1 monclient: found mon.noname-c 2024-09-16T03:46:08.975 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.948+0000 7f405cf67640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:46:08.975 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.948+0000 7f405cf67640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:46:08.975 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.948+0000 7f405cf67640 1 -- 172.21.15.42:0/3880401267 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f405813a100 con 0x7f40580a39b0 2024-09-16T03:46:08.975 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.948+0000 7f405cf67640 1 --2- 172.21.15.42:0/3880401267 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f40580a39b0 0x7f40581cf910 secure :-1 s=READY pgs=2424 cs=0 l=1 rev1=1 crypto rx=0x7f404800d100 tx=0x7f404800d5c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:46:08.975 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.948+0000 7f40557fa640 10 client.?.objecter ms_handle_connect 0x7f40580a39b0 2024-09-16T03:46:08.975 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.948+0000 7f40557fa640 10 client.?.objecter resend_mon_ops 2024-09-16T03:46:08.975 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.948+0000 7f40557fa640 1 -- 172.21.15.42:0/3880401267 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f404801a070 con 0x7f40580a39b0 2024-09-16T03:46:08.975 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.948+0000 7f40557fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:46:08.976 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.948+0000 7f40557fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:46:08.976 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.948+0000 7f40557fa640 10 monclient: dump: 2024-09-16T03:46:08.976 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:46:08.976 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:46:08.976 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:08.976 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:08.976 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:46:08.976 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:46:08.976 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:46:08.976 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:46:08.976 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:46:08.976 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:46:08.976 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.948+0000 7f40557fa640 10 monclient: _finish_auth 0 2024-09-16T03:46:08.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.948+0000 7f40557fa640 10 monclient: _check_auth_tickets 2024-09-16T03:46:08.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.948+0000 7f40557fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:46:08.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.948+0000 7f40557fa640 1 -- 172.21.15.42:0/3880401267 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4048003ba0 con 0x7f40580a39b0 2024-09-16T03:46:08.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.948+0000 7f40557fa640 10 monclient: handle_config config(0 keys) 2024-09-16T03:46:08.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.948+0000 7f40557fa640 1 -- 172.21.15.42:0/3880401267 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4048004030 con 0x7f40580a39b0 2024-09-16T03:46:08.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.948+0000 7f40557fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:46:08.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.948+0000 7f40557fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:46:08.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.948+0000 7f40557fa640 10 monclient: dump: 2024-09-16T03:46:08.977 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:46:08.977 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:46:08.977 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:08.977 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:08.977 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:46:08.978 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:46:08.978 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:46:08.978 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:46:08.978 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:46:08.978 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:46:08.978 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.948+0000 7f405f1cb640 5 monclient: authenticate success, global_id 13565 2024-09-16T03:46:08.978 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.948+0000 7f4056ffd640 4 set_mon_vals no callback set 2024-09-16T03:46:08.978 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.948+0000 7f405f1cb640 10 monclient: _renew_subs 2024-09-16T03:46:08.978 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.948+0000 7f405f1cb640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:46:08.978 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.948+0000 7f405f1cb640 1 -- 172.21.15.42:0/3880401267 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f405813f3a0 con 0x7f40580a39b0 2024-09-16T03:46:08.978 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.948+0000 7f405f1cb640 10 client.13565.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:46:08.978 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.948+0000 7f405f1cb640 10 monclient: _renew_subs 2024-09-16T03:46:08.978 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.948+0000 7f405f1cb640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:46:08.979 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.948+0000 7f405f1cb640 1 -- 172.21.15.42:0/3880401267 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f40580a36f0 con 0x7f40580a39b0 2024-09-16T03:46:08.979 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.948+0000 7f405f1cb640 1 librados: init done 2024-09-16T03:46:08.979 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.948+0000 7f40557fa640 1 -- 172.21.15.42:0/3880401267 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f404801d070 con 0x7f40580a39b0 2024-09-16T03:46:08.979 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.948+0000 7f405f1cb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:46:08.979 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.948+0000 7f405f1cb640 1 --2- 172.21.15.42:0/3880401267 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f405813ecd0 0x7f40580a1fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:46:08.979 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.948+0000 7f405f1cb640 10 monclient(hunting): start opening mon connection 2024-09-16T03:46:08.979 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.948+0000 7f405f1cb640 1 -- 172.21.15.42:0/3880401267 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f40581342d0 con 0x7f405813ecd0 2024-09-16T03:46:08.979 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.948+0000 7f4057fff640 1 --2- 172.21.15.42:0/3880401267 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f405813ecd0 0x7f40580a1fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:46:08.979 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.948+0000 7f4057fff640 10 monclient: get_auth_request con 0x7f405813ecd0 auth_method 0 2024-09-16T03:46:08.979 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.948+0000 7f4057fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:46:08.979 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.948+0000 7f4057fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:46:08.979 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.948+0000 7f4057fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:46:08.979 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.948+0000 7f40557fa640 1 --2- 172.21.15.42:0/3880401267 >> v2:172.21.15.154:6812/254845342 conn(0x7f4038037810 0x7f4038039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:46:08.979 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.948+0000 7f40557fa640 1 -- 172.21.15.42:0/3880401267 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(123..123 src has 1..123) ==== 6278+0+0 (secure 0 0 0) 0x7f4048049480 con 0x7f40580a39b0 2024-09-16T03:46:08.980 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.948+0000 7f4057fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:46:08.980 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.948+0000 7f4057fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:46:08.980 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.948+0000 7f40557fa640 10 client.13565.objecter ms_dispatch 0x7f4058000ba0 osd_map(123..123 src has 1..123) 2024-09-16T03:46:08.980 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.948+0000 7f40557fa640 3 client.13565.objecter handle_osd_map got epochs [123,123] > 0 2024-09-16T03:46:08.980 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.948+0000 7f40557fa640 3 client.13565.objecter handle_osd_map decoding full epoch 123 2024-09-16T03:46:08.980 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.948+0000 7f4057fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:46:08.980 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.948+0000 7f40557fa640 20 client.13565.objecter dump_active .. 0 homeless 2024-09-16T03:46:08.980 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.948+0000 7f4057fff640 10 monclient(hunting): handle_auth_done global_id 744426 payload 824 2024-09-16T03:46:08.980 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.948+0000 7f4057fff640 1 -- 172.21.15.42:0/3880401267 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4058133b80 con 0x7f405813ecd0 2024-09-16T03:46:08.980 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.948+0000 7f405d768640 1 --2- 172.21.15.42:0/3880401267 >> v2:172.21.15.154:6812/254845342 conn(0x7f4038037810 0x7f4038039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:46:08.980 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.948+0000 7f40557fa640 10 client.13565.objecter ms_handle_connect 0x7f405813ecd0 2024-09-16T03:46:08.980 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.948+0000 7f4057fff640 1 --2- 172.21.15.42:0/3880401267 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f405813ecd0 0x7f40580a1fe0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f40580a32f0 tx=0x7f404000e890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:46:08.980 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.948+0000 7f40557fa640 10 client.13565.objecter resend_mon_ops 2024-09-16T03:46:08.981 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.948+0000 7f405d768640 10 monclient: get_auth_request con 0x7f4038037810 auth_method 0 2024-09-16T03:46:08.981 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.948+0000 7f40557fa640 1 -- 172.21.15.42:0/3880401267 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4040019070 con 0x7f405813ecd0 2024-09-16T03:46:08.981 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.948+0000 7f40557fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:46:08.981 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.948+0000 7f40557fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:46:08.981 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.948+0000 7f40557fa640 10 monclient: dump: 2024-09-16T03:46:08.981 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:46:08.981 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:46:08.981 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:08.981 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:08.981 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:46:08.981 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:46:08.981 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:46:08.981 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:46:08.981 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:46:08.982 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:46:08.982 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.948+0000 7f40557fa640 1 -- 172.21.15.42:0/3880401267 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f40400052e0 con 0x7f405813ecd0 2024-09-16T03:46:08.982 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.948+0000 7f40557fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:46:08.982 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.948+0000 7f40557fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:46:08.982 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.948+0000 7f40557fa640 1 -- 172.21.15.42:0/3880401267 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f405813ecd0 msgr2=0x7f40580a1fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:46:08.982 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.948+0000 7f40557fa640 1 --2- 172.21.15.42:0/3880401267 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f405813ecd0 0x7f40580a1fe0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f40580a32f0 tx=0x7f404000e890 comp rx=0 tx=0).stop 2024-09-16T03:46:08.982 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.948+0000 7f40557fa640 1 -- 172.21.15.42:0/3880401267 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f405813ecd0 msgr2=0x7f40580a1fe0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:46:08.982 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.948+0000 7f40557fa640 1 --2- 172.21.15.42:0/3880401267 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f405813ecd0 0x7f40580a1fe0 secure :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f40580a32f0 tx=0x7f404000e890 comp rx=0 tx=0).stop 2024-09-16T03:46:08.982 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.952+0000 7f40557fa640 10 client.13565.objecter ms_handle_connect 0x7f4038037810 2024-09-16T03:46:08.982 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.952+0000 7f405d768640 1 --2- 172.21.15.42:0/3880401267 >> v2:172.21.15.154:6812/254845342 conn(0x7f4038037810 0x7f4038039cd0 secure :-1 s=READY pgs=3404 cs=0 l=1 rev1=1 crypto rx=0x7f40500ad000 tx=0x7f4050008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:46:08.982 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.956+0000 7f405f1cb640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:46:08.982 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.956+0000 7f405f1cb640 1 --2- 172.21.15.42:0/3880401267 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f40580a2520 0x7f405809d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:46:08.982 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.956+0000 7f405f1cb640 10 monclient(hunting): start opening mon connection 2024-09-16T03:46:08.983 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.956+0000 7f405f1cb640 1 -- 172.21.15.42:0/3880401267 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f40580a2930 con 0x7f40580a2520 2024-09-16T03:46:08.983 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.956+0000 7f4057fff640 1 --2- 172.21.15.42:0/3880401267 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f40580a2520 0x7f405809d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:46:08.983 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.956+0000 7f4057fff640 10 monclient: get_auth_request con 0x7f40580a2520 auth_method 0 2024-09-16T03:46:08.983 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.956+0000 7f4057fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:46:08.983 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.956+0000 7f4057fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:46:08.983 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.956+0000 7f4057fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:46:08.983 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.956+0000 7f4057fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:46:08.983 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.956+0000 7f4057fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:46:08.983 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.956+0000 7f4057fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:46:08.983 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.956+0000 7f4057fff640 10 monclient(hunting): handle_auth_done global_id 744429 payload 824 2024-09-16T03:46:08.983 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.956+0000 7f4057fff640 1 -- 172.21.15.42:0/3880401267 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f40580a5750 con 0x7f40580a2520 2024-09-16T03:46:08.983 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.956+0000 7f40557fa640 10 client.13565.objecter ms_handle_connect 0x7f40580a2520 2024-09-16T03:46:08.983 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.956+0000 7f40557fa640 10 client.13565.objecter resend_mon_ops 2024-09-16T03:46:08.984 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.956+0000 7f4057fff640 1 --2- 172.21.15.42:0/3880401267 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f40580a2520 0x7f405809d8e0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f40580a32f0 tx=0x7f4040000ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:46:08.984 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.956+0000 7f40557fa640 1 -- 172.21.15.42:0/3880401267 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4040019040 con 0x7f40580a2520 2024-09-16T03:46:08.984 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.956+0000 7f40557fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:46:08.984 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.956+0000 7f40557fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:46:08.984 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.956+0000 7f40557fa640 10 monclient: dump: 2024-09-16T03:46:08.984 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:46:08.984 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:46:08.984 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:08.984 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:08.984 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:46:08.984 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:46:08.984 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:46:08.984 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:46:08.984 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:46:08.985 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:46:08.985 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.956+0000 7f40557fa640 1 -- 172.21.15.42:0/3880401267 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4713 (secure 0 0 0) 0x7f4040003600 con 0x7f40580a2520 2024-09-16T03:46:08.985 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.956+0000 7f40557fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:46:08.985 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.956+0000 7f40557fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:46:08.985 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.956+0000 7f40557fa640 1 -- 172.21.15.42:0/3880401267 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f40580a2520 msgr2=0x7f405809d8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:46:08.985 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.956+0000 7f40557fa640 1 --2- 172.21.15.42:0/3880401267 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f40580a2520 0x7f405809d8e0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f40580a32f0 tx=0x7f4040000ee0 comp rx=0 tx=0).stop 2024-09-16T03:46:08.985 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.956+0000 7f40557fa640 1 -- 172.21.15.42:0/3880401267 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f40580a2520 msgr2=0x7f405809d8e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:46:08.985 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.956+0000 7f40557fa640 1 --2- 172.21.15.42:0/3880401267 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f40580a2520 0x7f405809d8e0 secure :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f40580a32f0 tx=0x7f4040000ee0 comp rx=0 tx=0).stop 2024-09-16T03:46:08.985 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:46:08.985 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:46:08.986 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:46:08.986 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "leader", 2024-09-16T03:46:08.986 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 34, 2024-09-16T03:46:08.986 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 56327, 2024-09-16T03:46:08.986 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:46:08.986 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:46:08.986 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T03:46:08.986 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T03:46:08.986 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:46:08.986 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 5, 2024-09-16T03:46:08.986 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:46:08.986 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:46:08.986 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:46:08.986 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:46:08.987 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:46:08.987 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:46:08.987 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:46:08.987 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:46:08.987 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:46:08.987 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:46:08.987 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:46:08.987 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:46:08.987 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:46:08.987 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:46:08.987 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:46:08.987 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:46:08.987 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:46:08.988 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:46:08.988 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:46:08.988 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:46:08.988 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:46:08.988 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:46:08.988 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:46:08.988 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:46:08.988 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:46:08.988 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:46:08.988 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:46:08.988 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:46:08.988 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:46:08.988 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:46:08.988 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:46:08.989 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:46:08.989 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:46:08.989 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:46:08.989 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:46:08.989 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:46:08.989 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:46:08.989 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:46:08.989 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:46:08.989 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:46:08.989 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:46:08.989 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:46:08.989 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:46:08.989 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:46:08.990 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:46:08.990 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:46:08.990 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:46:08.990 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:46:08.990 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:46:08.990 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:46:08.990 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:46:08.990 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:46:08.990 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:46:08.990 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:46:08.990 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:46:08.990 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:46:08.990 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:46:08.990 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:46:08.991 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:46:08.991 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:46:08.991 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:46:08.991 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:46:08.991 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:46:08.991 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:46:08.991 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:46:08.991 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:46:08.991 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:46:08.991 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:46:08.991 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:46:08.992 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:46:08.992 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:46:08.992 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:46:08.992 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:46:08.992 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:46:08.992 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:46:08.992 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:46:08.992 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:46:08.992 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:46:08.992 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:46:08.992 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:46:08.992 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:46:08.993 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:46:08.993 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:46:08.993 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:46:08.993 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:46:08.993 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:46:08.993 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:46:08.993 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:46:08.993 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:46:08.993 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:46:08.993 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:46:08.993 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:46:08.993 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:46:08.993 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:46:08.993 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:46:08.994 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:46:08.994 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:46:08.994 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:46:08.994 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:46:08.994 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:46:08.994 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:46:08.994 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:46:08.994 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:46:08.994 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:46:08.994 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:46:08.994 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:46:08.994 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:46:08.994 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:46:08.994 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:46:08.995 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:46:08.995 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:46:08.995 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:46:08.995 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:46:08.995 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:46:08.995 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:46:08.995 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:46:08.995 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:46:08.995 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:46:08.995 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:46:08.995 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:46:08.995 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:46:08.995 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:46:08.995 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:46:08.996 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:46:08.996 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:46:08.996 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:46:08.996 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:46:08.996 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:46:08.996 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:46:08.996 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:46:08.996 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:46:08.996 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:46:08.996 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:46:08.996 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:46:08.996 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:46:08.996 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:46:08.997 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:46:08.997 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:46:08.997 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:46:08.997 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:46:08.997 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:46:08.997 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T03:46:08.997 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:46:08.997 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:46:08.997 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:46:08.997 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:46:08.997 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:46:08.997 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:46:08.997 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:46:08.997 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:46:08.998 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:46:08.998 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:46:08.998 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 3 2024-09-16T03:46:08.998 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:46:08.998 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:46:08.998 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:46:08.998 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:46:08.998 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:46:09.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.956+0000 7f405f1cb640 10 librados: watch_flush enter 2024-09-16T03:46:09.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.956+0000 7f405f1cb640 10 librados: watch_flush exit 2024-09-16T03:46:09.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.956+0000 7f405f1cb640 20 client.13565.objecter shutdown clearing up homeless session... 2024-09-16T03:46:09.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.956+0000 7f405f1cb640 10 client.13565.objecter successfully canceled tick 2024-09-16T03:46:09.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.956+0000 7f405f1cb640 1 -- 172.21.15.42:0/3880401267 >> v2:172.21.15.154:6812/254845342 conn(0x7f4038037810 msgr2=0x7f4038039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:46:09.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.956+0000 7f405f1cb640 1 --2- 172.21.15.42:0/3880401267 >> v2:172.21.15.154:6812/254845342 conn(0x7f4038037810 0x7f4038039cd0 secure :-1 s=READY pgs=3404 cs=0 l=1 rev1=1 crypto rx=0x7f40500ad000 tx=0x7f4050008040 comp rx=0 tx=0).stop 2024-09-16T03:46:09.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.956+0000 7f405f1cb640 10 monclient: shutdown 2024-09-16T03:46:09.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.956+0000 7f405f1cb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:46:09.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.956+0000 7f405f1cb640 1 -- 172.21.15.42:0/3880401267 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f40580a39b0 msgr2=0x7f40581cf910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:46:09.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.956+0000 7f405f1cb640 1 --2- 172.21.15.42:0/3880401267 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f40580a39b0 0x7f40581cf910 secure :-1 s=READY pgs=2424 cs=0 l=1 rev1=1 crypto rx=0x7f404800d100 tx=0x7f404800d5c0 comp rx=0 tx=0).stop 2024-09-16T03:46:09.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.956+0000 7f405d768640 1 -- 172.21.15.42:0/3880401267 reap_dead start 2024-09-16T03:46:09.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.956+0000 7f405d768640 1 -- 172.21.15.42:0/3880401267 reap_dead start 2024-09-16T03:46:09.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.956+0000 7f405f1cb640 1 -- 172.21.15.42:0/3880401267 shutdown_connections 2024-09-16T03:46:09.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.956+0000 7f405f1cb640 1 -- 172.21.15.42:0/3880401267 >> 172.21.15.42:0/3880401267 conn(0x7f40580a9fb0 msgr2=0x7f4058131980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:46:09.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.956+0000 7f405f1cb640 1 -- 172.21.15.42:0/3880401267 shutdown_connections 2024-09-16T03:46:09.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.956+0000 7f405f1cb640 1 -- 172.21.15.42:0/3880401267 wait complete. 2024-09-16T03:46:09.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:08.956+0000 7f405f1cb640 1 librados: shutdown 2024-09-16T03:46:09.002 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:46:09.002 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T03:46:09.002 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T03:46:09.117 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.112+0000 7f0bfa751640 10 monclient: get_monmap_and_config 2024-09-16T03:46:09.117 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.112+0000 7f0bfa751640 10 monclient: build_initial_monmap 2024-09-16T03:46:09.117 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.112+0000 7f0bfa751640 10 monclient: monmap: 2024-09-16T03:46:09.117 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:46:09.117 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:46:09.117 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:46:09.116573+0000 2024-09-16T03:46:09.118 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:46:09.116573+0000 2024-09-16T03:46:09.118 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:46:09.118 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:46:09.118 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:46:09.118 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:46:09.118 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:46:09.118 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:46:09.118 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:46:09.118 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:46:09.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.112+0000 7f0bfa751640 1 Processor -- start 2024-09-16T03:46:09.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.112+0000 7f0bfa751640 1 -- start start 2024-09-16T03:46:09.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.112+0000 7f0bfa751640 10 monclient: init 2024-09-16T03:46:09.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.112+0000 7f0bfa751640 10 monclient: _reopen_session rank -1 2024-09-16T03:46:09.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.112+0000 7f0bfa751640 10 monclient: _add_conns ranks=[4,3,1,0,2] 2024-09-16T03:46:09.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.112+0000 7f0bfa751640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f0bf40a1330 0x7f0bf40a1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:46:09.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.112+0000 7f0bfa751640 10 monclient(hunting): picked mon.noname-c con 0x7f0bf40a1330 addr v2:172.21.15.154:3300/0 2024-09-16T03:46:09.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.112+0000 7f0bfa751640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f0bf409c920 0x7f0bf40a1c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:46:09.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.112+0000 7f0bfa751640 10 monclient(hunting): picked mon.noname-a con 0x7f0bf409c920 addr v2:172.21.15.42:3300/0 2024-09-16T03:46:09.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.112+0000 7f0bfa751640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0bf4072d40 addr v1:172.21.15.154:6789/0 2024-09-16T03:46:09.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.112+0000 7f0bfa751640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0bf4072350 con 0x7f0bf4072d40 2024-09-16T03:46:09.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.112+0000 7f0bfa751640 10 monclient(hunting): start opening mon connection 2024-09-16T03:46:09.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.112+0000 7f0bfa751640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f0bf4072b50 con 0x7f0bf409c920 2024-09-16T03:46:09.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.112+0000 7f0bfa751640 10 monclient(hunting): start opening mon connection 2024-09-16T03:46:09.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.112+0000 7f0bfa751640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f0bf40a2190 con 0x7f0bf40a1330 2024-09-16T03:46:09.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.112+0000 7f0bfa751640 10 monclient(hunting): _renew_subs 2024-09-16T03:46:09.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.112+0000 7f0bfa751640 10 monclient(hunting): authenticate will time out at 1216.086226s 2024-09-16T03:46:09.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.112+0000 7f0bf37fe640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f0bf409c920 0x7f0bf40a1c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:46:09.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.112+0000 7f0bf37fe640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f0bf409c920 0x7f0bf40a1c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:44362/0 (socket says 172.21.15.42:44362) 2024-09-16T03:46:09.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.112+0000 7f0bf37fe640 1 -- 172.21.15.42:0/1271242560 learned_addr learned my addr 172.21.15.42:0/1271242560 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:46:09.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.112+0000 7f0bf37fe640 10 monclient(hunting): get_auth_request con 0x7f0bf409c920 auth_method 0 2024-09-16T03:46:09.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.112+0000 7f0bf37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:46:09.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.112+0000 7f0bf37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:46:09.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.112+0000 7f0bf37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:46:09.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.112+0000 7f0bf37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:46:09.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.112+0000 7f0bf37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:46:09.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.112+0000 7f0bf37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:46:09.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.112+0000 7f0bf3fff640 1 --2- 172.21.15.42:0/1271242560 >> v2:172.21.15.154:3300/0 conn(0x7f0bf40a1330 0x7f0bf40a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:46:09.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.112+0000 7f0bf3fff640 10 monclient(hunting): get_auth_request con 0x7f0bf40a1330 auth_method 0 2024-09-16T03:46:09.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.112+0000 7f0bf3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:46:09.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.112+0000 7f0bf3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:46:09.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.112+0000 7f0bf3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:46:09.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.112+0000 7f0bf37fe640 10 monclient(hunting): handle_auth_done global_id 744432 payload 274 2024-09-16T03:46:09.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.112+0000 7f0bf37fe640 1 -- 172.21.15.42:0/1271242560 >> v2:172.21.15.154:3300/0 conn(0x7f0bf40a1330 msgr2=0x7f0bf40a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:46:09.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.112+0000 7f0bf37fe640 1 --2- 172.21.15.42:0/1271242560 >> v2:172.21.15.154:3300/0 conn(0x7f0bf40a1330 0x7f0bf40a1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:46:09.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.112+0000 7f0bf37fe640 1 -- 172.21.15.42:0/1271242560 >> v1:172.21.15.154:6789/0 conn(0x7f0bf4072d40 legacy=0x7f0bf4137120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:46:09.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.112+0000 7f0bf37fe640 10 monclient: _finish_hunting 0 2024-09-16T03:46:09.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.112+0000 7f0bf37fe640 1 monclient: found mon.noname-a 2024-09-16T03:46:09.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.112+0000 7f0bf37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:46:09.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.112+0000 7f0bf37fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:46:09.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.112+0000 7f0bf37fe640 1 -- 172.21.15.42:0/1271242560 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0bf409bc60 con 0x7f0bf409c920 2024-09-16T03:46:09.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.116+0000 7f0bf37fe640 1 --2- 172.21.15.42:0/1271242560 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0bf409c920 0x7f0bf40a1c50 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f0bf40a2340 tx=0x7f0be801c430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9764a288426e439 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:46:09.131 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.116+0000 7f0bf2ffd640 1 -- 172.21.15.42:0/1271242560 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0be80b0070 con 0x7f0bf409c920 2024-09-16T03:46:09.131 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.116+0000 7f0bf2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:46:09.131 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.116+0000 7f0bf2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:46:09.131 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.116+0000 7f0bf2ffd640 10 monclient: dump: 2024-09-16T03:46:09.131 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:46:09.131 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:46:09.131 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:09.131 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:09.131 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:46:09.131 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:46:09.131 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:46:09.131 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:46:09.132 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:46:09.132 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:46:09.132 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.116+0000 7f0bf2ffd640 10 monclient: _finish_auth 0 2024-09-16T03:46:09.132 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.116+0000 7f0bf2ffd640 10 monclient: _check_auth_tickets 2024-09-16T03:46:09.132 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.116+0000 7f0bf2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:46:09.132 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.116+0000 7f0bf2ffd640 1 -- 172.21.15.42:0/1271242560 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0be80ab600 con 0x7f0bf409c920 2024-09-16T03:46:09.132 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.116+0000 7f0bfa751640 5 monclient: authenticate success, global_id 744432 2024-09-16T03:46:09.132 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.116+0000 7f0bfa751640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:46:09.132 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.116+0000 7f0bf2ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:46:09.132 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.116+0000 7f0bfa751640 10 monclient: get_monmap_and_config success 2024-09-16T03:46:09.132 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.116+0000 7f0bfa751640 4 set_mon_vals no callback set 2024-09-16T03:46:09.132 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.116+0000 7f0bf2ffd640 1 -- 172.21.15.42:0/1271242560 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0be80ab9d0 con 0x7f0bf409c920 2024-09-16T03:46:09.132 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.116+0000 7f0bf2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:46:09.132 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.116+0000 7f0bf2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:46:09.133 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.116+0000 7f0bf2ffd640 10 monclient: dump: 2024-09-16T03:46:09.133 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:46:09.133 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:46:09.133 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:09.133 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:09.133 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:46:09.133 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:46:09.133 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:46:09.133 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:46:09.133 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:46:09.133 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:46:09.133 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.116+0000 7f0bfa751640 10 monclient: shutdown 2024-09-16T03:46:09.133 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.116+0000 7f0bfa751640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:46:09.133 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.116+0000 7f0bfa751640 1 -- 172.21.15.42:0/1271242560 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0bf409c920 msgr2=0x7f0bf40a1c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:46:09.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.116+0000 7f0bfa751640 1 --2- 172.21.15.42:0/1271242560 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0bf409c920 0x7f0bf40a1c50 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f0bf40a2340 tx=0x7f0be801c430 comp rx=0 tx=0).stop 2024-09-16T03:46:09.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.116+0000 7f0bfa751640 1 -- 172.21.15.42:0/1271242560 shutdown_connections 2024-09-16T03:46:09.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.116+0000 7f0bfa751640 1 --2- 172.21.15.42:0/1271242560 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0bf409c920 0x7f0bf40a1c50 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:46:09.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.116+0000 7f0bfa751640 1 --2- 172.21.15.42:0/1271242560 >> v2:172.21.15.154:3300/0 conn(0x7f0bf40a1330 0x7f0bf40a1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:46:09.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.116+0000 7f0bfa751640 1 -- 172.21.15.42:0/1271242560 >> 172.21.15.42:0/1271242560 conn(0x7f0bf40a2420 msgr2=0x7f0bf40a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:46:09.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.116+0000 7f0bfa751640 1 -- 172.21.15.42:0/1271242560 shutdown_connections 2024-09-16T03:46:09.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.116+0000 7f0bfa751640 1 -- 172.21.15.42:0/1271242560 wait complete. 2024-09-16T03:46:09.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.116+0000 7f0bfa751640 10 monclient: build_initial_monmap 2024-09-16T03:46:09.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.116+0000 7f0bfa751640 10 monclient: monmap: 2024-09-16T03:46:09.134 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:46:09.134 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:46:09.134 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:46:09.134 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:46:09.135 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:46:09.135 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:46:09.135 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:46:09.135 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:46:09.135 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:46:09.135 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:46:09.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.116+0000 7f0bfa751640 1 librados: starting msgr at 2024-09-16T03:46:09.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.116+0000 7f0bfa751640 1 librados: starting objecter 2024-09-16T03:46:09.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.116+0000 7f0bfa751640 1 Processor -- start 2024-09-16T03:46:09.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.116+0000 7f0bfa751640 1 -- start start 2024-09-16T03:46:09.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.116+0000 7f0bfa751640 1 librados: setting wanted keys 2024-09-16T03:46:09.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.116+0000 7f0bfa751640 1 librados: calling monclient init 2024-09-16T03:46:09.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.116+0000 7f0bfa751640 10 monclient: init 2024-09-16T03:46:09.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.116+0000 7f0bfa751640 10 monclient: _reopen_session rank -1 2024-09-16T03:46:09.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.116+0000 7f0bfa751640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:46:09.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.116+0000 7f0bfa751640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0bf4072d40 0x7f0bf41cf0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:46:09.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.116+0000 7f0bfa751640 10 monclient(hunting): picked mon.noname-c con 0x7f0bf4072d40 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:46:09.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.116+0000 7f0bfa751640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0bf409c920 0x7f0bf41334e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:46:09.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.116+0000 7f0bfa751640 10 monclient(hunting): picked mon.noname-a con 0x7f0bf409c920 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:46:09.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.116+0000 7f0bfa751640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0bf40a1330 0x7f0bf413cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:46:09.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.116+0000 7f0bfa751640 10 monclient(hunting): picked mon.noname-b con 0x7f0bf40a1330 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:46:09.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.116+0000 7f0bfa751640 10 monclient(hunting): start opening mon connection 2024-09-16T03:46:09.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.116+0000 7f0bfa751640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f0bf4141a60 con 0x7f0bf409c920 2024-09-16T03:46:09.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.116+0000 7f0bfa751640 10 monclient(hunting): start opening mon connection 2024-09-16T03:46:09.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.116+0000 7f0bfa751640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f0bf41418e0 con 0x7f0bf40a1330 2024-09-16T03:46:09.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.116+0000 7f0bfa751640 10 monclient(hunting): start opening mon connection 2024-09-16T03:46:09.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.116+0000 7f0bfa751640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f0bf4141be0 con 0x7f0bf4072d40 2024-09-16T03:46:09.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.116+0000 7f0bfa751640 10 monclient(hunting): _renew_subs 2024-09-16T03:46:09.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.116+0000 7f0bfa751640 10 monclient(hunting): authenticate will time out at 1216.091259s 2024-09-16T03:46:09.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.116+0000 7f0bf37fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0bf409c920 0x7f0bf41334e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:46:09.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.116+0000 7f0bf3fff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0bf4072d40 0x7f0bf41cf0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:46:09.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.116+0000 7f0bf37fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0bf409c920 0x7f0bf41334e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:44374/0 (socket says 172.21.15.42:44374) 2024-09-16T03:46:09.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.116+0000 7f0bf37fe640 1 -- 172.21.15.42:0/1247227672 learned_addr learned my addr 172.21.15.42:0/1247227672 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:46:09.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.116+0000 7f0bf3fff640 10 monclient(hunting): get_auth_request con 0x7f0bf4072d40 auth_method 0 2024-09-16T03:46:09.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.116+0000 7f0bf3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:46:09.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.116+0000 7f0bf3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:46:09.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.116+0000 7f0bf3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:46:09.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.116+0000 7f0bf37fe640 10 monclient(hunting): get_auth_request con 0x7f0bf409c920 auth_method 0 2024-09-16T03:46:09.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.116+0000 7f0bf37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:46:09.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.116+0000 7f0bf37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:46:09.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.116+0000 7f0bf37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:46:09.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.116+0000 7f0bf3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:46:09.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.116+0000 7f0bf3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:46:09.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.116+0000 7f0bf3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:46:09.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.116+0000 7f0bf37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:46:09.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.116+0000 7f0bf37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:46:09.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.116+0000 7f0bf37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:46:09.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.116+0000 7f0bf8cee640 1 --2- 172.21.15.42:0/1247227672 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0bf40a1330 0x7f0bf413cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:46:09.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.116+0000 7f0bf8cee640 10 monclient(hunting): get_auth_request con 0x7f0bf40a1330 auth_method 0 2024-09-16T03:46:09.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.116+0000 7f0bf8cee640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:46:09.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.116+0000 7f0bf8cee640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:46:09.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.116+0000 7f0bf8cee640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:46:09.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.116+0000 7f0bf3fff640 10 monclient(hunting): handle_auth_done global_id 13568 payload 824 2024-09-16T03:46:09.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.116+0000 7f0bf3fff640 1 -- 172.21.15.42:0/1247227672 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0bf40a1330 msgr2=0x7f0bf413cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:46:09.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.116+0000 7f0bf3fff640 1 --2- 172.21.15.42:0/1247227672 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0bf40a1330 0x7f0bf413cd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:46:09.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.116+0000 7f0bf3fff640 1 -- 172.21.15.42:0/1247227672 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0bf409c920 msgr2=0x7f0bf41334e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:46:09.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.116+0000 7f0bf3fff640 1 --2- 172.21.15.42:0/1247227672 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0bf409c920 0x7f0bf41334e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:46:09.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.116+0000 7f0bf3fff640 10 monclient: _finish_hunting 0 2024-09-16T03:46:09.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.116+0000 7f0bf3fff640 1 monclient: found mon.noname-c 2024-09-16T03:46:09.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.116+0000 7f0bf3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:46:09.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.116+0000 7f0bf3fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:46:09.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.116+0000 7f0bf3fff640 1 -- 172.21.15.42:0/1247227672 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0bf4073a30 con 0x7f0bf4072d40 2024-09-16T03:46:09.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.116+0000 7f0bf37fe640 1 --2- 172.21.15.42:0/1247227672 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0bf409c920 0x7f0bf41334e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:46:09.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.116+0000 7f0bf3fff640 1 --2- 172.21.15.42:0/1247227672 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0bf4072d40 0x7f0bf41cf0b0 secure :-1 s=READY pgs=2425 cs=0 l=1 rev1=1 crypto rx=0x7f0be400d5f0 tx=0x7f0be400dab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:46:09.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.120+0000 7f0bf0ff9640 10 client.?.objecter ms_handle_connect 0x7f0bf4072d40 2024-09-16T03:46:09.140 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.120+0000 7f0bf0ff9640 10 client.?.objecter resend_mon_ops 2024-09-16T03:46:09.140 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.120+0000 7f0bf0ff9640 1 -- 172.21.15.42:0/1247227672 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0be40125f0 con 0x7f0bf4072d40 2024-09-16T03:46:09.140 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.120+0000 7f0bf0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:46:09.140 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.120+0000 7f0bf0ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:46:09.140 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.120+0000 7f0bf0ff9640 10 monclient: dump: 2024-09-16T03:46:09.140 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:46:09.140 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:46:09.140 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:09.140 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:09.140 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:46:09.140 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:46:09.141 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:46:09.141 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:46:09.141 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:46:09.141 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:46:09.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.120+0000 7f0bf0ff9640 10 monclient: _finish_auth 0 2024-09-16T03:46:09.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.120+0000 7f0bf0ff9640 10 monclient: _check_auth_tickets 2024-09-16T03:46:09.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.120+0000 7f0bf0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:46:09.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.120+0000 7f0bfa751640 5 monclient: authenticate success, global_id 13568 2024-09-16T03:46:09.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.120+0000 7f0bfa751640 10 monclient: _renew_subs 2024-09-16T03:46:09.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.120+0000 7f0bfa751640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:46:09.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.120+0000 7f0bf0ff9640 1 -- 172.21.15.42:0/1247227672 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0be4019070 con 0x7f0bf4072d40 2024-09-16T03:46:09.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.120+0000 7f0bfa751640 1 -- 172.21.15.42:0/1247227672 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0bf409e730 con 0x7f0bf4072d40 2024-09-16T03:46:09.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.120+0000 7f0bf0ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T03:46:09.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.120+0000 7f0bf0ff9640 1 -- 172.21.15.42:0/1247227672 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0be40129b0 con 0x7f0bf4072d40 2024-09-16T03:46:09.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.120+0000 7f0bf0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:46:09.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.120+0000 7f0bf0ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:46:09.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.120+0000 7f0bf27fc640 4 set_mon_vals no callback set 2024-09-16T03:46:09.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.120+0000 7f0bfa751640 10 client.13568.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:46:09.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.120+0000 7f0bf0ff9640 10 monclient: dump: 2024-09-16T03:46:09.142 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:46:09.142 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:46:09.142 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:09.142 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:09.142 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:46:09.142 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:46:09.142 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:46:09.143 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:46:09.143 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:46:09.143 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:46:09.143 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.120+0000 7f0bfa751640 10 monclient: _renew_subs 2024-09-16T03:46:09.143 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.120+0000 7f0bfa751640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:46:09.143 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.120+0000 7f0bfa751640 1 -- 172.21.15.42:0/1247227672 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f0bf411db90 con 0x7f0bf4072d40 2024-09-16T03:46:09.143 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.120+0000 7f0bfa751640 1 librados: init done 2024-09-16T03:46:09.143 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.120+0000 7f0bfa751640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:46:09.143 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.120+0000 7f0bfa751640 1 --2- 172.21.15.42:0/1247227672 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0bf4133a20 0x7f0bf402c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:46:09.143 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.120+0000 7f0bfa751640 10 monclient(hunting): start opening mon connection 2024-09-16T03:46:09.143 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.120+0000 7f0bfa751640 1 -- 172.21.15.42:0/1247227672 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f0bf4137120 con 0x7f0bf4133a20 2024-09-16T03:46:09.143 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.120+0000 7f0bf37fe640 1 --2- 172.21.15.42:0/1247227672 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0bf4133a20 0x7f0bf402c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:46:09.143 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.120+0000 7f0bf37fe640 10 monclient: get_auth_request con 0x7f0bf4133a20 auth_method 0 2024-09-16T03:46:09.143 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.120+0000 7f0bf37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:46:09.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.120+0000 7f0bf37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:46:09.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.120+0000 7f0bf37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:46:09.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.120+0000 7f0bf37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:46:09.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.120+0000 7f0bf37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:46:09.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.120+0000 7f0bf37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:46:09.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.120+0000 7f0bf37fe640 10 monclient(hunting): handle_auth_done global_id 13571 payload 824 2024-09-16T03:46:09.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.120+0000 7f0bf37fe640 1 -- 172.21.15.42:0/1247227672 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0bf4139c60 con 0x7f0bf4133a20 2024-09-16T03:46:09.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.120+0000 7f0bf0ff9640 10 client.13568.objecter ms_handle_connect 0x7f0bf4133a20 2024-09-16T03:46:09.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.120+0000 7f0bf0ff9640 10 client.13568.objecter resend_mon_ops 2024-09-16T03:46:09.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.120+0000 7f0bf37fe640 1 --2- 172.21.15.42:0/1247227672 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0bf4133a20 0x7f0bf402c650 secure :-1 s=READY pgs=2426 cs=0 l=1 rev1=1 crypto rx=0x7f0be801a060 tx=0x7f0be801ac70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:46:09.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.120+0000 7f0bf0ff9640 1 -- 172.21.15.42:0/1247227672 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f0be4012b50 con 0x7f0bf4072d40 2024-09-16T03:46:09.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.120+0000 7f0bf0ff9640 1 --2- 172.21.15.42:0/1247227672 >> v2:172.21.15.154:6812/254845342 conn(0x7f0be0037810 0x7f0be0039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:46:09.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.120+0000 7f0bf0ff9640 1 -- 172.21.15.42:0/1247227672 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(123..123 src has 1..123) ==== 6278+0+0 (secure 0 0 0) 0x7f0be4049520 con 0x7f0bf4072d40 2024-09-16T03:46:09.145 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.120+0000 7f0bf0ff9640 10 client.13568.objecter ms_dispatch 0x7f0bf4000ba0 osd_map(123..123 src has 1..123) 2024-09-16T03:46:09.145 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.120+0000 7f0bf0ff9640 3 client.13568.objecter handle_osd_map got epochs [123,123] > 0 2024-09-16T03:46:09.145 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.120+0000 7f0bf0ff9640 3 client.13568.objecter handle_osd_map decoding full epoch 123 2024-09-16T03:46:09.145 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.120+0000 7f0bf0ff9640 20 client.13568.objecter dump_active .. 0 homeless 2024-09-16T03:46:09.145 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.120+0000 7f0bf0ff9640 1 -- 172.21.15.42:0/1247227672 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0be80b0040 con 0x7f0bf4133a20 2024-09-16T03:46:09.145 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.120+0000 7f0bf0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:46:09.145 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.120+0000 7f0bf0ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:46:09.145 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.120+0000 7f0bf0ff9640 10 monclient: dump: 2024-09-16T03:46:09.145 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:46:09.145 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:46:09.145 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:09.145 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:09.145 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:46:09.146 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:46:09.146 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:46:09.146 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:46:09.146 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:46:09.146 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:46:09.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.120+0000 7f0bf0ff9640 1 -- 172.21.15.42:0/1247227672 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f0be80b6670 con 0x7f0bf4133a20 2024-09-16T03:46:09.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.120+0000 7f0bf0ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:46:09.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.120+0000 7f0bf0ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:46:09.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.120+0000 7f0bf0ff9640 1 -- 172.21.15.42:0/1247227672 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0bf4133a20 msgr2=0x7f0bf402c650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:46:09.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.120+0000 7f0bf0ff9640 1 --2- 172.21.15.42:0/1247227672 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0bf4133a20 0x7f0bf402c650 secure :-1 s=READY pgs=2426 cs=0 l=1 rev1=1 crypto rx=0x7f0be801a060 tx=0x7f0be801ac70 comp rx=0 tx=0).stop 2024-09-16T03:46:09.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.120+0000 7f0bf0ff9640 1 -- 172.21.15.42:0/1247227672 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0bf4133a20 msgr2=0x7f0bf402c650 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:46:09.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.120+0000 7f0bf0ff9640 1 --2- 172.21.15.42:0/1247227672 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0bf4133a20 0x7f0bf402c650 secure :-1 s=CLOSED pgs=2426 cs=0 l=1 rev1=1 crypto rx=0x7f0be801a060 tx=0x7f0be801ac70 comp rx=0 tx=0).stop 2024-09-16T03:46:09.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.120+0000 7f0bf8cee640 1 --2- 172.21.15.42:0/1247227672 >> v2:172.21.15.154:6812/254845342 conn(0x7f0be0037810 0x7f0be0039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:46:09.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.120+0000 7f0bf8cee640 10 monclient: get_auth_request con 0x7f0be0037810 auth_method 0 2024-09-16T03:46:09.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.120+0000 7f0bf8cee640 1 --2- 172.21.15.42:0/1247227672 >> v2:172.21.15.154:6812/254845342 conn(0x7f0be0037810 0x7f0be0039cd0 secure :-1 s=READY pgs=3405 cs=0 l=1 rev1=1 crypto rx=0x7f0bec0054b0 tx=0x7f0bec009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:46:09.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.120+0000 7f0bf0ff9640 10 client.13568.objecter ms_handle_connect 0x7f0be0037810 2024-09-16T03:46:09.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.128+0000 7f0bfa751640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:46:09.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.128+0000 7f0bfa751640 1 --2- 172.21.15.42:0/1247227672 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0bf402cb90 0x7f0bf402cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:46:09.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.128+0000 7f0bfa751640 10 monclient(hunting): start opening mon connection 2024-09-16T03:46:09.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.128+0000 7f0bfa751640 1 -- 172.21.15.42:0/1247227672 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f0bf413ec40 con 0x7f0bf402cb90 2024-09-16T03:46:09.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.128+0000 7f0bf37fe640 1 --2- 172.21.15.42:0/1247227672 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0bf402cb90 0x7f0bf402cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:46:09.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.128+0000 7f0bf37fe640 10 monclient: get_auth_request con 0x7f0bf402cb90 auth_method 0 2024-09-16T03:46:09.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.128+0000 7f0bf37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:46:09.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.128+0000 7f0bf37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:46:09.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.128+0000 7f0bf37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:46:09.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.128+0000 7f0bf37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:46:09.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.128+0000 7f0bf37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:46:09.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.128+0000 7f0bf37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:46:09.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.128+0000 7f0bf37fe640 10 monclient(hunting): handle_auth_done global_id 13574 payload 824 2024-09-16T03:46:09.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.128+0000 7f0bf37fe640 1 -- 172.21.15.42:0/1247227672 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f0bf4069040 con 0x7f0bf402cb90 2024-09-16T03:46:09.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.128+0000 7f0bf37fe640 1 --2- 172.21.15.42:0/1247227672 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0bf402cb90 0x7f0bf402cfa0 secure :-1 s=READY pgs=2427 cs=0 l=1 rev1=1 crypto rx=0x7f0bf402dd90 tx=0x7f0be80191d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:46:09.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.128+0000 7f0bf0ff9640 10 client.13568.objecter ms_handle_connect 0x7f0bf402cb90 2024-09-16T03:46:09.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.128+0000 7f0bf0ff9640 10 client.13568.objecter resend_mon_ops 2024-09-16T03:46:09.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.128+0000 7f0bf0ff9640 1 -- 172.21.15.42:0/1247227672 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0be80b0040 con 0x7f0bf402cb90 2024-09-16T03:46:09.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.128+0000 7f0bf0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:46:09.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.128+0000 7f0bf0ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:46:09.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.128+0000 7f0bf0ff9640 10 monclient: dump: 2024-09-16T03:46:09.150 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:46:09.150 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:46:09.150 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:09.150 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:09.150 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:46:09.151 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:46:09.151 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:46:09.151 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:46:09.151 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:46:09.151 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:46:09.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.128+0000 7f0bf0ff9640 1 -- 172.21.15.42:0/1247227672 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4879 (secure 0 0 0) 0x7f0be80b7650 con 0x7f0bf402cb90 2024-09-16T03:46:09.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.128+0000 7f0bf0ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:46:09.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.128+0000 7f0bf0ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:46:09.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.128+0000 7f0bf0ff9640 1 -- 172.21.15.42:0/1247227672 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0bf402cb90 msgr2=0x7f0bf402cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:46:09.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.128+0000 7f0bf0ff9640 1 --2- 172.21.15.42:0/1247227672 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0bf402cb90 0x7f0bf402cfa0 secure :-1 s=READY pgs=2427 cs=0 l=1 rev1=1 crypto rx=0x7f0bf402dd90 tx=0x7f0be80191d0 comp rx=0 tx=0).stop 2024-09-16T03:46:09.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.128+0000 7f0bf0ff9640 1 -- 172.21.15.42:0/1247227672 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0bf402cb90 msgr2=0x7f0bf402cfa0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:46:09.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.128+0000 7f0bf0ff9640 1 --2- 172.21.15.42:0/1247227672 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0bf402cb90 0x7f0bf402cfa0 secure :-1 s=CLOSED pgs=2427 cs=0 l=1 rev1=1 crypto rx=0x7f0bf402dd90 tx=0x7f0be80191d0 comp rx=0 tx=0).stop 2024-09-16T03:46:09.152 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:46:09.152 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:46:09.152 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:46:09.152 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T03:46:09.152 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 34, 2024-09-16T03:46:09.152 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 248499, 2024-09-16T03:46:09.152 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:46:09.152 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:46:09.152 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T03:46:09.153 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T03:46:09.153 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:46:09.153 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 5, 2024-09-16T03:46:09.153 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:46:09.153 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:46:09.153 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:46:09.153 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:46:09.153 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:46:09.153 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:46:09.153 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:46:09.153 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:46:09.153 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:46:09.153 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:46:09.154 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:46:09.154 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:46:09.154 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:46:09.154 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:46:09.154 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:46:09.154 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:46:09.154 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:46:09.154 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:46:09.154 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:46:09.154 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:46:09.154 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:46:09.154 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:46:09.154 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:46:09.154 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:46:09.155 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:46:09.155 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:46:09.155 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:46:09.155 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:46:09.155 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:46:09.155 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:46:09.155 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:46:09.155 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:46:09.155 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:46:09.155 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:46:09.155 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:46:09.155 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:46:09.156 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:46:09.156 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:46:09.156 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:46:09.156 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:46:09.156 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:46:09.156 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:46:09.156 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:46:09.157 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:46:09.157 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:46:09.157 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:46:09.157 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:46:09.157 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:46:09.157 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:46:09.157 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:46:09.157 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:46:09.157 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:46:09.157 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:46:09.157 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:46:09.157 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:46:09.157 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:46:09.157 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:46:09.158 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:46:09.158 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:46:09.158 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:46:09.158 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:46:09.158 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:46:09.158 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:46:09.158 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:46:09.158 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:46:09.158 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:46:09.158 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:46:09.158 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:46:09.158 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:46:09.158 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:46:09.159 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:46:09.159 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:46:09.159 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:46:09.159 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:46:09.159 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:46:09.159 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:46:09.159 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:46:09.159 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:46:09.159 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:46:09.159 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:46:09.159 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:46:09.159 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:46:09.159 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:46:09.160 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:46:09.160 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:46:09.160 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:46:09.160 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:46:09.160 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:46:09.160 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:46:09.160 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:46:09.160 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:46:09.160 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:46:09.160 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:46:09.160 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:46:09.160 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:46:09.160 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:46:09.160 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:46:09.161 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:46:09.161 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:46:09.161 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:46:09.161 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:46:09.161 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:46:09.161 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:46:09.161 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:46:09.161 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:46:09.161 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:46:09.161 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:46:09.161 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:46:09.161 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:46:09.161 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:46:09.161 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:46:09.162 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:46:09.162 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:46:09.162 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:46:09.162 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:46:09.162 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:46:09.162 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:46:09.162 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:46:09.162 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:46:09.162 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:46:09.162 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:46:09.162 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:46:09.162 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:46:09.162 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:46:09.162 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:46:09.163 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:46:09.163 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:46:09.163 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:46:09.163 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:46:09.163 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:46:09.163 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:46:09.163 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:46:09.163 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:46:09.163 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:46:09.163 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:46:09.163 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:46:09.163 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:46:09.163 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:46:09.163 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:46:09.164 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:46:09.164 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:46:09.164 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T03:46:09.164 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:46:09.164 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:46:09.164 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:46:09.164 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 5 2024-09-16T03:46:09.164 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:46:09.164 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:46:09.164 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:46:09.164 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:46:09.164 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:46:09.164 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:46:09.164 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 8 2024-09-16T03:46:09.165 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:46:09.165 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:46:09.165 INFO:teuthology.orchestra.run.smithi042.stdout: "mgr": [ 2024-09-16T03:46:09.165 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:46:09.165 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:46:09.165 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:46:09.165 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:46:09.165 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:46:09.165 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:46:09.165 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:46:09.165 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:46:09.165 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:46:09.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.128+0000 7f0bfa751640 10 librados: watch_flush enter 2024-09-16T03:46:09.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.128+0000 7f0bfa751640 10 librados: watch_flush exit 2024-09-16T03:46:09.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.128+0000 7f0bfa751640 20 client.13568.objecter shutdown clearing up homeless session... 2024-09-16T03:46:09.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.128+0000 7f0bfa751640 10 client.13568.objecter successfully canceled tick 2024-09-16T03:46:09.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.128+0000 7f0bfa751640 1 -- 172.21.15.42:0/1247227672 >> v2:172.21.15.154:6812/254845342 conn(0x7f0be0037810 msgr2=0x7f0be0039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:46:09.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.128+0000 7f0bfa751640 1 --2- 172.21.15.42:0/1247227672 >> v2:172.21.15.154:6812/254845342 conn(0x7f0be0037810 0x7f0be0039cd0 secure :-1 s=READY pgs=3405 cs=0 l=1 rev1=1 crypto rx=0x7f0bec0054b0 tx=0x7f0bec009040 comp rx=0 tx=0).stop 2024-09-16T03:46:09.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.128+0000 7f0bfa751640 10 monclient: shutdown 2024-09-16T03:46:09.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.128+0000 7f0bfa751640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:46:09.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.128+0000 7f0bfa751640 1 -- 172.21.15.42:0/1247227672 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0bf4072d40 msgr2=0x7f0bf41cf0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:46:09.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.128+0000 7f0bfa751640 1 --2- 172.21.15.42:0/1247227672 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0bf4072d40 0x7f0bf41cf0b0 secure :-1 s=READY pgs=2425 cs=0 l=1 rev1=1 crypto rx=0x7f0be400d5f0 tx=0x7f0be400dab0 comp rx=0 tx=0).stop 2024-09-16T03:46:09.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.128+0000 7f0bf8cee640 1 -- 172.21.15.42:0/1247227672 reap_dead start 2024-09-16T03:46:09.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.128+0000 7f0bfa751640 1 -- 172.21.15.42:0/1247227672 shutdown_connections 2024-09-16T03:46:09.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.128+0000 7f0bfa751640 1 --2- 172.21.15.42:0/1247227672 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0bf4072d40 0x7f0bf41cf0b0 unknown :-1 s=CLOSED pgs=2425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:46:09.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.128+0000 7f0bfa751640 1 -- 172.21.15.42:0/1247227672 >> 172.21.15.42:0/1247227672 conn(0x7f0bf40a2420 msgr2=0x7f0bf4136310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:46:09.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.128+0000 7f0bfa751640 1 -- 172.21.15.42:0/1247227672 shutdown_connections 2024-09-16T03:46:09.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.128+0000 7f0bfa751640 1 -- 172.21.15.42:0/1247227672 wait complete. 2024-09-16T03:46:09.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.128+0000 7f0bfa751640 1 librados: shutdown 2024-09-16T03:46:09.168 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:46:09.169 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T03:46:09.169 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T03:46:09.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.280+0000 7f256edea640 10 monclient: get_monmap_and_config 2024-09-16T03:46:09.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.280+0000 7f256edea640 10 monclient: build_initial_monmap 2024-09-16T03:46:09.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.280+0000 7f256edea640 10 monclient: monmap: 2024-09-16T03:46:09.286 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:46:09.286 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:46:09.286 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:46:09.284991+0000 2024-09-16T03:46:09.286 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:46:09.284991+0000 2024-09-16T03:46:09.286 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:46:09.286 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:46:09.286 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:46:09.287 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:46:09.287 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:46:09.287 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:46:09.287 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:46:09.287 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:46:09.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.280+0000 7f256edea640 1 Processor -- start 2024-09-16T03:46:09.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.280+0000 7f256edea640 1 -- start start 2024-09-16T03:46:09.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.280+0000 7f256edea640 10 monclient: init 2024-09-16T03:46:09.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.280+0000 7f256edea640 10 monclient: _reopen_session rank -1 2024-09-16T03:46:09.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.280+0000 7f256edea640 10 monclient: _add_conns ranks=[0,3,1,2,4] 2024-09-16T03:46:09.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.280+0000 7f256edea640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f256809fb00 addr v1:172.21.15.42:6789/0 2024-09-16T03:46:09.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.280+0000 7f256edea640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f256809c160 0x7f2568139120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:46:09.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.280+0000 7f256edea640 10 monclient(hunting): picked mon.noname-a con 0x7f256809c160 addr v2:172.21.15.42:3300/0 2024-09-16T03:46:09.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.280+0000 7f256edea640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f256809c5d0 addr v1:172.21.15.154:6789/0 2024-09-16T03:46:09.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.280+0000 7f256edea640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f25680a1e20 con 0x7f256809fb00 2024-09-16T03:46:09.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.280+0000 7f256edea640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f256809ea30 con 0x7f256809c5d0 2024-09-16T03:46:09.291 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.280+0000 7f256edea640 10 monclient(hunting): start opening mon connection 2024-09-16T03:46:09.291 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.280+0000 7f256edea640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f256809e250 con 0x7f256809c160 2024-09-16T03:46:09.291 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.280+0000 7f256edea640 10 monclient(hunting): _renew_subs 2024-09-16T03:46:09.291 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.280+0000 7f256edea640 10 monclient(hunting): authenticate will time out at 1216.254933s 2024-09-16T03:46:09.291 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.280+0000 7f2567fff640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f256809c160 0x7f2568139120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:46:09.291 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.280+0000 7f256cb86640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f256809fb00 0x7f256809ba50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:48174/0 (socket says 172.21.15.42:48174) 2024-09-16T03:46:09.291 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.280+0000 7f256cb86640 1 -- 172.21.15.42:0/3736429960 learned_addr learned my addr 172.21.15.42:0/3736429960 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:46:09.291 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.280+0000 7f2567fff640 10 monclient(hunting): get_auth_request con 0x7f256809c160 auth_method 0 2024-09-16T03:46:09.291 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.280+0000 7f2567fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:46:09.291 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.280+0000 7f2567fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:46:09.291 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.280+0000 7f2567fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:46:09.292 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.280+0000 7f2567fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:46:09.292 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.280+0000 7f2567fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:46:09.292 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.280+0000 7f2567fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:46:09.292 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.280+0000 7f2567fff640 10 monclient(hunting): handle_auth_done global_id 744438 payload 274 2024-09-16T03:46:09.292 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.280+0000 7f2567fff640 1 -- 172.21.15.42:0/3736429960 >> v1:172.21.15.154:6789/0 conn(0x7f256809c5d0 legacy=0x7f256813b670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:46:09.292 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.280+0000 7f2567fff640 1 -- 172.21.15.42:0/3736429960 >> v1:172.21.15.42:6789/0 conn(0x7f256809fb00 legacy=0x7f256809ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:46:09.292 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.280+0000 7f2567fff640 10 monclient: _finish_hunting 0 2024-09-16T03:46:09.292 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.280+0000 7f2567fff640 1 monclient: found mon.noname-a 2024-09-16T03:46:09.292 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.280+0000 7f2567fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:46:09.292 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.280+0000 7f2567fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:46:09.292 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.280+0000 7f2567fff640 1 -- 172.21.15.42:0/3736429960 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f256809fee0 con 0x7f256809c160 2024-09-16T03:46:09.300 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.284+0000 7f2567fff640 1 --2- 172.21.15.42:0/3736429960 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f256809c160 0x7f2568139120 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f256809e3d0 tx=0x7f255401c360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70ce693b3b855802 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:46:09.300 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.284+0000 7f25677fe640 1 -- 172.21.15.42:0/3736429960 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25540b2070 con 0x7f256809c160 2024-09-16T03:46:09.300 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.284+0000 7f25677fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:46:09.300 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.284+0000 7f25677fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:46:09.300 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.284+0000 7f25677fe640 10 monclient: dump: 2024-09-16T03:46:09.301 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:46:09.301 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:46:09.301 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:09.301 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:09.301 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:46:09.301 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:46:09.301 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:46:09.301 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:46:09.301 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:46:09.301 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:46:09.301 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.284+0000 7f25677fe640 10 monclient: _finish_auth 0 2024-09-16T03:46:09.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.284+0000 7f25677fe640 10 monclient: _check_auth_tickets 2024-09-16T03:46:09.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.284+0000 7f25677fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:46:09.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.284+0000 7f256edea640 5 monclient: authenticate success, global_id 744438 2024-09-16T03:46:09.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.284+0000 7f25677fe640 1 -- 172.21.15.42:0/3736429960 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f25540a9070 con 0x7f256809c160 2024-09-16T03:46:09.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.284+0000 7f256edea640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:46:09.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.284+0000 7f25677fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:46:09.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.284+0000 7f256edea640 10 monclient: get_monmap_and_config success 2024-09-16T03:46:09.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.284+0000 7f256edea640 4 set_mon_vals no callback set 2024-09-16T03:46:09.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.284+0000 7f25677fe640 1 -- 172.21.15.42:0/3736429960 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25540ad880 con 0x7f256809c160 2024-09-16T03:46:09.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.284+0000 7f25677fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:46:09.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.284+0000 7f25677fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:46:09.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.284+0000 7f25677fe640 10 monclient: dump: 2024-09-16T03:46:09.303 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:46:09.303 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:46:09.303 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:09.303 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:09.303 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:46:09.303 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:46:09.303 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:46:09.303 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:46:09.303 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:46:09.303 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:46:09.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.284+0000 7f256edea640 10 monclient: shutdown 2024-09-16T03:46:09.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.284+0000 7f256edea640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:46:09.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.284+0000 7f256edea640 1 -- 172.21.15.42:0/3736429960 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f256809c160 msgr2=0x7f2568139120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:46:09.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.284+0000 7f256edea640 1 --2- 172.21.15.42:0/3736429960 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f256809c160 0x7f2568139120 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f256809e3d0 tx=0x7f255401c360 comp rx=0 tx=0).stop 2024-09-16T03:46:09.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.284+0000 7f256edea640 1 -- 172.21.15.42:0/3736429960 shutdown_connections 2024-09-16T03:46:09.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.284+0000 7f256edea640 1 --2- 172.21.15.42:0/3736429960 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f256809c160 0x7f2568139120 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:46:09.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.284+0000 7f256edea640 1 -- 172.21.15.42:0/3736429960 >> 172.21.15.42:0/3736429960 conn(0x7f25680a2420 msgr2=0x7f25680a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:46:09.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.284+0000 7f256edea640 1 -- 172.21.15.42:0/3736429960 shutdown_connections 2024-09-16T03:46:09.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.284+0000 7f256edea640 1 -- 172.21.15.42:0/3736429960 wait complete. 2024-09-16T03:46:09.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.284+0000 7f256edea640 10 monclient: build_initial_monmap 2024-09-16T03:46:09.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.284+0000 7f256edea640 10 monclient: monmap: 2024-09-16T03:46:09.304 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:46:09.305 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:46:09.305 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:46:09.305 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:46:09.305 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:46:09.305 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:46:09.305 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:46:09.305 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:46:09.305 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:46:09.305 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:46:09.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.284+0000 7f256edea640 1 librados: starting msgr at 2024-09-16T03:46:09.306 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.284+0000 7f256edea640 1 librados: starting objecter 2024-09-16T03:46:09.306 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.284+0000 7f256edea640 1 Processor -- start 2024-09-16T03:46:09.306 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.288+0000 7f256edea640 1 -- start start 2024-09-16T03:46:09.306 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.288+0000 7f256edea640 1 librados: setting wanted keys 2024-09-16T03:46:09.306 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.288+0000 7f256edea640 1 librados: calling monclient init 2024-09-16T03:46:09.306 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.288+0000 7f256edea640 10 monclient: init 2024-09-16T03:46:09.306 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.288+0000 7f256edea640 10 monclient: _reopen_session rank -1 2024-09-16T03:46:09.306 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.288+0000 7f256edea640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T03:46:09.306 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.288+0000 7f256edea640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f256809c160 0x7f25681cf5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:46:09.306 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.288+0000 7f256edea640 10 monclient(hunting): picked mon.noname-b con 0x7f256809c160 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:46:09.307 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.288+0000 7f256edea640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f256809c5d0 0x7f25681cfb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:46:09.307 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.288+0000 7f256edea640 10 monclient(hunting): picked mon.noname-c con 0x7f256809c5d0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:46:09.307 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.288+0000 7f256edea640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f256809fb00 0x7f2568120710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:46:09.307 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.288+0000 7f256edea640 10 monclient(hunting): picked mon.noname-a con 0x7f256809fb00 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:46:09.307 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.288+0000 7f256edea640 10 monclient(hunting): start opening mon connection 2024-09-16T03:46:09.307 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.288+0000 7f256edea640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f256809d380 con 0x7f256809fb00 2024-09-16T03:46:09.307 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.288+0000 7f256edea640 10 monclient(hunting): start opening mon connection 2024-09-16T03:46:09.307 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.288+0000 7f256edea640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f256809d500 con 0x7f256809c160 2024-09-16T03:46:09.307 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.288+0000 7f256edea640 10 monclient(hunting): start opening mon connection 2024-09-16T03:46:09.307 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.288+0000 7f256edea640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f25680a1330 con 0x7f256809c5d0 2024-09-16T03:46:09.307 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.288+0000 7f256edea640 10 monclient(hunting): _renew_subs 2024-09-16T03:46:09.308 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.288+0000 7f256edea640 10 monclient(hunting): authenticate will time out at 1216.260603s 2024-09-16T03:46:09.308 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.288+0000 7f2567fff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f256809c5d0 0x7f25681cfb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:46:09.308 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.288+0000 7f256d387640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f256809fb00 0x7f2568120710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:46:09.308 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.288+0000 7f2567fff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f256809c5d0 0x7f25681cfb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:51806/0 (socket says 172.21.15.42:51806) 2024-09-16T03:46:09.308 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.288+0000 7f256d387640 10 monclient(hunting): get_auth_request con 0x7f256809fb00 auth_method 0 2024-09-16T03:46:09.308 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.288+0000 7f2567fff640 1 -- 172.21.15.42:0/2639089672 learned_addr learned my addr 172.21.15.42:0/2639089672 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:46:09.308 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.288+0000 7f256d387640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:46:09.308 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.288+0000 7f256d387640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:46:09.308 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.288+0000 7f256d387640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:46:09.308 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.288+0000 7f2567fff640 10 monclient(hunting): get_auth_request con 0x7f256809c5d0 auth_method 0 2024-09-16T03:46:09.308 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.288+0000 7f2567fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:46:09.309 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.288+0000 7f2567fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:46:09.309 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.288+0000 7f2567fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:46:09.309 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.288+0000 7f256d387640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:46:09.309 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.288+0000 7f256d387640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:46:09.309 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.288+0000 7f256d387640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:46:09.309 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.288+0000 7f2567fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:46:09.309 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.288+0000 7f2567fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:46:09.309 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.288+0000 7f2567fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:46:09.309 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.288+0000 7f256cb86640 1 --2- 172.21.15.42:0/2639089672 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f256809c160 0x7f25681cf5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:46:09.309 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.288+0000 7f256cb86640 10 monclient(hunting): get_auth_request con 0x7f256809c160 auth_method 0 2024-09-16T03:46:09.309 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.288+0000 7f256cb86640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:46:09.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.288+0000 7f256cb86640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:46:09.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.288+0000 7f256cb86640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:46:09.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.288+0000 7f256d387640 10 monclient(hunting): handle_auth_done global_id 744444 payload 824 2024-09-16T03:46:09.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.288+0000 7f256d387640 1 -- 172.21.15.42:0/2639089672 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f256809c5d0 msgr2=0x7f25681cfb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:46:09.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.288+0000 7f256d387640 1 --2- 172.21.15.42:0/2639089672 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f256809c5d0 0x7f25681cfb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:46:09.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.288+0000 7f256d387640 1 -- 172.21.15.42:0/2639089672 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f256809c160 msgr2=0x7f25681cf5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:46:09.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.288+0000 7f256d387640 1 --2- 172.21.15.42:0/2639089672 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f256809c160 0x7f25681cf5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:46:09.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.288+0000 7f256d387640 10 monclient: _finish_hunting 0 2024-09-16T03:46:09.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.288+0000 7f256d387640 1 monclient: found mon.noname-a 2024-09-16T03:46:09.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.288+0000 7f256d387640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:46:09.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.288+0000 7f256d387640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:46:09.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.288+0000 7f256d387640 1 -- 172.21.15.42:0/2639089672 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25681284a0 con 0x7f256809fb00 2024-09-16T03:46:09.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.288+0000 7f2567fff640 1 --2- 172.21.15.42:0/2639089672 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f256809c5d0 0x7f25681cfb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:46:09.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.288+0000 7f256d387640 1 --2- 172.21.15.42:0/2639089672 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f256809fb00 0x7f2568120710 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f256000c700 tx=0x7f256000cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:46:09.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.288+0000 7f25657fa640 10 client.?.objecter ms_handle_connect 0x7f256809fb00 2024-09-16T03:46:09.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.288+0000 7f25657fa640 10 client.?.objecter resend_mon_ops 2024-09-16T03:46:09.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.288+0000 7f25657fa640 1 -- 172.21.15.42:0/2639089672 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2560003b80 con 0x7f256809fb00 2024-09-16T03:46:09.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.288+0000 7f25657fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:46:09.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.288+0000 7f25657fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:46:09.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.288+0000 7f25657fa640 10 monclient: dump: 2024-09-16T03:46:09.311 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:46:09.311 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:46:09.311 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:09.311 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:09.311 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:46:09.312 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:46:09.312 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:46:09.312 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:46:09.312 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:46:09.312 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:46:09.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.288+0000 7f25657fa640 10 monclient: _finish_auth 0 2024-09-16T03:46:09.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.288+0000 7f25657fa640 10 monclient: _check_auth_tickets 2024-09-16T03:46:09.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.288+0000 7f25657fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:46:09.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.288+0000 7f25657fa640 1 -- 172.21.15.42:0/2639089672 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2560003d20 con 0x7f256809fb00 2024-09-16T03:46:09.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.288+0000 7f25657fa640 10 monclient: handle_config config(0 keys) 2024-09-16T03:46:09.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.288+0000 7f25657fa640 1 -- 172.21.15.42:0/2639089672 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f256000aa30 con 0x7f256809fb00 2024-09-16T03:46:09.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.288+0000 7f256edea640 5 monclient: authenticate success, global_id 744444 2024-09-16T03:46:09.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.288+0000 7f256edea640 10 monclient: _renew_subs 2024-09-16T03:46:09.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.288+0000 7f256edea640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:46:09.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.288+0000 7f256edea640 1 -- 172.21.15.42:0/2639089672 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25680a0d70 con 0x7f256809fb00 2024-09-16T03:46:09.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.288+0000 7f2566ffd640 4 set_mon_vals no callback set 2024-09-16T03:46:09.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.288+0000 7f25657fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:46:09.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.288+0000 7f25657fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:46:09.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.288+0000 7f25657fa640 10 monclient: dump: 2024-09-16T03:46:09.313 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:46:09.313 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:46:09.313 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:09.314 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:09.314 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:46:09.314 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:46:09.314 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:46:09.314 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:46:09.314 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:46:09.314 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:46:09.314 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.288+0000 7f256edea640 10 client.744444.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:46:09.314 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.288+0000 7f256edea640 10 monclient: _renew_subs 2024-09-16T03:46:09.314 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.288+0000 7f256edea640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:46:09.314 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.288+0000 7f256edea640 1 -- 172.21.15.42:0/2639089672 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25681281c0 con 0x7f256809fb00 2024-09-16T03:46:09.314 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.288+0000 7f256edea640 1 librados: init done 2024-09-16T03:46:09.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.288+0000 7f256edea640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:46:09.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.288+0000 7f256edea640 1 --2- 172.21.15.42:0/2639089672 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2568120c50 0x7f2568121030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:46:09.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.288+0000 7f256edea640 10 monclient(hunting): start opening mon connection 2024-09-16T03:46:09.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.288+0000 7f256edea640 1 -- 172.21.15.42:0/2639089672 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f256809d1a0 con 0x7f2568120c50 2024-09-16T03:46:09.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.288+0000 7f25657fa640 1 -- 172.21.15.42:0/2639089672 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f256000abd0 con 0x7f256809fb00 2024-09-16T03:46:09.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.288+0000 7f256cb86640 1 --2- 172.21.15.42:0/2639089672 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2568120c50 0x7f2568121030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:46:09.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.288+0000 7f256cb86640 10 monclient: get_auth_request con 0x7f2568120c50 auth_method 0 2024-09-16T03:46:09.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.288+0000 7f256cb86640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:46:09.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.288+0000 7f256cb86640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:46:09.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.288+0000 7f256cb86640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:46:09.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.288+0000 7f25657fa640 1 --2- 172.21.15.42:0/2639089672 >> v2:172.21.15.154:6812/254845342 conn(0x7f254c037810 0x7f254c039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:46:09.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.288+0000 7f25657fa640 1 -- 172.21.15.42:0/2639089672 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(123..123 src has 1..123) ==== 6278+0+0 (secure 0 0 0) 0x7f2560048f50 con 0x7f256809fb00 2024-09-16T03:46:09.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.288+0000 7f25657fa640 10 client.744444.objecter ms_dispatch 0x7f2568000ba0 osd_map(123..123 src has 1..123) 2024-09-16T03:46:09.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.288+0000 7f25657fa640 3 client.744444.objecter handle_osd_map got epochs [123,123] > 0 2024-09-16T03:46:09.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.288+0000 7f25657fa640 3 client.744444.objecter handle_osd_map decoding full epoch 123 2024-09-16T03:46:09.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.288+0000 7f256cb86640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:46:09.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.288+0000 7f256cb86640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:46:09.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.288+0000 7f25657fa640 20 client.744444.objecter dump_active .. 0 homeless 2024-09-16T03:46:09.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.288+0000 7f256cb86640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:46:09.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.288+0000 7f2567fff640 1 --2- 172.21.15.42:0/2639089672 >> v2:172.21.15.154:6812/254845342 conn(0x7f254c037810 0x7f254c039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:46:09.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.288+0000 7f2567fff640 10 monclient: get_auth_request con 0x7f254c037810 auth_method 0 2024-09-16T03:46:09.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.288+0000 7f256cb86640 10 monclient(hunting): handle_auth_done global_id 754144 payload 824 2024-09-16T03:46:09.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.288+0000 7f256cb86640 1 -- 172.21.15.42:0/2639089672 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f25680724c0 con 0x7f2568120c50 2024-09-16T03:46:09.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.288+0000 7f25657fa640 10 client.744444.objecter ms_handle_connect 0x7f2568120c50 2024-09-16T03:46:09.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.288+0000 7f25657fa640 10 client.744444.objecter resend_mon_ops 2024-09-16T03:46:09.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.288+0000 7f256cb86640 1 --2- 172.21.15.42:0/2639089672 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2568120c50 0x7f2568121030 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f255c000f70 tx=0x7f255c00e960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:46:09.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.288+0000 7f25657fa640 1 -- 172.21.15.42:0/2639089672 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f255c005ad0 con 0x7f2568120c50 2024-09-16T03:46:09.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.288+0000 7f25657fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:46:09.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.288+0000 7f25657fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:46:09.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.288+0000 7f25657fa640 10 monclient: dump: 2024-09-16T03:46:09.317 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:46:09.318 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:46:09.318 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:09.318 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:09.318 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:46:09.318 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:46:09.318 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:46:09.318 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:46:09.318 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:46:09.318 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:46:09.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.292+0000 7f25657fa640 10 client.744444.objecter ms_handle_connect 0x7f254c037810 2024-09-16T03:46:09.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.292+0000 7f2567fff640 1 --2- 172.21.15.42:0/2639089672 >> v2:172.21.15.154:6812/254845342 conn(0x7f254c037810 0x7f254c039cd0 secure :-1 s=READY pgs=3406 cs=0 l=1 rev1=1 crypto rx=0x7f255401c3a0 tx=0x7f2554008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:46:09.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.292+0000 7f25657fa640 1 -- 172.21.15.42:0/2639089672 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f255c018070 con 0x7f2568120c50 2024-09-16T03:46:09.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.292+0000 7f25657fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:46:09.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.292+0000 7f25657fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:46:09.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.292+0000 7f25657fa640 1 -- 172.21.15.42:0/2639089672 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2568120c50 msgr2=0x7f2568121030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:46:09.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.292+0000 7f25657fa640 1 --2- 172.21.15.42:0/2639089672 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2568120c50 0x7f2568121030 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f255c000f70 tx=0x7f255c00e960 comp rx=0 tx=0).stop 2024-09-16T03:46:09.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.292+0000 7f25657fa640 1 -- 172.21.15.42:0/2639089672 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2568120c50 msgr2=0x7f2568121030 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:46:09.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.292+0000 7f25657fa640 1 --2- 172.21.15.42:0/2639089672 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2568120c50 0x7f2568121030 secure :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f255c000f70 tx=0x7f255c00e960 comp rx=0 tx=0).stop 2024-09-16T03:46:09.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.296+0000 7f256edea640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:46:09.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.296+0000 7f256edea640 1 --2- 172.21.15.42:0/2639089672 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2568121570 0x7f256811bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:46:09.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.296+0000 7f256edea640 10 monclient(hunting): start opening mon connection 2024-09-16T03:46:09.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.296+0000 7f256edea640 1 -- 172.21.15.42:0/2639089672 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f25680a16f0 con 0x7f2568121570 2024-09-16T03:46:09.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.296+0000 7f256cb86640 1 --2- 172.21.15.42:0/2639089672 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2568121570 0x7f256811bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:46:09.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.296+0000 7f256cb86640 10 monclient: get_auth_request con 0x7f2568121570 auth_method 0 2024-09-16T03:46:09.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.296+0000 7f256cb86640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:46:09.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.296+0000 7f256cb86640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:46:09.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.296+0000 7f256cb86640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:46:09.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.296+0000 7f256cb86640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:46:09.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.296+0000 7f256cb86640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:46:09.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.296+0000 7f256cb86640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:46:09.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.300+0000 7f256cb86640 10 monclient(hunting): handle_auth_done global_id 754147 payload 824 2024-09-16T03:46:09.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.300+0000 7f256cb86640 1 -- 172.21.15.42:0/2639089672 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f2568127020 con 0x7f2568121570 2024-09-16T03:46:09.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.300+0000 7f256cb86640 1 --2- 172.21.15.42:0/2639089672 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2568121570 0x7f256811bf70 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f255c000f70 tx=0x7f255c005800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:46:09.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.300+0000 7f25657fa640 10 client.744444.objecter ms_handle_connect 0x7f2568121570 2024-09-16T03:46:09.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.300+0000 7f25657fa640 10 client.744444.objecter resend_mon_ops 2024-09-16T03:46:09.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.300+0000 7f25657fa640 1 -- 172.21.15.42:0/2639089672 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f255c0072d0 con 0x7f2568121570 2024-09-16T03:46:09.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.300+0000 7f25657fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:46:09.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.300+0000 7f25657fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:46:09.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.300+0000 7f25657fa640 10 monclient: dump: 2024-09-16T03:46:09.323 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:46:09.323 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:46:09.323 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:09.323 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:09.323 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:46:09.324 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:46:09.324 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:46:09.324 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:46:09.324 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:46:09.324 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:46:09.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.300+0000 7f25657fa640 1 -- 172.21.15.42:0/2639089672 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4544 (secure 0 0 0) 0x7f255c0072d0 con 0x7f2568121570 2024-09-16T03:46:09.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.300+0000 7f25657fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:46:09.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.300+0000 7f25657fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:46:09.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.300+0000 7f25657fa640 1 -- 172.21.15.42:0/2639089672 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2568121570 msgr2=0x7f256811bf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:46:09.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.300+0000 7f25657fa640 1 --2- 172.21.15.42:0/2639089672 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2568121570 0x7f256811bf70 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f255c000f70 tx=0x7f255c005800 comp rx=0 tx=0).stop 2024-09-16T03:46:09.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.300+0000 7f25657fa640 1 -- 172.21.15.42:0/2639089672 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2568121570 msgr2=0x7f256811bf70 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:46:09.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.300+0000 7f25657fa640 1 --2- 172.21.15.42:0/2639089672 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2568121570 0x7f256811bf70 secure :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f255c000f70 tx=0x7f255c005800 comp rx=0 tx=0).stop 2024-09-16T03:46:09.325 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:46:09.325 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:46:09.325 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:46:09.325 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T03:46:09.325 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 34, 2024-09-16T03:46:09.326 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 11319, 2024-09-16T03:46:09.326 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:46:09.326 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:46:09.326 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T03:46:09.326 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T03:46:09.326 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:46:09.326 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 6, 2024-09-16T03:46:09.326 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:46:09.326 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:46:09.326 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:46:09.326 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:46:09.326 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:46:09.327 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:46:09.327 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:46:09.327 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:46:09.327 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:46:09.327 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:46:09.327 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:46:09.327 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:46:09.327 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:46:09.327 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:46:09.327 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:46:09.327 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:46:09.327 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:46:09.328 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:46:09.328 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:46:09.328 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:46:09.328 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:46:09.328 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:46:09.328 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:46:09.328 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:46:09.328 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:46:09.328 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:46:09.328 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:46:09.328 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:46:09.329 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:46:09.329 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:46:09.329 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:46:09.329 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:46:09.329 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:46:09.329 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:46:09.329 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:46:09.329 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:46:09.329 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:46:09.329 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:46:09.329 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:46:09.330 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:46:09.330 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:46:09.330 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:46:09.330 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:46:09.330 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:46:09.330 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:46:09.330 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:46:09.330 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:46:09.330 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:46:09.330 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:46:09.330 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:46:09.331 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:46:09.331 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:46:09.331 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:46:09.331 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:46:09.331 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:46:09.331 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:46:09.331 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:46:09.331 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:46:09.331 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:46:09.331 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:46:09.331 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:46:09.332 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:46:09.332 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:46:09.332 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:46:09.332 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:46:09.332 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:46:09.332 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:46:09.332 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:46:09.332 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:46:09.332 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:46:09.332 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:46:09.332 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:46:09.332 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:46:09.333 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:46:09.333 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:46:09.333 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:46:09.333 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:46:09.333 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:46:09.333 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:46:09.333 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:46:09.333 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:46:09.333 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:46:09.333 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:46:09.333 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:46:09.334 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:46:09.334 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:46:09.334 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:46:09.334 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:46:09.334 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:46:09.334 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:46:09.334 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:46:09.334 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:46:09.334 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:46:09.334 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:46:09.334 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:46:09.334 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:46:09.335 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:46:09.335 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:46:09.335 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:46:09.335 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:46:09.335 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:46:09.335 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:46:09.335 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:46:09.335 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:46:09.335 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:46:09.335 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:46:09.335 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:46:09.335 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:46:09.336 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:46:09.336 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:46:09.336 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:46:09.336 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:46:09.336 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:46:09.336 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:46:09.336 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:46:09.336 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:46:09.336 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:46:09.336 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:46:09.336 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:46:09.336 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:46:09.337 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:46:09.337 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:46:09.337 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:46:09.337 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:46:09.337 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:46:09.337 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:46:09.337 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:46:09.337 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:46:09.337 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:46:09.337 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:46:09.337 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:46:09.338 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:46:09.338 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:46:09.338 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:46:09.338 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:46:09.338 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:46:09.338 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:46:09.338 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:46:09.338 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:46:09.338 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:46:09.338 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:46:09.338 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:46:09.339 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:46:09.339 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:46:09.339 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:46:09.339 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 2 2024-09-16T03:46:09.339 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:46:09.339 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:46:09.339 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:46:09.339 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:46:09.339 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:46:09.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.300+0000 7f256edea640 10 librados: watch_flush enter 2024-09-16T03:46:09.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.300+0000 7f256edea640 10 librados: watch_flush exit 2024-09-16T03:46:09.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.300+0000 7f256edea640 20 client.744444.objecter shutdown clearing up homeless session... 2024-09-16T03:46:09.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.300+0000 7f256edea640 10 client.744444.objecter successfully canceled tick 2024-09-16T03:46:09.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.300+0000 7f256edea640 1 -- 172.21.15.42:0/2639089672 >> v2:172.21.15.154:6812/254845342 conn(0x7f254c037810 msgr2=0x7f254c039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:46:09.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.300+0000 7f256edea640 1 --2- 172.21.15.42:0/2639089672 >> v2:172.21.15.154:6812/254845342 conn(0x7f254c037810 0x7f254c039cd0 secure :-1 s=READY pgs=3406 cs=0 l=1 rev1=1 crypto rx=0x7f255401c3a0 tx=0x7f2554008040 comp rx=0 tx=0).stop 2024-09-16T03:46:09.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.300+0000 7f256edea640 10 monclient: shutdown 2024-09-16T03:46:09.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.300+0000 7f256edea640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:46:09.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.300+0000 7f256edea640 1 -- 172.21.15.42:0/2639089672 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f256809fb00 msgr2=0x7f2568120710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:46:09.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.300+0000 7f256edea640 1 --2- 172.21.15.42:0/2639089672 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f256809fb00 0x7f2568120710 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f256000c700 tx=0x7f256000cbc0 comp rx=0 tx=0).stop 2024-09-16T03:46:09.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.300+0000 7f256d387640 1 -- 172.21.15.42:0/2639089672 reap_dead start 2024-09-16T03:46:09.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.300+0000 7f256d387640 1 -- 172.21.15.42:0/2639089672 reap_dead start 2024-09-16T03:46:09.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.300+0000 7f256edea640 1 -- 172.21.15.42:0/2639089672 shutdown_connections 2024-09-16T03:46:09.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.300+0000 7f256edea640 1 -- 172.21.15.42:0/2639089672 >> 172.21.15.42:0/2639089672 conn(0x7f25680a2420 msgr2=0x7f256813c590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:46:09.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.300+0000 7f256edea640 1 -- 172.21.15.42:0/2639089672 shutdown_connections 2024-09-16T03:46:09.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.300+0000 7f256edea640 1 -- 172.21.15.42:0/2639089672 wait complete. 2024-09-16T03:46:09.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.300+0000 7f256edea640 1 librados: shutdown 2024-09-16T03:46:09.343 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:46:09.343 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-09-16T03:46:09.343 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-16T03:46:09.343 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-09-16T03:46:09.343 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-16T03:46:09.344 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-16T03:46:09.344 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T03:46:09.356 INFO:tasks.ceph.mon.a:Stopped 2024-09-16T03:46:09.356 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T03:46:09.356 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:46:09.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.512+0000 7ff5b9a2f640 10 monclient: get_monmap_and_config 2024-09-16T03:46:09.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.512+0000 7ff5b9a2f640 10 monclient: build_initial_monmap 2024-09-16T03:46:09.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.512+0000 7ff5b9a2f640 10 monclient: monmap: 2024-09-16T03:46:09.518 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:46:09.518 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:46:09.518 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:46:09.517540+0000 2024-09-16T03:46:09.518 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:46:09.517540+0000 2024-09-16T03:46:09.519 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:46:09.519 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:46:09.519 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:46:09.519 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:46:09.519 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:46:09.519 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:46:09.519 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:46:09.519 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:46:09.521 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.512+0000 7ff5b9a2f640 1 Processor -- start 2024-09-16T03:46:09.521 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.512+0000 7ff5b9a2f640 1 -- start start 2024-09-16T03:46:09.521 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.512+0000 7ff5b9a2f640 10 monclient: init 2024-09-16T03:46:09.521 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.512+0000 7ff5b9a2f640 10 monclient: _reopen_session rank -1 2024-09-16T03:46:09.521 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.512+0000 7ff5b9a2f640 10 monclient: _add_conns ranks=[0,2,1,4,3] 2024-09-16T03:46:09.521 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.512+0000 7ff5b9a2f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff5b40a1330 addr v1:172.21.15.42:6789/0 2024-09-16T03:46:09.521 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.512+0000 7ff5b9a2f640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff5b40a1710 0x7ff5b40a1b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:46:09.521 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.512+0000 7ff5b9a2f640 10 monclient(hunting): picked mon.noname-b con 0x7ff5b40a1710 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:46:09.522 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.512+0000 7ff5b9a2f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff5b41393a0 addr v1:172.21.15.154:6789/0 2024-09-16T03:46:09.522 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.512+0000 7ff5b9a2f640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff5b409e4c0 con 0x7ff5b40a1330 2024-09-16T03:46:09.522 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.512+0000 7ff5b2ffd640 1 -- >> v1:172.21.15.42:6789/0 conn(0x7ff5b40a1330 legacy=0x7ff5b409f180 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.42:6789/0 2024-09-16T03:46:09.522 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.512+0000 7ff5b9a2f640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff5b40a2050 con 0x7ff5b41393a0 2024-09-16T03:46:09.522 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.512+0000 7ff5b9a2f640 10 monclient(hunting): start opening mon connection 2024-09-16T03:46:09.522 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.512+0000 7ff5b9a2f640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7ff5b409edf0 con 0x7ff5b40a1710 2024-09-16T03:46:09.522 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.512+0000 7ff5b9a2f640 10 monclient(hunting): _renew_subs 2024-09-16T03:46:09.522 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.512+0000 7ff5b9a2f640 10 monclient(hunting): authenticate will time out at 1216.487124s 2024-09-16T03:46:09.522 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.512+0000 7ff5b27fc640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff5b40a1710 0x7ff5b40a1b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:46:09.522 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.512+0000 7ff5b27fc640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff5b40a1710 0x7ff5b40a1b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:51814/0 (socket says 172.21.15.42:51814) 2024-09-16T03:46:09.522 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.512+0000 7ff5b27fc640 1 -- 172.21.15.42:0/2074573688 learned_addr learned my addr 172.21.15.42:0/2074573688 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:46:09.522 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.512+0000 7ff5b27fc640 10 monclient(hunting): get_auth_request con 0x7ff5b40a1710 auth_method 0 2024-09-16T03:46:09.522 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.512+0000 7ff5b27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:46:09.522 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.512+0000 7ff5b27fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:46:09.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.512+0000 7ff5b27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:46:09.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.512+0000 7ff5b27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:46:09.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.512+0000 7ff5b27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:46:09.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.512+0000 7ff5b27fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:46:09.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.512+0000 7ff5b27fc640 10 monclient(hunting): handle_auth_done global_id 13580 payload 274 2024-09-16T03:46:09.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.512+0000 7ff5b27fc640 1 -- 172.21.15.42:0/2074573688 >> v1:172.21.15.154:6789/0 conn(0x7ff5b41393a0 legacy=0x7ff5b413b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:46:09.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.512+0000 7ff5b27fc640 1 -- 172.21.15.42:0/2074573688 >> v1:172.21.15.42:6789/0 conn(0x7ff5b40a1330 legacy=0x7ff5b409f180 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:46:09.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.512+0000 7ff5b27fc640 10 monclient: _finish_hunting 0 2024-09-16T03:46:09.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.512+0000 7ff5b27fc640 1 monclient: found mon.noname-b 2024-09-16T03:46:09.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.512+0000 7ff5b27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:46:09.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.512+0000 7ff5b27fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T03:46:09.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.512+0000 7ff5b27fc640 1 -- 172.21.15.42:0/2074573688 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5b40a3170 con 0x7ff5b40a1710 2024-09-16T03:46:09.529 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.516+0000 7ff5b27fc640 1 --2- 172.21.15.42:0/2074573688 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff5b40a1710 0x7ff5b40a1b10 secure :-1 s=READY pgs=2432 cs=0 l=1 rev1=1 crypto rx=0x7ff5a80098a0 tx=0x7ff5a801c3c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b477064c12a37c45 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:46:09.529 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.516+0000 7ff5b1ffb640 1 -- 172.21.15.42:0/2074573688 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5a80b0070 con 0x7ff5b40a1710 2024-09-16T03:46:09.529 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.516+0000 7ff5b1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:46:09.529 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.516+0000 7ff5b1ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:46:09.529 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.516+0000 7ff5b1ffb640 10 monclient: dump: 2024-09-16T03:46:09.529 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:46:09.529 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:46:09.529 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:09.529 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:09.530 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:46:09.530 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:46:09.530 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:46:09.530 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:46:09.530 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:46:09.530 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:46:09.530 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.516+0000 7ff5b1ffb640 10 monclient: _finish_auth 0 2024-09-16T03:46:09.530 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.516+0000 7ff5b1ffb640 10 monclient: _check_auth_tickets 2024-09-16T03:46:09.530 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.516+0000 7ff5b1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:46:09.530 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.516+0000 7ff5b1ffb640 1 -- 172.21.15.42:0/2074573688 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff5a80ab600 con 0x7ff5b40a1710 2024-09-16T03:46:09.530 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.516+0000 7ff5b9a2f640 5 monclient: authenticate success, global_id 13580 2024-09-16T03:46:09.531 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.516+0000 7ff5b9a2f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:46:09.531 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.516+0000 7ff5b1ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T03:46:09.531 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.516+0000 7ff5b9a2f640 10 monclient: get_monmap_and_config success 2024-09-16T03:46:09.531 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.516+0000 7ff5b1ffb640 1 -- 172.21.15.42:0/2074573688 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5a80ab9d0 con 0x7ff5b40a1710 2024-09-16T03:46:09.531 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.516+0000 7ff5b1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:46:09.531 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.516+0000 7ff5b9a2f640 4 set_mon_vals no callback set 2024-09-16T03:46:09.531 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.516+0000 7ff5b1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:46:09.531 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.516+0000 7ff5b1ffb640 10 monclient: dump: 2024-09-16T03:46:09.531 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:46:09.531 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:46:09.531 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:09.531 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:09.531 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:46:09.532 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:46:09.532 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:46:09.532 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:46:09.532 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:46:09.532 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:46:09.532 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.516+0000 7ff5b9a2f640 10 monclient: shutdown 2024-09-16T03:46:09.532 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.516+0000 7ff5b9a2f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:46:09.532 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.516+0000 7ff5b9a2f640 1 -- 172.21.15.42:0/2074573688 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff5b40a1710 msgr2=0x7ff5b40a1b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:46:09.532 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.516+0000 7ff5b9a2f640 1 --2- 172.21.15.42:0/2074573688 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff5b40a1710 0x7ff5b40a1b10 secure :-1 s=READY pgs=2432 cs=0 l=1 rev1=1 crypto rx=0x7ff5a80098a0 tx=0x7ff5a801c3c0 comp rx=0 tx=0).stop 2024-09-16T03:46:09.532 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.516+0000 7ff5b9a2f640 1 -- 172.21.15.42:0/2074573688 shutdown_connections 2024-09-16T03:46:09.532 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.516+0000 7ff5b9a2f640 1 --2- 172.21.15.42:0/2074573688 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff5b40a1710 0x7ff5b40a1b10 unknown :-1 s=CLOSED pgs=2432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:46:09.532 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.516+0000 7ff5b9a2f640 1 -- 172.21.15.42:0/2074573688 >> 172.21.15.42:0/2074573688 conn(0x7ff5b40a2420 msgr2=0x7ff5b40a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:46:09.532 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.516+0000 7ff5b9a2f640 1 -- 172.21.15.42:0/2074573688 shutdown_connections 2024-09-16T03:46:09.533 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.516+0000 7ff5b9a2f640 1 -- 172.21.15.42:0/2074573688 wait complete. 2024-09-16T03:46:09.533 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.516+0000 7ff5b9a2f640 10 monclient: build_initial_monmap 2024-09-16T03:46:09.533 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.516+0000 7ff5b9a2f640 10 monclient: monmap: 2024-09-16T03:46:09.533 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:46:09.533 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:46:09.533 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:46:09.533 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:46:09.533 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:46:09.533 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:46:09.533 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:46:09.533 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:46:09.533 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:46:09.533 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:46:09.533 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.516+0000 7ff5b9a2f640 1 librados: starting msgr at 2024-09-16T03:46:09.534 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.516+0000 7ff5b9a2f640 1 librados: starting objecter 2024-09-16T03:46:09.534 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.516+0000 7ff5b9a2f640 1 Processor -- start 2024-09-16T03:46:09.534 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.516+0000 7ff5b9a2f640 1 -- start start 2024-09-16T03:46:09.534 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.516+0000 7ff5b9a2f640 1 librados: setting wanted keys 2024-09-16T03:46:09.534 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.516+0000 7ff5b9a2f640 1 librados: calling monclient init 2024-09-16T03:46:09.534 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.516+0000 7ff5b9a2f640 10 monclient: init 2024-09-16T03:46:09.534 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.520+0000 7ff5b9a2f640 10 monclient: _reopen_session rank -1 2024-09-16T03:46:09.534 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.520+0000 7ff5b9a2f640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:46:09.534 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.520+0000 7ff5b9a2f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff5b40a1330 0x7ff5b4071860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:46:09.534 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.520+0000 7ff5b9a2f640 10 monclient(hunting): picked mon.noname-b con 0x7ff5b40a1330 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:46:09.534 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.520+0000 7ff5b9a2f640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff5b40a1710 0x7ff5b41cd550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:46:09.534 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.520+0000 7ff5b9a2f640 10 monclient(hunting): picked mon.noname-a con 0x7ff5b40a1710 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:46:09.534 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.520+0000 7ff5b9a2f640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff5b41393a0 0x7ff5b411bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:46:09.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.520+0000 7ff5b9a2f640 10 monclient(hunting): picked mon.noname-c con 0x7ff5b41393a0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:46:09.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.520+0000 7ff5b9a2f640 10 monclient(hunting): start opening mon connection 2024-09-16T03:46:09.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.520+0000 7ff5b9a2f640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7ff5b413f8e0 con 0x7ff5b40a1710 2024-09-16T03:46:09.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.520+0000 7ff5b9a2f640 10 monclient(hunting): start opening mon connection 2024-09-16T03:46:09.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.520+0000 7ff5b9a2f640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff5b409cd60 con 0x7ff5b40a1330 2024-09-16T03:46:09.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.520+0000 7ff5b9a2f640 10 monclient(hunting): start opening mon connection 2024-09-16T03:46:09.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.520+0000 7ff5b9a2f640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7ff5b413fa60 con 0x7ff5b41393a0 2024-09-16T03:46:09.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.520+0000 7ff5b9a2f640 10 monclient(hunting): _renew_subs 2024-09-16T03:46:09.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.520+0000 7ff5b9a2f640 10 monclient(hunting): authenticate will time out at 1216.492214s 2024-09-16T03:46:09.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.520+0000 7ff5b27fc640 1 -- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff5b40a1710 msgr2=0x7ff5b41cd550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.42:3300/0 2024-09-16T03:46:09.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.520+0000 7ff5b27fc640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff5b40a1710 0x7ff5b41cd550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:46:09.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.520+0000 7ff5b37fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff5b41393a0 0x7ff5b411bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:46:09.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.520+0000 7ff5b37fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff5b41393a0 0x7ff5b411bf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:51828/0 (socket says 172.21.15.42:51828) 2024-09-16T03:46:09.536 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.520+0000 7ff5b37fe640 1 -- 172.21.15.42:0/869615788 learned_addr learned my addr 172.21.15.42:0/869615788 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:46:09.536 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.520+0000 7ff5b37fe640 10 monclient(hunting): get_auth_request con 0x7ff5b41393a0 auth_method 0 2024-09-16T03:46:09.536 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.520+0000 7ff5b37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:46:09.536 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.520+0000 7ff5b37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:46:09.536 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.520+0000 7ff5b37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:46:09.536 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.520+0000 7ff5b37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:46:09.536 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.520+0000 7ff5b37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:46:09.536 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.520+0000 7ff5b37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:46:09.536 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.520+0000 7ff5b2ffd640 1 --2- 172.21.15.42:0/869615788 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff5b40a1330 0x7ff5b4071860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:46:09.536 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.520+0000 7ff5b37fe640 10 monclient(hunting): handle_auth_done global_id 13583 payload 824 2024-09-16T03:46:09.536 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.520+0000 7ff5b37fe640 1 -- 172.21.15.42:0/869615788 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff5b40a1330 msgr2=0x7ff5b4071860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:46:09.536 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.520+0000 7ff5b37fe640 1 --2- 172.21.15.42:0/869615788 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff5b40a1330 0x7ff5b4071860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:46:09.536 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.520+0000 7ff5b37fe640 1 -- 172.21.15.42:0/869615788 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff5b40a1710 msgr2=0x7ff5b41cd550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:46:09.537 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.520+0000 7ff5b37fe640 1 --2- 172.21.15.42:0/869615788 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff5b40a1710 0x7ff5b41cd550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:46:09.537 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.520+0000 7ff5b37fe640 10 monclient: _finish_hunting 0 2024-09-16T03:46:09.537 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.520+0000 7ff5b37fe640 1 monclient: found mon.noname-c 2024-09-16T03:46:09.537 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.520+0000 7ff5b37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:46:09.537 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.520+0000 7ff5b37fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:46:09.537 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.520+0000 7ff5b37fe640 1 -- 172.21.15.42:0/869615788 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5b413caa0 con 0x7ff5b41393a0 2024-09-16T03:46:09.537 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.520+0000 7ff5b2ffd640 10 monclient: get_auth_request con 0x7ff5b40a1330 auth_method 0 2024-09-16T03:46:09.537 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.520+0000 7ff5b2ffd640 1 --2- 172.21.15.42:0/869615788 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff5b40a1330 0x7ff5b4071860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T03:46:09.537 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.520+0000 7ff5b37fe640 1 --2- 172.21.15.42:0/869615788 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff5b41393a0 0x7ff5b411bf70 secure :-1 s=READY pgs=2433 cs=0 l=1 rev1=1 crypto rx=0x7ff5ac00c750 tx=0x7ff5ac00cc10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:46:09.537 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.520+0000 7ff59bfff640 10 client.?.objecter ms_handle_connect 0x7ff5b41393a0 2024-09-16T03:46:09.537 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.520+0000 7ff59bfff640 10 client.?.objecter resend_mon_ops 2024-09-16T03:46:09.537 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.520+0000 7ff59bfff640 1 -- 172.21.15.42:0/869615788 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5ac017070 con 0x7ff5b41393a0 2024-09-16T03:46:09.537 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.520+0000 7ff59bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:46:09.538 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.520+0000 7ff59bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:46:09.538 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.520+0000 7ff59bfff640 10 monclient: dump: 2024-09-16T03:46:09.538 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:46:09.538 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:46:09.538 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:09.538 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:09.538 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:46:09.538 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:46:09.538 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:46:09.538 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:46:09.538 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:46:09.538 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:46:09.538 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.520+0000 7ff59bfff640 10 monclient: _finish_auth 0 2024-09-16T03:46:09.538 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.520+0000 7ff59bfff640 10 monclient: _check_auth_tickets 2024-09-16T03:46:09.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.520+0000 7ff59bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:46:09.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.520+0000 7ff59bfff640 1 -- 172.21.15.42:0/869615788 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff5ac003b90 con 0x7ff5b41393a0 2024-09-16T03:46:09.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.520+0000 7ff59bfff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:46:09.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.520+0000 7ff59bfff640 1 -- 172.21.15.42:0/869615788 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5ac005070 con 0x7ff5b41393a0 2024-09-16T03:46:09.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.520+0000 7ff59bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:46:09.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.520+0000 7ff59bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:46:09.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.520+0000 7ff59bfff640 10 monclient: dump: 2024-09-16T03:46:09.539 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:46:09.539 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:46:09.539 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:09.539 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:09.539 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:46:09.539 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:46:09.539 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:46:09.540 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:46:09.540 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:46:09.540 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:46:09.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.520+0000 7ff5b9a2f640 5 monclient: authenticate success, global_id 13583 2024-09-16T03:46:09.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.520+0000 7ff5b9a2f640 10 monclient: _renew_subs 2024-09-16T03:46:09.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.520+0000 7ff5b0ff9640 4 set_mon_vals no callback set 2024-09-16T03:46:09.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.520+0000 7ff5b9a2f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:46:09.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.520+0000 7ff5b9a2f640 1 -- 172.21.15.42:0/869615788 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5b411ee10 con 0x7ff5b41393a0 2024-09-16T03:46:09.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.520+0000 7ff5b9a2f640 10 client.13583.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:46:09.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.520+0000 7ff5b9a2f640 10 monclient: _renew_subs 2024-09-16T03:46:09.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.520+0000 7ff5b9a2f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:46:09.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.520+0000 7ff5b9a2f640 1 -- 172.21.15.42:0/869615788 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5b411e570 con 0x7ff5b41393a0 2024-09-16T03:46:09.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.520+0000 7ff5b9a2f640 1 librados: init done 2024-09-16T03:46:09.541 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.520+0000 7ff5b9a2f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:46:09.541 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.520+0000 7ff59bfff640 1 -- 172.21.15.42:0/869615788 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7ff5ac005210 con 0x7ff5b41393a0 2024-09-16T03:46:09.541 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.520+0000 7ff5b9a2f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:46:09.541 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.520+0000 7ff5b9a2f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:46:09.541 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.520+0000 7ff5b9a2f640 1 -- 172.21.15.42:0/869615788 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff5b40008e0 con 0x7ff5b41393a0 2024-09-16T03:46:09.541 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.520+0000 7ff59bfff640 1 --2- 172.21.15.42:0/869615788 >> v2:172.21.15.154:6812/254845342 conn(0x7ff5a00377c0 0x7ff5a0039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:46:09.541 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.520+0000 7ff59bfff640 1 -- 172.21.15.42:0/869615788 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(123..123 src has 1..123) ==== 6278+0+0 (secure 0 0 0) 0x7ff5ac048600 con 0x7ff5b41393a0 2024-09-16T03:46:09.541 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.520+0000 7ff59bfff640 10 client.13583.objecter ms_dispatch 0x7ff5b4000ba0 osd_map(123..123 src has 1..123) 2024-09-16T03:46:09.541 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.520+0000 7ff59bfff640 3 client.13583.objecter handle_osd_map got epochs [123,123] > 0 2024-09-16T03:46:09.541 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.520+0000 7ff59bfff640 3 client.13583.objecter handle_osd_map decoding full epoch 123 2024-09-16T03:46:09.541 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.520+0000 7ff59bfff640 20 client.13583.objecter dump_active .. 0 homeless 2024-09-16T03:46:09.541 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.520+0000 7ff5b2ffd640 1 --2- 172.21.15.42:0/869615788 >> v2:172.21.15.154:6812/254845342 conn(0x7ff5a00377c0 0x7ff5a0039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:46:09.541 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.520+0000 7ff5b2ffd640 10 monclient: get_auth_request con 0x7ff5a00377c0 auth_method 0 2024-09-16T03:46:09.542 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.520+0000 7ff59bfff640 10 client.13583.objecter ms_handle_connect 0x7ff5a00377c0 2024-09-16T03:46:09.542 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.520+0000 7ff5b2ffd640 1 --2- 172.21.15.42:0/869615788 >> v2:172.21.15.154:6812/254845342 conn(0x7ff5a00377c0 0x7ff5a0039c80 secure :-1 s=READY pgs=3407 cs=0 l=1 rev1=1 crypto rx=0x7ff5a4003990 tx=0x7ff5a4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:46:09.542 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.524+0000 7ff59bfff640 1 -- 172.21.15.42:0/869615788 <== mon.2 v2:172.21.15.42:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7ff5b413caa0 con 0x7ff5b41393a0 2024-09-16T03:46:09.542 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.524+0000 7ff59bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:46:09.542 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.524+0000 7ff59bfff640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:46:09.761 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.752+0000 7ff5b9a2f640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:46:09.761 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.756+0000 7ff5b9a2f640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:46:09.761 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.756+0000 7ff5b9a2f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:46:09.761 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.756+0000 7ff5b9a2f640 1 -- 172.21.15.42:0/869615788 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7ff5b4134ac0 con 0x7ff5b41393a0 2024-09-16T03:46:09.761 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.756+0000 7ff59bfff640 1 -- 172.21.15.42:0/869615788 <== mon.2 v2:172.21.15.42:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7ff5ac019310 con 0x7ff5b41393a0 2024-09-16T03:46:09.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.756+0000 7ff59bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:46:09.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.756+0000 7ff59bfff640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:46:09.762 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":34,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:46:09.764 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.756+0000 7ff5b9a2f640 10 librados: watch_flush enter 2024-09-16T03:46:09.764 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.756+0000 7ff5b9a2f640 10 librados: watch_flush exit 2024-09-16T03:46:09.764 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.756+0000 7ff5b9a2f640 20 client.13583.objecter shutdown clearing up homeless session... 2024-09-16T03:46:09.765 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.756+0000 7ff5b9a2f640 10 client.13583.objecter successfully canceled tick 2024-09-16T03:46:09.765 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.756+0000 7ff5b9a2f640 1 -- 172.21.15.42:0/869615788 >> v2:172.21.15.154:6812/254845342 conn(0x7ff5a00377c0 msgr2=0x7ff5a0039c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:46:09.765 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.756+0000 7ff5b9a2f640 1 --2- 172.21.15.42:0/869615788 >> v2:172.21.15.154:6812/254845342 conn(0x7ff5a00377c0 0x7ff5a0039c80 secure :-1 s=READY pgs=3407 cs=0 l=1 rev1=1 crypto rx=0x7ff5a4003990 tx=0x7ff5a4008040 comp rx=0 tx=0).stop 2024-09-16T03:46:09.765 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.756+0000 7ff5b9a2f640 10 monclient: shutdown 2024-09-16T03:46:09.765 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.756+0000 7ff5b9a2f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:46:09.765 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.756+0000 7ff5b9a2f640 1 -- 172.21.15.42:0/869615788 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff5b41393a0 msgr2=0x7ff5b411bf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:46:09.765 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.756+0000 7ff5b9a2f640 1 --2- 172.21.15.42:0/869615788 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff5b41393a0 0x7ff5b411bf70 secure :-1 s=READY pgs=2433 cs=0 l=1 rev1=1 crypto rx=0x7ff5ac00c750 tx=0x7ff5ac00cc10 comp rx=0 tx=0).stop 2024-09-16T03:46:09.765 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.756+0000 7ff5b9a2f640 1 -- 172.21.15.42:0/869615788 shutdown_connections 2024-09-16T03:46:09.765 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.756+0000 7ff5b9a2f640 1 --2- 172.21.15.42:0/869615788 >> v2:172.21.15.154:6812/254845342 conn(0x7ff5a00377c0 0x7ff5a0039c80 unknown :-1 s=CLOSED pgs=3407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:46:09.766 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.756+0000 7ff5b9a2f640 1 --2- 172.21.15.42:0/869615788 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff5b41393a0 0x7ff5b411bf70 unknown :-1 s=CLOSED pgs=2433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:46:09.766 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.756+0000 7ff5b9a2f640 1 --2- 172.21.15.42:0/869615788 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff5b40a1710 0x7ff5b41cd550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:46:09.766 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.756+0000 7ff5b9a2f640 1 --2- 172.21.15.42:0/869615788 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff5b40a1330 0x7ff5b4071860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:46:09.766 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.756+0000 7ff5b9a2f640 1 -- 172.21.15.42:0/869615788 >> 172.21.15.42:0/869615788 conn(0x7ff5b40a2420 msgr2=0x7ff5b409ba50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:46:09.766 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.756+0000 7ff5b9a2f640 1 -- 172.21.15.42:0/869615788 shutdown_connections 2024-09-16T03:46:09.766 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.756+0000 7ff5b9a2f640 1 -- 172.21.15.42:0/869615788 wait complete. 2024-09-16T03:46:09.766 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:09.756+0000 7ff5b9a2f640 1 librados: shutdown 2024-09-16T03:46:12.783 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:46:12.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.908+0000 7fe27b4f9640 10 monclient: get_monmap_and_config 2024-09-16T03:46:12.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.908+0000 7fe27b4f9640 10 monclient: build_initial_monmap 2024-09-16T03:46:12.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.908+0000 7fe27b4f9640 10 monclient: monmap: 2024-09-16T03:46:12.915 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:46:12.915 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:46:12.916 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:46:12.913938+0000 2024-09-16T03:46:12.916 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:46:12.913938+0000 2024-09-16T03:46:12.916 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:46:12.916 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:46:12.916 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:46:12.917 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:46:12.917 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:46:12.917 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:46:12.917 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:46:12.917 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:46:12.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.908+0000 7fe27b4f9640 1 Processor -- start 2024-09-16T03:46:12.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.908+0000 7fe27b4f9640 1 -- start start 2024-09-16T03:46:12.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.908+0000 7fe27b4f9640 10 monclient: init 2024-09-16T03:46:12.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.908+0000 7fe27b4f9640 10 monclient: _reopen_session rank -1 2024-09-16T03:46:12.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.908+0000 7fe27b4f9640 10 monclient: _add_conns ranks=[2,3,0,1,4] 2024-09-16T03:46:12.938 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.908+0000 7fe27b4f9640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe2740a1330 0x7fe27409f180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:46:12.938 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.908+0000 7fe27b4f9640 10 monclient(hunting): picked mon.noname-b con 0x7fe2740a1330 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:46:12.938 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.908+0000 7fe27b4f9640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe2740a1710 0x7fe2740a1b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:46:12.939 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.908+0000 7fe27b4f9640 10 monclient(hunting): picked mon.noname-a con 0x7fe2740a1710 addr v2:172.21.15.42:3300/0 2024-09-16T03:46:12.939 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.908+0000 7fe27b4f9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe2741340f0 addr v1:172.21.15.42:6789/0 2024-09-16T03:46:12.939 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.908+0000 7fe27b4f9640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe27409e4c0 con 0x7fe2741340f0 2024-09-16T03:46:12.939 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.908+0000 7fe27b4f9640 10 monclient(hunting): start opening mon connection 2024-09-16T03:46:12.939 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.908+0000 7fe27b4f9640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7fe27409edf0 con 0x7fe2740a1710 2024-09-16T03:46:12.939 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.908+0000 7fe27b4f9640 10 monclient(hunting): start opening mon connection 2024-09-16T03:46:12.940 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.908+0000 7fe27b4f9640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fe27409cd60 con 0x7fe2740a1330 2024-09-16T03:46:12.941 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.908+0000 7fe27b4f9640 10 monclient(hunting): _renew_subs 2024-09-16T03:46:12.941 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.908+0000 7fe27b4f9640 10 monclient(hunting): authenticate will time out at 1219.883486s 2024-09-16T03:46:12.941 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.908+0000 7fe278a94640 1 -- >> v2:172.21.15.42:3300/0 conn(0x7fe2740a1710 msgr2=0x7fe2740a1b10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3300/0 2024-09-16T03:46:12.941 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.908+0000 7fe279a96640 1 -- >> v1:172.21.15.42:6789/0 conn(0x7fe2741340f0 legacy=0x7fe2741345a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.42:6789/0 2024-09-16T03:46:12.941 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.908+0000 7fe278a94640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe2740a1710 0x7fe2740a1b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:46:12.942 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.908+0000 7fe279295640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe2740a1330 0x7fe27409f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:46:12.942 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.908+0000 7fe279295640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe2740a1330 0x7fe27409f180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:51836/0 (socket says 172.21.15.42:51836) 2024-09-16T03:46:12.942 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.908+0000 7fe279295640 1 -- 172.21.15.42:0/433477521 learned_addr learned my addr 172.21.15.42:0/433477521 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:46:12.943 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.908+0000 7fe279295640 10 monclient(hunting): get_auth_request con 0x7fe2740a1330 auth_method 0 2024-09-16T03:46:12.943 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.908+0000 7fe279295640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:46:12.943 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.908+0000 7fe279295640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:46:12.943 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.908+0000 7fe279295640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:46:12.943 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.908+0000 7fe279295640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:46:12.943 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.908+0000 7fe279295640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:46:12.943 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.908+0000 7fe279295640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:46:12.943 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.912+0000 7fe279295640 10 monclient(hunting): handle_auth_done global_id 13586 payload 274 2024-09-16T03:46:12.943 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.912+0000 7fe279295640 1 -- 172.21.15.42:0/433477521 >> v2:172.21.15.42:3300/0 conn(0x7fe2740a1710 msgr2=0x7fe2740a1b10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:46:12.943 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.912+0000 7fe279295640 1 --2- 172.21.15.42:0/433477521 >> v2:172.21.15.42:3300/0 conn(0x7fe2740a1710 0x7fe2740a1b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:46:12.943 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.912+0000 7fe279295640 1 -- 172.21.15.42:0/433477521 >> v1:172.21.15.42:6789/0 conn(0x7fe2741340f0 legacy=0x7fe2741345a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:46:12.944 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.912+0000 7fe279295640 10 monclient: _finish_hunting 0 2024-09-16T03:46:12.944 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.912+0000 7fe279295640 1 monclient: found mon.noname-b 2024-09-16T03:46:12.944 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.912+0000 7fe279295640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:46:12.945 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.912+0000 7fe279295640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T03:46:12.945 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.912+0000 7fe279295640 1 -- 172.21.15.42:0/433477521 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2740a2050 con 0x7fe2740a1330 2024-09-16T03:46:12.945 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.912+0000 7fe279295640 1 --2- 172.21.15.42:0/433477521 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe2740a1330 0x7fe27409f180 secure :-1 s=READY pgs=2438 cs=0 l=1 rev1=1 crypto rx=0x7fe2640098a0 tx=0x7fe26401c3e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e46d9f2a20263c96 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:46:12.945 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.912+0000 7fe273fff640 1 -- 172.21.15.42:0/433477521 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2640b0070 con 0x7fe2740a1330 2024-09-16T03:46:12.945 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.912+0000 7fe273fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:46:12.945 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.912+0000 7fe273fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:46:12.945 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.912+0000 7fe273fff640 10 monclient: dump: 2024-09-16T03:46:12.945 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:46:12.945 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:46:12.945 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:12.946 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:12.946 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:46:12.946 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:46:12.946 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:46:12.946 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:46:12.946 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:46:12.946 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:46:12.946 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.912+0000 7fe273fff640 10 monclient: _finish_auth 0 2024-09-16T03:46:12.946 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.912+0000 7fe273fff640 10 monclient: _check_auth_tickets 2024-09-16T03:46:12.946 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.912+0000 7fe273fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:46:12.946 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.912+0000 7fe273fff640 1 -- 172.21.15.42:0/433477521 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe2640ab600 con 0x7fe2740a1330 2024-09-16T03:46:12.947 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.912+0000 7fe27b4f9640 5 monclient: authenticate success, global_id 13586 2024-09-16T03:46:12.947 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.912+0000 7fe27b4f9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:46:12.947 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.912+0000 7fe273fff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:46:12.947 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.912+0000 7fe273fff640 1 -- 172.21.15.42:0/433477521 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2640ab9d0 con 0x7fe2740a1330 2024-09-16T03:46:12.947 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.912+0000 7fe27b4f9640 10 monclient: get_monmap_and_config success 2024-09-16T03:46:12.947 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.912+0000 7fe27b4f9640 4 set_mon_vals no callback set 2024-09-16T03:46:12.947 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.912+0000 7fe273fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:46:12.947 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.912+0000 7fe273fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:46:12.947 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.912+0000 7fe273fff640 10 monclient: dump: 2024-09-16T03:46:12.947 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:46:12.947 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:46:12.948 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:12.948 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:12.948 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:46:12.948 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:46:12.948 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:46:12.948 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:46:12.948 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:46:12.948 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:46:12.948 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.912+0000 7fe27b4f9640 10 monclient: shutdown 2024-09-16T03:46:12.948 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.912+0000 7fe27b4f9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:46:12.948 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.912+0000 7fe27b4f9640 1 -- 172.21.15.42:0/433477521 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe2740a1330 msgr2=0x7fe27409f180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:46:12.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.912+0000 7fe27b4f9640 1 --2- 172.21.15.42:0/433477521 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe2740a1330 0x7fe27409f180 secure :-1 s=READY pgs=2438 cs=0 l=1 rev1=1 crypto rx=0x7fe2640098a0 tx=0x7fe26401c3e0 comp rx=0 tx=0).stop 2024-09-16T03:46:12.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.912+0000 7fe27b4f9640 1 -- 172.21.15.42:0/433477521 shutdown_connections 2024-09-16T03:46:12.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.912+0000 7fe27b4f9640 1 --2- 172.21.15.42:0/433477521 >> v2:172.21.15.42:3300/0 conn(0x7fe2740a1710 0x7fe2740a1b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:46:12.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.912+0000 7fe27b4f9640 1 --2- 172.21.15.42:0/433477521 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe2740a1330 0x7fe27409f180 unknown :-1 s=CLOSED pgs=2438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:46:12.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.912+0000 7fe27b4f9640 1 -- 172.21.15.42:0/433477521 >> 172.21.15.42:0/433477521 conn(0x7fe2740a2420 msgr2=0x7fe2740a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:46:12.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.912+0000 7fe27b4f9640 1 -- 172.21.15.42:0/433477521 shutdown_connections 2024-09-16T03:46:12.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.912+0000 7fe27b4f9640 1 -- 172.21.15.42:0/433477521 wait complete. 2024-09-16T03:46:12.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.912+0000 7fe27b4f9640 10 monclient: build_initial_monmap 2024-09-16T03:46:12.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.912+0000 7fe27b4f9640 10 monclient: monmap: 2024-09-16T03:46:12.949 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:46:12.949 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:46:12.950 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:46:12.950 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:46:12.950 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:46:12.950 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:46:12.950 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:46:12.950 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:46:12.950 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:46:12.950 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:46:12.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.912+0000 7fe27b4f9640 1 librados: starting msgr at 2024-09-16T03:46:12.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.912+0000 7fe27b4f9640 1 librados: starting objecter 2024-09-16T03:46:12.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.916+0000 7fe27b4f9640 1 Processor -- start 2024-09-16T03:46:12.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.916+0000 7fe27b4f9640 1 -- start start 2024-09-16T03:46:12.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.916+0000 7fe27b4f9640 1 librados: setting wanted keys 2024-09-16T03:46:12.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.916+0000 7fe27b4f9640 1 librados: calling monclient init 2024-09-16T03:46:12.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.916+0000 7fe27b4f9640 10 monclient: init 2024-09-16T03:46:12.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.916+0000 7fe27b4f9640 10 monclient: _reopen_session rank -1 2024-09-16T03:46:12.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.916+0000 7fe27b4f9640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T03:46:12.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.916+0000 7fe27b4f9640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe2740a1330 0x7fe27413a470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:46:12.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.916+0000 7fe27b4f9640 10 monclient(hunting): picked mon.noname-b con 0x7fe2740a1330 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:46:12.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.916+0000 7fe27b4f9640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe2740a1710 0x7fe2741cfd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:46:12.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.916+0000 7fe27b4f9640 10 monclient(hunting): picked mon.noname-c con 0x7fe2740a1710 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:46:12.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.916+0000 7fe27b4f9640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe2741340f0 0x7fe2741cc040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:46:12.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.916+0000 7fe27b4f9640 10 monclient(hunting): picked mon.noname-a con 0x7fe2741340f0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:46:12.952 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.916+0000 7fe27b4f9640 10 monclient(hunting): start opening mon connection 2024-09-16T03:46:12.952 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.916+0000 7fe27b4f9640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fe27413ed60 con 0x7fe2741340f0 2024-09-16T03:46:12.952 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.916+0000 7fe27b4f9640 10 monclient(hunting): start opening mon connection 2024-09-16T03:46:12.952 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.916+0000 7fe27b4f9640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe27413ebe0 con 0x7fe2740a1330 2024-09-16T03:46:12.952 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.916+0000 7fe27b4f9640 10 monclient(hunting): start opening mon connection 2024-09-16T03:46:12.952 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.916+0000 7fe27b4f9640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fe27413eee0 con 0x7fe2740a1710 2024-09-16T03:46:12.952 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.916+0000 7fe27b4f9640 10 monclient(hunting): _renew_subs 2024-09-16T03:46:12.952 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.916+0000 7fe27b4f9640 10 monclient(hunting): authenticate will time out at 1219.888600s 2024-09-16T03:46:12.952 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.916+0000 7fe279a96640 1 -- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe2741340f0 msgr2=0x7fe2741cc040 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.42:3300/0 2024-09-16T03:46:12.952 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.916+0000 7fe279a96640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe2741340f0 0x7fe2741cc040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:46:12.952 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.916+0000 7fe278a94640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe2740a1710 0x7fe2741cfd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:46:12.953 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.916+0000 7fe278a94640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe2740a1710 0x7fe2741cfd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:51846/0 (socket says 172.21.15.42:51846) 2024-09-16T03:46:12.953 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.916+0000 7fe278a94640 1 -- 172.21.15.42:0/2499464722 learned_addr learned my addr 172.21.15.42:0/2499464722 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:46:12.953 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.916+0000 7fe278a94640 10 monclient(hunting): get_auth_request con 0x7fe2740a1710 auth_method 0 2024-09-16T03:46:12.953 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.916+0000 7fe278a94640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:46:12.953 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.916+0000 7fe278a94640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:46:12.953 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.916+0000 7fe278a94640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:46:12.953 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.916+0000 7fe278a94640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:46:12.953 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.916+0000 7fe278a94640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:46:12.953 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.916+0000 7fe278a94640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:46:12.953 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.916+0000 7fe279295640 1 --2- 172.21.15.42:0/2499464722 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe2740a1330 0x7fe27413a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:46:12.953 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.916+0000 7fe278a94640 10 monclient(hunting): handle_auth_done global_id 13589 payload 824 2024-09-16T03:46:12.954 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.916+0000 7fe278a94640 1 -- 172.21.15.42:0/2499464722 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe2740a1330 msgr2=0x7fe27413a470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:46:12.954 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.916+0000 7fe278a94640 1 --2- 172.21.15.42:0/2499464722 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe2740a1330 0x7fe27413a470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:46:12.954 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.916+0000 7fe278a94640 1 -- 172.21.15.42:0/2499464722 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe2741340f0 msgr2=0x7fe2741cc040 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:46:12.954 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.916+0000 7fe278a94640 1 --2- 172.21.15.42:0/2499464722 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe2741340f0 0x7fe2741cc040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:46:12.954 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.916+0000 7fe278a94640 10 monclient: _finish_hunting 0 2024-09-16T03:46:12.954 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.916+0000 7fe278a94640 1 monclient: found mon.noname-c 2024-09-16T03:46:12.954 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.916+0000 7fe278a94640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:46:12.954 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.916+0000 7fe278a94640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:46:12.954 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.916+0000 7fe278a94640 1 -- 172.21.15.42:0/2499464722 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2740721c0 con 0x7fe2740a1710 2024-09-16T03:46:12.954 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.916+0000 7fe278a94640 1 --2- 172.21.15.42:0/2499464722 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe2740a1710 0x7fe2741cfd40 secure :-1 s=READY pgs=2439 cs=0 l=1 rev1=1 crypto rx=0x7fe26800b9e0 tx=0x7fe26800bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:46:12.954 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.916+0000 7fe271ffb640 10 client.?.objecter ms_handle_connect 0x7fe2740a1710 2024-09-16T03:46:12.954 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.916+0000 7fe271ffb640 10 client.?.objecter resend_mon_ops 2024-09-16T03:46:12.955 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.916+0000 7fe271ffb640 1 -- 172.21.15.42:0/2499464722 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe26800cba0 con 0x7fe2740a1710 2024-09-16T03:46:12.955 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.916+0000 7fe271ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:46:12.955 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.916+0000 7fe271ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:46:12.955 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.916+0000 7fe271ffb640 10 monclient: dump: 2024-09-16T03:46:12.955 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:46:12.955 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:46:12.955 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:12.955 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:12.955 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:46:12.955 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:46:12.955 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:46:12.956 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:46:12.956 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:46:12.956 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:46:12.956 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.916+0000 7fe271ffb640 10 monclient: _finish_auth 0 2024-09-16T03:46:12.956 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.916+0000 7fe271ffb640 10 monclient: _check_auth_tickets 2024-09-16T03:46:12.956 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.916+0000 7fe271ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:46:12.956 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.916+0000 7fe271ffb640 1 -- 172.21.15.42:0/2499464722 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe268010070 con 0x7fe2740a1710 2024-09-16T03:46:12.956 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.916+0000 7fe27b4f9640 5 monclient: authenticate success, global_id 13589 2024-09-16T03:46:12.956 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.916+0000 7fe27b4f9640 10 monclient: _renew_subs 2024-09-16T03:46:12.956 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.916+0000 7fe27b4f9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:46:12.957 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.916+0000 7fe27b4f9640 1 -- 172.21.15.42:0/2499464722 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe274121190 con 0x7fe2740a1710 2024-09-16T03:46:12.957 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.916+0000 7fe271ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T03:46:12.957 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.916+0000 7fe271ffb640 1 -- 172.21.15.42:0/2499464722 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe268015470 con 0x7fe2740a1710 2024-09-16T03:46:12.957 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.916+0000 7fe271ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:46:12.957 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.916+0000 7fe271ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:46:12.957 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.916+0000 7fe271ffb640 10 monclient: dump: 2024-09-16T03:46:12.957 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:46:12.957 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:46:12.957 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:12.957 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:12.957 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:46:12.957 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:46:12.958 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:46:12.958 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:46:12.958 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:46:12.958 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:46:12.958 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.916+0000 7fe27b4f9640 10 client.13589.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:46:12.958 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.916+0000 7fe27b4f9640 10 monclient: _renew_subs 2024-09-16T03:46:12.958 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.916+0000 7fe2737fe640 4 set_mon_vals no callback set 2024-09-16T03:46:12.958 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.916+0000 7fe27b4f9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:46:12.958 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.916+0000 7fe27b4f9640 1 -- 172.21.15.42:0/2499464722 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fe27406ef10 con 0x7fe2740a1710 2024-09-16T03:46:12.958 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.916+0000 7fe27b4f9640 1 librados: init done 2024-09-16T03:46:12.958 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.916+0000 7fe27b4f9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:46:12.959 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.916+0000 7fe27b4f9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:46:12.959 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.916+0000 7fe27b4f9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:46:12.959 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.916+0000 7fe271ffb640 1 -- 172.21.15.42:0/2499464722 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fe268003d10 con 0x7fe2740a1710 2024-09-16T03:46:12.959 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.916+0000 7fe27b4f9640 1 -- 172.21.15.42:0/2499464722 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe27409f180 con 0x7fe2740a1710 2024-09-16T03:46:12.959 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.916+0000 7fe271ffb640 1 --2- 172.21.15.42:0/2499464722 >> v2:172.21.15.154:6812/254845342 conn(0x7fe2600377f0 0x7fe260039cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:46:12.959 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.916+0000 7fe271ffb640 1 -- 172.21.15.42:0/2499464722 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(123..123 src has 1..123) ==== 6278+0+0 (secure 0 0 0) 0x7fe268049080 con 0x7fe2740a1710 2024-09-16T03:46:12.959 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.916+0000 7fe271ffb640 10 client.13589.objecter ms_dispatch 0x7fe274000ba0 osd_map(123..123 src has 1..123) 2024-09-16T03:46:12.959 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.916+0000 7fe271ffb640 3 client.13589.objecter handle_osd_map got epochs [123,123] > 0 2024-09-16T03:46:12.959 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.916+0000 7fe271ffb640 3 client.13589.objecter handle_osd_map decoding full epoch 123 2024-09-16T03:46:12.959 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.916+0000 7fe271ffb640 20 client.13589.objecter dump_active .. 0 homeless 2024-09-16T03:46:12.959 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.916+0000 7fe279295640 1 --2- 172.21.15.42:0/2499464722 >> v2:172.21.15.154:6812/254845342 conn(0x7fe2600377f0 0x7fe260039cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:46:12.959 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.916+0000 7fe279295640 10 monclient: get_auth_request con 0x7fe2600377f0 auth_method 0 2024-09-16T03:46:12.959 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.920+0000 7fe279295640 1 --2- 172.21.15.42:0/2499464722 >> v2:172.21.15.154:6812/254845342 conn(0x7fe2600377f0 0x7fe260039cb0 secure :-1 s=READY pgs=3408 cs=0 l=1 rev1=1 crypto rx=0x7fe2640ad000 tx=0x7fe264008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:46:12.960 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.920+0000 7fe271ffb640 10 client.13589.objecter ms_handle_connect 0x7fe2600377f0 2024-09-16T03:46:12.960 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.920+0000 7fe271ffb640 1 -- 172.21.15.42:0/2499464722 <== mon.2 v2:172.21.15.42:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7fe2740721c0 con 0x7fe2740a1710 2024-09-16T03:46:12.960 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.920+0000 7fe271ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:46:12.960 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:12.920+0000 7fe271ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:46:13.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:13.144+0000 7fe27b4f9640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:46:13.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:13.144+0000 7fe27b4f9640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:46:13.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:13.144+0000 7fe27b4f9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:46:13.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:13.144+0000 7fe27b4f9640 1 -- 172.21.15.42:0/2499464722 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fe274072710 con 0x7fe2740a1710 2024-09-16T03:46:13.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:13.144+0000 7fe271ffb640 1 -- 172.21.15.42:0/2499464722 <== mon.2 v2:172.21.15.42:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7fe268012320 con 0x7fe2740a1710 2024-09-16T03:46:13.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:13.144+0000 7fe271ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:46:13.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:13.144+0000 7fe271ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:46:13.153 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":34,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:46:13.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:13.148+0000 7fe27b4f9640 10 librados: watch_flush enter 2024-09-16T03:46:13.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:13.148+0000 7fe27b4f9640 10 librados: watch_flush exit 2024-09-16T03:46:13.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:13.148+0000 7fe27b4f9640 20 client.13589.objecter shutdown clearing up homeless session... 2024-09-16T03:46:13.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:13.148+0000 7fe27b4f9640 10 client.13589.objecter successfully canceled tick 2024-09-16T03:46:13.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:13.148+0000 7fe27b4f9640 1 -- 172.21.15.42:0/2499464722 >> v2:172.21.15.154:6812/254845342 conn(0x7fe2600377f0 msgr2=0x7fe260039cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:46:13.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:13.148+0000 7fe27b4f9640 1 --2- 172.21.15.42:0/2499464722 >> v2:172.21.15.154:6812/254845342 conn(0x7fe2600377f0 0x7fe260039cb0 secure :-1 s=READY pgs=3408 cs=0 l=1 rev1=1 crypto rx=0x7fe2640ad000 tx=0x7fe264008040 comp rx=0 tx=0).stop 2024-09-16T03:46:13.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:13.148+0000 7fe27b4f9640 10 monclient: shutdown 2024-09-16T03:46:13.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:13.148+0000 7fe27b4f9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:46:13.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:13.148+0000 7fe27b4f9640 1 -- 172.21.15.42:0/2499464722 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe2740a1710 msgr2=0x7fe2741cfd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:46:13.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:13.148+0000 7fe27b4f9640 1 --2- 172.21.15.42:0/2499464722 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe2740a1710 0x7fe2741cfd40 secure :-1 s=READY pgs=2439 cs=0 l=1 rev1=1 crypto rx=0x7fe26800b9e0 tx=0x7fe26800bea0 comp rx=0 tx=0).stop 2024-09-16T03:46:13.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:13.148+0000 7fe27b4f9640 1 -- 172.21.15.42:0/2499464722 shutdown_connections 2024-09-16T03:46:13.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:13.148+0000 7fe27b4f9640 1 --2- 172.21.15.42:0/2499464722 >> v2:172.21.15.154:6812/254845342 conn(0x7fe2600377f0 0x7fe260039cb0 unknown :-1 s=CLOSED pgs=3408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:46:13.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:13.148+0000 7fe27b4f9640 1 --2- 172.21.15.42:0/2499464722 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe2741340f0 0x7fe2741cc040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:46:13.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:13.148+0000 7fe27b4f9640 1 --2- 172.21.15.42:0/2499464722 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe2740a1710 0x7fe2741cfd40 unknown :-1 s=CLOSED pgs=2439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:46:13.157 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:13.148+0000 7fe27b4f9640 1 --2- 172.21.15.42:0/2499464722 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe2740a1330 0x7fe27413a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:46:13.157 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:13.148+0000 7fe27b4f9640 1 -- 172.21.15.42:0/2499464722 >> 172.21.15.42:0/2499464722 conn(0x7fe2740a2420 msgr2=0x7fe27409c270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:46:13.157 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:13.148+0000 7fe27b4f9640 1 -- 172.21.15.42:0/2499464722 shutdown_connections 2024-09-16T03:46:13.157 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:13.148+0000 7fe27b4f9640 1 -- 172.21.15.42:0/2499464722 wait complete. 2024-09-16T03:46:13.157 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:13.148+0000 7fe27b4f9640 1 librados: shutdown 2024-09-16T03:46:16.171 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:46:16.306 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.300+0000 7f274f0f0640 10 monclient: get_monmap_and_config 2024-09-16T03:46:16.307 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.300+0000 7f274f0f0640 10 monclient: build_initial_monmap 2024-09-16T03:46:16.307 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.300+0000 7f274f0f0640 10 monclient: monmap: 2024-09-16T03:46:16.307 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:46:16.307 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:46:16.307 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:46:16.305700+0000 2024-09-16T03:46:16.307 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:46:16.305700+0000 2024-09-16T03:46:16.308 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:46:16.308 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:46:16.308 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:46:16.308 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:46:16.308 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:46:16.308 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:46:16.308 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:46:16.308 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:46:16.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.300+0000 7f274f0f0640 1 Processor -- start 2024-09-16T03:46:16.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.300+0000 7f274f0f0640 1 -- start start 2024-09-16T03:46:16.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.300+0000 7f274f0f0640 10 monclient: init 2024-09-16T03:46:16.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.300+0000 7f274f0f0640 10 monclient: _reopen_session rank -1 2024-09-16T03:46:16.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.300+0000 7f274f0f0640 10 monclient: _add_conns ranks=[1,4,2,3,0] 2024-09-16T03:46:16.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.300+0000 7f274f0f0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f27480a39b0 addr v1:172.21.15.154:6789/0 2024-09-16T03:46:16.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.300+0000 7f274f0f0640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f27480a3d90 0x7f274813c380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:46:16.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.300+0000 7f274f0f0640 10 monclient(hunting): picked mon.noname-c con 0x7f27480a3d90 addr v2:172.21.15.154:3300/0 2024-09-16T03:46:16.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.300+0000 7f274f0f0640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f274813c8c0 0x7f274813ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:46:16.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.300+0000 7f274f0f0640 10 monclient(hunting): picked mon.noname-b con 0x7f274813c8c0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:46:16.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.300+0000 7f274f0f0640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2748134290 con 0x7f27480a39b0 2024-09-16T03:46:16.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.300+0000 7f274f0f0640 10 monclient(hunting): start opening mon connection 2024-09-16T03:46:16.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.300+0000 7f274f0f0640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f274809c140 con 0x7f27480a3d90 2024-09-16T03:46:16.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.300+0000 7f274f0f0640 10 monclient(hunting): start opening mon connection 2024-09-16T03:46:16.314 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.300+0000 7f274f0f0640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f274809c420 con 0x7f274813c8c0 2024-09-16T03:46:16.314 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.300+0000 7f274f0f0640 10 monclient(hunting): _renew_subs 2024-09-16T03:46:16.314 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.300+0000 7f274f0f0640 10 monclient(hunting): authenticate will time out at 1223.275760s 2024-09-16T03:46:16.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.300+0000 7f274d68d640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f274813c8c0 0x7f274813ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:46:16.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.304+0000 7f274d68d640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f274813c8c0 0x7f274813ecb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:34928/0 (socket says 172.21.15.42:34928) 2024-09-16T03:46:16.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.304+0000 7f274d68d640 1 -- 172.21.15.42:0/2479130413 learned_addr learned my addr 172.21.15.42:0/2479130413 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:46:16.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.304+0000 7f274d68d640 10 monclient(hunting): get_auth_request con 0x7f274813c8c0 auth_method 0 2024-09-16T03:46:16.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.304+0000 7f274d68d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:46:16.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.304+0000 7f274d68d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:46:16.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.304+0000 7f274d68d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:46:16.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.304+0000 7f274d68d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:46:16.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.304+0000 7f274d68d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:46:16.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.304+0000 7f273ffff640 1 --2- 172.21.15.42:0/2479130413 >> v2:172.21.15.154:3300/0 conn(0x7f27480a3d90 0x7f274813c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:46:16.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.304+0000 7f274d68d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:46:16.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.304+0000 7f273ffff640 10 monclient(hunting): get_auth_request con 0x7f27480a3d90 auth_method 0 2024-09-16T03:46:16.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.304+0000 7f273ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:46:16.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.304+0000 7f273ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:46:16.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.304+0000 7f273ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:46:16.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.304+0000 7f274d68d640 10 monclient(hunting): handle_auth_done global_id 13592 payload 274 2024-09-16T03:46:16.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.304+0000 7f274d68d640 1 -- 172.21.15.42:0/2479130413 >> v2:172.21.15.154:3300/0 conn(0x7f27480a3d90 msgr2=0x7f274813c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:46:16.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.304+0000 7f274d68d640 1 --2- 172.21.15.42:0/2479130413 >> v2:172.21.15.154:3300/0 conn(0x7f27480a3d90 0x7f274813c380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:46:16.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.304+0000 7f274d68d640 1 -- 172.21.15.42:0/2479130413 >> v1:172.21.15.154:6789/0 conn(0x7f27480a39b0 legacy=0x7f2748133b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:46:16.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.304+0000 7f274d68d640 10 monclient: _finish_hunting 0 2024-09-16T03:46:16.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.304+0000 7f274d68d640 1 monclient: found mon.noname-b 2024-09-16T03:46:16.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.304+0000 7f274d68d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:46:16.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.304+0000 7f274d68d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T03:46:16.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.304+0000 7f274d68d640 1 -- 172.21.15.42:0/2479130413 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f274806dd60 con 0x7f274813c8c0 2024-09-16T03:46:16.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.304+0000 7f274d68d640 1 --2- 172.21.15.42:0/2479130413 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f274813c8c0 0x7f274813ecb0 secure :-1 s=READY pgs=2441 cs=0 l=1 rev1=1 crypto rx=0x7f2740009e50 tx=0x7f274001c8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9089cea9cc991af2 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:46:16.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.304+0000 7f273f7fe640 1 -- 172.21.15.42:0/2479130413 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27400a5530 con 0x7f274813c8c0 2024-09-16T03:46:16.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.304+0000 7f273f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:46:16.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.304+0000 7f273f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:46:16.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.304+0000 7f273f7fe640 10 monclient: dump: 2024-09-16T03:46:16.327 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:46:16.327 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:46:16.327 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:16.327 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:16.327 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:46:16.328 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:46:16.328 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:46:16.328 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:46:16.328 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:46:16.328 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:46:16.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.304+0000 7f273f7fe640 10 monclient: _finish_auth 0 2024-09-16T03:46:16.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.304+0000 7f273f7fe640 10 monclient: _check_auth_tickets 2024-09-16T03:46:16.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.304+0000 7f273f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:46:16.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.304+0000 7f273f7fe640 1 -- 172.21.15.42:0/2479130413 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f27400a56d0 con 0x7f274813c8c0 2024-09-16T03:46:16.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.304+0000 7f274f0f0640 5 monclient: authenticate success, global_id 13592 2024-09-16T03:46:16.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.304+0000 7f274f0f0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:46:16.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.304+0000 7f273f7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:46:16.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.304+0000 7f274f0f0640 10 monclient: get_monmap_and_config success 2024-09-16T03:46:16.329 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.304+0000 7f274f0f0640 4 set_mon_vals no callback set 2024-09-16T03:46:16.329 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.304+0000 7f273f7fe640 1 -- 172.21.15.42:0/2479130413 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27400a5aa0 con 0x7f274813c8c0 2024-09-16T03:46:16.329 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.304+0000 7f273f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:46:16.329 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.304+0000 7f273f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:46:16.329 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.304+0000 7f273f7fe640 10 monclient: dump: 2024-09-16T03:46:16.329 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:46:16.329 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:46:16.329 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:16.329 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:16.329 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:46:16.329 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:46:16.329 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:46:16.329 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:46:16.329 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:46:16.330 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:46:16.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.308+0000 7f274f0f0640 10 monclient: shutdown 2024-09-16T03:46:16.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.308+0000 7f274f0f0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:46:16.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.308+0000 7f274f0f0640 1 -- 172.21.15.42:0/2479130413 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f274813c8c0 msgr2=0x7f274813ecb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:46:16.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.308+0000 7f274f0f0640 1 --2- 172.21.15.42:0/2479130413 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f274813c8c0 0x7f274813ecb0 secure :-1 s=READY pgs=2441 cs=0 l=1 rev1=1 crypto rx=0x7f2740009e50 tx=0x7f274001c8b0 comp rx=0 tx=0).stop 2024-09-16T03:46:16.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.308+0000 7f274f0f0640 1 -- 172.21.15.42:0/2479130413 shutdown_connections 2024-09-16T03:46:16.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.308+0000 7f274f0f0640 1 --2- 172.21.15.42:0/2479130413 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f274813c8c0 0x7f274813ecb0 unknown :-1 s=CLOSED pgs=2441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:46:16.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.308+0000 7f274f0f0640 1 --2- 172.21.15.42:0/2479130413 >> v2:172.21.15.154:3300/0 conn(0x7f27480a3d90 0x7f274813c380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:46:16.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.308+0000 7f274f0f0640 1 -- 172.21.15.42:0/2479130413 >> 172.21.15.42:0/2479130413 conn(0x7f27480a9fb0 msgr2=0x7f27480aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:46:16.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.308+0000 7f274f0f0640 1 -- 172.21.15.42:0/2479130413 shutdown_connections 2024-09-16T03:46:16.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.308+0000 7f274f0f0640 1 -- 172.21.15.42:0/2479130413 wait complete. 2024-09-16T03:46:16.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.308+0000 7f274f0f0640 10 monclient: build_initial_monmap 2024-09-16T03:46:16.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.308+0000 7f274f0f0640 10 monclient: monmap: 2024-09-16T03:46:16.331 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:46:16.331 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:46:16.331 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:46:16.331 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:46:16.331 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:46:16.331 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:46:16.331 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:46:16.331 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:46:16.331 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:46:16.331 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:46:16.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.308+0000 7f274f0f0640 1 librados: starting msgr at 2024-09-16T03:46:16.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.308+0000 7f274f0f0640 1 librados: starting objecter 2024-09-16T03:46:16.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.308+0000 7f274f0f0640 1 Processor -- start 2024-09-16T03:46:16.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.308+0000 7f274f0f0640 1 -- start start 2024-09-16T03:46:16.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.308+0000 7f274f0f0640 1 librados: setting wanted keys 2024-09-16T03:46:16.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.308+0000 7f274f0f0640 1 librados: calling monclient init 2024-09-16T03:46:16.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.308+0000 7f274f0f0640 10 monclient: init 2024-09-16T03:46:16.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.308+0000 7f274f0f0640 10 monclient: _reopen_session rank -1 2024-09-16T03:46:16.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.308+0000 7f274f0f0640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:46:16.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.308+0000 7f274f0f0640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f27480a39b0 0x7f27481cf430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:46:16.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.308+0000 7f274f0f0640 10 monclient(hunting): picked mon.noname-b con 0x7f27480a39b0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:46:16.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.308+0000 7f274f0f0640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f27480a3d90 0x7f274813b600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:46:16.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.308+0000 7f274f0f0640 10 monclient(hunting): picked mon.noname-a con 0x7f27480a3d90 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:46:16.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.308+0000 7f274f0f0640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f274813c8c0 0x7f274813bb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:46:16.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.308+0000 7f274f0f0640 10 monclient(hunting): picked mon.noname-c con 0x7f274813c8c0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:46:16.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.308+0000 7f274f0f0640 10 monclient(hunting): start opening mon connection 2024-09-16T03:46:16.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.308+0000 7f274f0f0640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f2748141da0 con 0x7f27480a3d90 2024-09-16T03:46:16.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.308+0000 7f274f0f0640 10 monclient(hunting): start opening mon connection 2024-09-16T03:46:16.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.308+0000 7f274f0f0640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f2748141c20 con 0x7f27480a39b0 2024-09-16T03:46:16.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.308+0000 7f274f0f0640 10 monclient(hunting): start opening mon connection 2024-09-16T03:46:16.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.308+0000 7f274f0f0640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f2748141f20 con 0x7f274813c8c0 2024-09-16T03:46:16.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.308+0000 7f274f0f0640 10 monclient(hunting): _renew_subs 2024-09-16T03:46:16.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.308+0000 7f274f0f0640 10 monclient(hunting): authenticate will time out at 1223.281801s 2024-09-16T03:46:16.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.308+0000 7f273ffff640 1 -- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f27480a3d90 msgr2=0x7f274813b600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.42:3300/0 2024-09-16T03:46:16.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.308+0000 7f273ffff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f27480a3d90 0x7f274813b600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:46:16.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.308+0000 7f274d68d640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f274813c8c0 0x7f274813bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:46:16.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.308+0000 7f274d68d640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f274813c8c0 0x7f274813bb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:34930/0 (socket says 172.21.15.42:34930) 2024-09-16T03:46:16.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.308+0000 7f274d68d640 1 -- 172.21.15.42:0/309862484 learned_addr learned my addr 172.21.15.42:0/309862484 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:46:16.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.308+0000 7f274d68d640 10 monclient(hunting): get_auth_request con 0x7f274813c8c0 auth_method 0 2024-09-16T03:46:16.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.308+0000 7f274d68d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:46:16.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.308+0000 7f274d68d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:46:16.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.308+0000 7f274d68d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:46:16.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.308+0000 7f274d68d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:46:16.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.308+0000 7f274d68d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:46:16.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.308+0000 7f274d68d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:46:16.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.308+0000 7f274ce8c640 1 --2- 172.21.15.42:0/309862484 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f27480a39b0 0x7f27481cf430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:46:16.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.308+0000 7f274ce8c640 10 monclient(hunting): get_auth_request con 0x7f27480a39b0 auth_method 0 2024-09-16T03:46:16.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.308+0000 7f274ce8c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:46:16.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.308+0000 7f274ce8c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:46:16.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.308+0000 7f274ce8c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:46:16.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.308+0000 7f274d68d640 10 monclient(hunting): handle_auth_done global_id 13595 payload 824 2024-09-16T03:46:16.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.308+0000 7f274d68d640 1 -- 172.21.15.42:0/309862484 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f27480a39b0 msgr2=0x7f27481cf430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:46:16.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.308+0000 7f274d68d640 1 --2- 172.21.15.42:0/309862484 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f27480a39b0 0x7f27481cf430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:46:16.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.308+0000 7f274d68d640 1 -- 172.21.15.42:0/309862484 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f27480a3d90 msgr2=0x7f274813b600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:46:16.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.308+0000 7f274d68d640 1 --2- 172.21.15.42:0/309862484 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f27480a3d90 0x7f274813b600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:46:16.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.308+0000 7f274d68d640 10 monclient: _finish_hunting 0 2024-09-16T03:46:16.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.308+0000 7f274d68d640 1 monclient: found mon.noname-c 2024-09-16T03:46:16.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.308+0000 7f274d68d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:46:16.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.308+0000 7f274d68d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:46:16.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.308+0000 7f274d68d640 1 -- 172.21.15.42:0/309862484 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2748138d90 con 0x7f274813c8c0 2024-09-16T03:46:16.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.308+0000 7f274d68d640 1 --2- 172.21.15.42:0/309862484 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f274813c8c0 0x7f274813bb40 secure :-1 s=READY pgs=2442 cs=0 l=1 rev1=1 crypto rx=0x7f27400af040 tx=0x7f2740019650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:46:16.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.308+0000 7f273d7fa640 10 client.?.objecter ms_handle_connect 0x7f274813c8c0 2024-09-16T03:46:16.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.308+0000 7f273d7fa640 10 client.?.objecter resend_mon_ops 2024-09-16T03:46:16.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.308+0000 7f273d7fa640 1 -- 172.21.15.42:0/309862484 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27400bc020 con 0x7f274813c8c0 2024-09-16T03:46:16.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.308+0000 7f273d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:46:16.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.308+0000 7f273d7fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:46:16.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.308+0000 7f273d7fa640 10 monclient: dump: 2024-09-16T03:46:16.341 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:46:16.341 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:46:16.341 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:16.341 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:16.341 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:46:16.342 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:46:16.342 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:46:16.342 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:46:16.342 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:46:16.342 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:46:16.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.308+0000 7f273d7fa640 10 monclient: _finish_auth 0 2024-09-16T03:46:16.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.308+0000 7f273d7fa640 10 monclient: _check_auth_tickets 2024-09-16T03:46:16.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.308+0000 7f273d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:46:16.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.308+0000 7f273d7fa640 1 -- 172.21.15.42:0/309862484 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f27400b8880 con 0x7f274813c8c0 2024-09-16T03:46:16.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.308+0000 7f273d7fa640 10 monclient: handle_config config(0 keys) 2024-09-16T03:46:16.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.308+0000 7f273d7fa640 1 -- 172.21.15.42:0/309862484 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27400a5530 con 0x7f274813c8c0 2024-09-16T03:46:16.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.308+0000 7f273d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:46:16.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.308+0000 7f273d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:46:16.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.308+0000 7f273d7fa640 10 monclient: dump: 2024-09-16T03:46:16.344 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:46:16.344 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:46:16.345 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:16.345 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:16.345 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:46:16.345 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:46:16.345 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:46:16.345 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:46:16.345 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:46:16.345 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:46:16.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.308+0000 7f274f0f0640 5 monclient: authenticate success, global_id 13595 2024-09-16T03:46:16.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.308+0000 7f274f0f0640 10 monclient: _renew_subs 2024-09-16T03:46:16.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.308+0000 7f274f0f0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:46:16.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.308+0000 7f274f0f0640 1 -- 172.21.15.42:0/309862484 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2748138210 con 0x7f274813c8c0 2024-09-16T03:46:16.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.308+0000 7f273e7fc640 4 set_mon_vals no callback set 2024-09-16T03:46:16.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.308+0000 7f274f0f0640 10 client.13595.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:46:16.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.308+0000 7f274f0f0640 10 monclient: _renew_subs 2024-09-16T03:46:16.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.308+0000 7f274f0f0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:46:16.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.308+0000 7f274f0f0640 1 -- 172.21.15.42:0/309862484 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f2748139df0 con 0x7f274813c8c0 2024-09-16T03:46:16.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.308+0000 7f274f0f0640 1 librados: init done 2024-09-16T03:46:16.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.308+0000 7f274f0f0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:46:16.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.308+0000 7f274f0f0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:46:16.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.308+0000 7f274f0f0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:46:16.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.308+0000 7f274f0f0640 1 -- 172.21.15.42:0/309862484 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2748133b80 con 0x7f274813c8c0 2024-09-16T03:46:16.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.308+0000 7f273d7fa640 1 -- 172.21.15.42:0/309862484 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f27400bf020 con 0x7f274813c8c0 2024-09-16T03:46:16.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.308+0000 7f273d7fa640 1 --2- 172.21.15.42:0/309862484 >> v2:172.21.15.154:6812/254845342 conn(0x7f27340377c0 0x7f2734039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:46:16.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.312+0000 7f273d7fa640 1 -- 172.21.15.42:0/309862484 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(123..123 src has 1..123) ==== 6278+0+0 (secure 0 0 0) 0x7f27400e4460 con 0x7f274813c8c0 2024-09-16T03:46:16.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.312+0000 7f273d7fa640 10 client.13595.objecter ms_dispatch 0x7f2748000ba0 osd_map(123..123 src has 1..123) 2024-09-16T03:46:16.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.312+0000 7f273d7fa640 3 client.13595.objecter handle_osd_map got epochs [123,123] > 0 2024-09-16T03:46:16.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.312+0000 7f273d7fa640 3 client.13595.objecter handle_osd_map decoding full epoch 123 2024-09-16T03:46:16.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.312+0000 7f273d7fa640 20 client.13595.objecter dump_active .. 0 homeless 2024-09-16T03:46:16.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.312+0000 7f274ce8c640 1 --2- 172.21.15.42:0/309862484 >> v2:172.21.15.154:6812/254845342 conn(0x7f27340377c0 0x7f2734039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:46:16.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.312+0000 7f274ce8c640 10 monclient: get_auth_request con 0x7f27340377c0 auth_method 0 2024-09-16T03:46:16.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.312+0000 7f274ce8c640 1 --2- 172.21.15.42:0/309862484 >> v2:172.21.15.154:6812/254845342 conn(0x7f27340377c0 0x7f2734039c80 secure :-1 s=READY pgs=3409 cs=0 l=1 rev1=1 crypto rx=0x7f2744004e30 tx=0x7f2744009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:46:16.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.312+0000 7f273d7fa640 10 client.13595.objecter ms_handle_connect 0x7f27340377c0 2024-09-16T03:46:16.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.316+0000 7f273d7fa640 1 -- 172.21.15.42:0/309862484 <== mon.2 v2:172.21.15.42:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f27400165c0 con 0x7f274813c8c0 2024-09-16T03:46:16.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.316+0000 7f273d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:46:16.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.316+0000 7f273d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:46:16.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.544+0000 7f274f0f0640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:46:16.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.544+0000 7f274f0f0640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:46:16.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.544+0000 7f274f0f0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:46:16.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.544+0000 7f274f0f0640 1 -- 172.21.15.42:0/309862484 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f27480a5750 con 0x7f274813c8c0 2024-09-16T03:46:16.551 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.544+0000 7f273d7fa640 1 -- 172.21.15.42:0/309862484 <== mon.2 v2:172.21.15.42:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7f27400039a0 con 0x7f274813c8c0 2024-09-16T03:46:16.551 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.544+0000 7f273d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:46:16.551 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.544+0000 7f273d7fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:46:16.551 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":34,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":13,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:46:16.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.544+0000 7f274f0f0640 10 librados: watch_flush enter 2024-09-16T03:46:16.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.544+0000 7f274f0f0640 10 librados: watch_flush exit 2024-09-16T03:46:16.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.544+0000 7f274f0f0640 20 client.13595.objecter shutdown clearing up homeless session... 2024-09-16T03:46:16.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.544+0000 7f274f0f0640 10 client.13595.objecter successfully canceled tick 2024-09-16T03:46:16.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.548+0000 7f274f0f0640 1 -- 172.21.15.42:0/309862484 >> v2:172.21.15.154:6812/254845342 conn(0x7f27340377c0 msgr2=0x7f2734039c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:46:16.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.548+0000 7f274f0f0640 1 --2- 172.21.15.42:0/309862484 >> v2:172.21.15.154:6812/254845342 conn(0x7f27340377c0 0x7f2734039c80 secure :-1 s=READY pgs=3409 cs=0 l=1 rev1=1 crypto rx=0x7f2744004e30 tx=0x7f2744009040 comp rx=0 tx=0).stop 2024-09-16T03:46:16.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.548+0000 7f274f0f0640 10 monclient: shutdown 2024-09-16T03:46:16.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.548+0000 7f274f0f0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:46:16.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.548+0000 7f274f0f0640 1 -- 172.21.15.42:0/309862484 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f274813c8c0 msgr2=0x7f274813bb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:46:16.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.548+0000 7f274f0f0640 1 --2- 172.21.15.42:0/309862484 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f274813c8c0 0x7f274813bb40 secure :-1 s=READY pgs=2442 cs=0 l=1 rev1=1 crypto rx=0x7f27400af040 tx=0x7f2740019650 comp rx=0 tx=0).stop 2024-09-16T03:46:16.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.548+0000 7f274f0f0640 1 -- 172.21.15.42:0/309862484 shutdown_connections 2024-09-16T03:46:16.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.548+0000 7f274f0f0640 1 --2- 172.21.15.42:0/309862484 >> v2:172.21.15.154:6812/254845342 conn(0x7f27340377c0 0x7f2734039c80 unknown :-1 s=CLOSED pgs=3409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:46:16.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.548+0000 7f274f0f0640 1 --2- 172.21.15.42:0/309862484 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f274813c8c0 0x7f274813bb40 unknown :-1 s=CLOSED pgs=2442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:46:16.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.548+0000 7f274f0f0640 1 --2- 172.21.15.42:0/309862484 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f27480a3d90 0x7f274813b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:46:16.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.548+0000 7f274f0f0640 1 --2- 172.21.15.42:0/309862484 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f27480a39b0 0x7f27481cf430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:46:16.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.548+0000 7f274f0f0640 1 -- 172.21.15.42:0/309862484 >> 172.21.15.42:0/309862484 conn(0x7f27480a9fb0 msgr2=0x7f274812d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:46:16.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.548+0000 7f274f0f0640 1 -- 172.21.15.42:0/309862484 shutdown_connections 2024-09-16T03:46:16.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.548+0000 7f274f0f0640 1 -- 172.21.15.42:0/309862484 wait complete. 2024-09-16T03:46:16.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:16.548+0000 7f274f0f0640 1 librados: shutdown 2024-09-16T03:46:19.569 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:46:19.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:19.696+0000 7fc240eac640 10 monclient: get_monmap_and_config 2024-09-16T03:46:19.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:19.696+0000 7fc240eac640 10 monclient: build_initial_monmap 2024-09-16T03:46:19.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:19.696+0000 7fc240eac640 10 monclient: monmap: 2024-09-16T03:46:19.707 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:46:19.708 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:46:19.708 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:46:19.702870+0000 2024-09-16T03:46:19.708 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:46:19.702870+0000 2024-09-16T03:46:19.708 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:46:19.708 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:46:19.708 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:46:19.708 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:46:19.709 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:46:19.709 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:46:19.709 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:46:19.709 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:46:19.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:19.696+0000 7fc240eac640 1 Processor -- start 2024-09-16T03:46:19.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:19.696+0000 7fc240eac640 1 -- start start 2024-09-16T03:46:19.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:19.696+0000 7fc240eac640 10 monclient: init 2024-09-16T03:46:19.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:19.700+0000 7fc240eac640 10 monclient: _reopen_session rank -1 2024-09-16T03:46:19.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:19.700+0000 7fc240eac640 10 monclient: _add_conns ranks=[2,3,4,0,1] 2024-09-16T03:46:19.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:19.700+0000 7fc240eac640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc23c0a1330 0x7fc23c0a1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:46:19.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:19.700+0000 7fc240eac640 10 monclient(hunting): picked mon.noname-b con 0x7fc23c0a1330 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:46:19.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:19.700+0000 7fc240eac640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fc23c06dac0 0x7fc23c0a1c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:46:19.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:19.700+0000 7fc240eac640 10 monclient(hunting): picked mon.noname-a con 0x7fc23c06dac0 addr v2:172.21.15.42:3300/0 2024-09-16T03:46:19.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:19.700+0000 7fc240eac640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fc23c135100 0x7fc23c135550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:46:19.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:19.700+0000 7fc240eac640 10 monclient(hunting): picked mon.noname-c con 0x7fc23c135100 addr v2:172.21.15.154:3300/0 2024-09-16T03:46:19.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:19.700+0000 7fc240eac640 10 monclient(hunting): start opening mon connection 2024-09-16T03:46:19.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:19.700+0000 7fc240eac640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7fc23c09c7d0 con 0x7fc23c06dac0 2024-09-16T03:46:19.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:19.700+0000 7fc240eac640 10 monclient(hunting): start opening mon connection 2024-09-16T03:46:19.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:19.700+0000 7fc240eac640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fc23c09c090 con 0x7fc23c135100 2024-09-16T03:46:19.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:19.700+0000 7fc240eac640 10 monclient(hunting): start opening mon connection 2024-09-16T03:46:19.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:19.700+0000 7fc240eac640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fc23c09f610 con 0x7fc23c0a1330 2024-09-16T03:46:19.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:19.700+0000 7fc240eac640 10 monclient(hunting): _renew_subs 2024-09-16T03:46:19.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:19.700+0000 7fc240eac640 10 monclient(hunting): authenticate will time out at 1226.672126s 2024-09-16T03:46:19.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:19.700+0000 7fc239d9b640 1 -- >> v2:172.21.15.42:3300/0 conn(0x7fc23c06dac0 msgr2=0x7fc23c0a1c50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3300/0 2024-09-16T03:46:19.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:19.700+0000 7fc239d9b640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fc23c06dac0 0x7fc23c0a1c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:46:19.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:19.700+0000 7fc23a59c640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc23c0a1330 0x7fc23c0a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:46:19.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:19.700+0000 7fc23a59c640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc23c0a1330 0x7fc23c0a1710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:34940/0 (socket says 172.21.15.42:34940) 2024-09-16T03:46:19.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:19.700+0000 7fc23a59c640 1 -- 172.21.15.42:0/1717680761 learned_addr learned my addr 172.21.15.42:0/1717680761 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:46:19.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:19.700+0000 7fc23a59c640 10 monclient(hunting): get_auth_request con 0x7fc23c0a1330 auth_method 0 2024-09-16T03:46:19.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:19.700+0000 7fc23a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:46:19.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:19.700+0000 7fc23a59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:46:19.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:19.700+0000 7fc23a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:46:19.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:19.700+0000 7fc23a59c640 1 -- 172.21.15.42:0/1717680761 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc23c0a1330 msgr2=0x7fc23c0a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T03:46:19.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:19.700+0000 7fc23a59c640 1 -- 172.21.15.42:0/1717680761 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc23c0a1330 msgr2=0x7fc23c0a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:46:19.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:19.700+0000 7fc23a59c640 1 --2- 172.21.15.42:0/1717680761 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc23c0a1330 0x7fc23c0a1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:46:19.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:19.700+0000 7fc23a59c640 1 --2- 172.21.15.42:0/1717680761 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc23c0a1330 0x7fc23c0a1710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:46:19.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:19.700+0000 7fc23ad9d640 1 --2- 172.21.15.42:0/1717680761 >> v2:172.21.15.154:3300/0 conn(0x7fc23c135100 0x7fc23c135550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:46:19.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:19.700+0000 7fc23ad9d640 10 monclient(hunting): get_auth_request con 0x7fc23c135100 auth_method 0 2024-09-16T03:46:19.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:19.700+0000 7fc23ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:46:19.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:19.700+0000 7fc23ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:46:19.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:19.700+0000 7fc23ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:46:19.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:19.700+0000 7fc23ad9d640 1 -- 172.21.15.42:0/1717680761 >> v2:172.21.15.154:3300/0 conn(0x7fc23c135100 msgr2=0x7fc23c135550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T03:46:19.714 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:19.700+0000 7fc23ad9d640 1 -- 172.21.15.42:0/1717680761 >> v2:172.21.15.154:3300/0 conn(0x7fc23c135100 msgr2=0x7fc23c135550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:46:19.714 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:19.700+0000 7fc23ad9d640 1 --2- 172.21.15.42:0/1717680761 >> v2:172.21.15.154:3300/0 conn(0x7fc23c135100 0x7fc23c135550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:46:19.714 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:19.700+0000 7fc23ad9d640 1 --2- 172.21.15.42:0/1717680761 >> v2:172.21.15.154:3300/0 conn(0x7fc23c135100 0x7fc23c135550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:46:19.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:19.904+0000 7fc239d9b640 1 -- 172.21.15.42:0/1717680761 >> v2:172.21.15.42:3300/0 conn(0x7fc23c06dac0 msgr2=0x7fc23c0a1c50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3300/0 2024-09-16T03:46:19.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:19.904+0000 7fc239d9b640 1 --2- 172.21.15.42:0/1717680761 >> v2:172.21.15.42:3300/0 conn(0x7fc23c06dac0 0x7fc23c0a1c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T03:46:19.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:19.904+0000 7fc23a59c640 1 --2- 172.21.15.42:0/1717680761 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc23c0a1330 0x7fc23c0a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:46:19.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:19.904+0000 7fc23a59c640 10 monclient(hunting): get_auth_request con 0x7fc23c0a1330 auth_method 0 2024-09-16T03:46:19.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:19.904+0000 7fc23a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:46:19.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:19.904+0000 7fc23a59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:46:19.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:19.904+0000 7fc23a59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:46:19.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:19.904+0000 7fc23ad9d640 1 --2- 172.21.15.42:0/1717680761 >> v2:172.21.15.154:3300/0 conn(0x7fc23c135100 0x7fc23c135550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:46:19.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:19.904+0000 7fc23ad9d640 10 monclient(hunting): get_auth_request con 0x7fc23c135100 auth_method 0 2024-09-16T03:46:19.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:19.904+0000 7fc23ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:46:19.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:19.904+0000 7fc23ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:46:19.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:19.904+0000 7fc23ad9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:46:19.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:19.904+0000 7fc23a59c640 1 -- 172.21.15.42:0/1717680761 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc23c0a1330 msgr2=0x7fc23c0a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T03:46:19.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:19.904+0000 7fc23a59c640 1 -- 172.21.15.42:0/1717680761 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc23c0a1330 msgr2=0x7fc23c0a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:46:19.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:19.904+0000 7fc23a59c640 1 --2- 172.21.15.42:0/1717680761 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc23c0a1330 0x7fc23c0a1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:46:19.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:19.904+0000 7fc23ad9d640 1 -- 172.21.15.42:0/1717680761 >> v2:172.21.15.154:3300/0 conn(0x7fc23c135100 msgr2=0x7fc23c135550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T03:46:19.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:19.904+0000 7fc23ad9d640 1 -- 172.21.15.42:0/1717680761 >> v2:172.21.15.154:3300/0 conn(0x7fc23c135100 msgr2=0x7fc23c135550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:46:19.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:19.904+0000 7fc23ad9d640 1 --2- 172.21.15.42:0/1717680761 >> v2:172.21.15.154:3300/0 conn(0x7fc23c135100 0x7fc23c135550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:46:19.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:19.904+0000 7fc23a59c640 1 --2- 172.21.15.42:0/1717680761 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc23c0a1330 0x7fc23c0a1710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T03:46:19.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:19.904+0000 7fc23ad9d640 1 --2- 172.21.15.42:0/1717680761 >> v2:172.21.15.154:3300/0 conn(0x7fc23c135100 0x7fc23c135550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T03:46:20.314 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:20.308+0000 7fc239d9b640 1 -- 172.21.15.42:0/1717680761 >> v2:172.21.15.42:3300/0 conn(0x7fc23c06dac0 msgr2=0x7fc23c0a1c50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3300/0 2024-09-16T03:46:20.314 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:20.308+0000 7fc239d9b640 1 --2- 172.21.15.42:0/1717680761 >> v2:172.21.15.42:3300/0 conn(0x7fc23c06dac0 0x7fc23c0a1c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T03:46:20.314 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:20.308+0000 7fc23a59c640 1 --2- 172.21.15.42:0/1717680761 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc23c0a1330 0x7fc23c0a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:46:20.314 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:20.308+0000 7fc23a59c640 10 monclient(hunting): get_auth_request con 0x7fc23c0a1330 auth_method 0 2024-09-16T03:46:20.314 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:20.308+0000 7fc23a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:46:20.314 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:20.308+0000 7fc23a59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:46:20.314 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:20.308+0000 7fc23a59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:46:20.314 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:20.308+0000 7fc23ad9d640 1 --2- 172.21.15.42:0/1717680761 >> v2:172.21.15.154:3300/0 conn(0x7fc23c135100 0x7fc23c135550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:46:20.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:20.308+0000 7fc23a59c640 1 -- 172.21.15.42:0/1717680761 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc23c0a1330 msgr2=0x7fc23c0a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T03:46:20.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:20.308+0000 7fc23a59c640 1 -- 172.21.15.42:0/1717680761 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc23c0a1330 msgr2=0x7fc23c0a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:46:20.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:20.308+0000 7fc23ad9d640 10 monclient(hunting): get_auth_request con 0x7fc23c135100 auth_method 0 2024-09-16T03:46:20.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:20.308+0000 7fc23a59c640 1 --2- 172.21.15.42:0/1717680761 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc23c0a1330 0x7fc23c0a1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:46:20.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:20.308+0000 7fc23ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:46:20.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:20.308+0000 7fc23ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:46:20.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:20.308+0000 7fc23ad9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:46:20.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:20.308+0000 7fc23a59c640 1 --2- 172.21.15.42:0/1717680761 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc23c0a1330 0x7fc23c0a1710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T03:46:20.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:20.308+0000 7fc23ad9d640 1 -- 172.21.15.42:0/1717680761 >> v2:172.21.15.154:3300/0 conn(0x7fc23c135100 msgr2=0x7fc23c135550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T03:46:20.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:20.308+0000 7fc23ad9d640 1 -- 172.21.15.42:0/1717680761 >> v2:172.21.15.154:3300/0 conn(0x7fc23c135100 msgr2=0x7fc23c135550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:46:20.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:20.308+0000 7fc23ad9d640 1 --2- 172.21.15.42:0/1717680761 >> v2:172.21.15.154:3300/0 conn(0x7fc23c135100 0x7fc23c135550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:46:20.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:20.308+0000 7fc23ad9d640 1 --2- 172.21.15.42:0/1717680761 >> v2:172.21.15.154:3300/0 conn(0x7fc23c135100 0x7fc23c135550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T03:46:21.118 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:21.112+0000 7fc239d9b640 1 -- 172.21.15.42:0/1717680761 >> v2:172.21.15.42:3300/0 conn(0x7fc23c06dac0 msgr2=0x7fc23c0a1c50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3300/0 2024-09-16T03:46:21.118 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:21.112+0000 7fc239d9b640 1 --2- 172.21.15.42:0/1717680761 >> v2:172.21.15.42:3300/0 conn(0x7fc23c06dac0 0x7fc23c0a1c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T03:46:21.118 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:21.112+0000 7fc23a59c640 1 --2- 172.21.15.42:0/1717680761 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc23c0a1330 0x7fc23c0a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:46:21.118 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:21.112+0000 7fc23a59c640 10 monclient(hunting): get_auth_request con 0x7fc23c0a1330 auth_method 0 2024-09-16T03:46:21.118 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:21.112+0000 7fc23a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:46:21.118 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:21.112+0000 7fc23a59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:46:21.118 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:21.112+0000 7fc23a59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:46:21.118 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:21.112+0000 7fc23ad9d640 1 --2- 172.21.15.42:0/1717680761 >> v2:172.21.15.154:3300/0 conn(0x7fc23c135100 0x7fc23c135550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:46:21.119 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:21.112+0000 7fc23ad9d640 10 monclient(hunting): get_auth_request con 0x7fc23c135100 auth_method 0 2024-09-16T03:46:21.119 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:21.112+0000 7fc23ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:46:21.119 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:21.112+0000 7fc23ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:46:21.119 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:21.112+0000 7fc23ad9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:46:21.119 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:21.112+0000 7fc23a59c640 1 -- 172.21.15.42:0/1717680761 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc23c0a1330 msgr2=0x7fc23c0a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T03:46:21.119 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:21.112+0000 7fc23a59c640 1 -- 172.21.15.42:0/1717680761 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc23c0a1330 msgr2=0x7fc23c0a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:46:21.119 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:21.112+0000 7fc23a59c640 1 --2- 172.21.15.42:0/1717680761 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc23c0a1330 0x7fc23c0a1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:46:21.119 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:21.112+0000 7fc23a59c640 1 --2- 172.21.15.42:0/1717680761 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc23c0a1330 0x7fc23c0a1710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T03:46:21.119 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:21.112+0000 7fc23ad9d640 1 -- 172.21.15.42:0/1717680761 >> v2:172.21.15.154:3300/0 conn(0x7fc23c135100 msgr2=0x7fc23c135550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T03:46:21.119 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:21.112+0000 7fc23ad9d640 1 -- 172.21.15.42:0/1717680761 >> v2:172.21.15.154:3300/0 conn(0x7fc23c135100 msgr2=0x7fc23c135550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:46:21.119 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:21.112+0000 7fc23ad9d640 1 --2- 172.21.15.42:0/1717680761 >> v2:172.21.15.154:3300/0 conn(0x7fc23c135100 0x7fc23c135550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:46:21.119 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:21.112+0000 7fc23ad9d640 1 --2- 172.21.15.42:0/1717680761 >> v2:172.21.15.154:3300/0 conn(0x7fc23c135100 0x7fc23c135550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T03:46:22.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:22.700+0000 7fc22bfff640 10 monclient(hunting): tick 2024-09-16T03:46:22.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:22.700+0000 7fc22bfff640 10 monclient(hunting): _check_auth_tickets 2024-09-16T03:46:22.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:22.700+0000 7fc22bfff640 1 monclient(hunting): continuing hunt 2024-09-16T03:46:22.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:22.700+0000 7fc22bfff640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T03:46:22.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:22.700+0000 7fc22bfff640 1 -- 172.21.15.42:0/1717680761 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc23c0a1330 msgr2=0x7fc23c0a1710 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:46:22.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:22.700+0000 7fc22bfff640 1 --2- 172.21.15.42:0/1717680761 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc23c0a1330 0x7fc23c0a1710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:46:22.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:22.700+0000 7fc22bfff640 1 -- 172.21.15.42:0/1717680761 >> v2:172.21.15.154:3300/0 conn(0x7fc23c135100 msgr2=0x7fc23c135550 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:46:22.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:22.700+0000 7fc22bfff640 1 --2- 172.21.15.42:0/1717680761 >> v2:172.21.15.154:3300/0 conn(0x7fc23c135100 0x7fc23c135550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:46:22.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:22.700+0000 7fc22bfff640 1 -- 172.21.15.42:0/1717680761 >> v2:172.21.15.42:3300/0 conn(0x7fc23c06dac0 msgr2=0x7fc23c0a1c50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:46:22.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:22.700+0000 7fc22bfff640 1 --2- 172.21.15.42:0/1717680761 >> v2:172.21.15.42:3300/0 conn(0x7fc23c06dac0 0x7fc23c0a1c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:46:22.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:22.700+0000 7fc22bfff640 10 monclient: _add_conns ranks=[0,1] 2024-09-16T03:46:22.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:22.700+0000 7fc22bfff640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc21c003680 addr v1:172.21.15.42:6789/0 2024-09-16T03:46:22.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:22.700+0000 7fc22bfff640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc21c007330 addr v1:172.21.15.154:6789/0 2024-09-16T03:46:22.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:22.700+0000 7fc22bfff640 1 -- 172.21.15.42:0/1717680761 --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc23c09f8f0 con 0x7fc21c003680 2024-09-16T03:46:22.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:22.700+0000 7fc22bfff640 1 -- 172.21.15.42:0/1717680761 --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc21c00c0b0 con 0x7fc21c007330 2024-09-16T03:46:22.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:22.700+0000 7fc22bfff640 10 monclient(hunting): _renew_subs 2024-09-16T03:46:22.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:22.700+0000 7fc23a59c640 1 -- 172.21.15.42:0/1717680761 >> v1:172.21.15.42:6789/0 conn(0x7fc21c003680 legacy=0x7fc21c005b00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.42:6789/0 2024-09-16T03:46:22.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:22.904+0000 7fc23a59c640 1 -- 172.21.15.42:0/1717680761 >> v1:172.21.15.42:6789/0 conn(0x7fc21c003680 legacy=0x7fc21c005b00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.42:6789/0 2024-09-16T03:46:23.309 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:23.308+0000 7fc23a59c640 1 -- 172.21.15.42:0/1717680761 >> v1:172.21.15.42:6789/0 conn(0x7fc21c003680 legacy=0x7fc21c005b00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.42:6789/0 2024-09-16T03:46:24.117 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:24.112+0000 7fc23a59c640 1 -- 172.21.15.42:0/1717680761 >> v1:172.21.15.42:6789/0 conn(0x7fc21c003680 legacy=0x7fc21c005b00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.42:6789/0 2024-09-16T03:46:25.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:25.704+0000 7fc22bfff640 10 monclient(hunting): tick 2024-09-16T03:46:25.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:25.704+0000 7fc22bfff640 10 monclient(hunting): _check_auth_tickets 2024-09-16T03:46:25.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:25.704+0000 7fc22bfff640 1 monclient(hunting): continuing hunt 2024-09-16T03:46:25.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:25.704+0000 7fc22bfff640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T03:46:25.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:25.704+0000 7fc22bfff640 1 -- 172.21.15.42:0/1717680761 >> v1:172.21.15.154:6789/0 conn(0x7fc21c007330 legacy=0x7fc21c0097f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:46:25.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:25.704+0000 7fc22bfff640 1 -- 172.21.15.42:0/1717680761 >> v1:172.21.15.42:6789/0 conn(0x7fc21c003680 legacy=0x7fc21c005b00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:46:25.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:25.704+0000 7fc23ad9d640 1 -- 172.21.15.42:0/1717680761 reap_dead start 2024-09-16T03:46:25.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:25.704+0000 7fc22bfff640 10 monclient: _add_conns ranks=[3,1,2,0,4] 2024-09-16T03:46:25.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:25.704+0000 7fc22bfff640 1 --2- 172.21.15.42:0/1717680761 >> v2:172.21.15.42:3300/0 conn(0x7fc21c00c3d0 0x7fc21c005b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:46:25.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:25.704+0000 7fc22bfff640 10 monclient(hunting): picked mon.noname-a con 0x7fc21c00c3d0 addr v2:172.21.15.42:3300/0 2024-09-16T03:46:25.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:25.704+0000 7fc22bfff640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc21c00c7b0 addr v1:172.21.15.154:6789/0 2024-09-16T03:46:25.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:25.704+0000 7fc22bfff640 1 --2- 172.21.15.42:0/1717680761 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc21c00cbf0 0x7fc21c00b060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:46:25.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:25.704+0000 7fc22bfff640 10 monclient(hunting): picked mon.noname-b con 0x7fc21c00cbf0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:46:25.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:25.704+0000 7fc22bfff640 1 -- 172.21.15.42:0/1717680761 --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc21c00d350 con 0x7fc21c00c7b0 2024-09-16T03:46:25.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:25.704+0000 7fc22bfff640 10 monclient(hunting): start opening mon connection 2024-09-16T03:46:25.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:25.704+0000 7fc22bfff640 1 -- 172.21.15.42:0/1717680761 --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7fc23c09c7d0 con 0x7fc21c00c3d0 2024-09-16T03:46:25.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:25.704+0000 7fc22bfff640 10 monclient(hunting): start opening mon connection 2024-09-16T03:46:25.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:25.704+0000 7fc22bfff640 1 -- 172.21.15.42:0/1717680761 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fc23c09c090 con 0x7fc21c00cbf0 2024-09-16T03:46:25.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:25.704+0000 7fc22bfff640 10 monclient(hunting): _renew_subs 2024-09-16T03:46:25.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:25.704+0000 7fc23a59c640 1 -- 172.21.15.42:0/1717680761 >> v2:172.21.15.42:3300/0 conn(0x7fc21c00c3d0 msgr2=0x7fc21c005b00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3300/0 2024-09-16T03:46:25.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:25.704+0000 7fc23a59c640 1 --2- 172.21.15.42:0/1717680761 >> v2:172.21.15.42:3300/0 conn(0x7fc21c00c3d0 0x7fc21c005b00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:46:25.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:25.704+0000 7fc23ad9d640 1 --2- 172.21.15.42:0/1717680761 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc21c00cbf0 0x7fc21c00b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:46:25.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:25.704+0000 7fc23ad9d640 10 monclient(hunting): get_auth_request con 0x7fc21c00cbf0 auth_method 0 2024-09-16T03:46:25.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:25.704+0000 7fc23ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:46:25.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:25.704+0000 7fc23ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:46:25.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:25.704+0000 7fc23ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:46:25.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:25.704+0000 7fc23ad9d640 1 -- 172.21.15.42:0/1717680761 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc21c00cbf0 msgr2=0x7fc21c00b060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T03:46:25.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:25.704+0000 7fc23ad9d640 1 -- 172.21.15.42:0/1717680761 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc21c00cbf0 msgr2=0x7fc21c00b060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:46:25.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:25.704+0000 7fc23ad9d640 1 --2- 172.21.15.42:0/1717680761 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc21c00cbf0 0x7fc21c00b060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:46:25.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:25.704+0000 7fc23ad9d640 1 --2- 172.21.15.42:0/1717680761 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc21c00cbf0 0x7fc21c00b060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:46:25.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:25.908+0000 7fc23a59c640 1 -- 172.21.15.42:0/1717680761 >> v2:172.21.15.42:3300/0 conn(0x7fc21c00c3d0 msgr2=0x7fc21c005b00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3300/0 2024-09-16T03:46:25.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:25.908+0000 7fc23a59c640 1 --2- 172.21.15.42:0/1717680761 >> v2:172.21.15.42:3300/0 conn(0x7fc21c00c3d0 0x7fc21c005b00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T03:46:25.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:25.908+0000 7fc23ad9d640 1 --2- 172.21.15.42:0/1717680761 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc21c00cbf0 0x7fc21c00b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:46:25.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:25.908+0000 7fc23ad9d640 10 monclient(hunting): get_auth_request con 0x7fc21c00cbf0 auth_method 0 2024-09-16T03:46:25.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:25.908+0000 7fc23ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:46:25.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:25.908+0000 7fc23ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:46:25.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:25.908+0000 7fc23ad9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:46:25.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:25.908+0000 7fc23ad9d640 1 -- 172.21.15.42:0/1717680761 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc21c00cbf0 msgr2=0x7fc21c00b060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T03:46:25.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:25.908+0000 7fc23ad9d640 1 -- 172.21.15.42:0/1717680761 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc21c00cbf0 msgr2=0x7fc21c00b060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:46:25.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:25.908+0000 7fc23ad9d640 1 --2- 172.21.15.42:0/1717680761 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc21c00cbf0 0x7fc21c00b060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:46:25.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:25.908+0000 7fc23ad9d640 1 --2- 172.21.15.42:0/1717680761 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc21c00cbf0 0x7fc21c00b060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T03:46:26.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:26.312+0000 7fc23a59c640 1 -- 172.21.15.42:0/1717680761 >> v2:172.21.15.42:3300/0 conn(0x7fc21c00c3d0 msgr2=0x7fc21c005b00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3300/0 2024-09-16T03:46:26.314 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:26.312+0000 7fc23a59c640 1 --2- 172.21.15.42:0/1717680761 >> v2:172.21.15.42:3300/0 conn(0x7fc21c00c3d0 0x7fc21c005b00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T03:46:26.314 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:26.312+0000 7fc23ad9d640 1 --2- 172.21.15.42:0/1717680761 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc21c00cbf0 0x7fc21c00b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:46:26.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:26.312+0000 7fc23ad9d640 10 monclient(hunting): get_auth_request con 0x7fc21c00cbf0 auth_method 0 2024-09-16T03:46:26.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:26.312+0000 7fc23ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:46:26.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:26.312+0000 7fc23ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:46:26.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:26.312+0000 7fc23ad9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:46:26.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:26.312+0000 7fc23ad9d640 1 -- 172.21.15.42:0/1717680761 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc21c00cbf0 msgr2=0x7fc21c00b060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T03:46:26.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:26.312+0000 7fc23ad9d640 1 -- 172.21.15.42:0/1717680761 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc21c00cbf0 msgr2=0x7fc21c00b060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:46:26.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:26.312+0000 7fc23ad9d640 1 --2- 172.21.15.42:0/1717680761 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc21c00cbf0 0x7fc21c00b060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:46:26.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:26.312+0000 7fc23ad9d640 1 --2- 172.21.15.42:0/1717680761 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc21c00cbf0 0x7fc21c00b060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T03:46:27.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:27.116+0000 7fc23a59c640 1 -- 172.21.15.42:0/1717680761 >> v2:172.21.15.42:3300/0 conn(0x7fc21c00c3d0 msgr2=0x7fc21c005b00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3300/0 2024-09-16T03:46:27.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:27.116+0000 7fc23a59c640 1 --2- 172.21.15.42:0/1717680761 >> v2:172.21.15.42:3300/0 conn(0x7fc21c00c3d0 0x7fc21c005b00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T03:46:27.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:27.116+0000 7fc23ad9d640 1 --2- 172.21.15.42:0/1717680761 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc21c00cbf0 0x7fc21c00b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:46:27.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:27.116+0000 7fc23ad9d640 10 monclient(hunting): get_auth_request con 0x7fc21c00cbf0 auth_method 0 2024-09-16T03:46:27.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:27.116+0000 7fc23ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:46:27.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:27.116+0000 7fc23ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:46:27.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:27.116+0000 7fc23ad9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:46:27.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:27.116+0000 7fc23ad9d640 1 -- 172.21.15.42:0/1717680761 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc21c00cbf0 msgr2=0x7fc21c00b060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T03:46:27.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:27.116+0000 7fc23ad9d640 1 -- 172.21.15.42:0/1717680761 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc21c00cbf0 msgr2=0x7fc21c00b060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:46:27.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:27.116+0000 7fc23ad9d640 1 --2- 172.21.15.42:0/1717680761 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc21c00cbf0 0x7fc21c00b060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:46:27.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:27.116+0000 7fc23ad9d640 1 --2- 172.21.15.42:0/1717680761 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc21c00cbf0 0x7fc21c00b060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T03:46:28.460 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.452+0000 7fc23959a640 1 -- 172.21.15.42:0/1717680761 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 303272113 0 0) 0x7fc21c00d350 con 0x7fc21c00c7b0 2024-09-16T03:46:28.460 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.452+0000 7fc23959a640 10 monclient(hunting): _negotiate 2024-09-16T03:46:28.460 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.452+0000 7fc23959a640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:46:28.460 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.452+0000 7fc23959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:46:28.460 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.452+0000 7fc23959a640 10 monclient(hunting): my global_id is 754156 2024-09-16T03:46:28.460 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.452+0000 7fc23959a640 1 -- 172.21.15.42:0/1717680761 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc224003530 con 0x7fc21c00c7b0 2024-09-16T03:46:28.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.500+0000 7fc23959a640 1 -- 172.21.15.42:0/1717680761 <== mon.1 v1:172.21.15.154:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4098373068 0 0) 0x7fc224003530 con 0x7fc21c00c7b0 2024-09-16T03:46:28.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.500+0000 7fc23959a640 1 -- 172.21.15.42:0/1717680761 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc21c00cbf0 msgr2=0x7fc21c00b060 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:46:28.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.500+0000 7fc23959a640 1 --2- 172.21.15.42:0/1717680761 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc21c00cbf0 0x7fc21c00b060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:46:28.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.500+0000 7fc23959a640 1 -- 172.21.15.42:0/1717680761 >> v2:172.21.15.42:3300/0 conn(0x7fc21c00c3d0 msgr2=0x7fc21c005b00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:46:28.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.500+0000 7fc23959a640 1 --2- 172.21.15.42:0/1717680761 >> v2:172.21.15.42:3300/0 conn(0x7fc21c00c3d0 0x7fc21c005b00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:46:28.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.500+0000 7fc23959a640 10 monclient: _finish_hunting 0 2024-09-16T03:46:28.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.500+0000 7fc23959a640 1 monclient: found mon.noname-c-legacy 2024-09-16T03:46:28.508 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.500+0000 7fc23959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:46:28.508 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.500+0000 7fc23959a640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.154:6789/0 2024-09-16T03:46:28.508 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.500+0000 7fc23959a640 1 -- 172.21.15.42:0/1717680761 --> v1:172.21.15.154:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc23c09f8f0 con 0x7fc21c00c7b0 2024-09-16T03:46:28.508 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.500+0000 7fc23959a640 10 monclient: _finish_auth 0 2024-09-16T03:46:28.508 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.500+0000 7fc23959a640 10 monclient: _check_auth_tickets 2024-09-16T03:46:28.508 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.500+0000 7fc23959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:46:28.508 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.500+0000 7fc23959a640 1 -- 172.21.15.42:0/1717680761 <== mon.1 v1:172.21.15.154:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2177413734 0 0) 0x7fc22c004200 con 0x7fc21c00c7b0 2024-09-16T03:46:28.508 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.500+0000 7fc23959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:46:28.508 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.500+0000 7fc23959a640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-16T03:46:28.508 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.500+0000 7fc23959a640 10 monclient: dump: 2024-09-16T03:46:28.508 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:46:28.508 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:46:28.508 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:28.509 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:28.509 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:46:28.509 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:46:28.509 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:46:28.509 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:46:28.509 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:46:28.509 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:46:28.509 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.500+0000 7fc23959a640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] but i'm connected to v1:172.21.15.154:6789/0, reconnecting 2024-09-16T03:46:28.509 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.500+0000 7fc23959a640 10 monclient: _reopen_session rank -1 2024-09-16T03:46:28.509 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.500+0000 7fc23959a640 1 -- 172.21.15.42:0/1717680761 >> v1:172.21.15.154:6789/0 conn(0x7fc21c00c7b0 legacy=0x7fc21c0097f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:46:28.509 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.500+0000 7fc23959a640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T03:46:28.509 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.500+0000 7fc23959a640 1 --2- 172.21.15.42:0/1717680761 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc2240039c0 0x7fc224003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:46:28.509 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.500+0000 7fc23959a640 10 monclient(hunting): picked mon.a con 0x7fc2240039c0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:46:28.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.500+0000 7fc23959a640 1 --2- 172.21.15.42:0/1717680761 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc224004810 0x7fc224004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:46:28.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.500+0000 7fc23959a640 10 monclient(hunting): picked mon.b con 0x7fc224004810 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:46:28.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.500+0000 7fc23959a640 1 --2- 172.21.15.42:0/1717680761 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc21c00cbf0 0x7fc224005280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:46:28.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.500+0000 7fc23959a640 10 monclient(hunting): picked mon.c con 0x7fc21c00cbf0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:46:28.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.500+0000 7fc23959a640 10 monclient(hunting): start opening mon connection 2024-09-16T03:46:28.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.500+0000 7fc23959a640 1 -- 172.21.15.42:0/1717680761 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fc23c09c7d0 con 0x7fc2240039c0 2024-09-16T03:46:28.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.500+0000 7fc23959a640 10 monclient(hunting): start opening mon connection 2024-09-16T03:46:28.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.500+0000 7fc23959a640 1 -- 172.21.15.42:0/1717680761 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc23c09c090 con 0x7fc224004810 2024-09-16T03:46:28.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.500+0000 7fc23a59c640 1 -- 172.21.15.42:0/1717680761 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc2240039c0 msgr2=0x7fc224003de0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3300/0 2024-09-16T03:46:28.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.500+0000 7fc23959a640 10 monclient(hunting): start opening mon connection 2024-09-16T03:46:28.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.500+0000 7fc23959a640 1 -- 172.21.15.42:0/1717680761 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fc22400bc10 con 0x7fc21c00cbf0 2024-09-16T03:46:28.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.500+0000 7fc23959a640 10 monclient(hunting): _renew_subs 2024-09-16T03:46:28.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.500+0000 7fc23a59c640 1 --2- 172.21.15.42:0/1717680761 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc2240039c0 0x7fc224003de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:46:28.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.500+0000 7fc23959a640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:46:28.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.500+0000 7fc23ad9d640 1 --2- 172.21.15.42:0/1717680761 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc21c00cbf0 0x7fc224005280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:46:28.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.500+0000 7fc23ad9d640 10 monclient(hunting): get_auth_request con 0x7fc21c00cbf0 auth_method 0 2024-09-16T03:46:28.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.500+0000 7fc23ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:46:28.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.500+0000 7fc23ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:46:28.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.500+0000 7fc23ad9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:46:28.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.500+0000 7fc239d9b640 1 --2- 172.21.15.42:0/1717680761 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc224004810 0x7fc224004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:46:28.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.500+0000 7fc23ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:46:28.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.500+0000 7fc23ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:46:28.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.500+0000 7fc23ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T03:46:28.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.500+0000 7fc239d9b640 10 monclient(hunting): get_auth_request con 0x7fc224004810 auth_method 0 2024-09-16T03:46:28.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.500+0000 7fc239d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:46:28.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.500+0000 7fc239d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:46:28.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.500+0000 7fc239d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:46:28.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.500+0000 7fc23ad9d640 10 monclient(hunting): handle_auth_done global_id 754156 payload 293 2024-09-16T03:46:28.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.500+0000 7fc23ad9d640 1 -- 172.21.15.42:0/1717680761 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc224004810 msgr2=0x7fc224004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:46:28.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.500+0000 7fc23ad9d640 1 --2- 172.21.15.42:0/1717680761 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc224004810 0x7fc224004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:46:28.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.500+0000 7fc23ad9d640 1 -- 172.21.15.42:0/1717680761 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc2240039c0 msgr2=0x7fc224003de0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:46:28.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.500+0000 7fc23ad9d640 1 --2- 172.21.15.42:0/1717680761 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc2240039c0 0x7fc224003de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:46:28.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.500+0000 7fc23ad9d640 10 monclient: _finish_hunting 0 2024-09-16T03:46:28.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.500+0000 7fc23ad9d640 1 monclient: found mon.c 2024-09-16T03:46:28.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.500+0000 7fc23ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:46:28.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.500+0000 7fc23ad9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:46:28.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.500+0000 7fc23ad9d640 1 -- 172.21.15.42:0/1717680761 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc224003530 con 0x7fc21c00cbf0 2024-09-16T03:46:28.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.500+0000 7fc23ad9d640 10 monclient: _finish_auth 0 2024-09-16T03:46:28.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.500+0000 7fc23ad9d640 10 monclient: _check_auth_tickets 2024-09-16T03:46:28.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.500+0000 7fc23ad9d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:46:28.513 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.500+0000 7fc240eac640 5 monclient: authenticate success, global_id 754156 2024-09-16T03:46:28.513 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.500+0000 7fc240eac640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:46:28.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.504+0000 7fc23ad9d640 1 --2- 172.21.15.42:0/1717680761 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc21c00cbf0 0x7fc224005280 secure :-1 s=READY pgs=2447 cs=0 l=1 rev1=1 crypto rx=0x7fc23c0a2190 tx=0x7fc234022a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3fbd173a2f8c6b17 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:46:28.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.504+0000 7fc23959a640 1 -- 172.21.15.42:0/1717680761 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2340b1070 con 0x7fc21c00cbf0 2024-09-16T03:46:28.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.504+0000 7fc23959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:46:28.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.504+0000 7fc23959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:46:28.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.504+0000 7fc23959a640 10 monclient: dump: 2024-09-16T03:46:28.518 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:46:28.518 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:46:28.518 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:28.518 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:28.518 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:46:28.518 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:46:28.518 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:46:28.519 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:46:28.519 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:46:28.519 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:46:28.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.504+0000 7fc23959a640 1 -- 172.21.15.42:0/1717680761 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2340ab7e0 con 0x7fc21c00cbf0 2024-09-16T03:46:28.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.504+0000 7fc240eac640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:46:28.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.504+0000 7fc23959a640 10 monclient: handle_config config(0 keys) 2024-09-16T03:46:28.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.504+0000 7fc240eac640 10 monclient: get_monmap_and_config success 2024-09-16T03:46:28.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.504+0000 7fc240eac640 4 set_mon_vals no callback set 2024-09-16T03:46:28.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.504+0000 7fc23959a640 1 -- 172.21.15.42:0/1717680761 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc234009b30 con 0x7fc21c00cbf0 2024-09-16T03:46:28.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.504+0000 7fc23959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:46:28.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.504+0000 7fc23959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:46:28.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.504+0000 7fc23959a640 10 monclient: dump: 2024-09-16T03:46:28.519 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:46:28.519 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:46:28.519 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:28.520 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:28.520 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:46:28.520 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:46:28.520 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:46:28.520 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:46:28.520 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:46:28.520 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:46:28.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.504+0000 7fc240eac640 10 monclient: shutdown 2024-09-16T03:46:28.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.504+0000 7fc240eac640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:46:28.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.504+0000 7fc240eac640 1 -- 172.21.15.42:0/1717680761 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc21c00cbf0 msgr2=0x7fc224005280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:46:28.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.504+0000 7fc240eac640 1 --2- 172.21.15.42:0/1717680761 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc21c00cbf0 0x7fc224005280 secure :-1 s=READY pgs=2447 cs=0 l=1 rev1=1 crypto rx=0x7fc23c0a2190 tx=0x7fc234022a40 comp rx=0 tx=0).stop 2024-09-16T03:46:28.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.504+0000 7fc23ad9d640 1 -- 172.21.15.42:0/1717680761 reap_dead start 2024-09-16T03:46:28.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.504+0000 7fc240eac640 1 -- 172.21.15.42:0/1717680761 shutdown_connections 2024-09-16T03:46:28.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.504+0000 7fc240eac640 1 -- 172.21.15.42:0/1717680761 >> 172.21.15.42:0/1717680761 conn(0x7fc23c0a2420 msgr2=0x7fc23c0a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:46:28.521 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.504+0000 7fc240eac640 1 -- 172.21.15.42:0/1717680761 shutdown_connections 2024-09-16T03:46:28.521 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.504+0000 7fc240eac640 1 -- 172.21.15.42:0/1717680761 wait complete. 2024-09-16T03:46:28.521 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.504+0000 7fc240eac640 10 monclient: build_initial_monmap 2024-09-16T03:46:28.521 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.504+0000 7fc240eac640 10 monclient: monmap: 2024-09-16T03:46:28.521 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:46:28.521 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:46:28.521 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:46:28.521 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:46:28.521 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:46:28.521 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:46:28.521 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:46:28.521 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:46:28.521 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:46:28.522 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:46:28.522 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.508+0000 7fc240eac640 1 librados: starting msgr at 2024-09-16T03:46:28.522 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.508+0000 7fc240eac640 1 librados: starting objecter 2024-09-16T03:46:28.522 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.508+0000 7fc240eac640 1 Processor -- start 2024-09-16T03:46:28.522 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.508+0000 7fc240eac640 1 -- start start 2024-09-16T03:46:28.522 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.508+0000 7fc240eac640 1 librados: setting wanted keys 2024-09-16T03:46:28.522 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.508+0000 7fc240eac640 1 librados: calling monclient init 2024-09-16T03:46:28.522 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.508+0000 7fc240eac640 10 monclient: init 2024-09-16T03:46:28.522 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.508+0000 7fc240eac640 10 monclient: _reopen_session rank -1 2024-09-16T03:46:28.522 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.508+0000 7fc240eac640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T03:46:28.522 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.508+0000 7fc240eac640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc21c00cbf0 0x7fc23c1cb390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:46:28.522 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.508+0000 7fc240eac640 10 monclient(hunting): picked mon.noname-a con 0x7fc21c00cbf0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:46:28.522 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.508+0000 7fc240eac640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc23c1cb8d0 0x7fc23c1229e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:46:28.522 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.508+0000 7fc240eac640 10 monclient(hunting): picked mon.noname-b con 0x7fc23c1cb8d0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:46:28.522 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.508+0000 7fc240eac640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc23c1cbcd0 0x7fc23c122f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:46:28.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.508+0000 7fc240eac640 10 monclient(hunting): picked mon.noname-c con 0x7fc23c1cbcd0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:46:28.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.508+0000 7fc240eac640 10 monclient(hunting): start opening mon connection 2024-09-16T03:46:28.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.508+0000 7fc240eac640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fc23c1358a0 con 0x7fc21c00cbf0 2024-09-16T03:46:28.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.508+0000 7fc240eac640 10 monclient(hunting): start opening mon connection 2024-09-16T03:46:28.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.508+0000 7fc240eac640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc23c135720 con 0x7fc23c1cb8d0 2024-09-16T03:46:28.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.508+0000 7fc240eac640 10 monclient(hunting): start opening mon connection 2024-09-16T03:46:28.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.508+0000 7fc240eac640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fc23c135a20 con 0x7fc23c1cbcd0 2024-09-16T03:46:28.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.508+0000 7fc240eac640 10 monclient(hunting): _renew_subs 2024-09-16T03:46:28.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.508+0000 7fc240eac640 10 monclient(hunting): authenticate will time out at 1235.481447s 2024-09-16T03:46:28.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.508+0000 7fc23a59c640 1 -- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc21c00cbf0 msgr2=0x7fc23c1cb390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.42:3300/0 2024-09-16T03:46:28.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.508+0000 7fc23a59c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc21c00cbf0 0x7fc23c1cb390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:46:28.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.508+0000 7fc23ad9d640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc23c1cbcd0 0x7fc23c122f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:46:28.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.508+0000 7fc23ad9d640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc23c1cbcd0 0x7fc23c122f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:39904/0 (socket says 172.21.15.42:39904) 2024-09-16T03:46:28.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.508+0000 7fc23ad9d640 1 -- 172.21.15.42:0/2867451538 learned_addr learned my addr 172.21.15.42:0/2867451538 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:46:28.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.508+0000 7fc23ad9d640 10 monclient(hunting): get_auth_request con 0x7fc23c1cbcd0 auth_method 0 2024-09-16T03:46:28.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.508+0000 7fc23ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:46:28.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.508+0000 7fc23ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:46:28.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.508+0000 7fc23ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:46:28.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.508+0000 7fc239d9b640 1 --2- 172.21.15.42:0/2867451538 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc23c1cb8d0 0x7fc23c1229e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:46:28.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.508+0000 7fc239d9b640 10 monclient(hunting): get_auth_request con 0x7fc23c1cb8d0 auth_method 0 2024-09-16T03:46:28.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.508+0000 7fc239d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:46:28.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.508+0000 7fc239d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:46:28.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.508+0000 7fc239d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:46:28.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.508+0000 7fc23ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:46:28.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.508+0000 7fc23ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:46:28.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.508+0000 7fc23ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:46:28.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.508+0000 7fc239d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:46:28.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.508+0000 7fc239d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:46:28.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.508+0000 7fc239d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:46:28.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.508+0000 7fc23ad9d640 10 monclient(hunting): handle_auth_done global_id 13598 payload 824 2024-09-16T03:46:28.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.508+0000 7fc23ad9d640 1 -- 172.21.15.42:0/2867451538 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc23c1cb8d0 msgr2=0x7fc23c1229e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:46:28.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.508+0000 7fc23ad9d640 1 --2- 172.21.15.42:0/2867451538 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc23c1cb8d0 0x7fc23c1229e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:46:28.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.508+0000 7fc23ad9d640 1 -- 172.21.15.42:0/2867451538 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc21c00cbf0 msgr2=0x7fc23c1cb390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:46:28.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.508+0000 7fc23ad9d640 1 --2- 172.21.15.42:0/2867451538 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc21c00cbf0 0x7fc23c1cb390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:46:28.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.508+0000 7fc23ad9d640 10 monclient: _finish_hunting 0 2024-09-16T03:46:28.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.508+0000 7fc23ad9d640 1 monclient: found mon.noname-c 2024-09-16T03:46:28.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.508+0000 7fc23ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:46:28.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.508+0000 7fc23ad9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:46:28.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.508+0000 7fc23ad9d640 1 -- 172.21.15.42:0/2867451538 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc23c1263b0 con 0x7fc23c1cbcd0 2024-09-16T03:46:28.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.508+0000 7fc23ad9d640 1 --2- 172.21.15.42:0/2867451538 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc23c1cbcd0 0x7fc23c122f20 secure :-1 s=READY pgs=2448 cs=0 l=1 rev1=1 crypto rx=0x7fc2340b7040 tx=0x7fc23c0a22d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:46:28.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.508+0000 7fc22affd640 10 client.?.objecter ms_handle_connect 0x7fc23c1cbcd0 2024-09-16T03:46:28.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.508+0000 7fc22affd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:46:28.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.512+0000 7fc22affd640 1 -- 172.21.15.42:0/2867451538 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2340b1040 con 0x7fc23c1cbcd0 2024-09-16T03:46:28.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.512+0000 7fc22affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:46:28.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.512+0000 7fc22affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:46:28.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.512+0000 7fc22affd640 10 monclient: dump: 2024-09-16T03:46:28.526 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:46:28.526 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:46:28.526 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:28.526 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:28.526 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:46:28.526 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:46:28.526 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:46:28.526 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:46:28.526 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:46:28.526 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:46:28.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.512+0000 7fc22affd640 10 monclient: _finish_auth 0 2024-09-16T03:46:28.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.512+0000 7fc22affd640 10 monclient: _check_auth_tickets 2024-09-16T03:46:28.527 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.512+0000 7fc22affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:46:28.527 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.512+0000 7fc22affd640 1 -- 172.21.15.42:0/2867451538 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2340ab5e0 con 0x7fc23c1cbcd0 2024-09-16T03:46:28.527 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.512+0000 7fc240eac640 5 monclient: authenticate success, global_id 13598 2024-09-16T03:46:28.527 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.512+0000 7fc22affd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:46:28.527 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.512+0000 7fc22affd640 1 -- 172.21.15.42:0/2867451538 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc234009b30 con 0x7fc23c1cbcd0 2024-09-16T03:46:28.527 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.512+0000 7fc240eac640 10 monclient: _renew_subs 2024-09-16T03:46:28.527 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.512+0000 7fc240eac640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:46:28.527 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.512+0000 7fc240eac640 1 -- 172.21.15.42:0/2867451538 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc23c126690 con 0x7fc23c1cbcd0 2024-09-16T03:46:28.527 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.512+0000 7fc238d99640 4 set_mon_vals no callback set 2024-09-16T03:46:28.527 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.512+0000 7fc22affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:46:28.527 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.512+0000 7fc22affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:46:28.527 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.512+0000 7fc22affd640 10 monclient: dump: 2024-09-16T03:46:28.527 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:46:28.528 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:46:28.528 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:28.528 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:28.528 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:46:28.528 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:46:28.528 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:46:28.528 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:46:28.528 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:46:28.528 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:46:28.528 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.512+0000 7fc240eac640 10 client.13598.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:46:28.528 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.512+0000 7fc240eac640 10 monclient: _renew_subs 2024-09-16T03:46:28.528 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.512+0000 7fc240eac640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:46:28.529 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.512+0000 7fc240eac640 1 -- 172.21.15.42:0/2867451538 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc23c1260d0 con 0x7fc23c1cbcd0 2024-09-16T03:46:28.529 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.512+0000 7fc240eac640 1 librados: init done 2024-09-16T03:46:28.530 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.512+0000 7fc240eac640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:46:28.530 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.512+0000 7fc240eac640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:46:28.530 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.512+0000 7fc240eac640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:46:28.530 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.512+0000 7fc240eac640 1 -- 172.21.15.42:0/2867451538 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc23c09f610 con 0x7fc23c1cbcd0 2024-09-16T03:46:28.530 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.512+0000 7fc22affd640 1 -- 172.21.15.42:0/2867451538 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fc2340c6020 con 0x7fc23c1cbcd0 2024-09-16T03:46:28.530 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.512+0000 7fc22affd640 1 --2- 172.21.15.42:0/2867451538 >> v2:172.21.15.154:6812/254845342 conn(0x7fc21c038d50 0x7fc21c03b210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:46:28.531 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.512+0000 7fc22affd640 1 -- 172.21.15.42:0/2867451538 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(123..123 src has 1..123) ==== 6278+0+0 (secure 0 0 0) 0x7fc2340e91d0 con 0x7fc23c1cbcd0 2024-09-16T03:46:28.531 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.512+0000 7fc22affd640 10 client.13598.objecter ms_dispatch 0x7fc23c000ba0 osd_map(123..123 src has 1..123) 2024-09-16T03:46:28.531 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.512+0000 7fc22affd640 3 client.13598.objecter handle_osd_map got epochs [123,123] > 0 2024-09-16T03:46:28.531 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.512+0000 7fc22affd640 3 client.13598.objecter handle_osd_map decoding full epoch 123 2024-09-16T03:46:28.531 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.512+0000 7fc22affd640 20 client.13598.objecter dump_active .. 0 homeless 2024-09-16T03:46:28.531 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.512+0000 7fc23a59c640 1 --2- 172.21.15.42:0/2867451538 >> v2:172.21.15.154:6812/254845342 conn(0x7fc21c038d50 0x7fc21c03b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:46:28.531 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.512+0000 7fc23a59c640 10 monclient: get_auth_request con 0x7fc21c038d50 auth_method 0 2024-09-16T03:46:28.531 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.512+0000 7fc22affd640 10 client.13598.objecter ms_handle_connect 0x7fc21c038d50 2024-09-16T03:46:28.531 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.512+0000 7fc23a59c640 1 --2- 172.21.15.42:0/2867451538 >> v2:172.21.15.154:6812/254845342 conn(0x7fc21c038d50 0x7fc21c03b210 secure :-1 s=READY pgs=3410 cs=0 l=1 rev1=1 crypto rx=0x7fc230009100 tx=0x7fc230008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:46:28.531 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.520+0000 7fc22affd640 1 -- 172.21.15.42:0/2867451538 <== mon.2 v2:172.21.15.42:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7fc234010040 con 0x7fc23c1cbcd0 2024-09-16T03:46:28.531 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.520+0000 7fc22affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:46:28.531 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.520+0000 7fc22affd640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:46:28.769 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.764+0000 7fc240eac640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:46:28.769 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.764+0000 7fc240eac640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:46:28.769 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.764+0000 7fc240eac640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:46:28.769 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.764+0000 7fc240eac640 1 -- 172.21.15.42:0/2867451538 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fc23c1268d0 con 0x7fc23c1cbcd0 2024-09-16T03:46:28.770 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.764+0000 7fc22affd640 1 -- 172.21.15.42:0/2867451538 <== mon.2 v2:172.21.15.42:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1554 (secure 0 0 0) 0x7fc2340ab7b0 con 0x7fc23c1cbcd0 2024-09-16T03:46:28.770 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.764+0000 7fc22affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:46:28.770 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.764+0000 7fc22affd640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:46:28.770 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":36,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:46:28.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.764+0000 7fc240eac640 10 librados: watch_flush enter 2024-09-16T03:46:28.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.764+0000 7fc240eac640 10 librados: watch_flush exit 2024-09-16T03:46:28.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.764+0000 7fc240eac640 20 client.13598.objecter shutdown clearing up homeless session... 2024-09-16T03:46:28.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.764+0000 7fc240eac640 10 client.13598.objecter successfully canceled tick 2024-09-16T03:46:28.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.764+0000 7fc240eac640 1 -- 172.21.15.42:0/2867451538 >> v2:172.21.15.154:6812/254845342 conn(0x7fc21c038d50 msgr2=0x7fc21c03b210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:46:28.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.764+0000 7fc240eac640 1 --2- 172.21.15.42:0/2867451538 >> v2:172.21.15.154:6812/254845342 conn(0x7fc21c038d50 0x7fc21c03b210 secure :-1 s=READY pgs=3410 cs=0 l=1 rev1=1 crypto rx=0x7fc230009100 tx=0x7fc230008040 comp rx=0 tx=0).stop 2024-09-16T03:46:28.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.764+0000 7fc240eac640 10 monclient: shutdown 2024-09-16T03:46:28.773 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.764+0000 7fc240eac640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:46:28.773 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.764+0000 7fc240eac640 1 -- 172.21.15.42:0/2867451538 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc23c1cbcd0 msgr2=0x7fc23c122f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:46:28.773 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.764+0000 7fc240eac640 1 --2- 172.21.15.42:0/2867451538 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc23c1cbcd0 0x7fc23c122f20 secure :-1 s=READY pgs=2448 cs=0 l=1 rev1=1 crypto rx=0x7fc2340b7040 tx=0x7fc23c0a22d0 comp rx=0 tx=0).stop 2024-09-16T03:46:28.773 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.764+0000 7fc240eac640 1 -- 172.21.15.42:0/2867451538 shutdown_connections 2024-09-16T03:46:28.773 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.764+0000 7fc240eac640 1 --2- 172.21.15.42:0/2867451538 >> v2:172.21.15.154:6812/254845342 conn(0x7fc21c038d50 0x7fc21c03b210 unknown :-1 s=CLOSED pgs=3410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:46:28.773 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.764+0000 7fc240eac640 1 --2- 172.21.15.42:0/2867451538 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc23c1cbcd0 0x7fc23c122f20 unknown :-1 s=CLOSED pgs=2448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:46:28.773 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.764+0000 7fc240eac640 1 --2- 172.21.15.42:0/2867451538 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc23c1cb8d0 0x7fc23c1229e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:46:28.773 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.764+0000 7fc240eac640 1 --2- 172.21.15.42:0/2867451538 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc21c00cbf0 0x7fc23c1cb390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:46:28.773 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.764+0000 7fc240eac640 1 -- 172.21.15.42:0/2867451538 >> 172.21.15.42:0/2867451538 conn(0x7fc23c0a2420 msgr2=0x7fc23c06f6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:46:28.773 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.764+0000 7fc240eac640 1 -- 172.21.15.42:0/2867451538 shutdown_connections 2024-09-16T03:46:28.773 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.764+0000 7fc240eac640 1 -- 172.21.15.42:0/2867451538 wait complete. 2024-09-16T03:46:28.773 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.764+0000 7fc240eac640 1 librados: shutdown 2024-09-16T03:46:28.790 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T03:46:28.790 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-16T03:46:28.790 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T03:46:28.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.904+0000 7fd274867640 10 monclient: get_monmap_and_config 2024-09-16T03:46:28.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.904+0000 7fd274867640 10 monclient: build_initial_monmap 2024-09-16T03:46:28.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.904+0000 7fd274867640 10 monclient: monmap: 2024-09-16T03:46:28.910 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:46:28.910 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:46:28.910 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:46:28.908816+0000 2024-09-16T03:46:28.910 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:46:28.908816+0000 2024-09-16T03:46:28.910 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:46:28.910 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:46:28.910 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:46:28.910 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:46:28.910 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:46:28.910 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:46:28.910 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:46:28.910 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:46:28.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.904+0000 7fd274867640 1 Processor -- start 2024-09-16T03:46:28.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.904+0000 7fd274867640 1 -- start start 2024-09-16T03:46:28.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.904+0000 7fd274867640 10 monclient: init 2024-09-16T03:46:28.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.904+0000 7fd274867640 10 monclient: _reopen_session rank -1 2024-09-16T03:46:28.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.904+0000 7fd274867640 10 monclient: _add_conns ranks=[0,4,1,2,3] 2024-09-16T03:46:28.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.904+0000 7fd274867640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd2700a39b0 addr v1:172.21.15.42:6789/0 2024-09-16T03:46:28.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.904+0000 7fd274867640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fd2700a3d90 0x7fd27013c380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:46:28.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.904+0000 7fd274867640 10 monclient(hunting): picked mon.noname-c con 0x7fd2700a3d90 addr v2:172.21.15.154:3300/0 2024-09-16T03:46:28.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.904+0000 7fd274867640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd27013c8c0 addr v1:172.21.15.154:6789/0 2024-09-16T03:46:28.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.904+0000 7fd274867640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd270134290 con 0x7fd2700a39b0 2024-09-16T03:46:28.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.904+0000 7fd274867640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd27009c140 con 0x7fd27013c8c0 2024-09-16T03:46:28.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.904+0000 7fd274867640 10 monclient(hunting): start opening mon connection 2024-09-16T03:46:28.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.904+0000 7fd274867640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fd27009c420 con 0x7fd2700a3d90 2024-09-16T03:46:28.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.904+0000 7fd274867640 10 monclient(hunting): _renew_subs 2024-09-16T03:46:28.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.904+0000 7fd274867640 10 monclient(hunting): authenticate will time out at 1235.878442s 2024-09-16T03:46:28.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.904+0000 7fd26e59c640 1 -- >> v1:172.21.15.42:6789/0 conn(0x7fd2700a39b0 legacy=0x7fd270133b80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.42:6789/0 2024-09-16T03:46:28.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.904+0000 7fd26dd9b640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fd2700a3d90 0x7fd27013c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:46:28.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.904+0000 7fd26dd9b640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fd2700a3d90 0x7fd27013c380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.42:36316/0 (socket says 172.21.15.42:36316) 2024-09-16T03:46:28.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.904+0000 7fd26dd9b640 1 -- 172.21.15.42:0/3701167166 learned_addr learned my addr 172.21.15.42:0/3701167166 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:46:28.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.904+0000 7fd26dd9b640 10 monclient(hunting): get_auth_request con 0x7fd2700a3d90 auth_method 0 2024-09-16T03:46:28.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.904+0000 7fd26dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:46:28.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.904+0000 7fd26dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:46:28.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.904+0000 7fd26dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:46:28.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.904+0000 7fd26ed9d640 1 --1- 172.21.15.42:0/3701167166 >> v1:172.21.15.154:6789/0 conn(0x7fd27013c8c0 0x7fd27013ecb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.42:57704/0 (socket says 172.21.15.42:57704) 2024-09-16T03:46:28.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.904+0000 7fd26dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:46:28.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.904+0000 7fd26dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:46:28.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.904+0000 7fd26dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:46:28.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.904+0000 7fd26dd9b640 10 monclient(hunting): handle_auth_done global_id 754162 payload 274 2024-09-16T03:46:28.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.904+0000 7fd26dd9b640 1 -- 172.21.15.42:0/3701167166 >> v1:172.21.15.154:6789/0 conn(0x7fd27013c8c0 legacy=0x7fd27013ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:46:28.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.904+0000 7fd26dd9b640 1 -- 172.21.15.42:0/3701167166 >> v1:172.21.15.42:6789/0 conn(0x7fd2700a39b0 legacy=0x7fd270133b80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:46:28.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.904+0000 7fd26dd9b640 10 monclient: _finish_hunting 0 2024-09-16T03:46:28.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.904+0000 7fd26dd9b640 1 monclient: found mon.noname-c 2024-09-16T03:46:28.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.904+0000 7fd26dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:46:28.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.904+0000 7fd26dd9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.154:3300/0 2024-09-16T03:46:28.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.904+0000 7fd26dd9b640 1 -- 172.21.15.42:0/3701167166 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd27006dd60 con 0x7fd2700a3d90 2024-09-16T03:46:28.917 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.908+0000 7fd26dd9b640 1 --2- 172.21.15.42:0/3701167166 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd2700a3d90 0x7fd27013c380 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fd264009870 tx=0x7fd26401c3e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=58591ed02488fafe server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:46:28.917 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.908+0000 7fd26d59a640 1 -- 172.21.15.42:0/3701167166 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2640b0070 con 0x7fd2700a3d90 2024-09-16T03:46:28.917 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.908+0000 7fd26d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:46:28.917 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.908+0000 7fd26d59a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:46:28.917 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.908+0000 7fd26d59a640 10 monclient: dump: 2024-09-16T03:46:28.917 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:46:28.917 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:46:28.918 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:28.918 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:28.918 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:46:28.918 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:46:28.918 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:46:28.918 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:46:28.918 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:46:28.918 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:46:28.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.908+0000 7fd26d59a640 10 monclient: _finish_auth 0 2024-09-16T03:46:28.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.908+0000 7fd26d59a640 10 monclient: _check_auth_tickets 2024-09-16T03:46:28.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.908+0000 7fd26d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:46:28.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.908+0000 7fd26d59a640 1 -- 172.21.15.42:0/3701167166 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2640ab600 con 0x7fd2700a3d90 2024-09-16T03:46:28.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.908+0000 7fd26d59a640 10 monclient: handle_config config(0 keys) 2024-09-16T03:46:28.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.908+0000 7fd26d59a640 1 -- 172.21.15.42:0/3701167166 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2640ab9d0 con 0x7fd2700a3d90 2024-09-16T03:46:28.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.908+0000 7fd274867640 5 monclient: authenticate success, global_id 754162 2024-09-16T03:46:28.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.908+0000 7fd274867640 10 monclient: get_monmap_and_config success 2024-09-16T03:46:28.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.908+0000 7fd274867640 4 set_mon_vals no callback set 2024-09-16T03:46:28.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.908+0000 7fd26d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:46:28.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.908+0000 7fd26d59a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:46:28.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.908+0000 7fd26d59a640 10 monclient: dump: 2024-09-16T03:46:28.919 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:46:28.919 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:46:28.919 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:28.919 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:28.919 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:46:28.919 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:46:28.919 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:46:28.920 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:46:28.920 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:46:28.920 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:46:28.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.908+0000 7fd274867640 10 monclient: shutdown 2024-09-16T03:46:28.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.908+0000 7fd274867640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:46:28.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.908+0000 7fd274867640 1 -- 172.21.15.42:0/3701167166 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd2700a3d90 msgr2=0x7fd27013c380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:46:28.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.908+0000 7fd274867640 1 --2- 172.21.15.42:0/3701167166 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd2700a3d90 0x7fd27013c380 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fd264009870 tx=0x7fd26401c3e0 comp rx=0 tx=0).stop 2024-09-16T03:46:28.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.908+0000 7fd274867640 1 -- 172.21.15.42:0/3701167166 shutdown_connections 2024-09-16T03:46:28.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.908+0000 7fd274867640 1 --2- 172.21.15.42:0/3701167166 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd2700a3d90 0x7fd27013c380 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:46:28.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.908+0000 7fd274867640 1 -- 172.21.15.42:0/3701167166 >> 172.21.15.42:0/3701167166 conn(0x7fd2700a9fb0 msgr2=0x7fd2700aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:46:28.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.908+0000 7fd274867640 1 -- 172.21.15.42:0/3701167166 shutdown_connections 2024-09-16T03:46:28.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.908+0000 7fd274867640 1 -- 172.21.15.42:0/3701167166 wait complete. 2024-09-16T03:46:28.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.912+0000 7fd274867640 10 monclient: build_initial_monmap 2024-09-16T03:46:28.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.912+0000 7fd274867640 10 monclient: monmap: 2024-09-16T03:46:28.921 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:46:28.921 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:46:28.921 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:46:28.921 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:46:28.921 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:46:28.921 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:46:28.921 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:46:28.921 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:46:28.921 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:46:28.921 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:46:28.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.912+0000 7fd274867640 1 librados: starting msgr at 2024-09-16T03:46:28.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.912+0000 7fd274867640 1 librados: starting objecter 2024-09-16T03:46:28.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.912+0000 7fd274867640 1 Processor -- start 2024-09-16T03:46:28.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.912+0000 7fd274867640 1 -- start start 2024-09-16T03:46:28.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.912+0000 7fd274867640 1 librados: setting wanted keys 2024-09-16T03:46:28.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.912+0000 7fd274867640 1 librados: calling monclient init 2024-09-16T03:46:28.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.912+0000 7fd274867640 10 monclient: init 2024-09-16T03:46:28.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.912+0000 7fd274867640 10 monclient: _reopen_session rank -1 2024-09-16T03:46:28.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.912+0000 7fd274867640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:46:28.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.912+0000 7fd274867640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd2700a39b0 0x7fd27012c970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:46:28.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.912+0000 7fd274867640 10 monclient(hunting): picked mon.noname-c con 0x7fd2700a39b0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:46:28.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.912+0000 7fd274867640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd2700a3d90 0x7fd2701393b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:46:28.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.912+0000 7fd274867640 10 monclient(hunting): picked mon.noname-b con 0x7fd2700a3d90 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:46:28.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.912+0000 7fd274867640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd27013c8c0 0x7fd2701398f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:46:28.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.912+0000 7fd274867640 10 monclient(hunting): picked mon.noname-a con 0x7fd27013c8c0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:46:28.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.912+0000 7fd274867640 10 monclient(hunting): start opening mon connection 2024-09-16T03:46:28.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.912+0000 7fd274867640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fd270142f80 con 0x7fd27013c8c0 2024-09-16T03:46:28.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.912+0000 7fd274867640 10 monclient(hunting): start opening mon connection 2024-09-16T03:46:28.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.912+0000 7fd274867640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd270142e00 con 0x7fd2700a3d90 2024-09-16T03:46:28.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.912+0000 7fd274867640 10 monclient(hunting): start opening mon connection 2024-09-16T03:46:28.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.912+0000 7fd274867640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fd270143100 con 0x7fd2700a39b0 2024-09-16T03:46:28.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.912+0000 7fd274867640 10 monclient(hunting): _renew_subs 2024-09-16T03:46:28.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.912+0000 7fd274867640 10 monclient(hunting): authenticate will time out at 1235.885216s 2024-09-16T03:46:28.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.912+0000 7fd26ed9d640 1 -- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd27013c8c0 msgr2=0x7fd2701398f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.42:3300/0 2024-09-16T03:46:28.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.912+0000 7fd26ed9d640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd27013c8c0 0x7fd2701398f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:46:28.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.912+0000 7fd26e59c640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd2700a39b0 0x7fd27012c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:46:28.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.912+0000 7fd26e59c640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd2700a39b0 0x7fd27012c970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:39918/0 (socket says 172.21.15.42:39918) 2024-09-16T03:46:28.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.912+0000 7fd26e59c640 1 -- 172.21.15.42:0/4023144989 learned_addr learned my addr 172.21.15.42:0/4023144989 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:46:28.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.912+0000 7fd26e59c640 10 monclient(hunting): get_auth_request con 0x7fd2700a39b0 auth_method 0 2024-09-16T03:46:28.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.912+0000 7fd26e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:46:28.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.912+0000 7fd26e59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:46:28.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.912+0000 7fd26e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:46:28.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.912+0000 7fd26e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:46:28.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.912+0000 7fd26e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:46:28.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.912+0000 7fd26e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:46:28.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.912+0000 7fd26dd9b640 1 --2- 172.21.15.42:0/4023144989 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd2700a3d90 0x7fd2701393b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:46:28.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.912+0000 7fd26dd9b640 10 monclient(hunting): get_auth_request con 0x7fd2700a3d90 auth_method 0 2024-09-16T03:46:28.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.912+0000 7fd26dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:46:28.929 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.912+0000 7fd26dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:46:28.929 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.912+0000 7fd26dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:46:28.929 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.912+0000 7fd26e59c640 10 monclient(hunting): handle_auth_done global_id 13601 payload 824 2024-09-16T03:46:28.929 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.912+0000 7fd26e59c640 1 -- 172.21.15.42:0/4023144989 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd2700a3d90 msgr2=0x7fd2701393b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:46:28.929 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.912+0000 7fd26e59c640 1 --2- 172.21.15.42:0/4023144989 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd2700a3d90 0x7fd2701393b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:46:28.929 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.912+0000 7fd26e59c640 1 -- 172.21.15.42:0/4023144989 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd27013c8c0 msgr2=0x7fd2701398f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:46:28.929 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.912+0000 7fd26e59c640 1 --2- 172.21.15.42:0/4023144989 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd27013c8c0 0x7fd2701398f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:46:28.929 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.912+0000 7fd26e59c640 10 monclient: _finish_hunting 0 2024-09-16T03:46:28.929 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.912+0000 7fd26e59c640 1 monclient: found mon.noname-c 2024-09-16T03:46:28.929 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.912+0000 7fd26e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:46:28.929 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.912+0000 7fd26e59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:46:28.929 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.912+0000 7fd26e59c640 1 -- 172.21.15.42:0/4023144989 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd27012e330 con 0x7fd2700a39b0 2024-09-16T03:46:28.929 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.912+0000 7fd26e59c640 1 --2- 172.21.15.42:0/4023144989 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd2700a39b0 0x7fd27012c970 secure :-1 s=READY pgs=2449 cs=0 l=1 rev1=1 crypto rx=0x7fd26000b560 tx=0x7fd26000ba20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:46:28.929 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.912+0000 7fd25effd640 10 client.?.objecter ms_handle_connect 0x7fd2700a39b0 2024-09-16T03:46:28.930 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.912+0000 7fd25effd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:46:28.930 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.912+0000 7fd25effd640 1 -- 172.21.15.42:0/4023144989 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd260013020 con 0x7fd2700a39b0 2024-09-16T03:46:28.930 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.912+0000 7fd25effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:46:28.930 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.912+0000 7fd25effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:46:28.930 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.912+0000 7fd25effd640 10 monclient: dump: 2024-09-16T03:46:28.930 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:46:28.930 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:46:28.930 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:28.930 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:28.930 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:46:28.930 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:46:28.930 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:46:28.930 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:46:28.930 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:46:28.931 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:46:28.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.912+0000 7fd25effd640 10 monclient: _finish_auth 0 2024-09-16T03:46:28.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.912+0000 7fd25effd640 10 monclient: _check_auth_tickets 2024-09-16T03:46:28.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.912+0000 7fd25effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:46:28.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.912+0000 7fd25effd640 1 -- 172.21.15.42:0/4023144989 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd260003df0 con 0x7fd2700a39b0 2024-09-16T03:46:28.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.912+0000 7fd25effd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:46:28.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.912+0000 7fd25effd640 1 -- 172.21.15.42:0/4023144989 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2600052b0 con 0x7fd2700a39b0 2024-09-16T03:46:28.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.912+0000 7fd25effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:46:28.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.912+0000 7fd25effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:46:28.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.912+0000 7fd25effd640 10 monclient: dump: 2024-09-16T03:46:28.931 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:46:28.931 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:46:28.931 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:28.932 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:28.932 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:46:28.932 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:46:28.932 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:46:28.932 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:46:28.932 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:46:28.932 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:46:28.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.912+0000 7fd274867640 5 monclient: authenticate success, global_id 13601 2024-09-16T03:46:28.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.912+0000 7fd274867640 10 monclient: _renew_subs 2024-09-16T03:46:28.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.912+0000 7fd274867640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:46:28.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.912+0000 7fd274867640 1 -- 172.21.15.42:0/4023144989 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd27006f680 con 0x7fd2700a39b0 2024-09-16T03:46:28.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.912+0000 7fd26cd99640 4 set_mon_vals no callback set 2024-09-16T03:46:28.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.912+0000 7fd274867640 10 client.13601.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:46:28.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.912+0000 7fd274867640 10 monclient: _renew_subs 2024-09-16T03:46:28.933 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.912+0000 7fd274867640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:46:28.933 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.912+0000 7fd274867640 1 -- 172.21.15.42:0/4023144989 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd27012dd70 con 0x7fd2700a39b0 2024-09-16T03:46:28.933 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.912+0000 7fd274867640 1 librados: init done 2024-09-16T03:46:28.933 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.912+0000 7fd274867640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:46:28.933 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.912+0000 7fd274867640 1 --2- 172.21.15.42:0/4023144989 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd27012ceb0 0x7fd270139e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:46:28.933 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.912+0000 7fd274867640 10 monclient(hunting): start opening mon connection 2024-09-16T03:46:28.933 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.912+0000 7fd274867640 1 -- 172.21.15.42:0/4023144989 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fd27013c200 con 0x7fd27012ceb0 2024-09-16T03:46:28.933 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.912+0000 7fd26dd9b640 1 --2- 172.21.15.42:0/4023144989 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd27012ceb0 0x7fd270139e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:46:28.933 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.912+0000 7fd25effd640 1 -- 172.21.15.42:0/4023144989 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fd260016020 con 0x7fd2700a39b0 2024-09-16T03:46:28.933 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.912+0000 7fd26dd9b640 10 monclient: get_auth_request con 0x7fd27012ceb0 auth_method 0 2024-09-16T03:46:28.933 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.912+0000 7fd26dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:46:28.933 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.912+0000 7fd26dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:46:28.933 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.912+0000 7fd26dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:46:28.934 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.912+0000 7fd26dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:46:28.934 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.912+0000 7fd26dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:46:28.934 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.912+0000 7fd26dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:46:28.934 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.912+0000 7fd25effd640 1 --2- 172.21.15.42:0/4023144989 >> v2:172.21.15.154:6812/254845342 conn(0x7fd2500377c0 0x7fd250039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:46:28.934 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.912+0000 7fd26dd9b640 10 monclient(hunting): handle_auth_done global_id 13604 payload 824 2024-09-16T03:46:28.934 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.912+0000 7fd26dd9b640 1 -- 172.21.15.42:0/4023144989 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd270133b80 con 0x7fd27012ceb0 2024-09-16T03:46:28.934 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.912+0000 7fd25effd640 1 -- 172.21.15.42:0/4023144989 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(124..124 src has 1..124) ==== 6770+0+0 (secure 0 0 0) 0x7fd260049180 con 0x7fd2700a39b0 2024-09-16T03:46:28.934 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.912+0000 7fd25effd640 10 client.13601.objecter ms_dispatch 0x7fd270000ba0 osd_map(124..124 src has 1..124) 2024-09-16T03:46:28.934 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.912+0000 7fd25effd640 3 client.13601.objecter handle_osd_map got epochs [124,124] > 0 2024-09-16T03:46:28.934 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.912+0000 7fd25effd640 3 client.13601.objecter handle_osd_map decoding full epoch 124 2024-09-16T03:46:28.934 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.912+0000 7fd25effd640 20 client.13601.objecter dump_active .. 0 homeless 2024-09-16T03:46:28.934 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.912+0000 7fd26dd9b640 1 --2- 172.21.15.42:0/4023144989 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd27012ceb0 0x7fd270139e30 secure :-1 s=READY pgs=2450 cs=0 l=1 rev1=1 crypto rx=0x7fd2700a7750 tx=0x7fd264019180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:46:28.934 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.912+0000 7fd25effd640 10 client.13601.objecter ms_handle_connect 0x7fd27012ceb0 2024-09-16T03:46:28.934 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.912+0000 7fd25effd640 10 client.13601.objecter resend_mon_ops 2024-09-16T03:46:28.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.912+0000 7fd25effd640 1 -- 172.21.15.42:0/4023144989 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2640bb070 con 0x7fd27012ceb0 2024-09-16T03:46:28.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.912+0000 7fd25effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:46:28.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.912+0000 7fd25effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:46:28.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.912+0000 7fd25effd640 10 monclient: dump: 2024-09-16T03:46:28.935 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:46:28.935 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:46:28.935 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:28.935 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:28.935 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:46:28.935 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:46:28.936 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:46:28.936 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:46:28.936 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:46:28.937 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:46:28.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.912+0000 7fd26ed9d640 1 --2- 172.21.15.42:0/4023144989 >> v2:172.21.15.154:6812/254845342 conn(0x7fd2500377c0 0x7fd250039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:46:28.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.912+0000 7fd26ed9d640 10 monclient: get_auth_request con 0x7fd2500377c0 auth_method 0 2024-09-16T03:46:28.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.916+0000 7fd25effd640 1 -- 172.21.15.42:0/4023144989 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fd2640b7460 con 0x7fd27012ceb0 2024-09-16T03:46:28.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.916+0000 7fd25effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:46:28.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.916+0000 7fd25effd640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:46:28.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.916+0000 7fd25effd640 1 -- 172.21.15.42:0/4023144989 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd27012ceb0 msgr2=0x7fd270139e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:46:28.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.916+0000 7fd25effd640 1 --2- 172.21.15.42:0/4023144989 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd27012ceb0 0x7fd270139e30 secure :-1 s=READY pgs=2450 cs=0 l=1 rev1=1 crypto rx=0x7fd2700a7750 tx=0x7fd264019180 comp rx=0 tx=0).stop 2024-09-16T03:46:28.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.916+0000 7fd25effd640 1 -- 172.21.15.42:0/4023144989 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd27012ceb0 msgr2=0x7fd270139e30 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:46:28.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.916+0000 7fd25effd640 1 --2- 172.21.15.42:0/4023144989 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd27012ceb0 0x7fd270139e30 secure :-1 s=CLOSED pgs=2450 cs=0 l=1 rev1=1 crypto rx=0x7fd2700a7750 tx=0x7fd264019180 comp rx=0 tx=0).stop 2024-09-16T03:46:28.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.916+0000 7fd25effd640 10 client.13601.objecter ms_handle_connect 0x7fd2500377c0 2024-09-16T03:46:28.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.916+0000 7fd26ed9d640 1 --2- 172.21.15.42:0/4023144989 >> v2:172.21.15.154:6812/254845342 conn(0x7fd2500377c0 0x7fd250039c80 secure :-1 s=READY pgs=3411 cs=0 l=1 rev1=1 crypto rx=0x7fd2680014b0 tx=0x7fd268007d30 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:46:28.939 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.920+0000 7fd274867640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:46:28.939 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.920+0000 7fd274867640 1 --2- 172.21.15.42:0/4023144989 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd27013a370 0x7fd27013ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:46:28.939 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.920+0000 7fd274867640 10 monclient(hunting): start opening mon connection 2024-09-16T03:46:28.939 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.920+0000 7fd274867640 1 -- 172.21.15.42:0/4023144989 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fd27012dbd0 con 0x7fd27013a370 2024-09-16T03:46:28.939 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.920+0000 7fd26dd9b640 1 --2- 172.21.15.42:0/4023144989 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd27013a370 0x7fd27013ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:46:28.939 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.920+0000 7fd26dd9b640 10 monclient: get_auth_request con 0x7fd27013a370 auth_method 0 2024-09-16T03:46:28.939 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.920+0000 7fd26dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:46:28.940 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.920+0000 7fd26dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:46:28.940 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.920+0000 7fd26dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:46:28.940 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.920+0000 7fd26dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:46:28.940 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.920+0000 7fd26dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:46:28.940 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.920+0000 7fd26dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:46:28.940 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.920+0000 7fd26dd9b640 10 monclient(hunting): handle_auth_done global_id 13607 payload 824 2024-09-16T03:46:28.940 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.920+0000 7fd26dd9b640 1 -- 172.21.15.42:0/4023144989 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fd27013a750 con 0x7fd27013a370 2024-09-16T03:46:28.940 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.920+0000 7fd26dd9b640 1 --2- 172.21.15.42:0/4023144989 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd27013a370 0x7fd27013ce80 secure :-1 s=READY pgs=2451 cs=0 l=1 rev1=1 crypto rx=0x7fd264009870 tx=0x7fd264009240 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:46:28.940 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.920+0000 7fd25effd640 10 client.13601.objecter ms_handle_connect 0x7fd27013a370 2024-09-16T03:46:28.940 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.920+0000 7fd25effd640 10 client.13601.objecter resend_mon_ops 2024-09-16T03:46:28.940 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.920+0000 7fd25effd640 1 -- 172.21.15.42:0/4023144989 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2640b0040 con 0x7fd27013a370 2024-09-16T03:46:28.940 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.920+0000 7fd25effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:46:28.940 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.920+0000 7fd25effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:46:28.941 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.920+0000 7fd25effd640 10 monclient: dump: 2024-09-16T03:46:28.941 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:46:28.941 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:46:28.941 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:28.941 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:28.941 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:46:28.941 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:46:28.941 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:46:28.941 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:46:28.941 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:46:28.941 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:46:28.941 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.920+0000 7fd25effd640 1 -- 172.21.15.42:0/4023144989 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4702 (secure 0 0 0) 0x7fd264003cc0 con 0x7fd27013a370 2024-09-16T03:46:28.941 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.920+0000 7fd25effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:46:28.941 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.920+0000 7fd25effd640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:46:28.942 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.920+0000 7fd25effd640 1 -- 172.21.15.42:0/4023144989 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd27013a370 msgr2=0x7fd27013ce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:46:28.942 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.920+0000 7fd25effd640 1 --2- 172.21.15.42:0/4023144989 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd27013a370 0x7fd27013ce80 secure :-1 s=READY pgs=2451 cs=0 l=1 rev1=1 crypto rx=0x7fd264009870 tx=0x7fd264009240 comp rx=0 tx=0).stop 2024-09-16T03:46:28.942 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.920+0000 7fd25effd640 1 -- 172.21.15.42:0/4023144989 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd27013a370 msgr2=0x7fd27013ce80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:46:28.942 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.920+0000 7fd25effd640 1 --2- 172.21.15.42:0/4023144989 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd27013a370 0x7fd27013ce80 secure :-1 s=CLOSED pgs=2451 cs=0 l=1 rev1=1 crypto rx=0x7fd264009870 tx=0x7fd264009240 comp rx=0 tx=0).stop 2024-09-16T03:46:28.942 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:46:28.942 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:46:28.942 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:46:28.942 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T03:46:28.942 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 36, 2024-09-16T03:46:28.942 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 268291, 2024-09-16T03:46:28.943 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:46:28.943 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T03:46:28.943 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T03:46:28.943 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:46:28.943 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 0, 2024-09-16T03:46:28.943 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:46:28.943 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:46:28.943 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:46:28.943 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:46:28.943 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:46:28.943 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:46:28.943 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:46:28.943 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:46:28.943 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:46:28.944 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:46:28.944 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:46:28.944 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:46:28.944 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:46:28.944 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:46:28.944 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:46:28.944 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:46:28.944 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:46:28.944 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:46:28.944 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:46:28.944 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:46:28.944 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:46:28.944 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:46:28.945 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:46:28.945 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:46:28.945 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:46:28.945 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:46:28.945 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:46:28.945 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:46:28.945 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:46:28.945 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:46:28.945 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:46:28.945 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:46:28.945 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:46:28.945 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:46:28.945 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:46:28.945 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:46:28.946 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:46:28.946 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:46:28.946 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:46:28.946 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:46:28.946 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:46:28.946 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:46:28.946 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:46:28.946 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:46:28.946 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:46:28.946 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:46:28.946 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:46:28.946 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:46:28.946 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:46:28.946 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:46:28.947 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:46:28.947 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:46:28.947 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:46:28.947 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:46:28.947 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:46:28.947 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:46:28.947 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:46:28.947 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:46:28.947 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:46:28.947 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:46:28.947 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:46:28.947 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:46:28.947 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:46:28.948 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:46:28.948 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:46:28.948 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:46:28.948 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:46:28.948 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:46:28.948 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:46:28.948 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:46:28.948 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:46:28.948 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:46:28.948 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:46:28.948 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:46:28.948 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:46:28.948 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:46:28.948 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:46:28.949 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:46:28.949 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:46:28.949 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:46:28.949 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:46:28.949 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:46:28.949 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:46:28.949 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:46:28.949 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:46:28.949 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:46:28.949 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:46:28.949 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:46:28.949 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:46:28.949 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:46:28.950 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:46:28.950 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:46:28.950 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:46:28.950 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:46:28.950 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:46:28.950 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:46:28.950 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:46:28.950 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:46:28.950 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:46:28.950 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:46:28.950 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:46:28.950 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:46:28.950 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:46:28.950 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:46:28.951 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:46:28.951 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:46:28.951 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:46:28.951 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:46:28.951 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:46:28.951 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:46:28.951 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:46:28.951 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:46:28.951 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:46:28.951 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:46:28.951 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:46:28.951 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:46:28.951 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:46:28.951 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:46:28.952 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:46:28.952 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:46:28.952 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:46:28.952 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:46:28.952 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:46:28.952 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:46:28.952 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:46:28.952 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:46:28.952 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:46:28.952 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:46:28.952 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:46:28.952 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:46:28.952 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:46:28.953 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:46:28.953 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:46:28.953 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:46:28.953 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:46:28.953 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:46:28.953 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:46:28.953 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:46:28.953 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:46:28.953 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:46:28.953 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:46:28.954 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T03:46:28.954 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:46:28.954 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:46:28.954 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:46:28.954 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 6 2024-09-16T03:46:28.954 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:46:28.954 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:46:28.954 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:46:28.954 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:46:28.954 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:46:28.954 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:46:28.954 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 10 2024-09-16T03:46:28.954 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:46:28.954 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:46:28.954 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:46:28.955 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:46:28.955 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:46:28.956 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.920+0000 7fd274867640 10 librados: watch_flush enter 2024-09-16T03:46:28.956 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.920+0000 7fd274867640 10 librados: watch_flush exit 2024-09-16T03:46:28.957 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.920+0000 7fd274867640 20 client.13601.objecter shutdown clearing up homeless session... 2024-09-16T03:46:28.957 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.920+0000 7fd274867640 10 client.13601.objecter successfully canceled tick 2024-09-16T03:46:28.957 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.920+0000 7fd274867640 1 -- 172.21.15.42:0/4023144989 >> v2:172.21.15.154:6812/254845342 conn(0x7fd2500377c0 msgr2=0x7fd250039c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:46:28.957 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.920+0000 7fd274867640 1 --2- 172.21.15.42:0/4023144989 >> v2:172.21.15.154:6812/254845342 conn(0x7fd2500377c0 0x7fd250039c80 secure :-1 s=READY pgs=3411 cs=0 l=1 rev1=1 crypto rx=0x7fd2680014b0 tx=0x7fd268007d30 comp rx=0 tx=0).stop 2024-09-16T03:46:28.957 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.920+0000 7fd274867640 10 monclient: shutdown 2024-09-16T03:46:28.957 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.920+0000 7fd274867640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:46:28.957 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.920+0000 7fd274867640 1 -- 172.21.15.42:0/4023144989 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd2700a39b0 msgr2=0x7fd27012c970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:46:28.957 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.920+0000 7fd274867640 1 --2- 172.21.15.42:0/4023144989 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd2700a39b0 0x7fd27012c970 secure :-1 s=READY pgs=2449 cs=0 l=1 rev1=1 crypto rx=0x7fd26000b560 tx=0x7fd26000ba20 comp rx=0 tx=0).stop 2024-09-16T03:46:28.957 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.920+0000 7fd26ed9d640 1 -- 172.21.15.42:0/4023144989 reap_dead start 2024-09-16T03:46:28.957 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.920+0000 7fd26ed9d640 1 -- 172.21.15.42:0/4023144989 reap_dead start 2024-09-16T03:46:28.957 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.920+0000 7fd274867640 1 -- 172.21.15.42:0/4023144989 shutdown_connections 2024-09-16T03:46:28.957 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.920+0000 7fd274867640 1 -- 172.21.15.42:0/4023144989 >> 172.21.15.42:0/4023144989 conn(0x7fd2700a9fb0 msgr2=0x7fd270131980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:46:28.957 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.920+0000 7fd274867640 1 -- 172.21.15.42:0/4023144989 shutdown_connections 2024-09-16T03:46:28.957 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.920+0000 7fd274867640 1 -- 172.21.15.42:0/4023144989 wait complete. 2024-09-16T03:46:28.958 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:28.920+0000 7fd274867640 1 librados: shutdown 2024-09-16T03:46:28.958 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T03:46:28.958 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-16T03:46:28.958 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T03:46:29.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.068+0000 7f9833176640 10 monclient: get_monmap_and_config 2024-09-16T03:46:29.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.068+0000 7f9833176640 10 monclient: build_initial_monmap 2024-09-16T03:46:29.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.068+0000 7f9833176640 10 monclient: monmap: 2024-09-16T03:46:29.074 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:46:29.074 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:46:29.074 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:46:29.073106+0000 2024-09-16T03:46:29.074 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:46:29.073106+0000 2024-09-16T03:46:29.074 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:46:29.074 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:46:29.074 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:46:29.075 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:46:29.075 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:46:29.075 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:46:29.075 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:46:29.075 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:46:29.077 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.068+0000 7f9833176640 1 Processor -- start 2024-09-16T03:46:29.077 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.068+0000 7f9833176640 1 -- start start 2024-09-16T03:46:29.077 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.068+0000 7f9833176640 10 monclient: init 2024-09-16T03:46:29.077 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.068+0000 7f9833176640 10 monclient: _reopen_session rank -1 2024-09-16T03:46:29.077 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.068+0000 7f9833176640 10 monclient: _add_conns ranks=[3,4,0,1,2] 2024-09-16T03:46:29.077 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.068+0000 7f9833176640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f982c0a1330 0x7f982c09f180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:46:29.077 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.068+0000 7f9833176640 10 monclient(hunting): picked mon.noname-a con 0x7f982c0a1330 addr v2:172.21.15.42:3300/0 2024-09-16T03:46:29.077 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.068+0000 7f9833176640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f982c0a1710 0x7f982c0a1b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:46:29.077 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.068+0000 7f9833176640 10 monclient(hunting): picked mon.noname-c con 0x7f982c0a1710 addr v2:172.21.15.154:3300/0 2024-09-16T03:46:29.077 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.068+0000 7f9833176640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f982c1340f0 addr v1:172.21.15.42:6789/0 2024-09-16T03:46:29.077 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.068+0000 7f9833176640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f982c09e4c0 con 0x7f982c1340f0 2024-09-16T03:46:29.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.068+0000 7f9833176640 10 monclient(hunting): start opening mon connection 2024-09-16T03:46:29.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.068+0000 7f9833176640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f982c09edf0 con 0x7f982c0a1330 2024-09-16T03:46:29.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.068+0000 7f9833176640 10 monclient(hunting): start opening mon connection 2024-09-16T03:46:29.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.068+0000 7f9833176640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f982c09cd60 con 0x7f982c0a1710 2024-09-16T03:46:29.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.068+0000 7f9833176640 10 monclient(hunting): _renew_subs 2024-09-16T03:46:29.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.068+0000 7f9833176640 10 monclient(hunting): authenticate will time out at 1236.042490s 2024-09-16T03:46:29.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.068+0000 7f9831713640 1 -- >> v1:172.21.15.42:6789/0 conn(0x7f982c1340f0 legacy=0x7f982c134540 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.42:6789/0 2024-09-16T03:46:29.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.068+0000 7f9830f12640 1 -- >> v2:172.21.15.42:3300/0 conn(0x7f982c0a1330 msgr2=0x7f982c09f180 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3300/0 2024-09-16T03:46:29.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.068+0000 7f9830f12640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f982c0a1330 0x7f982c09f180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:46:29.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.068+0000 7f982bfff640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f982c0a1710 0x7f982c0a1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:46:29.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.068+0000 7f982bfff640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f982c0a1710 0x7f982c0a1b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.42:36332/0 (socket says 172.21.15.42:36332) 2024-09-16T03:46:29.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.068+0000 7f982bfff640 1 -- 172.21.15.42:0/4269651134 learned_addr learned my addr 172.21.15.42:0/4269651134 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:46:29.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.068+0000 7f982bfff640 10 monclient(hunting): get_auth_request con 0x7f982c0a1710 auth_method 0 2024-09-16T03:46:29.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.068+0000 7f982bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:46:29.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.068+0000 7f982bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:46:29.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.068+0000 7f982bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:46:29.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.068+0000 7f982bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:46:29.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.068+0000 7f982bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:46:29.080 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.068+0000 7f982bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:46:29.080 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.068+0000 7f982bfff640 10 monclient(hunting): handle_auth_done global_id 754171 payload 274 2024-09-16T03:46:29.080 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.068+0000 7f982bfff640 1 -- 172.21.15.42:0/4269651134 >> v2:172.21.15.42:3300/0 conn(0x7f982c0a1330 msgr2=0x7f982c09f180 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:46:29.080 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.068+0000 7f982bfff640 1 --2- 172.21.15.42:0/4269651134 >> v2:172.21.15.42:3300/0 conn(0x7f982c0a1330 0x7f982c09f180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:46:29.080 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.068+0000 7f982bfff640 1 -- 172.21.15.42:0/4269651134 >> v1:172.21.15.42:6789/0 conn(0x7f982c1340f0 legacy=0x7f982c134540 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:46:29.080 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.068+0000 7f982bfff640 10 monclient: _finish_hunting 0 2024-09-16T03:46:29.080 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.068+0000 7f982bfff640 1 monclient: found mon.noname-c 2024-09-16T03:46:29.080 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.068+0000 7f982bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:46:29.080 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.068+0000 7f982bfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.154:3300/0 2024-09-16T03:46:29.080 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.068+0000 7f982bfff640 1 -- 172.21.15.42:0/4269651134 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f982c0a2080 con 0x7f982c0a1710 2024-09-16T03:46:29.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.072+0000 7f982bfff640 1 --2- 172.21.15.42:0/4269651134 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f982c0a1710 0x7f982c0a1b40 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f982c073490 tx=0x7f982001c450 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c176bfa33067250d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:46:29.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.072+0000 7f982b7fe640 1 -- 172.21.15.42:0/4269651134 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98200b2070 con 0x7f982c0a1710 2024-09-16T03:46:29.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.072+0000 7f982b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:46:29.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.072+0000 7f982b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:46:29.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.072+0000 7f982b7fe640 10 monclient: dump: 2024-09-16T03:46:29.089 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:46:29.089 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:46:29.089 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:29.089 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:29.089 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:46:29.089 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:46:29.090 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:46:29.090 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:46:29.090 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:46:29.090 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:46:29.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.072+0000 7f982b7fe640 10 monclient: _finish_auth 0 2024-09-16T03:46:29.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.072+0000 7f982b7fe640 10 monclient: _check_auth_tickets 2024-09-16T03:46:29.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.072+0000 7f982b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:46:29.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.072+0000 7f982b7fe640 1 -- 172.21.15.42:0/4269651134 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f98200a9070 con 0x7f982c0a1710 2024-09-16T03:46:29.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.072+0000 7f982b7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:46:29.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.072+0000 7f982b7fe640 1 -- 172.21.15.42:0/4269651134 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98200ad880 con 0x7f982c0a1710 2024-09-16T03:46:29.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.072+0000 7f9833176640 5 monclient: authenticate success, global_id 754171 2024-09-16T03:46:29.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.072+0000 7f9833176640 10 monclient: get_monmap_and_config success 2024-09-16T03:46:29.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.072+0000 7f9833176640 4 set_mon_vals no callback set 2024-09-16T03:46:29.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.072+0000 7f982b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:46:29.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.072+0000 7f982b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:46:29.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.072+0000 7f982b7fe640 10 monclient: dump: 2024-09-16T03:46:29.091 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:46:29.091 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:46:29.091 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:29.091 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:29.091 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:46:29.091 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:46:29.092 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:46:29.092 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:46:29.092 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:46:29.092 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:46:29.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.072+0000 7f9833176640 10 monclient: shutdown 2024-09-16T03:46:29.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.072+0000 7f9833176640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:46:29.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.072+0000 7f9833176640 1 -- 172.21.15.42:0/4269651134 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f982c0a1710 msgr2=0x7f982c0a1b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:46:29.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.072+0000 7f9833176640 1 --2- 172.21.15.42:0/4269651134 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f982c0a1710 0x7f982c0a1b40 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f982c073490 tx=0x7f982001c450 comp rx=0 tx=0).stop 2024-09-16T03:46:29.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.072+0000 7f9833176640 1 -- 172.21.15.42:0/4269651134 shutdown_connections 2024-09-16T03:46:29.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.072+0000 7f9833176640 1 --2- 172.21.15.42:0/4269651134 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f982c0a1710 0x7f982c0a1b40 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:46:29.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.072+0000 7f9833176640 1 --2- 172.21.15.42:0/4269651134 >> v2:172.21.15.42:3300/0 conn(0x7f982c0a1330 0x7f982c09f180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:46:29.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.072+0000 7f9833176640 1 -- 172.21.15.42:0/4269651134 >> 172.21.15.42:0/4269651134 conn(0x7f982c0a2420 msgr2=0x7f982c0a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:46:29.093 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.072+0000 7f9833176640 1 -- 172.21.15.42:0/4269651134 shutdown_connections 2024-09-16T03:46:29.093 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.072+0000 7f9833176640 1 -- 172.21.15.42:0/4269651134 wait complete. 2024-09-16T03:46:29.093 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.072+0000 7f9833176640 10 monclient: build_initial_monmap 2024-09-16T03:46:29.093 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.072+0000 7f9833176640 10 monclient: monmap: 2024-09-16T03:46:29.093 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:46:29.093 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:46:29.093 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:46:29.093 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:46:29.093 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:46:29.093 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:46:29.093 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:46:29.093 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:46:29.093 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:46:29.094 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:46:29.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.072+0000 7f9833176640 1 librados: starting msgr at 2024-09-16T03:46:29.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.072+0000 7f9833176640 1 librados: starting objecter 2024-09-16T03:46:29.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.072+0000 7f9833176640 1 Processor -- start 2024-09-16T03:46:29.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.076+0000 7f9833176640 1 -- start start 2024-09-16T03:46:29.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.076+0000 7f9833176640 1 librados: setting wanted keys 2024-09-16T03:46:29.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.076+0000 7f9833176640 1 librados: calling monclient init 2024-09-16T03:46:29.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.076+0000 7f9833176640 10 monclient: init 2024-09-16T03:46:29.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.076+0000 7f9833176640 10 monclient: _reopen_session rank -1 2024-09-16T03:46:29.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.076+0000 7f9833176640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:46:29.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.076+0000 7f9833176640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f982c0a1330 0x7f982c139790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:46:29.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.076+0000 7f9833176640 10 monclient(hunting): picked mon.noname-b con 0x7f982c0a1330 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:46:29.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.076+0000 7f9833176640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f982c0a1710 0x7f982c139cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:46:29.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.076+0000 7f9833176640 10 monclient(hunting): picked mon.noname-a con 0x7f982c0a1710 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:46:29.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.076+0000 7f9833176640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f982c1340f0 0x7f982c13a210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:46:29.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.076+0000 7f9833176640 10 monclient(hunting): picked mon.noname-c con 0x7f982c1340f0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:46:29.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.076+0000 7f9833176640 10 monclient(hunting): start opening mon connection 2024-09-16T03:46:29.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.076+0000 7f9833176640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f982c09bbd0 con 0x7f982c0a1710 2024-09-16T03:46:29.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.076+0000 7f9833176640 10 monclient(hunting): start opening mon connection 2024-09-16T03:46:29.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.076+0000 7f9833176640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f982c09ba50 con 0x7f982c0a1330 2024-09-16T03:46:29.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.076+0000 7f9833176640 10 monclient(hunting): start opening mon connection 2024-09-16T03:46:29.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.076+0000 7f9833176640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f982c09bd50 con 0x7f982c1340f0 2024-09-16T03:46:29.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.076+0000 7f9833176640 10 monclient(hunting): _renew_subs 2024-09-16T03:46:29.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.076+0000 7f9833176640 10 monclient(hunting): authenticate will time out at 1236.048573s 2024-09-16T03:46:29.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.076+0000 7f982bfff640 1 -- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f982c0a1710 msgr2=0x7f982c139cd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.42:3300/0 2024-09-16T03:46:29.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.076+0000 7f982bfff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f982c0a1710 0x7f982c139cd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:46:29.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.076+0000 7f9831713640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f982c1340f0 0x7f982c13a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:46:29.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.076+0000 7f9831713640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f982c1340f0 0x7f982c13a210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:39956/0 (socket says 172.21.15.42:39956) 2024-09-16T03:46:29.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.076+0000 7f9831713640 1 -- 172.21.15.42:0/3124160177 learned_addr learned my addr 172.21.15.42:0/3124160177 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:46:29.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.076+0000 7f9831713640 10 monclient(hunting): get_auth_request con 0x7f982c1340f0 auth_method 0 2024-09-16T03:46:29.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.076+0000 7f9831713640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:46:29.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.076+0000 7f9831713640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:46:29.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.076+0000 7f9831713640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:46:29.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.076+0000 7f9831713640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:46:29.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.076+0000 7f9831713640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:46:29.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.076+0000 7f9831713640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:46:29.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.076+0000 7f9830f12640 1 --2- 172.21.15.42:0/3124160177 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f982c0a1330 0x7f982c139790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:46:29.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.076+0000 7f9830f12640 10 monclient(hunting): get_auth_request con 0x7f982c0a1330 auth_method 0 2024-09-16T03:46:29.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.076+0000 7f9830f12640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:46:29.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.076+0000 7f9830f12640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:46:29.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.076+0000 7f9830f12640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:46:29.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.076+0000 7f9831713640 10 monclient(hunting): handle_auth_done global_id 13610 payload 824 2024-09-16T03:46:29.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.076+0000 7f9831713640 1 -- 172.21.15.42:0/3124160177 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f982c0a1330 msgr2=0x7f982c139790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:46:29.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.076+0000 7f9831713640 1 --2- 172.21.15.42:0/3124160177 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f982c0a1330 0x7f982c139790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:46:29.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.076+0000 7f9831713640 1 -- 172.21.15.42:0/3124160177 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f982c0a1710 msgr2=0x7f982c139cd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:46:29.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.076+0000 7f9831713640 1 --2- 172.21.15.42:0/3124160177 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f982c0a1710 0x7f982c139cd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:46:29.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.076+0000 7f9831713640 10 monclient: _finish_hunting 0 2024-09-16T03:46:29.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.076+0000 7f9831713640 1 monclient: found mon.noname-c 2024-09-16T03:46:29.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.076+0000 7f9831713640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:46:29.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.076+0000 7f9831713640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:46:29.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.076+0000 7f9831713640 1 -- 172.21.15.42:0/3124160177 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f982c11d020 con 0x7f982c1340f0 2024-09-16T03:46:29.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.076+0000 7f9831713640 1 --2- 172.21.15.42:0/3124160177 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f982c1340f0 0x7f982c13a210 secure :-1 s=READY pgs=2452 cs=0 l=1 rev1=1 crypto rx=0x7f981c00b410 tx=0x7f981c00b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:46:29.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.076+0000 7f98297fa640 10 client.?.objecter ms_handle_connect 0x7f982c1340f0 2024-09-16T03:46:29.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.076+0000 7f98297fa640 10 client.?.objecter resend_mon_ops 2024-09-16T03:46:29.098 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.076+0000 7f98297fa640 1 -- 172.21.15.42:0/3124160177 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f981c017070 con 0x7f982c1340f0 2024-09-16T03:46:29.098 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.076+0000 7f98297fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:46:29.098 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.076+0000 7f98297fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:46:29.098 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.076+0000 7f98297fa640 10 monclient: dump: 2024-09-16T03:46:29.098 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:46:29.098 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:46:29.098 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:29.098 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:29.098 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:46:29.098 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:46:29.098 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:46:29.098 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:46:29.098 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:46:29.098 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:46:29.099 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.076+0000 7f98297fa640 10 monclient: _finish_auth 0 2024-09-16T03:46:29.099 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.076+0000 7f98297fa640 10 monclient: _check_auth_tickets 2024-09-16T03:46:29.099 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.076+0000 7f98297fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:46:29.099 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.076+0000 7f9833176640 5 monclient: authenticate success, global_id 13610 2024-09-16T03:46:29.099 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.076+0000 7f98297fa640 1 -- 172.21.15.42:0/3124160177 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f981c003dd0 con 0x7f982c1340f0 2024-09-16T03:46:29.099 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.076+0000 7f98297fa640 10 monclient: handle_config config(0 keys) 2024-09-16T03:46:29.099 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.076+0000 7f98297fa640 1 -- 172.21.15.42:0/3124160177 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f981c006310 con 0x7f982c1340f0 2024-09-16T03:46:29.099 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.076+0000 7f98297fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:46:29.099 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.076+0000 7f98297fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:46:29.099 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.076+0000 7f98297fa640 10 monclient: dump: 2024-09-16T03:46:29.099 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:46:29.099 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:46:29.099 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:29.099 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:29.100 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:46:29.100 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:46:29.100 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:46:29.100 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:46:29.100 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:46:29.100 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:46:29.100 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.076+0000 7f982a7fc640 4 set_mon_vals no callback set 2024-09-16T03:46:29.100 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.076+0000 7f9833176640 10 monclient: _renew_subs 2024-09-16T03:46:29.100 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.076+0000 7f9833176640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:46:29.100 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.076+0000 7f9833176640 1 -- 172.21.15.42:0/3124160177 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f982c0a4d10 con 0x7f982c1340f0 2024-09-16T03:46:29.100 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.076+0000 7f9833176640 10 client.13610.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:46:29.100 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.076+0000 7f9833176640 10 monclient: _renew_subs 2024-09-16T03:46:29.100 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.076+0000 7f9833176640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:46:29.101 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.076+0000 7f9833176640 1 -- 172.21.15.42:0/3124160177 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f982c073a30 con 0x7f982c1340f0 2024-09-16T03:46:29.101 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.076+0000 7f9833176640 1 librados: init done 2024-09-16T03:46:29.101 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.076+0000 7f9833176640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:46:29.101 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.076+0000 7f9833176640 1 --2- 172.21.15.42:0/3124160177 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f982c06dce0 0x7f982c06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:46:29.101 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.076+0000 7f9833176640 10 monclient(hunting): start opening mon connection 2024-09-16T03:46:29.101 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.076+0000 7f9833176640 1 -- 172.21.15.42:0/3124160177 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f982c09f180 con 0x7f982c06dce0 2024-09-16T03:46:29.101 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.076+0000 7f98297fa640 1 -- 172.21.15.42:0/3124160177 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f981c0064b0 con 0x7f982c1340f0 2024-09-16T03:46:29.101 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.076+0000 7f9830f12640 1 --2- 172.21.15.42:0/3124160177 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f982c06dce0 0x7f982c06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:46:29.101 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.076+0000 7f9830f12640 10 monclient: get_auth_request con 0x7f982c06dce0 auth_method 0 2024-09-16T03:46:29.101 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.076+0000 7f9830f12640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:46:29.101 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.076+0000 7f9830f12640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:46:29.101 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.076+0000 7f9830f12640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:46:29.101 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.076+0000 7f98297fa640 1 --2- 172.21.15.42:0/3124160177 >> v2:172.21.15.154:6812/254845342 conn(0x7f98180377c0 0x7f9818039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:46:29.101 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.076+0000 7f9830f12640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:46:29.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.076+0000 7f9830f12640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:46:29.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.076+0000 7f98297fa640 1 -- 172.21.15.42:0/3124160177 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(124..124 src has 1..124) ==== 6770+0+0 (secure 0 0 0) 0x7f981c048450 con 0x7f982c1340f0 2024-09-16T03:46:29.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.076+0000 7f98297fa640 10 client.13610.objecter ms_dispatch 0x7f982c000ba0 osd_map(124..124 src has 1..124) 2024-09-16T03:46:29.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.076+0000 7f98297fa640 3 client.13610.objecter handle_osd_map got epochs [124,124] > 0 2024-09-16T03:46:29.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.076+0000 7f98297fa640 3 client.13610.objecter handle_osd_map decoding full epoch 124 2024-09-16T03:46:29.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.076+0000 7f9830f12640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:46:29.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.076+0000 7f98297fa640 20 client.13610.objecter dump_active .. 0 homeless 2024-09-16T03:46:29.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.076+0000 7f982bfff640 1 --2- 172.21.15.42:0/3124160177 >> v2:172.21.15.154:6812/254845342 conn(0x7f98180377c0 0x7f9818039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:46:29.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.076+0000 7f982bfff640 10 monclient: get_auth_request con 0x7f98180377c0 auth_method 0 2024-09-16T03:46:29.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.080+0000 7f9830f12640 10 monclient(hunting): handle_auth_done global_id 754177 payload 824 2024-09-16T03:46:29.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.080+0000 7f9830f12640 1 -- 172.21.15.42:0/3124160177 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f982c12fe70 con 0x7f982c06dce0 2024-09-16T03:46:29.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.080+0000 7f9830f12640 1 --2- 172.21.15.42:0/3124160177 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f982c06dce0 0x7f982c06e0c0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f9824009fd0 tx=0x7f9824005420 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:46:29.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.080+0000 7f98297fa640 10 client.13610.objecter ms_handle_connect 0x7f982c06dce0 2024-09-16T03:46:29.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.080+0000 7f98297fa640 10 client.13610.objecter resend_mon_ops 2024-09-16T03:46:29.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.080+0000 7f98297fa640 10 client.13610.objecter ms_handle_connect 0x7f98180377c0 2024-09-16T03:46:29.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.080+0000 7f982bfff640 1 --2- 172.21.15.42:0/3124160177 >> v2:172.21.15.154:6812/254845342 conn(0x7f98180377c0 0x7f9818039c80 secure :-1 s=READY pgs=3412 cs=0 l=1 rev1=1 crypto rx=0x7f98200af000 tx=0x7f9820008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:46:29.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.080+0000 7f98297fa640 1 -- 172.21.15.42:0/3124160177 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9824019070 con 0x7f982c06dce0 2024-09-16T03:46:29.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.080+0000 7f98297fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:46:29.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.080+0000 7f98297fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:46:29.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.080+0000 7f98297fa640 10 monclient: dump: 2024-09-16T03:46:29.103 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:46:29.103 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:46:29.103 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:29.103 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:29.103 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:46:29.103 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:46:29.103 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:46:29.103 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:46:29.104 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:46:29.104 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:46:29.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.080+0000 7f98297fa640 1 -- 172.21.15.42:0/3124160177 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f9824012520 con 0x7f982c06dce0 2024-09-16T03:46:29.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.080+0000 7f98297fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:46:29.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.080+0000 7f98297fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:46:29.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.080+0000 7f98297fa640 1 -- 172.21.15.42:0/3124160177 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f982c06dce0 msgr2=0x7f982c06e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:46:29.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.080+0000 7f98297fa640 1 --2- 172.21.15.42:0/3124160177 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f982c06dce0 0x7f982c06e0c0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f9824009fd0 tx=0x7f9824005420 comp rx=0 tx=0).stop 2024-09-16T03:46:29.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.080+0000 7f98297fa640 1 -- 172.21.15.42:0/3124160177 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f982c06dce0 msgr2=0x7f982c06e0c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:46:29.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.080+0000 7f98297fa640 1 --2- 172.21.15.42:0/3124160177 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f982c06dce0 0x7f982c06e0c0 secure :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f9824009fd0 tx=0x7f9824005420 comp rx=0 tx=0).stop 2024-09-16T03:46:29.106 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.084+0000 7f9833176640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:46:29.106 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.084+0000 7f9833176640 1 --2- 172.21.15.42:0/3124160177 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f982c06e600 0x7f982c1d9a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:46:29.106 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.084+0000 7f9833176640 10 monclient(hunting): start opening mon connection 2024-09-16T03:46:29.106 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.084+0000 7f9833176640 1 -- 172.21.15.42:0/3124160177 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f982c11d360 con 0x7f982c06e600 2024-09-16T03:46:29.106 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.088+0000 7f9830f12640 1 --2- 172.21.15.42:0/3124160177 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f982c06e600 0x7f982c1d9a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:46:29.106 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.088+0000 7f9830f12640 10 monclient: get_auth_request con 0x7f982c06e600 auth_method 0 2024-09-16T03:46:29.106 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.088+0000 7f9830f12640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:46:29.106 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.088+0000 7f9830f12640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:46:29.107 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.088+0000 7f9830f12640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:46:29.107 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.088+0000 7f9830f12640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:46:29.107 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.088+0000 7f9830f12640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:46:29.107 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.088+0000 7f9830f12640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:46:29.107 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.088+0000 7f9830f12640 10 monclient(hunting): handle_auth_done global_id 754180 payload 824 2024-09-16T03:46:29.107 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.088+0000 7f9830f12640 1 -- 172.21.15.42:0/3124160177 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f982c069040 con 0x7f982c06e600 2024-09-16T03:46:29.107 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.088+0000 7f9830f12640 1 --2- 172.21.15.42:0/3124160177 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f982c06e600 0x7f982c1d9a00 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f9824009fd0 tx=0x7f9824005d00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:46:29.107 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.088+0000 7f98297fa640 10 client.13610.objecter ms_handle_connect 0x7f982c06e600 2024-09-16T03:46:29.107 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.088+0000 7f98297fa640 10 client.13610.objecter resend_mon_ops 2024-09-16T03:46:29.107 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.088+0000 7f98297fa640 1 -- 172.21.15.42:0/3124160177 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9824019030 con 0x7f982c06e600 2024-09-16T03:46:29.107 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.088+0000 7f98297fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:46:29.107 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.088+0000 7f98297fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:46:29.107 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.088+0000 7f98297fa640 10 monclient: dump: 2024-09-16T03:46:29.107 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:46:29.108 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:46:29.108 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:29.108 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:29.108 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:46:29.108 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:46:29.108 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:46:29.108 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:46:29.108 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:46:29.108 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:46:29.108 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.088+0000 7f98297fa640 1 -- 172.21.15.42:0/3124160177 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4702 (secure 0 0 0) 0x7f9824024410 con 0x7f982c06e600 2024-09-16T03:46:29.108 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.088+0000 7f98297fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:46:29.108 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.088+0000 7f98297fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:46:29.108 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.088+0000 7f98297fa640 1 -- 172.21.15.42:0/3124160177 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f982c06e600 msgr2=0x7f982c1d9a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:46:29.109 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.088+0000 7f98297fa640 1 --2- 172.21.15.42:0/3124160177 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f982c06e600 0x7f982c1d9a00 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f9824009fd0 tx=0x7f9824005d00 comp rx=0 tx=0).stop 2024-09-16T03:46:29.109 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.088+0000 7f98297fa640 1 -- 172.21.15.42:0/3124160177 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f982c06e600 msgr2=0x7f982c1d9a00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:46:29.109 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.088+0000 7f98297fa640 1 --2- 172.21.15.42:0/3124160177 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f982c06e600 0x7f982c1d9a00 secure :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f9824009fd0 tx=0x7f9824005d00 comp rx=0 tx=0).stop 2024-09-16T03:46:29.109 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:46:29.109 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:46:29.109 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:46:29.109 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "leader", 2024-09-16T03:46:29.109 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 36, 2024-09-16T03:46:29.109 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 31107, 2024-09-16T03:46:29.109 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:46:29.110 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T03:46:29.110 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T03:46:29.110 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:46:29.110 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 0, 2024-09-16T03:46:29.110 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:46:29.110 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:46:29.110 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:46:29.110 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:46:29.110 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:46:29.110 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:46:29.110 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:46:29.110 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:46:29.110 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:46:29.110 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:46:29.111 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:46:29.111 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:46:29.111 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:46:29.111 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:46:29.111 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:46:29.111 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:46:29.111 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:46:29.111 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:46:29.111 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:46:29.111 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:46:29.111 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:46:29.111 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:46:29.111 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:46:29.111 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:46:29.112 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:46:29.112 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:46:29.112 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:46:29.112 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:46:29.112 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:46:29.112 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:46:29.112 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:46:29.112 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:46:29.112 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:46:29.112 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:46:29.112 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:46:29.112 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:46:29.112 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:46:29.113 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:46:29.113 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:46:29.113 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:46:29.113 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:46:29.113 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:46:29.113 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:46:29.113 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:46:29.113 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:46:29.113 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:46:29.113 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:46:29.113 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:46:29.113 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:46:29.113 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:46:29.114 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:46:29.114 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:46:29.114 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:46:29.114 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:46:29.114 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:46:29.114 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:46:29.114 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:46:29.114 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:46:29.114 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:46:29.114 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:46:29.114 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:46:29.114 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:46:29.114 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:46:29.115 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:46:29.115 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:46:29.115 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:46:29.115 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:46:29.115 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:46:29.115 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:46:29.115 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:46:29.115 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:46:29.115 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:46:29.115 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:46:29.115 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:46:29.115 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:46:29.115 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:46:29.115 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:46:29.116 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:46:29.116 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:46:29.116 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:46:29.116 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:46:29.116 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:46:29.116 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:46:29.116 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:46:29.116 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:46:29.116 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:46:29.116 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:46:29.116 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:46:29.116 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:46:29.116 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:46:29.117 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:46:29.117 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:46:29.117 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:46:29.117 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:46:29.117 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:46:29.117 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:46:29.117 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:46:29.117 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:46:29.117 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:46:29.117 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:46:29.117 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:46:29.117 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:46:29.117 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:46:29.117 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:46:29.118 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:46:29.118 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:46:29.118 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:46:29.118 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:46:29.118 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:46:29.118 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:46:29.118 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:46:29.118 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:46:29.118 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:46:29.118 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:46:29.118 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:46:29.118 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:46:29.118 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:46:29.118 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:46:29.119 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:46:29.119 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:46:29.119 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:46:29.119 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:46:29.119 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:46:29.119 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:46:29.119 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:46:29.119 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:46:29.119 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:46:29.119 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:46:29.119 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:46:29.119 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:46:29.119 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:46:29.120 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:46:29.120 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:46:29.120 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:46:29.120 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:46:29.120 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:46:29.120 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:46:29.120 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:46:29.120 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:46:29.120 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:46:29.120 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:46:29.120 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:46:29.120 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:46:29.120 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:46:29.121 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:46:29.121 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 2 2024-09-16T03:46:29.121 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:46:29.121 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:46:29.121 INFO:teuthology.orchestra.run.smithi042.stdout: "mgr": [ 2024-09-16T03:46:29.121 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:46:29.121 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:46:29.121 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:46:29.121 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:46:29.121 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:46:29.121 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:46:29.121 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:46:29.121 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:46:29.121 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:46:29.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.088+0000 7f9833176640 10 librados: watch_flush enter 2024-09-16T03:46:29.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.088+0000 7f9833176640 10 librados: watch_flush exit 2024-09-16T03:46:29.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.088+0000 7f9833176640 20 client.13610.objecter shutdown clearing up homeless session... 2024-09-16T03:46:29.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.088+0000 7f9833176640 10 client.13610.objecter successfully canceled tick 2024-09-16T03:46:29.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.088+0000 7f9833176640 1 -- 172.21.15.42:0/3124160177 >> v2:172.21.15.154:6812/254845342 conn(0x7f98180377c0 msgr2=0x7f9818039c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:46:29.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.088+0000 7f9833176640 1 --2- 172.21.15.42:0/3124160177 >> v2:172.21.15.154:6812/254845342 conn(0x7f98180377c0 0x7f9818039c80 secure :-1 s=READY pgs=3412 cs=0 l=1 rev1=1 crypto rx=0x7f98200af000 tx=0x7f9820008040 comp rx=0 tx=0).stop 2024-09-16T03:46:29.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.088+0000 7f9833176640 10 monclient: shutdown 2024-09-16T03:46:29.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.088+0000 7f9833176640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:46:29.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.088+0000 7f9833176640 1 -- 172.21.15.42:0/3124160177 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f982c1340f0 msgr2=0x7f982c13a210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:46:29.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.088+0000 7f9833176640 1 --2- 172.21.15.42:0/3124160177 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f982c1340f0 0x7f982c13a210 secure :-1 s=READY pgs=2452 cs=0 l=1 rev1=1 crypto rx=0x7f981c00b410 tx=0x7f981c00b8d0 comp rx=0 tx=0).stop 2024-09-16T03:46:29.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.088+0000 7f9831713640 1 -- 172.21.15.42:0/3124160177 reap_dead start 2024-09-16T03:46:29.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.088+0000 7f9831713640 1 -- 172.21.15.42:0/3124160177 reap_dead start 2024-09-16T03:46:29.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.088+0000 7f9833176640 1 -- 172.21.15.42:0/3124160177 shutdown_connections 2024-09-16T03:46:29.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.088+0000 7f9833176640 1 -- 172.21.15.42:0/3124160177 >> 172.21.15.42:0/3124160177 conn(0x7f982c0a2420 msgr2=0x7f982c0a3720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:46:29.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.088+0000 7f9833176640 1 -- 172.21.15.42:0/3124160177 shutdown_connections 2024-09-16T03:46:29.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.088+0000 7f9833176640 1 -- 172.21.15.42:0/3124160177 wait complete. 2024-09-16T03:46:29.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:29.088+0000 7f9833176640 1 librados: shutdown 2024-09-16T03:46:29.124 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T03:46:29.125 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-16T03:46:41.200 INFO:tasks.workunit.client.0.smithi154.stdout:ry=15 expected=15 2024-09-16T03:46:41.200 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: : seek to 12:6cac518f:::0:head 2024-09-16T03:46:41.200 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: : cursor()=12:6cac518f:::0:head expected=12:6cac518f:::0:head 2024-09-16T03:46:41.200 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: > 12:6cac518f:::0:head -> 0 2024-09-16T03:46:41.200 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: : entry=0 expected=0 2024-09-16T03:46:41.200 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: : seek to 12:566253c9:::13:head 2024-09-16T03:46:41.200 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: : cursor()=12:566253c9:::13:head expected=12:566253c9:::13:head 2024-09-16T03:46:41.200 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: > 12:566253c9:::13:head -> 13 2024-09-16T03:46:41.200 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: : entry=13 expected=13 2024-09-16T03:46:41.201 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: : seek to 12:e960b815:::9:head 2024-09-16T03:46:41.201 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: : cursor()=12:e960b815:::9:head expected=12:e960b815:::9:head 2024-09-16T03:46:41.201 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: > 12:e960b815:::9:head -> 9 2024-09-16T03:46:41.201 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: : entry=9 expected=9 2024-09-16T03:46:41.201 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: : seek to 12:de5d7c5f:::12:head 2024-09-16T03:46:41.201 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: : cursor()=12:de5d7c5f:::12:head expected=12:de5d7c5f:::12:head 2024-09-16T03:46:41.201 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: > 12:de5d7c5f:::12:head -> 12 2024-09-16T03:46:41.201 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: : entry=12 expected=12 2024-09-16T03:46:41.201 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: : seek to 12:d83876eb:::4:head 2024-09-16T03:46:41.201 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: : cursor()=12:d83876eb:::4:head expected=12:d83876eb:::4:head 2024-09-16T03:46:41.201 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: > 12:d83876eb:::4:head -> 4 2024-09-16T03:46:41.201 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: : entry=4 expected=4 2024-09-16T03:46:41.201 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: : seek to 12:c4fdafeb:::6:head 2024-09-16T03:46:41.201 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: : cursor()=12:c4fdafeb:::6:head expected=12:c4fdafeb:::6:head 2024-09-16T03:46:41.202 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: > 12:c4fdafeb:::6:head -> 6 2024-09-16T03:46:41.202 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: : entry=6 expected=6 2024-09-16T03:46:41.202 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: : seek to 12:62a1935d:::14:head 2024-09-16T03:46:41.202 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: : cursor()=12:62a1935d:::14:head expected=12:62a1935d:::14:head 2024-09-16T03:46:41.202 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: > 12:62a1935d:::14:head -> 14 2024-09-16T03:46:41.202 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: : entry=14 expected=14 2024-09-16T03:46:41.202 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: : seek to 12:5c6b0b28:::7:head 2024-09-16T03:46:41.202 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: : cursor()=12:5c6b0b28:::7:head expected=12:5c6b0b28:::7:head 2024-09-16T03:46:41.202 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: > 12:5c6b0b28:::7:head -> 7 2024-09-16T03:46:41.202 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: : entry=7 expected=7 2024-09-16T03:46:41.202 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: [ OK ] LibRadosList.ListObjectsCursor (298 ms) 2024-09-16T03:46:41.202 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: [ RUN ] LibRadosList.EnumerateObjects 2024-09-16T03:46:41.202 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: [ OK ] LibRadosList.EnumerateObjects (60416 ms) 2024-09-16T03:46:41.203 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: [ RUN ] LibRadosList.EnumerateObjectsSplit 2024-09-16T03:46:41.203 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: split 0/5 -> MIN 12:33333333::::head 2024-09-16T03:46:41.203 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: split 1/5 -> 12:33333333::::head 12:66666666::::head 2024-09-16T03:46:41.203 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: split 2/5 -> 12:66666666::::head 12:99999999::::head 2024-09-16T03:46:41.203 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: split 3/5 -> 12:99999999::::head 12:cccccccc::::head 2024-09-16T03:46:41.203 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: split 4/5 -> 12:cccccccc::::head MAX 2024-09-16T03:46:41.203 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: [ OK ] LibRadosList.EnumerateObjectsSplit (121570 ms) 2024-09-16T03:46:41.203 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: [----------] 7 tests from LibRadosList (182843 ms total) 2024-09-16T03:46:41.203 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: 2024-09-16T03:46:41.203 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: [----------] 3 tests from LibRadosListEC 2024-09-16T03:46:41.203 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: [ RUN ] LibRadosListEC.ListObjects 2024-09-16T03:46:41.203 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: [ OK ] LibRadosListEC.ListObjects (1062 ms) 2024-09-16T03:46:41.203 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: [ RUN ] LibRadosListEC.ListObjectsNS 2024-09-16T03:46:41.203 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: myset foo1,foo2,foo3 2024-09-16T03:46:41.204 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: foo1 2024-09-16T03:46:41.204 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: foo2 2024-09-16T03:46:41.204 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: foo3 2024-09-16T03:46:41.204 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: myset foo1,foo4,foo5 2024-09-16T03:46:41.204 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: foo4 2024-09-16T03:46:41.204 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: foo5 2024-09-16T03:46:41.204 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: foo1 2024-09-16T03:46:41.204 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: myset foo6,foo7 2024-09-16T03:46:41.204 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: foo7 2024-09-16T03:46:41.204 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: foo6 2024-09-16T03:46:41.204 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: myset :foo1,:foo2,:foo3,ns1:foo1,ns1:foo4,ns1:foo5,ns2:foo6,ns2:foo7 2024-09-16T03:46:41.204 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: ns1:foo4 2024-09-16T03:46:41.204 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: ns1:foo5 2024-09-16T03:46:41.205 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: ns2:foo7 2024-09-16T03:46:41.205 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: ns2:foo6 2024-09-16T03:46:41.205 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: ns1:foo1 2024-09-16T03:46:41.205 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: :foo1 2024-09-16T03:46:41.205 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: :foo2 2024-09-16T03:46:41.205 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: :foo3 2024-09-16T03:46:41.205 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: [ OK ] LibRadosListEC.ListObjectsNS (89 ms) 2024-09-16T03:46:47.058 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: [ RUN ] LibRadosListEC.ListObjectsStart 2024-09-16T03:46:47.059 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: 1 0 2024-09-16T03:46:47.059 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: 10 0 2024-09-16T03:46:47.059 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: 13 0 2024-09-16T03:46:47.059 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: 7 0 2024-09-16T03:46:47.059 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: 14 0 2024-09-16T03:46:47.059 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: 0 0 2024-09-16T03:46:47.059 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: 15 0 2024-09-16T03:46:47.059 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: 11 0 2024-09-16T03:46:47.059 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: 5 0 2024-09-16T03:46:47.059 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: 8 0 2024-09-16T03:46:47.059 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: 6 0 2024-09-16T03:46:47.060 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: 3 0 2024-09-16T03:46:47.060 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: 4 0 2024-09-16T03:46:47.060 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: 12 0 2024-09-16T03:46:47.060 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: 9 0 2024-09-16T03:46:47.060 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: 2 0 2024-09-16T03:46:47.060 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: have 1 expect one of 0,1,10,11,12,13,14,15,2,3,4,5,6,7,8,9 2024-09-16T03:46:47.060 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: [ OK ] LibRadosListEC.ListObjectsStart (151 ms) 2024-09-16T03:46:47.060 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: [----------] 3 tests from LibRadosListEC (1302 ms total) 2024-09-16T03:46:47.060 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: 2024-09-16T03:46:47.060 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: [----------] 1 test from LibRadosListNP 2024-09-16T03:46:47.060 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: [ RUN ] LibRadosListNP.ListObjectsError 2024-09-16T03:46:47.060 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: [ OK ] LibRadosListNP.ListObjectsError (3013 ms) 2024-09-16T03:46:47.061 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: [----------] 1 test from LibRadosListNP (3013 ms total) 2024-09-16T03:46:47.061 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: 2024-09-16T03:46:47.061 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: [----------] Global test environment tear-down 2024-09-16T03:46:47.061 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: [==========] 11 tests from 3 test suites ran. (197461 ms total) 2024-09-16T03:46:47.061 INFO:tasks.workunit.client.0.smithi154.stdout: api_list: [ PASSED ] 11 tests. 2024-09-16T03:46:49.126 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-16T03:46:49.126 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-09-16T03:46:49.126 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-16T03:46:49.127 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-16T03:46:49.130 INFO:tasks.ceph.mon.a:Started 2024-09-16T03:46:49.130 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T03:46:49.130 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:46:49.257 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.252+0000 7f857fa18640 10 monclient: get_monmap_and_config 2024-09-16T03:46:49.258 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.252+0000 7f857fa18640 10 monclient: build_initial_monmap 2024-09-16T03:46:49.258 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.252+0000 7f857fa18640 10 monclient: monmap: 2024-09-16T03:46:49.258 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:46:49.258 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:46:49.258 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:46:49.256682+0000 2024-09-16T03:46:49.258 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:46:49.256682+0000 2024-09-16T03:46:49.258 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:46:49.258 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:46:49.259 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:46:49.259 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:46:49.259 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:46:49.259 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:46:49.259 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:46:49.259 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:46:49.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.252+0000 7f857fa18640 1 Processor -- start 2024-09-16T03:46:49.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.252+0000 7f857fa18640 1 -- start start 2024-09-16T03:46:49.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.252+0000 7f857fa18640 10 monclient: init 2024-09-16T03:46:49.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.252+0000 7f857fa18640 10 monclient: _reopen_session rank -1 2024-09-16T03:46:49.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.252+0000 7f857fa18640 10 monclient: _add_conns ranks=[3,4,0,2,1] 2024-09-16T03:46:49.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.252+0000 7f857fa18640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f857806db90 0x7f85781340c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:46:49.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.252+0000 7f857fa18640 10 monclient(hunting): picked mon.noname-a con 0x7f857806db90 addr v2:172.21.15.42:3300/0 2024-09-16T03:46:49.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.252+0000 7f857fa18640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f85780a37f0 0x7f85780a3c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:46:49.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.252+0000 7f857fa18640 10 monclient(hunting): picked mon.noname-c con 0x7f85780a37f0 addr v2:172.21.15.154:3300/0 2024-09-16T03:46:49.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.252+0000 7f857fa18640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8578136e10 addr v1:172.21.15.42:6789/0 2024-09-16T03:46:49.271 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.252+0000 7f857fa18640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f857806ece0 con 0x7f8578136e10 2024-09-16T03:46:49.271 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.252+0000 7f857fa18640 10 monclient(hunting): start opening mon connection 2024-09-16T03:46:49.271 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.252+0000 7f857fa18640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f857806df70 con 0x7f857806db90 2024-09-16T03:46:49.271 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.252+0000 7f857fa18640 10 monclient(hunting): start opening mon connection 2024-09-16T03:46:49.271 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.252+0000 7f857fa18640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f857809bf80 con 0x7f85780a37f0 2024-09-16T03:46:49.271 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.252+0000 7f857fa18640 10 monclient(hunting): _renew_subs 2024-09-16T03:46:49.271 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.252+0000 7f857fa18640 10 monclient(hunting): authenticate will time out at 1256.225715s 2024-09-16T03:46:49.271 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.252+0000 7f857d7b4640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f857806db90 0x7f85781340c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:46:49.271 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.252+0000 7f857dfb5640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f8578136e10 0x7f8578137220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:59540/0 (socket says 172.21.15.42:59540) 2024-09-16T03:46:49.271 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.252+0000 7f857dfb5640 1 -- 172.21.15.42:0/3145371038 learned_addr learned my addr 172.21.15.42:0/3145371038 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:46:49.271 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.252+0000 7f857d7b4640 10 monclient(hunting): get_auth_request con 0x7f857806db90 auth_method 0 2024-09-16T03:46:49.271 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.252+0000 7f857d7b4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:46:49.272 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.252+0000 7f857d7b4640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:46:49.272 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.252+0000 7f857d7b4640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:46:49.272 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.252+0000 7f857d7b4640 1 -- 172.21.15.42:0/3145371038 >> v2:172.21.15.42:3300/0 conn(0x7f857806db90 msgr2=0x7f85781340c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T03:46:49.272 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.252+0000 7f857d7b4640 1 -- 172.21.15.42:0/3145371038 >> v2:172.21.15.42:3300/0 conn(0x7f857806db90 msgr2=0x7f85781340c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:46:49.272 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.252+0000 7f857d7b4640 1 --2- 172.21.15.42:0/3145371038 >> v2:172.21.15.42:3300/0 conn(0x7f857806db90 0x7f85781340c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:46:49.272 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.252+0000 7f857d7b4640 1 --2- 172.21.15.42:0/3145371038 >> v2:172.21.15.42:3300/0 conn(0x7f857806db90 0x7f85781340c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:46:49.272 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.252+0000 7f857cfb3640 1 --2- 172.21.15.42:0/3145371038 >> v2:172.21.15.154:3300/0 conn(0x7f85780a37f0 0x7f85780a3c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:46:49.272 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.252+0000 7f857cfb3640 10 monclient(hunting): get_auth_request con 0x7f85780a37f0 auth_method 0 2024-09-16T03:46:49.272 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.252+0000 7f857cfb3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:46:49.272 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.252+0000 7f857cfb3640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:46:49.272 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.252+0000 7f857cfb3640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:46:49.272 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.252+0000 7f857cfb3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:46:49.272 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.252+0000 7f857cfb3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:46:49.273 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.252+0000 7f857cfb3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:46:49.273 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.252+0000 7f857cfb3640 10 monclient(hunting): handle_auth_done global_id 754252 payload 274 2024-09-16T03:46:49.273 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.252+0000 7f857cfb3640 1 -- 172.21.15.42:0/3145371038 >> v2:172.21.15.42:3300/0 conn(0x7f857806db90 msgr2=0x7f85781340c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:46:49.273 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.252+0000 7f857cfb3640 1 --2- 172.21.15.42:0/3145371038 >> v2:172.21.15.42:3300/0 conn(0x7f857806db90 0x7f85781340c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:46:49.273 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.252+0000 7f857cfb3640 1 -- 172.21.15.42:0/3145371038 >> v1:172.21.15.42:6789/0 conn(0x7f8578136e10 legacy=0x7f8578137220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:46:49.273 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.252+0000 7f857cfb3640 10 monclient: _finish_hunting 0 2024-09-16T03:46:49.273 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.252+0000 7f857cfb3640 1 monclient: found mon.noname-c 2024-09-16T03:46:49.273 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.252+0000 7f857cfb3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:46:49.273 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.252+0000 7f857cfb3640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.154:3300/0 2024-09-16T03:46:49.273 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.252+0000 7f857cfb3640 1 -- 172.21.15.42:0/3145371038 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8578134780 con 0x7f85780a37f0 2024-09-16T03:46:49.273 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.256+0000 7f857cfb3640 1 --2- 172.21.15.42:0/3145371038 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f85780a37f0 0x7f85780a3c00 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f857806ce60 tx=0x7f856801cda0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2c41c88ef4a5d708 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:46:49.273 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.256+0000 7f856ffff640 1 -- 172.21.15.42:0/3145371038 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85680b0070 con 0x7f85780a37f0 2024-09-16T03:46:49.274 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.256+0000 7f856ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:46:49.274 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.256+0000 7f856ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:46:49.274 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.256+0000 7f856ffff640 10 monclient: dump: 2024-09-16T03:46:49.274 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:46:49.274 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:46:49.274 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:49.274 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:49.274 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:46:49.274 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:46:49.274 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:46:49.274 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:46:49.275 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:46:49.275 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:46:49.275 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.256+0000 7f856ffff640 10 monclient: _finish_auth 0 2024-09-16T03:46:49.275 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.256+0000 7f856ffff640 10 monclient: _check_auth_tickets 2024-09-16T03:46:49.275 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.256+0000 7f856ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:46:49.275 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.256+0000 7f857fa18640 5 monclient: authenticate success, global_id 754252 2024-09-16T03:46:49.275 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.256+0000 7f857fa18640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:46:49.275 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.256+0000 7f856ffff640 1 -- 172.21.15.42:0/3145371038 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f85680ab600 con 0x7f85780a37f0 2024-09-16T03:46:49.275 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.256+0000 7f856ffff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:46:49.275 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.256+0000 7f857fa18640 10 monclient: get_monmap_and_config success 2024-09-16T03:46:49.275 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.256+0000 7f857fa18640 4 set_mon_vals no callback set 2024-09-16T03:46:49.275 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.256+0000 7f856ffff640 1 -- 172.21.15.42:0/3145371038 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85680ab9d0 con 0x7f85780a37f0 2024-09-16T03:46:49.275 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.256+0000 7f856ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:46:49.276 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.256+0000 7f856ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:46:49.276 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.256+0000 7f856ffff640 10 monclient: dump: 2024-09-16T03:46:49.276 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:46:49.276 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:46:49.276 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:49.276 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:49.276 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:46:49.276 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:46:49.276 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:46:49.276 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:46:49.276 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:46:49.276 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:46:49.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.256+0000 7f857fa18640 10 monclient: shutdown 2024-09-16T03:46:49.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.256+0000 7f857fa18640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:46:49.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.256+0000 7f857fa18640 1 -- 172.21.15.42:0/3145371038 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f85780a37f0 msgr2=0x7f85780a3c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:46:49.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.256+0000 7f857fa18640 1 --2- 172.21.15.42:0/3145371038 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f85780a37f0 0x7f85780a3c00 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f857806ce60 tx=0x7f856801cda0 comp rx=0 tx=0).stop 2024-09-16T03:46:49.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.256+0000 7f857fa18640 1 -- 172.21.15.42:0/3145371038 shutdown_connections 2024-09-16T03:46:49.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.256+0000 7f857fa18640 1 --2- 172.21.15.42:0/3145371038 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f85780a37f0 0x7f85780a3c00 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:46:49.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.256+0000 7f857fa18640 1 --2- 172.21.15.42:0/3145371038 >> v2:172.21.15.42:3300/0 conn(0x7f857806db90 0x7f85781340c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:46:49.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.256+0000 7f857fa18640 1 -- 172.21.15.42:0/3145371038 >> 172.21.15.42:0/3145371038 conn(0x7f85780a9fb0 msgr2=0x7f85780aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:46:49.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.256+0000 7f857fa18640 1 -- 172.21.15.42:0/3145371038 shutdown_connections 2024-09-16T03:46:49.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.256+0000 7f857fa18640 1 -- 172.21.15.42:0/3145371038 wait complete. 2024-09-16T03:46:49.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.256+0000 7f857fa18640 10 monclient: build_initial_monmap 2024-09-16T03:46:49.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.256+0000 7f857fa18640 10 monclient: monmap: 2024-09-16T03:46:49.277 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:46:49.277 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:46:49.278 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:46:49.278 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:46:49.278 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:46:49.278 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:46:49.278 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:46:49.278 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:46:49.278 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:46:49.278 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:46:49.278 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.256+0000 7f857fa18640 1 librados: starting msgr at 2024-09-16T03:46:49.278 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.256+0000 7f857fa18640 1 librados: starting objecter 2024-09-16T03:46:49.278 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.256+0000 7f857fa18640 1 Processor -- start 2024-09-16T03:46:49.278 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.256+0000 7f857fa18640 1 -- start start 2024-09-16T03:46:49.278 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.256+0000 7f857fa18640 1 librados: setting wanted keys 2024-09-16T03:46:49.279 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.256+0000 7f857fa18640 1 librados: calling monclient init 2024-09-16T03:46:49.279 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.256+0000 7f857fa18640 10 monclient: init 2024-09-16T03:46:49.279 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.256+0000 7f857fa18640 10 monclient: _reopen_session rank -1 2024-09-16T03:46:49.279 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.256+0000 7f857fa18640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:46:49.279 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.256+0000 7f857fa18640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f857806db90 0x7f857812d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:46:49.279 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.256+0000 7f857fa18640 10 monclient(hunting): picked mon.noname-b con 0x7f857806db90 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:46:49.279 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.256+0000 7f857fa18640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f85780a37f0 0x7f857812dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:46:49.279 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.256+0000 7f857fa18640 10 monclient(hunting): picked mon.noname-a con 0x7f85780a37f0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:46:49.279 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.256+0000 7f857fa18640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f8578136e10 0x7f85781cf170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:46:49.279 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.256+0000 7f857fa18640 10 monclient(hunting): picked mon.noname-c con 0x7f8578136e10 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:46:49.279 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.256+0000 7f857fa18640 10 monclient(hunting): start opening mon connection 2024-09-16T03:46:49.279 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.256+0000 7f857fa18640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f8578141d60 con 0x7f85780a37f0 2024-09-16T03:46:49.279 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.256+0000 7f857fa18640 10 monclient(hunting): start opening mon connection 2024-09-16T03:46:49.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.256+0000 7f857fa18640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f8578141be0 con 0x7f857806db90 2024-09-16T03:46:49.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.256+0000 7f857fa18640 10 monclient(hunting): start opening mon connection 2024-09-16T03:46:49.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.256+0000 7f857fa18640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f8578141ee0 con 0x7f8578136e10 2024-09-16T03:46:49.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.256+0000 7f857fa18640 10 monclient(hunting): _renew_subs 2024-09-16T03:46:49.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.256+0000 7f857fa18640 10 monclient(hunting): authenticate will time out at 1256.231403s 2024-09-16T03:46:49.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.256+0000 7f857dfb5640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f8578136e10 0x7f85781cf170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:46:49.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.256+0000 7f857cfb3640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f85780a37f0 0x7f857812dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:46:49.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.256+0000 7f857cfb3640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f85780a37f0 0x7f857812dbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:60670/0 (socket says 172.21.15.42:60670) 2024-09-16T03:46:49.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.256+0000 7f857dfb5640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f8578136e10 0x7f85781cf170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:38962/0 (socket says 172.21.15.42:38962) 2024-09-16T03:46:49.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.256+0000 7f857cfb3640 1 -- 172.21.15.42:0/3368316972 learned_addr learned my addr 172.21.15.42:0/3368316972 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:46:49.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.256+0000 7f857cfb3640 10 monclient(hunting): get_auth_request con 0x7f85780a37f0 auth_method 0 2024-09-16T03:46:49.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.256+0000 7f857cfb3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:46:49.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.256+0000 7f857cfb3640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:46:49.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.256+0000 7f857cfb3640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:46:49.281 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.256+0000 7f857dfb5640 10 monclient(hunting): get_auth_request con 0x7f8578136e10 auth_method 0 2024-09-16T03:46:49.281 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.256+0000 7f857dfb5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:46:49.281 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.256+0000 7f857dfb5640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:46:49.281 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.256+0000 7f857dfb5640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:46:49.281 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.256+0000 7f857dfb5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:46:49.281 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.256+0000 7f857dfb5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:46:49.281 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.256+0000 7f857d7b4640 1 --2- 172.21.15.42:0/3368316972 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f857806db90 0x7f857812d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:46:49.281 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.256+0000 7f857cfb3640 1 -- 172.21.15.42:0/3368316972 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f85780a37f0 msgr2=0x7f857812dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-16T03:46:49.281 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.256+0000 7f857cfb3640 1 -- 172.21.15.42:0/3368316972 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f85780a37f0 msgr2=0x7f857812dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T03:46:49.281 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.256+0000 7f857cfb3640 1 --2- 172.21.15.42:0/3368316972 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f85780a37f0 0x7f857812dbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:46:49.281 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.256+0000 7f857dfb5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:46:49.281 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.256+0000 7f857cfb3640 1 --2- 172.21.15.42:0/3368316972 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f85780a37f0 0x7f857812dbd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:46:49.281 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.256+0000 7f857d7b4640 10 monclient(hunting): get_auth_request con 0x7f857806db90 auth_method 0 2024-09-16T03:46:49.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.256+0000 7f857d7b4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:46:49.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.256+0000 7f857d7b4640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:46:49.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.256+0000 7f857d7b4640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:46:49.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.256+0000 7f857dfb5640 10 monclient(hunting): handle_auth_done global_id 13661 payload 824 2024-09-16T03:46:49.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.256+0000 7f857dfb5640 1 -- 172.21.15.42:0/3368316972 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f857806db90 msgr2=0x7f857812d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:46:49.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.256+0000 7f857dfb5640 1 --2- 172.21.15.42:0/3368316972 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f857806db90 0x7f857812d690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:46:49.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.256+0000 7f857dfb5640 1 -- 172.21.15.42:0/3368316972 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f85780a37f0 msgr2=0x7f857812dbd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:46:49.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.256+0000 7f857dfb5640 1 --2- 172.21.15.42:0/3368316972 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f85780a37f0 0x7f857812dbd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:46:49.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.256+0000 7f857dfb5640 10 monclient: _finish_hunting 0 2024-09-16T03:46:49.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.256+0000 7f857dfb5640 1 monclient: found mon.noname-c 2024-09-16T03:46:49.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.256+0000 7f857dfb5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:46:49.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.256+0000 7f857dfb5640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:46:49.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.256+0000 7f857dfb5640 1 -- 172.21.15.42:0/3368316972 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f857812e110 con 0x7f8578136e10 2024-09-16T03:46:49.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.260+0000 7f857dfb5640 1 --2- 172.21.15.42:0/3368316972 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f8578136e10 0x7f85781cf170 secure :-1 s=READY pgs=2459 cs=0 l=1 rev1=1 crypto rx=0x7f857000caa0 tx=0x7f857000cf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:46:49.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.260+0000 7f856dffb640 10 client.?.objecter ms_handle_connect 0x7f8578136e10 2024-09-16T03:46:49.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.260+0000 7f856dffb640 10 client.?.objecter resend_mon_ops 2024-09-16T03:46:49.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.260+0000 7f856dffb640 1 -- 172.21.15.42:0/3368316972 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8570017070 con 0x7f8578136e10 2024-09-16T03:46:49.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.260+0000 7f856dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:46:49.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.260+0000 7f856dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:46:49.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.260+0000 7f856dffb640 10 monclient: dump: 2024-09-16T03:46:49.283 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:46:49.283 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:46:49.283 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:49.283 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:49.283 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:46:49.283 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:46:49.283 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:46:49.284 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:46:49.284 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:46:49.284 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:46:49.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.260+0000 7f856dffb640 10 monclient: _finish_auth 0 2024-09-16T03:46:49.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.260+0000 7f856dffb640 10 monclient: _check_auth_tickets 2024-09-16T03:46:49.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.260+0000 7f856dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:46:49.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.260+0000 7f857fa18640 5 monclient: authenticate success, global_id 13661 2024-09-16T03:46:49.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.260+0000 7f856dffb640 1 -- 172.21.15.42:0/3368316972 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f857000ac00 con 0x7f8578136e10 2024-09-16T03:46:49.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.260+0000 7f856dffb640 10 monclient: handle_config config(0 keys) 2024-09-16T03:46:49.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.260+0000 7f857fa18640 10 monclient: _renew_subs 2024-09-16T03:46:49.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.260+0000 7f856dffb640 1 -- 172.21.15.42:0/3368316972 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8570012400 con 0x7f8578136e10 2024-09-16T03:46:49.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.260+0000 7f857fa18640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:46:49.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.260+0000 7f857fa18640 1 -- 172.21.15.42:0/3368316972 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85780a26d0 con 0x7f8578136e10 2024-09-16T03:46:49.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.260+0000 7f856f7fe640 4 set_mon_vals no callback set 2024-09-16T03:46:49.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.260+0000 7f856dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:46:49.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.260+0000 7f856dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:46:49.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.260+0000 7f856dffb640 10 monclient: dump: 2024-09-16T03:46:49.285 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:46:49.285 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:46:49.285 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:49.285 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:46:49.285 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:46:49.285 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:46:49.285 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:46:49.286 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:46:49.286 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:46:49.286 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:46:49.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.260+0000 7f857fa18640 10 client.13661.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:46:49.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.260+0000 7f857fa18640 10 monclient: _renew_subs 2024-09-16T03:46:49.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.260+0000 7f857fa18640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:46:49.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.260+0000 7f857fa18640 1 -- 172.21.15.42:0/3368316972 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f8578133550 con 0x7f8578136e10 2024-09-16T03:46:49.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.260+0000 7f857fa18640 1 librados: init done 2024-09-16T03:46:49.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.260+0000 7f857fa18640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:46:49.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.260+0000 7f857fa18640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:46:49.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.260+0000 7f857fa18640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:46:49.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.260+0000 7f857fa18640 1 -- 172.21.15.42:0/3368316972 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f85781340c0 con 0x7f8578136e10 2024-09-16T03:46:49.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.260+0000 7f856dffb640 1 -- 172.21.15.42:0/3368316972 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f857001a070 con 0x7f8578136e10 2024-09-16T03:46:49.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.260+0000 7f856dffb640 1 --2- 172.21.15.42:0/3368316972 >> v2:172.21.15.154:6812/254845342 conn(0x7f855c037810 0x7f855c039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:46:49.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.260+0000 7f856dffb640 1 -- 172.21.15.42:0/3368316972 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(145..145 src has 1..145) ==== 7299+0+0 (secure 0 0 0) 0x7f8570049240 con 0x7f8578136e10 2024-09-16T03:46:49.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.260+0000 7f856dffb640 10 client.13661.objecter ms_dispatch 0x7f8578000ba0 osd_map(145..145 src has 1..145) 2024-09-16T03:46:49.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.260+0000 7f856dffb640 3 client.13661.objecter handle_osd_map got epochs [145,145] > 0 2024-09-16T03:46:49.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.260+0000 7f856dffb640 3 client.13661.objecter handle_osd_map decoding full epoch 145 2024-09-16T03:46:49.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.260+0000 7f856dffb640 20 client.13661.objecter dump_active .. 0 homeless 2024-09-16T03:46:49.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.260+0000 7f857d7b4640 1 --2- 172.21.15.42:0/3368316972 >> v2:172.21.15.154:6812/254845342 conn(0x7f855c037810 0x7f855c039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:46:49.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.260+0000 7f857d7b4640 10 monclient: get_auth_request con 0x7f855c037810 auth_method 0 2024-09-16T03:46:49.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.260+0000 7f857d7b4640 1 --2- 172.21.15.42:0/3368316972 >> v2:172.21.15.154:6812/254845342 conn(0x7f855c037810 0x7f855c039cd0 secure :-1 s=READY pgs=3426 cs=0 l=1 rev1=1 crypto rx=0x7f8574006aa0 tx=0x7f8574006a30 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:46:49.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.260+0000 7f856dffb640 10 client.13661.objecter ms_handle_connect 0x7f855c037810 2024-09-16T03:46:49.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.264+0000 7f856dffb640 1 -- 172.21.15.42:0/3368316972 <== mon.2 v2:172.21.15.42:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f857001a520 con 0x7f8578136e10 2024-09-16T03:46:49.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.264+0000 7f856dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:46:49.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.264+0000 7f856dffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:46:49.502 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.496+0000 7f857fa18640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:46:49.502 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.496+0000 7f857fa18640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:46:49.502 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.496+0000 7f857fa18640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:46:49.502 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.496+0000 7f857fa18640 1 -- 172.21.15.42:0/3368316972 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f85780a0830 con 0x7f8578136e10 2024-09-16T03:46:49.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.496+0000 7f856dffb640 1 -- 172.21.15.42:0/3368316972 <== mon.2 v2:172.21.15.42:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1555 (secure 0 0 0) 0x7f8570011df0 con 0x7f8578136e10 2024-09-16T03:46:49.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.496+0000 7f856dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:46:49.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.496+0000 7f856dffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:46:49.503 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":36,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":21,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:46:49.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.500+0000 7f857fa18640 10 librados: watch_flush enter 2024-09-16T03:46:49.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.500+0000 7f857fa18640 10 librados: watch_flush exit 2024-09-16T03:46:49.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.500+0000 7f857fa18640 20 client.13661.objecter shutdown clearing up homeless session... 2024-09-16T03:46:49.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.500+0000 7f857fa18640 10 client.13661.objecter successfully canceled tick 2024-09-16T03:46:49.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.500+0000 7f857fa18640 1 -- 172.21.15.42:0/3368316972 >> v2:172.21.15.154:6812/254845342 conn(0x7f855c037810 msgr2=0x7f855c039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:46:49.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.500+0000 7f857fa18640 1 --2- 172.21.15.42:0/3368316972 >> v2:172.21.15.154:6812/254845342 conn(0x7f855c037810 0x7f855c039cd0 secure :-1 s=READY pgs=3426 cs=0 l=1 rev1=1 crypto rx=0x7f8574006aa0 tx=0x7f8574006a30 comp rx=0 tx=0).stop 2024-09-16T03:46:49.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.500+0000 7f857fa18640 10 monclient: shutdown 2024-09-16T03:46:49.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.500+0000 7f857fa18640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:46:49.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.500+0000 7f857fa18640 1 -- 172.21.15.42:0/3368316972 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f8578136e10 msgr2=0x7f85781cf170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:46:49.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.500+0000 7f857fa18640 1 --2- 172.21.15.42:0/3368316972 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f8578136e10 0x7f85781cf170 secure :-1 s=READY pgs=2459 cs=0 l=1 rev1=1 crypto rx=0x7f857000caa0 tx=0x7f857000cf60 comp rx=0 tx=0).stop 2024-09-16T03:46:49.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.500+0000 7f857fa18640 1 -- 172.21.15.42:0/3368316972 shutdown_connections 2024-09-16T03:46:49.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.500+0000 7f857fa18640 1 --2- 172.21.15.42:0/3368316972 >> v2:172.21.15.154:6812/254845342 conn(0x7f855c037810 0x7f855c039cd0 unknown :-1 s=CLOSED pgs=3426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:46:49.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.500+0000 7f857fa18640 1 --2- 172.21.15.42:0/3368316972 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f8578136e10 0x7f85781cf170 unknown :-1 s=CLOSED pgs=2459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:46:49.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.500+0000 7f857fa18640 1 --2- 172.21.15.42:0/3368316972 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f85780a37f0 0x7f857812dbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:46:49.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.500+0000 7f857fa18640 1 --2- 172.21.15.42:0/3368316972 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f857806db90 0x7f857812d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:46:49.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.500+0000 7f857fa18640 1 -- 172.21.15.42:0/3368316972 >> 172.21.15.42:0/3368316972 conn(0x7f85780a9fb0 msgr2=0x7f85781321f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:46:49.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.500+0000 7f857fa18640 1 -- 172.21.15.42:0/3368316972 shutdown_connections 2024-09-16T03:46:49.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.500+0000 7f857fa18640 1 -- 172.21.15.42:0/3368316972 wait complete. 2024-09-16T03:46:49.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:49.500+0000 7f857fa18640 1 librados: shutdown 2024-09-16T03:46:52.523 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:46:52.658 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:52.652+0000 7ff1a1264640 10 monclient: get_monmap_and_config 2024-09-16T03:46:52.658 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:52.652+0000 7ff1a1264640 10 monclient: build_initial_monmap 2024-09-16T03:46:52.658 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:52.652+0000 7ff1a1264640 10 monclient: monmap: 2024-09-16T03:46:52.659 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:46:52.659 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:46:52.659 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:46:52.657726+0000 2024-09-16T03:46:52.659 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:46:52.657726+0000 2024-09-16T03:46:52.659 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:46:52.659 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:46:52.659 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:46:52.659 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:46:52.659 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:46:52.659 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:46:52.659 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:46:52.659 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:46:52.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:52.652+0000 7ff1a1264640 1 Processor -- start 2024-09-16T03:46:52.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:52.652+0000 7ff1a1264640 1 -- start start 2024-09-16T03:46:52.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:52.652+0000 7ff1a1264640 10 monclient: init 2024-09-16T03:46:52.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:52.652+0000 7ff1a1264640 10 monclient: _reopen_session rank -1 2024-09-16T03:46:52.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:52.652+0000 7ff1a1264640 10 monclient: _add_conns ranks=[0,4,1,2,3] 2024-09-16T03:46:52.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:52.652+0000 7ff1a1264640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff19c0a1330 addr v1:172.21.15.42:6789/0 2024-09-16T03:46:52.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:52.652+0000 7ff1a1264640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7ff19c06dac0 0x7ff19c13a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:46:52.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:52.652+0000 7ff1a1264640 10 monclient(hunting): picked mon.noname-c con 0x7ff19c06dac0 addr v2:172.21.15.154:3300/0 2024-09-16T03:46:52.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:52.652+0000 7ff1a1264640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff19c0a1e20 addr v1:172.21.15.154:6789/0 2024-09-16T03:46:52.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:52.652+0000 7ff1a1264640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff19c09c090 con 0x7ff19c0a1330 2024-09-16T03:46:52.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:52.652+0000 7ff1a1264640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff19c09f610 con 0x7ff19c0a1e20 2024-09-16T03:46:52.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:52.652+0000 7ff1a1264640 10 monclient(hunting): start opening mon connection 2024-09-16T03:46:52.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:52.652+0000 7ff1a1264640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7ff19c09f8f0 con 0x7ff19c06dac0 2024-09-16T03:46:52.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:52.652+0000 7ff1a1264640 10 monclient(hunting): _renew_subs 2024-09-16T03:46:52.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:52.652+0000 7ff1a1264640 10 monclient(hunting): authenticate will time out at 1259.627179s 2024-09-16T03:46:52.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:52.652+0000 7ff19affd640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7ff19c0a1330 0x7ff19c0a1710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:59542/0 (socket says 172.21.15.42:59542) 2024-09-16T03:46:52.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:52.652+0000 7ff19affd640 1 -- 172.21.15.42:0/3464194141 learned_addr learned my addr 172.21.15.42:0/3464194141 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:46:52.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:52.652+0000 7ff19a7fc640 1 --2- 172.21.15.42:0/3464194141 >> v2:172.21.15.154:3300/0 conn(0x7ff19c06dac0 0x7ff19c13a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:46:52.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:52.652+0000 7ff19a7fc640 10 monclient(hunting): get_auth_request con 0x7ff19c06dac0 auth_method 0 2024-09-16T03:46:52.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:52.652+0000 7ff19a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:46:52.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:52.652+0000 7ff19a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:46:52.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:52.652+0000 7ff19a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:46:52.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:52.652+0000 7ff19a7fc640 1 -- 172.21.15.42:0/3464194141 >> v2:172.21.15.154:3300/0 conn(0x7ff19c06dac0 msgr2=0x7ff19c13a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T03:46:52.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:52.652+0000 7ff19a7fc640 1 -- 172.21.15.42:0/3464194141 >> v2:172.21.15.154:3300/0 conn(0x7ff19c06dac0 msgr2=0x7ff19c13a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:46:52.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:52.652+0000 7ff19a7fc640 1 --2- 172.21.15.42:0/3464194141 >> v2:172.21.15.154:3300/0 conn(0x7ff19c06dac0 0x7ff19c13a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:46:52.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:52.652+0000 7ff19a7fc640 1 --2- 172.21.15.42:0/3464194141 >> v2:172.21.15.154:3300/0 conn(0x7ff19c06dac0 0x7ff19c13a340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:46:52.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:52.860+0000 7ff19a7fc640 1 --2- 172.21.15.42:0/3464194141 >> v2:172.21.15.154:3300/0 conn(0x7ff19c06dac0 0x7ff19c13a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:46:52.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:52.860+0000 7ff19a7fc640 10 monclient(hunting): get_auth_request con 0x7ff19c06dac0 auth_method 0 2024-09-16T03:46:52.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:52.860+0000 7ff19a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:46:52.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:52.860+0000 7ff19a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:46:52.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:52.860+0000 7ff19a7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:46:52.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:52.860+0000 7ff19a7fc640 1 -- 172.21.15.42:0/3464194141 >> v2:172.21.15.154:3300/0 conn(0x7ff19c06dac0 msgr2=0x7ff19c13a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T03:46:52.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:52.860+0000 7ff19a7fc640 1 -- 172.21.15.42:0/3464194141 >> v2:172.21.15.154:3300/0 conn(0x7ff19c06dac0 msgr2=0x7ff19c13a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:46:52.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:52.860+0000 7ff19a7fc640 1 --2- 172.21.15.42:0/3464194141 >> v2:172.21.15.154:3300/0 conn(0x7ff19c06dac0 0x7ff19c13a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:46:52.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:52.860+0000 7ff19a7fc640 1 --2- 172.21.15.42:0/3464194141 >> v2:172.21.15.154:3300/0 conn(0x7ff19c06dac0 0x7ff19c13a340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T03:46:53.269 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:53.264+0000 7ff19a7fc640 1 --2- 172.21.15.42:0/3464194141 >> v2:172.21.15.154:3300/0 conn(0x7ff19c06dac0 0x7ff19c13a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:46:53.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:53.264+0000 7ff19a7fc640 10 monclient(hunting): get_auth_request con 0x7ff19c06dac0 auth_method 0 2024-09-16T03:46:53.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:53.264+0000 7ff19a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:46:53.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:53.264+0000 7ff19a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:46:53.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:53.264+0000 7ff19a7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:46:53.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:53.264+0000 7ff19a7fc640 1 -- 172.21.15.42:0/3464194141 >> v2:172.21.15.154:3300/0 conn(0x7ff19c06dac0 msgr2=0x7ff19c13a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T03:46:53.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:53.264+0000 7ff19a7fc640 1 -- 172.21.15.42:0/3464194141 >> v2:172.21.15.154:3300/0 conn(0x7ff19c06dac0 msgr2=0x7ff19c13a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:46:53.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:53.264+0000 7ff19a7fc640 1 --2- 172.21.15.42:0/3464194141 >> v2:172.21.15.154:3300/0 conn(0x7ff19c06dac0 0x7ff19c13a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:46:53.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:53.264+0000 7ff19a7fc640 1 --2- 172.21.15.42:0/3464194141 >> v2:172.21.15.154:3300/0 conn(0x7ff19c06dac0 0x7ff19c13a340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T03:46:54.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:54.068+0000 7ff19a7fc640 1 --2- 172.21.15.42:0/3464194141 >> v2:172.21.15.154:3300/0 conn(0x7ff19c06dac0 0x7ff19c13a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:46:54.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:54.068+0000 7ff19a7fc640 10 monclient(hunting): get_auth_request con 0x7ff19c06dac0 auth_method 0 2024-09-16T03:46:54.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:54.068+0000 7ff19a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:46:54.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:54.068+0000 7ff19a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:46:54.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:54.068+0000 7ff19a7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:46:54.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:54.068+0000 7ff19a7fc640 1 -- 172.21.15.42:0/3464194141 >> v2:172.21.15.154:3300/0 conn(0x7ff19c06dac0 msgr2=0x7ff19c13a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T03:46:54.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:54.068+0000 7ff19a7fc640 1 -- 172.21.15.42:0/3464194141 >> v2:172.21.15.154:3300/0 conn(0x7ff19c06dac0 msgr2=0x7ff19c13a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:46:54.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:54.068+0000 7ff19a7fc640 1 --2- 172.21.15.42:0/3464194141 >> v2:172.21.15.154:3300/0 conn(0x7ff19c06dac0 0x7ff19c13a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:46:54.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:54.068+0000 7ff19a7fc640 1 --2- 172.21.15.42:0/3464194141 >> v2:172.21.15.154:3300/0 conn(0x7ff19c06dac0 0x7ff19c13a340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T03:46:55.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:55.656+0000 7ff198ff9640 10 monclient(hunting): tick 2024-09-16T03:46:55.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:55.656+0000 7ff198ff9640 10 monclient(hunting): _check_auth_tickets 2024-09-16T03:46:55.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:55.656+0000 7ff198ff9640 1 monclient(hunting): continuing hunt 2024-09-16T03:46:55.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:55.656+0000 7ff198ff9640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T03:46:55.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:55.656+0000 7ff198ff9640 1 -- 172.21.15.42:0/3464194141 >> v2:172.21.15.154:3300/0 conn(0x7ff19c06dac0 msgr2=0x7ff19c13a340 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:46:55.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:55.656+0000 7ff198ff9640 1 --2- 172.21.15.42:0/3464194141 >> v2:172.21.15.154:3300/0 conn(0x7ff19c06dac0 0x7ff19c13a340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:46:55.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:55.656+0000 7ff198ff9640 1 -- 172.21.15.42:0/3464194141 >> v1:172.21.15.154:6789/0 conn(0x7ff19c0a1e20 legacy=0x7ff19c13c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:46:55.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:55.656+0000 7ff198ff9640 1 -- 172.21.15.42:0/3464194141 >> v1:172.21.15.42:6789/0 conn(0x7ff19c0a1330 legacy=0x7ff19c0a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:46:55.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:55.656+0000 7ff198ff9640 10 monclient: _add_conns ranks=[3,2] 2024-09-16T03:46:55.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:55.656+0000 7ff198ff9640 1 --2- 172.21.15.42:0/3464194141 >> v2:172.21.15.42:3300/0 conn(0x7ff1880036f0 0x7ff188005b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:46:55.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:55.656+0000 7ff198ff9640 10 monclient(hunting): picked mon.noname-a con 0x7ff1880036f0 addr v2:172.21.15.42:3300/0 2024-09-16T03:46:55.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:55.656+0000 7ff198ff9640 1 --2- 172.21.15.42:0/3464194141 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff188006190 0x7ff188008630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:46:55.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:55.656+0000 7ff198ff9640 10 monclient(hunting): picked mon.noname-b con 0x7ff188006190 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:46:55.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:55.656+0000 7ff198ff9640 10 monclient(hunting): start opening mon connection 2024-09-16T03:46:55.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:55.656+0000 7ff198ff9640 1 -- 172.21.15.42:0/3464194141 --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7ff19c09f8f0 con 0x7ff1880036f0 2024-09-16T03:46:55.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:55.656+0000 7ff198ff9640 10 monclient(hunting): start opening mon connection 2024-09-16T03:46:55.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:55.656+0000 7ff198ff9640 1 -- 172.21.15.42:0/3464194141 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7ff188008d60 con 0x7ff188006190 2024-09-16T03:46:55.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:55.656+0000 7ff198ff9640 10 monclient(hunting): _renew_subs 2024-09-16T03:46:55.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:55.656+0000 7ff19affd640 1 --2- 172.21.15.42:0/3464194141 >> v2:172.21.15.42:3300/0 conn(0x7ff1880036f0 0x7ff188005b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:46:55.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:55.656+0000 7ff19a7fc640 1 --2- 172.21.15.42:0/3464194141 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff188006190 0x7ff188008630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:46:55.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:55.656+0000 7ff19affd640 10 monclient(hunting): get_auth_request con 0x7ff1880036f0 auth_method 0 2024-09-16T03:46:55.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:55.656+0000 7ff19affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:46:55.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:55.656+0000 7ff19affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:46:55.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:55.656+0000 7ff19affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:46:55.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:55.656+0000 7ff19a7fc640 10 monclient(hunting): get_auth_request con 0x7ff188006190 auth_method 0 2024-09-16T03:46:55.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:55.656+0000 7ff19a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:46:55.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:55.656+0000 7ff19a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:46:55.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:55.656+0000 7ff19a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:46:55.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:55.656+0000 7ff19a7fc640 1 -- 172.21.15.42:0/3464194141 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff188006190 msgr2=0x7ff188008630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T03:46:55.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:55.656+0000 7ff19affd640 1 -- 172.21.15.42:0/3464194141 >> v2:172.21.15.42:3300/0 conn(0x7ff1880036f0 msgr2=0x7ff188005b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T03:46:55.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:55.656+0000 7ff19affd640 1 -- 172.21.15.42:0/3464194141 >> v2:172.21.15.42:3300/0 conn(0x7ff1880036f0 msgr2=0x7ff188005b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:46:55.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:55.656+0000 7ff19a7fc640 1 -- 172.21.15.42:0/3464194141 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff188006190 msgr2=0x7ff188008630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:46:55.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:55.656+0000 7ff19affd640 1 --2- 172.21.15.42:0/3464194141 >> v2:172.21.15.42:3300/0 conn(0x7ff1880036f0 0x7ff188005b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:46:55.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:55.656+0000 7ff19a7fc640 1 --2- 172.21.15.42:0/3464194141 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff188006190 0x7ff188008630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:46:55.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:55.656+0000 7ff19affd640 1 --2- 172.21.15.42:0/3464194141 >> v2:172.21.15.42:3300/0 conn(0x7ff1880036f0 0x7ff188005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:46:55.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:55.656+0000 7ff19a7fc640 1 --2- 172.21.15.42:0/3464194141 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff188006190 0x7ff188008630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:46:55.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:55.860+0000 7ff19a7fc640 1 --2- 172.21.15.42:0/3464194141 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff188006190 0x7ff188008630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:46:55.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:55.860+0000 7ff19a7fc640 10 monclient(hunting): get_auth_request con 0x7ff188006190 auth_method 0 2024-09-16T03:46:55.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:55.860+0000 7ff19a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:46:55.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:55.860+0000 7ff19a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:46:55.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:55.860+0000 7ff19affd640 1 --2- 172.21.15.42:0/3464194141 >> v2:172.21.15.42:3300/0 conn(0x7ff1880036f0 0x7ff188005b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:46:55.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:55.860+0000 7ff19a7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:46:55.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:55.860+0000 7ff19affd640 10 monclient(hunting): get_auth_request con 0x7ff1880036f0 auth_method 0 2024-09-16T03:46:55.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:55.860+0000 7ff19affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:46:55.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:55.860+0000 7ff19affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:46:55.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:55.860+0000 7ff19affd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:46:55.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:55.860+0000 7ff19a7fc640 1 -- 172.21.15.42:0/3464194141 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff188006190 msgr2=0x7ff188008630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T03:46:55.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:55.860+0000 7ff19a7fc640 1 -- 172.21.15.42:0/3464194141 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff188006190 msgr2=0x7ff188008630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:46:55.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:55.860+0000 7ff19a7fc640 1 --2- 172.21.15.42:0/3464194141 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff188006190 0x7ff188008630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:46:55.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:55.860+0000 7ff19affd640 1 -- 172.21.15.42:0/3464194141 >> v2:172.21.15.42:3300/0 conn(0x7ff1880036f0 msgr2=0x7ff188005b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T03:46:55.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:55.860+0000 7ff19affd640 1 -- 172.21.15.42:0/3464194141 >> v2:172.21.15.42:3300/0 conn(0x7ff1880036f0 msgr2=0x7ff188005b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:46:55.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:55.860+0000 7ff19affd640 1 --2- 172.21.15.42:0/3464194141 >> v2:172.21.15.42:3300/0 conn(0x7ff1880036f0 0x7ff188005b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:46:55.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:55.860+0000 7ff19a7fc640 1 --2- 172.21.15.42:0/3464194141 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff188006190 0x7ff188008630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T03:46:55.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:55.860+0000 7ff19affd640 1 --2- 172.21.15.42:0/3464194141 >> v2:172.21.15.42:3300/0 conn(0x7ff1880036f0 0x7ff188005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T03:46:56.269 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:56.264+0000 7ff19affd640 1 --2- 172.21.15.42:0/3464194141 >> v2:172.21.15.42:3300/0 conn(0x7ff1880036f0 0x7ff188005b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:46:56.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:56.264+0000 7ff19a7fc640 1 --2- 172.21.15.42:0/3464194141 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff188006190 0x7ff188008630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:46:56.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:56.264+0000 7ff19a7fc640 10 monclient(hunting): get_auth_request con 0x7ff188006190 auth_method 0 2024-09-16T03:46:56.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:56.264+0000 7ff19a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:46:56.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:56.264+0000 7ff19a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:46:56.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:56.264+0000 7ff19a7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:46:56.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:56.264+0000 7ff19affd640 10 monclient(hunting): get_auth_request con 0x7ff1880036f0 auth_method 0 2024-09-16T03:46:56.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:56.264+0000 7ff19affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:46:56.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:56.264+0000 7ff19affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:46:56.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:56.264+0000 7ff19affd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:46:56.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:56.264+0000 7ff19a7fc640 1 -- 172.21.15.42:0/3464194141 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff188006190 msgr2=0x7ff188008630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T03:46:56.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:56.264+0000 7ff19a7fc640 1 -- 172.21.15.42:0/3464194141 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff188006190 msgr2=0x7ff188008630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:46:56.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:56.264+0000 7ff19a7fc640 1 --2- 172.21.15.42:0/3464194141 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff188006190 0x7ff188008630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:46:56.271 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:56.264+0000 7ff19affd640 1 -- 172.21.15.42:0/3464194141 >> v2:172.21.15.42:3300/0 conn(0x7ff1880036f0 msgr2=0x7ff188005b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T03:46:56.271 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:56.264+0000 7ff19affd640 1 -- 172.21.15.42:0/3464194141 >> v2:172.21.15.42:3300/0 conn(0x7ff1880036f0 msgr2=0x7ff188005b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:46:56.271 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:56.264+0000 7ff19affd640 1 --2- 172.21.15.42:0/3464194141 >> v2:172.21.15.42:3300/0 conn(0x7ff1880036f0 0x7ff188005b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:46:56.271 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:56.264+0000 7ff19a7fc640 1 --2- 172.21.15.42:0/3464194141 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff188006190 0x7ff188008630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T03:46:56.271 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:56.264+0000 7ff19affd640 1 --2- 172.21.15.42:0/3464194141 >> v2:172.21.15.42:3300/0 conn(0x7ff1880036f0 0x7ff188005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T03:46:57.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:57.068+0000 7ff19a7fc640 1 --2- 172.21.15.42:0/3464194141 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff188006190 0x7ff188008630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:46:57.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:57.068+0000 7ff19affd640 1 --2- 172.21.15.42:0/3464194141 >> v2:172.21.15.42:3300/0 conn(0x7ff1880036f0 0x7ff188005b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:46:57.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:57.068+0000 7ff19a7fc640 10 monclient(hunting): get_auth_request con 0x7ff188006190 auth_method 0 2024-09-16T03:46:57.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:57.068+0000 7ff19a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:46:57.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:57.068+0000 7ff19a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:46:57.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:57.068+0000 7ff19a7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:46:57.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:57.068+0000 7ff19affd640 10 monclient(hunting): get_auth_request con 0x7ff1880036f0 auth_method 0 2024-09-16T03:46:57.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:57.068+0000 7ff19affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:46:57.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:57.068+0000 7ff19affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:46:57.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:57.068+0000 7ff19affd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:46:57.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:57.068+0000 7ff19a7fc640 1 -- 172.21.15.42:0/3464194141 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff188006190 msgr2=0x7ff188008630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T03:46:57.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:57.068+0000 7ff19a7fc640 1 -- 172.21.15.42:0/3464194141 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff188006190 msgr2=0x7ff188008630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:46:57.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:57.068+0000 7ff19a7fc640 1 --2- 172.21.15.42:0/3464194141 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff188006190 0x7ff188008630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:46:57.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:57.068+0000 7ff19affd640 1 -- 172.21.15.42:0/3464194141 >> v2:172.21.15.42:3300/0 conn(0x7ff1880036f0 msgr2=0x7ff188005b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T03:46:57.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:57.068+0000 7ff19a7fc640 1 --2- 172.21.15.42:0/3464194141 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff188006190 0x7ff188008630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T03:46:57.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:57.068+0000 7ff19affd640 1 -- 172.21.15.42:0/3464194141 >> v2:172.21.15.42:3300/0 conn(0x7ff1880036f0 msgr2=0x7ff188005b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:46:57.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:57.068+0000 7ff19affd640 1 --2- 172.21.15.42:0/3464194141 >> v2:172.21.15.42:3300/0 conn(0x7ff1880036f0 0x7ff188005b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:46:57.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:57.068+0000 7ff19affd640 1 --2- 172.21.15.42:0/3464194141 >> v2:172.21.15.42:3300/0 conn(0x7ff1880036f0 0x7ff188005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T03:46:58.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:58.656+0000 7ff198ff9640 10 monclient(hunting): tick 2024-09-16T03:46:58.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:58.656+0000 7ff198ff9640 10 monclient(hunting): _check_auth_tickets 2024-09-16T03:46:58.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:58.656+0000 7ff198ff9640 1 monclient(hunting): continuing hunt 2024-09-16T03:46:58.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:58.656+0000 7ff198ff9640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T03:46:58.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:58.656+0000 7ff198ff9640 1 -- 172.21.15.42:0/3464194141 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff188006190 msgr2=0x7ff188008630 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:46:58.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:58.656+0000 7ff198ff9640 1 --2- 172.21.15.42:0/3464194141 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff188006190 0x7ff188008630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:46:58.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:58.656+0000 7ff198ff9640 1 -- 172.21.15.42:0/3464194141 >> v2:172.21.15.42:3300/0 conn(0x7ff1880036f0 msgr2=0x7ff188005b50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:46:58.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:58.656+0000 7ff198ff9640 1 --2- 172.21.15.42:0/3464194141 >> v2:172.21.15.42:3300/0 conn(0x7ff1880036f0 0x7ff188005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:46:58.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:58.656+0000 7ff198ff9640 10 monclient: _add_conns ranks=[1,0,3,4,2] 2024-09-16T03:46:58.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:58.656+0000 7ff198ff9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff19c0a1e20 addr v1:172.21.15.154:6789/0 2024-09-16T03:46:58.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:58.656+0000 7ff19b7fe640 1 -- 172.21.15.42:0/3464194141 reap_dead start 2024-09-16T03:46:58.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:58.656+0000 7ff198ff9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff19c0a1330 addr v1:172.21.15.42:6789/0 2024-09-16T03:46:58.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:58.656+0000 7ff198ff9640 1 --2- 172.21.15.42:0/3464194141 >> v2:172.21.15.42:3300/0 conn(0x7ff1880036f0 0x7ff188010090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:46:58.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:58.656+0000 7ff198ff9640 10 monclient(hunting): picked mon.noname-a con 0x7ff1880036f0 addr v2:172.21.15.42:3300/0 2024-09-16T03:46:58.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:58.656+0000 7ff198ff9640 1 -- 172.21.15.42:0/3464194141 --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff19c09fbd0 con 0x7ff19c0a1330 2024-09-16T03:46:58.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:58.656+0000 7ff198ff9640 1 -- 172.21.15.42:0/3464194141 --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff188005c60 con 0x7ff19c0a1e20 2024-09-16T03:46:58.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:58.656+0000 7ff198ff9640 10 monclient(hunting): start opening mon connection 2024-09-16T03:46:58.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:58.656+0000 7ff198ff9640 1 -- 172.21.15.42:0/3464194141 --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7ff19c09f8f0 con 0x7ff1880036f0 2024-09-16T03:46:58.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:58.656+0000 7ff198ff9640 10 monclient(hunting): _renew_subs 2024-09-16T03:46:58.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:58.656+0000 7ff19b7fe640 1 --2- 172.21.15.42:0/3464194141 >> v2:172.21.15.42:3300/0 conn(0x7ff1880036f0 0x7ff188010090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:46:58.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:58.656+0000 7ff19b7fe640 10 monclient(hunting): get_auth_request con 0x7ff1880036f0 auth_method 0 2024-09-16T03:46:58.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:58.656+0000 7ff19b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:46:58.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:58.656+0000 7ff19b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:46:58.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:58.656+0000 7ff19b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:46:58.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:58.656+0000 7ff19b7fe640 1 -- 172.21.15.42:0/3464194141 >> v2:172.21.15.42:3300/0 conn(0x7ff1880036f0 msgr2=0x7ff188010090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T03:46:58.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:58.656+0000 7ff19b7fe640 1 -- 172.21.15.42:0/3464194141 >> v2:172.21.15.42:3300/0 conn(0x7ff1880036f0 msgr2=0x7ff188010090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:46:58.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:58.656+0000 7ff19b7fe640 1 --2- 172.21.15.42:0/3464194141 >> v2:172.21.15.42:3300/0 conn(0x7ff1880036f0 0x7ff188010090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:46:58.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:58.656+0000 7ff19b7fe640 1 --2- 172.21.15.42:0/3464194141 >> v2:172.21.15.42:3300/0 conn(0x7ff1880036f0 0x7ff188010090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:46:58.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:58.860+0000 7ff19b7fe640 1 --2- 172.21.15.42:0/3464194141 >> v2:172.21.15.42:3300/0 conn(0x7ff1880036f0 0x7ff188010090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:46:58.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:58.860+0000 7ff19b7fe640 10 monclient(hunting): get_auth_request con 0x7ff1880036f0 auth_method 0 2024-09-16T03:46:58.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:58.860+0000 7ff19b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:46:58.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:58.860+0000 7ff19b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:46:58.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:58.860+0000 7ff19b7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:46:58.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:58.860+0000 7ff19b7fe640 1 -- 172.21.15.42:0/3464194141 >> v2:172.21.15.42:3300/0 conn(0x7ff1880036f0 msgr2=0x7ff188010090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T03:46:58.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:58.860+0000 7ff19b7fe640 1 -- 172.21.15.42:0/3464194141 >> v2:172.21.15.42:3300/0 conn(0x7ff1880036f0 msgr2=0x7ff188010090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:46:58.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:58.860+0000 7ff19b7fe640 1 --2- 172.21.15.42:0/3464194141 >> v2:172.21.15.42:3300/0 conn(0x7ff1880036f0 0x7ff188010090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:46:58.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:58.860+0000 7ff19b7fe640 1 --2- 172.21.15.42:0/3464194141 >> v2:172.21.15.42:3300/0 conn(0x7ff1880036f0 0x7ff188010090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T03:46:59.269 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:59.264+0000 7ff19b7fe640 1 --2- 172.21.15.42:0/3464194141 >> v2:172.21.15.42:3300/0 conn(0x7ff1880036f0 0x7ff188010090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:46:59.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:59.264+0000 7ff19b7fe640 10 monclient(hunting): get_auth_request con 0x7ff1880036f0 auth_method 0 2024-09-16T03:46:59.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:59.264+0000 7ff19b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:46:59.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:59.264+0000 7ff19b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:46:59.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:59.264+0000 7ff19b7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:46:59.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:59.264+0000 7ff19b7fe640 1 -- 172.21.15.42:0/3464194141 >> v2:172.21.15.42:3300/0 conn(0x7ff1880036f0 msgr2=0x7ff188010090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T03:46:59.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:59.264+0000 7ff19b7fe640 1 -- 172.21.15.42:0/3464194141 >> v2:172.21.15.42:3300/0 conn(0x7ff1880036f0 msgr2=0x7ff188010090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:46:59.271 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:59.264+0000 7ff19b7fe640 1 --2- 172.21.15.42:0/3464194141 >> v2:172.21.15.42:3300/0 conn(0x7ff1880036f0 0x7ff188010090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:46:59.271 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:46:59.264+0000 7ff19b7fe640 1 --2- 172.21.15.42:0/3464194141 >> v2:172.21.15.42:3300/0 conn(0x7ff1880036f0 0x7ff188010090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T03:47:00.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:00.068+0000 7ff19b7fe640 1 --2- 172.21.15.42:0/3464194141 >> v2:172.21.15.42:3300/0 conn(0x7ff1880036f0 0x7ff188010090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:00.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:00.068+0000 7ff19b7fe640 10 monclient(hunting): get_auth_request con 0x7ff1880036f0 auth_method 0 2024-09-16T03:47:00.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:00.068+0000 7ff19b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:00.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:00.068+0000 7ff19b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:00.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:00.068+0000 7ff19b7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:47:00.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:00.068+0000 7ff19b7fe640 1 -- 172.21.15.42:0/3464194141 >> v2:172.21.15.42:3300/0 conn(0x7ff1880036f0 msgr2=0x7ff188010090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T03:47:00.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:00.068+0000 7ff19b7fe640 1 -- 172.21.15.42:0/3464194141 >> v2:172.21.15.42:3300/0 conn(0x7ff1880036f0 msgr2=0x7ff188010090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:47:00.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:00.068+0000 7ff19b7fe640 1 --2- 172.21.15.42:0/3464194141 >> v2:172.21.15.42:3300/0 conn(0x7ff1880036f0 0x7ff188010090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:47:00.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:00.068+0000 7ff19b7fe640 1 --2- 172.21.15.42:0/3464194141 >> v2:172.21.15.42:3300/0 conn(0x7ff1880036f0 0x7ff188010090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T03:47:01.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:01.660+0000 7ff198ff9640 10 monclient(hunting): tick 2024-09-16T03:47:01.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:01.660+0000 7ff198ff9640 10 monclient(hunting): _check_auth_tickets 2024-09-16T03:47:01.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:01.660+0000 7ff198ff9640 1 monclient(hunting): continuing hunt 2024-09-16T03:47:01.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:01.660+0000 7ff198ff9640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T03:47:01.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:01.660+0000 7ff198ff9640 1 -- 172.21.15.42:0/3464194141 >> v2:172.21.15.42:3300/0 conn(0x7ff1880036f0 msgr2=0x7ff188010090 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:47:01.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:01.660+0000 7ff198ff9640 1 --2- 172.21.15.42:0/3464194141 >> v2:172.21.15.42:3300/0 conn(0x7ff1880036f0 0x7ff188010090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:01.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:01.660+0000 7ff198ff9640 1 -- 172.21.15.42:0/3464194141 >> v1:172.21.15.154:6789/0 conn(0x7ff19c0a1e20 legacy=0x7ff18800b240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:01.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:01.660+0000 7ff198ff9640 1 -- 172.21.15.42:0/3464194141 >> v1:172.21.15.42:6789/0 conn(0x7ff19c0a1330 legacy=0x7ff18800e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:01.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:01.660+0000 7ff198ff9640 10 monclient: _add_conns ranks=[4,2] 2024-09-16T03:47:01.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:01.660+0000 7ff198ff9640 1 --2- 172.21.15.42:0/3464194141 >> v2:172.21.15.154:3300/0 conn(0x7ff188006760 0x7ff188006b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:01.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:01.660+0000 7ff198ff9640 10 monclient(hunting): picked mon.noname-c con 0x7ff188006760 addr v2:172.21.15.154:3300/0 2024-09-16T03:47:01.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:01.660+0000 7ff198ff9640 1 --2- 172.21.15.42:0/3464194141 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff1880071e0 0x7ff1880075f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:01.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:01.660+0000 7ff198ff9640 10 monclient(hunting): picked mon.noname-b con 0x7ff1880071e0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:47:01.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:01.660+0000 7ff198ff9640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:01.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:01.660+0000 7ff198ff9640 1 -- 172.21.15.42:0/3464194141 --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7ff19c09f8f0 con 0x7ff188006760 2024-09-16T03:47:01.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:01.660+0000 7ff198ff9640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:01.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:01.660+0000 7ff198ff9640 1 -- 172.21.15.42:0/3464194141 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7ff188008d60 con 0x7ff1880071e0 2024-09-16T03:47:01.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:01.660+0000 7ff198ff9640 10 monclient(hunting): _renew_subs 2024-09-16T03:47:01.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:01.660+0000 7ff19a7fc640 1 --2- 172.21.15.42:0/3464194141 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff1880071e0 0x7ff1880075f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:01.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:01.660+0000 7ff19affd640 1 --2- 172.21.15.42:0/3464194141 >> v2:172.21.15.154:3300/0 conn(0x7ff188006760 0x7ff188006b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:01.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:01.660+0000 7ff19a7fc640 10 monclient(hunting): get_auth_request con 0x7ff1880071e0 auth_method 0 2024-09-16T03:47:01.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:01.660+0000 7ff19a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:01.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:01.660+0000 7ff19a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:01.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:01.660+0000 7ff19a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:47:01.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:01.660+0000 7ff19affd640 10 monclient(hunting): get_auth_request con 0x7ff188006760 auth_method 0 2024-09-16T03:47:01.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:01.660+0000 7ff19affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:01.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:01.660+0000 7ff19affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:01.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:01.660+0000 7ff19affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:47:01.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:01.660+0000 7ff19a7fc640 1 -- 172.21.15.42:0/3464194141 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff1880071e0 msgr2=0x7ff1880075f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T03:47:01.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:01.660+0000 7ff19a7fc640 1 -- 172.21.15.42:0/3464194141 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff1880071e0 msgr2=0x7ff1880075f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:47:01.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:01.660+0000 7ff19a7fc640 1 --2- 172.21.15.42:0/3464194141 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff1880071e0 0x7ff1880075f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:47:01.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:01.660+0000 7ff19a7fc640 1 --2- 172.21.15.42:0/3464194141 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff1880071e0 0x7ff1880075f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:47:01.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:01.660+0000 7ff19affd640 1 -- 172.21.15.42:0/3464194141 >> v2:172.21.15.154:3300/0 conn(0x7ff188006760 msgr2=0x7ff188006b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T03:47:01.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:01.660+0000 7ff19affd640 1 -- 172.21.15.42:0/3464194141 >> v2:172.21.15.154:3300/0 conn(0x7ff188006760 msgr2=0x7ff188006b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:47:01.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:01.660+0000 7ff19affd640 1 --2- 172.21.15.42:0/3464194141 >> v2:172.21.15.154:3300/0 conn(0x7ff188006760 0x7ff188006b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:47:01.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:01.660+0000 7ff19affd640 1 --2- 172.21.15.42:0/3464194141 >> v2:172.21.15.154:3300/0 conn(0x7ff188006760 0x7ff188006b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:47:01.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:01.864+0000 7ff19a7fc640 1 --2- 172.21.15.42:0/3464194141 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff1880071e0 0x7ff1880075f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:01.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:01.864+0000 7ff19a7fc640 10 monclient(hunting): get_auth_request con 0x7ff1880071e0 auth_method 0 2024-09-16T03:47:01.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:01.864+0000 7ff19a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:01.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:01.864+0000 7ff19a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:01.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:01.864+0000 7ff19a7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:47:01.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:01.864+0000 7ff19affd640 1 --2- 172.21.15.42:0/3464194141 >> v2:172.21.15.154:3300/0 conn(0x7ff188006760 0x7ff188006b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:01.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:01.864+0000 7ff19affd640 10 monclient(hunting): get_auth_request con 0x7ff188006760 auth_method 0 2024-09-16T03:47:01.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:01.864+0000 7ff19affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:01.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:01.864+0000 7ff19affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:01.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:01.864+0000 7ff19affd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:47:01.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:01.864+0000 7ff19a7fc640 1 -- 172.21.15.42:0/3464194141 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff1880071e0 msgr2=0x7ff1880075f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T03:47:01.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:01.864+0000 7ff19a7fc640 1 -- 172.21.15.42:0/3464194141 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff1880071e0 msgr2=0x7ff1880075f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:47:01.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:01.864+0000 7ff19a7fc640 1 --2- 172.21.15.42:0/3464194141 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff1880071e0 0x7ff1880075f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:47:01.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:01.864+0000 7ff19a7fc640 1 --2- 172.21.15.42:0/3464194141 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff1880071e0 0x7ff1880075f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T03:47:01.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:01.864+0000 7ff19affd640 1 -- 172.21.15.42:0/3464194141 >> v2:172.21.15.154:3300/0 conn(0x7ff188006760 msgr2=0x7ff188006b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T03:47:01.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:01.864+0000 7ff19affd640 1 -- 172.21.15.42:0/3464194141 >> v2:172.21.15.154:3300/0 conn(0x7ff188006760 msgr2=0x7ff188006b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:47:01.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:01.864+0000 7ff19affd640 1 --2- 172.21.15.42:0/3464194141 >> v2:172.21.15.154:3300/0 conn(0x7ff188006760 0x7ff188006b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:47:01.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:01.864+0000 7ff19affd640 1 --2- 172.21.15.42:0/3464194141 >> v2:172.21.15.154:3300/0 conn(0x7ff188006760 0x7ff188006b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T03:47:02.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.268+0000 7ff19a7fc640 1 --2- 172.21.15.42:0/3464194141 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff1880071e0 0x7ff1880075f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:02.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.268+0000 7ff19a7fc640 10 monclient(hunting): get_auth_request con 0x7ff1880071e0 auth_method 0 2024-09-16T03:47:02.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.268+0000 7ff19a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:02.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.268+0000 7ff19a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:02.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.268+0000 7ff19a7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:47:02.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.268+0000 7ff19a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:47:02.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.268+0000 7ff19a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:47:02.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.268+0000 7ff19a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:47:02.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.268+0000 7ff19a7fc640 10 monclient(hunting): handle_auth_done global_id 13664 payload 274 2024-09-16T03:47:02.271 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.268+0000 7ff19a7fc640 1 -- 172.21.15.42:0/3464194141 >> v2:172.21.15.154:3300/0 conn(0x7ff188006760 msgr2=0x7ff188006b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:47:02.271 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.268+0000 7ff19a7fc640 1 --2- 172.21.15.42:0/3464194141 >> v2:172.21.15.154:3300/0 conn(0x7ff188006760 0x7ff188006b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:02.271 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.268+0000 7ff19a7fc640 10 monclient: _finish_hunting 0 2024-09-16T03:47:02.271 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.268+0000 7ff19a7fc640 1 monclient: found mon.noname-b 2024-09-16T03:47:02.271 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.268+0000 7ff19a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:47:02.271 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.268+0000 7ff19a7fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T03:47:02.271 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.268+0000 7ff19a7fc640 1 -- 172.21.15.42:0/3464194141 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff188005e50 con 0x7ff1880071e0 2024-09-16T03:47:02.275 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.272+0000 7ff19a7fc640 1 --2- 172.21.15.42:0/3464194141 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff1880071e0 0x7ff1880075f0 secure :-1 s=READY pgs=2464 cs=0 l=1 rev1=1 crypto rx=0x7ff18c000ee0 tx=0x7ff18c0071e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c4e7af0f7990e372 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:47:02.276 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.272+0000 7ff199ffb640 1 -- 172.21.15.42:0/3464194141 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff18c00f040 con 0x7ff1880071e0 2024-09-16T03:47:02.276 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.272+0000 7ff199ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:47:02.276 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.272+0000 7ff199ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:47:02.276 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.272+0000 7ff199ffb640 10 monclient: dump: 2024-09-16T03:47:02.276 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:47:02.276 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:02.276 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:02.276 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:02.277 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:47:02.277 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:02.277 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:47:02.277 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:47:02.277 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:47:02.277 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:02.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.272+0000 7ff199ffb640 10 monclient: _finish_auth 0 2024-09-16T03:47:02.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.272+0000 7ff199ffb640 10 monclient: _check_auth_tickets 2024-09-16T03:47:02.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.272+0000 7ff199ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:47:02.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.272+0000 7ff199ffb640 1 -- 172.21.15.42:0/3464194141 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff18c01b030 con 0x7ff1880071e0 2024-09-16T03:47:02.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.272+0000 7ff199ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T03:47:02.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.272+0000 7ff199ffb640 1 -- 172.21.15.42:0/3464194141 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff18c01a450 con 0x7ff1880071e0 2024-09-16T03:47:02.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.272+0000 7ff1a1264640 5 monclient: authenticate success, global_id 13664 2024-09-16T03:47:02.278 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.272+0000 7ff1a1264640 10 monclient: get_monmap_and_config success 2024-09-16T03:47:02.278 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.272+0000 7ff1a1264640 4 set_mon_vals no callback set 2024-09-16T03:47:02.278 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.272+0000 7ff199ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:47:02.278 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.272+0000 7ff199ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:47:02.278 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.272+0000 7ff199ffb640 10 monclient: dump: 2024-09-16T03:47:02.278 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:47:02.278 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:02.278 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:02.278 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:02.278 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:47:02.278 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:02.278 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:47:02.279 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:47:02.279 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:47:02.279 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:02.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.276+0000 7ff1a1264640 10 monclient: shutdown 2024-09-16T03:47:02.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.276+0000 7ff1a1264640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:47:02.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.276+0000 7ff1a1264640 1 -- 172.21.15.42:0/3464194141 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff1880071e0 msgr2=0x7ff1880075f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:02.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.276+0000 7ff1a1264640 1 --2- 172.21.15.42:0/3464194141 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff1880071e0 0x7ff1880075f0 secure :-1 s=READY pgs=2464 cs=0 l=1 rev1=1 crypto rx=0x7ff18c000ee0 tx=0x7ff18c0071e0 comp rx=0 tx=0).stop 2024-09-16T03:47:02.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.276+0000 7ff19b7fe640 1 -- 172.21.15.42:0/3464194141 reap_dead start 2024-09-16T03:47:02.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.276+0000 7ff1a1264640 1 -- 172.21.15.42:0/3464194141 shutdown_connections 2024-09-16T03:47:02.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.276+0000 7ff1a1264640 1 -- 172.21.15.42:0/3464194141 >> 172.21.15.42:0/3464194141 conn(0x7ff19c0a2420 msgr2=0x7ff19c0a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:47:02.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.276+0000 7ff1a1264640 1 -- 172.21.15.42:0/3464194141 shutdown_connections 2024-09-16T03:47:02.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.276+0000 7ff1a1264640 1 -- 172.21.15.42:0/3464194141 wait complete. 2024-09-16T03:47:02.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.276+0000 7ff1a1264640 10 monclient: build_initial_monmap 2024-09-16T03:47:02.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.276+0000 7ff1a1264640 10 monclient: monmap: 2024-09-16T03:47:02.285 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:47:02.285 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:47:02.285 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:47:02.285 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:47:02.285 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:47:02.285 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:02.285 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:47:02.285 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:47:02.285 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:47:02.285 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:02.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.276+0000 7ff1a1264640 1 librados: starting msgr at 2024-09-16T03:47:02.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.276+0000 7ff1a1264640 1 librados: starting objecter 2024-09-16T03:47:02.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.276+0000 7ff1a1264640 1 Processor -- start 2024-09-16T03:47:02.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.276+0000 7ff1a1264640 1 -- start start 2024-09-16T03:47:02.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.276+0000 7ff1a1264640 1 librados: setting wanted keys 2024-09-16T03:47:02.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.276+0000 7ff1a1264640 1 librados: calling monclient init 2024-09-16T03:47:02.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.276+0000 7ff1a1264640 10 monclient: init 2024-09-16T03:47:02.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.276+0000 7ff1a1264640 10 monclient: _reopen_session rank -1 2024-09-16T03:47:02.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.276+0000 7ff1a1264640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:47:02.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.276+0000 7ff1a1264640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff19c1cd320 0x7ff19c134320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:02.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.276+0000 7ff1a1264640 10 monclient(hunting): picked mon.noname-a con 0x7ff19c1cd320 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:47:02.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.276+0000 7ff1a1264640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff19c134860 0x7ff19c122da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:02.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.276+0000 7ff1a1264640 10 monclient(hunting): picked mon.noname-c con 0x7ff19c134860 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:47:02.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.276+0000 7ff1a1264640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff19c1232e0 0x7ff19c123790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:02.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.276+0000 7ff1a1264640 10 monclient(hunting): picked mon.noname-b con 0x7ff19c1232e0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:47:02.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.276+0000 7ff1a1264640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:02.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.276+0000 7ff1a1264640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7ff19c135910 con 0x7ff19c1cd320 2024-09-16T03:47:02.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.276+0000 7ff1a1264640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:02.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.276+0000 7ff1a1264640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff19c09ef80 con 0x7ff19c1232e0 2024-09-16T03:47:02.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.276+0000 7ff1a1264640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:02.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.276+0000 7ff1a1264640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7ff19c135a90 con 0x7ff19c134860 2024-09-16T03:47:02.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.276+0000 7ff1a1264640 10 monclient(hunting): _renew_subs 2024-09-16T03:47:02.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.276+0000 7ff1a1264640 10 monclient(hunting): authenticate will time out at 1269.247666s 2024-09-16T03:47:02.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.276+0000 7ff19affd640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff19c1cd320 0x7ff19c134320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:02.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.276+0000 7ff19affd640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff19c1cd320 0x7ff19c134320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:57444/0 (socket says 172.21.15.42:57444) 2024-09-16T03:47:02.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.276+0000 7ff19affd640 1 -- 172.21.15.42:0/1935422415 learned_addr learned my addr 172.21.15.42:0/1935422415 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:47:02.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.276+0000 7ff19affd640 10 monclient(hunting): get_auth_request con 0x7ff19c1cd320 auth_method 0 2024-09-16T03:47:02.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.276+0000 7ff19affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:02.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.276+0000 7ff19affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:02.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.276+0000 7ff19affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:47:02.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.276+0000 7ff19a7fc640 1 --2- 172.21.15.42:0/1935422415 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff19c134860 0x7ff19c122da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:02.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.276+0000 7ff19a7fc640 10 monclient(hunting): get_auth_request con 0x7ff19c134860 auth_method 0 2024-09-16T03:47:02.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.276+0000 7ff19a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:02.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.276+0000 7ff19a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:02.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.276+0000 7ff19a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:47:02.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.276+0000 7ff19affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:47:02.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.276+0000 7ff19affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:47:02.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.280+0000 7ff19affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:47:02.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.280+0000 7ff19a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:47:02.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.280+0000 7ff19a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:47:02.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.280+0000 7ff19a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:47:02.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.280+0000 7ff19b7fe640 1 --2- 172.21.15.42:0/1935422415 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff19c1232e0 0x7ff19c123790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:02.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.280+0000 7ff19b7fe640 10 monclient(hunting): get_auth_request con 0x7ff19c1232e0 auth_method 0 2024-09-16T03:47:02.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.280+0000 7ff19b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:02.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.280+0000 7ff19b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:02.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.280+0000 7ff19b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:47:02.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.280+0000 7ff19affd640 10 monclient(hunting): handle_auth_done global_id 764097 payload 824 2024-09-16T03:47:02.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.280+0000 7ff19affd640 1 -- 172.21.15.42:0/1935422415 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff19c134860 msgr2=0x7ff19c122da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:02.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.280+0000 7ff19affd640 1 --2- 172.21.15.42:0/1935422415 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff19c134860 0x7ff19c122da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:02.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.280+0000 7ff19affd640 1 -- 172.21.15.42:0/1935422415 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff19c1232e0 msgr2=0x7ff19c123790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:02.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.280+0000 7ff19affd640 1 --2- 172.21.15.42:0/1935422415 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff19c1232e0 0x7ff19c123790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:02.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.280+0000 7ff19affd640 10 monclient: _finish_hunting 0 2024-09-16T03:47:02.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.280+0000 7ff19affd640 1 monclient: found mon.noname-a 2024-09-16T03:47:02.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.280+0000 7ff19affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:47:02.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.280+0000 7ff19affd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:47:02.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.280+0000 7ff19affd640 1 -- 172.21.15.42:0/1935422415 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff19c126a50 con 0x7ff19c1cd320 2024-09-16T03:47:02.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.280+0000 7ff19a7fc640 1 --2- 172.21.15.42:0/1935422415 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff19c134860 0x7ff19c122da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:47:02.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.280+0000 7ff19b7fe640 1 --2- 172.21.15.42:0/1935422415 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff19c1232e0 0x7ff19c123790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T03:47:02.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.280+0000 7ff19affd640 1 --2- 172.21.15.42:0/1935422415 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff19c1cd320 0x7ff19c134320 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7ff190007870 tx=0x7ff190018870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:47:02.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.280+0000 7ff1837fe640 10 client.?.objecter ms_handle_connect 0x7ff19c1cd320 2024-09-16T03:47:02.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.280+0000 7ff1837fe640 10 client.?.objecter resend_mon_ops 2024-09-16T03:47:02.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.280+0000 7ff1837fe640 1 -- 172.21.15.42:0/1935422415 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff190019720 con 0x7ff19c1cd320 2024-09-16T03:47:02.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.280+0000 7ff1837fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:47:02.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.280+0000 7ff1837fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:47:02.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.280+0000 7ff1837fe640 10 monclient: dump: 2024-09-16T03:47:02.290 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:47:02.290 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:02.291 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:02.291 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:02.291 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:47:02.291 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:02.291 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:47:02.291 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:47:02.291 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:47:02.291 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:02.291 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.280+0000 7ff1837fe640 10 monclient: _finish_auth 0 2024-09-16T03:47:02.291 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.280+0000 7ff1837fe640 10 monclient: _check_auth_tickets 2024-09-16T03:47:02.291 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.280+0000 7ff1837fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:47:02.291 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.280+0000 7ff1837fe640 1 -- 172.21.15.42:0/1935422415 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff19001d070 con 0x7ff19c1cd320 2024-09-16T03:47:02.291 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.280+0000 7ff1a1264640 5 monclient: authenticate success, global_id 764097 2024-09-16T03:47:02.291 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.280+0000 7ff1a1264640 10 monclient: _renew_subs 2024-09-16T03:47:02.292 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.280+0000 7ff1a1264640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:47:02.292 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.280+0000 7ff1a1264640 1 -- 172.21.15.42:0/1935422415 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff19c134ca0 con 0x7ff19c1cd320 2024-09-16T03:47:02.292 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.280+0000 7ff1837fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:47:02.292 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.280+0000 7ff1837fe640 1 -- 172.21.15.42:0/1935422415 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff190019b00 con 0x7ff19c1cd320 2024-09-16T03:47:02.292 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.280+0000 7ff1837fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:47:02.292 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.280+0000 7ff1837fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:47:02.292 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.280+0000 7ff1837fe640 10 monclient: dump: 2024-09-16T03:47:02.292 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:47:02.292 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:02.292 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:02.292 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:02.292 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:47:02.292 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:02.293 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:47:02.293 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:47:02.293 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:47:02.293 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:02.293 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.280+0000 7ff198ff9640 4 set_mon_vals no callback set 2024-09-16T03:47:02.293 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.280+0000 7ff1a1264640 10 client.764097.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:47:02.293 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.280+0000 7ff1a1264640 10 monclient: _renew_subs 2024-09-16T03:47:02.293 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.280+0000 7ff1a1264640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:47:02.293 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.280+0000 7ff1a1264640 1 -- 172.21.15.42:0/1935422415 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff19c06e4e0 con 0x7ff19c1cd320 2024-09-16T03:47:02.293 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.280+0000 7ff1a1264640 1 librados: init done 2024-09-16T03:47:02.293 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.280+0000 7ff1837fe640 1 -- 172.21.15.42:0/1935422415 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7ff19001f020 con 0x7ff19c1cd320 2024-09-16T03:47:02.293 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.280+0000 7ff1a1264640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:47:02.293 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.280+0000 7ff1a1264640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:47:02.293 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.280+0000 7ff1a1264640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:47:02.294 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.280+0000 7ff1a1264640 1 -- 172.21.15.42:0/1935422415 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff19c071ae0 con 0x7ff19c1cd320 2024-09-16T03:47:02.294 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.280+0000 7ff1837fe640 1 --2- 172.21.15.42:0/1935422415 >> v2:172.21.15.154:6812/254845342 conn(0x7ff188039480 0x7ff18803b940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:02.294 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.280+0000 7ff1837fe640 1 -- 172.21.15.42:0/1935422415 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(146..146 src has 1..146) ==== 7264+0+0 (secure 0 0 0) 0x7ff19004a240 con 0x7ff19c1cd320 2024-09-16T03:47:02.294 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.280+0000 7ff1837fe640 10 client.764097.objecter ms_dispatch 0x7ff19c000ba0 osd_map(146..146 src has 1..146) 2024-09-16T03:47:02.294 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.280+0000 7ff1837fe640 3 client.764097.objecter handle_osd_map got epochs [146,146] > 0 2024-09-16T03:47:02.294 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.280+0000 7ff1837fe640 3 client.764097.objecter handle_osd_map decoding full epoch 146 2024-09-16T03:47:02.294 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.280+0000 7ff1837fe640 20 client.764097.objecter dump_active .. 0 homeless 2024-09-16T03:47:02.294 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.280+0000 7ff19a7fc640 1 --2- 172.21.15.42:0/1935422415 >> v2:172.21.15.154:6812/254845342 conn(0x7ff188039480 0x7ff18803b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:02.294 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.280+0000 7ff19a7fc640 10 monclient: get_auth_request con 0x7ff188039480 auth_method 0 2024-09-16T03:47:02.295 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.280+0000 7ff1837fe640 10 client.764097.objecter ms_handle_connect 0x7ff188039480 2024-09-16T03:47:02.295 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.280+0000 7ff19a7fc640 1 --2- 172.21.15.42:0/1935422415 >> v2:172.21.15.154:6812/254845342 conn(0x7ff188039480 0x7ff18803b940 secure :-1 s=READY pgs=3427 cs=0 l=1 rev1=1 crypto rx=0x7ff18c0b0000 tx=0x7ff18c0a8000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:47:02.295 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.288+0000 7ff1837fe640 1 -- 172.21.15.42:0/1935422415 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7ff19002a8c0 con 0x7ff19c1cd320 2024-09-16T03:47:02.295 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.288+0000 7ff1837fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:47:02.295 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.288+0000 7ff1837fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:47:02.522 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.520+0000 7ff1a1264640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:47:02.522 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.520+0000 7ff1a1264640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:47:02.522 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.520+0000 7ff1a1264640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:47:02.522 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.520+0000 7ff1a1264640 1 -- 172.21.15.42:0/1935422415 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7ff19c135e50 con 0x7ff19c1cd320 2024-09-16T03:47:02.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.520+0000 7ff1837fe640 1 -- 172.21.15.42:0/1935422415 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7ff19002a3a0 con 0x7ff19c1cd320 2024-09-16T03:47:02.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.520+0000 7ff1837fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:47:02.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.520+0000 7ff1837fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:47:02.523 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":48,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:47:02.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.524+0000 7ff1a1264640 10 librados: watch_flush enter 2024-09-16T03:47:02.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.524+0000 7ff1a1264640 10 librados: watch_flush exit 2024-09-16T03:47:02.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.524+0000 7ff1a1264640 20 client.764097.objecter shutdown clearing up homeless session... 2024-09-16T03:47:02.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.524+0000 7ff1a1264640 10 client.764097.objecter successfully canceled tick 2024-09-16T03:47:02.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.524+0000 7ff1a1264640 1 -- 172.21.15.42:0/1935422415 >> v2:172.21.15.154:6812/254845342 conn(0x7ff188039480 msgr2=0x7ff18803b940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:02.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.524+0000 7ff1a1264640 1 --2- 172.21.15.42:0/1935422415 >> v2:172.21.15.154:6812/254845342 conn(0x7ff188039480 0x7ff18803b940 secure :-1 s=READY pgs=3427 cs=0 l=1 rev1=1 crypto rx=0x7ff18c0b0000 tx=0x7ff18c0a8000 comp rx=0 tx=0).stop 2024-09-16T03:47:02.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.524+0000 7ff1a1264640 10 monclient: shutdown 2024-09-16T03:47:02.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.524+0000 7ff1a1264640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:47:02.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.524+0000 7ff1a1264640 1 -- 172.21.15.42:0/1935422415 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff19c1cd320 msgr2=0x7ff19c134320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:02.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.524+0000 7ff1a1264640 1 --2- 172.21.15.42:0/1935422415 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff19c1cd320 0x7ff19c134320 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7ff190007870 tx=0x7ff190018870 comp rx=0 tx=0).stop 2024-09-16T03:47:02.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.524+0000 7ff1a1264640 1 -- 172.21.15.42:0/1935422415 shutdown_connections 2024-09-16T03:47:02.527 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.524+0000 7ff1a1264640 1 --2- 172.21.15.42:0/1935422415 >> v2:172.21.15.154:6812/254845342 conn(0x7ff188039480 0x7ff18803b940 unknown :-1 s=CLOSED pgs=3427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:02.527 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.524+0000 7ff1a1264640 1 --2- 172.21.15.42:0/1935422415 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff19c1232e0 0x7ff19c123790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:02.527 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.524+0000 7ff1a1264640 1 --2- 172.21.15.42:0/1935422415 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff19c134860 0x7ff19c122da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:02.527 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.524+0000 7ff1a1264640 1 --2- 172.21.15.42:0/1935422415 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff19c1cd320 0x7ff19c134320 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:02.527 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.524+0000 7ff1a1264640 1 -- 172.21.15.42:0/1935422415 >> 172.21.15.42:0/1935422415 conn(0x7ff19c0a2420 msgr2=0x7ff19c06ea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:47:02.527 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.524+0000 7ff1a1264640 1 -- 172.21.15.42:0/1935422415 shutdown_connections 2024-09-16T03:47:02.527 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.524+0000 7ff1a1264640 1 -- 172.21.15.42:0/1935422415 wait complete. 2024-09-16T03:47:02.527 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.524+0000 7ff1a1264640 1 librados: shutdown 2024-09-16T03:47:02.542 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T03:47:02.542 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T03:47:02.542 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T03:47:02.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.660+0000 7f0534161640 10 monclient: get_monmap_and_config 2024-09-16T03:47:02.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.660+0000 7f0534161640 10 monclient: build_initial_monmap 2024-09-16T03:47:02.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.660+0000 7f0534161640 10 monclient: monmap: 2024-09-16T03:47:02.661 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:47:02.661 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:02.661 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:47:02.660173+0000 2024-09-16T03:47:02.661 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:47:02.660173+0000 2024-09-16T03:47:02.661 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:47:02.661 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:02.662 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:47:02.662 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:47:02.662 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:47:02.662 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:47:02.662 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:47:02.662 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:02.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.660+0000 7f0534161640 1 Processor -- start 2024-09-16T03:47:02.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.660+0000 7f0534161640 1 -- start start 2024-09-16T03:47:02.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.660+0000 7f0534161640 10 monclient: init 2024-09-16T03:47:02.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.660+0000 7f0534161640 10 monclient: _reopen_session rank -1 2024-09-16T03:47:02.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.660+0000 7f0534161640 10 monclient: _add_conns ranks=[4,3,1,0,2] 2024-09-16T03:47:02.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.660+0000 7f0534161640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f052c0a1330 0x7f052c09f180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:02.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.660+0000 7f0534161640 10 monclient(hunting): picked mon.noname-c con 0x7f052c0a1330 addr v2:172.21.15.154:3300/0 2024-09-16T03:47:02.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.660+0000 7f0534161640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f052c0a1710 0x7f052c0a1b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:02.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.660+0000 7f0534161640 10 monclient(hunting): picked mon.noname-a con 0x7f052c0a1710 addr v2:172.21.15.42:3300/0 2024-09-16T03:47:02.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.660+0000 7f0534161640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f052c1340f0 addr v1:172.21.15.154:6789/0 2024-09-16T03:47:02.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.660+0000 7f0534161640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f052c09e4c0 con 0x7f052c1340f0 2024-09-16T03:47:02.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.660+0000 7f0534161640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:02.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.660+0000 7f0534161640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f052c09edf0 con 0x7f052c0a1710 2024-09-16T03:47:02.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.660+0000 7f0534161640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:02.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.660+0000 7f0534161640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f052c09cd60 con 0x7f052c0a1330 2024-09-16T03:47:02.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.660+0000 7f0534161640 10 monclient(hunting): _renew_subs 2024-09-16T03:47:02.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.660+0000 7f0534161640 10 monclient(hunting): authenticate will time out at 1269.629555s 2024-09-16T03:47:02.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.660+0000 7f05316fc640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f052c0a1710 0x7f052c0a1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:02.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.660+0000 7f05316fc640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f052c0a1710 0x7f052c0a1b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:57446/0 (socket says 172.21.15.42:57446) 2024-09-16T03:47:02.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.660+0000 7f05316fc640 1 -- 172.21.15.42:0/2964712152 learned_addr learned my addr 172.21.15.42:0/2964712152 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:47:02.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.660+0000 7f05316fc640 10 monclient(hunting): get_auth_request con 0x7f052c0a1710 auth_method 0 2024-09-16T03:47:02.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.660+0000 7f05316fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:02.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.660+0000 7f05316fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:02.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.660+0000 7f05316fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:47:02.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.660+0000 7f05316fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:47:02.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.660+0000 7f05316fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:47:02.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.660+0000 7f05316fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:47:02.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.660+0000 7f05316fc640 10 monclient(hunting): handle_auth_done global_id 764100 payload 274 2024-09-16T03:47:02.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.660+0000 7f05316fc640 1 -- 172.21.15.42:0/2964712152 >> v2:172.21.15.154:3300/0 conn(0x7f052c0a1330 msgr2=0x7f052c09f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:47:02.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.660+0000 7f05316fc640 1 --2- 172.21.15.42:0/2964712152 >> v2:172.21.15.154:3300/0 conn(0x7f052c0a1330 0x7f052c09f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:02.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.660+0000 7f05316fc640 1 -- 172.21.15.42:0/2964712152 >> v1:172.21.15.154:6789/0 conn(0x7f052c1340f0 legacy=0x7f052c134540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:47:02.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.660+0000 7f05316fc640 10 monclient: _finish_hunting 0 2024-09-16T03:47:02.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.660+0000 7f05316fc640 1 monclient: found mon.noname-a 2024-09-16T03:47:02.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.660+0000 7f05316fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:47:02.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.660+0000 7f05316fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:47:02.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.660+0000 7f05316fc640 1 -- 172.21.15.42:0/2964712152 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f052c0a2080 con 0x7f052c0a1710 2024-09-16T03:47:02.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.664+0000 7f05316fc640 1 --2- 172.21.15.42:0/2964712152 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f052c0a1710 0x7f052c0a1b40 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f052c06add0 tx=0x7f051c01c430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=129da2618ba9296e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:47:02.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.664+0000 7f0530efb640 1 -- 172.21.15.42:0/2964712152 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f051c0b0070 con 0x7f052c0a1710 2024-09-16T03:47:02.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.664+0000 7f0530efb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:47:02.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.664+0000 7f0530efb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:47:02.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.664+0000 7f0530efb640 10 monclient: dump: 2024-09-16T03:47:02.674 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:47:02.674 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:02.674 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:02.674 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:02.675 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:47:02.675 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:02.675 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:47:02.675 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:47:02.675 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:47:02.675 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:02.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.664+0000 7f0530efb640 10 monclient: _finish_auth 0 2024-09-16T03:47:02.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.664+0000 7f0530efb640 10 monclient: _check_auth_tickets 2024-09-16T03:47:02.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.664+0000 7f0530efb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:47:02.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.664+0000 7f0530efb640 1 -- 172.21.15.42:0/2964712152 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f051c0ab600 con 0x7f052c0a1710 2024-09-16T03:47:02.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.664+0000 7f0534161640 5 monclient: authenticate success, global_id 764100 2024-09-16T03:47:02.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.664+0000 7f0534161640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:47:02.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.664+0000 7f0530efb640 10 monclient: handle_config config(0 keys) 2024-09-16T03:47:02.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.664+0000 7f0530efb640 1 -- 172.21.15.42:0/2964712152 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f051c0ab9d0 con 0x7f052c0a1710 2024-09-16T03:47:02.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.664+0000 7f0530efb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:47:02.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.664+0000 7f0530efb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:47:02.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.664+0000 7f0530efb640 10 monclient: dump: 2024-09-16T03:47:02.676 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:47:02.676 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:02.676 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:02.676 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:02.676 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:47:02.676 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:02.676 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:47:02.676 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:47:02.676 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:47:02.677 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:02.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.664+0000 7f0534161640 10 monclient: get_monmap_and_config success 2024-09-16T03:47:02.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.664+0000 7f0534161640 4 set_mon_vals no callback set 2024-09-16T03:47:02.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.664+0000 7f0534161640 10 monclient: shutdown 2024-09-16T03:47:02.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.664+0000 7f0534161640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:47:02.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.664+0000 7f0534161640 1 -- 172.21.15.42:0/2964712152 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f052c0a1710 msgr2=0x7f052c0a1b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:02.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.664+0000 7f0534161640 1 --2- 172.21.15.42:0/2964712152 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f052c0a1710 0x7f052c0a1b40 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f052c06add0 tx=0x7f051c01c430 comp rx=0 tx=0).stop 2024-09-16T03:47:02.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.664+0000 7f0534161640 1 -- 172.21.15.42:0/2964712152 shutdown_connections 2024-09-16T03:47:02.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.664+0000 7f0534161640 1 --2- 172.21.15.42:0/2964712152 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f052c0a1710 0x7f052c0a1b40 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:02.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.664+0000 7f0534161640 1 --2- 172.21.15.42:0/2964712152 >> v2:172.21.15.154:3300/0 conn(0x7f052c0a1330 0x7f052c09f180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:02.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.664+0000 7f0534161640 1 -- 172.21.15.42:0/2964712152 >> 172.21.15.42:0/2964712152 conn(0x7f052c0a2420 msgr2=0x7f052c0a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:47:02.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.664+0000 7f0534161640 1 -- 172.21.15.42:0/2964712152 shutdown_connections 2024-09-16T03:47:02.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.664+0000 7f0534161640 1 -- 172.21.15.42:0/2964712152 wait complete. 2024-09-16T03:47:02.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.664+0000 7f0534161640 10 monclient: build_initial_monmap 2024-09-16T03:47:02.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.664+0000 7f0534161640 10 monclient: monmap: 2024-09-16T03:47:02.678 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:47:02.678 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:47:02.678 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:47:02.678 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:47:02.678 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:47:02.678 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:02.679 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:47:02.679 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:47:02.679 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:47:02.679 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:02.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.664+0000 7f0534161640 1 librados: starting msgr at 2024-09-16T03:47:02.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.664+0000 7f0534161640 1 librados: starting objecter 2024-09-16T03:47:02.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.664+0000 7f0534161640 1 Processor -- start 2024-09-16T03:47:02.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.664+0000 7f0534161640 1 -- start start 2024-09-16T03:47:02.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.664+0000 7f0534161640 1 librados: setting wanted keys 2024-09-16T03:47:02.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.664+0000 7f0534161640 1 librados: calling monclient init 2024-09-16T03:47:02.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.664+0000 7f0534161640 10 monclient: init 2024-09-16T03:47:02.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.664+0000 7f0534161640 10 monclient: _reopen_session rank -1 2024-09-16T03:47:02.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.664+0000 7f0534161640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:47:02.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.664+0000 7f0534161640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f052c0a1330 0x7f052c13a570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:02.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.664+0000 7f0534161640 10 monclient(hunting): picked mon.noname-c con 0x7f052c0a1330 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:47:02.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.664+0000 7f0534161640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f052c0a1710 0x7f052c133700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:02.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.664+0000 7f0534161640 10 monclient(hunting): picked mon.noname-a con 0x7f052c0a1710 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:47:02.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.664+0000 7f0534161640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f052c1340f0 0x7f052c06d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:02.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.664+0000 7f0534161640 10 monclient(hunting): picked mon.noname-b con 0x7f052c1340f0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:47:02.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.664+0000 7f0534161640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:02.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.664+0000 7f0534161640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f052c13ed60 con 0x7f052c0a1710 2024-09-16T03:47:02.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.664+0000 7f0534161640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:02.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.664+0000 7f0534161640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f052c13ebe0 con 0x7f052c1340f0 2024-09-16T03:47:02.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.664+0000 7f0534161640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:02.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.664+0000 7f0534161640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f052c13eee0 con 0x7f052c0a1330 2024-09-16T03:47:02.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.664+0000 7f0534161640 10 monclient(hunting): _renew_subs 2024-09-16T03:47:02.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.664+0000 7f0534161640 10 monclient(hunting): authenticate will time out at 1269.634799s 2024-09-16T03:47:02.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.664+0000 7f05316fc640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f052c0a1710 0x7f052c133700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:02.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.664+0000 7f0531efd640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f052c0a1330 0x7f052c13a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:02.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.664+0000 7f05316fc640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f052c0a1710 0x7f052c133700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:57452/0 (socket says 172.21.15.42:57452) 2024-09-16T03:47:02.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.664+0000 7f05316fc640 1 -- 172.21.15.42:0/534757398 learned_addr learned my addr 172.21.15.42:0/534757398 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:47:02.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.664+0000 7f05316fc640 10 monclient(hunting): get_auth_request con 0x7f052c0a1710 auth_method 0 2024-09-16T03:47:02.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.664+0000 7f05316fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:02.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.664+0000 7f05316fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:02.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.664+0000 7f05316fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:47:02.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.664+0000 7f0531efd640 10 monclient(hunting): get_auth_request con 0x7f052c0a1330 auth_method 0 2024-09-16T03:47:02.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.664+0000 7f0531efd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:02.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.664+0000 7f0531efd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:02.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.664+0000 7f0531efd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:47:02.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.664+0000 7f05316fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:47:02.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.664+0000 7f05316fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:47:02.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.664+0000 7f05316fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:47:02.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.664+0000 7f0531efd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:47:02.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.664+0000 7f0531efd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:47:02.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.664+0000 7f0531efd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:47:02.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.664+0000 7f05316fc640 10 monclient(hunting): handle_auth_done global_id 764103 payload 824 2024-09-16T03:47:02.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.664+0000 7f05326fe640 1 --2- 172.21.15.42:0/534757398 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f052c1340f0 0x7f052c06d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:02.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.664+0000 7f05316fc640 1 -- 172.21.15.42:0/534757398 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f052c0a1330 msgr2=0x7f052c13a570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:02.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.664+0000 7f05316fc640 1 --2- 172.21.15.42:0/534757398 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f052c0a1330 0x7f052c13a570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:02.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.664+0000 7f05316fc640 1 -- 172.21.15.42:0/534757398 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f052c1340f0 msgr2=0x7f052c06d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:02.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.664+0000 7f05316fc640 1 --2- 172.21.15.42:0/534757398 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f052c1340f0 0x7f052c06d900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:02.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.664+0000 7f05316fc640 10 monclient: _finish_hunting 0 2024-09-16T03:47:02.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.664+0000 7f05316fc640 1 monclient: found mon.noname-a 2024-09-16T03:47:02.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.664+0000 7f05316fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:47:02.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.664+0000 7f05316fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:47:02.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.664+0000 7f05316fc640 1 -- 172.21.15.42:0/534757398 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f052c06f080 con 0x7f052c0a1710 2024-09-16T03:47:02.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.664+0000 7f05316fc640 1 --2- 172.21.15.42:0/534757398 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f052c0a1710 0x7f052c133700 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f051c01aae0 tx=0x7f051c0191d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:47:02.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.664+0000 7f05227fc640 10 client.?.objecter ms_handle_connect 0x7f052c0a1710 2024-09-16T03:47:02.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.664+0000 7f05227fc640 10 client.?.objecter resend_mon_ops 2024-09-16T03:47:02.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.664+0000 7f05227fc640 1 -- 172.21.15.42:0/534757398 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f051c0bb070 con 0x7f052c0a1710 2024-09-16T03:47:02.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.664+0000 7f05227fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:47:02.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.664+0000 7f05227fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:47:02.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.664+0000 7f05227fc640 10 monclient: dump: 2024-09-16T03:47:02.685 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:47:02.685 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:02.685 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:02.685 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:02.685 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:47:02.685 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:02.685 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:47:02.685 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:47:02.686 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:47:02.686 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:02.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.664+0000 7f05227fc640 10 monclient: _finish_auth 0 2024-09-16T03:47:02.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.664+0000 7f05227fc640 10 monclient: _check_auth_tickets 2024-09-16T03:47:02.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.664+0000 7f05227fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:47:02.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.664+0000 7f05227fc640 1 -- 172.21.15.42:0/534757398 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f051c0b6640 con 0x7f052c0a1710 2024-09-16T03:47:02.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.664+0000 7f05227fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:47:02.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.664+0000 7f05227fc640 1 -- 172.21.15.42:0/534757398 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f051c0b0040 con 0x7f052c0a1710 2024-09-16T03:47:02.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.664+0000 7f05227fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:47:02.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.664+0000 7f05227fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:47:02.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.664+0000 7f05227fc640 10 monclient: dump: 2024-09-16T03:47:02.687 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:47:02.688 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:02.688 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:02.688 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:02.688 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:47:02.688 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:02.688 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:47:02.688 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:47:02.689 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:47:02.689 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:02.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.664+0000 7f0523fff640 4 set_mon_vals no callback set 2024-09-16T03:47:02.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.668+0000 7f0534161640 5 monclient: authenticate success, global_id 764103 2024-09-16T03:47:02.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.668+0000 7f0534161640 10 monclient: _renew_subs 2024-09-16T03:47:02.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.668+0000 7f0534161640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:47:02.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.668+0000 7f0534161640 1 -- 172.21.15.42:0/534757398 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f052c120b20 con 0x7f052c0a1710 2024-09-16T03:47:02.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.668+0000 7f0534161640 10 client.764103.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:47:02.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.668+0000 7f0534161640 10 monclient: _renew_subs 2024-09-16T03:47:02.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.668+0000 7f0534161640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:47:02.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.668+0000 7f0534161640 1 -- 172.21.15.42:0/534757398 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f052c0a3e00 con 0x7f052c0a1710 2024-09-16T03:47:02.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.668+0000 7f0534161640 1 librados: init done 2024-09-16T03:47:02.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.668+0000 7f0534161640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:47:02.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.668+0000 7f0534161640 1 --2- 172.21.15.42:0/534757398 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f052c133c40 0x7f052c02c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:02.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.668+0000 7f0534161640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:02.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.668+0000 7f0534161640 1 -- 172.21.15.42:0/534757398 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f052c0a4230 con 0x7f052c133c40 2024-09-16T03:47:02.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.668+0000 7f05227fc640 1 -- 172.21.15.42:0/534757398 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f051c0b4430 con 0x7f052c0a1710 2024-09-16T03:47:02.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.668+0000 7f0531efd640 1 --2- 172.21.15.42:0/534757398 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f052c133c40 0x7f052c02c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:02.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.668+0000 7f0531efd640 10 monclient: get_auth_request con 0x7f052c133c40 auth_method 0 2024-09-16T03:47:02.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.668+0000 7f0531efd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:02.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.668+0000 7f0531efd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:02.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.668+0000 7f0531efd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:47:02.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.668+0000 7f0531efd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:47:02.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.668+0000 7f0531efd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:47:02.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.668+0000 7f0531efd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:47:02.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.668+0000 7f05227fc640 1 --2- 172.21.15.42:0/534757398 >> v2:172.21.15.154:6812/254845342 conn(0x7f0510037840 0x7f0510039d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:02.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.668+0000 7f05227fc640 1 -- 172.21.15.42:0/534757398 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(146..146 src has 1..146) ==== 7264+0+0 (secure 0 0 0) 0x7f051c0e2f90 con 0x7f052c0a1710 2024-09-16T03:47:02.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.668+0000 7f05227fc640 10 client.764103.objecter ms_dispatch 0x7f052c000ba0 osd_map(146..146 src has 1..146) 2024-09-16T03:47:02.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.668+0000 7f05227fc640 3 client.764103.objecter handle_osd_map got epochs [146,146] > 0 2024-09-16T03:47:02.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.668+0000 7f05227fc640 3 client.764103.objecter handle_osd_map decoding full epoch 146 2024-09-16T03:47:02.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.668+0000 7f0531efd640 10 monclient(hunting): handle_auth_done global_id 764106 payload 824 2024-09-16T03:47:02.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.668+0000 7f0531efd640 1 -- 172.21.15.42:0/534757398 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f052c09f180 con 0x7f052c133c40 2024-09-16T03:47:02.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.668+0000 7f05227fc640 20 client.764103.objecter dump_active .. 0 homeless 2024-09-16T03:47:02.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.668+0000 7f05227fc640 10 client.764103.objecter ms_handle_connect 0x7f052c133c40 2024-09-16T03:47:02.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.668+0000 7f05227fc640 10 client.764103.objecter resend_mon_ops 2024-09-16T03:47:02.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.668+0000 7f0531efd640 1 --2- 172.21.15.42:0/534757398 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f052c133c40 0x7f052c02c650 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f052800e770 tx=0x7f052800ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:47:02.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.668+0000 7f05326fe640 1 --2- 172.21.15.42:0/534757398 >> v2:172.21.15.154:6812/254845342 conn(0x7f0510037840 0x7f0510039d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:02.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.668+0000 7f05227fc640 1 -- 172.21.15.42:0/534757398 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0528017070 con 0x7f052c133c40 2024-09-16T03:47:02.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.668+0000 7f05227fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:47:02.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.668+0000 7f05227fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:47:02.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.668+0000 7f05227fc640 10 monclient: dump: 2024-09-16T03:47:02.694 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:47:02.695 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:02.695 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:02.695 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:02.695 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:47:02.695 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:02.695 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:47:02.695 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:47:02.695 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:47:02.695 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:02.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.668+0000 7f05326fe640 10 monclient: get_auth_request con 0x7f0510037840 auth_method 0 2024-09-16T03:47:02.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.668+0000 7f05227fc640 1 -- 172.21.15.42:0/534757398 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f052800c7a0 con 0x7f052c133c40 2024-09-16T03:47:02.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.668+0000 7f05227fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:47:02.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.668+0000 7f05227fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:47:02.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.668+0000 7f05227fc640 1 -- 172.21.15.42:0/534757398 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f052c133c40 msgr2=0x7f052c02c650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:02.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.668+0000 7f05227fc640 1 --2- 172.21.15.42:0/534757398 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f052c133c40 0x7f052c02c650 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f052800e770 tx=0x7f052800ec30 comp rx=0 tx=0).stop 2024-09-16T03:47:02.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.668+0000 7f05227fc640 1 -- 172.21.15.42:0/534757398 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f052c133c40 msgr2=0x7f052c02c650 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:47:02.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.668+0000 7f05227fc640 1 --2- 172.21.15.42:0/534757398 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f052c133c40 0x7f052c02c650 secure :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f052800e770 tx=0x7f052800ec30 comp rx=0 tx=0).stop 2024-09-16T03:47:02.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.668+0000 7f05326fe640 1 --2- 172.21.15.42:0/534757398 >> v2:172.21.15.154:6812/254845342 conn(0x7f0510037840 0x7f0510039d00 secure :-1 s=READY pgs=3428 cs=0 l=1 rev1=1 crypto rx=0x7f05240043d0 tx=0x7f0524009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:47:02.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.668+0000 7f05227fc640 10 client.764103.objecter ms_handle_connect 0x7f0510037840 2024-09-16T03:47:02.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.676+0000 7f0534161640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:47:02.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.676+0000 7f0534161640 1 --2- 172.21.15.42:0/534757398 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f052c02cb90 0x7f052c02cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:02.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.676+0000 7f0534161640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:02.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.676+0000 7f0534161640 1 -- 172.21.15.42:0/534757398 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f052c136a90 con 0x7f052c02cb90 2024-09-16T03:47:02.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.676+0000 7f0531efd640 1 --2- 172.21.15.42:0/534757398 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f052c02cb90 0x7f052c02cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:02.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.676+0000 7f0531efd640 10 monclient: get_auth_request con 0x7f052c02cb90 auth_method 0 2024-09-16T03:47:02.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.676+0000 7f0531efd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:02.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.676+0000 7f0531efd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:02.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.676+0000 7f0531efd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:47:02.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.676+0000 7f0531efd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:47:02.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.676+0000 7f0531efd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:47:02.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.676+0000 7f0531efd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:47:02.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.676+0000 7f0531efd640 10 monclient(hunting): handle_auth_done global_id 764109 payload 824 2024-09-16T03:47:02.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.676+0000 7f0531efd640 1 -- 172.21.15.42:0/534757398 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f052c06ee00 con 0x7f052c02cb90 2024-09-16T03:47:02.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.676+0000 7f0531efd640 1 --2- 172.21.15.42:0/534757398 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f052c02cb90 0x7f052c02cfa0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f0528019000 tx=0x7f0528004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:47:02.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.676+0000 7f05227fc640 10 client.764103.objecter ms_handle_connect 0x7f052c02cb90 2024-09-16T03:47:02.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.676+0000 7f05227fc640 10 client.764103.objecter resend_mon_ops 2024-09-16T03:47:02.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.676+0000 7f05227fc640 1 -- 172.21.15.42:0/534757398 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0528017070 con 0x7f052c02cb90 2024-09-16T03:47:02.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.676+0000 7f05227fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:47:02.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.676+0000 7f05227fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:47:02.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.676+0000 7f05227fc640 10 monclient: dump: 2024-09-16T03:47:02.700 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:47:02.700 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:02.700 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:02.700 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:02.700 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:47:02.700 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:02.700 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:47:02.700 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:47:02.700 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:47:02.700 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:02.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.676+0000 7f05227fc640 1 -- 172.21.15.42:0/534757398 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4546 (secure 0 0 0) 0x7f0528006310 con 0x7f052c02cb90 2024-09-16T03:47:02.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.676+0000 7f05227fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:47:02.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.676+0000 7f05227fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:47:02.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.676+0000 7f05227fc640 1 -- 172.21.15.42:0/534757398 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f052c02cb90 msgr2=0x7f052c02cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:02.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.676+0000 7f05227fc640 1 --2- 172.21.15.42:0/534757398 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f052c02cb90 0x7f052c02cfa0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f0528019000 tx=0x7f0528004740 comp rx=0 tx=0).stop 2024-09-16T03:47:02.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.676+0000 7f05227fc640 1 -- 172.21.15.42:0/534757398 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f052c02cb90 msgr2=0x7f052c02cfa0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:47:02.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.676+0000 7f05227fc640 1 --2- 172.21.15.42:0/534757398 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f052c02cb90 0x7f052c02cfa0 secure :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f0528019000 tx=0x7f0528004740 comp rx=0 tx=0).stop 2024-09-16T03:47:02.701 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:47:02.701 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:47:02.701 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:47:02.701 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "leader", 2024-09-16T03:47:02.701 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 48, 2024-09-16T03:47:02.702 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 13467, 2024-09-16T03:47:02.702 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:47:02.702 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:47:02.702 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T03:47:02.702 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T03:47:02.702 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:47:02.702 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 0, 2024-09-16T03:47:02.702 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:47:02.702 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:47:02.702 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:47:02.702 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:47:02.702 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:47:02.703 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:47:02.703 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:47:02.703 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:47:02.703 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:47:02.703 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:47:02.703 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:47:02.703 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:47:02.703 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:47:02.703 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:47:02.703 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:47:02.703 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:47:02.704 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:47:02.704 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:47:02.704 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:47:02.704 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:47:02.704 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:47:02.704 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:47:02.704 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:47:02.704 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:47:02.705 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:47:02.705 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:47:02.705 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:47:02.705 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:47:02.705 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:47:02.705 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:02.705 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:47:02.705 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:47:02.705 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:47:02.706 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:47:02.706 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:47:02.706 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:47:02.706 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:47:02.706 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:47:02.706 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:47:02.706 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:47:02.706 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:47:02.706 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:47:02.706 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:47:02.706 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:47:02.706 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:47:02.706 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:47:02.707 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:47:02.707 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:47:02.707 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:47:02.707 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:47:02.707 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:47:02.707 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:47:02.707 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:47:02.707 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:47:02.707 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:47:02.707 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:47:02.708 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:47:02.708 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:47:02.708 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:47:02.708 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:47:02.708 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:02.708 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:47:02.708 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:02.708 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:47:02.708 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:47:02.708 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:47:02.708 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:47:02.709 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:02.709 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:47:02.709 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:47:02.709 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:47:02.709 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:02.709 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:02.709 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:47:02.709 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:47:02.709 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:47:02.709 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:47:02.709 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:47:02.709 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:02.710 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:47:02.710 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:47:02.710 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:47:02.710 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:47:02.710 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:47:02.710 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:02.710 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:02.710 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:47:02.710 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:47:02.710 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:47:02.710 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:47:02.710 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:02.710 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:47:02.711 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:47:02.711 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:47:02.711 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:02.711 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:02.711 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:47:02.711 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:47:02.711 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:47:02.711 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:47:02.711 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:47:02.711 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:02.711 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:47:02.711 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:47:02.711 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:47:02.711 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:47:02.712 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:47:02.712 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:02.712 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:02.712 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:47:02.712 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:47:02.712 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:47:02.712 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:47:02.712 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:02.712 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:47:02.712 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:47:02.712 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:47:02.712 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:02.712 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:02.713 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:47:02.713 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:47:02.713 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:47:02.713 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:47:02.713 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:47:02.713 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:02.713 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:47:02.713 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:47:02.713 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:47:02.713 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:47:02.713 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:47:02.713 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:47:02.713 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:47:02.714 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:02.714 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:47:02.714 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:47:02.714 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:02.714 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:47:02.714 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:47:02.714 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:47:02.714 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:47:02.714 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:47:02.714 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:47:02.714 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:02.714 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:47:02.714 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:47:02.715 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 2 2024-09-16T03:47:02.715 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:47:02.715 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:47:02.715 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:02.715 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:47:02.715 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:47:02.717 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.676+0000 7f0534161640 10 librados: watch_flush enter 2024-09-16T03:47:02.717 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.676+0000 7f0534161640 10 librados: watch_flush exit 2024-09-16T03:47:02.717 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.676+0000 7f0534161640 20 client.764103.objecter shutdown clearing up homeless session... 2024-09-16T03:47:02.717 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.676+0000 7f0534161640 10 client.764103.objecter successfully canceled tick 2024-09-16T03:47:02.717 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.676+0000 7f0534161640 1 -- 172.21.15.42:0/534757398 >> v2:172.21.15.154:6812/254845342 conn(0x7f0510037840 msgr2=0x7f0510039d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:02.718 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.676+0000 7f0534161640 1 --2- 172.21.15.42:0/534757398 >> v2:172.21.15.154:6812/254845342 conn(0x7f0510037840 0x7f0510039d00 secure :-1 s=READY pgs=3428 cs=0 l=1 rev1=1 crypto rx=0x7f05240043d0 tx=0x7f0524009040 comp rx=0 tx=0).stop 2024-09-16T03:47:02.718 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.676+0000 7f0534161640 10 monclient: shutdown 2024-09-16T03:47:02.718 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.676+0000 7f0534161640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:47:02.718 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.676+0000 7f0534161640 1 -- 172.21.15.42:0/534757398 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f052c0a1710 msgr2=0x7f052c133700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:02.718 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.676+0000 7f05326fe640 1 -- 172.21.15.42:0/534757398 reap_dead start 2024-09-16T03:47:02.718 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.676+0000 7f0534161640 1 --2- 172.21.15.42:0/534757398 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f052c0a1710 0x7f052c133700 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f051c01aae0 tx=0x7f051c0191d0 comp rx=0 tx=0).stop 2024-09-16T03:47:02.718 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.676+0000 7f0534161640 1 -- 172.21.15.42:0/534757398 shutdown_connections 2024-09-16T03:47:02.718 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.676+0000 7f0534161640 1 --2- 172.21.15.42:0/534757398 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f052c0a1710 0x7f052c133700 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:02.718 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.676+0000 7f0534161640 1 -- 172.21.15.42:0/534757398 >> 172.21.15.42:0/534757398 conn(0x7f052c0a2420 msgr2=0x7f052c13b920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:47:02.718 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.676+0000 7f0534161640 1 -- 172.21.15.42:0/534757398 shutdown_connections 2024-09-16T03:47:02.718 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.676+0000 7f0534161640 1 -- 172.21.15.42:0/534757398 wait complete. 2024-09-16T03:47:02.718 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.676+0000 7f0534161640 1 librados: shutdown 2024-09-16T03:47:02.719 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:47:02.719 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T03:47:02.719 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T03:47:02.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.836+0000 7ff9bbc2f640 10 monclient: get_monmap_and_config 2024-09-16T03:47:02.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.836+0000 7ff9bbc2f640 10 monclient: build_initial_monmap 2024-09-16T03:47:02.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.836+0000 7ff9bbc2f640 10 monclient: monmap: 2024-09-16T03:47:02.840 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:47:02.840 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:02.840 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:47:02.838974+0000 2024-09-16T03:47:02.840 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:47:02.838974+0000 2024-09-16T03:47:02.840 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:47:02.840 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:02.841 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:47:02.841 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:47:02.841 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:47:02.841 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:47:02.841 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:47:02.841 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:02.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.840+0000 7ff9bbc2f640 1 Processor -- start 2024-09-16T03:47:02.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.840+0000 7ff9bbc2f640 1 -- start start 2024-09-16T03:47:02.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.840+0000 7ff9bbc2f640 10 monclient: init 2024-09-16T03:47:02.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.840+0000 7ff9bbc2f640 10 monclient: _reopen_session rank -1 2024-09-16T03:47:02.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.840+0000 7ff9bbc2f640 10 monclient: _add_conns ranks=[2,3,4,0,1] 2024-09-16T03:47:02.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.840+0000 7ff9bbc2f640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff9b406db90 0x7ff9b41340c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:02.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.840+0000 7ff9bbc2f640 10 monclient(hunting): picked mon.noname-b con 0x7ff9b406db90 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:47:02.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.840+0000 7ff9bbc2f640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff9b40a37f0 0x7ff9b40a3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:02.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.840+0000 7ff9bbc2f640 10 monclient(hunting): picked mon.noname-a con 0x7ff9b40a37f0 addr v2:172.21.15.42:3300/0 2024-09-16T03:47:02.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.840+0000 7ff9bbc2f640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7ff9b4136e10 0x7ff9b4137220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:02.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.840+0000 7ff9bbc2f640 10 monclient(hunting): picked mon.noname-c con 0x7ff9b4136e10 addr v2:172.21.15.154:3300/0 2024-09-16T03:47:02.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.840+0000 7ff9bbc2f640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:02.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.840+0000 7ff9bbc2f640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7ff9b40a4110 con 0x7ff9b40a37f0 2024-09-16T03:47:02.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.840+0000 7ff9bbc2f640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:02.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.840+0000 7ff9bbc2f640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7ff9b406df70 con 0x7ff9b4136e10 2024-09-16T03:47:02.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.840+0000 7ff9bbc2f640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:02.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.840+0000 7ff9bbc2f640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7ff9b406ece0 con 0x7ff9b406db90 2024-09-16T03:47:02.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.840+0000 7ff9bbc2f640 10 monclient(hunting): _renew_subs 2024-09-16T03:47:02.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.840+0000 7ff9bbc2f640 10 monclient(hunting): authenticate will time out at 1269.809086s 2024-09-16T03:47:02.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.840+0000 7ff9b99cb640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff9b406db90 0x7ff9b41340c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:02.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.840+0000 7ff9b91ca640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff9b40a37f0 0x7ff9b40a3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:02.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.840+0000 7ff9b99cb640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff9b406db90 0x7ff9b41340c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:57384/0 (socket says 172.21.15.42:57384) 2024-09-16T03:47:02.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.840+0000 7ff9b99cb640 1 -- 172.21.15.42:0/3216154604 learned_addr learned my addr 172.21.15.42:0/3216154604 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:47:02.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.840+0000 7ff9b99cb640 10 monclient(hunting): get_auth_request con 0x7ff9b406db90 auth_method 0 2024-09-16T03:47:02.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.840+0000 7ff9b99cb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:02.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.840+0000 7ff9b99cb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:02.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.840+0000 7ff9b99cb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:47:02.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.840+0000 7ff9b91ca640 10 monclient(hunting): get_auth_request con 0x7ff9b40a37f0 auth_method 0 2024-09-16T03:47:02.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.840+0000 7ff9b91ca640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:02.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.840+0000 7ff9b91ca640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:02.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.840+0000 7ff9b91ca640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:47:02.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.840+0000 7ff9b99cb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:47:02.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.840+0000 7ff9b99cb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:47:02.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.840+0000 7ff9b99cb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:47:02.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.840+0000 7ff9b91ca640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:47:02.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.840+0000 7ff9b91ca640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:47:02.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.840+0000 7ff9b91ca640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:47:02.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.840+0000 7ff9ba1cc640 1 --2- 172.21.15.42:0/3216154604 >> v2:172.21.15.154:3300/0 conn(0x7ff9b4136e10 0x7ff9b4137220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:02.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.840+0000 7ff9b91ca640 10 monclient(hunting): handle_auth_done global_id 764112 payload 274 2024-09-16T03:47:02.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.840+0000 7ff9b91ca640 1 -- 172.21.15.42:0/3216154604 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff9b406db90 msgr2=0x7ff9b41340c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:47:02.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.840+0000 7ff9b91ca640 1 --2- 172.21.15.42:0/3216154604 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff9b406db90 0x7ff9b41340c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:02.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.840+0000 7ff9b91ca640 1 -- 172.21.15.42:0/3216154604 >> v2:172.21.15.154:3300/0 conn(0x7ff9b4136e10 msgr2=0x7ff9b4137220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:47:02.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.840+0000 7ff9b91ca640 1 --2- 172.21.15.42:0/3216154604 >> v2:172.21.15.154:3300/0 conn(0x7ff9b4136e10 0x7ff9b4137220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:02.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.840+0000 7ff9b91ca640 10 monclient: _finish_hunting 0 2024-09-16T03:47:02.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.840+0000 7ff9b91ca640 1 monclient: found mon.noname-a 2024-09-16T03:47:02.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.840+0000 7ff9b91ca640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:47:02.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.840+0000 7ff9b91ca640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:47:02.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.840+0000 7ff9b91ca640 1 -- 172.21.15.42:0/3216154604 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9b409bf80 con 0x7ff9b40a37f0 2024-09-16T03:47:02.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.840+0000 7ff9ba1cc640 10 monclient: get_auth_request con 0x7ff9b4136e10 auth_method 0 2024-09-16T03:47:02.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.840+0000 7ff9b99cb640 1 --2- 172.21.15.42:0/3216154604 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff9b406db90 0x7ff9b41340c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:47:02.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.840+0000 7ff9ba1cc640 1 --2- 172.21.15.42:0/3216154604 >> v2:172.21.15.154:3300/0 conn(0x7ff9b4136e10 0x7ff9b4137220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T03:47:02.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.840+0000 7ff9b91ca640 1 --2- 172.21.15.42:0/3216154604 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff9b40a37f0 0x7ff9b40a3bd0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7ff9b40a7a60 tx=0x7ff9a401c320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58199f6113f81601 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:47:02.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.840+0000 7ff9b89c9640 1 -- 172.21.15.42:0/3216154604 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9a40b0070 con 0x7ff9b40a37f0 2024-09-16T03:47:02.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.840+0000 7ff9b89c9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:47:02.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.840+0000 7ff9b89c9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:47:02.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.840+0000 7ff9b89c9640 10 monclient: dump: 2024-09-16T03:47:02.856 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:47:02.856 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:02.856 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:02.856 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:02.856 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:47:02.857 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:02.857 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:47:02.857 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:47:02.857 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:47:02.857 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:02.857 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.840+0000 7ff9b89c9640 10 monclient: _finish_auth 0 2024-09-16T03:47:02.857 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.840+0000 7ff9b89c9640 10 monclient: _check_auth_tickets 2024-09-16T03:47:02.857 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.840+0000 7ff9b89c9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:47:02.857 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.840+0000 7ff9b89c9640 1 -- 172.21.15.42:0/3216154604 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff9a40ab600 con 0x7ff9b40a37f0 2024-09-16T03:47:02.857 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.840+0000 7ff9bbc2f640 5 monclient: authenticate success, global_id 764112 2024-09-16T03:47:02.857 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.840+0000 7ff9bbc2f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:47:02.858 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.840+0000 7ff9b89c9640 10 monclient: handle_config config(0 keys) 2024-09-16T03:47:02.858 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.844+0000 7ff9bbc2f640 10 monclient: get_monmap_and_config success 2024-09-16T03:47:02.858 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.844+0000 7ff9bbc2f640 4 set_mon_vals no callback set 2024-09-16T03:47:02.858 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.844+0000 7ff9b89c9640 1 -- 172.21.15.42:0/3216154604 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9a40a4e60 con 0x7ff9b40a37f0 2024-09-16T03:47:02.858 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.844+0000 7ff9b89c9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:47:02.858 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.844+0000 7ff9b89c9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:47:02.858 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.844+0000 7ff9b89c9640 10 monclient: dump: 2024-09-16T03:47:02.858 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:47:02.858 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:02.858 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:02.858 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:02.858 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:47:02.858 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:02.859 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:47:02.859 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:47:02.859 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:47:02.859 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:02.859 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.844+0000 7ff9bbc2f640 10 monclient: shutdown 2024-09-16T03:47:02.859 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.844+0000 7ff9bbc2f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:47:02.859 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.844+0000 7ff9bbc2f640 1 -- 172.21.15.42:0/3216154604 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff9b40a37f0 msgr2=0x7ff9b40a3bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:02.859 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.844+0000 7ff9bbc2f640 1 --2- 172.21.15.42:0/3216154604 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff9b40a37f0 0x7ff9b40a3bd0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7ff9b40a7a60 tx=0x7ff9a401c320 comp rx=0 tx=0).stop 2024-09-16T03:47:02.859 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.844+0000 7ff9bbc2f640 1 -- 172.21.15.42:0/3216154604 shutdown_connections 2024-09-16T03:47:02.859 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.844+0000 7ff9bbc2f640 1 --2- 172.21.15.42:0/3216154604 >> v2:172.21.15.154:3300/0 conn(0x7ff9b4136e10 0x7ff9b4137220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:02.859 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.844+0000 7ff9bbc2f640 1 --2- 172.21.15.42:0/3216154604 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff9b40a37f0 0x7ff9b40a3bd0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:02.859 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.844+0000 7ff9bbc2f640 1 --2- 172.21.15.42:0/3216154604 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff9b406db90 0x7ff9b41340c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:02.859 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.844+0000 7ff9bbc2f640 1 -- 172.21.15.42:0/3216154604 >> 172.21.15.42:0/3216154604 conn(0x7ff9b40a9fb0 msgr2=0x7ff9b40aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:47:02.859 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.844+0000 7ff9bbc2f640 1 -- 172.21.15.42:0/3216154604 shutdown_connections 2024-09-16T03:47:02.860 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.844+0000 7ff9bbc2f640 1 -- 172.21.15.42:0/3216154604 wait complete. 2024-09-16T03:47:02.860 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.844+0000 7ff9bbc2f640 10 monclient: build_initial_monmap 2024-09-16T03:47:02.860 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.844+0000 7ff9bbc2f640 10 monclient: monmap: 2024-09-16T03:47:02.860 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:47:02.860 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:47:02.860 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:47:02.860 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:47:02.860 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:47:02.860 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:02.860 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:47:02.860 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:47:02.860 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:47:02.861 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:02.861 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.844+0000 7ff9bbc2f640 1 librados: starting msgr at 2024-09-16T03:47:02.861 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.844+0000 7ff9bbc2f640 1 librados: starting objecter 2024-09-16T03:47:02.861 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.844+0000 7ff9bbc2f640 1 Processor -- start 2024-09-16T03:47:02.861 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.844+0000 7ff9bbc2f640 1 -- start start 2024-09-16T03:47:02.861 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.844+0000 7ff9bbc2f640 1 librados: setting wanted keys 2024-09-16T03:47:02.861 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.844+0000 7ff9bbc2f640 1 librados: calling monclient init 2024-09-16T03:47:02.861 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.844+0000 7ff9bbc2f640 10 monclient: init 2024-09-16T03:47:02.861 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.844+0000 7ff9bbc2f640 10 monclient: _reopen_session rank -1 2024-09-16T03:47:02.861 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.844+0000 7ff9bbc2f640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:47:02.861 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.844+0000 7ff9bbc2f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff9b406db90 0x7ff9b413d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:02.861 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.844+0000 7ff9bbc2f640 10 monclient(hunting): picked mon.noname-b con 0x7ff9b406db90 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:47:02.861 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.844+0000 7ff9bbc2f640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff9b40a37f0 0x7ff9b41cc330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:02.862 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.844+0000 7ff9bbc2f640 10 monclient(hunting): picked mon.noname-a con 0x7ff9b40a37f0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:47:02.862 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.844+0000 7ff9bbc2f640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff9b4136e10 0x7ff9b41cc870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:02.862 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.844+0000 7ff9bbc2f640 10 monclient(hunting): picked mon.noname-c con 0x7ff9b4136e10 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:47:02.862 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.844+0000 7ff9bbc2f640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:02.862 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.844+0000 7ff9bbc2f640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7ff9b4140b40 con 0x7ff9b40a37f0 2024-09-16T03:47:02.862 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.844+0000 7ff9bbc2f640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:02.862 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.844+0000 7ff9bbc2f640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff9b41409c0 con 0x7ff9b406db90 2024-09-16T03:47:02.862 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.844+0000 7ff9bbc2f640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:02.862 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.844+0000 7ff9bbc2f640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7ff9b4140cc0 con 0x7ff9b4136e10 2024-09-16T03:47:02.862 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.844+0000 7ff9bbc2f640 10 monclient(hunting): _renew_subs 2024-09-16T03:47:02.862 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.844+0000 7ff9bbc2f640 10 monclient(hunting): authenticate will time out at 1269.814084s 2024-09-16T03:47:02.862 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.844+0000 7ff9b91ca640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff9b40a37f0 0x7ff9b41cc330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:02.862 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.844+0000 7ff9ba1cc640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff9b4136e10 0x7ff9b41cc870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:02.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.844+0000 7ff9b91ca640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff9b40a37f0 0x7ff9b41cc330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:57484/0 (socket says 172.21.15.42:57484) 2024-09-16T03:47:02.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.844+0000 7ff9b91ca640 1 -- 172.21.15.42:0/1576550259 learned_addr learned my addr 172.21.15.42:0/1576550259 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:47:02.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.844+0000 7ff9b91ca640 10 monclient(hunting): get_auth_request con 0x7ff9b40a37f0 auth_method 0 2024-09-16T03:47:02.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.844+0000 7ff9b91ca640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:02.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.844+0000 7ff9b91ca640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:02.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.844+0000 7ff9b91ca640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:47:02.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.844+0000 7ff9ba1cc640 10 monclient(hunting): get_auth_request con 0x7ff9b4136e10 auth_method 0 2024-09-16T03:47:02.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.844+0000 7ff9ba1cc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:02.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.844+0000 7ff9ba1cc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:02.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.844+0000 7ff9ba1cc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:47:02.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.844+0000 7ff9b91ca640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:47:02.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.844+0000 7ff9b91ca640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:47:02.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.844+0000 7ff9b91ca640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:47:02.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.844+0000 7ff9ba1cc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:47:02.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.844+0000 7ff9ba1cc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:47:02.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.844+0000 7ff9ba1cc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:47:02.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.844+0000 7ff9b99cb640 1 --2- 172.21.15.42:0/1576550259 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff9b406db90 0x7ff9b413d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:02.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.844+0000 7ff9b99cb640 10 monclient(hunting): get_auth_request con 0x7ff9b406db90 auth_method 0 2024-09-16T03:47:02.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.844+0000 7ff9b99cb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:02.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.844+0000 7ff9b99cb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:02.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.844+0000 7ff9b99cb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:47:02.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.844+0000 7ff9b91ca640 10 monclient(hunting): handle_auth_done global_id 764115 payload 824 2024-09-16T03:47:02.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.844+0000 7ff9b91ca640 1 -- 172.21.15.42:0/1576550259 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff9b4136e10 msgr2=0x7ff9b41cc870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:02.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.844+0000 7ff9b91ca640 1 --2- 172.21.15.42:0/1576550259 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff9b4136e10 0x7ff9b41cc870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:02.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.844+0000 7ff9b91ca640 1 -- 172.21.15.42:0/1576550259 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff9b406db90 msgr2=0x7ff9b413d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:02.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.844+0000 7ff9b91ca640 1 --2- 172.21.15.42:0/1576550259 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff9b406db90 0x7ff9b413d3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:02.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.844+0000 7ff9b91ca640 10 monclient: _finish_hunting 0 2024-09-16T03:47:02.865 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.844+0000 7ff9b91ca640 1 monclient: found mon.noname-a 2024-09-16T03:47:02.865 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.844+0000 7ff9b91ca640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:47:02.865 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.844+0000 7ff9b91ca640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:47:02.865 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.844+0000 7ff9b91ca640 1 -- 172.21.15.42:0/1576550259 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9b409f5f0 con 0x7ff9b40a37f0 2024-09-16T03:47:02.865 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.844+0000 7ff9ba1cc640 1 --2- 172.21.15.42:0/1576550259 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff9b4136e10 0x7ff9b41cc870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:47:02.865 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.844+0000 7ff9b91ca640 1 --2- 172.21.15.42:0/1576550259 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff9b40a37f0 0x7ff9b41cc330 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7ff9a401a9d0 tx=0x7ff9a40190c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:47:02.865 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.844+0000 7ff9aa7fc640 10 client.?.objecter ms_handle_connect 0x7ff9b40a37f0 2024-09-16T03:47:02.865 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.844+0000 7ff9aa7fc640 10 client.?.objecter resend_mon_ops 2024-09-16T03:47:02.865 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.844+0000 7ff9aa7fc640 1 -- 172.21.15.42:0/1576550259 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9a40b0040 con 0x7ff9b40a37f0 2024-09-16T03:47:02.865 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.844+0000 7ff9aa7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:47:02.865 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.844+0000 7ff9aa7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:47:02.865 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.844+0000 7ff9aa7fc640 10 monclient: dump: 2024-09-16T03:47:02.865 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:47:02.866 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:02.866 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:02.866 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:02.866 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:47:02.866 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:02.866 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:47:02.866 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:47:02.866 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:47:02.866 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:02.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.844+0000 7ff9aa7fc640 10 monclient: _finish_auth 0 2024-09-16T03:47:02.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.844+0000 7ff9aa7fc640 10 monclient: _check_auth_tickets 2024-09-16T03:47:02.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.844+0000 7ff9aa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:47:02.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.844+0000 7ff9bbc2f640 5 monclient: authenticate success, global_id 764115 2024-09-16T03:47:02.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.844+0000 7ff9aa7fc640 1 -- 172.21.15.42:0/1576550259 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff9a40b9070 con 0x7ff9b40a37f0 2024-09-16T03:47:02.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.844+0000 7ff9bbc2f640 10 monclient: _renew_subs 2024-09-16T03:47:02.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.844+0000 7ff9bbc2f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:47:02.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.844+0000 7ff9bbc2f640 1 -- 172.21.15.42:0/1576550259 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9b40a8b90 con 0x7ff9b40a37f0 2024-09-16T03:47:02.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.844+0000 7ff9aa7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:47:02.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.844+0000 7ff9aa7fc640 1 -- 172.21.15.42:0/1576550259 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9a40b4650 con 0x7ff9b40a37f0 2024-09-16T03:47:02.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.844+0000 7ff9aa7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:47:02.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.844+0000 7ff9aa7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:47:02.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.844+0000 7ff9aa7fc640 10 monclient: dump: 2024-09-16T03:47:02.867 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:47:02.867 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:02.867 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:02.868 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:02.868 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:47:02.868 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:02.868 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:47:02.868 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:47:02.868 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:47:02.868 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:02.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.844+0000 7ff9ab7fe640 4 set_mon_vals no callback set 2024-09-16T03:47:02.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.844+0000 7ff9bbc2f640 10 client.764115.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:47:02.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.844+0000 7ff9bbc2f640 10 monclient: _renew_subs 2024-09-16T03:47:02.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.844+0000 7ff9bbc2f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:47:02.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.844+0000 7ff9bbc2f640 1 -- 172.21.15.42:0/1576550259 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9b409e380 con 0x7ff9b40a37f0 2024-09-16T03:47:02.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.844+0000 7ff9bbc2f640 1 librados: init done 2024-09-16T03:47:02.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.844+0000 7ff9bbc2f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:47:02.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.844+0000 7ff9bbc2f640 1 --2- 172.21.15.42:0/1576550259 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff9b409c710 0x7ff9b409caf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:02.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.844+0000 7ff9bbc2f640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:02.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.844+0000 7ff9bbc2f640 1 -- 172.21.15.42:0/1576550259 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7ff9b409f030 con 0x7ff9b409c710 2024-09-16T03:47:02.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.844+0000 7ff9b99cb640 1 --2- 172.21.15.42:0/1576550259 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff9b409c710 0x7ff9b409caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:02.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.848+0000 7ff9b99cb640 10 monclient: get_auth_request con 0x7ff9b409c710 auth_method 0 2024-09-16T03:47:02.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.848+0000 7ff9b99cb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:02.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.848+0000 7ff9b99cb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:02.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.848+0000 7ff9b99cb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:47:02.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.848+0000 7ff9aa7fc640 1 -- 172.21.15.42:0/1576550259 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7ff9a40b47f0 con 0x7ff9b40a37f0 2024-09-16T03:47:02.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.848+0000 7ff9b99cb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:47:02.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.848+0000 7ff9b99cb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:47:02.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.848+0000 7ff9b99cb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:47:02.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.848+0000 7ff9aa7fc640 1 --2- 172.21.15.42:0/1576550259 >> v2:172.21.15.154:6812/254845342 conn(0x7ff998037810 0x7ff998039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:02.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.848+0000 7ff9b99cb640 10 monclient(hunting): handle_auth_done global_id 13679 payload 824 2024-09-16T03:47:02.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.848+0000 7ff9b99cb640 1 -- 172.21.15.42:0/1576550259 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff9b40a3bd0 con 0x7ff9b409c710 2024-09-16T03:47:02.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.848+0000 7ff9aa7fc640 1 -- 172.21.15.42:0/1576550259 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(146..146 src has 1..146) ==== 7264+0+0 (secure 0 0 0) 0x7ff9a40e31d0 con 0x7ff9b40a37f0 2024-09-16T03:47:02.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.848+0000 7ff9aa7fc640 10 client.764115.objecter ms_dispatch 0x7ff9b4000ba0 osd_map(146..146 src has 1..146) 2024-09-16T03:47:02.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.848+0000 7ff9aa7fc640 3 client.764115.objecter handle_osd_map got epochs [146,146] > 0 2024-09-16T03:47:02.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.848+0000 7ff9aa7fc640 3 client.764115.objecter handle_osd_map decoding full epoch 146 2024-09-16T03:47:02.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.848+0000 7ff9aa7fc640 20 client.764115.objecter dump_active .. 0 homeless 2024-09-16T03:47:02.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.848+0000 7ff9aa7fc640 10 client.764115.objecter ms_handle_connect 0x7ff9b409c710 2024-09-16T03:47:02.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.848+0000 7ff9aa7fc640 10 client.764115.objecter resend_mon_ops 2024-09-16T03:47:02.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.848+0000 7ff9b99cb640 1 --2- 172.21.15.42:0/1576550259 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff9b409c710 0x7ff9b409caf0 secure :-1 s=READY pgs=2465 cs=0 l=1 rev1=1 crypto rx=0x7ff9b0009fd0 tx=0x7ff9b0011990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:47:02.871 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.848+0000 7ff9aa7fc640 1 -- 172.21.15.42:0/1576550259 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9b0015870 con 0x7ff9b409c710 2024-09-16T03:47:02.871 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.848+0000 7ff9aa7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:47:02.871 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.848+0000 7ff9ba1cc640 1 --2- 172.21.15.42:0/1576550259 >> v2:172.21.15.154:6812/254845342 conn(0x7ff998037810 0x7ff998039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:02.871 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.848+0000 7ff9aa7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:47:02.871 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.848+0000 7ff9aa7fc640 10 monclient: dump: 2024-09-16T03:47:02.871 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:47:02.871 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:02.871 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:02.871 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:02.871 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:47:02.871 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:02.872 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:47:02.872 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:47:02.872 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:47:02.872 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:02.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.848+0000 7ff9ba1cc640 10 monclient: get_auth_request con 0x7ff998037810 auth_method 0 2024-09-16T03:47:02.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.848+0000 7ff9aa7fc640 1 -- 172.21.15.42:0/1576550259 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7ff9b0015870 con 0x7ff9b409c710 2024-09-16T03:47:02.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.848+0000 7ff9aa7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:47:02.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.848+0000 7ff9aa7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:47:02.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.848+0000 7ff9aa7fc640 1 -- 172.21.15.42:0/1576550259 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff9b409c710 msgr2=0x7ff9b409caf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:02.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.848+0000 7ff9aa7fc640 1 --2- 172.21.15.42:0/1576550259 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff9b409c710 0x7ff9b409caf0 secure :-1 s=READY pgs=2465 cs=0 l=1 rev1=1 crypto rx=0x7ff9b0009fd0 tx=0x7ff9b0011990 comp rx=0 tx=0).stop 2024-09-16T03:47:02.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.848+0000 7ff9aa7fc640 1 -- 172.21.15.42:0/1576550259 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff9b409c710 msgr2=0x7ff9b409caf0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:47:02.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.848+0000 7ff9aa7fc640 1 --2- 172.21.15.42:0/1576550259 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff9b409c710 0x7ff9b409caf0 secure :-1 s=CLOSED pgs=2465 cs=0 l=1 rev1=1 crypto rx=0x7ff9b0009fd0 tx=0x7ff9b0011990 comp rx=0 tx=0).stop 2024-09-16T03:47:02.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.848+0000 7ff9ba1cc640 1 --2- 172.21.15.42:0/1576550259 >> v2:172.21.15.154:6812/254845342 conn(0x7ff998037810 0x7ff998039cd0 secure :-1 s=READY pgs=3429 cs=0 l=1 rev1=1 crypto rx=0x7ff9ac007fd0 tx=0x7ff9ac007780 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:47:02.873 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.848+0000 7ff9aa7fc640 10 client.764115.objecter ms_handle_connect 0x7ff998037810 2024-09-16T03:47:02.874 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.852+0000 7ff9bbc2f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:47:02.875 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.852+0000 7ff9bbc2f640 1 --2- 172.21.15.42:0/1576550259 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff9b409d030 0x7ff9b4138f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:02.875 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.852+0000 7ff9bbc2f640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:02.875 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.852+0000 7ff9bbc2f640 1 -- 172.21.15.42:0/1576550259 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7ff9b409f370 con 0x7ff9b409d030 2024-09-16T03:47:02.875 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.852+0000 7ff9b99cb640 1 --2- 172.21.15.42:0/1576550259 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff9b409d030 0x7ff9b4138f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:02.875 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.852+0000 7ff9b99cb640 10 monclient: get_auth_request con 0x7ff9b409d030 auth_method 0 2024-09-16T03:47:02.875 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.852+0000 7ff9b99cb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:02.875 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.852+0000 7ff9b99cb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:02.875 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.852+0000 7ff9b99cb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:47:02.875 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.852+0000 7ff9b99cb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:47:02.875 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.852+0000 7ff9b99cb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:47:02.875 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.852+0000 7ff9b99cb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:47:02.875 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.852+0000 7ff9b99cb640 10 monclient(hunting): handle_auth_done global_id 13682 payload 824 2024-09-16T03:47:02.875 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.852+0000 7ff9b99cb640 1 -- 172.21.15.42:0/1576550259 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ff9b4068e40 con 0x7ff9b409d030 2024-09-16T03:47:02.876 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.852+0000 7ff9aa7fc640 10 client.764115.objecter ms_handle_connect 0x7ff9b409d030 2024-09-16T03:47:02.876 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.852+0000 7ff9aa7fc640 10 client.764115.objecter resend_mon_ops 2024-09-16T03:47:02.876 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.852+0000 7ff9b99cb640 1 --2- 172.21.15.42:0/1576550259 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff9b409d030 0x7ff9b4138f50 secure :-1 s=READY pgs=2466 cs=0 l=1 rev1=1 crypto rx=0x7ff9b0009fd0 tx=0x7ff9b0004740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:47:02.876 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.856+0000 7ff9aa7fc640 1 -- 172.21.15.42:0/1576550259 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9b0004020 con 0x7ff9b409d030 2024-09-16T03:47:02.876 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.856+0000 7ff9aa7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:47:02.876 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.856+0000 7ff9aa7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:47:02.876 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.856+0000 7ff9aa7fc640 10 monclient: dump: 2024-09-16T03:47:02.876 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:47:02.876 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:02.876 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:02.876 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:02.877 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:47:02.877 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:02.877 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:47:02.877 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:47:02.877 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:47:02.877 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:02.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.856+0000 7ff9aa7fc640 1 -- 172.21.15.42:0/1576550259 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4712 (secure 0 0 0) 0x7ff9b0004020 con 0x7ff9b409d030 2024-09-16T03:47:02.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.856+0000 7ff9aa7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:47:02.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.856+0000 7ff9aa7fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:47:02.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.856+0000 7ff9aa7fc640 1 -- 172.21.15.42:0/1576550259 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff9b409d030 msgr2=0x7ff9b4138f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:02.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.856+0000 7ff9aa7fc640 1 --2- 172.21.15.42:0/1576550259 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff9b409d030 0x7ff9b4138f50 secure :-1 s=READY pgs=2466 cs=0 l=1 rev1=1 crypto rx=0x7ff9b0009fd0 tx=0x7ff9b0004740 comp rx=0 tx=0).stop 2024-09-16T03:47:02.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.856+0000 7ff9aa7fc640 1 -- 172.21.15.42:0/1576550259 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff9b409d030 msgr2=0x7ff9b4138f50 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:47:02.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.856+0000 7ff9aa7fc640 1 --2- 172.21.15.42:0/1576550259 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff9b409d030 0x7ff9b4138f50 secure :-1 s=CLOSED pgs=2466 cs=0 l=1 rev1=1 crypto rx=0x7ff9b0009fd0 tx=0x7ff9b0004740 comp rx=0 tx=0).stop 2024-09-16T03:47:02.878 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:47:02.878 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:47:02.878 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:47:02.878 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T03:47:02.878 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 48, 2024-09-16T03:47:02.878 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 302226, 2024-09-16T03:47:02.879 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:47:02.879 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:47:02.879 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T03:47:02.879 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T03:47:02.879 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:47:02.879 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 0, 2024-09-16T03:47:02.879 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:47:02.879 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:47:02.879 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:47:02.879 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:47:02.879 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:47:02.879 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:47:02.879 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:47:02.880 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:47:02.880 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:47:02.880 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:47:02.880 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:47:02.880 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:47:02.880 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:47:02.880 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:47:02.880 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:47:02.880 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:47:02.880 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:47:02.880 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:47:02.880 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:47:02.880 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:47:02.881 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:47:02.881 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:47:02.881 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:47:02.881 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:47:02.881 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:47:02.881 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:47:02.881 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:47:02.881 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:47:02.881 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:47:02.881 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:02.881 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:47:02.881 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:47:02.881 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:47:02.882 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:47:02.882 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:47:02.882 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:47:02.882 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:47:02.882 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:47:02.882 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:47:02.882 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:47:02.882 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:47:02.882 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:47:02.882 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:47:02.882 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:47:02.882 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:47:02.882 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:47:02.883 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:47:02.883 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:47:02.883 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:47:02.883 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:47:02.883 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:47:02.883 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:47:02.883 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:47:02.883 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:47:02.883 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:47:02.883 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:47:02.883 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:47:02.883 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:47:02.883 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:47:02.884 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:47:02.884 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:02.884 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:47:02.884 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:02.884 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:47:02.884 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:47:02.884 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:47:02.884 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:47:02.884 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:02.885 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:47:02.885 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:47:02.885 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:47:02.885 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:02.885 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:02.885 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:47:02.885 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:47:02.885 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:47:02.885 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:47:02.885 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:47:02.885 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:02.885 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:47:02.885 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:47:02.886 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:47:02.886 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:47:02.886 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:47:02.886 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:02.886 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:02.886 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:47:02.886 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:47:02.886 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:47:02.886 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:47:02.886 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:02.886 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:47:02.886 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:47:02.886 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:47:02.887 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:02.887 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:02.887 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:47:02.887 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:47:02.887 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:47:02.887 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:47:02.887 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:47:02.887 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:02.887 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:47:02.887 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:47:02.887 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:47:02.887 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:47:02.887 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:47:02.887 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:02.888 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:02.888 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:47:02.888 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:47:02.888 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:47:02.888 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:47:02.888 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:02.888 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:47:02.888 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:47:02.888 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:47:02.888 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:02.888 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:02.888 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:47:02.889 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:47:02.889 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:47:02.889 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:47:02.889 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:47:02.889 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:02.889 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:47:02.889 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:47:02.889 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:47:02.889 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:47:02.889 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:47:02.889 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:47:02.889 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:47:02.890 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:02.890 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:47:02.890 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:47:02.890 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:02.890 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:47:02.890 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:47:02.890 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:47:02.890 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:47:02.890 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:47:02.890 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T03:47:02.890 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:02.890 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:47:02.890 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:47:02.890 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 6 2024-09-16T03:47:02.891 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:47:02.891 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:47:02.891 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:47:02.891 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:02.891 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:47:02.891 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:47:02.891 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 7 2024-09-16T03:47:02.891 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:47:02.891 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:47:02.891 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:02.891 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:47:02.891 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:47:02.893 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.856+0000 7ff9bbc2f640 10 librados: watch_flush enter 2024-09-16T03:47:02.893 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.856+0000 7ff9bbc2f640 10 librados: watch_flush exit 2024-09-16T03:47:02.893 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.856+0000 7ff9bbc2f640 20 client.764115.objecter shutdown clearing up homeless session... 2024-09-16T03:47:02.893 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.856+0000 7ff9bbc2f640 10 client.764115.objecter successfully canceled tick 2024-09-16T03:47:02.893 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.856+0000 7ff9bbc2f640 1 -- 172.21.15.42:0/1576550259 >> v2:172.21.15.154:6812/254845342 conn(0x7ff998037810 msgr2=0x7ff998039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:02.893 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.856+0000 7ff9bbc2f640 1 --2- 172.21.15.42:0/1576550259 >> v2:172.21.15.154:6812/254845342 conn(0x7ff998037810 0x7ff998039cd0 secure :-1 s=READY pgs=3429 cs=0 l=1 rev1=1 crypto rx=0x7ff9ac007fd0 tx=0x7ff9ac007780 comp rx=0 tx=0).stop 2024-09-16T03:47:02.893 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.856+0000 7ff9bbc2f640 10 monclient: shutdown 2024-09-16T03:47:02.894 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.856+0000 7ff9bbc2f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:47:02.894 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.856+0000 7ff9bbc2f640 1 -- 172.21.15.42:0/1576550259 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff9b40a37f0 msgr2=0x7ff9b41cc330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:02.894 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.856+0000 7ff9bbc2f640 1 --2- 172.21.15.42:0/1576550259 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff9b40a37f0 0x7ff9b41cc330 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7ff9a401a9d0 tx=0x7ff9a40190c0 comp rx=0 tx=0).stop 2024-09-16T03:47:02.894 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.856+0000 7ff9ba1cc640 1 -- 172.21.15.42:0/1576550259 reap_dead start 2024-09-16T03:47:02.894 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.856+0000 7ff9ba1cc640 1 -- 172.21.15.42:0/1576550259 reap_dead start 2024-09-16T03:47:02.894 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.856+0000 7ff9bbc2f640 1 -- 172.21.15.42:0/1576550259 shutdown_connections 2024-09-16T03:47:02.894 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.856+0000 7ff9bbc2f640 1 -- 172.21.15.42:0/1576550259 >> 172.21.15.42:0/1576550259 conn(0x7ff9b40a9fb0 msgr2=0x7ff9b41322e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:47:02.894 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.856+0000 7ff9bbc2f640 1 -- 172.21.15.42:0/1576550259 shutdown_connections 2024-09-16T03:47:02.894 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.856+0000 7ff9bbc2f640 1 -- 172.21.15.42:0/1576550259 wait complete. 2024-09-16T03:47:02.894 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:02.856+0000 7ff9bbc2f640 1 librados: shutdown 2024-09-16T03:47:02.894 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:47:02.895 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T03:47:02.895 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T03:47:03.019 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.016+0000 7fd7ca18a640 10 monclient: get_monmap_and_config 2024-09-16T03:47:03.020 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.016+0000 7fd7ca18a640 10 monclient: build_initial_monmap 2024-09-16T03:47:03.020 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.016+0000 7fd7ca18a640 10 monclient: monmap: 2024-09-16T03:47:03.020 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:47:03.020 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:03.020 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:47:03.018893+0000 2024-09-16T03:47:03.020 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:47:03.018893+0000 2024-09-16T03:47:03.020 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:47:03.020 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:03.020 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:47:03.020 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:47:03.020 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:47:03.020 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:47:03.021 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:47:03.021 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:03.023 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.016+0000 7fd7ca18a640 1 Processor -- start 2024-09-16T03:47:03.023 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.016+0000 7fd7ca18a640 1 -- start start 2024-09-16T03:47:03.023 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.016+0000 7fd7ca18a640 10 monclient: init 2024-09-16T03:47:03.023 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.020+0000 7fd7ca18a640 10 monclient: _reopen_session rank -1 2024-09-16T03:47:03.023 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.020+0000 7fd7ca18a640 10 monclient: _add_conns ranks=[1,3,2,4,0] 2024-09-16T03:47:03.023 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.020+0000 7fd7ca18a640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd7c40a67e0 addr v1:172.21.15.154:6789/0 2024-09-16T03:47:03.023 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.020+0000 7fd7ca18a640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd7c4072560 0x7fd7c413a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:03.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.020+0000 7fd7ca18a640 10 monclient(hunting): picked mon.noname-a con 0x7fd7c4072560 addr v2:172.21.15.42:3300/0 2024-09-16T03:47:03.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.020+0000 7fd7ca18a640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd7c40a72d0 0x7fd7c413c820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:03.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.020+0000 7fd7ca18a640 10 monclient(hunting): picked mon.noname-b con 0x7fd7c40a72d0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:47:03.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.020+0000 7fd7ca18a640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd7c4071b80 con 0x7fd7c40a67e0 2024-09-16T03:47:03.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.020+0000 7fd7ca18a640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:03.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.020+0000 7fd7ca18a640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7fd7c4072ed0 con 0x7fd7c4072560 2024-09-16T03:47:03.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.020+0000 7fd7ca18a640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:03.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.020+0000 7fd7ca18a640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fd7c4070030 con 0x7fd7c40a72d0 2024-09-16T03:47:03.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.020+0000 7fd7ca18a640 10 monclient(hunting): _renew_subs 2024-09-16T03:47:03.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.020+0000 7fd7ca18a640 10 monclient(hunting): authenticate will time out at 1269.988193s 2024-09-16T03:47:03.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.020+0000 7fd7bbfff640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd7c4072560 0x7fd7c413a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:03.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.020+0000 7fd7c3fff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd7c40a72d0 0x7fd7c413c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:03.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.020+0000 7fd7bbfff640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd7c4072560 0x7fd7c413a2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:57492/0 (socket says 172.21.15.42:57492) 2024-09-16T03:47:03.025 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.020+0000 7fd7bbfff640 1 -- 172.21.15.42:0/1628665543 learned_addr learned my addr 172.21.15.42:0/1628665543 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:47:03.025 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.020+0000 7fd7bbfff640 10 monclient(hunting): get_auth_request con 0x7fd7c4072560 auth_method 0 2024-09-16T03:47:03.025 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.020+0000 7fd7bbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:03.025 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.020+0000 7fd7bbfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:03.025 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.020+0000 7fd7bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:47:03.025 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.020+0000 7fd7c3fff640 10 monclient(hunting): get_auth_request con 0x7fd7c40a72d0 auth_method 0 2024-09-16T03:47:03.025 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.020+0000 7fd7c3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:03.025 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.020+0000 7fd7c3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:03.025 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.020+0000 7fd7c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:47:03.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.020+0000 7fd7bbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:47:03.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.020+0000 7fd7bbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:47:03.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.020+0000 7fd7bbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:47:03.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.020+0000 7fd7c3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:47:03.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.020+0000 7fd7c3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:47:03.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.020+0000 7fd7c3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:47:03.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.020+0000 7fd7c3fff640 10 monclient(hunting): handle_auth_done global_id 13685 payload 274 2024-09-16T03:47:03.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.020+0000 7fd7c3fff640 1 -- 172.21.15.42:0/1628665543 >> v2:172.21.15.42:3300/0 conn(0x7fd7c4072560 msgr2=0x7fd7c413a2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:47:03.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.020+0000 7fd7c3fff640 1 --2- 172.21.15.42:0/1628665543 >> v2:172.21.15.42:3300/0 conn(0x7fd7c4072560 0x7fd7c413a2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:03.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.020+0000 7fd7c3fff640 1 -- 172.21.15.42:0/1628665543 >> v1:172.21.15.154:6789/0 conn(0x7fd7c40a67e0 legacy=0x7fd7c40a6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:03.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.020+0000 7fd7c3fff640 10 monclient: _finish_hunting 0 2024-09-16T03:47:03.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.020+0000 7fd7c3fff640 1 monclient: found mon.noname-b 2024-09-16T03:47:03.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.020+0000 7fd7c3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:47:03.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.020+0000 7fd7c3fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T03:47:03.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.020+0000 7fd7c3fff640 1 -- 172.21.15.42:0/1628665543 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7c40712e0 con 0x7fd7c40a72d0 2024-09-16T03:47:03.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.020+0000 7fd7bbfff640 1 --2- 172.21.15.42:0/1628665543 >> v2:172.21.15.42:3300/0 conn(0x7fd7c4072560 0x7fd7c413a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:47:03.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.020+0000 7fd7c3fff640 1 --2- 172.21.15.42:0/1628665543 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd7c40a72d0 0x7fd7c413c820 secure :-1 s=READY pgs=2467 cs=0 l=1 rev1=1 crypto rx=0x7fd7bc009d90 tx=0x7fd7bc01c800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c7d7933c14aba0d2 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:47:03.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.020+0000 7fd7c2ffd640 1 -- 172.21.15.42:0/1628665543 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7bc0b2070 con 0x7fd7c40a72d0 2024-09-16T03:47:03.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.020+0000 7fd7c2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:47:03.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.020+0000 7fd7c2ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:47:03.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.020+0000 7fd7c2ffd640 10 monclient: dump: 2024-09-16T03:47:03.035 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:47:03.035 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:03.035 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:03.035 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:03.036 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:47:03.036 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:03.036 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:47:03.036 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:47:03.036 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:47:03.036 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:03.036 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.020+0000 7fd7c2ffd640 10 monclient: _finish_auth 0 2024-09-16T03:47:03.036 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.020+0000 7fd7c2ffd640 10 monclient: _check_auth_tickets 2024-09-16T03:47:03.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.020+0000 7fd7c2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:47:03.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.020+0000 7fd7c2ffd640 1 -- 172.21.15.42:0/1628665543 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7bc0a58e0 con 0x7fd7c40a72d0 2024-09-16T03:47:03.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.020+0000 7fd7ca18a640 5 monclient: authenticate success, global_id 13685 2024-09-16T03:47:03.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.020+0000 7fd7ca18a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:47:03.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.020+0000 7fd7c2ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:47:03.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.020+0000 7fd7ca18a640 10 monclient: get_monmap_and_config success 2024-09-16T03:47:03.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.020+0000 7fd7c2ffd640 1 -- 172.21.15.42:0/1628665543 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7bc0a5cb0 con 0x7fd7c40a72d0 2024-09-16T03:47:03.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.020+0000 7fd7c2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:47:03.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.020+0000 7fd7ca18a640 4 set_mon_vals no callback set 2024-09-16T03:47:03.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.020+0000 7fd7c2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:47:03.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.020+0000 7fd7c2ffd640 10 monclient: dump: 2024-09-16T03:47:03.037 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:47:03.037 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:03.038 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:03.038 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:03.038 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:47:03.038 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:03.038 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:47:03.038 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:47:03.038 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:47:03.038 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:03.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.020+0000 7fd7ca18a640 10 monclient: shutdown 2024-09-16T03:47:03.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.020+0000 7fd7ca18a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:47:03.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.020+0000 7fd7ca18a640 1 -- 172.21.15.42:0/1628665543 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd7c40a72d0 msgr2=0x7fd7c413c820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:03.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.020+0000 7fd7ca18a640 1 --2- 172.21.15.42:0/1628665543 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd7c40a72d0 0x7fd7c413c820 secure :-1 s=READY pgs=2467 cs=0 l=1 rev1=1 crypto rx=0x7fd7bc009d90 tx=0x7fd7bc01c800 comp rx=0 tx=0).stop 2024-09-16T03:47:03.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.020+0000 7fd7ca18a640 1 -- 172.21.15.42:0/1628665543 shutdown_connections 2024-09-16T03:47:03.039 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.020+0000 7fd7ca18a640 1 --2- 172.21.15.42:0/1628665543 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd7c40a72d0 0x7fd7c413c820 unknown :-1 s=CLOSED pgs=2467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:03.039 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.020+0000 7fd7ca18a640 1 --2- 172.21.15.42:0/1628665543 >> v2:172.21.15.42:3300/0 conn(0x7fd7c4072560 0x7fd7c413a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:03.039 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.020+0000 7fd7ca18a640 1 -- 172.21.15.42:0/1628665543 >> 172.21.15.42:0/1628665543 conn(0x7fd7c40a7ed0 msgr2=0x7fd7c40a43d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:47:03.039 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.020+0000 7fd7ca18a640 1 -- 172.21.15.42:0/1628665543 shutdown_connections 2024-09-16T03:47:03.039 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.020+0000 7fd7ca18a640 1 -- 172.21.15.42:0/1628665543 wait complete. 2024-09-16T03:47:03.039 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.024+0000 7fd7ca18a640 10 monclient: build_initial_monmap 2024-09-16T03:47:03.039 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.024+0000 7fd7ca18a640 10 monclient: monmap: 2024-09-16T03:47:03.039 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:47:03.039 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:47:03.040 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:47:03.040 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:47:03.040 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:47:03.040 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:03.040 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:47:03.040 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:47:03.040 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:47:03.040 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:03.040 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.024+0000 7fd7ca18a640 1 librados: starting msgr at 2024-09-16T03:47:03.040 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.024+0000 7fd7ca18a640 1 librados: starting objecter 2024-09-16T03:47:03.040 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.024+0000 7fd7ca18a640 1 Processor -- start 2024-09-16T03:47:03.040 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.024+0000 7fd7ca18a640 1 -- start start 2024-09-16T03:47:03.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.024+0000 7fd7ca18a640 1 librados: setting wanted keys 2024-09-16T03:47:03.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.024+0000 7fd7ca18a640 1 librados: calling monclient init 2024-09-16T03:47:03.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.024+0000 7fd7ca18a640 10 monclient: init 2024-09-16T03:47:03.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.024+0000 7fd7ca18a640 10 monclient: _reopen_session rank -1 2024-09-16T03:47:03.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.024+0000 7fd7ca18a640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:47:03.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.024+0000 7fd7ca18a640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd7c4072560 0x7fd7c41cc920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:03.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.024+0000 7fd7ca18a640 10 monclient(hunting): picked mon.noname-c con 0x7fd7c4072560 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:47:03.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.024+0000 7fd7ca18a640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd7c40a67e0 0x7fd7c4135160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:03.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.024+0000 7fd7ca18a640 10 monclient(hunting): picked mon.noname-b con 0x7fd7c40a67e0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:47:03.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.024+0000 7fd7ca18a640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd7c40a72d0 0x7fd7c413a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:03.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.024+0000 7fd7ca18a640 10 monclient(hunting): picked mon.noname-a con 0x7fd7c40a72d0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:47:03.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.024+0000 7fd7ca18a640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:03.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.024+0000 7fd7ca18a640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fd7c413f700 con 0x7fd7c40a72d0 2024-09-16T03:47:03.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.024+0000 7fd7ca18a640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:03.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.024+0000 7fd7ca18a640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd7c413f580 con 0x7fd7c40a67e0 2024-09-16T03:47:03.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.024+0000 7fd7ca18a640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:03.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.024+0000 7fd7ca18a640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fd7c413f880 con 0x7fd7c4072560 2024-09-16T03:47:03.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.024+0000 7fd7ca18a640 10 monclient(hunting): _renew_subs 2024-09-16T03:47:03.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.024+0000 7fd7ca18a640 10 monclient(hunting): authenticate will time out at 1269.993134s 2024-09-16T03:47:03.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.024+0000 7fd7c3fff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd7c40a72d0 0x7fd7c413a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:03.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.024+0000 7fd7c37fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd7c4072560 0x7fd7c41cc920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:03.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.024+0000 7fd7c3fff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd7c40a72d0 0x7fd7c413a0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:57494/0 (socket says 172.21.15.42:57494) 2024-09-16T03:47:03.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.024+0000 7fd7c3fff640 1 -- 172.21.15.42:0/1788492702 learned_addr learned my addr 172.21.15.42:0/1788492702 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:47:03.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.024+0000 7fd7c37fe640 10 monclient(hunting): get_auth_request con 0x7fd7c4072560 auth_method 0 2024-09-16T03:47:03.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.024+0000 7fd7c37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:03.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.024+0000 7fd7c37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:03.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.024+0000 7fd7c37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:47:03.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.024+0000 7fd7c3fff640 10 monclient(hunting): get_auth_request con 0x7fd7c40a72d0 auth_method 0 2024-09-16T03:47:03.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.024+0000 7fd7c3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:03.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.024+0000 7fd7c3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:03.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.024+0000 7fd7c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:47:03.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.024+0000 7fd7c37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:47:03.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.024+0000 7fd7c37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:47:03.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.024+0000 7fd7c37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:47:03.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.024+0000 7fd7c3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:47:03.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.024+0000 7fd7c3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:47:03.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.024+0000 7fd7c3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:47:03.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.024+0000 7fd7bbfff640 1 --2- 172.21.15.42:0/1788492702 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd7c40a67e0 0x7fd7c4135160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:03.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.024+0000 7fd7bbfff640 10 monclient(hunting): get_auth_request con 0x7fd7c40a67e0 auth_method 0 2024-09-16T03:47:03.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.024+0000 7fd7bbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:03.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.024+0000 7fd7bbfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:03.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.024+0000 7fd7bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:47:03.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.024+0000 7fd7c37fe640 10 monclient(hunting): handle_auth_done global_id 13688 payload 824 2024-09-16T03:47:03.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.024+0000 7fd7c37fe640 1 -- 172.21.15.42:0/1788492702 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd7c40a67e0 msgr2=0x7fd7c4135160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:03.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.024+0000 7fd7c37fe640 1 --2- 172.21.15.42:0/1788492702 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd7c40a67e0 0x7fd7c4135160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:03.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.024+0000 7fd7c37fe640 1 -- 172.21.15.42:0/1788492702 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd7c40a72d0 msgr2=0x7fd7c413a0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:03.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.024+0000 7fd7c37fe640 1 --2- 172.21.15.42:0/1788492702 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd7c40a72d0 0x7fd7c413a0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:03.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.024+0000 7fd7c37fe640 10 monclient: _finish_hunting 0 2024-09-16T03:47:03.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.024+0000 7fd7c37fe640 1 monclient: found mon.noname-c 2024-09-16T03:47:03.045 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.024+0000 7fd7c37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:47:03.045 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.024+0000 7fd7c37fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:47:03.045 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.024+0000 7fd7c37fe640 1 -- 172.21.15.42:0/1788492702 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7c411f810 con 0x7fd7c4072560 2024-09-16T03:47:03.045 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.024+0000 7fd7c37fe640 1 --2- 172.21.15.42:0/1788492702 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd7c4072560 0x7fd7c41cc920 secure :-1 s=READY pgs=2468 cs=0 l=1 rev1=1 crypto rx=0x7fd7b4001ec0 tx=0x7fd7b400c890 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:47:03.045 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.024+0000 7fd7c0ff9640 10 client.?.objecter ms_handle_connect 0x7fd7c4072560 2024-09-16T03:47:03.045 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.024+0000 7fd7c0ff9640 10 client.?.objecter resend_mon_ops 2024-09-16T03:47:03.045 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.024+0000 7fd7c0ff9640 1 -- 172.21.15.42:0/1788492702 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7b4017070 con 0x7fd7c4072560 2024-09-16T03:47:03.045 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.024+0000 7fd7c0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:47:03.045 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.024+0000 7fd7c0ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:47:03.045 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.024+0000 7fd7c0ff9640 10 monclient: dump: 2024-09-16T03:47:03.045 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:47:03.045 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:03.045 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:03.045 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:03.046 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:47:03.046 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:03.046 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:47:03.046 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:47:03.046 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:47:03.046 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:03.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.024+0000 7fd7c0ff9640 10 monclient: _finish_auth 0 2024-09-16T03:47:03.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.024+0000 7fd7c0ff9640 10 monclient: _check_auth_tickets 2024-09-16T03:47:03.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.024+0000 7fd7c0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:47:03.048 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.024+0000 7fd7c0ff9640 1 -- 172.21.15.42:0/1788492702 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7b40052d0 con 0x7fd7c4072560 2024-09-16T03:47:03.048 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.024+0000 7fd7ca18a640 5 monclient: authenticate success, global_id 13688 2024-09-16T03:47:03.048 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.024+0000 7fd7ca18a640 10 monclient: _renew_subs 2024-09-16T03:47:03.048 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.024+0000 7fd7ca18a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:47:03.048 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.024+0000 7fd7ca18a640 1 -- 172.21.15.42:0/1788492702 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7c409bbe0 con 0x7fd7c4072560 2024-09-16T03:47:03.048 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.024+0000 7fd7c0ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T03:47:03.048 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.024+0000 7fd7c0ff9640 1 -- 172.21.15.42:0/1788492702 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7b4003840 con 0x7fd7c4072560 2024-09-16T03:47:03.048 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.024+0000 7fd7c0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:47:03.048 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.024+0000 7fd7c0ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:47:03.048 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.024+0000 7fd7c0ff9640 10 monclient: dump: 2024-09-16T03:47:03.048 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:47:03.048 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:03.048 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:03.049 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:03.049 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:47:03.049 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:03.049 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:47:03.049 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:47:03.049 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:47:03.049 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:03.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.024+0000 7fd7ca18a640 10 client.13688.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:47:03.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.024+0000 7fd7ca18a640 10 monclient: _renew_subs 2024-09-16T03:47:03.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.024+0000 7fd7ca18a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:47:03.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.024+0000 7fd7ca18a640 1 -- 172.21.15.42:0/1788492702 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7c411f530 con 0x7fd7c4072560 2024-09-16T03:47:03.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.024+0000 7fd7ca18a640 1 librados: init done 2024-09-16T03:47:03.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.024+0000 7fd7c27fc640 4 set_mon_vals no callback set 2024-09-16T03:47:03.050 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.024+0000 7fd7ca18a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:47:03.050 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.024+0000 7fd7ca18a640 1 --2- 172.21.15.42:0/1788492702 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd7c41356a0 0x7fd7c413a600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:03.050 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.024+0000 7fd7ca18a640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:03.050 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.024+0000 7fd7ca18a640 1 -- 172.21.15.42:0/1788492702 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd7c40a6bc0 con 0x7fd7c41356a0 2024-09-16T03:47:03.050 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.024+0000 7fd7c0ff9640 1 -- 172.21.15.42:0/1788492702 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fd7b401a070 con 0x7fd7c4072560 2024-09-16T03:47:03.050 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.024+0000 7fd7bbfff640 1 --2- 172.21.15.42:0/1788492702 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd7c41356a0 0x7fd7c413a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:03.050 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.024+0000 7fd7bbfff640 10 monclient: get_auth_request con 0x7fd7c41356a0 auth_method 0 2024-09-16T03:47:03.050 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.024+0000 7fd7bbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:03.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.024+0000 7fd7bbfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:03.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.024+0000 7fd7bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:47:03.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.024+0000 7fd7c0ff9640 1 --2- 172.21.15.42:0/1788492702 >> v2:172.21.15.154:6812/254845342 conn(0x7fd7a4037810 0x7fd7a4039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:03.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.024+0000 7fd7bbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:47:03.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.024+0000 7fd7bbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:47:03.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.024+0000 7fd7bbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:47:03.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.024+0000 7fd7c0ff9640 1 -- 172.21.15.42:0/1788492702 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(147..147 src has 1..147) ==== 6868+0+0 (secure 0 0 0) 0x7fd7b4048780 con 0x7fd7c4072560 2024-09-16T03:47:03.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.024+0000 7fd7c0ff9640 10 client.13688.objecter ms_dispatch 0x7fd7c4000ba0 osd_map(147..147 src has 1..147) 2024-09-16T03:47:03.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.024+0000 7fd7c0ff9640 3 client.13688.objecter handle_osd_map got epochs [147,147] > 0 2024-09-16T03:47:03.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.024+0000 7fd7c0ff9640 3 client.13688.objecter handle_osd_map decoding full epoch 147 2024-09-16T03:47:03.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.024+0000 7fd7c0ff9640 20 client.13688.objecter dump_active .. 0 homeless 2024-09-16T03:47:03.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.024+0000 7fd7bbfff640 10 monclient(hunting): handle_auth_done global_id 754270 payload 824 2024-09-16T03:47:03.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.024+0000 7fd7bbfff640 1 -- 172.21.15.42:0/1788492702 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd7c411e0b0 con 0x7fd7c41356a0 2024-09-16T03:47:03.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.024+0000 7fd7c3fff640 1 --2- 172.21.15.42:0/1788492702 >> v2:172.21.15.154:6812/254845342 conn(0x7fd7a4037810 0x7fd7a4039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:03.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.024+0000 7fd7c3fff640 10 monclient: get_auth_request con 0x7fd7a4037810 auth_method 0 2024-09-16T03:47:03.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.024+0000 7fd7bbfff640 1 --2- 172.21.15.42:0/1788492702 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd7c41356a0 0x7fd7c413a600 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fd7c411eca0 tx=0x7fd7ac011a60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:47:03.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.024+0000 7fd7c0ff9640 10 client.13688.objecter ms_handle_connect 0x7fd7c41356a0 2024-09-16T03:47:03.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.024+0000 7fd7c0ff9640 10 client.13688.objecter resend_mon_ops 2024-09-16T03:47:03.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.028+0000 7fd7c0ff9640 1 -- 172.21.15.42:0/1788492702 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7ac015870 con 0x7fd7c41356a0 2024-09-16T03:47:03.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.028+0000 7fd7c0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:47:03.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.028+0000 7fd7c0ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:47:03.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.028+0000 7fd7c0ff9640 10 monclient: dump: 2024-09-16T03:47:03.053 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:47:03.053 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:03.053 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:03.053 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:03.053 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:47:03.053 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:03.053 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:47:03.053 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:47:03.053 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:47:03.053 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:03.053 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.028+0000 7fd7c0ff9640 10 client.13688.objecter ms_handle_connect 0x7fd7a4037810 2024-09-16T03:47:03.053 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.028+0000 7fd7c3fff640 1 --2- 172.21.15.42:0/1788492702 >> v2:172.21.15.154:6812/254845342 conn(0x7fd7a4037810 0x7fd7a4039cd0 secure :-1 s=READY pgs=3430 cs=0 l=1 rev1=1 crypto rx=0x7fd7bc01aeb0 tx=0x7fd7bc0a7070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:47:03.053 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.028+0000 7fd7c0ff9640 1 -- 172.21.15.42:0/1788492702 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fd7ac015870 con 0x7fd7c41356a0 2024-09-16T03:47:03.053 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.028+0000 7fd7c0ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:47:03.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.028+0000 7fd7c0ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:47:03.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.028+0000 7fd7c0ff9640 1 -- 172.21.15.42:0/1788492702 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd7c41356a0 msgr2=0x7fd7c413a600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:03.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.028+0000 7fd7c0ff9640 1 --2- 172.21.15.42:0/1788492702 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd7c41356a0 0x7fd7c413a600 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fd7c411eca0 tx=0x7fd7ac011a60 comp rx=0 tx=0).stop 2024-09-16T03:47:03.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.028+0000 7fd7c0ff9640 1 -- 172.21.15.42:0/1788492702 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd7c41356a0 msgr2=0x7fd7c413a600 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:47:03.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.028+0000 7fd7c0ff9640 1 --2- 172.21.15.42:0/1788492702 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd7c41356a0 0x7fd7c413a600 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:03.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.032+0000 7fd7ca18a640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:47:03.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.032+0000 7fd7ca18a640 1 --2- 172.21.15.42:0/1788492702 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd7c413ab40 0x7fd7c41362b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:03.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.032+0000 7fd7ca18a640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:03.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.032+0000 7fd7ca18a640 1 -- 172.21.15.42:0/1788492702 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd7c413af50 con 0x7fd7c413ab40 2024-09-16T03:47:03.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.032+0000 7fd7bbfff640 1 --2- 172.21.15.42:0/1788492702 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd7c413ab40 0x7fd7c41362b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:03.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.032+0000 7fd7bbfff640 10 monclient: get_auth_request con 0x7fd7c413ab40 auth_method 0 2024-09-16T03:47:03.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.032+0000 7fd7bbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:03.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.032+0000 7fd7bbfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:03.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.032+0000 7fd7bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:47:03.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.032+0000 7fd7bbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:47:03.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.032+0000 7fd7bbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:47:03.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.032+0000 7fd7bbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:47:03.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.032+0000 7fd7bbfff640 10 monclient(hunting): handle_auth_done global_id 754273 payload 824 2024-09-16T03:47:03.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.032+0000 7fd7bbfff640 1 -- 172.21.15.42:0/1788492702 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fd7c4068e40 con 0x7fd7c413ab40 2024-09-16T03:47:03.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.032+0000 7fd7bbfff640 1 --2- 172.21.15.42:0/1788492702 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd7c413ab40 0x7fd7c41362b0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fd7ac01c000 tx=0x7fd7ac004740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:47:03.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.032+0000 7fd7c0ff9640 10 client.13688.objecter ms_handle_connect 0x7fd7c413ab40 2024-09-16T03:47:03.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.032+0000 7fd7c0ff9640 10 client.13688.objecter resend_mon_ops 2024-09-16T03:47:03.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.036+0000 7fd7c0ff9640 1 -- 172.21.15.42:0/1788492702 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7ac0067d0 con 0x7fd7c413ab40 2024-09-16T03:47:03.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.036+0000 7fd7c0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:47:03.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.036+0000 7fd7c0ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:47:03.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.036+0000 7fd7c0ff9640 10 monclient: dump: 2024-09-16T03:47:03.057 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:47:03.057 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:03.057 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:03.057 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:03.058 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:47:03.058 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:03.058 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:47:03.058 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:47:03.058 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:47:03.058 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:03.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.036+0000 7fd7c0ff9640 1 -- 172.21.15.42:0/1788492702 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4711 (secure 0 0 0) 0x7fd7ac0067d0 con 0x7fd7c413ab40 2024-09-16T03:47:03.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.036+0000 7fd7c0ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:47:03.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.036+0000 7fd7c0ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:47:03.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.036+0000 7fd7c0ff9640 1 -- 172.21.15.42:0/1788492702 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd7c413ab40 msgr2=0x7fd7c41362b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:03.059 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.036+0000 7fd7c0ff9640 1 --2- 172.21.15.42:0/1788492702 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd7c413ab40 0x7fd7c41362b0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fd7ac01c000 tx=0x7fd7ac004740 comp rx=0 tx=0).stop 2024-09-16T03:47:03.059 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.036+0000 7fd7c0ff9640 1 -- 172.21.15.42:0/1788492702 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd7c413ab40 msgr2=0x7fd7c41362b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:47:03.059 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.036+0000 7fd7c0ff9640 1 --2- 172.21.15.42:0/1788492702 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd7c413ab40 0x7fd7c41362b0 secure :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fd7ac01c000 tx=0x7fd7ac004740 comp rx=0 tx=0).stop 2024-09-16T03:47:03.059 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:47:03.059 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:47:03.059 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:47:03.059 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T03:47:03.059 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 48, 2024-09-16T03:47:03.059 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 65051, 2024-09-16T03:47:03.060 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:47:03.060 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:47:03.060 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T03:47:03.060 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T03:47:03.060 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:47:03.060 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 1, 2024-09-16T03:47:03.060 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:47:03.060 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:47:03.060 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:47:03.060 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:47:03.061 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:47:03.061 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:47:03.061 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:47:03.061 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:47:03.061 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:47:03.061 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:47:03.061 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:47:03.061 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:47:03.061 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:47:03.061 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:47:03.062 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:47:03.062 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:47:03.062 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:47:03.062 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:47:03.062 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:47:03.062 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:47:03.062 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:47:03.062 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:47:03.062 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:47:03.062 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:47:03.062 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:47:03.062 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:47:03.062 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:47:03.063 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:47:03.063 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:47:03.063 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:03.063 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:47:03.063 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:47:03.063 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:47:03.063 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:47:03.063 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:47:03.063 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:47:03.063 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:47:03.063 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:47:03.063 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:47:03.063 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:47:03.064 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:47:03.064 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:47:03.064 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:47:03.064 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:47:03.064 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:47:03.064 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:47:03.064 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:47:03.064 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:47:03.064 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:47:03.064 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:47:03.064 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:47:03.064 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:47:03.064 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:47:03.065 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:47:03.065 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:47:03.065 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:47:03.065 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:47:03.065 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:47:03.065 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:47:03.065 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:47:03.065 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:03.065 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:47:03.065 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:03.065 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:47:03.065 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:47:03.065 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:47:03.065 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:47:03.066 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:03.066 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:47:03.066 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:47:03.066 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:47:03.066 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:03.066 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:03.066 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:47:03.066 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:47:03.066 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:47:03.066 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:47:03.066 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:47:03.066 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:03.067 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:47:03.067 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:47:03.067 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:47:03.067 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:47:03.067 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:47:03.067 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:03.067 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:03.067 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:47:03.068 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:47:03.068 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:47:03.068 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:47:03.068 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:03.068 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:47:03.068 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:47:03.068 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:47:03.068 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:03.068 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:03.068 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:47:03.068 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:47:03.069 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:47:03.069 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:47:03.069 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:47:03.069 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:03.069 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:47:03.069 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:47:03.070 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:47:03.070 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:47:03.070 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:47:03.070 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:03.070 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:03.070 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:47:03.070 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:47:03.070 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:47:03.070 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:47:03.070 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:03.070 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:47:03.070 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:47:03.070 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:47:03.071 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:03.071 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:03.071 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:47:03.071 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:47:03.071 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:47:03.071 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:47:03.071 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:47:03.071 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:03.071 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:47:03.071 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:47:03.072 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:47:03.072 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:47:03.072 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:47:03.072 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:47:03.072 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:47:03.072 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:03.072 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:47:03.072 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:47:03.072 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:03.073 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:47:03.073 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:47:03.073 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:47:03.073 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:47:03.073 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:47:03.073 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:47:03.073 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:03.073 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:47:03.073 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:47:03.073 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 3 2024-09-16T03:47:03.073 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:47:03.073 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:47:03.074 INFO:teuthology.orchestra.run.smithi042.stdout: "mgr": [ 2024-09-16T03:47:03.074 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:03.074 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:47:03.074 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:47:03.074 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:47:03.074 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:47:03.074 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:47:03.074 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:03.074 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:47:03.074 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:47:03.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.036+0000 7fd7ca18a640 10 librados: watch_flush enter 2024-09-16T03:47:03.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.036+0000 7fd7ca18a640 10 librados: watch_flush exit 2024-09-16T03:47:03.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.036+0000 7fd7ca18a640 20 client.13688.objecter shutdown clearing up homeless session... 2024-09-16T03:47:03.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.036+0000 7fd7ca18a640 10 client.13688.objecter successfully canceled tick 2024-09-16T03:47:03.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.036+0000 7fd7ca18a640 1 -- 172.21.15.42:0/1788492702 >> v2:172.21.15.154:6812/254845342 conn(0x7fd7a4037810 msgr2=0x7fd7a4039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:03.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.036+0000 7fd7ca18a640 1 --2- 172.21.15.42:0/1788492702 >> v2:172.21.15.154:6812/254845342 conn(0x7fd7a4037810 0x7fd7a4039cd0 secure :-1 s=READY pgs=3430 cs=0 l=1 rev1=1 crypto rx=0x7fd7bc01aeb0 tx=0x7fd7bc0a7070 comp rx=0 tx=0).stop 2024-09-16T03:47:03.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.036+0000 7fd7ca18a640 10 monclient: shutdown 2024-09-16T03:47:03.077 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.036+0000 7fd7ca18a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:47:03.077 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.036+0000 7fd7ca18a640 1 -- 172.21.15.42:0/1788492702 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd7c4072560 msgr2=0x7fd7c41cc920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:03.077 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.036+0000 7fd7ca18a640 1 --2- 172.21.15.42:0/1788492702 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd7c4072560 0x7fd7c41cc920 secure :-1 s=READY pgs=2468 cs=0 l=1 rev1=1 crypto rx=0x7fd7b4001ec0 tx=0x7fd7b400c890 comp rx=0 tx=0).stop 2024-09-16T03:47:03.077 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.036+0000 7fd7c3fff640 1 -- 172.21.15.42:0/1788492702 reap_dead start 2024-09-16T03:47:03.077 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.036+0000 7fd7c3fff640 1 -- 172.21.15.42:0/1788492702 reap_dead start 2024-09-16T03:47:03.077 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.036+0000 7fd7ca18a640 1 -- 172.21.15.42:0/1788492702 shutdown_connections 2024-09-16T03:47:03.077 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.036+0000 7fd7ca18a640 1 -- 172.21.15.42:0/1788492702 >> 172.21.15.42:0/1788492702 conn(0x7fd7c40a7ed0 msgr2=0x7fd7c40a2340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:47:03.077 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.036+0000 7fd7ca18a640 1 -- 172.21.15.42:0/1788492702 shutdown_connections 2024-09-16T03:47:03.077 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.036+0000 7fd7ca18a640 1 -- 172.21.15.42:0/1788492702 wait complete. 2024-09-16T03:47:03.077 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.036+0000 7fd7ca18a640 1 librados: shutdown 2024-09-16T03:47:03.077 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:47:03.078 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T03:47:03.078 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T03:47:03.199 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.196+0000 7fd16c9ef640 10 monclient: get_monmap_and_config 2024-09-16T03:47:03.200 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.196+0000 7fd16c9ef640 10 monclient: build_initial_monmap 2024-09-16T03:47:03.200 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.196+0000 7fd16c9ef640 10 monclient: monmap: 2024-09-16T03:47:03.200 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:47:03.200 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:03.201 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:47:03.198799+0000 2024-09-16T03:47:03.201 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:47:03.198799+0000 2024-09-16T03:47:03.201 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:47:03.201 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:03.201 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:47:03.201 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:47:03.201 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:47:03.201 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:47:03.201 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:47:03.201 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:03.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.196+0000 7fd16c9ef640 1 Processor -- start 2024-09-16T03:47:03.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.196+0000 7fd16c9ef640 1 -- start start 2024-09-16T03:47:03.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.196+0000 7fd16c9ef640 10 monclient: init 2024-09-16T03:47:03.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.200+0000 7fd16c9ef640 10 monclient: _reopen_session rank -1 2024-09-16T03:47:03.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.200+0000 7fd16c9ef640 10 monclient: _add_conns ranks=[3,1,0,4,2] 2024-09-16T03:47:03.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.200+0000 7fd16c9ef640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd16809fb00 0x7fd16809ba50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:03.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.200+0000 7fd16c9ef640 10 monclient(hunting): picked mon.noname-a con 0x7fd16809fb00 addr v2:172.21.15.42:3300/0 2024-09-16T03:47:03.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.200+0000 7fd16c9ef640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd16809bf90 addr v1:172.21.15.154:6789/0 2024-09-16T03:47:03.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.200+0000 7fd16c9ef640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd16809c400 addr v1:172.21.15.42:6789/0 2024-09-16T03:47:03.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.200+0000 7fd16c9ef640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd1680a1e20 con 0x7fd16809c400 2024-09-16T03:47:03.205 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.200+0000 7fd16c9ef640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd16809c850 con 0x7fd16809bf90 2024-09-16T03:47:03.205 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.200+0000 7fd16c9ef640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:03.205 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.200+0000 7fd16c9ef640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7fd16809e250 con 0x7fd16809fb00 2024-09-16T03:47:03.205 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.200+0000 7fd16c9ef640 10 monclient(hunting): _renew_subs 2024-09-16T03:47:03.205 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.200+0000 7fd16c9ef640 10 monclient(hunting): authenticate will time out at 1270.168325s 2024-09-16T03:47:03.205 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.200+0000 7fd16659c640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd16809fb00 0x7fd16809ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:03.205 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.200+0000 7fd166d9d640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fd16809c400 0x7fd16813b840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:55202/0 (socket says 172.21.15.42:55202) 2024-09-16T03:47:03.205 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.200+0000 7fd166d9d640 1 -- 172.21.15.42:0/1428993870 learned_addr learned my addr 172.21.15.42:0/1428993870 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:47:03.205 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.200+0000 7fd16659c640 10 monclient(hunting): get_auth_request con 0x7fd16809fb00 auth_method 0 2024-09-16T03:47:03.205 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.200+0000 7fd16659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:03.205 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.200+0000 7fd16659c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:03.205 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.200+0000 7fd16659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:47:03.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.200+0000 7fd16659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:47:03.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.200+0000 7fd16659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:47:03.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.200+0000 7fd16659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:47:03.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.200+0000 7fd16659c640 10 monclient(hunting): handle_auth_done global_id 764124 payload 274 2024-09-16T03:47:03.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.200+0000 7fd16659c640 1 -- 172.21.15.42:0/1428993870 >> v1:172.21.15.154:6789/0 conn(0x7fd16809bf90 legacy=0x7fd168138110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:03.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.200+0000 7fd16659c640 1 -- 172.21.15.42:0/1428993870 >> v1:172.21.15.42:6789/0 conn(0x7fd16809c400 legacy=0x7fd16813b840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:03.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.200+0000 7fd16659c640 10 monclient: _finish_hunting 0 2024-09-16T03:47:03.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.200+0000 7fd16659c640 1 monclient: found mon.noname-a 2024-09-16T03:47:03.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.200+0000 7fd16659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:47:03.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.200+0000 7fd16659c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:47:03.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.200+0000 7fd16659c640 1 -- 172.21.15.42:0/1428993870 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd16809ea30 con 0x7fd16809fb00 2024-09-16T03:47:03.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.200+0000 7fd16659c640 1 --2- 172.21.15.42:0/1428993870 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd16809fb00 0x7fd16809ba50 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fd15800ba50 tx=0x7fd15801e7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63c1d89a07e11916 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:47:03.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.200+0000 7fd16559a640 1 -- 172.21.15.42:0/1428993870 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1580ad070 con 0x7fd16809fb00 2024-09-16T03:47:03.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.200+0000 7fd16559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:47:03.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.200+0000 7fd16559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:47:03.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.200+0000 7fd16559a640 10 monclient: dump: 2024-09-16T03:47:03.207 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:47:03.207 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:03.207 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:03.207 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:03.207 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:47:03.207 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:03.207 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:47:03.208 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:47:03.208 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:47:03.208 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:03.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.200+0000 7fd16559a640 10 monclient: _finish_auth 0 2024-09-16T03:47:03.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.200+0000 7fd16559a640 10 monclient: _check_auth_tickets 2024-09-16T03:47:03.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.200+0000 7fd16559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:47:03.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.200+0000 7fd16559a640 1 -- 172.21.15.42:0/1428993870 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd1580a7840 con 0x7fd16809fb00 2024-09-16T03:47:03.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.200+0000 7fd16559a640 10 monclient: handle_config config(0 keys) 2024-09-16T03:47:03.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.200+0000 7fd16559a640 1 -- 172.21.15.42:0/1428993870 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1580a7c10 con 0x7fd16809fb00 2024-09-16T03:47:03.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.200+0000 7fd16c9ef640 5 monclient: authenticate success, global_id 764124 2024-09-16T03:47:03.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.200+0000 7fd16c9ef640 10 monclient: get_monmap_and_config success 2024-09-16T03:47:03.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.200+0000 7fd16c9ef640 4 set_mon_vals no callback set 2024-09-16T03:47:03.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.200+0000 7fd16559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:47:03.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.200+0000 7fd16559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:47:03.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.200+0000 7fd16559a640 10 monclient: dump: 2024-09-16T03:47:03.209 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:47:03.209 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:03.209 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:03.209 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:03.209 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:47:03.209 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:03.209 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:47:03.209 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:47:03.209 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:47:03.209 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:03.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.200+0000 7fd16c9ef640 10 monclient: shutdown 2024-09-16T03:47:03.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.200+0000 7fd16c9ef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:47:03.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.200+0000 7fd16c9ef640 1 -- 172.21.15.42:0/1428993870 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd16809fb00 msgr2=0x7fd16809ba50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:03.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.200+0000 7fd16c9ef640 1 --2- 172.21.15.42:0/1428993870 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd16809fb00 0x7fd16809ba50 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fd15800ba50 tx=0x7fd15801e7f0 comp rx=0 tx=0).stop 2024-09-16T03:47:03.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.200+0000 7fd16c9ef640 1 -- 172.21.15.42:0/1428993870 shutdown_connections 2024-09-16T03:47:03.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.200+0000 7fd16c9ef640 1 --2- 172.21.15.42:0/1428993870 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd16809fb00 0x7fd16809ba50 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:03.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.200+0000 7fd16c9ef640 1 -- 172.21.15.42:0/1428993870 >> 172.21.15.42:0/1428993870 conn(0x7fd1680a2420 msgr2=0x7fd1680a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:47:03.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.204+0000 7fd16c9ef640 1 -- 172.21.15.42:0/1428993870 shutdown_connections 2024-09-16T03:47:03.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.204+0000 7fd16c9ef640 1 -- 172.21.15.42:0/1428993870 wait complete. 2024-09-16T03:47:03.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.204+0000 7fd16c9ef640 10 monclient: build_initial_monmap 2024-09-16T03:47:03.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.204+0000 7fd16c9ef640 10 monclient: monmap: 2024-09-16T03:47:03.219 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:47:03.219 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:47:03.219 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:47:03.220 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:47:03.220 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:47:03.220 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:03.220 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:47:03.220 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:47:03.220 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:47:03.221 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:03.221 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.204+0000 7fd16c9ef640 1 librados: starting msgr at 2024-09-16T03:47:03.221 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.204+0000 7fd16c9ef640 1 librados: starting objecter 2024-09-16T03:47:03.221 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.204+0000 7fd16c9ef640 1 Processor -- start 2024-09-16T03:47:03.221 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.204+0000 7fd16c9ef640 1 -- start start 2024-09-16T03:47:03.221 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.204+0000 7fd16c9ef640 1 librados: setting wanted keys 2024-09-16T03:47:03.222 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.204+0000 7fd16c9ef640 1 librados: calling monclient init 2024-09-16T03:47:03.222 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.204+0000 7fd16c9ef640 10 monclient: init 2024-09-16T03:47:03.222 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.204+0000 7fd16c9ef640 10 monclient: _reopen_session rank -1 2024-09-16T03:47:03.222 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.204+0000 7fd16c9ef640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:47:03.222 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.204+0000 7fd16c9ef640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd16809bf90 0x7fd1681cee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:03.223 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.204+0000 7fd16c9ef640 10 monclient(hunting): picked mon.noname-c con 0x7fd16809bf90 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:47:03.223 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.204+0000 7fd16c9ef640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd16809c400 0x7fd16813c060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:03.223 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.204+0000 7fd16c9ef640 10 monclient(hunting): picked mon.noname-a con 0x7fd16809c400 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:47:03.223 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.204+0000 7fd16c9ef640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd16809fb00 0x7fd16813c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:03.223 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.204+0000 7fd16c9ef640 10 monclient(hunting): picked mon.noname-b con 0x7fd16809fb00 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:47:03.223 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.204+0000 7fd16c9ef640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:03.224 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.204+0000 7fd16c9ef640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fd1680a3170 con 0x7fd16809c400 2024-09-16T03:47:03.224 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.204+0000 7fd16c9ef640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:03.224 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.204+0000 7fd16c9ef640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd16809d0b0 con 0x7fd16809fb00 2024-09-16T03:47:03.224 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.204+0000 7fd16c9ef640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:03.224 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.204+0000 7fd16c9ef640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fd1680a32f0 con 0x7fd16809bf90 2024-09-16T03:47:03.224 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.204+0000 7fd16c9ef640 10 monclient(hunting): _renew_subs 2024-09-16T03:47:03.225 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.204+0000 7fd16c9ef640 10 monclient(hunting): authenticate will time out at 1270.173444s 2024-09-16T03:47:03.225 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.204+0000 7fd165d9b640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd16809c400 0x7fd16813c060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:03.225 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.204+0000 7fd16659c640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd16809bf90 0x7fd1681cee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:03.225 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.204+0000 7fd165d9b640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd16809c400 0x7fd16813c060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:57524/0 (socket says 172.21.15.42:57524) 2024-09-16T03:47:03.225 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.204+0000 7fd165d9b640 1 -- 172.21.15.42:0/1719000759 learned_addr learned my addr 172.21.15.42:0/1719000759 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:47:03.225 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.204+0000 7fd16659c640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd16809bf90 0x7fd1681cee10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:57426/0 (socket says 172.21.15.42:57426) 2024-09-16T03:47:03.225 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.204+0000 7fd16659c640 10 monclient(hunting): get_auth_request con 0x7fd16809bf90 auth_method 0 2024-09-16T03:47:03.225 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.204+0000 7fd16659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:03.225 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.204+0000 7fd16659c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:03.225 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.204+0000 7fd16659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:47:03.225 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.204+0000 7fd165d9b640 10 monclient(hunting): get_auth_request con 0x7fd16809c400 auth_method 0 2024-09-16T03:47:03.225 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.204+0000 7fd165d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:03.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.204+0000 7fd165d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:03.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.204+0000 7fd165d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:47:03.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.204+0000 7fd166d9d640 1 --2- 172.21.15.42:0/1719000759 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd16809fb00 0x7fd16813c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:03.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.204+0000 7fd166d9d640 10 monclient(hunting): get_auth_request con 0x7fd16809fb00 auth_method 0 2024-09-16T03:47:03.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.204+0000 7fd166d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:03.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.204+0000 7fd166d9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:03.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.204+0000 7fd166d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:47:03.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.204+0000 7fd16659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:47:03.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.204+0000 7fd16659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:47:03.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.204+0000 7fd16659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:47:03.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.204+0000 7fd165d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:47:03.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.204+0000 7fd165d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:47:03.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.204+0000 7fd165d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:47:03.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.204+0000 7fd166d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:47:03.227 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.204+0000 7fd166d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:47:03.227 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.204+0000 7fd166d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:47:03.227 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.204+0000 7fd165d9b640 10 monclient(hunting): handle_auth_done global_id 764127 payload 824 2024-09-16T03:47:03.227 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.204+0000 7fd165d9b640 1 -- 172.21.15.42:0/1719000759 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd16809bf90 msgr2=0x7fd1681cee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:03.227 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.204+0000 7fd165d9b640 1 --2- 172.21.15.42:0/1719000759 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd16809bf90 0x7fd1681cee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:03.227 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.204+0000 7fd165d9b640 1 -- 172.21.15.42:0/1719000759 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd16809fb00 msgr2=0x7fd16813c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:03.227 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.204+0000 7fd165d9b640 1 --2- 172.21.15.42:0/1719000759 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd16809fb00 0x7fd16813c5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:03.227 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.204+0000 7fd165d9b640 10 monclient: _finish_hunting 0 2024-09-16T03:47:03.227 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.204+0000 7fd165d9b640 1 monclient: found mon.noname-a 2024-09-16T03:47:03.227 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.204+0000 7fd165d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:47:03.227 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.204+0000 7fd165d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:47:03.227 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.204+0000 7fd165d9b640 1 -- 172.21.15.42:0/1719000759 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd16806deb0 con 0x7fd16809c400 2024-09-16T03:47:03.227 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.204+0000 7fd16659c640 1 --2- 172.21.15.42:0/1719000759 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd16809bf90 0x7fd1681cee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:47:03.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.204+0000 7fd165d9b640 1 --2- 172.21.15.42:0/1719000759 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd16809c400 0x7fd16813c060 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fd15c00c3c0 tx=0x7fd15c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:47:03.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.204+0000 7fd156ffd640 10 client.?.objecter ms_handle_connect 0x7fd16809c400 2024-09-16T03:47:03.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.204+0000 7fd156ffd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:47:03.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.204+0000 7fd156ffd640 1 -- 172.21.15.42:0/1719000759 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd15c017070 con 0x7fd16809c400 2024-09-16T03:47:03.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.204+0000 7fd156ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:47:03.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.204+0000 7fd156ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:47:03.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.204+0000 7fd156ffd640 10 monclient: dump: 2024-09-16T03:47:03.228 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:47:03.228 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:03.228 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:03.228 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:03.228 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:47:03.229 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:03.229 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:47:03.229 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:47:03.229 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:47:03.229 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:03.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.204+0000 7fd156ffd640 10 monclient: _finish_auth 0 2024-09-16T03:47:03.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.204+0000 7fd156ffd640 10 monclient: _check_auth_tickets 2024-09-16T03:47:03.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.204+0000 7fd156ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:47:03.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.204+0000 7fd156ffd640 1 -- 172.21.15.42:0/1719000759 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd15c0052d0 con 0x7fd16809c400 2024-09-16T03:47:03.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.204+0000 7fd156ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:47:03.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.204+0000 7fd156ffd640 1 -- 172.21.15.42:0/1719000759 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd15c003840 con 0x7fd16809c400 2024-09-16T03:47:03.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.204+0000 7fd156ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:47:03.230 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.204+0000 7fd156ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:47:03.230 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.204+0000 7fd156ffd640 10 monclient: dump: 2024-09-16T03:47:03.230 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:47:03.230 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:03.230 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:03.230 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:03.230 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:47:03.230 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:03.230 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:47:03.230 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:47:03.230 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:47:03.230 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:03.230 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.204+0000 7fd16c9ef640 5 monclient: authenticate success, global_id 764127 2024-09-16T03:47:03.230 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.204+0000 7fd16c9ef640 10 monclient: _renew_subs 2024-09-16T03:47:03.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.204+0000 7fd16c9ef640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:47:03.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.204+0000 7fd157fff640 4 set_mon_vals no callback set 2024-09-16T03:47:03.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.204+0000 7fd16c9ef640 1 -- 172.21.15.42:0/1719000759 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1681284b0 con 0x7fd16809c400 2024-09-16T03:47:03.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.204+0000 7fd16c9ef640 10 client.764127.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:47:03.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.204+0000 7fd16c9ef640 10 monclient: _renew_subs 2024-09-16T03:47:03.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.204+0000 7fd16c9ef640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:47:03.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.204+0000 7fd16c9ef640 1 -- 172.21.15.42:0/1719000759 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd168073a30 con 0x7fd16809c400 2024-09-16T03:47:03.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.204+0000 7fd16c9ef640 1 librados: init done 2024-09-16T03:47:03.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.204+0000 7fd156ffd640 1 -- 172.21.15.42:0/1719000759 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fd15c01a070 con 0x7fd16809c400 2024-09-16T03:47:03.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.204+0000 7fd16c9ef640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:47:03.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.204+0000 7fd16c9ef640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:47:03.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.204+0000 7fd16c9ef640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:47:03.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.204+0000 7fd16c9ef640 1 -- 172.21.15.42:0/1719000759 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd16809ba50 con 0x7fd16809c400 2024-09-16T03:47:03.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.204+0000 7fd156ffd640 1 --2- 172.21.15.42:0/1719000759 >> v2:172.21.15.154:6812/254845342 conn(0x7fd148037810 0x7fd148039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:03.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.204+0000 7fd156ffd640 1 -- 172.21.15.42:0/1719000759 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(147..147 src has 1..147) ==== 6868+0+0 (secure 0 0 0) 0x7fd15c048770 con 0x7fd16809c400 2024-09-16T03:47:03.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.204+0000 7fd156ffd640 10 client.764127.objecter ms_dispatch 0x7fd168000ba0 osd_map(147..147 src has 1..147) 2024-09-16T03:47:03.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.204+0000 7fd156ffd640 3 client.764127.objecter handle_osd_map got epochs [147,147] > 0 2024-09-16T03:47:03.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.204+0000 7fd156ffd640 3 client.764127.objecter handle_osd_map decoding full epoch 147 2024-09-16T03:47:03.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.204+0000 7fd156ffd640 20 client.764127.objecter dump_active .. 0 homeless 2024-09-16T03:47:03.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.208+0000 7fd16659c640 1 --2- 172.21.15.42:0/1719000759 >> v2:172.21.15.154:6812/254845342 conn(0x7fd148037810 0x7fd148039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:03.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.208+0000 7fd16659c640 10 monclient: get_auth_request con 0x7fd148037810 auth_method 0 2024-09-16T03:47:03.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.208+0000 7fd16659c640 1 --2- 172.21.15.42:0/1719000759 >> v2:172.21.15.154:6812/254845342 conn(0x7fd148037810 0x7fd148039cd0 secure :-1 s=READY pgs=3431 cs=0 l=1 rev1=1 crypto rx=0x7fd1580b2000 tx=0x7fd1580a9070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:47:03.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.208+0000 7fd156ffd640 10 client.764127.objecter ms_handle_connect 0x7fd148037810 2024-09-16T03:47:03.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.212+0000 7fd156ffd640 1 -- 172.21.15.42:0/1719000759 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7fd15c005480 con 0x7fd16809c400 2024-09-16T03:47:03.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.212+0000 7fd156ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:47:03.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.212+0000 7fd156ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:47:03.440 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.436+0000 7fd16c9ef640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T03:47:03.440 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.436+0000 7fd16c9ef640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T03:47:03.440 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.436+0000 7fd16c9ef640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:47:03.440 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.436+0000 7fd16c9ef640 1 -- 172.21.15.42:0/1719000759 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7fd168069040 con 0x7fd16809c400 2024-09-16T03:47:03.441 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.440+0000 7fd156ffd640 1 -- 172.21.15.42:0/1719000759 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7fd15c0039e0 con 0x7fd16809c400 2024-09-16T03:47:03.441 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.440+0000 7fd156ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T03:47:03.442 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.440+0000 7fd156ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:47:03.443 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.440+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:47:03.443 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.440+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=27,config=1,health=12,kv=10,logm=50} crc {auth=300305345,config=3678701417,health=4280765818,kv=3446188498,logm=749571742}) 2024-09-16T03:47:03.443 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.440+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=26,config=2,health=11,kv=10,logm=51} crc {auth=3087958604,config=692619927,health=394770834,kv=3446188498,logm=1003076183}) 2024-09-16T03:47:03.444 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.440+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:47:03.444 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.440+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=27,config=1,health=12,kv=10,logm=50} crc {auth=300305345,config=3678701417,health=4280765818,kv=3446188498,logm=749571742}) 2024-09-16T03:47:03.444 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.440+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=29,config=2,health=11,kv=9,logm=49} crc {auth=4256234101,config=692619927,health=1881049622,kv=542242125,logm=203451558}) 2024-09-16T03:47:03.445 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.440+0000 7fd16c9ef640 10 librados: watch_flush enter 2024-09-16T03:47:03.445 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.440+0000 7fd16c9ef640 10 librados: watch_flush exit 2024-09-16T03:47:03.445 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.440+0000 7fd16c9ef640 20 client.764127.objecter shutdown clearing up homeless session... 2024-09-16T03:47:03.445 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.440+0000 7fd16c9ef640 10 client.764127.objecter successfully canceled tick 2024-09-16T03:47:03.445 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.440+0000 7fd16c9ef640 1 -- 172.21.15.42:0/1719000759 >> v2:172.21.15.154:6812/254845342 conn(0x7fd148037810 msgr2=0x7fd148039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:03.445 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.440+0000 7fd16c9ef640 1 --2- 172.21.15.42:0/1719000759 >> v2:172.21.15.154:6812/254845342 conn(0x7fd148037810 0x7fd148039cd0 secure :-1 s=READY pgs=3431 cs=0 l=1 rev1=1 crypto rx=0x7fd1580b2000 tx=0x7fd1580a9070 comp rx=0 tx=0).stop 2024-09-16T03:47:03.445 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.440+0000 7fd16c9ef640 10 monclient: shutdown 2024-09-16T03:47:03.446 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.440+0000 7fd16c9ef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:47:03.446 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.440+0000 7fd16c9ef640 1 -- 172.21.15.42:0/1719000759 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd16809c400 msgr2=0x7fd16813c060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:03.446 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.440+0000 7fd16c9ef640 1 --2- 172.21.15.42:0/1719000759 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd16809c400 0x7fd16813c060 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fd15c00c3c0 tx=0x7fd15c00c880 comp rx=0 tx=0).stop 2024-09-16T03:47:03.446 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.440+0000 7fd16c9ef640 1 -- 172.21.15.42:0/1719000759 shutdown_connections 2024-09-16T03:47:03.446 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.440+0000 7fd16c9ef640 1 --2- 172.21.15.42:0/1719000759 >> v2:172.21.15.154:6812/254845342 conn(0x7fd148037810 0x7fd148039cd0 unknown :-1 s=CLOSED pgs=3431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:03.446 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.440+0000 7fd16c9ef640 1 --2- 172.21.15.42:0/1719000759 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd16809fb00 0x7fd16813c5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:03.446 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.440+0000 7fd16c9ef640 1 --2- 172.21.15.42:0/1719000759 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd16809c400 0x7fd16813c060 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:03.446 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.440+0000 7fd16c9ef640 1 --2- 172.21.15.42:0/1719000759 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd16809bf90 0x7fd1681cee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:03.446 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.440+0000 7fd16c9ef640 1 -- 172.21.15.42:0/1719000759 >> 172.21.15.42:0/1719000759 conn(0x7fd1680a2420 msgr2=0x7fd1680a46b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:47:03.446 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.440+0000 7fd16c9ef640 1 -- 172.21.15.42:0/1719000759 shutdown_connections 2024-09-16T03:47:03.446 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.440+0000 7fd16c9ef640 1 -- 172.21.15.42:0/1719000759 wait complete. 2024-09-16T03:47:03.446 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:03.440+0000 7fd16c9ef640 1 librados: shutdown 2024-09-16T03:47:03.447 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.444+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:47:03.447 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.444+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1390240265}) 2024-09-16T03:47:03.447 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.444+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3878213244}) 2024-09-16T03:47:03.447 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.444+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:47:03.447 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.444+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1390240265}) 2024-09-16T03:47:03.447 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.444+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1023471495}) 2024-09-16T03:47:03.448 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.444+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:47:03.448 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.444+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1270869965}) 2024-09-16T03:47:03.448 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.444+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3527189028}) 2024-09-16T03:47:03.448 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.444+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:47:03.448 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.444+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1270869965}) 2024-09-16T03:47:03.448 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.444+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1240296887}) 2024-09-16T03:47:03.450 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.448+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:47:03.450 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.448+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3848881145}) 2024-09-16T03:47:03.450 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.448+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2977100924}) 2024-09-16T03:47:03.450 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.448+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:47:03.450 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.448+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3848881145}) 2024-09-16T03:47:03.450 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.448+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3397716260}) 2024-09-16T03:47:03.451 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.448+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:47:03.451 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.448+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=701248907}) 2024-09-16T03:47:03.451 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.448+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1683452446}) 2024-09-16T03:47:03.451 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.448+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:47:03.451 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.448+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=701248907}) 2024-09-16T03:47:03.451 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.448+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4218906881}) 2024-09-16T03:47:03.453 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.452+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:47:03.453 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.452+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3538273435}) 2024-09-16T03:47:03.453 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.452+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3491742194}) 2024-09-16T03:47:03.453 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.452+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:47:03.453 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.452+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3538273435}) 2024-09-16T03:47:03.453 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.452+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4102913019}) 2024-09-16T03:47:03.456 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.452+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:47:03.456 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.452+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=546602548}) 2024-09-16T03:47:03.456 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.452+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4077264209}) 2024-09-16T03:47:03.456 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.452+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:47:03.456 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.452+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=546602548}) 2024-09-16T03:47:03.457 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.452+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1212105829}) 2024-09-16T03:47:03.457 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.452+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:47:03.457 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.452+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3086227830}) 2024-09-16T03:47:03.457 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.452+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2203285884}) 2024-09-16T03:47:03.457 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.452+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:47:03.457 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.452+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3086227830}) 2024-09-16T03:47:03.457 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.452+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3084873304}) 2024-09-16T03:47:03.458 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.456+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:47:03.458 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.456+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1859094715}) 2024-09-16T03:47:03.458 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.456+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4211013955}) 2024-09-16T03:47:03.458 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.456+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:47:03.458 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.456+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1859094715}) 2024-09-16T03:47:03.458 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.456+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2380896118}) 2024-09-16T03:47:03.460 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.456+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:47:03.460 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.456+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1891074497}) 2024-09-16T03:47:03.460 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.456+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1346441477}) 2024-09-16T03:47:03.460 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.456+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:47:03.460 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.456+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1891074497}) 2024-09-16T03:47:03.460 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.456+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=413777956}) 2024-09-16T03:47:03.462 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T03:47:03.462 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.460+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:47:03.462 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.460+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=64515744}) 2024-09-16T03:47:03.462 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.460+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1810558675}) 2024-09-16T03:47:03.463 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.460+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:47:03.463 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.460+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=64515744}) 2024-09-16T03:47:03.463 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.460+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2520414649}) 2024-09-16T03:47:03.463 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.460+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:47:03.463 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.460+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2485798107}) 2024-09-16T03:47:03.464 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.460+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1596104477}) 2024-09-16T03:47:03.464 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.460+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:47:03.464 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.460+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2485798107}) 2024-09-16T03:47:03.464 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.460+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3959787262}) 2024-09-16T03:47:03.465 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.464+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:47:03.466 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.464+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3763116981}) 2024-09-16T03:47:03.466 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.464+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2342347254}) 2024-09-16T03:47:03.466 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.464+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:47:03.466 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.464+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3763116981}) 2024-09-16T03:47:03.466 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.464+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=398802990}) 2024-09-16T03:47:03.467 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.464+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:47:03.467 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.464+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2688002020}) 2024-09-16T03:47:03.467 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.464+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1430724577}) 2024-09-16T03:47:03.467 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.464+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:47:03.467 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.464+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2688002020}) 2024-09-16T03:47:03.467 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.464+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3410180112}) 2024-09-16T03:47:03.469 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.468+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:47:03.469 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.468+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3982119480}) 2024-09-16T03:47:03.469 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.468+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2185400197}) 2024-09-16T03:47:03.469 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.468+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:47:03.469 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.468+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3982119480}) 2024-09-16T03:47:03.469 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.468+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=371821177}) 2024-09-16T03:47:03.471 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.468+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:47:03.471 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.468+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2682996504}) 2024-09-16T03:47:03.471 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.468+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2027105341}) 2024-09-16T03:47:03.471 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.468+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:47:03.471 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.468+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2682996504}) 2024-09-16T03:47:03.471 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.468+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3958764914}) 2024-09-16T03:47:03.473 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.468+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:47:03.473 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.472+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1193675779}) 2024-09-16T03:47:03.473 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.472+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3997104801}) 2024-09-16T03:47:03.473 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.472+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:47:03.473 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.472+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1193675779}) 2024-09-16T03:47:03.473 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.472+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=700096260}) 2024-09-16T03:47:03.474 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.472+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:47:03.474 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.472+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1215343182}) 2024-09-16T03:47:03.475 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.472+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2095333434}) 2024-09-16T03:47:03.475 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.472+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:47:03.475 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.472+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1215343182}) 2024-09-16T03:47:03.475 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.472+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2476486498}) 2024-09-16T03:47:03.476 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.472+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:47:03.476 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.472+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=355830912}) 2024-09-16T03:47:03.476 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.472+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=38140010}) 2024-09-16T03:47:03.476 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.472+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:47:03.477 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.472+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=355830912}) 2024-09-16T03:47:03.477 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.472+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=393979231}) 2024-09-16T03:47:03.478 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.476+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:47:03.478 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.476+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2347708188}) 2024-09-16T03:47:03.478 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.476+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2963433624}) 2024-09-16T03:47:03.478 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.476+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:47:03.478 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.476+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2347708188}) 2024-09-16T03:47:03.478 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.476+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=566927425}) 2024-09-16T03:47:03.480 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.476+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:47:03.480 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.476+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2244802115}) 2024-09-16T03:47:03.480 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.476+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2963880004}) 2024-09-16T03:47:03.480 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.476+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:47:03.480 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.476+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2244802115}) 2024-09-16T03:47:03.480 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.476+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1919827182}) 2024-09-16T03:47:03.482 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.480+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:47:03.482 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.480+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1227653587}) 2024-09-16T03:47:03.482 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.480+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3697321154}) 2024-09-16T03:47:03.482 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.480+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:47:03.483 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.480+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1227653587}) 2024-09-16T03:47:03.483 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.480+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3357154716}) 2024-09-16T03:47:03.484 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.480+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:47:03.484 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.480+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=468092377}) 2024-09-16T03:47:03.484 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.480+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=350796744}) 2024-09-16T03:47:03.484 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.480+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:47:03.484 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.480+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=468092377}) 2024-09-16T03:47:03.484 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.480+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1937967057}) 2024-09-16T03:47:03.486 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.484+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:47:03.486 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.484+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1979076787}) 2024-09-16T03:47:03.486 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.484+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3912635870}) 2024-09-16T03:47:03.486 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.484+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:47:03.486 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.484+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1979076787}) 2024-09-16T03:47:03.486 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.484+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4089272104}) 2024-09-16T03:47:03.487 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.484+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:47:03.487 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.484+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3000900639}) 2024-09-16T03:47:03.487 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.484+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=355067725}) 2024-09-16T03:47:03.487 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.484+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:47:03.488 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.484+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3000900639}) 2024-09-16T03:47:03.488 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.484+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=681961645}) 2024-09-16T03:47:03.490 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.488+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:47:03.490 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.488+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=45,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=10} crc {logm=1176426532,mdsmap=4088524775,mgr=1406439472,mgr_command_descs=2107673600,mgr_metadata=1367272202,mgrstat=3893678359,mon_config_key=85057112,monmap=3937513981,osd_metadata=939947925,osd_pg_creating=2007921489,osd_snap=2330958327}) 2024-09-16T03:47:03.490 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.488+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=44,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=10,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=14} crc {logm=2397303515,mdsmap=4088524775,mgr=911650295,mgr_command_descs=2107673600,mgr_metadata=1367272202,mgrstat=118055308,mon_config_key=1439144890,monmap=3937513981,osd_metadata=4182959540,osd_pg_creating=2007921489,osd_snap=1312259291}) 2024-09-16T03:47:03.490 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.488+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:47:03.490 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.488+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=45,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=10} crc {logm=1176426532,mdsmap=4088524775,mgr=1406439472,mgr_command_descs=2107673600,mgr_metadata=1367272202,mgrstat=3893678359,mon_config_key=85057112,monmap=3937513981,osd_metadata=939947925,osd_pg_creating=2007921489,osd_snap=2330958327}) 2024-09-16T03:47:03.490 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.488+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=46,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=9} crc {logm=2792140680,mdsmap=4088524775,mgr=460109776,mgr_command_descs=2107673600,mgr_metadata=1367272202,mgrstat=3412713481,mon_config_key=1019374798,monmap=3937513981,osd_metadata=4182959540,osd_pg_creating=2007921489,osd_snap=1205763489}) 2024-09-16T03:47:03.493 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.492+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:47:03.493 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.492+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=14,osdmap=86} crc {osd_snap=4135865792,osdmap=2113492416}) 2024-09-16T03:47:03.493 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.492+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=12,osdmap=88} crc {osd_snap=601032553,osdmap=2580096927}) 2024-09-16T03:47:03.493 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.492+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:47:03.493 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.492+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=14,osdmap=86} crc {osd_snap=4135865792,osdmap=2113492416}) 2024-09-16T03:47:03.493 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.492+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=14,osdmap=86} crc {osd_snap=4135865792,osdmap=3452952330}) 2024-09-16T03:47:03.496 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.492+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:47:03.496 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.492+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2505078201}) 2024-09-16T03:47:03.497 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.492+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1193516796}) 2024-09-16T03:47:03.497 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.492+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:47:03.497 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.492+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2505078201}) 2024-09-16T03:47:03.497 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.492+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1759949911}) 2024-09-16T03:47:03.512 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.508+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:47:03.512 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.508+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=45} crc {osdmap=3064046012}) 2024-09-16T03:47:03.512 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.508+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=44} crc {osdmap=1876658253}) 2024-09-16T03:47:03.512 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.508+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:47:03.512 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.508+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=45} crc {osdmap=3064046012}) 2024-09-16T03:47:03.512 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:03.508+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=45} crc {osdmap=3841097499}) 2024-09-16T03:47:04.462 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T03:47:04.463 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:47:04.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.592+0000 7f3cf5af5640 10 monclient: get_monmap_and_config 2024-09-16T03:47:04.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.592+0000 7f3cf5af5640 10 monclient: build_initial_monmap 2024-09-16T03:47:04.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.592+0000 7f3cf5af5640 10 monclient: monmap: 2024-09-16T03:47:04.594 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:47:04.594 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:04.595 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:47:04.593455+0000 2024-09-16T03:47:04.595 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:47:04.593455+0000 2024-09-16T03:47:04.595 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:47:04.595 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:04.595 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:47:04.595 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:47:04.595 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:47:04.595 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:47:04.595 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:47:04.595 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:04.597 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.592+0000 7f3cf5af5640 1 Processor -- start 2024-09-16T03:47:04.598 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.592+0000 7f3cf5af5640 1 -- start start 2024-09-16T03:47:04.598 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.592+0000 7f3cf5af5640 10 monclient: init 2024-09-16T03:47:04.598 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.592+0000 7f3cf5af5640 10 monclient: _reopen_session rank -1 2024-09-16T03:47:04.598 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.592+0000 7f3cf5af5640 10 monclient: _add_conns ranks=[1,4,2,0,3] 2024-09-16T03:47:04.598 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.592+0000 7f3cf5af5640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3cf00a1330 addr v1:172.21.15.154:6789/0 2024-09-16T03:47:04.598 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.592+0000 7f3cf5af5640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f3cf006dac0 0x7f3cf013a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:04.598 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.592+0000 7f3cf5af5640 10 monclient(hunting): picked mon.noname-c con 0x7f3cf006dac0 addr v2:172.21.15.154:3300/0 2024-09-16T03:47:04.598 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.592+0000 7f3cf5af5640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3cf00a1e20 0x7f3cf013c890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:04.598 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.592+0000 7f3cf5af5640 10 monclient(hunting): picked mon.noname-b con 0x7f3cf00a1e20 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:47:04.598 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.592+0000 7f3cf5af5640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3cf009c090 con 0x7f3cf00a1330 2024-09-16T03:47:04.598 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.592+0000 7f3cf5af5640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:04.598 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.592+0000 7f3cf5af5640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f3cf009c890 con 0x7f3cf006dac0 2024-09-16T03:47:04.598 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.592+0000 7f3cf5af5640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:04.599 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.592+0000 7f3cf5af5640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f3cf009f610 con 0x7f3cf00a1e20 2024-09-16T03:47:04.599 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.592+0000 7f3cf5af5640 10 monclient(hunting): _renew_subs 2024-09-16T03:47:04.599 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.592+0000 7f3cf5af5640 10 monclient(hunting): authenticate will time out at 1271.562913s 2024-09-16T03:47:04.599 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.592+0000 7f3ceffff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3cf00a1e20 0x7f3cf013c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:04.599 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.592+0000 7f3ceffff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3cf00a1e20 0x7f3cf013c890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:51554/0 (socket says 172.21.15.42:51554) 2024-09-16T03:47:04.599 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.592+0000 7f3ceffff640 1 -- 172.21.15.42:0/1399704013 learned_addr learned my addr 172.21.15.42:0/1399704013 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:47:04.599 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.592+0000 7f3ceffff640 10 monclient(hunting): get_auth_request con 0x7f3cf00a1e20 auth_method 0 2024-09-16T03:47:04.599 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.592+0000 7f3ceffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:04.599 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.592+0000 7f3ceffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:04.599 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.592+0000 7f3ceffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:47:04.599 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.592+0000 7f3ceffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:47:04.599 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.592+0000 7f3ceffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:47:04.599 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.592+0000 7f3ceffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:47:04.600 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.592+0000 7f3ceeffd640 1 --2- 172.21.15.42:0/1399704013 >> v2:172.21.15.154:3300/0 conn(0x7f3cf006dac0 0x7f3cf013a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:04.600 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.592+0000 7f3ceeffd640 10 monclient(hunting): get_auth_request con 0x7f3cf006dac0 auth_method 0 2024-09-16T03:47:04.600 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.592+0000 7f3ceeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:04.600 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.592+0000 7f3ceeffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:04.600 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.592+0000 7f3ceeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:47:04.600 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.592+0000 7f3ceffff640 10 monclient(hunting): handle_auth_done global_id 13697 payload 274 2024-09-16T03:47:04.600 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.592+0000 7f3ceffff640 1 -- 172.21.15.42:0/1399704013 >> v2:172.21.15.154:3300/0 conn(0x7f3cf006dac0 msgr2=0x7f3cf013a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:47:04.600 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.592+0000 7f3ceffff640 1 --2- 172.21.15.42:0/1399704013 >> v2:172.21.15.154:3300/0 conn(0x7f3cf006dac0 0x7f3cf013a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:04.600 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.592+0000 7f3ceffff640 1 -- 172.21.15.42:0/1399704013 >> v1:172.21.15.154:6789/0 conn(0x7f3cf00a1330 legacy=0x7f3cf00a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:04.600 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.592+0000 7f3ceffff640 10 monclient: _finish_hunting 0 2024-09-16T03:47:04.600 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.592+0000 7f3ceffff640 1 monclient: found mon.noname-b 2024-09-16T03:47:04.600 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.592+0000 7f3ceffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:47:04.600 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.592+0000 7f3ceffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T03:47:04.601 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.592+0000 7f3ceffff640 1 -- 172.21.15.42:0/1399704013 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3cf009f8f0 con 0x7f3cf00a1e20 2024-09-16T03:47:04.601 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.592+0000 7f3ceeffd640 1 --2- 172.21.15.42:0/1399704013 >> v2:172.21.15.154:3300/0 conn(0x7f3cf006dac0 0x7f3cf013a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T03:47:04.606 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.596+0000 7f3ceffff640 1 --2- 172.21.15.42:0/1399704013 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3cf00a1e20 0x7f3cf013c890 secure :-1 s=READY pgs=2469 cs=0 l=1 rev1=1 crypto rx=0x7f3ce8009e50 tx=0x7f3ce801c950 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8664dc7a75fcc04f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:47:04.606 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.596+0000 7f3cee7fc640 1 -- 172.21.15.42:0/1399704013 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ce80a5830 con 0x7f3cf00a1e20 2024-09-16T03:47:04.606 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.596+0000 7f3cee7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:47:04.606 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.596+0000 7f3cee7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:47:04.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.596+0000 7f3cee7fc640 10 monclient: dump: 2024-09-16T03:47:04.607 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:47:04.607 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:04.607 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:04.607 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:04.607 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:47:04.607 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:04.607 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:47:04.607 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:47:04.607 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:47:04.607 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:04.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.596+0000 7f3cee7fc640 10 monclient: _finish_auth 0 2024-09-16T03:47:04.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.596+0000 7f3cee7fc640 10 monclient: _check_auth_tickets 2024-09-16T03:47:04.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.596+0000 7f3cee7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:47:04.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.596+0000 7f3cee7fc640 1 -- 172.21.15.42:0/1399704013 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3ce80a59d0 con 0x7f3cf00a1e20 2024-09-16T03:47:04.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.596+0000 7f3cee7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:47:04.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.596+0000 7f3cf5af5640 5 monclient: authenticate success, global_id 13697 2024-09-16T03:47:04.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.596+0000 7f3cee7fc640 1 -- 172.21.15.42:0/1399704013 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ce80a5da0 con 0x7f3cf00a1e20 2024-09-16T03:47:04.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.596+0000 7f3cee7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:47:04.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.596+0000 7f3cee7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:47:04.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.596+0000 7f3cee7fc640 10 monclient: dump: 2024-09-16T03:47:04.608 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:47:04.608 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:04.608 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:04.608 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:04.608 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:47:04.608 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:04.609 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:47:04.609 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:47:04.609 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:47:04.609 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:04.609 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.596+0000 7f3cf5af5640 10 monclient: get_monmap_and_config success 2024-09-16T03:47:04.609 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.596+0000 7f3cf5af5640 4 set_mon_vals no callback set 2024-09-16T03:47:04.609 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.596+0000 7f3cf5af5640 10 monclient: shutdown 2024-09-16T03:47:04.609 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.596+0000 7f3cf5af5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:47:04.609 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.596+0000 7f3cf5af5640 1 -- 172.21.15.42:0/1399704013 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3cf00a1e20 msgr2=0x7f3cf013c890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:04.609 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.596+0000 7f3cf5af5640 1 --2- 172.21.15.42:0/1399704013 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3cf00a1e20 0x7f3cf013c890 secure :-1 s=READY pgs=2469 cs=0 l=1 rev1=1 crypto rx=0x7f3ce8009e50 tx=0x7f3ce801c950 comp rx=0 tx=0).stop 2024-09-16T03:47:04.609 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.596+0000 7f3cf5af5640 1 -- 172.21.15.42:0/1399704013 shutdown_connections 2024-09-16T03:47:04.609 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.596+0000 7f3cf5af5640 1 --2- 172.21.15.42:0/1399704013 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3cf00a1e20 0x7f3cf013c890 unknown :-1 s=CLOSED pgs=2469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:04.609 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.596+0000 7f3cf5af5640 1 --2- 172.21.15.42:0/1399704013 >> v2:172.21.15.154:3300/0 conn(0x7f3cf006dac0 0x7f3cf013a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:04.609 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.596+0000 7f3cf5af5640 1 -- 172.21.15.42:0/1399704013 >> 172.21.15.42:0/1399704013 conn(0x7f3cf00a2420 msgr2=0x7f3cf00a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:47:04.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.596+0000 7f3cf5af5640 1 -- 172.21.15.42:0/1399704013 shutdown_connections 2024-09-16T03:47:04.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.596+0000 7f3cf5af5640 1 -- 172.21.15.42:0/1399704013 wait complete. 2024-09-16T03:47:04.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.596+0000 7f3cf5af5640 10 monclient: build_initial_monmap 2024-09-16T03:47:04.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.596+0000 7f3cf5af5640 10 monclient: monmap: 2024-09-16T03:47:04.610 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:47:04.610 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:47:04.610 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:47:04.610 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:47:04.610 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:47:04.610 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:04.610 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:47:04.610 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:47:04.610 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:47:04.611 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:04.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.596+0000 7f3cf5af5640 1 librados: starting msgr at 2024-09-16T03:47:04.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.596+0000 7f3cf5af5640 1 librados: starting objecter 2024-09-16T03:47:04.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.596+0000 7f3cf5af5640 1 Processor -- start 2024-09-16T03:47:04.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.596+0000 7f3cf5af5640 1 -- start start 2024-09-16T03:47:04.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.596+0000 7f3cf5af5640 1 librados: setting wanted keys 2024-09-16T03:47:04.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.596+0000 7f3cf5af5640 1 librados: calling monclient init 2024-09-16T03:47:04.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.596+0000 7f3cf5af5640 10 monclient: init 2024-09-16T03:47:04.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.600+0000 7f3cf5af5640 10 monclient: _reopen_session rank -1 2024-09-16T03:47:04.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.600+0000 7f3cf5af5640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T03:47:04.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.600+0000 7f3cf5af5640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3cf00a1330 0x7f3cf0071050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:04.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.600+0000 7f3cf5af5640 10 monclient(hunting): picked mon.noname-b con 0x7f3cf00a1330 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:47:04.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.600+0000 7f3cf5af5640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3cf00a1e20 0x7f3cf0132460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:04.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.600+0000 7f3cf5af5640 10 monclient(hunting): picked mon.noname-c con 0x7f3cf00a1e20 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:47:04.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.600+0000 7f3cf5af5640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3cf01329a0 0x7f3cf011bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:04.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.600+0000 7f3cf5af5640 10 monclient(hunting): picked mon.noname-a con 0x7f3cf01329a0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:47:04.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.600+0000 7f3cf5af5640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:04.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.600+0000 7f3cf5af5640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f3cf013f5f0 con 0x7f3cf01329a0 2024-09-16T03:47:04.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.600+0000 7f3cf5af5640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:04.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.600+0000 7f3cf5af5640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f3cf009efb0 con 0x7f3cf00a1330 2024-09-16T03:47:04.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.600+0000 7f3cf5af5640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:04.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.600+0000 7f3cf5af5640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f3cf013f770 con 0x7f3cf00a1e20 2024-09-16T03:47:04.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.600+0000 7f3cf5af5640 10 monclient(hunting): _renew_subs 2024-09-16T03:47:04.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.600+0000 7f3cf5af5640 10 monclient(hunting): authenticate will time out at 1271.568401s 2024-09-16T03:47:04.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.600+0000 7f3ceeffd640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3cf00a1e20 0x7f3cf0132460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:04.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.600+0000 7f3ceeffd640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3cf00a1e20 0x7f3cf0132460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:51560/0 (socket says 172.21.15.42:51560) 2024-09-16T03:47:04.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.600+0000 7f3ceeffd640 1 -- 172.21.15.42:0/2452579811 learned_addr learned my addr 172.21.15.42:0/2452579811 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:47:04.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.600+0000 7f3ceeffd640 10 monclient(hunting): get_auth_request con 0x7f3cf00a1e20 auth_method 0 2024-09-16T03:47:04.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.600+0000 7f3ceeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:04.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.600+0000 7f3ceeffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:04.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.600+0000 7f3ceeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:47:04.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.600+0000 7f3ceffff640 1 --2- 172.21.15.42:0/2452579811 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3cf01329a0 0x7f3cf011bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:04.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.600+0000 7f3ceffff640 10 monclient(hunting): get_auth_request con 0x7f3cf01329a0 auth_method 0 2024-09-16T03:47:04.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.600+0000 7f3ceffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:04.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.600+0000 7f3ceffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:04.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.600+0000 7f3ceffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:47:04.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.600+0000 7f3ceeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:47:04.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.600+0000 7f3ceeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:47:04.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.600+0000 7f3ceeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:47:04.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.600+0000 7f3ceffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:47:04.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.600+0000 7f3ceffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:47:04.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.600+0000 7f3cef7fe640 1 --2- 172.21.15.42:0/2452579811 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3cf00a1330 0x7f3cf0071050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:04.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.600+0000 7f3ceffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:47:04.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.600+0000 7f3cef7fe640 10 monclient(hunting): get_auth_request con 0x7f3cf00a1330 auth_method 0 2024-09-16T03:47:04.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.600+0000 7f3cef7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:04.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.600+0000 7f3cef7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:04.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.600+0000 7f3cef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:47:04.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.600+0000 7f3ceeffd640 10 monclient(hunting): handle_auth_done global_id 13700 payload 824 2024-09-16T03:47:04.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.600+0000 7f3ceeffd640 1 -- 172.21.15.42:0/2452579811 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3cf00a1330 msgr2=0x7f3cf0071050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:04.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.600+0000 7f3ceeffd640 1 --2- 172.21.15.42:0/2452579811 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3cf00a1330 0x7f3cf0071050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:04.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.600+0000 7f3ceeffd640 1 -- 172.21.15.42:0/2452579811 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3cf01329a0 msgr2=0x7f3cf011bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:04.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.600+0000 7f3ceeffd640 1 --2- 172.21.15.42:0/2452579811 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3cf01329a0 0x7f3cf011bf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:04.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.600+0000 7f3ceeffd640 10 monclient: _finish_hunting 0 2024-09-16T03:47:04.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.600+0000 7f3ceeffd640 1 monclient: found mon.noname-c 2024-09-16T03:47:04.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.600+0000 7f3ceeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:47:04.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.600+0000 7f3ceeffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:47:04.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.600+0000 7f3ceeffd640 1 -- 172.21.15.42:0/2452579811 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3cf01cf130 con 0x7f3cf00a1e20 2024-09-16T03:47:04.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.600+0000 7f3ceeffd640 1 --2- 172.21.15.42:0/2452579811 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3cf00a1e20 0x7f3cf0132460 secure :-1 s=READY pgs=2470 cs=0 l=1 rev1=1 crypto rx=0x7f3cd800e840 tx=0x7f3cd800ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:47:04.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.600+0000 7f3ce7fff640 10 client.?.objecter ms_handle_connect 0x7f3cf00a1e20 2024-09-16T03:47:04.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.600+0000 7f3ce7fff640 10 client.?.objecter resend_mon_ops 2024-09-16T03:47:04.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.600+0000 7f3ce7fff640 1 -- 172.21.15.42:0/2452579811 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cd8017070 con 0x7f3cf00a1e20 2024-09-16T03:47:04.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.600+0000 7f3ce7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:47:04.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.600+0000 7f3ce7fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:47:04.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.600+0000 7f3ce7fff640 10 monclient: dump: 2024-09-16T03:47:04.615 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:47:04.616 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:04.616 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:04.616 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:04.616 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:47:04.616 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:04.616 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:47:04.616 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:47:04.616 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:47:04.616 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:04.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.600+0000 7f3ce7fff640 10 monclient: _finish_auth 0 2024-09-16T03:47:04.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.600+0000 7f3ce7fff640 10 monclient: _check_auth_tickets 2024-09-16T03:47:04.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.600+0000 7f3ce7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:47:04.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.600+0000 7f3cf5af5640 5 monclient: authenticate success, global_id 13700 2024-09-16T03:47:04.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.600+0000 7f3ce7fff640 1 -- 172.21.15.42:0/2452579811 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3cd800c8d0 con 0x7f3cf00a1e20 2024-09-16T03:47:04.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.600+0000 7f3cf5af5640 10 monclient: _renew_subs 2024-09-16T03:47:04.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.600+0000 7f3cf5af5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:47:04.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.600+0000 7f3cf5af5640 1 -- 172.21.15.42:0/2452579811 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3cf011eb30 con 0x7f3cf00a1e20 2024-09-16T03:47:04.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.600+0000 7f3ce7fff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:47:04.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.600+0000 7f3ce7fff640 1 -- 172.21.15.42:0/2452579811 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cd800f040 con 0x7f3cf00a1e20 2024-09-16T03:47:04.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.600+0000 7f3ce7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:47:04.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.600+0000 7f3ce7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:47:04.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.600+0000 7f3ce7fff640 10 monclient: dump: 2024-09-16T03:47:04.617 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:47:04.617 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:04.617 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:04.617 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:04.617 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:47:04.618 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:04.618 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:47:04.618 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:47:04.618 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:47:04.618 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:04.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.600+0000 7f3cf5af5640 10 client.13700.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:47:04.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.600+0000 7f3cf5af5640 10 monclient: _renew_subs 2024-09-16T03:47:04.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.600+0000 7f3cf5af5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:47:04.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.600+0000 7f3cf5af5640 1 -- 172.21.15.42:0/2452579811 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f3cf009cbd0 con 0x7f3cf00a1e20 2024-09-16T03:47:04.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.600+0000 7f3cedffb640 4 set_mon_vals no callback set 2024-09-16T03:47:04.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.600+0000 7f3cf5af5640 1 librados: init done 2024-09-16T03:47:04.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.600+0000 7f3cf5af5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:47:04.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.600+0000 7f3cf5af5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:47:04.619 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.600+0000 7f3cf5af5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:47:04.619 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.600+0000 7f3ce7fff640 1 -- 172.21.15.42:0/2452579811 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f3cd801a070 con 0x7f3cf00a1e20 2024-09-16T03:47:04.619 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.600+0000 7f3cf5af5640 1 -- 172.21.15.42:0/2452579811 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3cf0073a30 con 0x7f3cf00a1e20 2024-09-16T03:47:04.619 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.600+0000 7f3ce7fff640 1 --2- 172.21.15.42:0/2452579811 >> v2:172.21.15.154:6812/254845342 conn(0x7f3cd0037840 0x7f3cd0039d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:04.619 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.600+0000 7f3ce7fff640 1 -- 172.21.15.42:0/2452579811 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(149..149 src has 1..149) ==== 6100+0+0 (secure 0 0 0) 0x7f3cd8048fa0 con 0x7f3cf00a1e20 2024-09-16T03:47:04.619 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.600+0000 7f3ce7fff640 10 client.13700.objecter ms_dispatch 0x7f3cf0000ba0 osd_map(149..149 src has 1..149) 2024-09-16T03:47:04.619 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.600+0000 7f3ce7fff640 3 client.13700.objecter handle_osd_map got epochs [149,149] > 0 2024-09-16T03:47:04.619 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.600+0000 7f3ce7fff640 3 client.13700.objecter handle_osd_map decoding full epoch 149 2024-09-16T03:47:04.619 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.600+0000 7f3ce7fff640 20 client.13700.objecter dump_active .. 0 homeless 2024-09-16T03:47:04.619 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.600+0000 7f3cef7fe640 1 --2- 172.21.15.42:0/2452579811 >> v2:172.21.15.154:6812/254845342 conn(0x7f3cd0037840 0x7f3cd0039d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:04.619 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.600+0000 7f3cef7fe640 10 monclient: get_auth_request con 0x7f3cd0037840 auth_method 0 2024-09-16T03:47:04.619 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.600+0000 7f3ce7fff640 10 client.13700.objecter ms_handle_connect 0x7f3cd0037840 2024-09-16T03:47:04.620 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.600+0000 7f3cef7fe640 1 --2- 172.21.15.42:0/2452579811 >> v2:172.21.15.154:6812/254845342 conn(0x7f3cd0037840 0x7f3cd0039d00 secure :-1 s=READY pgs=3433 cs=0 l=1 rev1=1 crypto rx=0x7f3ce0001650 tx=0x7f3ce0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:47:04.620 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.604+0000 7f3ce7fff640 1 -- 172.21.15.42:0/2452579811 <== mon.2 v2:172.21.15.42:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f3cf01cf130 con 0x7f3cf00a1e20 2024-09-16T03:47:04.620 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.604+0000 7f3ce7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:47:04.620 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.604+0000 7f3ce7fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:47:04.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.836+0000 7f3cf5af5640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:47:04.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.836+0000 7f3cf5af5640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:47:04.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.836+0000 7f3cf5af5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:47:04.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.836+0000 7f3cf5af5640 1 -- 172.21.15.42:0/2452579811 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f3cf0071ef0 con 0x7f3cf00a1e20 2024-09-16T03:47:04.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.836+0000 7f3ce7fff640 1 -- 172.21.15.42:0/2452579811 <== mon.2 v2:172.21.15.42:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7f3cd80129c0 con 0x7f3cf00a1e20 2024-09-16T03:47:04.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.836+0000 7f3ce7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:47:04.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.836+0000 7f3ce7fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:47:04.840 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":48,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:47:04.842 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.840+0000 7f3cf5af5640 10 librados: watch_flush enter 2024-09-16T03:47:04.842 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.840+0000 7f3cf5af5640 10 librados: watch_flush exit 2024-09-16T03:47:04.842 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.840+0000 7f3cf5af5640 20 client.13700.objecter shutdown clearing up homeless session... 2024-09-16T03:47:04.843 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.840+0000 7f3cf5af5640 10 client.13700.objecter successfully canceled tick 2024-09-16T03:47:04.843 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.840+0000 7f3cf5af5640 1 -- 172.21.15.42:0/2452579811 >> v2:172.21.15.154:6812/254845342 conn(0x7f3cd0037840 msgr2=0x7f3cd0039d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:04.843 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.840+0000 7f3cf5af5640 1 --2- 172.21.15.42:0/2452579811 >> v2:172.21.15.154:6812/254845342 conn(0x7f3cd0037840 0x7f3cd0039d00 secure :-1 s=READY pgs=3433 cs=0 l=1 rev1=1 crypto rx=0x7f3ce0001650 tx=0x7f3ce0009040 comp rx=0 tx=0).stop 2024-09-16T03:47:04.843 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.840+0000 7f3cf5af5640 10 monclient: shutdown 2024-09-16T03:47:04.843 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.840+0000 7f3cf5af5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:47:04.843 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.840+0000 7f3cf5af5640 1 -- 172.21.15.42:0/2452579811 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3cf00a1e20 msgr2=0x7f3cf0132460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:04.843 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.840+0000 7f3cf5af5640 1 --2- 172.21.15.42:0/2452579811 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3cf00a1e20 0x7f3cf0132460 secure :-1 s=READY pgs=2470 cs=0 l=1 rev1=1 crypto rx=0x7f3cd800e840 tx=0x7f3cd800ed00 comp rx=0 tx=0).stop 2024-09-16T03:47:04.843 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.840+0000 7f3cf5af5640 1 -- 172.21.15.42:0/2452579811 shutdown_connections 2024-09-16T03:47:04.843 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.840+0000 7f3cf5af5640 1 --2- 172.21.15.42:0/2452579811 >> v2:172.21.15.154:6812/254845342 conn(0x7f3cd0037840 0x7f3cd0039d00 unknown :-1 s=CLOSED pgs=3433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:04.843 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.840+0000 7f3cf5af5640 1 --2- 172.21.15.42:0/2452579811 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3cf01329a0 0x7f3cf011bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:04.843 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.840+0000 7f3cf5af5640 1 --2- 172.21.15.42:0/2452579811 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3cf00a1e20 0x7f3cf0132460 unknown :-1 s=CLOSED pgs=2470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:04.843 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.840+0000 7f3cf5af5640 1 --2- 172.21.15.42:0/2452579811 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3cf00a1330 0x7f3cf0071050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:04.843 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.840+0000 7f3cf5af5640 1 -- 172.21.15.42:0/2452579811 >> 172.21.15.42:0/2452579811 conn(0x7f3cf00a2420 msgr2=0x7f3cf00a46b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:47:04.844 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.840+0000 7f3cf5af5640 1 -- 172.21.15.42:0/2452579811 shutdown_connections 2024-09-16T03:47:04.844 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.840+0000 7f3cf5af5640 1 -- 172.21.15.42:0/2452579811 wait complete. 2024-09-16T03:47:04.844 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.840+0000 7f3cf5af5640 1 librados: shutdown 2024-09-16T03:47:04.859 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T03:47:04.859 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T03:47:04.859 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T03:47:04.859 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T03:47:04.976 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.972+0000 7fdf8e1f9640 10 monclient: get_monmap_and_config 2024-09-16T03:47:04.976 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.972+0000 7fdf8e1f9640 10 monclient: build_initial_monmap 2024-09-16T03:47:04.976 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.972+0000 7fdf8e1f9640 10 monclient: monmap: 2024-09-16T03:47:04.976 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:47:04.976 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:04.977 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:47:04.975465+0000 2024-09-16T03:47:04.977 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:47:04.975465+0000 2024-09-16T03:47:04.977 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:47:04.977 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:04.977 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:47:04.977 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:47:04.977 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:47:04.977 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:47:04.977 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:47:04.977 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:04.981 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.976+0000 7fdf8e1f9640 1 Processor -- start 2024-09-16T03:47:04.981 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.976+0000 7fdf8e1f9640 1 -- start start 2024-09-16T03:47:04.981 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.976+0000 7fdf8e1f9640 10 monclient: init 2024-09-16T03:47:04.982 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.976+0000 7fdf8e1f9640 10 monclient: _reopen_session rank -1 2024-09-16T03:47:04.982 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.976+0000 7fdf8e1f9640 10 monclient: _add_conns ranks=[0,4,2,1,3] 2024-09-16T03:47:04.982 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.976+0000 7fdf8e1f9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fdf880a67e0 addr v1:172.21.15.42:6789/0 2024-09-16T03:47:04.982 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.976+0000 7fdf8e1f9640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fdf88072560 0x7fdf8813a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:04.982 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.976+0000 7fdf8e1f9640 10 monclient(hunting): picked mon.noname-c con 0x7fdf88072560 addr v2:172.21.15.154:3300/0 2024-09-16T03:47:04.982 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.976+0000 7fdf8e1f9640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fdf880a72d0 0x7fdf8813c820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:04.982 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.976+0000 7fdf8e1f9640 10 monclient(hunting): picked mon.noname-b con 0x7fdf880a72d0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:47:04.982 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.976+0000 7fdf8e1f9640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdf88071b80 con 0x7fdf880a67e0 2024-09-16T03:47:04.982 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.976+0000 7fdf8e1f9640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:04.982 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.976+0000 7fdf8e1f9640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fdf88072ed0 con 0x7fdf88072560 2024-09-16T03:47:04.982 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.976+0000 7fdf8e1f9640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:04.982 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.976+0000 7fdf8e1f9640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fdf88070030 con 0x7fdf880a72d0 2024-09-16T03:47:04.983 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.976+0000 7fdf8e1f9640 10 monclient(hunting): _renew_subs 2024-09-16T03:47:04.983 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.976+0000 7fdf8e1f9640 10 monclient(hunting): authenticate will time out at 1271.944681s 2024-09-16T03:47:04.983 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.976+0000 7fdf87fff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fdf880a72d0 0x7fdf8813c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:04.983 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.976+0000 7fdf877fe640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fdf880a67e0 0x7fdf880a6bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:46556/0 (socket says 172.21.15.42:46556) 2024-09-16T03:47:04.983 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.976+0000 7fdf87fff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fdf880a72d0 0x7fdf8813c820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:51564/0 (socket says 172.21.15.42:51564) 2024-09-16T03:47:04.983 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.976+0000 7fdf877fe640 1 -- 172.21.15.42:0/667112704 learned_addr learned my addr 172.21.15.42:0/667112704 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:47:04.983 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.976+0000 7fdf87fff640 10 monclient(hunting): get_auth_request con 0x7fdf880a72d0 auth_method 0 2024-09-16T03:47:04.983 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.976+0000 7fdf87fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:04.983 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.976+0000 7fdf87fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:04.983 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.976+0000 7fdf87fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:47:04.983 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.976+0000 7fdf87fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:47:04.983 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.976+0000 7fdf87fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:47:04.983 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.976+0000 7fdf87fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:47:04.983 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.976+0000 7fdf7effd640 1 --2- 172.21.15.42:0/667112704 >> v2:172.21.15.154:3300/0 conn(0x7fdf88072560 0x7fdf8813a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:04.984 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.976+0000 7fdf7effd640 10 monclient(hunting): get_auth_request con 0x7fdf88072560 auth_method 0 2024-09-16T03:47:04.984 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.976+0000 7fdf7effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:04.984 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.976+0000 7fdf7effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:04.984 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.976+0000 7fdf7effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:47:04.984 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.976+0000 7fdf87fff640 10 monclient(hunting): handle_auth_done global_id 13703 payload 274 2024-09-16T03:47:04.984 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.976+0000 7fdf87fff640 1 -- 172.21.15.42:0/667112704 >> v2:172.21.15.154:3300/0 conn(0x7fdf88072560 msgr2=0x7fdf8813a2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:47:04.984 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.976+0000 7fdf87fff640 1 --2- 172.21.15.42:0/667112704 >> v2:172.21.15.154:3300/0 conn(0x7fdf88072560 0x7fdf8813a2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:04.984 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.976+0000 7fdf87fff640 1 -- 172.21.15.42:0/667112704 >> v1:172.21.15.42:6789/0 conn(0x7fdf880a67e0 legacy=0x7fdf880a6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:04.984 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.976+0000 7fdf87fff640 10 monclient: _finish_hunting 0 2024-09-16T03:47:04.984 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.976+0000 7fdf87fff640 1 monclient: found mon.noname-b 2024-09-16T03:47:04.984 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.976+0000 7fdf87fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:47:04.984 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.976+0000 7fdf87fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T03:47:04.984 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.976+0000 7fdf86ffd640 1 -- 172.21.15.42:0/667112704 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 188650320 0 0) 0x7fdf88071b80 con 0x7fdf880a67e0 2024-09-16T03:47:04.984 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.976+0000 7fdf87fff640 1 -- 172.21.15.42:0/667112704 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf880712e0 con 0x7fdf880a72d0 2024-09-16T03:47:04.985 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.976+0000 7fdf86ffd640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T03:47:04.985 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.976+0000 7fdf87fff640 1 --2- 172.21.15.42:0/667112704 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fdf880a72d0 0x7fdf8813c820 secure :-1 s=READY pgs=2471 cs=0 l=1 rev1=1 crypto rx=0x7fdf80009d90 tx=0x7fdf8001c800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d7d4676b94555573 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:47:04.985 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.976+0000 7fdf86ffd640 1 -- 172.21.15.42:0/667112704 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf800b2070 con 0x7fdf880a72d0 2024-09-16T03:47:04.985 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.976+0000 7fdf86ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:47:04.985 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.976+0000 7fdf86ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:47:04.985 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.976+0000 7fdf86ffd640 10 monclient: dump: 2024-09-16T03:47:04.985 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:47:04.985 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:04.985 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:04.985 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:04.985 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:47:04.985 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:04.985 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:47:04.985 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:47:04.985 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:47:04.986 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:04.986 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.976+0000 7fdf86ffd640 10 monclient: _finish_auth 0 2024-09-16T03:47:04.986 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.976+0000 7fdf86ffd640 10 monclient: _check_auth_tickets 2024-09-16T03:47:04.986 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.976+0000 7fdf86ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:47:04.986 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.976+0000 7fdf86ffd640 1 -- 172.21.15.42:0/667112704 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdf800a58e0 con 0x7fdf880a72d0 2024-09-16T03:47:04.986 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.976+0000 7fdf8e1f9640 5 monclient: authenticate success, global_id 13703 2024-09-16T03:47:04.986 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.976+0000 7fdf8e1f9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:47:04.986 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.976+0000 7fdf86ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:47:04.986 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.976+0000 7fdf8e1f9640 10 monclient: get_monmap_and_config success 2024-09-16T03:47:04.986 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.976+0000 7fdf86ffd640 1 -- 172.21.15.42:0/667112704 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf800a5cb0 con 0x7fdf880a72d0 2024-09-16T03:47:04.986 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.976+0000 7fdf8e1f9640 4 set_mon_vals no callback set 2024-09-16T03:47:04.986 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.976+0000 7fdf86ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:47:04.986 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.976+0000 7fdf86ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:47:04.986 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.976+0000 7fdf86ffd640 10 monclient: dump: 2024-09-16T03:47:04.987 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:47:04.987 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:04.987 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:04.987 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:04.987 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:47:04.987 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:04.987 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:47:04.987 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:47:04.987 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:47:04.987 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:04.987 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.976+0000 7fdf8e1f9640 10 monclient: shutdown 2024-09-16T03:47:04.987 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.976+0000 7fdf8e1f9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:47:04.987 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.976+0000 7fdf8e1f9640 1 -- 172.21.15.42:0/667112704 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fdf880a72d0 msgr2=0x7fdf8813c820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:04.987 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.976+0000 7fdf8e1f9640 1 --2- 172.21.15.42:0/667112704 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fdf880a72d0 0x7fdf8813c820 secure :-1 s=READY pgs=2471 cs=0 l=1 rev1=1 crypto rx=0x7fdf80009d90 tx=0x7fdf8001c800 comp rx=0 tx=0).stop 2024-09-16T03:47:04.987 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.980+0000 7fdf8e1f9640 1 -- 172.21.15.42:0/667112704 shutdown_connections 2024-09-16T03:47:04.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.980+0000 7fdf8e1f9640 1 --2- 172.21.15.42:0/667112704 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fdf880a72d0 0x7fdf8813c820 unknown :-1 s=CLOSED pgs=2471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:04.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.980+0000 7fdf8e1f9640 1 --2- 172.21.15.42:0/667112704 >> v2:172.21.15.154:3300/0 conn(0x7fdf88072560 0x7fdf8813a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:04.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.980+0000 7fdf8e1f9640 1 -- 172.21.15.42:0/667112704 >> 172.21.15.42:0/667112704 conn(0x7fdf880a7ed0 msgr2=0x7fdf880a43d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:47:04.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.980+0000 7fdf8e1f9640 1 -- 172.21.15.42:0/667112704 shutdown_connections 2024-09-16T03:47:04.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.980+0000 7fdf8e1f9640 1 -- 172.21.15.42:0/667112704 wait complete. 2024-09-16T03:47:04.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.980+0000 7fdf8e1f9640 10 monclient: build_initial_monmap 2024-09-16T03:47:04.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.980+0000 7fdf8e1f9640 10 monclient: monmap: 2024-09-16T03:47:04.988 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:47:04.988 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:47:04.988 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:47:04.988 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:47:04.988 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:47:04.988 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:04.988 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:47:04.988 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:47:04.989 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:47:04.989 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:04.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.980+0000 7fdf8e1f9640 1 librados: starting msgr at 2024-09-16T03:47:04.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.980+0000 7fdf8e1f9640 1 librados: starting objecter 2024-09-16T03:47:04.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.980+0000 7fdf8e1f9640 1 Processor -- start 2024-09-16T03:47:04.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.980+0000 7fdf8e1f9640 1 -- start start 2024-09-16T03:47:04.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.980+0000 7fdf8e1f9640 1 librados: setting wanted keys 2024-09-16T03:47:04.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.980+0000 7fdf8e1f9640 1 librados: calling monclient init 2024-09-16T03:47:04.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.980+0000 7fdf8e1f9640 10 monclient: init 2024-09-16T03:47:04.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.980+0000 7fdf8e1f9640 10 monclient: _reopen_session rank -1 2024-09-16T03:47:04.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.980+0000 7fdf8e1f9640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T03:47:04.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.980+0000 7fdf8e1f9640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdf88072560 0x7fdf88134be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:04.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.980+0000 7fdf8e1f9640 10 monclient(hunting): picked mon.noname-a con 0x7fdf88072560 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:47:04.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.980+0000 7fdf8e1f9640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdf880a67e0 0x7fdf88135120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:04.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.980+0000 7fdf8e1f9640 10 monclient(hunting): picked mon.noname-b con 0x7fdf880a67e0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:47:04.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.980+0000 7fdf8e1f9640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fdf880a72d0 0x7fdf88139eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:04.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.980+0000 7fdf8e1f9640 10 monclient(hunting): picked mon.noname-c con 0x7fdf880a72d0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:47:04.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.980+0000 7fdf8e1f9640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:04.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.980+0000 7fdf8e1f9640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fdf8813f700 con 0x7fdf88072560 2024-09-16T03:47:04.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.980+0000 7fdf8e1f9640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:04.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.980+0000 7fdf8e1f9640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fdf8813f580 con 0x7fdf880a67e0 2024-09-16T03:47:04.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.980+0000 7fdf8e1f9640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:04.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.980+0000 7fdf8e1f9640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fdf8813f880 con 0x7fdf880a72d0 2024-09-16T03:47:04.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.980+0000 7fdf8e1f9640 10 monclient(hunting): _renew_subs 2024-09-16T03:47:04.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.980+0000 7fdf8e1f9640 10 monclient(hunting): authenticate will time out at 1271.949594s 2024-09-16T03:47:04.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.980+0000 7fdf877fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdf88072560 0x7fdf88134be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:04.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.980+0000 7fdf877fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdf88072560 0x7fdf88134be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:49676/0 (socket says 172.21.15.42:49676) 2024-09-16T03:47:04.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.980+0000 7fdf877fe640 1 -- 172.21.15.42:0/2341329730 learned_addr learned my addr 172.21.15.42:0/2341329730 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:47:04.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.980+0000 7fdf877fe640 10 monclient(hunting): get_auth_request con 0x7fdf88072560 auth_method 0 2024-09-16T03:47:04.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.980+0000 7fdf877fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:04.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.980+0000 7fdf877fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:04.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.980+0000 7fdf877fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:47:04.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.980+0000 7fdf87fff640 1 --2- 172.21.15.42:0/2341329730 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fdf880a72d0 0x7fdf88139eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:04.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.980+0000 7fdf87fff640 10 monclient(hunting): get_auth_request con 0x7fdf880a72d0 auth_method 0 2024-09-16T03:47:04.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.980+0000 7fdf87fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:04.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.980+0000 7fdf87fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:04.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.980+0000 7fdf87fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:47:04.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.980+0000 7fdf877fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:47:04.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.980+0000 7fdf877fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:47:04.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.980+0000 7fdf877fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:47:04.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.980+0000 7fdf87fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:47:04.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.980+0000 7fdf87fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:47:04.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.980+0000 7fdf87fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:47:04.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.980+0000 7fdf7effd640 1 --2- 172.21.15.42:0/2341329730 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdf880a67e0 0x7fdf88135120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:04.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.980+0000 7fdf877fe640 10 monclient(hunting): handle_auth_done global_id 764139 payload 824 2024-09-16T03:47:05.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.980+0000 7fdf877fe640 1 -- 172.21.15.42:0/2341329730 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fdf880a72d0 msgr2=0x7fdf88139eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:05.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.980+0000 7fdf877fe640 1 --2- 172.21.15.42:0/2341329730 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fdf880a72d0 0x7fdf88139eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:05.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.980+0000 7fdf877fe640 1 -- 172.21.15.42:0/2341329730 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdf880a67e0 msgr2=0x7fdf88135120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:05.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.980+0000 7fdf877fe640 1 --2- 172.21.15.42:0/2341329730 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdf880a67e0 0x7fdf88135120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:05.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.980+0000 7fdf877fe640 10 monclient: _finish_hunting 0 2024-09-16T03:47:05.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.980+0000 7fdf877fe640 1 monclient: found mon.noname-a 2024-09-16T03:47:05.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.980+0000 7fdf877fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:47:05.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.980+0000 7fdf877fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:47:05.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.980+0000 7fdf877fe640 1 -- 172.21.15.42:0/2341329730 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf8811fa10 con 0x7fdf88072560 2024-09-16T03:47:05.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.980+0000 7fdf7effd640 10 monclient: get_auth_request con 0x7fdf880a67e0 auth_method 0 2024-09-16T03:47:05.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.980+0000 7fdf7effd640 1 --2- 172.21.15.42:0/2341329730 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdf880a67e0 0x7fdf88135120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T03:47:05.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.980+0000 7fdf877fe640 1 --2- 172.21.15.42:0/2341329730 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdf88072560 0x7fdf88134be0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fdf780011f0 tx=0x7fdf7800c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:47:05.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.980+0000 7fdf84ff9640 10 client.?.objecter ms_handle_connect 0x7fdf88072560 2024-09-16T03:47:05.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.980+0000 7fdf84ff9640 10 client.?.objecter resend_mon_ops 2024-09-16T03:47:05.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.980+0000 7fdf84ff9640 1 -- 172.21.15.42:0/2341329730 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf78018070 con 0x7fdf88072560 2024-09-16T03:47:05.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.980+0000 7fdf84ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:47:05.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.980+0000 7fdf84ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:47:05.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.980+0000 7fdf84ff9640 10 monclient: dump: 2024-09-16T03:47:05.001 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:47:05.001 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:05.001 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:05.001 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:05.001 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:47:05.001 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:05.001 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:47:05.001 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:47:05.002 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:47:05.002 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:05.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.980+0000 7fdf84ff9640 10 monclient: _finish_auth 0 2024-09-16T03:47:05.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.980+0000 7fdf84ff9640 10 monclient: _check_auth_tickets 2024-09-16T03:47:05.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.980+0000 7fdf84ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:47:05.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.980+0000 7fdf8e1f9640 5 monclient: authenticate success, global_id 764139 2024-09-16T03:47:05.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.980+0000 7fdf8e1f9640 10 monclient: _renew_subs 2024-09-16T03:47:05.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.980+0000 7fdf84ff9640 1 -- 172.21.15.42:0/2341329730 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdf780074e0 con 0x7fdf88072560 2024-09-16T03:47:05.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.980+0000 7fdf8e1f9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:47:05.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.980+0000 7fdf8e1f9640 1 -- 172.21.15.42:0/2341329730 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf8809bf00 con 0x7fdf88072560 2024-09-16T03:47:05.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.980+0000 7fdf84ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T03:47:05.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.980+0000 7fdf84ff9640 1 -- 172.21.15.42:0/2341329730 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf78004a10 con 0x7fdf88072560 2024-09-16T03:47:05.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.980+0000 7fdf84ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:47:05.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.980+0000 7fdf8e1f9640 10 client.764139.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:47:05.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.980+0000 7fdf84ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:47:05.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.980+0000 7fdf867fc640 4 set_mon_vals no callback set 2024-09-16T03:47:05.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.980+0000 7fdf84ff9640 10 monclient: dump: 2024-09-16T03:47:05.003 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:47:05.003 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:05.003 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:05.003 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:05.003 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:47:05.003 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:05.003 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:47:05.003 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:47:05.003 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:47:05.004 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:05.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.980+0000 7fdf8e1f9640 10 monclient: _renew_subs 2024-09-16T03:47:05.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.980+0000 7fdf8e1f9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:47:05.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.980+0000 7fdf8e1f9640 1 -- 172.21.15.42:0/2341329730 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf8811f730 con 0x7fdf88072560 2024-09-16T03:47:05.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.980+0000 7fdf8e1f9640 1 librados: init done 2024-09-16T03:47:05.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.980+0000 7fdf8e1f9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:47:05.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.980+0000 7fdf8e1f9640 1 --2- 172.21.15.42:0/2341329730 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdf8811e280 0x7fdf8813a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:05.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.980+0000 7fdf8e1f9640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:05.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.980+0000 7fdf8e1f9640 1 -- 172.21.15.42:0/2341329730 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fdf8811f3b0 con 0x7fdf8811e280 2024-09-16T03:47:05.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.980+0000 7fdf84ff9640 1 -- 172.21.15.42:0/2341329730 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fdf7801b070 con 0x7fdf88072560 2024-09-16T03:47:05.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.980+0000 7fdf7effd640 1 --2- 172.21.15.42:0/2341329730 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdf8811e280 0x7fdf8813a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:05.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.980+0000 7fdf7effd640 10 monclient: get_auth_request con 0x7fdf8811e280 auth_method 0 2024-09-16T03:47:05.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.980+0000 7fdf7effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:05.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.980+0000 7fdf7effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:05.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.980+0000 7fdf7effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:47:05.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.980+0000 7fdf84ff9640 1 --2- 172.21.15.42:0/2341329730 >> v2:172.21.15.154:6812/254845342 conn(0x7fdf68037810 0x7fdf68039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:05.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.980+0000 7fdf7effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:47:05.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.980+0000 7fdf7effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:47:05.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.980+0000 7fdf7effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:47:05.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.980+0000 7fdf84ff9640 1 -- 172.21.15.42:0/2341329730 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(149..149 src has 1..149) ==== 6100+0+0 (secure 0 0 0) 0x7fdf78048480 con 0x7fdf88072560 2024-09-16T03:47:05.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.980+0000 7fdf84ff9640 10 client.764139.objecter ms_dispatch 0x7fdf88000ba0 osd_map(149..149 src has 1..149) 2024-09-16T03:47:05.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.980+0000 7fdf84ff9640 3 client.764139.objecter handle_osd_map got epochs [149,149] > 0 2024-09-16T03:47:05.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.980+0000 7fdf84ff9640 3 client.764139.objecter handle_osd_map decoding full epoch 149 2024-09-16T03:47:05.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.980+0000 7fdf7effd640 10 monclient(hunting): handle_auth_done global_id 764142 payload 824 2024-09-16T03:47:05.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.980+0000 7fdf7effd640 1 -- 172.21.15.42:0/2341329730 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fdf880a6bc0 con 0x7fdf8811e280 2024-09-16T03:47:05.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.980+0000 7fdf84ff9640 20 client.764139.objecter dump_active .. 0 homeless 2024-09-16T03:47:05.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.980+0000 7fdf84ff9640 10 client.764139.objecter ms_handle_connect 0x7fdf8811e280 2024-09-16T03:47:05.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.984+0000 7fdf84ff9640 10 client.764139.objecter resend_mon_ops 2024-09-16T03:47:05.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.980+0000 7fdf7effd640 1 --2- 172.21.15.42:0/2341329730 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdf8811e280 0x7fdf8813a3f0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fdf8813b630 tx=0x7fdf7000eba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:47:05.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.984+0000 7fdf87fff640 1 --2- 172.21.15.42:0/2341329730 >> v2:172.21.15.154:6812/254845342 conn(0x7fdf68037810 0x7fdf68039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:05.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.984+0000 7fdf84ff9640 1 -- 172.21.15.42:0/2341329730 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf70017020 con 0x7fdf8811e280 2024-09-16T03:47:05.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.984+0000 7fdf84ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:47:05.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.984+0000 7fdf84ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:47:05.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.984+0000 7fdf84ff9640 10 monclient: dump: 2024-09-16T03:47:05.007 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:47:05.007 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:05.007 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:05.007 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:05.007 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:47:05.007 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:05.007 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:47:05.007 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:47:05.007 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:47:05.007 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:05.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.984+0000 7fdf87fff640 10 monclient: get_auth_request con 0x7fdf68037810 auth_method 0 2024-09-16T03:47:05.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.984+0000 7fdf84ff9640 1 -- 172.21.15.42:0/2341329730 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fdf70004030 con 0x7fdf8811e280 2024-09-16T03:47:05.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.984+0000 7fdf84ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:47:05.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.984+0000 7fdf84ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:47:05.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.984+0000 7fdf84ff9640 1 -- 172.21.15.42:0/2341329730 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdf8811e280 msgr2=0x7fdf8813a3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:05.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.984+0000 7fdf84ff9640 1 --2- 172.21.15.42:0/2341329730 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdf8811e280 0x7fdf8813a3f0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fdf8813b630 tx=0x7fdf7000eba0 comp rx=0 tx=0).stop 2024-09-16T03:47:05.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.984+0000 7fdf84ff9640 1 -- 172.21.15.42:0/2341329730 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdf8811e280 msgr2=0x7fdf8813a3f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:47:05.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.984+0000 7fdf84ff9640 1 --2- 172.21.15.42:0/2341329730 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdf8811e280 0x7fdf8813a3f0 secure :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fdf8813b630 tx=0x7fdf7000eba0 comp rx=0 tx=0).stop 2024-09-16T03:47:05.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.984+0000 7fdf87fff640 1 --2- 172.21.15.42:0/2341329730 >> v2:172.21.15.154:6812/254845342 conn(0x7fdf68037810 0x7fdf68039cd0 secure :-1 s=READY pgs=3434 cs=0 l=1 rev1=1 crypto rx=0x7fdf800af000 tx=0x7fdf800a7070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:47:05.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.984+0000 7fdf84ff9640 10 client.764139.objecter ms_handle_connect 0x7fdf68037810 2024-09-16T03:47:05.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.988+0000 7fdf8e1f9640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:47:05.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.988+0000 7fdf8e1f9640 1 --2- 172.21.15.42:0/2341329730 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdf8813a930 0x7fdf8813ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:05.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.988+0000 7fdf8e1f9640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:05.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.988+0000 7fdf8e1f9640 1 -- 172.21.15.42:0/2341329730 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fdf8813b280 con 0x7fdf8813a930 2024-09-16T03:47:05.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.988+0000 7fdf7effd640 1 --2- 172.21.15.42:0/2341329730 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdf8813a930 0x7fdf8813ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:05.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.988+0000 7fdf7effd640 10 monclient: get_auth_request con 0x7fdf8813a930 auth_method 0 2024-09-16T03:47:05.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.988+0000 7fdf7effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:05.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.988+0000 7fdf7effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:05.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.988+0000 7fdf7effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:47:05.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.988+0000 7fdf7effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:47:05.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.988+0000 7fdf7effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:47:05.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.988+0000 7fdf7effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:47:05.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.988+0000 7fdf7effd640 10 monclient(hunting): handle_auth_done global_id 764145 payload 824 2024-09-16T03:47:05.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.988+0000 7fdf7effd640 1 -- 172.21.15.42:0/2341329730 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fdf88135660 con 0x7fdf8813a930 2024-09-16T03:47:05.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.988+0000 7fdf84ff9640 10 client.764139.objecter ms_handle_connect 0x7fdf8813a930 2024-09-16T03:47:05.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.988+0000 7fdf84ff9640 10 client.764139.objecter resend_mon_ops 2024-09-16T03:47:05.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.988+0000 7fdf7effd640 1 --2- 172.21.15.42:0/2341329730 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdf8813a930 0x7fdf8813ad40 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fdf8813b630 tx=0x7fdf70004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:47:05.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.988+0000 7fdf84ff9640 1 -- 172.21.15.42:0/2341329730 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf70017030 con 0x7fdf8813a930 2024-09-16T03:47:05.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.988+0000 7fdf84ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:47:05.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.988+0000 7fdf84ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:47:05.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.988+0000 7fdf84ff9640 10 monclient: dump: 2024-09-16T03:47:05.010 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:47:05.010 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:05.010 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:05.010 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:05.010 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:47:05.010 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:05.010 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:47:05.010 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:47:05.010 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:47:05.010 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:05.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.988+0000 7fdf84ff9640 1 -- 172.21.15.42:0/2341329730 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4546 (secure 0 0 0) 0x7fdf7000c9e0 con 0x7fdf8813a930 2024-09-16T03:47:05.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.988+0000 7fdf84ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:47:05.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.988+0000 7fdf84ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:47:05.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.988+0000 7fdf84ff9640 1 -- 172.21.15.42:0/2341329730 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdf8813a930 msgr2=0x7fdf8813ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:05.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.988+0000 7fdf84ff9640 1 --2- 172.21.15.42:0/2341329730 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdf8813a930 0x7fdf8813ad40 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fdf8813b630 tx=0x7fdf70004740 comp rx=0 tx=0).stop 2024-09-16T03:47:05.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.988+0000 7fdf84ff9640 1 -- 172.21.15.42:0/2341329730 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdf8813a930 msgr2=0x7fdf8813ad40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:47:05.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.988+0000 7fdf84ff9640 1 --2- 172.21.15.42:0/2341329730 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdf8813a930 0x7fdf8813ad40 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:05.011 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:47:05.011 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:47:05.011 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:47:05.011 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "leader", 2024-09-16T03:47:05.012 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 48, 2024-09-16T03:47:05.012 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 15779, 2024-09-16T03:47:05.012 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:47:05.012 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:47:05.012 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T03:47:05.012 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T03:47:05.012 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:47:05.012 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 3, 2024-09-16T03:47:05.012 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:47:05.012 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:47:05.012 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:47:05.012 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:47:05.012 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:47:05.013 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:47:05.013 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:47:05.013 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:47:05.013 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:47:05.013 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:47:05.013 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:47:05.013 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:47:05.013 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:47:05.013 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:47:05.013 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:47:05.013 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:47:05.013 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:47:05.013 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:47:05.013 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:47:05.013 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:47:05.014 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:47:05.014 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:47:05.014 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:47:05.014 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:47:05.014 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:47:05.014 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:47:05.015 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:47:05.015 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:47:05.015 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:47:05.015 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:05.015 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:47:05.016 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:47:05.016 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:47:05.016 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:47:05.016 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:47:05.016 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:47:05.016 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:47:05.016 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:47:05.016 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:47:05.016 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:47:05.016 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:47:05.016 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:47:05.016 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:47:05.016 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:47:05.016 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:47:05.017 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:47:05.017 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:47:05.017 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:47:05.017 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:47:05.017 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:47:05.017 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:47:05.017 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:47:05.017 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:47:05.017 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:47:05.017 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:47:05.017 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:47:05.017 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:47:05.017 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:47:05.017 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:47:05.018 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:47:05.018 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:05.018 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:47:05.018 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:05.018 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:47:05.018 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:47:05.018 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:47:05.018 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:47:05.018 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:05.018 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:47:05.018 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:47:05.018 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:47:05.018 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:05.019 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:05.019 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:47:05.019 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:47:05.019 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:47:05.019 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:47:05.019 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:47:05.019 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:05.019 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:47:05.019 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:47:05.019 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:47:05.019 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:47:05.019 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:47:05.019 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:05.019 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:05.020 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:47:05.020 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:47:05.020 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:47:05.020 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:47:05.020 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:05.020 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:47:05.020 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:47:05.020 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:47:05.020 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:05.020 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:05.020 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:47:05.020 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:47:05.020 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:47:05.020 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:47:05.021 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:47:05.021 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:05.021 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:47:05.021 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:47:05.021 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:47:05.021 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:47:05.021 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:47:05.021 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:05.021 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:05.021 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:47:05.021 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:47:05.021 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:47:05.021 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:47:05.022 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:05.022 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:47:05.022 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:47:05.022 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:47:05.022 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:05.022 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:05.022 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:47:05.022 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:47:05.022 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:47:05.022 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:47:05.022 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:47:05.022 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:05.022 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:47:05.022 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:47:05.023 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:47:05.023 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:47:05.023 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:47:05.023 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:47:05.023 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:47:05.023 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:05.023 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:47:05.023 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:47:05.023 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:05.023 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:47:05.023 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:47:05.023 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:47:05.023 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:47:05.023 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:47:05.023 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:47:05.024 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:05.024 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:47:05.024 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:47:05.024 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 2 2024-09-16T03:47:05.024 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:47:05.024 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:47:05.024 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:05.024 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:47:05.024 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:47:05.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.988+0000 7fdf8e1f9640 10 librados: watch_flush enter 2024-09-16T03:47:05.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.988+0000 7fdf8e1f9640 10 librados: watch_flush exit 2024-09-16T03:47:05.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.988+0000 7fdf8e1f9640 20 client.764139.objecter shutdown clearing up homeless session... 2024-09-16T03:47:05.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.988+0000 7fdf8e1f9640 10 client.764139.objecter successfully canceled tick 2024-09-16T03:47:05.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.988+0000 7fdf8e1f9640 1 -- 172.21.15.42:0/2341329730 >> v2:172.21.15.154:6812/254845342 conn(0x7fdf68037810 msgr2=0x7fdf68039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:05.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.988+0000 7fdf8e1f9640 1 --2- 172.21.15.42:0/2341329730 >> v2:172.21.15.154:6812/254845342 conn(0x7fdf68037810 0x7fdf68039cd0 secure :-1 s=READY pgs=3434 cs=0 l=1 rev1=1 crypto rx=0x7fdf800af000 tx=0x7fdf800a7070 comp rx=0 tx=0).stop 2024-09-16T03:47:05.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.988+0000 7fdf8e1f9640 10 monclient: shutdown 2024-09-16T03:47:05.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.988+0000 7fdf8e1f9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:47:05.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.988+0000 7fdf8e1f9640 1 -- 172.21.15.42:0/2341329730 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdf88072560 msgr2=0x7fdf88134be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:05.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.988+0000 7fdf8e1f9640 1 --2- 172.21.15.42:0/2341329730 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdf88072560 0x7fdf88134be0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fdf780011f0 tx=0x7fdf7800c9b0 comp rx=0 tx=0).stop 2024-09-16T03:47:05.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.988+0000 7fdf87fff640 1 -- 172.21.15.42:0/2341329730 reap_dead start 2024-09-16T03:47:05.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.988+0000 7fdf87fff640 1 -- 172.21.15.42:0/2341329730 reap_dead start 2024-09-16T03:47:05.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.988+0000 7fdf8e1f9640 1 -- 172.21.15.42:0/2341329730 shutdown_connections 2024-09-16T03:47:05.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.988+0000 7fdf8e1f9640 1 -- 172.21.15.42:0/2341329730 >> 172.21.15.42:0/2341329730 conn(0x7fdf880a7ed0 msgr2=0x7fdf880a2340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:47:05.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.988+0000 7fdf8e1f9640 1 -- 172.21.15.42:0/2341329730 shutdown_connections 2024-09-16T03:47:05.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.988+0000 7fdf8e1f9640 1 -- 172.21.15.42:0/2341329730 wait complete. 2024-09-16T03:47:05.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:04.992+0000 7fdf8e1f9640 1 librados: shutdown 2024-09-16T03:47:05.027 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:47:05.027 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T03:47:05.028 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T03:47:05.145 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.144+0000 7f262e32e640 10 monclient: get_monmap_and_config 2024-09-16T03:47:05.145 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.144+0000 7f262e32e640 10 monclient: build_initial_monmap 2024-09-16T03:47:05.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.144+0000 7f262e32e640 10 monclient: monmap: 2024-09-16T03:47:05.146 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:47:05.146 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:05.146 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:47:05.144496+0000 2024-09-16T03:47:05.146 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:47:05.144496+0000 2024-09-16T03:47:05.146 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:47:05.146 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:05.146 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:47:05.146 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:47:05.146 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:47:05.146 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:47:05.146 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:47:05.146 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:05.148 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.144+0000 7f262e32e640 1 Processor -- start 2024-09-16T03:47:05.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.144+0000 7f262e32e640 1 -- start start 2024-09-16T03:47:05.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.144+0000 7f262e32e640 10 monclient: init 2024-09-16T03:47:05.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.144+0000 7f262e32e640 10 monclient: _reopen_session rank -1 2024-09-16T03:47:05.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.144+0000 7f262e32e640 10 monclient: _add_conns ranks=[3,2,4,1,0] 2024-09-16T03:47:05.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.144+0000 7f262e32e640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f26280a1330 0x7f262809f180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:05.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.144+0000 7f262e32e640 10 monclient(hunting): picked mon.noname-a con 0x7f26280a1330 addr v2:172.21.15.42:3300/0 2024-09-16T03:47:05.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.144+0000 7f262e32e640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f26280a1710 0x7f26280a1b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:05.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.144+0000 7f262e32e640 10 monclient(hunting): picked mon.noname-b con 0x7f26280a1710 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:47:05.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.144+0000 7f262e32e640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f26281340f0 0x7f26281345a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:05.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.144+0000 7f262e32e640 10 monclient(hunting): picked mon.noname-c con 0x7f26281340f0 addr v2:172.21.15.154:3300/0 2024-09-16T03:47:05.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.144+0000 7f262e32e640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:05.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.144+0000 7f262e32e640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f262809edf0 con 0x7f26280a1330 2024-09-16T03:47:05.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.144+0000 7f262e32e640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:05.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.144+0000 7f262e32e640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f262809cd60 con 0x7f26281340f0 2024-09-16T03:47:05.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.144+0000 7f262e32e640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:05.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.144+0000 7f262e32e640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f262809e4c0 con 0x7f26280a1710 2024-09-16T03:47:05.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.144+0000 7f262e32e640 10 monclient(hunting): _renew_subs 2024-09-16T03:47:05.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.144+0000 7f262e32e640 10 monclient(hunting): authenticate will time out at 1272.113936s 2024-09-16T03:47:05.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.144+0000 7f26277fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f26280a1710 0x7f26280a1b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:05.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.144+0000 7f26277fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f26280a1710 0x7f26280a1b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:51580/0 (socket says 172.21.15.42:51580) 2024-09-16T03:47:05.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.144+0000 7f26277fe640 1 -- 172.21.15.42:0/2138403235 learned_addr learned my addr 172.21.15.42:0/2138403235 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:47:05.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.144+0000 7f26277fe640 10 monclient(hunting): get_auth_request con 0x7f26280a1710 auth_method 0 2024-09-16T03:47:05.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.144+0000 7f26277fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:05.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.144+0000 7f26277fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:05.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.144+0000 7f26277fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:47:05.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.144+0000 7f26277fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:47:05.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.144+0000 7f26277fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:47:05.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.144+0000 7f2627fff640 1 --2- 172.21.15.42:0/2138403235 >> v2:172.21.15.42:3300/0 conn(0x7f26280a1330 0x7f262809f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:05.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.144+0000 7f26277fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:47:05.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.144+0000 7f2627fff640 10 monclient(hunting): get_auth_request con 0x7f26280a1330 auth_method 0 2024-09-16T03:47:05.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.144+0000 7f2627fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:05.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.144+0000 7f2627fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:05.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.144+0000 7f2627fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:47:05.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.144+0000 7f262c8cb640 1 --2- 172.21.15.42:0/2138403235 >> v2:172.21.15.154:3300/0 conn(0x7f26281340f0 0x7f26281345a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:05.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.144+0000 7f2627fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:47:05.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.144+0000 7f2627fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:47:05.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.144+0000 7f2627fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:47:05.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.144+0000 7f262c8cb640 10 monclient(hunting): get_auth_request con 0x7f26281340f0 auth_method 0 2024-09-16T03:47:05.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.144+0000 7f262c8cb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:05.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.144+0000 7f262c8cb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:05.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.144+0000 7f262c8cb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:47:05.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.144+0000 7f26277fe640 10 monclient(hunting): handle_auth_done global_id 13709 payload 274 2024-09-16T03:47:05.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.144+0000 7f26277fe640 1 -- 172.21.15.42:0/2138403235 >> v2:172.21.15.154:3300/0 conn(0x7f26281340f0 msgr2=0x7f26281345a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:47:05.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.144+0000 7f26277fe640 1 --2- 172.21.15.42:0/2138403235 >> v2:172.21.15.154:3300/0 conn(0x7f26281340f0 0x7f26281345a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:05.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.144+0000 7f26277fe640 1 -- 172.21.15.42:0/2138403235 >> v2:172.21.15.42:3300/0 conn(0x7f26280a1330 msgr2=0x7f262809f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:47:05.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.144+0000 7f26277fe640 1 --2- 172.21.15.42:0/2138403235 >> v2:172.21.15.42:3300/0 conn(0x7f26280a1330 0x7f262809f180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:05.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.144+0000 7f26277fe640 10 monclient: _finish_hunting 0 2024-09-16T03:47:05.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.144+0000 7f26277fe640 1 monclient: found mon.noname-b 2024-09-16T03:47:05.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.144+0000 7f26277fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:47:05.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.144+0000 7f26277fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T03:47:05.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.144+0000 7f26277fe640 1 -- 172.21.15.42:0/2138403235 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26280a2050 con 0x7f26280a1710 2024-09-16T03:47:05.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.148+0000 7f26277fe640 1 --2- 172.21.15.42:0/2138403235 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f26280a1710 0x7f26280a1b10 secure :-1 s=READY pgs=2472 cs=0 l=1 rev1=1 crypto rx=0x7f2618009e50 tx=0x7f261801c8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8c03388f9ee22324 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:47:05.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.148+0000 7f2626ffd640 1 -- 172.21.15.42:0/2138403235 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26180a5640 con 0x7f26280a1710 2024-09-16T03:47:05.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.148+0000 7f2626ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:47:05.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.148+0000 7f2626ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:47:05.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.148+0000 7f2626ffd640 10 monclient: dump: 2024-09-16T03:47:05.159 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:47:05.159 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:05.159 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:05.159 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:05.159 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:47:05.159 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:05.159 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:47:05.160 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:47:05.160 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:47:05.160 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:05.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.148+0000 7f2626ffd640 10 monclient: _finish_auth 0 2024-09-16T03:47:05.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.148+0000 7f2626ffd640 10 monclient: _check_auth_tickets 2024-09-16T03:47:05.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.148+0000 7f2626ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:47:05.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.148+0000 7f262e32e640 5 monclient: authenticate success, global_id 13709 2024-09-16T03:47:05.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.148+0000 7f262e32e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:47:05.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.148+0000 7f2626ffd640 1 -- 172.21.15.42:0/2138403235 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f26180a57e0 con 0x7f26280a1710 2024-09-16T03:47:05.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.148+0000 7f2626ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:47:05.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.148+0000 7f262e32e640 10 monclient: get_monmap_and_config success 2024-09-16T03:47:05.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.148+0000 7f262e32e640 4 set_mon_vals no callback set 2024-09-16T03:47:05.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.148+0000 7f2626ffd640 1 -- 172.21.15.42:0/2138403235 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26180a5bb0 con 0x7f26280a1710 2024-09-16T03:47:05.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.148+0000 7f2626ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:47:05.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.148+0000 7f2626ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:47:05.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.148+0000 7f2626ffd640 10 monclient: dump: 2024-09-16T03:47:05.161 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:47:05.161 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:05.161 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:05.161 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:05.161 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:47:05.161 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:05.161 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:47:05.161 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:47:05.161 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:47:05.161 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:05.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.148+0000 7f262e32e640 10 monclient: shutdown 2024-09-16T03:47:05.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.148+0000 7f262e32e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:47:05.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.148+0000 7f262e32e640 1 -- 172.21.15.42:0/2138403235 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f26280a1710 msgr2=0x7f26280a1b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:05.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.148+0000 7f262e32e640 1 --2- 172.21.15.42:0/2138403235 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f26280a1710 0x7f26280a1b10 secure :-1 s=READY pgs=2472 cs=0 l=1 rev1=1 crypto rx=0x7f2618009e50 tx=0x7f261801c8b0 comp rx=0 tx=0).stop 2024-09-16T03:47:05.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.148+0000 7f262e32e640 1 -- 172.21.15.42:0/2138403235 shutdown_connections 2024-09-16T03:47:05.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.148+0000 7f262e32e640 1 --2- 172.21.15.42:0/2138403235 >> v2:172.21.15.154:3300/0 conn(0x7f26281340f0 0x7f26281345a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:05.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.148+0000 7f262e32e640 1 --2- 172.21.15.42:0/2138403235 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f26280a1710 0x7f26280a1b10 unknown :-1 s=CLOSED pgs=2472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:05.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.148+0000 7f262e32e640 1 --2- 172.21.15.42:0/2138403235 >> v2:172.21.15.42:3300/0 conn(0x7f26280a1330 0x7f262809f180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:05.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.148+0000 7f262e32e640 1 -- 172.21.15.42:0/2138403235 >> 172.21.15.42:0/2138403235 conn(0x7f26280a2420 msgr2=0x7f26280a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:47:05.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.148+0000 7f262e32e640 1 -- 172.21.15.42:0/2138403235 shutdown_connections 2024-09-16T03:47:05.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.148+0000 7f262e32e640 1 -- 172.21.15.42:0/2138403235 wait complete. 2024-09-16T03:47:05.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.148+0000 7f262e32e640 10 monclient: build_initial_monmap 2024-09-16T03:47:05.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.148+0000 7f262e32e640 10 monclient: monmap: 2024-09-16T03:47:05.162 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:47:05.162 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:47:05.163 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:47:05.163 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:47:05.163 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:47:05.163 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:05.163 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:47:05.163 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:47:05.163 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:47:05.163 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:05.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.148+0000 7f262e32e640 1 librados: starting msgr at 2024-09-16T03:47:05.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.148+0000 7f262e32e640 1 librados: starting objecter 2024-09-16T03:47:05.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.148+0000 7f262e32e640 1 Processor -- start 2024-09-16T03:47:05.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.148+0000 7f262e32e640 1 -- start start 2024-09-16T03:47:05.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.148+0000 7f262e32e640 1 librados: setting wanted keys 2024-09-16T03:47:05.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.148+0000 7f262e32e640 1 librados: calling monclient init 2024-09-16T03:47:05.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.148+0000 7f262e32e640 10 monclient: init 2024-09-16T03:47:05.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.148+0000 7f262e32e640 10 monclient: _reopen_session rank -1 2024-09-16T03:47:05.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.148+0000 7f262e32e640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:47:05.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.148+0000 7f262e32e640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f26280a1330 0x7f26281399e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:05.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.148+0000 7f262e32e640 10 monclient(hunting): picked mon.noname-a con 0x7f26280a1330 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:47:05.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.148+0000 7f262e32e640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f26280a1710 0x7f2628130730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:05.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.148+0000 7f262e32e640 10 monclient(hunting): picked mon.noname-c con 0x7f26280a1710 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:47:05.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.148+0000 7f262e32e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f26281340f0 0x7f262813dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:05.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.148+0000 7f262e32e640 10 monclient(hunting): picked mon.noname-b con 0x7f26281340f0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:47:05.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.148+0000 7f262e32e640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:05.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.148+0000 7f262e32e640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f262809d1b0 con 0x7f26280a1330 2024-09-16T03:47:05.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.148+0000 7f262e32e640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:05.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.148+0000 7f262e32e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f262809d030 con 0x7f26281340f0 2024-09-16T03:47:05.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.148+0000 7f262e32e640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:05.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.148+0000 7f262e32e640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f262809ba50 con 0x7f26280a1710 2024-09-16T03:47:05.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.148+0000 7f262e32e640 10 monclient(hunting): _renew_subs 2024-09-16T03:47:05.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.148+0000 7f262e32e640 10 monclient(hunting): authenticate will time out at 1272.119142s 2024-09-16T03:47:05.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.148+0000 7f26277fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f26280a1710 0x7f2628130730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:05.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.148+0000 7f2627fff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f26280a1330 0x7f26281399e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:05.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.148+0000 7f26277fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f26280a1710 0x7f2628130730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:51592/0 (socket says 172.21.15.42:51592) 2024-09-16T03:47:05.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.148+0000 7f2627fff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f26280a1330 0x7f26281399e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:49712/0 (socket says 172.21.15.42:49712) 2024-09-16T03:47:05.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.148+0000 7f26277fe640 1 -- 172.21.15.42:0/162863229 learned_addr learned my addr 172.21.15.42:0/162863229 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:47:05.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.148+0000 7f26277fe640 10 monclient(hunting): get_auth_request con 0x7f26280a1710 auth_method 0 2024-09-16T03:47:05.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.148+0000 7f26277fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:05.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.148+0000 7f26277fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:05.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.148+0000 7f26277fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:47:05.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.148+0000 7f2627fff640 10 monclient(hunting): get_auth_request con 0x7f26280a1330 auth_method 0 2024-09-16T03:47:05.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.148+0000 7f2627fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:05.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.148+0000 7f2627fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:05.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.148+0000 7f2627fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:47:05.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.148+0000 7f2627fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:47:05.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.148+0000 7f2627fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:47:05.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.148+0000 7f2627fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:47:05.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.148+0000 7f26277fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:47:05.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.148+0000 7f26277fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:47:05.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.148+0000 7f26277fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:47:05.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.148+0000 7f2627fff640 10 monclient(hunting): handle_auth_done global_id 764151 payload 824 2024-09-16T03:47:05.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.148+0000 7f2627fff640 1 -- 172.21.15.42:0/162863229 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f26280a1710 msgr2=0x7f2628130730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:05.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.148+0000 7f2627fff640 1 --2- 172.21.15.42:0/162863229 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f26280a1710 0x7f2628130730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:05.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.148+0000 7f2627fff640 1 -- 172.21.15.42:0/162863229 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f26281340f0 msgr2=0x7f262813dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:05.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.148+0000 7f2627fff640 1 --2- 172.21.15.42:0/162863229 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f26281340f0 0x7f262813dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:05.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.148+0000 7f2627fff640 10 monclient: _finish_hunting 0 2024-09-16T03:47:05.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.148+0000 7f2627fff640 1 monclient: found mon.noname-a 2024-09-16T03:47:05.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.148+0000 7f2627fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:47:05.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.148+0000 7f2627fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:47:05.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.148+0000 7f2627fff640 1 -- 172.21.15.42:0/162863229 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26281cf5a0 con 0x7f26280a1330 2024-09-16T03:47:05.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.148+0000 7f26277fe640 1 --2- 172.21.15.42:0/162863229 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f26280a1710 0x7f2628130730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:47:05.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.148+0000 7f2627fff640 1 --2- 172.21.15.42:0/162863229 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f26280a1330 0x7f26281399e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f261000e820 tx=0x7f261000ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:47:05.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.148+0000 7f2624ff9640 10 client.?.objecter ms_handle_connect 0x7f26280a1330 2024-09-16T03:47:05.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.148+0000 7f2624ff9640 10 client.?.objecter resend_mon_ops 2024-09-16T03:47:05.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.152+0000 7f2624ff9640 1 -- 172.21.15.42:0/162863229 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2610017070 con 0x7f26280a1330 2024-09-16T03:47:05.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.152+0000 7f2624ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:47:05.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.152+0000 7f2624ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:47:05.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.152+0000 7f2624ff9640 10 monclient: dump: 2024-09-16T03:47:05.167 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:47:05.168 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:05.168 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:05.168 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:05.168 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:47:05.168 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:05.168 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:47:05.168 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:47:05.168 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:47:05.168 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:05.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.152+0000 7f2624ff9640 10 monclient: _finish_auth 0 2024-09-16T03:47:05.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.152+0000 7f2624ff9640 10 monclient: _check_auth_tickets 2024-09-16T03:47:05.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.152+0000 7f2624ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:47:05.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.152+0000 7f2624ff9640 1 -- 172.21.15.42:0/162863229 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f261000c940 con 0x7f26280a1330 2024-09-16T03:47:05.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.152+0000 7f2624ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T03:47:05.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.152+0000 7f2624ff9640 1 -- 172.21.15.42:0/162863229 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f261000ccf0 con 0x7f26280a1330 2024-09-16T03:47:05.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.152+0000 7f2624ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:47:05.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.152+0000 7f2624ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:47:05.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.152+0000 7f2624ff9640 10 monclient: dump: 2024-09-16T03:47:05.169 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:47:05.169 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:05.169 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:05.169 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:05.169 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:47:05.169 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:05.170 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:47:05.170 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:47:05.170 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:47:05.170 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:05.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.152+0000 7f26267fc640 4 set_mon_vals no callback set 2024-09-16T03:47:05.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.152+0000 7f262e32e640 5 monclient: authenticate success, global_id 764151 2024-09-16T03:47:05.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.152+0000 7f262e32e640 10 monclient: _renew_subs 2024-09-16T03:47:05.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.152+0000 7f262e32e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:47:05.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.152+0000 7f262e32e640 1 -- 172.21.15.42:0/162863229 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f262811ce20 con 0x7f26280a1330 2024-09-16T03:47:05.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.152+0000 7f262e32e640 10 client.764151.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:47:05.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.152+0000 7f262e32e640 10 monclient: _renew_subs 2024-09-16T03:47:05.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.152+0000 7f262e32e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:47:05.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.152+0000 7f262e32e640 1 -- 172.21.15.42:0/162863229 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2628072490 con 0x7f26280a1330 2024-09-16T03:47:05.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.152+0000 7f262e32e640 1 librados: init done 2024-09-16T03:47:05.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.152+0000 7f262e32e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:47:05.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.152+0000 7f262e32e640 1 --2- 172.21.15.42:0/162863229 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f262811c9c0 0x7f262813e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:05.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.152+0000 7f262e32e640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:05.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.152+0000 7f262e32e640 1 -- 172.21.15.42:0/162863229 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f2628073a30 con 0x7f262811c9c0 2024-09-16T03:47:05.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.152+0000 7f26277fe640 1 --2- 172.21.15.42:0/162863229 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f262811c9c0 0x7f262813e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:05.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.152+0000 7f26277fe640 10 monclient: get_auth_request con 0x7f262811c9c0 auth_method 0 2024-09-16T03:47:05.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.152+0000 7f26277fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:05.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.152+0000 7f26277fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:05.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.152+0000 7f26277fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:47:05.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.152+0000 7f26277fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:47:05.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.152+0000 7f26277fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:47:05.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.152+0000 7f26277fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:47:05.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.152+0000 7f2624ff9640 1 -- 172.21.15.42:0/162863229 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f261001a070 con 0x7f26280a1330 2024-09-16T03:47:05.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.152+0000 7f26277fe640 10 monclient(hunting): handle_auth_done global_id 13715 payload 824 2024-09-16T03:47:05.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.152+0000 7f26277fe640 1 -- 172.21.15.42:0/162863229 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f26280a1b10 con 0x7f262811c9c0 2024-09-16T03:47:05.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.152+0000 7f2624ff9640 1 --2- 172.21.15.42:0/162863229 >> v2:172.21.15.154:6812/254845342 conn(0x7f2608037840 0x7f2608039d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:05.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.152+0000 7f2624ff9640 1 -- 172.21.15.42:0/162863229 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(149..149 src has 1..149) ==== 6100+0+0 (secure 0 0 0) 0x7f2610049060 con 0x7f26280a1330 2024-09-16T03:47:05.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.152+0000 7f2624ff9640 10 client.764151.objecter ms_dispatch 0x7f2628000ba0 osd_map(149..149 src has 1..149) 2024-09-16T03:47:05.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.152+0000 7f2624ff9640 3 client.764151.objecter handle_osd_map got epochs [149,149] > 0 2024-09-16T03:47:05.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.152+0000 7f2624ff9640 3 client.764151.objecter handle_osd_map decoding full epoch 149 2024-09-16T03:47:05.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.152+0000 7f26277fe640 1 --2- 172.21.15.42:0/162863229 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f262811c9c0 0x7f262813e110 secure :-1 s=READY pgs=2473 cs=0 l=1 rev1=1 crypto rx=0x7f261801d150 tx=0x7f2618019650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:47:05.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.152+0000 7f2624ff9640 20 client.764151.objecter dump_active .. 0 homeless 2024-09-16T03:47:05.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.152+0000 7f2624ff9640 10 client.764151.objecter ms_handle_connect 0x7f262811c9c0 2024-09-16T03:47:05.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.152+0000 7f2624ff9640 10 client.764151.objecter resend_mon_ops 2024-09-16T03:47:05.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.152+0000 7f262c8cb640 1 --2- 172.21.15.42:0/162863229 >> v2:172.21.15.154:6812/254845342 conn(0x7f2608037840 0x7f2608039d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:05.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.152+0000 7f262c8cb640 10 monclient: get_auth_request con 0x7f2608037840 auth_method 0 2024-09-16T03:47:05.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.152+0000 7f2624ff9640 1 -- 172.21.15.42:0/162863229 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2618009510 con 0x7f262811c9c0 2024-09-16T03:47:05.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.152+0000 7f2624ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:47:05.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.152+0000 7f2624ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:47:05.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.152+0000 7f2624ff9640 10 monclient: dump: 2024-09-16T03:47:05.173 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:47:05.173 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:05.173 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:05.173 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:05.173 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:47:05.173 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:05.173 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:47:05.174 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:47:05.174 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:47:05.174 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:05.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.152+0000 7f2624ff9640 1 -- 172.21.15.42:0/162863229 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f26180a5640 con 0x7f262811c9c0 2024-09-16T03:47:05.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.152+0000 7f2624ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:47:05.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.152+0000 7f2624ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:47:05.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.152+0000 7f2624ff9640 1 -- 172.21.15.42:0/162863229 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f262811c9c0 msgr2=0x7f262813e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:05.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.152+0000 7f2624ff9640 1 --2- 172.21.15.42:0/162863229 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f262811c9c0 0x7f262813e110 secure :-1 s=READY pgs=2473 cs=0 l=1 rev1=1 crypto rx=0x7f261801d150 tx=0x7f2618019650 comp rx=0 tx=0).stop 2024-09-16T03:47:05.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.152+0000 7f2624ff9640 1 -- 172.21.15.42:0/162863229 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f262811c9c0 msgr2=0x7f262813e110 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:47:05.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.152+0000 7f2624ff9640 1 --2- 172.21.15.42:0/162863229 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f262811c9c0 0x7f262813e110 secure :-1 s=CLOSED pgs=2473 cs=0 l=1 rev1=1 crypto rx=0x7f261801d150 tx=0x7f2618019650 comp rx=0 tx=0).stop 2024-09-16T03:47:05.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.152+0000 7f262c8cb640 1 --2- 172.21.15.42:0/162863229 >> v2:172.21.15.154:6812/254845342 conn(0x7f2608037840 0x7f2608039d00 secure :-1 s=READY pgs=3435 cs=0 l=1 rev1=1 crypto rx=0x7f262813f4f0 tx=0x7f2620008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:47:05.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.152+0000 7f2624ff9640 10 client.764151.objecter ms_handle_connect 0x7f2608037840 2024-09-16T03:47:05.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.160+0000 7f262e32e640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:47:05.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.160+0000 7f262e32e640 1 --2- 172.21.15.42:0/162863229 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2628130c70 0x7f262813e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:05.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.160+0000 7f262e32e640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:05.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.160+0000 7f262e32e640 1 -- 172.21.15.42:0/162863229 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f2628069040 con 0x7f2628130c70 2024-09-16T03:47:05.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.160+0000 7f26277fe640 1 --2- 172.21.15.42:0/162863229 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2628130c70 0x7f262813e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:05.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.160+0000 7f26277fe640 10 monclient: get_auth_request con 0x7f2628130c70 auth_method 0 2024-09-16T03:47:05.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.160+0000 7f26277fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:05.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.160+0000 7f26277fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:05.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.160+0000 7f26277fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:47:05.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.160+0000 7f26277fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:47:05.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.160+0000 7f26277fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:47:05.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.160+0000 7f26277fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:47:05.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.160+0000 7f26277fe640 10 monclient(hunting): handle_auth_done global_id 13718 payload 824 2024-09-16T03:47:05.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.160+0000 7f26277fe640 1 -- 172.21.15.42:0/162863229 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f26281da420 con 0x7f2628130c70 2024-09-16T03:47:05.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.160+0000 7f26277fe640 1 --2- 172.21.15.42:0/162863229 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2628130c70 0x7f262813e650 secure :-1 s=READY pgs=2474 cs=0 l=1 rev1=1 crypto rx=0x7f2628073bb0 tx=0x7f26180a7000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:47:05.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.160+0000 7f2624ff9640 10 client.764151.objecter ms_handle_connect 0x7f2628130c70 2024-09-16T03:47:05.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.160+0000 7f2624ff9640 10 client.764151.objecter resend_mon_ops 2024-09-16T03:47:05.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.160+0000 7f2624ff9640 1 -- 172.21.15.42:0/162863229 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26180c09e0 con 0x7f2628130c70 2024-09-16T03:47:05.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.160+0000 7f2624ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:47:05.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.160+0000 7f2624ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:47:05.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.160+0000 7f2624ff9640 10 monclient: dump: 2024-09-16T03:47:05.178 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:47:05.178 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:05.178 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:05.178 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:05.178 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:47:05.178 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:05.178 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:47:05.178 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:47:05.178 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:47:05.178 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:05.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.160+0000 7f2624ff9640 1 -- 172.21.15.42:0/162863229 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4712 (secure 0 0 0) 0x7f2618009af0 con 0x7f2628130c70 2024-09-16T03:47:05.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.160+0000 7f2624ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:47:05.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.160+0000 7f2624ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:47:05.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.160+0000 7f2624ff9640 1 -- 172.21.15.42:0/162863229 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2628130c70 msgr2=0x7f262813e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:05.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.160+0000 7f2624ff9640 1 --2- 172.21.15.42:0/162863229 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2628130c70 0x7f262813e650 secure :-1 s=READY pgs=2474 cs=0 l=1 rev1=1 crypto rx=0x7f2628073bb0 tx=0x7f26180a7000 comp rx=0 tx=0).stop 2024-09-16T03:47:05.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.160+0000 7f2624ff9640 1 -- 172.21.15.42:0/162863229 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2628130c70 msgr2=0x7f262813e650 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:47:05.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.160+0000 7f2624ff9640 1 --2- 172.21.15.42:0/162863229 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2628130c70 0x7f262813e650 secure :-1 s=CLOSED pgs=2474 cs=0 l=1 rev1=1 crypto rx=0x7f2628073bb0 tx=0x7f26180a7000 comp rx=0 tx=0).stop 2024-09-16T03:47:05.179 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:47:05.179 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:47:05.179 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:47:05.179 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T03:47:05.179 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 48, 2024-09-16T03:47:05.180 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 304530, 2024-09-16T03:47:05.180 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:47:05.180 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:47:05.180 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T03:47:05.180 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T03:47:05.180 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:47:05.180 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 3, 2024-09-16T03:47:05.180 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:47:05.180 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:47:05.180 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:47:05.180 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:47:05.180 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:47:05.180 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:47:05.180 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:47:05.181 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:47:05.181 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:47:05.181 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:47:05.181 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:47:05.181 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:47:05.181 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:47:05.181 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:47:05.181 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:47:05.181 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:47:05.181 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:47:05.181 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:47:05.181 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:47:05.181 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:47:05.182 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:47:05.182 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:47:05.182 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:47:05.182 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:47:05.182 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:47:05.182 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:47:05.182 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:47:05.182 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:47:05.182 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:47:05.182 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:05.182 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:47:05.182 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:47:05.182 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:47:05.183 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:47:05.183 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:47:05.183 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:47:05.183 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:47:05.183 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:47:05.183 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:47:05.183 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:47:05.183 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:47:05.183 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:47:05.183 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:47:05.183 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:47:05.183 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:47:05.183 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:47:05.183 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:47:05.183 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:47:05.184 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:47:05.184 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:47:05.184 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:47:05.184 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:47:05.184 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:47:05.184 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:47:05.184 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:47:05.184 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:47:05.184 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:47:05.184 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:47:05.184 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:47:05.184 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:47:05.184 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:05.184 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:47:05.185 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:05.185 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:47:05.185 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:47:05.185 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:47:05.185 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:47:05.185 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:05.185 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:47:05.185 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:47:05.185 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:47:05.185 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:05.185 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:05.185 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:47:05.185 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:47:05.185 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:47:05.186 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:47:05.186 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:47:05.186 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:05.186 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:47:05.186 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:47:05.186 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:47:05.186 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:47:05.186 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:47:05.186 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:05.186 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:05.186 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:47:05.186 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:47:05.186 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:47:05.187 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:47:05.187 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:05.187 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:47:05.187 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:47:05.187 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:47:05.187 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:05.187 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:05.187 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:47:05.187 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:47:05.187 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:47:05.187 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:47:05.187 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:47:05.188 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:05.188 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:47:05.188 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:47:05.188 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:47:05.188 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:47:05.188 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:47:05.188 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:05.188 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:05.188 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:47:05.188 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:47:05.188 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:47:05.188 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:47:05.188 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:05.188 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:47:05.189 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:47:05.189 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:47:05.189 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:05.189 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:05.189 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:47:05.189 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:47:05.189 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:47:05.189 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:47:05.189 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:47:05.189 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:05.189 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:47:05.189 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:47:05.189 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:47:05.189 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:47:05.190 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:47:05.190 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:47:05.190 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:47:05.190 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:05.190 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:47:05.190 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:47:05.190 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:05.190 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:47:05.190 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:47:05.190 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:47:05.190 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:47:05.190 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:47:05.190 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T03:47:05.190 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:05.191 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:47:05.191 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:47:05.191 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 6 2024-09-16T03:47:05.191 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:47:05.191 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:47:05.191 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:47:05.191 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:05.191 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:47:05.191 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:47:05.191 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 7 2024-09-16T03:47:05.191 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:47:05.191 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:47:05.191 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:05.191 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:47:05.192 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:47:05.193 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.160+0000 7f262e32e640 10 librados: watch_flush enter 2024-09-16T03:47:05.193 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.160+0000 7f262e32e640 10 librados: watch_flush exit 2024-09-16T03:47:05.193 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.160+0000 7f262e32e640 20 client.764151.objecter shutdown clearing up homeless session... 2024-09-16T03:47:05.193 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.160+0000 7f262e32e640 10 client.764151.objecter successfully canceled tick 2024-09-16T03:47:05.193 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.160+0000 7f262e32e640 1 -- 172.21.15.42:0/162863229 >> v2:172.21.15.154:6812/254845342 conn(0x7f2608037840 msgr2=0x7f2608039d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:05.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.160+0000 7f262e32e640 1 --2- 172.21.15.42:0/162863229 >> v2:172.21.15.154:6812/254845342 conn(0x7f2608037840 0x7f2608039d00 secure :-1 s=READY pgs=3435 cs=0 l=1 rev1=1 crypto rx=0x7f262813f4f0 tx=0x7f2620008040 comp rx=0 tx=0).stop 2024-09-16T03:47:05.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.160+0000 7f262e32e640 10 monclient: shutdown 2024-09-16T03:47:05.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.160+0000 7f262e32e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:47:05.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.160+0000 7f262e32e640 1 -- 172.21.15.42:0/162863229 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f26280a1330 msgr2=0x7f26281399e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:05.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.160+0000 7f262e32e640 1 --2- 172.21.15.42:0/162863229 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f26280a1330 0x7f26281399e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f261000e820 tx=0x7f261000ece0 comp rx=0 tx=0).stop 2024-09-16T03:47:05.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.160+0000 7f262c8cb640 1 -- 172.21.15.42:0/162863229 reap_dead start 2024-09-16T03:47:05.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.160+0000 7f262c8cb640 1 -- 172.21.15.42:0/162863229 reap_dead start 2024-09-16T03:47:05.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.160+0000 7f262e32e640 1 -- 172.21.15.42:0/162863229 shutdown_connections 2024-09-16T03:47:05.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.160+0000 7f262e32e640 1 -- 172.21.15.42:0/162863229 >> 172.21.15.42:0/162863229 conn(0x7f26280a2420 msgr2=0x7f26280a4cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:47:05.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.160+0000 7f262e32e640 1 -- 172.21.15.42:0/162863229 shutdown_connections 2024-09-16T03:47:05.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.160+0000 7f262e32e640 1 -- 172.21.15.42:0/162863229 wait complete. 2024-09-16T03:47:05.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.160+0000 7f262e32e640 1 librados: shutdown 2024-09-16T03:47:05.194 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:47:05.195 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T03:47:05.195 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T03:47:05.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.328+0000 7f535c5a2640 10 monclient: get_monmap_and_config 2024-09-16T03:47:05.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.328+0000 7f535c5a2640 10 monclient: build_initial_monmap 2024-09-16T03:47:05.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.328+0000 7f535c5a2640 10 monclient: monmap: 2024-09-16T03:47:05.333 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:47:05.333 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:05.333 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:47:05.331904+0000 2024-09-16T03:47:05.333 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:47:05.331904+0000 2024-09-16T03:47:05.333 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:47:05.333 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:05.333 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:47:05.333 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:47:05.334 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:47:05.334 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:47:05.334 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:47:05.334 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:05.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.332+0000 7f535c5a2640 1 Processor -- start 2024-09-16T03:47:05.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.332+0000 7f535c5a2640 1 -- start start 2024-09-16T03:47:05.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.332+0000 7f535c5a2640 10 monclient: init 2024-09-16T03:47:05.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.332+0000 7f535c5a2640 10 monclient: _reopen_session rank -1 2024-09-16T03:47:05.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.332+0000 7f535c5a2640 10 monclient: _add_conns ranks=[0,4,3,1,2] 2024-09-16T03:47:05.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.332+0000 7f535c5a2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5354131df0 addr v1:172.21.15.42:6789/0 2024-09-16T03:47:05.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.332+0000 7f535c5a2640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f53541319c0 0x7f535413a280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:05.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.332+0000 7f535c5a2640 10 monclient(hunting): picked mon.noname-c con 0x7f53541319c0 addr v2:172.21.15.154:3300/0 2024-09-16T03:47:05.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.332+0000 7f535c5a2640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f535413a7c0 0x7f535413cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:05.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.332+0000 7f535c5a2640 10 monclient(hunting): picked mon.noname-a con 0x7f535413a7c0 addr v2:172.21.15.42:3300/0 2024-09-16T03:47:05.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.332+0000 7f535c5a2640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53541328e0 con 0x7f5354131df0 2024-09-16T03:47:05.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.332+0000 7f535c5a2640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:05.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.332+0000 7f535c5a2640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f53540a1b80 con 0x7f535413a7c0 2024-09-16T03:47:05.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.332+0000 7f535c5a2640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:05.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.332+0000 7f535c5a2640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f53540a3710 con 0x7f53541319c0 2024-09-16T03:47:05.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.332+0000 7f535c5a2640 10 monclient(hunting): _renew_subs 2024-09-16T03:47:05.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.332+0000 7f535c5a2640 10 monclient(hunting): authenticate will time out at 1272.301181s 2024-09-16T03:47:05.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.332+0000 7f535a33e640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f5354131df0 0x7f53541321d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:46566/0 (socket says 172.21.15.42:46566) 2024-09-16T03:47:05.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.332+0000 7f535a33e640 1 -- 172.21.15.42:0/1160371715 learned_addr learned my addr 172.21.15.42:0/1160371715 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:47:05.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.332+0000 7f5359b3d640 1 --2- 172.21.15.42:0/1160371715 >> v2:172.21.15.154:3300/0 conn(0x7f53541319c0 0x7f535413a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:05.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.332+0000 7f5359b3d640 10 monclient(hunting): get_auth_request con 0x7f53541319c0 auth_method 0 2024-09-16T03:47:05.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.332+0000 7f5359b3d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:05.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.332+0000 7f5359b3d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:05.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.332+0000 7f5359b3d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:47:05.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.332+0000 7f535ab3f640 1 --2- 172.21.15.42:0/1160371715 >> v2:172.21.15.42:3300/0 conn(0x7f535413a7c0 0x7f535413cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:05.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.332+0000 7f535ab3f640 10 monclient(hunting): get_auth_request con 0x7f535413a7c0 auth_method 0 2024-09-16T03:47:05.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.332+0000 7f535ab3f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:05.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.332+0000 7f535ab3f640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:05.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.332+0000 7f535ab3f640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:47:05.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.332+0000 7f5359b3d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:47:05.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.332+0000 7f5359b3d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:47:05.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.332+0000 7f5359b3d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:47:05.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.332+0000 7f535ab3f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:47:05.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.332+0000 7f535ab3f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:47:05.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.332+0000 7f535ab3f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:47:05.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.332+0000 7f5359b3d640 10 monclient(hunting): handle_auth_done global_id 754312 payload 274 2024-09-16T03:47:05.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.332+0000 7f5359b3d640 1 -- 172.21.15.42:0/1160371715 >> v2:172.21.15.42:3300/0 conn(0x7f535413a7c0 msgr2=0x7f535413cbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:47:05.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.332+0000 7f5359b3d640 1 --2- 172.21.15.42:0/1160371715 >> v2:172.21.15.42:3300/0 conn(0x7f535413a7c0 0x7f535413cbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:05.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.332+0000 7f5359b3d640 1 -- 172.21.15.42:0/1160371715 >> v1:172.21.15.42:6789/0 conn(0x7f5354131df0 legacy=0x7f53541321d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:05.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.332+0000 7f5359b3d640 10 monclient: _finish_hunting 0 2024-09-16T03:47:05.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.332+0000 7f5359b3d640 1 monclient: found mon.noname-c 2024-09-16T03:47:05.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.332+0000 7f5359b3d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:47:05.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.332+0000 7f5359b3d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.154:3300/0 2024-09-16T03:47:05.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.332+0000 7f5359b3d640 1 -- 172.21.15.42:0/1160371715 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53540a0260 con 0x7f53541319c0 2024-09-16T03:47:05.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.336+0000 7f5359b3d640 1 --2- 172.21.15.42:0/1160371715 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f53541319c0 0x7f535413a280 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f5354075b90 tx=0x7f534401c020 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ff3fb91364245218 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:47:05.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.336+0000 7f535933c640 1 -- 172.21.15.42:0/1160371715 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53440b1070 con 0x7f53541319c0 2024-09-16T03:47:05.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.336+0000 7f535933c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:47:05.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.336+0000 7f535933c640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:47:05.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.336+0000 7f535933c640 10 monclient: dump: 2024-09-16T03:47:05.344 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:47:05.344 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:05.344 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:05.344 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:05.344 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:47:05.344 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:05.344 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:47:05.344 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:47:05.345 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:47:05.345 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:05.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.336+0000 7f535933c640 10 monclient: _finish_auth 0 2024-09-16T03:47:05.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.336+0000 7f535933c640 10 monclient: _check_auth_tickets 2024-09-16T03:47:05.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.336+0000 7f535933c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:47:05.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.336+0000 7f535933c640 1 -- 172.21.15.42:0/1160371715 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f53440a4860 con 0x7f53541319c0 2024-09-16T03:47:05.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.336+0000 7f535c5a2640 5 monclient: authenticate success, global_id 754312 2024-09-16T03:47:05.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.336+0000 7f535c5a2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:47:05.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.336+0000 7f535933c640 10 monclient: handle_config config(0 keys) 2024-09-16T03:47:05.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.336+0000 7f535c5a2640 10 monclient: get_monmap_and_config success 2024-09-16T03:47:05.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.336+0000 7f535933c640 1 -- 172.21.15.42:0/1160371715 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53440a4c30 con 0x7f53541319c0 2024-09-16T03:47:05.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.336+0000 7f535933c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:47:05.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.336+0000 7f535c5a2640 4 set_mon_vals no callback set 2024-09-16T03:47:05.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.336+0000 7f535933c640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:47:05.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.336+0000 7f535933c640 10 monclient: dump: 2024-09-16T03:47:05.346 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:47:05.346 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:05.346 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:05.346 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:05.346 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:47:05.346 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:05.346 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:47:05.346 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:47:05.346 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:47:05.347 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:05.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.336+0000 7f535c5a2640 10 monclient: shutdown 2024-09-16T03:47:05.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.336+0000 7f535c5a2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:47:05.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.336+0000 7f535c5a2640 1 -- 172.21.15.42:0/1160371715 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f53541319c0 msgr2=0x7f535413a280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:05.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.336+0000 7f535c5a2640 1 --2- 172.21.15.42:0/1160371715 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f53541319c0 0x7f535413a280 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f5354075b90 tx=0x7f534401c020 comp rx=0 tx=0).stop 2024-09-16T03:47:05.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.336+0000 7f535c5a2640 1 -- 172.21.15.42:0/1160371715 shutdown_connections 2024-09-16T03:47:05.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.336+0000 7f535c5a2640 1 --2- 172.21.15.42:0/1160371715 >> v2:172.21.15.42:3300/0 conn(0x7f535413a7c0 0x7f535413cbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:05.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.336+0000 7f535c5a2640 1 --2- 172.21.15.42:0/1160371715 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f53541319c0 0x7f535413a280 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:05.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.336+0000 7f535c5a2640 1 -- 172.21.15.42:0/1160371715 >> 172.21.15.42:0/1160371715 conn(0x7f53540a6640 msgr2=0x7f53540a2b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:47:05.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.336+0000 7f535c5a2640 1 -- 172.21.15.42:0/1160371715 shutdown_connections 2024-09-16T03:47:05.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.336+0000 7f535c5a2640 1 -- 172.21.15.42:0/1160371715 wait complete. 2024-09-16T03:47:05.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.336+0000 7f535c5a2640 10 monclient: build_initial_monmap 2024-09-16T03:47:05.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.336+0000 7f535c5a2640 10 monclient: monmap: 2024-09-16T03:47:05.348 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:47:05.348 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:47:05.348 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:47:05.348 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:47:05.348 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:47:05.348 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:05.348 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:47:05.349 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:47:05.349 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:47:05.349 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:05.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.336+0000 7f535c5a2640 1 librados: starting msgr at 2024-09-16T03:47:05.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.336+0000 7f535c5a2640 1 librados: starting objecter 2024-09-16T03:47:05.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.336+0000 7f535c5a2640 1 Processor -- start 2024-09-16T03:47:05.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.336+0000 7f535c5a2640 1 -- start start 2024-09-16T03:47:05.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.336+0000 7f535c5a2640 1 librados: setting wanted keys 2024-09-16T03:47:05.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.336+0000 7f535c5a2640 1 librados: calling monclient init 2024-09-16T03:47:05.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.336+0000 7f535c5a2640 10 monclient: init 2024-09-16T03:47:05.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.336+0000 7f535c5a2640 10 monclient: _reopen_session rank -1 2024-09-16T03:47:05.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.336+0000 7f535c5a2640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:47:05.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.336+0000 7f535c5a2640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f53541319c0 0x7f535412edf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:05.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.336+0000 7f535c5a2640 10 monclient(hunting): picked mon.noname-c con 0x7f53541319c0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:47:05.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.336+0000 7f535c5a2640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5354131df0 0x7f535412f330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:05.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.336+0000 7f535c5a2640 10 monclient(hunting): picked mon.noname-a con 0x7f5354131df0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:47:05.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.336+0000 7f535c5a2640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f535413a7c0 0x7f5354132ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:05.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.336+0000 7f535c5a2640 10 monclient(hunting): picked mon.noname-b con 0x7f535413a7c0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:47:05.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.336+0000 7f535c5a2640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:05.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.336+0000 7f535c5a2640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f53541404c0 con 0x7f5354131df0 2024-09-16T03:47:05.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.336+0000 7f535c5a2640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:05.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.336+0000 7f535c5a2640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f5354140340 con 0x7f535413a7c0 2024-09-16T03:47:05.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.336+0000 7f535c5a2640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:05.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.336+0000 7f535c5a2640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f5354140640 con 0x7f53541319c0 2024-09-16T03:47:05.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.336+0000 7f535c5a2640 10 monclient(hunting): _renew_subs 2024-09-16T03:47:05.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.336+0000 7f535c5a2640 10 monclient(hunting): authenticate will time out at 1272.306857s 2024-09-16T03:47:05.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.336+0000 7f5359b3d640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5354131df0 0x7f535412f330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:05.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.336+0000 7f5359b3d640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5354131df0 0x7f535412f330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:49744/0 (socket says 172.21.15.42:49744) 2024-09-16T03:47:05.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.336+0000 7f5359b3d640 1 -- 172.21.15.42:0/144670885 learned_addr learned my addr 172.21.15.42:0/144670885 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:47:05.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.336+0000 7f5359b3d640 10 monclient(hunting): get_auth_request con 0x7f5354131df0 auth_method 0 2024-09-16T03:47:05.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.336+0000 7f5359b3d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:05.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.336+0000 7f5359b3d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:05.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.336+0000 7f5359b3d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:47:05.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.336+0000 7f535a33e640 1 --2- 172.21.15.42:0/144670885 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f53541319c0 0x7f535412edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:05.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.336+0000 7f5359b3d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:47:05.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.336+0000 7f5359b3d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:47:05.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.336+0000 7f5359b3d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:47:05.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.336+0000 7f535a33e640 10 monclient(hunting): get_auth_request con 0x7f53541319c0 auth_method 0 2024-09-16T03:47:05.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.336+0000 7f535a33e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:05.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.336+0000 7f535a33e640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:05.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.336+0000 7f535a33e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:47:05.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.336+0000 7f535ab3f640 1 --2- 172.21.15.42:0/144670885 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f535413a7c0 0x7f5354132ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:05.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.336+0000 7f535ab3f640 10 monclient(hunting): get_auth_request con 0x7f535413a7c0 auth_method 0 2024-09-16T03:47:05.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.336+0000 7f535ab3f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:05.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.336+0000 7f535ab3f640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:05.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.336+0000 7f535ab3f640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:47:05.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.336+0000 7f535a33e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:47:05.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.336+0000 7f535a33e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:47:05.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.336+0000 7f535a33e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:47:05.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.336+0000 7f535ab3f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:47:05.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.336+0000 7f535ab3f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:47:05.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.336+0000 7f535ab3f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:47:05.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.336+0000 7f5359b3d640 10 monclient(hunting): handle_auth_done global_id 764172 payload 824 2024-09-16T03:47:05.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.336+0000 7f5359b3d640 1 -- 172.21.15.42:0/144670885 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f53541319c0 msgr2=0x7f535412edf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:05.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.336+0000 7f5359b3d640 1 --2- 172.21.15.42:0/144670885 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f53541319c0 0x7f535412edf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:05.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.336+0000 7f5359b3d640 1 -- 172.21.15.42:0/144670885 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f535413a7c0 msgr2=0x7f5354132ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:05.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.336+0000 7f5359b3d640 1 --2- 172.21.15.42:0/144670885 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f535413a7c0 0x7f5354132ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:05.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.336+0000 7f5359b3d640 10 monclient: _finish_hunting 0 2024-09-16T03:47:05.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.336+0000 7f5359b3d640 1 monclient: found mon.noname-a 2024-09-16T03:47:05.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.336+0000 7f5359b3d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:47:05.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.336+0000 7f5359b3d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:47:05.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.336+0000 7f5359b3d640 1 -- 172.21.15.42:0/144670885 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5354136320 con 0x7f5354131df0 2024-09-16T03:47:05.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.336+0000 7f535a33e640 1 --2- 172.21.15.42:0/144670885 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f53541319c0 0x7f535412edf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:47:05.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.336+0000 7f5359b3d640 1 --2- 172.21.15.42:0/144670885 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5354131df0 0x7f535412f330 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f53440af040 tx=0x7f53440190e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:47:05.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.336+0000 7f534affd640 10 client.?.objecter ms_handle_connect 0x7f5354131df0 2024-09-16T03:47:05.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.336+0000 7f534affd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:47:05.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.344+0000 7f534affd640 1 -- 172.21.15.42:0/144670885 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53440b9020 con 0x7f5354131df0 2024-09-16T03:47:05.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.344+0000 7f534affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:47:05.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.344+0000 7f534affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:47:05.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.344+0000 7f534affd640 10 monclient: dump: 2024-09-16T03:47:05.358 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:47:05.358 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:05.358 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:05.359 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:05.359 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:47:05.359 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:05.359 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:47:05.359 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:47:05.359 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:47:05.359 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:05.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.344+0000 7f534affd640 10 monclient: _finish_auth 0 2024-09-16T03:47:05.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.344+0000 7f534affd640 10 monclient: _check_auth_tickets 2024-09-16T03:47:05.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.344+0000 7f534affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:47:05.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.344+0000 7f534affd640 1 -- 172.21.15.42:0/144670885 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f53440ac640 con 0x7f5354131df0 2024-09-16T03:47:05.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.344+0000 7f535c5a2640 5 monclient: authenticate success, global_id 764172 2024-09-16T03:47:05.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.344+0000 7f535c5a2640 10 monclient: _renew_subs 2024-09-16T03:47:05.360 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.344+0000 7f535c5a2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:47:05.360 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.344+0000 7f535c5a2640 1 -- 172.21.15.42:0/144670885 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5354136600 con 0x7f5354131df0 2024-09-16T03:47:05.360 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.344+0000 7f534affd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:47:05.360 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.344+0000 7f534affd640 1 -- 172.21.15.42:0/144670885 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53440b1040 con 0x7f5354131df0 2024-09-16T03:47:05.360 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.344+0000 7f534affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:47:05.360 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.344+0000 7f534affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:47:05.360 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.344+0000 7f534affd640 10 monclient: dump: 2024-09-16T03:47:05.360 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:47:05.360 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:05.360 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:05.360 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:05.360 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:47:05.360 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:05.361 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:47:05.361 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:47:05.361 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:47:05.361 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:05.361 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.344+0000 7f535c5a2640 10 client.764172.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:47:05.361 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.344+0000 7f535c5a2640 10 monclient: _renew_subs 2024-09-16T03:47:05.361 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.344+0000 7f535c5a2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:47:05.361 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.344+0000 7f5358b3b640 4 set_mon_vals no callback set 2024-09-16T03:47:05.361 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.344+0000 7f535c5a2640 1 -- 172.21.15.42:0/144670885 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53541310b0 con 0x7f5354131df0 2024-09-16T03:47:05.361 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.344+0000 7f535c5a2640 1 librados: init done 2024-09-16T03:47:05.361 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.344+0000 7f535c5a2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:47:05.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.344+0000 7f535c5a2640 1 --2- 172.21.15.42:0/144670885 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f53541337d0 0x7f535402c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:05.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.344+0000 7f535c5a2640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:05.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.344+0000 7f535c5a2640 1 -- 172.21.15.42:0/144670885 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f53541321d0 con 0x7f53541337d0 2024-09-16T03:47:05.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.344+0000 7f534affd640 1 -- 172.21.15.42:0/144670885 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f53440bc020 con 0x7f5354131df0 2024-09-16T03:47:05.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.344+0000 7f535a33e640 1 --2- 172.21.15.42:0/144670885 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f53541337d0 0x7f535402c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:05.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.344+0000 7f535a33e640 10 monclient: get_auth_request con 0x7f53541337d0 auth_method 0 2024-09-16T03:47:05.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.344+0000 7f535a33e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:05.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.344+0000 7f535a33e640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:05.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.344+0000 7f535a33e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:47:05.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.344+0000 7f535a33e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:47:05.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.344+0000 7f535a33e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:47:05.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.344+0000 7f535a33e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:47:05.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.344+0000 7f534affd640 1 --2- 172.21.15.42:0/144670885 >> v2:172.21.15.154:6812/254845342 conn(0x7f5338037810 0x7f5338039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:05.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.344+0000 7f534affd640 1 -- 172.21.15.42:0/144670885 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(150..150 src has 1..150) ==== 5993+0+0 (secure 0 0 0) 0x7f53440e23e0 con 0x7f5354131df0 2024-09-16T03:47:05.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.344+0000 7f534affd640 10 client.764172.objecter ms_dispatch 0x7f5354000ba0 osd_map(150..150 src has 1..150) 2024-09-16T03:47:05.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.344+0000 7f534affd640 3 client.764172.objecter handle_osd_map got epochs [150,150] > 0 2024-09-16T03:47:05.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.344+0000 7f534affd640 3 client.764172.objecter handle_osd_map decoding full epoch 150 2024-09-16T03:47:05.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.344+0000 7f535a33e640 10 monclient(hunting): handle_auth_done global_id 754318 payload 824 2024-09-16T03:47:05.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.344+0000 7f534affd640 20 client.764172.objecter dump_active .. 0 homeless 2024-09-16T03:47:05.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.344+0000 7f535a33e640 1 -- 172.21.15.42:0/144670885 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f53541369a0 con 0x7f53541337d0 2024-09-16T03:47:05.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.344+0000 7f535ab3f640 1 --2- 172.21.15.42:0/144670885 >> v2:172.21.15.154:6812/254845342 conn(0x7f5338037810 0x7f5338039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:05.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.344+0000 7f535ab3f640 10 monclient: get_auth_request con 0x7f5338037810 auth_method 0 2024-09-16T03:47:05.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.344+0000 7f534affd640 10 client.764172.objecter ms_handle_connect 0x7f53541337d0 2024-09-16T03:47:05.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.344+0000 7f534affd640 10 client.764172.objecter resend_mon_ops 2024-09-16T03:47:05.364 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.344+0000 7f535a33e640 1 --2- 172.21.15.42:0/144670885 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f53541337d0 0x7f535402c650 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f5350001850 tx=0x7f5350001c50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:47:05.364 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.348+0000 7f534affd640 1 -- 172.21.15.42:0/144670885 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f535000f9e0 con 0x7f53541337d0 2024-09-16T03:47:05.364 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.348+0000 7f534affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:47:05.364 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.348+0000 7f534affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:47:05.364 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.348+0000 7f534affd640 10 monclient: dump: 2024-09-16T03:47:05.364 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:47:05.364 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:05.364 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:05.364 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:05.364 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:47:05.364 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:05.364 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:47:05.364 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:47:05.365 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:47:05.365 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:05.365 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.348+0000 7f535ab3f640 1 --2- 172.21.15.42:0/144670885 >> v2:172.21.15.154:6812/254845342 conn(0x7f5338037810 0x7f5338039cd0 secure :-1 s=READY pgs=3438 cs=0 l=1 rev1=1 crypto rx=0x7f53541342e0 tx=0x7f534c00b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:47:05.365 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.348+0000 7f534affd640 10 client.764172.objecter ms_handle_connect 0x7f5338037810 2024-09-16T03:47:05.365 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.348+0000 7f534affd640 1 -- 172.21.15.42:0/144670885 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f5350018070 con 0x7f53541337d0 2024-09-16T03:47:05.365 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.348+0000 7f534affd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:47:05.365 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.348+0000 7f534affd640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:47:05.365 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.348+0000 7f534affd640 1 -- 172.21.15.42:0/144670885 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f53541337d0 msgr2=0x7f535402c650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:05.366 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.348+0000 7f534affd640 1 --2- 172.21.15.42:0/144670885 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f53541337d0 0x7f535402c650 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f5350001850 tx=0x7f5350001c50 comp rx=0 tx=0).stop 2024-09-16T03:47:05.366 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.348+0000 7f534affd640 1 -- 172.21.15.42:0/144670885 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f53541337d0 msgr2=0x7f535402c650 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:47:05.366 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.348+0000 7f534affd640 1 --2- 172.21.15.42:0/144670885 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f53541337d0 0x7f535402c650 secure :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f5350001850 tx=0x7f5350001c50 comp rx=0 tx=0).stop 2024-09-16T03:47:05.366 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.352+0000 7f535c5a2640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:47:05.366 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.352+0000 7f535c5a2640 1 --2- 172.21.15.42:0/144670885 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5354133bb0 0x7f535402cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:05.366 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.352+0000 7f535c5a2640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:05.366 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.352+0000 7f535c5a2640 1 -- 172.21.15.42:0/144670885 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f53541372c0 con 0x7f5354133bb0 2024-09-16T03:47:05.366 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.352+0000 7f535a33e640 1 --2- 172.21.15.42:0/144670885 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5354133bb0 0x7f535402cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:05.366 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.352+0000 7f535a33e640 10 monclient: get_auth_request con 0x7f5354133bb0 auth_method 0 2024-09-16T03:47:05.366 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.352+0000 7f535a33e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:05.366 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.352+0000 7f535a33e640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:05.367 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.352+0000 7f535a33e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:47:05.367 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.352+0000 7f535a33e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:47:05.367 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.352+0000 7f535a33e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:47:05.367 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.352+0000 7f535a33e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:47:05.367 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.352+0000 7f535a33e640 10 monclient(hunting): handle_auth_done global_id 754321 payload 824 2024-09-16T03:47:05.367 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.352+0000 7f535a33e640 1 -- 172.21.15.42:0/144670885 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f535406ff80 con 0x7f5354133bb0 2024-09-16T03:47:05.367 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.352+0000 7f535a33e640 1 --2- 172.21.15.42:0/144670885 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5354133bb0 0x7f535402cb90 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f5350001850 tx=0x7f5350000ee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:47:05.367 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.352+0000 7f534affd640 10 client.764172.objecter ms_handle_connect 0x7f5354133bb0 2024-09-16T03:47:05.367 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.352+0000 7f534affd640 10 client.764172.objecter resend_mon_ops 2024-09-16T03:47:05.367 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.352+0000 7f534affd640 1 -- 172.21.15.42:0/144670885 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5350027410 con 0x7f5354133bb0 2024-09-16T03:47:05.367 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.352+0000 7f534affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:47:05.367 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.352+0000 7f534affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:47:05.367 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.352+0000 7f534affd640 10 monclient: dump: 2024-09-16T03:47:05.368 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:47:05.368 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:05.368 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:05.368 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:05.368 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:47:05.368 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:05.368 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:47:05.368 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:47:05.368 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:47:05.368 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:05.368 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.352+0000 7f534affd640 1 -- 172.21.15.42:0/144670885 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4711 (secure 0 0 0) 0x7f53500133f0 con 0x7f5354133bb0 2024-09-16T03:47:05.369 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.352+0000 7f534affd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:47:05.369 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.352+0000 7f534affd640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:47:05.369 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.352+0000 7f534affd640 1 -- 172.21.15.42:0/144670885 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5354133bb0 msgr2=0x7f535402cb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:05.369 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.352+0000 7f534affd640 1 --2- 172.21.15.42:0/144670885 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5354133bb0 0x7f535402cb90 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f5350001850 tx=0x7f5350000ee0 comp rx=0 tx=0).stop 2024-09-16T03:47:05.369 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.352+0000 7f534affd640 1 -- 172.21.15.42:0/144670885 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5354133bb0 msgr2=0x7f535402cb90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:47:05.369 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.352+0000 7f534affd640 1 --2- 172.21.15.42:0/144670885 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5354133bb0 0x7f535402cb90 secure :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f5350001850 tx=0x7f5350000ee0 comp rx=0 tx=0).stop 2024-09-16T03:47:05.369 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:47:05.369 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:47:05.369 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:47:05.369 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T03:47:05.370 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 48, 2024-09-16T03:47:05.370 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 67375, 2024-09-16T03:47:05.370 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:47:05.370 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:47:05.370 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T03:47:05.370 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T03:47:05.370 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:47:05.370 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 3, 2024-09-16T03:47:05.370 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:47:05.370 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:47:05.370 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:47:05.370 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:47:05.371 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:47:05.371 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:47:05.371 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:47:05.371 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:47:05.371 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:47:05.371 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:47:05.371 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:47:05.371 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:47:05.371 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:47:05.371 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:47:05.371 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:47:05.371 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:47:05.371 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:47:05.371 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:47:05.372 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:47:05.372 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:47:05.372 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:47:05.372 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:47:05.372 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:47:05.372 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:47:05.372 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:47:05.372 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:47:05.373 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:47:05.373 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:47:05.373 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:47:05.373 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:05.373 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:47:05.373 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:47:05.373 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:47:05.373 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:47:05.373 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:47:05.373 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:47:05.373 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:47:05.373 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:47:05.373 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:47:05.374 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:47:05.374 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:47:05.374 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:47:05.374 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:47:05.374 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:47:05.374 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:47:05.374 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:47:05.374 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:47:05.374 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:47:05.374 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:47:05.374 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:47:05.374 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:47:05.375 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:47:05.375 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:47:05.375 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:47:05.375 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:47:05.375 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:47:05.375 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:47:05.375 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:47:05.375 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:47:05.375 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:47:05.375 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:05.375 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:47:05.375 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:05.375 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:47:05.375 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:47:05.376 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:47:05.376 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:47:05.376 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:05.376 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:47:05.376 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:47:05.376 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:47:05.376 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:05.376 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:05.376 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:47:05.376 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:47:05.376 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:47:05.376 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:47:05.376 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:47:05.377 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:05.377 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:47:05.377 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:47:05.377 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:47:05.377 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:47:05.377 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:47:05.377 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:05.377 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:05.377 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:47:05.377 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:47:05.377 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:47:05.377 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:47:05.377 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:05.378 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:47:05.378 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:47:05.378 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:47:05.378 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:05.378 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:05.378 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:47:05.378 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:47:05.378 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:47:05.378 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:47:05.378 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:47:05.378 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:05.378 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:47:05.378 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:47:05.379 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:47:05.379 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:47:05.379 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:47:05.379 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:05.379 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:05.379 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:47:05.379 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:47:05.379 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:47:05.379 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:47:05.379 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:05.379 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:47:05.379 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:47:05.379 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:47:05.379 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:05.380 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:05.380 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:47:05.380 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:47:05.380 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:47:05.380 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:47:05.380 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:47:05.380 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:05.380 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:47:05.380 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:47:05.380 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:47:05.380 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:47:05.380 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:47:05.380 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:47:05.381 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:47:05.381 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:05.381 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:47:05.381 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:47:05.381 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:05.381 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:47:05.381 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:47:05.381 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:47:05.381 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:47:05.381 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:47:05.381 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:47:05.381 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:05.381 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:47:05.381 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:47:05.382 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 4 2024-09-16T03:47:05.382 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:47:05.382 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:47:05.382 INFO:teuthology.orchestra.run.smithi042.stdout: "mgr": [ 2024-09-16T03:47:05.382 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:05.382 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:47:05.382 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:47:05.382 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:47:05.382 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:47:05.382 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:47:05.382 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:05.382 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:47:05.383 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:47:05.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.356+0000 7f535c5a2640 10 librados: watch_flush enter 2024-09-16T03:47:05.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.356+0000 7f535c5a2640 10 librados: watch_flush exit 2024-09-16T03:47:05.385 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.356+0000 7f535c5a2640 20 client.764172.objecter shutdown clearing up homeless session... 2024-09-16T03:47:05.385 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.356+0000 7f535c5a2640 10 client.764172.objecter successfully canceled tick 2024-09-16T03:47:05.385 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.356+0000 7f535c5a2640 1 -- 172.21.15.42:0/144670885 >> v2:172.21.15.154:6812/254845342 conn(0x7f5338037810 msgr2=0x7f5338039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:05.385 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.356+0000 7f535c5a2640 1 --2- 172.21.15.42:0/144670885 >> v2:172.21.15.154:6812/254845342 conn(0x7f5338037810 0x7f5338039cd0 secure :-1 s=READY pgs=3438 cs=0 l=1 rev1=1 crypto rx=0x7f53541342e0 tx=0x7f534c00b040 comp rx=0 tx=0).stop 2024-09-16T03:47:05.385 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.356+0000 7f535c5a2640 10 monclient: shutdown 2024-09-16T03:47:05.385 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.356+0000 7f535c5a2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:47:05.385 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.356+0000 7f535c5a2640 1 -- 172.21.15.42:0/144670885 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5354131df0 msgr2=0x7f535412f330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:05.385 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.356+0000 7f535c5a2640 1 --2- 172.21.15.42:0/144670885 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5354131df0 0x7f535412f330 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f53440af040 tx=0x7f53440190e0 comp rx=0 tx=0).stop 2024-09-16T03:47:05.385 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.356+0000 7f535ab3f640 1 -- 172.21.15.42:0/144670885 reap_dead start 2024-09-16T03:47:05.385 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.356+0000 7f535ab3f640 1 -- 172.21.15.42:0/144670885 reap_dead start 2024-09-16T03:47:05.385 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.356+0000 7f535c5a2640 1 -- 172.21.15.42:0/144670885 shutdown_connections 2024-09-16T03:47:05.385 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.356+0000 7f535c5a2640 1 -- 172.21.15.42:0/144670885 >> 172.21.15.42:0/144670885 conn(0x7f53540a6640 msgr2=0x7f53540a0940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:47:05.386 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.356+0000 7f535c5a2640 1 -- 172.21.15.42:0/144670885 shutdown_connections 2024-09-16T03:47:05.386 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.356+0000 7f535c5a2640 1 -- 172.21.15.42:0/144670885 wait complete. 2024-09-16T03:47:05.386 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.356+0000 7f535c5a2640 1 librados: shutdown 2024-09-16T03:47:05.386 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:47:05.386 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-09-16T03:47:05.386 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-16T03:47:05.386 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-09-16T03:47:05.387 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-16T03:47:05.387 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-16T03:47:05.387 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T03:47:05.409 INFO:tasks.ceph.mon.b:Stopped 2024-09-16T03:47:05.410 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T03:47:05.410 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:47:05.530 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.528+0000 7fe848983640 10 monclient: get_monmap_and_config 2024-09-16T03:47:05.530 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.528+0000 7fe848983640 10 monclient: build_initial_monmap 2024-09-16T03:47:05.530 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.528+0000 7fe848983640 10 monclient: monmap: 2024-09-16T03:47:05.530 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:47:05.530 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:05.530 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:47:05.529278+0000 2024-09-16T03:47:05.531 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:47:05.529278+0000 2024-09-16T03:47:05.531 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:47:05.531 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:05.531 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:47:05.531 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:47:05.531 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:47:05.531 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:47:05.531 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:47:05.531 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:05.533 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.528+0000 7fe848983640 1 Processor -- start 2024-09-16T03:47:05.533 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.528+0000 7fe848983640 1 -- start start 2024-09-16T03:47:05.534 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.528+0000 7fe848983640 10 monclient: init 2024-09-16T03:47:05.534 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.528+0000 7fe848983640 10 monclient: _reopen_session rank -1 2024-09-16T03:47:05.534 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.528+0000 7fe848983640 10 monclient: _add_conns ranks=[4,3,0,1,2] 2024-09-16T03:47:05.534 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.528+0000 7fe848983640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fe8440a39b0 0x7fe844133b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:05.534 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.528+0000 7fe848983640 10 monclient(hunting): picked mon.noname-c con 0x7fe8440a39b0 addr v2:172.21.15.154:3300/0 2024-09-16T03:47:05.534 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.528+0000 7fe848983640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe8440a3d90 0x7fe844137000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:05.534 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.528+0000 7fe848983640 10 monclient(hunting): picked mon.noname-a con 0x7fe8440a3d90 addr v2:172.21.15.42:3300/0 2024-09-16T03:47:05.534 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.528+0000 7fe848983640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe844137540 addr v1:172.21.15.42:6789/0 2024-09-16T03:47:05.534 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.528+0000 7fe848983640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe84409c140 con 0x7fe844137540 2024-09-16T03:47:05.534 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.528+0000 7fe848983640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:05.534 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.528+0000 7fe848983640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7fe84409c420 con 0x7fe8440a3d90 2024-09-16T03:47:05.534 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.528+0000 7fe848983640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:05.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.528+0000 7fe848983640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fe84406dd60 con 0x7fe8440a39b0 2024-09-16T03:47:05.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.528+0000 7fe848983640 10 monclient(hunting): _renew_subs 2024-09-16T03:47:05.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.528+0000 7fe848983640 10 monclient(hunting): authenticate will time out at 1272.499045s 2024-09-16T03:47:05.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.528+0000 7fe84259c640 1 -- >> v2:172.21.15.154:3300/0 conn(0x7fe8440a39b0 msgr2=0x7fe844133b80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T03:47:05.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.528+0000 7fe84259c640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fe8440a39b0 0x7fe844133b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:47:05.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.528+0000 7fe842d9d640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fe844137540 0x7fe84413e390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:46580/0 (socket says 172.21.15.42:46580) 2024-09-16T03:47:05.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.528+0000 7fe842d9d640 1 -- 172.21.15.42:0/2746264889 learned_addr learned my addr 172.21.15.42:0/2746264889 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:47:05.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.528+0000 7fe841d9b640 1 --2- 172.21.15.42:0/2746264889 >> v2:172.21.15.42:3300/0 conn(0x7fe8440a3d90 0x7fe844137000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:05.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.528+0000 7fe841d9b640 10 monclient(hunting): get_auth_request con 0x7fe8440a3d90 auth_method 0 2024-09-16T03:47:05.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.528+0000 7fe841d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:05.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.528+0000 7fe841d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:05.536 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.528+0000 7fe841d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:47:05.536 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.528+0000 7fe841d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:47:05.536 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.528+0000 7fe841d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:47:05.536 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.528+0000 7fe841d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:47:05.536 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.528+0000 7fe84159a640 1 -- 172.21.15.42:0/2746264889 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1636487936 0 0) 0x7fe84409c140 con 0x7fe844137540 2024-09-16T03:47:05.536 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.528+0000 7fe84159a640 10 monclient(hunting): _negotiate 2024-09-16T03:47:05.536 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.528+0000 7fe84159a640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:05.536 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.528+0000 7fe84159a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:47:05.536 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.528+0000 7fe84159a640 10 monclient(hunting): my global_id is 764178 2024-09-16T03:47:05.536 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.528+0000 7fe84159a640 1 -- 172.21.15.42:0/2746264889 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe830003530 con 0x7fe844137540 2024-09-16T03:47:05.536 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.528+0000 7fe841d9b640 10 monclient(hunting): handle_auth_done global_id 764175 payload 274 2024-09-16T03:47:05.536 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.528+0000 7fe841d9b640 1 -- 172.21.15.42:0/2746264889 >> v2:172.21.15.154:3300/0 conn(0x7fe8440a39b0 msgr2=0x7fe844133b80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:47:05.536 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.528+0000 7fe841d9b640 1 --2- 172.21.15.42:0/2746264889 >> v2:172.21.15.154:3300/0 conn(0x7fe8440a39b0 0x7fe844133b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:05.537 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.528+0000 7fe841d9b640 1 -- 172.21.15.42:0/2746264889 >> v1:172.21.15.42:6789/0 conn(0x7fe844137540 legacy=0x7fe84413e390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:05.537 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.528+0000 7fe841d9b640 10 monclient: _finish_hunting 0 2024-09-16T03:47:05.537 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.528+0000 7fe841d9b640 1 monclient: found mon.noname-a 2024-09-16T03:47:05.537 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.528+0000 7fe841d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:47:05.537 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.528+0000 7fe841d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:47:05.537 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.528+0000 7fe841d9b640 1 -- 172.21.15.42:0/2746264889 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8440a5e20 con 0x7fe8440a3d90 2024-09-16T03:47:05.542 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.532+0000 7fe841d9b640 1 --2- 172.21.15.42:0/2746264889 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe8440a3d90 0x7fe844137000 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fe8440a7750 tx=0x7fe82c01c360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76f141ffd684d640 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:47:05.542 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.532+0000 7fe84159a640 1 -- 172.21.15.42:0/2746264889 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe82c0b0070 con 0x7fe8440a3d90 2024-09-16T03:47:05.542 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.532+0000 7fe84159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:47:05.542 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.532+0000 7fe84159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:47:05.542 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.532+0000 7fe84159a640 10 monclient: dump: 2024-09-16T03:47:05.543 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:47:05.543 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:05.543 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:05.543 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:05.543 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:47:05.543 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:05.543 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:47:05.543 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:47:05.543 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:47:05.543 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:05.543 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.532+0000 7fe84159a640 10 monclient: _finish_auth 0 2024-09-16T03:47:05.543 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.532+0000 7fe84159a640 10 monclient: _check_auth_tickets 2024-09-16T03:47:05.543 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.532+0000 7fe84159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:47:05.543 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.532+0000 7fe84159a640 1 -- 172.21.15.42:0/2746264889 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe82c0ab600 con 0x7fe8440a3d90 2024-09-16T03:47:05.543 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.532+0000 7fe848983640 5 monclient: authenticate success, global_id 764175 2024-09-16T03:47:05.543 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.532+0000 7fe848983640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:47:05.544 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.532+0000 7fe84159a640 10 monclient: handle_config config(0 keys) 2024-09-16T03:47:05.544 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.532+0000 7fe848983640 10 monclient: get_monmap_and_config success 2024-09-16T03:47:05.544 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.532+0000 7fe848983640 4 set_mon_vals no callback set 2024-09-16T03:47:05.544 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.532+0000 7fe84159a640 1 -- 172.21.15.42:0/2746264889 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe82c0ab9d0 con 0x7fe8440a3d90 2024-09-16T03:47:05.544 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.532+0000 7fe84159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:47:05.544 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.532+0000 7fe84159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:47:05.544 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.532+0000 7fe84159a640 10 monclient: dump: 2024-09-16T03:47:05.544 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:47:05.544 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:05.544 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:05.544 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:05.544 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:47:05.544 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:05.544 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:47:05.544 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:47:05.545 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:47:05.545 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:05.545 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.532+0000 7fe848983640 10 monclient: shutdown 2024-09-16T03:47:05.545 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.532+0000 7fe848983640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:47:05.545 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.532+0000 7fe848983640 1 -- 172.21.15.42:0/2746264889 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe8440a3d90 msgr2=0x7fe844137000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:05.545 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.532+0000 7fe848983640 1 --2- 172.21.15.42:0/2746264889 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe8440a3d90 0x7fe844137000 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fe8440a7750 tx=0x7fe82c01c360 comp rx=0 tx=0).stop 2024-09-16T03:47:05.545 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.532+0000 7fe848983640 1 -- 172.21.15.42:0/2746264889 shutdown_connections 2024-09-16T03:47:05.545 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.532+0000 7fe848983640 1 --2- 172.21.15.42:0/2746264889 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe8440a3d90 0x7fe844137000 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:05.545 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.532+0000 7fe848983640 1 --2- 172.21.15.42:0/2746264889 >> v2:172.21.15.154:3300/0 conn(0x7fe8440a39b0 0x7fe844133b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:05.545 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.532+0000 7fe848983640 1 -- 172.21.15.42:0/2746264889 >> 172.21.15.42:0/2746264889 conn(0x7fe8440a9fb0 msgr2=0x7fe8440aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:47:05.545 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.532+0000 7fe848983640 1 -- 172.21.15.42:0/2746264889 shutdown_connections 2024-09-16T03:47:05.545 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.532+0000 7fe848983640 1 -- 172.21.15.42:0/2746264889 wait complete. 2024-09-16T03:47:05.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.532+0000 7fe848983640 10 monclient: build_initial_monmap 2024-09-16T03:47:05.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.532+0000 7fe848983640 10 monclient: monmap: 2024-09-16T03:47:05.546 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:47:05.546 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:47:05.546 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:47:05.546 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:47:05.546 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:47:05.546 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:05.546 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:47:05.546 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:47:05.546 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:47:05.546 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:05.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.532+0000 7fe848983640 1 librados: starting msgr at 2024-09-16T03:47:05.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.532+0000 7fe848983640 1 librados: starting objecter 2024-09-16T03:47:05.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.532+0000 7fe848983640 1 Processor -- start 2024-09-16T03:47:05.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.532+0000 7fe848983640 1 -- start start 2024-09-16T03:47:05.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.532+0000 7fe848983640 1 librados: setting wanted keys 2024-09-16T03:47:05.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.532+0000 7fe848983640 1 librados: calling monclient init 2024-09-16T03:47:05.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.532+0000 7fe848983640 10 monclient: init 2024-09-16T03:47:05.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.536+0000 7fe848983640 10 monclient: _reopen_session rank -1 2024-09-16T03:47:05.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.536+0000 7fe848983640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:47:05.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.536+0000 7fe848983640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe8440a39b0 0x7fe844130560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:05.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.536+0000 7fe848983640 10 monclient(hunting): picked mon.noname-c con 0x7fe8440a39b0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:47:05.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.536+0000 7fe848983640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe8440a3d90 0x7fe844130aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:05.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.536+0000 7fe848983640 10 monclient(hunting): picked mon.noname-b con 0x7fe8440a3d90 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:47:05.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.536+0000 7fe848983640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe844137540 0x7fe84413d130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:05.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.536+0000 7fe848983640 10 monclient(hunting): picked mon.noname-a con 0x7fe844137540 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:47:05.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.536+0000 7fe848983640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:05.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.536+0000 7fe848983640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fe8441424e0 con 0x7fe844137540 2024-09-16T03:47:05.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.536+0000 7fe848983640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:05.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.536+0000 7fe848983640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe844142660 con 0x7fe8440a3d90 2024-09-16T03:47:05.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.536+0000 7fe848983640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:05.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.536+0000 7fe848983640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fe8441427e0 con 0x7fe8440a39b0 2024-09-16T03:47:05.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.536+0000 7fe848983640 10 monclient(hunting): _renew_subs 2024-09-16T03:47:05.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.536+0000 7fe848983640 10 monclient(hunting): authenticate will time out at 1272.504334s 2024-09-16T03:47:05.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.536+0000 7fe842d9d640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe844137540 0x7fe84413d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:05.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.536+0000 7fe84259c640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe8440a39b0 0x7fe844130560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:05.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.536+0000 7fe842d9d640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe844137540 0x7fe84413d130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:49750/0 (socket says 172.21.15.42:49750) 2024-09-16T03:47:05.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.536+0000 7fe84259c640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe8440a39b0 0x7fe844130560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:51624/0 (socket says 172.21.15.42:51624) 2024-09-16T03:47:05.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.536+0000 7fe841d9b640 1 -- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe8440a3d90 msgr2=0x7fe844130aa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T03:47:05.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.536+0000 7fe842d9d640 1 -- 172.21.15.42:0/421050704 learned_addr learned my addr 172.21.15.42:0/421050704 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:47:05.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.536+0000 7fe842d9d640 10 monclient(hunting): get_auth_request con 0x7fe844137540 auth_method 0 2024-09-16T03:47:05.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.536+0000 7fe842d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:05.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.536+0000 7fe842d9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:05.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.536+0000 7fe842d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:47:05.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.536+0000 7fe841d9b640 1 --2- 172.21.15.42:0/421050704 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe8440a3d90 0x7fe844130aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:47:05.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.536+0000 7fe84259c640 10 monclient(hunting): get_auth_request con 0x7fe8440a39b0 auth_method 0 2024-09-16T03:47:05.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.536+0000 7fe84259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:05.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.536+0000 7fe84259c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:05.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.536+0000 7fe84259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:47:05.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.536+0000 7fe842d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:47:05.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.536+0000 7fe842d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:47:05.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.536+0000 7fe842d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:47:05.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.536+0000 7fe84259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:47:05.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.536+0000 7fe84259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:47:05.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.536+0000 7fe84259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:47:05.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.536+0000 7fe842d9d640 10 monclient(hunting): handle_auth_done global_id 764181 payload 824 2024-09-16T03:47:05.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.536+0000 7fe842d9d640 1 -- 172.21.15.42:0/421050704 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe8440a39b0 msgr2=0x7fe844130560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:05.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.536+0000 7fe842d9d640 1 --2- 172.21.15.42:0/421050704 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe8440a39b0 0x7fe844130560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:05.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.536+0000 7fe842d9d640 1 -- 172.21.15.42:0/421050704 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe8440a3d90 msgr2=0x7fe844130aa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:47:05.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.536+0000 7fe842d9d640 1 --2- 172.21.15.42:0/421050704 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe8440a3d90 0x7fe844130aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:05.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.536+0000 7fe842d9d640 10 monclient: _finish_hunting 0 2024-09-16T03:47:05.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.536+0000 7fe842d9d640 1 monclient: found mon.noname-a 2024-09-16T03:47:05.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.536+0000 7fe842d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:47:05.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.536+0000 7fe842d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:47:05.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.536+0000 7fe842d9d640 1 -- 172.21.15.42:0/421050704 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe84409e210 con 0x7fe844137540 2024-09-16T03:47:05.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.536+0000 7fe842d9d640 1 --2- 172.21.15.42:0/421050704 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe844137540 0x7fe84413d130 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fe83c00cb40 tx=0x7fe83c00a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:47:05.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.536+0000 7fe83affd640 10 client.?.objecter ms_handle_connect 0x7fe844137540 2024-09-16T03:47:05.551 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.536+0000 7fe83affd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:47:05.551 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.536+0000 7fe83affd640 1 -- 172.21.15.42:0/421050704 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe83c017070 con 0x7fe844137540 2024-09-16T03:47:05.551 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.536+0000 7fe83affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:47:05.551 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.536+0000 7fe83affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:47:05.551 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.536+0000 7fe83affd640 10 monclient: dump: 2024-09-16T03:47:05.551 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:47:05.551 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:05.551 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:05.551 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:05.551 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:47:05.551 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:05.551 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:47:05.552 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:47:05.552 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:47:05.552 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:05.552 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.536+0000 7fe83affd640 10 monclient: _finish_auth 0 2024-09-16T03:47:05.552 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.536+0000 7fe83affd640 10 monclient: _check_auth_tickets 2024-09-16T03:47:05.552 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.536+0000 7fe83affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:47:05.552 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.536+0000 7fe83affd640 1 -- 172.21.15.42:0/421050704 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe83c00aca0 con 0x7fe844137540 2024-09-16T03:47:05.552 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.536+0000 7fe848983640 5 monclient: authenticate success, global_id 764181 2024-09-16T03:47:05.552 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.536+0000 7fe848983640 10 monclient: _renew_subs 2024-09-16T03:47:05.552 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.536+0000 7fe848983640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:47:05.552 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.536+0000 7fe848983640 1 -- 172.21.15.42:0/421050704 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe84409d040 con 0x7fe844137540 2024-09-16T03:47:05.552 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.536+0000 7fe83affd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:47:05.552 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.536+0000 7fe83affd640 1 -- 172.21.15.42:0/421050704 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe83c012620 con 0x7fe844137540 2024-09-16T03:47:05.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.536+0000 7fe83affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:47:05.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.536+0000 7fe83affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:47:05.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.536+0000 7fe83affd640 10 monclient: dump: 2024-09-16T03:47:05.553 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:47:05.553 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:05.553 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:05.553 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:05.553 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:47:05.553 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:05.553 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:47:05.553 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:47:05.553 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:47:05.553 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:05.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.536+0000 7fe83bfff640 4 set_mon_vals no callback set 2024-09-16T03:47:05.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.536+0000 7fe848983640 10 client.764181.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:47:05.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.536+0000 7fe848983640 10 monclient: _renew_subs 2024-09-16T03:47:05.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.536+0000 7fe848983640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:47:05.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.536+0000 7fe848983640 1 -- 172.21.15.42:0/421050704 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe84409df30 con 0x7fe844137540 2024-09-16T03:47:05.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.536+0000 7fe848983640 1 librados: init done 2024-09-16T03:47:05.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.536+0000 7fe848983640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:47:05.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.536+0000 7fe848983640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:47:05.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.536+0000 7fe848983640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:47:05.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.536+0000 7fe83affd640 1 -- 172.21.15.42:0/421050704 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fe83c01a070 con 0x7fe844137540 2024-09-16T03:47:05.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.536+0000 7fe848983640 1 -- 172.21.15.42:0/421050704 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe844133b80 con 0x7fe844137540 2024-09-16T03:47:05.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.536+0000 7fe83affd640 1 --2- 172.21.15.42:0/421050704 >> v2:172.21.15.154:6812/254845342 conn(0x7fe8240377c0 0x7fe824039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:05.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.536+0000 7fe83affd640 1 -- 172.21.15.42:0/421050704 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(150..150 src has 1..150) ==== 5993+0+0 (secure 0 0 0) 0x7fe83c048e10 con 0x7fe844137540 2024-09-16T03:47:05.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.536+0000 7fe83affd640 10 client.764181.objecter ms_dispatch 0x7fe844000ba0 osd_map(150..150 src has 1..150) 2024-09-16T03:47:05.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.536+0000 7fe83affd640 3 client.764181.objecter handle_osd_map got epochs [150,150] > 0 2024-09-16T03:47:05.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.536+0000 7fe83affd640 3 client.764181.objecter handle_osd_map decoding full epoch 150 2024-09-16T03:47:05.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.536+0000 7fe83affd640 20 client.764181.objecter dump_active .. 0 homeless 2024-09-16T03:47:05.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.536+0000 7fe84259c640 1 --2- 172.21.15.42:0/421050704 >> v2:172.21.15.154:6812/254845342 conn(0x7fe8240377c0 0x7fe824039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:05.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.536+0000 7fe84259c640 10 monclient: get_auth_request con 0x7fe8240377c0 auth_method 0 2024-09-16T03:47:05.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.536+0000 7fe83affd640 10 client.764181.objecter ms_handle_connect 0x7fe8240377c0 2024-09-16T03:47:05.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.536+0000 7fe84259c640 1 --2- 172.21.15.42:0/421050704 >> v2:172.21.15.154:6812/254845342 conn(0x7fe8240377c0 0x7fe824039c80 secure :-1 s=READY pgs=3439 cs=0 l=1 rev1=1 crypto rx=0x7fe8340093d0 tx=0x7fe834009360 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:47:05.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.540+0000 7fe83affd640 1 -- 172.21.15.42:0/421050704 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7fe83c0079c0 con 0x7fe844137540 2024-09-16T03:47:05.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.540+0000 7fe83affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:47:05.556 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.540+0000 7fe83affd640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:47:05.771 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.768+0000 7fe848983640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:47:05.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.768+0000 7fe848983640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:47:05.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.768+0000 7fe848983640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:47:05.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.768+0000 7fe848983640 1 -- 172.21.15.42:0/421050704 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fe8440a5750 con 0x7fe844137540 2024-09-16T03:47:05.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.768+0000 7fe83affd640 1 -- 172.21.15.42:0/421050704 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7fe83c0074a0 con 0x7fe844137540 2024-09-16T03:47:05.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.768+0000 7fe83affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:47:05.773 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.768+0000 7fe83affd640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:47:05.773 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":48,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:47:05.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.772+0000 7fe848983640 10 librados: watch_flush enter 2024-09-16T03:47:05.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.772+0000 7fe848983640 10 librados: watch_flush exit 2024-09-16T03:47:05.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.772+0000 7fe848983640 20 client.764181.objecter shutdown clearing up homeless session... 2024-09-16T03:47:05.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.772+0000 7fe848983640 10 client.764181.objecter successfully canceled tick 2024-09-16T03:47:05.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.772+0000 7fe848983640 1 -- 172.21.15.42:0/421050704 >> v2:172.21.15.154:6812/254845342 conn(0x7fe8240377c0 msgr2=0x7fe824039c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:05.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.772+0000 7fe848983640 1 --2- 172.21.15.42:0/421050704 >> v2:172.21.15.154:6812/254845342 conn(0x7fe8240377c0 0x7fe824039c80 secure :-1 s=READY pgs=3439 cs=0 l=1 rev1=1 crypto rx=0x7fe8340093d0 tx=0x7fe834009360 comp rx=0 tx=0).stop 2024-09-16T03:47:05.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.772+0000 7fe848983640 10 monclient: shutdown 2024-09-16T03:47:05.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.772+0000 7fe848983640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:47:05.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.772+0000 7fe848983640 1 -- 172.21.15.42:0/421050704 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe844137540 msgr2=0x7fe84413d130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:05.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.772+0000 7fe848983640 1 --2- 172.21.15.42:0/421050704 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe844137540 0x7fe84413d130 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fe83c00cb40 tx=0x7fe83c00a060 comp rx=0 tx=0).stop 2024-09-16T03:47:05.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.772+0000 7fe848983640 1 -- 172.21.15.42:0/421050704 shutdown_connections 2024-09-16T03:47:05.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.772+0000 7fe848983640 1 --2- 172.21.15.42:0/421050704 >> v2:172.21.15.154:6812/254845342 conn(0x7fe8240377c0 0x7fe824039c80 unknown :-1 s=CLOSED pgs=3439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:05.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.772+0000 7fe848983640 1 --2- 172.21.15.42:0/421050704 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe844137540 0x7fe84413d130 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:05.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.772+0000 7fe848983640 1 --2- 172.21.15.42:0/421050704 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe8440a3d90 0x7fe844130aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:05.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.772+0000 7fe848983640 1 --2- 172.21.15.42:0/421050704 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe8440a39b0 0x7fe844130560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:05.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.772+0000 7fe848983640 1 -- 172.21.15.42:0/421050704 >> 172.21.15.42:0/421050704 conn(0x7fe8440a9fb0 msgr2=0x7fe8440a93f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:47:05.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.772+0000 7fe848983640 1 -- 172.21.15.42:0/421050704 shutdown_connections 2024-09-16T03:47:05.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.772+0000 7fe848983640 1 -- 172.21.15.42:0/421050704 wait complete. 2024-09-16T03:47:05.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:05.772+0000 7fe848983640 1 librados: shutdown 2024-09-16T03:47:08.792 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:47:08.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.924+0000 7fd357bfb640 10 monclient: get_monmap_and_config 2024-09-16T03:47:08.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.924+0000 7fd357bfb640 10 monclient: build_initial_monmap 2024-09-16T03:47:08.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.924+0000 7fd357bfb640 10 monclient: monmap: 2024-09-16T03:47:08.927 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:47:08.927 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:08.927 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:47:08.925521+0000 2024-09-16T03:47:08.927 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:47:08.925521+0000 2024-09-16T03:47:08.927 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:47:08.927 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:08.927 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:47:08.927 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:47:08.927 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:47:08.927 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:47:08.927 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:47:08.927 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:08.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.924+0000 7fd357bfb640 1 Processor -- start 2024-09-16T03:47:08.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.924+0000 7fd357bfb640 1 -- start start 2024-09-16T03:47:08.938 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.924+0000 7fd357bfb640 10 monclient: init 2024-09-16T03:47:08.938 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.924+0000 7fd357bfb640 10 monclient: _reopen_session rank -1 2024-09-16T03:47:08.938 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.924+0000 7fd357bfb640 10 monclient: _add_conns ranks=[0,4,1,2,3] 2024-09-16T03:47:08.938 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.924+0000 7fd357bfb640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd3500a1330 addr v1:172.21.15.42:6789/0 2024-09-16T03:47:08.938 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.924+0000 7fd357bfb640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fd35006dac0 0x7fd35013a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:08.938 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.924+0000 7fd357bfb640 10 monclient(hunting): picked mon.noname-c con 0x7fd35006dac0 addr v2:172.21.15.154:3300/0 2024-09-16T03:47:08.938 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.924+0000 7fd357bfb640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd3500a1e20 addr v1:172.21.15.154:6789/0 2024-09-16T03:47:08.938 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.924+0000 7fd357bfb640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd35009c090 con 0x7fd3500a1330 2024-09-16T03:47:08.938 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.924+0000 7fd357bfb640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd35009f610 con 0x7fd3500a1e20 2024-09-16T03:47:08.938 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.924+0000 7fd357bfb640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:08.938 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.924+0000 7fd357bfb640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fd35009f8f0 con 0x7fd35006dac0 2024-09-16T03:47:08.938 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.924+0000 7fd357bfb640 10 monclient(hunting): _renew_subs 2024-09-16T03:47:08.938 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.924+0000 7fd357bfb640 10 monclient(hunting): authenticate will time out at 1275.895076s 2024-09-16T03:47:08.938 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.924+0000 7fd356198640 1 -- >> v1:172.21.15.154:6789/0 conn(0x7fd3500a1e20 legacy=0x7fd35013c890 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.154:6789/0 2024-09-16T03:47:08.939 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.924+0000 7fd355196640 1 -- >> v2:172.21.15.154:3300/0 conn(0x7fd35006dac0 msgr2=0x7fd35013a340 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T03:47:08.939 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.924+0000 7fd355997640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fd3500a1330 0x7fd3500a1710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:46586/0 (socket says 172.21.15.42:46586) 2024-09-16T03:47:08.939 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.924+0000 7fd355997640 1 -- 172.21.15.42:0/159606189 learned_addr learned my addr 172.21.15.42:0/159606189 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:47:08.939 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.924+0000 7fd355196640 1 --2- 172.21.15.42:0/159606189 >> v2:172.21.15.154:3300/0 conn(0x7fd35006dac0 0x7fd35013a340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:47:08.939 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.924+0000 7fd354995640 1 -- 172.21.15.42:0/159606189 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 845608884 0 0) 0x7fd35009c090 con 0x7fd3500a1330 2024-09-16T03:47:08.939 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.924+0000 7fd354995640 10 monclient(hunting): _negotiate 2024-09-16T03:47:08.939 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.924+0000 7fd354995640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:08.939 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.924+0000 7fd354995640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:47:08.939 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.924+0000 7fd354995640 10 monclient(hunting): my global_id is 764184 2024-09-16T03:47:08.939 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.924+0000 7fd354995640 1 -- 172.21.15.42:0/159606189 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd33c003530 con 0x7fd3500a1330 2024-09-16T03:47:08.939 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.924+0000 7fd354995640 1 -- 172.21.15.42:0/159606189 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2816552081 0 0) 0x7fd33c003530 con 0x7fd3500a1330 2024-09-16T03:47:08.939 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.924+0000 7fd354995640 1 -- 172.21.15.42:0/159606189 >> v2:172.21.15.154:3300/0 conn(0x7fd35006dac0 msgr2=0x7fd35013a340 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:47:08.940 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.924+0000 7fd354995640 1 --2- 172.21.15.42:0/159606189 >> v2:172.21.15.154:3300/0 conn(0x7fd35006dac0 0x7fd35013a340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:08.940 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.924+0000 7fd354995640 1 -- 172.21.15.42:0/159606189 >> v1:172.21.15.154:6789/0 conn(0x7fd3500a1e20 legacy=0x7fd35013c890 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:47:08.940 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.924+0000 7fd354995640 10 monclient: _finish_hunting 0 2024-09-16T03:47:08.940 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.924+0000 7fd354995640 1 monclient: found mon.noname-a-legacy 2024-09-16T03:47:08.940 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.924+0000 7fd354995640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:47:08.940 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.924+0000 7fd354995640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.42:6789/0 2024-09-16T03:47:08.940 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.924+0000 7fd354995640 1 -- 172.21.15.42:0/159606189 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd35009fbd0 con 0x7fd3500a1330 2024-09-16T03:47:08.940 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.924+0000 7fd354995640 10 monclient: _finish_auth 0 2024-09-16T03:47:08.940 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.928+0000 7fd354995640 10 monclient: _check_auth_tickets 2024-09-16T03:47:08.940 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.928+0000 7fd354995640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:47:08.940 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.928+0000 7fd354995640 1 -- 172.21.15.42:0/159606189 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2177413734 0 0) 0x7fd34c001680 con 0x7fd3500a1330 2024-09-16T03:47:08.940 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.928+0000 7fd357bfb640 5 monclient: authenticate success, global_id 764184 2024-09-16T03:47:08.941 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.928+0000 7fd357bfb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:47:08.941 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.928+0000 7fd354995640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:47:08.941 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.928+0000 7fd354995640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T03:47:08.941 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.928+0000 7fd354995640 10 monclient: dump: 2024-09-16T03:47:08.941 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:47:08.941 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:08.941 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:08.941 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:08.941 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:47:08.941 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:08.941 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:47:08.941 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:47:08.941 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:47:08.942 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:08.942 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.928+0000 7fd354995640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] but i'm connected to v1:172.21.15.42:6789/0, reconnecting 2024-09-16T03:47:08.942 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.928+0000 7fd354995640 10 monclient: _reopen_session rank -1 2024-09-16T03:47:08.942 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.928+0000 7fd354995640 1 -- 172.21.15.42:0/159606189 >> v1:172.21.15.42:6789/0 conn(0x7fd3500a1330 legacy=0x7fd3500a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:08.942 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.928+0000 7fd354995640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:47:08.942 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.928+0000 7fd354995640 1 --2- 172.21.15.42:0/159606189 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd33c0039c0 0x7fd33c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:08.942 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.928+0000 7fd354995640 10 monclient(hunting): picked mon.c con 0x7fd33c0039c0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:47:08.942 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.928+0000 7fd354995640 1 --2- 172.21.15.42:0/159606189 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd33c004810 0x7fd33c004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:08.942 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.928+0000 7fd354995640 10 monclient(hunting): picked mon.a con 0x7fd33c004810 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:47:08.942 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.928+0000 7fd354995640 1 --2- 172.21.15.42:0/159606189 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd33c0052f0 0x7fd33c00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:08.942 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.928+0000 7fd354995640 10 monclient(hunting): picked mon.b con 0x7fd33c0052f0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:47:08.942 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.928+0000 7fd354995640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:08.942 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.928+0000 7fd354995640 1 -- 172.21.15.42:0/159606189 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fd35009f8f0 con 0x7fd33c004810 2024-09-16T03:47:08.942 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.928+0000 7fd354995640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:08.943 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.928+0000 7fd354995640 1 -- 172.21.15.42:0/159606189 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd33c00c1f0 con 0x7fd33c0052f0 2024-09-16T03:47:08.943 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.928+0000 7fd354995640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:08.943 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.928+0000 7fd354995640 1 -- 172.21.15.42:0/159606189 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fd33c00c3f0 con 0x7fd33c0039c0 2024-09-16T03:47:08.943 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.928+0000 7fd354995640 10 monclient(hunting): _renew_subs 2024-09-16T03:47:08.943 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.928+0000 7fd354995640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:47:08.943 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.928+0000 7fd357bfb640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-16T03:47:08.943 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.928+0000 7fd355997640 1 --2- 172.21.15.42:0/159606189 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd33c0039c0 0x7fd33c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:08.943 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.928+0000 7fd355196640 1 --2- 172.21.15.42:0/159606189 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd33c004810 0x7fd33c004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:08.943 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.928+0000 7fd355997640 10 monclient(hunting): get_auth_request con 0x7fd33c0039c0 auth_method 0 2024-09-16T03:47:08.943 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.928+0000 7fd355997640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:08.943 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.928+0000 7fd355997640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:08.943 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.928+0000 7fd355997640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:47:08.943 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.928+0000 7fd355196640 10 monclient(hunting): get_auth_request con 0x7fd33c004810 auth_method 0 2024-09-16T03:47:08.944 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.928+0000 7fd355196640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:08.944 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.928+0000 7fd355196640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:08.944 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.928+0000 7fd355196640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:47:08.944 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.928+0000 7fd356198640 1 -- 172.21.15.42:0/159606189 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd33c0052f0 msgr2=0x7fd33c00b9d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T03:47:08.944 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.928+0000 7fd356198640 1 --2- 172.21.15.42:0/159606189 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd33c0052f0 0x7fd33c00b9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:47:08.944 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.928+0000 7fd355997640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:47:08.944 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.928+0000 7fd355997640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:47:08.944 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.928+0000 7fd355997640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T03:47:08.944 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.928+0000 7fd355196640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:47:08.944 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.928+0000 7fd355196640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:47:08.944 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.928+0000 7fd355196640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T03:47:08.944 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.928+0000 7fd355196640 10 monclient(hunting): handle_auth_done global_id 764184 payload 293 2024-09-16T03:47:08.945 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.928+0000 7fd355196640 1 -- 172.21.15.42:0/159606189 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd33c0039c0 msgr2=0x7fd33c003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:47:08.945 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.928+0000 7fd355196640 1 --2- 172.21.15.42:0/159606189 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd33c0039c0 0x7fd33c003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:08.945 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.928+0000 7fd355196640 1 -- 172.21.15.42:0/159606189 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd33c0052f0 msgr2=0x7fd33c00b9d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:47:08.945 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.928+0000 7fd355196640 1 --2- 172.21.15.42:0/159606189 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd33c0052f0 0x7fd33c00b9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:08.945 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.928+0000 7fd355196640 10 monclient: _finish_hunting 0 2024-09-16T03:47:08.945 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.928+0000 7fd355196640 1 monclient: found mon.a 2024-09-16T03:47:08.945 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.928+0000 7fd355196640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:47:08.945 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.928+0000 7fd355196640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:47:08.945 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.928+0000 7fd356198640 1 -- 172.21.15.42:0/159606189 reap_dead start 2024-09-16T03:47:08.945 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.928+0000 7fd355196640 1 -- 172.21.15.42:0/159606189 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd33c003530 con 0x7fd33c004810 2024-09-16T03:47:08.945 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.928+0000 7fd355196640 10 monclient: _finish_auth 0 2024-09-16T03:47:08.945 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.928+0000 7fd355196640 10 monclient: _check_auth_tickets 2024-09-16T03:47:08.945 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.928+0000 7fd355196640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:47:08.946 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.928+0000 7fd355997640 1 --2- 172.21.15.42:0/159606189 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd33c0039c0 0x7fd33c003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:47:08.946 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.928+0000 7fd355196640 1 --2- 172.21.15.42:0/159606189 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd33c004810 0x7fd33c004c90 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fd340006fd0 tx=0x7fd3400207f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca2c76eab1b7bf27 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:47:08.946 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.928+0000 7fd354995640 1 -- 172.21.15.42:0/159606189 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3400b2070 con 0x7fd33c004810 2024-09-16T03:47:08.946 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.928+0000 7fd354995640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:47:08.946 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.928+0000 7fd354995640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:47:08.946 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.928+0000 7fd354995640 10 monclient: dump: 2024-09-16T03:47:08.946 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:47:08.946 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:08.946 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:08.946 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:08.946 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:47:08.946 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:08.946 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:47:08.946 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:47:08.947 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:47:08.947 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:08.947 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.928+0000 7fd354995640 1 -- 172.21.15.42:0/159606189 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd340003bc0 con 0x7fd33c004810 2024-09-16T03:47:08.947 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.928+0000 7fd357bfb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:47:08.947 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.928+0000 7fd354995640 10 monclient: handle_config config(0 keys) 2024-09-16T03:47:08.947 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.928+0000 7fd354995640 1 -- 172.21.15.42:0/159606189 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd340009520 con 0x7fd33c004810 2024-09-16T03:47:08.947 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.928+0000 7fd357bfb640 10 monclient: get_monmap_and_config success 2024-09-16T03:47:08.947 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.928+0000 7fd357bfb640 4 set_mon_vals no callback set 2024-09-16T03:47:08.947 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.928+0000 7fd354995640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:47:08.947 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.928+0000 7fd354995640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:47:08.947 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.928+0000 7fd354995640 10 monclient: dump: 2024-09-16T03:47:08.947 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:47:08.947 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:08.947 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:08.948 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:08.948 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:47:08.948 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:08.948 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:47:08.948 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:47:08.948 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:47:08.948 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:08.948 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.928+0000 7fd357bfb640 10 monclient: shutdown 2024-09-16T03:47:08.948 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.928+0000 7fd357bfb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:47:08.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.928+0000 7fd357bfb640 1 -- 172.21.15.42:0/159606189 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd33c004810 msgr2=0x7fd33c004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:08.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.928+0000 7fd357bfb640 1 --2- 172.21.15.42:0/159606189 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd33c004810 0x7fd33c004c90 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fd340006fd0 tx=0x7fd3400207f0 comp rx=0 tx=0).stop 2024-09-16T03:47:08.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.928+0000 7fd357bfb640 1 -- 172.21.15.42:0/159606189 shutdown_connections 2024-09-16T03:47:08.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.928+0000 7fd357bfb640 1 --2- 172.21.15.42:0/159606189 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd33c004810 0x7fd33c004c90 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:08.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.928+0000 7fd357bfb640 1 -- 172.21.15.42:0/159606189 >> 172.21.15.42:0/159606189 conn(0x7fd3500a2420 msgr2=0x7fd3500a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:47:08.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.928+0000 7fd357bfb640 1 -- 172.21.15.42:0/159606189 shutdown_connections 2024-09-16T03:47:08.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.928+0000 7fd357bfb640 1 -- 172.21.15.42:0/159606189 wait complete. 2024-09-16T03:47:08.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.932+0000 7fd357bfb640 10 monclient: build_initial_monmap 2024-09-16T03:47:08.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.932+0000 7fd357bfb640 10 monclient: monmap: 2024-09-16T03:47:08.949 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:47:08.950 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:47:08.950 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:47:08.950 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:47:08.950 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:47:08.950 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:08.950 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:47:08.950 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:47:08.950 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:47:08.950 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:08.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.932+0000 7fd357bfb640 1 librados: starting msgr at 2024-09-16T03:47:08.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.932+0000 7fd357bfb640 1 librados: starting objecter 2024-09-16T03:47:08.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.932+0000 7fd357bfb640 1 Processor -- start 2024-09-16T03:47:08.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.932+0000 7fd357bfb640 1 -- start start 2024-09-16T03:47:08.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.932+0000 7fd357bfb640 1 librados: setting wanted keys 2024-09-16T03:47:08.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.932+0000 7fd357bfb640 1 librados: calling monclient init 2024-09-16T03:47:08.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.932+0000 7fd357bfb640 10 monclient: init 2024-09-16T03:47:08.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.932+0000 7fd357bfb640 10 monclient: _reopen_session rank -1 2024-09-16T03:47:08.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.932+0000 7fd357bfb640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:47:08.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.932+0000 7fd357bfb640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd33c004810 0x7fd350134250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:08.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.932+0000 7fd357bfb640 10 monclient(hunting): picked mon.noname-c con 0x7fd33c004810 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:47:08.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.932+0000 7fd357bfb640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd3501cb680 0x7fd350134790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:08.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.932+0000 7fd357bfb640 10 monclient(hunting): picked mon.noname-a con 0x7fd3501cb680 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:47:08.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.932+0000 7fd357bfb640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd350122cd0 0x7fd350123150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:08.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.932+0000 7fd357bfb640 10 monclient(hunting): picked mon.noname-b con 0x7fd350122cd0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:47:08.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.932+0000 7fd357bfb640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:08.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.932+0000 7fd357bfb640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fd350135ca0 con 0x7fd3501cb680 2024-09-16T03:47:08.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.932+0000 7fd357bfb640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:08.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.932+0000 7fd357bfb640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd350135b20 con 0x7fd350122cd0 2024-09-16T03:47:08.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.932+0000 7fd357bfb640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:08.952 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.932+0000 7fd357bfb640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fd350135e20 con 0x7fd33c004810 2024-09-16T03:47:08.952 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.932+0000 7fd357bfb640 10 monclient(hunting): _renew_subs 2024-09-16T03:47:08.952 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.932+0000 7fd357bfb640 10 monclient(hunting): authenticate will time out at 1275.901411s 2024-09-16T03:47:08.952 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.932+0000 7fd355997640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd33c004810 0x7fd350134250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:08.952 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.932+0000 7fd355196640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd3501cb680 0x7fd350134790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:08.952 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.932+0000 7fd355997640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd33c004810 0x7fd350134250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:51642/0 (socket says 172.21.15.42:51642) 2024-09-16T03:47:08.952 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.932+0000 7fd355196640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd3501cb680 0x7fd350134790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:49766/0 (socket says 172.21.15.42:49766) 2024-09-16T03:47:08.952 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.932+0000 7fd355997640 1 -- 172.21.15.42:0/2557247433 learned_addr learned my addr 172.21.15.42:0/2557247433 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:47:08.952 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.932+0000 7fd356198640 1 -- 172.21.15.42:0/2557247433 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd350122cd0 msgr2=0x7fd350123150 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T03:47:08.952 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.932+0000 7fd355997640 10 monclient(hunting): get_auth_request con 0x7fd33c004810 auth_method 0 2024-09-16T03:47:08.952 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.932+0000 7fd355997640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:08.953 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.932+0000 7fd355997640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:08.953 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.932+0000 7fd355997640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:47:08.953 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.932+0000 7fd355196640 10 monclient(hunting): get_auth_request con 0x7fd3501cb680 auth_method 0 2024-09-16T03:47:08.953 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.932+0000 7fd355196640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:08.953 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.932+0000 7fd355196640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:08.953 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.932+0000 7fd355196640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:47:08.953 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.932+0000 7fd356198640 1 --2- 172.21.15.42:0/2557247433 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd350122cd0 0x7fd350123150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:47:08.953 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.932+0000 7fd355196640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:47:08.953 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.932+0000 7fd355196640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:47:08.953 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.932+0000 7fd355196640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:47:08.953 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.932+0000 7fd355997640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:47:08.953 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.932+0000 7fd355997640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:47:08.953 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.932+0000 7fd355997640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:47:08.953 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.932+0000 7fd355196640 10 monclient(hunting): handle_auth_done global_id 764187 payload 824 2024-09-16T03:47:08.954 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.932+0000 7fd355196640 1 -- 172.21.15.42:0/2557247433 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd33c004810 msgr2=0x7fd350134250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:08.954 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.932+0000 7fd355196640 1 --2- 172.21.15.42:0/2557247433 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd33c004810 0x7fd350134250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:08.954 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.932+0000 7fd355196640 1 -- 172.21.15.42:0/2557247433 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd350122cd0 msgr2=0x7fd350123150 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:47:08.954 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.932+0000 7fd355196640 1 --2- 172.21.15.42:0/2557247433 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd350122cd0 0x7fd350123150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:08.954 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.932+0000 7fd355196640 10 monclient: _finish_hunting 0 2024-09-16T03:47:08.954 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.932+0000 7fd355196640 1 monclient: found mon.noname-a 2024-09-16T03:47:08.954 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.932+0000 7fd355196640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:47:08.954 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.932+0000 7fd355196640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:47:08.954 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.932+0000 7fd355196640 1 -- 172.21.15.42:0/2557247433 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd350073a30 con 0x7fd3501cb680 2024-09-16T03:47:08.954 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.932+0000 7fd355997640 1 --2- 172.21.15.42:0/2557247433 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd33c004810 0x7fd350134250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:47:08.954 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.932+0000 7fd355196640 1 --2- 172.21.15.42:0/2557247433 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd3501cb680 0x7fd350134790 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fd340008000 tx=0x7fd33c005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:47:08.954 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.932+0000 7fd3467fc640 10 client.?.objecter ms_handle_connect 0x7fd3501cb680 2024-09-16T03:47:08.954 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.932+0000 7fd3467fc640 10 client.?.objecter resend_mon_ops 2024-09-16T03:47:08.955 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.932+0000 7fd3467fc640 1 -- 172.21.15.42:0/2557247433 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3400bb020 con 0x7fd3501cb680 2024-09-16T03:47:08.955 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.932+0000 7fd3467fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:47:08.955 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.932+0000 7fd3467fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:47:08.955 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.932+0000 7fd3467fc640 10 monclient: dump: 2024-09-16T03:47:08.955 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:47:08.955 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:08.955 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:08.955 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:08.955 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:47:08.955 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:08.955 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:47:08.955 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:47:08.956 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:47:08.956 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:08.956 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.932+0000 7fd3467fc640 10 monclient: _finish_auth 0 2024-09-16T03:47:08.956 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.932+0000 7fd3467fc640 10 monclient: _check_auth_tickets 2024-09-16T03:47:08.956 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.932+0000 7fd3467fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:47:08.956 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.932+0000 7fd3467fc640 1 -- 172.21.15.42:0/2557247433 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd340009500 con 0x7fd3501cb680 2024-09-16T03:47:08.956 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.932+0000 7fd357bfb640 5 monclient: authenticate success, global_id 764187 2024-09-16T03:47:08.956 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.932+0000 7fd357bfb640 10 monclient: _renew_subs 2024-09-16T03:47:08.956 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.932+0000 7fd357bfb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:47:08.956 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.932+0000 7fd357bfb640 1 -- 172.21.15.42:0/2557247433 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd350126c60 con 0x7fd3501cb680 2024-09-16T03:47:08.956 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.932+0000 7fd3467fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:47:08.956 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.932+0000 7fd3467fc640 1 -- 172.21.15.42:0/2557247433 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3400b2040 con 0x7fd3501cb680 2024-09-16T03:47:08.956 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.932+0000 7fd3467fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:47:08.957 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.932+0000 7fd3467fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:47:08.957 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.932+0000 7fd3467fc640 10 monclient: dump: 2024-09-16T03:47:08.957 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:47:08.957 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:08.957 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:08.957 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:08.957 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:47:08.957 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:08.957 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:47:08.957 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:47:08.957 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:47:08.957 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:08.958 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.932+0000 7fd3477fe640 4 set_mon_vals no callback set 2024-09-16T03:47:08.958 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.932+0000 7fd357bfb640 10 client.764187.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:47:08.958 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.932+0000 7fd357bfb640 10 monclient: _renew_subs 2024-09-16T03:47:08.958 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.932+0000 7fd357bfb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:47:08.958 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.932+0000 7fd357bfb640 1 -- 172.21.15.42:0/2557247433 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd350124970 con 0x7fd3501cb680 2024-09-16T03:47:08.958 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.932+0000 7fd357bfb640 1 librados: init done 2024-09-16T03:47:08.958 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.932+0000 7fd357bfb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:47:08.958 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.932+0000 7fd357bfb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:47:08.958 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.932+0000 7fd357bfb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:47:08.958 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.932+0000 7fd357bfb640 1 -- 172.21.15.42:0/2557247433 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd350136320 con 0x7fd3501cb680 2024-09-16T03:47:08.958 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.932+0000 7fd3467fc640 1 -- 172.21.15.42:0/2557247433 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fd3400be020 con 0x7fd3501cb680 2024-09-16T03:47:08.958 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.932+0000 7fd3467fc640 1 --2- 172.21.15.42:0/2557247433 >> v2:172.21.15.154:6812/254845342 conn(0x7fd3340377f0 0x7fd334039cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:08.959 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.932+0000 7fd3467fc640 1 -- 172.21.15.42:0/2557247433 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(150..150 src has 1..150) ==== 5993+0+0 (secure 0 0 0) 0x7fd3400e6cc0 con 0x7fd3501cb680 2024-09-16T03:47:08.959 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.932+0000 7fd3467fc640 10 client.764187.objecter ms_dispatch 0x7fd350000ba0 osd_map(150..150 src has 1..150) 2024-09-16T03:47:08.959 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.932+0000 7fd3467fc640 3 client.764187.objecter handle_osd_map got epochs [150,150] > 0 2024-09-16T03:47:08.959 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.932+0000 7fd3467fc640 3 client.764187.objecter handle_osd_map decoding full epoch 150 2024-09-16T03:47:08.959 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.932+0000 7fd3467fc640 20 client.764187.objecter dump_active .. 0 homeless 2024-09-16T03:47:08.959 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.936+0000 7fd355997640 1 --2- 172.21.15.42:0/2557247433 >> v2:172.21.15.154:6812/254845342 conn(0x7fd3340377f0 0x7fd334039cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:08.959 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.936+0000 7fd355997640 10 monclient: get_auth_request con 0x7fd3340377f0 auth_method 0 2024-09-16T03:47:08.959 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.936+0000 7fd3467fc640 10 client.764187.objecter ms_handle_connect 0x7fd3340377f0 2024-09-16T03:47:08.959 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.936+0000 7fd355997640 1 --2- 172.21.15.42:0/2557247433 >> v2:172.21.15.154:6812/254845342 conn(0x7fd3340377f0 0x7fd334039cb0 secure :-1 s=READY pgs=3440 cs=0 l=1 rev1=1 crypto rx=0x7fd34c000c10 tx=0x7fd34c007900 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:47:08.960 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.940+0000 7fd3467fc640 1 -- 172.21.15.42:0/2557247433 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7fd3400c5030 con 0x7fd3501cb680 2024-09-16T03:47:08.960 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.940+0000 7fd3467fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:47:08.960 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:08.940+0000 7fd3467fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:47:09.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:09.168+0000 7fd357bfb640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:47:09.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:09.168+0000 7fd357bfb640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:47:09.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:09.168+0000 7fd357bfb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:47:09.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:09.168+0000 7fd357bfb640 1 -- 172.21.15.42:0/2557247433 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fd35006f1d0 con 0x7fd3501cb680 2024-09-16T03:47:09.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:09.168+0000 7fd3467fc640 1 -- 172.21.15.42:0/2557247433 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7fd350073a30 con 0x7fd3501cb680 2024-09-16T03:47:09.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:09.168+0000 7fd3467fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:47:09.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:09.168+0000 7fd3467fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:47:09.170 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":48,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:47:09.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:09.168+0000 7fd357bfb640 10 librados: watch_flush enter 2024-09-16T03:47:09.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:09.168+0000 7fd357bfb640 10 librados: watch_flush exit 2024-09-16T03:47:09.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:09.168+0000 7fd357bfb640 20 client.764187.objecter shutdown clearing up homeless session... 2024-09-16T03:47:09.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:09.168+0000 7fd357bfb640 10 client.764187.objecter successfully canceled tick 2024-09-16T03:47:09.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:09.168+0000 7fd357bfb640 1 -- 172.21.15.42:0/2557247433 >> v2:172.21.15.154:6812/254845342 conn(0x7fd3340377f0 msgr2=0x7fd334039cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:09.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:09.168+0000 7fd357bfb640 1 --2- 172.21.15.42:0/2557247433 >> v2:172.21.15.154:6812/254845342 conn(0x7fd3340377f0 0x7fd334039cb0 secure :-1 s=READY pgs=3440 cs=0 l=1 rev1=1 crypto rx=0x7fd34c000c10 tx=0x7fd34c007900 comp rx=0 tx=0).stop 2024-09-16T03:47:09.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:09.168+0000 7fd357bfb640 10 monclient: shutdown 2024-09-16T03:47:09.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:09.168+0000 7fd357bfb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:47:09.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:09.168+0000 7fd357bfb640 1 -- 172.21.15.42:0/2557247433 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd3501cb680 msgr2=0x7fd350134790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:09.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:09.168+0000 7fd357bfb640 1 --2- 172.21.15.42:0/2557247433 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd3501cb680 0x7fd350134790 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fd340008000 tx=0x7fd33c005840 comp rx=0 tx=0).stop 2024-09-16T03:47:09.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:09.168+0000 7fd357bfb640 1 -- 172.21.15.42:0/2557247433 shutdown_connections 2024-09-16T03:47:09.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:09.168+0000 7fd357bfb640 1 --2- 172.21.15.42:0/2557247433 >> v2:172.21.15.154:6812/254845342 conn(0x7fd3340377f0 0x7fd334039cb0 unknown :-1 s=CLOSED pgs=3440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:09.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:09.168+0000 7fd357bfb640 1 --2- 172.21.15.42:0/2557247433 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd350122cd0 0x7fd350123150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:09.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:09.168+0000 7fd357bfb640 1 --2- 172.21.15.42:0/2557247433 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd3501cb680 0x7fd350134790 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:09.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:09.168+0000 7fd357bfb640 1 --2- 172.21.15.42:0/2557247433 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd33c004810 0x7fd350134250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:09.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:09.168+0000 7fd357bfb640 1 -- 172.21.15.42:0/2557247433 >> 172.21.15.42:0/2557247433 conn(0x7fd3500a2420 msgr2=0x7fd35006f990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:47:09.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:09.168+0000 7fd357bfb640 1 -- 172.21.15.42:0/2557247433 shutdown_connections 2024-09-16T03:47:09.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:09.168+0000 7fd357bfb640 1 -- 172.21.15.42:0/2557247433 wait complete. 2024-09-16T03:47:09.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:09.168+0000 7fd357bfb640 1 librados: shutdown 2024-09-16T03:47:12.190 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:47:12.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.323+0000 7f45b39b7640 10 monclient: get_monmap_and_config 2024-09-16T03:47:12.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.323+0000 7f45b39b7640 10 monclient: build_initial_monmap 2024-09-16T03:47:12.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.323+0000 7f45b39b7640 10 monclient: monmap: 2024-09-16T03:47:12.326 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:47:12.326 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:12.326 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:47:12.325272+0000 2024-09-16T03:47:12.326 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:47:12.325272+0000 2024-09-16T03:47:12.326 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:47:12.327 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:12.327 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:47:12.327 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:47:12.327 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:47:12.327 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:47:12.327 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:47:12.327 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:12.329 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.323+0000 7f45b39b7640 1 Processor -- start 2024-09-16T03:47:12.329 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.323+0000 7f45b39b7640 1 -- start start 2024-09-16T03:47:12.329 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.323+0000 7f45b39b7640 10 monclient: init 2024-09-16T03:47:12.329 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.323+0000 7f45b39b7640 10 monclient: _reopen_session rank -1 2024-09-16T03:47:12.329 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.323+0000 7f45b39b7640 10 monclient: _add_conns ranks=[3,4,1,2,0] 2024-09-16T03:47:12.329 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.323+0000 7f45b39b7640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f45ac0a1330 0x7f45ac0a1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:12.329 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.323+0000 7f45b39b7640 10 monclient(hunting): picked mon.noname-a con 0x7f45ac0a1330 addr v2:172.21.15.42:3300/0 2024-09-16T03:47:12.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.323+0000 7f45b39b7640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f45ac06dac0 0x7f45ac0a1c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:12.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.323+0000 7f45b39b7640 10 monclient(hunting): picked mon.noname-c con 0x7f45ac06dac0 addr v2:172.21.15.154:3300/0 2024-09-16T03:47:12.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.323+0000 7f45b39b7640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f45ac135100 addr v1:172.21.15.154:6789/0 2024-09-16T03:47:12.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.323+0000 7f45b39b7640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f45ac09c7d0 con 0x7f45ac135100 2024-09-16T03:47:12.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.323+0000 7f45b39b7640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:12.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.323+0000 7f45b39b7640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f45ac09c090 con 0x7f45ac0a1330 2024-09-16T03:47:12.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.323+0000 7f45b39b7640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:12.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.323+0000 7f45b39b7640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f45ac09f610 con 0x7f45ac06dac0 2024-09-16T03:47:12.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.323+0000 7f45b39b7640 10 monclient(hunting): _renew_subs 2024-09-16T03:47:12.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.323+0000 7f45b39b7640 10 monclient(hunting): authenticate will time out at 1279.294668s 2024-09-16T03:47:12.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.323+0000 7f45b1f54640 1 -- >> v1:172.21.15.154:6789/0 conn(0x7f45ac135100 legacy=0x7f45ac135550 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.154:6789/0 2024-09-16T03:47:12.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.323+0000 7f45b0f52640 1 -- >> v2:172.21.15.154:3300/0 conn(0x7f45ac06dac0 msgr2=0x7f45ac0a1c50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T03:47:12.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.323+0000 7f45b1753640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f45ac0a1330 0x7f45ac0a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:12.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.323+0000 7f45b0f52640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f45ac06dac0 0x7f45ac0a1c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:47:12.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.323+0000 7f45b1753640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f45ac0a1330 0x7f45ac0a1710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:49776/0 (socket says 172.21.15.42:49776) 2024-09-16T03:47:12.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.323+0000 7f45b1753640 1 -- 172.21.15.42:0/1078735301 learned_addr learned my addr 172.21.15.42:0/1078735301 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:47:12.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.323+0000 7f45b1753640 10 monclient(hunting): get_auth_request con 0x7f45ac0a1330 auth_method 0 2024-09-16T03:47:12.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.323+0000 7f45b1753640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:12.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.323+0000 7f45b1753640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:12.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.323+0000 7f45b1753640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:47:12.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.323+0000 7f45b1753640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:47:12.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.323+0000 7f45b1753640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:47:12.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.323+0000 7f45b1753640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:47:12.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.323+0000 7f45b1753640 10 monclient(hunting): handle_auth_done global_id 764190 payload 274 2024-09-16T03:47:12.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.323+0000 7f45b1753640 1 -- 172.21.15.42:0/1078735301 >> v2:172.21.15.154:3300/0 conn(0x7f45ac06dac0 msgr2=0x7f45ac0a1c50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:47:12.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.323+0000 7f45b1753640 1 --2- 172.21.15.42:0/1078735301 >> v2:172.21.15.154:3300/0 conn(0x7f45ac06dac0 0x7f45ac0a1c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:12.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.323+0000 7f45b1753640 1 -- 172.21.15.42:0/1078735301 >> v1:172.21.15.154:6789/0 conn(0x7f45ac135100 legacy=0x7f45ac135550 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:47:12.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.323+0000 7f45b1753640 10 monclient: _finish_hunting 0 2024-09-16T03:47:12.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.323+0000 7f45b1753640 1 monclient: found mon.noname-a 2024-09-16T03:47:12.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.323+0000 7f45b1753640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:47:12.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.323+0000 7f45b1753640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:47:12.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.323+0000 7f45b1753640 1 -- 172.21.15.42:0/1078735301 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45ac09f8f0 con 0x7f45ac0a1330 2024-09-16T03:47:12.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.327+0000 7f45b1753640 1 --2- 172.21.15.42:0/1078735301 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f45ac0a1330 0x7f45ac0a1710 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f45ac06da10 tx=0x7f45a801c4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16e9d391c6db375c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:47:12.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.327+0000 7f45a3fff640 1 -- 172.21.15.42:0/1078735301 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45a80aa430 con 0x7f45ac0a1330 2024-09-16T03:47:12.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.327+0000 7f45a3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:47:12.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.327+0000 7f45a3fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:47:12.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.327+0000 7f45a3fff640 10 monclient: dump: 2024-09-16T03:47:12.338 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:47:12.339 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:12.339 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:12.339 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:12.339 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:47:12.339 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:12.339 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:47:12.339 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:47:12.339 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:47:12.339 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:12.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.327+0000 7f45a3fff640 10 monclient: _finish_auth 0 2024-09-16T03:47:12.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.327+0000 7f45a3fff640 10 monclient: _check_auth_tickets 2024-09-16T03:47:12.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.327+0000 7f45a3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:47:12.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.327+0000 7f45b39b7640 5 monclient: authenticate success, global_id 764190 2024-09-16T03:47:12.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.327+0000 7f45b39b7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:47:12.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.327+0000 7f45a3fff640 1 -- 172.21.15.42:0/1078735301 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f45a80af070 con 0x7f45ac0a1330 2024-09-16T03:47:12.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.327+0000 7f45a3fff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:47:12.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.327+0000 7f45b39b7640 10 monclient: get_monmap_and_config success 2024-09-16T03:47:12.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.327+0000 7f45b39b7640 4 set_mon_vals no callback set 2024-09-16T03:47:12.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.327+0000 7f45a3fff640 1 -- 172.21.15.42:0/1078735301 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45a80aa7f0 con 0x7f45ac0a1330 2024-09-16T03:47:12.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.327+0000 7f45a3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:47:12.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.327+0000 7f45a3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:47:12.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.327+0000 7f45a3fff640 10 monclient: dump: 2024-09-16T03:47:12.340 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:47:12.340 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:12.340 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:12.340 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:12.340 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:47:12.341 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:12.341 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:47:12.341 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:47:12.341 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:47:12.341 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:12.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.327+0000 7f45b39b7640 10 monclient: shutdown 2024-09-16T03:47:12.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.327+0000 7f45b39b7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:47:12.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.327+0000 7f45b39b7640 1 -- 172.21.15.42:0/1078735301 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f45ac0a1330 msgr2=0x7f45ac0a1710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:12.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.327+0000 7f45b39b7640 1 --2- 172.21.15.42:0/1078735301 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f45ac0a1330 0x7f45ac0a1710 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f45ac06da10 tx=0x7f45a801c4b0 comp rx=0 tx=0).stop 2024-09-16T03:47:12.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.327+0000 7f45b39b7640 1 -- 172.21.15.42:0/1078735301 shutdown_connections 2024-09-16T03:47:12.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.327+0000 7f45b39b7640 1 --2- 172.21.15.42:0/1078735301 >> v2:172.21.15.154:3300/0 conn(0x7f45ac06dac0 0x7f45ac0a1c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:12.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.327+0000 7f45b39b7640 1 --2- 172.21.15.42:0/1078735301 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f45ac0a1330 0x7f45ac0a1710 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:12.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.327+0000 7f45b39b7640 1 -- 172.21.15.42:0/1078735301 >> 172.21.15.42:0/1078735301 conn(0x7f45ac0a2420 msgr2=0x7f45ac0a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:47:12.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.327+0000 7f45b39b7640 1 -- 172.21.15.42:0/1078735301 shutdown_connections 2024-09-16T03:47:12.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.327+0000 7f45b39b7640 1 -- 172.21.15.42:0/1078735301 wait complete. 2024-09-16T03:47:12.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.327+0000 7f45b39b7640 10 monclient: build_initial_monmap 2024-09-16T03:47:12.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.327+0000 7f45b39b7640 10 monclient: monmap: 2024-09-16T03:47:12.342 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:47:12.342 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:47:12.342 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:47:12.342 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:47:12.342 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:47:12.342 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:12.342 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:47:12.342 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:47:12.342 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:47:12.342 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:12.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.327+0000 7f45b39b7640 1 librados: starting msgr at 2024-09-16T03:47:12.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.327+0000 7f45b39b7640 1 librados: starting objecter 2024-09-16T03:47:12.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.331+0000 7f45b39b7640 1 Processor -- start 2024-09-16T03:47:12.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.331+0000 7f45b39b7640 1 -- start start 2024-09-16T03:47:12.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.331+0000 7f45b39b7640 1 librados: setting wanted keys 2024-09-16T03:47:12.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.331+0000 7f45b39b7640 1 librados: calling monclient init 2024-09-16T03:47:12.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.331+0000 7f45b39b7640 10 monclient: init 2024-09-16T03:47:12.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.331+0000 7f45b39b7640 10 monclient: _reopen_session rank -1 2024-09-16T03:47:12.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.331+0000 7f45b39b7640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T03:47:12.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.331+0000 7f45b39b7640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f45ac0a1330 0x7f45ac1cd0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:12.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.331+0000 7f45b39b7640 10 monclient(hunting): picked mon.noname-a con 0x7f45ac0a1330 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:47:12.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.331+0000 7f45b39b7640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f45ac135100 0x7f45ac1cd5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:12.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.331+0000 7f45b39b7640 10 monclient(hunting): picked mon.noname-b con 0x7f45ac135100 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:47:12.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.331+0000 7f45b39b7640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f45ac11d7b0 0x7f45ac1cdb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:12.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.331+0000 7f45b39b7640 10 monclient(hunting): picked mon.noname-c con 0x7f45ac11d7b0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:47:12.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.331+0000 7f45b39b7640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:12.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.331+0000 7f45b39b7640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f45ac13fb80 con 0x7f45ac0a1330 2024-09-16T03:47:12.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.331+0000 7f45b39b7640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:12.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.331+0000 7f45b39b7640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f45ac09ef80 con 0x7f45ac135100 2024-09-16T03:47:12.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.331+0000 7f45b39b7640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:12.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.331+0000 7f45b39b7640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f45ac13fd00 con 0x7f45ac11d7b0 2024-09-16T03:47:12.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.331+0000 7f45b39b7640 10 monclient(hunting): _renew_subs 2024-09-16T03:47:12.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.331+0000 7f45b39b7640 10 monclient(hunting): authenticate will time out at 1279.300618s 2024-09-16T03:47:12.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.331+0000 7f45b1753640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f45ac0a1330 0x7f45ac1cd0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:12.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.331+0000 7f45b1753640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f45ac0a1330 0x7f45ac1cd0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:49786/0 (socket says 172.21.15.42:49786) 2024-09-16T03:47:12.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.331+0000 7f45b1753640 1 -- 172.21.15.42:0/304146435 learned_addr learned my addr 172.21.15.42:0/304146435 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:47:12.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.331+0000 7f45b1753640 10 monclient(hunting): get_auth_request con 0x7f45ac0a1330 auth_method 0 2024-09-16T03:47:12.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.331+0000 7f45b1f54640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f45ac11d7b0 0x7f45ac1cdb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:12.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.331+0000 7f45b1753640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:12.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.331+0000 7f45b1753640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:12.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.331+0000 7f45b1753640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:47:12.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.331+0000 7f45b1f54640 10 monclient(hunting): get_auth_request con 0x7f45ac11d7b0 auth_method 0 2024-09-16T03:47:12.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.331+0000 7f45b1f54640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:12.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.331+0000 7f45b1f54640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:12.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.331+0000 7f45b1f54640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:47:12.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.331+0000 7f45b1753640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:47:12.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.331+0000 7f45b1753640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:47:12.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.331+0000 7f45b1753640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:47:12.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.331+0000 7f45b1f54640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:47:12.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.331+0000 7f45b1f54640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:47:12.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.331+0000 7f45b1f54640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:47:12.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.331+0000 7f45b0f52640 1 -- 172.21.15.42:0/304146435 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f45ac135100 msgr2=0x7f45ac1cd5e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T03:47:12.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.331+0000 7f45b0f52640 1 --2- 172.21.15.42:0/304146435 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f45ac135100 0x7f45ac1cd5e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:47:12.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.331+0000 7f45b1753640 10 monclient(hunting): handle_auth_done global_id 764193 payload 824 2024-09-16T03:47:12.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.331+0000 7f45b1753640 1 -- 172.21.15.42:0/304146435 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f45ac11d7b0 msgr2=0x7f45ac1cdb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:12.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.331+0000 7f45b1753640 1 --2- 172.21.15.42:0/304146435 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f45ac11d7b0 0x7f45ac1cdb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:12.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.331+0000 7f45b1753640 1 -- 172.21.15.42:0/304146435 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f45ac135100 msgr2=0x7f45ac1cd5e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:47:12.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.331+0000 7f45b1753640 1 --2- 172.21.15.42:0/304146435 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f45ac135100 0x7f45ac1cd5e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:12.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.331+0000 7f45b1753640 10 monclient: _finish_hunting 0 2024-09-16T03:47:12.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.331+0000 7f45b1753640 1 monclient: found mon.noname-a 2024-09-16T03:47:12.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.331+0000 7f45b1753640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:47:12.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.331+0000 7f45b1753640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:47:12.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.331+0000 7f45b1753640 1 -- 172.21.15.42:0/304146435 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45ac071a90 con 0x7f45ac0a1330 2024-09-16T03:47:12.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.331+0000 7f45b1753640 1 --2- 172.21.15.42:0/304146435 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f45ac0a1330 0x7f45ac1cd0a0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f45a80ac040 tx=0x7f45a801acf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:47:12.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.331+0000 7f45a1ffb640 10 client.?.objecter ms_handle_connect 0x7f45ac0a1330 2024-09-16T03:47:12.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.331+0000 7f45a1ffb640 10 client.?.objecter resend_mon_ops 2024-09-16T03:47:12.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.331+0000 7f45a1ffb640 1 -- 172.21.15.42:0/304146435 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45a80b3800 con 0x7f45ac0a1330 2024-09-16T03:47:12.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.331+0000 7f45a1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:47:12.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.331+0000 7f45a1ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:47:12.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.331+0000 7f45a1ffb640 10 monclient: dump: 2024-09-16T03:47:12.347 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:47:12.347 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:12.347 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:12.347 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:12.347 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:47:12.347 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:12.347 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:47:12.347 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:47:12.347 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:47:12.348 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:12.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.331+0000 7f45a1ffb640 10 monclient: _finish_auth 0 2024-09-16T03:47:12.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.331+0000 7f45a1ffb640 10 monclient: _check_auth_tickets 2024-09-16T03:47:12.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.331+0000 7f45a1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:47:12.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.331+0000 7f45b39b7640 5 monclient: authenticate success, global_id 764193 2024-09-16T03:47:12.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.331+0000 7f45b39b7640 10 monclient: _renew_subs 2024-09-16T03:47:12.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.331+0000 7f45a1ffb640 1 -- 172.21.15.42:0/304146435 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f45a80b39a0 con 0x7f45ac0a1330 2024-09-16T03:47:12.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.331+0000 7f45b39b7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:47:12.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.331+0000 7f45b39b7640 1 -- 172.21.15.42:0/304146435 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45ac0a50e0 con 0x7f45ac0a1330 2024-09-16T03:47:12.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.331+0000 7f45a1ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T03:47:12.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.331+0000 7f45a1ffb640 1 -- 172.21.15.42:0/304146435 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45a80b3d90 con 0x7f45ac0a1330 2024-09-16T03:47:12.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.331+0000 7f45a1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:47:12.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.331+0000 7f45a1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:47:12.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.331+0000 7f45a1ffb640 10 monclient: dump: 2024-09-16T03:47:12.349 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:47:12.349 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:12.349 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:12.349 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:12.349 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:47:12.349 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:12.349 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:47:12.349 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:47:12.349 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:47:12.349 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:12.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.331+0000 7f45a37fe640 4 set_mon_vals no callback set 2024-09-16T03:47:12.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.331+0000 7f45b39b7640 10 client.764193.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:47:12.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.331+0000 7f45b39b7640 10 monclient: _renew_subs 2024-09-16T03:47:12.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.331+0000 7f45b39b7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:47:12.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.331+0000 7f45b39b7640 1 -- 172.21.15.42:0/304146435 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45ac11ea40 con 0x7f45ac0a1330 2024-09-16T03:47:12.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.331+0000 7f45b39b7640 1 librados: init done 2024-09-16T03:47:12.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.331+0000 7f45b39b7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:47:12.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.331+0000 7f45b39b7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:47:12.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.331+0000 7f45b39b7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:47:12.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.331+0000 7f45b39b7640 1 -- 172.21.15.42:0/304146435 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f45ac135550 con 0x7f45ac0a1330 2024-09-16T03:47:12.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.331+0000 7f45a1ffb640 1 -- 172.21.15.42:0/304146435 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f45a80c6440 con 0x7f45ac0a1330 2024-09-16T03:47:12.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.331+0000 7f45a1ffb640 1 --2- 172.21.15.42:0/304146435 >> v2:172.21.15.154:6812/254845342 conn(0x7f45900377c0 0x7f4590039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:12.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.331+0000 7f45a1ffb640 1 -- 172.21.15.42:0/304146435 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(150..150 src has 1..150) ==== 5993+0+0 (secure 0 0 0) 0x7f45a80e3b20 con 0x7f45ac0a1330 2024-09-16T03:47:12.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.331+0000 7f45a1ffb640 10 client.764193.objecter ms_dispatch 0x7f45ac000ba0 osd_map(150..150 src has 1..150) 2024-09-16T03:47:12.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.331+0000 7f45a1ffb640 3 client.764193.objecter handle_osd_map got epochs [150,150] > 0 2024-09-16T03:47:12.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.331+0000 7f45a1ffb640 3 client.764193.objecter handle_osd_map decoding full epoch 150 2024-09-16T03:47:12.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.331+0000 7f45a1ffb640 20 client.764193.objecter dump_active .. 0 homeless 2024-09-16T03:47:12.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.331+0000 7f45b0f52640 1 --2- 172.21.15.42:0/304146435 >> v2:172.21.15.154:6812/254845342 conn(0x7f45900377c0 0x7f4590039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:12.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.331+0000 7f45b0f52640 10 monclient: get_auth_request con 0x7f45900377c0 auth_method 0 2024-09-16T03:47:12.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.331+0000 7f45b0f52640 1 --2- 172.21.15.42:0/304146435 >> v2:172.21.15.154:6812/254845342 conn(0x7f45900377c0 0x7f4590039c80 secure :-1 s=READY pgs=3441 cs=0 l=1 rev1=1 crypto rx=0x7f459c006fd0 tx=0x7f459c006d40 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:47:12.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.331+0000 7f45a1ffb640 10 client.764193.objecter ms_handle_connect 0x7f45900377c0 2024-09-16T03:47:12.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.335+0000 7f45a1ffb640 1 -- 172.21.15.42:0/304146435 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f45a80160d0 con 0x7f45ac0a1330 2024-09-16T03:47:12.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.335+0000 7f45a1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:47:12.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.335+0000 7f45a1ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:47:12.565 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.563+0000 7f45b39b7640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:47:12.565 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.563+0000 7f45b39b7640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:47:12.565 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.563+0000 7f45b39b7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:47:12.565 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.563+0000 7f45b39b7640 1 -- 172.21.15.42:0/304146435 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f45ac13c3d0 con 0x7f45ac0a1330 2024-09-16T03:47:12.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.563+0000 7f45a1ffb640 1 -- 172.21.15.42:0/304146435 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7f45a80c6730 con 0x7f45ac0a1330 2024-09-16T03:47:12.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.563+0000 7f45a1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:47:12.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.563+0000 7f45a1ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:47:12.567 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":48,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:47:12.569 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.563+0000 7f45b39b7640 10 librados: watch_flush enter 2024-09-16T03:47:12.569 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.563+0000 7f45b39b7640 10 librados: watch_flush exit 2024-09-16T03:47:12.569 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.563+0000 7f45b39b7640 20 client.764193.objecter shutdown clearing up homeless session... 2024-09-16T03:47:12.569 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.563+0000 7f45b39b7640 10 client.764193.objecter successfully canceled tick 2024-09-16T03:47:12.569 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.563+0000 7f45b39b7640 1 -- 172.21.15.42:0/304146435 >> v2:172.21.15.154:6812/254845342 conn(0x7f45900377c0 msgr2=0x7f4590039c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:12.570 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.563+0000 7f45b39b7640 1 --2- 172.21.15.42:0/304146435 >> v2:172.21.15.154:6812/254845342 conn(0x7f45900377c0 0x7f4590039c80 secure :-1 s=READY pgs=3441 cs=0 l=1 rev1=1 crypto rx=0x7f459c006fd0 tx=0x7f459c006d40 comp rx=0 tx=0).stop 2024-09-16T03:47:12.570 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.563+0000 7f45b39b7640 10 monclient: shutdown 2024-09-16T03:47:12.570 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.563+0000 7f45b39b7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:47:12.570 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.563+0000 7f45b39b7640 1 -- 172.21.15.42:0/304146435 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f45ac0a1330 msgr2=0x7f45ac1cd0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:12.570 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.563+0000 7f45b39b7640 1 --2- 172.21.15.42:0/304146435 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f45ac0a1330 0x7f45ac1cd0a0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f45a80ac040 tx=0x7f45a801acf0 comp rx=0 tx=0).stop 2024-09-16T03:47:12.570 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.563+0000 7f45b39b7640 1 -- 172.21.15.42:0/304146435 shutdown_connections 2024-09-16T03:47:12.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.563+0000 7f45b39b7640 1 --2- 172.21.15.42:0/304146435 >> v2:172.21.15.154:6812/254845342 conn(0x7f45900377c0 0x7f4590039c80 unknown :-1 s=CLOSED pgs=3441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:12.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.563+0000 7f45b39b7640 1 --2- 172.21.15.42:0/304146435 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f45ac11d7b0 0x7f45ac1cdb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:12.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.563+0000 7f45b39b7640 1 --2- 172.21.15.42:0/304146435 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f45ac135100 0x7f45ac1cd5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:12.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.563+0000 7f45b39b7640 1 --2- 172.21.15.42:0/304146435 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f45ac0a1330 0x7f45ac1cd0a0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:12.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.563+0000 7f45b39b7640 1 -- 172.21.15.42:0/304146435 >> 172.21.15.42:0/304146435 conn(0x7f45ac0a2420 msgr2=0x7f45ac0a3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:47:12.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.563+0000 7f45b39b7640 1 -- 172.21.15.42:0/304146435 shutdown_connections 2024-09-16T03:47:12.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.563+0000 7f45b39b7640 1 -- 172.21.15.42:0/304146435 wait complete. 2024-09-16T03:47:12.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:12.567+0000 7f45b39b7640 1 librados: shutdown 2024-09-16T03:47:15.586 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:47:15.718 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:15.715+0000 7fd7909b4640 10 monclient: get_monmap_and_config 2024-09-16T03:47:15.718 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:15.715+0000 7fd7909b4640 10 monclient: build_initial_monmap 2024-09-16T03:47:15.719 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:15.715+0000 7fd7909b4640 10 monclient: monmap: 2024-09-16T03:47:15.719 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:47:15.719 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:15.719 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:47:15.717689+0000 2024-09-16T03:47:15.719 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:47:15.717689+0000 2024-09-16T03:47:15.719 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:47:15.719 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:15.719 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:47:15.719 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:47:15.719 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:47:15.719 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:47:15.719 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:47:15.719 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:15.721 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:15.715+0000 7fd7909b4640 1 Processor -- start 2024-09-16T03:47:15.721 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:15.715+0000 7fd7909b4640 1 -- start start 2024-09-16T03:47:15.721 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:15.715+0000 7fd7909b4640 10 monclient: init 2024-09-16T03:47:15.721 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:15.715+0000 7fd7909b4640 10 monclient: _reopen_session rank -1 2024-09-16T03:47:15.721 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:15.715+0000 7fd7909b4640 10 monclient: _add_conns ranks=[0,3,1,4,2] 2024-09-16T03:47:15.721 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:15.715+0000 7fd7909b4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd78c0a1330 addr v1:172.21.15.42:6789/0 2024-09-16T03:47:15.722 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:15.715+0000 7fd7909b4640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd78c06dac0 0x7fd78c13a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:15.722 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:15.715+0000 7fd7909b4640 10 monclient(hunting): picked mon.noname-a con 0x7fd78c06dac0 addr v2:172.21.15.42:3300/0 2024-09-16T03:47:15.722 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:15.715+0000 7fd7909b4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd78c0a1e20 addr v1:172.21.15.154:6789/0 2024-09-16T03:47:15.722 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:15.715+0000 7fd7909b4640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd78c09c090 con 0x7fd78c0a1330 2024-09-16T03:47:15.722 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:15.715+0000 7fd7909b4640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd78c09f610 con 0x7fd78c0a1e20 2024-09-16T03:47:15.722 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:15.715+0000 7fd7909b4640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:15.722 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:15.715+0000 7fd7909b4640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7fd78c09f8f0 con 0x7fd78c06dac0 2024-09-16T03:47:15.722 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:15.715+0000 7fd7909b4640 10 monclient(hunting): _renew_subs 2024-09-16T03:47:15.722 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:15.715+0000 7fd7909b4640 10 monclient(hunting): authenticate will time out at 1282.687117s 2024-09-16T03:47:15.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:15.715+0000 7fd78a59c640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fd78c0a1330 0x7fd78c0a1710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:48654/0 (socket says 172.21.15.42:48654) 2024-09-16T03:47:15.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:15.715+0000 7fd78a59c640 1 -- 172.21.15.42:0/547517290 learned_addr learned my addr 172.21.15.42:0/547517290 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:47:15.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:15.715+0000 7fd78ad9d640 1 -- 172.21.15.42:0/547517290 >> v1:172.21.15.154:6789/0 conn(0x7fd78c0a1e20 legacy=0x7fd78c13c890 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.154:6789/0 2024-09-16T03:47:15.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:15.715+0000 7fd789d9b640 1 --2- 172.21.15.42:0/547517290 >> v2:172.21.15.42:3300/0 conn(0x7fd78c06dac0 0x7fd78c13a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:15.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:15.715+0000 7fd789d9b640 10 monclient(hunting): get_auth_request con 0x7fd78c06dac0 auth_method 0 2024-09-16T03:47:15.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:15.715+0000 7fd789d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:15.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:15.715+0000 7fd789d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:15.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:15.715+0000 7fd789d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:47:15.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:15.715+0000 7fd789d9b640 1 -- 172.21.15.42:0/547517290 >> v2:172.21.15.42:3300/0 conn(0x7fd78c06dac0 msgr2=0x7fd78c13a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T03:47:15.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:15.715+0000 7fd789d9b640 1 -- 172.21.15.42:0/547517290 >> v2:172.21.15.42:3300/0 conn(0x7fd78c06dac0 msgr2=0x7fd78c13a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:47:15.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:15.715+0000 7fd789d9b640 1 --2- 172.21.15.42:0/547517290 >> v2:172.21.15.42:3300/0 conn(0x7fd78c06dac0 0x7fd78c13a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:47:15.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:15.715+0000 7fd789d9b640 1 --2- 172.21.15.42:0/547517290 >> v2:172.21.15.42:3300/0 conn(0x7fd78c06dac0 0x7fd78c13a340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:47:15.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:15.919+0000 7fd78ad9d640 1 -- 172.21.15.42:0/547517290 >> v1:172.21.15.154:6789/0 conn(0x7fd78c0a1e20 legacy=0x7fd78c13c890 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.154:6789/0 2024-09-16T03:47:15.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:15.919+0000 7fd789d9b640 1 --2- 172.21.15.42:0/547517290 >> v2:172.21.15.42:3300/0 conn(0x7fd78c06dac0 0x7fd78c13a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:15.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:15.919+0000 7fd789d9b640 10 monclient(hunting): get_auth_request con 0x7fd78c06dac0 auth_method 0 2024-09-16T03:47:15.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:15.919+0000 7fd789d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:15.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:15.919+0000 7fd789d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:15.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:15.919+0000 7fd789d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:47:15.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:15.919+0000 7fd789d9b640 1 -- 172.21.15.42:0/547517290 >> v2:172.21.15.42:3300/0 conn(0x7fd78c06dac0 msgr2=0x7fd78c13a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T03:47:15.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:15.919+0000 7fd789d9b640 1 -- 172.21.15.42:0/547517290 >> v2:172.21.15.42:3300/0 conn(0x7fd78c06dac0 msgr2=0x7fd78c13a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:47:15.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:15.919+0000 7fd789d9b640 1 --2- 172.21.15.42:0/547517290 >> v2:172.21.15.42:3300/0 conn(0x7fd78c06dac0 0x7fd78c13a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:47:15.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:15.919+0000 7fd789d9b640 1 --2- 172.21.15.42:0/547517290 >> v2:172.21.15.42:3300/0 conn(0x7fd78c06dac0 0x7fd78c13a340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T03:47:16.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:16.323+0000 7fd78ad9d640 1 -- 172.21.15.42:0/547517290 >> v1:172.21.15.154:6789/0 conn(0x7fd78c0a1e20 legacy=0x7fd78c13c890 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.154:6789/0 2024-09-16T03:47:16.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:16.323+0000 7fd789d9b640 1 --2- 172.21.15.42:0/547517290 >> v2:172.21.15.42:3300/0 conn(0x7fd78c06dac0 0x7fd78c13a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:16.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:16.323+0000 7fd789d9b640 10 monclient(hunting): get_auth_request con 0x7fd78c06dac0 auth_method 0 2024-09-16T03:47:16.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:16.323+0000 7fd789d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:16.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:16.323+0000 7fd789d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:16.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:16.323+0000 7fd789d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:47:16.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:16.323+0000 7fd789d9b640 1 -- 172.21.15.42:0/547517290 >> v2:172.21.15.42:3300/0 conn(0x7fd78c06dac0 msgr2=0x7fd78c13a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T03:47:16.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:16.323+0000 7fd789d9b640 1 -- 172.21.15.42:0/547517290 >> v2:172.21.15.42:3300/0 conn(0x7fd78c06dac0 msgr2=0x7fd78c13a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:47:16.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:16.323+0000 7fd789d9b640 1 --2- 172.21.15.42:0/547517290 >> v2:172.21.15.42:3300/0 conn(0x7fd78c06dac0 0x7fd78c13a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:47:16.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:16.323+0000 7fd789d9b640 1 --2- 172.21.15.42:0/547517290 >> v2:172.21.15.42:3300/0 conn(0x7fd78c06dac0 0x7fd78c13a340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T03:47:17.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:17.127+0000 7fd78ad9d640 1 -- 172.21.15.42:0/547517290 >> v1:172.21.15.154:6789/0 conn(0x7fd78c0a1e20 legacy=0x7fd78c13c890 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.154:6789/0 2024-09-16T03:47:17.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:17.127+0000 7fd789d9b640 1 --2- 172.21.15.42:0/547517290 >> v2:172.21.15.42:3300/0 conn(0x7fd78c06dac0 0x7fd78c13a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:17.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:17.127+0000 7fd789d9b640 10 monclient(hunting): get_auth_request con 0x7fd78c06dac0 auth_method 0 2024-09-16T03:47:17.131 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:17.127+0000 7fd789d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:17.131 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:17.127+0000 7fd789d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:17.131 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:17.127+0000 7fd789d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:47:17.131 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:17.127+0000 7fd789d9b640 1 -- 172.21.15.42:0/547517290 >> v2:172.21.15.42:3300/0 conn(0x7fd78c06dac0 msgr2=0x7fd78c13a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T03:47:17.131 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:17.127+0000 7fd789d9b640 1 -- 172.21.15.42:0/547517290 >> v2:172.21.15.42:3300/0 conn(0x7fd78c06dac0 msgr2=0x7fd78c13a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:47:17.131 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:17.127+0000 7fd789d9b640 1 --2- 172.21.15.42:0/547517290 >> v2:172.21.15.42:3300/0 conn(0x7fd78c06dac0 0x7fd78c13a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:47:17.132 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:17.127+0000 7fd789d9b640 1 --2- 172.21.15.42:0/547517290 >> v2:172.21.15.42:3300/0 conn(0x7fd78c06dac0 0x7fd78c13a340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T03:47:18.721 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:18.715+0000 7fd783fff640 10 monclient(hunting): tick 2024-09-16T03:47:18.721 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:18.715+0000 7fd783fff640 10 monclient(hunting): _check_auth_tickets 2024-09-16T03:47:18.721 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:18.715+0000 7fd783fff640 1 monclient(hunting): continuing hunt 2024-09-16T03:47:18.721 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:18.715+0000 7fd783fff640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T03:47:18.721 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:18.715+0000 7fd783fff640 1 -- 172.21.15.42:0/547517290 >> v2:172.21.15.42:3300/0 conn(0x7fd78c06dac0 msgr2=0x7fd78c13a340 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:47:18.721 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:18.715+0000 7fd783fff640 1 --2- 172.21.15.42:0/547517290 >> v2:172.21.15.42:3300/0 conn(0x7fd78c06dac0 0x7fd78c13a340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:18.721 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:18.715+0000 7fd783fff640 1 -- 172.21.15.42:0/547517290 >> v1:172.21.15.154:6789/0 conn(0x7fd78c0a1e20 legacy=0x7fd78c13c890 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:47:18.721 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:18.715+0000 7fd783fff640 1 -- 172.21.15.42:0/547517290 >> v1:172.21.15.42:6789/0 conn(0x7fd78c0a1330 legacy=0x7fd78c0a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:18.722 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:18.715+0000 7fd783fff640 10 monclient: _add_conns ranks=[4,2] 2024-09-16T03:47:18.722 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:18.715+0000 7fd783fff640 1 --2- 172.21.15.42:0/547517290 >> v2:172.21.15.154:3300/0 conn(0x7fd76c0036d0 0x7fd76c005b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:18.722 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:18.715+0000 7fd783fff640 10 monclient(hunting): picked mon.noname-c con 0x7fd76c0036d0 addr v2:172.21.15.154:3300/0 2024-09-16T03:47:18.722 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:18.715+0000 7fd783fff640 1 --2- 172.21.15.42:0/547517290 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd76c006120 0x7fd76c0085e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:18.722 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:18.715+0000 7fd783fff640 10 monclient(hunting): picked mon.noname-b con 0x7fd76c006120 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:47:18.722 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:18.715+0000 7fd783fff640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:18.722 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:18.715+0000 7fd783fff640 1 -- 172.21.15.42:0/547517290 --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fd78c09f8f0 con 0x7fd76c0036d0 2024-09-16T03:47:18.722 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:18.715+0000 7fd783fff640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:18.722 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:18.715+0000 7fd783fff640 1 -- 172.21.15.42:0/547517290 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fd76c008d10 con 0x7fd76c006120 2024-09-16T03:47:18.722 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:18.715+0000 7fd783fff640 10 monclient(hunting): _renew_subs 2024-09-16T03:47:18.722 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:18.715+0000 7fd78a59c640 1 -- 172.21.15.42:0/547517290 >> v2:172.21.15.154:3300/0 conn(0x7fd76c0036d0 msgr2=0x7fd76c005b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T03:47:18.722 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:18.715+0000 7fd789d9b640 1 --2- 172.21.15.42:0/547517290 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd76c006120 0x7fd76c0085e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:18.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:18.715+0000 7fd78a59c640 1 --2- 172.21.15.42:0/547517290 >> v2:172.21.15.154:3300/0 conn(0x7fd76c0036d0 0x7fd76c005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:47:18.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:18.715+0000 7fd789d9b640 10 monclient(hunting): get_auth_request con 0x7fd76c006120 auth_method 0 2024-09-16T03:47:18.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:18.715+0000 7fd789d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:18.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:18.715+0000 7fd789d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:18.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:18.715+0000 7fd789d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:47:18.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:18.715+0000 7fd789d9b640 1 -- 172.21.15.42:0/547517290 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd76c006120 msgr2=0x7fd76c0085e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T03:47:18.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:18.715+0000 7fd789d9b640 1 -- 172.21.15.42:0/547517290 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd76c006120 msgr2=0x7fd76c0085e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:47:18.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:18.719+0000 7fd789d9b640 1 --2- 172.21.15.42:0/547517290 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd76c006120 0x7fd76c0085e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:47:18.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:18.719+0000 7fd789d9b640 1 --2- 172.21.15.42:0/547517290 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd76c006120 0x7fd76c0085e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:47:18.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:18.919+0000 7fd78a59c640 1 -- 172.21.15.42:0/547517290 >> v2:172.21.15.154:3300/0 conn(0x7fd76c0036d0 msgr2=0x7fd76c005b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T03:47:18.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:18.919+0000 7fd78a59c640 1 --2- 172.21.15.42:0/547517290 >> v2:172.21.15.154:3300/0 conn(0x7fd76c0036d0 0x7fd76c005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T03:47:18.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:18.923+0000 7fd789d9b640 1 --2- 172.21.15.42:0/547517290 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd76c006120 0x7fd76c0085e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:18.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:18.923+0000 7fd789d9b640 10 monclient(hunting): get_auth_request con 0x7fd76c006120 auth_method 0 2024-09-16T03:47:18.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:18.923+0000 7fd789d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:18.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:18.923+0000 7fd789d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:18.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:18.923+0000 7fd789d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:47:18.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:18.923+0000 7fd789d9b640 1 -- 172.21.15.42:0/547517290 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd76c006120 msgr2=0x7fd76c0085e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T03:47:18.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:18.923+0000 7fd789d9b640 1 -- 172.21.15.42:0/547517290 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd76c006120 msgr2=0x7fd76c0085e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:47:18.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:18.923+0000 7fd789d9b640 1 --2- 172.21.15.42:0/547517290 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd76c006120 0x7fd76c0085e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:47:18.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:18.923+0000 7fd789d9b640 1 --2- 172.21.15.42:0/547517290 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd76c006120 0x7fd76c0085e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T03:47:19.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:19.323+0000 7fd78a59c640 1 -- 172.21.15.42:0/547517290 >> v2:172.21.15.154:3300/0 conn(0x7fd76c0036d0 msgr2=0x7fd76c005b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T03:47:19.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:19.323+0000 7fd78a59c640 1 --2- 172.21.15.42:0/547517290 >> v2:172.21.15.154:3300/0 conn(0x7fd76c0036d0 0x7fd76c005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T03:47:19.329 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:19.327+0000 7fd789d9b640 1 --2- 172.21.15.42:0/547517290 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd76c006120 0x7fd76c0085e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:19.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:19.327+0000 7fd789d9b640 10 monclient(hunting): get_auth_request con 0x7fd76c006120 auth_method 0 2024-09-16T03:47:19.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:19.327+0000 7fd789d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:19.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:19.327+0000 7fd789d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:19.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:19.327+0000 7fd789d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:47:19.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:19.327+0000 7fd789d9b640 1 -- 172.21.15.42:0/547517290 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd76c006120 msgr2=0x7fd76c0085e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T03:47:19.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:19.327+0000 7fd789d9b640 1 -- 172.21.15.42:0/547517290 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd76c006120 msgr2=0x7fd76c0085e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:47:19.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:19.327+0000 7fd789d9b640 1 --2- 172.21.15.42:0/547517290 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd76c006120 0x7fd76c0085e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:47:19.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:19.327+0000 7fd789d9b640 1 --2- 172.21.15.42:0/547517290 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd76c006120 0x7fd76c0085e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T03:47:20.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:20.127+0000 7fd78a59c640 1 -- 172.21.15.42:0/547517290 >> v2:172.21.15.154:3300/0 conn(0x7fd76c0036d0 msgr2=0x7fd76c005b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T03:47:20.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:20.127+0000 7fd78a59c640 1 --2- 172.21.15.42:0/547517290 >> v2:172.21.15.154:3300/0 conn(0x7fd76c0036d0 0x7fd76c005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T03:47:20.133 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:20.131+0000 7fd789d9b640 1 --2- 172.21.15.42:0/547517290 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd76c006120 0x7fd76c0085e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:20.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:20.131+0000 7fd789d9b640 10 monclient(hunting): get_auth_request con 0x7fd76c006120 auth_method 0 2024-09-16T03:47:20.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:20.131+0000 7fd789d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:20.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:20.131+0000 7fd789d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:20.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:20.131+0000 7fd789d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:47:20.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:20.131+0000 7fd789d9b640 1 -- 172.21.15.42:0/547517290 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd76c006120 msgr2=0x7fd76c0085e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T03:47:20.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:20.131+0000 7fd789d9b640 1 -- 172.21.15.42:0/547517290 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd76c006120 msgr2=0x7fd76c0085e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:47:20.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:20.131+0000 7fd789d9b640 1 --2- 172.21.15.42:0/547517290 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd76c006120 0x7fd76c0085e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:47:20.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:20.131+0000 7fd789d9b640 1 --2- 172.21.15.42:0/547517290 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd76c006120 0x7fd76c0085e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T03:47:21.721 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.715+0000 7fd783fff640 10 monclient(hunting): tick 2024-09-16T03:47:21.722 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.715+0000 7fd783fff640 10 monclient(hunting): _check_auth_tickets 2024-09-16T03:47:21.722 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.715+0000 7fd783fff640 1 monclient(hunting): continuing hunt 2024-09-16T03:47:21.722 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.715+0000 7fd783fff640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T03:47:21.722 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.715+0000 7fd783fff640 1 -- 172.21.15.42:0/547517290 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd76c006120 msgr2=0x7fd76c0085e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:47:21.722 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.715+0000 7fd783fff640 1 --2- 172.21.15.42:0/547517290 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd76c006120 0x7fd76c0085e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:21.722 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.715+0000 7fd783fff640 1 -- 172.21.15.42:0/547517290 >> v2:172.21.15.154:3300/0 conn(0x7fd76c0036d0 msgr2=0x7fd76c005b50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:47:21.722 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.715+0000 7fd783fff640 1 --2- 172.21.15.42:0/547517290 >> v2:172.21.15.154:3300/0 conn(0x7fd76c0036d0 0x7fd76c005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:21.722 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.715+0000 7fd783fff640 10 monclient: _add_conns ranks=[1,2,0,3,4] 2024-09-16T03:47:21.722 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.715+0000 7fd78ad9d640 1 -- 172.21.15.42:0/547517290 reap_dead start 2024-09-16T03:47:21.722 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.715+0000 7fd783fff640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd78c0a1e20 addr v1:172.21.15.154:6789/0 2024-09-16T03:47:21.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.715+0000 7fd783fff640 1 --2- 172.21.15.42:0/547517290 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd76c003b40 0x7fd76c0065d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:21.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.715+0000 7fd783fff640 10 monclient(hunting): picked mon.noname-b con 0x7fd76c003b40 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:47:21.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.715+0000 7fd783fff640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd76c006b10 addr v1:172.21.15.42:6789/0 2024-09-16T03:47:21.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.715+0000 7fd783fff640 1 -- 172.21.15.42:0/547517290 --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd78c09fbd0 con 0x7fd76c006b10 2024-09-16T03:47:21.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.715+0000 7fd783fff640 1 -- 172.21.15.42:0/547517290 --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd78c09f610 con 0x7fd78c0a1e20 2024-09-16T03:47:21.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.715+0000 7fd783fff640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:21.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.715+0000 7fd783fff640 1 -- 172.21.15.42:0/547517290 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fd78c09f8f0 con 0x7fd76c003b40 2024-09-16T03:47:21.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.715+0000 7fd783fff640 10 monclient(hunting): _renew_subs 2024-09-16T03:47:21.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.719+0000 7fd78a59c640 1 -- 172.21.15.42:0/547517290 >> v1:172.21.15.154:6789/0 conn(0x7fd78c0a1e20 legacy=0x7fd76c00b1d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.154:6789/0 2024-09-16T03:47:21.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.719+0000 7fd789d9b640 1 --2- 172.21.15.42:0/547517290 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd76c003b40 0x7fd76c0065d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:21.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.719+0000 7fd789d9b640 10 monclient(hunting): get_auth_request con 0x7fd76c003b40 auth_method 0 2024-09-16T03:47:21.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.719+0000 7fd789d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:21.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.719+0000 7fd789d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:21.724 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.719+0000 7fd789d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:47:21.724 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.719+0000 7fd789d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:47:21.724 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.719+0000 7fd789d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:47:21.724 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.719+0000 7fd789d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:47:21.724 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.719+0000 7fd789d9b640 10 monclient(hunting): handle_auth_done global_id 13742 payload 274 2024-09-16T03:47:21.724 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.719+0000 7fd789d9b640 1 -- 172.21.15.42:0/547517290 >> v1:172.21.15.154:6789/0 conn(0x7fd78c0a1e20 legacy=0x7fd76c00b1d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:47:21.724 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.719+0000 7fd789d9b640 1 -- 172.21.15.42:0/547517290 >> v1:172.21.15.42:6789/0 conn(0x7fd76c006b10 legacy=0x7fd76c006f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:21.724 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.719+0000 7fd789d9b640 10 monclient: _finish_hunting 0 2024-09-16T03:47:21.724 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.719+0000 7fd789d9b640 1 monclient: found mon.noname-b 2024-09-16T03:47:21.724 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.719+0000 7fd78959a640 1 -- 172.21.15.42:0/547517290 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2286859855 0 0) 0x7fd78c09fbd0 con 0x7fd76c006b10 2024-09-16T03:47:21.724 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.719+0000 7fd789d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:47:21.724 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.719+0000 7fd789d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T03:47:21.724 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.719+0000 7fd789d9b640 1 -- 172.21.15.42:0/547517290 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd76c008890 con 0x7fd76c003b40 2024-09-16T03:47:21.725 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.719+0000 7fd78959a640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T03:47:21.727 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.723+0000 7fd789d9b640 1 --2- 172.21.15.42:0/547517290 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd76c003b40 0x7fd76c0065d0 secure :-1 s=READY pgs=2480 cs=0 l=1 rev1=1 crypto rx=0x7fd774016930 tx=0x7fd77401dcc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a6720e5a7a532d95 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:47:21.728 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.723+0000 7fd78959a640 1 -- 172.21.15.42:0/547517290 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd77400f040 con 0x7fd76c003b40 2024-09-16T03:47:21.728 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.723+0000 7fd78959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:47:21.728 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.723+0000 7fd78959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:47:21.728 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.723+0000 7fd78959a640 10 monclient: dump: 2024-09-16T03:47:21.728 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:47:21.728 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:21.729 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:21.729 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:21.729 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:47:21.729 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:21.729 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:47:21.729 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:47:21.729 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:47:21.729 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:21.729 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.723+0000 7fd78959a640 10 monclient: _finish_auth 0 2024-09-16T03:47:21.729 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.723+0000 7fd78959a640 10 monclient: _check_auth_tickets 2024-09-16T03:47:21.729 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.723+0000 7fd78959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:47:21.729 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.723+0000 7fd78959a640 1 -- 172.21.15.42:0/547517290 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7740a6c50 con 0x7fd76c003b40 2024-09-16T03:47:21.729 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.723+0000 7fd7909b4640 5 monclient: authenticate success, global_id 13742 2024-09-16T03:47:21.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.723+0000 7fd7909b4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:47:21.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.723+0000 7fd78959a640 10 monclient: handle_config config(0 keys) 2024-09-16T03:47:21.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.723+0000 7fd7909b4640 10 monclient: get_monmap_and_config success 2024-09-16T03:47:21.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.723+0000 7fd7909b4640 4 set_mon_vals no callback set 2024-09-16T03:47:21.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.723+0000 7fd78959a640 1 -- 172.21.15.42:0/547517290 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd774004030 con 0x7fd76c003b40 2024-09-16T03:47:21.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.723+0000 7fd78959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:47:21.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.723+0000 7fd78959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:47:21.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.723+0000 7fd78959a640 10 monclient: dump: 2024-09-16T03:47:21.730 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:47:21.730 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:21.730 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:21.730 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:21.730 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:47:21.731 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:21.731 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:47:21.731 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:47:21.731 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:47:21.731 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:21.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.727+0000 7fd7909b4640 10 monclient: shutdown 2024-09-16T03:47:21.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.727+0000 7fd7909b4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:47:21.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.727+0000 7fd7909b4640 1 -- 172.21.15.42:0/547517290 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd76c003b40 msgr2=0x7fd76c0065d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:21.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.727+0000 7fd7909b4640 1 --2- 172.21.15.42:0/547517290 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd76c003b40 0x7fd76c0065d0 secure :-1 s=READY pgs=2480 cs=0 l=1 rev1=1 crypto rx=0x7fd774016930 tx=0x7fd77401dcc0 comp rx=0 tx=0).stop 2024-09-16T03:47:21.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.727+0000 7fd7909b4640 1 -- 172.21.15.42:0/547517290 shutdown_connections 2024-09-16T03:47:21.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.727+0000 7fd7909b4640 1 --2- 172.21.15.42:0/547517290 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd76c003b40 0x7fd76c0065d0 unknown :-1 s=CLOSED pgs=2480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:21.733 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.727+0000 7fd7909b4640 1 -- 172.21.15.42:0/547517290 >> 172.21.15.42:0/547517290 conn(0x7fd78c0a2420 msgr2=0x7fd78c0a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:47:21.733 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.727+0000 7fd7909b4640 1 -- 172.21.15.42:0/547517290 shutdown_connections 2024-09-16T03:47:21.733 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.727+0000 7fd7909b4640 1 -- 172.21.15.42:0/547517290 wait complete. 2024-09-16T03:47:21.733 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.727+0000 7fd7909b4640 10 monclient: build_initial_monmap 2024-09-16T03:47:21.733 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.727+0000 7fd7909b4640 10 monclient: monmap: 2024-09-16T03:47:21.733 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:47:21.733 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:47:21.733 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:47:21.733 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:47:21.733 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:47:21.733 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:21.733 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:47:21.733 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:47:21.733 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:47:21.734 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:21.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.727+0000 7fd7909b4640 1 librados: starting msgr at 2024-09-16T03:47:21.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.727+0000 7fd7909b4640 1 librados: starting objecter 2024-09-16T03:47:21.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.727+0000 7fd7909b4640 1 Processor -- start 2024-09-16T03:47:21.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.727+0000 7fd7909b4640 1 -- start start 2024-09-16T03:47:21.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.727+0000 7fd7909b4640 1 librados: setting wanted keys 2024-09-16T03:47:21.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.727+0000 7fd7909b4640 1 librados: calling monclient init 2024-09-16T03:47:21.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.727+0000 7fd7909b4640 10 monclient: init 2024-09-16T03:47:21.738 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.731+0000 7fd7909b4640 10 monclient: _reopen_session rank -1 2024-09-16T03:47:21.738 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.731+0000 7fd7909b4640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:47:21.738 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.731+0000 7fd7909b4640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd76c003b40 0x7fd78c1cc3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:21.738 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.731+0000 7fd7909b4640 10 monclient(hunting): picked mon.noname-a con 0x7fd76c003b40 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:47:21.738 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.731+0000 7fd7909b4640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd76c006b10 0x7fd78c1cc8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:21.738 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.731+0000 7fd7909b4640 10 monclient(hunting): picked mon.noname-c con 0x7fd76c006b10 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:47:21.738 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.731+0000 7fd7909b4640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd78c0a1e20 0x7fd78c123a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:21.738 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.731+0000 7fd7909b4640 10 monclient(hunting): picked mon.noname-b con 0x7fd78c0a1e20 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:47:21.738 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.731+0000 7fd7909b4640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:21.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.731+0000 7fd7909b4640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fd78c09eef0 con 0x7fd76c003b40 2024-09-16T03:47:21.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.731+0000 7fd7909b4640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:21.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.731+0000 7fd7909b4640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd78c09ed70 con 0x7fd78c0a1e20 2024-09-16T03:47:21.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.731+0000 7fd7909b4640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:21.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.731+0000 7fd7909b4640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fd78c0a1710 con 0x7fd76c006b10 2024-09-16T03:47:21.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.731+0000 7fd7909b4640 10 monclient(hunting): _renew_subs 2024-09-16T03:47:21.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.731+0000 7fd7909b4640 10 monclient(hunting): authenticate will time out at 1288.701024s 2024-09-16T03:47:21.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.731+0000 7fd78a59c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd76c003b40 0x7fd78c1cc3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:21.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.731+0000 7fd78a59c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd76c003b40 0x7fd78c1cc3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:60050/0 (socket says 172.21.15.42:60050) 2024-09-16T03:47:21.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.731+0000 7fd78a59c640 1 -- 172.21.15.42:0/939391007 learned_addr learned my addr 172.21.15.42:0/939391007 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:47:21.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.731+0000 7fd78a59c640 10 monclient(hunting): get_auth_request con 0x7fd76c003b40 auth_method 0 2024-09-16T03:47:21.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.731+0000 7fd78a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:21.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.731+0000 7fd78a59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:21.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.731+0000 7fd78a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:47:21.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.731+0000 7fd789d9b640 1 --2- 172.21.15.42:0/939391007 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd76c006b10 0x7fd78c1cc8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:21.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.731+0000 7fd78ad9d640 1 -- 172.21.15.42:0/939391007 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd78c0a1e20 msgr2=0x7fd78c123a60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T03:47:21.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.731+0000 7fd78a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:47:21.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.731+0000 7fd78a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:47:21.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.731+0000 7fd78a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:47:21.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.731+0000 7fd789d9b640 10 monclient(hunting): get_auth_request con 0x7fd76c006b10 auth_method 0 2024-09-16T03:47:21.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.731+0000 7fd789d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:21.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.731+0000 7fd78ad9d640 1 --2- 172.21.15.42:0/939391007 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd78c0a1e20 0x7fd78c123a60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:47:21.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.731+0000 7fd789d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:21.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.731+0000 7fd789d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:47:21.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.731+0000 7fd789d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:47:21.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.731+0000 7fd789d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:47:21.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.731+0000 7fd789d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:47:21.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.731+0000 7fd78a59c640 10 monclient(hunting): handle_auth_done global_id 764199 payload 824 2024-09-16T03:47:21.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.731+0000 7fd78a59c640 1 -- 172.21.15.42:0/939391007 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd76c006b10 msgr2=0x7fd78c1cc8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:21.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.731+0000 7fd78a59c640 1 --2- 172.21.15.42:0/939391007 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd76c006b10 0x7fd78c1cc8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:21.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.731+0000 7fd78a59c640 1 -- 172.21.15.42:0/939391007 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd78c0a1e20 msgr2=0x7fd78c123a60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:47:21.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.731+0000 7fd78a59c640 1 --2- 172.21.15.42:0/939391007 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd78c0a1e20 0x7fd78c123a60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:21.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.731+0000 7fd78a59c640 10 monclient: _finish_hunting 0 2024-09-16T03:47:21.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.731+0000 7fd78a59c640 1 monclient: found mon.noname-a 2024-09-16T03:47:21.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.731+0000 7fd78a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:47:21.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.731+0000 7fd78a59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:47:21.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.731+0000 7fd78a59c640 1 -- 172.21.15.42:0/939391007 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd78c127290 con 0x7fd76c003b40 2024-09-16T03:47:21.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.731+0000 7fd78a59c640 1 --2- 172.21.15.42:0/939391007 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd76c003b40 0x7fd78c1cc3a0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fd77c00cc20 tx=0x7fd77c00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:47:21.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.731+0000 7fd782ffd640 10 client.?.objecter ms_handle_connect 0x7fd76c003b40 2024-09-16T03:47:21.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.731+0000 7fd782ffd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:47:21.742 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.731+0000 7fd782ffd640 1 -- 172.21.15.42:0/939391007 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd77c017070 con 0x7fd76c003b40 2024-09-16T03:47:21.742 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.731+0000 7fd782ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:47:21.742 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.731+0000 7fd782ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:47:21.742 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.731+0000 7fd782ffd640 10 monclient: dump: 2024-09-16T03:47:21.742 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:47:21.742 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:21.742 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:21.742 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:21.742 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:47:21.742 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:21.742 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:47:21.742 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:47:21.742 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:47:21.743 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:21.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.731+0000 7fd782ffd640 10 monclient: _finish_auth 0 2024-09-16T03:47:21.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.731+0000 7fd782ffd640 10 monclient: _check_auth_tickets 2024-09-16T03:47:21.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.731+0000 7fd782ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:47:21.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.731+0000 7fd782ffd640 1 -- 172.21.15.42:0/939391007 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd77c00ce30 con 0x7fd76c003b40 2024-09-16T03:47:21.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.731+0000 7fd782ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:47:21.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.731+0000 7fd782ffd640 1 -- 172.21.15.42:0/939391007 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd77c012600 con 0x7fd76c003b40 2024-09-16T03:47:21.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.731+0000 7fd7909b4640 5 monclient: authenticate success, global_id 764199 2024-09-16T03:47:21.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.731+0000 7fd7909b4640 10 monclient: _renew_subs 2024-09-16T03:47:21.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.731+0000 7fd7909b4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:47:21.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.731+0000 7fd788d99640 4 set_mon_vals no callback set 2024-09-16T03:47:21.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.731+0000 7fd7909b4640 1 -- 172.21.15.42:0/939391007 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd78c127850 con 0x7fd76c003b40 2024-09-16T03:47:21.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.731+0000 7fd782ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:47:21.744 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.731+0000 7fd782ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:47:21.744 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.731+0000 7fd782ffd640 10 monclient: dump: 2024-09-16T03:47:21.744 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:47:21.744 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:21.744 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:21.744 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:21.744 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:47:21.744 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:21.744 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:47:21.744 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:47:21.744 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:47:21.744 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:21.744 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.731+0000 7fd7909b4640 10 client.764199.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:47:21.744 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.731+0000 7fd7909b4640 10 monclient: _renew_subs 2024-09-16T03:47:21.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.731+0000 7fd7909b4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:47:21.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.731+0000 7fd7909b4640 1 -- 172.21.15.42:0/939391007 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd78c126fb0 con 0x7fd76c003b40 2024-09-16T03:47:21.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.731+0000 7fd7909b4640 1 librados: init done 2024-09-16T03:47:21.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.731+0000 7fd7909b4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:47:21.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.731+0000 7fd7909b4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:47:21.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.731+0000 7fd7909b4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:47:21.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.731+0000 7fd782ffd640 1 -- 172.21.15.42:0/939391007 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fd77c01a070 con 0x7fd76c003b40 2024-09-16T03:47:21.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.731+0000 7fd7909b4640 1 -- 172.21.15.42:0/939391007 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd78c0a1aa0 con 0x7fd76c003b40 2024-09-16T03:47:21.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.731+0000 7fd782ffd640 1 --2- 172.21.15.42:0/939391007 >> v2:172.21.15.154:6812/254845342 conn(0x7fd76c03b720 0x7fd76c03dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:21.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.731+0000 7fd782ffd640 1 -- 172.21.15.42:0/939391007 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(151..151 src has 1..151) ==== 6739+0+0 (secure 0 0 0) 0x7fd77c049280 con 0x7fd76c003b40 2024-09-16T03:47:21.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.731+0000 7fd782ffd640 10 client.764199.objecter ms_dispatch 0x7fd78c000ba0 osd_map(151..151 src has 1..151) 2024-09-16T03:47:21.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.731+0000 7fd782ffd640 3 client.764199.objecter handle_osd_map got epochs [151,151] > 0 2024-09-16T03:47:21.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.731+0000 7fd782ffd640 3 client.764199.objecter handle_osd_map decoding full epoch 151 2024-09-16T03:47:21.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.731+0000 7fd782ffd640 20 client.764199.objecter dump_active .. 0 homeless 2024-09-16T03:47:21.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.735+0000 7fd789d9b640 1 --2- 172.21.15.42:0/939391007 >> v2:172.21.15.154:6812/254845342 conn(0x7fd76c03b720 0x7fd76c03dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:21.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.735+0000 7fd789d9b640 10 monclient: get_auth_request con 0x7fd76c03b720 auth_method 0 2024-09-16T03:47:21.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.735+0000 7fd789d9b640 1 --2- 172.21.15.42:0/939391007 >> v2:172.21.15.154:6812/254845342 conn(0x7fd76c03b720 0x7fd76c03dbe0 secure :-1 s=READY pgs=3442 cs=0 l=1 rev1=1 crypto rx=0x7fd7740af000 tx=0x7fd7740a8070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:47:21.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.735+0000 7fd782ffd640 10 client.764199.objecter ms_handle_connect 0x7fd76c03b720 2024-09-16T03:47:21.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.739+0000 7fd782ffd640 1 -- 172.21.15.42:0/939391007 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7fd77c007070 con 0x7fd76c003b40 2024-09-16T03:47:21.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.739+0000 7fd782ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:47:21.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.739+0000 7fd782ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:47:21.976 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.971+0000 7fd7909b4640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:47:21.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.971+0000 7fd7909b4640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:47:21.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.971+0000 7fd7909b4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:47:21.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.971+0000 7fd7909b4640 1 -- 172.21.15.42:0/939391007 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fd78c069040 con 0x7fd76c003b40 2024-09-16T03:47:21.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.975+0000 7fd782ffd640 1 -- 172.21.15.42:0/939391007 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1554 (secure 0 0 0) 0x7fd77c047410 con 0x7fd76c003b40 2024-09-16T03:47:21.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.975+0000 7fd782ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:47:21.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.975+0000 7fd782ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:47:21.978 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":52,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:47:21.980 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.975+0000 7fd7909b4640 10 librados: watch_flush enter 2024-09-16T03:47:21.980 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.975+0000 7fd7909b4640 10 librados: watch_flush exit 2024-09-16T03:47:21.980 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.975+0000 7fd7909b4640 20 client.764199.objecter shutdown clearing up homeless session... 2024-09-16T03:47:21.980 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.975+0000 7fd7909b4640 10 client.764199.objecter successfully canceled tick 2024-09-16T03:47:21.980 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.975+0000 7fd7909b4640 1 -- 172.21.15.42:0/939391007 >> v2:172.21.15.154:6812/254845342 conn(0x7fd76c03b720 msgr2=0x7fd76c03dbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:21.980 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.975+0000 7fd7909b4640 1 --2- 172.21.15.42:0/939391007 >> v2:172.21.15.154:6812/254845342 conn(0x7fd76c03b720 0x7fd76c03dbe0 secure :-1 s=READY pgs=3442 cs=0 l=1 rev1=1 crypto rx=0x7fd7740af000 tx=0x7fd7740a8070 comp rx=0 tx=0).stop 2024-09-16T03:47:21.980 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.975+0000 7fd7909b4640 10 monclient: shutdown 2024-09-16T03:47:21.981 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.975+0000 7fd7909b4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:47:21.981 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.975+0000 7fd7909b4640 1 -- 172.21.15.42:0/939391007 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd76c003b40 msgr2=0x7fd78c1cc3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:21.981 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.975+0000 7fd7909b4640 1 --2- 172.21.15.42:0/939391007 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd76c003b40 0x7fd78c1cc3a0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fd77c00cc20 tx=0x7fd77c00a430 comp rx=0 tx=0).stop 2024-09-16T03:47:21.981 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.975+0000 7fd7909b4640 1 -- 172.21.15.42:0/939391007 shutdown_connections 2024-09-16T03:47:21.981 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.975+0000 7fd7909b4640 1 --2- 172.21.15.42:0/939391007 >> v2:172.21.15.154:6812/254845342 conn(0x7fd76c03b720 0x7fd76c03dbe0 unknown :-1 s=CLOSED pgs=3442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:21.981 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.975+0000 7fd7909b4640 1 --2- 172.21.15.42:0/939391007 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd78c0a1e20 0x7fd78c123a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:21.981 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.975+0000 7fd7909b4640 1 --2- 172.21.15.42:0/939391007 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd76c006b10 0x7fd78c1cc8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:21.981 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.975+0000 7fd7909b4640 1 --2- 172.21.15.42:0/939391007 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd76c003b40 0x7fd78c1cc3a0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:21.981 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.975+0000 7fd7909b4640 1 -- 172.21.15.42:0/939391007 >> 172.21.15.42:0/939391007 conn(0x7fd78c0a2420 msgr2=0x7fd78c0715b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:47:21.981 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.975+0000 7fd7909b4640 1 -- 172.21.15.42:0/939391007 shutdown_connections 2024-09-16T03:47:21.981 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.975+0000 7fd7909b4640 1 -- 172.21.15.42:0/939391007 wait complete. 2024-09-16T03:47:21.981 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:21.975+0000 7fd7909b4640 1 librados: shutdown 2024-09-16T03:47:21.997 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T03:47:21.997 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-16T03:47:21.997 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T03:47:22.117 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.115+0000 7f925da1b640 10 monclient: get_monmap_and_config 2024-09-16T03:47:22.117 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.115+0000 7f925da1b640 10 monclient: build_initial_monmap 2024-09-16T03:47:22.117 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.115+0000 7f925da1b640 10 monclient: monmap: 2024-09-16T03:47:22.117 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:47:22.117 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:22.117 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:47:22.116247+0000 2024-09-16T03:47:22.117 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:47:22.116247+0000 2024-09-16T03:47:22.117 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:47:22.117 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:22.117 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:47:22.118 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:47:22.118 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:47:22.118 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:47:22.118 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:47:22.118 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:22.119 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.115+0000 7f925da1b640 1 Processor -- start 2024-09-16T03:47:22.120 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.115+0000 7f925da1b640 1 -- start start 2024-09-16T03:47:22.120 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.115+0000 7f925da1b640 10 monclient: init 2024-09-16T03:47:22.120 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.115+0000 7f925da1b640 10 monclient: _reopen_session rank -1 2024-09-16T03:47:22.120 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.115+0000 7f925da1b640 10 monclient: _add_conns ranks=[2,4,1,3,0] 2024-09-16T03:47:22.120 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.115+0000 7f925da1b640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f92580a39b0 0x7f9258133b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:22.120 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.115+0000 7f925da1b640 10 monclient(hunting): picked mon.noname-b con 0x7f92580a39b0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:47:22.120 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.115+0000 7f925da1b640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f92580a3d90 0x7f9258137000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:22.120 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.115+0000 7f925da1b640 10 monclient(hunting): picked mon.noname-c con 0x7f92580a3d90 addr v2:172.21.15.154:3300/0 2024-09-16T03:47:22.120 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.115+0000 7f925da1b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9258137540 addr v1:172.21.15.154:6789/0 2024-09-16T03:47:22.120 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.115+0000 7f925da1b640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f925809c140 con 0x7f9258137540 2024-09-16T03:47:22.120 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.115+0000 7f925da1b640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:22.120 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.115+0000 7f925da1b640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f925809c420 con 0x7f92580a3d90 2024-09-16T03:47:22.120 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.115+0000 7f925da1b640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:22.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.115+0000 7f925da1b640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f925806dd60 con 0x7f92580a39b0 2024-09-16T03:47:22.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.115+0000 7f925da1b640 10 monclient(hunting): _renew_subs 2024-09-16T03:47:22.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.115+0000 7f925da1b640 10 monclient(hunting): authenticate will time out at 1289.085545s 2024-09-16T03:47:22.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.115+0000 7f9256ffd640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f92580a39b0 0x7f9258133b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:22.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.115+0000 7f92567fc640 1 -- >> v2:172.21.15.154:3300/0 conn(0x7f92580a3d90 msgr2=0x7f9258137000 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T03:47:22.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.115+0000 7f9256ffd640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f92580a39b0 0x7f9258133b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:36908/0 (socket says 172.21.15.42:36908) 2024-09-16T03:47:22.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.115+0000 7f92577fe640 1 -- >> v1:172.21.15.154:6789/0 conn(0x7f9258137540 legacy=0x7f925813e390 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.154:6789/0 2024-09-16T03:47:22.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.115+0000 7f9256ffd640 1 -- 172.21.15.42:0/1458841839 learned_addr learned my addr 172.21.15.42:0/1458841839 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:47:22.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.115+0000 7f9256ffd640 10 monclient(hunting): get_auth_request con 0x7f92580a39b0 auth_method 0 2024-09-16T03:47:22.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.115+0000 7f9256ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:22.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.115+0000 7f9256ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:22.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.115+0000 7f9256ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:47:22.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.115+0000 7f92567fc640 1 --2- 172.21.15.42:0/1458841839 >> v2:172.21.15.154:3300/0 conn(0x7f92580a3d90 0x7f9258137000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:47:22.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.115+0000 7f9256ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:47:22.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.115+0000 7f9256ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:47:22.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.115+0000 7f9256ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:47:22.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.115+0000 7f9256ffd640 10 monclient(hunting): handle_auth_done global_id 13748 payload 274 2024-09-16T03:47:22.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.115+0000 7f9256ffd640 1 -- 172.21.15.42:0/1458841839 >> v2:172.21.15.154:3300/0 conn(0x7f92580a3d90 msgr2=0x7f9258137000 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:47:22.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.115+0000 7f9256ffd640 1 --2- 172.21.15.42:0/1458841839 >> v2:172.21.15.154:3300/0 conn(0x7f92580a3d90 0x7f9258137000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:22.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.115+0000 7f9256ffd640 1 -- 172.21.15.42:0/1458841839 >> v1:172.21.15.154:6789/0 conn(0x7f9258137540 legacy=0x7f925813e390 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:47:22.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.115+0000 7f9256ffd640 10 monclient: _finish_hunting 0 2024-09-16T03:47:22.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.115+0000 7f9256ffd640 1 monclient: found mon.noname-b 2024-09-16T03:47:22.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.115+0000 7f9256ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:47:22.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.115+0000 7f9256ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T03:47:22.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.115+0000 7f9256ffd640 1 -- 172.21.15.42:0/1458841839 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92580a5e20 con 0x7f92580a39b0 2024-09-16T03:47:22.128 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.119+0000 7f9256ffd640 1 --2- 172.21.15.42:0/1458841839 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f92580a39b0 0x7f9258133b80 secure :-1 s=READY pgs=2481 cs=0 l=1 rev1=1 crypto rx=0x7f9250009e50 tx=0x7f925001c8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bf463ccbbb6cc85c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:47:22.128 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.119+0000 7f9255ffb640 1 -- 172.21.15.42:0/1458841839 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92500a5550 con 0x7f92580a39b0 2024-09-16T03:47:22.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.119+0000 7f9255ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:47:22.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.119+0000 7f9255ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:47:22.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.119+0000 7f9255ffb640 10 monclient: dump: 2024-09-16T03:47:22.129 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:47:22.129 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:22.129 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:22.129 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:22.129 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:47:22.129 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:22.129 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:47:22.129 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:47:22.129 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:47:22.129 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:22.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.119+0000 7f9255ffb640 10 monclient: _finish_auth 0 2024-09-16T03:47:22.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.119+0000 7f9255ffb640 10 monclient: _check_auth_tickets 2024-09-16T03:47:22.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.119+0000 7f9255ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:47:22.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.119+0000 7f9255ffb640 1 -- 172.21.15.42:0/1458841839 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f92500a56f0 con 0x7f92580a39b0 2024-09-16T03:47:22.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.119+0000 7f9255ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T03:47:22.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.119+0000 7f9255ffb640 1 -- 172.21.15.42:0/1458841839 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92500a5ac0 con 0x7f92580a39b0 2024-09-16T03:47:22.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.119+0000 7f925da1b640 5 monclient: authenticate success, global_id 13748 2024-09-16T03:47:22.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.119+0000 7f925da1b640 10 monclient: get_monmap_and_config success 2024-09-16T03:47:22.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.119+0000 7f925da1b640 4 set_mon_vals no callback set 2024-09-16T03:47:22.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.119+0000 7f9255ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:47:22.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.119+0000 7f9255ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:47:22.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.119+0000 7f9255ffb640 10 monclient: dump: 2024-09-16T03:47:22.130 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:47:22.130 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:22.131 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:22.131 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:22.131 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:47:22.131 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:22.131 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:47:22.131 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:47:22.131 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:47:22.131 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:22.131 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.119+0000 7f925da1b640 10 monclient: shutdown 2024-09-16T03:47:22.131 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.119+0000 7f925da1b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:47:22.131 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.119+0000 7f925da1b640 1 -- 172.21.15.42:0/1458841839 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f92580a39b0 msgr2=0x7f9258133b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:22.131 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.119+0000 7f925da1b640 1 --2- 172.21.15.42:0/1458841839 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f92580a39b0 0x7f9258133b80 secure :-1 s=READY pgs=2481 cs=0 l=1 rev1=1 crypto rx=0x7f9250009e50 tx=0x7f925001c8d0 comp rx=0 tx=0).stop 2024-09-16T03:47:22.131 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.119+0000 7f925da1b640 1 -- 172.21.15.42:0/1458841839 shutdown_connections 2024-09-16T03:47:22.131 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.119+0000 7f925da1b640 1 --2- 172.21.15.42:0/1458841839 >> v2:172.21.15.154:3300/0 conn(0x7f92580a3d90 0x7f9258137000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:22.132 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.119+0000 7f925da1b640 1 --2- 172.21.15.42:0/1458841839 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f92580a39b0 0x7f9258133b80 unknown :-1 s=CLOSED pgs=2481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:22.132 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.119+0000 7f925da1b640 1 -- 172.21.15.42:0/1458841839 >> 172.21.15.42:0/1458841839 conn(0x7f92580a9fb0 msgr2=0x7f92580aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:47:22.132 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.119+0000 7f925da1b640 1 -- 172.21.15.42:0/1458841839 shutdown_connections 2024-09-16T03:47:22.132 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.119+0000 7f925da1b640 1 -- 172.21.15.42:0/1458841839 wait complete. 2024-09-16T03:47:22.132 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.119+0000 7f925da1b640 10 monclient: build_initial_monmap 2024-09-16T03:47:22.132 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.119+0000 7f925da1b640 10 monclient: monmap: 2024-09-16T03:47:22.132 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:47:22.132 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:47:22.132 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:47:22.132 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:47:22.132 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:47:22.132 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:22.132 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:47:22.132 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:47:22.133 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:47:22.133 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:22.133 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.119+0000 7f925da1b640 1 librados: starting msgr at 2024-09-16T03:47:22.133 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.119+0000 7f925da1b640 1 librados: starting objecter 2024-09-16T03:47:22.133 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.119+0000 7f925da1b640 1 Processor -- start 2024-09-16T03:47:22.133 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.119+0000 7f925da1b640 1 -- start start 2024-09-16T03:47:22.133 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.119+0000 7f925da1b640 1 librados: setting wanted keys 2024-09-16T03:47:22.133 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.119+0000 7f925da1b640 1 librados: calling monclient init 2024-09-16T03:47:22.133 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.119+0000 7f925da1b640 10 monclient: init 2024-09-16T03:47:22.133 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.119+0000 7f925da1b640 10 monclient: _reopen_session rank -1 2024-09-16T03:47:22.133 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.119+0000 7f925da1b640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:47:22.133 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.119+0000 7f925da1b640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f92580a39b0 0x7f92581cf900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:22.133 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.119+0000 7f925da1b640 10 monclient(hunting): picked mon.noname-a con 0x7f92580a39b0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:47:22.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.119+0000 7f925da1b640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f92580a3d90 0x7f925813e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:22.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.119+0000 7f925da1b640 10 monclient(hunting): picked mon.noname-c con 0x7f92580a3d90 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:47:22.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.119+0000 7f925da1b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9258137540 0x7f925813ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:22.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.119+0000 7f925da1b640 10 monclient(hunting): picked mon.noname-b con 0x7f9258137540 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:47:22.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.119+0000 7f925da1b640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:22.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.119+0000 7f925da1b640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f92581424e0 con 0x7f92580a39b0 2024-09-16T03:47:22.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.119+0000 7f925da1b640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:22.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.119+0000 7f925da1b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f9258142660 con 0x7f9258137540 2024-09-16T03:47:22.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.119+0000 7f925da1b640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:22.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.119+0000 7f925da1b640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f9258138ee0 con 0x7f92580a3d90 2024-09-16T03:47:22.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.119+0000 7f925da1b640 10 monclient(hunting): _renew_subs 2024-09-16T03:47:22.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.119+0000 7f925da1b640 10 monclient(hunting): authenticate will time out at 1289.090474s 2024-09-16T03:47:22.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.119+0000 7f9256ffd640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f92580a39b0 0x7f92581cf900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:22.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.119+0000 7f92567fc640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f92580a3d90 0x7f925813e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:22.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.119+0000 7f9256ffd640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f92580a39b0 0x7f92581cf900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:60066/0 (socket says 172.21.15.42:60066) 2024-09-16T03:47:22.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.119+0000 7f9256ffd640 1 -- 172.21.15.42:0/3269272905 learned_addr learned my addr 172.21.15.42:0/3269272905 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:47:22.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.119+0000 7f9256ffd640 10 monclient(hunting): get_auth_request con 0x7f92580a39b0 auth_method 0 2024-09-16T03:47:22.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.119+0000 7f9256ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:22.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.119+0000 7f9256ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:22.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.119+0000 7f9256ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:47:22.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.119+0000 7f92567fc640 10 monclient(hunting): get_auth_request con 0x7f92580a3d90 auth_method 0 2024-09-16T03:47:22.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.119+0000 7f92567fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:22.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.119+0000 7f92567fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:22.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.119+0000 7f92567fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:47:22.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.119+0000 7f92577fe640 1 -- 172.21.15.42:0/3269272905 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9258137540 msgr2=0x7f925813ecf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T03:47:22.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.119+0000 7f92577fe640 1 --2- 172.21.15.42:0/3269272905 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9258137540 0x7f925813ecf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:47:22.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.119+0000 7f9256ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:47:22.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.119+0000 7f9256ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:47:22.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.119+0000 7f9256ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:47:22.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.119+0000 7f92567fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:47:22.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.119+0000 7f92567fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:47:22.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.119+0000 7f92567fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:47:22.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.119+0000 7f92567fc640 10 monclient(hunting): handle_auth_done global_id 13751 payload 824 2024-09-16T03:47:22.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.119+0000 7f92567fc640 1 -- 172.21.15.42:0/3269272905 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9258137540 msgr2=0x7f925813ecf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:47:22.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.119+0000 7f92567fc640 1 --2- 172.21.15.42:0/3269272905 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9258137540 0x7f925813ecf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:22.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.119+0000 7f92567fc640 1 -- 172.21.15.42:0/3269272905 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f92580a39b0 msgr2=0x7f92581cf900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:22.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.119+0000 7f92567fc640 1 --2- 172.21.15.42:0/3269272905 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f92580a39b0 0x7f92581cf900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:22.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.119+0000 7f92567fc640 10 monclient: _finish_hunting 0 2024-09-16T03:47:22.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.119+0000 7f92567fc640 1 monclient: found mon.noname-c 2024-09-16T03:47:22.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.119+0000 7f92567fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:47:22.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.119+0000 7f92567fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:47:22.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.119+0000 7f92567fc640 1 -- 172.21.15.42:0/3269272905 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9258138400 con 0x7f92580a3d90 2024-09-16T03:47:22.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.119+0000 7f9256ffd640 1 --2- 172.21.15.42:0/3269272905 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f92580a39b0 0x7f92581cf900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:47:22.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.119+0000 7f92567fc640 1 --2- 172.21.15.42:0/3269272905 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f92580a3d90 0x7f925813e7b0 secure :-1 s=READY pgs=2482 cs=0 l=1 rev1=1 crypto rx=0x7f924000b4a0 tx=0x7f924000b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:47:22.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.119+0000 7f924ffff640 10 client.?.objecter ms_handle_connect 0x7f92580a3d90 2024-09-16T03:47:22.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.119+0000 7f924ffff640 10 client.?.objecter resend_mon_ops 2024-09-16T03:47:22.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.119+0000 7f924ffff640 1 -- 172.21.15.42:0/3269272905 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9240003bd0 con 0x7f92580a3d90 2024-09-16T03:47:22.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.119+0000 7f924ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:47:22.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.119+0000 7f924ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:47:22.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.119+0000 7f924ffff640 10 monclient: dump: 2024-09-16T03:47:22.137 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:47:22.137 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:22.137 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:22.137 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:22.137 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:47:22.138 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:22.138 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:47:22.138 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:47:22.138 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:47:22.138 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:22.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.119+0000 7f924ffff640 10 monclient: _finish_auth 0 2024-09-16T03:47:22.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.119+0000 7f924ffff640 10 monclient: _check_auth_tickets 2024-09-16T03:47:22.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.119+0000 7f924ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:47:22.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.119+0000 7f924ffff640 1 -- 172.21.15.42:0/3269272905 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9240015070 con 0x7f92580a3d90 2024-09-16T03:47:22.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.119+0000 7f924ffff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:47:22.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.119+0000 7f924ffff640 1 -- 172.21.15.42:0/3269272905 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9240005070 con 0x7f92580a3d90 2024-09-16T03:47:22.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.119+0000 7f925da1b640 5 monclient: authenticate success, global_id 13751 2024-09-16T03:47:22.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.119+0000 7f925da1b640 10 monclient: _renew_subs 2024-09-16T03:47:22.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.119+0000 7f925da1b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:47:22.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.119+0000 7f925da1b640 1 -- 172.21.15.42:0/3269272905 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9258139c90 con 0x7f92580a3d90 2024-09-16T03:47:22.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.119+0000 7f92557fa640 4 set_mon_vals no callback set 2024-09-16T03:47:22.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.119+0000 7f924ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:47:22.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.119+0000 7f924ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:47:22.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.119+0000 7f924ffff640 10 monclient: dump: 2024-09-16T03:47:22.139 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:47:22.139 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:22.139 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:22.139 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:22.139 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:47:22.139 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:22.140 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:47:22.140 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:47:22.140 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:47:22.140 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:22.140 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.119+0000 7f925da1b640 10 client.13751.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:47:22.140 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.119+0000 7f925da1b640 10 monclient: _renew_subs 2024-09-16T03:47:22.140 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.119+0000 7f925da1b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:47:22.140 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.119+0000 7f925da1b640 1 -- 172.21.15.42:0/3269272905 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f92581cfe40 con 0x7f92580a3d90 2024-09-16T03:47:22.140 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.119+0000 7f925da1b640 1 librados: init done 2024-09-16T03:47:22.140 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.123+0000 7f925da1b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:47:22.140 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.123+0000 7f925da1b640 1 --2- 172.21.15.42:0/3269272905 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f92580a1b00 0x7f92580a1ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:22.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.123+0000 7f925da1b640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:22.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.123+0000 7f925da1b640 1 -- 172.21.15.42:0/3269272905 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f92581342d0 con 0x7f92580a1b00 2024-09-16T03:47:22.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.123+0000 7f924ffff640 1 -- 172.21.15.42:0/3269272905 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f9240005210 con 0x7f92580a3d90 2024-09-16T03:47:22.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.123+0000 7f9256ffd640 1 --2- 172.21.15.42:0/3269272905 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f92580a1b00 0x7f92580a1ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:22.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.123+0000 7f9256ffd640 10 monclient: get_auth_request con 0x7f92580a1b00 auth_method 0 2024-09-16T03:47:22.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.123+0000 7f9256ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:22.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.123+0000 7f9256ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:22.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.123+0000 7f9256ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:47:22.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.123+0000 7f9256ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:47:22.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.123+0000 7f9256ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:47:22.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.123+0000 7f9256ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:47:22.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.123+0000 7f924ffff640 1 --2- 172.21.15.42:0/3269272905 >> v2:172.21.15.154:6812/254845342 conn(0x7f92380377c0 0x7f9238039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:22.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.123+0000 7f924ffff640 1 -- 172.21.15.42:0/3269272905 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(151..151 src has 1..151) ==== 6739+0+0 (secure 0 0 0) 0x7f9240048be0 con 0x7f92580a3d90 2024-09-16T03:47:22.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.123+0000 7f924ffff640 10 client.13751.objecter ms_dispatch 0x7f9258000ba0 osd_map(151..151 src has 1..151) 2024-09-16T03:47:22.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.123+0000 7f924ffff640 3 client.13751.objecter handle_osd_map got epochs [151,151] > 0 2024-09-16T03:47:22.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.123+0000 7f924ffff640 3 client.13751.objecter handle_osd_map decoding full epoch 151 2024-09-16T03:47:22.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.123+0000 7f9256ffd640 10 monclient(hunting): handle_auth_done global_id 764205 payload 824 2024-09-16T03:47:22.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.123+0000 7f9256ffd640 1 -- 172.21.15.42:0/3269272905 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9258133b80 con 0x7f92580a1b00 2024-09-16T03:47:22.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.123+0000 7f924ffff640 20 client.13751.objecter dump_active .. 0 homeless 2024-09-16T03:47:22.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.123+0000 7f924ffff640 10 client.13751.objecter ms_handle_connect 0x7f92580a1b00 2024-09-16T03:47:22.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.123+0000 7f924ffff640 10 client.13751.objecter resend_mon_ops 2024-09-16T03:47:22.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.123+0000 7f9256ffd640 1 --2- 172.21.15.42:0/3269272905 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f92580a1b00 0x7f92580a1ee0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f925001d170 tx=0x7f9250019670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:47:22.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.123+0000 7f92577fe640 1 --2- 172.21.15.42:0/3269272905 >> v2:172.21.15.154:6812/254845342 conn(0x7f92380377c0 0x7f9238039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:22.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.123+0000 7f92577fe640 10 monclient: get_auth_request con 0x7f92380377c0 auth_method 0 2024-09-16T03:47:22.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.123+0000 7f924ffff640 1 -- 172.21.15.42:0/3269272905 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92500c0640 con 0x7f92580a1b00 2024-09-16T03:47:22.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.123+0000 7f924ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:47:22.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.123+0000 7f924ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:47:22.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.123+0000 7f924ffff640 10 monclient: dump: 2024-09-16T03:47:22.143 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:47:22.143 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:22.143 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:22.143 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:22.143 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:47:22.143 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:22.143 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:47:22.143 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:47:22.143 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:47:22.143 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:22.143 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.123+0000 7f924ffff640 1 -- 172.21.15.42:0/3269272905 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f92500c0640 con 0x7f92580a1b00 2024-09-16T03:47:22.143 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.123+0000 7f924ffff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:47:22.143 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.123+0000 7f924ffff640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:47:22.143 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.123+0000 7f924ffff640 1 -- 172.21.15.42:0/3269272905 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f92580a1b00 msgr2=0x7f92580a1ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:22.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.123+0000 7f924ffff640 1 --2- 172.21.15.42:0/3269272905 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f92580a1b00 0x7f92580a1ee0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f925001d170 tx=0x7f9250019670 comp rx=0 tx=0).stop 2024-09-16T03:47:22.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.123+0000 7f924ffff640 1 -- 172.21.15.42:0/3269272905 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f92580a1b00 msgr2=0x7f92580a1ee0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:47:22.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.123+0000 7f924ffff640 1 --2- 172.21.15.42:0/3269272905 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f92580a1b00 0x7f92580a1ee0 secure :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f925001d170 tx=0x7f9250019670 comp rx=0 tx=0).stop 2024-09-16T03:47:22.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.123+0000 7f92577fe640 1 --2- 172.21.15.42:0/3269272905 >> v2:172.21.15.154:6812/254845342 conn(0x7f92380377c0 0x7f9238039c80 secure :-1 s=READY pgs=3443 cs=0 l=1 rev1=1 crypto rx=0x7f9248003940 tx=0x7f9248006c80 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:47:22.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.123+0000 7f924ffff640 10 client.13751.objecter ms_handle_connect 0x7f92380377c0 2024-09-16T03:47:22.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.127+0000 7f925da1b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:47:22.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.127+0000 7f925da1b640 1 --2- 172.21.15.42:0/3269272905 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f92580a2420 0x7f925809d910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:22.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.127+0000 7f925da1b640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:22.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.127+0000 7f925da1b640 1 -- 172.21.15.42:0/3269272905 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f925806df10 con 0x7f92580a2420 2024-09-16T03:47:22.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.127+0000 7f9256ffd640 1 --2- 172.21.15.42:0/3269272905 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f92580a2420 0x7f925809d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:22.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.127+0000 7f9256ffd640 10 monclient: get_auth_request con 0x7f92580a2420 auth_method 0 2024-09-16T03:47:22.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.127+0000 7f9256ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:22.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.127+0000 7f9256ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:22.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.127+0000 7f9256ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:47:22.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.131+0000 7f9256ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:47:22.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.131+0000 7f9256ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:47:22.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.131+0000 7f9256ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:47:22.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.131+0000 7f9256ffd640 10 monclient(hunting): handle_auth_done global_id 764208 payload 824 2024-09-16T03:47:22.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.131+0000 7f9256ffd640 1 -- 172.21.15.42:0/3269272905 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f92580a5750 con 0x7f92580a2420 2024-09-16T03:47:22.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.131+0000 7f9256ffd640 1 --2- 172.21.15.42:0/3269272905 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f92580a2420 0x7f925809d910 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f925001c910 tx=0x7f92500a7000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:47:22.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.131+0000 7f924ffff640 10 client.13751.objecter ms_handle_connect 0x7f92580a2420 2024-09-16T03:47:22.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.131+0000 7f924ffff640 10 client.13751.objecter resend_mon_ops 2024-09-16T03:47:22.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.131+0000 7f924ffff640 1 -- 172.21.15.42:0/3269272905 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92500c9070 con 0x7f92580a2420 2024-09-16T03:47:22.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.131+0000 7f924ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:47:22.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.131+0000 7f924ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:47:22.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.131+0000 7f924ffff640 10 monclient: dump: 2024-09-16T03:47:22.147 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:47:22.147 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:22.148 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:22.148 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:22.148 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:47:22.148 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:22.148 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:47:22.148 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:47:22.148 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:47:22.148 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:22.148 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.131+0000 7f924ffff640 1 -- 172.21.15.42:0/3269272905 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4535 (secure 0 0 0) 0x7f92500c0cf0 con 0x7f92580a2420 2024-09-16T03:47:22.148 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.131+0000 7f924ffff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:47:22.148 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.131+0000 7f924ffff640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:47:22.148 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.131+0000 7f924ffff640 1 -- 172.21.15.42:0/3269272905 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f92580a2420 msgr2=0x7f925809d910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:22.148 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.131+0000 7f924ffff640 1 --2- 172.21.15.42:0/3269272905 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f92580a2420 0x7f925809d910 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f925001c910 tx=0x7f92500a7000 comp rx=0 tx=0).stop 2024-09-16T03:47:22.148 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.131+0000 7f924ffff640 1 -- 172.21.15.42:0/3269272905 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f92580a2420 msgr2=0x7f925809d910 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:47:22.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.131+0000 7f924ffff640 1 --2- 172.21.15.42:0/3269272905 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f92580a2420 0x7f925809d910 secure :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f925001c910 tx=0x7f92500a7000 comp rx=0 tx=0).stop 2024-09-16T03:47:22.149 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:47:22.149 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:47:22.149 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:47:22.149 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "leader", 2024-09-16T03:47:22.149 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 52, 2024-09-16T03:47:22.149 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 32923, 2024-09-16T03:47:22.149 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:47:22.149 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:47:22.149 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T03:47:22.150 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:47:22.150 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 1, 2024-09-16T03:47:22.150 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:47:22.150 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:47:22.150 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:47:22.150 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:47:22.150 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:47:22.150 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:47:22.150 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:47:22.150 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:47:22.150 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:47:22.150 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:47:22.150 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:47:22.150 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:47:22.151 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:47:22.151 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:47:22.151 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:47:22.151 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:47:22.151 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:47:22.151 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:47:22.151 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:47:22.151 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:47:22.151 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:47:22.151 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:47:22.151 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:47:22.151 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:47:22.151 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:47:22.151 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:47:22.152 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:47:22.152 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:47:22.152 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:47:22.152 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:22.152 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:47:22.152 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:47:22.152 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:47:22.152 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:47:22.152 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:47:22.152 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:47:22.152 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:47:22.152 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:47:22.152 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:47:22.152 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:47:22.153 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:47:22.153 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:47:22.153 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:47:22.153 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:47:22.153 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:47:22.153 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:47:22.153 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:47:22.153 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:47:22.153 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:47:22.153 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:47:22.153 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:47:22.153 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:47:22.153 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:47:22.153 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:47:22.154 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:47:22.154 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:47:22.154 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:47:22.154 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:47:22.154 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:47:22.154 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:47:22.154 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:22.154 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:47:22.154 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:22.154 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:47:22.154 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:47:22.154 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:47:22.154 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:47:22.154 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:22.155 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:47:22.155 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:47:22.155 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:47:22.155 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:22.155 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:22.155 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:47:22.155 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:47:22.155 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:47:22.155 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:47:22.155 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:47:22.155 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:22.155 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:47:22.155 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:47:22.155 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:47:22.156 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:47:22.156 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:47:22.156 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:22.156 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:22.156 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:47:22.156 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:47:22.156 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:47:22.156 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:47:22.156 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:22.156 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:47:22.156 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:47:22.156 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:47:22.156 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:22.156 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:22.156 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:47:22.157 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:47:22.157 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:47:22.157 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:47:22.157 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:47:22.157 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:22.157 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:47:22.157 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:47:22.157 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:47:22.157 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:47:22.158 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:47:22.158 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:22.158 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:22.158 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:47:22.158 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:47:22.158 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:47:22.158 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:47:22.158 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:22.158 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:47:22.158 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:47:22.158 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:47:22.158 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:22.158 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:22.158 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:47:22.158 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:47:22.159 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:47:22.159 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:47:22.159 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:47:22.159 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:22.159 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:47:22.159 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:47:22.159 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:47:22.159 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:47:22.159 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:47:22.159 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:47:22.159 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:47:22.159 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:22.159 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:47:22.159 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:47:22.160 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:22.160 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:47:22.160 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:47:22.160 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:47:22.160 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:47:22.160 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:47:22.160 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:47:22.160 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:22.160 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:47:22.160 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:47:22.160 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 3 2024-09-16T03:47:22.160 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:47:22.160 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:47:22.161 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:22.161 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:47:22.161 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:47:22.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.131+0000 7f925da1b640 10 librados: watch_flush enter 2024-09-16T03:47:22.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.131+0000 7f925da1b640 10 librados: watch_flush exit 2024-09-16T03:47:22.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.131+0000 7f925da1b640 20 client.13751.objecter shutdown clearing up homeless session... 2024-09-16T03:47:22.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.131+0000 7f925da1b640 10 client.13751.objecter successfully canceled tick 2024-09-16T03:47:22.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.131+0000 7f925da1b640 1 -- 172.21.15.42:0/3269272905 >> v2:172.21.15.154:6812/254845342 conn(0x7f92380377c0 msgr2=0x7f9238039c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:22.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.131+0000 7f925da1b640 1 --2- 172.21.15.42:0/3269272905 >> v2:172.21.15.154:6812/254845342 conn(0x7f92380377c0 0x7f9238039c80 secure :-1 s=READY pgs=3443 cs=0 l=1 rev1=1 crypto rx=0x7f9248003940 tx=0x7f9248006c80 comp rx=0 tx=0).stop 2024-09-16T03:47:22.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.131+0000 7f925da1b640 10 monclient: shutdown 2024-09-16T03:47:22.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.131+0000 7f925da1b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:47:22.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.131+0000 7f925da1b640 1 -- 172.21.15.42:0/3269272905 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f92580a3d90 msgr2=0x7f925813e7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:22.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.131+0000 7f925da1b640 1 --2- 172.21.15.42:0/3269272905 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f92580a3d90 0x7f925813e7b0 secure :-1 s=READY pgs=2482 cs=0 l=1 rev1=1 crypto rx=0x7f924000b4a0 tx=0x7f924000b960 comp rx=0 tx=0).stop 2024-09-16T03:47:22.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.131+0000 7f92577fe640 1 -- 172.21.15.42:0/3269272905 reap_dead start 2024-09-16T03:47:22.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.131+0000 7f925da1b640 1 -- 172.21.15.42:0/3269272905 shutdown_connections 2024-09-16T03:47:22.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.131+0000 7f925da1b640 1 --2- 172.21.15.42:0/3269272905 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f92580a3d90 0x7f925813e7b0 unknown :-1 s=CLOSED pgs=2482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:22.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.131+0000 7f925da1b640 1 -- 172.21.15.42:0/3269272905 >> 172.21.15.42:0/3269272905 conn(0x7f92580a9fb0 msgr2=0x7f9258131980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:47:22.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.131+0000 7f925da1b640 1 -- 172.21.15.42:0/3269272905 shutdown_connections 2024-09-16T03:47:22.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.131+0000 7f925da1b640 1 -- 172.21.15.42:0/3269272905 wait complete. 2024-09-16T03:47:22.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.131+0000 7f925da1b640 1 librados: shutdown 2024-09-16T03:47:22.164 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T03:47:22.164 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-16T03:47:22.164 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T03:47:22.281 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.275+0000 7f2400978640 10 monclient: get_monmap_and_config 2024-09-16T03:47:22.281 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.279+0000 7f2400978640 10 monclient: build_initial_monmap 2024-09-16T03:47:22.281 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.279+0000 7f2400978640 10 monclient: monmap: 2024-09-16T03:47:22.281 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:47:22.281 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:22.281 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:47:22.280162+0000 2024-09-16T03:47:22.281 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:47:22.280162+0000 2024-09-16T03:47:22.281 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:47:22.281 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:22.281 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:47:22.281 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:47:22.282 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:47:22.282 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:47:22.282 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:47:22.282 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:22.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.279+0000 7f2400978640 1 Processor -- start 2024-09-16T03:47:22.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.279+0000 7f2400978640 1 -- start start 2024-09-16T03:47:22.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.279+0000 7f2400978640 10 monclient: init 2024-09-16T03:47:22.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.279+0000 7f2400978640 10 monclient: _reopen_session rank -1 2024-09-16T03:47:22.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.279+0000 7f2400978640 10 monclient: _add_conns ranks=[1,2,0,4,3] 2024-09-16T03:47:22.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.279+0000 7f2400978640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f23fc073080 addr v1:172.21.15.154:6789/0 2024-09-16T03:47:22.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.279+0000 7f2400978640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f23fc09f670 0x7f23fc139fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:22.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.279+0000 7f2400978640 10 monclient(hunting): picked mon.noname-b con 0x7f23fc09f670 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:47:22.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.279+0000 7f2400978640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f23fc13a510 addr v1:172.21.15.42:6789/0 2024-09-16T03:47:22.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.279+0000 7f2400978640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f23fc0a1cb0 con 0x7f23fc13a510 2024-09-16T03:47:22.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.279+0000 7f2400978640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f23fc0a1f90 con 0x7f23fc073080 2024-09-16T03:47:22.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.279+0000 7f2400978640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:22.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.279+0000 7f2400978640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f23fc075d40 con 0x7f23fc09f670 2024-09-16T03:47:22.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.279+0000 7f2400978640 10 monclient(hunting): _renew_subs 2024-09-16T03:47:22.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.279+0000 7f2400978640 10 monclient(hunting): authenticate will time out at 1289.250439s 2024-09-16T03:47:22.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.279+0000 7f23fa59c640 1 -- >> v1:172.21.15.154:6789/0 conn(0x7f23fc073080 legacy=0x7f23fc073460 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.154:6789/0 2024-09-16T03:47:22.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.279+0000 7f23fad9d640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f23fc13a510 0x7f23fc13c900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:48674/0 (socket says 172.21.15.42:48674) 2024-09-16T03:47:22.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.279+0000 7f23fad9d640 1 -- 172.21.15.42:0/3245318073 learned_addr learned my addr 172.21.15.42:0/3245318073 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:47:22.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.279+0000 7f23f1d9b640 1 --2- 172.21.15.42:0/3245318073 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f23fc09f670 0x7f23fc139fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:22.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.279+0000 7f23f1d9b640 10 monclient(hunting): get_auth_request con 0x7f23fc09f670 auth_method 0 2024-09-16T03:47:22.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.279+0000 7f23f1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:22.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.279+0000 7f23f1d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:22.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.279+0000 7f23f1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:47:22.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.279+0000 7f23f1d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:47:22.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.279+0000 7f23f1d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:47:22.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.279+0000 7f23f1d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:47:22.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.279+0000 7f23f1d9b640 10 monclient(hunting): handle_auth_done global_id 13754 payload 274 2024-09-16T03:47:22.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.279+0000 7f23f1d9b640 1 -- 172.21.15.42:0/3245318073 >> v1:172.21.15.154:6789/0 conn(0x7f23fc073080 legacy=0x7f23fc073460 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:47:22.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.279+0000 7f23f1d9b640 1 -- 172.21.15.42:0/3245318073 >> v1:172.21.15.42:6789/0 conn(0x7f23fc13a510 legacy=0x7f23fc13c900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:22.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.279+0000 7f23f1d9b640 10 monclient: _finish_hunting 0 2024-09-16T03:47:22.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.279+0000 7f23f1d9b640 1 monclient: found mon.noname-b 2024-09-16T03:47:22.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.279+0000 7f23f1d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:47:22.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.279+0000 7f23f1d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T03:47:22.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.279+0000 7f23f1d9b640 1 -- 172.21.15.42:0/3245318073 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23fc0a2690 con 0x7f23fc09f670 2024-09-16T03:47:22.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.283+0000 7f23f1d9b640 1 --2- 172.21.15.42:0/3245318073 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f23fc09f670 0x7f23fc139fd0 secure :-1 s=READY pgs=2483 cs=0 l=1 rev1=1 crypto rx=0x7f23e40097f0 tx=0x7f23e401c360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6352903375a9b783 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:47:22.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.283+0000 7f23f9d9b640 1 -- 172.21.15.42:0/3245318073 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23e40b0070 con 0x7f23fc09f670 2024-09-16T03:47:22.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.283+0000 7f23f9d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:47:22.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.283+0000 7f23f9d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:47:22.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.283+0000 7f23f9d9b640 10 monclient: dump: 2024-09-16T03:47:22.288 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:47:22.288 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:22.288 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:22.288 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:22.288 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:47:22.288 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:22.288 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:47:22.288 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:47:22.288 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:47:22.288 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:22.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.283+0000 7f23f9d9b640 10 monclient: _finish_auth 0 2024-09-16T03:47:22.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.283+0000 7f23f9d9b640 10 monclient: _check_auth_tickets 2024-09-16T03:47:22.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.283+0000 7f23f9d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:47:22.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.283+0000 7f23f9d9b640 1 -- 172.21.15.42:0/3245318073 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f23e40ab600 con 0x7f23fc09f670 2024-09-16T03:47:22.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.283+0000 7f2400978640 5 monclient: authenticate success, global_id 13754 2024-09-16T03:47:22.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.283+0000 7f2400978640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:47:22.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.283+0000 7f23f9d9b640 10 monclient: handle_config config(0 keys) 2024-09-16T03:47:22.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.283+0000 7f2400978640 10 monclient: get_monmap_and_config success 2024-09-16T03:47:22.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.283+0000 7f2400978640 4 set_mon_vals no callback set 2024-09-16T03:47:22.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.283+0000 7f23f9d9b640 1 -- 172.21.15.42:0/3245318073 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23e40ab9d0 con 0x7f23fc09f670 2024-09-16T03:47:22.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.283+0000 7f23f9d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:47:22.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.283+0000 7f23f9d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:47:22.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.283+0000 7f23f9d9b640 10 monclient: dump: 2024-09-16T03:47:22.289 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:47:22.289 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:22.289 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:22.290 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:22.290 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:47:22.290 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:22.290 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:47:22.290 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:47:22.290 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:47:22.290 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:22.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.283+0000 7f2400978640 10 monclient: shutdown 2024-09-16T03:47:22.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.283+0000 7f2400978640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:47:22.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.283+0000 7f2400978640 1 -- 172.21.15.42:0/3245318073 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f23fc09f670 msgr2=0x7f23fc139fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:22.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.283+0000 7f2400978640 1 --2- 172.21.15.42:0/3245318073 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f23fc09f670 0x7f23fc139fd0 secure :-1 s=READY pgs=2483 cs=0 l=1 rev1=1 crypto rx=0x7f23e40097f0 tx=0x7f23e401c360 comp rx=0 tx=0).stop 2024-09-16T03:47:22.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.283+0000 7f2400978640 1 -- 172.21.15.42:0/3245318073 shutdown_connections 2024-09-16T03:47:22.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.283+0000 7f2400978640 1 --2- 172.21.15.42:0/3245318073 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f23fc09f670 0x7f23fc139fd0 unknown :-1 s=CLOSED pgs=2483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:22.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.283+0000 7f2400978640 1 -- 172.21.15.42:0/3245318073 >> 172.21.15.42:0/3245318073 conn(0x7f23fc06bc40 msgr2=0x7f23fc06c050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:47:22.291 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.283+0000 7f2400978640 1 -- 172.21.15.42:0/3245318073 shutdown_connections 2024-09-16T03:47:22.291 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.283+0000 7f2400978640 1 -- 172.21.15.42:0/3245318073 wait complete. 2024-09-16T03:47:22.291 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.283+0000 7f2400978640 10 monclient: build_initial_monmap 2024-09-16T03:47:22.291 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.283+0000 7f2400978640 10 monclient: monmap: 2024-09-16T03:47:22.291 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:47:22.291 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:47:22.291 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:47:22.291 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:47:22.291 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:47:22.291 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:22.291 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:47:22.291 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:47:22.291 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:47:22.291 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:22.296 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.283+0000 7f2400978640 1 librados: starting msgr at 2024-09-16T03:47:22.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.283+0000 7f2400978640 1 librados: starting objecter 2024-09-16T03:47:22.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.283+0000 7f2400978640 1 Processor -- start 2024-09-16T03:47:22.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.283+0000 7f2400978640 1 -- start start 2024-09-16T03:47:22.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.283+0000 7f2400978640 1 librados: setting wanted keys 2024-09-16T03:47:22.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.283+0000 7f2400978640 1 librados: calling monclient init 2024-09-16T03:47:22.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.283+0000 7f2400978640 10 monclient: init 2024-09-16T03:47:22.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.283+0000 7f2400978640 10 monclient: _reopen_session rank -1 2024-09-16T03:47:22.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.283+0000 7f2400978640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:47:22.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.283+0000 7f2400978640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f23fc073080 0x7f23fc1cef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:22.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.283+0000 7f2400978640 10 monclient(hunting): picked mon.noname-c con 0x7f23fc073080 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:47:22.298 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.283+0000 7f2400978640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f23fc13a510 0x7f23fc130ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:22.298 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.283+0000 7f2400978640 10 monclient(hunting): picked mon.noname-a con 0x7f23fc13a510 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:47:22.298 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.283+0000 7f2400978640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f23fc130ff0 0x7f23fc137ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:22.298 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.283+0000 7f2400978640 10 monclient(hunting): picked mon.noname-b con 0x7f23fc130ff0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:47:22.298 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.283+0000 7f2400978640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:22.298 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.283+0000 7f2400978640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f23fc140bd0 con 0x7f23fc13a510 2024-09-16T03:47:22.298 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.283+0000 7f2400978640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:22.298 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.283+0000 7f2400978640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f23fc140a50 con 0x7f23fc130ff0 2024-09-16T03:47:22.298 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.283+0000 7f2400978640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:22.298 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.283+0000 7f2400978640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f23fc140d50 con 0x7f23fc073080 2024-09-16T03:47:22.298 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.283+0000 7f2400978640 10 monclient(hunting): _renew_subs 2024-09-16T03:47:22.298 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.283+0000 7f2400978640 10 monclient(hunting): authenticate will time out at 1289.255954s 2024-09-16T03:47:22.298 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.287+0000 7f23fa59c640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f23fc073080 0x7f23fc1cef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:22.298 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.287+0000 7f23f1d9b640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f23fc13a510 0x7f23fc130ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:22.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.287+0000 7f23fa59c640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f23fc073080 0x7f23fc1cef50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:36928/0 (socket says 172.21.15.42:36928) 2024-09-16T03:47:22.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.287+0000 7f23fa59c640 1 -- 172.21.15.42:0/3475651053 learned_addr learned my addr 172.21.15.42:0/3475651053 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:47:22.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.287+0000 7f23fa59c640 10 monclient(hunting): get_auth_request con 0x7f23fc073080 auth_method 0 2024-09-16T03:47:22.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.287+0000 7f23fa59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:22.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.287+0000 7f23fa59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:22.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.287+0000 7f23fa59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:47:22.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.287+0000 7f23f1d9b640 10 monclient(hunting): get_auth_request con 0x7f23fc13a510 auth_method 0 2024-09-16T03:47:22.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.287+0000 7f23f1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:22.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.287+0000 7f23f1d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:22.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.287+0000 7f23f1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:47:22.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.287+0000 7f23fa59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:47:22.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.287+0000 7f23fa59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:47:22.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.287+0000 7f23fa59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:47:22.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.287+0000 7f23fad9d640 1 -- 172.21.15.42:0/3475651053 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f23fc130ff0 msgr2=0x7f23fc137ba0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T03:47:22.300 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.287+0000 7f23fad9d640 1 --2- 172.21.15.42:0/3475651053 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f23fc130ff0 0x7f23fc137ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:47:22.300 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.287+0000 7f23f1d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:47:22.300 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.287+0000 7f23f1d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:47:22.300 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.287+0000 7f23f1d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:47:22.300 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.287+0000 7f23fa59c640 10 monclient(hunting): handle_auth_done global_id 13757 payload 824 2024-09-16T03:47:22.300 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.287+0000 7f23fa59c640 1 -- 172.21.15.42:0/3475651053 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f23fc130ff0 msgr2=0x7f23fc137ba0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:47:22.300 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.287+0000 7f23fa59c640 1 --2- 172.21.15.42:0/3475651053 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f23fc130ff0 0x7f23fc137ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:22.300 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.287+0000 7f23fa59c640 1 -- 172.21.15.42:0/3475651053 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f23fc13a510 msgr2=0x7f23fc130ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:22.300 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.287+0000 7f23fa59c640 1 --2- 172.21.15.42:0/3475651053 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f23fc13a510 0x7f23fc130ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:22.300 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.287+0000 7f23fa59c640 10 monclient: _finish_hunting 0 2024-09-16T03:47:22.300 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.287+0000 7f23fa59c640 1 monclient: found mon.noname-c 2024-09-16T03:47:22.300 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.287+0000 7f23fa59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:47:22.300 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.287+0000 7f23fa59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:47:22.301 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.287+0000 7f23fa59c640 1 -- 172.21.15.42:0/3475651053 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23fc09ef10 con 0x7f23fc073080 2024-09-16T03:47:22.301 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.287+0000 7f23fa59c640 1 --2- 172.21.15.42:0/3475651053 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f23fc073080 0x7f23fc1cef50 secure :-1 s=READY pgs=2484 cs=0 l=1 rev1=1 crypto rx=0x7f23ec00b530 tx=0x7f23ec00b9f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:47:22.301 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.287+0000 7f23f37fe640 10 client.?.objecter ms_handle_connect 0x7f23fc073080 2024-09-16T03:47:22.301 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.287+0000 7f23f37fe640 10 client.?.objecter resend_mon_ops 2024-09-16T03:47:22.301 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.287+0000 7f23f37fe640 1 -- 172.21.15.42:0/3475651053 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23ec013020 con 0x7f23fc073080 2024-09-16T03:47:22.301 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.287+0000 7f23f37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:47:22.301 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.287+0000 7f23f37fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:47:22.301 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.287+0000 7f23f37fe640 10 monclient: dump: 2024-09-16T03:47:22.301 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:47:22.301 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:22.301 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:22.301 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:22.301 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:47:22.301 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:22.302 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:47:22.302 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:47:22.302 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:47:22.302 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:22.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.287+0000 7f23f37fe640 10 monclient: _finish_auth 0 2024-09-16T03:47:22.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.287+0000 7f23f37fe640 10 monclient: _check_auth_tickets 2024-09-16T03:47:22.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.287+0000 7f23f37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:47:22.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.287+0000 7f23f37fe640 1 -- 172.21.15.42:0/3475651053 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f23ec003df0 con 0x7f23fc073080 2024-09-16T03:47:22.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.287+0000 7f23f37fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:47:22.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.287+0000 7f23f37fe640 1 -- 172.21.15.42:0/3475651053 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23ec0052b0 con 0x7f23fc073080 2024-09-16T03:47:22.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.287+0000 7f2400978640 5 monclient: authenticate success, global_id 13757 2024-09-16T03:47:22.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.287+0000 7f2400978640 10 monclient: _renew_subs 2024-09-16T03:47:22.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.287+0000 7f2400978640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:47:22.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.287+0000 7f2400978640 1 -- 172.21.15.42:0/3475651053 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23fc0a53c0 con 0x7f23fc073080 2024-09-16T03:47:22.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.287+0000 7f23f959a640 4 set_mon_vals no callback set 2024-09-16T03:47:22.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.287+0000 7f23f37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:47:22.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.287+0000 7f23f37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:47:22.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.287+0000 7f23f37fe640 10 monclient: dump: 2024-09-16T03:47:22.303 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:47:22.303 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:22.303 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:22.303 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:22.303 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:47:22.303 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:22.303 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:47:22.303 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:47:22.303 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:47:22.303 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:22.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.287+0000 7f2400978640 10 client.13757.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:47:22.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.287+0000 7f2400978640 10 monclient: _renew_subs 2024-09-16T03:47:22.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.287+0000 7f2400978640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:47:22.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.287+0000 7f2400978640 1 -- 172.21.15.42:0/3475651053 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f23fc09f1f0 con 0x7f23fc073080 2024-09-16T03:47:22.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.287+0000 7f2400978640 1 librados: init done 2024-09-16T03:47:22.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.287+0000 7f2400978640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:47:22.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.287+0000 7f2400978640 1 --2- 172.21.15.42:0/3475651053 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f23fc09f670 0x7f23fc02c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:22.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.287+0000 7f2400978640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:22.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.287+0000 7f2400978640 1 -- 172.21.15.42:0/3475651053 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f23fc09d150 con 0x7f23fc09f670 2024-09-16T03:47:22.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.287+0000 7f23f37fe640 1 -- 172.21.15.42:0/3475651053 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f23ec016020 con 0x7f23fc073080 2024-09-16T03:47:22.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.287+0000 7f23f1d9b640 1 --2- 172.21.15.42:0/3475651053 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f23fc09f670 0x7f23fc02c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:22.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.287+0000 7f23f37fe640 1 --2- 172.21.15.42:0/3475651053 >> v2:172.21.15.154:6812/254845342 conn(0x7f23dc0377c0 0x7f23dc039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:22.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.287+0000 7f23f1d9b640 10 monclient: get_auth_request con 0x7f23fc09f670 auth_method 0 2024-09-16T03:47:22.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.287+0000 7f23f1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:22.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.287+0000 7f23f1d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:22.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.287+0000 7f23f1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:47:22.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.287+0000 7f23f37fe640 1 -- 172.21.15.42:0/3475651053 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(152..152 src has 1..152) ==== 6861+0+0 (secure 0 0 0) 0x7f23ec048f20 con 0x7f23fc073080 2024-09-16T03:47:22.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.287+0000 7f23f37fe640 10 client.13757.objecter ms_dispatch 0x7f23fc000ba0 osd_map(152..152 src has 1..152) 2024-09-16T03:47:22.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.287+0000 7f23f37fe640 3 client.13757.objecter handle_osd_map got epochs [152,152] > 0 2024-09-16T03:47:22.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.287+0000 7f23f37fe640 3 client.13757.objecter handle_osd_map decoding full epoch 152 2024-09-16T03:47:22.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.287+0000 7f23f1d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:47:22.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.287+0000 7f23f1d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:47:22.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.287+0000 7f23f1d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:47:22.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.287+0000 7f23f37fe640 20 client.13757.objecter dump_active .. 0 homeless 2024-09-16T03:47:22.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.287+0000 7f23f1d9b640 10 monclient(hunting): handle_auth_done global_id 13760 payload 824 2024-09-16T03:47:22.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.287+0000 7f23f1d9b640 1 -- 172.21.15.42:0/3475651053 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f23fc073460 con 0x7f23fc09f670 2024-09-16T03:47:22.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.287+0000 7f23f37fe640 10 client.13757.objecter ms_handle_connect 0x7f23fc09f670 2024-09-16T03:47:22.306 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.287+0000 7f23f37fe640 10 client.13757.objecter resend_mon_ops 2024-09-16T03:47:22.306 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.287+0000 7f23f1d9b640 1 --2- 172.21.15.42:0/3475651053 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f23fc09f670 0x7f23fc02c650 secure :-1 s=READY pgs=2485 cs=0 l=1 rev1=1 crypto rx=0x7f23fc09ffe0 tx=0x7f23e4019100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:47:22.306 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.287+0000 7f23fad9d640 1 --2- 172.21.15.42:0/3475651053 >> v2:172.21.15.154:6812/254845342 conn(0x7f23dc0377c0 0x7f23dc039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:22.306 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.287+0000 7f23fad9d640 10 monclient: get_auth_request con 0x7f23dc0377c0 auth_method 0 2024-09-16T03:47:22.306 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.287+0000 7f23f37fe640 1 -- 172.21.15.42:0/3475651053 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23e40bc070 con 0x7f23fc09f670 2024-09-16T03:47:22.306 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.287+0000 7f23f37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:47:22.306 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.287+0000 7f23f37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:47:22.306 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.287+0000 7f23f37fe640 10 monclient: dump: 2024-09-16T03:47:22.306 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:47:22.306 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:22.306 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:22.306 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:22.306 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:47:22.306 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:22.306 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:47:22.307 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:47:22.307 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:47:22.307 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:22.307 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.287+0000 7f23f37fe640 1 -- 172.21.15.42:0/3475651053 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f23e40b1040 con 0x7f23fc09f670 2024-09-16T03:47:22.307 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.287+0000 7f23f37fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:47:22.307 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.287+0000 7f23f37fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:47:22.307 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.287+0000 7f23f37fe640 1 -- 172.21.15.42:0/3475651053 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f23fc09f670 msgr2=0x7f23fc02c650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:22.307 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.287+0000 7f23f37fe640 1 --2- 172.21.15.42:0/3475651053 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f23fc09f670 0x7f23fc02c650 secure :-1 s=READY pgs=2485 cs=0 l=1 rev1=1 crypto rx=0x7f23fc09ffe0 tx=0x7f23e4019100 comp rx=0 tx=0).stop 2024-09-16T03:47:22.307 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.287+0000 7f23f37fe640 1 -- 172.21.15.42:0/3475651053 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f23fc09f670 msgr2=0x7f23fc02c650 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:47:22.307 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.287+0000 7f23f37fe640 1 --2- 172.21.15.42:0/3475651053 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f23fc09f670 0x7f23fc02c650 secure :-1 s=CLOSED pgs=2485 cs=0 l=1 rev1=1 crypto rx=0x7f23fc09ffe0 tx=0x7f23e4019100 comp rx=0 tx=0).stop 2024-09-16T03:47:22.307 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.291+0000 7f23fad9d640 1 --2- 172.21.15.42:0/3475651053 >> v2:172.21.15.154:6812/254845342 conn(0x7f23dc0377c0 0x7f23dc039c80 secure :-1 s=READY pgs=3444 cs=0 l=1 rev1=1 crypto rx=0x7f23fc0a1be0 tx=0x7f23f4001660 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:47:22.307 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.291+0000 7f23f37fe640 10 client.13757.objecter ms_handle_connect 0x7f23dc0377c0 2024-09-16T03:47:22.309 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.295+0000 7f2400978640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:47:22.309 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.295+0000 7f2400978640 1 --2- 172.21.15.42:0/3475651053 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f23fc02cb90 0x7f23fc02cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:22.309 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.295+0000 7f2400978640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:22.309 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.295+0000 7f2400978640 1 -- 172.21.15.42:0/3475651053 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f23fc070750 con 0x7f23fc02cb90 2024-09-16T03:47:22.309 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.295+0000 7f23f1d9b640 1 --2- 172.21.15.42:0/3475651053 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f23fc02cb90 0x7f23fc02cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:22.309 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.295+0000 7f23f1d9b640 10 monclient: get_auth_request con 0x7f23fc02cb90 auth_method 0 2024-09-16T03:47:22.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.295+0000 7f23f1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:22.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.295+0000 7f23f1d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:22.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.295+0000 7f23f1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:47:22.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.295+0000 7f23f1d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:47:22.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.295+0000 7f23f1d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:47:22.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.295+0000 7f23f1d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:47:22.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.295+0000 7f23f1d9b640 10 monclient(hunting): handle_auth_done global_id 13763 payload 824 2024-09-16T03:47:22.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.295+0000 7f23f1d9b640 1 -- 172.21.15.42:0/3475651053 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f23fc09dac0 con 0x7f23fc02cb90 2024-09-16T03:47:22.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.295+0000 7f23f1d9b640 1 --2- 172.21.15.42:0/3475651053 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f23fc02cb90 0x7f23fc02cfa0 secure :-1 s=READY pgs=2486 cs=0 l=1 rev1=1 crypto rx=0x7f23e40ae040 tx=0x7f23e4009190 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:47:22.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.295+0000 7f23f37fe640 10 client.13757.objecter ms_handle_connect 0x7f23fc02cb90 2024-09-16T03:47:22.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.295+0000 7f23f37fe640 10 client.13757.objecter resend_mon_ops 2024-09-16T03:47:22.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.295+0000 7f23f37fe640 1 -- 172.21.15.42:0/3475651053 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23e40b0040 con 0x7f23fc02cb90 2024-09-16T03:47:22.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.295+0000 7f23f37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:47:22.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.295+0000 7f23f37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:47:22.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.295+0000 7f23f37fe640 10 monclient: dump: 2024-09-16T03:47:22.311 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:47:22.311 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:22.311 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:22.311 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:22.311 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:47:22.311 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:22.311 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:47:22.311 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:47:22.311 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:47:22.311 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:22.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.295+0000 7f23f37fe640 1 -- 172.21.15.42:0/3475651053 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4868 (secure 0 0 0) 0x7f23e4003d60 con 0x7f23fc02cb90 2024-09-16T03:47:22.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.295+0000 7f23f37fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:47:22.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.295+0000 7f23f37fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:47:22.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.295+0000 7f23f37fe640 1 -- 172.21.15.42:0/3475651053 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f23fc02cb90 msgr2=0x7f23fc02cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:22.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.295+0000 7f23f37fe640 1 --2- 172.21.15.42:0/3475651053 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f23fc02cb90 0x7f23fc02cfa0 secure :-1 s=READY pgs=2486 cs=0 l=1 rev1=1 crypto rx=0x7f23e40ae040 tx=0x7f23e4009190 comp rx=0 tx=0).stop 2024-09-16T03:47:22.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.295+0000 7f23f37fe640 1 -- 172.21.15.42:0/3475651053 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f23fc02cb90 msgr2=0x7f23fc02cfa0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:47:22.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.295+0000 7f23f37fe640 1 --2- 172.21.15.42:0/3475651053 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f23fc02cb90 0x7f23fc02cfa0 secure :-1 s=CLOSED pgs=2486 cs=0 l=1 rev1=1 crypto rx=0x7f23e40ae040 tx=0x7f23e4009190 comp rx=0 tx=0).stop 2024-09-16T03:47:22.312 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:47:22.312 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:47:22.312 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:47:22.312 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T03:47:22.313 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 52, 2024-09-16T03:47:22.313 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 321666, 2024-09-16T03:47:22.313 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:47:22.313 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:47:22.313 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T03:47:22.313 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:47:22.313 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 2, 2024-09-16T03:47:22.313 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:47:22.313 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:47:22.313 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:47:22.313 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:47:22.314 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:47:22.314 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:47:22.314 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:47:22.314 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:47:22.314 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:47:22.314 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:47:22.314 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:47:22.314 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:47:22.314 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:47:22.314 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:47:22.315 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:47:22.315 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:47:22.315 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:47:22.315 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:47:22.315 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:47:22.315 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:47:22.315 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:47:22.315 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:47:22.315 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:47:22.315 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:47:22.315 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:47:22.316 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:47:22.316 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:47:22.316 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:47:22.316 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:47:22.316 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:22.316 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:47:22.316 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:47:22.316 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:47:22.316 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:47:22.316 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:47:22.316 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:47:22.316 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:47:22.316 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:47:22.317 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:47:22.317 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:47:22.317 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:47:22.317 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:47:22.317 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:47:22.317 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:47:22.317 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:47:22.317 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:47:22.317 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:47:22.317 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:47:22.317 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:47:22.317 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:47:22.317 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:47:22.317 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:47:22.317 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:47:22.318 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:47:22.318 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:47:22.318 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:47:22.318 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:47:22.318 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:47:22.318 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:47:22.318 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:47:22.318 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:22.318 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:47:22.318 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:22.318 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:47:22.318 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:47:22.318 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:47:22.318 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:47:22.319 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:22.319 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:47:22.319 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:47:22.319 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:47:22.319 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:22.319 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:22.319 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:47:22.319 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:47:22.319 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:47:22.319 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:47:22.319 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:47:22.319 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:22.319 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:47:22.319 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:47:22.320 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:47:22.320 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:47:22.320 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:47:22.320 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:22.320 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:22.320 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:47:22.320 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:47:22.320 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:47:22.320 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:47:22.320 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:22.320 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:47:22.320 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:47:22.320 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:47:22.320 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:22.321 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:22.321 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:47:22.321 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:47:22.321 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:47:22.321 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:47:22.321 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:47:22.321 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:22.321 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:47:22.321 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:47:22.321 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:47:22.321 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:47:22.321 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:47:22.321 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:22.321 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:22.322 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:47:22.322 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:47:22.322 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:47:22.322 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:47:22.322 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:22.322 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:47:22.322 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:47:22.322 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:47:22.322 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:22.322 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:22.322 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:47:22.322 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:47:22.322 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:47:22.322 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:47:22.323 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:47:22.323 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:22.323 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:47:22.323 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:47:22.323 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:47:22.323 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:47:22.323 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:47:22.323 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:47:22.323 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:47:22.323 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:22.323 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:47:22.323 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:47:22.323 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:22.323 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:47:22.323 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:47:22.324 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:47:22.324 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:47:22.324 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:47:22.324 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T03:47:22.324 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:22.324 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:47:22.324 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:47:22.324 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 6 2024-09-16T03:47:22.324 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:47:22.324 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:47:22.324 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:47:22.324 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:22.324 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:47:22.325 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:47:22.325 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 9 2024-09-16T03:47:22.325 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:47:22.325 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:47:22.325 INFO:teuthology.orchestra.run.smithi042.stdout: "mgr": [ 2024-09-16T03:47:22.325 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:22.325 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:47:22.325 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:47:22.325 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:47:22.325 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:47:22.325 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:47:22.325 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:22.325 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:47:22.325 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:47:22.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.295+0000 7f2400978640 10 librados: watch_flush enter 2024-09-16T03:47:22.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.295+0000 7f2400978640 10 librados: watch_flush exit 2024-09-16T03:47:22.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.295+0000 7f2400978640 20 client.13757.objecter shutdown clearing up homeless session... 2024-09-16T03:47:22.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.295+0000 7f2400978640 10 client.13757.objecter successfully canceled tick 2024-09-16T03:47:22.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.295+0000 7f2400978640 1 -- 172.21.15.42:0/3475651053 >> v2:172.21.15.154:6812/254845342 conn(0x7f23dc0377c0 msgr2=0x7f23dc039c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:22.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.295+0000 7f2400978640 1 --2- 172.21.15.42:0/3475651053 >> v2:172.21.15.154:6812/254845342 conn(0x7f23dc0377c0 0x7f23dc039c80 secure :-1 s=READY pgs=3444 cs=0 l=1 rev1=1 crypto rx=0x7f23fc0a1be0 tx=0x7f23f4001660 comp rx=0 tx=0).stop 2024-09-16T03:47:22.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.295+0000 7f2400978640 10 monclient: shutdown 2024-09-16T03:47:22.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.295+0000 7f2400978640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:47:22.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.295+0000 7f2400978640 1 -- 172.21.15.42:0/3475651053 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f23fc073080 msgr2=0x7f23fc1cef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:22.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.295+0000 7f2400978640 1 --2- 172.21.15.42:0/3475651053 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f23fc073080 0x7f23fc1cef50 secure :-1 s=READY pgs=2484 cs=0 l=1 rev1=1 crypto rx=0x7f23ec00b530 tx=0x7f23ec00b9f0 comp rx=0 tx=0).stop 2024-09-16T03:47:22.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.295+0000 7f23fad9d640 1 -- 172.21.15.42:0/3475651053 reap_dead start 2024-09-16T03:47:22.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.295+0000 7f23fad9d640 1 -- 172.21.15.42:0/3475651053 reap_dead start 2024-09-16T03:47:22.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.295+0000 7f2400978640 1 -- 172.21.15.42:0/3475651053 shutdown_connections 2024-09-16T03:47:22.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.295+0000 7f2400978640 1 -- 172.21.15.42:0/3475651053 >> 172.21.15.42:0/3475651053 conn(0x7f23fc06bc40 msgr2=0x7f23fc13a8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:47:22.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.299+0000 7f2400978640 1 -- 172.21.15.42:0/3475651053 shutdown_connections 2024-09-16T03:47:22.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.299+0000 7f2400978640 1 -- 172.21.15.42:0/3475651053 wait complete. 2024-09-16T03:47:22.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:22.299+0000 7f2400978640 1 librados: shutdown 2024-09-16T03:47:22.329 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T03:47:22.329 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-16T03:47:42.330 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-16T03:47:42.330 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-09-16T03:47:42.330 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-16T03:47:42.330 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-16T03:47:42.334 INFO:tasks.ceph.mon.b:Started 2024-09-16T03:47:42.334 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T03:47:42.334 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:47:42.471 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.467+0000 7f305ca82640 10 monclient: get_monmap_and_config 2024-09-16T03:47:42.471 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.467+0000 7f305ca82640 10 monclient: build_initial_monmap 2024-09-16T03:47:42.471 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.467+0000 7f305ca82640 10 monclient: monmap: 2024-09-16T03:47:42.471 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:47:42.471 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:42.471 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:47:42.470178+0000 2024-09-16T03:47:42.471 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:47:42.470178+0000 2024-09-16T03:47:42.471 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:47:42.471 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:42.472 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:47:42.472 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:47:42.472 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:47:42.472 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:47:42.472 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:47:42.472 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:42.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.467+0000 7f305ca82640 1 Processor -- start 2024-09-16T03:47:42.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.467+0000 7f305ca82640 1 -- start start 2024-09-16T03:47:42.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.467+0000 7f305ca82640 10 monclient: init 2024-09-16T03:47:42.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.467+0000 7f305ca82640 10 monclient: _reopen_session rank -1 2024-09-16T03:47:42.475 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.467+0000 7f305ca82640 10 monclient: _add_conns ranks=[3,4,1,2,0] 2024-09-16T03:47:42.475 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.467+0000 7f305ca82640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f30580a1330 0x7f305809f180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:42.475 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.467+0000 7f305ca82640 10 monclient(hunting): picked mon.noname-a con 0x7f30580a1330 addr v2:172.21.15.42:3300/0 2024-09-16T03:47:42.475 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.467+0000 7f305ca82640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f30580a1710 0x7f30580a1b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:42.475 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.467+0000 7f305ca82640 10 monclient(hunting): picked mon.noname-c con 0x7f30580a1710 addr v2:172.21.15.154:3300/0 2024-09-16T03:47:42.475 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.467+0000 7f305ca82640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f30581340f0 addr v1:172.21.15.154:6789/0 2024-09-16T03:47:42.475 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.467+0000 7f305ca82640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f305809e4c0 con 0x7f30581340f0 2024-09-16T03:47:42.475 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.467+0000 7f305ca82640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:42.475 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.467+0000 7f305ca82640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f305809edf0 con 0x7f30580a1330 2024-09-16T03:47:42.475 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.467+0000 7f305ca82640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:42.475 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.467+0000 7f305ca82640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f305809cd60 con 0x7f30580a1710 2024-09-16T03:47:42.475 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.467+0000 7f305ca82640 10 monclient(hunting): _renew_subs 2024-09-16T03:47:42.476 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.467+0000 7f305ca82640 10 monclient(hunting): authenticate will time out at 1309.439719s 2024-09-16T03:47:42.476 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.467+0000 7f305659c640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f30580a1330 0x7f305809f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:42.476 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.471+0000 7f305659c640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f30580a1330 0x7f305809f180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:53546/0 (socket says 172.21.15.42:53546) 2024-09-16T03:47:42.476 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.471+0000 7f305659c640 1 -- 172.21.15.42:0/4083480230 learned_addr learned my addr 172.21.15.42:0/4083480230 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:47:42.476 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.471+0000 7f305659c640 10 monclient(hunting): get_auth_request con 0x7f30580a1330 auth_method 0 2024-09-16T03:47:42.476 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.471+0000 7f305659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:42.476 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.471+0000 7f305659c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:42.476 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.471+0000 7f305659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:47:42.476 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.471+0000 7f3055d9b640 1 --2- 172.21.15.42:0/4083480230 >> v2:172.21.15.154:3300/0 conn(0x7f30580a1710 0x7f30580a1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:42.476 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.471+0000 7f305659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:47:42.476 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.471+0000 7f305659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:47:42.476 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.471+0000 7f305659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:47:42.477 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.471+0000 7f3055d9b640 10 monclient(hunting): get_auth_request con 0x7f30580a1710 auth_method 0 2024-09-16T03:47:42.477 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.471+0000 7f3055d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:42.477 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.471+0000 7f3055d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:42.477 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.471+0000 7f3055d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:47:42.477 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.471+0000 7f305659c640 10 monclient(hunting): handle_auth_done global_id 764277 payload 274 2024-09-16T03:47:42.477 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.471+0000 7f3055d9b640 1 -- 172.21.15.42:0/4083480230 >> v2:172.21.15.154:3300/0 conn(0x7f30580a1710 msgr2=0x7f30580a1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T03:47:42.477 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.471+0000 7f3055d9b640 1 -- 172.21.15.42:0/4083480230 >> v2:172.21.15.154:3300/0 conn(0x7f30580a1710 msgr2=0x7f30580a1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:47:42.477 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.471+0000 7f3055d9b640 1 --2- 172.21.15.42:0/4083480230 >> v2:172.21.15.154:3300/0 conn(0x7f30580a1710 0x7f30580a1b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:47:42.477 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.471+0000 7f305659c640 1 -- 172.21.15.42:0/4083480230 >> v2:172.21.15.154:3300/0 conn(0x7f30580a1710 msgr2=0x7f30580a1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:47:42.477 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.471+0000 7f3055d9b640 1 --2- 172.21.15.42:0/4083480230 >> v2:172.21.15.154:3300/0 conn(0x7f30580a1710 0x7f30580a1b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:47:42.477 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.471+0000 7f305659c640 1 --2- 172.21.15.42:0/4083480230 >> v2:172.21.15.154:3300/0 conn(0x7f30580a1710 0x7f30580a1b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:42.477 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.471+0000 7f305659c640 1 -- 172.21.15.42:0/4083480230 >> v1:172.21.15.154:6789/0 conn(0x7f30581340f0 legacy=0x7f3058134540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:42.477 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.471+0000 7f305659c640 10 monclient: _finish_hunting 0 2024-09-16T03:47:42.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.471+0000 7f305659c640 1 monclient: found mon.noname-a 2024-09-16T03:47:42.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.471+0000 7f305659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:47:42.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.471+0000 7f305659c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:47:42.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.471+0000 7f305659c640 1 -- 172.21.15.42:0/4083480230 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30580a2080 con 0x7f30580a1330 2024-09-16T03:47:42.483 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.475+0000 7f305659c640 1 --2- 172.21.15.42:0/4083480230 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f30580a1330 0x7f305809f180 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f3058073490 tx=0x7f304001c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3d2afcccae0847f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:47:42.484 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.475+0000 7f305559a640 1 -- 172.21.15.42:0/4083480230 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30400b2070 con 0x7f30580a1330 2024-09-16T03:47:42.484 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.475+0000 7f305559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:47:42.484 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.475+0000 7f305559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:47:42.484 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.475+0000 7f305559a640 10 monclient: dump: 2024-09-16T03:47:42.484 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:47:42.484 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:42.484 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:42.484 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:42.484 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:47:42.484 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:42.484 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:47:42.484 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:47:42.485 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:47:42.485 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:42.485 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.475+0000 7f305559a640 10 monclient: _finish_auth 0 2024-09-16T03:47:42.485 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.475+0000 7f305559a640 10 monclient: _check_auth_tickets 2024-09-16T03:47:42.485 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.475+0000 7f305559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:47:42.485 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.475+0000 7f305559a640 1 -- 172.21.15.42:0/4083480230 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f30400ad680 con 0x7f30580a1330 2024-09-16T03:47:42.485 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.475+0000 7f305ca82640 5 monclient: authenticate success, global_id 764277 2024-09-16T03:47:42.485 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.475+0000 7f305ca82640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:47:42.485 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.475+0000 7f305559a640 10 monclient: handle_config config(0 keys) 2024-09-16T03:47:42.485 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.475+0000 7f305559a640 1 -- 172.21.15.42:0/4083480230 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30400ada50 con 0x7f30580a1330 2024-09-16T03:47:42.485 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.475+0000 7f305559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:47:42.485 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.475+0000 7f305559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:47:42.485 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.475+0000 7f305559a640 10 monclient: dump: 2024-09-16T03:47:42.486 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:47:42.486 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:42.486 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:42.486 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:42.486 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:47:42.486 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:42.486 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:47:42.486 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:47:42.486 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:47:42.486 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:42.486 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.475+0000 7f305ca82640 10 monclient: get_monmap_and_config success 2024-09-16T03:47:42.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.475+0000 7f305ca82640 4 set_mon_vals no callback set 2024-09-16T03:47:42.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.475+0000 7f305ca82640 10 monclient: shutdown 2024-09-16T03:47:42.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.475+0000 7f305ca82640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:47:42.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.475+0000 7f305ca82640 1 -- 172.21.15.42:0/4083480230 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f30580a1330 msgr2=0x7f305809f180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:42.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.475+0000 7f305ca82640 1 --2- 172.21.15.42:0/4083480230 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f30580a1330 0x7f305809f180 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f3058073490 tx=0x7f304001c470 comp rx=0 tx=0).stop 2024-09-16T03:47:42.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.475+0000 7f305ca82640 1 -- 172.21.15.42:0/4083480230 shutdown_connections 2024-09-16T03:47:42.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.475+0000 7f305ca82640 1 --2- 172.21.15.42:0/4083480230 >> v2:172.21.15.154:3300/0 conn(0x7f30580a1710 0x7f30580a1b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:42.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.475+0000 7f305ca82640 1 --2- 172.21.15.42:0/4083480230 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f30580a1330 0x7f305809f180 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:42.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.475+0000 7f305ca82640 1 -- 172.21.15.42:0/4083480230 >> 172.21.15.42:0/4083480230 conn(0x7f30580a2420 msgr2=0x7f30580a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:47:42.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.475+0000 7f305ca82640 1 -- 172.21.15.42:0/4083480230 shutdown_connections 2024-09-16T03:47:42.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.475+0000 7f305ca82640 1 -- 172.21.15.42:0/4083480230 wait complete. 2024-09-16T03:47:42.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.475+0000 7f305ca82640 10 monclient: build_initial_monmap 2024-09-16T03:47:42.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.475+0000 7f305ca82640 10 monclient: monmap: 2024-09-16T03:47:42.488 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:47:42.488 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:47:42.488 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:47:42.488 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:47:42.488 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:47:42.488 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:42.488 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:47:42.488 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:47:42.488 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:47:42.488 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:42.488 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.475+0000 7f305ca82640 1 librados: starting msgr at 2024-09-16T03:47:42.488 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.475+0000 7f305ca82640 1 librados: starting objecter 2024-09-16T03:47:42.488 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.475+0000 7f305ca82640 1 Processor -- start 2024-09-16T03:47:42.488 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.475+0000 7f305ca82640 1 -- start start 2024-09-16T03:47:42.489 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.475+0000 7f305ca82640 1 librados: setting wanted keys 2024-09-16T03:47:42.489 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.475+0000 7f305ca82640 1 librados: calling monclient init 2024-09-16T03:47:42.489 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.475+0000 7f305ca82640 10 monclient: init 2024-09-16T03:47:42.489 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.475+0000 7f305ca82640 10 monclient: _reopen_session rank -1 2024-09-16T03:47:42.489 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.475+0000 7f305ca82640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:47:42.489 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.475+0000 7f305ca82640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f30580a1330 0x7f305809ba50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:42.489 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.475+0000 7f305ca82640 10 monclient(hunting): picked mon.noname-a con 0x7f30580a1330 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:47:42.489 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.475+0000 7f305ca82640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f30580a1710 0x7f30581cd100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:42.489 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.475+0000 7f305ca82640 10 monclient(hunting): picked mon.noname-c con 0x7f30580a1710 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:47:42.489 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.475+0000 7f305ca82640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f30581340f0 0x7f3058133310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:42.489 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.475+0000 7f305ca82640 10 monclient(hunting): picked mon.noname-b con 0x7f30581340f0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:47:42.489 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.475+0000 7f305ca82640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:42.489 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.475+0000 7f305ca82640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f305813ed60 con 0x7f30580a1330 2024-09-16T03:47:42.489 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.475+0000 7f305ca82640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:42.490 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.475+0000 7f305ca82640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f305813ebe0 con 0x7f30581340f0 2024-09-16T03:47:42.490 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.475+0000 7f305ca82640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:42.490 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.475+0000 7f305ca82640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f305813eee0 con 0x7f30580a1710 2024-09-16T03:47:42.490 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.475+0000 7f305ca82640 10 monclient(hunting): _renew_subs 2024-09-16T03:47:42.490 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.475+0000 7f305ca82640 10 monclient(hunting): authenticate will time out at 1309.446742s 2024-09-16T03:47:42.490 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.475+0000 7f305659c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f30580a1330 0x7f305809ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:42.490 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.475+0000 7f305659c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f30580a1330 0x7f305809ba50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:53556/0 (socket says 172.21.15.42:53556) 2024-09-16T03:47:42.490 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.475+0000 7f305659c640 1 -- 172.21.15.42:0/3434647606 learned_addr learned my addr 172.21.15.42:0/3434647606 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:47:42.490 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.475+0000 7f305659c640 10 monclient(hunting): get_auth_request con 0x7f30580a1330 auth_method 0 2024-09-16T03:47:42.490 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.475+0000 7f305659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:42.490 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.475+0000 7f305659c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:42.490 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.475+0000 7f305659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:47:42.490 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.475+0000 7f3055d9b640 1 --2- 172.21.15.42:0/3434647606 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f30580a1710 0x7f30581cd100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:42.491 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.475+0000 7f3055d9b640 10 monclient(hunting): get_auth_request con 0x7f30580a1710 auth_method 0 2024-09-16T03:47:42.491 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.475+0000 7f3055d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:42.491 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.475+0000 7f3055d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:42.491 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.475+0000 7f3055d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:47:42.491 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.475+0000 7f305659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:47:42.491 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.475+0000 7f305659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:47:42.491 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.475+0000 7f305659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:47:42.491 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.475+0000 7f3055d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:47:42.491 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.475+0000 7f3055d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:47:42.491 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.475+0000 7f3055d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:47:42.491 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.475+0000 7f3056d9d640 1 --2- 172.21.15.42:0/3434647606 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f30581340f0 0x7f3058133310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:42.491 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.475+0000 7f305659c640 10 monclient(hunting): handle_auth_done global_id 764280 payload 824 2024-09-16T03:47:42.491 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.475+0000 7f305659c640 1 -- 172.21.15.42:0/3434647606 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f30580a1710 msgr2=0x7f30581cd100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:42.491 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.475+0000 7f305659c640 1 --2- 172.21.15.42:0/3434647606 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f30580a1710 0x7f30581cd100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:42.492 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.475+0000 7f305659c640 1 -- 172.21.15.42:0/3434647606 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f30581340f0 msgr2=0x7f3058133310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:42.492 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.475+0000 7f305659c640 1 --2- 172.21.15.42:0/3434647606 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f30581340f0 0x7f3058133310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:42.492 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.475+0000 7f305659c640 10 monclient: _finish_hunting 0 2024-09-16T03:47:42.492 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.475+0000 7f305659c640 1 monclient: found mon.noname-a 2024-09-16T03:47:42.492 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.475+0000 7f305659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:47:42.492 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.475+0000 7f305659c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:47:42.492 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.475+0000 7f305659c640 1 -- 172.21.15.42:0/3434647606 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3058073a30 con 0x7f30580a1330 2024-09-16T03:47:42.492 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.475+0000 7f3056d9d640 10 monclient: get_auth_request con 0x7f30581340f0 auth_method 0 2024-09-16T03:47:42.492 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.475+0000 7f3056d9d640 1 --2- 172.21.15.42:0/3434647606 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f30581340f0 0x7f3058133310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T03:47:42.492 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.475+0000 7f305659c640 1 --2- 172.21.15.42:0/3434647606 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f30580a1330 0x7f305809ba50 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f304001a0a0 tx=0x7f304001acb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:47:42.492 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.475+0000 7f304effd640 10 client.?.objecter ms_handle_connect 0x7f30580a1330 2024-09-16T03:47:42.492 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.475+0000 7f304effd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:47:42.492 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.475+0000 7f304effd640 1 -- 172.21.15.42:0/3434647606 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30400b2040 con 0x7f30580a1330 2024-09-16T03:47:42.493 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.475+0000 7f304effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:47:42.493 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.475+0000 7f304effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:47:42.493 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.475+0000 7f304effd640 10 monclient: dump: 2024-09-16T03:47:42.493 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:47:42.493 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:42.493 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:42.493 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:42.493 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:47:42.493 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:42.493 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:47:42.493 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:47:42.493 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:47:42.493 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:42.493 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.475+0000 7f304effd640 10 monclient: _finish_auth 0 2024-09-16T03:47:42.494 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.475+0000 7f304effd640 10 monclient: _check_auth_tickets 2024-09-16T03:47:42.494 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.475+0000 7f304effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:47:42.494 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.475+0000 7f304effd640 1 -- 172.21.15.42:0/3434647606 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f30400bd070 con 0x7f30580a1330 2024-09-16T03:47:42.494 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.475+0000 7f305ca82640 5 monclient: authenticate success, global_id 764280 2024-09-16T03:47:42.494 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.475+0000 7f305ca82640 10 monclient: _renew_subs 2024-09-16T03:47:42.494 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.475+0000 7f305ca82640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:47:42.494 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.475+0000 7f305ca82640 1 -- 172.21.15.42:0/3434647606 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f305809c5a0 con 0x7f30580a1330 2024-09-16T03:47:42.494 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.475+0000 7f304effd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:47:42.494 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.475+0000 7f304effd640 1 -- 172.21.15.42:0/3434647606 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30400ade30 con 0x7f30580a1330 2024-09-16T03:47:42.494 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.475+0000 7f304effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:47:42.494 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.475+0000 7f304effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:47:42.494 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.475+0000 7f304effd640 10 monclient: dump: 2024-09-16T03:47:42.494 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:47:42.494 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:42.494 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:42.495 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:42.495 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:47:42.495 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:42.495 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:47:42.495 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:47:42.495 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:47:42.495 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:42.495 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.475+0000 7f305ca82640 10 client.764280.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:47:42.495 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.475+0000 7f305ca82640 10 monclient: _renew_subs 2024-09-16T03:47:42.495 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.475+0000 7f305ca82640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:47:42.495 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.475+0000 7f305ca82640 1 -- 172.21.15.42:0/3434647606 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f305811e410 con 0x7f30580a1330 2024-09-16T03:47:42.495 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.475+0000 7f3054d99640 4 set_mon_vals no callback set 2024-09-16T03:47:42.495 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.475+0000 7f305ca82640 1 librados: init done 2024-09-16T03:47:42.495 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.479+0000 7f305ca82640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:47:42.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.479+0000 7f305ca82640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:47:42.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.479+0000 7f305ca82640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:47:42.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.479+0000 7f305ca82640 1 -- 172.21.15.42:0/3434647606 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f305809f180 con 0x7f30580a1330 2024-09-16T03:47:42.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.479+0000 7f304effd640 1 -- 172.21.15.42:0/3434647606 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f30400c2460 con 0x7f30580a1330 2024-09-16T03:47:42.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.479+0000 7f304effd640 1 --2- 172.21.15.42:0/3434647606 >> v2:172.21.15.154:6812/254845342 conn(0x7f3038037810 0x7f3038039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:42.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.479+0000 7f304effd640 1 -- 172.21.15.42:0/3434647606 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(172..172 src has 1..172) ==== 6853+0+0 (secure 0 0 0) 0x7f30400e3470 con 0x7f30580a1330 2024-09-16T03:47:42.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.479+0000 7f304effd640 10 client.764280.objecter ms_dispatch 0x7f3058000ba0 osd_map(172..172 src has 1..172) 2024-09-16T03:47:42.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.479+0000 7f304effd640 3 client.764280.objecter handle_osd_map got epochs [172,172] > 0 2024-09-16T03:47:42.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.479+0000 7f304effd640 3 client.764280.objecter handle_osd_map decoding full epoch 172 2024-09-16T03:47:42.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.479+0000 7f304effd640 20 client.764280.objecter dump_active .. 0 homeless 2024-09-16T03:47:42.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.479+0000 7f3055d9b640 1 --2- 172.21.15.42:0/3434647606 >> v2:172.21.15.154:6812/254845342 conn(0x7f3038037810 0x7f3038039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:42.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.479+0000 7f3055d9b640 10 monclient: get_auth_request con 0x7f3038037810 auth_method 0 2024-09-16T03:47:42.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.479+0000 7f3055d9b640 1 --2- 172.21.15.42:0/3434647606 >> v2:172.21.15.154:6812/254845342 conn(0x7f3038037810 0x7f3038039cd0 secure :-1 s=READY pgs=3455 cs=0 l=1 rev1=1 crypto rx=0x7f305813c390 tx=0x7f3048008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:47:42.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.479+0000 7f304effd640 10 client.764280.objecter ms_handle_connect 0x7f3038037810 2024-09-16T03:47:42.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.487+0000 7f304effd640 1 -- 172.21.15.42:0/3434647606 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f30400160f0 con 0x7f30580a1330 2024-09-16T03:47:42.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.487+0000 7f304effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:47:42.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.487+0000 7f304effd640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:47:42.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.719+0000 7f305ca82640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:47:42.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.719+0000 7f305ca82640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:47:42.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.719+0000 7f305ca82640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:47:42.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.719+0000 7f305ca82640 1 -- 172.21.15.42:0/3434647606 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f305811e5e0 con 0x7f30580a1330 2024-09-16T03:47:42.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.719+0000 7f304effd640 1 -- 172.21.15.42:0/3434647606 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1555 (secure 0 0 0) 0x7f30400b77b0 con 0x7f30580a1330 2024-09-16T03:47:42.724 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.719+0000 7f304effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:47:42.724 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.719+0000 7f304effd640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:47:42.724 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":52,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":22,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:47:42.726 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.723+0000 7f305ca82640 10 librados: watch_flush enter 2024-09-16T03:47:42.726 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.723+0000 7f305ca82640 10 librados: watch_flush exit 2024-09-16T03:47:42.726 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.723+0000 7f305ca82640 20 client.764280.objecter shutdown clearing up homeless session... 2024-09-16T03:47:42.726 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.723+0000 7f305ca82640 10 client.764280.objecter successfully canceled tick 2024-09-16T03:47:42.726 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.723+0000 7f305ca82640 1 -- 172.21.15.42:0/3434647606 >> v2:172.21.15.154:6812/254845342 conn(0x7f3038037810 msgr2=0x7f3038039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:42.726 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.723+0000 7f305ca82640 1 --2- 172.21.15.42:0/3434647606 >> v2:172.21.15.154:6812/254845342 conn(0x7f3038037810 0x7f3038039cd0 secure :-1 s=READY pgs=3455 cs=0 l=1 rev1=1 crypto rx=0x7f305813c390 tx=0x7f3048008040 comp rx=0 tx=0).stop 2024-09-16T03:47:42.726 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.723+0000 7f305ca82640 10 monclient: shutdown 2024-09-16T03:47:42.727 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.723+0000 7f305ca82640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:47:42.727 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.723+0000 7f305ca82640 1 -- 172.21.15.42:0/3434647606 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f30580a1330 msgr2=0x7f305809ba50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:42.727 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.723+0000 7f305ca82640 1 --2- 172.21.15.42:0/3434647606 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f30580a1330 0x7f305809ba50 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f304001a0a0 tx=0x7f304001acb0 comp rx=0 tx=0).stop 2024-09-16T03:47:42.727 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.723+0000 7f305ca82640 1 -- 172.21.15.42:0/3434647606 shutdown_connections 2024-09-16T03:47:42.727 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.723+0000 7f305ca82640 1 --2- 172.21.15.42:0/3434647606 >> v2:172.21.15.154:6812/254845342 conn(0x7f3038037810 0x7f3038039cd0 unknown :-1 s=CLOSED pgs=3455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:42.727 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.723+0000 7f305ca82640 1 --2- 172.21.15.42:0/3434647606 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f30581340f0 0x7f3058133310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:42.727 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.723+0000 7f305ca82640 1 --2- 172.21.15.42:0/3434647606 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f30580a1710 0x7f30581cd100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:42.727 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.723+0000 7f305ca82640 1 --2- 172.21.15.42:0/3434647606 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f30580a1330 0x7f305809ba50 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:42.727 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.723+0000 7f305ca82640 1 -- 172.21.15.42:0/3434647606 >> 172.21.15.42:0/3434647606 conn(0x7f30580a2420 msgr2=0x7f3058070e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:47:42.727 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.723+0000 7f305ca82640 1 -- 172.21.15.42:0/3434647606 shutdown_connections 2024-09-16T03:47:42.727 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.723+0000 7f305ca82640 1 -- 172.21.15.42:0/3434647606 wait complete. 2024-09-16T03:47:42.727 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:42.723+0000 7f305ca82640 1 librados: shutdown 2024-09-16T03:47:45.744 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:47:45.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.879+0000 7f847fb78640 10 monclient: get_monmap_and_config 2024-09-16T03:47:45.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.879+0000 7f847fb78640 10 monclient: build_initial_monmap 2024-09-16T03:47:45.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.879+0000 7f847fb78640 10 monclient: monmap: 2024-09-16T03:47:45.884 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:47:45.884 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:45.884 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:47:45.883044+0000 2024-09-16T03:47:45.884 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:47:45.883044+0000 2024-09-16T03:47:45.885 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:47:45.885 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:45.885 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:47:45.885 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:47:45.885 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:47:45.885 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:47:45.885 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:47:45.885 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:45.888 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.883+0000 7f847fb78640 1 Processor -- start 2024-09-16T03:47:45.888 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.883+0000 7f847fb78640 1 -- start start 2024-09-16T03:47:45.888 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.883+0000 7f847fb78640 10 monclient: init 2024-09-16T03:47:45.888 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.883+0000 7f847fb78640 10 monclient: _reopen_session rank -1 2024-09-16T03:47:45.888 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.883+0000 7f847fb78640 10 monclient: _add_conns ranks=[3,1,4,0,2] 2024-09-16T03:47:45.888 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.883+0000 7f847fb78640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f8478130bb0 0x7f8478130f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:45.888 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.883+0000 7f847fb78640 10 monclient(hunting): picked mon.noname-a con 0x7f8478130bb0 addr v2:172.21.15.42:3300/0 2024-09-16T03:47:45.888 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.883+0000 7f847fb78640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f84780a2030 addr v1:172.21.15.154:6789/0 2024-09-16T03:47:45.888 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.883+0000 7f847fb78640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f84780a2440 0x7f8478073350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:45.889 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.883+0000 7f847fb78640 10 monclient(hunting): picked mon.noname-c con 0x7f84780a2440 addr v2:172.21.15.154:3300/0 2024-09-16T03:47:45.889 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.883+0000 7f847fb78640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f84780a1750 con 0x7f84780a2030 2024-09-16T03:47:45.889 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.883+0000 7f847fb78640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:45.889 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.883+0000 7f847fb78640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f84780a1470 con 0x7f8478130bb0 2024-09-16T03:47:45.889 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.883+0000 7f847fb78640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:45.889 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.883+0000 7f847fb78640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f84780a1a30 con 0x7f84780a2440 2024-09-16T03:47:45.889 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.883+0000 7f847fb78640 10 monclient(hunting): _renew_subs 2024-09-16T03:47:45.889 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.883+0000 7f847fb78640 10 monclient(hunting): authenticate will time out at 1312.853349s 2024-09-16T03:47:45.889 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.883+0000 7f847d914640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f8478130bb0 0x7f8478130f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:45.889 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.883+0000 7f847d914640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f8478130bb0 0x7f8478130f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:43316/0 (socket says 172.21.15.42:43316) 2024-09-16T03:47:45.890 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.883+0000 7f847d914640 1 -- 172.21.15.42:0/342025339 learned_addr learned my addr 172.21.15.42:0/342025339 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:47:45.890 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.883+0000 7f847d914640 10 monclient(hunting): get_auth_request con 0x7f8478130bb0 auth_method 0 2024-09-16T03:47:45.890 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.883+0000 7f847d914640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:45.890 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.883+0000 7f847d914640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:45.890 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.883+0000 7f847d914640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:47:45.890 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.883+0000 7f847d914640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:47:45.890 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.883+0000 7f847d914640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:47:45.890 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.883+0000 7f847d914640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:47:45.890 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.883+0000 7f847d914640 10 monclient(hunting): handle_auth_done global_id 764283 payload 274 2024-09-16T03:47:45.890 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.883+0000 7f847d914640 1 -- 172.21.15.42:0/342025339 >> v2:172.21.15.154:3300/0 conn(0x7f84780a2440 msgr2=0x7f8478073350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:47:45.891 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.883+0000 7f847d914640 1 --2- 172.21.15.42:0/342025339 >> v2:172.21.15.154:3300/0 conn(0x7f84780a2440 0x7f8478073350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:45.891 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.883+0000 7f847d914640 1 -- 172.21.15.42:0/342025339 >> v1:172.21.15.154:6789/0 conn(0x7f84780a2030 legacy=0x7f8478072c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:47:45.891 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.883+0000 7f847d914640 10 monclient: _finish_hunting 0 2024-09-16T03:47:45.891 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.883+0000 7f847d914640 1 monclient: found mon.noname-a 2024-09-16T03:47:45.892 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.883+0000 7f847d914640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:47:45.892 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.883+0000 7f847d914640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:47:45.892 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.883+0000 7f847d914640 1 -- 172.21.15.42:0/342025339 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84781314d0 con 0x7f8478130bb0 2024-09-16T03:47:45.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.887+0000 7f847d914640 1 --2- 172.21.15.42:0/342025339 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8478130bb0 0x7f8478130f90 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f847809caf0 tx=0x7f847401c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bd69b3b3d7a6b09 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:47:45.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.887+0000 7f847c912640 1 -- 172.21.15.42:0/342025339 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84740b0070 con 0x7f8478130bb0 2024-09-16T03:47:45.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.887+0000 7f847c912640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:47:45.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.887+0000 7f847c912640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:47:45.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.887+0000 7f847c912640 10 monclient: dump: 2024-09-16T03:47:45.899 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:47:45.899 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:45.899 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:45.899 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:45.899 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:47:45.899 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:45.900 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:47:45.900 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:47:45.900 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:47:45.900 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:45.900 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.887+0000 7f847c912640 10 monclient: _finish_auth 0 2024-09-16T03:47:45.900 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.887+0000 7f847c912640 10 monclient: _check_auth_tickets 2024-09-16T03:47:45.900 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.887+0000 7f847c912640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:47:45.900 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.887+0000 7f847c912640 1 -- 172.21.15.42:0/342025339 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f84740ab600 con 0x7f8478130bb0 2024-09-16T03:47:45.900 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.887+0000 7f847c912640 10 monclient: handle_config config(0 keys) 2024-09-16T03:47:45.900 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.887+0000 7f847c912640 1 -- 172.21.15.42:0/342025339 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84740ab9d0 con 0x7f8478130bb0 2024-09-16T03:47:45.900 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.887+0000 7f847fb78640 5 monclient: authenticate success, global_id 764283 2024-09-16T03:47:45.900 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.887+0000 7f847fb78640 10 monclient: get_monmap_and_config success 2024-09-16T03:47:45.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.887+0000 7f847fb78640 4 set_mon_vals no callback set 2024-09-16T03:47:45.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.887+0000 7f847c912640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:47:45.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.887+0000 7f847c912640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:47:45.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.887+0000 7f847c912640 10 monclient: dump: 2024-09-16T03:47:45.901 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:47:45.901 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:45.901 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:45.901 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:45.901 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:47:45.901 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:45.901 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:47:45.902 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:47:45.902 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:47:45.902 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:45.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.887+0000 7f847fb78640 10 monclient: shutdown 2024-09-16T03:47:45.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.887+0000 7f847fb78640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:47:45.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.887+0000 7f847fb78640 1 -- 172.21.15.42:0/342025339 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8478130bb0 msgr2=0x7f8478130f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:45.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.887+0000 7f847fb78640 1 --2- 172.21.15.42:0/342025339 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8478130bb0 0x7f8478130f90 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f847809caf0 tx=0x7f847401c3a0 comp rx=0 tx=0).stop 2024-09-16T03:47:45.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.887+0000 7f847fb78640 1 -- 172.21.15.42:0/342025339 shutdown_connections 2024-09-16T03:47:45.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.887+0000 7f847fb78640 1 --2- 172.21.15.42:0/342025339 >> v2:172.21.15.154:3300/0 conn(0x7f84780a2440 0x7f8478073350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:45.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.887+0000 7f847fb78640 1 --2- 172.21.15.42:0/342025339 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8478130bb0 0x7f8478130f90 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:45.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.887+0000 7f847fb78640 1 -- 172.21.15.42:0/342025339 >> 172.21.15.42:0/342025339 conn(0x7f84780a2ad0 msgr2=0x7f84780a2ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:47:45.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.887+0000 7f847fb78640 1 -- 172.21.15.42:0/342025339 shutdown_connections 2024-09-16T03:47:45.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.887+0000 7f847fb78640 1 -- 172.21.15.42:0/342025339 wait complete. 2024-09-16T03:47:45.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.887+0000 7f847fb78640 10 monclient: build_initial_monmap 2024-09-16T03:47:45.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.887+0000 7f847fb78640 10 monclient: monmap: 2024-09-16T03:47:45.903 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:47:45.903 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:47:45.903 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:47:45.903 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:47:45.903 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:47:45.903 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:45.903 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:47:45.903 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:47:45.904 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:47:45.904 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:45.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.887+0000 7f847fb78640 1 librados: starting msgr at 2024-09-16T03:47:45.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.887+0000 7f847fb78640 1 librados: starting objecter 2024-09-16T03:47:45.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.887+0000 7f847fb78640 1 Processor -- start 2024-09-16T03:47:45.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.887+0000 7f847fb78640 1 -- start start 2024-09-16T03:47:45.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.887+0000 7f847fb78640 1 librados: setting wanted keys 2024-09-16T03:47:45.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.887+0000 7f847fb78640 1 librados: calling monclient init 2024-09-16T03:47:45.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.887+0000 7f847fb78640 10 monclient: init 2024-09-16T03:47:45.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.887+0000 7f847fb78640 10 monclient: _reopen_session rank -1 2024-09-16T03:47:45.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.887+0000 7f847fb78640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T03:47:45.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.887+0000 7f847fb78640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f84780a2030 0x7f84781cbf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:45.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.887+0000 7f847fb78640 10 monclient(hunting): picked mon.noname-a con 0x7f84780a2030 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:47:45.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.887+0000 7f847fb78640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f84780a2440 0x7f847813e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:45.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.887+0000 7f847fb78640 10 monclient(hunting): picked mon.noname-b con 0x7f84780a2440 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:47:45.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.887+0000 7f847fb78640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f8478130bb0 0x7f84781cf400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:45.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.887+0000 7f847fb78640 10 monclient(hunting): picked mon.noname-c con 0x7f8478130bb0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:47:45.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.887+0000 7f847fb78640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:45.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.887+0000 7f847fb78640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f8478141da0 con 0x7f84780a2030 2024-09-16T03:47:45.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.887+0000 7f847fb78640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:45.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.887+0000 7f847fb78640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f8478073aa0 con 0x7f84780a2440 2024-09-16T03:47:45.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.887+0000 7f847fb78640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:45.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.887+0000 7f847fb78640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f8478141f20 con 0x7f8478130bb0 2024-09-16T03:47:45.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.887+0000 7f847fb78640 10 monclient(hunting): _renew_subs 2024-09-16T03:47:45.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.887+0000 7f847fb78640 10 monclient(hunting): authenticate will time out at 1312.859471s 2024-09-16T03:47:45.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.887+0000 7f847d914640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f84780a2030 0x7f84781cbf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:45.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.887+0000 7f847d914640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f84780a2030 0x7f84781cbf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:43330/0 (socket says 172.21.15.42:43330) 2024-09-16T03:47:45.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.887+0000 7f847d914640 1 -- 172.21.15.42:0/4214973143 learned_addr learned my addr 172.21.15.42:0/4214973143 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:47:45.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.887+0000 7f847d914640 10 monclient(hunting): get_auth_request con 0x7f84780a2030 auth_method 0 2024-09-16T03:47:45.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.887+0000 7f847d914640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:45.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.887+0000 7f847d914640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:45.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.887+0000 7f847d914640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:47:45.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.887+0000 7f847e115640 1 --2- 172.21.15.42:0/4214973143 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f8478130bb0 0x7f84781cf400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:45.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.887+0000 7f847e115640 10 monclient(hunting): get_auth_request con 0x7f8478130bb0 auth_method 0 2024-09-16T03:47:45.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.887+0000 7f847e115640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:45.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.887+0000 7f847e115640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:45.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.887+0000 7f847e115640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:47:45.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.887+0000 7f847d914640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:47:45.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.887+0000 7f847d914640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:47:45.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.887+0000 7f847d914640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:47:45.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.887+0000 7f847e115640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:47:45.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.887+0000 7f847e115640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:47:45.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.887+0000 7f847e115640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:47:45.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.887+0000 7f847d113640 1 --2- 172.21.15.42:0/4214973143 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f84780a2440 0x7f847813e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:45.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.887+0000 7f847d914640 10 monclient(hunting): handle_auth_done global_id 764286 payload 824 2024-09-16T03:47:45.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.887+0000 7f847d914640 1 -- 172.21.15.42:0/4214973143 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f8478130bb0 msgr2=0x7f84781cf400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:45.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.887+0000 7f847d914640 1 --2- 172.21.15.42:0/4214973143 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f8478130bb0 0x7f84781cf400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:45.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.887+0000 7f847d914640 1 -- 172.21.15.42:0/4214973143 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f84780a2440 msgr2=0x7f847813e8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:45.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.891+0000 7f847d914640 1 --2- 172.21.15.42:0/4214973143 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f84780a2440 0x7f847813e8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:45.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.891+0000 7f847d914640 10 monclient: _finish_hunting 0 2024-09-16T03:47:45.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.891+0000 7f847d914640 1 monclient: found mon.noname-a 2024-09-16T03:47:45.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.891+0000 7f847d914640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:47:45.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.891+0000 7f847d914640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:47:45.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.891+0000 7f847d914640 1 -- 172.21.15.42:0/4214973143 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84781320f0 con 0x7f84780a2030 2024-09-16T03:47:45.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.891+0000 7f847d113640 10 monclient: get_auth_request con 0x7f84780a2440 auth_method 0 2024-09-16T03:47:45.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.891+0000 7f847e115640 1 --2- 172.21.15.42:0/4214973143 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f8478130bb0 0x7f84781cf400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:47:45.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.891+0000 7f847d113640 1 --2- 172.21.15.42:0/4214973143 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f84780a2440 0x7f847813e8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T03:47:45.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.891+0000 7f847d914640 1 --2- 172.21.15.42:0/4214973143 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f84780a2030 0x7f84781cbf00 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f84740ae040 tx=0x7f847401abe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:47:45.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.891+0000 7f846e7fc640 10 client.?.objecter ms_handle_connect 0x7f84780a2030 2024-09-16T03:47:45.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.891+0000 7f846e7fc640 10 client.?.objecter resend_mon_ops 2024-09-16T03:47:45.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.891+0000 7f846e7fc640 1 -- 172.21.15.42:0/4214973143 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84740ba020 con 0x7f84780a2030 2024-09-16T03:47:45.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.891+0000 7f846e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:47:45.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.891+0000 7f846e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:47:45.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.891+0000 7f846e7fc640 10 monclient: dump: 2024-09-16T03:47:45.909 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:47:45.909 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:45.909 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:45.909 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:45.909 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:47:45.909 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:45.909 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:47:45.910 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:47:45.910 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:47:45.910 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:45.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.891+0000 7f846e7fc640 10 monclient: _finish_auth 0 2024-09-16T03:47:45.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.891+0000 7f846e7fc640 10 monclient: _check_auth_tickets 2024-09-16T03:47:45.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.891+0000 7f846e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:47:45.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.891+0000 7f846e7fc640 1 -- 172.21.15.42:0/4214973143 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f84740b6640 con 0x7f84780a2030 2024-09-16T03:47:45.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.891+0000 7f846e7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:47:45.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.891+0000 7f846e7fc640 1 -- 172.21.15.42:0/4214973143 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84740b0040 con 0x7f84780a2030 2024-09-16T03:47:45.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.891+0000 7f846e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:47:45.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.891+0000 7f846e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:47:45.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.891+0000 7f846e7fc640 10 monclient: dump: 2024-09-16T03:47:45.910 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:47:45.911 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:45.911 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:45.911 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:45.911 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:47:45.911 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:45.911 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:47:45.911 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:47:45.911 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:47:45.911 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:45.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.891+0000 7f846ffff640 4 set_mon_vals no callback set 2024-09-16T03:47:45.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.891+0000 7f847fb78640 5 monclient: authenticate success, global_id 764286 2024-09-16T03:47:45.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.891+0000 7f847fb78640 10 monclient: _renew_subs 2024-09-16T03:47:45.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.891+0000 7f847fb78640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:47:45.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.891+0000 7f847fb78640 1 -- 172.21.15.42:0/4214973143 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f847809d1c0 con 0x7f84780a2030 2024-09-16T03:47:45.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.891+0000 7f847fb78640 10 client.764286.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:47:45.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.891+0000 7f847fb78640 10 monclient: _renew_subs 2024-09-16T03:47:45.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.891+0000 7f847fb78640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:47:45.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.891+0000 7f847fb78640 1 -- 172.21.15.42:0/4214973143 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f847813a060 con 0x7f84780a2030 2024-09-16T03:47:45.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.891+0000 7f847fb78640 1 librados: init done 2024-09-16T03:47:45.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.891+0000 7f847fb78640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:47:45.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.891+0000 7f847fb78640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:47:45.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.891+0000 7f847fb78640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:47:45.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.891+0000 7f847fb78640 1 -- 172.21.15.42:0/4214973143 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f847809fdb0 con 0x7f84780a2030 2024-09-16T03:47:45.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.891+0000 7f846e7fc640 1 -- 172.21.15.42:0/4214973143 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f84740bd020 con 0x7f84780a2030 2024-09-16T03:47:45.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.891+0000 7f846e7fc640 1 --2- 172.21.15.42:0/4214973143 >> v2:172.21.15.154:6812/254845342 conn(0x7f845c037810 0x7f845c039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:45.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.891+0000 7f846e7fc640 1 -- 172.21.15.42:0/4214973143 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(175..175 src has 1..175) ==== 6596+0+0 (secure 0 0 0) 0x7f84740e3730 con 0x7f84780a2030 2024-09-16T03:47:45.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.891+0000 7f846e7fc640 10 client.764286.objecter ms_dispatch 0x7f8478000ba0 osd_map(175..175 src has 1..175) 2024-09-16T03:47:45.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.891+0000 7f846e7fc640 3 client.764286.objecter handle_osd_map got epochs [175,175] > 0 2024-09-16T03:47:45.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.891+0000 7f846e7fc640 3 client.764286.objecter handle_osd_map decoding full epoch 175 2024-09-16T03:47:45.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.891+0000 7f846e7fc640 20 client.764286.objecter dump_active .. 0 homeless 2024-09-16T03:47:45.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.891+0000 7f847d113640 1 --2- 172.21.15.42:0/4214973143 >> v2:172.21.15.154:6812/254845342 conn(0x7f845c037810 0x7f845c039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:45.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.891+0000 7f847d113640 10 monclient: get_auth_request con 0x7f845c037810 auth_method 0 2024-09-16T03:47:45.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.891+0000 7f847d113640 1 --2- 172.21.15.42:0/4214973143 >> v2:172.21.15.154:6812/254845342 conn(0x7f845c037810 0x7f845c039cd0 secure :-1 s=READY pgs=3456 cs=0 l=1 rev1=1 crypto rx=0x7f8478132bd0 tx=0x7f8468009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:47:45.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.891+0000 7f846e7fc640 10 client.764286.objecter ms_handle_connect 0x7f845c037810 2024-09-16T03:47:45.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.895+0000 7f846e7fc640 1 -- 172.21.15.42:0/4214973143 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f847809fdb0 con 0x7f84780a2030 2024-09-16T03:47:45.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.895+0000 7f846e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:47:45.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:45.895+0000 7f846e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:47:46.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:46.151+0000 7f847fb78640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:47:46.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:46.151+0000 7f847fb78640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:47:46.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:46.151+0000 7f847fb78640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:47:46.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:46.151+0000 7f847fb78640 1 -- 172.21.15.42:0/4214973143 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f8478130f90 con 0x7f84780a2030 2024-09-16T03:47:46.157 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:46.155+0000 7f846e7fc640 1 -- 172.21.15.42:0/4214973143 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1555 (secure 0 0 0) 0x7f84740c4030 con 0x7f84780a2030 2024-09-16T03:47:46.157 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:46.155+0000 7f846e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:47:46.157 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:46.155+0000 7f846e7fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:47:46.157 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":52,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":25,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:47:46.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:46.155+0000 7f847fb78640 10 librados: watch_flush enter 2024-09-16T03:47:46.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:46.155+0000 7f847fb78640 10 librados: watch_flush exit 2024-09-16T03:47:46.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:46.155+0000 7f847fb78640 20 client.764286.objecter shutdown clearing up homeless session... 2024-09-16T03:47:46.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:46.155+0000 7f847fb78640 10 client.764286.objecter successfully canceled tick 2024-09-16T03:47:46.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:46.155+0000 7f847fb78640 1 -- 172.21.15.42:0/4214973143 >> v2:172.21.15.154:6812/254845342 conn(0x7f845c037810 msgr2=0x7f845c039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:46.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:46.155+0000 7f847fb78640 1 --2- 172.21.15.42:0/4214973143 >> v2:172.21.15.154:6812/254845342 conn(0x7f845c037810 0x7f845c039cd0 secure :-1 s=READY pgs=3456 cs=0 l=1 rev1=1 crypto rx=0x7f8478132bd0 tx=0x7f8468009040 comp rx=0 tx=0).stop 2024-09-16T03:47:46.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:46.155+0000 7f847fb78640 10 monclient: shutdown 2024-09-16T03:47:46.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:46.155+0000 7f847fb78640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:47:46.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:46.155+0000 7f847fb78640 1 -- 172.21.15.42:0/4214973143 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f84780a2030 msgr2=0x7f84781cbf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:46.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:46.155+0000 7f847fb78640 1 --2- 172.21.15.42:0/4214973143 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f84780a2030 0x7f84781cbf00 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f84740ae040 tx=0x7f847401abe0 comp rx=0 tx=0).stop 2024-09-16T03:47:46.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:46.155+0000 7f847fb78640 1 -- 172.21.15.42:0/4214973143 shutdown_connections 2024-09-16T03:47:46.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:46.155+0000 7f847fb78640 1 --2- 172.21.15.42:0/4214973143 >> v2:172.21.15.154:6812/254845342 conn(0x7f845c037810 0x7f845c039cd0 unknown :-1 s=CLOSED pgs=3456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:46.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:46.155+0000 7f847fb78640 1 --2- 172.21.15.42:0/4214973143 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f8478130bb0 0x7f84781cf400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:46.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:46.155+0000 7f847fb78640 1 --2- 172.21.15.42:0/4214973143 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f84780a2440 0x7f847813e8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:46.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:46.155+0000 7f847fb78640 1 --2- 172.21.15.42:0/4214973143 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f84780a2030 0x7f84781cbf00 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:46.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:46.155+0000 7f847fb78640 1 -- 172.21.15.42:0/4214973143 >> 172.21.15.42:0/4214973143 conn(0x7f84780a2ad0 msgr2=0x7f8478134160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:47:46.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:46.155+0000 7f847fb78640 1 -- 172.21.15.42:0/4214973143 shutdown_connections 2024-09-16T03:47:46.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:46.155+0000 7f847fb78640 1 -- 172.21.15.42:0/4214973143 wait complete. 2024-09-16T03:47:46.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:46.155+0000 7f847fb78640 1 librados: shutdown 2024-09-16T03:47:49.176 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:47:49.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.315+0000 7fec995e0640 10 monclient: get_monmap_and_config 2024-09-16T03:47:49.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.315+0000 7fec995e0640 10 monclient: build_initial_monmap 2024-09-16T03:47:49.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.315+0000 7fec995e0640 10 monclient: monmap: 2024-09-16T03:47:49.318 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:47:49.318 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:49.318 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:47:49.316269+0000 2024-09-16T03:47:49.318 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:47:49.316269+0000 2024-09-16T03:47:49.318 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:47:49.318 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:49.318 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:47:49.318 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:47:49.318 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:47:49.318 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:47:49.319 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:47:49.319 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:49.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.315+0000 7fec995e0640 1 Processor -- start 2024-09-16T03:47:49.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.315+0000 7fec995e0640 1 -- start start 2024-09-16T03:47:49.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.315+0000 7fec995e0640 10 monclient: init 2024-09-16T03:47:49.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.315+0000 7fec995e0640 10 monclient: _reopen_session rank -1 2024-09-16T03:47:49.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.315+0000 7fec995e0640 10 monclient: _add_conns ranks=[1,2,0,4,3] 2024-09-16T03:47:49.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.315+0000 7fec995e0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fec9406bc40 addr v1:172.21.15.154:6789/0 2024-09-16T03:47:49.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.315+0000 7fec995e0640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fec94072dd0 0x7fec9413a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:49.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.315+0000 7fec995e0640 10 monclient(hunting): picked mon.noname-b con 0x7fec94072dd0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:47:49.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.315+0000 7fec995e0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fec9406c730 addr v1:172.21.15.42:6789/0 2024-09-16T03:47:49.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.315+0000 7fec995e0640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fec940723c0 con 0x7fec9406c730 2024-09-16T03:47:49.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.315+0000 7fec995e0640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fec9409bc60 con 0x7fec9406bc40 2024-09-16T03:47:49.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.315+0000 7fec995e0640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:49.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.315+0000 7fec995e0640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fec9409c2b0 con 0x7fec94072dd0 2024-09-16T03:47:49.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.315+0000 7fec995e0640 10 monclient(hunting): _renew_subs 2024-09-16T03:47:49.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.315+0000 7fec995e0640 10 monclient(hunting): authenticate will time out at 1316.285657s 2024-09-16T03:47:49.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.315+0000 7fec927fc640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fec94072dd0 0x7fec9413a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:49.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.315+0000 7fec937fe640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fec9406c730 0x7fec9413c890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:51708/0 (socket says 172.21.15.42:51708) 2024-09-16T03:47:49.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.315+0000 7fec937fe640 1 -- 172.21.15.42:0/749459321 learned_addr learned my addr 172.21.15.42:0/749459321 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:47:49.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.315+0000 7fec927fc640 10 monclient(hunting): get_auth_request con 0x7fec94072dd0 auth_method 0 2024-09-16T03:47:49.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.315+0000 7fec927fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:49.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.315+0000 7fec927fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:49.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.315+0000 7fec927fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:47:49.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.315+0000 7fec927fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:47:49.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.315+0000 7fec927fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:47:49.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.315+0000 7fec927fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:47:49.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.315+0000 7fec927fc640 10 monclient(hunting): handle_auth_done global_id 13823 payload 274 2024-09-16T03:47:49.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.315+0000 7fec91ffb640 1 -- 172.21.15.42:0/749459321 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3401061820 0 0) 0x7fec940723c0 con 0x7fec9406c730 2024-09-16T03:47:49.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.315+0000 7fec927fc640 1 -- 172.21.15.42:0/749459321 >> v1:172.21.15.154:6789/0 conn(0x7fec9406bc40 legacy=0x7fec9406c020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:47:49.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.315+0000 7fec927fc640 1 -- 172.21.15.42:0/749459321 >> v1:172.21.15.42:6789/0 conn(0x7fec9406c730 legacy=0x7fec9413c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:49.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.315+0000 7fec927fc640 10 monclient: _finish_hunting 0 2024-09-16T03:47:49.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.315+0000 7fec927fc640 1 monclient: found mon.noname-b 2024-09-16T03:47:49.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.315+0000 7fec927fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:47:49.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.315+0000 7fec927fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T03:47:49.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.315+0000 7fec927fc640 1 -- 172.21.15.42:0/749459321 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec9409bf40 con 0x7fec94072dd0 2024-09-16T03:47:49.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.315+0000 7fec91ffb640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T03:47:49.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.319+0000 7fec927fc640 1 --2- 172.21.15.42:0/749459321 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fec94072dd0 0x7fec9413a340 secure :-1 s=READY pgs=2501 cs=0 l=1 rev1=1 crypto rx=0x7fec880097f0 tx=0x7fec8801c360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=68df86413251c59b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:47:49.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.319+0000 7fec91ffb640 1 -- 172.21.15.42:0/749459321 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec880b2070 con 0x7fec94072dd0 2024-09-16T03:47:49.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.319+0000 7fec91ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:47:49.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.319+0000 7fec91ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:47:49.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.319+0000 7fec91ffb640 10 monclient: dump: 2024-09-16T03:47:49.326 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:47:49.326 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:49.326 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:49.326 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:49.326 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:47:49.326 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:49.326 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:47:49.326 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:47:49.326 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:47:49.326 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:49.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.319+0000 7fec91ffb640 10 monclient: _finish_auth 0 2024-09-16T03:47:49.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.319+0000 7fec91ffb640 10 monclient: _check_auth_tickets 2024-09-16T03:47:49.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.319+0000 7fec91ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:47:49.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.319+0000 7fec91ffb640 1 -- 172.21.15.42:0/749459321 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fec880ad680 con 0x7fec94072dd0 2024-09-16T03:47:49.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.319+0000 7fec91ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T03:47:49.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.319+0000 7fec91ffb640 1 -- 172.21.15.42:0/749459321 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec880ada50 con 0x7fec94072dd0 2024-09-16T03:47:49.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.319+0000 7fec91ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:47:49.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.319+0000 7fec91ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:47:49.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.319+0000 7fec91ffb640 10 monclient: dump: 2024-09-16T03:47:49.327 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:47:49.327 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:49.327 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:49.327 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:49.327 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:47:49.328 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:49.328 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:47:49.328 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:47:49.328 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:47:49.328 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:49.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.319+0000 7fec995e0640 5 monclient: authenticate success, global_id 13823 2024-09-16T03:47:49.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.319+0000 7fec995e0640 10 monclient: get_monmap_and_config success 2024-09-16T03:47:49.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.319+0000 7fec995e0640 4 set_mon_vals no callback set 2024-09-16T03:47:49.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.319+0000 7fec995e0640 10 monclient: shutdown 2024-09-16T03:47:49.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.319+0000 7fec995e0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:47:49.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.319+0000 7fec995e0640 1 -- 172.21.15.42:0/749459321 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fec94072dd0 msgr2=0x7fec9413a340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:49.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.319+0000 7fec995e0640 1 --2- 172.21.15.42:0/749459321 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fec94072dd0 0x7fec9413a340 secure :-1 s=READY pgs=2501 cs=0 l=1 rev1=1 crypto rx=0x7fec880097f0 tx=0x7fec8801c360 comp rx=0 tx=0).stop 2024-09-16T03:47:49.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.319+0000 7fec995e0640 1 -- 172.21.15.42:0/749459321 shutdown_connections 2024-09-16T03:47:49.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.319+0000 7fec995e0640 1 --2- 172.21.15.42:0/749459321 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fec94072dd0 0x7fec9413a340 unknown :-1 s=CLOSED pgs=2501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:49.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.319+0000 7fec995e0640 1 -- 172.21.15.42:0/749459321 >> 172.21.15.42:0/749459321 conn(0x7fec940a3220 msgr2=0x7fec940a3630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:47:49.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.319+0000 7fec995e0640 1 -- 172.21.15.42:0/749459321 shutdown_connections 2024-09-16T03:47:49.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.319+0000 7fec995e0640 1 -- 172.21.15.42:0/749459321 wait complete. 2024-09-16T03:47:49.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.319+0000 7fec995e0640 10 monclient: build_initial_monmap 2024-09-16T03:47:49.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.319+0000 7fec995e0640 10 monclient: monmap: 2024-09-16T03:47:49.334 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:47:49.334 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:47:49.334 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:47:49.334 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:47:49.334 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:47:49.335 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:49.335 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:47:49.335 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:47:49.335 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:47:49.335 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:49.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.319+0000 7fec995e0640 1 librados: starting msgr at 2024-09-16T03:47:49.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.319+0000 7fec995e0640 1 librados: starting objecter 2024-09-16T03:47:49.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.319+0000 7fec995e0640 1 Processor -- start 2024-09-16T03:47:49.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.319+0000 7fec995e0640 1 -- start start 2024-09-16T03:47:49.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.319+0000 7fec995e0640 1 librados: setting wanted keys 2024-09-16T03:47:49.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.319+0000 7fec995e0640 1 librados: calling monclient init 2024-09-16T03:47:49.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.319+0000 7fec995e0640 10 monclient: init 2024-09-16T03:47:49.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.319+0000 7fec995e0640 10 monclient: _reopen_session rank -1 2024-09-16T03:47:49.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.319+0000 7fec995e0640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:47:49.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.319+0000 7fec995e0640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fec9406bc40 0x7fec94134b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:49.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.319+0000 7fec995e0640 10 monclient(hunting): picked mon.noname-a con 0x7fec9406bc40 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:47:49.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.319+0000 7fec995e0640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fec9406c730 0x7fec941350a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:49.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.319+0000 7fec995e0640 10 monclient(hunting): picked mon.noname-c con 0x7fec9406c730 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:47:49.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.319+0000 7fec995e0640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fec94072dd0 0x7fec9406f950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:49.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.319+0000 7fec995e0640 10 monclient(hunting): picked mon.noname-b con 0x7fec94072dd0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:47:49.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.319+0000 7fec995e0640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:49.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.319+0000 7fec995e0640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fec94140990 con 0x7fec9406bc40 2024-09-16T03:47:49.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.319+0000 7fec995e0640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:49.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.319+0000 7fec995e0640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fec94140810 con 0x7fec94072dd0 2024-09-16T03:47:49.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.319+0000 7fec995e0640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:49.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.319+0000 7fec995e0640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fec94140b10 con 0x7fec9406c730 2024-09-16T03:47:49.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.319+0000 7fec995e0640 10 monclient(hunting): _renew_subs 2024-09-16T03:47:49.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.319+0000 7fec995e0640 10 monclient(hunting): authenticate will time out at 1316.290617s 2024-09-16T03:47:49.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.319+0000 7fec92ffd640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fec9406bc40 0x7fec94134b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:49.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.319+0000 7fec927fc640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fec9406c730 0x7fec941350a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:49.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.319+0000 7fec92ffd640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fec9406bc40 0x7fec94134b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:43336/0 (socket says 172.21.15.42:43336) 2024-09-16T03:47:49.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.319+0000 7fec927fc640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fec9406c730 0x7fec941350a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:59420/0 (socket says 172.21.15.42:59420) 2024-09-16T03:47:49.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.319+0000 7fec92ffd640 1 -- 172.21.15.42:0/2063024224 learned_addr learned my addr 172.21.15.42:0/2063024224 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:47:49.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.319+0000 7fec92ffd640 10 monclient(hunting): get_auth_request con 0x7fec9406bc40 auth_method 0 2024-09-16T03:47:49.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.319+0000 7fec92ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:49.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.319+0000 7fec92ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:49.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.319+0000 7fec92ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:47:49.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.319+0000 7fec927fc640 10 monclient(hunting): get_auth_request con 0x7fec9406c730 auth_method 0 2024-09-16T03:47:49.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.319+0000 7fec927fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:49.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.319+0000 7fec927fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:49.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.319+0000 7fec927fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:47:49.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.319+0000 7fec927fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:47:49.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.319+0000 7fec927fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:47:49.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.319+0000 7fec927fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:47:49.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.319+0000 7fec92ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:47:49.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.319+0000 7fec92ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:47:49.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.319+0000 7fec92ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:47:49.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.319+0000 7fec937fe640 1 --2- 172.21.15.42:0/2063024224 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fec94072dd0 0x7fec9406f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:49.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.319+0000 7fec937fe640 10 monclient(hunting): get_auth_request con 0x7fec94072dd0 auth_method 0 2024-09-16T03:47:49.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.319+0000 7fec937fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:49.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.319+0000 7fec937fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:49.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.319+0000 7fec937fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:47:49.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.319+0000 7fec92ffd640 10 monclient(hunting): handle_auth_done global_id 764295 payload 824 2024-09-16T03:47:49.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.319+0000 7fec92ffd640 1 -- 172.21.15.42:0/2063024224 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fec9406c730 msgr2=0x7fec941350a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:49.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.319+0000 7fec92ffd640 1 --2- 172.21.15.42:0/2063024224 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fec9406c730 0x7fec941350a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:49.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.319+0000 7fec92ffd640 1 -- 172.21.15.42:0/2063024224 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fec94072dd0 msgr2=0x7fec9406f950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:49.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.319+0000 7fec92ffd640 1 --2- 172.21.15.42:0/2063024224 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fec94072dd0 0x7fec9406f950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:49.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.319+0000 7fec92ffd640 10 monclient: _finish_hunting 0 2024-09-16T03:47:49.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.319+0000 7fec92ffd640 1 monclient: found mon.noname-a 2024-09-16T03:47:49.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.319+0000 7fec92ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:47:49.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.319+0000 7fec92ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:47:49.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.319+0000 7fec92ffd640 1 -- 172.21.15.42:0/2063024224 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec9413c2e0 con 0x7fec9406bc40 2024-09-16T03:47:49.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.319+0000 7fec927fc640 1 --2- 172.21.15.42:0/2063024224 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fec9406c730 0x7fec941350a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:47:49.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.319+0000 7fec937fe640 1 --2- 172.21.15.42:0/2063024224 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fec94072dd0 0x7fec9406f950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T03:47:49.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.319+0000 7fec92ffd640 1 --2- 172.21.15.42:0/2063024224 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fec9406bc40 0x7fec94134b60 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fec8400c450 tx=0x7fec8400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:47:49.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.319+0000 7fec7b7fe640 10 client.?.objecter ms_handle_connect 0x7fec9406bc40 2024-09-16T03:47:49.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.319+0000 7fec7b7fe640 10 client.?.objecter resend_mon_ops 2024-09-16T03:47:49.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.319+0000 7fec7b7fe640 1 -- 172.21.15.42:0/2063024224 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec84012830 con 0x7fec9406bc40 2024-09-16T03:47:49.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.319+0000 7fec7b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:47:49.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.319+0000 7fec7b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:47:49.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.319+0000 7fec7b7fe640 10 monclient: dump: 2024-09-16T03:47:49.343 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:47:49.343 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:49.343 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:49.343 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:49.343 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:47:49.343 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:49.343 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:47:49.343 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:47:49.343 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:47:49.343 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:49.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.319+0000 7fec7b7fe640 10 monclient: _finish_auth 0 2024-09-16T03:47:49.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.319+0000 7fec7b7fe640 10 monclient: _check_auth_tickets 2024-09-16T03:47:49.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.319+0000 7fec7b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:47:49.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.319+0000 7fec7b7fe640 1 -- 172.21.15.42:0/2063024224 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fec84017070 con 0x7fec9406bc40 2024-09-16T03:47:49.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.319+0000 7fec995e0640 5 monclient: authenticate success, global_id 764295 2024-09-16T03:47:49.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.319+0000 7fec995e0640 10 monclient: _renew_subs 2024-09-16T03:47:49.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.319+0000 7fec995e0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:47:49.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.319+0000 7fec995e0640 1 -- 172.21.15.42:0/2063024224 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec9413c5c0 con 0x7fec9406bc40 2024-09-16T03:47:49.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.319+0000 7fec7b7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:47:49.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.319+0000 7fec7b7fe640 1 -- 172.21.15.42:0/2063024224 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec84012c30 con 0x7fec9406bc40 2024-09-16T03:47:49.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.319+0000 7fec7b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:47:49.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.319+0000 7fec7b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:47:49.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.319+0000 7fec7b7fe640 10 monclient: dump: 2024-09-16T03:47:49.345 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:47:49.345 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:49.345 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:49.345 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:49.345 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:47:49.345 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:49.345 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:47:49.345 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:47:49.345 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:47:49.345 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:49.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.319+0000 7fec995e0640 10 client.764295.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:47:49.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.319+0000 7fec995e0640 10 monclient: _renew_subs 2024-09-16T03:47:49.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.319+0000 7fec995e0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:47:49.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.319+0000 7fec995e0640 1 -- 172.21.15.42:0/2063024224 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec941355e0 con 0x7fec9406bc40 2024-09-16T03:47:49.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.319+0000 7fec917fa640 4 set_mon_vals no callback set 2024-09-16T03:47:49.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.319+0000 7fec995e0640 1 librados: init done 2024-09-16T03:47:49.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.323+0000 7fec995e0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:47:49.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.323+0000 7fec7b7fe640 1 -- 172.21.15.42:0/2063024224 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fec8401c460 con 0x7fec9406bc40 2024-09-16T03:47:49.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.323+0000 7fec995e0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:47:49.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.323+0000 7fec995e0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:47:49.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.323+0000 7fec995e0640 1 -- 172.21.15.42:0/2063024224 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fec940737d0 con 0x7fec9406bc40 2024-09-16T03:47:49.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.323+0000 7fec7b7fe640 1 --2- 172.21.15.42:0/2063024224 >> v2:172.21.15.154:6812/254845342 conn(0x7fec80037810 0x7fec80039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:49.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.323+0000 7fec7b7fe640 1 -- 172.21.15.42:0/2063024224 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(179..179 src has 1..179) ==== 6881+0+0 (secure 0 0 0) 0x7fec840490f0 con 0x7fec9406bc40 2024-09-16T03:47:49.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.323+0000 7fec7b7fe640 10 client.764295.objecter ms_dispatch 0x7fec94000ba0 osd_map(179..179 src has 1..179) 2024-09-16T03:47:49.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.323+0000 7fec7b7fe640 3 client.764295.objecter handle_osd_map got epochs [179,179] > 0 2024-09-16T03:47:49.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.323+0000 7fec7b7fe640 3 client.764295.objecter handle_osd_map decoding full epoch 179 2024-09-16T03:47:49.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.323+0000 7fec7b7fe640 20 client.764295.objecter dump_active .. 0 homeless 2024-09-16T03:47:49.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.323+0000 7fec927fc640 1 --2- 172.21.15.42:0/2063024224 >> v2:172.21.15.154:6812/254845342 conn(0x7fec80037810 0x7fec80039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:49.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.323+0000 7fec927fc640 10 monclient: get_auth_request con 0x7fec80037810 auth_method 0 2024-09-16T03:47:49.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.323+0000 7fec927fc640 1 --2- 172.21.15.42:0/2063024224 >> v2:172.21.15.154:6812/254845342 conn(0x7fec80037810 0x7fec80039cd0 secure :-1 s=READY pgs=3459 cs=0 l=1 rev1=1 crypto rx=0x7fec880af000 tx=0x7fec88004740 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:47:49.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.323+0000 7fec7b7fe640 10 client.764295.objecter ms_handle_connect 0x7fec80037810 2024-09-16T03:47:49.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.327+0000 7fec7b7fe640 1 -- 172.21.15.42:0/2063024224 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7fec8401a760 con 0x7fec9406bc40 2024-09-16T03:47:49.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.327+0000 7fec7b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:47:49.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.327+0000 7fec7b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:47:49.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.551+0000 7fec995e0640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:47:49.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.551+0000 7fec995e0640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:47:49.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.551+0000 7fec995e0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:47:49.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.551+0000 7fec995e0640 1 -- 172.21.15.42:0/2063024224 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fec9406c0a0 con 0x7fec9406bc40 2024-09-16T03:47:49.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.551+0000 7fec7b7fe640 1 -- 172.21.15.42:0/2063024224 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7fec8401b3c0 con 0x7fec9406bc40 2024-09-16T03:47:49.556 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.551+0000 7fec7b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:47:49.556 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.551+0000 7fec7b7fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:47:49.556 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":54,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:47:49.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.555+0000 7fec995e0640 10 librados: watch_flush enter 2024-09-16T03:47:49.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.555+0000 7fec995e0640 10 librados: watch_flush exit 2024-09-16T03:47:49.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.555+0000 7fec995e0640 20 client.764295.objecter shutdown clearing up homeless session... 2024-09-16T03:47:49.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.555+0000 7fec995e0640 10 client.764295.objecter successfully canceled tick 2024-09-16T03:47:49.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.555+0000 7fec995e0640 1 -- 172.21.15.42:0/2063024224 >> v2:172.21.15.154:6812/254845342 conn(0x7fec80037810 msgr2=0x7fec80039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:49.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.555+0000 7fec995e0640 1 --2- 172.21.15.42:0/2063024224 >> v2:172.21.15.154:6812/254845342 conn(0x7fec80037810 0x7fec80039cd0 secure :-1 s=READY pgs=3459 cs=0 l=1 rev1=1 crypto rx=0x7fec880af000 tx=0x7fec88004740 comp rx=0 tx=0).stop 2024-09-16T03:47:49.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.555+0000 7fec995e0640 10 monclient: shutdown 2024-09-16T03:47:49.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.555+0000 7fec995e0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:47:49.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.555+0000 7fec995e0640 1 -- 172.21.15.42:0/2063024224 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fec9406bc40 msgr2=0x7fec94134b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:49.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.555+0000 7fec995e0640 1 --2- 172.21.15.42:0/2063024224 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fec9406bc40 0x7fec94134b60 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fec8400c450 tx=0x7fec8400c910 comp rx=0 tx=0).stop 2024-09-16T03:47:49.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.555+0000 7fec995e0640 1 -- 172.21.15.42:0/2063024224 shutdown_connections 2024-09-16T03:47:49.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.555+0000 7fec995e0640 1 --2- 172.21.15.42:0/2063024224 >> v2:172.21.15.154:6812/254845342 conn(0x7fec80037810 0x7fec80039cd0 unknown :-1 s=CLOSED pgs=3459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:49.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.555+0000 7fec995e0640 1 --2- 172.21.15.42:0/2063024224 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fec94072dd0 0x7fec9406f950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:49.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.555+0000 7fec995e0640 1 --2- 172.21.15.42:0/2063024224 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fec9406c730 0x7fec941350a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:49.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.555+0000 7fec995e0640 1 --2- 172.21.15.42:0/2063024224 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fec9406bc40 0x7fec94134b60 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:49.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.555+0000 7fec995e0640 1 -- 172.21.15.42:0/2063024224 >> 172.21.15.42:0/2063024224 conn(0x7fec940a3220 msgr2=0x7fec940a5cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:47:49.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.555+0000 7fec995e0640 1 -- 172.21.15.42:0/2063024224 shutdown_connections 2024-09-16T03:47:49.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.555+0000 7fec995e0640 1 -- 172.21.15.42:0/2063024224 wait complete. 2024-09-16T03:47:49.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.555+0000 7fec995e0640 1 librados: shutdown 2024-09-16T03:47:49.574 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T03:47:49.575 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T03:47:49.575 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T03:47:49.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.699+0000 7fd14313c640 10 monclient: get_monmap_and_config 2024-09-16T03:47:49.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.699+0000 7fd14313c640 10 monclient: build_initial_monmap 2024-09-16T03:47:49.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.699+0000 7fd14313c640 10 monclient: monmap: 2024-09-16T03:47:49.704 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:47:49.704 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:49.704 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:47:49.702852+0000 2024-09-16T03:47:49.704 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:47:49.702852+0000 2024-09-16T03:47:49.704 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:47:49.704 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:49.704 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:47:49.705 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:47:49.705 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:47:49.705 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:47:49.705 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:47:49.705 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:49.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.699+0000 7fd14313c640 1 Processor -- start 2024-09-16T03:47:49.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.699+0000 7fd14313c640 1 -- start start 2024-09-16T03:47:49.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.699+0000 7fd14313c640 10 monclient: init 2024-09-16T03:47:49.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.703+0000 7fd14313c640 10 monclient: _reopen_session rank -1 2024-09-16T03:47:49.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.703+0000 7fd14313c640 10 monclient: _add_conns ranks=[1,3,2,4,0] 2024-09-16T03:47:49.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.703+0000 7fd14313c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd13c0a39b0 addr v1:172.21.15.154:6789/0 2024-09-16T03:47:49.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.703+0000 7fd14313c640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd13c0a3d90 0x7fd13c13c380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:49.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.703+0000 7fd14313c640 10 monclient(hunting): picked mon.noname-a con 0x7fd13c0a3d90 addr v2:172.21.15.42:3300/0 2024-09-16T03:47:49.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.703+0000 7fd14313c640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd13c13c8c0 0x7fd13c13ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:49.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.703+0000 7fd14313c640 10 monclient(hunting): picked mon.noname-b con 0x7fd13c13c8c0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:47:49.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.703+0000 7fd14313c640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd13c134290 con 0x7fd13c0a39b0 2024-09-16T03:47:49.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.703+0000 7fd14313c640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:49.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.703+0000 7fd14313c640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7fd13c09c140 con 0x7fd13c0a3d90 2024-09-16T03:47:49.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.703+0000 7fd14313c640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:49.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.703+0000 7fd14313c640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fd13c09c420 con 0x7fd13c13c8c0 2024-09-16T03:47:49.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.703+0000 7fd14313c640 10 monclient(hunting): _renew_subs 2024-09-16T03:47:49.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.703+0000 7fd14313c640 10 monclient(hunting): authenticate will time out at 1316.672423s 2024-09-16T03:47:49.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.703+0000 7fd13bfff640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd13c0a3d90 0x7fd13c13c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:49.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.703+0000 7fd1416d9640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd13c13c8c0 0x7fd13c13ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:49.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.703+0000 7fd13bfff640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd13c0a3d90 0x7fd13c13c380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:43346/0 (socket says 172.21.15.42:43346) 2024-09-16T03:47:49.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.703+0000 7fd1416d9640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd13c13c8c0 0x7fd13c13ecb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:59424/0 (socket says 172.21.15.42:59424) 2024-09-16T03:47:49.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.703+0000 7fd13bfff640 1 -- 172.21.15.42:0/4173031119 learned_addr learned my addr 172.21.15.42:0/4173031119 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:47:49.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.703+0000 7fd13bfff640 10 monclient(hunting): get_auth_request con 0x7fd13c0a3d90 auth_method 0 2024-09-16T03:47:49.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.703+0000 7fd13bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:49.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.703+0000 7fd13bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:49.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.703+0000 7fd13bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:47:49.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.703+0000 7fd1416d9640 10 monclient(hunting): get_auth_request con 0x7fd13c13c8c0 auth_method 0 2024-09-16T03:47:49.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.703+0000 7fd1416d9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:49.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.703+0000 7fd1416d9640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:49.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.703+0000 7fd1416d9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:47:49.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.703+0000 7fd13bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:47:49.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.703+0000 7fd13bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:47:49.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.703+0000 7fd13bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:47:49.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.703+0000 7fd1416d9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:47:49.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.703+0000 7fd1416d9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:47:49.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.703+0000 7fd1416d9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:47:49.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.703+0000 7fd13bfff640 10 monclient(hunting): handle_auth_done global_id 764298 payload 274 2024-09-16T03:47:49.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.703+0000 7fd13bfff640 1 -- 172.21.15.42:0/4173031119 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd13c13c8c0 msgr2=0x7fd13c13ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:47:49.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.703+0000 7fd13bfff640 1 --2- 172.21.15.42:0/4173031119 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd13c13c8c0 0x7fd13c13ecb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:49.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.703+0000 7fd13bfff640 1 -- 172.21.15.42:0/4173031119 >> v1:172.21.15.154:6789/0 conn(0x7fd13c0a39b0 legacy=0x7fd13c133b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:49.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.703+0000 7fd13bfff640 10 monclient: _finish_hunting 0 2024-09-16T03:47:49.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.703+0000 7fd13bfff640 1 monclient: found mon.noname-a 2024-09-16T03:47:49.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.703+0000 7fd13bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:47:49.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.703+0000 7fd13bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:47:49.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.703+0000 7fd13bfff640 1 -- 172.21.15.42:0/4173031119 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd13c06dd60 con 0x7fd13c0a3d90 2024-09-16T03:47:49.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.703+0000 7fd1416d9640 1 --2- 172.21.15.42:0/4173031119 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd13c13c8c0 0x7fd13c13ecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:47:49.722 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.703+0000 7fd13bfff640 1 --2- 172.21.15.42:0/4173031119 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd13c0a3d90 0x7fd13c13c380 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fd12c009e50 tx=0x7fd12c01c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69f443be9f47c57e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:47:49.722 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.703+0000 7fd13b7fe640 1 -- 172.21.15.42:0/4173031119 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd12c0a55b0 con 0x7fd13c0a3d90 2024-09-16T03:47:49.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.703+0000 7fd13b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:47:49.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.703+0000 7fd13b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:47:49.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.703+0000 7fd13b7fe640 10 monclient: dump: 2024-09-16T03:47:49.723 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:47:49.723 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:49.723 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:49.723 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:49.723 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:47:49.724 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:49.724 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:47:49.724 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:47:49.724 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:47:49.724 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:49.724 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.703+0000 7fd13b7fe640 10 monclient: _finish_auth 0 2024-09-16T03:47:49.724 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.703+0000 7fd13b7fe640 10 monclient: _check_auth_tickets 2024-09-16T03:47:49.724 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.703+0000 7fd13b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:47:49.725 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.703+0000 7fd14313c640 5 monclient: authenticate success, global_id 764298 2024-09-16T03:47:49.725 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.703+0000 7fd13b7fe640 1 -- 172.21.15.42:0/4173031119 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd12c0a5750 con 0x7fd13c0a3d90 2024-09-16T03:47:49.725 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.703+0000 7fd14313c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:47:49.725 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.703+0000 7fd13b7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:47:49.725 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.703+0000 7fd14313c640 10 monclient: get_monmap_and_config success 2024-09-16T03:47:49.725 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.703+0000 7fd14313c640 4 set_mon_vals no callback set 2024-09-16T03:47:49.725 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.703+0000 7fd13b7fe640 1 -- 172.21.15.42:0/4173031119 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd12c0a5b20 con 0x7fd13c0a3d90 2024-09-16T03:47:49.725 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.703+0000 7fd13b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:47:49.726 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.703+0000 7fd13b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:47:49.726 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.703+0000 7fd13b7fe640 10 monclient: dump: 2024-09-16T03:47:49.726 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:47:49.726 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:49.726 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:49.726 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:49.726 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:47:49.726 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:49.727 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:47:49.727 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:47:49.727 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:47:49.727 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:49.727 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.703+0000 7fd14313c640 10 monclient: shutdown 2024-09-16T03:47:49.727 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.703+0000 7fd14313c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:47:49.727 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.703+0000 7fd14313c640 1 -- 172.21.15.42:0/4173031119 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd13c0a3d90 msgr2=0x7fd13c13c380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:49.727 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.703+0000 7fd14313c640 1 --2- 172.21.15.42:0/4173031119 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd13c0a3d90 0x7fd13c13c380 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fd12c009e50 tx=0x7fd12c01c8f0 comp rx=0 tx=0).stop 2024-09-16T03:47:49.727 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.703+0000 7fd14313c640 1 -- 172.21.15.42:0/4173031119 shutdown_connections 2024-09-16T03:47:49.728 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.703+0000 7fd14313c640 1 --2- 172.21.15.42:0/4173031119 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd13c13c8c0 0x7fd13c13ecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:49.728 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.703+0000 7fd14313c640 1 --2- 172.21.15.42:0/4173031119 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd13c0a3d90 0x7fd13c13c380 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:49.728 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.703+0000 7fd14313c640 1 -- 172.21.15.42:0/4173031119 >> 172.21.15.42:0/4173031119 conn(0x7fd13c0a9fb0 msgr2=0x7fd13c0aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:47:49.728 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.703+0000 7fd14313c640 1 -- 172.21.15.42:0/4173031119 shutdown_connections 2024-09-16T03:47:49.728 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.703+0000 7fd14313c640 1 -- 172.21.15.42:0/4173031119 wait complete. 2024-09-16T03:47:49.728 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.707+0000 7fd14313c640 10 monclient: build_initial_monmap 2024-09-16T03:47:49.728 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.707+0000 7fd14313c640 10 monclient: monmap: 2024-09-16T03:47:49.728 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:47:49.729 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:47:49.729 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:47:49.729 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:47:49.729 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:47:49.729 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:49.729 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:47:49.729 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:47:49.729 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:47:49.730 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:49.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.707+0000 7fd14313c640 1 librados: starting msgr at 2024-09-16T03:47:49.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.707+0000 7fd14313c640 1 librados: starting objecter 2024-09-16T03:47:49.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.707+0000 7fd14313c640 1 Processor -- start 2024-09-16T03:47:49.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.707+0000 7fd14313c640 1 -- start start 2024-09-16T03:47:49.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.707+0000 7fd14313c640 1 librados: setting wanted keys 2024-09-16T03:47:49.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.707+0000 7fd14313c640 1 librados: calling monclient init 2024-09-16T03:47:49.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.707+0000 7fd14313c640 10 monclient: init 2024-09-16T03:47:49.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.707+0000 7fd14313c640 10 monclient: _reopen_session rank -1 2024-09-16T03:47:49.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.707+0000 7fd14313c640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T03:47:49.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.707+0000 7fd14313c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd13c0a39b0 0x7fd13c138130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:49.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.707+0000 7fd14313c640 10 monclient(hunting): picked mon.noname-a con 0x7fd13c0a39b0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:47:49.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.707+0000 7fd14313c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd13c0a3d90 0x7fd13c138670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:49.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.707+0000 7fd14313c640 10 monclient(hunting): picked mon.noname-b con 0x7fd13c0a3d90 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:47:49.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.707+0000 7fd14313c640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd13c13c8c0 0x7fd13c138bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:49.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.707+0000 7fd14313c640 10 monclient(hunting): picked mon.noname-c con 0x7fd13c13c8c0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:47:49.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.707+0000 7fd14313c640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:49.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.707+0000 7fd14313c640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fd13c141da0 con 0x7fd13c0a39b0 2024-09-16T03:47:49.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.707+0000 7fd14313c640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:49.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.707+0000 7fd14313c640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd13c141c20 con 0x7fd13c0a3d90 2024-09-16T03:47:49.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.707+0000 7fd14313c640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:49.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.707+0000 7fd14313c640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fd13c141f20 con 0x7fd13c13c8c0 2024-09-16T03:47:49.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.707+0000 7fd14313c640 10 monclient(hunting): _renew_subs 2024-09-16T03:47:49.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.707+0000 7fd14313c640 10 monclient(hunting): authenticate will time out at 1316.677403s 2024-09-16T03:47:49.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.707+0000 7fd140ed8640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd13c0a39b0 0x7fd13c138130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:49.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.707+0000 7fd1416d9640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd13c13c8c0 0x7fd13c138bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:49.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.707+0000 7fd140ed8640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd13c0a39b0 0x7fd13c138130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:43360/0 (socket says 172.21.15.42:43360) 2024-09-16T03:47:49.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.707+0000 7fd1416d9640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd13c13c8c0 0x7fd13c138bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:59440/0 (socket says 172.21.15.42:59440) 2024-09-16T03:47:49.733 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.707+0000 7fd140ed8640 1 -- 172.21.15.42:0/4059251985 learned_addr learned my addr 172.21.15.42:0/4059251985 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:47:49.733 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.707+0000 7fd140ed8640 10 monclient(hunting): get_auth_request con 0x7fd13c0a39b0 auth_method 0 2024-09-16T03:47:49.733 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.707+0000 7fd140ed8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:49.733 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.707+0000 7fd140ed8640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:49.733 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.707+0000 7fd140ed8640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:47:49.733 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.707+0000 7fd1416d9640 10 monclient(hunting): get_auth_request con 0x7fd13c13c8c0 auth_method 0 2024-09-16T03:47:49.733 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.707+0000 7fd1416d9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:49.733 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.707+0000 7fd1416d9640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:49.733 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.707+0000 7fd1416d9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:47:49.733 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.707+0000 7fd140ed8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:47:49.733 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.707+0000 7fd140ed8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:47:49.733 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.707+0000 7fd140ed8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:47:49.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.707+0000 7fd1416d9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:47:49.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.707+0000 7fd1416d9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:47:49.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.707+0000 7fd1416d9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:47:49.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.707+0000 7fd13bfff640 1 --2- 172.21.15.42:0/4059251985 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd13c0a3d90 0x7fd13c138670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:49.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.707+0000 7fd13bfff640 10 monclient(hunting): get_auth_request con 0x7fd13c0a3d90 auth_method 0 2024-09-16T03:47:49.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.707+0000 7fd13bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:49.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.707+0000 7fd13bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:49.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.707+0000 7fd13bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:47:49.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.707+0000 7fd140ed8640 10 monclient(hunting): handle_auth_done global_id 764301 payload 824 2024-09-16T03:47:49.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.707+0000 7fd140ed8640 1 -- 172.21.15.42:0/4059251985 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd13c13c8c0 msgr2=0x7fd13c138bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:49.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.707+0000 7fd140ed8640 1 --2- 172.21.15.42:0/4059251985 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd13c13c8c0 0x7fd13c138bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:49.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.707+0000 7fd140ed8640 1 -- 172.21.15.42:0/4059251985 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd13c0a3d90 msgr2=0x7fd13c138670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:49.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.707+0000 7fd140ed8640 1 --2- 172.21.15.42:0/4059251985 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd13c0a3d90 0x7fd13c138670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:49.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.707+0000 7fd140ed8640 10 monclient: _finish_hunting 0 2024-09-16T03:47:49.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.707+0000 7fd140ed8640 1 monclient: found mon.noname-a 2024-09-16T03:47:49.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.707+0000 7fd140ed8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:47:49.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.707+0000 7fd140ed8640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:47:49.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.707+0000 7fd140ed8640 1 -- 172.21.15.42:0/4059251985 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd13c06f680 con 0x7fd13c0a39b0 2024-09-16T03:47:49.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.707+0000 7fd1416d9640 1 --2- 172.21.15.42:0/4059251985 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd13c13c8c0 0x7fd13c138bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:47:49.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.707+0000 7fd140ed8640 1 --2- 172.21.15.42:0/4059251985 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd13c0a39b0 0x7fd13c138130 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fd12400c500 tx=0x7fd12400c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:47:49.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.707+0000 7fd1397fa640 10 client.?.objecter ms_handle_connect 0x7fd13c0a39b0 2024-09-16T03:47:49.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.707+0000 7fd1397fa640 10 client.?.objecter resend_mon_ops 2024-09-16T03:47:49.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.707+0000 7fd1397fa640 1 -- 172.21.15.42:0/4059251985 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd124017070 con 0x7fd13c0a39b0 2024-09-16T03:47:49.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.707+0000 7fd1397fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:47:49.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.707+0000 7fd1397fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:47:49.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.707+0000 7fd1397fa640 10 monclient: dump: 2024-09-16T03:47:49.736 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:47:49.736 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:49.736 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:49.736 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:49.736 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:47:49.736 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:49.736 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:47:49.736 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:47:49.736 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:47:49.736 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:49.736 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.707+0000 7fd1397fa640 10 monclient: _finish_auth 0 2024-09-16T03:47:49.736 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.707+0000 7fd1397fa640 10 monclient: _check_auth_tickets 2024-09-16T03:47:49.736 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.707+0000 7fd1397fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:47:49.737 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.707+0000 7fd1397fa640 1 -- 172.21.15.42:0/4059251985 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd1240052d0 con 0x7fd13c0a39b0 2024-09-16T03:47:49.737 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.707+0000 7fd14313c640 5 monclient: authenticate success, global_id 764301 2024-09-16T03:47:49.737 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.707+0000 7fd14313c640 10 monclient: _renew_subs 2024-09-16T03:47:49.737 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.707+0000 7fd14313c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:47:49.737 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.707+0000 7fd14313c640 1 -- 172.21.15.42:0/4059251985 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd13c132bb0 con 0x7fd13c0a39b0 2024-09-16T03:47:49.737 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.707+0000 7fd1397fa640 10 monclient: handle_config config(0 keys) 2024-09-16T03:47:49.737 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.707+0000 7fd1397fa640 1 -- 172.21.15.42:0/4059251985 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd124003840 con 0x7fd13c0a39b0 2024-09-16T03:47:49.737 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.707+0000 7fd1397fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:47:49.737 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.707+0000 7fd1397fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:47:49.737 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.707+0000 7fd1397fa640 10 monclient: dump: 2024-09-16T03:47:49.737 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:47:49.737 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:49.737 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:49.738 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:49.738 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:47:49.738 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:49.738 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:47:49.738 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:47:49.738 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:47:49.738 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:49.738 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.707+0000 7fd14313c640 10 client.764301.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:47:49.738 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.707+0000 7fd14313c640 10 monclient: _renew_subs 2024-09-16T03:47:49.738 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.707+0000 7fd13affd640 4 set_mon_vals no callback set 2024-09-16T03:47:49.738 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.707+0000 7fd14313c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:47:49.738 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.707+0000 7fd14313c640 1 -- 172.21.15.42:0/4059251985 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd13c130720 con 0x7fd13c0a39b0 2024-09-16T03:47:49.738 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.707+0000 7fd14313c640 1 librados: init done 2024-09-16T03:47:49.738 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.707+0000 7fd14313c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:47:49.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.707+0000 7fd14313c640 1 --2- 172.21.15.42:0/4059251985 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd13c1396e0 0x7fd13c12d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:49.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.707+0000 7fd14313c640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:49.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.707+0000 7fd14313c640 1 -- 172.21.15.42:0/4059251985 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fd13c13c270 con 0x7fd13c1396e0 2024-09-16T03:47:49.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.707+0000 7fd13bfff640 1 --2- 172.21.15.42:0/4059251985 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd13c1396e0 0x7fd13c12d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:49.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.707+0000 7fd13bfff640 10 monclient: get_auth_request con 0x7fd13c1396e0 auth_method 0 2024-09-16T03:47:49.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.707+0000 7fd13bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:49.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.707+0000 7fd13bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:49.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.707+0000 7fd13bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:47:49.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.707+0000 7fd13bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:47:49.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.707+0000 7fd13bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:47:49.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.707+0000 7fd13bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:47:49.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.707+0000 7fd1397fa640 1 -- 172.21.15.42:0/4059251985 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fd12401a070 con 0x7fd13c0a39b0 2024-09-16T03:47:49.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.707+0000 7fd13bfff640 10 monclient(hunting): handle_auth_done global_id 764304 payload 824 2024-09-16T03:47:49.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.707+0000 7fd13bfff640 1 -- 172.21.15.42:0/4059251985 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd13c13a760 con 0x7fd13c1396e0 2024-09-16T03:47:49.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.707+0000 7fd1397fa640 1 --2- 172.21.15.42:0/4059251985 >> v2:172.21.15.154:6812/254845342 conn(0x7fd11c037810 0x7fd11c039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:49.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.707+0000 7fd1397fa640 1 -- 172.21.15.42:0/4059251985 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(179..179 src has 1..179) ==== 6881+0+0 (secure 0 0 0) 0x7fd1240486f0 con 0x7fd13c0a39b0 2024-09-16T03:47:49.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.707+0000 7fd1397fa640 10 client.764301.objecter ms_dispatch 0x7fd13c000ba0 osd_map(179..179 src has 1..179) 2024-09-16T03:47:49.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.707+0000 7fd1397fa640 3 client.764301.objecter handle_osd_map got epochs [179,179] > 0 2024-09-16T03:47:49.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.707+0000 7fd1397fa640 3 client.764301.objecter handle_osd_map decoding full epoch 179 2024-09-16T03:47:49.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.707+0000 7fd13bfff640 1 --2- 172.21.15.42:0/4059251985 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd13c1396e0 0x7fd13c12d2d0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fd13c0a7750 tx=0x7fd12c019690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:47:49.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.707+0000 7fd1397fa640 20 client.764301.objecter dump_active .. 0 homeless 2024-09-16T03:47:49.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.707+0000 7fd1397fa640 10 client.764301.objecter ms_handle_connect 0x7fd13c1396e0 2024-09-16T03:47:49.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.707+0000 7fd1397fa640 10 client.764301.objecter resend_mon_ops 2024-09-16T03:47:49.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.711+0000 7fd1397fa640 1 -- 172.21.15.42:0/4059251985 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd12c009510 con 0x7fd13c1396e0 2024-09-16T03:47:49.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.711+0000 7fd1397fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:47:49.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.711+0000 7fd1397fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:47:49.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.711+0000 7fd1397fa640 10 monclient: dump: 2024-09-16T03:47:49.741 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:47:49.741 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:49.741 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:49.741 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:49.741 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:47:49.741 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:49.741 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:47:49.741 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:47:49.741 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:47:49.741 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:49.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.711+0000 7fd1416d9640 1 --2- 172.21.15.42:0/4059251985 >> v2:172.21.15.154:6812/254845342 conn(0x7fd11c037810 0x7fd11c039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:49.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.711+0000 7fd1416d9640 10 monclient: get_auth_request con 0x7fd11c037810 auth_method 0 2024-09-16T03:47:49.742 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.711+0000 7fd1397fa640 1 -- 172.21.15.42:0/4059251985 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fd12c0b6620 con 0x7fd13c1396e0 2024-09-16T03:47:49.742 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.711+0000 7fd1397fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:47:49.742 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.711+0000 7fd1397fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:47:49.742 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.711+0000 7fd1397fa640 1 -- 172.21.15.42:0/4059251985 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd13c1396e0 msgr2=0x7fd13c12d2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:49.742 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.711+0000 7fd1397fa640 1 --2- 172.21.15.42:0/4059251985 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd13c1396e0 0x7fd13c12d2d0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fd13c0a7750 tx=0x7fd12c019690 comp rx=0 tx=0).stop 2024-09-16T03:47:49.742 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.711+0000 7fd1397fa640 1 -- 172.21.15.42:0/4059251985 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd13c1396e0 msgr2=0x7fd13c12d2d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:47:49.742 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.711+0000 7fd1397fa640 1 --2- 172.21.15.42:0/4059251985 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd13c1396e0 0x7fd13c12d2d0 secure :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fd13c0a7750 tx=0x7fd12c019690 comp rx=0 tx=0).stop 2024-09-16T03:47:49.742 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.711+0000 7fd1416d9640 1 --2- 172.21.15.42:0/4059251985 >> v2:172.21.15.154:6812/254845342 conn(0x7fd11c037810 0x7fd11c039cd0 secure :-1 s=READY pgs=3460 cs=0 l=1 rev1=1 crypto rx=0x7fd13c1313d0 tx=0x7fd134009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:47:49.742 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.711+0000 7fd1397fa640 10 client.764301.objecter ms_handle_connect 0x7fd11c037810 2024-09-16T03:47:49.742 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.715+0000 7fd14313c640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:47:49.742 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.715+0000 7fd14313c640 1 --2- 172.21.15.42:0/4059251985 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd13c12d810 0x7fd13c12dc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:49.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.715+0000 7fd14313c640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:49.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.715+0000 7fd14313c640 1 -- 172.21.15.42:0/4059251985 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fd13c12e160 con 0x7fd13c12d810 2024-09-16T03:47:49.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.715+0000 7fd13bfff640 1 --2- 172.21.15.42:0/4059251985 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd13c12d810 0x7fd13c12dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:49.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.715+0000 7fd13bfff640 10 monclient: get_auth_request con 0x7fd13c12d810 auth_method 0 2024-09-16T03:47:49.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.715+0000 7fd13bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:49.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.715+0000 7fd13bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:49.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.715+0000 7fd13bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:47:49.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.715+0000 7fd13bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:47:49.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.715+0000 7fd13bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:47:49.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.715+0000 7fd13bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:47:49.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.715+0000 7fd13bfff640 10 monclient(hunting): handle_auth_done global_id 764307 payload 824 2024-09-16T03:47:49.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.715+0000 7fd13bfff640 1 -- 172.21.15.42:0/4059251985 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fd13c133b80 con 0x7fd13c12d810 2024-09-16T03:47:49.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.715+0000 7fd1397fa640 10 client.764301.objecter ms_handle_connect 0x7fd13c12d810 2024-09-16T03:47:49.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.715+0000 7fd1397fa640 10 client.764301.objecter resend_mon_ops 2024-09-16T03:47:49.744 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.715+0000 7fd13bfff640 1 --2- 172.21.15.42:0/4059251985 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd13c12d810 0x7fd13c12dc20 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fd12c009e50 tx=0x7fd12c0a7000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:47:49.744 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.715+0000 7fd1397fa640 1 -- 172.21.15.42:0/4059251985 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd12c0c2ab0 con 0x7fd13c12d810 2024-09-16T03:47:49.744 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.715+0000 7fd1397fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:47:49.744 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.715+0000 7fd1397fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:47:49.744 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.715+0000 7fd1397fa640 10 monclient: dump: 2024-09-16T03:47:49.744 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:47:49.744 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:49.744 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:49.744 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:49.744 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:47:49.744 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:49.744 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:47:49.745 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:47:49.745 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:47:49.745 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:49.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.715+0000 7fd1397fa640 1 -- 172.21.15.42:0/4059251985 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4546 (secure 0 0 0) 0x7fd12c003fa0 con 0x7fd13c12d810 2024-09-16T03:47:49.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.715+0000 7fd1397fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:47:49.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.715+0000 7fd1397fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:47:49.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.715+0000 7fd1397fa640 1 -- 172.21.15.42:0/4059251985 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd13c12d810 msgr2=0x7fd13c12dc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:49.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.715+0000 7fd1397fa640 1 --2- 172.21.15.42:0/4059251985 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd13c12d810 0x7fd13c12dc20 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fd12c009e50 tx=0x7fd12c0a7000 comp rx=0 tx=0).stop 2024-09-16T03:47:49.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.715+0000 7fd1397fa640 1 -- 172.21.15.42:0/4059251985 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd13c12d810 msgr2=0x7fd13c12dc20 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:47:49.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.715+0000 7fd1397fa640 1 --2- 172.21.15.42:0/4059251985 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd13c12d810 0x7fd13c12dc20 secure :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fd12c009e50 tx=0x7fd12c0a7000 comp rx=0 tx=0).stop 2024-09-16T03:47:49.745 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:47:49.746 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:47:49.746 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:47:49.746 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "leader", 2024-09-16T03:47:49.746 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 54, 2024-09-16T03:47:49.746 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 60507, 2024-09-16T03:47:49.746 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:47:49.746 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:47:49.746 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T03:47:49.746 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T03:47:49.746 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:47:49.746 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 3, 2024-09-16T03:47:49.746 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:47:49.747 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:47:49.747 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:47:49.747 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:47:49.747 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:47:49.747 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:47:49.747 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:47:49.747 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:47:49.747 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:47:49.747 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:47:49.747 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:47:49.747 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:47:49.747 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:47:49.747 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:47:49.748 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:47:49.748 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:47:49.748 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:47:49.748 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:47:49.748 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:47:49.748 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:47:49.748 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:47:49.748 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:47:49.748 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:47:49.748 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:47:49.748 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:47:49.749 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:47:49.749 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:47:49.749 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:47:49.749 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:47:49.749 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:49.749 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:47:49.749 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:47:49.749 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:47:49.749 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:47:49.749 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:47:49.749 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:47:49.750 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:47:49.750 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:47:49.750 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:47:49.750 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:47:49.750 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:47:49.750 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:47:49.750 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:47:49.750 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:47:49.750 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:47:49.750 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:47:49.750 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:47:49.750 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:47:49.750 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:47:49.750 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:47:49.750 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:47:49.751 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:47:49.751 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:47:49.751 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:47:49.751 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:47:49.751 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:47:49.751 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:47:49.751 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:47:49.751 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:47:49.751 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:47:49.751 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:49.751 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:47:49.751 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:49.751 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:47:49.751 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:47:49.751 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:47:49.752 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:47:49.752 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:49.752 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:47:49.752 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:47:49.752 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:47:49.752 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:49.752 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:49.752 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:47:49.752 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:47:49.752 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:47:49.752 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:47:49.752 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:47:49.752 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:49.753 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:47:49.753 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:47:49.753 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:47:49.753 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:47:49.753 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:47:49.753 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:49.753 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:49.753 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:47:49.753 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:47:49.753 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:47:49.753 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:47:49.753 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:49.753 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:47:49.754 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:47:49.754 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:47:49.754 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:49.754 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:49.754 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:47:49.754 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:47:49.754 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:47:49.754 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:47:49.754 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:47:49.754 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:49.754 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:47:49.754 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:47:49.754 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:47:49.754 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:47:49.754 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:47:49.755 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:49.755 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:49.755 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:47:49.755 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:47:49.755 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:47:49.755 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:47:49.755 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:49.755 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:47:49.755 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:47:49.755 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:47:49.755 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:49.755 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:49.755 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:47:49.755 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:47:49.755 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:47:49.756 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:47:49.756 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:47:49.756 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:49.756 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:47:49.756 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:47:49.756 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:47:49.756 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:47:49.756 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:47:49.756 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:47:49.756 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:47:49.756 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:49.756 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:47:49.757 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:47:49.757 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:49.757 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:47:49.757 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:47:49.757 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:47:49.757 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:47:49.757 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:47:49.757 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:47:49.757 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:49.757 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:47:49.757 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:47:49.757 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 4 2024-09-16T03:47:49.757 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:47:49.757 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:47:49.758 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:49.758 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:47:49.758 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:47:49.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.715+0000 7fd14313c640 10 librados: watch_flush enter 2024-09-16T03:47:49.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.715+0000 7fd14313c640 10 librados: watch_flush exit 2024-09-16T03:47:49.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.715+0000 7fd14313c640 20 client.764301.objecter shutdown clearing up homeless session... 2024-09-16T03:47:49.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.715+0000 7fd14313c640 10 client.764301.objecter successfully canceled tick 2024-09-16T03:47:49.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.715+0000 7fd14313c640 1 -- 172.21.15.42:0/4059251985 >> v2:172.21.15.154:6812/254845342 conn(0x7fd11c037810 msgr2=0x7fd11c039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:49.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.715+0000 7fd14313c640 1 --2- 172.21.15.42:0/4059251985 >> v2:172.21.15.154:6812/254845342 conn(0x7fd11c037810 0x7fd11c039cd0 secure :-1 s=READY pgs=3460 cs=0 l=1 rev1=1 crypto rx=0x7fd13c1313d0 tx=0x7fd134009190 comp rx=0 tx=0).stop 2024-09-16T03:47:49.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.715+0000 7fd14313c640 10 monclient: shutdown 2024-09-16T03:47:49.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.715+0000 7fd14313c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:47:49.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.715+0000 7fd14313c640 1 -- 172.21.15.42:0/4059251985 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd13c0a39b0 msgr2=0x7fd13c138130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:49.761 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.715+0000 7fd14313c640 1 --2- 172.21.15.42:0/4059251985 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd13c0a39b0 0x7fd13c138130 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fd12400c500 tx=0x7fd12400c900 comp rx=0 tx=0).stop 2024-09-16T03:47:49.761 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.715+0000 7fd1416d9640 1 -- 172.21.15.42:0/4059251985 reap_dead start 2024-09-16T03:47:49.761 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.719+0000 7fd1416d9640 1 -- 172.21.15.42:0/4059251985 reap_dead start 2024-09-16T03:47:49.761 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.719+0000 7fd14313c640 1 -- 172.21.15.42:0/4059251985 shutdown_connections 2024-09-16T03:47:49.761 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.719+0000 7fd14313c640 1 -- 172.21.15.42:0/4059251985 >> 172.21.15.42:0/4059251985 conn(0x7fd13c0a9fb0 msgr2=0x7fd13c131980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:47:49.761 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.719+0000 7fd14313c640 1 -- 172.21.15.42:0/4059251985 shutdown_connections 2024-09-16T03:47:49.761 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.719+0000 7fd14313c640 1 -- 172.21.15.42:0/4059251985 wait complete. 2024-09-16T03:47:49.761 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.719+0000 7fd14313c640 1 librados: shutdown 2024-09-16T03:47:49.761 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:47:49.761 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T03:47:49.762 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T03:47:49.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.875+0000 7f63a6b7a640 10 monclient: get_monmap_and_config 2024-09-16T03:47:49.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.875+0000 7f63a6b7a640 10 monclient: build_initial_monmap 2024-09-16T03:47:49.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.875+0000 7f63a6b7a640 10 monclient: monmap: 2024-09-16T03:47:49.879 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:47:49.879 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:49.879 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:47:49.878470+0000 2024-09-16T03:47:49.880 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:47:49.878470+0000 2024-09-16T03:47:49.880 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:47:49.880 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:49.880 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:47:49.880 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:47:49.880 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:47:49.880 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:47:49.880 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:47:49.880 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:49.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.875+0000 7f63a6b7a640 1 Processor -- start 2024-09-16T03:47:49.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.875+0000 7f63a6b7a640 1 -- start start 2024-09-16T03:47:49.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.875+0000 7f63a6b7a640 10 monclient: init 2024-09-16T03:47:49.883 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.875+0000 7f63a6b7a640 10 monclient: _reopen_session rank -1 2024-09-16T03:47:49.883 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.875+0000 7f63a6b7a640 10 monclient: _add_conns ranks=[3,1,4,2,0] 2024-09-16T03:47:49.883 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.875+0000 7f63a6b7a640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f63a00a1330 0x7f63a00a1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:49.883 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.875+0000 7f63a6b7a640 10 monclient(hunting): picked mon.noname-a con 0x7f63a00a1330 addr v2:172.21.15.42:3300/0 2024-09-16T03:47:49.883 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.875+0000 7f63a6b7a640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f63a006dac0 addr v1:172.21.15.154:6789/0 2024-09-16T03:47:49.883 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.875+0000 7f63a6b7a640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f63a00a1c50 0x7f63a0135810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:49.883 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.875+0000 7f63a6b7a640 10 monclient(hunting): picked mon.noname-c con 0x7f63a00a1c50 addr v2:172.21.15.154:3300/0 2024-09-16T03:47:49.883 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.875+0000 7f63a6b7a640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f63a009c090 con 0x7f63a006dac0 2024-09-16T03:47:49.883 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.875+0000 7f63a6b7a640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:49.883 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.875+0000 7f63a6b7a640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f63a009f610 con 0x7f63a00a1330 2024-09-16T03:47:49.883 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.875+0000 7f63a6b7a640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:49.883 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.875+0000 7f63a6b7a640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f63a009f8f0 con 0x7f63a00a1c50 2024-09-16T03:47:49.883 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.875+0000 7f63a6b7a640 10 monclient(hunting): _renew_subs 2024-09-16T03:47:49.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.875+0000 7f63a6b7a640 10 monclient(hunting): authenticate will time out at 1316.847941s 2024-09-16T03:47:49.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.879+0000 7f63a4916640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f63a00a1330 0x7f63a00a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:49.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.879+0000 7f63a4916640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f63a00a1330 0x7f63a00a1710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:43386/0 (socket says 172.21.15.42:43386) 2024-09-16T03:47:49.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.879+0000 7f63a4916640 1 -- 172.21.15.42:0/1662725569 learned_addr learned my addr 172.21.15.42:0/1662725569 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:47:49.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.879+0000 7f63a4916640 10 monclient(hunting): get_auth_request con 0x7f63a00a1330 auth_method 0 2024-09-16T03:47:49.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.879+0000 7f63a4916640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:49.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.879+0000 7f63a4916640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:49.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.879+0000 7f63a4916640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:47:49.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.879+0000 7f63a4916640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:47:49.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.879+0000 7f63a4916640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:47:49.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.879+0000 7f63a5117640 1 --2- 172.21.15.42:0/1662725569 >> v2:172.21.15.154:3300/0 conn(0x7f63a00a1c50 0x7f63a0135810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:49.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.879+0000 7f63a4916640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:47:49.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.879+0000 7f63a5117640 10 monclient(hunting): get_auth_request con 0x7f63a00a1c50 auth_method 0 2024-09-16T03:47:49.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.879+0000 7f63a5117640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:49.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.879+0000 7f63a5117640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:49.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.879+0000 7f63a5117640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:47:49.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.879+0000 7f63a4916640 10 monclient(hunting): handle_auth_done global_id 764310 payload 274 2024-09-16T03:47:49.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.879+0000 7f63a5117640 1 -- 172.21.15.42:0/1662725569 >> v2:172.21.15.154:3300/0 conn(0x7f63a00a1c50 msgr2=0x7f63a0135810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T03:47:49.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.879+0000 7f63a5117640 1 -- 172.21.15.42:0/1662725569 >> v2:172.21.15.154:3300/0 conn(0x7f63a00a1c50 msgr2=0x7f63a0135810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:47:49.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.879+0000 7f63a5117640 1 --2- 172.21.15.42:0/1662725569 >> v2:172.21.15.154:3300/0 conn(0x7f63a00a1c50 0x7f63a0135810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:47:49.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.879+0000 7f63a4916640 1 -- 172.21.15.42:0/1662725569 >> v2:172.21.15.154:3300/0 conn(0x7f63a00a1c50 msgr2=0x7f63a0135810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:47:49.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.879+0000 7f63a5117640 1 --2- 172.21.15.42:0/1662725569 >> v2:172.21.15.154:3300/0 conn(0x7f63a00a1c50 0x7f63a0135810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:47:49.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.879+0000 7f63a4916640 1 --2- 172.21.15.42:0/1662725569 >> v2:172.21.15.154:3300/0 conn(0x7f63a00a1c50 0x7f63a0135810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:49.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.879+0000 7f63a4916640 1 -- 172.21.15.42:0/1662725569 >> v1:172.21.15.154:6789/0 conn(0x7f63a006dac0 legacy=0x7f63a0135100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:47:49.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.879+0000 7f63a4916640 10 monclient: _finish_hunting 0 2024-09-16T03:47:49.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.879+0000 7f63a4916640 1 monclient: found mon.noname-a 2024-09-16T03:47:49.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.879+0000 7f63a4916640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:47:49.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.879+0000 7f63a4916640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:47:49.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.879+0000 7f63a4916640 1 -- 172.21.15.42:0/1662725569 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63a009fbd0 con 0x7f63a00a1330 2024-09-16T03:47:49.894 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.883+0000 7f63a4916640 1 --2- 172.21.15.42:0/1662725569 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f63a00a1330 0x7f63a00a1710 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f63a009c800 tx=0x7f638801c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d3286202b3f81a0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:47:49.895 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.883+0000 7f639f7fe640 1 -- 172.21.15.42:0/1662725569 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63880b1070 con 0x7f63a00a1330 2024-09-16T03:47:49.895 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.883+0000 7f639f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:47:49.895 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.883+0000 7f639f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:47:49.895 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.883+0000 7f639f7fe640 10 monclient: dump: 2024-09-16T03:47:49.895 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:47:49.895 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:49.895 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:49.895 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:49.896 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:47:49.896 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:49.896 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:47:49.896 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:47:49.896 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:47:49.896 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:49.896 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.883+0000 7f639f7fe640 10 monclient: _finish_auth 0 2024-09-16T03:47:49.896 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.883+0000 7f639f7fe640 10 monclient: _check_auth_tickets 2024-09-16T03:47:49.896 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.883+0000 7f639f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:47:49.896 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.883+0000 7f639f7fe640 1 -- 172.21.15.42:0/1662725569 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f63880ac620 con 0x7f63a00a1330 2024-09-16T03:47:49.896 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.883+0000 7f63a6b7a640 5 monclient: authenticate success, global_id 764310 2024-09-16T03:47:49.896 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.883+0000 7f63a6b7a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:47:49.896 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.883+0000 7f639f7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:47:49.897 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.883+0000 7f63a6b7a640 10 monclient: get_monmap_and_config success 2024-09-16T03:47:49.897 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.883+0000 7f639f7fe640 1 -- 172.21.15.42:0/1662725569 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63880ac9f0 con 0x7f63a00a1330 2024-09-16T03:47:49.897 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.883+0000 7f639f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:47:49.897 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.883+0000 7f63a6b7a640 4 set_mon_vals no callback set 2024-09-16T03:47:49.897 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.883+0000 7f639f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:47:49.897 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.883+0000 7f639f7fe640 10 monclient: dump: 2024-09-16T03:47:49.897 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:47:49.897 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:49.897 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:49.897 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:49.897 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:47:49.897 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:49.897 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:47:49.897 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:47:49.898 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:47:49.898 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:49.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.883+0000 7f63a6b7a640 10 monclient: shutdown 2024-09-16T03:47:49.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.883+0000 7f63a6b7a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:47:49.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.883+0000 7f63a6b7a640 1 -- 172.21.15.42:0/1662725569 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f63a00a1330 msgr2=0x7f63a00a1710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:49.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.883+0000 7f63a6b7a640 1 --2- 172.21.15.42:0/1662725569 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f63a00a1330 0x7f63a00a1710 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f63a009c800 tx=0x7f638801c3a0 comp rx=0 tx=0).stop 2024-09-16T03:47:49.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.883+0000 7f63a6b7a640 1 -- 172.21.15.42:0/1662725569 shutdown_connections 2024-09-16T03:47:49.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.883+0000 7f63a6b7a640 1 --2- 172.21.15.42:0/1662725569 >> v2:172.21.15.154:3300/0 conn(0x7f63a00a1c50 0x7f63a0135810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:49.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.883+0000 7f63a6b7a640 1 --2- 172.21.15.42:0/1662725569 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f63a00a1330 0x7f63a00a1710 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:49.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.883+0000 7f63a6b7a640 1 -- 172.21.15.42:0/1662725569 >> 172.21.15.42:0/1662725569 conn(0x7f63a00a2420 msgr2=0x7f63a00a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:47:49.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.883+0000 7f63a6b7a640 1 -- 172.21.15.42:0/1662725569 shutdown_connections 2024-09-16T03:47:49.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.883+0000 7f63a6b7a640 1 -- 172.21.15.42:0/1662725569 wait complete. 2024-09-16T03:47:49.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.883+0000 7f63a6b7a640 10 monclient: build_initial_monmap 2024-09-16T03:47:49.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.883+0000 7f63a6b7a640 10 monclient: monmap: 2024-09-16T03:47:49.899 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:47:49.899 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:47:49.899 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:47:49.899 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:47:49.899 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:47:49.899 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:49.899 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:47:49.899 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:47:49.899 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:47:49.899 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:49.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.883+0000 7f63a6b7a640 1 librados: starting msgr at 2024-09-16T03:47:49.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.883+0000 7f63a6b7a640 1 librados: starting objecter 2024-09-16T03:47:49.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.883+0000 7f63a6b7a640 1 Processor -- start 2024-09-16T03:47:49.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.883+0000 7f63a6b7a640 1 -- start start 2024-09-16T03:47:49.900 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.883+0000 7f63a6b7a640 1 librados: setting wanted keys 2024-09-16T03:47:49.900 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.883+0000 7f63a6b7a640 1 librados: calling monclient init 2024-09-16T03:47:49.900 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.883+0000 7f63a6b7a640 10 monclient: init 2024-09-16T03:47:49.900 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.883+0000 7f63a6b7a640 10 monclient: _reopen_session rank -1 2024-09-16T03:47:49.900 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.883+0000 7f63a6b7a640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:47:49.900 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.883+0000 7f63a6b7a640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f63a00a1330 0x7f63a01caa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:49.900 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.883+0000 7f63a6b7a640 10 monclient(hunting): picked mon.noname-c con 0x7f63a00a1330 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:47:49.900 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.883+0000 7f63a6b7a640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f63a00a1c50 0x7f63a01cb230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:49.900 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.883+0000 7f63a6b7a640 10 monclient(hunting): picked mon.noname-b con 0x7f63a00a1c50 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:47:49.900 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.883+0000 7f63a6b7a640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f63a01cb770 0x7f63a013aca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:49.900 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.883+0000 7f63a6b7a640 10 monclient(hunting): picked mon.noname-a con 0x7f63a01cb770 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:47:49.900 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.883+0000 7f63a6b7a640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:49.900 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.883+0000 7f63a6b7a640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f63a009cb60 con 0x7f63a01cb770 2024-09-16T03:47:49.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.883+0000 7f63a6b7a640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:49.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.883+0000 7f63a6b7a640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f63a0135f60 con 0x7f63a00a1c50 2024-09-16T03:47:49.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.883+0000 7f63a6b7a640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:49.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.883+0000 7f63a6b7a640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f63a009cce0 con 0x7f63a00a1330 2024-09-16T03:47:49.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.883+0000 7f63a6b7a640 10 monclient(hunting): _renew_subs 2024-09-16T03:47:49.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.883+0000 7f63a6b7a640 10 monclient(hunting): authenticate will time out at 1316.854931s 2024-09-16T03:47:49.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.883+0000 7f63a5117640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f63a01cb770 0x7f63a013aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:49.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.883+0000 7f63a4916640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f63a00a1330 0x7f63a01caa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:49.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.883+0000 7f63a5117640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f63a01cb770 0x7f63a013aca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:43392/0 (socket says 172.21.15.42:43392) 2024-09-16T03:47:49.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.883+0000 7f63a5117640 1 -- 172.21.15.42:0/2092736583 learned_addr learned my addr 172.21.15.42:0/2092736583 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:47:49.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.883+0000 7f63a5117640 10 monclient(hunting): get_auth_request con 0x7f63a01cb770 auth_method 0 2024-09-16T03:47:49.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.883+0000 7f63a5117640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:49.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.883+0000 7f63a5117640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:49.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.883+0000 7f63a5117640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:47:49.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.883+0000 7f63a4916640 10 monclient(hunting): get_auth_request con 0x7f63a00a1330 auth_method 0 2024-09-16T03:47:49.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.883+0000 7f63a4916640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:49.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.883+0000 7f63a4916640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:49.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.883+0000 7f63a4916640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:47:49.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.883+0000 7f63a5117640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:47:49.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.883+0000 7f63a5117640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:47:49.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.883+0000 7f63a5117640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:47:49.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.883+0000 7f63a4916640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:47:49.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.883+0000 7f639ffff640 1 --2- 172.21.15.42:0/2092736583 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f63a00a1c50 0x7f63a01cb230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:49.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.883+0000 7f63a4916640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:47:49.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.883+0000 7f63a4916640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:47:49.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.883+0000 7f639ffff640 10 monclient(hunting): get_auth_request con 0x7f63a00a1c50 auth_method 0 2024-09-16T03:47:49.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.883+0000 7f639ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:49.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.883+0000 7f639ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:49.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.883+0000 7f639ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:47:49.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.883+0000 7f63a5117640 10 monclient(hunting): handle_auth_done global_id 764313 payload 824 2024-09-16T03:47:49.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.883+0000 7f63a5117640 1 -- 172.21.15.42:0/2092736583 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f63a00a1330 msgr2=0x7f63a01caa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:49.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.883+0000 7f63a5117640 1 --2- 172.21.15.42:0/2092736583 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f63a00a1330 0x7f63a01caa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:49.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.883+0000 7f63a5117640 1 -- 172.21.15.42:0/2092736583 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f63a00a1c50 msgr2=0x7f63a01cb230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:49.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.883+0000 7f63a5117640 1 --2- 172.21.15.42:0/2092736583 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f63a00a1c50 0x7f63a01cb230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:49.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.883+0000 7f63a5117640 10 monclient: _finish_hunting 0 2024-09-16T03:47:49.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.883+0000 7f63a5117640 1 monclient: found mon.noname-a 2024-09-16T03:47:49.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.883+0000 7f63a5117640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:47:49.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.883+0000 7f63a5117640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:47:49.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.883+0000 7f63a5117640 1 -- 172.21.15.42:0/2092736583 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63a011d2d0 con 0x7f63a01cb770 2024-09-16T03:47:49.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.883+0000 7f63a5117640 1 --2- 172.21.15.42:0/2092736583 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f63a01cb770 0x7f63a013aca0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f6398006e90 tx=0x7f639800ea60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:47:49.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.883+0000 7f639d7fa640 10 client.?.objecter ms_handle_connect 0x7f63a01cb770 2024-09-16T03:47:49.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.883+0000 7f639d7fa640 10 client.?.objecter resend_mon_ops 2024-09-16T03:47:49.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.883+0000 7f639d7fa640 1 -- 172.21.15.42:0/2092736583 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f639800f040 con 0x7f63a01cb770 2024-09-16T03:47:49.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.883+0000 7f639d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:47:49.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.883+0000 7f639d7fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:47:49.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.883+0000 7f639d7fa640 10 monclient: dump: 2024-09-16T03:47:49.904 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:47:49.904 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:49.904 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:49.904 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:49.904 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:47:49.904 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:49.904 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:47:49.904 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:47:49.904 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:47:49.905 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:49.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.883+0000 7f639d7fa640 10 monclient: _finish_auth 0 2024-09-16T03:47:49.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.883+0000 7f639d7fa640 10 monclient: _check_auth_tickets 2024-09-16T03:47:49.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.883+0000 7f639d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:47:49.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.883+0000 7f63a6b7a640 5 monclient: authenticate success, global_id 764313 2024-09-16T03:47:49.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.883+0000 7f639d7fa640 1 -- 172.21.15.42:0/2092736583 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6398018070 con 0x7f63a01cb770 2024-09-16T03:47:49.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.883+0000 7f639d7fa640 10 monclient: handle_config config(0 keys) 2024-09-16T03:47:49.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.883+0000 7f63a6b7a640 10 monclient: _renew_subs 2024-09-16T03:47:49.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.883+0000 7f639d7fa640 1 -- 172.21.15.42:0/2092736583 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6398013870 con 0x7f63a01cb770 2024-09-16T03:47:49.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.883+0000 7f63a6b7a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:47:49.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.883+0000 7f63a6b7a640 1 -- 172.21.15.42:0/2092736583 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63a013cde0 con 0x7f63a01cb770 2024-09-16T03:47:49.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.883+0000 7f639d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:47:49.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.883+0000 7f639d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:47:49.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.883+0000 7f639d7fa640 10 monclient: dump: 2024-09-16T03:47:49.906 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:47:49.906 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:49.906 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:49.906 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:49.906 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:47:49.906 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:49.906 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:47:49.906 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:47:49.906 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:47:49.906 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:49.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.883+0000 7f639effd640 4 set_mon_vals no callback set 2024-09-16T03:47:49.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.883+0000 7f63a6b7a640 10 client.764313.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:47:49.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.883+0000 7f63a6b7a640 10 monclient: _renew_subs 2024-09-16T03:47:49.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.883+0000 7f63a6b7a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:47:49.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.883+0000 7f63a6b7a640 1 -- 172.21.15.42:0/2092736583 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63a00a4160 con 0x7f63a01cb770 2024-09-16T03:47:49.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.883+0000 7f63a6b7a640 1 librados: init done 2024-09-16T03:47:49.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.887+0000 7f63a6b7a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:47:49.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.887+0000 7f63a6b7a640 1 --2- 172.21.15.42:0/2092736583 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f63a006dac0 0x7f63a002c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:49.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.887+0000 7f63a6b7a640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:49.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.887+0000 7f63a6b7a640 1 -- 172.21.15.42:0/2092736583 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f63a0073a30 con 0x7f63a006dac0 2024-09-16T03:47:49.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.887+0000 7f639d7fa640 1 -- 172.21.15.42:0/2092736583 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f6398013a10 con 0x7f63a01cb770 2024-09-16T03:47:49.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.887+0000 7f63a4916640 1 --2- 172.21.15.42:0/2092736583 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f63a006dac0 0x7f63a002c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:49.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.887+0000 7f63a4916640 10 monclient: get_auth_request con 0x7f63a006dac0 auth_method 0 2024-09-16T03:47:49.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.887+0000 7f63a4916640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:49.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.887+0000 7f63a4916640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:49.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.887+0000 7f63a4916640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:47:49.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.887+0000 7f63a4916640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:47:49.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.887+0000 7f63a4916640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:47:49.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.887+0000 7f63a4916640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:47:49.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.887+0000 7f639d7fa640 1 --2- 172.21.15.42:0/2092736583 >> v2:172.21.15.154:6812/254845342 conn(0x7f6380037810 0x7f6380039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:49.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.887+0000 7f639d7fa640 1 -- 172.21.15.42:0/2092736583 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(179..179 src has 1..179) ==== 6881+0+0 (secure 0 0 0) 0x7f6398048d30 con 0x7f63a01cb770 2024-09-16T03:47:49.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.887+0000 7f639d7fa640 10 client.764313.objecter ms_dispatch 0x7f63a0000ba0 osd_map(179..179 src has 1..179) 2024-09-16T03:47:49.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.887+0000 7f639d7fa640 3 client.764313.objecter handle_osd_map got epochs [179,179] > 0 2024-09-16T03:47:49.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.887+0000 7f639d7fa640 3 client.764313.objecter handle_osd_map decoding full epoch 179 2024-09-16T03:47:49.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.887+0000 7f63a4916640 10 monclient(hunting): handle_auth_done global_id 13838 payload 824 2024-09-16T03:47:49.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.887+0000 7f63a4916640 1 -- 172.21.15.42:0/2092736583 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f63a00a1710 con 0x7f63a006dac0 2024-09-16T03:47:49.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.887+0000 7f639d7fa640 20 client.764313.objecter dump_active .. 0 homeless 2024-09-16T03:47:49.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.887+0000 7f639d7fa640 10 client.764313.objecter ms_handle_connect 0x7f63a006dac0 2024-09-16T03:47:49.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.887+0000 7f639d7fa640 10 client.764313.objecter resend_mon_ops 2024-09-16T03:47:49.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.887+0000 7f63a4916640 1 --2- 172.21.15.42:0/2092736583 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f63a006dac0 0x7f63a002c650 secure :-1 s=READY pgs=2502 cs=0 l=1 rev1=1 crypto rx=0x7f63a006da10 tx=0x7f6388019140 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:47:49.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.887+0000 7f639ffff640 1 --2- 172.21.15.42:0/2092736583 >> v2:172.21.15.154:6812/254845342 conn(0x7f6380037810 0x7f6380039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:49.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.887+0000 7f639ffff640 10 monclient: get_auth_request con 0x7f6380037810 auth_method 0 2024-09-16T03:47:49.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.887+0000 7f639d7fa640 1 -- 172.21.15.42:0/2092736583 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63880b1040 con 0x7f63a006dac0 2024-09-16T03:47:49.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.887+0000 7f639d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:47:49.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.887+0000 7f639d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:47:49.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.887+0000 7f639d7fa640 10 monclient: dump: 2024-09-16T03:47:49.909 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:47:49.909 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:49.909 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:49.909 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:49.909 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:47:49.909 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:49.909 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:47:49.909 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:47:49.909 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:47:49.909 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:49.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.887+0000 7f639d7fa640 1 -- 172.21.15.42:0/2092736583 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f63880b6440 con 0x7f63a006dac0 2024-09-16T03:47:49.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.887+0000 7f639d7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:47:49.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.887+0000 7f639d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:47:49.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.887+0000 7f639d7fa640 1 -- 172.21.15.42:0/2092736583 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f63a006dac0 msgr2=0x7f63a002c650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:49.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.887+0000 7f639d7fa640 1 --2- 172.21.15.42:0/2092736583 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f63a006dac0 0x7f63a002c650 secure :-1 s=READY pgs=2502 cs=0 l=1 rev1=1 crypto rx=0x7f63a006da10 tx=0x7f6388019140 comp rx=0 tx=0).stop 2024-09-16T03:47:49.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.887+0000 7f639d7fa640 1 -- 172.21.15.42:0/2092736583 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f63a006dac0 msgr2=0x7f63a002c650 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:47:49.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.887+0000 7f639d7fa640 1 --2- 172.21.15.42:0/2092736583 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f63a006dac0 0x7f63a002c650 secure :-1 s=CLOSED pgs=2502 cs=0 l=1 rev1=1 crypto rx=0x7f63a006da10 tx=0x7f6388019140 comp rx=0 tx=0).stop 2024-09-16T03:47:49.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.887+0000 7f639d7fa640 10 client.764313.objecter ms_handle_connect 0x7f6380037810 2024-09-16T03:47:49.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.887+0000 7f639ffff640 1 --2- 172.21.15.42:0/2092736583 >> v2:172.21.15.154:6812/254845342 conn(0x7f6380037810 0x7f6380039cd0 secure :-1 s=READY pgs=3461 cs=0 l=1 rev1=1 crypto rx=0x7f63a013d330 tx=0x7f6390009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:47:49.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.895+0000 7f63a6b7a640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:47:49.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.895+0000 7f63a6b7a640 1 --2- 172.21.15.42:0/2092736583 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f63a002cb90 0x7f63a002cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:49.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.895+0000 7f63a6b7a640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:49.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.895+0000 7f63a6b7a640 1 -- 172.21.15.42:0/2092736583 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f63a0137a30 con 0x7f63a002cb90 2024-09-16T03:47:49.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.895+0000 7f63a4916640 1 --2- 172.21.15.42:0/2092736583 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f63a002cb90 0x7f63a002cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:49.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.895+0000 7f63a4916640 10 monclient: get_auth_request con 0x7f63a002cb90 auth_method 0 2024-09-16T03:47:49.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.895+0000 7f63a4916640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:49.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.895+0000 7f63a4916640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:49.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.895+0000 7f63a4916640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:47:49.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.895+0000 7f63a4916640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:47:49.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.895+0000 7f63a4916640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:47:49.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.895+0000 7f63a4916640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:47:49.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.895+0000 7f63a4916640 10 monclient(hunting): handle_auth_done global_id 13841 payload 824 2024-09-16T03:47:49.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.895+0000 7f63a4916640 1 -- 172.21.15.42:0/2092736583 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f63a011cff0 con 0x7f63a002cb90 2024-09-16T03:47:49.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.895+0000 7f63a4916640 1 --2- 172.21.15.42:0/2092736583 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f63a002cb90 0x7f63a002cfa0 secure :-1 s=READY pgs=2503 cs=0 l=1 rev1=1 crypto rx=0x7f6388019fd0 tx=0x7f6388009240 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:47:49.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.895+0000 7f639d7fa640 10 client.764313.objecter ms_handle_connect 0x7f63a002cb90 2024-09-16T03:47:49.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.895+0000 7f639d7fa640 10 client.764313.objecter resend_mon_ops 2024-09-16T03:47:49.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.895+0000 7f639d7fa640 1 -- 172.21.15.42:0/2092736583 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63880c2c00 con 0x7f63a002cb90 2024-09-16T03:47:49.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.895+0000 7f639d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:47:49.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.895+0000 7f639d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:47:49.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.895+0000 7f639d7fa640 10 monclient: dump: 2024-09-16T03:47:49.913 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:47:49.913 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:49.913 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:49.914 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:49.914 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:47:49.914 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:49.914 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:47:49.914 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:47:49.914 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:47:49.914 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:49.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.895+0000 7f639d7fa640 1 -- 172.21.15.42:0/2092736583 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4879 (secure 0 0 0) 0x7f63880c2c00 con 0x7f63a002cb90 2024-09-16T03:47:49.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.895+0000 7f639d7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:47:49.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.895+0000 7f639d7fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:47:49.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.895+0000 7f639d7fa640 1 -- 172.21.15.42:0/2092736583 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f63a002cb90 msgr2=0x7f63a002cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:49.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.895+0000 7f639d7fa640 1 --2- 172.21.15.42:0/2092736583 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f63a002cb90 0x7f63a002cfa0 secure :-1 s=READY pgs=2503 cs=0 l=1 rev1=1 crypto rx=0x7f6388019fd0 tx=0x7f6388009240 comp rx=0 tx=0).stop 2024-09-16T03:47:49.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.895+0000 7f639d7fa640 1 -- 172.21.15.42:0/2092736583 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f63a002cb90 msgr2=0x7f63a002cfa0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:47:49.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.895+0000 7f639d7fa640 1 --2- 172.21.15.42:0/2092736583 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f63a002cb90 0x7f63a002cfa0 secure :-1 s=CLOSED pgs=2503 cs=0 l=1 rev1=1 crypto rx=0x7f6388019fd0 tx=0x7f6388009240 comp rx=0 tx=0).stop 2024-09-16T03:47:49.915 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:47:49.915 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:47:49.915 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:47:49.915 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T03:47:49.916 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 54, 2024-09-16T03:47:49.916 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 349266, 2024-09-16T03:47:49.916 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:47:49.916 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:47:49.916 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T03:47:49.916 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T03:47:49.916 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:47:49.916 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 3, 2024-09-16T03:47:49.916 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:47:49.916 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:47:49.916 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:47:49.916 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:47:49.916 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:47:49.916 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:47:49.917 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:47:49.917 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:47:49.917 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:47:49.917 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:47:49.917 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:47:49.917 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:47:49.917 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:47:49.917 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:47:49.917 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:47:49.917 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:47:49.917 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:47:49.917 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:47:49.917 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:47:49.918 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:47:49.918 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:47:49.918 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:47:49.918 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:47:49.918 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:47:49.918 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:47:49.918 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:47:49.918 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:47:49.918 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:47:49.918 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:47:49.918 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:49.918 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:47:49.918 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:47:49.918 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:47:49.919 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:47:49.919 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:47:49.919 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:47:49.919 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:47:49.919 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:47:49.919 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:47:49.919 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:47:49.919 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:47:49.919 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:47:49.919 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:47:49.919 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:47:49.919 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:47:49.919 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:47:49.919 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:47:49.920 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:47:49.920 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:47:49.920 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:47:49.920 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:47:49.920 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:47:49.920 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:47:49.920 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:47:49.920 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:47:49.920 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:47:49.920 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:47:49.920 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:47:49.920 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:47:49.920 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:47:49.921 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:49.921 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:47:49.921 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:49.921 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:47:49.921 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:47:49.921 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:47:49.921 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:47:49.921 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:49.921 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:47:49.921 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:47:49.921 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:47:49.921 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:49.921 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:49.921 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:47:49.921 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:47:49.922 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:47:49.922 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:47:49.922 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:47:49.922 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:49.922 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:47:49.922 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:47:49.922 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:47:49.922 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:47:49.922 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:47:49.922 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:49.922 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:49.922 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:47:49.922 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:47:49.922 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:47:49.923 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:47:49.923 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:49.923 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:47:49.923 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:47:49.923 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:47:49.923 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:49.923 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:49.923 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:47:49.923 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:47:49.923 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:47:49.923 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:47:49.923 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:47:49.923 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:49.924 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:47:49.924 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:47:49.924 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:47:49.924 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:47:49.924 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:47:49.924 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:49.924 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:49.924 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:47:49.924 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:47:49.924 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:47:49.924 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:47:49.924 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:49.924 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:47:49.924 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:47:49.925 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:47:49.925 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:49.925 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:49.925 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:47:49.925 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:47:49.925 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:47:49.925 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:47:49.925 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:47:49.925 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:49.925 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:47:49.925 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:47:49.925 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:47:49.925 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:47:49.926 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:47:49.926 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:47:49.926 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:47:49.926 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:49.926 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:47:49.926 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:47:49.926 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:49.926 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:47:49.926 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:47:49.927 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:47:49.927 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:47:49.927 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:47:49.927 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T03:47:49.927 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:49.927 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:47:49.927 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:47:49.927 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 6 2024-09-16T03:47:49.927 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:47:49.927 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:47:49.927 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:47:49.927 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:49.927 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:47:49.927 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:47:49.928 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 7 2024-09-16T03:47:49.928 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:47:49.928 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:47:49.928 INFO:teuthology.orchestra.run.smithi042.stdout: "mgr": [ 2024-09-16T03:47:49.928 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:49.928 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:47:49.928 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:47:49.928 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:47:49.928 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:47:49.928 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:47:49.928 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:49.928 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:47:49.928 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:47:49.930 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.895+0000 7f63a6b7a640 10 librados: watch_flush enter 2024-09-16T03:47:49.930 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.895+0000 7f63a6b7a640 10 librados: watch_flush exit 2024-09-16T03:47:49.930 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.895+0000 7f63a6b7a640 20 client.764313.objecter shutdown clearing up homeless session... 2024-09-16T03:47:49.930 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.895+0000 7f63a6b7a640 10 client.764313.objecter successfully canceled tick 2024-09-16T03:47:49.930 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.895+0000 7f63a6b7a640 1 -- 172.21.15.42:0/2092736583 >> v2:172.21.15.154:6812/254845342 conn(0x7f6380037810 msgr2=0x7f6380039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:49.930 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.895+0000 7f63a6b7a640 1 --2- 172.21.15.42:0/2092736583 >> v2:172.21.15.154:6812/254845342 conn(0x7f6380037810 0x7f6380039cd0 secure :-1 s=READY pgs=3461 cs=0 l=1 rev1=1 crypto rx=0x7f63a013d330 tx=0x7f6390009040 comp rx=0 tx=0).stop 2024-09-16T03:47:49.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.895+0000 7f63a6b7a640 10 monclient: shutdown 2024-09-16T03:47:49.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.895+0000 7f63a6b7a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:47:49.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.895+0000 7f63a6b7a640 1 -- 172.21.15.42:0/2092736583 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f63a01cb770 msgr2=0x7f63a013aca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:49.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.895+0000 7f63a5117640 1 -- 172.21.15.42:0/2092736583 reap_dead start 2024-09-16T03:47:49.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.895+0000 7f63a6b7a640 1 --2- 172.21.15.42:0/2092736583 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f63a01cb770 0x7f63a013aca0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f6398006e90 tx=0x7f639800ea60 comp rx=0 tx=0).stop 2024-09-16T03:47:49.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.895+0000 7f63a6b7a640 1 -- 172.21.15.42:0/2092736583 shutdown_connections 2024-09-16T03:47:49.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.895+0000 7f63a6b7a640 1 --2- 172.21.15.42:0/2092736583 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f63a01cb770 0x7f63a013aca0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:49.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.895+0000 7f63a6b7a640 1 -- 172.21.15.42:0/2092736583 >> 172.21.15.42:0/2092736583 conn(0x7f63a00a2420 msgr2=0x7f63a00a4570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:47:49.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.895+0000 7f63a6b7a640 1 -- 172.21.15.42:0/2092736583 shutdown_connections 2024-09-16T03:47:49.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.895+0000 7f63a6b7a640 1 -- 172.21.15.42:0/2092736583 wait complete. 2024-09-16T03:47:49.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:49.895+0000 7f63a6b7a640 1 librados: shutdown 2024-09-16T03:47:49.932 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:47:49.932 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T03:47:49.932 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T03:47:50.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.055+0000 7f6c436ba640 10 monclient: get_monmap_and_config 2024-09-16T03:47:50.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.055+0000 7f6c436ba640 10 monclient: build_initial_monmap 2024-09-16T03:47:50.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.055+0000 7f6c436ba640 10 monclient: monmap: 2024-09-16T03:47:50.062 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:47:50.062 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:50.062 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:47:50.059901+0000 2024-09-16T03:47:50.062 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:47:50.059901+0000 2024-09-16T03:47:50.062 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:47:50.062 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:50.062 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:47:50.062 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:47:50.062 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:47:50.063 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:47:50.063 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:47:50.063 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:50.063 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.059+0000 7f6c436ba640 1 Processor -- start 2024-09-16T03:47:50.063 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.059+0000 7f6c436ba640 1 -- start start 2024-09-16T03:47:50.063 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.059+0000 7f6c436ba640 10 monclient: init 2024-09-16T03:47:50.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.059+0000 7f6c436ba640 10 monclient: _reopen_session rank -1 2024-09-16T03:47:50.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.059+0000 7f6c436ba640 10 monclient: _add_conns ranks=[2,4,3,1,0] 2024-09-16T03:47:50.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.059+0000 7f6c436ba640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f6c3c06db90 0x7f6c3c1340c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:50.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.059+0000 7f6c436ba640 10 monclient(hunting): picked mon.noname-b con 0x7f6c3c06db90 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:47:50.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.059+0000 7f6c436ba640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f6c3c0a37f0 0x7f6c3c0a3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:50.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.059+0000 7f6c436ba640 10 monclient(hunting): picked mon.noname-c con 0x7f6c3c0a37f0 addr v2:172.21.15.154:3300/0 2024-09-16T03:47:50.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.059+0000 7f6c436ba640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f6c3c136e10 0x7f6c3c137220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:50.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.059+0000 7f6c436ba640 10 monclient(hunting): picked mon.noname-a con 0x7f6c3c136e10 addr v2:172.21.15.42:3300/0 2024-09-16T03:47:50.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.059+0000 7f6c436ba640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:50.067 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.059+0000 7f6c436ba640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f6c3c0a4110 con 0x7f6c3c136e10 2024-09-16T03:47:50.067 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.059+0000 7f6c436ba640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:50.067 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.059+0000 7f6c436ba640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f6c3c06df70 con 0x7f6c3c0a37f0 2024-09-16T03:47:50.067 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.059+0000 7f6c436ba640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:50.067 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.059+0000 7f6c436ba640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f6c3c06ece0 con 0x7f6c3c06db90 2024-09-16T03:47:50.067 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.059+0000 7f6c436ba640 10 monclient(hunting): _renew_subs 2024-09-16T03:47:50.067 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.059+0000 7f6c436ba640 10 monclient(hunting): authenticate will time out at 1317.029792s 2024-09-16T03:47:50.067 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.059+0000 7f6c41456640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f6c3c06db90 0x7f6c3c1340c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:50.067 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.059+0000 7f6c41c57640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f6c3c136e10 0x7f6c3c137220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:50.067 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.059+0000 7f6c41456640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f6c3c06db90 0x7f6c3c1340c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:59486/0 (socket says 172.21.15.42:59486) 2024-09-16T03:47:50.067 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.059+0000 7f6c41456640 1 -- 172.21.15.42:0/2801935320 learned_addr learned my addr 172.21.15.42:0/2801935320 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:47:50.067 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.059+0000 7f6c41456640 10 monclient(hunting): get_auth_request con 0x7f6c3c06db90 auth_method 0 2024-09-16T03:47:50.067 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.059+0000 7f6c41456640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:50.068 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.059+0000 7f6c41456640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:50.068 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.059+0000 7f6c41456640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:47:50.068 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.059+0000 7f6c41c57640 10 monclient(hunting): get_auth_request con 0x7f6c3c136e10 auth_method 0 2024-09-16T03:47:50.068 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.059+0000 7f6c41c57640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:50.068 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.059+0000 7f6c41c57640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:50.068 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.059+0000 7f6c41c57640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:47:50.068 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.059+0000 7f6c41c57640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:47:50.068 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.059+0000 7f6c41c57640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:47:50.068 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.059+0000 7f6c41c57640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:47:50.068 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.059+0000 7f6c41456640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:47:50.068 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.059+0000 7f6c41456640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:47:50.068 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.059+0000 7f6c41456640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:47:50.068 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.059+0000 7f6c40c55640 1 --2- 172.21.15.42:0/2801935320 >> v2:172.21.15.154:3300/0 conn(0x7f6c3c0a37f0 0x7f6c3c0a3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:50.068 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.059+0000 7f6c40c55640 10 monclient(hunting): get_auth_request con 0x7f6c3c0a37f0 auth_method 0 2024-09-16T03:47:50.069 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.059+0000 7f6c40c55640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:50.069 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.059+0000 7f6c40c55640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:50.069 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.059+0000 7f6c40c55640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:47:50.069 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.059+0000 7f6c41c57640 10 monclient(hunting): handle_auth_done global_id 764316 payload 274 2024-09-16T03:47:50.069 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.059+0000 7f6c41c57640 1 -- 172.21.15.42:0/2801935320 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f6c3c06db90 msgr2=0x7f6c3c1340c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:47:50.069 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.059+0000 7f6c41c57640 1 --2- 172.21.15.42:0/2801935320 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f6c3c06db90 0x7f6c3c1340c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:50.069 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.059+0000 7f6c41c57640 1 -- 172.21.15.42:0/2801935320 >> v2:172.21.15.154:3300/0 conn(0x7f6c3c0a37f0 msgr2=0x7f6c3c0a3bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:47:50.069 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.059+0000 7f6c41c57640 1 --2- 172.21.15.42:0/2801935320 >> v2:172.21.15.154:3300/0 conn(0x7f6c3c0a37f0 0x7f6c3c0a3bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:50.069 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.059+0000 7f6c41c57640 10 monclient: _finish_hunting 0 2024-09-16T03:47:50.069 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.059+0000 7f6c41c57640 1 monclient: found mon.noname-a 2024-09-16T03:47:50.069 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.059+0000 7f6c41c57640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:47:50.069 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.059+0000 7f6c41c57640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:47:50.069 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.059+0000 7f6c41c57640 1 -- 172.21.15.42:0/2801935320 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c3c09bf80 con 0x7f6c3c136e10 2024-09-16T03:47:50.070 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.059+0000 7f6c41456640 1 --2- 172.21.15.42:0/2801935320 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f6c3c06db90 0x7f6c3c1340c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:47:50.070 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.063+0000 7f6c41c57640 1 --2- 172.21.15.42:0/2801935320 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6c3c136e10 0x7f6c3c137220 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f6c3c06e4f0 tx=0x7f6c3401c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b480eac8256afc27 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:47:50.070 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.063+0000 7f6c33fff640 1 -- 172.21.15.42:0/2801935320 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c340b2070 con 0x7f6c3c136e10 2024-09-16T03:47:50.070 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.063+0000 7f6c33fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:47:50.070 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.063+0000 7f6c33fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:47:50.070 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.063+0000 7f6c33fff640 10 monclient: dump: 2024-09-16T03:47:50.070 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:47:50.070 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:50.070 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:50.070 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:50.071 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:47:50.071 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:50.071 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:47:50.071 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:47:50.071 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:47:50.071 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:50.071 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.063+0000 7f6c33fff640 10 monclient: _finish_auth 0 2024-09-16T03:47:50.071 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.063+0000 7f6c33fff640 10 monclient: _check_auth_tickets 2024-09-16T03:47:50.071 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.063+0000 7f6c33fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:47:50.071 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.063+0000 7f6c436ba640 5 monclient: authenticate success, global_id 764316 2024-09-16T03:47:50.071 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.063+0000 7f6c436ba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:47:50.071 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.063+0000 7f6c33fff640 1 -- 172.21.15.42:0/2801935320 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6c340a5760 con 0x7f6c3c136e10 2024-09-16T03:47:50.072 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.063+0000 7f6c33fff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:47:50.072 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.063+0000 7f6c436ba640 10 monclient: get_monmap_and_config success 2024-09-16T03:47:50.072 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.063+0000 7f6c436ba640 4 set_mon_vals no callback set 2024-09-16T03:47:50.072 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.063+0000 7f6c33fff640 1 -- 172.21.15.42:0/2801935320 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c340a5b30 con 0x7f6c3c136e10 2024-09-16T03:47:50.072 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.063+0000 7f6c33fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:47:50.072 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.063+0000 7f6c33fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:47:50.072 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.063+0000 7f6c33fff640 10 monclient: dump: 2024-09-16T03:47:50.072 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:47:50.072 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:50.072 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:50.072 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:50.072 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:47:50.072 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:50.073 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:47:50.073 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:47:50.073 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:47:50.073 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:50.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.063+0000 7f6c436ba640 10 monclient: shutdown 2024-09-16T03:47:50.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.063+0000 7f6c436ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:47:50.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.063+0000 7f6c436ba640 1 -- 172.21.15.42:0/2801935320 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6c3c136e10 msgr2=0x7f6c3c137220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:50.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.063+0000 7f6c436ba640 1 --2- 172.21.15.42:0/2801935320 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6c3c136e10 0x7f6c3c137220 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f6c3c06e4f0 tx=0x7f6c3401c7f0 comp rx=0 tx=0).stop 2024-09-16T03:47:50.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.063+0000 7f6c436ba640 1 -- 172.21.15.42:0/2801935320 shutdown_connections 2024-09-16T03:47:50.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.063+0000 7f6c436ba640 1 --2- 172.21.15.42:0/2801935320 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6c3c136e10 0x7f6c3c137220 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:50.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.063+0000 7f6c436ba640 1 --2- 172.21.15.42:0/2801935320 >> v2:172.21.15.154:3300/0 conn(0x7f6c3c0a37f0 0x7f6c3c0a3bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:50.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.063+0000 7f6c436ba640 1 --2- 172.21.15.42:0/2801935320 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f6c3c06db90 0x7f6c3c1340c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:50.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.063+0000 7f6c436ba640 1 -- 172.21.15.42:0/2801935320 >> 172.21.15.42:0/2801935320 conn(0x7f6c3c0a9fb0 msgr2=0x7f6c3c0aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:47:50.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.063+0000 7f6c436ba640 1 -- 172.21.15.42:0/2801935320 shutdown_connections 2024-09-16T03:47:50.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.063+0000 7f6c436ba640 1 -- 172.21.15.42:0/2801935320 wait complete. 2024-09-16T03:47:50.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.063+0000 7f6c436ba640 10 monclient: build_initial_monmap 2024-09-16T03:47:50.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.063+0000 7f6c436ba640 10 monclient: monmap: 2024-09-16T03:47:50.082 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:47:50.082 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:47:50.082 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:47:50.082 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:47:50.082 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:47:50.082 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:50.082 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:47:50.082 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:47:50.082 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:47:50.082 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:50.083 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.063+0000 7f6c436ba640 1 librados: starting msgr at 2024-09-16T03:47:50.083 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.063+0000 7f6c436ba640 1 librados: starting objecter 2024-09-16T03:47:50.083 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.063+0000 7f6c436ba640 1 Processor -- start 2024-09-16T03:47:50.083 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.063+0000 7f6c436ba640 1 -- start start 2024-09-16T03:47:50.083 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.063+0000 7f6c436ba640 1 librados: setting wanted keys 2024-09-16T03:47:50.083 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.063+0000 7f6c436ba640 1 librados: calling monclient init 2024-09-16T03:47:50.083 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.063+0000 7f6c436ba640 10 monclient: init 2024-09-16T03:47:50.083 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.063+0000 7f6c436ba640 10 monclient: _reopen_session rank -1 2024-09-16T03:47:50.083 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.063+0000 7f6c436ba640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:47:50.083 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.063+0000 7f6c436ba640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6c3c06db90 0x7f6c3c0a31e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:50.083 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.063+0000 7f6c436ba640 10 monclient(hunting): picked mon.noname-a con 0x7f6c3c06db90 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:47:50.083 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.063+0000 7f6c436ba640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f6c3c0a37f0 0x7f6c3c09c710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:50.083 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.063+0000 7f6c436ba640 10 monclient(hunting): picked mon.noname-c con 0x7f6c3c0a37f0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:47:50.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.063+0000 7f6c436ba640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6c3c136e10 0x7f6c3c09cc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:50.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.063+0000 7f6c436ba640 10 monclient(hunting): picked mon.noname-b con 0x7f6c3c136e10 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:47:50.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.063+0000 7f6c436ba640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:50.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.063+0000 7f6c436ba640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f6c3c140b40 con 0x7f6c3c06db90 2024-09-16T03:47:50.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.063+0000 7f6c436ba640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:50.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.063+0000 7f6c436ba640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f6c3c1409c0 con 0x7f6c3c136e10 2024-09-16T03:47:50.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.063+0000 7f6c436ba640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:50.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.063+0000 7f6c436ba640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f6c3c140cc0 con 0x7f6c3c0a37f0 2024-09-16T03:47:50.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.063+0000 7f6c436ba640 10 monclient(hunting): _renew_subs 2024-09-16T03:47:50.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.063+0000 7f6c436ba640 10 monclient(hunting): authenticate will time out at 1317.034810s 2024-09-16T03:47:50.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.063+0000 7f6c41456640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6c3c06db90 0x7f6c3c0a31e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:50.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.063+0000 7f6c40c55640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f6c3c0a37f0 0x7f6c3c09c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:50.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.063+0000 7f6c41456640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6c3c06db90 0x7f6c3c0a31e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:43412/0 (socket says 172.21.15.42:43412) 2024-09-16T03:47:50.085 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.063+0000 7f6c41456640 1 -- 172.21.15.42:0/1387445218 learned_addr learned my addr 172.21.15.42:0/1387445218 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:47:50.085 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.063+0000 7f6c40c55640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f6c3c0a37f0 0x7f6c3c09c710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:59490/0 (socket says 172.21.15.42:59490) 2024-09-16T03:47:50.085 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.063+0000 7f6c41456640 10 monclient(hunting): get_auth_request con 0x7f6c3c06db90 auth_method 0 2024-09-16T03:47:50.085 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.063+0000 7f6c41456640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:50.085 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.063+0000 7f6c41456640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:50.085 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.063+0000 7f6c41456640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:47:50.085 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.063+0000 7f6c40c55640 10 monclient(hunting): get_auth_request con 0x7f6c3c0a37f0 auth_method 0 2024-09-16T03:47:50.085 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.063+0000 7f6c40c55640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:50.085 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.063+0000 7f6c40c55640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:50.085 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.063+0000 7f6c40c55640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:47:50.085 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.063+0000 7f6c41456640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:47:50.085 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.063+0000 7f6c41456640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:47:50.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.063+0000 7f6c41456640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:47:50.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.063+0000 7f6c40c55640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:47:50.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.063+0000 7f6c40c55640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:47:50.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.063+0000 7f6c40c55640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:47:50.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.063+0000 7f6c41c57640 1 --2- 172.21.15.42:0/1387445218 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6c3c136e10 0x7f6c3c09cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:50.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.063+0000 7f6c41c57640 10 monclient(hunting): get_auth_request con 0x7f6c3c136e10 auth_method 0 2024-09-16T03:47:50.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.063+0000 7f6c41c57640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:50.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.063+0000 7f6c41c57640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:50.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.063+0000 7f6c41c57640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:47:50.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.063+0000 7f6c41456640 10 monclient(hunting): handle_auth_done global_id 764319 payload 824 2024-09-16T03:47:50.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.063+0000 7f6c41456640 1 -- 172.21.15.42:0/1387445218 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f6c3c0a37f0 msgr2=0x7f6c3c09c710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:50.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.063+0000 7f6c41456640 1 --2- 172.21.15.42:0/1387445218 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f6c3c0a37f0 0x7f6c3c09c710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:50.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.063+0000 7f6c41456640 1 -- 172.21.15.42:0/1387445218 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6c3c136e10 msgr2=0x7f6c3c09cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:50.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.063+0000 7f6c41456640 1 --2- 172.21.15.42:0/1387445218 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6c3c136e10 0x7f6c3c09cc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:50.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.063+0000 7f6c41456640 10 monclient: _finish_hunting 0 2024-09-16T03:47:50.087 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.063+0000 7f6c41456640 1 monclient: found mon.noname-a 2024-09-16T03:47:50.087 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.063+0000 7f6c41456640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:47:50.087 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.063+0000 7f6c41456640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:47:50.087 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.063+0000 7f6c41456640 1 -- 172.21.15.42:0/1387445218 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c3c09f6f0 con 0x7f6c3c06db90 2024-09-16T03:47:50.087 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.063+0000 7f6c40c55640 1 --2- 172.21.15.42:0/1387445218 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f6c3c0a37f0 0x7f6c3c09c710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:47:50.087 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.063+0000 7f6c41456640 1 --2- 172.21.15.42:0/1387445218 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6c3c06db90 0x7f6c3c0a31e0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f6c3800e960 tx=0x7f6c3800ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:47:50.087 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.063+0000 7f6c31ffb640 10 client.?.objecter ms_handle_connect 0x7f6c3c06db90 2024-09-16T03:47:50.087 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.063+0000 7f6c31ffb640 10 client.?.objecter resend_mon_ops 2024-09-16T03:47:50.087 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.063+0000 7f6c31ffb640 1 -- 172.21.15.42:0/1387445218 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c38017070 con 0x7f6c3c06db90 2024-09-16T03:47:50.087 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.063+0000 7f6c31ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:47:50.087 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.063+0000 7f6c31ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:47:50.087 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.063+0000 7f6c31ffb640 10 monclient: dump: 2024-09-16T03:47:50.087 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:47:50.088 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:50.088 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:50.088 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:50.088 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:47:50.088 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:50.088 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:47:50.088 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:47:50.088 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:47:50.088 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:50.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.063+0000 7f6c31ffb640 10 monclient: _finish_auth 0 2024-09-16T03:47:50.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.063+0000 7f6c31ffb640 10 monclient: _check_auth_tickets 2024-09-16T03:47:50.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.063+0000 7f6c31ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:47:50.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.063+0000 7f6c31ffb640 1 -- 172.21.15.42:0/1387445218 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6c3800c9a0 con 0x7f6c3c06db90 2024-09-16T03:47:50.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.063+0000 7f6c31ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T03:47:50.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.063+0000 7f6c31ffb640 1 -- 172.21.15.42:0/1387445218 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c3800cd50 con 0x7f6c3c06db90 2024-09-16T03:47:50.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.063+0000 7f6c436ba640 5 monclient: authenticate success, global_id 764319 2024-09-16T03:47:50.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.063+0000 7f6c31ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:47:50.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.063+0000 7f6c31ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:47:50.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.063+0000 7f6c31ffb640 10 monclient: dump: 2024-09-16T03:47:50.089 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:47:50.089 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:50.089 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:50.089 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:50.089 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:47:50.089 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:50.089 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:47:50.089 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:47:50.090 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:47:50.090 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:50.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.063+0000 7f6c32ffd640 4 set_mon_vals no callback set 2024-09-16T03:47:50.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.063+0000 7f6c436ba640 10 monclient: _renew_subs 2024-09-16T03:47:50.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.063+0000 7f6c436ba640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:47:50.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.063+0000 7f6c436ba640 1 -- 172.21.15.42:0/1387445218 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c3c09db80 con 0x7f6c3c06db90 2024-09-16T03:47:50.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.063+0000 7f6c436ba640 10 client.764319.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:47:50.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.063+0000 7f6c436ba640 10 monclient: _renew_subs 2024-09-16T03:47:50.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.063+0000 7f6c436ba640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:47:50.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.063+0000 7f6c436ba640 1 -- 172.21.15.42:0/1387445218 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c3c09f130 con 0x7f6c3c06db90 2024-09-16T03:47:50.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.063+0000 7f6c436ba640 1 librados: init done 2024-09-16T03:47:50.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.067+0000 7f6c436ba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:47:50.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.067+0000 7f6c436ba640 1 --2- 172.21.15.42:0/1387445218 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6c3c09ddc0 0x7f6c3c09d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:50.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.067+0000 7f6c436ba640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:50.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.067+0000 7f6c436ba640 1 -- 172.21.15.42:0/1387445218 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f6c3c0a3bd0 con 0x7f6c3c09ddc0 2024-09-16T03:47:50.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.067+0000 7f6c31ffb640 1 -- 172.21.15.42:0/1387445218 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f6c3801a070 con 0x7f6c3c06db90 2024-09-16T03:47:50.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.067+0000 7f6c31ffb640 1 --2- 172.21.15.42:0/1387445218 >> v2:172.21.15.154:6812/254845342 conn(0x7f6c20037810 0x7f6c20039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:50.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.067+0000 7f6c40c55640 1 --2- 172.21.15.42:0/1387445218 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6c3c09ddc0 0x7f6c3c09d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:50.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.067+0000 7f6c31ffb640 1 -- 172.21.15.42:0/1387445218 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(179..179 src has 1..179) ==== 6881+0+0 (secure 0 0 0) 0x7f6c38049340 con 0x7f6c3c06db90 2024-09-16T03:47:50.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.067+0000 7f6c31ffb640 10 client.764319.objecter ms_dispatch 0x7f6c3c000ba0 osd_map(179..179 src has 1..179) 2024-09-16T03:47:50.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.067+0000 7f6c31ffb640 3 client.764319.objecter handle_osd_map got epochs [179,179] > 0 2024-09-16T03:47:50.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.067+0000 7f6c31ffb640 3 client.764319.objecter handle_osd_map decoding full epoch 179 2024-09-16T03:47:50.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.067+0000 7f6c40c55640 10 monclient: get_auth_request con 0x7f6c3c09ddc0 auth_method 0 2024-09-16T03:47:50.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.067+0000 7f6c40c55640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:50.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.067+0000 7f6c40c55640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:50.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.067+0000 7f6c40c55640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:47:50.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.067+0000 7f6c31ffb640 20 client.764319.objecter dump_active .. 0 homeless 2024-09-16T03:47:50.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.067+0000 7f6c40c55640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:47:50.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.067+0000 7f6c40c55640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:47:50.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.067+0000 7f6c40c55640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:47:50.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.067+0000 7f6c41c57640 1 --2- 172.21.15.42:0/1387445218 >> v2:172.21.15.154:6812/254845342 conn(0x7f6c20037810 0x7f6c20039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:50.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.067+0000 7f6c41c57640 10 monclient: get_auth_request con 0x7f6c20037810 auth_method 0 2024-09-16T03:47:50.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.067+0000 7f6c40c55640 10 monclient(hunting): handle_auth_done global_id 774130 payload 824 2024-09-16T03:47:50.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.067+0000 7f6c40c55640 1 -- 172.21.15.42:0/1387445218 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6c3c0008e0 con 0x7f6c3c09ddc0 2024-09-16T03:47:50.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.067+0000 7f6c31ffb640 10 client.764319.objecter ms_handle_connect 0x7f6c3c09ddc0 2024-09-16T03:47:50.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.067+0000 7f6c31ffb640 10 client.764319.objecter resend_mon_ops 2024-09-16T03:47:50.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.067+0000 7f6c40c55640 1 --2- 172.21.15.42:0/1387445218 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6c3c09ddc0 0x7f6c3c09d190 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f6c3c13db80 tx=0x7f6c2c007f70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:47:50.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.067+0000 7f6c41c57640 1 --2- 172.21.15.42:0/1387445218 >> v2:172.21.15.154:6812/254845342 conn(0x7f6c20037810 0x7f6c20039cd0 secure :-1 s=READY pgs=3462 cs=0 l=1 rev1=1 crypto rx=0x7f6c340af000 tx=0x7f6c340a7070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:47:50.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.067+0000 7f6c31ffb640 10 client.764319.objecter ms_handle_connect 0x7f6c20037810 2024-09-16T03:47:50.093 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.067+0000 7f6c31ffb640 1 -- 172.21.15.42:0/1387445218 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c2c018020 con 0x7f6c3c09ddc0 2024-09-16T03:47:50.093 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.067+0000 7f6c31ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:47:50.093 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.067+0000 7f6c31ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:47:50.093 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.067+0000 7f6c31ffb640 10 monclient: dump: 2024-09-16T03:47:50.093 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:47:50.093 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:50.093 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:50.093 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:50.093 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:47:50.093 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:50.093 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:47:50.093 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:47:50.093 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:47:50.094 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:50.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.067+0000 7f6c31ffb640 1 -- 172.21.15.42:0/1387445218 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f6c2c00ecd0 con 0x7f6c3c09ddc0 2024-09-16T03:47:50.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.067+0000 7f6c31ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:47:50.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.067+0000 7f6c31ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:47:50.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.067+0000 7f6c31ffb640 1 -- 172.21.15.42:0/1387445218 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6c3c09ddc0 msgr2=0x7f6c3c09d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:50.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.067+0000 7f6c31ffb640 1 --2- 172.21.15.42:0/1387445218 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6c3c09ddc0 0x7f6c3c09d190 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f6c3c13db80 tx=0x7f6c2c007f70 comp rx=0 tx=0).stop 2024-09-16T03:47:50.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.067+0000 7f6c31ffb640 1 -- 172.21.15.42:0/1387445218 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6c3c09ddc0 msgr2=0x7f6c3c09d190 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:47:50.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.067+0000 7f6c31ffb640 1 --2- 172.21.15.42:0/1387445218 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6c3c09ddc0 0x7f6c3c09d190 secure :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f6c3c13db80 tx=0x7f6c2c007f70 comp rx=0 tx=0).stop 2024-09-16T03:47:50.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.075+0000 7f6c436ba640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:47:50.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.075+0000 7f6c436ba640 1 --2- 172.21.15.42:0/1387445218 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6c3c138f50 0x7f6c3c139360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:50.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.075+0000 7f6c436ba640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:50.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.075+0000 7f6c436ba640 1 -- 172.21.15.42:0/1387445218 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f6c3c0a2530 con 0x7f6c3c138f50 2024-09-16T03:47:50.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.075+0000 7f6c40c55640 1 --2- 172.21.15.42:0/1387445218 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6c3c138f50 0x7f6c3c139360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:50.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.075+0000 7f6c40c55640 10 monclient: get_auth_request con 0x7f6c3c138f50 auth_method 0 2024-09-16T03:47:50.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.075+0000 7f6c40c55640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:50.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.075+0000 7f6c40c55640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:50.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.075+0000 7f6c40c55640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:47:50.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.075+0000 7f6c40c55640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:47:50.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.075+0000 7f6c40c55640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:47:50.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.075+0000 7f6c40c55640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:47:50.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.075+0000 7f6c40c55640 10 monclient(hunting): handle_auth_done global_id 774133 payload 824 2024-09-16T03:47:50.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.075+0000 7f6c40c55640 1 -- 172.21.15.42:0/1387445218 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f6c3c1d9e40 con 0x7f6c3c138f50 2024-09-16T03:47:50.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.075+0000 7f6c40c55640 1 --2- 172.21.15.42:0/1387445218 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6c3c138f50 0x7f6c3c139360 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f6c3c13db80 tx=0x7f6c2c0046d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:47:50.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.075+0000 7f6c31ffb640 10 client.764319.objecter ms_handle_connect 0x7f6c3c138f50 2024-09-16T03:47:50.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.075+0000 7f6c31ffb640 10 client.764319.objecter resend_mon_ops 2024-09-16T03:47:50.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.075+0000 7f6c31ffb640 1 -- 172.21.15.42:0/1387445218 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c2c018070 con 0x7f6c3c138f50 2024-09-16T03:47:50.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.075+0000 7f6c31ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:47:50.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.075+0000 7f6c31ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:47:50.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.075+0000 7f6c31ffb640 10 monclient: dump: 2024-09-16T03:47:50.096 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:47:50.096 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:50.096 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:50.096 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:50.096 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:47:50.096 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:50.096 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:47:50.096 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:47:50.096 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:47:50.096 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:50.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.075+0000 7f6c31ffb640 1 -- 172.21.15.42:0/1387445218 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4543 (secure 0 0 0) 0x7f6c2c01f0d0 con 0x7f6c3c138f50 2024-09-16T03:47:50.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.075+0000 7f6c31ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:47:50.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.075+0000 7f6c31ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:47:50.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.075+0000 7f6c31ffb640 1 -- 172.21.15.42:0/1387445218 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6c3c138f50 msgr2=0x7f6c3c139360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:50.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.075+0000 7f6c31ffb640 1 --2- 172.21.15.42:0/1387445218 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6c3c138f50 0x7f6c3c139360 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f6c3c13db80 tx=0x7f6c2c0046d0 comp rx=0 tx=0).stop 2024-09-16T03:47:50.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.075+0000 7f6c31ffb640 1 -- 172.21.15.42:0/1387445218 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6c3c138f50 msgr2=0x7f6c3c139360 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:47:50.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.075+0000 7f6c31ffb640 1 --2- 172.21.15.42:0/1387445218 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6c3c138f50 0x7f6c3c139360 secure :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f6c3c13db80 tx=0x7f6c2c0046d0 comp rx=0 tx=0).stop 2024-09-16T03:47:50.097 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:47:50.097 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:47:50.097 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:47:50.097 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T03:47:50.098 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 54, 2024-09-16T03:47:50.098 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 7663, 2024-09-16T03:47:50.099 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:47:50.099 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:47:50.100 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T03:47:50.100 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T03:47:50.100 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:47:50.100 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 3, 2024-09-16T03:47:50.100 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:47:50.100 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:47:50.100 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:47:50.100 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:47:50.100 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:47:50.100 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:47:50.100 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:47:50.100 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:47:50.100 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:47:50.100 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:47:50.101 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:47:50.101 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:47:50.101 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:47:50.101 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:47:50.101 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:47:50.101 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:47:50.101 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:47:50.101 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:47:50.101 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:47:50.101 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:47:50.101 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:47:50.101 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:47:50.101 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:47:50.101 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:47:50.102 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:47:50.102 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:47:50.102 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:47:50.102 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:47:50.102 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:47:50.102 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:50.102 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:47:50.102 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:47:50.102 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:47:50.102 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:47:50.102 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:47:50.102 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:47:50.102 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:47:50.102 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:47:50.103 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:47:50.103 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:47:50.103 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:47:50.103 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:47:50.103 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:47:50.103 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:47:50.103 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:47:50.103 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:47:50.103 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:47:50.103 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:47:50.103 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:47:50.103 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:47:50.103 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:47:50.103 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:47:50.104 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:47:50.104 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:47:50.104 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:47:50.104 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:47:50.104 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:47:50.104 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:47:50.104 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:47:50.104 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:47:50.104 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:50.104 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:47:50.104 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:50.104 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:47:50.104 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:47:50.104 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:47:50.105 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:47:50.105 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:50.105 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:47:50.105 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:47:50.105 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:47:50.105 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:50.105 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:50.105 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:47:50.105 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:47:50.105 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:47:50.105 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:47:50.105 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:47:50.105 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:50.105 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:47:50.106 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:47:50.106 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:47:50.106 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:47:50.106 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:47:50.106 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:50.106 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:50.106 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:47:50.106 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:47:50.106 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:47:50.106 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:47:50.106 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:50.106 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:47:50.106 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:47:50.106 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:47:50.107 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:50.107 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:50.107 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:47:50.107 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:47:50.107 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:47:50.107 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:47:50.107 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:47:50.107 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:50.107 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:47:50.107 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:47:50.107 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:47:50.107 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:47:50.107 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:47:50.107 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:50.108 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:50.108 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:47:50.108 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:47:50.108 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:47:50.108 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:47:50.108 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:50.108 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:47:50.108 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:47:50.108 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:47:50.108 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:50.108 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:50.108 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:47:50.108 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:47:50.108 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:47:50.109 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:47:50.109 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:47:50.109 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:50.109 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:47:50.109 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:47:50.109 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:47:50.109 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:47:50.109 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:47:50.109 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:47:50.109 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:47:50.109 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:50.109 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:47:50.109 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:47:50.109 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:50.109 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:47:50.110 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:47:50.110 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:47:50.110 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:47:50.110 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:47:50.110 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:47:50.110 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:50.110 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:47:50.110 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:47:50.110 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:47:50.110 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:47:50.110 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:47:50.110 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:50.110 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:47:50.110 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:47:50.112 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.079+0000 7f6c436ba640 10 librados: watch_flush enter 2024-09-16T03:47:50.112 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.079+0000 7f6c436ba640 10 librados: watch_flush exit 2024-09-16T03:47:50.112 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.079+0000 7f6c436ba640 20 client.764319.objecter shutdown clearing up homeless session... 2024-09-16T03:47:50.112 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.079+0000 7f6c436ba640 10 client.764319.objecter successfully canceled tick 2024-09-16T03:47:50.112 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.079+0000 7f6c436ba640 1 -- 172.21.15.42:0/1387445218 >> v2:172.21.15.154:6812/254845342 conn(0x7f6c20037810 msgr2=0x7f6c20039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:50.113 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.079+0000 7f6c436ba640 1 --2- 172.21.15.42:0/1387445218 >> v2:172.21.15.154:6812/254845342 conn(0x7f6c20037810 0x7f6c20039cd0 secure :-1 s=READY pgs=3462 cs=0 l=1 rev1=1 crypto rx=0x7f6c340af000 tx=0x7f6c340a7070 comp rx=0 tx=0).stop 2024-09-16T03:47:50.113 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.079+0000 7f6c436ba640 10 monclient: shutdown 2024-09-16T03:47:50.113 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.079+0000 7f6c436ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:47:50.113 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.079+0000 7f6c436ba640 1 -- 172.21.15.42:0/1387445218 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6c3c06db90 msgr2=0x7f6c3c0a31e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:50.113 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.079+0000 7f6c436ba640 1 --2- 172.21.15.42:0/1387445218 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6c3c06db90 0x7f6c3c0a31e0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f6c3800e960 tx=0x7f6c3800ee20 comp rx=0 tx=0).stop 2024-09-16T03:47:50.113 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.079+0000 7f6c41c57640 1 -- 172.21.15.42:0/1387445218 reap_dead start 2024-09-16T03:47:50.113 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.079+0000 7f6c436ba640 1 -- 172.21.15.42:0/1387445218 shutdown_connections 2024-09-16T03:47:50.113 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.079+0000 7f6c436ba640 1 --2- 172.21.15.42:0/1387445218 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6c3c06db90 0x7f6c3c0a31e0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:50.113 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.079+0000 7f6c436ba640 1 -- 172.21.15.42:0/1387445218 >> 172.21.15.42:0/1387445218 conn(0x7f6c3c0a9fb0 msgr2=0x7f6c3c1322e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:47:50.113 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.079+0000 7f6c436ba640 1 -- 172.21.15.42:0/1387445218 shutdown_connections 2024-09-16T03:47:50.113 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.079+0000 7f6c436ba640 1 -- 172.21.15.42:0/1387445218 wait complete. 2024-09-16T03:47:50.113 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.079+0000 7f6c436ba640 1 librados: shutdown 2024-09-16T03:47:50.114 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:47:50.114 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T03:47:50.114 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T03:47:50.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.223+0000 7f73c3fec640 10 monclient: get_monmap_and_config 2024-09-16T03:47:50.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.223+0000 7f73c3fec640 10 monclient: build_initial_monmap 2024-09-16T03:47:50.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.223+0000 7f73c3fec640 10 monclient: monmap: 2024-09-16T03:47:50.229 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:47:50.229 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:50.229 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:47:50.227627+0000 2024-09-16T03:47:50.229 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:47:50.227627+0000 2024-09-16T03:47:50.229 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:47:50.229 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:50.229 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:47:50.229 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:47:50.229 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:47:50.229 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:47:50.229 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:47:50.229 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:50.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.227+0000 7f73c3fec640 1 Processor -- start 2024-09-16T03:47:50.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.227+0000 7f73c3fec640 1 -- start start 2024-09-16T03:47:50.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.227+0000 7f73c3fec640 10 monclient: init 2024-09-16T03:47:50.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.227+0000 7f73c3fec640 10 monclient: _reopen_session rank -1 2024-09-16T03:47:50.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.227+0000 7f73c3fec640 10 monclient: _add_conns ranks=[0,2,1,4,3] 2024-09-16T03:47:50.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.227+0000 7f73c3fec640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f73bc0a1330 addr v1:172.21.15.42:6789/0 2024-09-16T03:47:50.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.227+0000 7f73c3fec640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f73bc06dac0 0x7f73bc13a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:50.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.227+0000 7f73c3fec640 10 monclient(hunting): picked mon.noname-b con 0x7f73bc06dac0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:47:50.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.227+0000 7f73c3fec640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f73bc0a1e20 addr v1:172.21.15.154:6789/0 2024-09-16T03:47:50.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.227+0000 7f73c3fec640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f73bc09c7d0 con 0x7f73bc0a1330 2024-09-16T03:47:50.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.227+0000 7f73c3fec640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f73bc09c090 con 0x7f73bc0a1e20 2024-09-16T03:47:50.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.227+0000 7f73c3fec640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:50.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.227+0000 7f73c3fec640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f73bc09f610 con 0x7f73bc06dac0 2024-09-16T03:47:50.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.227+0000 7f73c3fec640 10 monclient(hunting): _renew_subs 2024-09-16T03:47:50.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.227+0000 7f73c3fec640 10 monclient(hunting): authenticate will time out at 1317.197051s 2024-09-16T03:47:50.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.227+0000 7f73c1587640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f73bc06dac0 0x7f73bc13a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:50.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.227+0000 7f73c1d88640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f73bc0a1330 0x7f73bc0a1710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:51718/0 (socket says 172.21.15.42:51718) 2024-09-16T03:47:50.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.227+0000 7f73c1587640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f73bc06dac0 0x7f73bc13a340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:59496/0 (socket says 172.21.15.42:59496) 2024-09-16T03:47:50.233 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.227+0000 7f73c1d88640 1 -- 172.21.15.42:0/3718434726 learned_addr learned my addr 172.21.15.42:0/3718434726 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:47:50.233 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.227+0000 7f73c1587640 10 monclient(hunting): get_auth_request con 0x7f73bc06dac0 auth_method 0 2024-09-16T03:47:50.233 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.227+0000 7f73c1587640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:50.233 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.227+0000 7f73c1587640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:50.233 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.227+0000 7f73c1587640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:47:50.233 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.227+0000 7f73c1587640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:47:50.233 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.227+0000 7f73c1587640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:47:50.233 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.227+0000 7f73c1587640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:47:50.233 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.227+0000 7f73c0d86640 1 -- 172.21.15.42:0/3718434726 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1895426098 0 0) 0x7f73bc09c7d0 con 0x7f73bc0a1330 2024-09-16T03:47:50.233 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.227+0000 7f73c0d86640 10 monclient(hunting): _negotiate 2024-09-16T03:47:50.233 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.227+0000 7f73c0d86640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:50.233 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.227+0000 7f73c0d86640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:47:50.233 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.227+0000 7f73c0d86640 10 monclient(hunting): my global_id is 764322 2024-09-16T03:47:50.233 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.227+0000 7f73c0d86640 1 -- 172.21.15.42:0/3718434726 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f73a8003530 con 0x7f73bc0a1330 2024-09-16T03:47:50.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.227+0000 7f73c1587640 10 monclient(hunting): handle_auth_done global_id 13850 payload 274 2024-09-16T03:47:50.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.227+0000 7f73c1587640 1 -- 172.21.15.42:0/3718434726 >> v1:172.21.15.154:6789/0 conn(0x7f73bc0a1e20 legacy=0x7f73bc13c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:50.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.227+0000 7f73c1587640 1 -- 172.21.15.42:0/3718434726 >> v1:172.21.15.42:6789/0 conn(0x7f73bc0a1330 legacy=0x7f73bc0a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:50.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.227+0000 7f73c1587640 10 monclient: _finish_hunting 0 2024-09-16T03:47:50.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.227+0000 7f73c1587640 1 monclient: found mon.noname-b 2024-09-16T03:47:50.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.227+0000 7f73c1587640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:47:50.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.227+0000 7f73c1587640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T03:47:50.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.227+0000 7f73c1587640 1 -- 172.21.15.42:0/3718434726 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73bc09f8f0 con 0x7f73bc06dac0 2024-09-16T03:47:50.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.227+0000 7f73c1587640 1 --2- 172.21.15.42:0/3718434726 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f73bc06dac0 0x7f73bc13a340 secure :-1 s=READY pgs=2504 cs=0 l=1 rev1=1 crypto rx=0x7f73ac0097f0 tx=0x7f73ac01c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d22ee69e41952d67 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:47:50.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.231+0000 7f73c0d86640 1 -- 172.21.15.42:0/3718434726 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73ac0aa610 con 0x7f73bc06dac0 2024-09-16T03:47:50.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.231+0000 7f73c0d86640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:47:50.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.231+0000 7f73c0d86640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:47:50.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.231+0000 7f73c0d86640 10 monclient: dump: 2024-09-16T03:47:50.240 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:47:50.240 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:50.240 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:50.240 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:50.240 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:47:50.240 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:50.240 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:47:50.240 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:47:50.240 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:47:50.240 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:50.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.231+0000 7f73c0d86640 10 monclient: _finish_auth 0 2024-09-16T03:47:50.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.231+0000 7f73c0d86640 10 monclient: _check_auth_tickets 2024-09-16T03:47:50.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.231+0000 7f73c0d86640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:47:50.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.231+0000 7f73c0d86640 1 -- 172.21.15.42:0/3718434726 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f73ac0af070 con 0x7f73bc06dac0 2024-09-16T03:47:50.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.231+0000 7f73c3fec640 5 monclient: authenticate success, global_id 13850 2024-09-16T03:47:50.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.231+0000 7f73c3fec640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:47:50.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.231+0000 7f73c0d86640 10 monclient: handle_config config(0 keys) 2024-09-16T03:47:50.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.231+0000 7f73c3fec640 10 monclient: get_monmap_and_config success 2024-09-16T03:47:50.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.231+0000 7f73c3fec640 4 set_mon_vals no callback set 2024-09-16T03:47:50.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.231+0000 7f73c0d86640 1 -- 172.21.15.42:0/3718434726 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73ac0aa9b0 con 0x7f73bc06dac0 2024-09-16T03:47:50.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.231+0000 7f73c0d86640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:47:50.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.231+0000 7f73c0d86640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:47:50.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.231+0000 7f73c0d86640 10 monclient: dump: 2024-09-16T03:47:50.241 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:47:50.242 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:50.242 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:50.242 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:50.242 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:47:50.242 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:50.242 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:47:50.242 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:47:50.242 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:47:50.242 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:50.242 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.231+0000 7f73c3fec640 10 monclient: shutdown 2024-09-16T03:47:50.242 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.231+0000 7f73c3fec640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:47:50.242 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.231+0000 7f73c3fec640 1 -- 172.21.15.42:0/3718434726 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f73bc06dac0 msgr2=0x7f73bc13a340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:50.242 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.231+0000 7f73c3fec640 1 --2- 172.21.15.42:0/3718434726 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f73bc06dac0 0x7f73bc13a340 secure :-1 s=READY pgs=2504 cs=0 l=1 rev1=1 crypto rx=0x7f73ac0097f0 tx=0x7f73ac01c490 comp rx=0 tx=0).stop 2024-09-16T03:47:50.242 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.231+0000 7f73c3fec640 1 -- 172.21.15.42:0/3718434726 shutdown_connections 2024-09-16T03:47:50.243 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.231+0000 7f73c3fec640 1 --2- 172.21.15.42:0/3718434726 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f73bc06dac0 0x7f73bc13a340 unknown :-1 s=CLOSED pgs=2504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:50.243 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.231+0000 7f73c3fec640 1 -- 172.21.15.42:0/3718434726 >> 172.21.15.42:0/3718434726 conn(0x7f73bc0a2420 msgr2=0x7f73bc0a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:47:50.243 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.231+0000 7f73c3fec640 1 -- 172.21.15.42:0/3718434726 shutdown_connections 2024-09-16T03:47:50.243 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.231+0000 7f73c3fec640 1 -- 172.21.15.42:0/3718434726 wait complete. 2024-09-16T03:47:50.243 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.231+0000 7f73c3fec640 10 monclient: build_initial_monmap 2024-09-16T03:47:50.243 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.231+0000 7f73c3fec640 10 monclient: monmap: 2024-09-16T03:47:50.243 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:47:50.243 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:47:50.243 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:47:50.243 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:47:50.243 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:47:50.243 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:50.243 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:47:50.244 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:47:50.244 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:47:50.244 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:50.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.231+0000 7f73c3fec640 1 librados: starting msgr at 2024-09-16T03:47:50.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.231+0000 7f73c3fec640 1 librados: starting objecter 2024-09-16T03:47:50.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.231+0000 7f73c3fec640 1 Processor -- start 2024-09-16T03:47:50.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.231+0000 7f73c3fec640 1 -- start start 2024-09-16T03:47:50.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.231+0000 7f73c3fec640 1 librados: setting wanted keys 2024-09-16T03:47:50.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.231+0000 7f73c3fec640 1 librados: calling monclient init 2024-09-16T03:47:50.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.231+0000 7f73c3fec640 10 monclient: init 2024-09-16T03:47:50.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.231+0000 7f73c3fec640 10 monclient: _reopen_session rank -1 2024-09-16T03:47:50.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.231+0000 7f73c3fec640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T03:47:50.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.231+0000 7f73c3fec640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f73bc0a1330 0x7f73bc134760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:50.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.231+0000 7f73c3fec640 10 monclient(hunting): picked mon.noname-a con 0x7f73bc0a1330 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:47:50.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.231+0000 7f73c3fec640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f73bc0a1e20 0x7f73bc134ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:50.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.231+0000 7f73c3fec640 10 monclient(hunting): picked mon.noname-b con 0x7f73bc0a1e20 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:47:50.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.231+0000 7f73c3fec640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f73bc1351e0 0x7f73bc11bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:50.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.231+0000 7f73c3fec640 10 monclient(hunting): picked mon.noname-c con 0x7f73bc1351e0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:47:50.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.231+0000 7f73c3fec640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:50.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.231+0000 7f73c3fec640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f73bc140950 con 0x7f73bc0a1330 2024-09-16T03:47:50.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.231+0000 7f73c3fec640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:50.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.231+0000 7f73c3fec640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f73bc1407d0 con 0x7f73bc0a1e20 2024-09-16T03:47:50.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.231+0000 7f73c3fec640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:50.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.231+0000 7f73c3fec640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f73bc140ad0 con 0x7f73bc1351e0 2024-09-16T03:47:50.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.231+0000 7f73c3fec640 10 monclient(hunting): _renew_subs 2024-09-16T03:47:50.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.231+0000 7f73c3fec640 10 monclient(hunting): authenticate will time out at 1317.202172s 2024-09-16T03:47:50.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.231+0000 7f73c2589640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f73bc1351e0 0x7f73bc11bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:50.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.231+0000 7f73c1d88640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f73bc0a1330 0x7f73bc134760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:50.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.231+0000 7f73c2589640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f73bc1351e0 0x7f73bc11bf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:59498/0 (socket says 172.21.15.42:59498) 2024-09-16T03:47:50.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.231+0000 7f73c2589640 1 -- 172.21.15.42:0/334121655 learned_addr learned my addr 172.21.15.42:0/334121655 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:47:50.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.231+0000 7f73c1d88640 10 monclient(hunting): get_auth_request con 0x7f73bc0a1330 auth_method 0 2024-09-16T03:47:50.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.231+0000 7f73c1d88640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:50.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.231+0000 7f73c1d88640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:50.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.231+0000 7f73c1d88640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:47:50.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.231+0000 7f73c2589640 10 monclient(hunting): get_auth_request con 0x7f73bc1351e0 auth_method 0 2024-09-16T03:47:50.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.231+0000 7f73c2589640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:50.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.231+0000 7f73c2589640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:50.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.231+0000 7f73c2589640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:47:50.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.231+0000 7f73c1d88640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:47:50.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.231+0000 7f73c1d88640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:47:50.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.231+0000 7f73c1d88640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:47:50.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.231+0000 7f73c2589640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:47:50.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.231+0000 7f73c2589640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:47:50.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.231+0000 7f73c2589640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:47:50.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.231+0000 7f73c1587640 1 --2- 172.21.15.42:0/334121655 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f73bc0a1e20 0x7f73bc134ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:50.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.231+0000 7f73c1587640 10 monclient(hunting): get_auth_request con 0x7f73bc0a1e20 auth_method 0 2024-09-16T03:47:50.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.231+0000 7f73c1587640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:50.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.231+0000 7f73c1587640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:50.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.231+0000 7f73c1587640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:47:50.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.231+0000 7f73c1d88640 10 monclient(hunting): handle_auth_done global_id 764325 payload 824 2024-09-16T03:47:50.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.231+0000 7f73c1d88640 1 -- 172.21.15.42:0/334121655 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f73bc1351e0 msgr2=0x7f73bc11bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:50.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.231+0000 7f73c1d88640 1 --2- 172.21.15.42:0/334121655 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f73bc1351e0 0x7f73bc11bf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:50.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.231+0000 7f73c1d88640 1 -- 172.21.15.42:0/334121655 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f73bc0a1e20 msgr2=0x7f73bc134ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:50.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.231+0000 7f73c1d88640 1 --2- 172.21.15.42:0/334121655 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f73bc0a1e20 0x7f73bc134ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:50.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.231+0000 7f73c1d88640 10 monclient: _finish_hunting 0 2024-09-16T03:47:50.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.231+0000 7f73c1d88640 1 monclient: found mon.noname-a 2024-09-16T03:47:50.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.231+0000 7f73c1d88640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:47:50.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.231+0000 7f73c1d88640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:47:50.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.231+0000 7f73c1d88640 1 -- 172.21.15.42:0/334121655 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73bc11e350 con 0x7f73bc0a1330 2024-09-16T03:47:50.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.231+0000 7f73c1d88640 1 --2- 172.21.15.42:0/334121655 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f73bc0a1330 0x7f73bc134760 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f73b800c400 tx=0x7f73b800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:47:50.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.231+0000 7f73b27fc640 10 client.?.objecter ms_handle_connect 0x7f73bc0a1330 2024-09-16T03:47:50.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.231+0000 7f73b27fc640 10 client.?.objecter resend_mon_ops 2024-09-16T03:47:50.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.231+0000 7f73b27fc640 1 -- 172.21.15.42:0/334121655 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73b8017070 con 0x7f73bc0a1330 2024-09-16T03:47:50.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.231+0000 7f73b27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:47:50.249 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.231+0000 7f73b27fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:47:50.249 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.231+0000 7f73b27fc640 10 monclient: dump: 2024-09-16T03:47:50.249 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:47:50.249 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:50.249 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:50.249 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:50.249 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:47:50.249 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:50.249 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:47:50.249 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:47:50.249 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:47:50.249 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:50.249 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.231+0000 7f73b27fc640 10 monclient: _finish_auth 0 2024-09-16T03:47:50.249 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.231+0000 7f73b27fc640 10 monclient: _check_auth_tickets 2024-09-16T03:47:50.250 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.231+0000 7f73b27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:47:50.250 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.231+0000 7f73c3fec640 5 monclient: authenticate success, global_id 764325 2024-09-16T03:47:50.250 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.231+0000 7f73b27fc640 1 -- 172.21.15.42:0/334121655 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f73b8005cf0 con 0x7f73bc0a1330 2024-09-16T03:47:50.250 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.231+0000 7f73b27fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:47:50.250 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.231+0000 7f73b27fc640 1 -- 172.21.15.42:0/334121655 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73b80072d0 con 0x7f73bc0a1330 2024-09-16T03:47:50.250 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.231+0000 7f73b27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:47:50.250 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.231+0000 7f73b27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:47:50.250 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.231+0000 7f73b3fff640 4 set_mon_vals no callback set 2024-09-16T03:47:50.250 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.231+0000 7f73b27fc640 10 monclient: dump: 2024-09-16T03:47:50.250 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:47:50.250 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:50.250 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:50.250 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:50.250 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:47:50.251 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:50.251 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:47:50.251 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:47:50.251 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:47:50.251 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:50.251 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.231+0000 7f73c3fec640 10 monclient: _renew_subs 2024-09-16T03:47:50.251 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.231+0000 7f73c3fec640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:47:50.251 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.231+0000 7f73c3fec640 1 -- 172.21.15.42:0/334121655 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73bc11e630 con 0x7f73bc0a1330 2024-09-16T03:47:50.251 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.231+0000 7f73c3fec640 10 client.764325.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:47:50.251 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.231+0000 7f73c3fec640 10 monclient: _renew_subs 2024-09-16T03:47:50.251 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.231+0000 7f73c3fec640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:47:50.251 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.231+0000 7f73c3fec640 1 -- 172.21.15.42:0/334121655 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73bc06e260 con 0x7f73bc0a1330 2024-09-16T03:47:50.251 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.231+0000 7f73c3fec640 1 librados: init done 2024-09-16T03:47:50.251 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.235+0000 7f73c3fec640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:47:50.251 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.235+0000 7f73c3fec640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:47:50.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.235+0000 7f73c3fec640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:47:50.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.235+0000 7f73c3fec640 1 -- 172.21.15.42:0/334121655 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f73bc11d4c0 con 0x7f73bc0a1330 2024-09-16T03:47:50.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.235+0000 7f73b27fc640 1 -- 172.21.15.42:0/334121655 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f73b801a070 con 0x7f73bc0a1330 2024-09-16T03:47:50.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.235+0000 7f73b27fc640 1 --2- 172.21.15.42:0/334121655 >> v2:172.21.15.154:6812/254845342 conn(0x7f73a0037810 0x7f73a0039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:50.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.235+0000 7f73b27fc640 1 -- 172.21.15.42:0/334121655 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(179..179 src has 1..179) ==== 6881+0+0 (secure 0 0 0) 0x7f73b8048740 con 0x7f73bc0a1330 2024-09-16T03:47:50.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.235+0000 7f73b27fc640 10 client.764325.objecter ms_dispatch 0x7f73bc000ba0 osd_map(179..179 src has 1..179) 2024-09-16T03:47:50.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.235+0000 7f73b27fc640 3 client.764325.objecter handle_osd_map got epochs [179,179] > 0 2024-09-16T03:47:50.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.235+0000 7f73b27fc640 3 client.764325.objecter handle_osd_map decoding full epoch 179 2024-09-16T03:47:50.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.235+0000 7f73b27fc640 20 client.764325.objecter dump_active .. 0 homeless 2024-09-16T03:47:50.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.235+0000 7f73c1587640 1 --2- 172.21.15.42:0/334121655 >> v2:172.21.15.154:6812/254845342 conn(0x7f73a0037810 0x7f73a0039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:50.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.235+0000 7f73c1587640 10 monclient: get_auth_request con 0x7f73a0037810 auth_method 0 2024-09-16T03:47:50.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.235+0000 7f73c1587640 1 --2- 172.21.15.42:0/334121655 >> v2:172.21.15.154:6812/254845342 conn(0x7f73a0037810 0x7f73a0039cd0 secure :-1 s=READY pgs=3463 cs=0 l=1 rev1=1 crypto rx=0x7f73ac0ae000 tx=0x7f73ac0a7000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:47:50.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.235+0000 7f73b27fc640 10 client.764325.objecter ms_handle_connect 0x7f73a0037810 2024-09-16T03:47:50.253 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.239+0000 7f73b27fc640 1 -- 172.21.15.42:0/334121655 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f73b80076b0 con 0x7f73bc0a1330 2024-09-16T03:47:50.253 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.239+0000 7f73b27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:47:50.253 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.239+0000 7f73b27fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:47:50.481 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.479+0000 7f73c3fec640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T03:47:50.482 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.479+0000 7f73c3fec640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T03:47:50.482 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.479+0000 7f73c3fec640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:47:50.482 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.479+0000 7f73c3fec640 1 -- 172.21.15.42:0/334121655 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f73bc073a30 con 0x7f73bc0a1330 2024-09-16T03:47:50.483 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.479+0000 7f73b27fc640 1 -- 172.21.15.42:0/334121655 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f73b8007470 con 0x7f73bc0a1330 2024-09-16T03:47:50.483 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.479+0000 7f73b27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T03:47:50.483 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.479+0000 7f73b27fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:47:50.484 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.479+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:47:50.484 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.479+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=30,config=1,health=12,kv=9,logm=48} crc {auth=2717393001,config=1228700967,health=940943734,kv=3276479304,logm=1152177066}) 2024-09-16T03:47:50.484 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.479+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=28,config=2,health=12,kv=9,logm=49} crc {auth=2349996054,config=692619927,health=940943734,kv=2094645990,logm=2874457272}) 2024-09-16T03:47:50.484 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.479+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:47:50.484 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.479+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=30,config=1,health=12,kv=9,logm=48} crc {auth=2717393001,config=1228700967,health=940943734,kv=3276479304,logm=1152177066}) 2024-09-16T03:47:50.485 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.479+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=30,config=2,health=11,kv=10,logm=47} crc {auth=2065014084,config=692619927,health=2630333188,kv=3446188498,logm=761121662}) 2024-09-16T03:47:50.486 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.483+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:47:50.486 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.483+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=126608431}) 2024-09-16T03:47:50.486 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.483+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4170863226}) 2024-09-16T03:47:50.486 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.483+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:47:50.486 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.483+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=126608431}) 2024-09-16T03:47:50.486 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.483+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2149920930}) 2024-09-16T03:47:50.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.483+0000 7f73c3fec640 10 librados: watch_flush enter 2024-09-16T03:47:50.488 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.483+0000 7f73c3fec640 10 librados: watch_flush exit 2024-09-16T03:47:50.488 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.483+0000 7f73c3fec640 20 client.764325.objecter shutdown clearing up homeless session... 2024-09-16T03:47:50.488 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.483+0000 7f73c3fec640 10 client.764325.objecter successfully canceled tick 2024-09-16T03:47:50.488 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.483+0000 7f73c3fec640 1 -- 172.21.15.42:0/334121655 >> v2:172.21.15.154:6812/254845342 conn(0x7f73a0037810 msgr2=0x7f73a0039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:50.488 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.483+0000 7f73c3fec640 1 --2- 172.21.15.42:0/334121655 >> v2:172.21.15.154:6812/254845342 conn(0x7f73a0037810 0x7f73a0039cd0 secure :-1 s=READY pgs=3463 cs=0 l=1 rev1=1 crypto rx=0x7f73ac0ae000 tx=0x7f73ac0a7000 comp rx=0 tx=0).stop 2024-09-16T03:47:50.488 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.483+0000 7f73c3fec640 10 monclient: shutdown 2024-09-16T03:47:50.488 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.483+0000 7f73c3fec640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:47:50.488 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.483+0000 7f73c3fec640 1 -- 172.21.15.42:0/334121655 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f73bc0a1330 msgr2=0x7f73bc134760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:50.488 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.483+0000 7f73c3fec640 1 --2- 172.21.15.42:0/334121655 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f73bc0a1330 0x7f73bc134760 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f73b800c400 tx=0x7f73b800c8c0 comp rx=0 tx=0).stop 2024-09-16T03:47:50.489 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.483+0000 7f73c3fec640 1 -- 172.21.15.42:0/334121655 shutdown_connections 2024-09-16T03:47:50.489 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.483+0000 7f73c3fec640 1 --2- 172.21.15.42:0/334121655 >> v2:172.21.15.154:6812/254845342 conn(0x7f73a0037810 0x7f73a0039cd0 unknown :-1 s=CLOSED pgs=3463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:50.489 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.483+0000 7f73c3fec640 1 --2- 172.21.15.42:0/334121655 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f73bc1351e0 0x7f73bc11bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:50.489 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.483+0000 7f73c3fec640 1 --2- 172.21.15.42:0/334121655 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f73bc0a1e20 0x7f73bc134ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:50.489 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.483+0000 7f73c3fec640 1 --2- 172.21.15.42:0/334121655 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f73bc0a1330 0x7f73bc134760 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:50.489 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.483+0000 7f73c3fec640 1 -- 172.21.15.42:0/334121655 >> 172.21.15.42:0/334121655 conn(0x7f73bc0a2420 msgr2=0x7f73bc0a4dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:47:50.489 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.483+0000 7f73c3fec640 1 -- 172.21.15.42:0/334121655 shutdown_connections 2024-09-16T03:47:50.489 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.483+0000 7f73c3fec640 1 -- 172.21.15.42:0/334121655 wait complete. 2024-09-16T03:47:50.489 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:50.483+0000 7f73c3fec640 1 librados: shutdown 2024-09-16T03:47:50.491 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.483+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:47:50.491 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.483+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3928269126}) 2024-09-16T03:47:50.491 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.483+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2529321307}) 2024-09-16T03:47:50.491 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.483+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:47:50.492 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.483+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3928269126}) 2024-09-16T03:47:50.492 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.483+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=244315177}) 2024-09-16T03:47:50.492 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.483+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:47:50.492 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.483+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2459439843}) 2024-09-16T03:47:50.492 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.483+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1589703353}) 2024-09-16T03:47:50.492 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.483+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:47:50.492 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.483+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2459439843}) 2024-09-16T03:47:50.492 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.483+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1461267251}) 2024-09-16T03:47:50.492 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.487+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:47:50.492 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.487+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2342486706}) 2024-09-16T03:47:50.492 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.487+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1251095267}) 2024-09-16T03:47:50.492 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.487+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:47:50.493 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.487+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2342486706}) 2024-09-16T03:47:50.493 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.487+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1432361246}) 2024-09-16T03:47:50.493 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.487+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:47:50.493 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.487+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=512106439}) 2024-09-16T03:47:50.493 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.487+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1253579081}) 2024-09-16T03:47:50.493 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.487+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:47:50.493 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.487+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=512106439}) 2024-09-16T03:47:50.493 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.487+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2831020930}) 2024-09-16T03:47:50.494 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.487+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:47:50.494 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.487+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=751268380}) 2024-09-16T03:47:50.494 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.487+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2678598329}) 2024-09-16T03:47:50.495 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.487+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:47:50.495 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.487+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=751268380}) 2024-09-16T03:47:50.495 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.487+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1507881594}) 2024-09-16T03:47:50.495 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.491+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:47:50.495 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.491+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=297133281}) 2024-09-16T03:47:50.495 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.491+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2448185752}) 2024-09-16T03:47:50.495 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.491+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:47:50.495 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.491+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=297133281}) 2024-09-16T03:47:50.495 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.491+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3294388732}) 2024-09-16T03:47:50.496 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.491+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:47:50.496 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.491+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2000203697}) 2024-09-16T03:47:50.497 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.491+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4233136382}) 2024-09-16T03:47:50.497 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.491+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:47:50.497 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.491+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2000203697}) 2024-09-16T03:47:50.497 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.491+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1326970766}) 2024-09-16T03:47:50.497 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.491+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:47:50.497 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.491+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=518256194}) 2024-09-16T03:47:50.497 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.491+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3158513174}) 2024-09-16T03:47:50.497 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.491+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:47:50.497 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.491+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=518256194}) 2024-09-16T03:47:50.497 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.491+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=92069967}) 2024-09-16T03:47:50.498 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.495+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:47:50.498 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.495+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2740937285}) 2024-09-16T03:47:50.498 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.495+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2266672826}) 2024-09-16T03:47:50.498 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.495+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:47:50.498 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.495+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2740937285}) 2024-09-16T03:47:50.499 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.495+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=634018728}) 2024-09-16T03:47:50.499 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.495+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:47:50.499 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.495+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1826259891}) 2024-09-16T03:47:50.499 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.495+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1986064088}) 2024-09-16T03:47:50.500 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.495+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:47:50.500 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.495+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1826259891}) 2024-09-16T03:47:50.500 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.495+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3096218489}) 2024-09-16T03:47:50.501 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.499+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:47:50.501 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.499+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2653499229}) 2024-09-16T03:47:50.501 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.499+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2198357110}) 2024-09-16T03:47:50.501 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.499+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:47:50.501 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.499+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2653499229}) 2024-09-16T03:47:50.501 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.499+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1221639530}) 2024-09-16T03:47:50.502 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.499+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:47:50.502 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.499+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1505304371}) 2024-09-16T03:47:50.502 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.499+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2211164985}) 2024-09-16T03:47:50.502 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.499+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:47:50.503 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.499+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1505304371}) 2024-09-16T03:47:50.503 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.499+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=634480417}) 2024-09-16T03:47:50.504 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T03:47:50.504 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.499+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:47:50.504 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.499+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2460441754}) 2024-09-16T03:47:50.504 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.499+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1379830002}) 2024-09-16T03:47:50.504 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.499+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:47:50.504 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.499+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2460441754}) 2024-09-16T03:47:50.505 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.499+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=54987367}) 2024-09-16T03:47:50.509 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.503+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:47:50.509 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.503+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=525885139}) 2024-09-16T03:47:50.509 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.503+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2770232642}) 2024-09-16T03:47:50.509 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.503+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:47:50.509 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.503+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=525885139}) 2024-09-16T03:47:50.509 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.503+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3707204295}) 2024-09-16T03:47:50.509 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.503+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:47:50.509 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.503+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2367478978}) 2024-09-16T03:47:50.509 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.503+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4213766467}) 2024-09-16T03:47:50.510 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.503+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:47:50.510 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.503+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2367478978}) 2024-09-16T03:47:50.510 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.503+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=110472980}) 2024-09-16T03:47:50.510 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.503+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:47:50.510 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.503+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1735013003}) 2024-09-16T03:47:50.510 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.503+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=200307089}) 2024-09-16T03:47:50.510 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.503+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:47:50.510 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.503+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1735013003}) 2024-09-16T03:47:50.510 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.503+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2955890777}) 2024-09-16T03:47:50.511 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.507+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:47:50.511 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.507+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2981290035}) 2024-09-16T03:47:50.511 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.507+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1793872869}) 2024-09-16T03:47:50.511 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.507+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:47:50.511 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.507+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2981290035}) 2024-09-16T03:47:50.511 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.507+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1854805105}) 2024-09-16T03:47:50.512 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.507+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:47:50.512 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.507+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2871638391}) 2024-09-16T03:47:50.512 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.507+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2113898328}) 2024-09-16T03:47:50.512 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.507+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:47:50.512 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.507+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2871638391}) 2024-09-16T03:47:50.512 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.507+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3206668440}) 2024-09-16T03:47:50.514 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.511+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:47:50.514 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.511+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2667274178}) 2024-09-16T03:47:50.514 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.511+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1246722017}) 2024-09-16T03:47:50.514 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.511+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:47:50.514 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.511+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2667274178}) 2024-09-16T03:47:50.514 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.511+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1790617856}) 2024-09-16T03:47:50.515 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.511+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:47:50.515 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.511+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2505408248}) 2024-09-16T03:47:50.516 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.511+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=854847729}) 2024-09-16T03:47:50.516 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.511+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:47:50.516 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.511+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2505408248}) 2024-09-16T03:47:50.516 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.511+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3032871705}) 2024-09-16T03:47:50.517 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.515+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:47:50.517 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.515+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2374366634}) 2024-09-16T03:47:50.517 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.515+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=590103997}) 2024-09-16T03:47:50.517 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.515+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:47:50.517 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.515+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2374366634}) 2024-09-16T03:47:50.518 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.515+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1010051978}) 2024-09-16T03:47:50.519 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.515+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:47:50.519 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.515+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=776496852}) 2024-09-16T03:47:50.519 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.515+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2507469726}) 2024-09-16T03:47:50.519 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.515+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:47:50.519 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.515+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=776496852}) 2024-09-16T03:47:50.519 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.515+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1699016909}) 2024-09-16T03:47:50.521 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.515+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:47:50.521 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.515+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2057632141}) 2024-09-16T03:47:50.521 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.519+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1244838736}) 2024-09-16T03:47:50.521 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.519+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:47:50.521 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.519+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2057632141}) 2024-09-16T03:47:50.521 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.519+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2301958053}) 2024-09-16T03:47:50.522 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.519+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:47:50.522 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.519+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4228259685}) 2024-09-16T03:47:50.522 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.519+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=157983059}) 2024-09-16T03:47:50.523 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.519+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:47:50.523 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.519+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4228259685}) 2024-09-16T03:47:50.523 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.519+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1256151400}) 2024-09-16T03:47:50.524 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.519+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:47:50.524 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.519+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3397292836}) 2024-09-16T03:47:50.524 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.519+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3380550296}) 2024-09-16T03:47:50.524 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.519+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:47:50.524 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.519+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3397292836}) 2024-09-16T03:47:50.524 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.519+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2930232445}) 2024-09-16T03:47:50.526 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.523+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:47:50.526 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.523+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1549187528}) 2024-09-16T03:47:50.526 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.523+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2831366760}) 2024-09-16T03:47:50.526 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.523+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:47:50.526 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.523+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1549187528}) 2024-09-16T03:47:50.526 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.523+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=824577422}) 2024-09-16T03:47:50.527 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.523+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:47:50.527 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.523+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2571093622}) 2024-09-16T03:47:50.528 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.523+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=184106437}) 2024-09-16T03:47:50.528 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.523+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:47:50.528 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.523+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2571093622}) 2024-09-16T03:47:50.528 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.523+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=578728341}) 2024-09-16T03:47:50.529 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.527+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:47:50.529 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.527+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=617559476}) 2024-09-16T03:47:50.529 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.527+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=506865462}) 2024-09-16T03:47:50.529 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.527+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:47:50.529 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.527+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=617559476}) 2024-09-16T03:47:50.530 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.527+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1875052831}) 2024-09-16T03:47:50.532 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.527+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:47:50.532 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.527+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=13,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=11,monmap=2,osd_metadata=5,osd_pg_creating=1,osd_snap=26,osdmap=22} crc {logm=118773712,mdsmap=4088524775,mgr=1669371261,mgr_command_descs=2107673600,mgr_metadata=1367272202,mgrstat=3749492927,mon_config_key=85057112,monmap=3140514009,osd_metadata=951161650,osd_pg_creating=3921117255,osd_snap=4002033399,osdmap=1349119484}) 2024-09-16T03:47:50.532 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.527+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=12,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=26,osdmap=19} crc {logm=2148270880,mdsmap=4088524775,mgr=460109776,mgr_command_descs=2107673600,mgr_metadata=1367272202,mgrstat=2287445166,mon_config_key=85057112,monmap=3937513981,osd_metadata=4182959540,osd_pg_creating=3921117255,osd_snap=292228967,osdmap=3100954064}) 2024-09-16T03:47:50.532 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.527+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:47:50.532 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.527+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=13,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=11,monmap=2,osd_metadata=5,osd_pg_creating=1,osd_snap=26,osdmap=22} crc {logm=118773712,mdsmap=4088524775,mgr=1669371261,mgr_command_descs=2107673600,mgr_metadata=1367272202,mgrstat=3749492927,mon_config_key=85057112,monmap=3140514009,osd_metadata=951161650,osd_pg_creating=3921117255,osd_snap=4002033399,osdmap=1349119484}) 2024-09-16T03:47:50.532 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.527+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=13,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=23,osdmap=21} crc {logm=118773712,mdsmap=4088524775,mgr=460109776,mgr_command_descs=2107673600,mgr_metadata=1367272202,mgrstat=4091924695,mon_config_key=85057112,monmap=3937513981,osd_metadata=4182959540,osd_pg_creating=3921117255,osd_snap=4079165823,osdmap=3200391902}) 2024-09-16T03:47:50.534 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.531+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:47:50.535 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.531+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=4214627738}) 2024-09-16T03:47:50.535 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.531+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1825922883}) 2024-09-16T03:47:50.535 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.531+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:47:50.535 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.531+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=4214627738}) 2024-09-16T03:47:50.535 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.531+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3385492105}) 2024-09-16T03:47:50.538 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.535+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:47:50.538 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.535+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=640233293}) 2024-09-16T03:47:50.538 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.535+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2207950957}) 2024-09-16T03:47:50.538 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.535+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:47:50.538 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.535+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=640233293}) 2024-09-16T03:47:50.538 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.535+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1375807828}) 2024-09-16T03:47:50.555 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.551+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:47:50.555 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.551+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=51} crc {osdmap=2702086259}) 2024-09-16T03:47:50.556 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.551+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=51} crc {osdmap=1428018814}) 2024-09-16T03:47:50.556 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.551+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:47:50.556 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.551+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=51} crc {osdmap=2702086259}) 2024-09-16T03:47:50.556 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:47:50.551+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=47} crc {osdmap=50681041}) 2024-09-16T03:47:51.504 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T03:47:51.505 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:47:51.638 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.635+0000 7f218270a640 10 monclient: get_monmap_and_config 2024-09-16T03:47:51.638 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.635+0000 7f218270a640 10 monclient: build_initial_monmap 2024-09-16T03:47:51.639 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.635+0000 7f218270a640 10 monclient: monmap: 2024-09-16T03:47:51.639 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:47:51.639 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:51.639 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:47:51.637552+0000 2024-09-16T03:47:51.639 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:47:51.637552+0000 2024-09-16T03:47:51.639 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:47:51.639 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:51.639 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:47:51.639 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:47:51.639 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:47:51.639 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:47:51.639 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:47:51.639 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:51.642 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.635+0000 7f218270a640 1 Processor -- start 2024-09-16T03:47:51.642 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.635+0000 7f218270a640 1 -- start start 2024-09-16T03:47:51.642 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.635+0000 7f218270a640 10 monclient: init 2024-09-16T03:47:51.642 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.635+0000 7f218270a640 10 monclient: _reopen_session rank -1 2024-09-16T03:47:51.642 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.635+0000 7f218270a640 10 monclient: _add_conns ranks=[2,1,3,0,4] 2024-09-16T03:47:51.642 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.635+0000 7f218270a640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f217c0a1330 0x7f217c0a1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:51.642 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.635+0000 7f218270a640 10 monclient(hunting): picked mon.noname-b con 0x7f217c0a1330 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:47:51.642 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.635+0000 7f218270a640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f217c06dac0 addr v1:172.21.15.154:6789/0 2024-09-16T03:47:51.642 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.635+0000 7f218270a640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f217c0a1c50 0x7f217c135810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:51.642 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.635+0000 7f218270a640 10 monclient(hunting): picked mon.noname-a con 0x7f217c0a1c50 addr v2:172.21.15.42:3300/0 2024-09-16T03:47:51.642 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.635+0000 7f218270a640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f217c09c090 con 0x7f217c06dac0 2024-09-16T03:47:51.643 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.635+0000 7f218270a640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:51.643 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.635+0000 7f218270a640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f217c09c890 con 0x7f217c0a1c50 2024-09-16T03:47:51.643 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.635+0000 7f218270a640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:51.643 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.635+0000 7f218270a640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f217c09f610 con 0x7f217c0a1330 2024-09-16T03:47:51.643 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.635+0000 7f218270a640 10 monclient(hunting): _renew_subs 2024-09-16T03:47:51.643 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.635+0000 7f218270a640 10 monclient(hunting): authenticate will time out at 1318.607077s 2024-09-16T03:47:51.643 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.635+0000 7f217bfff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f217c0a1330 0x7f217c0a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:51.643 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.635+0000 7f2180ca7640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f217c0a1c50 0x7f217c135810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:51.643 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.635+0000 7f2180ca7640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f217c0a1c50 0x7f217c135810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:43432/0 (socket says 172.21.15.42:43432) 2024-09-16T03:47:51.643 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.635+0000 7f2180ca7640 1 -- 172.21.15.42:0/452993245 learned_addr learned my addr 172.21.15.42:0/452993245 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:47:51.643 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.635+0000 7f217bfff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f217c0a1330 0x7f217c0a1710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:59504/0 (socket says 172.21.15.42:59504) 2024-09-16T03:47:51.643 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.635+0000 7f2180ca7640 10 monclient(hunting): get_auth_request con 0x7f217c0a1c50 auth_method 0 2024-09-16T03:47:51.643 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.635+0000 7f2180ca7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:51.644 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.635+0000 7f2180ca7640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:51.644 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.635+0000 7f2180ca7640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:47:51.644 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.635+0000 7f217bfff640 10 monclient(hunting): get_auth_request con 0x7f217c0a1330 auth_method 0 2024-09-16T03:47:51.644 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.635+0000 7f217bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:51.644 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.635+0000 7f217bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:51.644 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.635+0000 7f217bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:47:51.644 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.635+0000 7f217bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:47:51.644 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.635+0000 7f217bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:47:51.644 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.635+0000 7f217bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:47:51.644 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.635+0000 7f2180ca7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:47:51.644 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.635+0000 7f2180ca7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:47:51.644 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.635+0000 7f2180ca7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:47:51.644 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.635+0000 7f217bfff640 10 monclient(hunting): handle_auth_done global_id 13856 payload 274 2024-09-16T03:47:51.645 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.635+0000 7f217bfff640 1 -- 172.21.15.42:0/452993245 >> v2:172.21.15.42:3300/0 conn(0x7f217c0a1c50 msgr2=0x7f217c135810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:47:51.645 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.635+0000 7f217bfff640 1 --2- 172.21.15.42:0/452993245 >> v2:172.21.15.42:3300/0 conn(0x7f217c0a1c50 0x7f217c135810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:51.645 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.635+0000 7f217bfff640 1 -- 172.21.15.42:0/452993245 >> v1:172.21.15.154:6789/0 conn(0x7f217c06dac0 legacy=0x7f217c135100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:51.645 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.635+0000 7f217bfff640 10 monclient: _finish_hunting 0 2024-09-16T03:47:51.645 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.635+0000 7f217bfff640 1 monclient: found mon.noname-b 2024-09-16T03:47:51.645 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.635+0000 7f217bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:47:51.645 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.635+0000 7f217bfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T03:47:51.645 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.635+0000 7f217bfff640 1 -- 172.21.15.42:0/452993245 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f217c09f8f0 con 0x7f217c0a1330 2024-09-16T03:47:51.645 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.635+0000 7f2180ca7640 1 --2- 172.21.15.42:0/452993245 >> v2:172.21.15.42:3300/0 conn(0x7f217c0a1c50 0x7f217c135810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:47:51.652 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.639+0000 7f217bfff640 1 --2- 172.21.15.42:0/452993245 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f217c0a1330 0x7f217c0a1710 secure :-1 s=READY pgs=2507 cs=0 l=1 rev1=1 crypto rx=0x7f21640097c0 tx=0x7f216401c3c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fa09d5fbcf5ba175 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:47:51.652 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.639+0000 7f217affd640 1 -- 172.21.15.42:0/452993245 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21640b1070 con 0x7f217c0a1330 2024-09-16T03:47:51.652 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.639+0000 7f217affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:47:51.652 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.639+0000 7f217affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:47:51.652 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.639+0000 7f217affd640 10 monclient: dump: 2024-09-16T03:47:51.652 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:47:51.652 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:51.652 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:51.652 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:51.652 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:47:51.652 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:51.653 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:47:51.653 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:47:51.653 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:47:51.653 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:51.653 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.639+0000 7f217affd640 10 monclient: _finish_auth 0 2024-09-16T03:47:51.653 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.639+0000 7f217affd640 10 monclient: _check_auth_tickets 2024-09-16T03:47:51.653 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.639+0000 7f217affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:47:51.653 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.639+0000 7f217affd640 1 -- 172.21.15.42:0/452993245 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f21640a8050 con 0x7f217c0a1330 2024-09-16T03:47:51.653 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.639+0000 7f218270a640 5 monclient: authenticate success, global_id 13856 2024-09-16T03:47:51.653 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.639+0000 7f217affd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:47:51.653 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.639+0000 7f218270a640 10 monclient: get_monmap_and_config success 2024-09-16T03:47:51.653 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.639+0000 7f217affd640 1 -- 172.21.15.42:0/452993245 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21640ac860 con 0x7f217c0a1330 2024-09-16T03:47:51.653 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.639+0000 7f218270a640 4 set_mon_vals no callback set 2024-09-16T03:47:51.653 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.639+0000 7f217affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:47:51.654 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.639+0000 7f217affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:47:51.654 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.639+0000 7f217affd640 10 monclient: dump: 2024-09-16T03:47:51.654 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:47:51.654 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:51.654 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:51.654 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:51.654 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:47:51.654 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:51.654 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:47:51.654 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:47:51.654 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:47:51.654 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:51.654 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.639+0000 7f218270a640 10 monclient: shutdown 2024-09-16T03:47:51.655 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.639+0000 7f218270a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:47:51.655 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.639+0000 7f218270a640 1 -- 172.21.15.42:0/452993245 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f217c0a1330 msgr2=0x7f217c0a1710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:51.655 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.639+0000 7f218270a640 1 --2- 172.21.15.42:0/452993245 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f217c0a1330 0x7f217c0a1710 secure :-1 s=READY pgs=2507 cs=0 l=1 rev1=1 crypto rx=0x7f21640097c0 tx=0x7f216401c3c0 comp rx=0 tx=0).stop 2024-09-16T03:47:51.655 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.639+0000 7f218270a640 1 -- 172.21.15.42:0/452993245 shutdown_connections 2024-09-16T03:47:51.655 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.639+0000 7f218270a640 1 --2- 172.21.15.42:0/452993245 >> v2:172.21.15.42:3300/0 conn(0x7f217c0a1c50 0x7f217c135810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:51.655 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.639+0000 7f218270a640 1 --2- 172.21.15.42:0/452993245 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f217c0a1330 0x7f217c0a1710 unknown :-1 s=CLOSED pgs=2507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:51.655 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.639+0000 7f218270a640 1 -- 172.21.15.42:0/452993245 >> 172.21.15.42:0/452993245 conn(0x7f217c0a2420 msgr2=0x7f217c0a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:47:51.655 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.639+0000 7f218270a640 1 -- 172.21.15.42:0/452993245 shutdown_connections 2024-09-16T03:47:51.655 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.639+0000 7f218270a640 1 -- 172.21.15.42:0/452993245 wait complete. 2024-09-16T03:47:51.655 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.639+0000 7f218270a640 10 monclient: build_initial_monmap 2024-09-16T03:47:51.655 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.639+0000 7f218270a640 10 monclient: monmap: 2024-09-16T03:47:51.655 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:47:51.655 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:47:51.655 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:47:51.656 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:47:51.656 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:47:51.656 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:51.656 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:47:51.656 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:47:51.656 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:47:51.656 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:51.656 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.639+0000 7f218270a640 1 librados: starting msgr at 2024-09-16T03:47:51.656 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.639+0000 7f218270a640 1 librados: starting objecter 2024-09-16T03:47:51.656 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.639+0000 7f218270a640 1 Processor -- start 2024-09-16T03:47:51.656 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.639+0000 7f218270a640 1 -- start start 2024-09-16T03:47:51.656 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.639+0000 7f218270a640 1 librados: setting wanted keys 2024-09-16T03:47:51.656 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.639+0000 7f218270a640 1 librados: calling monclient init 2024-09-16T03:47:51.656 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.639+0000 7f218270a640 10 monclient: init 2024-09-16T03:47:51.657 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.643+0000 7f218270a640 10 monclient: _reopen_session rank -1 2024-09-16T03:47:51.657 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.643+0000 7f218270a640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T03:47:51.657 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.643+0000 7f218270a640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f217c0a1330 0x7f217c1cedb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:51.657 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.643+0000 7f218270a640 10 monclient(hunting): picked mon.noname-b con 0x7f217c0a1330 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:47:51.657 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.643+0000 7f218270a640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f217c0a1c50 0x7f217c1351a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:51.657 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.643+0000 7f218270a640 10 monclient(hunting): picked mon.noname-c con 0x7f217c0a1c50 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:47:51.657 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.643+0000 7f218270a640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f217c1356e0 0x7f217c09cb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:51.657 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.643+0000 7f218270a640 10 monclient(hunting): picked mon.noname-a con 0x7f217c1356e0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:47:51.657 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.643+0000 7f218270a640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:51.657 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.643+0000 7f218270a640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f217c09ef30 con 0x7f217c1356e0 2024-09-16T03:47:51.657 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.643+0000 7f218270a640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:51.657 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.643+0000 7f218270a640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f217c09edb0 con 0x7f217c0a1330 2024-09-16T03:47:51.657 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.643+0000 7f218270a640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:51.657 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.643+0000 7f218270a640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f217c135d50 con 0x7f217c0a1c50 2024-09-16T03:47:51.658 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.643+0000 7f218270a640 10 monclient(hunting): _renew_subs 2024-09-16T03:47:51.658 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.643+0000 7f218270a640 10 monclient(hunting): authenticate will time out at 1318.612400s 2024-09-16T03:47:51.658 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.643+0000 7f217b7fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f217c0a1c50 0x7f217c1351a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:51.658 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.643+0000 7f2180ca7640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f217c1356e0 0x7f217c09cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:51.658 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.643+0000 7f217b7fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f217c0a1c50 0x7f217c1351a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:59514/0 (socket says 172.21.15.42:59514) 2024-09-16T03:47:51.658 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.643+0000 7f217b7fe640 1 -- 172.21.15.42:0/1091824739 learned_addr learned my addr 172.21.15.42:0/1091824739 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:47:51.658 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.643+0000 7f217b7fe640 10 monclient(hunting): get_auth_request con 0x7f217c0a1c50 auth_method 0 2024-09-16T03:47:51.658 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.643+0000 7f217b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:51.658 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.643+0000 7f217b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:51.658 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.643+0000 7f217b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:47:51.658 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.643+0000 7f2180ca7640 10 monclient(hunting): get_auth_request con 0x7f217c1356e0 auth_method 0 2024-09-16T03:47:51.658 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.643+0000 7f2180ca7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:51.658 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.643+0000 7f2180ca7640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:51.659 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.643+0000 7f2180ca7640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:47:51.659 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.643+0000 7f217b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:47:51.659 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.643+0000 7f217b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:47:51.659 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.643+0000 7f217b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:47:51.659 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.643+0000 7f2180ca7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:47:51.659 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.643+0000 7f2180ca7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:47:51.659 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.643+0000 7f2180ca7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:47:51.659 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.643+0000 7f217bfff640 1 --2- 172.21.15.42:0/1091824739 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f217c0a1330 0x7f217c1cedb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:51.659 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.643+0000 7f217bfff640 10 monclient(hunting): get_auth_request con 0x7f217c0a1330 auth_method 0 2024-09-16T03:47:51.659 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.643+0000 7f217bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:51.659 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.643+0000 7f217bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:51.659 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.643+0000 7f217bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:47:51.659 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.643+0000 7f217b7fe640 10 monclient(hunting): handle_auth_done global_id 13859 payload 824 2024-09-16T03:47:51.659 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.643+0000 7f217b7fe640 1 -- 172.21.15.42:0/1091824739 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f217c0a1330 msgr2=0x7f217c1cedb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:51.660 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.643+0000 7f217b7fe640 1 --2- 172.21.15.42:0/1091824739 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f217c0a1330 0x7f217c1cedb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:51.660 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.643+0000 7f217b7fe640 1 -- 172.21.15.42:0/1091824739 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f217c1356e0 msgr2=0x7f217c09cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:51.660 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.643+0000 7f217b7fe640 1 --2- 172.21.15.42:0/1091824739 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f217c1356e0 0x7f217c09cb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:51.660 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.643+0000 7f217b7fe640 10 monclient: _finish_hunting 0 2024-09-16T03:47:51.660 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.643+0000 7f217b7fe640 1 monclient: found mon.noname-c 2024-09-16T03:47:51.660 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.643+0000 7f217b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:47:51.660 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.643+0000 7f217b7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:47:51.660 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.643+0000 7f217b7fe640 1 -- 172.21.15.42:0/1091824739 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f217c121b50 con 0x7f217c0a1c50 2024-09-16T03:47:51.660 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.643+0000 7f2180ca7640 1 --2- 172.21.15.42:0/1091824739 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f217c1356e0 0x7f217c09cb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:47:51.660 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.643+0000 7f217b7fe640 1 --2- 172.21.15.42:0/1091824739 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f217c0a1c50 0x7f217c1351a0 secure :-1 s=READY pgs=2508 cs=0 l=1 rev1=1 crypto rx=0x7f216c00c420 tx=0x7f216c00c8e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:47:51.660 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.643+0000 7f2178ff9640 10 client.?.objecter ms_handle_connect 0x7f217c0a1c50 2024-09-16T03:47:51.660 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.643+0000 7f2178ff9640 10 client.?.objecter resend_mon_ops 2024-09-16T03:47:51.660 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.643+0000 7f2178ff9640 1 -- 172.21.15.42:0/1091824739 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f216c0052d0 con 0x7f217c0a1c50 2024-09-16T03:47:51.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.643+0000 7f2178ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:47:51.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.643+0000 7f2178ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:47:51.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.643+0000 7f2178ff9640 10 monclient: dump: 2024-09-16T03:47:51.661 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:47:51.661 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:51.661 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:51.661 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:51.661 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:47:51.661 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:51.661 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:47:51.661 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:47:51.661 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:47:51.662 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:51.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.643+0000 7f2178ff9640 10 monclient: _finish_auth 0 2024-09-16T03:47:51.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.643+0000 7f2178ff9640 10 monclient: _check_auth_tickets 2024-09-16T03:47:51.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.643+0000 7f2178ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:47:51.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.643+0000 7f218270a640 5 monclient: authenticate success, global_id 13859 2024-09-16T03:47:51.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.643+0000 7f2178ff9640 1 -- 172.21.15.42:0/1091824739 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f216c005470 con 0x7f217c0a1c50 2024-09-16T03:47:51.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.643+0000 7f218270a640 10 monclient: _renew_subs 2024-09-16T03:47:51.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.643+0000 7f218270a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:47:51.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.643+0000 7f218270a640 1 -- 172.21.15.42:0/1091824739 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f217c071350 con 0x7f217c0a1c50 2024-09-16T03:47:51.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.643+0000 7f2178ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T03:47:51.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.643+0000 7f2178ff9640 1 -- 172.21.15.42:0/1091824739 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f216c007260 con 0x7f217c0a1c50 2024-09-16T03:47:51.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.643+0000 7f2178ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:47:51.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.643+0000 7f2178ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:47:51.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.643+0000 7f2178ff9640 10 monclient: dump: 2024-09-16T03:47:51.663 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:47:51.663 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:51.663 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:51.663 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:51.663 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:47:51.663 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:51.663 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:47:51.663 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:47:51.663 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:47:51.663 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:51.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.643+0000 7f218270a640 10 client.13859.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:47:51.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.643+0000 7f2179ffb640 4 set_mon_vals no callback set 2024-09-16T03:47:51.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.643+0000 7f218270a640 10 monclient: _renew_subs 2024-09-16T03:47:51.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.643+0000 7f218270a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:47:51.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.643+0000 7f218270a640 1 -- 172.21.15.42:0/1091824739 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f217c073a30 con 0x7f217c0a1c50 2024-09-16T03:47:51.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.643+0000 7f218270a640 1 librados: init done 2024-09-16T03:47:51.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.643+0000 7f218270a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:47:51.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.643+0000 7f218270a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:47:51.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.643+0000 7f218270a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:47:51.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.643+0000 7f218270a640 1 -- 172.21.15.42:0/1091824739 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f217c0a1710 con 0x7f217c0a1c50 2024-09-16T03:47:51.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.643+0000 7f2178ff9640 1 -- 172.21.15.42:0/1091824739 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f216c007400 con 0x7f217c0a1c50 2024-09-16T03:47:51.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.643+0000 7f2178ff9640 1 --2- 172.21.15.42:0/1091824739 >> v2:172.21.15.154:6812/254845342 conn(0x7f215c037810 0x7f215c039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:51.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.643+0000 7f2178ff9640 1 -- 172.21.15.42:0/1091824739 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(181..181 src has 1..181) ==== 6991+0+0 (secure 0 0 0) 0x7f216c048900 con 0x7f217c0a1c50 2024-09-16T03:47:51.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.643+0000 7f2178ff9640 10 client.13859.objecter ms_dispatch 0x7f217c000ba0 osd_map(181..181 src has 1..181) 2024-09-16T03:47:51.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.643+0000 7f2178ff9640 3 client.13859.objecter handle_osd_map got epochs [181,181] > 0 2024-09-16T03:47:51.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.643+0000 7f2178ff9640 3 client.13859.objecter handle_osd_map decoding full epoch 181 2024-09-16T03:47:51.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.643+0000 7f2178ff9640 20 client.13859.objecter dump_active .. 0 homeless 2024-09-16T03:47:51.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.643+0000 7f217bfff640 1 --2- 172.21.15.42:0/1091824739 >> v2:172.21.15.154:6812/254845342 conn(0x7f215c037810 0x7f215c039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:51.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.643+0000 7f217bfff640 10 monclient: get_auth_request con 0x7f215c037810 auth_method 0 2024-09-16T03:47:51.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.643+0000 7f2178ff9640 10 client.13859.objecter ms_handle_connect 0x7f215c037810 2024-09-16T03:47:51.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.643+0000 7f217bfff640 1 --2- 172.21.15.42:0/1091824739 >> v2:172.21.15.154:6812/254845342 conn(0x7f215c037810 0x7f215c039cd0 secure :-1 s=READY pgs=3464 cs=0 l=1 rev1=1 crypto rx=0x7f21640ae000 tx=0x7f2164008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:47:51.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.647+0000 7f2178ff9640 1 -- 172.21.15.42:0/1091824739 <== mon.2 v2:172.21.15.42:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f216c0067e0 con 0x7f217c0a1c50 2024-09-16T03:47:51.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.647+0000 7f2178ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:47:51.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.647+0000 7f2178ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:47:51.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.879+0000 7f218270a640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:47:51.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.879+0000 7f218270a640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:47:51.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.879+0000 7f218270a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:47:51.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.879+0000 7f218270a640 1 -- 172.21.15.42:0/1091824739 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f217c121930 con 0x7f217c0a1c50 2024-09-16T03:47:51.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.879+0000 7f2178ff9640 1 -- 172.21.15.42:0/1091824739 <== mon.2 v2:172.21.15.42:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7f216c012a30 con 0x7f217c0a1c50 2024-09-16T03:47:51.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.879+0000 7f2178ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:47:51.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.879+0000 7f2178ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:47:51.883 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":54,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:47:51.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.879+0000 7f218270a640 10 librados: watch_flush enter 2024-09-16T03:47:51.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.879+0000 7f218270a640 10 librados: watch_flush exit 2024-09-16T03:47:51.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.879+0000 7f218270a640 20 client.13859.objecter shutdown clearing up homeless session... 2024-09-16T03:47:51.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.879+0000 7f218270a640 10 client.13859.objecter successfully canceled tick 2024-09-16T03:47:51.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.879+0000 7f218270a640 1 -- 172.21.15.42:0/1091824739 >> v2:172.21.15.154:6812/254845342 conn(0x7f215c037810 msgr2=0x7f215c039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:51.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.879+0000 7f218270a640 1 --2- 172.21.15.42:0/1091824739 >> v2:172.21.15.154:6812/254845342 conn(0x7f215c037810 0x7f215c039cd0 secure :-1 s=READY pgs=3464 cs=0 l=1 rev1=1 crypto rx=0x7f21640ae000 tx=0x7f2164008040 comp rx=0 tx=0).stop 2024-09-16T03:47:51.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.879+0000 7f218270a640 10 monclient: shutdown 2024-09-16T03:47:51.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.879+0000 7f218270a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:47:51.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.879+0000 7f218270a640 1 -- 172.21.15.42:0/1091824739 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f217c0a1c50 msgr2=0x7f217c1351a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:51.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.879+0000 7f218270a640 1 --2- 172.21.15.42:0/1091824739 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f217c0a1c50 0x7f217c1351a0 secure :-1 s=READY pgs=2508 cs=0 l=1 rev1=1 crypto rx=0x7f216c00c420 tx=0x7f216c00c8e0 comp rx=0 tx=0).stop 2024-09-16T03:47:51.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.879+0000 7f218270a640 1 -- 172.21.15.42:0/1091824739 shutdown_connections 2024-09-16T03:47:51.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.879+0000 7f218270a640 1 --2- 172.21.15.42:0/1091824739 >> v2:172.21.15.154:6812/254845342 conn(0x7f215c037810 0x7f215c039cd0 unknown :-1 s=CLOSED pgs=3464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:51.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.879+0000 7f218270a640 1 --2- 172.21.15.42:0/1091824739 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f217c1356e0 0x7f217c09cb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:51.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.879+0000 7f218270a640 1 --2- 172.21.15.42:0/1091824739 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f217c0a1c50 0x7f217c1351a0 unknown :-1 s=CLOSED pgs=2508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:51.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.879+0000 7f218270a640 1 --2- 172.21.15.42:0/1091824739 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f217c0a1330 0x7f217c1cedb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:51.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.879+0000 7f218270a640 1 -- 172.21.15.42:0/1091824739 >> 172.21.15.42:0/1091824739 conn(0x7f217c0a2420 msgr2=0x7f217c0a0a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:47:51.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.879+0000 7f218270a640 1 -- 172.21.15.42:0/1091824739 shutdown_connections 2024-09-16T03:47:51.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.879+0000 7f218270a640 1 -- 172.21.15.42:0/1091824739 wait complete. 2024-09-16T03:47:51.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:51.883+0000 7f218270a640 1 librados: shutdown 2024-09-16T03:47:51.901 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T03:47:51.901 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T03:47:51.902 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T03:47:51.902 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T03:47:52.018 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.015+0000 7f1c1c119640 10 monclient: get_monmap_and_config 2024-09-16T03:47:52.018 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.015+0000 7f1c1c119640 10 monclient: build_initial_monmap 2024-09-16T03:47:52.018 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.015+0000 7f1c1c119640 10 monclient: monmap: 2024-09-16T03:47:52.018 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:47:52.018 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:52.019 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:47:52.017396+0000 2024-09-16T03:47:52.019 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:47:52.017396+0000 2024-09-16T03:47:52.019 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:47:52.019 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:52.019 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:47:52.019 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:47:52.019 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:47:52.019 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:47:52.019 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:47:52.019 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:52.022 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.015+0000 7f1c1c119640 1 Processor -- start 2024-09-16T03:47:52.022 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.015+0000 7f1c1c119640 1 -- start start 2024-09-16T03:47:52.022 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.015+0000 7f1c1c119640 10 monclient: init 2024-09-16T03:47:52.022 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.015+0000 7f1c1c119640 10 monclient: _reopen_session rank -1 2024-09-16T03:47:52.022 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.015+0000 7f1c1c119640 10 monclient: _add_conns ranks=[1,4,2,3,0] 2024-09-16T03:47:52.022 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.015+0000 7f1c1c119640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1c1406db90 addr v1:172.21.15.154:6789/0 2024-09-16T03:47:52.023 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.015+0000 7f1c1c119640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f1c141340c0 0x7f1c1413c190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:52.023 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.015+0000 7f1c1c119640 10 monclient(hunting): picked mon.noname-c con 0x7f1c141340c0 addr v2:172.21.15.154:3300/0 2024-09-16T03:47:52.023 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.015+0000 7f1c1c119640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1c1413c6d0 0x7f1c1413eac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:52.023 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.015+0000 7f1c1c119640 10 monclient(hunting): picked mon.noname-b con 0x7f1c1413c6d0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:47:52.023 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.015+0000 7f1c1c119640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1c1406ece0 con 0x7f1c1406db90 2024-09-16T03:47:52.023 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.015+0000 7f1c1c119640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:52.023 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.015+0000 7f1c1c119640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f1c1406df70 con 0x7f1c141340c0 2024-09-16T03:47:52.023 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.015+0000 7f1c1c119640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:52.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.015+0000 7f1c1c119640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f1c1409bf80 con 0x7f1c1413c6d0 2024-09-16T03:47:52.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.015+0000 7f1c1c119640 10 monclient(hunting): _renew_subs 2024-09-16T03:47:52.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.015+0000 7f1c1c119640 10 monclient(hunting): authenticate will time out at 1318.987527s 2024-09-16T03:47:52.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.015+0000 7f1c1a6b6640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1c1413c6d0 0x7f1c1413eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:52.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.015+0000 7f1c1a6b6640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1c1413c6d0 0x7f1c1413eac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:59522/0 (socket says 172.21.15.42:59522) 2024-09-16T03:47:52.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.015+0000 7f1c1a6b6640 1 -- 172.21.15.42:0/1865204659 learned_addr learned my addr 172.21.15.42:0/1865204659 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:47:52.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.015+0000 7f1c1a6b6640 10 monclient(hunting): get_auth_request con 0x7f1c1413c6d0 auth_method 0 2024-09-16T03:47:52.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.015+0000 7f1c1a6b6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:52.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.015+0000 7f1c1a6b6640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:52.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.015+0000 7f1c1a6b6640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:47:52.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.015+0000 7f1c1a6b6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:47:52.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.015+0000 7f1c1a6b6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:47:52.025 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.015+0000 7f1c1a6b6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:47:52.025 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.015+0000 7f1c196b4640 1 --2- 172.21.15.42:0/1865204659 >> v2:172.21.15.154:3300/0 conn(0x7f1c141340c0 0x7f1c1413c190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:52.025 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.015+0000 7f1c196b4640 10 monclient(hunting): get_auth_request con 0x7f1c141340c0 auth_method 0 2024-09-16T03:47:52.025 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.019+0000 7f1c196b4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:52.025 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.019+0000 7f1c196b4640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:52.025 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.019+0000 7f1c196b4640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:47:52.025 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.019+0000 7f1c1a6b6640 10 monclient(hunting): handle_auth_done global_id 13862 payload 274 2024-09-16T03:47:52.025 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.019+0000 7f1c1a6b6640 1 -- 172.21.15.42:0/1865204659 >> v2:172.21.15.154:3300/0 conn(0x7f1c141340c0 msgr2=0x7f1c1413c190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:47:52.025 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.019+0000 7f1c1a6b6640 1 --2- 172.21.15.42:0/1865204659 >> v2:172.21.15.154:3300/0 conn(0x7f1c141340c0 0x7f1c1413c190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:52.025 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.019+0000 7f1c1a6b6640 1 -- 172.21.15.42:0/1865204659 >> v1:172.21.15.154:6789/0 conn(0x7f1c1406db90 legacy=0x7f1c140a37f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:47:52.025 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.019+0000 7f1c1a6b6640 10 monclient: _finish_hunting 0 2024-09-16T03:47:52.025 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.019+0000 7f1c1a6b6640 1 monclient: found mon.noname-b 2024-09-16T03:47:52.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.019+0000 7f1c1a6b6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:47:52.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.019+0000 7f1c1a6b6640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T03:47:52.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.019+0000 7f1c1a6b6640 1 -- 172.21.15.42:0/1865204659 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c141344d0 con 0x7f1c1413c6d0 2024-09-16T03:47:52.034 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.019+0000 7f1c1a6b6640 1 --2- 172.21.15.42:0/1865204659 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1c1413c6d0 0x7f1c1413eac0 secure :-1 s=READY pgs=2509 cs=0 l=1 rev1=1 crypto rx=0x7f1c0c009e50 tx=0x7f1c0c01c8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f2dbb45320cc643d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:47:52.034 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.019+0000 7f1c18eb3640 1 -- 172.21.15.42:0/1865204659 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c0c0a5730 con 0x7f1c1413c6d0 2024-09-16T03:47:52.034 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.019+0000 7f1c18eb3640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:47:52.034 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.019+0000 7f1c18eb3640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:47:52.034 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.019+0000 7f1c18eb3640 10 monclient: dump: 2024-09-16T03:47:52.034 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:47:52.034 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:52.034 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:52.035 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:52.036 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:47:52.036 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:52.036 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:47:52.036 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:47:52.036 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:47:52.037 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:52.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.019+0000 7f1c18eb3640 10 monclient: _finish_auth 0 2024-09-16T03:47:52.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.019+0000 7f1c18eb3640 10 monclient: _check_auth_tickets 2024-09-16T03:47:52.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.019+0000 7f1c18eb3640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:47:52.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.019+0000 7f1c18eb3640 1 -- 172.21.15.42:0/1865204659 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1c0c0a58d0 con 0x7f1c1413c6d0 2024-09-16T03:47:52.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.019+0000 7f1c1c119640 5 monclient: authenticate success, global_id 13862 2024-09-16T03:47:52.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.019+0000 7f1c1c119640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:47:52.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.019+0000 7f1c18eb3640 10 monclient: handle_config config(0 keys) 2024-09-16T03:47:52.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.019+0000 7f1c1c119640 10 monclient: get_monmap_and_config success 2024-09-16T03:47:52.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.019+0000 7f1c1c119640 4 set_mon_vals no callback set 2024-09-16T03:47:52.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.019+0000 7f1c18eb3640 1 -- 172.21.15.42:0/1865204659 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c0c0a5ca0 con 0x7f1c1413c6d0 2024-09-16T03:47:52.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.019+0000 7f1c18eb3640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:47:52.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.019+0000 7f1c18eb3640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:47:52.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.019+0000 7f1c18eb3640 10 monclient: dump: 2024-09-16T03:47:52.038 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:47:52.038 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:52.038 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:52.038 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:52.038 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:47:52.038 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:52.038 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:47:52.038 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:47:52.038 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:47:52.038 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:52.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.019+0000 7f1c1c119640 10 monclient: shutdown 2024-09-16T03:47:52.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.019+0000 7f1c1c119640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:47:52.039 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.019+0000 7f1c1c119640 1 -- 172.21.15.42:0/1865204659 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1c1413c6d0 msgr2=0x7f1c1413eac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:52.039 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.019+0000 7f1c1c119640 1 --2- 172.21.15.42:0/1865204659 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1c1413c6d0 0x7f1c1413eac0 secure :-1 s=READY pgs=2509 cs=0 l=1 rev1=1 crypto rx=0x7f1c0c009e50 tx=0x7f1c0c01c8b0 comp rx=0 tx=0).stop 2024-09-16T03:47:52.039 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.019+0000 7f1c1c119640 1 -- 172.21.15.42:0/1865204659 shutdown_connections 2024-09-16T03:47:52.039 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.019+0000 7f1c1c119640 1 --2- 172.21.15.42:0/1865204659 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1c1413c6d0 0x7f1c1413eac0 unknown :-1 s=CLOSED pgs=2509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:52.039 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.019+0000 7f1c1c119640 1 --2- 172.21.15.42:0/1865204659 >> v2:172.21.15.154:3300/0 conn(0x7f1c141340c0 0x7f1c1413c190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:52.039 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.019+0000 7f1c1c119640 1 -- 172.21.15.42:0/1865204659 >> 172.21.15.42:0/1865204659 conn(0x7f1c140a9fb0 msgr2=0x7f1c140aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:47:52.039 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.019+0000 7f1c1c119640 1 -- 172.21.15.42:0/1865204659 shutdown_connections 2024-09-16T03:47:52.039 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.019+0000 7f1c1c119640 1 -- 172.21.15.42:0/1865204659 wait complete. 2024-09-16T03:47:52.039 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.019+0000 7f1c1c119640 10 monclient: build_initial_monmap 2024-09-16T03:47:52.039 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.019+0000 7f1c1c119640 10 monclient: monmap: 2024-09-16T03:47:52.039 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:47:52.039 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:47:52.039 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:47:52.039 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:47:52.040 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:47:52.040 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:52.040 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:47:52.040 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:47:52.040 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:47:52.040 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:52.040 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.019+0000 7f1c1c119640 1 librados: starting msgr at 2024-09-16T03:47:52.040 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.019+0000 7f1c1c119640 1 librados: starting objecter 2024-09-16T03:47:52.040 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.019+0000 7f1c1c119640 1 Processor -- start 2024-09-16T03:47:52.040 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.019+0000 7f1c1c119640 1 -- start start 2024-09-16T03:47:52.040 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.019+0000 7f1c1c119640 1 librados: setting wanted keys 2024-09-16T03:47:52.040 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.019+0000 7f1c1c119640 1 librados: calling monclient init 2024-09-16T03:47:52.040 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.019+0000 7f1c1c119640 10 monclient: init 2024-09-16T03:47:52.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.023+0000 7f1c1c119640 10 monclient: _reopen_session rank -1 2024-09-16T03:47:52.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.023+0000 7f1c1c119640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:47:52.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.023+0000 7f1c1c119640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1c1406db90 0x7f1c14139f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:52.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.023+0000 7f1c1c119640 10 monclient(hunting): picked mon.noname-c con 0x7f1c1406db90 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:47:52.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.023+0000 7f1c1c119640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1c141340c0 0x7f1c1413a470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:52.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.023+0000 7f1c1c119640 10 monclient(hunting): picked mon.noname-a con 0x7f1c141340c0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:47:52.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.023+0000 7f1c1c119640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1c1413c6d0 0x7f1c1413a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:52.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.023+0000 7f1c1c119640 10 monclient(hunting): picked mon.noname-b con 0x7f1c1413c6d0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:47:52.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.023+0000 7f1c1c119640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:52.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.023+0000 7f1c1c119640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f1c14141820 con 0x7f1c141340c0 2024-09-16T03:47:52.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.023+0000 7f1c1c119640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:52.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.023+0000 7f1c1c119640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f1c141419a0 con 0x7f1c1413c6d0 2024-09-16T03:47:52.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.023+0000 7f1c1c119640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:52.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.023+0000 7f1c1c119640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f1c14141b20 con 0x7f1c1406db90 2024-09-16T03:47:52.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.023+0000 7f1c1c119640 10 monclient(hunting): _renew_subs 2024-09-16T03:47:52.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.023+0000 7f1c1c119640 10 monclient(hunting): authenticate will time out at 1318.992490s 2024-09-16T03:47:52.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.023+0000 7f1c19eb5640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1c1406db90 0x7f1c14139f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:52.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.023+0000 7f1c196b4640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1c141340c0 0x7f1c1413a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:52.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.023+0000 7f1c19eb5640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1c1406db90 0x7f1c14139f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:59538/0 (socket says 172.21.15.42:59538) 2024-09-16T03:47:52.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.023+0000 7f1c19eb5640 1 -- 172.21.15.42:0/3795288361 learned_addr learned my addr 172.21.15.42:0/3795288361 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:47:52.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.023+0000 7f1c19eb5640 10 monclient(hunting): get_auth_request con 0x7f1c1406db90 auth_method 0 2024-09-16T03:47:52.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.023+0000 7f1c19eb5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:52.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.023+0000 7f1c19eb5640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:52.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.023+0000 7f1c19eb5640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:47:52.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.023+0000 7f1c196b4640 10 monclient(hunting): get_auth_request con 0x7f1c141340c0 auth_method 0 2024-09-16T03:47:52.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.023+0000 7f1c196b4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:52.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.023+0000 7f1c196b4640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:52.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.023+0000 7f1c196b4640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:47:52.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.023+0000 7f1c19eb5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:47:52.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.023+0000 7f1c19eb5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:47:52.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.023+0000 7f1c19eb5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:47:52.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.023+0000 7f1c196b4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:47:52.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.023+0000 7f1c196b4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:47:52.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.023+0000 7f1c196b4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:47:52.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.023+0000 7f1c1a6b6640 1 --2- 172.21.15.42:0/3795288361 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1c1413c6d0 0x7f1c1413a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:52.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.023+0000 7f1c19eb5640 10 monclient(hunting): handle_auth_done global_id 13865 payload 824 2024-09-16T03:47:52.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.023+0000 7f1c19eb5640 1 -- 172.21.15.42:0/3795288361 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1c1413c6d0 msgr2=0x7f1c1413a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:52.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.023+0000 7f1c19eb5640 1 --2- 172.21.15.42:0/3795288361 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1c1413c6d0 0x7f1c1413a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:52.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.023+0000 7f1c19eb5640 1 -- 172.21.15.42:0/3795288361 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1c141340c0 msgr2=0x7f1c1413a470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:52.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.023+0000 7f1c19eb5640 1 --2- 172.21.15.42:0/3795288361 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1c141340c0 0x7f1c1413a470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:52.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.023+0000 7f1c19eb5640 10 monclient: _finish_hunting 0 2024-09-16T03:47:52.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.023+0000 7f1c19eb5640 1 monclient: found mon.noname-c 2024-09-16T03:47:52.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.023+0000 7f1c19eb5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:47:52.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.023+0000 7f1c19eb5640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:47:52.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.023+0000 7f1c19eb5640 1 -- 172.21.15.42:0/3795288361 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c1413bfa0 con 0x7f1c1406db90 2024-09-16T03:47:52.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.023+0000 7f1c1a6b6640 10 monclient: get_auth_request con 0x7f1c1413c6d0 auth_method 0 2024-09-16T03:47:52.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.023+0000 7f1c196b4640 1 --2- 172.21.15.42:0/3795288361 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1c141340c0 0x7f1c1413a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:47:52.045 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.023+0000 7f1c1a6b6640 1 --2- 172.21.15.42:0/3795288361 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1c1413c6d0 0x7f1c1413a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T03:47:52.045 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.023+0000 7f1c19eb5640 1 --2- 172.21.15.42:0/3795288361 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1c1406db90 0x7f1c14139f30 secure :-1 s=READY pgs=2510 cs=0 l=1 rev1=1 crypto rx=0x7f1c1000b410 tx=0x7f1c1000b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:47:52.045 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.023+0000 7f1c0a7fc640 10 client.?.objecter ms_handle_connect 0x7f1c1406db90 2024-09-16T03:47:52.045 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.023+0000 7f1c0a7fc640 10 client.?.objecter resend_mon_ops 2024-09-16T03:47:52.045 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.023+0000 7f1c0a7fc640 1 -- 172.21.15.42:0/3795288361 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c10015030 con 0x7f1c1406db90 2024-09-16T03:47:52.045 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.023+0000 7f1c0a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:47:52.045 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.023+0000 7f1c0a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:47:52.045 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.023+0000 7f1c0a7fc640 10 monclient: dump: 2024-09-16T03:47:52.045 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:47:52.045 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:52.045 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:52.045 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:52.046 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:47:52.046 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:52.046 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:47:52.046 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:47:52.046 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:47:52.046 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:52.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.023+0000 7f1c0a7fc640 10 monclient: _finish_auth 0 2024-09-16T03:47:52.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.023+0000 7f1c0a7fc640 10 monclient: _check_auth_tickets 2024-09-16T03:47:52.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.023+0000 7f1c0a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:47:52.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.023+0000 7f1c1c119640 5 monclient: authenticate success, global_id 13865 2024-09-16T03:47:52.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.023+0000 7f1c0a7fc640 1 -- 172.21.15.42:0/3795288361 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1c10004020 con 0x7f1c1406db90 2024-09-16T03:47:52.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.023+0000 7f1c0a7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:47:52.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.023+0000 7f1c0a7fc640 1 -- 172.21.15.42:0/3795288361 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c10004450 con 0x7f1c1406db90 2024-09-16T03:47:52.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.023+0000 7f1c0a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:47:52.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.023+0000 7f1c0a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:47:52.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.023+0000 7f1c0bfff640 4 set_mon_vals no callback set 2024-09-16T03:47:52.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.023+0000 7f1c0a7fc640 10 monclient: dump: 2024-09-16T03:47:52.047 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:47:52.047 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:52.047 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:52.047 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:52.047 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:47:52.047 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:52.047 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:47:52.047 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:47:52.047 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:47:52.048 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:52.048 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.023+0000 7f1c1c119640 10 monclient: _renew_subs 2024-09-16T03:47:52.048 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.023+0000 7f1c1c119640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:47:52.048 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.023+0000 7f1c1c119640 1 -- 172.21.15.42:0/3795288361 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c1413ead0 con 0x7f1c1406db90 2024-09-16T03:47:52.048 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.023+0000 7f1c1c119640 10 client.13865.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:47:52.048 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.023+0000 7f1c1c119640 10 monclient: _renew_subs 2024-09-16T03:47:52.048 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.023+0000 7f1c1c119640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:47:52.048 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.023+0000 7f1c1c119640 1 -- 172.21.15.42:0/3795288361 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c1409e3a0 con 0x7f1c1406db90 2024-09-16T03:47:52.048 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.023+0000 7f1c1c119640 1 librados: init done 2024-09-16T03:47:52.048 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.023+0000 7f1c1c119640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:47:52.048 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.023+0000 7f1c1c119640 1 --2- 172.21.15.42:0/3795288361 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1c141cf770 0x7f1c141cfb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:52.048 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.023+0000 7f1c1c119640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:52.048 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.023+0000 7f1c1c119640 1 -- 172.21.15.42:0/3795288361 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f1c140a4110 con 0x7f1c141cf770 2024-09-16T03:47:52.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.023+0000 7f1c196b4640 1 --2- 172.21.15.42:0/3795288361 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1c141cf770 0x7f1c141cfb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:52.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.023+0000 7f1c196b4640 10 monclient: get_auth_request con 0x7f1c141cf770 auth_method 0 2024-09-16T03:47:52.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.023+0000 7f1c196b4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:52.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.023+0000 7f1c196b4640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:52.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.023+0000 7f1c196b4640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:47:52.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.023+0000 7f1c196b4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:47:52.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.023+0000 7f1c196b4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:47:52.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.023+0000 7f1c196b4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:47:52.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.023+0000 7f1c0a7fc640 1 -- 172.21.15.42:0/3795288361 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f1c10018070 con 0x7f1c1406db90 2024-09-16T03:47:52.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.023+0000 7f1c196b4640 10 monclient(hunting): handle_auth_done global_id 764337 payload 824 2024-09-16T03:47:52.050 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.023+0000 7f1c196b4640 1 -- 172.21.15.42:0/3795288361 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1c1409df90 con 0x7f1c141cf770 2024-09-16T03:47:52.050 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.023+0000 7f1c0a7fc640 1 --2- 172.21.15.42:0/3795288361 >> v2:172.21.15.154:6812/254845342 conn(0x7f1bf8037810 0x7f1bf8039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:52.050 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.023+0000 7f1c0a7fc640 1 -- 172.21.15.42:0/3795288361 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(181..181 src has 1..181) ==== 6991+0+0 (secure 0 0 0) 0x7f1c10049020 con 0x7f1c1406db90 2024-09-16T03:47:52.050 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.023+0000 7f1c0a7fc640 10 client.13865.objecter ms_dispatch 0x7f1c14000ba0 osd_map(181..181 src has 1..181) 2024-09-16T03:47:52.050 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.023+0000 7f1c0a7fc640 3 client.13865.objecter handle_osd_map got epochs [181,181] > 0 2024-09-16T03:47:52.050 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.023+0000 7f1c0a7fc640 3 client.13865.objecter handle_osd_map decoding full epoch 181 2024-09-16T03:47:52.050 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.023+0000 7f1c0a7fc640 20 client.13865.objecter dump_active .. 0 homeless 2024-09-16T03:47:52.050 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.023+0000 7f1c196b4640 1 --2- 172.21.15.42:0/3795288361 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1c141cf770 0x7f1c141cfb50 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f1c1413bf50 tx=0x7f1c0400e1d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:47:52.050 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.023+0000 7f1c0a7fc640 10 client.13865.objecter ms_handle_connect 0x7f1c141cf770 2024-09-16T03:47:52.050 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.023+0000 7f1c0a7fc640 10 client.13865.objecter resend_mon_ops 2024-09-16T03:47:52.050 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.023+0000 7f1c0a7fc640 1 -- 172.21.15.42:0/3795288361 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c04018020 con 0x7f1c141cf770 2024-09-16T03:47:52.050 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.023+0000 7f1c0a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:47:52.050 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.023+0000 7f1c0a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:47:52.050 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.023+0000 7f1c0a7fc640 10 monclient: dump: 2024-09-16T03:47:52.051 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:47:52.051 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:52.051 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:52.051 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:52.051 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:47:52.051 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:52.051 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:47:52.051 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:47:52.051 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:47:52.051 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:52.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.023+0000 7f1c1a6b6640 1 --2- 172.21.15.42:0/3795288361 >> v2:172.21.15.154:6812/254845342 conn(0x7f1bf8037810 0x7f1bf8039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:52.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.023+0000 7f1c1a6b6640 10 monclient: get_auth_request con 0x7f1bf8037810 auth_method 0 2024-09-16T03:47:52.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.023+0000 7f1c0a7fc640 1 -- 172.21.15.42:0/3795288361 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f1c0400ed40 con 0x7f1c141cf770 2024-09-16T03:47:52.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.023+0000 7f1c0a7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:47:52.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.023+0000 7f1c0a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:47:52.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.023+0000 7f1c0a7fc640 1 -- 172.21.15.42:0/3795288361 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1c141cf770 msgr2=0x7f1c141cfb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:52.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.023+0000 7f1c0a7fc640 1 --2- 172.21.15.42:0/3795288361 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1c141cf770 0x7f1c141cfb50 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f1c1413bf50 tx=0x7f1c0400e1d0 comp rx=0 tx=0).stop 2024-09-16T03:47:52.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.023+0000 7f1c0a7fc640 1 -- 172.21.15.42:0/3795288361 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1c141cf770 msgr2=0x7f1c141cfb50 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:47:52.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.023+0000 7f1c0a7fc640 1 --2- 172.21.15.42:0/3795288361 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1c141cf770 0x7f1c141cfb50 secure :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f1c1413bf50 tx=0x7f1c0400e1d0 comp rx=0 tx=0).stop 2024-09-16T03:47:52.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.027+0000 7f1c0a7fc640 10 client.13865.objecter ms_handle_connect 0x7f1bf8037810 2024-09-16T03:47:52.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.027+0000 7f1c1a6b6640 1 --2- 172.21.15.42:0/3795288361 >> v2:172.21.15.154:6812/254845342 conn(0x7f1bf8037810 0x7f1bf8039cd0 secure :-1 s=READY pgs=3465 cs=0 l=1 rev1=1 crypto rx=0x7f1c0c0ae000 tx=0x7f1c0c0a7070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:47:52.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.031+0000 7f1c1c119640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:47:52.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.031+0000 7f1c1c119640 1 --2- 172.21.15.42:0/3795288361 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1c1402c650 0x7f1c1402ca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:52.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.031+0000 7f1c1c119640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:52.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.031+0000 7f1c1c119640 1 -- 172.21.15.42:0/3795288361 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f1c140a37f0 con 0x7f1c1402c650 2024-09-16T03:47:52.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.031+0000 7f1c196b4640 1 --2- 172.21.15.42:0/3795288361 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1c1402c650 0x7f1c1402ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:52.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.031+0000 7f1c196b4640 10 monclient: get_auth_request con 0x7f1c1402c650 auth_method 0 2024-09-16T03:47:52.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.031+0000 7f1c196b4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:52.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.031+0000 7f1c196b4640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:52.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.031+0000 7f1c196b4640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:47:52.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.031+0000 7f1c196b4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:47:52.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.031+0000 7f1c196b4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:47:52.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.031+0000 7f1c196b4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:47:52.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.031+0000 7f1c196b4640 10 monclient(hunting): handle_auth_done global_id 764340 payload 824 2024-09-16T03:47:52.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.031+0000 7f1c196b4640 1 -- 172.21.15.42:0/3795288361 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f1c141360f0 con 0x7f1c1402c650 2024-09-16T03:47:52.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.031+0000 7f1c0a7fc640 10 client.13865.objecter ms_handle_connect 0x7f1c1402c650 2024-09-16T03:47:52.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.031+0000 7f1c0a7fc640 10 client.13865.objecter resend_mon_ops 2024-09-16T03:47:52.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.031+0000 7f1c196b4640 1 --2- 172.21.15.42:0/3795288361 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1c1402c650 0x7f1c1402ca60 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f1c1413bf50 tx=0x7f1c040046d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:47:52.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.031+0000 7f1c0a7fc640 1 -- 172.21.15.42:0/3795288361 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c04018030 con 0x7f1c1402c650 2024-09-16T03:47:52.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.031+0000 7f1c0a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:47:52.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.031+0000 7f1c0a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:47:52.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.031+0000 7f1c0a7fc640 10 monclient: dump: 2024-09-16T03:47:52.055 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:47:52.055 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:52.055 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:52.055 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:52.055 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:47:52.055 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:52.056 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:47:52.056 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:47:52.056 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:47:52.056 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:52.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.031+0000 7f1c0a7fc640 1 -- 172.21.15.42:0/3795288361 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4546 (secure 0 0 0) 0x7f1c0401f0d0 con 0x7f1c1402c650 2024-09-16T03:47:52.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.031+0000 7f1c0a7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:47:52.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.031+0000 7f1c0a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:47:52.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.031+0000 7f1c0a7fc640 1 -- 172.21.15.42:0/3795288361 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1c1402c650 msgr2=0x7f1c1402ca60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:52.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.031+0000 7f1c0a7fc640 1 --2- 172.21.15.42:0/3795288361 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1c1402c650 0x7f1c1402ca60 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f1c1413bf50 tx=0x7f1c040046d0 comp rx=0 tx=0).stop 2024-09-16T03:47:52.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.031+0000 7f1c0a7fc640 1 -- 172.21.15.42:0/3795288361 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1c1402c650 msgr2=0x7f1c1402ca60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:47:52.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.031+0000 7f1c0a7fc640 1 --2- 172.21.15.42:0/3795288361 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1c1402c650 0x7f1c1402ca60 secure :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f1c1413bf50 tx=0x7f1c040046d0 comp rx=0 tx=0).stop 2024-09-16T03:47:52.057 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:47:52.057 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:47:52.057 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:47:52.057 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "leader", 2024-09-16T03:47:52.057 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 54, 2024-09-16T03:47:52.057 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 62823, 2024-09-16T03:47:52.057 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:47:52.057 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:47:52.057 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T03:47:52.057 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T03:47:52.057 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:47:52.057 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 5, 2024-09-16T03:47:52.058 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:47:52.058 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:47:52.058 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:47:52.058 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:47:52.058 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:47:52.058 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:47:52.058 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:47:52.058 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:47:52.058 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:47:52.058 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:47:52.058 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:47:52.058 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:47:52.058 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:47:52.058 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:47:52.059 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:47:52.059 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:47:52.059 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:47:52.059 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:47:52.059 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:47:52.059 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:47:52.059 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:47:52.059 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:47:52.059 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:47:52.059 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:47:52.059 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:47:52.059 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:47:52.059 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:47:52.059 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:47:52.060 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:47:52.060 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:52.060 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:47:52.060 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:47:52.060 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:47:52.060 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:47:52.060 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:47:52.060 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:47:52.060 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:47:52.060 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:47:52.060 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:47:52.060 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:47:52.060 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:47:52.060 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:47:52.061 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:47:52.061 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:47:52.061 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:47:52.061 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:47:52.061 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:47:52.061 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:47:52.061 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:47:52.061 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:47:52.061 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:47:52.061 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:47:52.062 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:47:52.062 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:47:52.062 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:47:52.062 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:47:52.062 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:47:52.062 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:47:52.062 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:47:52.062 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:47:52.062 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:52.062 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:47:52.062 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:52.062 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:47:52.062 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:47:52.062 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:47:52.063 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:47:52.063 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:52.063 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:47:52.063 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:47:52.063 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:47:52.063 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:52.063 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:52.063 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:47:52.063 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:47:52.063 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:47:52.063 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:47:52.063 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:47:52.063 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:52.063 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:47:52.064 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:47:52.064 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:47:52.064 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:47:52.064 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:47:52.064 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:52.064 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:52.064 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:47:52.064 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:47:52.064 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:47:52.064 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:47:52.064 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:52.064 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:47:52.064 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:47:52.064 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:47:52.065 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:52.065 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:52.065 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:47:52.065 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:47:52.065 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:47:52.065 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:47:52.065 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:47:52.065 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:52.065 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:47:52.065 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:47:52.065 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:47:52.065 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:47:52.065 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:47:52.065 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:52.066 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:52.066 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:47:52.066 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:47:52.066 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:47:52.066 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:47:52.066 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:52.066 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:47:52.066 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:47:52.066 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:47:52.066 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:52.066 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:52.066 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:47:52.066 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:47:52.066 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:47:52.067 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:47:52.067 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:47:52.067 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:52.067 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:47:52.067 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:47:52.067 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:47:52.067 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:47:52.067 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:47:52.067 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:47:52.067 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:47:52.067 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:52.067 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:47:52.067 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:47:52.067 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:52.068 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:47:52.068 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:47:52.068 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:47:52.068 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:47:52.068 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:47:52.068 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:47:52.068 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:52.068 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:47:52.068 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:47:52.068 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 3 2024-09-16T03:47:52.068 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:47:52.068 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:47:52.068 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:52.068 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:47:52.069 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:47:52.070 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.031+0000 7f1c1c119640 10 librados: watch_flush enter 2024-09-16T03:47:52.070 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.031+0000 7f1c1c119640 10 librados: watch_flush exit 2024-09-16T03:47:52.070 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.031+0000 7f1c1c119640 20 client.13865.objecter shutdown clearing up homeless session... 2024-09-16T03:47:52.070 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.031+0000 7f1c1c119640 10 client.13865.objecter successfully canceled tick 2024-09-16T03:47:52.070 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.031+0000 7f1c1c119640 1 -- 172.21.15.42:0/3795288361 >> v2:172.21.15.154:6812/254845342 conn(0x7f1bf8037810 msgr2=0x7f1bf8039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:52.071 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.031+0000 7f1c1c119640 1 --2- 172.21.15.42:0/3795288361 >> v2:172.21.15.154:6812/254845342 conn(0x7f1bf8037810 0x7f1bf8039cd0 secure :-1 s=READY pgs=3465 cs=0 l=1 rev1=1 crypto rx=0x7f1c0c0ae000 tx=0x7f1c0c0a7070 comp rx=0 tx=0).stop 2024-09-16T03:47:52.071 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.031+0000 7f1c1c119640 10 monclient: shutdown 2024-09-16T03:47:52.071 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.031+0000 7f1c1c119640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:47:52.071 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.031+0000 7f1c1c119640 1 -- 172.21.15.42:0/3795288361 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1c1406db90 msgr2=0x7f1c14139f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:52.071 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.031+0000 7f1c1c119640 1 --2- 172.21.15.42:0/3795288361 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1c1406db90 0x7f1c14139f30 secure :-1 s=READY pgs=2510 cs=0 l=1 rev1=1 crypto rx=0x7f1c1000b410 tx=0x7f1c1000b8d0 comp rx=0 tx=0).stop 2024-09-16T03:47:52.071 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.031+0000 7f1c1a6b6640 1 -- 172.21.15.42:0/3795288361 reap_dead start 2024-09-16T03:47:52.071 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.031+0000 7f1c1a6b6640 1 -- 172.21.15.42:0/3795288361 reap_dead start 2024-09-16T03:47:52.071 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.031+0000 7f1c1c119640 1 -- 172.21.15.42:0/3795288361 shutdown_connections 2024-09-16T03:47:52.071 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.031+0000 7f1c1c119640 1 -- 172.21.15.42:0/3795288361 >> 172.21.15.42:0/3795288361 conn(0x7f1c140a9fb0 msgr2=0x7f1c140a8f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:47:52.071 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.031+0000 7f1c1c119640 1 -- 172.21.15.42:0/3795288361 shutdown_connections 2024-09-16T03:47:52.071 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.031+0000 7f1c1c119640 1 -- 172.21.15.42:0/3795288361 wait complete. 2024-09-16T03:47:52.071 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.031+0000 7f1c1c119640 1 librados: shutdown 2024-09-16T03:47:52.072 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:47:52.072 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T03:47:52.072 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T03:47:52.181 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.175+0000 7fbb3985e640 10 monclient: get_monmap_and_config 2024-09-16T03:47:52.181 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.175+0000 7fbb3985e640 10 monclient: build_initial_monmap 2024-09-16T03:47:52.181 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.175+0000 7fbb3985e640 10 monclient: monmap: 2024-09-16T03:47:52.181 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:47:52.181 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:52.182 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:47:52.179841+0000 2024-09-16T03:47:52.182 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:47:52.179841+0000 2024-09-16T03:47:52.182 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:47:52.182 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:52.182 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:47:52.182 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:47:52.182 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:47:52.182 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:47:52.182 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:47:52.182 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:52.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.179+0000 7fbb3985e640 1 Processor -- start 2024-09-16T03:47:52.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.179+0000 7fbb3985e640 1 -- start start 2024-09-16T03:47:52.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.179+0000 7fbb3985e640 10 monclient: init 2024-09-16T03:47:52.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.179+0000 7fbb3985e640 10 monclient: _reopen_session rank -1 2024-09-16T03:47:52.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.179+0000 7fbb3985e640 10 monclient: _add_conns ranks=[4,1,0,3,2] 2024-09-16T03:47:52.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.179+0000 7fbb3985e640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fbb340a1330 0x7fbb3409f180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:52.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.179+0000 7fbb3985e640 10 monclient(hunting): picked mon.noname-c con 0x7fbb340a1330 addr v2:172.21.15.154:3300/0 2024-09-16T03:47:52.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.179+0000 7fbb3985e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbb340a1710 addr v1:172.21.15.154:6789/0 2024-09-16T03:47:52.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.179+0000 7fbb3985e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbb341393a0 addr v1:172.21.15.42:6789/0 2024-09-16T03:47:52.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.179+0000 7fbb3985e640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbb3409e4c0 con 0x7fbb341393a0 2024-09-16T03:47:52.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.179+0000 7fbb3985e640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbb3409f6c0 con 0x7fbb340a1710 2024-09-16T03:47:52.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.179+0000 7fbb3985e640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:52.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.179+0000 7fbb3985e640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fbb3409edf0 con 0x7fbb340a1330 2024-09-16T03:47:52.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.179+0000 7fbb3985e640 10 monclient(hunting): _renew_subs 2024-09-16T03:47:52.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.179+0000 7fbb3985e640 10 monclient(hunting): authenticate will time out at 1319.149392s 2024-09-16T03:47:52.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.179+0000 7fbb337fe640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fbb341393a0 0x7fbb3413b790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:51722/0 (socket says 172.21.15.42:51722) 2024-09-16T03:47:52.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.179+0000 7fbb337fe640 1 -- 172.21.15.42:0/3546882277 learned_addr learned my addr 172.21.15.42:0/3546882277 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:47:52.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.179+0000 7fbb32ffd640 1 --2- 172.21.15.42:0/3546882277 >> v2:172.21.15.154:3300/0 conn(0x7fbb340a1330 0x7fbb3409f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:52.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.179+0000 7fbb32ffd640 10 monclient(hunting): get_auth_request con 0x7fbb340a1330 auth_method 0 2024-09-16T03:47:52.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.179+0000 7fbb32ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:52.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.179+0000 7fbb32ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:52.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.179+0000 7fbb32ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:47:52.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.179+0000 7fbb31ffb640 1 -- 172.21.15.42:0/3546882277 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2335547777 0 0) 0x7fbb3409e4c0 con 0x7fbb341393a0 2024-09-16T03:47:52.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.179+0000 7fbb31ffb640 10 monclient(hunting): _negotiate 2024-09-16T03:47:52.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.179+0000 7fbb31ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:52.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.179+0000 7fbb31ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:47:52.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.179+0000 7fbb31ffb640 10 monclient(hunting): my global_id is 764343 2024-09-16T03:47:52.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.179+0000 7fbb31ffb640 1 -- 172.21.15.42:0/3546882277 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbb1c005740 con 0x7fbb341393a0 2024-09-16T03:47:52.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.179+0000 7fbb32ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:47:52.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.179+0000 7fbb32ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:47:52.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.179+0000 7fbb32ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:47:52.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.179+0000 7fbb31ffb640 1 -- 172.21.15.42:0/3546882277 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1470478552 0 0) 0x7fbb1c005740 con 0x7fbb341393a0 2024-09-16T03:47:52.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.179+0000 7fbb31ffb640 1 -- 172.21.15.42:0/3546882277 >> v2:172.21.15.154:3300/0 conn(0x7fbb340a1330 msgr2=0x7fbb3409f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:47:52.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.179+0000 7fbb31ffb640 1 --2- 172.21.15.42:0/3546882277 >> v2:172.21.15.154:3300/0 conn(0x7fbb340a1330 0x7fbb3409f180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:52.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.179+0000 7fbb31ffb640 1 -- 172.21.15.42:0/3546882277 >> v1:172.21.15.154:6789/0 conn(0x7fbb340a1710 legacy=0x7fbb340a1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:52.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.179+0000 7fbb31ffb640 10 monclient: _finish_hunting 0 2024-09-16T03:47:52.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.179+0000 7fbb31ffb640 1 monclient: found mon.noname-a-legacy 2024-09-16T03:47:52.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.179+0000 7fbb31ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:47:52.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.179+0000 7fbb31ffb640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.42:6789/0 2024-09-16T03:47:52.197 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.179+0000 7fbb31ffb640 1 -- 172.21.15.42:0/3546882277 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb340a3170 con 0x7fbb341393a0 2024-09-16T03:47:52.197 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.179+0000 7fbb31ffb640 10 monclient: _finish_auth 0 2024-09-16T03:47:52.197 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.179+0000 7fbb31ffb640 10 monclient: _check_auth_tickets 2024-09-16T03:47:52.197 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.179+0000 7fbb31ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:47:52.197 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.179+0000 7fbb31ffb640 1 -- 172.21.15.42:0/3546882277 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2177413734 0 0) 0x7fbb24001a00 con 0x7fbb341393a0 2024-09-16T03:47:52.197 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.179+0000 7fbb3985e640 5 monclient: authenticate success, global_id 764343 2024-09-16T03:47:52.197 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.179+0000 7fbb3985e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:47:52.197 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.179+0000 7fbb31ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:47:52.197 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.179+0000 7fbb31ffb640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T03:47:52.197 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.179+0000 7fbb31ffb640 10 monclient: dump: 2024-09-16T03:47:52.197 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:47:52.197 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:52.197 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:52.198 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:52.198 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:47:52.198 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:52.199 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:47:52.199 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:47:52.199 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:47:52.199 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:52.199 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.179+0000 7fbb31ffb640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] but i'm connected to v1:172.21.15.42:6789/0, reconnecting 2024-09-16T03:47:52.199 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.179+0000 7fbb31ffb640 10 monclient: _reopen_session rank -1 2024-09-16T03:47:52.199 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.179+0000 7fbb31ffb640 1 -- 172.21.15.42:0/3546882277 >> v1:172.21.15.42:6789/0 conn(0x7fbb341393a0 legacy=0x7fbb3413b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:52.199 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.179+0000 7fbb31ffb640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T03:47:52.199 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.179+0000 7fbb31ffb640 1 --2- 172.21.15.42:0/3546882277 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbb1c005bd0 0x7fbb1c005ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:52.199 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.179+0000 7fbb31ffb640 10 monclient(hunting): picked mon.b con 0x7fbb1c005bd0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:47:52.199 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.179+0000 7fbb31ffb640 1 --2- 172.21.15.42:0/3546882277 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbb1c006a20 0x7fbb1c006ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:52.199 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.179+0000 7fbb31ffb640 10 monclient(hunting): picked mon.c con 0x7fbb1c006a20 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:47:52.199 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.179+0000 7fbb31ffb640 1 --2- 172.21.15.42:0/3546882277 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbb1c007500 0x7fbb1c00dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:52.200 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.179+0000 7fbb31ffb640 10 monclient(hunting): picked mon.a con 0x7fbb1c007500 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:47:52.200 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.179+0000 7fbb31ffb640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:52.200 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.179+0000 7fbb31ffb640 1 -- 172.21.15.42:0/3546882277 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fbb3409edf0 con 0x7fbb1c007500 2024-09-16T03:47:52.200 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.179+0000 7fbb31ffb640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:52.200 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.179+0000 7fbb31ffb640 1 -- 172.21.15.42:0/3546882277 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fbb1c00e3d0 con 0x7fbb1c005bd0 2024-09-16T03:47:52.200 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.179+0000 7fbb31ffb640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:52.200 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.179+0000 7fbb31ffb640 1 -- 172.21.15.42:0/3546882277 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fbb1c00e5d0 con 0x7fbb1c006a20 2024-09-16T03:47:52.200 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.179+0000 7fbb31ffb640 10 monclient(hunting): _renew_subs 2024-09-16T03:47:52.200 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.179+0000 7fbb31ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:47:52.200 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.179+0000 7fbb3985e640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-16T03:47:52.200 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.179+0000 7fbb337fe640 1 --2- 172.21.15.42:0/3546882277 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbb1c007500 0x7fbb1c00dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:52.200 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.179+0000 7fbb327fc640 1 --2- 172.21.15.42:0/3546882277 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbb1c006a20 0x7fbb1c006ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:52.200 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.179+0000 7fbb337fe640 10 monclient(hunting): get_auth_request con 0x7fbb1c007500 auth_method 0 2024-09-16T03:47:52.201 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.179+0000 7fbb337fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:52.201 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.179+0000 7fbb337fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:52.201 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.179+0000 7fbb337fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:47:52.201 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.179+0000 7fbb327fc640 10 monclient(hunting): get_auth_request con 0x7fbb1c006a20 auth_method 0 2024-09-16T03:47:52.201 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.179+0000 7fbb327fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:52.201 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.179+0000 7fbb327fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:52.201 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.179+0000 7fbb327fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:47:52.201 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.179+0000 7fbb32ffd640 1 --2- 172.21.15.42:0/3546882277 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbb1c005bd0 0x7fbb1c005ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:52.201 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.179+0000 7fbb337fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:47:52.201 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.179+0000 7fbb337fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:47:52.201 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.179+0000 7fbb337fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T03:47:52.201 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.179+0000 7fbb32ffd640 10 monclient(hunting): get_auth_request con 0x7fbb1c005bd0 auth_method 0 2024-09-16T03:47:52.201 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.179+0000 7fbb32ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:52.201 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.179+0000 7fbb32ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:52.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.179+0000 7fbb32ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:47:52.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.179+0000 7fbb327fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:47:52.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.179+0000 7fbb327fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:47:52.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.179+0000 7fbb327fc640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T03:47:52.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.179+0000 7fbb32ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:47:52.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.179+0000 7fbb32ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:47:52.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.179+0000 7fbb32ffd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T03:47:52.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.179+0000 7fbb337fe640 10 monclient(hunting): handle_auth_done global_id 764343 payload 293 2024-09-16T03:47:52.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.179+0000 7fbb337fe640 1 -- 172.21.15.42:0/3546882277 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbb1c006a20 msgr2=0x7fbb1c006ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:47:52.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.179+0000 7fbb337fe640 1 --2- 172.21.15.42:0/3546882277 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbb1c006a20 0x7fbb1c006ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:52.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.179+0000 7fbb337fe640 1 -- 172.21.15.42:0/3546882277 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbb1c005bd0 msgr2=0x7fbb1c005ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:47:52.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.179+0000 7fbb337fe640 1 --2- 172.21.15.42:0/3546882277 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbb1c005bd0 0x7fbb1c005ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:52.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.179+0000 7fbb337fe640 10 monclient: _finish_hunting 0 2024-09-16T03:47:52.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.179+0000 7fbb337fe640 1 monclient: found mon.a 2024-09-16T03:47:52.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.179+0000 7fbb337fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:47:52.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.179+0000 7fbb337fe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:47:52.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.179+0000 7fbb337fe640 1 -- 172.21.15.42:0/3546882277 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb1c005740 con 0x7fbb1c007500 2024-09-16T03:47:52.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.179+0000 7fbb337fe640 10 monclient: _finish_auth 0 2024-09-16T03:47:52.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.179+0000 7fbb337fe640 10 monclient: _check_auth_tickets 2024-09-16T03:47:52.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.179+0000 7fbb337fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:47:52.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.183+0000 7fbb337fe640 1 -- 172.21.15.42:0/3546882277 reap_dead start 2024-09-16T03:47:52.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.183+0000 7fbb337fe640 1 --2- 172.21.15.42:0/3546882277 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbb1c007500 0x7fbb1c00dbe0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fbb24002240 tx=0x7fbb2401d340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=867a37af69d53cb7 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:47:52.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.183+0000 7fbb31ffb640 1 -- 172.21.15.42:0/3546882277 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb2400a4a0 con 0x7fbb1c007500 2024-09-16T03:47:52.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.183+0000 7fbb31ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:47:52.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.183+0000 7fbb31ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:47:52.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.183+0000 7fbb31ffb640 10 monclient: dump: 2024-09-16T03:47:52.204 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:47:52.204 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:52.204 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:52.204 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:52.204 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:47:52.204 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:52.204 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:47:52.204 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:47:52.204 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:47:52.204 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:52.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.183+0000 7fbb3985e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:47:52.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.183+0000 7fbb31ffb640 1 -- 172.21.15.42:0/3546882277 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb240ad650 con 0x7fbb1c007500 2024-09-16T03:47:52.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.183+0000 7fbb31ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T03:47:52.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.183+0000 7fbb3985e640 10 monclient: get_monmap_and_config success 2024-09-16T03:47:52.205 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.183+0000 7fbb3985e640 4 set_mon_vals no callback set 2024-09-16T03:47:52.205 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.183+0000 7fbb31ffb640 1 -- 172.21.15.42:0/3546882277 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb240ada40 con 0x7fbb1c007500 2024-09-16T03:47:52.205 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.183+0000 7fbb31ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:47:52.205 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.183+0000 7fbb31ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:47:52.205 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.183+0000 7fbb31ffb640 10 monclient: dump: 2024-09-16T03:47:52.205 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:47:52.205 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:52.205 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:52.205 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:52.205 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:47:52.205 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:52.205 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:47:52.205 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:47:52.206 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:47:52.206 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:52.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.183+0000 7fbb3985e640 10 monclient: shutdown 2024-09-16T03:47:52.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.183+0000 7fbb3985e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:47:52.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.183+0000 7fbb3985e640 1 -- 172.21.15.42:0/3546882277 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbb1c007500 msgr2=0x7fbb1c00dbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:52.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.183+0000 7fbb3985e640 1 --2- 172.21.15.42:0/3546882277 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbb1c007500 0x7fbb1c00dbe0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fbb24002240 tx=0x7fbb2401d340 comp rx=0 tx=0).stop 2024-09-16T03:47:52.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.183+0000 7fbb3985e640 1 -- 172.21.15.42:0/3546882277 shutdown_connections 2024-09-16T03:47:52.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.183+0000 7fbb3985e640 1 --2- 172.21.15.42:0/3546882277 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbb1c007500 0x7fbb1c00dbe0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:52.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.183+0000 7fbb3985e640 1 -- 172.21.15.42:0/3546882277 >> 172.21.15.42:0/3546882277 conn(0x7fbb340a2420 msgr2=0x7fbb340a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:47:52.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.183+0000 7fbb3985e640 1 -- 172.21.15.42:0/3546882277 shutdown_connections 2024-09-16T03:47:52.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.183+0000 7fbb3985e640 1 -- 172.21.15.42:0/3546882277 wait complete. 2024-09-16T03:47:52.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.183+0000 7fbb3985e640 10 monclient: build_initial_monmap 2024-09-16T03:47:52.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.183+0000 7fbb3985e640 10 monclient: monmap: 2024-09-16T03:47:52.207 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:47:52.207 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:47:52.207 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:47:52.207 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:47:52.207 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:47:52.208 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:52.208 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:47:52.208 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:47:52.208 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:47:52.208 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:52.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.183+0000 7fbb3985e640 1 librados: starting msgr at 2024-09-16T03:47:52.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.183+0000 7fbb3985e640 1 librados: starting objecter 2024-09-16T03:47:52.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.183+0000 7fbb3985e640 1 Processor -- start 2024-09-16T03:47:52.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.183+0000 7fbb3985e640 1 -- start start 2024-09-16T03:47:52.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.183+0000 7fbb3985e640 1 librados: setting wanted keys 2024-09-16T03:47:52.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.183+0000 7fbb3985e640 1 librados: calling monclient init 2024-09-16T03:47:52.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.183+0000 7fbb3985e640 10 monclient: init 2024-09-16T03:47:52.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.183+0000 7fbb3985e640 10 monclient: _reopen_session rank -1 2024-09-16T03:47:52.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.183+0000 7fbb3985e640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:47:52.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.183+0000 7fbb3985e640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbb1c007500 0x7fbb34130780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:52.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.183+0000 7fbb3985e640 10 monclient(hunting): picked mon.noname-c con 0x7fbb1c007500 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:47:52.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.183+0000 7fbb3985e640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbb34130cc0 0x7fbb34131120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:52.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.183+0000 7fbb3985e640 10 monclient(hunting): picked mon.noname-a con 0x7fbb34130cc0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:47:52.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.183+0000 7fbb3985e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbb341cec60 0x7fbb341d1050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:52.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.183+0000 7fbb3985e640 10 monclient(hunting): picked mon.noname-b con 0x7fbb341cec60 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:47:52.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.183+0000 7fbb3985e640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:52.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.183+0000 7fbb3985e640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fbb3409f510 con 0x7fbb34130cc0 2024-09-16T03:47:52.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.183+0000 7fbb3985e640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:52.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.183+0000 7fbb3985e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fbb3409f390 con 0x7fbb341cec60 2024-09-16T03:47:52.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.183+0000 7fbb3985e640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:52.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.183+0000 7fbb3985e640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fbb3409cd40 con 0x7fbb1c007500 2024-09-16T03:47:52.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.183+0000 7fbb3985e640 10 monclient(hunting): _renew_subs 2024-09-16T03:47:52.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.183+0000 7fbb3985e640 10 monclient(hunting): authenticate will time out at 1319.155108s 2024-09-16T03:47:52.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.183+0000 7fbb327fc640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbb34130cc0 0x7fbb34131120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:52.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.183+0000 7fbb327fc640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbb34130cc0 0x7fbb34131120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:43480/0 (socket says 172.21.15.42:43480) 2024-09-16T03:47:52.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.183+0000 7fbb32ffd640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbb1c007500 0x7fbb34130780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:52.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.183+0000 7fbb327fc640 1 -- 172.21.15.42:0/3507214947 learned_addr learned my addr 172.21.15.42:0/3507214947 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:47:52.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.183+0000 7fbb327fc640 10 monclient(hunting): get_auth_request con 0x7fbb34130cc0 auth_method 0 2024-09-16T03:47:52.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.183+0000 7fbb327fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:52.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.183+0000 7fbb327fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:52.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.183+0000 7fbb327fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:47:52.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.183+0000 7fbb32ffd640 10 monclient(hunting): get_auth_request con 0x7fbb1c007500 auth_method 0 2024-09-16T03:47:52.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.183+0000 7fbb32ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:52.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.183+0000 7fbb32ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:52.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.183+0000 7fbb32ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:47:52.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.183+0000 7fbb327fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:47:52.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.183+0000 7fbb327fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:47:52.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.183+0000 7fbb327fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:47:52.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.183+0000 7fbb337fe640 1 --2- 172.21.15.42:0/3507214947 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbb341cec60 0x7fbb341d1050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:52.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.183+0000 7fbb337fe640 10 monclient(hunting): get_auth_request con 0x7fbb341cec60 auth_method 0 2024-09-16T03:47:52.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.183+0000 7fbb337fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:52.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.183+0000 7fbb337fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:52.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.183+0000 7fbb337fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:47:52.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.183+0000 7fbb32ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:47:52.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.183+0000 7fbb32ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:47:52.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.183+0000 7fbb32ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:47:52.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.183+0000 7fbb327fc640 10 monclient(hunting): handle_auth_done global_id 764346 payload 824 2024-09-16T03:47:52.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.183+0000 7fbb327fc640 1 -- 172.21.15.42:0/3507214947 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbb1c007500 msgr2=0x7fbb34130780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:52.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.183+0000 7fbb327fc640 1 --2- 172.21.15.42:0/3507214947 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbb1c007500 0x7fbb34130780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:52.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.183+0000 7fbb327fc640 1 -- 172.21.15.42:0/3507214947 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbb341cec60 msgr2=0x7fbb341d1050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:52.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.183+0000 7fbb327fc640 1 --2- 172.21.15.42:0/3507214947 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbb341cec60 0x7fbb341d1050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:52.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.183+0000 7fbb327fc640 10 monclient: _finish_hunting 0 2024-09-16T03:47:52.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.183+0000 7fbb327fc640 1 monclient: found mon.noname-a 2024-09-16T03:47:52.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.183+0000 7fbb327fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:47:52.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.183+0000 7fbb327fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:47:52.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.183+0000 7fbb327fc640 1 -- 172.21.15.42:0/3507214947 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb34073a30 con 0x7fbb34130cc0 2024-09-16T03:47:52.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.183+0000 7fbb337fe640 1 --2- 172.21.15.42:0/3507214947 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbb341cec60 0x7fbb341d1050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T03:47:52.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.183+0000 7fbb327fc640 1 --2- 172.21.15.42:0/3507214947 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbb34130cc0 0x7fbb34131120 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fbb2800f7f0 tx=0x7fbb2800fbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:47:52.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.183+0000 7fbb1bfff640 10 client.?.objecter ms_handle_connect 0x7fbb34130cc0 2024-09-16T03:47:52.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.183+0000 7fbb1bfff640 10 client.?.objecter resend_mon_ops 2024-09-16T03:47:52.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.183+0000 7fbb1bfff640 1 -- 172.21.15.42:0/3507214947 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb280050d0 con 0x7fbb34130cc0 2024-09-16T03:47:52.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.183+0000 7fbb1bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:47:52.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.187+0000 7fbb1bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:47:52.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.187+0000 7fbb1bfff640 10 monclient: dump: 2024-09-16T03:47:52.213 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:47:52.213 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:52.213 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:52.213 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:52.213 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:47:52.213 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:52.213 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:47:52.214 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:47:52.214 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:47:52.214 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:52.214 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.187+0000 7fbb1bfff640 10 monclient: _finish_auth 0 2024-09-16T03:47:52.214 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.187+0000 7fbb1bfff640 10 monclient: _check_auth_tickets 2024-09-16T03:47:52.214 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.187+0000 7fbb1bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:47:52.214 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.187+0000 7fbb1bfff640 1 -- 172.21.15.42:0/3507214947 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb28018070 con 0x7fbb34130cc0 2024-09-16T03:47:52.214 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.187+0000 7fbb3985e640 5 monclient: authenticate success, global_id 764346 2024-09-16T03:47:52.214 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.187+0000 7fbb3985e640 10 monclient: _renew_subs 2024-09-16T03:47:52.214 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.187+0000 7fbb3985e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:47:52.214 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.187+0000 7fbb3985e640 1 -- 172.21.15.42:0/3507214947 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb34133000 con 0x7fbb34130cc0 2024-09-16T03:47:52.214 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.187+0000 7fbb1bfff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:47:52.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.187+0000 7fbb1bfff640 1 -- 172.21.15.42:0/3507214947 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb28005490 con 0x7fbb34130cc0 2024-09-16T03:47:52.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.187+0000 7fbb1bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:47:52.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.187+0000 7fbb1bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:47:52.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.187+0000 7fbb1bfff640 10 monclient: dump: 2024-09-16T03:47:52.215 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:47:52.215 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:52.215 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:52.215 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:52.216 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:47:52.216 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:52.216 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:47:52.216 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:47:52.216 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:47:52.216 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:52.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.187+0000 7fbb30ff9640 4 set_mon_vals no callback set 2024-09-16T03:47:52.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.187+0000 7fbb3985e640 10 client.764346.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:47:52.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.187+0000 7fbb3985e640 10 monclient: _renew_subs 2024-09-16T03:47:52.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.187+0000 7fbb3985e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:47:52.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.187+0000 7fbb3985e640 1 -- 172.21.15.42:0/3507214947 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb34137ec0 con 0x7fbb34130cc0 2024-09-16T03:47:52.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.187+0000 7fbb3985e640 1 librados: init done 2024-09-16T03:47:52.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.187+0000 7fbb3985e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:47:52.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.187+0000 7fbb3985e640 1 --2- 172.21.15.42:0/3507214947 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbb34133240 0x7fbb3413a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:52.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.187+0000 7fbb3985e640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:52.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.187+0000 7fbb3985e640 1 -- 172.21.15.42:0/3507214947 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fbb340a5380 con 0x7fbb34133240 2024-09-16T03:47:52.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.187+0000 7fbb32ffd640 1 --2- 172.21.15.42:0/3507214947 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbb34133240 0x7fbb3413a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:52.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.187+0000 7fbb1bfff640 1 -- 172.21.15.42:0/3507214947 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fbb2800d730 con 0x7fbb34130cc0 2024-09-16T03:47:52.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.187+0000 7fbb32ffd640 10 monclient: get_auth_request con 0x7fbb34133240 auth_method 0 2024-09-16T03:47:52.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.187+0000 7fbb32ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:52.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.187+0000 7fbb32ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:52.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.187+0000 7fbb32ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:47:52.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.187+0000 7fbb32ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:47:52.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.187+0000 7fbb32ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:47:52.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.187+0000 7fbb32ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:47:52.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.187+0000 7fbb1bfff640 1 --2- 172.21.15.42:0/3507214947 >> v2:172.21.15.154:6812/254845342 conn(0x7fbb200379b0 0x7fbb20039e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:52.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.187+0000 7fbb32ffd640 10 monclient(hunting): handle_auth_done global_id 13871 payload 824 2024-09-16T03:47:52.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.187+0000 7fbb1bfff640 1 -- 172.21.15.42:0/3507214947 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(181..181 src has 1..181) ==== 6991+0+0 (secure 0 0 0) 0x7fbb28049190 con 0x7fbb34130cc0 2024-09-16T03:47:52.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.187+0000 7fbb1bfff640 10 client.764346.objecter ms_dispatch 0x7fbb34000ba0 osd_map(181..181 src has 1..181) 2024-09-16T03:47:52.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.187+0000 7fbb1bfff640 3 client.764346.objecter handle_osd_map got epochs [181,181] > 0 2024-09-16T03:47:52.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.187+0000 7fbb1bfff640 3 client.764346.objecter handle_osd_map decoding full epoch 181 2024-09-16T03:47:52.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.187+0000 7fbb32ffd640 1 -- 172.21.15.42:0/3507214947 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbb340a1b20 con 0x7fbb34133240 2024-09-16T03:47:52.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.187+0000 7fbb1bfff640 20 client.764346.objecter dump_active .. 0 homeless 2024-09-16T03:47:52.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.187+0000 7fbb1bfff640 10 client.764346.objecter ms_handle_connect 0x7fbb34133240 2024-09-16T03:47:52.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.187+0000 7fbb1bfff640 10 client.764346.objecter resend_mon_ops 2024-09-16T03:47:52.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.187+0000 7fbb32ffd640 1 --2- 172.21.15.42:0/3507214947 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbb34133240 0x7fbb3413a4b0 secure :-1 s=READY pgs=2511 cs=0 l=1 rev1=1 crypto rx=0x7fbb2c014a10 tx=0x7fbb2c014ed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:47:52.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.187+0000 7fbb337fe640 1 --2- 172.21.15.42:0/3507214947 >> v2:172.21.15.154:6812/254845342 conn(0x7fbb200379b0 0x7fbb20039e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:52.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.187+0000 7fbb337fe640 10 monclient: get_auth_request con 0x7fbb200379b0 auth_method 0 2024-09-16T03:47:52.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.187+0000 7fbb1bfff640 1 -- 172.21.15.42:0/3507214947 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb2c022070 con 0x7fbb34133240 2024-09-16T03:47:52.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.187+0000 7fbb1bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:47:52.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.187+0000 7fbb1bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:47:52.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.187+0000 7fbb1bfff640 10 monclient: dump: 2024-09-16T03:47:52.220 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:47:52.220 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:52.220 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:52.220 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:52.220 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:47:52.220 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:52.220 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:47:52.220 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:47:52.220 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:47:52.220 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:52.221 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.187+0000 7fbb1bfff640 1 -- 172.21.15.42:0/3507214947 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fbb2c012cb0 con 0x7fbb34133240 2024-09-16T03:47:52.221 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.187+0000 7fbb1bfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:47:52.221 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.187+0000 7fbb1bfff640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:47:52.221 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.187+0000 7fbb1bfff640 1 -- 172.21.15.42:0/3507214947 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbb34133240 msgr2=0x7fbb3413a4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:52.221 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.187+0000 7fbb1bfff640 1 --2- 172.21.15.42:0/3507214947 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbb34133240 0x7fbb3413a4b0 secure :-1 s=READY pgs=2511 cs=0 l=1 rev1=1 crypto rx=0x7fbb2c014a10 tx=0x7fbb2c014ed0 comp rx=0 tx=0).stop 2024-09-16T03:47:52.221 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.187+0000 7fbb1bfff640 1 -- 172.21.15.42:0/3507214947 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbb34133240 msgr2=0x7fbb3413a4b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:47:52.221 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.187+0000 7fbb1bfff640 1 --2- 172.21.15.42:0/3507214947 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbb34133240 0x7fbb3413a4b0 secure :-1 s=CLOSED pgs=2511 cs=0 l=1 rev1=1 crypto rx=0x7fbb2c014a10 tx=0x7fbb2c014ed0 comp rx=0 tx=0).stop 2024-09-16T03:47:52.221 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.187+0000 7fbb337fe640 1 --2- 172.21.15.42:0/3507214947 >> v2:172.21.15.154:6812/254845342 conn(0x7fbb200379b0 0x7fbb20039e70 secure :-1 s=READY pgs=3466 cs=0 l=1 rev1=1 crypto rx=0x7fbb1c00e3a0 tx=0x7fbb24009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:47:52.221 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.187+0000 7fbb1bfff640 10 client.764346.objecter ms_handle_connect 0x7fbb200379b0 2024-09-16T03:47:52.225 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.191+0000 7fbb3985e640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:47:52.225 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.191+0000 7fbb3985e640 1 --2- 172.21.15.42:0/3507214947 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbb3413a9f0 0x7fbb3413ae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:52.225 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.191+0000 7fbb3985e640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:52.225 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.191+0000 7fbb3985e640 1 -- 172.21.15.42:0/3507214947 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fbb34132170 con 0x7fbb3413a9f0 2024-09-16T03:47:52.225 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.195+0000 7fbb32ffd640 1 --2- 172.21.15.42:0/3507214947 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbb3413a9f0 0x7fbb3413ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:52.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.195+0000 7fbb32ffd640 10 monclient: get_auth_request con 0x7fbb3413a9f0 auth_method 0 2024-09-16T03:47:52.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.195+0000 7fbb32ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:52.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.195+0000 7fbb32ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:52.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.195+0000 7fbb32ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:47:52.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.195+0000 7fbb32ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:47:52.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.195+0000 7fbb32ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:47:52.227 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.195+0000 7fbb32ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:47:52.227 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.195+0000 7fbb32ffd640 10 monclient(hunting): handle_auth_done global_id 13874 payload 824 2024-09-16T03:47:52.227 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.195+0000 7fbb32ffd640 1 -- 172.21.15.42:0/3507214947 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fbb34133620 con 0x7fbb3413a9f0 2024-09-16T03:47:52.227 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.195+0000 7fbb1bfff640 10 client.764346.objecter ms_handle_connect 0x7fbb3413a9f0 2024-09-16T03:47:52.227 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.195+0000 7fbb1bfff640 10 client.764346.objecter resend_mon_ops 2024-09-16T03:47:52.227 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.195+0000 7fbb32ffd640 1 --2- 172.21.15.42:0/3507214947 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbb3413a9f0 0x7fbb3413ae00 secure :-1 s=READY pgs=2512 cs=0 l=1 rev1=1 crypto rx=0x7fbb2c014a10 tx=0x7fbb2c004740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:47:52.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.195+0000 7fbb1bfff640 1 -- 172.21.15.42:0/3507214947 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb2c022030 con 0x7fbb3413a9f0 2024-09-16T03:47:52.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.195+0000 7fbb1bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:47:52.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.195+0000 7fbb1bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:47:52.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.195+0000 7fbb1bfff640 10 monclient: dump: 2024-09-16T03:47:52.228 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:47:52.228 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:52.229 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:52.229 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:52.229 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:47:52.229 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:52.229 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:47:52.229 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:47:52.230 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:47:52.230 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:52.230 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.195+0000 7fbb1bfff640 1 -- 172.21.15.42:0/3507214947 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4879 (secure 0 0 0) 0x7fbb2c025d00 con 0x7fbb3413a9f0 2024-09-16T03:47:52.230 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.195+0000 7fbb1bfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:47:52.230 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.195+0000 7fbb1bfff640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:47:52.230 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.195+0000 7fbb1bfff640 1 -- 172.21.15.42:0/3507214947 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbb3413a9f0 msgr2=0x7fbb3413ae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:52.230 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.195+0000 7fbb1bfff640 1 --2- 172.21.15.42:0/3507214947 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbb3413a9f0 0x7fbb3413ae00 secure :-1 s=READY pgs=2512 cs=0 l=1 rev1=1 crypto rx=0x7fbb2c014a10 tx=0x7fbb2c004740 comp rx=0 tx=0).stop 2024-09-16T03:47:52.230 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.195+0000 7fbb1bfff640 1 -- 172.21.15.42:0/3507214947 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbb3413a9f0 msgr2=0x7fbb3413ae00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:47:52.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.195+0000 7fbb1bfff640 1 --2- 172.21.15.42:0/3507214947 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbb3413a9f0 0x7fbb3413ae00 secure :-1 s=CLOSED pgs=2512 cs=0 l=1 rev1=1 crypto rx=0x7fbb2c014a10 tx=0x7fbb2c004740 comp rx=0 tx=0).stop 2024-09-16T03:47:52.231 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:47:52.231 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:47:52.231 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:47:52.231 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T03:47:52.231 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 54, 2024-09-16T03:47:52.231 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 351566, 2024-09-16T03:47:52.232 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:47:52.232 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:47:52.232 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T03:47:52.232 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T03:47:52.232 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:47:52.232 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 5, 2024-09-16T03:47:52.232 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:47:52.232 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:47:52.232 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:47:52.232 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:47:52.232 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:47:52.232 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:47:52.233 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:47:52.233 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:47:52.233 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:47:52.233 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:47:52.233 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:47:52.233 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:47:52.233 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:47:52.233 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:47:52.233 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:47:52.233 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:47:52.233 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:47:52.233 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:47:52.233 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:47:52.234 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:47:52.234 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:47:52.234 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:47:52.234 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:47:52.234 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:47:52.234 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:47:52.234 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:47:52.234 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:47:52.234 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:47:52.234 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:47:52.234 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:52.234 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:47:52.234 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:47:52.234 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:47:52.234 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:47:52.235 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:47:52.235 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:47:52.235 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:47:52.235 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:47:52.235 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:47:52.235 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:47:52.235 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:47:52.235 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:47:52.235 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:47:52.235 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:47:52.235 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:47:52.235 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:47:52.235 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:47:52.235 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:47:52.235 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:47:52.236 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:47:52.236 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:47:52.236 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:47:52.236 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:47:52.236 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:47:52.236 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:47:52.236 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:47:52.236 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:47:52.236 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:47:52.236 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:47:52.236 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:47:52.236 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:52.236 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:47:52.236 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:52.236 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:47:52.237 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:47:52.237 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:47:52.237 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:47:52.237 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:52.237 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:47:52.237 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:47:52.237 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:47:52.237 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:52.237 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:52.238 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:47:52.238 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:47:52.238 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:47:52.238 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:47:52.238 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:47:52.238 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:52.238 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:47:52.238 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:47:52.238 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:47:52.238 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:47:52.238 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:47:52.238 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:52.239 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:52.239 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:47:52.239 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:47:52.239 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:47:52.239 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:47:52.239 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:52.239 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:47:52.239 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:47:52.239 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:47:52.239 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:52.239 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:52.239 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:47:52.239 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:47:52.239 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:47:52.240 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:47:52.240 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:47:52.240 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:52.240 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:47:52.240 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:47:52.240 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:47:52.240 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:47:52.240 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:47:52.240 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:52.240 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:52.240 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:47:52.240 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:47:52.240 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:47:52.240 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:47:52.240 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:52.240 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:47:52.241 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:47:52.241 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:47:52.241 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:52.241 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:52.241 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:47:52.241 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:47:52.241 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:47:52.241 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:47:52.241 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:47:52.241 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:52.241 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:47:52.241 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:47:52.241 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:47:52.241 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:47:52.242 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:47:52.242 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:47:52.242 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:47:52.242 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:52.242 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:47:52.242 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:47:52.242 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:52.242 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:47:52.242 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:47:52.242 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:47:52.242 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:47:52.242 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:47:52.242 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T03:47:52.242 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:52.242 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:47:52.243 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:47:52.243 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 6 2024-09-16T03:47:52.243 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:47:52.243 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:47:52.243 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:47:52.243 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:52.243 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:47:52.243 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:47:52.243 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 7 2024-09-16T03:47:52.243 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:47:52.243 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:47:52.243 INFO:teuthology.orchestra.run.smithi042.stdout: "mgr": [ 2024-09-16T03:47:52.244 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:52.244 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:47:52.244 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:47:52.244 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:47:52.244 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:47:52.244 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:47:52.244 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:52.244 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:47:52.244 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:47:52.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.195+0000 7fbb3985e640 10 librados: watch_flush enter 2024-09-16T03:47:52.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.195+0000 7fbb3985e640 10 librados: watch_flush exit 2024-09-16T03:47:52.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.195+0000 7fbb3985e640 20 client.764346.objecter shutdown clearing up homeless session... 2024-09-16T03:47:52.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.195+0000 7fbb3985e640 10 client.764346.objecter successfully canceled tick 2024-09-16T03:47:52.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.195+0000 7fbb3985e640 1 -- 172.21.15.42:0/3507214947 >> v2:172.21.15.154:6812/254845342 conn(0x7fbb200379b0 msgr2=0x7fbb20039e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:52.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.195+0000 7fbb3985e640 1 --2- 172.21.15.42:0/3507214947 >> v2:172.21.15.154:6812/254845342 conn(0x7fbb200379b0 0x7fbb20039e70 secure :-1 s=READY pgs=3466 cs=0 l=1 rev1=1 crypto rx=0x7fbb1c00e3a0 tx=0x7fbb24009040 comp rx=0 tx=0).stop 2024-09-16T03:47:52.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.195+0000 7fbb3985e640 10 monclient: shutdown 2024-09-16T03:47:52.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.195+0000 7fbb3985e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:47:52.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.195+0000 7fbb3985e640 1 -- 172.21.15.42:0/3507214947 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbb34130cc0 msgr2=0x7fbb34131120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:52.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.195+0000 7fbb3985e640 1 --2- 172.21.15.42:0/3507214947 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbb34130cc0 0x7fbb34131120 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fbb2800f7f0 tx=0x7fbb2800fbf0 comp rx=0 tx=0).stop 2024-09-16T03:47:52.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.195+0000 7fbb337fe640 1 -- 172.21.15.42:0/3507214947 reap_dead start 2024-09-16T03:47:52.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.195+0000 7fbb337fe640 1 -- 172.21.15.42:0/3507214947 reap_dead start 2024-09-16T03:47:52.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.195+0000 7fbb3985e640 1 -- 172.21.15.42:0/3507214947 shutdown_connections 2024-09-16T03:47:52.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.195+0000 7fbb3985e640 1 -- 172.21.15.42:0/3507214947 >> 172.21.15.42:0/3507214947 conn(0x7fbb340a2420 msgr2=0x7fbb340a3720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:47:52.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.195+0000 7fbb3985e640 1 -- 172.21.15.42:0/3507214947 shutdown_connections 2024-09-16T03:47:52.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.195+0000 7fbb3985e640 1 -- 172.21.15.42:0/3507214947 wait complete. 2024-09-16T03:47:52.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.195+0000 7fbb3985e640 1 librados: shutdown 2024-09-16T03:47:52.248 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:47:52.248 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T03:47:52.248 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T03:47:52.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.375+0000 7f41f095f640 10 monclient: get_monmap_and_config 2024-09-16T03:47:52.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.375+0000 7f41f095f640 10 monclient: build_initial_monmap 2024-09-16T03:47:52.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.375+0000 7f41f095f640 10 monclient: monmap: 2024-09-16T03:47:52.377 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:47:52.377 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:52.378 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:47:52.376442+0000 2024-09-16T03:47:52.378 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:47:52.376442+0000 2024-09-16T03:47:52.378 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:47:52.378 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:52.378 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:47:52.378 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:47:52.378 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:47:52.378 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:47:52.378 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:47:52.378 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:52.380 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.375+0000 7f41f095f640 1 Processor -- start 2024-09-16T03:47:52.380 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.375+0000 7f41f095f640 1 -- start start 2024-09-16T03:47:52.380 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.375+0000 7f41f095f640 10 monclient: init 2024-09-16T03:47:52.380 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.375+0000 7f41f095f640 10 monclient: _reopen_session rank -1 2024-09-16T03:47:52.380 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.375+0000 7f41f095f640 10 monclient: _add_conns ranks=[2,1,3,4,0] 2024-09-16T03:47:52.380 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.375+0000 7f41f095f640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f41ec0a3450 0x7f41ec0a3830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:52.380 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.375+0000 7f41f095f640 10 monclient(hunting): picked mon.noname-b con 0x7f41ec0a3450 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:47:52.380 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.375+0000 7f41f095f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f41ec0a0ea0 addr v1:172.21.15.154:6789/0 2024-09-16T03:47:52.380 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.375+0000 7f41f095f640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f41ec0a3d70 0x7f41ec137930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:52.380 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.375+0000 7f41f095f640 10 monclient(hunting): picked mon.noname-a con 0x7f41ec0a3d70 addr v2:172.21.15.42:3300/0 2024-09-16T03:47:52.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.375+0000 7f41f095f640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f41ec0a00e0 con 0x7f41ec0a0ea0 2024-09-16T03:47:52.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.375+0000 7f41f095f640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:52.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.375+0000 7f41f095f640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f41ec0a1b40 con 0x7f41ec0a3d70 2024-09-16T03:47:52.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.375+0000 7f41f095f640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:52.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.375+0000 7f41f095f640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f41ec0a1d90 con 0x7f41ec0a3450 2024-09-16T03:47:52.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.375+0000 7f41f095f640 10 monclient(hunting): _renew_subs 2024-09-16T03:47:52.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.375+0000 7f41f095f640 10 monclient(hunting): authenticate will time out at 1319.345761s 2024-09-16T03:47:52.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.375+0000 7f41ea59c640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f41ec0a3450 0x7f41ec0a3830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:52.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.375+0000 7f41ea59c640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f41ec0a3450 0x7f41ec0a3830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:59590/0 (socket says 172.21.15.42:59590) 2024-09-16T03:47:52.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.375+0000 7f41ea59c640 1 -- 172.21.15.42:0/3824947070 learned_addr learned my addr 172.21.15.42:0/3824947070 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:47:52.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.375+0000 7f41ea59c640 10 monclient(hunting): get_auth_request con 0x7f41ec0a3450 auth_method 0 2024-09-16T03:47:52.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.375+0000 7f41ea59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:52.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.375+0000 7f41ea59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:52.382 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.375+0000 7f41ea59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:47:52.382 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.375+0000 7f41ead9d640 1 --2- 172.21.15.42:0/3824947070 >> v2:172.21.15.42:3300/0 conn(0x7f41ec0a3d70 0x7f41ec137930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:52.382 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.375+0000 7f41ead9d640 10 monclient(hunting): get_auth_request con 0x7f41ec0a3d70 auth_method 0 2024-09-16T03:47:52.382 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.375+0000 7f41ead9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:52.382 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.375+0000 7f41ead9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:52.382 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.375+0000 7f41ead9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:47:52.382 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.375+0000 7f41ea59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:47:52.382 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.375+0000 7f41ea59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:47:52.382 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.375+0000 7f41ea59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:47:52.382 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.375+0000 7f41ead9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:47:52.382 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.375+0000 7f41ead9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:47:52.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.375+0000 7f41ead9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:47:52.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.375+0000 7f41ea59c640 10 monclient(hunting): handle_auth_done global_id 13877 payload 274 2024-09-16T03:47:52.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.375+0000 7f41ea59c640 1 -- 172.21.15.42:0/3824947070 >> v2:172.21.15.42:3300/0 conn(0x7f41ec0a3d70 msgr2=0x7f41ec137930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:47:52.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.375+0000 7f41ea59c640 1 --2- 172.21.15.42:0/3824947070 >> v2:172.21.15.42:3300/0 conn(0x7f41ec0a3d70 0x7f41ec137930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:52.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.375+0000 7f41ea59c640 1 -- 172.21.15.42:0/3824947070 >> v1:172.21.15.154:6789/0 conn(0x7f41ec0a0ea0 legacy=0x7f41ec137220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:47:52.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.375+0000 7f41ea59c640 10 monclient: _finish_hunting 0 2024-09-16T03:47:52.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.375+0000 7f41ea59c640 1 monclient: found mon.noname-b 2024-09-16T03:47:52.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.375+0000 7f41ea59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:47:52.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.375+0000 7f41ea59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T03:47:52.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.375+0000 7f41ea59c640 1 -- 172.21.15.42:0/3824947070 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41ec09fea0 con 0x7f41ec0a3450 2024-09-16T03:47:52.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.375+0000 7f41ead9d640 1 --2- 172.21.15.42:0/3824947070 >> v2:172.21.15.42:3300/0 conn(0x7f41ec0a3d70 0x7f41ec137930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:47:52.389 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.379+0000 7f41ea59c640 1 --2- 172.21.15.42:0/3824947070 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f41ec0a3450 0x7f41ec0a3830 secure :-1 s=READY pgs=2513 cs=0 l=1 rev1=1 crypto rx=0x7f41dc009870 tx=0x7f41dc01c3e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ed20e6abe1907717 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:47:52.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.379+0000 7f41e959a640 1 -- 172.21.15.42:0/3824947070 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41dc0b0070 con 0x7f41ec0a3450 2024-09-16T03:47:52.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.379+0000 7f41e959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:47:52.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.379+0000 7f41e959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:47:52.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.379+0000 7f41e959a640 10 monclient: dump: 2024-09-16T03:47:52.390 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:47:52.390 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:52.390 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:52.390 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:52.390 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:47:52.390 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:52.390 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:47:52.390 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:47:52.391 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:47:52.391 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:52.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.379+0000 7f41e959a640 10 monclient: _finish_auth 0 2024-09-16T03:47:52.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.379+0000 7f41e959a640 10 monclient: _check_auth_tickets 2024-09-16T03:47:52.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.379+0000 7f41e959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:47:52.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.379+0000 7f41e959a640 1 -- 172.21.15.42:0/3824947070 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f41dc0ab600 con 0x7f41ec0a3450 2024-09-16T03:47:52.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.379+0000 7f41e959a640 10 monclient: handle_config config(0 keys) 2024-09-16T03:47:52.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.379+0000 7f41e959a640 1 -- 172.21.15.42:0/3824947070 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41dc0ab9d0 con 0x7f41ec0a3450 2024-09-16T03:47:52.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.379+0000 7f41f095f640 5 monclient: authenticate success, global_id 13877 2024-09-16T03:47:52.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.379+0000 7f41f095f640 10 monclient: get_monmap_and_config success 2024-09-16T03:47:52.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.379+0000 7f41f095f640 4 set_mon_vals no callback set 2024-09-16T03:47:52.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.379+0000 7f41e959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:47:52.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.379+0000 7f41e959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:47:52.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.379+0000 7f41e959a640 10 monclient: dump: 2024-09-16T03:47:52.392 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:47:52.392 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:52.392 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:52.392 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:52.392 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:47:52.392 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:52.392 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:47:52.392 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:47:52.392 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:47:52.392 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:52.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.379+0000 7f41f095f640 10 monclient: shutdown 2024-09-16T03:47:52.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.379+0000 7f41f095f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:47:52.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.379+0000 7f41f095f640 1 -- 172.21.15.42:0/3824947070 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f41ec0a3450 msgr2=0x7f41ec0a3830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:52.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.379+0000 7f41f095f640 1 --2- 172.21.15.42:0/3824947070 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f41ec0a3450 0x7f41ec0a3830 secure :-1 s=READY pgs=2513 cs=0 l=1 rev1=1 crypto rx=0x7f41dc009870 tx=0x7f41dc01c3e0 comp rx=0 tx=0).stop 2024-09-16T03:47:52.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.379+0000 7f41f095f640 1 -- 172.21.15.42:0/3824947070 shutdown_connections 2024-09-16T03:47:52.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.379+0000 7f41f095f640 1 --2- 172.21.15.42:0/3824947070 >> v2:172.21.15.42:3300/0 conn(0x7f41ec0a3d70 0x7f41ec137930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:52.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.379+0000 7f41f095f640 1 --2- 172.21.15.42:0/3824947070 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f41ec0a3450 0x7f41ec0a3830 unknown :-1 s=CLOSED pgs=2513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:52.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.379+0000 7f41f095f640 1 -- 172.21.15.42:0/3824947070 >> 172.21.15.42:0/3824947070 conn(0x7f41ec0756e0 msgr2=0x7f41ec075af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:47:52.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.379+0000 7f41f095f640 1 -- 172.21.15.42:0/3824947070 shutdown_connections 2024-09-16T03:47:52.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.379+0000 7f41f095f640 1 -- 172.21.15.42:0/3824947070 wait complete. 2024-09-16T03:47:52.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.379+0000 7f41f095f640 10 monclient: build_initial_monmap 2024-09-16T03:47:52.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.379+0000 7f41f095f640 10 monclient: monmap: 2024-09-16T03:47:52.393 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:47:52.393 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:47:52.393 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:47:52.393 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:47:52.393 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:47:52.394 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:52.394 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:47:52.394 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:47:52.394 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:47:52.394 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:52.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.379+0000 7f41f095f640 1 librados: starting msgr at 2024-09-16T03:47:52.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.379+0000 7f41f095f640 1 librados: starting objecter 2024-09-16T03:47:52.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.379+0000 7f41f095f640 1 Processor -- start 2024-09-16T03:47:52.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.379+0000 7f41f095f640 1 -- start start 2024-09-16T03:47:52.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.379+0000 7f41f095f640 1 librados: setting wanted keys 2024-09-16T03:47:52.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.379+0000 7f41f095f640 1 librados: calling monclient init 2024-09-16T03:47:52.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.379+0000 7f41f095f640 10 monclient: init 2024-09-16T03:47:52.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.379+0000 7f41f095f640 10 monclient: _reopen_session rank -1 2024-09-16T03:47:52.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.379+0000 7f41f095f640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:47:52.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.379+0000 7f41f095f640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f41ec0a0ea0 0x7f41ec1cf180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:52.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.379+0000 7f41f095f640 10 monclient(hunting): picked mon.noname-a con 0x7f41ec0a0ea0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:47:52.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.379+0000 7f41f095f640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f41ec0a3450 0x7f41ec1372a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:52.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.379+0000 7f41f095f640 10 monclient(hunting): picked mon.noname-c con 0x7f41ec0a3450 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:47:52.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.379+0000 7f41f095f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f41ec0a3d70 0x7f41ec11e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:52.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.379+0000 7f41f095f640 10 monclient(hunting): picked mon.noname-b con 0x7f41ec0a3d70 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:47:52.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.379+0000 7f41f095f640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:52.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.379+0000 7f41f095f640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f41ec137ff0 con 0x7f41ec0a0ea0 2024-09-16T03:47:52.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.379+0000 7f41f095f640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:52.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.379+0000 7f41f095f640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f41ec137e70 con 0x7f41ec0a3d70 2024-09-16T03:47:52.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.379+0000 7f41f095f640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:52.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.379+0000 7f41f095f640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f41ec141ca0 con 0x7f41ec0a3450 2024-09-16T03:47:52.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.379+0000 7f41f095f640 10 monclient(hunting): _renew_subs 2024-09-16T03:47:52.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.379+0000 7f41f095f640 10 monclient(hunting): authenticate will time out at 1319.350884s 2024-09-16T03:47:52.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.379+0000 7f41e9d9b640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f41ec0a3450 0x7f41ec1372a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:52.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.379+0000 7f41ea59c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f41ec0a0ea0 0x7f41ec1cf180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:52.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.379+0000 7f41e9d9b640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f41ec0a3450 0x7f41ec1372a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:59592/0 (socket says 172.21.15.42:59592) 2024-09-16T03:47:52.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.379+0000 7f41e9d9b640 1 -- 172.21.15.42:0/3736970475 learned_addr learned my addr 172.21.15.42:0/3736970475 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:47:52.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.379+0000 7f41e9d9b640 10 monclient(hunting): get_auth_request con 0x7f41ec0a3450 auth_method 0 2024-09-16T03:47:52.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.379+0000 7f41e9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:52.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.379+0000 7f41e9d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:52.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.379+0000 7f41e9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:47:52.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.379+0000 7f41ea59c640 10 monclient(hunting): get_auth_request con 0x7f41ec0a0ea0 auth_method 0 2024-09-16T03:47:52.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.379+0000 7f41ea59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:52.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.379+0000 7f41ea59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:52.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.379+0000 7f41ea59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:47:52.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.379+0000 7f41ea59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:47:52.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.379+0000 7f41ea59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:47:52.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.379+0000 7f41ea59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:47:52.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.379+0000 7f41e9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:47:52.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.379+0000 7f41e9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:47:52.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.379+0000 7f41e9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:47:52.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.379+0000 7f41ead9d640 1 --2- 172.21.15.42:0/3736970475 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f41ec0a3d70 0x7f41ec11e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:52.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.379+0000 7f41ead9d640 10 monclient(hunting): get_auth_request con 0x7f41ec0a3d70 auth_method 0 2024-09-16T03:47:52.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.379+0000 7f41ead9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:52.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.379+0000 7f41ead9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:52.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.379+0000 7f41ead9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:47:52.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.379+0000 7f41ea59c640 10 monclient(hunting): handle_auth_done global_id 764352 payload 824 2024-09-16T03:47:52.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.379+0000 7f41ea59c640 1 -- 172.21.15.42:0/3736970475 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f41ec0a3450 msgr2=0x7f41ec1372a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:52.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.379+0000 7f41ea59c640 1 --2- 172.21.15.42:0/3736970475 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f41ec0a3450 0x7f41ec1372a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:52.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.379+0000 7f41ea59c640 1 -- 172.21.15.42:0/3736970475 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f41ec0a3d70 msgr2=0x7f41ec11e090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:52.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.379+0000 7f41ea59c640 1 --2- 172.21.15.42:0/3736970475 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f41ec0a3d70 0x7f41ec11e090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:52.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.379+0000 7f41ea59c640 10 monclient: _finish_hunting 0 2024-09-16T03:47:52.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.379+0000 7f41ea59c640 1 monclient: found mon.noname-a 2024-09-16T03:47:52.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.379+0000 7f41ea59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:47:52.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.379+0000 7f41ea59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:47:52.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.379+0000 7f41ea59c640 1 -- 172.21.15.42:0/3736970475 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41ec121c30 con 0x7f41ec0a0ea0 2024-09-16T03:47:52.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.379+0000 7f41e9d9b640 1 --2- 172.21.15.42:0/3736970475 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f41ec0a3450 0x7f41ec1372a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:47:52.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.379+0000 7f41ea59c640 1 --2- 172.21.15.42:0/3736970475 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f41ec0a0ea0 0x7f41ec1cf180 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f41dc0ab000 tx=0x7f41dc019180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:47:52.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.379+0000 7f41daffd640 10 client.?.objecter ms_handle_connect 0x7f41ec0a0ea0 2024-09-16T03:47:52.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.379+0000 7f41daffd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:47:52.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.379+0000 7f41daffd640 1 -- 172.21.15.42:0/3736970475 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41dc0b0040 con 0x7f41ec0a0ea0 2024-09-16T03:47:52.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.379+0000 7f41daffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:47:52.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.379+0000 7f41daffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:47:52.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.379+0000 7f41daffd640 10 monclient: dump: 2024-09-16T03:47:52.399 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:47:52.399 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:52.399 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:52.399 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:52.399 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:47:52.399 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:52.399 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:47:52.399 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:47:52.399 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:47:52.399 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:52.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.379+0000 7f41daffd640 10 monclient: _finish_auth 0 2024-09-16T03:47:52.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.379+0000 7f41daffd640 10 monclient: _check_auth_tickets 2024-09-16T03:47:52.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.379+0000 7f41daffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:47:52.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.379+0000 7f41f095f640 5 monclient: authenticate success, global_id 764352 2024-09-16T03:47:52.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.379+0000 7f41f095f640 10 monclient: _renew_subs 2024-09-16T03:47:52.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.379+0000 7f41daffd640 1 -- 172.21.15.42:0/3736970475 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f41dc0b4620 con 0x7f41ec0a0ea0 2024-09-16T03:47:52.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.379+0000 7f41f095f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:47:52.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.379+0000 7f41f095f640 1 -- 172.21.15.42:0/3736970475 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41ec06e2e0 con 0x7f41ec0a0ea0 2024-09-16T03:47:52.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.379+0000 7f41daffd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:47:52.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.379+0000 7f41daffd640 1 -- 172.21.15.42:0/3736970475 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41dc0b4a10 con 0x7f41ec0a0ea0 2024-09-16T03:47:52.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.379+0000 7f41daffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:47:52.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.379+0000 7f41daffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:47:52.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.379+0000 7f41daffd640 10 monclient: dump: 2024-09-16T03:47:52.400 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:47:52.400 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:52.400 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:52.401 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:52.401 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:47:52.401 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:52.401 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:47:52.401 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:47:52.401 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:47:52.401 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:52.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.379+0000 7f41dbfff640 4 set_mon_vals no callback set 2024-09-16T03:47:52.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.379+0000 7f41f095f640 10 client.764352.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:47:52.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.379+0000 7f41f095f640 10 monclient: _renew_subs 2024-09-16T03:47:52.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.379+0000 7f41f095f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:47:52.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.379+0000 7f41f095f640 1 -- 172.21.15.42:0/3736970475 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41ec121670 con 0x7f41ec0a0ea0 2024-09-16T03:47:52.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.379+0000 7f41f095f640 1 librados: init done 2024-09-16T03:47:52.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.383+0000 7f41daffd640 1 -- 172.21.15.42:0/3736970475 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f41dc0ab4c0 con 0x7f41ec0a0ea0 2024-09-16T03:47:52.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.383+0000 7f41f095f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:47:52.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.383+0000 7f41f095f640 1 --2- 172.21.15.42:0/3736970475 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f41ec1377e0 0x7f41ec02c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:52.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.383+0000 7f41f095f640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:52.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.383+0000 7f41f095f640 1 -- 172.21.15.42:0/3736970475 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f41ec1d0120 con 0x7f41ec1377e0 2024-09-16T03:47:52.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.383+0000 7f41daffd640 1 --2- 172.21.15.42:0/3736970475 >> v2:172.21.15.154:6812/254845342 conn(0x7f41cc037810 0x7f41cc039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:52.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.383+0000 7f41daffd640 1 -- 172.21.15.42:0/3736970475 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(182..182 src has 1..182) ==== 6676+0+0 (secure 0 0 0) 0x7f41dc0e2e50 con 0x7f41ec0a0ea0 2024-09-16T03:47:52.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.383+0000 7f41daffd640 10 client.764352.objecter ms_dispatch 0x7f41ec000ba0 osd_map(182..182 src has 1..182) 2024-09-16T03:47:52.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.383+0000 7f41daffd640 3 client.764352.objecter handle_osd_map got epochs [182,182] > 0 2024-09-16T03:47:52.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.383+0000 7f41daffd640 3 client.764352.objecter handle_osd_map decoding full epoch 182 2024-09-16T03:47:52.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.383+0000 7f41daffd640 20 client.764352.objecter dump_active .. 0 homeless 2024-09-16T03:47:52.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.383+0000 7f41e9d9b640 1 --2- 172.21.15.42:0/3736970475 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f41ec1377e0 0x7f41ec02c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:52.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.383+0000 7f41e9d9b640 10 monclient: get_auth_request con 0x7f41ec1377e0 auth_method 0 2024-09-16T03:47:52.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.383+0000 7f41e9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:52.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.383+0000 7f41e9d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:52.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.383+0000 7f41e9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:47:52.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.383+0000 7f41ead9d640 1 --2- 172.21.15.42:0/3736970475 >> v2:172.21.15.154:6812/254845342 conn(0x7f41cc037810 0x7f41cc039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:52.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.383+0000 7f41ead9d640 10 monclient: get_auth_request con 0x7f41cc037810 auth_method 0 2024-09-16T03:47:52.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.383+0000 7f41e9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:47:52.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.383+0000 7f41e9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:47:52.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.383+0000 7f41e9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:47:52.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.383+0000 7f41e9d9b640 10 monclient(hunting): handle_auth_done global_id 774160 payload 824 2024-09-16T03:47:52.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.383+0000 7f41e9d9b640 1 -- 172.21.15.42:0/3736970475 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f41ec0a3830 con 0x7f41ec1377e0 2024-09-16T03:47:52.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.383+0000 7f41e9d9b640 1 --2- 172.21.15.42:0/3736970475 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f41ec1377e0 0x7f41ec02c650 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f41ec1215a0 tx=0x7f41e000ff00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:47:52.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.383+0000 7f41daffd640 10 client.764352.objecter ms_handle_connect 0x7f41ec1377e0 2024-09-16T03:47:52.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.383+0000 7f41daffd640 10 client.764352.objecter resend_mon_ops 2024-09-16T03:47:52.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.383+0000 7f41daffd640 10 client.764352.objecter ms_handle_connect 0x7f41cc037810 2024-09-16T03:47:52.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.383+0000 7f41ead9d640 1 --2- 172.21.15.42:0/3736970475 >> v2:172.21.15.154:6812/254845342 conn(0x7f41cc037810 0x7f41cc039cd0 secure :-1 s=READY pgs=3467 cs=0 l=1 rev1=1 crypto rx=0x7f41ec120670 tx=0x7f41e400b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:47:52.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.383+0000 7f41daffd640 1 -- 172.21.15.42:0/3736970475 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41e0018020 con 0x7f41ec1377e0 2024-09-16T03:47:52.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.383+0000 7f41daffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:47:52.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.383+0000 7f41daffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:47:52.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.383+0000 7f41daffd640 10 monclient: dump: 2024-09-16T03:47:52.404 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:47:52.404 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:52.404 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:52.404 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:52.404 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:47:52.404 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:52.404 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:47:52.404 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:47:52.404 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:47:52.405 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:52.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.383+0000 7f41daffd640 1 -- 172.21.15.42:0/3736970475 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f41e000db00 con 0x7f41ec1377e0 2024-09-16T03:47:52.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.383+0000 7f41daffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:47:52.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.383+0000 7f41daffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:47:52.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.383+0000 7f41daffd640 1 -- 172.21.15.42:0/3736970475 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f41ec1377e0 msgr2=0x7f41ec02c650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:52.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.383+0000 7f41daffd640 1 --2- 172.21.15.42:0/3736970475 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f41ec1377e0 0x7f41ec02c650 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f41ec1215a0 tx=0x7f41e000ff00 comp rx=0 tx=0).stop 2024-09-16T03:47:52.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.383+0000 7f41daffd640 1 -- 172.21.15.42:0/3736970475 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f41ec1377e0 msgr2=0x7f41ec02c650 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:47:52.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.383+0000 7f41daffd640 1 --2- 172.21.15.42:0/3736970475 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f41ec1377e0 0x7f41ec02c650 secure :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f41ec1215a0 tx=0x7f41e000ff00 comp rx=0 tx=0).stop 2024-09-16T03:47:52.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.391+0000 7f41f095f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:47:52.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.391+0000 7f41f095f640 1 --2- 172.21.15.42:0/3736970475 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f41ec02cb90 0x7f41ec02cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:52.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.391+0000 7f41f095f640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:52.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.391+0000 7f41f095f640 1 -- 172.21.15.42:0/3736970475 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f41ec13b950 con 0x7f41ec02cb90 2024-09-16T03:47:52.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.391+0000 7f41e9d9b640 1 --2- 172.21.15.42:0/3736970475 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f41ec02cb90 0x7f41ec02cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:52.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.391+0000 7f41e9d9b640 10 monclient: get_auth_request con 0x7f41ec02cb90 auth_method 0 2024-09-16T03:47:52.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.391+0000 7f41e9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:52.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.391+0000 7f41e9d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:52.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.391+0000 7f41e9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:47:52.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.391+0000 7f41e9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:47:52.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.391+0000 7f41e9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:47:52.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.391+0000 7f41e9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:47:52.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.391+0000 7f41e9d9b640 10 monclient(hunting): handle_auth_done global_id 774163 payload 824 2024-09-16T03:47:52.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.391+0000 7f41e9d9b640 1 -- 172.21.15.42:0/3736970475 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f41ec120220 con 0x7f41ec02cb90 2024-09-16T03:47:52.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.391+0000 7f41e9d9b640 1 --2- 172.21.15.42:0/3736970475 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f41ec02cb90 0x7f41ec02cfa0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f41ec1215a0 tx=0x7f41e0007f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:47:52.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.391+0000 7f41daffd640 10 client.764352.objecter ms_handle_connect 0x7f41ec02cb90 2024-09-16T03:47:52.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.391+0000 7f41daffd640 10 client.764352.objecter resend_mon_ops 2024-09-16T03:47:52.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.391+0000 7f41daffd640 1 -- 172.21.15.42:0/3736970475 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41e0018070 con 0x7f41ec02cb90 2024-09-16T03:47:52.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.391+0000 7f41daffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:47:52.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.391+0000 7f41daffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:47:52.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.391+0000 7f41daffd640 10 monclient: dump: 2024-09-16T03:47:52.408 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:47:52.408 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:52.409 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:52.409 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:52.409 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:47:52.409 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:52.409 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:47:52.409 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:47:52.409 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:47:52.409 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:52.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.391+0000 7f41daffd640 1 -- 172.21.15.42:0/3736970475 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4543 (secure 0 0 0) 0x7f41e000de40 con 0x7f41ec02cb90 2024-09-16T03:47:52.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.391+0000 7f41daffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:47:52.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.391+0000 7f41daffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:47:52.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.391+0000 7f41daffd640 1 -- 172.21.15.42:0/3736970475 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f41ec02cb90 msgr2=0x7f41ec02cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:52.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.391+0000 7f41daffd640 1 --2- 172.21.15.42:0/3736970475 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f41ec02cb90 0x7f41ec02cfa0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f41ec1215a0 tx=0x7f41e0007f90 comp rx=0 tx=0).stop 2024-09-16T03:47:52.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.391+0000 7f41daffd640 1 -- 172.21.15.42:0/3736970475 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f41ec02cb90 msgr2=0x7f41ec02cfa0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:47:52.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.391+0000 7f41daffd640 1 --2- 172.21.15.42:0/3736970475 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f41ec02cb90 0x7f41ec02cfa0 secure :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f41ec1215a0 tx=0x7f41e0007f90 comp rx=0 tx=0).stop 2024-09-16T03:47:52.410 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:47:52.410 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:47:52.410 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:47:52.410 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T03:47:52.410 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 54, 2024-09-16T03:47:52.410 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 9979, 2024-09-16T03:47:52.410 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:47:52.410 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:47:52.411 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T03:47:52.411 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T03:47:52.411 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:47:52.411 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 6, 2024-09-16T03:47:52.411 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:47:52.411 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:47:52.411 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:47:52.411 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:47:52.411 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:47:52.411 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:47:52.411 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:47:52.411 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:47:52.411 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:47:52.412 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:47:52.412 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:47:52.412 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:47:52.412 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:47:52.412 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:47:52.412 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:47:52.412 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:47:52.412 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:47:52.412 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:47:52.412 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:47:52.412 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:47:52.412 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:47:52.412 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:47:52.412 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:47:52.413 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:47:52.413 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:47:52.413 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:47:52.413 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:47:52.413 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:47:52.413 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:47:52.413 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:52.413 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:47:52.413 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:47:52.413 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:47:52.413 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:47:52.413 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:47:52.413 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:47:52.414 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:47:52.414 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:47:52.414 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:47:52.414 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:47:52.414 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:47:52.414 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:47:52.414 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:47:52.414 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:47:52.414 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:47:52.414 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:47:52.414 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:47:52.414 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:47:52.414 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:47:52.414 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:47:52.415 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:47:52.415 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:47:52.415 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:47:52.415 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:47:52.415 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:47:52.415 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:47:52.415 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:47:52.415 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:47:52.415 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:47:52.415 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:47:52.415 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:52.415 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:47:52.415 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:52.416 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:47:52.416 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:47:52.416 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:47:52.416 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:47:52.416 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:52.416 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:47:52.416 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:47:52.416 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:47:52.416 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:52.416 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:52.416 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:47:52.416 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:47:52.416 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:47:52.417 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:47:52.417 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:47:52.417 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:52.417 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:47:52.417 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:47:52.417 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:47:52.417 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:47:52.417 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:47:52.417 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:52.417 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:52.417 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:47:52.417 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:47:52.417 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:47:52.417 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:47:52.418 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:52.418 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:47:52.418 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:47:52.418 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:47:52.418 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:52.418 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:52.418 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:47:52.418 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:47:52.418 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:47:52.418 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:47:52.418 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:47:52.418 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:52.418 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:47:52.419 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:47:52.419 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:47:52.419 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:47:52.419 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:47:52.419 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:52.419 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:52.419 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:47:52.419 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:47:52.419 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:47:52.419 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:47:52.419 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:52.419 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:47:52.419 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:47:52.419 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:47:52.419 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:52.420 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:52.420 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:47:52.420 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:47:52.420 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:47:52.420 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:47:52.420 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:47:52.420 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:52.420 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:47:52.420 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:47:52.420 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:47:52.420 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:47:52.420 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:47:52.420 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:47:52.421 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:47:52.421 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:52.421 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:47:52.421 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:47:52.421 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:52.421 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:47:52.421 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:47:52.421 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:47:52.421 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:47:52.421 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:47:52.421 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:47:52.421 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:47:52.421 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:47:52.421 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:47:52.421 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:47:52.422 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:47:52.422 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:47:52.422 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:47:52.422 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:47:52.422 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:47:52.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.391+0000 7f41f095f640 10 librados: watch_flush enter 2024-09-16T03:47:52.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.391+0000 7f41f095f640 10 librados: watch_flush exit 2024-09-16T03:47:52.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.391+0000 7f41f095f640 20 client.764352.objecter shutdown clearing up homeless session... 2024-09-16T03:47:52.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.391+0000 7f41f095f640 10 client.764352.objecter successfully canceled tick 2024-09-16T03:47:52.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.391+0000 7f41f095f640 1 -- 172.21.15.42:0/3736970475 >> v2:172.21.15.154:6812/254845342 conn(0x7f41cc037810 msgr2=0x7f41cc039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:52.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.391+0000 7f41f095f640 1 --2- 172.21.15.42:0/3736970475 >> v2:172.21.15.154:6812/254845342 conn(0x7f41cc037810 0x7f41cc039cd0 secure :-1 s=READY pgs=3467 cs=0 l=1 rev1=1 crypto rx=0x7f41ec120670 tx=0x7f41e400b040 comp rx=0 tx=0).stop 2024-09-16T03:47:52.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.391+0000 7f41f095f640 10 monclient: shutdown 2024-09-16T03:47:52.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.391+0000 7f41f095f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:47:52.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.391+0000 7f41f095f640 1 -- 172.21.15.42:0/3736970475 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f41ec0a0ea0 msgr2=0x7f41ec1cf180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:52.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.391+0000 7f41f095f640 1 --2- 172.21.15.42:0/3736970475 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f41ec0a0ea0 0x7f41ec1cf180 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f41dc0ab000 tx=0x7f41dc019180 comp rx=0 tx=0).stop 2024-09-16T03:47:52.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.391+0000 7f41ead9d640 1 -- 172.21.15.42:0/3736970475 reap_dead start 2024-09-16T03:47:52.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.391+0000 7f41f095f640 1 -- 172.21.15.42:0/3736970475 shutdown_connections 2024-09-16T03:47:52.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.391+0000 7f41f095f640 1 --2- 172.21.15.42:0/3736970475 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f41ec0a0ea0 0x7f41ec1cf180 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:52.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.391+0000 7f41f095f640 1 -- 172.21.15.42:0/3736970475 >> 172.21.15.42:0/3736970475 conn(0x7f41ec0756e0 msgr2=0x7f41ec070200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:47:52.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.391+0000 7f41f095f640 1 -- 172.21.15.42:0/3736970475 shutdown_connections 2024-09-16T03:47:52.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.391+0000 7f41f095f640 1 -- 172.21.15.42:0/3736970475 wait complete. 2024-09-16T03:47:52.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.391+0000 7f41f095f640 1 librados: shutdown 2024-09-16T03:47:52.425 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:47:52.425 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-09-16T03:47:52.425 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-16T03:47:52.425 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-09-16T03:47:52.425 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-16T03:47:52.426 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-16T03:47:52.426 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T03:47:52.441 INFO:tasks.ceph.mon.c:Stopped 2024-09-16T03:47:52.441 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T03:47:52.441 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:47:52.598 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.595+0000 7f7e67a17640 10 monclient: get_monmap_and_config 2024-09-16T03:47:52.598 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.595+0000 7f7e67a17640 10 monclient: build_initial_monmap 2024-09-16T03:47:52.598 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.595+0000 7f7e67a17640 10 monclient: monmap: 2024-09-16T03:47:52.598 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:47:52.598 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:52.598 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:47:52.597167+0000 2024-09-16T03:47:52.598 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:47:52.597167+0000 2024-09-16T03:47:52.598 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:47:52.599 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:52.599 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:47:52.599 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:47:52.599 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:47:52.599 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:47:52.599 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:47:52.599 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:52.605 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.595+0000 7f7e67a17640 1 Processor -- start 2024-09-16T03:47:52.605 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.595+0000 7f7e67a17640 1 -- start start 2024-09-16T03:47:52.605 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.595+0000 7f7e67a17640 10 monclient: init 2024-09-16T03:47:52.605 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.595+0000 7f7e67a17640 10 monclient: _reopen_session rank -1 2024-09-16T03:47:52.605 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.595+0000 7f7e67a17640 10 monclient: _add_conns ranks=[4,0,2,3,1] 2024-09-16T03:47:52.605 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.595+0000 7f7e67a17640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f7e600a1330 0x7f7e6009f180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:52.605 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.595+0000 7f7e67a17640 10 monclient(hunting): picked mon.noname-c con 0x7f7e600a1330 addr v2:172.21.15.154:3300/0 2024-09-16T03:47:52.605 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.595+0000 7f7e67a17640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7e600a1710 addr v1:172.21.15.42:6789/0 2024-09-16T03:47:52.605 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.595+0000 7f7e67a17640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7e601393a0 0x7f7e6013b790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:52.605 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.595+0000 7f7e67a17640 10 monclient(hunting): picked mon.noname-b con 0x7f7e601393a0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:47:52.606 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.595+0000 7f7e67a17640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7e6009e4c0 con 0x7f7e600a1710 2024-09-16T03:47:52.606 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.595+0000 7f7e67a17640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:52.606 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.595+0000 7f7e67a17640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f7e6009edf0 con 0x7f7e600a1330 2024-09-16T03:47:52.606 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.595+0000 7f7e67a17640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:52.606 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.595+0000 7f7e67a17640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f7e6009cd60 con 0x7f7e601393a0 2024-09-16T03:47:52.606 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.595+0000 7f7e67a17640 10 monclient(hunting): _renew_subs 2024-09-16T03:47:52.606 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.595+0000 7f7e67a17640 10 monclient(hunting): authenticate will time out at 1319.566742s 2024-09-16T03:47:52.606 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.595+0000 7f7e65fb4640 1 -- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7e601393a0 msgr2=0x7f7e6013b790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T03:47:52.606 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.595+0000 7f7e65fb4640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7e601393a0 0x7f7e6013b790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:47:52.606 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.595+0000 7f7e64fb2640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f7e600a1710 0x7f7e600a1b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:51738/0 (socket says 172.21.15.42:51738) 2024-09-16T03:47:52.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.595+0000 7f7e64fb2640 1 -- 172.21.15.42:0/1336162818 learned_addr learned my addr 172.21.15.42:0/1336162818 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:47:52.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.595+0000 7f7e657b3640 1 --2- 172.21.15.42:0/1336162818 >> v2:172.21.15.154:3300/0 conn(0x7f7e600a1330 0x7f7e6009f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:52.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.595+0000 7f7e657b3640 10 monclient(hunting): get_auth_request con 0x7f7e600a1330 auth_method 0 2024-09-16T03:47:52.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.595+0000 7f7e657b3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:52.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.595+0000 7f7e657b3640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:52.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.595+0000 7f7e657b3640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:47:52.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.595+0000 7f7e57fff640 1 -- 172.21.15.42:0/1336162818 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4247511164 0 0) 0x7f7e6009e4c0 con 0x7f7e600a1710 2024-09-16T03:47:52.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.595+0000 7f7e57fff640 10 monclient(hunting): _negotiate 2024-09-16T03:47:52.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.595+0000 7f7e57fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:52.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.595+0000 7f7e57fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:47:52.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.595+0000 7f7e57fff640 10 monclient(hunting): my global_id is 764355 2024-09-16T03:47:52.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.595+0000 7f7e57fff640 1 -- 172.21.15.42:0/1336162818 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7e4c003530 con 0x7f7e600a1710 2024-09-16T03:47:52.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.595+0000 7f7e657b3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:47:52.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.595+0000 7f7e657b3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:47:52.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.595+0000 7f7e657b3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:47:52.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.595+0000 7f7e57fff640 1 -- 172.21.15.42:0/1336162818 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 829812692 0 0) 0x7f7e4c003530 con 0x7f7e600a1710 2024-09-16T03:47:52.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.595+0000 7f7e57fff640 1 -- 172.21.15.42:0/1336162818 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7e601393a0 msgr2=0x7f7e6013b790 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:47:52.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.595+0000 7f7e57fff640 1 --2- 172.21.15.42:0/1336162818 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7e601393a0 0x7f7e6013b790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:52.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.595+0000 7f7e57fff640 1 -- 172.21.15.42:0/1336162818 >> v2:172.21.15.154:3300/0 conn(0x7f7e600a1330 msgr2=0x7f7e6009f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:47:52.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.595+0000 7f7e57fff640 1 --2- 172.21.15.42:0/1336162818 >> v2:172.21.15.154:3300/0 conn(0x7f7e600a1330 0x7f7e6009f180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:52.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.595+0000 7f7e57fff640 10 monclient: _finish_hunting 0 2024-09-16T03:47:52.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.595+0000 7f7e57fff640 1 monclient: found mon.noname-a-legacy 2024-09-16T03:47:52.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.595+0000 7f7e57fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:47:52.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.595+0000 7f7e57fff640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.42:6789/0 2024-09-16T03:47:52.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.595+0000 7f7e57fff640 1 -- 172.21.15.42:0/1336162818 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e6009f6c0 con 0x7f7e600a1710 2024-09-16T03:47:52.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.595+0000 7f7e57fff640 10 monclient: _finish_auth 0 2024-09-16T03:47:52.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.595+0000 7f7e57fff640 10 monclient: _check_auth_tickets 2024-09-16T03:47:52.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.595+0000 7f7e57fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:47:52.609 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.595+0000 7f7e57fff640 1 -- 172.21.15.42:0/1336162818 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2177413734 0 0) 0x7f7e500016a0 con 0x7f7e600a1710 2024-09-16T03:47:52.609 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.595+0000 7f7e67a17640 5 monclient: authenticate success, global_id 764355 2024-09-16T03:47:52.609 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.595+0000 7f7e67a17640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:47:52.609 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.595+0000 7f7e57fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:47:52.609 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.595+0000 7f7e57fff640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T03:47:52.609 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.595+0000 7f7e57fff640 10 monclient: dump: 2024-09-16T03:47:52.609 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:47:52.609 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:52.609 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:52.609 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:52.609 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:47:52.609 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:52.609 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:47:52.610 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:47:52.610 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:47:52.610 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:52.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.595+0000 7f7e57fff640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] but i'm connected to v1:172.21.15.42:6789/0, reconnecting 2024-09-16T03:47:52.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.595+0000 7f7e57fff640 10 monclient: _reopen_session rank -1 2024-09-16T03:47:52.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.595+0000 7f7e57fff640 1 -- 172.21.15.42:0/1336162818 >> v1:172.21.15.42:6789/0 conn(0x7f7e600a1710 legacy=0x7f7e600a1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:52.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.595+0000 7f7e57fff640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:47:52.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.595+0000 7f7e57fff640 1 --2- 172.21.15.42:0/1336162818 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7e601393a0 0x7f7e4c0039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:52.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.595+0000 7f7e57fff640 10 monclient(hunting): picked mon.c con 0x7f7e601393a0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:47:52.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.595+0000 7f7e57fff640 1 --2- 172.21.15.42:0/1336162818 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7e4c003f60 0x7f7e4c004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:52.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.595+0000 7f7e57fff640 10 monclient(hunting): picked mon.b con 0x7f7e4c003f60 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:47:52.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.595+0000 7f7e57fff640 1 --2- 172.21.15.42:0/1336162818 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7e4c004d50 0x7f7e4c0051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:52.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.595+0000 7f7e57fff640 10 monclient(hunting): picked mon.a con 0x7f7e4c004d50 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:47:52.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.595+0000 7f7e57fff640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:52.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.595+0000 7f7e57fff640 1 -- 172.21.15.42:0/1336162818 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f7e6009edf0 con 0x7f7e4c004d50 2024-09-16T03:47:52.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.595+0000 7f7e57fff640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:52.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.595+0000 7f7e57fff640 1 -- 172.21.15.42:0/1336162818 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f7e6009cd60 con 0x7f7e4c003f60 2024-09-16T03:47:52.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.595+0000 7f7e57fff640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:52.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.595+0000 7f7e57fff640 1 -- 172.21.15.42:0/1336162818 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f7e4c00bc40 con 0x7f7e601393a0 2024-09-16T03:47:52.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.595+0000 7f7e57fff640 10 monclient(hunting): _renew_subs 2024-09-16T03:47:52.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.595+0000 7f7e657b3640 1 -- 172.21.15.42:0/1336162818 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7e601393a0 msgr2=0x7f7e4c0039e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T03:47:52.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.595+0000 7f7e57fff640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:47:52.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.595+0000 7f7e657b3640 1 --2- 172.21.15.42:0/1336162818 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7e601393a0 0x7f7e4c0039e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:47:52.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.595+0000 7f7e67a17640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-16T03:47:52.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.595+0000 7f7e65fb4640 1 --2- 172.21.15.42:0/1336162818 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7e4c004d50 0x7f7e4c0051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:52.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.595+0000 7f7e65fb4640 10 monclient(hunting): get_auth_request con 0x7f7e4c004d50 auth_method 0 2024-09-16T03:47:52.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.595+0000 7f7e65fb4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:52.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.595+0000 7f7e65fb4640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:52.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.595+0000 7f7e65fb4640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:47:52.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.599+0000 7f7e65fb4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:47:52.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.599+0000 7f7e65fb4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:47:52.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.599+0000 7f7e64fb2640 1 --2- 172.21.15.42:0/1336162818 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7e4c003f60 0x7f7e4c004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:52.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.599+0000 7f7e65fb4640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T03:47:52.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.599+0000 7f7e64fb2640 10 monclient(hunting): get_auth_request con 0x7f7e4c003f60 auth_method 0 2024-09-16T03:47:52.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.599+0000 7f7e64fb2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:52.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.599+0000 7f7e64fb2640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:52.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.599+0000 7f7e64fb2640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:47:52.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.599+0000 7f7e65fb4640 10 monclient(hunting): handle_auth_done global_id 764355 payload 293 2024-09-16T03:47:52.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.599+0000 7f7e65fb4640 1 -- 172.21.15.42:0/1336162818 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7e601393a0 msgr2=0x7f7e4c0039e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:47:52.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.599+0000 7f7e65fb4640 1 --2- 172.21.15.42:0/1336162818 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7e601393a0 0x7f7e4c0039e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:52.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.599+0000 7f7e65fb4640 1 -- 172.21.15.42:0/1336162818 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7e4c003f60 msgr2=0x7f7e4c004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:47:52.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.599+0000 7f7e65fb4640 1 --2- 172.21.15.42:0/1336162818 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7e4c003f60 0x7f7e4c004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:52.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.599+0000 7f7e65fb4640 10 monclient: _finish_hunting 0 2024-09-16T03:47:52.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.599+0000 7f7e65fb4640 1 monclient: found mon.a 2024-09-16T03:47:52.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.599+0000 7f7e65fb4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:47:52.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.599+0000 7f7e65fb4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:47:52.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.599+0000 7f7e65fb4640 1 -- 172.21.15.42:0/1336162818 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e4c003530 con 0x7f7e4c004d50 2024-09-16T03:47:52.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.599+0000 7f7e65fb4640 10 monclient: _finish_auth 0 2024-09-16T03:47:52.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.599+0000 7f7e65fb4640 10 monclient: _check_auth_tickets 2024-09-16T03:47:52.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.599+0000 7f7e65fb4640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:47:52.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.599+0000 7f7e64fb2640 1 --2- 172.21.15.42:0/1336162818 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7e4c003f60 0x7f7e4c004810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T03:47:52.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.599+0000 7f7e65fb4640 1 --2- 172.21.15.42:0/1336162818 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7e4c004d50 0x7f7e4c0051d0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f7e58003d10 tx=0x7f7e5801cdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b85e5b62140c5aa2 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:47:52.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.599+0000 7f7e57fff640 1 -- 172.21.15.42:0/1336162818 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e58009a50 con 0x7f7e4c004d50 2024-09-16T03:47:52.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.599+0000 7f7e57fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:47:52.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.599+0000 7f7e57fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:47:52.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.599+0000 7f7e57fff640 10 monclient: dump: 2024-09-16T03:47:52.614 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:47:52.614 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:52.614 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:52.614 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:52.614 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:47:52.614 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:52.614 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:47:52.614 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:47:52.614 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:47:52.614 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:52.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.599+0000 7f7e57fff640 1 -- 172.21.15.42:0/1336162818 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e580a9070 con 0x7f7e4c004d50 2024-09-16T03:47:52.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.599+0000 7f7e57fff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:47:52.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.599+0000 7f7e57fff640 1 -- 172.21.15.42:0/1336162818 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e580a5de0 con 0x7f7e4c004d50 2024-09-16T03:47:52.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.599+0000 7f7e67a17640 10 monclient: get_monmap_and_config success 2024-09-16T03:47:52.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.599+0000 7f7e67a17640 4 set_mon_vals no callback set 2024-09-16T03:47:52.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.599+0000 7f7e57fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:47:52.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.599+0000 7f7e57fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:47:52.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.599+0000 7f7e57fff640 10 monclient: dump: 2024-09-16T03:47:52.615 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:47:52.615 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:52.615 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:52.615 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:52.615 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:47:52.615 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:52.615 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:47:52.615 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:47:52.616 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:47:52.616 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:52.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.599+0000 7f7e67a17640 10 monclient: shutdown 2024-09-16T03:47:52.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.599+0000 7f7e67a17640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:47:52.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.599+0000 7f7e67a17640 1 -- 172.21.15.42:0/1336162818 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7e4c004d50 msgr2=0x7f7e4c0051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:52.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.599+0000 7f7e67a17640 1 --2- 172.21.15.42:0/1336162818 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7e4c004d50 0x7f7e4c0051d0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f7e58003d10 tx=0x7f7e5801cdd0 comp rx=0 tx=0).stop 2024-09-16T03:47:52.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.599+0000 7f7e65fb4640 1 -- 172.21.15.42:0/1336162818 reap_dead start 2024-09-16T03:47:52.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.599+0000 7f7e67a17640 1 -- 172.21.15.42:0/1336162818 shutdown_connections 2024-09-16T03:47:52.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.599+0000 7f7e67a17640 1 -- 172.21.15.42:0/1336162818 >> 172.21.15.42:0/1336162818 conn(0x7f7e600a2420 msgr2=0x7f7e600a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:47:52.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.599+0000 7f7e67a17640 1 -- 172.21.15.42:0/1336162818 shutdown_connections 2024-09-16T03:47:52.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.599+0000 7f7e67a17640 1 -- 172.21.15.42:0/1336162818 wait complete. 2024-09-16T03:47:52.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.599+0000 7f7e67a17640 10 monclient: build_initial_monmap 2024-09-16T03:47:52.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.599+0000 7f7e67a17640 10 monclient: monmap: 2024-09-16T03:47:52.617 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:47:52.617 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:47:52.617 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:47:52.617 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:47:52.617 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:47:52.617 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:52.617 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:47:52.617 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:47:52.617 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:47:52.617 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:52.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.603+0000 7f7e67a17640 1 librados: starting msgr at 2024-09-16T03:47:52.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.603+0000 7f7e67a17640 1 librados: starting objecter 2024-09-16T03:47:52.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.603+0000 7f7e67a17640 1 Processor -- start 2024-09-16T03:47:52.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.603+0000 7f7e67a17640 1 -- start start 2024-09-16T03:47:52.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.603+0000 7f7e67a17640 1 librados: setting wanted keys 2024-09-16T03:47:52.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.603+0000 7f7e67a17640 1 librados: calling monclient init 2024-09-16T03:47:52.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.603+0000 7f7e67a17640 10 monclient: init 2024-09-16T03:47:52.622 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.603+0000 7f7e67a17640 10 monclient: _reopen_session rank -1 2024-09-16T03:47:52.622 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.603+0000 7f7e67a17640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T03:47:52.622 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.603+0000 7f7e67a17640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7e601c95c0 0x7f7e601329d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:52.622 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.603+0000 7f7e67a17640 10 monclient(hunting): picked mon.noname-a con 0x7f7e601c95c0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:47:52.622 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.603+0000 7f7e67a17640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7e60132f10 0x7f7e601ccae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:52.622 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.603+0000 7f7e67a17640 10 monclient(hunting): picked mon.noname-b con 0x7f7e60132f10 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:47:52.622 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.603+0000 7f7e67a17640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7e601cd020 0x7f7e601cd4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:52.622 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.603+0000 7f7e67a17640 10 monclient(hunting): picked mon.noname-c con 0x7f7e601cd020 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:47:52.622 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.603+0000 7f7e67a17640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:52.622 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.603+0000 7f7e67a17640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f7e6013c330 con 0x7f7e601c95c0 2024-09-16T03:47:52.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.603+0000 7f7e67a17640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:52.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.603+0000 7f7e67a17640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f7e6013c1b0 con 0x7f7e60132f10 2024-09-16T03:47:52.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.603+0000 7f7e67a17640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:52.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.603+0000 7f7e67a17640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f7e6013c4b0 con 0x7f7e601cd020 2024-09-16T03:47:52.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.603+0000 7f7e67a17640 10 monclient(hunting): _renew_subs 2024-09-16T03:47:52.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.603+0000 7f7e67a17640 10 monclient(hunting): authenticate will time out at 1319.572835s 2024-09-16T03:47:52.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.603+0000 7f7e65fb4640 1 -- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7e601cd020 msgr2=0x7f7e601cd4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T03:47:52.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.603+0000 7f7e65fb4640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7e601cd020 0x7f7e601cd4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:47:52.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.603+0000 7f7e657b3640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7e601c95c0 0x7f7e601329d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:52.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.603+0000 7f7e657b3640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7e601c95c0 0x7f7e601329d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:43514/0 (socket says 172.21.15.42:43514) 2024-09-16T03:47:52.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.603+0000 7f7e657b3640 1 -- 172.21.15.42:0/457810402 learned_addr learned my addr 172.21.15.42:0/457810402 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:47:52.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.603+0000 7f7e657b3640 10 monclient(hunting): get_auth_request con 0x7f7e601c95c0 auth_method 0 2024-09-16T03:47:52.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.603+0000 7f7e657b3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:52.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.603+0000 7f7e657b3640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:52.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.603+0000 7f7e657b3640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:47:52.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.603+0000 7f7e657b3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:47:52.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.603+0000 7f7e657b3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:47:52.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.603+0000 7f7e657b3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:47:52.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.603+0000 7f7e64fb2640 1 --2- 172.21.15.42:0/457810402 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7e60132f10 0x7f7e601ccae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:52.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.603+0000 7f7e64fb2640 10 monclient(hunting): get_auth_request con 0x7f7e60132f10 auth_method 0 2024-09-16T03:47:52.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.603+0000 7f7e64fb2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:52.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.603+0000 7f7e64fb2640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:52.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.603+0000 7f7e64fb2640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:47:52.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.603+0000 7f7e657b3640 10 monclient(hunting): handle_auth_done global_id 764358 payload 824 2024-09-16T03:47:52.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.603+0000 7f7e657b3640 1 -- 172.21.15.42:0/457810402 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7e601cd020 msgr2=0x7f7e601cd4a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:47:52.625 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.603+0000 7f7e657b3640 1 --2- 172.21.15.42:0/457810402 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7e601cd020 0x7f7e601cd4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:52.625 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.603+0000 7f7e657b3640 1 -- 172.21.15.42:0/457810402 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7e60132f10 msgr2=0x7f7e601ccae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:52.625 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.603+0000 7f7e657b3640 1 --2- 172.21.15.42:0/457810402 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7e60132f10 0x7f7e601ccae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:52.625 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.603+0000 7f7e657b3640 10 monclient: _finish_hunting 0 2024-09-16T03:47:52.625 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.603+0000 7f7e657b3640 1 monclient: found mon.noname-a 2024-09-16T03:47:52.625 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.603+0000 7f7e657b3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:47:52.625 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.603+0000 7f7e657b3640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:47:52.625 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.603+0000 7f7e657b3640 1 -- 172.21.15.42:0/457810402 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e60120390 con 0x7f7e601c95c0 2024-09-16T03:47:52.625 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.603+0000 7f7e64fb2640 1 --2- 172.21.15.42:0/457810402 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7e60132f10 0x7f7e601ccae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T03:47:52.625 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.603+0000 7f7e657b3640 1 --2- 172.21.15.42:0/457810402 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7e601c95c0 0x7f7e601329d0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f7e5c00e9e0 tx=0x7f7e5c00eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:47:52.625 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.603+0000 7f7e55ffb640 10 client.?.objecter ms_handle_connect 0x7f7e601c95c0 2024-09-16T03:47:52.625 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.603+0000 7f7e55ffb640 10 client.?.objecter resend_mon_ops 2024-09-16T03:47:52.625 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.603+0000 7f7e55ffb640 1 -- 172.21.15.42:0/457810402 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e5c017070 con 0x7f7e601c95c0 2024-09-16T03:47:52.626 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.603+0000 7f7e55ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:47:52.626 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.603+0000 7f7e55ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:47:52.626 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.603+0000 7f7e55ffb640 10 monclient: dump: 2024-09-16T03:47:52.626 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:47:52.626 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:52.626 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:52.626 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:52.626 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:47:52.626 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:52.626 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:47:52.626 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:47:52.626 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:47:52.626 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:52.627 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.603+0000 7f7e55ffb640 10 monclient: _finish_auth 0 2024-09-16T03:47:52.627 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.603+0000 7f7e55ffb640 10 monclient: _check_auth_tickets 2024-09-16T03:47:52.627 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.603+0000 7f7e55ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:47:52.627 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.603+0000 7f7e55ffb640 1 -- 172.21.15.42:0/457810402 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e5c00c9d0 con 0x7f7e601c95c0 2024-09-16T03:47:52.627 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.603+0000 7f7e55ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T03:47:52.627 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.603+0000 7f7e55ffb640 1 -- 172.21.15.42:0/457810402 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e5c00cda0 con 0x7f7e601c95c0 2024-09-16T03:47:52.627 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.603+0000 7f7e67a17640 5 monclient: authenticate success, global_id 764358 2024-09-16T03:47:52.627 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.603+0000 7f7e67a17640 10 monclient: _renew_subs 2024-09-16T03:47:52.627 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.603+0000 7f7e67a17640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:47:52.627 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.603+0000 7f7e67a17640 1 -- 172.21.15.42:0/457810402 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e600a4690 con 0x7f7e601c95c0 2024-09-16T03:47:52.627 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.603+0000 7f7e577fe640 4 set_mon_vals no callback set 2024-09-16T03:47:52.627 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.603+0000 7f7e55ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:47:52.627 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.603+0000 7f7e55ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:47:52.627 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.603+0000 7f7e55ffb640 10 monclient: dump: 2024-09-16T03:47:52.628 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:47:52.628 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:52.628 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:52.628 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:52.628 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:47:52.628 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:52.628 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:47:52.628 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:47:52.628 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:47:52.628 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:52.628 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.603+0000 7f7e67a17640 10 client.764358.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:47:52.628 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.603+0000 7f7e67a17640 10 monclient: _renew_subs 2024-09-16T03:47:52.628 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.603+0000 7f7e67a17640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:47:52.629 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.603+0000 7f7e67a17640 1 -- 172.21.15.42:0/457810402 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e60073a30 con 0x7f7e601c95c0 2024-09-16T03:47:52.629 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.603+0000 7f7e67a17640 1 librados: init done 2024-09-16T03:47:52.629 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.603+0000 7f7e67a17640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:47:52.629 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.603+0000 7f7e55ffb640 1 -- 172.21.15.42:0/457810402 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f7e5c01a070 con 0x7f7e601c95c0 2024-09-16T03:47:52.629 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.603+0000 7f7e67a17640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:47:52.629 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.603+0000 7f7e67a17640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:47:52.629 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.603+0000 7f7e67a17640 1 -- 172.21.15.42:0/457810402 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e6009f180 con 0x7f7e601c95c0 2024-09-16T03:47:52.629 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.603+0000 7f7e55ffb640 1 --2- 172.21.15.42:0/457810402 >> v2:172.21.15.154:6812/254845342 conn(0x7f7e440377c0 0x7f7e44039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:52.629 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.603+0000 7f7e55ffb640 1 -- 172.21.15.42:0/457810402 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(182..182 src has 1..182) ==== 6676+0+0 (secure 0 0 0) 0x7f7e5c0492a0 con 0x7f7e601c95c0 2024-09-16T03:47:52.629 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.603+0000 7f7e55ffb640 10 client.764358.objecter ms_dispatch 0x7f7e60000ba0 osd_map(182..182 src has 1..182) 2024-09-16T03:47:52.629 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.603+0000 7f7e55ffb640 3 client.764358.objecter handle_osd_map got epochs [182,182] > 0 2024-09-16T03:47:52.629 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.603+0000 7f7e55ffb640 3 client.764358.objecter handle_osd_map decoding full epoch 182 2024-09-16T03:47:52.629 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.603+0000 7f7e55ffb640 20 client.764358.objecter dump_active .. 0 homeless 2024-09-16T03:47:52.629 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.603+0000 7f7e64fb2640 1 --2- 172.21.15.42:0/457810402 >> v2:172.21.15.154:6812/254845342 conn(0x7f7e440377c0 0x7f7e44039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:52.630 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.603+0000 7f7e64fb2640 10 monclient: get_auth_request con 0x7f7e440377c0 auth_method 0 2024-09-16T03:47:52.630 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.603+0000 7f7e55ffb640 10 client.764358.objecter ms_handle_connect 0x7f7e440377c0 2024-09-16T03:47:52.630 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.603+0000 7f7e64fb2640 1 --2- 172.21.15.42:0/457810402 >> v2:172.21.15.154:6812/254845342 conn(0x7f7e440377c0 0x7f7e44039c80 secure :-1 s=READY pgs=3468 cs=0 l=1 rev1=1 crypto rx=0x7f7e50004850 tx=0x7f7e50009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:47:52.630 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.607+0000 7f7e55ffb640 1 -- 172.21.15.42:0/457810402 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f7e5c01a360 con 0x7f7e601c95c0 2024-09-16T03:47:52.630 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.607+0000 7f7e55ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:47:52.630 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.607+0000 7f7e55ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:47:52.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.827+0000 7f7e67a17640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:47:52.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.827+0000 7f7e67a17640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:47:52.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.827+0000 7f7e67a17640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:47:52.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.827+0000 7f7e67a17640 1 -- 172.21.15.42:0/457810402 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f7e6011e0a0 con 0x7f7e601c95c0 2024-09-16T03:47:52.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.827+0000 7f7e55ffb640 1 -- 172.21.15.42:0/457810402 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7f7e5c013660 con 0x7f7e601c95c0 2024-09-16T03:47:52.832 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.827+0000 7f7e55ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:47:52.832 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.827+0000 7f7e55ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:47:52.832 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":54,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:47:52.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.831+0000 7f7e67a17640 10 librados: watch_flush enter 2024-09-16T03:47:52.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.831+0000 7f7e67a17640 10 librados: watch_flush exit 2024-09-16T03:47:52.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.831+0000 7f7e67a17640 20 client.764358.objecter shutdown clearing up homeless session... 2024-09-16T03:47:52.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.831+0000 7f7e67a17640 10 client.764358.objecter successfully canceled tick 2024-09-16T03:47:52.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.831+0000 7f7e67a17640 1 -- 172.21.15.42:0/457810402 >> v2:172.21.15.154:6812/254845342 conn(0x7f7e440377c0 msgr2=0x7f7e44039c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:52.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.831+0000 7f7e67a17640 1 --2- 172.21.15.42:0/457810402 >> v2:172.21.15.154:6812/254845342 conn(0x7f7e440377c0 0x7f7e44039c80 secure :-1 s=READY pgs=3468 cs=0 l=1 rev1=1 crypto rx=0x7f7e50004850 tx=0x7f7e50009040 comp rx=0 tx=0).stop 2024-09-16T03:47:52.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.831+0000 7f7e67a17640 10 monclient: shutdown 2024-09-16T03:47:52.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.831+0000 7f7e67a17640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:47:52.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.831+0000 7f7e67a17640 1 -- 172.21.15.42:0/457810402 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7e601c95c0 msgr2=0x7f7e601329d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:52.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.831+0000 7f7e67a17640 1 --2- 172.21.15.42:0/457810402 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7e601c95c0 0x7f7e601329d0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f7e5c00e9e0 tx=0x7f7e5c00eea0 comp rx=0 tx=0).stop 2024-09-16T03:47:52.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.831+0000 7f7e67a17640 1 -- 172.21.15.42:0/457810402 shutdown_connections 2024-09-16T03:47:52.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.831+0000 7f7e67a17640 1 --2- 172.21.15.42:0/457810402 >> v2:172.21.15.154:6812/254845342 conn(0x7f7e440377c0 0x7f7e44039c80 unknown :-1 s=CLOSED pgs=3468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:52.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.831+0000 7f7e67a17640 1 --2- 172.21.15.42:0/457810402 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7e601cd020 0x7f7e601cd4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:52.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.831+0000 7f7e67a17640 1 --2- 172.21.15.42:0/457810402 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7e60132f10 0x7f7e601ccae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:52.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.831+0000 7f7e67a17640 1 --2- 172.21.15.42:0/457810402 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7e601c95c0 0x7f7e601329d0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:52.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.831+0000 7f7e67a17640 1 -- 172.21.15.42:0/457810402 >> 172.21.15.42:0/457810402 conn(0x7f7e600a2420 msgr2=0x7f7e60133410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:47:52.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.831+0000 7f7e67a17640 1 -- 172.21.15.42:0/457810402 shutdown_connections 2024-09-16T03:47:52.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.831+0000 7f7e67a17640 1 -- 172.21.15.42:0/457810402 wait complete. 2024-09-16T03:47:52.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:52.831+0000 7f7e67a17640 1 librados: shutdown 2024-09-16T03:47:55.852 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:47:55.982 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.979+0000 7f48590fb640 10 monclient: get_monmap_and_config 2024-09-16T03:47:55.983 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.979+0000 7f48590fb640 10 monclient: build_initial_monmap 2024-09-16T03:47:55.983 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.979+0000 7f48590fb640 10 monclient: monmap: 2024-09-16T03:47:55.983 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:47:55.983 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:55.983 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:47:55.981849+0000 2024-09-16T03:47:55.983 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:47:55.981849+0000 2024-09-16T03:47:55.983 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:47:55.983 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:55.983 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:47:55.983 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:47:55.983 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:47:55.983 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:47:55.983 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:47:55.984 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:55.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.979+0000 7f48590fb640 1 Processor -- start 2024-09-16T03:47:55.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.979+0000 7f48590fb640 1 -- start start 2024-09-16T03:47:55.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.979+0000 7f48590fb640 10 monclient: init 2024-09-16T03:47:55.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.979+0000 7f48590fb640 10 monclient: _reopen_session rank -1 2024-09-16T03:47:55.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.979+0000 7f48590fb640 10 monclient: _add_conns ranks=[2,0,1,4,3] 2024-09-16T03:47:55.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.979+0000 7f48590fb640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f485406db90 0x7f48541340c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:55.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.979+0000 7f48590fb640 10 monclient(hunting): picked mon.noname-b con 0x7f485406db90 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:47:55.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.979+0000 7f48590fb640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f48540a37f0 addr v1:172.21.15.42:6789/0 2024-09-16T03:47:55.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.979+0000 7f48590fb640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f48540a3bd0 addr v1:172.21.15.154:6789/0 2024-09-16T03:47:55.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.979+0000 7f48590fb640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f485406ece0 con 0x7f48540a37f0 2024-09-16T03:47:55.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.979+0000 7f48590fb640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f485409bf80 con 0x7f48540a3bd0 2024-09-16T03:47:55.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.979+0000 7f48590fb640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:55.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.979+0000 7f48590fb640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f485406df70 con 0x7f485406db90 2024-09-16T03:47:55.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.979+0000 7f48590fb640 10 monclient(hunting): _renew_subs 2024-09-16T03:47:55.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.979+0000 7f48590fb640 10 monclient(hunting): authenticate will time out at 1322.951337s 2024-09-16T03:47:55.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.979+0000 7f4852d9d640 1 -- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f485406db90 msgr2=0x7f48541340c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T03:47:55.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.979+0000 7f4852d9d640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f485406db90 0x7f48541340c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:47:55.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.979+0000 7f485259c640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f48540a37f0 0x7f4854136e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:50696/0 (socket says 172.21.15.42:50696) 2024-09-16T03:47:55.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.979+0000 7f485259c640 1 -- 172.21.15.42:0/4257293432 learned_addr learned my addr 172.21.15.42:0/4257293432 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:47:55.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.979+0000 7f4851d9b640 1 -- 172.21.15.42:0/4257293432 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1340742266 0 0) 0x7f485406ece0 con 0x7f48540a37f0 2024-09-16T03:47:55.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.979+0000 7f4851d9b640 10 monclient(hunting): _negotiate 2024-09-16T03:47:55.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.979+0000 7f4851d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:55.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.979+0000 7f4851d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:47:55.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.979+0000 7f4851d9b640 10 monclient(hunting): my global_id is 764361 2024-09-16T03:47:55.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.979+0000 7f4851d9b640 1 -- 172.21.15.42:0/4257293432 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f483c003530 con 0x7f48540a37f0 2024-09-16T03:47:55.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.983+0000 7f4851d9b640 1 -- 172.21.15.42:0/4257293432 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1712488369 0 0) 0x7f483c003530 con 0x7f48540a37f0 2024-09-16T03:47:55.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.983+0000 7f4851d9b640 1 -- 172.21.15.42:0/4257293432 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f485406db90 msgr2=0x7f48541340c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:47:55.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.983+0000 7f4851d9b640 1 --2- 172.21.15.42:0/4257293432 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f485406db90 0x7f48541340c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:55.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.983+0000 7f4851d9b640 1 -- 172.21.15.42:0/4257293432 >> v1:172.21.15.154:6789/0 conn(0x7f48540a3bd0 legacy=0x7f4854137520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:55.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.983+0000 7f4851d9b640 10 monclient: _finish_hunting 0 2024-09-16T03:47:55.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.983+0000 7f4851d9b640 1 monclient: found mon.noname-a-legacy 2024-09-16T03:47:55.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.983+0000 7f4851d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:47:55.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.983+0000 7f4851d9b640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.42:6789/0 2024-09-16T03:47:55.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.983+0000 7f4851d9b640 1 -- 172.21.15.42:0/4257293432 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4854137c30 con 0x7f48540a37f0 2024-09-16T03:47:55.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.983+0000 7f4851d9b640 10 monclient: _finish_auth 0 2024-09-16T03:47:55.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.983+0000 7f4851d9b640 10 monclient: _check_auth_tickets 2024-09-16T03:47:55.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.983+0000 7f4851d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:47:55.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.983+0000 7f4851d9b640 1 -- 172.21.15.42:0/4257293432 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2177413734 0 0) 0x7f4848001680 con 0x7f48540a37f0 2024-09-16T03:47:55.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.983+0000 7f4851d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:47:55.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.983+0000 7f4851d9b640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T03:47:55.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.983+0000 7f4851d9b640 10 monclient: dump: 2024-09-16T03:47:55.991 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:47:55.991 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:55.991 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:55.991 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:55.991 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:47:55.991 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:55.991 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:47:55.991 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:47:55.991 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:47:55.991 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:55.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.983+0000 7f4851d9b640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] but i'm connected to v1:172.21.15.42:6789/0, reconnecting 2024-09-16T03:47:55.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.983+0000 7f4851d9b640 10 monclient: _reopen_session rank -1 2024-09-16T03:47:55.992 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.983+0000 7f4851d9b640 1 -- 172.21.15.42:0/4257293432 >> v1:172.21.15.42:6789/0 conn(0x7f48540a37f0 legacy=0x7f4854136e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:55.992 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.983+0000 7f4851d9b640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:47:55.992 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.983+0000 7f4851d9b640 1 --2- 172.21.15.42:0/4257293432 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f483c0039f0 0x7f483c003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:55.992 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.983+0000 7f4851d9b640 10 monclient(hunting): picked mon.a con 0x7f483c0039f0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:47:55.992 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.983+0000 7f4851d9b640 1 --2- 172.21.15.42:0/4257293432 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f485406db90 0x7f483c004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:55.992 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.983+0000 7f4851d9b640 10 monclient(hunting): picked mon.c con 0x7f485406db90 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:47:55.992 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.983+0000 7f4851d9b640 1 --2- 172.21.15.42:0/4257293432 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f483c004e70 0x7f483c00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:55.992 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.983+0000 7f4851d9b640 10 monclient(hunting): picked mon.b con 0x7f483c004e70 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:47:55.992 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.983+0000 7f4851d9b640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:55.992 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.983+0000 7f4851d9b640 1 -- 172.21.15.42:0/4257293432 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f485406df70 con 0x7f483c0039f0 2024-09-16T03:47:55.992 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.983+0000 7f4851d9b640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:55.992 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.983+0000 7f4851d9b640 1 -- 172.21.15.42:0/4257293432 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f483c005600 con 0x7f483c004e70 2024-09-16T03:47:55.992 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.983+0000 7f4851d9b640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:55.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.983+0000 7f4851d9b640 1 -- 172.21.15.42:0/4257293432 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f483c00bf10 con 0x7f485406db90 2024-09-16T03:47:55.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.983+0000 7f4851d9b640 10 monclient(hunting): _renew_subs 2024-09-16T03:47:55.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.983+0000 7f4851d9b640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:47:55.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.983+0000 7f485259c640 1 -- 172.21.15.42:0/4257293432 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f485406db90 msgr2=0x7f483c004810 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T03:47:55.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.983+0000 7f485259c640 1 --2- 172.21.15.42:0/4257293432 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f485406db90 0x7f483c004810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:47:55.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.983+0000 7f4852d9d640 1 --2- 172.21.15.42:0/4257293432 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f483c0039f0 0x7f483c003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:55.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.983+0000 7f4852d9d640 10 monclient(hunting): get_auth_request con 0x7f483c0039f0 auth_method 0 2024-09-16T03:47:55.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.983+0000 7f4852d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:55.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.983+0000 7f4852d9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:55.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.983+0000 7f4852d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:47:55.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.983+0000 7f4852d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:47:55.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.983+0000 7f4852d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:47:55.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.983+0000 7f4852d9d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T03:47:55.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.983+0000 7f4852d9d640 10 monclient(hunting): handle_auth_done global_id 764361 payload 293 2024-09-16T03:47:55.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.983+0000 7f4852d9d640 1 -- 172.21.15.42:0/4257293432 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f485406db90 msgr2=0x7f483c004810 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:47:55.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.983+0000 7f4852d9d640 1 --2- 172.21.15.42:0/4257293432 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f485406db90 0x7f483c004810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:55.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.983+0000 7f4852d9d640 1 -- 172.21.15.42:0/4257293432 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f483c004e70 msgr2=0x7f483c00b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:47:55.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.983+0000 7f4852d9d640 1 --2- 172.21.15.42:0/4257293432 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f483c004e70 0x7f483c00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:55.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.983+0000 7f4852d9d640 10 monclient: _finish_hunting 0 2024-09-16T03:47:55.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.983+0000 7f4852d9d640 1 monclient: found mon.a 2024-09-16T03:47:55.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.983+0000 7f4852d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:47:55.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.983+0000 7f4852d9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:47:55.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.983+0000 7f4852d9d640 1 -- 172.21.15.42:0/4257293432 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f483c003530 con 0x7f483c0039f0 2024-09-16T03:47:55.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.983+0000 7f4852d9d640 10 monclient: _finish_auth 0 2024-09-16T03:47:55.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.983+0000 7f4852d9d640 10 monclient: _check_auth_tickets 2024-09-16T03:47:55.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.983+0000 7f4852d9d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:47:55.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.983+0000 7f48590fb640 5 monclient: authenticate success, global_id 764361 2024-09-16T03:47:55.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.983+0000 7f48590fb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:47:55.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.983+0000 7f4852d9d640 1 --2- 172.21.15.42:0/4257293432 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f483c0039f0 0x7f483c003e10 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f4844003990 tx=0x7f484401c420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a7beccd520161c9 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:47:55.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.983+0000 7f4851d9b640 1 -- 172.21.15.42:0/4257293432 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4844009520 con 0x7f483c0039f0 2024-09-16T03:47:55.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.983+0000 7f4851d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:47:55.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.983+0000 7f4851d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:47:55.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.983+0000 7f4851d9b640 10 monclient: dump: 2024-09-16T03:47:55.995 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:47:55.995 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:55.995 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:55.995 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:55.995 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:47:55.995 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:55.995 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:47:55.995 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:47:55.996 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:47:55.996 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:55.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.983+0000 7f48590fb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:47:55.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.983+0000 7f4851d9b640 1 -- 172.21.15.42:0/4257293432 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4844003bc0 con 0x7f483c0039f0 2024-09-16T03:47:55.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.983+0000 7f4851d9b640 10 monclient: handle_config config(0 keys) 2024-09-16T03:47:55.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.983+0000 7f48590fb640 10 monclient: get_monmap_and_config success 2024-09-16T03:47:55.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.983+0000 7f48590fb640 4 set_mon_vals no callback set 2024-09-16T03:47:55.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.983+0000 7f4851d9b640 1 -- 172.21.15.42:0/4257293432 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4844005070 con 0x7f483c0039f0 2024-09-16T03:47:55.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.983+0000 7f4851d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:47:55.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.983+0000 7f4851d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:47:55.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.983+0000 7f4851d9b640 10 monclient: dump: 2024-09-16T03:47:55.996 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:47:55.996 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:55.996 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:55.997 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:55.997 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:47:55.997 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:55.997 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:47:55.997 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:47:55.997 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:47:55.997 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:56.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.983+0000 7f48590fb640 10 monclient: shutdown 2024-09-16T03:47:56.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.983+0000 7f48590fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:47:56.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.983+0000 7f48590fb640 1 -- 172.21.15.42:0/4257293432 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f483c0039f0 msgr2=0x7f483c003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:56.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.983+0000 7f48590fb640 1 --2- 172.21.15.42:0/4257293432 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f483c0039f0 0x7f483c003e10 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f4844003990 tx=0x7f484401c420 comp rx=0 tx=0).stop 2024-09-16T03:47:56.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.983+0000 7f485359e640 1 -- 172.21.15.42:0/4257293432 reap_dead start 2024-09-16T03:47:56.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.983+0000 7f48590fb640 1 -- 172.21.15.42:0/4257293432 shutdown_connections 2024-09-16T03:47:56.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.983+0000 7f48590fb640 1 -- 172.21.15.42:0/4257293432 >> 172.21.15.42:0/4257293432 conn(0x7f48540a9fb0 msgr2=0x7f48540aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:47:56.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.983+0000 7f48590fb640 1 -- 172.21.15.42:0/4257293432 shutdown_connections 2024-09-16T03:47:56.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.983+0000 7f48590fb640 1 -- 172.21.15.42:0/4257293432 wait complete. 2024-09-16T03:47:56.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.987+0000 7f48590fb640 10 monclient: build_initial_monmap 2024-09-16T03:47:56.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.987+0000 7f48590fb640 10 monclient: monmap: 2024-09-16T03:47:56.003 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:47:56.003 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:47:56.003 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:47:56.004 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:47:56.004 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:47:56.004 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:56.004 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:47:56.004 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:47:56.004 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:47:56.004 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:56.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.987+0000 7f48590fb640 1 librados: starting msgr at 2024-09-16T03:47:56.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.987+0000 7f48590fb640 1 librados: starting objecter 2024-09-16T03:47:56.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.987+0000 7f48590fb640 1 Processor -- start 2024-09-16T03:47:56.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.987+0000 7f48590fb640 1 -- start start 2024-09-16T03:47:56.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.987+0000 7f48590fb640 1 librados: setting wanted keys 2024-09-16T03:47:56.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.987+0000 7f48590fb640 1 librados: calling monclient init 2024-09-16T03:47:56.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.987+0000 7f48590fb640 10 monclient: init 2024-09-16T03:47:56.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.987+0000 7f48590fb640 10 monclient: _reopen_session rank -1 2024-09-16T03:47:56.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.987+0000 7f48590fb640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T03:47:56.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.987+0000 7f48590fb640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f48541cbdc0 0x7f48540a2ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:56.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.987+0000 7f48590fb640 10 monclient(hunting): picked mon.noname-b con 0x7f48541cbdc0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:47:56.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.987+0000 7f48590fb640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f48540a33e0 0x7f48541cf300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:56.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.987+0000 7f48590fb640 10 monclient(hunting): picked mon.noname-c con 0x7f48540a33e0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:47:56.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.987+0000 7f48590fb640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f48541cf840 0x7f48541cfcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:56.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.987+0000 7f48590fb640 10 monclient(hunting): picked mon.noname-a con 0x7f48541cf840 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:47:56.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.987+0000 7f48590fb640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:56.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.987+0000 7f48590fb640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f4854140b50 con 0x7f48541cf840 2024-09-16T03:47:56.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.987+0000 7f48590fb640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:56.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.987+0000 7f48590fb640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f48541409d0 con 0x7f48541cbdc0 2024-09-16T03:47:56.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.987+0000 7f48590fb640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:56.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.987+0000 7f48590fb640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f4854140cd0 con 0x7f48540a33e0 2024-09-16T03:47:56.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.987+0000 7f48590fb640 10 monclient(hunting): _renew_subs 2024-09-16T03:47:56.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.987+0000 7f48590fb640 10 monclient(hunting): authenticate will time out at 1322.957348s 2024-09-16T03:47:56.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.987+0000 7f485259c640 1 -- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f48540a33e0 msgr2=0x7f48541cf300 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T03:47:56.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.987+0000 7f485259c640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f48540a33e0 0x7f48541cf300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:47:56.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.987+0000 7f485359e640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f48541cf840 0x7f48541cfcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:56.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.987+0000 7f485359e640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f48541cf840 0x7f48541cfcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:50934/0 (socket says 172.21.15.42:50934) 2024-09-16T03:47:56.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.987+0000 7f485359e640 1 -- 172.21.15.42:0/2787208107 learned_addr learned my addr 172.21.15.42:0/2787208107 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:47:56.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.987+0000 7f485359e640 10 monclient(hunting): get_auth_request con 0x7f48541cf840 auth_method 0 2024-09-16T03:47:56.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.987+0000 7f485359e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:56.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.987+0000 7f485359e640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:56.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.987+0000 7f485359e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:47:56.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.987+0000 7f485359e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:47:56.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.987+0000 7f485359e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:47:56.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.987+0000 7f485359e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:47:56.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.987+0000 7f4852d9d640 1 --2- 172.21.15.42:0/2787208107 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f48541cbdc0 0x7f48540a2ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:56.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.987+0000 7f4852d9d640 10 monclient(hunting): get_auth_request con 0x7f48541cbdc0 auth_method 0 2024-09-16T03:47:56.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.987+0000 7f4852d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:56.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.987+0000 7f4852d9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:56.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.987+0000 7f4852d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:47:56.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.987+0000 7f485359e640 10 monclient(hunting): handle_auth_done global_id 764364 payload 824 2024-09-16T03:47:56.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.987+0000 7f485359e640 1 -- 172.21.15.42:0/2787208107 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f48540a33e0 msgr2=0x7f48541cf300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:47:56.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.987+0000 7f485359e640 1 --2- 172.21.15.42:0/2787208107 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f48540a33e0 0x7f48541cf300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:56.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.987+0000 7f485359e640 1 -- 172.21.15.42:0/2787208107 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f48541cbdc0 msgr2=0x7f48540a2ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:56.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.987+0000 7f485359e640 1 --2- 172.21.15.42:0/2787208107 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f48541cbdc0 0x7f48540a2ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:56.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.987+0000 7f485359e640 10 monclient: _finish_hunting 0 2024-09-16T03:47:56.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.987+0000 7f485359e640 1 monclient: found mon.noname-a 2024-09-16T03:47:56.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.987+0000 7f485359e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:47:56.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.987+0000 7f485359e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:47:56.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.987+0000 7f485359e640 1 -- 172.21.15.42:0/2787208107 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4854132c30 con 0x7f48541cf840 2024-09-16T03:47:56.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.987+0000 7f485359e640 1 --2- 172.21.15.42:0/2787208107 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f48541cf840 0x7f48541cfcf0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f485409bbd0 tx=0x7f484c00cad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:47:56.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.987+0000 7f483b7fe640 10 client.?.objecter ms_handle_connect 0x7f48541cf840 2024-09-16T03:47:56.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.987+0000 7f483b7fe640 10 client.?.objecter resend_mon_ops 2024-09-16T03:47:56.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.987+0000 7f483b7fe640 1 -- 172.21.15.42:0/2787208107 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f484c017070 con 0x7f48541cf840 2024-09-16T03:47:56.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.987+0000 7f483b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:47:56.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.987+0000 7f483b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:47:56.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.987+0000 7f483b7fe640 10 monclient: dump: 2024-09-16T03:47:56.008 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:47:56.008 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:56.008 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:56.008 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:56.009 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:47:56.009 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:56.009 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:47:56.009 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:47:56.009 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:47:56.009 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:56.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.987+0000 7f483b7fe640 10 monclient: _finish_auth 0 2024-09-16T03:47:56.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.987+0000 7f483b7fe640 10 monclient: _check_auth_tickets 2024-09-16T03:47:56.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.987+0000 7f483b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:47:56.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.987+0000 7f48590fb640 5 monclient: authenticate success, global_id 764364 2024-09-16T03:47:56.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.987+0000 7f48590fb640 10 monclient: _renew_subs 2024-09-16T03:47:56.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.987+0000 7f483b7fe640 1 -- 172.21.15.42:0/2787208107 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f484c007070 con 0x7f48541cf840 2024-09-16T03:47:56.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.987+0000 7f48590fb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:47:56.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.987+0000 7f48590fb640 1 -- 172.21.15.42:0/2787208107 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f485409c920 con 0x7f48541cf840 2024-09-16T03:47:56.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.987+0000 7f483b7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:47:56.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.987+0000 7f483b7fe640 1 -- 172.21.15.42:0/2787208107 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f484c007460 con 0x7f48541cf840 2024-09-16T03:47:56.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.987+0000 7f483b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:47:56.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.987+0000 7f483b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:47:56.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.987+0000 7f483b7fe640 10 monclient: dump: 2024-09-16T03:47:56.010 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:47:56.010 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:56.010 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:56.010 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:56.010 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:47:56.010 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:56.010 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:47:56.010 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:47:56.011 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:47:56.011 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:56.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.987+0000 7f48590fb640 10 client.764364.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:47:56.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.987+0000 7f48590fb640 10 monclient: _renew_subs 2024-09-16T03:47:56.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.987+0000 7f48590fb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:47:56.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.987+0000 7f48590fb640 1 -- 172.21.15.42:0/2787208107 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48541326e0 con 0x7f48541cf840 2024-09-16T03:47:56.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.987+0000 7f48590fb640 1 librados: init done 2024-09-16T03:47:56.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.987+0000 7f485159a640 4 set_mon_vals no callback set 2024-09-16T03:47:56.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.987+0000 7f483b7fe640 1 -- 172.21.15.42:0/2787208107 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f484c007600 con 0x7f48541cf840 2024-09-16T03:47:56.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.987+0000 7f48590fb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:47:56.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.987+0000 7f48590fb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:47:56.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.987+0000 7f48590fb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:47:56.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.987+0000 7f48590fb640 1 -- 172.21.15.42:0/2787208107 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f485406f6d0 con 0x7f48541cf840 2024-09-16T03:47:56.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.987+0000 7f483b7fe640 1 --2- 172.21.15.42:0/2787208107 >> v2:172.21.15.154:6812/254845342 conn(0x7f48400377c0 0x7f4840039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:56.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.987+0000 7f483b7fe640 1 -- 172.21.15.42:0/2787208107 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(182..182 src has 1..182) ==== 6676+0+0 (secure 0 0 0) 0x7f484c0485a0 con 0x7f48541cf840 2024-09-16T03:47:56.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.987+0000 7f483b7fe640 10 client.764364.objecter ms_dispatch 0x7f4854000ba0 osd_map(182..182 src has 1..182) 2024-09-16T03:47:56.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.987+0000 7f483b7fe640 3 client.764364.objecter handle_osd_map got epochs [182,182] > 0 2024-09-16T03:47:56.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.987+0000 7f483b7fe640 3 client.764364.objecter handle_osd_map decoding full epoch 182 2024-09-16T03:47:56.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.987+0000 7f483b7fe640 20 client.764364.objecter dump_active .. 0 homeless 2024-09-16T03:47:56.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.987+0000 7f4852d9d640 1 --2- 172.21.15.42:0/2787208107 >> v2:172.21.15.154:6812/254845342 conn(0x7f48400377c0 0x7f4840039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:56.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.987+0000 7f4852d9d640 10 monclient: get_auth_request con 0x7f48400377c0 auth_method 0 2024-09-16T03:47:56.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.991+0000 7f4852d9d640 1 --2- 172.21.15.42:0/2787208107 >> v2:172.21.15.154:6812/254845342 conn(0x7f48400377c0 0x7f4840039c80 secure :-1 s=READY pgs=3469 cs=0 l=1 rev1=1 crypto rx=0x7f48440aa000 tx=0x7f4844005040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:47:56.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.991+0000 7f483b7fe640 10 client.764364.objecter ms_handle_connect 0x7f48400377c0 2024-09-16T03:47:56.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.995+0000 7f483b7fe640 1 -- 172.21.15.42:0/2787208107 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f485406f6d0 con 0x7f48541cf840 2024-09-16T03:47:56.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.995+0000 7f483b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:47:56.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:55.995+0000 7f483b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:47:56.224 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:56.219+0000 7f48590fb640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:47:56.224 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:56.219+0000 7f48590fb640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:47:56.224 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:56.219+0000 7f48590fb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:47:56.224 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:56.219+0000 7f48590fb640 1 -- 172.21.15.42:0/2787208107 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f4854136e10 con 0x7f48541cf840 2024-09-16T03:47:56.225 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:56.219+0000 7f483b7fe640 1 -- 172.21.15.42:0/2787208107 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7f4854132c30 con 0x7f48541cf840 2024-09-16T03:47:56.225 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:56.219+0000 7f483b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:47:56.225 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:56.219+0000 7f483b7fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:47:56.225 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":54,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":9,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:47:56.227 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:56.223+0000 7f48590fb640 10 librados: watch_flush enter 2024-09-16T03:47:56.227 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:56.223+0000 7f48590fb640 10 librados: watch_flush exit 2024-09-16T03:47:56.227 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:56.223+0000 7f48590fb640 20 client.764364.objecter shutdown clearing up homeless session... 2024-09-16T03:47:56.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:56.223+0000 7f48590fb640 10 client.764364.objecter successfully canceled tick 2024-09-16T03:47:56.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:56.223+0000 7f48590fb640 1 -- 172.21.15.42:0/2787208107 >> v2:172.21.15.154:6812/254845342 conn(0x7f48400377c0 msgr2=0x7f4840039c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:56.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:56.223+0000 7f48590fb640 1 --2- 172.21.15.42:0/2787208107 >> v2:172.21.15.154:6812/254845342 conn(0x7f48400377c0 0x7f4840039c80 secure :-1 s=READY pgs=3469 cs=0 l=1 rev1=1 crypto rx=0x7f48440aa000 tx=0x7f4844005040 comp rx=0 tx=0).stop 2024-09-16T03:47:56.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:56.223+0000 7f48590fb640 10 monclient: shutdown 2024-09-16T03:47:56.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:56.223+0000 7f48590fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:47:56.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:56.223+0000 7f48590fb640 1 -- 172.21.15.42:0/2787208107 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f48541cf840 msgr2=0x7f48541cfcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:56.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:56.223+0000 7f48590fb640 1 --2- 172.21.15.42:0/2787208107 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f48541cf840 0x7f48541cfcf0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f485409bbd0 tx=0x7f484c00cad0 comp rx=0 tx=0).stop 2024-09-16T03:47:56.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:56.223+0000 7f48590fb640 1 -- 172.21.15.42:0/2787208107 shutdown_connections 2024-09-16T03:47:56.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:56.223+0000 7f48590fb640 1 --2- 172.21.15.42:0/2787208107 >> v2:172.21.15.154:6812/254845342 conn(0x7f48400377c0 0x7f4840039c80 unknown :-1 s=CLOSED pgs=3469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:56.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:56.223+0000 7f48590fb640 1 --2- 172.21.15.42:0/2787208107 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f48541cf840 0x7f48541cfcf0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:56.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:56.223+0000 7f48590fb640 1 --2- 172.21.15.42:0/2787208107 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f48540a33e0 0x7f48541cf300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:56.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:56.223+0000 7f48590fb640 1 --2- 172.21.15.42:0/2787208107 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f48541cbdc0 0x7f48540a2ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:56.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:56.223+0000 7f48590fb640 1 -- 172.21.15.42:0/2787208107 >> 172.21.15.42:0/2787208107 conn(0x7f48540a9fb0 msgr2=0x7f48540a65b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:47:56.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:56.223+0000 7f48590fb640 1 -- 172.21.15.42:0/2787208107 shutdown_connections 2024-09-16T03:47:56.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:56.223+0000 7f48590fb640 1 -- 172.21.15.42:0/2787208107 wait complete. 2024-09-16T03:47:56.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:56.223+0000 7f48590fb640 1 librados: shutdown 2024-09-16T03:47:59.246 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:47:59.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.375+0000 7f6393e22640 10 monclient: get_monmap_and_config 2024-09-16T03:47:59.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.375+0000 7f6393e22640 10 monclient: build_initial_monmap 2024-09-16T03:47:59.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.375+0000 7f6393e22640 10 monclient: monmap: 2024-09-16T03:47:59.379 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:47:59.379 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:59.379 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:47:59.377838+0000 2024-09-16T03:47:59.379 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:47:59.377838+0000 2024-09-16T03:47:59.379 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:47:59.379 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:59.379 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:47:59.379 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:47:59.379 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:47:59.380 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:47:59.380 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:47:59.380 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:59.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.375+0000 7f6393e22640 1 Processor -- start 2024-09-16T03:47:59.382 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.375+0000 7f6393e22640 1 -- start start 2024-09-16T03:47:59.382 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.375+0000 7f6393e22640 10 monclient: init 2024-09-16T03:47:59.382 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.375+0000 7f6393e22640 10 monclient: _reopen_session rank -1 2024-09-16T03:47:59.382 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.375+0000 7f6393e22640 10 monclient: _add_conns ranks=[4,1,0,2,3] 2024-09-16T03:47:59.382 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.375+0000 7f6393e22640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f638c0a1330 0x7f638c09f180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:59.382 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.375+0000 7f6393e22640 10 monclient(hunting): picked mon.noname-c con 0x7f638c0a1330 addr v2:172.21.15.154:3300/0 2024-09-16T03:47:59.382 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.375+0000 7f6393e22640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f638c0a1710 addr v1:172.21.15.154:6789/0 2024-09-16T03:47:59.382 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.375+0000 7f6393e22640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f638c1393a0 addr v1:172.21.15.42:6789/0 2024-09-16T03:47:59.382 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.375+0000 7f6393e22640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f638c09e4c0 con 0x7f638c1393a0 2024-09-16T03:47:59.382 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.375+0000 7f6393e22640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f638c09f6c0 con 0x7f638c0a1710 2024-09-16T03:47:59.382 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.375+0000 7f6393e22640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:59.382 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.375+0000 7f6393e22640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f638c09edf0 con 0x7f638c0a1330 2024-09-16T03:47:59.382 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.375+0000 7f6393e22640 10 monclient(hunting): _renew_subs 2024-09-16T03:47:59.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.375+0000 7f6393e22640 10 monclient(hunting): authenticate will time out at 1326.347321s 2024-09-16T03:47:59.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.375+0000 7f63923bf640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f638c1393a0 0x7f638c13b790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:50706/0 (socket says 172.21.15.42:50706) 2024-09-16T03:47:59.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.375+0000 7f63923bf640 1 -- 172.21.15.42:0/3184462842 learned_addr learned my addr 172.21.15.42:0/3184462842 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:47:59.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.375+0000 7f6391bbe640 1 --2- 172.21.15.42:0/3184462842 >> v2:172.21.15.154:3300/0 conn(0x7f638c0a1330 0x7f638c09f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:59.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.375+0000 7f6391bbe640 10 monclient(hunting): get_auth_request con 0x7f638c0a1330 auth_method 0 2024-09-16T03:47:59.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.375+0000 7f6391bbe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:59.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.375+0000 7f6391bbe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:59.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.375+0000 7f6391bbe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:47:59.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.379+0000 7f6391bbe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:47:59.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.379+0000 7f6391bbe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:47:59.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.379+0000 7f6391bbe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:47:59.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.379+0000 7f6391bbe640 10 monclient(hunting): handle_auth_done global_id 774175 payload 274 2024-09-16T03:47:59.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.379+0000 7f6391bbe640 1 -- 172.21.15.42:0/3184462842 >> v1:172.21.15.154:6789/0 conn(0x7f638c0a1710 legacy=0x7f638c0a1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:59.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.379+0000 7f6391bbe640 1 -- 172.21.15.42:0/3184462842 >> v1:172.21.15.42:6789/0 conn(0x7f638c1393a0 legacy=0x7f638c13b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:59.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.379+0000 7f6391bbe640 10 monclient: _finish_hunting 0 2024-09-16T03:47:59.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.379+0000 7f6391bbe640 1 monclient: found mon.noname-c 2024-09-16T03:47:59.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.379+0000 7f6391bbe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:47:59.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.379+0000 7f6391bbe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.154:3300/0 2024-09-16T03:47:59.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.379+0000 7f6391bbe640 1 -- 172.21.15.42:0/3184462842 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f638c0a3170 con 0x7f638c0a1330 2024-09-16T03:47:59.385 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.379+0000 7f6391bbe640 1 --2- 172.21.15.42:0/3184462842 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f638c0a1330 0x7f638c09f180 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f6388009d90 tx=0x7f638801c830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=44407db0baf9c55 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:47:59.385 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.379+0000 7f6390bbc640 1 -- 172.21.15.42:0/3184462842 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63880b2070 con 0x7f638c0a1330 2024-09-16T03:47:59.385 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.379+0000 7f6390bbc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:47:59.385 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.379+0000 7f6390bbc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:47:59.385 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.379+0000 7f6390bbc640 10 monclient: dump: 2024-09-16T03:47:59.386 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:47:59.386 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:59.386 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:59.386 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:59.386 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:47:59.386 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:59.386 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:47:59.386 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:47:59.386 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:47:59.386 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:59.386 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.379+0000 7f6390bbc640 10 monclient: _finish_auth 0 2024-09-16T03:47:59.386 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.379+0000 7f6390bbc640 10 monclient: _check_auth_tickets 2024-09-16T03:47:59.386 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.379+0000 7f6390bbc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:47:59.387 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.379+0000 7f6390bbc640 1 -- 172.21.15.42:0/3184462842 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f63880a5800 con 0x7f638c0a1330 2024-09-16T03:47:59.387 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.379+0000 7f6390bbc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:47:59.387 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.379+0000 7f6390bbc640 1 -- 172.21.15.42:0/3184462842 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63880a5bd0 con 0x7f638c0a1330 2024-09-16T03:47:59.387 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.379+0000 7f6390bbc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:47:59.387 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.379+0000 7f6390bbc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:47:59.387 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.379+0000 7f6390bbc640 10 monclient: dump: 2024-09-16T03:47:59.387 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:47:59.387 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:59.387 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:59.388 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:59.388 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:47:59.388 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:59.388 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:47:59.388 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:47:59.388 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:47:59.388 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:59.388 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.379+0000 7f6393e22640 5 monclient: authenticate success, global_id 774175 2024-09-16T03:47:59.388 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.379+0000 7f6393e22640 10 monclient: get_monmap_and_config success 2024-09-16T03:47:59.388 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.379+0000 7f6393e22640 4 set_mon_vals no callback set 2024-09-16T03:47:59.388 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.383+0000 7f6393e22640 10 monclient: shutdown 2024-09-16T03:47:59.388 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.383+0000 7f6393e22640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:47:59.388 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.383+0000 7f6393e22640 1 -- 172.21.15.42:0/3184462842 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f638c0a1330 msgr2=0x7f638c09f180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:59.388 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.383+0000 7f6393e22640 1 --2- 172.21.15.42:0/3184462842 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f638c0a1330 0x7f638c09f180 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f6388009d90 tx=0x7f638801c830 comp rx=0 tx=0).stop 2024-09-16T03:47:59.389 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.383+0000 7f6393e22640 1 -- 172.21.15.42:0/3184462842 shutdown_connections 2024-09-16T03:47:59.389 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.383+0000 7f6393e22640 1 --2- 172.21.15.42:0/3184462842 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f638c0a1330 0x7f638c09f180 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:59.389 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.383+0000 7f6393e22640 1 -- 172.21.15.42:0/3184462842 >> 172.21.15.42:0/3184462842 conn(0x7f638c0a2420 msgr2=0x7f638c0a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:47:59.389 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.383+0000 7f6393e22640 1 -- 172.21.15.42:0/3184462842 shutdown_connections 2024-09-16T03:47:59.389 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.383+0000 7f6393e22640 1 -- 172.21.15.42:0/3184462842 wait complete. 2024-09-16T03:47:59.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.383+0000 7f6393e22640 10 monclient: build_initial_monmap 2024-09-16T03:47:59.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.383+0000 7f6393e22640 10 monclient: monmap: 2024-09-16T03:47:59.394 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:47:59.394 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:47:59.394 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:47:59.394 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:47:59.394 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:47:59.394 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:59.395 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:47:59.395 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:47:59.395 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:47:59.395 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:59.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.383+0000 7f6393e22640 1 librados: starting msgr at 2024-09-16T03:47:59.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.383+0000 7f6393e22640 1 librados: starting objecter 2024-09-16T03:47:59.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.383+0000 7f6393e22640 1 Processor -- start 2024-09-16T03:47:59.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.383+0000 7f6393e22640 1 -- start start 2024-09-16T03:47:59.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.383+0000 7f6393e22640 1 librados: setting wanted keys 2024-09-16T03:47:59.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.383+0000 7f6393e22640 1 librados: calling monclient init 2024-09-16T03:47:59.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.383+0000 7f6393e22640 10 monclient: init 2024-09-16T03:47:59.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.383+0000 7f6393e22640 10 monclient: _reopen_session rank -1 2024-09-16T03:47:59.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.383+0000 7f6393e22640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:47:59.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.383+0000 7f6393e22640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f638c0a1330 0x7f638c1343b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:59.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.383+0000 7f6393e22640 10 monclient(hunting): picked mon.noname-c con 0x7f638c0a1330 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:47:59.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.383+0000 7f6393e22640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f638c0a1710 0x7f638c1348f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:59.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.383+0000 7f6393e22640 10 monclient(hunting): picked mon.noname-b con 0x7f638c0a1710 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:47:59.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.383+0000 7f6393e22640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f638c1393a0 0x7f638c11cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:59.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.383+0000 7f6393e22640 10 monclient(hunting): picked mon.noname-a con 0x7f638c1393a0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:47:59.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.383+0000 7f6393e22640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:59.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.383+0000 7f6393e22640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f638c13f8e0 con 0x7f638c1393a0 2024-09-16T03:47:59.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.383+0000 7f6393e22640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:59.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.383+0000 7f6393e22640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f638c09cd60 con 0x7f638c0a1710 2024-09-16T03:47:59.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.383+0000 7f6393e22640 10 monclient(hunting): start opening mon connection 2024-09-16T03:47:59.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.383+0000 7f6393e22640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f638c13fa60 con 0x7f638c0a1330 2024-09-16T03:47:59.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.383+0000 7f6393e22640 10 monclient(hunting): _renew_subs 2024-09-16T03:47:59.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.383+0000 7f6393e22640 10 monclient(hunting): authenticate will time out at 1326.354123s 2024-09-16T03:47:59.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.383+0000 7f6391bbe640 1 -- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f638c0a1330 msgr2=0x7f638c1343b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T03:47:59.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.383+0000 7f6391bbe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f638c0a1330 0x7f638c1343b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:47:59.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.383+0000 7f63923bf640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f638c1393a0 0x7f638c11cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:59.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.383+0000 7f63923bf640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f638c1393a0 0x7f638c11cfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:50940/0 (socket says 172.21.15.42:50940) 2024-09-16T03:47:59.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.383+0000 7f63923bf640 1 -- 172.21.15.42:0/3083491564 learned_addr learned my addr 172.21.15.42:0/3083491564 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:47:59.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.383+0000 7f63923bf640 10 monclient(hunting): get_auth_request con 0x7f638c1393a0 auth_method 0 2024-09-16T03:47:59.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.383+0000 7f63923bf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:59.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.383+0000 7f63923bf640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:59.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.383+0000 7f63923bf640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:47:59.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.383+0000 7f63913bd640 1 --2- 172.21.15.42:0/3083491564 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f638c0a1710 0x7f638c1348f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:59.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.383+0000 7f63923bf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:47:59.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.383+0000 7f63923bf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:47:59.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.383+0000 7f63923bf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:47:59.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.383+0000 7f63913bd640 10 monclient(hunting): get_auth_request con 0x7f638c0a1710 auth_method 0 2024-09-16T03:47:59.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.383+0000 7f63913bd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:47:59.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.383+0000 7f63913bd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:47:59.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.383+0000 7f63913bd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:47:59.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.383+0000 7f63913bd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:47:59.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.383+0000 7f63913bd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:47:59.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.383+0000 7f63913bd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:47:59.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.383+0000 7f63923bf640 10 monclient(hunting): handle_auth_done global_id 764367 payload 824 2024-09-16T03:47:59.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.383+0000 7f63923bf640 1 -- 172.21.15.42:0/3083491564 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f638c0a1330 msgr2=0x7f638c1343b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:47:59.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.383+0000 7f63923bf640 1 --2- 172.21.15.42:0/3083491564 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f638c0a1330 0x7f638c1343b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:59.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.383+0000 7f63923bf640 1 -- 172.21.15.42:0/3083491564 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f638c0a1710 msgr2=0x7f638c1348f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:59.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.383+0000 7f63923bf640 1 --2- 172.21.15.42:0/3083491564 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f638c0a1710 0x7f638c1348f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:59.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.383+0000 7f63923bf640 10 monclient: _finish_hunting 0 2024-09-16T03:47:59.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.383+0000 7f63923bf640 1 monclient: found mon.noname-a 2024-09-16T03:47:59.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.383+0000 7f63923bf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:47:59.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.383+0000 7f63923bf640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:47:59.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.383+0000 7f63923bf640 1 -- 172.21.15.42:0/3083491564 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f638c120a80 con 0x7f638c1393a0 2024-09-16T03:47:59.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.383+0000 7f63923bf640 1 --2- 172.21.15.42:0/3083491564 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f638c1393a0 0x7f638c11cfc0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f638400c400 tx=0x7f638400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:47:59.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.383+0000 7f63827fc640 10 client.?.objecter ms_handle_connect 0x7f638c1393a0 2024-09-16T03:47:59.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.383+0000 7f63827fc640 10 client.?.objecter resend_mon_ops 2024-09-16T03:47:59.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.383+0000 7f63827fc640 1 -- 172.21.15.42:0/3083491564 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6384017070 con 0x7f638c1393a0 2024-09-16T03:47:59.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.383+0000 7f63827fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:47:59.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.383+0000 7f63827fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:47:59.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.383+0000 7f63827fc640 10 monclient: dump: 2024-09-16T03:47:59.400 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:47:59.400 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:59.400 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:59.400 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:59.400 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:47:59.400 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:59.400 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:47:59.400 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:47:59.400 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:47:59.400 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:59.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.383+0000 7f63827fc640 10 monclient: _finish_auth 0 2024-09-16T03:47:59.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.383+0000 7f63827fc640 10 monclient: _check_auth_tickets 2024-09-16T03:47:59.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.383+0000 7f63827fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:47:59.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.383+0000 7f6393e22640 5 monclient: authenticate success, global_id 764367 2024-09-16T03:47:59.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.383+0000 7f63827fc640 1 -- 172.21.15.42:0/3083491564 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6384005cf0 con 0x7f638c1393a0 2024-09-16T03:47:59.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.383+0000 7f6393e22640 10 monclient: _renew_subs 2024-09-16T03:47:59.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.383+0000 7f6393e22640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:47:59.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.383+0000 7f6393e22640 1 -- 172.21.15.42:0/3083491564 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f638c13c580 con 0x7f638c1393a0 2024-09-16T03:47:59.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.383+0000 7f63827fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:47:59.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.383+0000 7f63827fc640 1 -- 172.21.15.42:0/3083491564 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63840072d0 con 0x7f638c1393a0 2024-09-16T03:47:59.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.383+0000 7f63827fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:47:59.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.383+0000 7f63827fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:47:59.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.383+0000 7f63827fc640 10 monclient: dump: 2024-09-16T03:47:59.401 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:47:59.401 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:47:59.402 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:59.402 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:47:59.402 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:47:59.402 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:47:59.402 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:47:59.402 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:47:59.402 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:47:59.402 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:47:59.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.383+0000 7f6383fff640 4 set_mon_vals no callback set 2024-09-16T03:47:59.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.383+0000 7f6393e22640 10 client.764367.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:47:59.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.383+0000 7f6393e22640 10 monclient: _renew_subs 2024-09-16T03:47:59.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.383+0000 7f6393e22640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:47:59.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.383+0000 7f6393e22640 1 -- 172.21.15.42:0/3083491564 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f638c1207a0 con 0x7f638c1393a0 2024-09-16T03:47:59.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.383+0000 7f6393e22640 1 librados: init done 2024-09-16T03:47:59.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.387+0000 7f6393e22640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:47:59.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.387+0000 7f6393e22640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:47:59.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.387+0000 7f6393e22640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:47:59.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.387+0000 7f63827fc640 1 -- 172.21.15.42:0/3083491564 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f638401a070 con 0x7f638c1393a0 2024-09-16T03:47:59.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.387+0000 7f6393e22640 1 -- 172.21.15.42:0/3083491564 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f638c09f180 con 0x7f638c1393a0 2024-09-16T03:47:59.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.387+0000 7f63827fc640 1 --2- 172.21.15.42:0/3083491564 >> v2:172.21.15.154:6812/254845342 conn(0x7f63700377c0 0x7f6370039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:47:59.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.387+0000 7f63827fc640 1 -- 172.21.15.42:0/3083491564 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(182..182 src has 1..182) ==== 6676+0+0 (secure 0 0 0) 0x7f6384048620 con 0x7f638c1393a0 2024-09-16T03:47:59.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.387+0000 7f63827fc640 10 client.764367.objecter ms_dispatch 0x7f638c000ba0 osd_map(182..182 src has 1..182) 2024-09-16T03:47:59.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.387+0000 7f63827fc640 3 client.764367.objecter handle_osd_map got epochs [182,182] > 0 2024-09-16T03:47:59.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.387+0000 7f63827fc640 3 client.764367.objecter handle_osd_map decoding full epoch 182 2024-09-16T03:47:59.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.387+0000 7f63827fc640 20 client.764367.objecter dump_active .. 0 homeless 2024-09-16T03:47:59.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.387+0000 7f6391bbe640 1 --2- 172.21.15.42:0/3083491564 >> v2:172.21.15.154:6812/254845342 conn(0x7f63700377c0 0x7f6370039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:47:59.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.387+0000 7f6391bbe640 10 monclient: get_auth_request con 0x7f63700377c0 auth_method 0 2024-09-16T03:47:59.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.387+0000 7f6391bbe640 1 --2- 172.21.15.42:0/3083491564 >> v2:172.21.15.154:6812/254845342 conn(0x7f63700377c0 0x7f6370039c80 secure :-1 s=READY pgs=3470 cs=0 l=1 rev1=1 crypto rx=0x7f63880af000 tx=0x7f63880a7070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:47:59.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.387+0000 7f63827fc640 10 client.764367.objecter ms_handle_connect 0x7f63700377c0 2024-09-16T03:47:59.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.391+0000 7f63827fc640 1 -- 172.21.15.42:0/3083491564 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f63840076b0 con 0x7f638c1393a0 2024-09-16T03:47:59.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.391+0000 7f63827fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:47:59.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.391+0000 7f63827fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:47:59.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.615+0000 7f6393e22640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:47:59.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.615+0000 7f6393e22640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:47:59.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.615+0000 7f6393e22640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:47:59.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.615+0000 7f6393e22640 1 -- 172.21.15.42:0/3083491564 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f638c11f350 con 0x7f638c1393a0 2024-09-16T03:47:59.619 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.615+0000 7f63827fc640 1 -- 172.21.15.42:0/3083491564 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7f6384007470 con 0x7f638c1393a0 2024-09-16T03:47:59.619 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.615+0000 7f63827fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:47:59.619 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.615+0000 7f63827fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:47:59.619 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":54,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":13,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:47:59.621 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.615+0000 7f6393e22640 10 librados: watch_flush enter 2024-09-16T03:47:59.621 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.615+0000 7f6393e22640 10 librados: watch_flush exit 2024-09-16T03:47:59.621 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.615+0000 7f6393e22640 20 client.764367.objecter shutdown clearing up homeless session... 2024-09-16T03:47:59.622 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.615+0000 7f6393e22640 10 client.764367.objecter successfully canceled tick 2024-09-16T03:47:59.622 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.615+0000 7f6393e22640 1 -- 172.21.15.42:0/3083491564 >> v2:172.21.15.154:6812/254845342 conn(0x7f63700377c0 msgr2=0x7f6370039c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:59.622 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.615+0000 7f6393e22640 1 --2- 172.21.15.42:0/3083491564 >> v2:172.21.15.154:6812/254845342 conn(0x7f63700377c0 0x7f6370039c80 secure :-1 s=READY pgs=3470 cs=0 l=1 rev1=1 crypto rx=0x7f63880af000 tx=0x7f63880a7070 comp rx=0 tx=0).stop 2024-09-16T03:47:59.622 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.615+0000 7f6393e22640 10 monclient: shutdown 2024-09-16T03:47:59.622 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.615+0000 7f6393e22640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:47:59.622 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.615+0000 7f6393e22640 1 -- 172.21.15.42:0/3083491564 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f638c1393a0 msgr2=0x7f638c11cfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:47:59.622 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.615+0000 7f6393e22640 1 --2- 172.21.15.42:0/3083491564 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f638c1393a0 0x7f638c11cfc0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f638400c400 tx=0x7f638400c8c0 comp rx=0 tx=0).stop 2024-09-16T03:47:59.622 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.619+0000 7f6393e22640 1 -- 172.21.15.42:0/3083491564 shutdown_connections 2024-09-16T03:47:59.622 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.619+0000 7f6393e22640 1 --2- 172.21.15.42:0/3083491564 >> v2:172.21.15.154:6812/254845342 conn(0x7f63700377c0 0x7f6370039c80 unknown :-1 s=CLOSED pgs=3470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:59.622 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.619+0000 7f6393e22640 1 --2- 172.21.15.42:0/3083491564 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f638c1393a0 0x7f638c11cfc0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:59.622 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.619+0000 7f6393e22640 1 --2- 172.21.15.42:0/3083491564 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f638c0a1710 0x7f638c1348f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:59.622 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.619+0000 7f6393e22640 1 --2- 172.21.15.42:0/3083491564 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f638c0a1330 0x7f638c1343b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:47:59.622 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.619+0000 7f6393e22640 1 -- 172.21.15.42:0/3083491564 >> 172.21.15.42:0/3083491564 conn(0x7f638c0a2420 msgr2=0x7f638c071e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:47:59.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.619+0000 7f6393e22640 1 -- 172.21.15.42:0/3083491564 shutdown_connections 2024-09-16T03:47:59.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.619+0000 7f6393e22640 1 -- 172.21.15.42:0/3083491564 wait complete. 2024-09-16T03:47:59.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:47:59.619+0000 7f6393e22640 1 librados: shutdown 2024-09-16T03:48:02.638 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:48:02.770 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:02.767+0000 7f87a4899640 10 monclient: get_monmap_and_config 2024-09-16T03:48:02.771 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:02.767+0000 7f87a4899640 10 monclient: build_initial_monmap 2024-09-16T03:48:02.771 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:02.767+0000 7f87a4899640 10 monclient: monmap: 2024-09-16T03:48:02.771 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:48:02.771 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:48:02.771 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:48:02.769863+0000 2024-09-16T03:48:02.771 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:48:02.769863+0000 2024-09-16T03:48:02.771 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:48:02.771 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:48:02.772 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:48:02.772 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:48:02.772 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:48:02.772 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:48:02.772 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:48:02.772 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:48:02.774 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:02.767+0000 7f87a4899640 1 Processor -- start 2024-09-16T03:48:02.774 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:02.767+0000 7f87a4899640 1 -- start start 2024-09-16T03:48:02.774 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:02.767+0000 7f87a4899640 10 monclient: init 2024-09-16T03:48:02.774 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:02.767+0000 7f87a4899640 10 monclient: _reopen_session rank -1 2024-09-16T03:48:02.774 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:02.767+0000 7f87a4899640 10 monclient: _add_conns ranks=[3,4,1,0,2] 2024-09-16T03:48:02.774 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:02.767+0000 7f87a4899640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f87a006db90 0x7f87a01340c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:02.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:02.767+0000 7f87a4899640 10 monclient(hunting): picked mon.noname-a con 0x7f87a006db90 addr v2:172.21.15.42:3300/0 2024-09-16T03:48:02.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:02.767+0000 7f87a4899640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f87a00a37f0 0x7f87a00a3c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:02.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:02.767+0000 7f87a4899640 10 monclient(hunting): picked mon.noname-c con 0x7f87a00a37f0 addr v2:172.21.15.154:3300/0 2024-09-16T03:48:02.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:02.767+0000 7f87a4899640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f87a0136e10 addr v1:172.21.15.154:6789/0 2024-09-16T03:48:02.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:02.767+0000 7f87a4899640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f87a006ece0 con 0x7f87a0136e10 2024-09-16T03:48:02.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:02.767+0000 7f87a4899640 10 monclient(hunting): start opening mon connection 2024-09-16T03:48:02.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:02.767+0000 7f87a4899640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f87a006df70 con 0x7f87a006db90 2024-09-16T03:48:02.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:02.767+0000 7f87a4899640 10 monclient(hunting): start opening mon connection 2024-09-16T03:48:02.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:02.767+0000 7f87a4899640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f87a009bf80 con 0x7f87a00a37f0 2024-09-16T03:48:02.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:02.767+0000 7f87a4899640 10 monclient(hunting): _renew_subs 2024-09-16T03:48:02.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:02.767+0000 7f87a4899640 10 monclient(hunting): authenticate will time out at 1329.739482s 2024-09-16T03:48:02.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:02.767+0000 7f879e59c640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f87a006db90 0x7f87a01340c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:48:02.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:02.767+0000 7f879e59c640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f87a006db90 0x7f87a01340c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:50946/0 (socket says 172.21.15.42:50946) 2024-09-16T03:48:02.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:02.767+0000 7f879e59c640 1 -- 172.21.15.42:0/3353013708 learned_addr learned my addr 172.21.15.42:0/3353013708 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:48:02.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:02.767+0000 7f879e59c640 10 monclient(hunting): get_auth_request con 0x7f87a006db90 auth_method 0 2024-09-16T03:48:02.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:02.767+0000 7f879e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:48:02.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:02.767+0000 7f879e59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:48:02.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:02.767+0000 7f879e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:48:02.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:02.767+0000 7f879e59c640 1 -- 172.21.15.42:0/3353013708 >> v2:172.21.15.42:3300/0 conn(0x7f87a006db90 msgr2=0x7f87a01340c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T03:48:02.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:02.767+0000 7f879e59c640 1 -- 172.21.15.42:0/3353013708 >> v2:172.21.15.42:3300/0 conn(0x7f87a006db90 msgr2=0x7f87a01340c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:48:02.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:02.767+0000 7f879e59c640 1 --2- 172.21.15.42:0/3353013708 >> v2:172.21.15.42:3300/0 conn(0x7f87a006db90 0x7f87a01340c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:48:02.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:02.767+0000 7f879e59c640 1 --2- 172.21.15.42:0/3353013708 >> v2:172.21.15.42:3300/0 conn(0x7f87a006db90 0x7f87a01340c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:48:02.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:02.771+0000 7f879dd9b640 1 --2- 172.21.15.42:0/3353013708 >> v2:172.21.15.154:3300/0 conn(0x7f87a00a37f0 0x7f87a00a3c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:48:02.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:02.771+0000 7f879dd9b640 10 monclient(hunting): get_auth_request con 0x7f87a00a37f0 auth_method 0 2024-09-16T03:48:02.777 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:02.771+0000 7f879dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:48:02.777 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:02.771+0000 7f879dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:48:02.777 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:02.771+0000 7f879dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:48:02.777 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:02.771+0000 7f879dd9b640 1 -- 172.21.15.42:0/3353013708 >> v2:172.21.15.154:3300/0 conn(0x7f87a00a37f0 msgr2=0x7f87a00a3c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T03:48:02.777 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:02.771+0000 7f879dd9b640 1 -- 172.21.15.42:0/3353013708 >> v2:172.21.15.154:3300/0 conn(0x7f87a00a37f0 msgr2=0x7f87a00a3c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:48:02.777 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:02.771+0000 7f879dd9b640 1 --2- 172.21.15.42:0/3353013708 >> v2:172.21.15.154:3300/0 conn(0x7f87a00a37f0 0x7f87a00a3c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:48:02.777 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:02.771+0000 7f879dd9b640 1 --2- 172.21.15.42:0/3353013708 >> v2:172.21.15.154:3300/0 conn(0x7f87a00a37f0 0x7f87a00a3c00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:48:02.974 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:02.971+0000 7f879e59c640 1 --2- 172.21.15.42:0/3353013708 >> v2:172.21.15.42:3300/0 conn(0x7f87a006db90 0x7f87a01340c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:48:02.974 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:02.971+0000 7f879e59c640 10 monclient(hunting): get_auth_request con 0x7f87a006db90 auth_method 0 2024-09-16T03:48:02.974 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:02.971+0000 7f879e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:48:02.974 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:02.971+0000 7f879e59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:48:02.974 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:02.971+0000 7f879e59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:48:02.974 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:02.971+0000 7f879e59c640 1 -- 172.21.15.42:0/3353013708 >> v2:172.21.15.42:3300/0 conn(0x7f87a006db90 msgr2=0x7f87a01340c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T03:48:02.974 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:02.971+0000 7f879e59c640 1 -- 172.21.15.42:0/3353013708 >> v2:172.21.15.42:3300/0 conn(0x7f87a006db90 msgr2=0x7f87a01340c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:48:02.974 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:02.971+0000 7f879e59c640 1 --2- 172.21.15.42:0/3353013708 >> v2:172.21.15.42:3300/0 conn(0x7f87a006db90 0x7f87a01340c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:48:02.974 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:02.971+0000 7f879e59c640 1 --2- 172.21.15.42:0/3353013708 >> v2:172.21.15.42:3300/0 conn(0x7f87a006db90 0x7f87a01340c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T03:48:02.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:02.975+0000 7f879dd9b640 1 --2- 172.21.15.42:0/3353013708 >> v2:172.21.15.154:3300/0 conn(0x7f87a00a37f0 0x7f87a00a3c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:48:02.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:02.975+0000 7f879dd9b640 10 monclient(hunting): get_auth_request con 0x7f87a00a37f0 auth_method 0 2024-09-16T03:48:02.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:02.975+0000 7f879dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:48:02.978 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:02.975+0000 7f879dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:48:02.978 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:02.975+0000 7f879dd9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:48:02.978 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:02.975+0000 7f879dd9b640 1 -- 172.21.15.42:0/3353013708 >> v2:172.21.15.154:3300/0 conn(0x7f87a00a37f0 msgr2=0x7f87a00a3c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T03:48:02.978 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:02.975+0000 7f879dd9b640 1 -- 172.21.15.42:0/3353013708 >> v2:172.21.15.154:3300/0 conn(0x7f87a00a37f0 msgr2=0x7f87a00a3c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:48:02.978 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:02.975+0000 7f879dd9b640 1 --2- 172.21.15.42:0/3353013708 >> v2:172.21.15.154:3300/0 conn(0x7f87a00a37f0 0x7f87a00a3c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:48:02.978 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:02.975+0000 7f879dd9b640 1 --2- 172.21.15.42:0/3353013708 >> v2:172.21.15.154:3300/0 conn(0x7f87a00a37f0 0x7f87a00a3c00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T03:48:03.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:03.375+0000 7f879e59c640 1 --2- 172.21.15.42:0/3353013708 >> v2:172.21.15.42:3300/0 conn(0x7f87a006db90 0x7f87a01340c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:48:03.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:03.375+0000 7f879e59c640 10 monclient(hunting): get_auth_request con 0x7f87a006db90 auth_method 0 2024-09-16T03:48:03.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:03.375+0000 7f879e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:48:03.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:03.375+0000 7f879e59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:48:03.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:03.375+0000 7f879e59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:48:03.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:03.375+0000 7f879e59c640 1 -- 172.21.15.42:0/3353013708 >> v2:172.21.15.42:3300/0 conn(0x7f87a006db90 msgr2=0x7f87a01340c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T03:48:03.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:03.375+0000 7f879e59c640 1 -- 172.21.15.42:0/3353013708 >> v2:172.21.15.42:3300/0 conn(0x7f87a006db90 msgr2=0x7f87a01340c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:48:03.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:03.375+0000 7f879e59c640 1 --2- 172.21.15.42:0/3353013708 >> v2:172.21.15.42:3300/0 conn(0x7f87a006db90 0x7f87a01340c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:48:03.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:03.375+0000 7f879e59c640 1 --2- 172.21.15.42:0/3353013708 >> v2:172.21.15.42:3300/0 conn(0x7f87a006db90 0x7f87a01340c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T03:48:03.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:03.379+0000 7f879dd9b640 1 --2- 172.21.15.42:0/3353013708 >> v2:172.21.15.154:3300/0 conn(0x7f87a00a37f0 0x7f87a00a3c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:48:03.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:03.379+0000 7f879dd9b640 10 monclient(hunting): get_auth_request con 0x7f87a00a37f0 auth_method 0 2024-09-16T03:48:03.382 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:03.379+0000 7f879dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:48:03.382 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:03.379+0000 7f879dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:48:03.382 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:03.379+0000 7f879dd9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:48:03.382 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:03.379+0000 7f879dd9b640 1 -- 172.21.15.42:0/3353013708 >> v2:172.21.15.154:3300/0 conn(0x7f87a00a37f0 msgr2=0x7f87a00a3c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T03:48:03.382 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:03.379+0000 7f879dd9b640 1 -- 172.21.15.42:0/3353013708 >> v2:172.21.15.154:3300/0 conn(0x7f87a00a37f0 msgr2=0x7f87a00a3c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:48:03.382 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:03.379+0000 7f879dd9b640 1 --2- 172.21.15.42:0/3353013708 >> v2:172.21.15.154:3300/0 conn(0x7f87a00a37f0 0x7f87a00a3c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:48:03.382 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:03.379+0000 7f879dd9b640 1 --2- 172.21.15.42:0/3353013708 >> v2:172.21.15.154:3300/0 conn(0x7f87a00a37f0 0x7f87a00a3c00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T03:48:04.181 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:04.179+0000 7f879e59c640 1 --2- 172.21.15.42:0/3353013708 >> v2:172.21.15.42:3300/0 conn(0x7f87a006db90 0x7f87a01340c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:48:04.181 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:04.179+0000 7f879e59c640 10 monclient(hunting): get_auth_request con 0x7f87a006db90 auth_method 0 2024-09-16T03:48:04.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:04.179+0000 7f879e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:48:04.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:04.179+0000 7f879e59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:48:04.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:04.179+0000 7f879e59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:48:04.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:04.179+0000 7f879e59c640 1 -- 172.21.15.42:0/3353013708 >> v2:172.21.15.42:3300/0 conn(0x7f87a006db90 msgr2=0x7f87a01340c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T03:48:04.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:04.179+0000 7f879e59c640 1 -- 172.21.15.42:0/3353013708 >> v2:172.21.15.42:3300/0 conn(0x7f87a006db90 msgr2=0x7f87a01340c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:48:04.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:04.179+0000 7f879e59c640 1 --2- 172.21.15.42:0/3353013708 >> v2:172.21.15.42:3300/0 conn(0x7f87a006db90 0x7f87a01340c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:48:04.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:04.179+0000 7f879e59c640 1 --2- 172.21.15.42:0/3353013708 >> v2:172.21.15.42:3300/0 conn(0x7f87a006db90 0x7f87a01340c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T03:48:04.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:04.183+0000 7f879dd9b640 1 --2- 172.21.15.42:0/3353013708 >> v2:172.21.15.154:3300/0 conn(0x7f87a00a37f0 0x7f87a00a3c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:48:04.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:04.183+0000 7f879dd9b640 10 monclient(hunting): get_auth_request con 0x7f87a00a37f0 auth_method 0 2024-09-16T03:48:04.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:04.183+0000 7f879dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:48:04.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:04.183+0000 7f879dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:48:04.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:04.183+0000 7f879dd9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:48:04.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:04.183+0000 7f879dd9b640 1 -- 172.21.15.42:0/3353013708 >> v2:172.21.15.154:3300/0 conn(0x7f87a00a37f0 msgr2=0x7f87a00a3c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T03:48:04.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:04.183+0000 7f879dd9b640 1 -- 172.21.15.42:0/3353013708 >> v2:172.21.15.154:3300/0 conn(0x7f87a00a37f0 msgr2=0x7f87a00a3c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:48:04.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:04.183+0000 7f879dd9b640 1 --2- 172.21.15.42:0/3353013708 >> v2:172.21.15.154:3300/0 conn(0x7f87a00a37f0 0x7f87a00a3c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:48:04.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:04.183+0000 7f879dd9b640 1 --2- 172.21.15.42:0/3353013708 >> v2:172.21.15.154:3300/0 conn(0x7f87a00a37f0 0x7f87a00a3c00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T03:48:05.773 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:05.767+0000 7f878ffff640 10 monclient(hunting): tick 2024-09-16T03:48:05.773 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:05.767+0000 7f878ffff640 10 monclient(hunting): _check_auth_tickets 2024-09-16T03:48:05.773 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:05.767+0000 7f878ffff640 1 monclient(hunting): continuing hunt 2024-09-16T03:48:05.773 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:05.767+0000 7f878ffff640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T03:48:05.773 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:05.767+0000 7f878ffff640 1 -- 172.21.15.42:0/3353013708 >> v2:172.21.15.154:3300/0 conn(0x7f87a00a37f0 msgr2=0x7f87a00a3c00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:48:05.773 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:05.767+0000 7f878ffff640 1 --2- 172.21.15.42:0/3353013708 >> v2:172.21.15.154:3300/0 conn(0x7f87a00a37f0 0x7f87a00a3c00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:05.773 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:05.767+0000 7f878ffff640 1 -- 172.21.15.42:0/3353013708 >> v2:172.21.15.42:3300/0 conn(0x7f87a006db90 msgr2=0x7f87a01340c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:48:05.773 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:05.767+0000 7f878ffff640 1 --2- 172.21.15.42:0/3353013708 >> v2:172.21.15.42:3300/0 conn(0x7f87a006db90 0x7f87a01340c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:05.773 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:05.767+0000 7f878ffff640 1 -- 172.21.15.42:0/3353013708 >> v1:172.21.15.154:6789/0 conn(0x7f87a0136e10 legacy=0x7f87a0137220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:48:05.774 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:05.767+0000 7f878ffff640 10 monclient: _add_conns ranks=[2,0] 2024-09-16T03:48:05.774 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:05.767+0000 7f878ffff640 1 --2- 172.21.15.42:0/3353013708 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f87800036c0 0x7f8780005ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:05.774 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:05.767+0000 7f878ffff640 10 monclient(hunting): picked mon.noname-b con 0x7f87800036c0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:48:05.774 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:05.767+0000 7f878ffff640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8780006200 addr v1:172.21.15.42:6789/0 2024-09-16T03:48:05.774 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:05.767+0000 7f878ffff640 1 -- 172.21.15.42:0/3353013708 --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f87a0134780 con 0x7f8780006200 2024-09-16T03:48:05.774 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:05.767+0000 7f878ffff640 10 monclient(hunting): start opening mon connection 2024-09-16T03:48:05.774 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:05.767+0000 7f878ffff640 1 -- 172.21.15.42:0/3353013708 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f87a006df70 con 0x7f87800036c0 2024-09-16T03:48:05.774 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:05.767+0000 7f878ffff640 10 monclient(hunting): _renew_subs 2024-09-16T03:48:05.774 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:05.767+0000 7f879e59c640 1 -- 172.21.15.42:0/3353013708 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f87800036c0 msgr2=0x7f8780005ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T03:48:05.774 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:05.767+0000 7f879e59c640 1 --2- 172.21.15.42:0/3353013708 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f87800036c0 0x7f8780005ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:48:05.973 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:05.971+0000 7f879e59c640 1 -- 172.21.15.42:0/3353013708 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f87800036c0 msgr2=0x7f8780005ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T03:48:05.973 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:05.971+0000 7f879e59c640 1 --2- 172.21.15.42:0/3353013708 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f87800036c0 0x7f8780005ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T03:48:06.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:06.375+0000 7f879e59c640 1 -- 172.21.15.42:0/3353013708 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f87800036c0 msgr2=0x7f8780005ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T03:48:06.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:06.375+0000 7f879e59c640 1 --2- 172.21.15.42:0/3353013708 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f87800036c0 0x7f8780005ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T03:48:07.181 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.179+0000 7f879e59c640 1 -- 172.21.15.42:0/3353013708 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f87800036c0 msgr2=0x7f8780005ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T03:48:07.181 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.179+0000 7f879e59c640 1 --2- 172.21.15.42:0/3353013708 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f87800036c0 0x7f8780005ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T03:48:07.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.395+0000 7f879d59a640 1 -- 172.21.15.42:0/3353013708 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2683974979 0 0) 0x7f87a0134780 con 0x7f8780006200 2024-09-16T03:48:07.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.395+0000 7f879d59a640 10 monclient(hunting): _negotiate 2024-09-16T03:48:07.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.395+0000 7f879d59a640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:48:07.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.395+0000 7f879d59a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:48:07.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.395+0000 7f879d59a640 10 monclient(hunting): my global_id is 764370 2024-09-16T03:48:07.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.395+0000 7f879d59a640 1 -- 172.21.15.42:0/3353013708 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8788003530 con 0x7f8780006200 2024-09-16T03:48:07.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.403+0000 7f879d59a640 1 -- 172.21.15.42:0/3353013708 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 871823485 0 0) 0x7f8788003530 con 0x7f8780006200 2024-09-16T03:48:07.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.403+0000 7f879d59a640 1 -- 172.21.15.42:0/3353013708 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f87800036c0 msgr2=0x7f8780005ad0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:48:07.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.403+0000 7f879d59a640 1 --2- 172.21.15.42:0/3353013708 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f87800036c0 0x7f8780005ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:07.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.403+0000 7f879d59a640 10 monclient: _finish_hunting 0 2024-09-16T03:48:07.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.403+0000 7f879d59a640 1 monclient: found mon.noname-a-legacy 2024-09-16T03:48:07.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.403+0000 7f879d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:48:07.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.403+0000 7f879d59a640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.42:6789/0 2024-09-16T03:48:07.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.403+0000 7f879d59a640 1 -- 172.21.15.42:0/3353013708 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f878000b040 con 0x7f8780006200 2024-09-16T03:48:07.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.403+0000 7f879d59a640 10 monclient: _finish_auth 0 2024-09-16T03:48:07.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.403+0000 7f879d59a640 10 monclient: _check_auth_tickets 2024-09-16T03:48:07.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.403+0000 7f879d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:48:07.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.403+0000 7f879d59a640 1 -- 172.21.15.42:0/3353013708 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2177413734 0 0) 0x7f8790009a00 con 0x7f8780006200 2024-09-16T03:48:07.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.403+0000 7f879d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:48:07.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.403+0000 7f879d59a640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T03:48:07.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.403+0000 7f879d59a640 10 monclient: dump: 2024-09-16T03:48:07.409 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:48:07.409 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:48:07.409 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:07.409 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:07.409 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:48:07.409 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:48:07.409 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:48:07.409 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:48:07.409 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:48:07.409 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:48:07.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.403+0000 7f879d59a640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] but i'm connected to v1:172.21.15.42:6789/0, reconnecting 2024-09-16T03:48:07.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.403+0000 7f879d59a640 10 monclient: _reopen_session rank -1 2024-09-16T03:48:07.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.403+0000 7f879d59a640 1 -- 172.21.15.42:0/3353013708 >> v1:172.21.15.42:6789/0 conn(0x7f8780006200 legacy=0x7f87800086a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:48:07.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.403+0000 7f879d59a640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:48:07.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.403+0000 7f879ed9d640 1 -- 172.21.15.42:0/3353013708 reap_dead start 2024-09-16T03:48:07.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.403+0000 7f879d59a640 1 --2- 172.21.15.42:0/3353013708 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f87880039c0 0x7f8788003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:07.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.403+0000 7f879d59a640 10 monclient(hunting): picked mon.a con 0x7f87880039c0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:48:07.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.403+0000 7f879d59a640 1 --2- 172.21.15.42:0/3353013708 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f87800036c0 0x7f8788004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:07.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.403+0000 7f879d59a640 10 monclient(hunting): picked mon.c con 0x7f87800036c0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:48:07.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.403+0000 7f879d59a640 1 --2- 172.21.15.42:0/3353013708 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8788004da0 0x7f8788005220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:07.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.403+0000 7f879d59a640 10 monclient(hunting): picked mon.b con 0x7f8788004da0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:48:07.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.403+0000 7f879d59a640 10 monclient(hunting): start opening mon connection 2024-09-16T03:48:07.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.403+0000 7f879d59a640 1 -- 172.21.15.42:0/3353013708 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f87a006df70 con 0x7f87880039c0 2024-09-16T03:48:07.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.403+0000 7f879d59a640 10 monclient(hunting): start opening mon connection 2024-09-16T03:48:07.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.403+0000 7f879d59a640 1 -- 172.21.15.42:0/3353013708 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f878800bc50 con 0x7f8788004da0 2024-09-16T03:48:07.411 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.403+0000 7f879d59a640 10 monclient(hunting): start opening mon connection 2024-09-16T03:48:07.411 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.403+0000 7f879d59a640 1 -- 172.21.15.42:0/3353013708 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f878800be50 con 0x7f87800036c0 2024-09-16T03:48:07.411 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.403+0000 7f879d59a640 10 monclient(hunting): _renew_subs 2024-09-16T03:48:07.411 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.403+0000 7f879d59a640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:48:07.411 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.403+0000 7f879e59c640 1 --2- 172.21.15.42:0/3353013708 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f87880039c0 0x7f8788003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:48:07.411 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.403+0000 7f879dd9b640 1 -- 172.21.15.42:0/3353013708 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f87800036c0 msgr2=0x7f8788004810 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T03:48:07.411 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.403+0000 7f879dd9b640 1 --2- 172.21.15.42:0/3353013708 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f87800036c0 0x7f8788004810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:48:07.411 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.403+0000 7f879e59c640 10 monclient(hunting): get_auth_request con 0x7f87880039c0 auth_method 0 2024-09-16T03:48:07.411 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.403+0000 7f879e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:48:07.411 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.403+0000 7f879e59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:48:07.411 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.403+0000 7f879e59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:48:07.411 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.403+0000 7f879ed9d640 1 --2- 172.21.15.42:0/3353013708 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8788004da0 0x7f8788005220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:48:07.411 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.403+0000 7f879e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:48:07.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.403+0000 7f879e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:48:07.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.403+0000 7f879e59c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T03:48:07.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.403+0000 7f879ed9d640 10 monclient(hunting): get_auth_request con 0x7f8788004da0 auth_method 0 2024-09-16T03:48:07.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.403+0000 7f879ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:48:07.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.403+0000 7f879ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:48:07.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.403+0000 7f879ed9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:48:07.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.403+0000 7f879ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:48:07.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.403+0000 7f879ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:48:07.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.403+0000 7f879ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T03:48:07.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.403+0000 7f879e59c640 10 monclient(hunting): handle_auth_done global_id 764370 payload 293 2024-09-16T03:48:07.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.403+0000 7f879e59c640 1 -- 172.21.15.42:0/3353013708 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f87800036c0 msgr2=0x7f8788004810 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:48:07.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.403+0000 7f879e59c640 1 --2- 172.21.15.42:0/3353013708 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f87800036c0 0x7f8788004810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:07.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.403+0000 7f879e59c640 1 -- 172.21.15.42:0/3353013708 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8788004da0 msgr2=0x7f8788005220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:48:07.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.403+0000 7f879e59c640 1 --2- 172.21.15.42:0/3353013708 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8788004da0 0x7f8788005220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:07.413 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.403+0000 7f879e59c640 10 monclient: _finish_hunting 0 2024-09-16T03:48:07.413 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.403+0000 7f879e59c640 1 monclient: found mon.a 2024-09-16T03:48:07.413 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.403+0000 7f879e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:48:07.413 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.403+0000 7f879e59c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:48:07.413 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.403+0000 7f879e59c640 1 -- 172.21.15.42:0/3353013708 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8788003530 con 0x7f87880039c0 2024-09-16T03:48:07.413 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.403+0000 7f879e59c640 10 monclient: _finish_auth 0 2024-09-16T03:48:07.413 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.403+0000 7f879e59c640 10 monclient: _check_auth_tickets 2024-09-16T03:48:07.413 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.403+0000 7f879e59c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:48:07.413 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.403+0000 7f87a4899640 5 monclient: authenticate success, global_id 764370 2024-09-16T03:48:07.413 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.403+0000 7f87a4899640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:48:07.415 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.407+0000 7f879e59c640 1 --2- 172.21.15.42:0/3353013708 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f87880039c0 0x7f8788003dc0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f87940037d0 tx=0x7f879401d400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=928e548d326af9a1 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:48:07.415 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.407+0000 7f879d59a640 1 -- 172.21.15.42:0/3353013708 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87940b3070 con 0x7f87880039c0 2024-09-16T03:48:07.415 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.407+0000 7f879d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:48:07.415 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.407+0000 7f879d59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:48:07.415 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.407+0000 7f879d59a640 10 monclient: dump: 2024-09-16T03:48:07.415 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:48:07.415 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:48:07.415 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:07.415 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:07.415 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:48:07.416 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:48:07.416 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:48:07.416 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:48:07.416 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:48:07.416 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:48:07.416 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.407+0000 7f879d59a640 1 -- 172.21.15.42:0/3353013708 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f87940ae410 con 0x7f87880039c0 2024-09-16T03:48:07.416 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.407+0000 7f87a4899640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:48:07.416 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.407+0000 7f879d59a640 10 monclient: handle_config config(0 keys) 2024-09-16T03:48:07.416 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.407+0000 7f879d59a640 1 -- 172.21.15.42:0/3353013708 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f879400f040 con 0x7f87880039c0 2024-09-16T03:48:07.416 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.407+0000 7f87a4899640 10 monclient: get_monmap_and_config success 2024-09-16T03:48:07.416 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.407+0000 7f87a4899640 4 set_mon_vals no callback set 2024-09-16T03:48:07.416 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.407+0000 7f879d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:48:07.416 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.407+0000 7f879d59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:48:07.417 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.407+0000 7f879d59a640 10 monclient: dump: 2024-09-16T03:48:07.417 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:48:07.417 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:48:07.417 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:07.417 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:07.417 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:48:07.417 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:48:07.417 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:48:07.417 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:48:07.417 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:48:07.417 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:48:07.417 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.407+0000 7f87a4899640 10 monclient: shutdown 2024-09-16T03:48:07.417 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.407+0000 7f87a4899640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:48:07.417 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.407+0000 7f87a4899640 1 -- 172.21.15.42:0/3353013708 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f87880039c0 msgr2=0x7f8788003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:48:07.418 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.407+0000 7f87a4899640 1 --2- 172.21.15.42:0/3353013708 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f87880039c0 0x7f8788003dc0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f87940037d0 tx=0x7f879401d400 comp rx=0 tx=0).stop 2024-09-16T03:48:07.418 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.411+0000 7f87a4899640 1 -- 172.21.15.42:0/3353013708 shutdown_connections 2024-09-16T03:48:07.418 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.411+0000 7f87a4899640 1 --2- 172.21.15.42:0/3353013708 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8788004da0 0x7f8788005220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:07.418 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.411+0000 7f87a4899640 1 --2- 172.21.15.42:0/3353013708 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f87800036c0 0x7f8788004810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:07.418 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.411+0000 7f87a4899640 1 --2- 172.21.15.42:0/3353013708 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f87880039c0 0x7f8788003dc0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:07.418 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.411+0000 7f87a4899640 1 -- 172.21.15.42:0/3353013708 >> 172.21.15.42:0/3353013708 conn(0x7f87a00a9fb0 msgr2=0x7f87a00aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:48:07.418 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.411+0000 7f87a4899640 1 -- 172.21.15.42:0/3353013708 shutdown_connections 2024-09-16T03:48:07.418 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.411+0000 7f87a4899640 1 -- 172.21.15.42:0/3353013708 wait complete. 2024-09-16T03:48:07.418 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.411+0000 7f87a4899640 10 monclient: build_initial_monmap 2024-09-16T03:48:07.418 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.411+0000 7f87a4899640 10 monclient: monmap: 2024-09-16T03:48:07.418 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:48:07.418 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:48:07.418 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:48:07.418 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:48:07.419 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:48:07.419 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:48:07.419 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:48:07.419 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:48:07.419 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:48:07.419 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:48:07.419 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.411+0000 7f87a4899640 1 librados: starting msgr at 2024-09-16T03:48:07.419 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.411+0000 7f87a4899640 1 librados: starting objecter 2024-09-16T03:48:07.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.411+0000 7f87a4899640 1 Processor -- start 2024-09-16T03:48:07.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.411+0000 7f87a4899640 1 -- start start 2024-09-16T03:48:07.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.411+0000 7f87a4899640 1 librados: setting wanted keys 2024-09-16T03:48:07.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.411+0000 7f87a4899640 1 librados: calling monclient init 2024-09-16T03:48:07.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.411+0000 7f87a4899640 10 monclient: init 2024-09-16T03:48:07.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.411+0000 7f87a4899640 10 monclient: _reopen_session rank -1 2024-09-16T03:48:07.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.411+0000 7f87a4899640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:48:07.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.411+0000 7f87a4899640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f87800036c0 0x7f87a01ce570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:07.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.411+0000 7f87a4899640 10 monclient(hunting): picked mon.noname-c con 0x7f87800036c0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:48:07.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.415+0000 7f87a4899640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f87880039c0 0x7f87a01ceab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:07.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.415+0000 7f87a4899640 10 monclient(hunting): picked mon.noname-a con 0x7f87880039c0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:48:07.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.415+0000 7f87a4899640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8788004da0 0x7f87a01cf2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:07.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.415+0000 7f87a4899640 10 monclient(hunting): picked mon.noname-b con 0x7f8788004da0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:48:07.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.415+0000 7f87a4899640 10 monclient(hunting): start opening mon connection 2024-09-16T03:48:07.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.415+0000 7f87a4899640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f87a0137bb0 con 0x7f87880039c0 2024-09-16T03:48:07.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.415+0000 7f87a4899640 10 monclient(hunting): start opening mon connection 2024-09-16T03:48:07.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.415+0000 7f87a4899640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f87a0137a30 con 0x7f8788004da0 2024-09-16T03:48:07.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.415+0000 7f87a4899640 10 monclient(hunting): start opening mon connection 2024-09-16T03:48:07.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.415+0000 7f87a4899640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f87a013a180 con 0x7f87800036c0 2024-09-16T03:48:07.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.415+0000 7f87a4899640 10 monclient(hunting): _renew_subs 2024-09-16T03:48:07.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.415+0000 7f87a4899640 10 monclient(hunting): authenticate will time out at 1334.384113s 2024-09-16T03:48:07.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.415+0000 7f879e59c640 1 -- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f87800036c0 msgr2=0x7f87a01ce570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T03:48:07.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.415+0000 7f879e59c640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f87800036c0 0x7f87a01ce570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:48:07.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.415+0000 7f879dd9b640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f87880039c0 0x7f87a01ceab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:48:07.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.415+0000 7f879dd9b640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f87880039c0 0x7f87a01ceab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:60778/0 (socket says 172.21.15.42:60778) 2024-09-16T03:48:07.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.415+0000 7f879dd9b640 1 -- 172.21.15.42:0/1486097439 learned_addr learned my addr 172.21.15.42:0/1486097439 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:48:07.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.415+0000 7f879dd9b640 10 monclient(hunting): get_auth_request con 0x7f87880039c0 auth_method 0 2024-09-16T03:48:07.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.415+0000 7f879dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:48:07.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.415+0000 7f879dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:48:07.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.415+0000 7f879dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:48:07.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.415+0000 7f879ed9d640 1 --2- 172.21.15.42:0/1486097439 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8788004da0 0x7f87a01cf2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:48:07.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.415+0000 7f879dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:48:07.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.415+0000 7f879dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:48:07.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.415+0000 7f879dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:48:07.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.415+0000 7f879ed9d640 10 monclient(hunting): get_auth_request con 0x7f8788004da0 auth_method 0 2024-09-16T03:48:07.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.415+0000 7f879ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:48:07.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.415+0000 7f879ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:48:07.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.415+0000 7f879ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:48:07.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.415+0000 7f879dd9b640 10 monclient(hunting): handle_auth_done global_id 764373 payload 824 2024-09-16T03:48:07.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.415+0000 7f879dd9b640 1 -- 172.21.15.42:0/1486097439 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f87800036c0 msgr2=0x7f87a01ce570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:48:07.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.415+0000 7f879dd9b640 1 --2- 172.21.15.42:0/1486097439 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f87800036c0 0x7f87a01ce570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:07.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.415+0000 7f879dd9b640 1 -- 172.21.15.42:0/1486097439 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8788004da0 msgr2=0x7f87a01cf2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:48:07.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.415+0000 7f879dd9b640 1 --2- 172.21.15.42:0/1486097439 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8788004da0 0x7f87a01cf2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:07.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.415+0000 7f879dd9b640 10 monclient: _finish_hunting 0 2024-09-16T03:48:07.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.415+0000 7f879dd9b640 1 monclient: found mon.noname-a 2024-09-16T03:48:07.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.415+0000 7f879dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:48:07.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.415+0000 7f879dd9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:48:07.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.415+0000 7f879dd9b640 1 -- 172.21.15.42:0/1486097439 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87a0122e00 con 0x7f87880039c0 2024-09-16T03:48:07.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.415+0000 7f879ed9d640 1 --2- 172.21.15.42:0/1486097439 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8788004da0 0x7f87a01cf2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T03:48:07.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.415+0000 7f879dd9b640 1 --2- 172.21.15.42:0/1486097439 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f87880039c0 0x7f87a01ceab0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f8790003f00 tx=0x7f8790011600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:48:07.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.415+0000 7f878effd640 10 client.?.objecter ms_handle_connect 0x7f87880039c0 2024-09-16T03:48:07.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.415+0000 7f878effd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:48:07.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.415+0000 7f878effd640 1 -- 172.21.15.42:0/1486097439 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f879001b020 con 0x7f87880039c0 2024-09-16T03:48:07.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.415+0000 7f878effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:48:07.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.415+0000 7f878effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:48:07.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.415+0000 7f878effd640 10 monclient: dump: 2024-09-16T03:48:07.427 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:48:07.427 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:48:07.427 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:07.427 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:07.427 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:48:07.428 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:48:07.428 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:48:07.428 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:48:07.428 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:48:07.428 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:48:07.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.415+0000 7f878effd640 10 monclient: _finish_auth 0 2024-09-16T03:48:07.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.415+0000 7f878effd640 10 monclient: _check_auth_tickets 2024-09-16T03:48:07.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.415+0000 7f878effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:48:07.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.415+0000 7f87a4899640 5 monclient: authenticate success, global_id 764373 2024-09-16T03:48:07.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.415+0000 7f87a4899640 10 monclient: _renew_subs 2024-09-16T03:48:07.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.415+0000 7f87a4899640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:48:07.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.415+0000 7f87a4899640 1 -- 172.21.15.42:0/1486097439 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87a01349e0 con 0x7f87880039c0 2024-09-16T03:48:07.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.415+0000 7f878effd640 1 -- 172.21.15.42:0/1486097439 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8790017660 con 0x7f87880039c0 2024-09-16T03:48:07.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.415+0000 7f878effd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:48:07.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.415+0000 7f87a4899640 10 client.764373.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:48:07.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.415+0000 7f87a4899640 10 monclient: _renew_subs 2024-09-16T03:48:07.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.415+0000 7f87a4899640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:48:07.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.415+0000 7f87a4899640 1 -- 172.21.15.42:0/1486097439 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87a0122b20 con 0x7f87880039c0 2024-09-16T03:48:07.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.415+0000 7f87a4899640 1 librados: init done 2024-09-16T03:48:07.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.415+0000 7f879cd99640 4 set_mon_vals no callback set 2024-09-16T03:48:07.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.415+0000 7f878effd640 1 -- 172.21.15.42:0/1486097439 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8790009da0 con 0x7f87880039c0 2024-09-16T03:48:07.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.415+0000 7f878effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:48:07.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.415+0000 7f878effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:48:07.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.415+0000 7f878effd640 10 monclient: dump: 2024-09-16T03:48:07.429 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:48:07.429 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:48:07.429 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:07.430 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:07.430 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:48:07.430 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:48:07.430 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:48:07.430 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:48:07.430 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:48:07.430 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:48:07.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.415+0000 7f87a4899640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:48:07.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.415+0000 7f87a4899640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:48:07.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.415+0000 7f87a4899640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:48:07.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.415+0000 7f878effd640 1 -- 172.21.15.42:0/1486097439 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f879001e020 con 0x7f87880039c0 2024-09-16T03:48:07.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.415+0000 7f87a4899640 1 -- 172.21.15.42:0/1486097439 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f87a0134780 con 0x7f87880039c0 2024-09-16T03:48:07.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.415+0000 7f878effd640 1 --2- 172.21.15.42:0/1486097439 >> v2:172.21.15.154:6812/254845342 conn(0x7f8780038020 0x7f878003a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:07.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.415+0000 7f878effd640 1 -- 172.21.15.42:0/1486097439 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(183..183 src has 1..183) ==== 6151+0+0 (secure 0 0 0) 0x7f8790049ea0 con 0x7f87880039c0 2024-09-16T03:48:07.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.415+0000 7f878effd640 10 client.764373.objecter ms_dispatch 0x7f87a0000ba0 osd_map(183..183 src has 1..183) 2024-09-16T03:48:07.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.415+0000 7f878effd640 3 client.764373.objecter handle_osd_map got epochs [183,183] > 0 2024-09-16T03:48:07.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.415+0000 7f878effd640 3 client.764373.objecter handle_osd_map decoding full epoch 183 2024-09-16T03:48:07.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.415+0000 7f878effd640 20 client.764373.objecter dump_active .. 0 homeless 2024-09-16T03:48:07.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.415+0000 7f879e59c640 1 --2- 172.21.15.42:0/1486097439 >> v2:172.21.15.154:6812/254845342 conn(0x7f8780038020 0x7f878003a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:48:07.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.415+0000 7f879e59c640 10 monclient: get_auth_request con 0x7f8780038020 auth_method 0 2024-09-16T03:48:07.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.419+0000 7f879e59c640 1 --2- 172.21.15.42:0/1486097439 >> v2:172.21.15.154:6812/254845342 conn(0x7f8780038020 0x7f878003a4e0 secure :-1 s=READY pgs=3471 cs=0 l=1 rev1=1 crypto rx=0x7f87940b0000 tx=0x7f8794010040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:48:07.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.419+0000 7f878effd640 10 client.764373.objecter ms_handle_connect 0x7f8780038020 2024-09-16T03:48:07.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.423+0000 7f878effd640 1 -- 172.21.15.42:0/1486097439 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f8790025070 con 0x7f87880039c0 2024-09-16T03:48:07.432 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.423+0000 7f878effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:48:07.432 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.423+0000 7f878effd640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:48:07.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.659+0000 7f87a4899640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:48:07.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.659+0000 7f87a4899640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:48:07.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.659+0000 7f87a4899640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:48:07.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.659+0000 7f87a4899640 1 -- 172.21.15.42:0/1486097439 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f87a006f680 con 0x7f87880039c0 2024-09-16T03:48:07.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.659+0000 7f878effd640 1 -- 172.21.15.42:0/1486097439 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1554 (secure 0 0 0) 0x7f8790010040 con 0x7f87880039c0 2024-09-16T03:48:07.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.659+0000 7f878effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:48:07.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.659+0000 7f878effd640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:48:07.663 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":58,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:48:07.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.659+0000 7f87a4899640 10 librados: watch_flush enter 2024-09-16T03:48:07.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.659+0000 7f87a4899640 10 librados: watch_flush exit 2024-09-16T03:48:07.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.659+0000 7f87a4899640 20 client.764373.objecter shutdown clearing up homeless session... 2024-09-16T03:48:07.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.659+0000 7f87a4899640 10 client.764373.objecter successfully canceled tick 2024-09-16T03:48:07.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.659+0000 7f87a4899640 1 -- 172.21.15.42:0/1486097439 >> v2:172.21.15.154:6812/254845342 conn(0x7f8780038020 msgr2=0x7f878003a4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:48:07.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.659+0000 7f87a4899640 1 --2- 172.21.15.42:0/1486097439 >> v2:172.21.15.154:6812/254845342 conn(0x7f8780038020 0x7f878003a4e0 secure :-1 s=READY pgs=3471 cs=0 l=1 rev1=1 crypto rx=0x7f87940b0000 tx=0x7f8794010040 comp rx=0 tx=0).stop 2024-09-16T03:48:07.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.659+0000 7f87a4899640 10 monclient: shutdown 2024-09-16T03:48:07.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.659+0000 7f87a4899640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:48:07.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.659+0000 7f87a4899640 1 -- 172.21.15.42:0/1486097439 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f87880039c0 msgr2=0x7f87a01ceab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:48:07.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.659+0000 7f87a4899640 1 --2- 172.21.15.42:0/1486097439 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f87880039c0 0x7f87a01ceab0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f8790003f00 tx=0x7f8790011600 comp rx=0 tx=0).stop 2024-09-16T03:48:07.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.659+0000 7f87a4899640 1 -- 172.21.15.42:0/1486097439 shutdown_connections 2024-09-16T03:48:07.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.659+0000 7f87a4899640 1 --2- 172.21.15.42:0/1486097439 >> v2:172.21.15.154:6812/254845342 conn(0x7f8780038020 0x7f878003a4e0 unknown :-1 s=CLOSED pgs=3471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:07.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.659+0000 7f87a4899640 1 --2- 172.21.15.42:0/1486097439 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8788004da0 0x7f87a01cf2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:07.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.659+0000 7f87a4899640 1 --2- 172.21.15.42:0/1486097439 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f87880039c0 0x7f87a01ceab0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:07.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.659+0000 7f87a4899640 1 --2- 172.21.15.42:0/1486097439 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f87800036c0 0x7f87a01ce570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:07.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.659+0000 7f87a4899640 1 -- 172.21.15.42:0/1486097439 >> 172.21.15.42:0/1486097439 conn(0x7f87a00a9fb0 msgr2=0x7f87a0131790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:48:07.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.659+0000 7f87a4899640 1 -- 172.21.15.42:0/1486097439 shutdown_connections 2024-09-16T03:48:07.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.659+0000 7f87a4899640 1 -- 172.21.15.42:0/1486097439 wait complete. 2024-09-16T03:48:07.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.659+0000 7f87a4899640 1 librados: shutdown 2024-09-16T03:48:07.682 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T03:48:07.682 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-16T03:48:07.682 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T03:48:07.801 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.799+0000 7fed87ef5640 10 monclient: get_monmap_and_config 2024-09-16T03:48:07.801 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.799+0000 7fed87ef5640 10 monclient: build_initial_monmap 2024-09-16T03:48:07.801 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.799+0000 7fed87ef5640 10 monclient: monmap: 2024-09-16T03:48:07.801 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:48:07.801 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:48:07.801 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:48:07.800415+0000 2024-09-16T03:48:07.802 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:48:07.800415+0000 2024-09-16T03:48:07.802 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:48:07.802 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:48:07.802 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:48:07.802 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:48:07.802 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:48:07.802 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:48:07.802 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:48:07.802 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:48:07.808 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.799+0000 7fed87ef5640 1 Processor -- start 2024-09-16T03:48:07.808 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.799+0000 7fed87ef5640 1 -- start start 2024-09-16T03:48:07.808 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.799+0000 7fed87ef5640 10 monclient: init 2024-09-16T03:48:07.808 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.799+0000 7fed87ef5640 10 monclient: _reopen_session rank -1 2024-09-16T03:48:07.808 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.799+0000 7fed87ef5640 10 monclient: _add_conns ranks=[2,0,4,1,3] 2024-09-16T03:48:07.808 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.799+0000 7fed87ef5640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fed800a1330 0x7fed8009f180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:07.808 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.799+0000 7fed87ef5640 10 monclient(hunting): picked mon.noname-b con 0x7fed800a1330 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:48:07.808 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.799+0000 7fed87ef5640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fed800a1710 addr v1:172.21.15.42:6789/0 2024-09-16T03:48:07.808 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.799+0000 7fed87ef5640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fed801393a0 0x7fed8013b790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:07.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.799+0000 7fed87ef5640 10 monclient(hunting): picked mon.noname-c con 0x7fed801393a0 addr v2:172.21.15.154:3300/0 2024-09-16T03:48:07.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.799+0000 7fed87ef5640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fed8009e4c0 con 0x7fed800a1710 2024-09-16T03:48:07.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.799+0000 7fed87ef5640 10 monclient(hunting): start opening mon connection 2024-09-16T03:48:07.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.799+0000 7fed87ef5640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fed8009edf0 con 0x7fed801393a0 2024-09-16T03:48:07.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.799+0000 7fed87ef5640 10 monclient(hunting): start opening mon connection 2024-09-16T03:48:07.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.799+0000 7fed87ef5640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fed8009cd60 con 0x7fed800a1330 2024-09-16T03:48:07.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.799+0000 7fed87ef5640 10 monclient(hunting): _renew_subs 2024-09-16T03:48:07.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.799+0000 7fed87ef5640 10 monclient(hunting): authenticate will time out at 1334.769796s 2024-09-16T03:48:07.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.799+0000 7fed85c91640 1 -- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fed800a1330 msgr2=0x7fed8009f180 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T03:48:07.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.799+0000 7fed85c91640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fed800a1330 0x7fed8009f180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:48:07.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.799+0000 7fed85490640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fed800a1710 0x7fed800a1b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:34946/0 (socket says 172.21.15.42:34946) 2024-09-16T03:48:07.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.799+0000 7fed85490640 1 -- 172.21.15.42:0/442948849 learned_addr learned my addr 172.21.15.42:0/442948849 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:48:07.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.799+0000 7fed84c8f640 1 -- 172.21.15.42:0/442948849 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1977831274 0 0) 0x7fed8009e4c0 con 0x7fed800a1710 2024-09-16T03:48:07.810 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.799+0000 7fed84c8f640 10 monclient(hunting): _negotiate 2024-09-16T03:48:07.810 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.799+0000 7fed84c8f640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:48:07.810 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.799+0000 7fed84c8f640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:48:07.810 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.799+0000 7fed86492640 1 --2- 172.21.15.42:0/442948849 >> v2:172.21.15.154:3300/0 conn(0x7fed801393a0 0x7fed8013b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:48:07.810 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.799+0000 7fed84c8f640 10 monclient(hunting): my global_id is 764385 2024-09-16T03:48:07.810 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.799+0000 7fed84c8f640 1 -- 172.21.15.42:0/442948849 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fed6c003530 con 0x7fed800a1710 2024-09-16T03:48:07.810 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.799+0000 7fed86492640 10 monclient(hunting): get_auth_request con 0x7fed801393a0 auth_method 0 2024-09-16T03:48:07.810 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.799+0000 7fed86492640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:48:07.810 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.799+0000 7fed86492640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:48:07.810 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.799+0000 7fed86492640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:48:07.810 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.799+0000 7fed86492640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:48:07.810 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.799+0000 7fed86492640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:48:07.810 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.799+0000 7fed86492640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:48:07.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.799+0000 7fed84c8f640 1 -- 172.21.15.42:0/442948849 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 594377132 0 0) 0x7fed6c003530 con 0x7fed800a1710 2024-09-16T03:48:07.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.799+0000 7fed84c8f640 1 -- 172.21.15.42:0/442948849 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fed800a1330 msgr2=0x7fed8009f180 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:48:07.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.799+0000 7fed84c8f640 1 --2- 172.21.15.42:0/442948849 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fed800a1330 0x7fed8009f180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:07.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.799+0000 7fed84c8f640 1 -- 172.21.15.42:0/442948849 >> v2:172.21.15.154:3300/0 conn(0x7fed801393a0 msgr2=0x7fed8013b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:48:07.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.799+0000 7fed84c8f640 1 --2- 172.21.15.42:0/442948849 >> v2:172.21.15.154:3300/0 conn(0x7fed801393a0 0x7fed8013b790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:07.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.799+0000 7fed84c8f640 10 monclient: _finish_hunting 0 2024-09-16T03:48:07.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.799+0000 7fed84c8f640 1 monclient: found mon.noname-a-legacy 2024-09-16T03:48:07.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.799+0000 7fed84c8f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:48:07.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.799+0000 7fed84c8f640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.42:6789/0 2024-09-16T03:48:07.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.799+0000 7fed84c8f640 1 -- 172.21.15.42:0/442948849 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed8009f6c0 con 0x7fed800a1710 2024-09-16T03:48:07.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.799+0000 7fed84c8f640 10 monclient: _finish_auth 0 2024-09-16T03:48:07.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.799+0000 7fed84c8f640 10 monclient: _check_auth_tickets 2024-09-16T03:48:07.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.799+0000 7fed84c8f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:48:07.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.799+0000 7fed84c8f640 1 -- 172.21.15.42:0/442948849 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2177413734 0 0) 0x7fed70001680 con 0x7fed800a1710 2024-09-16T03:48:07.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.799+0000 7fed87ef5640 5 monclient: authenticate success, global_id 764385 2024-09-16T03:48:07.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.799+0000 7fed87ef5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:48:07.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.799+0000 7fed84c8f640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:48:07.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.799+0000 7fed84c8f640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T03:48:07.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.799+0000 7fed84c8f640 10 monclient: dump: 2024-09-16T03:48:07.812 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:48:07.812 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:48:07.812 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:07.812 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:07.812 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:48:07.812 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:48:07.812 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:48:07.812 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:48:07.813 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:48:07.813 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:48:07.813 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.799+0000 7fed84c8f640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] but i'm connected to v1:172.21.15.42:6789/0, reconnecting 2024-09-16T03:48:07.813 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.799+0000 7fed84c8f640 10 monclient: _reopen_session rank -1 2024-09-16T03:48:07.813 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.799+0000 7fed84c8f640 1 -- 172.21.15.42:0/442948849 >> v1:172.21.15.42:6789/0 conn(0x7fed800a1710 legacy=0x7fed800a1b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:48:07.813 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.799+0000 7fed84c8f640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:48:07.813 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.799+0000 7fed84c8f640 1 --2- 172.21.15.42:0/442948849 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fed800a1330 0x7fed6c0039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:07.813 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.799+0000 7fed84c8f640 10 monclient(hunting): picked mon.c con 0x7fed800a1330 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:48:07.813 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.799+0000 7fed84c8f640 1 --2- 172.21.15.42:0/442948849 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fed6c003f60 0x7fed6c004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:07.813 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.799+0000 7fed84c8f640 10 monclient(hunting): picked mon.b con 0x7fed6c003f60 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:48:07.813 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.799+0000 7fed84c8f640 1 --2- 172.21.15.42:0/442948849 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fed6c004d50 0x7fed6c0051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:07.813 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.799+0000 7fed84c8f640 10 monclient(hunting): picked mon.a con 0x7fed6c004d50 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:48:07.813 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.799+0000 7fed84c8f640 10 monclient(hunting): start opening mon connection 2024-09-16T03:48:07.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.799+0000 7fed84c8f640 1 -- 172.21.15.42:0/442948849 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fed8009edf0 con 0x7fed6c004d50 2024-09-16T03:48:07.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.799+0000 7fed84c8f640 10 monclient(hunting): start opening mon connection 2024-09-16T03:48:07.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.799+0000 7fed84c8f640 1 -- 172.21.15.42:0/442948849 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fed8009cd60 con 0x7fed6c003f60 2024-09-16T03:48:07.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.799+0000 7fed84c8f640 10 monclient(hunting): start opening mon connection 2024-09-16T03:48:07.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.799+0000 7fed84c8f640 1 -- 172.21.15.42:0/442948849 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fed6c00bc40 con 0x7fed800a1330 2024-09-16T03:48:07.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.799+0000 7fed84c8f640 10 monclient(hunting): _renew_subs 2024-09-16T03:48:07.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.799+0000 7fed84c8f640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:48:07.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.799+0000 7fed85c91640 1 -- 172.21.15.42:0/442948849 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fed800a1330 msgr2=0x7fed6c0039e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T03:48:07.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.799+0000 7fed87ef5640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-16T03:48:07.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.799+0000 7fed85c91640 1 --2- 172.21.15.42:0/442948849 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fed800a1330 0x7fed6c0039e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:48:07.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.799+0000 7fed86492640 1 --2- 172.21.15.42:0/442948849 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fed6c004d50 0x7fed6c0051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:48:07.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.799+0000 7fed86492640 10 monclient(hunting): get_auth_request con 0x7fed6c004d50 auth_method 0 2024-09-16T03:48:07.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.799+0000 7fed86492640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:48:07.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.799+0000 7fed86492640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:48:07.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.799+0000 7fed86492640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:48:07.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.799+0000 7fed86492640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:48:07.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.799+0000 7fed86492640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:48:07.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.799+0000 7fed86492640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T03:48:07.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.799+0000 7fed85490640 1 --2- 172.21.15.42:0/442948849 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fed6c003f60 0x7fed6c004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:48:07.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.799+0000 7fed85490640 10 monclient(hunting): get_auth_request con 0x7fed6c003f60 auth_method 0 2024-09-16T03:48:07.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.799+0000 7fed85490640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:48:07.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.799+0000 7fed85490640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:48:07.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.799+0000 7fed85490640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:48:07.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.799+0000 7fed86492640 10 monclient(hunting): handle_auth_done global_id 764385 payload 293 2024-09-16T03:48:07.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.799+0000 7fed86492640 1 -- 172.21.15.42:0/442948849 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fed800a1330 msgr2=0x7fed6c0039e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:48:07.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.799+0000 7fed86492640 1 --2- 172.21.15.42:0/442948849 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fed800a1330 0x7fed6c0039e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:07.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.799+0000 7fed86492640 1 -- 172.21.15.42:0/442948849 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fed6c003f60 msgr2=0x7fed6c004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:48:07.816 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.799+0000 7fed86492640 1 --2- 172.21.15.42:0/442948849 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fed6c003f60 0x7fed6c004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:07.816 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.799+0000 7fed86492640 10 monclient: _finish_hunting 0 2024-09-16T03:48:07.816 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.799+0000 7fed86492640 1 monclient: found mon.a 2024-09-16T03:48:07.816 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.799+0000 7fed86492640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:48:07.816 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.799+0000 7fed86492640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:48:07.816 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.799+0000 7fed86492640 1 -- 172.21.15.42:0/442948849 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed6c003530 con 0x7fed6c004d50 2024-09-16T03:48:07.816 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.799+0000 7fed86492640 10 monclient: _finish_auth 0 2024-09-16T03:48:07.816 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.799+0000 7fed86492640 10 monclient: _check_auth_tickets 2024-09-16T03:48:07.816 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.799+0000 7fed86492640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:48:07.816 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.803+0000 7fed86492640 1 --2- 172.21.15.42:0/442948849 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fed6c004d50 0x7fed6c0051d0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fed8009cba0 tx=0x7fed7801cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e072a98d8dec23c4 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:48:07.816 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.803+0000 7fed84c8f640 1 -- 172.21.15.42:0/442948849 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed780b2070 con 0x7fed6c004d50 2024-09-16T03:48:07.816 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.803+0000 7fed84c8f640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:48:07.816 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.803+0000 7fed84c8f640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:48:07.817 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.803+0000 7fed84c8f640 10 monclient: dump: 2024-09-16T03:48:07.817 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:48:07.817 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:48:07.817 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:07.817 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:07.817 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:48:07.817 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:48:07.817 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:48:07.817 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:48:07.817 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:48:07.817 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:48:07.817 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.803+0000 7fed87ef5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:48:07.817 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.803+0000 7fed84c8f640 1 -- 172.21.15.42:0/442948849 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fed780a5b40 con 0x7fed6c004d50 2024-09-16T03:48:07.817 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.803+0000 7fed84c8f640 10 monclient: handle_config config(0 keys) 2024-09-16T03:48:07.818 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.803+0000 7fed87ef5640 10 monclient: get_monmap_and_config success 2024-09-16T03:48:07.818 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.803+0000 7fed87ef5640 4 set_mon_vals no callback set 2024-09-16T03:48:07.818 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.803+0000 7fed84c8f640 1 -- 172.21.15.42:0/442948849 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed78009c10 con 0x7fed6c004d50 2024-09-16T03:48:07.818 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.803+0000 7fed84c8f640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:48:07.818 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.803+0000 7fed84c8f640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:48:07.818 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.803+0000 7fed84c8f640 10 monclient: dump: 2024-09-16T03:48:07.818 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:48:07.818 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:48:07.818 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:07.818 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:07.818 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:48:07.818 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:48:07.818 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:48:07.818 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:48:07.819 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:48:07.819 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:48:07.819 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.803+0000 7fed87ef5640 10 monclient: shutdown 2024-09-16T03:48:07.819 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.803+0000 7fed87ef5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:48:07.819 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.803+0000 7fed87ef5640 1 -- 172.21.15.42:0/442948849 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fed6c004d50 msgr2=0x7fed6c0051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:48:07.819 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.803+0000 7fed87ef5640 1 --2- 172.21.15.42:0/442948849 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fed6c004d50 0x7fed6c0051d0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fed8009cba0 tx=0x7fed7801cc40 comp rx=0 tx=0).stop 2024-09-16T03:48:07.819 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.803+0000 7fed86492640 1 -- 172.21.15.42:0/442948849 reap_dead start 2024-09-16T03:48:07.819 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.803+0000 7fed87ef5640 1 -- 172.21.15.42:0/442948849 shutdown_connections 2024-09-16T03:48:07.819 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.803+0000 7fed87ef5640 1 -- 172.21.15.42:0/442948849 >> 172.21.15.42:0/442948849 conn(0x7fed800a2420 msgr2=0x7fed800a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:48:07.819 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.803+0000 7fed87ef5640 1 -- 172.21.15.42:0/442948849 shutdown_connections 2024-09-16T03:48:07.819 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.803+0000 7fed87ef5640 1 -- 172.21.15.42:0/442948849 wait complete. 2024-09-16T03:48:07.819 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.803+0000 7fed87ef5640 10 monclient: build_initial_monmap 2024-09-16T03:48:07.819 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.803+0000 7fed87ef5640 10 monclient: monmap: 2024-09-16T03:48:07.819 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:48:07.819 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:48:07.820 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:48:07.820 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:48:07.820 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:48:07.820 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:48:07.820 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:48:07.820 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:48:07.820 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:48:07.820 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:48:07.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.803+0000 7fed87ef5640 1 librados: starting msgr at 2024-09-16T03:48:07.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.803+0000 7fed87ef5640 1 librados: starting objecter 2024-09-16T03:48:07.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.803+0000 7fed87ef5640 1 Processor -- start 2024-09-16T03:48:07.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.803+0000 7fed87ef5640 1 -- start start 2024-09-16T03:48:07.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.803+0000 7fed87ef5640 1 librados: setting wanted keys 2024-09-16T03:48:07.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.803+0000 7fed87ef5640 1 librados: calling monclient init 2024-09-16T03:48:07.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.803+0000 7fed87ef5640 10 monclient: init 2024-09-16T03:48:07.826 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.803+0000 7fed87ef5640 10 monclient: _reopen_session rank -1 2024-09-16T03:48:07.826 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.803+0000 7fed87ef5640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:48:07.826 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.803+0000 7fed87ef5640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fed6c004d50 0x7fed801cefb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:07.826 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.803+0000 7fed87ef5640 10 monclient(hunting): picked mon.noname-c con 0x7fed6c004d50 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:48:07.826 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.803+0000 7fed87ef5640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fed80132840 0x7fed801cba90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:07.826 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.803+0000 7fed87ef5640 10 monclient(hunting): picked mon.noname-b con 0x7fed80132840 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:48:07.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.803+0000 7fed87ef5640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fed80132ca0 0x7fed801cbfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:07.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.803+0000 7fed87ef5640 10 monclient(hunting): picked mon.noname-a con 0x7fed80132ca0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:48:07.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.803+0000 7fed87ef5640 10 monclient(hunting): start opening mon connection 2024-09-16T03:48:07.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.803+0000 7fed87ef5640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fed8013c870 con 0x7fed80132ca0 2024-09-16T03:48:07.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.803+0000 7fed87ef5640 10 monclient(hunting): start opening mon connection 2024-09-16T03:48:07.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.803+0000 7fed87ef5640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fed8013c6f0 con 0x7fed80132840 2024-09-16T03:48:07.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.803+0000 7fed87ef5640 10 monclient(hunting): start opening mon connection 2024-09-16T03:48:07.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.803+0000 7fed85c91640 1 -- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fed6c004d50 msgr2=0x7fed801cefb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T03:48:07.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.803+0000 7fed87ef5640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fed8013c9f0 con 0x7fed6c004d50 2024-09-16T03:48:07.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.803+0000 7fed87ef5640 10 monclient(hunting): _renew_subs 2024-09-16T03:48:07.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.803+0000 7fed87ef5640 10 monclient(hunting): authenticate will time out at 1334.775695s 2024-09-16T03:48:07.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.803+0000 7fed85c91640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fed6c004d50 0x7fed801cefb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:48:07.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.803+0000 7fed86492640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fed80132ca0 0x7fed801cbfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:48:07.828 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.803+0000 7fed86492640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fed80132ca0 0x7fed801cbfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:60788/0 (socket says 172.21.15.42:60788) 2024-09-16T03:48:07.828 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.803+0000 7fed86492640 1 -- 172.21.15.42:0/544072451 learned_addr learned my addr 172.21.15.42:0/544072451 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:48:07.828 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.803+0000 7fed86492640 10 monclient(hunting): get_auth_request con 0x7fed80132ca0 auth_method 0 2024-09-16T03:48:07.828 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.803+0000 7fed86492640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:48:07.828 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.803+0000 7fed86492640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:48:07.828 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.803+0000 7fed86492640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:48:07.828 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.803+0000 7fed86492640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:48:07.828 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.803+0000 7fed86492640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:48:07.828 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.803+0000 7fed86492640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:48:07.828 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.807+0000 7fed86492640 10 monclient(hunting): handle_auth_done global_id 764388 payload 824 2024-09-16T03:48:07.828 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.807+0000 7fed86492640 1 -- 172.21.15.42:0/544072451 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fed6c004d50 msgr2=0x7fed801cefb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:48:07.828 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.807+0000 7fed86492640 1 --2- 172.21.15.42:0/544072451 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fed6c004d50 0x7fed801cefb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:07.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.807+0000 7fed86492640 1 -- 172.21.15.42:0/544072451 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fed80132840 msgr2=0x7fed801cba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:48:07.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.807+0000 7fed86492640 1 --2- 172.21.15.42:0/544072451 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fed80132840 0x7fed801cba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:07.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.807+0000 7fed86492640 10 monclient: _finish_hunting 0 2024-09-16T03:48:07.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.807+0000 7fed86492640 1 monclient: found mon.noname-a 2024-09-16T03:48:07.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.807+0000 7fed86492640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:48:07.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.807+0000 7fed86492640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:48:07.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.807+0000 7fed86492640 1 -- 172.21.15.42:0/544072451 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed80121f60 con 0x7fed80132ca0 2024-09-16T03:48:07.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.807+0000 7fed86492640 1 --2- 172.21.15.42:0/544072451 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fed80132ca0 0x7fed801cbfd0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fed78002240 tx=0x7fed78009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:48:07.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.807+0000 7fed767fc640 10 client.?.objecter ms_handle_connect 0x7fed80132ca0 2024-09-16T03:48:07.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.807+0000 7fed767fc640 10 client.?.objecter resend_mon_ops 2024-09-16T03:48:07.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.807+0000 7fed767fc640 1 -- 172.21.15.42:0/544072451 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed780b2040 con 0x7fed80132ca0 2024-09-16T03:48:07.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.807+0000 7fed767fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:48:07.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.807+0000 7fed767fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:48:07.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.807+0000 7fed767fc640 10 monclient: dump: 2024-09-16T03:48:07.830 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:48:07.830 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:48:07.830 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:07.830 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:07.830 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:48:07.830 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:48:07.830 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:48:07.830 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:48:07.830 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:48:07.830 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:48:07.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.807+0000 7fed767fc640 10 monclient: _finish_auth 0 2024-09-16T03:48:07.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.807+0000 7fed767fc640 10 monclient: _check_auth_tickets 2024-09-16T03:48:07.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.807+0000 7fed767fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:48:07.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.807+0000 7fed87ef5640 5 monclient: authenticate success, global_id 764388 2024-09-16T03:48:07.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.807+0000 7fed87ef5640 10 monclient: _renew_subs 2024-09-16T03:48:07.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.807+0000 7fed767fc640 1 -- 172.21.15.42:0/544072451 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fed780bc070 con 0x7fed80132ca0 2024-09-16T03:48:07.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.807+0000 7fed87ef5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:48:07.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.807+0000 7fed87ef5640 1 -- 172.21.15.42:0/544072451 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed80122240 con 0x7fed80132ca0 2024-09-16T03:48:07.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.807+0000 7fed767fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:48:07.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.807+0000 7fed767fc640 1 -- 172.21.15.42:0/544072451 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed780ae930 con 0x7fed80132ca0 2024-09-16T03:48:07.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.807+0000 7fed767fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:48:07.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.807+0000 7fed767fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:48:07.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.807+0000 7fed767fc640 10 monclient: dump: 2024-09-16T03:48:07.831 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:48:07.831 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:48:07.831 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:07.832 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:07.832 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:48:07.832 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:48:07.832 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:48:07.832 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:48:07.832 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:48:07.832 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:48:07.832 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.807+0000 7fed77fff640 4 set_mon_vals no callback set 2024-09-16T03:48:07.832 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.807+0000 7fed87ef5640 10 client.764388.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:48:07.832 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.807+0000 7fed87ef5640 10 monclient: _renew_subs 2024-09-16T03:48:07.832 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.807+0000 7fed87ef5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:48:07.832 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.807+0000 7fed87ef5640 1 -- 172.21.15.42:0/544072451 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed80073a30 con 0x7fed80132ca0 2024-09-16T03:48:07.832 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.807+0000 7fed87ef5640 1 librados: init done 2024-09-16T03:48:07.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.807+0000 7fed87ef5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:48:07.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.807+0000 7fed87ef5640 1 --2- 172.21.15.42:0/544072451 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fed801cc510 0x7fed80133150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:07.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.807+0000 7fed87ef5640 10 monclient(hunting): start opening mon connection 2024-09-16T03:48:07.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.807+0000 7fed87ef5640 1 -- 172.21.15.42:0/544072451 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fed801cc8f0 con 0x7fed801cc510 2024-09-16T03:48:07.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.807+0000 7fed85c91640 1 --2- 172.21.15.42:0/544072451 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fed801cc510 0x7fed80133150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:48:07.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.807+0000 7fed85c91640 10 monclient: get_auth_request con 0x7fed801cc510 auth_method 0 2024-09-16T03:48:07.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.807+0000 7fed85c91640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:48:07.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.807+0000 7fed85c91640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:48:07.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.807+0000 7fed85c91640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:48:07.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.807+0000 7fed85c91640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:48:07.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.807+0000 7fed85c91640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:48:07.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.807+0000 7fed85c91640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:48:07.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.807+0000 7fed767fc640 1 -- 172.21.15.42:0/544072451 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fed780aead0 con 0x7fed80132ca0 2024-09-16T03:48:07.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.807+0000 7fed85c91640 10 monclient(hunting): handle_auth_done global_id 764391 payload 824 2024-09-16T03:48:07.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.807+0000 7fed85c91640 1 -- 172.21.15.42:0/544072451 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fed800a1b10 con 0x7fed801cc510 2024-09-16T03:48:07.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.807+0000 7fed767fc640 1 --2- 172.21.15.42:0/544072451 >> v2:172.21.15.154:6812/254845342 conn(0x7fed640377c0 0x7fed64039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:07.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.807+0000 7fed767fc640 1 -- 172.21.15.42:0/544072451 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(183..183 src has 1..183) ==== 6151+0+0 (secure 0 0 0) 0x7fed780e3820 con 0x7fed80132ca0 2024-09-16T03:48:07.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.807+0000 7fed767fc640 10 client.764388.objecter ms_dispatch 0x7fed80000ba0 osd_map(183..183 src has 1..183) 2024-09-16T03:48:07.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.807+0000 7fed767fc640 3 client.764388.objecter handle_osd_map got epochs [183,183] > 0 2024-09-16T03:48:07.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.807+0000 7fed767fc640 3 client.764388.objecter handle_osd_map decoding full epoch 183 2024-09-16T03:48:07.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.807+0000 7fed85c91640 1 --2- 172.21.15.42:0/544072451 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fed801cc510 0x7fed80133150 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fed7c00b430 tx=0x7fed7c00b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:48:07.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.807+0000 7fed767fc640 20 client.764388.objecter dump_active .. 0 homeless 2024-09-16T03:48:07.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.807+0000 7fed767fc640 10 client.764388.objecter ms_handle_connect 0x7fed801cc510 2024-09-16T03:48:07.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.807+0000 7fed767fc640 10 client.764388.objecter resend_mon_ops 2024-09-16T03:48:07.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.807+0000 7fed767fc640 1 -- 172.21.15.42:0/544072451 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed7c003e60 con 0x7fed801cc510 2024-09-16T03:48:07.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.807+0000 7fed767fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:48:07.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.807+0000 7fed767fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:48:07.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.807+0000 7fed767fc640 10 monclient: dump: 2024-09-16T03:48:07.835 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:48:07.835 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:48:07.835 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:07.835 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:07.835 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:48:07.835 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:48:07.835 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:48:07.835 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:48:07.835 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:48:07.835 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:48:07.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.807+0000 7fed85490640 1 --2- 172.21.15.42:0/544072451 >> v2:172.21.15.154:6812/254845342 conn(0x7fed640377c0 0x7fed64039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:48:07.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.807+0000 7fed85490640 10 monclient: get_auth_request con 0x7fed640377c0 auth_method 0 2024-09-16T03:48:07.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.807+0000 7fed767fc640 1 -- 172.21.15.42:0/544072451 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fed7c015070 con 0x7fed801cc510 2024-09-16T03:48:07.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.807+0000 7fed767fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:48:07.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.807+0000 7fed767fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:48:07.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.807+0000 7fed767fc640 1 -- 172.21.15.42:0/544072451 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fed801cc510 msgr2=0x7fed80133150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:48:07.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.807+0000 7fed767fc640 1 --2- 172.21.15.42:0/544072451 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fed801cc510 0x7fed80133150 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fed7c00b430 tx=0x7fed7c00b8f0 comp rx=0 tx=0).stop 2024-09-16T03:48:07.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.807+0000 7fed767fc640 1 -- 172.21.15.42:0/544072451 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fed801cc510 msgr2=0x7fed80133150 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:48:07.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.807+0000 7fed767fc640 1 --2- 172.21.15.42:0/544072451 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fed801cc510 0x7fed80133150 secure :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fed7c00b430 tx=0x7fed7c00b8f0 comp rx=0 tx=0).stop 2024-09-16T03:48:07.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.807+0000 7fed85490640 1 --2- 172.21.15.42:0/544072451 >> v2:172.21.15.154:6812/254845342 conn(0x7fed640377c0 0x7fed64039c80 secure :-1 s=READY pgs=3473 cs=0 l=1 rev1=1 crypto rx=0x7fed70001ec0 tx=0x7fed70001db0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:48:07.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.807+0000 7fed767fc640 10 client.764388.objecter ms_handle_connect 0x7fed640377c0 2024-09-16T03:48:07.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.815+0000 7fed87ef5640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:48:07.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.815+0000 7fed87ef5640 1 --2- 172.21.15.42:0/544072451 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fed80133690 0x7fed80133a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:07.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.815+0000 7fed87ef5640 10 monclient(hunting): start opening mon connection 2024-09-16T03:48:07.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.815+0000 7fed87ef5640 1 -- 172.21.15.42:0/544072451 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fed8013b2d0 con 0x7fed80133690 2024-09-16T03:48:07.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.815+0000 7fed85c91640 1 --2- 172.21.15.42:0/544072451 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fed80133690 0x7fed80133a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:48:07.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.815+0000 7fed85c91640 10 monclient: get_auth_request con 0x7fed80133690 auth_method 0 2024-09-16T03:48:07.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.815+0000 7fed85c91640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:48:07.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.815+0000 7fed85c91640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:48:07.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.815+0000 7fed85c91640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:48:07.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.815+0000 7fed85c91640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:48:07.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.815+0000 7fed85c91640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:48:07.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.815+0000 7fed85c91640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:48:07.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.815+0000 7fed85c91640 10 monclient(hunting): handle_auth_done global_id 764394 payload 824 2024-09-16T03:48:07.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.815+0000 7fed85c91640 1 -- 172.21.15.42:0/544072451 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fed8013b770 con 0x7fed80133690 2024-09-16T03:48:07.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.815+0000 7fed85c91640 1 --2- 172.21.15.42:0/544072451 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fed80133690 0x7fed80133a70 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fed7c00b430 tx=0x7fed7c005a70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:48:07.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.815+0000 7fed767fc640 10 client.764388.objecter ms_handle_connect 0x7fed80133690 2024-09-16T03:48:07.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.815+0000 7fed767fc640 10 client.764388.objecter resend_mon_ops 2024-09-16T03:48:07.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.815+0000 7fed767fc640 1 -- 172.21.15.42:0/544072451 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed7c0055d0 con 0x7fed80133690 2024-09-16T03:48:07.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.815+0000 7fed767fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:48:07.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.815+0000 7fed767fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:48:07.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.815+0000 7fed767fc640 10 monclient: dump: 2024-09-16T03:48:07.838 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:48:07.838 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:48:07.838 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:07.838 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:07.838 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:48:07.838 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:48:07.838 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:48:07.838 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:48:07.838 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:48:07.838 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:48:07.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.815+0000 7fed767fc640 1 -- 172.21.15.42:0/544072451 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4869 (secure 0 0 0) 0x7fed7c0055d0 con 0x7fed80133690 2024-09-16T03:48:07.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.815+0000 7fed767fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:48:07.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.815+0000 7fed767fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:48:07.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.815+0000 7fed767fc640 1 -- 172.21.15.42:0/544072451 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fed80133690 msgr2=0x7fed80133a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:48:07.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.815+0000 7fed767fc640 1 --2- 172.21.15.42:0/544072451 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fed80133690 0x7fed80133a70 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fed7c00b430 tx=0x7fed7c005a70 comp rx=0 tx=0).stop 2024-09-16T03:48:07.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.815+0000 7fed767fc640 1 -- 172.21.15.42:0/544072451 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fed80133690 msgr2=0x7fed80133a70 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:48:07.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.815+0000 7fed767fc640 1 --2- 172.21.15.42:0/544072451 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fed80133690 0x7fed80133a70 secure :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fed7c00b430 tx=0x7fed7c005a70 comp rx=0 tx=0).stop 2024-09-16T03:48:07.839 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:48:07.839 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:48:07.839 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:48:07.840 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "leader", 2024-09-16T03:48:07.840 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 58, 2024-09-16T03:48:07.840 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 78607, 2024-09-16T03:48:07.840 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:48:07.840 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:48:07.840 INFO:teuthology.orchestra.run.smithi042.stdout: 1 2024-09-16T03:48:07.840 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:48:07.840 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 0, 2024-09-16T03:48:07.840 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:48:07.840 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:48:07.840 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:48:07.840 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:48:07.841 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:48:07.841 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:48:07.841 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:48:07.841 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:48:07.841 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:48:07.841 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:48:07.841 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:48:07.841 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:48:07.841 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:48:07.841 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:48:07.841 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:48:07.841 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:48:07.841 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:48:07.842 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:48:07.842 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:48:07.842 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:48:07.842 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:48:07.842 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:48:07.842 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:48:07.842 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:48:07.842 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:48:07.842 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:48:07.842 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:48:07.842 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:48:07.842 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:48:07.842 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:48:07.843 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:48:07.843 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:48:07.843 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:48:07.843 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:48:07.843 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:48:07.843 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:48:07.843 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:48:07.843 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:48:07.843 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:48:07.843 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:48:07.843 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:48:07.843 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:48:07.843 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:48:07.844 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:48:07.844 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:48:07.844 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:48:07.844 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:48:07.844 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:48:07.844 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:48:07.844 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:48:07.844 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:48:07.844 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:48:07.844 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:48:07.844 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:48:07.844 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:48:07.844 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:48:07.844 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:48:07.845 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:48:07.845 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:48:07.845 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:48:07.845 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:48:07.845 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:48:07.845 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:48:07.845 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:48:07.845 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:48:07.845 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:48:07.845 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:48:07.845 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:48:07.845 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:48:07.845 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:48:07.846 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:48:07.846 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:48:07.846 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:48:07.846 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:48:07.846 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:48:07.846 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:48:07.846 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:48:07.846 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:48:07.846 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:48:07.846 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:48:07.846 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:48:07.846 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:48:07.846 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:48:07.846 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:48:07.847 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:48:07.847 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:48:07.847 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:48:07.847 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:48:07.847 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:48:07.847 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:48:07.847 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:48:07.847 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:48:07.847 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:48:07.847 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:48:07.847 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:48:07.847 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:48:07.847 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:48:07.847 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:48:07.848 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:48:07.848 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:48:07.848 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:48:07.848 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:48:07.848 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:48:07.848 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:48:07.848 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:48:07.848 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:48:07.848 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:48:07.848 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:48:07.848 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:48:07.848 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:48:07.848 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:48:07.849 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:48:07.849 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:48:07.849 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:48:07.849 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:48:07.849 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:48:07.849 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:48:07.849 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:48:07.849 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:48:07.849 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:48:07.849 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:48:07.849 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:48:07.849 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:48:07.849 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:48:07.849 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:48:07.850 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:48:07.850 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:48:07.850 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:48:07.850 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:48:07.850 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:48:07.850 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:48:07.850 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:48:07.850 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:48:07.850 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:48:07.850 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:48:07.850 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:48:07.850 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:48:07.850 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:48:07.850 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:48:07.851 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:48:07.851 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:48:07.851 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T03:48:07.851 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:48:07.851 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:48:07.851 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:48:07.851 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 5 2024-09-16T03:48:07.851 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:48:07.851 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:48:07.851 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:48:07.851 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:48:07.851 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:48:07.851 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:48:07.851 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 8 2024-09-16T03:48:07.852 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:48:07.852 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:48:07.852 INFO:teuthology.orchestra.run.smithi042.stdout: "mgr": [ 2024-09-16T03:48:07.852 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:48:07.852 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:48:07.852 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:48:07.852 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:48:07.852 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:48:07.852 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:48:07.852 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:48:07.852 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:48:07.852 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:48:07.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.815+0000 7fed87ef5640 10 librados: watch_flush enter 2024-09-16T03:48:07.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.815+0000 7fed87ef5640 10 librados: watch_flush exit 2024-09-16T03:48:07.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.815+0000 7fed87ef5640 20 client.764388.objecter shutdown clearing up homeless session... 2024-09-16T03:48:07.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.815+0000 7fed87ef5640 10 client.764388.objecter successfully canceled tick 2024-09-16T03:48:07.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.815+0000 7fed87ef5640 1 -- 172.21.15.42:0/544072451 >> v2:172.21.15.154:6812/254845342 conn(0x7fed640377c0 msgr2=0x7fed64039c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:48:07.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.815+0000 7fed87ef5640 1 --2- 172.21.15.42:0/544072451 >> v2:172.21.15.154:6812/254845342 conn(0x7fed640377c0 0x7fed64039c80 secure :-1 s=READY pgs=3473 cs=0 l=1 rev1=1 crypto rx=0x7fed70001ec0 tx=0x7fed70001db0 comp rx=0 tx=0).stop 2024-09-16T03:48:07.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.815+0000 7fed87ef5640 10 monclient: shutdown 2024-09-16T03:48:07.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.815+0000 7fed87ef5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:48:07.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.815+0000 7fed87ef5640 1 -- 172.21.15.42:0/544072451 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fed80132ca0 msgr2=0x7fed801cbfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:48:07.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.815+0000 7fed87ef5640 1 --2- 172.21.15.42:0/544072451 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fed80132ca0 0x7fed801cbfd0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fed78002240 tx=0x7fed78009750 comp rx=0 tx=0).stop 2024-09-16T03:48:07.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.815+0000 7fed86492640 1 -- 172.21.15.42:0/544072451 reap_dead start 2024-09-16T03:48:07.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.815+0000 7fed86492640 1 -- 172.21.15.42:0/544072451 reap_dead start 2024-09-16T03:48:07.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.815+0000 7fed87ef5640 1 -- 172.21.15.42:0/544072451 shutdown_connections 2024-09-16T03:48:07.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.815+0000 7fed87ef5640 1 -- 172.21.15.42:0/544072451 >> 172.21.15.42:0/544072451 conn(0x7fed800a2420 msgr2=0x7fed800a3d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:48:07.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.815+0000 7fed87ef5640 1 -- 172.21.15.42:0/544072451 shutdown_connections 2024-09-16T03:48:07.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.815+0000 7fed87ef5640 1 -- 172.21.15.42:0/544072451 wait complete. 2024-09-16T03:48:07.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.815+0000 7fed87ef5640 1 librados: shutdown 2024-09-16T03:48:07.856 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T03:48:07.856 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-16T03:48:07.856 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T03:48:07.972 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.967+0000 7fefe4f89640 10 monclient: get_monmap_and_config 2024-09-16T03:48:07.973 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.967+0000 7fefe4f89640 10 monclient: build_initial_monmap 2024-09-16T03:48:07.973 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.967+0000 7fefe4f89640 10 monclient: monmap: 2024-09-16T03:48:07.973 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:48:07.973 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:48:07.973 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:48:07.971849+0000 2024-09-16T03:48:07.973 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:48:07.971849+0000 2024-09-16T03:48:07.973 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:48:07.973 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:48:07.973 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:48:07.973 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:48:07.973 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:48:07.973 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:48:07.973 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:48:07.974 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:48:07.976 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.971+0000 7fefe4f89640 1 Processor -- start 2024-09-16T03:48:07.976 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.971+0000 7fefe4f89640 1 -- start start 2024-09-16T03:48:07.976 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.971+0000 7fefe4f89640 10 monclient: init 2024-09-16T03:48:07.976 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.971+0000 7fefe4f89640 10 monclient: _reopen_session rank -1 2024-09-16T03:48:07.976 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.971+0000 7fefe4f89640 10 monclient: _add_conns ranks=[4,3,0,1,2] 2024-09-16T03:48:07.976 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.971+0000 7fefe4f89640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fefe00a1330 0x7fefe00a1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:07.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.971+0000 7fefe4f89640 10 monclient(hunting): picked mon.noname-c con 0x7fefe00a1330 addr v2:172.21.15.154:3300/0 2024-09-16T03:48:07.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.971+0000 7fefe4f89640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fefe006dac0 0x7fefe00a1c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:07.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.971+0000 7fefe4f89640 10 monclient(hunting): picked mon.noname-a con 0x7fefe006dac0 addr v2:172.21.15.42:3300/0 2024-09-16T03:48:07.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.971+0000 7fefe4f89640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fefe0135100 addr v1:172.21.15.42:6789/0 2024-09-16T03:48:07.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.971+0000 7fefe4f89640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fefe009c7d0 con 0x7fefe0135100 2024-09-16T03:48:07.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.971+0000 7fefe4f89640 10 monclient(hunting): start opening mon connection 2024-09-16T03:48:07.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.971+0000 7fefe4f89640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7fefe009c090 con 0x7fefe006dac0 2024-09-16T03:48:07.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.971+0000 7fefe4f89640 10 monclient(hunting): start opening mon connection 2024-09-16T03:48:07.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.971+0000 7fefe4f89640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fefe009f610 con 0x7fefe00a1330 2024-09-16T03:48:07.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.971+0000 7fefe4f89640 10 monclient(hunting): _renew_subs 2024-09-16T03:48:07.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.971+0000 7fefe4f89640 10 monclient(hunting): authenticate will time out at 1334.941253s 2024-09-16T03:48:07.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.971+0000 7fefded9d640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fefe0135100 0x7fefe0135550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:34954/0 (socket says 172.21.15.42:34954) 2024-09-16T03:48:07.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.971+0000 7fefded9d640 1 -- 172.21.15.42:0/3244215307 learned_addr learned my addr 172.21.15.42:0/3244215307 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:48:07.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.971+0000 7fefddd9b640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fefe006dac0 0x7fefe00a1c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:48:07.978 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.971+0000 7fefddd9b640 10 monclient(hunting): get_auth_request con 0x7fefe006dac0 auth_method 0 2024-09-16T03:48:07.978 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.971+0000 7fefddd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:48:07.978 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.971+0000 7fefddd9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:48:07.978 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.971+0000 7fefddd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:48:07.978 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.971+0000 7fefddd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:48:07.978 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.971+0000 7fefde59c640 1 --2- 172.21.15.42:0/3244215307 >> v2:172.21.15.154:3300/0 conn(0x7fefe00a1330 0x7fefe00a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:48:07.978 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.971+0000 7fefddd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:48:07.978 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.971+0000 7fefddd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:48:07.978 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.971+0000 7fefde59c640 10 monclient(hunting): get_auth_request con 0x7fefe00a1330 auth_method 0 2024-09-16T03:48:07.978 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.971+0000 7fefde59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:48:07.978 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.971+0000 7fefde59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:48:07.978 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.971+0000 7fefde59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:48:07.978 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.971+0000 7fefdd59a640 1 -- 172.21.15.42:0/3244215307 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3890285661 0 0) 0x7fefe009c7d0 con 0x7fefe0135100 2024-09-16T03:48:07.978 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.971+0000 7fefdd59a640 10 monclient(hunting): _negotiate 2024-09-16T03:48:07.979 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.971+0000 7fefdd59a640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:48:07.979 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.971+0000 7fefdd59a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:48:07.979 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.971+0000 7fefdd59a640 10 monclient(hunting): my global_id is 764400 2024-09-16T03:48:07.979 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.971+0000 7fefdd59a640 1 -- 172.21.15.42:0/3244215307 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fefc8003530 con 0x7fefe0135100 2024-09-16T03:48:07.979 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.971+0000 7fefddd9b640 10 monclient(hunting): handle_auth_done global_id 764397 payload 274 2024-09-16T03:48:07.979 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.971+0000 7fefddd9b640 1 -- 172.21.15.42:0/3244215307 >> v2:172.21.15.154:3300/0 conn(0x7fefe00a1330 msgr2=0x7fefe00a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:48:07.979 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.971+0000 7fefddd9b640 1 --2- 172.21.15.42:0/3244215307 >> v2:172.21.15.154:3300/0 conn(0x7fefe00a1330 0x7fefe00a1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:07.979 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.971+0000 7fefddd9b640 1 -- 172.21.15.42:0/3244215307 >> v1:172.21.15.42:6789/0 conn(0x7fefe0135100 legacy=0x7fefe0135550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:48:07.979 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.971+0000 7fefddd9b640 10 monclient: _finish_hunting 0 2024-09-16T03:48:07.979 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.971+0000 7fefddd9b640 1 monclient: found mon.noname-a 2024-09-16T03:48:07.979 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.971+0000 7fefddd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:48:07.979 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.971+0000 7fefddd9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:48:07.979 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.971+0000 7fefddd9b640 1 -- 172.21.15.42:0/3244215307 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fefe009f8f0 con 0x7fefe006dac0 2024-09-16T03:48:07.979 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.971+0000 7fefde59c640 1 --2- 172.21.15.42:0/3244215307 >> v2:172.21.15.154:3300/0 conn(0x7fefe00a1330 0x7fefe00a1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T03:48:07.986 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.975+0000 7fefddd9b640 1 --2- 172.21.15.42:0/3244215307 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fefe006dac0 0x7fefe00a1c50 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fefe006add0 tx=0x7fefd001c430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bfd8c8fd63abd2d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:48:07.986 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.975+0000 7fefdd59a640 1 -- 172.21.15.42:0/3244215307 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefd00ab650 con 0x7fefe006dac0 2024-09-16T03:48:07.986 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.975+0000 7fefdd59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:48:07.986 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.975+0000 7fefdd59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:48:07.986 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.975+0000 7fefdd59a640 10 monclient: dump: 2024-09-16T03:48:07.987 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:48:07.987 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:48:07.987 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:07.987 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:07.987 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:48:07.987 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:48:07.987 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:48:07.987 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:48:07.987 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:48:07.987 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:48:07.987 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.975+0000 7fefdd59a640 10 monclient: _finish_auth 0 2024-09-16T03:48:07.987 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.975+0000 7fefdd59a640 10 monclient: _check_auth_tickets 2024-09-16T03:48:07.987 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.975+0000 7fefdd59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:48:07.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.975+0000 7fefdd59a640 1 -- 172.21.15.42:0/3244215307 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fefd00b0070 con 0x7fefe006dac0 2024-09-16T03:48:07.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.975+0000 7fefdd59a640 10 monclient: handle_config config(0 keys) 2024-09-16T03:48:07.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.975+0000 7fefdd59a640 1 -- 172.21.15.42:0/3244215307 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefd00aba10 con 0x7fefe006dac0 2024-09-16T03:48:07.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.975+0000 7fefe4f89640 5 monclient: authenticate success, global_id 764397 2024-09-16T03:48:07.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.975+0000 7fefe4f89640 10 monclient: get_monmap_and_config success 2024-09-16T03:48:07.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.975+0000 7fefe4f89640 4 set_mon_vals no callback set 2024-09-16T03:48:07.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.975+0000 7fefdd59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:48:07.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.975+0000 7fefdd59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:48:07.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.975+0000 7fefdd59a640 10 monclient: dump: 2024-09-16T03:48:07.988 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:48:07.988 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:48:07.988 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:07.988 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:07.989 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:48:07.989 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:48:07.989 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:48:07.989 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:48:07.989 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:48:07.989 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:48:07.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.975+0000 7fefe4f89640 10 monclient: shutdown 2024-09-16T03:48:07.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.975+0000 7fefe4f89640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:48:07.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.975+0000 7fefe4f89640 1 -- 172.21.15.42:0/3244215307 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fefe006dac0 msgr2=0x7fefe00a1c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:48:07.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.975+0000 7fefe4f89640 1 --2- 172.21.15.42:0/3244215307 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fefe006dac0 0x7fefe00a1c50 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fefe006add0 tx=0x7fefd001c430 comp rx=0 tx=0).stop 2024-09-16T03:48:07.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.975+0000 7fefe4f89640 1 -- 172.21.15.42:0/3244215307 shutdown_connections 2024-09-16T03:48:07.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.975+0000 7fefe4f89640 1 --2- 172.21.15.42:0/3244215307 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fefe006dac0 0x7fefe00a1c50 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:07.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.975+0000 7fefe4f89640 1 --2- 172.21.15.42:0/3244215307 >> v2:172.21.15.154:3300/0 conn(0x7fefe00a1330 0x7fefe00a1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:07.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.975+0000 7fefe4f89640 1 -- 172.21.15.42:0/3244215307 >> 172.21.15.42:0/3244215307 conn(0x7fefe00a2420 msgr2=0x7fefe00a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:48:07.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.975+0000 7fefe4f89640 1 -- 172.21.15.42:0/3244215307 shutdown_connections 2024-09-16T03:48:07.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.975+0000 7fefe4f89640 1 -- 172.21.15.42:0/3244215307 wait complete. 2024-09-16T03:48:07.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.975+0000 7fefe4f89640 10 monclient: build_initial_monmap 2024-09-16T03:48:07.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.975+0000 7fefe4f89640 10 monclient: monmap: 2024-09-16T03:48:07.990 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:48:07.990 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:48:07.990 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:48:07.990 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:48:07.990 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:48:07.990 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:48:07.990 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:48:07.990 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:48:07.990 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:48:07.990 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:48:07.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.975+0000 7fefe4f89640 1 librados: starting msgr at 2024-09-16T03:48:07.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.975+0000 7fefe4f89640 1 librados: starting objecter 2024-09-16T03:48:07.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.975+0000 7fefe4f89640 1 Processor -- start 2024-09-16T03:48:07.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.975+0000 7fefe4f89640 1 -- start start 2024-09-16T03:48:07.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.975+0000 7fefe4f89640 1 librados: setting wanted keys 2024-09-16T03:48:07.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.975+0000 7fefe4f89640 1 librados: calling monclient init 2024-09-16T03:48:07.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.975+0000 7fefe4f89640 10 monclient: init 2024-09-16T03:48:07.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.975+0000 7fefe4f89640 10 monclient: _reopen_session rank -1 2024-09-16T03:48:07.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.975+0000 7fefe4f89640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:48:07.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.975+0000 7fefe4f89640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fefe00a1330 0x7fefe011bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:07.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.975+0000 7fefe4f89640 10 monclient(hunting): picked mon.noname-c con 0x7fefe00a1330 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:48:07.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.975+0000 7fefe4f89640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fefe0135100 0x7fefe011c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:07.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.975+0000 7fefe4f89640 10 monclient(hunting): picked mon.noname-b con 0x7fefe0135100 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:48:07.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.975+0000 7fefe4f89640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fefe011c9f0 0x7fefe0133900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:07.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.975+0000 7fefe4f89640 10 monclient(hunting): picked mon.noname-a con 0x7fefe011c9f0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:48:07.992 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.975+0000 7fefe4f89640 10 monclient(hunting): start opening mon connection 2024-09-16T03:48:07.992 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.975+0000 7fefe4f89640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fefe013fb80 con 0x7fefe011c9f0 2024-09-16T03:48:07.992 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.975+0000 7fefe4f89640 10 monclient(hunting): start opening mon connection 2024-09-16T03:48:07.992 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.975+0000 7fefe4f89640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fefe009ef80 con 0x7fefe0135100 2024-09-16T03:48:07.992 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.975+0000 7fefe4f89640 10 monclient(hunting): start opening mon connection 2024-09-16T03:48:07.992 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.975+0000 7fefe4f89640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fefe013fd00 con 0x7fefe00a1330 2024-09-16T03:48:07.992 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.975+0000 7fefe4f89640 10 monclient(hunting): _renew_subs 2024-09-16T03:48:07.992 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.975+0000 7fefe4f89640 10 monclient(hunting): authenticate will time out at 1334.946508s 2024-09-16T03:48:07.992 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.975+0000 7fefde59c640 1 -- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fefe00a1330 msgr2=0x7fefe011bf70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T03:48:07.992 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.975+0000 7fefde59c640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fefe00a1330 0x7fefe011bf70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:48:07.992 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.975+0000 7fefded9d640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fefe011c9f0 0x7fefe0133900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:48:07.992 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.975+0000 7fefded9d640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fefe011c9f0 0x7fefe0133900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:60826/0 (socket says 172.21.15.42:60826) 2024-09-16T03:48:07.992 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.975+0000 7fefded9d640 1 -- 172.21.15.42:0/1631530123 learned_addr learned my addr 172.21.15.42:0/1631530123 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:48:07.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.975+0000 7fefded9d640 10 monclient(hunting): get_auth_request con 0x7fefe011c9f0 auth_method 0 2024-09-16T03:48:07.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.975+0000 7fefded9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:48:07.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.975+0000 7fefded9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:48:07.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.975+0000 7fefded9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:48:07.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.975+0000 7fefded9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:48:07.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.975+0000 7fefded9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:48:07.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.975+0000 7fefded9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:48:07.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.975+0000 7fefddd9b640 1 --2- 172.21.15.42:0/1631530123 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fefe0135100 0x7fefe011c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:48:07.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.975+0000 7fefddd9b640 10 monclient(hunting): get_auth_request con 0x7fefe0135100 auth_method 0 2024-09-16T03:48:07.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.975+0000 7fefddd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:48:07.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.975+0000 7fefddd9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:48:07.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.975+0000 7fefddd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:48:07.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.975+0000 7fefded9d640 10 monclient(hunting): handle_auth_done global_id 764403 payload 824 2024-09-16T03:48:07.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.975+0000 7fefded9d640 1 -- 172.21.15.42:0/1631530123 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fefe00a1330 msgr2=0x7fefe011bf70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:48:07.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.975+0000 7fefded9d640 1 --2- 172.21.15.42:0/1631530123 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fefe00a1330 0x7fefe011bf70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:07.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.975+0000 7fefded9d640 1 -- 172.21.15.42:0/1631530123 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fefe0135100 msgr2=0x7fefe011c4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:48:07.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.975+0000 7fefded9d640 1 --2- 172.21.15.42:0/1631530123 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fefe0135100 0x7fefe011c4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:07.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.975+0000 7fefded9d640 10 monclient: _finish_hunting 0 2024-09-16T03:48:07.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.975+0000 7fefded9d640 1 monclient: found mon.noname-a 2024-09-16T03:48:07.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.975+0000 7fefded9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:48:07.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.975+0000 7fefded9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:48:07.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.975+0000 7fefded9d640 1 -- 172.21.15.42:0/1631530123 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fefe00713a0 con 0x7fefe011c9f0 2024-09-16T03:48:07.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.975+0000 7fefded9d640 1 --2- 172.21.15.42:0/1631530123 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fefe011c9f0 0x7fefe0133900 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fefd800cb40 tx=0x7fefd800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:48:07.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.975+0000 7fefceffd640 10 client.?.objecter ms_handle_connect 0x7fefe011c9f0 2024-09-16T03:48:07.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.975+0000 7fefceffd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:48:07.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.975+0000 7fefceffd640 1 -- 172.21.15.42:0/1631530123 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefd8017070 con 0x7fefe011c9f0 2024-09-16T03:48:07.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.975+0000 7fefceffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:48:07.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.975+0000 7fefceffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:48:07.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.975+0000 7fefceffd640 10 monclient: dump: 2024-09-16T03:48:07.995 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:48:07.995 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:48:07.995 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:07.995 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:07.995 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:48:07.995 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:48:07.995 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:48:07.995 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:48:07.995 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:48:07.995 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:48:07.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.975+0000 7fefceffd640 10 monclient: _finish_auth 0 2024-09-16T03:48:07.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.975+0000 7fefceffd640 10 monclient: _check_auth_tickets 2024-09-16T03:48:07.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.975+0000 7fefceffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:48:07.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.975+0000 7fefceffd640 1 -- 172.21.15.42:0/1631530123 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fefd800ac40 con 0x7fefe011c9f0 2024-09-16T03:48:07.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.975+0000 7fefceffd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:48:07.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.975+0000 7fefceffd640 1 -- 172.21.15.42:0/1631530123 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefd8012400 con 0x7fefe011c9f0 2024-09-16T03:48:07.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.975+0000 7fefe4f89640 5 monclient: authenticate success, global_id 764403 2024-09-16T03:48:07.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.975+0000 7fefe4f89640 10 monclient: _renew_subs 2024-09-16T03:48:07.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.975+0000 7fefe4f89640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:48:07.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.975+0000 7fefe4f89640 1 -- 172.21.15.42:0/1631530123 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fefe00a5220 con 0x7fefe011c9f0 2024-09-16T03:48:07.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.975+0000 7fefdcd99640 4 set_mon_vals no callback set 2024-09-16T03:48:07.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.975+0000 7fefceffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:48:07.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.975+0000 7fefceffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:48:07.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.975+0000 7fefceffd640 10 monclient: dump: 2024-09-16T03:48:07.996 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:48:07.996 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:48:07.996 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:07.997 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:07.997 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:48:07.997 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:48:07.997 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:48:07.997 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:48:07.997 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:48:07.997 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:48:07.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.975+0000 7fefe4f89640 10 client.764403.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:48:07.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.975+0000 7fefe4f89640 10 monclient: _renew_subs 2024-09-16T03:48:07.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.975+0000 7fefe4f89640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:48:07.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.975+0000 7fefe4f89640 1 -- 172.21.15.42:0/1631530123 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fefe011e3d0 con 0x7fefe011c9f0 2024-09-16T03:48:07.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.975+0000 7fefe4f89640 1 librados: init done 2024-09-16T03:48:07.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.979+0000 7fefe4f89640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:48:07.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.979+0000 7fefe4f89640 1 --2- 172.21.15.42:0/1631530123 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fefe006dac0 0x7fefe011e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:07.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.979+0000 7fefe4f89640 10 monclient(hunting): start opening mon connection 2024-09-16T03:48:07.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.979+0000 7fefe4f89640 1 -- 172.21.15.42:0/1631530123 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fefe011d200 con 0x7fefe006dac0 2024-09-16T03:48:07.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.979+0000 7fefceffd640 1 -- 172.21.15.42:0/1631530123 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fefd801a070 con 0x7fefe011c9f0 2024-09-16T03:48:07.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.979+0000 7fefde59c640 1 --2- 172.21.15.42:0/1631530123 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fefe006dac0 0x7fefe011e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:48:07.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.979+0000 7fefde59c640 10 monclient: get_auth_request con 0x7fefe006dac0 auth_method 0 2024-09-16T03:48:07.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.979+0000 7fefde59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:48:07.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.979+0000 7fefde59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:48:07.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.979+0000 7fefde59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:48:07.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.979+0000 7fefceffd640 1 --2- 172.21.15.42:0/1631530123 >> v2:172.21.15.154:6812/254845342 conn(0x7fefc00377c0 0x7fefc0039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:07.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.979+0000 7fefceffd640 1 -- 172.21.15.42:0/1631530123 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(183..183 src has 1..183) ==== 6151+0+0 (secure 0 0 0) 0x7fefd8048e50 con 0x7fefe011c9f0 2024-09-16T03:48:07.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.979+0000 7fefceffd640 10 client.764403.objecter ms_dispatch 0x7fefe0000ba0 osd_map(183..183 src has 1..183) 2024-09-16T03:48:07.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.979+0000 7fefceffd640 3 client.764403.objecter handle_osd_map got epochs [183,183] > 0 2024-09-16T03:48:07.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.979+0000 7fefceffd640 3 client.764403.objecter handle_osd_map decoding full epoch 183 2024-09-16T03:48:07.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.979+0000 7fefde59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:48:07.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.979+0000 7fefde59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:48:07.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.979+0000 7fefde59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:48:07.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.979+0000 7fefceffd640 20 client.764403.objecter dump_active .. 0 homeless 2024-09-16T03:48:07.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.979+0000 7fefddd9b640 1 --2- 172.21.15.42:0/1631530123 >> v2:172.21.15.154:6812/254845342 conn(0x7fefc00377c0 0x7fefc0039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:48:07.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.979+0000 7fefddd9b640 10 monclient: get_auth_request con 0x7fefc00377c0 auth_method 0 2024-09-16T03:48:07.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.979+0000 7fefde59c640 10 monclient(hunting): handle_auth_done global_id 774196 payload 824 2024-09-16T03:48:07.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.979+0000 7fefde59c640 1 -- 172.21.15.42:0/1631530123 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fefe0135550 con 0x7fefe006dac0 2024-09-16T03:48:07.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.979+0000 7fefde59c640 1 --2- 172.21.15.42:0/1631530123 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fefe006dac0 0x7fefe011e610 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fefd400e710 tx=0x7fefd400ebd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:48:07.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.979+0000 7fefceffd640 10 client.764403.objecter ms_handle_connect 0x7fefe006dac0 2024-09-16T03:48:07.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.979+0000 7fefceffd640 10 client.764403.objecter resend_mon_ops 2024-09-16T03:48:07.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.979+0000 7fefceffd640 1 -- 172.21.15.42:0/1631530123 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefd400f040 con 0x7fefe006dac0 2024-09-16T03:48:07.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.979+0000 7fefceffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:48:07.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.979+0000 7fefceffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:48:08.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.979+0000 7fefceffd640 10 monclient: dump: 2024-09-16T03:48:08.000 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:48:08.000 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:48:08.000 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:08.000 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:08.000 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:48:08.000 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:48:08.000 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:48:08.000 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:48:08.000 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:48:08.000 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:48:08.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.979+0000 7fefceffd640 10 client.764403.objecter ms_handle_connect 0x7fefc00377c0 2024-09-16T03:48:08.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.979+0000 7fefddd9b640 1 --2- 172.21.15.42:0/1631530123 >> v2:172.21.15.154:6812/254845342 conn(0x7fefc00377c0 0x7fefc0039c80 secure :-1 s=READY pgs=3474 cs=0 l=1 rev1=1 crypto rx=0x7fefd00af000 tx=0x7fefd00a7000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:48:08.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.979+0000 7fefceffd640 1 -- 172.21.15.42:0/1631530123 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fefd4004030 con 0x7fefe006dac0 2024-09-16T03:48:08.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.979+0000 7fefceffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:48:08.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.979+0000 7fefceffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:48:08.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.979+0000 7fefceffd640 1 -- 172.21.15.42:0/1631530123 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fefe006dac0 msgr2=0x7fefe011e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:48:08.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.979+0000 7fefceffd640 1 --2- 172.21.15.42:0/1631530123 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fefe006dac0 0x7fefe011e610 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fefd400e710 tx=0x7fefd400ebd0 comp rx=0 tx=0).stop 2024-09-16T03:48:08.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.979+0000 7fefceffd640 1 -- 172.21.15.42:0/1631530123 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fefe006dac0 msgr2=0x7fefe011e610 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:48:08.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.979+0000 7fefceffd640 1 --2- 172.21.15.42:0/1631530123 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fefe006dac0 0x7fefe011e610 secure :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fefd400e710 tx=0x7fefd400ebd0 comp rx=0 tx=0).stop 2024-09-16T03:48:08.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.987+0000 7fefe4f89640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:48:08.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.987+0000 7fefe4f89640 1 --2- 172.21.15.42:0/1631530123 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fefe0133e40 0x7fefe0134250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:08.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.987+0000 7fefe4f89640 10 monclient(hunting): start opening mon connection 2024-09-16T03:48:08.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.987+0000 7fefe4f89640 1 -- 172.21.15.42:0/1631530123 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fefe0134790 con 0x7fefe0133e40 2024-09-16T03:48:08.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.987+0000 7fefde59c640 1 --2- 172.21.15.42:0/1631530123 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fefe0133e40 0x7fefe0134250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:48:08.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.987+0000 7fefde59c640 10 monclient: get_auth_request con 0x7fefe0133e40 auth_method 0 2024-09-16T03:48:08.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.987+0000 7fefde59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:48:08.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.987+0000 7fefde59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:48:08.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.987+0000 7fefde59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:48:08.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.987+0000 7fefde59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:48:08.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.987+0000 7fefde59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:48:08.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.987+0000 7fefde59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:48:08.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.987+0000 7fefde59c640 10 monclient(hunting): handle_auth_done global_id 774199 payload 824 2024-09-16T03:48:08.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.987+0000 7fefde59c640 1 -- 172.21.15.42:0/1631530123 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fefe0069040 con 0x7fefe0133e40 2024-09-16T03:48:08.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.987+0000 7fefceffd640 10 client.764403.objecter ms_handle_connect 0x7fefe0133e40 2024-09-16T03:48:08.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.987+0000 7fefceffd640 10 client.764403.objecter resend_mon_ops 2024-09-16T03:48:08.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.987+0000 7fefde59c640 1 --2- 172.21.15.42:0/1631530123 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fefe0133e40 0x7fefe0134250 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fefd400e710 tx=0x7fefd4004740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:48:08.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.987+0000 7fefceffd640 1 -- 172.21.15.42:0/1631530123 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefd4029020 con 0x7fefe0133e40 2024-09-16T03:48:08.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.987+0000 7fefceffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:48:08.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.987+0000 7fefceffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:48:08.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.987+0000 7fefceffd640 10 monclient: dump: 2024-09-16T03:48:08.004 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:48:08.004 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:48:08.004 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:08.004 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:08.004 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:48:08.005 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:48:08.005 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:48:08.005 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:48:08.005 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:48:08.005 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:48:08.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.987+0000 7fefceffd640 1 -- 172.21.15.42:0/1631530123 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4700 (secure 0 0 0) 0x7fefd4003af0 con 0x7fefe0133e40 2024-09-16T03:48:08.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.987+0000 7fefceffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:48:08.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.987+0000 7fefceffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:48:08.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.987+0000 7fefceffd640 1 -- 172.21.15.42:0/1631530123 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fefe0133e40 msgr2=0x7fefe0134250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:48:08.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.987+0000 7fefceffd640 1 --2- 172.21.15.42:0/1631530123 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fefe0133e40 0x7fefe0134250 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fefd400e710 tx=0x7fefd4004740 comp rx=0 tx=0).stop 2024-09-16T03:48:08.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.987+0000 7fefceffd640 1 -- 172.21.15.42:0/1631530123 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fefe0133e40 msgr2=0x7fefe0134250 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:48:08.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.987+0000 7fefceffd640 1 --2- 172.21.15.42:0/1631530123 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fefe0133e40 0x7fefe0134250 secure :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fefd400e710 tx=0x7fefd4004740 comp rx=0 tx=0).stop 2024-09-16T03:48:08.006 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:48:08.006 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:48:08.006 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:48:08.006 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T03:48:08.006 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 58, 2024-09-16T03:48:08.006 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 25571, 2024-09-16T03:48:08.006 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:48:08.006 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:48:08.006 INFO:teuthology.orchestra.run.smithi042.stdout: 1 2024-09-16T03:48:08.006 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:48:08.006 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 0, 2024-09-16T03:48:08.006 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:48:08.007 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:48:08.007 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:48:08.007 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:48:08.007 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:48:08.007 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:48:08.007 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:48:08.007 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:48:08.007 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:48:08.007 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:48:08.007 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:48:08.007 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:48:08.007 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:48:08.008 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:48:08.008 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:48:08.008 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:48:08.008 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:48:08.008 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:48:08.008 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:48:08.008 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:48:08.008 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:48:08.008 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:48:08.008 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:48:08.008 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:48:08.008 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:48:08.008 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:48:08.008 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:48:08.009 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:48:08.009 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:48:08.009 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:48:08.009 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:48:08.009 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:48:08.009 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:48:08.009 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:48:08.009 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:48:08.009 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:48:08.009 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:48:08.009 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:48:08.009 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:48:08.009 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:48:08.010 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:48:08.010 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:48:08.010 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:48:08.010 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:48:08.010 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:48:08.010 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:48:08.010 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:48:08.010 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:48:08.010 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:48:08.010 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:48:08.010 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:48:08.010 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:48:08.010 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:48:08.010 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:48:08.011 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:48:08.011 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:48:08.011 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:48:08.011 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:48:08.011 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:48:08.011 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:48:08.011 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:48:08.011 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:48:08.011 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:48:08.011 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:48:08.011 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:48:08.011 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:48:08.011 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:48:08.012 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:48:08.012 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:48:08.012 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:48:08.012 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:48:08.012 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:48:08.012 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:48:08.012 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:48:08.012 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:48:08.012 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:48:08.012 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:48:08.012 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:48:08.012 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:48:08.012 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:48:08.013 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:48:08.013 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:48:08.013 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:48:08.013 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:48:08.013 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:48:08.013 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:48:08.013 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:48:08.013 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:48:08.013 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:48:08.013 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:48:08.013 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:48:08.013 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:48:08.013 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:48:08.013 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:48:08.013 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:48:08.014 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:48:08.014 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:48:08.014 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:48:08.014 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:48:08.014 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:48:08.014 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:48:08.014 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:48:08.014 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:48:08.014 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:48:08.014 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:48:08.014 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:48:08.014 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:48:08.014 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:48:08.014 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:48:08.015 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:48:08.015 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:48:08.015 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:48:08.015 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:48:08.015 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:48:08.015 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:48:08.015 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:48:08.015 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:48:08.015 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:48:08.015 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:48:08.015 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:48:08.015 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:48:08.015 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:48:08.016 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:48:08.016 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:48:08.016 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:48:08.016 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:48:08.016 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:48:08.016 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:48:08.016 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:48:08.016 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:48:08.016 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:48:08.016 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:48:08.016 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:48:08.016 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:48:08.016 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:48:08.016 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:48:08.017 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:48:08.017 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:48:08.017 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:48:08.017 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:48:08.017 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:48:08.017 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T03:48:08.017 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:48:08.017 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:48:08.017 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:48:08.017 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:48:08.017 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:48:08.017 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:48:08.017 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:48:08.017 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:48:08.017 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:48:08.018 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:48:08.018 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 3 2024-09-16T03:48:08.018 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:48:08.018 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:48:08.018 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:48:08.018 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:48:08.018 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:48:08.019 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.987+0000 7fefe4f89640 10 librados: watch_flush enter 2024-09-16T03:48:08.020 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.987+0000 7fefe4f89640 10 librados: watch_flush exit 2024-09-16T03:48:08.020 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.987+0000 7fefe4f89640 20 client.764403.objecter shutdown clearing up homeless session... 2024-09-16T03:48:08.020 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.987+0000 7fefe4f89640 10 client.764403.objecter successfully canceled tick 2024-09-16T03:48:08.020 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.987+0000 7fefe4f89640 1 -- 172.21.15.42:0/1631530123 >> v2:172.21.15.154:6812/254845342 conn(0x7fefc00377c0 msgr2=0x7fefc0039c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:48:08.020 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.987+0000 7fefe4f89640 1 --2- 172.21.15.42:0/1631530123 >> v2:172.21.15.154:6812/254845342 conn(0x7fefc00377c0 0x7fefc0039c80 secure :-1 s=READY pgs=3474 cs=0 l=1 rev1=1 crypto rx=0x7fefd00af000 tx=0x7fefd00a7000 comp rx=0 tx=0).stop 2024-09-16T03:48:08.020 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.987+0000 7fefe4f89640 10 monclient: shutdown 2024-09-16T03:48:08.020 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.987+0000 7fefe4f89640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:48:08.020 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.987+0000 7fefded9d640 1 -- 172.21.15.42:0/1631530123 reap_dead start 2024-09-16T03:48:08.020 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.987+0000 7fefe4f89640 1 -- 172.21.15.42:0/1631530123 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fefe011c9f0 msgr2=0x7fefe0133900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:48:08.020 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.987+0000 7fefe4f89640 1 --2- 172.21.15.42:0/1631530123 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fefe011c9f0 0x7fefe0133900 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fefd800cb40 tx=0x7fefd800a060 comp rx=0 tx=0).stop 2024-09-16T03:48:08.020 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.987+0000 7fefe4f89640 1 -- 172.21.15.42:0/1631530123 shutdown_connections 2024-09-16T03:48:08.020 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.987+0000 7fefe4f89640 1 --2- 172.21.15.42:0/1631530123 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fefe011c9f0 0x7fefe0133900 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:08.021 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.987+0000 7fefe4f89640 1 -- 172.21.15.42:0/1631530123 >> 172.21.15.42:0/1631530123 conn(0x7fefe00a2420 msgr2=0x7fefe00718f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:48:08.021 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.987+0000 7fefe4f89640 1 -- 172.21.15.42:0/1631530123 shutdown_connections 2024-09-16T03:48:08.021 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.987+0000 7fefe4f89640 1 -- 172.21.15.42:0/1631530123 wait complete. 2024-09-16T03:48:08.021 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:07.987+0000 7fefe4f89640 1 librados: shutdown 2024-09-16T03:48:08.021 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T03:48:08.021 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-16T03:48:28.022 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-16T03:48:28.022 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-09-16T03:48:28.022 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-16T03:48:28.023 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-16T03:48:28.025 INFO:tasks.ceph.mon.c:Started 2024-09-16T03:48:28.026 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T03:48:28.026 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:48:28.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.147+0000 7fa630bbd640 10 monclient: get_monmap_and_config 2024-09-16T03:48:28.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.147+0000 7fa630bbd640 10 monclient: build_initial_monmap 2024-09-16T03:48:28.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.147+0000 7fa630bbd640 10 monclient: monmap: 2024-09-16T03:48:28.151 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:48:28.151 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:48:28.151 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:48:28.149843+0000 2024-09-16T03:48:28.151 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:48:28.149843+0000 2024-09-16T03:48:28.151 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:48:28.151 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:48:28.151 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:48:28.152 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:48:28.152 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:48:28.152 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:48:28.152 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:48:28.152 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:48:28.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.147+0000 7fa630bbd640 1 Processor -- start 2024-09-16T03:48:28.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.147+0000 7fa630bbd640 1 -- start start 2024-09-16T03:48:28.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.147+0000 7fa630bbd640 10 monclient: init 2024-09-16T03:48:28.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.147+0000 7fa630bbd640 10 monclient: _reopen_session rank -1 2024-09-16T03:48:28.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.147+0000 7fa630bbd640 10 monclient: _add_conns ranks=[4,0,1,2,3] 2024-09-16T03:48:28.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.147+0000 7fa630bbd640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fa62c130a40 0x7fa62c0a2250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:28.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.147+0000 7fa630bbd640 10 monclient(hunting): picked mon.noname-c con 0x7fa62c130a40 addr v2:172.21.15.154:3300/0 2024-09-16T03:48:28.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.147+0000 7fa630bbd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa62c130e20 addr v1:172.21.15.42:6789/0 2024-09-16T03:48:28.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.147+0000 7fa630bbd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa62c131290 addr v1:172.21.15.154:6789/0 2024-09-16T03:48:28.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.147+0000 7fa630bbd640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa62c073260 con 0x7fa62c130e20 2024-09-16T03:48:28.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.147+0000 7fa630bbd640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa62c0a0fd0 con 0x7fa62c131290 2024-09-16T03:48:28.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.147+0000 7fa630bbd640 10 monclient(hunting): start opening mon connection 2024-09-16T03:48:28.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.147+0000 7fa630bbd640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fa62c0a12b0 con 0x7fa62c130a40 2024-09-16T03:48:28.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.147+0000 7fa630bbd640 10 monclient(hunting): _renew_subs 2024-09-16T03:48:28.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.147+0000 7fa630bbd640 10 monclient(hunting): authenticate will time out at 1355.119138s 2024-09-16T03:48:28.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.147+0000 7fa629d9b640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fa62c130e20 0x7fa62c072440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:46828/0 (socket says 172.21.15.42:46828) 2024-09-16T03:48:28.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.147+0000 7fa629d9b640 1 -- 172.21.15.42:0/164112817 learned_addr learned my addr 172.21.15.42:0/164112817 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:48:28.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.147+0000 7fa62a59c640 1 --2- 172.21.15.42:0/164112817 >> v2:172.21.15.154:3300/0 conn(0x7fa62c130a40 0x7fa62c0a2250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:48:28.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.147+0000 7fa62959a640 1 -- 172.21.15.42:0/164112817 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4278672224 0 0) 0x7fa62c073260 con 0x7fa62c130e20 2024-09-16T03:48:28.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.147+0000 7fa62959a640 10 monclient(hunting): _negotiate 2024-09-16T03:48:28.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.147+0000 7fa62959a640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:48:28.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.147+0000 7fa62959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:48:28.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.147+0000 7fa62959a640 10 monclient(hunting): my global_id is 764430 2024-09-16T03:48:28.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.147+0000 7fa62959a640 1 -- 172.21.15.42:0/164112817 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa618003530 con 0x7fa62c130e20 2024-09-16T03:48:28.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.147+0000 7fa62a59c640 10 monclient(hunting): get_auth_request con 0x7fa62c130a40 auth_method 0 2024-09-16T03:48:28.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.147+0000 7fa62a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:48:28.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.147+0000 7fa62a59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:48:28.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.147+0000 7fa62a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:48:28.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.147+0000 7fa62a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:48:28.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.147+0000 7fa62a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:48:28.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.147+0000 7fa62959a640 1 -- 172.21.15.42:0/164112817 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1794441744 0 0) 0x7fa618003530 con 0x7fa62c130e20 2024-09-16T03:48:28.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.147+0000 7fa62a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:48:28.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.147+0000 7fa62959a640 1 -- 172.21.15.42:0/164112817 >> v2:172.21.15.154:3300/0 conn(0x7fa62c130a40 msgr2=0x7fa62c0a2250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:48:28.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.147+0000 7fa62959a640 1 --2- 172.21.15.42:0/164112817 >> v2:172.21.15.154:3300/0 conn(0x7fa62c130a40 0x7fa62c0a2250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:28.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.147+0000 7fa62959a640 1 -- 172.21.15.42:0/164112817 >> v1:172.21.15.154:6789/0 conn(0x7fa62c131290 legacy=0x7fa62c072b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:48:28.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.147+0000 7fa62959a640 10 monclient: _finish_hunting 0 2024-09-16T03:48:28.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.147+0000 7fa62959a640 1 monclient: found mon.noname-a-legacy 2024-09-16T03:48:28.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.147+0000 7fa62959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:48:28.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.147+0000 7fa62959a640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.42:6789/0 2024-09-16T03:48:28.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.147+0000 7fa62959a640 1 -- 172.21.15.42:0/164112817 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa62c0a1590 con 0x7fa62c130e20 2024-09-16T03:48:28.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.147+0000 7fa62959a640 10 monclient: _finish_auth 0 2024-09-16T03:48:28.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.147+0000 7fa62959a640 10 monclient: _check_auth_tickets 2024-09-16T03:48:28.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.147+0000 7fa62959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:48:28.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.147+0000 7fa62959a640 1 -- 172.21.15.42:0/164112817 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2177413734 0 0) 0x7fa614001680 con 0x7fa62c130e20 2024-09-16T03:48:28.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.147+0000 7fa62959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:48:28.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.147+0000 7fa62959a640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T03:48:28.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.147+0000 7fa62959a640 10 monclient: dump: 2024-09-16T03:48:28.166 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:48:28.166 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:48:28.166 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:28.166 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:28.166 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:48:28.166 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:48:28.166 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:48:28.166 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:48:28.166 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:48:28.166 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:48:28.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.147+0000 7fa62959a640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] but i'm connected to v1:172.21.15.42:6789/0, reconnecting 2024-09-16T03:48:28.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.147+0000 7fa62959a640 10 monclient: _reopen_session rank -1 2024-09-16T03:48:28.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.147+0000 7fa62959a640 1 -- 172.21.15.42:0/164112817 >> v1:172.21.15.42:6789/0 conn(0x7fa62c130e20 legacy=0x7fa62c072440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:48:28.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.147+0000 7fa62959a640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:48:28.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.151+0000 7fa62959a640 1 --2- 172.21.15.42:0/164112817 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa6180039c0 0x7fa618003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:28.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.151+0000 7fa62959a640 10 monclient(hunting): picked mon.b con 0x7fa6180039c0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:48:28.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.151+0000 7fa62959a640 1 --2- 172.21.15.42:0/164112817 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa618004810 0x7fa618004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:28.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.151+0000 7fa62959a640 10 monclient(hunting): picked mon.a con 0x7fa618004810 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:48:28.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.151+0000 7fa62959a640 1 --2- 172.21.15.42:0/164112817 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa6180052f0 0x7fa61800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:28.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.151+0000 7fa62959a640 10 monclient(hunting): picked mon.c con 0x7fa6180052f0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:48:28.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.151+0000 7fa62959a640 10 monclient(hunting): start opening mon connection 2024-09-16T03:48:28.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.151+0000 7fa62959a640 1 -- 172.21.15.42:0/164112817 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fa62c0a12b0 con 0x7fa618004810 2024-09-16T03:48:28.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.151+0000 7fa62959a640 10 monclient(hunting): start opening mon connection 2024-09-16T03:48:28.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.151+0000 7fa62959a640 1 -- 172.21.15.42:0/164112817 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa61800c1c0 con 0x7fa6180039c0 2024-09-16T03:48:28.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.151+0000 7fa62959a640 10 monclient(hunting): start opening mon connection 2024-09-16T03:48:28.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.151+0000 7fa62959a640 1 -- 172.21.15.42:0/164112817 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fa61800c3c0 con 0x7fa6180052f0 2024-09-16T03:48:28.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.151+0000 7fa62959a640 10 monclient(hunting): _renew_subs 2024-09-16T03:48:28.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.151+0000 7fa62959a640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:48:28.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.151+0000 7fa629d9b640 1 --2- 172.21.15.42:0/164112817 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa618004810 0x7fa618004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:48:28.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.151+0000 7fa629d9b640 10 monclient(hunting): get_auth_request con 0x7fa618004810 auth_method 0 2024-09-16T03:48:28.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.151+0000 7fa629d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:48:28.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.151+0000 7fa629d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:48:28.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.151+0000 7fa629d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:48:28.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.151+0000 7fa62ad9d640 1 --2- 172.21.15.42:0/164112817 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa6180052f0 0x7fa61800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:48:28.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.151+0000 7fa62ad9d640 10 monclient(hunting): get_auth_request con 0x7fa6180052f0 auth_method 0 2024-09-16T03:48:28.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.151+0000 7fa62ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:48:28.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.151+0000 7fa62ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:48:28.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.151+0000 7fa62ad9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:48:28.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.151+0000 7fa629d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:48:28.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.151+0000 7fa629d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:48:28.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.151+0000 7fa629d9b640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T03:48:28.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.151+0000 7fa62ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:48:28.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.151+0000 7fa62ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:48:28.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.151+0000 7fa62ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T03:48:28.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.151+0000 7fa629d9b640 10 monclient(hunting): handle_auth_done global_id 764430 payload 293 2024-09-16T03:48:28.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.151+0000 7fa629d9b640 1 -- 172.21.15.42:0/164112817 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa6180052f0 msgr2=0x7fa61800b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:48:28.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.151+0000 7fa629d9b640 1 --2- 172.21.15.42:0/164112817 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa6180052f0 0x7fa61800b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:28.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.151+0000 7fa629d9b640 1 -- 172.21.15.42:0/164112817 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa6180039c0 msgr2=0x7fa618003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:48:28.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.151+0000 7fa629d9b640 1 --2- 172.21.15.42:0/164112817 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa6180039c0 0x7fa618003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:28.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.151+0000 7fa629d9b640 10 monclient: _finish_hunting 0 2024-09-16T03:48:28.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.151+0000 7fa629d9b640 1 monclient: found mon.a 2024-09-16T03:48:28.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.151+0000 7fa629d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:48:28.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.151+0000 7fa629d9b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:48:28.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.151+0000 7fa629d9b640 1 -- 172.21.15.42:0/164112817 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa618003530 con 0x7fa618004810 2024-09-16T03:48:28.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.151+0000 7fa629d9b640 10 monclient: _finish_auth 0 2024-09-16T03:48:28.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.151+0000 7fa629d9b640 10 monclient: _check_auth_tickets 2024-09-16T03:48:28.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.151+0000 7fa629d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:48:28.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.151+0000 7fa62ad9d640 1 -- 172.21.15.42:0/164112817 reap_dead start 2024-09-16T03:48:28.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.151+0000 7fa630bbd640 5 monclient: authenticate success, global_id 764430 2024-09-16T03:48:28.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.151+0000 7fa630bbd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:48:28.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.151+0000 7fa629d9b640 1 --2- 172.21.15.42:0/164112817 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa618004810 0x7fa618004c90 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fa61400c190 tx=0x7fa61401cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f8783d995a62404 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:48:28.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.151+0000 7fa62959a640 1 -- 172.21.15.42:0/164112817 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa614007d00 con 0x7fa618004810 2024-09-16T03:48:28.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.151+0000 7fa62959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:48:28.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.151+0000 7fa62959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:48:28.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.151+0000 7fa62959a640 10 monclient: dump: 2024-09-16T03:48:28.171 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:48:28.171 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:48:28.171 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:28.172 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:28.172 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:48:28.172 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:48:28.172 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:48:28.172 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:48:28.172 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:48:28.172 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:48:28.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.151+0000 7fa62959a640 1 -- 172.21.15.42:0/164112817 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa6140a9020 con 0x7fa618004810 2024-09-16T03:48:28.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.151+0000 7fa630bbd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:48:28.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.151+0000 7fa62959a640 10 monclient: handle_config config(0 keys) 2024-09-16T03:48:28.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.151+0000 7fa630bbd640 10 monclient: get_monmap_and_config success 2024-09-16T03:48:28.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.151+0000 7fa630bbd640 4 set_mon_vals no callback set 2024-09-16T03:48:28.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.151+0000 7fa62959a640 1 -- 172.21.15.42:0/164112817 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa614005ae0 con 0x7fa618004810 2024-09-16T03:48:28.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.151+0000 7fa62959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:48:28.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.151+0000 7fa62959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:48:28.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.151+0000 7fa62959a640 10 monclient: dump: 2024-09-16T03:48:28.173 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:48:28.173 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:48:28.173 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:28.173 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:28.173 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:48:28.173 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:48:28.173 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:48:28.173 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:48:28.174 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:48:28.174 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:48:28.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.151+0000 7fa630bbd640 10 monclient: shutdown 2024-09-16T03:48:28.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.151+0000 7fa630bbd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:48:28.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.151+0000 7fa630bbd640 1 -- 172.21.15.42:0/164112817 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa618004810 msgr2=0x7fa618004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:48:28.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.151+0000 7fa630bbd640 1 --2- 172.21.15.42:0/164112817 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa618004810 0x7fa618004c90 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fa61400c190 tx=0x7fa61401cbc0 comp rx=0 tx=0).stop 2024-09-16T03:48:28.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.151+0000 7fa630bbd640 1 -- 172.21.15.42:0/164112817 shutdown_connections 2024-09-16T03:48:28.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.151+0000 7fa630bbd640 1 --2- 172.21.15.42:0/164112817 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa618004810 0x7fa618004c90 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:28.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.151+0000 7fa630bbd640 1 -- 172.21.15.42:0/164112817 >> 172.21.15.42:0/164112817 conn(0x7fa62c06bc40 msgr2=0x7fa62c06c050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:48:28.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.151+0000 7fa630bbd640 1 -- 172.21.15.42:0/164112817 shutdown_connections 2024-09-16T03:48:28.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.151+0000 7fa630bbd640 1 -- 172.21.15.42:0/164112817 wait complete. 2024-09-16T03:48:28.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.155+0000 7fa630bbd640 10 monclient: build_initial_monmap 2024-09-16T03:48:28.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.155+0000 7fa630bbd640 10 monclient: monmap: 2024-09-16T03:48:28.175 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:48:28.175 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:48:28.175 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:48:28.175 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:48:28.175 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:48:28.175 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:48:28.175 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:48:28.175 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:48:28.175 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:48:28.175 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:48:28.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.155+0000 7fa630bbd640 1 librados: starting msgr at 2024-09-16T03:48:28.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.155+0000 7fa630bbd640 1 librados: starting objecter 2024-09-16T03:48:28.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.155+0000 7fa630bbd640 1 Processor -- start 2024-09-16T03:48:28.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.155+0000 7fa630bbd640 1 -- start start 2024-09-16T03:48:28.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.155+0000 7fa630bbd640 1 librados: setting wanted keys 2024-09-16T03:48:28.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.155+0000 7fa630bbd640 1 librados: calling monclient init 2024-09-16T03:48:28.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.155+0000 7fa630bbd640 10 monclient: init 2024-09-16T03:48:28.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.155+0000 7fa630bbd640 10 monclient: _reopen_session rank -1 2024-09-16T03:48:28.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.155+0000 7fa630bbd640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:48:28.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.155+0000 7fa630bbd640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa618004810 0x7fa62c1cf420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:28.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.155+0000 7fa630bbd640 10 monclient(hunting): picked mon.noname-a con 0x7fa618004810 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:48:28.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.155+0000 7fa630bbd640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa62c1cf960 0x7fa62c11e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:28.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.155+0000 7fa630bbd640 10 monclient(hunting): picked mon.noname-c con 0x7fa62c1cf960 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:48:28.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.155+0000 7fa630bbd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa62c1cfd80 0x7fa62c11e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:28.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.155+0000 7fa630bbd640 10 monclient(hunting): picked mon.noname-b con 0x7fa62c1cfd80 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:48:28.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.155+0000 7fa630bbd640 10 monclient(hunting): start opening mon connection 2024-09-16T03:48:28.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.155+0000 7fa630bbd640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fa62c072fe0 con 0x7fa618004810 2024-09-16T03:48:28.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.155+0000 7fa630bbd640 10 monclient(hunting): start opening mon connection 2024-09-16T03:48:28.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.155+0000 7fa630bbd640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa62c072e60 con 0x7fa62c1cfd80 2024-09-16T03:48:28.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.155+0000 7fa630bbd640 10 monclient(hunting): start opening mon connection 2024-09-16T03:48:28.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.155+0000 7fa630bbd640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fa62c13a230 con 0x7fa62c1cf960 2024-09-16T03:48:28.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.155+0000 7fa630bbd640 10 monclient(hunting): _renew_subs 2024-09-16T03:48:28.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.155+0000 7fa630bbd640 10 monclient(hunting): authenticate will time out at 1355.125090s 2024-09-16T03:48:28.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.155+0000 7fa62a59c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa618004810 0x7fa62c1cf420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:48:28.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.155+0000 7fa629d9b640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa62c1cf960 0x7fa62c11e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:48:28.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.155+0000 7fa629d9b640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa62c1cf960 0x7fa62c11e070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:33496/0 (socket says 172.21.15.42:33496) 2024-09-16T03:48:28.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.155+0000 7fa629d9b640 1 -- 172.21.15.42:0/3027020599 learned_addr learned my addr 172.21.15.42:0/3027020599 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:48:28.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.155+0000 7fa62a59c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa618004810 0x7fa62c1cf420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:34572/0 (socket says 172.21.15.42:34572) 2024-09-16T03:48:28.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.155+0000 7fa629d9b640 10 monclient(hunting): get_auth_request con 0x7fa62c1cf960 auth_method 0 2024-09-16T03:48:28.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.155+0000 7fa629d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:48:28.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.155+0000 7fa629d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:48:28.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.155+0000 7fa629d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:48:28.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.155+0000 7fa62a59c640 10 monclient(hunting): get_auth_request con 0x7fa618004810 auth_method 0 2024-09-16T03:48:28.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.155+0000 7fa62a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:48:28.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.155+0000 7fa62a59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:48:28.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.155+0000 7fa62a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:48:28.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.155+0000 7fa629d9b640 1 -- 172.21.15.42:0/3027020599 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa62c1cf960 msgr2=0x7fa62c11e070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-16T03:48:28.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.155+0000 7fa629d9b640 1 -- 172.21.15.42:0/3027020599 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa62c1cf960 msgr2=0x7fa62c11e070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T03:48:28.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.155+0000 7fa629d9b640 1 --2- 172.21.15.42:0/3027020599 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa62c1cf960 0x7fa62c11e070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:48:28.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.155+0000 7fa62a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:48:28.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.155+0000 7fa62a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:48:28.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.155+0000 7fa62a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:48:28.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.155+0000 7fa629d9b640 1 --2- 172.21.15.42:0/3027020599 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa62c1cf960 0x7fa62c11e070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:48:28.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.155+0000 7fa62ad9d640 1 --2- 172.21.15.42:0/3027020599 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa62c1cfd80 0x7fa62c11e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:48:28.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.155+0000 7fa62ad9d640 10 monclient(hunting): get_auth_request con 0x7fa62c1cfd80 auth_method 0 2024-09-16T03:48:28.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.155+0000 7fa62ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:48:28.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.155+0000 7fa62ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:48:28.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.155+0000 7fa62ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:48:28.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.155+0000 7fa62a59c640 10 monclient(hunting): handle_auth_done global_id 764433 payload 824 2024-09-16T03:48:28.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.155+0000 7fa62a59c640 1 -- 172.21.15.42:0/3027020599 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa62c1cf960 msgr2=0x7fa62c11e070 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:48:28.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.155+0000 7fa62a59c640 1 --2- 172.21.15.42:0/3027020599 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa62c1cf960 0x7fa62c11e070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:28.180 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.155+0000 7fa62a59c640 1 -- 172.21.15.42:0/3027020599 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa62c1cfd80 msgr2=0x7fa62c11e5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:48:28.180 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.155+0000 7fa62a59c640 1 --2- 172.21.15.42:0/3027020599 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa62c1cfd80 0x7fa62c11e5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:28.180 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.155+0000 7fa62a59c640 10 monclient: _finish_hunting 0 2024-09-16T03:48:28.180 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.155+0000 7fa62a59c640 1 monclient: found mon.noname-a 2024-09-16T03:48:28.180 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.155+0000 7fa62a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:48:28.180 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.155+0000 7fa62a59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:48:28.180 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.155+0000 7fa62a59c640 1 -- 172.21.15.42:0/3027020599 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa62c121a00 con 0x7fa618004810 2024-09-16T03:48:28.180 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.155+0000 7fa62a59c640 1 --2- 172.21.15.42:0/3027020599 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa618004810 0x7fa62c1cf420 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fa62c09d020 tx=0x7fa61c00ec00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:48:28.180 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.155+0000 7fa622ffd640 10 client.?.objecter ms_handle_connect 0x7fa618004810 2024-09-16T03:48:28.180 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.155+0000 7fa622ffd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:48:28.180 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.155+0000 7fa622ffd640 1 -- 172.21.15.42:0/3027020599 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa61c017070 con 0x7fa618004810 2024-09-16T03:48:28.180 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.155+0000 7fa622ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:48:28.181 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.155+0000 7fa622ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:48:28.181 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.155+0000 7fa622ffd640 10 monclient: dump: 2024-09-16T03:48:28.181 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:48:28.181 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:48:28.181 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:28.181 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:28.181 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:48:28.181 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:48:28.181 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:48:28.181 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:48:28.181 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:48:28.181 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:48:28.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.155+0000 7fa622ffd640 10 monclient: _finish_auth 0 2024-09-16T03:48:28.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.155+0000 7fa622ffd640 10 monclient: _check_auth_tickets 2024-09-16T03:48:28.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.155+0000 7fa622ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:48:28.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.155+0000 7fa622ffd640 1 -- 172.21.15.42:0/3027020599 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa61c00c730 con 0x7fa618004810 2024-09-16T03:48:28.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.155+0000 7fa622ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:48:28.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.155+0000 7fa622ffd640 1 -- 172.21.15.42:0/3027020599 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa61c00cb00 con 0x7fa618004810 2024-09-16T03:48:28.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.155+0000 7fa622ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:48:28.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.155+0000 7fa622ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:48:28.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.155+0000 7fa622ffd640 10 monclient: dump: 2024-09-16T03:48:28.182 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:48:28.182 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:48:28.182 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:28.183 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:28.183 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:48:28.183 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:48:28.183 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:48:28.183 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:48:28.183 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:48:28.183 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:48:28.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.155+0000 7fa630bbd640 5 monclient: authenticate success, global_id 764433 2024-09-16T03:48:28.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.155+0000 7fa630bbd640 10 monclient: _renew_subs 2024-09-16T03:48:28.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.155+0000 7fa628d99640 4 set_mon_vals no callback set 2024-09-16T03:48:28.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.155+0000 7fa630bbd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:48:28.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.155+0000 7fa630bbd640 1 -- 172.21.15.42:0/3027020599 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa62c121ce0 con 0x7fa618004810 2024-09-16T03:48:28.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.155+0000 7fa630bbd640 10 client.764433.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:48:28.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.155+0000 7fa630bbd640 10 monclient: _renew_subs 2024-09-16T03:48:28.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.155+0000 7fa630bbd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:48:28.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.155+0000 7fa630bbd640 1 -- 172.21.15.42:0/3027020599 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa62c1328f0 con 0x7fa618004810 2024-09-16T03:48:28.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.155+0000 7fa630bbd640 1 librados: init done 2024-09-16T03:48:28.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.155+0000 7fa630bbd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:48:28.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.155+0000 7fa630bbd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:48:28.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.155+0000 7fa630bbd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:48:28.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.155+0000 7fa622ffd640 1 -- 172.21.15.42:0/3027020599 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fa61c01a070 con 0x7fa618004810 2024-09-16T03:48:28.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.155+0000 7fa630bbd640 1 -- 172.21.15.42:0/3027020599 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa62c0731f0 con 0x7fa618004810 2024-09-16T03:48:28.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.155+0000 7fa622ffd640 1 --2- 172.21.15.42:0/3027020599 >> v2:172.21.15.154:6812/254845342 conn(0x7fa60c037810 0x7fa60c039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:28.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.155+0000 7fa622ffd640 1 -- 172.21.15.42:0/3027020599 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(203..203 src has 1..203) ==== 6623+0+0 (secure 0 0 0) 0x7fa61c048ca0 con 0x7fa618004810 2024-09-16T03:48:28.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.155+0000 7fa622ffd640 10 client.764433.objecter ms_dispatch 0x7fa62c000ba0 osd_map(203..203 src has 1..203) 2024-09-16T03:48:28.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.155+0000 7fa622ffd640 3 client.764433.objecter handle_osd_map got epochs [203,203] > 0 2024-09-16T03:48:28.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.155+0000 7fa622ffd640 3 client.764433.objecter handle_osd_map decoding full epoch 203 2024-09-16T03:48:28.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.155+0000 7fa622ffd640 20 client.764433.objecter dump_active .. 0 homeless 2024-09-16T03:48:28.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.155+0000 7fa629d9b640 1 --2- 172.21.15.42:0/3027020599 >> v2:172.21.15.154:6812/254845342 conn(0x7fa60c037810 0x7fa60c039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:48:28.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.155+0000 7fa629d9b640 10 monclient: get_auth_request con 0x7fa60c037810 auth_method 0 2024-09-16T03:48:28.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.159+0000 7fa622ffd640 10 client.764433.objecter ms_handle_connect 0x7fa60c037810 2024-09-16T03:48:28.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.159+0000 7fa629d9b640 1 --2- 172.21.15.42:0/3027020599 >> v2:172.21.15.154:6812/254845342 conn(0x7fa60c037810 0x7fa60c039cd0 secure :-1 s=READY pgs=3481 cs=0 l=1 rev1=1 crypto rx=0x7fa61400c190 tx=0x7fa62c0a2790 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:48:28.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.163+0000 7fa622ffd640 1 -- 172.21.15.42:0/3027020599 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7fa61c012df0 con 0x7fa618004810 2024-09-16T03:48:28.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.163+0000 7fa622ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:48:28.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.163+0000 7fa622ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:48:28.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.395+0000 7fa630bbd640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:48:28.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.395+0000 7fa630bbd640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:48:28.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.395+0000 7fa630bbd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:48:28.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.395+0000 7fa630bbd640 1 -- 172.21.15.42:0/3027020599 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fa62c06b3e0 con 0x7fa618004810 2024-09-16T03:48:28.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.395+0000 7fa622ffd640 1 -- 172.21.15.42:0/3027020599 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1555 (secure 0 0 0) 0x7fa61c011df0 con 0x7fa618004810 2024-09-16T03:48:28.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.395+0000 7fa622ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:48:28.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.395+0000 7fa622ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:48:28.399 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":58,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":21,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:48:28.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.395+0000 7fa630bbd640 10 librados: watch_flush enter 2024-09-16T03:48:28.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.395+0000 7fa630bbd640 10 librados: watch_flush exit 2024-09-16T03:48:28.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.395+0000 7fa630bbd640 20 client.764433.objecter shutdown clearing up homeless session... 2024-09-16T03:48:28.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.395+0000 7fa630bbd640 10 client.764433.objecter successfully canceled tick 2024-09-16T03:48:28.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.395+0000 7fa630bbd640 1 -- 172.21.15.42:0/3027020599 >> v2:172.21.15.154:6812/254845342 conn(0x7fa60c037810 msgr2=0x7fa60c039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:48:28.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.395+0000 7fa630bbd640 1 --2- 172.21.15.42:0/3027020599 >> v2:172.21.15.154:6812/254845342 conn(0x7fa60c037810 0x7fa60c039cd0 secure :-1 s=READY pgs=3481 cs=0 l=1 rev1=1 crypto rx=0x7fa61400c190 tx=0x7fa62c0a2790 comp rx=0 tx=0).stop 2024-09-16T03:48:28.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.395+0000 7fa630bbd640 10 monclient: shutdown 2024-09-16T03:48:28.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.395+0000 7fa630bbd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:48:28.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.395+0000 7fa630bbd640 1 -- 172.21.15.42:0/3027020599 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa618004810 msgr2=0x7fa62c1cf420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:48:28.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.395+0000 7fa630bbd640 1 --2- 172.21.15.42:0/3027020599 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa618004810 0x7fa62c1cf420 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fa62c09d020 tx=0x7fa61c00ec00 comp rx=0 tx=0).stop 2024-09-16T03:48:28.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.395+0000 7fa630bbd640 1 -- 172.21.15.42:0/3027020599 shutdown_connections 2024-09-16T03:48:28.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.395+0000 7fa630bbd640 1 --2- 172.21.15.42:0/3027020599 >> v2:172.21.15.154:6812/254845342 conn(0x7fa60c037810 0x7fa60c039cd0 unknown :-1 s=CLOSED pgs=3481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:28.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.395+0000 7fa630bbd640 1 --2- 172.21.15.42:0/3027020599 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa62c1cfd80 0x7fa62c11e5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:28.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.395+0000 7fa630bbd640 1 --2- 172.21.15.42:0/3027020599 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa62c1cf960 0x7fa62c11e070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:28.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.395+0000 7fa630bbd640 1 --2- 172.21.15.42:0/3027020599 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa618004810 0x7fa62c1cf420 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:28.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.395+0000 7fa630bbd640 1 -- 172.21.15.42:0/3027020599 >> 172.21.15.42:0/3027020599 conn(0x7fa62c06bc40 msgr2=0x7fa62c09fdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:48:28.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.399+0000 7fa630bbd640 1 -- 172.21.15.42:0/3027020599 shutdown_connections 2024-09-16T03:48:28.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.399+0000 7fa630bbd640 1 -- 172.21.15.42:0/3027020599 wait complete. 2024-09-16T03:48:28.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:28.399+0000 7fa630bbd640 1 librados: shutdown 2024-09-16T03:48:31.418 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:48:31.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.551+0000 7fd7b0b91640 10 monclient: get_monmap_and_config 2024-09-16T03:48:31.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.551+0000 7fd7b0b91640 10 monclient: build_initial_monmap 2024-09-16T03:48:31.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.551+0000 7fd7b0b91640 10 monclient: monmap: 2024-09-16T03:48:31.554 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:48:31.554 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:48:31.554 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:48:31.553227+0000 2024-09-16T03:48:31.555 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:48:31.553227+0000 2024-09-16T03:48:31.555 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:48:31.555 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:48:31.555 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:48:31.555 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:48:31.555 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:48:31.555 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:48:31.555 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:48:31.555 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:48:31.557 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.551+0000 7fd7b0b91640 1 Processor -- start 2024-09-16T03:48:31.557 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.551+0000 7fd7b0b91640 1 -- start start 2024-09-16T03:48:31.557 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.551+0000 7fd7b0b91640 10 monclient: init 2024-09-16T03:48:31.557 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.551+0000 7fd7b0b91640 10 monclient: _reopen_session rank -1 2024-09-16T03:48:31.557 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.551+0000 7fd7b0b91640 10 monclient: _add_conns ranks=[3,1,4,0,2] 2024-09-16T03:48:31.557 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.551+0000 7fd7b0b91640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd7ac0a1330 0x7fd7ac0a1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:31.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.551+0000 7fd7b0b91640 10 monclient(hunting): picked mon.noname-a con 0x7fd7ac0a1330 addr v2:172.21.15.42:3300/0 2024-09-16T03:48:31.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.551+0000 7fd7b0b91640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd7ac06dac0 addr v1:172.21.15.154:6789/0 2024-09-16T03:48:31.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.551+0000 7fd7b0b91640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fd7ac0a1c50 0x7fd7ac135810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:31.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.551+0000 7fd7b0b91640 10 monclient(hunting): picked mon.noname-c con 0x7fd7ac0a1c50 addr v2:172.21.15.154:3300/0 2024-09-16T03:48:31.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.551+0000 7fd7b0b91640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd7ac09c090 con 0x7fd7ac06dac0 2024-09-16T03:48:31.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.551+0000 7fd7b0b91640 10 monclient(hunting): start opening mon connection 2024-09-16T03:48:31.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.551+0000 7fd7b0b91640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7fd7ac09f610 con 0x7fd7ac0a1330 2024-09-16T03:48:31.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.551+0000 7fd7b0b91640 10 monclient(hunting): start opening mon connection 2024-09-16T03:48:31.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.551+0000 7fd7b0b91640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fd7ac09f8f0 con 0x7fd7ac0a1c50 2024-09-16T03:48:31.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.551+0000 7fd7b0b91640 10 monclient(hunting): _renew_subs 2024-09-16T03:48:31.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.551+0000 7fd7b0b91640 10 monclient(hunting): authenticate will time out at 1358.522654s 2024-09-16T03:48:31.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.551+0000 7fd7aa59c640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd7ac0a1330 0x7fd7ac0a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:48:31.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.551+0000 7fd7aa59c640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd7ac0a1330 0x7fd7ac0a1710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:34614/0 (socket says 172.21.15.42:34614) 2024-09-16T03:48:31.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.551+0000 7fd7aa59c640 1 -- 172.21.15.42:0/3957495170 learned_addr learned my addr 172.21.15.42:0/3957495170 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:48:31.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.551+0000 7fd7aa59c640 10 monclient(hunting): get_auth_request con 0x7fd7ac0a1330 auth_method 0 2024-09-16T03:48:31.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.551+0000 7fd7aa59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:48:31.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.551+0000 7fd7aa59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:48:31.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.551+0000 7fd7aa59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:48:31.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.551+0000 7fd7aa59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:48:31.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.551+0000 7fd7aa59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:48:31.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.551+0000 7fd7aa59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:48:31.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.551+0000 7fd7aad9d640 1 --2- 172.21.15.42:0/3957495170 >> v2:172.21.15.154:3300/0 conn(0x7fd7ac0a1c50 0x7fd7ac135810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:48:31.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.551+0000 7fd7aad9d640 10 monclient(hunting): get_auth_request con 0x7fd7ac0a1c50 auth_method 0 2024-09-16T03:48:31.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.551+0000 7fd7aad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:48:31.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.551+0000 7fd7aad9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:48:31.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.551+0000 7fd7aad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:48:31.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.551+0000 7fd7aa59c640 10 monclient(hunting): handle_auth_done global_id 764448 payload 274 2024-09-16T03:48:31.560 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.551+0000 7fd7aa59c640 1 -- 172.21.15.42:0/3957495170 >> v2:172.21.15.154:3300/0 conn(0x7fd7ac0a1c50 msgr2=0x7fd7ac135810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:48:31.560 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.551+0000 7fd7aa59c640 1 --2- 172.21.15.42:0/3957495170 >> v2:172.21.15.154:3300/0 conn(0x7fd7ac0a1c50 0x7fd7ac135810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:31.560 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.551+0000 7fd7aa59c640 1 -- 172.21.15.42:0/3957495170 >> v1:172.21.15.154:6789/0 conn(0x7fd7ac06dac0 legacy=0x7fd7ac135100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:48:31.560 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.551+0000 7fd7aa59c640 10 monclient: _finish_hunting 0 2024-09-16T03:48:31.560 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.551+0000 7fd7aa59c640 1 monclient: found mon.noname-a 2024-09-16T03:48:31.560 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.551+0000 7fd7aa59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:48:31.560 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.551+0000 7fd7aa59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:48:31.560 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.551+0000 7fd7aa59c640 1 -- 172.21.15.42:0/3957495170 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7ac09fbd0 con 0x7fd7ac0a1330 2024-09-16T03:48:31.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.555+0000 7fd7aa59c640 1 --2- 172.21.15.42:0/3957495170 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd7ac0a1330 0x7fd7ac0a1710 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fd7ac09c800 tx=0x7fd79c01c870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72e42d0ef905716e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:48:31.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.555+0000 7fd7a959a640 1 -- 172.21.15.42:0/3957495170 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd79c0a5750 con 0x7fd7ac0a1330 2024-09-16T03:48:31.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.555+0000 7fd7a959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:48:31.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.555+0000 7fd7a959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:48:31.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.555+0000 7fd7a959a640 10 monclient: dump: 2024-09-16T03:48:31.567 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:48:31.567 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:48:31.567 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:31.567 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:31.567 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:48:31.567 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:48:31.567 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:48:31.567 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:48:31.567 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:48:31.567 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:48:31.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.555+0000 7fd7a959a640 10 monclient: _finish_auth 0 2024-09-16T03:48:31.568 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.555+0000 7fd7a959a640 10 monclient: _check_auth_tickets 2024-09-16T03:48:31.568 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.555+0000 7fd7a959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:48:31.568 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.555+0000 7fd7a959a640 1 -- 172.21.15.42:0/3957495170 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd79c0a58f0 con 0x7fd7ac0a1330 2024-09-16T03:48:31.568 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.555+0000 7fd7a959a640 10 monclient: handle_config config(0 keys) 2024-09-16T03:48:31.568 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.555+0000 7fd7a959a640 1 -- 172.21.15.42:0/3957495170 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd79c0a5cc0 con 0x7fd7ac0a1330 2024-09-16T03:48:31.568 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.555+0000 7fd7a959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:48:31.568 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.555+0000 7fd7a959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:48:31.568 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.555+0000 7fd7a959a640 10 monclient: dump: 2024-09-16T03:48:31.568 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:48:31.568 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:48:31.568 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:31.568 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:31.568 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:48:31.569 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:48:31.569 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:48:31.569 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:48:31.569 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:48:31.569 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:48:31.569 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.555+0000 7fd7b0b91640 5 monclient: authenticate success, global_id 764448 2024-09-16T03:48:31.569 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.555+0000 7fd7b0b91640 10 monclient: get_monmap_and_config success 2024-09-16T03:48:31.569 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.555+0000 7fd7b0b91640 4 set_mon_vals no callback set 2024-09-16T03:48:31.569 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.555+0000 7fd7b0b91640 10 monclient: shutdown 2024-09-16T03:48:31.569 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.555+0000 7fd7b0b91640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:48:31.569 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.555+0000 7fd7b0b91640 1 -- 172.21.15.42:0/3957495170 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd7ac0a1330 msgr2=0x7fd7ac0a1710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:48:31.569 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.555+0000 7fd7b0b91640 1 --2- 172.21.15.42:0/3957495170 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd7ac0a1330 0x7fd7ac0a1710 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fd7ac09c800 tx=0x7fd79c01c870 comp rx=0 tx=0).stop 2024-09-16T03:48:31.569 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.555+0000 7fd7b0b91640 1 -- 172.21.15.42:0/3957495170 shutdown_connections 2024-09-16T03:48:31.570 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.555+0000 7fd7b0b91640 1 --2- 172.21.15.42:0/3957495170 >> v2:172.21.15.154:3300/0 conn(0x7fd7ac0a1c50 0x7fd7ac135810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:31.570 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.555+0000 7fd7b0b91640 1 --2- 172.21.15.42:0/3957495170 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd7ac0a1330 0x7fd7ac0a1710 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:31.570 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.555+0000 7fd7b0b91640 1 -- 172.21.15.42:0/3957495170 >> 172.21.15.42:0/3957495170 conn(0x7fd7ac0a2420 msgr2=0x7fd7ac0a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:48:31.570 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.555+0000 7fd7b0b91640 1 -- 172.21.15.42:0/3957495170 shutdown_connections 2024-09-16T03:48:31.570 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.555+0000 7fd7b0b91640 1 -- 172.21.15.42:0/3957495170 wait complete. 2024-09-16T03:48:31.570 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.555+0000 7fd7b0b91640 10 monclient: build_initial_monmap 2024-09-16T03:48:31.570 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.555+0000 7fd7b0b91640 10 monclient: monmap: 2024-09-16T03:48:31.570 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:48:31.570 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:48:31.570 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:48:31.570 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:48:31.570 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:48:31.570 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:48:31.571 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:48:31.571 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:48:31.571 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:48:31.571 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:48:31.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.555+0000 7fd7b0b91640 1 librados: starting msgr at 2024-09-16T03:48:31.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.555+0000 7fd7b0b91640 1 librados: starting objecter 2024-09-16T03:48:31.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.555+0000 7fd7b0b91640 1 Processor -- start 2024-09-16T03:48:31.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.555+0000 7fd7b0b91640 1 -- start start 2024-09-16T03:48:31.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.555+0000 7fd7b0b91640 1 librados: setting wanted keys 2024-09-16T03:48:31.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.555+0000 7fd7b0b91640 1 librados: calling monclient init 2024-09-16T03:48:31.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.555+0000 7fd7b0b91640 10 monclient: init 2024-09-16T03:48:31.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.555+0000 7fd7b0b91640 10 monclient: _reopen_session rank -1 2024-09-16T03:48:31.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.555+0000 7fd7b0b91640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:48:31.572 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.555+0000 7fd7b0b91640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd7ac0a1330 0x7fd7ac1ccba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:31.572 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.555+0000 7fd7b0b91640 10 monclient(hunting): picked mon.noname-a con 0x7fd7ac0a1330 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:48:31.572 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.555+0000 7fd7b0b91640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd7ac0a1c50 0x7fd7ac134f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:31.572 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.555+0000 7fd7b0b91640 10 monclient(hunting): picked mon.noname-c con 0x7fd7ac0a1c50 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:48:31.572 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.555+0000 7fd7b0b91640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd7ac135470 0x7fd7ac11bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:31.572 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.559+0000 7fd7b0b91640 10 monclient(hunting): picked mon.noname-b con 0x7fd7ac135470 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:48:31.572 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.559+0000 7fd7b0b91640 10 monclient(hunting): start opening mon connection 2024-09-16T03:48:31.572 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.559+0000 7fd7b0b91640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fd7ac135d50 con 0x7fd7ac0a1330 2024-09-16T03:48:31.572 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.559+0000 7fd7b0b91640 10 monclient(hunting): start opening mon connection 2024-09-16T03:48:31.572 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.559+0000 7fd7b0b91640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd7ac09ef80 con 0x7fd7ac135470 2024-09-16T03:48:31.572 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.559+0000 7fd7b0b91640 10 monclient(hunting): start opening mon connection 2024-09-16T03:48:31.572 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.559+0000 7fd7b0b91640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fd7ac135ed0 con 0x7fd7ac0a1c50 2024-09-16T03:48:31.572 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.559+0000 7fd7b0b91640 10 monclient(hunting): _renew_subs 2024-09-16T03:48:31.573 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.559+0000 7fd7b0b91640 10 monclient(hunting): authenticate will time out at 1358.528037s 2024-09-16T03:48:31.573 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.559+0000 7fd7aa59c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd7ac0a1330 0x7fd7ac1ccba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:48:31.573 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.559+0000 7fd7aa59c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd7ac0a1330 0x7fd7ac1ccba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:34618/0 (socket says 172.21.15.42:34618) 2024-09-16T03:48:31.573 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.559+0000 7fd7aa59c640 1 -- 172.21.15.42:0/1458265870 learned_addr learned my addr 172.21.15.42:0/1458265870 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:48:31.573 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.559+0000 7fd7aa59c640 10 monclient(hunting): get_auth_request con 0x7fd7ac0a1330 auth_method 0 2024-09-16T03:48:31.573 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.559+0000 7fd7aa59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:48:31.573 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.559+0000 7fd7aa59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:48:31.573 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.559+0000 7fd7aa59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:48:31.573 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.559+0000 7fd7a9d9b640 1 --2- 172.21.15.42:0/1458265870 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd7ac0a1c50 0x7fd7ac134f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:48:31.573 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.559+0000 7fd7a9d9b640 10 monclient(hunting): get_auth_request con 0x7fd7ac0a1c50 auth_method 0 2024-09-16T03:48:31.573 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.559+0000 7fd7a9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:48:31.573 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.559+0000 7fd7a9d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:48:31.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.559+0000 7fd7a9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:48:31.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.559+0000 7fd7aa59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:48:31.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.559+0000 7fd7aa59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:48:31.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.559+0000 7fd7aa59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:48:31.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.559+0000 7fd7a9d9b640 1 -- 172.21.15.42:0/1458265870 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd7ac0a1c50 msgr2=0x7fd7ac134f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-16T03:48:31.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.559+0000 7fd7a9d9b640 1 -- 172.21.15.42:0/1458265870 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd7ac0a1c50 msgr2=0x7fd7ac134f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T03:48:31.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.559+0000 7fd7a9d9b640 1 --2- 172.21.15.42:0/1458265870 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd7ac0a1c50 0x7fd7ac134f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:48:31.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.559+0000 7fd7aad9d640 1 --2- 172.21.15.42:0/1458265870 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd7ac135470 0x7fd7ac11bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:48:31.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.559+0000 7fd7a9d9b640 1 --2- 172.21.15.42:0/1458265870 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd7ac0a1c50 0x7fd7ac134f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:48:31.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.559+0000 7fd7aad9d640 10 monclient(hunting): get_auth_request con 0x7fd7ac135470 auth_method 0 2024-09-16T03:48:31.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.559+0000 7fd7aad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:48:31.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.559+0000 7fd7aad9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:48:31.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.559+0000 7fd7aad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:48:31.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.559+0000 7fd7aa59c640 10 monclient(hunting): handle_auth_done global_id 764451 payload 824 2024-09-16T03:48:31.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.559+0000 7fd7aa59c640 1 -- 172.21.15.42:0/1458265870 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd7ac0a1c50 msgr2=0x7fd7ac134f30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:48:31.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.559+0000 7fd7aa59c640 1 --2- 172.21.15.42:0/1458265870 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd7ac0a1c50 0x7fd7ac134f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:31.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.559+0000 7fd7aa59c640 1 -- 172.21.15.42:0/1458265870 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd7ac135470 msgr2=0x7fd7ac11bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:48:31.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.559+0000 7fd7aa59c640 1 --2- 172.21.15.42:0/1458265870 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd7ac135470 0x7fd7ac11bf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:31.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.559+0000 7fd7aa59c640 10 monclient: _finish_hunting 0 2024-09-16T03:48:31.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.559+0000 7fd7aa59c640 1 monclient: found mon.noname-a 2024-09-16T03:48:31.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.559+0000 7fd7aa59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:48:31.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.559+0000 7fd7aa59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:48:31.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.559+0000 7fd7aa59c640 1 -- 172.21.15.42:0/1458265870 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7ac11f600 con 0x7fd7ac0a1330 2024-09-16T03:48:31.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.559+0000 7fd7aa59c640 1 --2- 172.21.15.42:0/1458265870 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd7ac0a1330 0x7fd7ac1ccba0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fd79c0af040 tx=0x7fd79c019610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:48:31.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.559+0000 7fd79affd640 10 client.?.objecter ms_handle_connect 0x7fd7ac0a1330 2024-09-16T03:48:31.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.559+0000 7fd79affd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:48:31.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.559+0000 7fd79affd640 1 -- 172.21.15.42:0/1458265870 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd79c0bc020 con 0x7fd7ac0a1330 2024-09-16T03:48:31.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.559+0000 7fd79affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:48:31.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.559+0000 7fd79affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:48:31.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.559+0000 7fd79affd640 10 monclient: dump: 2024-09-16T03:48:31.576 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:48:31.576 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:48:31.576 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:31.576 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:31.576 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:48:31.576 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:48:31.576 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:48:31.576 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:48:31.576 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:48:31.576 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:48:31.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.559+0000 7fd79affd640 10 monclient: _finish_auth 0 2024-09-16T03:48:31.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.559+0000 7fd79affd640 10 monclient: _check_auth_tickets 2024-09-16T03:48:31.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.559+0000 7fd79affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:48:31.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.559+0000 7fd7b0b91640 5 monclient: authenticate success, global_id 764451 2024-09-16T03:48:31.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.559+0000 7fd79affd640 1 -- 172.21.15.42:0/1458265870 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd79c0b86b0 con 0x7fd7ac0a1330 2024-09-16T03:48:31.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.559+0000 7fd7b0b91640 10 monclient: _renew_subs 2024-09-16T03:48:31.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.559+0000 7fd7b0b91640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:48:31.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.559+0000 7fd7b0b91640 1 -- 172.21.15.42:0/1458265870 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7ac11f8e0 con 0x7fd7ac0a1330 2024-09-16T03:48:31.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.559+0000 7fd79affd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:48:31.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.559+0000 7fd79affd640 1 -- 172.21.15.42:0/1458265870 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd79c0b8aa0 con 0x7fd7ac0a1330 2024-09-16T03:48:31.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.559+0000 7fd79affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:48:31.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.559+0000 7fd79affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:48:31.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.559+0000 7fd79affd640 10 monclient: dump: 2024-09-16T03:48:31.577 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:48:31.578 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:48:31.578 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:31.578 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:31.578 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:48:31.578 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:48:31.578 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:48:31.578 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:48:31.578 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:48:31.578 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:48:31.578 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.559+0000 7fd79bfff640 4 set_mon_vals no callback set 2024-09-16T03:48:31.578 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.559+0000 7fd7b0b91640 10 client.764451.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:48:31.578 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.559+0000 7fd7b0b91640 10 monclient: _renew_subs 2024-09-16T03:48:31.578 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.559+0000 7fd7b0b91640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:48:31.578 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.559+0000 7fd7b0b91640 1 -- 172.21.15.42:0/1458265870 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7ac073a30 con 0x7fd7ac0a1330 2024-09-16T03:48:31.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.559+0000 7fd7b0b91640 1 librados: init done 2024-09-16T03:48:31.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.559+0000 7fd79affd640 1 -- 172.21.15.42:0/1458265870 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fd79c0bf020 con 0x7fd7ac0a1330 2024-09-16T03:48:31.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.559+0000 7fd7b0b91640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:48:31.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.559+0000 7fd7b0b91640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:48:31.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.559+0000 7fd7b0b91640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:48:31.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.559+0000 7fd79affd640 1 --2- 172.21.15.42:0/1458265870 >> v2:172.21.15.154:6812/254845342 conn(0x7fd78c037840 0x7fd78c039d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:31.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.559+0000 7fd7b0b91640 1 -- 172.21.15.42:0/1458265870 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd7ac11d5f0 con 0x7fd7ac0a1330 2024-09-16T03:48:31.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.559+0000 7fd79affd640 1 -- 172.21.15.42:0/1458265870 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(207..207 src has 1..207) ==== 6060+0+0 (secure 0 0 0) 0x7fd79c0e4830 con 0x7fd7ac0a1330 2024-09-16T03:48:31.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.559+0000 7fd79affd640 10 client.764451.objecter ms_dispatch 0x7fd7ac000ba0 osd_map(207..207 src has 1..207) 2024-09-16T03:48:31.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.559+0000 7fd79affd640 3 client.764451.objecter handle_osd_map got epochs [207,207] > 0 2024-09-16T03:48:31.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.559+0000 7fd79affd640 3 client.764451.objecter handle_osd_map decoding full epoch 207 2024-09-16T03:48:31.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.559+0000 7fd79affd640 20 client.764451.objecter dump_active .. 0 homeless 2024-09-16T03:48:31.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.559+0000 7fd7a9d9b640 1 --2- 172.21.15.42:0/1458265870 >> v2:172.21.15.154:6812/254845342 conn(0x7fd78c037840 0x7fd78c039d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:48:31.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.559+0000 7fd7a9d9b640 10 monclient: get_auth_request con 0x7fd78c037840 auth_method 0 2024-09-16T03:48:31.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.559+0000 7fd7a9d9b640 1 --2- 172.21.15.42:0/1458265870 >> v2:172.21.15.154:6812/254845342 conn(0x7fd78c037840 0x7fd78c039d00 secure :-1 s=READY pgs=3485 cs=0 l=1 rev1=1 crypto rx=0x7fd7a0007fd0 tx=0x7fd7a0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:48:31.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.559+0000 7fd79affd640 10 client.764451.objecter ms_handle_connect 0x7fd78c037840 2024-09-16T03:48:31.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.563+0000 7fd79affd640 1 -- 172.21.15.42:0/1458265870 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7fd79c016580 con 0x7fd7ac0a1330 2024-09-16T03:48:31.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.563+0000 7fd79affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:48:31.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.563+0000 7fd79affd640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:48:31.798 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.795+0000 7fd7b0b91640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:48:31.798 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.795+0000 7fd7b0b91640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:48:31.799 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.795+0000 7fd7b0b91640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:48:31.799 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.795+0000 7fd7b0b91640 1 -- 172.21.15.42:0/1458265870 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fd7ac0a1710 con 0x7fd7ac0a1330 2024-09-16T03:48:31.799 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.795+0000 7fd79affd640 1 -- 172.21.15.42:0/1458265870 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1555 (secure 0 0 0) 0x7fd79c0b7cd0 con 0x7fd7ac0a1330 2024-09-16T03:48:31.799 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.795+0000 7fd79affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:48:31.799 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.795+0000 7fd79affd640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:48:31.799 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":58,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":24,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:48:31.802 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.799+0000 7fd7b0b91640 10 librados: watch_flush enter 2024-09-16T03:48:31.802 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.799+0000 7fd7b0b91640 10 librados: watch_flush exit 2024-09-16T03:48:31.802 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.799+0000 7fd7b0b91640 20 client.764451.objecter shutdown clearing up homeless session... 2024-09-16T03:48:31.802 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.799+0000 7fd7b0b91640 10 client.764451.objecter successfully canceled tick 2024-09-16T03:48:31.802 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.799+0000 7fd7b0b91640 1 -- 172.21.15.42:0/1458265870 >> v2:172.21.15.154:6812/254845342 conn(0x7fd78c037840 msgr2=0x7fd78c039d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:48:31.802 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.799+0000 7fd7b0b91640 1 --2- 172.21.15.42:0/1458265870 >> v2:172.21.15.154:6812/254845342 conn(0x7fd78c037840 0x7fd78c039d00 secure :-1 s=READY pgs=3485 cs=0 l=1 rev1=1 crypto rx=0x7fd7a0007fd0 tx=0x7fd7a0009040 comp rx=0 tx=0).stop 2024-09-16T03:48:31.802 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.799+0000 7fd7b0b91640 10 monclient: shutdown 2024-09-16T03:48:31.802 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.799+0000 7fd7b0b91640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:48:31.802 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.799+0000 7fd7b0b91640 1 -- 172.21.15.42:0/1458265870 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd7ac0a1330 msgr2=0x7fd7ac1ccba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:48:31.802 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.799+0000 7fd7b0b91640 1 --2- 172.21.15.42:0/1458265870 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd7ac0a1330 0x7fd7ac1ccba0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fd79c0af040 tx=0x7fd79c019610 comp rx=0 tx=0).stop 2024-09-16T03:48:31.802 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.799+0000 7fd7b0b91640 1 -- 172.21.15.42:0/1458265870 shutdown_connections 2024-09-16T03:48:31.802 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.799+0000 7fd7b0b91640 1 --2- 172.21.15.42:0/1458265870 >> v2:172.21.15.154:6812/254845342 conn(0x7fd78c037840 0x7fd78c039d00 unknown :-1 s=CLOSED pgs=3485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:31.803 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.799+0000 7fd7b0b91640 1 --2- 172.21.15.42:0/1458265870 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd7ac135470 0x7fd7ac11bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:31.803 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.799+0000 7fd7b0b91640 1 --2- 172.21.15.42:0/1458265870 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd7ac0a1c50 0x7fd7ac134f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:31.803 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.799+0000 7fd7b0b91640 1 --2- 172.21.15.42:0/1458265870 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd7ac0a1330 0x7fd7ac1ccba0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:31.803 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.799+0000 7fd7b0b91640 1 -- 172.21.15.42:0/1458265870 >> 172.21.15.42:0/1458265870 conn(0x7fd7ac0a2420 msgr2=0x7fd7ac06f920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:48:31.803 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.799+0000 7fd7b0b91640 1 -- 172.21.15.42:0/1458265870 shutdown_connections 2024-09-16T03:48:31.803 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.799+0000 7fd7b0b91640 1 -- 172.21.15.42:0/1458265870 wait complete. 2024-09-16T03:48:31.803 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:31.799+0000 7fd7b0b91640 1 librados: shutdown 2024-09-16T03:48:34.819 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:48:34.954 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.951+0000 7fd0f7615640 10 monclient: get_monmap_and_config 2024-09-16T03:48:34.954 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.951+0000 7fd0f7615640 10 monclient: build_initial_monmap 2024-09-16T03:48:34.954 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.951+0000 7fd0f7615640 10 monclient: monmap: 2024-09-16T03:48:34.954 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:48:34.954 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:48:34.954 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:48:34.953280+0000 2024-09-16T03:48:34.954 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:48:34.953280+0000 2024-09-16T03:48:34.955 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:48:34.955 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:48:34.955 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:48:34.955 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:48:34.955 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:48:34.955 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:48:34.955 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:48:34.955 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:48:34.958 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.951+0000 7fd0f7615640 1 Processor -- start 2024-09-16T03:48:34.958 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.951+0000 7fd0f7615640 1 -- start start 2024-09-16T03:48:34.958 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.951+0000 7fd0f7615640 10 monclient: init 2024-09-16T03:48:34.958 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.951+0000 7fd0f7615640 10 monclient: _reopen_session rank -1 2024-09-16T03:48:34.958 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.951+0000 7fd0f7615640 10 monclient: _add_conns ranks=[3,4,2,1,0] 2024-09-16T03:48:34.958 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.951+0000 7fd0f7615640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd0f006db90 0x7fd0f01340c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:34.959 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.951+0000 7fd0f7615640 10 monclient(hunting): picked mon.noname-a con 0x7fd0f006db90 addr v2:172.21.15.42:3300/0 2024-09-16T03:48:34.959 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.951+0000 7fd0f7615640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fd0f00a37f0 0x7fd0f00a3c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:34.959 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.951+0000 7fd0f7615640 10 monclient(hunting): picked mon.noname-c con 0x7fd0f00a37f0 addr v2:172.21.15.154:3300/0 2024-09-16T03:48:34.959 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.951+0000 7fd0f7615640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd0f0136e10 0x7fd0f0137220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:34.959 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.951+0000 7fd0f7615640 10 monclient(hunting): picked mon.noname-b con 0x7fd0f0136e10 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:48:34.959 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.951+0000 7fd0f7615640 10 monclient(hunting): start opening mon connection 2024-09-16T03:48:34.959 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.951+0000 7fd0f7615640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7fd0f006df70 con 0x7fd0f006db90 2024-09-16T03:48:34.959 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.951+0000 7fd0f7615640 10 monclient(hunting): start opening mon connection 2024-09-16T03:48:34.959 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.951+0000 7fd0f7615640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fd0f006ece0 con 0x7fd0f00a37f0 2024-09-16T03:48:34.959 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.951+0000 7fd0f7615640 10 monclient(hunting): start opening mon connection 2024-09-16T03:48:34.959 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.951+0000 7fd0f7615640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fd0f009bf80 con 0x7fd0f0136e10 2024-09-16T03:48:34.959 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.951+0000 7fd0f7615640 10 monclient(hunting): _renew_subs 2024-09-16T03:48:34.959 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.951+0000 7fd0f7615640 10 monclient(hunting): authenticate will time out at 1361.922984s 2024-09-16T03:48:34.959 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.951+0000 7fd0f53b1640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd0f006db90 0x7fd0f01340c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:48:34.960 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.951+0000 7fd0f5bb2640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd0f0136e10 0x7fd0f0137220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:48:34.960 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.951+0000 7fd0f53b1640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd0f006db90 0x7fd0f01340c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:44650/0 (socket says 172.21.15.42:44650) 2024-09-16T03:48:34.960 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.951+0000 7fd0f53b1640 1 -- 172.21.15.42:0/767064951 learned_addr learned my addr 172.21.15.42:0/767064951 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:48:34.960 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.951+0000 7fd0f53b1640 10 monclient(hunting): get_auth_request con 0x7fd0f006db90 auth_method 0 2024-09-16T03:48:34.960 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.951+0000 7fd0f53b1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:48:34.960 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.951+0000 7fd0f53b1640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:48:34.960 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.951+0000 7fd0f53b1640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:48:34.960 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.951+0000 7fd0f5bb2640 10 monclient(hunting): get_auth_request con 0x7fd0f0136e10 auth_method 0 2024-09-16T03:48:34.960 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.951+0000 7fd0f5bb2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:48:34.961 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.951+0000 7fd0f5bb2640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:48:34.961 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.951+0000 7fd0f5bb2640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:48:34.961 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.951+0000 7fd0f53b1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:48:34.961 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.951+0000 7fd0f53b1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:48:34.961 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.951+0000 7fd0f5bb2640 1 -- 172.21.15.42:0/767064951 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd0f0136e10 msgr2=0x7fd0f0137220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T03:48:34.961 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.951+0000 7fd0f5bb2640 1 -- 172.21.15.42:0/767064951 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd0f0136e10 msgr2=0x7fd0f0137220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:48:34.961 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.951+0000 7fd0f5bb2640 1 --2- 172.21.15.42:0/767064951 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd0f0136e10 0x7fd0f0137220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:48:34.961 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.951+0000 7fd0f53b1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:48:34.961 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.951+0000 7fd0f5bb2640 1 --2- 172.21.15.42:0/767064951 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd0f0136e10 0x7fd0f0137220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:48:34.961 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.951+0000 7fd0f4bb0640 1 --2- 172.21.15.42:0/767064951 >> v2:172.21.15.154:3300/0 conn(0x7fd0f00a37f0 0x7fd0f00a3c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:48:34.961 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.951+0000 7fd0f4bb0640 10 monclient(hunting): get_auth_request con 0x7fd0f00a37f0 auth_method 0 2024-09-16T03:48:34.961 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.951+0000 7fd0f4bb0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:48:34.961 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.951+0000 7fd0f4bb0640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:48:34.962 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.951+0000 7fd0f4bb0640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:48:34.962 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.951+0000 7fd0f53b1640 10 monclient(hunting): handle_auth_done global_id 764454 payload 274 2024-09-16T03:48:34.962 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.951+0000 7fd0f53b1640 1 -- 172.21.15.42:0/767064951 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd0f0136e10 msgr2=0x7fd0f0137220 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:48:34.962 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.951+0000 7fd0f53b1640 1 --2- 172.21.15.42:0/767064951 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd0f0136e10 0x7fd0f0137220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:34.962 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.951+0000 7fd0f53b1640 1 -- 172.21.15.42:0/767064951 >> v2:172.21.15.154:3300/0 conn(0x7fd0f00a37f0 msgr2=0x7fd0f00a3c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:48:34.962 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.951+0000 7fd0f53b1640 1 --2- 172.21.15.42:0/767064951 >> v2:172.21.15.154:3300/0 conn(0x7fd0f00a37f0 0x7fd0f00a3c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:34.962 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.951+0000 7fd0f53b1640 10 monclient: _finish_hunting 0 2024-09-16T03:48:34.962 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.951+0000 7fd0f53b1640 1 monclient: found mon.noname-a 2024-09-16T03:48:34.962 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.951+0000 7fd0f53b1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:48:34.962 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.951+0000 7fd0f53b1640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:48:34.962 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.951+0000 7fd0f53b1640 1 -- 172.21.15.42:0/767064951 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0f0134780 con 0x7fd0f006db90 2024-09-16T03:48:34.969 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.955+0000 7fd0f53b1640 1 --2- 172.21.15.42:0/767064951 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd0f006db90 0x7fd0f01340c0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fd0f009c130 tx=0x7fd0e801c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57356b218b46cb2 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:48:34.969 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.955+0000 7fd0effff640 1 -- 172.21.15.42:0/767064951 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0e80b0070 con 0x7fd0f006db90 2024-09-16T03:48:34.969 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.955+0000 7fd0effff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:48:34.969 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.955+0000 7fd0effff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:48:34.969 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.955+0000 7fd0effff640 10 monclient: dump: 2024-09-16T03:48:34.969 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:48:34.970 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:48:34.970 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:34.970 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:34.970 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:48:34.970 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:48:34.970 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:48:34.970 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:48:34.970 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:48:34.970 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:48:34.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.955+0000 7fd0effff640 10 monclient: _finish_auth 0 2024-09-16T03:48:34.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.955+0000 7fd0effff640 10 monclient: _check_auth_tickets 2024-09-16T03:48:34.971 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.955+0000 7fd0effff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:48:34.971 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.955+0000 7fd0effff640 1 -- 172.21.15.42:0/767064951 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd0e80ab600 con 0x7fd0f006db90 2024-09-16T03:48:34.971 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.955+0000 7fd0effff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:48:34.971 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.955+0000 7fd0effff640 1 -- 172.21.15.42:0/767064951 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0e80ab9d0 con 0x7fd0f006db90 2024-09-16T03:48:34.971 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.955+0000 7fd0f7615640 5 monclient: authenticate success, global_id 764454 2024-09-16T03:48:34.971 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.955+0000 7fd0f7615640 10 monclient: get_monmap_and_config success 2024-09-16T03:48:34.971 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.955+0000 7fd0f7615640 4 set_mon_vals no callback set 2024-09-16T03:48:34.971 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.955+0000 7fd0effff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:48:34.971 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.955+0000 7fd0effff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:48:34.971 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.955+0000 7fd0effff640 10 monclient: dump: 2024-09-16T03:48:34.971 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:48:34.971 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:48:34.971 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:34.971 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:34.972 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:48:34.972 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:48:34.972 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:48:34.972 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:48:34.972 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:48:34.972 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:48:34.972 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.955+0000 7fd0f7615640 10 monclient: shutdown 2024-09-16T03:48:34.972 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.955+0000 7fd0f7615640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:48:34.972 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.955+0000 7fd0f7615640 1 -- 172.21.15.42:0/767064951 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd0f006db90 msgr2=0x7fd0f01340c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:48:34.972 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.955+0000 7fd0f7615640 1 --2- 172.21.15.42:0/767064951 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd0f006db90 0x7fd0f01340c0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fd0f009c130 tx=0x7fd0e801c3c0 comp rx=0 tx=0).stop 2024-09-16T03:48:34.972 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.955+0000 7fd0f7615640 1 -- 172.21.15.42:0/767064951 shutdown_connections 2024-09-16T03:48:34.972 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.955+0000 7fd0f7615640 1 --2- 172.21.15.42:0/767064951 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd0f0136e10 0x7fd0f0137220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:34.972 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.955+0000 7fd0f7615640 1 --2- 172.21.15.42:0/767064951 >> v2:172.21.15.154:3300/0 conn(0x7fd0f00a37f0 0x7fd0f00a3c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:34.972 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.955+0000 7fd0f7615640 1 --2- 172.21.15.42:0/767064951 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd0f006db90 0x7fd0f01340c0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:34.973 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.955+0000 7fd0f7615640 1 -- 172.21.15.42:0/767064951 >> 172.21.15.42:0/767064951 conn(0x7fd0f00a9fb0 msgr2=0x7fd0f00aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:48:34.973 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.955+0000 7fd0f7615640 1 -- 172.21.15.42:0/767064951 shutdown_connections 2024-09-16T03:48:34.973 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.955+0000 7fd0f7615640 1 -- 172.21.15.42:0/767064951 wait complete. 2024-09-16T03:48:34.973 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.959+0000 7fd0f7615640 10 monclient: build_initial_monmap 2024-09-16T03:48:34.973 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.959+0000 7fd0f7615640 10 monclient: monmap: 2024-09-16T03:48:34.973 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:48:34.973 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:48:34.973 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:48:34.973 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:48:34.973 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:48:34.973 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:48:34.973 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:48:34.973 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:48:34.974 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:48:34.974 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:48:34.974 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.959+0000 7fd0f7615640 1 librados: starting msgr at 2024-09-16T03:48:34.974 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.959+0000 7fd0f7615640 1 librados: starting objecter 2024-09-16T03:48:34.974 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.959+0000 7fd0f7615640 1 Processor -- start 2024-09-16T03:48:34.974 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.959+0000 7fd0f7615640 1 -- start start 2024-09-16T03:48:34.974 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.959+0000 7fd0f7615640 1 librados: setting wanted keys 2024-09-16T03:48:34.974 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.959+0000 7fd0f7615640 1 librados: calling monclient init 2024-09-16T03:48:34.974 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.959+0000 7fd0f7615640 10 monclient: init 2024-09-16T03:48:34.974 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.959+0000 7fd0f7615640 10 monclient: _reopen_session rank -1 2024-09-16T03:48:34.974 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.959+0000 7fd0f7615640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T03:48:34.974 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.959+0000 7fd0f7615640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd0f006db90 0x7fd0f00a3200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:34.974 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.959+0000 7fd0f7615640 10 monclient(hunting): picked mon.noname-b con 0x7fd0f006db90 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:48:34.974 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.959+0000 7fd0f7615640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd0f00a37f0 0x7fd0f013a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:34.975 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.959+0000 7fd0f7615640 10 monclient(hunting): picked mon.noname-c con 0x7fd0f00a37f0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:48:34.975 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.959+0000 7fd0f7615640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd0f0136e10 0x7fd0f013ab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:34.975 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.959+0000 7fd0f7615640 10 monclient(hunting): picked mon.noname-a con 0x7fd0f0136e10 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:48:34.975 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.959+0000 7fd0f7615640 10 monclient(hunting): start opening mon connection 2024-09-16T03:48:34.975 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.959+0000 7fd0f7615640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fd0f0140bd0 con 0x7fd0f0136e10 2024-09-16T03:48:34.975 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.959+0000 7fd0f7615640 10 monclient(hunting): start opening mon connection 2024-09-16T03:48:34.975 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.959+0000 7fd0f7615640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd0f0140d50 con 0x7fd0f006db90 2024-09-16T03:48:34.975 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.959+0000 7fd0f7615640 10 monclient(hunting): start opening mon connection 2024-09-16T03:48:34.975 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.959+0000 7fd0f7615640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fd0f0140ed0 con 0x7fd0f00a37f0 2024-09-16T03:48:34.975 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.959+0000 7fd0f7615640 10 monclient(hunting): _renew_subs 2024-09-16T03:48:34.975 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.959+0000 7fd0f7615640 10 monclient(hunting): authenticate will time out at 1361.929114s 2024-09-16T03:48:34.975 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.959+0000 7fd0f4bb0640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd0f00a37f0 0x7fd0f013a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:48:34.975 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.959+0000 7fd0f5bb2640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd0f0136e10 0x7fd0f013ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:48:34.975 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.959+0000 7fd0f4bb0640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd0f00a37f0 0x7fd0f013a5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:37942/0 (socket says 172.21.15.42:37942) 2024-09-16T03:48:34.976 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.959+0000 7fd0f4bb0640 1 -- 172.21.15.42:0/4008058907 learned_addr learned my addr 172.21.15.42:0/4008058907 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:48:34.976 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.959+0000 7fd0f4bb0640 10 monclient(hunting): get_auth_request con 0x7fd0f00a37f0 auth_method 0 2024-09-16T03:48:34.976 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.959+0000 7fd0f4bb0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:48:34.976 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.959+0000 7fd0f4bb0640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:48:34.976 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.959+0000 7fd0f4bb0640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:48:34.976 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.959+0000 7fd0f5bb2640 10 monclient(hunting): get_auth_request con 0x7fd0f0136e10 auth_method 0 2024-09-16T03:48:34.976 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.959+0000 7fd0f5bb2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:48:34.976 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.959+0000 7fd0f5bb2640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:48:34.976 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.959+0000 7fd0f5bb2640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:48:34.976 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.959+0000 7fd0f5bb2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:48:34.976 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.959+0000 7fd0f5bb2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:48:34.976 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.959+0000 7fd0f4bb0640 1 -- 172.21.15.42:0/4008058907 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd0f00a37f0 msgr2=0x7fd0f013a5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-16T03:48:34.976 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.959+0000 7fd0f4bb0640 1 -- 172.21.15.42:0/4008058907 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd0f00a37f0 msgr2=0x7fd0f013a5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T03:48:34.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.959+0000 7fd0f4bb0640 1 --2- 172.21.15.42:0/4008058907 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd0f00a37f0 0x7fd0f013a5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:48:34.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.959+0000 7fd0f5bb2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:48:34.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.959+0000 7fd0f4bb0640 1 --2- 172.21.15.42:0/4008058907 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd0f00a37f0 0x7fd0f013a5f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:48:34.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.959+0000 7fd0f53b1640 1 --2- 172.21.15.42:0/4008058907 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd0f006db90 0x7fd0f00a3200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:48:34.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.959+0000 7fd0f5bb2640 10 monclient(hunting): handle_auth_done global_id 764457 payload 824 2024-09-16T03:48:34.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.959+0000 7fd0f5bb2640 1 -- 172.21.15.42:0/4008058907 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd0f00a37f0 msgr2=0x7fd0f013a5f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:48:34.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.959+0000 7fd0f5bb2640 1 --2- 172.21.15.42:0/4008058907 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd0f00a37f0 0x7fd0f013a5f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:34.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.959+0000 7fd0f5bb2640 1 -- 172.21.15.42:0/4008058907 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd0f006db90 msgr2=0x7fd0f00a3200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:48:34.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.959+0000 7fd0f5bb2640 1 --2- 172.21.15.42:0/4008058907 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd0f006db90 0x7fd0f00a3200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:34.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.959+0000 7fd0f5bb2640 10 monclient: _finish_hunting 0 2024-09-16T03:48:34.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.959+0000 7fd0f5bb2640 1 monclient: found mon.noname-a 2024-09-16T03:48:34.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.959+0000 7fd0f5bb2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:48:34.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.959+0000 7fd0f5bb2640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:48:34.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.959+0000 7fd0f5bb2640 1 -- 172.21.15.42:0/4008058907 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0f013c860 con 0x7fd0f0136e10 2024-09-16T03:48:34.978 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.959+0000 7fd0f53b1640 10 monclient: get_auth_request con 0x7fd0f006db90 auth_method 0 2024-09-16T03:48:34.978 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.959+0000 7fd0f53b1640 1 --2- 172.21.15.42:0/4008058907 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd0f006db90 0x7fd0f00a3200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T03:48:34.978 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.959+0000 7fd0f5bb2640 1 --2- 172.21.15.42:0/4008058907 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd0f0136e10 0x7fd0f013ab30 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fd0e0007cf0 tx=0x7fd0e000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:48:34.978 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.959+0000 7fd0edffb640 10 client.?.objecter ms_handle_connect 0x7fd0f0136e10 2024-09-16T03:48:34.978 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.959+0000 7fd0edffb640 10 client.?.objecter resend_mon_ops 2024-09-16T03:48:34.978 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.959+0000 7fd0edffb640 1 -- 172.21.15.42:0/4008058907 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0e000ed70 con 0x7fd0f0136e10 2024-09-16T03:48:34.978 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.959+0000 7fd0edffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:48:34.978 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.959+0000 7fd0edffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:48:34.979 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.959+0000 7fd0edffb640 10 monclient: dump: 2024-09-16T03:48:34.979 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:48:34.979 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:48:34.979 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:34.979 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:34.979 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:48:34.979 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:48:34.979 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:48:34.979 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:48:34.979 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:48:34.980 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:48:34.980 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.959+0000 7fd0edffb640 10 monclient: _finish_auth 0 2024-09-16T03:48:34.980 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.959+0000 7fd0edffb640 10 monclient: _check_auth_tickets 2024-09-16T03:48:34.980 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.959+0000 7fd0edffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:48:34.980 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.959+0000 7fd0edffb640 1 -- 172.21.15.42:0/4008058907 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd0e00039a0 con 0x7fd0f0136e10 2024-09-16T03:48:34.980 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.959+0000 7fd0f7615640 5 monclient: authenticate success, global_id 764457 2024-09-16T03:48:34.980 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.959+0000 7fd0f7615640 10 monclient: _renew_subs 2024-09-16T03:48:34.980 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.959+0000 7fd0f7615640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:48:34.980 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.959+0000 7fd0f7615640 1 -- 172.21.15.42:0/4008058907 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0f009c7b0 con 0x7fd0f0136e10 2024-09-16T03:48:34.980 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.959+0000 7fd0edffb640 10 monclient: handle_config config(0 keys) 2024-09-16T03:48:34.980 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.959+0000 7fd0edffb640 1 -- 172.21.15.42:0/4008058907 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0e0003d90 con 0x7fd0f0136e10 2024-09-16T03:48:34.980 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.959+0000 7fd0edffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:48:34.980 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.959+0000 7fd0edffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:48:34.980 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.959+0000 7fd0edffb640 10 monclient: dump: 2024-09-16T03:48:34.981 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:48:34.981 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:48:34.981 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:34.981 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:34.981 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:48:34.981 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:48:34.981 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:48:34.981 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:48:34.981 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:48:34.981 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:48:34.981 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.959+0000 7fd0f7615640 10 client.764457.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:48:34.982 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.959+0000 7fd0f7615640 10 monclient: _renew_subs 2024-09-16T03:48:34.982 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.959+0000 7fd0f7615640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:48:34.982 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.959+0000 7fd0f7615640 1 -- 172.21.15.42:0/4008058907 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0f009ca50 con 0x7fd0f0136e10 2024-09-16T03:48:34.982 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.959+0000 7fd0f7615640 1 librados: init done 2024-09-16T03:48:34.982 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.959+0000 7fd0ef7fe640 4 set_mon_vals no callback set 2024-09-16T03:48:34.982 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.959+0000 7fd0f7615640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:48:34.982 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.959+0000 7fd0f7615640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:48:34.982 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.959+0000 7fd0f7615640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:48:34.982 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.959+0000 7fd0f7615640 1 -- 172.21.15.42:0/4008058907 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd0f00a3c00 con 0x7fd0f0136e10 2024-09-16T03:48:34.982 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.959+0000 7fd0edffb640 1 -- 172.21.15.42:0/4008058907 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fd0e001b020 con 0x7fd0f0136e10 2024-09-16T03:48:34.982 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.959+0000 7fd0edffb640 1 --2- 172.21.15.42:0/4008058907 >> v2:172.21.15.154:6812/254845342 conn(0x7fd0dc037810 0x7fd0dc039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:34.982 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.959+0000 7fd0edffb640 1 -- 172.21.15.42:0/4008058907 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(210..210 src has 1..210) ==== 7442+0+0 (secure 0 0 0) 0x7fd0e0049660 con 0x7fd0f0136e10 2024-09-16T03:48:34.982 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.959+0000 7fd0edffb640 10 client.764457.objecter ms_dispatch 0x7fd0f0000ba0 osd_map(210..210 src has 1..210) 2024-09-16T03:48:34.982 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.959+0000 7fd0edffb640 3 client.764457.objecter handle_osd_map got epochs [210,210] > 0 2024-09-16T03:48:34.983 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.959+0000 7fd0edffb640 3 client.764457.objecter handle_osd_map decoding full epoch 210 2024-09-16T03:48:34.983 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.959+0000 7fd0edffb640 20 client.764457.objecter dump_active .. 0 homeless 2024-09-16T03:48:34.983 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.959+0000 7fd0f53b1640 1 --2- 172.21.15.42:0/4008058907 >> v2:172.21.15.154:6812/254845342 conn(0x7fd0dc037810 0x7fd0dc039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:48:34.983 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.959+0000 7fd0f53b1640 10 monclient: get_auth_request con 0x7fd0dc037810 auth_method 0 2024-09-16T03:48:34.983 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.963+0000 7fd0f53b1640 1 --2- 172.21.15.42:0/4008058907 >> v2:172.21.15.154:6812/254845342 conn(0x7fd0dc037810 0x7fd0dc039cd0 secure :-1 s=READY pgs=3486 cs=0 l=1 rev1=1 crypto rx=0x7fd0e80ad000 tx=0x7fd0e8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:48:34.983 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.963+0000 7fd0edffb640 10 client.764457.objecter ms_handle_connect 0x7fd0dc037810 2024-09-16T03:48:34.983 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.967+0000 7fd0edffb640 1 -- 172.21.15.42:0/4008058907 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7fd0e00041c0 con 0x7fd0f0136e10 2024-09-16T03:48:34.983 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.967+0000 7fd0edffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:48:34.983 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:34.967+0000 7fd0edffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:48:35.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.191+0000 7fd0f7615640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:48:35.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.191+0000 7fd0f7615640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:48:35.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.191+0000 7fd0f7615640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:48:35.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.191+0000 7fd0f7615640 1 -- 172.21.15.42:0/4008058907 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fd0f006f6a0 con 0x7fd0f0136e10 2024-09-16T03:48:35.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.191+0000 7fd0edffb640 1 -- 172.21.15.42:0/4008058907 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7fd0e0013740 con 0x7fd0f0136e10 2024-09-16T03:48:35.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.191+0000 7fd0edffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:48:35.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.191+0000 7fd0edffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:48:35.195 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":60,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:48:35.197 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.195+0000 7fd0f7615640 10 librados: watch_flush enter 2024-09-16T03:48:35.198 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.195+0000 7fd0f7615640 10 librados: watch_flush exit 2024-09-16T03:48:35.198 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.195+0000 7fd0f7615640 20 client.764457.objecter shutdown clearing up homeless session... 2024-09-16T03:48:35.198 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.195+0000 7fd0f7615640 10 client.764457.objecter successfully canceled tick 2024-09-16T03:48:35.198 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.195+0000 7fd0f7615640 1 -- 172.21.15.42:0/4008058907 >> v2:172.21.15.154:6812/254845342 conn(0x7fd0dc037810 msgr2=0x7fd0dc039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:48:35.198 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.195+0000 7fd0f7615640 1 --2- 172.21.15.42:0/4008058907 >> v2:172.21.15.154:6812/254845342 conn(0x7fd0dc037810 0x7fd0dc039cd0 secure :-1 s=READY pgs=3486 cs=0 l=1 rev1=1 crypto rx=0x7fd0e80ad000 tx=0x7fd0e8008040 comp rx=0 tx=0).stop 2024-09-16T03:48:35.198 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.195+0000 7fd0f7615640 10 monclient: shutdown 2024-09-16T03:48:35.198 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.195+0000 7fd0f7615640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:48:35.198 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.195+0000 7fd0f7615640 1 -- 172.21.15.42:0/4008058907 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd0f0136e10 msgr2=0x7fd0f013ab30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:48:35.198 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.195+0000 7fd0f7615640 1 --2- 172.21.15.42:0/4008058907 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd0f0136e10 0x7fd0f013ab30 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fd0e0007cf0 tx=0x7fd0e000e4b0 comp rx=0 tx=0).stop 2024-09-16T03:48:35.198 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.195+0000 7fd0f7615640 1 -- 172.21.15.42:0/4008058907 shutdown_connections 2024-09-16T03:48:35.198 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.195+0000 7fd0f7615640 1 --2- 172.21.15.42:0/4008058907 >> v2:172.21.15.154:6812/254845342 conn(0x7fd0dc037810 0x7fd0dc039cd0 unknown :-1 s=CLOSED pgs=3486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:35.198 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.195+0000 7fd0f7615640 1 --2- 172.21.15.42:0/4008058907 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd0f0136e10 0x7fd0f013ab30 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:35.198 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.195+0000 7fd0f7615640 1 --2- 172.21.15.42:0/4008058907 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd0f00a37f0 0x7fd0f013a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:35.199 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.195+0000 7fd0f7615640 1 --2- 172.21.15.42:0/4008058907 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd0f006db90 0x7fd0f00a3200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:35.199 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.195+0000 7fd0f7615640 1 -- 172.21.15.42:0/4008058907 >> 172.21.15.42:0/4008058907 conn(0x7fd0f00a9fb0 msgr2=0x7fd0f01307b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:48:35.199 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.195+0000 7fd0f7615640 1 -- 172.21.15.42:0/4008058907 shutdown_connections 2024-09-16T03:48:35.199 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.195+0000 7fd0f7615640 1 -- 172.21.15.42:0/4008058907 wait complete. 2024-09-16T03:48:35.199 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.195+0000 7fd0f7615640 1 librados: shutdown 2024-09-16T03:48:35.214 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T03:48:35.214 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T03:48:35.214 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T03:48:35.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.339+0000 7f878b92d640 10 monclient: get_monmap_and_config 2024-09-16T03:48:35.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.339+0000 7f878b92d640 10 monclient: build_initial_monmap 2024-09-16T03:48:35.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.339+0000 7f878b92d640 10 monclient: monmap: 2024-09-16T03:48:35.341 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:48:35.341 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:48:35.341 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:48:35.340504+0000 2024-09-16T03:48:35.342 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:48:35.340504+0000 2024-09-16T03:48:35.342 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:48:35.342 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:48:35.342 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:48:35.342 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:48:35.342 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:48:35.342 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:48:35.342 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:48:35.342 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:48:35.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.339+0000 7f878b92d640 1 Processor -- start 2024-09-16T03:48:35.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.339+0000 7f878b92d640 1 -- start start 2024-09-16T03:48:35.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.339+0000 7f878b92d640 10 monclient: init 2024-09-16T03:48:35.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.339+0000 7f878b92d640 10 monclient: _reopen_session rank -1 2024-09-16T03:48:35.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.339+0000 7f878b92d640 10 monclient: _add_conns ranks=[2,0,3,4,1] 2024-09-16T03:48:35.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.339+0000 7f878b92d640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f878406db90 0x7f87841340c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:35.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.339+0000 7f878b92d640 10 monclient(hunting): picked mon.noname-b con 0x7f878406db90 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:48:35.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.339+0000 7f878b92d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f87840a37f0 addr v1:172.21.15.42:6789/0 2024-09-16T03:48:35.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.339+0000 7f878b92d640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f87840a3bd0 0x7f8784137520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:35.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.339+0000 7f878b92d640 10 monclient(hunting): picked mon.noname-a con 0x7f87840a3bd0 addr v2:172.21.15.42:3300/0 2024-09-16T03:48:35.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.339+0000 7f878b92d640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f878406ece0 con 0x7f87840a37f0 2024-09-16T03:48:35.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.339+0000 7f878b92d640 10 monclient(hunting): start opening mon connection 2024-09-16T03:48:35.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.339+0000 7f878b92d640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f878406df70 con 0x7f87840a3bd0 2024-09-16T03:48:35.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.339+0000 7f878b92d640 10 monclient(hunting): start opening mon connection 2024-09-16T03:48:35.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.339+0000 7f878b92d640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f878409bf80 con 0x7f878406db90 2024-09-16T03:48:35.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.339+0000 7f878b92d640 10 monclient(hunting): _renew_subs 2024-09-16T03:48:35.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.339+0000 7f878b92d640 10 monclient(hunting): authenticate will time out at 1362.309992s 2024-09-16T03:48:35.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.339+0000 7f87896c9640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f878406db90 0x7f87841340c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:48:35.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.339+0000 7f8789eca640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f87840a3bd0 0x7f8784137520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:48:35.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.339+0000 7f8789eca640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f87840a3bd0 0x7f8784137520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:44670/0 (socket says 172.21.15.42:44670) 2024-09-16T03:48:35.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.339+0000 7f87896c9640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f878406db90 0x7f87841340c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:37950/0 (socket says 172.21.15.42:37950) 2024-09-16T03:48:35.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.339+0000 7f8789eca640 1 -- 172.21.15.42:0/4169837013 learned_addr learned my addr 172.21.15.42:0/4169837013 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:48:35.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.339+0000 7f8789eca640 10 monclient(hunting): get_auth_request con 0x7f87840a3bd0 auth_method 0 2024-09-16T03:48:35.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.339+0000 7f8789eca640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:48:35.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.339+0000 7f8789eca640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:48:35.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.339+0000 7f8789eca640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:48:35.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.339+0000 7f87896c9640 10 monclient(hunting): get_auth_request con 0x7f878406db90 auth_method 0 2024-09-16T03:48:35.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.339+0000 7f87896c9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:48:35.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.339+0000 7f87896c9640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:48:35.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.339+0000 7f87896c9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:48:35.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.339+0000 7f87896c9640 1 -- 172.21.15.42:0/4169837013 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f878406db90 msgr2=0x7f87841340c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T03:48:35.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.339+0000 7f87896c9640 1 -- 172.21.15.42:0/4169837013 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f878406db90 msgr2=0x7f87841340c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:48:35.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.339+0000 7f87896c9640 1 --2- 172.21.15.42:0/4169837013 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f878406db90 0x7f87841340c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:48:35.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.339+0000 7f87896c9640 1 --2- 172.21.15.42:0/4169837013 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f878406db90 0x7f87841340c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:48:35.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.339+0000 7f8789eca640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:48:35.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.339+0000 7f8789eca640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:48:35.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.339+0000 7f8789eca640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:48:35.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.339+0000 7f8789eca640 10 monclient(hunting): handle_auth_done global_id 764460 payload 274 2024-09-16T03:48:35.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.339+0000 7f8789eca640 1 -- 172.21.15.42:0/4169837013 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f878406db90 msgr2=0x7f87841340c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:48:35.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.339+0000 7f8789eca640 1 --2- 172.21.15.42:0/4169837013 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f878406db90 0x7f87841340c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:35.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.339+0000 7f8789eca640 1 -- 172.21.15.42:0/4169837013 >> v1:172.21.15.42:6789/0 conn(0x7f87840a37f0 legacy=0x7f8784136e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:48:35.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.339+0000 7f8789eca640 10 monclient: _finish_hunting 0 2024-09-16T03:48:35.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.339+0000 7f8789eca640 1 monclient: found mon.noname-a 2024-09-16T03:48:35.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.339+0000 7f8789eca640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:48:35.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.339+0000 7f8789eca640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:48:35.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.339+0000 7f8789eca640 1 -- 172.21.15.42:0/4169837013 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8784134780 con 0x7f87840a3bd0 2024-09-16T03:48:35.361 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.343+0000 7f8789eca640 1 --2- 172.21.15.42:0/4169837013 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f87840a3bd0 0x7f8784137520 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f877c009e50 tx=0x7f877c01d350 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c1764a757469673 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:48:35.361 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.343+0000 7f877bfff640 1 -- 172.21.15.42:0/4169837013 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f877c0a56c0 con 0x7f87840a3bd0 2024-09-16T03:48:35.361 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.343+0000 7f877bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:48:35.361 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.343+0000 7f877bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:48:35.361 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.343+0000 7f877bfff640 10 monclient: dump: 2024-09-16T03:48:35.361 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:48:35.362 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:48:35.362 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:35.362 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:35.362 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:48:35.362 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:48:35.362 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:48:35.362 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:48:35.362 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:48:35.362 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:48:35.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.343+0000 7f877bfff640 10 monclient: _finish_auth 0 2024-09-16T03:48:35.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.343+0000 7f877bfff640 10 monclient: _check_auth_tickets 2024-09-16T03:48:35.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.343+0000 7f877bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:48:35.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.343+0000 7f877bfff640 1 -- 172.21.15.42:0/4169837013 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f877c0a5860 con 0x7f87840a3bd0 2024-09-16T03:48:35.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.343+0000 7f878b92d640 5 monclient: authenticate success, global_id 764460 2024-09-16T03:48:35.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.343+0000 7f878b92d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:48:35.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.343+0000 7f877bfff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:48:35.364 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.343+0000 7f877bfff640 1 -- 172.21.15.42:0/4169837013 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f877c0a5c30 con 0x7f87840a3bd0 2024-09-16T03:48:35.364 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.343+0000 7f877bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:48:35.364 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.343+0000 7f877bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:48:35.364 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.343+0000 7f877bfff640 10 monclient: dump: 2024-09-16T03:48:35.364 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:48:35.364 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:48:35.364 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:35.364 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:35.364 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:48:35.364 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:48:35.365 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:48:35.365 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:48:35.365 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:48:35.365 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:48:35.365 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.343+0000 7f878b92d640 10 monclient: get_monmap_and_config success 2024-09-16T03:48:35.365 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.343+0000 7f878b92d640 4 set_mon_vals no callback set 2024-09-16T03:48:35.365 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.343+0000 7f878b92d640 10 monclient: shutdown 2024-09-16T03:48:35.365 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.343+0000 7f878b92d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:48:35.365 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.343+0000 7f878b92d640 1 -- 172.21.15.42:0/4169837013 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f87840a3bd0 msgr2=0x7f8784137520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:48:35.365 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.343+0000 7f878b92d640 1 --2- 172.21.15.42:0/4169837013 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f87840a3bd0 0x7f8784137520 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f877c009e50 tx=0x7f877c01d350 comp rx=0 tx=0).stop 2024-09-16T03:48:35.365 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.343+0000 7f878b92d640 1 -- 172.21.15.42:0/4169837013 shutdown_connections 2024-09-16T03:48:35.366 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.343+0000 7f878b92d640 1 --2- 172.21.15.42:0/4169837013 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f87840a3bd0 0x7f8784137520 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:35.366 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.343+0000 7f878b92d640 1 --2- 172.21.15.42:0/4169837013 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f878406db90 0x7f87841340c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:35.366 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.343+0000 7f878b92d640 1 -- 172.21.15.42:0/4169837013 >> 172.21.15.42:0/4169837013 conn(0x7f87840a9fb0 msgr2=0x7f87840aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:48:35.366 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.343+0000 7f878b92d640 1 -- 172.21.15.42:0/4169837013 shutdown_connections 2024-09-16T03:48:35.366 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.343+0000 7f878b92d640 1 -- 172.21.15.42:0/4169837013 wait complete. 2024-09-16T03:48:35.366 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.343+0000 7f878b92d640 10 monclient: build_initial_monmap 2024-09-16T03:48:35.366 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.343+0000 7f878b92d640 10 monclient: monmap: 2024-09-16T03:48:35.366 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:48:35.366 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:48:35.366 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:48:35.366 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:48:35.367 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:48:35.367 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:48:35.367 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:48:35.367 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:48:35.367 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:48:35.367 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:48:35.367 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.343+0000 7f878b92d640 1 librados: starting msgr at 2024-09-16T03:48:35.367 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.343+0000 7f878b92d640 1 librados: starting objecter 2024-09-16T03:48:35.367 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.343+0000 7f878b92d640 1 Processor -- start 2024-09-16T03:48:35.367 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.343+0000 7f878b92d640 1 -- start start 2024-09-16T03:48:35.367 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.343+0000 7f878b92d640 1 librados: setting wanted keys 2024-09-16T03:48:35.368 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.343+0000 7f878b92d640 1 librados: calling monclient init 2024-09-16T03:48:35.368 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.343+0000 7f878b92d640 10 monclient: init 2024-09-16T03:48:35.368 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.343+0000 7f878b92d640 10 monclient: _reopen_session rank -1 2024-09-16T03:48:35.368 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.343+0000 7f878b92d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:48:35.368 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.343+0000 7f878b92d640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f878406db90 0x7f878413cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:35.368 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.343+0000 7f878b92d640 10 monclient(hunting): picked mon.noname-a con 0x7f878406db90 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:48:35.368 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.343+0000 7f878b92d640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f87840a37f0 0x7f878413d430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:35.368 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.343+0000 7f878b92d640 10 monclient(hunting): picked mon.noname-c con 0x7f87840a37f0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:48:35.368 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.343+0000 7f878b92d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f87840a3bd0 0x7f878413d970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:35.368 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.343+0000 7f878b92d640 10 monclient(hunting): picked mon.noname-b con 0x7f87840a3bd0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:48:35.368 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.343+0000 7f878b92d640 10 monclient(hunting): start opening mon connection 2024-09-16T03:48:35.369 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.343+0000 7f878b92d640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f87841419d0 con 0x7f878406db90 2024-09-16T03:48:35.369 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.343+0000 7f878b92d640 10 monclient(hunting): start opening mon connection 2024-09-16T03:48:35.369 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.343+0000 7f878b92d640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f87840a4100 con 0x7f87840a3bd0 2024-09-16T03:48:35.369 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.343+0000 7f878b92d640 10 monclient(hunting): start opening mon connection 2024-09-16T03:48:35.369 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.343+0000 7f878b92d640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f8784141b50 con 0x7f87840a37f0 2024-09-16T03:48:35.369 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.343+0000 7f878b92d640 10 monclient(hunting): _renew_subs 2024-09-16T03:48:35.369 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.343+0000 7f878b92d640 10 monclient(hunting): authenticate will time out at 1362.315423s 2024-09-16T03:48:35.369 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.343+0000 7f87896c9640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f878406db90 0x7f878413cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:48:35.369 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.343+0000 7f8788ec8640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f87840a37f0 0x7f878413d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:48:35.369 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.343+0000 7f87896c9640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f878406db90 0x7f878413cef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:44678/0 (socket says 172.21.15.42:44678) 2024-09-16T03:48:35.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.343+0000 7f87896c9640 1 -- 172.21.15.42:0/900660455 learned_addr learned my addr 172.21.15.42:0/900660455 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:48:35.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.343+0000 7f8788ec8640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f87840a37f0 0x7f878413d430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:37966/0 (socket says 172.21.15.42:37966) 2024-09-16T03:48:35.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.343+0000 7f87896c9640 10 monclient(hunting): get_auth_request con 0x7f878406db90 auth_method 0 2024-09-16T03:48:35.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.343+0000 7f87896c9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:48:35.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.343+0000 7f87896c9640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:48:35.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.343+0000 7f87896c9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:48:35.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.343+0000 7f8788ec8640 10 monclient(hunting): get_auth_request con 0x7f87840a37f0 auth_method 0 2024-09-16T03:48:35.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.343+0000 7f8788ec8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:48:35.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.343+0000 7f8788ec8640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:48:35.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.343+0000 7f8788ec8640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:48:35.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.343+0000 7f87896c9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:48:35.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.343+0000 7f87896c9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:48:35.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.343+0000 7f87896c9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:48:35.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.343+0000 7f8788ec8640 1 -- 172.21.15.42:0/900660455 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f87840a37f0 msgr2=0x7f878413d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-16T03:48:35.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.343+0000 7f8788ec8640 1 -- 172.21.15.42:0/900660455 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f87840a37f0 msgr2=0x7f878413d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T03:48:35.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.343+0000 7f8788ec8640 1 --2- 172.21.15.42:0/900660455 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f87840a37f0 0x7f878413d430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:48:35.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.343+0000 7f8788ec8640 1 --2- 172.21.15.42:0/900660455 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f87840a37f0 0x7f878413d430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:48:35.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.343+0000 7f8789eca640 1 --2- 172.21.15.42:0/900660455 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f87840a3bd0 0x7f878413d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:48:35.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.343+0000 7f87896c9640 10 monclient(hunting): handle_auth_done global_id 764466 payload 824 2024-09-16T03:48:35.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.343+0000 7f87896c9640 1 -- 172.21.15.42:0/900660455 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f87840a37f0 msgr2=0x7f878413d430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:48:35.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.343+0000 7f87896c9640 1 --2- 172.21.15.42:0/900660455 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f87840a37f0 0x7f878413d430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:35.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.343+0000 7f87896c9640 1 -- 172.21.15.42:0/900660455 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f87840a3bd0 msgr2=0x7f878413d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:48:35.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.343+0000 7f87896c9640 1 --2- 172.21.15.42:0/900660455 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f87840a3bd0 0x7f878413d970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:35.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.343+0000 7f87896c9640 10 monclient: _finish_hunting 0 2024-09-16T03:48:35.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.343+0000 7f87896c9640 1 monclient: found mon.noname-a 2024-09-16T03:48:35.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.343+0000 7f87896c9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:48:35.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.343+0000 7f87896c9640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:48:35.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.343+0000 7f87896c9640 1 -- 172.21.15.42:0/900660455 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f878409ce00 con 0x7f878406db90 2024-09-16T03:48:35.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.343+0000 7f8789eca640 10 monclient: get_auth_request con 0x7f87840a3bd0 auth_method 0 2024-09-16T03:48:35.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.343+0000 7f8789eca640 1 --2- 172.21.15.42:0/900660455 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f87840a3bd0 0x7f878413d970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T03:48:35.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.347+0000 7f87896c9640 1 --2- 172.21.15.42:0/900660455 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f878406db90 0x7f878413cef0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f878000e7e0 tx=0x7f878000eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:48:35.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.347+0000 7f8779ffb640 10 client.?.objecter ms_handle_connect 0x7f878406db90 2024-09-16T03:48:35.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.347+0000 7f8779ffb640 10 client.?.objecter resend_mon_ops 2024-09-16T03:48:35.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.347+0000 7f8779ffb640 1 -- 172.21.15.42:0/900660455 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8780017020 con 0x7f878406db90 2024-09-16T03:48:35.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.347+0000 7f8779ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:48:35.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.347+0000 7f8779ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:48:35.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.347+0000 7f8779ffb640 10 monclient: dump: 2024-09-16T03:48:35.374 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:48:35.374 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:48:35.374 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:35.374 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:35.374 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:48:35.374 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:48:35.374 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:48:35.374 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:48:35.374 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:48:35.374 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:48:35.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.347+0000 7f8779ffb640 10 monclient: _finish_auth 0 2024-09-16T03:48:35.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.347+0000 7f8779ffb640 10 monclient: _check_auth_tickets 2024-09-16T03:48:35.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.347+0000 7f8779ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:48:35.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.347+0000 7f8779ffb640 1 -- 172.21.15.42:0/900660455 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f878000c6d0 con 0x7f878406db90 2024-09-16T03:48:35.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.347+0000 7f878b92d640 5 monclient: authenticate success, global_id 764466 2024-09-16T03:48:35.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.347+0000 7f878b92d640 10 monclient: _renew_subs 2024-09-16T03:48:35.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.347+0000 7f878b92d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:48:35.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.347+0000 7f878b92d640 1 -- 172.21.15.42:0/900660455 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f878409dc60 con 0x7f878406db90 2024-09-16T03:48:35.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.347+0000 7f8779ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T03:48:35.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.347+0000 7f8779ffb640 1 -- 172.21.15.42:0/900660455 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f878000f040 con 0x7f878406db90 2024-09-16T03:48:35.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.347+0000 7f8779ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:48:35.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.347+0000 7f8779ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:48:35.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.347+0000 7f8779ffb640 10 monclient: dump: 2024-09-16T03:48:35.376 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:48:35.376 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:48:35.376 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:35.376 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:35.376 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:48:35.376 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:48:35.376 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:48:35.377 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:48:35.377 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:48:35.377 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:48:35.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.347+0000 7f877b7fe640 4 set_mon_vals no callback set 2024-09-16T03:48:35.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.347+0000 7f878b92d640 10 client.764466.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:48:35.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.347+0000 7f878b92d640 10 monclient: _renew_subs 2024-09-16T03:48:35.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.347+0000 7f878b92d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:48:35.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.347+0000 7f878b92d640 1 -- 172.21.15.42:0/900660455 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87841cedb0 con 0x7f878406db90 2024-09-16T03:48:35.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.347+0000 7f878b92d640 1 librados: init done 2024-09-16T03:48:35.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.347+0000 7f8779ffb640 1 -- 172.21.15.42:0/900660455 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f878001a020 con 0x7f878406db90 2024-09-16T03:48:35.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.347+0000 7f878b92d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:48:35.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.347+0000 7f878b92d640 1 --2- 172.21.15.42:0/900660455 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f87841cf0a0 0x7f87841cf480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:35.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.347+0000 7f878b92d640 10 monclient(hunting): start opening mon connection 2024-09-16T03:48:35.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.347+0000 7f878b92d640 1 -- 172.21.15.42:0/900660455 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f878409de30 con 0x7f87841cf0a0 2024-09-16T03:48:35.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.347+0000 7f8788ec8640 1 --2- 172.21.15.42:0/900660455 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f87841cf0a0 0x7f87841cf480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:48:35.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.347+0000 7f8788ec8640 10 monclient: get_auth_request con 0x7f87841cf0a0 auth_method 0 2024-09-16T03:48:35.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.347+0000 7f8788ec8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:48:35.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.347+0000 7f8788ec8640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:48:35.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.347+0000 7f8788ec8640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:48:35.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.347+0000 7f8779ffb640 1 --2- 172.21.15.42:0/900660455 >> v2:172.21.15.154:6812/254845342 conn(0x7f8768037810 0x7f8768039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:35.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.347+0000 7f8779ffb640 1 -- 172.21.15.42:0/900660455 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(210..210 src has 1..210) ==== 7442+0+0 (secure 0 0 0) 0x7f87800493c0 con 0x7f878406db90 2024-09-16T03:48:35.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.347+0000 7f8788ec8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:48:35.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.347+0000 7f8788ec8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:48:35.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.347+0000 7f8779ffb640 10 client.764466.objecter ms_dispatch 0x7f8784000ba0 osd_map(210..210 src has 1..210) 2024-09-16T03:48:35.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.347+0000 7f8779ffb640 3 client.764466.objecter handle_osd_map got epochs [210,210] > 0 2024-09-16T03:48:35.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.347+0000 7f8779ffb640 3 client.764466.objecter handle_osd_map decoding full epoch 210 2024-09-16T03:48:35.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.347+0000 7f8788ec8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:48:35.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.347+0000 7f8779ffb640 20 client.764466.objecter dump_active .. 0 homeless 2024-09-16T03:48:35.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.347+0000 7f8788ec8640 10 monclient(hunting): handle_auth_done global_id 764469 payload 824 2024-09-16T03:48:35.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.347+0000 7f8788ec8640 1 -- 172.21.15.42:0/900660455 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f87841340c0 con 0x7f87841cf0a0 2024-09-16T03:48:35.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.347+0000 7f8788ec8640 1 --2- 172.21.15.42:0/900660455 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f87841cf0a0 0x7f87841cf480 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f8784132fc0 tx=0x7f877400e8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:48:35.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.347+0000 7f8779ffb640 10 client.764466.objecter ms_handle_connect 0x7f87841cf0a0 2024-09-16T03:48:35.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.347+0000 7f8779ffb640 10 client.764466.objecter resend_mon_ops 2024-09-16T03:48:35.380 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.347+0000 7f8789eca640 1 --2- 172.21.15.42:0/900660455 >> v2:172.21.15.154:6812/254845342 conn(0x7f8768037810 0x7f8768039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:48:35.380 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.347+0000 7f8789eca640 10 monclient: get_auth_request con 0x7f8768037810 auth_method 0 2024-09-16T03:48:35.380 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.347+0000 7f8779ffb640 1 -- 172.21.15.42:0/900660455 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8774019070 con 0x7f87841cf0a0 2024-09-16T03:48:35.380 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.347+0000 7f8779ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:48:35.380 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.347+0000 7f8779ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:48:35.380 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.347+0000 7f8779ffb640 10 monclient: dump: 2024-09-16T03:48:35.380 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:48:35.380 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:48:35.380 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:35.380 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:35.380 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:48:35.381 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:48:35.381 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:48:35.381 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:48:35.381 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:48:35.381 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:48:35.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.347+0000 7f8779ffb640 1 -- 172.21.15.42:0/900660455 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f87740052e0 con 0x7f87841cf0a0 2024-09-16T03:48:35.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.347+0000 7f8779ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:48:35.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.347+0000 7f8779ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:48:35.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.347+0000 7f8779ffb640 1 -- 172.21.15.42:0/900660455 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f87841cf0a0 msgr2=0x7f87841cf480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:48:35.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.347+0000 7f8779ffb640 1 --2- 172.21.15.42:0/900660455 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f87841cf0a0 0x7f87841cf480 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f8784132fc0 tx=0x7f877400e8b0 comp rx=0 tx=0).stop 2024-09-16T03:48:35.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.347+0000 7f8779ffb640 1 -- 172.21.15.42:0/900660455 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f87841cf0a0 msgr2=0x7f87841cf480 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:48:35.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.347+0000 7f8779ffb640 1 --2- 172.21.15.42:0/900660455 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f87841cf0a0 0x7f87841cf480 secure :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f8784132fc0 tx=0x7f877400e8b0 comp rx=0 tx=0).stop 2024-09-16T03:48:35.382 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.347+0000 7f8779ffb640 10 client.764466.objecter ms_handle_connect 0x7f8768037810 2024-09-16T03:48:35.382 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.347+0000 7f8789eca640 1 --2- 172.21.15.42:0/900660455 >> v2:172.21.15.154:6812/254845342 conn(0x7f8768037810 0x7f8768039cd0 secure :-1 s=READY pgs=3487 cs=0 l=1 rev1=1 crypto rx=0x7f877c0ae000 tx=0x7f877c004ac0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:48:35.382 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.355+0000 7f878b92d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:48:35.382 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.355+0000 7f878b92d640 1 --2- 172.21.15.42:0/900660455 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f87841cf9c0 0x7f8784137f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:35.382 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.355+0000 7f878b92d640 10 monclient(hunting): start opening mon connection 2024-09-16T03:48:35.382 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.355+0000 7f878b92d640 1 -- 172.21.15.42:0/900660455 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f87841330c0 con 0x7f87841cf9c0 2024-09-16T03:48:35.382 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.355+0000 7f8788ec8640 1 --2- 172.21.15.42:0/900660455 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f87841cf9c0 0x7f8784137f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:48:35.382 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.355+0000 7f8788ec8640 10 monclient: get_auth_request con 0x7f87841cf9c0 auth_method 0 2024-09-16T03:48:35.382 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.355+0000 7f8788ec8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:48:35.382 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.355+0000 7f8788ec8640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:48:35.382 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.355+0000 7f8788ec8640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:48:35.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.355+0000 7f8788ec8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:48:35.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.355+0000 7f8788ec8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:48:35.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.355+0000 7f8788ec8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:48:35.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.355+0000 7f8788ec8640 10 monclient(hunting): handle_auth_done global_id 764472 payload 824 2024-09-16T03:48:35.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.355+0000 7f8788ec8640 1 -- 172.21.15.42:0/900660455 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f87841335d0 con 0x7f87841cf9c0 2024-09-16T03:48:35.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.355+0000 7f8788ec8640 1 --2- 172.21.15.42:0/900660455 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f87841cf9c0 0x7f8784137f40 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f8784132fc0 tx=0x7f8774000ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:48:35.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.355+0000 7f8779ffb640 10 client.764466.objecter ms_handle_connect 0x7f87841cf9c0 2024-09-16T03:48:35.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.355+0000 7f8779ffb640 10 client.764466.objecter resend_mon_ops 2024-09-16T03:48:35.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.355+0000 7f8779ffb640 1 -- 172.21.15.42:0/900660455 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8774019040 con 0x7f87841cf9c0 2024-09-16T03:48:35.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.355+0000 7f8779ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:48:35.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.355+0000 7f8779ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:48:35.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.355+0000 7f8779ffb640 10 monclient: dump: 2024-09-16T03:48:35.384 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:48:35.384 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:48:35.384 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:35.384 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:35.384 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:48:35.384 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:48:35.384 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:48:35.384 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:48:35.384 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:48:35.384 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:48:35.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.355+0000 7f8779ffb640 1 -- 172.21.15.42:0/900660455 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4881 (secure 0 0 0) 0x7f8774003600 con 0x7f87841cf9c0 2024-09-16T03:48:35.385 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.355+0000 7f8779ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:48:35.385 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.355+0000 7f8779ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:48:35.385 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.355+0000 7f8779ffb640 1 -- 172.21.15.42:0/900660455 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f87841cf9c0 msgr2=0x7f8784137f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:48:35.385 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.355+0000 7f8779ffb640 1 --2- 172.21.15.42:0/900660455 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f87841cf9c0 0x7f8784137f40 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f8784132fc0 tx=0x7f8774000ee0 comp rx=0 tx=0).stop 2024-09-16T03:48:35.385 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.355+0000 7f8779ffb640 1 -- 172.21.15.42:0/900660455 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f87841cf9c0 msgr2=0x7f8784137f40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:48:35.385 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.355+0000 7f8779ffb640 1 --2- 172.21.15.42:0/900660455 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f87841cf9c0 0x7f8784137f40 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:35.385 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:48:35.385 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:48:35.386 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:48:35.386 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "leader", 2024-09-16T03:48:35.386 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 60, 2024-09-16T03:48:35.386 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 106147, 2024-09-16T03:48:35.386 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:48:35.386 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:48:35.386 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T03:48:35.386 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T03:48:35.386 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:48:35.386 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 1, 2024-09-16T03:48:35.386 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:48:35.387 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:48:35.387 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:48:35.387 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:48:35.387 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:48:35.387 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:48:35.387 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:48:35.387 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:48:35.387 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:48:35.387 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:48:35.387 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:48:35.387 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:48:35.387 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:48:35.388 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:48:35.388 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:48:35.388 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:48:35.388 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:48:35.388 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:48:35.388 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:48:35.388 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:48:35.388 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:48:35.388 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:48:35.388 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:48:35.388 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:48:35.389 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:48:35.389 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:48:35.389 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:48:35.389 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:48:35.389 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:48:35.389 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:48:35.389 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:48:35.389 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:48:35.389 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:48:35.389 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:48:35.389 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:48:35.390 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:48:35.390 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:48:35.390 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:48:35.390 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:48:35.390 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:48:35.390 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:48:35.390 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:48:35.390 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:48:35.390 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:48:35.391 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:48:35.391 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:48:35.391 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:48:35.391 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:48:35.391 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:48:35.391 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:48:35.391 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:48:35.391 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:48:35.391 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:48:35.391 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:48:35.391 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:48:35.391 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:48:35.391 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:48:35.391 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:48:35.392 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:48:35.392 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:48:35.392 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:48:35.392 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:48:35.392 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:48:35.392 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:48:35.392 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:48:35.392 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:48:35.392 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:48:35.392 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:48:35.392 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:48:35.392 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:48:35.393 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:48:35.393 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:48:35.393 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:48:35.393 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:48:35.393 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:48:35.393 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:48:35.393 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:48:35.393 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:48:35.393 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:48:35.393 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:48:35.393 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:48:35.393 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:48:35.393 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:48:35.393 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:48:35.394 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:48:35.394 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:48:35.394 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:48:35.394 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:48:35.394 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:48:35.394 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:48:35.394 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:48:35.394 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:48:35.394 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:48:35.394 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:48:35.394 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:48:35.394 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:48:35.394 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:48:35.394 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:48:35.395 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:48:35.395 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:48:35.395 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:48:35.395 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:48:35.395 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:48:35.395 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:48:35.395 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:48:35.395 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:48:35.395 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:48:35.395 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:48:35.395 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:48:35.395 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:48:35.395 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:48:35.396 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:48:35.396 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:48:35.396 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:48:35.396 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:48:35.396 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:48:35.396 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:48:35.396 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:48:35.396 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:48:35.396 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:48:35.396 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:48:35.396 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:48:35.396 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:48:35.396 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:48:35.396 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:48:35.397 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:48:35.397 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:48:35.397 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:48:35.397 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:48:35.397 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:48:35.397 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:48:35.397 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:48:35.397 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:48:35.397 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:48:35.397 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:48:35.397 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:48:35.397 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:48:35.397 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:48:35.397 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:48:35.398 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:48:35.398 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:48:35.398 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T03:48:35.398 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:48:35.398 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:48:35.398 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:48:35.398 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 5 2024-09-16T03:48:35.398 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:48:35.398 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:48:35.398 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:48:35.398 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:48:35.398 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:48:35.398 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:48:35.398 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 7 2024-09-16T03:48:35.399 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:48:35.399 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:48:35.399 INFO:teuthology.orchestra.run.smithi042.stdout: "mgr": [ 2024-09-16T03:48:35.399 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:48:35.399 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:48:35.399 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:48:35.399 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:48:35.399 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:48:35.399 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:48:35.399 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:48:35.399 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:48:35.399 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:48:35.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.355+0000 7f878b92d640 10 librados: watch_flush enter 2024-09-16T03:48:35.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.355+0000 7f878b92d640 10 librados: watch_flush exit 2024-09-16T03:48:35.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.355+0000 7f878b92d640 20 client.764466.objecter shutdown clearing up homeless session... 2024-09-16T03:48:35.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.355+0000 7f878b92d640 10 client.764466.objecter successfully canceled tick 2024-09-16T03:48:35.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.355+0000 7f878b92d640 1 -- 172.21.15.42:0/900660455 >> v2:172.21.15.154:6812/254845342 conn(0x7f8768037810 msgr2=0x7f8768039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:48:35.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.355+0000 7f878b92d640 1 --2- 172.21.15.42:0/900660455 >> v2:172.21.15.154:6812/254845342 conn(0x7f8768037810 0x7f8768039cd0 secure :-1 s=READY pgs=3487 cs=0 l=1 rev1=1 crypto rx=0x7f877c0ae000 tx=0x7f877c004ac0 comp rx=0 tx=0).stop 2024-09-16T03:48:35.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.355+0000 7f878b92d640 10 monclient: shutdown 2024-09-16T03:48:35.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.355+0000 7f878b92d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:48:35.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.355+0000 7f878b92d640 1 -- 172.21.15.42:0/900660455 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f878406db90 msgr2=0x7f878413cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:48:35.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.355+0000 7f878b92d640 1 --2- 172.21.15.42:0/900660455 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f878406db90 0x7f878413cef0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f878000e7e0 tx=0x7f878000eca0 comp rx=0 tx=0).stop 2024-09-16T03:48:35.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.355+0000 7f8789eca640 1 -- 172.21.15.42:0/900660455 reap_dead start 2024-09-16T03:48:35.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.355+0000 7f8789eca640 1 -- 172.21.15.42:0/900660455 reap_dead start 2024-09-16T03:48:35.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.355+0000 7f878b92d640 1 -- 172.21.15.42:0/900660455 shutdown_connections 2024-09-16T03:48:35.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.355+0000 7f878b92d640 1 -- 172.21.15.42:0/900660455 >> 172.21.15.42:0/900660455 conn(0x7f87840a9fb0 msgr2=0x7f87841326c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:48:35.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.355+0000 7f878b92d640 1 -- 172.21.15.42:0/900660455 shutdown_connections 2024-09-16T03:48:35.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.355+0000 7f878b92d640 1 -- 172.21.15.42:0/900660455 wait complete. 2024-09-16T03:48:35.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.355+0000 7f878b92d640 1 librados: shutdown 2024-09-16T03:48:35.402 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:48:35.403 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T03:48:35.403 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T03:48:35.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.519+0000 7fbe5b36a640 10 monclient: get_monmap_and_config 2024-09-16T03:48:35.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.519+0000 7fbe5b36a640 10 monclient: build_initial_monmap 2024-09-16T03:48:35.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.519+0000 7fbe5b36a640 10 monclient: monmap: 2024-09-16T03:48:35.525 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:48:35.525 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:48:35.525 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:48:35.520689+0000 2024-09-16T03:48:35.525 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:48:35.520689+0000 2024-09-16T03:48:35.525 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:48:35.525 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:48:35.525 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:48:35.525 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:48:35.525 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:48:35.525 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:48:35.525 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:48:35.525 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:48:35.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.519+0000 7fbe5b36a640 1 Processor -- start 2024-09-16T03:48:35.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.519+0000 7fbe5b36a640 1 -- start start 2024-09-16T03:48:35.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.519+0000 7fbe5b36a640 10 monclient: init 2024-09-16T03:48:35.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.519+0000 7fbe5b36a640 10 monclient: _reopen_session rank -1 2024-09-16T03:48:35.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.519+0000 7fbe5b36a640 10 monclient: _add_conns ranks=[1,4,3,2,0] 2024-09-16T03:48:35.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.519+0000 7fbe5b36a640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbe540a1330 addr v1:172.21.15.154:6789/0 2024-09-16T03:48:35.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.519+0000 7fbe5b36a640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fbe5406dac0 0x7fbe5413a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:35.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.519+0000 7fbe5b36a640 10 monclient(hunting): picked mon.noname-c con 0x7fbe5406dac0 addr v2:172.21.15.154:3300/0 2024-09-16T03:48:35.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.519+0000 7fbe5b36a640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fbe540a1e20 0x7fbe5413c890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:35.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.519+0000 7fbe5b36a640 10 monclient(hunting): picked mon.noname-a con 0x7fbe540a1e20 addr v2:172.21.15.42:3300/0 2024-09-16T03:48:35.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.519+0000 7fbe5b36a640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbe5409c090 con 0x7fbe540a1330 2024-09-16T03:48:35.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.519+0000 7fbe5b36a640 10 monclient(hunting): start opening mon connection 2024-09-16T03:48:35.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.519+0000 7fbe5b36a640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7fbe5409f610 con 0x7fbe540a1e20 2024-09-16T03:48:35.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.519+0000 7fbe5b36a640 10 monclient(hunting): start opening mon connection 2024-09-16T03:48:35.527 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.519+0000 7fbe5b36a640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fbe5409f8f0 con 0x7fbe5406dac0 2024-09-16T03:48:35.527 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.519+0000 7fbe5b36a640 10 monclient(hunting): _renew_subs 2024-09-16T03:48:35.527 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.519+0000 7fbe5b36a640 10 monclient(hunting): authenticate will time out at 1362.490056s 2024-09-16T03:48:35.527 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.519+0000 7fbe59907640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fbe540a1e20 0x7fbe5413c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:48:35.527 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.519+0000 7fbe59907640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fbe540a1e20 0x7fbe5413c890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:44690/0 (socket says 172.21.15.42:44690) 2024-09-16T03:48:35.527 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.519+0000 7fbe59907640 1 -- 172.21.15.42:0/2077703136 learned_addr learned my addr 172.21.15.42:0/2077703136 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:48:35.527 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.519+0000 7fbe59907640 10 monclient(hunting): get_auth_request con 0x7fbe540a1e20 auth_method 0 2024-09-16T03:48:35.527 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.519+0000 7fbe59907640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:48:35.527 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.519+0000 7fbe59907640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:48:35.527 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.519+0000 7fbe59907640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:48:35.527 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.519+0000 7fbe59907640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:48:35.527 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.519+0000 7fbe59907640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:48:35.528 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.519+0000 7fbe59907640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:48:35.528 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.519+0000 7fbe58905640 1 --2- 172.21.15.42:0/2077703136 >> v2:172.21.15.154:3300/0 conn(0x7fbe5406dac0 0x7fbe5413a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:48:35.528 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.519+0000 7fbe58905640 10 monclient(hunting): get_auth_request con 0x7fbe5406dac0 auth_method 0 2024-09-16T03:48:35.528 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.519+0000 7fbe58905640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:48:35.528 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.519+0000 7fbe58905640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:48:35.528 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.519+0000 7fbe58905640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:48:35.528 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.519+0000 7fbe59907640 10 monclient(hunting): handle_auth_done global_id 764475 payload 274 2024-09-16T03:48:35.528 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.519+0000 7fbe59907640 1 -- 172.21.15.42:0/2077703136 >> v2:172.21.15.154:3300/0 conn(0x7fbe5406dac0 msgr2=0x7fbe5413a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:48:35.528 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.519+0000 7fbe59907640 1 --2- 172.21.15.42:0/2077703136 >> v2:172.21.15.154:3300/0 conn(0x7fbe5406dac0 0x7fbe5413a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:35.528 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.519+0000 7fbe59907640 1 -- 172.21.15.42:0/2077703136 >> v1:172.21.15.154:6789/0 conn(0x7fbe540a1330 legacy=0x7fbe540a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:48:35.528 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.519+0000 7fbe59907640 10 monclient: _finish_hunting 0 2024-09-16T03:48:35.528 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.519+0000 7fbe59907640 1 monclient: found mon.noname-a 2024-09-16T03:48:35.529 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.519+0000 7fbe59907640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:48:35.529 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.519+0000 7fbe59907640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:48:35.529 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.519+0000 7fbe59907640 1 -- 172.21.15.42:0/2077703136 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe5409fbd0 con 0x7fbe540a1e20 2024-09-16T03:48:35.537 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.523+0000 7fbe59907640 1 --2- 172.21.15.42:0/2077703136 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbe540a1e20 0x7fbe5413c890 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fbe5406da10 tx=0x7fbe4c01c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fc31c3f85dd59ec server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:48:35.538 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.523+0000 7fbe4bfff640 1 -- 172.21.15.42:0/2077703136 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe4c0a5650 con 0x7fbe540a1e20 2024-09-16T03:48:35.538 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.523+0000 7fbe4bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:48:35.538 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.523+0000 7fbe4bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:48:35.538 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.523+0000 7fbe4bfff640 10 monclient: dump: 2024-09-16T03:48:35.538 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:48:35.538 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:48:35.538 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:35.538 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:35.538 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:48:35.539 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:48:35.539 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:48:35.539 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:48:35.539 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:48:35.539 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:48:35.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.523+0000 7fbe4bfff640 10 monclient: _finish_auth 0 2024-09-16T03:48:35.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.523+0000 7fbe4bfff640 10 monclient: _check_auth_tickets 2024-09-16T03:48:35.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.523+0000 7fbe4bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:48:35.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.523+0000 7fbe5b36a640 5 monclient: authenticate success, global_id 764475 2024-09-16T03:48:35.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.523+0000 7fbe4bfff640 1 -- 172.21.15.42:0/2077703136 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbe4c0a57f0 con 0x7fbe540a1e20 2024-09-16T03:48:35.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.523+0000 7fbe4bfff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:48:35.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.523+0000 7fbe5b36a640 10 monclient: get_monmap_and_config success 2024-09-16T03:48:35.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.523+0000 7fbe5b36a640 4 set_mon_vals no callback set 2024-09-16T03:48:35.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.523+0000 7fbe4bfff640 1 -- 172.21.15.42:0/2077703136 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe4c0a5bc0 con 0x7fbe540a1e20 2024-09-16T03:48:35.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.523+0000 7fbe4bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:48:35.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.523+0000 7fbe4bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:48:35.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.523+0000 7fbe4bfff640 10 monclient: dump: 2024-09-16T03:48:35.540 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:48:35.541 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:48:35.541 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:35.541 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:35.541 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:48:35.541 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:48:35.541 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:48:35.541 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:48:35.541 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:48:35.541 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:48:35.541 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.523+0000 7fbe5b36a640 10 monclient: shutdown 2024-09-16T03:48:35.541 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.523+0000 7fbe5b36a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:48:35.542 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.523+0000 7fbe5b36a640 1 -- 172.21.15.42:0/2077703136 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbe540a1e20 msgr2=0x7fbe5413c890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:48:35.542 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.523+0000 7fbe5b36a640 1 --2- 172.21.15.42:0/2077703136 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbe540a1e20 0x7fbe5413c890 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fbe5406da10 tx=0x7fbe4c01c990 comp rx=0 tx=0).stop 2024-09-16T03:48:35.542 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.523+0000 7fbe5b36a640 1 -- 172.21.15.42:0/2077703136 shutdown_connections 2024-09-16T03:48:35.543 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.523+0000 7fbe5b36a640 1 --2- 172.21.15.42:0/2077703136 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbe540a1e20 0x7fbe5413c890 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:35.543 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.523+0000 7fbe5b36a640 1 --2- 172.21.15.42:0/2077703136 >> v2:172.21.15.154:3300/0 conn(0x7fbe5406dac0 0x7fbe5413a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:35.543 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.523+0000 7fbe5b36a640 1 -- 172.21.15.42:0/2077703136 >> 172.21.15.42:0/2077703136 conn(0x7fbe540a2420 msgr2=0x7fbe540a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:48:35.543 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.523+0000 7fbe5b36a640 1 -- 172.21.15.42:0/2077703136 shutdown_connections 2024-09-16T03:48:35.543 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.523+0000 7fbe5b36a640 1 -- 172.21.15.42:0/2077703136 wait complete. 2024-09-16T03:48:35.543 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.523+0000 7fbe5b36a640 10 monclient: build_initial_monmap 2024-09-16T03:48:35.543 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.523+0000 7fbe5b36a640 10 monclient: monmap: 2024-09-16T03:48:35.543 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:48:35.543 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:48:35.544 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:48:35.544 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:48:35.544 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:48:35.544 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:48:35.544 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:48:35.544 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:48:35.544 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:48:35.544 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:48:35.544 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.523+0000 7fbe5b36a640 1 librados: starting msgr at 2024-09-16T03:48:35.544 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.523+0000 7fbe5b36a640 1 librados: starting objecter 2024-09-16T03:48:35.544 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.523+0000 7fbe5b36a640 1 Processor -- start 2024-09-16T03:48:35.545 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.523+0000 7fbe5b36a640 1 -- start start 2024-09-16T03:48:35.545 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.523+0000 7fbe5b36a640 1 librados: setting wanted keys 2024-09-16T03:48:35.545 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.523+0000 7fbe5b36a640 1 librados: calling monclient init 2024-09-16T03:48:35.545 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.523+0000 7fbe5b36a640 10 monclient: init 2024-09-16T03:48:35.545 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.527+0000 7fbe5b36a640 10 monclient: _reopen_session rank -1 2024-09-16T03:48:35.545 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.527+0000 7fbe5b36a640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T03:48:35.545 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.527+0000 7fbe5b36a640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbe540a1330 0x7fbe54132ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:35.545 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.527+0000 7fbe5b36a640 10 monclient(hunting): picked mon.noname-b con 0x7fbe540a1330 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:48:35.545 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.527+0000 7fbe5b36a640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbe540a1e20 0x7fbe5413b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:35.545 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.527+0000 7fbe5b36a640 10 monclient(hunting): picked mon.noname-c con 0x7fbe540a1e20 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:48:35.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.527+0000 7fbe5b36a640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbe5409cb60 0x7fbe5409cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:35.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.527+0000 7fbe5b36a640 10 monclient(hunting): picked mon.noname-a con 0x7fbe5409cb60 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:48:35.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.527+0000 7fbe5b36a640 10 monclient(hunting): start opening mon connection 2024-09-16T03:48:35.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.527+0000 7fbe5b36a640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fbe5413f800 con 0x7fbe5409cb60 2024-09-16T03:48:35.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.527+0000 7fbe5b36a640 10 monclient(hunting): start opening mon connection 2024-09-16T03:48:35.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.527+0000 7fbe5b36a640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fbe5409ef80 con 0x7fbe540a1330 2024-09-16T03:48:35.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.527+0000 7fbe5b36a640 10 monclient(hunting): start opening mon connection 2024-09-16T03:48:35.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.527+0000 7fbe5b36a640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fbe5413f980 con 0x7fbe540a1e20 2024-09-16T03:48:35.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.527+0000 7fbe5b36a640 10 monclient(hunting): _renew_subs 2024-09-16T03:48:35.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.527+0000 7fbe5b36a640 10 monclient(hunting): authenticate will time out at 1362.496210s 2024-09-16T03:48:35.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.527+0000 7fbe59907640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbe5409cb60 0x7fbe5409cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:48:35.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.527+0000 7fbe59907640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbe5409cb60 0x7fbe5409cfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:44692/0 (socket says 172.21.15.42:44692) 2024-09-16T03:48:35.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.527+0000 7fbe59907640 1 -- 172.21.15.42:0/4228994471 learned_addr learned my addr 172.21.15.42:0/4228994471 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:48:35.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.527+0000 7fbe59907640 10 monclient(hunting): get_auth_request con 0x7fbe5409cb60 auth_method 0 2024-09-16T03:48:35.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.527+0000 7fbe59907640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:48:35.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.527+0000 7fbe59907640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:48:35.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.527+0000 7fbe59907640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:48:35.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.527+0000 7fbe58905640 1 --2- 172.21.15.42:0/4228994471 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbe540a1e20 0x7fbe5413b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:48:35.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.527+0000 7fbe58905640 10 monclient(hunting): get_auth_request con 0x7fbe540a1e20 auth_method 0 2024-09-16T03:48:35.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.527+0000 7fbe58905640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:48:35.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.527+0000 7fbe58905640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:48:35.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.527+0000 7fbe58905640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:48:35.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.527+0000 7fbe59907640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:48:35.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.527+0000 7fbe59907640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:48:35.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.527+0000 7fbe59907640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:48:35.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.527+0000 7fbe59106640 1 --2- 172.21.15.42:0/4228994471 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbe540a1330 0x7fbe54132ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:48:35.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.527+0000 7fbe58905640 1 -- 172.21.15.42:0/4228994471 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbe540a1e20 msgr2=0x7fbe5413b8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-16T03:48:35.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.527+0000 7fbe58905640 1 -- 172.21.15.42:0/4228994471 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbe540a1e20 msgr2=0x7fbe5413b8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T03:48:35.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.527+0000 7fbe58905640 1 --2- 172.21.15.42:0/4228994471 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbe540a1e20 0x7fbe5413b8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:48:35.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.527+0000 7fbe58905640 1 --2- 172.21.15.42:0/4228994471 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbe540a1e20 0x7fbe5413b8a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:48:35.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.527+0000 7fbe59106640 10 monclient(hunting): get_auth_request con 0x7fbe540a1330 auth_method 0 2024-09-16T03:48:35.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.527+0000 7fbe59106640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:48:35.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.527+0000 7fbe59106640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:48:35.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.527+0000 7fbe59106640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:48:35.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.527+0000 7fbe59907640 10 monclient(hunting): handle_auth_done global_id 764478 payload 824 2024-09-16T03:48:35.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.527+0000 7fbe59907640 1 -- 172.21.15.42:0/4228994471 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbe540a1e20 msgr2=0x7fbe5413b8a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:48:35.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.527+0000 7fbe59907640 1 --2- 172.21.15.42:0/4228994471 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbe540a1e20 0x7fbe5413b8a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:35.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.527+0000 7fbe59907640 1 -- 172.21.15.42:0/4228994471 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbe540a1330 msgr2=0x7fbe54132ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:48:35.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.527+0000 7fbe59907640 1 --2- 172.21.15.42:0/4228994471 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbe540a1330 0x7fbe54132ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:35.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.527+0000 7fbe59907640 10 monclient: _finish_hunting 0 2024-09-16T03:48:35.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.527+0000 7fbe59907640 1 monclient: found mon.noname-a 2024-09-16T03:48:35.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.527+0000 7fbe59907640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:48:35.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.527+0000 7fbe59907640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:48:35.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.527+0000 7fbe59907640 1 -- 172.21.15.42:0/4228994471 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe5413bde0 con 0x7fbe5409cb60 2024-09-16T03:48:35.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.527+0000 7fbe59907640 1 --2- 172.21.15.42:0/4228994471 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbe5409cb60 0x7fbe5409cfd0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fbe4c0af040 tx=0x7fbe4c019730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:48:35.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.527+0000 7fbe49ffb640 10 client.?.objecter ms_handle_connect 0x7fbe5409cb60 2024-09-16T03:48:35.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.527+0000 7fbe49ffb640 10 client.?.objecter resend_mon_ops 2024-09-16T03:48:35.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.527+0000 7fbe49ffb640 1 -- 172.21.15.42:0/4228994471 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe4c0bc020 con 0x7fbe5409cb60 2024-09-16T03:48:35.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.527+0000 7fbe49ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:48:35.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.527+0000 7fbe49ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:48:35.551 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.527+0000 7fbe49ffb640 10 monclient: dump: 2024-09-16T03:48:35.551 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:48:35.551 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:48:35.551 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:35.551 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:35.551 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:48:35.551 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:48:35.551 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:48:35.552 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:48:35.552 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:48:35.552 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:48:35.552 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.527+0000 7fbe49ffb640 10 monclient: _finish_auth 0 2024-09-16T03:48:35.552 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.527+0000 7fbe49ffb640 10 monclient: _check_auth_tickets 2024-09-16T03:48:35.552 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.527+0000 7fbe49ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:48:35.552 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.527+0000 7fbe49ffb640 1 -- 172.21.15.42:0/4228994471 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbe4c0b8670 con 0x7fbe5409cb60 2024-09-16T03:48:35.552 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.527+0000 7fbe49ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T03:48:35.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.527+0000 7fbe49ffb640 1 -- 172.21.15.42:0/4228994471 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe4c009510 con 0x7fbe5409cb60 2024-09-16T03:48:35.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.527+0000 7fbe5b36a640 5 monclient: authenticate success, global_id 764478 2024-09-16T03:48:35.556 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.527+0000 7fbe5b36a640 10 monclient: _renew_subs 2024-09-16T03:48:35.556 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.527+0000 7fbe5b36a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:48:35.556 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.527+0000 7fbe5b36a640 1 -- 172.21.15.42:0/4228994471 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe5413a8b0 con 0x7fbe5409cb60 2024-09-16T03:48:35.556 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.527+0000 7fbe4b7fe640 4 set_mon_vals no callback set 2024-09-16T03:48:35.556 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.527+0000 7fbe49ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:48:35.557 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.527+0000 7fbe49ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:48:35.557 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.527+0000 7fbe49ffb640 10 monclient: dump: 2024-09-16T03:48:35.557 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:48:35.557 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:48:35.557 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:35.557 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:35.557 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:48:35.557 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:48:35.557 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:48:35.557 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:48:35.557 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:48:35.557 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:48:35.557 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.527+0000 7fbe5b36a640 10 client.764478.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:48:35.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.527+0000 7fbe5b36a640 10 monclient: _renew_subs 2024-09-16T03:48:35.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.527+0000 7fbe5b36a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:48:35.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.527+0000 7fbe5b36a640 1 -- 172.21.15.42:0/4228994471 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe5413c810 con 0x7fbe5409cb60 2024-09-16T03:48:35.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.527+0000 7fbe5b36a640 1 librados: init done 2024-09-16T03:48:35.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.527+0000 7fbe5b36a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:48:35.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.527+0000 7fbe5b36a640 1 --2- 172.21.15.42:0/4228994471 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbe5406dac0 0x7fbe5402d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:35.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.527+0000 7fbe5b36a640 10 monclient(hunting): start opening mon connection 2024-09-16T03:48:35.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.527+0000 7fbe5b36a640 1 -- 172.21.15.42:0/4228994471 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fbe5409e7f0 con 0x7fbe5406dac0 2024-09-16T03:48:35.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.527+0000 7fbe59106640 1 --2- 172.21.15.42:0/4228994471 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbe5406dac0 0x7fbe5402d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:48:35.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.527+0000 7fbe59106640 10 monclient: get_auth_request con 0x7fbe5406dac0 auth_method 0 2024-09-16T03:48:35.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.527+0000 7fbe59106640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:48:35.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.527+0000 7fbe59106640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:48:35.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.527+0000 7fbe59106640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:48:35.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.527+0000 7fbe59106640 1 -- 172.21.15.42:0/4228994471 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbe5406dac0 msgr2=0x7fbe5402d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-16T03:48:35.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.527+0000 7fbe59106640 1 -- 172.21.15.42:0/4228994471 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbe5406dac0 msgr2=0x7fbe5402d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T03:48:35.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.527+0000 7fbe59106640 1 --2- 172.21.15.42:0/4228994471 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbe5406dac0 0x7fbe5402d7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:48:35.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.527+0000 7fbe59106640 1 --2- 172.21.15.42:0/4228994471 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbe5406dac0 0x7fbe5402d7d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:48:35.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.527+0000 7fbe49ffb640 1 -- 172.21.15.42:0/4228994471 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fbe4c0bf020 con 0x7fbe5409cb60 2024-09-16T03:48:35.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.527+0000 7fbe49ffb640 1 --2- 172.21.15.42:0/4228994471 >> v2:172.21.15.154:6812/254845342 conn(0x7fbe38037810 0x7fbe38039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:35.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.527+0000 7fbe49ffb640 1 -- 172.21.15.42:0/4228994471 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(211..211 src has 1..211) ==== 7042+0+0 (secure 0 0 0) 0x7fbe4c0e4510 con 0x7fbe5409cb60 2024-09-16T03:48:35.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.527+0000 7fbe49ffb640 10 client.764478.objecter ms_dispatch 0x7fbe54000ba0 osd_map(211..211 src has 1..211) 2024-09-16T03:48:35.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.527+0000 7fbe49ffb640 3 client.764478.objecter handle_osd_map got epochs [211,211] > 0 2024-09-16T03:48:35.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.527+0000 7fbe49ffb640 3 client.764478.objecter handle_osd_map decoding full epoch 211 2024-09-16T03:48:35.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.527+0000 7fbe49ffb640 20 client.764478.objecter dump_active .. 0 homeless 2024-09-16T03:48:35.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.527+0000 7fbe58905640 1 --2- 172.21.15.42:0/4228994471 >> v2:172.21.15.154:6812/254845342 conn(0x7fbe38037810 0x7fbe38039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:48:35.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.527+0000 7fbe58905640 10 monclient: get_auth_request con 0x7fbe38037810 auth_method 0 2024-09-16T03:48:35.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.527+0000 7fbe49ffb640 10 client.764478.objecter ms_handle_connect 0x7fbe38037810 2024-09-16T03:48:35.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.527+0000 7fbe58905640 1 --2- 172.21.15.42:0/4228994471 >> v2:172.21.15.154:6812/254845342 conn(0x7fbe38037810 0x7fbe38039cd0 secure :-1 s=READY pgs=3488 cs=0 l=1 rev1=1 crypto rx=0x7fbe54071b80 tx=0x7fbe50008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:48:35.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.731+0000 7fbe59106640 1 --2- 172.21.15.42:0/4228994471 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbe5406dac0 0x7fbe5402d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:48:35.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.731+0000 7fbe59106640 10 monclient: get_auth_request con 0x7fbe5406dac0 auth_method 0 2024-09-16T03:48:35.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.731+0000 7fbe59106640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:48:35.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.731+0000 7fbe59106640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:48:35.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.731+0000 7fbe59106640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:48:35.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.731+0000 7fbe59106640 1 -- 172.21.15.42:0/4228994471 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbe5406dac0 msgr2=0x7fbe5402d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-16T03:48:35.736 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.731+0000 7fbe59106640 1 -- 172.21.15.42:0/4228994471 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbe5406dac0 msgr2=0x7fbe5402d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T03:48:35.736 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.731+0000 7fbe59106640 1 --2- 172.21.15.42:0/4228994471 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbe5406dac0 0x7fbe5402d7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:48:35.736 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:35.731+0000 7fbe59106640 1 --2- 172.21.15.42:0/4228994471 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbe5406dac0 0x7fbe5402d7d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T03:48:36.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:36.135+0000 7fbe59106640 1 --2- 172.21.15.42:0/4228994471 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbe5406dac0 0x7fbe5402d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:48:36.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:36.135+0000 7fbe59106640 10 monclient: get_auth_request con 0x7fbe5406dac0 auth_method 0 2024-09-16T03:48:36.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:36.135+0000 7fbe59106640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:48:36.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:36.135+0000 7fbe59106640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:48:36.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:36.135+0000 7fbe59106640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:48:36.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:36.135+0000 7fbe59106640 1 -- 172.21.15.42:0/4228994471 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbe5406dac0 msgr2=0x7fbe5402d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-16T03:48:36.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:36.135+0000 7fbe59106640 1 -- 172.21.15.42:0/4228994471 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbe5406dac0 msgr2=0x7fbe5402d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T03:48:36.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:36.135+0000 7fbe59106640 1 --2- 172.21.15.42:0/4228994471 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbe5406dac0 0x7fbe5402d7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:48:36.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:36.135+0000 7fbe59106640 1 --2- 172.21.15.42:0/4228994471 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbe5406dac0 0x7fbe5402d7d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T03:48:36.941 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:36.939+0000 7fbe59106640 1 --2- 172.21.15.42:0/4228994471 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbe5406dac0 0x7fbe5402d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:48:36.942 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:36.939+0000 7fbe59106640 10 monclient: get_auth_request con 0x7fbe5406dac0 auth_method 0 2024-09-16T03:48:36.942 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:36.939+0000 7fbe59106640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:48:36.942 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:36.939+0000 7fbe59106640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:48:36.942 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:36.939+0000 7fbe59106640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:48:36.942 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:36.939+0000 7fbe59106640 1 -- 172.21.15.42:0/4228994471 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbe5406dac0 msgr2=0x7fbe5402d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-16T03:48:36.942 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:36.939+0000 7fbe59106640 1 -- 172.21.15.42:0/4228994471 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbe5406dac0 msgr2=0x7fbe5402d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T03:48:36.942 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:36.939+0000 7fbe59106640 1 --2- 172.21.15.42:0/4228994471 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbe5406dac0 0x7fbe5402d7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:48:36.942 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:36.939+0000 7fbe59106640 1 --2- 172.21.15.42:0/4228994471 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbe5406dac0 0x7fbe5402d7d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T03:48:38.529 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.527+0000 7fbe48ff9640 10 monclient: tick 2024-09-16T03:48:38.529 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.527+0000 7fbe48ff9640 10 monclient: _check_auth_tickets 2024-09-16T03:48:38.529 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.527+0000 7fbe48ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:48:38.530 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.527+0000 7fbe48ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:48:38.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.543+0000 7fbe59106640 1 --2- 172.21.15.42:0/4228994471 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbe5406dac0 0x7fbe5402d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:48:38.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.543+0000 7fbe59106640 10 monclient: get_auth_request con 0x7fbe5406dac0 auth_method 0 2024-09-16T03:48:38.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.543+0000 7fbe59106640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:48:38.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.543+0000 7fbe59106640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:48:38.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.543+0000 7fbe59106640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:48:38.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.543+0000 7fbe59106640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:48:38.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.543+0000 7fbe59106640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:48:38.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.543+0000 7fbe59106640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:48:38.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.543+0000 7fbe59106640 10 monclient(hunting): handle_auth_done global_id 784103 payload 824 2024-09-16T03:48:38.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.543+0000 7fbe59106640 1 -- 172.21.15.42:0/4228994471 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbe540a1710 con 0x7fbe5406dac0 2024-09-16T03:48:38.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.543+0000 7fbe59106640 1 --2- 172.21.15.42:0/4228994471 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbe5406dac0 0x7fbe5402d7d0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fbe44015580 tx=0x7fbe44015a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:48:38.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.543+0000 7fbe49ffb640 10 client.764478.objecter ms_handle_connect 0x7fbe5406dac0 2024-09-16T03:48:38.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.543+0000 7fbe49ffb640 10 client.764478.objecter resend_mon_ops 2024-09-16T03:48:38.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.543+0000 7fbe49ffb640 1 -- 172.21.15.42:0/4228994471 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe4401b070 con 0x7fbe5406dac0 2024-09-16T03:48:38.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.543+0000 7fbe49ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:48:38.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.543+0000 7fbe49ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:48:38.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.543+0000 7fbe49ffb640 10 monclient: dump: 2024-09-16T03:48:38.548 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:48:38.548 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:48:38.548 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:38.548 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:38.548 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:48:38.548 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:48:38.548 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:48:38.548 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:48:38.548 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:48:38.549 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:48:38.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.543+0000 7fbe49ffb640 1 -- 172.21.15.42:0/4228994471 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fbe440067e0 con 0x7fbe5406dac0 2024-09-16T03:48:38.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.543+0000 7fbe49ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:48:38.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.543+0000 7fbe49ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:48:38.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.543+0000 7fbe49ffb640 1 -- 172.21.15.42:0/4228994471 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbe5406dac0 msgr2=0x7fbe5402d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:48:38.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.543+0000 7fbe49ffb640 1 --2- 172.21.15.42:0/4228994471 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbe5406dac0 0x7fbe5402d7d0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fbe44015580 tx=0x7fbe44015a40 comp rx=0 tx=0).stop 2024-09-16T03:48:38.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.543+0000 7fbe49ffb640 1 -- 172.21.15.42:0/4228994471 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbe5406dac0 msgr2=0x7fbe5402d7d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:48:38.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.543+0000 7fbe49ffb640 1 --2- 172.21.15.42:0/4228994471 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbe5406dac0 0x7fbe5402d7d0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:38.556 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.551+0000 7fbe5b36a640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:48:38.556 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.551+0000 7fbe5b36a640 1 --2- 172.21.15.42:0/4228994471 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbe5413aaf0 0x7fbe5409d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:38.557 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.551+0000 7fbe5b36a640 10 monclient(hunting): start opening mon connection 2024-09-16T03:48:38.557 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.551+0000 7fbe5b36a640 1 -- 172.21.15.42:0/4228994471 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fbe5413af00 con 0x7fbe5413aaf0 2024-09-16T03:48:38.557 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.551+0000 7fbe59106640 1 --2- 172.21.15.42:0/4228994471 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbe5413aaf0 0x7fbe5409d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:48:38.557 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.551+0000 7fbe59106640 10 monclient: get_auth_request con 0x7fbe5413aaf0 auth_method 0 2024-09-16T03:48:38.557 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.551+0000 7fbe59106640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:48:38.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.551+0000 7fbe59106640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:48:38.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.551+0000 7fbe59106640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:48:38.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.551+0000 7fbe59106640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:48:38.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.551+0000 7fbe59106640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:48:38.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.551+0000 7fbe59106640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:48:38.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.551+0000 7fbe59106640 10 monclient(hunting): handle_auth_done global_id 784106 payload 824 2024-09-16T03:48:38.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.551+0000 7fbe59106640 1 -- 172.21.15.42:0/4228994471 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fbe54071610 con 0x7fbe5413aaf0 2024-09-16T03:48:38.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.551+0000 7fbe59106640 1 --2- 172.21.15.42:0/4228994471 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbe5413aaf0 0x7fbe5409d510 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fbe44003fd0 tx=0x7fbe44003770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:48:38.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.551+0000 7fbe49ffb640 10 client.764478.objecter ms_handle_connect 0x7fbe5413aaf0 2024-09-16T03:48:38.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.551+0000 7fbe49ffb640 10 client.764478.objecter resend_mon_ops 2024-09-16T03:48:38.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.555+0000 7fbe49ffb640 1 -- 172.21.15.42:0/4228994471 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe44013980 con 0x7fbe5413aaf0 2024-09-16T03:48:38.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.555+0000 7fbe49ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:48:38.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.555+0000 7fbe49ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:48:38.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.555+0000 7fbe49ffb640 10 monclient: dump: 2024-09-16T03:48:38.562 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:48:38.562 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:48:38.562 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:38.562 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:38.562 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:48:38.562 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:48:38.563 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:48:38.563 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:48:38.563 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:48:38.563 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:48:38.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.555+0000 7fbe49ffb640 1 -- 172.21.15.42:0/4228994471 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4544 (secure 0 0 0) 0x7fbe44013960 con 0x7fbe5413aaf0 2024-09-16T03:48:38.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.555+0000 7fbe49ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:48:38.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.555+0000 7fbe49ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:48:38.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.555+0000 7fbe49ffb640 1 -- 172.21.15.42:0/4228994471 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbe5413aaf0 msgr2=0x7fbe5409d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:48:38.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.555+0000 7fbe49ffb640 1 --2- 172.21.15.42:0/4228994471 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbe5413aaf0 0x7fbe5409d510 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fbe44003fd0 tx=0x7fbe44003770 comp rx=0 tx=0).stop 2024-09-16T03:48:38.565 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.555+0000 7fbe49ffb640 1 -- 172.21.15.42:0/4228994471 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbe5413aaf0 msgr2=0x7fbe5409d510 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:48:38.565 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.555+0000 7fbe49ffb640 1 --2- 172.21.15.42:0/4228994471 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbe5413aaf0 0x7fbe5409d510 secure :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fbe44003fd0 tx=0x7fbe44003770 comp rx=0 tx=0).stop 2024-09-16T03:48:38.565 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:48:38.566 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:48:38.566 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:48:38.566 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T03:48:38.566 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 60, 2024-09-16T03:48:38.566 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 10455, 2024-09-16T03:48:38.567 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:48:38.567 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:48:38.567 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T03:48:38.567 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T03:48:38.567 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:48:38.568 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 5, 2024-09-16T03:48:38.568 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:48:38.568 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:48:38.568 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:48:38.568 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:48:38.569 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:48:38.569 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:48:38.569 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:48:38.569 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:48:38.569 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:48:38.569 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:48:38.570 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:48:38.570 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:48:38.570 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:48:38.570 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:48:38.570 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:48:38.571 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:48:38.571 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:48:38.571 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:48:38.571 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:48:38.571 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:48:38.571 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:48:38.572 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:48:38.572 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:48:38.572 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:48:38.572 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:48:38.572 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:48:38.573 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:48:38.573 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:48:38.573 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:48:38.573 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:48:38.573 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:48:38.574 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:48:38.574 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:48:38.574 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:48:38.574 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:48:38.574 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:48:38.575 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:48:38.575 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:48:38.575 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:48:38.575 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:48:38.575 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:48:38.576 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:48:38.576 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:48:38.576 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:48:38.576 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:48:38.576 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:48:38.577 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:48:38.577 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:48:38.577 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:48:38.577 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:48:38.577 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:48:38.577 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:48:38.578 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:48:38.578 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:48:38.578 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:48:38.578 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:48:38.578 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:48:38.579 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:48:38.579 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:48:38.579 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:48:38.579 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:48:38.579 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:48:38.580 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:48:38.580 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:48:38.580 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:48:38.580 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:48:38.580 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:48:38.581 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:48:38.581 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:48:38.581 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:48:38.581 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:48:38.581 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:48:38.582 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:48:38.582 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:48:38.582 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:48:38.582 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:48:38.583 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:48:38.583 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:48:38.583 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:48:38.583 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:48:38.584 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:48:38.584 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:48:38.584 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:48:38.584 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:48:38.584 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:48:38.585 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:48:38.585 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:48:38.585 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:48:38.585 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:48:38.585 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:48:38.586 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:48:38.586 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:48:38.586 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:48:38.586 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:48:38.586 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:48:38.586 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:48:38.587 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:48:38.587 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:48:38.587 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:48:38.587 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:48:38.587 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:48:38.588 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:48:38.588 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:48:38.588 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:48:38.588 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:48:38.588 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:48:38.588 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:48:38.589 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:48:38.589 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:48:38.589 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:48:38.589 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:48:38.589 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:48:38.590 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:48:38.590 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:48:38.590 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:48:38.590 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:48:38.590 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:48:38.591 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:48:38.591 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:48:38.591 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:48:38.591 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:48:38.591 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:48:38.592 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:48:38.592 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:48:38.592 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:48:38.592 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:48:38.592 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:48:38.593 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:48:38.593 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:48:38.593 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:48:38.593 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:48:38.593 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:48:38.593 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:48:38.594 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:48:38.594 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:48:38.594 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:48:38.594 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:48:38.594 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:48:38.595 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:48:38.595 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:48:38.595 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:48:38.595 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:48:38.595 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:48:38.595 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:48:38.596 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:48:38.596 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:48:38.596 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:48:38.596 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:48:38.596 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:48:38.597 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:48:38.597 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:48:38.600 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.555+0000 7fbe5b36a640 10 librados: watch_flush enter 2024-09-16T03:48:38.600 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.555+0000 7fbe5b36a640 10 librados: watch_flush exit 2024-09-16T03:48:38.601 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.555+0000 7fbe5b36a640 20 client.764478.objecter shutdown clearing up homeless session... 2024-09-16T03:48:38.601 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.555+0000 7fbe5b36a640 10 client.764478.objecter successfully canceled tick 2024-09-16T03:48:38.601 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.555+0000 7fbe5b36a640 1 -- 172.21.15.42:0/4228994471 >> v2:172.21.15.154:6812/254845342 conn(0x7fbe38037810 msgr2=0x7fbe38039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:48:38.601 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.555+0000 7fbe5b36a640 1 --2- 172.21.15.42:0/4228994471 >> v2:172.21.15.154:6812/254845342 conn(0x7fbe38037810 0x7fbe38039cd0 secure :-1 s=READY pgs=3488 cs=0 l=1 rev1=1 crypto rx=0x7fbe54071b80 tx=0x7fbe50008040 comp rx=0 tx=0).stop 2024-09-16T03:48:38.601 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.555+0000 7fbe5b36a640 10 monclient: shutdown 2024-09-16T03:48:38.602 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.555+0000 7fbe5b36a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:48:38.602 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.555+0000 7fbe5b36a640 1 -- 172.21.15.42:0/4228994471 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbe5409cb60 msgr2=0x7fbe5409cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:48:38.602 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.555+0000 7fbe5b36a640 1 --2- 172.21.15.42:0/4228994471 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbe5409cb60 0x7fbe5409cfd0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fbe4c0af040 tx=0x7fbe4c019730 comp rx=0 tx=0).stop 2024-09-16T03:48:38.602 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.555+0000 7fbe59907640 1 -- 172.21.15.42:0/4228994471 reap_dead start 2024-09-16T03:48:38.602 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.555+0000 7fbe59907640 1 -- 172.21.15.42:0/4228994471 reap_dead start 2024-09-16T03:48:38.603 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.555+0000 7fbe5b36a640 1 -- 172.21.15.42:0/4228994471 shutdown_connections 2024-09-16T03:48:38.603 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.555+0000 7fbe5b36a640 1 -- 172.21.15.42:0/4228994471 >> 172.21.15.42:0/4228994471 conn(0x7fbe540a2420 msgr2=0x7fbe54135300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:48:38.603 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.555+0000 7fbe5b36a640 1 -- 172.21.15.42:0/4228994471 shutdown_connections 2024-09-16T03:48:38.603 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.555+0000 7fbe5b36a640 1 -- 172.21.15.42:0/4228994471 wait complete. 2024-09-16T03:48:38.603 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.555+0000 7fbe5b36a640 1 librados: shutdown 2024-09-16T03:48:38.604 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:48:38.604 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T03:48:38.604 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T03:48:38.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.727+0000 7f9d97ee0640 10 monclient: get_monmap_and_config 2024-09-16T03:48:38.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.727+0000 7f9d97ee0640 10 monclient: build_initial_monmap 2024-09-16T03:48:38.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.727+0000 7f9d97ee0640 10 monclient: monmap: 2024-09-16T03:48:38.730 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:48:38.730 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:48:38.730 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:48:38.729310+0000 2024-09-16T03:48:38.730 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:48:38.729310+0000 2024-09-16T03:48:38.731 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:48:38.731 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:48:38.731 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:48:38.731 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:48:38.731 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:48:38.731 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:48:38.731 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:48:38.731 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:48:38.733 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.727+0000 7f9d97ee0640 1 Processor -- start 2024-09-16T03:48:38.733 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.727+0000 7f9d97ee0640 1 -- start start 2024-09-16T03:48:38.733 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.727+0000 7f9d97ee0640 10 monclient: init 2024-09-16T03:48:38.733 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.727+0000 7f9d97ee0640 10 monclient: _reopen_session rank -1 2024-09-16T03:48:38.733 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.727+0000 7f9d97ee0640 10 monclient: _add_conns ranks=[3,4,2,0,1] 2024-09-16T03:48:38.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.727+0000 7f9d97ee0640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f9d900a1330 0x7f9d900a1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:38.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.727+0000 7f9d97ee0640 10 monclient(hunting): picked mon.noname-a con 0x7f9d900a1330 addr v2:172.21.15.42:3300/0 2024-09-16T03:48:38.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.727+0000 7f9d97ee0640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f9d9006dac0 0x7f9d900a1c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:38.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.727+0000 7f9d97ee0640 10 monclient(hunting): picked mon.noname-c con 0x7f9d9006dac0 addr v2:172.21.15.154:3300/0 2024-09-16T03:48:38.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.727+0000 7f9d97ee0640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9d90135100 0x7f9d90135550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:38.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.727+0000 7f9d97ee0640 10 monclient(hunting): picked mon.noname-b con 0x7f9d90135100 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:48:38.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.727+0000 7f9d97ee0640 10 monclient(hunting): start opening mon connection 2024-09-16T03:48:38.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.727+0000 7f9d97ee0640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f9d9009c7d0 con 0x7f9d900a1330 2024-09-16T03:48:38.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.727+0000 7f9d97ee0640 10 monclient(hunting): start opening mon connection 2024-09-16T03:48:38.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.727+0000 7f9d97ee0640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f9d9009c090 con 0x7f9d9006dac0 2024-09-16T03:48:38.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.727+0000 7f9d97ee0640 10 monclient(hunting): start opening mon connection 2024-09-16T03:48:38.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.727+0000 7f9d97ee0640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f9d9009f610 con 0x7f9d90135100 2024-09-16T03:48:38.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.727+0000 7f9d97ee0640 10 monclient(hunting): _renew_subs 2024-09-16T03:48:38.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.727+0000 7f9d97ee0640 10 monclient(hunting): authenticate will time out at 1365.698801s 2024-09-16T03:48:38.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.727+0000 7f9d95c7c640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f9d900a1330 0x7f9d900a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:48:38.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.727+0000 7f9d9647d640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9d90135100 0x7f9d90135550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:48:38.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.727+0000 7f9d95c7c640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f9d900a1330 0x7f9d900a1710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:44708/0 (socket says 172.21.15.42:44708) 2024-09-16T03:48:38.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.727+0000 7f9d95c7c640 1 -- 172.21.15.42:0/312914270 learned_addr learned my addr 172.21.15.42:0/312914270 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:48:38.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.727+0000 7f9d95c7c640 10 monclient(hunting): get_auth_request con 0x7f9d900a1330 auth_method 0 2024-09-16T03:48:38.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.727+0000 7f9d95c7c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:48:38.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.727+0000 7f9d95c7c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:48:38.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.727+0000 7f9d95c7c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:48:38.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.727+0000 7f9d9647d640 10 monclient(hunting): get_auth_request con 0x7f9d90135100 auth_method 0 2024-09-16T03:48:38.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.727+0000 7f9d9647d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:48:38.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.727+0000 7f9d9647d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:48:38.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.727+0000 7f9d9647d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:48:38.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.727+0000 7f9d95c7c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:48:38.736 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.727+0000 7f9d95c7c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:48:38.736 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.727+0000 7f9d95c7c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:48:38.736 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.727+0000 7f9d9647d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:48:38.736 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.727+0000 7f9d9647d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:48:38.736 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.727+0000 7f9d9647d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:48:38.736 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.727+0000 7f9d9547b640 1 --2- 172.21.15.42:0/312914270 >> v2:172.21.15.154:3300/0 conn(0x7f9d9006dac0 0x7f9d900a1c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:48:38.736 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.727+0000 7f9d95c7c640 10 monclient(hunting): handle_auth_done global_id 764490 payload 274 2024-09-16T03:48:38.736 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.727+0000 7f9d95c7c640 1 -- 172.21.15.42:0/312914270 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9d90135100 msgr2=0x7f9d90135550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:48:38.736 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.727+0000 7f9d95c7c640 1 --2- 172.21.15.42:0/312914270 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9d90135100 0x7f9d90135550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:38.736 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.727+0000 7f9d95c7c640 1 -- 172.21.15.42:0/312914270 >> v2:172.21.15.154:3300/0 conn(0x7f9d9006dac0 msgr2=0x7f9d900a1c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:48:38.736 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.727+0000 7f9d95c7c640 1 --2- 172.21.15.42:0/312914270 >> v2:172.21.15.154:3300/0 conn(0x7f9d9006dac0 0x7f9d900a1c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:38.736 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.727+0000 7f9d95c7c640 10 monclient: _finish_hunting 0 2024-09-16T03:48:38.736 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.727+0000 7f9d95c7c640 1 monclient: found mon.noname-a 2024-09-16T03:48:38.736 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.727+0000 7f9d95c7c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:48:38.737 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.727+0000 7f9d95c7c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:48:38.737 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.727+0000 7f9d95c7c640 1 -- 172.21.15.42:0/312914270 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d9009f8f0 con 0x7f9d900a1330 2024-09-16T03:48:38.737 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.727+0000 7f9d9547b640 10 monclient: get_auth_request con 0x7f9d9006dac0 auth_method 0 2024-09-16T03:48:38.737 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.727+0000 7f9d9547b640 1 --2- 172.21.15.42:0/312914270 >> v2:172.21.15.154:3300/0 conn(0x7f9d9006dac0 0x7f9d900a1c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T03:48:38.737 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.727+0000 7f9d9647d640 1 --2- 172.21.15.42:0/312914270 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9d90135100 0x7f9d90135550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:48:38.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.731+0000 7f9d95c7c640 1 --2- 172.21.15.42:0/312914270 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9d900a1330 0x7f9d900a1710 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f9d9006da10 tx=0x7f9d8c01c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=faad3c371be7a1d0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:48:38.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.731+0000 7f9d94c7a640 1 -- 172.21.15.42:0/312914270 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d8c0b1070 con 0x7f9d900a1330 2024-09-16T03:48:38.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.731+0000 7f9d94c7a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:48:38.744 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.731+0000 7f9d94c7a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:48:38.744 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.731+0000 7f9d94c7a640 10 monclient: dump: 2024-09-16T03:48:38.744 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:48:38.744 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:48:38.744 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:38.744 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:38.744 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:48:38.744 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:48:38.744 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:48:38.744 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:48:38.744 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:48:38.744 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:48:38.744 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.731+0000 7f9d94c7a640 10 monclient: _finish_auth 0 2024-09-16T03:48:38.744 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.731+0000 7f9d94c7a640 10 monclient: _check_auth_tickets 2024-09-16T03:48:38.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.731+0000 7f9d94c7a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:48:38.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.731+0000 7f9d94c7a640 1 -- 172.21.15.42:0/312914270 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9d8c0ac620 con 0x7f9d900a1330 2024-09-16T03:48:38.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.731+0000 7f9d97ee0640 5 monclient: authenticate success, global_id 764490 2024-09-16T03:48:38.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.731+0000 7f9d97ee0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:48:38.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.731+0000 7f9d94c7a640 10 monclient: handle_config config(0 keys) 2024-09-16T03:48:38.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.731+0000 7f9d97ee0640 10 monclient: get_monmap_and_config success 2024-09-16T03:48:38.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.731+0000 7f9d97ee0640 4 set_mon_vals no callback set 2024-09-16T03:48:38.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.731+0000 7f9d94c7a640 1 -- 172.21.15.42:0/312914270 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d8c0ac9f0 con 0x7f9d900a1330 2024-09-16T03:48:38.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.731+0000 7f9d94c7a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:48:38.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.731+0000 7f9d94c7a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:48:38.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.731+0000 7f9d94c7a640 10 monclient: dump: 2024-09-16T03:48:38.745 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:48:38.745 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:48:38.746 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:38.746 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:38.746 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:48:38.746 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:48:38.746 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:48:38.746 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:48:38.746 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:48:38.746 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:48:38.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.731+0000 7f9d97ee0640 10 monclient: shutdown 2024-09-16T03:48:38.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.731+0000 7f9d97ee0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:48:38.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.731+0000 7f9d97ee0640 1 -- 172.21.15.42:0/312914270 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9d900a1330 msgr2=0x7f9d900a1710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:48:38.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.731+0000 7f9d97ee0640 1 --2- 172.21.15.42:0/312914270 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9d900a1330 0x7f9d900a1710 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f9d9006da10 tx=0x7f9d8c01c440 comp rx=0 tx=0).stop 2024-09-16T03:48:38.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.731+0000 7f9d97ee0640 1 -- 172.21.15.42:0/312914270 shutdown_connections 2024-09-16T03:48:38.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.731+0000 7f9d97ee0640 1 --2- 172.21.15.42:0/312914270 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9d90135100 0x7f9d90135550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:38.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.731+0000 7f9d97ee0640 1 --2- 172.21.15.42:0/312914270 >> v2:172.21.15.154:3300/0 conn(0x7f9d9006dac0 0x7f9d900a1c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:38.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.731+0000 7f9d97ee0640 1 --2- 172.21.15.42:0/312914270 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9d900a1330 0x7f9d900a1710 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:38.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.731+0000 7f9d97ee0640 1 -- 172.21.15.42:0/312914270 >> 172.21.15.42:0/312914270 conn(0x7f9d900a2420 msgr2=0x7f9d900a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:48:38.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.731+0000 7f9d97ee0640 1 -- 172.21.15.42:0/312914270 shutdown_connections 2024-09-16T03:48:38.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.731+0000 7f9d97ee0640 1 -- 172.21.15.42:0/312914270 wait complete. 2024-09-16T03:48:38.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.731+0000 7f9d97ee0640 10 monclient: build_initial_monmap 2024-09-16T03:48:38.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.731+0000 7f9d97ee0640 10 monclient: monmap: 2024-09-16T03:48:38.747 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:48:38.747 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:48:38.747 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:48:38.747 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:48:38.747 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:48:38.748 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:48:38.748 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:48:38.748 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:48:38.748 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:48:38.748 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:48:38.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.731+0000 7f9d97ee0640 1 librados: starting msgr at 2024-09-16T03:48:38.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.731+0000 7f9d97ee0640 1 librados: starting objecter 2024-09-16T03:48:38.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.731+0000 7f9d97ee0640 1 Processor -- start 2024-09-16T03:48:38.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.731+0000 7f9d97ee0640 1 -- start start 2024-09-16T03:48:38.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.731+0000 7f9d97ee0640 1 librados: setting wanted keys 2024-09-16T03:48:38.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.731+0000 7f9d97ee0640 1 librados: calling monclient init 2024-09-16T03:48:38.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.731+0000 7f9d97ee0640 10 monclient: init 2024-09-16T03:48:38.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.731+0000 7f9d97ee0640 10 monclient: _reopen_session rank -1 2024-09-16T03:48:38.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.731+0000 7f9d97ee0640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T03:48:38.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.731+0000 7f9d97ee0640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9d900a1330 0x7f9d901cef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:38.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.731+0000 7f9d97ee0640 10 monclient(hunting): picked mon.noname-b con 0x7f9d900a1330 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:48:38.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.731+0000 7f9d97ee0640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9d90135100 0x7f9d90134590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:38.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.731+0000 7f9d97ee0640 10 monclient(hunting): picked mon.noname-c con 0x7f9d90135100 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:48:38.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.735+0000 7f9d97ee0640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9d90134ad0 0x7f9d9009cb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:38.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.735+0000 7f9d97ee0640 10 monclient(hunting): picked mon.noname-a con 0x7f9d90134ad0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:48:38.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.735+0000 7f9d97ee0640 10 monclient(hunting): start opening mon connection 2024-09-16T03:48:38.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.735+0000 7f9d97ee0640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f9d9009eef0 con 0x7f9d90134ad0 2024-09-16T03:48:38.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.735+0000 7f9d97ee0640 10 monclient(hunting): start opening mon connection 2024-09-16T03:48:38.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.735+0000 7f9d97ee0640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f9d9009ed70 con 0x7f9d900a1330 2024-09-16T03:48:38.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.735+0000 7f9d97ee0640 10 monclient(hunting): start opening mon connection 2024-09-16T03:48:38.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.735+0000 7f9d97ee0640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f9d9013eb70 con 0x7f9d90135100 2024-09-16T03:48:38.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.735+0000 7f9d97ee0640 10 monclient(hunting): _renew_subs 2024-09-16T03:48:38.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.735+0000 7f9d97ee0640 10 monclient(hunting): authenticate will time out at 1365.704039s 2024-09-16T03:48:38.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.735+0000 7f9d9547b640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9d90135100 0x7f9d90134590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:48:38.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.735+0000 7f9d9647d640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9d90134ad0 0x7f9d9009cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:48:38.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.735+0000 7f9d9547b640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9d90135100 0x7f9d90134590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:38056/0 (socket says 172.21.15.42:38056) 2024-09-16T03:48:38.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.735+0000 7f9d9547b640 1 -- 172.21.15.42:0/529828030 learned_addr learned my addr 172.21.15.42:0/529828030 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:48:38.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.735+0000 7f9d9647d640 10 monclient(hunting): get_auth_request con 0x7f9d90134ad0 auth_method 0 2024-09-16T03:48:38.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.735+0000 7f9d9647d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:48:38.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.735+0000 7f9d9647d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:48:38.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.735+0000 7f9d9647d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:48:38.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.735+0000 7f9d9547b640 10 monclient(hunting): get_auth_request con 0x7f9d90135100 auth_method 0 2024-09-16T03:48:38.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.735+0000 7f9d9547b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:48:38.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.735+0000 7f9d9547b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:48:38.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.735+0000 7f9d9547b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:48:38.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.735+0000 7f9d9647d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:48:38.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.735+0000 7f9d9647d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:48:38.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.735+0000 7f9d9647d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:48:38.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.735+0000 7f9d9547b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:48:38.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.735+0000 7f9d9547b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:48:38.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.735+0000 7f9d9547b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:48:38.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.735+0000 7f9d95c7c640 1 --2- 172.21.15.42:0/529828030 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9d900a1330 0x7f9d901cef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:48:38.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.735+0000 7f9d95c7c640 10 monclient(hunting): get_auth_request con 0x7f9d900a1330 auth_method 0 2024-09-16T03:48:38.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.735+0000 7f9d95c7c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:48:38.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.735+0000 7f9d95c7c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:48:38.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.735+0000 7f9d95c7c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:48:38.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.735+0000 7f9d9547b640 10 monclient(hunting): handle_auth_done global_id 784112 payload 824 2024-09-16T03:48:38.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.735+0000 7f9d9547b640 1 -- 172.21.15.42:0/529828030 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9d900a1330 msgr2=0x7f9d901cef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:48:38.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.735+0000 7f9d9547b640 1 --2- 172.21.15.42:0/529828030 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9d900a1330 0x7f9d901cef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:38.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.735+0000 7f9d9547b640 1 -- 172.21.15.42:0/529828030 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9d90134ad0 msgr2=0x7f9d9009cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:48:38.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.735+0000 7f9d9547b640 1 --2- 172.21.15.42:0/529828030 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9d90134ad0 0x7f9d9009cb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:38.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.735+0000 7f9d9547b640 10 monclient: _finish_hunting 0 2024-09-16T03:48:38.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.735+0000 7f9d9547b640 1 monclient: found mon.noname-c 2024-09-16T03:48:38.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.735+0000 7f9d9547b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:48:38.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.735+0000 7f9d9547b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:48:38.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.735+0000 7f9d9547b640 1 -- 172.21.15.42:0/529828030 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d90071b40 con 0x7f9d90135100 2024-09-16T03:48:38.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.735+0000 7f9d9647d640 1 --2- 172.21.15.42:0/529828030 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9d90134ad0 0x7f9d9009cb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:48:38.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.735+0000 7f9d9547b640 1 --2- 172.21.15.42:0/529828030 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9d90135100 0x7f9d90134590 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f9d8000c7f0 tx=0x7f9d8000cbf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:48:38.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.735+0000 7f9d867fc640 10 client.?.objecter ms_handle_connect 0x7f9d90135100 2024-09-16T03:48:38.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.735+0000 7f9d867fc640 10 client.?.objecter resend_mon_ops 2024-09-16T03:48:38.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.735+0000 7f9d867fc640 1 -- 172.21.15.42:0/529828030 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d80007820 con 0x7f9d90135100 2024-09-16T03:48:38.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.735+0000 7f9d867fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:48:38.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.735+0000 7f9d867fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:48:38.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.735+0000 7f9d867fc640 10 monclient: dump: 2024-09-16T03:48:38.753 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:48:38.753 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:48:38.753 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:38.753 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:38.754 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:48:38.754 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:48:38.754 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:48:38.754 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:48:38.754 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:48:38.754 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:48:38.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.735+0000 7f9d867fc640 10 monclient: _finish_auth 0 2024-09-16T03:48:38.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.735+0000 7f9d867fc640 10 monclient: _check_auth_tickets 2024-09-16T03:48:38.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.735+0000 7f9d867fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:48:38.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.735+0000 7f9d867fc640 1 -- 172.21.15.42:0/529828030 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9d800079c0 con 0x7f9d90135100 2024-09-16T03:48:38.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.735+0000 7f9d97ee0640 5 monclient: authenticate success, global_id 784112 2024-09-16T03:48:38.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.735+0000 7f9d97ee0640 10 monclient: _renew_subs 2024-09-16T03:48:38.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.735+0000 7f9d97ee0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:48:38.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.735+0000 7f9d97ee0640 1 -- 172.21.15.42:0/529828030 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d900a51e0 con 0x7f9d90135100 2024-09-16T03:48:38.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.735+0000 7f9d867fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:48:38.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.735+0000 7f9d867fc640 1 -- 172.21.15.42:0/529828030 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d80007db0 con 0x7f9d90135100 2024-09-16T03:48:38.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.735+0000 7f9d867fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:48:38.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.735+0000 7f9d867fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:48:38.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.735+0000 7f9d867fc640 10 monclient: dump: 2024-09-16T03:48:38.755 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:48:38.755 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:48:38.755 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:38.755 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:38.755 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:48:38.755 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:48:38.755 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:48:38.755 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:48:38.755 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:48:38.755 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:48:38.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.735+0000 7f9d87fff640 4 set_mon_vals no callback set 2024-09-16T03:48:38.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.735+0000 7f9d97ee0640 10 client.784112.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:48:38.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.735+0000 7f9d97ee0640 10 monclient: _renew_subs 2024-09-16T03:48:38.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.735+0000 7f9d97ee0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:48:38.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.735+0000 7f9d97ee0640 1 -- 172.21.15.42:0/529828030 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d901216b0 con 0x7f9d90135100 2024-09-16T03:48:38.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.735+0000 7f9d97ee0640 1 librados: init done 2024-09-16T03:48:38.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.735+0000 7f9d97ee0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:48:38.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.735+0000 7f9d97ee0640 1 --2- 172.21.15.42:0/529828030 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9d9006dac0 0x7f9d9009d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:38.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.735+0000 7f9d97ee0640 10 monclient(hunting): start opening mon connection 2024-09-16T03:48:38.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.735+0000 7f9d97ee0640 1 -- 172.21.15.42:0/529828030 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f9d901208d0 con 0x7f9d9006dac0 2024-09-16T03:48:38.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.735+0000 7f9d867fc640 1 -- 172.21.15.42:0/529828030 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f9d800113f0 con 0x7f9d90135100 2024-09-16T03:48:38.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.735+0000 7f9d95c7c640 1 --2- 172.21.15.42:0/529828030 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9d9006dac0 0x7f9d9009d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:48:38.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.735+0000 7f9d867fc640 1 --2- 172.21.15.42:0/529828030 >> v2:172.21.15.154:6812/254845342 conn(0x7f9d74037810 0x7f9d74039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:38.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.735+0000 7f9d95c7c640 10 monclient: get_auth_request con 0x7f9d9006dac0 auth_method 0 2024-09-16T03:48:38.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.735+0000 7f9d95c7c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:48:38.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.735+0000 7f9d95c7c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:48:38.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.735+0000 7f9d95c7c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:48:38.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.735+0000 7f9d867fc640 1 -- 172.21.15.42:0/529828030 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(214..214 src has 1..214) ==== 6058+0+0 (secure 0 0 0) 0x7f9d800493b0 con 0x7f9d90135100 2024-09-16T03:48:38.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.735+0000 7f9d867fc640 10 client.784112.objecter ms_dispatch 0x7f9d90000ba0 osd_map(214..214 src has 1..214) 2024-09-16T03:48:38.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.735+0000 7f9d867fc640 3 client.784112.objecter handle_osd_map got epochs [214,214] > 0 2024-09-16T03:48:38.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.735+0000 7f9d867fc640 3 client.784112.objecter handle_osd_map decoding full epoch 214 2024-09-16T03:48:38.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.735+0000 7f9d95c7c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:48:38.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.735+0000 7f9d95c7c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:48:38.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.735+0000 7f9d867fc640 20 client.784112.objecter dump_active .. 0 homeless 2024-09-16T03:48:38.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.735+0000 7f9d95c7c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:48:38.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.735+0000 7f9d9647d640 1 --2- 172.21.15.42:0/529828030 >> v2:172.21.15.154:6812/254845342 conn(0x7f9d74037810 0x7f9d74039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:48:38.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.735+0000 7f9d9647d640 10 monclient: get_auth_request con 0x7f9d74037810 auth_method 0 2024-09-16T03:48:38.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.735+0000 7f9d95c7c640 10 monclient(hunting): handle_auth_done global_id 774292 payload 824 2024-09-16T03:48:38.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.735+0000 7f9d95c7c640 1 -- 172.21.15.42:0/529828030 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9d90135550 con 0x7f9d9006dac0 2024-09-16T03:48:38.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.735+0000 7f9d867fc640 10 client.784112.objecter ms_handle_connect 0x7f9d9006dac0 2024-09-16T03:48:38.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.735+0000 7f9d867fc640 10 client.784112.objecter resend_mon_ops 2024-09-16T03:48:38.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.735+0000 7f9d95c7c640 1 --2- 172.21.15.42:0/529828030 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9d9006dac0 0x7f9d9009d0a0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f9d9006add0 tx=0x7f9d8c01ac80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:48:38.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.735+0000 7f9d867fc640 10 client.784112.objecter ms_handle_connect 0x7f9d74037810 2024-09-16T03:48:38.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.735+0000 7f9d9647d640 1 --2- 172.21.15.42:0/529828030 >> v2:172.21.15.154:6812/254845342 conn(0x7f9d74037810 0x7f9d74039cd0 secure :-1 s=READY pgs=3491 cs=0 l=1 rev1=1 crypto rx=0x7f9d88003c10 tx=0x7f9d8800b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:48:38.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.735+0000 7f9d867fc640 1 -- 172.21.15.42:0/529828030 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d8c0b1040 con 0x7f9d9006dac0 2024-09-16T03:48:38.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.735+0000 7f9d867fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:48:38.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.735+0000 7f9d867fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:48:38.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.735+0000 7f9d867fc640 10 monclient: dump: 2024-09-16T03:48:38.758 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:48:38.759 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:48:38.759 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:38.759 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:38.759 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:48:38.759 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:48:38.759 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:48:38.759 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:48:38.759 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:48:38.759 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:48:38.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.739+0000 7f9d867fc640 1 -- 172.21.15.42:0/529828030 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f9d8c009490 con 0x7f9d9006dac0 2024-09-16T03:48:38.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.739+0000 7f9d867fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:48:38.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.739+0000 7f9d867fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:48:38.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.739+0000 7f9d867fc640 1 -- 172.21.15.42:0/529828030 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9d9006dac0 msgr2=0x7f9d9009d0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:48:38.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.739+0000 7f9d867fc640 1 --2- 172.21.15.42:0/529828030 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9d9006dac0 0x7f9d9009d0a0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f9d9006add0 tx=0x7f9d8c01ac80 comp rx=0 tx=0).stop 2024-09-16T03:48:38.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.739+0000 7f9d867fc640 1 -- 172.21.15.42:0/529828030 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9d9006dac0 msgr2=0x7f9d9009d0a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:48:38.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.739+0000 7f9d867fc640 1 --2- 172.21.15.42:0/529828030 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9d9006dac0 0x7f9d9009d0a0 secure :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f9d9006add0 tx=0x7f9d8c01ac80 comp rx=0 tx=0).stop 2024-09-16T03:48:38.761 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.743+0000 7f9d97ee0640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:48:38.761 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.743+0000 7f9d97ee0640 1 --2- 172.21.15.42:0/529828030 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9d9009d5e0 0x7f9d90137330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:38.761 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.743+0000 7f9d97ee0640 10 monclient(hunting): start opening mon connection 2024-09-16T03:48:38.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.743+0000 7f9d97ee0640 1 -- 172.21.15.42:0/529828030 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f9d9009d9f0 con 0x7f9d9009d5e0 2024-09-16T03:48:38.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.743+0000 7f9d95c7c640 1 --2- 172.21.15.42:0/529828030 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9d9009d5e0 0x7f9d90137330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:48:38.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.743+0000 7f9d95c7c640 10 monclient: get_auth_request con 0x7f9d9009d5e0 auth_method 0 2024-09-16T03:48:38.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.743+0000 7f9d95c7c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:48:38.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.743+0000 7f9d95c7c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:48:38.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.743+0000 7f9d95c7c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:48:38.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.743+0000 7f9d95c7c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:48:38.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.743+0000 7f9d95c7c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:48:38.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.743+0000 7f9d95c7c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:48:38.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.743+0000 7f9d95c7c640 10 monclient(hunting): handle_auth_done global_id 774295 payload 824 2024-09-16T03:48:38.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.743+0000 7f9d95c7c640 1 -- 172.21.15.42:0/529828030 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f9d90121380 con 0x7f9d9009d5e0 2024-09-16T03:48:38.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.747+0000 7f9d95c7c640 1 --2- 172.21.15.42:0/529828030 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9d9009d5e0 0x7f9d90137330 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f9d8c01cce0 tx=0x7f9d8c0191e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:48:38.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.747+0000 7f9d867fc640 10 client.784112.objecter ms_handle_connect 0x7f9d9009d5e0 2024-09-16T03:48:38.763 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.747+0000 7f9d867fc640 10 client.784112.objecter resend_mon_ops 2024-09-16T03:48:38.763 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.747+0000 7f9d867fc640 1 -- 172.21.15.42:0/529828030 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d8c0b1040 con 0x7f9d9009d5e0 2024-09-16T03:48:38.763 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.747+0000 7f9d867fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:48:38.763 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.747+0000 7f9d867fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:48:38.763 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.747+0000 7f9d867fc640 10 monclient: dump: 2024-09-16T03:48:38.763 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:48:38.763 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:48:38.763 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:38.763 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:38.763 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:48:38.763 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:48:38.763 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:48:38.763 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:48:38.763 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:48:38.764 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:48:38.764 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.747+0000 7f9d867fc640 1 -- 172.21.15.42:0/529828030 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4711 (secure 0 0 0) 0x7f9d8c0c07f0 con 0x7f9d9009d5e0 2024-09-16T03:48:38.764 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.747+0000 7f9d867fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:48:38.764 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.747+0000 7f9d867fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:48:38.764 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.747+0000 7f9d867fc640 1 -- 172.21.15.42:0/529828030 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9d9009d5e0 msgr2=0x7f9d90137330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:48:38.764 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.747+0000 7f9d867fc640 1 --2- 172.21.15.42:0/529828030 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9d9009d5e0 0x7f9d90137330 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f9d8c01cce0 tx=0x7f9d8c0191e0 comp rx=0 tx=0).stop 2024-09-16T03:48:38.764 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.747+0000 7f9d867fc640 1 -- 172.21.15.42:0/529828030 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9d9009d5e0 msgr2=0x7f9d90137330 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:48:38.764 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.747+0000 7f9d867fc640 1 --2- 172.21.15.42:0/529828030 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9d9009d5e0 0x7f9d90137330 secure :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f9d8c01cce0 tx=0x7f9d8c0191e0 comp rx=0 tx=0).stop 2024-09-16T03:48:38.764 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:48:38.765 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:48:38.765 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:48:38.765 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T03:48:38.765 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 60, 2024-09-16T03:48:38.765 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 56331, 2024-09-16T03:48:38.765 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:48:38.765 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:48:38.765 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T03:48:38.765 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T03:48:38.765 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:48:38.765 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 5, 2024-09-16T03:48:38.765 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:48:38.765 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:48:38.766 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:48:38.766 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:48:38.766 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:48:38.766 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:48:38.766 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:48:38.766 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:48:38.766 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:48:38.766 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:48:38.766 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:48:38.766 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:48:38.766 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:48:38.766 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:48:38.766 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:48:38.766 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:48:38.766 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:48:38.767 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:48:38.767 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:48:38.767 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:48:38.767 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:48:38.767 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:48:38.767 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:48:38.767 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:48:38.767 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:48:38.767 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:48:38.767 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:48:38.767 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:48:38.767 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:48:38.767 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:48:38.768 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:48:38.768 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:48:38.768 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:48:38.768 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:48:38.768 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:48:38.768 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:48:38.768 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:48:38.768 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:48:38.768 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:48:38.768 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:48:38.768 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:48:38.768 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:48:38.768 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:48:38.769 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:48:38.769 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:48:38.769 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:48:38.769 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:48:38.769 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:48:38.769 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:48:38.769 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:48:38.769 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:48:38.769 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:48:38.769 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:48:38.769 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:48:38.769 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:48:38.769 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:48:38.769 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:48:38.770 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:48:38.770 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:48:38.770 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:48:38.770 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:48:38.770 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:48:38.770 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:48:38.770 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:48:38.770 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:48:38.770 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:48:38.770 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:48:38.770 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:48:38.771 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:48:38.771 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:48:38.771 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:48:38.771 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:48:38.771 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:48:38.771 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:48:38.771 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:48:38.771 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:48:38.771 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:48:38.771 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:48:38.771 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:48:38.771 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:48:38.771 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:48:38.772 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:48:38.772 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:48:38.772 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:48:38.772 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:48:38.772 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:48:38.772 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:48:38.772 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:48:38.772 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:48:38.772 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:48:38.772 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:48:38.772 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:48:38.772 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:48:38.772 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:48:38.772 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:48:38.773 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:48:38.773 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:48:38.773 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:48:38.773 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:48:38.773 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:48:38.773 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:48:38.773 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:48:38.773 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:48:38.773 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:48:38.773 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:48:38.773 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:48:38.773 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:48:38.773 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:48:38.773 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:48:38.774 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:48:38.774 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:48:38.774 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:48:38.774 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:48:38.774 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:48:38.774 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:48:38.774 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:48:38.774 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:48:38.774 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:48:38.774 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:48:38.775 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:48:38.775 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:48:38.775 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:48:38.775 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:48:38.775 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:48:38.775 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:48:38.775 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:48:38.775 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:48:38.775 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:48:38.776 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:48:38.776 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:48:38.776 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:48:38.776 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:48:38.776 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:48:38.776 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:48:38.776 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:48:38.776 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:48:38.776 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:48:38.776 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:48:38.776 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:48:38.776 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:48:38.776 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:48:38.776 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T03:48:38.777 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:48:38.777 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:48:38.777 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:48:38.777 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:48:38.777 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:48:38.777 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:48:38.777 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:48:38.777 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:48:38.777 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:48:38.777 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:48:38.777 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 4 2024-09-16T03:48:38.777 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:48:38.777 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:48:38.777 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:48:38.777 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:48:38.778 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:48:38.779 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.747+0000 7f9d97ee0640 10 librados: watch_flush enter 2024-09-16T03:48:38.779 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.747+0000 7f9d97ee0640 10 librados: watch_flush exit 2024-09-16T03:48:38.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.747+0000 7f9d97ee0640 20 client.784112.objecter shutdown clearing up homeless session... 2024-09-16T03:48:38.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.747+0000 7f9d97ee0640 10 client.784112.objecter successfully canceled tick 2024-09-16T03:48:38.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.747+0000 7f9d97ee0640 1 -- 172.21.15.42:0/529828030 >> v2:172.21.15.154:6812/254845342 conn(0x7f9d74037810 msgr2=0x7f9d74039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:48:38.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.747+0000 7f9d97ee0640 1 --2- 172.21.15.42:0/529828030 >> v2:172.21.15.154:6812/254845342 conn(0x7f9d74037810 0x7f9d74039cd0 secure :-1 s=READY pgs=3491 cs=0 l=1 rev1=1 crypto rx=0x7f9d88003c10 tx=0x7f9d8800b040 comp rx=0 tx=0).stop 2024-09-16T03:48:38.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.747+0000 7f9d97ee0640 10 monclient: shutdown 2024-09-16T03:48:38.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.747+0000 7f9d97ee0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:48:38.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.747+0000 7f9d97ee0640 1 -- 172.21.15.42:0/529828030 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9d90135100 msgr2=0x7f9d90134590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:48:38.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.747+0000 7f9d97ee0640 1 --2- 172.21.15.42:0/529828030 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9d90135100 0x7f9d90134590 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f9d8000c7f0 tx=0x7f9d8000cbf0 comp rx=0 tx=0).stop 2024-09-16T03:48:38.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.747+0000 7f9d9647d640 1 -- 172.21.15.42:0/529828030 reap_dead start 2024-09-16T03:48:38.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.747+0000 7f9d97ee0640 1 -- 172.21.15.42:0/529828030 shutdown_connections 2024-09-16T03:48:38.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.747+0000 7f9d97ee0640 1 --2- 172.21.15.42:0/529828030 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9d90135100 0x7f9d90134590 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:38.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.747+0000 7f9d97ee0640 1 -- 172.21.15.42:0/529828030 >> 172.21.15.42:0/529828030 conn(0x7f9d900a2420 msgr2=0x7f9d900a3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:48:38.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.747+0000 7f9d97ee0640 1 -- 172.21.15.42:0/529828030 shutdown_connections 2024-09-16T03:48:38.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.747+0000 7f9d97ee0640 1 -- 172.21.15.42:0/529828030 wait complete. 2024-09-16T03:48:38.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.747+0000 7f9d97ee0640 1 librados: shutdown 2024-09-16T03:48:38.781 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:48:38.781 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T03:48:38.781 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T03:48:38.896 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.891+0000 7f587b177640 10 monclient: get_monmap_and_config 2024-09-16T03:48:38.896 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.891+0000 7f587b177640 10 monclient: build_initial_monmap 2024-09-16T03:48:38.896 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.891+0000 7f587b177640 10 monclient: monmap: 2024-09-16T03:48:38.896 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:48:38.896 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:48:38.897 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:48:38.895496+0000 2024-09-16T03:48:38.897 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:48:38.895496+0000 2024-09-16T03:48:38.897 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:48:38.897 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:48:38.897 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:48:38.897 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:48:38.897 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:48:38.897 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:48:38.897 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:48:38.897 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:48:38.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.895+0000 7f587b177640 1 Processor -- start 2024-09-16T03:48:38.900 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.895+0000 7f587b177640 1 -- start start 2024-09-16T03:48:38.900 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.895+0000 7f587b177640 10 monclient: init 2024-09-16T03:48:38.900 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.895+0000 7f587b177640 10 monclient: _reopen_session rank -1 2024-09-16T03:48:38.900 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.895+0000 7f587b177640 10 monclient: _add_conns ranks=[3,4,2,0,1] 2024-09-16T03:48:38.900 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.895+0000 7f587b177640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f58740a39b0 0x7f5874133b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:38.900 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.895+0000 7f587b177640 10 monclient(hunting): picked mon.noname-a con 0x7f58740a39b0 addr v2:172.21.15.42:3300/0 2024-09-16T03:48:38.900 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.895+0000 7f587b177640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f58740a3d90 0x7f5874137000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:38.900 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.895+0000 7f587b177640 10 monclient(hunting): picked mon.noname-c con 0x7f58740a3d90 addr v2:172.21.15.154:3300/0 2024-09-16T03:48:38.900 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.895+0000 7f587b177640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5874137540 0x7f5874137990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:38.900 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.895+0000 7f587b177640 10 monclient(hunting): picked mon.noname-b con 0x7f5874137540 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:48:38.900 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.895+0000 7f587b177640 10 monclient(hunting): start opening mon connection 2024-09-16T03:48:38.900 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.895+0000 7f587b177640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f587409c140 con 0x7f58740a39b0 2024-09-16T03:48:38.900 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.895+0000 7f587b177640 10 monclient(hunting): start opening mon connection 2024-09-16T03:48:38.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.895+0000 7f587b177640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f587409c420 con 0x7f58740a3d90 2024-09-16T03:48:38.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.895+0000 7f587b177640 10 monclient(hunting): start opening mon connection 2024-09-16T03:48:38.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.895+0000 7f587b177640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f587406dd60 con 0x7f5874137540 2024-09-16T03:48:38.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.895+0000 7f587b177640 10 monclient(hunting): _renew_subs 2024-09-16T03:48:38.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.895+0000 7f587b177640 10 monclient(hunting): authenticate will time out at 1365.865029s 2024-09-16T03:48:38.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.895+0000 7f5878f13640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f58740a39b0 0x7f5874133b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:48:38.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.895+0000 7f5878f13640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f58740a39b0 0x7f5874133b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:44720/0 (socket says 172.21.15.42:44720) 2024-09-16T03:48:38.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.895+0000 7f5878f13640 1 -- 172.21.15.42:0/3078430568 learned_addr learned my addr 172.21.15.42:0/3078430568 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:48:38.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.895+0000 7f5878f13640 10 monclient(hunting): get_auth_request con 0x7f58740a39b0 auth_method 0 2024-09-16T03:48:38.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.895+0000 7f5878f13640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:48:38.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.895+0000 7f5878f13640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:48:38.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.895+0000 7f5878f13640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:48:38.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.895+0000 7f5878f13640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:48:38.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.895+0000 7f5878f13640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:48:38.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.895+0000 7f5879714640 1 --2- 172.21.15.42:0/3078430568 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5874137540 0x7f5874137990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:48:38.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.895+0000 7f5878f13640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:48:38.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.895+0000 7f5879714640 10 monclient(hunting): get_auth_request con 0x7f5874137540 auth_method 0 2024-09-16T03:48:38.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.895+0000 7f5879714640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:48:38.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.895+0000 7f5879714640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:48:38.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.895+0000 7f5879714640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:48:38.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.895+0000 7f5873fff640 1 --2- 172.21.15.42:0/3078430568 >> v2:172.21.15.154:3300/0 conn(0x7f58740a3d90 0x7f5874137000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:48:38.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.895+0000 7f5873fff640 10 monclient(hunting): get_auth_request con 0x7f58740a3d90 auth_method 0 2024-09-16T03:48:38.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.895+0000 7f5873fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:48:38.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.895+0000 7f5873fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:48:38.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.895+0000 7f5873fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:48:38.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.895+0000 7f5878f13640 10 monclient(hunting): handle_auth_done global_id 764496 payload 274 2024-09-16T03:48:38.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.895+0000 7f5878f13640 1 -- 172.21.15.42:0/3078430568 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5874137540 msgr2=0x7f5874137990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:48:38.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.895+0000 7f5878f13640 1 --2- 172.21.15.42:0/3078430568 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5874137540 0x7f5874137990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:38.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.895+0000 7f5878f13640 1 -- 172.21.15.42:0/3078430568 >> v2:172.21.15.154:3300/0 conn(0x7f58740a3d90 msgr2=0x7f5874137000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:48:38.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.895+0000 7f5878f13640 1 --2- 172.21.15.42:0/3078430568 >> v2:172.21.15.154:3300/0 conn(0x7f58740a3d90 0x7f5874137000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:38.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.895+0000 7f5878f13640 10 monclient: _finish_hunting 0 2024-09-16T03:48:38.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.895+0000 7f5878f13640 1 monclient: found mon.noname-a 2024-09-16T03:48:38.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.895+0000 7f5878f13640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:48:38.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.895+0000 7f5878f13640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:48:38.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.895+0000 7f5878f13640 1 -- 172.21.15.42:0/3078430568 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58740a5e20 con 0x7f58740a39b0 2024-09-16T03:48:38.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.895+0000 7f5879714640 1 --2- 172.21.15.42:0/3078430568 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5874137540 0x7f5874137990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T03:48:38.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.895+0000 7f5878f13640 1 --2- 172.21.15.42:0/3078430568 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f58740a39b0 0x7f5874133b80 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f58740a56c0 tx=0x7f586801c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5eca3d78e3ec17ce server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:48:38.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.899+0000 7f58737fe640 1 -- 172.21.15.42:0/3078430568 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58680b0070 con 0x7f58740a39b0 2024-09-16T03:48:38.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.899+0000 7f58737fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:48:38.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.899+0000 7f58737fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:48:38.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.899+0000 7f58737fe640 10 monclient: dump: 2024-09-16T03:48:38.909 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:48:38.909 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:48:38.909 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:38.910 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:38.910 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:48:38.910 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:48:38.910 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:48:38.910 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:48:38.910 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:48:38.910 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:48:38.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.899+0000 7f58737fe640 10 monclient: _finish_auth 0 2024-09-16T03:48:38.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.899+0000 7f58737fe640 10 monclient: _check_auth_tickets 2024-09-16T03:48:38.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.899+0000 7f58737fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:48:38.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.899+0000 7f58737fe640 1 -- 172.21.15.42:0/3078430568 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f58680ab600 con 0x7f58740a39b0 2024-09-16T03:48:38.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.899+0000 7f58737fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:48:38.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.899+0000 7f58737fe640 1 -- 172.21.15.42:0/3078430568 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58680ab9d0 con 0x7f58740a39b0 2024-09-16T03:48:38.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.899+0000 7f587b177640 5 monclient: authenticate success, global_id 764496 2024-09-16T03:48:38.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.899+0000 7f587b177640 10 monclient: get_monmap_and_config success 2024-09-16T03:48:38.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.899+0000 7f587b177640 4 set_mon_vals no callback set 2024-09-16T03:48:38.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.899+0000 7f58737fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:48:38.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.899+0000 7f58737fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:48:38.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.899+0000 7f58737fe640 10 monclient: dump: 2024-09-16T03:48:38.911 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:48:38.911 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:48:38.911 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:38.911 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:38.911 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:48:38.911 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:48:38.911 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:48:38.911 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:48:38.912 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:48:38.912 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:48:38.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.899+0000 7f587b177640 10 monclient: shutdown 2024-09-16T03:48:38.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.899+0000 7f587b177640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:48:38.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.899+0000 7f587b177640 1 -- 172.21.15.42:0/3078430568 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f58740a39b0 msgr2=0x7f5874133b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:48:38.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.899+0000 7f587b177640 1 --2- 172.21.15.42:0/3078430568 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f58740a39b0 0x7f5874133b80 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f58740a56c0 tx=0x7f586801c3e0 comp rx=0 tx=0).stop 2024-09-16T03:48:38.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.899+0000 7f587b177640 1 -- 172.21.15.42:0/3078430568 shutdown_connections 2024-09-16T03:48:38.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.899+0000 7f587b177640 1 --2- 172.21.15.42:0/3078430568 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5874137540 0x7f5874137990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:38.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.899+0000 7f587b177640 1 --2- 172.21.15.42:0/3078430568 >> v2:172.21.15.154:3300/0 conn(0x7f58740a3d90 0x7f5874137000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:38.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.899+0000 7f587b177640 1 --2- 172.21.15.42:0/3078430568 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f58740a39b0 0x7f5874133b80 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:38.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.899+0000 7f587b177640 1 -- 172.21.15.42:0/3078430568 >> 172.21.15.42:0/3078430568 conn(0x7f58740a9fb0 msgr2=0x7f58740aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:48:38.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.899+0000 7f587b177640 1 -- 172.21.15.42:0/3078430568 shutdown_connections 2024-09-16T03:48:38.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.899+0000 7f587b177640 1 -- 172.21.15.42:0/3078430568 wait complete. 2024-09-16T03:48:38.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.899+0000 7f587b177640 10 monclient: build_initial_monmap 2024-09-16T03:48:38.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.899+0000 7f587b177640 10 monclient: monmap: 2024-09-16T03:48:38.913 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:48:38.913 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:48:38.913 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:48:38.913 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:48:38.913 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:48:38.913 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:48:38.913 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:48:38.913 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:48:38.913 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:48:38.913 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:48:38.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.899+0000 7f587b177640 1 librados: starting msgr at 2024-09-16T03:48:38.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.899+0000 7f587b177640 1 librados: starting objecter 2024-09-16T03:48:38.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.899+0000 7f587b177640 1 Processor -- start 2024-09-16T03:48:38.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.899+0000 7f587b177640 1 -- start start 2024-09-16T03:48:38.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.899+0000 7f587b177640 1 librados: setting wanted keys 2024-09-16T03:48:38.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.899+0000 7f587b177640 1 librados: calling monclient init 2024-09-16T03:48:38.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.899+0000 7f587b177640 10 monclient: init 2024-09-16T03:48:38.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.899+0000 7f587b177640 10 monclient: _reopen_session rank -1 2024-09-16T03:48:38.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.899+0000 7f587b177640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:48:38.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.899+0000 7f587b177640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f58740a39b0 0x7f58741cf2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:38.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.899+0000 7f587b177640 10 monclient(hunting): picked mon.noname-b con 0x7f58740a39b0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:48:38.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.899+0000 7f587b177640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f58740a3d90 0x7f58741cf800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:38.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.899+0000 7f587b177640 10 monclient(hunting): picked mon.noname-a con 0x7f58740a3d90 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:48:38.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.899+0000 7f587b177640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5874137540 0x7f587413b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:38.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.899+0000 7f587b177640 10 monclient(hunting): picked mon.noname-c con 0x7f5874137540 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:48:38.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.899+0000 7f587b177640 10 monclient(hunting): start opening mon connection 2024-09-16T03:48:38.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.899+0000 7f587b177640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f5874140dc0 con 0x7f58740a3d90 2024-09-16T03:48:38.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.899+0000 7f587b177640 10 monclient(hunting): start opening mon connection 2024-09-16T03:48:38.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.899+0000 7f587b177640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f5874140f40 con 0x7f58740a39b0 2024-09-16T03:48:38.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.899+0000 7f587b177640 10 monclient(hunting): start opening mon connection 2024-09-16T03:48:38.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.899+0000 7f587b177640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f5874132eb0 con 0x7f5874137540 2024-09-16T03:48:38.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.899+0000 7f587b177640 10 monclient(hunting): _renew_subs 2024-09-16T03:48:38.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.899+0000 7f587b177640 10 monclient(hunting): authenticate will time out at 1365.870331s 2024-09-16T03:48:38.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.899+0000 7f5873fff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f58740a3d90 0x7f58741cf800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:48:38.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.899+0000 7f5879714640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5874137540 0x7f587413b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:48:38.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.899+0000 7f5873fff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f58740a3d90 0x7f58741cf800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:44722/0 (socket says 172.21.15.42:44722) 2024-09-16T03:48:38.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.899+0000 7f5873fff640 1 -- 172.21.15.42:0/2456370336 learned_addr learned my addr 172.21.15.42:0/2456370336 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:48:38.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.899+0000 7f5873fff640 10 monclient(hunting): get_auth_request con 0x7f58740a3d90 auth_method 0 2024-09-16T03:48:38.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.899+0000 7f5873fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:48:38.916 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.899+0000 7f5873fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:48:38.916 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.899+0000 7f5873fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:48:38.916 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.899+0000 7f5879714640 10 monclient(hunting): get_auth_request con 0x7f5874137540 auth_method 0 2024-09-16T03:48:38.916 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.899+0000 7f5879714640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:48:38.916 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.899+0000 7f5879714640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:48:38.916 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.899+0000 7f5879714640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:48:38.916 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.899+0000 7f5873fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:48:38.916 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.899+0000 7f5873fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:48:38.916 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.899+0000 7f5873fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:48:38.916 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.899+0000 7f5879714640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:48:38.916 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.899+0000 7f5879714640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:48:38.916 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.899+0000 7f5879714640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:48:38.916 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.899+0000 7f5878f13640 1 --2- 172.21.15.42:0/2456370336 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f58740a39b0 0x7f58741cf2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:48:38.917 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.899+0000 7f5878f13640 10 monclient(hunting): get_auth_request con 0x7f58740a39b0 auth_method 0 2024-09-16T03:48:38.917 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.899+0000 7f5878f13640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:48:38.917 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.899+0000 7f5878f13640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:48:38.917 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.899+0000 7f5878f13640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:48:38.917 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.899+0000 7f5873fff640 10 monclient(hunting): handle_auth_done global_id 764499 payload 824 2024-09-16T03:48:38.917 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.899+0000 7f5873fff640 1 -- 172.21.15.42:0/2456370336 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5874137540 msgr2=0x7f587413b170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:48:38.917 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.899+0000 7f5873fff640 1 --2- 172.21.15.42:0/2456370336 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5874137540 0x7f587413b170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:38.917 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.899+0000 7f5873fff640 1 -- 172.21.15.42:0/2456370336 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f58740a39b0 msgr2=0x7f58741cf2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:48:38.917 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.899+0000 7f5873fff640 1 --2- 172.21.15.42:0/2456370336 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f58740a39b0 0x7f58741cf2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:38.917 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.899+0000 7f5873fff640 10 monclient: _finish_hunting 0 2024-09-16T03:48:38.917 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.899+0000 7f5873fff640 1 monclient: found mon.noname-a 2024-09-16T03:48:38.917 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.899+0000 7f5873fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:48:38.917 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.899+0000 7f5873fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:48:38.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.899+0000 7f5873fff640 1 -- 172.21.15.42:0/2456370336 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f587409d5a0 con 0x7f58740a3d90 2024-09-16T03:48:38.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.899+0000 7f5879714640 1 --2- 172.21.15.42:0/2456370336 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5874137540 0x7f587413b170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:48:38.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.899+0000 7f5873fff640 1 --2- 172.21.15.42:0/2456370336 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f58740a3d90 0x7f58741cf800 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f586000d800 tx=0x7f586000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:48:38.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.899+0000 7f58717fa640 10 client.?.objecter ms_handle_connect 0x7f58740a3d90 2024-09-16T03:48:38.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.899+0000 7f58717fa640 10 client.?.objecter resend_mon_ops 2024-09-16T03:48:38.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.899+0000 7f58717fa640 1 -- 172.21.15.42:0/2456370336 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f586000b660 con 0x7f58740a3d90 2024-09-16T03:48:38.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.899+0000 7f58717fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:48:38.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.899+0000 7f58717fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:48:38.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.899+0000 7f58717fa640 10 monclient: dump: 2024-09-16T03:48:38.918 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:48:38.918 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:48:38.918 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:38.918 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:38.919 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:48:38.919 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:48:38.919 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:48:38.919 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:48:38.919 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:48:38.919 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:48:38.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.899+0000 7f58717fa640 10 monclient: _finish_auth 0 2024-09-16T03:48:38.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.899+0000 7f58717fa640 10 monclient: _check_auth_tickets 2024-09-16T03:48:38.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.899+0000 7f58717fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:48:38.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.899+0000 7f58717fa640 1 -- 172.21.15.42:0/2456370336 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5860019070 con 0x7f58740a3d90 2024-09-16T03:48:38.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.899+0000 7f587b177640 5 monclient: authenticate success, global_id 764499 2024-09-16T03:48:38.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.899+0000 7f587b177640 10 monclient: _renew_subs 2024-09-16T03:48:38.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.899+0000 7f587b177640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:48:38.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.899+0000 7f587b177640 1 -- 172.21.15.42:0/2456370336 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5874137190 con 0x7f58740a3d90 2024-09-16T03:48:38.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.899+0000 7f58717fa640 10 monclient: handle_config config(0 keys) 2024-09-16T03:48:38.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.899+0000 7f58717fa640 1 -- 172.21.15.42:0/2456370336 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f586000ba00 con 0x7f58740a3d90 2024-09-16T03:48:38.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.899+0000 7f58717fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:48:38.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.899+0000 7f58717fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:48:38.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.899+0000 7f58717fa640 10 monclient: dump: 2024-09-16T03:48:38.920 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:48:38.920 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:48:38.920 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:38.920 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:38.920 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:48:38.920 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:48:38.920 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:48:38.920 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:48:38.920 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:48:38.921 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:48:38.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.899+0000 7f587b177640 10 client.764499.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:48:38.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.899+0000 7f587b177640 10 monclient: _renew_subs 2024-09-16T03:48:38.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.899+0000 7f58727fc640 4 set_mon_vals no callback set 2024-09-16T03:48:38.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.899+0000 7f587b177640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:48:38.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.899+0000 7f587b177640 1 -- 172.21.15.42:0/2456370336 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f587409d2c0 con 0x7f58740a3d90 2024-09-16T03:48:38.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.899+0000 7f587b177640 1 librados: init done 2024-09-16T03:48:38.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.903+0000 7f587b177640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:48:38.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.903+0000 7f587b177640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:48:38.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.903+0000 7f587b177640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:48:38.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.903+0000 7f587b177640 1 -- 172.21.15.42:0/2456370336 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f587409cd00 con 0x7f58740a3d90 2024-09-16T03:48:38.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.903+0000 7f58717fa640 1 -- 172.21.15.42:0/2456370336 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f586000bba0 con 0x7f58740a3d90 2024-09-16T03:48:38.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.903+0000 7f58717fa640 1 --2- 172.21.15.42:0/2456370336 >> v2:172.21.15.154:6812/254845342 conn(0x7f5858037810 0x7f5858039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:38.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.903+0000 7f58717fa640 1 -- 172.21.15.42:0/2456370336 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(214..214 src has 1..214) ==== 6058+0+0 (secure 0 0 0) 0x7f5860049440 con 0x7f58740a3d90 2024-09-16T03:48:38.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.903+0000 7f58717fa640 10 client.764499.objecter ms_dispatch 0x7f5874000ba0 osd_map(214..214 src has 1..214) 2024-09-16T03:48:38.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.903+0000 7f58717fa640 3 client.764499.objecter handle_osd_map got epochs [214,214] > 0 2024-09-16T03:48:38.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.903+0000 7f58717fa640 3 client.764499.objecter handle_osd_map decoding full epoch 214 2024-09-16T03:48:38.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.903+0000 7f58717fa640 20 client.764499.objecter dump_active .. 0 homeless 2024-09-16T03:48:38.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.903+0000 7f5878f13640 1 --2- 172.21.15.42:0/2456370336 >> v2:172.21.15.154:6812/254845342 conn(0x7f5858037810 0x7f5858039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:48:38.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.903+0000 7f5878f13640 10 monclient: get_auth_request con 0x7f5858037810 auth_method 0 2024-09-16T03:48:38.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.903+0000 7f58717fa640 10 client.764499.objecter ms_handle_connect 0x7f5858037810 2024-09-16T03:48:38.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.903+0000 7f5878f13640 1 --2- 172.21.15.42:0/2456370336 >> v2:172.21.15.154:6812/254845342 conn(0x7f5858037810 0x7f5858039cd0 secure :-1 s=READY pgs=3492 cs=0 l=1 rev1=1 crypto rx=0x7f58680ad000 tx=0x7f5868008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:48:38.923 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.907+0000 7f58717fa640 1 -- 172.21.15.42:0/2456370336 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f586001e840 con 0x7f58740a3d90 2024-09-16T03:48:38.923 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.907+0000 7f58717fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:48:38.923 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:38.907+0000 7f58717fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:48:39.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:39.135+0000 7f587b177640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T03:48:39.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:39.135+0000 7f587b177640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T03:48:39.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:39.135+0000 7f587b177640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:48:39.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:39.135+0000 7f587b177640 1 -- 172.21.15.42:0/2456370336 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f5874133b80 con 0x7f58740a3d90 2024-09-16T03:48:39.140 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:39.135+0000 7f58717fa640 1 -- 172.21.15.42:0/2456370336 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f58600490c0 con 0x7f58740a3d90 2024-09-16T03:48:39.140 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:39.135+0000 7f58717fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T03:48:39.140 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:39.135+0000 7f58717fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:48:39.141 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.135+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:48:39.141 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.135+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=29,config=2,health=12,kv=10,logm=47} crc {auth=719323565,config=692619927,health=4193908912,kv=1987072213,logm=3432149384}) 2024-09-16T03:48:39.141 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.139+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=27,config=2,health=11,kv=10,logm=50} crc {auth=3593939884,config=692619927,health=2936019667,kv=3446188498,logm=1949204479}) 2024-09-16T03:48:39.141 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.139+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:48:39.141 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.139+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=29,config=2,health=12,kv=10,logm=47} crc {auth=719323565,config=692619927,health=4193908912,kv=1987072213,logm=3432149384}) 2024-09-16T03:48:39.141 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.139+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=30,config=2,health=11,kv=10,logm=47} crc {auth=3296876796,config=692619927,health=3068285802,kv=3446188498,logm=586871310}) 2024-09-16T03:48:39.143 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:39.139+0000 7f587b177640 10 librados: watch_flush enter 2024-09-16T03:48:39.143 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:39.139+0000 7f587b177640 10 librados: watch_flush exit 2024-09-16T03:48:39.143 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:39.139+0000 7f587b177640 20 client.764499.objecter shutdown clearing up homeless session... 2024-09-16T03:48:39.143 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:39.139+0000 7f587b177640 10 client.764499.objecter successfully canceled tick 2024-09-16T03:48:39.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:39.139+0000 7f587b177640 1 -- 172.21.15.42:0/2456370336 >> v2:172.21.15.154:6812/254845342 conn(0x7f5858037810 msgr2=0x7f5858039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:48:39.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:39.139+0000 7f587b177640 1 --2- 172.21.15.42:0/2456370336 >> v2:172.21.15.154:6812/254845342 conn(0x7f5858037810 0x7f5858039cd0 secure :-1 s=READY pgs=3492 cs=0 l=1 rev1=1 crypto rx=0x7f58680ad000 tx=0x7f5868008040 comp rx=0 tx=0).stop 2024-09-16T03:48:39.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:39.139+0000 7f587b177640 10 monclient: shutdown 2024-09-16T03:48:39.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:39.139+0000 7f587b177640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:48:39.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:39.139+0000 7f587b177640 1 -- 172.21.15.42:0/2456370336 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f58740a3d90 msgr2=0x7f58741cf800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:48:39.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:39.139+0000 7f587b177640 1 --2- 172.21.15.42:0/2456370336 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f58740a3d90 0x7f58741cf800 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f586000d800 tx=0x7f586000dcc0 comp rx=0 tx=0).stop 2024-09-16T03:48:39.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:39.139+0000 7f587b177640 1 -- 172.21.15.42:0/2456370336 shutdown_connections 2024-09-16T03:48:39.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:39.139+0000 7f587b177640 1 --2- 172.21.15.42:0/2456370336 >> v2:172.21.15.154:6812/254845342 conn(0x7f5858037810 0x7f5858039cd0 unknown :-1 s=CLOSED pgs=3492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:39.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:39.139+0000 7f587b177640 1 --2- 172.21.15.42:0/2456370336 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5874137540 0x7f587413b170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:39.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:39.139+0000 7f587b177640 1 --2- 172.21.15.42:0/2456370336 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f58740a3d90 0x7f58741cf800 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:39.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:39.139+0000 7f587b177640 1 --2- 172.21.15.42:0/2456370336 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f58740a39b0 0x7f58741cf2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:39.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:39.139+0000 7f587b177640 1 -- 172.21.15.42:0/2456370336 >> 172.21.15.42:0/2456370336 conn(0x7f58740a9fb0 msgr2=0x7f5874131980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:48:39.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:39.139+0000 7f587b177640 1 -- 172.21.15.42:0/2456370336 shutdown_connections 2024-09-16T03:48:39.145 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:39.139+0000 7f587b177640 1 -- 172.21.15.42:0/2456370336 wait complete. 2024-09-16T03:48:39.145 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:39.139+0000 7f587b177640 1 librados: shutdown 2024-09-16T03:48:39.145 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.139+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:48:39.146 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.139+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4265367916}) 2024-09-16T03:48:39.146 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.139+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2539082483}) 2024-09-16T03:48:39.146 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.139+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:48:39.146 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.139+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4265367916}) 2024-09-16T03:48:39.146 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.139+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=546700029}) 2024-09-16T03:48:39.147 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.143+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:48:39.147 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.143+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1349946014}) 2024-09-16T03:48:39.147 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.143+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3616109116}) 2024-09-16T03:48:39.147 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.143+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:48:39.147 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.143+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1349946014}) 2024-09-16T03:48:39.147 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.143+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2394250374}) 2024-09-16T03:48:39.149 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.147+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:48:39.149 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.147+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1187398199}) 2024-09-16T03:48:39.149 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.147+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=171784713}) 2024-09-16T03:48:39.149 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.147+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:48:39.149 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.147+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1187398199}) 2024-09-16T03:48:39.149 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.147+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3935090143}) 2024-09-16T03:48:39.151 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.147+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:48:39.151 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.147+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=763028117}) 2024-09-16T03:48:39.151 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.147+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=519576628}) 2024-09-16T03:48:39.151 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.147+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:48:39.151 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.147+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=763028117}) 2024-09-16T03:48:39.151 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.147+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2903355596}) 2024-09-16T03:48:39.153 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.151+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:48:39.153 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.151+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2177099840}) 2024-09-16T03:48:39.153 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.151+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=956530699}) 2024-09-16T03:48:39.153 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.151+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:48:39.153 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.151+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2177099840}) 2024-09-16T03:48:39.153 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.151+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1685724200}) 2024-09-16T03:48:39.155 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.151+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:48:39.155 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.151+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2147865452}) 2024-09-16T03:48:39.155 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.151+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2268528399}) 2024-09-16T03:48:39.155 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.151+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:48:39.155 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.151+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2147865452}) 2024-09-16T03:48:39.155 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.151+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3432500980}) 2024-09-16T03:48:39.157 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.155+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:48:39.157 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.155+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2459798541}) 2024-09-16T03:48:39.157 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.155+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2196621747}) 2024-09-16T03:48:39.157 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.155+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:48:39.157 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.155+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2459798541}) 2024-09-16T03:48:39.157 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.155+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1727244195}) 2024-09-16T03:48:39.159 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.155+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:48:39.159 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.155+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3387491608}) 2024-09-16T03:48:39.159 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.155+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3205083073}) 2024-09-16T03:48:39.159 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.155+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:48:39.159 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.155+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3387491608}) 2024-09-16T03:48:39.159 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.155+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2267042425}) 2024-09-16T03:48:39.160 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.155+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:48:39.161 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.155+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1594083855}) 2024-09-16T03:48:39.161 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.155+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=483859469}) 2024-09-16T03:48:39.161 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.155+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:48:39.161 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.155+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1594083855}) 2024-09-16T03:48:39.161 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.155+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=419654036}) 2024-09-16T03:48:39.163 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T03:48:39.163 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.159+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:48:39.163 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.159+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2898924843}) 2024-09-16T03:48:39.163 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.159+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3956183801}) 2024-09-16T03:48:39.163 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.159+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:48:39.163 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.159+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2898924843}) 2024-09-16T03:48:39.163 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.159+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=146316816}) 2024-09-16T03:48:39.164 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.159+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:48:39.164 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.159+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=183420584}) 2024-09-16T03:48:39.164 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.159+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1489800076}) 2024-09-16T03:48:39.164 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.159+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:48:39.164 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.159+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=183420584}) 2024-09-16T03:48:39.164 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.159+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3859733576}) 2024-09-16T03:48:39.166 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.163+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:48:39.166 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.163+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=493014750}) 2024-09-16T03:48:39.166 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.163+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1351582581}) 2024-09-16T03:48:39.166 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.163+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:48:39.166 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.163+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=493014750}) 2024-09-16T03:48:39.166 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.163+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=49055509}) 2024-09-16T03:48:39.168 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.163+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:48:39.168 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.163+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2303610364}) 2024-09-16T03:48:39.168 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.163+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1354566421}) 2024-09-16T03:48:39.168 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.163+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:48:39.168 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.163+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2303610364}) 2024-09-16T03:48:39.168 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.163+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2960265527}) 2024-09-16T03:48:39.169 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.167+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:48:39.170 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.167+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=240811063}) 2024-09-16T03:48:39.170 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.167+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1801718739}) 2024-09-16T03:48:39.170 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.167+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:48:39.170 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.167+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=240811063}) 2024-09-16T03:48:39.170 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.167+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2541962201}) 2024-09-16T03:48:39.171 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.167+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:48:39.172 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.167+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1000277740}) 2024-09-16T03:48:39.172 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.167+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3430086850}) 2024-09-16T03:48:39.172 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.167+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:48:39.172 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.167+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1000277740}) 2024-09-16T03:48:39.172 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.167+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1722118832}) 2024-09-16T03:48:39.173 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.171+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:48:39.174 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.171+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2942984194}) 2024-09-16T03:48:39.174 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.171+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1625871468}) 2024-09-16T03:48:39.174 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.171+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:48:39.174 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.171+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2942984194}) 2024-09-16T03:48:39.174 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.171+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3805964400}) 2024-09-16T03:48:39.175 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.171+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:48:39.176 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.171+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1241651521}) 2024-09-16T03:48:39.176 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.171+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=888701939}) 2024-09-16T03:48:39.176 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.171+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:48:39.176 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.171+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1241651521}) 2024-09-16T03:48:39.176 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.171+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2463090537}) 2024-09-16T03:48:39.178 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.175+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:48:39.178 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.175+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3275836626}) 2024-09-16T03:48:39.178 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.175+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3641804056}) 2024-09-16T03:48:39.178 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.175+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:48:39.178 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.175+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3275836626}) 2024-09-16T03:48:39.178 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.175+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=977046344}) 2024-09-16T03:48:39.180 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.175+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:48:39.180 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.175+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=121961681}) 2024-09-16T03:48:39.180 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.175+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1080406345}) 2024-09-16T03:48:39.180 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.175+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:48:39.180 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.175+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=121961681}) 2024-09-16T03:48:39.180 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.175+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2206580820}) 2024-09-16T03:48:39.181 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.179+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:48:39.182 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.179+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1977764912}) 2024-09-16T03:48:39.182 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.179+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=939256354}) 2024-09-16T03:48:39.182 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.179+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:48:39.182 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.179+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1977764912}) 2024-09-16T03:48:39.182 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.179+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2894888065}) 2024-09-16T03:48:39.183 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.179+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:48:39.183 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.179+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3986263826}) 2024-09-16T03:48:39.183 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.179+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=270478044}) 2024-09-16T03:48:39.184 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.179+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:48:39.184 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.179+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3986263826}) 2024-09-16T03:48:39.184 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.179+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1969012264}) 2024-09-16T03:48:39.185 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.183+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:48:39.186 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.183+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4130327772}) 2024-09-16T03:48:39.186 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.183+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1421896177}) 2024-09-16T03:48:39.186 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.183+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:48:39.186 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.183+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4130327772}) 2024-09-16T03:48:39.186 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.183+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2371702904}) 2024-09-16T03:48:39.187 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.183+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:48:39.187 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.183+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=154368409}) 2024-09-16T03:48:39.187 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.183+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=792014003}) 2024-09-16T03:48:39.187 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.183+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:48:39.187 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.183+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=154368409}) 2024-09-16T03:48:39.187 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.183+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2569421969}) 2024-09-16T03:48:39.189 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.187+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:48:39.189 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.187+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3075881781}) 2024-09-16T03:48:39.189 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.187+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2104293536}) 2024-09-16T03:48:39.189 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.187+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:48:39.189 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.187+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3075881781}) 2024-09-16T03:48:39.189 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.187+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2939078516}) 2024-09-16T03:48:39.190 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.187+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:48:39.190 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.187+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2912704413}) 2024-09-16T03:48:39.190 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.187+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=288796367}) 2024-09-16T03:48:39.191 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.187+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:48:39.191 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.187+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2912704413}) 2024-09-16T03:48:39.191 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.187+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=778684974}) 2024-09-16T03:48:39.192 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.187+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:48:39.192 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.187+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2452622033}) 2024-09-16T03:48:39.192 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.187+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2593930898}) 2024-09-16T03:48:39.192 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.187+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:48:39.192 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.187+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2452622033}) 2024-09-16T03:48:39.193 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.187+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3463093915}) 2024-09-16T03:48:39.194 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.191+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:48:39.194 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.191+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=487082047}) 2024-09-16T03:48:39.194 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.191+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=250056216}) 2024-09-16T03:48:39.194 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.191+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:48:39.194 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.191+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=487082047}) 2024-09-16T03:48:39.194 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.191+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1142288360}) 2024-09-16T03:48:39.195 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.191+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:48:39.196 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.191+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=360567967}) 2024-09-16T03:48:39.196 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.191+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3451058739}) 2024-09-16T03:48:39.196 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.191+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:48:39.196 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.191+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=360567967}) 2024-09-16T03:48:39.196 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.191+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2591443207}) 2024-09-16T03:48:39.197 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.195+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:48:39.197 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.195+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3876608028}) 2024-09-16T03:48:39.198 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.195+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3602049625}) 2024-09-16T03:48:39.198 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.195+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:48:39.198 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.195+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3876608028}) 2024-09-16T03:48:39.198 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.195+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=507894409}) 2024-09-16T03:48:39.199 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.195+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:48:39.199 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.195+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1363202285}) 2024-09-16T03:48:39.199 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.195+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=791313477}) 2024-09-16T03:48:39.199 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.195+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:48:39.199 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.195+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1363202285}) 2024-09-16T03:48:39.199 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.195+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1252994835}) 2024-09-16T03:48:39.201 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.195+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:48:39.201 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.195+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2120435475}) 2024-09-16T03:48:39.201 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.195+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2346537096}) 2024-09-16T03:48:39.201 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.195+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:48:39.201 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.199+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2120435475}) 2024-09-16T03:48:39.201 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.199+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4138437093}) 2024-09-16T03:48:39.203 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.199+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:48:39.203 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.199+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3938357989}) 2024-09-16T03:48:39.203 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.199+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1823257439}) 2024-09-16T03:48:39.203 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.199+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:48:39.203 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.199+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3938357989}) 2024-09-16T03:48:39.203 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.199+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4239563150}) 2024-09-16T03:48:39.204 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.199+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:48:39.204 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.199+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1081314353}) 2024-09-16T03:48:39.204 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.199+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3583287794}) 2024-09-16T03:48:39.204 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.199+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:48:39.204 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.199+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1081314353}) 2024-09-16T03:48:39.204 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.199+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=5882528}) 2024-09-16T03:48:39.206 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.203+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:48:39.206 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.203+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=670024823}) 2024-09-16T03:48:39.206 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.203+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=168646529}) 2024-09-16T03:48:39.206 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.203+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:48:39.206 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.203+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=670024823}) 2024-09-16T03:48:39.206 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.203+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2973196934}) 2024-09-16T03:48:39.208 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.203+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:48:39.208 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.203+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=39,mdsmap=1,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=10,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=19} crc {logm=1945018796,mdsmap=2354140621,mgr=460109776,mgr_command_descs=2107673600,mgr_metadata=1367272202,mgrstat=809185679,mon_config_key=3969788172,monmap=3937513981,osd_metadata=4182959540,osd_pg_creating=1728642274,osd_snap=3249910894}) 2024-09-16T03:48:39.208 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.203+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=39,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=11,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=18} crc {logm=3040605666,mdsmap=4088524775,mgr=460109776,mgr_command_descs=2107673600,mgr_metadata=1367272202,mgrstat=3449421947,mon_config_key=85057112,monmap=3937513981,osd_metadata=191738038,osd_pg_creating=1728642274,osd_snap=17291021}) 2024-09-16T03:48:39.209 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.203+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:48:39.209 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.203+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=39,mdsmap=1,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=10,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=19} crc {logm=1945018796,mdsmap=2354140621,mgr=460109776,mgr_command_descs=2107673600,mgr_metadata=1367272202,mgrstat=809185679,mon_config_key=3969788172,monmap=3937513981,osd_metadata=4182959540,osd_pg_creating=1728642274,osd_snap=3249910894}) 2024-09-16T03:48:39.209 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.203+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=39,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=11,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=17} crc {logm=3147191963,mdsmap=687171726,mgr=460109776,mgr_command_descs=2107673600,mgr_metadata=1367272202,mgrstat=809185679,mon_config_key=85057112,monmap=3937513981,osd_metadata=4021991691,osd_pg_creating=1728642274,osd_snap=306965537}) 2024-09-16T03:48:39.210 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.207+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:48:39.211 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.207+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=8,osdmap=92} crc {osd_snap=802082919,osdmap=3921612440}) 2024-09-16T03:48:39.211 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.207+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=8,osdmap=92} crc {osd_snap=802082919,osdmap=522157400}) 2024-09-16T03:48:39.211 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.207+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:48:39.211 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.207+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=8,osdmap=92} crc {osd_snap=802082919,osdmap=3921612440}) 2024-09-16T03:48:39.211 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.207+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=6,osdmap=94} crc {osd_snap=2926684946,osdmap=701378289}) 2024-09-16T03:48:39.213 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.211+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:48:39.213 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.211+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=4080284489}) 2024-09-16T03:48:39.213 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.211+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1792485013}) 2024-09-16T03:48:39.213 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.211+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:48:39.213 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.211+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=4080284489}) 2024-09-16T03:48:39.213 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.211+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1910739706}) 2024-09-16T03:48:39.217 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.215+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:48:39.217 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.215+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3250663946}) 2024-09-16T03:48:39.217 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.215+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2759583720}) 2024-09-16T03:48:39.217 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.215+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:48:39.218 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.215+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3250663946}) 2024-09-16T03:48:39.218 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.215+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2396458732}) 2024-09-16T03:48:39.236 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.231+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:48:39.237 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.231+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=36} crc {osdmap=2501158448}) 2024-09-16T03:48:39.237 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.231+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=34} crc {osdmap=2641699796}) 2024-09-16T03:48:39.237 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.231+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:48:39.237 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.231+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=36} crc {osdmap=2501158448}) 2024-09-16T03:48:39.237 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:48:39.231+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=33} crc {osdmap=2389492786}) 2024-09-16T03:48:40.164 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T03:48:40.165 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:48:40.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.299+0000 7f69c9041640 10 monclient: get_monmap_and_config 2024-09-16T03:48:40.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.299+0000 7f69c9041640 10 monclient: build_initial_monmap 2024-09-16T03:48:40.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.299+0000 7f69c9041640 10 monclient: monmap: 2024-09-16T03:48:40.303 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:48:40.303 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:48:40.303 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:48:40.302502+0000 2024-09-16T03:48:40.304 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:48:40.302502+0000 2024-09-16T03:48:40.304 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:48:40.304 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:48:40.304 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:48:40.304 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:48:40.304 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:48:40.304 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:48:40.304 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:48:40.305 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:48:40.308 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.299+0000 7f69c9041640 1 Processor -- start 2024-09-16T03:48:40.308 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.299+0000 7f69c9041640 1 -- start start 2024-09-16T03:48:40.308 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.299+0000 7f69c9041640 10 monclient: init 2024-09-16T03:48:40.308 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.299+0000 7f69c9041640 10 monclient: _reopen_session rank -1 2024-09-16T03:48:40.308 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.299+0000 7f69c9041640 10 monclient: _add_conns ranks=[3,0,4,1,2] 2024-09-16T03:48:40.308 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.299+0000 7f69c9041640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f69c406bc40 0x7f69c40a09d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:40.309 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.299+0000 7f69c9041640 10 monclient(hunting): picked mon.noname-a con 0x7f69c406bc40 addr v2:172.21.15.42:3300/0 2024-09-16T03:48:40.309 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.299+0000 7f69c9041640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f69c406c020 addr v1:172.21.15.42:6789/0 2024-09-16T03:48:40.309 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.299+0000 7f69c9041640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f69c413c5b0 0x7f69c413e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:40.309 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.299+0000 7f69c9041640 10 monclient(hunting): picked mon.noname-c con 0x7f69c413c5b0 addr v2:172.21.15.154:3300/0 2024-09-16T03:48:40.309 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.299+0000 7f69c9041640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f69c40a1d50 con 0x7f69c406c020 2024-09-16T03:48:40.309 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.299+0000 7f69c9041640 10 monclient(hunting): start opening mon connection 2024-09-16T03:48:40.309 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.299+0000 7f69c9041640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f69c40a2030 con 0x7f69c406bc40 2024-09-16T03:48:40.309 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.299+0000 7f69c9041640 10 monclient(hunting): start opening mon connection 2024-09-16T03:48:40.309 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.299+0000 7f69c9041640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f69c4072a50 con 0x7f69c413c5b0 2024-09-16T03:48:40.309 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.299+0000 7f69c9041640 10 monclient(hunting): _renew_subs 2024-09-16T03:48:40.309 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.299+0000 7f69c9041640 10 monclient(hunting): authenticate will time out at 1367.271762s 2024-09-16T03:48:40.309 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.299+0000 7f69c259c640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f69c406c020 0x7f69c406c430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:37180/0 (socket says 172.21.15.42:37180) 2024-09-16T03:48:40.309 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.299+0000 7f69c259c640 1 -- 172.21.15.42:0/3812810597 learned_addr learned my addr 172.21.15.42:0/3812810597 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:48:40.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.303+0000 7f69c359e640 1 --2- 172.21.15.42:0/3812810597 >> v2:172.21.15.154:3300/0 conn(0x7f69c413c5b0 0x7f69c413e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:48:40.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.303+0000 7f69c2d9d640 1 --2- 172.21.15.42:0/3812810597 >> v2:172.21.15.42:3300/0 conn(0x7f69c406bc40 0x7f69c40a09d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:48:40.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.303+0000 7f69c359e640 10 monclient(hunting): get_auth_request con 0x7f69c413c5b0 auth_method 0 2024-09-16T03:48:40.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.303+0000 7f69c359e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:48:40.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.303+0000 7f69c359e640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:48:40.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.303+0000 7f69c359e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:48:40.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.303+0000 7f69c2d9d640 10 monclient(hunting): get_auth_request con 0x7f69c406bc40 auth_method 0 2024-09-16T03:48:40.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.303+0000 7f69c2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:48:40.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.303+0000 7f69c2d9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:48:40.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.303+0000 7f69c2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:48:40.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.303+0000 7f69c359e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:48:40.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.303+0000 7f69c359e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:48:40.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.303+0000 7f69c359e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:48:40.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.303+0000 7f69c2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:48:40.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.303+0000 7f69c2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:48:40.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.303+0000 7f69c2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:48:40.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.303+0000 7f69c2d9d640 10 monclient(hunting): handle_auth_done global_id 764502 payload 274 2024-09-16T03:48:40.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.303+0000 7f69c2d9d640 1 -- 172.21.15.42:0/3812810597 >> v2:172.21.15.154:3300/0 conn(0x7f69c413c5b0 msgr2=0x7f69c413e9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:48:40.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.303+0000 7f69c2d9d640 1 --2- 172.21.15.42:0/3812810597 >> v2:172.21.15.154:3300/0 conn(0x7f69c413c5b0 0x7f69c413e9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:40.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.303+0000 7f69c2d9d640 1 -- 172.21.15.42:0/3812810597 >> v1:172.21.15.42:6789/0 conn(0x7f69c406c020 legacy=0x7f69c406c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:48:40.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.303+0000 7f69c2d9d640 10 monclient: _finish_hunting 0 2024-09-16T03:48:40.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.303+0000 7f69c2d9d640 1 monclient: found mon.noname-a 2024-09-16T03:48:40.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.303+0000 7f69c2d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:48:40.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.303+0000 7f69c2d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:48:40.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.303+0000 7f69c2d9d640 1 -- 172.21.15.42:0/3812810597 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69c40a1a70 con 0x7f69c406bc40 2024-09-16T03:48:40.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.303+0000 7f69c359e640 1 --2- 172.21.15.42:0/3812810597 >> v2:172.21.15.154:3300/0 conn(0x7f69c413c5b0 0x7f69c413e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:48:40.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.303+0000 7f69c2d9d640 1 --2- 172.21.15.42:0/3812810597 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f69c406bc40 0x7f69c40a09d0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f69c406cb40 tx=0x7f69b401c340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b94811716fcc5a6 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:48:40.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.303+0000 7f69c1d9b640 1 -- 172.21.15.42:0/3812810597 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69b40b2070 con 0x7f69c406bc40 2024-09-16T03:48:40.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.303+0000 7f69c1d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:48:40.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.303+0000 7f69c1d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:48:40.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.303+0000 7f69c1d9b640 10 monclient: dump: 2024-09-16T03:48:40.312 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:48:40.313 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:48:40.313 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:40.313 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:40.313 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:48:40.313 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:48:40.313 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:48:40.314 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:48:40.314 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:48:40.314 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:48:40.314 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.303+0000 7f69c1d9b640 10 monclient: _finish_auth 0 2024-09-16T03:48:40.314 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.303+0000 7f69c1d9b640 10 monclient: _check_auth_tickets 2024-09-16T03:48:40.314 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.303+0000 7f69c1d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:48:40.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.303+0000 7f69c1d9b640 1 -- 172.21.15.42:0/3812810597 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f69b40a9070 con 0x7f69c406bc40 2024-09-16T03:48:40.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.303+0000 7f69c1d9b640 10 monclient: handle_config config(0 keys) 2024-09-16T03:48:40.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.303+0000 7f69c1d9b640 1 -- 172.21.15.42:0/3812810597 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69b40ad880 con 0x7f69c406bc40 2024-09-16T03:48:40.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.303+0000 7f69c9041640 5 monclient: authenticate success, global_id 764502 2024-09-16T03:48:40.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.303+0000 7f69c9041640 10 monclient: get_monmap_and_config success 2024-09-16T03:48:40.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.303+0000 7f69c9041640 4 set_mon_vals no callback set 2024-09-16T03:48:40.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.303+0000 7f69c1d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:48:40.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.303+0000 7f69c1d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:48:40.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.303+0000 7f69c1d9b640 10 monclient: dump: 2024-09-16T03:48:40.316 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:48:40.316 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:48:40.316 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:40.316 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:40.316 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:48:40.317 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:48:40.317 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:48:40.317 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:48:40.317 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:48:40.317 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:48:40.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.307+0000 7f69c9041640 10 monclient: shutdown 2024-09-16T03:48:40.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.307+0000 7f69c9041640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:48:40.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.307+0000 7f69c9041640 1 -- 172.21.15.42:0/3812810597 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f69c406bc40 msgr2=0x7f69c40a09d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:48:40.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.307+0000 7f69c9041640 1 --2- 172.21.15.42:0/3812810597 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f69c406bc40 0x7f69c40a09d0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f69c406cb40 tx=0x7f69b401c340 comp rx=0 tx=0).stop 2024-09-16T03:48:40.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.307+0000 7f69c9041640 1 -- 172.21.15.42:0/3812810597 shutdown_connections 2024-09-16T03:48:40.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.307+0000 7f69c9041640 1 --2- 172.21.15.42:0/3812810597 >> v2:172.21.15.154:3300/0 conn(0x7f69c413c5b0 0x7f69c413e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:40.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.307+0000 7f69c9041640 1 --2- 172.21.15.42:0/3812810597 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f69c406bc40 0x7f69c40a09d0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:40.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.307+0000 7f69c9041640 1 -- 172.21.15.42:0/3812810597 >> 172.21.15.42:0/3812810597 conn(0x7f69c40a3220 msgr2=0x7f69c40a3630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:48:40.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.307+0000 7f69c9041640 1 -- 172.21.15.42:0/3812810597 shutdown_connections 2024-09-16T03:48:40.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.307+0000 7f69c9041640 1 -- 172.21.15.42:0/3812810597 wait complete. 2024-09-16T03:48:40.329 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.307+0000 7f69c9041640 10 monclient: build_initial_monmap 2024-09-16T03:48:40.329 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.307+0000 7f69c9041640 10 monclient: monmap: 2024-09-16T03:48:40.329 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:48:40.329 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:48:40.329 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:48:40.329 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:48:40.329 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:48:40.329 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:48:40.329 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:48:40.330 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:48:40.330 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:48:40.330 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:48:40.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.307+0000 7f69c9041640 1 librados: starting msgr at 2024-09-16T03:48:40.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.307+0000 7f69c9041640 1 librados: starting objecter 2024-09-16T03:48:40.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.307+0000 7f69c9041640 1 Processor -- start 2024-09-16T03:48:40.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.307+0000 7f69c9041640 1 -- start start 2024-09-16T03:48:40.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.307+0000 7f69c9041640 1 librados: setting wanted keys 2024-09-16T03:48:40.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.307+0000 7f69c9041640 1 librados: calling monclient init 2024-09-16T03:48:40.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.307+0000 7f69c9041640 10 monclient: init 2024-09-16T03:48:40.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.307+0000 7f69c9041640 10 monclient: _reopen_session rank -1 2024-09-16T03:48:40.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.307+0000 7f69c9041640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:48:40.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.307+0000 7f69c9041640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f69c406bc40 0x7f69c41ceed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:40.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.307+0000 7f69c9041640 10 monclient(hunting): picked mon.noname-a con 0x7f69c406bc40 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:48:40.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.307+0000 7f69c9041640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f69c406c020 0x7f69c4137330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:40.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.307+0000 7f69c9041640 10 monclient(hunting): picked mon.noname-c con 0x7f69c406c020 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:48:40.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.307+0000 7f69c9041640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f69c413c5b0 0x7f69c411ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:40.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.307+0000 7f69c9041640 10 monclient(hunting): picked mon.noname-b con 0x7f69c413c5b0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:48:40.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.307+0000 7f69c9041640 10 monclient(hunting): start opening mon connection 2024-09-16T03:48:40.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.307+0000 7f69c9041640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f69c4141a90 con 0x7f69c406bc40 2024-09-16T03:48:40.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.307+0000 7f69c9041640 10 monclient(hunting): start opening mon connection 2024-09-16T03:48:40.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.307+0000 7f69c9041640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f69c4141910 con 0x7f69c413c5b0 2024-09-16T03:48:40.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.307+0000 7f69c9041640 10 monclient(hunting): start opening mon connection 2024-09-16T03:48:40.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.307+0000 7f69c9041640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f69c4141c10 con 0x7f69c406c020 2024-09-16T03:48:40.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.307+0000 7f69c9041640 10 monclient(hunting): _renew_subs 2024-09-16T03:48:40.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.307+0000 7f69c9041640 10 monclient(hunting): authenticate will time out at 1367.278049s 2024-09-16T03:48:40.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.307+0000 7f69c2d9d640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f69c406bc40 0x7f69c41ceed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:48:40.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.307+0000 7f69c259c640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f69c406c020 0x7f69c4137330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:48:40.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.307+0000 7f69c2d9d640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f69c406bc40 0x7f69c41ceed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:44748/0 (socket says 172.21.15.42:44748) 2024-09-16T03:48:40.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.307+0000 7f69c2d9d640 1 -- 172.21.15.42:0/1842613555 learned_addr learned my addr 172.21.15.42:0/1842613555 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:48:40.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.307+0000 7f69c2d9d640 10 monclient(hunting): get_auth_request con 0x7f69c406bc40 auth_method 0 2024-09-16T03:48:40.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.307+0000 7f69c2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:48:40.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.307+0000 7f69c2d9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:48:40.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.307+0000 7f69c2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:48:40.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.307+0000 7f69c259c640 10 monclient(hunting): get_auth_request con 0x7f69c406c020 auth_method 0 2024-09-16T03:48:40.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.307+0000 7f69c259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:48:40.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.307+0000 7f69c259c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:48:40.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.307+0000 7f69c259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:48:40.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.307+0000 7f69c2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:48:40.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.307+0000 7f69c2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:48:40.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.307+0000 7f69c2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:48:40.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.307+0000 7f69c259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:48:40.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.307+0000 7f69c259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:48:40.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.307+0000 7f69c259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:48:40.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.307+0000 7f69c359e640 1 --2- 172.21.15.42:0/1842613555 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f69c413c5b0 0x7f69c411ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:48:40.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.307+0000 7f69c359e640 10 monclient(hunting): get_auth_request con 0x7f69c413c5b0 auth_method 0 2024-09-16T03:48:40.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.307+0000 7f69c359e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:48:40.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.307+0000 7f69c359e640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:48:40.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.307+0000 7f69c359e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:48:40.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.307+0000 7f69c2d9d640 10 monclient(hunting): handle_auth_done global_id 764505 payload 824 2024-09-16T03:48:40.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.307+0000 7f69c2d9d640 1 -- 172.21.15.42:0/1842613555 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f69c406c020 msgr2=0x7f69c4137330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:48:40.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.307+0000 7f69c2d9d640 1 --2- 172.21.15.42:0/1842613555 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f69c406c020 0x7f69c4137330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:40.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.307+0000 7f69c2d9d640 1 -- 172.21.15.42:0/1842613555 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f69c413c5b0 msgr2=0x7f69c411ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:48:40.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.307+0000 7f69c2d9d640 1 --2- 172.21.15.42:0/1842613555 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f69c413c5b0 0x7f69c411ddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:40.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.307+0000 7f69c2d9d640 10 monclient: _finish_hunting 0 2024-09-16T03:48:40.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.307+0000 7f69c2d9d640 1 monclient: found mon.noname-a 2024-09-16T03:48:40.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.307+0000 7f69c2d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:48:40.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.307+0000 7f69c2d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:48:40.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.307+0000 7f69c2d9d640 1 -- 172.21.15.42:0/1842613555 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69c41219e0 con 0x7f69c406bc40 2024-09-16T03:48:40.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.307+0000 7f69c259c640 1 --2- 172.21.15.42:0/1842613555 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f69c406c020 0x7f69c4137330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:48:40.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.307+0000 7f69c2d9d640 1 --2- 172.21.15.42:0/1842613555 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f69c406bc40 0x7f69c41ceed0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f69b40b0040 tx=0x7f69b40190e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:48:40.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.307+0000 7f69ab7fe640 10 client.?.objecter ms_handle_connect 0x7f69c406bc40 2024-09-16T03:48:40.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.307+0000 7f69ab7fe640 10 client.?.objecter resend_mon_ops 2024-09-16T03:48:40.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.307+0000 7f69ab7fe640 1 -- 172.21.15.42:0/1842613555 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69b40ba020 con 0x7f69c406bc40 2024-09-16T03:48:40.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.307+0000 7f69ab7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:48:40.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.307+0000 7f69ab7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:48:40.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.307+0000 7f69ab7fe640 10 monclient: dump: 2024-09-16T03:48:40.337 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:48:40.337 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:48:40.338 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:40.338 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:40.338 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:48:40.338 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:48:40.338 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:48:40.338 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:48:40.338 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:48:40.339 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:48:40.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.307+0000 7f69ab7fe640 10 monclient: _finish_auth 0 2024-09-16T03:48:40.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.307+0000 7f69ab7fe640 10 monclient: _check_auth_tickets 2024-09-16T03:48:40.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.307+0000 7f69ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:48:40.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.307+0000 7f69ab7fe640 1 -- 172.21.15.42:0/1842613555 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f69b40add80 con 0x7f69c406bc40 2024-09-16T03:48:40.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.307+0000 7f69c9041640 5 monclient: authenticate success, global_id 764505 2024-09-16T03:48:40.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.307+0000 7f69c9041640 10 monclient: _renew_subs 2024-09-16T03:48:40.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.307+0000 7f69c9041640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:48:40.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.307+0000 7f69c9041640 1 -- 172.21.15.42:0/1842613555 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69c4121cc0 con 0x7f69c406bc40 2024-09-16T03:48:40.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.307+0000 7f69ab7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:48:40.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.307+0000 7f69ab7fe640 1 -- 172.21.15.42:0/1842613555 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69b40b2040 con 0x7f69c406bc40 2024-09-16T03:48:40.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.307+0000 7f69ab7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:48:40.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.307+0000 7f69ab7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:48:40.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.307+0000 7f69ab7fe640 10 monclient: dump: 2024-09-16T03:48:40.341 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:48:40.341 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:48:40.341 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:40.341 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:40.341 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:48:40.342 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:48:40.342 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:48:40.342 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:48:40.342 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:48:40.342 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:48:40.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.307+0000 7f69c159a640 4 set_mon_vals no callback set 2024-09-16T03:48:40.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.307+0000 7f69c9041640 10 client.764505.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:48:40.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.307+0000 7f69c9041640 10 monclient: _renew_subs 2024-09-16T03:48:40.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.307+0000 7f69c9041640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:48:40.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.307+0000 7f69c9041640 1 -- 172.21.15.42:0/1842613555 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69c413dc00 con 0x7f69c406bc40 2024-09-16T03:48:40.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.307+0000 7f69c9041640 1 librados: init done 2024-09-16T03:48:40.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.311+0000 7f69c9041640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:48:40.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.311+0000 7f69c9041640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:48:40.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.311+0000 7f69c9041640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:48:40.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.311+0000 7f69c9041640 1 -- 172.21.15.42:0/1842613555 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f69c40a09d0 con 0x7f69c406bc40 2024-09-16T03:48:40.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.311+0000 7f69ab7fe640 1 -- 172.21.15.42:0/1842613555 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f69b40bd020 con 0x7f69c406bc40 2024-09-16T03:48:40.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.311+0000 7f69ab7fe640 1 --2- 172.21.15.42:0/1842613555 >> v2:172.21.15.154:6812/254845342 conn(0x7f69b0037840 0x7f69b0039d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:40.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.311+0000 7f69ab7fe640 1 -- 172.21.15.42:0/1842613555 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(216..216 src has 1..216) ==== 6980+0+0 (secure 0 0 0) 0x7f69b40e2fe0 con 0x7f69c406bc40 2024-09-16T03:48:40.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.311+0000 7f69ab7fe640 10 client.764505.objecter ms_dispatch 0x7f69c4000ba0 osd_map(216..216 src has 1..216) 2024-09-16T03:48:40.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.311+0000 7f69ab7fe640 3 client.764505.objecter handle_osd_map got epochs [216,216] > 0 2024-09-16T03:48:40.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.311+0000 7f69ab7fe640 3 client.764505.objecter handle_osd_map decoding full epoch 216 2024-09-16T03:48:40.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.311+0000 7f69ab7fe640 20 client.764505.objecter dump_active .. 0 homeless 2024-09-16T03:48:40.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.311+0000 7f69c259c640 1 --2- 172.21.15.42:0/1842613555 >> v2:172.21.15.154:6812/254845342 conn(0x7f69b0037840 0x7f69b0039d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:48:40.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.311+0000 7f69c259c640 10 monclient: get_auth_request con 0x7f69b0037840 auth_method 0 2024-09-16T03:48:40.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.311+0000 7f69c259c640 1 --2- 172.21.15.42:0/1842613555 >> v2:172.21.15.154:6812/254845342 conn(0x7f69b0037840 0x7f69b0039d00 secure :-1 s=READY pgs=3493 cs=0 l=1 rev1=1 crypto rx=0x7f69c411f5d0 tx=0x7f69b8007e90 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:48:40.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.311+0000 7f69ab7fe640 10 client.764505.objecter ms_handle_connect 0x7f69b0037840 2024-09-16T03:48:40.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.315+0000 7f69ab7fe640 1 -- 172.21.15.42:0/1842613555 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f69b40c4030 con 0x7f69c406bc40 2024-09-16T03:48:40.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.315+0000 7f69ab7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:48:40.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.315+0000 7f69ab7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:48:40.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.547+0000 7f69c9041640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:48:40.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.547+0000 7f69c9041640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:48:40.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.547+0000 7f69c9041640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:48:40.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.547+0000 7f69c9041640 1 -- 172.21.15.42:0/1842613555 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f69c41202b0 con 0x7f69c406bc40 2024-09-16T03:48:40.551 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.547+0000 7f69ab7fe640 1 -- 172.21.15.42:0/1842613555 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7f69b4016050 con 0x7f69c406bc40 2024-09-16T03:48:40.551 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.547+0000 7f69ab7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:48:40.551 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.547+0000 7f69ab7fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:48:40.551 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":60,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:48:40.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.551+0000 7f69c9041640 10 librados: watch_flush enter 2024-09-16T03:48:40.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.551+0000 7f69c9041640 10 librados: watch_flush exit 2024-09-16T03:48:40.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.551+0000 7f69c9041640 20 client.764505.objecter shutdown clearing up homeless session... 2024-09-16T03:48:40.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.551+0000 7f69c9041640 10 client.764505.objecter successfully canceled tick 2024-09-16T03:48:40.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.551+0000 7f69c9041640 1 -- 172.21.15.42:0/1842613555 >> v2:172.21.15.154:6812/254845342 conn(0x7f69b0037840 msgr2=0x7f69b0039d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:48:40.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.551+0000 7f69c9041640 1 --2- 172.21.15.42:0/1842613555 >> v2:172.21.15.154:6812/254845342 conn(0x7f69b0037840 0x7f69b0039d00 secure :-1 s=READY pgs=3493 cs=0 l=1 rev1=1 crypto rx=0x7f69c411f5d0 tx=0x7f69b8007e90 comp rx=0 tx=0).stop 2024-09-16T03:48:40.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.551+0000 7f69c9041640 10 monclient: shutdown 2024-09-16T03:48:40.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.551+0000 7f69c9041640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:48:40.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.551+0000 7f69c9041640 1 -- 172.21.15.42:0/1842613555 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f69c406bc40 msgr2=0x7f69c41ceed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:48:40.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.551+0000 7f69c9041640 1 --2- 172.21.15.42:0/1842613555 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f69c406bc40 0x7f69c41ceed0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f69b40b0040 tx=0x7f69b40190e0 comp rx=0 tx=0).stop 2024-09-16T03:48:40.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.551+0000 7f69c9041640 1 -- 172.21.15.42:0/1842613555 shutdown_connections 2024-09-16T03:48:40.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.551+0000 7f69c9041640 1 --2- 172.21.15.42:0/1842613555 >> v2:172.21.15.154:6812/254845342 conn(0x7f69b0037840 0x7f69b0039d00 unknown :-1 s=CLOSED pgs=3493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:40.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.551+0000 7f69c9041640 1 --2- 172.21.15.42:0/1842613555 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f69c413c5b0 0x7f69c411ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:40.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.551+0000 7f69c9041640 1 --2- 172.21.15.42:0/1842613555 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f69c406c020 0x7f69c4137330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:40.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.551+0000 7f69c9041640 1 --2- 172.21.15.42:0/1842613555 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f69c406bc40 0x7f69c41ceed0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:40.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.551+0000 7f69c9041640 1 -- 172.21.15.42:0/1842613555 >> 172.21.15.42:0/1842613555 conn(0x7f69c40a3220 msgr2=0x7f69c413b350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:48:40.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.551+0000 7f69c9041640 1 -- 172.21.15.42:0/1842613555 shutdown_connections 2024-09-16T03:48:40.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.551+0000 7f69c9041640 1 -- 172.21.15.42:0/1842613555 wait complete. 2024-09-16T03:48:40.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.551+0000 7f69c9041640 1 librados: shutdown 2024-09-16T03:48:40.570 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T03:48:40.570 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T03:48:40.570 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T03:48:40.570 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T03:48:40.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.683+0000 7f0f7134b640 10 monclient: get_monmap_and_config 2024-09-16T03:48:40.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.683+0000 7f0f7134b640 10 monclient: build_initial_monmap 2024-09-16T03:48:40.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.683+0000 7f0f7134b640 10 monclient: monmap: 2024-09-16T03:48:40.686 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:48:40.686 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:48:40.686 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:48:40.685414+0000 2024-09-16T03:48:40.687 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:48:40.685414+0000 2024-09-16T03:48:40.687 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:48:40.687 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:48:40.687 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:48:40.687 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:48:40.687 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:48:40.687 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:48:40.687 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:48:40.687 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:48:40.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.683+0000 7f0f7134b640 1 Processor -- start 2024-09-16T03:48:40.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.683+0000 7f0f7134b640 1 -- start start 2024-09-16T03:48:40.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.683+0000 7f0f7134b640 10 monclient: init 2024-09-16T03:48:40.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.683+0000 7f0f7134b640 10 monclient: _reopen_session rank -1 2024-09-16T03:48:40.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.683+0000 7f0f7134b640 10 monclient: _add_conns ranks=[0,1,3,4,2] 2024-09-16T03:48:40.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.683+0000 7f0f7134b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0f6c06db90 addr v1:172.21.15.42:6789/0 2024-09-16T03:48:40.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.683+0000 7f0f7134b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0f6c1340c0 addr v1:172.21.15.154:6789/0 2024-09-16T03:48:40.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.683+0000 7f0f7134b640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f0f6c13d8b0 0x7f0f6c13fca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:40.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.683+0000 7f0f7134b640 10 monclient(hunting): picked mon.noname-a con 0x7f0f6c13d8b0 addr v2:172.21.15.42:3300/0 2024-09-16T03:48:40.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.683+0000 7f0f7134b640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0f6c06ece0 con 0x7f0f6c06db90 2024-09-16T03:48:40.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.683+0000 7f0f7134b640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0f6c09bf80 con 0x7f0f6c1340c0 2024-09-16T03:48:40.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.683+0000 7f0f7134b640 10 monclient(hunting): start opening mon connection 2024-09-16T03:48:40.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.683+0000 7f0f7134b640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f0f6c06df70 con 0x7f0f6c13d8b0 2024-09-16T03:48:40.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.683+0000 7f0f7134b640 10 monclient(hunting): _renew_subs 2024-09-16T03:48:40.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.683+0000 7f0f7134b640 10 monclient(hunting): authenticate will time out at 1367.655173s 2024-09-16T03:48:40.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.683+0000 7f0f6b7fe640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f0f6c13d8b0 0x7f0f6c13fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:48:40.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.683+0000 7f0f6b7fe640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f0f6c13d8b0 0x7f0f6c13fca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:44760/0 (socket says 172.21.15.42:44760) 2024-09-16T03:48:40.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.683+0000 7f0f6affd640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f0f6c06db90 0x7f0f6c0a37f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:37196/0 (socket says 172.21.15.42:37196) 2024-09-16T03:48:40.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.683+0000 7f0f6b7fe640 1 -- 172.21.15.42:0/973662968 learned_addr learned my addr 172.21.15.42:0/973662968 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:48:40.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.683+0000 7f0f6b7fe640 10 monclient(hunting): get_auth_request con 0x7f0f6c13d8b0 auth_method 0 2024-09-16T03:48:40.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.683+0000 7f0f6b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:48:40.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.683+0000 7f0f6b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:48:40.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.683+0000 7f0f6b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:48:40.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.683+0000 7f0f6b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:48:40.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.683+0000 7f0f6b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:48:40.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.683+0000 7f0f6b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:48:40.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.683+0000 7f0f6b7fe640 10 monclient(hunting): handle_auth_done global_id 764508 payload 274 2024-09-16T03:48:40.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.683+0000 7f0f6b7fe640 1 -- 172.21.15.42:0/973662968 >> v1:172.21.15.154:6789/0 conn(0x7f0f6c1340c0 legacy=0x7f0f6c13c190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:48:40.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.683+0000 7f0f6b7fe640 1 -- 172.21.15.42:0/973662968 >> v1:172.21.15.42:6789/0 conn(0x7f0f6c06db90 legacy=0x7f0f6c0a37f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:48:40.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.683+0000 7f0f6b7fe640 10 monclient: _finish_hunting 0 2024-09-16T03:48:40.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.683+0000 7f0f6b7fe640 1 monclient: found mon.noname-a 2024-09-16T03:48:40.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.683+0000 7f0f6b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:48:40.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.683+0000 7f0f6b7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:48:40.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.683+0000 7f0f6b7fe640 1 -- 172.21.15.42:0/973662968 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f6c1344d0 con 0x7f0f6c13d8b0 2024-09-16T03:48:40.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.687+0000 7f0f6b7fe640 1 --2- 172.21.15.42:0/973662968 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0f6c13d8b0 0x7f0f6c13fca0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f0f64009870 tx=0x7f0f6401cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9deab7f4c1fdb7c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:48:40.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.687+0000 7f0f69ffb640 1 -- 172.21.15.42:0/973662968 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f640b0070 con 0x7f0f6c13d8b0 2024-09-16T03:48:40.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.687+0000 7f0f69ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:48:40.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.687+0000 7f0f69ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:48:40.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.687+0000 7f0f69ffb640 10 monclient: dump: 2024-09-16T03:48:40.698 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:48:40.698 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:48:40.698 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:40.698 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:40.698 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:48:40.698 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:48:40.698 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:48:40.698 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:48:40.698 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:48:40.698 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:48:40.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.687+0000 7f0f69ffb640 10 monclient: _finish_auth 0 2024-09-16T03:48:40.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.687+0000 7f0f69ffb640 10 monclient: _check_auth_tickets 2024-09-16T03:48:40.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.687+0000 7f0f69ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:48:40.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.687+0000 7f0f69ffb640 1 -- 172.21.15.42:0/973662968 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0f640ab600 con 0x7f0f6c13d8b0 2024-09-16T03:48:40.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.687+0000 7f0f7134b640 5 monclient: authenticate success, global_id 764508 2024-09-16T03:48:40.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.687+0000 7f0f7134b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:48:40.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.687+0000 7f0f69ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T03:48:40.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.687+0000 7f0f7134b640 10 monclient: get_monmap_and_config success 2024-09-16T03:48:40.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.687+0000 7f0f69ffb640 1 -- 172.21.15.42:0/973662968 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f640ab9d0 con 0x7f0f6c13d8b0 2024-09-16T03:48:40.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.687+0000 7f0f7134b640 4 set_mon_vals no callback set 2024-09-16T03:48:40.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.687+0000 7f0f69ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:48:40.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.687+0000 7f0f69ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:48:40.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.687+0000 7f0f69ffb640 10 monclient: dump: 2024-09-16T03:48:40.699 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:48:40.700 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:48:40.700 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:40.700 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:40.700 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:48:40.700 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:48:40.700 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:48:40.700 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:48:40.700 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:48:40.700 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:48:40.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.687+0000 7f0f7134b640 10 monclient: shutdown 2024-09-16T03:48:40.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.687+0000 7f0f7134b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:48:40.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.687+0000 7f0f7134b640 1 -- 172.21.15.42:0/973662968 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0f6c13d8b0 msgr2=0x7f0f6c13fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:48:40.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.687+0000 7f0f7134b640 1 --2- 172.21.15.42:0/973662968 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0f6c13d8b0 0x7f0f6c13fca0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f0f64009870 tx=0x7f0f6401cc60 comp rx=0 tx=0).stop 2024-09-16T03:48:40.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.687+0000 7f0f7134b640 1 -- 172.21.15.42:0/973662968 shutdown_connections 2024-09-16T03:48:40.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.687+0000 7f0f7134b640 1 --2- 172.21.15.42:0/973662968 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0f6c13d8b0 0x7f0f6c13fca0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:40.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.687+0000 7f0f7134b640 1 -- 172.21.15.42:0/973662968 >> 172.21.15.42:0/973662968 conn(0x7f0f6c0a9fb0 msgr2=0x7f0f6c0aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:48:40.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.687+0000 7f0f7134b640 1 -- 172.21.15.42:0/973662968 shutdown_connections 2024-09-16T03:48:40.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.687+0000 7f0f7134b640 1 -- 172.21.15.42:0/973662968 wait complete. 2024-09-16T03:48:40.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.687+0000 7f0f7134b640 10 monclient: build_initial_monmap 2024-09-16T03:48:40.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.687+0000 7f0f7134b640 10 monclient: monmap: 2024-09-16T03:48:40.701 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:48:40.701 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:48:40.701 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:48:40.701 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:48:40.701 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:48:40.701 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:48:40.701 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:48:40.702 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:48:40.702 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:48:40.702 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:48:40.702 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.687+0000 7f0f7134b640 1 librados: starting msgr at 2024-09-16T03:48:40.702 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.687+0000 7f0f7134b640 1 librados: starting objecter 2024-09-16T03:48:40.702 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.687+0000 7f0f7134b640 1 Processor -- start 2024-09-16T03:48:40.702 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.687+0000 7f0f7134b640 1 -- start start 2024-09-16T03:48:40.702 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.687+0000 7f0f7134b640 1 librados: setting wanted keys 2024-09-16T03:48:40.702 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.687+0000 7f0f7134b640 1 librados: calling monclient init 2024-09-16T03:48:40.702 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.687+0000 7f0f7134b640 10 monclient: init 2024-09-16T03:48:40.702 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.691+0000 7f0f7134b640 10 monclient: _reopen_session rank -1 2024-09-16T03:48:40.702 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.691+0000 7f0f7134b640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:48:40.702 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.691+0000 7f0f7134b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0f6c06db90 0x7f0f6c139d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:40.702 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.691+0000 7f0f7134b640 10 monclient(hunting): picked mon.noname-b con 0x7f0f6c06db90 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:48:40.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.691+0000 7f0f7134b640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0f6c1340c0 0x7f0f6c13a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:40.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.691+0000 7f0f7134b640 10 monclient(hunting): picked mon.noname-a con 0x7f0f6c1340c0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:48:40.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.691+0000 7f0f7134b640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0f6c13d8b0 0x7f0f6c13a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:40.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.691+0000 7f0f7134b640 10 monclient(hunting): picked mon.noname-c con 0x7f0f6c13d8b0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:48:40.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.691+0000 7f0f7134b640 10 monclient(hunting): start opening mon connection 2024-09-16T03:48:40.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.691+0000 7f0f7134b640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f0f6c142b80 con 0x7f0f6c1340c0 2024-09-16T03:48:40.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.691+0000 7f0f7134b640 10 monclient(hunting): start opening mon connection 2024-09-16T03:48:40.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.691+0000 7f0f7134b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f0f6c142a00 con 0x7f0f6c06db90 2024-09-16T03:48:40.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.691+0000 7f0f7134b640 10 monclient(hunting): start opening mon connection 2024-09-16T03:48:40.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.691+0000 7f0f7134b640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f0f6c142d00 con 0x7f0f6c13d8b0 2024-09-16T03:48:40.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.691+0000 7f0f7134b640 10 monclient(hunting): _renew_subs 2024-09-16T03:48:40.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.691+0000 7f0f7134b640 10 monclient(hunting): authenticate will time out at 1367.660132s 2024-09-16T03:48:40.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.691+0000 7f0f6b7fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0f6c13d8b0 0x7f0f6c13a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:48:40.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.691+0000 7f0f6a7fc640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0f6c1340c0 0x7f0f6c13a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:48:40.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.691+0000 7f0f6b7fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0f6c13d8b0 0x7f0f6c13a7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:38090/0 (socket says 172.21.15.42:38090) 2024-09-16T03:48:40.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.691+0000 7f0f6b7fe640 1 -- 172.21.15.42:0/1861837650 learned_addr learned my addr 172.21.15.42:0/1861837650 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:48:40.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.691+0000 7f0f6a7fc640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0f6c1340c0 0x7f0f6c13a280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:44776/0 (socket says 172.21.15.42:44776) 2024-09-16T03:48:40.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.691+0000 7f0f6b7fe640 10 monclient(hunting): get_auth_request con 0x7f0f6c13d8b0 auth_method 0 2024-09-16T03:48:40.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.691+0000 7f0f6b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:48:40.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.691+0000 7f0f6b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:48:40.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.691+0000 7f0f6b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:48:40.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.691+0000 7f0f6a7fc640 10 monclient(hunting): get_auth_request con 0x7f0f6c1340c0 auth_method 0 2024-09-16T03:48:40.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.691+0000 7f0f6a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:48:40.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.691+0000 7f0f6a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:48:40.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.691+0000 7f0f6a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:48:40.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.691+0000 7f0f6b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:48:40.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.691+0000 7f0f6b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:48:40.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.691+0000 7f0f6b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:48:40.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.691+0000 7f0f6a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:48:40.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.691+0000 7f0f6a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:48:40.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.691+0000 7f0f6a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:48:40.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.691+0000 7f0f6b7fe640 10 monclient(hunting): handle_auth_done global_id 784124 payload 824 2024-09-16T03:48:40.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.691+0000 7f0f6affd640 1 --2- 172.21.15.42:0/1861837650 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0f6c06db90 0x7f0f6c139d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:48:40.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.691+0000 7f0f6b7fe640 1 -- 172.21.15.42:0/1861837650 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0f6c06db90 msgr2=0x7f0f6c139d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:48:40.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.691+0000 7f0f6b7fe640 1 --2- 172.21.15.42:0/1861837650 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0f6c06db90 0x7f0f6c139d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:40.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.691+0000 7f0f6b7fe640 1 -- 172.21.15.42:0/1861837650 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0f6c1340c0 msgr2=0x7f0f6c13a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:48:40.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.691+0000 7f0f6b7fe640 1 --2- 172.21.15.42:0/1861837650 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0f6c1340c0 0x7f0f6c13a280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:40.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.691+0000 7f0f6b7fe640 10 monclient: _finish_hunting 0 2024-09-16T03:48:40.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.691+0000 7f0f6b7fe640 1 monclient: found mon.noname-c 2024-09-16T03:48:40.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.691+0000 7f0f6b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:48:40.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.691+0000 7f0f6b7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:48:40.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.691+0000 7f0f6b7fe640 1 -- 172.21.15.42:0/1861837650 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f6c09d4c0 con 0x7f0f6c13d8b0 2024-09-16T03:48:40.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.691+0000 7f0f6a7fc640 1 --2- 172.21.15.42:0/1861837650 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0f6c1340c0 0x7f0f6c13a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:48:40.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.691+0000 7f0f6b7fe640 1 --2- 172.21.15.42:0/1861837650 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0f6c13d8b0 0x7f0f6c13a7c0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f0f64009870 tx=0x7f0f6401aa40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:48:40.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.691+0000 7f0f637fe640 10 client.?.objecter ms_handle_connect 0x7f0f6c13d8b0 2024-09-16T03:48:40.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.691+0000 7f0f637fe640 10 client.?.objecter resend_mon_ops 2024-09-16T03:48:40.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.691+0000 7f0f637fe640 1 -- 172.21.15.42:0/1861837650 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f640ba020 con 0x7f0f6c13d8b0 2024-09-16T03:48:40.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.691+0000 7f0f637fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:48:40.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.691+0000 7f0f637fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:48:40.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.691+0000 7f0f637fe640 10 monclient: dump: 2024-09-16T03:48:40.706 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:48:40.707 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:48:40.707 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:40.707 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:40.707 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:48:40.707 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:48:40.707 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:48:40.707 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:48:40.707 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:48:40.707 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:48:40.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.691+0000 7f0f637fe640 10 monclient: _finish_auth 0 2024-09-16T03:48:40.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.691+0000 7f0f637fe640 10 monclient: _check_auth_tickets 2024-09-16T03:48:40.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.691+0000 7f0f637fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:48:40.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.691+0000 7f0f637fe640 1 -- 172.21.15.42:0/1861837650 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0f640b6620 con 0x7f0f6c13d8b0 2024-09-16T03:48:40.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.691+0000 7f0f7134b640 5 monclient: authenticate success, global_id 784124 2024-09-16T03:48:40.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.691+0000 7f0f7134b640 10 monclient: _renew_subs 2024-09-16T03:48:40.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.691+0000 7f0f7134b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:48:40.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.691+0000 7f0f7134b640 1 -- 172.21.15.42:0/1861837650 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f6c0a8e10 con 0x7f0f6c13d8b0 2024-09-16T03:48:40.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.691+0000 7f0f637fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:48:40.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.691+0000 7f0f637fe640 1 -- 172.21.15.42:0/1861837650 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f640b0040 con 0x7f0f6c13d8b0 2024-09-16T03:48:40.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.691+0000 7f0f637fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:48:40.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.691+0000 7f0f637fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:48:40.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.691+0000 7f0f637fe640 10 monclient: dump: 2024-09-16T03:48:40.708 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:48:40.708 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:48:40.708 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:40.709 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:40.709 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:48:40.709 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:48:40.709 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:48:40.709 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:48:40.709 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:48:40.709 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:48:40.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.691+0000 7f0f7134b640 10 client.784124.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:48:40.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.691+0000 7f0f7134b640 10 monclient: _renew_subs 2024-09-16T03:48:40.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.691+0000 7f0f7134b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:48:40.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.691+0000 7f0f7134b640 1 -- 172.21.15.42:0/1861837650 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f6c09d1e0 con 0x7f0f6c13d8b0 2024-09-16T03:48:40.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.691+0000 7f0f7134b640 1 librados: init done 2024-09-16T03:48:40.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.691+0000 7f0f697fa640 4 set_mon_vals no callback set 2024-09-16T03:48:40.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.691+0000 7f0f7134b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:48:40.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.691+0000 7f0f7134b640 1 --2- 172.21.15.42:0/1861837650 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0f6c1cf6e0 0x7f0f6c1cfac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:40.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.691+0000 7f0f7134b640 10 monclient(hunting): start opening mon connection 2024-09-16T03:48:40.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.691+0000 7f0f7134b640 1 -- 172.21.15.42:0/1861837650 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f0f6c09c780 con 0x7f0f6c1cf6e0 2024-09-16T03:48:40.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.691+0000 7f0f6affd640 1 --2- 172.21.15.42:0/1861837650 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0f6c1cf6e0 0x7f0f6c1cfac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:48:40.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.691+0000 7f0f6affd640 10 monclient: get_auth_request con 0x7f0f6c1cf6e0 auth_method 0 2024-09-16T03:48:40.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.691+0000 7f0f6affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:48:40.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.691+0000 7f0f6affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:48:40.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.691+0000 7f0f6affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:48:40.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.691+0000 7f0f637fe640 1 -- 172.21.15.42:0/1861837650 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f0f640bd020 con 0x7f0f6c13d8b0 2024-09-16T03:48:40.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.691+0000 7f0f6affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:48:40.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.691+0000 7f0f6affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:48:40.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.691+0000 7f0f6affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:48:40.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.691+0000 7f0f6affd640 10 monclient(hunting): handle_auth_done global_id 764514 payload 824 2024-09-16T03:48:40.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.691+0000 7f0f637fe640 1 --2- 172.21.15.42:0/1861837650 >> v2:172.21.15.154:6812/254845342 conn(0x7f0f4c037810 0x7f0f4c039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:40.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.691+0000 7f0f6affd640 1 -- 172.21.15.42:0/1861837650 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0f6c06f6a0 con 0x7f0f6c1cf6e0 2024-09-16T03:48:40.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.691+0000 7f0f637fe640 1 -- 172.21.15.42:0/1861837650 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(216..216 src has 1..216) ==== 6980+0+0 (secure 0 0 0) 0x7f0f640e3670 con 0x7f0f6c13d8b0 2024-09-16T03:48:40.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.691+0000 7f0f637fe640 10 client.784124.objecter ms_dispatch 0x7f0f6c000ba0 osd_map(216..216 src has 1..216) 2024-09-16T03:48:40.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.691+0000 7f0f637fe640 3 client.784124.objecter handle_osd_map got epochs [216,216] > 0 2024-09-16T03:48:40.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.691+0000 7f0f637fe640 3 client.784124.objecter handle_osd_map decoding full epoch 216 2024-09-16T03:48:40.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.691+0000 7f0f637fe640 20 client.784124.objecter dump_active .. 0 homeless 2024-09-16T03:48:40.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.691+0000 7f0f637fe640 10 client.784124.objecter ms_handle_connect 0x7f0f6c1cf6e0 2024-09-16T03:48:40.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.691+0000 7f0f637fe640 10 client.784124.objecter resend_mon_ops 2024-09-16T03:48:40.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.691+0000 7f0f6affd640 1 --2- 172.21.15.42:0/1861837650 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0f6c1cf6e0 0x7f0f6c1cfac0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f0f5400d380 tx=0x7f0f54001e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:48:40.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.691+0000 7f0f6a7fc640 1 --2- 172.21.15.42:0/1861837650 >> v2:172.21.15.154:6812/254845342 conn(0x7f0f4c037810 0x7f0f4c039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:48:40.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.691+0000 7f0f637fe640 1 -- 172.21.15.42:0/1861837650 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f54018070 con 0x7f0f6c1cf6e0 2024-09-16T03:48:40.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.691+0000 7f0f637fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:48:40.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.691+0000 7f0f637fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:48:40.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.691+0000 7f0f637fe640 10 monclient: dump: 2024-09-16T03:48:40.712 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:48:40.712 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:48:40.712 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:40.712 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:40.712 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:48:40.712 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:48:40.712 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:48:40.712 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:48:40.712 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:48:40.713 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:48:40.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.691+0000 7f0f6a7fc640 10 monclient: get_auth_request con 0x7f0f4c037810 auth_method 0 2024-09-16T03:48:40.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.691+0000 7f0f637fe640 1 -- 172.21.15.42:0/1861837650 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f0f540074e0 con 0x7f0f6c1cf6e0 2024-09-16T03:48:40.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.691+0000 7f0f637fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:48:40.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.691+0000 7f0f637fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:48:40.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.691+0000 7f0f637fe640 1 -- 172.21.15.42:0/1861837650 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0f6c1cf6e0 msgr2=0x7f0f6c1cfac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:48:40.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.691+0000 7f0f637fe640 1 --2- 172.21.15.42:0/1861837650 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0f6c1cf6e0 0x7f0f6c1cfac0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f0f5400d380 tx=0x7f0f54001e80 comp rx=0 tx=0).stop 2024-09-16T03:48:40.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.691+0000 7f0f637fe640 1 -- 172.21.15.42:0/1861837650 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0f6c1cf6e0 msgr2=0x7f0f6c1cfac0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:48:40.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.691+0000 7f0f637fe640 1 --2- 172.21.15.42:0/1861837650 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0f6c1cf6e0 0x7f0f6c1cfac0 secure :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f0f5400d380 tx=0x7f0f54001e80 comp rx=0 tx=0).stop 2024-09-16T03:48:40.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.691+0000 7f0f6a7fc640 1 --2- 172.21.15.42:0/1861837650 >> v2:172.21.15.154:6812/254845342 conn(0x7f0f4c037810 0x7f0f4c039cd0 secure :-1 s=READY pgs=3494 cs=0 l=1 rev1=1 crypto rx=0x7f0f6c13b5e0 tx=0x7f0f5c003dd0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:48:40.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.691+0000 7f0f637fe640 10 client.784124.objecter ms_handle_connect 0x7f0f4c037810 2024-09-16T03:48:40.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.699+0000 7f0f7134b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:48:40.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.699+0000 7f0f7134b640 1 --2- 172.21.15.42:0/1861837650 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0f6c02c650 0x7f0f6c02ca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:40.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.699+0000 7f0f7134b640 10 monclient(hunting): start opening mon connection 2024-09-16T03:48:40.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.699+0000 7f0f7134b640 1 -- 172.21.15.42:0/1861837650 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f0f6c0a37d0 con 0x7f0f6c02c650 2024-09-16T03:48:40.716 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.699+0000 7f0f6affd640 1 --2- 172.21.15.42:0/1861837650 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0f6c02c650 0x7f0f6c02ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:48:40.716 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.699+0000 7f0f6affd640 10 monclient: get_auth_request con 0x7f0f6c02c650 auth_method 0 2024-09-16T03:48:40.716 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.699+0000 7f0f6affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:48:40.716 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.699+0000 7f0f6affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:48:40.716 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.699+0000 7f0f6affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:48:40.716 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.699+0000 7f0f6affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:48:40.716 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.699+0000 7f0f6affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:48:40.716 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.699+0000 7f0f6affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:48:40.716 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.699+0000 7f0f6affd640 10 monclient(hunting): handle_auth_done global_id 764517 payload 824 2024-09-16T03:48:40.716 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.699+0000 7f0f6affd640 1 -- 172.21.15.42:0/1861837650 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f0f6c1360f0 con 0x7f0f6c02c650 2024-09-16T03:48:40.716 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.699+0000 7f0f637fe640 10 client.784124.objecter ms_handle_connect 0x7f0f6c02c650 2024-09-16T03:48:40.717 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.699+0000 7f0f6affd640 1 --2- 172.21.15.42:0/1861837650 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0f6c02c650 0x7f0f6c02ca60 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f0f54015000 tx=0x7f0f54000ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:48:40.717 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.699+0000 7f0f637fe640 10 client.784124.objecter resend_mon_ops 2024-09-16T03:48:40.717 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.699+0000 7f0f637fe640 1 -- 172.21.15.42:0/1861837650 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f54018040 con 0x7f0f6c02c650 2024-09-16T03:48:40.717 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.699+0000 7f0f637fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:48:40.717 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.699+0000 7f0f637fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:48:40.717 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.699+0000 7f0f637fe640 10 monclient: dump: 2024-09-16T03:48:40.717 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:48:40.717 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:48:40.717 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:40.717 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:40.717 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:48:40.717 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:48:40.717 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:48:40.718 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:48:40.718 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:48:40.718 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:48:40.718 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.699+0000 7f0f637fe640 1 -- 172.21.15.42:0/1861837650 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4881 (secure 0 0 0) 0x7f0f54005e20 con 0x7f0f6c02c650 2024-09-16T03:48:40.718 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.699+0000 7f0f637fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:48:40.718 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.699+0000 7f0f637fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:48:40.718 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.699+0000 7f0f637fe640 1 -- 172.21.15.42:0/1861837650 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0f6c02c650 msgr2=0x7f0f6c02ca60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:48:40.718 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.699+0000 7f0f637fe640 1 --2- 172.21.15.42:0/1861837650 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0f6c02c650 0x7f0f6c02ca60 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f0f54015000 tx=0x7f0f54000ee0 comp rx=0 tx=0).stop 2024-09-16T03:48:40.718 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.699+0000 7f0f637fe640 1 -- 172.21.15.42:0/1861837650 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0f6c02c650 msgr2=0x7f0f6c02ca60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:48:40.718 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.699+0000 7f0f637fe640 1 --2- 172.21.15.42:0/1861837650 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0f6c02c650 0x7f0f6c02ca60 secure :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f0f54015000 tx=0x7f0f54000ee0 comp rx=0 tx=0).stop 2024-09-16T03:48:40.719 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:48:40.719 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:48:40.719 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:48:40.719 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "leader", 2024-09-16T03:48:40.719 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 60, 2024-09-16T03:48:40.719 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 111490, 2024-09-16T03:48:40.719 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:48:40.719 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:48:40.719 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T03:48:40.719 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T03:48:40.719 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:48:40.719 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 7, 2024-09-16T03:48:40.720 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:48:40.720 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:48:40.720 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:48:40.720 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:48:40.720 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:48:40.720 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:48:40.720 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:48:40.720 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:48:40.720 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:48:40.720 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:48:40.720 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:48:40.720 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:48:40.720 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:48:40.721 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:48:40.721 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:48:40.721 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:48:40.721 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:48:40.721 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:48:40.721 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:48:40.721 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:48:40.721 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:48:40.721 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:48:40.721 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:48:40.721 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:48:40.721 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:48:40.721 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:48:40.722 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:48:40.722 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:48:40.722 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:48:40.722 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:48:40.722 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:48:40.722 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:48:40.722 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:48:40.722 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:48:40.722 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:48:40.722 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:48:40.722 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:48:40.722 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:48:40.722 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:48:40.722 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:48:40.723 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:48:40.723 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:48:40.723 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:48:40.723 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:48:40.723 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:48:40.723 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:48:40.723 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:48:40.723 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:48:40.723 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:48:40.723 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:48:40.723 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:48:40.723 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:48:40.723 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:48:40.724 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:48:40.724 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:48:40.724 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:48:40.724 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:48:40.724 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:48:40.724 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:48:40.724 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:48:40.724 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:48:40.724 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:48:40.724 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:48:40.724 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:48:40.724 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:48:40.724 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:48:40.725 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:48:40.725 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:48:40.725 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:48:40.725 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:48:40.725 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:48:40.725 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:48:40.725 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:48:40.725 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:48:40.725 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:48:40.725 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:48:40.725 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:48:40.725 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:48:40.726 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:48:40.726 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:48:40.726 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:48:40.726 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:48:40.726 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:48:40.726 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:48:40.726 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:48:40.726 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:48:40.726 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:48:40.726 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:48:40.726 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:48:40.726 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:48:40.726 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:48:40.727 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:48:40.727 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:48:40.727 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:48:40.727 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:48:40.727 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:48:40.727 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:48:40.727 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:48:40.727 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:48:40.727 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:48:40.727 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:48:40.727 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:48:40.727 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:48:40.727 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:48:40.727 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:48:40.728 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:48:40.728 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:48:40.728 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:48:40.728 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:48:40.728 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:48:40.728 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:48:40.728 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:48:40.728 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:48:40.728 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:48:40.728 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:48:40.728 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:48:40.728 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:48:40.728 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:48:40.728 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:48:40.729 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:48:40.729 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:48:40.729 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:48:40.729 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:48:40.729 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:48:40.729 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:48:40.729 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:48:40.729 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:48:40.729 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:48:40.729 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:48:40.729 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:48:40.729 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:48:40.729 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:48:40.729 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:48:40.730 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:48:40.730 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:48:40.730 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:48:40.730 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:48:40.730 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:48:40.730 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:48:40.730 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:48:40.730 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:48:40.730 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T03:48:40.730 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:48:40.730 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:48:40.730 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:48:40.730 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 5 2024-09-16T03:48:40.731 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:48:40.731 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:48:40.731 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:48:40.731 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:48:40.731 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:48:40.731 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:48:40.731 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 6 2024-09-16T03:48:40.731 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:48:40.731 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:48:40.731 INFO:teuthology.orchestra.run.smithi042.stdout: "mgr": [ 2024-09-16T03:48:40.731 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:48:40.731 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:48:40.731 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:48:40.731 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:48:40.732 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:48:40.732 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:48:40.732 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:48:40.732 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:48:40.732 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:48:40.733 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.699+0000 7f0f7134b640 10 librados: watch_flush enter 2024-09-16T03:48:40.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.699+0000 7f0f7134b640 10 librados: watch_flush exit 2024-09-16T03:48:40.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.699+0000 7f0f7134b640 20 client.784124.objecter shutdown clearing up homeless session... 2024-09-16T03:48:40.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.699+0000 7f0f7134b640 10 client.784124.objecter successfully canceled tick 2024-09-16T03:48:40.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.699+0000 7f0f7134b640 1 -- 172.21.15.42:0/1861837650 >> v2:172.21.15.154:6812/254845342 conn(0x7f0f4c037810 msgr2=0x7f0f4c039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:48:40.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.699+0000 7f0f7134b640 1 --2- 172.21.15.42:0/1861837650 >> v2:172.21.15.154:6812/254845342 conn(0x7f0f4c037810 0x7f0f4c039cd0 secure :-1 s=READY pgs=3494 cs=0 l=1 rev1=1 crypto rx=0x7f0f6c13b5e0 tx=0x7f0f5c003dd0 comp rx=0 tx=0).stop 2024-09-16T03:48:40.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.699+0000 7f0f7134b640 10 monclient: shutdown 2024-09-16T03:48:40.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.699+0000 7f0f7134b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:48:40.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.699+0000 7f0f7134b640 1 -- 172.21.15.42:0/1861837650 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0f6c13d8b0 msgr2=0x7f0f6c13a7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:48:40.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.699+0000 7f0f7134b640 1 --2- 172.21.15.42:0/1861837650 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0f6c13d8b0 0x7f0f6c13a7c0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f0f64009870 tx=0x7f0f6401aa40 comp rx=0 tx=0).stop 2024-09-16T03:48:40.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.699+0000 7f0f6b7fe640 1 -- 172.21.15.42:0/1861837650 reap_dead start 2024-09-16T03:48:40.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.699+0000 7f0f6b7fe640 1 -- 172.21.15.42:0/1861837650 reap_dead start 2024-09-16T03:48:40.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.699+0000 7f0f7134b640 1 -- 172.21.15.42:0/1861837650 shutdown_connections 2024-09-16T03:48:40.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.699+0000 7f0f7134b640 1 -- 172.21.15.42:0/1861837650 >> 172.21.15.42:0/1861837650 conn(0x7f0f6c0a9fb0 msgr2=0x7f0f6c13d1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:48:40.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.699+0000 7f0f7134b640 1 -- 172.21.15.42:0/1861837650 shutdown_connections 2024-09-16T03:48:40.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.699+0000 7f0f7134b640 1 -- 172.21.15.42:0/1861837650 wait complete. 2024-09-16T03:48:40.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.699+0000 7f0f7134b640 1 librados: shutdown 2024-09-16T03:48:40.735 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:48:40.735 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T03:48:40.735 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T03:48:40.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.851+0000 7f5b50599640 10 monclient: get_monmap_and_config 2024-09-16T03:48:40.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.851+0000 7f5b50599640 10 monclient: build_initial_monmap 2024-09-16T03:48:40.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.851+0000 7f5b50599640 10 monclient: monmap: 2024-09-16T03:48:40.853 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:48:40.854 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:48:40.854 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:48:40.852485+0000 2024-09-16T03:48:40.854 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:48:40.852485+0000 2024-09-16T03:48:40.854 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:48:40.854 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:48:40.854 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:48:40.854 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:48:40.854 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:48:40.854 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:48:40.854 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:48:40.854 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:48:40.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.851+0000 7f5b50599640 1 Processor -- start 2024-09-16T03:48:40.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.851+0000 7f5b50599640 1 -- start start 2024-09-16T03:48:40.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.851+0000 7f5b50599640 10 monclient: init 2024-09-16T03:48:40.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.851+0000 7f5b50599640 10 monclient: _reopen_session rank -1 2024-09-16T03:48:40.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.851+0000 7f5b50599640 10 monclient: _add_conns ranks=[0,3,1,2,4] 2024-09-16T03:48:40.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.851+0000 7f5b50599640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5b48133a60 addr v1:172.21.15.42:6789/0 2024-09-16T03:48:40.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.851+0000 7f5b50599640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f5b480a1be0 0x7f5b4809ca60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:40.857 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.851+0000 7f5b50599640 10 monclient(hunting): picked mon.noname-a con 0x7f5b480a1be0 addr v2:172.21.15.42:3300/0 2024-09-16T03:48:40.857 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.851+0000 7f5b50599640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5b48138a40 addr v1:172.21.15.154:6789/0 2024-09-16T03:48:40.857 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.851+0000 7f5b50599640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5b480a11a0 con 0x7f5b48133a60 2024-09-16T03:48:40.857 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.851+0000 7f5b50599640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5b480a18a0 con 0x7f5b48138a40 2024-09-16T03:48:40.857 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.851+0000 7f5b50599640 10 monclient(hunting): start opening mon connection 2024-09-16T03:48:40.857 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.851+0000 7f5b50599640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f5b4809d6e0 con 0x7f5b480a1be0 2024-09-16T03:48:40.857 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.851+0000 7f5b50599640 10 monclient(hunting): _renew_subs 2024-09-16T03:48:40.857 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.851+0000 7f5b50599640 10 monclient(hunting): authenticate will time out at 1367.821951s 2024-09-16T03:48:40.857 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.851+0000 7f5b4db34640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f5b480a1be0 0x7f5b4809ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:48:40.857 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.851+0000 7f5b4db34640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f5b480a1be0 0x7f5b4809ca60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:44814/0 (socket says 172.21.15.42:44814) 2024-09-16T03:48:40.857 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.851+0000 7f5b4db34640 1 -- 172.21.15.42:0/3197240845 learned_addr learned my addr 172.21.15.42:0/3197240845 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:48:40.857 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.851+0000 7f5b4db34640 10 monclient(hunting): get_auth_request con 0x7f5b480a1be0 auth_method 0 2024-09-16T03:48:40.857 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.851+0000 7f5b4db34640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:48:40.857 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.851+0000 7f5b4db34640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:48:40.858 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.851+0000 7f5b4db34640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:48:40.858 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.851+0000 7f5b4db34640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:48:40.858 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.851+0000 7f5b4db34640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:48:40.858 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.851+0000 7f5b4db34640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:48:40.858 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.851+0000 7f5b4db34640 10 monclient(hunting): handle_auth_done global_id 764520 payload 274 2024-09-16T03:48:40.858 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.851+0000 7f5b4db34640 1 -- 172.21.15.42:0/3197240845 >> v1:172.21.15.154:6789/0 conn(0x7f5b48138a40 legacy=0x7f5b4813ae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:48:40.858 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.851+0000 7f5b4db34640 1 -- 172.21.15.42:0/3197240845 >> v1:172.21.15.42:6789/0 conn(0x7f5b48133a60 legacy=0x7f5b48133e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:48:40.858 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.851+0000 7f5b4db34640 10 monclient: _finish_hunting 0 2024-09-16T03:48:40.858 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.851+0000 7f5b4db34640 1 monclient: found mon.noname-a 2024-09-16T03:48:40.858 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.851+0000 7f5b4db34640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:48:40.858 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.851+0000 7f5b4db34640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:48:40.858 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.851+0000 7f5b4db34640 1 -- 172.21.15.42:0/3197240845 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b480a15e0 con 0x7f5b480a1be0 2024-09-16T03:48:40.865 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.855+0000 7f5b4db34640 1 --2- 172.21.15.42:0/3197240845 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5b480a1be0 0x7f5b4809ca60 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f5b3c009870 tx=0x7f5b3c01c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=badd9f6eeca8aff4 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:48:40.865 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.855+0000 7f5b4d333640 1 -- 172.21.15.42:0/3197240845 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b3c0b0070 con 0x7f5b480a1be0 2024-09-16T03:48:40.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.855+0000 7f5b4d333640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:48:40.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.855+0000 7f5b4d333640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:48:40.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.855+0000 7f5b4d333640 10 monclient: dump: 2024-09-16T03:48:40.866 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:48:40.866 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:48:40.866 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:40.866 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:40.866 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:48:40.866 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:48:40.866 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:48:40.866 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:48:40.866 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:48:40.866 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:48:40.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.855+0000 7f5b4d333640 10 monclient: _finish_auth 0 2024-09-16T03:48:40.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.855+0000 7f5b4d333640 10 monclient: _check_auth_tickets 2024-09-16T03:48:40.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.855+0000 7f5b4d333640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:48:40.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.855+0000 7f5b50599640 5 monclient: authenticate success, global_id 764520 2024-09-16T03:48:40.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.855+0000 7f5b50599640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:48:40.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.855+0000 7f5b4d333640 1 -- 172.21.15.42:0/3197240845 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5b3c0ab600 con 0x7f5b480a1be0 2024-09-16T03:48:40.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.855+0000 7f5b4d333640 10 monclient: handle_config config(0 keys) 2024-09-16T03:48:40.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.855+0000 7f5b50599640 10 monclient: get_monmap_and_config success 2024-09-16T03:48:40.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.855+0000 7f5b50599640 4 set_mon_vals no callback set 2024-09-16T03:48:40.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.855+0000 7f5b4d333640 1 -- 172.21.15.42:0/3197240845 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b3c0ab9d0 con 0x7f5b480a1be0 2024-09-16T03:48:40.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.855+0000 7f5b4d333640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:48:40.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.855+0000 7f5b4d333640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:48:40.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.855+0000 7f5b4d333640 10 monclient: dump: 2024-09-16T03:48:40.868 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:48:40.868 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:48:40.868 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:40.868 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:40.868 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:48:40.869 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:48:40.869 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:48:40.869 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:48:40.869 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:48:40.869 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:48:40.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.855+0000 7f5b50599640 10 monclient: shutdown 2024-09-16T03:48:40.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.855+0000 7f5b50599640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:48:40.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.855+0000 7f5b50599640 1 -- 172.21.15.42:0/3197240845 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5b480a1be0 msgr2=0x7f5b4809ca60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:48:40.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.855+0000 7f5b50599640 1 --2- 172.21.15.42:0/3197240845 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5b480a1be0 0x7f5b4809ca60 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f5b3c009870 tx=0x7f5b3c01c3e0 comp rx=0 tx=0).stop 2024-09-16T03:48:40.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.855+0000 7f5b50599640 1 -- 172.21.15.42:0/3197240845 shutdown_connections 2024-09-16T03:48:40.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.855+0000 7f5b50599640 1 --2- 172.21.15.42:0/3197240845 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5b480a1be0 0x7f5b4809ca60 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:40.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.855+0000 7f5b50599640 1 -- 172.21.15.42:0/3197240845 >> 172.21.15.42:0/3197240845 conn(0x7f5b480a6210 msgr2=0x7f5b480a6620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:48:40.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.855+0000 7f5b50599640 1 -- 172.21.15.42:0/3197240845 shutdown_connections 2024-09-16T03:48:40.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.855+0000 7f5b50599640 1 -- 172.21.15.42:0/3197240845 wait complete. 2024-09-16T03:48:40.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.855+0000 7f5b50599640 10 monclient: build_initial_monmap 2024-09-16T03:48:40.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.855+0000 7f5b50599640 10 monclient: monmap: 2024-09-16T03:48:40.872 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:48:40.872 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:48:40.872 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:48:40.872 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:48:40.872 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:48:40.872 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:48:40.872 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:48:40.872 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:48:40.873 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:48:40.873 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:48:40.873 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.855+0000 7f5b50599640 1 librados: starting msgr at 2024-09-16T03:48:40.873 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.855+0000 7f5b50599640 1 librados: starting objecter 2024-09-16T03:48:40.873 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.855+0000 7f5b50599640 1 Processor -- start 2024-09-16T03:48:40.873 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.855+0000 7f5b50599640 1 -- start start 2024-09-16T03:48:40.873 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.855+0000 7f5b50599640 1 librados: setting wanted keys 2024-09-16T03:48:40.873 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.855+0000 7f5b50599640 1 librados: calling monclient init 2024-09-16T03:48:40.873 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.855+0000 7f5b50599640 10 monclient: init 2024-09-16T03:48:40.873 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.855+0000 7f5b50599640 10 monclient: _reopen_session rank -1 2024-09-16T03:48:40.873 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.855+0000 7f5b50599640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:48:40.873 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.855+0000 7f5b50599640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5b480a1be0 0x7f5b481cf010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:40.873 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.855+0000 7f5b50599640 10 monclient(hunting): picked mon.noname-c con 0x7f5b480a1be0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:48:40.873 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.855+0000 7f5b50599640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5b48133a60 0x7f5b48132b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:40.874 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.855+0000 7f5b50599640 10 monclient(hunting): picked mon.noname-b con 0x7f5b48133a60 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:48:40.874 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.855+0000 7f5b50599640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5b48138a40 0x7f5b48135a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:40.874 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.855+0000 7f5b50599640 10 monclient(hunting): picked mon.noname-a con 0x7f5b48138a40 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:48:40.874 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.855+0000 7f5b50599640 10 monclient(hunting): start opening mon connection 2024-09-16T03:48:40.874 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.855+0000 7f5b50599640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f5b48075a70 con 0x7f5b48138a40 2024-09-16T03:48:40.874 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.855+0000 7f5b50599640 10 monclient(hunting): start opening mon connection 2024-09-16T03:48:40.874 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.855+0000 7f5b50599640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f5b480758f0 con 0x7f5b48133a60 2024-09-16T03:48:40.874 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.855+0000 7f5b50599640 10 monclient(hunting): start opening mon connection 2024-09-16T03:48:40.874 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.855+0000 7f5b50599640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f5b48075bf0 con 0x7f5b480a1be0 2024-09-16T03:48:40.874 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.855+0000 7f5b50599640 10 monclient(hunting): _renew_subs 2024-09-16T03:48:40.874 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.855+0000 7f5b50599640 10 monclient(hunting): authenticate will time out at 1367.826959s 2024-09-16T03:48:40.874 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.855+0000 7f5b4e335640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5b480a1be0 0x7f5b481cf010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:48:40.874 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.855+0000 7f5b4eb36640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5b48138a40 0x7f5b48135a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:48:40.875 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.855+0000 7f5b4e335640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5b480a1be0 0x7f5b481cf010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:38100/0 (socket says 172.21.15.42:38100) 2024-09-16T03:48:40.875 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.855+0000 7f5b4e335640 1 -- 172.21.15.42:0/2765343386 learned_addr learned my addr 172.21.15.42:0/2765343386 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:48:40.875 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.855+0000 7f5b4eb36640 10 monclient(hunting): get_auth_request con 0x7f5b48138a40 auth_method 0 2024-09-16T03:48:40.875 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.855+0000 7f5b4eb36640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:48:40.875 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.855+0000 7f5b4eb36640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:48:40.875 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.855+0000 7f5b4eb36640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:48:40.875 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.855+0000 7f5b4e335640 10 monclient(hunting): get_auth_request con 0x7f5b480a1be0 auth_method 0 2024-09-16T03:48:40.876 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.855+0000 7f5b4e335640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:48:40.876 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.855+0000 7f5b4e335640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:48:40.876 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.855+0000 7f5b4e335640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:48:40.876 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.855+0000 7f5b4eb36640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:48:40.876 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.855+0000 7f5b4eb36640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:48:40.876 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.855+0000 7f5b4eb36640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:48:40.876 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.855+0000 7f5b4e335640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:48:40.876 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.855+0000 7f5b4e335640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:48:40.876 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.855+0000 7f5b4e335640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:48:40.876 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.855+0000 7f5b4db34640 1 --2- 172.21.15.42:0/2765343386 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5b48133a60 0x7f5b48132b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:48:40.876 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.855+0000 7f5b4eb36640 10 monclient(hunting): handle_auth_done global_id 764526 payload 824 2024-09-16T03:48:40.876 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.855+0000 7f5b4eb36640 1 -- 172.21.15.42:0/2765343386 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5b480a1be0 msgr2=0x7f5b481cf010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:48:40.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.855+0000 7f5b4eb36640 1 --2- 172.21.15.42:0/2765343386 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5b480a1be0 0x7f5b481cf010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:40.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.855+0000 7f5b4eb36640 1 -- 172.21.15.42:0/2765343386 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5b48133a60 msgr2=0x7f5b48132b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:48:40.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.855+0000 7f5b4eb36640 1 --2- 172.21.15.42:0/2765343386 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5b48133a60 0x7f5b48132b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:40.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.855+0000 7f5b4eb36640 10 monclient: _finish_hunting 0 2024-09-16T03:48:40.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.855+0000 7f5b4eb36640 1 monclient: found mon.noname-a 2024-09-16T03:48:40.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.855+0000 7f5b4eb36640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:48:40.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.855+0000 7f5b4eb36640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:48:40.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.855+0000 7f5b4eb36640 1 -- 172.21.15.42:0/2765343386 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b481cf550 con 0x7f5b48138a40 2024-09-16T03:48:40.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.855+0000 7f5b4e335640 1 --2- 172.21.15.42:0/2765343386 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5b480a1be0 0x7f5b481cf010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:48:40.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.855+0000 7f5b4eb36640 1 --2- 172.21.15.42:0/2765343386 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5b48138a40 0x7f5b48135a30 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f5b4000c750 tx=0x7f5b4000cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:48:40.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.855+0000 7f5b3affd640 10 client.?.objecter ms_handle_connect 0x7f5b48138a40 2024-09-16T03:48:40.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.855+0000 7f5b3affd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:48:40.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.855+0000 7f5b3affd640 1 -- 172.21.15.42:0/2765343386 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b40016020 con 0x7f5b48138a40 2024-09-16T03:48:40.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.855+0000 7f5b3affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:48:40.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.855+0000 7f5b3affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:48:40.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.855+0000 7f5b3affd640 10 monclient: dump: 2024-09-16T03:48:40.878 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:48:40.878 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:48:40.878 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:40.878 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:40.878 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:48:40.878 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:48:40.878 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:48:40.878 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:48:40.878 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:48:40.878 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:48:40.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.855+0000 7f5b3affd640 10 monclient: _finish_auth 0 2024-09-16T03:48:40.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.855+0000 7f5b3affd640 10 monclient: _check_auth_tickets 2024-09-16T03:48:40.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.855+0000 7f5b3affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:48:40.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.855+0000 7f5b3affd640 1 -- 172.21.15.42:0/2765343386 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5b4000a850 con 0x7f5b48138a40 2024-09-16T03:48:40.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.855+0000 7f5b50599640 5 monclient: authenticate success, global_id 764526 2024-09-16T03:48:40.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.855+0000 7f5b50599640 10 monclient: _renew_subs 2024-09-16T03:48:40.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.855+0000 7f5b50599640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:48:40.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.855+0000 7f5b50599640 1 -- 172.21.15.42:0/2765343386 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b4811e500 con 0x7f5b48138a40 2024-09-16T03:48:40.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.855+0000 7f5b3affd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:48:40.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.855+0000 7f5b3affd640 1 -- 172.21.15.42:0/2765343386 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b4000ac80 con 0x7f5b48138a40 2024-09-16T03:48:40.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.855+0000 7f5b3affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:48:40.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.855+0000 7f5b3affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:48:40.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.855+0000 7f5b3affd640 10 monclient: dump: 2024-09-16T03:48:40.879 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:48:40.879 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:48:40.879 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:40.880 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:40.880 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:48:40.880 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:48:40.880 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:48:40.880 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:48:40.880 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:48:40.880 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:48:40.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.855+0000 7f5b50599640 10 client.764526.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:48:40.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.855+0000 7f5b4cb32640 4 set_mon_vals no callback set 2024-09-16T03:48:40.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.859+0000 7f5b50599640 10 monclient: _renew_subs 2024-09-16T03:48:40.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.859+0000 7f5b50599640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:48:40.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.859+0000 7f5b50599640 1 -- 172.21.15.42:0/2765343386 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b4811df40 con 0x7f5b48138a40 2024-09-16T03:48:40.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.859+0000 7f5b50599640 1 librados: init done 2024-09-16T03:48:40.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.859+0000 7f5b50599640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:48:40.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.859+0000 7f5b3affd640 1 -- 172.21.15.42:0/2765343386 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f5b40019020 con 0x7f5b48138a40 2024-09-16T03:48:40.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.859+0000 7f5b50599640 1 --2- 172.21.15.42:0/2765343386 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5b481330a0 0x7f5b48135f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:40.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.859+0000 7f5b50599640 10 monclient(hunting): start opening mon connection 2024-09-16T03:48:40.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.859+0000 7f5b50599640 1 -- 172.21.15.42:0/2765343386 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f5b48133e40 con 0x7f5b481330a0 2024-09-16T03:48:40.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.859+0000 7f5b4e335640 1 --2- 172.21.15.42:0/2765343386 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5b481330a0 0x7f5b48135f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:48:40.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.859+0000 7f5b4e335640 10 monclient: get_auth_request con 0x7f5b481330a0 auth_method 0 2024-09-16T03:48:40.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.859+0000 7f5b4e335640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:48:40.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.859+0000 7f5b4e335640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:48:40.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.859+0000 7f5b4e335640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:48:40.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.859+0000 7f5b3affd640 1 --2- 172.21.15.42:0/2765343386 >> v2:172.21.15.154:6812/254845342 conn(0x7f5b2c037810 0x7f5b2c039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:40.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.859+0000 7f5b4e335640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:48:40.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.859+0000 7f5b4e335640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:48:40.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.859+0000 7f5b4e335640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:48:40.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.859+0000 7f5b3affd640 1 -- 172.21.15.42:0/2765343386 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(216..216 src has 1..216) ==== 6980+0+0 (secure 0 0 0) 0x7f5b40049250 con 0x7f5b48138a40 2024-09-16T03:48:40.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.859+0000 7f5b3affd640 10 client.764526.objecter ms_dispatch 0x7f5b48000ba0 osd_map(216..216 src has 1..216) 2024-09-16T03:48:40.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.859+0000 7f5b3affd640 3 client.764526.objecter handle_osd_map got epochs [216,216] > 0 2024-09-16T03:48:40.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.859+0000 7f5b3affd640 3 client.764526.objecter handle_osd_map decoding full epoch 216 2024-09-16T03:48:40.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.859+0000 7f5b3affd640 20 client.764526.objecter dump_active .. 0 homeless 2024-09-16T03:48:40.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.859+0000 7f5b4e335640 10 monclient(hunting): handle_auth_done global_id 784130 payload 824 2024-09-16T03:48:40.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.859+0000 7f5b4e335640 1 -- 172.21.15.42:0/2765343386 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5b4811caf0 con 0x7f5b481330a0 2024-09-16T03:48:40.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.859+0000 7f5b4db34640 1 --2- 172.21.15.42:0/2765343386 >> v2:172.21.15.154:6812/254845342 conn(0x7f5b2c037810 0x7f5b2c039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:48:40.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.859+0000 7f5b4db34640 10 monclient: get_auth_request con 0x7f5b2c037810 auth_method 0 2024-09-16T03:48:40.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.859+0000 7f5b3affd640 10 client.764526.objecter ms_handle_connect 0x7f5b481330a0 2024-09-16T03:48:40.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.859+0000 7f5b3affd640 10 client.764526.objecter resend_mon_ops 2024-09-16T03:48:40.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.859+0000 7f5b4e335640 1 --2- 172.21.15.42:0/2765343386 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5b481330a0 0x7f5b48135f70 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f5b4400fae0 tx=0x7f5b44001e00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:48:40.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.859+0000 7f5b3affd640 1 -- 172.21.15.42:0/2765343386 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b440052e0 con 0x7f5b481330a0 2024-09-16T03:48:40.883 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.859+0000 7f5b3affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:48:40.883 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.859+0000 7f5b3affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:48:40.883 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.859+0000 7f5b3affd640 10 monclient: dump: 2024-09-16T03:48:40.883 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:48:40.883 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:48:40.883 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:40.883 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:40.883 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:48:40.883 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:48:40.883 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:48:40.883 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:48:40.883 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:48:40.883 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:48:40.883 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.859+0000 7f5b3affd640 1 -- 172.21.15.42:0/2765343386 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f5b44018070 con 0x7f5b481330a0 2024-09-16T03:48:40.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.859+0000 7f5b3affd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:48:40.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.859+0000 7f5b3affd640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:48:40.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.859+0000 7f5b3affd640 1 -- 172.21.15.42:0/2765343386 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5b481330a0 msgr2=0x7f5b48135f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:48:40.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.859+0000 7f5b3affd640 1 --2- 172.21.15.42:0/2765343386 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5b481330a0 0x7f5b48135f70 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f5b4400fae0 tx=0x7f5b44001e00 comp rx=0 tx=0).stop 2024-09-16T03:48:40.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.859+0000 7f5b3affd640 1 -- 172.21.15.42:0/2765343386 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5b481330a0 msgr2=0x7f5b48135f70 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:48:40.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.859+0000 7f5b3affd640 1 --2- 172.21.15.42:0/2765343386 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5b481330a0 0x7f5b48135f70 secure :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f5b4400fae0 tx=0x7f5b44001e00 comp rx=0 tx=0).stop 2024-09-16T03:48:40.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.859+0000 7f5b4db34640 1 --2- 172.21.15.42:0/2765343386 >> v2:172.21.15.154:6812/254845342 conn(0x7f5b2c037810 0x7f5b2c039cd0 secure :-1 s=READY pgs=3495 cs=0 l=1 rev1=1 crypto rx=0x7f5b3c0ad000 tx=0x7f5b3c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:48:40.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.859+0000 7f5b3affd640 10 client.764526.objecter ms_handle_connect 0x7f5b2c037810 2024-09-16T03:48:40.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.867+0000 7f5b50599640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:48:40.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.867+0000 7f5b50599640 1 --2- 172.21.15.42:0/2765343386 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5b481364b0 0x7f5b48138e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:40.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.867+0000 7f5b50599640 10 monclient(hunting): start opening mon connection 2024-09-16T03:48:40.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.867+0000 7f5b50599640 1 -- 172.21.15.42:0/2765343386 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f5b481368c0 con 0x7f5b481364b0 2024-09-16T03:48:40.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.867+0000 7f5b4e335640 1 --2- 172.21.15.42:0/2765343386 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5b481364b0 0x7f5b48138e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:48:40.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.867+0000 7f5b4e335640 10 monclient: get_auth_request con 0x7f5b481364b0 auth_method 0 2024-09-16T03:48:40.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.867+0000 7f5b4e335640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:48:40.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.867+0000 7f5b4e335640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:48:40.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.867+0000 7f5b4e335640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:48:40.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.867+0000 7f5b4e335640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:48:40.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.867+0000 7f5b4e335640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:48:40.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.867+0000 7f5b4e335640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:48:40.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.867+0000 7f5b4e335640 10 monclient(hunting): handle_auth_done global_id 784133 payload 824 2024-09-16T03:48:40.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.867+0000 7f5b4e335640 1 -- 172.21.15.42:0/2765343386 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f5b4806b490 con 0x7f5b481364b0 2024-09-16T03:48:40.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.867+0000 7f5b4e335640 1 --2- 172.21.15.42:0/2765343386 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5b481364b0 0x7f5b48138e20 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f5b4400fae0 tx=0x7f5b44000ee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:48:40.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.867+0000 7f5b3affd640 10 client.764526.objecter ms_handle_connect 0x7f5b481364b0 2024-09-16T03:48:40.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.867+0000 7f5b3affd640 10 client.764526.objecter resend_mon_ops 2024-09-16T03:48:40.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.867+0000 7f5b3affd640 1 -- 172.21.15.42:0/2765343386 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b4401ecb0 con 0x7f5b481364b0 2024-09-16T03:48:40.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.867+0000 7f5b3affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:48:40.888 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.867+0000 7f5b3affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:48:40.888 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.867+0000 7f5b3affd640 10 monclient: dump: 2024-09-16T03:48:40.888 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:48:40.888 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:48:40.888 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:40.888 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:40.888 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:48:40.888 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:48:40.888 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:48:40.888 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:48:40.888 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:48:40.888 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:48:40.888 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.867+0000 7f5b3affd640 1 -- 172.21.15.42:0/2765343386 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4711 (secure 0 0 0) 0x7f5b44003600 con 0x7f5b481364b0 2024-09-16T03:48:40.889 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.867+0000 7f5b3affd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:48:40.889 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.867+0000 7f5b3affd640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:48:40.889 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.867+0000 7f5b3affd640 1 -- 172.21.15.42:0/2765343386 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5b481364b0 msgr2=0x7f5b48138e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:48:40.889 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.867+0000 7f5b3affd640 1 --2- 172.21.15.42:0/2765343386 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5b481364b0 0x7f5b48138e20 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f5b4400fae0 tx=0x7f5b44000ee0 comp rx=0 tx=0).stop 2024-09-16T03:48:40.889 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.867+0000 7f5b3affd640 1 -- 172.21.15.42:0/2765343386 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5b481364b0 msgr2=0x7f5b48138e20 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:48:40.889 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.867+0000 7f5b3affd640 1 --2- 172.21.15.42:0/2765343386 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5b481364b0 0x7f5b48138e20 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:40.890 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:48:40.890 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:48:40.890 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:48:40.890 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T03:48:40.890 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 60, 2024-09-16T03:48:40.890 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 12767, 2024-09-16T03:48:40.891 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:48:40.891 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:48:40.891 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T03:48:40.891 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T03:48:40.892 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:48:40.892 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 7, 2024-09-16T03:48:40.892 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:48:40.892 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:48:40.892 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:48:40.892 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:48:40.892 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:48:40.892 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:48:40.892 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:48:40.892 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:48:40.892 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:48:40.892 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:48:40.892 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:48:40.893 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:48:40.893 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:48:40.893 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:48:40.893 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:48:40.893 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:48:40.893 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:48:40.893 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:48:40.893 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:48:40.893 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:48:40.894 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:48:40.894 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:48:40.894 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:48:40.894 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:48:40.894 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:48:40.894 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:48:40.894 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:48:40.894 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:48:40.894 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:48:40.894 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:48:40.894 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:48:40.894 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:48:40.894 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:48:40.895 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:48:40.895 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:48:40.895 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:48:40.895 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:48:40.895 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:48:40.895 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:48:40.895 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:48:40.895 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:48:40.895 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:48:40.895 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:48:40.895 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:48:40.895 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:48:40.895 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:48:40.895 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:48:40.896 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:48:40.896 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:48:40.896 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:48:40.896 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:48:40.896 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:48:40.896 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:48:40.896 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:48:40.896 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:48:40.896 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:48:40.896 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:48:40.896 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:48:40.896 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:48:40.896 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:48:40.897 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:48:40.897 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:48:40.897 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:48:40.897 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:48:40.897 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:48:40.897 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:48:40.897 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:48:40.897 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:48:40.897 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:48:40.897 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:48:40.897 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:48:40.897 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:48:40.898 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:48:40.898 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:48:40.898 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:48:40.898 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:48:40.898 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:48:40.898 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:48:40.898 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:48:40.898 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:48:40.898 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:48:40.898 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:48:40.898 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:48:40.898 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:48:40.898 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:48:40.899 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:48:40.899 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:48:40.899 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:48:40.899 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:48:40.899 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:48:40.899 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:48:40.899 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:48:40.899 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:48:40.899 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:48:40.899 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:48:40.899 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:48:40.899 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:48:40.899 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:48:40.900 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:48:40.900 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:48:40.900 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:48:40.900 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:48:40.900 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:48:40.901 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:48:40.901 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:48:40.901 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:48:40.901 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:48:40.901 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:48:40.901 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:48:40.901 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:48:40.901 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:48:40.901 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:48:40.901 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:48:40.901 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:48:40.901 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:48:40.902 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:48:40.902 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:48:40.902 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:48:40.902 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:48:40.902 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:48:40.902 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:48:40.902 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:48:40.902 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:48:40.902 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:48:40.902 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:48:40.902 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:48:40.902 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:48:40.902 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:48:40.902 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:48:40.903 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:48:40.903 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:48:40.903 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:48:40.903 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:48:40.903 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:48:40.903 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:48:40.903 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:48:40.903 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:48:40.903 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:48:40.904 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:48:40.904 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:48:40.904 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:48:40.904 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T03:48:40.904 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:48:40.904 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:48:40.904 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:48:40.904 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:48:40.904 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:48:40.904 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:48:40.905 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:48:40.905 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:48:40.905 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:48:40.905 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:48:40.905 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:48:40.905 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:48:40.905 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:48:40.905 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:48:40.905 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:48:40.905 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:48:40.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.867+0000 7f5b50599640 10 librados: watch_flush enter 2024-09-16T03:48:40.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.867+0000 7f5b50599640 10 librados: watch_flush exit 2024-09-16T03:48:40.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.867+0000 7f5b50599640 20 client.764526.objecter shutdown clearing up homeless session... 2024-09-16T03:48:40.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.867+0000 7f5b50599640 10 client.764526.objecter successfully canceled tick 2024-09-16T03:48:40.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.867+0000 7f5b50599640 1 -- 172.21.15.42:0/2765343386 >> v2:172.21.15.154:6812/254845342 conn(0x7f5b2c037810 msgr2=0x7f5b2c039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:48:40.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.867+0000 7f5b50599640 1 --2- 172.21.15.42:0/2765343386 >> v2:172.21.15.154:6812/254845342 conn(0x7f5b2c037810 0x7f5b2c039cd0 secure :-1 s=READY pgs=3495 cs=0 l=1 rev1=1 crypto rx=0x7f5b3c0ad000 tx=0x7f5b3c008040 comp rx=0 tx=0).stop 2024-09-16T03:48:40.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.867+0000 7f5b50599640 10 monclient: shutdown 2024-09-16T03:48:40.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.867+0000 7f5b50599640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:48:40.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.867+0000 7f5b50599640 1 -- 172.21.15.42:0/2765343386 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5b48138a40 msgr2=0x7f5b48135a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:48:40.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.867+0000 7f5b4eb36640 1 -- 172.21.15.42:0/2765343386 reap_dead start 2024-09-16T03:48:40.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.867+0000 7f5b50599640 1 --2- 172.21.15.42:0/2765343386 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5b48138a40 0x7f5b48135a30 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f5b4000c750 tx=0x7f5b4000cc10 comp rx=0 tx=0).stop 2024-09-16T03:48:40.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.867+0000 7f5b50599640 1 -- 172.21.15.42:0/2765343386 shutdown_connections 2024-09-16T03:48:40.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.867+0000 7f5b50599640 1 --2- 172.21.15.42:0/2765343386 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5b48138a40 0x7f5b48135a30 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:40.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.867+0000 7f5b50599640 1 -- 172.21.15.42:0/2765343386 >> 172.21.15.42:0/2765343386 conn(0x7f5b480a6210 msgr2=0x7f5b4809c270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:48:40.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.867+0000 7f5b50599640 1 -- 172.21.15.42:0/2765343386 shutdown_connections 2024-09-16T03:48:40.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.867+0000 7f5b50599640 1 -- 172.21.15.42:0/2765343386 wait complete. 2024-09-16T03:48:40.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:40.867+0000 7f5b50599640 1 librados: shutdown 2024-09-16T03:48:40.909 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:48:40.909 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T03:48:40.909 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T03:48:41.025 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.023+0000 7ffa753d0640 10 monclient: get_monmap_and_config 2024-09-16T03:48:41.025 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.023+0000 7ffa753d0640 10 monclient: build_initial_monmap 2024-09-16T03:48:41.025 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.023+0000 7ffa753d0640 10 monclient: monmap: 2024-09-16T03:48:41.026 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:48:41.026 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:48:41.026 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:48:41.024480+0000 2024-09-16T03:48:41.026 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:48:41.024480+0000 2024-09-16T03:48:41.026 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:48:41.026 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:48:41.026 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:48:41.026 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:48:41.026 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:48:41.026 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:48:41.026 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:48:41.026 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:48:41.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.023+0000 7ffa753d0640 1 Processor -- start 2024-09-16T03:48:41.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.023+0000 7ffa753d0640 1 -- start start 2024-09-16T03:48:41.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.023+0000 7ffa753d0640 10 monclient: init 2024-09-16T03:48:41.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.023+0000 7ffa753d0640 10 monclient: _reopen_session rank -1 2024-09-16T03:48:41.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.023+0000 7ffa753d0640 10 monclient: _add_conns ranks=[0,4,1,2,3] 2024-09-16T03:48:41.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.023+0000 7ffa753d0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ffa700a1330 addr v1:172.21.15.42:6789/0 2024-09-16T03:48:41.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.023+0000 7ffa753d0640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7ffa7006dac0 0x7ffa7013a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:41.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.023+0000 7ffa753d0640 10 monclient(hunting): picked mon.noname-c con 0x7ffa7006dac0 addr v2:172.21.15.154:3300/0 2024-09-16T03:48:41.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.023+0000 7ffa753d0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ffa700a1e20 addr v1:172.21.15.154:6789/0 2024-09-16T03:48:41.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.023+0000 7ffa753d0640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffa7009c090 con 0x7ffa700a1330 2024-09-16T03:48:41.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.023+0000 7ffa753d0640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffa7009f610 con 0x7ffa700a1e20 2024-09-16T03:48:41.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.023+0000 7ffa753d0640 10 monclient(hunting): start opening mon connection 2024-09-16T03:48:41.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.023+0000 7ffa753d0640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7ffa7009f8f0 con 0x7ffa7006dac0 2024-09-16T03:48:41.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.023+0000 7ffa753d0640 10 monclient(hunting): _renew_subs 2024-09-16T03:48:41.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.023+0000 7ffa753d0640 10 monclient(hunting): authenticate will time out at 1367.993935s 2024-09-16T03:48:41.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.023+0000 7ffa6effd640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7ffa700a1330 0x7ffa700a1710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:37222/0 (socket says 172.21.15.42:37222) 2024-09-16T03:48:41.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.023+0000 7ffa6effd640 1 -- 172.21.15.42:0/3003204849 learned_addr learned my addr 172.21.15.42:0/3003204849 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:48:41.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.023+0000 7ffa6e7fc640 1 --2- 172.21.15.42:0/3003204849 >> v2:172.21.15.154:3300/0 conn(0x7ffa7006dac0 0x7ffa7013a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:48:41.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.023+0000 7ffa6e7fc640 10 monclient(hunting): get_auth_request con 0x7ffa7006dac0 auth_method 0 2024-09-16T03:48:41.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.023+0000 7ffa6e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:48:41.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.023+0000 7ffa6e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:48:41.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.023+0000 7ffa6e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:48:41.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.023+0000 7ffa6dffb640 1 -- 172.21.15.42:0/3003204849 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4206837673 0 0) 0x7ffa7009c090 con 0x7ffa700a1330 2024-09-16T03:48:41.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.023+0000 7ffa6dffb640 10 monclient(hunting): _negotiate 2024-09-16T03:48:41.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.023+0000 7ffa6dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:48:41.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.023+0000 7ffa6dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:48:41.045 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.023+0000 7ffa6dffb640 10 monclient(hunting): my global_id is 764529 2024-09-16T03:48:41.045 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.023+0000 7ffa6dffb640 1 -- 172.21.15.42:0/3003204849 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffa50003530 con 0x7ffa700a1330 2024-09-16T03:48:41.045 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.023+0000 7ffa6e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:48:41.045 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.023+0000 7ffa6e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:48:41.045 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.023+0000 7ffa6e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:48:41.045 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.023+0000 7ffa6dffb640 1 -- 172.21.15.42:0/3003204849 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 43556932 0 0) 0x7ffa7009f610 con 0x7ffa700a1e20 2024-09-16T03:48:41.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.023+0000 7ffa6dffb640 10 monclient(hunting): _negotiate 2024-09-16T03:48:41.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.023+0000 7ffa6dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:48:41.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.023+0000 7ffa6dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:48:41.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.023+0000 7ffa6dffb640 10 monclient(hunting): my global_id is 774313 2024-09-16T03:48:41.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.023+0000 7ffa6dffb640 1 -- 172.21.15.42:0/3003204849 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffa7009c090 con 0x7ffa700a1e20 2024-09-16T03:48:41.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.023+0000 7ffa6dffb640 1 -- 172.21.15.42:0/3003204849 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4181045976 0 0) 0x7ffa50003530 con 0x7ffa700a1330 2024-09-16T03:48:41.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.023+0000 7ffa6dffb640 1 -- 172.21.15.42:0/3003204849 >> v2:172.21.15.154:3300/0 conn(0x7ffa7006dac0 msgr2=0x7ffa7013a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:48:41.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.023+0000 7ffa6dffb640 1 --2- 172.21.15.42:0/3003204849 >> v2:172.21.15.154:3300/0 conn(0x7ffa7006dac0 0x7ffa7013a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:41.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.023+0000 7ffa6dffb640 1 -- 172.21.15.42:0/3003204849 >> v1:172.21.15.154:6789/0 conn(0x7ffa700a1e20 legacy=0x7ffa7013c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:48:41.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.023+0000 7ffa6dffb640 10 monclient: _finish_hunting 0 2024-09-16T03:48:41.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.023+0000 7ffa6dffb640 1 monclient: found mon.noname-a-legacy 2024-09-16T03:48:41.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.023+0000 7ffa6dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:48:41.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.023+0000 7ffa6dffb640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.42:6789/0 2024-09-16T03:48:41.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.023+0000 7ffa6dffb640 1 -- 172.21.15.42:0/3003204849 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa7009fbd0 con 0x7ffa700a1330 2024-09-16T03:48:41.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.023+0000 7ffa6dffb640 10 monclient: _finish_auth 0 2024-09-16T03:48:41.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.023+0000 7ffa6dffb640 10 monclient: _check_auth_tickets 2024-09-16T03:48:41.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.023+0000 7ffa6dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:48:41.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.023+0000 7ffa6dffb640 1 -- 172.21.15.42:0/3003204849 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2177413734 0 0) 0x7ffa60001a00 con 0x7ffa700a1330 2024-09-16T03:48:41.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.023+0000 7ffa6dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:48:41.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.023+0000 7ffa6e7fc640 1 --2- 172.21.15.42:0/3003204849 >> v2:172.21.15.154:3300/0 conn(0x7ffa7006dac0 0x7ffa7013a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:48:41.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.023+0000 7ffa6dffb640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T03:48:41.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.023+0000 7ffa6dffb640 10 monclient: dump: 2024-09-16T03:48:41.047 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:48:41.047 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:48:41.047 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:41.047 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:41.047 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:48:41.048 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:48:41.048 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:48:41.048 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:48:41.048 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:48:41.048 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:48:41.048 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.023+0000 7ffa6dffb640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] but i'm connected to v1:172.21.15.42:6789/0, reconnecting 2024-09-16T03:48:41.048 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.023+0000 7ffa6dffb640 10 monclient: _reopen_session rank -1 2024-09-16T03:48:41.048 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.023+0000 7ffa6dffb640 1 -- 172.21.15.42:0/3003204849 >> v1:172.21.15.42:6789/0 conn(0x7ffa700a1330 legacy=0x7ffa700a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:48:41.048 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.023+0000 7ffa6dffb640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T03:48:41.048 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.023+0000 7ffa6dffb640 1 --2- 172.21.15.42:0/3003204849 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffa50003ba0 0x7ffa50003fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:41.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.023+0000 7ffa6dffb640 10 monclient(hunting): picked mon.b con 0x7ffa50003ba0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:48:41.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.023+0000 7ffa6dffb640 1 --2- 172.21.15.42:0/3003204849 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ffa50004810 0x7ffa50004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:41.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.023+0000 7ffa6dffb640 10 monclient(hunting): picked mon.c con 0x7ffa50004810 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:48:41.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.023+0000 7ffa6dffb640 1 --2- 172.21.15.42:0/3003204849 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ffa50005220 0x7ffa5000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:41.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.023+0000 7ffa6dffb640 10 monclient(hunting): picked mon.a con 0x7ffa50005220 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:48:41.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.023+0000 7ffa6dffb640 10 monclient(hunting): start opening mon connection 2024-09-16T03:48:41.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.023+0000 7ffa6dffb640 1 -- 172.21.15.42:0/3003204849 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7ffa7009f8f0 con 0x7ffa50005220 2024-09-16T03:48:41.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.023+0000 7ffa6dffb640 10 monclient(hunting): start opening mon connection 2024-09-16T03:48:41.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.023+0000 7ffa6dffb640 1 -- 172.21.15.42:0/3003204849 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ffa5000c150 con 0x7ffa50003ba0 2024-09-16T03:48:41.050 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.023+0000 7ffa6dffb640 10 monclient(hunting): start opening mon connection 2024-09-16T03:48:41.050 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.023+0000 7ffa6dffb640 1 -- 172.21.15.42:0/3003204849 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7ffa5000c350 con 0x7ffa50004810 2024-09-16T03:48:41.050 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.023+0000 7ffa6dffb640 10 monclient(hunting): _renew_subs 2024-09-16T03:48:41.050 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.023+0000 7ffa6dffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:48:41.050 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.023+0000 7ffa6effd640 1 --2- 172.21.15.42:0/3003204849 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffa50003ba0 0x7ffa50003fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:48:41.050 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.023+0000 7ffa6f7fe640 1 --2- 172.21.15.42:0/3003204849 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ffa50005220 0x7ffa5000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:48:41.050 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.023+0000 7ffa6effd640 10 monclient(hunting): get_auth_request con 0x7ffa50003ba0 auth_method 0 2024-09-16T03:48:41.050 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.023+0000 7ffa6effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:48:41.050 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.023+0000 7ffa6effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:48:41.050 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.023+0000 7ffa6effd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:48:41.050 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.023+0000 7ffa6f7fe640 10 monclient(hunting): get_auth_request con 0x7ffa50005220 auth_method 0 2024-09-16T03:48:41.050 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.023+0000 7ffa6f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:48:41.050 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.023+0000 7ffa6f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:48:41.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.023+0000 7ffa6f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:48:41.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.023+0000 7ffa6f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:48:41.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.023+0000 7ffa6f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:48:41.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.023+0000 7ffa6f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T03:48:41.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.023+0000 7ffa6e7fc640 1 --2- 172.21.15.42:0/3003204849 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ffa50004810 0x7ffa50004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:48:41.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.023+0000 7ffa6effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:48:41.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.023+0000 7ffa6effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:48:41.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.023+0000 7ffa6effd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T03:48:41.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.023+0000 7ffa6e7fc640 10 monclient(hunting): get_auth_request con 0x7ffa50004810 auth_method 0 2024-09-16T03:48:41.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.023+0000 7ffa6e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:48:41.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.023+0000 7ffa6e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:48:41.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.023+0000 7ffa6e7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:48:41.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.023+0000 7ffa6e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:48:41.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.023+0000 7ffa6e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:48:41.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.023+0000 7ffa6e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T03:48:41.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.023+0000 7ffa6f7fe640 10 monclient(hunting): handle_auth_done global_id 764529 payload 293 2024-09-16T03:48:41.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.023+0000 7ffa6f7fe640 1 -- 172.21.15.42:0/3003204849 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ffa50004810 msgr2=0x7ffa50004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:48:41.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.023+0000 7ffa6f7fe640 1 --2- 172.21.15.42:0/3003204849 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ffa50004810 0x7ffa50004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:41.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.023+0000 7ffa6f7fe640 1 -- 172.21.15.42:0/3003204849 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffa50003ba0 msgr2=0x7ffa50003fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:48:41.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.023+0000 7ffa6f7fe640 1 --2- 172.21.15.42:0/3003204849 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffa50003ba0 0x7ffa50003fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:41.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.023+0000 7ffa6f7fe640 10 monclient: _finish_hunting 0 2024-09-16T03:48:41.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.023+0000 7ffa6f7fe640 1 monclient: found mon.a 2024-09-16T03:48:41.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.023+0000 7ffa6f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:48:41.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.023+0000 7ffa6f7fe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:48:41.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.023+0000 7ffa6f7fe640 1 -- 172.21.15.42:0/3003204849 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa50003530 con 0x7ffa50005220 2024-09-16T03:48:41.053 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.023+0000 7ffa6f7fe640 10 monclient: _finish_auth 0 2024-09-16T03:48:41.053 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.023+0000 7ffa6f7fe640 10 monclient: _check_auth_tickets 2024-09-16T03:48:41.053 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.023+0000 7ffa6f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:48:41.053 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.023+0000 7ffa753d0640 5 monclient: authenticate success, global_id 764529 2024-09-16T03:48:41.053 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.023+0000 7ffa753d0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:48:41.053 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.027+0000 7ffa6f7fe640 1 -- 172.21.15.42:0/3003204849 reap_dead start 2024-09-16T03:48:41.053 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.027+0000 7ffa6f7fe640 1 --2- 172.21.15.42:0/3003204849 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ffa50005220 0x7ffa5000b9d0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7ffa680018d0 tx=0x7ffa6801ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f37d6adb0ff2c033 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:48:41.053 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.027+0000 7ffa6dffb640 1 -- 172.21.15.42:0/3003204849 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa68001520 con 0x7ffa50005220 2024-09-16T03:48:41.053 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.027+0000 7ffa6dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:48:41.053 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.027+0000 7ffa6dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:48:41.053 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.027+0000 7ffa6dffb640 10 monclient: dump: 2024-09-16T03:48:41.053 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:48:41.054 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:48:41.054 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:41.054 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:41.054 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:48:41.054 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:48:41.054 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:48:41.054 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:48:41.054 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:48:41.054 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:48:41.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.027+0000 7ffa753d0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:48:41.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.027+0000 7ffa6dffb640 1 -- 172.21.15.42:0/3003204849 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffa680052d0 con 0x7ffa50005220 2024-09-16T03:48:41.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.027+0000 7ffa6dffb640 10 monclient: handle_config config(0 keys) 2024-09-16T03:48:41.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.027+0000 7ffa753d0640 10 monclient: get_monmap_and_config success 2024-09-16T03:48:41.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.027+0000 7ffa6dffb640 1 -- 172.21.15.42:0/3003204849 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa680a8930 con 0x7ffa50005220 2024-09-16T03:48:41.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.027+0000 7ffa6dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:48:41.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.027+0000 7ffa753d0640 4 set_mon_vals no callback set 2024-09-16T03:48:41.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.027+0000 7ffa6dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:48:41.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.027+0000 7ffa6dffb640 10 monclient: dump: 2024-09-16T03:48:41.055 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:48:41.055 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:48:41.055 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:41.055 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:41.055 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:48:41.056 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:48:41.056 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:48:41.056 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:48:41.056 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:48:41.056 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:48:41.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.027+0000 7ffa753d0640 10 monclient: shutdown 2024-09-16T03:48:41.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.027+0000 7ffa753d0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:48:41.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.027+0000 7ffa753d0640 1 -- 172.21.15.42:0/3003204849 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ffa50005220 msgr2=0x7ffa5000b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:48:41.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.027+0000 7ffa753d0640 1 --2- 172.21.15.42:0/3003204849 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ffa50005220 0x7ffa5000b9d0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7ffa680018d0 tx=0x7ffa6801ecd0 comp rx=0 tx=0).stop 2024-09-16T03:48:41.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.027+0000 7ffa753d0640 1 -- 172.21.15.42:0/3003204849 shutdown_connections 2024-09-16T03:48:41.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.027+0000 7ffa753d0640 1 --2- 172.21.15.42:0/3003204849 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ffa50005220 0x7ffa5000b9d0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:41.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.027+0000 7ffa753d0640 1 -- 172.21.15.42:0/3003204849 >> 172.21.15.42:0/3003204849 conn(0x7ffa700a2420 msgr2=0x7ffa700a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:48:41.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.027+0000 7ffa753d0640 1 -- 172.21.15.42:0/3003204849 shutdown_connections 2024-09-16T03:48:41.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.027+0000 7ffa753d0640 1 -- 172.21.15.42:0/3003204849 wait complete. 2024-09-16T03:48:41.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.027+0000 7ffa753d0640 10 monclient: build_initial_monmap 2024-09-16T03:48:41.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.031+0000 7ffa753d0640 10 monclient: monmap: 2024-09-16T03:48:41.057 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:48:41.057 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:48:41.057 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:48:41.057 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:48:41.057 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:48:41.057 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:48:41.057 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:48:41.057 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:48:41.057 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:48:41.057 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:48:41.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.031+0000 7ffa753d0640 1 librados: starting msgr at 2024-09-16T03:48:41.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.031+0000 7ffa753d0640 1 librados: starting objecter 2024-09-16T03:48:41.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.031+0000 7ffa753d0640 1 Processor -- start 2024-09-16T03:48:41.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.031+0000 7ffa753d0640 1 -- start start 2024-09-16T03:48:41.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.031+0000 7ffa753d0640 1 librados: setting wanted keys 2024-09-16T03:48:41.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.031+0000 7ffa753d0640 1 librados: calling monclient init 2024-09-16T03:48:41.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.031+0000 7ffa753d0640 10 monclient: init 2024-09-16T03:48:41.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.031+0000 7ffa753d0640 10 monclient: _reopen_session rank -1 2024-09-16T03:48:41.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.031+0000 7ffa753d0640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:48:41.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.031+0000 7ffa753d0640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ffa50005220 0x7ffa701ca380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:41.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.031+0000 7ffa753d0640 10 monclient(hunting): picked mon.noname-c con 0x7ffa50005220 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:48:41.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.031+0000 7ffa753d0640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffa701ca8c0 0x7ffa70121ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:41.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.031+0000 7ffa753d0640 10 monclient(hunting): picked mon.noname-b con 0x7ffa701ca8c0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:48:41.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.031+0000 7ffa753d0640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ffa701caca0 0x7ffa70122000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:41.059 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.031+0000 7ffa753d0640 10 monclient(hunting): picked mon.noname-a con 0x7ffa701caca0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:48:41.059 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.031+0000 7ffa753d0640 10 monclient(hunting): start opening mon connection 2024-09-16T03:48:41.059 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.031+0000 7ffa753d0640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7ffa7009eef0 con 0x7ffa701caca0 2024-09-16T03:48:41.059 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.031+0000 7ffa753d0640 10 monclient(hunting): start opening mon connection 2024-09-16T03:48:41.059 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.031+0000 7ffa753d0640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ffa7009ed70 con 0x7ffa701ca8c0 2024-09-16T03:48:41.059 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.031+0000 7ffa753d0640 10 monclient(hunting): start opening mon connection 2024-09-16T03:48:41.059 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.031+0000 7ffa753d0640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7ffa700a1710 con 0x7ffa50005220 2024-09-16T03:48:41.059 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.031+0000 7ffa753d0640 10 monclient(hunting): _renew_subs 2024-09-16T03:48:41.059 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.031+0000 7ffa753d0640 10 monclient(hunting): authenticate will time out at 1368.001071s 2024-09-16T03:48:41.059 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.031+0000 7ffa6f7fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ffa701caca0 0x7ffa70122000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:48:41.059 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.031+0000 7ffa6f7fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ffa701caca0 0x7ffa70122000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:44850/0 (socket says 172.21.15.42:44850) 2024-09-16T03:48:41.060 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.031+0000 7ffa6f7fe640 1 -- 172.21.15.42:0/1544097314 learned_addr learned my addr 172.21.15.42:0/1544097314 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:48:41.060 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.031+0000 7ffa6f7fe640 10 monclient(hunting): get_auth_request con 0x7ffa701caca0 auth_method 0 2024-09-16T03:48:41.060 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.031+0000 7ffa6f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:48:41.060 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.031+0000 7ffa6f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:48:41.060 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.031+0000 7ffa6f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:48:41.060 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.031+0000 7ffa6effd640 1 --2- 172.21.15.42:0/1544097314 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ffa50005220 0x7ffa701ca380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:48:41.060 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.031+0000 7ffa6effd640 10 monclient(hunting): get_auth_request con 0x7ffa50005220 auth_method 0 2024-09-16T03:48:41.060 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.031+0000 7ffa6effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:48:41.060 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.031+0000 7ffa6effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:48:41.060 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.031+0000 7ffa6effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:48:41.060 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.031+0000 7ffa6f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:48:41.060 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.031+0000 7ffa6f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:48:41.061 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.031+0000 7ffa6f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:48:41.061 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.031+0000 7ffa6effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:48:41.061 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.031+0000 7ffa6effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:48:41.061 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.031+0000 7ffa6effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:48:41.061 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.031+0000 7ffa6e7fc640 1 --2- 172.21.15.42:0/1544097314 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffa701ca8c0 0x7ffa70121ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:48:41.061 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.031+0000 7ffa6f7fe640 10 monclient(hunting): handle_auth_done global_id 764532 payload 824 2024-09-16T03:48:41.061 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.031+0000 7ffa6f7fe640 1 -- 172.21.15.42:0/1544097314 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ffa50005220 msgr2=0x7ffa701ca380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:48:41.061 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.031+0000 7ffa6f7fe640 1 --2- 172.21.15.42:0/1544097314 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ffa50005220 0x7ffa701ca380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:41.061 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.031+0000 7ffa6f7fe640 1 -- 172.21.15.42:0/1544097314 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffa701ca8c0 msgr2=0x7ffa70121ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:48:41.061 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.031+0000 7ffa6f7fe640 1 --2- 172.21.15.42:0/1544097314 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffa701ca8c0 0x7ffa70121ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:41.061 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.031+0000 7ffa6f7fe640 10 monclient: _finish_hunting 0 2024-09-16T03:48:41.061 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.031+0000 7ffa6f7fe640 1 monclient: found mon.noname-a 2024-09-16T03:48:41.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.031+0000 7ffa6f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:48:41.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.031+0000 7ffa6f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:48:41.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.031+0000 7ffa6f7fe640 1 -- 172.21.15.42:0/1544097314 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa70125400 con 0x7ffa701caca0 2024-09-16T03:48:41.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.031+0000 7ffa6e7fc640 10 monclient: get_auth_request con 0x7ffa701ca8c0 auth_method 0 2024-09-16T03:48:41.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.031+0000 7ffa6e7fc640 1 --2- 172.21.15.42:0/1544097314 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffa701ca8c0 0x7ffa70121ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T03:48:41.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.031+0000 7ffa6f7fe640 1 --2- 172.21.15.42:0/1544097314 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ffa701caca0 0x7ffa70122000 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7ffa6801eb00 tx=0x7ffa50005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:48:41.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.031+0000 7ffa677fe640 10 client.?.objecter ms_handle_connect 0x7ffa701caca0 2024-09-16T03:48:41.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.031+0000 7ffa677fe640 10 client.?.objecter resend_mon_ops 2024-09-16T03:48:41.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.031+0000 7ffa677fe640 1 -- 172.21.15.42:0/1544097314 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa68001500 con 0x7ffa701caca0 2024-09-16T03:48:41.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.031+0000 7ffa677fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:48:41.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.031+0000 7ffa677fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:48:41.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.031+0000 7ffa677fe640 10 monclient: dump: 2024-09-16T03:48:41.062 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:48:41.063 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:48:41.063 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:41.063 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:41.063 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:48:41.063 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:48:41.063 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:48:41.063 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:48:41.063 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:48:41.063 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:48:41.063 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.031+0000 7ffa677fe640 10 monclient: _finish_auth 0 2024-09-16T03:48:41.064 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.031+0000 7ffa677fe640 10 monclient: _check_auth_tickets 2024-09-16T03:48:41.064 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.031+0000 7ffa677fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:48:41.064 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.031+0000 7ffa677fe640 1 -- 172.21.15.42:0/1544097314 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffa68005480 con 0x7ffa701caca0 2024-09-16T03:48:41.064 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.031+0000 7ffa753d0640 5 monclient: authenticate success, global_id 764532 2024-09-16T03:48:41.064 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.031+0000 7ffa753d0640 10 monclient: _renew_subs 2024-09-16T03:48:41.064 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.031+0000 7ffa753d0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:48:41.064 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.031+0000 7ffa753d0640 1 -- 172.21.15.42:0/1544097314 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa70123b30 con 0x7ffa701caca0 2024-09-16T03:48:41.064 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.031+0000 7ffa677fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:48:41.064 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.031+0000 7ffa677fe640 1 -- 172.21.15.42:0/1544097314 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa68003600 con 0x7ffa701caca0 2024-09-16T03:48:41.064 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.031+0000 7ffa677fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:48:41.064 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.031+0000 7ffa677fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:48:41.065 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.031+0000 7ffa677fe640 10 monclient: dump: 2024-09-16T03:48:41.065 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:48:41.065 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:48:41.065 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:41.065 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:41.065 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:48:41.065 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:48:41.065 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:48:41.065 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:48:41.065 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:48:41.065 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:48:41.065 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.031+0000 7ffa753d0640 10 client.764532.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:48:41.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.031+0000 7ffa753d0640 10 monclient: _renew_subs 2024-09-16T03:48:41.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.031+0000 7ffa753d0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:48:41.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.031+0000 7ffa753d0640 1 -- 172.21.15.42:0/1544097314 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa701259c0 con 0x7ffa701caca0 2024-09-16T03:48:41.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.031+0000 7ffa753d0640 1 librados: init done 2024-09-16T03:48:41.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.031+0000 7ffa6d7fa640 4 set_mon_vals no callback set 2024-09-16T03:48:41.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.031+0000 7ffa753d0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:48:41.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.031+0000 7ffa753d0640 1 --2- 172.21.15.42:0/1544097314 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffa70122540 0x7ffa70122920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:41.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.031+0000 7ffa753d0640 10 monclient(hunting): start opening mon connection 2024-09-16T03:48:41.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.031+0000 7ffa753d0640 1 -- 172.21.15.42:0/1544097314 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ffa700a1aa0 con 0x7ffa70122540 2024-09-16T03:48:41.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.031+0000 7ffa677fe640 1 -- 172.21.15.42:0/1544097314 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7ffa680037a0 con 0x7ffa701caca0 2024-09-16T03:48:41.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.031+0000 7ffa6effd640 1 --2- 172.21.15.42:0/1544097314 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffa70122540 0x7ffa70122920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:48:41.067 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.031+0000 7ffa677fe640 1 --2- 172.21.15.42:0/1544097314 >> v2:172.21.15.154:6812/254845342 conn(0x7ffa54037840 0x7ffa54039d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:41.067 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.031+0000 7ffa6effd640 10 monclient: get_auth_request con 0x7ffa70122540 auth_method 0 2024-09-16T03:48:41.067 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.031+0000 7ffa677fe640 1 -- 172.21.15.42:0/1544097314 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(216..216 src has 1..216) ==== 6980+0+0 (secure 0 0 0) 0x7ffa680e5d80 con 0x7ffa701caca0 2024-09-16T03:48:41.067 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.031+0000 7ffa6effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:48:41.067 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.031+0000 7ffa6effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:48:41.067 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.031+0000 7ffa6effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:48:41.067 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.031+0000 7ffa677fe640 10 client.764532.objecter ms_dispatch 0x7ffa70000ba0 osd_map(216..216 src has 1..216) 2024-09-16T03:48:41.067 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.031+0000 7ffa677fe640 3 client.764532.objecter handle_osd_map got epochs [216,216] > 0 2024-09-16T03:48:41.067 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.031+0000 7ffa677fe640 3 client.764532.objecter handle_osd_map decoding full epoch 216 2024-09-16T03:48:41.067 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.031+0000 7ffa677fe640 20 client.764532.objecter dump_active .. 0 homeless 2024-09-16T03:48:41.067 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.031+0000 7ffa6effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:48:41.067 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.031+0000 7ffa6effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:48:41.068 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.031+0000 7ffa6effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:48:41.068 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.031+0000 7ffa6e7fc640 1 --2- 172.21.15.42:0/1544097314 >> v2:172.21.15.154:6812/254845342 conn(0x7ffa54037840 0x7ffa54039d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:48:41.068 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.031+0000 7ffa6e7fc640 10 monclient: get_auth_request con 0x7ffa54037840 auth_method 0 2024-09-16T03:48:41.068 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.031+0000 7ffa6effd640 10 monclient(hunting): handle_auth_done global_id 774316 payload 824 2024-09-16T03:48:41.068 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.035+0000 7ffa6effd640 1 -- 172.21.15.42:0/1544097314 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ffa70122e60 con 0x7ffa70122540 2024-09-16T03:48:41.068 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.035+0000 7ffa677fe640 10 client.764532.objecter ms_handle_connect 0x7ffa70122540 2024-09-16T03:48:41.068 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.035+0000 7ffa677fe640 10 client.764532.objecter resend_mon_ops 2024-09-16T03:48:41.068 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.035+0000 7ffa6effd640 1 --2- 172.21.15.42:0/1544097314 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffa70122540 0x7ffa70122920 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7ffa600077a0 tx=0x7ffa600122c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:48:41.068 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.035+0000 7ffa6e7fc640 1 --2- 172.21.15.42:0/1544097314 >> v2:172.21.15.154:6812/254845342 conn(0x7ffa54037840 0x7ffa54039d00 secure :-1 s=READY pgs=3496 cs=0 l=1 rev1=1 crypto rx=0x7ffa58006b20 tx=0x7ffa58008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:48:41.068 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.035+0000 7ffa677fe640 10 client.764532.objecter ms_handle_connect 0x7ffa54037840 2024-09-16T03:48:41.069 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.035+0000 7ffa677fe640 1 -- 172.21.15.42:0/1544097314 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa60019070 con 0x7ffa70122540 2024-09-16T03:48:41.069 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.035+0000 7ffa677fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:48:41.069 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.035+0000 7ffa677fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:48:41.069 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.035+0000 7ffa677fe640 10 monclient: dump: 2024-09-16T03:48:41.069 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:48:41.069 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:48:41.069 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:41.069 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:41.069 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:48:41.070 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:48:41.070 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:48:41.070 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:48:41.070 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:48:41.070 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:48:41.070 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.035+0000 7ffa677fe640 1 -- 172.21.15.42:0/1544097314 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7ffa60012d10 con 0x7ffa70122540 2024-09-16T03:48:41.070 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.035+0000 7ffa677fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:48:41.070 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.035+0000 7ffa677fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:48:41.070 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.035+0000 7ffa677fe640 1 -- 172.21.15.42:0/1544097314 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffa70122540 msgr2=0x7ffa70122920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:48:41.070 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.035+0000 7ffa677fe640 1 --2- 172.21.15.42:0/1544097314 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffa70122540 0x7ffa70122920 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7ffa600077a0 tx=0x7ffa600122c0 comp rx=0 tx=0).stop 2024-09-16T03:48:41.070 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.035+0000 7ffa677fe640 1 -- 172.21.15.42:0/1544097314 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffa70122540 msgr2=0x7ffa70122920 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:48:41.071 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.035+0000 7ffa677fe640 1 --2- 172.21.15.42:0/1544097314 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffa70122540 0x7ffa70122920 secure :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7ffa600077a0 tx=0x7ffa600122c0 comp rx=0 tx=0).stop 2024-09-16T03:48:41.072 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.039+0000 7ffa753d0640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:48:41.072 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.039+0000 7ffa753d0640 1 --2- 172.21.15.42:0/1544097314 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffa7011cf80 0x7ffa7011d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:41.072 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.039+0000 7ffa753d0640 10 monclient(hunting): start opening mon connection 2024-09-16T03:48:41.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.039+0000 7ffa753d0640 1 -- 172.21.15.42:0/1544097314 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ffa7006ed10 con 0x7ffa7011cf80 2024-09-16T03:48:41.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.043+0000 7ffa6effd640 1 --2- 172.21.15.42:0/1544097314 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffa7011cf80 0x7ffa7011d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:48:41.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.043+0000 7ffa6effd640 10 monclient: get_auth_request con 0x7ffa7011cf80 auth_method 0 2024-09-16T03:48:41.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.043+0000 7ffa6effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:48:41.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.043+0000 7ffa6effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:48:41.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.043+0000 7ffa6effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:48:41.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.043+0000 7ffa6effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:48:41.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.043+0000 7ffa6effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:48:41.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.043+0000 7ffa6effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:48:41.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.043+0000 7ffa6effd640 10 monclient(hunting): handle_auth_done global_id 774319 payload 824 2024-09-16T03:48:41.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.043+0000 7ffa6effd640 1 -- 172.21.15.42:0/1544097314 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ffa70069040 con 0x7ffa7011cf80 2024-09-16T03:48:41.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.043+0000 7ffa6effd640 1 --2- 172.21.15.42:0/1544097314 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffa7011cf80 0x7ffa7011d390 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7ffa600077a0 tx=0x7ffa60002120 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:48:41.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.043+0000 7ffa677fe640 10 client.764532.objecter ms_handle_connect 0x7ffa7011cf80 2024-09-16T03:48:41.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.043+0000 7ffa677fe640 10 client.764532.objecter resend_mon_ops 2024-09-16T03:48:41.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.043+0000 7ffa677fe640 1 -- 172.21.15.42:0/1544097314 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa60019050 con 0x7ffa7011cf80 2024-09-16T03:48:41.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.043+0000 7ffa677fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:48:41.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.043+0000 7ffa677fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:48:41.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.043+0000 7ffa677fe640 10 monclient: dump: 2024-09-16T03:48:41.074 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:48:41.074 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:48:41.074 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:41.074 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:41.075 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:48:41.075 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:48:41.075 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:48:41.075 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:48:41.075 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:48:41.075 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:48:41.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.043+0000 7ffa677fe640 1 -- 172.21.15.42:0/1544097314 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4544 (secure 0 0 0) 0x7ffa600159e0 con 0x7ffa7011cf80 2024-09-16T03:48:41.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.043+0000 7ffa677fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:48:41.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.043+0000 7ffa677fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:48:41.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.043+0000 7ffa677fe640 1 -- 172.21.15.42:0/1544097314 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffa7011cf80 msgr2=0x7ffa7011d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:48:41.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.043+0000 7ffa677fe640 1 --2- 172.21.15.42:0/1544097314 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffa7011cf80 0x7ffa7011d390 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7ffa600077a0 tx=0x7ffa60002120 comp rx=0 tx=0).stop 2024-09-16T03:48:41.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.043+0000 7ffa677fe640 1 -- 172.21.15.42:0/1544097314 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffa7011cf80 msgr2=0x7ffa7011d390 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:48:41.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.043+0000 7ffa677fe640 1 --2- 172.21.15.42:0/1544097314 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffa7011cf80 0x7ffa7011d390 secure :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7ffa600077a0 tx=0x7ffa60002120 comp rx=0 tx=0).stop 2024-09-16T03:48:41.076 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:48:41.076 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:48:41.076 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:48:41.076 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T03:48:41.076 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 60, 2024-09-16T03:48:41.076 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 58627, 2024-09-16T03:48:41.077 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:48:41.077 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:48:41.077 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T03:48:41.077 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T03:48:41.077 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:48:41.077 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 7, 2024-09-16T03:48:41.077 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:48:41.077 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:48:41.077 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:48:41.077 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:48:41.077 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:48:41.078 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:48:41.078 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:48:41.078 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:48:41.078 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:48:41.078 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:48:41.078 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:48:41.078 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:48:41.078 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:48:41.078 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:48:41.078 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:48:41.078 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:48:41.079 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:48:41.079 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:48:41.079 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:48:41.079 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:48:41.079 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:48:41.079 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:48:41.079 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:48:41.079 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:48:41.079 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:48:41.079 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:48:41.079 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:48:41.080 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:48:41.080 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:48:41.080 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:48:41.080 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:48:41.080 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:48:41.080 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:48:41.080 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:48:41.080 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:48:41.080 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:48:41.080 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:48:41.080 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:48:41.080 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:48:41.081 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:48:41.081 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:48:41.081 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:48:41.081 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:48:41.081 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:48:41.081 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:48:41.081 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:48:41.081 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:48:41.081 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:48:41.081 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:48:41.081 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:48:41.082 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:48:41.082 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:48:41.082 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:48:41.082 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:48:41.082 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:48:41.082 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:48:41.082 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:48:41.082 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:48:41.082 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:48:41.082 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:48:41.082 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:48:41.083 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:48:41.083 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:48:41.083 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:48:41.083 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:48:41.083 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:48:41.083 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:48:41.083 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:48:41.083 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:48:41.083 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:48:41.083 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:48:41.083 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:48:41.084 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:48:41.084 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:48:41.084 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:48:41.084 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:48:41.084 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:48:41.084 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:48:41.084 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:48:41.084 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:48:41.084 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:48:41.084 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:48:41.084 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:48:41.085 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:48:41.085 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:48:41.085 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:48:41.085 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:48:41.085 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:48:41.085 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:48:41.085 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:48:41.085 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:48:41.085 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:48:41.085 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:48:41.085 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:48:41.086 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:48:41.086 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:48:41.086 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:48:41.086 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:48:41.086 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:48:41.086 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:48:41.086 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:48:41.086 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:48:41.086 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:48:41.086 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:48:41.086 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:48:41.086 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:48:41.087 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:48:41.087 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:48:41.087 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:48:41.087 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:48:41.087 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:48:41.087 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:48:41.087 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:48:41.087 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:48:41.087 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:48:41.087 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:48:41.087 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:48:41.088 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:48:41.088 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:48:41.088 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:48:41.088 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:48:41.088 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:48:41.088 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:48:41.088 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:48:41.088 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:48:41.088 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:48:41.088 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:48:41.088 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:48:41.088 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:48:41.089 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:48:41.089 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:48:41.089 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:48:41.089 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:48:41.089 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:48:41.089 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:48:41.089 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:48:41.089 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:48:41.089 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:48:41.089 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:48:41.089 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:48:41.090 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:48:41.090 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:48:41.090 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:48:41.090 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:48:41.090 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:48:41.090 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 4 2024-09-16T03:48:41.090 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:48:41.090 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:48:41.090 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:48:41.090 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:48:41.090 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:48:41.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.043+0000 7ffa753d0640 10 librados: watch_flush enter 2024-09-16T03:48:41.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.043+0000 7ffa753d0640 10 librados: watch_flush exit 2024-09-16T03:48:41.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.043+0000 7ffa753d0640 20 client.764532.objecter shutdown clearing up homeless session... 2024-09-16T03:48:41.093 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.043+0000 7ffa753d0640 10 client.764532.objecter successfully canceled tick 2024-09-16T03:48:41.093 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.043+0000 7ffa753d0640 1 -- 172.21.15.42:0/1544097314 >> v2:172.21.15.154:6812/254845342 conn(0x7ffa54037840 msgr2=0x7ffa54039d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:48:41.093 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.043+0000 7ffa753d0640 1 --2- 172.21.15.42:0/1544097314 >> v2:172.21.15.154:6812/254845342 conn(0x7ffa54037840 0x7ffa54039d00 secure :-1 s=READY pgs=3496 cs=0 l=1 rev1=1 crypto rx=0x7ffa58006b20 tx=0x7ffa58008040 comp rx=0 tx=0).stop 2024-09-16T03:48:41.093 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.043+0000 7ffa753d0640 10 monclient: shutdown 2024-09-16T03:48:41.093 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.043+0000 7ffa753d0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:48:41.093 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.043+0000 7ffa753d0640 1 -- 172.21.15.42:0/1544097314 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ffa701caca0 msgr2=0x7ffa70122000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:48:41.093 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.043+0000 7ffa753d0640 1 --2- 172.21.15.42:0/1544097314 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ffa701caca0 0x7ffa70122000 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7ffa6801eb00 tx=0x7ffa50005840 comp rx=0 tx=0).stop 2024-09-16T03:48:41.093 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.043+0000 7ffa6f7fe640 1 -- 172.21.15.42:0/1544097314 reap_dead start 2024-09-16T03:48:41.093 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.043+0000 7ffa753d0640 1 -- 172.21.15.42:0/1544097314 shutdown_connections 2024-09-16T03:48:41.093 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.043+0000 7ffa753d0640 1 --2- 172.21.15.42:0/1544097314 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ffa701caca0 0x7ffa70122000 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:41.093 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.043+0000 7ffa753d0640 1 -- 172.21.15.42:0/1544097314 >> 172.21.15.42:0/1544097314 conn(0x7ffa700a2420 msgr2=0x7ffa700717d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:48:41.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.043+0000 7ffa753d0640 1 -- 172.21.15.42:0/1544097314 shutdown_connections 2024-09-16T03:48:41.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.043+0000 7ffa753d0640 1 -- 172.21.15.42:0/1544097314 wait complete. 2024-09-16T03:48:41.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.043+0000 7ffa753d0640 1 librados: shutdown 2024-09-16T03:48:41.094 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:48:41.094 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-09-16T03:48:41.094 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-16T03:48:41.094 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-09-16T03:48:41.095 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-16T03:48:41.095 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-16T03:48:41.095 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T03:48:41.117 INFO:tasks.ceph.mon.b:Stopped 2024-09-16T03:48:41.117 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T03:48:41.118 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:48:41.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.227+0000 7fdee884d640 10 monclient: get_monmap_and_config 2024-09-16T03:48:41.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.227+0000 7fdee884d640 10 monclient: build_initial_monmap 2024-09-16T03:48:41.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.227+0000 7fdee884d640 10 monclient: monmap: 2024-09-16T03:48:41.231 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:48:41.231 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:48:41.232 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:48:41.230386+0000 2024-09-16T03:48:41.232 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:48:41.230386+0000 2024-09-16T03:48:41.232 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:48:41.232 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:48:41.232 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:48:41.232 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:48:41.232 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:48:41.232 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:48:41.232 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:48:41.232 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:48:41.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.227+0000 7fdee884d640 1 Processor -- start 2024-09-16T03:48:41.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.227+0000 7fdee884d640 1 -- start start 2024-09-16T03:48:41.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.227+0000 7fdee884d640 10 monclient: init 2024-09-16T03:48:41.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.227+0000 7fdee884d640 10 monclient: _reopen_session rank -1 2024-09-16T03:48:41.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.227+0000 7fdee884d640 10 monclient: _add_conns ranks=[0,3,1,4,2] 2024-09-16T03:48:41.235 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.227+0000 7fdee884d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fdee40a4720 addr v1:172.21.15.42:6789/0 2024-09-16T03:48:41.235 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.227+0000 7fdee884d640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fdee4137bf0 0x7fdee413c720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:41.235 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.227+0000 7fdee884d640 10 monclient(hunting): picked mon.noname-a con 0x7fdee4137bf0 addr v2:172.21.15.42:3300/0 2024-09-16T03:48:41.235 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.227+0000 7fdee884d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fdee4138000 addr v1:172.21.15.154:6789/0 2024-09-16T03:48:41.235 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.227+0000 7fdee884d640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdee40a1470 con 0x7fdee40a4720 2024-09-16T03:48:41.235 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.231+0000 7fdee884d640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdee40a1750 con 0x7fdee4138000 2024-09-16T03:48:41.235 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.231+0000 7fdee884d640 10 monclient(hunting): start opening mon connection 2024-09-16T03:48:41.235 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.231+0000 7fdee884d640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7fdee406e8f0 con 0x7fdee4137bf0 2024-09-16T03:48:41.235 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.231+0000 7fdee884d640 10 monclient(hunting): _renew_subs 2024-09-16T03:48:41.235 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.231+0000 7fdee884d640 10 monclient(hunting): authenticate will time out at 1368.200036s 2024-09-16T03:48:41.235 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.231+0000 7fdee259c640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fdee40a4720 0x7fdee41374e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:37234/0 (socket says 172.21.15.42:37234) 2024-09-16T03:48:41.235 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.231+0000 7fdee259c640 1 -- 172.21.15.42:0/1123004954 learned_addr learned my addr 172.21.15.42:0/1123004954 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:48:41.235 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.231+0000 7fdee1d9b640 1 --2- 172.21.15.42:0/1123004954 >> v2:172.21.15.42:3300/0 conn(0x7fdee4137bf0 0x7fdee413c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:48:41.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.231+0000 7fdee2d9d640 1 -- 172.21.15.42:0/1123004954 >> v1:172.21.15.154:6789/0 conn(0x7fdee4138000 legacy=0x7fdee413ec70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.154:6789/0 2024-09-16T03:48:41.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.231+0000 7fdee1d9b640 10 monclient(hunting): get_auth_request con 0x7fdee4137bf0 auth_method 0 2024-09-16T03:48:41.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.231+0000 7fdee1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:48:41.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.231+0000 7fdee1d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:48:41.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.231+0000 7fdee1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:48:41.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.231+0000 7fdee1d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:48:41.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.231+0000 7fdee1d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:48:41.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.231+0000 7fdee1d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:48:41.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.231+0000 7fdee1d9b640 10 monclient(hunting): handle_auth_done global_id 764535 payload 274 2024-09-16T03:48:41.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.231+0000 7fdee1d9b640 1 -- 172.21.15.42:0/1123004954 >> v1:172.21.15.154:6789/0 conn(0x7fdee4138000 legacy=0x7fdee413ec70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:48:41.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.231+0000 7fdee1d9b640 1 -- 172.21.15.42:0/1123004954 >> v1:172.21.15.42:6789/0 conn(0x7fdee40a4720 legacy=0x7fdee41374e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:48:41.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.231+0000 7fdee1d9b640 10 monclient: _finish_hunting 0 2024-09-16T03:48:41.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.231+0000 7fdee1d9b640 1 monclient: found mon.noname-a 2024-09-16T03:48:41.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.231+0000 7fdee1d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:48:41.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.231+0000 7fdee1d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:48:41.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.231+0000 7fdee1d9b640 1 -- 172.21.15.42:0/1123004954 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdee40a75a0 con 0x7fdee4137bf0 2024-09-16T03:48:41.242 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.231+0000 7fdee1d9b640 1 --2- 172.21.15.42:0/1123004954 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdee4137bf0 0x7fdee413c720 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fded80097c0 tx=0x7fded801c320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c18769983c30a4f0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:48:41.242 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.231+0000 7fdee159a640 1 -- 172.21.15.42:0/1123004954 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fded80b0070 con 0x7fdee4137bf0 2024-09-16T03:48:41.242 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.231+0000 7fdee159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:48:41.242 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.231+0000 7fdee159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:48:41.242 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.231+0000 7fdee159a640 10 monclient: dump: 2024-09-16T03:48:41.242 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:48:41.242 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:48:41.242 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:41.243 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:41.243 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:48:41.243 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:48:41.243 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:48:41.243 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:48:41.243 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:48:41.243 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:48:41.243 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.231+0000 7fdee159a640 10 monclient: _finish_auth 0 2024-09-16T03:48:41.243 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.231+0000 7fdee159a640 10 monclient: _check_auth_tickets 2024-09-16T03:48:41.243 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.231+0000 7fdee159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:48:41.243 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.231+0000 7fdee159a640 1 -- 172.21.15.42:0/1123004954 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fded80ab600 con 0x7fdee4137bf0 2024-09-16T03:48:41.243 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.231+0000 7fdee159a640 10 monclient: handle_config config(0 keys) 2024-09-16T03:48:41.243 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.231+0000 7fdee159a640 1 -- 172.21.15.42:0/1123004954 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fded80ab9d0 con 0x7fdee4137bf0 2024-09-16T03:48:41.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.231+0000 7fdee884d640 5 monclient: authenticate success, global_id 764535 2024-09-16T03:48:41.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.231+0000 7fdee884d640 10 monclient: get_monmap_and_config success 2024-09-16T03:48:41.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.231+0000 7fdee884d640 4 set_mon_vals no callback set 2024-09-16T03:48:41.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.231+0000 7fdee159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:48:41.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.231+0000 7fdee159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:48:41.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.231+0000 7fdee159a640 10 monclient: dump: 2024-09-16T03:48:41.244 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:48:41.244 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:48:41.244 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:41.245 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:41.245 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:48:41.245 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:48:41.245 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:48:41.245 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:48:41.245 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:48:41.245 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:48:41.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.231+0000 7fdee884d640 10 monclient: shutdown 2024-09-16T03:48:41.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.231+0000 7fdee884d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:48:41.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.231+0000 7fdee884d640 1 -- 172.21.15.42:0/1123004954 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdee4137bf0 msgr2=0x7fdee413c720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:48:41.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.231+0000 7fdee884d640 1 --2- 172.21.15.42:0/1123004954 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdee4137bf0 0x7fdee413c720 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fded80097c0 tx=0x7fded801c320 comp rx=0 tx=0).stop 2024-09-16T03:48:41.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.231+0000 7fdee884d640 1 -- 172.21.15.42:0/1123004954 shutdown_connections 2024-09-16T03:48:41.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.231+0000 7fdee884d640 1 --2- 172.21.15.42:0/1123004954 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdee4137bf0 0x7fdee413c720 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:41.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.231+0000 7fdee884d640 1 -- 172.21.15.42:0/1123004954 >> 172.21.15.42:0/1123004954 conn(0x7fdee40a6820 msgr2=0x7fdee40a6c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:48:41.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.231+0000 7fdee884d640 1 -- 172.21.15.42:0/1123004954 shutdown_connections 2024-09-16T03:48:41.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.231+0000 7fdee884d640 1 -- 172.21.15.42:0/1123004954 wait complete. 2024-09-16T03:48:41.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.235+0000 7fdee884d640 10 monclient: build_initial_monmap 2024-09-16T03:48:41.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.235+0000 7fdee884d640 10 monclient: monmap: 2024-09-16T03:48:41.246 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:48:41.246 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:48:41.246 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:48:41.246 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:48:41.246 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:48:41.246 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:48:41.246 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:48:41.246 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:48:41.246 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:48:41.247 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:48:41.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.235+0000 7fdee884d640 1 librados: starting msgr at 2024-09-16T03:48:41.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.235+0000 7fdee884d640 1 librados: starting objecter 2024-09-16T03:48:41.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.235+0000 7fdee884d640 1 Processor -- start 2024-09-16T03:48:41.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.235+0000 7fdee884d640 1 -- start start 2024-09-16T03:48:41.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.235+0000 7fdee884d640 1 librados: setting wanted keys 2024-09-16T03:48:41.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.235+0000 7fdee884d640 1 librados: calling monclient init 2024-09-16T03:48:41.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.235+0000 7fdee884d640 10 monclient: init 2024-09-16T03:48:41.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.235+0000 7fdee884d640 10 monclient: _reopen_session rank -1 2024-09-16T03:48:41.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.235+0000 7fdee884d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:48:41.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.235+0000 7fdee884d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdee40a4720 0x7fdee41cf1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:41.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.235+0000 7fdee884d640 10 monclient(hunting): picked mon.noname-b con 0x7fdee40a4720 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:48:41.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.235+0000 7fdee884d640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdee4137bf0 0x7fdee41cf8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:41.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.235+0000 7fdee884d640 10 monclient(hunting): picked mon.noname-a con 0x7fdee4137bf0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:48:41.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.235+0000 7fdee884d640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fdee4138000 0x7fdee413b470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:41.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.235+0000 7fdee884d640 10 monclient(hunting): picked mon.noname-c con 0x7fdee4138000 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:48:41.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.235+0000 7fdee884d640 10 monclient(hunting): start opening mon connection 2024-09-16T03:48:41.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.235+0000 7fdee884d640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fdee4142dc0 con 0x7fdee4137bf0 2024-09-16T03:48:41.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.235+0000 7fdee884d640 10 monclient(hunting): start opening mon connection 2024-09-16T03:48:41.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.235+0000 7fdee884d640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fdee4142f40 con 0x7fdee40a4720 2024-09-16T03:48:41.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.235+0000 7fdee884d640 10 monclient(hunting): start opening mon connection 2024-09-16T03:48:41.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.235+0000 7fdee884d640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fdee41430c0 con 0x7fdee4138000 2024-09-16T03:48:41.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.235+0000 7fdee884d640 10 monclient(hunting): _renew_subs 2024-09-16T03:48:41.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.235+0000 7fdee884d640 10 monclient(hunting): authenticate will time out at 1368.205311s 2024-09-16T03:48:41.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.235+0000 7fdee1d9b640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdee4137bf0 0x7fdee41cf8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:48:41.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.235+0000 7fdee1d9b640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdee4137bf0 0x7fdee41cf8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:44878/0 (socket says 172.21.15.42:44878) 2024-09-16T03:48:41.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.235+0000 7fdee1d9b640 1 -- 172.21.15.42:0/1775909926 learned_addr learned my addr 172.21.15.42:0/1775909926 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:48:41.249 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.235+0000 7fdee1d9b640 10 monclient(hunting): get_auth_request con 0x7fdee4137bf0 auth_method 0 2024-09-16T03:48:41.249 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.235+0000 7fdee1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:48:41.249 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.235+0000 7fdee1d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:48:41.249 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.235+0000 7fdee1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:48:41.249 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.235+0000 7fdee2d9d640 1 --2- 172.21.15.42:0/1775909926 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fdee4138000 0x7fdee413b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:48:41.249 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.235+0000 7fdee259c640 1 -- 172.21.15.42:0/1775909926 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdee40a4720 msgr2=0x7fdee41cf1c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T03:48:41.249 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.235+0000 7fdee259c640 1 --2- 172.21.15.42:0/1775909926 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdee40a4720 0x7fdee41cf1c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:48:41.249 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.235+0000 7fdee2d9d640 10 monclient(hunting): get_auth_request con 0x7fdee4138000 auth_method 0 2024-09-16T03:48:41.249 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.235+0000 7fdee2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:48:41.249 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.235+0000 7fdee2d9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:48:41.249 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.235+0000 7fdee2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:48:41.249 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.235+0000 7fdee1d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:48:41.249 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.235+0000 7fdee1d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:48:41.250 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.235+0000 7fdee1d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:48:41.250 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.235+0000 7fdee2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:48:41.250 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.235+0000 7fdee2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:48:41.250 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.235+0000 7fdee2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:48:41.250 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.235+0000 7fdee1d9b640 10 monclient(hunting): handle_auth_done global_id 764541 payload 824 2024-09-16T03:48:41.250 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.235+0000 7fdee1d9b640 1 -- 172.21.15.42:0/1775909926 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fdee4138000 msgr2=0x7fdee413b470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:48:41.250 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.235+0000 7fdee1d9b640 1 --2- 172.21.15.42:0/1775909926 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fdee4138000 0x7fdee413b470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:41.250 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.235+0000 7fdee1d9b640 1 -- 172.21.15.42:0/1775909926 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdee40a4720 msgr2=0x7fdee41cf1c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:48:41.250 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.235+0000 7fdee1d9b640 1 --2- 172.21.15.42:0/1775909926 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdee40a4720 0x7fdee41cf1c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:41.250 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.235+0000 7fdee1d9b640 10 monclient: _finish_hunting 0 2024-09-16T03:48:41.250 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.235+0000 7fdee1d9b640 1 monclient: found mon.noname-a 2024-09-16T03:48:41.250 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.235+0000 7fdee1d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:48:41.250 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.235+0000 7fdee1d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:48:41.250 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.235+0000 7fdee1d9b640 1 -- 172.21.15.42:0/1775909926 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdee413f0d0 con 0x7fdee4137bf0 2024-09-16T03:48:41.251 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.235+0000 7fdee1d9b640 1 --2- 172.21.15.42:0/1775909926 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdee4137bf0 0x7fdee41cf8f0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fded8018f30 tx=0x7fded80190c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:48:41.251 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.235+0000 7fded2ffd640 10 client.?.objecter ms_handle_connect 0x7fdee4137bf0 2024-09-16T03:48:41.251 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.235+0000 7fded2ffd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:48:41.251 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.235+0000 7fded2ffd640 1 -- 172.21.15.42:0/1775909926 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fded80b8020 con 0x7fdee4137bf0 2024-09-16T03:48:41.251 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.235+0000 7fded2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:48:41.251 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.235+0000 7fded2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:48:41.251 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.235+0000 7fded2ffd640 10 monclient: dump: 2024-09-16T03:48:41.251 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:48:41.251 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:48:41.251 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:41.251 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:41.251 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:48:41.251 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:48:41.252 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:48:41.252 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:48:41.252 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:48:41.252 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:48:41.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.235+0000 7fded2ffd640 10 monclient: _finish_auth 0 2024-09-16T03:48:41.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.235+0000 7fded2ffd640 10 monclient: _check_auth_tickets 2024-09-16T03:48:41.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.235+0000 7fded2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:48:41.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.235+0000 7fded2ffd640 1 -- 172.21.15.42:0/1775909926 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fded80aa620 con 0x7fdee4137bf0 2024-09-16T03:48:41.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.235+0000 7fdee884d640 5 monclient: authenticate success, global_id 764541 2024-09-16T03:48:41.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.235+0000 7fdee884d640 10 monclient: _renew_subs 2024-09-16T03:48:41.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.235+0000 7fdee884d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:48:41.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.235+0000 7fdee884d640 1 -- 172.21.15.42:0/1775909926 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdee413f3b0 con 0x7fdee4137bf0 2024-09-16T03:48:41.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.235+0000 7fded2ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:48:41.253 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.235+0000 7fded2ffd640 1 -- 172.21.15.42:0/1775909926 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fded80b0040 con 0x7fdee4137bf0 2024-09-16T03:48:41.253 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.235+0000 7fded2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:48:41.253 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.235+0000 7fded2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:48:41.253 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.235+0000 7fded2ffd640 10 monclient: dump: 2024-09-16T03:48:41.253 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:48:41.253 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:48:41.253 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:41.253 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:41.253 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:48:41.254 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:48:41.254 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:48:41.254 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:48:41.254 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:48:41.254 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:48:41.254 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.235+0000 7fded3fff640 4 set_mon_vals no callback set 2024-09-16T03:48:41.254 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.235+0000 7fdee884d640 10 client.764541.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:48:41.254 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.235+0000 7fdee884d640 10 monclient: _renew_subs 2024-09-16T03:48:41.254 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.235+0000 7fdee884d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:48:41.254 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.235+0000 7fdee884d640 1 -- 172.21.15.42:0/1775909926 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdee40a8510 con 0x7fdee4137bf0 2024-09-16T03:48:41.254 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.235+0000 7fdee884d640 1 librados: init done 2024-09-16T03:48:41.254 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.235+0000 7fdee884d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:48:41.254 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.235+0000 7fdee884d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:48:41.254 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.235+0000 7fdee884d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:48:41.255 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.235+0000 7fdee884d640 1 -- 172.21.15.42:0/1775909926 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdee41374e0 con 0x7fdee4137bf0 2024-09-16T03:48:41.255 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.235+0000 7fded2ffd640 1 -- 172.21.15.42:0/1775909926 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fded80bb020 con 0x7fdee4137bf0 2024-09-16T03:48:41.255 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.239+0000 7fded2ffd640 1 --2- 172.21.15.42:0/1775909926 >> v2:172.21.15.154:6812/254845342 conn(0x7fdec40377c0 0x7fdec4039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:41.255 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.239+0000 7fded2ffd640 1 -- 172.21.15.42:0/1775909926 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(216..216 src has 1..216) ==== 6980+0+0 (secure 0 0 0) 0x7fded80e37b0 con 0x7fdee4137bf0 2024-09-16T03:48:41.255 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.239+0000 7fded2ffd640 10 client.764541.objecter ms_dispatch 0x7fdee4000ba0 osd_map(216..216 src has 1..216) 2024-09-16T03:48:41.255 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.239+0000 7fded2ffd640 3 client.764541.objecter handle_osd_map got epochs [216,216] > 0 2024-09-16T03:48:41.255 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.239+0000 7fded2ffd640 3 client.764541.objecter handle_osd_map decoding full epoch 216 2024-09-16T03:48:41.255 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.239+0000 7fded2ffd640 20 client.764541.objecter dump_active .. 0 homeless 2024-09-16T03:48:41.255 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.239+0000 7fdee259c640 1 --2- 172.21.15.42:0/1775909926 >> v2:172.21.15.154:6812/254845342 conn(0x7fdec40377c0 0x7fdec4039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:48:41.255 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.239+0000 7fdee259c640 10 monclient: get_auth_request con 0x7fdec40377c0 auth_method 0 2024-09-16T03:48:41.255 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.239+0000 7fdee259c640 1 --2- 172.21.15.42:0/1775909926 >> v2:172.21.15.154:6812/254845342 conn(0x7fdec40377c0 0x7fdec4039c80 secure :-1 s=READY pgs=3497 cs=0 l=1 rev1=1 crypto rx=0x7fded4004820 tx=0x7fded4007af0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:48:41.255 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.239+0000 7fded2ffd640 10 client.764541.objecter ms_handle_connect 0x7fdec40377c0 2024-09-16T03:48:41.256 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.243+0000 7fded2ffd640 1 -- 172.21.15.42:0/1775909926 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7fded80c2030 con 0x7fdee4137bf0 2024-09-16T03:48:41.256 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.243+0000 7fded2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:48:41.256 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.243+0000 7fded2ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:48:41.468 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.463+0000 7fdee884d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:48:41.469 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.463+0000 7fdee884d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:48:41.469 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.463+0000 7fdee884d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:48:41.469 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.463+0000 7fdee884d640 1 -- 172.21.15.42:0/1775909926 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fdee413d9a0 con 0x7fdee4137bf0 2024-09-16T03:48:41.469 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.467+0000 7fded2ffd640 1 -- 172.21.15.42:0/1775909926 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7fded8016030 con 0x7fdee4137bf0 2024-09-16T03:48:41.469 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.467+0000 7fded2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:48:41.469 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.467+0000 7fded2ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:48:41.470 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":60,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:48:41.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.467+0000 7fdee884d640 10 librados: watch_flush enter 2024-09-16T03:48:41.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.467+0000 7fdee884d640 10 librados: watch_flush exit 2024-09-16T03:48:41.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.467+0000 7fdee884d640 20 client.764541.objecter shutdown clearing up homeless session... 2024-09-16T03:48:41.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.467+0000 7fdee884d640 10 client.764541.objecter successfully canceled tick 2024-09-16T03:48:41.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.467+0000 7fdee884d640 1 -- 172.21.15.42:0/1775909926 >> v2:172.21.15.154:6812/254845342 conn(0x7fdec40377c0 msgr2=0x7fdec4039c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:48:41.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.467+0000 7fdee884d640 1 --2- 172.21.15.42:0/1775909926 >> v2:172.21.15.154:6812/254845342 conn(0x7fdec40377c0 0x7fdec4039c80 secure :-1 s=READY pgs=3497 cs=0 l=1 rev1=1 crypto rx=0x7fded4004820 tx=0x7fded4007af0 comp rx=0 tx=0).stop 2024-09-16T03:48:41.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.467+0000 7fdee884d640 10 monclient: shutdown 2024-09-16T03:48:41.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.467+0000 7fdee884d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:48:41.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.467+0000 7fdee884d640 1 -- 172.21.15.42:0/1775909926 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdee4137bf0 msgr2=0x7fdee41cf8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:48:41.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.467+0000 7fdee884d640 1 --2- 172.21.15.42:0/1775909926 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdee4137bf0 0x7fdee41cf8f0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fded8018f30 tx=0x7fded80190c0 comp rx=0 tx=0).stop 2024-09-16T03:48:41.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.467+0000 7fdee884d640 1 -- 172.21.15.42:0/1775909926 shutdown_connections 2024-09-16T03:48:41.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.467+0000 7fdee884d640 1 --2- 172.21.15.42:0/1775909926 >> v2:172.21.15.154:6812/254845342 conn(0x7fdec40377c0 0x7fdec4039c80 unknown :-1 s=CLOSED pgs=3497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:41.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.467+0000 7fdee884d640 1 --2- 172.21.15.42:0/1775909926 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fdee4138000 0x7fdee413b470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:41.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.467+0000 7fdee884d640 1 --2- 172.21.15.42:0/1775909926 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdee4137bf0 0x7fdee41cf8f0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:41.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.467+0000 7fdee884d640 1 --2- 172.21.15.42:0/1775909926 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdee40a4720 0x7fdee41cf1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:41.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.467+0000 7fdee884d640 1 -- 172.21.15.42:0/1775909926 >> 172.21.15.42:0/1775909926 conn(0x7fdee40a6820 msgr2=0x7fdee409da60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:48:41.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.467+0000 7fdee884d640 1 -- 172.21.15.42:0/1775909926 shutdown_connections 2024-09-16T03:48:41.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.467+0000 7fdee884d640 1 -- 172.21.15.42:0/1775909926 wait complete. 2024-09-16T03:48:41.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:41.467+0000 7fdee884d640 1 librados: shutdown 2024-09-16T03:48:44.488 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:48:44.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.615+0000 7f20f100c640 10 monclient: get_monmap_and_config 2024-09-16T03:48:44.619 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.615+0000 7f20f100c640 10 monclient: build_initial_monmap 2024-09-16T03:48:44.619 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.615+0000 7f20f100c640 10 monclient: monmap: 2024-09-16T03:48:44.619 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:48:44.619 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:48:44.619 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:48:44.617923+0000 2024-09-16T03:48:44.619 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:48:44.617923+0000 2024-09-16T03:48:44.619 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:48:44.619 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:48:44.619 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:48:44.619 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:48:44.619 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:48:44.620 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:48:44.620 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:48:44.620 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:48:44.622 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.615+0000 7f20f100c640 1 Processor -- start 2024-09-16T03:48:44.622 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.615+0000 7f20f100c640 1 -- start start 2024-09-16T03:48:44.622 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.615+0000 7f20f100c640 10 monclient: init 2024-09-16T03:48:44.622 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.615+0000 7f20f100c640 10 monclient: _reopen_session rank -1 2024-09-16T03:48:44.622 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.615+0000 7f20f100c640 10 monclient: _add_conns ranks=[0,2,3,4,1] 2024-09-16T03:48:44.622 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.615+0000 7f20f100c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f20ec0a1330 addr v1:172.21.15.42:6789/0 2024-09-16T03:48:44.622 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.615+0000 7f20f100c640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f20ec06dac0 0x7f20ec13a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:44.622 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.615+0000 7f20f100c640 10 monclient(hunting): picked mon.noname-b con 0x7f20ec06dac0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:48:44.622 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.615+0000 7f20f100c640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f20ec0a1e20 0x7f20ec13c890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:44.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.615+0000 7f20f100c640 10 monclient(hunting): picked mon.noname-a con 0x7f20ec0a1e20 addr v2:172.21.15.42:3300/0 2024-09-16T03:48:44.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.615+0000 7f20f100c640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f20ec09c7d0 con 0x7f20ec0a1330 2024-09-16T03:48:44.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.615+0000 7f20f100c640 10 monclient(hunting): start opening mon connection 2024-09-16T03:48:44.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.615+0000 7f20f100c640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f20ec09c090 con 0x7f20ec0a1e20 2024-09-16T03:48:44.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.615+0000 7f20f100c640 10 monclient(hunting): start opening mon connection 2024-09-16T03:48:44.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.615+0000 7f20f100c640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f20ec09f610 con 0x7f20ec06dac0 2024-09-16T03:48:44.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.615+0000 7f20f100c640 10 monclient(hunting): _renew_subs 2024-09-16T03:48:44.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.615+0000 7f20f100c640 10 monclient(hunting): authenticate will time out at 1371.587394s 2024-09-16T03:48:44.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.615+0000 7f20ea59c640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f20ec06dac0 0x7f20ec13a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:48:44.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.615+0000 7f20eb59e640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f20ec0a1e20 0x7f20ec13c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:48:44.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.615+0000 7f20ea59c640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f20ec06dac0 0x7f20ec13a340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:39882/0 (socket says 172.21.15.42:39882) 2024-09-16T03:48:44.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.615+0000 7f20eb59e640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f20ec0a1e20 0x7f20ec13c890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:55228/0 (socket says 172.21.15.42:55228) 2024-09-16T03:48:44.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.615+0000 7f20ea59c640 1 -- 172.21.15.42:0/3304208844 learned_addr learned my addr 172.21.15.42:0/3304208844 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:48:44.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.615+0000 7f20ea59c640 10 monclient(hunting): get_auth_request con 0x7f20ec06dac0 auth_method 0 2024-09-16T03:48:44.625 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.615+0000 7f20ea59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:48:44.625 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.615+0000 7f20ea59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:48:44.625 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.615+0000 7f20ea59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:48:44.625 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.615+0000 7f20eb59e640 10 monclient(hunting): get_auth_request con 0x7f20ec0a1e20 auth_method 0 2024-09-16T03:48:44.625 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.615+0000 7f20eb59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:48:44.625 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.615+0000 7f20eb59e640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:48:44.626 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.615+0000 7f20eb59e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:48:44.626 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.615+0000 7f20ea59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:48:44.626 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.615+0000 7f20ea59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:48:44.626 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.615+0000 7f20ea59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:48:44.626 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.615+0000 7f20eb59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:48:44.626 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.615+0000 7f20eb59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:48:44.627 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.615+0000 7f20eb59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:48:44.627 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.615+0000 7f20ea59c640 10 monclient(hunting): handle_auth_done global_id 784142 payload 274 2024-09-16T03:48:44.627 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.615+0000 7f20ea59c640 1 -- 172.21.15.42:0/3304208844 >> v2:172.21.15.42:3300/0 conn(0x7f20ec0a1e20 msgr2=0x7f20ec13c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:48:44.627 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.615+0000 7f20ea59c640 1 --2- 172.21.15.42:0/3304208844 >> v2:172.21.15.42:3300/0 conn(0x7f20ec0a1e20 0x7f20ec13c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:44.627 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.615+0000 7f20ea59c640 1 -- 172.21.15.42:0/3304208844 >> v1:172.21.15.42:6789/0 conn(0x7f20ec0a1330 legacy=0x7f20ec0a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:48:44.627 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.615+0000 7f20ea59c640 10 monclient: _finish_hunting 0 2024-09-16T03:48:44.627 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.615+0000 7f20ea59c640 1 monclient: found mon.noname-b 2024-09-16T03:48:44.628 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.615+0000 7f20ea59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:48:44.628 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.615+0000 7f20ea59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T03:48:44.628 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.615+0000 7f20ea59c640 1 -- 172.21.15.42:0/3304208844 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20ec09f8f0 con 0x7f20ec06dac0 2024-09-16T03:48:44.636 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.619+0000 7f20ea59c640 1 --2- 172.21.15.42:0/3304208844 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f20ec06dac0 0x7f20ec13a340 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f20dc009e50 tx=0x7f20dc01c9e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=40beced45b0094f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:48:44.637 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.619+0000 7f20e9d9b640 1 -- 172.21.15.42:0/3304208844 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20dc0a58d0 con 0x7f20ec06dac0 2024-09-16T03:48:44.637 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.619+0000 7f20e9d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:48:44.637 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.619+0000 7f20e9d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:48:44.637 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.619+0000 7f20e9d9b640 10 monclient: dump: 2024-09-16T03:48:44.637 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:48:44.637 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:48:44.638 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:44.638 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:44.638 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:48:44.638 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:48:44.638 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:48:44.638 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:48:44.638 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:48:44.639 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:48:44.639 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.619+0000 7f20e9d9b640 10 monclient: _finish_auth 0 2024-09-16T03:48:44.639 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.619+0000 7f20e9d9b640 10 monclient: _check_auth_tickets 2024-09-16T03:48:44.639 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.619+0000 7f20e9d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:48:44.639 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.619+0000 7f20e9d9b640 1 -- 172.21.15.42:0/3304208844 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f20dc0a5a70 con 0x7f20ec06dac0 2024-09-16T03:48:44.639 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.619+0000 7f20e9d9b640 10 monclient: handle_config config(0 keys) 2024-09-16T03:48:44.639 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.619+0000 7f20e9d9b640 1 -- 172.21.15.42:0/3304208844 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20dc0a5e40 con 0x7f20ec06dac0 2024-09-16T03:48:44.639 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.619+0000 7f20f100c640 5 monclient: authenticate success, global_id 784142 2024-09-16T03:48:44.640 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.619+0000 7f20e9d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:48:44.640 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.619+0000 7f20e9d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:48:44.640 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.619+0000 7f20e9d9b640 10 monclient: dump: 2024-09-16T03:48:44.640 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:48:44.640 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:48:44.640 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:44.640 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:44.640 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:48:44.640 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:48:44.641 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:48:44.641 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:48:44.641 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:48:44.641 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:48:44.641 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.619+0000 7f20f100c640 10 monclient: get_monmap_and_config success 2024-09-16T03:48:44.641 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.619+0000 7f20f100c640 4 set_mon_vals no callback set 2024-09-16T03:48:44.641 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.619+0000 7f20f100c640 10 monclient: shutdown 2024-09-16T03:48:44.642 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.619+0000 7f20f100c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:48:44.642 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.619+0000 7f20f100c640 1 -- 172.21.15.42:0/3304208844 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f20ec06dac0 msgr2=0x7f20ec13a340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:48:44.642 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.619+0000 7f20f100c640 1 --2- 172.21.15.42:0/3304208844 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f20ec06dac0 0x7f20ec13a340 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f20dc009e50 tx=0x7f20dc01c9e0 comp rx=0 tx=0).stop 2024-09-16T03:48:44.642 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.619+0000 7f20f100c640 1 -- 172.21.15.42:0/3304208844 shutdown_connections 2024-09-16T03:48:44.642 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.619+0000 7f20f100c640 1 --2- 172.21.15.42:0/3304208844 >> v2:172.21.15.42:3300/0 conn(0x7f20ec0a1e20 0x7f20ec13c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:44.642 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.619+0000 7f20f100c640 1 --2- 172.21.15.42:0/3304208844 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f20ec06dac0 0x7f20ec13a340 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:44.642 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.619+0000 7f20f100c640 1 -- 172.21.15.42:0/3304208844 >> 172.21.15.42:0/3304208844 conn(0x7f20ec0a2420 msgr2=0x7f20ec0a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:48:44.643 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.619+0000 7f20f100c640 1 -- 172.21.15.42:0/3304208844 shutdown_connections 2024-09-16T03:48:44.643 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.619+0000 7f20f100c640 1 -- 172.21.15.42:0/3304208844 wait complete. 2024-09-16T03:48:44.643 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.619+0000 7f20f100c640 10 monclient: build_initial_monmap 2024-09-16T03:48:44.643 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.619+0000 7f20f100c640 10 monclient: monmap: 2024-09-16T03:48:44.643 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:48:44.643 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:48:44.643 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:48:44.643 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:48:44.643 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:48:44.644 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:48:44.644 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:48:44.644 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:48:44.644 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:48:44.644 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:48:44.644 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.623+0000 7f20f100c640 1 librados: starting msgr at 2024-09-16T03:48:44.644 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.623+0000 7f20f100c640 1 librados: starting objecter 2024-09-16T03:48:44.644 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.623+0000 7f20f100c640 1 Processor -- start 2024-09-16T03:48:44.644 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.623+0000 7f20f100c640 1 -- start start 2024-09-16T03:48:44.644 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.623+0000 7f20f100c640 1 librados: setting wanted keys 2024-09-16T03:48:44.644 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.623+0000 7f20f100c640 1 librados: calling monclient init 2024-09-16T03:48:44.644 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.623+0000 7f20f100c640 10 monclient: init 2024-09-16T03:48:44.644 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.623+0000 7f20f100c640 10 monclient: _reopen_session rank -1 2024-09-16T03:48:44.645 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.623+0000 7f20f100c640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:48:44.645 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.623+0000 7f20f100c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f20ec0a1330 0x7f20ec1ccb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:44.645 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.623+0000 7f20f100c640 10 monclient(hunting): picked mon.noname-a con 0x7f20ec0a1330 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:48:44.645 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.623+0000 7f20f100c640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f20ec0a1e20 0x7f20ec1322e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:44.645 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.623+0000 7f20f100c640 10 monclient(hunting): picked mon.noname-c con 0x7f20ec0a1e20 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:48:44.645 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.623+0000 7f20f100c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f20ec1392e0 0x7f20ec139790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:44.645 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.623+0000 7f20f100c640 10 monclient(hunting): picked mon.noname-b con 0x7f20ec1392e0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:48:44.645 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.623+0000 7f20f100c640 10 monclient(hunting): start opening mon connection 2024-09-16T03:48:44.645 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.623+0000 7f20f100c640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f20ec13f770 con 0x7f20ec0a1330 2024-09-16T03:48:44.645 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.623+0000 7f20f100c640 10 monclient(hunting): start opening mon connection 2024-09-16T03:48:44.645 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.623+0000 7f20f100c640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f20ec13f5f0 con 0x7f20ec1392e0 2024-09-16T03:48:44.645 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.623+0000 7f20f100c640 10 monclient(hunting): start opening mon connection 2024-09-16T03:48:44.646 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.623+0000 7f20f100c640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f20ec13f8f0 con 0x7f20ec0a1e20 2024-09-16T03:48:44.646 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.623+0000 7f20f100c640 10 monclient(hunting): _renew_subs 2024-09-16T03:48:44.646 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.623+0000 7f20f100c640 10 monclient(hunting): authenticate will time out at 1371.593209s 2024-09-16T03:48:44.646 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.623+0000 7f20ea59c640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f20ec0a1e20 0x7f20ec1322e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:48:44.646 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.623+0000 7f20eb59e640 1 -- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f20ec1392e0 msgr2=0x7f20ec139790 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T03:48:44.646 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.623+0000 7f20ea59c640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f20ec0a1e20 0x7f20ec1322e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:39894/0 (socket says 172.21.15.42:39894) 2024-09-16T03:48:44.646 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.623+0000 7f20ea59c640 1 -- 172.21.15.42:0/3943249104 learned_addr learned my addr 172.21.15.42:0/3943249104 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:48:44.646 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.623+0000 7f20ead9d640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f20ec0a1330 0x7f20ec1ccb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:48:44.647 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.623+0000 7f20ea59c640 10 monclient(hunting): get_auth_request con 0x7f20ec0a1e20 auth_method 0 2024-09-16T03:48:44.647 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.623+0000 7f20ea59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:48:44.647 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.623+0000 7f20eb59e640 1 --2- 172.21.15.42:0/3943249104 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f20ec1392e0 0x7f20ec139790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:48:44.647 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.623+0000 7f20ea59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:48:44.647 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.623+0000 7f20ea59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:48:44.647 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.623+0000 7f20ead9d640 10 monclient(hunting): get_auth_request con 0x7f20ec0a1330 auth_method 0 2024-09-16T03:48:44.647 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.623+0000 7f20ead9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:48:44.647 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.623+0000 7f20ead9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:48:44.647 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.623+0000 7f20ead9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:48:44.647 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.623+0000 7f20ea59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:48:44.647 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.623+0000 7f20ea59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:48:44.647 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.623+0000 7f20ea59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:48:44.647 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.623+0000 7f20ead9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:48:44.648 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.623+0000 7f20ead9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:48:44.648 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.623+0000 7f20ead9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:48:44.648 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.623+0000 7f20ea59c640 10 monclient(hunting): handle_auth_done global_id 784145 payload 824 2024-09-16T03:48:44.648 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.623+0000 7f20ea59c640 1 -- 172.21.15.42:0/3943249104 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f20ec1392e0 msgr2=0x7f20ec139790 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:48:44.648 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.623+0000 7f20ea59c640 1 --2- 172.21.15.42:0/3943249104 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f20ec1392e0 0x7f20ec139790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:44.648 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.623+0000 7f20ea59c640 1 -- 172.21.15.42:0/3943249104 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f20ec0a1330 msgr2=0x7f20ec1ccb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:48:44.648 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.623+0000 7f20ea59c640 1 --2- 172.21.15.42:0/3943249104 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f20ec0a1330 0x7f20ec1ccb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:44.648 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.623+0000 7f20ea59c640 10 monclient: _finish_hunting 0 2024-09-16T03:48:44.648 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.623+0000 7f20ea59c640 1 monclient: found mon.noname-c 2024-09-16T03:48:44.648 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.623+0000 7f20ea59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:48:44.648 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.623+0000 7f20ea59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:48:44.648 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.623+0000 7f20ea59c640 1 -- 172.21.15.42:0/3943249104 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20ec11c9b0 con 0x7f20ec0a1e20 2024-09-16T03:48:44.648 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.623+0000 7f20ead9d640 1 --2- 172.21.15.42:0/3943249104 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f20ec0a1330 0x7f20ec1ccb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:48:44.649 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.623+0000 7f20ea59c640 1 --2- 172.21.15.42:0/3943249104 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f20ec0a1e20 0x7f20ec1322e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f20dc0af040 tx=0x7f20dc019780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:48:44.649 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.623+0000 7f20e37fe640 10 client.?.objecter ms_handle_connect 0x7f20ec0a1e20 2024-09-16T03:48:44.649 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.623+0000 7f20e37fe640 10 client.?.objecter resend_mon_ops 2024-09-16T03:48:44.649 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.623+0000 7f20e37fe640 1 -- 172.21.15.42:0/3943249104 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20dc0bc020 con 0x7f20ec0a1e20 2024-09-16T03:48:44.649 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.623+0000 7f20e37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:48:44.649 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.623+0000 7f20e37fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:48:44.649 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.623+0000 7f20e37fe640 10 monclient: dump: 2024-09-16T03:48:44.649 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:48:44.649 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:48:44.649 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:44.649 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:44.649 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:48:44.649 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:48:44.650 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:48:44.650 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:48:44.650 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:48:44.650 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:48:44.650 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.623+0000 7f20e37fe640 10 monclient: _finish_auth 0 2024-09-16T03:48:44.650 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.623+0000 7f20e37fe640 10 monclient: _check_auth_tickets 2024-09-16T03:48:44.650 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.623+0000 7f20e37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:48:44.650 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.623+0000 7f20e37fe640 1 -- 172.21.15.42:0/3943249104 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f20dc0b8470 con 0x7f20ec0a1e20 2024-09-16T03:48:44.650 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.623+0000 7f20e37fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:48:44.650 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.623+0000 7f20e37fe640 1 -- 172.21.15.42:0/3943249104 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20dc0a58d0 con 0x7f20ec0a1e20 2024-09-16T03:48:44.650 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.623+0000 7f20e37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:48:44.650 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.623+0000 7f20e37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:48:44.650 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.623+0000 7f20e37fe640 10 monclient: dump: 2024-09-16T03:48:44.651 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:48:44.651 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:48:44.651 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:44.651 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:44.651 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:48:44.651 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:48:44.651 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:48:44.651 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:48:44.651 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:48:44.651 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:48:44.651 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.623+0000 7f20f100c640 5 monclient: authenticate success, global_id 784145 2024-09-16T03:48:44.651 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.623+0000 7f20e959a640 4 set_mon_vals no callback set 2024-09-16T03:48:44.651 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.623+0000 7f20f100c640 10 monclient: _renew_subs 2024-09-16T03:48:44.651 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.623+0000 7f20f100c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:48:44.652 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.623+0000 7f20f100c640 1 -- 172.21.15.42:0/3943249104 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20ec06e060 con 0x7f20ec0a1e20 2024-09-16T03:48:44.652 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.623+0000 7f20f100c640 10 client.784145.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:48:44.652 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.623+0000 7f20f100c640 10 monclient: _renew_subs 2024-09-16T03:48:44.652 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.623+0000 7f20f100c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:48:44.652 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.623+0000 7f20f100c640 1 -- 172.21.15.42:0/3943249104 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f20ec13af40 con 0x7f20ec0a1e20 2024-09-16T03:48:44.652 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.623+0000 7f20f100c640 1 librados: init done 2024-09-16T03:48:44.652 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.623+0000 7f20f100c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:48:44.652 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.623+0000 7f20f100c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:48:44.652 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.623+0000 7f20e37fe640 1 -- 172.21.15.42:0/3943249104 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f20dc0bf020 con 0x7f20ec0a1e20 2024-09-16T03:48:44.652 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.623+0000 7f20f100c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:48:44.652 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.623+0000 7f20f100c640 1 -- 172.21.15.42:0/3943249104 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f20ec0a1710 con 0x7f20ec0a1e20 2024-09-16T03:48:44.652 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.623+0000 7f20e37fe640 1 --2- 172.21.15.42:0/3943249104 >> v2:172.21.15.154:6812/254845342 conn(0x7f20cc0377c0 0x7f20cc039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:44.652 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.623+0000 7f20e37fe640 1 -- 172.21.15.42:0/3943249104 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(216..216 src has 1..216) ==== 6980+0+0 (secure 0 0 0) 0x7f20dc0e4800 con 0x7f20ec0a1e20 2024-09-16T03:48:44.653 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.623+0000 7f20e37fe640 10 client.784145.objecter ms_dispatch 0x7f20ec000ba0 osd_map(216..216 src has 1..216) 2024-09-16T03:48:44.653 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.623+0000 7f20e37fe640 3 client.784145.objecter handle_osd_map got epochs [216,216] > 0 2024-09-16T03:48:44.653 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.623+0000 7f20e37fe640 3 client.784145.objecter handle_osd_map decoding full epoch 216 2024-09-16T03:48:44.653 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.623+0000 7f20e37fe640 20 client.784145.objecter dump_active .. 0 homeless 2024-09-16T03:48:44.653 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.623+0000 7f20ead9d640 1 --2- 172.21.15.42:0/3943249104 >> v2:172.21.15.154:6812/254845342 conn(0x7f20cc0377c0 0x7f20cc039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:48:44.653 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.623+0000 7f20ead9d640 10 monclient: get_auth_request con 0x7f20cc0377c0 auth_method 0 2024-09-16T03:48:44.653 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.627+0000 7f20ead9d640 1 --2- 172.21.15.42:0/3943249104 >> v2:172.21.15.154:6812/254845342 conn(0x7f20cc0377c0 0x7f20cc039c80 secure :-1 s=READY pgs=3498 cs=0 l=1 rev1=1 crypto rx=0x7f20d4001c00 tx=0x7f20d4009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:48:44.653 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.627+0000 7f20e37fe640 10 client.784145.objecter ms_handle_connect 0x7f20cc0377c0 2024-09-16T03:48:44.653 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.631+0000 7f20e37fe640 1 -- 172.21.15.42:0/3943249104 <== mon.2 v2:172.21.15.42:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f20dc0c6070 con 0x7f20ec0a1e20 2024-09-16T03:48:44.653 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.631+0000 7f20e37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:48:44.653 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.631+0000 7f20e37fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:48:44.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.851+0000 7f20f100c640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:48:44.857 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.851+0000 7f20f100c640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:48:44.857 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.851+0000 7f20f100c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:48:44.857 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.851+0000 7f20f100c640 1 -- 172.21.15.42:0/3943249104 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f20ec1cd9d0 con 0x7f20ec0a1e20 2024-09-16T03:48:44.857 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.855+0000 7f20e37fe640 1 -- 172.21.15.42:0/3943249104 <== mon.2 v2:172.21.15.42:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7f20dc016600 con 0x7f20ec0a1e20 2024-09-16T03:48:44.857 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.855+0000 7f20e37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:48:44.857 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.855+0000 7f20e37fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:48:44.858 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":60,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:48:44.860 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.855+0000 7f20f100c640 10 librados: watch_flush enter 2024-09-16T03:48:44.860 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.855+0000 7f20f100c640 10 librados: watch_flush exit 2024-09-16T03:48:44.860 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.855+0000 7f20f100c640 20 client.784145.objecter shutdown clearing up homeless session... 2024-09-16T03:48:44.860 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.855+0000 7f20f100c640 10 client.784145.objecter successfully canceled tick 2024-09-16T03:48:44.860 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.855+0000 7f20f100c640 1 -- 172.21.15.42:0/3943249104 >> v2:172.21.15.154:6812/254845342 conn(0x7f20cc0377c0 msgr2=0x7f20cc039c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:48:44.860 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.855+0000 7f20f100c640 1 --2- 172.21.15.42:0/3943249104 >> v2:172.21.15.154:6812/254845342 conn(0x7f20cc0377c0 0x7f20cc039c80 secure :-1 s=READY pgs=3498 cs=0 l=1 rev1=1 crypto rx=0x7f20d4001c00 tx=0x7f20d4009040 comp rx=0 tx=0).stop 2024-09-16T03:48:44.861 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.855+0000 7f20f100c640 10 monclient: shutdown 2024-09-16T03:48:44.861 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.855+0000 7f20f100c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:48:44.861 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.855+0000 7f20f100c640 1 -- 172.21.15.42:0/3943249104 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f20ec0a1e20 msgr2=0x7f20ec1322e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:48:44.861 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.855+0000 7f20f100c640 1 --2- 172.21.15.42:0/3943249104 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f20ec0a1e20 0x7f20ec1322e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f20dc0af040 tx=0x7f20dc019780 comp rx=0 tx=0).stop 2024-09-16T03:48:44.861 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.855+0000 7f20f100c640 1 -- 172.21.15.42:0/3943249104 shutdown_connections 2024-09-16T03:48:44.861 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.855+0000 7f20f100c640 1 --2- 172.21.15.42:0/3943249104 >> v2:172.21.15.154:6812/254845342 conn(0x7f20cc0377c0 0x7f20cc039c80 unknown :-1 s=CLOSED pgs=3498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:44.861 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.855+0000 7f20f100c640 1 --2- 172.21.15.42:0/3943249104 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f20ec1392e0 0x7f20ec139790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:44.861 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.855+0000 7f20f100c640 1 --2- 172.21.15.42:0/3943249104 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f20ec0a1e20 0x7f20ec1322e0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:44.862 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.855+0000 7f20f100c640 1 --2- 172.21.15.42:0/3943249104 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f20ec0a1330 0x7f20ec1ccb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:44.862 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.855+0000 7f20f100c640 1 -- 172.21.15.42:0/3943249104 >> 172.21.15.42:0/3943249104 conn(0x7f20ec0a2420 msgr2=0x7f20ec135300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:48:44.862 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.855+0000 7f20f100c640 1 -- 172.21.15.42:0/3943249104 shutdown_connections 2024-09-16T03:48:44.862 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.855+0000 7f20f100c640 1 -- 172.21.15.42:0/3943249104 wait complete. 2024-09-16T03:48:44.862 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:44.855+0000 7f20f100c640 1 librados: shutdown 2024-09-16T03:48:47.875 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:48:48.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.007+0000 7efedbe9b640 10 monclient: get_monmap_and_config 2024-09-16T03:48:48.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.007+0000 7efedbe9b640 10 monclient: build_initial_monmap 2024-09-16T03:48:48.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.007+0000 7efedbe9b640 10 monclient: monmap: 2024-09-16T03:48:48.010 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:48:48.010 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:48:48.011 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:48:48.009518+0000 2024-09-16T03:48:48.011 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:48:48.009518+0000 2024-09-16T03:48:48.011 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:48:48.011 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:48:48.011 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:48:48.011 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:48:48.011 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:48:48.011 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:48:48.011 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:48:48.011 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:48:48.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.007+0000 7efedbe9b640 1 Processor -- start 2024-09-16T03:48:48.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.007+0000 7efedbe9b640 1 -- start start 2024-09-16T03:48:48.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.007+0000 7efedbe9b640 10 monclient: init 2024-09-16T03:48:48.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.007+0000 7efedbe9b640 10 monclient: _reopen_session rank -1 2024-09-16T03:48:48.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.007+0000 7efedbe9b640 10 monclient: _add_conns ranks=[2,0,4,3,1] 2024-09-16T03:48:48.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.007+0000 7efedbe9b640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7efed406db90 0x7efed41340c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:48.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.007+0000 7efedbe9b640 10 monclient(hunting): picked mon.noname-b con 0x7efed406db90 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:48:48.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.007+0000 7efedbe9b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7efed40a37f0 addr v1:172.21.15.42:6789/0 2024-09-16T03:48:48.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.007+0000 7efedbe9b640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7efed40a3bd0 0x7efed4137520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:48.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.007+0000 7efedbe9b640 10 monclient(hunting): picked mon.noname-c con 0x7efed40a3bd0 addr v2:172.21.15.154:3300/0 2024-09-16T03:48:48.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.007+0000 7efedbe9b640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efed406ece0 con 0x7efed40a37f0 2024-09-16T03:48:48.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.007+0000 7efedbe9b640 10 monclient(hunting): start opening mon connection 2024-09-16T03:48:48.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.007+0000 7efedbe9b640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7efed406df70 con 0x7efed40a3bd0 2024-09-16T03:48:48.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.007+0000 7efedbe9b640 10 monclient(hunting): start opening mon connection 2024-09-16T03:48:48.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.007+0000 7efedbe9b640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7efed409bf80 con 0x7efed406db90 2024-09-16T03:48:48.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.007+0000 7efedbe9b640 10 monclient(hunting): _renew_subs 2024-09-16T03:48:48.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.007+0000 7efedbe9b640 10 monclient(hunting): authenticate will time out at 1374.978984s 2024-09-16T03:48:48.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.007+0000 7efed9c37640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7efed406db90 0x7efed41340c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:48:48.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.007+0000 7efed9436640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7efed40a37f0 0x7efed4136e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:46170/0 (socket says 172.21.15.42:46170) 2024-09-16T03:48:48.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.007+0000 7efed9436640 1 -- 172.21.15.42:0/1991997555 learned_addr learned my addr 172.21.15.42:0/1991997555 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:48:48.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.007+0000 7efeda438640 1 -- 172.21.15.42:0/1991997555 >> v2:172.21.15.154:3300/0 conn(0x7efed40a3bd0 msgr2=0x7efed4137520 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T03:48:48.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.007+0000 7efed9c37640 10 monclient(hunting): get_auth_request con 0x7efed406db90 auth_method 0 2024-09-16T03:48:48.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.007+0000 7efed9c37640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:48:48.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.007+0000 7efed9c37640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:48:48.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.007+0000 7efed9c37640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:48:48.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.007+0000 7efeda438640 1 --2- 172.21.15.42:0/1991997555 >> v2:172.21.15.154:3300/0 conn(0x7efed40a3bd0 0x7efed4137520 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:48:48.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.007+0000 7efed9c37640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:48:48.016 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.007+0000 7efed9c37640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:48:48.016 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.007+0000 7efed9c37640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:48:48.016 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.007+0000 7efed9c37640 10 monclient(hunting): handle_auth_done global_id 784148 payload 274 2024-09-16T03:48:48.016 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.007+0000 7efed9c37640 1 -- 172.21.15.42:0/1991997555 >> v2:172.21.15.154:3300/0 conn(0x7efed40a3bd0 msgr2=0x7efed4137520 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:48:48.016 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.007+0000 7efed9c37640 1 --2- 172.21.15.42:0/1991997555 >> v2:172.21.15.154:3300/0 conn(0x7efed40a3bd0 0x7efed4137520 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:48.016 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.007+0000 7efed9c37640 1 -- 172.21.15.42:0/1991997555 >> v1:172.21.15.42:6789/0 conn(0x7efed40a37f0 legacy=0x7efed4136e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:48:48.016 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.007+0000 7efed9c37640 10 monclient: _finish_hunting 0 2024-09-16T03:48:48.016 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.007+0000 7efed9c37640 1 monclient: found mon.noname-b 2024-09-16T03:48:48.016 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.007+0000 7efed9c37640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:48:48.016 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.007+0000 7efed9c37640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T03:48:48.016 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.007+0000 7efed9c37640 1 -- 172.21.15.42:0/1991997555 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efed4134780 con 0x7efed406db90 2024-09-16T03:48:48.025 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.011+0000 7efed9c37640 1 --2- 172.21.15.42:0/1991997555 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7efed406db90 0x7efed41340c0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7efec40097c0 tx=0x7efec401c380 comp rx=0 tx=0).ready entity=mon.2 client_cookie=23487457467acda3 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:48:48.025 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.011+0000 7efed8c35640 1 -- 172.21.15.42:0/1991997555 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efec40b0070 con 0x7efed406db90 2024-09-16T03:48:48.025 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.011+0000 7efed8c35640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:48:48.025 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.011+0000 7efed8c35640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:48:48.025 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.011+0000 7efed8c35640 10 monclient: dump: 2024-09-16T03:48:48.025 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:48:48.025 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:48:48.025 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:48.025 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:48.025 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:48:48.026 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:48:48.026 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:48:48.026 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:48:48.026 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:48:48.026 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:48:48.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.011+0000 7efed8c35640 10 monclient: _finish_auth 0 2024-09-16T03:48:48.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.011+0000 7efed8c35640 10 monclient: _check_auth_tickets 2024-09-16T03:48:48.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.011+0000 7efed8c35640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:48:48.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.011+0000 7efed8c35640 1 -- 172.21.15.42:0/1991997555 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efec40ab600 con 0x7efed406db90 2024-09-16T03:48:48.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.011+0000 7efedbe9b640 5 monclient: authenticate success, global_id 784148 2024-09-16T03:48:48.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.011+0000 7efed8c35640 10 monclient: handle_config config(0 keys) 2024-09-16T03:48:48.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.011+0000 7efed8c35640 1 -- 172.21.15.42:0/1991997555 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efec40ab9d0 con 0x7efed406db90 2024-09-16T03:48:48.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.011+0000 7efed8c35640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:48:48.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.011+0000 7efed8c35640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:48:48.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.011+0000 7efed8c35640 10 monclient: dump: 2024-09-16T03:48:48.027 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:48:48.027 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:48:48.027 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:48.027 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:48.027 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:48:48.027 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:48:48.027 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:48:48.027 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:48:48.027 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:48:48.027 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:48:48.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.011+0000 7efedbe9b640 10 monclient: get_monmap_and_config success 2024-09-16T03:48:48.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.011+0000 7efedbe9b640 4 set_mon_vals no callback set 2024-09-16T03:48:48.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.011+0000 7efedbe9b640 10 monclient: shutdown 2024-09-16T03:48:48.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.011+0000 7efedbe9b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:48:48.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.011+0000 7efedbe9b640 1 -- 172.21.15.42:0/1991997555 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7efed406db90 msgr2=0x7efed41340c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:48:48.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.011+0000 7efedbe9b640 1 --2- 172.21.15.42:0/1991997555 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7efed406db90 0x7efed41340c0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7efec40097c0 tx=0x7efec401c380 comp rx=0 tx=0).stop 2024-09-16T03:48:48.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.011+0000 7efedbe9b640 1 -- 172.21.15.42:0/1991997555 shutdown_connections 2024-09-16T03:48:48.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.011+0000 7efedbe9b640 1 --2- 172.21.15.42:0/1991997555 >> v2:172.21.15.154:3300/0 conn(0x7efed40a3bd0 0x7efed4137520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:48.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.011+0000 7efedbe9b640 1 --2- 172.21.15.42:0/1991997555 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7efed406db90 0x7efed41340c0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:48.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.011+0000 7efedbe9b640 1 -- 172.21.15.42:0/1991997555 >> 172.21.15.42:0/1991997555 conn(0x7efed40a9fb0 msgr2=0x7efed40aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:48:48.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.011+0000 7efedbe9b640 1 -- 172.21.15.42:0/1991997555 shutdown_connections 2024-09-16T03:48:48.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.011+0000 7efedbe9b640 1 -- 172.21.15.42:0/1991997555 wait complete. 2024-09-16T03:48:48.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.011+0000 7efedbe9b640 10 monclient: build_initial_monmap 2024-09-16T03:48:48.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.011+0000 7efedbe9b640 10 monclient: monmap: 2024-09-16T03:48:48.029 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:48:48.029 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:48:48.029 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:48:48.029 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:48:48.029 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:48:48.029 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:48:48.029 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:48:48.029 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:48:48.029 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:48:48.029 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:48:48.029 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.011+0000 7efedbe9b640 1 librados: starting msgr at 2024-09-16T03:48:48.029 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.011+0000 7efedbe9b640 1 librados: starting objecter 2024-09-16T03:48:48.029 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.011+0000 7efedbe9b640 1 Processor -- start 2024-09-16T03:48:48.030 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.011+0000 7efedbe9b640 1 -- start start 2024-09-16T03:48:48.030 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.011+0000 7efedbe9b640 1 librados: setting wanted keys 2024-09-16T03:48:48.030 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.011+0000 7efedbe9b640 1 librados: calling monclient init 2024-09-16T03:48:48.030 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.011+0000 7efedbe9b640 10 monclient: init 2024-09-16T03:48:48.030 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.015+0000 7efedbe9b640 10 monclient: _reopen_session rank -1 2024-09-16T03:48:48.030 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.015+0000 7efedbe9b640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:48:48.030 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.015+0000 7efedbe9b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efed406db90 0x7efed40a31d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:48.030 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.015+0000 7efedbe9b640 10 monclient(hunting): picked mon.noname-b con 0x7efed406db90 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:48:48.030 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.015+0000 7efedbe9b640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efed40a37f0 0x7efed413a440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:48.030 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.015+0000 7efedbe9b640 10 monclient(hunting): picked mon.noname-a con 0x7efed40a37f0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:48:48.030 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.015+0000 7efedbe9b640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7efed40a3bd0 0x7efed413a980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:48.030 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.015+0000 7efedbe9b640 10 monclient(hunting): picked mon.noname-c con 0x7efed40a3bd0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:48:48.030 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.015+0000 7efedbe9b640 10 monclient(hunting): start opening mon connection 2024-09-16T03:48:48.030 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.015+0000 7efedbe9b640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7efed41419d0 con 0x7efed40a37f0 2024-09-16T03:48:48.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.015+0000 7efedbe9b640 10 monclient(hunting): start opening mon connection 2024-09-16T03:48:48.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.015+0000 7efedbe9b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7efed40a4100 con 0x7efed406db90 2024-09-16T03:48:48.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.015+0000 7efedbe9b640 10 monclient(hunting): start opening mon connection 2024-09-16T03:48:48.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.015+0000 7efedbe9b640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7efed4141b50 con 0x7efed40a3bd0 2024-09-16T03:48:48.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.015+0000 7efedbe9b640 10 monclient(hunting): _renew_subs 2024-09-16T03:48:48.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.015+0000 7efedbe9b640 10 monclient(hunting): authenticate will time out at 1374.984245s 2024-09-16T03:48:48.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.015+0000 7efed9436640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efed40a37f0 0x7efed413a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:48:48.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.015+0000 7efed9c37640 1 -- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efed406db90 msgr2=0x7efed40a31d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T03:48:48.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.015+0000 7efeda438640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7efed40a3bd0 0x7efed413a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:48:48.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.015+0000 7efed9436640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efed40a37f0 0x7efed413a440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:55240/0 (socket says 172.21.15.42:55240) 2024-09-16T03:48:48.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.015+0000 7efed9436640 1 -- 172.21.15.42:0/2245913307 learned_addr learned my addr 172.21.15.42:0/2245913307 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:48:48.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.015+0000 7efed9436640 10 monclient(hunting): get_auth_request con 0x7efed40a37f0 auth_method 0 2024-09-16T03:48:48.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.015+0000 7efed9436640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:48:48.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.015+0000 7efed9c37640 1 --2- 172.21.15.42:0/2245913307 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efed406db90 0x7efed40a31d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:48:48.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.015+0000 7efed9436640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:48:48.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.015+0000 7efed9436640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:48:48.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.015+0000 7efeda438640 10 monclient(hunting): get_auth_request con 0x7efed40a3bd0 auth_method 0 2024-09-16T03:48:48.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.015+0000 7efeda438640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:48:48.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.015+0000 7efeda438640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:48:48.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.015+0000 7efeda438640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:48:48.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.015+0000 7efed9436640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:48:48.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.015+0000 7efed9436640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:48:48.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.015+0000 7efed9436640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:48:48.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.015+0000 7efeda438640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:48:48.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.015+0000 7efeda438640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:48:48.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.015+0000 7efeda438640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:48:48.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.015+0000 7efeda438640 10 monclient(hunting): handle_auth_done global_id 784151 payload 824 2024-09-16T03:48:48.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.015+0000 7efeda438640 1 -- 172.21.15.42:0/2245913307 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efed406db90 msgr2=0x7efed40a31d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:48:48.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.015+0000 7efeda438640 1 --2- 172.21.15.42:0/2245913307 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efed406db90 0x7efed40a31d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:48.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.015+0000 7efeda438640 1 -- 172.21.15.42:0/2245913307 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efed40a37f0 msgr2=0x7efed413a440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:48:48.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.015+0000 7efeda438640 1 --2- 172.21.15.42:0/2245913307 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efed40a37f0 0x7efed413a440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:48.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.015+0000 7efeda438640 10 monclient: _finish_hunting 0 2024-09-16T03:48:48.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.015+0000 7efeda438640 1 monclient: found mon.noname-c 2024-09-16T03:48:48.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.015+0000 7efeda438640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:48:48.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.015+0000 7efeda438640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:48:48.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.015+0000 7efeda438640 1 -- 172.21.15.42:0/2245913307 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efed40a9840 con 0x7efed40a3bd0 2024-09-16T03:48:48.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.015+0000 7efed9436640 1 --2- 172.21.15.42:0/2245913307 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efed40a37f0 0x7efed413a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:48:48.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.015+0000 7efeda438640 1 --2- 172.21.15.42:0/2245913307 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7efed40a3bd0 0x7efed413a980 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7efecc00ba60 tx=0x7efecc00bf20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:48:48.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.015+0000 7efeca7fc640 10 client.?.objecter ms_handle_connect 0x7efed40a3bd0 2024-09-16T03:48:48.034 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.015+0000 7efeca7fc640 10 client.?.objecter resend_mon_ops 2024-09-16T03:48:48.034 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.015+0000 7efeca7fc640 1 -- 172.21.15.42:0/2245913307 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efecc00cbe0 con 0x7efed40a3bd0 2024-09-16T03:48:48.034 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.015+0000 7efeca7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:48:48.034 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.015+0000 7efeca7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:48:48.034 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.015+0000 7efeca7fc640 10 monclient: dump: 2024-09-16T03:48:48.034 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:48:48.034 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:48:48.034 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:48.034 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:48.034 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:48:48.034 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:48:48.034 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:48:48.034 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:48:48.035 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:48:48.035 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:48:48.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.015+0000 7efeca7fc640 10 monclient: _finish_auth 0 2024-09-16T03:48:48.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.015+0000 7efeca7fc640 10 monclient: _check_auth_tickets 2024-09-16T03:48:48.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.015+0000 7efeca7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:48:48.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.015+0000 7efeca7fc640 1 -- 172.21.15.42:0/2245913307 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efecc010070 con 0x7efed40a3bd0 2024-09-16T03:48:48.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.015+0000 7efedbe9b640 5 monclient: authenticate success, global_id 784151 2024-09-16T03:48:48.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.015+0000 7efedbe9b640 10 monclient: _renew_subs 2024-09-16T03:48:48.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.015+0000 7efedbe9b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:48:48.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.015+0000 7efedbe9b640 1 -- 172.21.15.42:0/2245913307 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efed4131a90 con 0x7efed40a3bd0 2024-09-16T03:48:48.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.015+0000 7efeca7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:48:48.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.015+0000 7efeca7fc640 1 -- 172.21.15.42:0/2245913307 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efecc015470 con 0x7efed40a3bd0 2024-09-16T03:48:48.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.015+0000 7efeca7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:48:48.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.015+0000 7efeca7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:48:48.036 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.015+0000 7efeca7fc640 10 monclient: dump: 2024-09-16T03:48:48.036 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:48:48.036 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:48:48.036 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:48.036 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:48.036 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:48:48.036 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:48:48.036 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:48:48.036 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:48:48.036 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:48:48.036 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:48:48.036 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.015+0000 7efecb7fe640 4 set_mon_vals no callback set 2024-09-16T03:48:48.036 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.015+0000 7efedbe9b640 10 client.784151.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:48:48.036 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.015+0000 7efedbe9b640 10 monclient: _renew_subs 2024-09-16T03:48:48.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.015+0000 7efedbe9b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:48:48.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.015+0000 7efedbe9b640 1 -- 172.21.15.42:0/2245913307 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7efed413dc00 con 0x7efed40a3bd0 2024-09-16T03:48:48.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.015+0000 7efedbe9b640 1 librados: init done 2024-09-16T03:48:48.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.015+0000 7efeca7fc640 1 -- 172.21.15.42:0/2245913307 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7efecc015610 con 0x7efed40a3bd0 2024-09-16T03:48:48.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.015+0000 7efedbe9b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:48:48.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.015+0000 7efedbe9b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:48:48.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.015+0000 7efedbe9b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:48:48.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.015+0000 7efedbe9b640 1 -- 172.21.15.42:0/2245913307 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efed41340c0 con 0x7efed40a3bd0 2024-09-16T03:48:48.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.015+0000 7efeca7fc640 1 --2- 172.21.15.42:0/2245913307 >> v2:172.21.15.154:6812/254845342 conn(0x7efeb80377c0 0x7efeb8039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:48.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.015+0000 7efeca7fc640 1 -- 172.21.15.42:0/2245913307 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(216..216 src has 1..216) ==== 6980+0+0 (secure 0 0 0) 0x7efecc049610 con 0x7efed40a3bd0 2024-09-16T03:48:48.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.015+0000 7efeca7fc640 10 client.784151.objecter ms_dispatch 0x7efed4000ba0 osd_map(216..216 src has 1..216) 2024-09-16T03:48:48.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.015+0000 7efeca7fc640 3 client.784151.objecter handle_osd_map got epochs [216,216] > 0 2024-09-16T03:48:48.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.015+0000 7efeca7fc640 3 client.784151.objecter handle_osd_map decoding full epoch 216 2024-09-16T03:48:48.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.015+0000 7efeca7fc640 20 client.784151.objecter dump_active .. 0 homeless 2024-09-16T03:48:48.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.015+0000 7efed9c37640 1 --2- 172.21.15.42:0/2245913307 >> v2:172.21.15.154:6812/254845342 conn(0x7efeb80377c0 0x7efeb8039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:48:48.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.015+0000 7efed9c37640 10 monclient: get_auth_request con 0x7efeb80377c0 auth_method 0 2024-09-16T03:48:48.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.015+0000 7efeca7fc640 10 client.784151.objecter ms_handle_connect 0x7efeb80377c0 2024-09-16T03:48:48.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.015+0000 7efed9c37640 1 --2- 172.21.15.42:0/2245913307 >> v2:172.21.15.154:6812/254845342 conn(0x7efeb80377c0 0x7efeb8039c80 secure :-1 s=READY pgs=3499 cs=0 l=1 rev1=1 crypto rx=0x7efec40ad000 tx=0x7efec4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:48:48.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.019+0000 7efeca7fc640 1 -- 172.21.15.42:0/2245913307 <== mon.2 v2:172.21.15.42:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7efecc015ae0 con 0x7efed40a3bd0 2024-09-16T03:48:48.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.019+0000 7efeca7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:48:48.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.019+0000 7efeca7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:48:48.250 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.247+0000 7efedbe9b640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:48:48.250 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.247+0000 7efedbe9b640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:48:48.250 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.247+0000 7efedbe9b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:48:48.250 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.247+0000 7efedbe9b640 1 -- 172.21.15.42:0/2245913307 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7efed4068e40 con 0x7efed40a3bd0 2024-09-16T03:48:48.251 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.247+0000 7efeca7fc640 1 -- 172.21.15.42:0/2245913307 <== mon.2 v2:172.21.15.42:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7efecc04b6e0 con 0x7efed40a3bd0 2024-09-16T03:48:48.251 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.247+0000 7efeca7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:48:48.251 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.247+0000 7efeca7fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:48:48.251 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":60,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":14,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:48:48.254 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.251+0000 7efedbe9b640 10 librados: watch_flush enter 2024-09-16T03:48:48.254 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.251+0000 7efedbe9b640 10 librados: watch_flush exit 2024-09-16T03:48:48.254 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.251+0000 7efedbe9b640 20 client.784151.objecter shutdown clearing up homeless session... 2024-09-16T03:48:48.254 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.251+0000 7efedbe9b640 10 client.784151.objecter successfully canceled tick 2024-09-16T03:48:48.254 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.251+0000 7efedbe9b640 1 -- 172.21.15.42:0/2245913307 >> v2:172.21.15.154:6812/254845342 conn(0x7efeb80377c0 msgr2=0x7efeb8039c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:48:48.254 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.251+0000 7efedbe9b640 1 --2- 172.21.15.42:0/2245913307 >> v2:172.21.15.154:6812/254845342 conn(0x7efeb80377c0 0x7efeb8039c80 secure :-1 s=READY pgs=3499 cs=0 l=1 rev1=1 crypto rx=0x7efec40ad000 tx=0x7efec4008040 comp rx=0 tx=0).stop 2024-09-16T03:48:48.255 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.251+0000 7efedbe9b640 10 monclient: shutdown 2024-09-16T03:48:48.255 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.251+0000 7efedbe9b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:48:48.255 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.251+0000 7efedbe9b640 1 -- 172.21.15.42:0/2245913307 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7efed40a3bd0 msgr2=0x7efed413a980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:48:48.255 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.251+0000 7efedbe9b640 1 --2- 172.21.15.42:0/2245913307 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7efed40a3bd0 0x7efed413a980 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7efecc00ba60 tx=0x7efecc00bf20 comp rx=0 tx=0).stop 2024-09-16T03:48:48.255 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.251+0000 7efedbe9b640 1 -- 172.21.15.42:0/2245913307 shutdown_connections 2024-09-16T03:48:48.255 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.251+0000 7efedbe9b640 1 --2- 172.21.15.42:0/2245913307 >> v2:172.21.15.154:6812/254845342 conn(0x7efeb80377c0 0x7efeb8039c80 unknown :-1 s=CLOSED pgs=3499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:48.255 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.251+0000 7efedbe9b640 1 --2- 172.21.15.42:0/2245913307 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7efed40a3bd0 0x7efed413a980 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:48.255 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.251+0000 7efedbe9b640 1 --2- 172.21.15.42:0/2245913307 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efed40a37f0 0x7efed413a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:48.255 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.251+0000 7efedbe9b640 1 --2- 172.21.15.42:0/2245913307 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efed406db90 0x7efed40a31d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:48.255 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.251+0000 7efedbe9b640 1 -- 172.21.15.42:0/2245913307 >> 172.21.15.42:0/2245913307 conn(0x7efed40a9fb0 msgr2=0x7efed40a7ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:48:48.255 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.251+0000 7efedbe9b640 1 -- 172.21.15.42:0/2245913307 shutdown_connections 2024-09-16T03:48:48.256 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.251+0000 7efedbe9b640 1 -- 172.21.15.42:0/2245913307 wait complete. 2024-09-16T03:48:48.256 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:48.251+0000 7efedbe9b640 1 librados: shutdown 2024-09-16T03:48:51.270 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:48:51.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:51.399+0000 7f5421e6d640 10 monclient: get_monmap_and_config 2024-09-16T03:48:51.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:51.399+0000 7f5421e6d640 10 monclient: build_initial_monmap 2024-09-16T03:48:51.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:51.399+0000 7f5421e6d640 10 monclient: monmap: 2024-09-16T03:48:51.403 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:48:51.403 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:48:51.403 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:48:51.401638+0000 2024-09-16T03:48:51.403 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:48:51.401638+0000 2024-09-16T03:48:51.403 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:48:51.403 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:48:51.404 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:48:51.404 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:48:51.404 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:48:51.404 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:48:51.404 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:48:51.404 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:48:51.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:51.399+0000 7f5421e6d640 1 Processor -- start 2024-09-16T03:48:51.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:51.399+0000 7f5421e6d640 1 -- start start 2024-09-16T03:48:51.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:51.399+0000 7f5421e6d640 10 monclient: init 2024-09-16T03:48:51.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:51.399+0000 7f5421e6d640 10 monclient: _reopen_session rank -1 2024-09-16T03:48:51.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:51.399+0000 7f5421e6d640 10 monclient: _add_conns ranks=[0,2,3,4,1] 2024-09-16T03:48:51.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:51.399+0000 7f5421e6d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f541c0a1330 addr v1:172.21.15.42:6789/0 2024-09-16T03:48:51.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:51.399+0000 7f5421e6d640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f541c06dac0 0x7f541c13a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:51.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:51.399+0000 7f5421e6d640 10 monclient(hunting): picked mon.noname-b con 0x7f541c06dac0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:48:51.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:51.399+0000 7f5421e6d640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f541c0a1e20 0x7f541c13c890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:51.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:51.399+0000 7f5421e6d640 10 monclient(hunting): picked mon.noname-a con 0x7f541c0a1e20 addr v2:172.21.15.42:3300/0 2024-09-16T03:48:51.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:51.399+0000 7f5421e6d640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f541c09c7d0 con 0x7f541c0a1330 2024-09-16T03:48:51.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:51.399+0000 7f5421e6d640 10 monclient(hunting): start opening mon connection 2024-09-16T03:48:51.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:51.399+0000 7f5421e6d640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f541c09c090 con 0x7f541c0a1e20 2024-09-16T03:48:51.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:51.399+0000 7f5421e6d640 10 monclient(hunting): start opening mon connection 2024-09-16T03:48:51.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:51.399+0000 7f5421e6d640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f541c09f610 con 0x7f541c06dac0 2024-09-16T03:48:51.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:51.399+0000 7f5421e6d640 10 monclient(hunting): _renew_subs 2024-09-16T03:48:51.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:51.399+0000 7f5421e6d640 10 monclient(hunting): authenticate will time out at 1378.371152s 2024-09-16T03:48:51.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:51.399+0000 7f541b7fe640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f541c0a1330 0x7f541c0a1710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:46178/0 (socket says 172.21.15.42:46178) 2024-09-16T03:48:51.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:51.399+0000 7f541b7fe640 1 -- 172.21.15.42:0/4115485458 learned_addr learned my addr 172.21.15.42:0/4115485458 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:48:51.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:51.399+0000 7f5412ffd640 1 --2- 172.21.15.42:0/4115485458 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f541c06dac0 0x7f541c13a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:48:51.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:51.399+0000 7f5412ffd640 10 monclient(hunting): get_auth_request con 0x7f541c06dac0 auth_method 0 2024-09-16T03:48:51.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:51.399+0000 7f5412ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:48:51.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:51.399+0000 7f5412ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:48:51.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:51.399+0000 7f5412ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:48:51.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:51.399+0000 7f541bfff640 1 --2- 172.21.15.42:0/4115485458 >> v2:172.21.15.42:3300/0 conn(0x7f541c0a1e20 0x7f541c13c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:48:51.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:51.399+0000 7f541bfff640 10 monclient(hunting): get_auth_request con 0x7f541c0a1e20 auth_method 0 2024-09-16T03:48:51.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:51.399+0000 7f541bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:48:51.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:51.399+0000 7f541bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:48:51.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:51.399+0000 7f541bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:48:51.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:51.399+0000 7f5412ffd640 1 -- 172.21.15.42:0/4115485458 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f541c06dac0 msgr2=0x7f541c13a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T03:48:51.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:51.399+0000 7f5412ffd640 1 -- 172.21.15.42:0/4115485458 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f541c06dac0 msgr2=0x7f541c13a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:48:51.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:51.399+0000 7f5412ffd640 1 --2- 172.21.15.42:0/4115485458 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f541c06dac0 0x7f541c13a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:48:51.411 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:51.399+0000 7f541bfff640 1 -- 172.21.15.42:0/4115485458 >> v2:172.21.15.42:3300/0 conn(0x7f541c0a1e20 msgr2=0x7f541c13c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T03:48:51.411 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:51.399+0000 7f541bfff640 1 -- 172.21.15.42:0/4115485458 >> v2:172.21.15.42:3300/0 conn(0x7f541c0a1e20 msgr2=0x7f541c13c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:48:51.411 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:51.399+0000 7f5412ffd640 1 --2- 172.21.15.42:0/4115485458 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f541c06dac0 0x7f541c13a340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:48:51.411 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:51.399+0000 7f541bfff640 1 --2- 172.21.15.42:0/4115485458 >> v2:172.21.15.42:3300/0 conn(0x7f541c0a1e20 0x7f541c13c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:48:51.411 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:51.399+0000 7f541bfff640 1 --2- 172.21.15.42:0/4115485458 >> v2:172.21.15.42:3300/0 conn(0x7f541c0a1e20 0x7f541c13c890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:48:51.606 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:51.603+0000 7f5412ffd640 1 --2- 172.21.15.42:0/4115485458 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f541c06dac0 0x7f541c13a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:48:51.606 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:51.603+0000 7f541bfff640 1 --2- 172.21.15.42:0/4115485458 >> v2:172.21.15.42:3300/0 conn(0x7f541c0a1e20 0x7f541c13c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:48:51.606 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:51.603+0000 7f541bfff640 10 monclient(hunting): get_auth_request con 0x7f541c0a1e20 auth_method 0 2024-09-16T03:48:51.606 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:51.603+0000 7f541bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:48:51.606 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:51.603+0000 7f541bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:48:51.606 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:51.603+0000 7f541bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:48:51.606 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:51.603+0000 7f5412ffd640 10 monclient(hunting): get_auth_request con 0x7f541c06dac0 auth_method 0 2024-09-16T03:48:51.606 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:51.603+0000 7f5412ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:48:51.606 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:51.603+0000 7f5412ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:48:51.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:51.603+0000 7f5412ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:48:51.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:51.603+0000 7f5412ffd640 1 -- 172.21.15.42:0/4115485458 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f541c06dac0 msgr2=0x7f541c13a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T03:48:51.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:51.603+0000 7f541bfff640 1 -- 172.21.15.42:0/4115485458 >> v2:172.21.15.42:3300/0 conn(0x7f541c0a1e20 msgr2=0x7f541c13c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T03:48:51.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:51.603+0000 7f5412ffd640 1 -- 172.21.15.42:0/4115485458 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f541c06dac0 msgr2=0x7f541c13a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:48:51.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:51.603+0000 7f541bfff640 1 -- 172.21.15.42:0/4115485458 >> v2:172.21.15.42:3300/0 conn(0x7f541c0a1e20 msgr2=0x7f541c13c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:48:51.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:51.603+0000 7f541bfff640 1 --2- 172.21.15.42:0/4115485458 >> v2:172.21.15.42:3300/0 conn(0x7f541c0a1e20 0x7f541c13c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:48:51.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:51.603+0000 7f5412ffd640 1 --2- 172.21.15.42:0/4115485458 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f541c06dac0 0x7f541c13a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:48:51.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:51.603+0000 7f5412ffd640 1 --2- 172.21.15.42:0/4115485458 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f541c06dac0 0x7f541c13a340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T03:48:51.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:51.603+0000 7f541bfff640 1 --2- 172.21.15.42:0/4115485458 >> v2:172.21.15.42:3300/0 conn(0x7f541c0a1e20 0x7f541c13c890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T03:48:52.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:52.007+0000 7f5412ffd640 1 --2- 172.21.15.42:0/4115485458 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f541c06dac0 0x7f541c13a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:48:52.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:52.007+0000 7f541bfff640 1 --2- 172.21.15.42:0/4115485458 >> v2:172.21.15.42:3300/0 conn(0x7f541c0a1e20 0x7f541c13c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:48:52.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:52.007+0000 7f541bfff640 10 monclient(hunting): get_auth_request con 0x7f541c0a1e20 auth_method 0 2024-09-16T03:48:52.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:52.007+0000 7f541bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:48:52.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:52.007+0000 7f541bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:48:52.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:52.007+0000 7f541bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:48:52.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:52.007+0000 7f5412ffd640 10 monclient(hunting): get_auth_request con 0x7f541c06dac0 auth_method 0 2024-09-16T03:48:52.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:52.007+0000 7f5412ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:48:52.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:52.007+0000 7f5412ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:48:52.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:52.007+0000 7f5412ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:48:52.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:52.007+0000 7f5412ffd640 1 -- 172.21.15.42:0/4115485458 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f541c06dac0 msgr2=0x7f541c13a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T03:48:52.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:52.007+0000 7f5412ffd640 1 -- 172.21.15.42:0/4115485458 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f541c06dac0 msgr2=0x7f541c13a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:48:52.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:52.007+0000 7f541bfff640 1 -- 172.21.15.42:0/4115485458 >> v2:172.21.15.42:3300/0 conn(0x7f541c0a1e20 msgr2=0x7f541c13c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T03:48:52.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:52.007+0000 7f5412ffd640 1 --2- 172.21.15.42:0/4115485458 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f541c06dac0 0x7f541c13a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:48:52.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:52.007+0000 7f541bfff640 1 -- 172.21.15.42:0/4115485458 >> v2:172.21.15.42:3300/0 conn(0x7f541c0a1e20 msgr2=0x7f541c13c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:48:52.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:52.007+0000 7f541bfff640 1 --2- 172.21.15.42:0/4115485458 >> v2:172.21.15.42:3300/0 conn(0x7f541c0a1e20 0x7f541c13c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:48:52.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:52.007+0000 7f541bfff640 1 --2- 172.21.15.42:0/4115485458 >> v2:172.21.15.42:3300/0 conn(0x7f541c0a1e20 0x7f541c13c890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T03:48:52.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:52.007+0000 7f5412ffd640 1 --2- 172.21.15.42:0/4115485458 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f541c06dac0 0x7f541c13a340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T03:48:52.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:52.811+0000 7f541bfff640 1 --2- 172.21.15.42:0/4115485458 >> v2:172.21.15.42:3300/0 conn(0x7f541c0a1e20 0x7f541c13c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:48:52.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:52.811+0000 7f5412ffd640 1 --2- 172.21.15.42:0/4115485458 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f541c06dac0 0x7f541c13a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:48:52.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:52.811+0000 7f541bfff640 10 monclient(hunting): get_auth_request con 0x7f541c0a1e20 auth_method 0 2024-09-16T03:48:52.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:52.811+0000 7f541bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:48:52.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:52.811+0000 7f541bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:48:52.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:52.811+0000 7f541bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:48:52.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:52.811+0000 7f5412ffd640 10 monclient(hunting): get_auth_request con 0x7f541c06dac0 auth_method 0 2024-09-16T03:48:52.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:52.811+0000 7f5412ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:48:52.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:52.811+0000 7f5412ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:48:52.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:52.811+0000 7f5412ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:48:52.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:52.811+0000 7f541bfff640 1 -- 172.21.15.42:0/4115485458 >> v2:172.21.15.42:3300/0 conn(0x7f541c0a1e20 msgr2=0x7f541c13c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T03:48:52.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:52.811+0000 7f541bfff640 1 -- 172.21.15.42:0/4115485458 >> v2:172.21.15.42:3300/0 conn(0x7f541c0a1e20 msgr2=0x7f541c13c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:48:52.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:52.811+0000 7f541bfff640 1 --2- 172.21.15.42:0/4115485458 >> v2:172.21.15.42:3300/0 conn(0x7f541c0a1e20 0x7f541c13c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:48:52.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:52.811+0000 7f5412ffd640 1 -- 172.21.15.42:0/4115485458 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f541c06dac0 msgr2=0x7f541c13a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T03:48:52.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:52.811+0000 7f5412ffd640 1 -- 172.21.15.42:0/4115485458 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f541c06dac0 msgr2=0x7f541c13a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:48:52.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:52.811+0000 7f5412ffd640 1 --2- 172.21.15.42:0/4115485458 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f541c06dac0 0x7f541c13a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:48:52.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:52.811+0000 7f541bfff640 1 --2- 172.21.15.42:0/4115485458 >> v2:172.21.15.42:3300/0 conn(0x7f541c0a1e20 0x7f541c13c890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T03:48:52.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:52.811+0000 7f5412ffd640 1 --2- 172.21.15.42:0/4115485458 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f541c06dac0 0x7f541c13a340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T03:48:54.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:54.399+0000 7f5419ffb640 10 monclient(hunting): tick 2024-09-16T03:48:54.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:54.399+0000 7f5419ffb640 10 monclient(hunting): _check_auth_tickets 2024-09-16T03:48:54.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:54.399+0000 7f5419ffb640 1 monclient(hunting): continuing hunt 2024-09-16T03:48:54.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:54.399+0000 7f5419ffb640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T03:48:54.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:54.399+0000 7f5419ffb640 1 -- 172.21.15.42:0/4115485458 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f541c06dac0 msgr2=0x7f541c13a340 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:48:54.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:54.399+0000 7f5419ffb640 1 --2- 172.21.15.42:0/4115485458 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f541c06dac0 0x7f541c13a340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:54.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:54.399+0000 7f5419ffb640 1 -- 172.21.15.42:0/4115485458 >> v2:172.21.15.42:3300/0 conn(0x7f541c0a1e20 msgr2=0x7f541c13c890 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:48:54.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:54.399+0000 7f5419ffb640 1 --2- 172.21.15.42:0/4115485458 >> v2:172.21.15.42:3300/0 conn(0x7f541c0a1e20 0x7f541c13c890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:54.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:54.399+0000 7f5419ffb640 1 -- 172.21.15.42:0/4115485458 >> v1:172.21.15.42:6789/0 conn(0x7f541c0a1330 legacy=0x7f541c0a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:48:54.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:54.399+0000 7f5419ffb640 10 monclient: _add_conns ranks=[4,1] 2024-09-16T03:48:54.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:54.399+0000 7f5419ffb640 1 --2- 172.21.15.42:0/4115485458 >> v2:172.21.15.154:3300/0 conn(0x7f53fc0036d0 0x7f53fc005b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:54.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:54.399+0000 7f5419ffb640 10 monclient(hunting): picked mon.noname-c con 0x7f53fc0036d0 addr v2:172.21.15.154:3300/0 2024-09-16T03:48:54.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:54.399+0000 7f5419ffb640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f53fc006180 addr v1:172.21.15.154:6789/0 2024-09-16T03:48:54.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:54.399+0000 7f5419ffb640 1 -- 172.21.15.42:0/4115485458 --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f541c09f8f0 con 0x7f53fc006180 2024-09-16T03:48:54.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:54.399+0000 7f5419ffb640 10 monclient(hunting): start opening mon connection 2024-09-16T03:48:54.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:54.399+0000 7f5419ffb640 1 -- 172.21.15.42:0/4115485458 --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f541c09c090 con 0x7f53fc0036d0 2024-09-16T03:48:54.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:54.399+0000 7f5419ffb640 10 monclient(hunting): _renew_subs 2024-09-16T03:48:54.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:54.399+0000 7f541b7fe640 1 -- 172.21.15.42:0/4115485458 >> v2:172.21.15.154:3300/0 conn(0x7f53fc0036d0 msgr2=0x7f53fc005b30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T03:48:54.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:54.399+0000 7f541b7fe640 1 --2- 172.21.15.42:0/4115485458 >> v2:172.21.15.154:3300/0 conn(0x7f53fc0036d0 0x7f53fc005b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:48:54.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:54.399+0000 7f5412ffd640 1 -- 172.21.15.42:0/4115485458 >> v1:172.21.15.154:6789/0 conn(0x7f53fc006180 legacy=0x7f53fc008620 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.154:6789/0 2024-09-16T03:48:54.605 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:54.603+0000 7f541b7fe640 1 -- 172.21.15.42:0/4115485458 >> v2:172.21.15.154:3300/0 conn(0x7f53fc0036d0 msgr2=0x7f53fc005b30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T03:48:54.605 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:54.603+0000 7f541b7fe640 1 --2- 172.21.15.42:0/4115485458 >> v2:172.21.15.154:3300/0 conn(0x7f53fc0036d0 0x7f53fc005b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T03:48:54.605 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:54.603+0000 7f5412ffd640 1 -- 172.21.15.42:0/4115485458 >> v1:172.21.15.154:6789/0 conn(0x7f53fc006180 legacy=0x7f53fc008620 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.154:6789/0 2024-09-16T03:48:55.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:55.007+0000 7f541b7fe640 1 -- 172.21.15.42:0/4115485458 >> v2:172.21.15.154:3300/0 conn(0x7f53fc0036d0 msgr2=0x7f53fc005b30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T03:48:55.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:55.007+0000 7f5412ffd640 1 -- 172.21.15.42:0/4115485458 >> v1:172.21.15.154:6789/0 conn(0x7f53fc006180 legacy=0x7f53fc008620 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.154:6789/0 2024-09-16T03:48:55.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:55.007+0000 7f541b7fe640 1 --2- 172.21.15.42:0/4115485458 >> v2:172.21.15.154:3300/0 conn(0x7f53fc0036d0 0x7f53fc005b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T03:48:55.813 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:55.811+0000 7f541b7fe640 1 -- 172.21.15.42:0/4115485458 >> v2:172.21.15.154:3300/0 conn(0x7f53fc0036d0 msgr2=0x7f53fc005b30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T03:48:55.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:55.811+0000 7f5412ffd640 1 -- 172.21.15.42:0/4115485458 >> v1:172.21.15.154:6789/0 conn(0x7f53fc006180 legacy=0x7f53fc008620 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.154:6789/0 2024-09-16T03:48:55.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:55.811+0000 7f541b7fe640 1 --2- 172.21.15.42:0/4115485458 >> v2:172.21.15.154:3300/0 conn(0x7f53fc0036d0 0x7f53fc005b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T03:48:57.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.399+0000 7f5419ffb640 10 monclient(hunting): tick 2024-09-16T03:48:57.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.399+0000 7f5419ffb640 10 monclient(hunting): _check_auth_tickets 2024-09-16T03:48:57.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.399+0000 7f5419ffb640 1 monclient(hunting): continuing hunt 2024-09-16T03:48:57.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.399+0000 7f5419ffb640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T03:48:57.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.399+0000 7f5419ffb640 1 -- 172.21.15.42:0/4115485458 >> v2:172.21.15.154:3300/0 conn(0x7f53fc0036d0 msgr2=0x7f53fc005b30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:48:57.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.399+0000 7f5419ffb640 1 --2- 172.21.15.42:0/4115485458 >> v2:172.21.15.154:3300/0 conn(0x7f53fc0036d0 0x7f53fc005b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:57.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.399+0000 7f5419ffb640 1 -- 172.21.15.42:0/4115485458 >> v1:172.21.15.154:6789/0 conn(0x7f53fc006180 legacy=0x7f53fc008620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:48:57.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.399+0000 7f5419ffb640 10 monclient: _add_conns ranks=[0,4,3,2,1] 2024-09-16T03:48:57.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.399+0000 7f541bfff640 1 -- 172.21.15.42:0/4115485458 reap_dead start 2024-09-16T03:48:57.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.399+0000 7f5419ffb640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f541c0a1330 addr v1:172.21.15.42:6789/0 2024-09-16T03:48:57.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.399+0000 7f5419ffb640 1 --2- 172.21.15.42:0/4115485458 >> v2:172.21.15.154:3300/0 conn(0x7f53fc00a5b0 0x7f53fc005b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:57.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.399+0000 7f5419ffb640 10 monclient(hunting): picked mon.noname-c con 0x7f53fc00a5b0 addr v2:172.21.15.154:3300/0 2024-09-16T03:48:57.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.399+0000 7f5419ffb640 1 --2- 172.21.15.42:0/4115485458 >> v2:172.21.15.42:3300/0 conn(0x7f53fc008d50 0x7f53fc008570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:57.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.399+0000 7f5419ffb640 10 monclient(hunting): picked mon.noname-a con 0x7f53fc008d50 addr v2:172.21.15.42:3300/0 2024-09-16T03:48:57.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.399+0000 7f5419ffb640 1 -- 172.21.15.42:0/4115485458 --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53fc00af60 con 0x7f541c0a1330 2024-09-16T03:48:57.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.399+0000 7f5419ffb640 10 monclient(hunting): start opening mon connection 2024-09-16T03:48:57.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.399+0000 7f5419ffb640 1 -- 172.21.15.42:0/4115485458 --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f541c09c090 con 0x7f53fc008d50 2024-09-16T03:48:57.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.399+0000 7f5419ffb640 10 monclient(hunting): start opening mon connection 2024-09-16T03:48:57.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.399+0000 7f5419ffb640 1 -- 172.21.15.42:0/4115485458 --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f541c09f610 con 0x7f53fc00a5b0 2024-09-16T03:48:57.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.399+0000 7f5419ffb640 10 monclient(hunting): _renew_subs 2024-09-16T03:48:57.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.403+0000 7f5412ffd640 1 -- 172.21.15.42:0/4115485458 >> v2:172.21.15.154:3300/0 conn(0x7f53fc00a5b0 msgr2=0x7f53fc005b30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T03:48:57.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.403+0000 7f5412ffd640 1 --2- 172.21.15.42:0/4115485458 >> v2:172.21.15.154:3300/0 conn(0x7f53fc00a5b0 0x7f53fc005b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:48:57.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.403+0000 7f541bfff640 1 --2- 172.21.15.42:0/4115485458 >> v2:172.21.15.42:3300/0 conn(0x7f53fc008d50 0x7f53fc008570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:48:57.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.403+0000 7f541bfff640 10 monclient(hunting): get_auth_request con 0x7f53fc008d50 auth_method 0 2024-09-16T03:48:57.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.403+0000 7f541bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:48:57.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.403+0000 7f541bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:48:57.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.403+0000 7f541bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:48:57.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.403+0000 7f541bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:48:57.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.403+0000 7f541bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:48:57.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.403+0000 7f541bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:48:57.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.403+0000 7f541bfff640 10 monclient(hunting): handle_auth_done global_id 764556 payload 274 2024-09-16T03:48:57.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.403+0000 7f541bfff640 1 -- 172.21.15.42:0/4115485458 >> v2:172.21.15.154:3300/0 conn(0x7f53fc00a5b0 msgr2=0x7f53fc005b30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:48:57.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.403+0000 7f541bfff640 1 --2- 172.21.15.42:0/4115485458 >> v2:172.21.15.154:3300/0 conn(0x7f53fc00a5b0 0x7f53fc005b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:57.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.403+0000 7f541bfff640 1 -- 172.21.15.42:0/4115485458 >> v1:172.21.15.42:6789/0 conn(0x7f541c0a1330 legacy=0x7f53fc009ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:48:57.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.403+0000 7f541bfff640 10 monclient: _finish_hunting 0 2024-09-16T03:48:57.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.403+0000 7f541bfff640 1 monclient: found mon.noname-a 2024-09-16T03:48:57.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.403+0000 7f541bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:48:57.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.403+0000 7f541bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:48:57.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.403+0000 7f541bfff640 1 -- 172.21.15.42:0/4115485458 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f541c09f8f0 con 0x7f53fc008d50 2024-09-16T03:48:57.411 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.407+0000 7f541bfff640 1 --2- 172.21.15.42:0/4115485458 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f53fc008d50 0x7f53fc008570 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f541c06da10 tx=0x7f541401d1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c0aeedd73662c1 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:48:57.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.407+0000 7f541affd640 1 -- 172.21.15.42:0/4115485458 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54140b3070 con 0x7f53fc008d50 2024-09-16T03:48:57.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.407+0000 7f541affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:48:57.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.407+0000 7f541affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:48:57.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.407+0000 7f541affd640 10 monclient: dump: 2024-09-16T03:48:57.413 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:48:57.413 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:48:57.413 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:57.413 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:57.413 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:48:57.413 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:48:57.413 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:48:57.413 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:48:57.413 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:48:57.413 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:48:57.413 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.407+0000 7f541affd640 10 monclient: _finish_auth 0 2024-09-16T03:48:57.413 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.407+0000 7f541affd640 10 monclient: _check_auth_tickets 2024-09-16T03:48:57.413 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.407+0000 7f541affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:48:57.414 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.407+0000 7f541affd640 1 -- 172.21.15.42:0/4115485458 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f54140ae410 con 0x7f53fc008d50 2024-09-16T03:48:57.414 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.407+0000 7f541affd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:48:57.414 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.407+0000 7f541affd640 1 -- 172.21.15.42:0/4115485458 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54140ae5e0 con 0x7f53fc008d50 2024-09-16T03:48:57.414 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.407+0000 7f5421e6d640 5 monclient: authenticate success, global_id 764556 2024-09-16T03:48:57.414 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.407+0000 7f5421e6d640 10 monclient: get_monmap_and_config success 2024-09-16T03:48:57.414 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.407+0000 7f5421e6d640 4 set_mon_vals no callback set 2024-09-16T03:48:57.414 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.407+0000 7f541affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:48:57.414 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.407+0000 7f541affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:48:57.414 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.407+0000 7f541affd640 10 monclient: dump: 2024-09-16T03:48:57.414 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:48:57.414 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:48:57.414 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:57.414 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:57.415 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:48:57.415 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:48:57.415 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:48:57.415 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:48:57.415 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:48:57.415 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:48:57.420 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.411+0000 7f5421e6d640 10 monclient: shutdown 2024-09-16T03:48:57.420 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.411+0000 7f5421e6d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:48:57.420 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.411+0000 7f5421e6d640 1 -- 172.21.15.42:0/4115485458 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f53fc008d50 msgr2=0x7f53fc008570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:48:57.421 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.411+0000 7f5421e6d640 1 --2- 172.21.15.42:0/4115485458 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f53fc008d50 0x7f53fc008570 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f541c06da10 tx=0x7f541401d1e0 comp rx=0 tx=0).stop 2024-09-16T03:48:57.421 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.411+0000 7f5421e6d640 1 -- 172.21.15.42:0/4115485458 shutdown_connections 2024-09-16T03:48:57.421 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.411+0000 7f5421e6d640 1 --2- 172.21.15.42:0/4115485458 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f53fc008d50 0x7f53fc008570 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:57.421 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.411+0000 7f5421e6d640 1 --2- 172.21.15.42:0/4115485458 >> v2:172.21.15.154:3300/0 conn(0x7f53fc00a5b0 0x7f53fc005b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:57.421 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.411+0000 7f5421e6d640 1 -- 172.21.15.42:0/4115485458 >> 172.21.15.42:0/4115485458 conn(0x7f541c0a2420 msgr2=0x7f541c0a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:48:57.421 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.411+0000 7f5421e6d640 1 -- 172.21.15.42:0/4115485458 shutdown_connections 2024-09-16T03:48:57.421 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.411+0000 7f5421e6d640 1 -- 172.21.15.42:0/4115485458 wait complete. 2024-09-16T03:48:57.421 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.411+0000 7f5421e6d640 10 monclient: build_initial_monmap 2024-09-16T03:48:57.421 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.411+0000 7f5421e6d640 10 monclient: monmap: 2024-09-16T03:48:57.421 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:48:57.421 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:48:57.421 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:48:57.421 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:48:57.421 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:48:57.421 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:48:57.422 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:48:57.422 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:48:57.422 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:48:57.422 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:48:57.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.411+0000 7f5421e6d640 1 librados: starting msgr at 2024-09-16T03:48:57.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.411+0000 7f5421e6d640 1 librados: starting objecter 2024-09-16T03:48:57.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.411+0000 7f5421e6d640 1 Processor -- start 2024-09-16T03:48:57.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.411+0000 7f5421e6d640 1 -- start start 2024-09-16T03:48:57.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.411+0000 7f5421e6d640 1 librados: setting wanted keys 2024-09-16T03:48:57.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.411+0000 7f5421e6d640 1 librados: calling monclient init 2024-09-16T03:48:57.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.411+0000 7f5421e6d640 10 monclient: init 2024-09-16T03:48:57.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.411+0000 7f5421e6d640 10 monclient: _reopen_session rank -1 2024-09-16T03:48:57.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.411+0000 7f5421e6d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:48:57.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.411+0000 7f5421e6d640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f53fc008d50 0x7f541c1ca3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:57.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.411+0000 7f5421e6d640 10 monclient(hunting): picked mon.noname-c con 0x7f53fc008d50 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:48:57.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.411+0000 7f5421e6d640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f53fc00a5b0 0x7f541c1ca930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:57.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.411+0000 7f5421e6d640 10 monclient(hunting): picked mon.noname-a con 0x7f53fc00a5b0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:48:57.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.411+0000 7f5421e6d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f541c0a1330 0x7f541c122910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:57.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.411+0000 7f5421e6d640 10 monclient(hunting): picked mon.noname-b con 0x7f541c0a1330 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:48:57.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.411+0000 7f5421e6d640 10 monclient(hunting): start opening mon connection 2024-09-16T03:48:57.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.411+0000 7f5421e6d640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f541c0a32f0 con 0x7f53fc00a5b0 2024-09-16T03:48:57.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.411+0000 7f5421e6d640 10 monclient(hunting): start opening mon connection 2024-09-16T03:48:57.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.411+0000 7f5421e6d640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f541c0a3170 con 0x7f541c0a1330 2024-09-16T03:48:57.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.411+0000 7f5421e6d640 10 monclient(hunting): start opening mon connection 2024-09-16T03:48:57.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.411+0000 7f5421e6d640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f541c09ed70 con 0x7f53fc008d50 2024-09-16T03:48:57.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.411+0000 7f5421e6d640 10 monclient(hunting): _renew_subs 2024-09-16T03:48:57.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.411+0000 7f5421e6d640 10 monclient(hunting): authenticate will time out at 1384.383674s 2024-09-16T03:48:57.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.411+0000 7f541bfff640 1 -- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f541c0a1330 msgr2=0x7f541c122910 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T03:48:57.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.411+0000 7f5412ffd640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f53fc00a5b0 0x7f541c1ca930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:48:57.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.411+0000 7f541bfff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f541c0a1330 0x7f541c122910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:48:57.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.411+0000 7f5412ffd640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f53fc00a5b0 0x7f541c1ca930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:50590/0 (socket says 172.21.15.42:50590) 2024-09-16T03:48:57.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.411+0000 7f5412ffd640 1 -- 172.21.15.42:0/363795215 learned_addr learned my addr 172.21.15.42:0/363795215 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:48:57.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.411+0000 7f541b7fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f53fc008d50 0x7f541c1ca3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:48:57.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.415+0000 7f5412ffd640 10 monclient(hunting): get_auth_request con 0x7f53fc00a5b0 auth_method 0 2024-09-16T03:48:57.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.415+0000 7f5412ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:48:57.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.415+0000 7f5412ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:48:57.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.415+0000 7f5412ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:48:57.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.415+0000 7f541b7fe640 10 monclient(hunting): get_auth_request con 0x7f53fc008d50 auth_method 0 2024-09-16T03:48:57.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.415+0000 7f541b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:48:57.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.415+0000 7f541b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:48:57.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.415+0000 7f541b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:48:57.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.415+0000 7f5412ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:48:57.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.415+0000 7f5412ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:48:57.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.415+0000 7f5412ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:48:57.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.415+0000 7f541b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:48:57.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.415+0000 7f541b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:48:57.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.415+0000 7f541b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:48:57.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.415+0000 7f5412ffd640 10 monclient(hunting): handle_auth_done global_id 764562 payload 824 2024-09-16T03:48:57.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.415+0000 7f5412ffd640 1 -- 172.21.15.42:0/363795215 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f53fc008d50 msgr2=0x7f541c1ca3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:48:57.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.415+0000 7f5412ffd640 1 --2- 172.21.15.42:0/363795215 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f53fc008d50 0x7f541c1ca3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:57.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.415+0000 7f5412ffd640 1 -- 172.21.15.42:0/363795215 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f541c0a1330 msgr2=0x7f541c122910 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:48:57.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.415+0000 7f5412ffd640 1 --2- 172.21.15.42:0/363795215 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f541c0a1330 0x7f541c122910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:57.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.415+0000 7f5412ffd640 10 monclient: _finish_hunting 0 2024-09-16T03:48:57.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.415+0000 7f5412ffd640 1 monclient: found mon.noname-a 2024-09-16T03:48:57.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.415+0000 7f5412ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:48:57.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.415+0000 7f5412ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:48:57.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.415+0000 7f5412ffd640 1 -- 172.21.15.42:0/363795215 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f541c1271e0 con 0x7f53fc00a5b0 2024-09-16T03:48:57.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.415+0000 7f5412ffd640 1 --2- 172.21.15.42:0/363795215 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f53fc00a5b0 0x7f541c1ca930 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f5404011fd0 tx=0x7f5404013e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:48:57.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.415+0000 7f5418ff9640 10 client.?.objecter ms_handle_connect 0x7f53fc00a5b0 2024-09-16T03:48:57.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.415+0000 7f5418ff9640 10 client.?.objecter resend_mon_ops 2024-09-16T03:48:57.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.415+0000 7f5418ff9640 1 -- 172.21.15.42:0/363795215 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f540401a070 con 0x7f53fc00a5b0 2024-09-16T03:48:57.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.415+0000 7f5418ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:48:57.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.415+0000 7f5418ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:48:57.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.415+0000 7f5418ff9640 10 monclient: dump: 2024-09-16T03:48:57.427 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:48:57.427 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:48:57.427 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:57.427 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:57.427 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:48:57.427 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:48:57.427 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:48:57.427 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:48:57.427 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:48:57.427 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:48:57.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.415+0000 7f5418ff9640 10 monclient: _finish_auth 0 2024-09-16T03:48:57.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.415+0000 7f5418ff9640 10 monclient: _check_auth_tickets 2024-09-16T03:48:57.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.415+0000 7f5418ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:48:57.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.415+0000 7f5421e6d640 5 monclient: authenticate success, global_id 764562 2024-09-16T03:48:57.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.415+0000 7f5418ff9640 1 -- 172.21.15.42:0/363795215 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f540400eb10 con 0x7f53fc00a5b0 2024-09-16T03:48:57.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.415+0000 7f5421e6d640 10 monclient: _renew_subs 2024-09-16T03:48:57.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.415+0000 7f5421e6d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:48:57.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.415+0000 7f5421e6d640 1 -- 172.21.15.42:0/363795215 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f541c126520 con 0x7f53fc00a5b0 2024-09-16T03:48:57.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.415+0000 7f5418ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T03:48:57.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.415+0000 7f5418ff9640 1 -- 172.21.15.42:0/363795215 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f540400f040 con 0x7f53fc00a5b0 2024-09-16T03:48:57.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.415+0000 7f5418ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:48:57.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.415+0000 7f5418ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:48:57.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.415+0000 7f5418ff9640 10 monclient: dump: 2024-09-16T03:48:57.428 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:48:57.428 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:48:57.429 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:57.429 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:57.432 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:48:57.432 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:48:57.432 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:48:57.432 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:48:57.432 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:48:57.432 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:48:57.432 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.415+0000 7f541a7fc640 4 set_mon_vals no callback set 2024-09-16T03:48:57.432 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.415+0000 7f5421e6d640 10 client.764562.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:48:57.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.415+0000 7f5421e6d640 10 monclient: _renew_subs 2024-09-16T03:48:57.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.415+0000 7f5421e6d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:48:57.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.415+0000 7f5421e6d640 1 -- 172.21.15.42:0/363795215 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f541c126800 con 0x7f53fc00a5b0 2024-09-16T03:48:57.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.415+0000 7f5421e6d640 1 librados: init done 2024-09-16T03:48:57.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.415+0000 7f5418ff9640 1 -- 172.21.15.42:0/363795215 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f540401d070 con 0x7f53fc00a5b0 2024-09-16T03:48:57.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.415+0000 7f5421e6d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:48:57.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.415+0000 7f5421e6d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:48:57.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.415+0000 7f5421e6d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:48:57.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.415+0000 7f5421e6d640 1 -- 172.21.15.42:0/363795215 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f541c071130 con 0x7f53fc00a5b0 2024-09-16T03:48:57.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.415+0000 7f5418ff9640 1 --2- 172.21.15.42:0/363795215 >> v2:172.21.15.154:6812/254845342 conn(0x7f53fc03c430 0x7f53fc03e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:57.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.415+0000 7f5418ff9640 1 -- 172.21.15.42:0/363795215 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(218..218 src has 1..218) ==== 6981+0+0 (secure 0 0 0) 0x7f5404049930 con 0x7f53fc00a5b0 2024-09-16T03:48:57.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.415+0000 7f5418ff9640 10 client.764562.objecter ms_dispatch 0x7f541c000ba0 osd_map(218..218 src has 1..218) 2024-09-16T03:48:57.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.415+0000 7f5418ff9640 3 client.764562.objecter handle_osd_map got epochs [218,218] > 0 2024-09-16T03:48:57.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.415+0000 7f5418ff9640 3 client.764562.objecter handle_osd_map decoding full epoch 218 2024-09-16T03:48:57.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.415+0000 7f5418ff9640 20 client.764562.objecter dump_active .. 0 homeless 2024-09-16T03:48:57.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.415+0000 7f541b7fe640 1 --2- 172.21.15.42:0/363795215 >> v2:172.21.15.154:6812/254845342 conn(0x7f53fc03c430 0x7f53fc03e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:48:57.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.415+0000 7f541b7fe640 10 monclient: get_auth_request con 0x7f53fc03c430 auth_method 0 2024-09-16T03:48:57.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.415+0000 7f5418ff9640 10 client.764562.objecter ms_handle_connect 0x7f53fc03c430 2024-09-16T03:48:57.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.415+0000 7f541b7fe640 1 --2- 172.21.15.42:0/363795215 >> v2:172.21.15.154:6812/254845342 conn(0x7f53fc03c430 0x7f53fc03e8f0 secure :-1 s=READY pgs=3500 cs=0 l=1 rev1=1 crypto rx=0x7f540c000f70 tx=0x7f540c007c20 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:48:57.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.423+0000 7f5418ff9640 1 -- 172.21.15.42:0/363795215 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f541c1271e0 con 0x7f53fc00a5b0 2024-09-16T03:48:57.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.423+0000 7f5418ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:48:57.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.423+0000 7f5418ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:48:57.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.671+0000 7f5421e6d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:48:57.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.671+0000 7f5421e6d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:48:57.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.671+0000 7f5421e6d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:48:57.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.671+0000 7f5421e6d640 1 -- 172.21.15.42:0/363795215 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f541c06e560 con 0x7f53fc00a5b0 2024-09-16T03:48:57.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.675+0000 7f5418ff9640 1 -- 172.21.15.42:0/363795215 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1554 (secure 0 0 0) 0x7f5404014350 con 0x7f53fc00a5b0 2024-09-16T03:48:57.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.675+0000 7f5418ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:48:57.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.675+0000 7f5418ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:48:57.678 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":64,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:48:57.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.675+0000 7f5421e6d640 10 librados: watch_flush enter 2024-09-16T03:48:57.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.675+0000 7f5421e6d640 10 librados: watch_flush exit 2024-09-16T03:48:57.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.675+0000 7f5421e6d640 20 client.764562.objecter shutdown clearing up homeless session... 2024-09-16T03:48:57.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.675+0000 7f5421e6d640 10 client.764562.objecter successfully canceled tick 2024-09-16T03:48:57.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.675+0000 7f5421e6d640 1 -- 172.21.15.42:0/363795215 >> v2:172.21.15.154:6812/254845342 conn(0x7f53fc03c430 msgr2=0x7f53fc03e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:48:57.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.675+0000 7f5421e6d640 1 --2- 172.21.15.42:0/363795215 >> v2:172.21.15.154:6812/254845342 conn(0x7f53fc03c430 0x7f53fc03e8f0 secure :-1 s=READY pgs=3500 cs=0 l=1 rev1=1 crypto rx=0x7f540c000f70 tx=0x7f540c007c20 comp rx=0 tx=0).stop 2024-09-16T03:48:57.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.675+0000 7f5421e6d640 10 monclient: shutdown 2024-09-16T03:48:57.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.675+0000 7f5421e6d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:48:57.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.675+0000 7f5421e6d640 1 -- 172.21.15.42:0/363795215 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f53fc00a5b0 msgr2=0x7f541c1ca930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:48:57.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.675+0000 7f5421e6d640 1 --2- 172.21.15.42:0/363795215 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f53fc00a5b0 0x7f541c1ca930 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f5404011fd0 tx=0x7f5404013e00 comp rx=0 tx=0).stop 2024-09-16T03:48:57.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.675+0000 7f5421e6d640 1 -- 172.21.15.42:0/363795215 shutdown_connections 2024-09-16T03:48:57.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.675+0000 7f5421e6d640 1 --2- 172.21.15.42:0/363795215 >> v2:172.21.15.154:6812/254845342 conn(0x7f53fc03c430 0x7f53fc03e8f0 unknown :-1 s=CLOSED pgs=3500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:57.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.675+0000 7f5421e6d640 1 --2- 172.21.15.42:0/363795215 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f541c0a1330 0x7f541c122910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:57.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.675+0000 7f5421e6d640 1 --2- 172.21.15.42:0/363795215 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f53fc00a5b0 0x7f541c1ca930 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:57.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.675+0000 7f5421e6d640 1 --2- 172.21.15.42:0/363795215 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f53fc008d50 0x7f541c1ca3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:57.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.675+0000 7f5421e6d640 1 -- 172.21.15.42:0/363795215 >> 172.21.15.42:0/363795215 conn(0x7f541c0a2420 msgr2=0x7f541c0709a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:48:57.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.675+0000 7f5421e6d640 1 -- 172.21.15.42:0/363795215 shutdown_connections 2024-09-16T03:48:57.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.675+0000 7f5421e6d640 1 -- 172.21.15.42:0/363795215 wait complete. 2024-09-16T03:48:57.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.675+0000 7f5421e6d640 1 librados: shutdown 2024-09-16T03:48:57.696 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T03:48:57.697 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-16T03:48:57.697 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T03:48:57.817 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.811+0000 7f716f3f5640 10 monclient: get_monmap_and_config 2024-09-16T03:48:57.817 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.811+0000 7f716f3f5640 10 monclient: build_initial_monmap 2024-09-16T03:48:57.817 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.811+0000 7f716f3f5640 10 monclient: monmap: 2024-09-16T03:48:57.817 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:48:57.817 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:48:57.817 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:48:57.815974+0000 2024-09-16T03:48:57.817 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:48:57.815974+0000 2024-09-16T03:48:57.817 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:48:57.817 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:48:57.818 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:48:57.818 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:48:57.818 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:48:57.818 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:48:57.818 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:48:57.818 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:48:57.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.815+0000 7f716f3f5640 1 Processor -- start 2024-09-16T03:48:57.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.815+0000 7f716f3f5640 1 -- start start 2024-09-16T03:48:57.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.815+0000 7f716f3f5640 10 monclient: init 2024-09-16T03:48:57.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.815+0000 7f716f3f5640 10 monclient: _reopen_session rank -1 2024-09-16T03:48:57.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.815+0000 7f716f3f5640 10 monclient: _add_conns ranks=[2,4,0,1,3] 2024-09-16T03:48:57.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.815+0000 7f716f3f5640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f71680a1330 0x7f71680a1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:57.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.815+0000 7f716f3f5640 10 monclient(hunting): picked mon.noname-b con 0x7f71680a1330 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:48:57.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.815+0000 7f716f3f5640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f716806dac0 0x7f71680a1c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:57.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.815+0000 7f716f3f5640 10 monclient(hunting): picked mon.noname-c con 0x7f716806dac0 addr v2:172.21.15.154:3300/0 2024-09-16T03:48:57.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.815+0000 7f716f3f5640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7168135100 addr v1:172.21.15.42:6789/0 2024-09-16T03:48:57.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.815+0000 7f716f3f5640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f716809c7d0 con 0x7f7168135100 2024-09-16T03:48:57.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.815+0000 7f716f3f5640 10 monclient(hunting): start opening mon connection 2024-09-16T03:48:57.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.815+0000 7f716f3f5640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f716809c090 con 0x7f716806dac0 2024-09-16T03:48:57.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.815+0000 7f716f3f5640 10 monclient(hunting): start opening mon connection 2024-09-16T03:48:57.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.815+0000 7f716f3f5640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f716809f610 con 0x7f71680a1330 2024-09-16T03:48:57.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.815+0000 7f716f3f5640 10 monclient(hunting): _renew_subs 2024-09-16T03:48:57.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.815+0000 7f716f3f5640 10 monclient(hunting): authenticate will time out at 1384.785367s 2024-09-16T03:48:57.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.815+0000 7f716d191640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f71680a1330 0x7f71680a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:48:57.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.815+0000 7f716d992640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f7168135100 0x7f7168135550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:48924/0 (socket says 172.21.15.42:48924) 2024-09-16T03:48:57.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.815+0000 7f716d992640 1 -- 172.21.15.42:0/1115049143 learned_addr learned my addr 172.21.15.42:0/1115049143 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:48:57.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.815+0000 7f716d191640 10 monclient(hunting): get_auth_request con 0x7f71680a1330 auth_method 0 2024-09-16T03:48:57.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.815+0000 7f716d191640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:48:57.823 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.815+0000 7f716d191640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:48:57.823 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.815+0000 7f716d191640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:48:57.823 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.815+0000 7f716c990640 1 -- 172.21.15.42:0/1115049143 >> v2:172.21.15.154:3300/0 conn(0x7f716806dac0 msgr2=0x7f71680a1c50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T03:48:57.823 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.815+0000 7f716c990640 1 --2- 172.21.15.42:0/1115049143 >> v2:172.21.15.154:3300/0 conn(0x7f716806dac0 0x7f71680a1c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:48:57.823 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.815+0000 7f716d191640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:48:57.823 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.815+0000 7f716d191640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:48:57.823 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.815+0000 7f716d191640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:48:57.823 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.815+0000 7f716d191640 10 monclient(hunting): handle_auth_done global_id 784157 payload 274 2024-09-16T03:48:57.823 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.815+0000 7f716d191640 1 -- 172.21.15.42:0/1115049143 >> v2:172.21.15.154:3300/0 conn(0x7f716806dac0 msgr2=0x7f71680a1c50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:48:57.823 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.815+0000 7f716d191640 1 --2- 172.21.15.42:0/1115049143 >> v2:172.21.15.154:3300/0 conn(0x7f716806dac0 0x7f71680a1c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:57.823 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.815+0000 7f716d191640 1 -- 172.21.15.42:0/1115049143 >> v1:172.21.15.42:6789/0 conn(0x7f7168135100 legacy=0x7f7168135550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:48:57.824 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.815+0000 7f716d191640 10 monclient: _finish_hunting 0 2024-09-16T03:48:57.824 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.815+0000 7f716d191640 1 monclient: found mon.noname-b 2024-09-16T03:48:57.824 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.815+0000 7f716d191640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:48:57.824 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.815+0000 7f716d191640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T03:48:57.824 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.815+0000 7f716d191640 1 -- 172.21.15.42:0/1115049143 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f716809f8f0 con 0x7f71680a1330 2024-09-16T03:48:57.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.815+0000 7f716d191640 1 --2- 172.21.15.42:0/1115049143 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f71680a1330 0x7f71680a1710 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f7164009d90 tx=0x7f716401c970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eebbe8882a529654 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:48:57.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.819+0000 7f715ffff640 1 -- 172.21.15.42:0/1115049143 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71640b2070 con 0x7f71680a1330 2024-09-16T03:48:57.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.819+0000 7f715ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:48:57.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.819+0000 7f715ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:48:57.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.819+0000 7f715ffff640 10 monclient: dump: 2024-09-16T03:48:57.832 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:48:57.832 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:48:57.832 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:57.832 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:57.832 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:48:57.832 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:48:57.832 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:48:57.832 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:48:57.832 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:48:57.832 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:48:57.832 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.819+0000 7f715ffff640 10 monclient: _finish_auth 0 2024-09-16T03:48:57.832 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.819+0000 7f715ffff640 10 monclient: _check_auth_tickets 2024-09-16T03:48:57.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.819+0000 7f715ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:48:57.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.819+0000 7f715ffff640 1 -- 172.21.15.42:0/1115049143 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f71640a5940 con 0x7f71680a1330 2024-09-16T03:48:57.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.819+0000 7f715ffff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:48:57.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.819+0000 7f715ffff640 1 -- 172.21.15.42:0/1115049143 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71640a5d10 con 0x7f71680a1330 2024-09-16T03:48:57.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.819+0000 7f716f3f5640 5 monclient: authenticate success, global_id 784157 2024-09-16T03:48:57.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.819+0000 7f716f3f5640 10 monclient: get_monmap_and_config success 2024-09-16T03:48:57.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.819+0000 7f716f3f5640 4 set_mon_vals no callback set 2024-09-16T03:48:57.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.819+0000 7f715ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:48:57.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.819+0000 7f715ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:48:57.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.819+0000 7f715ffff640 10 monclient: dump: 2024-09-16T03:48:57.833 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:48:57.833 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:48:57.834 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:57.834 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:57.834 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:48:57.834 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:48:57.834 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:48:57.834 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:48:57.834 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:48:57.834 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:48:57.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.819+0000 7f716f3f5640 10 monclient: shutdown 2024-09-16T03:48:57.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.819+0000 7f716f3f5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:48:57.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.819+0000 7f716f3f5640 1 -- 172.21.15.42:0/1115049143 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f71680a1330 msgr2=0x7f71680a1710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:48:57.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.819+0000 7f716f3f5640 1 --2- 172.21.15.42:0/1115049143 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f71680a1330 0x7f71680a1710 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f7164009d90 tx=0x7f716401c970 comp rx=0 tx=0).stop 2024-09-16T03:48:57.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.819+0000 7f716f3f5640 1 -- 172.21.15.42:0/1115049143 shutdown_connections 2024-09-16T03:48:57.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.819+0000 7f716f3f5640 1 --2- 172.21.15.42:0/1115049143 >> v2:172.21.15.154:3300/0 conn(0x7f716806dac0 0x7f71680a1c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:57.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.819+0000 7f716f3f5640 1 --2- 172.21.15.42:0/1115049143 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f71680a1330 0x7f71680a1710 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:57.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.819+0000 7f716f3f5640 1 -- 172.21.15.42:0/1115049143 >> 172.21.15.42:0/1115049143 conn(0x7f71680a2420 msgr2=0x7f71680a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:48:57.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.819+0000 7f716f3f5640 1 -- 172.21.15.42:0/1115049143 shutdown_connections 2024-09-16T03:48:57.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.819+0000 7f716f3f5640 1 -- 172.21.15.42:0/1115049143 wait complete. 2024-09-16T03:48:57.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.819+0000 7f716f3f5640 10 monclient: build_initial_monmap 2024-09-16T03:48:57.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.819+0000 7f716f3f5640 10 monclient: monmap: 2024-09-16T03:48:57.835 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:48:57.835 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:48:57.835 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:48:57.835 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:48:57.835 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:48:57.835 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:48:57.835 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:48:57.836 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:48:57.836 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:48:57.836 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:48:57.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.819+0000 7f716f3f5640 1 librados: starting msgr at 2024-09-16T03:48:57.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.819+0000 7f716f3f5640 1 librados: starting objecter 2024-09-16T03:48:57.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.819+0000 7f716f3f5640 1 Processor -- start 2024-09-16T03:48:57.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.819+0000 7f716f3f5640 1 -- start start 2024-09-16T03:48:57.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.819+0000 7f716f3f5640 1 librados: setting wanted keys 2024-09-16T03:48:57.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.819+0000 7f716f3f5640 1 librados: calling monclient init 2024-09-16T03:48:57.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.819+0000 7f716f3f5640 10 monclient: init 2024-09-16T03:48:57.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.819+0000 7f716f3f5640 10 monclient: _reopen_session rank -1 2024-09-16T03:48:57.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.819+0000 7f716f3f5640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:48:57.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.819+0000 7f716f3f5640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f71680a1330 0x7f716813c120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:57.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.819+0000 7f716f3f5640 10 monclient(hunting): picked mon.noname-b con 0x7f71680a1330 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:48:57.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.819+0000 7f716f3f5640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7168135100 0x7f716811bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:57.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.819+0000 7f716f3f5640 10 monclient(hunting): picked mon.noname-a con 0x7f7168135100 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:48:57.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.819+0000 7f716f3f5640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f71681342e0 0x7f71681cd020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:57.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.819+0000 7f716f3f5640 10 monclient(hunting): picked mon.noname-c con 0x7f71681342e0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:48:57.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.819+0000 7f716f3f5640 10 monclient(hunting): start opening mon connection 2024-09-16T03:48:57.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.819+0000 7f716f3f5640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f716813fd00 con 0x7f7168135100 2024-09-16T03:48:57.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.819+0000 7f716f3f5640 10 monclient(hunting): start opening mon connection 2024-09-16T03:48:57.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.819+0000 7f716f3f5640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f716813fb80 con 0x7f71680a1330 2024-09-16T03:48:57.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.819+0000 7f716f3f5640 10 monclient(hunting): start opening mon connection 2024-09-16T03:48:57.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.819+0000 7f716f3f5640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f716813fe80 con 0x7f71681342e0 2024-09-16T03:48:57.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.819+0000 7f716f3f5640 10 monclient(hunting): _renew_subs 2024-09-16T03:48:57.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.819+0000 7f716f3f5640 10 monclient(hunting): authenticate will time out at 1384.790379s 2024-09-16T03:48:57.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.819+0000 7f716d992640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f71681342e0 0x7f71681cd020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:48:57.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.819+0000 7f716d992640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f71681342e0 0x7f71681cd020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:39122/0 (socket says 172.21.15.42:39122) 2024-09-16T03:48:57.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.819+0000 7f716d992640 1 -- 172.21.15.42:0/1391896125 learned_addr learned my addr 172.21.15.42:0/1391896125 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:48:57.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.819+0000 7f716d992640 10 monclient(hunting): get_auth_request con 0x7f71681342e0 auth_method 0 2024-09-16T03:48:57.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.819+0000 7f716d992640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:48:57.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.819+0000 7f716d992640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:48:57.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.819+0000 7f716d992640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:48:57.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.819+0000 7f716d191640 1 -- 172.21.15.42:0/1391896125 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f71680a1330 msgr2=0x7f716813c120 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T03:48:57.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.819+0000 7f716c990640 1 --2- 172.21.15.42:0/1391896125 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7168135100 0x7f716811bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:48:57.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.819+0000 7f716d191640 1 --2- 172.21.15.42:0/1391896125 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f71680a1330 0x7f716813c120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:48:57.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.819+0000 7f716c990640 10 monclient(hunting): get_auth_request con 0x7f7168135100 auth_method 0 2024-09-16T03:48:57.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.819+0000 7f716c990640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:48:57.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.819+0000 7f716c990640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:48:57.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.819+0000 7f716c990640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:48:57.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.819+0000 7f716d992640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:48:57.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.819+0000 7f716d992640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:48:57.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.819+0000 7f716d992640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:48:57.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.819+0000 7f716c990640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:48:57.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.819+0000 7f716c990640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:48:57.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.819+0000 7f716c990640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:48:57.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.819+0000 7f716d992640 10 monclient(hunting): handle_auth_done global_id 784160 payload 824 2024-09-16T03:48:57.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.819+0000 7f716d992640 1 -- 172.21.15.42:0/1391896125 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f71680a1330 msgr2=0x7f716813c120 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:48:57.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.819+0000 7f716d992640 1 --2- 172.21.15.42:0/1391896125 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f71680a1330 0x7f716813c120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:57.841 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.819+0000 7f716d992640 1 -- 172.21.15.42:0/1391896125 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7168135100 msgr2=0x7f716811bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:48:57.841 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.819+0000 7f716d992640 1 --2- 172.21.15.42:0/1391896125 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7168135100 0x7f716811bf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:57.841 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.819+0000 7f716d992640 10 monclient: _finish_hunting 0 2024-09-16T03:48:57.841 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.819+0000 7f716d992640 1 monclient: found mon.noname-c 2024-09-16T03:48:57.841 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.819+0000 7f716d992640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:48:57.841 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.819+0000 7f716d992640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:48:57.841 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.819+0000 7f716d992640 1 -- 172.21.15.42:0/1391896125 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f716811c4b0 con 0x7f71681342e0 2024-09-16T03:48:57.841 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.819+0000 7f716d992640 1 --2- 172.21.15.42:0/1391896125 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f71681342e0 0x7f71681cd020 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f716000c430 tx=0x7f716000c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:48:57.841 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.819+0000 7f715dffb640 10 client.?.objecter ms_handle_connect 0x7f71681342e0 2024-09-16T03:48:57.841 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.819+0000 7f715dffb640 10 client.?.objecter resend_mon_ops 2024-09-16T03:48:57.841 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.819+0000 7f715dffb640 1 -- 172.21.15.42:0/1391896125 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7160017070 con 0x7f71681342e0 2024-09-16T03:48:57.841 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.819+0000 7f715dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:48:57.842 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.819+0000 7f715dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:48:57.842 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.819+0000 7f715dffb640 10 monclient: dump: 2024-09-16T03:48:57.842 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:48:57.842 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:48:57.842 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:57.842 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:57.843 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:48:57.843 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:48:57.843 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:48:57.843 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:48:57.844 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:48:57.844 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:48:57.844 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.819+0000 7f715dffb640 10 monclient: _finish_auth 0 2024-09-16T03:48:57.844 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.819+0000 7f715dffb640 10 monclient: _check_auth_tickets 2024-09-16T03:48:57.844 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.819+0000 7f715dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:48:57.844 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.819+0000 7f715dffb640 1 -- 172.21.15.42:0/1391896125 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7160005cf0 con 0x7f71681342e0 2024-09-16T03:48:57.844 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.819+0000 7f715dffb640 10 monclient: handle_config config(0 keys) 2024-09-16T03:48:57.844 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.819+0000 7f715dffb640 1 -- 172.21.15.42:0/1391896125 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71600072d0 con 0x7f71681342e0 2024-09-16T03:48:57.844 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.819+0000 7f716f3f5640 5 monclient: authenticate success, global_id 784160 2024-09-16T03:48:57.845 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.819+0000 7f716f3f5640 10 monclient: _renew_subs 2024-09-16T03:48:57.845 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.819+0000 7f716f3f5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:48:57.845 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.819+0000 7f716f3f5640 1 -- 172.21.15.42:0/1391896125 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f716813cd60 con 0x7f71681342e0 2024-09-16T03:48:57.845 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.819+0000 7f715f7fe640 4 set_mon_vals no callback set 2024-09-16T03:48:57.845 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.819+0000 7f715dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:48:57.845 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.819+0000 7f715dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:48:57.845 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.819+0000 7f715dffb640 10 monclient: dump: 2024-09-16T03:48:57.845 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:48:57.845 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:48:57.845 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:57.845 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:57.846 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:48:57.846 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:48:57.846 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:48:57.846 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:48:57.846 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:48:57.846 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:48:57.846 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.819+0000 7f716f3f5640 10 client.784160.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:48:57.846 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.819+0000 7f716f3f5640 10 monclient: _renew_subs 2024-09-16T03:48:57.846 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.819+0000 7f716f3f5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:48:57.847 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.819+0000 7f716f3f5640 1 -- 172.21.15.42:0/1391896125 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f7168134770 con 0x7f71681342e0 2024-09-16T03:48:57.847 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.819+0000 7f716f3f5640 1 librados: init done 2024-09-16T03:48:57.847 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.823+0000 7f716f3f5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:48:57.847 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.823+0000 7f716f3f5640 1 --2- 172.21.15.42:0/1391896125 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f716806dac0 0x7f716802c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:57.847 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.823+0000 7f716f3f5640 10 monclient(hunting): start opening mon connection 2024-09-16T03:48:57.848 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.823+0000 7f716f3f5640 1 -- 172.21.15.42:0/1391896125 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f716811d090 con 0x7f716806dac0 2024-09-16T03:48:57.848 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.823+0000 7f716d191640 1 --2- 172.21.15.42:0/1391896125 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f716806dac0 0x7f716802c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:48:57.848 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.823+0000 7f716d191640 10 monclient: get_auth_request con 0x7f716806dac0 auth_method 0 2024-09-16T03:48:57.849 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.823+0000 7f716d191640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:48:57.849 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.823+0000 7f716d191640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:48:57.849 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.823+0000 7f716d191640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:48:57.849 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.823+0000 7f715dffb640 1 -- 172.21.15.42:0/1391896125 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f716001a070 con 0x7f71681342e0 2024-09-16T03:48:57.849 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.823+0000 7f716d191640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:48:57.849 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.823+0000 7f716d191640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:48:57.849 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.823+0000 7f716d191640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:48:57.849 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.823+0000 7f716d191640 10 monclient(hunting): handle_auth_done global_id 764571 payload 824 2024-09-16T03:48:57.849 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.823+0000 7f715dffb640 1 --2- 172.21.15.42:0/1391896125 >> v2:172.21.15.154:6812/254845342 conn(0x7f714c0377c0 0x7f714c039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:57.849 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.823+0000 7f716d191640 1 -- 172.21.15.42:0/1391896125 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7168073a30 con 0x7f716806dac0 2024-09-16T03:48:57.849 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.823+0000 7f715dffb640 1 -- 172.21.15.42:0/1391896125 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(219..219 src has 1..219) ==== 6246+0+0 (secure 0 0 0) 0x7f7160048630 con 0x7f71681342e0 2024-09-16T03:48:57.849 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.823+0000 7f715dffb640 10 client.784160.objecter ms_dispatch 0x7f7168000ba0 osd_map(219..219 src has 1..219) 2024-09-16T03:48:57.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.823+0000 7f715dffb640 3 client.784160.objecter handle_osd_map got epochs [219,219] > 0 2024-09-16T03:48:57.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.823+0000 7f715dffb640 3 client.784160.objecter handle_osd_map decoding full epoch 219 2024-09-16T03:48:57.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.823+0000 7f715dffb640 20 client.784160.objecter dump_active .. 0 homeless 2024-09-16T03:48:57.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.823+0000 7f715dffb640 10 client.784160.objecter ms_handle_connect 0x7f716806dac0 2024-09-16T03:48:57.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.823+0000 7f716d191640 1 --2- 172.21.15.42:0/1391896125 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f716806dac0 0x7f716802c650 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f716809bb00 tx=0x7f71680a22d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:48:57.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.823+0000 7f715dffb640 10 client.784160.objecter resend_mon_ops 2024-09-16T03:48:57.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.823+0000 7f716c990640 1 --2- 172.21.15.42:0/1391896125 >> v2:172.21.15.154:6812/254845342 conn(0x7f714c0377c0 0x7f714c039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:48:57.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.823+0000 7f715dffb640 1 -- 172.21.15.42:0/1391896125 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71640bd070 con 0x7f716806dac0 2024-09-16T03:48:57.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.823+0000 7f715dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:48:57.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.823+0000 7f715dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:48:57.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.823+0000 7f715dffb640 10 monclient: dump: 2024-09-16T03:48:57.851 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:48:57.851 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:48:57.851 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:57.851 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:57.851 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:48:57.851 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:48:57.851 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:48:57.851 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:48:57.851 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:48:57.851 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:48:57.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.823+0000 7f716c990640 10 monclient: get_auth_request con 0x7f714c0377c0 auth_method 0 2024-09-16T03:48:57.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.823+0000 7f715dffb640 1 -- 172.21.15.42:0/1391896125 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f7164003ba0 con 0x7f716806dac0 2024-09-16T03:48:57.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.823+0000 7f715dffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:48:57.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.823+0000 7f715dffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:48:57.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.823+0000 7f715dffb640 1 -- 172.21.15.42:0/1391896125 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f716806dac0 msgr2=0x7f716802c650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:48:57.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.823+0000 7f715dffb640 1 --2- 172.21.15.42:0/1391896125 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f716806dac0 0x7f716802c650 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f716809bb00 tx=0x7f71680a22d0 comp rx=0 tx=0).stop 2024-09-16T03:48:57.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.823+0000 7f715dffb640 1 -- 172.21.15.42:0/1391896125 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f716806dac0 msgr2=0x7f716802c650 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:48:57.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.823+0000 7f715dffb640 1 --2- 172.21.15.42:0/1391896125 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f716806dac0 0x7f716802c650 secure :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f716809bb00 tx=0x7f71680a22d0 comp rx=0 tx=0).stop 2024-09-16T03:48:57.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.823+0000 7f715dffb640 10 client.784160.objecter ms_handle_connect 0x7f714c0377c0 2024-09-16T03:48:57.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.823+0000 7f716c990640 1 --2- 172.21.15.42:0/1391896125 >> v2:172.21.15.154:6812/254845342 conn(0x7f714c0377c0 0x7f714c039c80 secure :-1 s=READY pgs=3501 cs=0 l=1 rev1=1 crypto rx=0x7f71580093d0 tx=0x7f7158009360 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:48:57.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.827+0000 7f716f3f5640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:48:57.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.827+0000 7f716f3f5640 1 --2- 172.21.15.42:0/1391896125 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f716802cb90 0x7f716802cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:57.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.827+0000 7f716f3f5640 10 monclient(hunting): start opening mon connection 2024-09-16T03:48:57.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.827+0000 7f716f3f5640 1 -- 172.21.15.42:0/1391896125 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f7168074230 con 0x7f716802cb90 2024-09-16T03:48:57.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.827+0000 7f716d191640 1 --2- 172.21.15.42:0/1391896125 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f716802cb90 0x7f716802cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:48:57.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.827+0000 7f716d191640 10 monclient: get_auth_request con 0x7f716802cb90 auth_method 0 2024-09-16T03:48:57.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.827+0000 7f716d191640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:48:57.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.827+0000 7f716d191640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:48:57.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.827+0000 7f716d191640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:48:57.857 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.827+0000 7f716d191640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:48:57.857 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.827+0000 7f716d191640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:48:57.857 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.827+0000 7f716d191640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:48:57.857 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.827+0000 7f716d191640 10 monclient(hunting): handle_auth_done global_id 764574 payload 824 2024-09-16T03:48:57.857 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.827+0000 7f716d191640 1 -- 172.21.15.42:0/1391896125 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f7168139150 con 0x7f716802cb90 2024-09-16T03:48:57.857 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.831+0000 7f716d191640 1 --2- 172.21.15.42:0/1391896125 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f716802cb90 0x7f716802cfa0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f716401b020 tx=0x7f7164009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:48:57.857 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.831+0000 7f715dffb640 10 client.784160.objecter ms_handle_connect 0x7f716802cb90 2024-09-16T03:48:57.857 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.831+0000 7f715dffb640 10 client.784160.objecter resend_mon_ops 2024-09-16T03:48:57.857 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.831+0000 7f715dffb640 1 -- 172.21.15.42:0/1391896125 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71640b2040 con 0x7f716802cb90 2024-09-16T03:48:57.857 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.831+0000 7f715dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:48:57.857 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.831+0000 7f715dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:48:57.857 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.831+0000 7f715dffb640 10 monclient: dump: 2024-09-16T03:48:57.857 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:48:57.857 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:48:57.858 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:57.858 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:57.858 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:48:57.858 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:48:57.858 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:48:57.858 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:48:57.858 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:48:57.858 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:48:57.858 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.831+0000 7f715dffb640 1 -- 172.21.15.42:0/1391896125 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4870 (secure 0 0 0) 0x7f71640c3a50 con 0x7f716802cb90 2024-09-16T03:48:57.858 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.831+0000 7f715dffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:48:57.858 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.831+0000 7f715dffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:48:57.858 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.831+0000 7f715dffb640 1 -- 172.21.15.42:0/1391896125 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f716802cb90 msgr2=0x7f716802cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:48:57.858 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.831+0000 7f715dffb640 1 --2- 172.21.15.42:0/1391896125 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f716802cb90 0x7f716802cfa0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f716401b020 tx=0x7f7164009750 comp rx=0 tx=0).stop 2024-09-16T03:48:57.858 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.831+0000 7f715dffb640 1 -- 172.21.15.42:0/1391896125 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f716802cb90 msgr2=0x7f716802cfa0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:48:57.859 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.831+0000 7f715dffb640 1 --2- 172.21.15.42:0/1391896125 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f716802cb90 0x7f716802cfa0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:57.859 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:48:57.859 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:48:57.859 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:48:57.859 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "leader", 2024-09-16T03:48:57.859 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 64, 2024-09-16T03:48:57.859 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 128622, 2024-09-16T03:48:57.859 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:48:57.859 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:48:57.860 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T03:48:57.860 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:48:57.860 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 2, 2024-09-16T03:48:57.860 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:48:57.860 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:48:57.860 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:48:57.860 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:48:57.860 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:48:57.860 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:48:57.860 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:48:57.860 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:48:57.860 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:48:57.860 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:48:57.861 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:48:57.861 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:48:57.861 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:48:57.861 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:48:57.861 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:48:57.861 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:48:57.861 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:48:57.861 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:48:57.861 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:48:57.861 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:48:57.862 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:48:57.862 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:48:57.862 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:48:57.862 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:48:57.862 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:48:57.862 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:48:57.862 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:48:57.862 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:48:57.862 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:48:57.862 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:48:57.862 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:48:57.862 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:48:57.862 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:48:57.862 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:48:57.862 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:48:57.863 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:48:57.863 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:48:57.863 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:48:57.863 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:48:57.863 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:48:57.863 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:48:57.863 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:48:57.863 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:48:57.863 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:48:57.864 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:48:57.864 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:48:57.864 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:48:57.864 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:48:57.864 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:48:57.865 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:48:57.865 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:48:57.865 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:48:57.865 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:48:57.865 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:48:57.865 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:48:57.865 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:48:57.865 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:48:57.865 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:48:57.865 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:48:57.865 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:48:57.865 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:48:57.865 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:48:57.866 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:48:57.866 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:48:57.866 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:48:57.866 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:48:57.866 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:48:57.866 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:48:57.866 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:48:57.866 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:48:57.866 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:48:57.866 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:48:57.867 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:48:57.867 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:48:57.867 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:48:57.867 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:48:57.867 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:48:57.867 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:48:57.867 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:48:57.867 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:48:57.867 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:48:57.867 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:48:57.867 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:48:57.867 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:48:57.868 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:48:57.868 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:48:57.868 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:48:57.868 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:48:57.868 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:48:57.868 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:48:57.868 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:48:57.868 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:48:57.868 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:48:57.868 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:48:57.868 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:48:57.868 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:48:57.869 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:48:57.869 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:48:57.869 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:48:57.869 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:48:57.869 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:48:57.869 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:48:57.869 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:48:57.869 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:48:57.869 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:48:57.869 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:48:57.869 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:48:57.869 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:48:57.869 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:48:57.869 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:48:57.870 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:48:57.870 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:48:57.870 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:48:57.870 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:48:57.870 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:48:57.870 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:48:57.870 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:48:57.870 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:48:57.870 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:48:57.870 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:48:57.870 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:48:57.870 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:48:57.870 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:48:57.871 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:48:57.871 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:48:57.871 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:48:57.871 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:48:57.871 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:48:57.871 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:48:57.871 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:48:57.871 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:48:57.871 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:48:57.872 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:48:57.872 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:48:57.872 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:48:57.872 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:48:57.872 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:48:57.872 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:48:57.873 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:48:57.873 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:48:57.873 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:48:57.873 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T03:48:57.873 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:48:57.873 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:48:57.874 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:48:57.874 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 5 2024-09-16T03:48:57.874 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:48:57.874 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:48:57.874 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:48:57.874 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:48:57.874 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:48:57.875 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:48:57.875 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 8 2024-09-16T03:48:57.875 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:48:57.875 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:48:57.875 INFO:teuthology.orchestra.run.smithi042.stdout: "mgr": [ 2024-09-16T03:48:57.875 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:48:57.875 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:48:57.876 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:48:57.876 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:48:57.876 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:48:57.876 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:48:57.876 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:48:57.876 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:48:57.876 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:48:57.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.831+0000 7f716f3f5640 10 librados: watch_flush enter 2024-09-16T03:48:57.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.831+0000 7f716f3f5640 10 librados: watch_flush exit 2024-09-16T03:48:57.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.831+0000 7f716f3f5640 20 client.784160.objecter shutdown clearing up homeless session... 2024-09-16T03:48:57.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.831+0000 7f716f3f5640 10 client.784160.objecter successfully canceled tick 2024-09-16T03:48:57.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.831+0000 7f716f3f5640 1 -- 172.21.15.42:0/1391896125 >> v2:172.21.15.154:6812/254845342 conn(0x7f714c0377c0 msgr2=0x7f714c039c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:48:57.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.831+0000 7f716f3f5640 1 --2- 172.21.15.42:0/1391896125 >> v2:172.21.15.154:6812/254845342 conn(0x7f714c0377c0 0x7f714c039c80 secure :-1 s=READY pgs=3501 cs=0 l=1 rev1=1 crypto rx=0x7f71580093d0 tx=0x7f7158009360 comp rx=0 tx=0).stop 2024-09-16T03:48:57.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.831+0000 7f716f3f5640 10 monclient: shutdown 2024-09-16T03:48:57.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.831+0000 7f716f3f5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:48:57.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.831+0000 7f716f3f5640 1 -- 172.21.15.42:0/1391896125 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f71681342e0 msgr2=0x7f71681cd020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:48:57.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.831+0000 7f716f3f5640 1 --2- 172.21.15.42:0/1391896125 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f71681342e0 0x7f71681cd020 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f716000c430 tx=0x7f716000c8f0 comp rx=0 tx=0).stop 2024-09-16T03:48:57.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.831+0000 7f716d992640 1 -- 172.21.15.42:0/1391896125 reap_dead start 2024-09-16T03:48:57.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.831+0000 7f716d992640 1 -- 172.21.15.42:0/1391896125 reap_dead start 2024-09-16T03:48:57.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.831+0000 7f716f3f5640 1 -- 172.21.15.42:0/1391896125 shutdown_connections 2024-09-16T03:48:57.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.831+0000 7f716f3f5640 1 -- 172.21.15.42:0/1391896125 >> 172.21.15.42:0/1391896125 conn(0x7f71680a2420 msgr2=0x7f7168136320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:48:57.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.831+0000 7f716f3f5640 1 -- 172.21.15.42:0/1391896125 shutdown_connections 2024-09-16T03:48:57.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.831+0000 7f716f3f5640 1 -- 172.21.15.42:0/1391896125 wait complete. 2024-09-16T03:48:57.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.831+0000 7f716f3f5640 1 librados: shutdown 2024-09-16T03:48:57.882 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T03:48:57.882 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-16T03:48:57.882 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T03:48:58.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.999+0000 7f5a037ef640 10 monclient: get_monmap_and_config 2024-09-16T03:48:58.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.999+0000 7f5a037ef640 10 monclient: build_initial_monmap 2024-09-16T03:48:58.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.999+0000 7f5a037ef640 10 monclient: monmap: 2024-09-16T03:48:58.002 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:48:58.002 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:48:58.002 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:48:58.000492+0000 2024-09-16T03:48:58.002 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:48:58.000492+0000 2024-09-16T03:48:58.002 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:48:58.002 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:48:58.002 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:48:58.002 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:48:58.002 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:48:58.002 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:48:58.002 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:48:58.002 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:48:58.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.999+0000 7f5a037ef640 1 Processor -- start 2024-09-16T03:48:58.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.999+0000 7f5a037ef640 1 -- start start 2024-09-16T03:48:58.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.999+0000 7f5a037ef640 10 monclient: init 2024-09-16T03:48:58.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.999+0000 7f5a037ef640 10 monclient: _reopen_session rank -1 2024-09-16T03:48:58.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.999+0000 7f5a037ef640 10 monclient: _add_conns ranks=[1,4,2,0,3] 2024-09-16T03:48:58.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.999+0000 7f5a037ef640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f59fc0a1330 addr v1:172.21.15.154:6789/0 2024-09-16T03:48:58.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.999+0000 7f5a037ef640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f59fc06dac0 0x7f59fc13a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:58.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.999+0000 7f5a037ef640 10 monclient(hunting): picked mon.noname-c con 0x7f59fc06dac0 addr v2:172.21.15.154:3300/0 2024-09-16T03:48:58.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.999+0000 7f5a037ef640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f59fc0a1e20 0x7f59fc13c890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:58.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.999+0000 7f5a037ef640 10 monclient(hunting): picked mon.noname-b con 0x7f59fc0a1e20 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:48:58.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.999+0000 7f5a037ef640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f59fc09c090 con 0x7f59fc0a1330 2024-09-16T03:48:58.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.999+0000 7f5a037ef640 10 monclient(hunting): start opening mon connection 2024-09-16T03:48:58.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.999+0000 7f5a037ef640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f59fc09c890 con 0x7f59fc06dac0 2024-09-16T03:48:58.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.999+0000 7f5a037ef640 10 monclient(hunting): start opening mon connection 2024-09-16T03:48:58.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.999+0000 7f5a037ef640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f59fc09f610 con 0x7f59fc0a1e20 2024-09-16T03:48:58.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.999+0000 7f5a037ef640 10 monclient(hunting): _renew_subs 2024-09-16T03:48:58.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.999+0000 7f5a037ef640 10 monclient(hunting): authenticate will time out at 1384.969980s 2024-09-16T03:48:58.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.999+0000 7f5a0158b640 1 -- >> v1:172.21.15.154:6789/0 conn(0x7f59fc0a1330 legacy=0x7f59fc0a1710 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.154:6789/0 2024-09-16T03:48:58.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.999+0000 7f5a01d8c640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f59fc0a1e20 0x7f59fc13c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:48:58.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.999+0000 7f5a00d8a640 1 -- >> v2:172.21.15.154:3300/0 conn(0x7f59fc06dac0 msgr2=0x7f59fc13a340 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T03:48:58.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.999+0000 7f5a01d8c640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f59fc0a1e20 0x7f59fc13c890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:39132/0 (socket says 172.21.15.42:39132) 2024-09-16T03:48:58.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.999+0000 7f5a01d8c640 1 -- 172.21.15.42:0/2815552722 learned_addr learned my addr 172.21.15.42:0/2815552722 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:48:58.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.999+0000 7f5a00d8a640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f59fc06dac0 0x7f59fc13a340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:48:58.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.999+0000 7f5a01d8c640 10 monclient(hunting): get_auth_request con 0x7f59fc0a1e20 auth_method 0 2024-09-16T03:48:58.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.999+0000 7f5a01d8c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:48:58.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.999+0000 7f5a01d8c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:48:58.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.999+0000 7f5a01d8c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:48:58.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.999+0000 7f5a01d8c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:48:58.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.999+0000 7f5a01d8c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:48:58.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.999+0000 7f5a01d8c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:48:58.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.999+0000 7f5a01d8c640 10 monclient(hunting): handle_auth_done global_id 784163 payload 274 2024-09-16T03:48:58.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.999+0000 7f5a01d8c640 1 -- 172.21.15.42:0/2815552722 >> v2:172.21.15.154:3300/0 conn(0x7f59fc06dac0 msgr2=0x7f59fc13a340 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:48:58.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.999+0000 7f5a01d8c640 1 --2- 172.21.15.42:0/2815552722 >> v2:172.21.15.154:3300/0 conn(0x7f59fc06dac0 0x7f59fc13a340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:58.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.999+0000 7f5a01d8c640 1 -- 172.21.15.42:0/2815552722 >> v1:172.21.15.154:6789/0 conn(0x7f59fc0a1330 legacy=0x7f59fc0a1710 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:48:58.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.999+0000 7f5a01d8c640 10 monclient: _finish_hunting 0 2024-09-16T03:48:58.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.999+0000 7f5a01d8c640 1 monclient: found mon.noname-b 2024-09-16T03:48:58.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.999+0000 7f5a01d8c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:48:58.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.999+0000 7f5a01d8c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T03:48:58.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:57.999+0000 7f5a01d8c640 1 -- 172.21.15.42:0/2815552722 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59fc09f8f0 con 0x7f59fc0a1e20 2024-09-16T03:48:58.019 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.003+0000 7f5a01d8c640 1 --2- 172.21.15.42:0/2815552722 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f59fc0a1e20 0x7f59fc13c890 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f59ec009870 tx=0x7f59ec01c460 comp rx=0 tx=0).ready entity=mon.2 client_cookie=511a108d48028b31 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:48:58.019 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.003+0000 7f59fbfff640 1 -- 172.21.15.42:0/2815552722 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59ec0ab640 con 0x7f59fc0a1e20 2024-09-16T03:48:58.019 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.003+0000 7f59fbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:48:58.019 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.003+0000 7f59fbfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:48:58.019 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.003+0000 7f59fbfff640 10 monclient: dump: 2024-09-16T03:48:58.019 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:48:58.019 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:48:58.019 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:58.019 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:58.020 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:48:58.020 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:48:58.020 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:48:58.020 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:48:58.020 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:48:58.020 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:48:58.020 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.003+0000 7f59fbfff640 10 monclient: _finish_auth 0 2024-09-16T03:48:58.020 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.003+0000 7f59fbfff640 10 monclient: _check_auth_tickets 2024-09-16T03:48:58.020 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.003+0000 7f59fbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:48:58.020 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.003+0000 7f59fbfff640 1 -- 172.21.15.42:0/2815552722 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f59ec0b0070 con 0x7f59fc0a1e20 2024-09-16T03:48:58.020 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.003+0000 7f59fbfff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:48:58.020 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.003+0000 7f59fbfff640 1 -- 172.21.15.42:0/2815552722 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59ec0aba00 con 0x7f59fc0a1e20 2024-09-16T03:48:58.021 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.003+0000 7f5a037ef640 5 monclient: authenticate success, global_id 784163 2024-09-16T03:48:58.021 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.003+0000 7f5a037ef640 10 monclient: get_monmap_and_config success 2024-09-16T03:48:58.021 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.003+0000 7f5a037ef640 4 set_mon_vals no callback set 2024-09-16T03:48:58.021 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.003+0000 7f59fbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:48:58.021 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.003+0000 7f59fbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:48:58.021 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.003+0000 7f59fbfff640 10 monclient: dump: 2024-09-16T03:48:58.021 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:48:58.021 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:48:58.021 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:58.021 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:58.021 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:48:58.022 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:48:58.022 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:48:58.022 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:48:58.022 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:48:58.022 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:48:58.022 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.003+0000 7f5a037ef640 10 monclient: shutdown 2024-09-16T03:48:58.022 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.003+0000 7f5a037ef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:48:58.022 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.003+0000 7f5a037ef640 1 -- 172.21.15.42:0/2815552722 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f59fc0a1e20 msgr2=0x7f59fc13c890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:48:58.022 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.003+0000 7f5a037ef640 1 --2- 172.21.15.42:0/2815552722 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f59fc0a1e20 0x7f59fc13c890 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f59ec009870 tx=0x7f59ec01c460 comp rx=0 tx=0).stop 2024-09-16T03:48:58.022 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.003+0000 7f5a037ef640 1 -- 172.21.15.42:0/2815552722 shutdown_connections 2024-09-16T03:48:58.022 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.003+0000 7f5a037ef640 1 --2- 172.21.15.42:0/2815552722 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f59fc0a1e20 0x7f59fc13c890 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:58.023 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.003+0000 7f5a037ef640 1 --2- 172.21.15.42:0/2815552722 >> v2:172.21.15.154:3300/0 conn(0x7f59fc06dac0 0x7f59fc13a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:58.023 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.003+0000 7f5a037ef640 1 -- 172.21.15.42:0/2815552722 >> 172.21.15.42:0/2815552722 conn(0x7f59fc0a2420 msgr2=0x7f59fc0a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:48:58.023 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.003+0000 7f5a037ef640 1 -- 172.21.15.42:0/2815552722 shutdown_connections 2024-09-16T03:48:58.023 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.003+0000 7f5a037ef640 1 -- 172.21.15.42:0/2815552722 wait complete. 2024-09-16T03:48:58.023 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.003+0000 7f5a037ef640 10 monclient: build_initial_monmap 2024-09-16T03:48:58.023 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.003+0000 7f5a037ef640 10 monclient: monmap: 2024-09-16T03:48:58.023 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:48:58.023 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:48:58.023 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:48:58.023 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:48:58.023 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:48:58.023 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:48:58.024 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:48:58.024 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:48:58.024 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:48:58.024 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:48:58.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.003+0000 7f5a037ef640 1 librados: starting msgr at 2024-09-16T03:48:58.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.003+0000 7f5a037ef640 1 librados: starting objecter 2024-09-16T03:48:58.025 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.003+0000 7f5a037ef640 1 Processor -- start 2024-09-16T03:48:58.025 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.003+0000 7f5a037ef640 1 -- start start 2024-09-16T03:48:58.025 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.003+0000 7f5a037ef640 1 librados: setting wanted keys 2024-09-16T03:48:58.025 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.003+0000 7f5a037ef640 1 librados: calling monclient init 2024-09-16T03:48:58.025 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.003+0000 7f5a037ef640 10 monclient: init 2024-09-16T03:48:58.025 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.003+0000 7f5a037ef640 10 monclient: _reopen_session rank -1 2024-09-16T03:48:58.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.003+0000 7f5a037ef640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:48:58.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.003+0000 7f5a037ef640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f59fc0a1330 0x7f59fc1ccae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:58.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.003+0000 7f5a037ef640 10 monclient(hunting): picked mon.noname-a con 0x7f59fc0a1330 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:48:58.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.003+0000 7f5a037ef640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f59fc0a1e20 0x7f59fc1352b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:58.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.003+0000 7f5a037ef640 10 monclient(hunting): picked mon.noname-c con 0x7f59fc0a1e20 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:48:58.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.003+0000 7f5a037ef640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f59fc11bf70 0x7f59fc11c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:58.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.003+0000 7f5a037ef640 10 monclient(hunting): picked mon.noname-b con 0x7f59fc11bf70 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:48:58.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.003+0000 7f5a037ef640 10 monclient(hunting): start opening mon connection 2024-09-16T03:48:58.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.003+0000 7f5a037ef640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f59fc13f770 con 0x7f59fc0a1330 2024-09-16T03:48:58.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.003+0000 7f5a037ef640 10 monclient(hunting): start opening mon connection 2024-09-16T03:48:58.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.003+0000 7f5a037ef640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f59fc13f5f0 con 0x7f59fc11bf70 2024-09-16T03:48:58.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.003+0000 7f5a037ef640 10 monclient(hunting): start opening mon connection 2024-09-16T03:48:58.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.003+0000 7f5a037ef640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f59fc13f8f0 con 0x7f59fc0a1e20 2024-09-16T03:48:58.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.003+0000 7f5a037ef640 10 monclient(hunting): _renew_subs 2024-09-16T03:48:58.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.003+0000 7f5a037ef640 10 monclient(hunting): authenticate will time out at 1384.974941s 2024-09-16T03:48:58.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.003+0000 7f5a0158b640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f59fc0a1330 0x7f59fc1ccae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:48:58.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.003+0000 7f5a00d8a640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f59fc0a1e20 0x7f59fc1352b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:48:58.029 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.003+0000 7f5a0158b640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f59fc0a1330 0x7f59fc1ccae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:50638/0 (socket says 172.21.15.42:50638) 2024-09-16T03:48:58.029 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.003+0000 7f5a0158b640 1 -- 172.21.15.42:0/756460306 learned_addr learned my addr 172.21.15.42:0/756460306 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:48:58.029 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.003+0000 7f5a0158b640 10 monclient(hunting): get_auth_request con 0x7f59fc0a1330 auth_method 0 2024-09-16T03:48:58.029 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.003+0000 7f5a0158b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:48:58.029 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.003+0000 7f5a0158b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:48:58.029 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.003+0000 7f5a0158b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:48:58.030 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.003+0000 7f5a00d8a640 10 monclient(hunting): get_auth_request con 0x7f59fc0a1e20 auth_method 0 2024-09-16T03:48:58.030 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.003+0000 7f5a00d8a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:48:58.030 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.003+0000 7f5a00d8a640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:48:58.030 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.003+0000 7f5a00d8a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:48:58.030 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.003+0000 7f5a01d8c640 1 -- 172.21.15.42:0/756460306 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f59fc11bf70 msgr2=0x7f59fc11c3f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T03:48:58.030 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.003+0000 7f5a01d8c640 1 --2- 172.21.15.42:0/756460306 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f59fc11bf70 0x7f59fc11c3f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:48:58.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.003+0000 7f5a0158b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:48:58.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.003+0000 7f5a0158b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:48:58.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.003+0000 7f5a0158b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:48:58.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.003+0000 7f5a00d8a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:48:58.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.003+0000 7f5a00d8a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:48:58.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.003+0000 7f5a00d8a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:48:58.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.003+0000 7f5a0158b640 10 monclient(hunting): handle_auth_done global_id 764577 payload 824 2024-09-16T03:48:58.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.003+0000 7f5a0158b640 1 -- 172.21.15.42:0/756460306 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f59fc0a1e20 msgr2=0x7f59fc1352b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:48:58.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.003+0000 7f5a0158b640 1 --2- 172.21.15.42:0/756460306 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f59fc0a1e20 0x7f59fc1352b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:58.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.003+0000 7f5a0158b640 1 -- 172.21.15.42:0/756460306 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f59fc11bf70 msgr2=0x7f59fc11c3f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:48:58.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.003+0000 7f5a0158b640 1 --2- 172.21.15.42:0/756460306 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f59fc11bf70 0x7f59fc11c3f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:48:58.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.003+0000 7f5a0158b640 10 monclient: _finish_hunting 0 2024-09-16T03:48:58.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.003+0000 7f5a0158b640 1 monclient: found mon.noname-a 2024-09-16T03:48:58.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.003+0000 7f5a0158b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:48:58.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.003+0000 7f5a0158b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:48:58.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.003+0000 7f5a0158b640 1 -- 172.21.15.42:0/756460306 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59fc11fc40 con 0x7f59fc0a1330 2024-09-16T03:48:58.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.003+0000 7f5a00d8a640 1 --2- 172.21.15.42:0/756460306 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f59fc0a1e20 0x7f59fc1352b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:48:58.034 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.003+0000 7f5a0158b640 1 --2- 172.21.15.42:0/756460306 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f59fc0a1330 0x7f59fc1ccae0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f59f400b9e0 tx=0x7f59f400bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:48:58.034 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.003+0000 7f59f9ffb640 10 client.?.objecter ms_handle_connect 0x7f59fc0a1330 2024-09-16T03:48:58.034 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.003+0000 7f59f9ffb640 10 client.?.objecter resend_mon_ops 2024-09-16T03:48:58.034 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.003+0000 7f59f9ffb640 1 -- 172.21.15.42:0/756460306 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59f400cba0 con 0x7f59fc0a1330 2024-09-16T03:48:58.034 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.003+0000 7f59f9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:48:58.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.003+0000 7f59f9ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:48:58.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.003+0000 7f59f9ffb640 10 monclient: dump: 2024-09-16T03:48:58.035 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:48:58.035 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:48:58.035 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:58.035 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:58.036 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:48:58.036 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:48:58.036 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:48:58.036 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:48:58.036 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:48:58.036 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:48:58.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.003+0000 7f59f9ffb640 10 monclient: _finish_auth 0 2024-09-16T03:48:58.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.003+0000 7f59f9ffb640 10 monclient: _check_auth_tickets 2024-09-16T03:48:58.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.003+0000 7f59f9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:48:58.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.003+0000 7f59f9ffb640 1 -- 172.21.15.42:0/756460306 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f59f4010070 con 0x7f59fc0a1330 2024-09-16T03:48:58.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.003+0000 7f5a037ef640 5 monclient: authenticate success, global_id 764577 2024-09-16T03:48:58.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.003+0000 7f5a037ef640 10 monclient: _renew_subs 2024-09-16T03:48:58.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.003+0000 7f5a037ef640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:48:58.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.003+0000 7f5a037ef640 1 -- 172.21.15.42:0/756460306 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59fc06df20 con 0x7f59fc0a1330 2024-09-16T03:48:58.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.003+0000 7f59f9ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T03:48:58.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.003+0000 7f59f9ffb640 1 -- 172.21.15.42:0/756460306 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59f4015470 con 0x7f59fc0a1330 2024-09-16T03:48:58.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.003+0000 7f59f9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:48:58.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.003+0000 7f59f9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:48:58.039 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.003+0000 7f59f9ffb640 10 monclient: dump: 2024-09-16T03:48:58.039 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:48:58.039 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:48:58.039 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:58.039 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:58.039 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:48:58.040 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:48:58.040 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:48:58.040 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:48:58.040 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:48:58.040 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:48:58.040 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.003+0000 7f5a037ef640 10 client.764577.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:48:58.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.003+0000 7f5a037ef640 10 monclient: _renew_subs 2024-09-16T03:48:58.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.003+0000 7f59faffd640 4 set_mon_vals no callback set 2024-09-16T03:48:58.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.003+0000 7f5a037ef640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:48:58.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.003+0000 7f5a037ef640 1 -- 172.21.15.42:0/756460306 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59fc073a30 con 0x7f59fc0a1330 2024-09-16T03:48:58.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.003+0000 7f5a037ef640 1 librados: init done 2024-09-16T03:48:58.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.007+0000 7f5a037ef640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:48:58.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.007+0000 7f5a037ef640 1 --2- 172.21.15.42:0/756460306 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f59fc06dac0 0x7f59fc11c930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:58.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.007+0000 7f5a037ef640 10 monclient(hunting): start opening mon connection 2024-09-16T03:48:58.045 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.007+0000 7f5a037ef640 1 -- 172.21.15.42:0/756460306 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f59fc11e660 con 0x7f59fc06dac0 2024-09-16T03:48:58.045 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.007+0000 7f5a00d8a640 1 --2- 172.21.15.42:0/756460306 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f59fc06dac0 0x7f59fc11c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:48:58.045 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.007+0000 7f5a00d8a640 10 monclient: get_auth_request con 0x7f59fc06dac0 auth_method 0 2024-09-16T03:48:58.045 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.007+0000 7f5a00d8a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:48:58.045 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.007+0000 7f5a00d8a640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:48:58.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.007+0000 7f5a00d8a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:48:58.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.007+0000 7f5a00d8a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:48:58.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.007+0000 7f5a00d8a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:48:58.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.007+0000 7f5a00d8a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:48:58.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.007+0000 7f59f9ffb640 1 -- 172.21.15.42:0/756460306 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f59f401e3f0 con 0x7f59fc0a1330 2024-09-16T03:48:58.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.007+0000 7f5a00d8a640 10 monclient(hunting): handle_auth_done global_id 784169 payload 824 2024-09-16T03:48:58.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.007+0000 7f5a00d8a640 1 -- 172.21.15.42:0/756460306 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f59fc0a1710 con 0x7f59fc06dac0 2024-09-16T03:48:58.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.007+0000 7f59f9ffb640 1 --2- 172.21.15.42:0/756460306 >> v2:172.21.15.154:6812/254845342 conn(0x7f59e80377c0 0x7f59e8039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:58.048 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.007+0000 7f59f9ffb640 1 -- 172.21.15.42:0/756460306 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(219..219 src has 1..219) ==== 6246+0+0 (secure 0 0 0) 0x7f59f40492a0 con 0x7f59fc0a1330 2024-09-16T03:48:58.048 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.007+0000 7f59f9ffb640 10 client.764577.objecter ms_dispatch 0x7f59fc000ba0 osd_map(219..219 src has 1..219) 2024-09-16T03:48:58.048 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.007+0000 7f59f9ffb640 3 client.764577.objecter handle_osd_map got epochs [219,219] > 0 2024-09-16T03:48:58.048 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.007+0000 7f59f9ffb640 3 client.764577.objecter handle_osd_map decoding full epoch 219 2024-09-16T03:48:58.048 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.007+0000 7f59f9ffb640 20 client.764577.objecter dump_active .. 0 homeless 2024-09-16T03:48:58.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.007+0000 7f59f9ffb640 10 client.764577.objecter ms_handle_connect 0x7f59fc06dac0 2024-09-16T03:48:58.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.007+0000 7f59f9ffb640 10 client.764577.objecter resend_mon_ops 2024-09-16T03:48:58.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.007+0000 7f5a00d8a640 1 --2- 172.21.15.42:0/756460306 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f59fc06dac0 0x7f59fc11c930 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f59fc11e930 tx=0x7f59f0005850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:48:58.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.007+0000 7f59f9ffb640 1 -- 172.21.15.42:0/756460306 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59f0016070 con 0x7f59fc06dac0 2024-09-16T03:48:58.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.007+0000 7f59f9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:48:58.050 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.007+0000 7f59f9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:48:58.050 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.007+0000 7f5a01d8c640 1 --2- 172.21.15.42:0/756460306 >> v2:172.21.15.154:6812/254845342 conn(0x7f59e80377c0 0x7f59e8039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:48:58.050 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.007+0000 7f59f9ffb640 10 monclient: dump: 2024-09-16T03:48:58.050 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:48:58.050 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:48:58.050 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:58.051 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:58.051 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:48:58.051 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:48:58.051 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:48:58.051 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:48:58.051 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:48:58.052 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:48:58.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.007+0000 7f5a01d8c640 10 monclient: get_auth_request con 0x7f59e80377c0 auth_method 0 2024-09-16T03:48:58.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.007+0000 7f59f9ffb640 1 -- 172.21.15.42:0/756460306 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f59f000e9e0 con 0x7f59fc06dac0 2024-09-16T03:48:58.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.007+0000 7f59f9ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:48:58.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.007+0000 7f59f9ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:48:58.053 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.007+0000 7f59f9ffb640 1 -- 172.21.15.42:0/756460306 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f59fc06dac0 msgr2=0x7f59fc11c930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:48:58.053 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.007+0000 7f59f9ffb640 1 --2- 172.21.15.42:0/756460306 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f59fc06dac0 0x7f59fc11c930 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f59fc11e930 tx=0x7f59f0005850 comp rx=0 tx=0).stop 2024-09-16T03:48:58.053 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.007+0000 7f59f9ffb640 1 -- 172.21.15.42:0/756460306 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f59fc06dac0 msgr2=0x7f59fc11c930 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:48:58.053 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.007+0000 7f59f9ffb640 1 --2- 172.21.15.42:0/756460306 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f59fc06dac0 0x7f59fc11c930 secure :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f59fc11e930 tx=0x7f59f0005850 comp rx=0 tx=0).stop 2024-09-16T03:48:58.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.007+0000 7f59f9ffb640 10 client.764577.objecter ms_handle_connect 0x7f59e80377c0 2024-09-16T03:48:58.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.007+0000 7f5a01d8c640 1 --2- 172.21.15.42:0/756460306 >> v2:172.21.15.154:6812/254845342 conn(0x7f59e80377c0 0x7f59e8039c80 secure :-1 s=READY pgs=3502 cs=0 l=1 rev1=1 crypto rx=0x7f59ec0af000 tx=0x7f59ec0a7000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:48:58.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.015+0000 7f5a037ef640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:48:58.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.015+0000 7f5a037ef640 1 --2- 172.21.15.42:0/756460306 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f59fc11f5e0 0x7f59fc138330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:48:58.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.015+0000 7f5a037ef640 10 monclient(hunting): start opening mon connection 2024-09-16T03:48:58.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.015+0000 7f5a037ef640 1 -- 172.21.15.42:0/756460306 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f59fc11e270 con 0x7f59fc11f5e0 2024-09-16T03:48:58.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.015+0000 7f5a00d8a640 1 --2- 172.21.15.42:0/756460306 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f59fc11f5e0 0x7f59fc138330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:48:58.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.015+0000 7f5a00d8a640 10 monclient: get_auth_request con 0x7f59fc11f5e0 auth_method 0 2024-09-16T03:48:58.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.015+0000 7f5a00d8a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:48:58.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.015+0000 7f5a00d8a640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:48:58.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.015+0000 7f5a00d8a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:48:58.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.015+0000 7f5a00d8a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:48:58.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.015+0000 7f5a00d8a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:48:58.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.015+0000 7f5a00d8a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:48:58.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.015+0000 7f5a00d8a640 10 monclient(hunting): handle_auth_done global_id 784172 payload 824 2024-09-16T03:48:58.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.015+0000 7f5a00d8a640 1 -- 172.21.15.42:0/756460306 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f59fc11f9f0 con 0x7f59fc11f5e0 2024-09-16T03:48:58.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.015+0000 7f59f9ffb640 10 client.764577.objecter ms_handle_connect 0x7f59fc11f5e0 2024-09-16T03:48:58.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.015+0000 7f5a00d8a640 1 --2- 172.21.15.42:0/756460306 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f59fc11f5e0 0x7f59fc138330 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f59fc11e930 tx=0x7f59f00059e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:48:58.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.015+0000 7f59f9ffb640 10 client.764577.objecter resend_mon_ops 2024-09-16T03:48:58.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.015+0000 7f59f9ffb640 1 -- 172.21.15.42:0/756460306 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59f0016040 con 0x7f59fc11f5e0 2024-09-16T03:48:58.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.015+0000 7f59f9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:48:58.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.015+0000 7f59f9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:48:58.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.015+0000 7f59f9ffb640 10 monclient: dump: 2024-09-16T03:48:58.058 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:48:58.058 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:48:58.058 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:58.058 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:48:58.058 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:48:58.059 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:48:58.059 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:48:58.059 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:48:58.059 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:48:58.059 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:48:58.059 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.015+0000 7f59f9ffb640 1 -- 172.21.15.42:0/756460306 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4700 (secure 0 0 0) 0x7f59f0005a50 con 0x7f59fc11f5e0 2024-09-16T03:48:58.060 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.015+0000 7f59f9ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:48:58.060 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.015+0000 7f59f9ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:48:58.060 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.015+0000 7f59f9ffb640 1 -- 172.21.15.42:0/756460306 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f59fc11f5e0 msgr2=0x7f59fc138330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:48:58.060 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.015+0000 7f59f9ffb640 1 --2- 172.21.15.42:0/756460306 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f59fc11f5e0 0x7f59fc138330 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f59fc11e930 tx=0x7f59f00059e0 comp rx=0 tx=0).stop 2024-09-16T03:48:58.060 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.015+0000 7f59f9ffb640 1 -- 172.21.15.42:0/756460306 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f59fc11f5e0 msgr2=0x7f59fc138330 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:48:58.061 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.015+0000 7f59f9ffb640 1 --2- 172.21.15.42:0/756460306 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f59fc11f5e0 0x7f59fc138330 secure :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f59fc11e930 tx=0x7f59f00059e0 comp rx=0 tx=0).stop 2024-09-16T03:48:58.061 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:48:58.062 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:48:58.063 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:48:58.063 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T03:48:58.063 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 64, 2024-09-16T03:48:58.063 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 29915, 2024-09-16T03:48:58.063 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:48:58.063 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:48:58.063 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T03:48:58.063 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:48:58.063 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 2, 2024-09-16T03:48:58.063 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:48:58.063 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:48:58.063 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:48:58.063 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:48:58.063 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:48:58.064 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:48:58.064 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:48:58.064 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:48:58.064 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:48:58.064 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:48:58.064 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:48:58.064 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:48:58.064 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:48:58.064 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:48:58.064 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:48:58.064 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:48:58.064 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:48:58.064 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:48:58.065 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:48:58.065 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:48:58.065 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:48:58.065 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:48:58.065 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:48:58.065 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:48:58.065 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:48:58.065 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:48:58.065 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:48:58.065 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:48:58.065 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:48:58.065 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:48:58.065 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:48:58.066 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:48:58.066 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:48:58.066 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:48:58.066 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:48:58.066 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:48:58.066 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:48:58.066 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:48:58.066 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:48:58.066 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:48:58.066 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:48:58.066 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:48:58.066 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:48:58.066 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:48:58.066 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:48:58.067 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:48:58.067 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:48:58.067 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:48:58.067 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:48:58.067 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:48:58.067 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:48:58.067 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:48:58.067 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:48:58.067 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:48:58.067 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:48:58.068 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:48:58.068 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:48:58.068 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:48:58.068 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:48:58.068 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:48:58.068 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:48:58.068 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:48:58.068 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:48:58.068 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:48:58.068 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:48:58.068 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:48:58.068 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:48:58.068 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:48:58.068 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:48:58.069 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:48:58.069 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:48:58.069 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:48:58.069 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:48:58.069 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:48:58.069 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:48:58.069 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:48:58.069 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:48:58.069 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:48:58.069 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:48:58.069 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:48:58.069 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:48:58.069 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:48:58.069 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:48:58.070 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:48:58.070 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:48:58.070 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:48:58.070 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:48:58.070 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:48:58.070 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:48:58.070 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:48:58.070 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:48:58.070 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:48:58.070 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:48:58.071 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:48:58.071 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:48:58.071 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:48:58.071 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:48:58.071 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:48:58.071 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:48:58.071 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:48:58.071 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:48:58.071 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:48:58.071 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:48:58.072 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:48:58.072 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:48:58.072 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:48:58.072 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:48:58.072 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:48:58.072 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:48:58.072 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:48:58.072 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:48:58.072 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:48:58.072 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:48:58.072 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:48:58.072 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:48:58.072 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:48:58.073 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:48:58.073 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:48:58.073 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:48:58.073 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:48:58.073 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:48:58.073 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:48:58.073 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:48:58.073 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:48:58.073 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:48:58.073 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:48:58.073 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:48:58.073 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:48:58.073 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:48:58.073 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:48:58.074 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:48:58.074 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:48:58.074 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:48:58.074 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:48:58.074 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:48:58.074 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:48:58.074 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:48:58.074 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:48:58.074 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:48:58.074 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:48:58.074 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:48:58.074 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T03:48:58.074 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:48:58.074 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:48:58.074 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:48:58.075 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:48:58.075 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:48:58.075 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:48:58.075 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:48:58.075 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:48:58.075 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:48:58.075 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:48:58.075 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 2 2024-09-16T03:48:58.075 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:48:58.075 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:48:58.075 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:48:58.075 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:48:58.075 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:48:58.077 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.015+0000 7f5a037ef640 10 librados: watch_flush enter 2024-09-16T03:48:58.077 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.015+0000 7f5a037ef640 10 librados: watch_flush exit 2024-09-16T03:48:58.077 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.015+0000 7f5a037ef640 20 client.764577.objecter shutdown clearing up homeless session... 2024-09-16T03:48:58.077 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.015+0000 7f5a037ef640 10 client.764577.objecter successfully canceled tick 2024-09-16T03:48:58.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.015+0000 7f5a037ef640 1 -- 172.21.15.42:0/756460306 >> v2:172.21.15.154:6812/254845342 conn(0x7f59e80377c0 msgr2=0x7f59e8039c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:48:58.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.015+0000 7f5a037ef640 1 --2- 172.21.15.42:0/756460306 >> v2:172.21.15.154:6812/254845342 conn(0x7f59e80377c0 0x7f59e8039c80 secure :-1 s=READY pgs=3502 cs=0 l=1 rev1=1 crypto rx=0x7f59ec0af000 tx=0x7f59ec0a7000 comp rx=0 tx=0).stop 2024-09-16T03:48:58.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.015+0000 7f5a037ef640 10 monclient: shutdown 2024-09-16T03:48:58.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.015+0000 7f5a037ef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:48:58.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.015+0000 7f5a037ef640 1 -- 172.21.15.42:0/756460306 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f59fc0a1330 msgr2=0x7f59fc1ccae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:48:58.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.015+0000 7f5a037ef640 1 --2- 172.21.15.42:0/756460306 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f59fc0a1330 0x7f59fc1ccae0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f59f400b9e0 tx=0x7f59f400bea0 comp rx=0 tx=0).stop 2024-09-16T03:48:58.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.015+0000 7f5a01d8c640 1 -- 172.21.15.42:0/756460306 reap_dead start 2024-09-16T03:48:58.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.015+0000 7f5a01d8c640 1 -- 172.21.15.42:0/756460306 reap_dead start 2024-09-16T03:48:58.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.015+0000 7f5a037ef640 1 -- 172.21.15.42:0/756460306 shutdown_connections 2024-09-16T03:48:58.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.015+0000 7f5a037ef640 1 -- 172.21.15.42:0/756460306 >> 172.21.15.42:0/756460306 conn(0x7f59fc0a2420 msgr2=0x7f59fc0a46b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:48:58.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.015+0000 7f5a037ef640 1 -- 172.21.15.42:0/756460306 shutdown_connections 2024-09-16T03:48:58.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.015+0000 7f5a037ef640 1 -- 172.21.15.42:0/756460306 wait complete. 2024-09-16T03:48:58.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:48:58.015+0000 7f5a037ef640 1 librados: shutdown 2024-09-16T03:48:58.079 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T03:48:58.079 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-16T03:48:58.467 INFO:tasks.workunit.client.0.smithi154.stdout:ite 2024-09-16T03:48:58.467 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleWrite (7093 ms) 2024-09-16T03:48:58.467 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio: [ RUN ] LibRadosAioEC.WaitForComplete 2024-09-16T03:48:58.467 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio: [ OK ] LibRadosAioEC.WaitForComplete (7084 ms) 2024-09-16T03:48:58.467 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTrip 2024-09-16T03:48:58.467 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTrip (25159 ms) 2024-09-16T03:48:58.467 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTrip2 2024-09-16T03:48:58.467 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTrip2 (7047 ms) 2024-09-16T03:48:58.467 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTripAppend 2024-09-16T03:48:58.467 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTripAppend (6990 ms) 2024-09-16T03:48:58.468 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio: [ RUN ] LibRadosAioEC.IsComplete 2024-09-16T03:48:58.468 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio: [ OK ] LibRadosAioEC.IsComplete (18205 ms) 2024-09-16T03:48:58.468 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio: [ RUN ] LibRadosAioEC.IsSafe 2024-09-16T03:48:58.468 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio: [ OK ] LibRadosAioEC.IsSafe (22108 ms) 2024-09-16T03:48:58.468 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio: [ RUN ] LibRadosAioEC.ReturnValue 2024-09-16T03:48:58.468 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio: [ OK ] LibRadosAioEC.ReturnValue (6918 ms) 2024-09-16T03:48:58.468 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio: [ RUN ] LibRadosAioEC.Flush 2024-09-16T03:48:58.468 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio: [ OK ] LibRadosAioEC.Flush (7029 ms) 2024-09-16T03:48:58.468 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio: [ RUN ] LibRadosAioEC.FlushAsync 2024-09-16T03:48:58.468 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio: [ OK ] LibRadosAioEC.FlushAsync (6969 ms) 2024-09-16T03:48:58.468 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTripWriteFull 2024-09-16T03:48:58.468 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTripWriteFull (21025 ms) 2024-09-16T03:48:58.468 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleStat 2024-09-16T03:48:58.469 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleStat (6986 ms) 2024-09-16T03:48:58.469 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleStatNS 2024-09-16T03:48:58.469 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleStatNS (6992 ms) 2024-09-16T03:48:58.469 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio: [ RUN ] LibRadosAioEC.StatRemove 2024-09-16T03:48:58.469 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio: [ OK ] LibRadosAioEC.StatRemove (6996 ms) 2024-09-16T03:48:58.469 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio: [ RUN ] LibRadosAioEC.ExecuteClass 2024-09-16T03:48:58.469 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio: [ OK ] LibRadosAioEC.ExecuteClass (7116 ms) 2024-09-16T03:48:58.469 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio: [ RUN ] LibRadosAioEC.MultiWrite 2024-09-16T03:48:58.469 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio: [ OK ] LibRadosAioEC.MultiWrite (21074 ms) 2024-09-16T03:48:58.469 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio: [----------] 16 tests from LibRadosAioEC (184791 ms total) 2024-09-16T03:48:58.469 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio: 2024-09-16T03:48:58.469 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio: [----------] Global test environment tear-down 2024-09-16T03:48:58.469 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio: [==========] 42 tests from 2 test suites ran. (328911 ms total) 2024-09-16T03:48:58.469 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio: [ PASSED ] 42 tests. 2024-09-16T03:49:03.281 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [==========] Running 77 tests from 4 test suites. 2024-09-16T03:49:03.281 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [----------] Global test environment set-up. 2024-09-16T03:49:03.281 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [----------] 3 tests from LibRadosTierPP 2024-09-16T03:49:03.281 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: seed 28216 2024-09-16T03:49:03.281 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.Dirty 2024-09-16T03:49:03.281 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ OK ] LibRadosTierPP.Dirty (229 ms) 2024-09-16T03:49:03.281 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.FlushWriteRaces 2024-09-16T03:49:03.282 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ OK ] LibRadosTierPP.FlushWriteRaces (31732 ms) 2024-09-16T03:49:03.282 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.HitSetNone 2024-09-16T03:49:03.282 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ OK ] LibRadosTierPP.HitSetNone (108 ms) 2024-09-16T03:49:03.282 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [----------] 3 tests from LibRadosTierPP (32069 ms total) 2024-09-16T03:49:03.282 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: 2024-09-16T03:49:03.282 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [----------] 48 tests from LibRadosTwoPoolsPP 2024-09-16T03:49:03.282 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Overlay 2024-09-16T03:49:03.282 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Overlay (7002 ms) 2024-09-16T03:49:03.282 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Promote 2024-09-16T03:49:03.282 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Promote (37091 ms) 2024-09-16T03:49:03.282 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnap 2024-09-16T03:49:03.282 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnap (10011 ms) 2024-09-16T03:49:03.282 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnapScrub 2024-09-16T03:49:03.282 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: my_snaps [3] 2024-09-16T03:49:03.283 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: my_snaps [4,3] 2024-09-16T03:49:03.283 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: my_snaps [5,4,3] 2024-09-16T03:49:03.283 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: my_snaps [6,5,4,3] 2024-09-16T03:49:03.283 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: promoting some heads 2024-09-16T03:49:03.283 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: promoting from clones for snap 6 2024-09-16T03:49:03.283 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: promoting from clones for snap 5 2024-09-16T03:49:03.283 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: promoting from clones for snap 4 2024-09-16T03:49:03.283 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: promoting from clones for snap 3 2024-09-16T03:49:03.283 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: waiting for scrubs... 2024-09-16T03:49:03.283 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: done waiting 2024-09-16T03:49:03.283 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnapScrub (47347 ms) 2024-09-16T03:49:03.283 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnapTrimRace 2024-09-16T03:49:03.283 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnapTrimRace (10077 ms) 2024-09-16T03:49:03.283 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Whiteout 2024-09-16T03:49:03.284 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Whiteout (8077 ms) 2024-09-16T03:49:03.284 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.WhiteoutDeleteCreate 2024-09-16T03:49:03.284 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.WhiteoutDeleteCreate (26188 ms) 2024-09-16T03:49:03.284 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Evict 2024-09-16T03:49:03.284 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Evict (8075 ms) 2024-09-16T03:49:03.284 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnap 2024-09-16T03:49:03.284 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnap (21903 ms) 2024-09-16T03:49:03.284 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnap2 2024-09-16T03:49:03.284 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnap2 (23343 ms) 2024-09-16T03:49:03.284 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ListSnap 2024-09-16T03:49:03.284 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ListSnap (9948 ms) 2024-09-16T03:49:03.284 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnapRollbackReadRace 2024-09-16T03:49:03.284 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnapRollbackReadRace (13002 ms) 2024-09-16T03:49:03.285 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TryFlush 2024-09-16T03:49:03.285 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TryFlush (21973 ms) 2024-09-16T03:49:03.285 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Flush 2024-09-16T03:49:03.285 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Flush (7998 ms) 2024-09-16T03:49:03.285 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.FlushSnap 2024-09-16T03:49:03.285 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.FlushSnap (13062 ms) 2024-09-16T03:49:03.285 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.FlushTryFlushRaces 2024-09-16T03:49:03.285 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.FlushTryFlushRaces (22158 ms) 2024-09-16T03:49:03.285 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TryFlushReadRace 2024-09-16T03:49:03.285 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TryFlushReadRace (7807 ms) 2024-09-16T03:49:18.080 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-16T03:49:18.080 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-09-16T03:49:18.080 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-16T03:49:18.080 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-16T03:49:18.083 INFO:tasks.ceph.mon.b:Started 2024-09-16T03:49:18.084 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T03:49:18.084 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:49:18.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.214+0000 7f74652c3640 10 monclient: get_monmap_and_config 2024-09-16T03:49:18.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.214+0000 7f74652c3640 10 monclient: build_initial_monmap 2024-09-16T03:49:18.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.214+0000 7f74652c3640 10 monclient: monmap: 2024-09-16T03:49:18.219 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:49:18.220 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:49:18.220 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:49:18.218446+0000 2024-09-16T03:49:18.220 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:49:18.218446+0000 2024-09-16T03:49:18.220 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:49:18.220 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:49:18.220 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:49:18.220 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:49:18.220 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:49:18.220 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:49:18.220 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:49:18.220 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:49:18.222 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.214+0000 7f74652c3640 1 Processor -- start 2024-09-16T03:49:18.222 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.214+0000 7f74652c3640 1 -- start start 2024-09-16T03:49:18.222 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.214+0000 7f74652c3640 10 monclient: init 2024-09-16T03:49:18.222 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.214+0000 7f74652c3640 10 monclient: _reopen_session rank -1 2024-09-16T03:49:18.222 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.214+0000 7f74652c3640 10 monclient: _add_conns ranks=[2,3,0,1,4] 2024-09-16T03:49:18.222 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.214+0000 7f74652c3640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f74600a3450 0x7f74600a3830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:18.223 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.214+0000 7f74652c3640 10 monclient(hunting): picked mon.noname-b con 0x7f74600a3450 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:49:18.223 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.214+0000 7f74652c3640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f74600a0ea0 0x7f74600a3d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:18.223 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.214+0000 7f74652c3640 10 monclient(hunting): picked mon.noname-a con 0x7f74600a0ea0 addr v2:172.21.15.42:3300/0 2024-09-16T03:49:18.223 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.214+0000 7f74652c3640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7460137220 addr v1:172.21.15.42:6789/0 2024-09-16T03:49:18.223 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.214+0000 7f74652c3640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f74600a00e0 con 0x7f7460137220 2024-09-16T03:49:18.223 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.214+0000 7f74652c3640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:18.223 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.214+0000 7f74652c3640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f74600a1b40 con 0x7f74600a0ea0 2024-09-16T03:49:18.223 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.214+0000 7f74652c3640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:18.223 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.214+0000 7f74652c3640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f74600a1d90 con 0x7f74600a3450 2024-09-16T03:49:18.223 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.214+0000 7f74652c3640 10 monclient(hunting): _renew_subs 2024-09-16T03:49:18.223 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.214+0000 7f74652c3640 10 monclient(hunting): authenticate will time out at 1405.187856s 2024-09-16T03:49:18.223 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.218+0000 7f745effd640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f74600a3450 0x7f74600a3830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:18.223 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.218+0000 7f745e7fc640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f74600a0ea0 0x7f74600a3d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:18.223 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.218+0000 7f745f7fe640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f7460137220 0x7f7460137630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:49712/0 (socket says 172.21.15.42:49712) 2024-09-16T03:49:18.224 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.218+0000 7f745e7fc640 10 monclient(hunting): get_auth_request con 0x7f74600a0ea0 auth_method 0 2024-09-16T03:49:18.224 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.218+0000 7f745f7fe640 1 -- 172.21.15.42:0/359767615 learned_addr learned my addr 172.21.15.42:0/359767615 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:49:18.224 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.218+0000 7f745e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:18.224 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.218+0000 7f745e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:18.224 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.218+0000 7f745e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:18.224 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.218+0000 7f745effd640 10 monclient(hunting): get_auth_request con 0x7f74600a3450 auth_method 0 2024-09-16T03:49:18.224 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.218+0000 7f745effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:18.224 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.218+0000 7f745effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:18.224 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.218+0000 7f745effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:18.224 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.218+0000 7f745e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:18.224 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.218+0000 7f745e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:18.224 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.218+0000 7f745e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:18.224 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.218+0000 7f745effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:18.225 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.218+0000 7f745effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:18.225 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.218+0000 7f745effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:18.225 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.218+0000 7f745e7fc640 10 monclient(hunting): handle_auth_done global_id 764601 payload 274 2024-09-16T03:49:18.225 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.218+0000 7f745e7fc640 1 -- 172.21.15.42:0/359767615 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f74600a3450 msgr2=0x7f74600a3830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:49:18.225 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.218+0000 7f745e7fc640 1 --2- 172.21.15.42:0/359767615 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f74600a3450 0x7f74600a3830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:18.225 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.218+0000 7f745e7fc640 1 -- 172.21.15.42:0/359767615 >> v1:172.21.15.42:6789/0 conn(0x7f7460137220 legacy=0x7f7460137630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:18.225 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.218+0000 7f745e7fc640 10 monclient: _finish_hunting 0 2024-09-16T03:49:18.225 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.218+0000 7f745e7fc640 1 monclient: found mon.noname-a 2024-09-16T03:49:18.225 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.218+0000 7f745e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:49:18.225 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.218+0000 7f745e7fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:49:18.225 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.218+0000 7f745e7fc640 1 -- 172.21.15.42:0/359767615 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f746009fea0 con 0x7f74600a0ea0 2024-09-16T03:49:18.225 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.218+0000 7f745dffb640 1 -- 172.21.15.42:0/359767615 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4006380992 0 0) 0x7f74600a00e0 con 0x7f7460137220 2024-09-16T03:49:18.225 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.218+0000 7f745dffb640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T03:49:18.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.218+0000 7f745e7fc640 1 --2- 172.21.15.42:0/359767615 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f74600a0ea0 0x7f74600a3d70 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f74540097c0 tx=0x7f745401c360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2902586c0e16051 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:18.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.218+0000 7f745dffb640 1 -- 172.21.15.42:0/359767615 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74540b0070 con 0x7f74600a0ea0 2024-09-16T03:49:18.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.218+0000 7f745dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:18.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.218+0000 7f745dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:49:18.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.218+0000 7f745dffb640 10 monclient: dump: 2024-09-16T03:49:18.231 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:49:18.231 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:49:18.231 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:18.231 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:18.232 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:18.232 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:49:18.232 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:49:18.232 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:49:18.232 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:49:18.232 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:49:18.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.218+0000 7f745dffb640 10 monclient: _finish_auth 0 2024-09-16T03:49:18.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.218+0000 7f745dffb640 10 monclient: _check_auth_tickets 2024-09-16T03:49:18.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.218+0000 7f745dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:49:18.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.218+0000 7f745dffb640 1 -- 172.21.15.42:0/359767615 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f74540ab600 con 0x7f74600a0ea0 2024-09-16T03:49:18.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.218+0000 7f74652c3640 5 monclient: authenticate success, global_id 764601 2024-09-16T03:49:18.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.218+0000 7f74652c3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:49:18.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.218+0000 7f745dffb640 10 monclient: handle_config config(0 keys) 2024-09-16T03:49:18.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.218+0000 7f74652c3640 10 monclient: get_monmap_and_config success 2024-09-16T03:49:18.233 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.218+0000 7f74652c3640 4 set_mon_vals no callback set 2024-09-16T03:49:18.233 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.218+0000 7f745dffb640 1 -- 172.21.15.42:0/359767615 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74540ab9d0 con 0x7f74600a0ea0 2024-09-16T03:49:18.233 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.218+0000 7f745dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:18.233 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.218+0000 7f745dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:49:18.233 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.218+0000 7f745dffb640 10 monclient: dump: 2024-09-16T03:49:18.233 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:49:18.233 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:49:18.233 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:18.233 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:18.233 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:18.233 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:49:18.233 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:49:18.233 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:49:18.234 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:49:18.234 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:49:18.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.218+0000 7f74652c3640 10 monclient: shutdown 2024-09-16T03:49:18.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.218+0000 7f74652c3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:49:18.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.218+0000 7f74652c3640 1 -- 172.21.15.42:0/359767615 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f74600a0ea0 msgr2=0x7f74600a3d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:18.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.218+0000 7f74652c3640 1 --2- 172.21.15.42:0/359767615 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f74600a0ea0 0x7f74600a3d70 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f74540097c0 tx=0x7f745401c360 comp rx=0 tx=0).stop 2024-09-16T03:49:18.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.218+0000 7f74652c3640 1 -- 172.21.15.42:0/359767615 shutdown_connections 2024-09-16T03:49:18.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.218+0000 7f74652c3640 1 --2- 172.21.15.42:0/359767615 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f74600a0ea0 0x7f74600a3d70 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:18.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.218+0000 7f74652c3640 1 --2- 172.21.15.42:0/359767615 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f74600a3450 0x7f74600a3830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:18.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.218+0000 7f74652c3640 1 -- 172.21.15.42:0/359767615 >> 172.21.15.42:0/359767615 conn(0x7f74600756e0 msgr2=0x7f7460075af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:49:18.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.218+0000 7f74652c3640 1 -- 172.21.15.42:0/359767615 shutdown_connections 2024-09-16T03:49:18.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.218+0000 7f74652c3640 1 -- 172.21.15.42:0/359767615 wait complete. 2024-09-16T03:49:18.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.222+0000 7f74652c3640 10 monclient: build_initial_monmap 2024-09-16T03:49:18.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.222+0000 7f74652c3640 10 monclient: monmap: 2024-09-16T03:49:18.235 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:49:18.235 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:49:18.235 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:49:18.235 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:49:18.235 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:49:18.235 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:49:18.235 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:49:18.235 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:49:18.235 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:49:18.235 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:49:18.235 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.222+0000 7f74652c3640 1 librados: starting msgr at 2024-09-16T03:49:18.235 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.222+0000 7f74652c3640 1 librados: starting objecter 2024-09-16T03:49:18.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.222+0000 7f74652c3640 1 Processor -- start 2024-09-16T03:49:18.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.222+0000 7f74652c3640 1 -- start start 2024-09-16T03:49:18.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.222+0000 7f74652c3640 1 librados: setting wanted keys 2024-09-16T03:49:18.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.222+0000 7f74652c3640 1 librados: calling monclient init 2024-09-16T03:49:18.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.222+0000 7f74652c3640 10 monclient: init 2024-09-16T03:49:18.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.222+0000 7f74652c3640 10 monclient: _reopen_session rank -1 2024-09-16T03:49:18.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.222+0000 7f74652c3640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:49:18.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.222+0000 7f74652c3640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f74600a0ea0 0x7f746013df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:18.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.222+0000 7f74652c3640 10 monclient(hunting): picked mon.noname-c con 0x7f74600a0ea0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:49:18.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.222+0000 7f74652c3640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f74600a3450 0x7f7460136270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:18.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.222+0000 7f74652c3640 10 monclient(hunting): picked mon.noname-a con 0x7f74600a3450 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:49:18.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.222+0000 7f74652c3640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7460137220 0x7f746011e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:18.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.222+0000 7f74652c3640 10 monclient(hunting): picked mon.noname-b con 0x7f7460137220 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:49:18.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.222+0000 7f74652c3640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:18.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.222+0000 7f74652c3640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f7460142030 con 0x7f74600a3450 2024-09-16T03:49:18.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.222+0000 7f74652c3640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:18.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.222+0000 7f74652c3640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f7460141eb0 con 0x7f7460137220 2024-09-16T03:49:18.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.222+0000 7f74652c3640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:18.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.222+0000 7f74652c3640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f74601421b0 con 0x7f74600a0ea0 2024-09-16T03:49:18.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.222+0000 7f74652c3640 10 monclient(hunting): _renew_subs 2024-09-16T03:49:18.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.222+0000 7f74652c3640 10 monclient(hunting): authenticate will time out at 1405.193202s 2024-09-16T03:49:18.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.222+0000 7f745effd640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f74600a0ea0 0x7f746013df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:18.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.222+0000 7f745e7fc640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f74600a3450 0x7f7460136270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:18.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.222+0000 7f745effd640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f74600a0ea0 0x7f746013df70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:33308/0 (socket says 172.21.15.42:33308) 2024-09-16T03:49:18.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.222+0000 7f745effd640 1 -- 172.21.15.42:0/4014478153 learned_addr learned my addr 172.21.15.42:0/4014478153 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:49:18.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.222+0000 7f745effd640 10 monclient(hunting): get_auth_request con 0x7f74600a0ea0 auth_method 0 2024-09-16T03:49:18.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.222+0000 7f745effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:18.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.222+0000 7f745effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:18.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.222+0000 7f745effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:18.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.222+0000 7f745e7fc640 10 monclient(hunting): get_auth_request con 0x7f74600a3450 auth_method 0 2024-09-16T03:49:18.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.222+0000 7f745e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:18.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.222+0000 7f745e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:18.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.222+0000 7f745e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:18.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.222+0000 7f745e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:18.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.222+0000 7f745e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:18.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.222+0000 7f745e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:18.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.222+0000 7f745effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:18.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.222+0000 7f745effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:18.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.222+0000 7f745effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:18.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.222+0000 7f745f7fe640 1 --2- 172.21.15.42:0/4014478153 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7460137220 0x7f746011e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:18.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.222+0000 7f745f7fe640 10 monclient(hunting): get_auth_request con 0x7f7460137220 auth_method 0 2024-09-16T03:49:18.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.222+0000 7f745f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:18.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.222+0000 7f745f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:18.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.222+0000 7f745f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:18.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.222+0000 7f745effd640 10 monclient(hunting): handle_auth_done global_id 784190 payload 824 2024-09-16T03:49:18.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.222+0000 7f745effd640 1 -- 172.21.15.42:0/4014478153 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7460137220 msgr2=0x7f746011e090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:18.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.222+0000 7f745effd640 1 --2- 172.21.15.42:0/4014478153 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7460137220 0x7f746011e090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:18.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.222+0000 7f745effd640 1 -- 172.21.15.42:0/4014478153 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f74600a3450 msgr2=0x7f7460136270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:18.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.222+0000 7f745effd640 1 --2- 172.21.15.42:0/4014478153 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f74600a3450 0x7f7460136270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:18.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.222+0000 7f745effd640 10 monclient: _finish_hunting 0 2024-09-16T03:49:18.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.222+0000 7f745effd640 1 monclient: found mon.noname-c 2024-09-16T03:49:18.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.222+0000 7f745effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:49:18.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.222+0000 7f745effd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:49:18.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.222+0000 7f745effd640 1 -- 172.21.15.42:0/4014478153 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f746013ec00 con 0x7f74600a0ea0 2024-09-16T03:49:18.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.222+0000 7f745e7fc640 1 --2- 172.21.15.42:0/4014478153 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f74600a3450 0x7f7460136270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:49:18.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.222+0000 7f745effd640 1 --2- 172.21.15.42:0/4014478153 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f74600a0ea0 0x7f746013df70 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f745000e930 tx=0x7f745000edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:18.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.222+0000 7f74477fe640 10 client.?.objecter ms_handle_connect 0x7f74600a0ea0 2024-09-16T03:49:18.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.222+0000 7f74477fe640 10 client.?.objecter resend_mon_ops 2024-09-16T03:49:18.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.222+0000 7f74477fe640 1 -- 172.21.15.42:0/4014478153 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7450017070 con 0x7f74600a0ea0 2024-09-16T03:49:18.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.222+0000 7f74477fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:18.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.222+0000 7f74477fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:49:18.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.222+0000 7f74477fe640 10 monclient: dump: 2024-09-16T03:49:18.240 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:49:18.240 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:49:18.240 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:18.240 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:18.241 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:18.241 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:49:18.241 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:49:18.241 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:49:18.241 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:49:18.241 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:49:18.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.222+0000 7f74477fe640 10 monclient: _finish_auth 0 2024-09-16T03:49:18.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.222+0000 7f74477fe640 10 monclient: _check_auth_tickets 2024-09-16T03:49:18.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.222+0000 7f74477fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:49:18.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.222+0000 7f74477fe640 1 -- 172.21.15.42:0/4014478153 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f745000ca00 con 0x7f74600a0ea0 2024-09-16T03:49:18.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.222+0000 7f74652c3640 5 monclient: authenticate success, global_id 784190 2024-09-16T03:49:18.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.222+0000 7f74652c3640 10 monclient: _renew_subs 2024-09-16T03:49:18.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.222+0000 7f74652c3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:49:18.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.222+0000 7f74652c3640 1 -- 172.21.15.42:0/4014478153 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7460120c00 con 0x7f74600a0ea0 2024-09-16T03:49:18.242 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.222+0000 7f74477fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:49:18.242 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.222+0000 7f74477fe640 1 -- 172.21.15.42:0/4014478153 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f745000cdb0 con 0x7f74600a0ea0 2024-09-16T03:49:18.242 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.222+0000 7f74477fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:18.242 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.222+0000 7f74477fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:49:18.242 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.222+0000 7f74477fe640 10 monclient: dump: 2024-09-16T03:49:18.242 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:49:18.242 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:49:18.242 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:18.242 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:18.242 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:18.242 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:49:18.242 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:49:18.242 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:49:18.242 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:49:18.243 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:49:18.243 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.222+0000 7f745d7fa640 4 set_mon_vals no callback set 2024-09-16T03:49:18.243 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.222+0000 7f74652c3640 10 client.784190.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:49:18.243 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.222+0000 7f74652c3640 10 monclient: _renew_subs 2024-09-16T03:49:18.243 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.222+0000 7f74652c3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:49:18.243 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.222+0000 7f74652c3640 1 -- 172.21.15.42:0/4014478153 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f7460073b40 con 0x7f74600a0ea0 2024-09-16T03:49:18.243 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.222+0000 7f74652c3640 1 librados: init done 2024-09-16T03:49:18.243 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.222+0000 7f74652c3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:49:18.243 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.222+0000 7f74652c3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:49:18.243 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.222+0000 7f74652c3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:49:18.243 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.222+0000 7f74652c3640 1 -- 172.21.15.42:0/4014478153 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7460137630 con 0x7f74600a0ea0 2024-09-16T03:49:18.243 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.222+0000 7f74477fe640 1 -- 172.21.15.42:0/4014478153 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f745001a070 con 0x7f74600a0ea0 2024-09-16T03:49:18.243 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.222+0000 7f74477fe640 1 --2- 172.21.15.42:0/4014478153 >> v2:172.21.15.154:6812/254845342 conn(0x7f744c037810 0x7f744c039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:18.243 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.222+0000 7f74477fe640 1 -- 172.21.15.42:0/4014478153 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(239..239 src has 1..239) ==== 6556+0+0 (secure 0 0 0) 0x7f7450049460 con 0x7f74600a0ea0 2024-09-16T03:49:18.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.222+0000 7f74477fe640 10 client.784190.objecter ms_dispatch 0x7f7460000ba0 osd_map(239..239 src has 1..239) 2024-09-16T03:49:18.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.222+0000 7f74477fe640 3 client.784190.objecter handle_osd_map got epochs [239,239] > 0 2024-09-16T03:49:18.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.222+0000 7f74477fe640 3 client.784190.objecter handle_osd_map decoding full epoch 239 2024-09-16T03:49:18.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.222+0000 7f74477fe640 20 client.784190.objecter dump_active .. 0 homeless 2024-09-16T03:49:18.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.222+0000 7f745e7fc640 1 --2- 172.21.15.42:0/4014478153 >> v2:172.21.15.154:6812/254845342 conn(0x7f744c037810 0x7f744c039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:18.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.222+0000 7f745e7fc640 10 monclient: get_auth_request con 0x7f744c037810 auth_method 0 2024-09-16T03:49:18.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.226+0000 7f74477fe640 10 client.784190.objecter ms_handle_connect 0x7f744c037810 2024-09-16T03:49:18.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.226+0000 7f745e7fc640 1 --2- 172.21.15.42:0/4014478153 >> v2:172.21.15.154:6812/254845342 conn(0x7f744c037810 0x7f744c039cd0 secure :-1 s=READY pgs=3507 cs=0 l=1 rev1=1 crypto rx=0x7f74540ad000 tx=0x7f74540a6000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:18.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.230+0000 7f74477fe640 1 -- 172.21.15.42:0/4014478153 <== mon.2 v2:172.21.15.42:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f7450011da0 con 0x7f74600a0ea0 2024-09-16T03:49:18.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.230+0000 7f74477fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:49:18.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.230+0000 7f74477fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:49:18.489 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.486+0000 7f74652c3640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:49:18.490 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.486+0000 7f74652c3640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:49:18.490 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.486+0000 7f74652c3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:49:18.490 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.486+0000 7f74652c3640 1 -- 172.21.15.42:0/4014478153 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f74600008e0 con 0x7f74600a0ea0 2024-09-16T03:49:18.490 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.486+0000 7f74477fe640 1 -- 172.21.15.42:0/4014478153 <== mon.2 v2:172.21.15.42:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1555 (secure 0 0 0) 0x7f7450013470 con 0x7f74600a0ea0 2024-09-16T03:49:18.490 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.486+0000 7f74477fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:49:18.491 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.486+0000 7f74477fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:49:18.491 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":64,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":23,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:49:18.494 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.490+0000 7f74652c3640 10 librados: watch_flush enter 2024-09-16T03:49:18.494 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.490+0000 7f74652c3640 10 librados: watch_flush exit 2024-09-16T03:49:18.494 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.490+0000 7f74652c3640 20 client.784190.objecter shutdown clearing up homeless session... 2024-09-16T03:49:18.494 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.490+0000 7f74652c3640 10 client.784190.objecter successfully canceled tick 2024-09-16T03:49:18.494 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.490+0000 7f74652c3640 1 -- 172.21.15.42:0/4014478153 >> v2:172.21.15.154:6812/254845342 conn(0x7f744c037810 msgr2=0x7f744c039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:18.495 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.490+0000 7f74652c3640 1 --2- 172.21.15.42:0/4014478153 >> v2:172.21.15.154:6812/254845342 conn(0x7f744c037810 0x7f744c039cd0 secure :-1 s=READY pgs=3507 cs=0 l=1 rev1=1 crypto rx=0x7f74540ad000 tx=0x7f74540a6000 comp rx=0 tx=0).stop 2024-09-16T03:49:18.495 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.490+0000 7f74652c3640 10 monclient: shutdown 2024-09-16T03:49:18.495 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.490+0000 7f74652c3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:49:18.495 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.490+0000 7f74652c3640 1 -- 172.21.15.42:0/4014478153 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f74600a0ea0 msgr2=0x7f746013df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:18.495 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.490+0000 7f74652c3640 1 --2- 172.21.15.42:0/4014478153 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f74600a0ea0 0x7f746013df70 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f745000e930 tx=0x7f745000edf0 comp rx=0 tx=0).stop 2024-09-16T03:49:18.495 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.490+0000 7f74652c3640 1 -- 172.21.15.42:0/4014478153 shutdown_connections 2024-09-16T03:49:18.495 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.490+0000 7f74652c3640 1 --2- 172.21.15.42:0/4014478153 >> v2:172.21.15.154:6812/254845342 conn(0x7f744c037810 0x7f744c039cd0 unknown :-1 s=CLOSED pgs=3507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:18.495 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.490+0000 7f74652c3640 1 --2- 172.21.15.42:0/4014478153 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7460137220 0x7f746011e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:18.495 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.490+0000 7f74652c3640 1 --2- 172.21.15.42:0/4014478153 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f74600a3450 0x7f7460136270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:18.495 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.490+0000 7f74652c3640 1 --2- 172.21.15.42:0/4014478153 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f74600a0ea0 0x7f746013df70 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:18.495 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.490+0000 7f74652c3640 1 -- 172.21.15.42:0/4014478153 >> 172.21.15.42:0/4014478153 conn(0x7f74600756e0 msgr2=0x7f74600a6380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:49:18.495 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.490+0000 7f74652c3640 1 -- 172.21.15.42:0/4014478153 shutdown_connections 2024-09-16T03:49:18.495 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.490+0000 7f74652c3640 1 -- 172.21.15.42:0/4014478153 wait complete. 2024-09-16T03:49:18.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:18.490+0000 7f74652c3640 1 librados: shutdown 2024-09-16T03:49:19.013 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetRead 2024-09-16T03:49:19.013 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: hmm, no HitSet yet 2024-09-16T03:49:19.013 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: ok, hit_set contains 269:602f83fe:::foo:head 2024-09-16T03:49:19.013 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetRead (9000 ms) 2024-09-16T03:49:19.013 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetWrite 2024-09-16T03:49:19.013 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: pg_num = 32 2024-09-16T03:49:19.013 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: pg 0 ls 1726458559,0 2024-09-16T03:49:19.013 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: pg 1 ls 1726458559,0 2024-09-16T03:49:19.014 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: pg 2 ls 1726458559,0 2024-09-16T03:49:19.014 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: pg 3 ls 1726458559,0 2024-09-16T03:49:19.014 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: pg 4 ls 1726458559,0 2024-09-16T03:49:19.014 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: pg 5 ls 1726458559,0 2024-09-16T03:49:19.014 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: pg 6 ls 1726458559,0 2024-09-16T03:49:19.014 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: pg 7 ls 1726458559,0 2024-09-16T03:49:19.014 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: pg 8 ls 1726458559,0 2024-09-16T03:49:19.014 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: pg 9 ls 1726458559,0 2024-09-16T03:49:19.014 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: pg 10 ls 1726458559,0 2024-09-16T03:49:19.014 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: pg 11 ls 1726458559,0 2024-09-16T03:49:19.014 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: pg 12 ls 1726458559,0 2024-09-16T03:49:19.014 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: pg 13 ls 1726458559,0 2024-09-16T03:49:19.014 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: pg 14 ls 1726458559,0 2024-09-16T03:49:19.015 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: pg 15 ls 1726458559,0 2024-09-16T03:49:19.015 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: pg 16 ls 1726458559,0 2024-09-16T03:49:19.015 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: pg 17 ls 1726458559,0 2024-09-16T03:49:19.015 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: pg 18 ls 1726458559,0 2024-09-16T03:49:19.015 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: pg 19 ls 1726458559,0 2024-09-16T03:49:19.015 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: pg 20 ls 1726458559,0 2024-09-16T03:49:19.015 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: pg 21 ls 1726458559,0 2024-09-16T03:49:19.015 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: pg 22 ls 1726458559,0 2024-09-16T03:49:19.015 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: pg 23 ls 1726458559,0 2024-09-16T03:49:19.015 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: pg 24 ls 1726458559,0 2024-09-16T03:49:19.015 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: pg 25 ls 1726458559,0 2024-09-16T03:49:19.015 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: pg 26 ls 1726458559,0 2024-09-16T03:49:19.015 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: pg 27 ls 1726458559,0 2024-09-16T03:49:19.016 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: pg 28 ls 1726458559,0 2024-09-16T03:49:19.016 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: pg 29 ls 1726458559,0 2024-09-16T03:49:19.016 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: pg 30 ls 1726458559,0 2024-09-16T03:49:19.016 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: pg 31 ls 1726458559,0 2024-09-16T03:49:19.016 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: pg_num = 32 2024-09-16T03:49:19.016 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:6cac518f:::0:head 2024-09-16T03:49:19.016 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:02547ec2:::1:head 2024-09-16T03:49:19.016 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:f905c69b:::2:head 2024-09-16T03:49:19.016 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:cfc208b3:::3:head 2024-09-16T03:49:19.016 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:d83876eb:::4:head 2024-09-16T03:49:19.016 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:b29083e3:::5:head 2024-09-16T03:49:19.016 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:c4fdafeb:::6:head 2024-09-16T03:49:19.016 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:5c6b0b28:::7:head 2024-09-16T03:49:19.017 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:bd63b0f1:::8:head 2024-09-16T03:49:19.017 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:e960b815:::9:head 2024-09-16T03:49:19.017 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:52ea6a34:::10:head 2024-09-16T03:49:19.017 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:89d3ae78:::11:head 2024-09-16T03:49:19.017 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:de5d7c5f:::12:head 2024-09-16T03:49:19.017 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:566253c9:::13:head 2024-09-16T03:49:19.017 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:62a1935d:::14:head 2024-09-16T03:49:19.017 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:863748b0:::15:head 2024-09-16T03:49:19.017 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:3958e169:::16:head 2024-09-16T03:49:19.017 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:4d4dabf9:::17:head 2024-09-16T03:49:19.017 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:8391935d:::18:head 2024-09-16T03:49:19.017 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:28883081:::19:head 2024-09-16T03:49:19.017 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:69259c59:::20:head 2024-09-16T03:49:19.017 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:4bdb80b7:::21:head 2024-09-16T03:49:19.018 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:a11c5d71:::22:head 2024-09-16T03:49:19.018 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:271af37b:::23:head 2024-09-16T03:49:19.018 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:95b121be:::24:head 2024-09-16T03:49:19.018 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:58d1031b:::25:head 2024-09-16T03:49:19.018 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:0a050783:::26:head 2024-09-16T03:49:19.018 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:c709704c:::27:head 2024-09-16T03:49:19.018 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:cbe56eaf:::28:head 2024-09-16T03:49:19.018 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:86b4b162:::29:head 2024-09-16T03:49:19.018 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:70d89383:::30:head 2024-09-16T03:49:19.018 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:dd450c7c:::31:head 2024-09-16T03:49:19.018 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:6d5729b1:::32:head 2024-09-16T03:49:19.019 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:c388f3fb:::33:head 2024-09-16T03:49:19.019 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:56cfea31:::34:head 2024-09-16T03:49:19.019 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:9dbc1bf7:::35:head 2024-09-16T03:49:19.019 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:40b74ccd:::36:head 2024-09-16T03:49:19.019 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:4d5aaf42:::37:head 2024-09-16T03:49:19.019 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:920f362c:::38:head 2024-09-16T03:49:19.019 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:6cc53222:::39:head 2024-09-16T03:49:19.019 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:9cad833f:::40:head 2024-09-16T03:49:19.019 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:1ea84d41:::41:head 2024-09-16T03:49:19.019 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:c4480ef6:::42:head 2024-09-16T03:49:19.019 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:a694361e:::43:head 2024-09-16T03:49:19.019 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:d1bd33e9:::44:head 2024-09-16T03:49:19.020 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:ddc2cd5d:::45:head 2024-09-16T03:49:19.020 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:2b782207:::46:head 2024-09-16T03:49:19.020 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:7b187fca:::47:head 2024-09-16T03:49:19.020 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:90ecdf6f:::48:head 2024-09-16T03:49:19.020 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:a5ed95fe:::49:head 2024-09-16T03:49:19.020 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:ea0eaa55:::50:head 2024-09-16T03:49:19.020 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:f33ef17b:::51:head 2024-09-16T03:49:19.020 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:a0d1b2f6:::52:head 2024-09-16T03:49:19.020 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:60c5229e:::53:head 2024-09-16T03:49:19.020 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:edcbc575:::54:head 2024-09-16T03:49:19.020 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:102cf253:::55:head 2024-09-16T03:49:19.020 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:efb7fb0b:::56:head 2024-09-16T03:49:19.020 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:50d0a326:::57:head 2024-09-16T03:49:19.020 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:d4dc5daf:::58:head 2024-09-16T03:49:19.021 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:3a130462:::59:head 2024-09-16T03:49:19.021 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:ec87ed71:::60:head 2024-09-16T03:49:19.021 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:d5bc9454:::61:head 2024-09-16T03:49:19.021 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:3ddfe313:::62:head 2024-09-16T03:49:19.021 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:7c2816b9:::63:head 2024-09-16T03:49:19.021 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:47e00e4d:::64:head 2024-09-16T03:49:19.021 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:c6410c18:::65:head 2024-09-16T03:49:19.021 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:b48ed237:::66:head 2024-09-16T03:49:19.021 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:cd63ad31:::67:head 2024-09-16T03:49:19.021 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:b179e92b:::68:head 2024-09-16T03:49:19.021 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:0d9f741a:::69:head 2024-09-16T03:49:19.021 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:6d3352ae:::70:head 2024-09-16T03:49:19.021 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:c6d5c19e:::71:head 2024-09-16T03:49:19.021 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:bc4729c3:::72:head 2024-09-16T03:49:19.022 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:77e930b9:::73:head 2024-09-16T03:49:19.022 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:0abeecfd:::74:head 2024-09-16T03:49:19.022 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:b7c37e15:::75:head 2024-09-16T03:49:19.022 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:b6378398:::76:head 2024-09-16T03:49:19.022 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:02bd68de:::77:head 2024-09-16T03:49:19.022 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:cc795d2d:::78:head 2024-09-16T03:49:19.022 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:630d4fea:::79:head 2024-09-16T03:49:19.022 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:e0d29ef5:::80:head 2024-09-16T03:49:19.022 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:fd6f13d2:::81:head 2024-09-16T03:49:19.022 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:606461d5:::82:head 2024-09-16T03:49:19.022 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:eadbdc43:::83:head 2024-09-16T03:49:19.022 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:8761d0bb:::84:head 2024-09-16T03:49:19.022 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:9ef0186f:::85:head 2024-09-16T03:49:19.022 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:e0d41294:::86:head 2024-09-16T03:49:19.023 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:961de695:::87:head 2024-09-16T03:49:19.023 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:1423148f:::88:head 2024-09-16T03:49:19.023 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:633a8fa2:::89:head 2024-09-16T03:49:19.023 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:a8653809:::90:head 2024-09-16T03:49:19.023 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:3dac8b33:::91:head 2024-09-16T03:49:19.023 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:35aad435:::92:head 2024-09-16T03:49:19.023 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:f6dcc343:::93:head 2024-09-16T03:49:19.023 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:dbbdad87:::94:head 2024-09-16T03:49:19.023 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:1cb48ce0:::95:head 2024-09-16T03:49:19.023 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:03cd461c:::96:head 2024-09-16T03:49:19.023 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:17a4ea99:::97:head 2024-09-16T03:49:19.023 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:9993c9a7:::98:head 2024-09-16T03:49:19.023 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:6394211c:::99:head 2024-09-16T03:49:19.024 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:94c7ae57:::100:head 2024-09-16T03:49:19.024 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:6fdee5bb:::101:head 2024-09-16T03:49:19.024 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:9a477fd1:::102:head 2024-09-16T03:49:19.024 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:eb850916:::103:head 2024-09-16T03:49:19.024 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:affc56b9:::104:head 2024-09-16T03:49:19.024 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:b42dc814:::105:head 2024-09-16T03:49:19.024 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:f319f8f0:::106:head 2024-09-16T03:49:19.024 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:9a40b9de:::107:head 2024-09-16T03:49:19.024 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:8b524f28:::108:head 2024-09-16T03:49:19.024 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:e3de589f:::109:head 2024-09-16T03:49:19.024 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:90f90a5b:::110:head 2024-09-16T03:49:19.024 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:a7b4f1d7:::111:head 2024-09-16T03:49:19.024 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:af51766e:::112:head 2024-09-16T03:49:19.025 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:b6f90bd1:::113:head 2024-09-16T03:49:19.025 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:e0261208:::114:head 2024-09-16T03:49:19.025 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:c9569ef7:::115:head 2024-09-16T03:49:19.025 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:61bebe50:::116:head 2024-09-16T03:49:19.025 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:fe93412b:::117:head 2024-09-16T03:49:19.025 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:d3d38bee:::118:head 2024-09-16T03:49:19.025 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:3100ba0c:::119:head 2024-09-16T03:49:19.025 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:d0560ada:::120:head 2024-09-16T03:49:19.025 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:f0ea8b35:::121:head 2024-09-16T03:49:19.025 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:766f231a:::122:head 2024-09-16T03:49:19.025 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:a07a2582:::123:head 2024-09-16T03:49:19.025 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:bd7c6b3a:::124:head 2024-09-16T03:49:19.025 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:fb2ddaff:::125:head 2024-09-16T03:49:19.025 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:4408e1fe:::126:head 2024-09-16T03:49:19.026 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:ee1df7a7:::127:head 2024-09-16T03:49:19.026 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:c3002909:::128:head 2024-09-16T03:49:19.026 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:4f48ffa9:::129:head 2024-09-16T03:49:19.026 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:edf38733:::130:head 2024-09-16T03:49:19.026 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:c08425c0:::131:head 2024-09-16T03:49:19.026 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:5f902d98:::132:head 2024-09-16T03:49:19.026 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:41ea2c93:::133:head 2024-09-16T03:49:19.026 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:813cee13:::134:head 2024-09-16T03:49:19.026 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:0131818d:::135:head 2024-09-16T03:49:19.026 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:26ba5a85:::136:head 2024-09-16T03:49:19.026 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:381b8a5a:::137:head 2024-09-16T03:49:19.026 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:28797e47:::138:head 2024-09-16T03:49:19.026 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:bfca7f22:::139:head 2024-09-16T03:49:19.026 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:36807075:::140:head 2024-09-16T03:49:19.027 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:80b03975:::141:head 2024-09-16T03:49:19.027 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:5c15709b:::142:head 2024-09-16T03:49:19.027 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:f39ea15e:::143:head 2024-09-16T03:49:19.027 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:ea992956:::144:head 2024-09-16T03:49:19.027 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:48887b1c:::145:head 2024-09-16T03:49:19.027 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:9f24a9dd:::146:head 2024-09-16T03:49:19.027 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:987f100b:::147:head 2024-09-16T03:49:19.027 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:d2dd3581:::148:head 2024-09-16T03:49:19.027 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:7fed1808:::149:head 2024-09-16T03:49:19.027 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:c80b70e9:::150:head 2024-09-16T03:49:19.027 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:85ed90f9:::151:head 2024-09-16T03:49:19.027 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:36428b24:::152:head 2024-09-16T03:49:19.027 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:d044c34a:::153:head 2024-09-16T03:49:19.027 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:7c18bf58:::154:head 2024-09-16T03:49:19.028 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:d1c21232:::155:head 2024-09-16T03:49:19.028 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:a7a3c575:::156:head 2024-09-16T03:49:19.028 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:87da0633:::157:head 2024-09-16T03:49:19.028 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:d5ac3822:::158:head 2024-09-16T03:49:19.028 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:3f20522d:::159:head 2024-09-16T03:49:19.028 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:6ca26563:::160:head 2024-09-16T03:49:19.028 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:532ce135:::161:head 2024-09-16T03:49:19.028 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:c78863e6:::162:head 2024-09-16T03:49:21.511 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:49:21.647 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.642+0000 7f97ba619640 10 monclient: get_monmap_and_config 2024-09-16T03:49:21.647 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.642+0000 7f97ba619640 10 monclient: build_initial_monmap 2024-09-16T03:49:21.647 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.642+0000 7f97ba619640 10 monclient: monmap: 2024-09-16T03:49:21.647 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:49:21.647 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:49:21.647 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:49:21.646100+0000 2024-09-16T03:49:21.648 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:49:21.646100+0000 2024-09-16T03:49:21.648 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:49:21.648 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:49:21.648 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:49:21.648 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:49:21.648 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:49:21.648 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:49:21.648 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:49:21.648 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:49:21.651 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.642+0000 7f97ba619640 1 Processor -- start 2024-09-16T03:49:21.651 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.642+0000 7f97ba619640 1 -- start start 2024-09-16T03:49:21.651 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.642+0000 7f97ba619640 10 monclient: init 2024-09-16T03:49:21.652 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.642+0000 7f97ba619640 10 monclient: _reopen_session rank -1 2024-09-16T03:49:21.652 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.642+0000 7f97ba619640 10 monclient: _add_conns ranks=[3,2,0,1,4] 2024-09-16T03:49:21.652 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.642+0000 7f97ba619640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f97b40a1330 0x7f97b40a1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:21.652 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.642+0000 7f97ba619640 10 monclient(hunting): picked mon.noname-a con 0x7f97b40a1330 addr v2:172.21.15.42:3300/0 2024-09-16T03:49:21.652 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.642+0000 7f97ba619640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f97b406dac0 0x7f97b40a1c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:21.652 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.642+0000 7f97ba619640 10 monclient(hunting): picked mon.noname-b con 0x7f97b406dac0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:49:21.652 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.642+0000 7f97ba619640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f97b4135100 addr v1:172.21.15.42:6789/0 2024-09-16T03:49:21.652 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.642+0000 7f97ba619640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f97b409c7d0 con 0x7f97b4135100 2024-09-16T03:49:21.652 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.642+0000 7f97ba619640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:21.652 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.642+0000 7f97ba619640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f97b409c090 con 0x7f97b40a1330 2024-09-16T03:49:21.652 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.642+0000 7f97ba619640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:21.653 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.642+0000 7f97ba619640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f97b409f610 con 0x7f97b406dac0 2024-09-16T03:49:21.653 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.642+0000 7f97ba619640 10 monclient(hunting): _renew_subs 2024-09-16T03:49:21.653 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.642+0000 7f97ba619640 10 monclient(hunting): authenticate will time out at 1408.615475s 2024-09-16T03:49:21.653 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.642+0000 7f97b37fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f97b406dac0 0x7f97b40a1c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:21.653 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.642+0000 7f97b37fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f97b406dac0 0x7f97b40a1c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:33318/0 (socket says 172.21.15.42:33318) 2024-09-16T03:49:21.653 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.642+0000 7f97b37fe640 1 -- 172.21.15.42:0/1604406768 learned_addr learned my addr 172.21.15.42:0/1604406768 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:49:21.653 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.642+0000 7f97b37fe640 10 monclient(hunting): get_auth_request con 0x7f97b406dac0 auth_method 0 2024-09-16T03:49:21.653 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.642+0000 7f97b37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:21.653 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.642+0000 7f97b37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:21.653 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.642+0000 7f97b37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:21.653 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.642+0000 7f97b3fff640 1 --2- 172.21.15.42:0/1604406768 >> v2:172.21.15.42:3300/0 conn(0x7f97b40a1330 0x7f97b40a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:21.654 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.642+0000 7f97b3fff640 10 monclient(hunting): get_auth_request con 0x7f97b40a1330 auth_method 0 2024-09-16T03:49:21.654 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.642+0000 7f97b3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:21.654 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.642+0000 7f97b3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:21.654 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.642+0000 7f97b3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:21.654 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.642+0000 7f97b37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:21.654 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.642+0000 7f97b37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:21.654 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.642+0000 7f97b37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:21.654 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.642+0000 7f97b3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:21.654 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.642+0000 7f97b3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:21.654 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.642+0000 7f97b3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:21.654 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.646+0000 7f97b37fe640 10 monclient(hunting): handle_auth_done global_id 784199 payload 274 2024-09-16T03:49:21.655 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.646+0000 7f97b37fe640 1 -- 172.21.15.42:0/1604406768 >> v2:172.21.15.42:3300/0 conn(0x7f97b40a1330 msgr2=0x7f97b40a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:49:21.655 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.646+0000 7f97b37fe640 1 --2- 172.21.15.42:0/1604406768 >> v2:172.21.15.42:3300/0 conn(0x7f97b40a1330 0x7f97b40a1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:21.655 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.646+0000 7f97b37fe640 1 -- 172.21.15.42:0/1604406768 >> v1:172.21.15.42:6789/0 conn(0x7f97b4135100 legacy=0x7f97b4135550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:21.655 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.646+0000 7f97b37fe640 10 monclient: _finish_hunting 0 2024-09-16T03:49:21.655 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.646+0000 7f97b37fe640 1 monclient: found mon.noname-b 2024-09-16T03:49:21.655 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.646+0000 7f97b37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:49:21.655 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.646+0000 7f97b37fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T03:49:21.655 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.646+0000 7f97b37fe640 1 -- 172.21.15.42:0/1604406768 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97b409f8f0 con 0x7f97b406dac0 2024-09-16T03:49:21.655 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.646+0000 7f97b3fff640 1 --2- 172.21.15.42:0/1604406768 >> v2:172.21.15.42:3300/0 conn(0x7f97b40a1330 0x7f97b40a1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:49:21.655 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.646+0000 7f97b37fe640 1 --2- 172.21.15.42:0/1604406768 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f97b406dac0 0x7f97b40a1c50 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f97a8009870 tx=0x7f97a801c540 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a08a8147c833a708 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:21.655 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.646+0000 7f97b2ffd640 1 -- 172.21.15.42:0/1604406768 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97a80ab650 con 0x7f97b406dac0 2024-09-16T03:49:21.656 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.646+0000 7f97b2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:21.656 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.646+0000 7f97b2ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:49:21.656 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.646+0000 7f97b2ffd640 10 monclient: dump: 2024-09-16T03:49:21.656 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:49:21.656 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:49:21.656 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:21.656 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:21.656 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:21.656 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:49:21.656 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:49:21.656 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:49:21.657 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:49:21.657 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:49:21.657 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.646+0000 7f97b2ffd640 10 monclient: _finish_auth 0 2024-09-16T03:49:21.657 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.646+0000 7f97b2ffd640 10 monclient: _check_auth_tickets 2024-09-16T03:49:21.657 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.646+0000 7f97b2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:49:21.657 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.646+0000 7f97b2ffd640 1 -- 172.21.15.42:0/1604406768 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f97a80b0070 con 0x7f97b406dac0 2024-09-16T03:49:21.657 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.646+0000 7f97ba619640 5 monclient: authenticate success, global_id 784199 2024-09-16T03:49:21.657 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.646+0000 7f97ba619640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:49:21.657 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.646+0000 7f97b2ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:49:21.657 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.646+0000 7f97ba619640 10 monclient: get_monmap_and_config success 2024-09-16T03:49:21.657 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.646+0000 7f97ba619640 4 set_mon_vals no callback set 2024-09-16T03:49:21.658 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.646+0000 7f97b2ffd640 1 -- 172.21.15.42:0/1604406768 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97a80aba10 con 0x7f97b406dac0 2024-09-16T03:49:21.658 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.646+0000 7f97b2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:21.658 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.646+0000 7f97b2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:49:21.658 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.646+0000 7f97b2ffd640 10 monclient: dump: 2024-09-16T03:49:21.658 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:49:21.658 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:49:21.658 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:21.658 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:21.658 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:21.658 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:49:21.658 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:49:21.659 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:49:21.659 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:49:21.659 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:49:21.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.646+0000 7f97ba619640 10 monclient: shutdown 2024-09-16T03:49:21.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.646+0000 7f97ba619640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:49:21.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.646+0000 7f97ba619640 1 -- 172.21.15.42:0/1604406768 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f97b406dac0 msgr2=0x7f97b40a1c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:21.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.646+0000 7f97ba619640 1 --2- 172.21.15.42:0/1604406768 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f97b406dac0 0x7f97b40a1c50 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f97a8009870 tx=0x7f97a801c540 comp rx=0 tx=0).stop 2024-09-16T03:49:21.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.646+0000 7f97ba619640 1 -- 172.21.15.42:0/1604406768 shutdown_connections 2024-09-16T03:49:21.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.646+0000 7f97ba619640 1 --2- 172.21.15.42:0/1604406768 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f97b406dac0 0x7f97b40a1c50 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:21.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.646+0000 7f97ba619640 1 --2- 172.21.15.42:0/1604406768 >> v2:172.21.15.42:3300/0 conn(0x7f97b40a1330 0x7f97b40a1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:21.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.646+0000 7f97ba619640 1 -- 172.21.15.42:0/1604406768 >> 172.21.15.42:0/1604406768 conn(0x7f97b40a2420 msgr2=0x7f97b40a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:49:21.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.646+0000 7f97ba619640 1 -- 172.21.15.42:0/1604406768 shutdown_connections 2024-09-16T03:49:21.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.646+0000 7f97ba619640 1 -- 172.21.15.42:0/1604406768 wait complete. 2024-09-16T03:49:21.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.646+0000 7f97ba619640 10 monclient: build_initial_monmap 2024-09-16T03:49:21.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.646+0000 7f97ba619640 10 monclient: monmap: 2024-09-16T03:49:21.665 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:49:21.665 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:49:21.666 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:49:21.666 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:49:21.666 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:49:21.666 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:49:21.666 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:49:21.666 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:49:21.666 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:49:21.666 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:49:21.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.650+0000 7f97ba619640 1 librados: starting msgr at 2024-09-16T03:49:21.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.650+0000 7f97ba619640 1 librados: starting objecter 2024-09-16T03:49:21.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.650+0000 7f97ba619640 1 Processor -- start 2024-09-16T03:49:21.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.650+0000 7f97ba619640 1 -- start start 2024-09-16T03:49:21.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.650+0000 7f97ba619640 1 librados: setting wanted keys 2024-09-16T03:49:21.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.650+0000 7f97ba619640 1 librados: calling monclient init 2024-09-16T03:49:21.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.650+0000 7f97ba619640 10 monclient: init 2024-09-16T03:49:21.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.650+0000 7f97ba619640 10 monclient: _reopen_session rank -1 2024-09-16T03:49:21.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.650+0000 7f97ba619640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:49:21.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.650+0000 7f97ba619640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f97b40a1330 0x7f97b413b110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:21.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.650+0000 7f97ba619640 10 monclient(hunting): picked mon.noname-c con 0x7f97b40a1330 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:49:21.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.650+0000 7f97ba619640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f97b4135100 0x7f97b4134530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:21.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.650+0000 7f97ba619640 10 monclient(hunting): picked mon.noname-b con 0x7f97b4135100 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:49:21.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.650+0000 7f97ba619640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f97b4134a70 0x7f97b409cb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:21.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.650+0000 7f97ba619640 10 monclient(hunting): picked mon.noname-a con 0x7f97b4134a70 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:49:21.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.650+0000 7f97ba619640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:21.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.650+0000 7f97ba619640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f97b413fd00 con 0x7f97b4134a70 2024-09-16T03:49:21.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.650+0000 7f97ba619640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:21.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.650+0000 7f97ba619640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f97b413fb80 con 0x7f97b4135100 2024-09-16T03:49:21.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.650+0000 7f97ba619640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:21.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.650+0000 7f97ba619640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f97b413fe80 con 0x7f97b40a1330 2024-09-16T03:49:21.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.650+0000 7f97ba619640 10 monclient(hunting): _renew_subs 2024-09-16T03:49:21.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.650+0000 7f97ba619640 10 monclient(hunting): authenticate will time out at 1408.620824s 2024-09-16T03:49:21.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.650+0000 7f97b8bb6640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f97b4134a70 0x7f97b409cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:21.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.650+0000 7f97b3fff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f97b40a1330 0x7f97b413b110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:21.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.650+0000 7f97b8bb6640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f97b4134a70 0x7f97b409cb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:47010/0 (socket says 172.21.15.42:47010) 2024-09-16T03:49:21.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.650+0000 7f97b8bb6640 1 -- 172.21.15.42:0/1143432177 learned_addr learned my addr 172.21.15.42:0/1143432177 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:49:21.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.650+0000 7f97b8bb6640 10 monclient(hunting): get_auth_request con 0x7f97b4134a70 auth_method 0 2024-09-16T03:49:21.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.650+0000 7f97b3fff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f97b40a1330 0x7f97b413b110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:33330/0 (socket says 172.21.15.42:33330) 2024-09-16T03:49:21.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.650+0000 7f97b8bb6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:21.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.650+0000 7f97b8bb6640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:21.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.650+0000 7f97b8bb6640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:21.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.650+0000 7f97b3fff640 10 monclient(hunting): get_auth_request con 0x7f97b40a1330 auth_method 0 2024-09-16T03:49:21.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.650+0000 7f97b3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:21.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.650+0000 7f97b3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:21.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.650+0000 7f97b3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:21.670 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.650+0000 7f97b8bb6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:21.670 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.650+0000 7f97b8bb6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:21.670 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.650+0000 7f97b8bb6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:21.670 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.650+0000 7f97b3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:21.670 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.650+0000 7f97b3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:21.670 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.650+0000 7f97b3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:21.670 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.650+0000 7f97b37fe640 1 --2- 172.21.15.42:0/1143432177 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f97b4135100 0x7f97b4134530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:21.670 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.650+0000 7f97b37fe640 10 monclient(hunting): get_auth_request con 0x7f97b4135100 auth_method 0 2024-09-16T03:49:21.670 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.650+0000 7f97b37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:21.670 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.650+0000 7f97b37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:21.670 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.650+0000 7f97b37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:21.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.650+0000 7f97b8bb6640 10 monclient(hunting): handle_auth_done global_id 764619 payload 824 2024-09-16T03:49:21.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.650+0000 7f97b8bb6640 1 -- 172.21.15.42:0/1143432177 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f97b40a1330 msgr2=0x7f97b413b110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:21.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.650+0000 7f97b8bb6640 1 --2- 172.21.15.42:0/1143432177 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f97b40a1330 0x7f97b413b110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:21.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.650+0000 7f97b8bb6640 1 -- 172.21.15.42:0/1143432177 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f97b4135100 msgr2=0x7f97b4134530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:21.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.650+0000 7f97b8bb6640 1 --2- 172.21.15.42:0/1143432177 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f97b4135100 0x7f97b4134530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:21.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.650+0000 7f97b8bb6640 10 monclient: _finish_hunting 0 2024-09-16T03:49:21.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.650+0000 7f97b8bb6640 1 monclient: found mon.noname-a 2024-09-16T03:49:21.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.650+0000 7f97b8bb6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:49:21.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.650+0000 7f97b8bb6640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:49:21.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.650+0000 7f97b8bb6640 1 -- 172.21.15.42:0/1143432177 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97b41cf6f0 con 0x7f97b4134a70 2024-09-16T03:49:21.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.650+0000 7f97b3fff640 1 --2- 172.21.15.42:0/1143432177 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f97b40a1330 0x7f97b413b110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:49:21.672 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.650+0000 7f97b8bb6640 1 --2- 172.21.15.42:0/1143432177 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f97b4134a70 0x7f97b409cb60 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f97ac00c430 tx=0x7f97ac00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:21.672 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.650+0000 7f97b0ff9640 10 client.?.objecter ms_handle_connect 0x7f97b4134a70 2024-09-16T03:49:21.672 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.650+0000 7f97b0ff9640 10 client.?.objecter resend_mon_ops 2024-09-16T03:49:21.672 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.650+0000 7f97b0ff9640 1 -- 172.21.15.42:0/1143432177 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97ac005220 con 0x7f97b4134a70 2024-09-16T03:49:21.672 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.650+0000 7f97b0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:21.672 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.650+0000 7f97b0ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:49:21.672 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.650+0000 7f97b0ff9640 10 monclient: dump: 2024-09-16T03:49:21.672 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:49:21.672 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:49:21.672 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:21.672 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:21.672 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:21.673 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:49:21.673 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:49:21.673 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:49:21.673 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:49:21.673 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:49:21.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.650+0000 7f97b0ff9640 10 monclient: _finish_auth 0 2024-09-16T03:49:21.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.650+0000 7f97b0ff9640 10 monclient: _check_auth_tickets 2024-09-16T03:49:21.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.650+0000 7f97b0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:49:21.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.650+0000 7f97b0ff9640 1 -- 172.21.15.42:0/1143432177 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f97ac0053c0 con 0x7f97b4134a70 2024-09-16T03:49:21.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.650+0000 7f97b0ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T03:49:21.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.650+0000 7f97ba619640 5 monclient: authenticate success, global_id 764619 2024-09-16T03:49:21.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.650+0000 7f97ba619640 10 monclient: _renew_subs 2024-09-16T03:49:21.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.650+0000 7f97b0ff9640 1 -- 172.21.15.42:0/1143432177 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97ac007270 con 0x7f97b4134a70 2024-09-16T03:49:21.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.650+0000 7f97ba619640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:49:21.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.650+0000 7f97ba619640 1 -- 172.21.15.42:0/1143432177 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97b4120920 con 0x7f97b4134a70 2024-09-16T03:49:21.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.650+0000 7f97b0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:21.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.650+0000 7f97b27fc640 4 set_mon_vals no callback set 2024-09-16T03:49:21.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.650+0000 7f97b0ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:49:21.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.650+0000 7f97b0ff9640 10 monclient: dump: 2024-09-16T03:49:21.674 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:49:21.674 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:49:21.674 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:21.675 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:21.675 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:21.675 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:49:21.675 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:49:21.675 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:49:21.675 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:49:21.675 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:49:21.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.650+0000 7f97ba619640 10 client.764619.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:49:21.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.650+0000 7f97ba619640 10 monclient: _renew_subs 2024-09-16T03:49:21.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.650+0000 7f97ba619640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:49:21.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.650+0000 7f97ba619640 1 -- 172.21.15.42:0/1143432177 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97b4120c00 con 0x7f97b4134a70 2024-09-16T03:49:21.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.650+0000 7f97ba619640 1 librados: init done 2024-09-16T03:49:21.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.650+0000 7f97ba619640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:49:21.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.650+0000 7f97ba619640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:49:21.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.650+0000 7f97ba619640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:49:21.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.650+0000 7f97ba619640 1 -- 172.21.15.42:0/1143432177 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97b40719c0 con 0x7f97b4134a70 2024-09-16T03:49:21.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.650+0000 7f97b0ff9640 1 -- 172.21.15.42:0/1143432177 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f97ac019020 con 0x7f97b4134a70 2024-09-16T03:49:21.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.650+0000 7f97b0ff9640 1 --2- 172.21.15.42:0/1143432177 >> v2:172.21.15.154:6812/254845342 conn(0x7f97a0037810 0x7f97a0039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:21.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.650+0000 7f97b0ff9640 1 -- 172.21.15.42:0/1143432177 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(243..243 src has 1..243) ==== 5634+0+0 (secure 0 0 0) 0x7f97ac0481f0 con 0x7f97b4134a70 2024-09-16T03:49:21.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.650+0000 7f97b0ff9640 10 client.764619.objecter ms_dispatch 0x7f97b4000ba0 osd_map(243..243 src has 1..243) 2024-09-16T03:49:21.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.650+0000 7f97b0ff9640 3 client.764619.objecter handle_osd_map got epochs [243,243] > 0 2024-09-16T03:49:21.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.650+0000 7f97b0ff9640 3 client.764619.objecter handle_osd_map decoding full epoch 243 2024-09-16T03:49:21.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.650+0000 7f97b0ff9640 20 client.764619.objecter dump_active .. 0 homeless 2024-09-16T03:49:21.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.650+0000 7f97b3fff640 1 --2- 172.21.15.42:0/1143432177 >> v2:172.21.15.154:6812/254845342 conn(0x7f97a0037810 0x7f97a0039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:21.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.650+0000 7f97b3fff640 10 monclient: get_auth_request con 0x7f97a0037810 auth_method 0 2024-09-16T03:49:21.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.654+0000 7f97b0ff9640 10 client.764619.objecter ms_handle_connect 0x7f97a0037810 2024-09-16T03:49:21.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.654+0000 7f97b3fff640 1 --2- 172.21.15.42:0/1143432177 >> v2:172.21.15.154:6812/254845342 conn(0x7f97a0037810 0x7f97a0039cd0 secure :-1 s=READY pgs=3509 cs=0 l=1 rev1=1 crypto rx=0x7f97a4009fd0 tx=0x7f97a400b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:21.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.654+0000 7f97b0ff9640 1 -- 172.21.15.42:0/1143432177 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f97b41cf6f0 con 0x7f97b4134a70 2024-09-16T03:49:21.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.654+0000 7f97b0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:49:21.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.654+0000 7f97b0ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:49:21.889 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.882+0000 7f97ba619640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:49:21.889 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.882+0000 7f97ba619640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:49:21.889 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.882+0000 7f97ba619640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:49:21.889 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.882+0000 7f97ba619640 1 -- 172.21.15.42:0/1143432177 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f97b4073a30 con 0x7f97b4134a70 2024-09-16T03:49:21.889 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.886+0000 7f97b0ff9640 1 -- 172.21.15.42:0/1143432177 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7f97ac0075f0 con 0x7f97b4134a70 2024-09-16T03:49:21.889 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.886+0000 7f97b0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:49:21.890 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.886+0000 7f97b0ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:49:21.890 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":66,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:49:21.892 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.886+0000 7f97ba619640 10 librados: watch_flush enter 2024-09-16T03:49:21.892 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.886+0000 7f97ba619640 10 librados: watch_flush exit 2024-09-16T03:49:21.892 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.886+0000 7f97ba619640 20 client.764619.objecter shutdown clearing up homeless session... 2024-09-16T03:49:21.892 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.886+0000 7f97ba619640 10 client.764619.objecter successfully canceled tick 2024-09-16T03:49:21.892 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.886+0000 7f97ba619640 1 -- 172.21.15.42:0/1143432177 >> v2:172.21.15.154:6812/254845342 conn(0x7f97a0037810 msgr2=0x7f97a0039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:21.892 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.886+0000 7f97ba619640 1 --2- 172.21.15.42:0/1143432177 >> v2:172.21.15.154:6812/254845342 conn(0x7f97a0037810 0x7f97a0039cd0 secure :-1 s=READY pgs=3509 cs=0 l=1 rev1=1 crypto rx=0x7f97a4009fd0 tx=0x7f97a400b040 comp rx=0 tx=0).stop 2024-09-16T03:49:21.893 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.886+0000 7f97ba619640 10 monclient: shutdown 2024-09-16T03:49:21.893 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.886+0000 7f97ba619640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:49:21.893 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.886+0000 7f97ba619640 1 -- 172.21.15.42:0/1143432177 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f97b4134a70 msgr2=0x7f97b409cb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:21.893 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.886+0000 7f97ba619640 1 --2- 172.21.15.42:0/1143432177 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f97b4134a70 0x7f97b409cb60 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f97ac00c430 tx=0x7f97ac00c8f0 comp rx=0 tx=0).stop 2024-09-16T03:49:21.893 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.886+0000 7f97ba619640 1 -- 172.21.15.42:0/1143432177 shutdown_connections 2024-09-16T03:49:21.893 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.886+0000 7f97ba619640 1 --2- 172.21.15.42:0/1143432177 >> v2:172.21.15.154:6812/254845342 conn(0x7f97a0037810 0x7f97a0039cd0 unknown :-1 s=CLOSED pgs=3509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:21.893 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.886+0000 7f97ba619640 1 --2- 172.21.15.42:0/1143432177 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f97b4134a70 0x7f97b409cb60 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:21.893 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.886+0000 7f97ba619640 1 --2- 172.21.15.42:0/1143432177 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f97b4135100 0x7f97b4134530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:21.893 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.886+0000 7f97ba619640 1 --2- 172.21.15.42:0/1143432177 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f97b40a1330 0x7f97b413b110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:21.893 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.886+0000 7f97ba619640 1 -- 172.21.15.42:0/1143432177 >> 172.21.15.42:0/1143432177 conn(0x7f97b40a2420 msgr2=0x7f97b413c590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:49:21.893 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.886+0000 7f97ba619640 1 -- 172.21.15.42:0/1143432177 shutdown_connections 2024-09-16T03:49:21.893 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.886+0000 7f97ba619640 1 -- 172.21.15.42:0/1143432177 wait complete. 2024-09-16T03:49:21.893 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:21.886+0000 7f97ba619640 1 librados: shutdown 2024-09-16T03:49:21.908 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T03:49:21.908 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T03:49:21.909 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T03:49:22.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.022+0000 7f4dc69cd640 10 monclient: get_monmap_and_config 2024-09-16T03:49:22.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.022+0000 7f4dc69cd640 10 monclient: build_initial_monmap 2024-09-16T03:49:22.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.022+0000 7f4dc69cd640 10 monclient: monmap: 2024-09-16T03:49:22.028 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:49:22.028 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:49:22.028 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:49:22.026984+0000 2024-09-16T03:49:22.028 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:49:22.026984+0000 2024-09-16T03:49:22.028 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:49:22.029 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:49:22.029 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:49:22.029 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:49:22.029 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:49:22.029 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:49:22.029 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:49:22.029 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:49:22.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.022+0000 7f4dc69cd640 1 Processor -- start 2024-09-16T03:49:22.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.022+0000 7f4dc69cd640 1 -- start start 2024-09-16T03:49:22.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.022+0000 7f4dc69cd640 10 monclient: init 2024-09-16T03:49:22.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.026+0000 7f4dc69cd640 10 monclient: _reopen_session rank -1 2024-09-16T03:49:22.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.026+0000 7f4dc69cd640 10 monclient: _add_conns ranks=[2,1,0,3,4] 2024-09-16T03:49:22.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.026+0000 7f4dc69cd640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4dc00a1330 0x7f4dc009f180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:22.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.026+0000 7f4dc69cd640 10 monclient(hunting): picked mon.noname-b con 0x7f4dc00a1330 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:49:22.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.026+0000 7f4dc69cd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4dc00a1710 addr v1:172.21.15.154:6789/0 2024-09-16T03:49:22.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.026+0000 7f4dc69cd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4dc01393a0 addr v1:172.21.15.42:6789/0 2024-09-16T03:49:22.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.026+0000 7f4dc69cd640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4dc009e4c0 con 0x7f4dc01393a0 2024-09-16T03:49:22.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.026+0000 7f4dc69cd640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4dc009f6c0 con 0x7f4dc00a1710 2024-09-16T03:49:22.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.026+0000 7f4dc69cd640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:22.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.026+0000 7f4dc69cd640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f4dc009edf0 con 0x7f4dc00a1330 2024-09-16T03:49:22.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.026+0000 7f4dc69cd640 10 monclient(hunting): _renew_subs 2024-09-16T03:49:22.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.026+0000 7f4dc69cd640 10 monclient(hunting): authenticate will time out at 1408.996503s 2024-09-16T03:49:22.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.026+0000 7f4dbffff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4dc00a1330 0x7f4dc009f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:22.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.026+0000 7f4dbffff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4dc00a1330 0x7f4dc009f180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:33346/0 (socket says 172.21.15.42:33346) 2024-09-16T03:49:22.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.026+0000 7f4dc4f6a640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f4dc01393a0 0x7f4dc013b790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:49730/0 (socket says 172.21.15.42:49730) 2024-09-16T03:49:22.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.026+0000 7f4dbffff640 1 -- 172.21.15.42:0/1943037785 learned_addr learned my addr 172.21.15.42:0/1943037785 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:49:22.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.026+0000 7f4dbffff640 10 monclient(hunting): get_auth_request con 0x7f4dc00a1330 auth_method 0 2024-09-16T03:49:22.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.026+0000 7f4dbffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:22.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.026+0000 7f4dbffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:22.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.026+0000 7f4dbffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:22.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.026+0000 7f4dbffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:22.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.026+0000 7f4dbffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:22.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.026+0000 7f4dbffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:22.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.026+0000 7f4dbffff640 10 monclient(hunting): handle_auth_done global_id 784205 payload 274 2024-09-16T03:49:22.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.026+0000 7f4dbffff640 1 -- 172.21.15.42:0/1943037785 >> v1:172.21.15.154:6789/0 conn(0x7f4dc00a1710 legacy=0x7f4dc00a1b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:49:22.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.026+0000 7f4dbffff640 1 -- 172.21.15.42:0/1943037785 >> v1:172.21.15.42:6789/0 conn(0x7f4dc01393a0 legacy=0x7f4dc013b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:22.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.026+0000 7f4dbffff640 10 monclient: _finish_hunting 0 2024-09-16T03:49:22.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.026+0000 7f4dbffff640 1 monclient: found mon.noname-b 2024-09-16T03:49:22.034 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.026+0000 7f4dbffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:49:22.034 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.026+0000 7f4dbffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T03:49:22.034 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.026+0000 7f4dbffff640 1 -- 172.21.15.42:0/1943037785 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4dc00a3170 con 0x7f4dc00a1330 2024-09-16T03:49:22.034 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.026+0000 7f4dbeffd640 1 -- 172.21.15.42:0/1943037785 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1380170687 0 0) 0x7f4dc009e4c0 con 0x7f4dc01393a0 2024-09-16T03:49:22.034 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.026+0000 7f4dbeffd640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T03:49:22.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.026+0000 7f4dbffff640 1 --2- 172.21.15.42:0/1943037785 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4dc00a1330 0x7f4dc009f180 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f4db00098a0 tx=0x7f4db001c400 comp rx=0 tx=0).ready entity=mon.2 client_cookie=60ca0bd7de785e51 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:22.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.030+0000 7f4dbeffd640 1 -- 172.21.15.42:0/1943037785 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4db00b1070 con 0x7f4dc00a1330 2024-09-16T03:49:22.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.030+0000 7f4dbeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:22.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.030+0000 7f4dbeffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:49:22.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.030+0000 7f4dbeffd640 10 monclient: dump: 2024-09-16T03:49:22.042 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:49:22.042 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:49:22.042 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:22.042 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:22.042 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:22.042 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:49:22.042 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:49:22.042 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:49:22.042 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:49:22.043 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:49:22.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.030+0000 7f4dbeffd640 10 monclient: _finish_auth 0 2024-09-16T03:49:22.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.030+0000 7f4dbeffd640 10 monclient: _check_auth_tickets 2024-09-16T03:49:22.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.030+0000 7f4dbeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:49:22.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.030+0000 7f4dbeffd640 1 -- 172.21.15.42:0/1943037785 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4db00ac640 con 0x7f4dc00a1330 2024-09-16T03:49:22.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.030+0000 7f4dc69cd640 5 monclient: authenticate success, global_id 784205 2024-09-16T03:49:22.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.030+0000 7f4dc69cd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:49:22.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.030+0000 7f4dbeffd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:49:22.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.030+0000 7f4dc69cd640 10 monclient: get_monmap_and_config success 2024-09-16T03:49:22.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.030+0000 7f4dbeffd640 1 -- 172.21.15.42:0/1943037785 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4db00aca10 con 0x7f4dc00a1330 2024-09-16T03:49:22.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.030+0000 7f4dbeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:22.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.030+0000 7f4dc69cd640 4 set_mon_vals no callback set 2024-09-16T03:49:22.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.030+0000 7f4dbeffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:49:22.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.030+0000 7f4dbeffd640 10 monclient: dump: 2024-09-16T03:49:22.044 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:49:22.044 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:49:22.044 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:22.044 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:22.044 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:22.044 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:49:22.044 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:49:22.044 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:49:22.045 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:49:22.045 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:49:22.045 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.030+0000 7f4dc69cd640 10 monclient: shutdown 2024-09-16T03:49:22.045 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.030+0000 7f4dc69cd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:49:22.045 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.030+0000 7f4dc69cd640 1 -- 172.21.15.42:0/1943037785 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4dc00a1330 msgr2=0x7f4dc009f180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:22.045 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.030+0000 7f4dc69cd640 1 --2- 172.21.15.42:0/1943037785 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4dc00a1330 0x7f4dc009f180 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f4db00098a0 tx=0x7f4db001c400 comp rx=0 tx=0).stop 2024-09-16T03:49:22.045 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.030+0000 7f4dc69cd640 1 -- 172.21.15.42:0/1943037785 shutdown_connections 2024-09-16T03:49:22.045 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.030+0000 7f4dc69cd640 1 --2- 172.21.15.42:0/1943037785 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4dc00a1330 0x7f4dc009f180 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:22.045 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.030+0000 7f4dc69cd640 1 -- 172.21.15.42:0/1943037785 >> 172.21.15.42:0/1943037785 conn(0x7f4dc00a2420 msgr2=0x7f4dc00a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:49:22.045 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.030+0000 7f4dc69cd640 1 -- 172.21.15.42:0/1943037785 shutdown_connections 2024-09-16T03:49:22.045 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.030+0000 7f4dc69cd640 1 -- 172.21.15.42:0/1943037785 wait complete. 2024-09-16T03:49:22.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.030+0000 7f4dc69cd640 10 monclient: build_initial_monmap 2024-09-16T03:49:22.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.030+0000 7f4dc69cd640 10 monclient: monmap: 2024-09-16T03:49:22.046 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:49:22.046 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:49:22.046 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:49:22.046 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:49:22.046 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:49:22.046 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:49:22.046 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:49:22.046 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:49:22.046 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:49:22.047 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:49:22.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.030+0000 7f4dc69cd640 1 librados: starting msgr at 2024-09-16T03:49:22.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.030+0000 7f4dc69cd640 1 librados: starting objecter 2024-09-16T03:49:22.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.030+0000 7f4dc69cd640 1 Processor -- start 2024-09-16T03:49:22.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.030+0000 7f4dc69cd640 1 -- start start 2024-09-16T03:49:22.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.030+0000 7f4dc69cd640 1 librados: setting wanted keys 2024-09-16T03:49:22.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.030+0000 7f4dc69cd640 1 librados: calling monclient init 2024-09-16T03:49:22.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.030+0000 7f4dc69cd640 10 monclient: init 2024-09-16T03:49:22.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.030+0000 7f4dc69cd640 10 monclient: _reopen_session rank -1 2024-09-16T03:49:22.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.030+0000 7f4dc69cd640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:49:22.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.030+0000 7f4dc69cd640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4dc00a1330 0x7f4dc01ced50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:22.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.030+0000 7f4dc69cd640 10 monclient(hunting): picked mon.noname-c con 0x7f4dc00a1330 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:49:22.048 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.030+0000 7f4dc69cd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4dc00a1710 0x7f4dc0134300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:22.048 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.030+0000 7f4dc69cd640 10 monclient(hunting): picked mon.noname-b con 0x7f4dc00a1710 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:49:22.048 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.030+0000 7f4dc69cd640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4dc01393a0 0x7f4dc0134840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:22.048 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.030+0000 7f4dc69cd640 10 monclient(hunting): picked mon.noname-a con 0x7f4dc01393a0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:49:22.048 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.030+0000 7f4dc69cd640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:22.048 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.030+0000 7f4dc69cd640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f4dc013f8e0 con 0x7f4dc01393a0 2024-09-16T03:49:22.048 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.030+0000 7f4dc69cd640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:22.048 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.030+0000 7f4dc69cd640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f4dc009cd60 con 0x7f4dc00a1710 2024-09-16T03:49:22.048 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.030+0000 7f4dc69cd640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:22.048 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.030+0000 7f4dc69cd640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f4dc013fa60 con 0x7f4dc00a1330 2024-09-16T03:49:22.048 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.030+0000 7f4dc69cd640 10 monclient(hunting): _renew_subs 2024-09-16T03:49:22.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.030+0000 7f4dc69cd640 10 monclient(hunting): authenticate will time out at 1409.002372s 2024-09-16T03:49:22.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.030+0000 7f4dbffff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4dc00a1330 0x7f4dc01ced50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:22.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.030+0000 7f4dc4f6a640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4dc01393a0 0x7f4dc0134840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:22.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.030+0000 7f4dbffff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4dc00a1330 0x7f4dc01ced50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:33362/0 (socket says 172.21.15.42:33362) 2024-09-16T03:49:22.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.030+0000 7f4dc4f6a640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4dc01393a0 0x7f4dc0134840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:47026/0 (socket says 172.21.15.42:47026) 2024-09-16T03:49:22.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.030+0000 7f4dbffff640 1 -- 172.21.15.42:0/2229983029 learned_addr learned my addr 172.21.15.42:0/2229983029 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:49:22.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.030+0000 7f4dbffff640 10 monclient(hunting): get_auth_request con 0x7f4dc00a1330 auth_method 0 2024-09-16T03:49:22.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.030+0000 7f4dbffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:22.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.030+0000 7f4dbffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:22.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.030+0000 7f4dbffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:22.050 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.030+0000 7f4dc4f6a640 10 monclient(hunting): get_auth_request con 0x7f4dc01393a0 auth_method 0 2024-09-16T03:49:22.050 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.030+0000 7f4dc4f6a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:22.050 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.030+0000 7f4dc4f6a640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:22.050 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.030+0000 7f4dc4f6a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:22.050 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.030+0000 7f4dc4f6a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:22.050 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.030+0000 7f4dc4f6a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:22.050 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.030+0000 7f4dc4f6a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:22.050 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.030+0000 7f4dbf7fe640 1 --2- 172.21.15.42:0/2229983029 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4dc00a1710 0x7f4dc0134300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:22.050 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.030+0000 7f4dbffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:22.050 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.030+0000 7f4dbffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:22.050 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.030+0000 7f4dbffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:22.050 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.030+0000 7f4dbf7fe640 10 monclient(hunting): get_auth_request con 0x7f4dc00a1710 auth_method 0 2024-09-16T03:49:22.050 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.030+0000 7f4dbf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:22.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.030+0000 7f4dbf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:22.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.030+0000 7f4dbf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:22.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.030+0000 7f4dc4f6a640 10 monclient(hunting): handle_auth_done global_id 764625 payload 824 2024-09-16T03:49:22.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.030+0000 7f4dc4f6a640 1 -- 172.21.15.42:0/2229983029 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4dc00a1330 msgr2=0x7f4dc01ced50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:22.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.030+0000 7f4dc4f6a640 1 --2- 172.21.15.42:0/2229983029 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4dc00a1330 0x7f4dc01ced50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:22.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.030+0000 7f4dc4f6a640 1 -- 172.21.15.42:0/2229983029 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4dc00a1710 msgr2=0x7f4dc0134300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:22.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.030+0000 7f4dbf7fe640 1 -- 172.21.15.42:0/2229983029 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4dc00a1710 msgr2=0x7f4dc0134300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-16T03:49:22.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.030+0000 7f4dbf7fe640 1 -- 172.21.15.42:0/2229983029 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4dc00a1710 msgr2=0x7f4dc0134300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T03:49:22.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.030+0000 7f4dbf7fe640 1 --2- 172.21.15.42:0/2229983029 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4dc00a1710 0x7f4dc0134300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:49:22.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.030+0000 7f4dbf7fe640 1 --2- 172.21.15.42:0/2229983029 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4dc00a1710 0x7f4dc0134300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:49:22.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.030+0000 7f4dc4f6a640 1 --2- 172.21.15.42:0/2229983029 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4dc00a1710 0x7f4dc0134300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:22.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.030+0000 7f4dc4f6a640 10 monclient: _finish_hunting 0 2024-09-16T03:49:22.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.030+0000 7f4dc4f6a640 1 monclient: found mon.noname-a 2024-09-16T03:49:22.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.030+0000 7f4dc4f6a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:49:22.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.030+0000 7f4dc4f6a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:49:22.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.030+0000 7f4dc4f6a640 1 -- 172.21.15.42:0/2229983029 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4dc0120cb0 con 0x7f4dc01393a0 2024-09-16T03:49:22.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.030+0000 7f4dc4f6a640 1 --2- 172.21.15.42:0/2229983029 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4dc01393a0 0x7f4dc0134840 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f4dc009df70 tx=0x7f4db800cbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:22.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.030+0000 7f4dbcff9640 10 client.?.objecter ms_handle_connect 0x7f4dc01393a0 2024-09-16T03:49:22.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.030+0000 7f4dbcff9640 10 client.?.objecter resend_mon_ops 2024-09-16T03:49:22.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.030+0000 7f4dbcff9640 1 -- 172.21.15.42:0/2229983029 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4db8018070 con 0x7f4dc01393a0 2024-09-16T03:49:22.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.030+0000 7f4dbcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:22.053 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.030+0000 7f4dbcff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:49:22.053 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.030+0000 7f4dbcff9640 10 monclient: dump: 2024-09-16T03:49:22.053 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:49:22.053 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:49:22.053 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:22.053 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:22.053 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:22.053 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:49:22.053 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:49:22.053 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:49:22.053 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:49:22.054 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:49:22.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.030+0000 7f4dbcff9640 10 monclient: _finish_auth 0 2024-09-16T03:49:22.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.030+0000 7f4dbcff9640 10 monclient: _check_auth_tickets 2024-09-16T03:49:22.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.030+0000 7f4dbcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:49:22.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.030+0000 7f4dbcff9640 1 -- 172.21.15.42:0/2229983029 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4db800ce30 con 0x7f4dc01393a0 2024-09-16T03:49:22.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.030+0000 7f4dc69cd640 5 monclient: authenticate success, global_id 764625 2024-09-16T03:49:22.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.030+0000 7f4dc69cd640 10 monclient: _renew_subs 2024-09-16T03:49:22.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.030+0000 7f4dc69cd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:49:22.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.030+0000 7f4dc69cd640 1 -- 172.21.15.42:0/2229983029 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4dc0073a30 con 0x7f4dc01393a0 2024-09-16T03:49:22.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.030+0000 7f4dbcff9640 10 monclient: handle_config config(0 keys) 2024-09-16T03:49:22.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.030+0000 7f4dbcff9640 1 -- 172.21.15.42:0/2229983029 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4db800a7d0 con 0x7f4dc01393a0 2024-09-16T03:49:22.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.030+0000 7f4dbcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:22.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.030+0000 7f4dbcff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:49:22.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.030+0000 7f4dbe7fc640 4 set_mon_vals no callback set 2024-09-16T03:49:22.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.030+0000 7f4dc69cd640 10 client.764625.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:49:22.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.030+0000 7f4dbcff9640 10 monclient: dump: 2024-09-16T03:49:22.055 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:49:22.055 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:49:22.055 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:22.055 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:22.055 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:22.055 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:49:22.055 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:49:22.056 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:49:22.056 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:49:22.056 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:49:22.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.030+0000 7f4dc69cd640 10 monclient: _renew_subs 2024-09-16T03:49:22.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.030+0000 7f4dc69cd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:49:22.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.030+0000 7f4dc69cd640 1 -- 172.21.15.42:0/2229983029 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4dc00a0d70 con 0x7f4dc01393a0 2024-09-16T03:49:22.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.030+0000 7f4dc69cd640 1 librados: init done 2024-09-16T03:49:22.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.034+0000 7f4dc69cd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:49:22.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.034+0000 7f4dc69cd640 1 --2- 172.21.15.42:0/2229983029 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4dc011cfc0 0x7f4dc011d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:22.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.034+0000 7f4dc69cd640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:22.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.034+0000 7f4dc69cd640 1 -- 172.21.15.42:0/2229983029 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f4dc009f180 con 0x7f4dc011cfc0 2024-09-16T03:49:22.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.034+0000 7f4dbffff640 1 --2- 172.21.15.42:0/2229983029 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4dc011cfc0 0x7f4dc011d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:22.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.034+0000 7f4dbffff640 10 monclient: get_auth_request con 0x7f4dc011cfc0 auth_method 0 2024-09-16T03:49:22.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.034+0000 7f4dbffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:22.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.034+0000 7f4dbffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:22.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.034+0000 7f4dbffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:22.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.034+0000 7f4dbcff9640 1 -- 172.21.15.42:0/2229983029 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f4db801b070 con 0x7f4dc01393a0 2024-09-16T03:49:22.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.034+0000 7f4dbffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:22.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.034+0000 7f4dbffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:22.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.034+0000 7f4dbffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:22.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.034+0000 7f4dbcff9640 1 --2- 172.21.15.42:0/2229983029 >> v2:172.21.15.154:6812/254845342 conn(0x7f4dac037810 0x7f4dac039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:22.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.034+0000 7f4dbcff9640 1 -- 172.21.15.42:0/2229983029 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(243..243 src has 1..243) ==== 5634+0+0 (secure 0 0 0) 0x7f4db80483f0 con 0x7f4dc01393a0 2024-09-16T03:49:22.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.034+0000 7f4dbffff640 10 monclient(hunting): handle_auth_done global_id 764628 payload 824 2024-09-16T03:49:22.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.034+0000 7f4dbcff9640 10 client.764625.objecter ms_dispatch 0x7f4dc0000ba0 osd_map(243..243 src has 1..243) 2024-09-16T03:49:22.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.034+0000 7f4dbcff9640 3 client.764625.objecter handle_osd_map got epochs [243,243] > 0 2024-09-16T03:49:22.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.034+0000 7f4dbcff9640 3 client.764625.objecter handle_osd_map decoding full epoch 243 2024-09-16T03:49:22.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.034+0000 7f4dbffff640 1 -- 172.21.15.42:0/2229983029 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4dc00008e0 con 0x7f4dc011cfc0 2024-09-16T03:49:22.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.034+0000 7f4dbcff9640 20 client.764625.objecter dump_active .. 0 homeless 2024-09-16T03:49:22.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.034+0000 7f4dbcff9640 10 client.764625.objecter ms_handle_connect 0x7f4dc011cfc0 2024-09-16T03:49:22.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.034+0000 7f4dbcff9640 10 client.764625.objecter resend_mon_ops 2024-09-16T03:49:22.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.034+0000 7f4dbffff640 1 --2- 172.21.15.42:0/2229983029 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4dc011cfc0 0x7f4dc011d3a0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f4db00af040 tx=0x7f4db00191a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:22.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.034+0000 7f4dbf7fe640 1 --2- 172.21.15.42:0/2229983029 >> v2:172.21.15.154:6812/254845342 conn(0x7f4dac037810 0x7f4dac039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:22.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.034+0000 7f4dbcff9640 1 -- 172.21.15.42:0/2229983029 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4db00bd070 con 0x7f4dc011cfc0 2024-09-16T03:49:22.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.034+0000 7f4dbcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:22.059 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.034+0000 7f4dbcff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:49:22.059 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.034+0000 7f4dbcff9640 10 monclient: dump: 2024-09-16T03:49:22.059 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:49:22.059 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:49:22.059 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:22.059 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:22.059 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:22.060 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:49:22.060 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:49:22.060 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:49:22.060 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:49:22.060 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:49:22.060 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.034+0000 7f4dbf7fe640 10 monclient: get_auth_request con 0x7f4dac037810 auth_method 0 2024-09-16T03:49:22.060 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.034+0000 7f4dbcff9640 1 -- 172.21.15.42:0/2229983029 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f4db00b6640 con 0x7f4dc011cfc0 2024-09-16T03:49:22.060 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.034+0000 7f4dbcff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:49:22.060 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.034+0000 7f4dbcff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:49:22.060 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.034+0000 7f4dbcff9640 1 -- 172.21.15.42:0/2229983029 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4dc011cfc0 msgr2=0x7f4dc011d3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:22.060 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.034+0000 7f4dbcff9640 1 --2- 172.21.15.42:0/2229983029 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4dc011cfc0 0x7f4dc011d3a0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f4db00af040 tx=0x7f4db00191a0 comp rx=0 tx=0).stop 2024-09-16T03:49:22.061 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.034+0000 7f4dbcff9640 1 -- 172.21.15.42:0/2229983029 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4dc011cfc0 msgr2=0x7f4dc011d3a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:49:22.061 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.034+0000 7f4dbcff9640 1 --2- 172.21.15.42:0/2229983029 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4dc011cfc0 0x7f4dc011d3a0 secure :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f4db00af040 tx=0x7f4db00191a0 comp rx=0 tx=0).stop 2024-09-16T03:49:22.061 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.034+0000 7f4dbf7fe640 1 --2- 172.21.15.42:0/2229983029 >> v2:172.21.15.154:6812/254845342 conn(0x7f4dac037810 0x7f4dac039cd0 secure :-1 s=READY pgs=3510 cs=0 l=1 rev1=1 crypto rx=0x7f4dc011e5e0 tx=0x7f4db4009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:22.061 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.034+0000 7f4dbcff9640 10 client.764625.objecter ms_handle_connect 0x7f4dac037810 2024-09-16T03:49:22.063 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.038+0000 7f4dc69cd640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:49:22.063 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.038+0000 7f4dc69cd640 1 --2- 172.21.15.42:0/2229983029 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4dc011d8e0 0x7f4dc0137390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:22.063 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.038+0000 7f4dc69cd640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:22.063 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.038+0000 7f4dc69cd640 1 -- 172.21.15.42:0/2229983029 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f4dc011f520 con 0x7f4dc011d8e0 2024-09-16T03:49:22.063 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.038+0000 7f4dbffff640 1 --2- 172.21.15.42:0/2229983029 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4dc011d8e0 0x7f4dc0137390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:22.063 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.038+0000 7f4dbffff640 10 monclient: get_auth_request con 0x7f4dc011d8e0 auth_method 0 2024-09-16T03:49:22.063 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.038+0000 7f4dbffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:22.063 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.038+0000 7f4dbffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:22.063 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.038+0000 7f4dbffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:22.064 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.038+0000 7f4dbffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:22.064 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.038+0000 7f4dbffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:22.064 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.038+0000 7f4dbffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:22.064 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.042+0000 7f4dbffff640 10 monclient(hunting): handle_auth_done global_id 764631 payload 824 2024-09-16T03:49:22.064 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.042+0000 7f4dbffff640 1 -- 172.21.15.42:0/2229983029 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f4dc011dcf0 con 0x7f4dc011d8e0 2024-09-16T03:49:22.064 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.042+0000 7f4dbcff9640 10 client.764625.objecter ms_handle_connect 0x7f4dc011d8e0 2024-09-16T03:49:22.064 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.042+0000 7f4dbcff9640 10 client.764625.objecter resend_mon_ops 2024-09-16T03:49:22.064 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.042+0000 7f4dbffff640 1 --2- 172.21.15.42:0/2229983029 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4dc011d8e0 0x7f4dc0137390 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f4db0000c10 tx=0x7f4db0009240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:22.064 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.042+0000 7f4dbcff9640 1 -- 172.21.15.42:0/2229983029 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4db00b1040 con 0x7f4dc011d8e0 2024-09-16T03:49:22.064 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.042+0000 7f4dbcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:22.064 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.042+0000 7f4dbcff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:49:22.065 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.042+0000 7f4dbcff9640 10 monclient: dump: 2024-09-16T03:49:22.065 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:49:22.065 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:49:22.065 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:22.065 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:22.065 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:22.065 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:49:22.065 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:49:22.065 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:49:22.065 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:49:22.065 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:49:22.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.042+0000 7f4dbcff9640 1 -- 172.21.15.42:0/2229983029 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4881 (secure 0 0 0) 0x7f4db00b6e40 con 0x7f4dc011d8e0 2024-09-16T03:49:22.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.042+0000 7f4dbcff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:49:22.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.042+0000 7f4dbcff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:49:22.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.042+0000 7f4dbcff9640 1 -- 172.21.15.42:0/2229983029 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4dc011d8e0 msgr2=0x7f4dc0137390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:22.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.042+0000 7f4dbcff9640 1 --2- 172.21.15.42:0/2229983029 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4dc011d8e0 0x7f4dc0137390 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f4db0000c10 tx=0x7f4db0009240 comp rx=0 tx=0).stop 2024-09-16T03:49:22.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.042+0000 7f4dbcff9640 1 -- 172.21.15.42:0/2229983029 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4dc011d8e0 msgr2=0x7f4dc0137390 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:49:22.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.042+0000 7f4dbcff9640 1 --2- 172.21.15.42:0/2229983029 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4dc011d8e0 0x7f4dc0137390 secure :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f4db0000c10 tx=0x7f4db0009240 comp rx=0 tx=0).stop 2024-09-16T03:49:22.066 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:49:22.067 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:49:22.067 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:49:22.067 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "leader", 2024-09-16T03:49:22.067 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 66, 2024-09-16T03:49:22.067 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 152834, 2024-09-16T03:49:22.067 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:49:22.067 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:49:22.067 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T03:49:22.067 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T03:49:22.067 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:49:22.067 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 0, 2024-09-16T03:49:22.067 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:49:22.068 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:49:22.068 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:49:22.068 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:49:22.068 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:49:22.068 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:49:22.068 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:49:22.068 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:49:22.068 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:49:22.068 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:49:22.068 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:49:22.068 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:49:22.069 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:49:22.069 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:49:22.069 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:49:22.069 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:49:22.069 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:49:22.069 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:49:22.070 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:49:22.070 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:49:22.070 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:49:22.070 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:49:22.070 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:49:22.070 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:49:22.070 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:49:22.070 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:49:22.070 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:49:22.070 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:49:22.070 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:49:22.071 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:49:22.071 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:49:22.071 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:49:22.071 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:49:22.071 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:49:22.071 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:49:22.071 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:49:22.071 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:49:22.071 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:49:22.071 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:49:22.072 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:49:22.072 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:49:22.072 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:49:22.072 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:49:22.072 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:49:22.072 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:49:22.072 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:49:22.072 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:49:22.072 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:49:22.072 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:49:22.072 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:49:22.073 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:49:22.073 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:49:22.073 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:49:22.073 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:49:22.073 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:49:22.073 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:49:22.073 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:49:22.073 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:49:22.073 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:49:22.073 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:49:22.073 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:49:22.073 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:49:22.074 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:49:22.074 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:49:22.074 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:49:22.074 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:49:22.074 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:49:22.074 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:49:22.074 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:49:22.074 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:49:22.074 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:49:22.074 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:49:22.074 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:49:22.074 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:49:22.075 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:49:22.075 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:49:22.075 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:49:22.075 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:49:22.075 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:49:22.075 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:49:22.075 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:49:22.075 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:49:22.075 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:49:22.075 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:49:22.075 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:49:22.076 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:49:22.076 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:49:22.076 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:49:22.076 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:49:22.076 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:49:22.076 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:49:22.076 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:49:22.076 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:49:22.076 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:49:22.076 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:49:22.076 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:49:22.077 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:49:22.077 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:49:22.077 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:49:22.077 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:49:22.077 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:49:22.077 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:49:22.077 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:49:22.077 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:49:22.077 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:49:22.077 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:49:22.078 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:49:22.078 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:49:22.078 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:49:22.078 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:49:22.078 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:49:22.078 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:49:22.078 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:49:22.078 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:49:22.078 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:49:22.078 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:49:22.078 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:49:22.079 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:49:22.079 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:49:22.079 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:49:22.079 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:49:22.079 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:49:22.079 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:49:22.079 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:49:22.079 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:49:22.079 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:49:22.079 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:49:22.079 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:49:22.079 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:49:22.080 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:49:22.080 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:49:22.080 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:49:22.080 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:49:22.080 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:49:22.080 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:49:22.080 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:49:22.080 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:49:22.080 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:49:22.080 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:49:22.080 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:49:22.080 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:49:22.081 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T03:49:22.081 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:49:22.081 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:49:22.081 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:49:22.081 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 5 2024-09-16T03:49:22.081 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:49:22.081 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:49:22.081 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:49:22.081 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:49:22.081 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:49:22.081 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:49:22.081 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 8 2024-09-16T03:49:22.082 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:49:22.082 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:49:22.082 INFO:teuthology.orchestra.run.smithi042.stdout: "mgr": [ 2024-09-16T03:49:22.082 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:49:22.082 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:49:22.082 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:49:22.082 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:49:22.082 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:49:22.082 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:49:22.082 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:49:22.082 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:49:22.082 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:49:22.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.042+0000 7f4dc69cd640 10 librados: watch_flush enter 2024-09-16T03:49:22.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.042+0000 7f4dc69cd640 10 librados: watch_flush exit 2024-09-16T03:49:22.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.042+0000 7f4dc69cd640 20 client.764625.objecter shutdown clearing up homeless session... 2024-09-16T03:49:22.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.042+0000 7f4dc69cd640 10 client.764625.objecter successfully canceled tick 2024-09-16T03:49:22.085 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.042+0000 7f4dc69cd640 1 -- 172.21.15.42:0/2229983029 >> v2:172.21.15.154:6812/254845342 conn(0x7f4dac037810 msgr2=0x7f4dac039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:22.085 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.042+0000 7f4dc69cd640 1 --2- 172.21.15.42:0/2229983029 >> v2:172.21.15.154:6812/254845342 conn(0x7f4dac037810 0x7f4dac039cd0 secure :-1 s=READY pgs=3510 cs=0 l=1 rev1=1 crypto rx=0x7f4dc011e5e0 tx=0x7f4db4009040 comp rx=0 tx=0).stop 2024-09-16T03:49:22.085 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.042+0000 7f4dc69cd640 10 monclient: shutdown 2024-09-16T03:49:22.085 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.042+0000 7f4dc69cd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:49:22.085 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.042+0000 7f4dc69cd640 1 -- 172.21.15.42:0/2229983029 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4dc01393a0 msgr2=0x7f4dc0134840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:22.085 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.042+0000 7f4dc69cd640 1 --2- 172.21.15.42:0/2229983029 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4dc01393a0 0x7f4dc0134840 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f4dc009df70 tx=0x7f4db800cbe0 comp rx=0 tx=0).stop 2024-09-16T03:49:22.085 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.042+0000 7f4dc4f6a640 1 -- 172.21.15.42:0/2229983029 reap_dead start 2024-09-16T03:49:22.085 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.042+0000 7f4dc4f6a640 1 -- 172.21.15.42:0/2229983029 reap_dead start 2024-09-16T03:49:22.085 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.042+0000 7f4dc69cd640 1 -- 172.21.15.42:0/2229983029 shutdown_connections 2024-09-16T03:49:22.085 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.042+0000 7f4dc69cd640 1 -- 172.21.15.42:0/2229983029 >> 172.21.15.42:0/2229983029 conn(0x7f4dc00a2420 msgr2=0x7f4dc0071d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:49:22.085 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.042+0000 7f4dc69cd640 1 -- 172.21.15.42:0/2229983029 shutdown_connections 2024-09-16T03:49:22.085 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.042+0000 7f4dc69cd640 1 -- 172.21.15.42:0/2229983029 wait complete. 2024-09-16T03:49:22.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.042+0000 7f4dc69cd640 1 librados: shutdown 2024-09-16T03:49:22.086 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:49:22.086 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T03:49:22.086 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T03:49:22.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.198+0000 7f5fb7823640 10 monclient: get_monmap_and_config 2024-09-16T03:49:22.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.198+0000 7f5fb7823640 10 monclient: build_initial_monmap 2024-09-16T03:49:22.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.198+0000 7f5fb7823640 10 monclient: monmap: 2024-09-16T03:49:22.203 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:49:22.203 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:49:22.203 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:49:22.201872+0000 2024-09-16T03:49:22.203 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:49:22.201872+0000 2024-09-16T03:49:22.203 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:49:22.203 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:49:22.203 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:49:22.204 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:49:22.204 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:49:22.204 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:49:22.204 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:49:22.204 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:49:22.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.198+0000 7f5fb7823640 1 Processor -- start 2024-09-16T03:49:22.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.198+0000 7f5fb7823640 1 -- start start 2024-09-16T03:49:22.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.198+0000 7f5fb7823640 10 monclient: init 2024-09-16T03:49:22.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.198+0000 7f5fb7823640 10 monclient: _reopen_session rank -1 2024-09-16T03:49:22.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.198+0000 7f5fb7823640 10 monclient: _add_conns ranks=[0,4,2,1,3] 2024-09-16T03:49:22.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.198+0000 7f5fb7823640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5fb00a1330 addr v1:172.21.15.42:6789/0 2024-09-16T03:49:22.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.198+0000 7f5fb7823640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f5fb006dac0 0x7f5fb013a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:22.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.198+0000 7f5fb7823640 10 monclient(hunting): picked mon.noname-c con 0x7f5fb006dac0 addr v2:172.21.15.154:3300/0 2024-09-16T03:49:22.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.198+0000 7f5fb7823640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5fb00a1e20 0x7f5fb013c890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:22.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.198+0000 7f5fb7823640 10 monclient(hunting): picked mon.noname-b con 0x7f5fb00a1e20 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:49:22.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.198+0000 7f5fb7823640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5fb009c090 con 0x7f5fb00a1330 2024-09-16T03:49:22.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.198+0000 7f5fb7823640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:22.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.198+0000 7f5fb7823640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f5fb009c890 con 0x7f5fb006dac0 2024-09-16T03:49:22.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.198+0000 7f5fb7823640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:22.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.198+0000 7f5fb7823640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f5fb009f610 con 0x7f5fb00a1e20 2024-09-16T03:49:22.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.198+0000 7f5fb7823640 10 monclient(hunting): _renew_subs 2024-09-16T03:49:22.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.198+0000 7f5fb7823640 10 monclient(hunting): authenticate will time out at 1409.171344s 2024-09-16T03:49:22.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.198+0000 7f5fb55bf640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f5fb00a1330 0x7f5fb00a1710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:49742/0 (socket says 172.21.15.42:49742) 2024-09-16T03:49:22.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.198+0000 7f5fb55bf640 1 -- 172.21.15.42:0/3955841886 learned_addr learned my addr 172.21.15.42:0/3955841886 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:49:22.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.198+0000 7f5fb5dc0640 1 --2- 172.21.15.42:0/3955841886 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5fb00a1e20 0x7f5fb013c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:22.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.198+0000 7f5fb5dc0640 10 monclient(hunting): get_auth_request con 0x7f5fb00a1e20 auth_method 0 2024-09-16T03:49:22.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.198+0000 7f5fb5dc0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:22.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.198+0000 7f5fb5dc0640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:22.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.198+0000 7f5fb5dc0640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:22.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.198+0000 7f5fb4dbe640 1 --2- 172.21.15.42:0/3955841886 >> v2:172.21.15.154:3300/0 conn(0x7f5fb006dac0 0x7f5fb013a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:22.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.198+0000 7f5fb5dc0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:22.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.198+0000 7f5fb5dc0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:22.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.198+0000 7f5fb5dc0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:22.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.198+0000 7f5fb4dbe640 10 monclient(hunting): get_auth_request con 0x7f5fb006dac0 auth_method 0 2024-09-16T03:49:22.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.198+0000 7f5fb4dbe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:22.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.198+0000 7f5fb4dbe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:22.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.198+0000 7f5fb4dbe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:22.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.198+0000 7f5fa7fff640 1 -- 172.21.15.42:0/3955841886 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 162666927 0 0) 0x7f5fb009c090 con 0x7f5fb00a1330 2024-09-16T03:49:22.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.198+0000 7f5fa7fff640 10 monclient(hunting): _negotiate 2024-09-16T03:49:22.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.198+0000 7f5fa7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:22.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.198+0000 7f5fa7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:22.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.198+0000 7f5fa7fff640 10 monclient(hunting): my global_id is 764634 2024-09-16T03:49:22.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.202+0000 7f5fb4dbe640 1 -- 172.21.15.42:0/3955841886 >> v2:172.21.15.154:3300/0 conn(0x7f5fb006dac0 msgr2=0x7f5fb013a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T03:49:22.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.202+0000 7f5fb4dbe640 1 -- 172.21.15.42:0/3955841886 >> v2:172.21.15.154:3300/0 conn(0x7f5fb006dac0 msgr2=0x7f5fb013a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:49:22.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.202+0000 7f5fb4dbe640 1 --2- 172.21.15.42:0/3955841886 >> v2:172.21.15.154:3300/0 conn(0x7f5fb006dac0 0x7f5fb013a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:49:22.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.202+0000 7f5fa7fff640 1 -- 172.21.15.42:0/3955841886 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5f9c003530 con 0x7f5fb00a1330 2024-09-16T03:49:22.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.202+0000 7f5fb4dbe640 1 --2- 172.21.15.42:0/3955841886 >> v2:172.21.15.154:3300/0 conn(0x7f5fb006dac0 0x7f5fb013a340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:49:22.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.202+0000 7f5fb5dc0640 10 monclient(hunting): handle_auth_done global_id 784211 payload 274 2024-09-16T03:49:22.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.202+0000 7f5fb5dc0640 1 -- 172.21.15.42:0/3955841886 >> v2:172.21.15.154:3300/0 conn(0x7f5fb006dac0 msgr2=0x7f5fb013a340 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:49:22.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.202+0000 7f5fb5dc0640 1 --2- 172.21.15.42:0/3955841886 >> v2:172.21.15.154:3300/0 conn(0x7f5fb006dac0 0x7f5fb013a340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:22.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.202+0000 7f5fb5dc0640 1 -- 172.21.15.42:0/3955841886 >> v1:172.21.15.42:6789/0 conn(0x7f5fb00a1330 legacy=0x7f5fb00a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:22.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.202+0000 7f5fb5dc0640 10 monclient: _finish_hunting 0 2024-09-16T03:49:22.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.202+0000 7f5fb5dc0640 1 monclient: found mon.noname-b 2024-09-16T03:49:22.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.202+0000 7f5fb5dc0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:49:22.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.202+0000 7f5fb5dc0640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T03:49:22.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.202+0000 7f5fb5dc0640 1 -- 172.21.15.42:0/3955841886 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5fb009f8f0 con 0x7f5fb00a1e20 2024-09-16T03:49:22.224 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.202+0000 7f5fb5dc0640 1 --2- 172.21.15.42:0/3955841886 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5fb00a1e20 0x7f5fb013c890 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f5fa8009d90 tx=0x7f5fa801c890 comp rx=0 tx=0).ready entity=mon.2 client_cookie=53f23a1637bfaab6 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:22.224 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.202+0000 7f5fa7fff640 1 -- 172.21.15.42:0/3955841886 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fa80b2070 con 0x7f5fb00a1e20 2024-09-16T03:49:22.224 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.202+0000 7f5fa7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:22.224 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.202+0000 7f5fa7fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:49:22.224 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.202+0000 7f5fa7fff640 10 monclient: dump: 2024-09-16T03:49:22.224 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:49:22.224 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:49:22.224 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:22.225 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:22.225 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:22.225 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:49:22.225 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:49:22.225 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:49:22.225 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:49:22.225 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:49:22.225 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.202+0000 7f5fa7fff640 10 monclient: _finish_auth 0 2024-09-16T03:49:22.225 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.202+0000 7f5fa7fff640 10 monclient: _check_auth_tickets 2024-09-16T03:49:22.225 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.202+0000 7f5fa7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:49:22.225 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.202+0000 7f5fa7fff640 1 -- 172.21.15.42:0/3955841886 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5fa80a5970 con 0x7f5fb00a1e20 2024-09-16T03:49:22.225 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.202+0000 7f5fb7823640 5 monclient: authenticate success, global_id 784211 2024-09-16T03:49:22.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.202+0000 7f5fb7823640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:49:22.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.202+0000 7f5fa7fff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:49:22.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.202+0000 7f5fb7823640 10 monclient: get_monmap_and_config success 2024-09-16T03:49:22.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.202+0000 7f5fa7fff640 1 -- 172.21.15.42:0/3955841886 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fa80a5d40 con 0x7f5fb00a1e20 2024-09-16T03:49:22.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.202+0000 7f5fa7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:22.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.202+0000 7f5fb7823640 4 set_mon_vals no callback set 2024-09-16T03:49:22.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.202+0000 7f5fa7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:49:22.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.202+0000 7f5fa7fff640 10 monclient: dump: 2024-09-16T03:49:22.226 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:49:22.226 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:49:22.226 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:22.226 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:22.227 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:22.227 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:49:22.227 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:49:22.227 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:49:22.227 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:49:22.227 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:49:22.227 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.202+0000 7f5fb7823640 10 monclient: shutdown 2024-09-16T03:49:22.227 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.202+0000 7f5fb7823640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:49:22.227 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.202+0000 7f5fb7823640 1 -- 172.21.15.42:0/3955841886 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5fb00a1e20 msgr2=0x7f5fb013c890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:22.227 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.202+0000 7f5fb7823640 1 --2- 172.21.15.42:0/3955841886 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5fb00a1e20 0x7f5fb013c890 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f5fa8009d90 tx=0x7f5fa801c890 comp rx=0 tx=0).stop 2024-09-16T03:49:22.227 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.202+0000 7f5fb7823640 1 -- 172.21.15.42:0/3955841886 shutdown_connections 2024-09-16T03:49:22.227 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.202+0000 7f5fb7823640 1 --2- 172.21.15.42:0/3955841886 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5fb00a1e20 0x7f5fb013c890 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:22.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.202+0000 7f5fb7823640 1 --2- 172.21.15.42:0/3955841886 >> v2:172.21.15.154:3300/0 conn(0x7f5fb006dac0 0x7f5fb013a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:22.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.202+0000 7f5fb7823640 1 -- 172.21.15.42:0/3955841886 >> 172.21.15.42:0/3955841886 conn(0x7f5fb00a2420 msgr2=0x7f5fb00a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:49:22.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.202+0000 7f5fb7823640 1 -- 172.21.15.42:0/3955841886 shutdown_connections 2024-09-16T03:49:22.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.202+0000 7f5fb7823640 1 -- 172.21.15.42:0/3955841886 wait complete. 2024-09-16T03:49:22.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.202+0000 7f5fb7823640 10 monclient: build_initial_monmap 2024-09-16T03:49:22.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.202+0000 7f5fb7823640 10 monclient: monmap: 2024-09-16T03:49:22.228 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:49:22.228 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:49:22.228 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:49:22.228 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:49:22.228 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:49:22.229 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:49:22.229 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:49:22.229 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:49:22.229 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:49:22.229 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:49:22.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.206+0000 7f5fb7823640 1 librados: starting msgr at 2024-09-16T03:49:22.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.206+0000 7f5fb7823640 1 librados: starting objecter 2024-09-16T03:49:22.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.206+0000 7f5fb7823640 1 Processor -- start 2024-09-16T03:49:22.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.206+0000 7f5fb7823640 1 -- start start 2024-09-16T03:49:22.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.206+0000 7f5fb7823640 1 librados: setting wanted keys 2024-09-16T03:49:22.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.206+0000 7f5fb7823640 1 librados: calling monclient init 2024-09-16T03:49:22.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.206+0000 7f5fb7823640 10 monclient: init 2024-09-16T03:49:22.230 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.206+0000 7f5fb7823640 10 monclient: _reopen_session rank -1 2024-09-16T03:49:22.230 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.206+0000 7f5fb7823640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:49:22.230 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.206+0000 7f5fb7823640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5fb00a1330 0x7f5fb01cecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:22.230 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.206+0000 7f5fb7823640 10 monclient(hunting): picked mon.noname-c con 0x7f5fb00a1330 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:49:22.230 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.206+0000 7f5fb7823640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5fb00a1e20 0x7f5fb01354c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:22.230 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.206+0000 7f5fb7823640 10 monclient(hunting): picked mon.noname-b con 0x7f5fb00a1e20 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:49:22.230 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.206+0000 7f5fb7823640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5fb0135a00 0x7f5fb009cb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:22.230 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.206+0000 7f5fb7823640 10 monclient(hunting): picked mon.noname-a con 0x7f5fb0135a00 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:49:22.230 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.206+0000 7f5fb7823640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:22.230 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.206+0000 7f5fb7823640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f5fb013f770 con 0x7f5fb0135a00 2024-09-16T03:49:22.230 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.206+0000 7f5fb7823640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:22.230 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.206+0000 7f5fb7823640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f5fb013f5f0 con 0x7f5fb00a1e20 2024-09-16T03:49:22.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.206+0000 7f5fb7823640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:22.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.206+0000 7f5fb7823640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f5fb013f8f0 con 0x7f5fb00a1330 2024-09-16T03:49:22.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.206+0000 7f5fb7823640 10 monclient(hunting): _renew_subs 2024-09-16T03:49:22.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.206+0000 7f5fb7823640 10 monclient(hunting): authenticate will time out at 1409.177142s 2024-09-16T03:49:22.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.206+0000 7f5fb55bf640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5fb00a1330 0x7f5fb01cecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:22.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.206+0000 7f5fb55bf640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5fb00a1330 0x7f5fb01cecf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:33374/0 (socket says 172.21.15.42:33374) 2024-09-16T03:49:22.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.206+0000 7f5fb55bf640 1 -- 172.21.15.42:0/3868083236 learned_addr learned my addr 172.21.15.42:0/3868083236 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:49:22.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.206+0000 7f5fb55bf640 10 monclient(hunting): get_auth_request con 0x7f5fb00a1330 auth_method 0 2024-09-16T03:49:22.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.206+0000 7f5fb55bf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:22.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.206+0000 7f5fb55bf640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:22.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.206+0000 7f5fb55bf640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:22.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.206+0000 7f5fb5dc0640 1 --2- 172.21.15.42:0/3868083236 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5fb0135a00 0x7f5fb009cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:22.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.206+0000 7f5fb5dc0640 10 monclient(hunting): get_auth_request con 0x7f5fb0135a00 auth_method 0 2024-09-16T03:49:22.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.206+0000 7f5fb5dc0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:22.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.206+0000 7f5fb5dc0640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:22.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.206+0000 7f5fb5dc0640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:22.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.206+0000 7f5fb55bf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:22.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.206+0000 7f5fb55bf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:22.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.206+0000 7f5fb55bf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:22.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.206+0000 7f5fb5dc0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:22.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.206+0000 7f5fb5dc0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:22.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.206+0000 7f5fb5dc0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:22.233 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.206+0000 7f5fb4dbe640 1 --2- 172.21.15.42:0/3868083236 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5fb00a1e20 0x7f5fb01354c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:22.233 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.206+0000 7f5fb55bf640 10 monclient(hunting): handle_auth_done global_id 784214 payload 824 2024-09-16T03:49:22.233 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.206+0000 7f5fb55bf640 1 -- 172.21.15.42:0/3868083236 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5fb00a1e20 msgr2=0x7f5fb01354c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:22.233 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.206+0000 7f5fb55bf640 1 --2- 172.21.15.42:0/3868083236 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5fb00a1e20 0x7f5fb01354c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:22.233 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.206+0000 7f5fb55bf640 1 -- 172.21.15.42:0/3868083236 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5fb0135a00 msgr2=0x7f5fb009cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:22.233 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.206+0000 7f5fb55bf640 1 --2- 172.21.15.42:0/3868083236 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5fb0135a00 0x7f5fb009cb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:22.233 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.206+0000 7f5fb55bf640 10 monclient: _finish_hunting 0 2024-09-16T03:49:22.233 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.206+0000 7f5fb55bf640 1 monclient: found mon.noname-c 2024-09-16T03:49:22.233 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.206+0000 7f5fb55bf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:49:22.233 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.206+0000 7f5fb55bf640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:49:22.233 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.206+0000 7f5fb55bf640 1 -- 172.21.15.42:0/3868083236 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5fb013c380 con 0x7f5fb00a1330 2024-09-16T03:49:22.233 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.206+0000 7f5fb55bf640 1 --2- 172.21.15.42:0/3868083236 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5fb00a1330 0x7f5fb01cecf0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f5fa000c450 tx=0x7f5fa000c910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:22.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.206+0000 7f5fa5ffb640 10 client.?.objecter ms_handle_connect 0x7f5fb00a1330 2024-09-16T03:49:22.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.206+0000 7f5fa5ffb640 10 client.?.objecter resend_mon_ops 2024-09-16T03:49:22.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.206+0000 7f5fa5ffb640 1 -- 172.21.15.42:0/3868083236 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fa0017070 con 0x7f5fb00a1330 2024-09-16T03:49:22.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.206+0000 7f5fa5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:22.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.206+0000 7f5fa5ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:49:22.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.206+0000 7f5fa5ffb640 10 monclient: dump: 2024-09-16T03:49:22.234 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:49:22.234 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:49:22.234 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:22.234 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:22.234 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:22.235 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:49:22.235 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:49:22.235 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:49:22.235 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:49:22.235 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:49:22.235 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.206+0000 7f5fa5ffb640 10 monclient: _finish_auth 0 2024-09-16T03:49:22.235 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.206+0000 7f5fa5ffb640 10 monclient: _check_auth_tickets 2024-09-16T03:49:22.235 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.206+0000 7f5fa5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:49:22.235 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.206+0000 7f5fb7823640 5 monclient: authenticate success, global_id 784214 2024-09-16T03:49:22.235 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.206+0000 7f5fb7823640 10 monclient: _renew_subs 2024-09-16T03:49:22.235 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.206+0000 7f5fa5ffb640 1 -- 172.21.15.42:0/3868083236 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5fa0005cf0 con 0x7f5fb00a1330 2024-09-16T03:49:22.235 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.206+0000 7f5fb7823640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:49:22.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.206+0000 7f5fb7823640 1 -- 172.21.15.42:0/3868083236 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5fb01cf230 con 0x7f5fb00a1330 2024-09-16T03:49:22.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.206+0000 7f5fa5ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T03:49:22.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.206+0000 7f5fa5ffb640 1 -- 172.21.15.42:0/3868083236 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fa0007290 con 0x7f5fb00a1330 2024-09-16T03:49:22.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.206+0000 7f5fa5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:22.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.206+0000 7f5fa5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:49:22.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.206+0000 7f5fa5ffb640 10 monclient: dump: 2024-09-16T03:49:22.236 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:49:22.236 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:49:22.236 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:22.236 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:22.236 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:22.236 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:49:22.237 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:49:22.237 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:49:22.237 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:49:22.237 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:49:22.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.206+0000 7f5fb7823640 10 client.784214.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:49:22.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.206+0000 7f5fb7823640 10 monclient: _renew_subs 2024-09-16T03:49:22.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.206+0000 7f5fa77fe640 4 set_mon_vals no callback set 2024-09-16T03:49:22.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.206+0000 7f5fb7823640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:49:22.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.206+0000 7f5fb7823640 1 -- 172.21.15.42:0/3868083236 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f5fb0073a30 con 0x7f5fb00a1330 2024-09-16T03:49:22.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.206+0000 7f5fb7823640 1 librados: init done 2024-09-16T03:49:22.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.206+0000 7f5fb7823640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:49:22.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.206+0000 7f5fb7823640 1 --2- 172.21.15.42:0/3868083236 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5fb006dac0 0x7f5fb0121910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:22.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.206+0000 7f5fb7823640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:22.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.206+0000 7f5fb7823640 1 -- 172.21.15.42:0/3868083236 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f5fb00a1710 con 0x7f5fb006dac0 2024-09-16T03:49:22.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.206+0000 7f5fb4dbe640 1 --2- 172.21.15.42:0/3868083236 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5fb006dac0 0x7f5fb0121910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:22.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.206+0000 7f5fb4dbe640 10 monclient: get_auth_request con 0x7f5fb006dac0 auth_method 0 2024-09-16T03:49:22.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.206+0000 7f5fb4dbe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:22.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.206+0000 7f5fb4dbe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:22.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.206+0000 7f5fb4dbe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:22.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.206+0000 7f5fa5ffb640 1 -- 172.21.15.42:0/3868083236 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f5fa001a070 con 0x7f5fb00a1330 2024-09-16T03:49:22.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.206+0000 7f5fb4dbe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:22.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.206+0000 7f5fb4dbe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:22.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.206+0000 7f5fb4dbe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:22.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.206+0000 7f5fa5ffb640 1 --2- 172.21.15.42:0/3868083236 >> v2:172.21.15.154:6812/254845342 conn(0x7f5f94037810 0x7f5f94039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:22.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.206+0000 7f5fb4dbe640 10 monclient(hunting): handle_auth_done global_id 784217 payload 824 2024-09-16T03:49:22.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.206+0000 7f5fa5ffb640 1 -- 172.21.15.42:0/3868083236 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(243..243 src has 1..243) ==== 5634+0+0 (secure 0 0 0) 0x7f5fa0048340 con 0x7f5fb00a1330 2024-09-16T03:49:22.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.206+0000 7f5fa5ffb640 10 client.784214.objecter ms_dispatch 0x7f5fb0000ba0 osd_map(243..243 src has 1..243) 2024-09-16T03:49:22.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.206+0000 7f5fb4dbe640 1 -- 172.21.15.42:0/3868083236 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5fb01203f0 con 0x7f5fb006dac0 2024-09-16T03:49:22.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.206+0000 7f5fa5ffb640 3 client.784214.objecter handle_osd_map got epochs [243,243] > 0 2024-09-16T03:49:22.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.206+0000 7f5fa5ffb640 3 client.784214.objecter handle_osd_map decoding full epoch 243 2024-09-16T03:49:22.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.206+0000 7f5fa5ffb640 20 client.784214.objecter dump_active .. 0 homeless 2024-09-16T03:49:22.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.206+0000 7f5fa5ffb640 10 client.784214.objecter ms_handle_connect 0x7f5fb006dac0 2024-09-16T03:49:22.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.206+0000 7f5fa5ffb640 10 client.784214.objecter resend_mon_ops 2024-09-16T03:49:22.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.206+0000 7f5fb4dbe640 1 --2- 172.21.15.42:0/3868083236 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5fb006dac0 0x7f5fb0121910 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f5fb0071800 tx=0x7f5fac00f8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:22.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.206+0000 7f5fb5dc0640 1 --2- 172.21.15.42:0/3868083236 >> v2:172.21.15.154:6812/254845342 conn(0x7f5f94037810 0x7f5f94039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:22.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.206+0000 7f5fa5ffb640 1 -- 172.21.15.42:0/3868083236 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fac013810 con 0x7f5fb006dac0 2024-09-16T03:49:22.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.206+0000 7f5fa5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:22.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.206+0000 7f5fa5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:49:22.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.206+0000 7f5fa5ffb640 10 monclient: dump: 2024-09-16T03:49:22.240 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:49:22.240 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:49:22.240 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:22.240 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:22.240 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:22.240 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:49:22.240 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:49:22.240 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:49:22.241 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:49:22.241 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:49:22.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.206+0000 7f5fb5dc0640 10 monclient: get_auth_request con 0x7f5f94037810 auth_method 0 2024-09-16T03:49:22.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.206+0000 7f5fa5ffb640 1 -- 172.21.15.42:0/3868083236 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f5fac018070 con 0x7f5fb006dac0 2024-09-16T03:49:22.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.206+0000 7f5fa5ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:49:22.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.206+0000 7f5fa5ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:49:22.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.206+0000 7f5fa5ffb640 1 -- 172.21.15.42:0/3868083236 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5fb006dac0 msgr2=0x7f5fb0121910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:22.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.206+0000 7f5fa5ffb640 1 --2- 172.21.15.42:0/3868083236 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5fb006dac0 0x7f5fb0121910 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f5fb0071800 tx=0x7f5fac00f8a0 comp rx=0 tx=0).stop 2024-09-16T03:49:22.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.206+0000 7f5fa5ffb640 1 -- 172.21.15.42:0/3868083236 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5fb006dac0 msgr2=0x7f5fb0121910 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:49:22.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.206+0000 7f5fa5ffb640 1 --2- 172.21.15.42:0/3868083236 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5fb006dac0 0x7f5fb0121910 secure :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f5fb0071800 tx=0x7f5fac00f8a0 comp rx=0 tx=0).stop 2024-09-16T03:49:22.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.210+0000 7f5fa5ffb640 10 client.784214.objecter ms_handle_connect 0x7f5f94037810 2024-09-16T03:49:22.242 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.210+0000 7f5fb5dc0640 1 --2- 172.21.15.42:0/3868083236 >> v2:172.21.15.154:6812/254845342 conn(0x7f5f94037810 0x7f5f94039cd0 secure :-1 s=READY pgs=3511 cs=0 l=1 rev1=1 crypto rx=0x7f5fa80af000 tx=0x7f5fa80a7070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:22.242 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.214+0000 7f5fb7823640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:49:22.242 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.214+0000 7f5fb7823640 1 --2- 172.21.15.42:0/3868083236 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5fb002c650 0x7f5fb002ca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:22.242 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.214+0000 7f5fb7823640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:22.242 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.214+0000 7f5fb7823640 1 -- 172.21.15.42:0/3868083236 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f5fb0138a40 con 0x7f5fb002c650 2024-09-16T03:49:22.242 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.214+0000 7f5fb4dbe640 1 --2- 172.21.15.42:0/3868083236 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5fb002c650 0x7f5fb002ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:22.242 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.214+0000 7f5fb4dbe640 10 monclient: get_auth_request con 0x7f5fb002c650 auth_method 0 2024-09-16T03:49:22.242 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.214+0000 7f5fb4dbe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:22.242 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.214+0000 7f5fb4dbe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:22.242 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.214+0000 7f5fb4dbe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:22.242 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.214+0000 7f5fb4dbe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:22.243 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.214+0000 7f5fb4dbe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:22.243 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.214+0000 7f5fb4dbe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:22.243 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.214+0000 7f5fb4dbe640 10 monclient(hunting): handle_auth_done global_id 784220 payload 824 2024-09-16T03:49:22.243 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.214+0000 7f5fb4dbe640 1 -- 172.21.15.42:0/3868083236 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f5fb0069040 con 0x7f5fb002c650 2024-09-16T03:49:22.243 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.214+0000 7f5fb4dbe640 1 --2- 172.21.15.42:0/3868083236 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5fb002c650 0x7f5fb002ca30 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f5fac017000 tx=0x7f5fac0046d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:22.243 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.214+0000 7f5fa5ffb640 10 client.784214.objecter ms_handle_connect 0x7f5fb002c650 2024-09-16T03:49:22.243 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.214+0000 7f5fa5ffb640 10 client.784214.objecter resend_mon_ops 2024-09-16T03:49:22.243 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.214+0000 7f5fa5ffb640 1 -- 172.21.15.42:0/3868083236 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fac01db90 con 0x7f5fb002c650 2024-09-16T03:49:22.243 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.214+0000 7f5fa5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:22.243 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.214+0000 7f5fa5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:49:22.243 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.214+0000 7f5fa5ffb640 10 monclient: dump: 2024-09-16T03:49:22.244 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:49:22.244 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:49:22.244 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:22.244 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:22.244 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:22.244 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:49:22.244 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:49:22.244 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:49:22.244 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:49:22.244 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:49:22.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.214+0000 7f5fa5ffb640 1 -- 172.21.15.42:0/3868083236 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4711 (secure 0 0 0) 0x7f5fac01db90 con 0x7f5fb002c650 2024-09-16T03:49:22.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.214+0000 7f5fa5ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:49:22.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.214+0000 7f5fa5ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:49:22.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.214+0000 7f5fa5ffb640 1 -- 172.21.15.42:0/3868083236 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5fb002c650 msgr2=0x7f5fb002ca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:22.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.214+0000 7f5fa5ffb640 1 --2- 172.21.15.42:0/3868083236 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5fb002c650 0x7f5fb002ca30 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f5fac017000 tx=0x7f5fac0046d0 comp rx=0 tx=0).stop 2024-09-16T03:49:22.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.214+0000 7f5fa5ffb640 1 -- 172.21.15.42:0/3868083236 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5fb002c650 msgr2=0x7f5fb002ca30 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:49:22.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.214+0000 7f5fa5ffb640 1 --2- 172.21.15.42:0/3868083236 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5fb002c650 0x7f5fb002ca30 secure :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f5fac017000 tx=0x7f5fac0046d0 comp rx=0 tx=0).stop 2024-09-16T03:49:22.245 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:49:22.245 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:49:22.245 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:49:22.246 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T03:49:22.246 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 66, 2024-09-16T03:49:22.246 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 54115, 2024-09-16T03:49:22.246 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:49:22.246 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:49:22.246 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T03:49:22.246 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T03:49:22.246 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:49:22.246 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 0, 2024-09-16T03:49:22.246 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:49:22.246 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:49:22.246 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:49:22.247 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:49:22.247 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:49:22.247 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:49:22.247 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:49:22.247 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:49:22.247 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:49:22.247 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:49:22.247 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:49:22.247 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:49:22.247 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:49:22.247 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:49:22.248 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:49:22.248 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:49:22.248 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:49:22.248 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:49:22.248 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:49:22.248 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:49:22.248 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:49:22.248 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:49:22.248 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:49:22.248 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:49:22.249 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:49:22.249 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:49:22.249 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:49:22.249 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:49:22.249 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:49:22.249 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:49:22.249 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:49:22.249 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:49:22.249 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:49:22.249 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:49:22.249 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:49:22.250 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:49:22.250 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:49:22.250 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:49:22.250 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:49:22.250 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:49:22.250 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:49:22.250 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:49:22.250 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:49:22.250 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:49:22.250 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:49:22.251 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:49:22.251 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:49:22.251 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:49:22.251 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:49:22.251 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:49:22.251 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:49:22.251 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:49:22.251 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:49:22.251 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:49:22.251 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:49:22.252 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:49:22.252 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:49:22.252 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:49:22.252 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:49:22.252 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:49:22.252 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:49:22.252 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:49:22.252 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:49:22.253 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:49:22.253 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:49:22.253 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:49:22.253 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:49:22.253 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:49:22.253 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:49:22.253 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:49:22.253 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:49:22.253 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:49:22.253 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:49:22.253 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:49:22.253 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:49:22.253 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:49:22.254 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:49:22.254 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:49:22.254 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:49:22.254 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:49:22.254 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:49:22.254 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:49:22.254 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:49:22.254 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:49:22.254 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:49:22.254 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:49:22.254 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:49:22.254 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:49:22.254 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:49:22.254 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:49:22.255 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:49:22.255 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:49:22.255 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:49:22.255 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:49:22.255 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:49:22.255 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:49:22.255 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:49:22.255 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:49:22.255 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:49:22.255 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:49:22.255 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:49:22.255 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:49:22.255 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:49:22.255 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:49:22.256 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:49:22.256 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:49:22.256 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:49:22.256 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:49:22.256 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:49:22.256 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:49:22.256 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:49:22.256 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:49:22.256 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:49:22.256 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:49:22.256 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:49:22.256 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:49:22.256 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:49:22.256 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:49:22.256 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:49:22.257 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:49:22.257 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:49:22.257 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:49:22.257 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:49:22.257 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:49:22.257 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:49:22.257 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:49:22.257 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:49:22.257 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:49:22.257 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:49:22.257 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:49:22.257 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:49:22.257 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:49:22.257 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:49:22.257 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:49:22.258 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:49:22.258 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:49:22.258 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:49:22.258 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:49:22.258 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:49:22.258 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:49:22.258 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:49:22.258 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T03:49:22.258 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:49:22.258 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:49:22.258 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:49:22.258 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:49:22.259 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:49:22.259 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:49:22.259 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:49:22.259 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:49:22.259 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:49:22.259 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:49:22.259 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 3 2024-09-16T03:49:22.259 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:49:22.259 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:49:22.259 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:49:22.259 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:49:22.259 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:49:22.261 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.214+0000 7f5fb7823640 10 librados: watch_flush enter 2024-09-16T03:49:22.261 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.214+0000 7f5fb7823640 10 librados: watch_flush exit 2024-09-16T03:49:22.261 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.214+0000 7f5fb7823640 20 client.784214.objecter shutdown clearing up homeless session... 2024-09-16T03:49:22.261 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.214+0000 7f5fb7823640 10 client.784214.objecter successfully canceled tick 2024-09-16T03:49:22.261 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.214+0000 7f5fb7823640 1 -- 172.21.15.42:0/3868083236 >> v2:172.21.15.154:6812/254845342 conn(0x7f5f94037810 msgr2=0x7f5f94039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:22.261 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.214+0000 7f5fb7823640 1 --2- 172.21.15.42:0/3868083236 >> v2:172.21.15.154:6812/254845342 conn(0x7f5f94037810 0x7f5f94039cd0 secure :-1 s=READY pgs=3511 cs=0 l=1 rev1=1 crypto rx=0x7f5fa80af000 tx=0x7f5fa80a7070 comp rx=0 tx=0).stop 2024-09-16T03:49:22.262 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.214+0000 7f5fb7823640 10 monclient: shutdown 2024-09-16T03:49:22.262 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.214+0000 7f5fb7823640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:49:22.262 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.214+0000 7f5fb7823640 1 -- 172.21.15.42:0/3868083236 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5fb00a1330 msgr2=0x7f5fb01cecf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:22.262 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.214+0000 7f5fb7823640 1 --2- 172.21.15.42:0/3868083236 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5fb00a1330 0x7f5fb01cecf0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f5fa000c450 tx=0x7f5fa000c910 comp rx=0 tx=0).stop 2024-09-16T03:49:22.262 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.214+0000 7f5fb5dc0640 1 -- 172.21.15.42:0/3868083236 reap_dead start 2024-09-16T03:49:22.262 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.214+0000 7f5fb7823640 1 -- 172.21.15.42:0/3868083236 shutdown_connections 2024-09-16T03:49:22.262 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.214+0000 7f5fb7823640 1 --2- 172.21.15.42:0/3868083236 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5fb00a1330 0x7f5fb01cecf0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:22.262 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.214+0000 7f5fb7823640 1 -- 172.21.15.42:0/3868083236 >> 172.21.15.42:0/3868083236 conn(0x7f5fb00a2420 msgr2=0x7f5fb00a4dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:49:22.262 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.214+0000 7f5fb7823640 1 -- 172.21.15.42:0/3868083236 shutdown_connections 2024-09-16T03:49:22.262 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.214+0000 7f5fb7823640 1 -- 172.21.15.42:0/3868083236 wait complete. 2024-09-16T03:49:22.262 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.214+0000 7f5fb7823640 1 librados: shutdown 2024-09-16T03:49:22.263 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:49:22.263 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T03:49:22.263 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T03:49:22.385 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.382+0000 7f2d3dbad640 10 monclient: get_monmap_and_config 2024-09-16T03:49:22.386 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.382+0000 7f2d3dbad640 10 monclient: build_initial_monmap 2024-09-16T03:49:22.386 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.382+0000 7f2d3dbad640 10 monclient: monmap: 2024-09-16T03:49:22.386 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:49:22.386 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:49:22.386 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:49:22.384910+0000 2024-09-16T03:49:22.386 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:49:22.384910+0000 2024-09-16T03:49:22.386 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:49:22.386 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:49:22.386 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:49:22.386 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:49:22.387 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:49:22.387 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:49:22.387 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:49:22.387 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:49:22.389 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.382+0000 7f2d3dbad640 1 Processor -- start 2024-09-16T03:49:22.389 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.382+0000 7f2d3dbad640 1 -- start start 2024-09-16T03:49:22.389 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.382+0000 7f2d3dbad640 10 monclient: init 2024-09-16T03:49:22.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.382+0000 7f2d3dbad640 10 monclient: _reopen_session rank -1 2024-09-16T03:49:22.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.382+0000 7f2d3dbad640 10 monclient: _add_conns ranks=[2,4,1,3,0] 2024-09-16T03:49:22.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.382+0000 7f2d3dbad640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2d380a39b0 0x7f2d38133b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:22.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.382+0000 7f2d3dbad640 10 monclient(hunting): picked mon.noname-b con 0x7f2d380a39b0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:49:22.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.382+0000 7f2d3dbad640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f2d380a3d90 0x7f2d38137000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:22.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.382+0000 7f2d3dbad640 10 monclient(hunting): picked mon.noname-c con 0x7f2d380a3d90 addr v2:172.21.15.154:3300/0 2024-09-16T03:49:22.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.382+0000 7f2d3dbad640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2d38137540 addr v1:172.21.15.154:6789/0 2024-09-16T03:49:22.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.382+0000 7f2d3dbad640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2d3809c140 con 0x7f2d38137540 2024-09-16T03:49:22.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.382+0000 7f2d3dbad640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:22.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.382+0000 7f2d3dbad640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f2d3809c420 con 0x7f2d380a3d90 2024-09-16T03:49:22.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.382+0000 7f2d3dbad640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:22.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.382+0000 7f2d3dbad640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f2d3806dd60 con 0x7f2d380a39b0 2024-09-16T03:49:22.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.382+0000 7f2d3dbad640 10 monclient(hunting): _renew_subs 2024-09-16T03:49:22.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.382+0000 7f2d3dbad640 10 monclient(hunting): authenticate will time out at 1409.354362s 2024-09-16T03:49:22.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.382+0000 7f2d377fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2d380a39b0 0x7f2d38133b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:22.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.382+0000 7f2d377fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2d380a39b0 0x7f2d38133b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:33382/0 (socket says 172.21.15.42:33382) 2024-09-16T03:49:22.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.382+0000 7f2d377fe640 1 -- 172.21.15.42:0/1036362674 learned_addr learned my addr 172.21.15.42:0/1036362674 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:49:22.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.382+0000 7f2d377fe640 10 monclient(hunting): get_auth_request con 0x7f2d380a39b0 auth_method 0 2024-09-16T03:49:22.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.382+0000 7f2d377fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:22.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.382+0000 7f2d377fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:22.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.382+0000 7f2d377fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:22.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.382+0000 7f2d377fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:22.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.382+0000 7f2d377fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:22.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.382+0000 7f2d377fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:22.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.382+0000 7f2d36ffd640 1 --2- 172.21.15.42:0/1036362674 >> v2:172.21.15.154:3300/0 conn(0x7f2d380a3d90 0x7f2d38137000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:22.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.382+0000 7f2d36ffd640 10 monclient(hunting): get_auth_request con 0x7f2d380a3d90 auth_method 0 2024-09-16T03:49:22.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.382+0000 7f2d36ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:22.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.382+0000 7f2d36ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:22.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.382+0000 7f2d36ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:22.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.382+0000 7f2d377fe640 10 monclient(hunting): handle_auth_done global_id 784223 payload 274 2024-09-16T03:49:22.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.382+0000 7f2d377fe640 1 -- 172.21.15.42:0/1036362674 >> v2:172.21.15.154:3300/0 conn(0x7f2d380a3d90 msgr2=0x7f2d38137000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:49:22.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.382+0000 7f2d377fe640 1 --2- 172.21.15.42:0/1036362674 >> v2:172.21.15.154:3300/0 conn(0x7f2d380a3d90 0x7f2d38137000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:22.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.382+0000 7f2d377fe640 1 -- 172.21.15.42:0/1036362674 >> v1:172.21.15.154:6789/0 conn(0x7f2d38137540 legacy=0x7f2d3813e390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:49:22.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.382+0000 7f2d377fe640 10 monclient: _finish_hunting 0 2024-09-16T03:49:22.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.382+0000 7f2d377fe640 1 monclient: found mon.noname-b 2024-09-16T03:49:22.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.382+0000 7f2d377fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:49:22.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.382+0000 7f2d377fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T03:49:22.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.382+0000 7f2d377fe640 1 -- 172.21.15.42:0/1036362674 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d380a5e20 con 0x7f2d380a39b0 2024-09-16T03:49:22.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.386+0000 7f2d377fe640 1 --2- 172.21.15.42:0/1036362674 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2d380a39b0 0x7f2d38133b80 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f2d28009870 tx=0x7f2d2801c3e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f293ba6bb71cf22e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:22.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.386+0000 7f2d367fc640 1 -- 172.21.15.42:0/1036362674 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d280b0070 con 0x7f2d380a39b0 2024-09-16T03:49:22.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.386+0000 7f2d367fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:22.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.386+0000 7f2d367fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:49:22.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.386+0000 7f2d367fc640 10 monclient: dump: 2024-09-16T03:49:22.400 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:49:22.400 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:49:22.400 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:22.400 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:22.400 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:22.400 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:49:22.400 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:49:22.400 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:49:22.400 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:49:22.401 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:49:22.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.386+0000 7f2d367fc640 10 monclient: _finish_auth 0 2024-09-16T03:49:22.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.386+0000 7f2d367fc640 10 monclient: _check_auth_tickets 2024-09-16T03:49:22.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.386+0000 7f2d367fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:49:22.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.386+0000 7f2d367fc640 1 -- 172.21.15.42:0/1036362674 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d280ab600 con 0x7f2d380a39b0 2024-09-16T03:49:22.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.386+0000 7f2d367fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:49:22.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.386+0000 7f2d367fc640 1 -- 172.21.15.42:0/1036362674 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d280ab9d0 con 0x7f2d380a39b0 2024-09-16T03:49:22.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.386+0000 7f2d3dbad640 5 monclient: authenticate success, global_id 784223 2024-09-16T03:49:22.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.386+0000 7f2d3dbad640 10 monclient: get_monmap_and_config success 2024-09-16T03:49:22.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.386+0000 7f2d3dbad640 4 set_mon_vals no callback set 2024-09-16T03:49:22.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.386+0000 7f2d367fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:22.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.386+0000 7f2d367fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:49:22.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.386+0000 7f2d367fc640 10 monclient: dump: 2024-09-16T03:49:22.402 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:49:22.402 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:49:22.402 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:22.402 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:22.402 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:22.402 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:49:22.402 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:49:22.402 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:49:22.402 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:49:22.402 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:49:22.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.386+0000 7f2d3dbad640 10 monclient: shutdown 2024-09-16T03:49:22.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.386+0000 7f2d3dbad640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:49:22.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.386+0000 7f2d3dbad640 1 -- 172.21.15.42:0/1036362674 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2d380a39b0 msgr2=0x7f2d38133b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:22.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.386+0000 7f2d3dbad640 1 --2- 172.21.15.42:0/1036362674 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2d380a39b0 0x7f2d38133b80 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f2d28009870 tx=0x7f2d2801c3e0 comp rx=0 tx=0).stop 2024-09-16T03:49:22.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.386+0000 7f2d3dbad640 1 -- 172.21.15.42:0/1036362674 shutdown_connections 2024-09-16T03:49:22.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.386+0000 7f2d3dbad640 1 --2- 172.21.15.42:0/1036362674 >> v2:172.21.15.154:3300/0 conn(0x7f2d380a3d90 0x7f2d38137000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:22.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.386+0000 7f2d3dbad640 1 --2- 172.21.15.42:0/1036362674 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2d380a39b0 0x7f2d38133b80 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:22.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.386+0000 7f2d3dbad640 1 -- 172.21.15.42:0/1036362674 >> 172.21.15.42:0/1036362674 conn(0x7f2d380a9fb0 msgr2=0x7f2d380aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:49:22.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.386+0000 7f2d3dbad640 1 -- 172.21.15.42:0/1036362674 shutdown_connections 2024-09-16T03:49:22.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.386+0000 7f2d3dbad640 1 -- 172.21.15.42:0/1036362674 wait complete. 2024-09-16T03:49:22.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.386+0000 7f2d3dbad640 10 monclient: build_initial_monmap 2024-09-16T03:49:22.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.386+0000 7f2d3dbad640 10 monclient: monmap: 2024-09-16T03:49:22.403 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:49:22.403 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:49:22.403 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:49:22.404 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:49:22.404 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:49:22.404 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:49:22.404 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:49:22.404 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:49:22.404 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:49:22.404 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:49:22.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.386+0000 7f2d3dbad640 1 librados: starting msgr at 2024-09-16T03:49:22.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.386+0000 7f2d3dbad640 1 librados: starting objecter 2024-09-16T03:49:22.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.386+0000 7f2d3dbad640 1 Processor -- start 2024-09-16T03:49:22.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.386+0000 7f2d3dbad640 1 -- start start 2024-09-16T03:49:22.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.386+0000 7f2d3dbad640 1 librados: setting wanted keys 2024-09-16T03:49:22.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.386+0000 7f2d3dbad640 1 librados: calling monclient init 2024-09-16T03:49:22.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.386+0000 7f2d3dbad640 10 monclient: init 2024-09-16T03:49:22.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.386+0000 7f2d3dbad640 10 monclient: _reopen_session rank -1 2024-09-16T03:49:22.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.386+0000 7f2d3dbad640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:49:22.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.386+0000 7f2d3dbad640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2d380a39b0 0x7f2d3813e600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:22.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.386+0000 7f2d3dbad640 10 monclient(hunting): picked mon.noname-c con 0x7f2d380a39b0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:49:22.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.386+0000 7f2d3dbad640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d380a3d90 0x7f2d3813eb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:22.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.386+0000 7f2d3dbad640 10 monclient(hunting): picked mon.noname-b con 0x7f2d380a3d90 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:49:22.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.386+0000 7f2d3dbad640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2d38137540 0x7f2d380a1910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:22.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.386+0000 7f2d3dbad640 10 monclient(hunting): picked mon.noname-a con 0x7f2d38137540 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:49:22.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.386+0000 7f2d3dbad640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:22.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.386+0000 7f2d3dbad640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f2d381424e0 con 0x7f2d38137540 2024-09-16T03:49:22.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.386+0000 7f2d3dbad640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:22.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.386+0000 7f2d3dbad640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f2d38142660 con 0x7f2d380a3d90 2024-09-16T03:49:22.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.386+0000 7f2d3dbad640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:22.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.386+0000 7f2d3dbad640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f2d38136e90 con 0x7f2d380a39b0 2024-09-16T03:49:22.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.386+0000 7f2d3dbad640 10 monclient(hunting): _renew_subs 2024-09-16T03:49:22.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.386+0000 7f2d3dbad640 10 monclient(hunting): authenticate will time out at 1409.359918s 2024-09-16T03:49:22.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.390+0000 7f2d377fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2d380a39b0 0x7f2d3813e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:22.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.390+0000 7f2d37fff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2d38137540 0x7f2d380a1910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:22.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.390+0000 7f2d37fff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2d38137540 0x7f2d380a1910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:47068/0 (socket says 172.21.15.42:47068) 2024-09-16T03:49:22.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.390+0000 7f2d37fff640 1 -- 172.21.15.42:0/1428171669 learned_addr learned my addr 172.21.15.42:0/1428171669 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:49:22.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.390+0000 7f2d37fff640 10 monclient(hunting): get_auth_request con 0x7f2d38137540 auth_method 0 2024-09-16T03:49:22.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.390+0000 7f2d37fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:22.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.390+0000 7f2d37fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:22.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.390+0000 7f2d37fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:22.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.390+0000 7f2d377fe640 10 monclient(hunting): get_auth_request con 0x7f2d380a39b0 auth_method 0 2024-09-16T03:49:22.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.390+0000 7f2d377fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:22.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.390+0000 7f2d377fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:22.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.390+0000 7f2d377fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:22.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.390+0000 7f2d36ffd640 1 --2- 172.21.15.42:0/1428171669 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d380a3d90 0x7f2d3813eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:22.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.390+0000 7f2d377fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:22.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.390+0000 7f2d377fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:22.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.390+0000 7f2d377fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:22.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.390+0000 7f2d36ffd640 10 monclient(hunting): get_auth_request con 0x7f2d380a3d90 auth_method 0 2024-09-16T03:49:22.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.390+0000 7f2d36ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:22.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.390+0000 7f2d36ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:22.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.390+0000 7f2d36ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:22.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.390+0000 7f2d37fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:22.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.390+0000 7f2d37fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:22.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.390+0000 7f2d37fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:22.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.390+0000 7f2d377fe640 10 monclient(hunting): handle_auth_done global_id 784226 payload 824 2024-09-16T03:49:22.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.390+0000 7f2d377fe640 1 -- 172.21.15.42:0/1428171669 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d380a3d90 msgr2=0x7f2d3813eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:22.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.390+0000 7f2d36ffd640 1 -- 172.21.15.42:0/1428171669 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d380a3d90 msgr2=0x7f2d3813eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-16T03:49:22.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.390+0000 7f2d36ffd640 1 -- 172.21.15.42:0/1428171669 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d380a3d90 msgr2=0x7f2d3813eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T03:49:22.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.390+0000 7f2d36ffd640 1 --2- 172.21.15.42:0/1428171669 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d380a3d90 0x7f2d3813eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:49:22.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.390+0000 7f2d36ffd640 1 --2- 172.21.15.42:0/1428171669 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d380a3d90 0x7f2d3813eb40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:49:22.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.390+0000 7f2d377fe640 1 --2- 172.21.15.42:0/1428171669 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d380a3d90 0x7f2d3813eb40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:22.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.390+0000 7f2d377fe640 1 -- 172.21.15.42:0/1428171669 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2d38137540 msgr2=0x7f2d380a1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:22.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.390+0000 7f2d377fe640 1 --2- 172.21.15.42:0/1428171669 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2d38137540 0x7f2d380a1910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:22.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.390+0000 7f2d377fe640 10 monclient: _finish_hunting 0 2024-09-16T03:49:22.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.390+0000 7f2d377fe640 1 monclient: found mon.noname-c 2024-09-16T03:49:22.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.390+0000 7f2d377fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:49:22.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.390+0000 7f2d377fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:49:22.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.390+0000 7f2d377fe640 1 -- 172.21.15.42:0/1428171669 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d3813a0a0 con 0x7f2d380a39b0 2024-09-16T03:49:22.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.390+0000 7f2d377fe640 1 --2- 172.21.15.42:0/1428171669 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2d380a39b0 0x7f2d3813e600 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f2d3806dee0 tx=0x7f2d28019180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:22.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.390+0000 7f2d1ffff640 10 client.?.objecter ms_handle_connect 0x7f2d380a39b0 2024-09-16T03:49:22.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.390+0000 7f2d1ffff640 10 client.?.objecter resend_mon_ops 2024-09-16T03:49:22.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.390+0000 7f2d1ffff640 1 -- 172.21.15.42:0/1428171669 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d280b0040 con 0x7f2d380a39b0 2024-09-16T03:49:22.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.390+0000 7f2d1ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:22.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.390+0000 7f2d1ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:49:22.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.390+0000 7f2d1ffff640 10 monclient: dump: 2024-09-16T03:49:22.410 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:49:22.410 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:49:22.410 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:22.410 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:22.410 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:22.410 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:49:22.410 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:49:22.410 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:49:22.410 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:49:22.410 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:49:22.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.390+0000 7f2d1ffff640 10 monclient: _finish_auth 0 2024-09-16T03:49:22.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.390+0000 7f2d1ffff640 10 monclient: _check_auth_tickets 2024-09-16T03:49:22.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.390+0000 7f2d1ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:49:22.411 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.390+0000 7f2d1ffff640 1 -- 172.21.15.42:0/1428171669 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d280b9070 con 0x7f2d380a39b0 2024-09-16T03:49:22.411 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.390+0000 7f2d1ffff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:49:22.411 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.390+0000 7f2d1ffff640 1 -- 172.21.15.42:0/1428171669 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d280b4880 con 0x7f2d380a39b0 2024-09-16T03:49:22.411 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.390+0000 7f2d3dbad640 5 monclient: authenticate success, global_id 784226 2024-09-16T03:49:22.411 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.390+0000 7f2d3dbad640 10 monclient: _renew_subs 2024-09-16T03:49:22.411 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.390+0000 7f2d3dbad640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:49:22.411 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.390+0000 7f2d3dbad640 1 -- 172.21.15.42:0/1428171669 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d3806f680 con 0x7f2d380a39b0 2024-09-16T03:49:22.411 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.390+0000 7f2d1ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:22.411 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.390+0000 7f2d1ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:49:22.411 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.390+0000 7f2d1ffff640 10 monclient: dump: 2024-09-16T03:49:22.411 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:49:22.411 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:49:22.411 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:22.412 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:22.412 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:22.412 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:49:22.412 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:49:22.412 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:49:22.412 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:49:22.412 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:49:22.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.390+0000 7f2d3dbad640 10 client.784226.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:49:22.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.390+0000 7f2d3dbad640 10 monclient: _renew_subs 2024-09-16T03:49:22.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.390+0000 7f2d3dbad640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:49:22.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.390+0000 7f2d3dbad640 1 -- 172.21.15.42:0/1428171669 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d38139dc0 con 0x7f2d380a39b0 2024-09-16T03:49:22.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.390+0000 7f2d3dbad640 1 librados: init done 2024-09-16T03:49:22.413 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.390+0000 7f2d35ffb640 4 set_mon_vals no callback set 2024-09-16T03:49:22.413 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.390+0000 7f2d3dbad640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:49:22.413 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.390+0000 7f2d3dbad640 1 --2- 172.21.15.42:0/1428171669 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d381cfa90 0x7f2d380a1e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:22.413 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.390+0000 7f2d3dbad640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:22.413 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.390+0000 7f2d3dbad640 1 -- 172.21.15.42:0/1428171669 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f2d381342d0 con 0x7f2d381cfa90 2024-09-16T03:49:22.413 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.390+0000 7f2d1ffff640 1 -- 172.21.15.42:0/1428171669 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f2d280ab4c0 con 0x7f2d380a39b0 2024-09-16T03:49:22.413 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.390+0000 7f2d1ffff640 1 --2- 172.21.15.42:0/1428171669 >> v2:172.21.15.154:6812/254845342 conn(0x7f2d24037810 0x7f2d24039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:22.413 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.390+0000 7f2d36ffd640 1 --2- 172.21.15.42:0/1428171669 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d381cfa90 0x7f2d380a1e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:22.413 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.390+0000 7f2d36ffd640 10 monclient: get_auth_request con 0x7f2d381cfa90 auth_method 0 2024-09-16T03:49:22.413 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.390+0000 7f2d36ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:22.414 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.390+0000 7f2d36ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:22.414 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.390+0000 7f2d36ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:22.414 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.390+0000 7f2d1ffff640 1 -- 172.21.15.42:0/1428171669 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(244..244 src has 1..244) ==== 6108+0+0 (secure 0 0 0) 0x7f2d280e2d80 con 0x7f2d380a39b0 2024-09-16T03:49:22.414 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.390+0000 7f2d1ffff640 10 client.784226.objecter ms_dispatch 0x7f2d38000ba0 osd_map(244..244 src has 1..244) 2024-09-16T03:49:22.414 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.390+0000 7f2d1ffff640 3 client.784226.objecter handle_osd_map got epochs [244,244] > 0 2024-09-16T03:49:22.414 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.390+0000 7f2d1ffff640 3 client.784226.objecter handle_osd_map decoding full epoch 244 2024-09-16T03:49:22.414 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.390+0000 7f2d1ffff640 20 client.784226.objecter dump_active .. 0 homeless 2024-09-16T03:49:22.414 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.390+0000 7f2d36ffd640 1 -- 172.21.15.42:0/1428171669 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d381cfa90 msgr2=0x7f2d380a1e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-16T03:49:22.414 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.390+0000 7f2d36ffd640 1 -- 172.21.15.42:0/1428171669 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d381cfa90 msgr2=0x7f2d380a1e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T03:49:22.414 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.390+0000 7f2d36ffd640 1 --2- 172.21.15.42:0/1428171669 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d381cfa90 0x7f2d380a1e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:49:22.414 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.390+0000 7f2d36ffd640 1 --2- 172.21.15.42:0/1428171669 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d381cfa90 0x7f2d380a1e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:49:22.414 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.390+0000 7f2d37fff640 1 --2- 172.21.15.42:0/1428171669 >> v2:172.21.15.154:6812/254845342 conn(0x7f2d24037810 0x7f2d24039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:22.414 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.390+0000 7f2d37fff640 10 monclient: get_auth_request con 0x7f2d24037810 auth_method 0 2024-09-16T03:49:22.415 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.390+0000 7f2d1ffff640 10 client.784226.objecter ms_handle_connect 0x7f2d24037810 2024-09-16T03:49:22.415 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.390+0000 7f2d37fff640 1 --2- 172.21.15.42:0/1428171669 >> v2:172.21.15.154:6812/254845342 conn(0x7f2d24037810 0x7f2d24039cd0 secure :-1 s=READY pgs=3512 cs=0 l=1 rev1=1 crypto rx=0x7f2d300056c0 tx=0x7f2d30003df0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:22.598 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.594+0000 7f2d36ffd640 1 --2- 172.21.15.42:0/1428171669 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d381cfa90 0x7f2d380a1e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:22.598 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.594+0000 7f2d36ffd640 10 monclient: get_auth_request con 0x7f2d381cfa90 auth_method 0 2024-09-16T03:49:22.598 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.594+0000 7f2d36ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:22.598 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.594+0000 7f2d36ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:22.598 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.594+0000 7f2d36ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:49:22.598 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.594+0000 7f2d36ffd640 1 -- 172.21.15.42:0/1428171669 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d381cfa90 msgr2=0x7f2d380a1e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-16T03:49:22.598 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.594+0000 7f2d36ffd640 1 -- 172.21.15.42:0/1428171669 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d381cfa90 msgr2=0x7f2d380a1e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T03:49:22.598 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.594+0000 7f2d36ffd640 1 --2- 172.21.15.42:0/1428171669 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d381cfa90 0x7f2d380a1e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:49:22.598 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.594+0000 7f2d36ffd640 1 --2- 172.21.15.42:0/1428171669 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d381cfa90 0x7f2d380a1e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T03:49:23.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.998+0000 7f2d36ffd640 1 --2- 172.21.15.42:0/1428171669 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d381cfa90 0x7f2d380a1e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:23.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.998+0000 7f2d36ffd640 10 monclient: get_auth_request con 0x7f2d381cfa90 auth_method 0 2024-09-16T03:49:23.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.998+0000 7f2d36ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:23.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.998+0000 7f2d36ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:23.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.998+0000 7f2d36ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:49:23.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.998+0000 7f2d36ffd640 1 -- 172.21.15.42:0/1428171669 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d381cfa90 msgr2=0x7f2d380a1e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-16T03:49:23.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.998+0000 7f2d36ffd640 1 -- 172.21.15.42:0/1428171669 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d381cfa90 msgr2=0x7f2d380a1e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T03:49:23.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.998+0000 7f2d36ffd640 1 --2- 172.21.15.42:0/1428171669 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d381cfa90 0x7f2d380a1e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:49:23.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:22.998+0000 7f2d36ffd640 1 --2- 172.21.15.42:0/1428171669 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d381cfa90 0x7f2d380a1e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T03:49:23.806 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.802+0000 7f2d36ffd640 1 --2- 172.21.15.42:0/1428171669 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d381cfa90 0x7f2d380a1e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:23.806 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.802+0000 7f2d36ffd640 10 monclient: get_auth_request con 0x7f2d381cfa90 auth_method 0 2024-09-16T03:49:23.806 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.802+0000 7f2d36ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:23.806 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.802+0000 7f2d36ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:23.807 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.802+0000 7f2d36ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:49:23.807 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.802+0000 7f2d36ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:23.807 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.802+0000 7f2d36ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:23.807 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.802+0000 7f2d36ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:23.807 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.802+0000 7f2d36ffd640 10 monclient(hunting): handle_auth_done global_id 794098 payload 824 2024-09-16T03:49:23.807 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.802+0000 7f2d36ffd640 1 -- 172.21.15.42:0/1428171669 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2d38133b80 con 0x7f2d381cfa90 2024-09-16T03:49:23.808 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.802+0000 7f2d36ffd640 1 --2- 172.21.15.42:0/1428171669 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d381cfa90 0x7f2d380a1e50 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f2d38139660 tx=0x7f2d2c015b60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:23.808 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.802+0000 7f2d1ffff640 10 client.784226.objecter ms_handle_connect 0x7f2d381cfa90 2024-09-16T03:49:23.808 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.802+0000 7f2d1ffff640 10 client.784226.objecter resend_mon_ops 2024-09-16T03:49:23.808 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.802+0000 7f2d1ffff640 1 -- 172.21.15.42:0/1428171669 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d2c011910 con 0x7f2d381cfa90 2024-09-16T03:49:23.808 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.802+0000 7f2d1ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:23.808 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.802+0000 7f2d1ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:49:23.808 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.802+0000 7f2d1ffff640 10 monclient: dump: 2024-09-16T03:49:23.808 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:49:23.808 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:49:23.808 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:23.809 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:23.809 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:23.809 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:49:23.809 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:49:23.809 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:49:23.809 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:49:23.809 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:49:23.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.802+0000 7f2d1ffff640 1 -- 172.21.15.42:0/1428171669 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f2d2c011910 con 0x7f2d381cfa90 2024-09-16T03:49:23.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.802+0000 7f2d1ffff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:49:23.810 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.802+0000 7f2d1ffff640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:49:23.810 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.802+0000 7f2d1ffff640 1 -- 172.21.15.42:0/1428171669 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d381cfa90 msgr2=0x7f2d380a1e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:23.810 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.802+0000 7f2d1ffff640 1 --2- 172.21.15.42:0/1428171669 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d381cfa90 0x7f2d380a1e50 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f2d38139660 tx=0x7f2d2c015b60 comp rx=0 tx=0).stop 2024-09-16T03:49:23.810 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.802+0000 7f2d1ffff640 1 -- 172.21.15.42:0/1428171669 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d381cfa90 msgr2=0x7f2d380a1e50 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:49:23.810 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.802+0000 7f2d1ffff640 1 --2- 172.21.15.42:0/1428171669 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d381cfa90 0x7f2d380a1e50 secure :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f2d38139660 tx=0x7f2d2c015b60 comp rx=0 tx=0).stop 2024-09-16T03:49:23.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.810+0000 7f2d3dbad640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:49:23.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.810+0000 7f2d3dbad640 1 --2- 172.21.15.42:0/1428171669 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d380a2390 0x7f2d3809d920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:23.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.810+0000 7f2d3dbad640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:23.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.810+0000 7f2d3dbad640 1 -- 172.21.15.42:0/1428171669 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f2d380a27a0 con 0x7f2d380a2390 2024-09-16T03:49:23.816 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.810+0000 7f2d36ffd640 1 --2- 172.21.15.42:0/1428171669 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d380a2390 0x7f2d3809d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:23.816 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.810+0000 7f2d36ffd640 10 monclient: get_auth_request con 0x7f2d380a2390 auth_method 0 2024-09-16T03:49:23.816 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.810+0000 7f2d36ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:23.816 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.810+0000 7f2d36ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:23.816 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.810+0000 7f2d36ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:23.816 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.810+0000 7f2d36ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:23.816 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.810+0000 7f2d36ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:23.816 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.810+0000 7f2d36ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:23.817 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.810+0000 7f2d36ffd640 10 monclient(hunting): handle_auth_done global_id 794101 payload 824 2024-09-16T03:49:23.817 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.810+0000 7f2d36ffd640 1 -- 172.21.15.42:0/1428171669 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f2d38133410 con 0x7f2d380a2390 2024-09-16T03:49:23.817 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.814+0000 7f2d36ffd640 1 --2- 172.21.15.42:0/1428171669 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d380a2390 0x7f2d3809d920 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f2d38139660 tx=0x7f2d2c004740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:23.817 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.814+0000 7f2d1ffff640 10 client.784226.objecter ms_handle_connect 0x7f2d380a2390 2024-09-16T03:49:23.817 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.814+0000 7f2d1ffff640 10 client.784226.objecter resend_mon_ops 2024-09-16T03:49:23.818 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.814+0000 7f2d1ffff640 1 -- 172.21.15.42:0/1428171669 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d2c017bb0 con 0x7f2d380a2390 2024-09-16T03:49:23.818 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.814+0000 7f2d1ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:23.818 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.814+0000 7f2d1ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:49:23.818 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.814+0000 7f2d1ffff640 10 monclient: dump: 2024-09-16T03:49:23.818 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:49:23.818 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:49:23.818 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:23.818 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:23.818 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:23.819 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:49:23.819 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:49:23.819 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:49:23.819 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:49:23.819 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:49:23.819 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.814+0000 7f2d1ffff640 1 -- 172.21.15.42:0/1428171669 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4543 (secure 0 0 0) 0x7f2d2c017bb0 con 0x7f2d380a2390 2024-09-16T03:49:23.819 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.814+0000 7f2d1ffff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:49:23.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.814+0000 7f2d1ffff640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:49:23.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.814+0000 7f2d1ffff640 1 -- 172.21.15.42:0/1428171669 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d380a2390 msgr2=0x7f2d3809d920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:23.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.814+0000 7f2d1ffff640 1 --2- 172.21.15.42:0/1428171669 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d380a2390 0x7f2d3809d920 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f2d38139660 tx=0x7f2d2c004740 comp rx=0 tx=0).stop 2024-09-16T03:49:23.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.814+0000 7f2d1ffff640 1 -- 172.21.15.42:0/1428171669 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d380a2390 msgr2=0x7f2d3809d920 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:49:23.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.814+0000 7f2d1ffff640 1 --2- 172.21.15.42:0/1428171669 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d380a2390 0x7f2d3809d920 secure :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f2d38139660 tx=0x7f2d2c004740 comp rx=0 tx=0).stop 2024-09-16T03:49:23.820 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:49:23.820 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:49:23.820 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:49:23.820 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T03:49:23.821 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 66, 2024-09-16T03:49:23.821 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 5651, 2024-09-16T03:49:23.821 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:49:23.821 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:49:23.821 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T03:49:23.821 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T03:49:23.821 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:49:23.821 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 2, 2024-09-16T03:49:23.821 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:49:23.821 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:49:23.821 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:49:23.821 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:49:23.822 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:49:23.822 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:49:23.822 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:49:23.822 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:49:23.822 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:49:23.822 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:49:23.822 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:49:23.822 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:49:23.822 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:49:23.822 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:49:23.822 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:49:23.822 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:49:23.822 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:49:23.822 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:49:23.823 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:49:23.823 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:49:23.823 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:49:23.823 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:49:23.823 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:49:23.823 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:49:23.823 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:49:23.823 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:49:23.823 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:49:23.823 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:49:23.823 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:49:23.823 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:49:23.823 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:49:23.823 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:49:23.824 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:49:23.824 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:49:23.824 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:49:23.824 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:49:23.824 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:49:23.824 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:49:23.824 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:49:23.824 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:49:23.824 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:49:23.824 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:49:23.824 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:49:23.824 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:49:23.825 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:49:23.825 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:49:23.825 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:49:23.825 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:49:23.825 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:49:23.825 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:49:23.825 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:49:23.825 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:49:23.825 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:49:23.825 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:49:23.825 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:49:23.825 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:49:23.825 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:49:23.826 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:49:23.826 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:49:23.826 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:49:23.826 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:49:23.826 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:49:23.826 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:49:23.826 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:49:23.826 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:49:23.826 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:49:23.826 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:49:23.826 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:49:23.827 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:49:23.827 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:49:23.827 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:49:23.827 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:49:23.827 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:49:23.827 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:49:23.827 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:49:23.827 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:49:23.827 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:49:23.827 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:49:23.827 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:49:23.827 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:49:23.827 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:49:23.827 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:49:23.828 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:49:23.828 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:49:23.828 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:49:23.828 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:49:23.828 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:49:23.828 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:49:23.828 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:49:23.828 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:49:23.828 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:49:23.828 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:49:23.828 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:49:23.828 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:49:23.828 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:49:23.829 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:49:23.829 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:49:23.829 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:49:23.829 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:49:23.829 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:49:23.829 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:49:23.829 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:49:23.829 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:49:23.829 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:49:23.829 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:49:23.829 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:49:23.829 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:49:23.829 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:49:23.830 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:49:23.830 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:49:23.830 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:49:23.830 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:49:23.830 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:49:23.830 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:49:23.830 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:49:23.830 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:49:23.830 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:49:23.830 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:49:23.830 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:49:23.830 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:49:23.830 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:49:23.830 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:49:23.831 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:49:23.831 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:49:23.831 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:49:23.831 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:49:23.831 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:49:23.831 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:49:23.831 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:49:23.831 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:49:23.831 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:49:23.831 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:49:23.831 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:49:23.831 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:49:23.831 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:49:23.831 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:49:23.832 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:49:23.832 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:49:23.832 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:49:23.832 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:49:23.832 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:49:23.832 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:49:23.832 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:49:23.832 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:49:23.832 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:49:23.832 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:49:23.832 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:49:23.832 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:49:23.832 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:49:23.833 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:49:23.833 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:49:23.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.814+0000 7f2d3dbad640 10 librados: watch_flush enter 2024-09-16T03:49:23.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.814+0000 7f2d3dbad640 10 librados: watch_flush exit 2024-09-16T03:49:23.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.814+0000 7f2d3dbad640 20 client.784226.objecter shutdown clearing up homeless session... 2024-09-16T03:49:23.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.814+0000 7f2d3dbad640 10 client.784226.objecter successfully canceled tick 2024-09-16T03:49:23.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.814+0000 7f2d3dbad640 1 -- 172.21.15.42:0/1428171669 >> v2:172.21.15.154:6812/254845342 conn(0x7f2d24037810 msgr2=0x7f2d24039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:23.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.814+0000 7f2d3dbad640 1 --2- 172.21.15.42:0/1428171669 >> v2:172.21.15.154:6812/254845342 conn(0x7f2d24037810 0x7f2d24039cd0 secure :-1 s=READY pgs=3512 cs=0 l=1 rev1=1 crypto rx=0x7f2d300056c0 tx=0x7f2d30003df0 comp rx=0 tx=0).stop 2024-09-16T03:49:23.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.814+0000 7f2d3dbad640 10 monclient: shutdown 2024-09-16T03:49:23.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.814+0000 7f2d3dbad640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:49:23.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.814+0000 7f2d3dbad640 1 -- 172.21.15.42:0/1428171669 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2d380a39b0 msgr2=0x7f2d3813e600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:23.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.814+0000 7f2d3dbad640 1 --2- 172.21.15.42:0/1428171669 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2d380a39b0 0x7f2d3813e600 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f2d3806dee0 tx=0x7f2d28019180 comp rx=0 tx=0).stop 2024-09-16T03:49:23.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.814+0000 7f2d37fff640 1 -- 172.21.15.42:0/1428171669 reap_dead start 2024-09-16T03:49:23.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.814+0000 7f2d37fff640 1 -- 172.21.15.42:0/1428171669 reap_dead start 2024-09-16T03:49:23.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.814+0000 7f2d3dbad640 1 -- 172.21.15.42:0/1428171669 shutdown_connections 2024-09-16T03:49:23.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.814+0000 7f2d3dbad640 1 -- 172.21.15.42:0/1428171669 >> 172.21.15.42:0/1428171669 conn(0x7f2d380a9fb0 msgr2=0x7f2d38131980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:49:23.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.814+0000 7f2d3dbad640 1 -- 172.21.15.42:0/1428171669 shutdown_connections 2024-09-16T03:49:23.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.814+0000 7f2d3dbad640 1 -- 172.21.15.42:0/1428171669 wait complete. 2024-09-16T03:49:23.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.814+0000 7f2d3dbad640 1 librados: shutdown 2024-09-16T03:49:23.850 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:49:23.850 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T03:49:23.850 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T03:49:23.984 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.978+0000 7f1fe6a60640 10 monclient: get_monmap_and_config 2024-09-16T03:49:23.984 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.978+0000 7f1fe6a60640 10 monclient: build_initial_monmap 2024-09-16T03:49:23.984 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.978+0000 7f1fe6a60640 10 monclient: monmap: 2024-09-16T03:49:23.984 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:49:23.984 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:49:23.985 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:49:23.983339+0000 2024-09-16T03:49:23.985 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:49:23.983339+0000 2024-09-16T03:49:23.985 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:49:23.985 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:49:23.985 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:49:23.985 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:49:23.985 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:49:23.985 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:49:23.985 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:49:23.985 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:49:23.987 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.978+0000 7f1fe6a60640 1 Processor -- start 2024-09-16T03:49:23.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.982+0000 7f1fe6a60640 1 -- start start 2024-09-16T03:49:23.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.982+0000 7f1fe6a60640 10 monclient: init 2024-09-16T03:49:23.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.982+0000 7f1fe6a60640 10 monclient: _reopen_session rank -1 2024-09-16T03:49:23.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.982+0000 7f1fe6a60640 10 monclient: _add_conns ranks=[3,1,4,2,0] 2024-09-16T03:49:23.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.982+0000 7f1fe6a60640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f1fe00a1330 0x7f1fe009f180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:23.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.982+0000 7f1fe6a60640 10 monclient(hunting): picked mon.noname-a con 0x7f1fe00a1330 addr v2:172.21.15.42:3300/0 2024-09-16T03:49:23.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.982+0000 7f1fe6a60640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1fe00a1710 addr v1:172.21.15.154:6789/0 2024-09-16T03:49:23.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.982+0000 7f1fe6a60640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f1fe01393a0 0x7f1fe013b790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:23.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.982+0000 7f1fe6a60640 10 monclient(hunting): picked mon.noname-c con 0x7f1fe01393a0 addr v2:172.21.15.154:3300/0 2024-09-16T03:49:23.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.982+0000 7f1fe6a60640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1fe009e4c0 con 0x7f1fe00a1710 2024-09-16T03:49:23.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.982+0000 7f1fe6a60640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:23.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.982+0000 7f1fe6a60640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f1fe009edf0 con 0x7f1fe00a1330 2024-09-16T03:49:23.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.982+0000 7f1fe6a60640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:23.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.982+0000 7f1fe6a60640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f1fe009cd60 con 0x7f1fe01393a0 2024-09-16T03:49:23.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.982+0000 7f1fe6a60640 10 monclient(hunting): _renew_subs 2024-09-16T03:49:23.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.982+0000 7f1fe6a60640 10 monclient(hunting): authenticate will time out at 1410.952800s 2024-09-16T03:49:23.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.982+0000 7f1fdffff640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f1fe00a1330 0x7f1fe009f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:23.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.982+0000 7f1fdffff640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f1fe00a1330 0x7f1fe009f180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:47072/0 (socket says 172.21.15.42:47072) 2024-09-16T03:49:23.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.982+0000 7f1fdffff640 1 -- 172.21.15.42:0/3782814431 learned_addr learned my addr 172.21.15.42:0/3782814431 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:49:23.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.982+0000 7f1fdffff640 10 monclient(hunting): get_auth_request con 0x7f1fe00a1330 auth_method 0 2024-09-16T03:49:23.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.982+0000 7f1fdffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:23.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.982+0000 7f1fdffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:23.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.982+0000 7f1fdffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:23.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.982+0000 7f1fdffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:23.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.982+0000 7f1fdffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:23.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.982+0000 7f1fdffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:23.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.982+0000 7f1fe4ffd640 1 --2- 172.21.15.42:0/3782814431 >> v2:172.21.15.154:3300/0 conn(0x7f1fe01393a0 0x7f1fe013b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:23.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.982+0000 7f1fe4ffd640 10 monclient(hunting): get_auth_request con 0x7f1fe01393a0 auth_method 0 2024-09-16T03:49:23.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.982+0000 7f1fe4ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:23.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.982+0000 7f1fe4ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:23.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.982+0000 7f1fe4ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:23.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.982+0000 7f1fdf7fe640 1 -- 172.21.15.42:0/3782814431 >> v1:172.21.15.154:6789/0 conn(0x7f1fe00a1710 legacy=0x7f1fe00a1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T03:49:23.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.982+0000 7f1fdf7fe640 1 -- 172.21.15.42:0/3782814431 >> v1:172.21.15.154:6789/0 conn(0x7f1fe00a1710 legacy=0x7f1fe00a1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:49:23.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.982+0000 7f1fdf7fe640 1 --1- 172.21.15.42:0/3782814431 >> v1:172.21.15.154:6789/0 conn(0x7f1fe00a1710 0x7f1fe00a1b40 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=0).handle_connect_reply_1 read connect reply failed 2024-09-16T03:49:23.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.982+0000 7f1fdffff640 10 monclient(hunting): handle_auth_done global_id 764643 payload 274 2024-09-16T03:49:23.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.982+0000 7f1fdffff640 1 -- 172.21.15.42:0/3782814431 >> v2:172.21.15.154:3300/0 conn(0x7f1fe01393a0 msgr2=0x7f1fe013b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:49:23.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.982+0000 7f1fdffff640 1 --2- 172.21.15.42:0/3782814431 >> v2:172.21.15.154:3300/0 conn(0x7f1fe01393a0 0x7f1fe013b790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:23.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.982+0000 7f1fdffff640 1 -- 172.21.15.42:0/3782814431 >> v1:172.21.15.154:6789/0 conn(0x7f1fe00a1710 legacy=0x7f1fe00a1b40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:49:23.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.982+0000 7f1fdffff640 10 monclient: _finish_hunting 0 2024-09-16T03:49:23.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.982+0000 7f1fdffff640 1 monclient: found mon.noname-a 2024-09-16T03:49:23.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.982+0000 7f1fdffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:49:23.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.982+0000 7f1fdffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:49:23.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.982+0000 7f1fdffff640 1 -- 172.21.15.42:0/3782814431 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1fe009f6c0 con 0x7f1fe00a1330 2024-09-16T03:49:23.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.982+0000 7f1fdffff640 1 --2- 172.21.15.42:0/3782814431 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1fe00a1330 0x7f1fe009f180 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f1fe009df50 tx=0x7f1fd001ea70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1f11e7bd24f078f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:23.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.982+0000 7f1fdeffd640 1 -- 172.21.15.42:0/3782814431 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fd00b4070 con 0x7f1fe00a1330 2024-09-16T03:49:23.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.982+0000 7f1fdeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:23.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.982+0000 7f1fdeffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:49:23.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.982+0000 7f1fdeffd640 10 monclient: dump: 2024-09-16T03:49:23.996 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:49:23.996 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:49:23.996 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:23.997 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:23.997 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:23.997 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:49:23.997 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:49:23.997 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:49:23.997 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:49:23.997 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:49:23.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.982+0000 7f1fdeffd640 10 monclient: _finish_auth 0 2024-09-16T03:49:23.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.982+0000 7f1fdeffd640 10 monclient: _check_auth_tickets 2024-09-16T03:49:23.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.982+0000 7f1fdeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:49:23.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.982+0000 7f1fe6a60640 5 monclient: authenticate success, global_id 764643 2024-09-16T03:49:23.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.982+0000 7f1fe6a60640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:49:23.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.982+0000 7f1fdeffd640 1 -- 172.21.15.42:0/3782814431 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1fd00a7950 con 0x7f1fe00a1330 2024-09-16T03:49:23.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.982+0000 7f1fdeffd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:49:23.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.982+0000 7f1fdeffd640 1 -- 172.21.15.42:0/3782814431 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fd00a7d20 con 0x7f1fe00a1330 2024-09-16T03:49:23.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.982+0000 7f1fdeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:23.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.982+0000 7f1fdeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:49:23.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.982+0000 7f1fdeffd640 10 monclient: dump: 2024-09-16T03:49:23.998 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:49:23.998 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:49:23.998 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:23.998 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:23.998 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:23.999 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:49:23.999 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:49:23.999 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:49:23.999 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:49:23.999 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:49:23.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.982+0000 7f1fe6a60640 10 monclient: get_monmap_and_config success 2024-09-16T03:49:23.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.982+0000 7f1fe6a60640 4 set_mon_vals no callback set 2024-09-16T03:49:23.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.986+0000 7f1fe6a60640 10 monclient: shutdown 2024-09-16T03:49:23.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.986+0000 7f1fe6a60640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:49:23.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.986+0000 7f1fe6a60640 1 -- 172.21.15.42:0/3782814431 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1fe00a1330 msgr2=0x7f1fe009f180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:24.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.986+0000 7f1fe6a60640 1 --2- 172.21.15.42:0/3782814431 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1fe00a1330 0x7f1fe009f180 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f1fe009df50 tx=0x7f1fd001ea70 comp rx=0 tx=0).stop 2024-09-16T03:49:24.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.986+0000 7f1fe6a60640 1 -- 172.21.15.42:0/3782814431 shutdown_connections 2024-09-16T03:49:24.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.986+0000 7f1fe6a60640 1 --2- 172.21.15.42:0/3782814431 >> v2:172.21.15.154:3300/0 conn(0x7f1fe01393a0 0x7f1fe013b790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:24.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.986+0000 7f1fe6a60640 1 --2- 172.21.15.42:0/3782814431 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1fe00a1330 0x7f1fe009f180 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:24.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.986+0000 7f1fe6a60640 1 -- 172.21.15.42:0/3782814431 >> 172.21.15.42:0/3782814431 conn(0x7f1fe00a2420 msgr2=0x7f1fe00a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:49:24.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.986+0000 7f1fe6a60640 1 -- 172.21.15.42:0/3782814431 shutdown_connections 2024-09-16T03:49:24.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.986+0000 7f1fe6a60640 1 -- 172.21.15.42:0/3782814431 wait complete. 2024-09-16T03:49:24.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.986+0000 7f1fe6a60640 10 monclient: build_initial_monmap 2024-09-16T03:49:24.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.986+0000 7f1fe6a60640 10 monclient: monmap: 2024-09-16T03:49:24.000 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:49:24.001 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:49:24.001 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:49:24.001 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:49:24.001 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:49:24.001 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:49:24.001 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:49:24.001 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:49:24.001 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:49:24.001 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:49:24.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.986+0000 7f1fe6a60640 1 librados: starting msgr at 2024-09-16T03:49:24.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.986+0000 7f1fe6a60640 1 librados: starting objecter 2024-09-16T03:49:24.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.986+0000 7f1fe6a60640 1 Processor -- start 2024-09-16T03:49:24.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.986+0000 7f1fe6a60640 1 -- start start 2024-09-16T03:49:24.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.986+0000 7f1fe6a60640 1 librados: setting wanted keys 2024-09-16T03:49:24.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.986+0000 7f1fe6a60640 1 librados: calling monclient init 2024-09-16T03:49:24.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.986+0000 7f1fe6a60640 10 monclient: init 2024-09-16T03:49:24.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.986+0000 7f1fe6a60640 10 monclient: _reopen_session rank -1 2024-09-16T03:49:24.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.986+0000 7f1fe6a60640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:49:24.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.986+0000 7f1fe6a60640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1fe00a1330 0x7f1fe011bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:24.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.986+0000 7f1fe6a60640 10 monclient(hunting): picked mon.noname-b con 0x7f1fe00a1330 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:49:24.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.986+0000 7f1fe6a60640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1fe00a1710 0x7f1fe0134750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:24.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.986+0000 7f1fe6a60640 10 monclient(hunting): picked mon.noname-a con 0x7f1fe00a1710 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:49:24.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.986+0000 7f1fe6a60640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1fe01393a0 0x7f1fe011c730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:24.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.986+0000 7f1fe6a60640 10 monclient(hunting): picked mon.noname-c con 0x7f1fe01393a0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:49:24.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.986+0000 7f1fe6a60640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:24.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.986+0000 7f1fe6a60640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f1fe009c630 con 0x7f1fe00a1710 2024-09-16T03:49:24.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.986+0000 7f1fe6a60640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:24.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.986+0000 7f1fe6a60640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f1fe009c4b0 con 0x7f1fe00a1330 2024-09-16T03:49:24.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.986+0000 7f1fe6a60640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:24.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.986+0000 7f1fe6a60640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f1fe009c7b0 con 0x7f1fe01393a0 2024-09-16T03:49:24.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.986+0000 7f1fe6a60640 10 monclient(hunting): _renew_subs 2024-09-16T03:49:24.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.986+0000 7f1fe6a60640 10 monclient(hunting): authenticate will time out at 1410.958033s 2024-09-16T03:49:24.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.986+0000 7f1fe4ffd640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1fe01393a0 0x7f1fe011c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:24.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.986+0000 7f1fdf7fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1fe00a1710 0x7f1fe0134750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:24.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.986+0000 7f1fdf7fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1fe00a1710 0x7f1fe0134750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:47088/0 (socket says 172.21.15.42:47088) 2024-09-16T03:49:24.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.986+0000 7f1fdf7fe640 1 -- 172.21.15.42:0/3448206890 learned_addr learned my addr 172.21.15.42:0/3448206890 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:49:24.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.986+0000 7f1fe4ffd640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1fe01393a0 0x7f1fe011c730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:33396/0 (socket says 172.21.15.42:33396) 2024-09-16T03:49:24.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.986+0000 7f1fdf7fe640 10 monclient(hunting): get_auth_request con 0x7f1fe00a1710 auth_method 0 2024-09-16T03:49:24.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.986+0000 7f1fdf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:24.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.986+0000 7f1fdf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:24.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.986+0000 7f1fdf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:24.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.986+0000 7f1fe4ffd640 10 monclient(hunting): get_auth_request con 0x7f1fe01393a0 auth_method 0 2024-09-16T03:49:24.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.986+0000 7f1fe4ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:24.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.986+0000 7f1fe4ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:24.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.986+0000 7f1fe4ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:24.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.986+0000 7f1fdf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:24.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.986+0000 7f1fdf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:24.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.986+0000 7f1fdf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:24.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.986+0000 7f1fe4ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:24.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.986+0000 7f1fe4ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:24.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.986+0000 7f1fe4ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:24.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.986+0000 7f1fdffff640 1 --2- 172.21.15.42:0/3448206890 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1fe00a1330 0x7f1fe011bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:24.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.986+0000 7f1fdffff640 10 monclient(hunting): get_auth_request con 0x7f1fe00a1330 auth_method 0 2024-09-16T03:49:24.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.986+0000 7f1fdffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:24.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.986+0000 7f1fdffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:24.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.986+0000 7f1fdffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:24.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.986+0000 7f1fdf7fe640 10 monclient(hunting): handle_auth_done global_id 764646 payload 824 2024-09-16T03:49:24.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.986+0000 7f1fdf7fe640 1 -- 172.21.15.42:0/3448206890 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1fe01393a0 msgr2=0x7f1fe011c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:24.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.986+0000 7f1fdf7fe640 1 --2- 172.21.15.42:0/3448206890 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1fe01393a0 0x7f1fe011c730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:24.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.986+0000 7f1fdf7fe640 1 -- 172.21.15.42:0/3448206890 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1fe00a1330 msgr2=0x7f1fe011bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:24.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.986+0000 7f1fdf7fe640 1 --2- 172.21.15.42:0/3448206890 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1fe00a1330 0x7f1fe011bf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:24.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.986+0000 7f1fdf7fe640 10 monclient: _finish_hunting 0 2024-09-16T03:49:24.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.986+0000 7f1fdf7fe640 1 monclient: found mon.noname-a 2024-09-16T03:49:24.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.986+0000 7f1fdf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:49:24.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.986+0000 7f1fdf7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:49:24.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.986+0000 7f1fdf7fe640 1 -- 172.21.15.42:0/3448206890 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1fe0072840 con 0x7f1fe00a1710 2024-09-16T03:49:24.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.986+0000 7f1fdf7fe640 1 --2- 172.21.15.42:0/3448206890 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1fe00a1710 0x7f1fe0134750 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f1fc800c460 tx=0x7f1fc800c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:24.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.986+0000 7f1fdcff9640 10 client.?.objecter ms_handle_connect 0x7f1fe00a1710 2024-09-16T03:49:24.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.986+0000 7f1fdcff9640 10 client.?.objecter resend_mon_ops 2024-09-16T03:49:24.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.986+0000 7f1fdcff9640 1 -- 172.21.15.42:0/3448206890 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fc8017070 con 0x7f1fe00a1710 2024-09-16T03:49:24.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.986+0000 7f1fdcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:24.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.986+0000 7f1fdcff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:49:24.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.986+0000 7f1fdcff9640 10 monclient: dump: 2024-09-16T03:49:24.007 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:49:24.007 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:49:24.007 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:24.007 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:24.007 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:24.007 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:49:24.007 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:49:24.007 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:49:24.008 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:49:24.008 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:49:24.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.986+0000 7f1fdcff9640 10 monclient: _finish_auth 0 2024-09-16T03:49:24.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.986+0000 7f1fdcff9640 10 monclient: _check_auth_tickets 2024-09-16T03:49:24.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.986+0000 7f1fdcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:49:24.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.986+0000 7f1fdcff9640 1 -- 172.21.15.42:0/3448206890 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1fc80074c0 con 0x7f1fe00a1710 2024-09-16T03:49:24.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.986+0000 7f1fe6a60640 5 monclient: authenticate success, global_id 764646 2024-09-16T03:49:24.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.986+0000 7f1fe6a60640 10 monclient: _renew_subs 2024-09-16T03:49:24.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.986+0000 7f1fe6a60640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:49:24.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.986+0000 7f1fe6a60640 1 -- 172.21.15.42:0/3448206890 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1fe0073a30 con 0x7f1fe00a1710 2024-09-16T03:49:24.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.986+0000 7f1fdcff9640 10 monclient: handle_config config(0 keys) 2024-09-16T03:49:24.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.986+0000 7f1fdcff9640 1 -- 172.21.15.42:0/3448206890 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fc8007670 con 0x7f1fe00a1710 2024-09-16T03:49:24.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.986+0000 7f1fdcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:24.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.986+0000 7f1fdcff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:49:24.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.986+0000 7f1fdcff9640 10 monclient: dump: 2024-09-16T03:49:24.009 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:49:24.009 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:49:24.009 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:24.009 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:24.009 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:24.009 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:49:24.009 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:49:24.009 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:49:24.009 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:49:24.009 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:49:24.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.986+0000 7f1fe6a60640 10 client.764646.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:49:24.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.986+0000 7f1fe6a60640 10 monclient: _renew_subs 2024-09-16T03:49:24.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.986+0000 7f1fe6a60640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:49:24.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.986+0000 7f1fe6a60640 1 -- 172.21.15.42:0/3448206890 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1fe0071970 con 0x7f1fe00a1710 2024-09-16T03:49:24.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.986+0000 7f1fde7fc640 4 set_mon_vals no callback set 2024-09-16T03:49:24.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.986+0000 7f1fe6a60640 1 librados: init done 2024-09-16T03:49:24.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.986+0000 7f1fe6a60640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:49:24.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.986+0000 7f1fe6a60640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:49:24.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.986+0000 7f1fe6a60640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:49:24.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.986+0000 7f1fe6a60640 1 -- 172.21.15.42:0/3448206890 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1fe009f180 con 0x7f1fe00a1710 2024-09-16T03:49:24.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.990+0000 7f1fdcff9640 1 -- 172.21.15.42:0/3448206890 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f1fc801a070 con 0x7f1fe00a1710 2024-09-16T03:49:24.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.990+0000 7f1fdcff9640 1 --2- 172.21.15.42:0/3448206890 >> v2:172.21.15.154:6812/254845342 conn(0x7f1fc4037810 0x7f1fc4039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:24.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.990+0000 7f1fdcff9640 1 -- 172.21.15.42:0/3448206890 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(245..245 src has 1..245) ==== 6524+0+0 (secure 0 0 0) 0x7f1fc8048580 con 0x7f1fe00a1710 2024-09-16T03:49:24.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.990+0000 7f1fdcff9640 10 client.764646.objecter ms_dispatch 0x7f1fe0000ba0 osd_map(245..245 src has 1..245) 2024-09-16T03:49:24.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.990+0000 7f1fdcff9640 3 client.764646.objecter handle_osd_map got epochs [245,245] > 0 2024-09-16T03:49:24.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.990+0000 7f1fdcff9640 3 client.764646.objecter handle_osd_map decoding full epoch 245 2024-09-16T03:49:24.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.990+0000 7f1fdcff9640 20 client.764646.objecter dump_active .. 0 homeless 2024-09-16T03:49:24.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.990+0000 7f1fdffff640 1 --2- 172.21.15.42:0/3448206890 >> v2:172.21.15.154:6812/254845342 conn(0x7f1fc4037810 0x7f1fc4039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:24.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.990+0000 7f1fdffff640 10 monclient: get_auth_request con 0x7f1fc4037810 auth_method 0 2024-09-16T03:49:24.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.990+0000 7f1fdffff640 1 --2- 172.21.15.42:0/3448206890 >> v2:172.21.15.154:6812/254845342 conn(0x7f1fc4037810 0x7f1fc4039cd0 secure :-1 s=READY pgs=3513 cs=0 l=1 rev1=1 crypto rx=0x7f1fd00b1000 tx=0x7f1fd00a9070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:24.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.990+0000 7f1fdcff9640 10 client.764646.objecter ms_handle_connect 0x7f1fc4037810 2024-09-16T03:49:24.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.994+0000 7f1fdcff9640 1 -- 172.21.15.42:0/3448206890 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f1fc8005a10 con 0x7f1fe00a1710 2024-09-16T03:49:24.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.994+0000 7f1fdcff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:49:24.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:23.994+0000 7f1fdcff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:49:24.223 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:24.218+0000 7f1fe6a60640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T03:49:24.223 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:24.218+0000 7f1fe6a60640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T03:49:24.223 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:24.218+0000 7f1fe6a60640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:49:24.223 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:24.218+0000 7f1fe6a60640 1 -- 172.21.15.42:0/3448206890 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f1fe0069040 con 0x7f1fe00a1710 2024-09-16T03:49:24.224 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:24.218+0000 7f1fdcff9640 1 -- 172.21.15.42:0/3448206890 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f1fc80117a0 con 0x7f1fe00a1710 2024-09-16T03:49:24.225 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:24.218+0000 7f1fdcff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T03:49:24.225 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:24.218+0000 7f1fdcff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:49:24.226 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.222+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:49:24.226 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.222+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=27,config=2,health=12,kv=10,logm=49} crc {auth=3539578372,config=692619927,health=2911299891,kv=3446188498,logm=2528175729}) 2024-09-16T03:49:24.226 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.222+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=24,config=2,health=12,kv=10,logm=52} crc {auth=1211043911,config=692619927,health=2174347234,kv=3446188498,logm=1492951816}) 2024-09-16T03:49:24.226 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.222+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:49:24.226 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.222+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=27,config=2,health=12,kv=10,logm=49} crc {auth=3539578372,config=692619927,health=2911299891,kv=3446188498,logm=2528175729}) 2024-09-16T03:49:24.226 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.222+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=26,config=1,health=11,kv=10,logm=52} crc {auth=1653729655,config=1228700967,health=2243981552,kv=2728889361,logm=2223596728}) 2024-09-16T03:49:24.227 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:24.222+0000 7f1fe6a60640 10 librados: watch_flush enter 2024-09-16T03:49:24.227 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:24.222+0000 7f1fe6a60640 10 librados: watch_flush exit 2024-09-16T03:49:24.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:24.222+0000 7f1fe6a60640 20 client.764646.objecter shutdown clearing up homeless session... 2024-09-16T03:49:24.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:24.222+0000 7f1fe6a60640 10 client.764646.objecter successfully canceled tick 2024-09-16T03:49:24.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:24.222+0000 7f1fe6a60640 1 -- 172.21.15.42:0/3448206890 >> v2:172.21.15.154:6812/254845342 conn(0x7f1fc4037810 msgr2=0x7f1fc4039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:24.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:24.222+0000 7f1fe6a60640 1 --2- 172.21.15.42:0/3448206890 >> v2:172.21.15.154:6812/254845342 conn(0x7f1fc4037810 0x7f1fc4039cd0 secure :-1 s=READY pgs=3513 cs=0 l=1 rev1=1 crypto rx=0x7f1fd00b1000 tx=0x7f1fd00a9070 comp rx=0 tx=0).stop 2024-09-16T03:49:24.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:24.222+0000 7f1fe6a60640 10 monclient: shutdown 2024-09-16T03:49:24.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:24.222+0000 7f1fe6a60640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:49:24.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:24.222+0000 7f1fe6a60640 1 -- 172.21.15.42:0/3448206890 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1fe00a1710 msgr2=0x7f1fe0134750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:24.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:24.222+0000 7f1fe6a60640 1 --2- 172.21.15.42:0/3448206890 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1fe00a1710 0x7f1fe0134750 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f1fc800c460 tx=0x7f1fc800c920 comp rx=0 tx=0).stop 2024-09-16T03:49:24.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:24.222+0000 7f1fe6a60640 1 -- 172.21.15.42:0/3448206890 shutdown_connections 2024-09-16T03:49:24.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:24.222+0000 7f1fe6a60640 1 --2- 172.21.15.42:0/3448206890 >> v2:172.21.15.154:6812/254845342 conn(0x7f1fc4037810 0x7f1fc4039cd0 unknown :-1 s=CLOSED pgs=3513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:24.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:24.222+0000 7f1fe6a60640 1 --2- 172.21.15.42:0/3448206890 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1fe01393a0 0x7f1fe011c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:24.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:24.222+0000 7f1fe6a60640 1 --2- 172.21.15.42:0/3448206890 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1fe00a1710 0x7f1fe0134750 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:24.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:24.222+0000 7f1fe6a60640 1 --2- 172.21.15.42:0/3448206890 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1fe00a1330 0x7f1fe011bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:24.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:24.222+0000 7f1fe6a60640 1 -- 172.21.15.42:0/3448206890 >> 172.21.15.42:0/3448206890 conn(0x7f1fe00a2420 msgr2=0x7f1fe00a3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:49:24.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:24.222+0000 7f1fe6a60640 1 -- 172.21.15.42:0/3448206890 shutdown_connections 2024-09-16T03:49:24.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:24.222+0000 7f1fe6a60640 1 -- 172.21.15.42:0/3448206890 wait complete. 2024-09-16T03:49:24.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:24.222+0000 7f1fe6a60640 1 librados: shutdown 2024-09-16T03:49:24.230 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.222+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:49:24.230 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.222+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2523415699}) 2024-09-16T03:49:24.230 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.222+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1356443658}) 2024-09-16T03:49:24.230 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.226+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:49:24.230 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.226+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2523415699}) 2024-09-16T03:49:24.230 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.226+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2016111796}) 2024-09-16T03:49:24.232 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.226+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:49:24.232 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.226+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=858960200}) 2024-09-16T03:49:24.232 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.226+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1252342100}) 2024-09-16T03:49:24.232 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.226+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:49:24.232 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.226+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=858960200}) 2024-09-16T03:49:24.232 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.226+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2700918473}) 2024-09-16T03:49:24.234 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.230+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:49:24.234 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.230+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2533987569}) 2024-09-16T03:49:24.234 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.230+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3802315211}) 2024-09-16T03:49:24.234 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.230+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:49:24.234 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.230+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2533987569}) 2024-09-16T03:49:24.234 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.230+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1135224751}) 2024-09-16T03:49:24.236 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.230+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:49:24.236 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.230+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2840217786}) 2024-09-16T03:49:24.236 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.230+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1732598669}) 2024-09-16T03:49:24.236 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.230+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:49:24.236 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.230+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2840217786}) 2024-09-16T03:49:24.237 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.230+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3505993889}) 2024-09-16T03:49:24.238 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.234+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:49:24.239 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.234+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2076037779}) 2024-09-16T03:49:24.239 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.234+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3937922885}) 2024-09-16T03:49:24.239 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.234+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:49:24.239 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.234+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2076037779}) 2024-09-16T03:49:24.239 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.234+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3017160488}) 2024-09-16T03:49:24.240 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.234+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:49:24.240 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.234+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2049395029}) 2024-09-16T03:49:24.240 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.234+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2286986119}) 2024-09-16T03:49:24.240 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.234+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:49:24.241 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.234+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2049395029}) 2024-09-16T03:49:24.241 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.234+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3287902127}) 2024-09-16T03:49:24.242 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.238+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:49:24.242 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.238+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=876577545}) 2024-09-16T03:49:24.242 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.238+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1239045859}) 2024-09-16T03:49:24.242 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.238+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:49:24.242 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.238+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=876577545}) 2024-09-16T03:49:24.242 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.238+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=682613241}) 2024-09-16T03:49:24.243 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.238+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:49:24.244 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.238+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2754817853}) 2024-09-16T03:49:24.244 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.238+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=681889408}) 2024-09-16T03:49:24.244 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.238+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:49:24.244 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.238+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2754817853}) 2024-09-16T03:49:24.244 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.238+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3203975744}) 2024-09-16T03:49:24.245 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.242+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:49:24.245 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.242+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3207219122}) 2024-09-16T03:49:24.245 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.242+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1728063574}) 2024-09-16T03:49:24.245 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.242+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:49:24.246 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.242+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3207219122}) 2024-09-16T03:49:24.246 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.242+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=686110507}) 2024-09-16T03:49:24.247 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T03:49:24.247 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.242+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:49:24.247 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.242+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1976134003}) 2024-09-16T03:49:24.248 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.242+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=422908304}) 2024-09-16T03:49:24.248 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.242+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:49:24.248 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.242+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1976134003}) 2024-09-16T03:49:24.248 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.242+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2851951468}) 2024-09-16T03:49:24.248 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.242+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:49:24.249 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.242+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4068064316}) 2024-09-16T03:49:24.249 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.242+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1422178655}) 2024-09-16T03:49:24.249 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.242+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:49:24.249 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.242+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4068064316}) 2024-09-16T03:49:24.249 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.242+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3428627818}) 2024-09-16T03:49:24.250 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.246+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:49:24.250 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.246+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4259984816}) 2024-09-16T03:49:24.250 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.246+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1292096645}) 2024-09-16T03:49:24.251 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.246+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:49:24.251 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.246+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4259984816}) 2024-09-16T03:49:24.251 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.246+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=472106667}) 2024-09-16T03:49:24.252 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.246+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:49:24.252 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.246+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4155357478}) 2024-09-16T03:49:24.252 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.246+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3296099585}) 2024-09-16T03:49:24.252 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.246+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:49:24.252 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.246+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4155357478}) 2024-09-16T03:49:24.252 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.246+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4276632858}) 2024-09-16T03:49:24.254 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.250+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:49:24.255 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.250+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4174070558}) 2024-09-16T03:49:24.255 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.250+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3512368924}) 2024-09-16T03:49:24.255 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.250+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:49:24.255 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.250+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4174070558}) 2024-09-16T03:49:24.255 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.250+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3507590547}) 2024-09-16T03:49:24.256 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.250+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:49:24.257 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.250+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=461097840}) 2024-09-16T03:49:24.257 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.250+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2926419382}) 2024-09-16T03:49:24.257 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.250+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:49:24.257 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.250+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=461097840}) 2024-09-16T03:49:24.257 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.250+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=536854915}) 2024-09-16T03:49:24.258 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.254+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:49:24.258 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.254+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2221031588}) 2024-09-16T03:49:24.258 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.254+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2336260529}) 2024-09-16T03:49:24.258 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.254+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:49:24.258 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.254+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2221031588}) 2024-09-16T03:49:24.258 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.254+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1636271199}) 2024-09-16T03:49:24.259 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.254+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:49:24.260 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.254+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=175722576}) 2024-09-16T03:49:24.260 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.254+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1356359114}) 2024-09-16T03:49:24.260 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.254+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:49:24.260 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.254+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=175722576}) 2024-09-16T03:49:24.260 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.254+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1608528934}) 2024-09-16T03:49:24.261 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.258+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:49:24.262 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.258+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=385701666}) 2024-09-16T03:49:24.262 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.258+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2211927303}) 2024-09-16T03:49:24.262 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.258+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:49:24.262 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.258+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=385701666}) 2024-09-16T03:49:24.262 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.258+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1324270053}) 2024-09-16T03:49:24.263 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.258+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:49:24.264 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.258+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=282973364}) 2024-09-16T03:49:24.264 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.258+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=645300696}) 2024-09-16T03:49:24.264 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.258+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:49:24.264 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.258+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=282973364}) 2024-09-16T03:49:24.264 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.258+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2220558126}) 2024-09-16T03:49:24.270 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.262+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:49:24.270 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.262+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=555625405}) 2024-09-16T03:49:24.270 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.262+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3474908539}) 2024-09-16T03:49:24.270 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.262+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:49:24.270 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.262+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=555625405}) 2024-09-16T03:49:24.270 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.262+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2235732241}) 2024-09-16T03:49:24.270 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.262+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:49:24.271 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.262+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1093802339}) 2024-09-16T03:49:24.271 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.262+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2436134829}) 2024-09-16T03:49:24.271 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.262+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:49:24.271 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.262+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1093802339}) 2024-09-16T03:49:24.271 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.262+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1925462953}) 2024-09-16T03:49:24.271 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.266+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:49:24.271 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.266+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4143433899}) 2024-09-16T03:49:24.271 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.266+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1294454626}) 2024-09-16T03:49:24.271 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.266+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:49:24.271 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.266+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4143433899}) 2024-09-16T03:49:24.271 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.266+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=25447440}) 2024-09-16T03:49:24.272 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.266+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:49:24.272 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.266+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2602013508}) 2024-09-16T03:49:24.272 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.266+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1903042063}) 2024-09-16T03:49:24.272 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.266+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:49:24.272 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.266+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2602013508}) 2024-09-16T03:49:24.272 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.266+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3199688079}) 2024-09-16T03:49:24.274 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.270+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:49:24.274 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.270+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3086988736}) 2024-09-16T03:49:24.274 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.270+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2182810621}) 2024-09-16T03:49:24.275 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.270+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:49:24.275 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.270+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3086988736}) 2024-09-16T03:49:24.275 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.270+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3706570360}) 2024-09-16T03:49:24.276 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.270+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:49:24.276 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.270+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=792440624}) 2024-09-16T03:49:24.276 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.270+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1191750009}) 2024-09-16T03:49:24.276 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.270+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:49:24.276 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.270+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=792440624}) 2024-09-16T03:49:24.277 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.270+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3409227869}) 2024-09-16T03:49:24.278 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.274+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:49:24.278 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.274+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2374415974}) 2024-09-16T03:49:24.278 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.274+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4190688253}) 2024-09-16T03:49:24.279 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.274+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:49:24.279 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.274+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2374415974}) 2024-09-16T03:49:24.279 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.274+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1062511830}) 2024-09-16T03:49:24.280 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.274+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:49:24.281 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.274+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=758909664}) 2024-09-16T03:49:24.281 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.274+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1242079520}) 2024-09-16T03:49:24.281 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.274+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:49:24.281 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.274+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=758909664}) 2024-09-16T03:49:24.281 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.274+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=352230762}) 2024-09-16T03:49:24.283 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.278+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:49:24.283 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.278+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3676960000}) 2024-09-16T03:49:24.283 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.278+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3046666190}) 2024-09-16T03:49:24.283 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.278+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:49:24.283 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.278+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3676960000}) 2024-09-16T03:49:24.283 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.278+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2312035338}) 2024-09-16T03:49:24.284 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.278+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:49:24.285 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.278+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1421493897}) 2024-09-16T03:49:24.285 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.278+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2826331038}) 2024-09-16T03:49:24.285 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.278+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:49:24.285 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.278+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1421493897}) 2024-09-16T03:49:24.285 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.278+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1078773267}) 2024-09-16T03:49:24.287 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.282+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:49:24.287 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.282+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1876314587}) 2024-09-16T03:49:24.287 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.282+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4216767158}) 2024-09-16T03:49:24.287 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.282+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:49:24.287 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.282+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1876314587}) 2024-09-16T03:49:24.287 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.282+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2096761239}) 2024-09-16T03:49:24.288 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.282+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:49:24.289 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.282+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=907546684}) 2024-09-16T03:49:24.289 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.282+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3458827436}) 2024-09-16T03:49:24.289 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.282+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:49:24.289 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.282+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=907546684}) 2024-09-16T03:49:24.289 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.282+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1615279543}) 2024-09-16T03:49:24.291 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.286+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:49:24.291 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.286+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2894864733}) 2024-09-16T03:49:24.291 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.286+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3204792809}) 2024-09-16T03:49:24.291 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.286+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:49:24.291 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.286+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2894864733}) 2024-09-16T03:49:24.291 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.286+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=200613140}) 2024-09-16T03:49:24.293 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.290+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:49:24.293 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.290+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2133551582}) 2024-09-16T03:49:24.293 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.290+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3957259465}) 2024-09-16T03:49:24.293 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.290+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:49:24.293 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.290+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2133551582}) 2024-09-16T03:49:24.293 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.290+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1566183477}) 2024-09-16T03:49:24.295 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.290+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:49:24.295 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.290+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1467888984}) 2024-09-16T03:49:24.295 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.290+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3530591560}) 2024-09-16T03:49:24.295 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.290+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:49:24.295 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.290+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1467888984}) 2024-09-16T03:49:24.295 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.290+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=287224350}) 2024-09-16T03:49:24.297 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.294+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:49:24.297 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.294+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2404837253}) 2024-09-16T03:49:24.297 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.294+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4077127679}) 2024-09-16T03:49:24.297 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.294+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:49:24.297 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.294+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2404837253}) 2024-09-16T03:49:24.297 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.294+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2808556674}) 2024-09-16T03:49:24.299 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.294+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:49:24.299 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.294+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3807765528}) 2024-09-16T03:49:24.299 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.294+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2779052169}) 2024-09-16T03:49:24.299 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.294+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:49:24.299 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.294+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3807765528}) 2024-09-16T03:49:24.299 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.294+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2655598289}) 2024-09-16T03:49:24.301 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.298+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:49:24.301 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.298+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=120195760}) 2024-09-16T03:49:24.302 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.298+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2535705259}) 2024-09-16T03:49:24.302 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.298+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:49:24.302 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.298+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=120195760}) 2024-09-16T03:49:24.302 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.298+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2670719280}) 2024-09-16T03:49:24.303 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.298+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:49:24.303 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.298+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1688443289}) 2024-09-16T03:49:24.303 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.298+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1489986117}) 2024-09-16T03:49:24.303 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.298+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:49:24.303 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.298+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1688443289}) 2024-09-16T03:49:24.303 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.298+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1669292018}) 2024-09-16T03:49:24.305 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.298+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:49:24.305 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.298+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2100450065}) 2024-09-16T03:49:24.305 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.298+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=427795470}) 2024-09-16T03:49:24.305 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.298+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:49:24.305 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.298+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2100450065}) 2024-09-16T03:49:24.305 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.298+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3217826408}) 2024-09-16T03:49:24.307 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.302+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:49:24.308 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.302+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=19,mdsmap=3,mgr=6,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=29,osdmap=11} crc {logm=3612951343,mdsmap=4088524775,mgr=741776913,mgr_command_descs=2107673600,mgr_metadata=1367272202,mgrstat=1810567429,mon_config_key=85057112,monmap=3937513981,osd_metadata=4182959540,osd_pg_creating=1948605615,osd_snap=3767380019,osdmap=4101484021}) 2024-09-16T03:49:24.308 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.302+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=18,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=11,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=28,osdmap=12} crc {logm=2830654662,mdsmap=4088524775,mgr=460109776,mgr_command_descs=2107673600,mgr_metadata=1367272202,mgrstat=1810567429,mon_config_key=85057112,monmap=3937513981,osd_metadata=191738038,osd_pg_creating=1948605615,osd_snap=2617741307,osdmap=2602084612}) 2024-09-16T03:49:24.308 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.302+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:49:24.308 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.302+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=19,mdsmap=3,mgr=6,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=29,osdmap=11} crc {logm=3612951343,mdsmap=4088524775,mgr=741776913,mgr_command_descs=2107673600,mgr_metadata=1367272202,mgrstat=1810567429,mon_config_key=85057112,monmap=3937513981,osd_metadata=4182959540,osd_pg_creating=1948605615,osd_snap=3767380019,osdmap=4101484021}) 2024-09-16T03:49:24.308 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.302+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=19,mdsmap=3,mgr=8,mgr_command_descs=1,mgrstat=9,mon_config_key=10,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=28,osdmap=13} crc {logm=2037361213,mdsmap=4088524775,mgr=460109776,mgr_command_descs=2107673600,mgrstat=1810567429,mon_config_key=1822420877,monmap=3937513981,osd_metadata=4021991691,osd_pg_creating=1948605615,osd_snap=1635149380,osdmap=4058061877}) 2024-09-16T03:49:24.310 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.306+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:49:24.311 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.306+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=336201527}) 2024-09-16T03:49:24.311 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.306+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1441688901}) 2024-09-16T03:49:24.311 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.306+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:49:24.311 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.306+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=336201527}) 2024-09-16T03:49:24.311 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.306+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3757994939}) 2024-09-16T03:49:24.313 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.310+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:49:24.313 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.310+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2421976912}) 2024-09-16T03:49:24.313 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.310+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3346698775}) 2024-09-16T03:49:24.313 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.310+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:49:24.313 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.310+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2421976912}) 2024-09-16T03:49:24.313 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.310+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2291145748}) 2024-09-16T03:49:24.316 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.310+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:49:24.317 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.310+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=944954439}) 2024-09-16T03:49:24.317 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.310+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=769055499}) 2024-09-16T03:49:24.317 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.310+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:49:24.317 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.314+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=944954439}) 2024-09-16T03:49:24.317 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.314+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=803081958}) 2024-09-16T03:49:24.338 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.334+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:49:24.338 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.334+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=50} crc {osdmap=339455250}) 2024-09-16T03:49:24.338 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.334+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=53} crc {osdmap=3107394432}) 2024-09-16T03:49:24.338 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.334+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:49:24.338 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.334+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=50} crc {osdmap=339455250}) 2024-09-16T03:49:24.338 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:49:24.334+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=54} crc {osdmap=1911661446}) 2024-09-16T03:49:25.248 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T03:49:25.248 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:49:25.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.374+0000 7fbd4d815640 10 monclient: get_monmap_and_config 2024-09-16T03:49:25.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.374+0000 7fbd4d815640 10 monclient: build_initial_monmap 2024-09-16T03:49:25.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.374+0000 7fbd4d815640 10 monclient: monmap: 2024-09-16T03:49:25.379 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:49:25.379 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:49:25.379 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:49:25.378124+0000 2024-09-16T03:49:25.379 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:49:25.378124+0000 2024-09-16T03:49:25.380 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:49:25.380 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:49:25.380 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:49:25.380 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:49:25.380 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:49:25.380 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:49:25.380 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:49:25.380 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:49:25.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.374+0000 7fbd4d815640 1 Processor -- start 2024-09-16T03:49:25.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.374+0000 7fbd4d815640 1 -- start start 2024-09-16T03:49:25.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.374+0000 7fbd4d815640 10 monclient: init 2024-09-16T03:49:25.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.374+0000 7fbd4d815640 10 monclient: _reopen_session rank -1 2024-09-16T03:49:25.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.374+0000 7fbd4d815640 10 monclient: _add_conns ranks=[2,1,3,4,0] 2024-09-16T03:49:25.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.374+0000 7fbd4d815640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbd480a1330 0x7fbd4809f180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:25.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.374+0000 7fbd4d815640 10 monclient(hunting): picked mon.noname-b con 0x7fbd480a1330 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:49:25.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.374+0000 7fbd4d815640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbd480a1710 addr v1:172.21.15.154:6789/0 2024-09-16T03:49:25.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.374+0000 7fbd4d815640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fbd481393a0 0x7fbd4813b790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:25.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.374+0000 7fbd4d815640 10 monclient(hunting): picked mon.noname-a con 0x7fbd481393a0 addr v2:172.21.15.42:3300/0 2024-09-16T03:49:25.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.374+0000 7fbd4d815640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbd4809e4c0 con 0x7fbd480a1710 2024-09-16T03:49:25.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.374+0000 7fbd4d815640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:25.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.374+0000 7fbd4d815640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7fbd4809edf0 con 0x7fbd481393a0 2024-09-16T03:49:25.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.374+0000 7fbd4d815640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:25.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.374+0000 7fbd4d815640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fbd4809cd60 con 0x7fbd480a1330 2024-09-16T03:49:25.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.374+0000 7fbd4d815640 10 monclient(hunting): _renew_subs 2024-09-16T03:49:25.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.374+0000 7fbd4d815640 10 monclient(hunting): authenticate will time out at 1412.347664s 2024-09-16T03:49:25.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.374+0000 7fbd477fe640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fbd481393a0 0x7fbd4813b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:25.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.374+0000 7fbd477fe640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fbd481393a0 0x7fbd4813b790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:59144/0 (socket says 172.21.15.42:59144) 2024-09-16T03:49:25.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.374+0000 7fbd477fe640 1 -- 172.21.15.42:0/1291462774 learned_addr learned my addr 172.21.15.42:0/1291462774 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:49:25.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.374+0000 7fbd477fe640 10 monclient(hunting): get_auth_request con 0x7fbd481393a0 auth_method 0 2024-09-16T03:49:25.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.374+0000 7fbd477fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:25.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.374+0000 7fbd477fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:25.385 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.374+0000 7fbd477fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:25.385 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.374+0000 7fbd46ffd640 1 --2- 172.21.15.42:0/1291462774 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbd480a1330 0x7fbd4809f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:25.385 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.378+0000 7fbd46ffd640 10 monclient(hunting): get_auth_request con 0x7fbd480a1330 auth_method 0 2024-09-16T03:49:25.385 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.378+0000 7fbd46ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:25.385 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.378+0000 7fbd46ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:25.385 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.378+0000 7fbd46ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:25.385 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.378+0000 7fbd477fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:25.385 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.378+0000 7fbd477fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:25.385 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.378+0000 7fbd477fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:25.385 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.378+0000 7fbd46ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:25.385 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.378+0000 7fbd46ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:25.385 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.378+0000 7fbd46ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:25.385 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.378+0000 7fbd477fe640 10 monclient(hunting): handle_auth_done global_id 764649 payload 274 2024-09-16T03:49:25.386 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.378+0000 7fbd477fe640 1 -- 172.21.15.42:0/1291462774 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbd480a1330 msgr2=0x7fbd4809f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:49:25.386 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.378+0000 7fbd477fe640 1 --2- 172.21.15.42:0/1291462774 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbd480a1330 0x7fbd4809f180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:25.386 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.378+0000 7fbd477fe640 1 -- 172.21.15.42:0/1291462774 >> v1:172.21.15.154:6789/0 conn(0x7fbd480a1710 legacy=0x7fbd480a1b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:25.386 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.378+0000 7fbd477fe640 10 monclient: _finish_hunting 0 2024-09-16T03:49:25.386 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.378+0000 7fbd477fe640 1 monclient: found mon.noname-a 2024-09-16T03:49:25.386 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.378+0000 7fbd477fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:49:25.386 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.378+0000 7fbd477fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:49:25.386 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.378+0000 7fbd477fe640 1 -- 172.21.15.42:0/1291462774 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd4809f6c0 con 0x7fbd481393a0 2024-09-16T03:49:25.386 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.378+0000 7fbd46ffd640 1 --2- 172.21.15.42:0/1291462774 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbd480a1330 0x7fbd4809f180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:49:25.386 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.378+0000 7fbd477fe640 1 --2- 172.21.15.42:0/1291462774 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbd481393a0 0x7fbd4813b790 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fbd40009870 tx=0x7fbd4001c420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3df3e10db4aa7e4 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:25.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.378+0000 7fbd45ffb640 1 -- 172.21.15.42:0/1291462774 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd400b0070 con 0x7fbd481393a0 2024-09-16T03:49:25.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.378+0000 7fbd45ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:25.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.378+0000 7fbd45ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:49:25.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.378+0000 7fbd45ffb640 10 monclient: dump: 2024-09-16T03:49:25.392 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:49:25.392 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:49:25.392 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:25.392 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:25.393 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:25.393 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:49:25.393 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:49:25.393 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:49:25.393 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:49:25.393 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:49:25.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.378+0000 7fbd45ffb640 10 monclient: _finish_auth 0 2024-09-16T03:49:25.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.378+0000 7fbd45ffb640 10 monclient: _check_auth_tickets 2024-09-16T03:49:25.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.378+0000 7fbd45ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:49:25.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.378+0000 7fbd45ffb640 1 -- 172.21.15.42:0/1291462774 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbd400ab600 con 0x7fbd481393a0 2024-09-16T03:49:25.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.378+0000 7fbd4d815640 5 monclient: authenticate success, global_id 764649 2024-09-16T03:49:25.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.378+0000 7fbd4d815640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:49:25.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.378+0000 7fbd45ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T03:49:25.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.378+0000 7fbd4d815640 10 monclient: get_monmap_and_config success 2024-09-16T03:49:25.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.378+0000 7fbd4d815640 4 set_mon_vals no callback set 2024-09-16T03:49:25.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.378+0000 7fbd45ffb640 1 -- 172.21.15.42:0/1291462774 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd400ab9d0 con 0x7fbd481393a0 2024-09-16T03:49:25.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.378+0000 7fbd45ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:25.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.378+0000 7fbd45ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:49:25.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.378+0000 7fbd45ffb640 10 monclient: dump: 2024-09-16T03:49:25.394 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:49:25.394 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:49:25.394 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:25.394 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:25.394 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:25.394 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:49:25.394 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:49:25.394 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:49:25.395 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:49:25.395 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:49:25.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.378+0000 7fbd4d815640 10 monclient: shutdown 2024-09-16T03:49:25.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.378+0000 7fbd4d815640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:49:25.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.378+0000 7fbd4d815640 1 -- 172.21.15.42:0/1291462774 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbd481393a0 msgr2=0x7fbd4813b790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:25.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.378+0000 7fbd4d815640 1 --2- 172.21.15.42:0/1291462774 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbd481393a0 0x7fbd4813b790 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fbd40009870 tx=0x7fbd4001c420 comp rx=0 tx=0).stop 2024-09-16T03:49:25.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.378+0000 7fbd4d815640 1 -- 172.21.15.42:0/1291462774 shutdown_connections 2024-09-16T03:49:25.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.378+0000 7fbd4d815640 1 --2- 172.21.15.42:0/1291462774 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbd481393a0 0x7fbd4813b790 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:25.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.378+0000 7fbd4d815640 1 --2- 172.21.15.42:0/1291462774 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbd480a1330 0x7fbd4809f180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:25.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.378+0000 7fbd4d815640 1 -- 172.21.15.42:0/1291462774 >> 172.21.15.42:0/1291462774 conn(0x7fbd480a2420 msgr2=0x7fbd480a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:49:25.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.378+0000 7fbd4d815640 1 -- 172.21.15.42:0/1291462774 shutdown_connections 2024-09-16T03:49:25.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.378+0000 7fbd4d815640 1 -- 172.21.15.42:0/1291462774 wait complete. 2024-09-16T03:49:25.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.378+0000 7fbd4d815640 10 monclient: build_initial_monmap 2024-09-16T03:49:25.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.378+0000 7fbd4d815640 10 monclient: monmap: 2024-09-16T03:49:25.396 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:49:25.396 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:49:25.396 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:49:25.396 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:49:25.396 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:49:25.396 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:49:25.396 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:49:25.396 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:49:25.396 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:49:25.396 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:49:25.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.382+0000 7fbd4d815640 1 librados: starting msgr at 2024-09-16T03:49:25.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.382+0000 7fbd4d815640 1 librados: starting objecter 2024-09-16T03:49:25.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.382+0000 7fbd4d815640 1 Processor -- start 2024-09-16T03:49:25.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.382+0000 7fbd4d815640 1 -- start start 2024-09-16T03:49:25.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.382+0000 7fbd4d815640 1 librados: setting wanted keys 2024-09-16T03:49:25.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.382+0000 7fbd4d815640 1 librados: calling monclient init 2024-09-16T03:49:25.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.382+0000 7fbd4d815640 10 monclient: init 2024-09-16T03:49:25.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.382+0000 7fbd4d815640 10 monclient: _reopen_session rank -1 2024-09-16T03:49:25.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.382+0000 7fbd4d815640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:49:25.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.382+0000 7fbd4d815640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbd480a1330 0x7fbd480716d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:25.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.382+0000 7fbd4d815640 10 monclient(hunting): picked mon.noname-c con 0x7fbd480a1330 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:49:25.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.382+0000 7fbd4d815640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbd480a1710 0x7fbd48071c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:25.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.382+0000 7fbd4d815640 10 monclient(hunting): picked mon.noname-b con 0x7fbd480a1710 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:49:25.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.382+0000 7fbd4d815640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbd481393a0 0x7fbd48072150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:25.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.382+0000 7fbd4d815640 10 monclient(hunting): picked mon.noname-a con 0x7fbd481393a0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:49:25.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.382+0000 7fbd4d815640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:25.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.382+0000 7fbd4d815640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fbd4813e880 con 0x7fbd481393a0 2024-09-16T03:49:25.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.382+0000 7fbd4d815640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:25.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.382+0000 7fbd4d815640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fbd4813e700 con 0x7fbd480a1710 2024-09-16T03:49:25.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.382+0000 7fbd4d815640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:25.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.382+0000 7fbd4d815640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fbd4813ea00 con 0x7fbd480a1330 2024-09-16T03:49:25.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.382+0000 7fbd4d815640 10 monclient(hunting): _renew_subs 2024-09-16T03:49:25.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.382+0000 7fbd4d815640 10 monclient(hunting): authenticate will time out at 1412.352832s 2024-09-16T03:49:25.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.382+0000 7fbd46ffd640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbd480a1330 0x7fbd480716d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:25.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.382+0000 7fbd477fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbd481393a0 0x7fbd48072150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:25.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.382+0000 7fbd46ffd640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbd480a1330 0x7fbd480716d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:60976/0 (socket says 172.21.15.42:60976) 2024-09-16T03:49:25.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.382+0000 7fbd46ffd640 1 -- 172.21.15.42:0/4069713786 learned_addr learned my addr 172.21.15.42:0/4069713786 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:49:25.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.382+0000 7fbd46ffd640 10 monclient(hunting): get_auth_request con 0x7fbd480a1330 auth_method 0 2024-09-16T03:49:25.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.382+0000 7fbd46ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:25.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.382+0000 7fbd46ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:25.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.382+0000 7fbd46ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:25.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.382+0000 7fbd477fe640 10 monclient(hunting): get_auth_request con 0x7fbd481393a0 auth_method 0 2024-09-16T03:49:25.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.382+0000 7fbd477fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:25.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.382+0000 7fbd477fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:25.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.382+0000 7fbd477fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:25.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.382+0000 7fbd46ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:25.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.382+0000 7fbd46ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:25.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.382+0000 7fbd46ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:25.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.382+0000 7fbd477fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:25.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.382+0000 7fbd477fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:25.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.382+0000 7fbd477fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:25.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.382+0000 7fbd467fc640 1 --2- 172.21.15.42:0/4069713786 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbd480a1710 0x7fbd48071c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:25.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.382+0000 7fbd467fc640 10 monclient(hunting): get_auth_request con 0x7fbd480a1710 auth_method 0 2024-09-16T03:49:25.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.382+0000 7fbd467fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:25.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.382+0000 7fbd467fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:25.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.382+0000 7fbd467fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:25.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.382+0000 7fbd46ffd640 10 monclient(hunting): handle_auth_done global_id 784235 payload 824 2024-09-16T03:49:25.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.382+0000 7fbd46ffd640 1 -- 172.21.15.42:0/4069713786 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbd480a1710 msgr2=0x7fbd48071c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:25.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.382+0000 7fbd46ffd640 1 --2- 172.21.15.42:0/4069713786 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbd480a1710 0x7fbd48071c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:25.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.382+0000 7fbd46ffd640 1 -- 172.21.15.42:0/4069713786 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbd481393a0 msgr2=0x7fbd48072150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:25.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.382+0000 7fbd46ffd640 1 --2- 172.21.15.42:0/4069713786 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbd481393a0 0x7fbd48072150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:25.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.382+0000 7fbd46ffd640 10 monclient: _finish_hunting 0 2024-09-16T03:49:25.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.382+0000 7fbd46ffd640 1 monclient: found mon.noname-c 2024-09-16T03:49:25.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.382+0000 7fbd46ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:49:25.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.382+0000 7fbd46ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:49:25.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.382+0000 7fbd46ffd640 1 -- 172.21.15.42:0/4069713786 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd4811d710 con 0x7fbd480a1330 2024-09-16T03:49:25.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.382+0000 7fbd477fe640 1 --2- 172.21.15.42:0/4069713786 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbd481393a0 0x7fbd48072150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:49:25.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.382+0000 7fbd46ffd640 1 --2- 172.21.15.42:0/4069713786 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbd480a1330 0x7fbd480716d0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fbd3800ebe0 tx=0x7fbd3800c520 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:25.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.382+0000 7fbd3ffff640 10 client.?.objecter ms_handle_connect 0x7fbd480a1330 2024-09-16T03:49:25.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.382+0000 7fbd3ffff640 10 client.?.objecter resend_mon_ops 2024-09-16T03:49:25.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.382+0000 7fbd3ffff640 1 -- 172.21.15.42:0/4069713786 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd38017070 con 0x7fbd480a1330 2024-09-16T03:49:25.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.382+0000 7fbd3ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:25.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.382+0000 7fbd3ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:49:25.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.382+0000 7fbd3ffff640 10 monclient: dump: 2024-09-16T03:49:25.401 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:49:25.401 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:49:25.401 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:25.401 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:25.401 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:25.402 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:49:25.402 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:49:25.402 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:49:25.402 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:49:25.402 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:49:25.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.382+0000 7fbd3ffff640 10 monclient: _finish_auth 0 2024-09-16T03:49:25.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.382+0000 7fbd3ffff640 10 monclient: _check_auth_tickets 2024-09-16T03:49:25.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.382+0000 7fbd3ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:49:25.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.382+0000 7fbd4d815640 5 monclient: authenticate success, global_id 784235 2024-09-16T03:49:25.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.382+0000 7fbd3ffff640 1 -- 172.21.15.42:0/4069713786 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbd3800cc30 con 0x7fbd480a1330 2024-09-16T03:49:25.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.382+0000 7fbd3ffff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:49:25.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.382+0000 7fbd3ffff640 1 -- 172.21.15.42:0/4069713786 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd38013470 con 0x7fbd480a1330 2024-09-16T03:49:25.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.382+0000 7fbd3ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:25.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.382+0000 7fbd3ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:49:25.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.382+0000 7fbd3ffff640 10 monclient: dump: 2024-09-16T03:49:25.403 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:49:25.403 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:49:25.403 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:25.403 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:25.403 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:25.403 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:49:25.403 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:49:25.403 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:49:25.403 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:49:25.403 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:49:25.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.382+0000 7fbd457fa640 4 set_mon_vals no callback set 2024-09-16T03:49:25.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.382+0000 7fbd4d815640 10 monclient: _renew_subs 2024-09-16T03:49:25.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.382+0000 7fbd4d815640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:49:25.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.382+0000 7fbd4d815640 1 -- 172.21.15.42:0/4069713786 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd4811d9f0 con 0x7fbd480a1330 2024-09-16T03:49:25.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.382+0000 7fbd4d815640 10 client.784235.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:49:25.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.382+0000 7fbd4d815640 10 monclient: _renew_subs 2024-09-16T03:49:25.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.382+0000 7fbd4d815640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:49:25.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.382+0000 7fbd4d815640 1 -- 172.21.15.42:0/4069713786 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd480a0e90 con 0x7fbd480a1330 2024-09-16T03:49:25.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.382+0000 7fbd4d815640 1 librados: init done 2024-09-16T03:49:25.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.382+0000 7fbd3ffff640 1 -- 172.21.15.42:0/4069713786 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fbd3801a070 con 0x7fbd480a1330 2024-09-16T03:49:25.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.382+0000 7fbd4d815640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:49:25.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.382+0000 7fbd4d815640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:49:25.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.382+0000 7fbd4d815640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:49:25.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.382+0000 7fbd4d815640 1 -- 172.21.15.42:0/4069713786 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbd4809f180 con 0x7fbd480a1330 2024-09-16T03:49:25.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.382+0000 7fbd3ffff640 1 --2- 172.21.15.42:0/4069713786 >> v2:172.21.15.154:6812/254845342 conn(0x7fbd28037810 0x7fbd28039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:25.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.382+0000 7fbd3ffff640 1 -- 172.21.15.42:0/4069713786 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(247..247 src has 1..247) ==== 6129+0+0 (secure 0 0 0) 0x7fbd38049050 con 0x7fbd480a1330 2024-09-16T03:49:25.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.382+0000 7fbd3ffff640 10 client.784235.objecter ms_dispatch 0x7fbd48000ba0 osd_map(247..247 src has 1..247) 2024-09-16T03:49:25.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.382+0000 7fbd3ffff640 3 client.784235.objecter handle_osd_map got epochs [247,247] > 0 2024-09-16T03:49:25.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.382+0000 7fbd3ffff640 3 client.784235.objecter handle_osd_map decoding full epoch 247 2024-09-16T03:49:25.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.382+0000 7fbd3ffff640 20 client.784235.objecter dump_active .. 0 homeless 2024-09-16T03:49:25.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.382+0000 7fbd467fc640 1 --2- 172.21.15.42:0/4069713786 >> v2:172.21.15.154:6812/254845342 conn(0x7fbd28037810 0x7fbd28039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:25.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.382+0000 7fbd467fc640 10 monclient: get_auth_request con 0x7fbd28037810 auth_method 0 2024-09-16T03:49:25.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.386+0000 7fbd3ffff640 10 client.784235.objecter ms_handle_connect 0x7fbd28037810 2024-09-16T03:49:25.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.386+0000 7fbd467fc640 1 --2- 172.21.15.42:0/4069713786 >> v2:172.21.15.154:6812/254845342 conn(0x7fbd28037810 0x7fbd28039cd0 secure :-1 s=READY pgs=3514 cs=0 l=1 rev1=1 crypto rx=0x7fbd48072840 tx=0x7fbd30009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:25.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.390+0000 7fbd3ffff640 1 -- 172.21.15.42:0/4069713786 <== mon.2 v2:172.21.15.42:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7fbd38011350 con 0x7fbd480a1330 2024-09-16T03:49:25.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.390+0000 7fbd3ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:49:25.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.390+0000 7fbd3ffff640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:49:25.619 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.614+0000 7fbd4d815640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:49:25.619 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.614+0000 7fbd4d815640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:49:25.619 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.614+0000 7fbd4d815640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:49:25.619 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.614+0000 7fbd4d815640 1 -- 172.21.15.42:0/4069713786 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fbd48069040 con 0x7fbd480a1330 2024-09-16T03:49:25.620 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.614+0000 7fbd3ffff640 1 -- 172.21.15.42:0/4069713786 <== mon.2 v2:172.21.15.42:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7fbd3801ad70 con 0x7fbd480a1330 2024-09-16T03:49:25.620 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.614+0000 7fbd3ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:49:25.620 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.614+0000 7fbd3ffff640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:49:25.620 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":66,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:49:25.622 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.618+0000 7fbd4d815640 10 librados: watch_flush enter 2024-09-16T03:49:25.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.618+0000 7fbd4d815640 10 librados: watch_flush exit 2024-09-16T03:49:25.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.618+0000 7fbd4d815640 20 client.784235.objecter shutdown clearing up homeless session... 2024-09-16T03:49:25.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.618+0000 7fbd4d815640 10 client.784235.objecter successfully canceled tick 2024-09-16T03:49:25.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.618+0000 7fbd4d815640 1 -- 172.21.15.42:0/4069713786 >> v2:172.21.15.154:6812/254845342 conn(0x7fbd28037810 msgr2=0x7fbd28039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:25.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.618+0000 7fbd4d815640 1 --2- 172.21.15.42:0/4069713786 >> v2:172.21.15.154:6812/254845342 conn(0x7fbd28037810 0x7fbd28039cd0 secure :-1 s=READY pgs=3514 cs=0 l=1 rev1=1 crypto rx=0x7fbd48072840 tx=0x7fbd30009040 comp rx=0 tx=0).stop 2024-09-16T03:49:25.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.618+0000 7fbd4d815640 10 monclient: shutdown 2024-09-16T03:49:25.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.618+0000 7fbd4d815640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:49:25.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.618+0000 7fbd4d815640 1 -- 172.21.15.42:0/4069713786 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbd480a1330 msgr2=0x7fbd480716d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:25.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.618+0000 7fbd4d815640 1 --2- 172.21.15.42:0/4069713786 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbd480a1330 0x7fbd480716d0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fbd3800ebe0 tx=0x7fbd3800c520 comp rx=0 tx=0).stop 2024-09-16T03:49:25.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.618+0000 7fbd4d815640 1 -- 172.21.15.42:0/4069713786 shutdown_connections 2024-09-16T03:49:25.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.618+0000 7fbd4d815640 1 --2- 172.21.15.42:0/4069713786 >> v2:172.21.15.154:6812/254845342 conn(0x7fbd28037810 0x7fbd28039cd0 unknown :-1 s=CLOSED pgs=3514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:25.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.618+0000 7fbd4d815640 1 --2- 172.21.15.42:0/4069713786 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbd481393a0 0x7fbd48072150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:25.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.618+0000 7fbd4d815640 1 --2- 172.21.15.42:0/4069713786 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbd480a1710 0x7fbd48071c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:25.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.618+0000 7fbd4d815640 1 --2- 172.21.15.42:0/4069713786 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbd480a1330 0x7fbd480716d0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:25.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.618+0000 7fbd4d815640 1 -- 172.21.15.42:0/4069713786 >> 172.21.15.42:0/4069713786 conn(0x7fbd480a2420 msgr2=0x7fbd480a45b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:49:25.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.618+0000 7fbd4d815640 1 -- 172.21.15.42:0/4069713786 shutdown_connections 2024-09-16T03:49:25.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.618+0000 7fbd4d815640 1 -- 172.21.15.42:0/4069713786 wait complete. 2024-09-16T03:49:25.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.618+0000 7fbd4d815640 1 librados: shutdown 2024-09-16T03:49:25.638 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T03:49:25.639 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T03:49:25.639 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T03:49:25.639 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T03:49:25.764 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.758+0000 7f52ffe95640 10 monclient: get_monmap_and_config 2024-09-16T03:49:25.764 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.758+0000 7f52ffe95640 10 monclient: build_initial_monmap 2024-09-16T03:49:25.764 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.758+0000 7f52ffe95640 10 monclient: monmap: 2024-09-16T03:49:25.764 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:49:25.764 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:49:25.764 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:49:25.763108+0000 2024-09-16T03:49:25.764 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:49:25.763108+0000 2024-09-16T03:49:25.764 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:49:25.765 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:49:25.765 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:49:25.765 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:49:25.765 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:49:25.765 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:49:25.765 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:49:25.765 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:49:25.767 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.758+0000 7f52ffe95640 1 Processor -- start 2024-09-16T03:49:25.767 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.758+0000 7f52ffe95640 1 -- start start 2024-09-16T03:49:25.768 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.758+0000 7f52ffe95640 10 monclient: init 2024-09-16T03:49:25.768 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.762+0000 7f52ffe95640 10 monclient: _reopen_session rank -1 2024-09-16T03:49:25.768 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.762+0000 7f52ffe95640 10 monclient: _add_conns ranks=[1,2,4,3,0] 2024-09-16T03:49:25.768 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.762+0000 7f52ffe95640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f52f80a2f10 addr v1:172.21.15.154:6789/0 2024-09-16T03:49:25.768 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.762+0000 7f52ffe95640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f52f809e070 0x7f52f80a1a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:25.768 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.762+0000 7f52ffe95640 10 monclient(hunting): picked mon.noname-b con 0x7f52f809e070 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:49:25.768 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.762+0000 7f52ffe95640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f52f8140b30 0x7f52f8142f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:25.768 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.762+0000 7f52ffe95640 10 monclient(hunting): picked mon.noname-c con 0x7f52f8140b30 addr v2:172.21.15.154:3300/0 2024-09-16T03:49:25.768 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.762+0000 7f52ffe95640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f52f809dc50 con 0x7f52f80a2f10 2024-09-16T03:49:25.768 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.762+0000 7f52ffe95640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:25.768 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.762+0000 7f52ffe95640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f52f80a32f0 con 0x7f52f8140b30 2024-09-16T03:49:25.769 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.762+0000 7f52ffe95640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:25.769 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.762+0000 7f52ffe95640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f52f8073290 con 0x7f52f809e070 2024-09-16T03:49:25.769 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.762+0000 7f52ffe95640 10 monclient(hunting): _renew_subs 2024-09-16T03:49:25.769 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.762+0000 7f52ffe95640 10 monclient(hunting): authenticate will time out at 1412.732575s 2024-09-16T03:49:25.769 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.762+0000 7f52fd430640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f52f809e070 0x7f52f80a1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:25.769 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.762+0000 7f52fd430640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f52f809e070 0x7f52f80a1a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:60988/0 (socket says 172.21.15.42:60988) 2024-09-16T03:49:25.769 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.762+0000 7f52fd430640 1 -- 172.21.15.42:0/32007976 learned_addr learned my addr 172.21.15.42:0/32007976 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:49:25.769 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.762+0000 7f52fd430640 10 monclient(hunting): get_auth_request con 0x7f52f809e070 auth_method 0 2024-09-16T03:49:25.770 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.762+0000 7f52fd430640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:25.770 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.762+0000 7f52fd430640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:25.770 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.762+0000 7f52fd430640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:25.770 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.762+0000 7f52fd430640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:25.770 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.762+0000 7f52fd430640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:25.770 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.762+0000 7f52fd430640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:25.770 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.762+0000 7f52fe432640 1 --2- 172.21.15.42:0/32007976 >> v2:172.21.15.154:3300/0 conn(0x7f52f8140b30 0x7f52f8142f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:25.770 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.762+0000 7f52fe432640 10 monclient(hunting): get_auth_request con 0x7f52f8140b30 auth_method 0 2024-09-16T03:49:25.770 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.762+0000 7f52fe432640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:25.770 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.762+0000 7f52fe432640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:25.770 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.762+0000 7f52fe432640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:25.770 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.762+0000 7f52fd430640 10 monclient(hunting): handle_auth_done global_id 784238 payload 274 2024-09-16T03:49:25.770 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.762+0000 7f52fd430640 1 -- 172.21.15.42:0/32007976 >> v2:172.21.15.154:3300/0 conn(0x7f52f8140b30 msgr2=0x7f52f8142f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:49:25.771 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.762+0000 7f52fd430640 1 --2- 172.21.15.42:0/32007976 >> v2:172.21.15.154:3300/0 conn(0x7f52f8140b30 0x7f52f8142f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:25.771 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.762+0000 7f52fd430640 1 -- 172.21.15.42:0/32007976 >> v1:172.21.15.154:6789/0 conn(0x7f52f80a2f10 legacy=0x7f52f80a1330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:25.771 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.762+0000 7f52fd430640 10 monclient: _finish_hunting 0 2024-09-16T03:49:25.771 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.762+0000 7f52fd430640 1 monclient: found mon.noname-b 2024-09-16T03:49:25.771 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.762+0000 7f52fd430640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:49:25.771 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.762+0000 7f52fd430640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T03:49:25.771 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.762+0000 7f52fd430640 1 -- 172.21.15.42:0/32007976 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52f809e600 con 0x7f52f809e070 2024-09-16T03:49:25.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.762+0000 7f52fd430640 1 --2- 172.21.15.42:0/32007976 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f52f809e070 0x7f52f80a1a40 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f52e8009870 tx=0x7f52e801c3f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=790298a04edc0444 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:25.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.762+0000 7f52fcc2f640 1 -- 172.21.15.42:0/32007976 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52e80ab5f0 con 0x7f52f809e070 2024-09-16T03:49:25.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.762+0000 7f52fcc2f640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:25.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.762+0000 7f52fcc2f640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:49:25.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.762+0000 7f52fcc2f640 10 monclient: dump: 2024-09-16T03:49:25.780 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:49:25.780 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:49:25.781 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:25.781 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:25.781 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:25.781 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:49:25.781 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:49:25.781 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:49:25.781 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:49:25.781 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:49:25.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.762+0000 7f52fcc2f640 10 monclient: _finish_auth 0 2024-09-16T03:49:25.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.762+0000 7f52fcc2f640 10 monclient: _check_auth_tickets 2024-09-16T03:49:25.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.762+0000 7f52fcc2f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:49:25.782 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.762+0000 7f52ffe95640 5 monclient: authenticate success, global_id 784238 2024-09-16T03:49:25.782 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.762+0000 7f52ffe95640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:49:25.782 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.762+0000 7f52fcc2f640 1 -- 172.21.15.42:0/32007976 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f52e80b0070 con 0x7f52f809e070 2024-09-16T03:49:25.782 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.762+0000 7f52fcc2f640 10 monclient: handle_config config(0 keys) 2024-09-16T03:49:25.782 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.762+0000 7f52ffe95640 10 monclient: get_monmap_and_config success 2024-09-16T03:49:25.782 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.762+0000 7f52fcc2f640 1 -- 172.21.15.42:0/32007976 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52e80ab9b0 con 0x7f52f809e070 2024-09-16T03:49:25.782 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.762+0000 7f52fcc2f640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:25.782 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.762+0000 7f52ffe95640 4 set_mon_vals no callback set 2024-09-16T03:49:25.782 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.762+0000 7f52fcc2f640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:49:25.782 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.762+0000 7f52fcc2f640 10 monclient: dump: 2024-09-16T03:49:25.782 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:49:25.782 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:49:25.782 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:25.783 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:25.783 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:25.783 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:49:25.783 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:49:25.783 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:49:25.783 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:49:25.783 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:49:25.783 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.766+0000 7f52ffe95640 10 monclient: shutdown 2024-09-16T03:49:25.783 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.766+0000 7f52ffe95640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:49:25.783 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.766+0000 7f52ffe95640 1 -- 172.21.15.42:0/32007976 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f52f809e070 msgr2=0x7f52f80a1a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:25.783 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.766+0000 7f52ffe95640 1 --2- 172.21.15.42:0/32007976 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f52f809e070 0x7f52f80a1a40 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f52e8009870 tx=0x7f52e801c3f0 comp rx=0 tx=0).stop 2024-09-16T03:49:25.783 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.766+0000 7f52ffe95640 1 -- 172.21.15.42:0/32007976 shutdown_connections 2024-09-16T03:49:25.783 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.766+0000 7f52ffe95640 1 --2- 172.21.15.42:0/32007976 >> v2:172.21.15.154:3300/0 conn(0x7f52f8140b30 0x7f52f8142f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:25.784 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.766+0000 7f52ffe95640 1 --2- 172.21.15.42:0/32007976 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f52f809e070 0x7f52f80a1a40 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:25.784 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.766+0000 7f52ffe95640 1 -- 172.21.15.42:0/32007976 >> 172.21.15.42:0/32007976 conn(0x7f52f80a2420 msgr2=0x7f52f80a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:49:25.784 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.766+0000 7f52ffe95640 1 -- 172.21.15.42:0/32007976 shutdown_connections 2024-09-16T03:49:25.784 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.766+0000 7f52ffe95640 1 -- 172.21.15.42:0/32007976 wait complete. 2024-09-16T03:49:25.784 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.766+0000 7f52ffe95640 10 monclient: build_initial_monmap 2024-09-16T03:49:25.784 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.766+0000 7f52ffe95640 10 monclient: monmap: 2024-09-16T03:49:25.784 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:49:25.784 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:49:25.784 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:49:25.784 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:49:25.784 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:49:25.784 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:49:25.785 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:49:25.785 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:49:25.785 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:49:25.785 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:49:25.785 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.766+0000 7f52ffe95640 1 librados: starting msgr at 2024-09-16T03:49:25.785 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.766+0000 7f52ffe95640 1 librados: starting objecter 2024-09-16T03:49:25.785 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.766+0000 7f52ffe95640 1 Processor -- start 2024-09-16T03:49:25.785 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.766+0000 7f52ffe95640 1 -- start start 2024-09-16T03:49:25.785 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.766+0000 7f52ffe95640 1 librados: setting wanted keys 2024-09-16T03:49:25.785 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.766+0000 7f52ffe95640 1 librados: calling monclient init 2024-09-16T03:49:25.785 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.766+0000 7f52ffe95640 10 monclient: init 2024-09-16T03:49:25.785 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.766+0000 7f52ffe95640 10 monclient: _reopen_session rank -1 2024-09-16T03:49:25.785 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.766+0000 7f52ffe95640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T03:49:25.785 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.766+0000 7f52ffe95640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f52f809e070 0x7f52f81d3290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:25.786 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.766+0000 7f52ffe95640 10 monclient(hunting): picked mon.noname-b con 0x7f52f809e070 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:49:25.786 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.766+0000 7f52ffe95640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f52f80a2f10 0x7f52f813bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:25.786 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.766+0000 7f52ffe95640 10 monclient(hunting): picked mon.noname-c con 0x7f52f80a2f10 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:49:25.786 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.766+0000 7f52ffe95640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f52f8140b30 0x7f52f8070b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:25.786 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.766+0000 7f52ffe95640 10 monclient(hunting): picked mon.noname-a con 0x7f52f8140b30 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:49:25.786 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.766+0000 7f52ffe95640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:25.786 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.766+0000 7f52ffe95640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f52f8145c80 con 0x7f52f8140b30 2024-09-16T03:49:25.786 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.766+0000 7f52ffe95640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:25.786 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.766+0000 7f52ffe95640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f52f80a2190 con 0x7f52f809e070 2024-09-16T03:49:25.786 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.766+0000 7f52ffe95640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:25.786 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.766+0000 7f52ffe95640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f52f8145e00 con 0x7f52f80a2f10 2024-09-16T03:49:25.786 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.766+0000 7f52ffe95640 10 monclient(hunting): _renew_subs 2024-09-16T03:49:25.786 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.766+0000 7f52ffe95640 10 monclient(hunting): authenticate will time out at 1412.738144s 2024-09-16T03:49:25.786 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.766+0000 7f52fd430640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f52f80a2f10 0x7f52f813bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:25.787 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.766+0000 7f52fd430640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f52f80a2f10 0x7f52f813bdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:60996/0 (socket says 172.21.15.42:60996) 2024-09-16T03:49:25.787 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.766+0000 7f52fd430640 1 -- 172.21.15.42:0/467814263 learned_addr learned my addr 172.21.15.42:0/467814263 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:49:25.787 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.766+0000 7f52fd430640 10 monclient(hunting): get_auth_request con 0x7f52f80a2f10 auth_method 0 2024-09-16T03:49:25.787 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.766+0000 7f52fd430640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:25.787 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.766+0000 7f52fd430640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:25.787 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.766+0000 7f52fd430640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:25.787 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.766+0000 7f52fe432640 1 --2- 172.21.15.42:0/467814263 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f52f8140b30 0x7f52f8070b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:25.787 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.766+0000 7f52fe432640 10 monclient(hunting): get_auth_request con 0x7f52f8140b30 auth_method 0 2024-09-16T03:49:25.787 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.766+0000 7f52fe432640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:25.787 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.766+0000 7f52fe432640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:25.787 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.766+0000 7f52fe432640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:25.787 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.766+0000 7f52fd430640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:25.787 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.766+0000 7f52fd430640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:25.788 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.766+0000 7f52fd430640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:25.788 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.766+0000 7f52fe432640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:25.788 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.766+0000 7f52fe432640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:25.788 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.766+0000 7f52fe432640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:25.788 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.766+0000 7f52fdc31640 1 --2- 172.21.15.42:0/467814263 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f52f809e070 0x7f52f81d3290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:25.788 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.766+0000 7f52fdc31640 10 monclient(hunting): get_auth_request con 0x7f52f809e070 auth_method 0 2024-09-16T03:49:25.788 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.766+0000 7f52fdc31640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:25.788 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.766+0000 7f52fdc31640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:25.788 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.766+0000 7f52fdc31640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:25.788 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.766+0000 7f52fe432640 10 monclient(hunting): handle_auth_done global_id 764655 payload 824 2024-09-16T03:49:25.788 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.766+0000 7f52fe432640 1 -- 172.21.15.42:0/467814263 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f52f80a2f10 msgr2=0x7f52f813bdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:25.788 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.766+0000 7f52fe432640 1 --2- 172.21.15.42:0/467814263 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f52f80a2f10 0x7f52f813bdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:25.788 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.766+0000 7f52fe432640 1 -- 172.21.15.42:0/467814263 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f52f809e070 msgr2=0x7f52f81d3290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:25.789 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.766+0000 7f52fe432640 1 --2- 172.21.15.42:0/467814263 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f52f809e070 0x7f52f81d3290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:25.789 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.766+0000 7f52fe432640 10 monclient: _finish_hunting 0 2024-09-16T03:49:25.789 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.766+0000 7f52fe432640 1 monclient: found mon.noname-a 2024-09-16T03:49:25.789 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.766+0000 7f52fe432640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:49:25.789 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.766+0000 7f52fe432640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:49:25.789 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.766+0000 7f52fe432640 1 -- 172.21.15.42:0/467814263 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52f8126480 con 0x7f52f8140b30 2024-09-16T03:49:25.789 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.766+0000 7f52fd430640 1 --2- 172.21.15.42:0/467814263 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f52f80a2f10 0x7f52f813bdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:49:25.789 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.766+0000 7f52fe432640 1 --2- 172.21.15.42:0/467814263 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f52f8140b30 0x7f52f8070b50 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f52f0007d90 tx=0x7f52f000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:25.789 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.766+0000 7f52ee7fc640 10 client.?.objecter ms_handle_connect 0x7f52f8140b30 2024-09-16T03:49:25.789 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.766+0000 7f52ee7fc640 10 client.?.objecter resend_mon_ops 2024-09-16T03:49:25.789 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.766+0000 7f52ee7fc640 1 -- 172.21.15.42:0/467814263 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52f000edd0 con 0x7f52f8140b30 2024-09-16T03:49:25.789 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.766+0000 7f52ee7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:25.789 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.766+0000 7f52ee7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:49:25.789 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.766+0000 7f52ee7fc640 10 monclient: dump: 2024-09-16T03:49:25.790 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:49:25.790 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:49:25.790 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:25.790 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:25.790 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:25.790 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:49:25.790 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:49:25.790 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:49:25.790 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:49:25.790 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:49:25.790 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.766+0000 7f52ee7fc640 10 monclient: _finish_auth 0 2024-09-16T03:49:25.790 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.766+0000 7f52ee7fc640 10 monclient: _check_auth_tickets 2024-09-16T03:49:25.790 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.766+0000 7f52ee7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:49:25.790 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.766+0000 7f52ee7fc640 1 -- 172.21.15.42:0/467814263 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f52f00039a0 con 0x7f52f8140b30 2024-09-16T03:49:25.791 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.766+0000 7f52ffe95640 5 monclient: authenticate success, global_id 764655 2024-09-16T03:49:25.791 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.766+0000 7f52ffe95640 10 monclient: _renew_subs 2024-09-16T03:49:25.791 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.766+0000 7f52ffe95640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:49:25.791 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.766+0000 7f52ffe95640 1 -- 172.21.15.42:0/467814263 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52f8126760 con 0x7f52f8140b30 2024-09-16T03:49:25.791 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.766+0000 7f52ee7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:49:25.791 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.766+0000 7f52ee7fc640 1 -- 172.21.15.42:0/467814263 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52f0003d90 con 0x7f52f8140b30 2024-09-16T03:49:25.791 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.766+0000 7f52ee7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:25.791 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.766+0000 7f52ee7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:49:25.791 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.766+0000 7f52ee7fc640 10 monclient: dump: 2024-09-16T03:49:25.791 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:49:25.791 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:49:25.791 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:25.791 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:25.791 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:25.791 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:49:25.792 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:49:25.792 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:49:25.792 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:49:25.792 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:49:25.792 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.766+0000 7f52ffe95640 10 client.764655.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:49:25.792 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.766+0000 7f52ffe95640 10 monclient: _renew_subs 2024-09-16T03:49:25.792 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.766+0000 7f52effff640 4 set_mon_vals no callback set 2024-09-16T03:49:25.792 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.766+0000 7f52ffe95640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:49:25.792 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.766+0000 7f52ffe95640 1 -- 172.21.15.42:0/467814263 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52f81251f0 con 0x7f52f8140b30 2024-09-16T03:49:25.792 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.766+0000 7f52ffe95640 1 librados: init done 2024-09-16T03:49:25.792 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.766+0000 7f52ffe95640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:49:25.792 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.766+0000 7f52ffe95640 1 --2- 172.21.15.42:0/467814263 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f52f813c310 0x7f52f802c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:25.793 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.766+0000 7f52ffe95640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:25.793 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.766+0000 7f52ffe95640 1 -- 172.21.15.42:0/467814263 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f52f80a1330 con 0x7f52f813c310 2024-09-16T03:49:25.793 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.770+0000 7f52fdc31640 1 --2- 172.21.15.42:0/467814263 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f52f813c310 0x7f52f802c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:25.793 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.770+0000 7f52fdc31640 10 monclient: get_auth_request con 0x7f52f813c310 auth_method 0 2024-09-16T03:49:25.793 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.770+0000 7f52fdc31640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:25.793 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.770+0000 7f52fdc31640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:25.793 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.770+0000 7f52fdc31640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:25.793 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.770+0000 7f52fdc31640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:25.793 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.770+0000 7f52fdc31640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:25.793 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.770+0000 7f52fdc31640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:25.793 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.770+0000 7f52fdc31640 10 monclient(hunting): handle_auth_done global_id 764658 payload 824 2024-09-16T03:49:25.793 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.770+0000 7f52fdc31640 1 -- 172.21.15.42:0/467814263 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f52f8140230 con 0x7f52f813c310 2024-09-16T03:49:25.793 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.770+0000 7f52ee7fc640 10 client.764655.objecter ms_handle_connect 0x7f52f813c310 2024-09-16T03:49:25.794 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.770+0000 7f52ee7fc640 10 client.764655.objecter resend_mon_ops 2024-09-16T03:49:25.794 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.770+0000 7f52fdc31640 1 --2- 172.21.15.42:0/467814263 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f52f813c310 0x7f52f802c650 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f52f8072350 tx=0x7f52f400e890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:25.794 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.770+0000 7f52ee7fc640 1 -- 172.21.15.42:0/467814263 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52f4019070 con 0x7f52f813c310 2024-09-16T03:49:25.794 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.770+0000 7f52ee7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:25.794 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.770+0000 7f52ee7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:49:25.794 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.770+0000 7f52ee7fc640 10 monclient: dump: 2024-09-16T03:49:25.794 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:49:25.794 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:49:25.794 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:25.794 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:25.794 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:25.794 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:49:25.794 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:49:25.795 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:49:25.795 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:49:25.795 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:49:25.795 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.770+0000 7f52ee7fc640 1 -- 172.21.15.42:0/467814263 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f52f001b020 con 0x7f52f8140b30 2024-09-16T03:49:25.795 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.770+0000 7f52ee7fc640 1 --2- 172.21.15.42:0/467814263 >> v2:172.21.15.154:6812/254845342 conn(0x7f52dc037740 0x7f52dc039c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:25.795 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.770+0000 7f52ee7fc640 1 -- 172.21.15.42:0/467814263 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(247..247 src has 1..247) ==== 6129+0+0 (secure 0 0 0) 0x7f52f00492f0 con 0x7f52f8140b30 2024-09-16T03:49:25.795 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.770+0000 7f52ee7fc640 10 client.764655.objecter ms_dispatch 0x7f52f8000ba0 osd_map(247..247 src has 1..247) 2024-09-16T03:49:25.795 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.770+0000 7f52ee7fc640 3 client.764655.objecter handle_osd_map got epochs [247,247] > 0 2024-09-16T03:49:25.795 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.770+0000 7f52ee7fc640 3 client.764655.objecter handle_osd_map decoding full epoch 247 2024-09-16T03:49:25.795 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.770+0000 7f52ee7fc640 20 client.764655.objecter dump_active .. 0 homeless 2024-09-16T03:49:25.795 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.770+0000 7f52ee7fc640 1 -- 172.21.15.42:0/467814263 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f52f40052e0 con 0x7f52f813c310 2024-09-16T03:49:25.795 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.770+0000 7f52ee7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:49:25.795 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.770+0000 7f52ee7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:49:25.795 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.770+0000 7f52ee7fc640 1 -- 172.21.15.42:0/467814263 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f52f813c310 msgr2=0x7f52f802c650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:25.796 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.770+0000 7f52ee7fc640 1 --2- 172.21.15.42:0/467814263 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f52f813c310 0x7f52f802c650 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f52f8072350 tx=0x7f52f400e890 comp rx=0 tx=0).stop 2024-09-16T03:49:25.796 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.770+0000 7f52ee7fc640 1 -- 172.21.15.42:0/467814263 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f52f813c310 msgr2=0x7f52f802c650 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:49:25.796 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.770+0000 7f52ee7fc640 1 --2- 172.21.15.42:0/467814263 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f52f813c310 0x7f52f802c650 secure :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f52f8072350 tx=0x7f52f400e890 comp rx=0 tx=0).stop 2024-09-16T03:49:25.796 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.770+0000 7f52fd430640 1 --2- 172.21.15.42:0/467814263 >> v2:172.21.15.154:6812/254845342 conn(0x7f52dc037740 0x7f52dc039c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:25.796 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.770+0000 7f52fd430640 10 monclient: get_auth_request con 0x7f52dc037740 auth_method 0 2024-09-16T03:49:25.796 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.770+0000 7f52ee7fc640 10 client.764655.objecter ms_handle_connect 0x7f52dc037740 2024-09-16T03:49:25.796 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.770+0000 7f52fd430640 1 --2- 172.21.15.42:0/467814263 >> v2:172.21.15.154:6812/254845342 conn(0x7f52dc037740 0x7f52dc039c00 secure :-1 s=READY pgs=3515 cs=0 l=1 rev1=1 crypto rx=0x7f52e801aaa0 tx=0x7f52e80a7000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:25.798 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.774+0000 7f52ffe95640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:49:25.798 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.774+0000 7f52ffe95640 1 --2- 172.21.15.42:0/467814263 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f52f802cb90 0x7f52f802cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:25.798 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.774+0000 7f52ffe95640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:25.798 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.774+0000 7f52ffe95640 1 -- 172.21.15.42:0/467814263 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f52f8069040 con 0x7f52f802cb90 2024-09-16T03:49:25.798 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.778+0000 7f52fdc31640 1 --2- 172.21.15.42:0/467814263 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f52f802cb90 0x7f52f802cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:25.798 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.778+0000 7f52fdc31640 10 monclient: get_auth_request con 0x7f52f802cb90 auth_method 0 2024-09-16T03:49:25.798 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.778+0000 7f52fdc31640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:25.798 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.778+0000 7f52fdc31640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:25.799 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.778+0000 7f52fdc31640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:25.799 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.778+0000 7f52fdc31640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:25.799 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.778+0000 7f52fdc31640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:25.799 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.778+0000 7f52fdc31640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:25.799 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.778+0000 7f52fdc31640 10 monclient(hunting): handle_auth_done global_id 764661 payload 824 2024-09-16T03:49:25.799 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.778+0000 7f52fdc31640 1 -- 172.21.15.42:0/467814263 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f52f81d4080 con 0x7f52f802cb90 2024-09-16T03:49:25.799 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.778+0000 7f52ee7fc640 10 client.764655.objecter ms_handle_connect 0x7f52f802cb90 2024-09-16T03:49:25.799 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.778+0000 7f52fdc31640 1 --2- 172.21.15.42:0/467814263 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f52f802cb90 0x7f52f802cfa0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f52f8072350 tx=0x7f52f4000ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:25.799 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.778+0000 7f52ee7fc640 10 client.764655.objecter resend_mon_ops 2024-09-16T03:49:25.800 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.778+0000 7f52ee7fc640 1 -- 172.21.15.42:0/467814263 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52f4003600 con 0x7f52f802cb90 2024-09-16T03:49:25.800 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.778+0000 7f52ee7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:25.800 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.778+0000 7f52ee7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:49:25.800 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.778+0000 7f52ee7fc640 10 monclient: dump: 2024-09-16T03:49:25.800 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:49:25.800 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:49:25.800 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:25.800 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:25.800 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:25.800 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:49:25.800 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:49:25.800 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:49:25.800 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:49:25.801 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:49:25.801 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.778+0000 7f52ee7fc640 1 -- 172.21.15.42:0/467814263 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4881 (secure 0 0 0) 0x7f52f4003600 con 0x7f52f802cb90 2024-09-16T03:49:25.801 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.778+0000 7f52ee7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:49:25.801 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.778+0000 7f52ee7fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:49:25.801 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.778+0000 7f52ee7fc640 1 -- 172.21.15.42:0/467814263 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f52f802cb90 msgr2=0x7f52f802cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:25.801 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.778+0000 7f52ee7fc640 1 --2- 172.21.15.42:0/467814263 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f52f802cb90 0x7f52f802cfa0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f52f8072350 tx=0x7f52f4000ee0 comp rx=0 tx=0).stop 2024-09-16T03:49:25.801 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.778+0000 7f52ee7fc640 1 -- 172.21.15.42:0/467814263 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f52f802cb90 msgr2=0x7f52f802cfa0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:49:25.801 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.778+0000 7f52ee7fc640 1 --2- 172.21.15.42:0/467814263 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f52f802cb90 0x7f52f802cfa0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:25.801 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:49:25.801 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:49:25.802 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:49:25.802 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "leader", 2024-09-16T03:49:25.802 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 66, 2024-09-16T03:49:25.802 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 156570, 2024-09-16T03:49:25.802 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:49:25.802 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:49:25.802 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T03:49:25.802 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T03:49:25.802 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:49:25.802 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 4, 2024-09-16T03:49:25.802 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:49:25.802 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:49:25.802 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:49:25.803 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:49:25.803 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:49:25.803 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:49:25.803 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:49:25.803 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:49:25.803 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:49:25.803 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:49:25.803 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:49:25.803 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:49:25.803 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:49:25.803 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:49:25.803 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:49:25.803 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:49:25.803 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:49:25.804 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:49:25.804 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:49:25.804 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:49:25.804 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:49:25.804 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:49:25.804 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:49:25.804 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:49:25.804 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:49:25.804 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:49:25.804 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:49:25.804 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:49:25.804 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:49:25.804 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:49:25.804 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:49:25.805 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:49:25.805 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:49:25.805 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:49:25.805 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:49:25.805 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:49:25.805 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:49:25.805 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:49:25.805 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:49:25.805 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:49:25.805 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:49:25.805 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:49:25.805 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:49:25.805 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:49:25.806 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:49:25.806 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:49:25.806 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:49:25.806 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:49:25.806 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:49:25.806 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:49:25.806 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:49:25.806 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:49:25.806 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:49:25.806 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:49:25.806 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:49:25.806 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:49:25.806 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:49:25.807 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:49:25.807 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:49:25.807 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:49:25.807 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:49:25.807 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:49:25.807 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:49:25.807 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:49:25.807 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:49:25.807 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:49:25.807 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:49:25.807 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:49:25.807 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:49:25.807 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:49:25.807 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:49:25.808 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:49:25.808 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:49:25.808 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:49:25.808 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:49:25.808 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:49:25.808 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:49:25.808 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:49:25.808 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:49:25.808 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:49:25.808 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:49:25.808 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:49:25.808 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:49:25.808 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:49:25.808 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:49:25.809 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:49:25.809 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:49:25.809 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:49:25.809 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:49:25.809 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:49:25.809 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:49:25.809 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:49:25.809 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:49:25.809 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:49:25.809 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:49:25.809 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:49:25.809 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:49:25.809 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:49:25.809 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:49:25.810 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:49:25.810 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:49:25.810 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:49:25.810 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:49:25.810 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:49:25.810 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:49:25.810 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:49:25.810 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:49:25.810 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:49:25.810 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:49:25.810 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:49:25.810 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:49:25.810 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:49:25.810 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:49:25.811 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:49:25.811 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:49:25.811 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:49:25.811 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:49:25.811 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:49:25.811 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:49:25.811 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:49:25.811 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:49:25.811 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:49:25.811 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:49:25.811 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:49:25.811 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:49:25.811 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:49:25.811 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:49:25.811 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:49:25.812 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:49:25.812 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:49:25.812 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:49:25.812 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:49:25.812 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:49:25.812 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:49:25.812 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:49:25.812 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:49:25.812 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:49:25.812 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:49:25.812 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:49:25.812 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:49:25.812 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:49:25.812 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T03:49:25.813 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:49:25.813 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:49:25.813 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:49:25.813 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 5 2024-09-16T03:49:25.813 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:49:25.813 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:49:25.813 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:49:25.813 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:49:25.813 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:49:25.813 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:49:25.813 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 8 2024-09-16T03:49:25.813 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:49:25.813 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:49:25.813 INFO:teuthology.orchestra.run.smithi042.stdout: "mgr": [ 2024-09-16T03:49:25.814 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:49:25.814 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:49:25.814 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:49:25.814 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:49:25.814 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:49:25.814 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:49:25.814 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:49:25.814 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:49:25.814 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:49:25.816 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.778+0000 7f52ffe95640 10 librados: watch_flush enter 2024-09-16T03:49:25.816 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.778+0000 7f52ffe95640 10 librados: watch_flush exit 2024-09-16T03:49:25.816 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.778+0000 7f52ffe95640 20 client.764655.objecter shutdown clearing up homeless session... 2024-09-16T03:49:25.816 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.778+0000 7f52ffe95640 10 client.764655.objecter successfully canceled tick 2024-09-16T03:49:25.816 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.778+0000 7f52ffe95640 1 -- 172.21.15.42:0/467814263 >> v2:172.21.15.154:6812/254845342 conn(0x7f52dc037740 msgr2=0x7f52dc039c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:25.816 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.778+0000 7f52ffe95640 1 --2- 172.21.15.42:0/467814263 >> v2:172.21.15.154:6812/254845342 conn(0x7f52dc037740 0x7f52dc039c00 secure :-1 s=READY pgs=3515 cs=0 l=1 rev1=1 crypto rx=0x7f52e801aaa0 tx=0x7f52e80a7000 comp rx=0 tx=0).stop 2024-09-16T03:49:25.816 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.778+0000 7f52ffe95640 10 monclient: shutdown 2024-09-16T03:49:25.816 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.778+0000 7f52ffe95640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:49:25.816 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.778+0000 7f52ffe95640 1 -- 172.21.15.42:0/467814263 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f52f8140b30 msgr2=0x7f52f8070b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:25.816 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.778+0000 7f52ffe95640 1 --2- 172.21.15.42:0/467814263 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f52f8140b30 0x7f52f8070b50 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f52f0007d90 tx=0x7f52f000e4b0 comp rx=0 tx=0).stop 2024-09-16T03:49:25.816 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.778+0000 7f52fe432640 1 -- 172.21.15.42:0/467814263 reap_dead start 2024-09-16T03:49:25.817 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.778+0000 7f52fe432640 1 -- 172.21.15.42:0/467814263 reap_dead start 2024-09-16T03:49:25.817 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.778+0000 7f52ffe95640 1 -- 172.21.15.42:0/467814263 shutdown_connections 2024-09-16T03:49:25.817 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.778+0000 7f52ffe95640 1 -- 172.21.15.42:0/467814263 >> 172.21.15.42:0/467814263 conn(0x7f52f80a2420 msgr2=0x7f52f8141da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:49:25.817 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.778+0000 7f52ffe95640 1 -- 172.21.15.42:0/467814263 shutdown_connections 2024-09-16T03:49:25.817 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.778+0000 7f52ffe95640 1 -- 172.21.15.42:0/467814263 wait complete. 2024-09-16T03:49:25.817 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.778+0000 7f52ffe95640 1 librados: shutdown 2024-09-16T03:49:25.817 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:49:25.817 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T03:49:25.817 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T03:49:25.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.926+0000 7fd8cc91e640 10 monclient: get_monmap_and_config 2024-09-16T03:49:25.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.926+0000 7fd8cc91e640 10 monclient: build_initial_monmap 2024-09-16T03:49:25.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.926+0000 7fd8cc91e640 10 monclient: monmap: 2024-09-16T03:49:25.932 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:49:25.932 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:49:25.932 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:49:25.931308+0000 2024-09-16T03:49:25.933 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:49:25.931308+0000 2024-09-16T03:49:25.933 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:49:25.933 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:49:25.933 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:49:25.933 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:49:25.933 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:49:25.933 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:49:25.933 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:49:25.933 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:49:25.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.930+0000 7fd8cc91e640 1 Processor -- start 2024-09-16T03:49:25.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.930+0000 7fd8cc91e640 1 -- start start 2024-09-16T03:49:25.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.930+0000 7fd8cc91e640 10 monclient: init 2024-09-16T03:49:25.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.930+0000 7fd8cc91e640 10 monclient: _reopen_session rank -1 2024-09-16T03:49:25.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.930+0000 7fd8cc91e640 10 monclient: _add_conns ranks=[1,3,2,0,4] 2024-09-16T03:49:25.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.930+0000 7fd8cc91e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd8c80a39b0 addr v1:172.21.15.154:6789/0 2024-09-16T03:49:25.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.930+0000 7fd8cc91e640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd8c80a3d90 0x7fd8c813c380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:25.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.930+0000 7fd8cc91e640 10 monclient(hunting): picked mon.noname-a con 0x7fd8c80a3d90 addr v2:172.21.15.42:3300/0 2024-09-16T03:49:25.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.930+0000 7fd8cc91e640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd8c813c8c0 0x7fd8c813ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:25.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.930+0000 7fd8cc91e640 10 monclient(hunting): picked mon.noname-b con 0x7fd8c813c8c0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:49:25.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.930+0000 7fd8cc91e640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd8c8134290 con 0x7fd8c80a39b0 2024-09-16T03:49:25.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.930+0000 7fd8cc91e640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:25.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.930+0000 7fd8cc91e640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7fd8c809c140 con 0x7fd8c80a3d90 2024-09-16T03:49:25.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.930+0000 7fd8cc91e640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:25.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.930+0000 7fd8cc91e640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fd8c809c420 con 0x7fd8c813c8c0 2024-09-16T03:49:25.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.930+0000 7fd8cc91e640 10 monclient(hunting): _renew_subs 2024-09-16T03:49:25.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.930+0000 7fd8cc91e640 10 monclient(hunting): authenticate will time out at 1412.901255s 2024-09-16T03:49:25.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.930+0000 7fd8c6d9d640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd8c813c8c0 0x7fd8c813ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:25.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.930+0000 7fd8c6d9d640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd8c813c8c0 0x7fd8c813ecb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:32778/0 (socket says 172.21.15.42:32778) 2024-09-16T03:49:25.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.930+0000 7fd8c6d9d640 1 -- 172.21.15.42:0/3792936281 learned_addr learned my addr 172.21.15.42:0/3792936281 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:49:25.938 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.930+0000 7fd8c6d9d640 10 monclient(hunting): get_auth_request con 0x7fd8c813c8c0 auth_method 0 2024-09-16T03:49:25.938 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.930+0000 7fd8c6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:25.938 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.930+0000 7fd8c6d9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:25.938 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.930+0000 7fd8c6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:25.938 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.930+0000 7fd8c5d9b640 1 --2- 172.21.15.42:0/3792936281 >> v2:172.21.15.42:3300/0 conn(0x7fd8c80a3d90 0x7fd8c813c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:25.938 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.930+0000 7fd8c5d9b640 10 monclient(hunting): get_auth_request con 0x7fd8c80a3d90 auth_method 0 2024-09-16T03:49:25.938 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.930+0000 7fd8c5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:25.938 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.930+0000 7fd8c5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:25.938 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.930+0000 7fd8c5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:25.939 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.930+0000 7fd8c6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:25.939 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.930+0000 7fd8c6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:25.939 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.930+0000 7fd8c6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:25.939 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.930+0000 7fd8c5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:25.939 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.930+0000 7fd8c5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:25.939 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.930+0000 7fd8c5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:25.939 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.930+0000 7fd8c6d9d640 10 monclient(hunting): handle_auth_done global_id 784244 payload 274 2024-09-16T03:49:25.939 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.930+0000 7fd8c6d9d640 1 -- 172.21.15.42:0/3792936281 >> v2:172.21.15.42:3300/0 conn(0x7fd8c80a3d90 msgr2=0x7fd8c813c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:49:25.939 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.930+0000 7fd8c6d9d640 1 --2- 172.21.15.42:0/3792936281 >> v2:172.21.15.42:3300/0 conn(0x7fd8c80a3d90 0x7fd8c813c380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:25.939 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.930+0000 7fd8c6d9d640 1 -- 172.21.15.42:0/3792936281 >> v1:172.21.15.154:6789/0 conn(0x7fd8c80a39b0 legacy=0x7fd8c8133b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:25.940 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.930+0000 7fd8c6d9d640 10 monclient: _finish_hunting 0 2024-09-16T03:49:25.940 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.930+0000 7fd8c6d9d640 1 monclient: found mon.noname-b 2024-09-16T03:49:25.940 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.930+0000 7fd8c6d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:49:25.940 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.930+0000 7fd8c6d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T03:49:25.940 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.930+0000 7fd8c6d9d640 1 -- 172.21.15.42:0/3792936281 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8c806dd60 con 0x7fd8c813c8c0 2024-09-16T03:49:25.940 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.930+0000 7fd8c5d9b640 1 --2- 172.21.15.42:0/3792936281 >> v2:172.21.15.42:3300/0 conn(0x7fd8c80a3d90 0x7fd8c813c380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:49:25.946 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.930+0000 7fd8c6d9d640 1 --2- 172.21.15.42:0/3792936281 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd8c813c8c0 0x7fd8c813ecb0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fd8c0009e50 tx=0x7fd8c001c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a3e019418bb686b8 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:25.947 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.934+0000 7fd8c559a640 1 -- 172.21.15.42:0/3792936281 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8c00a55b0 con 0x7fd8c813c8c0 2024-09-16T03:49:25.947 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.934+0000 7fd8c559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:25.947 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.934+0000 7fd8c559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:49:25.947 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.934+0000 7fd8c559a640 10 monclient: dump: 2024-09-16T03:49:25.947 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:49:25.947 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:49:25.947 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:25.947 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:25.947 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:25.947 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:49:25.947 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:49:25.947 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:49:25.948 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:49:25.948 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:49:25.948 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.934+0000 7fd8c559a640 10 monclient: _finish_auth 0 2024-09-16T03:49:25.948 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.934+0000 7fd8c559a640 10 monclient: _check_auth_tickets 2024-09-16T03:49:25.948 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.934+0000 7fd8c559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:49:25.948 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.934+0000 7fd8cc91e640 5 monclient: authenticate success, global_id 784244 2024-09-16T03:49:25.948 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.934+0000 7fd8cc91e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:49:25.948 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.934+0000 7fd8c559a640 1 -- 172.21.15.42:0/3792936281 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd8c00a5750 con 0x7fd8c813c8c0 2024-09-16T03:49:25.948 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.934+0000 7fd8c559a640 10 monclient: handle_config config(0 keys) 2024-09-16T03:49:25.948 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.934+0000 7fd8cc91e640 10 monclient: get_monmap_and_config success 2024-09-16T03:49:25.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.934+0000 7fd8cc91e640 4 set_mon_vals no callback set 2024-09-16T03:49:25.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.934+0000 7fd8c559a640 1 -- 172.21.15.42:0/3792936281 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8c00a5b20 con 0x7fd8c813c8c0 2024-09-16T03:49:25.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.934+0000 7fd8c559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:25.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.934+0000 7fd8c559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:49:25.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.934+0000 7fd8c559a640 10 monclient: dump: 2024-09-16T03:49:25.950 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:49:25.950 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:49:25.950 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:25.950 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:25.950 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:25.950 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:49:25.950 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:49:25.950 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:49:25.950 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:49:25.950 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:49:25.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.934+0000 7fd8cc91e640 10 monclient: shutdown 2024-09-16T03:49:25.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.934+0000 7fd8cc91e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:49:25.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.934+0000 7fd8cc91e640 1 -- 172.21.15.42:0/3792936281 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd8c813c8c0 msgr2=0x7fd8c813ecb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:25.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.934+0000 7fd8cc91e640 1 --2- 172.21.15.42:0/3792936281 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd8c813c8c0 0x7fd8c813ecb0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fd8c0009e50 tx=0x7fd8c001c8f0 comp rx=0 tx=0).stop 2024-09-16T03:49:25.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.934+0000 7fd8cc91e640 1 -- 172.21.15.42:0/3792936281 shutdown_connections 2024-09-16T03:49:25.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.934+0000 7fd8cc91e640 1 --2- 172.21.15.42:0/3792936281 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd8c813c8c0 0x7fd8c813ecb0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:25.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.934+0000 7fd8cc91e640 1 --2- 172.21.15.42:0/3792936281 >> v2:172.21.15.42:3300/0 conn(0x7fd8c80a3d90 0x7fd8c813c380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:25.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.934+0000 7fd8cc91e640 1 -- 172.21.15.42:0/3792936281 >> 172.21.15.42:0/3792936281 conn(0x7fd8c80a9fb0 msgr2=0x7fd8c80aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:49:25.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.934+0000 7fd8cc91e640 1 -- 172.21.15.42:0/3792936281 shutdown_connections 2024-09-16T03:49:25.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.934+0000 7fd8cc91e640 1 -- 172.21.15.42:0/3792936281 wait complete. 2024-09-16T03:49:25.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.934+0000 7fd8cc91e640 10 monclient: build_initial_monmap 2024-09-16T03:49:25.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.934+0000 7fd8cc91e640 10 monclient: monmap: 2024-09-16T03:49:25.952 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:49:25.952 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:49:25.952 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:49:25.952 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:49:25.952 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:49:25.952 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:49:25.952 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:49:25.952 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:49:25.952 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:49:25.952 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:49:25.953 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.934+0000 7fd8cc91e640 1 librados: starting msgr at 2024-09-16T03:49:25.953 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.934+0000 7fd8cc91e640 1 librados: starting objecter 2024-09-16T03:49:25.953 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.934+0000 7fd8cc91e640 1 Processor -- start 2024-09-16T03:49:25.953 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.934+0000 7fd8cc91e640 1 -- start start 2024-09-16T03:49:25.953 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.934+0000 7fd8cc91e640 1 librados: setting wanted keys 2024-09-16T03:49:25.953 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.934+0000 7fd8cc91e640 1 librados: calling monclient init 2024-09-16T03:49:25.953 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.934+0000 7fd8cc91e640 10 monclient: init 2024-09-16T03:49:25.953 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.934+0000 7fd8cc91e640 10 monclient: _reopen_session rank -1 2024-09-16T03:49:25.953 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.934+0000 7fd8cc91e640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:49:25.953 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.934+0000 7fd8cc91e640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd8c80a39b0 0x7fd8c812d620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:25.953 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.934+0000 7fd8cc91e640 10 monclient(hunting): picked mon.noname-a con 0x7fd8c80a39b0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:49:25.953 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.934+0000 7fd8cc91e640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd8c80a3d90 0x7fd8c812db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:25.954 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.934+0000 7fd8cc91e640 10 monclient(hunting): picked mon.noname-c con 0x7fd8c80a3d90 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:49:25.954 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.934+0000 7fd8cc91e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd8c813c8c0 0x7fd8c81cf130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:25.954 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.934+0000 7fd8cc91e640 10 monclient(hunting): picked mon.noname-b con 0x7fd8c813c8c0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:49:25.954 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.934+0000 7fd8cc91e640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:25.954 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.934+0000 7fd8cc91e640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fd8c8141da0 con 0x7fd8c80a39b0 2024-09-16T03:49:25.954 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.934+0000 7fd8cc91e640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:25.954 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.934+0000 7fd8cc91e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd8c8141c20 con 0x7fd8c813c8c0 2024-09-16T03:49:25.954 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.934+0000 7fd8cc91e640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:25.954 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.934+0000 7fd8cc91e640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fd8c8141f20 con 0x7fd8c80a3d90 2024-09-16T03:49:25.954 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.934+0000 7fd8cc91e640 10 monclient(hunting): _renew_subs 2024-09-16T03:49:25.954 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.934+0000 7fd8cc91e640 10 monclient(hunting): authenticate will time out at 1412.906195s 2024-09-16T03:49:25.954 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.934+0000 7fd8c5d9b640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd8c80a3d90 0x7fd8c812db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:25.955 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.934+0000 7fd8c5d9b640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd8c80a3d90 0x7fd8c812db60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:32780/0 (socket says 172.21.15.42:32780) 2024-09-16T03:49:25.955 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.934+0000 7fd8c5d9b640 1 -- 172.21.15.42:0/293355197 learned_addr learned my addr 172.21.15.42:0/293355197 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:49:25.955 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.934+0000 7fd8c5d9b640 10 monclient(hunting): get_auth_request con 0x7fd8c80a3d90 auth_method 0 2024-09-16T03:49:25.955 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.934+0000 7fd8c5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:25.955 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.934+0000 7fd8c5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:25.955 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.934+0000 7fd8c5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:25.955 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.934+0000 7fd8c659c640 1 --2- 172.21.15.42:0/293355197 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd8c80a39b0 0x7fd8c812d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:25.955 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.934+0000 7fd8c659c640 10 monclient(hunting): get_auth_request con 0x7fd8c80a39b0 auth_method 0 2024-09-16T03:49:25.955 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.934+0000 7fd8c659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:25.955 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.934+0000 7fd8c659c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:25.955 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.934+0000 7fd8c659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:25.955 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.934+0000 7fd8c5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:25.956 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.934+0000 7fd8c5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:25.956 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.934+0000 7fd8c5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:25.956 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.934+0000 7fd8c659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:25.956 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.934+0000 7fd8c659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:25.956 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.934+0000 7fd8c659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:25.956 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.934+0000 7fd8c6d9d640 1 --2- 172.21.15.42:0/293355197 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd8c813c8c0 0x7fd8c81cf130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:25.956 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.934+0000 7fd8c6d9d640 10 monclient(hunting): get_auth_request con 0x7fd8c813c8c0 auth_method 0 2024-09-16T03:49:25.956 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.934+0000 7fd8c6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:25.956 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.934+0000 7fd8c6d9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:25.956 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.934+0000 7fd8c6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:25.956 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.934+0000 7fd8c5d9b640 10 monclient(hunting): handle_auth_done global_id 784247 payload 824 2024-09-16T03:49:25.956 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.934+0000 7fd8c5d9b640 1 -- 172.21.15.42:0/293355197 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd8c813c8c0 msgr2=0x7fd8c81cf130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:25.956 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.934+0000 7fd8c5d9b640 1 --2- 172.21.15.42:0/293355197 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd8c813c8c0 0x7fd8c81cf130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:25.957 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.934+0000 7fd8c5d9b640 1 -- 172.21.15.42:0/293355197 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd8c80a39b0 msgr2=0x7fd8c812d620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:25.957 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.934+0000 7fd8c5d9b640 1 --2- 172.21.15.42:0/293355197 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd8c80a39b0 0x7fd8c812d620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:25.957 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.934+0000 7fd8c5d9b640 10 monclient: _finish_hunting 0 2024-09-16T03:49:25.957 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.934+0000 7fd8c5d9b640 1 monclient: found mon.noname-c 2024-09-16T03:49:25.957 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.934+0000 7fd8c5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:49:25.957 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.934+0000 7fd8c5d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:49:25.957 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.934+0000 7fd8c5d9b640 1 -- 172.21.15.42:0/293355197 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8c813edc0 con 0x7fd8c80a3d90 2024-09-16T03:49:25.957 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.934+0000 7fd8c5d9b640 1 --2- 172.21.15.42:0/293355197 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd8c80a3d90 0x7fd8c812db60 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fd8bc00ec10 tx=0x7fd8bc00c520 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:25.957 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.934+0000 7fd8b6ffd640 10 client.?.objecter ms_handle_connect 0x7fd8c80a3d90 2024-09-16T03:49:25.957 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.934+0000 7fd8b6ffd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:49:25.957 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.934+0000 7fd8b6ffd640 1 -- 172.21.15.42:0/293355197 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8bc017070 con 0x7fd8c80a3d90 2024-09-16T03:49:25.957 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.934+0000 7fd8b6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:25.957 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.934+0000 7fd8b6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:49:25.958 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.934+0000 7fd8b6ffd640 10 monclient: dump: 2024-09-16T03:49:25.958 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:49:25.958 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:49:25.958 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:25.958 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:25.958 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:25.958 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:49:25.958 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:49:25.958 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:49:25.958 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:49:25.958 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:49:25.958 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.934+0000 7fd8b6ffd640 10 monclient: _finish_auth 0 2024-09-16T03:49:25.958 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.934+0000 7fd8b6ffd640 10 monclient: _check_auth_tickets 2024-09-16T03:49:25.959 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.934+0000 7fd8b6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:49:25.959 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.934+0000 7fd8b6ffd640 1 -- 172.21.15.42:0/293355197 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd8bc00cc30 con 0x7fd8c80a3d90 2024-09-16T03:49:25.959 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.934+0000 7fd8cc91e640 5 monclient: authenticate success, global_id 784247 2024-09-16T03:49:25.959 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.934+0000 7fd8cc91e640 10 monclient: _renew_subs 2024-09-16T03:49:25.959 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.934+0000 7fd8cc91e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:49:25.959 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.934+0000 7fd8cc91e640 1 -- 172.21.15.42:0/293355197 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8c806f680 con 0x7fd8c80a3d90 2024-09-16T03:49:25.959 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.934+0000 7fd8b6ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:49:25.959 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.934+0000 7fd8b6ffd640 1 -- 172.21.15.42:0/293355197 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8bc013470 con 0x7fd8c80a3d90 2024-09-16T03:49:25.959 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.934+0000 7fd8b6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:25.959 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.934+0000 7fd8b6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:49:25.959 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.934+0000 7fd8b6ffd640 10 monclient: dump: 2024-09-16T03:49:25.959 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:49:25.959 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:49:25.959 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:25.960 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:25.960 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:25.960 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:49:25.960 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:49:25.960 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:49:25.960 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:49:25.960 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:49:25.960 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.934+0000 7fd8cc91e640 10 client.784247.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:49:25.960 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.934+0000 7fd8c4d99640 4 set_mon_vals no callback set 2024-09-16T03:49:25.960 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.934+0000 7fd8cc91e640 10 monclient: _renew_subs 2024-09-16T03:49:25.960 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.934+0000 7fd8cc91e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:49:25.960 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.934+0000 7fd8cc91e640 1 -- 172.21.15.42:0/293355197 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8c813a760 con 0x7fd8c80a3d90 2024-09-16T03:49:25.961 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.934+0000 7fd8cc91e640 1 librados: init done 2024-09-16T03:49:25.961 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.934+0000 7fd8cc91e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:49:25.961 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.934+0000 7fd8cc91e640 1 --2- 172.21.15.42:0/293355197 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd8c81cf670 0x7fd8c81cfa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:25.961 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.934+0000 7fd8cc91e640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:25.961 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.934+0000 7fd8cc91e640 1 -- 172.21.15.42:0/293355197 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fd8c81cff90 con 0x7fd8c81cf670 2024-09-16T03:49:25.961 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.938+0000 7fd8b6ffd640 1 -- 172.21.15.42:0/293355197 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fd8bc01a070 con 0x7fd8c80a3d90 2024-09-16T03:49:25.961 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.938+0000 7fd8b6ffd640 1 --2- 172.21.15.42:0/293355197 >> v2:172.21.15.154:6812/254845342 conn(0x7fd8a8037840 0x7fd8a8039d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:25.961 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.938+0000 7fd8b6ffd640 1 -- 172.21.15.42:0/293355197 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(247..247 src has 1..247) ==== 6129+0+0 (secure 0 0 0) 0x7fd8bc049080 con 0x7fd8c80a3d90 2024-09-16T03:49:25.961 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.938+0000 7fd8b6ffd640 10 client.784247.objecter ms_dispatch 0x7fd8c8000ba0 osd_map(247..247 src has 1..247) 2024-09-16T03:49:25.961 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.938+0000 7fd8b6ffd640 3 client.784247.objecter handle_osd_map got epochs [247,247] > 0 2024-09-16T03:49:25.961 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.938+0000 7fd8b6ffd640 3 client.784247.objecter handle_osd_map decoding full epoch 247 2024-09-16T03:49:25.961 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.938+0000 7fd8b6ffd640 20 client.784247.objecter dump_active .. 0 homeless 2024-09-16T03:49:25.961 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.938+0000 7fd8c659c640 1 --2- 172.21.15.42:0/293355197 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd8c81cf670 0x7fd8c81cfa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:25.962 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.938+0000 7fd8c659c640 10 monclient: get_auth_request con 0x7fd8c81cf670 auth_method 0 2024-09-16T03:49:25.962 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.938+0000 7fd8c659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:25.962 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.938+0000 7fd8c659c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:25.962 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.938+0000 7fd8c659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:25.962 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.938+0000 7fd8c659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:25.962 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.938+0000 7fd8c659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:25.962 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.938+0000 7fd8c659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:25.962 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.938+0000 7fd8c6d9d640 1 --2- 172.21.15.42:0/293355197 >> v2:172.21.15.154:6812/254845342 conn(0x7fd8a8037840 0x7fd8a8039d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:25.962 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.938+0000 7fd8c659c640 10 monclient(hunting): handle_auth_done global_id 784250 payload 824 2024-09-16T03:49:25.962 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.938+0000 7fd8c659c640 1 -- 172.21.15.42:0/293355197 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd8c8133b80 con 0x7fd8c81cf670 2024-09-16T03:49:25.962 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.938+0000 7fd8c6d9d640 10 monclient: get_auth_request con 0x7fd8a8037840 auth_method 0 2024-09-16T03:49:25.962 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.938+0000 7fd8c659c640 1 --2- 172.21.15.42:0/293355197 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd8c81cf670 0x7fd8c81cfa50 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fd8b800f540 tx=0x7fd8b800fa00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:25.963 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.938+0000 7fd8b6ffd640 10 client.784247.objecter ms_handle_connect 0x7fd8c81cf670 2024-09-16T03:49:25.963 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.938+0000 7fd8b6ffd640 10 client.784247.objecter resend_mon_ops 2024-09-16T03:49:25.963 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.938+0000 7fd8b6ffd640 1 -- 172.21.15.42:0/293355197 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8b8019070 con 0x7fd8c81cf670 2024-09-16T03:49:25.963 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.938+0000 7fd8b6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:25.963 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.938+0000 7fd8b6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:49:25.963 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.938+0000 7fd8b6ffd640 10 monclient: dump: 2024-09-16T03:49:25.963 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:49:25.963 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:49:25.963 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:25.963 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:25.963 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:25.963 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:49:25.963 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:49:25.964 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:49:25.964 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:49:25.964 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:49:25.964 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.938+0000 7fd8b6ffd640 1 -- 172.21.15.42:0/293355197 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fd8b8005320 con 0x7fd8c81cf670 2024-09-16T03:49:25.964 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.938+0000 7fd8b6ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:49:25.964 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.938+0000 7fd8b6ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:49:25.964 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.938+0000 7fd8b6ffd640 1 -- 172.21.15.42:0/293355197 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd8c81cf670 msgr2=0x7fd8c81cfa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:25.964 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.938+0000 7fd8b6ffd640 1 --2- 172.21.15.42:0/293355197 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd8c81cf670 0x7fd8c81cfa50 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fd8b800f540 tx=0x7fd8b800fa00 comp rx=0 tx=0).stop 2024-09-16T03:49:25.964 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.938+0000 7fd8b6ffd640 1 -- 172.21.15.42:0/293355197 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd8c81cf670 msgr2=0x7fd8c81cfa50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:49:25.964 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.938+0000 7fd8b6ffd640 1 --2- 172.21.15.42:0/293355197 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd8c81cf670 0x7fd8c81cfa50 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:25.964 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.938+0000 7fd8b6ffd640 10 client.784247.objecter ms_handle_connect 0x7fd8a8037840 2024-09-16T03:49:25.964 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.938+0000 7fd8c6d9d640 1 --2- 172.21.15.42:0/293355197 >> v2:172.21.15.154:6812/254845342 conn(0x7fd8a8037840 0x7fd8a8039d00 secure :-1 s=READY pgs=3516 cs=0 l=1 rev1=1 crypto rx=0x7fd8c00ae000 tx=0x7fd8c00a7070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:25.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.942+0000 7fd8cc91e640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:49:25.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.942+0000 7fd8cc91e640 1 --2- 172.21.15.42:0/293355197 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd8c812f500 0x7fd8c812f910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:25.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.942+0000 7fd8cc91e640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:25.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.942+0000 7fd8cc91e640 1 -- 172.21.15.42:0/293355197 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fd8c812e0a0 con 0x7fd8c812f500 2024-09-16T03:49:25.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.942+0000 7fd8c659c640 1 --2- 172.21.15.42:0/293355197 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd8c812f500 0x7fd8c812f910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:25.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.942+0000 7fd8c659c640 10 monclient: get_auth_request con 0x7fd8c812f500 auth_method 0 2024-09-16T03:49:25.967 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.942+0000 7fd8c659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:25.967 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.942+0000 7fd8c659c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:25.967 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.942+0000 7fd8c659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:25.967 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.942+0000 7fd8c659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:25.967 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.942+0000 7fd8c659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:25.967 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.942+0000 7fd8c659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:25.967 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.942+0000 7fd8c659c640 10 monclient(hunting): handle_auth_done global_id 784253 payload 824 2024-09-16T03:49:25.967 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.942+0000 7fd8c659c640 1 -- 172.21.15.42:0/293355197 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fd8c8068e40 con 0x7fd8c812f500 2024-09-16T03:49:25.967 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.946+0000 7fd8b6ffd640 10 client.784247.objecter ms_handle_connect 0x7fd8c812f500 2024-09-16T03:49:25.967 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.946+0000 7fd8c659c640 1 --2- 172.21.15.42:0/293355197 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd8c812f500 0x7fd8c812f910 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fd8b800f540 tx=0x7fd8b8007f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:25.967 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.946+0000 7fd8b6ffd640 10 client.784247.objecter resend_mon_ops 2024-09-16T03:49:25.967 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.946+0000 7fd8b6ffd640 1 -- 172.21.15.42:0/293355197 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8b8019070 con 0x7fd8c812f500 2024-09-16T03:49:25.967 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.946+0000 7fd8b6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:25.967 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.946+0000 7fd8b6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:49:25.968 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.946+0000 7fd8b6ffd640 10 monclient: dump: 2024-09-16T03:49:25.968 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:49:25.968 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:49:25.968 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:25.968 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:25.968 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:25.968 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:49:25.968 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:49:25.968 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:49:25.968 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:49:25.968 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:49:25.968 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.946+0000 7fd8b6ffd640 1 -- 172.21.15.42:0/293355197 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4711 (secure 0 0 0) 0x7fd8b8007610 con 0x7fd8c812f500 2024-09-16T03:49:25.968 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.946+0000 7fd8b6ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:49:25.968 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.946+0000 7fd8b6ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:49:25.969 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.946+0000 7fd8b6ffd640 1 -- 172.21.15.42:0/293355197 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd8c812f500 msgr2=0x7fd8c812f910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:25.969 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.946+0000 7fd8b6ffd640 1 --2- 172.21.15.42:0/293355197 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd8c812f500 0x7fd8c812f910 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fd8b800f540 tx=0x7fd8b8007f90 comp rx=0 tx=0).stop 2024-09-16T03:49:25.969 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.946+0000 7fd8b6ffd640 1 -- 172.21.15.42:0/293355197 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd8c812f500 msgr2=0x7fd8c812f910 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:49:25.969 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.946+0000 7fd8b6ffd640 1 --2- 172.21.15.42:0/293355197 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd8c812f500 0x7fd8c812f910 secure :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fd8b800f540 tx=0x7fd8b8007f90 comp rx=0 tx=0).stop 2024-09-16T03:49:25.969 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:49:25.969 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:49:25.969 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:49:25.969 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T03:49:25.969 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 66, 2024-09-16T03:49:25.969 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 57847, 2024-09-16T03:49:25.970 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:49:25.970 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:49:25.970 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T03:49:25.970 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T03:49:25.970 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:49:25.970 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 4, 2024-09-16T03:49:25.970 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:49:25.970 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:49:25.970 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:49:25.970 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:49:25.970 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:49:25.970 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:49:25.970 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:49:25.970 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:49:25.971 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:49:25.971 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:49:25.971 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:49:25.971 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:49:25.971 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:49:25.971 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:49:25.971 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:49:25.971 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:49:25.971 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:49:25.971 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:49:25.971 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:49:25.971 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:49:25.971 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:49:25.972 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:49:25.972 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:49:25.972 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:49:25.972 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:49:25.972 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:49:25.972 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:49:25.972 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:49:25.972 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:49:25.972 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:49:25.972 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:49:25.972 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:49:25.973 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:49:25.973 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:49:25.973 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:49:25.973 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:49:25.973 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:49:25.973 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:49:25.973 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:49:25.973 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:49:25.973 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:49:25.973 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:49:25.973 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:49:25.973 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:49:25.973 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:49:25.974 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:49:25.974 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:49:25.974 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:49:25.974 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:49:25.974 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:49:25.974 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:49:25.974 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:49:25.974 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:49:25.974 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:49:25.974 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:49:25.974 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:49:25.974 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:49:25.974 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:49:25.974 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:49:25.975 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:49:25.975 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:49:25.975 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:49:25.975 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:49:25.975 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:49:25.975 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:49:25.975 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:49:25.975 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:49:25.976 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:49:25.976 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:49:25.976 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:49:25.976 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:49:25.976 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:49:25.976 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:49:25.976 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:49:25.976 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:49:25.976 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:49:25.976 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:49:25.976 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:49:25.976 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:49:25.977 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:49:25.977 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:49:25.977 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:49:25.977 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:49:25.977 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:49:25.977 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:49:25.977 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:49:25.977 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:49:25.977 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:49:25.977 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:49:25.977 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:49:25.977 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:49:25.977 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:49:25.978 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:49:25.978 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:49:25.978 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:49:25.978 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:49:25.978 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:49:25.978 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:49:25.978 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:49:25.978 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:49:25.978 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:49:25.978 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:49:25.978 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:49:25.978 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:49:25.978 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:49:25.979 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:49:25.979 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:49:25.979 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:49:25.979 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:49:25.979 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:49:25.979 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:49:25.979 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:49:25.979 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:49:25.979 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:49:25.979 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:49:25.979 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:49:25.979 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:49:25.979 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:49:25.979 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:49:25.980 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:49:25.980 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:49:25.980 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:49:25.980 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:49:25.980 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:49:25.980 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:49:25.980 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:49:25.980 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:49:25.980 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:49:25.980 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:49:25.980 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:49:25.980 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:49:25.980 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:49:25.981 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:49:25.981 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:49:25.981 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:49:25.981 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:49:25.981 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:49:25.981 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:49:25.981 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:49:25.981 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:49:25.981 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:49:25.981 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T03:49:25.981 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:49:25.981 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:49:25.981 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:49:25.981 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:49:25.982 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:49:25.982 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:49:25.982 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:49:25.982 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:49:25.982 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:49:25.982 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:49:25.982 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 3 2024-09-16T03:49:25.982 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:49:25.982 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:49:25.982 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:49:25.982 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:49:25.982 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:49:25.984 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.946+0000 7fd8cc91e640 10 librados: watch_flush enter 2024-09-16T03:49:25.984 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.946+0000 7fd8cc91e640 10 librados: watch_flush exit 2024-09-16T03:49:25.984 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.946+0000 7fd8cc91e640 20 client.784247.objecter shutdown clearing up homeless session... 2024-09-16T03:49:25.984 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.946+0000 7fd8cc91e640 10 client.784247.objecter successfully canceled tick 2024-09-16T03:49:25.984 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.946+0000 7fd8cc91e640 1 -- 172.21.15.42:0/293355197 >> v2:172.21.15.154:6812/254845342 conn(0x7fd8a8037840 msgr2=0x7fd8a8039d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:25.984 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.946+0000 7fd8cc91e640 1 --2- 172.21.15.42:0/293355197 >> v2:172.21.15.154:6812/254845342 conn(0x7fd8a8037840 0x7fd8a8039d00 secure :-1 s=READY pgs=3516 cs=0 l=1 rev1=1 crypto rx=0x7fd8c00ae000 tx=0x7fd8c00a7070 comp rx=0 tx=0).stop 2024-09-16T03:49:25.984 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.946+0000 7fd8cc91e640 10 monclient: shutdown 2024-09-16T03:49:25.985 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.946+0000 7fd8cc91e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:49:25.985 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.946+0000 7fd8cc91e640 1 -- 172.21.15.42:0/293355197 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd8c80a3d90 msgr2=0x7fd8c812db60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:25.985 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.946+0000 7fd8cc91e640 1 --2- 172.21.15.42:0/293355197 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd8c80a3d90 0x7fd8c812db60 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fd8bc00ec10 tx=0x7fd8bc00c520 comp rx=0 tx=0).stop 2024-09-16T03:49:25.985 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.946+0000 7fd8c6d9d640 1 -- 172.21.15.42:0/293355197 reap_dead start 2024-09-16T03:49:25.985 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.946+0000 7fd8c6d9d640 1 -- 172.21.15.42:0/293355197 reap_dead start 2024-09-16T03:49:25.985 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.946+0000 7fd8cc91e640 1 -- 172.21.15.42:0/293355197 shutdown_connections 2024-09-16T03:49:25.985 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.946+0000 7fd8cc91e640 1 -- 172.21.15.42:0/293355197 >> 172.21.15.42:0/293355197 conn(0x7fd8c80a9fb0 msgr2=0x7fd8c8131980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:49:25.985 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.946+0000 7fd8cc91e640 1 -- 172.21.15.42:0/293355197 shutdown_connections 2024-09-16T03:49:25.985 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.946+0000 7fd8cc91e640 1 -- 172.21.15.42:0/293355197 wait complete. 2024-09-16T03:49:25.985 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:25.946+0000 7fd8cc91e640 1 librados: shutdown 2024-09-16T03:49:25.986 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:49:25.986 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T03:49:25.986 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T03:49:26.113 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.106+0000 7f20bf966640 10 monclient: get_monmap_and_config 2024-09-16T03:49:26.113 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.106+0000 7f20bf966640 10 monclient: build_initial_monmap 2024-09-16T03:49:26.113 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.106+0000 7f20bf966640 10 monclient: monmap: 2024-09-16T03:49:26.113 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:49:26.113 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:49:26.113 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:49:26.112001+0000 2024-09-16T03:49:26.113 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:49:26.112001+0000 2024-09-16T03:49:26.113 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:49:26.114 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:49:26.114 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:49:26.114 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:49:26.114 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:49:26.114 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:49:26.114 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:49:26.114 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:49:26.116 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.110+0000 7f20bf966640 1 Processor -- start 2024-09-16T03:49:26.116 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.110+0000 7f20bf966640 1 -- start start 2024-09-16T03:49:26.116 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.110+0000 7f20bf966640 10 monclient: init 2024-09-16T03:49:26.117 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.110+0000 7f20bf966640 10 monclient: _reopen_session rank -1 2024-09-16T03:49:26.117 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.110+0000 7f20bf966640 10 monclient: _add_conns ranks=[1,3,2,4,0] 2024-09-16T03:49:26.117 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.110+0000 7f20bf966640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f20b80a1330 addr v1:172.21.15.154:6789/0 2024-09-16T03:49:26.117 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.110+0000 7f20bf966640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f20b809c920 0x7f20b813c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:26.117 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.110+0000 7f20bf966640 10 monclient(hunting): picked mon.noname-a con 0x7f20b809c920 addr v2:172.21.15.42:3300/0 2024-09-16T03:49:26.117 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.110+0000 7f20bf966640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f20b8072d40 0x7f20b813eac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:26.117 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.110+0000 7f20bf966640 10 monclient(hunting): picked mon.noname-b con 0x7f20b8072d40 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:49:26.117 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.110+0000 7f20bf966640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f20b8072350 con 0x7f20b80a1330 2024-09-16T03:49:26.117 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.110+0000 7f20bf966640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:26.117 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.110+0000 7f20bf966640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f20b8072b50 con 0x7f20b809c920 2024-09-16T03:49:26.117 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.110+0000 7f20bf966640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:26.117 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.110+0000 7f20bf966640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f20b809bc60 con 0x7f20b8072d40 2024-09-16T03:49:26.117 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.110+0000 7f20bf966640 10 monclient(hunting): _renew_subs 2024-09-16T03:49:26.118 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.110+0000 7f20bf966640 10 monclient(hunting): authenticate will time out at 1413.081383s 2024-09-16T03:49:26.118 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.110+0000 7f20bcf01640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f20b809c920 0x7f20b813c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:26.118 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.110+0000 7f20bdf03640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f20b8072d40 0x7f20b813eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:26.118 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.110+0000 7f20bcf01640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f20b809c920 0x7f20b813c570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:59200/0 (socket says 172.21.15.42:59200) 2024-09-16T03:49:26.118 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.110+0000 7f20bcf01640 1 -- 172.21.15.42:0/1234466955 learned_addr learned my addr 172.21.15.42:0/1234466955 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:49:26.118 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.110+0000 7f20bcf01640 10 monclient(hunting): get_auth_request con 0x7f20b809c920 auth_method 0 2024-09-16T03:49:26.118 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.110+0000 7f20bcf01640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:26.118 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.110+0000 7f20bcf01640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:26.118 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.110+0000 7f20bcf01640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:26.118 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.110+0000 7f20bdf03640 10 monclient(hunting): get_auth_request con 0x7f20b8072d40 auth_method 0 2024-09-16T03:49:26.118 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.110+0000 7f20bdf03640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:26.118 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.110+0000 7f20bdf03640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:26.118 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.110+0000 7f20bdf03640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:26.119 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.110+0000 7f20bcf01640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:26.119 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.110+0000 7f20bcf01640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:26.119 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.110+0000 7f20bcf01640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:26.119 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.110+0000 7f20bdf03640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:26.119 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.110+0000 7f20bdf03640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:26.119 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.110+0000 7f20bdf03640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:26.119 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.110+0000 7f20bcf01640 10 monclient(hunting): handle_auth_done global_id 764670 payload 274 2024-09-16T03:49:26.119 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.110+0000 7f20bcf01640 1 -- 172.21.15.42:0/1234466955 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f20b8072d40 msgr2=0x7f20b813eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:49:26.119 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.110+0000 7f20bcf01640 1 --2- 172.21.15.42:0/1234466955 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f20b8072d40 0x7f20b813eac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:26.119 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.110+0000 7f20bcf01640 1 -- 172.21.15.42:0/1234466955 >> v1:172.21.15.154:6789/0 conn(0x7f20b80a1330 legacy=0x7f20b80a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:49:26.119 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.110+0000 7f20bcf01640 10 monclient: _finish_hunting 0 2024-09-16T03:49:26.119 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.110+0000 7f20bcf01640 1 monclient: found mon.noname-a 2024-09-16T03:49:26.119 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.110+0000 7f20bcf01640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:49:26.119 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.110+0000 7f20bcf01640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:49:26.120 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.110+0000 7f20bcf01640 1 -- 172.21.15.42:0/1234466955 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20b809bf40 con 0x7f20b809c920 2024-09-16T03:49:26.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.114+0000 7f20bcf01640 1 --2- 172.21.15.42:0/1234466955 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f20b809c920 0x7f20b813c570 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f20a8009870 tx=0x7f20a801c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d60b29c2159bc41f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:26.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.114+0000 7f20affff640 1 -- 172.21.15.42:0/1234466955 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20a80b0070 con 0x7f20b809c920 2024-09-16T03:49:26.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.114+0000 7f20affff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:26.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.114+0000 7f20affff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:49:26.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.114+0000 7f20affff640 10 monclient: dump: 2024-09-16T03:49:26.127 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:49:26.127 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:49:26.127 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:26.127 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:26.127 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:26.127 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:49:26.128 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:49:26.128 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:49:26.128 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:49:26.128 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:49:26.128 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.114+0000 7f20affff640 10 monclient: _finish_auth 0 2024-09-16T03:49:26.128 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.114+0000 7f20affff640 10 monclient: _check_auth_tickets 2024-09-16T03:49:26.128 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.114+0000 7f20affff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:49:26.128 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.114+0000 7f20affff640 1 -- 172.21.15.42:0/1234466955 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f20a80ab600 con 0x7f20b809c920 2024-09-16T03:49:26.128 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.114+0000 7f20bf966640 5 monclient: authenticate success, global_id 764670 2024-09-16T03:49:26.128 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.114+0000 7f20bf966640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:49:26.128 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.114+0000 7f20affff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:49:26.128 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.114+0000 7f20bf966640 10 monclient: get_monmap_and_config success 2024-09-16T03:49:26.128 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.114+0000 7f20affff640 1 -- 172.21.15.42:0/1234466955 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20a80ab9d0 con 0x7f20b809c920 2024-09-16T03:49:26.128 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.114+0000 7f20bf966640 4 set_mon_vals no callback set 2024-09-16T03:49:26.128 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.114+0000 7f20affff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:26.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.114+0000 7f20affff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:49:26.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.114+0000 7f20affff640 10 monclient: dump: 2024-09-16T03:49:26.129 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:49:26.129 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:49:26.129 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:26.129 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:26.129 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:26.129 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:49:26.129 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:49:26.129 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:49:26.129 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:49:26.129 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:49:26.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.114+0000 7f20bf966640 10 monclient: shutdown 2024-09-16T03:49:26.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.114+0000 7f20bf966640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:49:26.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.114+0000 7f20bf966640 1 -- 172.21.15.42:0/1234466955 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f20b809c920 msgr2=0x7f20b813c570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:26.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.114+0000 7f20bf966640 1 --2- 172.21.15.42:0/1234466955 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f20b809c920 0x7f20b813c570 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f20a8009870 tx=0x7f20a801c3e0 comp rx=0 tx=0).stop 2024-09-16T03:49:26.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.114+0000 7f20bf966640 1 -- 172.21.15.42:0/1234466955 shutdown_connections 2024-09-16T03:49:26.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.114+0000 7f20bf966640 1 --2- 172.21.15.42:0/1234466955 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f20b8072d40 0x7f20b813eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:26.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.114+0000 7f20bf966640 1 --2- 172.21.15.42:0/1234466955 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f20b809c920 0x7f20b813c570 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:26.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.114+0000 7f20bf966640 1 -- 172.21.15.42:0/1234466955 >> 172.21.15.42:0/1234466955 conn(0x7f20b80a2420 msgr2=0x7f20b80a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:49:26.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.114+0000 7f20bf966640 1 -- 172.21.15.42:0/1234466955 shutdown_connections 2024-09-16T03:49:26.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.114+0000 7f20bf966640 1 -- 172.21.15.42:0/1234466955 wait complete. 2024-09-16T03:49:26.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.114+0000 7f20bf966640 10 monclient: build_initial_monmap 2024-09-16T03:49:26.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.114+0000 7f20bf966640 10 monclient: monmap: 2024-09-16T03:49:26.130 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:49:26.131 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:49:26.131 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:49:26.131 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:49:26.131 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:49:26.131 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:49:26.131 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:49:26.131 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:49:26.131 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:49:26.131 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:49:26.131 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.114+0000 7f20bf966640 1 librados: starting msgr at 2024-09-16T03:49:26.131 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.114+0000 7f20bf966640 1 librados: starting objecter 2024-09-16T03:49:26.131 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.114+0000 7f20bf966640 1 Processor -- start 2024-09-16T03:49:26.131 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.114+0000 7f20bf966640 1 -- start start 2024-09-16T03:49:26.132 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.114+0000 7f20bf966640 1 librados: setting wanted keys 2024-09-16T03:49:26.132 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.114+0000 7f20bf966640 1 librados: calling monclient init 2024-09-16T03:49:26.132 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.114+0000 7f20bf966640 10 monclient: init 2024-09-16T03:49:26.132 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.114+0000 7f20bf966640 10 monclient: _reopen_session rank -1 2024-09-16T03:49:26.132 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.114+0000 7f20bf966640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T03:49:26.132 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.114+0000 7f20bf966640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f20b8072d40 0x7f20b81cf110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:26.132 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.114+0000 7f20bf966640 10 monclient(hunting): picked mon.noname-b con 0x7f20b8072d40 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:49:26.132 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.114+0000 7f20bf966640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f20b809c920 0x7f20b8133260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:26.132 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.114+0000 7f20bf966640 10 monclient(hunting): picked mon.noname-c con 0x7f20b809c920 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:49:26.132 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.114+0000 7f20bf966640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f20b80a1330 0x7f20b813c310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:26.132 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.114+0000 7f20bf966640 10 monclient(hunting): picked mon.noname-a con 0x7f20b80a1330 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:49:26.132 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.114+0000 7f20bf966640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:26.132 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.114+0000 7f20bf966640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f20b8141cc0 con 0x7f20b80a1330 2024-09-16T03:49:26.132 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.114+0000 7f20bf966640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:26.133 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.114+0000 7f20bf966640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f20b8141b40 con 0x7f20b8072d40 2024-09-16T03:49:26.133 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.114+0000 7f20bf966640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:26.133 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.114+0000 7f20bf966640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f20b8141e40 con 0x7f20b809c920 2024-09-16T03:49:26.133 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.114+0000 7f20bf966640 10 monclient(hunting): _renew_subs 2024-09-16T03:49:26.133 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.114+0000 7f20bf966640 10 monclient(hunting): authenticate will time out at 1413.086664s 2024-09-16T03:49:26.133 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.114+0000 7f20bcf01640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f20b809c920 0x7f20b8133260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:26.133 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.114+0000 7f20bdf03640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f20b80a1330 0x7f20b813c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:26.133 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.114+0000 7f20bcf01640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f20b809c920 0x7f20b8133260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:32816/0 (socket says 172.21.15.42:32816) 2024-09-16T03:49:26.133 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.114+0000 7f20bcf01640 1 -- 172.21.15.42:0/2158394205 learned_addr learned my addr 172.21.15.42:0/2158394205 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:49:26.133 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.114+0000 7f20bcf01640 10 monclient(hunting): get_auth_request con 0x7f20b809c920 auth_method 0 2024-09-16T03:49:26.133 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.114+0000 7f20bcf01640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:26.133 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.114+0000 7f20bcf01640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:26.133 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.114+0000 7f20bcf01640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:26.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.114+0000 7f20bdf03640 10 monclient(hunting): get_auth_request con 0x7f20b80a1330 auth_method 0 2024-09-16T03:49:26.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.114+0000 7f20bdf03640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:26.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.114+0000 7f20bdf03640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:26.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.114+0000 7f20bdf03640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:26.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.114+0000 7f20bcf01640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:26.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.114+0000 7f20bcf01640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:26.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.114+0000 7f20bcf01640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:26.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.114+0000 7f20bdf03640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:26.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.114+0000 7f20bdf03640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:26.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.114+0000 7f20bdf03640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:26.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.114+0000 7f20bd702640 1 --2- 172.21.15.42:0/2158394205 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f20b8072d40 0x7f20b81cf110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:26.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.114+0000 7f20bd702640 10 monclient(hunting): get_auth_request con 0x7f20b8072d40 auth_method 0 2024-09-16T03:49:26.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.114+0000 7f20bd702640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:26.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.114+0000 7f20bd702640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:26.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.114+0000 7f20bd702640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:26.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.114+0000 7f20bdf03640 10 monclient(hunting): handle_auth_done global_id 764673 payload 824 2024-09-16T03:49:26.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.114+0000 7f20bdf03640 1 -- 172.21.15.42:0/2158394205 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f20b809c920 msgr2=0x7f20b8133260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:26.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.114+0000 7f20bdf03640 1 --2- 172.21.15.42:0/2158394205 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f20b809c920 0x7f20b8133260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:26.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.114+0000 7f20bdf03640 1 -- 172.21.15.42:0/2158394205 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f20b8072d40 msgr2=0x7f20b81cf110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:26.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.114+0000 7f20bdf03640 1 --2- 172.21.15.42:0/2158394205 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f20b8072d40 0x7f20b81cf110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:26.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.114+0000 7f20bdf03640 10 monclient: _finish_hunting 0 2024-09-16T03:49:26.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.114+0000 7f20bdf03640 1 monclient: found mon.noname-a 2024-09-16T03:49:26.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.114+0000 7f20bdf03640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:49:26.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.114+0000 7f20bdf03640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:49:26.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.114+0000 7f20bdf03640 1 -- 172.21.15.42:0/2158394205 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20b811dbf0 con 0x7f20b80a1330 2024-09-16T03:49:26.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.114+0000 7f20bcf01640 1 --2- 172.21.15.42:0/2158394205 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f20b809c920 0x7f20b8133260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:49:26.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.114+0000 7f20bdf03640 1 --2- 172.21.15.42:0/2158394205 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f20b80a1330 0x7f20b813c310 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f20b000efc0 tx=0x7f20b000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:26.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.114+0000 7f20adffb640 10 client.?.objecter ms_handle_connect 0x7f20b80a1330 2024-09-16T03:49:26.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.114+0000 7f20adffb640 10 client.?.objecter resend_mon_ops 2024-09-16T03:49:26.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.114+0000 7f20adffb640 1 -- 172.21.15.42:0/2158394205 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20b0018020 con 0x7f20b80a1330 2024-09-16T03:49:26.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.114+0000 7f20adffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:26.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.114+0000 7f20adffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:49:26.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.114+0000 7f20adffb640 10 monclient: dump: 2024-09-16T03:49:26.136 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:49:26.136 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:49:26.136 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:26.136 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:26.136 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:26.136 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:49:26.136 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:49:26.136 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:49:26.136 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:49:26.136 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:49:26.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.114+0000 7f20adffb640 10 monclient: _finish_auth 0 2024-09-16T03:49:26.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.114+0000 7f20adffb640 10 monclient: _check_auth_tickets 2024-09-16T03:49:26.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.114+0000 7f20adffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:49:26.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.114+0000 7f20adffb640 1 -- 172.21.15.42:0/2158394205 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f20b0014480 con 0x7f20b80a1330 2024-09-16T03:49:26.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.114+0000 7f20adffb640 10 monclient: handle_config config(0 keys) 2024-09-16T03:49:26.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.114+0000 7f20adffb640 1 -- 172.21.15.42:0/2158394205 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20b0014870 con 0x7f20b80a1330 2024-09-16T03:49:26.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.114+0000 7f20bf966640 5 monclient: authenticate success, global_id 764673 2024-09-16T03:49:26.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.114+0000 7f20bf966640 10 monclient: _renew_subs 2024-09-16T03:49:26.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.114+0000 7f20bf966640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:49:26.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.114+0000 7f20bf966640 1 -- 172.21.15.42:0/2158394205 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20b80a40c0 con 0x7f20b80a1330 2024-09-16T03:49:26.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.114+0000 7f20af7fe640 4 set_mon_vals no callback set 2024-09-16T03:49:26.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.114+0000 7f20adffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:26.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.114+0000 7f20adffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:49:26.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.114+0000 7f20adffb640 10 monclient: dump: 2024-09-16T03:49:26.137 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:49:26.137 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:49:26.138 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:26.138 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:26.138 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:26.138 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:49:26.138 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:49:26.138 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:49:26.138 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:49:26.138 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:49:26.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.114+0000 7f20bf966640 10 client.764673.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:49:26.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.114+0000 7f20bf966640 10 monclient: _renew_subs 2024-09-16T03:49:26.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.114+0000 7f20bf966640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:49:26.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.114+0000 7f20bf966640 1 -- 172.21.15.42:0/2158394205 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20b8073a30 con 0x7f20b80a1330 2024-09-16T03:49:26.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.114+0000 7f20bf966640 1 librados: init done 2024-09-16T03:49:26.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.118+0000 7f20bf966640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:49:26.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.118+0000 7f20bf966640 1 --2- 172.21.15.42:0/2158394205 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f20b81337a0 0x7f20b813c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:26.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.118+0000 7f20bf966640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:26.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.118+0000 7f20bf966640 1 -- 172.21.15.42:0/2158394205 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f20b811d470 con 0x7f20b81337a0 2024-09-16T03:49:26.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.118+0000 7f20adffb640 1 -- 172.21.15.42:0/2158394205 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f20b001b020 con 0x7f20b80a1330 2024-09-16T03:49:26.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.118+0000 7f20adffb640 1 --2- 172.21.15.42:0/2158394205 >> v2:172.21.15.154:6812/254845342 conn(0x7f209c037810 0x7f209c039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:26.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.118+0000 7f20bd702640 1 --2- 172.21.15.42:0/2158394205 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f20b81337a0 0x7f20b813c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:26.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.118+0000 7f20adffb640 1 -- 172.21.15.42:0/2158394205 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(247..247 src has 1..247) ==== 6129+0+0 (secure 0 0 0) 0x7f20b0049e70 con 0x7f20b80a1330 2024-09-16T03:49:26.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.118+0000 7f20adffb640 10 client.764673.objecter ms_dispatch 0x7f20b8000ba0 osd_map(247..247 src has 1..247) 2024-09-16T03:49:26.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.118+0000 7f20adffb640 3 client.764673.objecter handle_osd_map got epochs [247,247] > 0 2024-09-16T03:49:26.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.118+0000 7f20adffb640 3 client.764673.objecter handle_osd_map decoding full epoch 247 2024-09-16T03:49:26.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.118+0000 7f20bd702640 10 monclient: get_auth_request con 0x7f20b81337a0 auth_method 0 2024-09-16T03:49:26.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.118+0000 7f20bd702640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:26.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.118+0000 7f20bd702640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:26.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.118+0000 7f20bd702640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:26.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.118+0000 7f20adffb640 20 client.764673.objecter dump_active .. 0 homeless 2024-09-16T03:49:26.140 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.118+0000 7f20bd702640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:26.140 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.118+0000 7f20bd702640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:26.140 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.118+0000 7f20bd702640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:26.140 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.118+0000 7f20bcf01640 1 --2- 172.21.15.42:0/2158394205 >> v2:172.21.15.154:6812/254845342 conn(0x7f209c037810 0x7f209c039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:26.140 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.118+0000 7f20bcf01640 10 monclient: get_auth_request con 0x7f209c037810 auth_method 0 2024-09-16T03:49:26.140 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.118+0000 7f20bd702640 10 monclient(hunting): handle_auth_done global_id 794128 payload 824 2024-09-16T03:49:26.140 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.118+0000 7f20bd702640 1 -- 172.21.15.42:0/2158394205 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f20b80a1710 con 0x7f20b81337a0 2024-09-16T03:49:26.140 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.118+0000 7f20adffb640 10 client.764673.objecter ms_handle_connect 0x7f209c037810 2024-09-16T03:49:26.140 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.118+0000 7f20bcf01640 1 --2- 172.21.15.42:0/2158394205 >> v2:172.21.15.154:6812/254845342 conn(0x7f209c037810 0x7f209c039cd0 secure :-1 s=READY pgs=3517 cs=0 l=1 rev1=1 crypto rx=0x7f20a8009870 tx=0x7f20a8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:26.140 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.118+0000 7f20adffb640 10 client.764673.objecter ms_handle_connect 0x7f20b81337a0 2024-09-16T03:49:26.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.118+0000 7f20adffb640 10 client.764673.objecter resend_mon_ops 2024-09-16T03:49:26.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.118+0000 7f20bd702640 1 --2- 172.21.15.42:0/2158394205 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f20b81337a0 0x7f20b813c850 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f20b400e450 tx=0x7f20b400e910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:26.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.118+0000 7f20adffb640 1 -- 172.21.15.42:0/2158394205 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20b4007240 con 0x7f20b81337a0 2024-09-16T03:49:26.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.118+0000 7f20adffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:26.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.118+0000 7f20adffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:49:26.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.118+0000 7f20adffb640 10 monclient: dump: 2024-09-16T03:49:26.141 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:49:26.141 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:49:26.141 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:26.141 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:26.141 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:26.141 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:49:26.142 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:49:26.142 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:49:26.142 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:49:26.142 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:49:26.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.118+0000 7f20adffb640 1 -- 172.21.15.42:0/2158394205 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f20b4007240 con 0x7f20b81337a0 2024-09-16T03:49:26.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.118+0000 7f20adffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:49:26.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.118+0000 7f20adffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:49:26.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.118+0000 7f20adffb640 1 -- 172.21.15.42:0/2158394205 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f20b81337a0 msgr2=0x7f20b813c850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:26.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.118+0000 7f20adffb640 1 --2- 172.21.15.42:0/2158394205 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f20b81337a0 0x7f20b813c850 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f20b400e450 tx=0x7f20b400e910 comp rx=0 tx=0).stop 2024-09-16T03:49:26.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.118+0000 7f20adffb640 1 -- 172.21.15.42:0/2158394205 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f20b81337a0 msgr2=0x7f20b813c850 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:49:26.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.118+0000 7f20adffb640 1 --2- 172.21.15.42:0/2158394205 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f20b81337a0 0x7f20b813c850 secure :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f20b400e450 tx=0x7f20b400e910 comp rx=0 tx=0).stop 2024-09-16T03:49:26.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.126+0000 7f20bf966640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:49:26.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.126+0000 7f20bf966640 1 --2- 172.21.15.42:0/2158394205 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f20b813cd90 0x7f20b8138550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:26.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.126+0000 7f20bf966640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:26.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.126+0000 7f20bf966640 1 -- 172.21.15.42:0/2158394205 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f20b813d1a0 con 0x7f20b813cd90 2024-09-16T03:49:26.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.126+0000 7f20bd702640 1 --2- 172.21.15.42:0/2158394205 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f20b813cd90 0x7f20b8138550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:26.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.126+0000 7f20bd702640 10 monclient: get_auth_request con 0x7f20b813cd90 auth_method 0 2024-09-16T03:49:26.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.126+0000 7f20bd702640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:26.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.126+0000 7f20bd702640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:26.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.126+0000 7f20bd702640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:26.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.126+0000 7f20bd702640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:26.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.126+0000 7f20bd702640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:26.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.126+0000 7f20bd702640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:26.145 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.126+0000 7f20bd702640 10 monclient(hunting): handle_auth_done global_id 794131 payload 824 2024-09-16T03:49:26.145 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.126+0000 7f20bd702640 1 -- 172.21.15.42:0/2158394205 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f20b811bf70 con 0x7f20b813cd90 2024-09-16T03:49:26.145 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.126+0000 7f20bd702640 1 --2- 172.21.15.42:0/2158394205 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f20b813cd90 0x7f20b8138550 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f20b400e450 tx=0x7f20b4003770 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:26.145 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.126+0000 7f20adffb640 10 client.764673.objecter ms_handle_connect 0x7f20b813cd90 2024-09-16T03:49:26.145 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.126+0000 7f20adffb640 10 client.764673.objecter resend_mon_ops 2024-09-16T03:49:26.145 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.126+0000 7f20adffb640 1 -- 172.21.15.42:0/2158394205 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20b401e960 con 0x7f20b813cd90 2024-09-16T03:49:26.145 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.126+0000 7f20adffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:26.145 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.126+0000 7f20adffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:49:26.145 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.126+0000 7f20adffb640 10 monclient: dump: 2024-09-16T03:49:26.145 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:49:26.145 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:49:26.145 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:26.145 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:26.145 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:26.146 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:49:26.146 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:49:26.146 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:49:26.146 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:49:26.146 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:49:26.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.126+0000 7f20adffb640 1 -- 172.21.15.42:0/2158394205 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4543 (secure 0 0 0) 0x7f20b4013420 con 0x7f20b813cd90 2024-09-16T03:49:26.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.126+0000 7f20adffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:49:26.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.126+0000 7f20adffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:49:26.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.126+0000 7f20adffb640 1 -- 172.21.15.42:0/2158394205 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f20b813cd90 msgr2=0x7f20b8138550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:26.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.126+0000 7f20adffb640 1 --2- 172.21.15.42:0/2158394205 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f20b813cd90 0x7f20b8138550 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f20b400e450 tx=0x7f20b4003770 comp rx=0 tx=0).stop 2024-09-16T03:49:26.146 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:49:26.147 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:49:26.147 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:49:26.147 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T03:49:26.147 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 66, 2024-09-16T03:49:26.147 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 7967, 2024-09-16T03:49:26.147 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:49:26.147 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:49:26.147 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T03:49:26.147 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T03:49:26.147 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:49:26.147 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 4, 2024-09-16T03:49:26.147 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:49:26.147 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:49:26.147 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:49:26.147 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:49:26.148 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:49:26.148 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:49:26.148 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:49:26.148 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:49:26.148 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:49:26.148 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:49:26.148 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:49:26.148 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:49:26.148 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:49:26.148 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:49:26.148 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:49:26.148 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:49:26.148 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:49:26.148 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:49:26.148 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:49:26.149 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:49:26.149 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:49:26.149 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:49:26.149 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:49:26.149 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:49:26.149 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:49:26.149 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:49:26.149 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:49:26.149 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:49:26.149 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:49:26.149 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:49:26.149 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:49:26.149 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:49:26.149 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:49:26.149 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:49:26.150 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:49:26.150 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:49:26.150 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:49:26.150 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:49:26.150 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:49:26.150 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:49:26.150 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:49:26.150 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:49:26.150 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:49:26.150 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:49:26.150 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:49:26.150 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:49:26.150 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:49:26.151 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:49:26.151 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:49:26.151 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:49:26.151 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:49:26.151 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:49:26.151 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:49:26.151 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:49:26.151 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:49:26.151 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:49:26.151 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:49:26.151 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:49:26.151 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:49:26.152 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:49:26.152 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:49:26.152 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:49:26.152 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:49:26.152 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:49:26.152 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:49:26.152 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:49:26.152 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:49:26.152 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:49:26.152 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:49:26.152 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:49:26.152 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:49:26.152 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:49:26.152 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:49:26.152 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:49:26.153 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:49:26.153 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:49:26.153 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:49:26.153 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:49:26.153 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:49:26.153 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:49:26.153 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:49:26.153 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:49:26.153 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:49:26.154 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:49:26.154 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:49:26.154 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:49:26.154 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:49:26.154 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:49:26.154 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:49:26.154 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:49:26.154 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:49:26.154 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:49:26.154 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:49:26.154 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:49:26.154 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:49:26.154 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:49:26.155 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:49:26.155 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:49:26.155 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:49:26.155 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:49:26.155 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:49:26.155 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:49:26.155 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:49:26.155 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:49:26.155 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:49:26.155 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:49:26.155 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:49:26.155 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:49:26.155 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:49:26.155 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:49:26.155 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:49:26.156 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:49:26.156 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:49:26.156 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:49:26.156 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:49:26.156 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:49:26.156 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:49:26.156 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:49:26.156 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:49:26.156 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:49:26.156 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:49:26.156 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:49:26.156 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:49:26.156 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:49:26.157 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:49:26.157 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:49:26.157 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:49:26.157 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:49:26.157 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:49:26.157 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:49:26.157 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:49:26.157 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:49:26.157 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:49:26.157 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:49:26.157 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:49:26.157 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:49:26.157 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:49:26.158 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:49:26.158 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:49:26.158 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:49:26.158 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:49:26.158 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:49:26.158 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:49:26.158 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:49:26.158 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:49:26.158 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:49:26.158 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:49:26.158 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:49:26.158 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:49:26.158 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:49:26.158 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:49:26.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.126+0000 7f20adffb640 1 -- 172.21.15.42:0/2158394205 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f20b813cd90 msgr2=0x7f20b8138550 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:49:26.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.126+0000 7f20adffb640 1 --2- 172.21.15.42:0/2158394205 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f20b813cd90 0x7f20b8138550 secure :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f20b400e450 tx=0x7f20b4003770 comp rx=0 tx=0).stop 2024-09-16T03:49:26.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.126+0000 7f20bf966640 10 librados: watch_flush enter 2024-09-16T03:49:26.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.126+0000 7f20bf966640 10 librados: watch_flush exit 2024-09-16T03:49:26.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.126+0000 7f20bf966640 20 client.764673.objecter shutdown clearing up homeless session... 2024-09-16T03:49:26.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.126+0000 7f20bf966640 10 client.764673.objecter successfully canceled tick 2024-09-16T03:49:26.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.126+0000 7f20bf966640 1 -- 172.21.15.42:0/2158394205 >> v2:172.21.15.154:6812/254845342 conn(0x7f209c037810 msgr2=0x7f209c039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:26.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.126+0000 7f20bf966640 1 --2- 172.21.15.42:0/2158394205 >> v2:172.21.15.154:6812/254845342 conn(0x7f209c037810 0x7f209c039cd0 secure :-1 s=READY pgs=3517 cs=0 l=1 rev1=1 crypto rx=0x7f20a8009870 tx=0x7f20a8008040 comp rx=0 tx=0).stop 2024-09-16T03:49:26.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.126+0000 7f20bf966640 10 monclient: shutdown 2024-09-16T03:49:26.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.126+0000 7f20bf966640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:49:26.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.126+0000 7f20bf966640 1 -- 172.21.15.42:0/2158394205 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f20b80a1330 msgr2=0x7f20b813c310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:26.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.126+0000 7f20bf966640 1 --2- 172.21.15.42:0/2158394205 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f20b80a1330 0x7f20b813c310 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f20b000efc0 tx=0x7f20b000c490 comp rx=0 tx=0).stop 2024-09-16T03:49:26.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.126+0000 7f20bdf03640 1 -- 172.21.15.42:0/2158394205 reap_dead start 2024-09-16T03:49:26.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.126+0000 7f20bdf03640 1 -- 172.21.15.42:0/2158394205 reap_dead start 2024-09-16T03:49:26.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.126+0000 7f20bf966640 1 -- 172.21.15.42:0/2158394205 shutdown_connections 2024-09-16T03:49:26.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.126+0000 7f20bf966640 1 -- 172.21.15.42:0/2158394205 >> 172.21.15.42:0/2158394205 conn(0x7f20b80a2420 msgr2=0x7f20b80a0a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:49:26.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.126+0000 7f20bf966640 1 -- 172.21.15.42:0/2158394205 shutdown_connections 2024-09-16T03:49:26.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.126+0000 7f20bf966640 1 -- 172.21.15.42:0/2158394205 wait complete. 2024-09-16T03:49:26.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.126+0000 7f20bf966640 1 librados: shutdown 2024-09-16T03:49:26.162 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:49:26.162 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-09-16T03:49:26.162 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-16T03:49:26.162 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-09-16T03:49:26.163 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-16T03:49:26.163 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-16T03:49:26.163 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T03:49:26.174 INFO:tasks.ceph.mon.c:Stopped 2024-09-16T03:49:26.175 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T03:49:26.175 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:49:26.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.326+0000 7f463db06640 10 monclient: get_monmap_and_config 2024-09-16T03:49:26.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.326+0000 7f463db06640 10 monclient: build_initial_monmap 2024-09-16T03:49:26.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.326+0000 7f463db06640 10 monclient: monmap: 2024-09-16T03:49:26.331 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:49:26.331 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:49:26.331 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:49:26.329576+0000 2024-09-16T03:49:26.331 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:49:26.329576+0000 2024-09-16T03:49:26.331 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:49:26.331 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:49:26.331 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:49:26.331 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:49:26.331 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:49:26.331 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:49:26.331 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:49:26.331 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:49:26.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.326+0000 7f463db06640 1 Processor -- start 2024-09-16T03:49:26.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.326+0000 7f463db06640 1 -- start start 2024-09-16T03:49:26.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.326+0000 7f463db06640 10 monclient: init 2024-09-16T03:49:26.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.326+0000 7f463db06640 10 monclient: _reopen_session rank -1 2024-09-16T03:49:26.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.326+0000 7f463db06640 10 monclient: _add_conns ranks=[4,0,1,2,3] 2024-09-16T03:49:26.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.326+0000 7f463db06640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f46380a1330 0x7f463809f180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:26.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.326+0000 7f463db06640 10 monclient(hunting): picked mon.noname-c con 0x7f46380a1330 addr v2:172.21.15.154:3300/0 2024-09-16T03:49:26.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.326+0000 7f463db06640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f46380a1710 addr v1:172.21.15.42:6789/0 2024-09-16T03:49:26.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.326+0000 7f463db06640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f46381393a0 addr v1:172.21.15.154:6789/0 2024-09-16T03:49:26.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.326+0000 7f463db06640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f463809e4c0 con 0x7f46380a1710 2024-09-16T03:49:26.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.326+0000 7f463db06640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f463809f6c0 con 0x7f46381393a0 2024-09-16T03:49:26.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.326+0000 7f463db06640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:26.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.326+0000 7f463db06640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f463809edf0 con 0x7f46380a1330 2024-09-16T03:49:26.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.326+0000 7f463db06640 10 monclient(hunting): _renew_subs 2024-09-16T03:49:26.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.326+0000 7f463db06640 10 monclient(hunting): authenticate will time out at 1413.299527s 2024-09-16T03:49:26.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.326+0000 7f4636ffd640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f46380a1710 0x7f46380a1b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:52170/0 (socket says 172.21.15.42:52170) 2024-09-16T03:49:26.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.326+0000 7f4636ffd640 1 -- 172.21.15.42:0/2061707995 learned_addr learned my addr 172.21.15.42:0/2061707995 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:49:26.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.326+0000 7f46377fe640 1 --2- 172.21.15.42:0/2061707995 >> v2:172.21.15.154:3300/0 conn(0x7f46380a1330 0x7f463809f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:26.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.326+0000 7f46367fc640 1 -- 172.21.15.42:0/2061707995 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3280027284 0 0) 0x7f463809e4c0 con 0x7f46380a1710 2024-09-16T03:49:26.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.330+0000 7f46377fe640 10 monclient(hunting): get_auth_request con 0x7f46380a1330 auth_method 0 2024-09-16T03:49:26.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.330+0000 7f46377fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:26.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.330+0000 7f46377fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:26.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.330+0000 7f46377fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:26.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.330+0000 7f46367fc640 10 monclient(hunting): _negotiate 2024-09-16T03:49:26.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.330+0000 7f46367fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:26.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.330+0000 7f46367fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:26.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.330+0000 7f46367fc640 10 monclient(hunting): my global_id is 764676 2024-09-16T03:49:26.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.330+0000 7f46367fc640 1 -- 172.21.15.42:0/2061707995 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4620003530 con 0x7f46380a1710 2024-09-16T03:49:26.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.330+0000 7f46377fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:26.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.330+0000 7f46377fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:26.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.330+0000 7f46377fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:26.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.330+0000 7f46367fc640 1 -- 172.21.15.42:0/2061707995 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3688770194 0 0) 0x7f4620003530 con 0x7f46380a1710 2024-09-16T03:49:26.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.330+0000 7f46367fc640 1 -- 172.21.15.42:0/2061707995 >> v2:172.21.15.154:3300/0 conn(0x7f46380a1330 msgr2=0x7f463809f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:49:26.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.330+0000 7f46367fc640 1 --2- 172.21.15.42:0/2061707995 >> v2:172.21.15.154:3300/0 conn(0x7f46380a1330 0x7f463809f180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:26.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.330+0000 7f46367fc640 1 -- 172.21.15.42:0/2061707995 >> v1:172.21.15.154:6789/0 conn(0x7f46381393a0 legacy=0x7f463813b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:26.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.330+0000 7f46367fc640 10 monclient: _finish_hunting 0 2024-09-16T03:49:26.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.330+0000 7f46367fc640 1 monclient: found mon.noname-a-legacy 2024-09-16T03:49:26.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.330+0000 7f46367fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:49:26.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.330+0000 7f46367fc640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.42:6789/0 2024-09-16T03:49:26.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.330+0000 7f46367fc640 1 -- 172.21.15.42:0/2061707995 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46380a3170 con 0x7f46380a1710 2024-09-16T03:49:26.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.330+0000 7f46367fc640 10 monclient: _finish_auth 0 2024-09-16T03:49:26.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.330+0000 7f46367fc640 10 monclient: _check_auth_tickets 2024-09-16T03:49:26.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.330+0000 7f46367fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:49:26.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.330+0000 7f46367fc640 1 -- 172.21.15.42:0/2061707995 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2177413734 0 0) 0x7f462c001680 con 0x7f46380a1710 2024-09-16T03:49:26.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.330+0000 7f46367fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:26.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.330+0000 7f46367fc640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T03:49:26.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.330+0000 7f46367fc640 10 monclient: dump: 2024-09-16T03:49:26.342 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:49:26.342 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:49:26.342 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:26.342 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:26.342 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:26.342 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:49:26.342 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:49:26.343 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:49:26.343 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:49:26.343 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:49:26.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.330+0000 7f46367fc640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] but i'm connected to v1:172.21.15.42:6789/0, reconnecting 2024-09-16T03:49:26.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.330+0000 7f46367fc640 10 monclient: _reopen_session rank -1 2024-09-16T03:49:26.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.330+0000 7f46367fc640 1 -- 172.21.15.42:0/2061707995 >> v1:172.21.15.42:6789/0 conn(0x7f46380a1710 legacy=0x7f46380a1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:26.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.330+0000 7f46367fc640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:49:26.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.330+0000 7f46367fc640 1 --2- 172.21.15.42:0/2061707995 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f46200039c0 0x7f4620003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:26.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.330+0000 7f46367fc640 10 monclient(hunting): picked mon.b con 0x7f46200039c0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:49:26.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.330+0000 7f46367fc640 1 --2- 172.21.15.42:0/2061707995 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4620004810 0x7f4620004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:26.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.330+0000 7f46367fc640 10 monclient(hunting): picked mon.a con 0x7f4620004810 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:49:26.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.330+0000 7f46367fc640 1 --2- 172.21.15.42:0/2061707995 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f46200052f0 0x7f462000dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:26.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.330+0000 7f46367fc640 10 monclient(hunting): picked mon.c con 0x7f46200052f0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:49:26.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.330+0000 7f46367fc640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:26.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.330+0000 7f46367fc640 1 -- 172.21.15.42:0/2061707995 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f463809edf0 con 0x7f4620004810 2024-09-16T03:49:26.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.330+0000 7f46367fc640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:26.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.330+0000 7f46367fc640 1 -- 172.21.15.42:0/2061707995 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f462000e3d0 con 0x7f46200039c0 2024-09-16T03:49:26.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.330+0000 7f46367fc640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:26.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.330+0000 7f46367fc640 1 -- 172.21.15.42:0/2061707995 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f462000e5d0 con 0x7f46200052f0 2024-09-16T03:49:26.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.330+0000 7f46367fc640 10 monclient(hunting): _renew_subs 2024-09-16T03:49:26.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.330+0000 7f46367fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:49:26.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.330+0000 7f4637fff640 1 -- 172.21.15.42:0/2061707995 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f46200052f0 msgr2=0x7f462000dbe0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T03:49:26.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.330+0000 7f4637fff640 1 --2- 172.21.15.42:0/2061707995 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f46200052f0 0x7f462000dbe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:49:26.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.330+0000 7f4636ffd640 1 --2- 172.21.15.42:0/2061707995 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4620004810 0x7f4620004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:26.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.330+0000 7f4636ffd640 10 monclient(hunting): get_auth_request con 0x7f4620004810 auth_method 0 2024-09-16T03:49:26.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.330+0000 7f4636ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:26.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.330+0000 7f4636ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:26.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.330+0000 7f4636ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:49:26.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.330+0000 7f46377fe640 1 --2- 172.21.15.42:0/2061707995 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f46200039c0 0x7f4620003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:26.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.330+0000 7f46377fe640 10 monclient(hunting): get_auth_request con 0x7f46200039c0 auth_method 0 2024-09-16T03:49:26.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.330+0000 7f46377fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:26.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.330+0000 7f46377fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:26.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.330+0000 7f46377fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:49:26.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.330+0000 7f4636ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:26.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.330+0000 7f4636ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:26.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.330+0000 7f4636ffd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T03:49:26.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.330+0000 7f46377fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:26.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.330+0000 7f46377fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:26.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.330+0000 7f46377fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T03:49:26.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.330+0000 7f4636ffd640 10 monclient(hunting): handle_auth_done global_id 764676 payload 293 2024-09-16T03:49:26.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.330+0000 7f4636ffd640 1 -- 172.21.15.42:0/2061707995 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f46200052f0 msgr2=0x7f462000dbe0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:49:26.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.330+0000 7f4636ffd640 1 --2- 172.21.15.42:0/2061707995 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f46200052f0 0x7f462000dbe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:26.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.330+0000 7f4636ffd640 1 -- 172.21.15.42:0/2061707995 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f46200039c0 msgr2=0x7f4620003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:49:26.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.330+0000 7f4636ffd640 1 --2- 172.21.15.42:0/2061707995 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f46200039c0 0x7f4620003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:26.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.330+0000 7f4636ffd640 10 monclient: _finish_hunting 0 2024-09-16T03:49:26.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.330+0000 7f4636ffd640 1 monclient: found mon.a 2024-09-16T03:49:26.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.330+0000 7f4637fff640 1 -- 172.21.15.42:0/2061707995 reap_dead start 2024-09-16T03:49:26.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.330+0000 7f4636ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:49:26.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.330+0000 7f4636ffd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:49:26.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.330+0000 7f4636ffd640 1 -- 172.21.15.42:0/2061707995 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4620003530 con 0x7f4620004810 2024-09-16T03:49:26.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.330+0000 7f4636ffd640 10 monclient: _finish_auth 0 2024-09-16T03:49:26.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.330+0000 7f4636ffd640 10 monclient: _check_auth_tickets 2024-09-16T03:49:26.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.330+0000 7f4636ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:49:26.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.330+0000 7f463db06640 5 monclient: authenticate success, global_id 764676 2024-09-16T03:49:26.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.330+0000 7f463db06640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:49:26.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.330+0000 7f4636ffd640 1 --2- 172.21.15.42:0/2061707995 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4620004810 0x7f4620004c90 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f462c001ec0 tx=0x7f462c01cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=992a9670114d3507 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:26.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.330+0000 7f46367fc640 1 -- 172.21.15.42:0/2061707995 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f462c007d40 con 0x7f4620004810 2024-09-16T03:49:26.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.330+0000 7f46367fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:26.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.330+0000 7f46367fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:49:26.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.330+0000 7f46367fc640 10 monclient: dump: 2024-09-16T03:49:26.348 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:49:26.348 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:49:26.348 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:26.348 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:26.348 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:26.348 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:49:26.348 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:49:26.348 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:49:26.348 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:49:26.348 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:49:26.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.330+0000 7f463db06640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:49:26.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.330+0000 7f46367fc640 1 -- 172.21.15.42:0/2061707995 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f462c0a9070 con 0x7f4620004810 2024-09-16T03:49:26.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.330+0000 7f46367fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:49:26.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.330+0000 7f463db06640 10 monclient: get_monmap_and_config success 2024-09-16T03:49:26.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.330+0000 7f463db06640 4 set_mon_vals no callback set 2024-09-16T03:49:26.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.330+0000 7f46367fc640 1 -- 172.21.15.42:0/2061707995 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f462c005d40 con 0x7f4620004810 2024-09-16T03:49:26.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.330+0000 7f46367fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:26.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.330+0000 7f46367fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:49:26.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.330+0000 7f46367fc640 10 monclient: dump: 2024-09-16T03:49:26.349 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:49:26.349 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:49:26.350 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:26.350 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:26.350 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:26.350 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:49:26.350 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:49:26.350 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:49:26.350 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:49:26.350 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:49:26.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.330+0000 7f463db06640 10 monclient: shutdown 2024-09-16T03:49:26.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.330+0000 7f463db06640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:49:26.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.330+0000 7f463db06640 1 -- 172.21.15.42:0/2061707995 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4620004810 msgr2=0x7f4620004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:26.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.330+0000 7f463db06640 1 --2- 172.21.15.42:0/2061707995 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4620004810 0x7f4620004c90 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f462c001ec0 tx=0x7f462c01cc20 comp rx=0 tx=0).stop 2024-09-16T03:49:26.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.330+0000 7f463db06640 1 -- 172.21.15.42:0/2061707995 shutdown_connections 2024-09-16T03:49:26.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.330+0000 7f463db06640 1 --2- 172.21.15.42:0/2061707995 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4620004810 0x7f4620004c90 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:26.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.330+0000 7f463db06640 1 -- 172.21.15.42:0/2061707995 >> 172.21.15.42:0/2061707995 conn(0x7f46380a2420 msgr2=0x7f46380a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:49:26.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.330+0000 7f463db06640 1 -- 172.21.15.42:0/2061707995 shutdown_connections 2024-09-16T03:49:26.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.330+0000 7f463db06640 1 -- 172.21.15.42:0/2061707995 wait complete. 2024-09-16T03:49:26.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.334+0000 7f463db06640 10 monclient: build_initial_monmap 2024-09-16T03:49:26.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.334+0000 7f463db06640 10 monclient: monmap: 2024-09-16T03:49:26.351 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:49:26.351 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:49:26.351 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:49:26.351 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:49:26.351 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:49:26.351 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:49:26.351 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:49:26.351 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:49:26.351 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:49:26.352 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:49:26.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.334+0000 7f463db06640 1 librados: starting msgr at 2024-09-16T03:49:26.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.334+0000 7f463db06640 1 librados: starting objecter 2024-09-16T03:49:26.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.334+0000 7f463db06640 1 Processor -- start 2024-09-16T03:49:26.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.334+0000 7f463db06640 1 -- start start 2024-09-16T03:49:26.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.334+0000 7f463db06640 1 librados: setting wanted keys 2024-09-16T03:49:26.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.334+0000 7f463db06640 1 librados: calling monclient init 2024-09-16T03:49:26.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.334+0000 7f463db06640 10 monclient: init 2024-09-16T03:49:26.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.334+0000 7f463db06640 10 monclient: _reopen_session rank -1 2024-09-16T03:49:26.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.334+0000 7f463db06640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T03:49:26.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.334+0000 7f463db06640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4620004810 0x7f4638133240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:26.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.334+0000 7f463db06640 10 monclient(hunting): picked mon.noname-a con 0x7f4620004810 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:49:26.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.334+0000 7f463db06640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f46381cf8e0 0x7f4638133780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:26.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.334+0000 7f463db06640 10 monclient(hunting): picked mon.noname-b con 0x7f46381cf8e0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:49:26.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.334+0000 7f463db06640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4638122cd0 0x7f4638123150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:26.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.334+0000 7f463db06640 10 monclient(hunting): picked mon.noname-c con 0x7f4638122cd0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:49:26.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.334+0000 7f463db06640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:26.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.334+0000 7f463db06640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f4638134c90 con 0x7f4620004810 2024-09-16T03:49:26.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.334+0000 7f463db06640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:26.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.334+0000 7f463db06640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f4638134b10 con 0x7f46381cf8e0 2024-09-16T03:49:26.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.334+0000 7f463db06640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:26.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.334+0000 7f463db06640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f4638134e10 con 0x7f4638122cd0 2024-09-16T03:49:26.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.334+0000 7f463db06640 10 monclient(hunting): _renew_subs 2024-09-16T03:49:26.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.334+0000 7f463db06640 10 monclient(hunting): authenticate will time out at 1413.305503s 2024-09-16T03:49:26.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.334+0000 7f4637fff640 1 -- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4638122cd0 msgr2=0x7f4638123150 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T03:49:26.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.334+0000 7f4637fff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4638122cd0 0x7f4638123150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:49:26.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.334+0000 7f46377fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4620004810 0x7f4638133240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:26.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.334+0000 7f46377fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4620004810 0x7f4638133240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:59230/0 (socket says 172.21.15.42:59230) 2024-09-16T03:49:26.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.334+0000 7f46377fe640 1 -- 172.21.15.42:0/693294287 learned_addr learned my addr 172.21.15.42:0/693294287 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:49:26.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.334+0000 7f46377fe640 10 monclient(hunting): get_auth_request con 0x7f4620004810 auth_method 0 2024-09-16T03:49:26.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.334+0000 7f46377fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:26.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.334+0000 7f46377fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:26.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.334+0000 7f46377fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:26.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.334+0000 7f46377fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:26.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.334+0000 7f46377fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:26.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.334+0000 7f46377fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:26.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.334+0000 7f4636ffd640 1 --2- 172.21.15.42:0/693294287 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f46381cf8e0 0x7f4638133780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:26.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.334+0000 7f4636ffd640 10 monclient(hunting): get_auth_request con 0x7f46381cf8e0 auth_method 0 2024-09-16T03:49:26.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.334+0000 7f4636ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:26.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.334+0000 7f4636ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:26.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.334+0000 7f4636ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:26.360 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.334+0000 7f46377fe640 10 monclient(hunting): handle_auth_done global_id 764679 payload 824 2024-09-16T03:49:26.360 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.334+0000 7f46377fe640 1 -- 172.21.15.42:0/693294287 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4638122cd0 msgr2=0x7f4638123150 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:49:26.360 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.334+0000 7f46377fe640 1 --2- 172.21.15.42:0/693294287 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4638122cd0 0x7f4638123150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:26.360 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.334+0000 7f46377fe640 1 -- 172.21.15.42:0/693294287 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f46381cf8e0 msgr2=0x7f4638133780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:26.360 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.334+0000 7f46377fe640 1 --2- 172.21.15.42:0/693294287 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f46381cf8e0 0x7f4638133780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:26.360 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.334+0000 7f46377fe640 10 monclient: _finish_hunting 0 2024-09-16T03:49:26.360 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.334+0000 7f46377fe640 1 monclient: found mon.noname-a 2024-09-16T03:49:26.360 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.334+0000 7f46377fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:49:26.360 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.334+0000 7f46377fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:49:26.360 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.334+0000 7f46377fe640 1 -- 172.21.15.42:0/693294287 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4638073a30 con 0x7f4620004810 2024-09-16T03:49:26.360 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.334+0000 7f46377fe640 1 --2- 172.21.15.42:0/693294287 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4620004810 0x7f4638133240 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f4628009f60 tx=0x7f4628009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:26.360 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.334+0000 7f461ffff640 10 client.?.objecter ms_handle_connect 0x7f4620004810 2024-09-16T03:49:26.360 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.334+0000 7f461ffff640 10 client.?.objecter resend_mon_ops 2024-09-16T03:49:26.361 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.334+0000 7f461ffff640 1 -- 172.21.15.42:0/693294287 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4628019070 con 0x7f4620004810 2024-09-16T03:49:26.361 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.334+0000 7f461ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:26.361 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.334+0000 7f461ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:49:26.361 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.334+0000 7f461ffff640 10 monclient: dump: 2024-09-16T03:49:26.361 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:49:26.361 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:49:26.361 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:26.361 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:26.361 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:26.361 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:49:26.361 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:49:26.361 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:49:26.361 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:49:26.362 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:49:26.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.334+0000 7f461ffff640 10 monclient: _finish_auth 0 2024-09-16T03:49:26.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.334+0000 7f461ffff640 10 monclient: _check_auth_tickets 2024-09-16T03:49:26.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.334+0000 7f461ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:49:26.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.334+0000 7f461ffff640 1 -- 172.21.15.42:0/693294287 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f46280039a0 con 0x7f4620004810 2024-09-16T03:49:26.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.334+0000 7f461ffff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:49:26.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.334+0000 7f461ffff640 1 -- 172.21.15.42:0/693294287 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4628003d50 con 0x7f4620004810 2024-09-16T03:49:26.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.334+0000 7f461ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:26.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.334+0000 7f461ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:49:26.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.334+0000 7f461ffff640 10 monclient: dump: 2024-09-16T03:49:26.363 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:49:26.363 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:49:26.363 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:26.363 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:26.363 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:26.363 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:49:26.363 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:49:26.363 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:49:26.363 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:49:26.363 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:49:26.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.334+0000 7f463db06640 5 monclient: authenticate success, global_id 764679 2024-09-16T03:49:26.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.334+0000 7f463db06640 10 monclient: _renew_subs 2024-09-16T03:49:26.364 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.334+0000 7f463db06640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:49:26.364 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.334+0000 7f4635ffb640 4 set_mon_vals no callback set 2024-09-16T03:49:26.364 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.334+0000 7f463db06640 1 -- 172.21.15.42:0/693294287 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4638126c60 con 0x7f4620004810 2024-09-16T03:49:26.364 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.334+0000 7f463db06640 10 client.764679.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:49:26.364 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.334+0000 7f463db06640 10 monclient: _renew_subs 2024-09-16T03:49:26.364 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.334+0000 7f463db06640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:49:26.364 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.334+0000 7f463db06640 1 -- 172.21.15.42:0/693294287 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4638124970 con 0x7f4620004810 2024-09-16T03:49:26.364 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.334+0000 7f463db06640 1 librados: init done 2024-09-16T03:49:26.364 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.334+0000 7f463db06640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:49:26.364 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.334+0000 7f463db06640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:49:26.364 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.334+0000 7f463db06640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:49:26.364 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.334+0000 7f463db06640 1 -- 172.21.15.42:0/693294287 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f463809ba50 con 0x7f4620004810 2024-09-16T03:49:26.364 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.334+0000 7f461ffff640 1 -- 172.21.15.42:0/693294287 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f462801c070 con 0x7f4620004810 2024-09-16T03:49:26.365 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.334+0000 7f461ffff640 1 --2- 172.21.15.42:0/693294287 >> v2:172.21.15.154:6812/254845342 conn(0x7f46240377c0 0x7f4624039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:26.365 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.334+0000 7f461ffff640 1 -- 172.21.15.42:0/693294287 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(247..247 src has 1..247) ==== 6129+0+0 (secure 0 0 0) 0x7f4628049d30 con 0x7f4620004810 2024-09-16T03:49:26.365 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.334+0000 7f461ffff640 10 client.764679.objecter ms_dispatch 0x7f4638000ba0 osd_map(247..247 src has 1..247) 2024-09-16T03:49:26.365 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.334+0000 7f461ffff640 3 client.764679.objecter handle_osd_map got epochs [247,247] > 0 2024-09-16T03:49:26.365 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.334+0000 7f461ffff640 3 client.764679.objecter handle_osd_map decoding full epoch 247 2024-09-16T03:49:26.365 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.334+0000 7f461ffff640 20 client.764679.objecter dump_active .. 0 homeless 2024-09-16T03:49:26.365 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.338+0000 7f4636ffd640 1 --2- 172.21.15.42:0/693294287 >> v2:172.21.15.154:6812/254845342 conn(0x7f46240377c0 0x7f4624039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:26.365 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.338+0000 7f4636ffd640 10 monclient: get_auth_request con 0x7f46240377c0 auth_method 0 2024-09-16T03:49:26.365 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.338+0000 7f4636ffd640 1 --2- 172.21.15.42:0/693294287 >> v2:172.21.15.154:6812/254845342 conn(0x7f46240377c0 0x7f4624039c80 secure :-1 s=READY pgs=3518 cs=0 l=1 rev1=1 crypto rx=0x7f462c0ab000 tx=0x7f462c0b0040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:26.365 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.338+0000 7f461ffff640 10 client.764679.objecter ms_handle_connect 0x7f46240377c0 2024-09-16T03:49:26.365 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.342+0000 7f461ffff640 1 -- 172.21.15.42:0/693294287 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f4638073a30 con 0x7f4620004810 2024-09-16T03:49:26.365 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.342+0000 7f461ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:49:26.365 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.342+0000 7f461ffff640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:49:26.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.558+0000 7f463db06640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:49:26.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.558+0000 7f463db06640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:49:26.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.558+0000 7f463db06640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:49:26.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.558+0000 7f463db06640 1 -- 172.21.15.42:0/693294287 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f4638071020 con 0x7f4620004810 2024-09-16T03:49:26.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.558+0000 7f461ffff640 1 -- 172.21.15.42:0/693294287 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7f4628010040 con 0x7f4620004810 2024-09-16T03:49:26.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.558+0000 7f461ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:49:26.565 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.558+0000 7f461ffff640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:49:26.565 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":66,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":4,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:49:26.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.562+0000 7f463db06640 10 librados: watch_flush enter 2024-09-16T03:49:26.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.562+0000 7f463db06640 10 librados: watch_flush exit 2024-09-16T03:49:26.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.562+0000 7f463db06640 20 client.764679.objecter shutdown clearing up homeless session... 2024-09-16T03:49:26.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.562+0000 7f463db06640 10 client.764679.objecter successfully canceled tick 2024-09-16T03:49:26.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.562+0000 7f463db06640 1 -- 172.21.15.42:0/693294287 >> v2:172.21.15.154:6812/254845342 conn(0x7f46240377c0 msgr2=0x7f4624039c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:26.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.562+0000 7f463db06640 1 --2- 172.21.15.42:0/693294287 >> v2:172.21.15.154:6812/254845342 conn(0x7f46240377c0 0x7f4624039c80 secure :-1 s=READY pgs=3518 cs=0 l=1 rev1=1 crypto rx=0x7f462c0ab000 tx=0x7f462c0b0040 comp rx=0 tx=0).stop 2024-09-16T03:49:26.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.562+0000 7f463db06640 10 monclient: shutdown 2024-09-16T03:49:26.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.562+0000 7f463db06640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:49:26.568 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.562+0000 7f463db06640 1 -- 172.21.15.42:0/693294287 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4620004810 msgr2=0x7f4638133240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:26.568 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.562+0000 7f463db06640 1 --2- 172.21.15.42:0/693294287 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4620004810 0x7f4638133240 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f4628009f60 tx=0x7f4628009f90 comp rx=0 tx=0).stop 2024-09-16T03:49:26.568 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.562+0000 7f463db06640 1 -- 172.21.15.42:0/693294287 shutdown_connections 2024-09-16T03:49:26.568 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.562+0000 7f463db06640 1 --2- 172.21.15.42:0/693294287 >> v2:172.21.15.154:6812/254845342 conn(0x7f46240377c0 0x7f4624039c80 unknown :-1 s=CLOSED pgs=3518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:26.568 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.562+0000 7f463db06640 1 --2- 172.21.15.42:0/693294287 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4638122cd0 0x7f4638123150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:26.568 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.562+0000 7f463db06640 1 --2- 172.21.15.42:0/693294287 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f46381cf8e0 0x7f4638133780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:26.568 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.562+0000 7f463db06640 1 --2- 172.21.15.42:0/693294287 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4620004810 0x7f4638133240 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:26.568 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.562+0000 7f463db06640 1 -- 172.21.15.42:0/693294287 >> 172.21.15.42:0/693294287 conn(0x7f46380a2420 msgr2=0x7f46380a5120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:49:26.568 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.562+0000 7f463db06640 1 -- 172.21.15.42:0/693294287 shutdown_connections 2024-09-16T03:49:26.568 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.562+0000 7f463db06640 1 -- 172.21.15.42:0/693294287 wait complete. 2024-09-16T03:49:26.569 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:26.562+0000 7f463db06640 1 librados: shutdown 2024-09-16T03:49:29.586 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:49:29.722 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.718+0000 7f53b65d2640 10 monclient: get_monmap_and_config 2024-09-16T03:49:29.722 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.718+0000 7f53b65d2640 10 monclient: build_initial_monmap 2024-09-16T03:49:29.722 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.718+0000 7f53b65d2640 10 monclient: monmap: 2024-09-16T03:49:29.722 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:49:29.723 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:49:29.723 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:49:29.721433+0000 2024-09-16T03:49:29.723 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:49:29.721433+0000 2024-09-16T03:49:29.723 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:49:29.723 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:49:29.723 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:49:29.723 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:49:29.723 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:49:29.723 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:49:29.723 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:49:29.723 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:49:29.727 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.718+0000 7f53b65d2640 1 Processor -- start 2024-09-16T03:49:29.728 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.718+0000 7f53b65d2640 1 -- start start 2024-09-16T03:49:29.728 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.718+0000 7f53b65d2640 10 monclient: init 2024-09-16T03:49:29.728 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.718+0000 7f53b65d2640 10 monclient: _reopen_session rank -1 2024-09-16T03:49:29.728 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.718+0000 7f53b65d2640 10 monclient: _add_conns ranks=[2,0,1,4,3] 2024-09-16T03:49:29.728 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.718+0000 7f53b65d2640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f53b00a1330 0x7f53b009f180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:29.728 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.718+0000 7f53b65d2640 10 monclient(hunting): picked mon.noname-b con 0x7f53b00a1330 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:49:29.728 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.718+0000 7f53b65d2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f53b00a1710 addr v1:172.21.15.42:6789/0 2024-09-16T03:49:29.728 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.718+0000 7f53b65d2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f53b01393a0 addr v1:172.21.15.154:6789/0 2024-09-16T03:49:29.728 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.718+0000 7f53b65d2640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53b009e4c0 con 0x7f53b00a1710 2024-09-16T03:49:29.728 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.718+0000 7f53b65d2640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53b009f6c0 con 0x7f53b01393a0 2024-09-16T03:49:29.728 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.718+0000 7f53b65d2640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:29.728 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.718+0000 7f53b65d2640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f53b009edf0 con 0x7f53b00a1330 2024-09-16T03:49:29.729 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.718+0000 7f53b65d2640 10 monclient(hunting): _renew_subs 2024-09-16T03:49:29.729 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.718+0000 7f53b65d2640 10 monclient(hunting): authenticate will time out at 1416.691381s 2024-09-16T03:49:29.729 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.718+0000 7f53affff640 1 -- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f53b00a1330 msgr2=0x7f53b009f180 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T03:49:29.729 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.718+0000 7f53affff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f53b00a1330 0x7f53b009f180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:49:29.729 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.718+0000 7f53af7fe640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f53b00a1710 0x7f53b00a1b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:52184/0 (socket says 172.21.15.42:52184) 2024-09-16T03:49:29.729 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.718+0000 7f53af7fe640 1 -- 172.21.15.42:0/1070734833 learned_addr learned my addr 172.21.15.42:0/1070734833 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:49:29.729 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.718+0000 7f53aeffd640 1 -- 172.21.15.42:0/1070734833 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2577193180 0 0) 0x7f53b009e4c0 con 0x7f53b00a1710 2024-09-16T03:49:29.729 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.718+0000 7f53aeffd640 10 monclient(hunting): _negotiate 2024-09-16T03:49:29.729 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.718+0000 7f53aeffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:29.729 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.718+0000 7f53aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:29.729 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.718+0000 7f53aeffd640 10 monclient(hunting): my global_id is 764682 2024-09-16T03:49:29.729 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.718+0000 7f53aeffd640 1 -- 172.21.15.42:0/1070734833 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5398003530 con 0x7f53b00a1710 2024-09-16T03:49:29.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.722+0000 7f53aeffd640 1 -- 172.21.15.42:0/1070734833 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 984776756 0 0) 0x7f5398003530 con 0x7f53b00a1710 2024-09-16T03:49:29.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.722+0000 7f53aeffd640 1 -- 172.21.15.42:0/1070734833 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f53b00a1330 msgr2=0x7f53b009f180 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:49:29.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.722+0000 7f53aeffd640 1 --2- 172.21.15.42:0/1070734833 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f53b00a1330 0x7f53b009f180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:29.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.722+0000 7f53aeffd640 1 -- 172.21.15.42:0/1070734833 >> v1:172.21.15.154:6789/0 conn(0x7f53b01393a0 legacy=0x7f53b013b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:29.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.722+0000 7f53aeffd640 10 monclient: _finish_hunting 0 2024-09-16T03:49:29.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.722+0000 7f53aeffd640 1 monclient: found mon.noname-a-legacy 2024-09-16T03:49:29.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.722+0000 7f53aeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:49:29.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.722+0000 7f53aeffd640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.42:6789/0 2024-09-16T03:49:29.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.722+0000 7f53aeffd640 1 -- 172.21.15.42:0/1070734833 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53b00a3170 con 0x7f53b00a1710 2024-09-16T03:49:29.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.722+0000 7f53aeffd640 10 monclient: _finish_auth 0 2024-09-16T03:49:29.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.722+0000 7f53aeffd640 10 monclient: _check_auth_tickets 2024-09-16T03:49:29.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.722+0000 7f53aeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:49:29.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.722+0000 7f53aeffd640 1 -- 172.21.15.42:0/1070734833 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2177413734 0 0) 0x7f53a4001680 con 0x7f53b00a1710 2024-09-16T03:49:29.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.722+0000 7f53b65d2640 5 monclient: authenticate success, global_id 764682 2024-09-16T03:49:29.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.722+0000 7f53b65d2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:49:29.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.722+0000 7f53aeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:29.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.722+0000 7f53aeffd640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T03:49:29.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.722+0000 7f53aeffd640 10 monclient: dump: 2024-09-16T03:49:29.731 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:49:29.731 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:49:29.731 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:29.731 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:29.731 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:29.731 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:49:29.731 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:49:29.732 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:49:29.732 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:49:29.732 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:49:29.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.722+0000 7f53aeffd640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] but i'm connected to v1:172.21.15.42:6789/0, reconnecting 2024-09-16T03:49:29.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.722+0000 7f53aeffd640 10 monclient: _reopen_session rank -1 2024-09-16T03:49:29.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.722+0000 7f53aeffd640 1 -- 172.21.15.42:0/1070734833 >> v1:172.21.15.42:6789/0 conn(0x7f53b00a1710 legacy=0x7f53b00a1b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:29.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.722+0000 7f53aeffd640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T03:49:29.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.722+0000 7f53aeffd640 1 --2- 172.21.15.42:0/1070734833 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f53980039f0 0x7f5398003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:29.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.722+0000 7f53aeffd640 10 monclient(hunting): picked mon.a con 0x7f53980039f0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:49:29.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.722+0000 7f53aeffd640 1 --2- 172.21.15.42:0/1070734833 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5398004810 0x7f5398004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:29.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.722+0000 7f53aeffd640 10 monclient(hunting): picked mon.b con 0x7f5398004810 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:49:29.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.722+0000 7f53aeffd640 1 --2- 172.21.15.42:0/1070734833 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f53b00a1330 0x7f539800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:29.733 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.722+0000 7f53aeffd640 10 monclient(hunting): picked mon.c con 0x7f53b00a1330 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:49:29.733 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.722+0000 7f53aeffd640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:29.733 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.722+0000 7f53aeffd640 1 -- 172.21.15.42:0/1070734833 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f53b009edf0 con 0x7f53980039f0 2024-09-16T03:49:29.733 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.722+0000 7f53aeffd640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:29.733 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.722+0000 7f53aeffd640 1 -- 172.21.15.42:0/1070734833 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f5398005600 con 0x7f5398004810 2024-09-16T03:49:29.733 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.722+0000 7f53aeffd640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:29.733 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.722+0000 7f53aeffd640 1 -- 172.21.15.42:0/1070734833 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f539800e120 con 0x7f53b00a1330 2024-09-16T03:49:29.733 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.722+0000 7f53aeffd640 10 monclient(hunting): _renew_subs 2024-09-16T03:49:29.733 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.722+0000 7f53aeffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:49:29.733 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.722+0000 7f53b65d2640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-16T03:49:29.733 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.722+0000 7f53b4b6f640 1 -- 172.21.15.42:0/1070734833 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f53b00a1330 msgr2=0x7f539800b9d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T03:49:29.733 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.722+0000 7f53b4b6f640 1 --2- 172.21.15.42:0/1070734833 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f53b00a1330 0x7f539800b9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:49:29.733 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.722+0000 7f53affff640 1 --2- 172.21.15.42:0/1070734833 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f53980039f0 0x7f5398003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:29.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.722+0000 7f53affff640 10 monclient(hunting): get_auth_request con 0x7f53980039f0 auth_method 0 2024-09-16T03:49:29.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.722+0000 7f53affff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:29.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.722+0000 7f53affff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:29.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.722+0000 7f53affff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:49:29.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.722+0000 7f53affff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:29.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.722+0000 7f53affff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:29.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.722+0000 7f53affff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T03:49:29.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.722+0000 7f53af7fe640 1 --2- 172.21.15.42:0/1070734833 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5398004810 0x7f5398004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:29.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.722+0000 7f53af7fe640 10 monclient(hunting): get_auth_request con 0x7f5398004810 auth_method 0 2024-09-16T03:49:29.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.722+0000 7f53af7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:29.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.722+0000 7f53af7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:29.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.722+0000 7f53af7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:49:29.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.722+0000 7f53affff640 10 monclient(hunting): handle_auth_done global_id 764682 payload 293 2024-09-16T03:49:29.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.722+0000 7f53affff640 1 -- 172.21.15.42:0/1070734833 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f53b00a1330 msgr2=0x7f539800b9d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:49:29.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.722+0000 7f53affff640 1 --2- 172.21.15.42:0/1070734833 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f53b00a1330 0x7f539800b9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:29.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.722+0000 7f53affff640 1 -- 172.21.15.42:0/1070734833 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5398004810 msgr2=0x7f5398004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:49:29.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.722+0000 7f53affff640 1 --2- 172.21.15.42:0/1070734833 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5398004810 0x7f5398004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:29.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.722+0000 7f53affff640 10 monclient: _finish_hunting 0 2024-09-16T03:49:29.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.722+0000 7f53affff640 1 monclient: found mon.a 2024-09-16T03:49:29.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.722+0000 7f53affff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:49:29.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.722+0000 7f53affff640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:49:29.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.722+0000 7f53affff640 1 -- 172.21.15.42:0/1070734833 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5398003530 con 0x7f53980039f0 2024-09-16T03:49:29.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.722+0000 7f53affff640 10 monclient: _finish_auth 0 2024-09-16T03:49:29.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.722+0000 7f53affff640 10 monclient: _check_auth_tickets 2024-09-16T03:49:29.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.722+0000 7f53affff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:49:29.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.726+0000 7f53affff640 1 --2- 172.21.15.42:0/1070734833 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f53980039f0 0x7f5398003e10 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f53a0009e30 tx=0x7f53a001caf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4488167b7316cc6 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:29.742 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.726+0000 7f53aeffd640 1 -- 172.21.15.42:0/1070734833 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53a0009a30 con 0x7f53980039f0 2024-09-16T03:49:29.742 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.726+0000 7f53aeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:29.742 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.726+0000 7f53aeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:49:29.742 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.726+0000 7f53aeffd640 10 monclient: dump: 2024-09-16T03:49:29.742 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:49:29.742 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:49:29.742 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:29.742 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:29.742 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:29.742 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:49:29.742 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:49:29.742 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:49:29.743 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:49:29.743 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:49:29.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.726+0000 7f53b65d2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:49:29.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.726+0000 7f53aeffd640 1 -- 172.21.15.42:0/1070734833 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f53a00a9070 con 0x7f53980039f0 2024-09-16T03:49:29.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.726+0000 7f53aeffd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:49:29.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.726+0000 7f53aeffd640 1 -- 172.21.15.42:0/1070734833 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53a00a5b20 con 0x7f53980039f0 2024-09-16T03:49:29.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.726+0000 7f53b65d2640 10 monclient: get_monmap_and_config success 2024-09-16T03:49:29.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.726+0000 7f53b65d2640 4 set_mon_vals no callback set 2024-09-16T03:49:29.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.726+0000 7f53aeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:29.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.726+0000 7f53aeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:49:29.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.726+0000 7f53aeffd640 10 monclient: dump: 2024-09-16T03:49:29.743 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:49:29.743 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:49:29.744 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:29.744 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:29.744 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:29.744 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:49:29.744 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:49:29.744 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:49:29.744 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:49:29.744 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:49:29.744 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.726+0000 7f53b65d2640 10 monclient: shutdown 2024-09-16T03:49:29.744 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.726+0000 7f53b65d2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:49:29.744 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.726+0000 7f53b65d2640 1 -- 172.21.15.42:0/1070734833 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f53980039f0 msgr2=0x7f5398003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:29.744 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.726+0000 7f53b65d2640 1 --2- 172.21.15.42:0/1070734833 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f53980039f0 0x7f5398003e10 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f53a0009e30 tx=0x7f53a001caf0 comp rx=0 tx=0).stop 2024-09-16T03:49:29.744 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.726+0000 7f53b4b6f640 1 -- 172.21.15.42:0/1070734833 reap_dead start 2024-09-16T03:49:29.744 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.726+0000 7f53b65d2640 1 -- 172.21.15.42:0/1070734833 shutdown_connections 2024-09-16T03:49:29.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.726+0000 7f53b65d2640 1 -- 172.21.15.42:0/1070734833 >> 172.21.15.42:0/1070734833 conn(0x7f53b00a2420 msgr2=0x7f53b00a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:49:29.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.726+0000 7f53b65d2640 1 -- 172.21.15.42:0/1070734833 shutdown_connections 2024-09-16T03:49:29.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.726+0000 7f53b65d2640 1 -- 172.21.15.42:0/1070734833 wait complete. 2024-09-16T03:49:29.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.726+0000 7f53b65d2640 10 monclient: build_initial_monmap 2024-09-16T03:49:29.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.726+0000 7f53b65d2640 10 monclient: monmap: 2024-09-16T03:49:29.745 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:49:29.745 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:49:29.745 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:49:29.745 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:49:29.745 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:49:29.745 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:49:29.745 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:49:29.745 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:49:29.745 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:49:29.746 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:49:29.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.726+0000 7f53b65d2640 1 librados: starting msgr at 2024-09-16T03:49:29.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.726+0000 7f53b65d2640 1 librados: starting objecter 2024-09-16T03:49:29.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.726+0000 7f53b65d2640 1 Processor -- start 2024-09-16T03:49:29.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.726+0000 7f53b65d2640 1 -- start start 2024-09-16T03:49:29.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.726+0000 7f53b65d2640 1 librados: setting wanted keys 2024-09-16T03:49:29.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.726+0000 7f53b65d2640 1 librados: calling monclient init 2024-09-16T03:49:29.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.726+0000 7f53b65d2640 10 monclient: init 2024-09-16T03:49:29.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.726+0000 7f53b65d2640 10 monclient: _reopen_session rank -1 2024-09-16T03:49:29.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.726+0000 7f53b65d2640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:49:29.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.726+0000 7f53b65d2640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f53980039f0 0x7f53b0070e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:29.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.726+0000 7f53b65d2640 10 monclient(hunting): picked mon.noname-a con 0x7f53980039f0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:49:29.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.726+0000 7f53b65d2640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f53b0071370 0x7f53b00717f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:29.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.726+0000 7f53b65d2640 10 monclient(hunting): picked mon.noname-c con 0x7f53b0071370 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:49:29.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.726+0000 7f53b65d2640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f53b0071d30 0x7f53b01d1590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:29.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.726+0000 7f53b65d2640 10 monclient(hunting): picked mon.noname-b con 0x7f53b0071d30 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:49:29.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.726+0000 7f53b65d2640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:29.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.726+0000 7f53b65d2640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f53b009c5f0 con 0x7f53980039f0 2024-09-16T03:49:29.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.726+0000 7f53b65d2640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:29.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.726+0000 7f53b65d2640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f53b009c470 con 0x7f53b0071d30 2024-09-16T03:49:29.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.726+0000 7f53b65d2640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:29.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.726+0000 7f53b65d2640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f53b009c770 con 0x7f53b0071370 2024-09-16T03:49:29.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.726+0000 7f53b65d2640 10 monclient(hunting): _renew_subs 2024-09-16T03:49:29.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.726+0000 7f53b65d2640 10 monclient(hunting): authenticate will time out at 1416.699055s 2024-09-16T03:49:29.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.726+0000 7f53af7fe640 1 -- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f53b0071370 msgr2=0x7f53b00717f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T03:49:29.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.726+0000 7f53af7fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f53b0071370 0x7f53b00717f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:49:29.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.726+0000 7f53affff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f53980039f0 0x7f53b0070e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:29.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.726+0000 7f53affff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f53980039f0 0x7f53b0070e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:59242/0 (socket says 172.21.15.42:59242) 2024-09-16T03:49:29.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.726+0000 7f53affff640 1 -- 172.21.15.42:0/3691105147 learned_addr learned my addr 172.21.15.42:0/3691105147 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:49:29.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.726+0000 7f53affff640 10 monclient(hunting): get_auth_request con 0x7f53980039f0 auth_method 0 2024-09-16T03:49:29.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.726+0000 7f53affff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:29.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.726+0000 7f53affff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:29.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.726+0000 7f53affff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:29.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.726+0000 7f53affff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:29.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.726+0000 7f53affff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:29.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.726+0000 7f53affff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:29.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.726+0000 7f53b4b6f640 1 --2- 172.21.15.42:0/3691105147 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f53b0071d30 0x7f53b01d1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:29.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.726+0000 7f53affff640 10 monclient(hunting): handle_auth_done global_id 764685 payload 824 2024-09-16T03:49:29.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.726+0000 7f53affff640 1 -- 172.21.15.42:0/3691105147 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f53b0071370 msgr2=0x7f53b00717f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:49:29.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.726+0000 7f53affff640 1 --2- 172.21.15.42:0/3691105147 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f53b0071370 0x7f53b00717f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:29.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.726+0000 7f53affff640 1 -- 172.21.15.42:0/3691105147 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f53b0071d30 msgr2=0x7f53b01d1590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:29.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.726+0000 7f53affff640 1 --2- 172.21.15.42:0/3691105147 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f53b0071d30 0x7f53b01d1590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:29.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.726+0000 7f53affff640 10 monclient: _finish_hunting 0 2024-09-16T03:49:29.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.726+0000 7f53affff640 1 monclient: found mon.noname-a 2024-09-16T03:49:29.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.726+0000 7f53affff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:49:29.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.726+0000 7f53affff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:49:29.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.726+0000 7f53affff640 1 -- 172.21.15.42:0/3691105147 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53b00a3c80 con 0x7f53980039f0 2024-09-16T03:49:29.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.726+0000 7f53b4b6f640 10 monclient: get_auth_request con 0x7f53b0071d30 auth_method 0 2024-09-16T03:49:29.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.726+0000 7f53b4b6f640 1 --2- 172.21.15.42:0/3691105147 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f53b0071d30 0x7f53b01d1590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T03:49:29.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.726+0000 7f53affff640 1 --2- 172.21.15.42:0/3691105147 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f53980039f0 0x7f53b0070e30 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f53a001cb30 tx=0x7f5398005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:29.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.726+0000 7f53acff9640 10 client.?.objecter ms_handle_connect 0x7f53980039f0 2024-09-16T03:49:29.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.726+0000 7f53acff9640 10 client.?.objecter resend_mon_ops 2024-09-16T03:49:29.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.726+0000 7f53acff9640 1 -- 172.21.15.42:0/3691105147 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53a0009a30 con 0x7f53980039f0 2024-09-16T03:49:29.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.726+0000 7f53acff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:29.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.726+0000 7f53acff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:49:29.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.726+0000 7f53acff9640 10 monclient: dump: 2024-09-16T03:49:29.750 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:49:29.750 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:49:29.750 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:29.750 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:29.750 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:29.750 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:49:29.750 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:49:29.750 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:49:29.750 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:49:29.751 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:49:29.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.726+0000 7f53acff9640 10 monclient: _finish_auth 0 2024-09-16T03:49:29.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.726+0000 7f53acff9640 10 monclient: _check_auth_tickets 2024-09-16T03:49:29.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.726+0000 7f53acff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:49:29.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.726+0000 7f53acff9640 1 -- 172.21.15.42:0/3691105147 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f53a00ae670 con 0x7f53980039f0 2024-09-16T03:49:29.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.726+0000 7f53b65d2640 5 monclient: authenticate success, global_id 764685 2024-09-16T03:49:29.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.726+0000 7f53b65d2640 10 monclient: _renew_subs 2024-09-16T03:49:29.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.726+0000 7f53b65d2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:49:29.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.726+0000 7f53b65d2640 1 -- 172.21.15.42:0/3691105147 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53b01320d0 con 0x7f53980039f0 2024-09-16T03:49:29.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.730+0000 7f53acff9640 10 monclient: handle_config config(0 keys) 2024-09-16T03:49:29.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.730+0000 7f53acff9640 1 -- 172.21.15.42:0/3691105147 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53a00aea40 con 0x7f53980039f0 2024-09-16T03:49:29.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.730+0000 7f53acff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:29.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.730+0000 7f53acff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:49:29.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.730+0000 7f53acff9640 10 monclient: dump: 2024-09-16T03:49:29.752 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:49:29.752 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:49:29.752 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:29.752 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:29.752 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:29.752 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:49:29.752 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:49:29.752 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:49:29.752 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:49:29.752 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:49:29.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.730+0000 7f53b65d2640 10 client.764685.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:49:29.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.730+0000 7f53b65d2640 10 monclient: _renew_subs 2024-09-16T03:49:29.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.730+0000 7f53b65d2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:49:29.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.730+0000 7f53b65d2640 1 -- 172.21.15.42:0/3691105147 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53b00721b0 con 0x7f53980039f0 2024-09-16T03:49:29.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.730+0000 7f53b65d2640 1 librados: init done 2024-09-16T03:49:29.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.730+0000 7f53ae7fc640 4 set_mon_vals no callback set 2024-09-16T03:49:29.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.730+0000 7f53b65d2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:49:29.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.730+0000 7f53b65d2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:49:29.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.730+0000 7f53b65d2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:49:29.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.730+0000 7f53b65d2640 1 -- 172.21.15.42:0/3691105147 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f53b00a1b10 con 0x7f53980039f0 2024-09-16T03:49:29.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.730+0000 7f53acff9640 1 -- 172.21.15.42:0/3691105147 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f53a00b83d0 con 0x7f53980039f0 2024-09-16T03:49:29.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.730+0000 7f53acff9640 1 --2- 172.21.15.42:0/3691105147 >> v2:172.21.15.154:6812/254845342 conn(0x7f539c0377c0 0x7f539c039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:29.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.730+0000 7f53acff9640 1 -- 172.21.15.42:0/3691105147 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(247..247 src has 1..247) ==== 6129+0+0 (secure 0 0 0) 0x7f53a00e2ce0 con 0x7f53980039f0 2024-09-16T03:49:29.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.730+0000 7f53acff9640 10 client.764685.objecter ms_dispatch 0x7f53b0000ba0 osd_map(247..247 src has 1..247) 2024-09-16T03:49:29.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.730+0000 7f53acff9640 3 client.764685.objecter handle_osd_map got epochs [247,247] > 0 2024-09-16T03:49:29.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.730+0000 7f53acff9640 3 client.764685.objecter handle_osd_map decoding full epoch 247 2024-09-16T03:49:29.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.730+0000 7f53acff9640 20 client.764685.objecter dump_active .. 0 homeless 2024-09-16T03:49:29.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.730+0000 7f53af7fe640 1 --2- 172.21.15.42:0/3691105147 >> v2:172.21.15.154:6812/254845342 conn(0x7f539c0377c0 0x7f539c039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:29.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.730+0000 7f53af7fe640 10 monclient: get_auth_request con 0x7f539c0377c0 auth_method 0 2024-09-16T03:49:29.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.730+0000 7f53acff9640 10 client.764685.objecter ms_handle_connect 0x7f539c0377c0 2024-09-16T03:49:29.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.730+0000 7f53af7fe640 1 --2- 172.21.15.42:0/3691105147 >> v2:172.21.15.154:6812/254845342 conn(0x7f539c0377c0 0x7f539c039c80 secure :-1 s=READY pgs=3519 cs=0 l=1 rev1=1 crypto rx=0x7f53a4001ec0 tx=0x7f53a4001160 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:29.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.734+0000 7f53acff9640 1 -- 172.21.15.42:0/3691105147 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f53b00a3c80 con 0x7f53980039f0 2024-09-16T03:49:29.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.734+0000 7f53acff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:49:29.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.734+0000 7f53acff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:49:29.962 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.958+0000 7f53b65d2640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:49:29.962 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.958+0000 7f53b65d2640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:49:29.963 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.958+0000 7f53b65d2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:49:29.963 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.958+0000 7f53b65d2640 1 -- 172.21.15.42:0/3691105147 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f53b011c0a0 con 0x7f53980039f0 2024-09-16T03:49:29.963 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.958+0000 7f53acff9640 1 -- 172.21.15.42:0/3691105147 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7f53a0016ac0 con 0x7f53980039f0 2024-09-16T03:49:29.963 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.958+0000 7f53acff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:49:29.963 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.958+0000 7f53acff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:49:29.963 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":66,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:49:29.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.962+0000 7f53b65d2640 10 librados: watch_flush enter 2024-09-16T03:49:29.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.962+0000 7f53b65d2640 10 librados: watch_flush exit 2024-09-16T03:49:29.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.962+0000 7f53b65d2640 20 client.764685.objecter shutdown clearing up homeless session... 2024-09-16T03:49:29.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.962+0000 7f53b65d2640 10 client.764685.objecter successfully canceled tick 2024-09-16T03:49:29.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.962+0000 7f53b65d2640 1 -- 172.21.15.42:0/3691105147 >> v2:172.21.15.154:6812/254845342 conn(0x7f539c0377c0 msgr2=0x7f539c039c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:29.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.962+0000 7f53b65d2640 1 --2- 172.21.15.42:0/3691105147 >> v2:172.21.15.154:6812/254845342 conn(0x7f539c0377c0 0x7f539c039c80 secure :-1 s=READY pgs=3519 cs=0 l=1 rev1=1 crypto rx=0x7f53a4001ec0 tx=0x7f53a4001160 comp rx=0 tx=0).stop 2024-09-16T03:49:29.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.962+0000 7f53b65d2640 10 monclient: shutdown 2024-09-16T03:49:29.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.962+0000 7f53b65d2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:49:29.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.962+0000 7f53b65d2640 1 -- 172.21.15.42:0/3691105147 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f53980039f0 msgr2=0x7f53b0070e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:29.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.962+0000 7f53b65d2640 1 --2- 172.21.15.42:0/3691105147 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f53980039f0 0x7f53b0070e30 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f53a001cb30 tx=0x7f5398005840 comp rx=0 tx=0).stop 2024-09-16T03:49:29.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.962+0000 7f53b65d2640 1 -- 172.21.15.42:0/3691105147 shutdown_connections 2024-09-16T03:49:29.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.962+0000 7f53b65d2640 1 --2- 172.21.15.42:0/3691105147 >> v2:172.21.15.154:6812/254845342 conn(0x7f539c0377c0 0x7f539c039c80 unknown :-1 s=CLOSED pgs=3519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:29.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.962+0000 7f53b65d2640 1 --2- 172.21.15.42:0/3691105147 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f53b0071d30 0x7f53b01d1590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:29.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.962+0000 7f53b65d2640 1 --2- 172.21.15.42:0/3691105147 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f53b0071370 0x7f53b00717f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:29.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.962+0000 7f53b65d2640 1 --2- 172.21.15.42:0/3691105147 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f53980039f0 0x7f53b0070e30 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:29.967 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.962+0000 7f53b65d2640 1 -- 172.21.15.42:0/3691105147 >> 172.21.15.42:0/3691105147 conn(0x7f53b00a2420 msgr2=0x7f53b01315f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:49:29.967 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.962+0000 7f53b65d2640 1 -- 172.21.15.42:0/3691105147 shutdown_connections 2024-09-16T03:49:29.967 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.962+0000 7f53b65d2640 1 -- 172.21.15.42:0/3691105147 wait complete. 2024-09-16T03:49:29.967 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:29.962+0000 7f53b65d2640 1 librados: shutdown 2024-09-16T03:49:32.984 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:49:33.118 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.114+0000 7fd701e79640 10 monclient: get_monmap_and_config 2024-09-16T03:49:33.118 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.114+0000 7fd701e79640 10 monclient: build_initial_monmap 2024-09-16T03:49:33.118 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.114+0000 7fd701e79640 10 monclient: monmap: 2024-09-16T03:49:33.118 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:49:33.119 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:49:33.119 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:49:33.117437+0000 2024-09-16T03:49:33.119 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:49:33.117437+0000 2024-09-16T03:49:33.119 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:49:33.119 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:49:33.119 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:49:33.119 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:49:33.119 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:49:33.119 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:49:33.119 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:49:33.119 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:49:33.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.114+0000 7fd701e79640 1 Processor -- start 2024-09-16T03:49:33.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.114+0000 7fd701e79640 1 -- start start 2024-09-16T03:49:33.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.114+0000 7fd701e79640 10 monclient: init 2024-09-16T03:49:33.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.114+0000 7fd701e79640 10 monclient: _reopen_session rank -1 2024-09-16T03:49:33.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.114+0000 7fd701e79640 10 monclient: _add_conns ranks=[1,2,4,3,0] 2024-09-16T03:49:33.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.114+0000 7fd701e79640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd6fc09fb00 addr v1:172.21.15.154:6789/0 2024-09-16T03:49:33.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.114+0000 7fd701e79640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd6fc09c160 0x7fd6fc139120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:33.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.114+0000 7fd701e79640 10 monclient(hunting): picked mon.noname-b con 0x7fd6fc09c160 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:49:33.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.114+0000 7fd701e79640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fd6fc09c5a0 0x7fd6fc13b670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:33.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.114+0000 7fd701e79640 10 monclient(hunting): picked mon.noname-c con 0x7fd6fc09c5a0 addr v2:172.21.15.154:3300/0 2024-09-16T03:49:33.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.114+0000 7fd701e79640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd6fc0a1e20 con 0x7fd6fc09fb00 2024-09-16T03:49:33.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.114+0000 7fd701e79640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:33.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.114+0000 7fd701e79640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fd6fc09e250 con 0x7fd6fc09c5a0 2024-09-16T03:49:33.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.114+0000 7fd701e79640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:33.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.114+0000 7fd701e79640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fd6fc09ea30 con 0x7fd6fc09c160 2024-09-16T03:49:33.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.114+0000 7fd701e79640 10 monclient(hunting): _renew_subs 2024-09-16T03:49:33.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.114+0000 7fd701e79640 10 monclient(hunting): authenticate will time out at 1420.087011s 2024-09-16T03:49:33.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.114+0000 7fd6faffd640 1 -- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd6fc09c160 msgr2=0x7fd6fc139120 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T03:49:33.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.114+0000 7fd6faffd640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd6fc09c160 0x7fd6fc139120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:49:33.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.114+0000 7fd6fbfff640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fd6fc09c5a0 0x7fd6fc13b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:33.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.114+0000 7fd6fb7fe640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7fd6fc09fb00 0x7fd6fc09ba50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.42:37068/0 (socket says 172.21.15.42:37068) 2024-09-16T03:49:33.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.114+0000 7fd6fb7fe640 1 -- 172.21.15.42:0/3796432341 learned_addr learned my addr 172.21.15.42:0/3796432341 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:49:33.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.114+0000 7fd6fbfff640 10 monclient(hunting): get_auth_request con 0x7fd6fc09c5a0 auth_method 0 2024-09-16T03:49:33.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.114+0000 7fd6fbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:33.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.114+0000 7fd6fbfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:33.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.114+0000 7fd6fbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:33.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.114+0000 7fd6fbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:33.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.114+0000 7fd6fbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:33.125 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.114+0000 7fd6fbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:33.125 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.114+0000 7fd6fbfff640 10 monclient(hunting): handle_auth_done global_id 794143 payload 274 2024-09-16T03:49:33.125 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.114+0000 7fd6fbfff640 1 -- 172.21.15.42:0/3796432341 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd6fc09c160 msgr2=0x7fd6fc139120 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:49:33.125 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.114+0000 7fd6fbfff640 1 --2- 172.21.15.42:0/3796432341 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd6fc09c160 0x7fd6fc139120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:33.125 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.114+0000 7fd6fbfff640 1 -- 172.21.15.42:0/3796432341 >> v1:172.21.15.154:6789/0 conn(0x7fd6fc09fb00 legacy=0x7fd6fc09ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:33.125 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.118+0000 7fd6fbfff640 10 monclient: _finish_hunting 0 2024-09-16T03:49:33.125 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.118+0000 7fd6fbfff640 1 monclient: found mon.noname-c 2024-09-16T03:49:33.125 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.118+0000 7fd6fbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:49:33.125 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.118+0000 7fd6fbfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.154:3300/0 2024-09-16T03:49:33.125 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.118+0000 7fd6fbfff640 1 -- 172.21.15.42:0/3796432341 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6fc09fee0 con 0x7fd6fc09c5a0 2024-09-16T03:49:33.131 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.118+0000 7fd6fbfff640 1 --2- 172.21.15.42:0/3796432341 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd6fc09c5a0 0x7fd6fc13b670 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fd6f4009d90 tx=0x7fd6f401c850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ea3913fa69bcb84d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:33.132 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.118+0000 7fd6fa7fc640 1 -- 172.21.15.42:0/3796432341 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6f40b2070 con 0x7fd6fc09c5a0 2024-09-16T03:49:33.132 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.118+0000 7fd6fa7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:33.132 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.118+0000 7fd6fa7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:49:33.132 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.118+0000 7fd6fa7fc640 10 monclient: dump: 2024-09-16T03:49:33.132 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:49:33.132 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:49:33.132 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:33.132 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:33.132 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:33.132 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:49:33.132 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:49:33.132 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:49:33.133 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:49:33.133 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:49:33.133 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.118+0000 7fd6fa7fc640 10 monclient: _finish_auth 0 2024-09-16T03:49:33.133 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.118+0000 7fd6fa7fc640 10 monclient: _check_auth_tickets 2024-09-16T03:49:33.133 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.118+0000 7fd6fa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:49:33.133 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.118+0000 7fd6fa7fc640 1 -- 172.21.15.42:0/3796432341 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd6f40a5910 con 0x7fd6fc09c5a0 2024-09-16T03:49:33.133 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.118+0000 7fd6fa7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:49:33.133 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.118+0000 7fd6fa7fc640 1 -- 172.21.15.42:0/3796432341 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6f40a5ce0 con 0x7fd6fc09c5a0 2024-09-16T03:49:33.133 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.118+0000 7fd6fa7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:33.133 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.118+0000 7fd6fa7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:49:33.133 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.118+0000 7fd6fa7fc640 10 monclient: dump: 2024-09-16T03:49:33.133 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:49:33.133 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:49:33.134 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:33.134 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:33.134 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:33.134 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:49:33.134 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:49:33.134 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:49:33.134 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:49:33.134 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:49:33.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.118+0000 7fd701e79640 5 monclient: authenticate success, global_id 794143 2024-09-16T03:49:33.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.118+0000 7fd701e79640 10 monclient: get_monmap_and_config success 2024-09-16T03:49:33.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.118+0000 7fd701e79640 4 set_mon_vals no callback set 2024-09-16T03:49:33.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.122+0000 7fd701e79640 10 monclient: shutdown 2024-09-16T03:49:33.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.122+0000 7fd701e79640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:49:33.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.122+0000 7fd701e79640 1 -- 172.21.15.42:0/3796432341 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd6fc09c5a0 msgr2=0x7fd6fc13b670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:33.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.122+0000 7fd701e79640 1 --2- 172.21.15.42:0/3796432341 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd6fc09c5a0 0x7fd6fc13b670 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fd6f4009d90 tx=0x7fd6f401c850 comp rx=0 tx=0).stop 2024-09-16T03:49:33.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.122+0000 7fd701e79640 1 -- 172.21.15.42:0/3796432341 shutdown_connections 2024-09-16T03:49:33.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.122+0000 7fd701e79640 1 --2- 172.21.15.42:0/3796432341 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd6fc09c5a0 0x7fd6fc13b670 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:33.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.122+0000 7fd701e79640 1 --2- 172.21.15.42:0/3796432341 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd6fc09c160 0x7fd6fc139120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:33.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.122+0000 7fd701e79640 1 -- 172.21.15.42:0/3796432341 >> 172.21.15.42:0/3796432341 conn(0x7fd6fc0a2420 msgr2=0x7fd6fc0a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:49:33.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.122+0000 7fd701e79640 1 -- 172.21.15.42:0/3796432341 shutdown_connections 2024-09-16T03:49:33.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.122+0000 7fd701e79640 1 -- 172.21.15.42:0/3796432341 wait complete. 2024-09-16T03:49:33.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.122+0000 7fd701e79640 10 monclient: build_initial_monmap 2024-09-16T03:49:33.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.122+0000 7fd701e79640 10 monclient: monmap: 2024-09-16T03:49:33.136 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:49:33.136 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:49:33.136 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:49:33.136 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:49:33.136 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:49:33.136 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:49:33.136 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:49:33.136 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:49:33.136 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:49:33.136 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:49:33.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.122+0000 7fd701e79640 1 librados: starting msgr at 2024-09-16T03:49:33.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.122+0000 7fd701e79640 1 librados: starting objecter 2024-09-16T03:49:33.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.122+0000 7fd701e79640 1 Processor -- start 2024-09-16T03:49:33.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.122+0000 7fd701e79640 1 -- start start 2024-09-16T03:49:33.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.122+0000 7fd701e79640 1 librados: setting wanted keys 2024-09-16T03:49:33.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.122+0000 7fd701e79640 1 librados: calling monclient init 2024-09-16T03:49:33.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.122+0000 7fd701e79640 10 monclient: init 2024-09-16T03:49:33.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.122+0000 7fd701e79640 10 monclient: _reopen_session rank -1 2024-09-16T03:49:33.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.122+0000 7fd701e79640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T03:49:33.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.122+0000 7fd701e79640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd6fc09c160 0x7fd6fc06db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:33.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.122+0000 7fd701e79640 10 monclient(hunting): picked mon.noname-a con 0x7fd6fc09c160 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:49:33.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.122+0000 7fd701e79640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd6fc09c5a0 0x7fd6fc06e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:33.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.122+0000 7fd701e79640 10 monclient(hunting): picked mon.noname-b con 0x7fd6fc09c5a0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:49:33.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.122+0000 7fd701e79640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd6fc09fb00 0x7fd6fc06e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:33.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.122+0000 7fd701e79640 10 monclient(hunting): picked mon.noname-c con 0x7fd6fc09fb00 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:49:33.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.122+0000 7fd701e79640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:33.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.122+0000 7fd701e79640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fd6fc09d380 con 0x7fd6fc09c160 2024-09-16T03:49:33.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.122+0000 7fd701e79640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:33.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.122+0000 7fd701e79640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd6fc09d140 con 0x7fd6fc09c5a0 2024-09-16T03:49:33.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.122+0000 7fd701e79640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:33.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.122+0000 7fd701e79640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fd6fc09d500 con 0x7fd6fc09fb00 2024-09-16T03:49:33.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.122+0000 7fd701e79640 10 monclient(hunting): _renew_subs 2024-09-16T03:49:33.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.122+0000 7fd701e79640 10 monclient(hunting): authenticate will time out at 1420.093806s 2024-09-16T03:49:33.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.122+0000 7fd6fbfff640 1 -- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd6fc09fb00 msgr2=0x7fd6fc06e610 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T03:49:33.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.122+0000 7fd6fbfff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd6fc09fb00 0x7fd6fc06e610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:49:33.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.122+0000 7fd6fb7fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd6fc09c160 0x7fd6fc06db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:33.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.122+0000 7fd6fb7fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd6fc09c160 0x7fd6fc06db90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:59252/0 (socket says 172.21.15.42:59252) 2024-09-16T03:49:33.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.122+0000 7fd6fb7fe640 1 -- 172.21.15.42:0/2508489998 learned_addr learned my addr 172.21.15.42:0/2508489998 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:49:33.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.122+0000 7fd6fb7fe640 10 monclient(hunting): get_auth_request con 0x7fd6fc09c160 auth_method 0 2024-09-16T03:49:33.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.122+0000 7fd6fb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:33.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.122+0000 7fd6fb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:33.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.122+0000 7fd6fb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:33.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.122+0000 7fd6fb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:33.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.122+0000 7fd6fb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:33.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.122+0000 7fd6fb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:33.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.122+0000 7fd6faffd640 1 --2- 172.21.15.42:0/2508489998 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd6fc09c5a0 0x7fd6fc06e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:33.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.122+0000 7fd6faffd640 10 monclient(hunting): get_auth_request con 0x7fd6fc09c5a0 auth_method 0 2024-09-16T03:49:33.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.122+0000 7fd6faffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:33.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.122+0000 7fd6faffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:33.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.122+0000 7fd6faffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:33.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.122+0000 7fd6fb7fe640 10 monclient(hunting): handle_auth_done global_id 764688 payload 824 2024-09-16T03:49:33.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.122+0000 7fd6fb7fe640 1 -- 172.21.15.42:0/2508489998 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd6fc09fb00 msgr2=0x7fd6fc06e610 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:49:33.140 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.122+0000 7fd6fb7fe640 1 --2- 172.21.15.42:0/2508489998 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd6fc09fb00 0x7fd6fc06e610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:33.140 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.122+0000 7fd6fb7fe640 1 -- 172.21.15.42:0/2508489998 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd6fc09c5a0 msgr2=0x7fd6fc06e0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:33.140 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.122+0000 7fd6fb7fe640 1 --2- 172.21.15.42:0/2508489998 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd6fc09c5a0 0x7fd6fc06e0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:33.140 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.122+0000 7fd6fb7fe640 10 monclient: _finish_hunting 0 2024-09-16T03:49:33.140 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.122+0000 7fd6fb7fe640 1 monclient: found mon.noname-a 2024-09-16T03:49:33.140 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.122+0000 7fd6fb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:49:33.140 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.122+0000 7fd6fb7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:49:33.140 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.122+0000 7fd6fb7fe640 1 -- 172.21.15.42:0/2508489998 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6fc126410 con 0x7fd6fc09c160 2024-09-16T03:49:33.140 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.122+0000 7fd6fb7fe640 1 --2- 172.21.15.42:0/2508489998 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd6fc09c160 0x7fd6fc06db90 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fd6ec00c500 tx=0x7fd6ec00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:33.140 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.122+0000 7fd6e3fff640 10 client.?.objecter ms_handle_connect 0x7fd6fc09c160 2024-09-16T03:49:33.140 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.122+0000 7fd6e3fff640 10 client.?.objecter resend_mon_ops 2024-09-16T03:49:33.140 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.122+0000 7fd6e3fff640 1 -- 172.21.15.42:0/2508489998 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6ec017070 con 0x7fd6fc09c160 2024-09-16T03:49:33.140 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.122+0000 7fd6e3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:33.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.122+0000 7fd6e3fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:49:33.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.122+0000 7fd6e3fff640 10 monclient: dump: 2024-09-16T03:49:33.141 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:49:33.141 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:49:33.141 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:33.141 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:33.141 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:33.141 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:49:33.141 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:49:33.141 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:49:33.141 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:49:33.141 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:49:33.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.122+0000 7fd6e3fff640 10 monclient: _finish_auth 0 2024-09-16T03:49:33.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.122+0000 7fd6e3fff640 10 monclient: _check_auth_tickets 2024-09-16T03:49:33.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.122+0000 7fd6e3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:49:33.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.122+0000 7fd701e79640 5 monclient: authenticate success, global_id 764688 2024-09-16T03:49:33.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.122+0000 7fd701e79640 10 monclient: _renew_subs 2024-09-16T03:49:33.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.122+0000 7fd701e79640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:49:33.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.122+0000 7fd701e79640 1 -- 172.21.15.42:0/2508489998 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6fc0a53f0 con 0x7fd6fc09c160 2024-09-16T03:49:33.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.122+0000 7fd701e79640 10 client.764688.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:49:33.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.122+0000 7fd6e3fff640 1 -- 172.21.15.42:0/2508489998 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd6ec005cf0 con 0x7fd6fc09c160 2024-09-16T03:49:33.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.122+0000 7fd701e79640 10 monclient: _renew_subs 2024-09-16T03:49:33.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.122+0000 7fd701e79640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:49:33.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.122+0000 7fd701e79640 1 -- 172.21.15.42:0/2508489998 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6fc126130 con 0x7fd6fc09c160 2024-09-16T03:49:33.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.122+0000 7fd701e79640 1 librados: init done 2024-09-16T03:49:33.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.122+0000 7fd6e3fff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:49:33.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.122+0000 7fd6e3fff640 1 -- 172.21.15.42:0/2508489998 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6ec0072f0 con 0x7fd6fc09c160 2024-09-16T03:49:33.143 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.122+0000 7fd6e3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:33.143 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.122+0000 7fd6e3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:49:33.143 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.122+0000 7fd6e3fff640 10 monclient: dump: 2024-09-16T03:49:33.143 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:49:33.143 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:49:33.143 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:33.143 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:33.143 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:33.143 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:49:33.143 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:49:33.143 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:49:33.143 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:49:33.143 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:49:33.143 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.122+0000 7fd6f9ffb640 4 set_mon_vals no callback set 2024-09-16T03:49:33.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.122+0000 7fd701e79640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:49:33.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.122+0000 7fd701e79640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:49:33.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.122+0000 7fd701e79640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:49:33.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.122+0000 7fd701e79640 1 -- 172.21.15.42:0/2508489998 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd6fc124cd0 con 0x7fd6fc09c160 2024-09-16T03:49:33.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.122+0000 7fd6e3fff640 1 -- 172.21.15.42:0/2508489998 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fd6ec01a070 con 0x7fd6fc09c160 2024-09-16T03:49:33.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.126+0000 7fd6e3fff640 1 --2- 172.21.15.42:0/2508489998 >> v2:172.21.15.154:6812/254845342 conn(0x7fd6e8037810 0x7fd6e8039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:33.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.126+0000 7fd6e3fff640 1 -- 172.21.15.42:0/2508489998 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(247..247 src has 1..247) ==== 6129+0+0 (secure 0 0 0) 0x7fd6ec048430 con 0x7fd6fc09c160 2024-09-16T03:49:33.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.126+0000 7fd6e3fff640 10 client.764688.objecter ms_dispatch 0x7fd6fc000ba0 osd_map(247..247 src has 1..247) 2024-09-16T03:49:33.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.126+0000 7fd6e3fff640 3 client.764688.objecter handle_osd_map got epochs [247,247] > 0 2024-09-16T03:49:33.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.126+0000 7fd6e3fff640 3 client.764688.objecter handle_osd_map decoding full epoch 247 2024-09-16T03:49:33.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.126+0000 7fd6e3fff640 20 client.764688.objecter dump_active .. 0 homeless 2024-09-16T03:49:33.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.126+0000 7fd6faffd640 1 --2- 172.21.15.42:0/2508489998 >> v2:172.21.15.154:6812/254845342 conn(0x7fd6e8037810 0x7fd6e8039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:33.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.126+0000 7fd6faffd640 10 monclient: get_auth_request con 0x7fd6e8037810 auth_method 0 2024-09-16T03:49:33.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.126+0000 7fd6faffd640 1 --2- 172.21.15.42:0/2508489998 >> v2:172.21.15.154:6812/254845342 conn(0x7fd6e8037810 0x7fd6e8039cd0 secure :-1 s=READY pgs=3520 cs=0 l=1 rev1=1 crypto rx=0x7fd6f0006fd0 tx=0x7fd6f0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:33.145 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.126+0000 7fd6e3fff640 10 client.764688.objecter ms_handle_connect 0x7fd6e8037810 2024-09-16T03:49:33.145 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.130+0000 7fd6e3fff640 1 -- 172.21.15.42:0/2508489998 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7fd6fc124cd0 con 0x7fd6fc09c160 2024-09-16T03:49:33.145 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.130+0000 7fd6e3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:49:33.145 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.130+0000 7fd6e3fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:49:33.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.354+0000 7fd701e79640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:49:33.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.354+0000 7fd701e79640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:49:33.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.354+0000 7fd701e79640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:49:33.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.354+0000 7fd701e79640 1 -- 172.21.15.42:0/2508489998 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fd6fc069040 con 0x7fd6fc09c160 2024-09-16T03:49:33.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.354+0000 7fd6e3fff640 1 -- 172.21.15.42:0/2508489998 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7fd6ec01ac60 con 0x7fd6fc09c160 2024-09-16T03:49:33.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.354+0000 7fd6e3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:49:33.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.354+0000 7fd6e3fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:49:33.360 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":66,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:49:33.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.358+0000 7fd701e79640 10 librados: watch_flush enter 2024-09-16T03:49:33.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.358+0000 7fd701e79640 10 librados: watch_flush exit 2024-09-16T03:49:33.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.358+0000 7fd701e79640 20 client.764688.objecter shutdown clearing up homeless session... 2024-09-16T03:49:33.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.358+0000 7fd701e79640 10 client.764688.objecter successfully canceled tick 2024-09-16T03:49:33.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.358+0000 7fd701e79640 1 -- 172.21.15.42:0/2508489998 >> v2:172.21.15.154:6812/254845342 conn(0x7fd6e8037810 msgr2=0x7fd6e8039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:33.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.358+0000 7fd701e79640 1 --2- 172.21.15.42:0/2508489998 >> v2:172.21.15.154:6812/254845342 conn(0x7fd6e8037810 0x7fd6e8039cd0 secure :-1 s=READY pgs=3520 cs=0 l=1 rev1=1 crypto rx=0x7fd6f0006fd0 tx=0x7fd6f0008040 comp rx=0 tx=0).stop 2024-09-16T03:49:33.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.358+0000 7fd701e79640 10 monclient: shutdown 2024-09-16T03:49:33.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.358+0000 7fd701e79640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:49:33.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.358+0000 7fd701e79640 1 -- 172.21.15.42:0/2508489998 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd6fc09c160 msgr2=0x7fd6fc06db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:33.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.358+0000 7fd701e79640 1 --2- 172.21.15.42:0/2508489998 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd6fc09c160 0x7fd6fc06db90 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fd6ec00c500 tx=0x7fd6ec00c9c0 comp rx=0 tx=0).stop 2024-09-16T03:49:33.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.358+0000 7fd701e79640 1 -- 172.21.15.42:0/2508489998 shutdown_connections 2024-09-16T03:49:33.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.358+0000 7fd701e79640 1 --2- 172.21.15.42:0/2508489998 >> v2:172.21.15.154:6812/254845342 conn(0x7fd6e8037810 0x7fd6e8039cd0 unknown :-1 s=CLOSED pgs=3520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:33.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.358+0000 7fd701e79640 1 --2- 172.21.15.42:0/2508489998 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd6fc09fb00 0x7fd6fc06e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:33.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.358+0000 7fd701e79640 1 --2- 172.21.15.42:0/2508489998 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd6fc09c5a0 0x7fd6fc06e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:33.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.358+0000 7fd701e79640 1 --2- 172.21.15.42:0/2508489998 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd6fc09c160 0x7fd6fc06db90 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:33.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.358+0000 7fd701e79640 1 -- 172.21.15.42:0/2508489998 >> 172.21.15.42:0/2508489998 conn(0x7fd6fc0a2420 msgr2=0x7fd6fc0a0a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:49:33.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.358+0000 7fd701e79640 1 -- 172.21.15.42:0/2508489998 shutdown_connections 2024-09-16T03:49:33.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.358+0000 7fd701e79640 1 -- 172.21.15.42:0/2508489998 wait complete. 2024-09-16T03:49:33.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:33.358+0000 7fd701e79640 1 librados: shutdown 2024-09-16T03:49:36.380 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:49:36.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:36.506+0000 7f7a9d6f4640 10 monclient: get_monmap_and_config 2024-09-16T03:49:36.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:36.506+0000 7f7a9d6f4640 10 monclient: build_initial_monmap 2024-09-16T03:49:36.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:36.506+0000 7f7a9d6f4640 10 monclient: monmap: 2024-09-16T03:49:36.512 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:49:36.512 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:49:36.512 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:49:36.511313+0000 2024-09-16T03:49:36.513 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:49:36.511313+0000 2024-09-16T03:49:36.513 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:49:36.513 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:49:36.513 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:49:36.513 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:49:36.513 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:49:36.513 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:49:36.513 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:49:36.513 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:49:36.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:36.506+0000 7f7a9d6f4640 1 Processor -- start 2024-09-16T03:49:36.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:36.510+0000 7f7a9d6f4640 1 -- start start 2024-09-16T03:49:36.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:36.510+0000 7f7a9d6f4640 10 monclient: init 2024-09-16T03:49:36.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:36.510+0000 7f7a9d6f4640 10 monclient: _reopen_session rank -1 2024-09-16T03:49:36.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:36.510+0000 7f7a9d6f4640 10 monclient: _add_conns ranks=[4,2,0,3,1] 2024-09-16T03:49:36.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:36.510+0000 7f7a9d6f4640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f7a980a1330 0x7f7a980a1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:36.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:36.510+0000 7f7a9d6f4640 10 monclient(hunting): picked mon.noname-c con 0x7f7a980a1330 addr v2:172.21.15.154:3300/0 2024-09-16T03:49:36.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:36.510+0000 7f7a9d6f4640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7a9806dac0 0x7f7a980a1c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:36.516 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:36.510+0000 7f7a9d6f4640 10 monclient(hunting): picked mon.noname-b con 0x7f7a9806dac0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:49:36.516 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:36.510+0000 7f7a9d6f4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7a98135100 addr v1:172.21.15.42:6789/0 2024-09-16T03:49:36.516 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:36.510+0000 7f7a9d6f4640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7a9809c7d0 con 0x7f7a98135100 2024-09-16T03:49:36.516 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:36.510+0000 7f7a9d6f4640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:36.516 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:36.510+0000 7f7a9d6f4640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f7a9809c090 con 0x7f7a980a1330 2024-09-16T03:49:36.516 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:36.510+0000 7f7a9d6f4640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:36.516 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:36.510+0000 7f7a9d6f4640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f7a9809f610 con 0x7f7a9806dac0 2024-09-16T03:49:36.516 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:36.510+0000 7f7a9d6f4640 10 monclient(hunting): _renew_subs 2024-09-16T03:49:36.516 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:36.510+0000 7f7a9d6f4640 10 monclient(hunting): authenticate will time out at 1423.480786s 2024-09-16T03:49:36.516 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:36.510+0000 7f7a967fc640 1 -- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7a9806dac0 msgr2=0x7f7a980a1c50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T03:49:36.516 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:36.510+0000 7f7a967fc640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7a9806dac0 0x7f7a980a1c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:49:36.516 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:36.510+0000 7f7a977fe640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f7a98135100 0x7f7a98135550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:43842/0 (socket says 172.21.15.42:43842) 2024-09-16T03:49:36.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:36.510+0000 7f7a977fe640 1 -- 172.21.15.42:0/2907460512 learned_addr learned my addr 172.21.15.42:0/2907460512 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:49:36.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:36.510+0000 7f7a96ffd640 1 --2- 172.21.15.42:0/2907460512 >> v2:172.21.15.154:3300/0 conn(0x7f7a980a1330 0x7f7a980a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:36.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:36.510+0000 7f7a96ffd640 10 monclient(hunting): get_auth_request con 0x7f7a980a1330 auth_method 0 2024-09-16T03:49:36.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:36.510+0000 7f7a96ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:36.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:36.510+0000 7f7a96ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:36.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:36.510+0000 7f7a96ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:36.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:36.510+0000 7f7a96ffd640 1 -- 172.21.15.42:0/2907460512 >> v2:172.21.15.154:3300/0 conn(0x7f7a980a1330 msgr2=0x7f7a980a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T03:49:36.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:36.510+0000 7f7a96ffd640 1 -- 172.21.15.42:0/2907460512 >> v2:172.21.15.154:3300/0 conn(0x7f7a980a1330 msgr2=0x7f7a980a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:49:36.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:36.510+0000 7f7a96ffd640 1 --2- 172.21.15.42:0/2907460512 >> v2:172.21.15.154:3300/0 conn(0x7f7a980a1330 0x7f7a980a1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:49:36.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:36.510+0000 7f7a96ffd640 1 --2- 172.21.15.42:0/2907460512 >> v2:172.21.15.154:3300/0 conn(0x7f7a980a1330 0x7f7a980a1710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:49:36.717 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:36.714+0000 7f7a967fc640 1 -- 172.21.15.42:0/2907460512 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7a9806dac0 msgr2=0x7f7a980a1c50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T03:49:36.717 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:36.714+0000 7f7a967fc640 1 --2- 172.21.15.42:0/2907460512 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7a9806dac0 0x7f7a980a1c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T03:49:36.718 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:36.714+0000 7f7a96ffd640 1 --2- 172.21.15.42:0/2907460512 >> v2:172.21.15.154:3300/0 conn(0x7f7a980a1330 0x7f7a980a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:36.718 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:36.714+0000 7f7a96ffd640 10 monclient(hunting): get_auth_request con 0x7f7a980a1330 auth_method 0 2024-09-16T03:49:36.718 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:36.714+0000 7f7a96ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:36.718 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:36.714+0000 7f7a96ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:36.718 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:36.714+0000 7f7a96ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:49:36.718 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:36.714+0000 7f7a96ffd640 1 -- 172.21.15.42:0/2907460512 >> v2:172.21.15.154:3300/0 conn(0x7f7a980a1330 msgr2=0x7f7a980a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T03:49:36.718 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:36.714+0000 7f7a96ffd640 1 -- 172.21.15.42:0/2907460512 >> v2:172.21.15.154:3300/0 conn(0x7f7a980a1330 msgr2=0x7f7a980a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:49:36.718 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:36.714+0000 7f7a96ffd640 1 --2- 172.21.15.42:0/2907460512 >> v2:172.21.15.154:3300/0 conn(0x7f7a980a1330 0x7f7a980a1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:49:36.718 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:36.714+0000 7f7a96ffd640 1 --2- 172.21.15.42:0/2907460512 >> v2:172.21.15.154:3300/0 conn(0x7f7a980a1330 0x7f7a980a1710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T03:49:37.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:37.118+0000 7f7a967fc640 1 -- 172.21.15.42:0/2907460512 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7a9806dac0 msgr2=0x7f7a980a1c50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T03:49:37.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:37.118+0000 7f7a967fc640 1 --2- 172.21.15.42:0/2907460512 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7a9806dac0 0x7f7a980a1c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T03:49:37.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:37.118+0000 7f7a96ffd640 1 --2- 172.21.15.42:0/2907460512 >> v2:172.21.15.154:3300/0 conn(0x7f7a980a1330 0x7f7a980a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:37.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:37.118+0000 7f7a96ffd640 10 monclient(hunting): get_auth_request con 0x7f7a980a1330 auth_method 0 2024-09-16T03:49:37.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:37.118+0000 7f7a96ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:37.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:37.118+0000 7f7a96ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:37.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:37.118+0000 7f7a96ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:49:37.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:37.118+0000 7f7a96ffd640 1 -- 172.21.15.42:0/2907460512 >> v2:172.21.15.154:3300/0 conn(0x7f7a980a1330 msgr2=0x7f7a980a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T03:49:37.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:37.118+0000 7f7a96ffd640 1 -- 172.21.15.42:0/2907460512 >> v2:172.21.15.154:3300/0 conn(0x7f7a980a1330 msgr2=0x7f7a980a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:49:37.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:37.118+0000 7f7a96ffd640 1 --2- 172.21.15.42:0/2907460512 >> v2:172.21.15.154:3300/0 conn(0x7f7a980a1330 0x7f7a980a1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:49:37.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:37.118+0000 7f7a96ffd640 1 --2- 172.21.15.42:0/2907460512 >> v2:172.21.15.154:3300/0 conn(0x7f7a980a1330 0x7f7a980a1710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T03:49:37.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:37.922+0000 7f7a967fc640 1 -- 172.21.15.42:0/2907460512 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7a9806dac0 msgr2=0x7f7a980a1c50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T03:49:37.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:37.922+0000 7f7a967fc640 1 --2- 172.21.15.42:0/2907460512 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7a9806dac0 0x7f7a980a1c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T03:49:37.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:37.922+0000 7f7a96ffd640 1 --2- 172.21.15.42:0/2907460512 >> v2:172.21.15.154:3300/0 conn(0x7f7a980a1330 0x7f7a980a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:37.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:37.922+0000 7f7a96ffd640 10 monclient(hunting): get_auth_request con 0x7f7a980a1330 auth_method 0 2024-09-16T03:49:37.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:37.922+0000 7f7a96ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:37.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:37.922+0000 7f7a96ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:37.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:37.922+0000 7f7a96ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:49:37.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:37.922+0000 7f7a96ffd640 1 -- 172.21.15.42:0/2907460512 >> v2:172.21.15.154:3300/0 conn(0x7f7a980a1330 msgr2=0x7f7a980a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T03:49:37.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:37.922+0000 7f7a96ffd640 1 -- 172.21.15.42:0/2907460512 >> v2:172.21.15.154:3300/0 conn(0x7f7a980a1330 msgr2=0x7f7a980a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:49:37.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:37.922+0000 7f7a96ffd640 1 --2- 172.21.15.42:0/2907460512 >> v2:172.21.15.154:3300/0 conn(0x7f7a980a1330 0x7f7a980a1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:49:37.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:37.922+0000 7f7a96ffd640 1 --2- 172.21.15.42:0/2907460512 >> v2:172.21.15.154:3300/0 conn(0x7f7a980a1330 0x7f7a980a1710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T03:49:39.514 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:39.510+0000 7f7a94ff9640 10 monclient(hunting): tick 2024-09-16T03:49:39.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:39.510+0000 7f7a94ff9640 10 monclient(hunting): _check_auth_tickets 2024-09-16T03:49:39.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:39.510+0000 7f7a94ff9640 1 monclient(hunting): continuing hunt 2024-09-16T03:49:39.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:39.510+0000 7f7a94ff9640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T03:49:39.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:39.510+0000 7f7a94ff9640 1 -- 172.21.15.42:0/2907460512 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7a9806dac0 msgr2=0x7f7a980a1c50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:49:39.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:39.510+0000 7f7a94ff9640 1 --2- 172.21.15.42:0/2907460512 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7a9806dac0 0x7f7a980a1c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:39.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:39.510+0000 7f7a94ff9640 1 -- 172.21.15.42:0/2907460512 >> v2:172.21.15.154:3300/0 conn(0x7f7a980a1330 msgr2=0x7f7a980a1710 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:49:39.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:39.510+0000 7f7a94ff9640 1 --2- 172.21.15.42:0/2907460512 >> v2:172.21.15.154:3300/0 conn(0x7f7a980a1330 0x7f7a980a1710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:39.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:39.510+0000 7f7a94ff9640 1 -- 172.21.15.42:0/2907460512 >> v1:172.21.15.42:6789/0 conn(0x7f7a98135100 legacy=0x7f7a98135550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:39.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:39.510+0000 7f7a94ff9640 10 monclient: _add_conns ranks=[1,3] 2024-09-16T03:49:39.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:39.510+0000 7f7a94ff9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7a840036f0 addr v1:172.21.15.154:6789/0 2024-09-16T03:49:39.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:39.510+0000 7f7a94ff9640 1 --2- 172.21.15.42:0/2907460512 >> v2:172.21.15.42:3300/0 conn(0x7f7a84007380 0x7f7a84009820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:39.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:39.510+0000 7f7a94ff9640 10 monclient(hunting): picked mon.noname-a con 0x7f7a84007380 addr v2:172.21.15.42:3300/0 2024-09-16T03:49:39.516 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:39.510+0000 7f7a94ff9640 1 -- 172.21.15.42:0/2907460512 --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7a9809f8f0 con 0x7f7a840036f0 2024-09-16T03:49:39.516 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:39.510+0000 7f7a94ff9640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:39.516 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:39.510+0000 7f7a94ff9640 1 -- 172.21.15.42:0/2907460512 --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f7a9809c090 con 0x7f7a84007380 2024-09-16T03:49:39.516 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:39.510+0000 7f7a94ff9640 10 monclient(hunting): _renew_subs 2024-09-16T03:49:39.516 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:39.510+0000 7f7a967fc640 1 --2- 172.21.15.42:0/2907460512 >> v2:172.21.15.42:3300/0 conn(0x7f7a84007380 0x7f7a84009820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:39.516 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:39.510+0000 7f7a967fc640 10 monclient(hunting): get_auth_request con 0x7f7a84007380 auth_method 0 2024-09-16T03:49:39.516 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:39.510+0000 7f7a967fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:39.516 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:39.510+0000 7f7a967fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:39.516 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:39.510+0000 7f7a967fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:39.516 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:39.510+0000 7f7a967fc640 1 -- 172.21.15.42:0/2907460512 >> v2:172.21.15.42:3300/0 conn(0x7f7a84007380 msgr2=0x7f7a84009820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T03:49:39.516 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:39.510+0000 7f7a967fc640 1 -- 172.21.15.42:0/2907460512 >> v2:172.21.15.42:3300/0 conn(0x7f7a84007380 msgr2=0x7f7a84009820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:49:39.516 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:39.510+0000 7f7a967fc640 1 --2- 172.21.15.42:0/2907460512 >> v2:172.21.15.42:3300/0 conn(0x7f7a84007380 0x7f7a84009820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:49:39.516 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:39.510+0000 7f7a967fc640 1 --2- 172.21.15.42:0/2907460512 >> v2:172.21.15.42:3300/0 conn(0x7f7a84007380 0x7f7a84009820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:49:39.717 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:39.714+0000 7f7a967fc640 1 --2- 172.21.15.42:0/2907460512 >> v2:172.21.15.42:3300/0 conn(0x7f7a84007380 0x7f7a84009820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:39.717 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:39.714+0000 7f7a967fc640 10 monclient(hunting): get_auth_request con 0x7f7a84007380 auth_method 0 2024-09-16T03:49:39.718 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:39.714+0000 7f7a967fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:39.718 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:39.714+0000 7f7a967fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:39.718 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:39.714+0000 7f7a967fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:49:39.718 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:39.714+0000 7f7a967fc640 1 -- 172.21.15.42:0/2907460512 >> v2:172.21.15.42:3300/0 conn(0x7f7a84007380 msgr2=0x7f7a84009820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T03:49:39.718 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:39.714+0000 7f7a967fc640 1 -- 172.21.15.42:0/2907460512 >> v2:172.21.15.42:3300/0 conn(0x7f7a84007380 msgr2=0x7f7a84009820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:49:39.718 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:39.714+0000 7f7a967fc640 1 --2- 172.21.15.42:0/2907460512 >> v2:172.21.15.42:3300/0 conn(0x7f7a84007380 0x7f7a84009820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:49:39.718 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:39.714+0000 7f7a967fc640 1 --2- 172.21.15.42:0/2907460512 >> v2:172.21.15.42:3300/0 conn(0x7f7a84007380 0x7f7a84009820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T03:49:40.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.118+0000 7f7a967fc640 1 --2- 172.21.15.42:0/2907460512 >> v2:172.21.15.42:3300/0 conn(0x7f7a84007380 0x7f7a84009820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:40.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.118+0000 7f7a967fc640 10 monclient(hunting): get_auth_request con 0x7f7a84007380 auth_method 0 2024-09-16T03:49:40.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.118+0000 7f7a967fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:40.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.118+0000 7f7a967fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:40.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.118+0000 7f7a967fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:49:40.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.118+0000 7f7a967fc640 1 -- 172.21.15.42:0/2907460512 >> v2:172.21.15.42:3300/0 conn(0x7f7a84007380 msgr2=0x7f7a84009820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T03:49:40.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.118+0000 7f7a967fc640 1 -- 172.21.15.42:0/2907460512 >> v2:172.21.15.42:3300/0 conn(0x7f7a84007380 msgr2=0x7f7a84009820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:49:40.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.118+0000 7f7a967fc640 1 --2- 172.21.15.42:0/2907460512 >> v2:172.21.15.42:3300/0 conn(0x7f7a84007380 0x7f7a84009820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:49:40.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.118+0000 7f7a967fc640 1 --2- 172.21.15.42:0/2907460512 >> v2:172.21.15.42:3300/0 conn(0x7f7a84007380 0x7f7a84009820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T03:49:40.456 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.450+0000 7f7a95ffb640 1 -- 172.21.15.42:0/2907460512 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2160555259 0 0) 0x7f7a9809f8f0 con 0x7f7a840036f0 2024-09-16T03:49:40.456 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.450+0000 7f7a95ffb640 10 monclient(hunting): _negotiate 2024-09-16T03:49:40.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.450+0000 7f7a95ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:40.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.450+0000 7f7a95ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:40.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.450+0000 7f7a95ffb640 10 monclient(hunting): my global_id is 794149 2024-09-16T03:49:40.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.450+0000 7f7a95ffb640 1 -- 172.21.15.42:0/2907460512 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7a80003530 con 0x7f7a840036f0 2024-09-16T03:49:40.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.450+0000 7f7a95ffb640 1 -- 172.21.15.42:0/2907460512 <== mon.1 v1:172.21.15.154:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 662139311 0 0) 0x7f7a80003530 con 0x7f7a840036f0 2024-09-16T03:49:40.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.450+0000 7f7a95ffb640 1 -- 172.21.15.42:0/2907460512 >> v2:172.21.15.42:3300/0 conn(0x7f7a84007380 msgr2=0x7f7a84009820 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:49:40.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.450+0000 7f7a95ffb640 1 --2- 172.21.15.42:0/2907460512 >> v2:172.21.15.42:3300/0 conn(0x7f7a84007380 0x7f7a84009820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:40.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.450+0000 7f7a95ffb640 10 monclient: _finish_hunting 0 2024-09-16T03:49:40.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.450+0000 7f7a95ffb640 1 monclient: found mon.noname-c-legacy 2024-09-16T03:49:40.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.450+0000 7f7a95ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:49:40.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.450+0000 7f7a95ffb640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.154:6789/0 2024-09-16T03:49:40.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.450+0000 7f7a95ffb640 1 -- 172.21.15.42:0/2907460512 --> v1:172.21.15.154:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a8400af90 con 0x7f7a840036f0 2024-09-16T03:49:40.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.450+0000 7f7a95ffb640 10 monclient: _finish_auth 0 2024-09-16T03:49:40.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.450+0000 7f7a95ffb640 10 monclient: _check_auth_tickets 2024-09-16T03:49:40.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.450+0000 7f7a95ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:49:40.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.450+0000 7f7a95ffb640 1 -- 172.21.15.42:0/2907460512 <== mon.1 v1:172.21.15.154:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2177413734 0 0) 0x7f7a8800f040 con 0x7f7a840036f0 2024-09-16T03:49:40.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.450+0000 7f7a95ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:40.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.450+0000 7f7a95ffb640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-16T03:49:40.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.450+0000 7f7a95ffb640 10 monclient: dump: 2024-09-16T03:49:40.458 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:49:40.458 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:49:40.458 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:40.458 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:40.458 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:40.458 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:49:40.458 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:49:40.459 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:49:40.459 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:49:40.459 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:49:40.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.450+0000 7f7a95ffb640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] but i'm connected to v1:172.21.15.154:6789/0, reconnecting 2024-09-16T03:49:40.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.450+0000 7f7a95ffb640 10 monclient: _reopen_session rank -1 2024-09-16T03:49:40.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.450+0000 7f7a95ffb640 1 -- 172.21.15.42:0/2907460512 >> v1:172.21.15.154:6789/0 conn(0x7f7a840036f0 legacy=0x7f7a84005b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:40.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.450+0000 7f7a977fe640 1 -- 172.21.15.42:0/2907460512 reap_dead start 2024-09-16T03:49:40.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.450+0000 7f7a95ffb640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:49:40.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.450+0000 7f7a95ffb640 1 --2- 172.21.15.42:0/2907460512 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7a800039c0 0x7f7a80003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:40.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.450+0000 7f7a95ffb640 10 monclient(hunting): picked mon.a con 0x7f7a800039c0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:49:40.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.450+0000 7f7a95ffb640 1 --2- 172.21.15.42:0/2907460512 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7a80004810 0x7f7a80004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:40.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.450+0000 7f7a95ffb640 10 monclient(hunting): picked mon.c con 0x7f7a80004810 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:49:40.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.450+0000 7f7a95ffb640 1 --2- 172.21.15.42:0/2907460512 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7a80005220 0x7f7a8000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:40.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.450+0000 7f7a95ffb640 10 monclient(hunting): picked mon.b con 0x7f7a80005220 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:49:40.460 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.450+0000 7f7a95ffb640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:40.460 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.450+0000 7f7a95ffb640 1 -- 172.21.15.42:0/2907460512 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f7a9809c090 con 0x7f7a800039c0 2024-09-16T03:49:40.460 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.450+0000 7f7a967fc640 1 -- 172.21.15.42:0/2907460512 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7a80004810 msgr2=0x7f7a80004c90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T03:49:40.460 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.450+0000 7f7a967fc640 1 --2- 172.21.15.42:0/2907460512 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7a80004810 0x7f7a80004c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:49:40.460 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.450+0000 7f7a95ffb640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:40.460 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.450+0000 7f7a95ffb640 1 -- 172.21.15.42:0/2907460512 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f7a8000c150 con 0x7f7a80005220 2024-09-16T03:49:40.460 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.450+0000 7f7a95ffb640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:40.460 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.450+0000 7f7a95ffb640 1 -- 172.21.15.42:0/2907460512 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f7a8000c350 con 0x7f7a80004810 2024-09-16T03:49:40.460 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.450+0000 7f7a95ffb640 10 monclient(hunting): _renew_subs 2024-09-16T03:49:40.460 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.450+0000 7f7a96ffd640 1 --2- 172.21.15.42:0/2907460512 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7a800039c0 0x7f7a80003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:40.460 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.450+0000 7f7a95ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:49:40.460 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.450+0000 7f7a96ffd640 10 monclient(hunting): get_auth_request con 0x7f7a800039c0 auth_method 0 2024-09-16T03:49:40.460 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.450+0000 7f7a96ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:40.461 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.450+0000 7f7a96ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:40.461 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.450+0000 7f7a96ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:49:40.461 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.450+0000 7f7a96ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:40.461 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.450+0000 7f7a96ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:40.461 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.450+0000 7f7a96ffd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T03:49:40.461 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.450+0000 7f7a977fe640 1 --2- 172.21.15.42:0/2907460512 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7a80005220 0x7f7a8000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:40.461 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.450+0000 7f7a96ffd640 10 monclient(hunting): handle_auth_done global_id 794149 payload 293 2024-09-16T03:49:40.461 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.450+0000 7f7a96ffd640 1 -- 172.21.15.42:0/2907460512 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7a80004810 msgr2=0x7f7a80004c90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:49:40.461 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.450+0000 7f7a96ffd640 1 --2- 172.21.15.42:0/2907460512 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7a80004810 0x7f7a80004c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:40.461 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.450+0000 7f7a96ffd640 1 -- 172.21.15.42:0/2907460512 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7a80005220 msgr2=0x7f7a8000b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:49:40.461 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.450+0000 7f7a96ffd640 1 --2- 172.21.15.42:0/2907460512 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7a80005220 0x7f7a8000b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:40.461 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.450+0000 7f7a96ffd640 10 monclient: _finish_hunting 0 2024-09-16T03:49:40.462 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.450+0000 7f7a96ffd640 1 monclient: found mon.a 2024-09-16T03:49:40.462 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.450+0000 7f7a96ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:49:40.462 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.450+0000 7f7a96ffd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:49:40.462 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.450+0000 7f7a96ffd640 1 -- 172.21.15.42:0/2907460512 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a80003530 con 0x7f7a800039c0 2024-09-16T03:49:40.462 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.450+0000 7f7a96ffd640 10 monclient: _finish_auth 0 2024-09-16T03:49:40.462 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.450+0000 7f7a96ffd640 10 monclient: _check_auth_tickets 2024-09-16T03:49:40.462 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.450+0000 7f7a96ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:49:40.462 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.450+0000 7f7a9d6f4640 5 monclient: authenticate success, global_id 794149 2024-09-16T03:49:40.462 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.450+0000 7f7a9d6f4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:49:40.462 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.450+0000 7f7a977fe640 10 monclient: get_auth_request con 0x7f7a80005220 auth_method 0 2024-09-16T03:49:40.462 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.450+0000 7f7a977fe640 1 --2- 172.21.15.42:0/2907460512 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7a80005220 0x7f7a8000b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T03:49:40.463 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.454+0000 7f7a96ffd640 1 --2- 172.21.15.42:0/2907460512 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7a800039c0 0x7f7a80003dc0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f7a88000ee0 tx=0x7f7a8801d820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab1ffee656b25eda server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:40.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.482+0000 7f7a95ffb640 1 -- 172.21.15.42:0/2907460512 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a880b4070 con 0x7f7a800039c0 2024-09-16T03:49:40.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.482+0000 7f7a95ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:40.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.482+0000 7f7a95ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:49:40.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.482+0000 7f7a95ffb640 10 monclient: dump: 2024-09-16T03:49:40.487 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:49:40.487 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:49:40.487 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:40.487 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:40.487 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:40.487 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:49:40.487 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:49:40.488 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:49:40.488 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:49:40.488 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:49:40.488 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.482+0000 7f7a9d6f4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:49:40.489 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.482+0000 7f7a95ffb640 1 -- 172.21.15.42:0/2907460512 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7a880af400 con 0x7f7a800039c0 2024-09-16T03:49:40.489 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.482+0000 7f7a95ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T03:49:40.489 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.482+0000 7f7a9d6f4640 10 monclient: get_monmap_and_config success 2024-09-16T03:49:40.489 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.482+0000 7f7a95ffb640 1 -- 172.21.15.42:0/2907460512 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a8800eb80 con 0x7f7a800039c0 2024-09-16T03:49:40.489 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.482+0000 7f7a9d6f4640 4 set_mon_vals no callback set 2024-09-16T03:49:40.489 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.482+0000 7f7a95ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:40.490 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.482+0000 7f7a95ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:49:40.490 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.482+0000 7f7a95ffb640 10 monclient: dump: 2024-09-16T03:49:40.490 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:49:40.490 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:49:40.490 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:40.490 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:40.490 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:40.490 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:49:40.490 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:49:40.490 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:49:40.490 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:49:40.490 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:49:40.490 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.482+0000 7f7a9d6f4640 10 monclient: shutdown 2024-09-16T03:49:40.490 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.482+0000 7f7a9d6f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:49:40.491 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.482+0000 7f7a9d6f4640 1 -- 172.21.15.42:0/2907460512 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7a800039c0 msgr2=0x7f7a80003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:40.491 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.482+0000 7f7a9d6f4640 1 --2- 172.21.15.42:0/2907460512 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7a800039c0 0x7f7a80003dc0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f7a88000ee0 tx=0x7f7a8801d820 comp rx=0 tx=0).stop 2024-09-16T03:49:40.491 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.482+0000 7f7a9d6f4640 1 -- 172.21.15.42:0/2907460512 shutdown_connections 2024-09-16T03:49:40.491 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.482+0000 7f7a9d6f4640 1 --2- 172.21.15.42:0/2907460512 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7a80005220 0x7f7a8000b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:40.491 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.482+0000 7f7a9d6f4640 1 --2- 172.21.15.42:0/2907460512 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7a80004810 0x7f7a80004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:40.491 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.482+0000 7f7a9d6f4640 1 --2- 172.21.15.42:0/2907460512 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7a800039c0 0x7f7a80003dc0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:40.491 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.482+0000 7f7a9d6f4640 1 -- 172.21.15.42:0/2907460512 >> 172.21.15.42:0/2907460512 conn(0x7f7a980a2420 msgr2=0x7f7a980a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:49:40.491 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.482+0000 7f7a9d6f4640 1 -- 172.21.15.42:0/2907460512 shutdown_connections 2024-09-16T03:49:40.491 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.482+0000 7f7a9d6f4640 1 -- 172.21.15.42:0/2907460512 wait complete. 2024-09-16T03:49:40.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.486+0000 7f7a9d6f4640 10 monclient: build_initial_monmap 2024-09-16T03:49:40.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.486+0000 7f7a9d6f4640 10 monclient: monmap: 2024-09-16T03:49:40.496 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:49:40.496 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:49:40.496 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:49:40.496 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:49:40.496 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:49:40.496 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:49:40.496 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:49:40.496 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:49:40.496 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:49:40.496 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:49:40.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.486+0000 7f7a9d6f4640 1 librados: starting msgr at 2024-09-16T03:49:40.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.486+0000 7f7a9d6f4640 1 librados: starting objecter 2024-09-16T03:49:40.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.486+0000 7f7a9d6f4640 1 Processor -- start 2024-09-16T03:49:40.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.486+0000 7f7a9d6f4640 1 -- start start 2024-09-16T03:49:40.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.486+0000 7f7a9d6f4640 1 librados: setting wanted keys 2024-09-16T03:49:40.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.486+0000 7f7a9d6f4640 1 librados: calling monclient init 2024-09-16T03:49:40.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.486+0000 7f7a9d6f4640 10 monclient: init 2024-09-16T03:49:40.498 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.486+0000 7f7a9d6f4640 10 monclient: _reopen_session rank -1 2024-09-16T03:49:40.498 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.486+0000 7f7a9d6f4640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:49:40.498 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.486+0000 7f7a9d6f4640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7a800039c0 0x7f7a981cc390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:40.498 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.486+0000 7f7a9d6f4640 10 monclient(hunting): picked mon.noname-c con 0x7f7a800039c0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:49:40.498 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.486+0000 7f7a9d6f4640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7a80004810 0x7f7a981cc8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:40.498 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.486+0000 7f7a9d6f4640 10 monclient(hunting): picked mon.noname-a con 0x7f7a80004810 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:49:40.498 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.486+0000 7f7a9d6f4640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7a80005220 0x7f7a98121970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:40.498 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.486+0000 7f7a9d6f4640 10 monclient(hunting): picked mon.noname-b con 0x7f7a80005220 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:49:40.498 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.486+0000 7f7a9d6f4640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:40.498 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.486+0000 7f7a9d6f4640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f7a9809eef0 con 0x7f7a80004810 2024-09-16T03:49:40.498 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.486+0000 7f7a9d6f4640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:40.498 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.486+0000 7f7a9d6f4640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f7a9809ed70 con 0x7f7a80005220 2024-09-16T03:49:40.499 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.486+0000 7f7a9d6f4640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:40.499 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.486+0000 7f7a9d6f4640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f7a980a1710 con 0x7f7a800039c0 2024-09-16T03:49:40.499 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.486+0000 7f7a9d6f4640 10 monclient(hunting): _renew_subs 2024-09-16T03:49:40.499 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.486+0000 7f7a9d6f4640 10 monclient(hunting): authenticate will time out at 1427.459813s 2024-09-16T03:49:40.499 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.486+0000 7f7a96ffd640 1 -- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7a800039c0 msgr2=0x7f7a981cc390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T03:49:40.499 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.486+0000 7f7a96ffd640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7a800039c0 0x7f7a981cc390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:49:40.499 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.490+0000 7f7a967fc640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7a80004810 0x7f7a981cc8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:40.499 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.490+0000 7f7a967fc640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7a80004810 0x7f7a981cc8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:32886/0 (socket says 172.21.15.42:32886) 2024-09-16T03:49:40.499 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.490+0000 7f7a967fc640 1 -- 172.21.15.42:0/1712997461 learned_addr learned my addr 172.21.15.42:0/1712997461 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:49:40.499 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.490+0000 7f7a967fc640 10 monclient(hunting): get_auth_request con 0x7f7a80004810 auth_method 0 2024-09-16T03:49:40.499 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.490+0000 7f7a967fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:40.499 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.490+0000 7f7a967fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:40.500 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.490+0000 7f7a967fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:40.500 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.490+0000 7f7a967fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:40.500 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.490+0000 7f7a967fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:40.500 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.490+0000 7f7a967fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:40.500 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.490+0000 7f7a977fe640 1 --2- 172.21.15.42:0/1712997461 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7a80005220 0x7f7a98121970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:40.500 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.490+0000 7f7a977fe640 10 monclient(hunting): get_auth_request con 0x7f7a80005220 auth_method 0 2024-09-16T03:49:40.500 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.490+0000 7f7a977fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:40.500 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.490+0000 7f7a977fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:40.500 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.490+0000 7f7a977fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:40.500 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.490+0000 7f7a967fc640 10 monclient(hunting): handle_auth_done global_id 764691 payload 824 2024-09-16T03:49:40.500 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.490+0000 7f7a967fc640 1 -- 172.21.15.42:0/1712997461 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7a800039c0 msgr2=0x7f7a981cc390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:49:40.500 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.490+0000 7f7a967fc640 1 --2- 172.21.15.42:0/1712997461 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7a800039c0 0x7f7a981cc390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:40.500 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.490+0000 7f7a967fc640 1 -- 172.21.15.42:0/1712997461 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7a80005220 msgr2=0x7f7a98121970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:40.501 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.490+0000 7f7a967fc640 1 --2- 172.21.15.42:0/1712997461 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7a80005220 0x7f7a98121970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:40.501 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.490+0000 7f7a967fc640 10 monclient: _finish_hunting 0 2024-09-16T03:49:40.501 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.490+0000 7f7a967fc640 1 monclient: found mon.noname-a 2024-09-16T03:49:40.501 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.490+0000 7f7a967fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:49:40.501 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.490+0000 7f7a967fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:49:40.501 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.490+0000 7f7a967fc640 1 -- 172.21.15.42:0/1712997461 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a98124170 con 0x7f7a80004810 2024-09-16T03:49:40.501 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.490+0000 7f7a967fc640 1 --2- 172.21.15.42:0/1712997461 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7a80004810 0x7f7a981cc8d0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f7a8c00bab0 tx=0x7f7a8c00bf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:40.501 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.490+0000 7f7a7f7fe640 10 client.?.objecter ms_handle_connect 0x7f7a80004810 2024-09-16T03:49:40.501 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.490+0000 7f7a7f7fe640 10 client.?.objecter resend_mon_ops 2024-09-16T03:49:40.501 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.490+0000 7f7a7f7fe640 1 -- 172.21.15.42:0/1712997461 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a8c016410 con 0x7f7a80004810 2024-09-16T03:49:40.501 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.490+0000 7f7a7f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:40.501 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.490+0000 7f7a7f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:49:40.501 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.490+0000 7f7a7f7fe640 10 monclient: dump: 2024-09-16T03:49:40.502 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:49:40.502 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:49:40.502 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:40.502 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:40.502 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:40.502 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:49:40.502 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:49:40.502 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:49:40.502 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:49:40.502 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:49:40.502 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.490+0000 7f7a7f7fe640 10 monclient: _finish_auth 0 2024-09-16T03:49:40.502 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.490+0000 7f7a7f7fe640 10 monclient: _check_auth_tickets 2024-09-16T03:49:40.502 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.490+0000 7f7a7f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:49:40.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.490+0000 7f7a7f7fe640 1 -- 172.21.15.42:0/1712997461 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7a8c01b070 con 0x7f7a80004810 2024-09-16T03:49:40.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.490+0000 7f7a7f7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:49:40.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.490+0000 7f7a7f7fe640 1 -- 172.21.15.42:0/1712997461 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a8c016810 con 0x7f7a80004810 2024-09-16T03:49:40.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.490+0000 7f7a9d6f4640 5 monclient: authenticate success, global_id 764691 2024-09-16T03:49:40.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.490+0000 7f7a9d6f4640 10 monclient: _renew_subs 2024-09-16T03:49:40.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.490+0000 7f7a9d6f4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:49:40.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.490+0000 7f7a9d6f4640 1 -- 172.21.15.42:0/1712997461 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a98125340 con 0x7f7a80004810 2024-09-16T03:49:40.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.490+0000 7f7a957fa640 4 set_mon_vals no callback set 2024-09-16T03:49:40.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.490+0000 7f7a7f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:40.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.490+0000 7f7a7f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:49:40.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.490+0000 7f7a7f7fe640 10 monclient: dump: 2024-09-16T03:49:40.503 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:49:40.503 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:49:40.504 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:40.504 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:40.504 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:40.504 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:49:40.504 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:49:40.504 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:49:40.504 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:49:40.504 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:49:40.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.490+0000 7f7a9d6f4640 10 client.764691.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:49:40.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.490+0000 7f7a9d6f4640 10 monclient: _renew_subs 2024-09-16T03:49:40.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.490+0000 7f7a9d6f4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:49:40.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.490+0000 7f7a9d6f4640 1 -- 172.21.15.42:0/1712997461 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a98125620 con 0x7f7a80004810 2024-09-16T03:49:40.505 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.490+0000 7f7a9d6f4640 1 librados: init done 2024-09-16T03:49:40.505 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.490+0000 7f7a7f7fe640 1 -- 172.21.15.42:0/1712997461 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f7a8c0169b0 con 0x7f7a80004810 2024-09-16T03:49:40.505 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.490+0000 7f7a9d6f4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:49:40.505 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.490+0000 7f7a9d6f4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:49:40.505 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.490+0000 7f7a9d6f4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:49:40.505 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.490+0000 7f7a9d6f4640 1 -- 172.21.15.42:0/1712997461 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7a98071280 con 0x7f7a80004810 2024-09-16T03:49:40.505 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.490+0000 7f7a7f7fe640 1 --2- 172.21.15.42:0/1712997461 >> v2:172.21.15.154:6812/254845342 conn(0x7f7a84038130 0x7f7a8403a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:40.505 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.490+0000 7f7a7f7fe640 1 -- 172.21.15.42:0/1712997461 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(248..248 src has 1..248) ==== 5807+0+0 (secure 0 0 0) 0x7f7a8c049aa0 con 0x7f7a80004810 2024-09-16T03:49:40.505 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.490+0000 7f7a7f7fe640 10 client.764691.objecter ms_dispatch 0x7f7a98000ba0 osd_map(248..248 src has 1..248) 2024-09-16T03:49:40.505 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.490+0000 7f7a7f7fe640 3 client.764691.objecter handle_osd_map got epochs [248,248] > 0 2024-09-16T03:49:40.505 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.490+0000 7f7a7f7fe640 3 client.764691.objecter handle_osd_map decoding full epoch 248 2024-09-16T03:49:40.505 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.490+0000 7f7a7f7fe640 20 client.764691.objecter dump_active .. 0 homeless 2024-09-16T03:49:40.505 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.490+0000 7f7a96ffd640 1 --2- 172.21.15.42:0/1712997461 >> v2:172.21.15.154:6812/254845342 conn(0x7f7a84038130 0x7f7a8403a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:40.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.490+0000 7f7a96ffd640 10 monclient: get_auth_request con 0x7f7a84038130 auth_method 0 2024-09-16T03:49:40.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.490+0000 7f7a96ffd640 1 --2- 172.21.15.42:0/1712997461 >> v2:172.21.15.154:6812/254845342 conn(0x7f7a84038130 0x7f7a8403a5f0 secure :-1 s=READY pgs=3521 cs=0 l=1 rev1=1 crypto rx=0x7f7a880b1000 tx=0x7f7a88010040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:40.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.490+0000 7f7a7f7fe640 10 client.764691.objecter ms_handle_connect 0x7f7a84038130 2024-09-16T03:49:40.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.494+0000 7f7a7f7fe640 1 -- 172.21.15.42:0/1712997461 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f7a8c01ee40 con 0x7f7a80004810 2024-09-16T03:49:40.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.494+0000 7f7a7f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:49:40.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.494+0000 7f7a7f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:49:40.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.726+0000 7f7a9d6f4640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:49:40.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.726+0000 7f7a9d6f4640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:49:40.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.726+0000 7f7a9d6f4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:49:40.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.726+0000 7f7a9d6f4640 1 -- 172.21.15.42:0/1712997461 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f7a98125860 con 0x7f7a80004810 2024-09-16T03:49:40.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.726+0000 7f7a7f7fe640 1 -- 172.21.15.42:0/1712997461 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1554 (secure 0 0 0) 0x7f7a8c0206f0 con 0x7f7a80004810 2024-09-16T03:49:40.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.726+0000 7f7a7f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:49:40.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.726+0000 7f7a7f7fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:49:40.731 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":70,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:49:40.733 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.726+0000 7f7a9d6f4640 10 librados: watch_flush enter 2024-09-16T03:49:40.733 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.726+0000 7f7a9d6f4640 10 librados: watch_flush exit 2024-09-16T03:49:40.733 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.726+0000 7f7a9d6f4640 20 client.764691.objecter shutdown clearing up homeless session... 2024-09-16T03:49:40.733 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.726+0000 7f7a9d6f4640 10 client.764691.objecter successfully canceled tick 2024-09-16T03:49:40.733 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.726+0000 7f7a9d6f4640 1 -- 172.21.15.42:0/1712997461 >> v2:172.21.15.154:6812/254845342 conn(0x7f7a84038130 msgr2=0x7f7a8403a5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:40.733 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.726+0000 7f7a9d6f4640 1 --2- 172.21.15.42:0/1712997461 >> v2:172.21.15.154:6812/254845342 conn(0x7f7a84038130 0x7f7a8403a5f0 secure :-1 s=READY pgs=3521 cs=0 l=1 rev1=1 crypto rx=0x7f7a880b1000 tx=0x7f7a88010040 comp rx=0 tx=0).stop 2024-09-16T03:49:40.733 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.726+0000 7f7a9d6f4640 10 monclient: shutdown 2024-09-16T03:49:40.733 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.726+0000 7f7a9d6f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:49:40.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.726+0000 7f7a9d6f4640 1 -- 172.21.15.42:0/1712997461 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7a80004810 msgr2=0x7f7a981cc8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:40.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.726+0000 7f7a9d6f4640 1 --2- 172.21.15.42:0/1712997461 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7a80004810 0x7f7a981cc8d0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f7a8c00bab0 tx=0x7f7a8c00bf70 comp rx=0 tx=0).stop 2024-09-16T03:49:40.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.726+0000 7f7a9d6f4640 1 -- 172.21.15.42:0/1712997461 shutdown_connections 2024-09-16T03:49:40.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.726+0000 7f7a9d6f4640 1 --2- 172.21.15.42:0/1712997461 >> v2:172.21.15.154:6812/254845342 conn(0x7f7a84038130 0x7f7a8403a5f0 unknown :-1 s=CLOSED pgs=3521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:40.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.726+0000 7f7a9d6f4640 1 --2- 172.21.15.42:0/1712997461 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7a80005220 0x7f7a98121970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:40.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.726+0000 7f7a9d6f4640 1 --2- 172.21.15.42:0/1712997461 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7a80004810 0x7f7a981cc8d0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:40.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.726+0000 7f7a9d6f4640 1 --2- 172.21.15.42:0/1712997461 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7a800039c0 0x7f7a981cc390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:40.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.726+0000 7f7a9d6f4640 1 -- 172.21.15.42:0/1712997461 >> 172.21.15.42:0/1712997461 conn(0x7f7a980a2420 msgr2=0x7f7a9811bf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:49:40.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.730+0000 7f7a9d6f4640 1 -- 172.21.15.42:0/1712997461 shutdown_connections 2024-09-16T03:49:40.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.730+0000 7f7a9d6f4640 1 -- 172.21.15.42:0/1712997461 wait complete. 2024-09-16T03:49:40.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.730+0000 7f7a9d6f4640 1 librados: shutdown 2024-09-16T03:49:40.750 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T03:49:40.751 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-16T03:49:40.751 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T03:49:40.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.866+0000 7f3826382640 10 monclient: get_monmap_and_config 2024-09-16T03:49:40.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.866+0000 7f3826382640 10 monclient: build_initial_monmap 2024-09-16T03:49:40.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.866+0000 7f3826382640 10 monclient: monmap: 2024-09-16T03:49:40.870 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:49:40.870 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:49:40.870 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:49:40.868626+0000 2024-09-16T03:49:40.870 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:49:40.868626+0000 2024-09-16T03:49:40.870 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:49:40.870 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:49:40.870 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:49:40.870 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:49:40.870 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:49:40.870 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:49:40.870 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:49:40.870 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:49:40.875 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.866+0000 7f3826382640 1 Processor -- start 2024-09-16T03:49:40.875 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.866+0000 7f3826382640 1 -- start start 2024-09-16T03:49:40.876 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.866+0000 7f3826382640 10 monclient: init 2024-09-16T03:49:40.876 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.866+0000 7f3826382640 10 monclient: _reopen_session rank -1 2024-09-16T03:49:40.876 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.866+0000 7f3826382640 10 monclient: _add_conns ranks=[0,2,4,1,3] 2024-09-16T03:49:40.876 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.866+0000 7f3826382640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f38200a1330 addr v1:172.21.15.42:6789/0 2024-09-16T03:49:40.876 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.866+0000 7f3826382640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f382006dac0 0x7f382013a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:40.876 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.866+0000 7f3826382640 10 monclient(hunting): picked mon.noname-b con 0x7f382006dac0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:49:40.876 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.866+0000 7f3826382640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f38200a1e20 0x7f382013c890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:40.876 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.866+0000 7f3826382640 10 monclient(hunting): picked mon.noname-c con 0x7f38200a1e20 addr v2:172.21.15.154:3300/0 2024-09-16T03:49:40.876 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.866+0000 7f3826382640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f382009c7d0 con 0x7f38200a1330 2024-09-16T03:49:40.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.866+0000 7f3826382640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:40.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.866+0000 7f3826382640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f382009c090 con 0x7f38200a1e20 2024-09-16T03:49:40.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.866+0000 7f3826382640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:40.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.866+0000 7f3826382640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f382009f610 con 0x7f382006dac0 2024-09-16T03:49:40.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.866+0000 7f3826382640 10 monclient(hunting): _renew_subs 2024-09-16T03:49:40.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.866+0000 7f3826382640 10 monclient(hunting): authenticate will time out at 1427.838234s 2024-09-16T03:49:40.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.866+0000 7f381f7fe640 1 -- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f382006dac0 msgr2=0x7f382013a340 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T03:49:40.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.866+0000 7f381f7fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f382006dac0 0x7f382013a340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:49:40.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.866+0000 7f381ffff640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f38200a1330 0x7f38200a1710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:43852/0 (socket says 172.21.15.42:43852) 2024-09-16T03:49:40.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.866+0000 7f381ffff640 1 -- 172.21.15.42:0/4288844508 learned_addr learned my addr 172.21.15.42:0/4288844508 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:49:40.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.866+0000 7f382491f640 1 --2- 172.21.15.42:0/4288844508 >> v2:172.21.15.154:3300/0 conn(0x7f38200a1e20 0x7f382013c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:40.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.866+0000 7f382491f640 10 monclient(hunting): get_auth_request con 0x7f38200a1e20 auth_method 0 2024-09-16T03:49:40.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.866+0000 7f382491f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:40.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.866+0000 7f382491f640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:40.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.866+0000 7f382491f640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:40.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.866+0000 7f381effd640 1 -- 172.21.15.42:0/4288844508 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2136804797 0 0) 0x7f382009c7d0 con 0x7f38200a1330 2024-09-16T03:49:40.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.866+0000 7f381effd640 10 monclient(hunting): _negotiate 2024-09-16T03:49:40.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.866+0000 7f381effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:40.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.866+0000 7f381effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:40.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.866+0000 7f381effd640 10 monclient(hunting): my global_id is 764694 2024-09-16T03:49:40.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.866+0000 7f381effd640 1 -- 172.21.15.42:0/4288844508 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3800003530 con 0x7f38200a1330 2024-09-16T03:49:40.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.866+0000 7f382491f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:40.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.866+0000 7f382491f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:40.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.866+0000 7f382491f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:40.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.866+0000 7f381effd640 1 -- 172.21.15.42:0/4288844508 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3106981104 0 0) 0x7f3800003530 con 0x7f38200a1330 2024-09-16T03:49:40.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.866+0000 7f381effd640 1 -- 172.21.15.42:0/4288844508 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f382006dac0 msgr2=0x7f382013a340 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:49:40.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.866+0000 7f381effd640 1 --2- 172.21.15.42:0/4288844508 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f382006dac0 0x7f382013a340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:40.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.866+0000 7f381effd640 1 -- 172.21.15.42:0/4288844508 >> v2:172.21.15.154:3300/0 conn(0x7f38200a1e20 msgr2=0x7f382013c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:49:40.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.866+0000 7f381effd640 1 --2- 172.21.15.42:0/4288844508 >> v2:172.21.15.154:3300/0 conn(0x7f38200a1e20 0x7f382013c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:40.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.866+0000 7f381effd640 10 monclient: _finish_hunting 0 2024-09-16T03:49:40.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.866+0000 7f381effd640 1 monclient: found mon.noname-a-legacy 2024-09-16T03:49:40.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.866+0000 7f381effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:49:40.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.866+0000 7f381effd640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.42:6789/0 2024-09-16T03:49:40.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.866+0000 7f381effd640 1 -- 172.21.15.42:0/4288844508 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f382009f8f0 con 0x7f38200a1330 2024-09-16T03:49:40.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.866+0000 7f381effd640 10 monclient: _finish_auth 0 2024-09-16T03:49:40.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.866+0000 7f381effd640 10 monclient: _check_auth_tickets 2024-09-16T03:49:40.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.866+0000 7f381effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:49:40.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.866+0000 7f3826382640 5 monclient: authenticate success, global_id 764694 2024-09-16T03:49:40.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.866+0000 7f381effd640 1 -- 172.21.15.42:0/4288844508 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2177413734 0 0) 0x7f3818001680 con 0x7f38200a1330 2024-09-16T03:49:40.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.866+0000 7f3826382640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:49:40.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.866+0000 7f381effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:40.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.866+0000 7f381effd640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T03:49:40.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.866+0000 7f381effd640 10 monclient: dump: 2024-09-16T03:49:40.880 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:49:40.880 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:49:40.880 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:40.880 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:40.880 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:40.880 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:49:40.880 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:49:40.880 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:49:40.880 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:49:40.881 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:49:40.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.866+0000 7f381effd640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] but i'm connected to v1:172.21.15.42:6789/0, reconnecting 2024-09-16T03:49:40.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.866+0000 7f381effd640 10 monclient: _reopen_session rank -1 2024-09-16T03:49:40.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.866+0000 7f381effd640 1 -- 172.21.15.42:0/4288844508 >> v1:172.21.15.42:6789/0 conn(0x7f38200a1330 legacy=0x7f38200a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:40.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.866+0000 7f381effd640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:49:40.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.866+0000 7f381effd640 1 --2- 172.21.15.42:0/4288844508 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f38000039c0 0x7f3800003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:40.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.866+0000 7f381effd640 10 monclient(hunting): picked mon.c con 0x7f38000039c0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:49:40.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.866+0000 7f381effd640 1 --2- 172.21.15.42:0/4288844508 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3800004340 0x7f3800004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:40.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.866+0000 7f381effd640 10 monclient(hunting): picked mon.a con 0x7f3800004340 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:49:40.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.866+0000 7f381effd640 1 --2- 172.21.15.42:0/4288844508 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3800004fc0 0x7f380000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:40.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.866+0000 7f381effd640 10 monclient(hunting): picked mon.b con 0x7f3800004fc0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:49:40.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.866+0000 7f381effd640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:40.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.866+0000 7f381effd640 1 -- 172.21.15.42:0/4288844508 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f382009c090 con 0x7f3800004340 2024-09-16T03:49:40.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.866+0000 7f381effd640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:40.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.866+0000 7f381effd640 1 -- 172.21.15.42:0/4288844508 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f382009f610 con 0x7f3800004fc0 2024-09-16T03:49:40.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.866+0000 7f381effd640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:40.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.866+0000 7f381effd640 1 -- 172.21.15.42:0/4288844508 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f380000bf10 con 0x7f38000039c0 2024-09-16T03:49:40.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.866+0000 7f381effd640 10 monclient(hunting): _renew_subs 2024-09-16T03:49:40.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.866+0000 7f381ffff640 1 -- 172.21.15.42:0/4288844508 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f38000039c0 msgr2=0x7f3800003dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T03:49:40.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.866+0000 7f381effd640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:49:40.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.866+0000 7f3826382640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-16T03:49:40.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.866+0000 7f381ffff640 1 --2- 172.21.15.42:0/4288844508 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f38000039c0 0x7f3800003dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:49:40.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.866+0000 7f381f7fe640 1 --2- 172.21.15.42:0/4288844508 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3800004340 0x7f3800004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:40.883 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.866+0000 7f381f7fe640 10 monclient(hunting): get_auth_request con 0x7f3800004340 auth_method 0 2024-09-16T03:49:40.883 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.866+0000 7f381f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:40.883 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.866+0000 7f381f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:40.883 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.866+0000 7f381f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:49:40.883 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.866+0000 7f381f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:40.883 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.866+0000 7f381f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:40.883 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.866+0000 7f381f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T03:49:40.883 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.866+0000 7f381f7fe640 10 monclient(hunting): handle_auth_done global_id 764694 payload 293 2024-09-16T03:49:40.883 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.866+0000 7f381f7fe640 1 -- 172.21.15.42:0/4288844508 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f38000039c0 msgr2=0x7f3800003dc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:49:40.883 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.866+0000 7f381f7fe640 1 --2- 172.21.15.42:0/4288844508 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f38000039c0 0x7f3800003dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:40.883 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.866+0000 7f381f7fe640 1 -- 172.21.15.42:0/4288844508 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3800004fc0 msgr2=0x7f380000b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:49:40.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.866+0000 7f382491f640 1 --2- 172.21.15.42:0/4288844508 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3800004fc0 0x7f380000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:40.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.866+0000 7f381f7fe640 1 --2- 172.21.15.42:0/4288844508 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3800004fc0 0x7f380000b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:40.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.866+0000 7f381f7fe640 10 monclient: _finish_hunting 0 2024-09-16T03:49:40.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.866+0000 7f381f7fe640 1 monclient: found mon.a 2024-09-16T03:49:40.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.866+0000 7f381f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:49:40.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.866+0000 7f381f7fe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:49:40.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.866+0000 7f381f7fe640 1 -- 172.21.15.42:0/4288844508 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3800003530 con 0x7f3800004340 2024-09-16T03:49:40.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.866+0000 7f381f7fe640 10 monclient: _finish_auth 0 2024-09-16T03:49:40.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.866+0000 7f381f7fe640 10 monclient: _check_auth_tickets 2024-09-16T03:49:40.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.866+0000 7f381f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:49:40.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.866+0000 7f382491f640 10 monclient: get_auth_request con 0x7f3800004fc0 auth_method 0 2024-09-16T03:49:40.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.866+0000 7f382491f640 1 --2- 172.21.15.42:0/4288844508 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3800004fc0 0x7f380000b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T03:49:40.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.870+0000 7f381f7fe640 1 --2- 172.21.15.42:0/4288844508 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3800004340 0x7f3800004810 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f3810009e30 tx=0x7f381001ca90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a235ce536c24d091 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:40.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.870+0000 7f381effd640 1 -- 172.21.15.42:0/4288844508 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3810009a30 con 0x7f3800004340 2024-09-16T03:49:40.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.870+0000 7f381effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:40.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.870+0000 7f381effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:49:40.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.870+0000 7f381effd640 10 monclient: dump: 2024-09-16T03:49:40.885 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:49:40.885 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:49:40.885 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:40.885 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:40.885 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:40.885 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:49:40.885 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:49:40.885 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:49:40.885 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:49:40.885 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:49:40.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.870+0000 7f381effd640 1 -- 172.21.15.42:0/4288844508 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f38100a9070 con 0x7f3800004340 2024-09-16T03:49:40.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.870+0000 7f381effd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:49:40.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.870+0000 7f381effd640 1 -- 172.21.15.42:0/4288844508 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38100a5c30 con 0x7f3800004340 2024-09-16T03:49:40.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.870+0000 7f3826382640 10 monclient: get_monmap_and_config success 2024-09-16T03:49:40.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.870+0000 7f3826382640 4 set_mon_vals no callback set 2024-09-16T03:49:40.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.870+0000 7f381effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:40.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.870+0000 7f381effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:49:40.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.870+0000 7f381effd640 10 monclient: dump: 2024-09-16T03:49:40.886 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:49:40.886 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:49:40.886 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:40.886 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:40.886 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:40.886 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:49:40.887 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:49:40.887 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:49:40.887 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:49:40.887 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:49:40.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.870+0000 7f3826382640 10 monclient: shutdown 2024-09-16T03:49:40.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.870+0000 7f3826382640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:49:40.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.870+0000 7f3826382640 1 -- 172.21.15.42:0/4288844508 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3800004340 msgr2=0x7f3800004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:40.888 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.870+0000 7f3826382640 1 --2- 172.21.15.42:0/4288844508 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3800004340 0x7f3800004810 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f3810009e30 tx=0x7f381001ca90 comp rx=0 tx=0).stop 2024-09-16T03:49:40.888 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.870+0000 7f382491f640 1 -- 172.21.15.42:0/4288844508 reap_dead start 2024-09-16T03:49:40.888 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.870+0000 7f3826382640 1 -- 172.21.15.42:0/4288844508 shutdown_connections 2024-09-16T03:49:40.888 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.870+0000 7f3826382640 1 -- 172.21.15.42:0/4288844508 >> 172.21.15.42:0/4288844508 conn(0x7f38200a2420 msgr2=0x7f38200a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:49:40.888 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.870+0000 7f3826382640 1 -- 172.21.15.42:0/4288844508 shutdown_connections 2024-09-16T03:49:40.888 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.870+0000 7f3826382640 1 -- 172.21.15.42:0/4288844508 wait complete. 2024-09-16T03:49:40.895 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.870+0000 7f3826382640 10 monclient: build_initial_monmap 2024-09-16T03:49:40.895 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.870+0000 7f3826382640 10 monclient: monmap: 2024-09-16T03:49:40.895 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:49:40.896 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:49:40.896 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:49:40.896 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:49:40.896 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:49:40.896 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:49:40.896 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:49:40.896 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:49:40.896 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:49:40.896 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:49:40.896 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.870+0000 7f3826382640 1 librados: starting msgr at 2024-09-16T03:49:40.896 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.870+0000 7f3826382640 1 librados: starting objecter 2024-09-16T03:49:40.896 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.870+0000 7f3826382640 1 Processor -- start 2024-09-16T03:49:40.896 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.870+0000 7f3826382640 1 -- start start 2024-09-16T03:49:40.896 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.870+0000 7f3826382640 1 librados: setting wanted keys 2024-09-16T03:49:40.897 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.870+0000 7f3826382640 1 librados: calling monclient init 2024-09-16T03:49:40.897 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.870+0000 7f3826382640 10 monclient: init 2024-09-16T03:49:40.897 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.874+0000 7f3826382640 10 monclient: _reopen_session rank -1 2024-09-16T03:49:40.897 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.874+0000 7f3826382640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:49:40.897 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.874+0000 7f3826382640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f38201c9be0 0x7f38201c9fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:40.897 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.874+0000 7f3826382640 10 monclient(hunting): picked mon.noname-a con 0x7f38201c9be0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:49:40.897 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.874+0000 7f3826382640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f38201346b0 0x7f3820134a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:40.897 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.874+0000 7f3826382640 10 monclient(hunting): picked mon.noname-c con 0x7f38201346b0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:49:40.897 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.874+0000 7f3826382640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f382011e0e0 0x7f382011e550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:40.897 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.874+0000 7f3826382640 10 monclient(hunting): picked mon.noname-b con 0x7f382011e0e0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:49:40.897 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.874+0000 7f3826382640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:40.897 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.874+0000 7f3826382640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f38200a32f0 con 0x7f38201c9be0 2024-09-16T03:49:40.897 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.874+0000 7f3826382640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:40.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.874+0000 7f3826382640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f38200a3170 con 0x7f382011e0e0 2024-09-16T03:49:40.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.874+0000 7f3826382640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:40.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.874+0000 7f3826382640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f382009ed70 con 0x7f38201346b0 2024-09-16T03:49:40.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.874+0000 7f3826382640 10 monclient(hunting): _renew_subs 2024-09-16T03:49:40.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.874+0000 7f3826382640 10 monclient(hunting): authenticate will time out at 1427.844520s 2024-09-16T03:49:40.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.874+0000 7f381f7fe640 1 -- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f38201346b0 msgr2=0x7f3820134a90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T03:49:40.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.874+0000 7f381f7fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f38201346b0 0x7f3820134a90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:49:40.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.874+0000 7f381ffff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f38201c9be0 0x7f38201c9fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:40.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.874+0000 7f381ffff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f38201c9be0 0x7f38201c9fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:32904/0 (socket says 172.21.15.42:32904) 2024-09-16T03:49:40.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.874+0000 7f381ffff640 1 -- 172.21.15.42:0/2099506568 learned_addr learned my addr 172.21.15.42:0/2099506568 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:49:40.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.874+0000 7f381ffff640 10 monclient(hunting): get_auth_request con 0x7f38201c9be0 auth_method 0 2024-09-16T03:49:40.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.874+0000 7f381ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:40.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.874+0000 7f381ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:40.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.874+0000 7f381ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:40.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.874+0000 7f381ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:40.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.874+0000 7f381ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:40.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.874+0000 7f381ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:40.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.874+0000 7f382491f640 1 --2- 172.21.15.42:0/2099506568 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f382011e0e0 0x7f382011e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:40.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.874+0000 7f382491f640 10 monclient(hunting): get_auth_request con 0x7f382011e0e0 auth_method 0 2024-09-16T03:49:40.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.874+0000 7f382491f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:40.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.874+0000 7f382491f640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:40.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.874+0000 7f382491f640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:40.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.874+0000 7f381ffff640 10 monclient(hunting): handle_auth_done global_id 764697 payload 824 2024-09-16T03:49:40.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.874+0000 7f381ffff640 1 -- 172.21.15.42:0/2099506568 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f38201346b0 msgr2=0x7f3820134a90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:49:40.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.874+0000 7f381ffff640 1 --2- 172.21.15.42:0/2099506568 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f38201346b0 0x7f3820134a90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:40.900 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.874+0000 7f381ffff640 1 -- 172.21.15.42:0/2099506568 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f382011e0e0 msgr2=0x7f382011e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:40.900 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.874+0000 7f381ffff640 1 --2- 172.21.15.42:0/2099506568 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f382011e0e0 0x7f382011e550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:40.900 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.874+0000 7f381ffff640 10 monclient: _finish_hunting 0 2024-09-16T03:49:40.900 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.874+0000 7f381ffff640 1 monclient: found mon.noname-a 2024-09-16T03:49:40.900 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.874+0000 7f381ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:49:40.900 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.874+0000 7f381ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:49:40.900 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.874+0000 7f381ffff640 1 -- 172.21.15.42:0/2099506568 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3820122070 con 0x7f38201c9be0 2024-09-16T03:49:40.900 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.874+0000 7f381ffff640 1 --2- 172.21.15.42:0/2099506568 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f38201c9be0 0x7f38201c9fc0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f381800f410 tx=0x7f381800f8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:40.900 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.874+0000 7f381cff9640 10 client.?.objecter ms_handle_connect 0x7f38201c9be0 2024-09-16T03:49:40.900 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.874+0000 7f381cff9640 10 client.?.objecter resend_mon_ops 2024-09-16T03:49:40.900 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.874+0000 7f381cff9640 1 -- 172.21.15.42:0/2099506568 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3818007d00 con 0x7f38201c9be0 2024-09-16T03:49:40.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.874+0000 7f381cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:40.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.874+0000 7f381cff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:49:40.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.874+0000 7f381cff9640 10 monclient: dump: 2024-09-16T03:49:40.901 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:49:40.901 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:49:40.901 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:40.901 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:40.901 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:40.901 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:49:40.901 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:49:40.901 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:49:40.901 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:49:40.902 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:49:40.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.874+0000 7f381cff9640 10 monclient: _finish_auth 0 2024-09-16T03:49:40.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.874+0000 7f381cff9640 10 monclient: _check_auth_tickets 2024-09-16T03:49:40.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.874+0000 7f381cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:49:40.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.874+0000 7f3826382640 5 monclient: authenticate success, global_id 764697 2024-09-16T03:49:40.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.874+0000 7f381cff9640 1 -- 172.21.15.42:0/2099506568 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f381800a0d0 con 0x7f38201c9be0 2024-09-16T03:49:40.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.874+0000 7f3826382640 10 monclient: _renew_subs 2024-09-16T03:49:40.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.874+0000 7f3826382640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:49:40.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.874+0000 7f3826382640 1 -- 172.21.15.42:0/2099506568 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f382006e060 con 0x7f38201c9be0 2024-09-16T03:49:40.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.874+0000 7f381cff9640 10 monclient: handle_config config(0 keys) 2024-09-16T03:49:40.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.874+0000 7f381cff9640 1 -- 172.21.15.42:0/2099506568 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f381800a4c0 con 0x7f38201c9be0 2024-09-16T03:49:40.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.874+0000 7f381cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:40.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.874+0000 7f381cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:49:40.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.874+0000 7f381cff9640 10 monclient: dump: 2024-09-16T03:49:40.903 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:49:40.903 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:49:40.903 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:40.903 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:40.903 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:40.903 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:49:40.903 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:49:40.903 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:49:40.903 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:49:40.903 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:49:40.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.874+0000 7f381e7fc640 4 set_mon_vals no callback set 2024-09-16T03:49:40.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.874+0000 7f3826382640 10 client.764697.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:49:40.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.874+0000 7f3826382640 10 monclient: _renew_subs 2024-09-16T03:49:40.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.874+0000 7f3826382640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:49:40.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.874+0000 7f3826382640 1 -- 172.21.15.42:0/2099506568 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3820120660 con 0x7f38201c9be0 2024-09-16T03:49:40.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.874+0000 7f3826382640 1 librados: init done 2024-09-16T03:49:40.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.874+0000 7f3826382640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:49:40.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.874+0000 7f3826382640 1 --2- 172.21.15.42:0/2099506568 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f382011ea90 0x7f382006f410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:40.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.874+0000 7f3826382640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:40.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.874+0000 7f3826382640 1 -- 172.21.15.42:0/2099506568 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f382006dac0 con 0x7f382011ea90 2024-09-16T03:49:40.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.874+0000 7f381cff9640 1 -- 172.21.15.42:0/2099506568 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f3818014070 con 0x7f38201c9be0 2024-09-16T03:49:40.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.874+0000 7f381f7fe640 1 --2- 172.21.15.42:0/2099506568 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f382011ea90 0x7f382006f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:40.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.874+0000 7f381f7fe640 10 monclient: get_auth_request con 0x7f382011ea90 auth_method 0 2024-09-16T03:49:40.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.874+0000 7f381f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:40.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.874+0000 7f381f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:40.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.874+0000 7f381f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:40.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.874+0000 7f381cff9640 1 --2- 172.21.15.42:0/2099506568 >> v2:172.21.15.154:6812/254845342 conn(0x7f38040377c0 0x7f3804039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:40.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.874+0000 7f381f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:40.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.874+0000 7f381f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:40.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.874+0000 7f381f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:40.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.874+0000 7f381cff9640 1 -- 172.21.15.42:0/2099506568 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(248..248 src has 1..248) ==== 5807+0+0 (secure 0 0 0) 0x7f381804ba20 con 0x7f38201c9be0 2024-09-16T03:49:40.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.874+0000 7f381cff9640 10 client.764697.objecter ms_dispatch 0x7f3820000ba0 osd_map(248..248 src has 1..248) 2024-09-16T03:49:40.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.874+0000 7f381cff9640 3 client.764697.objecter handle_osd_map got epochs [248,248] > 0 2024-09-16T03:49:40.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.874+0000 7f381cff9640 3 client.764697.objecter handle_osd_map decoding full epoch 248 2024-09-16T03:49:40.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.874+0000 7f381cff9640 20 client.764697.objecter dump_active .. 0 homeless 2024-09-16T03:49:40.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.874+0000 7f381f7fe640 10 monclient(hunting): handle_auth_done global_id 764700 payload 824 2024-09-16T03:49:40.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.874+0000 7f381f7fe640 1 -- 172.21.15.42:0/2099506568 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f382011ee70 con 0x7f382011ea90 2024-09-16T03:49:40.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.874+0000 7f382491f640 1 --2- 172.21.15.42:0/2099506568 >> v2:172.21.15.154:6812/254845342 conn(0x7f38040377c0 0x7f3804039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:40.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.874+0000 7f381cff9640 10 client.764697.objecter ms_handle_connect 0x7f382011ea90 2024-09-16T03:49:40.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.874+0000 7f381cff9640 10 client.764697.objecter resend_mon_ops 2024-09-16T03:49:40.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.874+0000 7f381f7fe640 1 --2- 172.21.15.42:0/2099506568 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f382011ea90 0x7f382006f410 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f3810003d10 tx=0x7f3800005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:40.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.874+0000 7f382491f640 10 monclient: get_auth_request con 0x7f38040377c0 auth_method 0 2024-09-16T03:49:40.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.874+0000 7f381cff9640 1 -- 172.21.15.42:0/2099506568 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38100ae670 con 0x7f382011ea90 2024-09-16T03:49:40.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.874+0000 7f381cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:40.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.874+0000 7f381cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:49:40.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.874+0000 7f381cff9640 10 monclient: dump: 2024-09-16T03:49:40.906 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:49:40.907 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:49:40.907 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:40.907 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:40.907 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:40.907 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:49:40.907 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:49:40.907 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:49:40.907 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:49:40.907 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:49:40.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.874+0000 7f381cff9640 1 -- 172.21.15.42:0/2099506568 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f38100a9050 con 0x7f382011ea90 2024-09-16T03:49:40.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.874+0000 7f381cff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:49:40.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.874+0000 7f381cff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:49:40.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.874+0000 7f381cff9640 1 -- 172.21.15.42:0/2099506568 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f382011ea90 msgr2=0x7f382006f410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:40.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.874+0000 7f381cff9640 1 --2- 172.21.15.42:0/2099506568 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f382011ea90 0x7f382006f410 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f3810003d10 tx=0x7f3800005840 comp rx=0 tx=0).stop 2024-09-16T03:49:40.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.874+0000 7f381cff9640 1 -- 172.21.15.42:0/2099506568 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f382011ea90 msgr2=0x7f382006f410 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:49:40.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.874+0000 7f381cff9640 1 --2- 172.21.15.42:0/2099506568 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f382011ea90 0x7f382006f410 secure :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f3810003d10 tx=0x7f3800005840 comp rx=0 tx=0).stop 2024-09-16T03:49:40.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.874+0000 7f381cff9640 10 client.764697.objecter ms_handle_connect 0x7f38040377c0 2024-09-16T03:49:40.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.874+0000 7f382491f640 1 --2- 172.21.15.42:0/2099506568 >> v2:172.21.15.154:6812/254845342 conn(0x7f38040377c0 0x7f3804039c80 secure :-1 s=READY pgs=3522 cs=0 l=1 rev1=1 crypto rx=0x7f382006c630 tx=0x7f3808008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:40.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.882+0000 7f3826382640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:49:40.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.882+0000 7f3826382640 1 --2- 172.21.15.42:0/2099506568 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f382006f950 0x7f382006fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:40.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.882+0000 7f3826382640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:40.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.882+0000 7f3826382640 1 -- 172.21.15.42:0/2099506568 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f3820069040 con 0x7f382006f950 2024-09-16T03:49:40.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.882+0000 7f381f7fe640 1 --2- 172.21.15.42:0/2099506568 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f382006f950 0x7f382006fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:40.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.882+0000 7f381f7fe640 10 monclient: get_auth_request con 0x7f382006f950 auth_method 0 2024-09-16T03:49:40.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.882+0000 7f381f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:40.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.882+0000 7f381f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:40.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.882+0000 7f381f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:40.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.882+0000 7f381f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:40.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.882+0000 7f381f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:40.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.882+0000 7f381f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:40.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.882+0000 7f381f7fe640 10 monclient(hunting): handle_auth_done global_id 764703 payload 824 2024-09-16T03:49:40.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.882+0000 7f381f7fe640 1 -- 172.21.15.42:0/2099506568 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f38200702a0 con 0x7f382006f950 2024-09-16T03:49:40.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.882+0000 7f381f7fe640 1 --2- 172.21.15.42:0/2099506568 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f382006f950 0x7f382006fd60 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f3810009e30 tx=0x7f3810009520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:40.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.882+0000 7f381cff9640 10 client.764697.objecter ms_handle_connect 0x7f382006f950 2024-09-16T03:49:40.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.882+0000 7f381cff9640 10 client.764697.objecter resend_mon_ops 2024-09-16T03:49:40.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.882+0000 7f381cff9640 1 -- 172.21.15.42:0/2099506568 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38100b8ba0 con 0x7f382006f950 2024-09-16T03:49:40.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.882+0000 7f381cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:40.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.882+0000 7f381cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:49:40.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.882+0000 7f381cff9640 10 monclient: dump: 2024-09-16T03:49:40.910 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:49:40.910 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:49:40.910 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:40.910 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:40.910 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:40.910 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:49:40.910 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:49:40.910 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:49:40.910 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:49:40.910 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:49:40.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.882+0000 7f381cff9640 1 -- 172.21.15.42:0/2099506568 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4870 (secure 0 0 0) 0x7f38100b4070 con 0x7f382006f950 2024-09-16T03:49:40.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.882+0000 7f381cff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:49:40.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.882+0000 7f381cff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:49:40.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.882+0000 7f381cff9640 1 -- 172.21.15.42:0/2099506568 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f382006f950 msgr2=0x7f382006fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:40.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.882+0000 7f381cff9640 1 --2- 172.21.15.42:0/2099506568 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f382006f950 0x7f382006fd60 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f3810009e30 tx=0x7f3810009520 comp rx=0 tx=0).stop 2024-09-16T03:49:40.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.882+0000 7f381cff9640 1 -- 172.21.15.42:0/2099506568 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f382006f950 msgr2=0x7f382006fd60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:49:40.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.882+0000 7f381cff9640 1 --2- 172.21.15.42:0/2099506568 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f382006f950 0x7f382006fd60 secure :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f3810009e30 tx=0x7f3810009520 comp rx=0 tx=0).stop 2024-09-16T03:49:40.911 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:49:40.911 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:49:40.911 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:49:40.912 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "leader", 2024-09-16T03:49:40.912 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 70, 2024-09-16T03:49:40.912 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 171674, 2024-09-16T03:49:40.912 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:49:40.912 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:49:40.912 INFO:teuthology.orchestra.run.smithi042.stdout: 1 2024-09-16T03:49:40.912 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:49:40.912 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 0, 2024-09-16T03:49:40.912 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:49:40.912 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:49:40.912 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:49:40.912 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:49:40.912 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:49:40.913 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:49:40.913 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:49:40.913 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:49:40.913 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:49:40.913 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:49:40.913 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:49:40.913 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:49:40.913 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:49:40.913 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:49:40.913 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:49:40.913 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:49:40.913 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:49:40.913 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:49:40.913 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:49:40.914 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:49:40.914 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:49:40.914 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:49:40.914 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:49:40.914 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:49:40.914 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:49:40.914 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:49:40.914 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:49:40.914 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:49:40.914 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:49:40.914 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:49:40.914 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:49:40.914 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:49:40.915 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:49:40.915 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:49:40.915 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:49:40.915 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:49:40.915 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:49:40.915 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:49:40.915 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:49:40.915 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:49:40.915 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:49:40.915 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:49:40.915 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:49:40.915 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:49:40.916 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:49:40.916 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:49:40.916 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:49:40.916 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:49:40.916 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:49:40.916 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:49:40.916 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:49:40.916 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:49:40.916 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:49:40.916 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:49:40.916 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:49:40.916 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:49:40.917 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:49:40.917 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:49:40.917 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:49:40.917 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:49:40.917 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:49:40.917 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:49:40.917 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:49:40.917 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:49:40.917 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:49:40.917 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:49:40.917 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:49:40.917 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:49:40.918 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:49:40.918 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:49:40.918 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:49:40.918 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:49:40.918 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:49:40.918 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:49:40.918 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:49:40.918 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:49:40.918 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:49:40.918 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:49:40.918 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:49:40.918 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:49:40.918 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:49:40.918 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:49:40.919 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:49:40.919 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:49:40.919 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:49:40.919 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:49:40.919 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:49:40.919 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:49:40.919 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:49:40.919 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:49:40.919 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:49:40.919 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:49:40.919 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:49:40.919 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:49:40.919 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:49:40.920 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:49:40.920 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:49:40.920 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:49:40.920 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:49:40.920 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:49:40.920 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:49:40.920 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:49:40.920 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:49:40.920 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:49:40.920 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:49:40.921 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:49:40.921 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:49:40.921 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:49:40.921 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:49:40.921 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:49:40.921 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:49:40.921 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:49:40.921 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:49:40.921 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:49:40.921 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:49:40.921 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:49:40.921 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:49:40.921 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:49:40.922 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:49:40.922 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:49:40.922 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:49:40.922 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:49:40.922 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:49:40.922 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:49:40.922 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:49:40.922 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:49:40.922 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:49:40.922 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:49:40.922 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:49:40.922 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:49:40.922 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:49:40.922 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:49:40.923 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:49:40.923 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:49:40.923 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:49:40.923 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:49:40.923 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:49:40.923 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:49:40.923 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:49:40.923 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:49:40.923 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:49:40.923 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T03:49:40.923 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:49:40.923 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:49:40.923 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:49:40.923 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 6 2024-09-16T03:49:40.924 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:49:40.924 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:49:40.924 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:49:40.924 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:49:40.924 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:49:40.924 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:49:40.924 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 9 2024-09-16T03:49:40.924 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:49:40.924 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:49:40.924 INFO:teuthology.orchestra.run.smithi042.stdout: "mgr": [ 2024-09-16T03:49:40.924 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:49:40.924 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:49:40.925 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:49:40.925 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:49:40.925 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:49:40.925 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:49:40.925 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:49:40.925 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:49:40.925 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:49:40.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.882+0000 7f3826382640 10 librados: watch_flush enter 2024-09-16T03:49:40.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.882+0000 7f3826382640 10 librados: watch_flush exit 2024-09-16T03:49:40.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.882+0000 7f3826382640 20 client.764697.objecter shutdown clearing up homeless session... 2024-09-16T03:49:40.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.882+0000 7f3826382640 10 client.764697.objecter successfully canceled tick 2024-09-16T03:49:40.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.882+0000 7f3826382640 1 -- 172.21.15.42:0/2099506568 >> v2:172.21.15.154:6812/254845342 conn(0x7f38040377c0 msgr2=0x7f3804039c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:40.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.882+0000 7f3826382640 1 --2- 172.21.15.42:0/2099506568 >> v2:172.21.15.154:6812/254845342 conn(0x7f38040377c0 0x7f3804039c80 secure :-1 s=READY pgs=3522 cs=0 l=1 rev1=1 crypto rx=0x7f382006c630 tx=0x7f3808008040 comp rx=0 tx=0).stop 2024-09-16T03:49:40.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.882+0000 7f3826382640 10 monclient: shutdown 2024-09-16T03:49:40.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.882+0000 7f3826382640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:49:40.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.882+0000 7f3826382640 1 -- 172.21.15.42:0/2099506568 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f38201c9be0 msgr2=0x7f38201c9fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:40.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.882+0000 7f3826382640 1 --2- 172.21.15.42:0/2099506568 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f38201c9be0 0x7f38201c9fc0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f381800f410 tx=0x7f381800f8d0 comp rx=0 tx=0).stop 2024-09-16T03:49:40.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.882+0000 7f382491f640 1 -- 172.21.15.42:0/2099506568 reap_dead start 2024-09-16T03:49:40.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.882+0000 7f382491f640 1 -- 172.21.15.42:0/2099506568 reap_dead start 2024-09-16T03:49:40.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.882+0000 7f3826382640 1 -- 172.21.15.42:0/2099506568 shutdown_connections 2024-09-16T03:49:40.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.882+0000 7f3826382640 1 -- 172.21.15.42:0/2099506568 >> 172.21.15.42:0/2099506568 conn(0x7f38200a2420 msgr2=0x7f38200a46b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:49:40.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.882+0000 7f3826382640 1 -- 172.21.15.42:0/2099506568 shutdown_connections 2024-09-16T03:49:40.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.882+0000 7f3826382640 1 -- 172.21.15.42:0/2099506568 wait complete. 2024-09-16T03:49:40.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:40.882+0000 7f3826382640 1 librados: shutdown 2024-09-16T03:49:40.928 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T03:49:40.929 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-16T03:49:40.929 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T03:49:41.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.042+0000 7fcf13f79640 10 monclient: get_monmap_and_config 2024-09-16T03:49:41.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.042+0000 7fcf13f79640 10 monclient: build_initial_monmap 2024-09-16T03:49:41.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.042+0000 7fcf13f79640 10 monclient: monmap: 2024-09-16T03:49:41.046 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:49:41.046 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:49:41.046 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:49:41.044923+0000 2024-09-16T03:49:41.046 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:49:41.044923+0000 2024-09-16T03:49:41.046 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:49:41.046 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:49:41.046 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:49:41.046 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:49:41.047 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:49:41.047 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:49:41.047 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:49:41.047 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:49:41.048 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.042+0000 7fcf13f79640 1 Processor -- start 2024-09-16T03:49:41.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.042+0000 7fcf13f79640 1 -- start start 2024-09-16T03:49:41.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.042+0000 7fcf13f79640 10 monclient: init 2024-09-16T03:49:41.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.042+0000 7fcf13f79640 10 monclient: _reopen_session rank -1 2024-09-16T03:49:41.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.042+0000 7fcf13f79640 10 monclient: _add_conns ranks=[3,2,0,4,1] 2024-09-16T03:49:41.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.042+0000 7fcf13f79640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fcf0c06db90 0x7fcf0c1340c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:41.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.042+0000 7fcf13f79640 10 monclient(hunting): picked mon.noname-a con 0x7fcf0c06db90 addr v2:172.21.15.42:3300/0 2024-09-16T03:49:41.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.042+0000 7fcf13f79640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fcf0c0a37f0 0x7fcf0c0a3c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:41.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.042+0000 7fcf13f79640 10 monclient(hunting): picked mon.noname-b con 0x7fcf0c0a37f0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:49:41.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.042+0000 7fcf13f79640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fcf0c136e10 addr v1:172.21.15.42:6789/0 2024-09-16T03:49:41.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.042+0000 7fcf13f79640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcf0c06ece0 con 0x7fcf0c136e10 2024-09-16T03:49:41.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.042+0000 7fcf13f79640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:41.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.042+0000 7fcf13f79640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7fcf0c06df70 con 0x7fcf0c06db90 2024-09-16T03:49:41.050 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.042+0000 7fcf13f79640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:41.050 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.042+0000 7fcf13f79640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fcf0c09bf80 con 0x7fcf0c0a37f0 2024-09-16T03:49:41.050 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.042+0000 7fcf13f79640 10 monclient(hunting): _renew_subs 2024-09-16T03:49:41.050 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.042+0000 7fcf13f79640 10 monclient(hunting): authenticate will time out at 1428.014990s 2024-09-16T03:49:41.050 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.042+0000 7fcf11514640 1 -- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fcf0c0a37f0 msgr2=0x7fcf0c0a3c00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T03:49:41.050 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.042+0000 7fcf11514640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fcf0c0a37f0 0x7fcf0c0a3c00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:49:41.050 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.042+0000 7fcf11d15640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fcf0c06db90 0x7fcf0c1340c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:41.050 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.042+0000 7fcf12516640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fcf0c136e10 0x7fcf0c137240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:43866/0 (socket says 172.21.15.42:43866) 2024-09-16T03:49:41.050 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.042+0000 7fcf11d15640 10 monclient(hunting): get_auth_request con 0x7fcf0c06db90 auth_method 0 2024-09-16T03:49:41.050 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.042+0000 7fcf11d15640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:41.050 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.042+0000 7fcf11d15640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:41.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.042+0000 7fcf11d15640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:41.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.042+0000 7fcf12516640 1 -- 172.21.15.42:0/4118644335 learned_addr learned my addr 172.21.15.42:0/4118644335 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:49:41.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.042+0000 7fcf11d15640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:41.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.042+0000 7fcf11d15640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:41.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.042+0000 7fcf11d15640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:41.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.042+0000 7fcf11d15640 10 monclient(hunting): handle_auth_done global_id 764706 payload 274 2024-09-16T03:49:41.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.042+0000 7fcf11d15640 1 -- 172.21.15.42:0/4118644335 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fcf0c0a37f0 msgr2=0x7fcf0c0a3c00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:49:41.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.042+0000 7fcf11d15640 1 --2- 172.21.15.42:0/4118644335 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fcf0c0a37f0 0x7fcf0c0a3c00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:41.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.042+0000 7fcf11d15640 1 -- 172.21.15.42:0/4118644335 >> v1:172.21.15.42:6789/0 conn(0x7fcf0c136e10 legacy=0x7fcf0c137240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:41.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.042+0000 7fcf11d15640 10 monclient: _finish_hunting 0 2024-09-16T03:49:41.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.042+0000 7fcf11d15640 1 monclient: found mon.noname-a 2024-09-16T03:49:41.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.042+0000 7fcf11d15640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:49:41.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.042+0000 7fcf11d15640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:49:41.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.042+0000 7fcf11d15640 1 -- 172.21.15.42:0/4118644335 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf0c134780 con 0x7fcf0c06db90 2024-09-16T03:49:41.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.042+0000 7fcf10d13640 1 -- 172.21.15.42:0/4118644335 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2052008361 0 0) 0x7fcf0c06ece0 con 0x7fcf0c136e10 2024-09-16T03:49:41.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.042+0000 7fcf10d13640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T03:49:41.060 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.046+0000 7fcf11d15640 1 --2- 172.21.15.42:0/4118644335 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcf0c06db90 0x7fcf0c1340c0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fcf080097c0 tx=0x7fcf0801c380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79bb3f7a019d8c60 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:41.060 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.046+0000 7fcf10d13640 1 -- 172.21.15.42:0/4118644335 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf080b0070 con 0x7fcf0c06db90 2024-09-16T03:49:41.060 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.046+0000 7fcf10d13640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:41.060 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.046+0000 7fcf10d13640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:49:41.060 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.046+0000 7fcf10d13640 10 monclient: dump: 2024-09-16T03:49:41.061 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:49:41.061 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:49:41.061 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:41.061 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:41.061 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:41.061 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:49:41.061 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:49:41.061 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:49:41.061 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:49:41.061 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:49:41.061 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.046+0000 7fcf10d13640 10 monclient: _finish_auth 0 2024-09-16T03:49:41.061 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.046+0000 7fcf10d13640 10 monclient: _check_auth_tickets 2024-09-16T03:49:41.061 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.046+0000 7fcf10d13640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:49:41.061 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.046+0000 7fcf10d13640 1 -- 172.21.15.42:0/4118644335 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcf080ab600 con 0x7fcf0c06db90 2024-09-16T03:49:41.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.046+0000 7fcf13f79640 5 monclient: authenticate success, global_id 764706 2024-09-16T03:49:41.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.046+0000 7fcf13f79640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:49:41.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.046+0000 7fcf10d13640 10 monclient: handle_config config(0 keys) 2024-09-16T03:49:41.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.046+0000 7fcf10d13640 1 -- 172.21.15.42:0/4118644335 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf080ab9d0 con 0x7fcf0c06db90 2024-09-16T03:49:41.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.046+0000 7fcf13f79640 10 monclient: get_monmap_and_config success 2024-09-16T03:49:41.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.046+0000 7fcf13f79640 4 set_mon_vals no callback set 2024-09-16T03:49:41.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.046+0000 7fcf10d13640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:41.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.046+0000 7fcf10d13640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:49:41.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.046+0000 7fcf10d13640 10 monclient: dump: 2024-09-16T03:49:41.062 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:49:41.062 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:49:41.062 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:41.062 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:41.063 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:41.063 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:49:41.063 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:49:41.063 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:49:41.063 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:49:41.063 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:49:41.063 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.046+0000 7fcf13f79640 10 monclient: shutdown 2024-09-16T03:49:41.063 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.046+0000 7fcf13f79640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:49:41.063 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.046+0000 7fcf13f79640 1 -- 172.21.15.42:0/4118644335 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcf0c06db90 msgr2=0x7fcf0c1340c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:41.063 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.046+0000 7fcf13f79640 1 --2- 172.21.15.42:0/4118644335 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcf0c06db90 0x7fcf0c1340c0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fcf080097c0 tx=0x7fcf0801c380 comp rx=0 tx=0).stop 2024-09-16T03:49:41.063 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.046+0000 7fcf13f79640 1 -- 172.21.15.42:0/4118644335 shutdown_connections 2024-09-16T03:49:41.064 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.046+0000 7fcf13f79640 1 --2- 172.21.15.42:0/4118644335 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fcf0c0a37f0 0x7fcf0c0a3c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:41.064 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.046+0000 7fcf13f79640 1 --2- 172.21.15.42:0/4118644335 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcf0c06db90 0x7fcf0c1340c0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:41.064 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.046+0000 7fcf13f79640 1 -- 172.21.15.42:0/4118644335 >> 172.21.15.42:0/4118644335 conn(0x7fcf0c0a9fb0 msgr2=0x7fcf0c0aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:49:41.064 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.046+0000 7fcf13f79640 1 -- 172.21.15.42:0/4118644335 shutdown_connections 2024-09-16T03:49:41.064 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.046+0000 7fcf13f79640 1 -- 172.21.15.42:0/4118644335 wait complete. 2024-09-16T03:49:41.064 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.046+0000 7fcf13f79640 10 monclient: build_initial_monmap 2024-09-16T03:49:41.064 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.046+0000 7fcf13f79640 10 monclient: monmap: 2024-09-16T03:49:41.064 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:49:41.064 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:49:41.064 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:49:41.064 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:49:41.064 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:49:41.064 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:49:41.065 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:49:41.065 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:49:41.065 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:49:41.065 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:49:41.065 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.046+0000 7fcf13f79640 1 librados: starting msgr at 2024-09-16T03:49:41.065 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.046+0000 7fcf13f79640 1 librados: starting objecter 2024-09-16T03:49:41.065 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.046+0000 7fcf13f79640 1 Processor -- start 2024-09-16T03:49:41.065 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.046+0000 7fcf13f79640 1 -- start start 2024-09-16T03:49:41.065 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.046+0000 7fcf13f79640 1 librados: setting wanted keys 2024-09-16T03:49:41.065 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.046+0000 7fcf13f79640 1 librados: calling monclient init 2024-09-16T03:49:41.065 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.046+0000 7fcf13f79640 10 monclient: init 2024-09-16T03:49:41.065 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.050+0000 7fcf13f79640 10 monclient: _reopen_session rank -1 2024-09-16T03:49:41.065 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.050+0000 7fcf13f79640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:49:41.065 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.050+0000 7fcf13f79640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcf0c06db90 0x7fcf0c1cf500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:41.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.050+0000 7fcf13f79640 10 monclient(hunting): picked mon.noname-b con 0x7fcf0c06db90 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:49:41.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.050+0000 7fcf13f79640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcf0c0a37f0 0x7fcf0c0a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:41.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.050+0000 7fcf13f79640 10 monclient(hunting): picked mon.noname-a con 0x7fcf0c0a37f0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:49:41.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.050+0000 7fcf13f79640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fcf0c136e10 0x7fcf0c0a3200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:41.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.050+0000 7fcf13f79640 10 monclient(hunting): picked mon.noname-c con 0x7fcf0c136e10 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:49:41.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.050+0000 7fcf13f79640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:41.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.050+0000 7fcf13f79640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fcf0c141e40 con 0x7fcf0c0a37f0 2024-09-16T03:49:41.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.050+0000 7fcf13f79640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:41.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.050+0000 7fcf13f79640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fcf0c141cc0 con 0x7fcf0c06db90 2024-09-16T03:49:41.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.050+0000 7fcf13f79640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:41.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.050+0000 7fcf13f79640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fcf0c141fc0 con 0x7fcf0c136e10 2024-09-16T03:49:41.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.050+0000 7fcf13f79640 10 monclient(hunting): _renew_subs 2024-09-16T03:49:41.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.050+0000 7fcf13f79640 10 monclient(hunting): authenticate will time out at 1428.020222s 2024-09-16T03:49:41.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.050+0000 7fcf12516640 1 -- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fcf0c136e10 msgr2=0x7fcf0c0a3200 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T03:49:41.067 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.050+0000 7fcf12516640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fcf0c136e10 0x7fcf0c0a3200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:49:41.067 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.050+0000 7fcf11514640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcf0c0a37f0 0x7fcf0c0a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:41.067 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.050+0000 7fcf11514640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcf0c0a37f0 0x7fcf0c0a2cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:32930/0 (socket says 172.21.15.42:32930) 2024-09-16T03:49:41.067 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.050+0000 7fcf11514640 1 -- 172.21.15.42:0/902680873 learned_addr learned my addr 172.21.15.42:0/902680873 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:49:41.067 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.050+0000 7fcf11514640 10 monclient(hunting): get_auth_request con 0x7fcf0c0a37f0 auth_method 0 2024-09-16T03:49:41.067 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.050+0000 7fcf11514640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:41.067 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.050+0000 7fcf11514640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:41.067 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.050+0000 7fcf11514640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:41.067 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.050+0000 7fcf11514640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:41.067 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.050+0000 7fcf11514640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:41.067 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.050+0000 7fcf11514640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:41.067 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.050+0000 7fcf11514640 10 monclient(hunting): handle_auth_done global_id 764712 payload 824 2024-09-16T03:49:41.068 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.050+0000 7fcf11514640 1 -- 172.21.15.42:0/902680873 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fcf0c136e10 msgr2=0x7fcf0c0a3200 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:49:41.068 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.050+0000 7fcf11d15640 1 --2- 172.21.15.42:0/902680873 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcf0c06db90 0x7fcf0c1cf500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:41.068 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.050+0000 7fcf11514640 1 --2- 172.21.15.42:0/902680873 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fcf0c136e10 0x7fcf0c0a3200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:41.068 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.050+0000 7fcf11514640 1 -- 172.21.15.42:0/902680873 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcf0c06db90 msgr2=0x7fcf0c1cf500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:41.068 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.050+0000 7fcf11514640 1 --2- 172.21.15.42:0/902680873 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcf0c06db90 0x7fcf0c1cf500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:41.068 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.050+0000 7fcf11514640 10 monclient: _finish_hunting 0 2024-09-16T03:49:41.068 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.050+0000 7fcf11514640 1 monclient: found mon.noname-a 2024-09-16T03:49:41.068 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.050+0000 7fcf11514640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:49:41.068 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.050+0000 7fcf11514640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:49:41.068 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.050+0000 7fcf11514640 1 -- 172.21.15.42:0/902680873 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf0c132c80 con 0x7fcf0c0a37f0 2024-09-16T03:49:41.068 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.050+0000 7fcf11514640 1 --2- 172.21.15.42:0/902680873 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcf0c0a37f0 0x7fcf0c0a2cc0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fcefc00b550 tx=0x7fcefc00ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:41.069 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.050+0000 7fcf027fc640 10 client.?.objecter ms_handle_connect 0x7fcf0c0a37f0 2024-09-16T03:49:41.069 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.050+0000 7fcf027fc640 10 client.?.objecter resend_mon_ops 2024-09-16T03:49:41.069 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.050+0000 7fcf027fc640 1 -- 172.21.15.42:0/902680873 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcefc003dd0 con 0x7fcf0c0a37f0 2024-09-16T03:49:41.069 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.050+0000 7fcf027fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:41.069 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.050+0000 7fcf027fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:49:41.069 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.050+0000 7fcf027fc640 10 monclient: dump: 2024-09-16T03:49:41.069 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:49:41.069 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:49:41.069 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:41.069 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:41.069 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:41.069 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:49:41.069 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:49:41.070 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:49:41.070 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:49:41.070 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:49:41.070 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.050+0000 7fcf027fc640 10 monclient: _finish_auth 0 2024-09-16T03:49:41.070 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.050+0000 7fcf027fc640 10 monclient: _check_auth_tickets 2024-09-16T03:49:41.070 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.050+0000 7fcf027fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:49:41.070 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.050+0000 7fcf027fc640 1 -- 172.21.15.42:0/902680873 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcefc015070 con 0x7fcf0c0a37f0 2024-09-16T03:49:41.070 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.050+0000 7fcf027fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:49:41.070 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.050+0000 7fcf027fc640 1 -- 172.21.15.42:0/902680873 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcefc005330 con 0x7fcf0c0a37f0 2024-09-16T03:49:41.070 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.050+0000 7fcf13f79640 5 monclient: authenticate success, global_id 764712 2024-09-16T03:49:41.070 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.050+0000 7fcf13f79640 10 monclient: _renew_subs 2024-09-16T03:49:41.070 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.050+0000 7fcf13f79640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:49:41.070 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.050+0000 7fcf13f79640 1 -- 172.21.15.42:0/902680873 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf0c09e480 con 0x7fcf0c0a37f0 2024-09-16T03:49:41.070 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.050+0000 7fcf03fff640 4 set_mon_vals no callback set 2024-09-16T03:49:41.071 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.050+0000 7fcf027fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:41.071 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.050+0000 7fcf027fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:49:41.071 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.050+0000 7fcf027fc640 10 monclient: dump: 2024-09-16T03:49:41.071 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:49:41.071 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:49:41.071 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:41.071 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:41.071 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:41.071 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:49:41.071 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:49:41.071 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:49:41.071 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:49:41.071 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:49:41.072 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.050+0000 7fcf13f79640 10 client.764712.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:49:41.072 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.050+0000 7fcf13f79640 10 monclient: _renew_subs 2024-09-16T03:49:41.072 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.050+0000 7fcf13f79640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:49:41.072 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.050+0000 7fcf13f79640 1 -- 172.21.15.42:0/902680873 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf0c133460 con 0x7fcf0c0a37f0 2024-09-16T03:49:41.072 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.050+0000 7fcf13f79640 1 librados: init done 2024-09-16T03:49:41.072 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.050+0000 7fcf13f79640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:49:41.072 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.050+0000 7fcf13f79640 1 --2- 172.21.15.42:0/902680873 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcf0c13d7b0 0x7fcf0c02c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:41.072 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.050+0000 7fcf13f79640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:41.072 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.050+0000 7fcf13f79640 1 -- 172.21.15.42:0/902680873 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fcf0c06f700 con 0x7fcf0c13d7b0 2024-09-16T03:49:41.072 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.050+0000 7fcf027fc640 1 -- 172.21.15.42:0/902680873 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fcefc0054d0 con 0x7fcf0c0a37f0 2024-09-16T03:49:41.072 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.050+0000 7fcf027fc640 1 --2- 172.21.15.42:0/902680873 >> v2:172.21.15.154:6812/254845342 conn(0x7fcef00377c0 0x7fcef0039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:41.072 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.050+0000 7fcf11d15640 1 --2- 172.21.15.42:0/902680873 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcf0c13d7b0 0x7fcf0c02c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:41.072 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.050+0000 7fcf027fc640 1 -- 172.21.15.42:0/902680873 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(248..248 src has 1..248) ==== 5807+0+0 (secure 0 0 0) 0x7fcefc048ea0 con 0x7fcf0c0a37f0 2024-09-16T03:49:41.072 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.050+0000 7fcf027fc640 10 client.764712.objecter ms_dispatch 0x7fcf0c000ba0 osd_map(248..248 src has 1..248) 2024-09-16T03:49:41.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.050+0000 7fcf11d15640 10 monclient: get_auth_request con 0x7fcf0c13d7b0 auth_method 0 2024-09-16T03:49:41.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.050+0000 7fcf027fc640 3 client.764712.objecter handle_osd_map got epochs [248,248] > 0 2024-09-16T03:49:41.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.050+0000 7fcf027fc640 3 client.764712.objecter handle_osd_map decoding full epoch 248 2024-09-16T03:49:41.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.050+0000 7fcf11d15640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:41.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.050+0000 7fcf11d15640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:41.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.050+0000 7fcf11d15640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:41.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.050+0000 7fcf027fc640 20 client.764712.objecter dump_active .. 0 homeless 2024-09-16T03:49:41.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.050+0000 7fcf11d15640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:41.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.050+0000 7fcf11d15640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:41.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.050+0000 7fcf11d15640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:41.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.050+0000 7fcf12516640 1 --2- 172.21.15.42:0/902680873 >> v2:172.21.15.154:6812/254845342 conn(0x7fcef00377c0 0x7fcef0039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:41.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.050+0000 7fcf12516640 10 monclient: get_auth_request con 0x7fcef00377c0 auth_method 0 2024-09-16T03:49:41.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.050+0000 7fcf11d15640 10 monclient(hunting): handle_auth_done global_id 794161 payload 824 2024-09-16T03:49:41.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.050+0000 7fcf11d15640 1 -- 172.21.15.42:0/902680873 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fcf0c0a5290 con 0x7fcf0c13d7b0 2024-09-16T03:49:41.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.050+0000 7fcf027fc640 10 client.764712.objecter ms_handle_connect 0x7fcf0c13d7b0 2024-09-16T03:49:41.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.050+0000 7fcf027fc640 10 client.764712.objecter resend_mon_ops 2024-09-16T03:49:41.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.050+0000 7fcf11d15640 1 --2- 172.21.15.42:0/902680873 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcf0c13d7b0 0x7fcf0c02c650 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fcf0c131410 tx=0x7fcf0801abc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:41.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.050+0000 7fcf027fc640 10 client.764712.objecter ms_handle_connect 0x7fcef00377c0 2024-09-16T03:49:41.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.050+0000 7fcf12516640 1 --2- 172.21.15.42:0/902680873 >> v2:172.21.15.154:6812/254845342 conn(0x7fcef00377c0 0x7fcef0039c80 secure :-1 s=READY pgs=3523 cs=0 l=1 rev1=1 crypto rx=0x7fcf04001470 tx=0x7fcf04007cd0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:41.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.050+0000 7fcf027fc640 1 -- 172.21.15.42:0/902680873 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf080bb030 con 0x7fcf0c13d7b0 2024-09-16T03:49:41.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.050+0000 7fcf027fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:41.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.050+0000 7fcf027fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:49:41.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.050+0000 7fcf027fc640 10 monclient: dump: 2024-09-16T03:49:41.074 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:49:41.074 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:49:41.074 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:41.074 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:41.074 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:41.075 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:49:41.075 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:49:41.075 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:49:41.075 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:49:41.075 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:49:41.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.054+0000 7fcf027fc640 1 -- 172.21.15.42:0/902680873 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fcf080abb20 con 0x7fcf0c13d7b0 2024-09-16T03:49:41.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.054+0000 7fcf027fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:49:41.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.054+0000 7fcf027fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:49:41.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.054+0000 7fcf027fc640 1 -- 172.21.15.42:0/902680873 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcf0c13d7b0 msgr2=0x7fcf0c02c650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:41.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.054+0000 7fcf027fc640 1 --2- 172.21.15.42:0/902680873 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcf0c13d7b0 0x7fcf0c02c650 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fcf0c131410 tx=0x7fcf0801abc0 comp rx=0 tx=0).stop 2024-09-16T03:49:41.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.054+0000 7fcf027fc640 1 -- 172.21.15.42:0/902680873 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcf0c13d7b0 msgr2=0x7fcf0c02c650 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:49:41.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.054+0000 7fcf027fc640 1 --2- 172.21.15.42:0/902680873 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcf0c13d7b0 0x7fcf0c02c650 secure :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fcf0c131410 tx=0x7fcf0801abc0 comp rx=0 tx=0).stop 2024-09-16T03:49:41.077 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.058+0000 7fcf13f79640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:49:41.077 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.058+0000 7fcf13f79640 1 --2- 172.21.15.42:0/902680873 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcf0c13db90 0x7fcf0c02cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:49:41.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.058+0000 7fcf13f79640 10 monclient(hunting): start opening mon connection 2024-09-16T03:49:41.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.058+0000 7fcf13f79640 1 -- 172.21.15.42:0/902680873 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fcf0c1340c0 con 0x7fcf0c13db90 2024-09-16T03:49:41.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.058+0000 7fcf11d15640 1 --2- 172.21.15.42:0/902680873 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcf0c13db90 0x7fcf0c02cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:49:41.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.058+0000 7fcf11d15640 10 monclient: get_auth_request con 0x7fcf0c13db90 auth_method 0 2024-09-16T03:49:41.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.058+0000 7fcf11d15640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:49:41.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.058+0000 7fcf11d15640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:49:41.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.058+0000 7fcf11d15640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:49:41.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.058+0000 7fcf11d15640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:49:41.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.058+0000 7fcf11d15640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:49:41.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.058+0000 7fcf11d15640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:49:41.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.058+0000 7fcf11d15640 10 monclient(hunting): handle_auth_done global_id 794164 payload 824 2024-09-16T03:49:41.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.058+0000 7fcf11d15640 1 -- 172.21.15.42:0/902680873 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fcf0c139530 con 0x7fcf0c13db90 2024-09-16T03:49:41.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.058+0000 7fcf11d15640 1 --2- 172.21.15.42:0/902680873 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcf0c13db90 0x7fcf0c02cb90 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fcf080097c0 tx=0x7fcf08019120 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:49:41.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.058+0000 7fcf027fc640 10 client.764712.objecter ms_handle_connect 0x7fcf0c13db90 2024-09-16T03:49:41.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.058+0000 7fcf027fc640 10 client.764712.objecter resend_mon_ops 2024-09-16T03:49:41.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.062+0000 7fcf027fc640 1 -- 172.21.15.42:0/902680873 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf080bb040 con 0x7fcf0c13db90 2024-09-16T03:49:41.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.062+0000 7fcf027fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:49:41.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.062+0000 7fcf027fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:49:41.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.062+0000 7fcf027fc640 10 monclient: dump: 2024-09-16T03:49:41.079 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:49:41.079 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:49:41.079 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:41.079 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:49:41.079 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:49:41.079 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:49:41.080 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:49:41.080 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:49:41.080 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:49:41.080 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:49:41.080 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.062+0000 7fcf027fc640 1 -- 172.21.15.42:0/902680873 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4533 (secure 0 0 0) 0x7fcf080c1c20 con 0x7fcf0c13db90 2024-09-16T03:49:41.080 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.062+0000 7fcf027fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:49:41.080 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.062+0000 7fcf027fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:49:41.080 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.062+0000 7fcf027fc640 1 -- 172.21.15.42:0/902680873 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcf0c13db90 msgr2=0x7fcf0c02cb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:41.080 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.062+0000 7fcf027fc640 1 --2- 172.21.15.42:0/902680873 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcf0c13db90 0x7fcf0c02cb90 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fcf080097c0 tx=0x7fcf08019120 comp rx=0 tx=0).stop 2024-09-16T03:49:41.080 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.062+0000 7fcf027fc640 1 -- 172.21.15.42:0/902680873 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcf0c13db90 msgr2=0x7fcf0c02cb90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:49:41.080 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.062+0000 7fcf027fc640 1 --2- 172.21.15.42:0/902680873 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcf0c13db90 0x7fcf0c02cb90 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:41.080 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:49:41.081 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:49:41.081 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:49:41.081 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T03:49:41.081 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 70, 2024-09-16T03:49:41.081 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 22899, 2024-09-16T03:49:41.081 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:49:41.081 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:49:41.081 INFO:teuthology.orchestra.run.smithi042.stdout: 1 2024-09-16T03:49:41.081 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:49:41.081 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 0, 2024-09-16T03:49:41.081 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:49:41.081 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:49:41.082 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:49:41.082 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:49:41.082 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:49:41.082 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:49:41.082 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:49:41.082 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:49:41.082 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:49:41.082 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:49:41.082 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:49:41.082 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:49:41.082 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:49:41.082 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:49:41.082 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:49:41.083 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:49:41.083 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:49:41.083 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:49:41.083 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:49:41.083 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:49:41.083 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:49:41.083 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:49:41.083 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:49:41.083 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:49:41.083 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:49:41.083 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:49:41.083 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:49:41.084 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:49:41.084 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:49:41.084 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:49:41.084 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:49:41.084 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:49:41.084 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:49:41.084 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:49:41.084 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:49:41.084 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:49:41.084 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:49:41.084 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:49:41.084 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:49:41.085 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:49:41.085 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:49:41.085 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:49:41.085 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:49:41.085 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:49:41.085 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:49:41.085 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:49:41.085 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:49:41.085 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:49:41.085 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:49:41.085 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:49:41.085 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:49:41.085 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:49:41.085 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:49:41.086 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:49:41.086 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:49:41.086 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:49:41.086 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:49:41.086 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:49:41.086 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:49:41.086 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:49:41.086 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:49:41.086 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:49:41.086 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:49:41.086 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:49:41.086 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:49:41.086 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:49:41.086 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:49:41.087 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:49:41.087 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:49:41.087 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:49:41.087 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:49:41.087 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:49:41.087 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:49:41.087 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:49:41.087 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:49:41.087 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:49:41.087 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:49:41.087 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:49:41.087 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:49:41.087 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:49:41.087 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:49:41.087 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:49:41.088 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:49:41.088 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:49:41.088 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:49:41.088 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:49:41.088 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:49:41.088 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:49:41.088 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:49:41.088 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:49:41.088 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:49:41.088 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:49:41.088 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:49:41.088 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:49:41.088 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:49:41.089 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:49:41.089 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:49:41.089 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:49:41.089 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:49:41.089 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:49:41.089 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:49:41.089 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:49:41.089 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:49:41.089 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:49:41.089 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:49:41.089 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:49:41.089 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:49:41.089 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:49:41.090 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:49:41.090 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:49:41.090 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:49:41.090 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:49:41.090 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:49:41.090 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:49:41.090 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:49:41.090 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:49:41.090 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:49:41.090 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:49:41.090 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:49:41.090 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:49:41.090 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:49:41.090 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:49:41.091 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:49:41.091 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:49:41.091 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:49:41.091 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:49:41.091 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:49:41.091 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:49:41.091 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:49:41.091 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:49:41.091 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:49:41.091 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:49:41.091 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:49:41.091 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:49:41.091 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:49:41.092 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:49:41.092 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:49:41.092 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:49:41.092 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:49:41.092 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:49:41.092 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:49:41.092 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:49:41.092 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:49:41.092 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:49:41.092 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:49:41.092 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:49:41.092 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:49:41.092 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:49:41.092 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:49:41.093 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:49:41.093 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:49:41.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.062+0000 7fcf13f79640 10 librados: watch_flush enter 2024-09-16T03:49:41.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.062+0000 7fcf13f79640 10 librados: watch_flush exit 2024-09-16T03:49:41.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.062+0000 7fcf13f79640 20 client.764712.objecter shutdown clearing up homeless session... 2024-09-16T03:49:41.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.062+0000 7fcf13f79640 10 client.764712.objecter successfully canceled tick 2024-09-16T03:49:41.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.062+0000 7fcf13f79640 1 -- 172.21.15.42:0/902680873 >> v2:172.21.15.154:6812/254845342 conn(0x7fcef00377c0 msgr2=0x7fcef0039c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:41.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.062+0000 7fcf13f79640 1 --2- 172.21.15.42:0/902680873 >> v2:172.21.15.154:6812/254845342 conn(0x7fcef00377c0 0x7fcef0039c80 secure :-1 s=READY pgs=3523 cs=0 l=1 rev1=1 crypto rx=0x7fcf04001470 tx=0x7fcf04007cd0 comp rx=0 tx=0).stop 2024-09-16T03:49:41.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.062+0000 7fcf13f79640 10 monclient: shutdown 2024-09-16T03:49:41.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.062+0000 7fcf13f79640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:49:41.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.062+0000 7fcf13f79640 1 -- 172.21.15.42:0/902680873 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcf0c0a37f0 msgr2=0x7fcf0c0a2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:49:41.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.062+0000 7fcf13f79640 1 --2- 172.21.15.42:0/902680873 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcf0c0a37f0 0x7fcf0c0a2cc0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fcefc00b550 tx=0x7fcefc00ba10 comp rx=0 tx=0).stop 2024-09-16T03:49:41.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.062+0000 7fcf12516640 1 -- 172.21.15.42:0/902680873 reap_dead start 2024-09-16T03:49:41.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.062+0000 7fcf13f79640 1 -- 172.21.15.42:0/902680873 shutdown_connections 2024-09-16T03:49:41.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.062+0000 7fcf13f79640 1 --2- 172.21.15.42:0/902680873 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcf0c0a37f0 0x7fcf0c0a2cc0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:49:41.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.062+0000 7fcf13f79640 1 -- 172.21.15.42:0/902680873 >> 172.21.15.42:0/902680873 conn(0x7fcf0c0a9fb0 msgr2=0x7fcf0c132570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:49:41.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.062+0000 7fcf13f79640 1 -- 172.21.15.42:0/902680873 shutdown_connections 2024-09-16T03:49:41.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.062+0000 7fcf13f79640 1 -- 172.21.15.42:0/902680873 wait complete. 2024-09-16T03:49:41.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:49:41.062+0000 7fcf13f79640 1 librados: shutdown 2024-09-16T03:49:41.096 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T03:49:41.096 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-16T03:49:55.326 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:7462dAio.RoundTripAppendPP (3084 ms) 2024-09-16T03:49:55.327 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio_pp: [ RUN ] LibRadosAio.RacingRemovePP 2024-09-16T03:49:55.327 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio_pp: [ OK ] LibRadosAio.RacingRemovePP (3041 ms) 2024-09-16T03:49:55.327 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripCmpExtPP 2024-09-16T03:49:55.327 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripCmpExtPP (21724 ms) 2024-09-16T03:49:55.327 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripCmpExtPP2 2024-09-16T03:49:55.327 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripCmpExtPP2 (2446 ms) 2024-09-16T03:49:55.327 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio_pp: [ RUN ] LibRadosAio.PoolEIOFlag 2024-09-16T03:49:55.327 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio_pp: setting pool EIO 2024-09-16T03:49:55.327 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio_pp: max_success 98, min_failed 99 2024-09-16T03:49:55.327 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio_pp: [ OK ] LibRadosAio.PoolEIOFlag (4041 ms) 2024-09-16T03:49:55.327 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio_pp: [ RUN ] LibRadosAio.MultiReads 2024-09-16T03:49:55.327 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio_pp: [ OK ] LibRadosAio.MultiReads (3005 ms) 2024-09-16T03:49:55.327 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio_pp: [----------] 32 tests from LibRadosAio (190500 ms total) 2024-09-16T03:49:55.328 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio_pp: 2024-09-16T03:49:55.328 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio_pp: [----------] 4 tests from LibRadosAioPP 2024-09-16T03:49:55.328 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.ReadIntoBufferlist 2024-09-16T03:49:55.328 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio_pp: [ OK ] LibRadosAioPP.ReadIntoBufferlist (2960 ms) 2024-09-16T03:49:55.328 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.XattrsRoundTripPP 2024-09-16T03:49:55.328 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio_pp: [ OK ] LibRadosAioPP.XattrsRoundTripPP (20985 ms) 2024-09-16T03:49:55.328 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.RmXattrPP 2024-09-16T03:49:55.328 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio_pp: [ OK ] LibRadosAioPP.RmXattrPP (29266 ms) 2024-09-16T03:49:55.328 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.RemoveTestPP 2024-09-16T03:49:55.328 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio_pp: [ OK ] LibRadosAioPP.RemoveTestPP (3062 ms) 2024-09-16T03:49:55.328 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio_pp: [----------] 4 tests from LibRadosAioPP (56273 ms total) 2024-09-16T03:49:55.328 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio_pp: 2024-09-16T03:49:55.328 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio_pp: [----------] 1 test from LibRadosIoPP 2024-09-16T03:49:55.328 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio_pp: [ RUN ] LibRadosIoPP.XattrListPP 2024-09-16T03:49:55.329 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio_pp: [ OK ] LibRadosIoPP.XattrListPP (2927 ms) 2024-09-16T03:49:55.329 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio_pp: [----------] 1 test from LibRadosIoPP (2927 ms total) 2024-09-16T03:49:55.329 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio_pp: 2024-09-16T03:49:55.329 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio_pp: [----------] 20 tests from LibRadosAioEC 2024-09-16T03:49:55.329 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleWritePP 2024-09-16T03:49:55.329 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleWritePP (28157 ms) 2024-09-16T03:49:55.329 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.WaitForSafePP 2024-09-16T03:49:55.329 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio_pp: [ OK ] LibRadosAioEC.WaitForSafePP (6877 ms) 2024-09-16T03:49:55.329 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP 2024-09-16T03:49:55.329 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP (6986 ms) 2024-09-16T03:49:55.329 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP2 2024-09-16T03:49:55.329 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP2 (7027 ms) 2024-09-16T03:49:55.329 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP3 2024-09-16T03:49:55.329 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP3 (2977 ms) 2024-09-16T03:49:55.330 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripSparseReadPP 2024-09-16T03:49:55.330 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripSparseReadPP (7123 ms) 2024-09-16T03:49:55.330 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripAppendPP 2024-09-16T03:49:55.330 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripAppendPP (20872 ms) 2024-09-16T03:49:55.330 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.IsCompletePP 2024-09-16T03:49:55.330 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio_pp: [ OK ] LibRadosAioEC.IsCompletePP (7029 ms) 2024-09-16T03:49:55.330 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.IsSafePP 2024-09-16T03:49:55.330 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio_pp: [ OK ] LibRadosAioEC.IsSafePP (6991 ms) 2024-09-16T03:49:55.330 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.ReturnValuePP 2024-09-16T03:49:55.330 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio_pp: [ OK ] LibRadosAioEC.ReturnValuePP (6998 ms) 2024-09-16T03:49:55.330 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.FlushPP 2024-09-16T03:49:55.330 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio_pp: [ OK ] LibRadosAioEC.FlushPP (21180 ms) 2024-09-16T03:49:55.330 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.FlushAsyncPP 2024-09-16T03:49:55.330 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio_pp: [ OK ] LibRadosAioEC.FlushAsyncPP (7037 ms) 2024-09-16T03:49:55.331 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripWriteFullPP 2024-09-16T03:49:55.331 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripWriteFullPP (6822 ms) 2024-09-16T03:50:01.096 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-16T03:50:01.097 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-09-16T03:50:01.097 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-16T03:50:01.097 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-16T03:50:01.100 INFO:tasks.ceph.mon.c:Started 2024-09-16T03:50:01.100 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T03:50:01.100 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:50:01.222 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.218+0000 7f1240273640 10 monclient: get_monmap_and_config 2024-09-16T03:50:01.222 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.218+0000 7f1240273640 10 monclient: build_initial_monmap 2024-09-16T03:50:01.222 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.218+0000 7f1240273640 10 monclient: monmap: 2024-09-16T03:50:01.222 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:50:01.222 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:50:01.223 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:50:01.221323+0000 2024-09-16T03:50:01.223 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:50:01.221323+0000 2024-09-16T03:50:01.223 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:50:01.223 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:50:01.223 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:50:01.223 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:50:01.223 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:50:01.223 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:50:01.223 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:50:01.223 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:50:01.233 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.218+0000 7f1240273640 1 Processor -- start 2024-09-16T03:50:01.233 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.218+0000 7f1240273640 1 -- start start 2024-09-16T03:50:01.233 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.218+0000 7f1240273640 10 monclient: init 2024-09-16T03:50:01.233 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.218+0000 7f1240273640 10 monclient: _reopen_session rank -1 2024-09-16T03:50:01.233 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.218+0000 7f1240273640 10 monclient: _add_conns ranks=[0,2,1,4,3] 2024-09-16T03:50:01.233 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.218+0000 7f1240273640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f12380a39b0 addr v1:172.21.15.42:6789/0 2024-09-16T03:50:01.233 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.218+0000 7f1240273640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f12380a3d90 0x7f123813c380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:50:01.233 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.218+0000 7f1240273640 10 monclient(hunting): picked mon.noname-b con 0x7f12380a3d90 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:50:01.233 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.218+0000 7f1240273640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f123813c8c0 addr v1:172.21.15.154:6789/0 2024-09-16T03:50:01.233 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.218+0000 7f1240273640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1238134290 con 0x7f12380a39b0 2024-09-16T03:50:01.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.218+0000 7f1240273640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f123809c140 con 0x7f123813c8c0 2024-09-16T03:50:01.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.218+0000 7f1240273640 10 monclient(hunting): start opening mon connection 2024-09-16T03:50:01.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.218+0000 7f1240273640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f123809c420 con 0x7f12380a3d90 2024-09-16T03:50:01.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.218+0000 7f1240273640 10 monclient(hunting): _renew_subs 2024-09-16T03:50:01.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.218+0000 7f1240273640 10 monclient(hunting): authenticate will time out at 1448.190453s 2024-09-16T03:50:01.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.218+0000 7f123e00f640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f12380a39b0 0x7f1238133b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:38320/0 (socket says 172.21.15.42:38320) 2024-09-16T03:50:01.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.218+0000 7f123e00f640 1 -- 172.21.15.42:0/978117728 learned_addr learned my addr 172.21.15.42:0/978117728 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:50:01.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.218+0000 7f123d80e640 1 --2- 172.21.15.42:0/978117728 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f12380a3d90 0x7f123813c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:50:01.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.218+0000 7f123d80e640 10 monclient(hunting): get_auth_request con 0x7f12380a3d90 auth_method 0 2024-09-16T03:50:01.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.218+0000 7f123d80e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:50:01.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.218+0000 7f123d80e640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:50:01.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.218+0000 7f123d80e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:50:01.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.218+0000 7f123d80e640 1 -- 172.21.15.42:0/978117728 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f12380a3d90 msgr2=0x7f123813c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T03:50:01.235 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.218+0000 7f123d80e640 1 -- 172.21.15.42:0/978117728 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f12380a3d90 msgr2=0x7f123813c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:50:01.235 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.218+0000 7f123d80e640 1 --2- 172.21.15.42:0/978117728 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f12380a3d90 0x7f123813c380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:50:01.235 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.218+0000 7f123d80e640 1 --2- 172.21.15.42:0/978117728 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f12380a3d90 0x7f123813c380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:50:01.235 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.218+0000 7f123d00d640 1 -- 172.21.15.42:0/978117728 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 731283469 0 0) 0x7f1238134290 con 0x7f12380a39b0 2024-09-16T03:50:01.235 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.218+0000 7f123d00d640 10 monclient(hunting): _negotiate 2024-09-16T03:50:01.235 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.218+0000 7f123d00d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:50:01.235 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.218+0000 7f123d00d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:50:01.235 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.218+0000 7f123d00d640 10 monclient(hunting): my global_id is 764733 2024-09-16T03:50:01.235 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.218+0000 7f123d00d640 1 -- 172.21.15.42:0/978117728 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1224003530 con 0x7f12380a39b0 2024-09-16T03:50:01.235 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.218+0000 7f123d00d640 1 -- 172.21.15.42:0/978117728 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3297842373 0 0) 0x7f1224003530 con 0x7f12380a39b0 2024-09-16T03:50:01.235 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.218+0000 7f123d00d640 1 -- 172.21.15.42:0/978117728 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f12380a3d90 msgr2=0x7f123813c380 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:50:01.235 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.218+0000 7f123d00d640 1 --2- 172.21.15.42:0/978117728 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f12380a3d90 0x7f123813c380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:50:01.235 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.218+0000 7f123d00d640 1 -- 172.21.15.42:0/978117728 >> v1:172.21.15.154:6789/0 conn(0x7f123813c8c0 legacy=0x7f123813ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:50:01.235 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.218+0000 7f123d00d640 10 monclient: _finish_hunting 0 2024-09-16T03:50:01.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.218+0000 7f123d00d640 1 monclient: found mon.noname-a-legacy 2024-09-16T03:50:01.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.218+0000 7f123d00d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:50:01.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.218+0000 7f123d00d640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.42:6789/0 2024-09-16T03:50:01.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.218+0000 7f123d00d640 1 -- 172.21.15.42:0/978117728 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f123806dd60 con 0x7f12380a39b0 2024-09-16T03:50:01.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.218+0000 7f123d00d640 10 monclient: _finish_auth 0 2024-09-16T03:50:01.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.218+0000 7f123d00d640 10 monclient: _check_auth_tickets 2024-09-16T03:50:01.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.218+0000 7f123d00d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:50:01.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.218+0000 7f123d00d640 1 -- 172.21.15.42:0/978117728 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2177413734 0 0) 0x7f1234001680 con 0x7f12380a39b0 2024-09-16T03:50:01.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.218+0000 7f1240273640 5 monclient: authenticate success, global_id 764733 2024-09-16T03:50:01.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.218+0000 7f1240273640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:50:01.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.218+0000 7f123d00d640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:50:01.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.218+0000 7f123d00d640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T03:50:01.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.218+0000 7f123d00d640 10 monclient: dump: 2024-09-16T03:50:01.237 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:50:01.237 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:50:01.237 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:01.237 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:01.237 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:50:01.237 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:50:01.237 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:50:01.237 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:50:01.237 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:50:01.237 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:50:01.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.218+0000 7f123d00d640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] but i'm connected to v1:172.21.15.42:6789/0, reconnecting 2024-09-16T03:50:01.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.218+0000 7f123d00d640 10 monclient: _reopen_session rank -1 2024-09-16T03:50:01.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.218+0000 7f123d00d640 1 -- 172.21.15.42:0/978117728 >> v1:172.21.15.42:6789/0 conn(0x7f12380a39b0 legacy=0x7f1238133b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:50:01.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.218+0000 7f123d00d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T03:50:01.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.218+0000 7f123d00d640 1 --2- 172.21.15.42:0/978117728 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f12240039f0 0x7f1224003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:50:01.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.218+0000 7f123d00d640 10 monclient(hunting): picked mon.b con 0x7f12240039f0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:50:01.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.218+0000 7f123d00d640 1 --2- 172.21.15.42:0/978117728 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f12380a3d90 0x7f1224004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:50:01.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.218+0000 7f123d00d640 10 monclient(hunting): picked mon.c con 0x7f12380a3d90 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:50:01.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.218+0000 7f123d00d640 1 --2- 172.21.15.42:0/978117728 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1224004da0 0x7f1224005220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:50:01.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.218+0000 7f123d00d640 10 monclient(hunting): picked mon.a con 0x7f1224004da0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:50:01.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.218+0000 7f123d00d640 10 monclient(hunting): start opening mon connection 2024-09-16T03:50:01.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.218+0000 7f123d00d640 1 -- 172.21.15.42:0/978117728 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f123809c420 con 0x7f1224004da0 2024-09-16T03:50:01.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.218+0000 7f123d00d640 10 monclient(hunting): start opening mon connection 2024-09-16T03:50:01.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.218+0000 7f123d00d640 1 -- 172.21.15.42:0/978117728 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f122400bc80 con 0x7f12240039f0 2024-09-16T03:50:01.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.218+0000 7f123d00d640 10 monclient(hunting): start opening mon connection 2024-09-16T03:50:01.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.218+0000 7f123d00d640 1 -- 172.21.15.42:0/978117728 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f122400be80 con 0x7f12380a3d90 2024-09-16T03:50:01.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.218+0000 7f123d00d640 10 monclient(hunting): _renew_subs 2024-09-16T03:50:01.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.218+0000 7f123d00d640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:50:01.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.218+0000 7f1240273640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-16T03:50:01.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.218+0000 7f123d80e640 1 --2- 172.21.15.42:0/978117728 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f12380a3d90 0x7f1224004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:50:01.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.218+0000 7f123d80e640 10 monclient(hunting): get_auth_request con 0x7f12380a3d90 auth_method 0 2024-09-16T03:50:01.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.218+0000 7f123d80e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:50:01.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.218+0000 7f123d80e640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:50:01.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.218+0000 7f123d80e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:50:01.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.218+0000 7f123e810640 1 --2- 172.21.15.42:0/978117728 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1224004da0 0x7f1224005220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:50:01.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.218+0000 7f123e810640 10 monclient(hunting): get_auth_request con 0x7f1224004da0 auth_method 0 2024-09-16T03:50:01.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.218+0000 7f123e810640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:50:01.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.218+0000 7f123e810640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:50:01.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.218+0000 7f123e810640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:50:01.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.218+0000 7f123d80e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:50:01.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.218+0000 7f123d80e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:50:01.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.218+0000 7f123d80e640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T03:50:01.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.218+0000 7f123e810640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:50:01.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.218+0000 7f123e810640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:50:01.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.218+0000 7f123e00f640 1 --2- 172.21.15.42:0/978117728 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f12240039f0 0x7f1224003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:50:01.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.218+0000 7f123e810640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T03:50:01.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.218+0000 7f123e00f640 10 monclient(hunting): get_auth_request con 0x7f12240039f0 auth_method 0 2024-09-16T03:50:01.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.218+0000 7f123e00f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:50:01.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.218+0000 7f123e00f640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:50:01.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.218+0000 7f123e00f640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:50:01.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.218+0000 7f123d80e640 10 monclient(hunting): handle_auth_done global_id 764733 payload 293 2024-09-16T03:50:01.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.218+0000 7f123d80e640 1 -- 172.21.15.42:0/978117728 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f12240039f0 msgr2=0x7f1224003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:50:01.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.218+0000 7f123d80e640 1 --2- 172.21.15.42:0/978117728 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f12240039f0 0x7f1224003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:50:01.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.218+0000 7f123d80e640 1 -- 172.21.15.42:0/978117728 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1224004da0 msgr2=0x7f1224005220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:50:01.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.218+0000 7f123d80e640 1 --2- 172.21.15.42:0/978117728 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1224004da0 0x7f1224005220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:50:01.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.218+0000 7f123d80e640 10 monclient: _finish_hunting 0 2024-09-16T03:50:01.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.218+0000 7f123d80e640 1 monclient: found mon.c 2024-09-16T03:50:01.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.218+0000 7f123d80e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:50:01.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.218+0000 7f123d80e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:50:01.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.218+0000 7f123d80e640 1 -- 172.21.15.42:0/978117728 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1224003530 con 0x7f12380a3d90 2024-09-16T03:50:01.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.218+0000 7f123d80e640 10 monclient: _finish_auth 0 2024-09-16T03:50:01.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.218+0000 7f123d80e640 10 monclient: _check_auth_tickets 2024-09-16T03:50:01.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.218+0000 7f123d80e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:50:01.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.222+0000 7f123d80e640 1 --2- 172.21.15.42:0/978117728 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f12380a3d90 0x7f1224004810 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f1228000ee0 tx=0x7f122801c4f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ba486308e39e91d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:50:01.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.222+0000 7f123d00d640 1 -- 172.21.15.42:0/978117728 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12280b0070 con 0x7f12380a3d90 2024-09-16T03:50:01.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.222+0000 7f123d00d640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:50:01.242 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.222+0000 7f123d00d640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:50:01.242 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.222+0000 7f123d00d640 10 monclient: dump: 2024-09-16T03:50:01.242 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:50:01.242 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:50:01.242 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:01.242 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:01.242 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:50:01.242 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:50:01.242 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:50:01.242 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:50:01.242 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:50:01.242 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:50:01.242 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.222+0000 7f123d00d640 1 -- 172.21.15.42:0/978117728 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f12280ab600 con 0x7f12380a3d90 2024-09-16T03:50:01.242 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.222+0000 7f123d00d640 10 monclient: handle_config config(0 keys) 2024-09-16T03:50:01.243 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.222+0000 7f123d00d640 1 -- 172.21.15.42:0/978117728 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f122800f040 con 0x7f12380a3d90 2024-09-16T03:50:01.243 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.222+0000 7f1240273640 10 monclient: get_monmap_and_config success 2024-09-16T03:50:01.243 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.222+0000 7f1240273640 4 set_mon_vals no callback set 2024-09-16T03:50:01.243 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.222+0000 7f123d00d640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:50:01.243 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.222+0000 7f123d00d640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:50:01.243 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.222+0000 7f123d00d640 10 monclient: dump: 2024-09-16T03:50:01.243 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:50:01.243 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:50:01.243 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:01.243 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:01.243 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:50:01.243 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:50:01.243 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:50:01.244 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:50:01.244 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:50:01.244 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:50:01.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.222+0000 7f1240273640 10 monclient: shutdown 2024-09-16T03:50:01.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.222+0000 7f1240273640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:50:01.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.222+0000 7f1240273640 1 -- 172.21.15.42:0/978117728 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f12380a3d90 msgr2=0x7f1224004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:50:01.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.222+0000 7f1240273640 1 --2- 172.21.15.42:0/978117728 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f12380a3d90 0x7f1224004810 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f1228000ee0 tx=0x7f122801c4f0 comp rx=0 tx=0).stop 2024-09-16T03:50:01.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.222+0000 7f123e810640 1 -- 172.21.15.42:0/978117728 reap_dead start 2024-09-16T03:50:01.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.222+0000 7f1240273640 1 -- 172.21.15.42:0/978117728 shutdown_connections 2024-09-16T03:50:01.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.222+0000 7f1240273640 1 -- 172.21.15.42:0/978117728 >> 172.21.15.42:0/978117728 conn(0x7f12380a9fb0 msgr2=0x7f12380aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:50:01.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.222+0000 7f1240273640 1 -- 172.21.15.42:0/978117728 shutdown_connections 2024-09-16T03:50:01.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.222+0000 7f1240273640 1 -- 172.21.15.42:0/978117728 wait complete. 2024-09-16T03:50:01.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.222+0000 7f1240273640 10 monclient: build_initial_monmap 2024-09-16T03:50:01.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.222+0000 7f1240273640 10 monclient: monmap: 2024-09-16T03:50:01.245 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:50:01.245 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:50:01.245 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:50:01.245 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:50:01.245 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:50:01.245 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:50:01.245 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:50:01.245 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:50:01.245 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:50:01.245 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:50:01.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.222+0000 7f1240273640 1 librados: starting msgr at 2024-09-16T03:50:01.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.222+0000 7f1240273640 1 librados: starting objecter 2024-09-16T03:50:01.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.222+0000 7f1240273640 1 Processor -- start 2024-09-16T03:50:01.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.222+0000 7f1240273640 1 -- start start 2024-09-16T03:50:01.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.222+0000 7f1240273640 1 librados: setting wanted keys 2024-09-16T03:50:01.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.222+0000 7f1240273640 1 librados: calling monclient init 2024-09-16T03:50:01.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.222+0000 7f1240273640 10 monclient: init 2024-09-16T03:50:01.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.226+0000 7f1240273640 10 monclient: _reopen_session rank -1 2024-09-16T03:50:01.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.226+0000 7f1240273640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:50:01.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.226+0000 7f1240273640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f12380a3d90 0x7f12380a2580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:50:01.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.226+0000 7f1240273640 10 monclient(hunting): picked mon.noname-c con 0x7f12380a3d90 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:50:01.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.226+0000 7f1240273640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f123809c940 0x7f123809cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:50:01.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.226+0000 7f1240273640 10 monclient(hunting): picked mon.noname-a con 0x7f123809c940 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:50:01.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.226+0000 7f1240273640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f123809d2a0 0x7f12381d61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:50:01.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.226+0000 7f1240273640 10 monclient(hunting): picked mon.noname-b con 0x7f123809d2a0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:50:01.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.226+0000 7f1240273640 10 monclient(hunting): start opening mon connection 2024-09-16T03:50:01.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.226+0000 7f1240273640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f1238140f70 con 0x7f123809c940 2024-09-16T03:50:01.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.226+0000 7f1240273640 10 monclient(hunting): start opening mon connection 2024-09-16T03:50:01.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.226+0000 7f1240273640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f1238140df0 con 0x7f123809d2a0 2024-09-16T03:50:01.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.226+0000 7f1240273640 10 monclient(hunting): start opening mon connection 2024-09-16T03:50:01.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.226+0000 7f1240273640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f12381410f0 con 0x7f12380a3d90 2024-09-16T03:50:01.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.226+0000 7f1240273640 10 monclient(hunting): _renew_subs 2024-09-16T03:50:01.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.226+0000 7f1240273640 10 monclient(hunting): authenticate will time out at 1448.196273s 2024-09-16T03:50:01.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.226+0000 7f123d80e640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f123809c940 0x7f123809cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:50:01.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.226+0000 7f123e00f640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f12380a3d90 0x7f12380a2580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:50:01.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.226+0000 7f123d80e640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f123809c940 0x7f123809cd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:38700/0 (socket says 172.21.15.42:38700) 2024-09-16T03:50:01.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.226+0000 7f123e00f640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f12380a3d90 0x7f12380a2580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:49424/0 (socket says 172.21.15.42:49424) 2024-09-16T03:50:01.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.226+0000 7f123d80e640 1 -- 172.21.15.42:0/3170720833 learned_addr learned my addr 172.21.15.42:0/3170720833 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:50:01.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.226+0000 7f123d80e640 10 monclient(hunting): get_auth_request con 0x7f123809c940 auth_method 0 2024-09-16T03:50:01.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.226+0000 7f123d80e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:50:01.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.226+0000 7f123d80e640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:50:01.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.226+0000 7f123d80e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:50:01.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.226+0000 7f123e00f640 10 monclient(hunting): get_auth_request con 0x7f12380a3d90 auth_method 0 2024-09-16T03:50:01.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.226+0000 7f123e00f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:50:01.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.226+0000 7f123e00f640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:50:01.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.226+0000 7f123e00f640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:50:01.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.226+0000 7f123d80e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:50:01.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.226+0000 7f123d80e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:50:01.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.226+0000 7f123d80e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:50:01.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.226+0000 7f123e00f640 1 -- 172.21.15.42:0/3170720833 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f12380a3d90 msgr2=0x7f12380a2580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-16T03:50:01.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.226+0000 7f123e00f640 1 -- 172.21.15.42:0/3170720833 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f12380a3d90 msgr2=0x7f12380a2580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T03:50:01.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.226+0000 7f123e00f640 1 --2- 172.21.15.42:0/3170720833 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f12380a3d90 0x7f12380a2580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:50:01.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.226+0000 7f123e00f640 1 --2- 172.21.15.42:0/3170720833 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f12380a3d90 0x7f12380a2580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:50:01.249 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.226+0000 7f123e810640 1 --2- 172.21.15.42:0/3170720833 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f123809d2a0 0x7f12381d61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:50:01.249 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.226+0000 7f123e810640 10 monclient(hunting): get_auth_request con 0x7f123809d2a0 auth_method 0 2024-09-16T03:50:01.249 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.226+0000 7f123e810640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:50:01.249 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.226+0000 7f123e810640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:50:01.249 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.226+0000 7f123e810640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:50:01.249 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.226+0000 7f123d80e640 10 monclient(hunting): handle_auth_done global_id 764736 payload 824 2024-09-16T03:50:01.249 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.226+0000 7f123d80e640 1 -- 172.21.15.42:0/3170720833 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f12380a3d90 msgr2=0x7f12380a2580 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:50:01.249 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.226+0000 7f123d80e640 1 --2- 172.21.15.42:0/3170720833 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f12380a3d90 0x7f12380a2580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:50:01.249 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.226+0000 7f123d80e640 1 -- 172.21.15.42:0/3170720833 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f123809d2a0 msgr2=0x7f12381d61b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:50:01.249 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.226+0000 7f123d80e640 1 --2- 172.21.15.42:0/3170720833 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f123809d2a0 0x7f12381d61b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:50:01.249 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.226+0000 7f123d80e640 10 monclient: _finish_hunting 0 2024-09-16T03:50:01.249 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.226+0000 7f123d80e640 1 monclient: found mon.noname-a 2024-09-16T03:50:01.249 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.226+0000 7f123d80e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:50:01.250 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.226+0000 7f123d80e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:50:01.250 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.226+0000 7f123d80e640 1 -- 172.21.15.42:0/3170720833 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f123809dab0 con 0x7f123809c940 2024-09-16T03:50:01.250 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.226+0000 7f123d80e640 1 --2- 172.21.15.42:0/3170720833 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f123809c940 0x7f123809cd60 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f12280ae040 tx=0x7f1224005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:50:01.250 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.226+0000 7f122effd640 10 client.?.objecter ms_handle_connect 0x7f123809c940 2024-09-16T03:50:01.250 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.226+0000 7f122effd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:50:01.250 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.226+0000 7f122effd640 1 -- 172.21.15.42:0/3170720833 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12280bb070 con 0x7f123809c940 2024-09-16T03:50:01.250 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.226+0000 7f122effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:50:01.250 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.226+0000 7f122effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:50:01.250 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.226+0000 7f122effd640 10 monclient: dump: 2024-09-16T03:50:01.250 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:50:01.250 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:50:01.250 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:01.250 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:01.251 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:50:01.251 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:50:01.251 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:50:01.251 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:50:01.251 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:50:01.251 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:50:01.251 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.226+0000 7f122effd640 10 monclient: _finish_auth 0 2024-09-16T03:50:01.251 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.226+0000 7f122effd640 10 monclient: _check_auth_tickets 2024-09-16T03:50:01.251 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.226+0000 7f122effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:50:01.251 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.226+0000 7f122effd640 1 -- 172.21.15.42:0/3170720833 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f12280abcb0 con 0x7f123809c940 2024-09-16T03:50:01.251 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.226+0000 7f1240273640 5 monclient: authenticate success, global_id 764736 2024-09-16T03:50:01.251 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.226+0000 7f1240273640 10 monclient: _renew_subs 2024-09-16T03:50:01.251 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.226+0000 7f1240273640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:50:01.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.226+0000 7f1240273640 1 -- 172.21.15.42:0/3170720833 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f123806f680 con 0x7f123809c940 2024-09-16T03:50:01.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.226+0000 7f122effd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:50:01.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.226+0000 7f122effd640 1 -- 172.21.15.42:0/3170720833 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12280b0040 con 0x7f123809c940 2024-09-16T03:50:01.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.226+0000 7f122effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:50:01.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.226+0000 7f122effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:50:01.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.226+0000 7f123c80c640 4 set_mon_vals no callback set 2024-09-16T03:50:01.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.226+0000 7f1240273640 10 client.764736.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:50:01.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.226+0000 7f122effd640 10 monclient: dump: 2024-09-16T03:50:01.252 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:50:01.252 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:50:01.252 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:01.252 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:01.252 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:50:01.253 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:50:01.253 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:50:01.253 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:50:01.253 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:50:01.253 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:50:01.253 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.226+0000 7f1240273640 10 monclient: _renew_subs 2024-09-16T03:50:01.253 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.226+0000 7f1240273640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:50:01.253 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.226+0000 7f1240273640 1 -- 172.21.15.42:0/3170720833 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f123809d720 con 0x7f123809c940 2024-09-16T03:50:01.253 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.226+0000 7f1240273640 1 librados: init done 2024-09-16T03:50:01.253 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.226+0000 7f1240273640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:50:01.253 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.226+0000 7f1240273640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:50:01.253 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.226+0000 7f1240273640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:50:01.253 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.226+0000 7f1240273640 1 -- 172.21.15.42:0/3170720833 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1238133b80 con 0x7f123809c940 2024-09-16T03:50:01.253 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.226+0000 7f122effd640 1 -- 172.21.15.42:0/3170720833 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f12280be070 con 0x7f123809c940 2024-09-16T03:50:01.254 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.226+0000 7f122effd640 1 --2- 172.21.15.42:0/3170720833 >> v2:172.21.15.154:6812/254845342 conn(0x7f121c037810 0x7f121c039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:50:01.254 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.226+0000 7f122effd640 1 -- 172.21.15.42:0/3170720833 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(268..268 src has 1..268) ==== 6139+0+0 (secure 0 0 0) 0x7f12280e38e0 con 0x7f123809c940 2024-09-16T03:50:01.254 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.226+0000 7f122effd640 10 client.764736.objecter ms_dispatch 0x7f1238000ba0 osd_map(268..268 src has 1..268) 2024-09-16T03:50:01.254 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.226+0000 7f122effd640 3 client.764736.objecter handle_osd_map got epochs [268,268] > 0 2024-09-16T03:50:01.254 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.226+0000 7f122effd640 3 client.764736.objecter handle_osd_map decoding full epoch 268 2024-09-16T03:50:01.254 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.226+0000 7f122effd640 20 client.764736.objecter dump_active .. 0 homeless 2024-09-16T03:50:01.254 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.226+0000 7f123e00f640 1 --2- 172.21.15.42:0/3170720833 >> v2:172.21.15.154:6812/254845342 conn(0x7f121c037810 0x7f121c039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:50:01.254 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.226+0000 7f123e00f640 10 monclient: get_auth_request con 0x7f121c037810 auth_method 0 2024-09-16T03:50:01.254 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.230+0000 7f123e00f640 1 --2- 172.21.15.42:0/3170720833 >> v2:172.21.15.154:6812/254845342 conn(0x7f121c037810 0x7f121c039cd0 secure :-1 s=READY pgs=3530 cs=0 l=1 rev1=1 crypto rx=0x7f1234001210 tx=0x7f1234009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:50:01.254 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.230+0000 7f122effd640 10 client.764736.objecter ms_handle_connect 0x7f121c037810 2024-09-16T03:50:01.255 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.230+0000 7f122effd640 1 -- 172.21.15.42:0/3170720833 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f1228016200 con 0x7f123809c940 2024-09-16T03:50:01.255 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.230+0000 7f122effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:50:01.255 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.230+0000 7f122effd640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:50:01.469 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.466+0000 7f1240273640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:50:01.469 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.466+0000 7f1240273640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:50:01.469 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.466+0000 7f1240273640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:50:01.469 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.466+0000 7f1240273640 1 -- 172.21.15.42:0/3170720833 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f12380a5750 con 0x7f123809c940 2024-09-16T03:50:01.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.466+0000 7f122effd640 1 -- 172.21.15.42:0/3170720833 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1555 (secure 0 0 0) 0x7f12280b69c0 con 0x7f123809c940 2024-09-16T03:50:01.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.466+0000 7f122effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:50:01.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.466+0000 7f122effd640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:50:01.470 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":70,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":21,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:50:01.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.466+0000 7f1240273640 10 librados: watch_flush enter 2024-09-16T03:50:01.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.466+0000 7f1240273640 10 librados: watch_flush exit 2024-09-16T03:50:01.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.466+0000 7f1240273640 20 client.764736.objecter shutdown clearing up homeless session... 2024-09-16T03:50:01.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.466+0000 7f1240273640 10 client.764736.objecter successfully canceled tick 2024-09-16T03:50:01.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.466+0000 7f1240273640 1 -- 172.21.15.42:0/3170720833 >> v2:172.21.15.154:6812/254845342 conn(0x7f121c037810 msgr2=0x7f121c039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:50:01.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.466+0000 7f1240273640 1 --2- 172.21.15.42:0/3170720833 >> v2:172.21.15.154:6812/254845342 conn(0x7f121c037810 0x7f121c039cd0 secure :-1 s=READY pgs=3530 cs=0 l=1 rev1=1 crypto rx=0x7f1234001210 tx=0x7f1234009040 comp rx=0 tx=0).stop 2024-09-16T03:50:01.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.466+0000 7f1240273640 10 monclient: shutdown 2024-09-16T03:50:01.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.466+0000 7f1240273640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:50:01.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.466+0000 7f1240273640 1 -- 172.21.15.42:0/3170720833 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f123809c940 msgr2=0x7f123809cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:50:01.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.466+0000 7f1240273640 1 --2- 172.21.15.42:0/3170720833 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f123809c940 0x7f123809cd60 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f12280ae040 tx=0x7f1224005840 comp rx=0 tx=0).stop 2024-09-16T03:50:01.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.466+0000 7f1240273640 1 -- 172.21.15.42:0/3170720833 shutdown_connections 2024-09-16T03:50:01.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.466+0000 7f1240273640 1 --2- 172.21.15.42:0/3170720833 >> v2:172.21.15.154:6812/254845342 conn(0x7f121c037810 0x7f121c039cd0 unknown :-1 s=CLOSED pgs=3530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:50:01.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.466+0000 7f1240273640 1 --2- 172.21.15.42:0/3170720833 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f123809d2a0 0x7f12381d61b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:50:01.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.466+0000 7f1240273640 1 --2- 172.21.15.42:0/3170720833 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f123809c940 0x7f123809cd60 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:50:01.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.466+0000 7f1240273640 1 --2- 172.21.15.42:0/3170720833 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f12380a3d90 0x7f12380a2580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:50:01.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.466+0000 7f1240273640 1 -- 172.21.15.42:0/3170720833 >> 172.21.15.42:0/3170720833 conn(0x7f12380a9fb0 msgr2=0x7f12381321a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:50:01.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.466+0000 7f1240273640 1 -- 172.21.15.42:0/3170720833 shutdown_connections 2024-09-16T03:50:01.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.466+0000 7f1240273640 1 -- 172.21.15.42:0/3170720833 wait complete. 2024-09-16T03:50:01.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:01.470+0000 7f1240273640 1 librados: shutdown 2024-09-16T03:50:04.490 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:50:04.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.618+0000 7f62f08fe640 10 monclient: get_monmap_and_config 2024-09-16T03:50:04.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.618+0000 7f62f08fe640 10 monclient: build_initial_monmap 2024-09-16T03:50:04.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.618+0000 7f62f08fe640 10 monclient: monmap: 2024-09-16T03:50:04.624 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:50:04.624 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:50:04.624 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:50:04.622747+0000 2024-09-16T03:50:04.624 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:50:04.622747+0000 2024-09-16T03:50:04.624 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:50:04.624 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:50:04.624 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:50:04.624 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:50:04.624 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:50:04.625 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:50:04.625 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:50:04.625 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:50:04.627 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.618+0000 7f62f08fe640 1 Processor -- start 2024-09-16T03:50:04.627 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.618+0000 7f62f08fe640 1 -- start start 2024-09-16T03:50:04.627 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.618+0000 7f62f08fe640 10 monclient: init 2024-09-16T03:50:04.627 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.622+0000 7f62f08fe640 10 monclient: _reopen_session rank -1 2024-09-16T03:50:04.627 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.622+0000 7f62f08fe640 10 monclient: _add_conns ranks=[4,2,3,0,1] 2024-09-16T03:50:04.627 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.622+0000 7f62f08fe640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f62ec0a1330 0x7f62ec0a1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:50:04.627 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.622+0000 7f62f08fe640 10 monclient(hunting): picked mon.noname-c con 0x7f62ec0a1330 addr v2:172.21.15.154:3300/0 2024-09-16T03:50:04.627 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.622+0000 7f62f08fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f62ec06dac0 0x7f62ec0a1c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:50:04.628 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.622+0000 7f62f08fe640 10 monclient(hunting): picked mon.noname-b con 0x7f62ec06dac0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:50:04.628 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.622+0000 7f62f08fe640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f62ec135100 0x7f62ec135550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:50:04.628 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.622+0000 7f62f08fe640 10 monclient(hunting): picked mon.noname-a con 0x7f62ec135100 addr v2:172.21.15.42:3300/0 2024-09-16T03:50:04.628 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.622+0000 7f62f08fe640 10 monclient(hunting): start opening mon connection 2024-09-16T03:50:04.628 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.622+0000 7f62f08fe640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f62ec09c7d0 con 0x7f62ec135100 2024-09-16T03:50:04.628 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.622+0000 7f62f08fe640 10 monclient(hunting): start opening mon connection 2024-09-16T03:50:04.628 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.622+0000 7f62f08fe640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f62ec09c090 con 0x7f62ec0a1330 2024-09-16T03:50:04.628 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.622+0000 7f62f08fe640 10 monclient(hunting): start opening mon connection 2024-09-16T03:50:04.628 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.622+0000 7f62f08fe640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f62ec09f610 con 0x7f62ec06dac0 2024-09-16T03:50:04.628 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.622+0000 7f62f08fe640 10 monclient(hunting): _renew_subs 2024-09-16T03:50:04.628 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.622+0000 7f62f08fe640 10 monclient(hunting): authenticate will time out at 1451.592256s 2024-09-16T03:50:04.628 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.622+0000 7f62ead9d640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f62ec135100 0x7f62ec135550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:50:04.629 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.622+0000 7f62ead9d640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f62ec135100 0x7f62ec135550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:55186/0 (socket says 172.21.15.42:55186) 2024-09-16T03:50:04.629 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.622+0000 7f62e9d9b640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f62ec06dac0 0x7f62ec0a1c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:50:04.629 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.622+0000 7f62ead9d640 1 -- 172.21.15.42:0/2932737304 learned_addr learned my addr 172.21.15.42:0/2932737304 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:50:04.629 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.622+0000 7f62ead9d640 10 monclient(hunting): get_auth_request con 0x7f62ec135100 auth_method 0 2024-09-16T03:50:04.629 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.622+0000 7f62ead9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:50:04.629 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.622+0000 7f62ead9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:50:04.629 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.622+0000 7f62ead9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:50:04.629 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.622+0000 7f62e9d9b640 10 monclient(hunting): get_auth_request con 0x7f62ec06dac0 auth_method 0 2024-09-16T03:50:04.629 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.622+0000 7f62e9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:50:04.629 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.622+0000 7f62e9d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:50:04.629 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.622+0000 7f62e9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:50:04.629 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.622+0000 7f62ead9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:50:04.629 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.622+0000 7f62ead9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:50:04.629 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.622+0000 7f62ead9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:50:04.630 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.622+0000 7f62e9d9b640 1 -- 172.21.15.42:0/2932737304 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f62ec06dac0 msgr2=0x7f62ec0a1c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T03:50:04.630 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.622+0000 7f62e9d9b640 1 -- 172.21.15.42:0/2932737304 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f62ec06dac0 msgr2=0x7f62ec0a1c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:50:04.630 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.622+0000 7f62e9d9b640 1 --2- 172.21.15.42:0/2932737304 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f62ec06dac0 0x7f62ec0a1c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:50:04.630 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.622+0000 7f62e9d9b640 1 --2- 172.21.15.42:0/2932737304 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f62ec06dac0 0x7f62ec0a1c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:50:04.630 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.622+0000 7f62ea59c640 1 --2- 172.21.15.42:0/2932737304 >> v2:172.21.15.154:3300/0 conn(0x7f62ec0a1330 0x7f62ec0a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:50:04.630 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.622+0000 7f62ead9d640 10 monclient(hunting): handle_auth_done global_id 764739 payload 274 2024-09-16T03:50:04.630 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.622+0000 7f62ead9d640 1 -- 172.21.15.42:0/2932737304 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f62ec06dac0 msgr2=0x7f62ec0a1c50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:50:04.630 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.622+0000 7f62ead9d640 1 --2- 172.21.15.42:0/2932737304 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f62ec06dac0 0x7f62ec0a1c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:50:04.630 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.622+0000 7f62ead9d640 1 -- 172.21.15.42:0/2932737304 >> v2:172.21.15.154:3300/0 conn(0x7f62ec0a1330 msgr2=0x7f62ec0a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:50:04.630 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.622+0000 7f62ead9d640 1 --2- 172.21.15.42:0/2932737304 >> v2:172.21.15.154:3300/0 conn(0x7f62ec0a1330 0x7f62ec0a1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:50:04.630 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.622+0000 7f62ead9d640 10 monclient: _finish_hunting 0 2024-09-16T03:50:04.630 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.622+0000 7f62ead9d640 1 monclient: found mon.noname-a 2024-09-16T03:50:04.630 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.622+0000 7f62ead9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:50:04.631 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.622+0000 7f62ead9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:50:04.631 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.622+0000 7f62ead9d640 1 -- 172.21.15.42:0/2932737304 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62ec09f8f0 con 0x7f62ec135100 2024-09-16T03:50:04.631 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.622+0000 7f62ea59c640 10 monclient: get_auth_request con 0x7f62ec0a1330 auth_method 0 2024-09-16T03:50:04.631 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.622+0000 7f62ea59c640 1 --2- 172.21.15.42:0/2932737304 >> v2:172.21.15.154:3300/0 conn(0x7f62ec0a1330 0x7f62ec0a1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T03:50:04.636 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.622+0000 7f62ead9d640 1 --2- 172.21.15.42:0/2932737304 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f62ec135100 0x7f62ec135550 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f62ec06add0 tx=0x7f62e401c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4704e32a07a97796 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:50:04.636 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.622+0000 7f62e959a640 1 -- 172.21.15.42:0/2932737304 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62e40b0070 con 0x7f62ec135100 2024-09-16T03:50:04.636 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.622+0000 7f62e959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:50:04.636 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.622+0000 7f62e959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:50:04.636 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.622+0000 7f62e959a640 10 monclient: dump: 2024-09-16T03:50:04.637 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:50:04.637 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:50:04.637 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:04.637 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:04.637 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:50:04.637 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:50:04.637 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:50:04.637 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:50:04.637 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:50:04.637 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:50:04.637 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.622+0000 7f62e959a640 10 monclient: _finish_auth 0 2024-09-16T03:50:04.637 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.622+0000 7f62e959a640 10 monclient: _check_auth_tickets 2024-09-16T03:50:04.637 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.622+0000 7f62e959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:50:04.638 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.622+0000 7f62e959a640 1 -- 172.21.15.42:0/2932737304 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f62e40ab600 con 0x7f62ec135100 2024-09-16T03:50:04.638 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.622+0000 7f62f08fe640 5 monclient: authenticate success, global_id 764739 2024-09-16T03:50:04.638 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.622+0000 7f62f08fe640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:50:04.638 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.622+0000 7f62e959a640 10 monclient: handle_config config(0 keys) 2024-09-16T03:50:04.638 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.622+0000 7f62e959a640 1 -- 172.21.15.42:0/2932737304 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62e40ab9d0 con 0x7f62ec135100 2024-09-16T03:50:04.638 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.622+0000 7f62f08fe640 10 monclient: get_monmap_and_config success 2024-09-16T03:50:04.638 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.622+0000 7f62f08fe640 4 set_mon_vals no callback set 2024-09-16T03:50:04.638 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.622+0000 7f62e959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:50:04.638 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.622+0000 7f62e959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:50:04.638 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.622+0000 7f62e959a640 10 monclient: dump: 2024-09-16T03:50:04.638 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:50:04.638 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:50:04.638 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:04.639 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:04.639 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:50:04.639 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:50:04.639 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:50:04.639 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:50:04.639 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:50:04.639 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:50:04.639 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.626+0000 7f62f08fe640 10 monclient: shutdown 2024-09-16T03:50:04.639 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.626+0000 7f62f08fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:50:04.639 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.626+0000 7f62f08fe640 1 -- 172.21.15.42:0/2932737304 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f62ec135100 msgr2=0x7f62ec135550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:50:04.639 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.626+0000 7f62f08fe640 1 --2- 172.21.15.42:0/2932737304 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f62ec135100 0x7f62ec135550 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f62ec06add0 tx=0x7f62e401c500 comp rx=0 tx=0).stop 2024-09-16T03:50:04.639 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.626+0000 7f62f08fe640 1 -- 172.21.15.42:0/2932737304 shutdown_connections 2024-09-16T03:50:04.639 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.626+0000 7f62f08fe640 1 --2- 172.21.15.42:0/2932737304 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f62ec135100 0x7f62ec135550 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:50:04.639 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.626+0000 7f62f08fe640 1 --2- 172.21.15.42:0/2932737304 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f62ec06dac0 0x7f62ec0a1c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:50:04.640 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.626+0000 7f62f08fe640 1 --2- 172.21.15.42:0/2932737304 >> v2:172.21.15.154:3300/0 conn(0x7f62ec0a1330 0x7f62ec0a1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:50:04.640 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.626+0000 7f62f08fe640 1 -- 172.21.15.42:0/2932737304 >> 172.21.15.42:0/2932737304 conn(0x7f62ec0a2420 msgr2=0x7f62ec0a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:50:04.640 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.626+0000 7f62f08fe640 1 -- 172.21.15.42:0/2932737304 shutdown_connections 2024-09-16T03:50:04.640 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.626+0000 7f62f08fe640 1 -- 172.21.15.42:0/2932737304 wait complete. 2024-09-16T03:50:04.640 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.626+0000 7f62f08fe640 10 monclient: build_initial_monmap 2024-09-16T03:50:04.640 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.626+0000 7f62f08fe640 10 monclient: monmap: 2024-09-16T03:50:04.640 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:50:04.640 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:50:04.640 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:50:04.640 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:50:04.640 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:50:04.640 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:50:04.641 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:50:04.641 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:50:04.641 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:50:04.641 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:50:04.641 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.626+0000 7f62f08fe640 1 librados: starting msgr at 2024-09-16T03:50:04.641 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.626+0000 7f62f08fe640 1 librados: starting objecter 2024-09-16T03:50:04.641 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.626+0000 7f62f08fe640 1 Processor -- start 2024-09-16T03:50:04.641 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.626+0000 7f62f08fe640 1 -- start start 2024-09-16T03:50:04.641 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.626+0000 7f62f08fe640 1 librados: setting wanted keys 2024-09-16T03:50:04.641 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.626+0000 7f62f08fe640 1 librados: calling monclient init 2024-09-16T03:50:04.641 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.626+0000 7f62f08fe640 10 monclient: init 2024-09-16T03:50:04.641 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.626+0000 7f62f08fe640 10 monclient: _reopen_session rank -1 2024-09-16T03:50:04.641 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.626+0000 7f62f08fe640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:50:04.641 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.626+0000 7f62f08fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f62ec0a1330 0x7f62ec1ccd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:50:04.642 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.626+0000 7f62f08fe640 10 monclient(hunting): picked mon.noname-a con 0x7f62ec0a1330 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:50:04.642 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.626+0000 7f62f08fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f62ec135100 0x7f62ec134340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:50:04.642 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.626+0000 7f62f08fe640 10 monclient(hunting): picked mon.noname-c con 0x7f62ec135100 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:50:04.642 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.626+0000 7f62f08fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f62ec134880 0x7f62ec11bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:50:04.642 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.626+0000 7f62f08fe640 10 monclient(hunting): picked mon.noname-b con 0x7f62ec134880 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:50:04.642 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.626+0000 7f62f08fe640 10 monclient(hunting): start opening mon connection 2024-09-16T03:50:04.642 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.626+0000 7f62f08fe640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f62ec09eef0 con 0x7f62ec0a1330 2024-09-16T03:50:04.642 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.626+0000 7f62f08fe640 10 monclient(hunting): start opening mon connection 2024-09-16T03:50:04.642 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.626+0000 7f62f08fe640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f62ec09ed70 con 0x7f62ec134880 2024-09-16T03:50:04.642 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.626+0000 7f62f08fe640 10 monclient(hunting): start opening mon connection 2024-09-16T03:50:04.642 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.626+0000 7f62f08fe640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f62ec13eb70 con 0x7f62ec135100 2024-09-16T03:50:04.642 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.626+0000 7f62f08fe640 10 monclient(hunting): _renew_subs 2024-09-16T03:50:04.642 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.626+0000 7f62f08fe640 10 monclient(hunting): authenticate will time out at 1451.597777s 2024-09-16T03:50:04.642 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.626+0000 7f62ea59c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f62ec0a1330 0x7f62ec1ccd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:50:04.643 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.626+0000 7f62ea59c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f62ec0a1330 0x7f62ec1ccd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:55198/0 (socket says 172.21.15.42:55198) 2024-09-16T03:50:04.643 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.626+0000 7f62e9d9b640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f62ec135100 0x7f62ec134340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:50:04.643 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.626+0000 7f62ea59c640 1 -- 172.21.15.42:0/2815543645 learned_addr learned my addr 172.21.15.42:0/2815543645 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:50:04.643 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.626+0000 7f62ea59c640 10 monclient(hunting): get_auth_request con 0x7f62ec0a1330 auth_method 0 2024-09-16T03:50:04.643 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.626+0000 7f62ea59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:50:04.643 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.626+0000 7f62ea59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:50:04.643 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.626+0000 7f62ea59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:50:04.643 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.626+0000 7f62e9d9b640 10 monclient(hunting): get_auth_request con 0x7f62ec135100 auth_method 0 2024-09-16T03:50:04.643 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.626+0000 7f62e9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:50:04.643 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.626+0000 7f62e9d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:50:04.643 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.626+0000 7f62e9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:50:04.643 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.626+0000 7f62ea59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:50:04.643 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.626+0000 7f62ea59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:50:04.643 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.626+0000 7f62ea59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:50:04.644 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.626+0000 7f62e9d9b640 1 -- 172.21.15.42:0/2815543645 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f62ec135100 msgr2=0x7f62ec134340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-16T03:50:04.644 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.626+0000 7f62e9d9b640 1 -- 172.21.15.42:0/2815543645 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f62ec135100 msgr2=0x7f62ec134340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T03:50:04.644 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.626+0000 7f62e9d9b640 1 --2- 172.21.15.42:0/2815543645 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f62ec135100 0x7f62ec134340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:50:04.644 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.626+0000 7f62e9d9b640 1 --2- 172.21.15.42:0/2815543645 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f62ec135100 0x7f62ec134340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:50:04.644 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.626+0000 7f62ead9d640 1 --2- 172.21.15.42:0/2815543645 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f62ec134880 0x7f62ec11bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:50:04.644 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.626+0000 7f62ead9d640 10 monclient(hunting): get_auth_request con 0x7f62ec134880 auth_method 0 2024-09-16T03:50:04.644 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.626+0000 7f62ead9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:50:04.644 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.626+0000 7f62ead9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:50:04.644 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.626+0000 7f62ead9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:50:04.644 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.626+0000 7f62ea59c640 10 monclient(hunting): handle_auth_done global_id 764742 payload 824 2024-09-16T03:50:04.644 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.626+0000 7f62ea59c640 1 -- 172.21.15.42:0/2815543645 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f62ec135100 msgr2=0x7f62ec134340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:50:04.644 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.626+0000 7f62ea59c640 1 --2- 172.21.15.42:0/2815543645 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f62ec135100 0x7f62ec134340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:50:04.644 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.626+0000 7f62ea59c640 1 -- 172.21.15.42:0/2815543645 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f62ec134880 msgr2=0x7f62ec11bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:50:04.645 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.626+0000 7f62ea59c640 1 --2- 172.21.15.42:0/2815543645 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f62ec134880 0x7f62ec11bf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:50:04.645 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.626+0000 7f62ea59c640 10 monclient: _finish_hunting 0 2024-09-16T03:50:04.645 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.626+0000 7f62ea59c640 1 monclient: found mon.noname-a 2024-09-16T03:50:04.645 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.626+0000 7f62ea59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:50:04.645 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.626+0000 7f62ea59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:50:04.645 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.626+0000 7f62ea59c640 1 -- 172.21.15.42:0/2815543645 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62ec071ac0 con 0x7f62ec0a1330 2024-09-16T03:50:04.645 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.626+0000 7f62ea59c640 1 --2- 172.21.15.42:0/2815543645 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f62ec0a1330 0x7f62ec1ccd20 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f62d400cf60 tx=0x7f62d400cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:50:04.645 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.626+0000 7f62e2ffd640 10 client.?.objecter ms_handle_connect 0x7f62ec0a1330 2024-09-16T03:50:04.645 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.626+0000 7f62e2ffd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:50:04.645 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.626+0000 7f62e2ffd640 1 -- 172.21.15.42:0/2815543645 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62d401a070 con 0x7f62ec0a1330 2024-09-16T03:50:04.645 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.626+0000 7f62e2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:50:04.645 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.626+0000 7f62e2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:50:04.645 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.626+0000 7f62e2ffd640 10 monclient: dump: 2024-09-16T03:50:04.645 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:50:04.646 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:50:04.646 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:04.646 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:04.646 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:50:04.646 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:50:04.646 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:50:04.646 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:50:04.646 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:50:04.646 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:50:04.646 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.626+0000 7f62e2ffd640 10 monclient: _finish_auth 0 2024-09-16T03:50:04.646 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.626+0000 7f62e2ffd640 10 monclient: _check_auth_tickets 2024-09-16T03:50:04.646 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.626+0000 7f62e2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:50:04.646 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.626+0000 7f62e2ffd640 1 -- 172.21.15.42:0/2815543645 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f62d400d840 con 0x7f62ec0a1330 2024-09-16T03:50:04.646 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.626+0000 7f62f08fe640 5 monclient: authenticate success, global_id 764742 2024-09-16T03:50:04.646 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.626+0000 7f62f08fe640 10 monclient: _renew_subs 2024-09-16T03:50:04.647 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.626+0000 7f62f08fe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:50:04.647 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.626+0000 7f62f08fe640 1 -- 172.21.15.42:0/2815543645 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62ec073a30 con 0x7f62ec0a1330 2024-09-16T03:50:04.647 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.626+0000 7f62e2ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:50:04.647 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.626+0000 7f62e2ffd640 1 -- 172.21.15.42:0/2815543645 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62d400dc10 con 0x7f62ec0a1330 2024-09-16T03:50:04.647 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.626+0000 7f62e2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:50:04.647 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.626+0000 7f62e2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:50:04.647 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.626+0000 7f62e2ffd640 10 monclient: dump: 2024-09-16T03:50:04.647 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:50:04.647 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:50:04.647 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:04.647 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:04.647 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:50:04.647 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:50:04.648 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:50:04.648 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:50:04.648 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:50:04.648 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:50:04.648 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.626+0000 7f62e3fff640 4 set_mon_vals no callback set 2024-09-16T03:50:04.648 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.626+0000 7f62f08fe640 10 client.764742.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:50:04.648 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.626+0000 7f62f08fe640 10 monclient: _renew_subs 2024-09-16T03:50:04.648 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.626+0000 7f62f08fe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:50:04.648 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.626+0000 7f62f08fe640 1 -- 172.21.15.42:0/2815543645 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62ec11fce0 con 0x7f62ec0a1330 2024-09-16T03:50:04.648 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.626+0000 7f62f08fe640 1 librados: init done 2024-09-16T03:50:04.648 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.626+0000 7f62e2ffd640 1 -- 172.21.15.42:0/2815543645 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f62d401d070 con 0x7f62ec0a1330 2024-09-16T03:50:04.648 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.630+0000 7f62f08fe640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:50:04.648 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.630+0000 7f62f08fe640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:50:04.648 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.630+0000 7f62f08fe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:50:04.649 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.630+0000 7f62f08fe640 1 -- 172.21.15.42:0/2815543645 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f62ec135550 con 0x7f62ec0a1330 2024-09-16T03:50:04.649 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.630+0000 7f62e2ffd640 1 --2- 172.21.15.42:0/2815543645 >> v2:172.21.15.154:6812/254845342 conn(0x7f62d0037810 0x7f62d0039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:50:04.649 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.630+0000 7f62e2ffd640 1 -- 172.21.15.42:0/2815543645 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(272..272 src has 1..272) ==== 5812+0+0 (secure 0 0 0) 0x7f62d4049840 con 0x7f62ec0a1330 2024-09-16T03:50:04.649 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.630+0000 7f62e2ffd640 10 client.764742.objecter ms_dispatch 0x7f62ec000ba0 osd_map(272..272 src has 1..272) 2024-09-16T03:50:04.649 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.630+0000 7f62e2ffd640 3 client.764742.objecter handle_osd_map got epochs [272,272] > 0 2024-09-16T03:50:04.649 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.630+0000 7f62e2ffd640 3 client.764742.objecter handle_osd_map decoding full epoch 272 2024-09-16T03:50:04.649 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.630+0000 7f62e2ffd640 20 client.764742.objecter dump_active .. 0 homeless 2024-09-16T03:50:04.649 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.630+0000 7f62e9d9b640 1 --2- 172.21.15.42:0/2815543645 >> v2:172.21.15.154:6812/254845342 conn(0x7f62d0037810 0x7f62d0039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:50:04.649 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.630+0000 7f62e9d9b640 10 monclient: get_auth_request con 0x7f62d0037810 auth_method 0 2024-09-16T03:50:04.649 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.630+0000 7f62e2ffd640 10 client.764742.objecter ms_handle_connect 0x7f62d0037810 2024-09-16T03:50:04.649 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.630+0000 7f62e9d9b640 1 --2- 172.21.15.42:0/2815543645 >> v2:172.21.15.154:6812/254845342 conn(0x7f62d0037810 0x7f62d0039cd0 secure :-1 s=READY pgs=3531 cs=0 l=1 rev1=1 crypto rx=0x7f62dc006aa0 tx=0x7f62dc006a30 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:50:04.650 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.638+0000 7f62e2ffd640 1 -- 172.21.15.42:0/2815543645 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f62d4013c40 con 0x7f62ec0a1330 2024-09-16T03:50:04.650 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.638+0000 7f62e2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:50:04.650 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.638+0000 7f62e2ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:50:04.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.862+0000 7f62f08fe640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:50:04.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.862+0000 7f62f08fe640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:50:04.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.862+0000 7f62f08fe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:50:04.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.862+0000 7f62f08fe640 1 -- 172.21.15.42:0/2815543645 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f62ec069040 con 0x7f62ec0a1330 2024-09-16T03:50:04.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.862+0000 7f62e2ffd640 1 -- 172.21.15.42:0/2815543645 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1555 (secure 0 0 0) 0x7f62d4047d70 con 0x7f62ec0a1330 2024-09-16T03:50:04.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.862+0000 7f62e2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:50:04.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.862+0000 7f62e2ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:50:04.868 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":70,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":24,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:50:04.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.866+0000 7f62f08fe640 10 librados: watch_flush enter 2024-09-16T03:50:04.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.866+0000 7f62f08fe640 10 librados: watch_flush exit 2024-09-16T03:50:04.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.866+0000 7f62f08fe640 20 client.764742.objecter shutdown clearing up homeless session... 2024-09-16T03:50:04.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.866+0000 7f62f08fe640 10 client.764742.objecter successfully canceled tick 2024-09-16T03:50:04.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.866+0000 7f62f08fe640 1 -- 172.21.15.42:0/2815543645 >> v2:172.21.15.154:6812/254845342 conn(0x7f62d0037810 msgr2=0x7f62d0039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:50:04.871 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.866+0000 7f62f08fe640 1 --2- 172.21.15.42:0/2815543645 >> v2:172.21.15.154:6812/254845342 conn(0x7f62d0037810 0x7f62d0039cd0 secure :-1 s=READY pgs=3531 cs=0 l=1 rev1=1 crypto rx=0x7f62dc006aa0 tx=0x7f62dc006a30 comp rx=0 tx=0).stop 2024-09-16T03:50:04.871 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.866+0000 7f62f08fe640 10 monclient: shutdown 2024-09-16T03:50:04.871 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.866+0000 7f62f08fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:50:04.871 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.866+0000 7f62f08fe640 1 -- 172.21.15.42:0/2815543645 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f62ec0a1330 msgr2=0x7f62ec1ccd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:50:04.871 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.866+0000 7f62f08fe640 1 --2- 172.21.15.42:0/2815543645 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f62ec0a1330 0x7f62ec1ccd20 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f62d400cf60 tx=0x7f62d400cf90 comp rx=0 tx=0).stop 2024-09-16T03:50:04.871 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.866+0000 7f62f08fe640 1 -- 172.21.15.42:0/2815543645 shutdown_connections 2024-09-16T03:50:04.871 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.866+0000 7f62f08fe640 1 --2- 172.21.15.42:0/2815543645 >> v2:172.21.15.154:6812/254845342 conn(0x7f62d0037810 0x7f62d0039cd0 unknown :-1 s=CLOSED pgs=3531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:50:04.871 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.866+0000 7f62f08fe640 1 --2- 172.21.15.42:0/2815543645 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f62ec134880 0x7f62ec11bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:50:04.871 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.866+0000 7f62f08fe640 1 --2- 172.21.15.42:0/2815543645 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f62ec135100 0x7f62ec134340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:50:04.871 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.866+0000 7f62f08fe640 1 --2- 172.21.15.42:0/2815543645 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f62ec0a1330 0x7f62ec1ccd20 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:50:04.871 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.866+0000 7f62f08fe640 1 -- 172.21.15.42:0/2815543645 >> 172.21.15.42:0/2815543645 conn(0x7f62ec0a2420 msgr2=0x7f62ec13bb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:50:04.871 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.866+0000 7f62f08fe640 1 -- 172.21.15.42:0/2815543645 shutdown_connections 2024-09-16T03:50:04.871 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.866+0000 7f62f08fe640 1 -- 172.21.15.42:0/2815543645 wait complete. 2024-09-16T03:50:04.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:04.866+0000 7f62f08fe640 1 librados: shutdown 2024-09-16T03:50:07.887 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:50:08.019 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.014+0000 7f0ccb839640 10 monclient: get_monmap_and_config 2024-09-16T03:50:08.019 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.014+0000 7f0ccb839640 10 monclient: build_initial_monmap 2024-09-16T03:50:08.019 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.014+0000 7f0ccb839640 10 monclient: monmap: 2024-09-16T03:50:08.019 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:50:08.019 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:50:08.019 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:50:08.017930+0000 2024-09-16T03:50:08.019 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:50:08.017930+0000 2024-09-16T03:50:08.019 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:50:08.019 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:50:08.019 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:50:08.019 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:50:08.020 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:50:08.020 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:50:08.020 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:50:08.020 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:50:08.021 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.014+0000 7f0ccb839640 1 Processor -- start 2024-09-16T03:50:08.021 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.014+0000 7f0ccb839640 1 -- start start 2024-09-16T03:50:08.022 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.014+0000 7f0ccb839640 10 monclient: init 2024-09-16T03:50:08.022 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.014+0000 7f0ccb839640 10 monclient: _reopen_session rank -1 2024-09-16T03:50:08.022 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.014+0000 7f0ccb839640 10 monclient: _add_conns ranks=[3,1,0,2,4] 2024-09-16T03:50:08.022 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.014+0000 7f0ccb839640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f0cc40a2f10 0x7f0cc40a1330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:50:08.022 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.014+0000 7f0ccb839640 10 monclient(hunting): picked mon.noname-a con 0x7f0cc40a2f10 addr v2:172.21.15.42:3300/0 2024-09-16T03:50:08.022 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.014+0000 7f0ccb839640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0cc409e070 addr v1:172.21.15.154:6789/0 2024-09-16T03:50:08.022 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.014+0000 7f0ccb839640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0cc4140b30 addr v1:172.21.15.42:6789/0 2024-09-16T03:50:08.022 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.014+0000 7f0ccb839640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0cc409dc50 con 0x7f0cc4140b30 2024-09-16T03:50:08.022 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.014+0000 7f0ccb839640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0cc4073290 con 0x7f0cc409e070 2024-09-16T03:50:08.022 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.014+0000 7f0ccb839640 10 monclient(hunting): start opening mon connection 2024-09-16T03:50:08.022 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.014+0000 7f0ccb839640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f0cc40a32f0 con 0x7f0cc40a2f10 2024-09-16T03:50:08.022 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.014+0000 7f0ccb839640 10 monclient(hunting): _renew_subs 2024-09-16T03:50:08.023 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.014+0000 7f0ccb839640 10 monclient(hunting): authenticate will time out at 1454.987402s 2024-09-16T03:50:08.023 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.014+0000 7f0cc95d5640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f0cc40a2f10 0x7f0cc40a1330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:50:08.023 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.014+0000 7f0cc9dd6640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f0cc4140b30 0x7f0cc4142f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:56370/0 (socket says 172.21.15.42:56370) 2024-09-16T03:50:08.023 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.014+0000 7f0cc9dd6640 1 -- 172.21.15.42:0/454959531 learned_addr learned my addr 172.21.15.42:0/454959531 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:50:08.023 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.014+0000 7f0cc95d5640 10 monclient(hunting): get_auth_request con 0x7f0cc40a2f10 auth_method 0 2024-09-16T03:50:08.023 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.014+0000 7f0cc95d5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:50:08.023 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.014+0000 7f0cc95d5640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:50:08.023 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.014+0000 7f0cc95d5640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:50:08.023 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.014+0000 7f0cc95d5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:50:08.023 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.014+0000 7f0cc95d5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:50:08.023 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.014+0000 7f0cc95d5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:50:08.023 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.014+0000 7f0cc95d5640 10 monclient(hunting): handle_auth_done global_id 764751 payload 274 2024-09-16T03:50:08.023 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.018+0000 7f0cc95d5640 1 -- 172.21.15.42:0/454959531 >> v1:172.21.15.154:6789/0 conn(0x7f0cc409e070 legacy=0x7f0cc40a1870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:50:08.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.018+0000 7f0cc95d5640 1 -- 172.21.15.42:0/454959531 >> v1:172.21.15.42:6789/0 conn(0x7f0cc4140b30 legacy=0x7f0cc4142f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:50:08.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.018+0000 7f0cc95d5640 10 monclient: _finish_hunting 0 2024-09-16T03:50:08.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.018+0000 7f0cc95d5640 1 monclient: found mon.noname-a 2024-09-16T03:50:08.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.018+0000 7f0cc95d5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:50:08.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.018+0000 7f0cc95d5640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:50:08.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.018+0000 7f0cc95d5640 1 -- 172.21.15.42:0/454959531 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0cc409e600 con 0x7f0cc40a2f10 2024-09-16T03:50:08.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.018+0000 7f0cb7fff640 1 -- 172.21.15.42:0/454959531 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2502223507 0 0) 0x7f0cc409dc50 con 0x7f0cc4140b30 2024-09-16T03:50:08.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.018+0000 7f0cb7fff640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T03:50:08.025 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.018+0000 7f0cc95d5640 1 --2- 172.21.15.42:0/454959531 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0cc40a2f10 0x7f0cc40a1330 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f0cc0009d90 tx=0x7f0cc001c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de519871849b99d7 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:50:08.025 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.022+0000 7f0cb7fff640 1 -- 172.21.15.42:0/454959531 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cc00b2070 con 0x7f0cc40a2f10 2024-09-16T03:50:08.025 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.022+0000 7f0cb7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:50:08.025 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.022+0000 7f0cb7fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:50:08.025 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.022+0000 7f0cb7fff640 10 monclient: dump: 2024-09-16T03:50:08.025 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:50:08.025 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:50:08.025 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:08.026 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:08.026 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:50:08.026 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:50:08.026 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:50:08.026 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:50:08.026 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:50:08.026 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:50:08.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.022+0000 7f0cb7fff640 10 monclient: _finish_auth 0 2024-09-16T03:50:08.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.022+0000 7f0cb7fff640 10 monclient: _check_auth_tickets 2024-09-16T03:50:08.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.022+0000 7f0cb7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:50:08.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.022+0000 7f0cb7fff640 1 -- 172.21.15.42:0/454959531 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0cc00a57e0 con 0x7f0cc40a2f10 2024-09-16T03:50:08.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.022+0000 7f0ccb839640 5 monclient: authenticate success, global_id 764751 2024-09-16T03:50:08.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.022+0000 7f0ccb839640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:50:08.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.022+0000 7f0cb7fff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:50:08.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.022+0000 7f0ccb839640 10 monclient: get_monmap_and_config success 2024-09-16T03:50:08.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.022+0000 7f0cb7fff640 1 -- 172.21.15.42:0/454959531 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cc00a5bb0 con 0x7f0cc40a2f10 2024-09-16T03:50:08.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.022+0000 7f0ccb839640 4 set_mon_vals no callback set 2024-09-16T03:50:08.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.022+0000 7f0cb7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:50:08.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.022+0000 7f0cb7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:50:08.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.022+0000 7f0cb7fff640 10 monclient: dump: 2024-09-16T03:50:08.027 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:50:08.027 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:50:08.027 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:08.027 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:08.027 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:50:08.027 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:50:08.027 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:50:08.027 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:50:08.027 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:50:08.028 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:50:08.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.022+0000 7f0ccb839640 10 monclient: shutdown 2024-09-16T03:50:08.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.022+0000 7f0ccb839640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:50:08.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.022+0000 7f0ccb839640 1 -- 172.21.15.42:0/454959531 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0cc40a2f10 msgr2=0x7f0cc40a1330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:50:08.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.022+0000 7f0ccb839640 1 --2- 172.21.15.42:0/454959531 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0cc40a2f10 0x7f0cc40a1330 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f0cc0009d90 tx=0x7f0cc001c830 comp rx=0 tx=0).stop 2024-09-16T03:50:08.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.022+0000 7f0ccb839640 1 -- 172.21.15.42:0/454959531 shutdown_connections 2024-09-16T03:50:08.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.022+0000 7f0ccb839640 1 --2- 172.21.15.42:0/454959531 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0cc40a2f10 0x7f0cc40a1330 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:50:08.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.022+0000 7f0ccb839640 1 -- 172.21.15.42:0/454959531 >> 172.21.15.42:0/454959531 conn(0x7f0cc40a2420 msgr2=0x7f0cc40a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:50:08.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.022+0000 7f0ccb839640 1 -- 172.21.15.42:0/454959531 shutdown_connections 2024-09-16T03:50:08.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.022+0000 7f0ccb839640 1 -- 172.21.15.42:0/454959531 wait complete. 2024-09-16T03:50:08.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.022+0000 7f0ccb839640 10 monclient: build_initial_monmap 2024-09-16T03:50:08.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.022+0000 7f0ccb839640 10 monclient: monmap: 2024-09-16T03:50:08.038 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:50:08.038 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:50:08.038 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:50:08.038 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:50:08.038 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:50:08.038 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:50:08.038 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:50:08.038 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:50:08.039 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:50:08.039 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:50:08.039 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.022+0000 7f0ccb839640 1 librados: starting msgr at 2024-09-16T03:50:08.039 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.022+0000 7f0ccb839640 1 librados: starting objecter 2024-09-16T03:50:08.039 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.022+0000 7f0ccb839640 1 Processor -- start 2024-09-16T03:50:08.039 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.022+0000 7f0ccb839640 1 -- start start 2024-09-16T03:50:08.039 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.022+0000 7f0ccb839640 1 librados: setting wanted keys 2024-09-16T03:50:08.039 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.022+0000 7f0ccb839640 1 librados: calling monclient init 2024-09-16T03:50:08.039 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.022+0000 7f0ccb839640 10 monclient: init 2024-09-16T03:50:08.039 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.022+0000 7f0ccb839640 10 monclient: _reopen_session rank -1 2024-09-16T03:50:08.039 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.022+0000 7f0ccb839640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:50:08.039 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.022+0000 7f0ccb839640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0cc409e070 0x7f0cc413b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:50:08.039 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.022+0000 7f0ccb839640 10 monclient(hunting): picked mon.noname-c con 0x7f0cc409e070 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:50:08.039 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.022+0000 7f0ccb839640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0cc40a2f10 0x7f0cc413b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:50:08.040 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.022+0000 7f0ccb839640 10 monclient(hunting): picked mon.noname-b con 0x7f0cc40a2f10 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:50:08.040 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.022+0000 7f0ccb839640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0cc4140b30 0x7f0cc406f520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:50:08.040 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.022+0000 7f0ccb839640 10 monclient(hunting): picked mon.noname-a con 0x7f0cc4140b30 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:50:08.040 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.022+0000 7f0ccb839640 10 monclient(hunting): start opening mon connection 2024-09-16T03:50:08.040 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.022+0000 7f0ccb839640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f0cc4146fe0 con 0x7f0cc4140b30 2024-09-16T03:50:08.040 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.022+0000 7f0ccb839640 10 monclient(hunting): start opening mon connection 2024-09-16T03:50:08.040 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.022+0000 7f0ccb839640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f0cc4146e60 con 0x7f0cc40a2f10 2024-09-16T03:50:08.040 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.022+0000 7f0ccb839640 10 monclient(hunting): start opening mon connection 2024-09-16T03:50:08.040 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.022+0000 7f0ccb839640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f0cc4147160 con 0x7f0cc409e070 2024-09-16T03:50:08.040 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.022+0000 7f0ccb839640 10 monclient(hunting): _renew_subs 2024-09-16T03:50:08.040 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.022+0000 7f0ccb839640 10 monclient(hunting): authenticate will time out at 1454.994608s 2024-09-16T03:50:08.040 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.022+0000 7f0cc9dd6640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0cc4140b30 0x7f0cc406f520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:50:08.040 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.022+0000 7f0cc95d5640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0cc409e070 0x7f0cc413b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:50:08.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.022+0000 7f0cc9dd6640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0cc4140b30 0x7f0cc406f520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:55208/0 (socket says 172.21.15.42:55208) 2024-09-16T03:50:08.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.022+0000 7f0cc9dd6640 1 -- 172.21.15.42:0/539771918 learned_addr learned my addr 172.21.15.42:0/539771918 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:50:08.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.022+0000 7f0cc9dd6640 10 monclient(hunting): get_auth_request con 0x7f0cc4140b30 auth_method 0 2024-09-16T03:50:08.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.022+0000 7f0cc9dd6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:50:08.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.022+0000 7f0cc9dd6640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:50:08.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.022+0000 7f0cc9dd6640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:50:08.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.022+0000 7f0cc95d5640 10 monclient(hunting): get_auth_request con 0x7f0cc409e070 auth_method 0 2024-09-16T03:50:08.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.022+0000 7f0cc95d5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:50:08.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.022+0000 7f0cc95d5640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:50:08.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.022+0000 7f0cc95d5640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:50:08.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.022+0000 7f0cc9dd6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:50:08.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.022+0000 7f0cc9dd6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:50:08.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.022+0000 7f0cc9dd6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:50:08.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.022+0000 7f0cc95d5640 1 -- 172.21.15.42:0/539771918 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0cc409e070 msgr2=0x7f0cc413b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-16T03:50:08.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.022+0000 7f0cc95d5640 1 -- 172.21.15.42:0/539771918 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0cc409e070 msgr2=0x7f0cc413b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T03:50:08.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.022+0000 7f0cc95d5640 1 --2- 172.21.15.42:0/539771918 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0cc409e070 0x7f0cc413b1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:50:08.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.022+0000 7f0cc95d5640 1 --2- 172.21.15.42:0/539771918 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0cc409e070 0x7f0cc413b1b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:50:08.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.022+0000 7f0cc9dd6640 10 monclient(hunting): handle_auth_done global_id 764757 payload 824 2024-09-16T03:50:08.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.022+0000 7f0cc8dd4640 1 --2- 172.21.15.42:0/539771918 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0cc40a2f10 0x7f0cc413b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:50:08.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.022+0000 7f0cc9dd6640 1 -- 172.21.15.42:0/539771918 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0cc409e070 msgr2=0x7f0cc413b1b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:50:08.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.022+0000 7f0cc9dd6640 1 --2- 172.21.15.42:0/539771918 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0cc409e070 0x7f0cc413b1b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:50:08.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.022+0000 7f0cc9dd6640 1 -- 172.21.15.42:0/539771918 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0cc40a2f10 msgr2=0x7f0cc413b6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:50:08.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.022+0000 7f0cc9dd6640 1 --2- 172.21.15.42:0/539771918 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0cc40a2f10 0x7f0cc413b6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:50:08.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.022+0000 7f0cc9dd6640 10 monclient: _finish_hunting 0 2024-09-16T03:50:08.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.022+0000 7f0cc9dd6640 1 monclient: found mon.noname-a 2024-09-16T03:50:08.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.022+0000 7f0cc9dd6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:50:08.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.022+0000 7f0cc9dd6640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:50:08.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.022+0000 7f0cc9dd6640 1 -- 172.21.15.42:0/539771918 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0cc4125170 con 0x7f0cc4140b30 2024-09-16T03:50:08.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.022+0000 7f0cc8dd4640 10 monclient: get_auth_request con 0x7f0cc40a2f10 auth_method 0 2024-09-16T03:50:08.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.022+0000 7f0cc8dd4640 1 --2- 172.21.15.42:0/539771918 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0cc40a2f10 0x7f0cc413b6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T03:50:08.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.022+0000 7f0cc9dd6640 1 --2- 172.21.15.42:0/539771918 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0cc4140b30 0x7f0cc406f520 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f0cbc00c480 tx=0x7f0cbc00c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:50:08.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.022+0000 7f0cb5ffb640 10 client.?.objecter ms_handle_connect 0x7f0cc4140b30 2024-09-16T03:50:08.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.022+0000 7f0cb5ffb640 10 client.?.objecter resend_mon_ops 2024-09-16T03:50:08.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.022+0000 7f0cb5ffb640 1 -- 172.21.15.42:0/539771918 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cbc018070 con 0x7f0cc4140b30 2024-09-16T03:50:08.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.022+0000 7f0cb5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:50:08.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.022+0000 7f0cb5ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:50:08.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.022+0000 7f0cb5ffb640 10 monclient: dump: 2024-09-16T03:50:08.043 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:50:08.043 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:50:08.043 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:08.043 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:08.044 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:50:08.044 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:50:08.044 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:50:08.044 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:50:08.044 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:50:08.044 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:50:08.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.022+0000 7f0cb5ffb640 10 monclient: _finish_auth 0 2024-09-16T03:50:08.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.022+0000 7f0cb5ffb640 10 monclient: _check_auth_tickets 2024-09-16T03:50:08.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.022+0000 7f0cb5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:50:08.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.022+0000 7f0cb5ffb640 1 -- 172.21.15.42:0/539771918 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0cbc0074e0 con 0x7f0cc4140b30 2024-09-16T03:50:08.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.022+0000 7f0cb5ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T03:50:08.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.022+0000 7f0cb5ffb640 1 -- 172.21.15.42:0/539771918 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cbc004a30 con 0x7f0cc4140b30 2024-09-16T03:50:08.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.022+0000 7f0cb5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:50:08.045 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.022+0000 7f0cb5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:50:08.045 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.022+0000 7f0cb5ffb640 10 monclient: dump: 2024-09-16T03:50:08.045 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:50:08.045 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:50:08.045 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:08.045 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:08.045 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:50:08.045 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:50:08.045 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:50:08.045 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:50:08.045 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:50:08.045 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:50:08.045 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.022+0000 7f0ccb839640 5 monclient: authenticate success, global_id 764757 2024-09-16T03:50:08.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.022+0000 7f0ccb839640 10 monclient: _renew_subs 2024-09-16T03:50:08.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.022+0000 7f0cb77fe640 4 set_mon_vals no callback set 2024-09-16T03:50:08.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.022+0000 7f0ccb839640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:50:08.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.022+0000 7f0ccb839640 1 -- 172.21.15.42:0/539771918 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0cc4071aa0 con 0x7f0cc4140b30 2024-09-16T03:50:08.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.022+0000 7f0ccb839640 10 client.764757.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:50:08.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.022+0000 7f0ccb839640 10 monclient: _renew_subs 2024-09-16T03:50:08.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.022+0000 7f0ccb839640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:50:08.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.022+0000 7f0ccb839640 1 -- 172.21.15.42:0/539771918 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0cc40711c0 con 0x7f0cc4140b30 2024-09-16T03:50:08.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.022+0000 7f0ccb839640 1 librados: init done 2024-09-16T03:50:08.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.026+0000 7f0ccb839640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:50:08.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.026+0000 7f0ccb839640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:50:08.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.026+0000 7f0ccb839640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:50:08.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.026+0000 7f0ccb839640 1 -- 172.21.15.42:0/539771918 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0cc40a1330 con 0x7f0cc4140b30 2024-09-16T03:50:08.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.026+0000 7f0cb5ffb640 1 -- 172.21.15.42:0/539771918 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f0cbc004bd0 con 0x7f0cc4140b30 2024-09-16T03:50:08.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.026+0000 7f0cb5ffb640 1 --2- 172.21.15.42:0/539771918 >> v2:172.21.15.154:6812/254845342 conn(0x7f0ca8037810 0x7f0ca8039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:50:08.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.026+0000 7f0cb5ffb640 1 -- 172.21.15.42:0/539771918 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(275..275 src has 1..275) ==== 6169+0+0 (secure 0 0 0) 0x7f0cbc0483e0 con 0x7f0cc4140b30 2024-09-16T03:50:08.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.026+0000 7f0cb5ffb640 10 client.764757.objecter ms_dispatch 0x7f0cc4000ba0 osd_map(275..275 src has 1..275) 2024-09-16T03:50:08.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.026+0000 7f0cb5ffb640 3 client.764757.objecter handle_osd_map got epochs [275,275] > 0 2024-09-16T03:50:08.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.026+0000 7f0cb5ffb640 3 client.764757.objecter handle_osd_map decoding full epoch 275 2024-09-16T03:50:08.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.026+0000 7f0cb5ffb640 20 client.764757.objecter dump_active .. 0 homeless 2024-09-16T03:50:08.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.026+0000 7f0cc95d5640 1 --2- 172.21.15.42:0/539771918 >> v2:172.21.15.154:6812/254845342 conn(0x7f0ca8037810 0x7f0ca8039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:50:08.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.026+0000 7f0cc95d5640 10 monclient: get_auth_request con 0x7f0ca8037810 auth_method 0 2024-09-16T03:50:08.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.026+0000 7f0cb5ffb640 10 client.764757.objecter ms_handle_connect 0x7f0ca8037810 2024-09-16T03:50:08.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.026+0000 7f0cc95d5640 1 --2- 172.21.15.42:0/539771918 >> v2:172.21.15.154:6812/254845342 conn(0x7f0ca8037810 0x7f0ca8039cd0 secure :-1 s=READY pgs=3533 cs=0 l=1 rev1=1 crypto rx=0x7f0cc00af000 tx=0x7f0cc00a7070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:50:08.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.030+0000 7f0cb5ffb640 1 -- 172.21.15.42:0/539771918 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f0cbc01ad20 con 0x7f0cc4140b30 2024-09-16T03:50:08.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.030+0000 7f0cb5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:50:08.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.030+0000 7f0cb5ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:50:08.261 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.254+0000 7f0ccb839640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:50:08.261 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.254+0000 7f0ccb839640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:50:08.261 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.254+0000 7f0ccb839640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:50:08.261 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.254+0000 7f0ccb839640 1 -- 172.21.15.42:0/539771918 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f0cc4143ab0 con 0x7f0cc4140b30 2024-09-16T03:50:08.261 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.258+0000 7f0cb5ffb640 1 -- 172.21.15.42:0/539771918 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7f0cbc01bc60 con 0x7f0cc4140b30 2024-09-16T03:50:08.261 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.258+0000 7f0cb5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:50:08.262 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.258+0000 7f0cb5ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:50:08.262 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":72,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:50:08.264 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.258+0000 7f0ccb839640 10 librados: watch_flush enter 2024-09-16T03:50:08.264 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.258+0000 7f0ccb839640 10 librados: watch_flush exit 2024-09-16T03:50:08.265 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.258+0000 7f0ccb839640 20 client.764757.objecter shutdown clearing up homeless session... 2024-09-16T03:50:08.265 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.258+0000 7f0ccb839640 10 client.764757.objecter successfully canceled tick 2024-09-16T03:50:08.265 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.258+0000 7f0ccb839640 1 -- 172.21.15.42:0/539771918 >> v2:172.21.15.154:6812/254845342 conn(0x7f0ca8037810 msgr2=0x7f0ca8039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:50:08.265 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.258+0000 7f0ccb839640 1 --2- 172.21.15.42:0/539771918 >> v2:172.21.15.154:6812/254845342 conn(0x7f0ca8037810 0x7f0ca8039cd0 secure :-1 s=READY pgs=3533 cs=0 l=1 rev1=1 crypto rx=0x7f0cc00af000 tx=0x7f0cc00a7070 comp rx=0 tx=0).stop 2024-09-16T03:50:08.265 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.258+0000 7f0ccb839640 10 monclient: shutdown 2024-09-16T03:50:08.265 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.258+0000 7f0ccb839640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:50:08.265 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.258+0000 7f0ccb839640 1 -- 172.21.15.42:0/539771918 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0cc4140b30 msgr2=0x7f0cc406f520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:50:08.265 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.258+0000 7f0ccb839640 1 --2- 172.21.15.42:0/539771918 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0cc4140b30 0x7f0cc406f520 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f0cbc00c480 tx=0x7f0cbc00c940 comp rx=0 tx=0).stop 2024-09-16T03:50:08.265 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.258+0000 7f0ccb839640 1 -- 172.21.15.42:0/539771918 shutdown_connections 2024-09-16T03:50:08.265 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.258+0000 7f0ccb839640 1 --2- 172.21.15.42:0/539771918 >> v2:172.21.15.154:6812/254845342 conn(0x7f0ca8037810 0x7f0ca8039cd0 unknown :-1 s=CLOSED pgs=3533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:50:08.265 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.258+0000 7f0ccb839640 1 --2- 172.21.15.42:0/539771918 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0cc4140b30 0x7f0cc406f520 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:50:08.265 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.258+0000 7f0ccb839640 1 --2- 172.21.15.42:0/539771918 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0cc40a2f10 0x7f0cc413b6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:50:08.265 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.258+0000 7f0ccb839640 1 --2- 172.21.15.42:0/539771918 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0cc409e070 0x7f0cc413b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:50:08.266 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.258+0000 7f0ccb839640 1 -- 172.21.15.42:0/539771918 >> 172.21.15.42:0/539771918 conn(0x7f0cc40a2420 msgr2=0x7f0cc409e7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:50:08.266 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.258+0000 7f0ccb839640 1 -- 172.21.15.42:0/539771918 shutdown_connections 2024-09-16T03:50:08.266 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.258+0000 7f0ccb839640 1 -- 172.21.15.42:0/539771918 wait complete. 2024-09-16T03:50:08.266 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.258+0000 7f0ccb839640 1 librados: shutdown 2024-09-16T03:50:08.281 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T03:50:08.281 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T03:50:08.281 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T03:50:08.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.406+0000 7f7d8ce20640 10 monclient: get_monmap_and_config 2024-09-16T03:50:08.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.406+0000 7f7d8ce20640 10 monclient: build_initial_monmap 2024-09-16T03:50:08.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.406+0000 7f7d8ce20640 10 monclient: monmap: 2024-09-16T03:50:08.410 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:50:08.410 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:50:08.410 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:50:08.408678+0000 2024-09-16T03:50:08.410 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:50:08.408678+0000 2024-09-16T03:50:08.410 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:50:08.410 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:50:08.411 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:50:08.411 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:50:08.411 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:50:08.411 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:50:08.411 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:50:08.411 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:50:08.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.406+0000 7f7d8ce20640 1 Processor -- start 2024-09-16T03:50:08.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.406+0000 7f7d8ce20640 1 -- start start 2024-09-16T03:50:08.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.406+0000 7f7d8ce20640 10 monclient: init 2024-09-16T03:50:08.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.406+0000 7f7d8ce20640 10 monclient: _reopen_session rank -1 2024-09-16T03:50:08.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.406+0000 7f7d8ce20640 10 monclient: _add_conns ranks=[4,0,2,3,1] 2024-09-16T03:50:08.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.406+0000 7f7d8ce20640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f7d880a39b0 0x7f7d88133b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:50:08.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.406+0000 7f7d8ce20640 10 monclient(hunting): picked mon.noname-c con 0x7f7d880a39b0 addr v2:172.21.15.154:3300/0 2024-09-16T03:50:08.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.406+0000 7f7d8ce20640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7d880a3d90 addr v1:172.21.15.42:6789/0 2024-09-16T03:50:08.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.406+0000 7f7d8ce20640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7d88137710 0x7f7d8813f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:50:08.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.406+0000 7f7d8ce20640 10 monclient(hunting): picked mon.noname-b con 0x7f7d88137710 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:50:08.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.406+0000 7f7d8ce20640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7d8809c140 con 0x7f7d880a3d90 2024-09-16T03:50:08.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.406+0000 7f7d8ce20640 10 monclient(hunting): start opening mon connection 2024-09-16T03:50:08.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.406+0000 7f7d8ce20640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f7d8809c420 con 0x7f7d880a39b0 2024-09-16T03:50:08.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.406+0000 7f7d8ce20640 10 monclient(hunting): start opening mon connection 2024-09-16T03:50:08.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.406+0000 7f7d8ce20640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f7d8806dd60 con 0x7f7d88137710 2024-09-16T03:50:08.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.406+0000 7f7d8ce20640 10 monclient(hunting): _renew_subs 2024-09-16T03:50:08.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.406+0000 7f7d8ce20640 10 monclient(hunting): authenticate will time out at 1455.378201s 2024-09-16T03:50:08.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.406+0000 7f7d86d9d640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7d88137710 0x7f7d8813f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:50:08.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.406+0000 7f7d7ffff640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f7d880a3d90 0x7f7d88137000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:56384/0 (socket says 172.21.15.42:56384) 2024-09-16T03:50:08.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.406+0000 7f7d7ffff640 1 -- 172.21.15.42:0/481039101 learned_addr learned my addr 172.21.15.42:0/481039101 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:50:08.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.406+0000 7f7d86d9d640 10 monclient(hunting): get_auth_request con 0x7f7d88137710 auth_method 0 2024-09-16T03:50:08.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.406+0000 7f7d86d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:50:08.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.406+0000 7f7d86d9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:50:08.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.406+0000 7f7d86d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:50:08.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.406+0000 7f7d86d9d640 1 -- 172.21.15.42:0/481039101 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7d88137710 msgr2=0x7f7d8813f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T03:50:08.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.406+0000 7f7d86d9d640 1 -- 172.21.15.42:0/481039101 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7d88137710 msgr2=0x7f7d8813f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:50:08.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.406+0000 7f7d86d9d640 1 --2- 172.21.15.42:0/481039101 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7d88137710 0x7f7d8813f3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:50:08.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.406+0000 7f7d8659c640 1 --2- 172.21.15.42:0/481039101 >> v2:172.21.15.154:3300/0 conn(0x7f7d880a39b0 0x7f7d88133b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:50:08.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.406+0000 7f7d86d9d640 1 --2- 172.21.15.42:0/481039101 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7d88137710 0x7f7d8813f3a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:50:08.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.406+0000 7f7d8659c640 10 monclient(hunting): get_auth_request con 0x7f7d880a39b0 auth_method 0 2024-09-16T03:50:08.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.406+0000 7f7d8659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:50:08.432 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.406+0000 7f7d8659c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:50:08.432 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.406+0000 7f7d8659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:50:08.432 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.406+0000 7f7d85d9b640 1 -- 172.21.15.42:0/481039101 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3175111432 0 0) 0x7f7d8809c140 con 0x7f7d880a3d90 2024-09-16T03:50:08.432 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.406+0000 7f7d85d9b640 10 monclient(hunting): _negotiate 2024-09-16T03:50:08.432 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.406+0000 7f7d85d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:50:08.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.406+0000 7f7d85d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:50:08.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.406+0000 7f7d85d9b640 10 monclient(hunting): my global_id is 764760 2024-09-16T03:50:08.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.406+0000 7f7d85d9b640 1 -- 172.21.15.42:0/481039101 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7d74003530 con 0x7f7d880a3d90 2024-09-16T03:50:08.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.406+0000 7f7d8659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:50:08.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.406+0000 7f7d8659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:50:08.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.406+0000 7f7d8659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:50:08.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.406+0000 7f7d85d9b640 1 -- 172.21.15.42:0/481039101 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3660151362 0 0) 0x7f7d74003530 con 0x7f7d880a3d90 2024-09-16T03:50:08.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.406+0000 7f7d85d9b640 1 -- 172.21.15.42:0/481039101 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7d88137710 msgr2=0x7f7d8813f3a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:50:08.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.406+0000 7f7d85d9b640 1 --2- 172.21.15.42:0/481039101 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7d88137710 0x7f7d8813f3a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:50:08.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.406+0000 7f7d85d9b640 1 -- 172.21.15.42:0/481039101 >> v2:172.21.15.154:3300/0 conn(0x7f7d880a39b0 msgr2=0x7f7d88133b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:50:08.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.406+0000 7f7d85d9b640 1 --2- 172.21.15.42:0/481039101 >> v2:172.21.15.154:3300/0 conn(0x7f7d880a39b0 0x7f7d88133b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:50:08.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.406+0000 7f7d85d9b640 10 monclient: _finish_hunting 0 2024-09-16T03:50:08.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.406+0000 7f7d85d9b640 1 monclient: found mon.noname-a-legacy 2024-09-16T03:50:08.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.406+0000 7f7d85d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:50:08.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.406+0000 7f7d85d9b640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.42:6789/0 2024-09-16T03:50:08.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.406+0000 7f7d85d9b640 1 -- 172.21.15.42:0/481039101 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d880a5e20 con 0x7f7d880a3d90 2024-09-16T03:50:08.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.406+0000 7f7d85d9b640 10 monclient: _finish_auth 0 2024-09-16T03:50:08.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.406+0000 7f7d85d9b640 10 monclient: _check_auth_tickets 2024-09-16T03:50:08.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.406+0000 7f7d85d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:50:08.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.406+0000 7f7d85d9b640 1 -- 172.21.15.42:0/481039101 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2177413734 0 0) 0x7f7d70001680 con 0x7f7d880a3d90 2024-09-16T03:50:08.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.406+0000 7f7d85d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:50:08.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.406+0000 7f7d85d9b640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T03:50:08.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.406+0000 7f7d85d9b640 10 monclient: dump: 2024-09-16T03:50:08.434 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:50:08.434 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:50:08.434 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:08.435 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:08.435 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:50:08.435 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:50:08.435 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:50:08.435 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:50:08.435 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:50:08.435 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:50:08.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.406+0000 7f7d85d9b640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] but i'm connected to v1:172.21.15.42:6789/0, reconnecting 2024-09-16T03:50:08.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.406+0000 7f7d85d9b640 10 monclient: _reopen_session rank -1 2024-09-16T03:50:08.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.406+0000 7f7d85d9b640 1 -- 172.21.15.42:0/481039101 >> v1:172.21.15.42:6789/0 conn(0x7f7d880a3d90 legacy=0x7f7d88137000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:50:08.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.406+0000 7f7d85d9b640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:50:08.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.406+0000 7f7d85d9b640 1 --2- 172.21.15.42:0/481039101 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7d740039c0 0x7f7d74003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:50:08.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.406+0000 7f7d85d9b640 10 monclient(hunting): picked mon.a con 0x7f7d740039c0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:50:08.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.406+0000 7f7d85d9b640 1 --2- 172.21.15.42:0/481039101 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7d88137710 0x7f7d74004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:50:08.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.406+0000 7f7d85d9b640 10 monclient(hunting): picked mon.c con 0x7f7d88137710 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:50:08.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.406+0000 7f7d85d9b640 1 --2- 172.21.15.42:0/481039101 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7d74004d50 0x7f7d740051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:50:08.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.406+0000 7f7d85d9b640 10 monclient(hunting): picked mon.b con 0x7f7d74004d50 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:50:08.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.406+0000 7f7d85d9b640 10 monclient(hunting): start opening mon connection 2024-09-16T03:50:08.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.406+0000 7f7d85d9b640 1 -- 172.21.15.42:0/481039101 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f7d8809c420 con 0x7f7d740039c0 2024-09-16T03:50:08.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.406+0000 7f7d85d9b640 10 monclient(hunting): start opening mon connection 2024-09-16T03:50:08.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.406+0000 7f7d85d9b640 1 -- 172.21.15.42:0/481039101 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f7d8806dd60 con 0x7f7d74004d50 2024-09-16T03:50:08.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.406+0000 7f7d85d9b640 10 monclient(hunting): start opening mon connection 2024-09-16T03:50:08.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.406+0000 7f7d85d9b640 1 -- 172.21.15.42:0/481039101 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f7d7400bb00 con 0x7f7d88137710 2024-09-16T03:50:08.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.406+0000 7f7d85d9b640 10 monclient(hunting): _renew_subs 2024-09-16T03:50:08.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.406+0000 7f7d85d9b640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:50:08.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.406+0000 7f7d8659c640 1 --2- 172.21.15.42:0/481039101 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7d740039c0 0x7f7d74003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:50:08.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.406+0000 7f7d8659c640 10 monclient(hunting): get_auth_request con 0x7f7d740039c0 auth_method 0 2024-09-16T03:50:08.437 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.406+0000 7f7d8659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:50:08.437 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.406+0000 7f7d8659c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:50:08.437 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.406+0000 7f7d8659c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:50:08.437 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.406+0000 7f7d7ffff640 1 --2- 172.21.15.42:0/481039101 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7d88137710 0x7f7d74004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:50:08.437 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.406+0000 7f7d7ffff640 10 monclient(hunting): get_auth_request con 0x7f7d88137710 auth_method 0 2024-09-16T03:50:08.437 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.406+0000 7f7d7ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:50:08.437 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.406+0000 7f7d7ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:50:08.437 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.406+0000 7f7d7ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:50:08.437 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.406+0000 7f7d8659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:50:08.437 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.406+0000 7f7d8659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:50:08.437 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.406+0000 7f7d8659c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T03:50:08.437 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.406+0000 7f7d7ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:50:08.437 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.406+0000 7f7d7ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:50:08.438 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.406+0000 7f7d7ffff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T03:50:08.441 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.406+0000 7f7d8659c640 10 monclient(hunting): handle_auth_done global_id 764760 payload 293 2024-09-16T03:50:08.441 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.406+0000 7f7d8659c640 1 -- 172.21.15.42:0/481039101 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7d88137710 msgr2=0x7f7d74004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:50:08.441 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.406+0000 7f7d8659c640 1 --2- 172.21.15.42:0/481039101 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7d88137710 0x7f7d74004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:50:08.441 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.406+0000 7f7d8659c640 1 -- 172.21.15.42:0/481039101 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7d74004d50 msgr2=0x7f7d740051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:50:08.441 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.406+0000 7f7d8659c640 1 --2- 172.21.15.42:0/481039101 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7d74004d50 0x7f7d740051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:50:08.441 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.406+0000 7f7d8659c640 10 monclient: _finish_hunting 0 2024-09-16T03:50:08.441 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.406+0000 7f7d8659c640 1 monclient: found mon.a 2024-09-16T03:50:08.442 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.406+0000 7f7d8659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:50:08.442 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.406+0000 7f7d8659c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:50:08.442 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.406+0000 7f7d8659c640 1 -- 172.21.15.42:0/481039101 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d74003530 con 0x7f7d740039c0 2024-09-16T03:50:08.442 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.406+0000 7f7d8659c640 10 monclient: _finish_auth 0 2024-09-16T03:50:08.442 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.406+0000 7f7d8659c640 10 monclient: _check_auth_tickets 2024-09-16T03:50:08.442 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.406+0000 7f7d8659c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:50:08.442 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.406+0000 7f7d8ce20640 5 monclient: authenticate success, global_id 764760 2024-09-16T03:50:08.442 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.406+0000 7f7d8ce20640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:50:08.442 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.410+0000 7f7d8659c640 1 --2- 172.21.15.42:0/481039101 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7d740039c0 0x7f7d74003de0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f7d78009ae0 tx=0x7f7d7801ca90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f53689f5ca6090c6 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:50:08.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.410+0000 7f7d85d9b640 1 -- 172.21.15.42:0/481039101 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d780096b0 con 0x7f7d740039c0 2024-09-16T03:50:08.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.410+0000 7f7d85d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:50:08.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.410+0000 7f7d85d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:50:08.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.410+0000 7f7d85d9b640 10 monclient: dump: 2024-09-16T03:50:08.443 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:50:08.443 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:50:08.443 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:08.443 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:08.443 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:50:08.443 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:50:08.443 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:50:08.443 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:50:08.443 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:50:08.444 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:50:08.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.410+0000 7f7d8ce20640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:50:08.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.410+0000 7f7d85d9b640 1 -- 172.21.15.42:0/481039101 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d780a6470 con 0x7f7d740039c0 2024-09-16T03:50:08.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.410+0000 7f7d85d9b640 10 monclient: handle_config config(0 keys) 2024-09-16T03:50:08.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.410+0000 7f7d8ce20640 10 monclient: get_monmap_and_config success 2024-09-16T03:50:08.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.410+0000 7f7d8ce20640 4 set_mon_vals no callback set 2024-09-16T03:50:08.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.410+0000 7f7d85d9b640 1 -- 172.21.15.42:0/481039101 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d780a6860 con 0x7f7d740039c0 2024-09-16T03:50:08.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.410+0000 7f7d85d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:50:08.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.410+0000 7f7d85d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:50:08.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.410+0000 7f7d85d9b640 10 monclient: dump: 2024-09-16T03:50:08.444 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:50:08.444 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:50:08.444 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:08.445 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:08.445 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:50:08.445 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:50:08.445 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:50:08.445 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:50:08.445 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:50:08.445 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:50:08.445 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.410+0000 7f7d8ce20640 10 monclient: shutdown 2024-09-16T03:50:08.445 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.410+0000 7f7d8ce20640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:50:08.445 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.410+0000 7f7d8ce20640 1 -- 172.21.15.42:0/481039101 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7d740039c0 msgr2=0x7f7d74003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:50:08.445 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.410+0000 7f7d8ce20640 1 --2- 172.21.15.42:0/481039101 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7d740039c0 0x7f7d74003de0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f7d78009ae0 tx=0x7f7d7801ca90 comp rx=0 tx=0).stop 2024-09-16T03:50:08.445 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.410+0000 7f7d86d9d640 1 -- 172.21.15.42:0/481039101 reap_dead start 2024-09-16T03:50:08.445 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.410+0000 7f7d8ce20640 1 -- 172.21.15.42:0/481039101 shutdown_connections 2024-09-16T03:50:08.445 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.410+0000 7f7d8ce20640 1 -- 172.21.15.42:0/481039101 >> 172.21.15.42:0/481039101 conn(0x7f7d880a9fb0 msgr2=0x7f7d880aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:50:08.446 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.410+0000 7f7d8ce20640 1 -- 172.21.15.42:0/481039101 shutdown_connections 2024-09-16T03:50:08.446 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.410+0000 7f7d8ce20640 1 -- 172.21.15.42:0/481039101 wait complete. 2024-09-16T03:50:08.446 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.410+0000 7f7d8ce20640 10 monclient: build_initial_monmap 2024-09-16T03:50:08.446 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.410+0000 7f7d8ce20640 10 monclient: monmap: 2024-09-16T03:50:08.446 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:50:08.446 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:50:08.446 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:50:08.446 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:50:08.446 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:50:08.446 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:50:08.446 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:50:08.446 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:50:08.446 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:50:08.446 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:50:08.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.410+0000 7f7d8ce20640 1 librados: starting msgr at 2024-09-16T03:50:08.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.410+0000 7f7d8ce20640 1 librados: starting objecter 2024-09-16T03:50:08.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.410+0000 7f7d8ce20640 1 Processor -- start 2024-09-16T03:50:08.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.410+0000 7f7d8ce20640 1 -- start start 2024-09-16T03:50:08.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.410+0000 7f7d8ce20640 1 librados: setting wanted keys 2024-09-16T03:50:08.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.410+0000 7f7d8ce20640 1 librados: calling monclient init 2024-09-16T03:50:08.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.410+0000 7f7d8ce20640 10 monclient: init 2024-09-16T03:50:08.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.410+0000 7f7d8ce20640 10 monclient: _reopen_session rank -1 2024-09-16T03:50:08.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.410+0000 7f7d8ce20640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T03:50:08.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.414+0000 7f7d8ce20640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7d740039c0 0x7f7d8813c480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:50:08.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.414+0000 7f7d8ce20640 10 monclient(hunting): picked mon.noname-b con 0x7f7d740039c0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:50:08.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.414+0000 7f7d8ce20640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7d881cfa80 0x7f7d881cff00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:50:08.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.414+0000 7f7d8ce20640 10 monclient(hunting): picked mon.noname-c con 0x7f7d881cfa80 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:50:08.448 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.414+0000 7f7d8ce20640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7d8809c8d0 0x7f7d8809cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:50:08.448 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.414+0000 7f7d8ce20640 10 monclient(hunting): picked mon.noname-a con 0x7f7d8809c8d0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:50:08.448 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.414+0000 7f7d8ce20640 10 monclient(hunting): start opening mon connection 2024-09-16T03:50:08.448 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.414+0000 7f7d8ce20640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f7d8813fdc0 con 0x7f7d8809c8d0 2024-09-16T03:50:08.448 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.414+0000 7f7d8ce20640 10 monclient(hunting): start opening mon connection 2024-09-16T03:50:08.448 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.414+0000 7f7d8ce20640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f7d8813ff40 con 0x7f7d740039c0 2024-09-16T03:50:08.448 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.414+0000 7f7d8ce20640 10 monclient(hunting): start opening mon connection 2024-09-16T03:50:08.448 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.414+0000 7f7d8ce20640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f7d881400c0 con 0x7f7d881cfa80 2024-09-16T03:50:08.448 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.414+0000 7f7d8ce20640 10 monclient(hunting): _renew_subs 2024-09-16T03:50:08.448 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.414+0000 7f7d8ce20640 10 monclient(hunting): authenticate will time out at 1455.384075s 2024-09-16T03:50:08.448 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.414+0000 7f7d7ffff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7d881cfa80 0x7f7d881cff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:50:08.448 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.414+0000 7f7d86d9d640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7d8809c8d0 0x7f7d8809cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:50:08.448 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.414+0000 7f7d86d9d640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7d8809c8d0 0x7f7d8809cd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:55222/0 (socket says 172.21.15.42:55222) 2024-09-16T03:50:08.448 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.414+0000 7f7d7ffff640 10 monclient(hunting): get_auth_request con 0x7f7d881cfa80 auth_method 0 2024-09-16T03:50:08.449 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.414+0000 7f7d86d9d640 1 -- 172.21.15.42:0/833875395 learned_addr learned my addr 172.21.15.42:0/833875395 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:50:08.449 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.414+0000 7f7d7ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:50:08.449 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.414+0000 7f7d7ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:50:08.449 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.414+0000 7f7d7ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:50:08.449 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.414+0000 7f7d86d9d640 10 monclient(hunting): get_auth_request con 0x7f7d8809c8d0 auth_method 0 2024-09-16T03:50:08.449 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.414+0000 7f7d86d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:50:08.449 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.414+0000 7f7d86d9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:50:08.449 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.414+0000 7f7d86d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:50:08.449 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.414+0000 7f7d7ffff640 1 -- 172.21.15.42:0/833875395 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7d881cfa80 msgr2=0x7f7d881cff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-16T03:50:08.449 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.414+0000 7f7d7ffff640 1 -- 172.21.15.42:0/833875395 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7d881cfa80 msgr2=0x7f7d881cff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T03:50:08.449 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.414+0000 7f7d7ffff640 1 --2- 172.21.15.42:0/833875395 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7d881cfa80 0x7f7d881cff00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:50:08.449 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.414+0000 7f7d86d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:50:08.449 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.414+0000 7f7d86d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:50:08.450 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.414+0000 7f7d8659c640 1 --2- 172.21.15.42:0/833875395 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7d740039c0 0x7f7d8813c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:50:08.450 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.414+0000 7f7d86d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:50:08.450 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.414+0000 7f7d7ffff640 1 --2- 172.21.15.42:0/833875395 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7d881cfa80 0x7f7d881cff00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:50:08.450 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.414+0000 7f7d8659c640 10 monclient(hunting): get_auth_request con 0x7f7d740039c0 auth_method 0 2024-09-16T03:50:08.450 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.414+0000 7f7d8659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:50:08.450 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.414+0000 7f7d8659c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:50:08.450 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.414+0000 7f7d8659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:50:08.450 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.414+0000 7f7d86d9d640 10 monclient(hunting): handle_auth_done global_id 764763 payload 824 2024-09-16T03:50:08.450 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.414+0000 7f7d86d9d640 1 -- 172.21.15.42:0/833875395 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7d881cfa80 msgr2=0x7f7d881cff00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:50:08.450 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.414+0000 7f7d86d9d640 1 --2- 172.21.15.42:0/833875395 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7d881cfa80 0x7f7d881cff00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:50:08.450 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.414+0000 7f7d86d9d640 1 -- 172.21.15.42:0/833875395 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7d740039c0 msgr2=0x7f7d8813c480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:50:08.450 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.414+0000 7f7d86d9d640 1 --2- 172.21.15.42:0/833875395 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7d740039c0 0x7f7d8813c480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:50:08.450 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.414+0000 7f7d86d9d640 10 monclient: _finish_hunting 0 2024-09-16T03:50:08.450 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.414+0000 7f7d86d9d640 1 monclient: found mon.noname-a 2024-09-16T03:50:08.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.414+0000 7f7d86d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:50:08.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.414+0000 7f7d86d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:50:08.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.414+0000 7f7d86d9d640 1 -- 172.21.15.42:0/833875395 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d880a8fa0 con 0x7f7d8809c8d0 2024-09-16T03:50:08.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.414+0000 7f7d8659c640 1 --2- 172.21.15.42:0/833875395 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7d740039c0 0x7f7d8813c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T03:50:08.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.414+0000 7f7d86d9d640 1 --2- 172.21.15.42:0/833875395 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7d8809c8d0 0x7f7d8809cd80 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f7d8809bf10 tx=0x7f7d8000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:50:08.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.414+0000 7f7d7effd640 10 client.?.objecter ms_handle_connect 0x7f7d8809c8d0 2024-09-16T03:50:08.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.414+0000 7f7d7effd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:50:08.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.414+0000 7f7d7effd640 1 -- 172.21.15.42:0/833875395 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d8000eb90 con 0x7f7d8809c8d0 2024-09-16T03:50:08.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.414+0000 7f7d7effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:50:08.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.414+0000 7f7d7effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:50:08.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.414+0000 7f7d7effd640 10 monclient: dump: 2024-09-16T03:50:08.451 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:50:08.451 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:50:08.451 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:08.452 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:08.452 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:50:08.452 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:50:08.452 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:50:08.452 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:50:08.452 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:50:08.452 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:50:08.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.414+0000 7f7d7effd640 10 monclient: _finish_auth 0 2024-09-16T03:50:08.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.414+0000 7f7d7effd640 10 monclient: _check_auth_tickets 2024-09-16T03:50:08.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.414+0000 7f7d7effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:50:08.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.414+0000 7f7d8ce20640 5 monclient: authenticate success, global_id 764763 2024-09-16T03:50:08.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.414+0000 7f7d7effd640 1 -- 172.21.15.42:0/833875395 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d8000ed50 con 0x7f7d8809c8d0 2024-09-16T03:50:08.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.414+0000 7f7d7effd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:50:08.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.414+0000 7f7d7effd640 1 -- 172.21.15.42:0/833875395 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d80003be0 con 0x7f7d8809c8d0 2024-09-16T03:50:08.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.414+0000 7f7d7effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:50:08.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.414+0000 7f7d7effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:50:08.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.414+0000 7f7d8559a640 4 set_mon_vals no callback set 2024-09-16T03:50:08.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.414+0000 7f7d7effd640 10 monclient: dump: 2024-09-16T03:50:08.453 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:50:08.453 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:50:08.453 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:08.453 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:08.453 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:50:08.453 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:50:08.453 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:50:08.453 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:50:08.453 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:50:08.453 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:50:08.454 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.414+0000 7f7d8ce20640 10 monclient: _renew_subs 2024-09-16T03:50:08.454 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.414+0000 7f7d8ce20640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:50:08.454 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.414+0000 7f7d8ce20640 1 -- 172.21.15.42:0/833875395 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d88137100 con 0x7f7d8809c8d0 2024-09-16T03:50:08.454 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.414+0000 7f7d8ce20640 10 client.764763.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:50:08.454 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.414+0000 7f7d8ce20640 10 monclient: _renew_subs 2024-09-16T03:50:08.454 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.414+0000 7f7d8ce20640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:50:08.454 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.414+0000 7f7d8ce20640 1 -- 172.21.15.42:0/833875395 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d8809f740 con 0x7f7d8809c8d0 2024-09-16T03:50:08.454 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.414+0000 7f7d8ce20640 1 librados: init done 2024-09-16T03:50:08.454 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.414+0000 7f7d8ce20640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:50:08.454 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.414+0000 7f7d8ce20640 1 --2- 172.21.15.42:0/833875395 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7d8809d2c0 0x7f7d88138130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:50:08.454 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.414+0000 7f7d8ce20640 10 monclient(hunting): start opening mon connection 2024-09-16T03:50:08.454 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.414+0000 7f7d8ce20640 1 -- 172.21.15.42:0/833875395 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f7d8809f980 con 0x7f7d8809d2c0 2024-09-16T03:50:08.454 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.414+0000 7f7d7effd640 1 -- 172.21.15.42:0/833875395 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f7d8001b030 con 0x7f7d8809c8d0 2024-09-16T03:50:08.455 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.414+0000 7f7d8659c640 1 --2- 172.21.15.42:0/833875395 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7d8809d2c0 0x7f7d88138130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:50:08.455 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.414+0000 7f7d8659c640 10 monclient: get_auth_request con 0x7f7d8809d2c0 auth_method 0 2024-09-16T03:50:08.455 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.414+0000 7f7d8659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:50:08.455 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.414+0000 7f7d8659c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:50:08.455 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.414+0000 7f7d8659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:50:08.455 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.414+0000 7f7d8659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:50:08.455 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.414+0000 7f7d8659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:50:08.455 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.414+0000 7f7d8659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:50:08.455 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.414+0000 7f7d7effd640 1 --2- 172.21.15.42:0/833875395 >> v2:172.21.15.154:6812/254845342 conn(0x7f7d68037810 0x7f7d68039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:50:08.455 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.414+0000 7f7d7effd640 1 -- 172.21.15.42:0/833875395 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(276..276 src has 1..276) ==== 6586+0+0 (secure 0 0 0) 0x7f7d80049270 con 0x7f7d8809c8d0 2024-09-16T03:50:08.455 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.414+0000 7f7d7effd640 10 client.764763.objecter ms_dispatch 0x7f7d88000ba0 osd_map(276..276 src has 1..276) 2024-09-16T03:50:08.455 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.414+0000 7f7d7effd640 3 client.764763.objecter handle_osd_map got epochs [276,276] > 0 2024-09-16T03:50:08.455 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.414+0000 7f7d7effd640 3 client.764763.objecter handle_osd_map decoding full epoch 276 2024-09-16T03:50:08.455 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.414+0000 7f7d8659c640 10 monclient(hunting): handle_auth_done global_id 764766 payload 824 2024-09-16T03:50:08.456 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.414+0000 7f7d7effd640 20 client.764763.objecter dump_active .. 0 homeless 2024-09-16T03:50:08.456 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.414+0000 7f7d8659c640 1 -- 172.21.15.42:0/833875395 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7d88133b80 con 0x7f7d8809d2c0 2024-09-16T03:50:08.456 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.414+0000 7f7d7ffff640 1 --2- 172.21.15.42:0/833875395 >> v2:172.21.15.154:6812/254845342 conn(0x7f7d68037810 0x7f7d68039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:50:08.456 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.414+0000 7f7d7effd640 10 client.764763.objecter ms_handle_connect 0x7f7d8809d2c0 2024-09-16T03:50:08.456 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.414+0000 7f7d7effd640 10 client.764763.objecter resend_mon_ops 2024-09-16T03:50:08.456 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.414+0000 7f7d8659c640 1 --2- 172.21.15.42:0/833875395 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7d8809d2c0 0x7f7d88138130 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f7d78009ab0 tx=0x7f7d78019830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:50:08.456 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.414+0000 7f7d7ffff640 10 monclient: get_auth_request con 0x7f7d68037810 auth_method 0 2024-09-16T03:50:08.456 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.414+0000 7f7d7effd640 1 -- 172.21.15.42:0/833875395 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d780a6990 con 0x7f7d8809d2c0 2024-09-16T03:50:08.456 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.414+0000 7f7d7effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:50:08.456 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.414+0000 7f7d7effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:50:08.456 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.414+0000 7f7d7effd640 10 monclient: dump: 2024-09-16T03:50:08.456 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:50:08.456 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:50:08.456 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:08.457 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:08.457 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:50:08.457 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:50:08.457 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:50:08.457 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:50:08.457 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:50:08.457 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:50:08.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.414+0000 7f7d7ffff640 1 -- 172.21.15.42:0/833875395 >> v2:172.21.15.154:6812/254845342 conn(0x7f7d68037810 msgr2=0x7f7d68039cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-16T03:50:08.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.414+0000 7f7d7ffff640 1 -- 172.21.15.42:0/833875395 >> v2:172.21.15.154:6812/254845342 conn(0x7f7d68037810 msgr2=0x7f7d68039cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T03:50:08.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.414+0000 7f7d7ffff640 1 --2- 172.21.15.42:0/833875395 >> v2:172.21.15.154:6812/254845342 conn(0x7f7d68037810 0x7f7d68039cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:50:08.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.414+0000 7f7d7ffff640 1 --2- 172.21.15.42:0/833875395 >> v2:172.21.15.154:6812/254845342 conn(0x7f7d68037810 0x7f7d68039cd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:50:08.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.414+0000 7f7d7effd640 1 -- 172.21.15.42:0/833875395 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f7d780a6990 con 0x7f7d8809d2c0 2024-09-16T03:50:08.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.414+0000 7f7d7effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:50:08.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.414+0000 7f7d7effd640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:50:08.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.414+0000 7f7d7effd640 1 -- 172.21.15.42:0/833875395 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7d8809d2c0 msgr2=0x7f7d88138130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:50:08.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.414+0000 7f7d7effd640 1 --2- 172.21.15.42:0/833875395 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7d8809d2c0 0x7f7d88138130 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f7d78009ab0 tx=0x7f7d78019830 comp rx=0 tx=0).stop 2024-09-16T03:50:08.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.414+0000 7f7d7effd640 1 -- 172.21.15.42:0/833875395 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7d8809d2c0 msgr2=0x7f7d88138130 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:50:08.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.414+0000 7f7d7effd640 1 --2- 172.21.15.42:0/833875395 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7d8809d2c0 0x7f7d88138130 secure :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f7d78009ab0 tx=0x7f7d78019830 comp rx=0 tx=0).stop 2024-09-16T03:50:08.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.422+0000 7f7d8ce20640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:50:08.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.422+0000 7f7d8ce20640 1 --2- 172.21.15.42:0/833875395 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7d88138670 0x7f7d88138a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:50:08.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.422+0000 7f7d8ce20640 10 monclient(hunting): start opening mon connection 2024-09-16T03:50:08.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.422+0000 7f7d8ce20640 1 -- 172.21.15.42:0/833875395 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f7d881342d0 con 0x7f7d88138670 2024-09-16T03:50:08.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.422+0000 7f7d8659c640 1 --2- 172.21.15.42:0/833875395 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7d88138670 0x7f7d88138a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:50:08.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.422+0000 7f7d8659c640 10 monclient: get_auth_request con 0x7f7d88138670 auth_method 0 2024-09-16T03:50:08.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.422+0000 7f7d8659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:50:08.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.422+0000 7f7d8659c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:50:08.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.422+0000 7f7d8659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:50:08.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.422+0000 7f7d8659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:50:08.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.422+0000 7f7d8659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:50:08.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.422+0000 7f7d8659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:50:08.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.422+0000 7f7d8659c640 10 monclient(hunting): handle_auth_done global_id 764769 payload 824 2024-09-16T03:50:08.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.422+0000 7f7d8659c640 1 -- 172.21.15.42:0/833875395 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f7d880a5750 con 0x7f7d88138670 2024-09-16T03:50:08.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.422+0000 7f7d8659c640 1 --2- 172.21.15.42:0/833875395 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7d88138670 0x7f7d88138a80 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f7d78009ae0 tx=0x7f7d74005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:50:08.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.422+0000 7f7d7effd640 10 client.764763.objecter ms_handle_connect 0x7f7d88138670 2024-09-16T03:50:08.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.422+0000 7f7d7effd640 10 client.764763.objecter resend_mon_ops 2024-09-16T03:50:08.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.422+0000 7f7d7effd640 1 -- 172.21.15.42:0/833875395 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d780c3920 con 0x7f7d88138670 2024-09-16T03:50:08.460 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.422+0000 7f7d7effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:50:08.460 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.422+0000 7f7d7effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:50:08.460 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.422+0000 7f7d7effd640 10 monclient: dump: 2024-09-16T03:50:08.460 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:50:08.460 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:50:08.460 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:08.460 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:08.460 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:50:08.460 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:50:08.460 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:50:08.460 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:50:08.460 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:50:08.460 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:50:08.460 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.422+0000 7f7d7effd640 1 -- 172.21.15.42:0/833875395 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4881 (secure 0 0 0) 0x7f7d780c3920 con 0x7f7d88138670 2024-09-16T03:50:08.461 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.422+0000 7f7d7effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:50:08.461 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.422+0000 7f7d7effd640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:50:08.461 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.422+0000 7f7d7effd640 1 -- 172.21.15.42:0/833875395 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7d88138670 msgr2=0x7f7d88138a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:50:08.461 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.422+0000 7f7d7effd640 1 --2- 172.21.15.42:0/833875395 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7d88138670 0x7f7d88138a80 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f7d78009ae0 tx=0x7f7d74005840 comp rx=0 tx=0).stop 2024-09-16T03:50:08.461 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.422+0000 7f7d7effd640 1 -- 172.21.15.42:0/833875395 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7d88138670 msgr2=0x7f7d88138a80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:50:08.461 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.422+0000 7f7d7effd640 1 --2- 172.21.15.42:0/833875395 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7d88138670 0x7f7d88138a80 secure :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f7d78009ae0 tx=0x7f7d74005840 comp rx=0 tx=0).stop 2024-09-16T03:50:08.461 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:50:08.461 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:50:08.461 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:50:08.462 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "leader", 2024-09-16T03:50:08.462 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 72, 2024-09-16T03:50:08.462 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 199214, 2024-09-16T03:50:08.462 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:50:08.462 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:50:08.462 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T03:50:08.462 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T03:50:08.462 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:50:08.462 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 1, 2024-09-16T03:50:08.462 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:50:08.462 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:50:08.462 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:50:08.462 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:50:08.462 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:50:08.463 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:50:08.463 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:50:08.463 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:50:08.463 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:50:08.463 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:50:08.463 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:50:08.463 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:50:08.463 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:50:08.463 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:50:08.463 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:50:08.463 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:50:08.463 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:50:08.463 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:50:08.463 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:50:08.464 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:50:08.464 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:50:08.464 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:50:08.464 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:50:08.464 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:50:08.464 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:50:08.464 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:50:08.464 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:50:08.464 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:50:08.464 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:50:08.464 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:50:08.464 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:50:08.464 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:50:08.464 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:50:08.465 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:50:08.465 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:50:08.465 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:50:08.465 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:50:08.465 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:50:08.465 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:50:08.465 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:50:08.465 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:50:08.465 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:50:08.465 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:50:08.465 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:50:08.465 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:50:08.465 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:50:08.466 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:50:08.466 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:50:08.466 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:50:08.466 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:50:08.466 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:50:08.466 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:50:08.466 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:50:08.466 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:50:08.466 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:50:08.466 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:50:08.466 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:50:08.466 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:50:08.466 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:50:08.466 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:50:08.467 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:50:08.467 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:50:08.467 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:50:08.467 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:50:08.467 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:50:08.467 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:50:08.467 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:50:08.467 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:50:08.467 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:50:08.467 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:50:08.467 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:50:08.467 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:50:08.467 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:50:08.467 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:50:08.468 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:50:08.468 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:50:08.468 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:50:08.468 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:50:08.468 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:50:08.468 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:50:08.468 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:50:08.468 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:50:08.468 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:50:08.468 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:50:08.468 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:50:08.468 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:50:08.468 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:50:08.469 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:50:08.469 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:50:08.469 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:50:08.469 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:50:08.469 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:50:08.469 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:50:08.469 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:50:08.469 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:50:08.469 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:50:08.469 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:50:08.469 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:50:08.469 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:50:08.469 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:50:08.470 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:50:08.470 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:50:08.470 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:50:08.470 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:50:08.470 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:50:08.470 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:50:08.470 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:50:08.470 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:50:08.470 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:50:08.470 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:50:08.470 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:50:08.470 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:50:08.470 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:50:08.470 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:50:08.471 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:50:08.471 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:50:08.471 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:50:08.471 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:50:08.471 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:50:08.471 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:50:08.471 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:50:08.471 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:50:08.471 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:50:08.471 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:50:08.471 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:50:08.471 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:50:08.471 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:50:08.471 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:50:08.472 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:50:08.472 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:50:08.472 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:50:08.472 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:50:08.472 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:50:08.472 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:50:08.472 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:50:08.472 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:50:08.472 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:50:08.472 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:50:08.472 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:50:08.472 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:50:08.472 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:50:08.473 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T03:50:08.473 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:50:08.473 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:50:08.473 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:50:08.473 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 6 2024-09-16T03:50:08.473 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:50:08.473 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:50:08.473 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:50:08.473 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:50:08.473 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:50:08.473 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:50:08.473 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 8 2024-09-16T03:50:08.473 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:50:08.473 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:50:08.473 INFO:teuthology.orchestra.run.smithi042.stdout: "mgr": [ 2024-09-16T03:50:08.474 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:50:08.474 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:50:08.474 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:50:08.474 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:50:08.474 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:50:08.474 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:50:08.474 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:50:08.474 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:50:08.474 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:50:08.476 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.422+0000 7f7d8ce20640 10 librados: watch_flush enter 2024-09-16T03:50:08.476 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.422+0000 7f7d8ce20640 10 librados: watch_flush exit 2024-09-16T03:50:08.476 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.422+0000 7f7d8ce20640 20 client.764763.objecter shutdown clearing up homeless session... 2024-09-16T03:50:08.476 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.422+0000 7f7d8ce20640 10 client.764763.objecter successfully canceled tick 2024-09-16T03:50:08.476 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.422+0000 7f7d8ce20640 1 -- 172.21.15.42:0/833875395 >> v2:172.21.15.154:6812/254845342 conn(0x7f7d68037810 msgr2=0x7f7d68039cd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:50:08.476 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.422+0000 7f7d8ce20640 1 --2- 172.21.15.42:0/833875395 >> v2:172.21.15.154:6812/254845342 conn(0x7f7d68037810 0x7f7d68039cd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:50:08.476 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.422+0000 7f7d8ce20640 10 monclient: shutdown 2024-09-16T03:50:08.476 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.422+0000 7f7d8ce20640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:50:08.476 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.422+0000 7f7d8ce20640 1 -- 172.21.15.42:0/833875395 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7d8809c8d0 msgr2=0x7f7d8809cd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:50:08.476 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.422+0000 7f7d86d9d640 1 -- 172.21.15.42:0/833875395 reap_dead start 2024-09-16T03:50:08.477 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.422+0000 7f7d8ce20640 1 --2- 172.21.15.42:0/833875395 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7d8809c8d0 0x7f7d8809cd80 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f7d8809bf10 tx=0x7f7d8000e4b0 comp rx=0 tx=0).stop 2024-09-16T03:50:08.477 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.422+0000 7f7d8ce20640 1 -- 172.21.15.42:0/833875395 shutdown_connections 2024-09-16T03:50:08.477 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.422+0000 7f7d8ce20640 1 --2- 172.21.15.42:0/833875395 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7d8809c8d0 0x7f7d8809cd80 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:50:08.477 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.422+0000 7f7d8ce20640 1 -- 172.21.15.42:0/833875395 >> 172.21.15.42:0/833875395 conn(0x7f7d880a9fb0 msgr2=0x7f7d880a79b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:50:08.477 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.422+0000 7f7d8ce20640 1 -- 172.21.15.42:0/833875395 shutdown_connections 2024-09-16T03:50:08.477 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.422+0000 7f7d8ce20640 1 -- 172.21.15.42:0/833875395 wait complete. 2024-09-16T03:50:08.477 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.422+0000 7f7d8ce20640 1 librados: shutdown 2024-09-16T03:50:08.477 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:50:08.477 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T03:50:08.478 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T03:50:08.598 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.594+0000 7fd65c347640 10 monclient: get_monmap_and_config 2024-09-16T03:50:08.598 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.594+0000 7fd65c347640 10 monclient: build_initial_monmap 2024-09-16T03:50:08.598 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.594+0000 7fd65c347640 10 monclient: monmap: 2024-09-16T03:50:08.598 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:50:08.598 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:50:08.598 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:50:08.596775+0000 2024-09-16T03:50:08.598 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:50:08.596775+0000 2024-09-16T03:50:08.598 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:50:08.598 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:50:08.598 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:50:08.599 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:50:08.599 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:50:08.599 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:50:08.599 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:50:08.599 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:50:08.604 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.594+0000 7fd65c347640 1 Processor -- start 2024-09-16T03:50:08.604 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.594+0000 7fd65c347640 1 -- start start 2024-09-16T03:50:08.604 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.594+0000 7fd65c347640 10 monclient: init 2024-09-16T03:50:08.604 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.594+0000 7fd65c347640 10 monclient: _reopen_session rank -1 2024-09-16T03:50:08.605 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.594+0000 7fd65c347640 10 monclient: _add_conns ranks=[0,4,2,3,1] 2024-09-16T03:50:08.605 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.594+0000 7fd65c347640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd6540a1330 addr v1:172.21.15.42:6789/0 2024-09-16T03:50:08.605 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.594+0000 7fd65c347640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fd65406dac0 0x7fd65413a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:50:08.605 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.594+0000 7fd65c347640 10 monclient(hunting): picked mon.noname-c con 0x7fd65406dac0 addr v2:172.21.15.154:3300/0 2024-09-16T03:50:08.605 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.594+0000 7fd65c347640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd6540a1e20 0x7fd65413c890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:50:08.605 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.594+0000 7fd65c347640 10 monclient(hunting): picked mon.noname-b con 0x7fd6540a1e20 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:50:08.605 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.594+0000 7fd65c347640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd65409c090 con 0x7fd6540a1330 2024-09-16T03:50:08.605 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.594+0000 7fd65c347640 10 monclient(hunting): start opening mon connection 2024-09-16T03:50:08.605 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.594+0000 7fd65c347640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fd65409c890 con 0x7fd65406dac0 2024-09-16T03:50:08.605 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.594+0000 7fd65c347640 10 monclient(hunting): start opening mon connection 2024-09-16T03:50:08.605 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.594+0000 7fd65c347640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fd65409f610 con 0x7fd6540a1e20 2024-09-16T03:50:08.605 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.594+0000 7fd65c347640 10 monclient(hunting): _renew_subs 2024-09-16T03:50:08.605 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.594+0000 7fd65c347640 10 monclient(hunting): authenticate will time out at 1455.566287s 2024-09-16T03:50:08.606 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.594+0000 7fd65a0e3640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fd6540a1330 0x7fd6540a1710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:56394/0 (socket says 172.21.15.42:56394) 2024-09-16T03:50:08.606 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.594+0000 7fd65a0e3640 1 -- 172.21.15.42:0/3238588433 learned_addr learned my addr 172.21.15.42:0/3238588433 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:50:08.606 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.594+0000 7fd65a8e4640 1 --2- 172.21.15.42:0/3238588433 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd6540a1e20 0x7fd65413c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:50:08.606 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.594+0000 7fd65a8e4640 10 monclient(hunting): get_auth_request con 0x7fd6540a1e20 auth_method 0 2024-09-16T03:50:08.606 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.594+0000 7fd65a8e4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:50:08.606 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.594+0000 7fd65a8e4640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:50:08.606 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.594+0000 7fd65a8e4640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:50:08.606 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.594+0000 7fd65a8e4640 1 -- 172.21.15.42:0/3238588433 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd6540a1e20 msgr2=0x7fd65413c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T03:50:08.606 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.594+0000 7fd65a8e4640 1 -- 172.21.15.42:0/3238588433 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd6540a1e20 msgr2=0x7fd65413c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:50:08.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.594+0000 7fd65a8e4640 1 --2- 172.21.15.42:0/3238588433 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd6540a1e20 0x7fd65413c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:50:08.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.594+0000 7fd6598e2640 1 --2- 172.21.15.42:0/3238588433 >> v2:172.21.15.154:3300/0 conn(0x7fd65406dac0 0x7fd65413a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:50:08.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.594+0000 7fd6598e2640 10 monclient(hunting): get_auth_request con 0x7fd65406dac0 auth_method 0 2024-09-16T03:50:08.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.594+0000 7fd6598e2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:50:08.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.594+0000 7fd6598e2640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:50:08.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.594+0000 7fd6598e2640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:50:08.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.594+0000 7fd65a8e4640 1 --2- 172.21.15.42:0/3238588433 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd6540a1e20 0x7fd65413c890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:50:08.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.594+0000 7fd6590e1640 1 -- 172.21.15.42:0/3238588433 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3369104254 0 0) 0x7fd65409c090 con 0x7fd6540a1330 2024-09-16T03:50:08.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.594+0000 7fd6590e1640 10 monclient(hunting): _negotiate 2024-09-16T03:50:08.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.594+0000 7fd6590e1640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:50:08.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.594+0000 7fd6590e1640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:50:08.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.594+0000 7fd6590e1640 10 monclient(hunting): my global_id is 764772 2024-09-16T03:50:08.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.594+0000 7fd6590e1640 1 -- 172.21.15.42:0/3238588433 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd640003530 con 0x7fd6540a1330 2024-09-16T03:50:08.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.594+0000 7fd6598e2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:50:08.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.594+0000 7fd6598e2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:50:08.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.594+0000 7fd6598e2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:50:08.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.594+0000 7fd6590e1640 1 -- 172.21.15.42:0/3238588433 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2501802427 0 0) 0x7fd640003530 con 0x7fd6540a1330 2024-09-16T03:50:08.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.594+0000 7fd6590e1640 1 -- 172.21.15.42:0/3238588433 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd6540a1e20 msgr2=0x7fd65413c890 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:50:08.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.594+0000 7fd6590e1640 1 --2- 172.21.15.42:0/3238588433 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd6540a1e20 0x7fd65413c890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:50:08.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.594+0000 7fd6590e1640 1 -- 172.21.15.42:0/3238588433 >> v2:172.21.15.154:3300/0 conn(0x7fd65406dac0 msgr2=0x7fd65413a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:50:08.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.594+0000 7fd6590e1640 1 --2- 172.21.15.42:0/3238588433 >> v2:172.21.15.154:3300/0 conn(0x7fd65406dac0 0x7fd65413a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:50:08.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.594+0000 7fd6590e1640 10 monclient: _finish_hunting 0 2024-09-16T03:50:08.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.594+0000 7fd6590e1640 1 monclient: found mon.noname-a-legacy 2024-09-16T03:50:08.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.594+0000 7fd6590e1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:50:08.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.594+0000 7fd6590e1640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.42:6789/0 2024-09-16T03:50:08.609 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.594+0000 7fd6590e1640 1 -- 172.21.15.42:0/3238588433 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd65409f8f0 con 0x7fd6540a1330 2024-09-16T03:50:08.609 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.594+0000 7fd6590e1640 10 monclient: _finish_auth 0 2024-09-16T03:50:08.609 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.594+0000 7fd6590e1640 10 monclient: _check_auth_tickets 2024-09-16T03:50:08.609 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.594+0000 7fd6590e1640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:50:08.609 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.594+0000 7fd6590e1640 1 -- 172.21.15.42:0/3238588433 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2177413734 0 0) 0x7fd650001680 con 0x7fd6540a1330 2024-09-16T03:50:08.609 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.594+0000 7fd65c347640 5 monclient: authenticate success, global_id 764772 2024-09-16T03:50:08.609 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.594+0000 7fd65c347640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:50:08.609 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.594+0000 7fd6598e2640 1 --2- 172.21.15.42:0/3238588433 >> v2:172.21.15.154:3300/0 conn(0x7fd65406dac0 0x7fd65413a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:50:08.609 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.594+0000 7fd6590e1640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:50:08.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.594+0000 7fd6590e1640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T03:50:08.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.594+0000 7fd6590e1640 10 monclient: dump: 2024-09-16T03:50:08.610 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:50:08.610 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:50:08.610 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:08.610 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:08.610 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:50:08.610 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:50:08.610 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:50:08.610 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:50:08.610 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:50:08.610 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:50:08.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.594+0000 7fd6590e1640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] but i'm connected to v1:172.21.15.42:6789/0, reconnecting 2024-09-16T03:50:08.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.594+0000 7fd6590e1640 10 monclient: _reopen_session rank -1 2024-09-16T03:50:08.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.594+0000 7fd6590e1640 1 -- 172.21.15.42:0/3238588433 >> v1:172.21.15.42:6789/0 conn(0x7fd6540a1330 legacy=0x7fd6540a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:50:08.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.594+0000 7fd6590e1640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:50:08.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.594+0000 7fd6590e1640 1 --2- 172.21.15.42:0/3238588433 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd6400039c0 0x7fd640003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:50:08.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.594+0000 7fd6590e1640 10 monclient(hunting): picked mon.a con 0x7fd6400039c0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:50:08.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.594+0000 7fd6590e1640 1 --2- 172.21.15.42:0/3238588433 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd6540a1e20 0x7fd640004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:50:08.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.594+0000 7fd6590e1640 10 monclient(hunting): picked mon.c con 0x7fd6540a1e20 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:50:08.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.594+0000 7fd6590e1640 1 --2- 172.21.15.42:0/3238588433 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd640004d50 0x7fd6400051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:50:08.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.594+0000 7fd6590e1640 10 monclient(hunting): picked mon.b con 0x7fd640004d50 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:50:08.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.594+0000 7fd6590e1640 10 monclient(hunting): start opening mon connection 2024-09-16T03:50:08.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.594+0000 7fd6590e1640 1 -- 172.21.15.42:0/3238588433 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fd65409c890 con 0x7fd6400039c0 2024-09-16T03:50:08.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.594+0000 7fd6590e1640 10 monclient(hunting): start opening mon connection 2024-09-16T03:50:08.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.594+0000 7fd6590e1640 1 -- 172.21.15.42:0/3238588433 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd65409f610 con 0x7fd640004d50 2024-09-16T03:50:08.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.594+0000 7fd6590e1640 10 monclient(hunting): start opening mon connection 2024-09-16T03:50:08.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.594+0000 7fd6590e1640 1 -- 172.21.15.42:0/3238588433 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fd64000ba80 con 0x7fd6540a1e20 2024-09-16T03:50:08.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.594+0000 7fd6590e1640 10 monclient(hunting): _renew_subs 2024-09-16T03:50:08.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.594+0000 7fd6590e1640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:50:08.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.594+0000 7fd65c347640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-16T03:50:08.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.594+0000 7fd65a0e3640 1 --2- 172.21.15.42:0/3238588433 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd6400039c0 0x7fd640003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:50:08.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.594+0000 7fd6598e2640 1 --2- 172.21.15.42:0/3238588433 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd6540a1e20 0x7fd640004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:50:08.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.594+0000 7fd65a0e3640 10 monclient(hunting): get_auth_request con 0x7fd6400039c0 auth_method 0 2024-09-16T03:50:08.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.594+0000 7fd65a0e3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:50:08.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.594+0000 7fd65a0e3640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:50:08.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.594+0000 7fd65a0e3640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:50:08.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.594+0000 7fd6598e2640 10 monclient(hunting): get_auth_request con 0x7fd6540a1e20 auth_method 0 2024-09-16T03:50:08.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.594+0000 7fd6598e2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:50:08.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.594+0000 7fd6598e2640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:50:08.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.594+0000 7fd6598e2640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:50:08.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.594+0000 7fd65a0e3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:50:08.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.594+0000 7fd65a0e3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:50:08.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.594+0000 7fd65a0e3640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T03:50:08.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.594+0000 7fd6598e2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:50:08.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.594+0000 7fd6598e2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:50:08.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.594+0000 7fd6598e2640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T03:50:08.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.594+0000 7fd65a0e3640 10 monclient(hunting): handle_auth_done global_id 764772 payload 293 2024-09-16T03:50:08.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.594+0000 7fd65a0e3640 1 -- 172.21.15.42:0/3238588433 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd6540a1e20 msgr2=0x7fd640004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:50:08.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.594+0000 7fd65a0e3640 1 --2- 172.21.15.42:0/3238588433 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd6540a1e20 0x7fd640004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:50:08.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.594+0000 7fd65a0e3640 1 -- 172.21.15.42:0/3238588433 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd640004d50 msgr2=0x7fd6400051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:50:08.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.594+0000 7fd65a0e3640 1 --2- 172.21.15.42:0/3238588433 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd640004d50 0x7fd6400051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:50:08.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.594+0000 7fd65a0e3640 10 monclient: _finish_hunting 0 2024-09-16T03:50:08.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.594+0000 7fd65a0e3640 1 monclient: found mon.a 2024-09-16T03:50:08.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.594+0000 7fd65a0e3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:50:08.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.594+0000 7fd65a0e3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:50:08.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.594+0000 7fd65a0e3640 1 -- 172.21.15.42:0/3238588433 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd640003530 con 0x7fd6400039c0 2024-09-16T03:50:08.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.594+0000 7fd65a0e3640 10 monclient: _finish_auth 0 2024-09-16T03:50:08.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.594+0000 7fd65a0e3640 10 monclient: _check_auth_tickets 2024-09-16T03:50:08.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.594+0000 7fd65a0e3640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:50:08.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.598+0000 7fd65a0e3640 1 --2- 172.21.15.42:0/3238588433 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd6400039c0 0x7fd640003de0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fd65000c210 tx=0x7fd65001cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c8c91f6ef555e66 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:50:08.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.598+0000 7fd6590e1640 1 -- 172.21.15.42:0/3238588433 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd650007d60 con 0x7fd6400039c0 2024-09-16T03:50:08.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.598+0000 7fd6590e1640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:50:08.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.598+0000 7fd6590e1640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:50:08.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.598+0000 7fd6590e1640 10 monclient: dump: 2024-09-16T03:50:08.614 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:50:08.614 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:50:08.614 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:08.615 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:08.615 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:50:08.615 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:50:08.615 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:50:08.615 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:50:08.615 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:50:08.615 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:50:08.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.598+0000 7fd65c347640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:50:08.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.598+0000 7fd6590e1640 1 -- 172.21.15.42:0/3238588433 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd6500a9020 con 0x7fd6400039c0 2024-09-16T03:50:08.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.598+0000 7fd6590e1640 10 monclient: handle_config config(0 keys) 2024-09-16T03:50:08.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.598+0000 7fd65c347640 10 monclient: get_monmap_and_config success 2024-09-16T03:50:08.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.598+0000 7fd6590e1640 1 -- 172.21.15.42:0/3238588433 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6500a5b90 con 0x7fd6400039c0 2024-09-16T03:50:08.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.598+0000 7fd65c347640 4 set_mon_vals no callback set 2024-09-16T03:50:08.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.598+0000 7fd6590e1640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:50:08.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.598+0000 7fd6590e1640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:50:08.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.598+0000 7fd6590e1640 10 monclient: dump: 2024-09-16T03:50:08.616 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:50:08.616 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:50:08.616 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:08.616 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:08.616 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:50:08.616 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:50:08.616 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:50:08.616 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:50:08.616 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:50:08.616 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:50:08.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.598+0000 7fd65c347640 10 monclient: shutdown 2024-09-16T03:50:08.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.598+0000 7fd65c347640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:50:08.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.598+0000 7fd65c347640 1 -- 172.21.15.42:0/3238588433 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd6400039c0 msgr2=0x7fd640003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:50:08.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.598+0000 7fd65c347640 1 --2- 172.21.15.42:0/3238588433 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd6400039c0 0x7fd640003de0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fd65000c210 tx=0x7fd65001cc50 comp rx=0 tx=0).stop 2024-09-16T03:50:08.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.598+0000 7fd65a8e4640 1 -- 172.21.15.42:0/3238588433 reap_dead start 2024-09-16T03:50:08.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.598+0000 7fd65c347640 1 -- 172.21.15.42:0/3238588433 shutdown_connections 2024-09-16T03:50:08.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.598+0000 7fd65c347640 1 -- 172.21.15.42:0/3238588433 >> 172.21.15.42:0/3238588433 conn(0x7fd6540a2420 msgr2=0x7fd6540a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:50:08.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.598+0000 7fd65c347640 1 -- 172.21.15.42:0/3238588433 shutdown_connections 2024-09-16T03:50:08.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.598+0000 7fd65c347640 1 -- 172.21.15.42:0/3238588433 wait complete. 2024-09-16T03:50:08.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.598+0000 7fd65c347640 10 monclient: build_initial_monmap 2024-09-16T03:50:08.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.598+0000 7fd65c347640 10 monclient: monmap: 2024-09-16T03:50:08.618 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:50:08.618 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:50:08.618 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:50:08.618 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:50:08.618 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:50:08.618 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:50:08.618 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:50:08.618 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:50:08.618 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:50:08.618 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:50:08.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.598+0000 7fd65c347640 1 librados: starting msgr at 2024-09-16T03:50:08.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.598+0000 7fd65c347640 1 librados: starting objecter 2024-09-16T03:50:08.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.598+0000 7fd65c347640 1 Processor -- start 2024-09-16T03:50:08.619 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.598+0000 7fd65c347640 1 -- start start 2024-09-16T03:50:08.619 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.598+0000 7fd65c347640 1 librados: setting wanted keys 2024-09-16T03:50:08.619 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.598+0000 7fd65c347640 1 librados: calling monclient init 2024-09-16T03:50:08.619 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.598+0000 7fd65c347640 10 monclient: init 2024-09-16T03:50:08.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.602+0000 7fd65c347640 10 monclient: _reopen_session rank -1 2024-09-16T03:50:08.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.602+0000 7fd65c347640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:50:08.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.602+0000 7fd65c347640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd6400039c0 0x7fd654130b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:50:08.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.602+0000 7fd65c347640 10 monclient(hunting): picked mon.noname-c con 0x7fd6400039c0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:50:08.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.602+0000 7fd65c347640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd65411bf70 0x7fd65411c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:50:08.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.602+0000 7fd65c347640 10 monclient(hunting): picked mon.noname-b con 0x7fd65411bf70 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:50:08.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.602+0000 7fd65c347640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd65411c910 0x7fd6541d5410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:50:08.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.602+0000 7fd65c347640 10 monclient(hunting): picked mon.noname-a con 0x7fd65411c910 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:50:08.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.602+0000 7fd65c347640 10 monclient(hunting): start opening mon connection 2024-09-16T03:50:08.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.602+0000 7fd65c347640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fd65413d0a0 con 0x7fd65411c910 2024-09-16T03:50:08.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.602+0000 7fd65c347640 10 monclient(hunting): start opening mon connection 2024-09-16T03:50:08.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.602+0000 7fd65c347640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd65409ef80 con 0x7fd65411bf70 2024-09-16T03:50:08.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.602+0000 7fd65c347640 10 monclient(hunting): start opening mon connection 2024-09-16T03:50:08.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.602+0000 7fd65c347640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fd65413d220 con 0x7fd6400039c0 2024-09-16T03:50:08.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.602+0000 7fd65c347640 10 monclient(hunting): _renew_subs 2024-09-16T03:50:08.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.602+0000 7fd65c347640 10 monclient(hunting): authenticate will time out at 1455.572258s 2024-09-16T03:50:08.625 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.602+0000 7fd65a0e3640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd6400039c0 0x7fd654130b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:50:08.625 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.602+0000 7fd65a8e4640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd65411c910 0x7fd6541d5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:50:08.625 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.602+0000 7fd65a0e3640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd6400039c0 0x7fd654130b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:44306/0 (socket says 172.21.15.42:44306) 2024-09-16T03:50:08.625 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.602+0000 7fd65a0e3640 1 -- 172.21.15.42:0/3856017930 learned_addr learned my addr 172.21.15.42:0/3856017930 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:50:08.625 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.602+0000 7fd65a8e4640 10 monclient(hunting): get_auth_request con 0x7fd65411c910 auth_method 0 2024-09-16T03:50:08.625 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.602+0000 7fd65a8e4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:50:08.625 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.602+0000 7fd65a8e4640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:50:08.625 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.602+0000 7fd65a8e4640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:50:08.625 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.602+0000 7fd65a0e3640 10 monclient(hunting): get_auth_request con 0x7fd6400039c0 auth_method 0 2024-09-16T03:50:08.625 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.602+0000 7fd65a0e3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:50:08.625 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.602+0000 7fd65a0e3640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:50:08.625 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.602+0000 7fd65a0e3640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:50:08.625 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.602+0000 7fd65a8e4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:50:08.625 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.602+0000 7fd65a8e4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:50:08.626 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.602+0000 7fd65a8e4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:50:08.626 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.602+0000 7fd65a0e3640 1 -- 172.21.15.42:0/3856017930 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd6400039c0 msgr2=0x7fd654130b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-16T03:50:08.626 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.602+0000 7fd65a0e3640 1 -- 172.21.15.42:0/3856017930 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd6400039c0 msgr2=0x7fd654130b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T03:50:08.626 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.602+0000 7fd65a0e3640 1 --2- 172.21.15.42:0/3856017930 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd6400039c0 0x7fd654130b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:50:08.626 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.602+0000 7fd6598e2640 1 --2- 172.21.15.42:0/3856017930 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd65411bf70 0x7fd65411c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:50:08.626 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.602+0000 7fd65a0e3640 1 --2- 172.21.15.42:0/3856017930 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd6400039c0 0x7fd654130b00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:50:08.626 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.602+0000 7fd6598e2640 10 monclient(hunting): get_auth_request con 0x7fd65411bf70 auth_method 0 2024-09-16T03:50:08.626 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.602+0000 7fd6598e2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:50:08.626 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.602+0000 7fd6598e2640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:50:08.626 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.602+0000 7fd6598e2640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:50:08.626 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.602+0000 7fd65a8e4640 10 monclient(hunting): handle_auth_done global_id 764775 payload 824 2024-09-16T03:50:08.626 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.602+0000 7fd65a8e4640 1 -- 172.21.15.42:0/3856017930 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd6400039c0 msgr2=0x7fd654130b00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:50:08.626 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.602+0000 7fd65a8e4640 1 --2- 172.21.15.42:0/3856017930 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd6400039c0 0x7fd654130b00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:50:08.627 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.602+0000 7fd65a8e4640 1 -- 172.21.15.42:0/3856017930 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd65411bf70 msgr2=0x7fd65411c3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:50:08.627 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.602+0000 7fd65a8e4640 1 --2- 172.21.15.42:0/3856017930 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd65411bf70 0x7fd65411c3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:50:08.627 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.602+0000 7fd65a8e4640 10 monclient: _finish_hunting 0 2024-09-16T03:50:08.627 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.602+0000 7fd65a8e4640 1 monclient: found mon.noname-a 2024-09-16T03:50:08.627 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.602+0000 7fd65a8e4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:50:08.627 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.602+0000 7fd65a8e4640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:50:08.627 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.602+0000 7fd65a8e4640 1 -- 172.21.15.42:0/3856017930 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd65411f210 con 0x7fd65411c910 2024-09-16T03:50:08.627 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.602+0000 7fd65a8e4640 1 --2- 172.21.15.42:0/3856017930 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd65411c910 0x7fd6541d5410 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fd64000bc50 tx=0x7fd64c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:50:08.627 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.602+0000 7fd64affd640 10 client.?.objecter ms_handle_connect 0x7fd65411c910 2024-09-16T03:50:08.627 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.602+0000 7fd64affd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:50:08.627 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.602+0000 7fd64affd640 1 -- 172.21.15.42:0/3856017930 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd64c00eb90 con 0x7fd65411c910 2024-09-16T03:50:08.627 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.602+0000 7fd64affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:50:08.627 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.602+0000 7fd64affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:50:08.628 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.602+0000 7fd64affd640 10 monclient: dump: 2024-09-16T03:50:08.628 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:50:08.628 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:50:08.628 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:08.628 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:08.628 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:50:08.628 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:50:08.628 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:50:08.628 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:50:08.628 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:50:08.628 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:50:08.628 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.602+0000 7fd64affd640 10 monclient: _finish_auth 0 2024-09-16T03:50:08.628 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.602+0000 7fd64affd640 10 monclient: _check_auth_tickets 2024-09-16T03:50:08.628 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.602+0000 7fd64affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:50:08.629 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.602+0000 7fd64affd640 1 -- 172.21.15.42:0/3856017930 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd64c00ed50 con 0x7fd65411c910 2024-09-16T03:50:08.629 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.602+0000 7fd65c347640 5 monclient: authenticate success, global_id 764775 2024-09-16T03:50:08.629 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.602+0000 7fd65c347640 10 monclient: _renew_subs 2024-09-16T03:50:08.629 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.602+0000 7fd65c347640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:50:08.629 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.602+0000 7fd65c347640 1 -- 172.21.15.42:0/3856017930 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd65411d800 con 0x7fd65411c910 2024-09-16T03:50:08.629 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.602+0000 7fd64affd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:50:08.629 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.602+0000 7fd64affd640 1 -- 172.21.15.42:0/3856017930 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd64c003be0 con 0x7fd65411c910 2024-09-16T03:50:08.629 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.602+0000 7fd64affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:50:08.629 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.602+0000 7fd64affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:50:08.629 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.602+0000 7fd64affd640 10 monclient: dump: 2024-09-16T03:50:08.629 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:50:08.629 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:50:08.629 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:08.629 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:08.630 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:50:08.630 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:50:08.630 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:50:08.630 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:50:08.630 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:50:08.630 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:50:08.630 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.602+0000 7fd6588e0640 4 set_mon_vals no callback set 2024-09-16T03:50:08.630 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.602+0000 7fd65c347640 10 client.764775.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:50:08.630 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.602+0000 7fd65c347640 10 monclient: _renew_subs 2024-09-16T03:50:08.630 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.602+0000 7fd65c347640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:50:08.630 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.602+0000 7fd65c347640 1 -- 172.21.15.42:0/3856017930 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd65411da80 con 0x7fd65411c910 2024-09-16T03:50:08.630 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.602+0000 7fd65c347640 1 librados: init done 2024-09-16T03:50:08.631 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.602+0000 7fd65c347640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:50:08.631 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.602+0000 7fd65c347640 1 --2- 172.21.15.42:0/3856017930 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd65406deb0 0x7fd65406e290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:50:08.631 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.602+0000 7fd65c347640 10 monclient(hunting): start opening mon connection 2024-09-16T03:50:08.631 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.602+0000 7fd65c347640 1 -- 172.21.15.42:0/3856017930 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fd65411e970 con 0x7fd65406deb0 2024-09-16T03:50:08.631 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.602+0000 7fd65a0e3640 1 --2- 172.21.15.42:0/3856017930 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd65406deb0 0x7fd65406e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:50:08.631 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.602+0000 7fd65a0e3640 10 monclient: get_auth_request con 0x7fd65406deb0 auth_method 0 2024-09-16T03:50:08.631 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.602+0000 7fd65a0e3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:50:08.631 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.602+0000 7fd65a0e3640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:50:08.631 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.602+0000 7fd65a0e3640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:50:08.631 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.602+0000 7fd64affd640 1 -- 172.21.15.42:0/3856017930 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fd64c01b030 con 0x7fd65411c910 2024-09-16T03:50:08.631 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.602+0000 7fd65a0e3640 1 -- 172.21.15.42:0/3856017930 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd65406deb0 msgr2=0x7fd65406e290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-16T03:50:08.631 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.602+0000 7fd65a0e3640 1 -- 172.21.15.42:0/3856017930 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd65406deb0 msgr2=0x7fd65406e290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T03:50:08.631 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.602+0000 7fd65a0e3640 1 --2- 172.21.15.42:0/3856017930 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd65406deb0 0x7fd65406e290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:50:08.631 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.602+0000 7fd65a0e3640 1 --2- 172.21.15.42:0/3856017930 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd65406deb0 0x7fd65406e290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:50:08.632 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.602+0000 7fd64affd640 1 --2- 172.21.15.42:0/3856017930 >> v2:172.21.15.154:6812/254845342 conn(0x7fd638037840 0x7fd638039d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:50:08.632 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.602+0000 7fd64affd640 1 -- 172.21.15.42:0/3856017930 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(276..276 src has 1..276) ==== 6586+0+0 (secure 0 0 0) 0x7fd64c0491a0 con 0x7fd65411c910 2024-09-16T03:50:08.632 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.602+0000 7fd64affd640 10 client.764775.objecter ms_dispatch 0x7fd654000ba0 osd_map(276..276 src has 1..276) 2024-09-16T03:50:08.632 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.602+0000 7fd64affd640 3 client.764775.objecter handle_osd_map got epochs [276,276] > 0 2024-09-16T03:50:08.632 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.602+0000 7fd64affd640 3 client.764775.objecter handle_osd_map decoding full epoch 276 2024-09-16T03:50:08.632 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.602+0000 7fd64affd640 20 client.764775.objecter dump_active .. 0 homeless 2024-09-16T03:50:08.632 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.602+0000 7fd6598e2640 1 --2- 172.21.15.42:0/3856017930 >> v2:172.21.15.154:6812/254845342 conn(0x7fd638037840 0x7fd638039d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:50:08.632 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.602+0000 7fd6598e2640 10 monclient: get_auth_request con 0x7fd638037840 auth_method 0 2024-09-16T03:50:08.632 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.602+0000 7fd64affd640 10 client.764775.objecter ms_handle_connect 0x7fd638037840 2024-09-16T03:50:08.632 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.602+0000 7fd6598e2640 1 --2- 172.21.15.42:0/3856017930 >> v2:172.21.15.154:6812/254845342 conn(0x7fd638037840 0x7fd638039d00 secure :-1 s=READY pgs=3534 cs=0 l=1 rev1=1 crypto rx=0x7fd65411da10 tx=0x7fd644008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:50:08.808 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.806+0000 7fd65a0e3640 1 --2- 172.21.15.42:0/3856017930 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd65406deb0 0x7fd65406e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:50:08.808 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.806+0000 7fd65a0e3640 10 monclient: get_auth_request con 0x7fd65406deb0 auth_method 0 2024-09-16T03:50:08.808 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.806+0000 7fd65a0e3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:50:08.808 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.806+0000 7fd65a0e3640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:50:08.808 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.806+0000 7fd65a0e3640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:50:08.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.806+0000 7fd65a0e3640 1 -- 172.21.15.42:0/3856017930 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd65406deb0 msgr2=0x7fd65406e290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-16T03:50:08.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.806+0000 7fd65a0e3640 1 -- 172.21.15.42:0/3856017930 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd65406deb0 msgr2=0x7fd65406e290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T03:50:08.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.806+0000 7fd65a0e3640 1 --2- 172.21.15.42:0/3856017930 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd65406deb0 0x7fd65406e290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:50:08.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:08.806+0000 7fd65a0e3640 1 --2- 172.21.15.42:0/3856017930 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd65406deb0 0x7fd65406e290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T03:50:09.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:09.210+0000 7fd65a0e3640 1 --2- 172.21.15.42:0/3856017930 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd65406deb0 0x7fd65406e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:50:09.214 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:09.210+0000 7fd65a0e3640 10 monclient: get_auth_request con 0x7fd65406deb0 auth_method 0 2024-09-16T03:50:09.214 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:09.210+0000 7fd65a0e3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:50:09.214 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:09.210+0000 7fd65a0e3640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:50:09.214 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:09.210+0000 7fd65a0e3640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:50:09.214 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:09.210+0000 7fd65a0e3640 1 -- 172.21.15.42:0/3856017930 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd65406deb0 msgr2=0x7fd65406e290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-16T03:50:09.214 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:09.210+0000 7fd65a0e3640 1 -- 172.21.15.42:0/3856017930 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd65406deb0 msgr2=0x7fd65406e290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T03:50:09.214 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:09.210+0000 7fd65a0e3640 1 --2- 172.21.15.42:0/3856017930 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd65406deb0 0x7fd65406e290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:50:09.214 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:09.210+0000 7fd65a0e3640 1 --2- 172.21.15.42:0/3856017930 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd65406deb0 0x7fd65406e290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T03:50:10.018 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:10.014+0000 7fd65a0e3640 1 --2- 172.21.15.42:0/3856017930 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd65406deb0 0x7fd65406e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:50:10.018 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:10.014+0000 7fd65a0e3640 10 monclient: get_auth_request con 0x7fd65406deb0 auth_method 0 2024-09-16T03:50:10.018 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:10.014+0000 7fd65a0e3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:50:10.018 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:10.014+0000 7fd65a0e3640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:50:10.018 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:10.014+0000 7fd65a0e3640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:50:10.018 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:10.014+0000 7fd65a0e3640 1 -- 172.21.15.42:0/3856017930 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd65406deb0 msgr2=0x7fd65406e290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-16T03:50:10.018 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:10.014+0000 7fd65a0e3640 1 -- 172.21.15.42:0/3856017930 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd65406deb0 msgr2=0x7fd65406e290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T03:50:10.018 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:10.014+0000 7fd65a0e3640 1 --2- 172.21.15.42:0/3856017930 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd65406deb0 0x7fd65406e290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:50:10.018 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:10.014+0000 7fd65a0e3640 1 --2- 172.21.15.42:0/3856017930 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd65406deb0 0x7fd65406e290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T03:50:11.605 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.602+0000 7fd649ffb640 10 monclient: tick 2024-09-16T03:50:11.605 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.602+0000 7fd649ffb640 10 monclient: _check_auth_tickets 2024-09-16T03:50:11.605 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.602+0000 7fd649ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:50:11.605 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.602+0000 7fd649ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:50:11.621 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.618+0000 7fd65a0e3640 1 --2- 172.21.15.42:0/3856017930 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd65406deb0 0x7fd65406e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:50:11.622 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.618+0000 7fd65a0e3640 10 monclient: get_auth_request con 0x7fd65406deb0 auth_method 0 2024-09-16T03:50:11.622 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.618+0000 7fd65a0e3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:50:11.622 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.618+0000 7fd65a0e3640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:50:11.622 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.618+0000 7fd65a0e3640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:50:11.622 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.618+0000 7fd65a0e3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:50:11.622 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.618+0000 7fd65a0e3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:50:11.622 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.618+0000 7fd65a0e3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:50:11.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.618+0000 7fd65a0e3640 10 monclient(hunting): handle_auth_done global_id 804101 payload 824 2024-09-16T03:50:11.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.618+0000 7fd65a0e3640 1 -- 172.21.15.42:0/3856017930 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd65409cdd0 con 0x7fd65406deb0 2024-09-16T03:50:11.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.618+0000 7fd65a0e3640 1 --2- 172.21.15.42:0/3856017930 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd65406deb0 0x7fd65406e290 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fd650001980 tx=0x7fd640005840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:50:11.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.618+0000 7fd64affd640 10 client.764775.objecter ms_handle_connect 0x7fd65406deb0 2024-09-16T03:50:11.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.618+0000 7fd64affd640 10 client.764775.objecter resend_mon_ops 2024-09-16T03:50:11.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.618+0000 7fd64affd640 1 -- 172.21.15.42:0/3856017930 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6500a5b70 con 0x7fd65406deb0 2024-09-16T03:50:11.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.618+0000 7fd64affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:50:11.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.618+0000 7fd64affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:50:11.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.618+0000 7fd64affd640 10 monclient: dump: 2024-09-16T03:50:11.623 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:50:11.623 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:50:11.624 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:11.624 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:11.624 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:50:11.624 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:50:11.624 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:50:11.624 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:50:11.624 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:50:11.624 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:50:11.625 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.618+0000 7fd64affd640 1 -- 172.21.15.42:0/3856017930 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fd6500a9070 con 0x7fd65406deb0 2024-09-16T03:50:11.625 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.618+0000 7fd64affd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:50:11.625 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.618+0000 7fd64affd640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:50:11.625 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.618+0000 7fd64affd640 1 -- 172.21.15.42:0/3856017930 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd65406deb0 msgr2=0x7fd65406e290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:50:11.625 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.618+0000 7fd64affd640 1 --2- 172.21.15.42:0/3856017930 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd65406deb0 0x7fd65406e290 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fd650001980 tx=0x7fd640005840 comp rx=0 tx=0).stop 2024-09-16T03:50:11.625 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.618+0000 7fd64affd640 1 -- 172.21.15.42:0/3856017930 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd65406deb0 msgr2=0x7fd65406e290 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:50:11.625 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.618+0000 7fd64affd640 1 --2- 172.21.15.42:0/3856017930 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd65406deb0 0x7fd65406e290 secure :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fd650001980 tx=0x7fd640005840 comp rx=0 tx=0).stop 2024-09-16T03:50:11.632 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.626+0000 7fd65c347640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:50:11.632 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.626+0000 7fd65c347640 1 --2- 172.21.15.42:0/3856017930 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd65406e7d0 0x7fd6541d9970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:50:11.632 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.626+0000 7fd65c347640 10 monclient(hunting): start opening mon connection 2024-09-16T03:50:11.632 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.626+0000 7fd65c347640 1 -- 172.21.15.42:0/3856017930 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fd654073a30 con 0x7fd65406e7d0 2024-09-16T03:50:11.633 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.626+0000 7fd65a0e3640 1 --2- 172.21.15.42:0/3856017930 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd65406e7d0 0x7fd6541d9970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:50:11.633 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.626+0000 7fd65a0e3640 10 monclient: get_auth_request con 0x7fd65406e7d0 auth_method 0 2024-09-16T03:50:11.633 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.626+0000 7fd65a0e3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:50:11.633 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.626+0000 7fd65a0e3640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:50:11.633 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.626+0000 7fd65a0e3640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:50:11.633 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.626+0000 7fd65a0e3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:50:11.633 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.626+0000 7fd65a0e3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:50:11.633 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.626+0000 7fd65a0e3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:50:11.633 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.630+0000 7fd65a0e3640 10 monclient(hunting): handle_auth_done global_id 804104 payload 824 2024-09-16T03:50:11.633 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.630+0000 7fd65a0e3640 1 -- 172.21.15.42:0/3856017930 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fd6540a1710 con 0x7fd65406e7d0 2024-09-16T03:50:11.634 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.630+0000 7fd64affd640 10 client.764775.objecter ms_handle_connect 0x7fd65406e7d0 2024-09-16T03:50:11.634 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.630+0000 7fd64affd640 10 client.764775.objecter resend_mon_ops 2024-09-16T03:50:11.634 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.630+0000 7fd65a0e3640 1 --2- 172.21.15.42:0/3856017930 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd65406e7d0 0x7fd6541d9970 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fd65001cc90 tx=0x7fd6500a7070 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:50:11.634 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.630+0000 7fd64affd640 1 -- 172.21.15.42:0/3856017930 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6500047f0 con 0x7fd65406e7d0 2024-09-16T03:50:11.635 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.630+0000 7fd64affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:50:11.635 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.630+0000 7fd64affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:50:11.635 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.630+0000 7fd64affd640 10 monclient: dump: 2024-09-16T03:50:11.635 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:50:11.635 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:50:11.635 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:11.635 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:11.635 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:50:11.635 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:50:11.635 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:50:11.635 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:50:11.635 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:50:11.636 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:50:11.636 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.630+0000 7fd64affd640 1 -- 172.21.15.42:0/3856017930 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4544 (secure 0 0 0) 0x7fd6500070d0 con 0x7fd65406e7d0 2024-09-16T03:50:11.636 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.630+0000 7fd64affd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:50:11.636 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.630+0000 7fd64affd640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:50:11.636 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.630+0000 7fd64affd640 1 -- 172.21.15.42:0/3856017930 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd65406e7d0 msgr2=0x7fd6541d9970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:50:11.636 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.630+0000 7fd64affd640 1 --2- 172.21.15.42:0/3856017930 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd65406e7d0 0x7fd6541d9970 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fd65001cc90 tx=0x7fd6500a7070 comp rx=0 tx=0).stop 2024-09-16T03:50:11.636 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.630+0000 7fd64affd640 1 -- 172.21.15.42:0/3856017930 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd65406e7d0 msgr2=0x7fd6541d9970 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:50:11.636 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.630+0000 7fd64affd640 1 --2- 172.21.15.42:0/3856017930 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd65406e7d0 0x7fd6541d9970 secure :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fd65001cc90 tx=0x7fd6500a7070 comp rx=0 tx=0).stop 2024-09-16T03:50:11.636 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:50:11.636 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:50:11.637 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:50:11.637 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T03:50:11.637 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 72, 2024-09-16T03:50:11.637 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 10455, 2024-09-16T03:50:11.637 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:50:11.637 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:50:11.637 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T03:50:11.637 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T03:50:11.637 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:50:11.637 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 4, 2024-09-16T03:50:11.637 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:50:11.637 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:50:11.637 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:50:11.638 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:50:11.638 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:50:11.638 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:50:11.638 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:50:11.638 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:50:11.638 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:50:11.638 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:50:11.638 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:50:11.638 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:50:11.638 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:50:11.638 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:50:11.638 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:50:11.638 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:50:11.639 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:50:11.639 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:50:11.639 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:50:11.639 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:50:11.639 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:50:11.639 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:50:11.639 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:50:11.639 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:50:11.639 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:50:11.639 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:50:11.639 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:50:11.639 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:50:11.639 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:50:11.639 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:50:11.640 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:50:11.640 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:50:11.640 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:50:11.640 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:50:11.640 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:50:11.640 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:50:11.640 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:50:11.640 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:50:11.640 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:50:11.640 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:50:11.640 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:50:11.640 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:50:11.641 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:50:11.641 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:50:11.641 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:50:11.641 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:50:11.641 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:50:11.641 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:50:11.641 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:50:11.641 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:50:11.641 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:50:11.641 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:50:11.641 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:50:11.641 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:50:11.641 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:50:11.641 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:50:11.642 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:50:11.642 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:50:11.642 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:50:11.642 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:50:11.642 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:50:11.642 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:50:11.642 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:50:11.642 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:50:11.642 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:50:11.642 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:50:11.642 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:50:11.642 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:50:11.642 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:50:11.642 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:50:11.643 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:50:11.643 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:50:11.643 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:50:11.643 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:50:11.643 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:50:11.643 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:50:11.643 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:50:11.643 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:50:11.643 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:50:11.643 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:50:11.643 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:50:11.643 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:50:11.643 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:50:11.643 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:50:11.643 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:50:11.644 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:50:11.644 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:50:11.644 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:50:11.644 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:50:11.644 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:50:11.644 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:50:11.644 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:50:11.644 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:50:11.644 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:50:11.644 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:50:11.644 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:50:11.644 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:50:11.644 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:50:11.645 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:50:11.645 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:50:11.645 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:50:11.645 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:50:11.645 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:50:11.645 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:50:11.645 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:50:11.645 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:50:11.645 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:50:11.645 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:50:11.645 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:50:11.645 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:50:11.645 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:50:11.646 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:50:11.646 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:50:11.646 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:50:11.646 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:50:11.646 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:50:11.646 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:50:11.646 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:50:11.646 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:50:11.646 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:50:11.646 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:50:11.646 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:50:11.646 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:50:11.646 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:50:11.647 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:50:11.647 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:50:11.647 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:50:11.647 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:50:11.647 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:50:11.647 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:50:11.647 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:50:11.647 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:50:11.647 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:50:11.647 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:50:11.647 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:50:11.647 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:50:11.647 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:50:11.648 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:50:11.648 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:50:11.648 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:50:11.648 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:50:11.648 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:50:11.648 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:50:11.648 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:50:11.648 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:50:11.648 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:50:11.648 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:50:11.648 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:50:11.648 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:50:11.648 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:50:11.649 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:50:11.650 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.630+0000 7fd65c347640 10 librados: watch_flush enter 2024-09-16T03:50:11.651 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.630+0000 7fd65c347640 10 librados: watch_flush exit 2024-09-16T03:50:11.651 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.630+0000 7fd65c347640 20 client.764775.objecter shutdown clearing up homeless session... 2024-09-16T03:50:11.651 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.630+0000 7fd65c347640 10 client.764775.objecter successfully canceled tick 2024-09-16T03:50:11.651 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.630+0000 7fd65c347640 1 -- 172.21.15.42:0/3856017930 >> v2:172.21.15.154:6812/254845342 conn(0x7fd638037840 msgr2=0x7fd638039d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:50:11.651 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.630+0000 7fd65c347640 1 --2- 172.21.15.42:0/3856017930 >> v2:172.21.15.154:6812/254845342 conn(0x7fd638037840 0x7fd638039d00 secure :-1 s=READY pgs=3534 cs=0 l=1 rev1=1 crypto rx=0x7fd65411da10 tx=0x7fd644008040 comp rx=0 tx=0).stop 2024-09-16T03:50:11.651 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.630+0000 7fd65c347640 10 monclient: shutdown 2024-09-16T03:50:11.651 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.630+0000 7fd65c347640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:50:11.651 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.630+0000 7fd65c347640 1 -- 172.21.15.42:0/3856017930 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd65411c910 msgr2=0x7fd6541d5410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:50:11.651 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.630+0000 7fd65c347640 1 --2- 172.21.15.42:0/3856017930 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd65411c910 0x7fd6541d5410 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fd64000bc50 tx=0x7fd64c00e4b0 comp rx=0 tx=0).stop 2024-09-16T03:50:11.651 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.630+0000 7fd65a8e4640 1 -- 172.21.15.42:0/3856017930 reap_dead start 2024-09-16T03:50:11.651 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.630+0000 7fd65a8e4640 1 -- 172.21.15.42:0/3856017930 reap_dead start 2024-09-16T03:50:11.651 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.630+0000 7fd65c347640 1 -- 172.21.15.42:0/3856017930 shutdown_connections 2024-09-16T03:50:11.651 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.630+0000 7fd65c347640 1 -- 172.21.15.42:0/3856017930 >> 172.21.15.42:0/3856017930 conn(0x7fd6540a2420 msgr2=0x7fd65409e230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:50:11.652 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.630+0000 7fd65c347640 1 -- 172.21.15.42:0/3856017930 shutdown_connections 2024-09-16T03:50:11.652 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.630+0000 7fd65c347640 1 -- 172.21.15.42:0/3856017930 wait complete. 2024-09-16T03:50:11.652 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.630+0000 7fd65c347640 1 librados: shutdown 2024-09-16T03:50:11.658 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:50:11.658 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T03:50:11.658 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T03:50:11.789 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.786+0000 7fb4d54d1640 10 monclient: get_monmap_and_config 2024-09-16T03:50:11.789 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.786+0000 7fb4d54d1640 10 monclient: build_initial_monmap 2024-09-16T03:50:11.790 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.786+0000 7fb4d54d1640 10 monclient: monmap: 2024-09-16T03:50:11.790 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:50:11.790 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:50:11.790 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:50:11.788524+0000 2024-09-16T03:50:11.790 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:50:11.788524+0000 2024-09-16T03:50:11.790 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:50:11.790 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:50:11.790 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:50:11.790 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:50:11.790 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:50:11.790 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:50:11.790 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:50:11.790 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:50:11.792 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.786+0000 7fb4d54d1640 1 Processor -- start 2024-09-16T03:50:11.793 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.786+0000 7fb4d54d1640 1 -- start start 2024-09-16T03:50:11.793 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.786+0000 7fb4d54d1640 10 monclient: init 2024-09-16T03:50:11.793 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.786+0000 7fb4d54d1640 10 monclient: _reopen_session rank -1 2024-09-16T03:50:11.793 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.786+0000 7fb4d54d1640 10 monclient: _add_conns ranks=[3,2,0,4,1] 2024-09-16T03:50:11.793 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.786+0000 7fb4d54d1640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb4d00a1330 0x7fb4d00a1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:50:11.793 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.786+0000 7fb4d54d1640 10 monclient(hunting): picked mon.noname-a con 0x7fb4d00a1330 addr v2:172.21.15.42:3300/0 2024-09-16T03:50:11.793 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.786+0000 7fb4d54d1640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb4d006dac0 0x7fb4d00a1c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:50:11.793 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.786+0000 7fb4d54d1640 10 monclient(hunting): picked mon.noname-b con 0x7fb4d006dac0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:50:11.793 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.786+0000 7fb4d54d1640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb4d0135100 addr v1:172.21.15.42:6789/0 2024-09-16T03:50:11.793 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.786+0000 7fb4d54d1640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb4d009c7d0 con 0x7fb4d0135100 2024-09-16T03:50:11.793 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.786+0000 7fb4d54d1640 10 monclient(hunting): start opening mon connection 2024-09-16T03:50:11.794 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.786+0000 7fb4d54d1640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7fb4d009c090 con 0x7fb4d00a1330 2024-09-16T03:50:11.794 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.786+0000 7fb4d54d1640 10 monclient(hunting): start opening mon connection 2024-09-16T03:50:11.794 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.786+0000 7fb4d54d1640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fb4d009f610 con 0x7fb4d006dac0 2024-09-16T03:50:11.794 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.786+0000 7fb4d54d1640 10 monclient(hunting): _renew_subs 2024-09-16T03:50:11.794 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.786+0000 7fb4d54d1640 10 monclient(hunting): authenticate will time out at 1458.757921s 2024-09-16T03:50:11.794 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.786+0000 7fb4ceffd640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb4d00a1330 0x7fb4d00a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:50:11.794 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.786+0000 7fb4ce7fc640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb4d006dac0 0x7fb4d00a1c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:50:11.794 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.786+0000 7fb4cf7fe640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fb4d0135100 0x7fb4d0135550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:56404/0 (socket says 172.21.15.42:56404) 2024-09-16T03:50:11.794 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.786+0000 7fb4cf7fe640 1 -- 172.21.15.42:0/4024647749 learned_addr learned my addr 172.21.15.42:0/4024647749 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:50:11.794 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.786+0000 7fb4ceffd640 10 monclient(hunting): get_auth_request con 0x7fb4d00a1330 auth_method 0 2024-09-16T03:50:11.794 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.786+0000 7fb4ceffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:50:11.795 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.786+0000 7fb4ceffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:50:11.795 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.786+0000 7fb4ceffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:50:11.795 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.786+0000 7fb4ce7fc640 10 monclient(hunting): get_auth_request con 0x7fb4d006dac0 auth_method 0 2024-09-16T03:50:11.795 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.786+0000 7fb4ce7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:50:11.795 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.786+0000 7fb4ce7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:50:11.795 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.786+0000 7fb4ce7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:50:11.795 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.786+0000 7fb4ceffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:50:11.795 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.786+0000 7fb4ceffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:50:11.795 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.786+0000 7fb4ceffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:50:11.795 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.786+0000 7fb4ce7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:50:11.795 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.786+0000 7fb4ce7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:50:11.795 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.786+0000 7fb4ce7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:50:11.795 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.786+0000 7fb4ce7fc640 10 monclient(hunting): handle_auth_done global_id 804107 payload 274 2024-09-16T03:50:11.795 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.786+0000 7fb4ce7fc640 1 -- 172.21.15.42:0/4024647749 >> v2:172.21.15.42:3300/0 conn(0x7fb4d00a1330 msgr2=0x7fb4d00a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:50:11.796 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.786+0000 7fb4ce7fc640 1 --2- 172.21.15.42:0/4024647749 >> v2:172.21.15.42:3300/0 conn(0x7fb4d00a1330 0x7fb4d00a1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:50:11.796 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.786+0000 7fb4ce7fc640 1 -- 172.21.15.42:0/4024647749 >> v1:172.21.15.42:6789/0 conn(0x7fb4d0135100 legacy=0x7fb4d0135550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:50:11.796 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.786+0000 7fb4ce7fc640 10 monclient: _finish_hunting 0 2024-09-16T03:50:11.796 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.786+0000 7fb4ce7fc640 1 monclient: found mon.noname-b 2024-09-16T03:50:11.796 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.786+0000 7fb4ce7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:50:11.796 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.786+0000 7fb4ce7fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T03:50:11.796 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.786+0000 7fb4cdffb640 1 -- 172.21.15.42:0/4024647749 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3952658680 0 0) 0x7fb4d009c7d0 con 0x7fb4d0135100 2024-09-16T03:50:11.796 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.786+0000 7fb4ce7fc640 1 -- 172.21.15.42:0/4024647749 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4d009f8f0 con 0x7fb4d006dac0 2024-09-16T03:50:11.796 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.786+0000 7fb4ceffd640 1 --2- 172.21.15.42:0/4024647749 >> v2:172.21.15.42:3300/0 conn(0x7fb4d00a1330 0x7fb4d00a1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:50:11.796 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.786+0000 7fb4cdffb640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T03:50:11.803 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.790+0000 7fb4ce7fc640 1 --2- 172.21.15.42:0/4024647749 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb4d006dac0 0x7fb4d00a1c50 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fb4c40097c0 tx=0x7fb4c401c480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7bcc480a51ebf9c4 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:50:11.803 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.790+0000 7fb4cdffb640 1 -- 172.21.15.42:0/4024647749 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4c40b2070 con 0x7fb4d006dac0 2024-09-16T03:50:11.803 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.790+0000 7fb4cdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:50:11.803 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.790+0000 7fb4cdffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:50:11.803 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.790+0000 7fb4cdffb640 10 monclient: dump: 2024-09-16T03:50:11.803 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:50:11.803 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:50:11.803 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:11.803 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:11.803 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:50:11.803 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:50:11.804 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:50:11.804 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:50:11.804 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:50:11.804 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:50:11.804 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.790+0000 7fb4cdffb640 10 monclient: _finish_auth 0 2024-09-16T03:50:11.804 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.790+0000 7fb4cdffb640 10 monclient: _check_auth_tickets 2024-09-16T03:50:11.804 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.790+0000 7fb4cdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:50:11.804 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.790+0000 7fb4cdffb640 1 -- 172.21.15.42:0/4024647749 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb4c40ad640 con 0x7fb4d006dac0 2024-09-16T03:50:11.804 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.790+0000 7fb4cdffb640 10 monclient: handle_config config(0 keys) 2024-09-16T03:50:11.804 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.790+0000 7fb4cdffb640 1 -- 172.21.15.42:0/4024647749 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4c40ada10 con 0x7fb4d006dac0 2024-09-16T03:50:11.804 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.790+0000 7fb4d54d1640 5 monclient: authenticate success, global_id 804107 2024-09-16T03:50:11.804 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.790+0000 7fb4d54d1640 10 monclient: get_monmap_and_config success 2024-09-16T03:50:11.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.790+0000 7fb4d54d1640 4 set_mon_vals no callback set 2024-09-16T03:50:11.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.790+0000 7fb4cdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:50:11.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.790+0000 7fb4cdffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:50:11.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.790+0000 7fb4cdffb640 10 monclient: dump: 2024-09-16T03:50:11.805 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:50:11.805 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:50:11.805 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:11.805 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:11.805 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:50:11.805 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:50:11.805 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:50:11.805 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:50:11.805 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:50:11.805 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:50:11.806 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.790+0000 7fb4d54d1640 10 monclient: shutdown 2024-09-16T03:50:11.806 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.790+0000 7fb4d54d1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:50:11.806 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.790+0000 7fb4d54d1640 1 -- 172.21.15.42:0/4024647749 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb4d006dac0 msgr2=0x7fb4d00a1c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:50:11.806 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.790+0000 7fb4d54d1640 1 --2- 172.21.15.42:0/4024647749 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb4d006dac0 0x7fb4d00a1c50 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fb4c40097c0 tx=0x7fb4c401c480 comp rx=0 tx=0).stop 2024-09-16T03:50:11.806 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.790+0000 7fb4d54d1640 1 -- 172.21.15.42:0/4024647749 shutdown_connections 2024-09-16T03:50:11.806 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.790+0000 7fb4d54d1640 1 --2- 172.21.15.42:0/4024647749 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb4d006dac0 0x7fb4d00a1c50 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:50:11.806 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.790+0000 7fb4d54d1640 1 --2- 172.21.15.42:0/4024647749 >> v2:172.21.15.42:3300/0 conn(0x7fb4d00a1330 0x7fb4d00a1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:50:11.806 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.790+0000 7fb4d54d1640 1 -- 172.21.15.42:0/4024647749 >> 172.21.15.42:0/4024647749 conn(0x7fb4d00a2420 msgr2=0x7fb4d00a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:50:11.806 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.790+0000 7fb4d54d1640 1 -- 172.21.15.42:0/4024647749 shutdown_connections 2024-09-16T03:50:11.806 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.790+0000 7fb4d54d1640 1 -- 172.21.15.42:0/4024647749 wait complete. 2024-09-16T03:50:11.806 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.790+0000 7fb4d54d1640 10 monclient: build_initial_monmap 2024-09-16T03:50:11.806 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.790+0000 7fb4d54d1640 10 monclient: monmap: 2024-09-16T03:50:11.806 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:50:11.806 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:50:11.807 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:50:11.807 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:50:11.807 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:50:11.807 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:50:11.807 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:50:11.807 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:50:11.807 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:50:11.807 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:50:11.807 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.790+0000 7fb4d54d1640 1 librados: starting msgr at 2024-09-16T03:50:11.807 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.790+0000 7fb4d54d1640 1 librados: starting objecter 2024-09-16T03:50:11.807 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.790+0000 7fb4d54d1640 1 Processor -- start 2024-09-16T03:50:11.807 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.790+0000 7fb4d54d1640 1 -- start start 2024-09-16T03:50:11.807 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.790+0000 7fb4d54d1640 1 librados: setting wanted keys 2024-09-16T03:50:11.808 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.790+0000 7fb4d54d1640 1 librados: calling monclient init 2024-09-16T03:50:11.808 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.790+0000 7fb4d54d1640 10 monclient: init 2024-09-16T03:50:11.808 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.790+0000 7fb4d54d1640 10 monclient: _reopen_session rank -1 2024-09-16T03:50:11.808 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.790+0000 7fb4d54d1640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:50:11.808 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.790+0000 7fb4d54d1640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb4d00a1330 0x7fb4d013c050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:50:11.808 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.790+0000 7fb4d54d1640 10 monclient(hunting): picked mon.noname-c con 0x7fb4d00a1330 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:50:11.808 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.790+0000 7fb4d54d1640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb4d0135100 0x7fb4d01343a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:50:11.808 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.790+0000 7fb4d54d1640 10 monclient(hunting): picked mon.noname-a con 0x7fb4d0135100 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:50:11.808 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.790+0000 7fb4d54d1640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb4d01348e0 0x7fb4d01cf150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:50:11.808 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.790+0000 7fb4d54d1640 10 monclient(hunting): picked mon.noname-b con 0x7fb4d01348e0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:50:11.808 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.790+0000 7fb4d54d1640 10 monclient(hunting): start opening mon connection 2024-09-16T03:50:11.808 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.790+0000 7fb4d54d1640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fb4d013fd00 con 0x7fb4d0135100 2024-09-16T03:50:11.808 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.790+0000 7fb4d54d1640 10 monclient(hunting): start opening mon connection 2024-09-16T03:50:11.808 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.790+0000 7fb4d54d1640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fb4d013fb80 con 0x7fb4d01348e0 2024-09-16T03:50:11.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.790+0000 7fb4d54d1640 10 monclient(hunting): start opening mon connection 2024-09-16T03:50:11.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.790+0000 7fb4d54d1640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fb4d013fe80 con 0x7fb4d00a1330 2024-09-16T03:50:11.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.790+0000 7fb4d54d1640 10 monclient(hunting): _renew_subs 2024-09-16T03:50:11.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.790+0000 7fb4d54d1640 10 monclient(hunting): authenticate will time out at 1458.762951s 2024-09-16T03:50:11.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.790+0000 7fb4ceffd640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb4d00a1330 0x7fb4d013c050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:50:11.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.790+0000 7fb4ce7fc640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb4d0135100 0x7fb4d01343a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:50:11.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.790+0000 7fb4ceffd640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb4d00a1330 0x7fb4d013c050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:44396/0 (socket says 172.21.15.42:44396) 2024-09-16T03:50:11.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.790+0000 7fb4ceffd640 1 -- 172.21.15.42:0/3898591137 learned_addr learned my addr 172.21.15.42:0/3898591137 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:50:11.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.790+0000 7fb4ceffd640 10 monclient(hunting): get_auth_request con 0x7fb4d00a1330 auth_method 0 2024-09-16T03:50:11.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.790+0000 7fb4ceffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:50:11.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.790+0000 7fb4ceffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:50:11.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.790+0000 7fb4ceffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:50:11.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.790+0000 7fb4ce7fc640 10 monclient(hunting): get_auth_request con 0x7fb4d0135100 auth_method 0 2024-09-16T03:50:11.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.790+0000 7fb4ce7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:50:11.810 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.790+0000 7fb4ce7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:50:11.810 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.790+0000 7fb4ce7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:50:11.810 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.790+0000 7fb4ceffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:50:11.810 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.790+0000 7fb4ceffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:50:11.810 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.790+0000 7fb4ceffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:50:11.810 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.790+0000 7fb4ce7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:50:11.810 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.790+0000 7fb4ce7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:50:11.810 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.790+0000 7fb4ce7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:50:11.810 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.790+0000 7fb4ceffd640 10 monclient(hunting): handle_auth_done global_id 804110 payload 824 2024-09-16T03:50:11.810 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.790+0000 7fb4cf7fe640 1 --2- 172.21.15.42:0/3898591137 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb4d01348e0 0x7fb4d01cf150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:50:11.810 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.790+0000 7fb4ceffd640 1 -- 172.21.15.42:0/3898591137 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb4d01348e0 msgr2=0x7fb4d01cf150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:50:11.810 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.790+0000 7fb4ceffd640 1 --2- 172.21.15.42:0/3898591137 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb4d01348e0 0x7fb4d01cf150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:50:11.810 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.790+0000 7fb4ceffd640 1 -- 172.21.15.42:0/3898591137 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb4d0135100 msgr2=0x7fb4d01343a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:50:11.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.790+0000 7fb4ceffd640 1 --2- 172.21.15.42:0/3898591137 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb4d0135100 0x7fb4d01343a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:50:11.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.790+0000 7fb4ceffd640 10 monclient: _finish_hunting 0 2024-09-16T03:50:11.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.790+0000 7fb4ceffd640 1 monclient: found mon.noname-c 2024-09-16T03:50:11.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.790+0000 7fb4ceffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:50:11.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.790+0000 7fb4ceffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:50:11.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.790+0000 7fb4ceffd640 1 -- 172.21.15.42:0/3898591137 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4d0120d50 con 0x7fb4d00a1330 2024-09-16T03:50:11.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.790+0000 7fb4cf7fe640 10 monclient: get_auth_request con 0x7fb4d01348e0 auth_method 0 2024-09-16T03:50:11.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.790+0000 7fb4cf7fe640 1 --2- 172.21.15.42:0/3898591137 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb4d01348e0 0x7fb4d01cf150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T03:50:11.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.790+0000 7fb4ceffd640 1 --2- 172.21.15.42:0/3898591137 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb4d00a1330 0x7fb4d013c050 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fb4c000ec10 tx=0x7fb4c000c520 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:50:11.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.790+0000 7fb4b77fe640 10 client.?.objecter ms_handle_connect 0x7fb4d00a1330 2024-09-16T03:50:11.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.790+0000 7fb4b77fe640 10 client.?.objecter resend_mon_ops 2024-09-16T03:50:11.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.790+0000 7fb4b77fe640 1 -- 172.21.15.42:0/3898591137 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4c0017070 con 0x7fb4d00a1330 2024-09-16T03:50:11.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.790+0000 7fb4b77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:50:11.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.790+0000 7fb4b77fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:50:11.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.790+0000 7fb4b77fe640 10 monclient: dump: 2024-09-16T03:50:11.812 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:50:11.812 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:50:11.812 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:11.812 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:11.812 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:50:11.812 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:50:11.812 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:50:11.812 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:50:11.812 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:50:11.812 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:50:11.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.790+0000 7fb4b77fe640 10 monclient: _finish_auth 0 2024-09-16T03:50:11.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.790+0000 7fb4b77fe640 10 monclient: _check_auth_tickets 2024-09-16T03:50:11.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.790+0000 7fb4b77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:50:11.813 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.790+0000 7fb4b77fe640 1 -- 172.21.15.42:0/3898591137 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb4c000cbd0 con 0x7fb4d00a1330 2024-09-16T03:50:11.813 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.790+0000 7fb4d54d1640 5 monclient: authenticate success, global_id 804110 2024-09-16T03:50:11.813 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.790+0000 7fb4d54d1640 10 monclient: _renew_subs 2024-09-16T03:50:11.813 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.790+0000 7fb4d54d1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:50:11.813 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.790+0000 7fb4d54d1640 1 -- 172.21.15.42:0/3898591137 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4d00719c0 con 0x7fb4d00a1330 2024-09-16T03:50:11.813 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.790+0000 7fb4b77fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:50:11.813 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.790+0000 7fb4b77fe640 1 -- 172.21.15.42:0/3898591137 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4c0013470 con 0x7fb4d00a1330 2024-09-16T03:50:11.813 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.790+0000 7fb4b77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:50:11.813 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.790+0000 7fb4b77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:50:11.813 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.790+0000 7fb4b77fe640 10 monclient: dump: 2024-09-16T03:50:11.813 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:50:11.813 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:50:11.813 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:11.814 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:11.814 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:50:11.814 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:50:11.814 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:50:11.814 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:50:11.814 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:50:11.814 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:50:11.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.790+0000 7fb4cd7fa640 4 set_mon_vals no callback set 2024-09-16T03:50:11.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.790+0000 7fb4d54d1640 10 client.804110.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:50:11.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.790+0000 7fb4d54d1640 10 monclient: _renew_subs 2024-09-16T03:50:11.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.790+0000 7fb4d54d1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:50:11.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.790+0000 7fb4d54d1640 1 -- 172.21.15.42:0/3898591137 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4d013cc10 con 0x7fb4d00a1330 2024-09-16T03:50:11.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.790+0000 7fb4d54d1640 1 librados: init done 2024-09-16T03:50:11.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.794+0000 7fb4d54d1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:50:11.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.794+0000 7fb4d54d1640 1 --2- 172.21.15.42:0/3898591137 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb4d006dac0 0x7fb4d002c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:50:11.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.794+0000 7fb4d54d1640 10 monclient(hunting): start opening mon connection 2024-09-16T03:50:11.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.794+0000 7fb4d54d1640 1 -- 172.21.15.42:0/3898591137 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fb4d0138a40 con 0x7fb4d006dac0 2024-09-16T03:50:11.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.794+0000 7fb4b77fe640 1 -- 172.21.15.42:0/3898591137 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fb4c001a070 con 0x7fb4d00a1330 2024-09-16T03:50:11.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.794+0000 7fb4b77fe640 1 --2- 172.21.15.42:0/3898591137 >> v2:172.21.15.154:6812/254845342 conn(0x7fb4bc037810 0x7fb4bc039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:50:11.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.794+0000 7fb4ce7fc640 1 --2- 172.21.15.42:0/3898591137 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb4d006dac0 0x7fb4d002c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:50:11.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.794+0000 7fb4b77fe640 1 -- 172.21.15.42:0/3898591137 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(279..279 src has 1..279) ==== 5421+0+0 (secure 0 0 0) 0x7fb4c0048db0 con 0x7fb4d00a1330 2024-09-16T03:50:11.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.794+0000 7fb4ce7fc640 10 monclient: get_auth_request con 0x7fb4d006dac0 auth_method 0 2024-09-16T03:50:11.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.794+0000 7fb4b77fe640 10 client.804110.objecter ms_dispatch 0x7fb4d0000ba0 osd_map(279..279 src has 1..279) 2024-09-16T03:50:11.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.794+0000 7fb4ce7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:50:11.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.794+0000 7fb4ce7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:50:11.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.794+0000 7fb4b77fe640 3 client.804110.objecter handle_osd_map got epochs [279,279] > 0 2024-09-16T03:50:11.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.794+0000 7fb4ce7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:50:11.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.794+0000 7fb4b77fe640 3 client.804110.objecter handle_osd_map decoding full epoch 279 2024-09-16T03:50:11.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.794+0000 7fb4b77fe640 20 client.804110.objecter dump_active .. 0 homeless 2024-09-16T03:50:11.816 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.794+0000 7fb4ce7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:50:11.816 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.794+0000 7fb4ce7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:50:11.816 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.794+0000 7fb4ce7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:50:11.816 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.794+0000 7fb4cf7fe640 1 --2- 172.21.15.42:0/3898591137 >> v2:172.21.15.154:6812/254845342 conn(0x7fb4bc037810 0x7fb4bc039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:50:11.816 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.794+0000 7fb4cf7fe640 10 monclient: get_auth_request con 0x7fb4bc037810 auth_method 0 2024-09-16T03:50:11.816 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.794+0000 7fb4ce7fc640 10 monclient(hunting): handle_auth_done global_id 794227 payload 824 2024-09-16T03:50:11.816 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.794+0000 7fb4ce7fc640 1 -- 172.21.15.42:0/3898591137 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb4d0135510 con 0x7fb4d006dac0 2024-09-16T03:50:11.816 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.794+0000 7fb4b77fe640 10 client.804110.objecter ms_handle_connect 0x7fb4d006dac0 2024-09-16T03:50:11.816 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.794+0000 7fb4b77fe640 10 client.804110.objecter resend_mon_ops 2024-09-16T03:50:11.816 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.794+0000 7fb4ce7fc640 1 --2- 172.21.15.42:0/3898591137 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb4d006dac0 0x7fb4d002c650 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fb4d009bb00 tx=0x7fb4c4019220 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:50:11.816 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.794+0000 7fb4cf7fe640 1 --2- 172.21.15.42:0/3898591137 >> v2:172.21.15.154:6812/254845342 conn(0x7fb4bc037810 0x7fb4bc039cd0 secure :-1 s=READY pgs=3535 cs=0 l=1 rev1=1 crypto rx=0x7fb4c80012f0 tx=0x7fb4c8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:50:11.816 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.794+0000 7fb4b77fe640 10 client.804110.objecter ms_handle_connect 0x7fb4bc037810 2024-09-16T03:50:11.816 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.794+0000 7fb4b77fe640 1 -- 172.21.15.42:0/3898591137 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4c40bd030 con 0x7fb4d006dac0 2024-09-16T03:50:11.817 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.794+0000 7fb4b77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:50:11.817 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.794+0000 7fb4b77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:50:11.817 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.794+0000 7fb4b77fe640 10 monclient: dump: 2024-09-16T03:50:11.817 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:50:11.817 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:50:11.817 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:11.817 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:11.817 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:50:11.817 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:50:11.817 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:50:11.817 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:50:11.817 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:50:11.817 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:50:11.817 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.794+0000 7fb4b77fe640 1 -- 172.21.15.42:0/3898591137 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fb4c40b8470 con 0x7fb4d006dac0 2024-09-16T03:50:11.818 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.794+0000 7fb4b77fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:50:11.818 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.794+0000 7fb4b77fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:50:11.818 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.794+0000 7fb4b77fe640 1 -- 172.21.15.42:0/3898591137 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb4d006dac0 msgr2=0x7fb4d002c650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:50:11.818 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.794+0000 7fb4b77fe640 1 --2- 172.21.15.42:0/3898591137 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb4d006dac0 0x7fb4d002c650 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fb4d009bb00 tx=0x7fb4c4019220 comp rx=0 tx=0).stop 2024-09-16T03:50:11.818 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.794+0000 7fb4b77fe640 1 -- 172.21.15.42:0/3898591137 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb4d006dac0 msgr2=0x7fb4d002c650 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:50:11.818 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.794+0000 7fb4b77fe640 1 --2- 172.21.15.42:0/3898591137 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb4d006dac0 0x7fb4d002c650 secure :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fb4d009bb00 tx=0x7fb4c4019220 comp rx=0 tx=0).stop 2024-09-16T03:50:11.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.802+0000 7fb4d54d1640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:50:11.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.802+0000 7fb4d54d1640 1 --2- 172.21.15.42:0/3898591137 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb4d002cb90 0x7fb4d002cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:50:11.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.802+0000 7fb4d54d1640 10 monclient(hunting): start opening mon connection 2024-09-16T03:50:11.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.802+0000 7fb4d54d1640 1 -- 172.21.15.42:0/3898591137 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fb4d009db10 con 0x7fb4d002cb90 2024-09-16T03:50:11.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.802+0000 7fb4ce7fc640 1 --2- 172.21.15.42:0/3898591137 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb4d002cb90 0x7fb4d002cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:50:11.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.802+0000 7fb4ce7fc640 10 monclient: get_auth_request con 0x7fb4d002cb90 auth_method 0 2024-09-16T03:50:11.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.802+0000 7fb4ce7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:50:11.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.802+0000 7fb4ce7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:50:11.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.802+0000 7fb4ce7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:50:11.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.802+0000 7fb4ce7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:50:11.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.802+0000 7fb4ce7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:50:11.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.802+0000 7fb4ce7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:50:11.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.802+0000 7fb4ce7fc640 10 monclient(hunting): handle_auth_done global_id 794230 payload 824 2024-09-16T03:50:11.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.802+0000 7fb4ce7fc640 1 -- 172.21.15.42:0/3898591137 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fb4d0069040 con 0x7fb4d002cb90 2024-09-16T03:50:11.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.802+0000 7fb4b77fe640 10 client.804110.objecter ms_handle_connect 0x7fb4d002cb90 2024-09-16T03:50:11.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.802+0000 7fb4b77fe640 10 client.804110.objecter resend_mon_ops 2024-09-16T03:50:11.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.802+0000 7fb4ce7fc640 1 --2- 172.21.15.42:0/3898591137 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb4d002cb90 0x7fb4d002cfa0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fb4c401ab30 tx=0x7fb4d00a22d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:50:11.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.802+0000 7fb4b77fe640 1 -- 172.21.15.42:0/3898591137 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4c40bd040 con 0x7fb4d002cb90 2024-09-16T03:50:11.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.802+0000 7fb4b77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:50:11.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.802+0000 7fb4b77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:50:11.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.802+0000 7fb4b77fe640 10 monclient: dump: 2024-09-16T03:50:11.821 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:50:11.821 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:50:11.821 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:11.822 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:11.822 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:50:11.822 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:50:11.822 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:50:11.822 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:50:11.822 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:50:11.822 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:50:11.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.802+0000 7fb4b77fe640 1 -- 172.21.15.42:0/3898591137 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4544 (secure 0 0 0) 0x7fb4c40c3a10 con 0x7fb4d002cb90 2024-09-16T03:50:11.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.802+0000 7fb4b77fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:50:11.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.802+0000 7fb4b77fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:50:11.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.802+0000 7fb4b77fe640 1 -- 172.21.15.42:0/3898591137 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb4d002cb90 msgr2=0x7fb4d002cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:50:11.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.802+0000 7fb4b77fe640 1 --2- 172.21.15.42:0/3898591137 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb4d002cb90 0x7fb4d002cfa0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fb4c401ab30 tx=0x7fb4d00a22d0 comp rx=0 tx=0).stop 2024-09-16T03:50:11.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.802+0000 7fb4b77fe640 1 -- 172.21.15.42:0/3898591137 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb4d002cb90 msgr2=0x7fb4d002cfa0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:50:11.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.802+0000 7fb4b77fe640 1 --2- 172.21.15.42:0/3898591137 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb4d002cb90 0x7fb4d002cfa0 secure :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fb4c401ab30 tx=0x7fb4d00a22d0 comp rx=0 tx=0).stop 2024-09-16T03:50:11.823 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:50:11.823 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:50:11.823 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:50:11.823 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T03:50:11.823 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 72, 2024-09-16T03:50:11.823 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 53643, 2024-09-16T03:50:11.823 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:50:11.823 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:50:11.823 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T03:50:11.823 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T03:50:11.824 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:50:11.824 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 4, 2024-09-16T03:50:11.824 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:50:11.824 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:50:11.824 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:50:11.824 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:50:11.824 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:50:11.824 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:50:11.824 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:50:11.824 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:50:11.824 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:50:11.824 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:50:11.824 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:50:11.824 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:50:11.825 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:50:11.825 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:50:11.825 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:50:11.825 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:50:11.825 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:50:11.825 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:50:11.825 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:50:11.825 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:50:11.825 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:50:11.825 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:50:11.825 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:50:11.825 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:50:11.825 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:50:11.826 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:50:11.826 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:50:11.826 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:50:11.826 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:50:11.826 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:50:11.826 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:50:11.826 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:50:11.826 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:50:11.826 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:50:11.826 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:50:11.826 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:50:11.826 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:50:11.826 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:50:11.827 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:50:11.827 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:50:11.827 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:50:11.827 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:50:11.827 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:50:11.827 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:50:11.827 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:50:11.827 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:50:11.827 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:50:11.827 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:50:11.827 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:50:11.827 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:50:11.827 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:50:11.827 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:50:11.828 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:50:11.828 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:50:11.828 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:50:11.828 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:50:11.828 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:50:11.828 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:50:11.828 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:50:11.828 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:50:11.828 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:50:11.828 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:50:11.829 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:50:11.829 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:50:11.829 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:50:11.829 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:50:11.829 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:50:11.829 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:50:11.829 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:50:11.829 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:50:11.829 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:50:11.829 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:50:11.829 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:50:11.829 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:50:11.830 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:50:11.830 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:50:11.830 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:50:11.830 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:50:11.830 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:50:11.830 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:50:11.830 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:50:11.830 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:50:11.830 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:50:11.830 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:50:11.830 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:50:11.830 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:50:11.831 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:50:11.831 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:50:11.831 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:50:11.831 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:50:11.831 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:50:11.831 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:50:11.831 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:50:11.831 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:50:11.831 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:50:11.831 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:50:11.831 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:50:11.831 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:50:11.831 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:50:11.831 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:50:11.832 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:50:11.832 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:50:11.832 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:50:11.832 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:50:11.832 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:50:11.832 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:50:11.832 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:50:11.832 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:50:11.832 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:50:11.832 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:50:11.832 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:50:11.832 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:50:11.832 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:50:11.833 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:50:11.833 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:50:11.833 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:50:11.833 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:50:11.833 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:50:11.833 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:50:11.833 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:50:11.833 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:50:11.833 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:50:11.833 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:50:11.833 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:50:11.833 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:50:11.833 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:50:11.834 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:50:11.834 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:50:11.834 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:50:11.834 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:50:11.834 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:50:11.834 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:50:11.834 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:50:11.834 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:50:11.834 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:50:11.834 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:50:11.834 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:50:11.834 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:50:11.834 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:50:11.834 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:50:11.835 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:50:11.835 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:50:11.835 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:50:11.835 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:50:11.835 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:50:11.835 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 2 2024-09-16T03:50:11.835 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:50:11.835 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:50:11.835 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:50:11.835 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:50:11.835 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:50:11.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.802+0000 7fb4d54d1640 10 librados: watch_flush enter 2024-09-16T03:50:11.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.802+0000 7fb4d54d1640 10 librados: watch_flush exit 2024-09-16T03:50:11.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.802+0000 7fb4d54d1640 20 client.804110.objecter shutdown clearing up homeless session... 2024-09-16T03:50:11.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.802+0000 7fb4d54d1640 10 client.804110.objecter successfully canceled tick 2024-09-16T03:50:11.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.802+0000 7fb4d54d1640 1 -- 172.21.15.42:0/3898591137 >> v2:172.21.15.154:6812/254845342 conn(0x7fb4bc037810 msgr2=0x7fb4bc039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:50:11.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.802+0000 7fb4d54d1640 1 --2- 172.21.15.42:0/3898591137 >> v2:172.21.15.154:6812/254845342 conn(0x7fb4bc037810 0x7fb4bc039cd0 secure :-1 s=READY pgs=3535 cs=0 l=1 rev1=1 crypto rx=0x7fb4c80012f0 tx=0x7fb4c8009040 comp rx=0 tx=0).stop 2024-09-16T03:50:11.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.802+0000 7fb4d54d1640 10 monclient: shutdown 2024-09-16T03:50:11.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.802+0000 7fb4d54d1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:50:11.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.802+0000 7fb4d54d1640 1 -- 172.21.15.42:0/3898591137 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb4d00a1330 msgr2=0x7fb4d013c050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:50:11.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.802+0000 7fb4d54d1640 1 --2- 172.21.15.42:0/3898591137 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb4d00a1330 0x7fb4d013c050 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fb4c000ec10 tx=0x7fb4c000c520 comp rx=0 tx=0).stop 2024-09-16T03:50:11.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.802+0000 7fb4cf7fe640 1 -- 172.21.15.42:0/3898591137 reap_dead start 2024-09-16T03:50:11.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.802+0000 7fb4cf7fe640 1 -- 172.21.15.42:0/3898591137 reap_dead start 2024-09-16T03:50:11.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.802+0000 7fb4d54d1640 1 -- 172.21.15.42:0/3898591137 shutdown_connections 2024-09-16T03:50:11.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.802+0000 7fb4d54d1640 1 -- 172.21.15.42:0/3898591137 >> 172.21.15.42:0/3898591137 conn(0x7fb4d00a2420 msgr2=0x7fb4d00a3f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:50:11.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.806+0000 7fb4d54d1640 1 -- 172.21.15.42:0/3898591137 shutdown_connections 2024-09-16T03:50:11.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.806+0000 7fb4d54d1640 1 -- 172.21.15.42:0/3898591137 wait complete. 2024-09-16T03:50:11.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.806+0000 7fb4d54d1640 1 librados: shutdown 2024-09-16T03:50:11.839 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:50:11.839 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T03:50:11.839 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T03:50:11.952 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.946+0000 7f3b30d4d640 10 monclient: get_monmap_and_config 2024-09-16T03:50:11.953 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.946+0000 7f3b30d4d640 10 monclient: build_initial_monmap 2024-09-16T03:50:11.953 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.946+0000 7f3b30d4d640 10 monclient: monmap: 2024-09-16T03:50:11.953 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:50:11.953 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:50:11.953 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:50:11.951538+0000 2024-09-16T03:50:11.953 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:50:11.951538+0000 2024-09-16T03:50:11.953 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:50:11.953 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:50:11.953 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:50:11.953 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:50:11.953 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:50:11.953 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:50:11.954 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:50:11.954 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:50:11.958 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.950+0000 7f3b30d4d640 1 Processor -- start 2024-09-16T03:50:11.958 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.950+0000 7f3b30d4d640 1 -- start start 2024-09-16T03:50:11.958 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.950+0000 7f3b30d4d640 10 monclient: init 2024-09-16T03:50:11.958 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.950+0000 7f3b30d4d640 10 monclient: _reopen_session rank -1 2024-09-16T03:50:11.958 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.950+0000 7f3b30d4d640 10 monclient: _add_conns ranks=[2,4,3,1,0] 2024-09-16T03:50:11.958 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.950+0000 7f3b30d4d640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3b2c09fb00 0x7f3b2c09ba50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:50:11.958 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.950+0000 7f3b30d4d640 10 monclient(hunting): picked mon.noname-b con 0x7f3b2c09fb00 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:50:11.958 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.950+0000 7f3b30d4d640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f3b2c09bf90 0x7f3b2c09c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:50:11.959 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.950+0000 7f3b30d4d640 10 monclient(hunting): picked mon.noname-c con 0x7f3b2c09bf90 addr v2:172.21.15.154:3300/0 2024-09-16T03:50:11.959 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.950+0000 7f3b30d4d640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f3b2c138110 0x7f3b2c13a500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:50:11.959 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.950+0000 7f3b30d4d640 10 monclient(hunting): picked mon.noname-a con 0x7f3b2c138110 addr v2:172.21.15.42:3300/0 2024-09-16T03:50:11.959 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.950+0000 7f3b30d4d640 10 monclient(hunting): start opening mon connection 2024-09-16T03:50:11.959 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.950+0000 7f3b30d4d640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f3b2c09e250 con 0x7f3b2c138110 2024-09-16T03:50:11.959 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.950+0000 7f3b30d4d640 10 monclient(hunting): start opening mon connection 2024-09-16T03:50:11.959 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.950+0000 7f3b30d4d640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f3b2c0a1e20 con 0x7f3b2c09bf90 2024-09-16T03:50:11.959 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.950+0000 7f3b30d4d640 10 monclient(hunting): start opening mon connection 2024-09-16T03:50:11.959 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.950+0000 7f3b30d4d640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f3b2c09ea30 con 0x7f3b2c09fb00 2024-09-16T03:50:11.959 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.950+0000 7f3b30d4d640 10 monclient(hunting): _renew_subs 2024-09-16T03:50:11.959 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.950+0000 7f3b30d4d640 10 monclient(hunting): authenticate will time out at 1458.921069s 2024-09-16T03:50:11.960 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.950+0000 7f3b2a59c640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3b2c09fb00 0x7f3b2c09ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:50:11.960 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.950+0000 7f3b2a59c640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3b2c09fb00 0x7f3b2c09ba50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:44404/0 (socket says 172.21.15.42:44404) 2024-09-16T03:50:11.960 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.950+0000 7f3b2ad9d640 1 --2- 172.21.15.42:0/3148534781 >> v2:172.21.15.42:3300/0 conn(0x7f3b2c138110 0x7f3b2c13a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:50:11.960 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.950+0000 7f3b2a59c640 1 -- 172.21.15.42:0/3148534781 learned_addr learned my addr 172.21.15.42:0/3148534781 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:50:11.960 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.950+0000 7f3b2a59c640 10 monclient(hunting): get_auth_request con 0x7f3b2c09fb00 auth_method 0 2024-09-16T03:50:11.960 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.950+0000 7f3b2a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:50:11.960 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.950+0000 7f3b2a59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:50:11.960 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.950+0000 7f3b2a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:50:11.960 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.950+0000 7f3b2ad9d640 10 monclient(hunting): get_auth_request con 0x7f3b2c138110 auth_method 0 2024-09-16T03:50:11.960 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.950+0000 7f3b2ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:50:11.960 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.950+0000 7f3b2ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:50:11.960 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.950+0000 7f3b2ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:50:11.960 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.950+0000 7f3b23fff640 1 --2- 172.21.15.42:0/3148534781 >> v2:172.21.15.154:3300/0 conn(0x7f3b2c09bf90 0x7f3b2c09c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:50:11.961 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.950+0000 7f3b23fff640 10 monclient(hunting): get_auth_request con 0x7f3b2c09bf90 auth_method 0 2024-09-16T03:50:11.961 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.950+0000 7f3b23fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:50:11.961 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.950+0000 7f3b23fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:50:11.961 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.950+0000 7f3b23fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:50:11.961 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.950+0000 7f3b2a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:50:11.961 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.950+0000 7f3b2a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:50:11.961 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.950+0000 7f3b2a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:50:11.961 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.950+0000 7f3b2ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:50:11.961 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.950+0000 7f3b2ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:50:11.961 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.950+0000 7f3b2ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:50:11.961 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.950+0000 7f3b23fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:50:11.961 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.950+0000 7f3b23fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:50:11.961 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.950+0000 7f3b23fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:50:11.962 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.950+0000 7f3b2a59c640 10 monclient(hunting): handle_auth_done global_id 804113 payload 274 2024-09-16T03:50:11.962 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.950+0000 7f3b2a59c640 1 -- 172.21.15.42:0/3148534781 >> v2:172.21.15.154:3300/0 conn(0x7f3b2c09bf90 msgr2=0x7f3b2c09c3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:50:11.962 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.950+0000 7f3b2a59c640 1 --2- 172.21.15.42:0/3148534781 >> v2:172.21.15.154:3300/0 conn(0x7f3b2c09bf90 0x7f3b2c09c3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:50:11.962 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.950+0000 7f3b2a59c640 1 -- 172.21.15.42:0/3148534781 >> v2:172.21.15.42:3300/0 conn(0x7f3b2c138110 msgr2=0x7f3b2c13a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:50:11.962 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.950+0000 7f3b2a59c640 1 --2- 172.21.15.42:0/3148534781 >> v2:172.21.15.42:3300/0 conn(0x7f3b2c138110 0x7f3b2c13a500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:50:11.962 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.950+0000 7f3b2a59c640 10 monclient: _finish_hunting 0 2024-09-16T03:50:11.962 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.950+0000 7f3b2a59c640 1 monclient: found mon.noname-b 2024-09-16T03:50:11.962 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.950+0000 7f3b2a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:50:11.962 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.950+0000 7f3b2a59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T03:50:11.962 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.950+0000 7f3b2a59c640 1 -- 172.21.15.42:0/3148534781 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b2c09fee0 con 0x7f3b2c09fb00 2024-09-16T03:50:11.963 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.950+0000 7f3b2ad9d640 1 --2- 172.21.15.42:0/3148534781 >> v2:172.21.15.42:3300/0 conn(0x7f3b2c138110 0x7f3b2c13a500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:50:11.963 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.950+0000 7f3b2a59c640 1 --2- 172.21.15.42:0/3148534781 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3b2c09fb00 0x7f3b2c09ba50 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f3b2c0a69e0 tx=0x7f3b1c01e5f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=36f6d3dab1c309fc server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:50:11.963 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.954+0000 7f3b29d9b640 1 -- 172.21.15.42:0/3148534781 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b1c0ad650 con 0x7f3b2c09fb00 2024-09-16T03:50:11.963 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.954+0000 7f3b29d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:50:11.963 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.954+0000 7f3b29d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:50:11.963 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.954+0000 7f3b29d9b640 10 monclient: dump: 2024-09-16T03:50:11.964 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:50:11.964 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:50:11.964 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:11.964 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:11.964 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:50:11.964 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:50:11.964 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:50:11.964 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:50:11.964 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:50:11.964 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:50:11.964 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.954+0000 7f3b29d9b640 10 monclient: _finish_auth 0 2024-09-16T03:50:11.964 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.954+0000 7f3b29d9b640 10 monclient: _check_auth_tickets 2024-09-16T03:50:11.964 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.954+0000 7f3b29d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:50:11.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.954+0000 7f3b30d4d640 5 monclient: authenticate success, global_id 804113 2024-09-16T03:50:11.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.954+0000 7f3b29d9b640 1 -- 172.21.15.42:0/3148534781 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3b1c0b2070 con 0x7f3b2c09fb00 2024-09-16T03:50:11.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.954+0000 7f3b30d4d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:50:11.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.954+0000 7f3b29d9b640 10 monclient: handle_config config(0 keys) 2024-09-16T03:50:11.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.954+0000 7f3b30d4d640 10 monclient: get_monmap_and_config success 2024-09-16T03:50:11.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.954+0000 7f3b29d9b640 1 -- 172.21.15.42:0/3148534781 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b1c0ada10 con 0x7f3b2c09fb00 2024-09-16T03:50:11.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.954+0000 7f3b29d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:50:11.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.954+0000 7f3b30d4d640 4 set_mon_vals no callback set 2024-09-16T03:50:11.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.954+0000 7f3b29d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:50:11.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.954+0000 7f3b29d9b640 10 monclient: dump: 2024-09-16T03:50:11.965 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:50:11.965 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:50:11.965 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:11.966 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:11.966 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:50:11.966 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:50:11.966 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:50:11.966 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:50:11.966 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:50:11.966 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:50:11.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.954+0000 7f3b30d4d640 10 monclient: shutdown 2024-09-16T03:50:11.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.954+0000 7f3b30d4d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:50:11.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.954+0000 7f3b30d4d640 1 -- 172.21.15.42:0/3148534781 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3b2c09fb00 msgr2=0x7f3b2c09ba50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:50:11.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.954+0000 7f3b30d4d640 1 --2- 172.21.15.42:0/3148534781 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3b2c09fb00 0x7f3b2c09ba50 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f3b2c0a69e0 tx=0x7f3b1c01e5f0 comp rx=0 tx=0).stop 2024-09-16T03:50:11.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.954+0000 7f3b30d4d640 1 -- 172.21.15.42:0/3148534781 shutdown_connections 2024-09-16T03:50:11.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.954+0000 7f3b30d4d640 1 --2- 172.21.15.42:0/3148534781 >> v2:172.21.15.42:3300/0 conn(0x7f3b2c138110 0x7f3b2c13a500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:50:11.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.954+0000 7f3b30d4d640 1 --2- 172.21.15.42:0/3148534781 >> v2:172.21.15.154:3300/0 conn(0x7f3b2c09bf90 0x7f3b2c09c3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:50:11.967 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.954+0000 7f3b30d4d640 1 --2- 172.21.15.42:0/3148534781 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3b2c09fb00 0x7f3b2c09ba50 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:50:11.967 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.954+0000 7f3b30d4d640 1 -- 172.21.15.42:0/3148534781 >> 172.21.15.42:0/3148534781 conn(0x7f3b2c0a2420 msgr2=0x7f3b2c0a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:50:11.967 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.954+0000 7f3b30d4d640 1 -- 172.21.15.42:0/3148534781 shutdown_connections 2024-09-16T03:50:11.967 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.954+0000 7f3b30d4d640 1 -- 172.21.15.42:0/3148534781 wait complete. 2024-09-16T03:50:11.972 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.954+0000 7f3b30d4d640 10 monclient: build_initial_monmap 2024-09-16T03:50:11.972 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.954+0000 7f3b30d4d640 10 monclient: monmap: 2024-09-16T03:50:11.972 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:50:11.972 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:50:11.972 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:50:11.972 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:50:11.972 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:50:11.972 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:50:11.973 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:50:11.973 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:50:11.973 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:50:11.973 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:50:11.973 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.954+0000 7f3b30d4d640 1 librados: starting msgr at 2024-09-16T03:50:11.973 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.954+0000 7f3b30d4d640 1 librados: starting objecter 2024-09-16T03:50:11.973 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.954+0000 7f3b30d4d640 1 Processor -- start 2024-09-16T03:50:11.973 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.954+0000 7f3b30d4d640 1 -- start start 2024-09-16T03:50:11.973 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.954+0000 7f3b30d4d640 1 librados: setting wanted keys 2024-09-16T03:50:11.973 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.954+0000 7f3b30d4d640 1 librados: calling monclient init 2024-09-16T03:50:11.973 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.954+0000 7f3b30d4d640 10 monclient: init 2024-09-16T03:50:11.973 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.954+0000 7f3b30d4d640 10 monclient: _reopen_session rank -1 2024-09-16T03:50:11.973 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.954+0000 7f3b30d4d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:50:11.973 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.954+0000 7f3b30d4d640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3b2c09bf90 0x7f3b2c1ced80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:50:11.974 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.954+0000 7f3b30d4d640 10 monclient(hunting): picked mon.noname-c con 0x7f3b2c09bf90 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:50:11.974 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.954+0000 7f3b30d4d640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3b2c09fb00 0x7f3b2c136690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:50:11.974 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.954+0000 7f3b30d4d640 10 monclient(hunting): picked mon.noname-a con 0x7f3b2c09fb00 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:50:11.974 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.954+0000 7f3b30d4d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3b2c138110 0x7f3b2c136bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:50:11.974 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.954+0000 7f3b30d4d640 10 monclient(hunting): picked mon.noname-b con 0x7f3b2c138110 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:50:11.974 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.954+0000 7f3b30d4d640 10 monclient(hunting): start opening mon connection 2024-09-16T03:50:11.974 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.954+0000 7f3b30d4d640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f3b2c0a32f0 con 0x7f3b2c09fb00 2024-09-16T03:50:11.974 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.954+0000 7f3b30d4d640 10 monclient(hunting): start opening mon connection 2024-09-16T03:50:11.974 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.954+0000 7f3b30d4d640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f3b2c0a3170 con 0x7f3b2c138110 2024-09-16T03:50:11.974 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.954+0000 7f3b30d4d640 10 monclient(hunting): start opening mon connection 2024-09-16T03:50:11.975 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.954+0000 7f3b30d4d640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f3b2c09d380 con 0x7f3b2c09bf90 2024-09-16T03:50:11.975 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.954+0000 7f3b30d4d640 10 monclient(hunting): _renew_subs 2024-09-16T03:50:11.975 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.954+0000 7f3b30d4d640 10 monclient(hunting): authenticate will time out at 1458.926853s 2024-09-16T03:50:11.975 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.954+0000 7f3b2a59c640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3b2c09bf90 0x7f3b2c1ced80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:50:11.975 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.954+0000 7f3b2a59c640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3b2c09bf90 0x7f3b2c1ced80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:44410/0 (socket says 172.21.15.42:44410) 2024-09-16T03:50:11.975 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.954+0000 7f3b2a59c640 1 -- 172.21.15.42:0/1325753526 learned_addr learned my addr 172.21.15.42:0/1325753526 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:50:11.975 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.954+0000 7f3b2a59c640 10 monclient(hunting): get_auth_request con 0x7f3b2c09bf90 auth_method 0 2024-09-16T03:50:11.975 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.954+0000 7f3b2a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:50:11.975 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.954+0000 7f3b2a59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:50:11.975 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.954+0000 7f3b2a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:50:11.975 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.954+0000 7f3b23fff640 1 --2- 172.21.15.42:0/1325753526 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3b2c09fb00 0x7f3b2c136690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:50:11.975 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.954+0000 7f3b23fff640 10 monclient(hunting): get_auth_request con 0x7f3b2c09fb00 auth_method 0 2024-09-16T03:50:11.975 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.954+0000 7f3b23fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:50:11.975 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.954+0000 7f3b23fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:50:11.976 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.954+0000 7f3b23fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:50:11.976 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.954+0000 7f3b2a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:50:11.976 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.954+0000 7f3b2a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:50:11.976 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.954+0000 7f3b2a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:50:11.976 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.954+0000 7f3b23fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:50:11.976 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.954+0000 7f3b23fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:50:11.976 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.954+0000 7f3b23fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:50:11.976 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.954+0000 7f3b2ad9d640 1 --2- 172.21.15.42:0/1325753526 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3b2c138110 0x7f3b2c136bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:50:11.976 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.954+0000 7f3b2ad9d640 10 monclient(hunting): get_auth_request con 0x7f3b2c138110 auth_method 0 2024-09-16T03:50:11.976 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.954+0000 7f3b2ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:50:11.976 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.954+0000 7f3b2ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:50:11.976 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.954+0000 7f3b2ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:50:11.976 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.954+0000 7f3b2a59c640 10 monclient(hunting): handle_auth_done global_id 804116 payload 824 2024-09-16T03:50:11.976 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.954+0000 7f3b2a59c640 1 -- 172.21.15.42:0/1325753526 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3b2c138110 msgr2=0x7f3b2c136bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:50:11.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.954+0000 7f3b2a59c640 1 --2- 172.21.15.42:0/1325753526 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3b2c138110 0x7f3b2c136bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:50:11.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.954+0000 7f3b2a59c640 1 -- 172.21.15.42:0/1325753526 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3b2c09fb00 msgr2=0x7f3b2c136690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:50:11.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.954+0000 7f3b2a59c640 1 --2- 172.21.15.42:0/1325753526 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3b2c09fb00 0x7f3b2c136690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:50:11.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.954+0000 7f3b2a59c640 10 monclient: _finish_hunting 0 2024-09-16T03:50:11.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.954+0000 7f3b2a59c640 1 monclient: found mon.noname-c 2024-09-16T03:50:11.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.954+0000 7f3b2a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:50:11.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.954+0000 7f3b2a59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:50:11.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.954+0000 7f3b2a59c640 1 -- 172.21.15.42:0/1325753526 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b2c1cf2c0 con 0x7f3b2c09bf90 2024-09-16T03:50:11.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.954+0000 7f3b23fff640 1 --2- 172.21.15.42:0/1325753526 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3b2c09fb00 0x7f3b2c136690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:50:11.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.954+0000 7f3b2a59c640 1 --2- 172.21.15.42:0/1325753526 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3b2c09bf90 0x7f3b2c1ced80 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f3b1c01ee90 tx=0x7f3b1c01ce30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:50:11.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.954+0000 7f3b22ffd640 10 client.?.objecter ms_handle_connect 0x7f3b2c09bf90 2024-09-16T03:50:11.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.954+0000 7f3b22ffd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:50:11.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.954+0000 7f3b22ffd640 1 -- 172.21.15.42:0/1325753526 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b1c0bc070 con 0x7f3b2c09bf90 2024-09-16T03:50:11.978 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.954+0000 7f3b22ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:50:11.978 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.954+0000 7f3b22ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:50:11.978 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.954+0000 7f3b22ffd640 10 monclient: dump: 2024-09-16T03:50:11.978 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:50:11.978 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:50:11.978 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:11.978 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:11.978 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:50:11.978 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:50:11.978 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:50:11.978 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:50:11.978 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:50:11.978 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:50:11.978 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.954+0000 7f3b22ffd640 10 monclient: _finish_auth 0 2024-09-16T03:50:11.979 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.954+0000 7f3b22ffd640 10 monclient: _check_auth_tickets 2024-09-16T03:50:11.979 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.954+0000 7f3b22ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:50:11.979 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.954+0000 7f3b22ffd640 1 -- 172.21.15.42:0/1325753526 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3b1c0b8640 con 0x7f3b2c09bf90 2024-09-16T03:50:11.979 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.954+0000 7f3b30d4d640 5 monclient: authenticate success, global_id 804116 2024-09-16T03:50:11.979 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.954+0000 7f3b30d4d640 10 monclient: _renew_subs 2024-09-16T03:50:11.979 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.954+0000 7f3b30d4d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:50:11.979 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.954+0000 7f3b30d4d640 1 -- 172.21.15.42:0/1325753526 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b2c13a250 con 0x7f3b2c09bf90 2024-09-16T03:50:11.979 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.954+0000 7f3b22ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:50:11.979 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.954+0000 7f3b22ffd640 1 -- 172.21.15.42:0/1325753526 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b1c0b8a30 con 0x7f3b2c09bf90 2024-09-16T03:50:11.979 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.954+0000 7f3b22ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:50:11.979 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.954+0000 7f3b22ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:50:11.979 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.954+0000 7f3b22ffd640 10 monclient: dump: 2024-09-16T03:50:11.979 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:50:11.979 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:50:11.980 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:11.980 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:11.980 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:50:11.980 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:50:11.980 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:50:11.980 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:50:11.980 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:50:11.980 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:50:11.980 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.954+0000 7f3b2959a640 4 set_mon_vals no callback set 2024-09-16T03:50:11.980 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.954+0000 7f3b30d4d640 10 client.804116.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:50:11.980 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.954+0000 7f3b30d4d640 10 monclient: _renew_subs 2024-09-16T03:50:11.980 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.954+0000 7f3b30d4d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:50:11.980 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.954+0000 7f3b30d4d640 1 -- 172.21.15.42:0/1325753526 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b2c073a30 con 0x7f3b2c09bf90 2024-09-16T03:50:11.981 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.958+0000 7f3b30d4d640 1 librados: init done 2024-09-16T03:50:11.981 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.958+0000 7f3b22ffd640 1 -- 172.21.15.42:0/1325753526 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f3b1c0bf070 con 0x7f3b2c09bf90 2024-09-16T03:50:11.981 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.958+0000 7f3b30d4d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:50:11.981 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.958+0000 7f3b30d4d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:50:11.981 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.958+0000 7f3b30d4d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:50:11.981 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.958+0000 7f3b30d4d640 1 -- 172.21.15.42:0/1325753526 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3b2c09c3d0 con 0x7f3b2c09bf90 2024-09-16T03:50:11.981 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.958+0000 7f3b22ffd640 1 --2- 172.21.15.42:0/1325753526 >> v2:172.21.15.154:6812/254845342 conn(0x7f3b0c037810 0x7f3b0c039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:50:11.981 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.958+0000 7f3b22ffd640 1 -- 172.21.15.42:0/1325753526 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(279..279 src has 1..279) ==== 5421+0+0 (secure 0 0 0) 0x7f3b1c0e3520 con 0x7f3b2c09bf90 2024-09-16T03:50:11.981 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.958+0000 7f3b22ffd640 10 client.804116.objecter ms_dispatch 0x7f3b2c000ba0 osd_map(279..279 src has 1..279) 2024-09-16T03:50:11.981 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.958+0000 7f3b22ffd640 3 client.804116.objecter handle_osd_map got epochs [279,279] > 0 2024-09-16T03:50:11.981 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.958+0000 7f3b22ffd640 3 client.804116.objecter handle_osd_map decoding full epoch 279 2024-09-16T03:50:11.982 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.958+0000 7f3b22ffd640 20 client.804116.objecter dump_active .. 0 homeless 2024-09-16T03:50:11.982 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.958+0000 7f3b23fff640 1 --2- 172.21.15.42:0/1325753526 >> v2:172.21.15.154:6812/254845342 conn(0x7f3b0c037810 0x7f3b0c039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:50:11.982 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.958+0000 7f3b23fff640 10 monclient: get_auth_request con 0x7f3b0c037810 auth_method 0 2024-09-16T03:50:11.982 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.958+0000 7f3b23fff640 1 --2- 172.21.15.42:0/1325753526 >> v2:172.21.15.154:6812/254845342 conn(0x7f3b0c037810 0x7f3b0c039cd0 secure :-1 s=READY pgs=3536 cs=0 l=1 rev1=1 crypto rx=0x7f3b2c123bb0 tx=0x7f3b14009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:50:11.982 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.958+0000 7f3b22ffd640 10 client.804116.objecter ms_handle_connect 0x7f3b0c037810 2024-09-16T03:50:11.982 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.962+0000 7f3b22ffd640 1 -- 172.21.15.42:0/1325753526 <== mon.2 v2:172.21.15.42:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f3b1c018300 con 0x7f3b2c09bf90 2024-09-16T03:50:11.982 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.962+0000 7f3b22ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:50:11.982 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:11.962+0000 7f3b22ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:50:12.197 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:12.194+0000 7f3b30d4d640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T03:50:12.197 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:12.194+0000 7f3b30d4d640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T03:50:12.197 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:12.194+0000 7f3b30d4d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:50:12.197 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:12.194+0000 7f3b30d4d640 1 -- 172.21.15.42:0/1325753526 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f3b2c124660 con 0x7f3b2c09bf90 2024-09-16T03:50:12.201 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.198+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:50:12.201 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.198+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=28,config=2,health=12,kv=9,logm=49} crc {auth=604439021,config=692619927,health=1812866750,kv=2818795783,logm=2178098247}) 2024-09-16T03:50:12.201 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.198+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=25,config=2,health=10,kv=10,logm=53} crc {auth=614387175,config=692619927,health=2157799236,kv=3446188498,logm=103609447}) 2024-09-16T03:50:12.201 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.198+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:50:12.201 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.198+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=28,config=2,health=12,kv=9,logm=49} crc {auth=604439021,config=692619927,health=1812866750,kv=2818795783,logm=2178098247}) 2024-09-16T03:50:12.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:12.198+0000 7f3b22ffd640 1 -- 172.21.15.42:0/1325753526 <== mon.2 v2:172.21.15.42:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f3b1c0bf520 con 0x7f3b2c09bf90 2024-09-16T03:50:12.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:12.198+0000 7f3b22ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T03:50:12.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:12.198+0000 7f3b22ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:50:12.202 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.198+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=29,config=2,health=12,kv=10,logm=47} crc {auth=4002353474,config=692619927,health=1812866750,kv=3446188498,logm=50838247}) 2024-09-16T03:50:12.204 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.198+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:50:12.204 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.198+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2898738458}) 2024-09-16T03:50:12.204 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.198+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1723144721}) 2024-09-16T03:50:12.204 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.198+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:50:12.205 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:12.198+0000 7f3b30d4d640 10 librados: watch_flush enter 2024-09-16T03:50:12.205 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.198+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2898738458}) 2024-09-16T03:50:12.205 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.198+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1835365210}) 2024-09-16T03:50:12.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:12.198+0000 7f3b30d4d640 10 librados: watch_flush exit 2024-09-16T03:50:12.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:12.198+0000 7f3b30d4d640 20 client.804116.objecter shutdown clearing up homeless session... 2024-09-16T03:50:12.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:12.198+0000 7f3b30d4d640 10 client.804116.objecter successfully canceled tick 2024-09-16T03:50:12.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:12.198+0000 7f3b30d4d640 1 -- 172.21.15.42:0/1325753526 >> v2:172.21.15.154:6812/254845342 conn(0x7f3b0c037810 msgr2=0x7f3b0c039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:50:12.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:12.198+0000 7f3b30d4d640 1 --2- 172.21.15.42:0/1325753526 >> v2:172.21.15.154:6812/254845342 conn(0x7f3b0c037810 0x7f3b0c039cd0 secure :-1 s=READY pgs=3536 cs=0 l=1 rev1=1 crypto rx=0x7f3b2c123bb0 tx=0x7f3b14009190 comp rx=0 tx=0).stop 2024-09-16T03:50:12.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:12.198+0000 7f3b30d4d640 10 monclient: shutdown 2024-09-16T03:50:12.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:12.198+0000 7f3b30d4d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:50:12.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:12.198+0000 7f3b30d4d640 1 -- 172.21.15.42:0/1325753526 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3b2c09bf90 msgr2=0x7f3b2c1ced80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:50:12.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:12.198+0000 7f3b30d4d640 1 --2- 172.21.15.42:0/1325753526 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3b2c09bf90 0x7f3b2c1ced80 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f3b1c01ee90 tx=0x7f3b1c01ce30 comp rx=0 tx=0).stop 2024-09-16T03:50:12.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:12.198+0000 7f3b30d4d640 1 -- 172.21.15.42:0/1325753526 shutdown_connections 2024-09-16T03:50:12.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:12.198+0000 7f3b30d4d640 1 --2- 172.21.15.42:0/1325753526 >> v2:172.21.15.154:6812/254845342 conn(0x7f3b0c037810 0x7f3b0c039cd0 unknown :-1 s=CLOSED pgs=3536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:50:12.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:12.198+0000 7f3b30d4d640 1 --2- 172.21.15.42:0/1325753526 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3b2c138110 0x7f3b2c136bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:50:12.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:12.198+0000 7f3b30d4d640 1 --2- 172.21.15.42:0/1325753526 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3b2c09fb00 0x7f3b2c136690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:50:12.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:12.198+0000 7f3b30d4d640 1 --2- 172.21.15.42:0/1325753526 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3b2c09bf90 0x7f3b2c1ced80 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:50:12.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:12.198+0000 7f3b30d4d640 1 -- 172.21.15.42:0/1325753526 >> 172.21.15.42:0/1325753526 conn(0x7f3b2c0a2420 msgr2=0x7f3b2c071d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:50:12.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:12.202+0000 7f3b30d4d640 1 -- 172.21.15.42:0/1325753526 shutdown_connections 2024-09-16T03:50:12.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:12.202+0000 7f3b30d4d640 1 -- 172.21.15.42:0/1325753526 wait complete. 2024-09-16T03:50:12.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:12.202+0000 7f3b30d4d640 1 librados: shutdown 2024-09-16T03:50:12.208 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.202+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:50:12.208 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.202+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2611423637}) 2024-09-16T03:50:12.208 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.202+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1581544975}) 2024-09-16T03:50:12.208 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.202+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:50:12.208 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.202+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2611423637}) 2024-09-16T03:50:12.208 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.202+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1215607435}) 2024-09-16T03:50:12.209 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.202+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:50:12.209 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.202+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3172529125}) 2024-09-16T03:50:12.209 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.202+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=823779483}) 2024-09-16T03:50:12.209 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.202+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:50:12.209 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.202+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3172529125}) 2024-09-16T03:50:12.209 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.202+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1944333694}) 2024-09-16T03:50:12.210 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.206+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:50:12.211 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.206+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=753385352}) 2024-09-16T03:50:12.211 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.206+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=324504626}) 2024-09-16T03:50:12.211 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.206+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:50:12.211 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.206+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=753385352}) 2024-09-16T03:50:12.211 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.206+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3949233725}) 2024-09-16T03:50:12.212 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.206+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:50:12.212 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.206+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=741619428}) 2024-09-16T03:50:12.212 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.206+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2135749077}) 2024-09-16T03:50:12.212 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.206+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:50:12.212 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.206+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=741619428}) 2024-09-16T03:50:12.213 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.206+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1019064194}) 2024-09-16T03:50:12.214 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.210+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:50:12.214 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.210+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1306519154}) 2024-09-16T03:50:12.214 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.210+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1678258736}) 2024-09-16T03:50:12.214 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.210+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:50:12.214 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.210+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1306519154}) 2024-09-16T03:50:12.214 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.210+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2030580164}) 2024-09-16T03:50:12.216 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.210+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:50:12.216 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.210+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3393670566}) 2024-09-16T03:50:12.216 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.210+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3724942798}) 2024-09-16T03:50:12.216 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.210+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:50:12.216 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.210+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3393670566}) 2024-09-16T03:50:12.216 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.210+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1207846776}) 2024-09-16T03:50:12.217 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.214+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:50:12.217 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.214+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4273678394}) 2024-09-16T03:50:12.217 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.214+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4289598867}) 2024-09-16T03:50:12.217 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.214+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:50:12.218 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.214+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4273678394}) 2024-09-16T03:50:12.218 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.214+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3024364713}) 2024-09-16T03:50:12.219 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.214+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:50:12.219 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.214+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3794762579}) 2024-09-16T03:50:12.219 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.214+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2660100777}) 2024-09-16T03:50:12.220 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.214+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:50:12.220 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.214+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3794762579}) 2024-09-16T03:50:12.220 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.214+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4247729133}) 2024-09-16T03:50:12.221 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.214+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:50:12.221 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.218+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1557847432}) 2024-09-16T03:50:12.221 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.218+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=660637973}) 2024-09-16T03:50:12.221 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.218+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:50:12.221 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.218+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1557847432}) 2024-09-16T03:50:12.221 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.218+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2410363843}) 2024-09-16T03:50:12.223 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.218+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:50:12.223 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.218+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2909764730}) 2024-09-16T03:50:12.223 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.218+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3739816454}) 2024-09-16T03:50:12.223 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.218+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:50:12.223 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.218+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2909764730}) 2024-09-16T03:50:12.223 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.218+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2819599961}) 2024-09-16T03:50:12.224 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T03:50:12.224 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.218+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:50:12.224 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.218+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1955413584}) 2024-09-16T03:50:12.224 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.218+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=516387363}) 2024-09-16T03:50:12.224 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.218+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:50:12.224 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.218+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1955413584}) 2024-09-16T03:50:12.225 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.218+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3340586390}) 2024-09-16T03:50:12.226 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.222+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:50:12.226 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.222+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2115041203}) 2024-09-16T03:50:12.226 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.222+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=912396437}) 2024-09-16T03:50:12.226 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.222+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:50:12.226 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.222+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2115041203}) 2024-09-16T03:50:12.226 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.222+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=260489729}) 2024-09-16T03:50:12.228 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.222+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:50:12.228 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.222+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=686920442}) 2024-09-16T03:50:12.228 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.222+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1792472296}) 2024-09-16T03:50:12.228 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.222+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:50:12.228 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.222+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=686920442}) 2024-09-16T03:50:12.228 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.222+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3314821916}) 2024-09-16T03:50:12.229 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.226+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:50:12.229 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.226+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3501394309}) 2024-09-16T03:50:12.229 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.226+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=656565511}) 2024-09-16T03:50:12.229 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.226+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:50:12.229 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.226+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3501394309}) 2024-09-16T03:50:12.229 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.226+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3128198860}) 2024-09-16T03:50:12.231 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.226+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:50:12.231 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.226+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3120796975}) 2024-09-16T03:50:12.231 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.226+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1307412231}) 2024-09-16T03:50:12.231 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.226+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:50:12.231 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.226+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3120796975}) 2024-09-16T03:50:12.231 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.226+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2596126207}) 2024-09-16T03:50:12.232 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.226+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:50:12.233 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.226+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3720541879}) 2024-09-16T03:50:12.233 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.226+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2968648998}) 2024-09-16T03:50:12.233 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.230+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:50:12.233 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.230+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3720541879}) 2024-09-16T03:50:12.233 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.230+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=275160268}) 2024-09-16T03:50:12.234 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.230+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:50:12.234 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.230+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=447284563}) 2024-09-16T03:50:12.234 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.230+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1919786654}) 2024-09-16T03:50:12.234 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.230+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:50:12.234 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.230+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=447284563}) 2024-09-16T03:50:12.234 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.230+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2206349257}) 2024-09-16T03:50:12.236 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.230+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:50:12.236 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.230+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1799823947}) 2024-09-16T03:50:12.236 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.230+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1225940608}) 2024-09-16T03:50:12.236 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.230+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:50:12.236 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.230+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1799823947}) 2024-09-16T03:50:12.236 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.230+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1778159367}) 2024-09-16T03:50:12.238 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.234+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:50:12.238 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.234+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2107690403}) 2024-09-16T03:50:12.238 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.234+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3753013530}) 2024-09-16T03:50:12.238 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.234+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:50:12.238 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.234+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2107690403}) 2024-09-16T03:50:12.238 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.234+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3581497483}) 2024-09-16T03:50:12.239 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.234+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:50:12.239 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.234+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1020427524}) 2024-09-16T03:50:12.239 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.234+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=529242837}) 2024-09-16T03:50:12.239 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.234+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:50:12.239 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.234+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1020427524}) 2024-09-16T03:50:12.240 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.234+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=151286632}) 2024-09-16T03:50:12.241 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.238+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:50:12.241 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.238+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2991640058}) 2024-09-16T03:50:12.241 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.238+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1203407400}) 2024-09-16T03:50:12.241 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.238+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:50:12.241 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.238+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2991640058}) 2024-09-16T03:50:12.241 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.238+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1881682715}) 2024-09-16T03:50:12.243 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.238+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:50:12.243 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.238+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3791041072}) 2024-09-16T03:50:12.243 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.238+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2175821665}) 2024-09-16T03:50:12.243 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.238+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:50:12.243 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.238+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3791041072}) 2024-09-16T03:50:12.243 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.238+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3364123839}) 2024-09-16T03:50:12.244 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.238+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:50:12.244 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.238+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3737895386}) 2024-09-16T03:50:12.244 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.238+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=309986750}) 2024-09-16T03:50:12.245 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.238+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:50:12.245 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.238+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3737895386}) 2024-09-16T03:50:12.245 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.238+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4272358343}) 2024-09-16T03:50:12.246 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.242+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:50:12.246 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.242+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3269475546}) 2024-09-16T03:50:12.246 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.242+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1880487750}) 2024-09-16T03:50:12.246 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.242+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:50:12.246 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.242+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3269475546}) 2024-09-16T03:50:12.247 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.242+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2121760743}) 2024-09-16T03:50:12.248 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.242+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:50:12.248 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.242+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1241524259}) 2024-09-16T03:50:12.248 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.242+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=618091563}) 2024-09-16T03:50:12.248 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.242+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:50:12.248 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.242+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1241524259}) 2024-09-16T03:50:12.248 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.242+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1682917584}) 2024-09-16T03:50:12.249 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.246+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:50:12.250 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.246+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1027886167}) 2024-09-16T03:50:12.250 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.246+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3797591511}) 2024-09-16T03:50:12.250 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.246+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:50:12.250 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.246+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1027886167}) 2024-09-16T03:50:12.250 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.246+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3552507492}) 2024-09-16T03:50:12.251 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.246+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:50:12.251 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.246+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=799373464}) 2024-09-16T03:50:12.251 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.246+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=374829510}) 2024-09-16T03:50:12.251 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.246+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:50:12.252 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.246+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=799373464}) 2024-09-16T03:50:12.252 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.246+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3426126505}) 2024-09-16T03:50:12.253 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.246+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:50:12.253 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.246+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2909469178}) 2024-09-16T03:50:12.253 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.246+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1247616482}) 2024-09-16T03:50:12.253 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.246+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:50:12.253 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.250+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2909469178}) 2024-09-16T03:50:12.253 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.250+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3791749586}) 2024-09-16T03:50:12.255 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.250+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:50:12.255 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.250+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3125076190}) 2024-09-16T03:50:12.255 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.250+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2262611911}) 2024-09-16T03:50:12.255 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.250+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:50:12.255 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.250+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3125076190}) 2024-09-16T03:50:12.255 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.250+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=848330090}) 2024-09-16T03:50:12.256 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.250+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:50:12.256 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.250+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1763194745}) 2024-09-16T03:50:12.256 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.250+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2032608090}) 2024-09-16T03:50:12.256 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.250+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:50:12.256 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.250+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1763194745}) 2024-09-16T03:50:12.257 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.250+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=826315239}) 2024-09-16T03:50:12.257 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.254+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:50:12.257 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.254+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2028546233}) 2024-09-16T03:50:12.258 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.254+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2854068886}) 2024-09-16T03:50:12.258 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.254+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:50:12.258 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.254+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2028546233}) 2024-09-16T03:50:12.258 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.254+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1200193043}) 2024-09-16T03:50:12.259 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.254+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:50:12.259 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.254+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3638276933}) 2024-09-16T03:50:12.259 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.254+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2563019753}) 2024-09-16T03:50:12.259 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.254+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:50:12.259 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.254+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3638276933}) 2024-09-16T03:50:12.259 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.254+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2856835450}) 2024-09-16T03:50:12.261 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.254+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:50:12.261 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.254+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=570179947}) 2024-09-16T03:50:12.261 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.254+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=997228953}) 2024-09-16T03:50:12.261 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.254+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:50:12.261 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.254+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=570179947}) 2024-09-16T03:50:12.261 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.254+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2684000087}) 2024-09-16T03:50:12.262 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.258+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:50:12.262 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.258+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1153422199}) 2024-09-16T03:50:12.262 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.258+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2696256893}) 2024-09-16T03:50:12.262 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.258+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:50:12.262 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.258+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1153422199}) 2024-09-16T03:50:12.263 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.258+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3324710654}) 2024-09-16T03:50:12.264 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.258+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:50:12.264 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.258+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3505263077}) 2024-09-16T03:50:12.264 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.258+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=105188083}) 2024-09-16T03:50:12.264 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.258+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:50:12.264 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.258+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3505263077}) 2024-09-16T03:50:12.264 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.258+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3925661086}) 2024-09-16T03:50:12.265 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.262+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:50:12.265 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.262+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2709559678}) 2024-09-16T03:50:12.265 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.262+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1289435416}) 2024-09-16T03:50:12.265 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.262+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:50:12.265 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.262+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2709559678}) 2024-09-16T03:50:12.266 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.262+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=642505816}) 2024-09-16T03:50:12.267 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.262+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:50:12.267 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.262+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2056236064}) 2024-09-16T03:50:12.267 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.262+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1211178357}) 2024-09-16T03:50:12.267 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.262+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:50:12.267 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.262+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2056236064}) 2024-09-16T03:50:12.267 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.262+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3984728492}) 2024-09-16T03:50:12.268 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.262+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:50:12.269 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.262+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1467473266}) 2024-09-16T03:50:12.269 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.262+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2594021084}) 2024-09-16T03:50:12.269 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.262+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:50:12.269 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.262+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1467473266}) 2024-09-16T03:50:12.269 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.262+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3060809597}) 2024-09-16T03:50:12.270 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.266+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:50:12.270 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.266+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=593853618}) 2024-09-16T03:50:12.270 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.266+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=41434814}) 2024-09-16T03:50:12.270 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.266+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:50:12.270 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.266+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=593853618}) 2024-09-16T03:50:12.271 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.266+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3024361972}) 2024-09-16T03:50:12.271 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.266+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:50:12.271 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.266+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2787120442}) 2024-09-16T03:50:12.272 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.266+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2004410144}) 2024-09-16T03:50:12.272 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.266+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:50:12.272 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.266+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2787120442}) 2024-09-16T03:50:12.272 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.266+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3301015085}) 2024-09-16T03:50:12.273 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.270+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:50:12.273 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.270+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3178318647}) 2024-09-16T03:50:12.273 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.270+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2190001958}) 2024-09-16T03:50:12.273 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.270+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:50:12.273 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.270+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3178318647}) 2024-09-16T03:50:12.273 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.270+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1172260583}) 2024-09-16T03:50:12.275 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.270+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:50:12.275 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.270+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=360902659}) 2024-09-16T03:50:12.275 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.270+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3490041447}) 2024-09-16T03:50:12.275 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.270+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:50:12.275 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.270+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=360902659}) 2024-09-16T03:50:12.275 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.270+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3463212000}) 2024-09-16T03:50:12.276 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.270+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:50:12.276 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.270+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3235359147}) 2024-09-16T03:50:12.276 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.270+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1026379}) 2024-09-16T03:50:12.277 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.270+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:50:12.277 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.270+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3235359147}) 2024-09-16T03:50:12.277 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.270+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=281088713}) 2024-09-16T03:50:12.279 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.274+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:50:12.279 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.274+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=22,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=7,mon_config_key=10,monmap=2,osd_metadata=6,osd_pg_creating=1,osd_snap=27,osdmap=12} crc {logm=3839235145,mdsmap=4088524775,mgr=460109776,mgr_command_descs=2107673600,mgr_metadata=1367272202,mgrstat=3041297542,mon_config_key=1439144890,monmap=3140514009,osd_metadata=4182959540,osd_pg_creating=774665589,osd_snap=322996913,osdmap=1643359314}) 2024-09-16T03:50:12.279 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.274+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=23,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=10,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=28,osdmap=8} crc {logm=1971756865,mdsmap=4088524775,mgr=460109776,mgr_command_descs=2107673600,mgr_metadata=1367272202,mgrstat=1062430450,mon_config_key=1439144890,monmap=3937513981,osd_metadata=4182959540,osd_pg_creating=774665589,osd_snap=159644537,osdmap=3933853893}) 2024-09-16T03:50:12.279 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.274+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:50:12.279 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.274+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=22,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=7,mon_config_key=10,monmap=2,osd_metadata=6,osd_pg_creating=1,osd_snap=27,osdmap=12} crc {logm=3839235145,mdsmap=4088524775,mgr=460109776,mgr_command_descs=2107673600,mgr_metadata=1367272202,mgrstat=3041297542,mon_config_key=1439144890,monmap=3140514009,osd_metadata=4182959540,osd_pg_creating=774665589,osd_snap=322996913,osdmap=1643359314}) 2024-09-16T03:50:12.279 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.274+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=21,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=10,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=29,osdmap=9} crc {logm=296079009,mdsmap=4088524775,mgr=1483503728,mgr_command_descs=2107673600,mgr_metadata=1367272202,mgrstat=1981168935,mon_config_key=541432806,monmap=3937513981,osd_metadata=4182959540,osd_pg_creating=774665589,osd_snap=1972131160,osdmap=929079470}) 2024-09-16T03:50:12.281 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.278+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:50:12.282 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.278+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3952800091}) 2024-09-16T03:50:12.282 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.278+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3360436203}) 2024-09-16T03:50:12.282 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.278+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:50:12.282 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.278+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3952800091}) 2024-09-16T03:50:12.282 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.278+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=87898925}) 2024-09-16T03:50:12.285 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.278+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:50:12.285 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.278+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2761648554}) 2024-09-16T03:50:12.285 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.282+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3064137818}) 2024-09-16T03:50:12.285 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.282+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:50:12.285 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.282+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2761648554}) 2024-09-16T03:50:12.285 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.282+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3481615441}) 2024-09-16T03:50:12.292 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.286+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:50:12.292 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.286+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=4004785298}) 2024-09-16T03:50:12.292 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.286+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1275580374}) 2024-09-16T03:50:12.293 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.286+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:50:12.293 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.286+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=4004785298}) 2024-09-16T03:50:12.293 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.286+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2388094007}) 2024-09-16T03:50:12.306 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.302+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:50:12.306 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.302+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2760840338}) 2024-09-16T03:50:12.306 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.302+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2359726672}) 2024-09-16T03:50:12.306 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.302+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:50:12.306 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.302+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2760840338}) 2024-09-16T03:50:12.306 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:50:12.302+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=153902477}) 2024-09-16T03:50:13.225 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T03:50:13.225 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:50:13.366 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.362+0000 7fb2adceb640 10 monclient: get_monmap_and_config 2024-09-16T03:50:13.366 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.362+0000 7fb2adceb640 10 monclient: build_initial_monmap 2024-09-16T03:50:13.366 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.362+0000 7fb2adceb640 10 monclient: monmap: 2024-09-16T03:50:13.366 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:50:13.366 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:50:13.366 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:50:13.364897+0000 2024-09-16T03:50:13.366 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:50:13.364897+0000 2024-09-16T03:50:13.366 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:50:13.366 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:50:13.366 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:50:13.366 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:50:13.366 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:50:13.367 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:50:13.367 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:50:13.367 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:50:13.369 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.362+0000 7fb2adceb640 1 Processor -- start 2024-09-16T03:50:13.369 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.362+0000 7fb2adceb640 1 -- start start 2024-09-16T03:50:13.369 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.362+0000 7fb2adceb640 10 monclient: init 2024-09-16T03:50:13.369 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.362+0000 7fb2adceb640 10 monclient: _reopen_session rank -1 2024-09-16T03:50:13.369 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.362+0000 7fb2adceb640 10 monclient: _add_conns ranks=[2,0,4,1,3] 2024-09-16T03:50:13.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.362+0000 7fb2adceb640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb2a80a2340 0x7fb2a80a2720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:50:13.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.362+0000 7fb2adceb640 10 monclient(hunting): picked mon.noname-b con 0x7fb2a80a2340 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:50:13.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.362+0000 7fb2adceb640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb2a809fe60 addr v1:172.21.15.42:6789/0 2024-09-16T03:50:13.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.362+0000 7fb2adceb640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fb2a80a2c60 0x7fb2a8137830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:50:13.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.362+0000 7fb2adceb640 10 monclient(hunting): picked mon.noname-c con 0x7fb2a80a2c60 addr v2:172.21.15.154:3300/0 2024-09-16T03:50:13.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.362+0000 7fb2adceb640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb2a80a0d70 con 0x7fb2a809fe60 2024-09-16T03:50:13.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.362+0000 7fb2adceb640 10 monclient(hunting): start opening mon connection 2024-09-16T03:50:13.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.362+0000 7fb2adceb640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fb2a80726e0 con 0x7fb2a80a2c60 2024-09-16T03:50:13.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.362+0000 7fb2adceb640 10 monclient(hunting): start opening mon connection 2024-09-16T03:50:13.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.362+0000 7fb2adceb640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fb2a806cae0 con 0x7fb2a80a2340 2024-09-16T03:50:13.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.362+0000 7fb2adceb640 10 monclient(hunting): _renew_subs 2024-09-16T03:50:13.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.362+0000 7fb2adceb640 10 monclient(hunting): authenticate will time out at 1460.334471s 2024-09-16T03:50:13.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.362+0000 7fb2a6ffd640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fb2a809fe60 0x7fb2a8137120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:56414/0 (socket says 172.21.15.42:56414) 2024-09-16T03:50:13.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.362+0000 7fb2a6ffd640 1 -- 172.21.15.42:0/2070589265 learned_addr learned my addr 172.21.15.42:0/2070589265 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:50:13.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.362+0000 7fb2a77fe640 1 --2- 172.21.15.42:0/2070589265 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb2a80a2340 0x7fb2a80a2720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:50:13.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.362+0000 7fb2a77fe640 10 monclient(hunting): get_auth_request con 0x7fb2a80a2340 auth_method 0 2024-09-16T03:50:13.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.362+0000 7fb2a77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:50:13.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.362+0000 7fb2a77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:50:13.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.362+0000 7fb2a77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:50:13.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.362+0000 7fb2a7fff640 1 --2- 172.21.15.42:0/2070589265 >> v2:172.21.15.154:3300/0 conn(0x7fb2a80a2c60 0x7fb2a8137830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:50:13.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.362+0000 7fb2a7fff640 10 monclient(hunting): get_auth_request con 0x7fb2a80a2c60 auth_method 0 2024-09-16T03:50:13.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.362+0000 7fb2a7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:50:13.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.362+0000 7fb2a7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:50:13.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.362+0000 7fb2a7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:50:13.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.362+0000 7fb2a77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:50:13.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.362+0000 7fb2a77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:50:13.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.362+0000 7fb2a77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:50:13.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.362+0000 7fb2a67fc640 1 -- 172.21.15.42:0/2070589265 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4066415689 0 0) 0x7fb2a80a0d70 con 0x7fb2a809fe60 2024-09-16T03:50:13.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.362+0000 7fb2a67fc640 10 monclient(hunting): _negotiate 2024-09-16T03:50:13.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.362+0000 7fb2a67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:50:13.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.362+0000 7fb2a67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:50:13.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.362+0000 7fb2a67fc640 10 monclient(hunting): my global_id is 764799 2024-09-16T03:50:13.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.362+0000 7fb2a67fc640 1 -- 172.21.15.42:0/2070589265 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb290003530 con 0x7fb2a809fe60 2024-09-16T03:50:13.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.362+0000 7fb2a7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:50:13.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.362+0000 7fb2a7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:50:13.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.362+0000 7fb2a7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:50:13.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.362+0000 7fb2a77fe640 10 monclient(hunting): handle_auth_done global_id 804125 payload 274 2024-09-16T03:50:13.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.362+0000 7fb2a77fe640 1 -- 172.21.15.42:0/2070589265 >> v2:172.21.15.154:3300/0 conn(0x7fb2a80a2c60 msgr2=0x7fb2a8137830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:50:13.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.362+0000 7fb2a77fe640 1 --2- 172.21.15.42:0/2070589265 >> v2:172.21.15.154:3300/0 conn(0x7fb2a80a2c60 0x7fb2a8137830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:50:13.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.362+0000 7fb2a77fe640 1 -- 172.21.15.42:0/2070589265 >> v1:172.21.15.42:6789/0 conn(0x7fb2a809fe60 legacy=0x7fb2a8137120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:50:13.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.362+0000 7fb2a77fe640 10 monclient: _finish_hunting 0 2024-09-16T03:50:13.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.362+0000 7fb2a77fe640 1 monclient: found mon.noname-b 2024-09-16T03:50:13.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.362+0000 7fb2a77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:50:13.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.362+0000 7fb2a77fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T03:50:13.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.362+0000 7fb2a77fe640 1 -- 172.21.15.42:0/2070589265 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2a806c350 con 0x7fb2a80a2340 2024-09-16T03:50:13.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.366+0000 7fb2a77fe640 1 --2- 172.21.15.42:0/2070589265 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb2a80a2340 0x7fb2a80a2720 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fb2980097c0 tx=0x7fb29801c360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4ca392a35b5362f9 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:50:13.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.366+0000 7fb2a67fc640 1 -- 172.21.15.42:0/2070589265 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2980b2070 con 0x7fb2a80a2340 2024-09-16T03:50:13.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.366+0000 7fb2a67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:50:13.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.366+0000 7fb2a67fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:50:13.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.366+0000 7fb2a67fc640 10 monclient: dump: 2024-09-16T03:50:13.379 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:50:13.379 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:50:13.379 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:13.379 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:13.379 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:50:13.379 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:50:13.379 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:50:13.380 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:50:13.380 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:50:13.380 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:50:13.380 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.366+0000 7fb2a67fc640 10 monclient: _finish_auth 0 2024-09-16T03:50:13.380 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.366+0000 7fb2a67fc640 10 monclient: _check_auth_tickets 2024-09-16T03:50:13.380 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.366+0000 7fb2a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:50:13.380 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.366+0000 7fb2adceb640 5 monclient: authenticate success, global_id 804125 2024-09-16T03:50:13.380 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.366+0000 7fb2adceb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:50:13.380 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.366+0000 7fb2a67fc640 1 -- 172.21.15.42:0/2070589265 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb2980ad680 con 0x7fb2a80a2340 2024-09-16T03:50:13.380 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.366+0000 7fb2a67fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:50:13.380 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.366+0000 7fb2adceb640 10 monclient: get_monmap_and_config success 2024-09-16T03:50:13.380 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.366+0000 7fb2adceb640 4 set_mon_vals no callback set 2024-09-16T03:50:13.380 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.366+0000 7fb2a67fc640 1 -- 172.21.15.42:0/2070589265 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2980ada30 con 0x7fb2a80a2340 2024-09-16T03:50:13.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.366+0000 7fb2a67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:50:13.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.366+0000 7fb2a67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:50:13.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.366+0000 7fb2a67fc640 10 monclient: dump: 2024-09-16T03:50:13.381 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:50:13.381 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:50:13.381 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:13.381 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:13.381 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:50:13.381 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:50:13.381 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:50:13.381 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:50:13.381 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:50:13.381 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:50:13.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.366+0000 7fb2adceb640 10 monclient: shutdown 2024-09-16T03:50:13.382 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.366+0000 7fb2adceb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:50:13.382 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.366+0000 7fb2adceb640 1 -- 172.21.15.42:0/2070589265 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb2a80a2340 msgr2=0x7fb2a80a2720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:50:13.382 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.366+0000 7fb2adceb640 1 --2- 172.21.15.42:0/2070589265 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb2a80a2340 0x7fb2a80a2720 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fb2980097c0 tx=0x7fb29801c360 comp rx=0 tx=0).stop 2024-09-16T03:50:13.382 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.366+0000 7fb2adceb640 1 -- 172.21.15.42:0/2070589265 shutdown_connections 2024-09-16T03:50:13.382 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.366+0000 7fb2adceb640 1 --2- 172.21.15.42:0/2070589265 >> v2:172.21.15.154:3300/0 conn(0x7fb2a80a2c60 0x7fb2a8137830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:50:13.382 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.366+0000 7fb2adceb640 1 --2- 172.21.15.42:0/2070589265 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb2a80a2340 0x7fb2a80a2720 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:50:13.382 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.366+0000 7fb2adceb640 1 -- 172.21.15.42:0/2070589265 >> 172.21.15.42:0/2070589265 conn(0x7fb2a80a3430 msgr2=0x7fb2a80a3810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:50:13.382 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.366+0000 7fb2adceb640 1 -- 172.21.15.42:0/2070589265 shutdown_connections 2024-09-16T03:50:13.382 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.366+0000 7fb2adceb640 1 -- 172.21.15.42:0/2070589265 wait complete. 2024-09-16T03:50:13.382 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.366+0000 7fb2adceb640 10 monclient: build_initial_monmap 2024-09-16T03:50:13.382 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.366+0000 7fb2adceb640 10 monclient: monmap: 2024-09-16T03:50:13.382 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:50:13.382 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:50:13.383 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:50:13.383 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:50:13.383 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:50:13.383 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:50:13.383 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:50:13.383 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:50:13.383 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:50:13.383 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:50:13.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.366+0000 7fb2adceb640 1 librados: starting msgr at 2024-09-16T03:50:13.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.366+0000 7fb2adceb640 1 librados: starting objecter 2024-09-16T03:50:13.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.366+0000 7fb2adceb640 1 Processor -- start 2024-09-16T03:50:13.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.366+0000 7fb2adceb640 1 -- start start 2024-09-16T03:50:13.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.366+0000 7fb2adceb640 1 librados: setting wanted keys 2024-09-16T03:50:13.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.366+0000 7fb2adceb640 1 librados: calling monclient init 2024-09-16T03:50:13.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.366+0000 7fb2adceb640 10 monclient: init 2024-09-16T03:50:13.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.366+0000 7fb2adceb640 10 monclient: _reopen_session rank -1 2024-09-16T03:50:13.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.366+0000 7fb2adceb640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:50:13.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.366+0000 7fb2adceb640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb2a80a2340 0x7fb2a81cef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:50:13.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.366+0000 7fb2adceb640 10 monclient(hunting): picked mon.noname-a con 0x7fb2a80a2340 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:50:13.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.366+0000 7fb2adceb640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb2a80a2c60 0x7fb2a8136e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:50:13.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.366+0000 7fb2adceb640 10 monclient(hunting): picked mon.noname-c con 0x7fb2a80a2c60 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:50:13.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.366+0000 7fb2adceb640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb2a81373a0 0x7fb2a811d860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:50:13.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.366+0000 7fb2adceb640 10 monclient(hunting): picked mon.noname-b con 0x7fb2a81373a0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:50:13.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.366+0000 7fb2adceb640 10 monclient(hunting): start opening mon connection 2024-09-16T03:50:13.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.366+0000 7fb2adceb640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fb2a8137d70 con 0x7fb2a80a2340 2024-09-16T03:50:13.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.370+0000 7fb2adceb640 10 monclient(hunting): start opening mon connection 2024-09-16T03:50:13.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.370+0000 7fb2adceb640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fb2a8137ef0 con 0x7fb2a81373a0 2024-09-16T03:50:13.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.370+0000 7fb2adceb640 10 monclient(hunting): start opening mon connection 2024-09-16T03:50:13.385 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.370+0000 7fb2adceb640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fb2a8141db0 con 0x7fb2a80a2c60 2024-09-16T03:50:13.385 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.370+0000 7fb2adceb640 10 monclient(hunting): _renew_subs 2024-09-16T03:50:13.385 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.370+0000 7fb2adceb640 10 monclient(hunting): authenticate will time out at 1460.340024s 2024-09-16T03:50:13.385 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.370+0000 7fb2a77fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb2a80a2340 0x7fb2a81cef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:50:13.385 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.370+0000 7fb2a6ffd640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb2a80a2c60 0x7fb2a8136e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:50:13.385 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.370+0000 7fb2a77fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb2a80a2340 0x7fb2a81cef30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:55302/0 (socket says 172.21.15.42:55302) 2024-09-16T03:50:13.385 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.370+0000 7fb2a6ffd640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb2a80a2c60 0x7fb2a8136e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:44438/0 (socket says 172.21.15.42:44438) 2024-09-16T03:50:13.385 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.370+0000 7fb2a77fe640 1 -- 172.21.15.42:0/338977761 learned_addr learned my addr 172.21.15.42:0/338977761 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:50:13.385 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.370+0000 7fb2a77fe640 10 monclient(hunting): get_auth_request con 0x7fb2a80a2340 auth_method 0 2024-09-16T03:50:13.385 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.370+0000 7fb2a77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:50:13.385 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.370+0000 7fb2a77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:50:13.385 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.370+0000 7fb2a77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:50:13.385 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.370+0000 7fb2a6ffd640 10 monclient(hunting): get_auth_request con 0x7fb2a80a2c60 auth_method 0 2024-09-16T03:50:13.385 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.370+0000 7fb2a6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:50:13.386 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.370+0000 7fb2a6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:50:13.386 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.370+0000 7fb2a6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:50:13.386 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.370+0000 7fb2a6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:50:13.386 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.370+0000 7fb2a6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:50:13.386 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.370+0000 7fb2a6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:50:13.386 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.370+0000 7fb2a77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:50:13.386 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.370+0000 7fb2a77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:50:13.386 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.370+0000 7fb2a77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:50:13.386 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.370+0000 7fb2a7fff640 1 --2- 172.21.15.42:0/338977761 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb2a81373a0 0x7fb2a811d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:50:13.386 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.370+0000 7fb2a77fe640 10 monclient(hunting): handle_auth_done global_id 764802 payload 824 2024-09-16T03:50:13.386 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.370+0000 7fb2a77fe640 1 -- 172.21.15.42:0/338977761 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb2a80a2c60 msgr2=0x7fb2a8136e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:50:13.386 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.370+0000 7fb2a77fe640 1 --2- 172.21.15.42:0/338977761 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb2a80a2c60 0x7fb2a8136e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:50:13.386 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.370+0000 7fb2a77fe640 1 -- 172.21.15.42:0/338977761 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb2a81373a0 msgr2=0x7fb2a811d860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:50:13.387 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.370+0000 7fb2a77fe640 1 --2- 172.21.15.42:0/338977761 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb2a81373a0 0x7fb2a811d860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:50:13.387 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.370+0000 7fb2a77fe640 10 monclient: _finish_hunting 0 2024-09-16T03:50:13.387 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.370+0000 7fb2a77fe640 1 monclient: found mon.noname-a 2024-09-16T03:50:13.387 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.370+0000 7fb2a77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:50:13.387 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.370+0000 7fb2a77fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:50:13.387 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.370+0000 7fb2a77fe640 1 -- 172.21.15.42:0/338977761 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2a8121510 con 0x7fb2a80a2340 2024-09-16T03:50:13.387 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.370+0000 7fb2a7fff640 10 monclient: get_auth_request con 0x7fb2a81373a0 auth_method 0 2024-09-16T03:50:13.387 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.370+0000 7fb2a6ffd640 1 --2- 172.21.15.42:0/338977761 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb2a80a2c60 0x7fb2a8136e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:50:13.387 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.370+0000 7fb2a7fff640 1 --2- 172.21.15.42:0/338977761 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb2a81373a0 0x7fb2a811d860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T03:50:13.387 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.370+0000 7fb2a77fe640 1 --2- 172.21.15.42:0/338977761 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb2a80a2340 0x7fb2a81cef30 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fb2980097c0 tx=0x7fb298019100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:50:13.387 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.370+0000 7fb28ffff640 10 client.?.objecter ms_handle_connect 0x7fb2a80a2340 2024-09-16T03:50:13.387 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.370+0000 7fb28ffff640 10 client.?.objecter resend_mon_ops 2024-09-16T03:50:13.387 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.370+0000 7fb28ffff640 1 -- 172.21.15.42:0/338977761 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2980b2040 con 0x7fb2a80a2340 2024-09-16T03:50:13.388 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.370+0000 7fb28ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:50:13.388 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.370+0000 7fb28ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:50:13.388 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.370+0000 7fb28ffff640 10 monclient: dump: 2024-09-16T03:50:13.388 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:50:13.388 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:50:13.388 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:13.388 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:13.388 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:50:13.388 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:50:13.388 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:50:13.388 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:50:13.388 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:50:13.388 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:50:13.388 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.370+0000 7fb28ffff640 10 monclient: _finish_auth 0 2024-09-16T03:50:13.389 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.370+0000 7fb28ffff640 10 monclient: _check_auth_tickets 2024-09-16T03:50:13.389 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.370+0000 7fb28ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:50:13.389 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.370+0000 7fb2adceb640 5 monclient: authenticate success, global_id 764802 2024-09-16T03:50:13.389 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.370+0000 7fb28ffff640 1 -- 172.21.15.42:0/338977761 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb298004230 con 0x7fb2a80a2340 2024-09-16T03:50:13.389 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.370+0000 7fb2adceb640 10 monclient: _renew_subs 2024-09-16T03:50:13.389 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.370+0000 7fb2adceb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:50:13.389 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.370+0000 7fb2adceb640 1 -- 172.21.15.42:0/338977761 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2a81217f0 con 0x7fb2a80a2340 2024-09-16T03:50:13.389 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.370+0000 7fb28ffff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:50:13.389 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.370+0000 7fb28ffff640 1 -- 172.21.15.42:0/338977761 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2980ad830 con 0x7fb2a80a2340 2024-09-16T03:50:13.389 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.370+0000 7fb28ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:50:13.389 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.370+0000 7fb28ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:50:13.389 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.370+0000 7fb28ffff640 10 monclient: dump: 2024-09-16T03:50:13.389 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:50:13.390 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:50:13.390 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:13.390 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:13.390 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:50:13.390 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:50:13.390 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:50:13.390 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:50:13.390 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:50:13.390 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:50:13.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.370+0000 7fb2a5ffb640 4 set_mon_vals no callback set 2024-09-16T03:50:13.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.370+0000 7fb2adceb640 10 client.764802.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:50:13.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.370+0000 7fb2adceb640 10 monclient: _renew_subs 2024-09-16T03:50:13.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.370+0000 7fb2adceb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:50:13.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.370+0000 7fb2adceb640 1 -- 172.21.15.42:0/338977761 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2a806f630 con 0x7fb2a80a2340 2024-09-16T03:50:13.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.370+0000 7fb2adceb640 1 librados: init done 2024-09-16T03:50:13.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.370+0000 7fb2adceb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:50:13.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.370+0000 7fb2adceb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:50:13.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.370+0000 7fb2adceb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:50:13.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.370+0000 7fb28ffff640 1 -- 172.21.15.42:0/338977761 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fb2980ad9d0 con 0x7fb2a80a2340 2024-09-16T03:50:13.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.370+0000 7fb2adceb640 1 -- 172.21.15.42:0/338977761 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb2a80a2720 con 0x7fb2a80a2340 2024-09-16T03:50:13.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.370+0000 7fb28ffff640 1 --2- 172.21.15.42:0/338977761 >> v2:172.21.15.154:6812/254845342 conn(0x7fb294037810 0x7fb294039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:50:13.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.370+0000 7fb28ffff640 1 -- 172.21.15.42:0/338977761 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(281..281 src has 1..281) ==== 6024+0+0 (secure 0 0 0) 0x7fb2980e2e10 con 0x7fb2a80a2340 2024-09-16T03:50:13.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.370+0000 7fb28ffff640 10 client.764802.objecter ms_dispatch 0x7fb2a8000ba0 osd_map(281..281 src has 1..281) 2024-09-16T03:50:13.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.370+0000 7fb28ffff640 3 client.764802.objecter handle_osd_map got epochs [281,281] > 0 2024-09-16T03:50:13.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.370+0000 7fb28ffff640 3 client.764802.objecter handle_osd_map decoding full epoch 281 2024-09-16T03:50:13.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.370+0000 7fb28ffff640 20 client.764802.objecter dump_active .. 0 homeless 2024-09-16T03:50:13.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.370+0000 7fb2a6ffd640 1 --2- 172.21.15.42:0/338977761 >> v2:172.21.15.154:6812/254845342 conn(0x7fb294037810 0x7fb294039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:50:13.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.370+0000 7fb2a6ffd640 10 monclient: get_auth_request con 0x7fb294037810 auth_method 0 2024-09-16T03:50:13.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.370+0000 7fb2a6ffd640 1 --2- 172.21.15.42:0/338977761 >> v2:172.21.15.154:6812/254845342 conn(0x7fb294037810 0x7fb294039cd0 secure :-1 s=READY pgs=3538 cs=0 l=1 rev1=1 crypto rx=0x7fb2a8120e40 tx=0x7fb29c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:50:13.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.370+0000 7fb28ffff640 10 client.764802.objecter ms_handle_connect 0x7fb294037810 2024-09-16T03:50:13.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.374+0000 7fb28ffff640 1 -- 172.21.15.42:0/338977761 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7fb2980ba020 con 0x7fb2a80a2340 2024-09-16T03:50:13.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.374+0000 7fb28ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:50:13.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.374+0000 7fb28ffff640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:50:13.597 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.594+0000 7fb2adceb640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:50:13.597 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.594+0000 7fb2adceb640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:50:13.597 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.594+0000 7fb2adceb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:50:13.597 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.594+0000 7fb2adceb640 1 -- 172.21.15.42:0/338977761 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fb2a80739f0 con 0x7fb2a80a2340 2024-09-16T03:50:13.598 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.594+0000 7fb28ffff640 1 -- 172.21.15.42:0/338977761 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7fb298016070 con 0x7fb2a80a2340 2024-09-16T03:50:13.598 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.594+0000 7fb28ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:50:13.598 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.594+0000 7fb28ffff640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:50:13.598 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":72,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:50:13.600 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.594+0000 7fb2adceb640 10 librados: watch_flush enter 2024-09-16T03:50:13.600 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.594+0000 7fb2adceb640 10 librados: watch_flush exit 2024-09-16T03:50:13.600 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.594+0000 7fb2adceb640 20 client.764802.objecter shutdown clearing up homeless session... 2024-09-16T03:50:13.600 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.594+0000 7fb2adceb640 10 client.764802.objecter successfully canceled tick 2024-09-16T03:50:13.600 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.594+0000 7fb2adceb640 1 -- 172.21.15.42:0/338977761 >> v2:172.21.15.154:6812/254845342 conn(0x7fb294037810 msgr2=0x7fb294039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:50:13.600 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.594+0000 7fb2adceb640 1 --2- 172.21.15.42:0/338977761 >> v2:172.21.15.154:6812/254845342 conn(0x7fb294037810 0x7fb294039cd0 secure :-1 s=READY pgs=3538 cs=0 l=1 rev1=1 crypto rx=0x7fb2a8120e40 tx=0x7fb29c009040 comp rx=0 tx=0).stop 2024-09-16T03:50:13.600 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.594+0000 7fb2adceb640 10 monclient: shutdown 2024-09-16T03:50:13.601 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.594+0000 7fb2adceb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:50:13.601 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.594+0000 7fb2adceb640 1 -- 172.21.15.42:0/338977761 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb2a80a2340 msgr2=0x7fb2a81cef30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:50:13.601 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.594+0000 7fb2adceb640 1 --2- 172.21.15.42:0/338977761 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb2a80a2340 0x7fb2a81cef30 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fb2980097c0 tx=0x7fb298019100 comp rx=0 tx=0).stop 2024-09-16T03:50:13.601 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.594+0000 7fb2adceb640 1 -- 172.21.15.42:0/338977761 shutdown_connections 2024-09-16T03:50:13.601 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.594+0000 7fb2adceb640 1 --2- 172.21.15.42:0/338977761 >> v2:172.21.15.154:6812/254845342 conn(0x7fb294037810 0x7fb294039cd0 unknown :-1 s=CLOSED pgs=3538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:50:13.601 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.594+0000 7fb2adceb640 1 --2- 172.21.15.42:0/338977761 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb2a81373a0 0x7fb2a811d860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:50:13.601 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.594+0000 7fb2adceb640 1 --2- 172.21.15.42:0/338977761 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb2a80a2c60 0x7fb2a8136e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:50:13.601 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.594+0000 7fb2adceb640 1 --2- 172.21.15.42:0/338977761 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb2a80a2340 0x7fb2a81cef30 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:50:13.601 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.594+0000 7fb2adceb640 1 -- 172.21.15.42:0/338977761 >> 172.21.15.42:0/338977761 conn(0x7fb2a80a3430 msgr2=0x7fb2a8070a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:50:13.601 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.594+0000 7fb2adceb640 1 -- 172.21.15.42:0/338977761 shutdown_connections 2024-09-16T03:50:13.601 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.594+0000 7fb2adceb640 1 -- 172.21.15.42:0/338977761 wait complete. 2024-09-16T03:50:13.601 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.594+0000 7fb2adceb640 1 librados: shutdown 2024-09-16T03:50:13.615 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T03:50:13.615 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T03:50:13.615 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T03:50:13.615 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T03:50:13.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.726+0000 7fe9a68e4640 10 monclient: get_monmap_and_config 2024-09-16T03:50:13.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.726+0000 7fe9a68e4640 10 monclient: build_initial_monmap 2024-09-16T03:50:13.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.726+0000 7fe9a68e4640 10 monclient: monmap: 2024-09-16T03:50:13.730 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:50:13.730 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:50:13.730 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:50:13.729305+0000 2024-09-16T03:50:13.731 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:50:13.729305+0000 2024-09-16T03:50:13.731 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:50:13.731 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:50:13.731 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:50:13.731 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:50:13.731 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:50:13.731 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:50:13.731 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:50:13.731 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:50:13.733 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.726+0000 7fe9a68e4640 1 Processor -- start 2024-09-16T03:50:13.733 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.726+0000 7fe9a68e4640 1 -- start start 2024-09-16T03:50:13.733 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.726+0000 7fe9a68e4640 10 monclient: init 2024-09-16T03:50:13.733 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.726+0000 7fe9a68e4640 10 monclient: _reopen_session rank -1 2024-09-16T03:50:13.733 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.726+0000 7fe9a68e4640 10 monclient: _add_conns ranks=[1,0,2,4,3] 2024-09-16T03:50:13.733 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.726+0000 7fe9a68e4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe9a00a1330 addr v1:172.21.15.154:6789/0 2024-09-16T03:50:13.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.726+0000 7fe9a68e4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe9a006dac0 addr v1:172.21.15.42:6789/0 2024-09-16T03:50:13.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.726+0000 7fe9a68e4640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe9a00a1e20 0x7fe9a013da70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:50:13.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.726+0000 7fe9a68e4640 10 monclient(hunting): picked mon.noname-b con 0x7fe9a00a1e20 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:50:13.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.726+0000 7fe9a68e4640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe9a009c090 con 0x7fe9a006dac0 2024-09-16T03:50:13.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.726+0000 7fe9a68e4640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe9a009f610 con 0x7fe9a00a1330 2024-09-16T03:50:13.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.726+0000 7fe9a68e4640 10 monclient(hunting): start opening mon connection 2024-09-16T03:50:13.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.726+0000 7fe9a68e4640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fe9a009c890 con 0x7fe9a00a1e20 2024-09-16T03:50:13.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.726+0000 7fe9a68e4640 10 monclient(hunting): _renew_subs 2024-09-16T03:50:13.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.726+0000 7fe9a68e4640 10 monclient(hunting): authenticate will time out at 1460.698920s 2024-09-16T03:50:13.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.726+0000 7fe9a4e81640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe9a00a1e20 0x7fe9a013da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:50:13.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.726+0000 7fe9a4e81640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe9a00a1e20 0x7fe9a013da70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:44442/0 (socket says 172.21.15.42:44442) 2024-09-16T03:50:13.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.726+0000 7fe9a4e81640 1 -- 172.21.15.42:0/923641754 learned_addr learned my addr 172.21.15.42:0/923641754 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:50:13.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.726+0000 7fe9a4e81640 10 monclient(hunting): get_auth_request con 0x7fe9a00a1e20 auth_method 0 2024-09-16T03:50:13.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.726+0000 7fe9a4e81640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:50:13.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.726+0000 7fe9a4e81640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:50:13.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.726+0000 7fe9a4e81640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:50:13.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.726+0000 7fe9a4e81640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:50:13.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.726+0000 7fe9a4e81640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:50:13.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.726+0000 7fe9a4e81640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:50:13.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.726+0000 7fe9a4e81640 10 monclient(hunting): handle_auth_done global_id 804131 payload 274 2024-09-16T03:50:13.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.726+0000 7fe9a4e81640 1 -- 172.21.15.42:0/923641754 >> v1:172.21.15.154:6789/0 conn(0x7fe9a00a1330 legacy=0x7fe9a00a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:50:13.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.726+0000 7fe9a4e81640 1 -- 172.21.15.42:0/923641754 >> v1:172.21.15.42:6789/0 conn(0x7fe9a006dac0 legacy=0x7fe9a013a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:50:13.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.726+0000 7fe9a4e81640 10 monclient: _finish_hunting 0 2024-09-16T03:50:13.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.726+0000 7fe9a4e81640 1 monclient: found mon.noname-b 2024-09-16T03:50:13.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.726+0000 7fe9a4e81640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:50:13.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.726+0000 7fe9a4e81640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T03:50:13.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.726+0000 7fe9a4e81640 1 -- 172.21.15.42:0/923641754 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9a009f8f0 con 0x7fe9a00a1e20 2024-09-16T03:50:13.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.730+0000 7fe9a4e81640 1 --2- 172.21.15.42:0/923641754 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe9a00a1e20 0x7fe9a013da70 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fe9980098a0 tx=0x7fe99801c460 comp rx=0 tx=0).ready entity=mon.2 client_cookie=480f89c1c7fd26f2 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:50:13.742 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.730+0000 7fe99effd640 1 -- 172.21.15.42:0/923641754 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9980ab680 con 0x7fe9a00a1e20 2024-09-16T03:50:13.742 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.730+0000 7fe99effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:50:13.742 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.730+0000 7fe99effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:50:13.742 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.730+0000 7fe99effd640 10 monclient: dump: 2024-09-16T03:50:13.742 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:50:13.742 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:50:13.742 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:13.742 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:13.742 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:50:13.742 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:50:13.742 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:50:13.742 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:50:13.742 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:50:13.743 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:50:13.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.730+0000 7fe99effd640 10 monclient: _finish_auth 0 2024-09-16T03:50:13.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.730+0000 7fe99effd640 10 monclient: _check_auth_tickets 2024-09-16T03:50:13.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.730+0000 7fe99effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:50:13.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.730+0000 7fe99effd640 1 -- 172.21.15.42:0/923641754 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe9980b0070 con 0x7fe9a00a1e20 2024-09-16T03:50:13.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.730+0000 7fe99effd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:50:13.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.730+0000 7fe99effd640 1 -- 172.21.15.42:0/923641754 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9980aba40 con 0x7fe9a00a1e20 2024-09-16T03:50:13.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.730+0000 7fe9a68e4640 5 monclient: authenticate success, global_id 804131 2024-09-16T03:50:13.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.730+0000 7fe9a68e4640 10 monclient: get_monmap_and_config success 2024-09-16T03:50:13.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.730+0000 7fe9a68e4640 4 set_mon_vals no callback set 2024-09-16T03:50:13.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.730+0000 7fe99effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:50:13.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.730+0000 7fe99effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:50:13.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.730+0000 7fe99effd640 10 monclient: dump: 2024-09-16T03:50:13.743 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:50:13.744 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:50:13.744 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:13.744 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:13.744 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:50:13.744 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:50:13.744 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:50:13.744 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:50:13.744 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:50:13.744 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:50:13.744 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.730+0000 7fe9a68e4640 10 monclient: shutdown 2024-09-16T03:50:13.744 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.730+0000 7fe9a68e4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:50:13.744 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.730+0000 7fe9a68e4640 1 -- 172.21.15.42:0/923641754 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe9a00a1e20 msgr2=0x7fe9a013da70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:50:13.744 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.730+0000 7fe9a68e4640 1 --2- 172.21.15.42:0/923641754 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe9a00a1e20 0x7fe9a013da70 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fe9980098a0 tx=0x7fe99801c460 comp rx=0 tx=0).stop 2024-09-16T03:50:13.744 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.730+0000 7fe9a68e4640 1 -- 172.21.15.42:0/923641754 shutdown_connections 2024-09-16T03:50:13.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.730+0000 7fe9a68e4640 1 --2- 172.21.15.42:0/923641754 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe9a00a1e20 0x7fe9a013da70 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:50:13.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.730+0000 7fe9a68e4640 1 -- 172.21.15.42:0/923641754 >> 172.21.15.42:0/923641754 conn(0x7fe9a00a2420 msgr2=0x7fe9a00a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:50:13.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.730+0000 7fe9a68e4640 1 -- 172.21.15.42:0/923641754 shutdown_connections 2024-09-16T03:50:13.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.730+0000 7fe9a68e4640 1 -- 172.21.15.42:0/923641754 wait complete. 2024-09-16T03:50:13.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.730+0000 7fe9a68e4640 10 monclient: build_initial_monmap 2024-09-16T03:50:13.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.730+0000 7fe9a68e4640 10 monclient: monmap: 2024-09-16T03:50:13.745 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:50:13.745 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:50:13.745 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:50:13.745 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:50:13.745 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:50:13.745 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:50:13.745 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:50:13.746 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:50:13.746 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:50:13.746 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:50:13.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.730+0000 7fe9a68e4640 1 librados: starting msgr at 2024-09-16T03:50:13.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.730+0000 7fe9a68e4640 1 librados: starting objecter 2024-09-16T03:50:13.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.730+0000 7fe9a68e4640 1 Processor -- start 2024-09-16T03:50:13.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.730+0000 7fe9a68e4640 1 -- start start 2024-09-16T03:50:13.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.730+0000 7fe9a68e4640 1 librados: setting wanted keys 2024-09-16T03:50:13.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.730+0000 7fe9a68e4640 1 librados: calling monclient init 2024-09-16T03:50:13.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.730+0000 7fe9a68e4640 10 monclient: init 2024-09-16T03:50:13.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.734+0000 7fe9a68e4640 10 monclient: _reopen_session rank -1 2024-09-16T03:50:13.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.734+0000 7fe9a68e4640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:50:13.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.734+0000 7fe9a68e4640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe9a00a1330 0x7fe9a011bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:50:13.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.734+0000 7fe9a68e4640 10 monclient(hunting): picked mon.noname-a con 0x7fe9a00a1330 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:50:13.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.734+0000 7fe9a68e4640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe9a00a1e20 0x7fe9a011c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:50:13.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.734+0000 7fe9a68e4640 10 monclient(hunting): picked mon.noname-c con 0x7fe9a00a1e20 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:50:13.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.734+0000 7fe9a68e4640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe9a011d3c0 0x7fe9a011c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:50:13.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.734+0000 7fe9a68e4640 10 monclient(hunting): picked mon.noname-b con 0x7fe9a011d3c0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:50:13.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.734+0000 7fe9a68e4640 10 monclient(hunting): start opening mon connection 2024-09-16T03:50:13.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.734+0000 7fe9a68e4640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fe9a0140950 con 0x7fe9a00a1330 2024-09-16T03:50:13.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.734+0000 7fe9a68e4640 10 monclient(hunting): start opening mon connection 2024-09-16T03:50:13.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.734+0000 7fe9a68e4640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe9a01407d0 con 0x7fe9a011d3c0 2024-09-16T03:50:13.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.734+0000 7fe9a68e4640 10 monclient(hunting): start opening mon connection 2024-09-16T03:50:13.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.734+0000 7fe9a68e4640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fe9a0140ad0 con 0x7fe9a00a1e20 2024-09-16T03:50:13.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.734+0000 7fe9a68e4640 10 monclient(hunting): _renew_subs 2024-09-16T03:50:13.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.734+0000 7fe9a68e4640 10 monclient(hunting): authenticate will time out at 1460.704552s 2024-09-16T03:50:13.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.734+0000 7fe99f7fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe9a00a1e20 0x7fe9a011c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:50:13.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.734+0000 7fe99ffff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe9a00a1330 0x7fe9a011bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:50:13.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.734+0000 7fe99f7fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe9a00a1e20 0x7fe9a011c4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:44454/0 (socket says 172.21.15.42:44454) 2024-09-16T03:50:13.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.734+0000 7fe99f7fe640 1 -- 172.21.15.42:0/1315859822 learned_addr learned my addr 172.21.15.42:0/1315859822 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:50:13.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.734+0000 7fe99f7fe640 10 monclient(hunting): get_auth_request con 0x7fe9a00a1e20 auth_method 0 2024-09-16T03:50:13.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.734+0000 7fe99f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:50:13.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.734+0000 7fe99f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:50:13.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.734+0000 7fe99f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:50:13.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.734+0000 7fe99ffff640 10 monclient(hunting): get_auth_request con 0x7fe9a00a1330 auth_method 0 2024-09-16T03:50:13.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.734+0000 7fe99ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:50:13.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.734+0000 7fe99ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:50:13.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.734+0000 7fe99ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:50:13.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.734+0000 7fe99f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:50:13.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.734+0000 7fe99f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:50:13.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.734+0000 7fe99f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:50:13.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.734+0000 7fe99ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:50:13.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.734+0000 7fe99ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:50:13.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.734+0000 7fe99ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:50:13.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.734+0000 7fe9a4e81640 1 --2- 172.21.15.42:0/1315859822 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe9a011d3c0 0x7fe9a011c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:50:13.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.734+0000 7fe9a4e81640 10 monclient(hunting): get_auth_request con 0x7fe9a011d3c0 auth_method 0 2024-09-16T03:50:13.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.734+0000 7fe9a4e81640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:50:13.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.734+0000 7fe9a4e81640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:50:13.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.734+0000 7fe9a4e81640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:50:13.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.734+0000 7fe99ffff640 10 monclient(hunting): handle_auth_done global_id 764805 payload 824 2024-09-16T03:50:13.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.734+0000 7fe99ffff640 1 -- 172.21.15.42:0/1315859822 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe9a00a1e20 msgr2=0x7fe9a011c4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:50:13.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.734+0000 7fe99ffff640 1 --2- 172.21.15.42:0/1315859822 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe9a00a1e20 0x7fe9a011c4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:50:13.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.734+0000 7fe99ffff640 1 -- 172.21.15.42:0/1315859822 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe9a011d3c0 msgr2=0x7fe9a011c9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:50:13.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.734+0000 7fe99ffff640 1 --2- 172.21.15.42:0/1315859822 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe9a011d3c0 0x7fe9a011c9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:50:13.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.734+0000 7fe99ffff640 10 monclient: _finish_hunting 0 2024-09-16T03:50:13.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.734+0000 7fe99ffff640 1 monclient: found mon.noname-a 2024-09-16T03:50:13.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.734+0000 7fe99ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:50:13.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.734+0000 7fe99ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:50:13.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.734+0000 7fe99ffff640 1 -- 172.21.15.42:0/1315859822 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9a011e590 con 0x7fe9a00a1330 2024-09-16T03:50:13.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.734+0000 7fe99f7fe640 1 --2- 172.21.15.42:0/1315859822 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe9a00a1e20 0x7fe9a011c4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:50:13.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.734+0000 7fe99ffff640 1 --2- 172.21.15.42:0/1315859822 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe9a00a1330 0x7fe9a011bf70 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fe98800c410 tx=0x7fe98800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:50:13.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.734+0000 7fe99cff9640 10 client.?.objecter ms_handle_connect 0x7fe9a00a1330 2024-09-16T03:50:13.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.734+0000 7fe99cff9640 10 client.?.objecter resend_mon_ops 2024-09-16T03:50:13.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.734+0000 7fe99cff9640 1 -- 172.21.15.42:0/1315859822 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe988017070 con 0x7fe9a00a1330 2024-09-16T03:50:13.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.734+0000 7fe99cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:50:13.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.734+0000 7fe99cff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:50:13.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.734+0000 7fe99cff9640 10 monclient: dump: 2024-09-16T03:50:13.750 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:50:13.750 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:50:13.751 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:13.751 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:13.751 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:50:13.751 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:50:13.751 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:50:13.751 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:50:13.751 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:50:13.751 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:50:13.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.734+0000 7fe99cff9640 10 monclient: _finish_auth 0 2024-09-16T03:50:13.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.734+0000 7fe99cff9640 10 monclient: _check_auth_tickets 2024-09-16T03:50:13.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.734+0000 7fe99cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:50:13.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.734+0000 7fe99cff9640 1 -- 172.21.15.42:0/1315859822 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe9880069e0 con 0x7fe9a00a1330 2024-09-16T03:50:13.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.734+0000 7fe99cff9640 10 monclient: handle_config config(0 keys) 2024-09-16T03:50:13.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.734+0000 7fe99cff9640 1 -- 172.21.15.42:0/1315859822 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe988006e10 con 0x7fe9a00a1330 2024-09-16T03:50:13.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.734+0000 7fe9a68e4640 5 monclient: authenticate success, global_id 764805 2024-09-16T03:50:13.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.734+0000 7fe9a68e4640 10 monclient: _renew_subs 2024-09-16T03:50:13.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.734+0000 7fe9a68e4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:50:13.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.734+0000 7fe9a68e4640 1 -- 172.21.15.42:0/1315859822 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9a011e870 con 0x7fe9a00a1330 2024-09-16T03:50:13.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.734+0000 7fe99e7fc640 4 set_mon_vals no callback set 2024-09-16T03:50:13.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.734+0000 7fe99cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:50:13.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.734+0000 7fe99cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:50:13.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.734+0000 7fe99cff9640 10 monclient: dump: 2024-09-16T03:50:13.752 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:50:13.752 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:50:13.752 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:13.752 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:13.752 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:50:13.753 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:50:13.753 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:50:13.753 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:50:13.753 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:50:13.753 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:50:13.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.734+0000 7fe9a68e4640 10 client.764805.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:50:13.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.734+0000 7fe9a68e4640 10 monclient: _renew_subs 2024-09-16T03:50:13.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.734+0000 7fe9a68e4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:50:13.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.734+0000 7fe9a68e4640 1 -- 172.21.15.42:0/1315859822 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9a013cd20 con 0x7fe9a00a1330 2024-09-16T03:50:13.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.734+0000 7fe9a68e4640 1 librados: init done 2024-09-16T03:50:13.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.734+0000 7fe9a68e4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:50:13.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.734+0000 7fe9a68e4640 1 --2- 172.21.15.42:0/1315859822 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe9a006dac0 0x7fe9a0135220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:50:13.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.734+0000 7fe9a68e4640 10 monclient(hunting): start opening mon connection 2024-09-16T03:50:13.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.734+0000 7fe9a68e4640 1 -- 172.21.15.42:0/1315859822 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fe9a00a1710 con 0x7fe9a006dac0 2024-09-16T03:50:13.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.734+0000 7fe99f7fe640 1 --2- 172.21.15.42:0/1315859822 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe9a006dac0 0x7fe9a0135220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:50:13.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.734+0000 7fe99f7fe640 10 monclient: get_auth_request con 0x7fe9a006dac0 auth_method 0 2024-09-16T03:50:13.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.734+0000 7fe99f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:50:13.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.734+0000 7fe99f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:50:13.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.734+0000 7fe99f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:50:13.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.734+0000 7fe99cff9640 1 -- 172.21.15.42:0/1315859822 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fe98801a070 con 0x7fe9a00a1330 2024-09-16T03:50:13.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.734+0000 7fe99f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:50:13.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.734+0000 7fe99f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:50:13.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.734+0000 7fe99f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:50:13.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.734+0000 7fe99f7fe640 10 monclient(hunting): handle_auth_done global_id 764808 payload 824 2024-09-16T03:50:13.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.734+0000 7fe99f7fe640 1 -- 172.21.15.42:0/1315859822 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe9a00008e0 con 0x7fe9a006dac0 2024-09-16T03:50:13.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.734+0000 7fe99cff9640 1 --2- 172.21.15.42:0/1315859822 >> v2:172.21.15.154:6812/254845342 conn(0x7fe980037810 0x7fe980039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:50:13.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.734+0000 7fe99cff9640 1 -- 172.21.15.42:0/1315859822 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(281..281 src has 1..281) ==== 6024+0+0 (secure 0 0 0) 0x7fe9880483e0 con 0x7fe9a00a1330 2024-09-16T03:50:13.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.734+0000 7fe99cff9640 10 client.764805.objecter ms_dispatch 0x7fe9a0000ba0 osd_map(281..281 src has 1..281) 2024-09-16T03:50:13.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.734+0000 7fe99cff9640 3 client.764805.objecter handle_osd_map got epochs [281,281] > 0 2024-09-16T03:50:13.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.734+0000 7fe99cff9640 3 client.764805.objecter handle_osd_map decoding full epoch 281 2024-09-16T03:50:13.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.734+0000 7fe99f7fe640 1 --2- 172.21.15.42:0/1315859822 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe9a006dac0 0x7fe9a0135220 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fe9a013c920 tx=0x7fe99000d520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:50:13.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.734+0000 7fe99cff9640 20 client.764805.objecter dump_active .. 0 homeless 2024-09-16T03:50:13.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.734+0000 7fe99cff9640 10 client.764805.objecter ms_handle_connect 0x7fe9a006dac0 2024-09-16T03:50:13.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.734+0000 7fe99cff9640 10 client.764805.objecter resend_mon_ops 2024-09-16T03:50:13.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.734+0000 7fe99cff9640 1 -- 172.21.15.42:0/1315859822 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe990006b60 con 0x7fe9a006dac0 2024-09-16T03:50:13.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.734+0000 7fe99cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:50:13.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.734+0000 7fe99cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:50:13.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.734+0000 7fe99cff9640 10 monclient: dump: 2024-09-16T03:50:13.756 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:50:13.756 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:50:13.756 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:13.756 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:13.756 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:50:13.756 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:50:13.756 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:50:13.756 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:50:13.756 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:50:13.757 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:50:13.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.734+0000 7fe9a4e81640 1 --2- 172.21.15.42:0/1315859822 >> v2:172.21.15.154:6812/254845342 conn(0x7fe980037810 0x7fe980039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:50:13.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.734+0000 7fe9a4e81640 10 monclient: get_auth_request con 0x7fe980037810 auth_method 0 2024-09-16T03:50:13.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.734+0000 7fe99cff9640 1 -- 172.21.15.42:0/1315859822 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fe990006b60 con 0x7fe9a006dac0 2024-09-16T03:50:13.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.734+0000 7fe99cff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:50:13.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.734+0000 7fe99cff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:50:13.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.734+0000 7fe99cff9640 1 -- 172.21.15.42:0/1315859822 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe9a006dac0 msgr2=0x7fe9a0135220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:50:13.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.734+0000 7fe99cff9640 1 --2- 172.21.15.42:0/1315859822 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe9a006dac0 0x7fe9a0135220 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fe9a013c920 tx=0x7fe99000d520 comp rx=0 tx=0).stop 2024-09-16T03:50:13.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.734+0000 7fe99cff9640 1 -- 172.21.15.42:0/1315859822 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe9a006dac0 msgr2=0x7fe9a0135220 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:50:13.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.734+0000 7fe99cff9640 1 --2- 172.21.15.42:0/1315859822 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe9a006dac0 0x7fe9a0135220 secure :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fe9a013c920 tx=0x7fe99000d520 comp rx=0 tx=0).stop 2024-09-16T03:50:13.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.734+0000 7fe99cff9640 10 client.764805.objecter ms_handle_connect 0x7fe980037810 2024-09-16T03:50:13.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.734+0000 7fe9a4e81640 1 --2- 172.21.15.42:0/1315859822 >> v2:172.21.15.154:6812/254845342 conn(0x7fe980037810 0x7fe980039cd0 secure :-1 s=READY pgs=3539 cs=0 l=1 rev1=1 crypto rx=0x7fe9980af000 tx=0x7fe9980a7000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:50:13.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.742+0000 7fe9a68e4640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:50:13.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.742+0000 7fe9a68e4640 1 --2- 172.21.15.42:0/1315859822 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe9a002c650 0x7fe9a002ca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:50:13.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.742+0000 7fe9a68e4640 10 monclient(hunting): start opening mon connection 2024-09-16T03:50:13.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.742+0000 7fe9a68e4640 1 -- 172.21.15.42:0/1315859822 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fe9a0138a40 con 0x7fe9a002c650 2024-09-16T03:50:13.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.742+0000 7fe99f7fe640 1 --2- 172.21.15.42:0/1315859822 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe9a002c650 0x7fe9a002ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:50:13.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.742+0000 7fe99f7fe640 10 monclient: get_auth_request con 0x7fe9a002c650 auth_method 0 2024-09-16T03:50:13.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.742+0000 7fe99f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:50:13.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.742+0000 7fe99f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:50:13.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.742+0000 7fe99f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:50:13.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.742+0000 7fe99f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:50:13.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.742+0000 7fe99f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:50:13.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.742+0000 7fe99f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:50:13.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.742+0000 7fe99f7fe640 10 monclient(hunting): handle_auth_done global_id 764811 payload 824 2024-09-16T03:50:13.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.742+0000 7fe99f7fe640 1 -- 172.21.15.42:0/1315859822 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fe9a0135760 con 0x7fe9a002c650 2024-09-16T03:50:13.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.742+0000 7fe99cff9640 10 client.764805.objecter ms_handle_connect 0x7fe9a002c650 2024-09-16T03:50:13.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.742+0000 7fe99f7fe640 1 --2- 172.21.15.42:0/1315859822 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe9a002c650 0x7fe9a002ca60 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fe99000d560 tx=0x7fe990001260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:50:13.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.742+0000 7fe99cff9640 10 client.764805.objecter resend_mon_ops 2024-09-16T03:50:13.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.742+0000 7fe99cff9640 1 -- 172.21.15.42:0/1315859822 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe99001dce0 con 0x7fe9a002c650 2024-09-16T03:50:13.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.742+0000 7fe99cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:50:13.761 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.742+0000 7fe99cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:50:13.761 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.742+0000 7fe99cff9640 10 monclient: dump: 2024-09-16T03:50:13.761 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:50:13.761 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:50:13.761 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:13.761 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:13.761 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:50:13.761 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:50:13.761 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:50:13.761 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:50:13.761 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:50:13.761 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:50:13.761 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.742+0000 7fe99cff9640 1 -- 172.21.15.42:0/1315859822 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4881 (secure 0 0 0) 0x7fe9900050d0 con 0x7fe9a002c650 2024-09-16T03:50:13.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.742+0000 7fe99cff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:50:13.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.742+0000 7fe99cff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:50:13.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.742+0000 7fe99cff9640 1 -- 172.21.15.42:0/1315859822 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe9a002c650 msgr2=0x7fe9a002ca60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:50:13.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.742+0000 7fe99cff9640 1 --2- 172.21.15.42:0/1315859822 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe9a002c650 0x7fe9a002ca60 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fe99000d560 tx=0x7fe990001260 comp rx=0 tx=0).stop 2024-09-16T03:50:13.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.742+0000 7fe99cff9640 1 -- 172.21.15.42:0/1315859822 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe9a002c650 msgr2=0x7fe9a002ca60 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:50:13.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.742+0000 7fe99cff9640 1 --2- 172.21.15.42:0/1315859822 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe9a002c650 0x7fe9a002ca60 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:50:13.762 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:50:13.762 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:50:13.762 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:50:13.762 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "leader", 2024-09-16T03:50:13.762 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 72, 2024-09-16T03:50:13.763 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 204534, 2024-09-16T03:50:13.763 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:50:13.763 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:50:13.763 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T03:50:13.763 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T03:50:13.763 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:50:13.763 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 6, 2024-09-16T03:50:13.763 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:50:13.763 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:50:13.763 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:50:13.763 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:50:13.763 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:50:13.763 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:50:13.764 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:50:13.764 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:50:13.764 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:50:13.764 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:50:13.764 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:50:13.764 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:50:13.764 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:50:13.764 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:50:13.764 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:50:13.764 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:50:13.764 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:50:13.764 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:50:13.764 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:50:13.765 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:50:13.765 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:50:13.765 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:50:13.765 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:50:13.765 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:50:13.765 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:50:13.765 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:50:13.765 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:50:13.765 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:50:13.765 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:50:13.765 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:50:13.765 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:50:13.765 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:50:13.766 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:50:13.766 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:50:13.766 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:50:13.766 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:50:13.766 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:50:13.766 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:50:13.766 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:50:13.766 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:50:13.766 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:50:13.766 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:50:13.766 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:50:13.766 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:50:13.766 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:50:13.767 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:50:13.767 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:50:13.767 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:50:13.767 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:50:13.767 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:50:13.767 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:50:13.767 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:50:13.767 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:50:13.767 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:50:13.767 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:50:13.767 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:50:13.767 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:50:13.767 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:50:13.767 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:50:13.768 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:50:13.768 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:50:13.768 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:50:13.768 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:50:13.768 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:50:13.768 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:50:13.768 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:50:13.768 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:50:13.768 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:50:13.768 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:50:13.768 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:50:13.768 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:50:13.768 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:50:13.769 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:50:13.769 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:50:13.769 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:50:13.769 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:50:13.769 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:50:13.769 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:50:13.769 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:50:13.769 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:50:13.769 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:50:13.769 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:50:13.769 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:50:13.769 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:50:13.769 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:50:13.770 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:50:13.770 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:50:13.770 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:50:13.770 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:50:13.770 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:50:13.770 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:50:13.770 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:50:13.770 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:50:13.770 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:50:13.770 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:50:13.770 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:50:13.770 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:50:13.770 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:50:13.770 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:50:13.771 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:50:13.771 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:50:13.771 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:50:13.771 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:50:13.771 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:50:13.771 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:50:13.771 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:50:13.771 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:50:13.771 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:50:13.771 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:50:13.771 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:50:13.771 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:50:13.771 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:50:13.772 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:50:13.772 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:50:13.772 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:50:13.772 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:50:13.772 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:50:13.772 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:50:13.772 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:50:13.772 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:50:13.772 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:50:13.772 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:50:13.772 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:50:13.772 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:50:13.772 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:50:13.772 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:50:13.773 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:50:13.773 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:50:13.773 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:50:13.773 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:50:13.773 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:50:13.773 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:50:13.773 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:50:13.773 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:50:13.773 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:50:13.773 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:50:13.773 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:50:13.776 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:50:13.776 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:50:13.776 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:50:13.776 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:50:13.776 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T03:50:13.776 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:50:13.777 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:50:13.777 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:50:13.777 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 6 2024-09-16T03:50:13.777 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:50:13.777 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:50:13.777 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:50:13.777 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:50:13.777 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:50:13.777 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:50:13.777 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 8 2024-09-16T03:50:13.777 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:50:13.777 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:50:13.777 INFO:teuthology.orchestra.run.smithi042.stdout: "mgr": [ 2024-09-16T03:50:13.777 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:50:13.778 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:50:13.778 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:50:13.778 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:50:13.778 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:50:13.778 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:50:13.778 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:50:13.778 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:50:13.778 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:50:13.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.742+0000 7fe9a68e4640 10 librados: watch_flush enter 2024-09-16T03:50:13.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.742+0000 7fe9a68e4640 10 librados: watch_flush exit 2024-09-16T03:50:13.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.742+0000 7fe9a68e4640 20 client.764805.objecter shutdown clearing up homeless session... 2024-09-16T03:50:13.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.742+0000 7fe9a68e4640 10 client.764805.objecter successfully canceled tick 2024-09-16T03:50:13.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.742+0000 7fe9a68e4640 1 -- 172.21.15.42:0/1315859822 >> v2:172.21.15.154:6812/254845342 conn(0x7fe980037810 msgr2=0x7fe980039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:50:13.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.742+0000 7fe9a68e4640 1 --2- 172.21.15.42:0/1315859822 >> v2:172.21.15.154:6812/254845342 conn(0x7fe980037810 0x7fe980039cd0 secure :-1 s=READY pgs=3539 cs=0 l=1 rev1=1 crypto rx=0x7fe9980af000 tx=0x7fe9980a7000 comp rx=0 tx=0).stop 2024-09-16T03:50:13.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.742+0000 7fe9a68e4640 10 monclient: shutdown 2024-09-16T03:50:13.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.742+0000 7fe9a68e4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:50:13.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.742+0000 7fe9a68e4640 1 -- 172.21.15.42:0/1315859822 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe9a00a1330 msgr2=0x7fe9a011bf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:50:13.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.742+0000 7fe9a68e4640 1 --2- 172.21.15.42:0/1315859822 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe9a00a1330 0x7fe9a011bf70 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fe98800c410 tx=0x7fe98800c8d0 comp rx=0 tx=0).stop 2024-09-16T03:50:13.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.742+0000 7fe9a4e81640 1 -- 172.21.15.42:0/1315859822 reap_dead start 2024-09-16T03:50:13.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.742+0000 7fe9a68e4640 1 -- 172.21.15.42:0/1315859822 shutdown_connections 2024-09-16T03:50:13.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.742+0000 7fe9a68e4640 1 --2- 172.21.15.42:0/1315859822 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe9a00a1330 0x7fe9a011bf70 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:50:13.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.742+0000 7fe9a68e4640 1 -- 172.21.15.42:0/1315859822 >> 172.21.15.42:0/1315859822 conn(0x7fe9a00a2420 msgr2=0x7fe9a00a4ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:50:13.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.742+0000 7fe9a68e4640 1 -- 172.21.15.42:0/1315859822 shutdown_connections 2024-09-16T03:50:13.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.742+0000 7fe9a68e4640 1 -- 172.21.15.42:0/1315859822 wait complete. 2024-09-16T03:50:13.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.742+0000 7fe9a68e4640 1 librados: shutdown 2024-09-16T03:50:13.781 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:50:13.781 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T03:50:13.782 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T03:50:13.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.902+0000 7fb568ada640 10 monclient: get_monmap_and_config 2024-09-16T03:50:13.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.902+0000 7fb568ada640 10 monclient: build_initial_monmap 2024-09-16T03:50:13.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.902+0000 7fb568ada640 10 monclient: monmap: 2024-09-16T03:50:13.909 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:50:13.909 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:50:13.909 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:50:13.908026+0000 2024-09-16T03:50:13.909 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:50:13.908026+0000 2024-09-16T03:50:13.909 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:50:13.909 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:50:13.909 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:50:13.910 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:50:13.910 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:50:13.910 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:50:13.910 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:50:13.910 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:50:13.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.906+0000 7fb568ada640 1 Processor -- start 2024-09-16T03:50:13.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.906+0000 7fb568ada640 1 -- start start 2024-09-16T03:50:13.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.906+0000 7fb568ada640 10 monclient: init 2024-09-16T03:50:13.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.906+0000 7fb568ada640 10 monclient: _reopen_session rank -1 2024-09-16T03:50:13.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.906+0000 7fb568ada640 10 monclient: _add_conns ranks=[2,0,1,3,4] 2024-09-16T03:50:13.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.906+0000 7fb568ada640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb5640a1330 0x7fb5640a1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:50:13.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.906+0000 7fb568ada640 10 monclient(hunting): picked mon.noname-b con 0x7fb5640a1330 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:50:13.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.906+0000 7fb568ada640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb56406dac0 addr v1:172.21.15.42:6789/0 2024-09-16T03:50:13.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.906+0000 7fb568ada640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb5640a1c50 addr v1:172.21.15.154:6789/0 2024-09-16T03:50:13.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.906+0000 7fb568ada640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb56409c7d0 con 0x7fb56406dac0 2024-09-16T03:50:13.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.906+0000 7fb568ada640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb56409c090 con 0x7fb5640a1c50 2024-09-16T03:50:13.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.906+0000 7fb568ada640 10 monclient(hunting): start opening mon connection 2024-09-16T03:50:13.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.906+0000 7fb568ada640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fb56409f610 con 0x7fb5640a1330 2024-09-16T03:50:13.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.906+0000 7fb568ada640 10 monclient(hunting): _renew_subs 2024-09-16T03:50:13.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.906+0000 7fb568ada640 10 monclient(hunting): authenticate will time out at 1460.877454s 2024-09-16T03:50:13.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.906+0000 7fb561d9b640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fb56406dac0 0x7fb564135100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:56428/0 (socket says 172.21.15.42:56428) 2024-09-16T03:50:13.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.906+0000 7fb561d9b640 1 -- 172.21.15.42:0/1536459665 learned_addr learned my addr 172.21.15.42:0/1536459665 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:50:13.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.906+0000 7fb56259c640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb5640a1330 0x7fb5640a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:50:13.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.906+0000 7fb56259c640 10 monclient(hunting): get_auth_request con 0x7fb5640a1330 auth_method 0 2024-09-16T03:50:13.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.906+0000 7fb56259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:50:13.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.906+0000 7fb56259c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:50:13.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.906+0000 7fb56259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:50:13.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.906+0000 7fb56259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:50:13.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.906+0000 7fb56259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:50:13.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.906+0000 7fb56259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:50:13.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.906+0000 7fb56259c640 10 monclient(hunting): handle_auth_done global_id 804137 payload 274 2024-09-16T03:50:13.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.906+0000 7fb56259c640 1 -- 172.21.15.42:0/1536459665 >> v1:172.21.15.154:6789/0 conn(0x7fb5640a1c50 legacy=0x7fb564135810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:50:13.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.906+0000 7fb56259c640 1 -- 172.21.15.42:0/1536459665 >> v1:172.21.15.42:6789/0 conn(0x7fb56406dac0 legacy=0x7fb564135100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:50:13.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.906+0000 7fb56159a640 1 -- 172.21.15.42:0/1536459665 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1056042086 0 0) 0x7fb56409c7d0 con 0x7fb56406dac0 2024-09-16T03:50:13.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.906+0000 7fb56259c640 10 monclient: _finish_hunting 0 2024-09-16T03:50:13.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.906+0000 7fb56259c640 1 monclient: found mon.noname-b 2024-09-16T03:50:13.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.906+0000 7fb56259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:50:13.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.906+0000 7fb56259c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T03:50:13.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.906+0000 7fb56259c640 1 -- 172.21.15.42:0/1536459665 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb56409f8f0 con 0x7fb5640a1330 2024-09-16T03:50:13.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.906+0000 7fb56159a640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T03:50:13.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.910+0000 7fb56259c640 1 --2- 172.21.15.42:0/1536459665 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb5640a1330 0x7fb5640a1710 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fb5540097f0 tx=0x7fb55401c3f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d455dec9d1d72eaf server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:50:13.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.910+0000 7fb56159a640 1 -- 172.21.15.42:0/1536459665 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5540b0070 con 0x7fb5640a1330 2024-09-16T03:50:13.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.910+0000 7fb56159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:50:13.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.910+0000 7fb56159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:50:13.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.910+0000 7fb56159a640 10 monclient: dump: 2024-09-16T03:50:13.921 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:50:13.921 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:50:13.921 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:13.921 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:13.921 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:50:13.921 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:50:13.921 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:50:13.921 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:50:13.921 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:50:13.921 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:50:13.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.910+0000 7fb56159a640 10 monclient: _finish_auth 0 2024-09-16T03:50:13.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.910+0000 7fb56159a640 10 monclient: _check_auth_tickets 2024-09-16T03:50:13.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.910+0000 7fb56159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:50:13.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.910+0000 7fb56159a640 1 -- 172.21.15.42:0/1536459665 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb5540ab600 con 0x7fb5640a1330 2024-09-16T03:50:13.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.910+0000 7fb56159a640 10 monclient: handle_config config(0 keys) 2024-09-16T03:50:13.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.910+0000 7fb56159a640 1 -- 172.21.15.42:0/1536459665 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5540ab9d0 con 0x7fb5640a1330 2024-09-16T03:50:13.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.910+0000 7fb568ada640 5 monclient: authenticate success, global_id 804137 2024-09-16T03:50:13.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.910+0000 7fb568ada640 10 monclient: get_monmap_and_config success 2024-09-16T03:50:13.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.910+0000 7fb568ada640 4 set_mon_vals no callback set 2024-09-16T03:50:13.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.910+0000 7fb56159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:50:13.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.910+0000 7fb56159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:50:13.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.910+0000 7fb56159a640 10 monclient: dump: 2024-09-16T03:50:13.922 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:50:13.923 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:50:13.923 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:13.923 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:13.923 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:50:13.923 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:50:13.923 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:50:13.923 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:50:13.923 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:50:13.923 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:50:13.923 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.910+0000 7fb568ada640 10 monclient: shutdown 2024-09-16T03:50:13.923 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.910+0000 7fb568ada640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:50:13.923 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.910+0000 7fb568ada640 1 -- 172.21.15.42:0/1536459665 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb5640a1330 msgr2=0x7fb5640a1710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:50:13.923 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.910+0000 7fb568ada640 1 --2- 172.21.15.42:0/1536459665 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb5640a1330 0x7fb5640a1710 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fb5540097f0 tx=0x7fb55401c3f0 comp rx=0 tx=0).stop 2024-09-16T03:50:13.923 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.910+0000 7fb568ada640 1 -- 172.21.15.42:0/1536459665 shutdown_connections 2024-09-16T03:50:13.924 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.910+0000 7fb568ada640 1 --2- 172.21.15.42:0/1536459665 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb5640a1330 0x7fb5640a1710 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:50:13.924 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.910+0000 7fb568ada640 1 -- 172.21.15.42:0/1536459665 >> 172.21.15.42:0/1536459665 conn(0x7fb5640a2420 msgr2=0x7fb5640a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:50:13.924 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.910+0000 7fb568ada640 1 -- 172.21.15.42:0/1536459665 shutdown_connections 2024-09-16T03:50:13.924 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.910+0000 7fb568ada640 1 -- 172.21.15.42:0/1536459665 wait complete. 2024-09-16T03:50:13.924 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.910+0000 7fb568ada640 10 monclient: build_initial_monmap 2024-09-16T03:50:13.924 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.910+0000 7fb568ada640 10 monclient: monmap: 2024-09-16T03:50:13.924 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:50:13.924 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:50:13.924 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:50:13.924 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:50:13.924 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:50:13.924 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:50:13.924 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:50:13.924 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:50:13.925 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:50:13.925 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:50:13.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.910+0000 7fb568ada640 1 librados: starting msgr at 2024-09-16T03:50:13.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.910+0000 7fb568ada640 1 librados: starting objecter 2024-09-16T03:50:13.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.910+0000 7fb568ada640 1 Processor -- start 2024-09-16T03:50:13.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.910+0000 7fb568ada640 1 -- start start 2024-09-16T03:50:13.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.910+0000 7fb568ada640 1 librados: setting wanted keys 2024-09-16T03:50:13.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.910+0000 7fb568ada640 1 librados: calling monclient init 2024-09-16T03:50:13.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.910+0000 7fb568ada640 10 monclient: init 2024-09-16T03:50:13.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.910+0000 7fb568ada640 10 monclient: _reopen_session rank -1 2024-09-16T03:50:13.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.910+0000 7fb568ada640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T03:50:13.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.910+0000 7fb568ada640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb5640a1330 0x7fb5641ced70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:50:13.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.910+0000 7fb568ada640 10 monclient(hunting): picked mon.noname-a con 0x7fb5640a1330 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:50:13.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.910+0000 7fb568ada640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb5640a1c50 0x7fb564135040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:50:13.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.910+0000 7fb568ada640 10 monclient(hunting): picked mon.noname-b con 0x7fb5640a1c50 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:50:13.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.910+0000 7fb568ada640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb56409dbb0 0x7fb56413cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:50:13.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.910+0000 7fb568ada640 10 monclient(hunting): picked mon.noname-c con 0x7fb56409dbb0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:50:13.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.910+0000 7fb568ada640 10 monclient(hunting): start opening mon connection 2024-09-16T03:50:13.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.910+0000 7fb568ada640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fb56409ef30 con 0x7fb5640a1330 2024-09-16T03:50:13.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.910+0000 7fb568ada640 10 monclient(hunting): start opening mon connection 2024-09-16T03:50:13.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.910+0000 7fb568ada640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fb56409edb0 con 0x7fb5640a1c50 2024-09-16T03:50:13.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.910+0000 7fb568ada640 10 monclient(hunting): start opening mon connection 2024-09-16T03:50:13.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.910+0000 7fb568ada640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fb564140b90 con 0x7fb56409dbb0 2024-09-16T03:50:13.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.910+0000 7fb568ada640 10 monclient(hunting): _renew_subs 2024-09-16T03:50:13.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.910+0000 7fb568ada640 10 monclient(hunting): authenticate will time out at 1460.882518s 2024-09-16T03:50:13.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.910+0000 7fb56259c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb5640a1330 0x7fb5641ced70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:50:13.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.910+0000 7fb562d9d640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb56409dbb0 0x7fb56413cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:50:13.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.910+0000 7fb562d9d640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb56409dbb0 0x7fb56413cbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:44486/0 (socket says 172.21.15.42:44486) 2024-09-16T03:50:13.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.910+0000 7fb562d9d640 1 -- 172.21.15.42:0/3604263304 learned_addr learned my addr 172.21.15.42:0/3604263304 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:50:13.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.910+0000 7fb562d9d640 10 monclient(hunting): get_auth_request con 0x7fb56409dbb0 auth_method 0 2024-09-16T03:50:13.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.910+0000 7fb562d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:50:13.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.910+0000 7fb562d9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:50:13.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.910+0000 7fb562d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:50:13.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.910+0000 7fb56259c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb5640a1330 0x7fb5641ced70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:55330/0 (socket says 172.21.15.42:55330) 2024-09-16T03:50:13.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.910+0000 7fb56259c640 10 monclient(hunting): get_auth_request con 0x7fb5640a1330 auth_method 0 2024-09-16T03:50:13.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.910+0000 7fb56259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:50:13.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.910+0000 7fb56259c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:50:13.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.910+0000 7fb56259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:50:13.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.910+0000 7fb562d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:50:13.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.910+0000 7fb562d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:50:13.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.910+0000 7fb562d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:50:13.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.910+0000 7fb56259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:50:13.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.910+0000 7fb56259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:50:13.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.910+0000 7fb56259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:50:13.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.910+0000 7fb561d9b640 1 --2- 172.21.15.42:0/3604263304 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb5640a1c50 0x7fb564135040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:50:13.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.910+0000 7fb561d9b640 10 monclient(hunting): get_auth_request con 0x7fb5640a1c50 auth_method 0 2024-09-16T03:50:13.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.910+0000 7fb561d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:50:13.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.910+0000 7fb561d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:50:13.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.910+0000 7fb561d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:50:13.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.910+0000 7fb562d9d640 10 monclient(hunting): handle_auth_done global_id 804140 payload 824 2024-09-16T03:50:13.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.910+0000 7fb562d9d640 1 -- 172.21.15.42:0/3604263304 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb5640a1c50 msgr2=0x7fb564135040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:50:13.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.910+0000 7fb562d9d640 1 --2- 172.21.15.42:0/3604263304 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb5640a1c50 0x7fb564135040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:50:13.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.910+0000 7fb562d9d640 1 -- 172.21.15.42:0/3604263304 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb5640a1330 msgr2=0x7fb5641ced70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:50:13.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.910+0000 7fb562d9d640 1 --2- 172.21.15.42:0/3604263304 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb5640a1330 0x7fb5641ced70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:50:13.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.910+0000 7fb562d9d640 10 monclient: _finish_hunting 0 2024-09-16T03:50:13.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.910+0000 7fb562d9d640 1 monclient: found mon.noname-c 2024-09-16T03:50:13.929 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.910+0000 7fb562d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:50:13.929 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.910+0000 7fb562d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:50:13.929 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.910+0000 7fb562d9d640 1 -- 172.21.15.42:0/3604263304 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5640710a0 con 0x7fb56409dbb0 2024-09-16T03:50:13.929 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.910+0000 7fb562d9d640 1 --2- 172.21.15.42:0/3604263304 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb56409dbb0 0x7fb56413cbc0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fb55c00c7b0 tx=0x7fb55c00cc70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:50:13.929 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.910+0000 7fb552ffd640 10 client.?.objecter ms_handle_connect 0x7fb56409dbb0 2024-09-16T03:50:13.929 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.910+0000 7fb552ffd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:50:13.929 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.910+0000 7fb552ffd640 1 -- 172.21.15.42:0/3604263304 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb55c004b70 con 0x7fb56409dbb0 2024-09-16T03:50:13.929 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.910+0000 7fb552ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:50:13.929 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.910+0000 7fb552ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:50:13.929 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.910+0000 7fb552ffd640 10 monclient: dump: 2024-09-16T03:50:13.929 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:50:13.929 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:50:13.929 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:13.930 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:13.930 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:50:13.930 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:50:13.930 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:50:13.930 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:50:13.930 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:50:13.930 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:50:13.930 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.910+0000 7fb552ffd640 10 monclient: _finish_auth 0 2024-09-16T03:50:13.930 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.910+0000 7fb552ffd640 10 monclient: _check_auth_tickets 2024-09-16T03:50:13.930 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.910+0000 7fb552ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:50:13.930 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.910+0000 7fb552ffd640 1 -- 172.21.15.42:0/3604263304 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb55c004d10 con 0x7fb56409dbb0 2024-09-16T03:50:13.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.910+0000 7fb552ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:50:13.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.910+0000 7fb552ffd640 1 -- 172.21.15.42:0/3604263304 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb55c00aa90 con 0x7fb56409dbb0 2024-09-16T03:50:13.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.910+0000 7fb552ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:50:13.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.910+0000 7fb552ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:50:13.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.910+0000 7fb552ffd640 10 monclient: dump: 2024-09-16T03:50:13.931 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:50:13.931 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:50:13.931 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:13.931 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:13.931 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:50:13.931 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:50:13.931 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:50:13.931 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:50:13.931 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:50:13.931 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:50:13.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.910+0000 7fb560d99640 4 set_mon_vals no callback set 2024-09-16T03:50:13.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.910+0000 7fb568ada640 5 monclient: authenticate success, global_id 804140 2024-09-16T03:50:13.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.910+0000 7fb568ada640 10 monclient: _renew_subs 2024-09-16T03:50:13.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.910+0000 7fb568ada640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:50:13.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.910+0000 7fb568ada640 1 -- 172.21.15.42:0/3604263304 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb56409eaa0 con 0x7fb56409dbb0 2024-09-16T03:50:13.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.910+0000 7fb568ada640 10 client.804140.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:50:13.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.910+0000 7fb568ada640 10 monclient: _renew_subs 2024-09-16T03:50:13.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.910+0000 7fb568ada640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:50:13.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.910+0000 7fb568ada640 1 -- 172.21.15.42:0/3604263304 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb564135750 con 0x7fb56409dbb0 2024-09-16T03:50:13.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.910+0000 7fb568ada640 1 librados: init done 2024-09-16T03:50:13.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.914+0000 7fb568ada640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:50:13.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.914+0000 7fb568ada640 1 --2- 172.21.15.42:0/3604263304 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb56406dac0 0x7fb56402c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:50:13.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.914+0000 7fb568ada640 10 monclient(hunting): start opening mon connection 2024-09-16T03:50:13.933 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.914+0000 7fb568ada640 1 -- 172.21.15.42:0/3604263304 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fb5641cfcb0 con 0x7fb56406dac0 2024-09-16T03:50:13.933 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.914+0000 7fb56259c640 1 --2- 172.21.15.42:0/3604263304 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb56406dac0 0x7fb56402c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:50:13.933 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.914+0000 7fb552ffd640 1 -- 172.21.15.42:0/3604263304 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fb55c00ac30 con 0x7fb56409dbb0 2024-09-16T03:50:13.933 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.914+0000 7fb56259c640 10 monclient: get_auth_request con 0x7fb56406dac0 auth_method 0 2024-09-16T03:50:13.933 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.914+0000 7fb56259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:50:13.933 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.914+0000 7fb56259c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:50:13.933 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.914+0000 7fb56259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:50:13.933 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.914+0000 7fb56259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:50:13.933 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.914+0000 7fb56259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:50:13.933 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.914+0000 7fb56259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:50:13.933 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.914+0000 7fb552ffd640 1 --2- 172.21.15.42:0/3604263304 >> v2:172.21.15.154:6812/254845342 conn(0x7fb544037810 0x7fb544039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:50:13.933 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.914+0000 7fb552ffd640 1 -- 172.21.15.42:0/3604263304 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(281..281 src has 1..281) ==== 6024+0+0 (secure 0 0 0) 0x7fb55c049290 con 0x7fb56409dbb0 2024-09-16T03:50:13.933 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.914+0000 7fb552ffd640 10 client.804140.objecter ms_dispatch 0x7fb564000ba0 osd_map(281..281 src has 1..281) 2024-09-16T03:50:13.933 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.914+0000 7fb552ffd640 3 client.804140.objecter handle_osd_map got epochs [281,281] > 0 2024-09-16T03:50:13.934 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.914+0000 7fb552ffd640 3 client.804140.objecter handle_osd_map decoding full epoch 281 2024-09-16T03:50:13.934 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.914+0000 7fb56259c640 10 monclient(hunting): handle_auth_done global_id 804143 payload 824 2024-09-16T03:50:13.934 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.914+0000 7fb552ffd640 20 client.804140.objecter dump_active .. 0 homeless 2024-09-16T03:50:13.934 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.914+0000 7fb56259c640 1 -- 172.21.15.42:0/3604263304 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb56406e530 con 0x7fb56406dac0 2024-09-16T03:50:13.934 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.914+0000 7fb561d9b640 1 --2- 172.21.15.42:0/3604263304 >> v2:172.21.15.154:6812/254845342 conn(0x7fb544037810 0x7fb544039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:50:13.934 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.914+0000 7fb552ffd640 10 client.804140.objecter ms_handle_connect 0x7fb56406dac0 2024-09-16T03:50:13.934 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.914+0000 7fb552ffd640 10 client.804140.objecter resend_mon_ops 2024-09-16T03:50:13.934 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.914+0000 7fb56259c640 1 --2- 172.21.15.42:0/3604263304 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb56406dac0 0x7fb56402c650 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fb56406c630 tx=0x7fb554019190 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:50:13.934 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.914+0000 7fb561d9b640 10 monclient: get_auth_request con 0x7fb544037810 auth_method 0 2024-09-16T03:50:13.934 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.914+0000 7fb552ffd640 1 -- 172.21.15.42:0/3604263304 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5540b0040 con 0x7fb56406dac0 2024-09-16T03:50:13.934 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.914+0000 7fb552ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:50:13.934 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.914+0000 7fb552ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:50:13.934 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.914+0000 7fb552ffd640 10 monclient: dump: 2024-09-16T03:50:13.934 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:50:13.935 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:50:13.935 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:13.935 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:13.935 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:50:13.935 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:50:13.935 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:50:13.935 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:50:13.935 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:50:13.935 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:50:13.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.914+0000 7fb552ffd640 1 -- 172.21.15.42:0/3604263304 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fb5540b1040 con 0x7fb56406dac0 2024-09-16T03:50:13.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.914+0000 7fb552ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:50:13.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.914+0000 7fb552ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:50:13.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.914+0000 7fb552ffd640 1 -- 172.21.15.42:0/3604263304 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb56406dac0 msgr2=0x7fb56402c650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:50:13.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.914+0000 7fb552ffd640 1 --2- 172.21.15.42:0/3604263304 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb56406dac0 0x7fb56402c650 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fb56406c630 tx=0x7fb554019190 comp rx=0 tx=0).stop 2024-09-16T03:50:13.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.914+0000 7fb552ffd640 1 -- 172.21.15.42:0/3604263304 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb56406dac0 msgr2=0x7fb56402c650 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:50:13.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.914+0000 7fb552ffd640 1 --2- 172.21.15.42:0/3604263304 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb56406dac0 0x7fb56402c650 secure :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fb56406c630 tx=0x7fb554019190 comp rx=0 tx=0).stop 2024-09-16T03:50:13.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.914+0000 7fb552ffd640 10 client.804140.objecter ms_handle_connect 0x7fb544037810 2024-09-16T03:50:13.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.914+0000 7fb561d9b640 1 --2- 172.21.15.42:0/3604263304 >> v2:172.21.15.154:6812/254845342 conn(0x7fb544037810 0x7fb544039cd0 secure :-1 s=READY pgs=3540 cs=0 l=1 rev1=1 crypto rx=0x7fb56409cc50 tx=0x7fb558007b60 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:50:13.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.918+0000 7fb568ada640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:50:13.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.918+0000 7fb568ada640 1 --2- 172.21.15.42:0/3604263304 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb56402cb90 0x7fb56402cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:50:13.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.918+0000 7fb568ada640 10 monclient(hunting): start opening mon connection 2024-09-16T03:50:13.938 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.918+0000 7fb568ada640 1 -- 172.21.15.42:0/3604263304 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fb564073a30 con 0x7fb56402cb90 2024-09-16T03:50:13.938 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.918+0000 7fb56259c640 1 --2- 172.21.15.42:0/3604263304 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb56402cb90 0x7fb56402cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:50:13.938 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.918+0000 7fb56259c640 10 monclient: get_auth_request con 0x7fb56402cb90 auth_method 0 2024-09-16T03:50:13.938 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.918+0000 7fb56259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:50:13.938 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.918+0000 7fb56259c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:50:13.938 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.918+0000 7fb56259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:50:13.938 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.918+0000 7fb56259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:50:13.938 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.918+0000 7fb56259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:50:13.938 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.918+0000 7fb56259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:50:13.938 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.918+0000 7fb56259c640 10 monclient(hunting): handle_auth_done global_id 804146 payload 824 2024-09-16T03:50:13.938 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.918+0000 7fb56259c640 1 -- 172.21.15.42:0/3604263304 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fb5640a1710 con 0x7fb56402cb90 2024-09-16T03:50:13.938 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.922+0000 7fb552ffd640 10 client.804140.objecter ms_handle_connect 0x7fb56402cb90 2024-09-16T03:50:13.938 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.922+0000 7fb552ffd640 10 client.804140.objecter resend_mon_ops 2024-09-16T03:50:13.939 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.922+0000 7fb56259c640 1 --2- 172.21.15.42:0/3604263304 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb56402cb90 0x7fb56402cfa0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fb554009740 tx=0x7fb554009190 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:50:13.939 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.922+0000 7fb552ffd640 1 -- 172.21.15.42:0/3604263304 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5540c17f0 con 0x7fb56402cb90 2024-09-16T03:50:13.939 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.922+0000 7fb552ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:50:13.939 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.922+0000 7fb552ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:50:13.939 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.922+0000 7fb552ffd640 10 monclient: dump: 2024-09-16T03:50:13.939 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:50:13.939 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:50:13.939 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:13.939 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:13.939 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:50:13.939 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:50:13.939 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:50:13.939 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:50:13.939 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:50:13.940 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:50:13.940 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.922+0000 7fb552ffd640 1 -- 172.21.15.42:0/3604263304 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4544 (secure 0 0 0) 0x7fb5540c5070 con 0x7fb56402cb90 2024-09-16T03:50:13.940 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.922+0000 7fb552ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:50:13.940 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.922+0000 7fb552ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:50:13.940 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.922+0000 7fb552ffd640 1 -- 172.21.15.42:0/3604263304 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb56402cb90 msgr2=0x7fb56402cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:50:13.940 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.922+0000 7fb552ffd640 1 --2- 172.21.15.42:0/3604263304 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb56402cb90 0x7fb56402cfa0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fb554009740 tx=0x7fb554009190 comp rx=0 tx=0).stop 2024-09-16T03:50:13.940 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.922+0000 7fb552ffd640 1 -- 172.21.15.42:0/3604263304 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb56402cb90 msgr2=0x7fb56402cfa0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:50:13.940 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.922+0000 7fb552ffd640 1 --2- 172.21.15.42:0/3604263304 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb56402cb90 0x7fb56402cfa0 secure :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fb554009740 tx=0x7fb554009190 comp rx=0 tx=0).stop 2024-09-16T03:50:13.940 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:50:13.940 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:50:13.940 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:50:13.941 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T03:50:13.941 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 72, 2024-09-16T03:50:13.941 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 12747, 2024-09-16T03:50:13.941 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:50:13.941 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:50:13.941 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T03:50:13.941 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T03:50:13.941 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:50:13.941 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 7, 2024-09-16T03:50:13.941 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:50:13.941 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:50:13.941 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:50:13.941 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:50:13.942 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:50:13.942 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:50:13.942 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:50:13.942 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:50:13.942 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:50:13.942 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:50:13.942 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:50:13.942 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:50:13.942 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:50:13.942 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:50:13.942 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:50:13.942 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:50:13.942 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:50:13.942 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:50:13.943 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:50:13.943 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:50:13.943 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:50:13.943 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:50:13.943 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:50:13.943 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:50:13.943 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:50:13.943 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:50:13.943 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:50:13.943 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:50:13.943 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:50:13.943 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:50:13.943 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:50:13.943 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:50:13.944 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:50:13.944 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:50:13.944 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:50:13.944 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:50:13.944 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:50:13.944 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:50:13.944 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:50:13.944 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:50:13.944 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:50:13.944 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:50:13.944 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:50:13.944 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:50:13.944 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:50:13.945 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:50:13.945 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:50:13.945 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:50:13.945 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:50:13.945 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:50:13.945 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:50:13.945 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:50:13.945 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:50:13.945 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:50:13.945 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:50:13.945 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:50:13.945 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:50:13.945 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:50:13.945 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:50:13.946 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:50:13.946 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:50:13.946 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:50:13.946 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:50:13.946 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:50:13.946 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:50:13.946 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:50:13.946 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:50:13.946 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:50:13.946 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:50:13.946 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:50:13.946 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:50:13.946 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:50:13.946 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:50:13.947 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:50:13.947 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:50:13.947 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:50:13.947 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:50:13.947 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:50:13.947 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:50:13.947 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:50:13.947 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:50:13.947 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:50:13.947 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:50:13.947 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:50:13.947 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:50:13.947 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:50:13.947 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:50:13.948 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:50:13.948 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:50:13.948 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:50:13.948 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:50:13.948 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:50:13.948 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:50:13.948 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:50:13.948 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:50:13.948 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:50:13.948 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:50:13.948 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:50:13.948 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:50:13.948 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:50:13.948 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:50:13.949 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:50:13.949 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:50:13.949 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:50:13.949 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:50:13.949 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:50:13.949 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:50:13.949 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:50:13.949 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:50:13.949 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:50:13.949 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:50:13.949 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:50:13.949 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:50:13.949 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:50:13.949 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:50:13.950 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:50:13.950 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:50:13.950 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:50:13.950 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:50:13.950 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:50:13.950 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:50:13.950 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:50:13.950 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:50:13.950 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:50:13.950 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:50:13.950 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:50:13.950 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:50:13.950 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:50:13.951 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:50:13.951 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:50:13.951 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:50:13.951 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:50:13.951 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:50:13.951 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:50:13.951 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:50:13.951 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:50:13.951 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:50:13.951 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:50:13.951 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:50:13.951 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:50:13.951 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:50:13.951 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:50:13.951 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:50:13.952 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:50:13.952 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:50:13.952 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 2 2024-09-16T03:50:13.952 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:50:13.952 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:50:13.952 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:50:13.952 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:50:13.952 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:50:13.954 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.922+0000 7fb568ada640 10 librados: watch_flush enter 2024-09-16T03:50:13.954 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.922+0000 7fb568ada640 10 librados: watch_flush exit 2024-09-16T03:50:13.954 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.922+0000 7fb568ada640 20 client.804140.objecter shutdown clearing up homeless session... 2024-09-16T03:50:13.954 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.922+0000 7fb568ada640 10 client.804140.objecter successfully canceled tick 2024-09-16T03:50:13.954 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.922+0000 7fb568ada640 1 -- 172.21.15.42:0/3604263304 >> v2:172.21.15.154:6812/254845342 conn(0x7fb544037810 msgr2=0x7fb544039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:50:13.954 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.922+0000 7fb568ada640 1 --2- 172.21.15.42:0/3604263304 >> v2:172.21.15.154:6812/254845342 conn(0x7fb544037810 0x7fb544039cd0 secure :-1 s=READY pgs=3540 cs=0 l=1 rev1=1 crypto rx=0x7fb56409cc50 tx=0x7fb558007b60 comp rx=0 tx=0).stop 2024-09-16T03:50:13.954 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.922+0000 7fb568ada640 10 monclient: shutdown 2024-09-16T03:50:13.954 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.922+0000 7fb568ada640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:50:13.954 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.922+0000 7fb568ada640 1 -- 172.21.15.42:0/3604263304 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb56409dbb0 msgr2=0x7fb56413cbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:50:13.955 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.922+0000 7fb562d9d640 1 -- 172.21.15.42:0/3604263304 reap_dead start 2024-09-16T03:50:13.955 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.922+0000 7fb568ada640 1 --2- 172.21.15.42:0/3604263304 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb56409dbb0 0x7fb56413cbc0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fb55c00c7b0 tx=0x7fb55c00cc70 comp rx=0 tx=0).stop 2024-09-16T03:50:13.955 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.922+0000 7fb568ada640 1 -- 172.21.15.42:0/3604263304 shutdown_connections 2024-09-16T03:50:13.955 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.922+0000 7fb568ada640 1 --2- 172.21.15.42:0/3604263304 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb56409dbb0 0x7fb56413cbc0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:50:13.955 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.922+0000 7fb568ada640 1 -- 172.21.15.42:0/3604263304 >> 172.21.15.42:0/3604263304 conn(0x7fb5640a2420 msgr2=0x7fb564070940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:50:13.955 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.922+0000 7fb568ada640 1 -- 172.21.15.42:0/3604263304 shutdown_connections 2024-09-16T03:50:13.955 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.922+0000 7fb568ada640 1 -- 172.21.15.42:0/3604263304 wait complete. 2024-09-16T03:50:13.955 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:13.922+0000 7fb568ada640 1 librados: shutdown 2024-09-16T03:50:13.955 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:50:13.955 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T03:50:13.956 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T03:50:14.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.070+0000 7f253ea11640 10 monclient: get_monmap_and_config 2024-09-16T03:50:14.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.070+0000 7f253ea11640 10 monclient: build_initial_monmap 2024-09-16T03:50:14.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.070+0000 7f253ea11640 10 monclient: monmap: 2024-09-16T03:50:14.074 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:50:14.074 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:50:14.074 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:50:14.072512+0000 2024-09-16T03:50:14.074 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:50:14.072512+0000 2024-09-16T03:50:14.074 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:50:14.074 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:50:14.074 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:50:14.074 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:50:14.074 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:50:14.074 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:50:14.074 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:50:14.074 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:50:14.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.070+0000 7f253ea11640 1 Processor -- start 2024-09-16T03:50:14.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.070+0000 7f253ea11640 1 -- start start 2024-09-16T03:50:14.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.070+0000 7f253ea11640 10 monclient: init 2024-09-16T03:50:14.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.070+0000 7f253ea11640 10 monclient: _reopen_session rank -1 2024-09-16T03:50:14.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.070+0000 7f253ea11640 10 monclient: _add_conns ranks=[3,4,2,1,0] 2024-09-16T03:50:14.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.070+0000 7f253ea11640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f25380a1330 0x7f25380a1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:50:14.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.070+0000 7f253ea11640 10 monclient(hunting): picked mon.noname-a con 0x7f25380a1330 addr v2:172.21.15.42:3300/0 2024-09-16T03:50:14.080 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.070+0000 7f253ea11640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f253806dac0 0x7f25380a1c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:50:14.080 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.070+0000 7f253ea11640 10 monclient(hunting): picked mon.noname-c con 0x7f253806dac0 addr v2:172.21.15.154:3300/0 2024-09-16T03:50:14.080 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.070+0000 7f253ea11640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2538135100 0x7f2538135550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:50:14.080 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.070+0000 7f253ea11640 10 monclient(hunting): picked mon.noname-b con 0x7f2538135100 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:50:14.080 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.070+0000 7f253ea11640 10 monclient(hunting): start opening mon connection 2024-09-16T03:50:14.080 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.070+0000 7f253ea11640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f253809c7d0 con 0x7f25380a1330 2024-09-16T03:50:14.080 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.070+0000 7f253ea11640 10 monclient(hunting): start opening mon connection 2024-09-16T03:50:14.080 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.070+0000 7f253ea11640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f253809c090 con 0x7f253806dac0 2024-09-16T03:50:14.080 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.070+0000 7f253ea11640 10 monclient(hunting): start opening mon connection 2024-09-16T03:50:14.080 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.070+0000 7f253ea11640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f253809f610 con 0x7f2538135100 2024-09-16T03:50:14.080 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.070+0000 7f253ea11640 10 monclient(hunting): _renew_subs 2024-09-16T03:50:14.080 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.070+0000 7f253ea11640 10 monclient(hunting): authenticate will time out at 1461.041983s 2024-09-16T03:50:14.080 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.070+0000 7f2537fff640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f25380a1330 0x7f25380a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:50:14.081 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.070+0000 7f2537fff640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f25380a1330 0x7f25380a1710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:37084/0 (socket says 172.21.15.42:37084) 2024-09-16T03:50:14.081 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.070+0000 7f2537fff640 1 -- 172.21.15.42:0/4177335987 learned_addr learned my addr 172.21.15.42:0/4177335987 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:50:14.081 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.070+0000 7f2537fff640 10 monclient(hunting): get_auth_request con 0x7f25380a1330 auth_method 0 2024-09-16T03:50:14.081 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.070+0000 7f2537fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:50:14.081 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.070+0000 7f2537fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:50:14.081 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.070+0000 7f2537fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:50:14.081 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.070+0000 7f253cfae640 1 --2- 172.21.15.42:0/4177335987 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2538135100 0x7f2538135550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:50:14.081 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.070+0000 7f253cfae640 10 monclient(hunting): get_auth_request con 0x7f2538135100 auth_method 0 2024-09-16T03:50:14.081 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.070+0000 7f253cfae640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:50:14.081 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.070+0000 7f253cfae640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:50:14.081 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.070+0000 7f253cfae640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:50:14.081 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.070+0000 7f2537fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:50:14.081 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.070+0000 7f2537fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:50:14.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.070+0000 7f2537fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:50:14.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.070+0000 7f253cfae640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:50:14.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.070+0000 7f253cfae640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:50:14.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.070+0000 7f253cfae640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:50:14.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.070+0000 7f25377fe640 1 --2- 172.21.15.42:0/4177335987 >> v2:172.21.15.154:3300/0 conn(0x7f253806dac0 0x7f25380a1c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:50:14.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.070+0000 7f25377fe640 10 monclient(hunting): get_auth_request con 0x7f253806dac0 auth_method 0 2024-09-16T03:50:14.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.070+0000 7f25377fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:50:14.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.070+0000 7f25377fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:50:14.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.070+0000 7f25377fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:50:14.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.070+0000 7f253cfae640 10 monclient(hunting): handle_auth_done global_id 804149 payload 274 2024-09-16T03:50:14.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.070+0000 7f253cfae640 1 -- 172.21.15.42:0/4177335987 >> v2:172.21.15.154:3300/0 conn(0x7f253806dac0 msgr2=0x7f25380a1c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:50:14.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.070+0000 7f253cfae640 1 --2- 172.21.15.42:0/4177335987 >> v2:172.21.15.154:3300/0 conn(0x7f253806dac0 0x7f25380a1c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:50:14.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.070+0000 7f253cfae640 1 -- 172.21.15.42:0/4177335987 >> v2:172.21.15.42:3300/0 conn(0x7f25380a1330 msgr2=0x7f25380a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:50:14.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.070+0000 7f253cfae640 1 --2- 172.21.15.42:0/4177335987 >> v2:172.21.15.42:3300/0 conn(0x7f25380a1330 0x7f25380a1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:50:14.083 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.070+0000 7f253cfae640 10 monclient: _finish_hunting 0 2024-09-16T03:50:14.083 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.070+0000 7f253cfae640 1 monclient: found mon.noname-b 2024-09-16T03:50:14.083 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.070+0000 7f253cfae640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:50:14.083 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.070+0000 7f253cfae640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T03:50:14.083 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.070+0000 7f253cfae640 1 -- 172.21.15.42:0/4177335987 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f253809f8f0 con 0x7f2538135100 2024-09-16T03:50:14.083 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.074+0000 7f253cfae640 1 --2- 172.21.15.42:0/4177335987 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2538135100 0x7f2538135550 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f253806da10 tx=0x7f253001c870 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3af6395f0a88f23d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:50:14.083 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.074+0000 7f2536ffd640 1 -- 172.21.15.42:0/4177335987 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25300b2070 con 0x7f2538135100 2024-09-16T03:50:14.083 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.074+0000 7f2536ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:50:14.083 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.074+0000 7f2536ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:50:14.083 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.074+0000 7f2536ffd640 10 monclient: dump: 2024-09-16T03:50:14.083 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:50:14.083 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:50:14.083 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:14.083 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:14.084 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:50:14.084 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:50:14.084 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:50:14.084 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:50:14.084 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:50:14.084 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:50:14.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.074+0000 7f2536ffd640 10 monclient: _finish_auth 0 2024-09-16T03:50:14.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.074+0000 7f2536ffd640 10 monclient: _check_auth_tickets 2024-09-16T03:50:14.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.074+0000 7f2536ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:50:14.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.074+0000 7f2536ffd640 1 -- 172.21.15.42:0/4177335987 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f25300a5890 con 0x7f2538135100 2024-09-16T03:50:14.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.074+0000 7f253ea11640 5 monclient: authenticate success, global_id 804149 2024-09-16T03:50:14.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.074+0000 7f253ea11640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:50:14.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.074+0000 7f2536ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:50:14.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.074+0000 7f253ea11640 10 monclient: get_monmap_and_config success 2024-09-16T03:50:14.085 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.074+0000 7f2536ffd640 1 -- 172.21.15.42:0/4177335987 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25300a5c60 con 0x7f2538135100 2024-09-16T03:50:14.085 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.074+0000 7f253ea11640 4 set_mon_vals no callback set 2024-09-16T03:50:14.085 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.074+0000 7f2536ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:50:14.085 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.074+0000 7f2536ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:50:14.085 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.074+0000 7f2536ffd640 10 monclient: dump: 2024-09-16T03:50:14.085 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:50:14.085 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:50:14.085 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:14.085 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:14.085 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:50:14.085 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:50:14.085 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:50:14.085 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:50:14.086 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:50:14.086 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:50:14.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.074+0000 7f253ea11640 10 monclient: shutdown 2024-09-16T03:50:14.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.074+0000 7f253ea11640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:50:14.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.074+0000 7f253ea11640 1 -- 172.21.15.42:0/4177335987 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2538135100 msgr2=0x7f2538135550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:50:14.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.074+0000 7f253ea11640 1 --2- 172.21.15.42:0/4177335987 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2538135100 0x7f2538135550 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f253806da10 tx=0x7f253001c870 comp rx=0 tx=0).stop 2024-09-16T03:50:14.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.074+0000 7f253ea11640 1 -- 172.21.15.42:0/4177335987 shutdown_connections 2024-09-16T03:50:14.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.074+0000 7f253ea11640 1 --2- 172.21.15.42:0/4177335987 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2538135100 0x7f2538135550 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:50:14.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.074+0000 7f253ea11640 1 --2- 172.21.15.42:0/4177335987 >> v2:172.21.15.154:3300/0 conn(0x7f253806dac0 0x7f25380a1c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:50:14.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.074+0000 7f253ea11640 1 --2- 172.21.15.42:0/4177335987 >> v2:172.21.15.42:3300/0 conn(0x7f25380a1330 0x7f25380a1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:50:14.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.074+0000 7f253ea11640 1 -- 172.21.15.42:0/4177335987 >> 172.21.15.42:0/4177335987 conn(0x7f25380a2420 msgr2=0x7f25380a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:50:14.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.074+0000 7f253ea11640 1 -- 172.21.15.42:0/4177335987 shutdown_connections 2024-09-16T03:50:14.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.074+0000 7f253ea11640 1 -- 172.21.15.42:0/4177335987 wait complete. 2024-09-16T03:50:14.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.074+0000 7f253ea11640 10 monclient: build_initial_monmap 2024-09-16T03:50:14.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.074+0000 7f253ea11640 10 monclient: monmap: 2024-09-16T03:50:14.087 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:50:14.087 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:50:14.087 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:50:14.087 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:50:14.087 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:50:14.087 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:50:14.087 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:50:14.087 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:50:14.087 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:50:14.087 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:50:14.087 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.074+0000 7f253ea11640 1 librados: starting msgr at 2024-09-16T03:50:14.087 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.074+0000 7f253ea11640 1 librados: starting objecter 2024-09-16T03:50:14.087 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.074+0000 7f253ea11640 1 Processor -- start 2024-09-16T03:50:14.087 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.074+0000 7f253ea11640 1 -- start start 2024-09-16T03:50:14.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.074+0000 7f253ea11640 1 librados: setting wanted keys 2024-09-16T03:50:14.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.074+0000 7f253ea11640 1 librados: calling monclient init 2024-09-16T03:50:14.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.074+0000 7f253ea11640 10 monclient: init 2024-09-16T03:50:14.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.074+0000 7f253ea11640 10 monclient: _reopen_session rank -1 2024-09-16T03:50:14.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.074+0000 7f253ea11640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:50:14.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.074+0000 7f253ea11640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f25380a1330 0x7f25381ccc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:50:14.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.074+0000 7f253ea11640 10 monclient(hunting): picked mon.noname-a con 0x7f25380a1330 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:50:14.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.074+0000 7f253ea11640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2538135100 0x7f2538134240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:50:14.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.074+0000 7f253ea11640 10 monclient(hunting): picked mon.noname-c con 0x7f2538135100 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:50:14.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.074+0000 7f253ea11640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2538134780 0x7f253811bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:50:14.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.074+0000 7f253ea11640 10 monclient(hunting): picked mon.noname-b con 0x7f2538134780 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:50:14.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.074+0000 7f253ea11640 10 monclient(hunting): start opening mon connection 2024-09-16T03:50:14.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.074+0000 7f253ea11640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f253809eef0 con 0x7f25380a1330 2024-09-16T03:50:14.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.074+0000 7f253ea11640 10 monclient(hunting): start opening mon connection 2024-09-16T03:50:14.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.074+0000 7f253ea11640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f253809ed70 con 0x7f2538134780 2024-09-16T03:50:14.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.074+0000 7f253ea11640 10 monclient(hunting): start opening mon connection 2024-09-16T03:50:14.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.074+0000 7f253ea11640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f253813eb70 con 0x7f2538135100 2024-09-16T03:50:14.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.074+0000 7f253ea11640 10 monclient(hunting): _renew_subs 2024-09-16T03:50:14.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.074+0000 7f253ea11640 10 monclient(hunting): authenticate will time out at 1461.047042s 2024-09-16T03:50:14.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.074+0000 7f2537fff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f25380a1330 0x7f25381ccc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:50:14.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.074+0000 7f25377fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2538135100 0x7f2538134240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:50:14.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.074+0000 7f2537fff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f25380a1330 0x7f25381ccc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:37098/0 (socket says 172.21.15.42:37098) 2024-09-16T03:50:14.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.074+0000 7f2537fff640 1 -- 172.21.15.42:0/3742723013 learned_addr learned my addr 172.21.15.42:0/3742723013 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:50:14.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.074+0000 7f2537fff640 10 monclient(hunting): get_auth_request con 0x7f25380a1330 auth_method 0 2024-09-16T03:50:14.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.074+0000 7f2537fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:50:14.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.074+0000 7f2537fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:50:14.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.074+0000 7f2537fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:50:14.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.074+0000 7f25377fe640 10 monclient(hunting): get_auth_request con 0x7f2538135100 auth_method 0 2024-09-16T03:50:14.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.074+0000 7f25377fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:50:14.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.074+0000 7f25377fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:50:14.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.074+0000 7f25377fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:50:14.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.074+0000 7f2537fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:50:14.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.074+0000 7f2537fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:50:14.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.074+0000 7f2537fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:50:14.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.074+0000 7f25377fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:50:14.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.074+0000 7f25377fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:50:14.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.074+0000 7f25377fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:50:14.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.074+0000 7f253cfae640 1 --2- 172.21.15.42:0/3742723013 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2538134780 0x7f253811bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:50:14.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.074+0000 7f2537fff640 10 monclient(hunting): handle_auth_done global_id 764823 payload 824 2024-09-16T03:50:14.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.074+0000 7f2537fff640 1 -- 172.21.15.42:0/3742723013 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2538135100 msgr2=0x7f2538134240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:50:14.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.074+0000 7f2537fff640 1 --2- 172.21.15.42:0/3742723013 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2538135100 0x7f2538134240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:50:14.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.074+0000 7f2537fff640 1 -- 172.21.15.42:0/3742723013 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2538134780 msgr2=0x7f253811bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:50:14.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.074+0000 7f2537fff640 1 --2- 172.21.15.42:0/3742723013 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2538134780 0x7f253811bf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:50:14.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.074+0000 7f2537fff640 10 monclient: _finish_hunting 0 2024-09-16T03:50:14.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.074+0000 7f2537fff640 1 monclient: found mon.noname-a 2024-09-16T03:50:14.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.074+0000 7f2537fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:50:14.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.074+0000 7f2537fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:50:14.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.074+0000 7f2537fff640 1 -- 172.21.15.42:0/3742723013 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f253811d8f0 con 0x7f25380a1330 2024-09-16T03:50:14.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.074+0000 7f253cfae640 10 monclient: get_auth_request con 0x7f2538134780 auth_method 0 2024-09-16T03:50:14.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.074+0000 7f253cfae640 1 --2- 172.21.15.42:0/3742723013 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2538134780 0x7f253811bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T03:50:14.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.074+0000 7f2537fff640 1 --2- 172.21.15.42:0/3742723013 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f25380a1330 0x7f25381ccc90 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f252c00e890 tx=0x7f252c00ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:50:14.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.074+0000 7f2534ff9640 10 client.?.objecter ms_handle_connect 0x7f25380a1330 2024-09-16T03:50:14.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.074+0000 7f2534ff9640 10 client.?.objecter resend_mon_ops 2024-09-16T03:50:14.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.074+0000 7f2534ff9640 1 -- 172.21.15.42:0/3742723013 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f252c017070 con 0x7f25380a1330 2024-09-16T03:50:14.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.078+0000 7f2534ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:50:14.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.078+0000 7f2534ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:50:14.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.078+0000 7f2534ff9640 10 monclient: dump: 2024-09-16T03:50:14.098 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:50:14.098 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:50:14.098 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:14.098 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:14.098 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:50:14.098 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:50:14.098 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:50:14.098 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:50:14.098 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:50:14.098 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:50:14.098 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.078+0000 7f2534ff9640 10 monclient: _finish_auth 0 2024-09-16T03:50:14.098 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.078+0000 7f2534ff9640 10 monclient: _check_auth_tickets 2024-09-16T03:50:14.098 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.078+0000 7f2534ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:50:14.098 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.078+0000 7f2534ff9640 1 -- 172.21.15.42:0/3742723013 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f252c00c940 con 0x7f25380a1330 2024-09-16T03:50:14.099 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.078+0000 7f253ea11640 5 monclient: authenticate success, global_id 764823 2024-09-16T03:50:14.099 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.078+0000 7f253ea11640 10 monclient: _renew_subs 2024-09-16T03:50:14.099 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.078+0000 7f253ea11640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:50:14.099 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.078+0000 7f253ea11640 1 -- 172.21.15.42:0/3742723013 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f253811fbe0 con 0x7f25380a1330 2024-09-16T03:50:14.099 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.078+0000 7f2534ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T03:50:14.099 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.078+0000 7f2534ff9640 1 -- 172.21.15.42:0/3742723013 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f252c00cd30 con 0x7f25380a1330 2024-09-16T03:50:14.099 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.078+0000 7f2534ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:50:14.099 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.078+0000 7f2534ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:50:14.099 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.078+0000 7f2534ff9640 10 monclient: dump: 2024-09-16T03:50:14.099 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:50:14.099 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:50:14.099 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:14.099 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:14.099 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:50:14.100 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:50:14.100 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:50:14.100 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:50:14.100 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:50:14.100 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:50:14.100 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.078+0000 7f253ea11640 10 client.764823.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:50:14.100 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.078+0000 7f253ea11640 10 monclient: _renew_subs 2024-09-16T03:50:14.100 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.078+0000 7f253ea11640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:50:14.100 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.078+0000 7f253ea11640 1 -- 172.21.15.42:0/3742723013 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f253811f900 con 0x7f25380a1330 2024-09-16T03:50:14.100 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.078+0000 7f2535ffb640 4 set_mon_vals no callback set 2024-09-16T03:50:14.100 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.078+0000 7f253ea11640 1 librados: init done 2024-09-16T03:50:14.100 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.078+0000 7f253ea11640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:50:14.100 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.078+0000 7f253ea11640 1 --2- 172.21.15.42:0/3742723013 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f253806dac0 0x7f253811c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:50:14.100 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.078+0000 7f253ea11640 10 monclient(hunting): start opening mon connection 2024-09-16T03:50:14.101 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.078+0000 7f253ea11640 1 -- 172.21.15.42:0/3742723013 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f2538073a30 con 0x7f253806dac0 2024-09-16T03:50:14.101 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.078+0000 7f2534ff9640 1 -- 172.21.15.42:0/3742723013 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f252c01a070 con 0x7f25380a1330 2024-09-16T03:50:14.101 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.078+0000 7f2534ff9640 1 --2- 172.21.15.42:0/3742723013 >> v2:172.21.15.154:6812/254845342 conn(0x7f2524037810 0x7f2524039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:50:14.101 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.078+0000 7f25377fe640 1 --2- 172.21.15.42:0/3742723013 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f253806dac0 0x7f253811c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:50:14.101 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.078+0000 7f2534ff9640 1 -- 172.21.15.42:0/3742723013 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(281..281 src has 1..281) ==== 6024+0+0 (secure 0 0 0) 0x7f252c048f30 con 0x7f25380a1330 2024-09-16T03:50:14.101 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.078+0000 7f2534ff9640 10 client.764823.objecter ms_dispatch 0x7f2538000ba0 osd_map(281..281 src has 1..281) 2024-09-16T03:50:14.101 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.078+0000 7f2534ff9640 3 client.764823.objecter handle_osd_map got epochs [281,281] > 0 2024-09-16T03:50:14.101 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.078+0000 7f2534ff9640 3 client.764823.objecter handle_osd_map decoding full epoch 281 2024-09-16T03:50:14.101 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.078+0000 7f25377fe640 10 monclient: get_auth_request con 0x7f253806dac0 auth_method 0 2024-09-16T03:50:14.101 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.078+0000 7f25377fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:50:14.101 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.078+0000 7f25377fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:50:14.101 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.078+0000 7f25377fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:50:14.101 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.078+0000 7f2534ff9640 20 client.764823.objecter dump_active .. 0 homeless 2024-09-16T03:50:14.101 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.078+0000 7f25377fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:50:14.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.078+0000 7f25377fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:50:14.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.078+0000 7f25377fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:50:14.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.078+0000 7f253cfae640 1 --2- 172.21.15.42:0/3742723013 >> v2:172.21.15.154:6812/254845342 conn(0x7f2524037810 0x7f2524039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:50:14.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.078+0000 7f253cfae640 10 monclient: get_auth_request con 0x7f2524037810 auth_method 0 2024-09-16T03:50:14.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.078+0000 7f25377fe640 10 monclient(hunting): handle_auth_done global_id 794257 payload 824 2024-09-16T03:50:14.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.078+0000 7f25377fe640 1 -- 172.21.15.42:0/3742723013 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f25380a5100 con 0x7f253806dac0 2024-09-16T03:50:14.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.078+0000 7f2534ff9640 10 client.764823.objecter ms_handle_connect 0x7f253806dac0 2024-09-16T03:50:14.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.078+0000 7f25377fe640 1 --2- 172.21.15.42:0/3742723013 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f253806dac0 0x7f253811c4b0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f25280107f0 tx=0x7f2528010cb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:50:14.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.078+0000 7f2534ff9640 10 client.764823.objecter resend_mon_ops 2024-09-16T03:50:14.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.078+0000 7f2534ff9640 1 -- 172.21.15.42:0/3742723013 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2528018020 con 0x7f253806dac0 2024-09-16T03:50:14.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.078+0000 7f2534ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:50:14.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.078+0000 7f2534ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:50:14.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.078+0000 7f253cfae640 1 --2- 172.21.15.42:0/3742723013 >> v2:172.21.15.154:6812/254845342 conn(0x7f2524037810 0x7f2524039cd0 secure :-1 s=READY pgs=3541 cs=0 l=1 rev1=1 crypto rx=0x7f25300af000 tx=0x7f25300a7070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:50:14.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.078+0000 7f2534ff9640 10 monclient: dump: 2024-09-16T03:50:14.103 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:50:14.103 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:50:14.103 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:14.103 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:14.103 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:50:14.103 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:50:14.103 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:50:14.103 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:50:14.103 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:50:14.103 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:50:14.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.078+0000 7f2534ff9640 10 client.764823.objecter ms_handle_connect 0x7f2524037810 2024-09-16T03:50:14.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.078+0000 7f2534ff9640 1 -- 172.21.15.42:0/3742723013 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f2528004240 con 0x7f253806dac0 2024-09-16T03:50:14.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.078+0000 7f2534ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:50:14.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.078+0000 7f2534ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:50:14.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.078+0000 7f2534ff9640 1 -- 172.21.15.42:0/3742723013 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f253806dac0 msgr2=0x7f253811c4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:50:14.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.078+0000 7f2534ff9640 1 --2- 172.21.15.42:0/3742723013 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f253806dac0 0x7f253811c4b0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f25280107f0 tx=0x7f2528010cb0 comp rx=0 tx=0).stop 2024-09-16T03:50:14.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.078+0000 7f2534ff9640 1 -- 172.21.15.42:0/3742723013 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f253806dac0 msgr2=0x7f253811c4b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:50:14.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.078+0000 7f2534ff9640 1 --2- 172.21.15.42:0/3742723013 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f253806dac0 0x7f253811c4b0 secure :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f25280107f0 tx=0x7f2528010cb0 comp rx=0 tx=0).stop 2024-09-16T03:50:14.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.086+0000 7f253ea11640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:50:14.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.086+0000 7f253ea11640 1 --2- 172.21.15.42:0/3742723013 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f253811c9f0 0x7f2538138330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:50:14.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.086+0000 7f253ea11640 10 monclient(hunting): start opening mon connection 2024-09-16T03:50:14.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.086+0000 7f253ea11640 1 -- 172.21.15.42:0/3742723013 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f253811ce00 con 0x7f253811c9f0 2024-09-16T03:50:14.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.086+0000 7f25377fe640 1 --2- 172.21.15.42:0/3742723013 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f253811c9f0 0x7f2538138330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:50:14.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.086+0000 7f25377fe640 10 monclient: get_auth_request con 0x7f253811c9f0 auth_method 0 2024-09-16T03:50:14.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.086+0000 7f25377fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:50:14.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.086+0000 7f25377fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:50:14.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.086+0000 7f25377fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:50:14.105 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.086+0000 7f25377fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:50:14.105 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.086+0000 7f25377fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:50:14.105 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.086+0000 7f25377fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:50:14.105 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.086+0000 7f25377fe640 10 monclient(hunting): handle_auth_done global_id 794260 payload 824 2024-09-16T03:50:14.105 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.086+0000 7f25377fe640 1 -- 172.21.15.42:0/3742723013 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f2538135550 con 0x7f253811c9f0 2024-09-16T03:50:14.105 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.086+0000 7f2534ff9640 10 client.764823.objecter ms_handle_connect 0x7f253811c9f0 2024-09-16T03:50:14.105 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.086+0000 7f2534ff9640 10 client.764823.objecter resend_mon_ops 2024-09-16T03:50:14.105 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.086+0000 7f25377fe640 1 --2- 172.21.15.42:0/3742723013 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f253811c9f0 0x7f2538138330 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f25280107f0 tx=0x7f25280046d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:50:14.105 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.086+0000 7f2534ff9640 1 -- 172.21.15.42:0/3742723013 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2528018070 con 0x7f253811c9f0 2024-09-16T03:50:14.105 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.086+0000 7f2534ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:50:14.105 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.086+0000 7f2534ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:50:14.105 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.086+0000 7f2534ff9640 10 monclient: dump: 2024-09-16T03:50:14.105 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:50:14.105 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:50:14.105 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:14.106 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:14.106 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:50:14.106 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:50:14.106 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:50:14.106 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:50:14.106 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:50:14.106 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:50:14.106 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.086+0000 7f2534ff9640 1 -- 172.21.15.42:0/3742723013 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4544 (secure 0 0 0) 0x7f252800eaf0 con 0x7f253811c9f0 2024-09-16T03:50:14.106 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.086+0000 7f2534ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:50:14.106 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.086+0000 7f2534ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:50:14.106 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.086+0000 7f2534ff9640 1 -- 172.21.15.42:0/3742723013 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f253811c9f0 msgr2=0x7f2538138330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:50:14.106 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.086+0000 7f2534ff9640 1 --2- 172.21.15.42:0/3742723013 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f253811c9f0 0x7f2538138330 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f25280107f0 tx=0x7f25280046d0 comp rx=0 tx=0).stop 2024-09-16T03:50:14.106 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.086+0000 7f2534ff9640 1 -- 172.21.15.42:0/3742723013 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f253811c9f0 msgr2=0x7f2538138330 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:50:14.106 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.086+0000 7f2534ff9640 1 --2- 172.21.15.42:0/3742723013 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f253811c9f0 0x7f2538138330 secure :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f25280107f0 tx=0x7f25280046d0 comp rx=0 tx=0).stop 2024-09-16T03:50:14.107 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:50:14.107 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:50:14.107 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:50:14.107 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T03:50:14.107 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 72, 2024-09-16T03:50:14.107 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 55927, 2024-09-16T03:50:14.107 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:50:14.107 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:50:14.107 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T03:50:14.108 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T03:50:14.108 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:50:14.108 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 7, 2024-09-16T03:50:14.108 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:50:14.108 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:50:14.108 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:50:14.108 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:50:14.108 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:50:14.108 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:50:14.108 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:50:14.108 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:50:14.108 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:50:14.108 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:50:14.108 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:50:14.109 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:50:14.109 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:50:14.109 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:50:14.109 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:50:14.109 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:50:14.109 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:50:14.109 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:50:14.109 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:50:14.109 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:50:14.109 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:50:14.109 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:50:14.109 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:50:14.109 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:50:14.110 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:50:14.110 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:50:14.110 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:50:14.110 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:50:14.110 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:50:14.110 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:50:14.110 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:50:14.110 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:50:14.110 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:50:14.110 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:50:14.110 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:50:14.110 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:50:14.110 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:50:14.110 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:50:14.111 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:50:14.111 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:50:14.111 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:50:14.111 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:50:14.111 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:50:14.111 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:50:14.111 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:50:14.111 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:50:14.111 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:50:14.111 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:50:14.111 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:50:14.111 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:50:14.111 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:50:14.112 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:50:14.112 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:50:14.112 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:50:14.112 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:50:14.112 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:50:14.112 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:50:14.112 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:50:14.112 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:50:14.112 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:50:14.112 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:50:14.112 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:50:14.112 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:50:14.112 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:50:14.112 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:50:14.113 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:50:14.113 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:50:14.113 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:50:14.113 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:50:14.113 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:50:14.113 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:50:14.113 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:50:14.113 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:50:14.113 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:50:14.113 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:50:14.113 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:50:14.113 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:50:14.113 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:50:14.114 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:50:14.114 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:50:14.114 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:50:14.114 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:50:14.114 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:50:14.114 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:50:14.114 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:50:14.114 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:50:14.114 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:50:14.114 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:50:14.114 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:50:14.114 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:50:14.114 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:50:14.114 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:50:14.114 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:50:14.115 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:50:14.115 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:50:14.115 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:50:14.115 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:50:14.115 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:50:14.115 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:50:14.115 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:50:14.115 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:50:14.115 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:50:14.115 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:50:14.115 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:50:14.115 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:50:14.115 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:50:14.116 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:50:14.116 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:50:14.116 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:50:14.116 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:50:14.116 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:50:14.116 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:50:14.116 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:50:14.116 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:50:14.116 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:50:14.116 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:50:14.116 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:50:14.116 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:50:14.116 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:50:14.116 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:50:14.117 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:50:14.117 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:50:14.117 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:50:14.117 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:50:14.117 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:50:14.117 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:50:14.117 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:50:14.117 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:50:14.117 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:50:14.117 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:50:14.117 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:50:14.117 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:50:14.117 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:50:14.117 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:50:14.117 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:50:14.118 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:50:14.118 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:50:14.118 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:50:14.118 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:50:14.118 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:50:14.118 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:50:14.118 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:50:14.118 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:50:14.118 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:50:14.118 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:50:14.118 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 2 2024-09-16T03:50:14.118 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:50:14.118 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:50:14.118 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:50:14.119 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:50:14.119 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:50:14.120 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.086+0000 7f253ea11640 10 librados: watch_flush enter 2024-09-16T03:50:14.120 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.086+0000 7f253ea11640 10 librados: watch_flush exit 2024-09-16T03:50:14.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.086+0000 7f253ea11640 20 client.764823.objecter shutdown clearing up homeless session... 2024-09-16T03:50:14.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.090+0000 7f253ea11640 10 client.764823.objecter successfully canceled tick 2024-09-16T03:50:14.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.090+0000 7f253ea11640 1 -- 172.21.15.42:0/3742723013 >> v2:172.21.15.154:6812/254845342 conn(0x7f2524037810 msgr2=0x7f2524039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:50:14.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.090+0000 7f253ea11640 1 --2- 172.21.15.42:0/3742723013 >> v2:172.21.15.154:6812/254845342 conn(0x7f2524037810 0x7f2524039cd0 secure :-1 s=READY pgs=3541 cs=0 l=1 rev1=1 crypto rx=0x7f25300af000 tx=0x7f25300a7070 comp rx=0 tx=0).stop 2024-09-16T03:50:14.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.090+0000 7f253ea11640 10 monclient: shutdown 2024-09-16T03:50:14.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.090+0000 7f253ea11640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:50:14.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.090+0000 7f253ea11640 1 -- 172.21.15.42:0/3742723013 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f25380a1330 msgr2=0x7f25381ccc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:50:14.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.090+0000 7f253ea11640 1 --2- 172.21.15.42:0/3742723013 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f25380a1330 0x7f25381ccc90 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f252c00e890 tx=0x7f252c00ed50 comp rx=0 tx=0).stop 2024-09-16T03:50:14.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.090+0000 7f253cfae640 1 -- 172.21.15.42:0/3742723013 reap_dead start 2024-09-16T03:50:14.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.090+0000 7f253cfae640 1 -- 172.21.15.42:0/3742723013 reap_dead start 2024-09-16T03:50:14.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.090+0000 7f253ea11640 1 -- 172.21.15.42:0/3742723013 shutdown_connections 2024-09-16T03:50:14.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.090+0000 7f253ea11640 1 -- 172.21.15.42:0/3742723013 >> 172.21.15.42:0/3742723013 conn(0x7f25380a2420 msgr2=0x7f25380a3720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:50:14.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.090+0000 7f253ea11640 1 -- 172.21.15.42:0/3742723013 shutdown_connections 2024-09-16T03:50:14.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.090+0000 7f253ea11640 1 -- 172.21.15.42:0/3742723013 wait complete. 2024-09-16T03:50:14.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:14.090+0000 7f253ea11640 1 librados: shutdown 2024-09-16T03:50:14.122 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:50:14.122 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-09-16T03:50:14.122 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['a'] 2024-09-16T03:50:14.122 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-09-16T03:50:14.122 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-16T03:50:14.122 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-16T03:50:14.123 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T03:50:14.133 INFO:tasks.ceph.mon.c:Stopped 2024-09-16T03:50:14.134 INFO:tasks.mon_thrash:Sending STOP to mon a 2024-09-16T03:50:14.134 INFO:tasks.ceph.mon.a:Sent signal 19 2024-09-16T03:50:14.134 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-16T03:50:29.135 INFO:tasks.mon_thrash:Sending CONT to mon a 2024-09-16T03:50:29.136 INFO:tasks.ceph.mon.a:Sent signal 18 2024-09-16T03:50:29.136 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T03:50:29.136 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:50:29.266 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:29.262+0000 7f3bcc671640 10 monclient: get_monmap_and_config 2024-09-16T03:50:29.267 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:29.262+0000 7f3bcc671640 10 monclient: build_initial_monmap 2024-09-16T03:50:29.267 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:29.262+0000 7f3bcc671640 10 monclient: monmap: 2024-09-16T03:50:29.267 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:50:29.267 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:50:29.267 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:50:29.265818+0000 2024-09-16T03:50:29.267 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:50:29.265818+0000 2024-09-16T03:50:29.267 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:50:29.267 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:50:29.267 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:50:29.267 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:50:29.267 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:50:29.268 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:50:29.268 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:50:29.268 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:50:29.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:29.262+0000 7f3bcc671640 1 Processor -- start 2024-09-16T03:50:29.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:29.262+0000 7f3bcc671640 1 -- start start 2024-09-16T03:50:29.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:29.262+0000 7f3bcc671640 10 monclient: init 2024-09-16T03:50:29.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:29.262+0000 7f3bcc671640 10 monclient: _reopen_session rank -1 2024-09-16T03:50:29.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:29.262+0000 7f3bcc671640 10 monclient: _add_conns ranks=[1,2,4,0,3] 2024-09-16T03:50:29.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:29.262+0000 7f3bcc671640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3bc40a1330 addr v1:172.21.15.154:6789/0 2024-09-16T03:50:29.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:29.262+0000 7f3bcc671640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3bc406dac0 0x7f3bc413a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:50:29.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:29.262+0000 7f3bcc671640 10 monclient(hunting): picked mon.noname-b con 0x7f3bc406dac0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:50:29.271 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:29.262+0000 7f3bcc671640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f3bc40a1e20 0x7f3bc413c890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:50:29.271 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:29.262+0000 7f3bcc671640 10 monclient(hunting): picked mon.noname-c con 0x7f3bc40a1e20 addr v2:172.21.15.154:3300/0 2024-09-16T03:50:29.271 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:29.262+0000 7f3bcc671640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3bc409c7d0 con 0x7f3bc40a1330 2024-09-16T03:50:29.271 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:29.262+0000 7f3bcc671640 10 monclient(hunting): start opening mon connection 2024-09-16T03:50:29.271 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:29.262+0000 7f3bcc671640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f3bc409c090 con 0x7f3bc40a1e20 2024-09-16T03:50:29.271 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:29.262+0000 7f3bcc671640 10 monclient(hunting): start opening mon connection 2024-09-16T03:50:29.271 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:29.262+0000 7f3bcc671640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f3bc409f610 con 0x7f3bc406dac0 2024-09-16T03:50:29.271 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:29.262+0000 7f3bcc671640 10 monclient(hunting): _renew_subs 2024-09-16T03:50:29.271 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:29.262+0000 7f3bcc671640 10 monclient(hunting): authenticate will time out at 1476.235290s 2024-09-16T03:50:29.271 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:29.262+0000 7f3bc9c0c640 1 -- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3bc406dac0 msgr2=0x7f3bc413a340 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T03:50:29.271 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:29.262+0000 7f3bc9c0c640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3bc406dac0 0x7f3bc413a340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:50:29.271 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:29.262+0000 7f3bcac0e640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f3bc40a1e20 0x7f3bc413c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:50:29.271 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:29.262+0000 7f3bca40d640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f3bc40a1330 0x7f3bc40a1710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.42:38470/0 (socket says 172.21.15.42:38470) 2024-09-16T03:50:29.272 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:29.262+0000 7f3bca40d640 1 -- 172.21.15.42:0/879723185 learned_addr learned my addr 172.21.15.42:0/879723185 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:50:29.272 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:29.262+0000 7f3bcac0e640 10 monclient(hunting): get_auth_request con 0x7f3bc40a1e20 auth_method 0 2024-09-16T03:50:29.272 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:29.262+0000 7f3bcac0e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:50:29.272 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:29.262+0000 7f3bcac0e640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:50:29.272 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:29.262+0000 7f3bcac0e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:50:29.272 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:29.262+0000 7f3bcac0e640 1 -- 172.21.15.42:0/879723185 >> v2:172.21.15.154:3300/0 conn(0x7f3bc40a1e20 msgr2=0x7f3bc413c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T03:50:29.272 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:29.262+0000 7f3bcac0e640 1 -- 172.21.15.42:0/879723185 >> v2:172.21.15.154:3300/0 conn(0x7f3bc40a1e20 msgr2=0x7f3bc413c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:50:29.272 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:29.262+0000 7f3bcac0e640 1 --2- 172.21.15.42:0/879723185 >> v2:172.21.15.154:3300/0 conn(0x7f3bc40a1e20 0x7f3bc413c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:50:29.272 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:29.266+0000 7f3bcac0e640 1 --2- 172.21.15.42:0/879723185 >> v2:172.21.15.154:3300/0 conn(0x7f3bc40a1e20 0x7f3bc413c890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:50:29.469 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:29.466+0000 7f3bc9c0c640 1 -- 172.21.15.42:0/879723185 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3bc406dac0 msgr2=0x7f3bc413a340 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T03:50:29.469 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:29.466+0000 7f3bc9c0c640 1 --2- 172.21.15.42:0/879723185 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3bc406dac0 0x7f3bc413a340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T03:50:29.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:29.470+0000 7f3bcac0e640 1 --2- 172.21.15.42:0/879723185 >> v2:172.21.15.154:3300/0 conn(0x7f3bc40a1e20 0x7f3bc413c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:50:29.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:29.470+0000 7f3bcac0e640 10 monclient(hunting): get_auth_request con 0x7f3bc40a1e20 auth_method 0 2024-09-16T03:50:29.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:29.470+0000 7f3bcac0e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:50:29.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:29.470+0000 7f3bcac0e640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:50:29.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:29.470+0000 7f3bcac0e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:50:29.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:29.470+0000 7f3bcac0e640 1 -- 172.21.15.42:0/879723185 >> v2:172.21.15.154:3300/0 conn(0x7f3bc40a1e20 msgr2=0x7f3bc413c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T03:50:29.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:29.470+0000 7f3bcac0e640 1 -- 172.21.15.42:0/879723185 >> v2:172.21.15.154:3300/0 conn(0x7f3bc40a1e20 msgr2=0x7f3bc413c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:50:29.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:29.470+0000 7f3bcac0e640 1 --2- 172.21.15.42:0/879723185 >> v2:172.21.15.154:3300/0 conn(0x7f3bc40a1e20 0x7f3bc413c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:50:29.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:29.470+0000 7f3bcac0e640 1 --2- 172.21.15.42:0/879723185 >> v2:172.21.15.154:3300/0 conn(0x7f3bc40a1e20 0x7f3bc413c890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T03:50:29.873 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:29.870+0000 7f3bc9c0c640 1 -- 172.21.15.42:0/879723185 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3bc406dac0 msgr2=0x7f3bc413a340 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T03:50:29.873 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:29.870+0000 7f3bc9c0c640 1 --2- 172.21.15.42:0/879723185 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3bc406dac0 0x7f3bc413a340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T03:50:29.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:29.874+0000 7f3bcac0e640 1 --2- 172.21.15.42:0/879723185 >> v2:172.21.15.154:3300/0 conn(0x7f3bc40a1e20 0x7f3bc413c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:50:29.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:29.874+0000 7f3bcac0e640 10 monclient(hunting): get_auth_request con 0x7f3bc40a1e20 auth_method 0 2024-09-16T03:50:29.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:29.874+0000 7f3bcac0e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:50:29.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:29.874+0000 7f3bcac0e640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:50:29.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:29.874+0000 7f3bcac0e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:50:29.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:29.874+0000 7f3bcac0e640 1 -- 172.21.15.42:0/879723185 >> v2:172.21.15.154:3300/0 conn(0x7f3bc40a1e20 msgr2=0x7f3bc413c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T03:50:29.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:29.874+0000 7f3bcac0e640 1 -- 172.21.15.42:0/879723185 >> v2:172.21.15.154:3300/0 conn(0x7f3bc40a1e20 msgr2=0x7f3bc413c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:50:29.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:29.874+0000 7f3bcac0e640 1 --2- 172.21.15.42:0/879723185 >> v2:172.21.15.154:3300/0 conn(0x7f3bc40a1e20 0x7f3bc413c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:50:29.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:29.874+0000 7f3bcac0e640 1 --2- 172.21.15.42:0/879723185 >> v2:172.21.15.154:3300/0 conn(0x7f3bc40a1e20 0x7f3bc413c890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T03:50:30.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:30.674+0000 7f3bc9c0c640 1 -- 172.21.15.42:0/879723185 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3bc406dac0 msgr2=0x7f3bc413a340 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T03:50:30.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:30.674+0000 7f3bc9c0c640 1 --2- 172.21.15.42:0/879723185 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3bc406dac0 0x7f3bc413a340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T03:50:30.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:30.678+0000 7f3bcac0e640 1 --2- 172.21.15.42:0/879723185 >> v2:172.21.15.154:3300/0 conn(0x7f3bc40a1e20 0x7f3bc413c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:50:30.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:30.678+0000 7f3bcac0e640 10 monclient(hunting): get_auth_request con 0x7f3bc40a1e20 auth_method 0 2024-09-16T03:50:30.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:30.678+0000 7f3bcac0e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:50:30.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:30.678+0000 7f3bcac0e640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:50:30.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:30.678+0000 7f3bcac0e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:50:30.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:30.678+0000 7f3bcac0e640 1 -- 172.21.15.42:0/879723185 >> v2:172.21.15.154:3300/0 conn(0x7f3bc40a1e20 msgr2=0x7f3bc413c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T03:50:30.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:30.678+0000 7f3bcac0e640 1 -- 172.21.15.42:0/879723185 >> v2:172.21.15.154:3300/0 conn(0x7f3bc40a1e20 msgr2=0x7f3bc413c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:50:30.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:30.678+0000 7f3bcac0e640 1 --2- 172.21.15.42:0/879723185 >> v2:172.21.15.154:3300/0 conn(0x7f3bc40a1e20 0x7f3bc413c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:50:30.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:30.678+0000 7f3bcac0e640 1 --2- 172.21.15.42:0/879723185 >> v2:172.21.15.154:3300/0 conn(0x7f3bc40a1e20 0x7f3bc413c890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T03:50:32.269 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:32.262+0000 7f3bbbfff640 10 monclient(hunting): tick 2024-09-16T03:50:32.269 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:32.262+0000 7f3bbbfff640 10 monclient(hunting): _check_auth_tickets 2024-09-16T03:50:32.269 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:32.262+0000 7f3bbbfff640 1 monclient(hunting): continuing hunt 2024-09-16T03:50:32.269 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:32.262+0000 7f3bbbfff640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T03:50:32.269 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:32.262+0000 7f3bbbfff640 1 -- 172.21.15.42:0/879723185 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3bc406dac0 msgr2=0x7f3bc413a340 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:50:32.269 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:32.262+0000 7f3bbbfff640 1 --2- 172.21.15.42:0/879723185 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3bc406dac0 0x7f3bc413a340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:50:32.269 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:32.262+0000 7f3bbbfff640 1 -- 172.21.15.42:0/879723185 >> v2:172.21.15.154:3300/0 conn(0x7f3bc40a1e20 msgr2=0x7f3bc413c890 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:50:32.269 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:32.262+0000 7f3bbbfff640 1 --2- 172.21.15.42:0/879723185 >> v2:172.21.15.154:3300/0 conn(0x7f3bc40a1e20 0x7f3bc413c890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:50:32.269 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:32.262+0000 7f3bbbfff640 1 -- 172.21.15.42:0/879723185 >> v1:172.21.15.154:6789/0 conn(0x7f3bc40a1330 legacy=0x7f3bc40a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:50:32.269 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:32.262+0000 7f3bbbfff640 10 monclient: _add_conns ranks=[3,0] 2024-09-16T03:50:32.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:32.262+0000 7f3bbbfff640 1 --2- 172.21.15.42:0/879723185 >> v2:172.21.15.42:3300/0 conn(0x7f3ba80036d0 0x7f3ba8005b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:50:32.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:32.262+0000 7f3bbbfff640 10 monclient(hunting): picked mon.noname-a con 0x7f3ba80036d0 addr v2:172.21.15.42:3300/0 2024-09-16T03:50:32.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:32.262+0000 7f3bbbfff640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3ba8006180 addr v1:172.21.15.42:6789/0 2024-09-16T03:50:32.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:32.262+0000 7f3bbbfff640 1 -- 172.21.15.42:0/879723185 --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3bc409f8f0 con 0x7f3ba8006180 2024-09-16T03:50:32.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:32.262+0000 7f3bbbfff640 10 monclient(hunting): start opening mon connection 2024-09-16T03:50:32.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:32.262+0000 7f3bbbfff640 1 -- 172.21.15.42:0/879723185 --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f3bc409c090 con 0x7f3ba80036d0 2024-09-16T03:50:32.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:32.262+0000 7f3bbbfff640 10 monclient(hunting): _renew_subs 2024-09-16T03:50:32.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:32.262+0000 7f3bca40d640 1 --2- 172.21.15.42:0/879723185 >> v2:172.21.15.42:3300/0 conn(0x7f3ba80036d0 0x7f3ba8005b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:50:32.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:32.266+0000 7f3bca40d640 10 monclient(hunting): get_auth_request con 0x7f3ba80036d0 auth_method 0 2024-09-16T03:50:32.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:32.266+0000 7f3bca40d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:50:32.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:32.266+0000 7f3bca40d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:50:32.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:32.266+0000 7f3bca40d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:50:32.271 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:32.266+0000 7f3bca40d640 1 -- 172.21.15.42:0/879723185 >> v2:172.21.15.42:3300/0 conn(0x7f3ba80036d0 msgr2=0x7f3ba8005b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T03:50:32.271 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:32.266+0000 7f3bca40d640 1 -- 172.21.15.42:0/879723185 >> v2:172.21.15.42:3300/0 conn(0x7f3ba80036d0 msgr2=0x7f3ba8005b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:50:32.271 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:32.266+0000 7f3bca40d640 1 --2- 172.21.15.42:0/879723185 >> v2:172.21.15.42:3300/0 conn(0x7f3ba80036d0 0x7f3ba8005b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:50:32.271 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:32.266+0000 7f3bca40d640 1 --2- 172.21.15.42:0/879723185 >> v2:172.21.15.42:3300/0 conn(0x7f3ba80036d0 0x7f3ba8005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:50:32.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:32.470+0000 7f3bca40d640 1 -- 172.21.15.42:0/879723185 >> v2:172.21.15.42:3300/0 conn(0x7f3ba80036d0 msgr2=0x7f3ba8005b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T03:50:32.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:32.470+0000 7f3bca40d640 1 -- 172.21.15.42:0/879723185 >> v2:172.21.15.42:3300/0 conn(0x7f3ba80036d0 msgr2=0x7f3ba8005b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:50:32.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:32.470+0000 7f3bca40d640 1 --2- 172.21.15.42:0/879723185 >> v2:172.21.15.42:3300/0 conn(0x7f3ba80036d0 0x7f3ba8005b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner read peer banner failed r=-1 ((1) Operation not permitted) 2024-09-16T03:50:32.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:32.470+0000 7f3bca40d640 1 --2- 172.21.15.42:0/879723185 >> v2:172.21.15.42:3300/0 conn(0x7f3ba80036d0 0x7f3ba8005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T03:50:32.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:32.874+0000 7f3bca40d640 1 --2- 172.21.15.42:0/879723185 >> v2:172.21.15.42:3300/0 conn(0x7f3ba80036d0 0x7f3ba8005b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:50:32.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:32.874+0000 7f3bca40d640 10 monclient(hunting): get_auth_request con 0x7f3ba80036d0 auth_method 0 2024-09-16T03:50:32.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:32.874+0000 7f3bca40d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:50:32.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:32.874+0000 7f3bca40d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:50:32.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:32.874+0000 7f3bca40d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:50:32.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:32.874+0000 7f3bca40d640 1 -- 172.21.15.42:0/879723185 >> v2:172.21.15.42:3300/0 conn(0x7f3ba80036d0 msgr2=0x7f3ba8005b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T03:50:32.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:32.874+0000 7f3bca40d640 1 -- 172.21.15.42:0/879723185 >> v2:172.21.15.42:3300/0 conn(0x7f3ba80036d0 msgr2=0x7f3ba8005b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:50:32.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:32.874+0000 7f3bca40d640 1 --2- 172.21.15.42:0/879723185 >> v2:172.21.15.42:3300/0 conn(0x7f3ba80036d0 0x7f3ba8005b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:50:32.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:32.874+0000 7f3bca40d640 1 --2- 172.21.15.42:0/879723185 >> v2:172.21.15.42:3300/0 conn(0x7f3ba80036d0 0x7f3ba8005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T03:50:33.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:33.678+0000 7f3bca40d640 1 --2- 172.21.15.42:0/879723185 >> v2:172.21.15.42:3300/0 conn(0x7f3ba80036d0 0x7f3ba8005b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:50:33.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:33.678+0000 7f3bca40d640 10 monclient(hunting): get_auth_request con 0x7f3ba80036d0 auth_method 0 2024-09-16T03:50:33.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:33.678+0000 7f3bca40d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:50:33.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:33.678+0000 7f3bca40d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:50:33.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:33.678+0000 7f3bca40d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:50:33.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:33.678+0000 7f3bca40d640 1 -- 172.21.15.42:0/879723185 >> v2:172.21.15.42:3300/0 conn(0x7f3ba80036d0 msgr2=0x7f3ba8005b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T03:50:33.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:33.678+0000 7f3bca40d640 1 -- 172.21.15.42:0/879723185 >> v2:172.21.15.42:3300/0 conn(0x7f3ba80036d0 msgr2=0x7f3ba8005b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:50:33.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:33.678+0000 7f3bca40d640 1 --2- 172.21.15.42:0/879723185 >> v2:172.21.15.42:3300/0 conn(0x7f3ba80036d0 0x7f3ba8005b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:50:33.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:33.678+0000 7f3bca40d640 1 --2- 172.21.15.42:0/879723185 >> v2:172.21.15.42:3300/0 conn(0x7f3ba80036d0 0x7f3ba8005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T03:50:34.214 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.210+0000 7f3bc940b640 1 -- 172.21.15.42:0/879723185 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2730002135 0 0) 0x7f3bc409f8f0 con 0x7f3ba8006180 2024-09-16T03:50:34.214 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.210+0000 7f3bc940b640 10 monclient(hunting): _negotiate 2024-09-16T03:50:34.214 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.210+0000 7f3bc940b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:50:34.214 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.210+0000 7f3bc940b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:50:34.214 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.210+0000 7f3bc940b640 10 monclient(hunting): my global_id is 764826 2024-09-16T03:50:34.214 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.210+0000 7f3bc940b640 1 -- 172.21.15.42:0/879723185 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3bb0003530 con 0x7f3ba8006180 2024-09-16T03:50:34.221 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.214+0000 7f3bc940b640 1 -- 172.21.15.42:0/879723185 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3677197238 0 0) 0x7f3bb0003530 con 0x7f3ba8006180 2024-09-16T03:50:34.221 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.214+0000 7f3bc940b640 1 -- 172.21.15.42:0/879723185 >> v2:172.21.15.42:3300/0 conn(0x7f3ba80036d0 msgr2=0x7f3ba8005b50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:50:34.221 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.214+0000 7f3bc940b640 1 --2- 172.21.15.42:0/879723185 >> v2:172.21.15.42:3300/0 conn(0x7f3ba80036d0 0x7f3ba8005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:50:34.222 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.214+0000 7f3bc940b640 10 monclient: _finish_hunting 0 2024-09-16T03:50:34.222 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.214+0000 7f3bc940b640 1 monclient: found mon.noname-a-legacy 2024-09-16T03:50:34.222 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.214+0000 7f3bc940b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:50:34.222 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.214+0000 7f3bc940b640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.42:6789/0 2024-09-16T03:50:34.222 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.214+0000 7f3bc940b640 1 -- 172.21.15.42:0/879723185 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ba800af80 con 0x7f3ba8006180 2024-09-16T03:50:34.222 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.214+0000 7f3bc940b640 10 monclient: _finish_auth 0 2024-09-16T03:50:34.222 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.214+0000 7f3bc940b640 10 monclient: _check_auth_tickets 2024-09-16T03:50:34.222 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.214+0000 7f3bc940b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:50:34.222 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.214+0000 7f3bc940b640 1 -- 172.21.15.42:0/879723185 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2177413734 0 0) 0x7f3bb4004350 con 0x7f3ba8006180 2024-09-16T03:50:34.222 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.214+0000 7f3bc940b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:50:34.222 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.214+0000 7f3bc940b640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T03:50:34.222 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.214+0000 7f3bc940b640 10 monclient: dump: 2024-09-16T03:50:34.222 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:50:34.223 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:50:34.223 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:34.223 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:34.223 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:50:34.223 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:50:34.223 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:50:34.223 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:50:34.223 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:50:34.223 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:50:34.223 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.214+0000 7f3bc940b640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] but i'm connected to v1:172.21.15.42:6789/0, reconnecting 2024-09-16T03:50:34.223 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.214+0000 7f3bc940b640 10 monclient: _reopen_session rank -1 2024-09-16T03:50:34.223 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.214+0000 7f3bc940b640 1 -- 172.21.15.42:0/879723185 >> v1:172.21.15.42:6789/0 conn(0x7f3ba8006180 legacy=0x7f3ba8008640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:50:34.223 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.214+0000 7f3bc940b640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:50:34.224 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.214+0000 7f3bcac0e640 1 -- 172.21.15.42:0/879723185 reap_dead start 2024-09-16T03:50:34.224 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.214+0000 7f3bc940b640 1 --2- 172.21.15.42:0/879723185 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3bb00039c0 0x7f3bb0003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:50:34.224 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.214+0000 7f3bc940b640 10 monclient(hunting): picked mon.b con 0x7f3bb00039c0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:50:34.224 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.214+0000 7f3bc940b640 1 --2- 172.21.15.42:0/879723185 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3bb0004810 0x7f3bb0004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:50:34.224 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.214+0000 7f3bc940b640 10 monclient(hunting): picked mon.a con 0x7f3bb0004810 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:50:34.224 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.214+0000 7f3bc940b640 1 --2- 172.21.15.42:0/879723185 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3bb0005220 0x7f3bb000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:50:34.224 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.214+0000 7f3bc940b640 10 monclient(hunting): picked mon.c con 0x7f3bb0005220 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:50:34.224 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.214+0000 7f3bc940b640 10 monclient(hunting): start opening mon connection 2024-09-16T03:50:34.224 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.214+0000 7f3bc940b640 1 -- 172.21.15.42:0/879723185 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f3bc409c090 con 0x7f3bb0004810 2024-09-16T03:50:34.224 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.214+0000 7f3bc940b640 10 monclient(hunting): start opening mon connection 2024-09-16T03:50:34.224 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.214+0000 7f3bc940b640 1 -- 172.21.15.42:0/879723185 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f3bb000c150 con 0x7f3bb00039c0 2024-09-16T03:50:34.224 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.214+0000 7f3bc940b640 10 monclient(hunting): start opening mon connection 2024-09-16T03:50:34.224 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.214+0000 7f3bc940b640 1 -- 172.21.15.42:0/879723185 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f3bb000c350 con 0x7f3bb0005220 2024-09-16T03:50:34.224 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.214+0000 7f3bc940b640 10 monclient(hunting): _renew_subs 2024-09-16T03:50:34.225 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.214+0000 7f3bc940b640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:50:34.225 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.214+0000 7f3bcac0e640 1 -- 172.21.15.42:0/879723185 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3bb0005220 msgr2=0x7f3bb000b9d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T03:50:34.225 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.214+0000 7f3bc9c0c640 1 --2- 172.21.15.42:0/879723185 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3bb0004810 0x7f3bb0004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:50:34.225 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.214+0000 7f3bcac0e640 1 --2- 172.21.15.42:0/879723185 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3bb0005220 0x7f3bb000b9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:50:34.225 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.214+0000 7f3bc9c0c640 10 monclient(hunting): get_auth_request con 0x7f3bb0004810 auth_method 0 2024-09-16T03:50:34.225 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.214+0000 7f3bca40d640 1 --2- 172.21.15.42:0/879723185 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3bb00039c0 0x7f3bb0003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:50:34.225 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.214+0000 7f3bc9c0c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:50:34.225 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.214+0000 7f3bc9c0c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:50:34.225 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.214+0000 7f3bc9c0c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:50:34.225 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.214+0000 7f3bca40d640 10 monclient(hunting): get_auth_request con 0x7f3bb00039c0 auth_method 0 2024-09-16T03:50:34.225 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.214+0000 7f3bca40d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:50:34.225 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.214+0000 7f3bca40d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:50:34.225 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.214+0000 7f3bca40d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:50:34.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.214+0000 7f3bc9c0c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:50:34.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.214+0000 7f3bc9c0c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:50:34.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.218+0000 7f3bc9c0c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T03:50:34.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.218+0000 7f3bca40d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:50:34.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.218+0000 7f3bca40d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:50:34.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.218+0000 7f3bca40d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T03:50:34.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.218+0000 7f3bc9c0c640 10 monclient(hunting): handle_auth_done global_id 764826 payload 293 2024-09-16T03:50:34.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.218+0000 7f3bc9c0c640 1 -- 172.21.15.42:0/879723185 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3bb0005220 msgr2=0x7f3bb000b9d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:50:34.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.218+0000 7f3bc9c0c640 1 --2- 172.21.15.42:0/879723185 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3bb0005220 0x7f3bb000b9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:50:34.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.218+0000 7f3bc9c0c640 1 -- 172.21.15.42:0/879723185 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3bb00039c0 msgr2=0x7f3bb0003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:50:34.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.218+0000 7f3bc9c0c640 1 --2- 172.21.15.42:0/879723185 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3bb00039c0 0x7f3bb0003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:50:34.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.218+0000 7f3bc9c0c640 10 monclient: _finish_hunting 0 2024-09-16T03:50:34.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.218+0000 7f3bc9c0c640 1 monclient: found mon.a 2024-09-16T03:50:34.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.218+0000 7f3bc9c0c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:50:34.227 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.218+0000 7f3bc9c0c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:50:34.227 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.218+0000 7f3bc9c0c640 1 -- 172.21.15.42:0/879723185 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3bb0003530 con 0x7f3bb0004810 2024-09-16T03:50:34.227 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.218+0000 7f3bc9c0c640 10 monclient: _finish_auth 0 2024-09-16T03:50:34.227 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.218+0000 7f3bc9c0c640 10 monclient: _check_auth_tickets 2024-09-16T03:50:34.227 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.218+0000 7f3bc9c0c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:50:34.227 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.218+0000 7f3bcc671640 5 monclient: authenticate success, global_id 764826 2024-09-16T03:50:34.227 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.218+0000 7f3bcc671640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:50:34.227 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.218+0000 7f3bca40d640 1 --2- 172.21.15.42:0/879723185 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3bb00039c0 0x7f3bb0003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:50:34.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.222+0000 7f3bc9c0c640 1 --2- 172.21.15.42:0/879723185 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3bb0004810 0x7f3bb0004c90 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f3bb4003990 tx=0x7f3bb401cd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c1ab4b88a346e60 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:50:34.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.222+0000 7f3bc940b640 1 -- 172.21.15.42:0/879723185 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bb400c0e0 con 0x7f3bb0004810 2024-09-16T03:50:34.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.222+0000 7f3bc940b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:50:34.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.222+0000 7f3bc940b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:50:34.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.222+0000 7f3bc940b640 10 monclient: dump: 2024-09-16T03:50:34.228 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:50:34.228 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:50:34.228 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:34.228 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:34.228 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:50:34.229 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:50:34.229 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:50:34.229 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:50:34.229 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:50:34.229 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:50:34.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.222+0000 7f3bc940b640 1 -- 172.21.15.42:0/879723185 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3bb40a9070 con 0x7f3bb0004810 2024-09-16T03:50:34.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.222+0000 7f3bcc671640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:50:34.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.222+0000 7f3bc940b640 10 monclient: handle_config config(0 keys) 2024-09-16T03:50:34.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.222+0000 7f3bcc671640 10 monclient: get_monmap_and_config success 2024-09-16T03:50:34.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.222+0000 7f3bc940b640 1 -- 172.21.15.42:0/879723185 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bb40a5940 con 0x7f3bb0004810 2024-09-16T03:50:34.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.222+0000 7f3bcc671640 4 set_mon_vals no callback set 2024-09-16T03:50:34.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.222+0000 7f3bc940b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:50:34.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.222+0000 7f3bc940b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:50:34.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.222+0000 7f3bc940b640 10 monclient: dump: 2024-09-16T03:50:34.230 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:50:34.230 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:50:34.230 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:34.230 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:34.230 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:50:34.230 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:50:34.230 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:50:34.230 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:50:34.230 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:50:34.230 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:50:34.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.222+0000 7f3bcc671640 10 monclient: shutdown 2024-09-16T03:50:34.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.222+0000 7f3bcc671640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:50:34.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.222+0000 7f3bcc671640 1 -- 172.21.15.42:0/879723185 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3bb0004810 msgr2=0x7f3bb0004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:50:34.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.222+0000 7f3bcc671640 1 --2- 172.21.15.42:0/879723185 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3bb0004810 0x7f3bb0004c90 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f3bb4003990 tx=0x7f3bb401cd30 comp rx=0 tx=0).stop 2024-09-16T03:50:34.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.226+0000 7f3bcc671640 1 -- 172.21.15.42:0/879723185 shutdown_connections 2024-09-16T03:50:34.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.226+0000 7f3bcc671640 1 --2- 172.21.15.42:0/879723185 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3bb0005220 0x7f3bb000b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:50:34.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.226+0000 7f3bcc671640 1 --2- 172.21.15.42:0/879723185 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3bb00039c0 0x7f3bb0003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:50:34.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.226+0000 7f3bcc671640 1 --2- 172.21.15.42:0/879723185 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3bb0004810 0x7f3bb0004c90 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:50:34.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.226+0000 7f3bcc671640 1 -- 172.21.15.42:0/879723185 >> 172.21.15.42:0/879723185 conn(0x7f3bc40a2420 msgr2=0x7f3bc40a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:50:34.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.226+0000 7f3bcc671640 1 -- 172.21.15.42:0/879723185 shutdown_connections 2024-09-16T03:50:34.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.226+0000 7f3bcc671640 1 -- 172.21.15.42:0/879723185 wait complete. 2024-09-16T03:50:34.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.226+0000 7f3bcc671640 10 monclient: build_initial_monmap 2024-09-16T03:50:34.233 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.226+0000 7f3bcc671640 10 monclient: monmap: 2024-09-16T03:50:34.233 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:50:34.233 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:50:34.233 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:50:34.233 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:50:34.233 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:50:34.233 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:50:34.233 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:50:34.233 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:50:34.233 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:50:34.233 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:50:34.233 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.226+0000 7f3bcc671640 1 librados: starting msgr at 2024-09-16T03:50:34.233 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.226+0000 7f3bcc671640 1 librados: starting objecter 2024-09-16T03:50:34.233 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.226+0000 7f3bcc671640 1 Processor -- start 2024-09-16T03:50:34.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.226+0000 7f3bcc671640 1 -- start start 2024-09-16T03:50:34.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.226+0000 7f3bcc671640 1 librados: setting wanted keys 2024-09-16T03:50:34.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.226+0000 7f3bcc671640 1 librados: calling monclient init 2024-09-16T03:50:34.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.226+0000 7f3bcc671640 10 monclient: init 2024-09-16T03:50:34.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.226+0000 7f3bcc671640 10 monclient: _reopen_session rank -1 2024-09-16T03:50:34.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.230+0000 7f3bcc671640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:50:34.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.230+0000 7f3bcc671640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3bb00039c0 0x7f3bc41cd0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:50:34.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.230+0000 7f3bcc671640 10 monclient(hunting): picked mon.noname-c con 0x7f3bb00039c0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:50:34.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.230+0000 7f3bcc671640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3bb0004810 0x7f3bc41cd600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:50:34.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.230+0000 7f3bcc671640 10 monclient(hunting): picked mon.noname-a con 0x7f3bb0004810 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:50:34.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.230+0000 7f3bcc671640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3bb0005220 0x7f3bc411f630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:50:34.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.230+0000 7f3bcc671640 10 monclient(hunting): picked mon.noname-b con 0x7f3bb0005220 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:50:34.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.230+0000 7f3bcc671640 10 monclient(hunting): start opening mon connection 2024-09-16T03:50:34.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.230+0000 7f3bcc671640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f3bc409eef0 con 0x7f3bb0004810 2024-09-16T03:50:34.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.230+0000 7f3bcc671640 10 monclient(hunting): start opening mon connection 2024-09-16T03:50:34.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.230+0000 7f3bca40d640 1 -- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3bb00039c0 msgr2=0x7f3bc41cd0c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T03:50:34.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.230+0000 7f3bcc671640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f3bc409ed70 con 0x7f3bb0005220 2024-09-16T03:50:34.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.230+0000 7f3bcc671640 10 monclient(hunting): start opening mon connection 2024-09-16T03:50:34.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.230+0000 7f3bcc671640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f3bc4135910 con 0x7f3bb00039c0 2024-09-16T03:50:34.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.230+0000 7f3bca40d640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3bb00039c0 0x7f3bc41cd0c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:50:34.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.230+0000 7f3bcc671640 10 monclient(hunting): _renew_subs 2024-09-16T03:50:34.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.230+0000 7f3bcc671640 10 monclient(hunting): authenticate will time out at 1481.200236s 2024-09-16T03:50:34.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.230+0000 7f3bc9c0c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3bb0004810 0x7f3bc41cd600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:50:34.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.230+0000 7f3bc9c0c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3bb0004810 0x7f3bc41cd600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:38672/0 (socket says 172.21.15.42:38672) 2024-09-16T03:50:34.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.230+0000 7f3bc9c0c640 1 -- 172.21.15.42:0/4056082298 learned_addr learned my addr 172.21.15.42:0/4056082298 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:50:34.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.230+0000 7f3bc9c0c640 10 monclient(hunting): get_auth_request con 0x7f3bb0004810 auth_method 0 2024-09-16T03:50:34.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.230+0000 7f3bc9c0c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:50:34.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.230+0000 7f3bc9c0c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:50:34.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.230+0000 7f3bc9c0c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:50:34.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.230+0000 7f3bc9c0c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:50:34.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.230+0000 7f3bc9c0c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:50:34.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.230+0000 7f3bcac0e640 1 --2- 172.21.15.42:0/4056082298 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3bb0005220 0x7f3bc411f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:50:34.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.230+0000 7f3bc9c0c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:50:34.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.230+0000 7f3bcac0e640 10 monclient(hunting): get_auth_request con 0x7f3bb0005220 auth_method 0 2024-09-16T03:50:34.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.230+0000 7f3bcac0e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:50:34.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.230+0000 7f3bcac0e640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:50:34.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.230+0000 7f3bcac0e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:50:34.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.230+0000 7f3bc9c0c640 10 monclient(hunting): handle_auth_done global_id 764829 payload 824 2024-09-16T03:50:34.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.230+0000 7f3bc9c0c640 1 -- 172.21.15.42:0/4056082298 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3bb00039c0 msgr2=0x7f3bc41cd0c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:50:34.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.230+0000 7f3bc9c0c640 1 --2- 172.21.15.42:0/4056082298 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3bb00039c0 0x7f3bc41cd0c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:50:34.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.230+0000 7f3bc9c0c640 1 -- 172.21.15.42:0/4056082298 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3bb0005220 msgr2=0x7f3bc411f630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:50:34.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.230+0000 7f3bc9c0c640 1 --2- 172.21.15.42:0/4056082298 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3bb0005220 0x7f3bc411f630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:50:34.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.230+0000 7f3bc9c0c640 10 monclient: _finish_hunting 0 2024-09-16T03:50:34.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.230+0000 7f3bc9c0c640 1 monclient: found mon.noname-a 2024-09-16T03:50:34.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.230+0000 7f3bc9c0c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:50:34.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.230+0000 7f3bc9c0c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:50:34.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.230+0000 7f3bc9c0c640 1 -- 172.21.15.42:0/4056082298 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3bc4121bb0 con 0x7f3bb0004810 2024-09-16T03:50:34.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.230+0000 7f3bcac0e640 1 --2- 172.21.15.42:0/4056082298 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3bb0005220 0x7f3bc411f630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T03:50:34.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.230+0000 7f3bc9c0c640 1 --2- 172.21.15.42:0/4056082298 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3bb0004810 0x7f3bc41cd600 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f3bb401c7a0 tx=0x7f3bb0005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:50:34.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.230+0000 7f3bbaffd640 10 client.?.objecter ms_handle_connect 0x7f3bb0004810 2024-09-16T03:50:34.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.230+0000 7f3bbaffd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:50:34.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.230+0000 7f3bbaffd640 1 -- 172.21.15.42:0/4056082298 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bb40bc070 con 0x7f3bb0004810 2024-09-16T03:50:34.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.230+0000 7f3bbaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:50:34.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.230+0000 7f3bbaffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:50:34.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.230+0000 7f3bbaffd640 10 monclient: dump: 2024-09-16T03:50:34.245 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:50:34.245 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:50:34.245 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:34.245 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:34.245 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:50:34.246 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:50:34.246 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:50:34.246 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:50:34.246 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:50:34.246 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:50:34.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.230+0000 7f3bbaffd640 10 monclient: _finish_auth 0 2024-09-16T03:50:34.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.230+0000 7f3bbaffd640 10 monclient: _check_auth_tickets 2024-09-16T03:50:34.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.230+0000 7f3bbaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:50:34.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.230+0000 7f3bcc671640 5 monclient: authenticate success, global_id 764829 2024-09-16T03:50:34.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.230+0000 7f3bcc671640 10 monclient: _renew_subs 2024-09-16T03:50:34.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.230+0000 7f3bbaffd640 1 -- 172.21.15.42:0/4056082298 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3bb4003970 con 0x7f3bb0004810 2024-09-16T03:50:34.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.230+0000 7f3bcc671640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:50:34.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.230+0000 7f3bcc671640 1 -- 172.21.15.42:0/4056082298 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3bc41235c0 con 0x7f3bb0004810 2024-09-16T03:50:34.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.230+0000 7f3bbaffd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:50:34.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.230+0000 7f3bbaffd640 1 -- 172.21.15.42:0/4056082298 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bb40a5b80 con 0x7f3bb0004810 2024-09-16T03:50:34.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.230+0000 7f3bbaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:50:34.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.230+0000 7f3bbaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:50:34.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.230+0000 7f3bbaffd640 10 monclient: dump: 2024-09-16T03:50:34.247 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:50:34.247 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:50:34.247 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:34.247 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:34.247 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:50:34.247 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:50:34.247 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:50:34.247 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:50:34.247 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:50:34.247 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:50:34.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.230+0000 7f3bbbfff640 4 set_mon_vals no callback set 2024-09-16T03:50:34.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.230+0000 7f3bcc671640 10 client.764829.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:50:34.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.230+0000 7f3bcc671640 10 monclient: _renew_subs 2024-09-16T03:50:34.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.230+0000 7f3bcc671640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:50:34.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.230+0000 7f3bcc671640 1 -- 172.21.15.42:0/4056082298 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3bc406df40 con 0x7f3bb0004810 2024-09-16T03:50:34.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.230+0000 7f3bcc671640 1 librados: init done 2024-09-16T03:50:34.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.230+0000 7f3bcc671640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:50:34.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.230+0000 7f3bcc671640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:50:34.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.230+0000 7f3bcc671640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:50:34.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.230+0000 7f3bbaffd640 1 -- 172.21.15.42:0/4056082298 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f3bb40ae460 con 0x7f3bb0004810 2024-09-16T03:50:34.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.230+0000 7f3bcc671640 1 -- 172.21.15.42:0/4056082298 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3bc4073a30 con 0x7f3bb0004810 2024-09-16T03:50:34.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.230+0000 7f3bbaffd640 1 --2- 172.21.15.42:0/4056082298 >> v2:172.21.15.154:6812/254845342 conn(0x7f3ba8038000 0x7f3ba803a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:50:34.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.230+0000 7f3bbaffd640 1 -- 172.21.15.42:0/4056082298 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(281..281 src has 1..281) ==== 6024+0+0 (secure 0 0 0) 0x7f3bb40e2550 con 0x7f3bb0004810 2024-09-16T03:50:34.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.230+0000 7f3bbaffd640 10 client.764829.objecter ms_dispatch 0x7f3bc4000ba0 osd_map(281..281 src has 1..281) 2024-09-16T03:50:34.249 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.230+0000 7f3bbaffd640 3 client.764829.objecter handle_osd_map got epochs [281,281] > 0 2024-09-16T03:50:34.249 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.230+0000 7f3bbaffd640 3 client.764829.objecter handle_osd_map decoding full epoch 281 2024-09-16T03:50:34.249 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.230+0000 7f3bbaffd640 20 client.764829.objecter dump_active .. 0 homeless 2024-09-16T03:50:34.249 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.230+0000 7f3bca40d640 1 --2- 172.21.15.42:0/4056082298 >> v2:172.21.15.154:6812/254845342 conn(0x7f3ba8038000 0x7f3ba803a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:50:34.249 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.230+0000 7f3bca40d640 10 monclient: get_auth_request con 0x7f3ba8038000 auth_method 0 2024-09-16T03:50:34.249 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.234+0000 7f3bca40d640 1 --2- 172.21.15.42:0/4056082298 >> v2:172.21.15.154:6812/254845342 conn(0x7f3ba8038000 0x7f3ba803a4c0 secure :-1 s=READY pgs=3543 cs=0 l=1 rev1=1 crypto rx=0x7f3bc0007d30 tx=0x7f3bc0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:50:34.249 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.234+0000 7f3bbaffd640 10 client.764829.objecter ms_handle_connect 0x7f3ba8038000 2024-09-16T03:50:34.249 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.238+0000 7f3bbaffd640 1 -- 172.21.15.42:0/4056082298 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f3bb4016a70 con 0x7f3bb0004810 2024-09-16T03:50:34.249 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.238+0000 7f3bbaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:50:34.250 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.238+0000 7f3bbaffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:50:34.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.466+0000 7f3bcc671640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:50:34.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.466+0000 7f3bcc671640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:50:34.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.466+0000 7f3bcc671640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:50:34.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.466+0000 7f3bcc671640 1 -- 172.21.15.42:0/4056082298 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f3bc409f610 con 0x7f3bb0004810 2024-09-16T03:50:34.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.470+0000 7f3bbaffd640 1 -- 172.21.15.42:0/4056082298 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1554 (secure 0 0 0) 0x7f3bb40e0a50 con 0x7f3bb0004810 2024-09-16T03:50:34.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.470+0000 7f3bbaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:50:34.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.470+0000 7f3bbaffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:50:34.474 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":76,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:50:34.477 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.470+0000 7f3bcc671640 10 librados: watch_flush enter 2024-09-16T03:50:34.477 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.470+0000 7f3bcc671640 10 librados: watch_flush exit 2024-09-16T03:50:34.477 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.470+0000 7f3bcc671640 20 client.764829.objecter shutdown clearing up homeless session... 2024-09-16T03:50:34.477 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.470+0000 7f3bcc671640 10 client.764829.objecter successfully canceled tick 2024-09-16T03:50:34.477 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.470+0000 7f3bcc671640 1 -- 172.21.15.42:0/4056082298 >> v2:172.21.15.154:6812/254845342 conn(0x7f3ba8038000 msgr2=0x7f3ba803a4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:50:34.477 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.470+0000 7f3bcc671640 1 --2- 172.21.15.42:0/4056082298 >> v2:172.21.15.154:6812/254845342 conn(0x7f3ba8038000 0x7f3ba803a4c0 secure :-1 s=READY pgs=3543 cs=0 l=1 rev1=1 crypto rx=0x7f3bc0007d30 tx=0x7f3bc0009040 comp rx=0 tx=0).stop 2024-09-16T03:50:34.477 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.470+0000 7f3bcc671640 10 monclient: shutdown 2024-09-16T03:50:34.477 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.470+0000 7f3bcc671640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:50:34.477 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.470+0000 7f3bcc671640 1 -- 172.21.15.42:0/4056082298 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3bb0004810 msgr2=0x7f3bc41cd600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:50:34.477 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.470+0000 7f3bcc671640 1 --2- 172.21.15.42:0/4056082298 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3bb0004810 0x7f3bc41cd600 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f3bb401c7a0 tx=0x7f3bb0005840 comp rx=0 tx=0).stop 2024-09-16T03:50:34.477 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.470+0000 7f3bcc671640 1 -- 172.21.15.42:0/4056082298 shutdown_connections 2024-09-16T03:50:34.477 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.470+0000 7f3bcc671640 1 --2- 172.21.15.42:0/4056082298 >> v2:172.21.15.154:6812/254845342 conn(0x7f3ba8038000 0x7f3ba803a4c0 unknown :-1 s=CLOSED pgs=3543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:50:34.477 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.470+0000 7f3bcc671640 1 --2- 172.21.15.42:0/4056082298 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3bb0005220 0x7f3bc411f630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:50:34.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.470+0000 7f3bcc671640 1 --2- 172.21.15.42:0/4056082298 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3bb0004810 0x7f3bc41cd600 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:50:34.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.470+0000 7f3bcc671640 1 --2- 172.21.15.42:0/4056082298 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3bb00039c0 0x7f3bc41cd0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:50:34.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.470+0000 7f3bcc671640 1 -- 172.21.15.42:0/4056082298 >> 172.21.15.42:0/4056082298 conn(0x7f3bc40a2420 msgr2=0x7f3bc40a46b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:50:34.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.470+0000 7f3bcc671640 1 -- 172.21.15.42:0/4056082298 shutdown_connections 2024-09-16T03:50:34.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.470+0000 7f3bcc671640 1 -- 172.21.15.42:0/4056082298 wait complete. 2024-09-16T03:50:34.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.470+0000 7f3bcc671640 1 librados: shutdown 2024-09-16T03:50:34.494 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T03:50:34.494 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-16T03:50:34.494 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T03:50:34.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.610+0000 7fefc121a640 10 monclient: get_monmap_and_config 2024-09-16T03:50:34.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.610+0000 7fefc121a640 10 monclient: build_initial_monmap 2024-09-16T03:50:34.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.610+0000 7fefc121a640 10 monclient: monmap: 2024-09-16T03:50:34.614 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:50:34.614 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:50:34.614 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:50:34.613014+0000 2024-09-16T03:50:34.614 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:50:34.613014+0000 2024-09-16T03:50:34.614 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:50:34.614 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:50:34.614 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:50:34.615 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:50:34.615 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:50:34.615 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:50:34.615 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:50:34.615 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:50:34.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.610+0000 7fefc121a640 1 Processor -- start 2024-09-16T03:50:34.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.610+0000 7fefc121a640 1 -- start start 2024-09-16T03:50:34.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.610+0000 7fefc121a640 10 monclient: init 2024-09-16T03:50:34.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.610+0000 7fefc121a640 10 monclient: _reopen_session rank -1 2024-09-16T03:50:34.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.610+0000 7fefc121a640 10 monclient: _add_conns ranks=[4,0,3,1,2] 2024-09-16T03:50:34.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.610+0000 7fefc121a640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fefbc06db90 0x7fefbc1340c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:50:34.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.610+0000 7fefc121a640 10 monclient(hunting): picked mon.noname-c con 0x7fefbc06db90 addr v2:172.21.15.154:3300/0 2024-09-16T03:50:34.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.610+0000 7fefc121a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fefbc0a37f0 addr v1:172.21.15.42:6789/0 2024-09-16T03:50:34.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.610+0000 7fefc121a640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fefbc0a3c00 0x7fefbc137520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:50:34.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.610+0000 7fefc121a640 10 monclient(hunting): picked mon.noname-a con 0x7fefbc0a3c00 addr v2:172.21.15.42:3300/0 2024-09-16T03:50:34.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.610+0000 7fefc121a640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fefbc06ece0 con 0x7fefbc0a37f0 2024-09-16T03:50:34.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.610+0000 7fefc121a640 10 monclient(hunting): start opening mon connection 2024-09-16T03:50:34.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.610+0000 7fefc121a640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7fefbc06df70 con 0x7fefbc0a3c00 2024-09-16T03:50:34.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.610+0000 7fefc121a640 10 monclient(hunting): start opening mon connection 2024-09-16T03:50:34.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.610+0000 7fefc121a640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fefbc09bf80 con 0x7fefbc06db90 2024-09-16T03:50:34.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.610+0000 7fefc121a640 10 monclient(hunting): _renew_subs 2024-09-16T03:50:34.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.610+0000 7fefc121a640 10 monclient(hunting): authenticate will time out at 1481.583101s 2024-09-16T03:50:34.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.610+0000 7fefbb59e640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fefbc0a3c00 0x7fefbc137520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:50:34.619 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.610+0000 7fefbb59e640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fefbc0a3c00 0x7fefbc137520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:38680/0 (socket says 172.21.15.42:38680) 2024-09-16T03:50:34.619 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.610+0000 7fefb3fff640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fefbc0a37f0 0x7fefbc136e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:46152/0 (socket says 172.21.15.42:46152) 2024-09-16T03:50:34.619 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.610+0000 7fefbb59e640 1 -- 172.21.15.42:0/2531538913 learned_addr learned my addr 172.21.15.42:0/2531538913 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:50:34.619 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.610+0000 7fefbb59e640 10 monclient(hunting): get_auth_request con 0x7fefbc0a3c00 auth_method 0 2024-09-16T03:50:34.619 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.610+0000 7fefbb59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:50:34.619 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.610+0000 7fefbb59e640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:50:34.619 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.610+0000 7fefbb59e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:50:34.619 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.610+0000 7fefbb59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:50:34.619 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.610+0000 7fefbb59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:50:34.619 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.610+0000 7fefbb59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:50:34.619 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.610+0000 7fefbad9d640 1 --2- 172.21.15.42:0/2531538913 >> v2:172.21.15.154:3300/0 conn(0x7fefbc06db90 0x7fefbc1340c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:50:34.619 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.610+0000 7fefbad9d640 10 monclient(hunting): get_auth_request con 0x7fefbc06db90 auth_method 0 2024-09-16T03:50:34.619 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.610+0000 7fefbad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:50:34.619 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.610+0000 7fefbad9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:50:34.620 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.610+0000 7fefbad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:50:34.620 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.610+0000 7fefbb59e640 10 monclient(hunting): handle_auth_done global_id 764832 payload 274 2024-09-16T03:50:34.620 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.610+0000 7fefbb59e640 1 -- 172.21.15.42:0/2531538913 >> v2:172.21.15.154:3300/0 conn(0x7fefbc06db90 msgr2=0x7fefbc1340c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:50:34.620 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.610+0000 7fefbb59e640 1 --2- 172.21.15.42:0/2531538913 >> v2:172.21.15.154:3300/0 conn(0x7fefbc06db90 0x7fefbc1340c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:50:34.620 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.610+0000 7fefbb59e640 1 -- 172.21.15.42:0/2531538913 >> v1:172.21.15.42:6789/0 conn(0x7fefbc0a37f0 legacy=0x7fefbc136e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:50:34.620 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.610+0000 7fefbb59e640 10 monclient: _finish_hunting 0 2024-09-16T03:50:34.620 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.610+0000 7fefbb59e640 1 monclient: found mon.noname-a 2024-09-16T03:50:34.620 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.610+0000 7fefbb59e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:50:34.620 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.610+0000 7fefbb59e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:50:34.620 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.610+0000 7fefbb59e640 1 -- 172.21.15.42:0/2531538913 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fefbc134780 con 0x7fefbc0a3c00 2024-09-16T03:50:34.620 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.610+0000 7fefbad9d640 1 --2- 172.21.15.42:0/2531538913 >> v2:172.21.15.154:3300/0 conn(0x7fefbc06db90 0x7fefbc1340c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T03:50:34.627 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.614+0000 7fefbb59e640 1 --2- 172.21.15.42:0/2531538913 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fefbc0a3c00 0x7fefbc137520 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fefbc0a76f0 tx=0x7fefb401c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8cbc8227254f057 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:50:34.627 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.614+0000 7fefba59c640 1 -- 172.21.15.42:0/2531538913 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefb40a4ad0 con 0x7fefbc0a3c00 2024-09-16T03:50:34.627 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.614+0000 7fefba59c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:50:34.627 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.614+0000 7fefba59c640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:50:34.627 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.614+0000 7fefba59c640 10 monclient: dump: 2024-09-16T03:50:34.627 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:50:34.627 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:50:34.627 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:34.627 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:34.627 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:50:34.628 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:50:34.628 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:50:34.628 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:50:34.628 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:50:34.628 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:50:34.628 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.614+0000 7fefba59c640 10 monclient: _finish_auth 0 2024-09-16T03:50:34.628 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.614+0000 7fefba59c640 10 monclient: _check_auth_tickets 2024-09-16T03:50:34.628 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.614+0000 7fefba59c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:50:34.628 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.614+0000 7fefba59c640 1 -- 172.21.15.42:0/2531538913 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fefbc137c70 con 0x7fefbc0a3c00 2024-09-16T03:50:34.628 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.614+0000 7fefba59c640 10 monclient: handle_config config(0 keys) 2024-09-16T03:50:34.628 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.614+0000 7fefba59c640 1 -- 172.21.15.42:0/2531538913 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefb40ac3f0 con 0x7fefbc0a3c00 2024-09-16T03:50:34.628 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.614+0000 7fefc121a640 5 monclient: authenticate success, global_id 764832 2024-09-16T03:50:34.628 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.614+0000 7fefc121a640 10 monclient: get_monmap_and_config success 2024-09-16T03:50:34.628 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.614+0000 7fefc121a640 4 set_mon_vals no callback set 2024-09-16T03:50:34.629 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.614+0000 7fefba59c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:50:34.629 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.614+0000 7fefba59c640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:50:34.629 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.614+0000 7fefba59c640 10 monclient: dump: 2024-09-16T03:50:34.629 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:50:34.629 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:50:34.629 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:34.629 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:34.629 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:50:34.629 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:50:34.629 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:50:34.629 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:50:34.629 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:50:34.629 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:50:34.629 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.614+0000 7fefc121a640 10 monclient: shutdown 2024-09-16T03:50:34.630 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.614+0000 7fefc121a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:50:34.630 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.614+0000 7fefc121a640 1 -- 172.21.15.42:0/2531538913 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fefbc0a3c00 msgr2=0x7fefbc137520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:50:34.630 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.614+0000 7fefc121a640 1 --2- 172.21.15.42:0/2531538913 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fefbc0a3c00 0x7fefbc137520 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fefbc0a76f0 tx=0x7fefb401c480 comp rx=0 tx=0).stop 2024-09-16T03:50:34.630 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.614+0000 7fefc121a640 1 -- 172.21.15.42:0/2531538913 shutdown_connections 2024-09-16T03:50:34.630 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.614+0000 7fefc121a640 1 --2- 172.21.15.42:0/2531538913 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fefbc0a3c00 0x7fefbc137520 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:50:34.630 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.614+0000 7fefc121a640 1 --2- 172.21.15.42:0/2531538913 >> v2:172.21.15.154:3300/0 conn(0x7fefbc06db90 0x7fefbc1340c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:50:34.630 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.614+0000 7fefc121a640 1 -- 172.21.15.42:0/2531538913 >> 172.21.15.42:0/2531538913 conn(0x7fefbc0a9fb0 msgr2=0x7fefbc0aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:50:34.630 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.614+0000 7fefc121a640 1 -- 172.21.15.42:0/2531538913 shutdown_connections 2024-09-16T03:50:34.630 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.614+0000 7fefc121a640 1 -- 172.21.15.42:0/2531538913 wait complete. 2024-09-16T03:50:34.630 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.618+0000 7fefc121a640 10 monclient: build_initial_monmap 2024-09-16T03:50:34.630 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.618+0000 7fefc121a640 10 monclient: monmap: 2024-09-16T03:50:34.630 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:50:34.631 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:50:34.631 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:50:34.631 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:50:34.631 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:50:34.631 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:50:34.631 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:50:34.631 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:50:34.631 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:50:34.631 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:50:34.631 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.618+0000 7fefc121a640 1 librados: starting msgr at 2024-09-16T03:50:34.631 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.618+0000 7fefc121a640 1 librados: starting objecter 2024-09-16T03:50:34.631 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.618+0000 7fefc121a640 1 Processor -- start 2024-09-16T03:50:34.631 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.618+0000 7fefc121a640 1 -- start start 2024-09-16T03:50:34.631 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.618+0000 7fefc121a640 1 librados: setting wanted keys 2024-09-16T03:50:34.632 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.618+0000 7fefc121a640 1 librados: calling monclient init 2024-09-16T03:50:34.632 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.618+0000 7fefc121a640 10 monclient: init 2024-09-16T03:50:34.632 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.618+0000 7fefc121a640 10 monclient: _reopen_session rank -1 2024-09-16T03:50:34.632 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.618+0000 7fefc121a640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:50:34.632 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.618+0000 7fefc121a640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fefbc06db90 0x7fefbc1cfb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:50:34.632 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.618+0000 7fefc121a640 10 monclient(hunting): picked mon.noname-c con 0x7fefbc06db90 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:50:34.632 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.618+0000 7fefc121a640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fefbc0a37f0 0x7fefbc13e360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:50:34.632 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.618+0000 7fefc121a640 10 monclient(hunting): picked mon.noname-b con 0x7fefbc0a37f0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:50:34.632 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.618+0000 7fefc121a640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fefbc0a3c00 0x7fefbc13e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:50:34.632 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.618+0000 7fefc121a640 10 monclient(hunting): picked mon.noname-a con 0x7fefbc0a3c00 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:50:34.632 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.618+0000 7fefc121a640 10 monclient(hunting): start opening mon connection 2024-09-16T03:50:34.632 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.618+0000 7fefc121a640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fefbc1425e0 con 0x7fefbc0a3c00 2024-09-16T03:50:34.632 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.618+0000 7fefc121a640 10 monclient(hunting): start opening mon connection 2024-09-16T03:50:34.633 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.618+0000 7fefc121a640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fefbc142460 con 0x7fefbc0a37f0 2024-09-16T03:50:34.633 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.618+0000 7fefc121a640 10 monclient(hunting): start opening mon connection 2024-09-16T03:50:34.633 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.618+0000 7fefc121a640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fefbc142760 con 0x7fefbc06db90 2024-09-16T03:50:34.633 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.618+0000 7fefc121a640 10 monclient(hunting): _renew_subs 2024-09-16T03:50:34.633 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.618+0000 7fefc121a640 10 monclient(hunting): authenticate will time out at 1481.589229s 2024-09-16T03:50:34.633 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.618+0000 7fefbad9d640 1 -- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fefbc06db90 msgr2=0x7fefbc1cfb60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T03:50:34.633 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.618+0000 7fefbad9d640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fefbc06db90 0x7fefbc1cfb60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:50:34.633 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.618+0000 7fefbb59e640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fefbc0a3c00 0x7fefbc13e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:50:34.633 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.618+0000 7fefbb59e640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fefbc0a3c00 0x7fefbc13e8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:38684/0 (socket says 172.21.15.42:38684) 2024-09-16T03:50:34.633 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.618+0000 7fefbb59e640 1 -- 172.21.15.42:0/3639921756 learned_addr learned my addr 172.21.15.42:0/3639921756 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:50:34.633 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.618+0000 7fefbb59e640 10 monclient(hunting): get_auth_request con 0x7fefbc0a3c00 auth_method 0 2024-09-16T03:50:34.633 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.618+0000 7fefbb59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:50:34.634 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.618+0000 7fefbb59e640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:50:34.634 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.618+0000 7fefbb59e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:50:34.634 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.618+0000 7fefbb59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:50:34.634 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.618+0000 7fefbb59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:50:34.634 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.618+0000 7fefbb59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:50:34.634 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.618+0000 7fefb3fff640 1 --2- 172.21.15.42:0/3639921756 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fefbc0a37f0 0x7fefbc13e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:50:34.634 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.618+0000 7fefbb59e640 10 monclient(hunting): handle_auth_done global_id 764838 payload 824 2024-09-16T03:50:34.634 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.618+0000 7fefbb59e640 1 -- 172.21.15.42:0/3639921756 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fefbc06db90 msgr2=0x7fefbc1cfb60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:50:34.634 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.618+0000 7fefbb59e640 1 --2- 172.21.15.42:0/3639921756 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fefbc06db90 0x7fefbc1cfb60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:50:34.634 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.618+0000 7fefbb59e640 1 -- 172.21.15.42:0/3639921756 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fefbc0a37f0 msgr2=0x7fefbc13e360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:50:34.634 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.618+0000 7fefbb59e640 1 --2- 172.21.15.42:0/3639921756 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fefbc0a37f0 0x7fefbc13e360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:50:34.634 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.618+0000 7fefbb59e640 10 monclient: _finish_hunting 0 2024-09-16T03:50:34.634 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.618+0000 7fefbb59e640 1 monclient: found mon.noname-a 2024-09-16T03:50:34.634 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.618+0000 7fefbb59e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:50:34.635 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.618+0000 7fefbb59e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:50:34.635 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.618+0000 7fefbb59e640 1 -- 172.21.15.42:0/3639921756 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fefbc139b20 con 0x7fefbc0a3c00 2024-09-16T03:50:34.635 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.618+0000 7fefb3fff640 10 monclient: get_auth_request con 0x7fefbc0a37f0 auth_method 0 2024-09-16T03:50:34.635 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.618+0000 7fefb3fff640 1 --2- 172.21.15.42:0/3639921756 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fefbc0a37f0 0x7fefbc13e360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T03:50:34.635 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.618+0000 7fefbb59e640 1 --2- 172.21.15.42:0/3639921756 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fefbc0a3c00 0x7fefbc13e8a0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fefb40ae040 tx=0x7fefb4019540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:50:34.635 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.618+0000 7fefb37fe640 10 client.?.objecter ms_handle_connect 0x7fefbc0a3c00 2024-09-16T03:50:34.635 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.618+0000 7fefb37fe640 10 client.?.objecter resend_mon_ops 2024-09-16T03:50:34.635 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.618+0000 7fefb37fe640 1 -- 172.21.15.42:0/3639921756 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefb40bb030 con 0x7fefbc0a3c00 2024-09-16T03:50:34.635 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.618+0000 7fefb37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:50:34.635 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.618+0000 7fefb37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:50:34.635 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.618+0000 7fefb37fe640 10 monclient: dump: 2024-09-16T03:50:34.635 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:50:34.635 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:50:34.635 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:34.636 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:34.636 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:50:34.636 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:50:34.636 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:50:34.636 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:50:34.636 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:50:34.636 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:50:34.636 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.618+0000 7fefb37fe640 10 monclient: _finish_auth 0 2024-09-16T03:50:34.636 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.618+0000 7fefb37fe640 10 monclient: _check_auth_tickets 2024-09-16T03:50:34.636 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.618+0000 7fefb37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:50:34.636 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.618+0000 7fefb37fe640 1 -- 172.21.15.42:0/3639921756 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fefb40091d0 con 0x7fefbc0a3c00 2024-09-16T03:50:34.636 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.618+0000 7fefc121a640 5 monclient: authenticate success, global_id 764838 2024-09-16T03:50:34.636 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.618+0000 7fefc121a640 10 monclient: _renew_subs 2024-09-16T03:50:34.637 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.618+0000 7fefc121a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:50:34.637 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.618+0000 7fefc121a640 1 -- 172.21.15.42:0/3639921756 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fefbc1d00a0 con 0x7fefbc0a3c00 2024-09-16T03:50:34.637 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.618+0000 7fefb37fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:50:34.637 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.618+0000 7fefb37fe640 1 -- 172.21.15.42:0/3639921756 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefb40b76b0 con 0x7fefbc0a3c00 2024-09-16T03:50:34.637 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.618+0000 7fefb37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:50:34.637 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.618+0000 7fefb37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:50:34.637 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.618+0000 7fefb37fe640 10 monclient: dump: 2024-09-16T03:50:34.637 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:50:34.637 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:50:34.637 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:34.637 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:34.637 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:50:34.637 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:50:34.638 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:50:34.638 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:50:34.638 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:50:34.638 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:50:34.638 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.618+0000 7fefc121a640 10 client.764838.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:50:34.638 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.618+0000 7fefc121a640 10 monclient: _renew_subs 2024-09-16T03:50:34.638 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.618+0000 7fefc121a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:50:34.638 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.618+0000 7fefc121a640 1 -- 172.21.15.42:0/3639921756 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fefbc133170 con 0x7fefbc0a3c00 2024-09-16T03:50:34.638 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.618+0000 7fefc121a640 1 librados: init done 2024-09-16T03:50:34.638 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.618+0000 7fefb9d9b640 4 set_mon_vals no callback set 2024-09-16T03:50:34.638 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.618+0000 7fefc121a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:50:34.638 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.618+0000 7fefc121a640 1 --2- 172.21.15.42:0/3639921756 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fefbc0a1ff0 0x7fefbc02c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:50:34.638 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.618+0000 7fefc121a640 10 monclient(hunting): start opening mon connection 2024-09-16T03:50:34.639 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.618+0000 7fefc121a640 1 -- 172.21.15.42:0/3639921756 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fefbc137c70 con 0x7fefbc0a1ff0 2024-09-16T03:50:34.639 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.618+0000 7fefbad9d640 1 --2- 172.21.15.42:0/3639921756 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fefbc0a1ff0 0x7fefbc02c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:50:34.639 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.618+0000 7fefbad9d640 10 monclient: get_auth_request con 0x7fefbc0a1ff0 auth_method 0 2024-09-16T03:50:34.639 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.618+0000 7fefbad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:50:34.639 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.618+0000 7fefbad9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:50:34.639 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.618+0000 7fefbad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:50:34.639 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.618+0000 7fefb37fe640 1 -- 172.21.15.42:0/3639921756 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fefb40be070 con 0x7fefbc0a3c00 2024-09-16T03:50:34.639 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.618+0000 7fefbad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:50:34.639 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.618+0000 7fefbad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:50:34.639 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.618+0000 7fefbad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:50:34.639 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.618+0000 7fefbad9d640 10 monclient(hunting): handle_auth_done global_id 764841 payload 824 2024-09-16T03:50:34.639 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.618+0000 7fefb37fe640 1 --2- 172.21.15.42:0/3639921756 >> v2:172.21.15.154:6812/254845342 conn(0x7fef9c0377c0 0x7fef9c039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:50:34.639 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.618+0000 7fefbad9d640 1 -- 172.21.15.42:0/3639921756 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fefbc06f6a0 con 0x7fefbc0a1ff0 2024-09-16T03:50:34.639 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.618+0000 7fefb37fe640 1 -- 172.21.15.42:0/3639921756 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(281..281 src has 1..281) ==== 6024+0+0 (secure 0 0 0) 0x7fefb40e2990 con 0x7fefbc0a3c00 2024-09-16T03:50:34.640 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.618+0000 7fefb37fe640 10 client.764838.objecter ms_dispatch 0x7fefbc000ba0 osd_map(281..281 src has 1..281) 2024-09-16T03:50:34.640 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.618+0000 7fefb37fe640 3 client.764838.objecter handle_osd_map got epochs [281,281] > 0 2024-09-16T03:50:34.640 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.618+0000 7fefb37fe640 3 client.764838.objecter handle_osd_map decoding full epoch 281 2024-09-16T03:50:34.640 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.618+0000 7fefb37fe640 20 client.764838.objecter dump_active .. 0 homeless 2024-09-16T03:50:34.640 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.618+0000 7fefb37fe640 10 client.764838.objecter ms_handle_connect 0x7fefbc0a1ff0 2024-09-16T03:50:34.640 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.618+0000 7fefb37fe640 10 client.764838.objecter resend_mon_ops 2024-09-16T03:50:34.640 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.618+0000 7fefbad9d640 1 --2- 172.21.15.42:0/3639921756 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fefbc0a1ff0 0x7fefbc02c650 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fefac009fd0 tx=0x7fefac00e970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:50:34.640 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.618+0000 7fefb37fe640 1 -- 172.21.15.42:0/3639921756 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefac018070 con 0x7fefbc0a1ff0 2024-09-16T03:50:34.640 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.618+0000 7fefb37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:50:34.640 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.618+0000 7fefb37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:50:34.640 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.618+0000 7fefb37fe640 10 monclient: dump: 2024-09-16T03:50:34.640 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:50:34.640 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:50:34.641 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:34.641 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:34.641 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:50:34.641 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:50:34.641 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:50:34.641 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:50:34.641 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:50:34.641 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:50:34.641 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.618+0000 7fefb3fff640 1 --2- 172.21.15.42:0/3639921756 >> v2:172.21.15.154:6812/254845342 conn(0x7fef9c0377c0 0x7fef9c039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:50:34.641 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.618+0000 7fefb3fff640 10 monclient: get_auth_request con 0x7fef9c0377c0 auth_method 0 2024-09-16T03:50:34.641 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.622+0000 7fefb37fe640 1 -- 172.21.15.42:0/3639921756 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fefac004230 con 0x7fefbc0a1ff0 2024-09-16T03:50:34.641 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.622+0000 7fefb37fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:50:34.641 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.622+0000 7fefb37fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:50:34.642 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.622+0000 7fefb37fe640 1 -- 172.21.15.42:0/3639921756 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fefbc0a1ff0 msgr2=0x7fefbc02c650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:50:34.642 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.622+0000 7fefb37fe640 1 --2- 172.21.15.42:0/3639921756 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fefbc0a1ff0 0x7fefbc02c650 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fefac009fd0 tx=0x7fefac00e970 comp rx=0 tx=0).stop 2024-09-16T03:50:34.642 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.622+0000 7fefb37fe640 1 -- 172.21.15.42:0/3639921756 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fefbc0a1ff0 msgr2=0x7fefbc02c650 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:50:34.642 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.622+0000 7fefb37fe640 1 --2- 172.21.15.42:0/3639921756 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fefbc0a1ff0 0x7fefbc02c650 secure :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fefac009fd0 tx=0x7fefac00e970 comp rx=0 tx=0).stop 2024-09-16T03:50:34.642 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.622+0000 7fefb3fff640 1 --2- 172.21.15.42:0/3639921756 >> v2:172.21.15.154:6812/254845342 conn(0x7fef9c0377c0 0x7fef9c039c80 secure :-1 s=READY pgs=3544 cs=0 l=1 rev1=1 crypto rx=0x7fefbc139d30 tx=0x7fefa4009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:50:34.642 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.622+0000 7fefb37fe640 10 client.764838.objecter ms_handle_connect 0x7fef9c0377c0 2024-09-16T03:50:34.644 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.626+0000 7fefc121a640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:50:34.644 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.626+0000 7fefc121a640 1 --2- 172.21.15.42:0/3639921756 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fefbc0a23d0 0x7fefbc02cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:50:34.644 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.626+0000 7fefc121a640 10 monclient(hunting): start opening mon connection 2024-09-16T03:50:34.644 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.626+0000 7fefc121a640 1 -- 172.21.15.42:0/3639921756 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fefbc09de20 con 0x7fefbc0a23d0 2024-09-16T03:50:34.644 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.626+0000 7fefbad9d640 1 --2- 172.21.15.42:0/3639921756 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fefbc0a23d0 0x7fefbc02cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:50:34.644 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.626+0000 7fefbad9d640 10 monclient: get_auth_request con 0x7fefbc0a23d0 auth_method 0 2024-09-16T03:50:34.644 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.626+0000 7fefbad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:50:34.644 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.626+0000 7fefbad9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:50:34.644 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.626+0000 7fefbad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:50:34.644 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.626+0000 7fefbad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:50:34.644 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.626+0000 7fefbad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:50:34.645 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.626+0000 7fefbad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:50:34.645 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.626+0000 7fefbad9d640 10 monclient(hunting): handle_auth_done global_id 764844 payload 824 2024-09-16T03:50:34.645 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.626+0000 7fefbad9d640 1 -- 172.21.15.42:0/3639921756 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fefbc1383f0 con 0x7fefbc0a23d0 2024-09-16T03:50:34.645 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.626+0000 7fefb37fe640 10 client.764838.objecter ms_handle_connect 0x7fefbc0a23d0 2024-09-16T03:50:34.645 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.626+0000 7fefb37fe640 10 client.764838.objecter resend_mon_ops 2024-09-16T03:50:34.645 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.626+0000 7fefbad9d640 1 --2- 172.21.15.42:0/3639921756 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fefbc0a23d0 0x7fefbc02cb90 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fefac015000 tx=0x7fefac004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:50:34.645 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.626+0000 7fefb37fe640 1 -- 172.21.15.42:0/3639921756 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefac018040 con 0x7fefbc0a23d0 2024-09-16T03:50:34.645 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.626+0000 7fefb37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:50:34.645 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.626+0000 7fefb37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:50:34.645 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.626+0000 7fefb37fe640 10 monclient: dump: 2024-09-16T03:50:34.645 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:50:34.645 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:50:34.645 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:34.646 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:34.646 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:50:34.646 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:50:34.646 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:50:34.646 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:50:34.646 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:50:34.646 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:50:34.646 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.626+0000 7fefb37fe640 1 -- 172.21.15.42:0/3639921756 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4870 (secure 0 0 0) 0x7fefac003620 con 0x7fefbc0a23d0 2024-09-16T03:50:34.646 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.626+0000 7fefb37fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:50:34.646 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.626+0000 7fefb37fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:50:34.646 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.626+0000 7fefb37fe640 1 -- 172.21.15.42:0/3639921756 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fefbc0a23d0 msgr2=0x7fefbc02cb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:50:34.646 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.626+0000 7fefb37fe640 1 --2- 172.21.15.42:0/3639921756 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fefbc0a23d0 0x7fefbc02cb90 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fefac015000 tx=0x7fefac004740 comp rx=0 tx=0).stop 2024-09-16T03:50:34.646 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.626+0000 7fefb37fe640 1 -- 172.21.15.42:0/3639921756 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fefbc0a23d0 msgr2=0x7fefbc02cb90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:50:34.647 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.626+0000 7fefb37fe640 1 --2- 172.21.15.42:0/3639921756 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fefbc0a23d0 0x7fefbc02cb90 secure :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fefac015000 tx=0x7fefac004740 comp rx=0 tx=0).stop 2024-09-16T03:50:34.647 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:50:34.647 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:50:34.647 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:50:34.647 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "leader", 2024-09-16T03:50:34.647 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 76, 2024-09-16T03:50:34.647 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 225417, 2024-09-16T03:50:34.647 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:50:34.647 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:50:34.647 INFO:teuthology.orchestra.run.smithi042.stdout: 1 2024-09-16T03:50:34.648 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:50:34.648 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 0, 2024-09-16T03:50:34.648 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:50:34.648 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:50:34.648 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:50:34.648 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:50:34.648 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:50:34.648 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:50:34.648 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:50:34.648 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:50:34.648 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:50:34.648 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:50:34.648 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:50:34.649 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:50:34.649 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:50:34.649 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:50:34.649 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:50:34.649 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:50:34.649 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:50:34.649 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:50:34.649 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:50:34.649 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:50:34.649 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:50:34.649 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:50:34.649 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:50:34.649 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:50:34.649 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:50:34.650 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:50:34.650 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:50:34.650 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:50:34.650 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:50:34.650 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:50:34.650 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:50:34.650 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:50:34.650 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:50:34.650 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:50:34.650 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:50:34.650 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:50:34.650 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:50:34.650 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:50:34.650 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:50:34.651 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:50:34.651 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:50:34.651 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:50:34.651 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:50:34.651 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:50:34.651 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:50:34.651 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:50:34.651 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:50:34.651 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:50:34.651 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:50:34.651 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:50:34.651 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:50:34.651 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:50:34.652 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:50:34.652 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:50:34.652 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:50:34.652 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:50:34.652 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:50:34.652 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:50:34.652 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:50:34.652 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:50:34.652 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:50:34.652 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:50:34.652 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:50:34.652 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:50:34.652 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:50:34.652 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:50:34.653 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:50:34.653 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:50:34.653 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:50:34.653 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:50:34.653 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:50:34.653 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:50:34.653 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:50:34.653 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:50:34.653 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:50:34.653 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:50:34.653 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:50:34.653 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:50:34.653 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:50:34.654 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:50:34.654 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:50:34.654 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:50:34.654 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:50:34.654 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:50:34.654 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:50:34.654 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:50:34.654 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:50:34.654 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:50:34.654 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:50:34.654 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:50:34.654 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:50:34.655 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:50:34.655 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:50:34.655 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:50:34.655 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:50:34.655 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:50:34.655 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:50:34.655 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:50:34.655 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:50:34.655 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:50:34.655 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:50:34.655 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:50:34.655 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:50:34.655 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:50:34.656 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:50:34.656 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:50:34.656 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:50:34.656 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:50:34.656 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:50:34.656 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:50:34.656 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:50:34.656 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:50:34.656 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:50:34.656 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:50:34.656 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:50:34.656 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:50:34.656 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:50:34.657 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:50:34.657 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:50:34.657 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:50:34.657 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:50:34.657 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:50:34.657 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:50:34.657 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:50:34.657 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:50:34.657 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:50:34.657 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:50:34.657 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:50:34.657 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:50:34.657 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:50:34.658 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:50:34.658 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:50:34.658 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:50:34.658 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:50:34.658 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:50:34.658 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:50:34.658 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:50:34.658 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:50:34.658 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:50:34.658 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:50:34.658 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:50:34.658 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T03:50:34.658 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:50:34.658 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:50:34.658 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:50:34.658 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 6 2024-09-16T03:50:34.659 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:50:34.659 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:50:34.659 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:50:34.661 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:50:34.661 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:50:34.661 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:50:34.661 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 8 2024-09-16T03:50:34.662 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:50:34.662 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:50:34.662 INFO:teuthology.orchestra.run.smithi042.stdout: "mgr": [ 2024-09-16T03:50:34.662 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:50:34.662 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:50:34.662 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:50:34.662 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:50:34.663 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:50:34.663 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:50:34.663 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:50:34.663 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:50:34.663 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:50:34.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.626+0000 7fefc121a640 10 librados: watch_flush enter 2024-09-16T03:50:34.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.626+0000 7fefc121a640 10 librados: watch_flush exit 2024-09-16T03:50:34.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.626+0000 7fefc121a640 20 client.764838.objecter shutdown clearing up homeless session... 2024-09-16T03:50:34.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.626+0000 7fefc121a640 10 client.764838.objecter successfully canceled tick 2024-09-16T03:50:34.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.626+0000 7fefc121a640 1 -- 172.21.15.42:0/3639921756 >> v2:172.21.15.154:6812/254845342 conn(0x7fef9c0377c0 msgr2=0x7fef9c039c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:50:34.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.626+0000 7fefc121a640 1 --2- 172.21.15.42:0/3639921756 >> v2:172.21.15.154:6812/254845342 conn(0x7fef9c0377c0 0x7fef9c039c80 secure :-1 s=READY pgs=3544 cs=0 l=1 rev1=1 crypto rx=0x7fefbc139d30 tx=0x7fefa4009040 comp rx=0 tx=0).stop 2024-09-16T03:50:34.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.626+0000 7fefc121a640 10 monclient: shutdown 2024-09-16T03:50:34.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.626+0000 7fefc121a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:50:34.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.626+0000 7fefc121a640 1 -- 172.21.15.42:0/3639921756 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fefbc0a3c00 msgr2=0x7fefbc13e8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:50:34.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.626+0000 7fefbb59e640 1 -- 172.21.15.42:0/3639921756 reap_dead start 2024-09-16T03:50:34.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.626+0000 7fefc121a640 1 --2- 172.21.15.42:0/3639921756 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fefbc0a3c00 0x7fefbc13e8a0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fefb40ae040 tx=0x7fefb4019540 comp rx=0 tx=0).stop 2024-09-16T03:50:34.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.626+0000 7fefc121a640 1 -- 172.21.15.42:0/3639921756 shutdown_connections 2024-09-16T03:50:34.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.626+0000 7fefc121a640 1 --2- 172.21.15.42:0/3639921756 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fefbc0a3c00 0x7fefbc13e8a0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:50:34.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.626+0000 7fefc121a640 1 -- 172.21.15.42:0/3639921756 >> 172.21.15.42:0/3639921756 conn(0x7fefbc0a9fb0 msgr2=0x7fefbc136c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:50:34.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.626+0000 7fefc121a640 1 -- 172.21.15.42:0/3639921756 shutdown_connections 2024-09-16T03:50:34.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.626+0000 7fefc121a640 1 -- 172.21.15.42:0/3639921756 wait complete. 2024-09-16T03:50:34.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.626+0000 7fefc121a640 1 librados: shutdown 2024-09-16T03:50:34.669 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T03:50:34.670 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-16T03:50:34.670 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T03:50:34.790 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.786+0000 7f508b780640 10 monclient: get_monmap_and_config 2024-09-16T03:50:34.791 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.786+0000 7f508b780640 10 monclient: build_initial_monmap 2024-09-16T03:50:34.791 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.786+0000 7f508b780640 10 monclient: monmap: 2024-09-16T03:50:34.791 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:50:34.791 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:50:34.791 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:50:34.789781+0000 2024-09-16T03:50:34.791 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:50:34.789781+0000 2024-09-16T03:50:34.791 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:50:34.791 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:50:34.791 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:50:34.791 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:50:34.791 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:50:34.791 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:50:34.792 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:50:34.792 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:50:34.793 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.786+0000 7f508b780640 1 Processor -- start 2024-09-16T03:50:34.793 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.786+0000 7f508b780640 1 -- start start 2024-09-16T03:50:34.793 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.786+0000 7f508b780640 10 monclient: init 2024-09-16T03:50:34.793 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.786+0000 7f508b780640 10 monclient: _reopen_session rank -1 2024-09-16T03:50:34.793 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.786+0000 7f508b780640 10 monclient: _add_conns ranks=[2,1,4,3,0] 2024-09-16T03:50:34.793 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.786+0000 7f508b780640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f50840a1330 0x7f50840a1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:50:34.793 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.786+0000 7f508b780640 10 monclient(hunting): picked mon.noname-b con 0x7f50840a1330 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:50:34.794 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.786+0000 7f508b780640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f508406dac0 addr v1:172.21.15.154:6789/0 2024-09-16T03:50:34.794 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.786+0000 7f508b780640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f50840a1c50 0x7f5084135810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:50:34.794 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.786+0000 7f508b780640 10 monclient(hunting): picked mon.noname-c con 0x7f50840a1c50 addr v2:172.21.15.154:3300/0 2024-09-16T03:50:34.794 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.786+0000 7f508b780640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f508409c090 con 0x7f508406dac0 2024-09-16T03:50:34.794 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.786+0000 7f508b780640 10 monclient(hunting): start opening mon connection 2024-09-16T03:50:34.794 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.786+0000 7f508b780640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f508409c890 con 0x7f50840a1c50 2024-09-16T03:50:34.794 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.786+0000 7f508b780640 10 monclient(hunting): start opening mon connection 2024-09-16T03:50:34.794 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.786+0000 7f508b780640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f508409f610 con 0x7f50840a1330 2024-09-16T03:50:34.794 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.786+0000 7f508b780640 10 monclient(hunting): _renew_subs 2024-09-16T03:50:34.794 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.786+0000 7f508b780640 10 monclient(hunting): authenticate will time out at 1481.759736s 2024-09-16T03:50:34.794 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.786+0000 7f508951c640 1 -- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f50840a1330 msgr2=0x7f50840a1710 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T03:50:34.794 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.786+0000 7f508951c640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f50840a1330 0x7f50840a1710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:50:34.794 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.790+0000 7f5089d1d640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f50840a1c50 0x7f5084135810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:50:34.794 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.790+0000 7f5089d1d640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f50840a1c50 0x7f5084135810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.42:49052/0 (socket says 172.21.15.42:49052) 2024-09-16T03:50:34.795 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.790+0000 7f5089d1d640 1 -- 172.21.15.42:0/3653286367 learned_addr learned my addr 172.21.15.42:0/3653286367 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:50:34.795 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.790+0000 7f5089d1d640 10 monclient(hunting): get_auth_request con 0x7f50840a1c50 auth_method 0 2024-09-16T03:50:34.795 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.790+0000 7f5089d1d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:50:34.795 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.790+0000 7f5089d1d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:50:34.795 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.790+0000 7f5089d1d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:50:34.795 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.790+0000 7f5089d1d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:50:34.795 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.790+0000 7f5089d1d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:50:34.795 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.790+0000 7f5089d1d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:50:34.796 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.790+0000 7f5089d1d640 10 monclient(hunting): handle_auth_done global_id 794269 payload 274 2024-09-16T03:50:34.796 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.790+0000 7f5089d1d640 1 -- 172.21.15.42:0/3653286367 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f50840a1330 msgr2=0x7f50840a1710 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:50:34.796 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.790+0000 7f5089d1d640 1 --2- 172.21.15.42:0/3653286367 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f50840a1330 0x7f50840a1710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:50:34.796 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.790+0000 7f5089d1d640 1 -- 172.21.15.42:0/3653286367 >> v1:172.21.15.154:6789/0 conn(0x7f508406dac0 legacy=0x7f5084135100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:50:34.796 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.790+0000 7f5089d1d640 10 monclient: _finish_hunting 0 2024-09-16T03:50:34.796 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.790+0000 7f5089d1d640 1 monclient: found mon.noname-c 2024-09-16T03:50:34.796 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.790+0000 7f5089d1d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:50:34.796 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.790+0000 7f5089d1d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.154:3300/0 2024-09-16T03:50:34.796 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.790+0000 7f5089d1d640 1 -- 172.21.15.42:0/3653286367 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f508409f8f0 con 0x7f50840a1c50 2024-09-16T03:50:34.796 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.790+0000 7f507bfff640 1 -- 172.21.15.42:0/3653286367 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1694693590 0 0) 0x7f508409c090 con 0x7f508406dac0 2024-09-16T03:50:34.797 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.790+0000 7f507bfff640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T03:50:34.797 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.790+0000 7f5089d1d640 1 --2- 172.21.15.42:0/3653286367 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f50840a1c50 0x7f5084135810 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f507c009e50 tx=0x7f507c01c970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b225abeb45dff880 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:50:34.802 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.790+0000 7f507bfff640 1 -- 172.21.15.42:0/3653286367 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f507c0a57c0 con 0x7f50840a1c50 2024-09-16T03:50:34.802 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.790+0000 7f507bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:50:34.802 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.790+0000 7f507bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:50:34.802 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.790+0000 7f507bfff640 10 monclient: dump: 2024-09-16T03:50:34.802 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:50:34.803 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:50:34.803 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:34.803 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:34.803 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:50:34.803 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:50:34.803 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:50:34.803 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:50:34.803 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:50:34.803 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:50:34.803 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.790+0000 7f507bfff640 10 monclient: _finish_auth 0 2024-09-16T03:50:34.803 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.790+0000 7f507bfff640 10 monclient: _check_auth_tickets 2024-09-16T03:50:34.803 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.790+0000 7f507bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:50:34.803 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.790+0000 7f508b780640 5 monclient: authenticate success, global_id 794269 2024-09-16T03:50:34.803 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.790+0000 7f508b780640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:50:34.804 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.790+0000 7f507bfff640 1 -- 172.21.15.42:0/3653286367 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f507c0a5960 con 0x7f50840a1c50 2024-09-16T03:50:34.804 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.790+0000 7f507bfff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:50:34.804 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.790+0000 7f508b780640 10 monclient: get_monmap_and_config success 2024-09-16T03:50:34.804 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.790+0000 7f508b780640 4 set_mon_vals no callback set 2024-09-16T03:50:34.804 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.790+0000 7f507bfff640 1 -- 172.21.15.42:0/3653286367 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f507c0a5d30 con 0x7f50840a1c50 2024-09-16T03:50:34.804 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.790+0000 7f507bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:50:34.804 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.790+0000 7f507bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:50:34.804 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.790+0000 7f507bfff640 10 monclient: dump: 2024-09-16T03:50:34.804 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:50:34.804 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:50:34.804 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:34.804 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:34.804 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:50:34.805 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:50:34.805 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:50:34.805 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:50:34.805 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:50:34.805 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:50:34.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.794+0000 7f508b780640 10 monclient: shutdown 2024-09-16T03:50:34.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.794+0000 7f508b780640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:50:34.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.794+0000 7f508b780640 1 -- 172.21.15.42:0/3653286367 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f50840a1c50 msgr2=0x7f5084135810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:50:34.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.794+0000 7f508b780640 1 --2- 172.21.15.42:0/3653286367 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f50840a1c50 0x7f5084135810 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f507c009e50 tx=0x7f507c01c970 comp rx=0 tx=0).stop 2024-09-16T03:50:34.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.794+0000 7f508b780640 1 -- 172.21.15.42:0/3653286367 shutdown_connections 2024-09-16T03:50:34.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.794+0000 7f508b780640 1 --2- 172.21.15.42:0/3653286367 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f50840a1c50 0x7f5084135810 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:50:34.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.794+0000 7f508b780640 1 --2- 172.21.15.42:0/3653286367 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f50840a1330 0x7f50840a1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:50:34.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.794+0000 7f508b780640 1 -- 172.21.15.42:0/3653286367 >> 172.21.15.42:0/3653286367 conn(0x7f50840a2420 msgr2=0x7f50840a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:50:34.806 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.794+0000 7f508b780640 1 -- 172.21.15.42:0/3653286367 shutdown_connections 2024-09-16T03:50:34.806 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.794+0000 7f508b780640 1 -- 172.21.15.42:0/3653286367 wait complete. 2024-09-16T03:50:34.806 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.794+0000 7f508b780640 10 monclient: build_initial_monmap 2024-09-16T03:50:34.806 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.794+0000 7f508b780640 10 monclient: monmap: 2024-09-16T03:50:34.806 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:50:34.806 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:50:34.806 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:50:34.806 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:50:34.806 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:50:34.806 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:50:34.806 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:50:34.806 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:50:34.806 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:50:34.806 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:50:34.806 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.794+0000 7f508b780640 1 librados: starting msgr at 2024-09-16T03:50:34.807 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.794+0000 7f508b780640 1 librados: starting objecter 2024-09-16T03:50:34.807 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.794+0000 7f508b780640 1 Processor -- start 2024-09-16T03:50:34.807 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.794+0000 7f508b780640 1 -- start start 2024-09-16T03:50:34.807 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.794+0000 7f508b780640 1 librados: setting wanted keys 2024-09-16T03:50:34.807 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.794+0000 7f508b780640 1 librados: calling monclient init 2024-09-16T03:50:34.807 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.794+0000 7f508b780640 10 monclient: init 2024-09-16T03:50:34.807 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.794+0000 7f508b780640 10 monclient: _reopen_session rank -1 2024-09-16T03:50:34.807 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.794+0000 7f508b780640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:50:34.807 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.794+0000 7f508b780640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f50840a1330 0x7f508411bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:50:34.807 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.794+0000 7f508b780640 10 monclient(hunting): picked mon.noname-c con 0x7f50840a1330 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:50:34.807 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.794+0000 7f508b780640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f50840a1c50 0x7f508411c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:50:34.807 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.794+0000 7f508b780640 10 monclient(hunting): picked mon.noname-a con 0x7f50840a1c50 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:50:34.807 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.794+0000 7f508b780640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f508411c9f0 0x7f5084134700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:50:34.807 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.794+0000 7f508b780640 10 monclient(hunting): picked mon.noname-b con 0x7f508411c9f0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:50:34.808 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.794+0000 7f508b780640 10 monclient(hunting): start opening mon connection 2024-09-16T03:50:34.808 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.794+0000 7f508b780640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f508409ef30 con 0x7f50840a1c50 2024-09-16T03:50:34.808 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.794+0000 7f508b780640 10 monclient(hunting): start opening mon connection 2024-09-16T03:50:34.808 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.794+0000 7f508b780640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f508409edb0 con 0x7f508411c9f0 2024-09-16T03:50:34.808 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.794+0000 7f508b780640 10 monclient(hunting): start opening mon connection 2024-09-16T03:50:34.808 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.794+0000 7f508b780640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f5084135d50 con 0x7f50840a1330 2024-09-16T03:50:34.808 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.794+0000 7f508b780640 10 monclient(hunting): _renew_subs 2024-09-16T03:50:34.808 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.794+0000 7f508b780640 10 monclient(hunting): authenticate will time out at 1481.765797s 2024-09-16T03:50:34.808 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.794+0000 7f508951c640 1 -- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f50840a1330 msgr2=0x7f508411bf70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T03:50:34.808 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.794+0000 7f508951c640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f50840a1330 0x7f508411bf70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:50:34.808 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.794+0000 7f5088d1b640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f50840a1c50 0x7f508411c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:50:34.808 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.794+0000 7f5088d1b640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f50840a1c50 0x7f508411c4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:38706/0 (socket says 172.21.15.42:38706) 2024-09-16T03:50:34.808 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.794+0000 7f5088d1b640 1 -- 172.21.15.42:0/302669442 learned_addr learned my addr 172.21.15.42:0/302669442 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:50:34.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.794+0000 7f5088d1b640 10 monclient(hunting): get_auth_request con 0x7f50840a1c50 auth_method 0 2024-09-16T03:50:34.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.794+0000 7f5088d1b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:50:34.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.794+0000 7f5088d1b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:50:34.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.794+0000 7f5088d1b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:50:34.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.794+0000 7f5088d1b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:50:34.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.794+0000 7f5088d1b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:50:34.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.794+0000 7f5089d1d640 1 --2- 172.21.15.42:0/302669442 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f508411c9f0 0x7f5084134700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:50:34.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.794+0000 7f5088d1b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:50:34.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.794+0000 7f5089d1d640 10 monclient(hunting): get_auth_request con 0x7f508411c9f0 auth_method 0 2024-09-16T03:50:34.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.794+0000 7f5089d1d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:50:34.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.794+0000 7f5089d1d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:50:34.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.794+0000 7f5089d1d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:50:34.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.794+0000 7f5089d1d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:50:34.810 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.794+0000 7f5089d1d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:50:34.810 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.794+0000 7f5089d1d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:50:34.810 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.794+0000 7f5088d1b640 10 monclient(hunting): handle_auth_done global_id 764847 payload 824 2024-09-16T03:50:34.810 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.794+0000 7f5088d1b640 1 -- 172.21.15.42:0/302669442 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f50840a1330 msgr2=0x7f508411bf70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:50:34.810 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.794+0000 7f5088d1b640 1 --2- 172.21.15.42:0/302669442 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f50840a1330 0x7f508411bf70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:50:34.810 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.794+0000 7f5088d1b640 1 -- 172.21.15.42:0/302669442 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f508411c9f0 msgr2=0x7f5084134700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:50:34.810 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.794+0000 7f5088d1b640 1 --2- 172.21.15.42:0/302669442 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f508411c9f0 0x7f5084134700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:50:34.810 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.794+0000 7f5088d1b640 10 monclient: _finish_hunting 0 2024-09-16T03:50:34.810 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.794+0000 7f5088d1b640 1 monclient: found mon.noname-a 2024-09-16T03:50:34.810 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.794+0000 7f5088d1b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:50:34.810 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.794+0000 7f5088d1b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:50:34.810 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.794+0000 7f5088d1b640 1 -- 172.21.15.42:0/302669442 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f508411e220 con 0x7f50840a1c50 2024-09-16T03:50:34.810 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.794+0000 7f5088d1b640 1 --2- 172.21.15.42:0/302669442 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f50840a1c50 0x7f508411c4b0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f507400c4d0 tx=0x7f507400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:50:34.810 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.794+0000 7f5079ffb640 10 client.?.objecter ms_handle_connect 0x7f50840a1c50 2024-09-16T03:50:34.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.794+0000 7f5079ffb640 10 client.?.objecter resend_mon_ops 2024-09-16T03:50:34.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.794+0000 7f5079ffb640 1 -- 172.21.15.42:0/302669442 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5074017070 con 0x7f50840a1c50 2024-09-16T03:50:34.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.794+0000 7f5079ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:50:34.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.794+0000 7f5079ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:50:34.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.794+0000 7f5079ffb640 10 monclient: dump: 2024-09-16T03:50:34.811 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:50:34.811 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:50:34.811 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:34.811 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:34.811 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:50:34.811 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:50:34.811 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:50:34.811 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:50:34.811 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:50:34.812 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:50:34.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.794+0000 7f5079ffb640 10 monclient: _finish_auth 0 2024-09-16T03:50:34.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.794+0000 7f5079ffb640 10 monclient: _check_auth_tickets 2024-09-16T03:50:34.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.794+0000 7f5079ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:50:34.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.794+0000 7f5079ffb640 1 -- 172.21.15.42:0/302669442 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5074005480 con 0x7f50840a1c50 2024-09-16T03:50:34.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.794+0000 7f5079ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T03:50:34.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.794+0000 7f5079ffb640 1 -- 172.21.15.42:0/302669442 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5074003840 con 0x7f50840a1c50 2024-09-16T03:50:34.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.794+0000 7f508b780640 5 monclient: authenticate success, global_id 764847 2024-09-16T03:50:34.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.794+0000 7f508b780640 10 monclient: _renew_subs 2024-09-16T03:50:34.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.794+0000 7f508b780640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:50:34.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.794+0000 7f508b780640 1 -- 172.21.15.42:0/302669442 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f508413c380 con 0x7f50840a1c50 2024-09-16T03:50:34.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.794+0000 7f507affd640 4 set_mon_vals no callback set 2024-09-16T03:50:34.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.794+0000 7f5079ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:50:34.813 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.794+0000 7f5079ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:50:34.813 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.794+0000 7f5079ffb640 10 monclient: dump: 2024-09-16T03:50:34.813 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:50:34.813 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:50:34.813 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:34.813 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:34.813 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:50:34.813 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:50:34.813 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:50:34.813 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:50:34.813 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:50:34.813 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:50:34.813 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.794+0000 7f508b780640 10 client.764847.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:50:34.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.794+0000 7f508b780640 10 monclient: _renew_subs 2024-09-16T03:50:34.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.794+0000 7f508b780640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:50:34.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.794+0000 7f508b780640 1 -- 172.21.15.42:0/302669442 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f508411f1d0 con 0x7f50840a1c50 2024-09-16T03:50:34.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.794+0000 7f508b780640 1 librados: init done 2024-09-16T03:50:34.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.794+0000 7f508b780640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:50:34.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.794+0000 7f508b780640 1 --2- 172.21.15.42:0/302669442 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f508406dac0 0x7f508402c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:50:34.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.794+0000 7f508b780640 10 monclient(hunting): start opening mon connection 2024-09-16T03:50:34.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.794+0000 7f508b780640 1 -- 172.21.15.42:0/302669442 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f50840a1710 con 0x7f508406dac0 2024-09-16T03:50:34.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.794+0000 7f5079ffb640 1 -- 172.21.15.42:0/302669442 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f507401a070 con 0x7f50840a1c50 2024-09-16T03:50:34.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.798+0000 7f508951c640 1 --2- 172.21.15.42:0/302669442 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f508406dac0 0x7f508402c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:50:34.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.798+0000 7f5079ffb640 1 --2- 172.21.15.42:0/302669442 >> v2:172.21.15.154:6812/254845342 conn(0x7f50680377c0 0x7f5068039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:50:34.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.798+0000 7f508951c640 10 monclient: get_auth_request con 0x7f508406dac0 auth_method 0 2024-09-16T03:50:34.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.798+0000 7f508951c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:50:34.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.798+0000 7f508951c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:50:34.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.798+0000 7f508951c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:50:34.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.798+0000 7f5079ffb640 1 -- 172.21.15.42:0/302669442 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(281..281 src has 1..281) ==== 6024+0+0 (secure 0 0 0) 0x7f50740484e0 con 0x7f50840a1c50 2024-09-16T03:50:34.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.798+0000 7f5079ffb640 10 client.764847.objecter ms_dispatch 0x7f5084000ba0 osd_map(281..281 src has 1..281) 2024-09-16T03:50:34.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.798+0000 7f5079ffb640 3 client.764847.objecter handle_osd_map got epochs [281,281] > 0 2024-09-16T03:50:34.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.798+0000 7f5079ffb640 3 client.764847.objecter handle_osd_map decoding full epoch 281 2024-09-16T03:50:34.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.798+0000 7f5079ffb640 20 client.764847.objecter dump_active .. 0 homeless 2024-09-16T03:50:34.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.798+0000 7f508951c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:50:34.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.798+0000 7f508951c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:50:34.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.798+0000 7f508951c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:50:34.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.798+0000 7f5089d1d640 1 --2- 172.21.15.42:0/302669442 >> v2:172.21.15.154:6812/254845342 conn(0x7f50680377c0 0x7f5068039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:50:34.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.798+0000 7f5089d1d640 10 monclient: get_auth_request con 0x7f50680377c0 auth_method 0 2024-09-16T03:50:34.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.798+0000 7f508951c640 10 monclient(hunting): handle_auth_done global_id 794278 payload 824 2024-09-16T03:50:34.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.798+0000 7f508951c640 1 -- 172.21.15.42:0/302669442 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f508411f850 con 0x7f508406dac0 2024-09-16T03:50:34.816 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.798+0000 7f5079ffb640 10 client.764847.objecter ms_handle_connect 0x7f508406dac0 2024-09-16T03:50:34.816 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.798+0000 7f5079ffb640 10 client.764847.objecter resend_mon_ops 2024-09-16T03:50:34.816 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.798+0000 7f508951c640 1 --2- 172.21.15.42:0/302669442 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f508406dac0 0x7f508402c650 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f508000b450 tx=0x7f508000b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:50:34.816 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.798+0000 7f5079ffb640 10 client.764847.objecter ms_handle_connect 0x7f50680377c0 2024-09-16T03:50:34.816 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.798+0000 7f5089d1d640 1 --2- 172.21.15.42:0/302669442 >> v2:172.21.15.154:6812/254845342 conn(0x7f50680377c0 0x7f5068039c80 secure :-1 s=READY pgs=3545 cs=0 l=1 rev1=1 crypto rx=0x7f507c0ae000 tx=0x7f507c0a7070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:50:34.816 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.798+0000 7f5079ffb640 1 -- 172.21.15.42:0/302669442 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5080003dd0 con 0x7f508406dac0 2024-09-16T03:50:34.816 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.798+0000 7f5079ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:50:34.816 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.798+0000 7f5079ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:50:34.816 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.798+0000 7f5079ffb640 10 monclient: dump: 2024-09-16T03:50:34.816 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:50:34.816 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:50:34.816 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:34.816 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:34.817 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:50:34.817 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:50:34.817 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:50:34.817 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:50:34.817 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:50:34.817 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:50:34.819 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.798+0000 7f5079ffb640 1 -- 172.21.15.42:0/302669442 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f5080015070 con 0x7f508406dac0 2024-09-16T03:50:34.819 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.798+0000 7f5079ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:50:34.819 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.798+0000 7f5079ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:50:34.819 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.798+0000 7f5079ffb640 1 -- 172.21.15.42:0/302669442 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f508406dac0 msgr2=0x7f508402c650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:50:34.819 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.798+0000 7f5079ffb640 1 --2- 172.21.15.42:0/302669442 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f508406dac0 0x7f508402c650 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f508000b450 tx=0x7f508000b910 comp rx=0 tx=0).stop 2024-09-16T03:50:34.819 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.798+0000 7f5079ffb640 1 -- 172.21.15.42:0/302669442 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f508406dac0 msgr2=0x7f508402c650 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:50:34.819 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.798+0000 7f5079ffb640 1 --2- 172.21.15.42:0/302669442 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f508406dac0 0x7f508402c650 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:50:34.819 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.806+0000 7f508b780640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:50:34.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.806+0000 7f508b780640 1 --2- 172.21.15.42:0/302669442 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f508402cb90 0x7f508402cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:50:34.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.806+0000 7f508b780640 10 monclient(hunting): start opening mon connection 2024-09-16T03:50:34.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.806+0000 7f508b780640 1 -- 172.21.15.42:0/302669442 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f508411d4a0 con 0x7f508402cb90 2024-09-16T03:50:34.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.806+0000 7f508951c640 1 --2- 172.21.15.42:0/302669442 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f508402cb90 0x7f508402cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:50:34.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.806+0000 7f508951c640 10 monclient: get_auth_request con 0x7f508402cb90 auth_method 0 2024-09-16T03:50:34.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.806+0000 7f508951c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:50:34.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.806+0000 7f508951c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:50:34.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.806+0000 7f508951c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:50:34.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.806+0000 7f508951c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:50:34.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.806+0000 7f508951c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:50:34.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.806+0000 7f508951c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:50:34.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.806+0000 7f508951c640 10 monclient(hunting): handle_auth_done global_id 794281 payload 824 2024-09-16T03:50:34.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.806+0000 7f508951c640 1 -- 172.21.15.42:0/302669442 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f5084070da0 con 0x7f508402cb90 2024-09-16T03:50:34.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.806+0000 7f508951c640 1 --2- 172.21.15.42:0/302669442 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f508402cb90 0x7f508402cfa0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f508000b450 tx=0x7f50800059e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:50:34.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.806+0000 7f5079ffb640 10 client.764847.objecter ms_handle_connect 0x7f508402cb90 2024-09-16T03:50:34.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.806+0000 7f5079ffb640 10 client.764847.objecter resend_mon_ops 2024-09-16T03:50:34.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.806+0000 7f5079ffb640 1 -- 172.21.15.42:0/302669442 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5080010480 con 0x7f508402cb90 2024-09-16T03:50:34.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.806+0000 7f5079ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:50:34.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.806+0000 7f5079ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:50:34.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.806+0000 7f5079ffb640 10 monclient: dump: 2024-09-16T03:50:34.821 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:50:34.821 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:50:34.821 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:34.821 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:50:34.821 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:50:34.821 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:50:34.821 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:50:34.822 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:50:34.822 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:50:34.822 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:50:34.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.806+0000 7f5079ffb640 1 -- 172.21.15.42:0/302669442 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4533 (secure 0 0 0) 0x7f5080010480 con 0x7f508402cb90 2024-09-16T03:50:34.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.806+0000 7f5079ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:50:34.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.806+0000 7f5079ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:50:34.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.806+0000 7f5079ffb640 1 -- 172.21.15.42:0/302669442 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f508402cb90 msgr2=0x7f508402cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:50:34.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.806+0000 7f5079ffb640 1 --2- 172.21.15.42:0/302669442 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f508402cb90 0x7f508402cfa0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f508000b450 tx=0x7f50800059e0 comp rx=0 tx=0).stop 2024-09-16T03:50:34.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.806+0000 7f5079ffb640 1 -- 172.21.15.42:0/302669442 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f508402cb90 msgr2=0x7f508402cfa0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:50:34.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.806+0000 7f5079ffb640 1 --2- 172.21.15.42:0/302669442 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f508402cb90 0x7f508402cfa0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:50:34.822 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:50:34.823 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:50:34.823 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:50:34.823 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T03:50:34.823 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 76, 2024-09-16T03:50:34.823 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 76646, 2024-09-16T03:50:34.823 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:50:34.823 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:50:34.823 INFO:teuthology.orchestra.run.smithi042.stdout: 1 2024-09-16T03:50:34.823 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:50:34.823 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 0, 2024-09-16T03:50:34.823 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:50:34.823 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:50:34.824 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:50:34.824 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:50:34.824 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:50:34.824 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:50:34.824 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:50:34.824 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:50:34.824 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:50:34.824 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:50:34.824 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:50:34.824 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:50:34.824 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:50:34.824 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:50:34.824 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:50:34.825 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:50:34.825 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:50:34.825 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:50:34.825 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:50:34.825 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:50:34.825 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:50:34.825 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:50:34.825 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:50:34.825 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:50:34.825 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:50:34.825 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:50:34.825 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:50:34.825 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:50:34.826 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:50:34.826 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:50:34.826 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:50:34.826 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:50:34.826 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:50:34.826 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:50:34.826 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:50:34.826 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:50:34.826 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:50:34.826 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:50:34.826 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:50:34.826 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:50:34.826 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:50:34.826 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:50:34.827 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:50:34.827 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:50:34.827 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:50:34.827 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:50:34.827 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:50:34.827 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:50:34.827 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:50:34.827 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:50:34.827 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:50:34.827 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:50:34.827 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:50:34.827 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:50:34.827 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:50:34.827 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:50:34.827 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:50:34.828 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:50:34.828 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:50:34.828 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:50:34.828 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:50:34.828 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:50:34.828 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:50:34.828 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:50:34.828 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:50:34.828 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:50:34.828 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:50:34.828 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:50:34.828 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:50:34.829 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:50:34.829 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:50:34.829 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:50:34.829 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:50:34.829 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:50:34.829 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:50:34.829 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:50:34.829 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:50:34.829 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:50:34.829 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:50:34.829 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:50:34.829 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:50:34.829 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:50:34.829 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:50:34.830 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:50:34.830 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:50:34.830 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:50:34.830 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:50:34.830 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:50:34.830 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:50:34.830 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:50:34.830 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:50:34.830 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:50:34.830 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:50:34.830 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:50:34.830 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:50:34.830 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:50:34.830 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:50:34.831 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:50:34.831 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:50:34.831 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:50:34.831 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:50:34.831 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:50:34.831 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:50:34.831 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:50:34.831 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:50:34.831 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:50:34.831 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:50:34.831 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:50:34.831 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:50:34.831 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:50:34.832 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:50:34.832 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:50:34.832 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:50:34.832 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:50:34.832 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:50:34.832 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:50:34.832 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:50:34.832 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:50:34.832 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:50:34.832 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:50:34.832 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:50:34.832 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:50:34.832 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:50:34.833 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:50:34.833 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:50:34.833 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:50:34.833 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:50:34.833 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:50:34.833 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:50:34.833 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:50:34.833 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:50:34.833 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:50:34.833 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:50:34.833 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:50:34.833 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:50:34.833 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:50:34.833 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:50:34.834 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:50:34.834 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:50:34.834 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:50:34.834 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:50:34.834 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:50:34.834 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:50:34.834 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:50:34.834 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:50:34.834 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 2 2024-09-16T03:50:34.834 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:50:34.834 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:50:34.834 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:50:34.834 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:50:34.834 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:50:34.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.806+0000 7f508b780640 10 librados: watch_flush enter 2024-09-16T03:50:34.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.806+0000 7f508b780640 10 librados: watch_flush exit 2024-09-16T03:50:34.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.806+0000 7f508b780640 20 client.764847.objecter shutdown clearing up homeless session... 2024-09-16T03:50:34.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.806+0000 7f508b780640 10 client.764847.objecter successfully canceled tick 2024-09-16T03:50:34.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.806+0000 7f508b780640 1 -- 172.21.15.42:0/302669442 >> v2:172.21.15.154:6812/254845342 conn(0x7f50680377c0 msgr2=0x7f5068039c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:50:34.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.806+0000 7f508b780640 1 --2- 172.21.15.42:0/302669442 >> v2:172.21.15.154:6812/254845342 conn(0x7f50680377c0 0x7f5068039c80 secure :-1 s=READY pgs=3545 cs=0 l=1 rev1=1 crypto rx=0x7f507c0ae000 tx=0x7f507c0a7070 comp rx=0 tx=0).stop 2024-09-16T03:50:34.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.806+0000 7f508b780640 10 monclient: shutdown 2024-09-16T03:50:34.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.806+0000 7f508b780640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:50:34.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.806+0000 7f508b780640 1 -- 172.21.15.42:0/302669442 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f50840a1c50 msgr2=0x7f508411c4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:50:34.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.806+0000 7f508b780640 1 --2- 172.21.15.42:0/302669442 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f50840a1c50 0x7f508411c4b0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f507400c4d0 tx=0x7f507400c990 comp rx=0 tx=0).stop 2024-09-16T03:50:34.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.806+0000 7f5089d1d640 1 -- 172.21.15.42:0/302669442 reap_dead start 2024-09-16T03:50:34.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.806+0000 7f5089d1d640 1 -- 172.21.15.42:0/302669442 reap_dead start 2024-09-16T03:50:34.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.806+0000 7f508b780640 1 -- 172.21.15.42:0/302669442 shutdown_connections 2024-09-16T03:50:34.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.806+0000 7f508b780640 1 -- 172.21.15.42:0/302669442 >> 172.21.15.42:0/302669442 conn(0x7f50840a2420 msgr2=0x7f5084071750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:50:34.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.806+0000 7f508b780640 1 -- 172.21.15.42:0/302669442 shutdown_connections 2024-09-16T03:50:34.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.806+0000 7f508b780640 1 -- 172.21.15.42:0/302669442 wait complete. 2024-09-16T03:50:34.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:50:34.806+0000 7f508b780640 1 librados: shutdown 2024-09-16T03:50:34.837 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T03:50:34.838 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-16T03:50:50.186 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripWriteFudf6:::163:head 2024-09-16T03:50:50.186 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:5d165639:::164:head 2024-09-16T03:50:50.187 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:f43765fc:::165:head 2024-09-16T03:50:50.187 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:b4c720e9:::166:head 2024-09-16T03:50:50.187 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:e694b040:::167:head 2024-09-16T03:50:50.187 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:afa38db2:::168:head 2024-09-16T03:50:50.187 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:77ba9f53:::169:head 2024-09-16T03:50:50.187 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:87495034:::170:head 2024-09-16T03:50:50.187 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:7c96bf0e:::171:head 2024-09-16T03:50:50.187 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:dbe346cc:::172:head 2024-09-16T03:50:50.187 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:e943ec24:::173:head 2024-09-16T03:50:50.187 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:f97a9c0c:::174:head 2024-09-16T03:50:50.187 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:6f26e74d:::175:head 2024-09-16T03:50:50.187 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:4f95e106:::176:head 2024-09-16T03:50:50.187 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:0e6f2f8f:::177:head 2024-09-16T03:50:50.188 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:05db05f1:::178:head 2024-09-16T03:50:50.188 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:38a78d66:::179:head 2024-09-16T03:50:50.188 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:d095610b:::180:head 2024-09-16T03:50:50.188 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:a1a9d709:::181:head 2024-09-16T03:50:50.188 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:1e5d39db:::182:head 2024-09-16T03:50:50.188 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:f7df4fb9:::183:head 2024-09-16T03:50:50.188 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:03a7f161:::184:head 2024-09-16T03:50:50.188 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:ba70721e:::185:head 2024-09-16T03:50:50.188 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:28e5662d:::186:head 2024-09-16T03:50:50.188 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:973d52de:::187:head 2024-09-16T03:50:50.189 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:4303eb1c:::188:head 2024-09-16T03:50:50.189 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:b990b48e:::189:head 2024-09-16T03:50:50.189 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:29b8165b:::190:head 2024-09-16T03:50:50.189 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:3547f197:::191:head 2024-09-16T03:50:50.189 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:7e260936:::192:head 2024-09-16T03:50:50.189 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:1abec7b1:::193:head 2024-09-16T03:50:50.189 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:10fdda93:::194:head 2024-09-16T03:50:50.189 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:15817eea:::195:head 2024-09-16T03:50:50.189 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:770bab57:::196:head 2024-09-16T03:50:50.189 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:ed9e13e7:::197:head 2024-09-16T03:50:50.189 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:71471a8f:::198:head 2024-09-16T03:50:50.190 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: checking for 271:10fb1d02:::199:head 2024-09-16T03:50:50.190 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetWrite (9075 ms) 2024-09-16T03:50:50.190 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetTrim 2024-09-16T03:50:50.190 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: got ls 1726458584,0 2024-09-16T03:50:50.190 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: first is 1726458584 2024-09-16T03:50:50.190 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: got ls 1726458584,0 2024-09-16T03:50:50.190 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: got ls 1726458584,0 2024-09-16T03:50:50.190 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: got ls 1726458584,1726458586,1726458587,0 2024-09-16T03:50:50.190 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: got ls 1726458584,1726458586,1726458587,0 2024-09-16T03:50:50.190 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: got ls 1726458584,1726458586,1726458587,0 2024-09-16T03:50:50.190 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: got ls 1726458584,1726458586,1726458587,1726458589,1726458590,0 2024-09-16T03:50:50.190 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: got ls 1726458584,1726458586,1726458587,1726458589,1726458590,0 2024-09-16T03:50:50.191 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: got ls 1726458584,1726458586,1726458587,1726458589,1726458590,0 2024-09-16T03:50:50.191 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: got ls 1726458584,1726458586,1726458587,1726458589,1726458590,1726458592,1726458593,0 2024-09-16T03:50:50.191 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: got ls 1726458584,1726458586,1726458587,1726458589,1726458590,1726458592,1726458593,0 2024-09-16T03:50:50.191 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: got ls 1726458584,1726458586,1726458587,1726458589,1726458590,1726458592,1726458593,0 2024-09-16T03:50:50.191 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: got ls 1726458587,1726458589,1726458590,1726458592,1726458593,1726458595,1726458596,0 2024-09-16T03:50:50.191 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: first now 1726458587, trimmed 2024-09-16T03:50:50.191 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetTrim (35985 ms) 2024-09-16T03:50:50.191 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteOn2ndRead 2024-09-16T03:50:50.191 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: foo0 2024-09-16T03:50:50.191 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: verifying foo0 is eventually promoted 2024-09-16T03:50:50.192 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteOn2ndRead (13951 ms) 2024-09-16T03:50:50.192 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ProxyRead 2024-09-16T03:50:54.838 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-16T03:50:54.839 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-09-16T03:50:54.839 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-16T03:50:54.839 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-16T03:50:54.842 INFO:tasks.ceph.mon.c:Started 2024-09-16T03:50:54.842 INFO:tasks.mon_thrash:Sending STOP to mon a 2024-09-16T03:50:54.842 INFO:tasks.ceph.mon.a:Sent signal 19 2024-09-16T03:50:54.842 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-16T03:51:09.843 INFO:tasks.mon_thrash:Sending CONT to mon a 2024-09-16T03:51:09.844 INFO:tasks.ceph.mon.a:Sent signal 18 2024-09-16T03:51:09.844 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T03:51:09.844 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:51:09.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.989+0000 7fac0068d640 10 monclient: get_monmap_and_config 2024-09-16T03:51:09.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.989+0000 7fac0068d640 10 monclient: build_initial_monmap 2024-09-16T03:51:09.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.989+0000 7fac0068d640 10 monclient: monmap: 2024-09-16T03:51:09.993 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:51:09.994 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:09.994 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:51:09.992323+0000 2024-09-16T03:51:09.994 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:51:09.992323+0000 2024-09-16T03:51:09.994 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:51:09.994 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:09.994 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:51:09.994 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:51:09.994 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:51:09.994 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:51:09.994 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:51:09.994 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:09.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.989+0000 7fac0068d640 1 Processor -- start 2024-09-16T03:51:09.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.989+0000 7fac0068d640 1 -- start start 2024-09-16T03:51:09.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.989+0000 7fac0068d640 10 monclient: init 2024-09-16T03:51:09.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.989+0000 7fac0068d640 10 monclient: _reopen_session rank -1 2024-09-16T03:51:09.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.989+0000 7fac0068d640 10 monclient: _add_conns ranks=[0,2,4,3,1] 2024-09-16T03:51:09.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.989+0000 7fac0068d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fabf80a39b0 addr v1:172.21.15.42:6789/0 2024-09-16T03:51:09.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.989+0000 7fac0068d640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fabf80a3d90 0x7fabf813c380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:09.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.989+0000 7fac0068d640 10 monclient(hunting): picked mon.noname-b con 0x7fabf80a3d90 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:51:09.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.989+0000 7fac0068d640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fabf813c8c0 0x7fabf813ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:09.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.989+0000 7fac0068d640 10 monclient(hunting): picked mon.noname-c con 0x7fabf813c8c0 addr v2:172.21.15.154:3300/0 2024-09-16T03:51:09.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.989+0000 7fac0068d640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fabf8134290 con 0x7fabf80a39b0 2024-09-16T03:51:09.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.989+0000 7fac0068d640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:09.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.989+0000 7fac0068d640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fabf809c140 con 0x7fabf813c8c0 2024-09-16T03:51:09.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.989+0000 7fac0068d640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:09.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.989+0000 7fac0068d640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fabf809c420 con 0x7fabf80a3d90 2024-09-16T03:51:09.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.989+0000 7fac0068d640 10 monclient(hunting): _renew_subs 2024-09-16T03:51:09.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.989+0000 7fac0068d640 10 monclient(hunting): authenticate will time out at 1516.961765s 2024-09-16T03:51:09.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.989+0000 7fabfdc28640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fabf80a3d90 0x7fabf813c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:09.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.989+0000 7fabfe429640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fabf80a39b0 0x7fabf8133b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:50484/0 (socket says 172.21.15.42:50484) 2024-09-16T03:51:09.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.989+0000 7fabfe429640 1 -- 172.21.15.42:0/599195008 learned_addr learned my addr 172.21.15.42:0/599195008 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:51:09.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.989+0000 7fabfdc28640 10 monclient(hunting): get_auth_request con 0x7fabf80a3d90 auth_method 0 2024-09-16T03:51:09.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.989+0000 7fabfdc28640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:09.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.989+0000 7fabfdc28640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:09.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.989+0000 7fabfdc28640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:51:09.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.989+0000 7fabfdc28640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:51:09.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.989+0000 7fabfdc28640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:51:09.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.989+0000 7fabfdc28640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:51:09.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.989+0000 7fabfec2a640 1 --2- 172.21.15.42:0/599195008 >> v2:172.21.15.154:3300/0 conn(0x7fabf813c8c0 0x7fabf813ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:09.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.989+0000 7fabfec2a640 10 monclient(hunting): get_auth_request con 0x7fabf813c8c0 auth_method 0 2024-09-16T03:51:09.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.989+0000 7fabfec2a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:09.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.989+0000 7fabfec2a640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:09.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.989+0000 7fabfec2a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:51:09.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.989+0000 7fabfd427640 1 -- 172.21.15.42:0/599195008 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3489475563 0 0) 0x7fabf8134290 con 0x7fabf80a39b0 2024-09-16T03:51:09.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.989+0000 7fabfd427640 10 monclient(hunting): _negotiate 2024-09-16T03:51:09.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.989+0000 7fabfd427640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:09.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.989+0000 7fabfd427640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:51:09.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.989+0000 7fabfd427640 10 monclient(hunting): my global_id is 764868 2024-09-16T03:51:09.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.989+0000 7fabfd427640 1 -- 172.21.15.42:0/599195008 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fabe4003530 con 0x7fabf80a39b0 2024-09-16T03:51:09.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.989+0000 7fabfdc28640 10 monclient(hunting): handle_auth_done global_id 814100 payload 274 2024-09-16T03:51:09.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.989+0000 7fabfdc28640 1 -- 172.21.15.42:0/599195008 >> v2:172.21.15.154:3300/0 conn(0x7fabf813c8c0 msgr2=0x7fabf813ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:51:09.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.989+0000 7fabfdc28640 1 --2- 172.21.15.42:0/599195008 >> v2:172.21.15.154:3300/0 conn(0x7fabf813c8c0 0x7fabf813ecb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:09.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.989+0000 7fabfdc28640 1 -- 172.21.15.42:0/599195008 >> v1:172.21.15.42:6789/0 conn(0x7fabf80a39b0 legacy=0x7fabf8133b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:09.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.989+0000 7fabfdc28640 10 monclient: _finish_hunting 0 2024-09-16T03:51:09.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.989+0000 7fabfdc28640 1 monclient: found mon.noname-b 2024-09-16T03:51:09.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.989+0000 7fabfdc28640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:51:09.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.989+0000 7fabfdc28640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T03:51:09.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.989+0000 7fabfdc28640 1 -- 172.21.15.42:0/599195008 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fabf806dd60 con 0x7fabf80a3d90 2024-09-16T03:51:10.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.993+0000 7fabfdc28640 1 --2- 172.21.15.42:0/599195008 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fabf80a3d90 0x7fabf813c380 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fabe80097c0 tx=0x7fabe801c360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e0d4812dbb8de8f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:51:10.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.993+0000 7fabfdc28640 1 -- 172.21.15.42:0/599195008 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fabf80a3d90 msgr2=0x7fabf813c380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-09-16T03:51:10.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.993+0000 7fabfdc28640 1 -- 172.21.15.42:0/599195008 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fabf80a3d90 msgr2=0x7fabf813c380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T03:51:10.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.993+0000 7fabfdc28640 1 --2- 172.21.15.42:0/599195008 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fabf80a3d90 0x7fabf813c380 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fabe80097c0 tx=0x7fabe801c360 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:51:10.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.993+0000 7fabfdc28640 1 --2- 172.21.15.42:0/599195008 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fabf80a3d90 0x7fabf813c380 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fabe80097c0 tx=0x7fabe801c360 comp rx=0 tx=0).stop 2024-09-16T03:51:10.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.993+0000 7fabfd427640 10 monclient: ms_handle_reset current mon [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:51:10.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.993+0000 7fabfd427640 10 monclient: _reopen_session rank -1 2024-09-16T03:51:10.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.993+0000 7fabfd427640 1 -- 172.21.15.42:0/599195008 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fabf80a3d90 msgr2=0x7fabf813c380 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:51:10.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.993+0000 7fabfd427640 1 --2- 172.21.15.42:0/599195008 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fabf80a3d90 0x7fabf813c380 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:10.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.993+0000 7fabfd427640 10 monclient: _add_conns ranks=[1,3] 2024-09-16T03:51:10.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.993+0000 7fabfd427640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fabe4003810 addr v1:172.21.15.154:6789/0 2024-09-16T03:51:10.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.993+0000 7fabfd427640 1 --2- 172.21.15.42:0/599195008 >> v2:172.21.15.42:3300/0 conn(0x7fabe40079b0 0x7fabe4009da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:10.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.993+0000 7fabfd427640 10 monclient(hunting): picked mon.noname-a con 0x7fabe40079b0 addr v2:172.21.15.42:3300/0 2024-09-16T03:51:10.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.993+0000 7fabfd427640 1 -- 172.21.15.42:0/599195008 --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fabf8134290 con 0x7fabe4003810 2024-09-16T03:51:10.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.993+0000 7fabfd427640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:10.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.993+0000 7fabfd427640 1 -- 172.21.15.42:0/599195008 --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7fabe400b5c0 con 0x7fabe40079b0 2024-09-16T03:51:10.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.993+0000 7fabfd427640 10 monclient(hunting): _renew_subs 2024-09-16T03:51:10.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.993+0000 7fabfd427640 10 monclient(hunting): ms_handle_reset stray mon [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:51:10.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.993+0000 7fabfdc28640 1 --2- 172.21.15.42:0/599195008 >> v2:172.21.15.42:3300/0 conn(0x7fabe40079b0 0x7fabe4009da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:10.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.993+0000 7fabfdc28640 10 monclient(hunting): get_auth_request con 0x7fabe40079b0 auth_method 0 2024-09-16T03:51:10.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.993+0000 7fabfdc28640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:10.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.993+0000 7fabfdc28640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:10.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.993+0000 7fabfdc28640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:51:10.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.993+0000 7fabfdc28640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:51:10.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.993+0000 7fabfdc28640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:51:10.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.993+0000 7fabfdc28640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T03:51:10.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.993+0000 7fabfdc28640 10 monclient(hunting): handle_auth_done global_id 814100 payload 293 2024-09-16T03:51:10.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.993+0000 7fabfdc28640 1 -- 172.21.15.42:0/599195008 >> v1:172.21.15.154:6789/0 conn(0x7fabe4003810 legacy=0x7fabe4003c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:51:10.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.993+0000 7fabfdc28640 10 monclient: _finish_hunting 0 2024-09-16T03:51:10.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.993+0000 7fabfdc28640 1 monclient: found mon.noname-a 2024-09-16T03:51:10.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.993+0000 7fabfdc28640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:51:10.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.993+0000 7fabfdc28640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:51:10.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.993+0000 7fabfdc28640 1 -- 172.21.15.42:0/599195008 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fabe400b830 con 0x7fabe40079b0 2024-09-16T03:51:10.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.993+0000 7fabfdc28640 1 --2- 172.21.15.42:0/599195008 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fabe40079b0 0x7fabe4009da0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fabf809bff0 tx=0x7fabe8009190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78b1c41e5ac0105d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:51:10.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.993+0000 7fabfd427640 1 -- 172.21.15.42:0/599195008 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabe80ac710 con 0x7fabe40079b0 2024-09-16T03:51:10.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.993+0000 7fabfd427640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:51:10.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.993+0000 7fabfd427640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:51:10.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.993+0000 7fabfd427640 10 monclient: dump: 2024-09-16T03:51:10.010 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:51:10.010 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:10.010 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:10.010 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:10.010 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:51:10.010 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:10.010 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:51:10.010 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:51:10.010 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:51:10.010 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:10.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.993+0000 7fabfd427640 10 monclient: _finish_auth 0 2024-09-16T03:51:10.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.993+0000 7fabfd427640 10 monclient: _check_auth_tickets 2024-09-16T03:51:10.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.993+0000 7fabfd427640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:51:10.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.993+0000 7fac0068d640 5 monclient: authenticate success, global_id 814100 2024-09-16T03:51:10.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.993+0000 7fac0068d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:51:10.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.993+0000 7fabfd427640 1 -- 172.21.15.42:0/599195008 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fabe80ac9b0 con 0x7fabe40079b0 2024-09-16T03:51:10.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.993+0000 7fabfd427640 10 monclient: handle_config config(0 keys) 2024-09-16T03:51:10.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.993+0000 7fac0068d640 10 monclient: get_monmap_and_config success 2024-09-16T03:51:10.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.993+0000 7fabfd427640 1 -- 172.21.15.42:0/599195008 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabe80acd60 con 0x7fabe40079b0 2024-09-16T03:51:10.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.993+0000 7fabfd427640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:51:10.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.993+0000 7fac0068d640 4 set_mon_vals no callback set 2024-09-16T03:51:10.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.993+0000 7fabfd427640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:51:10.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.993+0000 7fabfd427640 10 monclient: dump: 2024-09-16T03:51:10.011 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:51:10.011 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:10.011 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:10.011 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:10.011 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:51:10.012 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:10.012 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:51:10.012 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:51:10.012 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:51:10.012 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:10.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.993+0000 7fac0068d640 10 monclient: shutdown 2024-09-16T03:51:10.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.993+0000 7fac0068d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:51:10.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.993+0000 7fac0068d640 1 -- 172.21.15.42:0/599195008 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fabe40079b0 msgr2=0x7fabe4009da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:10.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.993+0000 7fac0068d640 1 --2- 172.21.15.42:0/599195008 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fabe40079b0 0x7fabe4009da0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fabf809bff0 tx=0x7fabe8009190 comp rx=0 tx=0).stop 2024-09-16T03:51:10.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.993+0000 7fabfec2a640 1 -- 172.21.15.42:0/599195008 reap_dead start 2024-09-16T03:51:10.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.993+0000 7fac0068d640 1 -- 172.21.15.42:0/599195008 shutdown_connections 2024-09-16T03:51:10.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.993+0000 7fac0068d640 1 --2- 172.21.15.42:0/599195008 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fabe40079b0 0x7fabe4009da0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:10.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.993+0000 7fac0068d640 1 -- 172.21.15.42:0/599195008 >> 172.21.15.42:0/599195008 conn(0x7fabf80a9fb0 msgr2=0x7fabf80aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:51:10.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.993+0000 7fac0068d640 1 -- 172.21.15.42:0/599195008 shutdown_connections 2024-09-16T03:51:10.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.993+0000 7fac0068d640 1 -- 172.21.15.42:0/599195008 wait complete. 2024-09-16T03:51:10.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.993+0000 7fac0068d640 10 monclient: build_initial_monmap 2024-09-16T03:51:10.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.993+0000 7fac0068d640 10 monclient: monmap: 2024-09-16T03:51:10.013 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:51:10.013 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:51:10.013 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:51:10.013 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:51:10.013 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:51:10.013 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:10.014 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:51:10.014 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:51:10.014 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:51:10.014 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:10.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.993+0000 7fac0068d640 1 librados: starting msgr at 2024-09-16T03:51:10.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.993+0000 7fac0068d640 1 librados: starting objecter 2024-09-16T03:51:10.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.993+0000 7fac0068d640 1 Processor -- start 2024-09-16T03:51:10.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.993+0000 7fac0068d640 1 -- start start 2024-09-16T03:51:10.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.993+0000 7fac0068d640 1 librados: setting wanted keys 2024-09-16T03:51:10.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.993+0000 7fac0068d640 1 librados: calling monclient init 2024-09-16T03:51:10.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.993+0000 7fac0068d640 10 monclient: init 2024-09-16T03:51:10.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.993+0000 7fac0068d640 10 monclient: _reopen_session rank -1 2024-09-16T03:51:10.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.993+0000 7fac0068d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:51:10.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.993+0000 7fac0068d640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fabe40079b0 0x7fabf81cf010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:10.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.993+0000 7fac0068d640 10 monclient(hunting): picked mon.noname-c con 0x7fabe40079b0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:51:10.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.993+0000 7fac0068d640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fabf8139560 0x7fabf81cf550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:10.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.993+0000 7fac0068d640 10 monclient(hunting): picked mon.noname-a con 0x7fabf8139560 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:51:10.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.993+0000 7fac0068d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fabf81cfa90 0x7fabf812d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:10.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.993+0000 7fac0068d640 10 monclient(hunting): picked mon.noname-b con 0x7fabf81cfa90 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:51:10.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.993+0000 7fac0068d640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:10.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.993+0000 7fac0068d640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fabf8141da0 con 0x7fabf8139560 2024-09-16T03:51:10.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.993+0000 7fac0068d640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:10.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.993+0000 7fac0068d640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fabf8141c20 con 0x7fabf81cfa90 2024-09-16T03:51:10.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.993+0000 7fac0068d640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:10.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.993+0000 7fac0068d640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fabf8141f20 con 0x7fabe40079b0 2024-09-16T03:51:10.016 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.993+0000 7fac0068d640 10 monclient(hunting): _renew_subs 2024-09-16T03:51:10.016 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.993+0000 7fac0068d640 10 monclient(hunting): authenticate will time out at 1516.967935s 2024-09-16T03:51:10.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.997+0000 7fabfdc28640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fabf8139560 0x7fabf81cf550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:10.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.997+0000 7fabfdc28640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fabf8139560 0x7fabf81cf550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:52998/0 (socket says 172.21.15.42:52998) 2024-09-16T03:51:10.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.997+0000 7fabfdc28640 1 -- 172.21.15.42:0/3329792371 learned_addr learned my addr 172.21.15.42:0/3329792371 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:51:10.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.997+0000 7fabfdc28640 10 monclient(hunting): get_auth_request con 0x7fabf8139560 auth_method 0 2024-09-16T03:51:10.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.997+0000 7fabfdc28640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:10.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.997+0000 7fabfdc28640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:10.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.997+0000 7fabfdc28640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:51:10.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.997+0000 7fabfe429640 1 --2- 172.21.15.42:0/3329792371 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fabe40079b0 0x7fabf81cf010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:10.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.997+0000 7fabfe429640 10 monclient(hunting): get_auth_request con 0x7fabe40079b0 auth_method 0 2024-09-16T03:51:10.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.997+0000 7fabfe429640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:10.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.997+0000 7fabfe429640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:10.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.997+0000 7fabfe429640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:51:10.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.997+0000 7fabfdc28640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:51:10.018 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.997+0000 7fabfdc28640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:51:10.018 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.997+0000 7fabfdc28640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:51:10.018 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.997+0000 7fabfe429640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:51:10.018 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.997+0000 7fabfe429640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:51:10.018 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.997+0000 7fabfec2a640 1 --2- 172.21.15.42:0/3329792371 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fabf81cfa90 0x7fabf812d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:10.018 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.997+0000 7fabfe429640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:51:10.018 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.997+0000 7fabfec2a640 10 monclient(hunting): get_auth_request con 0x7fabf81cfa90 auth_method 0 2024-09-16T03:51:10.018 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.997+0000 7fabfec2a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:10.018 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.997+0000 7fabfec2a640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:10.018 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.997+0000 7fabfec2a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:51:10.018 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.997+0000 7fabfdc28640 10 monclient(hunting): handle_auth_done global_id 764871 payload 824 2024-09-16T03:51:10.018 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.997+0000 7fabfdc28640 1 -- 172.21.15.42:0/3329792371 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fabe40079b0 msgr2=0x7fabf81cf010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:10.018 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.997+0000 7fabfdc28640 1 --2- 172.21.15.42:0/3329792371 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fabe40079b0 0x7fabf81cf010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:10.019 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.997+0000 7fabfdc28640 1 -- 172.21.15.42:0/3329792371 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fabf81cfa90 msgr2=0x7fabf812d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:10.019 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.997+0000 7fabfdc28640 1 --2- 172.21.15.42:0/3329792371 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fabf81cfa90 0x7fabf812d2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:10.019 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.997+0000 7fabfdc28640 10 monclient: _finish_hunting 0 2024-09-16T03:51:10.019 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.997+0000 7fabfdc28640 1 monclient: found mon.noname-a 2024-09-16T03:51:10.019 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.997+0000 7fabfdc28640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:51:10.019 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.997+0000 7fabfdc28640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:51:10.019 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.997+0000 7fabfdc28640 1 -- 172.21.15.42:0/3329792371 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fabf806f680 con 0x7fabf8139560 2024-09-16T03:51:10.019 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.997+0000 7fabfdc28640 1 --2- 172.21.15.42:0/3329792371 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fabf8139560 0x7fabf81cf550 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fabe80b3040 tx=0x7fabe8019100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:51:10.019 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.997+0000 7fabeeffd640 10 client.?.objecter ms_handle_connect 0x7fabf8139560 2024-09-16T03:51:10.019 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.997+0000 7fabeeffd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:51:10.019 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.997+0000 7fabeeffd640 1 -- 172.21.15.42:0/3329792371 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabe80c5020 con 0x7fabf8139560 2024-09-16T03:51:10.019 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.997+0000 7fabeeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:51:10.019 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.997+0000 7fabeeffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:51:10.020 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.997+0000 7fabeeffd640 10 monclient: dump: 2024-09-16T03:51:10.020 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:51:10.020 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:10.020 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:10.020 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:10.020 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:51:10.020 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:10.020 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:51:10.020 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:51:10.020 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:51:10.020 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:10.020 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.997+0000 7fabeeffd640 10 monclient: _finish_auth 0 2024-09-16T03:51:10.020 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.997+0000 7fabeeffd640 10 monclient: _check_auth_tickets 2024-09-16T03:51:10.020 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.997+0000 7fabeeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:51:10.021 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.997+0000 7fabeeffd640 1 -- 172.21.15.42:0/3329792371 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fabe80059a0 con 0x7fabf8139560 2024-09-16T03:51:10.021 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.997+0000 7fac0068d640 5 monclient: authenticate success, global_id 764871 2024-09-16T03:51:10.021 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.997+0000 7fac0068d640 10 monclient: _renew_subs 2024-09-16T03:51:10.021 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.997+0000 7fac0068d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:51:10.021 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.997+0000 7fac0068d640 1 -- 172.21.15.42:0/3329792371 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fabf8131140 con 0x7fabf8139560 2024-09-16T03:51:10.021 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.997+0000 7fabeeffd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:51:10.021 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.997+0000 7fabeeffd640 1 -- 172.21.15.42:0/3329792371 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabe8005d70 con 0x7fabf8139560 2024-09-16T03:51:10.021 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.997+0000 7fac0068d640 10 client.764871.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:51:10.021 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.997+0000 7fac0068d640 10 monclient: _renew_subs 2024-09-16T03:51:10.021 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.997+0000 7fac0068d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:51:10.021 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.997+0000 7fabfcc26640 4 set_mon_vals no callback set 2024-09-16T03:51:10.021 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.997+0000 7fac0068d640 1 -- 172.21.15.42:0/3329792371 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fabf813c140 con 0x7fabf8139560 2024-09-16T03:51:10.021 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.997+0000 7fac0068d640 1 librados: init done 2024-09-16T03:51:10.022 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.997+0000 7fabeeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:51:10.022 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.997+0000 7fabeeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:51:10.022 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.997+0000 7fabeeffd640 10 monclient: dump: 2024-09-16T03:51:10.022 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:51:10.022 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:10.022 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:10.022 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:10.022 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:51:10.022 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:10.022 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:51:10.022 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:51:10.022 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:51:10.022 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:10.022 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.997+0000 7fac0068d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:51:10.023 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.997+0000 7fac0068d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:51:10.023 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.997+0000 7fac0068d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:51:10.023 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.997+0000 7fac0068d640 1 -- 172.21.15.42:0/3329792371 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fabf8133b80 con 0x7fabf8139560 2024-09-16T03:51:10.023 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.997+0000 7fabeeffd640 1 -- 172.21.15.42:0/3329792371 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fabe80c8020 con 0x7fabf8139560 2024-09-16T03:51:10.023 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.997+0000 7fabeeffd640 1 --2- 172.21.15.42:0/3329792371 >> v2:172.21.15.154:6812/254845342 conn(0x7fabdc037810 0x7fabdc039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:10.023 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.997+0000 7fabeeffd640 1 -- 172.21.15.42:0/3329792371 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(306..306 src has 1..306) ==== 6552+0+0 (secure 0 0 0) 0x7fabe80e4640 con 0x7fabf8139560 2024-09-16T03:51:10.023 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.997+0000 7fabeeffd640 10 client.764871.objecter ms_dispatch 0x7fabf8000ba0 osd_map(306..306 src has 1..306) 2024-09-16T03:51:10.023 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.997+0000 7fabeeffd640 3 client.764871.objecter handle_osd_map got epochs [306,306] > 0 2024-09-16T03:51:10.023 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.997+0000 7fabeeffd640 3 client.764871.objecter handle_osd_map decoding full epoch 306 2024-09-16T03:51:10.023 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.997+0000 7fabeeffd640 20 client.764871.objecter dump_active .. 0 homeless 2024-09-16T03:51:10.023 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.997+0000 7fabfe429640 1 --2- 172.21.15.42:0/3329792371 >> v2:172.21.15.154:6812/254845342 conn(0x7fabdc037810 0x7fabdc039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:10.023 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.997+0000 7fabfe429640 10 monclient: get_auth_request con 0x7fabdc037810 auth_method 0 2024-09-16T03:51:10.023 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.997+0000 7fabfe429640 1 --2- 172.21.15.42:0/3329792371 >> v2:172.21.15.154:6812/254845342 conn(0x7fabdc037810 0x7fabdc039cd0 secure :-1 s=READY pgs=3550 cs=0 l=1 rev1=1 crypto rx=0x7fabf40012f0 tx=0x7fabf4009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:51:10.023 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:09.997+0000 7fabeeffd640 10 client.764871.objecter ms_handle_connect 0x7fabdc037810 2024-09-16T03:51:10.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.001+0000 7fabeeffd640 1 -- 172.21.15.42:0/3329792371 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7fabe80cf070 con 0x7fabf8139560 2024-09-16T03:51:10.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.001+0000 7fabeeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:51:10.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.001+0000 7fabeeffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:51:10.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.233+0000 7fac0068d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:51:10.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.233+0000 7fac0068d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:51:10.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.233+0000 7fac0068d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:51:10.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.233+0000 7fac0068d640 1 -- 172.21.15.42:0/3329792371 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fabf80a5750 con 0x7fabf8139560 2024-09-16T03:51:10.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.233+0000 7fabeeffd640 1 -- 172.21.15.42:0/3329792371 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7fabe8016070 con 0x7fabf8139560 2024-09-16T03:51:10.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.233+0000 7fabeeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:51:10.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.233+0000 7fabeeffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:51:10.240 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":80,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:51:10.242 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.237+0000 7fac0068d640 10 librados: watch_flush enter 2024-09-16T03:51:10.242 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.237+0000 7fac0068d640 10 librados: watch_flush exit 2024-09-16T03:51:10.242 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.237+0000 7fac0068d640 20 client.764871.objecter shutdown clearing up homeless session... 2024-09-16T03:51:10.242 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.237+0000 7fac0068d640 10 client.764871.objecter successfully canceled tick 2024-09-16T03:51:10.242 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.237+0000 7fac0068d640 1 -- 172.21.15.42:0/3329792371 >> v2:172.21.15.154:6812/254845342 conn(0x7fabdc037810 msgr2=0x7fabdc039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:10.242 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.237+0000 7fac0068d640 1 --2- 172.21.15.42:0/3329792371 >> v2:172.21.15.154:6812/254845342 conn(0x7fabdc037810 0x7fabdc039cd0 secure :-1 s=READY pgs=3550 cs=0 l=1 rev1=1 crypto rx=0x7fabf40012f0 tx=0x7fabf4009040 comp rx=0 tx=0).stop 2024-09-16T03:51:10.242 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.237+0000 7fac0068d640 10 monclient: shutdown 2024-09-16T03:51:10.242 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.237+0000 7fac0068d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:51:10.243 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.237+0000 7fac0068d640 1 -- 172.21.15.42:0/3329792371 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fabf8139560 msgr2=0x7fabf81cf550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:10.243 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.237+0000 7fac0068d640 1 --2- 172.21.15.42:0/3329792371 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fabf8139560 0x7fabf81cf550 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fabe80b3040 tx=0x7fabe8019100 comp rx=0 tx=0).stop 2024-09-16T03:51:10.243 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.237+0000 7fac0068d640 1 -- 172.21.15.42:0/3329792371 shutdown_connections 2024-09-16T03:51:10.243 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.237+0000 7fac0068d640 1 --2- 172.21.15.42:0/3329792371 >> v2:172.21.15.154:6812/254845342 conn(0x7fabdc037810 0x7fabdc039cd0 unknown :-1 s=CLOSED pgs=3550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:10.243 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.237+0000 7fac0068d640 1 --2- 172.21.15.42:0/3329792371 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fabf81cfa90 0x7fabf812d2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:10.243 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.237+0000 7fac0068d640 1 --2- 172.21.15.42:0/3329792371 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fabf8139560 0x7fabf81cf550 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:10.243 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.237+0000 7fac0068d640 1 --2- 172.21.15.42:0/3329792371 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fabe40079b0 0x7fabf81cf010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:10.243 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.237+0000 7fac0068d640 1 -- 172.21.15.42:0/3329792371 >> 172.21.15.42:0/3329792371 conn(0x7fabf80a9fb0 msgr2=0x7fabf8131980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:51:10.243 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.237+0000 7fac0068d640 1 -- 172.21.15.42:0/3329792371 shutdown_connections 2024-09-16T03:51:10.243 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.237+0000 7fac0068d640 1 -- 172.21.15.42:0/3329792371 wait complete. 2024-09-16T03:51:10.243 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.237+0000 7fac0068d640 1 librados: shutdown 2024-09-16T03:51:10.259 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T03:51:10.259 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T03:51:10.259 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T03:51:10.386 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.381+0000 7fa969892640 10 monclient: get_monmap_and_config 2024-09-16T03:51:10.386 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.381+0000 7fa969892640 10 monclient: build_initial_monmap 2024-09-16T03:51:10.386 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.381+0000 7fa969892640 10 monclient: monmap: 2024-09-16T03:51:10.386 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:51:10.386 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:10.386 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:51:10.384924+0000 2024-09-16T03:51:10.386 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:51:10.384924+0000 2024-09-16T03:51:10.386 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:51:10.386 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:10.387 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:51:10.387 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:51:10.387 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:51:10.387 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:51:10.387 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:51:10.387 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:10.389 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.381+0000 7fa969892640 1 Processor -- start 2024-09-16T03:51:10.389 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.381+0000 7fa969892640 1 -- start start 2024-09-16T03:51:10.389 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.381+0000 7fa969892640 10 monclient: init 2024-09-16T03:51:10.389 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.381+0000 7fa969892640 10 monclient: _reopen_session rank -1 2024-09-16T03:51:10.389 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.381+0000 7fa969892640 10 monclient: _add_conns ranks=[0,4,2,3,1] 2024-09-16T03:51:10.389 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.381+0000 7fa969892640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa96409fb00 addr v1:172.21.15.42:6789/0 2024-09-16T03:51:10.389 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.381+0000 7fa969892640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fa96409c160 0x7fa964139120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:10.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.381+0000 7fa969892640 10 monclient(hunting): picked mon.noname-c con 0x7fa96409c160 addr v2:172.21.15.154:3300/0 2024-09-16T03:51:10.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.381+0000 7fa969892640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa96409c5d0 0x7fa96413b670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:10.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.381+0000 7fa969892640 10 monclient(hunting): picked mon.noname-b con 0x7fa96409c5d0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:51:10.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.381+0000 7fa969892640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa9640a1e20 con 0x7fa96409fb00 2024-09-16T03:51:10.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.381+0000 7fa969892640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:10.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.381+0000 7fa969892640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fa96409e250 con 0x7fa96409c160 2024-09-16T03:51:10.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.381+0000 7fa969892640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:10.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.381+0000 7fa969892640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fa96409ea30 con 0x7fa96409c5d0 2024-09-16T03:51:10.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.381+0000 7fa969892640 10 monclient(hunting): _renew_subs 2024-09-16T03:51:10.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.381+0000 7fa969892640 10 monclient(hunting): authenticate will time out at 1517.354502s 2024-09-16T03:51:10.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.381+0000 7fa9637fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa96409c5d0 0x7fa96413b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:10.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.381+0000 7fa962ffd640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fa96409fb00 0x7fa96409ba50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:50498/0 (socket says 172.21.15.42:50498) 2024-09-16T03:51:10.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.381+0000 7fa962ffd640 1 -- 172.21.15.42:0/3176782205 learned_addr learned my addr 172.21.15.42:0/3176782205 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:51:10.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.381+0000 7fa9637fe640 10 monclient(hunting): get_auth_request con 0x7fa96409c5d0 auth_method 0 2024-09-16T03:51:10.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.381+0000 7fa9637fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:10.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.381+0000 7fa9637fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:10.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.381+0000 7fa9637fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:51:10.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.381+0000 7fa9637fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:51:10.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.381+0000 7fa9637fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:51:10.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.381+0000 7fa9637fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:51:10.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.381+0000 7fa9627fc640 1 --2- 172.21.15.42:0/3176782205 >> v2:172.21.15.154:3300/0 conn(0x7fa96409c160 0x7fa964139120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:10.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.381+0000 7fa9627fc640 10 monclient(hunting): get_auth_request con 0x7fa96409c160 auth_method 0 2024-09-16T03:51:10.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.381+0000 7fa9627fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:10.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.381+0000 7fa9627fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:10.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.381+0000 7fa9627fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:51:10.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.381+0000 7fa9637fe640 10 monclient(hunting): handle_auth_done global_id 814106 payload 274 2024-09-16T03:51:10.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.381+0000 7fa9637fe640 1 -- 172.21.15.42:0/3176782205 >> v2:172.21.15.154:3300/0 conn(0x7fa96409c160 msgr2=0x7fa964139120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:51:10.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.381+0000 7fa9637fe640 1 --2- 172.21.15.42:0/3176782205 >> v2:172.21.15.154:3300/0 conn(0x7fa96409c160 0x7fa964139120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:10.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.381+0000 7fa9637fe640 1 -- 172.21.15.42:0/3176782205 >> v1:172.21.15.42:6789/0 conn(0x7fa96409fb00 legacy=0x7fa96409ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:10.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.381+0000 7fa9637fe640 10 monclient: _finish_hunting 0 2024-09-16T03:51:10.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.381+0000 7fa9637fe640 1 monclient: found mon.noname-b 2024-09-16T03:51:10.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.381+0000 7fa961ffb640 1 -- 172.21.15.42:0/3176782205 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4085672215 0 0) 0x7fa9640a1e20 con 0x7fa96409fb00 2024-09-16T03:51:10.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.381+0000 7fa9637fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:51:10.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.381+0000 7fa9637fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T03:51:10.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.381+0000 7fa9637fe640 1 -- 172.21.15.42:0/3176782205 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa96409fee0 con 0x7fa96409c5d0 2024-09-16T03:51:10.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.381+0000 7fa961ffb640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T03:51:10.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.385+0000 7fa9637fe640 1 --2- 172.21.15.42:0/3176782205 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa96409c5d0 0x7fa96413b670 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fa9540097c0 tx=0x7fa95401d470 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a1893bc7eb45110d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:51:10.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.385+0000 7fa961ffb640 1 -- 172.21.15.42:0/3176782205 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9540b2070 con 0x7fa96409c5d0 2024-09-16T03:51:10.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.385+0000 7fa961ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:51:10.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.385+0000 7fa961ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:51:10.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.385+0000 7fa961ffb640 10 monclient: dump: 2024-09-16T03:51:10.398 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:51:10.398 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:10.398 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:10.399 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:10.399 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:51:10.399 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:10.399 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:51:10.399 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:51:10.399 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:51:10.399 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:10.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.385+0000 7fa961ffb640 10 monclient: _finish_auth 0 2024-09-16T03:51:10.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.385+0000 7fa961ffb640 10 monclient: _check_auth_tickets 2024-09-16T03:51:10.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.385+0000 7fa961ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:51:10.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.385+0000 7fa961ffb640 1 -- 172.21.15.42:0/3176782205 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa9540a9050 con 0x7fa96409c5d0 2024-09-16T03:51:10.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.385+0000 7fa969892640 5 monclient: authenticate success, global_id 814106 2024-09-16T03:51:10.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.385+0000 7fa969892640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:51:10.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.385+0000 7fa961ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T03:51:10.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.385+0000 7fa969892640 10 monclient: get_monmap_and_config success 2024-09-16T03:51:10.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.385+0000 7fa969892640 4 set_mon_vals no callback set 2024-09-16T03:51:10.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.385+0000 7fa961ffb640 1 -- 172.21.15.42:0/3176782205 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9540ad860 con 0x7fa96409c5d0 2024-09-16T03:51:10.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.385+0000 7fa961ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:51:10.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.385+0000 7fa961ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:51:10.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.385+0000 7fa961ffb640 10 monclient: dump: 2024-09-16T03:51:10.400 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:51:10.400 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:10.400 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:10.400 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:10.400 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:51:10.400 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:10.400 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:51:10.401 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:51:10.401 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:51:10.401 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:10.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.385+0000 7fa969892640 10 monclient: shutdown 2024-09-16T03:51:10.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.385+0000 7fa969892640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:51:10.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.385+0000 7fa969892640 1 -- 172.21.15.42:0/3176782205 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa96409c5d0 msgr2=0x7fa96413b670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:10.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.385+0000 7fa969892640 1 --2- 172.21.15.42:0/3176782205 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa96409c5d0 0x7fa96413b670 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fa9540097c0 tx=0x7fa95401d470 comp rx=0 tx=0).stop 2024-09-16T03:51:10.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.385+0000 7fa969892640 1 -- 172.21.15.42:0/3176782205 shutdown_connections 2024-09-16T03:51:10.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.385+0000 7fa969892640 1 --2- 172.21.15.42:0/3176782205 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa96409c5d0 0x7fa96413b670 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:10.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.385+0000 7fa969892640 1 --2- 172.21.15.42:0/3176782205 >> v2:172.21.15.154:3300/0 conn(0x7fa96409c160 0x7fa964139120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:10.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.385+0000 7fa969892640 1 -- 172.21.15.42:0/3176782205 >> 172.21.15.42:0/3176782205 conn(0x7fa9640a2420 msgr2=0x7fa9640a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:51:10.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.385+0000 7fa969892640 1 -- 172.21.15.42:0/3176782205 shutdown_connections 2024-09-16T03:51:10.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.385+0000 7fa969892640 1 -- 172.21.15.42:0/3176782205 wait complete. 2024-09-16T03:51:10.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.385+0000 7fa969892640 10 monclient: build_initial_monmap 2024-09-16T03:51:10.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.385+0000 7fa969892640 10 monclient: monmap: 2024-09-16T03:51:10.402 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:51:10.402 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:51:10.402 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:51:10.402 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:51:10.402 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:51:10.402 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:10.402 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:51:10.402 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:51:10.402 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:51:10.402 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:10.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.385+0000 7fa969892640 1 librados: starting msgr at 2024-09-16T03:51:10.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.385+0000 7fa969892640 1 librados: starting objecter 2024-09-16T03:51:10.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.385+0000 7fa969892640 1 Processor -- start 2024-09-16T03:51:10.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.385+0000 7fa969892640 1 -- start start 2024-09-16T03:51:10.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.385+0000 7fa969892640 1 librados: setting wanted keys 2024-09-16T03:51:10.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.385+0000 7fa969892640 1 librados: calling monclient init 2024-09-16T03:51:10.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.385+0000 7fa969892640 10 monclient: init 2024-09-16T03:51:10.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.385+0000 7fa969892640 10 monclient: _reopen_session rank -1 2024-09-16T03:51:10.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.385+0000 7fa969892640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:51:10.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.385+0000 7fa969892640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa96409c160 0x7fa96413a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:10.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.385+0000 7fa969892640 10 monclient(hunting): picked mon.noname-b con 0x7fa96409c160 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:51:10.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.385+0000 7fa969892640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa96409c5d0 0x7fa9641321c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:10.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.385+0000 7fa969892640 10 monclient(hunting): picked mon.noname-a con 0x7fa96409c5d0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:51:10.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.385+0000 7fa969892640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa96409fb00 0x7fa964132700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:10.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.385+0000 7fa969892640 10 monclient(hunting): picked mon.noname-c con 0x7fa96409fb00 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:51:10.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.385+0000 7fa969892640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:10.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.385+0000 7fa969892640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fa96409d380 con 0x7fa96409c5d0 2024-09-16T03:51:10.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.385+0000 7fa969892640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:10.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.385+0000 7fa969892640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa96409d130 con 0x7fa96409c160 2024-09-16T03:51:10.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.385+0000 7fa969892640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:10.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.385+0000 7fa969892640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fa96409d500 con 0x7fa96409fb00 2024-09-16T03:51:10.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.385+0000 7fa969892640 10 monclient(hunting): _renew_subs 2024-09-16T03:51:10.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.385+0000 7fa969892640 10 monclient(hunting): authenticate will time out at 1517.359556s 2024-09-16T03:51:10.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.385+0000 7fa9627fc640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa96409c5d0 0x7fa9641321c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:10.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.385+0000 7fa9627fc640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa96409c5d0 0x7fa9641321c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:53008/0 (socket says 172.21.15.42:53008) 2024-09-16T03:51:10.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.385+0000 7fa9637fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa96409fb00 0x7fa964132700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:10.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.385+0000 7fa9627fc640 1 -- 172.21.15.42:0/186660321 learned_addr learned my addr 172.21.15.42:0/186660321 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:51:10.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.385+0000 7fa9627fc640 10 monclient(hunting): get_auth_request con 0x7fa96409c5d0 auth_method 0 2024-09-16T03:51:10.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.385+0000 7fa9627fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:10.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.385+0000 7fa9627fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:10.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.385+0000 7fa9627fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:51:10.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.385+0000 7fa9637fe640 10 monclient(hunting): get_auth_request con 0x7fa96409fb00 auth_method 0 2024-09-16T03:51:10.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.385+0000 7fa9637fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:10.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.385+0000 7fa9637fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:10.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.385+0000 7fa9637fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:51:10.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.385+0000 7fa9637fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:51:10.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.385+0000 7fa9637fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:51:10.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.385+0000 7fa9637fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:51:10.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.385+0000 7fa9627fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:51:10.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.385+0000 7fa9627fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:51:10.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.385+0000 7fa9627fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:51:10.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.389+0000 7fa9637fe640 10 monclient(hunting): handle_auth_done global_id 814109 payload 824 2024-09-16T03:51:10.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.389+0000 7fa962ffd640 1 --2- 172.21.15.42:0/186660321 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa96409c160 0x7fa96413a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:10.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.389+0000 7fa9637fe640 1 -- 172.21.15.42:0/186660321 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa96409c160 msgr2=0x7fa96413a1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:10.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.389+0000 7fa9637fe640 1 --2- 172.21.15.42:0/186660321 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa96409c160 0x7fa96413a1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:10.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.389+0000 7fa9637fe640 1 -- 172.21.15.42:0/186660321 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa96409c5d0 msgr2=0x7fa9641321c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:10.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.389+0000 7fa9637fe640 1 --2- 172.21.15.42:0/186660321 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa96409c5d0 0x7fa9641321c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:10.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.389+0000 7fa9637fe640 10 monclient: _finish_hunting 0 2024-09-16T03:51:10.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.389+0000 7fa9637fe640 1 monclient: found mon.noname-c 2024-09-16T03:51:10.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.389+0000 7fa9637fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:51:10.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.389+0000 7fa9637fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:51:10.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.389+0000 7fa9637fe640 1 -- 172.21.15.42:0/186660321 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9641cefe0 con 0x7fa96409fb00 2024-09-16T03:51:10.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.389+0000 7fa9627fc640 1 --2- 172.21.15.42:0/186660321 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa96409c5d0 0x7fa9641321c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:51:10.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.389+0000 7fa962ffd640 10 monclient: get_auth_request con 0x7fa96409c160 auth_method 0 2024-09-16T03:51:10.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.389+0000 7fa962ffd640 1 --2- 172.21.15.42:0/186660321 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa96409c160 0x7fa96413a1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T03:51:10.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.389+0000 7fa9637fe640 1 --2- 172.21.15.42:0/186660321 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa96409fb00 0x7fa964132700 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fa9540ad000 tx=0x7fa95401a210 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:51:10.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.389+0000 7fa94bfff640 10 client.?.objecter ms_handle_connect 0x7fa96409fb00 2024-09-16T03:51:10.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.389+0000 7fa94bfff640 10 client.?.objecter resend_mon_ops 2024-09-16T03:51:10.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.389+0000 7fa94bfff640 1 -- 172.21.15.42:0/186660321 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9540ba070 con 0x7fa96409fb00 2024-09-16T03:51:10.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.389+0000 7fa94bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:51:10.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.389+0000 7fa94bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:51:10.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.389+0000 7fa94bfff640 10 monclient: dump: 2024-09-16T03:51:10.407 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:51:10.407 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:10.407 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:10.407 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:10.407 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:51:10.407 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:10.407 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:51:10.407 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:51:10.408 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:51:10.408 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:10.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.389+0000 7fa94bfff640 10 monclient: _finish_auth 0 2024-09-16T03:51:10.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.389+0000 7fa94bfff640 10 monclient: _check_auth_tickets 2024-09-16T03:51:10.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.389+0000 7fa94bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:51:10.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.389+0000 7fa94bfff640 1 -- 172.21.15.42:0/186660321 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa9540b6410 con 0x7fa96409fb00 2024-09-16T03:51:10.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.389+0000 7fa94bfff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:51:10.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.389+0000 7fa94bfff640 1 -- 172.21.15.42:0/186660321 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9540b2040 con 0x7fa96409fb00 2024-09-16T03:51:10.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.389+0000 7fa94bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:51:10.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.389+0000 7fa94bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:51:10.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.389+0000 7fa94bfff640 10 monclient: dump: 2024-09-16T03:51:10.408 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:51:10.408 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:10.408 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:10.409 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:10.409 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:51:10.409 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:10.409 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:51:10.409 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:51:10.409 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:51:10.409 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:10.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.389+0000 7fa969892640 5 monclient: authenticate success, global_id 814109 2024-09-16T03:51:10.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.389+0000 7fa969892640 10 monclient: _renew_subs 2024-09-16T03:51:10.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.389+0000 7fa969892640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:51:10.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.389+0000 7fa9617fa640 4 set_mon_vals no callback set 2024-09-16T03:51:10.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.389+0000 7fa969892640 1 -- 172.21.15.42:0/186660321 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa964120b90 con 0x7fa96409fb00 2024-09-16T03:51:10.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.389+0000 7fa969892640 10 client.814109.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:51:10.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.389+0000 7fa969892640 10 monclient: _renew_subs 2024-09-16T03:51:10.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.389+0000 7fa969892640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:51:10.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.389+0000 7fa969892640 1 -- 172.21.15.42:0/186660321 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa96406da90 con 0x7fa96409fb00 2024-09-16T03:51:10.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.389+0000 7fa969892640 1 librados: init done 2024-09-16T03:51:10.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.389+0000 7fa969892640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:51:10.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.389+0000 7fa969892640 1 --2- 172.21.15.42:0/186660321 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa964120530 0x7fa964137500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:10.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.389+0000 7fa969892640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:10.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.389+0000 7fa969892640 1 -- 172.21.15.42:0/186660321 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fa96409ba50 con 0x7fa964120530 2024-09-16T03:51:10.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.389+0000 7fa94bfff640 1 -- 172.21.15.42:0/186660321 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fa9540bd070 con 0x7fa96409fb00 2024-09-16T03:51:10.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.389+0000 7fa962ffd640 1 --2- 172.21.15.42:0/186660321 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa964120530 0x7fa964137500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:10.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.389+0000 7fa962ffd640 10 monclient: get_auth_request con 0x7fa964120530 auth_method 0 2024-09-16T03:51:10.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.389+0000 7fa962ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:10.411 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.389+0000 7fa962ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:10.411 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.389+0000 7fa962ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:51:10.411 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.389+0000 7fa962ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:51:10.411 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.389+0000 7fa962ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:51:10.411 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.389+0000 7fa962ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:51:10.411 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.389+0000 7fa94bfff640 1 --2- 172.21.15.42:0/186660321 >> v2:172.21.15.154:6812/254845342 conn(0x7fa950037840 0x7fa950039d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:10.411 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.389+0000 7fa962ffd640 10 monclient(hunting): handle_auth_done global_id 764880 payload 824 2024-09-16T03:51:10.411 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.389+0000 7fa94bfff640 1 -- 172.21.15.42:0/186660321 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(306..306 src has 1..306) ==== 6552+0+0 (secure 0 0 0) 0x7fa9540e2da0 con 0x7fa96409fb00 2024-09-16T03:51:10.411 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.389+0000 7fa962ffd640 1 -- 172.21.15.42:0/186660321 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa964120910 con 0x7fa964120530 2024-09-16T03:51:10.411 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.389+0000 7fa94bfff640 10 client.814109.objecter ms_dispatch 0x7fa964000ba0 osd_map(306..306 src has 1..306) 2024-09-16T03:51:10.411 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.389+0000 7fa94bfff640 3 client.814109.objecter handle_osd_map got epochs [306,306] > 0 2024-09-16T03:51:10.411 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.389+0000 7fa94bfff640 3 client.814109.objecter handle_osd_map decoding full epoch 306 2024-09-16T03:51:10.411 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.389+0000 7fa94bfff640 20 client.814109.objecter dump_active .. 0 homeless 2024-09-16T03:51:10.411 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.389+0000 7fa94bfff640 10 client.814109.objecter ms_handle_connect 0x7fa964120530 2024-09-16T03:51:10.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.389+0000 7fa962ffd640 1 --2- 172.21.15.42:0/186660321 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa964120530 0x7fa964137500 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fa95c00da60 tx=0x7fa95c00df20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:51:10.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.389+0000 7fa94bfff640 10 client.814109.objecter resend_mon_ops 2024-09-16T03:51:10.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.389+0000 7fa9627fc640 1 --2- 172.21.15.42:0/186660321 >> v2:172.21.15.154:6812/254845342 conn(0x7fa950037840 0x7fa950039d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:10.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.389+0000 7fa94bfff640 1 -- 172.21.15.42:0/186660321 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa95c018070 con 0x7fa964120530 2024-09-16T03:51:10.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.389+0000 7fa94bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:51:10.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.389+0000 7fa94bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:51:10.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.389+0000 7fa94bfff640 10 monclient: dump: 2024-09-16T03:51:10.412 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:51:10.412 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:10.412 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:10.412 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:10.412 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:51:10.412 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:10.413 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:51:10.413 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:51:10.413 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:51:10.413 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:10.413 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.389+0000 7fa9627fc640 10 monclient: get_auth_request con 0x7fa950037840 auth_method 0 2024-09-16T03:51:10.413 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.389+0000 7fa94bfff640 1 -- 172.21.15.42:0/186660321 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fa95c00b720 con 0x7fa964120530 2024-09-16T03:51:10.413 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.389+0000 7fa94bfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:51:10.413 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.389+0000 7fa94bfff640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:51:10.413 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.389+0000 7fa94bfff640 1 -- 172.21.15.42:0/186660321 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa964120530 msgr2=0x7fa964137500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:10.413 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.389+0000 7fa94bfff640 1 --2- 172.21.15.42:0/186660321 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa964120530 0x7fa964137500 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fa95c00da60 tx=0x7fa95c00df20 comp rx=0 tx=0).stop 2024-09-16T03:51:10.413 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.389+0000 7fa94bfff640 1 -- 172.21.15.42:0/186660321 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa964120530 msgr2=0x7fa964137500 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:51:10.413 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.389+0000 7fa94bfff640 1 --2- 172.21.15.42:0/186660321 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa964120530 0x7fa964137500 secure :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fa95c00da60 tx=0x7fa95c00df20 comp rx=0 tx=0).stop 2024-09-16T03:51:10.413 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.389+0000 7fa94bfff640 10 client.814109.objecter ms_handle_connect 0x7fa950037840 2024-09-16T03:51:10.413 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.389+0000 7fa9627fc640 1 --2- 172.21.15.42:0/186660321 >> v2:172.21.15.154:6812/254845342 conn(0x7fa950037840 0x7fa950039d00 secure :-1 s=READY pgs=3551 cs=0 l=1 rev1=1 crypto rx=0x7fa964070e60 tx=0x7fa958008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:51:10.415 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.397+0000 7fa969892640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:51:10.415 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.397+0000 7fa969892640 1 --2- 172.21.15.42:0/186660321 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa964137a40 0x7fa964137e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:10.415 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.397+0000 7fa969892640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:10.415 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.397+0000 7fa969892640 1 -- 172.21.15.42:0/186660321 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fa964138390 con 0x7fa964137a40 2024-09-16T03:51:10.415 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.397+0000 7fa962ffd640 1 --2- 172.21.15.42:0/186660321 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa964137a40 0x7fa964137e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:10.415 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.397+0000 7fa962ffd640 10 monclient: get_auth_request con 0x7fa964137a40 auth_method 0 2024-09-16T03:51:10.415 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.397+0000 7fa962ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:10.416 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.397+0000 7fa962ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:10.416 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.397+0000 7fa962ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:51:10.416 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.397+0000 7fa962ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:51:10.416 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.397+0000 7fa962ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:51:10.416 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.397+0000 7fa962ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:51:10.416 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.397+0000 7fa962ffd640 10 monclient(hunting): handle_auth_done global_id 764883 payload 824 2024-09-16T03:51:10.416 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.397+0000 7fa962ffd640 1 -- 172.21.15.42:0/186660321 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fa9641d9e90 con 0x7fa964137a40 2024-09-16T03:51:10.416 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.397+0000 7fa94bfff640 10 client.814109.objecter ms_handle_connect 0x7fa964137a40 2024-09-16T03:51:10.416 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.397+0000 7fa94bfff640 10 client.814109.objecter resend_mon_ops 2024-09-16T03:51:10.416 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.397+0000 7fa962ffd640 1 --2- 172.21.15.42:0/186660321 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa964137a40 0x7fa964137e50 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fa95c00da60 tx=0x7fa95c001410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:51:10.416 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.397+0000 7fa94bfff640 1 -- 172.21.15.42:0/186660321 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa95c018050 con 0x7fa964137a40 2024-09-16T03:51:10.416 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.397+0000 7fa94bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:51:10.416 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.397+0000 7fa94bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:51:10.417 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.397+0000 7fa94bfff640 10 monclient: dump: 2024-09-16T03:51:10.417 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:51:10.417 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:10.417 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:10.417 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:10.417 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:51:10.417 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:10.417 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:51:10.417 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:51:10.417 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:51:10.417 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:10.417 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.397+0000 7fa94bfff640 1 -- 172.21.15.42:0/186660321 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4881 (secure 0 0 0) 0x7fa95c019040 con 0x7fa964137a40 2024-09-16T03:51:10.417 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.397+0000 7fa94bfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:51:10.417 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.397+0000 7fa94bfff640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:51:10.418 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.397+0000 7fa94bfff640 1 -- 172.21.15.42:0/186660321 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa964137a40 msgr2=0x7fa964137e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:10.418 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.397+0000 7fa94bfff640 1 --2- 172.21.15.42:0/186660321 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa964137a40 0x7fa964137e50 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fa95c00da60 tx=0x7fa95c001410 comp rx=0 tx=0).stop 2024-09-16T03:51:10.418 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.397+0000 7fa94bfff640 1 -- 172.21.15.42:0/186660321 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa964137a40 msgr2=0x7fa964137e50 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:51:10.418 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.397+0000 7fa94bfff640 1 --2- 172.21.15.42:0/186660321 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa964137a40 0x7fa964137e50 secure :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fa95c00da60 tx=0x7fa95c001410 comp rx=0 tx=0).stop 2024-09-16T03:51:10.418 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:51:10.418 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:51:10.418 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:51:10.418 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "leader", 2024-09-16T03:51:10.418 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 80, 2024-09-16T03:51:10.418 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 261189, 2024-09-16T03:51:10.418 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:51:10.419 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:51:10.419 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T03:51:10.419 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T03:51:10.419 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:51:10.419 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 0, 2024-09-16T03:51:10.419 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:51:10.419 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:51:10.419 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:51:10.419 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:51:10.419 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:51:10.419 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:51:10.419 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:51:10.419 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:51:10.419 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:51:10.420 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:51:10.420 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:51:10.420 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:51:10.420 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:51:10.420 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:51:10.420 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:51:10.420 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:51:10.420 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:51:10.420 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:51:10.420 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:51:10.420 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:51:10.420 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:51:10.420 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:51:10.421 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:51:10.421 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:51:10.421 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:51:10.421 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:51:10.421 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:51:10.421 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:51:10.421 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:51:10.421 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:10.421 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:51:10.421 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:51:10.421 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:51:10.421 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:51:10.421 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:51:10.422 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:51:10.422 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:51:10.422 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:51:10.422 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:51:10.422 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:51:10.422 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:51:10.422 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:51:10.422 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:51:10.422 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:51:10.422 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:51:10.422 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:51:10.422 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:51:10.422 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:51:10.422 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:51:10.423 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:51:10.423 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:51:10.423 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:51:10.423 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:51:10.423 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:51:10.423 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:51:10.423 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:51:10.423 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:51:10.423 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:51:10.423 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:51:10.423 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:51:10.423 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:10.423 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:51:10.423 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:10.424 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:51:10.424 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:51:10.424 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:51:10.424 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:51:10.424 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:10.424 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:51:10.424 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:51:10.424 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:51:10.424 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:10.424 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:10.424 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:51:10.424 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:51:10.424 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:51:10.425 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:51:10.425 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:51:10.425 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:10.425 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:51:10.425 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:51:10.425 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:51:10.425 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:51:10.425 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:51:10.425 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:10.425 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:10.425 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:51:10.425 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:51:10.425 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:51:10.425 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:51:10.425 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:10.426 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:51:10.426 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:51:10.426 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:51:10.426 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:10.426 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:10.426 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:51:10.426 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:51:10.426 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:51:10.426 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:51:10.426 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:51:10.426 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:10.426 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:51:10.426 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:51:10.427 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:51:10.427 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:51:10.427 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:51:10.427 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:10.427 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:10.427 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:51:10.427 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:51:10.427 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:51:10.427 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:51:10.427 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:10.427 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:51:10.427 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:51:10.427 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:51:10.427 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:10.428 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:10.428 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:51:10.428 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:51:10.428 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:51:10.428 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:51:10.428 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:51:10.428 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:10.428 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:51:10.428 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:51:10.428 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:51:10.428 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:51:10.428 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:51:10.428 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:51:10.428 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:51:10.429 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:10.429 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:51:10.429 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:51:10.429 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:10.429 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:51:10.429 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:51:10.429 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:51:10.429 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:51:10.429 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:51:10.429 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T03:51:10.429 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:10.429 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:51:10.429 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:51:10.430 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 6 2024-09-16T03:51:10.430 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:51:10.430 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:51:10.430 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:51:10.430 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:10.430 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:51:10.430 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:51:10.430 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 6 2024-09-16T03:51:10.430 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:51:10.430 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:51:10.430 INFO:teuthology.orchestra.run.smithi042.stdout: "mgr": [ 2024-09-16T03:51:10.430 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:10.430 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:51:10.430 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:51:10.431 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:51:10.431 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:51:10.431 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:51:10.431 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:10.431 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:51:10.431 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:51:10.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.397+0000 7fa969892640 10 librados: watch_flush enter 2024-09-16T03:51:10.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.397+0000 7fa969892640 10 librados: watch_flush exit 2024-09-16T03:51:10.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.397+0000 7fa969892640 20 client.814109.objecter shutdown clearing up homeless session... 2024-09-16T03:51:10.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.397+0000 7fa969892640 10 client.814109.objecter successfully canceled tick 2024-09-16T03:51:10.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.397+0000 7fa969892640 1 -- 172.21.15.42:0/186660321 >> v2:172.21.15.154:6812/254845342 conn(0x7fa950037840 msgr2=0x7fa950039d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:10.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.397+0000 7fa969892640 1 --2- 172.21.15.42:0/186660321 >> v2:172.21.15.154:6812/254845342 conn(0x7fa950037840 0x7fa950039d00 secure :-1 s=READY pgs=3551 cs=0 l=1 rev1=1 crypto rx=0x7fa964070e60 tx=0x7fa958008040 comp rx=0 tx=0).stop 2024-09-16T03:51:10.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.397+0000 7fa969892640 10 monclient: shutdown 2024-09-16T03:51:10.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.397+0000 7fa969892640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:51:10.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.397+0000 7fa969892640 1 -- 172.21.15.42:0/186660321 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa96409fb00 msgr2=0x7fa964132700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:10.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.397+0000 7fa969892640 1 --2- 172.21.15.42:0/186660321 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa96409fb00 0x7fa964132700 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fa9540ad000 tx=0x7fa95401a210 comp rx=0 tx=0).stop 2024-09-16T03:51:10.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.397+0000 7fa9637fe640 1 -- 172.21.15.42:0/186660321 reap_dead start 2024-09-16T03:51:10.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.397+0000 7fa9637fe640 1 -- 172.21.15.42:0/186660321 reap_dead start 2024-09-16T03:51:10.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.397+0000 7fa969892640 1 -- 172.21.15.42:0/186660321 shutdown_connections 2024-09-16T03:51:10.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.397+0000 7fa969892640 1 -- 172.21.15.42:0/186660321 >> 172.21.15.42:0/186660321 conn(0x7fa9640a2420 msgr2=0x7fa9641332f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:51:10.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.397+0000 7fa969892640 1 -- 172.21.15.42:0/186660321 shutdown_connections 2024-09-16T03:51:10.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.397+0000 7fa969892640 1 -- 172.21.15.42:0/186660321 wait complete. 2024-09-16T03:51:10.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.397+0000 7fa969892640 1 librados: shutdown 2024-09-16T03:51:10.434 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:51:10.434 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T03:51:10.435 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T03:51:10.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.545+0000 7fac26051640 10 monclient: get_monmap_and_config 2024-09-16T03:51:10.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.545+0000 7fac26051640 10 monclient: build_initial_monmap 2024-09-16T03:51:10.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.545+0000 7fac26051640 10 monclient: monmap: 2024-09-16T03:51:10.550 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:51:10.550 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:10.550 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:51:10.549025+0000 2024-09-16T03:51:10.550 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:51:10.549025+0000 2024-09-16T03:51:10.550 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:51:10.551 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:10.551 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:51:10.551 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:51:10.551 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:51:10.551 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:51:10.551 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:51:10.551 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:10.552 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.545+0000 7fac26051640 1 Processor -- start 2024-09-16T03:51:10.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.545+0000 7fac26051640 1 -- start start 2024-09-16T03:51:10.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.545+0000 7fac26051640 10 monclient: init 2024-09-16T03:51:10.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.545+0000 7fac26051640 10 monclient: _reopen_session rank -1 2024-09-16T03:51:10.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.545+0000 7fac26051640 10 monclient: _add_conns ranks=[3,1,0,4,2] 2024-09-16T03:51:10.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.545+0000 7fac26051640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fac200a1330 0x7fac200a1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:10.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.545+0000 7fac26051640 10 monclient(hunting): picked mon.noname-a con 0x7fac200a1330 addr v2:172.21.15.42:3300/0 2024-09-16T03:51:10.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.545+0000 7fac26051640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fac2006dac0 addr v1:172.21.15.154:6789/0 2024-09-16T03:51:10.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.545+0000 7fac26051640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fac200a1c50 addr v1:172.21.15.42:6789/0 2024-09-16T03:51:10.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.545+0000 7fac26051640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fac20135f20 con 0x7fac200a1c50 2024-09-16T03:51:10.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.545+0000 7fac26051640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fac2009c090 con 0x7fac2006dac0 2024-09-16T03:51:10.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.545+0000 7fac26051640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:10.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.545+0000 7fac26051640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7fac2009f610 con 0x7fac200a1330 2024-09-16T03:51:10.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.545+0000 7fac26051640 10 monclient(hunting): _renew_subs 2024-09-16T03:51:10.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.545+0000 7fac26051640 10 monclient(hunting): authenticate will time out at 1517.518564s 2024-09-16T03:51:10.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.545+0000 7fac1ffff640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fac200a1c50 0x7fac20135810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:50502/0 (socket says 172.21.15.42:50502) 2024-09-16T03:51:10.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.545+0000 7fac1ffff640 1 -- 172.21.15.42:0/2298552664 learned_addr learned my addr 172.21.15.42:0/2298552664 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:51:10.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.545+0000 7fac1f7fe640 1 --2- 172.21.15.42:0/2298552664 >> v2:172.21.15.42:3300/0 conn(0x7fac200a1330 0x7fac200a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:10.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.545+0000 7fac1f7fe640 10 monclient(hunting): get_auth_request con 0x7fac200a1330 auth_method 0 2024-09-16T03:51:10.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.545+0000 7fac1f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:10.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.545+0000 7fac1f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:10.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.545+0000 7fac1f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:51:10.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.545+0000 7fac1f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:51:10.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.545+0000 7fac1f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:51:10.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.545+0000 7fac1f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:51:10.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.545+0000 7fac1f7fe640 10 monclient(hunting): handle_auth_done global_id 764886 payload 274 2024-09-16T03:51:10.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.545+0000 7fac1f7fe640 1 -- 172.21.15.42:0/2298552664 >> v1:172.21.15.154:6789/0 conn(0x7fac2006dac0 legacy=0x7fac20135100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:10.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.545+0000 7fac1f7fe640 1 -- 172.21.15.42:0/2298552664 >> v1:172.21.15.42:6789/0 conn(0x7fac200a1c50 legacy=0x7fac20135810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:10.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.545+0000 7fac1f7fe640 10 monclient: _finish_hunting 0 2024-09-16T03:51:10.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.545+0000 7fac1f7fe640 1 monclient: found mon.noname-a 2024-09-16T03:51:10.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.545+0000 7fac1f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:51:10.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.545+0000 7fac1f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:51:10.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.545+0000 7fac1f7fe640 1 -- 172.21.15.42:0/2298552664 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac2009f8f0 con 0x7fac200a1330 2024-09-16T03:51:10.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.549+0000 7fac1f7fe640 1 --2- 172.21.15.42:0/2298552664 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fac200a1330 0x7fac200a1710 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fac180097c0 tx=0x7fac1801c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=984212cd52baffa0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:51:10.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.549+0000 7fac1e7fc640 1 -- 172.21.15.42:0/2298552664 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac180b1070 con 0x7fac200a1330 2024-09-16T03:51:10.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.549+0000 7fac1e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:51:10.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.549+0000 7fac1e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:51:10.565 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.549+0000 7fac1e7fc640 10 monclient: dump: 2024-09-16T03:51:10.565 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:51:10.565 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:10.565 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:10.565 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:10.565 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:51:10.565 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:10.565 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:51:10.565 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:51:10.565 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:51:10.565 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:10.565 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.549+0000 7fac1e7fc640 10 monclient: _finish_auth 0 2024-09-16T03:51:10.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.549+0000 7fac1e7fc640 10 monclient: _check_auth_tickets 2024-09-16T03:51:10.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.549+0000 7fac1e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:51:10.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.549+0000 7fac1e7fc640 1 -- 172.21.15.42:0/2298552664 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac180a8050 con 0x7fac200a1330 2024-09-16T03:51:10.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.549+0000 7fac26051640 5 monclient: authenticate success, global_id 764886 2024-09-16T03:51:10.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.549+0000 7fac26051640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:51:10.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.549+0000 7fac1e7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:51:10.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.549+0000 7fac26051640 10 monclient: get_monmap_and_config success 2024-09-16T03:51:10.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.549+0000 7fac26051640 4 set_mon_vals no callback set 2024-09-16T03:51:10.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.549+0000 7fac1e7fc640 1 -- 172.21.15.42:0/2298552664 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac180ac860 con 0x7fac200a1330 2024-09-16T03:51:10.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.549+0000 7fac1e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:51:10.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.549+0000 7fac1e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:51:10.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.549+0000 7fac1e7fc640 10 monclient: dump: 2024-09-16T03:51:10.567 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:51:10.567 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:10.567 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:10.567 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:10.567 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:51:10.567 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:10.567 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:51:10.567 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:51:10.567 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:51:10.568 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:10.568 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.549+0000 7fac26051640 10 monclient: shutdown 2024-09-16T03:51:10.568 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.549+0000 7fac26051640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:51:10.568 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.549+0000 7fac26051640 1 -- 172.21.15.42:0/2298552664 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fac200a1330 msgr2=0x7fac200a1710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:10.568 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.549+0000 7fac26051640 1 --2- 172.21.15.42:0/2298552664 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fac200a1330 0x7fac200a1710 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fac180097c0 tx=0x7fac1801c3c0 comp rx=0 tx=0).stop 2024-09-16T03:51:10.568 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.549+0000 7fac26051640 1 -- 172.21.15.42:0/2298552664 shutdown_connections 2024-09-16T03:51:10.568 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.549+0000 7fac26051640 1 --2- 172.21.15.42:0/2298552664 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fac200a1330 0x7fac200a1710 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:10.568 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.549+0000 7fac26051640 1 -- 172.21.15.42:0/2298552664 >> 172.21.15.42:0/2298552664 conn(0x7fac200a2420 msgr2=0x7fac200a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:51:10.569 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.549+0000 7fac26051640 1 -- 172.21.15.42:0/2298552664 shutdown_connections 2024-09-16T03:51:10.569 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.549+0000 7fac26051640 1 -- 172.21.15.42:0/2298552664 wait complete. 2024-09-16T03:51:10.569 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.549+0000 7fac26051640 10 monclient: build_initial_monmap 2024-09-16T03:51:10.569 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.549+0000 7fac26051640 10 monclient: monmap: 2024-09-16T03:51:10.569 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:51:10.569 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:51:10.569 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:51:10.569 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:51:10.569 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:51:10.569 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:10.569 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:51:10.570 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:51:10.570 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:51:10.570 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:10.570 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.549+0000 7fac26051640 1 librados: starting msgr at 2024-09-16T03:51:10.570 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.549+0000 7fac26051640 1 librados: starting objecter 2024-09-16T03:51:10.570 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.549+0000 7fac26051640 1 Processor -- start 2024-09-16T03:51:10.570 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.549+0000 7fac26051640 1 -- start start 2024-09-16T03:51:10.570 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.549+0000 7fac26051640 1 librados: setting wanted keys 2024-09-16T03:51:10.570 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.549+0000 7fac26051640 1 librados: calling monclient init 2024-09-16T03:51:10.570 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.549+0000 7fac26051640 10 monclient: init 2024-09-16T03:51:10.570 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.549+0000 7fac26051640 10 monclient: _reopen_session rank -1 2024-09-16T03:51:10.570 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.549+0000 7fac26051640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T03:51:10.570 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.549+0000 7fac26051640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fac200a1330 0x7fac2013cf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:10.570 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac26051640 10 monclient(hunting): picked mon.noname-b con 0x7fac200a1330 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:51:10.570 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac26051640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fac200a1c50 0x7fac201352b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:10.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac26051640 10 monclient(hunting): picked mon.noname-c con 0x7fac200a1c50 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:51:10.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac26051640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fac201357f0 0x7fac2009cb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:10.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac26051640 10 monclient(hunting): picked mon.noname-a con 0x7fac201357f0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:51:10.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac26051640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:10.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac26051640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fac2009eef0 con 0x7fac201357f0 2024-09-16T03:51:10.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac26051640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:10.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac26051640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fac2009ed70 con 0x7fac200a1330 2024-09-16T03:51:10.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac26051640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:10.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac26051640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fac20140b90 con 0x7fac200a1c50 2024-09-16T03:51:10.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac26051640 10 monclient(hunting): _renew_subs 2024-09-16T03:51:10.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac26051640 10 monclient(hunting): authenticate will time out at 1517.524061s 2024-09-16T03:51:10.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac1ffff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fac201357f0 0x7fac2009cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:10.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac1effd640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fac200a1c50 0x7fac201352b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:10.572 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac1ffff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fac201357f0 0x7fac2009cb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:53032/0 (socket says 172.21.15.42:53032) 2024-09-16T03:51:10.572 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac1ffff640 1 -- 172.21.15.42:0/1520243905 learned_addr learned my addr 172.21.15.42:0/1520243905 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:51:10.572 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac1effd640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fac200a1c50 0x7fac201352b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:43428/0 (socket says 172.21.15.42:43428) 2024-09-16T03:51:10.572 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac1ffff640 10 monclient(hunting): get_auth_request con 0x7fac201357f0 auth_method 0 2024-09-16T03:51:10.572 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac1ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:10.572 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac1ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:10.572 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac1ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:51:10.572 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac1effd640 10 monclient(hunting): get_auth_request con 0x7fac200a1c50 auth_method 0 2024-09-16T03:51:10.572 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac1effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:10.572 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac1effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:10.572 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac1effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:51:10.572 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac1f7fe640 1 --2- 172.21.15.42:0/1520243905 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fac200a1330 0x7fac2013cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:10.572 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac1effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:51:10.573 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac1effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:51:10.573 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac1effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:51:10.573 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac1ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:51:10.573 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac1ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:51:10.573 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac1ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:51:10.573 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac1f7fe640 10 monclient(hunting): get_auth_request con 0x7fac200a1330 auth_method 0 2024-09-16T03:51:10.573 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac1f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:10.573 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac1f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:10.573 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac1f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:51:10.573 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac1f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:51:10.573 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac1f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:51:10.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac1f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:51:10.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac1effd640 10 monclient(hunting): handle_auth_done global_id 814112 payload 824 2024-09-16T03:51:10.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac1effd640 1 -- 172.21.15.42:0/1520243905 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fac200a1330 msgr2=0x7fac2013cf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:10.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac1effd640 1 --2- 172.21.15.42:0/1520243905 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fac200a1330 0x7fac2013cf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:10.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac1effd640 1 -- 172.21.15.42:0/1520243905 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fac201357f0 msgr2=0x7fac2009cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:10.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac1effd640 1 --2- 172.21.15.42:0/1520243905 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fac201357f0 0x7fac2009cb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:10.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac1effd640 10 monclient: _finish_hunting 0 2024-09-16T03:51:10.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac1effd640 1 monclient: found mon.noname-c 2024-09-16T03:51:10.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac1effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:51:10.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac1effd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:51:10.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac1effd640 1 -- 172.21.15.42:0/1520243905 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac2006e710 con 0x7fac200a1c50 2024-09-16T03:51:10.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac1ffff640 1 --2- 172.21.15.42:0/1520243905 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fac201357f0 0x7fac2009cb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:51:10.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac1effd640 1 --2- 172.21.15.42:0/1520243905 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fac200a1c50 0x7fac201352b0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fac0800c760 tx=0x7fac0800cc20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:51:10.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac17fff640 10 client.?.objecter ms_handle_connect 0x7fac200a1c50 2024-09-16T03:51:10.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac17fff640 10 client.?.objecter resend_mon_ops 2024-09-16T03:51:10.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac17fff640 1 -- 172.21.15.42:0/1520243905 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac08016020 con 0x7fac200a1c50 2024-09-16T03:51:10.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac17fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:51:10.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac17fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:51:10.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac17fff640 10 monclient: dump: 2024-09-16T03:51:10.575 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:51:10.575 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:10.575 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:10.575 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:10.575 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:51:10.575 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:10.575 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:51:10.576 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:51:10.576 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:51:10.576 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:10.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac17fff640 10 monclient: _finish_auth 0 2024-09-16T03:51:10.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac17fff640 10 monclient: _check_auth_tickets 2024-09-16T03:51:10.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac17fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:51:10.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac17fff640 1 -- 172.21.15.42:0/1520243905 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac08006d60 con 0x7fac200a1c50 2024-09-16T03:51:10.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac26051640 5 monclient: authenticate success, global_id 814112 2024-09-16T03:51:10.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac26051640 10 monclient: _renew_subs 2024-09-16T03:51:10.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac26051640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:51:10.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac26051640 1 -- 172.21.15.42:0/1520243905 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac20073a30 con 0x7fac200a1c50 2024-09-16T03:51:10.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac17fff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:51:10.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac17fff640 1 -- 172.21.15.42:0/1520243905 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac08003bc0 con 0x7fac200a1c50 2024-09-16T03:51:10.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac17fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:51:10.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac17fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:51:10.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac17fff640 10 monclient: dump: 2024-09-16T03:51:10.577 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:51:10.577 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:10.577 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:10.577 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:10.577 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:51:10.577 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:10.577 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:51:10.578 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:51:10.578 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:51:10.578 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:10.578 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac1dffb640 4 set_mon_vals no callback set 2024-09-16T03:51:10.578 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac26051640 10 client.814112.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:51:10.578 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac26051640 10 monclient: _renew_subs 2024-09-16T03:51:10.578 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac26051640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:51:10.578 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac26051640 1 -- 172.21.15.42:0/1520243905 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fac20120c50 con 0x7fac200a1c50 2024-09-16T03:51:10.578 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac26051640 1 librados: init done 2024-09-16T03:51:10.578 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac1effd640 1 -- 172.21.15.42:0/1520243905 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fac200a1c50 msgr2=0x7fac201352b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-16T03:51:10.578 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac1effd640 1 -- 172.21.15.42:0/1520243905 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fac200a1c50 msgr2=0x7fac201352b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T03:51:10.578 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac1effd640 1 --2- 172.21.15.42:0/1520243905 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fac200a1c50 0x7fac201352b0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fac0800c760 tx=0x7fac0800cc20 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:51:10.578 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac1effd640 1 --2- 172.21.15.42:0/1520243905 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fac200a1c50 0x7fac201352b0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fac0800c760 tx=0x7fac0800cc20 comp rx=0 tx=0).stop 2024-09-16T03:51:10.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac17fff640 10 monclient: ms_handle_reset current mon [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:51:10.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac17fff640 10 monclient: _reopen_session rank -1 2024-09-16T03:51:10.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac17fff640 1 -- 172.21.15.42:0/1520243905 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fac200a1c50 msgr2=0x7fac201352b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:51:10.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac17fff640 1 --2- 172.21.15.42:0/1520243905 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fac200a1c50 0x7fac201352b0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:10.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac17fff640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:51:10.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac17fff640 1 --2- 172.21.15.42:0/1520243905 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fac201357f0 0x7fac00007210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:10.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac17fff640 10 monclient(hunting): picked mon.a con 0x7fac201357f0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:51:10.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac17fff640 1 --2- 172.21.15.42:0/1520243905 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fac00007be0 0x7fac0000a070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:10.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac17fff640 10 monclient(hunting): picked mon.c con 0x7fac00007be0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:51:10.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac17fff640 1 --2- 172.21.15.42:0/1520243905 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fac200a1330 0x7fac0000cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:10.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac17fff640 10 monclient(hunting): picked mon.b con 0x7fac200a1330 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:51:10.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac17fff640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:10.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac17fff640 1 -- 172.21.15.42:0/1520243905 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fac0000d5a0 con 0x7fac201357f0 2024-09-16T03:51:10.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac17fff640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:10.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac17fff640 1 -- 172.21.15.42:0/1520243905 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fac0000d720 con 0x7fac200a1330 2024-09-16T03:51:10.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac17fff640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:10.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac17fff640 1 -- 172.21.15.42:0/1520243905 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fac0000d8a0 con 0x7fac00007be0 2024-09-16T03:51:10.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac17fff640 10 monclient(hunting): _renew_subs 2024-09-16T03:51:10.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac1effd640 1 --2- 172.21.15.42:0/1520243905 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fac00007be0 0x7fac0000a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:10.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac1effd640 10 monclient(hunting): get_auth_request con 0x7fac00007be0 auth_method 0 2024-09-16T03:51:10.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac1effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:10.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac1effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:10.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac1effd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:51:10.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac1f7fe640 1 --2- 172.21.15.42:0/1520243905 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fac201357f0 0x7fac00007210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:10.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac26051640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:51:10.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac26051640 1 --2- 172.21.15.42:0/1520243905 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fac2006dac0 0x7fac201352b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:10.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac26051640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:10.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac26051640 1 -- 172.21.15.42:0/1520243905 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fac201206f0 con 0x7fac2006dac0 2024-09-16T03:51:10.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac1f7fe640 10 monclient(hunting): get_auth_request con 0x7fac201357f0 auth_method 0 2024-09-16T03:51:10.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac1f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:10.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac1f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:10.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac1f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:51:10.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac1effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:51:10.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac1effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:51:10.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac1effd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T03:51:10.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac1f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:51:10.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac1f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:51:10.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac1f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T03:51:10.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac1ffff640 1 --2- 172.21.15.42:0/1520243905 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fac200a1330 0x7fac0000cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:10.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac1ffff640 10 monclient(hunting): get_auth_request con 0x7fac200a1330 auth_method 0 2024-09-16T03:51:10.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac1ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:10.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac1ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:10.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac1ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:51:10.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac1effd640 10 monclient(hunting): handle_auth_done global_id 814112 payload 293 2024-09-16T03:51:10.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac1effd640 1 -- 172.21.15.42:0/1520243905 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fac200a1330 msgr2=0x7fac0000cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:10.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac1effd640 1 --2- 172.21.15.42:0/1520243905 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fac200a1330 0x7fac0000cb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:10.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac1effd640 1 -- 172.21.15.42:0/1520243905 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fac201357f0 msgr2=0x7fac00007210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:10.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac1effd640 1 --2- 172.21.15.42:0/1520243905 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fac201357f0 0x7fac00007210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:10.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac1effd640 10 monclient: _finish_hunting 0 2024-09-16T03:51:10.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac1effd640 1 monclient: found mon.c 2024-09-16T03:51:10.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac1effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:51:10.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac1effd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:51:10.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac1effd640 1 -- 172.21.15.42:0/1520243905 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+,osdmap=0}) -- 0x7fac0000dbc0 con 0x7fac00007be0 2024-09-16T03:51:10.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac1effd640 10 monclient: _finish_auth 0 2024-09-16T03:51:10.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac1effd640 10 monclient: _check_auth_tickets 2024-09-16T03:51:10.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac1effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:51:10.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac1f7fe640 1 --2- 172.21.15.42:0/1520243905 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fac2006dac0 0x7fac201352b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:10.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac1f7fe640 10 monclient: get_auth_request con 0x7fac2006dac0 auth_method 0 2024-09-16T03:51:10.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac1f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:10.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac1f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:10.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac1f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:51:10.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac1f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:51:10.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac1f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:51:10.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac1f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:51:10.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac1effd640 1 --2- 172.21.15.42:0/1520243905 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fac00007be0 0x7fac0000a070 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fac08014000 tx=0x7fac08001690 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:51:10.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac17fff640 10 client.814112.objecter ms_handle_connect 0x7fac00007be0 2024-09-16T03:51:10.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac17fff640 10 client.814112.objecter resend_mon_ops 2024-09-16T03:51:10.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac1f7fe640 10 monclient(hunting): handle_auth_done global_id 814115 payload 824 2024-09-16T03:51:10.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac1f7fe640 1 -- 172.21.15.42:0/1520243905 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fac200a1710 con 0x7fac2006dac0 2024-09-16T03:51:10.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac17fff640 1 -- 172.21.15.42:0/1520243905 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac0802c070 con 0x7fac00007be0 2024-09-16T03:51:10.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac17fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:51:10.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac17fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:51:10.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac17fff640 10 monclient: dump: 2024-09-16T03:51:10.584 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:51:10.585 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:10.585 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:10.585 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:10.585 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:51:10.585 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:10.585 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:51:10.585 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:51:10.585 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:51:10.585 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:10.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac1f7fe640 1 --2- 172.21.15.42:0/1520243905 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fac2006dac0 0x7fac201352b0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fac2009c800 tx=0x7fac1801ac00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:51:10.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac17fff640 10 client.814112.objecter ms_handle_connect 0x7fac2006dac0 2024-09-16T03:51:10.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac17fff640 10 client.814112.objecter resend_mon_ops 2024-09-16T03:51:10.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac17fff640 1 -- 172.21.15.42:0/1520243905 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac08003e50 con 0x7fac00007be0 2024-09-16T03:51:10.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac17fff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:51:10.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac1d7fa640 4 set_mon_vals no callback set 2024-09-16T03:51:10.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac17fff640 1 -- 172.21.15.42:0/1520243905 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac180bd070 con 0x7fac2006dac0 2024-09-16T03:51:10.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac17fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:51:10.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac17fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:51:10.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac17fff640 10 monclient: dump: 2024-09-16T03:51:10.586 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:51:10.586 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:10.586 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:10.586 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:10.587 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:51:10.587 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:10.587 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:51:10.587 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:51:10.587 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:51:10.587 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:10.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac17fff640 1 -- 172.21.15.42:0/1520243905 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fac0802f070 con 0x7fac00007be0 2024-09-16T03:51:10.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac17fff640 1 --2- 172.21.15.42:0/1520243905 >> v2:172.21.15.154:6812/254845342 conn(0x7fac200a1c50 0x7fac00041dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:10.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac17fff640 1 -- 172.21.15.42:0/1520243905 <== mon.2 v2:172.21.15.42:3301/0 4 ==== osd_map(306..306 src has 1..306) ==== 6552+0+0 (secure 0 0 0) 0x7fac0804ba50 con 0x7fac00007be0 2024-09-16T03:51:10.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac17fff640 10 client.814112.objecter ms_dispatch 0x7fac20000ba0 osd_map(306..306 src has 1..306) 2024-09-16T03:51:10.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac17fff640 3 client.814112.objecter handle_osd_map got epochs [306,306] > 0 2024-09-16T03:51:10.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.553+0000 7fac17fff640 3 client.814112.objecter handle_osd_map decoding full epoch 306 2024-09-16T03:51:10.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.557+0000 7fac17fff640 20 client.814112.objecter dump_active .. 0 homeless 2024-09-16T03:51:10.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.557+0000 7fac17fff640 1 -- 172.21.15.42:0/1520243905 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fac180b9670 con 0x7fac2006dac0 2024-09-16T03:51:10.588 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.557+0000 7fac17fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:51:10.588 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.557+0000 7fac17fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:51:10.588 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.557+0000 7fac17fff640 1 -- 172.21.15.42:0/1520243905 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fac2006dac0 msgr2=0x7fac201352b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:10.588 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.557+0000 7fac17fff640 1 --2- 172.21.15.42:0/1520243905 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fac2006dac0 0x7fac201352b0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fac2009c800 tx=0x7fac1801ac00 comp rx=0 tx=0).stop 2024-09-16T03:51:10.588 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.557+0000 7fac17fff640 1 -- 172.21.15.42:0/1520243905 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fac2006dac0 msgr2=0x7fac201352b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:51:10.588 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.557+0000 7fac17fff640 1 --2- 172.21.15.42:0/1520243905 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fac2006dac0 0x7fac201352b0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:10.588 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.557+0000 7fac1ffff640 1 --2- 172.21.15.42:0/1520243905 >> v2:172.21.15.154:6812/254845342 conn(0x7fac200a1c50 0x7fac00041dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:10.588 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.557+0000 7fac1ffff640 10 monclient: get_auth_request con 0x7fac200a1c50 auth_method 0 2024-09-16T03:51:10.588 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.557+0000 7fac17fff640 10 client.814112.objecter ms_handle_connect 0x7fac200a1c50 2024-09-16T03:51:10.589 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.557+0000 7fac1ffff640 1 --2- 172.21.15.42:0/1520243905 >> v2:172.21.15.154:6812/254845342 conn(0x7fac200a1c50 0x7fac00041dc0 secure :-1 s=READY pgs=3552 cs=0 l=1 rev1=1 crypto rx=0x7fac10005890 tx=0x7fac10009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:51:10.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.561+0000 7fac26051640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:51:10.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.561+0000 7fac26051640 1 --2- 172.21.15.42:0/1520243905 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fac2013cf00 0x7fac2009d320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:10.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.561+0000 7fac26051640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:10.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.561+0000 7fac26051640 1 -- 172.21.15.42:0/1520243905 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fac201cf360 con 0x7fac2013cf00 2024-09-16T03:51:10.596 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.561+0000 7fac1f7fe640 1 --2- 172.21.15.42:0/1520243905 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fac2013cf00 0x7fac2009d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:10.596 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.561+0000 7fac1f7fe640 10 monclient: get_auth_request con 0x7fac2013cf00 auth_method 0 2024-09-16T03:51:10.596 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.561+0000 7fac1f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:10.596 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.561+0000 7fac1f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:10.596 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.561+0000 7fac1f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:51:10.596 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.561+0000 7fac1f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:51:10.596 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.561+0000 7fac1f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:51:10.596 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.561+0000 7fac1f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:51:10.596 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.561+0000 7fac1f7fe640 10 monclient(hunting): handle_auth_done global_id 814118 payload 824 2024-09-16T03:51:10.596 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.561+0000 7fac1f7fe640 1 -- 172.21.15.42:0/1520243905 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fac20069040 con 0x7fac2013cf00 2024-09-16T03:51:10.597 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.561+0000 7fac1f7fe640 1 --2- 172.21.15.42:0/1520243905 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fac2013cf00 0x7fac2009d320 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fac18019ff0 tx=0x7fac18019160 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:51:10.597 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.561+0000 7fac17fff640 10 client.814112.objecter ms_handle_connect 0x7fac2013cf00 2024-09-16T03:51:10.597 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.561+0000 7fac17fff640 10 client.814112.objecter resend_mon_ops 2024-09-16T03:51:10.597 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.561+0000 7fac17fff640 1 -- 172.21.15.42:0/1520243905 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac180b1040 con 0x7fac2013cf00 2024-09-16T03:51:10.597 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.561+0000 7fac17fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:51:10.597 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.561+0000 7fac17fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:51:10.597 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.561+0000 7fac17fff640 10 monclient: dump: 2024-09-16T03:51:10.597 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:51:10.597 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:10.597 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:10.597 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:10.597 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:51:10.597 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:10.598 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:51:10.598 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:51:10.598 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:51:10.598 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:10.598 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.561+0000 7fac17fff640 1 -- 172.21.15.42:0/1520243905 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4544 (secure 0 0 0) 0x7fac18003ab0 con 0x7fac2013cf00 2024-09-16T03:51:10.598 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.561+0000 7fac17fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:51:10.598 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.561+0000 7fac17fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:51:10.598 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.561+0000 7fac17fff640 1 -- 172.21.15.42:0/1520243905 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fac2013cf00 msgr2=0x7fac2009d320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:10.598 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.561+0000 7fac17fff640 1 --2- 172.21.15.42:0/1520243905 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fac2013cf00 0x7fac2009d320 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fac18019ff0 tx=0x7fac18019160 comp rx=0 tx=0).stop 2024-09-16T03:51:10.598 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.561+0000 7fac17fff640 1 -- 172.21.15.42:0/1520243905 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fac2013cf00 msgr2=0x7fac2009d320 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:51:10.598 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.561+0000 7fac17fff640 1 --2- 172.21.15.42:0/1520243905 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fac2013cf00 0x7fac2009d320 secure :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fac18019ff0 tx=0x7fac18019160 comp rx=0 tx=0).stop 2024-09-16T03:51:10.599 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:51:10.599 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:51:10.599 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:51:10.599 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T03:51:10.599 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 80, 2024-09-16T03:51:10.599 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 15635, 2024-09-16T03:51:10.599 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:51:10.599 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:51:10.599 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T03:51:10.599 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T03:51:10.599 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:51:10.599 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 0, 2024-09-16T03:51:10.599 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:51:10.600 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:51:10.600 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:51:10.600 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:51:10.600 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:51:10.600 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:51:10.600 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:51:10.600 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:51:10.600 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:51:10.600 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:51:10.600 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:51:10.600 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:51:10.600 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:51:10.600 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:51:10.601 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:51:10.601 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:51:10.601 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:51:10.601 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:51:10.601 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:51:10.601 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:51:10.601 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:51:10.601 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:51:10.601 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:51:10.601 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:51:10.601 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:51:10.601 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:51:10.601 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:51:10.602 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:51:10.602 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:51:10.602 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:10.602 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:51:10.602 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:51:10.602 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:51:10.602 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:51:10.602 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:51:10.602 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:51:10.602 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:51:10.602 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:51:10.602 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:51:10.602 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:51:10.603 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:51:10.603 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:51:10.603 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:51:10.603 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:51:10.603 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:51:10.603 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:51:10.603 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:51:10.603 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:51:10.603 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:51:10.603 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:51:10.603 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:51:10.603 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:51:10.603 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:51:10.603 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:51:10.604 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:51:10.604 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:51:10.604 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:51:10.604 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:51:10.604 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:51:10.604 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:51:10.604 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:10.604 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:51:10.604 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:10.604 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:51:10.604 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:51:10.604 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:51:10.605 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:51:10.605 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:10.605 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:51:10.605 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:51:10.605 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:51:10.605 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:10.605 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:10.605 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:51:10.605 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:51:10.605 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:51:10.605 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:51:10.605 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:51:10.605 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:10.605 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:51:10.606 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:51:10.606 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:51:10.606 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:51:10.606 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:51:10.606 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:10.606 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:10.606 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:51:10.606 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:51:10.606 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:51:10.606 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:51:10.606 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:10.606 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:51:10.606 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:51:10.607 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:51:10.607 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:10.607 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:10.607 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:51:10.607 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:51:10.607 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:51:10.607 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:51:10.607 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:51:10.607 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:10.607 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:51:10.607 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:51:10.607 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:51:10.607 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:51:10.607 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:51:10.608 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:10.608 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:10.608 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:51:10.608 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:51:10.608 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:51:10.608 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:51:10.608 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:10.608 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:51:10.608 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:51:10.608 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:51:10.608 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:10.608 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:10.608 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:51:10.609 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:51:10.609 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:51:10.609 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:51:10.609 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:51:10.609 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:10.609 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:51:10.609 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:51:10.609 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:51:10.609 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:51:10.609 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:51:10.609 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:51:10.609 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:51:10.609 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:10.610 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:51:10.610 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:51:10.610 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:10.610 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:51:10.610 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:51:10.610 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:51:10.610 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:51:10.610 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:51:10.610 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:51:10.610 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:10.610 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:51:10.610 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:51:10.610 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 2 2024-09-16T03:51:10.610 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:51:10.611 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:51:10.611 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:10.611 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:51:10.611 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:51:10.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.565+0000 7fac26051640 10 librados: watch_flush enter 2024-09-16T03:51:10.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.565+0000 7fac26051640 10 librados: watch_flush exit 2024-09-16T03:51:10.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.565+0000 7fac26051640 20 client.814112.objecter shutdown clearing up homeless session... 2024-09-16T03:51:10.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.565+0000 7fac26051640 10 client.814112.objecter successfully canceled tick 2024-09-16T03:51:10.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.565+0000 7fac26051640 1 -- 172.21.15.42:0/1520243905 >> v2:172.21.15.154:6812/254845342 conn(0x7fac200a1c50 msgr2=0x7fac00041dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:10.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.565+0000 7fac26051640 1 --2- 172.21.15.42:0/1520243905 >> v2:172.21.15.154:6812/254845342 conn(0x7fac200a1c50 0x7fac00041dc0 secure :-1 s=READY pgs=3552 cs=0 l=1 rev1=1 crypto rx=0x7fac10005890 tx=0x7fac10009040 comp rx=0 tx=0).stop 2024-09-16T03:51:10.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.565+0000 7fac26051640 10 monclient: shutdown 2024-09-16T03:51:10.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.565+0000 7fac26051640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:51:10.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.565+0000 7fac26051640 1 -- 172.21.15.42:0/1520243905 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fac00007be0 msgr2=0x7fac0000a070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:10.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.565+0000 7fac26051640 1 --2- 172.21.15.42:0/1520243905 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fac00007be0 0x7fac0000a070 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fac08014000 tx=0x7fac08001690 comp rx=0 tx=0).stop 2024-09-16T03:51:10.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.565+0000 7fac1ffff640 1 -- 172.21.15.42:0/1520243905 reap_dead start 2024-09-16T03:51:10.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.565+0000 7fac26051640 1 -- 172.21.15.42:0/1520243905 shutdown_connections 2024-09-16T03:51:10.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.565+0000 7fac26051640 1 --2- 172.21.15.42:0/1520243905 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fac00007be0 0x7fac0000a070 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:10.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.565+0000 7fac26051640 1 -- 172.21.15.42:0/1520243905 >> 172.21.15.42:0/1520243905 conn(0x7fac200a2420 msgr2=0x7fac200a4ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:51:10.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.565+0000 7fac26051640 1 -- 172.21.15.42:0/1520243905 shutdown_connections 2024-09-16T03:51:10.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.565+0000 7fac26051640 1 -- 172.21.15.42:0/1520243905 wait complete. 2024-09-16T03:51:10.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.565+0000 7fac26051640 1 librados: shutdown 2024-09-16T03:51:10.614 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:51:10.615 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T03:51:10.615 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T03:51:10.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.729+0000 7fbeed1f8640 10 monclient: get_monmap_and_config 2024-09-16T03:51:10.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.729+0000 7fbeed1f8640 10 monclient: build_initial_monmap 2024-09-16T03:51:10.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.729+0000 7fbeed1f8640 10 monclient: monmap: 2024-09-16T03:51:10.734 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:51:10.734 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:10.734 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:51:10.732482+0000 2024-09-16T03:51:10.734 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:51:10.732482+0000 2024-09-16T03:51:10.734 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:51:10.734 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:10.734 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:51:10.735 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:51:10.735 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:51:10.735 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:51:10.735 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:51:10.735 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:10.737 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.729+0000 7fbeed1f8640 1 Processor -- start 2024-09-16T03:51:10.737 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.729+0000 7fbeed1f8640 1 -- start start 2024-09-16T03:51:10.737 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.729+0000 7fbeed1f8640 10 monclient: init 2024-09-16T03:51:10.737 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.729+0000 7fbeed1f8640 10 monclient: _reopen_session rank -1 2024-09-16T03:51:10.737 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.729+0000 7fbeed1f8640 10 monclient: _add_conns ranks=[0,2,1,4,3] 2024-09-16T03:51:10.737 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.729+0000 7fbeed1f8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbee80a1330 addr v1:172.21.15.42:6789/0 2024-09-16T03:51:10.737 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.729+0000 7fbeed1f8640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbee806dac0 0x7fbee813a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:10.737 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.729+0000 7fbeed1f8640 10 monclient(hunting): picked mon.noname-b con 0x7fbee806dac0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:51:10.737 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.729+0000 7fbeed1f8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbee80a1e20 addr v1:172.21.15.154:6789/0 2024-09-16T03:51:10.738 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.729+0000 7fbeed1f8640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbee809c7d0 con 0x7fbee80a1330 2024-09-16T03:51:10.738 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.729+0000 7fbeed1f8640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbee809c090 con 0x7fbee80a1e20 2024-09-16T03:51:10.738 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.729+0000 7fbeed1f8640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:10.738 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.729+0000 7fbeed1f8640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fbee809f610 con 0x7fbee806dac0 2024-09-16T03:51:10.738 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.729+0000 7fbeed1f8640 10 monclient(hunting): _renew_subs 2024-09-16T03:51:10.738 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.729+0000 7fbeed1f8640 10 monclient(hunting): authenticate will time out at 1517.701958s 2024-09-16T03:51:10.738 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.729+0000 7fbee6d9d640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fbee80a1330 0x7fbee80a1710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:50506/0 (socket says 172.21.15.42:50506) 2024-09-16T03:51:10.738 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.729+0000 7fbee6d9d640 1 -- 172.21.15.42:0/2226511720 learned_addr learned my addr 172.21.15.42:0/2226511720 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:51:10.738 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.729+0000 7fbee659c640 1 --2- 172.21.15.42:0/2226511720 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbee806dac0 0x7fbee813a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:10.738 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.729+0000 7fbee659c640 10 monclient(hunting): get_auth_request con 0x7fbee806dac0 auth_method 0 2024-09-16T03:51:10.738 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.729+0000 7fbee659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:10.738 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.729+0000 7fbee659c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:10.738 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.729+0000 7fbee659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:51:10.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.729+0000 7fbee659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:51:10.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.729+0000 7fbee659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:51:10.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.729+0000 7fbee659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:51:10.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.729+0000 7fbee659c640 10 monclient(hunting): handle_auth_done global_id 814121 payload 274 2024-09-16T03:51:10.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.729+0000 7fbee5d9b640 1 -- 172.21.15.42:0/2226511720 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 958284824 0 0) 0x7fbee809c7d0 con 0x7fbee80a1330 2024-09-16T03:51:10.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.729+0000 7fbee659c640 1 -- 172.21.15.42:0/2226511720 >> v1:172.21.15.154:6789/0 conn(0x7fbee80a1e20 legacy=0x7fbee813c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:10.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.729+0000 7fbee659c640 1 -- 172.21.15.42:0/2226511720 >> v1:172.21.15.42:6789/0 conn(0x7fbee80a1330 legacy=0x7fbee80a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:10.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.729+0000 7fbee659c640 10 monclient: _finish_hunting 0 2024-09-16T03:51:10.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.729+0000 7fbee659c640 1 monclient: found mon.noname-b 2024-09-16T03:51:10.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.729+0000 7fbee659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:51:10.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.729+0000 7fbee659c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T03:51:10.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.729+0000 7fbee659c640 1 -- 172.21.15.42:0/2226511720 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbee809f8f0 con 0x7fbee806dac0 2024-09-16T03:51:10.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.729+0000 7fbee5d9b640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T03:51:10.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.733+0000 7fbee659c640 1 --2- 172.21.15.42:0/2226511720 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbee806dac0 0x7fbee813a340 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fbedc0097f0 tx=0x7fbedc01c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e38f69aeaa8c937f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:51:10.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.733+0000 7fbee5d9b640 1 -- 172.21.15.42:0/2226511720 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbedc0aa610 con 0x7fbee806dac0 2024-09-16T03:51:10.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.733+0000 7fbee5d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:51:10.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.733+0000 7fbee5d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:51:10.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.733+0000 7fbee5d9b640 10 monclient: dump: 2024-09-16T03:51:10.747 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:51:10.747 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:10.747 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:10.747 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:10.747 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:51:10.747 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:10.747 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:51:10.747 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:51:10.747 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:51:10.747 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:10.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.733+0000 7fbee5d9b640 10 monclient: _finish_auth 0 2024-09-16T03:51:10.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.733+0000 7fbee5d9b640 10 monclient: _check_auth_tickets 2024-09-16T03:51:10.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.733+0000 7fbee5d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:51:10.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.733+0000 7fbee5d9b640 1 -- 172.21.15.42:0/2226511720 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbedc0af070 con 0x7fbee806dac0 2024-09-16T03:51:10.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.733+0000 7fbee5d9b640 10 monclient: handle_config config(0 keys) 2024-09-16T03:51:10.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.733+0000 7fbee5d9b640 1 -- 172.21.15.42:0/2226511720 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbedc0aa9d0 con 0x7fbee806dac0 2024-09-16T03:51:10.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.733+0000 7fbee5d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:51:10.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.733+0000 7fbee5d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:51:10.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.733+0000 7fbee5d9b640 10 monclient: dump: 2024-09-16T03:51:10.748 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:51:10.748 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:10.748 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:10.748 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:10.748 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:51:10.748 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:10.748 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:51:10.748 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:51:10.749 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:51:10.749 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:10.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.733+0000 7fbeed1f8640 5 monclient: authenticate success, global_id 814121 2024-09-16T03:51:10.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.733+0000 7fbeed1f8640 10 monclient: get_monmap_and_config success 2024-09-16T03:51:10.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.733+0000 7fbeed1f8640 4 set_mon_vals no callback set 2024-09-16T03:51:10.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.733+0000 7fbeed1f8640 10 monclient: shutdown 2024-09-16T03:51:10.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.733+0000 7fbeed1f8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:51:10.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.733+0000 7fbeed1f8640 1 -- 172.21.15.42:0/2226511720 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbee806dac0 msgr2=0x7fbee813a340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:10.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.733+0000 7fbeed1f8640 1 --2- 172.21.15.42:0/2226511720 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbee806dac0 0x7fbee813a340 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fbedc0097f0 tx=0x7fbedc01c490 comp rx=0 tx=0).stop 2024-09-16T03:51:10.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.733+0000 7fbeed1f8640 1 -- 172.21.15.42:0/2226511720 shutdown_connections 2024-09-16T03:51:10.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.733+0000 7fbeed1f8640 1 --2- 172.21.15.42:0/2226511720 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbee806dac0 0x7fbee813a340 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:10.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.733+0000 7fbeed1f8640 1 -- 172.21.15.42:0/2226511720 >> 172.21.15.42:0/2226511720 conn(0x7fbee80a2420 msgr2=0x7fbee80a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:51:10.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.733+0000 7fbeed1f8640 1 -- 172.21.15.42:0/2226511720 shutdown_connections 2024-09-16T03:51:10.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.733+0000 7fbeed1f8640 1 -- 172.21.15.42:0/2226511720 wait complete. 2024-09-16T03:51:10.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.733+0000 7fbeed1f8640 10 monclient: build_initial_monmap 2024-09-16T03:51:10.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.733+0000 7fbeed1f8640 10 monclient: monmap: 2024-09-16T03:51:10.750 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:51:10.750 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:51:10.750 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:51:10.750 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:51:10.750 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:51:10.750 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:10.750 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:51:10.750 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:51:10.750 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:51:10.750 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:10.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.733+0000 7fbeed1f8640 1 librados: starting msgr at 2024-09-16T03:51:10.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.733+0000 7fbeed1f8640 1 librados: starting objecter 2024-09-16T03:51:10.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.733+0000 7fbeed1f8640 1 Processor -- start 2024-09-16T03:51:10.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.733+0000 7fbeed1f8640 1 -- start start 2024-09-16T03:51:10.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.733+0000 7fbeed1f8640 1 librados: setting wanted keys 2024-09-16T03:51:10.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.733+0000 7fbeed1f8640 1 librados: calling monclient init 2024-09-16T03:51:10.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.733+0000 7fbeed1f8640 10 monclient: init 2024-09-16T03:51:10.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.733+0000 7fbeed1f8640 10 monclient: _reopen_session rank -1 2024-09-16T03:51:10.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.733+0000 7fbeed1f8640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T03:51:10.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.733+0000 7fbeed1f8640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbee80a1330 0x7fbee81cedb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:10.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.733+0000 7fbeed1f8640 10 monclient(hunting): picked mon.noname-a con 0x7fbee80a1330 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:51:10.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.733+0000 7fbeed1f8640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbee80a1e20 0x7fbee813bf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:10.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.733+0000 7fbeed1f8640 10 monclient(hunting): picked mon.noname-b con 0x7fbee80a1e20 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:51:10.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.733+0000 7fbeed1f8640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbee8135320 0x7fbee81357b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:10.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.733+0000 7fbeed1f8640 10 monclient(hunting): picked mon.noname-c con 0x7fbee8135320 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:51:10.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.733+0000 7fbeed1f8640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:10.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.733+0000 7fbeed1f8640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fbee8140950 con 0x7fbee80a1330 2024-09-16T03:51:10.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.733+0000 7fbeed1f8640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:10.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.733+0000 7fbeed1f8640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fbee81407d0 con 0x7fbee80a1e20 2024-09-16T03:51:10.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.733+0000 7fbeed1f8640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:10.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.733+0000 7fbeed1f8640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fbee8140ad0 con 0x7fbee8135320 2024-09-16T03:51:10.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.733+0000 7fbeed1f8640 10 monclient(hunting): _renew_subs 2024-09-16T03:51:10.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.733+0000 7fbeed1f8640 10 monclient(hunting): authenticate will time out at 1517.707231s 2024-09-16T03:51:10.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.733+0000 7fbee6d9d640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbee80a1330 0x7fbee81cedb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:10.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.733+0000 7fbee759e640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbee8135320 0x7fbee81357b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:10.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.733+0000 7fbee6d9d640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbee80a1330 0x7fbee81cedb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:53050/0 (socket says 172.21.15.42:53050) 2024-09-16T03:51:10.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.733+0000 7fbee6d9d640 1 -- 172.21.15.42:0/536783003 learned_addr learned my addr 172.21.15.42:0/536783003 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:51:10.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.733+0000 7fbee6d9d640 10 monclient(hunting): get_auth_request con 0x7fbee80a1330 auth_method 0 2024-09-16T03:51:10.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.733+0000 7fbee6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:10.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.733+0000 7fbee6d9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:10.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.733+0000 7fbee6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:51:10.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.733+0000 7fbee759e640 10 monclient(hunting): get_auth_request con 0x7fbee8135320 auth_method 0 2024-09-16T03:51:10.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.733+0000 7fbee759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:10.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.733+0000 7fbee759e640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:10.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.733+0000 7fbee759e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:51:10.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.733+0000 7fbee6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:51:10.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.733+0000 7fbee6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:51:10.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.733+0000 7fbee6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:51:10.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.733+0000 7fbee759e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:51:10.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.733+0000 7fbee759e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:51:10.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.733+0000 7fbee759e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:51:10.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.733+0000 7fbee659c640 1 --2- 172.21.15.42:0/536783003 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbee80a1e20 0x7fbee813bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:10.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.733+0000 7fbee659c640 10 monclient(hunting): get_auth_request con 0x7fbee80a1e20 auth_method 0 2024-09-16T03:51:10.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.733+0000 7fbee659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:10.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.733+0000 7fbee659c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:10.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.733+0000 7fbee659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:51:10.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.733+0000 7fbee6d9d640 10 monclient(hunting): handle_auth_done global_id 764898 payload 824 2024-09-16T03:51:10.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.733+0000 7fbee6d9d640 1 -- 172.21.15.42:0/536783003 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbee8135320 msgr2=0x7fbee81357b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:10.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.733+0000 7fbee6d9d640 1 --2- 172.21.15.42:0/536783003 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbee8135320 0x7fbee81357b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:10.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.733+0000 7fbee6d9d640 1 -- 172.21.15.42:0/536783003 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbee80a1e20 msgr2=0x7fbee813bf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:10.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.733+0000 7fbee6d9d640 1 --2- 172.21.15.42:0/536783003 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbee80a1e20 0x7fbee813bf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:10.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.733+0000 7fbee6d9d640 10 monclient: _finish_hunting 0 2024-09-16T03:51:10.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.733+0000 7fbee6d9d640 1 monclient: found mon.noname-a 2024-09-16T03:51:10.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.733+0000 7fbee6d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:51:10.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.733+0000 7fbee6d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:51:10.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.733+0000 7fbee6d9d640 1 -- 172.21.15.42:0/536783003 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbee813d420 con 0x7fbee80a1330 2024-09-16T03:51:10.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.733+0000 7fbee6d9d640 1 --2- 172.21.15.42:0/536783003 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbee80a1330 0x7fbee81cedb0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fbed800c4a0 tx=0x7fbed800c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:51:10.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.733+0000 7fbecf7fe640 10 client.?.objecter ms_handle_connect 0x7fbee80a1330 2024-09-16T03:51:10.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.733+0000 7fbecf7fe640 10 client.?.objecter resend_mon_ops 2024-09-16T03:51:10.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.737+0000 7fbecf7fe640 1 -- 172.21.15.42:0/536783003 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbed8018070 con 0x7fbee80a1330 2024-09-16T03:51:10.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.737+0000 7fbecf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:51:10.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.737+0000 7fbecf7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:51:10.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.737+0000 7fbecf7fe640 10 monclient: dump: 2024-09-16T03:51:10.755 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:51:10.755 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:10.755 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:10.755 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:10.755 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:51:10.755 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:10.756 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:51:10.756 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:51:10.756 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:51:10.756 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:10.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.737+0000 7fbecf7fe640 10 monclient: _finish_auth 0 2024-09-16T03:51:10.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.737+0000 7fbecf7fe640 10 monclient: _check_auth_tickets 2024-09-16T03:51:10.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.737+0000 7fbecf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:51:10.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.737+0000 7fbecf7fe640 1 -- 172.21.15.42:0/536783003 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbed80074e0 con 0x7fbee80a1330 2024-09-16T03:51:10.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.737+0000 7fbeed1f8640 5 monclient: authenticate success, global_id 764898 2024-09-16T03:51:10.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.737+0000 7fbeed1f8640 10 monclient: _renew_subs 2024-09-16T03:51:10.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.737+0000 7fbeed1f8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:51:10.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.737+0000 7fbeed1f8640 1 -- 172.21.15.42:0/536783003 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbee806e340 con 0x7fbee80a1330 2024-09-16T03:51:10.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.737+0000 7fbecf7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:51:10.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.737+0000 7fbecf7fe640 1 -- 172.21.15.42:0/536783003 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbed8004a30 con 0x7fbee80a1330 2024-09-16T03:51:10.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.737+0000 7fbecf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:51:10.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.737+0000 7fbecf7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:51:10.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.737+0000 7fbecf7fe640 10 monclient: dump: 2024-09-16T03:51:10.757 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:51:10.757 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:10.757 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:10.757 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:10.757 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:51:10.757 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:10.758 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:51:10.758 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:51:10.758 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:51:10.758 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:10.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.737+0000 7fbeed1f8640 10 client.764898.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:51:10.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.737+0000 7fbeed1f8640 10 monclient: _renew_subs 2024-09-16T03:51:10.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.737+0000 7fbeed1f8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:51:10.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.737+0000 7fbeed1f8640 1 -- 172.21.15.42:0/536783003 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbee81209e0 con 0x7fbee80a1330 2024-09-16T03:51:10.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.737+0000 7fbeed1f8640 1 librados: init done 2024-09-16T03:51:10.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.737+0000 7fbee4d99640 4 set_mon_vals no callback set 2024-09-16T03:51:10.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.737+0000 7fbeed1f8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:51:10.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.737+0000 7fbeed1f8640 1 --2- 172.21.15.42:0/536783003 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbee806dac0 0x7fbee809cb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:10.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.737+0000 7fbeed1f8640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:10.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.737+0000 7fbeed1f8640 1 -- 172.21.15.42:0/536783003 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fbee8074230 con 0x7fbee806dac0 2024-09-16T03:51:10.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.737+0000 7fbecf7fe640 1 -- 172.21.15.42:0/536783003 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fbed801b070 con 0x7fbee80a1330 2024-09-16T03:51:10.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.737+0000 7fbee659c640 1 --2- 172.21.15.42:0/536783003 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbee806dac0 0x7fbee809cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:10.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.737+0000 7fbee659c640 10 monclient: get_auth_request con 0x7fbee806dac0 auth_method 0 2024-09-16T03:51:10.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.737+0000 7fbee659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:10.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.737+0000 7fbee659c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:10.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.737+0000 7fbee659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:51:10.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.737+0000 7fbecf7fe640 1 --2- 172.21.15.42:0/536783003 >> v2:172.21.15.154:6812/254845342 conn(0x7fbed4037810 0x7fbed4039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:10.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.737+0000 7fbecf7fe640 1 -- 172.21.15.42:0/536783003 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(306..306 src has 1..306) ==== 6552+0+0 (secure 0 0 0) 0x7fbed8048590 con 0x7fbee80a1330 2024-09-16T03:51:10.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.737+0000 7fbecf7fe640 10 client.764898.objecter ms_dispatch 0x7fbee8000ba0 osd_map(306..306 src has 1..306) 2024-09-16T03:51:10.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.737+0000 7fbecf7fe640 3 client.764898.objecter handle_osd_map got epochs [306,306] > 0 2024-09-16T03:51:10.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.737+0000 7fbecf7fe640 3 client.764898.objecter handle_osd_map decoding full epoch 306 2024-09-16T03:51:10.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.737+0000 7fbee659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:51:10.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.737+0000 7fbee659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:51:10.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.737+0000 7fbee659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:51:10.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.737+0000 7fbecf7fe640 20 client.764898.objecter dump_active .. 0 homeless 2024-09-16T03:51:10.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.737+0000 7fbee759e640 1 --2- 172.21.15.42:0/536783003 >> v2:172.21.15.154:6812/254845342 conn(0x7fbed4037810 0x7fbed4039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:10.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.737+0000 7fbee759e640 10 monclient: get_auth_request con 0x7fbed4037810 auth_method 0 2024-09-16T03:51:10.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.737+0000 7fbee659c640 10 monclient(hunting): handle_auth_done global_id 794317 payload 824 2024-09-16T03:51:10.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.737+0000 7fbee659c640 1 -- 172.21.15.42:0/536783003 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbee80a1710 con 0x7fbee806dac0 2024-09-16T03:51:10.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.737+0000 7fbecf7fe640 10 client.764898.objecter ms_handle_connect 0x7fbee806dac0 2024-09-16T03:51:10.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.737+0000 7fbecf7fe640 10 client.764898.objecter resend_mon_ops 2024-09-16T03:51:10.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.737+0000 7fbee659c640 1 --2- 172.21.15.42:0/536783003 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbee806dac0 0x7fbee809cb60 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fbee806add0 tx=0x7fbedc019230 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:51:10.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.737+0000 7fbecf7fe640 1 -- 172.21.15.42:0/536783003 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbedc003820 con 0x7fbee806dac0 2024-09-16T03:51:10.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.737+0000 7fbecf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:51:10.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.737+0000 7fbecf7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:51:10.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.737+0000 7fbecf7fe640 10 monclient: dump: 2024-09-16T03:51:10.761 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:51:10.761 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:10.761 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:10.761 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:10.761 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:51:10.761 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:10.761 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:51:10.761 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:51:10.761 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:51:10.761 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:10.761 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.737+0000 7fbecf7fe640 10 client.764898.objecter ms_handle_connect 0x7fbed4037810 2024-09-16T03:51:10.761 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.737+0000 7fbee759e640 1 --2- 172.21.15.42:0/536783003 >> v2:172.21.15.154:6812/254845342 conn(0x7fbed4037810 0x7fbed4039cd0 secure :-1 s=READY pgs=3553 cs=0 l=1 rev1=1 crypto rx=0x7fbee00012f0 tx=0x7fbee0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:51:10.761 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.737+0000 7fbecf7fe640 1 -- 172.21.15.42:0/536783003 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fbedc0af050 con 0x7fbee806dac0 2024-09-16T03:51:10.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.737+0000 7fbecf7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:51:10.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.737+0000 7fbecf7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:51:10.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.737+0000 7fbecf7fe640 1 -- 172.21.15.42:0/536783003 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbee806dac0 msgr2=0x7fbee809cb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:10.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.737+0000 7fbecf7fe640 1 --2- 172.21.15.42:0/536783003 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbee806dac0 0x7fbee809cb60 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fbee806add0 tx=0x7fbedc019230 comp rx=0 tx=0).stop 2024-09-16T03:51:10.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.737+0000 7fbecf7fe640 1 -- 172.21.15.42:0/536783003 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbee806dac0 msgr2=0x7fbee809cb60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:51:10.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.737+0000 7fbecf7fe640 1 --2- 172.21.15.42:0/536783003 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbee806dac0 0x7fbee809cb60 secure :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fbee806add0 tx=0x7fbedc019230 comp rx=0 tx=0).stop 2024-09-16T03:51:10.763 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.745+0000 7fbeed1f8640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:51:10.763 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.745+0000 7fbeed1f8640 1 --2- 172.21.15.42:0/536783003 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbee809d0a0 0x7fbee809d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:10.763 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.745+0000 7fbeed1f8640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:10.764 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.745+0000 7fbeed1f8640 1 -- 172.21.15.42:0/536783003 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fbee809d9f0 con 0x7fbee809d0a0 2024-09-16T03:51:10.764 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.745+0000 7fbee659c640 1 --2- 172.21.15.42:0/536783003 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbee809d0a0 0x7fbee809d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:10.764 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.745+0000 7fbee659c640 10 monclient: get_auth_request con 0x7fbee809d0a0 auth_method 0 2024-09-16T03:51:10.764 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.745+0000 7fbee659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:10.764 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.745+0000 7fbee659c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:10.764 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.745+0000 7fbee659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:51:10.764 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.745+0000 7fbee659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:51:10.764 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.745+0000 7fbee659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:51:10.764 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.745+0000 7fbee659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:51:10.764 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.745+0000 7fbee659c640 10 monclient(hunting): handle_auth_done global_id 794320 payload 824 2024-09-16T03:51:10.764 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.745+0000 7fbee659c640 1 -- 172.21.15.42:0/536783003 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fbee81205b0 con 0x7fbee809d0a0 2024-09-16T03:51:10.764 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.745+0000 7fbee659c640 1 --2- 172.21.15.42:0/536783003 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbee809d0a0 0x7fbee809d4b0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fbedc01a0c0 tx=0x7fbedc009190 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:51:10.765 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.745+0000 7fbecf7fe640 10 client.764898.objecter ms_handle_connect 0x7fbee809d0a0 2024-09-16T03:51:10.765 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.745+0000 7fbecf7fe640 10 client.764898.objecter resend_mon_ops 2024-09-16T03:51:10.765 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.745+0000 7fbecf7fe640 1 -- 172.21.15.42:0/536783003 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbedc0bdbf0 con 0x7fbee809d0a0 2024-09-16T03:51:10.765 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.745+0000 7fbecf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:51:10.765 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.745+0000 7fbecf7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:51:10.765 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.745+0000 7fbecf7fe640 10 monclient: dump: 2024-09-16T03:51:10.765 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:51:10.765 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:10.765 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:10.765 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:10.765 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:51:10.765 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:10.765 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:51:10.765 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:51:10.766 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:51:10.766 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:10.766 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.745+0000 7fbecf7fe640 1 -- 172.21.15.42:0/536783003 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4545 (secure 0 0 0) 0x7fbedc0c1570 con 0x7fbee809d0a0 2024-09-16T03:51:10.766 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.745+0000 7fbecf7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:51:10.766 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.745+0000 7fbecf7fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:51:10.766 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.745+0000 7fbecf7fe640 1 -- 172.21.15.42:0/536783003 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbee809d0a0 msgr2=0x7fbee809d4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:10.766 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.745+0000 7fbecf7fe640 1 --2- 172.21.15.42:0/536783003 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbee809d0a0 0x7fbee809d4b0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fbedc01a0c0 tx=0x7fbedc009190 comp rx=0 tx=0).stop 2024-09-16T03:51:10.766 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.745+0000 7fbecf7fe640 1 -- 172.21.15.42:0/536783003 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbee809d0a0 msgr2=0x7fbee809d4b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:51:10.766 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.745+0000 7fbecf7fe640 1 --2- 172.21.15.42:0/536783003 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbee809d0a0 0x7fbee809d4b0 secure :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fbedc01a0c0 tx=0x7fbedc009190 comp rx=0 tx=0).stop 2024-09-16T03:51:10.766 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:51:10.767 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:51:10.767 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:51:10.767 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T03:51:10.767 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 80, 2024-09-16T03:51:10.767 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 112586, 2024-09-16T03:51:10.767 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:51:10.767 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:51:10.767 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T03:51:10.767 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T03:51:10.767 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:51:10.767 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 0, 2024-09-16T03:51:10.767 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:51:10.767 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:51:10.767 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:51:10.768 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:51:10.768 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:51:10.768 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:51:10.768 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:51:10.768 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:51:10.768 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:51:10.768 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:51:10.768 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:51:10.768 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:51:10.768 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:51:10.768 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:51:10.768 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:51:10.768 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:51:10.769 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:51:10.769 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:51:10.769 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:51:10.769 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:51:10.769 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:51:10.769 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:51:10.769 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:51:10.769 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:51:10.769 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:51:10.769 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:51:10.769 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:51:10.769 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:51:10.769 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:51:10.769 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:10.770 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:51:10.770 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:51:10.770 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:51:10.770 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:51:10.770 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:51:10.770 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:51:10.770 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:51:10.770 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:51:10.770 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:51:10.770 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:51:10.770 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:51:10.770 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:51:10.770 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:51:10.771 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:51:10.771 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:51:10.771 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:51:10.771 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:51:10.771 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:51:10.771 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:51:10.771 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:51:10.771 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:51:10.771 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:51:10.771 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:51:10.771 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:51:10.771 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:51:10.771 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:51:10.771 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:51:10.772 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:51:10.772 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:51:10.772 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:51:10.772 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:10.772 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:51:10.772 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:10.772 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:51:10.772 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:51:10.772 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:51:10.772 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:51:10.772 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:10.772 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:51:10.772 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:51:10.773 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:51:10.773 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:10.773 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:10.773 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:51:10.773 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:51:10.773 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:51:10.773 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:51:10.773 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:51:10.773 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:10.773 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:51:10.773 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:51:10.773 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:51:10.773 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:51:10.773 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:51:10.774 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:10.774 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:10.774 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:51:10.774 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:51:10.774 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:51:10.774 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:51:10.774 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:10.774 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:51:10.774 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:51:10.774 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:51:10.774 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:10.774 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:10.774 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:51:10.774 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:51:10.775 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:51:10.775 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:51:10.775 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:51:10.775 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:10.775 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:51:10.775 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:51:10.775 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:51:10.775 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:51:10.775 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:51:10.775 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:10.776 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:10.776 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:51:10.776 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:51:10.776 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:51:10.776 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:51:10.776 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:10.776 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:51:10.776 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:51:10.776 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:51:10.776 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:10.776 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:10.776 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:51:10.776 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:51:10.777 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:51:10.777 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:51:10.777 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:51:10.777 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:10.777 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:51:10.777 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:51:10.777 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:51:10.777 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:51:10.777 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:51:10.777 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:51:10.777 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:51:10.777 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:10.777 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:51:10.777 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:51:10.778 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:10.778 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:51:10.778 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:51:10.778 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:51:10.778 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:51:10.778 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:51:10.778 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:51:10.778 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:10.778 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:51:10.778 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:51:10.778 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 3 2024-09-16T03:51:10.778 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:51:10.778 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:51:10.778 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:10.778 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:51:10.779 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:51:10.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.749+0000 7fbeed1f8640 10 librados: watch_flush enter 2024-09-16T03:51:10.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.749+0000 7fbeed1f8640 10 librados: watch_flush exit 2024-09-16T03:51:10.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.749+0000 7fbeed1f8640 20 client.764898.objecter shutdown clearing up homeless session... 2024-09-16T03:51:10.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.749+0000 7fbeed1f8640 10 client.764898.objecter successfully canceled tick 2024-09-16T03:51:10.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.749+0000 7fbeed1f8640 1 -- 172.21.15.42:0/536783003 >> v2:172.21.15.154:6812/254845342 conn(0x7fbed4037810 msgr2=0x7fbed4039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:10.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.749+0000 7fbeed1f8640 1 --2- 172.21.15.42:0/536783003 >> v2:172.21.15.154:6812/254845342 conn(0x7fbed4037810 0x7fbed4039cd0 secure :-1 s=READY pgs=3553 cs=0 l=1 rev1=1 crypto rx=0x7fbee00012f0 tx=0x7fbee0009040 comp rx=0 tx=0).stop 2024-09-16T03:51:10.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.749+0000 7fbeed1f8640 10 monclient: shutdown 2024-09-16T03:51:10.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.749+0000 7fbeed1f8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:51:10.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.749+0000 7fbeed1f8640 1 -- 172.21.15.42:0/536783003 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbee80a1330 msgr2=0x7fbee81cedb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:10.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.749+0000 7fbeed1f8640 1 --2- 172.21.15.42:0/536783003 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbee80a1330 0x7fbee81cedb0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fbed800c4a0 tx=0x7fbed800c960 comp rx=0 tx=0).stop 2024-09-16T03:51:10.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.749+0000 7fbee759e640 1 -- 172.21.15.42:0/536783003 reap_dead start 2024-09-16T03:51:10.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.749+0000 7fbee759e640 1 -- 172.21.15.42:0/536783003 reap_dead start 2024-09-16T03:51:10.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.749+0000 7fbeed1f8640 1 -- 172.21.15.42:0/536783003 shutdown_connections 2024-09-16T03:51:10.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.749+0000 7fbeed1f8640 1 -- 172.21.15.42:0/536783003 >> 172.21.15.42:0/536783003 conn(0x7fbee80a2420 msgr2=0x7fbee80a45b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:51:10.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.749+0000 7fbeed1f8640 1 -- 172.21.15.42:0/536783003 shutdown_connections 2024-09-16T03:51:10.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.749+0000 7fbeed1f8640 1 -- 172.21.15.42:0/536783003 wait complete. 2024-09-16T03:51:10.782 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.749+0000 7fbeed1f8640 1 librados: shutdown 2024-09-16T03:51:10.782 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:51:10.782 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T03:51:10.782 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T03:51:10.895 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.889+0000 7f28f472e640 10 monclient: get_monmap_and_config 2024-09-16T03:51:10.895 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.889+0000 7f28f472e640 10 monclient: build_initial_monmap 2024-09-16T03:51:10.895 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.889+0000 7f28f472e640 10 monclient: monmap: 2024-09-16T03:51:10.895 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:51:10.896 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:10.896 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:51:10.894334+0000 2024-09-16T03:51:10.896 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:51:10.894334+0000 2024-09-16T03:51:10.896 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:51:10.896 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:10.896 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:51:10.896 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:51:10.896 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:51:10.896 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:51:10.896 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:51:10.896 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:10.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.889+0000 7f28f472e640 1 Processor -- start 2024-09-16T03:51:10.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.889+0000 7f28f472e640 1 -- start start 2024-09-16T03:51:10.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.889+0000 7f28f472e640 10 monclient: init 2024-09-16T03:51:10.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.889+0000 7f28f472e640 10 monclient: _reopen_session rank -1 2024-09-16T03:51:10.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.889+0000 7f28f472e640 10 monclient: _add_conns ranks=[3,2,1,0,4] 2024-09-16T03:51:10.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.889+0000 7f28f472e640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f28ec0a39b0 0x7f28ec133b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:10.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.889+0000 7f28f472e640 10 monclient(hunting): picked mon.noname-a con 0x7f28ec0a39b0 addr v2:172.21.15.42:3300/0 2024-09-16T03:51:10.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.889+0000 7f28f472e640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f28ec0a3d90 0x7f28ec137000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:10.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.889+0000 7f28f472e640 10 monclient(hunting): picked mon.noname-b con 0x7f28ec0a3d90 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:51:10.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.889+0000 7f28f472e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f28ec137540 addr v1:172.21.15.154:6789/0 2024-09-16T03:51:10.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.889+0000 7f28f472e640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f28ec09c140 con 0x7f28ec137540 2024-09-16T03:51:10.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.889+0000 7f28f472e640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:10.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.889+0000 7f28f472e640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f28ec09c420 con 0x7f28ec0a39b0 2024-09-16T03:51:10.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.889+0000 7f28f472e640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:10.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.889+0000 7f28f472e640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f28ec06dd60 con 0x7f28ec0a3d90 2024-09-16T03:51:10.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.889+0000 7f28f472e640 10 monclient(hunting): _renew_subs 2024-09-16T03:51:10.900 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.889+0000 7f28f472e640 10 monclient(hunting): authenticate will time out at 1517.863709s 2024-09-16T03:51:10.900 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.889+0000 7f28f24ca640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f28ec0a39b0 0x7f28ec133b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:10.900 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.889+0000 7f28f1cc9640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f28ec0a3d90 0x7f28ec137000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:10.900 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.889+0000 7f28f24ca640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f28ec0a39b0 0x7f28ec133b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:53066/0 (socket says 172.21.15.42:53066) 2024-09-16T03:51:10.900 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.889+0000 7f28f24ca640 1 -- 172.21.15.42:0/2360570411 learned_addr learned my addr 172.21.15.42:0/2360570411 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:51:10.900 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.889+0000 7f28f24ca640 10 monclient(hunting): get_auth_request con 0x7f28ec0a39b0 auth_method 0 2024-09-16T03:51:10.900 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.889+0000 7f28f24ca640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:10.900 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.889+0000 7f28f24ca640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:10.900 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.889+0000 7f28f24ca640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:51:10.900 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.889+0000 7f28f1cc9640 10 monclient(hunting): get_auth_request con 0x7f28ec0a3d90 auth_method 0 2024-09-16T03:51:10.900 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.889+0000 7f28f1cc9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:10.900 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.889+0000 7f28f1cc9640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:10.900 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.889+0000 7f28f1cc9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:51:10.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.893+0000 7f28f24ca640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:51:10.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.893+0000 7f28f24ca640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:51:10.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.893+0000 7f28f24ca640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:51:10.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.893+0000 7f28f1cc9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:51:10.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.893+0000 7f28f1cc9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:51:10.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.893+0000 7f28f1cc9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:51:10.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.893+0000 7f28f1cc9640 10 monclient(hunting): handle_auth_done global_id 814127 payload 274 2024-09-16T03:51:10.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.893+0000 7f28f1cc9640 1 -- 172.21.15.42:0/2360570411 >> v2:172.21.15.42:3300/0 conn(0x7f28ec0a39b0 msgr2=0x7f28ec133b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:51:10.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.893+0000 7f28f1cc9640 1 --2- 172.21.15.42:0/2360570411 >> v2:172.21.15.42:3300/0 conn(0x7f28ec0a39b0 0x7f28ec133b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:10.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.893+0000 7f28f1cc9640 1 -- 172.21.15.42:0/2360570411 >> v1:172.21.15.154:6789/0 conn(0x7f28ec137540 legacy=0x7f28ec13e390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:51:10.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.893+0000 7f28f1cc9640 10 monclient: _finish_hunting 0 2024-09-16T03:51:10.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.893+0000 7f28f1cc9640 1 monclient: found mon.noname-b 2024-09-16T03:51:10.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.893+0000 7f28f1cc9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:51:10.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.893+0000 7f28f1cc9640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T03:51:10.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.893+0000 7f28f1cc9640 1 -- 172.21.15.42:0/2360570411 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28ec0a5e20 con 0x7f28ec0a3d90 2024-09-16T03:51:10.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.893+0000 7f28f24ca640 1 --2- 172.21.15.42:0/2360570411 >> v2:172.21.15.42:3300/0 conn(0x7f28ec0a39b0 0x7f28ec133b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:51:10.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.893+0000 7f28f1cc9640 1 --2- 172.21.15.42:0/2360570411 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f28ec0a3d90 0x7f28ec137000 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f28dc0097c0 tx=0x7f28dc01c320 comp rx=0 tx=0).ready entity=mon.2 client_cookie=15d0d53e2b7cf799 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:51:10.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.893+0000 7f28f14c8640 1 -- 172.21.15.42:0/2360570411 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28dc0b0070 con 0x7f28ec0a3d90 2024-09-16T03:51:10.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.893+0000 7f28f14c8640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:51:10.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.893+0000 7f28f14c8640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:51:10.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.893+0000 7f28f14c8640 10 monclient: dump: 2024-09-16T03:51:10.908 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:51:10.908 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:10.908 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:10.908 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:10.908 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:51:10.908 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:10.908 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:51:10.908 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:51:10.908 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:51:10.908 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:10.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.893+0000 7f28f14c8640 10 monclient: _finish_auth 0 2024-09-16T03:51:10.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.893+0000 7f28f14c8640 10 monclient: _check_auth_tickets 2024-09-16T03:51:10.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.893+0000 7f28f14c8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:51:10.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.893+0000 7f28f472e640 5 monclient: authenticate success, global_id 814127 2024-09-16T03:51:10.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.893+0000 7f28f472e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:51:10.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.893+0000 7f28f14c8640 1 -- 172.21.15.42:0/2360570411 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f28dc0ab600 con 0x7f28ec0a3d90 2024-09-16T03:51:10.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.893+0000 7f28f14c8640 10 monclient: handle_config config(0 keys) 2024-09-16T03:51:10.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.893+0000 7f28f472e640 10 monclient: get_monmap_and_config success 2024-09-16T03:51:10.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.893+0000 7f28f472e640 4 set_mon_vals no callback set 2024-09-16T03:51:10.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.893+0000 7f28f14c8640 1 -- 172.21.15.42:0/2360570411 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28dc0ab9d0 con 0x7f28ec0a3d90 2024-09-16T03:51:10.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.893+0000 7f28f14c8640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:51:10.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.893+0000 7f28f14c8640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:51:10.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.893+0000 7f28f14c8640 10 monclient: dump: 2024-09-16T03:51:10.910 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:51:10.910 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:10.910 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:10.910 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:10.910 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:51:10.910 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:10.910 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:51:10.910 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:51:10.910 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:51:10.910 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:10.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.893+0000 7f28f472e640 10 monclient: shutdown 2024-09-16T03:51:10.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.893+0000 7f28f472e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:51:10.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.893+0000 7f28f472e640 1 -- 172.21.15.42:0/2360570411 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f28ec0a3d90 msgr2=0x7f28ec137000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:10.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.893+0000 7f28f472e640 1 --2- 172.21.15.42:0/2360570411 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f28ec0a3d90 0x7f28ec137000 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f28dc0097c0 tx=0x7f28dc01c320 comp rx=0 tx=0).stop 2024-09-16T03:51:10.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.893+0000 7f28f472e640 1 -- 172.21.15.42:0/2360570411 shutdown_connections 2024-09-16T03:51:10.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.893+0000 7f28f472e640 1 --2- 172.21.15.42:0/2360570411 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f28ec0a3d90 0x7f28ec137000 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:10.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.893+0000 7f28f472e640 1 --2- 172.21.15.42:0/2360570411 >> v2:172.21.15.42:3300/0 conn(0x7f28ec0a39b0 0x7f28ec133b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:10.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.893+0000 7f28f472e640 1 -- 172.21.15.42:0/2360570411 >> 172.21.15.42:0/2360570411 conn(0x7f28ec0a9fb0 msgr2=0x7f28ec0aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:51:10.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.893+0000 7f28f472e640 1 -- 172.21.15.42:0/2360570411 shutdown_connections 2024-09-16T03:51:10.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.893+0000 7f28f472e640 1 -- 172.21.15.42:0/2360570411 wait complete. 2024-09-16T03:51:10.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.893+0000 7f28f472e640 10 monclient: build_initial_monmap 2024-09-16T03:51:10.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.893+0000 7f28f472e640 10 monclient: monmap: 2024-09-16T03:51:10.911 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:51:10.911 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:51:10.911 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:51:10.911 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:51:10.911 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:51:10.912 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:10.912 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:51:10.912 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:51:10.912 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:51:10.912 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:10.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.897+0000 7f28f472e640 1 librados: starting msgr at 2024-09-16T03:51:10.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.897+0000 7f28f472e640 1 librados: starting objecter 2024-09-16T03:51:10.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.897+0000 7f28f472e640 1 Processor -- start 2024-09-16T03:51:10.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.897+0000 7f28f472e640 1 -- start start 2024-09-16T03:51:10.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.897+0000 7f28f472e640 1 librados: setting wanted keys 2024-09-16T03:51:10.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.897+0000 7f28f472e640 1 librados: calling monclient init 2024-09-16T03:51:10.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.897+0000 7f28f472e640 10 monclient: init 2024-09-16T03:51:10.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.897+0000 7f28f472e640 10 monclient: _reopen_session rank -1 2024-09-16T03:51:10.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.897+0000 7f28f472e640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T03:51:10.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.897+0000 7f28f472e640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f28ec0a39b0 0x7f28ec13d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:10.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.897+0000 7f28f472e640 10 monclient(hunting): picked mon.noname-a con 0x7f28ec0a39b0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:51:10.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.897+0000 7f28f472e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f28ec0a3d90 0x7f28ec13d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:10.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.897+0000 7f28f472e640 10 monclient(hunting): picked mon.noname-b con 0x7f28ec0a3d90 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:51:10.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.897+0000 7f28f472e640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f28ec137540 0x7f28ec13dde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:10.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.897+0000 7f28f472e640 10 monclient(hunting): picked mon.noname-c con 0x7f28ec137540 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:51:10.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.897+0000 7f28f472e640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:10.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.897+0000 7f28f472e640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f28ec1424e0 con 0x7f28ec0a39b0 2024-09-16T03:51:10.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.897+0000 7f28f472e640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:10.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.897+0000 7f28f472e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f28ec142660 con 0x7f28ec0a3d90 2024-09-16T03:51:10.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.897+0000 7f28f472e640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:10.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.897+0000 7f28f472e640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f28ec1427e0 con 0x7f28ec137540 2024-09-16T03:51:10.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.897+0000 7f28f472e640 10 monclient(hunting): _renew_subs 2024-09-16T03:51:10.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.897+0000 7f28f472e640 10 monclient(hunting): authenticate will time out at 1517.868898s 2024-09-16T03:51:10.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.897+0000 7f28f24ca640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f28ec0a39b0 0x7f28ec13d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:10.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.897+0000 7f28f2ccb640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f28ec137540 0x7f28ec13dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:10.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.897+0000 7f28f24ca640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f28ec0a39b0 0x7f28ec13d360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:53082/0 (socket says 172.21.15.42:53082) 2024-09-16T03:51:10.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.897+0000 7f28f24ca640 1 -- 172.21.15.42:0/3123106790 learned_addr learned my addr 172.21.15.42:0/3123106790 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:51:10.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.897+0000 7f28f2ccb640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f28ec137540 0x7f28ec13dde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:43494/0 (socket says 172.21.15.42:43494) 2024-09-16T03:51:10.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.897+0000 7f28f24ca640 10 monclient(hunting): get_auth_request con 0x7f28ec0a39b0 auth_method 0 2024-09-16T03:51:10.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.897+0000 7f28f24ca640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:10.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.897+0000 7f28f24ca640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:10.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.897+0000 7f28f24ca640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:51:10.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.897+0000 7f28f2ccb640 10 monclient(hunting): get_auth_request con 0x7f28ec137540 auth_method 0 2024-09-16T03:51:10.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.897+0000 7f28f2ccb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:10.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.897+0000 7f28f2ccb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:10.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.897+0000 7f28f2ccb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:51:10.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.897+0000 7f28f2ccb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:51:10.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.897+0000 7f28f2ccb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:51:10.916 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.897+0000 7f28f2ccb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:51:10.916 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.897+0000 7f28f24ca640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:51:10.916 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.897+0000 7f28f24ca640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:51:10.916 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.897+0000 7f28f24ca640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:51:10.916 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.897+0000 7f28f1cc9640 1 --2- 172.21.15.42:0/3123106790 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f28ec0a3d90 0x7f28ec13d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:10.916 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.897+0000 7f28f2ccb640 10 monclient(hunting): handle_auth_done global_id 814130 payload 824 2024-09-16T03:51:10.916 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.897+0000 7f28f2ccb640 1 -- 172.21.15.42:0/3123106790 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f28ec0a3d90 msgr2=0x7f28ec13d8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:10.916 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.897+0000 7f28f2ccb640 1 --2- 172.21.15.42:0/3123106790 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f28ec0a3d90 0x7f28ec13d8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:10.916 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.897+0000 7f28f2ccb640 1 -- 172.21.15.42:0/3123106790 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f28ec0a39b0 msgr2=0x7f28ec13d360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:10.916 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.897+0000 7f28f2ccb640 1 --2- 172.21.15.42:0/3123106790 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f28ec0a39b0 0x7f28ec13d360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:10.916 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.897+0000 7f28f2ccb640 10 monclient: _finish_hunting 0 2024-09-16T03:51:10.916 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.897+0000 7f28f2ccb640 1 monclient: found mon.noname-c 2024-09-16T03:51:10.916 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.897+0000 7f28f2ccb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:51:10.917 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.897+0000 7f28f2ccb640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:51:10.917 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.897+0000 7f28f2ccb640 1 -- 172.21.15.42:0/3123106790 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28ec09e4d0 con 0x7f28ec137540 2024-09-16T03:51:10.917 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.897+0000 7f28f1cc9640 10 monclient: get_auth_request con 0x7f28ec0a3d90 auth_method 0 2024-09-16T03:51:10.917 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.897+0000 7f28f24ca640 1 --2- 172.21.15.42:0/3123106790 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f28ec0a39b0 0x7f28ec13d360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:51:10.917 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.897+0000 7f28f1cc9640 1 --2- 172.21.15.42:0/3123106790 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f28ec0a3d90 0x7f28ec13d8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T03:51:10.917 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.897+0000 7f28f2ccb640 1 --2- 172.21.15.42:0/3123106790 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f28ec137540 0x7f28ec13dde0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f28e800c410 tx=0x7f28e800c8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:51:10.917 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.897+0000 7f28e2ffd640 10 client.?.objecter ms_handle_connect 0x7f28ec137540 2024-09-16T03:51:10.917 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.897+0000 7f28e2ffd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:51:10.917 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.897+0000 7f28e2ffd640 1 -- 172.21.15.42:0/3123106790 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28e8017070 con 0x7f28ec137540 2024-09-16T03:51:10.917 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.897+0000 7f28e2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:51:10.917 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.897+0000 7f28e2ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:51:10.917 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.897+0000 7f28e2ffd640 10 monclient: dump: 2024-09-16T03:51:10.918 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:51:10.918 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:10.918 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:10.918 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:10.918 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:51:10.918 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:10.918 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:51:10.918 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:51:10.918 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:51:10.918 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:10.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.897+0000 7f28e2ffd640 10 monclient: _finish_auth 0 2024-09-16T03:51:10.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.897+0000 7f28e2ffd640 10 monclient: _check_auth_tickets 2024-09-16T03:51:10.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.897+0000 7f28e2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:51:10.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.897+0000 7f28e2ffd640 1 -- 172.21.15.42:0/3123106790 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f28e8005230 con 0x7f28ec137540 2024-09-16T03:51:10.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.897+0000 7f28e2ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:51:10.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.897+0000 7f28f472e640 5 monclient: authenticate success, global_id 814130 2024-09-16T03:51:10.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.897+0000 7f28e2ffd640 1 -- 172.21.15.42:0/3123106790 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28e8004590 con 0x7f28ec137540 2024-09-16T03:51:10.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.897+0000 7f28f472e640 10 monclient: _renew_subs 2024-09-16T03:51:10.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.897+0000 7f28f472e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:51:10.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.897+0000 7f28f472e640 1 -- 172.21.15.42:0/3123106790 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28ec136e00 con 0x7f28ec137540 2024-09-16T03:51:10.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.897+0000 7f28e2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:51:10.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.897+0000 7f28f0cc7640 4 set_mon_vals no callback set 2024-09-16T03:51:10.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.897+0000 7f28e2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:51:10.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.897+0000 7f28e2ffd640 10 monclient: dump: 2024-09-16T03:51:10.919 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:51:10.920 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:10.920 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:10.920 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:10.920 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:51:10.920 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:10.920 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:51:10.920 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:51:10.920 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:51:10.920 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:10.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.897+0000 7f28f472e640 10 client.814130.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:51:10.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.897+0000 7f28f472e640 10 monclient: _renew_subs 2024-09-16T03:51:10.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.897+0000 7f28f472e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:51:10.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.897+0000 7f28f472e640 1 -- 172.21.15.42:0/3123106790 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f28ec06f680 con 0x7f28ec137540 2024-09-16T03:51:10.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.897+0000 7f28f472e640 1 librados: init done 2024-09-16T03:51:10.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.897+0000 7f28e2ffd640 1 -- 172.21.15.42:0/3123106790 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f28e801a070 con 0x7f28ec137540 2024-09-16T03:51:10.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.897+0000 7f28f472e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:51:10.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.897+0000 7f28f472e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:51:10.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.897+0000 7f28f472e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:51:10.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.897+0000 7f28f472e640 1 -- 172.21.15.42:0/3123106790 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f28ec133b80 con 0x7f28ec137540 2024-09-16T03:51:10.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.897+0000 7f28e2ffd640 1 --2- 172.21.15.42:0/3123106790 >> v2:172.21.15.154:6812/254845342 conn(0x7f28d0037810 0x7f28d0039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:10.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.897+0000 7f28e2ffd640 1 -- 172.21.15.42:0/3123106790 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(306..306 src has 1..306) ==== 6552+0+0 (secure 0 0 0) 0x7f28e80483e0 con 0x7f28ec137540 2024-09-16T03:51:10.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.897+0000 7f28e2ffd640 10 client.814130.objecter ms_dispatch 0x7f28ec000ba0 osd_map(306..306 src has 1..306) 2024-09-16T03:51:10.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.897+0000 7f28e2ffd640 3 client.814130.objecter handle_osd_map got epochs [306,306] > 0 2024-09-16T03:51:10.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.897+0000 7f28e2ffd640 3 client.814130.objecter handle_osd_map decoding full epoch 306 2024-09-16T03:51:10.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.897+0000 7f28e2ffd640 20 client.814130.objecter dump_active .. 0 homeless 2024-09-16T03:51:10.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.897+0000 7f28f24ca640 1 --2- 172.21.15.42:0/3123106790 >> v2:172.21.15.154:6812/254845342 conn(0x7f28d0037810 0x7f28d0039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:10.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.897+0000 7f28f24ca640 10 monclient: get_auth_request con 0x7f28d0037810 auth_method 0 2024-09-16T03:51:10.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.901+0000 7f28e2ffd640 10 client.814130.objecter ms_handle_connect 0x7f28d0037810 2024-09-16T03:51:10.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.901+0000 7f28f24ca640 1 --2- 172.21.15.42:0/3123106790 >> v2:172.21.15.154:6812/254845342 conn(0x7f28d0037810 0x7f28d0039cd0 secure :-1 s=READY pgs=3554 cs=0 l=1 rev1=1 crypto rx=0x7f28e4009c90 tx=0x7f28e400b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:51:10.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.901+0000 7f28e2ffd640 1 -- 172.21.15.42:0/3123106790 <== mon.2 v2:172.21.15.42:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f28e8019e10 con 0x7f28ec137540 2024-09-16T03:51:10.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.905+0000 7f28e2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:51:10.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:10.905+0000 7f28e2ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:51:11.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:11.129+0000 7f28f472e640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T03:51:11.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:11.129+0000 7f28f472e640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T03:51:11.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:11.129+0000 7f28f472e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:51:11.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:11.129+0000 7f28f472e640 1 -- 172.21.15.42:0/3123106790 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f28ec0a5750 con 0x7f28ec137540 2024-09-16T03:51:11.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:11.133+0000 7f28e2ffd640 1 -- 172.21.15.42:0/3123106790 <== mon.2 v2:172.21.15.42:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f28e801a760 con 0x7f28ec137540 2024-09-16T03:51:11.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:11.133+0000 7f28e2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T03:51:11.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:11.133+0000 7f28e2ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:51:11.139 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.133+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:11.139 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.133+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=30,config=2,health=12,kv=8,logm=48} crc {auth=1618090503,config=692619927,health=3101215555,kv=2024801497,logm=197409876}) 2024-09-16T03:51:11.139 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.133+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=29,config=2,health=12,kv=10,logm=47} crc {auth=1096877274,config=692619927,health=3101215555,kv=3446188498,logm=205651503}) 2024-09-16T03:51:11.139 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.133+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:11.139 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.133+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=30,config=2,health=12,kv=8,logm=48} crc {auth=1618090503,config=692619927,health=3101215555,kv=2024801497,logm=197409876}) 2024-09-16T03:51:11.139 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.133+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=30,config=2,health=11,kv=10,logm=47} crc {auth=3046209846,config=692619927,health=169489802,kv=3446188498,logm=817271117}) 2024-09-16T03:51:11.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:11.133+0000 7f28f472e640 10 librados: watch_flush enter 2024-09-16T03:51:11.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:11.133+0000 7f28f472e640 10 librados: watch_flush exit 2024-09-16T03:51:11.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:11.133+0000 7f28f472e640 20 client.814130.objecter shutdown clearing up homeless session... 2024-09-16T03:51:11.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:11.133+0000 7f28f472e640 10 client.814130.objecter successfully canceled tick 2024-09-16T03:51:11.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:11.133+0000 7f28f472e640 1 -- 172.21.15.42:0/3123106790 >> v2:172.21.15.154:6812/254845342 conn(0x7f28d0037810 msgr2=0x7f28d0039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:11.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:11.133+0000 7f28f472e640 1 --2- 172.21.15.42:0/3123106790 >> v2:172.21.15.154:6812/254845342 conn(0x7f28d0037810 0x7f28d0039cd0 secure :-1 s=READY pgs=3554 cs=0 l=1 rev1=1 crypto rx=0x7f28e4009c90 tx=0x7f28e400b040 comp rx=0 tx=0).stop 2024-09-16T03:51:11.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:11.133+0000 7f28f472e640 10 monclient: shutdown 2024-09-16T03:51:11.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:11.133+0000 7f28f472e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:51:11.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:11.133+0000 7f28f472e640 1 -- 172.21.15.42:0/3123106790 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f28ec137540 msgr2=0x7f28ec13dde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:11.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:11.133+0000 7f28f472e640 1 --2- 172.21.15.42:0/3123106790 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f28ec137540 0x7f28ec13dde0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f28e800c410 tx=0x7f28e800c8d0 comp rx=0 tx=0).stop 2024-09-16T03:51:11.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:11.133+0000 7f28f472e640 1 -- 172.21.15.42:0/3123106790 shutdown_connections 2024-09-16T03:51:11.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:11.133+0000 7f28f472e640 1 --2- 172.21.15.42:0/3123106790 >> v2:172.21.15.154:6812/254845342 conn(0x7f28d0037810 0x7f28d0039cd0 unknown :-1 s=CLOSED pgs=3554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:11.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:11.133+0000 7f28f472e640 1 --2- 172.21.15.42:0/3123106790 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f28ec137540 0x7f28ec13dde0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:11.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:11.133+0000 7f28f472e640 1 --2- 172.21.15.42:0/3123106790 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f28ec0a3d90 0x7f28ec13d8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:11.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:11.133+0000 7f28f472e640 1 --2- 172.21.15.42:0/3123106790 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f28ec0a39b0 0x7f28ec13d360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:11.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:11.133+0000 7f28f472e640 1 -- 172.21.15.42:0/3123106790 >> 172.21.15.42:0/3123106790 conn(0x7f28ec0a9fb0 msgr2=0x7f28ec131e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:51:11.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:11.133+0000 7f28f472e640 1 -- 172.21.15.42:0/3123106790 shutdown_connections 2024-09-16T03:51:11.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:11.133+0000 7f28f472e640 1 -- 172.21.15.42:0/3123106790 wait complete. 2024-09-16T03:51:11.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:11.133+0000 7f28f472e640 1 librados: shutdown 2024-09-16T03:51:11.143 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.137+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:11.143 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.137+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=77578162}) 2024-09-16T03:51:11.143 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.137+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4042208063}) 2024-09-16T03:51:11.143 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.137+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:11.143 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.137+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=77578162}) 2024-09-16T03:51:11.143 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.137+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3631493141}) 2024-09-16T03:51:11.145 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.137+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:11.145 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.137+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=920140417}) 2024-09-16T03:51:11.145 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.137+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2426744997}) 2024-09-16T03:51:11.145 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.137+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:11.145 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.137+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=920140417}) 2024-09-16T03:51:11.145 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.137+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1359661909}) 2024-09-16T03:51:11.146 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.141+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:11.147 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.141+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=441725972}) 2024-09-16T03:51:11.147 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.141+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1472386164}) 2024-09-16T03:51:11.147 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.141+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:11.147 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.141+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=441725972}) 2024-09-16T03:51:11.147 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.141+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2867274912}) 2024-09-16T03:51:11.148 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.141+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:11.148 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.141+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1854228680}) 2024-09-16T03:51:11.148 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.141+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1316756945}) 2024-09-16T03:51:11.148 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.141+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:11.148 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.141+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1854228680}) 2024-09-16T03:51:11.149 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.141+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3158446817}) 2024-09-16T03:51:11.150 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.145+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:11.150 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.145+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4111076031}) 2024-09-16T03:51:11.150 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.145+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3312338054}) 2024-09-16T03:51:11.150 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.145+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:11.150 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.145+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4111076031}) 2024-09-16T03:51:11.151 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.145+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=786237369}) 2024-09-16T03:51:11.152 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.145+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:11.152 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.145+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1659585199}) 2024-09-16T03:51:11.152 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.145+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=976862660}) 2024-09-16T03:51:11.152 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.145+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:11.153 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.145+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1659585199}) 2024-09-16T03:51:11.153 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.145+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2534382353}) 2024-09-16T03:51:11.154 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.149+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:11.154 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.149+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2737994318}) 2024-09-16T03:51:11.154 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.149+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1888485494}) 2024-09-16T03:51:11.154 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.149+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:11.154 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.149+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2737994318}) 2024-09-16T03:51:11.154 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.149+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1769057541}) 2024-09-16T03:51:11.156 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.149+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:11.156 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.149+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=200678221}) 2024-09-16T03:51:11.156 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.149+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1418944540}) 2024-09-16T03:51:11.156 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.149+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:11.156 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.149+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=200678221}) 2024-09-16T03:51:11.156 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.149+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2419161557}) 2024-09-16T03:51:11.157 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T03:51:11.158 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.153+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:11.158 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.153+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1626781266}) 2024-09-16T03:51:11.158 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.153+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2503664551}) 2024-09-16T03:51:11.158 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.153+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:11.158 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.153+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1626781266}) 2024-09-16T03:51:11.158 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.153+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3679559575}) 2024-09-16T03:51:11.159 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.153+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:11.159 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.153+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2675185268}) 2024-09-16T03:51:11.159 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.153+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=640213994}) 2024-09-16T03:51:11.159 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.153+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:11.159 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.153+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2675185268}) 2024-09-16T03:51:11.159 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.153+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2151710923}) 2024-09-16T03:51:11.161 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.153+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:11.161 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.153+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2827973622}) 2024-09-16T03:51:11.161 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.153+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4024751050}) 2024-09-16T03:51:11.161 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.157+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:11.161 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.157+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2827973622}) 2024-09-16T03:51:11.161 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.157+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=806017570}) 2024-09-16T03:51:11.163 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.157+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:11.163 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.157+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1879750073}) 2024-09-16T03:51:11.163 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.157+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3356289624}) 2024-09-16T03:51:11.163 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.157+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:11.163 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.157+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1879750073}) 2024-09-16T03:51:11.163 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.157+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1810008363}) 2024-09-16T03:51:11.165 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.157+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:11.165 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.157+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3560204003}) 2024-09-16T03:51:11.165 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.157+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1768746859}) 2024-09-16T03:51:11.165 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.157+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:11.165 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.157+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3560204003}) 2024-09-16T03:51:11.165 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.157+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1362029656}) 2024-09-16T03:51:11.166 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.161+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:11.166 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.161+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3462724840}) 2024-09-16T03:51:11.166 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.161+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=551831095}) 2024-09-16T03:51:11.166 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.161+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:11.166 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.161+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3462724840}) 2024-09-16T03:51:11.166 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.161+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=658937601}) 2024-09-16T03:51:11.167 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.161+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:11.168 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.161+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1109271993}) 2024-09-16T03:51:11.168 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.161+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=738046417}) 2024-09-16T03:51:11.168 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.161+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:11.168 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.161+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1109271993}) 2024-09-16T03:51:11.168 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.161+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1470667123}) 2024-09-16T03:51:11.169 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.165+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:11.169 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.165+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3536905304}) 2024-09-16T03:51:11.169 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.165+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1130024762}) 2024-09-16T03:51:11.169 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.165+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:11.169 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.165+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3536905304}) 2024-09-16T03:51:11.170 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.165+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2216962265}) 2024-09-16T03:51:11.171 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.165+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:11.171 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.165+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=408938878}) 2024-09-16T03:51:11.171 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.165+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=707709406}) 2024-09-16T03:51:11.171 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.165+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:11.171 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.165+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=408938878}) 2024-09-16T03:51:11.171 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.165+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=712912988}) 2024-09-16T03:51:11.173 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.165+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:11.173 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.165+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2656146675}) 2024-09-16T03:51:11.173 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.169+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2355307299}) 2024-09-16T03:51:11.173 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.169+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:11.173 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.169+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2656146675}) 2024-09-16T03:51:11.173 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.169+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3722269163}) 2024-09-16T03:51:11.175 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.169+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:11.175 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.169+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2183772499}) 2024-09-16T03:51:11.175 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.169+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2566725298}) 2024-09-16T03:51:11.175 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.169+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:11.175 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.169+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2183772499}) 2024-09-16T03:51:11.175 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.169+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1420803852}) 2024-09-16T03:51:11.176 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.169+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:11.176 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.169+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2745058508}) 2024-09-16T03:51:11.177 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.169+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=993697954}) 2024-09-16T03:51:11.177 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.169+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:11.177 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.169+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2745058508}) 2024-09-16T03:51:11.177 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.169+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3276443888}) 2024-09-16T03:51:11.178 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.173+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:11.178 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.173+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2799785350}) 2024-09-16T03:51:11.178 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.173+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=823009006}) 2024-09-16T03:51:11.178 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.173+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:11.178 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.173+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2799785350}) 2024-09-16T03:51:11.178 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.173+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=558605815}) 2024-09-16T03:51:11.180 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.173+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:11.180 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.173+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=307515427}) 2024-09-16T03:51:11.180 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.173+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1051842758}) 2024-09-16T03:51:11.180 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.173+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:11.180 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.173+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=307515427}) 2024-09-16T03:51:11.180 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.173+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3293609241}) 2024-09-16T03:51:11.182 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.177+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:11.182 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.177+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3624148866}) 2024-09-16T03:51:11.182 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.177+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3362048651}) 2024-09-16T03:51:11.182 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.177+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:11.182 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.177+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3624148866}) 2024-09-16T03:51:11.182 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.177+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2418546787}) 2024-09-16T03:51:11.183 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.177+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:11.183 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.177+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1670361800}) 2024-09-16T03:51:11.183 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.177+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2680056741}) 2024-09-16T03:51:11.183 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.177+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:11.183 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.177+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1670361800}) 2024-09-16T03:51:11.183 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.177+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1256350125}) 2024-09-16T03:51:11.185 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.177+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:11.185 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.177+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3411149604}) 2024-09-16T03:51:11.185 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.177+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1046009363}) 2024-09-16T03:51:11.185 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.181+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:11.185 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.181+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3411149604}) 2024-09-16T03:51:11.185 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.181+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2427569391}) 2024-09-16T03:51:11.187 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.181+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:11.187 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.181+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1884840169}) 2024-09-16T03:51:11.187 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.181+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4253375812}) 2024-09-16T03:51:11.187 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.181+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:11.187 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.181+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1884840169}) 2024-09-16T03:51:11.187 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.181+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2774017163}) 2024-09-16T03:51:11.188 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.181+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:11.188 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.181+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3546637614}) 2024-09-16T03:51:11.189 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.181+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2566766187}) 2024-09-16T03:51:11.189 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.181+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:11.189 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.181+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3546637614}) 2024-09-16T03:51:11.189 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.181+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=945868019}) 2024-09-16T03:51:11.190 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.185+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:11.190 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.185+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1154444353}) 2024-09-16T03:51:11.190 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.185+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3628063930}) 2024-09-16T03:51:11.190 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.185+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:11.190 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.185+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1154444353}) 2024-09-16T03:51:11.191 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.185+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2884681259}) 2024-09-16T03:51:11.192 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.185+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:11.192 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.185+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1032521079}) 2024-09-16T03:51:11.192 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.185+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3894775980}) 2024-09-16T03:51:11.192 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.185+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:11.192 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.185+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1032521079}) 2024-09-16T03:51:11.192 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.185+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1065151284}) 2024-09-16T03:51:11.194 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.189+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:11.194 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.189+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=568608919}) 2024-09-16T03:51:11.194 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.189+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1201182180}) 2024-09-16T03:51:11.194 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.189+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:11.194 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.189+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=568608919}) 2024-09-16T03:51:11.194 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.189+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2442876295}) 2024-09-16T03:51:11.196 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.189+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:11.196 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.189+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=15564633}) 2024-09-16T03:51:11.196 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.189+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1398623537}) 2024-09-16T03:51:11.196 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.189+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:11.196 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.189+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=15564633}) 2024-09-16T03:51:11.196 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.189+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2227568828}) 2024-09-16T03:51:11.197 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.193+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:11.198 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.193+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4125488205}) 2024-09-16T03:51:11.198 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.193+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4126094040}) 2024-09-16T03:51:11.198 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.193+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:11.198 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.193+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4125488205}) 2024-09-16T03:51:11.198 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.193+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2800456501}) 2024-09-16T03:51:11.199 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.193+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:11.199 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.193+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1788750037}) 2024-09-16T03:51:11.199 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.193+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1335059789}) 2024-09-16T03:51:11.199 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.193+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:11.199 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.193+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1788750037}) 2024-09-16T03:51:11.200 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.193+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3316030770}) 2024-09-16T03:51:11.201 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.193+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:11.201 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.193+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3815663074}) 2024-09-16T03:51:11.201 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.193+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3345597743}) 2024-09-16T03:51:11.201 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.193+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:11.201 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.193+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3815663074}) 2024-09-16T03:51:11.201 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.193+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=727249244}) 2024-09-16T03:51:11.202 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.197+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:11.203 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.197+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1273807472}) 2024-09-16T03:51:11.203 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.197+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3860033384}) 2024-09-16T03:51:11.203 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.197+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:11.203 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.197+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1273807472}) 2024-09-16T03:51:11.203 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.197+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3801887649}) 2024-09-16T03:51:11.204 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.197+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:11.204 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.197+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3056423294}) 2024-09-16T03:51:11.204 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.197+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4038914991}) 2024-09-16T03:51:11.204 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.197+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:11.204 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.197+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3056423294}) 2024-09-16T03:51:11.204 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.197+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4123273740}) 2024-09-16T03:51:11.206 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.201+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:11.206 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.201+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2997751854}) 2024-09-16T03:51:11.206 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.201+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3543582361}) 2024-09-16T03:51:11.206 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.201+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:11.206 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.201+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2997751854}) 2024-09-16T03:51:11.206 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.201+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1212319608}) 2024-09-16T03:51:11.208 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.201+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:11.208 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.201+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=801903234}) 2024-09-16T03:51:11.208 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.201+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2351598086}) 2024-09-16T03:51:11.208 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.201+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:11.208 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.201+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=801903234}) 2024-09-16T03:51:11.208 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.201+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1998251092}) 2024-09-16T03:51:11.210 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.205+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:11.210 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.205+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=594923274}) 2024-09-16T03:51:11.210 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.205+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1058338066}) 2024-09-16T03:51:11.210 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.205+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:11.210 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.205+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=594923274}) 2024-09-16T03:51:11.210 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.205+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2179806970}) 2024-09-16T03:51:11.212 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.205+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:11.212 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.205+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1914888563}) 2024-09-16T03:51:11.212 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.205+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4025560291}) 2024-09-16T03:51:11.212 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.205+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:11.212 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.205+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1914888563}) 2024-09-16T03:51:11.212 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.205+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1055195233}) 2024-09-16T03:51:11.214 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.209+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:11.214 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.209+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=340572314}) 2024-09-16T03:51:11.214 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.209+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3551566142}) 2024-09-16T03:51:11.214 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.209+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:11.214 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.209+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=340572314}) 2024-09-16T03:51:11.214 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.209+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=33892470}) 2024-09-16T03:51:11.215 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.209+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:11.216 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.209+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=557978853}) 2024-09-16T03:51:11.216 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.209+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2056540245}) 2024-09-16T03:51:11.216 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.209+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:11.216 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.209+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=557978853}) 2024-09-16T03:51:11.216 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.209+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3389379757}) 2024-09-16T03:51:11.218 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.213+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:11.218 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.213+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1468185945}) 2024-09-16T03:51:11.218 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.213+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4041648471}) 2024-09-16T03:51:11.218 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.213+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:11.218 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.213+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1468185945}) 2024-09-16T03:51:11.218 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.213+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=705741710}) 2024-09-16T03:51:11.219 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.213+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:11.220 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.213+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4087711097}) 2024-09-16T03:51:11.220 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.213+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=908055050}) 2024-09-16T03:51:11.220 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.213+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:11.220 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.213+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4087711097}) 2024-09-16T03:51:11.220 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.213+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=54703912}) 2024-09-16T03:51:11.221 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.217+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:11.221 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.217+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3472225093}) 2024-09-16T03:51:11.222 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.217+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1547861249}) 2024-09-16T03:51:11.222 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.217+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:11.222 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.217+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3472225093}) 2024-09-16T03:51:11.222 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.217+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1921792682}) 2024-09-16T03:51:11.223 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.217+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:11.224 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.217+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1271730081}) 2024-09-16T03:51:11.224 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.217+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3428716514}) 2024-09-16T03:51:11.224 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.217+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:11.224 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.217+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1271730081}) 2024-09-16T03:51:11.224 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.217+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4106402961}) 2024-09-16T03:51:11.225 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.221+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:11.225 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.221+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2563745955}) 2024-09-16T03:51:11.225 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.221+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1569192843}) 2024-09-16T03:51:11.225 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.221+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:11.226 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.221+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2563745955}) 2024-09-16T03:51:11.226 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.221+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1399914633}) 2024-09-16T03:51:11.227 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.221+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:11.227 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.221+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2311990706}) 2024-09-16T03:51:11.227 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.221+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=98526814}) 2024-09-16T03:51:11.228 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.221+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:11.228 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.221+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2311990706}) 2024-09-16T03:51:11.228 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.221+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3069375237}) 2024-09-16T03:51:11.229 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.225+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:11.229 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.225+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2582634285}) 2024-09-16T03:51:11.229 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.225+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3804512143}) 2024-09-16T03:51:11.229 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.225+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:11.229 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.225+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2582634285}) 2024-09-16T03:51:11.230 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.225+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1742881542}) 2024-09-16T03:51:11.232 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.225+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:11.232 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.225+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=40,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=14} crc {logm=2728718739,mdsmap=4088524775,mgr=460109776,mgr_command_descs=2107673600,mgr_metadata=1367272202,mgrstat=297158471,mon_config_key=85057112,monmap=3937513981,osd_metadata=4182959540,osd_pg_creating=3299912523,osd_snap=3869766749}) 2024-09-16T03:51:11.232 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.225+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=39,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=9,monmap=2,osd_metadata=6,osd_pg_creating=1,osd_snap=19} crc {logm=3665378539,mdsmap=4088524775,mgr=460109776,mgr_command_descs=2107673600,mgr_metadata=1367272202,mgrstat=520346077,mon_config_key=3882128926,monmap=3140514009,osd_metadata=4182959540,osd_pg_creating=3299912523,osd_snap=667664684}) 2024-09-16T03:51:11.232 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.225+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:11.232 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.225+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=40,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=14} crc {logm=2728718739,mdsmap=4088524775,mgr=460109776,mgr_command_descs=2107673600,mgr_metadata=1367272202,mgrstat=297158471,mon_config_key=85057112,monmap=3937513981,osd_metadata=4182959540,osd_pg_creating=3299912523,osd_snap=3869766749}) 2024-09-16T03:51:11.232 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.225+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=38,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=11,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=18} crc {logm=3146656557,mdsmap=4088524775,mgr=390803354,mgr_command_descs=2107673600,mgr_metadata=1367272202,mgrstat=297158471,mon_config_key=85057112,monmap=3937513981,osd_metadata=191738038,osd_pg_creating=3299912523,osd_snap=1122454157}) 2024-09-16T03:51:11.234 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.229+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:11.234 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.229+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=14,osdmap=86} crc {osd_snap=3333857336,osdmap=3909436547}) 2024-09-16T03:51:11.234 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.229+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=15,osdmap=85} crc {osd_snap=402161950,osdmap=925804272}) 2024-09-16T03:51:11.235 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.229+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:11.235 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.229+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=14,osdmap=86} crc {osd_snap=3333857336,osdmap=3909436547}) 2024-09-16T03:51:11.235 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.229+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=15,osdmap=85} crc {osd_snap=402161950,osdmap=1694866683}) 2024-09-16T03:51:11.237 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.233+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:11.237 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.233+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3527971570}) 2024-09-16T03:51:11.237 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.233+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3252895317}) 2024-09-16T03:51:11.237 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.233+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:11.237 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.233+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3527971570}) 2024-09-16T03:51:11.238 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.233+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=182066911}) 2024-09-16T03:51:11.240 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.233+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:11.240 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.233+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=4249730228}) 2024-09-16T03:51:11.240 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.233+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=984275679}) 2024-09-16T03:51:11.240 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.233+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:11.241 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.233+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=4249730228}) 2024-09-16T03:51:11.241 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.233+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=843792054}) 2024-09-16T03:51:11.244 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.237+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:11.244 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.237+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=823252043}) 2024-09-16T03:51:11.245 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.237+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1645557596}) 2024-09-16T03:51:11.245 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.237+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:11.245 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.237+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=823252043}) 2024-09-16T03:51:11.245 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.237+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=910255205}) 2024-09-16T03:51:11.261 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.257+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:11.262 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.257+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=70} crc {osdmap=245930909}) 2024-09-16T03:51:11.262 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.257+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=68} crc {osdmap=3490461186}) 2024-09-16T03:51:11.262 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.257+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:11.262 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.257+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=70} crc {osdmap=245930909}) 2024-09-16T03:51:11.262 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:11.257+0000 7f5d61d1c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=68} crc {osdmap=949180764}) 2024-09-16T03:51:12.158 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T03:51:12.159 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:51:12.291 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.285+0000 7fd79119c640 10 monclient: get_monmap_and_config 2024-09-16T03:51:12.291 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.285+0000 7fd79119c640 10 monclient: build_initial_monmap 2024-09-16T03:51:12.291 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.285+0000 7fd79119c640 10 monclient: monmap: 2024-09-16T03:51:12.291 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:51:12.291 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:12.291 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:51:12.290214+0000 2024-09-16T03:51:12.292 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:51:12.290214+0000 2024-09-16T03:51:12.292 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:51:12.292 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:12.292 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:51:12.292 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:51:12.292 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:51:12.292 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:51:12.292 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:51:12.292 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:12.295 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.285+0000 7fd79119c640 1 Processor -- start 2024-09-16T03:51:12.295 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.285+0000 7fd79119c640 1 -- start start 2024-09-16T03:51:12.295 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.285+0000 7fd79119c640 10 monclient: init 2024-09-16T03:51:12.295 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.289+0000 7fd79119c640 10 monclient: _reopen_session rank -1 2024-09-16T03:51:12.295 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.289+0000 7fd79119c640 10 monclient: _add_conns ranks=[4,0,3,1,2] 2024-09-16T03:51:12.295 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.289+0000 7fd79119c640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fd78c09fb00 0x7fd78c09ba50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:12.295 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.289+0000 7fd79119c640 10 monclient(hunting): picked mon.noname-c con 0x7fd78c09fb00 addr v2:172.21.15.154:3300/0 2024-09-16T03:51:12.295 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.289+0000 7fd79119c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd78c09bf90 addr v1:172.21.15.42:6789/0 2024-09-16T03:51:12.295 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.289+0000 7fd79119c640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd78c09c400 0x7fd78c13b840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:12.295 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.289+0000 7fd79119c640 10 monclient(hunting): picked mon.noname-a con 0x7fd78c09c400 addr v2:172.21.15.42:3300/0 2024-09-16T03:51:12.295 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.289+0000 7fd79119c640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd78c0a1e20 con 0x7fd78c09bf90 2024-09-16T03:51:12.296 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.289+0000 7fd79119c640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:12.296 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.289+0000 7fd79119c640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7fd78c09e250 con 0x7fd78c09c400 2024-09-16T03:51:12.296 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.289+0000 7fd79119c640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:12.296 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.289+0000 7fd79119c640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fd78c09c850 con 0x7fd78c09fb00 2024-09-16T03:51:12.296 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.289+0000 7fd79119c640 10 monclient(hunting): _renew_subs 2024-09-16T03:51:12.296 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.289+0000 7fd79119c640 10 monclient(hunting): authenticate will time out at 1519.260418s 2024-09-16T03:51:12.296 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.289+0000 7fd78a59c640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fd78c09bf90 0x7fd78c138110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:50522/0 (socket says 172.21.15.42:50522) 2024-09-16T03:51:12.296 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.289+0000 7fd78a59c640 1 -- 172.21.15.42:0/3909396382 learned_addr learned my addr 172.21.15.42:0/3909396382 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:51:12.296 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.289+0000 7fd78b59e640 1 --2- 172.21.15.42:0/3909396382 >> v2:172.21.15.42:3300/0 conn(0x7fd78c09c400 0x7fd78c13b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:12.296 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.289+0000 7fd78b59e640 10 monclient(hunting): get_auth_request con 0x7fd78c09c400 auth_method 0 2024-09-16T03:51:12.296 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.289+0000 7fd78b59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:12.296 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.289+0000 7fd78b59e640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:12.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.289+0000 7fd78b59e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:51:12.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.289+0000 7fd78ad9d640 1 --2- 172.21.15.42:0/3909396382 >> v2:172.21.15.154:3300/0 conn(0x7fd78c09fb00 0x7fd78c09ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:12.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.289+0000 7fd78b59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:51:12.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.289+0000 7fd78b59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:51:12.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.289+0000 7fd78b59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:51:12.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.289+0000 7fd78ad9d640 10 monclient(hunting): get_auth_request con 0x7fd78c09fb00 auth_method 0 2024-09-16T03:51:12.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.289+0000 7fd78ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:12.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.289+0000 7fd78ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:12.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.289+0000 7fd78ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:51:12.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.289+0000 7fd789d9b640 1 -- 172.21.15.42:0/3909396382 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2841662235 0 0) 0x7fd78c0a1e20 con 0x7fd78c09bf90 2024-09-16T03:51:12.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.289+0000 7fd789d9b640 10 monclient(hunting): _negotiate 2024-09-16T03:51:12.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.289+0000 7fd789d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:12.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.289+0000 7fd789d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:51:12.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.289+0000 7fd789d9b640 10 monclient(hunting): my global_id is 764907 2024-09-16T03:51:12.298 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.289+0000 7fd789d9b640 1 -- 172.21.15.42:0/3909396382 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd774005740 con 0x7fd78c09bf90 2024-09-16T03:51:12.298 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.289+0000 7fd78ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:51:12.298 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.289+0000 7fd78ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:51:12.298 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.289+0000 7fd78ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:51:12.298 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.289+0000 7fd78b59e640 10 monclient(hunting): handle_auth_done global_id 764910 payload 274 2024-09-16T03:51:12.298 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.289+0000 7fd78b59e640 1 -- 172.21.15.42:0/3909396382 >> v2:172.21.15.154:3300/0 conn(0x7fd78c09fb00 msgr2=0x7fd78c09ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:51:12.298 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.289+0000 7fd78b59e640 1 --2- 172.21.15.42:0/3909396382 >> v2:172.21.15.154:3300/0 conn(0x7fd78c09fb00 0x7fd78c09ba50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:12.298 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.289+0000 7fd78b59e640 1 -- 172.21.15.42:0/3909396382 >> v1:172.21.15.42:6789/0 conn(0x7fd78c09bf90 legacy=0x7fd78c138110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:12.298 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.289+0000 7fd78b59e640 10 monclient: _finish_hunting 0 2024-09-16T03:51:12.298 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.289+0000 7fd78b59e640 1 monclient: found mon.noname-a 2024-09-16T03:51:12.298 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.289+0000 7fd78b59e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:51:12.298 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.289+0000 7fd78b59e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:51:12.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.289+0000 7fd78b59e640 1 -- 172.21.15.42:0/3909396382 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd78c09ea30 con 0x7fd78c09c400 2024-09-16T03:51:12.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.289+0000 7fd78b59e640 1 --2- 172.21.15.42:0/3909396382 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd78c09c400 0x7fd78c13b840 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fd78c0a68e0 tx=0x7fd78401c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc9ca176abe9600f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:51:12.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.289+0000 7fd789d9b640 1 -- 172.21.15.42:0/3909396382 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7840b2070 con 0x7fd78c09c400 2024-09-16T03:51:12.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.289+0000 7fd789d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:51:12.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.289+0000 7fd789d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:51:12.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.289+0000 7fd789d9b640 10 monclient: dump: 2024-09-16T03:51:12.304 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:51:12.304 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:12.304 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:12.304 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:12.304 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:51:12.304 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:12.304 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:51:12.304 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:51:12.304 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:51:12.305 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:12.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.289+0000 7fd789d9b640 10 monclient: _finish_auth 0 2024-09-16T03:51:12.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.289+0000 7fd789d9b640 10 monclient: _check_auth_tickets 2024-09-16T03:51:12.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.289+0000 7fd789d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:51:12.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.289+0000 7fd79119c640 5 monclient: authenticate success, global_id 764910 2024-09-16T03:51:12.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.289+0000 7fd79119c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:51:12.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.289+0000 7fd789d9b640 1 -- 172.21.15.42:0/3909396382 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7840a5830 con 0x7fd78c09c400 2024-09-16T03:51:12.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.289+0000 7fd789d9b640 10 monclient: handle_config config(0 keys) 2024-09-16T03:51:12.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.289+0000 7fd79119c640 10 monclient: get_monmap_and_config success 2024-09-16T03:51:12.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.289+0000 7fd79119c640 4 set_mon_vals no callback set 2024-09-16T03:51:12.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.289+0000 7fd789d9b640 1 -- 172.21.15.42:0/3909396382 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7840a5be0 con 0x7fd78c09c400 2024-09-16T03:51:12.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.289+0000 7fd789d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:51:12.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.289+0000 7fd789d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:51:12.306 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.289+0000 7fd789d9b640 10 monclient: dump: 2024-09-16T03:51:12.306 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:51:12.306 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:12.306 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:12.306 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:12.306 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:51:12.306 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:12.306 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:51:12.306 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:51:12.306 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:51:12.306 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:12.306 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.293+0000 7fd79119c640 10 monclient: shutdown 2024-09-16T03:51:12.306 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.293+0000 7fd79119c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:51:12.307 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.293+0000 7fd79119c640 1 -- 172.21.15.42:0/3909396382 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd78c09c400 msgr2=0x7fd78c13b840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:12.307 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.293+0000 7fd79119c640 1 --2- 172.21.15.42:0/3909396382 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd78c09c400 0x7fd78c13b840 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fd78c0a68e0 tx=0x7fd78401c7f0 comp rx=0 tx=0).stop 2024-09-16T03:51:12.307 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.293+0000 7fd79119c640 1 -- 172.21.15.42:0/3909396382 shutdown_connections 2024-09-16T03:51:12.307 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.293+0000 7fd79119c640 1 --2- 172.21.15.42:0/3909396382 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd78c09c400 0x7fd78c13b840 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:12.307 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.293+0000 7fd79119c640 1 --2- 172.21.15.42:0/3909396382 >> v2:172.21.15.154:3300/0 conn(0x7fd78c09fb00 0x7fd78c09ba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:12.307 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.293+0000 7fd79119c640 1 -- 172.21.15.42:0/3909396382 >> 172.21.15.42:0/3909396382 conn(0x7fd78c0a2420 msgr2=0x7fd78c0a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:51:12.307 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.293+0000 7fd79119c640 1 -- 172.21.15.42:0/3909396382 shutdown_connections 2024-09-16T03:51:12.307 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.293+0000 7fd79119c640 1 -- 172.21.15.42:0/3909396382 wait complete. 2024-09-16T03:51:12.307 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.293+0000 7fd79119c640 10 monclient: build_initial_monmap 2024-09-16T03:51:12.307 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.293+0000 7fd79119c640 10 monclient: monmap: 2024-09-16T03:51:12.307 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:51:12.307 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:51:12.307 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:51:12.307 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:51:12.308 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:51:12.308 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:12.308 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:51:12.308 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:51:12.308 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:51:12.308 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:12.308 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.293+0000 7fd79119c640 1 librados: starting msgr at 2024-09-16T03:51:12.308 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.293+0000 7fd79119c640 1 librados: starting objecter 2024-09-16T03:51:12.308 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.293+0000 7fd79119c640 1 Processor -- start 2024-09-16T03:51:12.308 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.293+0000 7fd79119c640 1 -- start start 2024-09-16T03:51:12.308 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.293+0000 7fd79119c640 1 librados: setting wanted keys 2024-09-16T03:51:12.308 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.293+0000 7fd79119c640 1 librados: calling monclient init 2024-09-16T03:51:12.309 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.293+0000 7fd79119c640 10 monclient: init 2024-09-16T03:51:12.309 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.293+0000 7fd79119c640 10 monclient: _reopen_session rank -1 2024-09-16T03:51:12.309 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.293+0000 7fd79119c640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:51:12.309 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.293+0000 7fd79119c640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd78c09bf90 0x7fd78c133100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:12.309 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.293+0000 7fd79119c640 10 monclient(hunting): picked mon.noname-c con 0x7fd78c09bf90 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:51:12.309 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.293+0000 7fd79119c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd78c09c400 0x7fd78c133640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:12.309 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.293+0000 7fd79119c640 10 monclient(hunting): picked mon.noname-b con 0x7fd78c09c400 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:51:12.309 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.293+0000 7fd79119c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd78c09fb00 0x7fd78c138fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:12.309 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.293+0000 7fd79119c640 10 monclient(hunting): picked mon.noname-a con 0x7fd78c09fb00 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:51:12.309 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.293+0000 7fd79119c640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:12.309 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.293+0000 7fd79119c640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fd78c0a3170 con 0x7fd78c09fb00 2024-09-16T03:51:12.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.293+0000 7fd79119c640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:12.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.293+0000 7fd79119c640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd78c09e000 con 0x7fd78c09c400 2024-09-16T03:51:12.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.293+0000 7fd79119c640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:12.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.293+0000 7fd79119c640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fd78c0a32f0 con 0x7fd78c09bf90 2024-09-16T03:51:12.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.293+0000 7fd79119c640 10 monclient(hunting): _renew_subs 2024-09-16T03:51:12.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.293+0000 7fd79119c640 10 monclient(hunting): authenticate will time out at 1519.266056s 2024-09-16T03:51:12.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.293+0000 7fd78ad9d640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd78c09bf90 0x7fd78c133100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:12.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.293+0000 7fd78b59e640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd78c09fb00 0x7fd78c138fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:12.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.293+0000 7fd78ad9d640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd78c09bf90 0x7fd78c133100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:43508/0 (socket says 172.21.15.42:43508) 2024-09-16T03:51:12.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.293+0000 7fd78ad9d640 1 -- 172.21.15.42:0/591793374 learned_addr learned my addr 172.21.15.42:0/591793374 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:51:12.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.293+0000 7fd78b59e640 10 monclient(hunting): get_auth_request con 0x7fd78c09fb00 auth_method 0 2024-09-16T03:51:12.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.293+0000 7fd78b59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:12.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.293+0000 7fd78b59e640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:12.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.293+0000 7fd78b59e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:51:12.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.293+0000 7fd78ad9d640 10 monclient(hunting): get_auth_request con 0x7fd78c09bf90 auth_method 0 2024-09-16T03:51:12.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.293+0000 7fd78ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:12.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.293+0000 7fd78ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:12.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.293+0000 7fd78ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:51:12.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.293+0000 7fd78b59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:51:12.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.293+0000 7fd78b59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:51:12.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.293+0000 7fd78b59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:51:12.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.293+0000 7fd78ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:51:12.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.293+0000 7fd78ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:51:12.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.293+0000 7fd78ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:51:12.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.293+0000 7fd78a59c640 1 --2- 172.21.15.42:0/591793374 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd78c09c400 0x7fd78c133640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:12.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.293+0000 7fd78a59c640 10 monclient(hunting): get_auth_request con 0x7fd78c09c400 auth_method 0 2024-09-16T03:51:12.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.293+0000 7fd78a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:12.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.293+0000 7fd78a59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:12.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.293+0000 7fd78a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:51:12.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.293+0000 7fd78b59e640 10 monclient(hunting): handle_auth_done global_id 764913 payload 824 2024-09-16T03:51:12.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.293+0000 7fd78b59e640 1 -- 172.21.15.42:0/591793374 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd78c09bf90 msgr2=0x7fd78c133100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:12.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.293+0000 7fd78b59e640 1 --2- 172.21.15.42:0/591793374 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd78c09bf90 0x7fd78c133100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:12.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.293+0000 7fd78b59e640 1 -- 172.21.15.42:0/591793374 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd78c09c400 msgr2=0x7fd78c133640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:12.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.293+0000 7fd78b59e640 1 --2- 172.21.15.42:0/591793374 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd78c09c400 0x7fd78c133640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:12.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.293+0000 7fd78b59e640 10 monclient: _finish_hunting 0 2024-09-16T03:51:12.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.293+0000 7fd78b59e640 1 monclient: found mon.noname-a 2024-09-16T03:51:12.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.293+0000 7fd78b59e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:51:12.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.293+0000 7fd78b59e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:51:12.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.293+0000 7fd78b59e640 1 -- 172.21.15.42:0/591793374 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd78c06db00 con 0x7fd78c09fb00 2024-09-16T03:51:12.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.293+0000 7fd78b59e640 1 --2- 172.21.15.42:0/591793374 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd78c09fb00 0x7fd78c138fa0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fd7840ac000 tx=0x7fd784019590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:51:12.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.293+0000 7fd7737fe640 10 client.?.objecter ms_handle_connect 0x7fd78c09fb00 2024-09-16T03:51:12.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.293+0000 7fd7737fe640 10 client.?.objecter resend_mon_ops 2024-09-16T03:51:12.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.293+0000 7fd7737fe640 1 -- 172.21.15.42:0/591793374 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7840b2050 con 0x7fd78c09fb00 2024-09-16T03:51:12.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.293+0000 7fd7737fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:51:12.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.293+0000 7fd7737fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:51:12.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.293+0000 7fd7737fe640 10 monclient: dump: 2024-09-16T03:51:12.313 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:51:12.313 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:12.313 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:12.313 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:12.313 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:51:12.313 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:12.313 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:51:12.314 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:51:12.314 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:51:12.314 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:12.314 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.293+0000 7fd7737fe640 10 monclient: _finish_auth 0 2024-09-16T03:51:12.314 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.293+0000 7fd7737fe640 10 monclient: _check_auth_tickets 2024-09-16T03:51:12.314 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.293+0000 7fd7737fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:51:12.314 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.293+0000 7fd7737fe640 1 -- 172.21.15.42:0/591793374 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7840be070 con 0x7fd78c09fb00 2024-09-16T03:51:12.314 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.293+0000 7fd7737fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:51:12.314 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.293+0000 7fd7737fe640 1 -- 172.21.15.42:0/591793374 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7840ad870 con 0x7fd78c09fb00 2024-09-16T03:51:12.314 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.293+0000 7fd7737fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:51:12.314 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.293+0000 7fd7737fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:51:12.314 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.293+0000 7fd7737fe640 10 monclient: dump: 2024-09-16T03:51:12.314 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:51:12.314 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:12.315 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:12.315 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:12.315 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:51:12.315 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:12.315 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:51:12.315 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:51:12.315 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:51:12.315 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:12.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.293+0000 7fd78959a640 4 set_mon_vals no callback set 2024-09-16T03:51:12.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.293+0000 7fd79119c640 5 monclient: authenticate success, global_id 764913 2024-09-16T03:51:12.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.293+0000 7fd79119c640 10 monclient: _renew_subs 2024-09-16T03:51:12.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.293+0000 7fd79119c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:51:12.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.293+0000 7fd79119c640 1 -- 172.21.15.42:0/591793374 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd78c073a30 con 0x7fd78c09fb00 2024-09-16T03:51:12.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.293+0000 7fd79119c640 10 client.764913.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:51:12.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.293+0000 7fd79119c640 10 monclient: _renew_subs 2024-09-16T03:51:12.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.293+0000 7fd79119c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:51:12.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.293+0000 7fd79119c640 1 -- 172.21.15.42:0/591793374 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd78c121230 con 0x7fd78c09fb00 2024-09-16T03:51:12.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.293+0000 7fd79119c640 1 librados: init done 2024-09-16T03:51:12.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.297+0000 7fd79119c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:51:12.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.297+0000 7fd79119c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:51:12.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.297+0000 7fd7737fe640 1 -- 172.21.15.42:0/591793374 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fd7840ada10 con 0x7fd78c09fb00 2024-09-16T03:51:12.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.297+0000 7fd79119c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:51:12.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.297+0000 7fd79119c640 1 -- 172.21.15.42:0/591793374 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd78c09ba50 con 0x7fd78c09fb00 2024-09-16T03:51:12.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.297+0000 7fd7737fe640 1 --2- 172.21.15.42:0/591793374 >> v2:172.21.15.154:6812/254845342 conn(0x7fd778037810 0x7fd778039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:12.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.297+0000 7fd7737fe640 1 -- 172.21.15.42:0/591793374 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(308..308 src has 1..308) ==== 5812+0+0 (secure 0 0 0) 0x7fd7840e3e90 con 0x7fd78c09fb00 2024-09-16T03:51:12.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.297+0000 7fd7737fe640 10 client.764913.objecter ms_dispatch 0x7fd78c000ba0 osd_map(308..308 src has 1..308) 2024-09-16T03:51:12.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.297+0000 7fd7737fe640 3 client.764913.objecter handle_osd_map got epochs [308,308] > 0 2024-09-16T03:51:12.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.297+0000 7fd7737fe640 3 client.764913.objecter handle_osd_map decoding full epoch 308 2024-09-16T03:51:12.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.297+0000 7fd7737fe640 20 client.764913.objecter dump_active .. 0 homeless 2024-09-16T03:51:12.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.297+0000 7fd78ad9d640 1 --2- 172.21.15.42:0/591793374 >> v2:172.21.15.154:6812/254845342 conn(0x7fd778037810 0x7fd778039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:12.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.297+0000 7fd78ad9d640 10 monclient: get_auth_request con 0x7fd778037810 auth_method 0 2024-09-16T03:51:12.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.297+0000 7fd7737fe640 10 client.764913.objecter ms_handle_connect 0x7fd778037810 2024-09-16T03:51:12.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.297+0000 7fd78ad9d640 1 --2- 172.21.15.42:0/591793374 >> v2:172.21.15.154:6812/254845342 conn(0x7fd778037810 0x7fd778039cd0 secure :-1 s=READY pgs=3555 cs=0 l=1 rev1=1 crypto rx=0x7fd77c009c20 tx=0x7fd77c009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:51:12.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.301+0000 7fd7737fe640 1 -- 172.21.15.42:0/591793374 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7fd784016500 con 0x7fd78c09fb00 2024-09-16T03:51:12.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.301+0000 7fd7737fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:51:12.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.301+0000 7fd7737fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:51:12.527 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.521+0000 7fd79119c640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:51:12.527 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.521+0000 7fd79119c640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:51:12.527 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.521+0000 7fd79119c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:51:12.527 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.521+0000 7fd79119c640 1 -- 172.21.15.42:0/591793374 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fd78c069040 con 0x7fd78c09fb00 2024-09-16T03:51:12.528 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.521+0000 7fd7737fe640 1 -- 172.21.15.42:0/591793374 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7fd7840c23c0 con 0x7fd78c09fb00 2024-09-16T03:51:12.528 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.521+0000 7fd7737fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:51:12.528 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.521+0000 7fd7737fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:51:12.528 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":80,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:51:12.530 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.525+0000 7fd79119c640 10 librados: watch_flush enter 2024-09-16T03:51:12.530 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.525+0000 7fd79119c640 10 librados: watch_flush exit 2024-09-16T03:51:12.530 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.525+0000 7fd79119c640 20 client.764913.objecter shutdown clearing up homeless session... 2024-09-16T03:51:12.530 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.525+0000 7fd79119c640 10 client.764913.objecter successfully canceled tick 2024-09-16T03:51:12.530 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.525+0000 7fd79119c640 1 -- 172.21.15.42:0/591793374 >> v2:172.21.15.154:6812/254845342 conn(0x7fd778037810 msgr2=0x7fd778039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:12.530 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.525+0000 7fd79119c640 1 --2- 172.21.15.42:0/591793374 >> v2:172.21.15.154:6812/254845342 conn(0x7fd778037810 0x7fd778039cd0 secure :-1 s=READY pgs=3555 cs=0 l=1 rev1=1 crypto rx=0x7fd77c009c20 tx=0x7fd77c009190 comp rx=0 tx=0).stop 2024-09-16T03:51:12.530 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.525+0000 7fd79119c640 10 monclient: shutdown 2024-09-16T03:51:12.530 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.525+0000 7fd79119c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:51:12.530 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.525+0000 7fd79119c640 1 -- 172.21.15.42:0/591793374 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd78c09fb00 msgr2=0x7fd78c138fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:12.530 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.525+0000 7fd79119c640 1 --2- 172.21.15.42:0/591793374 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd78c09fb00 0x7fd78c138fa0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fd7840ac000 tx=0x7fd784019590 comp rx=0 tx=0).stop 2024-09-16T03:51:12.531 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.525+0000 7fd79119c640 1 -- 172.21.15.42:0/591793374 shutdown_connections 2024-09-16T03:51:12.531 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.525+0000 7fd79119c640 1 --2- 172.21.15.42:0/591793374 >> v2:172.21.15.154:6812/254845342 conn(0x7fd778037810 0x7fd778039cd0 unknown :-1 s=CLOSED pgs=3555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:12.531 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.525+0000 7fd79119c640 1 --2- 172.21.15.42:0/591793374 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd78c09fb00 0x7fd78c138fa0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:12.531 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.525+0000 7fd79119c640 1 --2- 172.21.15.42:0/591793374 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd78c09c400 0x7fd78c133640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:12.531 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.525+0000 7fd79119c640 1 --2- 172.21.15.42:0/591793374 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd78c09bf90 0x7fd78c133100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:12.531 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.525+0000 7fd79119c640 1 -- 172.21.15.42:0/591793374 >> 172.21.15.42:0/591793374 conn(0x7fd78c0a2420 msgr2=0x7fd78c134300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:51:12.531 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.525+0000 7fd79119c640 1 -- 172.21.15.42:0/591793374 shutdown_connections 2024-09-16T03:51:12.531 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.525+0000 7fd79119c640 1 -- 172.21.15.42:0/591793374 wait complete. 2024-09-16T03:51:12.531 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.525+0000 7fd79119c640 1 librados: shutdown 2024-09-16T03:51:12.549 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T03:51:12.549 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T03:51:12.549 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T03:51:12.549 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T03:51:12.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.661+0000 7efd74eac640 10 monclient: get_monmap_and_config 2024-09-16T03:51:12.670 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.661+0000 7efd74eac640 10 monclient: build_initial_monmap 2024-09-16T03:51:12.670 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.665+0000 7efd74eac640 10 monclient: monmap: 2024-09-16T03:51:12.670 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:51:12.670 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:12.670 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:51:12.668142+0000 2024-09-16T03:51:12.670 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:51:12.668142+0000 2024-09-16T03:51:12.670 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:51:12.670 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:12.670 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:51:12.670 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:51:12.670 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:51:12.670 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:51:12.671 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:51:12.671 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:12.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.665+0000 7efd74eac640 1 Processor -- start 2024-09-16T03:51:12.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.665+0000 7efd74eac640 1 -- start start 2024-09-16T03:51:12.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.665+0000 7efd74eac640 10 monclient: init 2024-09-16T03:51:12.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.665+0000 7efd74eac640 10 monclient: _reopen_session rank -1 2024-09-16T03:51:12.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.665+0000 7efd74eac640 10 monclient: _add_conns ranks=[0,2,4,3,1] 2024-09-16T03:51:12.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.665+0000 7efd74eac640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7efd7009fb00 addr v1:172.21.15.42:6789/0 2024-09-16T03:51:12.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.665+0000 7efd74eac640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7efd7009c160 0x7efd70139120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:12.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.665+0000 7efd74eac640 10 monclient(hunting): picked mon.noname-b con 0x7efd7009c160 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:51:12.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.665+0000 7efd74eac640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7efd7009c5a0 0x7efd7013b670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:12.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.665+0000 7efd74eac640 10 monclient(hunting): picked mon.noname-c con 0x7efd7009c5a0 addr v2:172.21.15.154:3300/0 2024-09-16T03:51:12.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.665+0000 7efd74eac640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efd700a1e20 con 0x7efd7009fb00 2024-09-16T03:51:12.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.665+0000 7efd74eac640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:12.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.665+0000 7efd74eac640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7efd7009e250 con 0x7efd7009c5a0 2024-09-16T03:51:12.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.665+0000 7efd74eac640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:12.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.665+0000 7efd74eac640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7efd7009ea30 con 0x7efd7009c160 2024-09-16T03:51:12.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.665+0000 7efd74eac640 10 monclient(hunting): _renew_subs 2024-09-16T03:51:12.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.665+0000 7efd74eac640 10 monclient(hunting): authenticate will time out at 1519.637557s 2024-09-16T03:51:12.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.665+0000 7efd6e59c640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7efd7009fb00 0x7efd7009ba50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:50530/0 (socket says 172.21.15.42:50530) 2024-09-16T03:51:12.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.665+0000 7efd6e59c640 1 -- 172.21.15.42:0/1743197926 learned_addr learned my addr 172.21.15.42:0/1743197926 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:51:12.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.665+0000 7efd67fff640 1 --2- 172.21.15.42:0/1743197926 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7efd7009c160 0x7efd70139120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:12.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.665+0000 7efd67fff640 10 monclient(hunting): get_auth_request con 0x7efd7009c160 auth_method 0 2024-09-16T03:51:12.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.665+0000 7efd67fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:12.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.665+0000 7efd67fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:12.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.665+0000 7efd67fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:51:12.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.665+0000 7efd67fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:51:12.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.665+0000 7efd67fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:51:12.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.665+0000 7efd67fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:51:12.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.665+0000 7efd6ed9d640 1 --2- 172.21.15.42:0/1743197926 >> v2:172.21.15.154:3300/0 conn(0x7efd7009c5a0 0x7efd7013b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:12.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.665+0000 7efd6ed9d640 10 monclient(hunting): get_auth_request con 0x7efd7009c5a0 auth_method 0 2024-09-16T03:51:12.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.665+0000 7efd6ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:12.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.665+0000 7efd6ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:12.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.665+0000 7efd6ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:51:12.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.665+0000 7efd6dd9b640 1 -- 172.21.15.42:0/1743197926 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2751317382 0 0) 0x7efd700a1e20 con 0x7efd7009fb00 2024-09-16T03:51:12.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.665+0000 7efd6dd9b640 10 monclient(hunting): _negotiate 2024-09-16T03:51:12.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.665+0000 7efd6dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:12.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.665+0000 7efd6dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:51:12.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.665+0000 7efd6dd9b640 10 monclient(hunting): my global_id is 764916 2024-09-16T03:51:12.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.665+0000 7efd6dd9b640 1 -- 172.21.15.42:0/1743197926 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efd5c003530 con 0x7efd7009fb00 2024-09-16T03:51:12.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.665+0000 7efd67fff640 10 monclient(hunting): handle_auth_done global_id 814136 payload 274 2024-09-16T03:51:12.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.665+0000 7efd67fff640 1 -- 172.21.15.42:0/1743197926 >> v2:172.21.15.154:3300/0 conn(0x7efd7009c5a0 msgr2=0x7efd7013b670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:51:12.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.665+0000 7efd67fff640 1 --2- 172.21.15.42:0/1743197926 >> v2:172.21.15.154:3300/0 conn(0x7efd7009c5a0 0x7efd7013b670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:12.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.665+0000 7efd67fff640 1 -- 172.21.15.42:0/1743197926 >> v1:172.21.15.42:6789/0 conn(0x7efd7009fb00 legacy=0x7efd7009ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:12.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.665+0000 7efd67fff640 10 monclient: _finish_hunting 0 2024-09-16T03:51:12.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.665+0000 7efd67fff640 1 monclient: found mon.noname-b 2024-09-16T03:51:12.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.665+0000 7efd67fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:51:12.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.665+0000 7efd67fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T03:51:12.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.665+0000 7efd67fff640 1 -- 172.21.15.42:0/1743197926 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd7009fee0 con 0x7efd7009c160 2024-09-16T03:51:12.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.669+0000 7efd67fff640 1 --2- 172.21.15.42:0/1743197926 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7efd7009c160 0x7efd70139120 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7efd580097c0 tx=0x7efd5801c360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=65834ab0072d573f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:51:12.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.669+0000 7efd6dd9b640 1 -- 172.21.15.42:0/1743197926 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd580b0070 con 0x7efd7009c160 2024-09-16T03:51:12.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.669+0000 7efd6dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:51:12.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.669+0000 7efd6dd9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:51:12.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.669+0000 7efd6dd9b640 10 monclient: dump: 2024-09-16T03:51:12.693 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:51:12.693 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:12.693 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:12.693 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:12.693 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:51:12.694 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:12.694 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:51:12.694 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:51:12.694 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:51:12.694 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:12.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.669+0000 7efd6dd9b640 10 monclient: _finish_auth 0 2024-09-16T03:51:12.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.669+0000 7efd6dd9b640 10 monclient: _check_auth_tickets 2024-09-16T03:51:12.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.669+0000 7efd6dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:51:12.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.669+0000 7efd6dd9b640 1 -- 172.21.15.42:0/1743197926 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efd580ab600 con 0x7efd7009c160 2024-09-16T03:51:12.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.669+0000 7efd74eac640 5 monclient: authenticate success, global_id 814136 2024-09-16T03:51:12.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.669+0000 7efd6dd9b640 10 monclient: handle_config config(0 keys) 2024-09-16T03:51:12.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.669+0000 7efd74eac640 10 monclient: get_monmap_and_config success 2024-09-16T03:51:12.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.669+0000 7efd6dd9b640 1 -- 172.21.15.42:0/1743197926 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd580a4e60 con 0x7efd7009c160 2024-09-16T03:51:12.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.669+0000 7efd74eac640 4 set_mon_vals no callback set 2024-09-16T03:51:12.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.669+0000 7efd6dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:51:12.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.669+0000 7efd6dd9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:51:12.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.669+0000 7efd6dd9b640 10 monclient: dump: 2024-09-16T03:51:12.695 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:51:12.695 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:12.695 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:12.695 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:12.695 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:51:12.695 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:12.695 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:51:12.696 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:51:12.696 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:51:12.696 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:12.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.669+0000 7efd74eac640 10 monclient: shutdown 2024-09-16T03:51:12.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.669+0000 7efd74eac640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:51:12.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.669+0000 7efd74eac640 1 -- 172.21.15.42:0/1743197926 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7efd7009c160 msgr2=0x7efd70139120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:12.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.669+0000 7efd74eac640 1 --2- 172.21.15.42:0/1743197926 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7efd7009c160 0x7efd70139120 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7efd580097c0 tx=0x7efd5801c360 comp rx=0 tx=0).stop 2024-09-16T03:51:12.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.669+0000 7efd74eac640 1 -- 172.21.15.42:0/1743197926 shutdown_connections 2024-09-16T03:51:12.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.669+0000 7efd74eac640 1 --2- 172.21.15.42:0/1743197926 >> v2:172.21.15.154:3300/0 conn(0x7efd7009c5a0 0x7efd7013b670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:12.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.669+0000 7efd74eac640 1 --2- 172.21.15.42:0/1743197926 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7efd7009c160 0x7efd70139120 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:12.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.669+0000 7efd74eac640 1 -- 172.21.15.42:0/1743197926 >> 172.21.15.42:0/1743197926 conn(0x7efd700a2420 msgr2=0x7efd700a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:51:12.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.669+0000 7efd74eac640 1 -- 172.21.15.42:0/1743197926 shutdown_connections 2024-09-16T03:51:12.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.669+0000 7efd74eac640 1 -- 172.21.15.42:0/1743197926 wait complete. 2024-09-16T03:51:12.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.669+0000 7efd74eac640 10 monclient: build_initial_monmap 2024-09-16T03:51:12.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.669+0000 7efd74eac640 10 monclient: monmap: 2024-09-16T03:51:12.697 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:51:12.697 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:51:12.697 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:51:12.697 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:51:12.697 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:51:12.697 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:12.697 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:51:12.697 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:51:12.698 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:51:12.698 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:12.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.669+0000 7efd74eac640 1 librados: starting msgr at 2024-09-16T03:51:12.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.669+0000 7efd74eac640 1 librados: starting objecter 2024-09-16T03:51:12.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.669+0000 7efd74eac640 1 Processor -- start 2024-09-16T03:51:12.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.669+0000 7efd74eac640 1 -- start start 2024-09-16T03:51:12.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.669+0000 7efd74eac640 1 librados: setting wanted keys 2024-09-16T03:51:12.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.669+0000 7efd74eac640 1 librados: calling monclient init 2024-09-16T03:51:12.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.669+0000 7efd74eac640 10 monclient: init 2024-09-16T03:51:12.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.669+0000 7efd74eac640 10 monclient: _reopen_session rank -1 2024-09-16T03:51:12.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.669+0000 7efd74eac640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T03:51:12.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.669+0000 7efd74eac640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efd7009c160 0x7efd701327a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:12.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.669+0000 7efd74eac640 10 monclient(hunting): picked mon.noname-a con 0x7efd7009c160 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:51:12.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.669+0000 7efd74eac640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efd7009c5a0 0x7efd70132ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:12.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.669+0000 7efd74eac640 10 monclient(hunting): picked mon.noname-b con 0x7efd7009c5a0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:51:12.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.669+0000 7efd74eac640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7efd7009fb00 0x7efd701391f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:12.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.669+0000 7efd74eac640 10 monclient(hunting): picked mon.noname-c con 0x7efd7009fb00 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:51:12.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.669+0000 7efd74eac640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:12.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.669+0000 7efd74eac640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7efd7009d380 con 0x7efd7009c160 2024-09-16T03:51:12.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.669+0000 7efd74eac640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:12.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.669+0000 7efd74eac640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7efd7009d130 con 0x7efd7009c5a0 2024-09-16T03:51:12.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.669+0000 7efd74eac640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:12.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.669+0000 7efd74eac640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7efd7009d500 con 0x7efd7009fb00 2024-09-16T03:51:12.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.669+0000 7efd74eac640 10 monclient(hunting): _renew_subs 2024-09-16T03:51:12.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.669+0000 7efd74eac640 10 monclient(hunting): authenticate will time out at 1519.642523s 2024-09-16T03:51:12.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.669+0000 7efd6e59c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efd7009c160 0x7efd701327a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:12.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.669+0000 7efd6ed9d640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7efd7009fb00 0x7efd701391f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:12.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.669+0000 7efd6e59c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efd7009c160 0x7efd701327a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:53120/0 (socket says 172.21.15.42:53120) 2024-09-16T03:51:12.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.669+0000 7efd6e59c640 1 -- 172.21.15.42:0/1501823970 learned_addr learned my addr 172.21.15.42:0/1501823970 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:51:12.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.669+0000 7efd6e59c640 10 monclient(hunting): get_auth_request con 0x7efd7009c160 auth_method 0 2024-09-16T03:51:12.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.669+0000 7efd6e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:12.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.669+0000 7efd6e59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:12.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.669+0000 7efd6e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:51:12.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.669+0000 7efd6ed9d640 10 monclient(hunting): get_auth_request con 0x7efd7009fb00 auth_method 0 2024-09-16T03:51:12.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.669+0000 7efd6ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:12.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.669+0000 7efd6ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:12.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.669+0000 7efd6ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:51:12.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.669+0000 7efd6e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:51:12.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.669+0000 7efd6e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:51:12.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.669+0000 7efd6e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:51:12.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.669+0000 7efd6ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:51:12.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.669+0000 7efd6ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:51:12.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.669+0000 7efd6ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:51:12.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.669+0000 7efd67fff640 1 --2- 172.21.15.42:0/1501823970 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efd7009c5a0 0x7efd70132ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:12.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.669+0000 7efd67fff640 10 monclient(hunting): get_auth_request con 0x7efd7009c5a0 auth_method 0 2024-09-16T03:51:12.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.669+0000 7efd67fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:12.702 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.669+0000 7efd67fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:12.702 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.669+0000 7efd67fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:51:12.702 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.669+0000 7efd6e59c640 10 monclient(hunting): handle_auth_done global_id 764919 payload 824 2024-09-16T03:51:12.702 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.669+0000 7efd6e59c640 1 -- 172.21.15.42:0/1501823970 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7efd7009fb00 msgr2=0x7efd701391f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:12.702 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.669+0000 7efd6e59c640 1 --2- 172.21.15.42:0/1501823970 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7efd7009fb00 0x7efd701391f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:12.702 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.669+0000 7efd6e59c640 1 -- 172.21.15.42:0/1501823970 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efd7009c5a0 msgr2=0x7efd70132ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:12.702 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.669+0000 7efd6e59c640 1 --2- 172.21.15.42:0/1501823970 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efd7009c5a0 0x7efd70132ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:12.702 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.669+0000 7efd6e59c640 10 monclient: _finish_hunting 0 2024-09-16T03:51:12.702 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.669+0000 7efd6e59c640 1 monclient: found mon.noname-a 2024-09-16T03:51:12.702 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.669+0000 7efd6e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:51:12.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.669+0000 7efd6e59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:51:12.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.669+0000 7efd6e59c640 1 -- 172.21.15.42:0/1501823970 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd7011d340 con 0x7efd7009c160 2024-09-16T03:51:12.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.669+0000 7efd6ed9d640 1 --2- 172.21.15.42:0/1501823970 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7efd7009fb00 0x7efd701391f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:51:12.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.669+0000 7efd6e59c640 1 --2- 172.21.15.42:0/1501823970 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efd7009c160 0x7efd701327a0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7efd6000c480 tx=0x7efd6000c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:51:12.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.669+0000 7efd66ffd640 10 client.?.objecter ms_handle_connect 0x7efd7009c160 2024-09-16T03:51:12.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.669+0000 7efd66ffd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:51:12.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.669+0000 7efd66ffd640 1 -- 172.21.15.42:0/1501823970 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd60017070 con 0x7efd7009c160 2024-09-16T03:51:12.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.669+0000 7efd66ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:51:12.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.669+0000 7efd66ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:51:12.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.669+0000 7efd66ffd640 10 monclient: dump: 2024-09-16T03:51:12.704 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:51:12.704 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:12.704 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:12.704 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:12.704 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:51:12.704 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:12.704 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:51:12.704 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:51:12.704 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:51:12.704 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:12.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.669+0000 7efd66ffd640 10 monclient: _finish_auth 0 2024-09-16T03:51:12.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.669+0000 7efd66ffd640 10 monclient: _check_auth_tickets 2024-09-16T03:51:12.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.669+0000 7efd66ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:51:12.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.669+0000 7efd74eac640 5 monclient: authenticate success, global_id 764919 2024-09-16T03:51:12.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.669+0000 7efd66ffd640 1 -- 172.21.15.42:0/1501823970 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efd60005cf0 con 0x7efd7009c160 2024-09-16T03:51:12.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.669+0000 7efd74eac640 10 monclient: _renew_subs 2024-09-16T03:51:12.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.669+0000 7efd74eac640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:51:12.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.669+0000 7efd74eac640 1 -- 172.21.15.42:0/1501823970 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd7011d620 con 0x7efd7009c160 2024-09-16T03:51:12.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.669+0000 7efd66ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:51:12.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.669+0000 7efd66ffd640 1 -- 172.21.15.42:0/1501823970 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd60007290 con 0x7efd7009c160 2024-09-16T03:51:12.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.669+0000 7efd66ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:51:12.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.669+0000 7efd66ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:51:12.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.669+0000 7efd66ffd640 10 monclient: dump: 2024-09-16T03:51:12.705 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:51:12.705 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:12.705 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:12.705 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:12.706 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:51:12.706 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:12.706 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:51:12.706 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:51:12.706 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:51:12.706 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:12.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.669+0000 7efd74eac640 10 client.764919.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:51:12.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.669+0000 7efd6cd99640 4 set_mon_vals no callback set 2024-09-16T03:51:12.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.669+0000 7efd74eac640 10 monclient: _renew_subs 2024-09-16T03:51:12.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.669+0000 7efd74eac640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:51:12.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.669+0000 7efd74eac640 1 -- 172.21.15.42:0/1501823970 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd7013b330 con 0x7efd7009c160 2024-09-16T03:51:12.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.669+0000 7efd74eac640 1 librados: init done 2024-09-16T03:51:12.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.673+0000 7efd74eac640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:51:12.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.673+0000 7efd74eac640 1 --2- 172.21.15.42:0/1501823970 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efd70139ef0 0x7efd7002c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:12.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.673+0000 7efd74eac640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:12.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.673+0000 7efd74eac640 1 -- 172.21.15.42:0/1501823970 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7efd70073a30 con 0x7efd70139ef0 2024-09-16T03:51:12.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.673+0000 7efd67fff640 1 --2- 172.21.15.42:0/1501823970 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efd70139ef0 0x7efd7002c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:12.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.673+0000 7efd67fff640 10 monclient: get_auth_request con 0x7efd70139ef0 auth_method 0 2024-09-16T03:51:12.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.673+0000 7efd67fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:12.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.673+0000 7efd67fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:12.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.673+0000 7efd67fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:51:12.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.673+0000 7efd67fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:51:12.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.673+0000 7efd67fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:51:12.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.673+0000 7efd67fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:51:12.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.673+0000 7efd66ffd640 1 -- 172.21.15.42:0/1501823970 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7efd6001a070 con 0x7efd7009c160 2024-09-16T03:51:12.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.673+0000 7efd67fff640 10 monclient(hunting): handle_auth_done global_id 764922 payload 824 2024-09-16T03:51:12.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.673+0000 7efd67fff640 1 -- 172.21.15.42:0/1501823970 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7efd7009ba50 con 0x7efd70139ef0 2024-09-16T03:51:12.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.673+0000 7efd66ffd640 1 --2- 172.21.15.42:0/1501823970 >> v2:172.21.15.154:6812/254845342 conn(0x7efd50037810 0x7efd50039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:12.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.673+0000 7efd66ffd640 1 -- 172.21.15.42:0/1501823970 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(308..308 src has 1..308) ==== 5812+0+0 (secure 0 0 0) 0x7efd600483d0 con 0x7efd7009c160 2024-09-16T03:51:12.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.673+0000 7efd66ffd640 10 client.764919.objecter ms_dispatch 0x7efd70000ba0 osd_map(308..308 src has 1..308) 2024-09-16T03:51:12.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.673+0000 7efd66ffd640 3 client.764919.objecter handle_osd_map got epochs [308,308] > 0 2024-09-16T03:51:12.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.673+0000 7efd66ffd640 3 client.764919.objecter handle_osd_map decoding full epoch 308 2024-09-16T03:51:12.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.673+0000 7efd67fff640 1 --2- 172.21.15.42:0/1501823970 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efd70139ef0 0x7efd7002c650 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7efd700a68e0 tx=0x7efd5801aba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:51:12.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.673+0000 7efd66ffd640 20 client.764919.objecter dump_active .. 0 homeless 2024-09-16T03:51:12.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.673+0000 7efd66ffd640 10 client.764919.objecter ms_handle_connect 0x7efd70139ef0 2024-09-16T03:51:12.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.673+0000 7efd66ffd640 10 client.764919.objecter resend_mon_ops 2024-09-16T03:51:12.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.673+0000 7efd66ffd640 1 -- 172.21.15.42:0/1501823970 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd580bb030 con 0x7efd70139ef0 2024-09-16T03:51:12.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.673+0000 7efd66ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:51:12.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.673+0000 7efd66ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:51:12.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.673+0000 7efd66ffd640 10 monclient: dump: 2024-09-16T03:51:12.709 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:51:12.709 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:12.709 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:12.709 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:12.709 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:51:12.710 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:12.710 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:51:12.710 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:51:12.710 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:51:12.710 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:12.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.673+0000 7efd6ed9d640 1 --2- 172.21.15.42:0/1501823970 >> v2:172.21.15.154:6812/254845342 conn(0x7efd50037810 0x7efd50039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:12.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.673+0000 7efd6ed9d640 10 monclient: get_auth_request con 0x7efd50037810 auth_method 0 2024-09-16T03:51:12.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.673+0000 7efd66ffd640 1 -- 172.21.15.42:0/1501823970 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7efd580093e0 con 0x7efd70139ef0 2024-09-16T03:51:12.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.673+0000 7efd66ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:51:12.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.673+0000 7efd66ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:51:12.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.673+0000 7efd66ffd640 1 -- 172.21.15.42:0/1501823970 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efd70139ef0 msgr2=0x7efd7002c650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:12.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.673+0000 7efd66ffd640 1 --2- 172.21.15.42:0/1501823970 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efd70139ef0 0x7efd7002c650 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7efd700a68e0 tx=0x7efd5801aba0 comp rx=0 tx=0).stop 2024-09-16T03:51:12.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.673+0000 7efd66ffd640 1 -- 172.21.15.42:0/1501823970 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efd70139ef0 msgr2=0x7efd7002c650 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:51:12.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.673+0000 7efd66ffd640 1 --2- 172.21.15.42:0/1501823970 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efd70139ef0 0x7efd7002c650 secure :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7efd700a68e0 tx=0x7efd5801aba0 comp rx=0 tx=0).stop 2024-09-16T03:51:12.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.673+0000 7efd66ffd640 10 client.764919.objecter ms_handle_connect 0x7efd50037810 2024-09-16T03:51:12.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.673+0000 7efd6ed9d640 1 --2- 172.21.15.42:0/1501823970 >> v2:172.21.15.154:6812/254845342 conn(0x7efd50037810 0x7efd50039cd0 secure :-1 s=READY pgs=3556 cs=0 l=1 rev1=1 crypto rx=0x7efd680077c0 tx=0x7efd68009820 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:51:12.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.677+0000 7efd74eac640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:51:12.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.677+0000 7efd74eac640 1 --2- 172.21.15.42:0/1501823970 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efd7013a2d0 0x7efd7002cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:12.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.677+0000 7efd74eac640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:12.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.677+0000 7efd74eac640 1 -- 172.21.15.42:0/1501823970 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7efd70070590 con 0x7efd7013a2d0 2024-09-16T03:51:12.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.677+0000 7efd67fff640 1 --2- 172.21.15.42:0/1501823970 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efd7013a2d0 0x7efd7002cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:12.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.677+0000 7efd67fff640 10 monclient: get_auth_request con 0x7efd7013a2d0 auth_method 0 2024-09-16T03:51:12.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.677+0000 7efd67fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:12.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.677+0000 7efd67fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:12.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.677+0000 7efd67fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:51:12.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.677+0000 7efd67fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:51:12.714 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.677+0000 7efd67fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:51:12.714 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.677+0000 7efd67fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:51:12.714 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.677+0000 7efd67fff640 10 monclient(hunting): handle_auth_done global_id 764925 payload 824 2024-09-16T03:51:12.714 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.677+0000 7efd67fff640 1 -- 172.21.15.42:0/1501823970 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7efd7011d960 con 0x7efd7013a2d0 2024-09-16T03:51:12.714 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.681+0000 7efd66ffd640 10 client.764919.objecter ms_handle_connect 0x7efd7013a2d0 2024-09-16T03:51:12.714 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.681+0000 7efd66ffd640 10 client.764919.objecter resend_mon_ops 2024-09-16T03:51:12.714 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.681+0000 7efd67fff640 1 --2- 172.21.15.42:0/1501823970 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efd7013a2d0 0x7efd7002cb90 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7efd580097c0 tx=0x7efd58019100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:51:12.714 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.681+0000 7efd66ffd640 1 -- 172.21.15.42:0/1501823970 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd580bb040 con 0x7efd7013a2d0 2024-09-16T03:51:12.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.681+0000 7efd66ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:51:12.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.681+0000 7efd66ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:51:12.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.681+0000 7efd66ffd640 10 monclient: dump: 2024-09-16T03:51:12.715 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:51:12.715 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:12.715 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:12.715 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:12.716 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:51:12.716 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:12.716 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:51:12.716 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:51:12.716 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:51:12.716 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:12.716 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.681+0000 7efd66ffd640 1 -- 172.21.15.42:0/1501823970 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4881 (secure 0 0 0) 0x7efd580c1a10 con 0x7efd7013a2d0 2024-09-16T03:51:12.716 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.681+0000 7efd66ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:51:12.716 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.681+0000 7efd66ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:51:12.717 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.681+0000 7efd66ffd640 1 -- 172.21.15.42:0/1501823970 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efd7013a2d0 msgr2=0x7efd7002cb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:12.717 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.681+0000 7efd66ffd640 1 --2- 172.21.15.42:0/1501823970 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efd7013a2d0 0x7efd7002cb90 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7efd580097c0 tx=0x7efd58019100 comp rx=0 tx=0).stop 2024-09-16T03:51:12.717 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.681+0000 7efd66ffd640 1 -- 172.21.15.42:0/1501823970 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efd7013a2d0 msgr2=0x7efd7002cb90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:51:12.717 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.681+0000 7efd66ffd640 1 --2- 172.21.15.42:0/1501823970 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efd7013a2d0 0x7efd7002cb90 secure :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7efd580097c0 tx=0x7efd58019100 comp rx=0 tx=0).stop 2024-09-16T03:51:12.718 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:51:12.718 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:51:12.718 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:51:12.718 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "leader", 2024-09-16T03:51:12.718 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 80, 2024-09-16T03:51:12.718 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 263473, 2024-09-16T03:51:12.718 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:51:12.719 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:51:12.719 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T03:51:12.719 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T03:51:12.719 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:51:12.719 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 2, 2024-09-16T03:51:12.719 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:51:12.719 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:51:12.719 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:51:12.719 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:51:12.719 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:51:12.720 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:51:12.720 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:51:12.720 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:51:12.720 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:51:12.720 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:51:12.720 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:51:12.720 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:51:12.720 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:51:12.720 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:51:12.720 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:51:12.720 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:51:12.720 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:51:12.720 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:51:12.721 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:51:12.721 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:51:12.721 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:51:12.721 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:51:12.721 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:51:12.721 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:51:12.721 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:51:12.721 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:51:12.721 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:51:12.721 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:51:12.721 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:51:12.721 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:12.722 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:51:12.722 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:51:12.722 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:51:12.722 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:51:12.722 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:51:12.722 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:51:12.722 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:51:12.722 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:51:12.722 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:51:12.722 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:51:12.722 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:51:12.722 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:51:12.722 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:51:12.723 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:51:12.723 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:51:12.723 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:51:12.723 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:51:12.723 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:51:12.723 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:51:12.723 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:51:12.723 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:51:12.723 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:51:12.723 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:51:12.723 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:51:12.723 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:51:12.723 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:51:12.723 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:51:12.724 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:51:12.724 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:51:12.724 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:51:12.724 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:12.724 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:51:12.724 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:12.724 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:51:12.724 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:51:12.724 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:51:12.724 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:51:12.724 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:12.724 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:51:12.724 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:51:12.725 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:51:12.725 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:12.725 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:12.725 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:51:12.725 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:51:12.725 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:51:12.725 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:51:12.725 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:51:12.725 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:12.725 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:51:12.725 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:51:12.725 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:51:12.726 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:51:12.726 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:51:12.726 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:12.726 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:12.726 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:51:12.726 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:51:12.726 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:51:12.726 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:51:12.726 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:12.727 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:51:12.727 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:51:12.727 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:51:12.727 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:12.727 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:12.727 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:51:12.727 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:51:12.727 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:51:12.727 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:51:12.727 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:51:12.727 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:12.727 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:51:12.727 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:51:12.727 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:51:12.728 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:51:12.728 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:51:12.728 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:12.728 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:12.728 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:51:12.728 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:51:12.728 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:51:12.728 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:51:12.728 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:12.728 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:51:12.728 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:51:12.728 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:51:12.728 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:12.728 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:12.729 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:51:12.729 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:51:12.729 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:51:12.729 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:51:12.729 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:51:12.729 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:12.729 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:51:12.729 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:51:12.729 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:51:12.729 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:51:12.729 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:51:12.729 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:51:12.729 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:51:12.729 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:12.730 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:51:12.730 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:51:12.730 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:12.730 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:51:12.730 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:51:12.730 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:51:12.730 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:51:12.730 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:51:12.730 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T03:51:12.730 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:12.730 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:51:12.730 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:51:12.730 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 6 2024-09-16T03:51:12.730 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:51:12.730 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:51:12.731 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:51:12.731 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:12.731 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:51:12.731 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:51:12.731 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 7 2024-09-16T03:51:12.731 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:51:12.731 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:51:12.731 INFO:teuthology.orchestra.run.smithi042.stdout: "mgr": [ 2024-09-16T03:51:12.731 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:12.731 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:51:12.731 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:51:12.731 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:51:12.731 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:51:12.731 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:51:12.732 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:12.732 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:51:12.732 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:51:12.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.681+0000 7efd74eac640 10 librados: watch_flush enter 2024-09-16T03:51:12.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.681+0000 7efd74eac640 10 librados: watch_flush exit 2024-09-16T03:51:12.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.681+0000 7efd74eac640 20 client.764919.objecter shutdown clearing up homeless session... 2024-09-16T03:51:12.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.681+0000 7efd74eac640 10 client.764919.objecter successfully canceled tick 2024-09-16T03:51:12.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.681+0000 7efd74eac640 1 -- 172.21.15.42:0/1501823970 >> v2:172.21.15.154:6812/254845342 conn(0x7efd50037810 msgr2=0x7efd50039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:12.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.681+0000 7efd74eac640 1 --2- 172.21.15.42:0/1501823970 >> v2:172.21.15.154:6812/254845342 conn(0x7efd50037810 0x7efd50039cd0 secure :-1 s=READY pgs=3556 cs=0 l=1 rev1=1 crypto rx=0x7efd680077c0 tx=0x7efd68009820 comp rx=0 tx=0).stop 2024-09-16T03:51:12.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.681+0000 7efd74eac640 10 monclient: shutdown 2024-09-16T03:51:12.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.681+0000 7efd74eac640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:51:12.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.681+0000 7efd74eac640 1 -- 172.21.15.42:0/1501823970 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efd7009c160 msgr2=0x7efd701327a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:12.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.681+0000 7efd74eac640 1 --2- 172.21.15.42:0/1501823970 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efd7009c160 0x7efd701327a0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7efd6000c480 tx=0x7efd6000c940 comp rx=0 tx=0).stop 2024-09-16T03:51:12.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.681+0000 7efd6ed9d640 1 -- 172.21.15.42:0/1501823970 reap_dead start 2024-09-16T03:51:12.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.681+0000 7efd74eac640 1 -- 172.21.15.42:0/1501823970 shutdown_connections 2024-09-16T03:51:12.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.681+0000 7efd74eac640 1 --2- 172.21.15.42:0/1501823970 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efd7009c160 0x7efd701327a0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:12.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.681+0000 7efd74eac640 1 -- 172.21.15.42:0/1501823970 >> 172.21.15.42:0/1501823970 conn(0x7efd700a2420 msgr2=0x7efd70136db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:51:12.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.681+0000 7efd74eac640 1 -- 172.21.15.42:0/1501823970 shutdown_connections 2024-09-16T03:51:12.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.681+0000 7efd74eac640 1 -- 172.21.15.42:0/1501823970 wait complete. 2024-09-16T03:51:12.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.681+0000 7efd74eac640 1 librados: shutdown 2024-09-16T03:51:12.735 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:51:12.735 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T03:51:12.735 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T03:51:12.849 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.845+0000 7f464b27d640 10 monclient: get_monmap_and_config 2024-09-16T03:51:12.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.845+0000 7f464b27d640 10 monclient: build_initial_monmap 2024-09-16T03:51:12.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.845+0000 7f464b27d640 10 monclient: monmap: 2024-09-16T03:51:12.850 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:51:12.850 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:12.850 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:51:12.848502+0000 2024-09-16T03:51:12.850 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:51:12.848502+0000 2024-09-16T03:51:12.850 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:51:12.850 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:12.850 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:51:12.850 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:51:12.850 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:51:12.850 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:51:12.850 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:51:12.851 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:12.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.845+0000 7f464b27d640 1 Processor -- start 2024-09-16T03:51:12.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.845+0000 7f464b27d640 1 -- start start 2024-09-16T03:51:12.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.845+0000 7f464b27d640 10 monclient: init 2024-09-16T03:51:12.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.845+0000 7f464b27d640 10 monclient: _reopen_session rank -1 2024-09-16T03:51:12.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.845+0000 7f464b27d640 10 monclient: _add_conns ranks=[2,1,0,4,3] 2024-09-16T03:51:12.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.845+0000 7f464b27d640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f46440a1330 0x7f46440a1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:12.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.845+0000 7f464b27d640 10 monclient(hunting): picked mon.noname-b con 0x7f46440a1330 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:51:12.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.845+0000 7f464b27d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f464406dac0 addr v1:172.21.15.154:6789/0 2024-09-16T03:51:12.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.845+0000 7f464b27d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f46440a1c50 addr v1:172.21.15.42:6789/0 2024-09-16T03:51:12.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.845+0000 7f464b27d640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f464409c7d0 con 0x7f46440a1c50 2024-09-16T03:51:12.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.845+0000 7f464b27d640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f464409c090 con 0x7f464406dac0 2024-09-16T03:51:12.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.845+0000 7f464b27d640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:12.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.845+0000 7f464b27d640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f464409f610 con 0x7f46440a1330 2024-09-16T03:51:12.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.845+0000 7f464b27d640 10 monclient(hunting): _renew_subs 2024-09-16T03:51:12.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.845+0000 7f464b27d640 10 monclient(hunting): authenticate will time out at 1519.817960s 2024-09-16T03:51:12.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.845+0000 7f464981a640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f46440a1c50 0x7f4644135810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:50542/0 (socket says 172.21.15.42:50542) 2024-09-16T03:51:12.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.845+0000 7f464981a640 1 -- 172.21.15.42:0/364780166 learned_addr learned my addr 172.21.15.42:0/364780166 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:51:12.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.845+0000 7f4649019640 1 --2- 172.21.15.42:0/364780166 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f46440a1330 0x7f46440a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:12.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.845+0000 7f4649019640 10 monclient(hunting): get_auth_request con 0x7f46440a1330 auth_method 0 2024-09-16T03:51:12.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.845+0000 7f4649019640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:12.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.845+0000 7f4649019640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:12.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.845+0000 7f4649019640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:51:12.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.845+0000 7f4649019640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:51:12.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.845+0000 7f4649019640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:51:12.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.845+0000 7f4649019640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:51:12.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.845+0000 7f4649019640 10 monclient(hunting): handle_auth_done global_id 814142 payload 274 2024-09-16T03:51:12.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.845+0000 7f463bfff640 1 -- 172.21.15.42:0/364780166 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1873673340 0 0) 0x7f464409c7d0 con 0x7f46440a1c50 2024-09-16T03:51:12.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.845+0000 7f4649019640 1 -- 172.21.15.42:0/364780166 >> v1:172.21.15.154:6789/0 conn(0x7f464406dac0 legacy=0x7f4644135100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:12.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.845+0000 7f4649019640 1 -- 172.21.15.42:0/364780166 >> v1:172.21.15.42:6789/0 conn(0x7f46440a1c50 legacy=0x7f4644135810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:12.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.845+0000 7f4649019640 10 monclient: _finish_hunting 0 2024-09-16T03:51:12.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.845+0000 7f4649019640 1 monclient: found mon.noname-b 2024-09-16T03:51:12.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.845+0000 7f4649019640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:51:12.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.845+0000 7f4649019640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T03:51:12.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.845+0000 7f4649019640 1 -- 172.21.15.42:0/364780166 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f464409f8f0 con 0x7f46440a1330 2024-09-16T03:51:12.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.845+0000 7f463bfff640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T03:51:12.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.849+0000 7f4649019640 1 --2- 172.21.15.42:0/364780166 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f46440a1330 0x7f46440a1710 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f46400097f0 tx=0x7f464001c4c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b2a408d213f0c722 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:51:12.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.849+0000 7f463bfff640 1 -- 172.21.15.42:0/364780166 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46400b1070 con 0x7f46440a1330 2024-09-16T03:51:12.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.849+0000 7f463bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:51:12.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.849+0000 7f463bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:51:12.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.849+0000 7f463bfff640 10 monclient: dump: 2024-09-16T03:51:12.857 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:51:12.857 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:12.857 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:12.857 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:12.857 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:51:12.857 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:12.857 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:51:12.857 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:51:12.857 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:51:12.857 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:12.857 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.849+0000 7f463bfff640 10 monclient: _finish_auth 0 2024-09-16T03:51:12.857 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.849+0000 7f463bfff640 10 monclient: _check_auth_tickets 2024-09-16T03:51:12.858 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.849+0000 7f463bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:51:12.858 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.849+0000 7f463bfff640 1 -- 172.21.15.42:0/364780166 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f46400ac630 con 0x7f46440a1330 2024-09-16T03:51:12.858 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.849+0000 7f463bfff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:51:12.858 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.849+0000 7f463bfff640 1 -- 172.21.15.42:0/364780166 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46400a8050 con 0x7f46440a1330 2024-09-16T03:51:12.858 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.849+0000 7f464b27d640 5 monclient: authenticate success, global_id 814142 2024-09-16T03:51:12.858 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.849+0000 7f464b27d640 10 monclient: get_monmap_and_config success 2024-09-16T03:51:12.858 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.849+0000 7f464b27d640 4 set_mon_vals no callback set 2024-09-16T03:51:12.858 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.849+0000 7f463bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:51:12.858 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.849+0000 7f463bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:51:12.858 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.849+0000 7f463bfff640 10 monclient: dump: 2024-09-16T03:51:12.858 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:51:12.858 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:12.858 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:12.859 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:12.859 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:51:12.859 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:12.859 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:51:12.859 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:51:12.859 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:51:12.859 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:12.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.849+0000 7f464b27d640 10 monclient: shutdown 2024-09-16T03:51:12.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.849+0000 7f464b27d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:51:12.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.849+0000 7f464b27d640 1 -- 172.21.15.42:0/364780166 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f46440a1330 msgr2=0x7f46440a1710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:12.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.849+0000 7f464b27d640 1 --2- 172.21.15.42:0/364780166 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f46440a1330 0x7f46440a1710 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f46400097f0 tx=0x7f464001c4c0 comp rx=0 tx=0).stop 2024-09-16T03:51:12.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.849+0000 7f464b27d640 1 -- 172.21.15.42:0/364780166 shutdown_connections 2024-09-16T03:51:12.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.849+0000 7f464b27d640 1 --2- 172.21.15.42:0/364780166 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f46440a1330 0x7f46440a1710 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:12.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.849+0000 7f464b27d640 1 -- 172.21.15.42:0/364780166 >> 172.21.15.42:0/364780166 conn(0x7f46440a2420 msgr2=0x7f46440a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:51:12.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.849+0000 7f464b27d640 1 -- 172.21.15.42:0/364780166 shutdown_connections 2024-09-16T03:51:12.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.849+0000 7f464b27d640 1 -- 172.21.15.42:0/364780166 wait complete. 2024-09-16T03:51:12.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.849+0000 7f464b27d640 10 monclient: build_initial_monmap 2024-09-16T03:51:12.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.849+0000 7f464b27d640 10 monclient: monmap: 2024-09-16T03:51:12.868 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:51:12.868 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:51:12.868 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:51:12.868 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:51:12.868 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:51:12.868 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:12.868 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:51:12.869 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:51:12.869 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:51:12.869 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:12.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.849+0000 7f464b27d640 1 librados: starting msgr at 2024-09-16T03:51:12.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.849+0000 7f464b27d640 1 librados: starting objecter 2024-09-16T03:51:12.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.849+0000 7f464b27d640 1 Processor -- start 2024-09-16T03:51:12.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.849+0000 7f464b27d640 1 -- start start 2024-09-16T03:51:12.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.849+0000 7f464b27d640 1 librados: setting wanted keys 2024-09-16T03:51:12.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.849+0000 7f464b27d640 1 librados: calling monclient init 2024-09-16T03:51:12.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.849+0000 7f464b27d640 10 monclient: init 2024-09-16T03:51:12.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.849+0000 7f464b27d640 10 monclient: _reopen_session rank -1 2024-09-16T03:51:12.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.849+0000 7f464b27d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T03:51:12.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.849+0000 7f464b27d640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f46440a1330 0x7f46441351e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:12.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.849+0000 7f464b27d640 10 monclient(hunting): picked mon.noname-a con 0x7f46440a1330 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:51:12.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.849+0000 7f464b27d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f46440a1c50 0x7f464413cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:12.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.849+0000 7f464b27d640 10 monclient(hunting): picked mon.noname-b con 0x7f46440a1c50 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:51:12.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.849+0000 7f464b27d640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f464413d0d0 0x7f464413d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:12.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.849+0000 7f464b27d640 10 monclient(hunting): picked mon.noname-c con 0x7f464413d0d0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:51:12.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.849+0000 7f464b27d640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:12.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.849+0000 7f464b27d640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f464409ef30 con 0x7f46440a1330 2024-09-16T03:51:12.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.849+0000 7f464b27d640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:12.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.849+0000 7f464b27d640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f464409edb0 con 0x7f46440a1c50 2024-09-16T03:51:12.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.849+0000 7f464b27d640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:12.871 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.849+0000 7f464b27d640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f4644140b90 con 0x7f464413d0d0 2024-09-16T03:51:12.871 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.849+0000 7f464b27d640 10 monclient(hunting): _renew_subs 2024-09-16T03:51:12.871 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.849+0000 7f464b27d640 10 monclient(hunting): authenticate will time out at 1519.822970s 2024-09-16T03:51:12.871 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.849+0000 7f4649019640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f46440a1330 0x7f46441351e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:12.871 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.849+0000 7f4649019640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f46440a1330 0x7f46441351e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:53150/0 (socket says 172.21.15.42:53150) 2024-09-16T03:51:12.871 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.849+0000 7f4649019640 1 -- 172.21.15.42:0/2228913051 learned_addr learned my addr 172.21.15.42:0/2228913051 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:51:12.871 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.849+0000 7f4649019640 10 monclient(hunting): get_auth_request con 0x7f46440a1330 auth_method 0 2024-09-16T03:51:12.871 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.849+0000 7f4649019640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:12.871 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.849+0000 7f4649019640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:12.871 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.849+0000 7f4649019640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:51:12.871 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.849+0000 7f464981a640 1 --2- 172.21.15.42:0/2228913051 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f464413d0d0 0x7f464413d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:12.871 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.849+0000 7f464981a640 10 monclient(hunting): get_auth_request con 0x7f464413d0d0 auth_method 0 2024-09-16T03:51:12.871 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.849+0000 7f464981a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:12.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.849+0000 7f464981a640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:12.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.849+0000 7f464981a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:51:12.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.849+0000 7f4649019640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:51:12.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.849+0000 7f4649019640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:51:12.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.849+0000 7f4649019640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:51:12.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.849+0000 7f464981a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:51:12.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.849+0000 7f464981a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:51:12.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.849+0000 7f464981a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:51:12.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.849+0000 7f4649019640 10 monclient(hunting): handle_auth_done global_id 764931 payload 824 2024-09-16T03:51:12.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.849+0000 7f4649019640 1 -- 172.21.15.42:0/2228913051 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f464413d0d0 msgr2=0x7f464413d550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:12.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.849+0000 7f4649019640 1 --2- 172.21.15.42:0/2228913051 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f464413d0d0 0x7f464413d550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:12.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.849+0000 7f4649019640 1 -- 172.21.15.42:0/2228913051 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f46440a1c50 msgr2=0x7f464413cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:12.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.849+0000 7f4648818640 1 --2- 172.21.15.42:0/2228913051 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f46440a1c50 0x7f464413cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:12.873 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.849+0000 7f4649019640 1 --2- 172.21.15.42:0/2228913051 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f46440a1c50 0x7f464413cb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:12.873 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.849+0000 7f4649019640 10 monclient: _finish_hunting 0 2024-09-16T03:51:12.873 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.849+0000 7f4649019640 1 monclient: found mon.noname-a 2024-09-16T03:51:12.873 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.849+0000 7f4649019640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:51:12.873 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.849+0000 7f4649019640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:51:12.873 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.849+0000 7f4649019640 1 -- 172.21.15.42:0/2228913051 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46441c6130 con 0x7f46440a1330 2024-09-16T03:51:12.873 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.849+0000 7f464981a640 1 --2- 172.21.15.42:0/2228913051 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f464413d0d0 0x7f464413d550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:51:12.873 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.849+0000 7f4649019640 1 --2- 172.21.15.42:0/2228913051 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f46440a1330 0x7f46441351e0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f46400097f0 tx=0x7f464001ad00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:51:12.873 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.849+0000 7f4639ffb640 10 client.?.objecter ms_handle_connect 0x7f46440a1330 2024-09-16T03:51:12.873 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.849+0000 7f4639ffb640 10 client.?.objecter resend_mon_ops 2024-09-16T03:51:12.873 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.849+0000 7f4639ffb640 1 -- 172.21.15.42:0/2228913051 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46400b1050 con 0x7f46440a1330 2024-09-16T03:51:12.873 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.849+0000 7f4639ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:51:12.873 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.849+0000 7f4639ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:51:12.874 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.849+0000 7f4639ffb640 10 monclient: dump: 2024-09-16T03:51:12.874 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:51:12.874 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:12.874 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:12.874 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:12.874 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:51:12.874 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:12.874 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:51:12.874 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:51:12.874 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:51:12.874 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:12.874 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.849+0000 7f4639ffb640 10 monclient: _finish_auth 0 2024-09-16T03:51:12.874 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.849+0000 7f4639ffb640 10 monclient: _check_auth_tickets 2024-09-16T03:51:12.874 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.849+0000 7f4639ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:51:12.875 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.849+0000 7f464b27d640 5 monclient: authenticate success, global_id 764931 2024-09-16T03:51:12.875 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.849+0000 7f4639ffb640 1 -- 172.21.15.42:0/2228913051 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f46400bd070 con 0x7f46440a1330 2024-09-16T03:51:12.875 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.849+0000 7f464b27d640 10 monclient: _renew_subs 2024-09-16T03:51:12.875 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.849+0000 7f464b27d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:51:12.875 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.849+0000 7f464b27d640 1 -- 172.21.15.42:0/2228913051 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4644120920 con 0x7f46440a1330 2024-09-16T03:51:12.875 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.849+0000 7f4639ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T03:51:12.875 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.849+0000 7f4639ffb640 1 -- 172.21.15.42:0/2228913051 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46400b6850 con 0x7f46440a1330 2024-09-16T03:51:12.875 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.849+0000 7f4639ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:51:12.875 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.853+0000 7f4639ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:51:12.875 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.853+0000 7f4639ffb640 10 monclient: dump: 2024-09-16T03:51:12.875 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:51:12.875 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:12.875 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:12.875 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:12.876 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:51:12.876 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:12.876 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:51:12.876 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:51:12.876 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:51:12.876 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:12.876 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.853+0000 7f464b27d640 10 client.764931.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:51:12.876 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.853+0000 7f464b27d640 10 monclient: _renew_subs 2024-09-16T03:51:12.876 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.853+0000 7f463b7fe640 4 set_mon_vals no callback set 2024-09-16T03:51:12.876 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.853+0000 7f464b27d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:51:12.876 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.853+0000 7f464b27d640 1 -- 172.21.15.42:0/2228913051 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4644073a30 con 0x7f46440a1330 2024-09-16T03:51:12.876 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.853+0000 7f464b27d640 1 librados: init done 2024-09-16T03:51:12.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.853+0000 7f464b27d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:51:12.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.853+0000 7f464b27d640 1 --2- 172.21.15.42:0/2228913051 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f464406dac0 0x7f464402d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:12.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.853+0000 7f464b27d640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:12.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.853+0000 7f464b27d640 1 -- 172.21.15.42:0/2228913051 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f4644071070 con 0x7f464406dac0 2024-09-16T03:51:12.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.853+0000 7f4648818640 1 --2- 172.21.15.42:0/2228913051 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f464406dac0 0x7f464402d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:12.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.853+0000 7f4639ffb640 1 -- 172.21.15.42:0/2228913051 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f46400b69f0 con 0x7f46440a1330 2024-09-16T03:51:12.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.853+0000 7f4648818640 10 monclient: get_auth_request con 0x7f464406dac0 auth_method 0 2024-09-16T03:51:12.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.853+0000 7f4648818640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:12.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.853+0000 7f4648818640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:12.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.853+0000 7f4648818640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:51:12.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.853+0000 7f4648818640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:51:12.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.853+0000 7f4648818640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:51:12.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.853+0000 7f4648818640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:51:12.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.853+0000 7f4639ffb640 1 --2- 172.21.15.42:0/2228913051 >> v2:172.21.15.154:6812/254845342 conn(0x7f4628037810 0x7f4628039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:12.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.853+0000 7f4648818640 10 monclient(hunting): handle_auth_done global_id 814148 payload 824 2024-09-16T03:51:12.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.853+0000 7f4648818640 1 -- 172.21.15.42:0/2228913051 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f46440a1710 con 0x7f464406dac0 2024-09-16T03:51:12.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.853+0000 7f4639ffb640 1 -- 172.21.15.42:0/2228913051 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(308..308 src has 1..308) ==== 5812+0+0 (secure 0 0 0) 0x7f46400e3450 con 0x7f46440a1330 2024-09-16T03:51:12.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.853+0000 7f4639ffb640 10 client.764931.objecter ms_dispatch 0x7f4644000ba0 osd_map(308..308 src has 1..308) 2024-09-16T03:51:12.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.853+0000 7f4639ffb640 3 client.764931.objecter handle_osd_map got epochs [308,308] > 0 2024-09-16T03:51:12.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.853+0000 7f4639ffb640 3 client.764931.objecter handle_osd_map decoding full epoch 308 2024-09-16T03:51:12.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.853+0000 7f4639ffb640 20 client.764931.objecter dump_active .. 0 homeless 2024-09-16T03:51:12.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.853+0000 7f4648818640 1 --2- 172.21.15.42:0/2228913051 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f464406dac0 0x7f464402d7d0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f464409cc20 tx=0x7f463400d810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:51:12.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.853+0000 7f4639ffb640 10 client.764931.objecter ms_handle_connect 0x7f464406dac0 2024-09-16T03:51:12.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.853+0000 7f4639ffb640 10 client.764931.objecter resend_mon_ops 2024-09-16T03:51:12.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.853+0000 7f4639ffb640 1 -- 172.21.15.42:0/2228913051 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4634018070 con 0x7f464406dac0 2024-09-16T03:51:12.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.853+0000 7f4639ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:51:12.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.853+0000 7f464981a640 1 --2- 172.21.15.42:0/2228913051 >> v2:172.21.15.154:6812/254845342 conn(0x7f4628037810 0x7f4628039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:12.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.853+0000 7f4639ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:51:12.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.853+0000 7f4639ffb640 10 monclient: dump: 2024-09-16T03:51:12.879 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:51:12.879 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:12.879 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:12.879 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:12.879 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:51:12.879 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:12.879 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:51:12.880 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:51:12.880 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:51:12.880 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:12.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.853+0000 7f464981a640 10 monclient: get_auth_request con 0x7f4628037810 auth_method 0 2024-09-16T03:51:12.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.853+0000 7f4639ffb640 1 -- 172.21.15.42:0/2228913051 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f4634003620 con 0x7f464406dac0 2024-09-16T03:51:12.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.853+0000 7f4639ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:51:12.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.853+0000 7f4639ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:51:12.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.853+0000 7f4639ffb640 1 -- 172.21.15.42:0/2228913051 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f464406dac0 msgr2=0x7f464402d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:12.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.853+0000 7f4639ffb640 1 --2- 172.21.15.42:0/2228913051 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f464406dac0 0x7f464402d7d0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f464409cc20 tx=0x7f463400d810 comp rx=0 tx=0).stop 2024-09-16T03:51:12.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.853+0000 7f4639ffb640 1 -- 172.21.15.42:0/2228913051 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f464406dac0 msgr2=0x7f464402d7d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:51:12.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.853+0000 7f4639ffb640 1 --2- 172.21.15.42:0/2228913051 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f464406dac0 0x7f464402d7d0 secure :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f464409cc20 tx=0x7f463400d810 comp rx=0 tx=0).stop 2024-09-16T03:51:12.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.853+0000 7f4639ffb640 10 client.764931.objecter ms_handle_connect 0x7f4628037810 2024-09-16T03:51:12.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.853+0000 7f464981a640 1 --2- 172.21.15.42:0/2228913051 >> v2:172.21.15.154:6812/254845342 conn(0x7f4628037810 0x7f4628039cd0 secure :-1 s=READY pgs=3557 cs=0 l=1 rev1=1 crypto rx=0x7f463c0012f0 tx=0x7f463c0016c0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:51:12.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.857+0000 7f464b27d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:51:12.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.857+0000 7f464b27d640 1 --2- 172.21.15.42:0/2228913051 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f464413da90 0x7f4644062ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:12.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.857+0000 7f464b27d640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:12.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.857+0000 7f464b27d640 1 -- 172.21.15.42:0/2228913051 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f4644139a40 con 0x7f464413da90 2024-09-16T03:51:12.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.857+0000 7f4648818640 1 --2- 172.21.15.42:0/2228913051 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f464413da90 0x7f4644062ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:12.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.857+0000 7f4648818640 10 monclient: get_auth_request con 0x7f464413da90 auth_method 0 2024-09-16T03:51:12.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.857+0000 7f4648818640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:12.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.857+0000 7f4648818640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:12.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.857+0000 7f4648818640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:51:12.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.861+0000 7f4648818640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:51:12.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.861+0000 7f4648818640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:51:12.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.861+0000 7f4648818640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:51:12.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.861+0000 7f4648818640 10 monclient(hunting): handle_auth_done global_id 814151 payload 824 2024-09-16T03:51:12.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.861+0000 7f4648818640 1 -- 172.21.15.42:0/2228913051 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f464409d900 con 0x7f464413da90 2024-09-16T03:51:12.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.861+0000 7f4639ffb640 10 client.764931.objecter ms_handle_connect 0x7f464413da90 2024-09-16T03:51:12.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.861+0000 7f4639ffb640 10 client.764931.objecter resend_mon_ops 2024-09-16T03:51:12.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.861+0000 7f4648818640 1 --2- 172.21.15.42:0/2228913051 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f464413da90 0x7f4644062ea0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f4634009010 tx=0x7f4634000ee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:51:12.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.861+0000 7f4639ffb640 1 -- 172.21.15.42:0/2228913051 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4634018040 con 0x7f464413da90 2024-09-16T03:51:12.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.861+0000 7f4639ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:51:12.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.861+0000 7f4639ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:51:12.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.861+0000 7f4639ffb640 10 monclient: dump: 2024-09-16T03:51:12.882 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:51:12.882 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:12.882 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:12.882 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:12.882 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:51:12.882 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:12.883 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:51:12.883 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:51:12.883 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:51:12.883 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:12.883 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.861+0000 7f4639ffb640 1 -- 172.21.15.42:0/2228913051 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4544 (secure 0 0 0) 0x7f4634023450 con 0x7f464413da90 2024-09-16T03:51:12.883 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.861+0000 7f4639ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:51:12.883 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.861+0000 7f4639ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:51:12.883 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.861+0000 7f4639ffb640 1 -- 172.21.15.42:0/2228913051 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f464413da90 msgr2=0x7f4644062ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:12.883 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.861+0000 7f4639ffb640 1 --2- 172.21.15.42:0/2228913051 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f464413da90 0x7f4644062ea0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f4634009010 tx=0x7f4634000ee0 comp rx=0 tx=0).stop 2024-09-16T03:51:12.883 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.861+0000 7f4639ffb640 1 -- 172.21.15.42:0/2228913051 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f464413da90 msgr2=0x7f4644062ea0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:51:12.883 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.861+0000 7f4639ffb640 1 --2- 172.21.15.42:0/2228913051 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f464413da90 0x7f4644062ea0 secure :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f4634009010 tx=0x7f4634000ee0 comp rx=0 tx=0).stop 2024-09-16T03:51:12.884 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:51:12.884 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:51:12.884 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:51:12.884 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T03:51:12.884 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 80, 2024-09-16T03:51:12.884 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 17935, 2024-09-16T03:51:12.884 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:51:12.884 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:51:12.884 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T03:51:12.884 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T03:51:12.884 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:51:12.885 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 2, 2024-09-16T03:51:12.885 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:51:12.885 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:51:12.885 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:51:12.885 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:51:12.885 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:51:12.885 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:51:12.885 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:51:12.885 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:51:12.885 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:51:12.885 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:51:12.885 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:51:12.885 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:51:12.885 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:51:12.886 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:51:12.886 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:51:12.886 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:51:12.886 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:51:12.886 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:51:12.886 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:51:12.886 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:51:12.886 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:51:12.886 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:51:12.886 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:51:12.886 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:51:12.886 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:51:12.886 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:51:12.886 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:51:12.887 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:51:12.887 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:51:12.887 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:12.887 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:51:12.887 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:51:12.887 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:51:12.887 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:51:12.887 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:51:12.887 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:51:12.887 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:51:12.887 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:51:12.887 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:51:12.887 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:51:12.888 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:51:12.888 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:51:12.888 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:51:12.888 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:51:12.888 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:51:12.888 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:51:12.888 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:51:12.888 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:51:12.888 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:51:12.888 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:51:12.888 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:51:12.888 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:51:12.888 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:51:12.888 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:51:12.889 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:51:12.889 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:51:12.889 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:51:12.889 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:51:12.889 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:51:12.889 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:51:12.889 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:12.889 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:51:12.889 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:12.889 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:51:12.889 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:51:12.889 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:51:12.889 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:51:12.890 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:12.890 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:51:12.890 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:51:12.890 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:51:12.890 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:12.890 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:12.890 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:51:12.890 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:51:12.890 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:51:12.890 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:51:12.890 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:51:12.890 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:12.890 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:51:12.890 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:51:12.891 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:51:12.891 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:51:12.891 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:51:12.891 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:12.891 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:12.891 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:51:12.891 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:51:12.891 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:51:12.891 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:51:12.891 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:12.891 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:51:12.891 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:51:12.891 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:51:12.892 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:12.892 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:12.892 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:51:12.892 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:51:12.892 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:51:12.892 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:51:12.892 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:51:12.892 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:12.892 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:51:12.892 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:51:12.892 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:51:12.892 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:51:12.892 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:51:12.892 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:12.893 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:12.893 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:51:12.893 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:51:12.893 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:51:12.893 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:51:12.893 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:12.893 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:51:12.893 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:51:12.893 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:51:12.893 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:12.893 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:12.893 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:51:12.893 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:51:12.893 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:51:12.894 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:51:12.894 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:51:12.894 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:12.894 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:51:12.894 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:51:12.894 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:51:12.894 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:51:12.894 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:51:12.894 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:51:12.894 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:51:12.894 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:12.894 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:51:12.894 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:51:12.895 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:12.895 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:51:12.895 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:51:12.895 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:51:12.895 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:51:12.895 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:51:12.895 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:51:12.895 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:12.895 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:51:12.895 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:51:12.895 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:51:12.895 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:51:12.896 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:51:12.896 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:12.896 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:51:12.896 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:51:12.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.861+0000 7f464b27d640 10 librados: watch_flush enter 2024-09-16T03:51:12.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.861+0000 7f464b27d640 10 librados: watch_flush exit 2024-09-16T03:51:12.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.861+0000 7f464b27d640 20 client.764931.objecter shutdown clearing up homeless session... 2024-09-16T03:51:12.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.861+0000 7f464b27d640 10 client.764931.objecter successfully canceled tick 2024-09-16T03:51:12.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.861+0000 7f464b27d640 1 -- 172.21.15.42:0/2228913051 >> v2:172.21.15.154:6812/254845342 conn(0x7f4628037810 msgr2=0x7f4628039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:12.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.861+0000 7f464b27d640 1 --2- 172.21.15.42:0/2228913051 >> v2:172.21.15.154:6812/254845342 conn(0x7f4628037810 0x7f4628039cd0 secure :-1 s=READY pgs=3557 cs=0 l=1 rev1=1 crypto rx=0x7f463c0012f0 tx=0x7f463c0016c0 comp rx=0 tx=0).stop 2024-09-16T03:51:12.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.861+0000 7f464b27d640 10 monclient: shutdown 2024-09-16T03:51:12.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.861+0000 7f464b27d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:51:12.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.861+0000 7f464b27d640 1 -- 172.21.15.42:0/2228913051 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f46440a1330 msgr2=0x7f46441351e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:12.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.861+0000 7f464b27d640 1 --2- 172.21.15.42:0/2228913051 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f46440a1330 0x7f46441351e0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f46400097f0 tx=0x7f464001ad00 comp rx=0 tx=0).stop 2024-09-16T03:51:12.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.861+0000 7f464981a640 1 -- 172.21.15.42:0/2228913051 reap_dead start 2024-09-16T03:51:12.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.861+0000 7f464981a640 1 -- 172.21.15.42:0/2228913051 reap_dead start 2024-09-16T03:51:12.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.861+0000 7f464b27d640 1 -- 172.21.15.42:0/2228913051 shutdown_connections 2024-09-16T03:51:12.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.861+0000 7f464b27d640 1 -- 172.21.15.42:0/2228913051 >> 172.21.15.42:0/2228913051 conn(0x7f46440a2420 msgr2=0x7f46440715c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:51:12.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.861+0000 7f464b27d640 1 -- 172.21.15.42:0/2228913051 shutdown_connections 2024-09-16T03:51:12.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.861+0000 7f464b27d640 1 -- 172.21.15.42:0/2228913051 wait complete. 2024-09-16T03:51:12.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:12.861+0000 7f464b27d640 1 librados: shutdown 2024-09-16T03:51:12.899 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:51:12.899 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T03:51:12.900 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T03:51:12.986 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp:llPP2 2024-09-16T03:51:12.986 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripWriteFullPP2 (3034 ms) 2024-09-16T03:51:12.986 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleStatPP 2024-09-16T03:51:12.986 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleStatPP (6970 ms) 2024-09-16T03:51:12.986 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleStatPPNS 2024-09-16T03:51:12.986 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleStatPPNS (6981 ms) 2024-09-16T03:51:12.986 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.StatRemovePP 2024-09-16T03:51:12.986 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio_pp: [ OK ] LibRadosAioEC.StatRemovePP (27912 ms) 2024-09-16T03:51:12.986 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.ExecuteClassPP 2024-09-16T03:51:12.986 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio_pp: [ OK ] LibRadosAioEC.ExecuteClassPP (6952 ms) 2024-09-16T03:51:12.987 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.OmapPP 2024-09-16T03:51:12.987 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio_pp: [ OK ] LibRadosAioEC.OmapPP (6982 ms) 2024-09-16T03:51:12.987 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.MultiWritePP 2024-09-16T03:51:12.987 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio_pp: [ OK ] LibRadosAioEC.MultiWritePP (18823 ms) 2024-09-16T03:51:12.987 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio_pp: [----------] 20 tests from LibRadosAioEC (213730 ms total) 2024-09-16T03:51:12.987 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio_pp: 2024-09-16T03:51:12.987 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio_pp: [----------] Global test environment tear-down 2024-09-16T03:51:12.987 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio_pp: [==========] 57 tests from 4 test suites ran. (463430 ms total) 2024-09-16T03:51:12.987 INFO:tasks.workunit.client.0.smithi154.stdout: api_aio_pp: [ PASSED ] 57 tests. 2024-09-16T03:51:13.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.005+0000 7fcb2a99d640 10 monclient: get_monmap_and_config 2024-09-16T03:51:13.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.005+0000 7fcb2a99d640 10 monclient: build_initial_monmap 2024-09-16T03:51:13.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.005+0000 7fcb2a99d640 10 monclient: monmap: 2024-09-16T03:51:13.012 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:51:13.012 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:13.012 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:51:13.011174+0000 2024-09-16T03:51:13.013 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:51:13.011174+0000 2024-09-16T03:51:13.013 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:51:13.013 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:13.013 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:51:13.013 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:51:13.013 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:51:13.013 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:51:13.013 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:51:13.013 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:13.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.005+0000 7fcb2a99d640 1 Processor -- start 2024-09-16T03:51:13.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.005+0000 7fcb2a99d640 1 -- start start 2024-09-16T03:51:13.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.005+0000 7fcb2a99d640 10 monclient: init 2024-09-16T03:51:13.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.009+0000 7fcb2a99d640 10 monclient: _reopen_session rank -1 2024-09-16T03:51:13.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.009+0000 7fcb2a99d640 10 monclient: _add_conns ranks=[1,4,3,0,2] 2024-09-16T03:51:13.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.009+0000 7fcb2a99d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fcb240a39b0 addr v1:172.21.15.154:6789/0 2024-09-16T03:51:13.016 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.009+0000 7fcb2a99d640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fcb240a3d90 0x7fcb2413c380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:13.016 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.009+0000 7fcb2a99d640 10 monclient(hunting): picked mon.noname-c con 0x7fcb240a3d90 addr v2:172.21.15.154:3300/0 2024-09-16T03:51:13.016 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.009+0000 7fcb2a99d640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fcb2413c8c0 0x7fcb2413ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:13.016 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.009+0000 7fcb2a99d640 10 monclient(hunting): picked mon.noname-a con 0x7fcb2413c8c0 addr v2:172.21.15.42:3300/0 2024-09-16T03:51:13.016 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.009+0000 7fcb2a99d640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcb24134290 con 0x7fcb240a39b0 2024-09-16T03:51:13.016 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.009+0000 7fcb2a99d640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:13.016 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.009+0000 7fcb2a99d640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7fcb2409c140 con 0x7fcb2413c8c0 2024-09-16T03:51:13.016 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.009+0000 7fcb2a99d640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:13.016 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.009+0000 7fcb2a99d640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fcb2409c420 con 0x7fcb240a3d90 2024-09-16T03:51:13.016 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.009+0000 7fcb2a99d640 10 monclient(hunting): _renew_subs 2024-09-16T03:51:13.016 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.009+0000 7fcb2a99d640 10 monclient(hunting): authenticate will time out at 1519.980580s 2024-09-16T03:51:13.016 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.009+0000 7fcb28f3a640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fcb2413c8c0 0x7fcb2413ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:13.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.009+0000 7fcb28f3a640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fcb2413c8c0 0x7fcb2413ecb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:53160/0 (socket says 172.21.15.42:53160) 2024-09-16T03:51:13.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.009+0000 7fcb28f3a640 1 -- 172.21.15.42:0/4200734247 learned_addr learned my addr 172.21.15.42:0/4200734247 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:51:13.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.009+0000 7fcb28f3a640 10 monclient(hunting): get_auth_request con 0x7fcb2413c8c0 auth_method 0 2024-09-16T03:51:13.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.009+0000 7fcb28f3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:13.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.009+0000 7fcb28f3a640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:13.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.009+0000 7fcb28f3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:51:13.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.009+0000 7fcb28f3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:51:13.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.009+0000 7fcb28f3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:51:13.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.009+0000 7fcb28f3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:51:13.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.009+0000 7fcb28f3a640 10 monclient(hunting): handle_auth_done global_id 764934 payload 274 2024-09-16T03:51:13.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.009+0000 7fcb28f3a640 1 -- 172.21.15.42:0/4200734247 >> v2:172.21.15.154:3300/0 conn(0x7fcb240a3d90 msgr2=0x7fcb2413c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:51:13.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.009+0000 7fcb28f3a640 1 --2- 172.21.15.42:0/4200734247 >> v2:172.21.15.154:3300/0 conn(0x7fcb240a3d90 0x7fcb2413c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:13.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.009+0000 7fcb28f3a640 1 -- 172.21.15.42:0/4200734247 >> v1:172.21.15.154:6789/0 conn(0x7fcb240a39b0 legacy=0x7fcb24133b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:51:13.018 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.009+0000 7fcb28f3a640 10 monclient: _finish_hunting 0 2024-09-16T03:51:13.018 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.009+0000 7fcb28f3a640 1 monclient: found mon.noname-a 2024-09-16T03:51:13.018 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.009+0000 7fcb28f3a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:51:13.018 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.009+0000 7fcb28f3a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:51:13.018 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.009+0000 7fcb28f3a640 1 -- 172.21.15.42:0/4200734247 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb2406dd60 con 0x7fcb2413c8c0 2024-09-16T03:51:13.025 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.009+0000 7fcb28f3a640 1 --2- 172.21.15.42:0/4200734247 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcb2413c8c0 0x7fcb2413ecb0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fcb240a56c0 tx=0x7fcb1c01c870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6d043b00ad9fd4c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:51:13.025 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.009+0000 7fcb22ffd640 1 -- 172.21.15.42:0/4200734247 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb1c0a5530 con 0x7fcb2413c8c0 2024-09-16T03:51:13.025 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.009+0000 7fcb22ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:51:13.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.009+0000 7fcb22ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:51:13.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.009+0000 7fcb22ffd640 10 monclient: dump: 2024-09-16T03:51:13.026 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:51:13.026 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:13.026 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:13.026 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:13.026 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:51:13.026 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:13.026 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:51:13.026 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:51:13.026 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:51:13.026 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:13.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.009+0000 7fcb22ffd640 10 monclient: _finish_auth 0 2024-09-16T03:51:13.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.009+0000 7fcb22ffd640 10 monclient: _check_auth_tickets 2024-09-16T03:51:13.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.009+0000 7fcb22ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:51:13.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.009+0000 7fcb22ffd640 1 -- 172.21.15.42:0/4200734247 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcb1c0a56d0 con 0x7fcb2413c8c0 2024-09-16T03:51:13.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.009+0000 7fcb22ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:51:13.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.009+0000 7fcb2a99d640 5 monclient: authenticate success, global_id 764934 2024-09-16T03:51:13.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.009+0000 7fcb22ffd640 1 -- 172.21.15.42:0/4200734247 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb1c0a5aa0 con 0x7fcb2413c8c0 2024-09-16T03:51:13.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.009+0000 7fcb2a99d640 10 monclient: get_monmap_and_config success 2024-09-16T03:51:13.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.009+0000 7fcb2a99d640 4 set_mon_vals no callback set 2024-09-16T03:51:13.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.009+0000 7fcb22ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:51:13.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.009+0000 7fcb22ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:51:13.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.009+0000 7fcb22ffd640 10 monclient: dump: 2024-09-16T03:51:13.027 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:51:13.027 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:13.027 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:13.027 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:13.028 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:51:13.028 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:13.028 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:51:13.028 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:51:13.028 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:51:13.028 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:13.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.009+0000 7fcb2a99d640 10 monclient: shutdown 2024-09-16T03:51:13.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.009+0000 7fcb2a99d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:51:13.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.009+0000 7fcb2a99d640 1 -- 172.21.15.42:0/4200734247 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcb2413c8c0 msgr2=0x7fcb2413ecb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:13.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.009+0000 7fcb2a99d640 1 --2- 172.21.15.42:0/4200734247 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcb2413c8c0 0x7fcb2413ecb0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fcb240a56c0 tx=0x7fcb1c01c870 comp rx=0 tx=0).stop 2024-09-16T03:51:13.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.009+0000 7fcb2a99d640 1 -- 172.21.15.42:0/4200734247 shutdown_connections 2024-09-16T03:51:13.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.009+0000 7fcb2a99d640 1 --2- 172.21.15.42:0/4200734247 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcb2413c8c0 0x7fcb2413ecb0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:13.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.009+0000 7fcb2a99d640 1 --2- 172.21.15.42:0/4200734247 >> v2:172.21.15.154:3300/0 conn(0x7fcb240a3d90 0x7fcb2413c380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:13.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.009+0000 7fcb2a99d640 1 -- 172.21.15.42:0/4200734247 >> 172.21.15.42:0/4200734247 conn(0x7fcb240a9fb0 msgr2=0x7fcb240aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:51:13.029 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.009+0000 7fcb2a99d640 1 -- 172.21.15.42:0/4200734247 shutdown_connections 2024-09-16T03:51:13.029 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.009+0000 7fcb2a99d640 1 -- 172.21.15.42:0/4200734247 wait complete. 2024-09-16T03:51:13.029 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.013+0000 7fcb2a99d640 10 monclient: build_initial_monmap 2024-09-16T03:51:13.029 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.013+0000 7fcb2a99d640 10 monclient: monmap: 2024-09-16T03:51:13.029 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:51:13.029 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:51:13.029 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:51:13.029 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:51:13.029 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:51:13.029 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:13.029 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:51:13.029 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:51:13.029 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:51:13.029 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:13.030 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.013+0000 7fcb2a99d640 1 librados: starting msgr at 2024-09-16T03:51:13.030 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.013+0000 7fcb2a99d640 1 librados: starting objecter 2024-09-16T03:51:13.030 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.013+0000 7fcb2a99d640 1 Processor -- start 2024-09-16T03:51:13.030 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.013+0000 7fcb2a99d640 1 -- start start 2024-09-16T03:51:13.030 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.013+0000 7fcb2a99d640 1 librados: setting wanted keys 2024-09-16T03:51:13.030 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.013+0000 7fcb2a99d640 1 librados: calling monclient init 2024-09-16T03:51:13.030 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.013+0000 7fcb2a99d640 10 monclient: init 2024-09-16T03:51:13.030 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.013+0000 7fcb2a99d640 10 monclient: _reopen_session rank -1 2024-09-16T03:51:13.030 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.013+0000 7fcb2a99d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:51:13.030 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.013+0000 7fcb2a99d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcb240a39b0 0x7fcb24138130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:13.030 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.013+0000 7fcb2a99d640 10 monclient(hunting): picked mon.noname-b con 0x7fcb240a39b0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:51:13.030 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.013+0000 7fcb2a99d640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcb240a3d90 0x7fcb24138670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:13.030 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.013+0000 7fcb2a99d640 10 monclient(hunting): picked mon.noname-a con 0x7fcb240a3d90 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:51:13.030 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.013+0000 7fcb2a99d640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fcb2413c8c0 0x7fcb24138bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:13.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.013+0000 7fcb2a99d640 10 monclient(hunting): picked mon.noname-c con 0x7fcb2413c8c0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:51:13.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.013+0000 7fcb2a99d640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:13.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.013+0000 7fcb2a99d640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fcb24141da0 con 0x7fcb240a3d90 2024-09-16T03:51:13.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.013+0000 7fcb2a99d640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:13.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.013+0000 7fcb2a99d640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fcb24141c20 con 0x7fcb240a39b0 2024-09-16T03:51:13.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.013+0000 7fcb2a99d640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:13.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.013+0000 7fcb2a99d640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fcb24141f20 con 0x7fcb2413c8c0 2024-09-16T03:51:13.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.013+0000 7fcb2a99d640 10 monclient(hunting): _renew_subs 2024-09-16T03:51:13.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.013+0000 7fcb2a99d640 10 monclient(hunting): authenticate will time out at 1519.985290s 2024-09-16T03:51:13.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.013+0000 7fcb237fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcb240a3d90 0x7fcb24138670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:13.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.013+0000 7fcb237fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcb240a3d90 0x7fcb24138670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:53168/0 (socket says 172.21.15.42:53168) 2024-09-16T03:51:13.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.013+0000 7fcb237fe640 1 -- 172.21.15.42:0/918041113 learned_addr learned my addr 172.21.15.42:0/918041113 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:51:13.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.013+0000 7fcb237fe640 10 monclient(hunting): get_auth_request con 0x7fcb240a3d90 auth_method 0 2024-09-16T03:51:13.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.013+0000 7fcb237fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:13.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.013+0000 7fcb237fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:13.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.013+0000 7fcb237fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:51:13.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.013+0000 7fcb28f3a640 1 --2- 172.21.15.42:0/918041113 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fcb2413c8c0 0x7fcb24138bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:13.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.013+0000 7fcb28f3a640 10 monclient(hunting): get_auth_request con 0x7fcb2413c8c0 auth_method 0 2024-09-16T03:51:13.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.013+0000 7fcb28f3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:13.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.013+0000 7fcb28f3a640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:13.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.013+0000 7fcb28f3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:51:13.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.013+0000 7fcb237fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:51:13.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.013+0000 7fcb237fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:51:13.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.013+0000 7fcb237fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:51:13.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.013+0000 7fcb28f3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:51:13.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.013+0000 7fcb28f3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:51:13.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.013+0000 7fcb28f3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:51:13.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.013+0000 7fcb23fff640 1 --2- 172.21.15.42:0/918041113 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcb240a39b0 0x7fcb24138130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:13.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.013+0000 7fcb23fff640 10 monclient(hunting): get_auth_request con 0x7fcb240a39b0 auth_method 0 2024-09-16T03:51:13.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.013+0000 7fcb23fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:13.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.013+0000 7fcb23fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:13.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.013+0000 7fcb23fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:51:13.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.013+0000 7fcb23fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:51:13.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.013+0000 7fcb23fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:51:13.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.013+0000 7fcb23fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:51:13.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.013+0000 7fcb237fe640 10 monclient(hunting): handle_auth_done global_id 764937 payload 824 2024-09-16T03:51:13.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.013+0000 7fcb237fe640 1 -- 172.21.15.42:0/918041113 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fcb2413c8c0 msgr2=0x7fcb24138bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:13.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.013+0000 7fcb237fe640 1 --2- 172.21.15.42:0/918041113 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fcb2413c8c0 0x7fcb24138bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:13.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.013+0000 7fcb237fe640 1 -- 172.21.15.42:0/918041113 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcb240a39b0 msgr2=0x7fcb24138130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:13.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.013+0000 7fcb237fe640 1 --2- 172.21.15.42:0/918041113 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcb240a39b0 0x7fcb24138130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:13.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.013+0000 7fcb237fe640 10 monclient: _finish_hunting 0 2024-09-16T03:51:13.034 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.013+0000 7fcb237fe640 1 monclient: found mon.noname-a 2024-09-16T03:51:13.034 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.013+0000 7fcb237fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:51:13.034 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.013+0000 7fcb237fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:51:13.034 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.013+0000 7fcb237fe640 1 -- 172.21.15.42:0/918041113 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb2406f680 con 0x7fcb240a3d90 2024-09-16T03:51:13.034 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.013+0000 7fcb28f3a640 1 --2- 172.21.15.42:0/918041113 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fcb2413c8c0 0x7fcb24138bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:51:13.034 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.013+0000 7fcb237fe640 1 --2- 172.21.15.42:0/918041113 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcb240a3d90 0x7fcb24138670 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fcb1800b550 tx=0x7fcb1800ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:51:13.034 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.013+0000 7fcb20ff9640 10 client.?.objecter ms_handle_connect 0x7fcb240a3d90 2024-09-16T03:51:13.034 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.013+0000 7fcb20ff9640 10 client.?.objecter resend_mon_ops 2024-09-16T03:51:13.034 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.013+0000 7fcb20ff9640 1 -- 172.21.15.42:0/918041113 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb18014070 con 0x7fcb240a3d90 2024-09-16T03:51:13.034 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.013+0000 7fcb20ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:51:13.034 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.013+0000 7fcb20ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:51:13.034 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.013+0000 7fcb20ff9640 10 monclient: dump: 2024-09-16T03:51:13.034 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:51:13.034 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:13.035 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:13.035 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:13.035 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:51:13.035 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:13.035 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:51:13.035 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:51:13.035 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:51:13.035 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:13.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.013+0000 7fcb20ff9640 10 monclient: _finish_auth 0 2024-09-16T03:51:13.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.013+0000 7fcb20ff9640 10 monclient: _check_auth_tickets 2024-09-16T03:51:13.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.013+0000 7fcb20ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:51:13.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.013+0000 7fcb20ff9640 1 -- 172.21.15.42:0/918041113 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcb18010890 con 0x7fcb240a3d90 2024-09-16T03:51:13.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.013+0000 7fcb2a99d640 5 monclient: authenticate success, global_id 764937 2024-09-16T03:51:13.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.013+0000 7fcb2a99d640 10 monclient: _renew_subs 2024-09-16T03:51:13.036 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.013+0000 7fcb2a99d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:51:13.036 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.013+0000 7fcb2a99d640 1 -- 172.21.15.42:0/918041113 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb24139bf0 con 0x7fcb240a3d90 2024-09-16T03:51:13.036 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.013+0000 7fcb20ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T03:51:13.036 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.013+0000 7fcb20ff9640 1 -- 172.21.15.42:0/918041113 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb18010c80 con 0x7fcb240a3d90 2024-09-16T03:51:13.036 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.013+0000 7fcb20ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:51:13.036 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.013+0000 7fcb20ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:51:13.036 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.013+0000 7fcb20ff9640 10 monclient: dump: 2024-09-16T03:51:13.036 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:51:13.036 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:13.036 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:13.036 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:13.036 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:51:13.036 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:13.036 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:51:13.036 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:51:13.037 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:51:13.037 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:13.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.013+0000 7fcb227fc640 4 set_mon_vals no callback set 2024-09-16T03:51:13.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.013+0000 7fcb2a99d640 10 client.764937.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:51:13.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.013+0000 7fcb2a99d640 10 monclient: _renew_subs 2024-09-16T03:51:13.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.013+0000 7fcb2a99d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:51:13.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.013+0000 7fcb2a99d640 1 -- 172.21.15.42:0/918041113 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb2413acf0 con 0x7fcb240a3d90 2024-09-16T03:51:13.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.013+0000 7fcb2a99d640 1 librados: init done 2024-09-16T03:51:13.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.013+0000 7fcb2a99d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:51:13.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.013+0000 7fcb2a99d640 1 --2- 172.21.15.42:0/918041113 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcb2412d2d0 0x7fcb2412d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:13.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.013+0000 7fcb2a99d640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:13.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.013+0000 7fcb2a99d640 1 -- 172.21.15.42:0/918041113 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fcb24133b80 con 0x7fcb2412d2d0 2024-09-16T03:51:13.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.013+0000 7fcb20ff9640 1 -- 172.21.15.42:0/918041113 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fcb18017070 con 0x7fcb240a3d90 2024-09-16T03:51:13.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.013+0000 7fcb23fff640 1 --2- 172.21.15.42:0/918041113 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcb2412d2d0 0x7fcb2412d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:13.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.013+0000 7fcb20ff9640 1 --2- 172.21.15.42:0/918041113 >> v2:172.21.15.154:6812/254845342 conn(0x7fcb10037810 0x7fcb10039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:13.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.013+0000 7fcb23fff640 10 monclient: get_auth_request con 0x7fcb2412d2d0 auth_method 0 2024-09-16T03:51:13.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.013+0000 7fcb23fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:13.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.013+0000 7fcb23fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:13.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.013+0000 7fcb23fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:51:13.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.013+0000 7fcb20ff9640 1 -- 172.21.15.42:0/918041113 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(309..309 src has 1..309) ==== 5731+0+0 (secure 0 0 0) 0x7fcb18048d50 con 0x7fcb240a3d90 2024-09-16T03:51:13.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.013+0000 7fcb20ff9640 10 client.764937.objecter ms_dispatch 0x7fcb24000ba0 osd_map(309..309 src has 1..309) 2024-09-16T03:51:13.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.013+0000 7fcb20ff9640 3 client.764937.objecter handle_osd_map got epochs [309,309] > 0 2024-09-16T03:51:13.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.013+0000 7fcb20ff9640 3 client.764937.objecter handle_osd_map decoding full epoch 309 2024-09-16T03:51:13.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.013+0000 7fcb23fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:51:13.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.013+0000 7fcb23fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:51:13.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.013+0000 7fcb23fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:51:13.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.013+0000 7fcb20ff9640 20 client.764937.objecter dump_active .. 0 homeless 2024-09-16T03:51:13.039 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.013+0000 7fcb23fff640 10 monclient(hunting): handle_auth_done global_id 794338 payload 824 2024-09-16T03:51:13.039 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.013+0000 7fcb23fff640 1 -- 172.21.15.42:0/918041113 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fcb240a5750 con 0x7fcb2412d2d0 2024-09-16T03:51:13.039 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.013+0000 7fcb28f3a640 1 --2- 172.21.15.42:0/918041113 >> v2:172.21.15.154:6812/254845342 conn(0x7fcb10037810 0x7fcb10039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:13.039 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.013+0000 7fcb28f3a640 10 monclient: get_auth_request con 0x7fcb10037810 auth_method 0 2024-09-16T03:51:13.039 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.013+0000 7fcb23fff640 1 --2- 172.21.15.42:0/918041113 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcb2412d2d0 0x7fcb2412d6b0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fcb1400f560 tx=0x7fcb1400fa20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:51:13.039 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.013+0000 7fcb20ff9640 10 client.764937.objecter ms_handle_connect 0x7fcb2412d2d0 2024-09-16T03:51:13.039 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.013+0000 7fcb20ff9640 10 client.764937.objecter resend_mon_ops 2024-09-16T03:51:13.039 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.017+0000 7fcb20ff9640 1 -- 172.21.15.42:0/918041113 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb140052e0 con 0x7fcb2412d2d0 2024-09-16T03:51:13.039 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.017+0000 7fcb20ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:51:13.039 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.017+0000 7fcb20ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:51:13.039 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.017+0000 7fcb20ff9640 10 monclient: dump: 2024-09-16T03:51:13.039 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:51:13.039 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:13.039 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:13.040 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:13.040 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:51:13.040 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:13.040 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:51:13.040 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:51:13.040 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:51:13.040 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:13.040 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.017+0000 7fcb20ff9640 10 client.764937.objecter ms_handle_connect 0x7fcb10037810 2024-09-16T03:51:13.040 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.017+0000 7fcb28f3a640 1 --2- 172.21.15.42:0/918041113 >> v2:172.21.15.154:6812/254845342 conn(0x7fcb10037810 0x7fcb10039cd0 secure :-1 s=READY pgs=3558 cs=0 l=1 rev1=1 crypto rx=0x7fcb1c0ae000 tx=0x7fcb1c0a7070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:51:13.040 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.017+0000 7fcb20ff9640 1 -- 172.21.15.42:0/918041113 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fcb14018070 con 0x7fcb2412d2d0 2024-09-16T03:51:13.040 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.017+0000 7fcb20ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:51:13.040 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.017+0000 7fcb20ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:51:13.040 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.017+0000 7fcb20ff9640 1 -- 172.21.15.42:0/918041113 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcb2412d2d0 msgr2=0x7fcb2412d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:13.040 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.017+0000 7fcb20ff9640 1 --2- 172.21.15.42:0/918041113 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcb2412d2d0 0x7fcb2412d6b0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fcb1400f560 tx=0x7fcb1400fa20 comp rx=0 tx=0).stop 2024-09-16T03:51:13.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.017+0000 7fcb20ff9640 1 -- 172.21.15.42:0/918041113 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcb2412d2d0 msgr2=0x7fcb2412d6b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:51:13.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.017+0000 7fcb20ff9640 1 --2- 172.21.15.42:0/918041113 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcb2412d2d0 0x7fcb2412d6b0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:13.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.021+0000 7fcb2a99d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:51:13.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.021+0000 7fcb2a99d640 1 --2- 172.21.15.42:0/918041113 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcb2412dbf0 0x7fcb24135bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:13.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.021+0000 7fcb2a99d640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:13.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.021+0000 7fcb2a99d640 1 -- 172.21.15.42:0/918041113 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fcb24139840 con 0x7fcb2412dbf0 2024-09-16T03:51:13.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.021+0000 7fcb23fff640 1 --2- 172.21.15.42:0/918041113 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcb2412dbf0 0x7fcb24135bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:13.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.021+0000 7fcb23fff640 10 monclient: get_auth_request con 0x7fcb2412dbf0 auth_method 0 2024-09-16T03:51:13.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.021+0000 7fcb23fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:13.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.021+0000 7fcb23fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:13.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.021+0000 7fcb23fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:51:13.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.021+0000 7fcb23fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:51:13.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.021+0000 7fcb23fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:51:13.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.021+0000 7fcb23fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:51:13.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.025+0000 7fcb23fff640 10 monclient(hunting): handle_auth_done global_id 794341 payload 824 2024-09-16T03:51:13.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.025+0000 7fcb23fff640 1 -- 172.21.15.42:0/918041113 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fcb24068e40 con 0x7fcb2412dbf0 2024-09-16T03:51:13.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.025+0000 7fcb23fff640 1 --2- 172.21.15.42:0/918041113 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcb2412dbf0 0x7fcb24135bc0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fcb1400f560 tx=0x7fcb14001030 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:51:13.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.025+0000 7fcb20ff9640 10 client.764937.objecter ms_handle_connect 0x7fcb2412dbf0 2024-09-16T03:51:13.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.025+0000 7fcb20ff9640 10 client.764937.objecter resend_mon_ops 2024-09-16T03:51:13.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.025+0000 7fcb20ff9640 1 -- 172.21.15.42:0/918041113 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb14003cf0 con 0x7fcb2412dbf0 2024-09-16T03:51:13.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.025+0000 7fcb20ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:51:13.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.025+0000 7fcb20ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:51:13.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.025+0000 7fcb20ff9640 10 monclient: dump: 2024-09-16T03:51:13.044 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:51:13.044 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:13.044 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:13.044 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:13.044 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:51:13.044 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:13.044 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:51:13.044 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:51:13.044 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:51:13.044 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:13.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.025+0000 7fcb20ff9640 1 -- 172.21.15.42:0/918041113 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4545 (secure 0 0 0) 0x7fcb14003cf0 con 0x7fcb2412dbf0 2024-09-16T03:51:13.045 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.025+0000 7fcb20ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:51:13.045 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.025+0000 7fcb20ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:51:13.045 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.025+0000 7fcb20ff9640 1 -- 172.21.15.42:0/918041113 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcb2412dbf0 msgr2=0x7fcb24135bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:13.045 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.025+0000 7fcb20ff9640 1 --2- 172.21.15.42:0/918041113 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcb2412dbf0 0x7fcb24135bc0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fcb1400f560 tx=0x7fcb14001030 comp rx=0 tx=0).stop 2024-09-16T03:51:13.045 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.025+0000 7fcb20ff9640 1 -- 172.21.15.42:0/918041113 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcb2412dbf0 msgr2=0x7fcb24135bc0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:51:13.045 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.025+0000 7fcb20ff9640 1 --2- 172.21.15.42:0/918041113 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcb2412dbf0 0x7fcb24135bc0 secure :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fcb1400f560 tx=0x7fcb14001030 comp rx=0 tx=0).stop 2024-09-16T03:51:13.045 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:51:13.045 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:51:13.045 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:51:13.045 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T03:51:13.046 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 80, 2024-09-16T03:51:13.046 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 114866, 2024-09-16T03:51:13.046 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:51:13.046 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:51:13.046 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T03:51:13.046 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T03:51:13.046 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:51:13.046 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 3, 2024-09-16T03:51:13.046 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:51:13.046 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:51:13.046 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:51:13.046 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:51:13.046 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:51:13.046 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:51:13.047 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:51:13.047 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:51:13.047 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:51:13.047 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:51:13.047 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:51:13.047 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:51:13.047 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:51:13.047 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:51:13.047 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:51:13.047 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:51:13.047 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:51:13.047 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:51:13.047 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:51:13.047 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:51:13.048 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:51:13.048 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:51:13.048 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:51:13.048 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:51:13.048 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:51:13.048 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:51:13.048 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:51:13.048 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:51:13.048 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:51:13.048 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:13.048 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:51:13.048 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:51:13.048 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:51:13.049 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:51:13.049 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:51:13.049 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:51:13.049 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:51:13.049 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:51:13.049 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:51:13.049 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:51:13.049 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:51:13.049 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:51:13.049 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:51:13.049 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:51:13.049 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:51:13.049 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:51:13.049 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:51:13.050 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:51:13.050 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:51:13.050 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:51:13.050 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:51:13.050 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:51:13.050 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:51:13.050 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:51:13.050 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:51:13.050 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:51:13.050 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:51:13.050 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:51:13.050 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:51:13.050 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:51:13.050 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:13.050 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:51:13.051 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:13.051 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:51:13.051 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:51:13.051 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:51:13.051 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:51:13.051 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:13.051 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:51:13.051 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:51:13.051 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:51:13.051 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:13.051 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:13.051 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:51:13.051 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:51:13.051 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:51:13.052 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:51:13.052 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:51:13.052 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:13.052 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:51:13.052 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:51:13.052 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:51:13.052 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:51:13.052 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:51:13.052 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:13.052 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:13.052 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:51:13.052 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:51:13.052 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:51:13.052 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:51:13.053 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:13.053 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:51:13.053 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:51:13.053 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:51:13.053 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:13.053 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:13.053 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:51:13.053 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:51:13.053 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:51:13.053 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:51:13.053 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:51:13.053 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:13.053 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:51:13.053 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:51:13.053 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:51:13.054 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:51:13.054 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:51:13.054 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:13.054 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:13.054 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:51:13.054 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:51:13.054 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:51:13.054 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:51:13.054 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:13.054 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:51:13.054 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:51:13.054 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:51:13.054 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:13.054 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:13.055 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:51:13.055 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:51:13.055 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:51:13.055 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:51:13.055 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:51:13.055 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:13.055 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:51:13.055 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:51:13.055 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:51:13.055 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:51:13.055 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:51:13.055 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:51:13.055 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:51:13.055 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:13.055 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:51:13.056 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:51:13.056 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:13.056 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:51:13.056 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:51:13.056 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:51:13.056 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:51:13.056 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:51:13.056 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:51:13.056 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:13.056 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:51:13.056 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:51:13.056 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 2 2024-09-16T03:51:13.056 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:51:13.056 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:51:13.057 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:13.057 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:51:13.057 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:51:13.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.025+0000 7fcb2a99d640 10 librados: watch_flush enter 2024-09-16T03:51:13.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.025+0000 7fcb2a99d640 10 librados: watch_flush exit 2024-09-16T03:51:13.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.025+0000 7fcb2a99d640 20 client.764937.objecter shutdown clearing up homeless session... 2024-09-16T03:51:13.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.025+0000 7fcb2a99d640 10 client.764937.objecter successfully canceled tick 2024-09-16T03:51:13.059 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.025+0000 7fcb2a99d640 1 -- 172.21.15.42:0/918041113 >> v2:172.21.15.154:6812/254845342 conn(0x7fcb10037810 msgr2=0x7fcb10039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:13.059 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.025+0000 7fcb2a99d640 1 --2- 172.21.15.42:0/918041113 >> v2:172.21.15.154:6812/254845342 conn(0x7fcb10037810 0x7fcb10039cd0 secure :-1 s=READY pgs=3558 cs=0 l=1 rev1=1 crypto rx=0x7fcb1c0ae000 tx=0x7fcb1c0a7070 comp rx=0 tx=0).stop 2024-09-16T03:51:13.059 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.025+0000 7fcb2a99d640 10 monclient: shutdown 2024-09-16T03:51:13.059 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.025+0000 7fcb2a99d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:51:13.059 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.025+0000 7fcb2a99d640 1 -- 172.21.15.42:0/918041113 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcb240a3d90 msgr2=0x7fcb24138670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:13.059 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.025+0000 7fcb2a99d640 1 --2- 172.21.15.42:0/918041113 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcb240a3d90 0x7fcb24138670 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fcb1800b550 tx=0x7fcb1800ba10 comp rx=0 tx=0).stop 2024-09-16T03:51:13.059 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.025+0000 7fcb28f3a640 1 -- 172.21.15.42:0/918041113 reap_dead start 2024-09-16T03:51:13.059 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.025+0000 7fcb28f3a640 1 -- 172.21.15.42:0/918041113 reap_dead start 2024-09-16T03:51:13.059 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.025+0000 7fcb2a99d640 1 -- 172.21.15.42:0/918041113 shutdown_connections 2024-09-16T03:51:13.059 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.025+0000 7fcb2a99d640 1 -- 172.21.15.42:0/918041113 >> 172.21.15.42:0/918041113 conn(0x7fcb240a9fb0 msgr2=0x7fcb24131980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:51:13.059 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.025+0000 7fcb2a99d640 1 -- 172.21.15.42:0/918041113 shutdown_connections 2024-09-16T03:51:13.059 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.025+0000 7fcb2a99d640 1 -- 172.21.15.42:0/918041113 wait complete. 2024-09-16T03:51:13.059 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.025+0000 7fcb2a99d640 1 librados: shutdown 2024-09-16T03:51:13.060 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:51:13.060 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-09-16T03:51:13.060 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-16T03:51:13.060 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-09-16T03:51:13.060 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-16T03:51:13.060 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-16T03:51:13.060 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T03:51:13.080 INFO:tasks.ceph.mon.a:Stopped 2024-09-16T03:51:13.080 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T03:51:13.080 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:51:13.243 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.237+0000 7fb6e735f640 10 monclient: get_monmap_and_config 2024-09-16T03:51:13.243 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.237+0000 7fb6e735f640 10 monclient: build_initial_monmap 2024-09-16T03:51:13.243 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.237+0000 7fb6e735f640 10 monclient: monmap: 2024-09-16T03:51:13.243 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:51:13.243 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:13.243 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:51:13.241936+0000 2024-09-16T03:51:13.243 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:51:13.241936+0000 2024-09-16T03:51:13.243 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:51:13.243 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:13.244 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:51:13.244 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:51:13.244 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:51:13.244 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:51:13.244 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:51:13.244 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:13.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.237+0000 7fb6e735f640 1 Processor -- start 2024-09-16T03:51:13.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.237+0000 7fb6e735f640 1 -- start start 2024-09-16T03:51:13.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.237+0000 7fb6e735f640 10 monclient: init 2024-09-16T03:51:13.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.237+0000 7fb6e735f640 10 monclient: _reopen_session rank -1 2024-09-16T03:51:13.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.237+0000 7fb6e735f640 10 monclient: _add_conns ranks=[4,1,3,0,2] 2024-09-16T03:51:13.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.237+0000 7fb6e735f640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fb6e00a39b0 0x7fb6e0133b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:13.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.237+0000 7fb6e735f640 10 monclient(hunting): picked mon.noname-c con 0x7fb6e00a39b0 addr v2:172.21.15.154:3300/0 2024-09-16T03:51:13.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.237+0000 7fb6e735f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb6e00a3d90 addr v1:172.21.15.154:6789/0 2024-09-16T03:51:13.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.237+0000 7fb6e735f640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb6e0137710 0x7fb6e013f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:13.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.237+0000 7fb6e735f640 10 monclient(hunting): picked mon.noname-a con 0x7fb6e0137710 addr v2:172.21.15.42:3300/0 2024-09-16T03:51:13.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.237+0000 7fb6e735f640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb6e009c140 con 0x7fb6e00a3d90 2024-09-16T03:51:13.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.237+0000 7fb6e735f640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:13.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.237+0000 7fb6e735f640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7fb6e009c420 con 0x7fb6e0137710 2024-09-16T03:51:13.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.237+0000 7fb6e735f640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:13.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.237+0000 7fb6e735f640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fb6e006dd60 con 0x7fb6e00a39b0 2024-09-16T03:51:13.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.237+0000 7fb6e735f640 10 monclient(hunting): _renew_subs 2024-09-16T03:51:13.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.237+0000 7fb6e735f640 10 monclient(hunting): authenticate will time out at 1520.211445s 2024-09-16T03:51:13.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.237+0000 7fb6e58fc640 1 -- >> v2:172.21.15.42:3300/0 conn(0x7fb6e0137710 msgr2=0x7fb6e013f3a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3300/0 2024-09-16T03:51:13.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.237+0000 7fb6e58fc640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb6e0137710 0x7fb6e013f3a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:51:13.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.237+0000 7fb6e50fb640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fb6e00a39b0 0x7fb6e0133b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:13.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.237+0000 7fb6e50fb640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fb6e00a39b0 0x7fb6e0133b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.42:39044/0 (socket says 172.21.15.42:39044) 2024-09-16T03:51:13.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.237+0000 7fb6e50fb640 1 -- 172.21.15.42:0/33966113 learned_addr learned my addr 172.21.15.42:0/33966113 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:51:13.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.237+0000 7fb6e50fb640 10 monclient(hunting): get_auth_request con 0x7fb6e00a39b0 auth_method 0 2024-09-16T03:51:13.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.237+0000 7fb6e50fb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:13.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.237+0000 7fb6e50fb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:13.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.237+0000 7fb6e50fb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:51:13.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.241+0000 7fb6e50fb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:51:13.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.241+0000 7fb6e50fb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:51:13.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.241+0000 7fb6e50fb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:51:13.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.241+0000 7fb6e50fb640 10 monclient(hunting): handle_auth_done global_id 794344 payload 274 2024-09-16T03:51:13.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.241+0000 7fb6e50fb640 1 -- 172.21.15.42:0/33966113 >> v2:172.21.15.42:3300/0 conn(0x7fb6e0137710 msgr2=0x7fb6e013f3a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:51:13.249 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.241+0000 7fb6e50fb640 1 --2- 172.21.15.42:0/33966113 >> v2:172.21.15.42:3300/0 conn(0x7fb6e0137710 0x7fb6e013f3a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:13.249 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.241+0000 7fb6e50fb640 1 -- 172.21.15.42:0/33966113 >> v1:172.21.15.154:6789/0 conn(0x7fb6e00a3d90 legacy=0x7fb6e0137000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:13.249 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.241+0000 7fb6e50fb640 10 monclient: _finish_hunting 0 2024-09-16T03:51:13.249 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.241+0000 7fb6e50fb640 1 monclient: found mon.noname-c 2024-09-16T03:51:13.249 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.241+0000 7fb6e50fb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:51:13.249 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.241+0000 7fb6e50fb640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.154:3300/0 2024-09-16T03:51:13.249 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.241+0000 7fb6e50fb640 1 -- 172.21.15.42:0/33966113 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6e00a5e20 con 0x7fb6e00a39b0 2024-09-16T03:51:13.254 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.241+0000 7fb6e50fb640 1 --2- 172.21.15.42:0/33966113 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb6e00a39b0 0x7fb6e0133b80 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fb6e00a7750 tx=0x7fb6dc01c400 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b5b89fc801244f67 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:51:13.255 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.241+0000 7fb6d7fff640 1 -- 172.21.15.42:0/33966113 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6dc0b0070 con 0x7fb6e00a39b0 2024-09-16T03:51:13.255 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.241+0000 7fb6d7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:51:13.255 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.241+0000 7fb6d7fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:51:13.255 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.241+0000 7fb6d7fff640 10 monclient: dump: 2024-09-16T03:51:13.255 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:51:13.255 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:13.255 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:13.255 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:13.255 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:51:13.255 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:13.255 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:51:13.255 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:51:13.255 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:51:13.256 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:13.256 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.241+0000 7fb6d7fff640 10 monclient: _finish_auth 0 2024-09-16T03:51:13.256 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.241+0000 7fb6d7fff640 10 monclient: _check_auth_tickets 2024-09-16T03:51:13.256 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.241+0000 7fb6d7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:51:13.256 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.241+0000 7fb6d7fff640 1 -- 172.21.15.42:0/33966113 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb6dc0ab600 con 0x7fb6e00a39b0 2024-09-16T03:51:13.256 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.241+0000 7fb6d7fff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:51:13.256 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.241+0000 7fb6d7fff640 1 -- 172.21.15.42:0/33966113 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6dc0ab9d0 con 0x7fb6e00a39b0 2024-09-16T03:51:13.256 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.241+0000 7fb6d7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:51:13.256 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.241+0000 7fb6d7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:51:13.256 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.241+0000 7fb6d7fff640 10 monclient: dump: 2024-09-16T03:51:13.256 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:51:13.256 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:13.257 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:13.257 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:13.257 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:51:13.257 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:13.257 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:51:13.257 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:51:13.257 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:51:13.257 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:13.257 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.241+0000 7fb6e735f640 5 monclient: authenticate success, global_id 794344 2024-09-16T03:51:13.257 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.241+0000 7fb6e735f640 10 monclient: get_monmap_and_config success 2024-09-16T03:51:13.257 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.241+0000 7fb6e735f640 4 set_mon_vals no callback set 2024-09-16T03:51:13.257 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.241+0000 7fb6e735f640 10 monclient: shutdown 2024-09-16T03:51:13.257 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.241+0000 7fb6e735f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:51:13.257 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.241+0000 7fb6e735f640 1 -- 172.21.15.42:0/33966113 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb6e00a39b0 msgr2=0x7fb6e0133b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:13.258 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.241+0000 7fb6e735f640 1 --2- 172.21.15.42:0/33966113 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb6e00a39b0 0x7fb6e0133b80 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fb6e00a7750 tx=0x7fb6dc01c400 comp rx=0 tx=0).stop 2024-09-16T03:51:13.258 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.241+0000 7fb6e735f640 1 -- 172.21.15.42:0/33966113 shutdown_connections 2024-09-16T03:51:13.258 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.241+0000 7fb6e735f640 1 --2- 172.21.15.42:0/33966113 >> v2:172.21.15.42:3300/0 conn(0x7fb6e0137710 0x7fb6e013f3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:13.258 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.241+0000 7fb6e735f640 1 --2- 172.21.15.42:0/33966113 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb6e00a39b0 0x7fb6e0133b80 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:13.258 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.241+0000 7fb6e735f640 1 -- 172.21.15.42:0/33966113 >> 172.21.15.42:0/33966113 conn(0x7fb6e00a9fb0 msgr2=0x7fb6e00aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:51:13.258 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.241+0000 7fb6e735f640 1 -- 172.21.15.42:0/33966113 shutdown_connections 2024-09-16T03:51:13.258 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.241+0000 7fb6e735f640 1 -- 172.21.15.42:0/33966113 wait complete. 2024-09-16T03:51:13.258 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.245+0000 7fb6e735f640 10 monclient: build_initial_monmap 2024-09-16T03:51:13.258 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.245+0000 7fb6e735f640 10 monclient: monmap: 2024-09-16T03:51:13.258 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:51:13.258 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:51:13.258 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:51:13.258 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:51:13.259 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:51:13.259 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:13.259 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:51:13.259 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:51:13.259 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:51:13.259 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:13.259 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.245+0000 7fb6e735f640 1 librados: starting msgr at 2024-09-16T03:51:13.259 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.245+0000 7fb6e735f640 1 librados: starting objecter 2024-09-16T03:51:13.259 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.245+0000 7fb6e735f640 1 Processor -- start 2024-09-16T03:51:13.259 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.245+0000 7fb6e735f640 1 -- start start 2024-09-16T03:51:13.259 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.245+0000 7fb6e735f640 1 librados: setting wanted keys 2024-09-16T03:51:13.259 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.245+0000 7fb6e735f640 1 librados: calling monclient init 2024-09-16T03:51:13.259 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.245+0000 7fb6e735f640 10 monclient: init 2024-09-16T03:51:13.260 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.245+0000 7fb6e735f640 10 monclient: _reopen_session rank -1 2024-09-16T03:51:13.260 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.245+0000 7fb6e735f640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:51:13.260 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.245+0000 7fb6e735f640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb6e00a39b0 0x7fb6e01cf690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:13.260 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.245+0000 7fb6e735f640 10 monclient(hunting): picked mon.noname-c con 0x7fb6e00a39b0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:51:13.260 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.245+0000 7fb6e735f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb6e00a3d90 0x7fb6e01cfbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:13.260 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.245+0000 7fb6e735f640 10 monclient(hunting): picked mon.noname-b con 0x7fb6e00a3d90 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:51:13.260 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.245+0000 7fb6e735f640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb6e0137710 0x7fb6e01d0110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:13.260 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.245+0000 7fb6e735f640 10 monclient(hunting): picked mon.noname-a con 0x7fb6e0137710 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:51:13.260 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.245+0000 7fb6e735f640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:13.260 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.245+0000 7fb6e735f640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fb6e0142310 con 0x7fb6e0137710 2024-09-16T03:51:13.260 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.245+0000 7fb6e735f640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:13.260 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.245+0000 7fb6e735f640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fb6e0142490 con 0x7fb6e00a3d90 2024-09-16T03:51:13.260 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.245+0000 7fb6e735f640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:13.261 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.245+0000 7fb6e735f640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fb6e0142610 con 0x7fb6e00a39b0 2024-09-16T03:51:13.261 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.245+0000 7fb6e735f640 10 monclient(hunting): _renew_subs 2024-09-16T03:51:13.261 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.245+0000 7fb6e735f640 10 monclient(hunting): authenticate will time out at 1520.217291s 2024-09-16T03:51:13.261 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.245+0000 7fb6e58fc640 1 -- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb6e0137710 msgr2=0x7fb6e01d0110 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.42:3300/0 2024-09-16T03:51:13.261 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.245+0000 7fb6e58fc640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb6e0137710 0x7fb6e01d0110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:51:13.261 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.245+0000 7fb6e50fb640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb6e00a39b0 0x7fb6e01cf690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:13.261 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.245+0000 7fb6e50fb640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb6e00a39b0 0x7fb6e01cf690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:43608/0 (socket says 172.21.15.42:43608) 2024-09-16T03:51:13.261 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.245+0000 7fb6e50fb640 1 -- 172.21.15.42:0/2531031533 learned_addr learned my addr 172.21.15.42:0/2531031533 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:51:13.261 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.245+0000 7fb6e50fb640 10 monclient(hunting): get_auth_request con 0x7fb6e00a39b0 auth_method 0 2024-09-16T03:51:13.261 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.245+0000 7fb6e50fb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:13.261 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.245+0000 7fb6e50fb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:13.261 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.245+0000 7fb6e50fb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:51:13.261 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.245+0000 7fb6e50fb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:51:13.262 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.245+0000 7fb6e50fb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:51:13.262 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.245+0000 7fb6e50fb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:51:13.262 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.245+0000 7fb6e48fa640 1 --2- 172.21.15.42:0/2531031533 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb6e00a3d90 0x7fb6e01cfbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:13.262 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.245+0000 7fb6e50fb640 10 monclient(hunting): handle_auth_done global_id 814157 payload 824 2024-09-16T03:51:13.262 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.245+0000 7fb6e50fb640 1 -- 172.21.15.42:0/2531031533 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb6e00a3d90 msgr2=0x7fb6e01cfbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:13.262 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.245+0000 7fb6e50fb640 1 --2- 172.21.15.42:0/2531031533 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb6e00a3d90 0x7fb6e01cfbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:13.262 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.245+0000 7fb6e50fb640 1 -- 172.21.15.42:0/2531031533 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb6e0137710 msgr2=0x7fb6e01d0110 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:51:13.262 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.245+0000 7fb6e50fb640 1 --2- 172.21.15.42:0/2531031533 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb6e0137710 0x7fb6e01d0110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:13.262 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.245+0000 7fb6e50fb640 10 monclient: _finish_hunting 0 2024-09-16T03:51:13.262 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.245+0000 7fb6e50fb640 1 monclient: found mon.noname-c 2024-09-16T03:51:13.262 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.245+0000 7fb6e50fb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:51:13.262 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.245+0000 7fb6e50fb640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:51:13.262 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.245+0000 7fb6e50fb640 1 -- 172.21.15.42:0/2531031533 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6e00a2080 con 0x7fb6e00a39b0 2024-09-16T03:51:13.263 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.245+0000 7fb6e48fa640 10 monclient: get_auth_request con 0x7fb6e00a3d90 auth_method 0 2024-09-16T03:51:13.263 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.245+0000 7fb6e48fa640 1 --2- 172.21.15.42:0/2531031533 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb6e00a3d90 0x7fb6e01cfbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T03:51:13.263 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.245+0000 7fb6e50fb640 1 --2- 172.21.15.42:0/2531031533 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb6e00a39b0 0x7fb6e01cf690 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fb6dc0ae040 tx=0x7fb6dc0191a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:51:13.263 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.245+0000 7fb6d5ffb640 10 client.?.objecter ms_handle_connect 0x7fb6e00a39b0 2024-09-16T03:51:13.263 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.245+0000 7fb6d5ffb640 10 client.?.objecter resend_mon_ops 2024-09-16T03:51:13.263 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.245+0000 7fb6d5ffb640 1 -- 172.21.15.42:0/2531031533 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6dc0b8020 con 0x7fb6e00a39b0 2024-09-16T03:51:13.263 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.245+0000 7fb6d5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:51:13.263 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.245+0000 7fb6d5ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:51:13.263 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.245+0000 7fb6d5ffb640 10 monclient: dump: 2024-09-16T03:51:13.263 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:51:13.263 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:13.263 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:13.263 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:13.263 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:51:13.264 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:13.264 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:51:13.264 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:51:13.264 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:51:13.264 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:13.264 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.245+0000 7fb6d5ffb640 10 monclient: _finish_auth 0 2024-09-16T03:51:13.264 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.245+0000 7fb6d5ffb640 10 monclient: _check_auth_tickets 2024-09-16T03:51:13.264 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.245+0000 7fb6d5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:51:13.264 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.245+0000 7fb6e735f640 5 monclient: authenticate success, global_id 814157 2024-09-16T03:51:13.264 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.245+0000 7fb6e735f640 10 monclient: _renew_subs 2024-09-16T03:51:13.264 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.245+0000 7fb6d5ffb640 1 -- 172.21.15.42:0/2531031533 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb6dc0aa720 con 0x7fb6e00a39b0 2024-09-16T03:51:13.264 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.245+0000 7fb6e735f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:51:13.265 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.245+0000 7fb6e735f640 1 -- 172.21.15.42:0/2531031533 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6e00a10d0 con 0x7fb6e00a39b0 2024-09-16T03:51:13.265 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.245+0000 7fb6d5ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T03:51:13.265 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.245+0000 7fb6d5ffb640 1 -- 172.21.15.42:0/2531031533 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6dc0b0040 con 0x7fb6e00a39b0 2024-09-16T03:51:13.265 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.245+0000 7fb6d5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:51:13.265 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.245+0000 7fb6d5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:51:13.265 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.245+0000 7fb6d5ffb640 10 monclient: dump: 2024-09-16T03:51:13.265 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:51:13.265 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:13.265 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:13.265 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:13.265 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:51:13.265 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:13.265 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:51:13.265 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:51:13.266 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:51:13.266 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:13.266 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.245+0000 7fb6e735f640 10 client.814157.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:51:13.266 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.245+0000 7fb6e735f640 10 monclient: _renew_subs 2024-09-16T03:51:13.266 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.245+0000 7fb6e735f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:51:13.266 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.245+0000 7fb6e735f640 1 -- 172.21.15.42:0/2531031533 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6e00a0330 con 0x7fb6e00a39b0 2024-09-16T03:51:13.266 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.245+0000 7fb6e735f640 1 librados: init done 2024-09-16T03:51:13.266 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.245+0000 7fb6d77fe640 4 set_mon_vals no callback set 2024-09-16T03:51:13.266 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.245+0000 7fb6e735f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:51:13.266 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.245+0000 7fb6e735f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:51:13.266 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.245+0000 7fb6e735f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:51:13.266 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.245+0000 7fb6d5ffb640 1 -- 172.21.15.42:0/2531031533 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fb6dc0bb020 con 0x7fb6e00a39b0 2024-09-16T03:51:13.266 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.245+0000 7fb6e735f640 1 -- 172.21.15.42:0/2531031533 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb6e00a1d80 con 0x7fb6e00a39b0 2024-09-16T03:51:13.266 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.245+0000 7fb6d5ffb640 1 --2- 172.21.15.42:0/2531031533 >> v2:172.21.15.154:6812/254845342 conn(0x7fb6c40377c0 0x7fb6c4039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:13.267 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.245+0000 7fb6d5ffb640 1 -- 172.21.15.42:0/2531031533 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(309..309 src has 1..309) ==== 5731+0+0 (secure 0 0 0) 0x7fb6dc0e2ee0 con 0x7fb6e00a39b0 2024-09-16T03:51:13.267 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.245+0000 7fb6d5ffb640 10 client.814157.objecter ms_dispatch 0x7fb6e0000ba0 osd_map(309..309 src has 1..309) 2024-09-16T03:51:13.267 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.245+0000 7fb6d5ffb640 3 client.814157.objecter handle_osd_map got epochs [309,309] > 0 2024-09-16T03:51:13.267 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.245+0000 7fb6d5ffb640 3 client.814157.objecter handle_osd_map decoding full epoch 309 2024-09-16T03:51:13.267 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.245+0000 7fb6d5ffb640 20 client.814157.objecter dump_active .. 0 homeless 2024-09-16T03:51:13.267 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.245+0000 7fb6e48fa640 1 --2- 172.21.15.42:0/2531031533 >> v2:172.21.15.154:6812/254845342 conn(0x7fb6c40377c0 0x7fb6c4039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:13.267 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.245+0000 7fb6e48fa640 10 monclient: get_auth_request con 0x7fb6c40377c0 auth_method 0 2024-09-16T03:51:13.267 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.249+0000 7fb6e48fa640 1 --2- 172.21.15.42:0/2531031533 >> v2:172.21.15.154:6812/254845342 conn(0x7fb6c40377c0 0x7fb6c4039c80 secure :-1 s=READY pgs=3559 cs=0 l=1 rev1=1 crypto rx=0x7fb6e0132600 tx=0x7fb6d0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:51:13.267 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.249+0000 7fb6d5ffb640 10 client.814157.objecter ms_handle_connect 0x7fb6c40377c0 2024-09-16T03:51:13.267 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.253+0000 7fb6d5ffb640 1 -- 172.21.15.42:0/2531031533 <== mon.2 v2:172.21.15.42:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7fb6dc0c2030 con 0x7fb6e00a39b0 2024-09-16T03:51:13.268 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.253+0000 7fb6d5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:51:13.268 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.253+0000 7fb6d5ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:51:13.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.473+0000 7fb6e735f640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:51:13.481 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.473+0000 7fb6e735f640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:51:13.481 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.473+0000 7fb6e735f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:51:13.481 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.473+0000 7fb6e735f640 1 -- 172.21.15.42:0/2531031533 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fb6e0133b80 con 0x7fb6e00a39b0 2024-09-16T03:51:13.481 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.477+0000 7fb6d5ffb640 1 -- 172.21.15.42:0/2531031533 <== mon.2 v2:172.21.15.42:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7fb6dc016110 con 0x7fb6e00a39b0 2024-09-16T03:51:13.481 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.477+0000 7fb6d5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:51:13.481 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.477+0000 7fb6d5ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:51:13.482 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":80,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:51:13.483 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.477+0000 7fb6e735f640 10 librados: watch_flush enter 2024-09-16T03:51:13.483 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.477+0000 7fb6e735f640 10 librados: watch_flush exit 2024-09-16T03:51:13.484 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.477+0000 7fb6e735f640 20 client.814157.objecter shutdown clearing up homeless session... 2024-09-16T03:51:13.484 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.477+0000 7fb6e735f640 10 client.814157.objecter successfully canceled tick 2024-09-16T03:51:13.484 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.477+0000 7fb6e735f640 1 -- 172.21.15.42:0/2531031533 >> v2:172.21.15.154:6812/254845342 conn(0x7fb6c40377c0 msgr2=0x7fb6c4039c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:13.484 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.477+0000 7fb6e735f640 1 --2- 172.21.15.42:0/2531031533 >> v2:172.21.15.154:6812/254845342 conn(0x7fb6c40377c0 0x7fb6c4039c80 secure :-1 s=READY pgs=3559 cs=0 l=1 rev1=1 crypto rx=0x7fb6e0132600 tx=0x7fb6d0009040 comp rx=0 tx=0).stop 2024-09-16T03:51:13.484 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.477+0000 7fb6e735f640 10 monclient: shutdown 2024-09-16T03:51:13.484 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.477+0000 7fb6e735f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:51:13.484 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.477+0000 7fb6e735f640 1 -- 172.21.15.42:0/2531031533 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb6e00a39b0 msgr2=0x7fb6e01cf690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:13.484 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.477+0000 7fb6e735f640 1 --2- 172.21.15.42:0/2531031533 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb6e00a39b0 0x7fb6e01cf690 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fb6dc0ae040 tx=0x7fb6dc0191a0 comp rx=0 tx=0).stop 2024-09-16T03:51:13.484 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.477+0000 7fb6e735f640 1 -- 172.21.15.42:0/2531031533 shutdown_connections 2024-09-16T03:51:13.484 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.477+0000 7fb6e735f640 1 --2- 172.21.15.42:0/2531031533 >> v2:172.21.15.154:6812/254845342 conn(0x7fb6c40377c0 0x7fb6c4039c80 unknown :-1 s=CLOSED pgs=3559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:13.484 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.477+0000 7fb6e735f640 1 --2- 172.21.15.42:0/2531031533 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb6e0137710 0x7fb6e01d0110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:13.484 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.477+0000 7fb6e735f640 1 --2- 172.21.15.42:0/2531031533 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb6e00a3d90 0x7fb6e01cfbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:13.485 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.477+0000 7fb6e735f640 1 --2- 172.21.15.42:0/2531031533 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb6e00a39b0 0x7fb6e01cf690 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:13.485 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.477+0000 7fb6e735f640 1 -- 172.21.15.42:0/2531031533 >> 172.21.15.42:0/2531031533 conn(0x7fb6e00a9fb0 msgr2=0x7fb6e00a8a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:51:13.485 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.477+0000 7fb6e735f640 1 -- 172.21.15.42:0/2531031533 shutdown_connections 2024-09-16T03:51:13.485 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.477+0000 7fb6e735f640 1 -- 172.21.15.42:0/2531031533 wait complete. 2024-09-16T03:51:13.485 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:13.477+0000 7fb6e735f640 1 librados: shutdown 2024-09-16T03:51:16.502 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:51:16.638 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.629+0000 7f7a50e08640 10 monclient: get_monmap_and_config 2024-09-16T03:51:16.638 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.629+0000 7f7a50e08640 10 monclient: build_initial_monmap 2024-09-16T03:51:16.639 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.629+0000 7f7a50e08640 10 monclient: monmap: 2024-09-16T03:51:16.639 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:51:16.639 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:16.639 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:51:16.633960+0000 2024-09-16T03:51:16.639 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:51:16.633960+0000 2024-09-16T03:51:16.639 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:51:16.639 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:16.639 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:51:16.639 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:51:16.639 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:51:16.639 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:51:16.639 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:51:16.639 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:16.640 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.629+0000 7f7a50e08640 1 Processor -- start 2024-09-16T03:51:16.640 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.629+0000 7f7a50e08640 1 -- start start 2024-09-16T03:51:16.640 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.629+0000 7f7a50e08640 10 monclient: init 2024-09-16T03:51:16.640 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.629+0000 7f7a50e08640 10 monclient: _reopen_session rank -1 2024-09-16T03:51:16.640 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.629+0000 7f7a50e08640 10 monclient: _add_conns ranks=[1,0,4,3,2] 2024-09-16T03:51:16.640 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.629+0000 7f7a50e08640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7a4c0a1330 addr v1:172.21.15.154:6789/0 2024-09-16T03:51:16.640 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.629+0000 7f7a50e08640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7a4c06dac0 addr v1:172.21.15.42:6789/0 2024-09-16T03:51:16.640 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.629+0000 7f7a50e08640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f7a4c0a1e20 0x7f7a4c13da70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:16.640 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.629+0000 7f7a50e08640 10 monclient(hunting): picked mon.noname-c con 0x7f7a4c0a1e20 addr v2:172.21.15.154:3300/0 2024-09-16T03:51:16.640 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.629+0000 7f7a50e08640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7a4c09c090 con 0x7f7a4c06dac0 2024-09-16T03:51:16.640 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.629+0000 7f7a50e08640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7a4c09f610 con 0x7f7a4c0a1330 2024-09-16T03:51:16.640 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.629+0000 7f7a50e08640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:16.641 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.629+0000 7f7a50e08640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f7a4c09f8f0 con 0x7f7a4c0a1e20 2024-09-16T03:51:16.641 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.629+0000 7f7a50e08640 10 monclient(hunting): _renew_subs 2024-09-16T03:51:16.641 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.629+0000 7f7a50e08640 10 monclient(hunting): authenticate will time out at 1523.603447s 2024-09-16T03:51:16.641 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.629+0000 7f7a49d9b640 1 -- >> v1:172.21.15.42:6789/0 conn(0x7f7a4c06dac0 legacy=0x7f7a4c13a340 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.42:6789/0 2024-09-16T03:51:16.641 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.629+0000 7f7a4ad9d640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f7a4c0a1e20 0x7f7a4c13da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:16.641 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.633+0000 7f7a4ad9d640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f7a4c0a1e20 0x7f7a4c13da70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.42:33028/0 (socket says 172.21.15.42:33028) 2024-09-16T03:51:16.641 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.633+0000 7f7a4ad9d640 1 -- 172.21.15.42:0/3403789298 learned_addr learned my addr 172.21.15.42:0/3403789298 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:51:16.641 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.633+0000 7f7a4ad9d640 10 monclient(hunting): get_auth_request con 0x7f7a4c0a1e20 auth_method 0 2024-09-16T03:51:16.641 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.633+0000 7f7a4ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:16.641 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.633+0000 7f7a4ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:16.642 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.633+0000 7f7a4ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:51:16.642 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.633+0000 7f7a4ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:51:16.642 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.633+0000 7f7a4ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:51:16.642 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.633+0000 7f7a4ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:51:16.642 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.633+0000 7f7a4ad9d640 10 monclient(hunting): handle_auth_done global_id 794347 payload 274 2024-09-16T03:51:16.642 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.633+0000 7f7a4ad9d640 1 -- 172.21.15.42:0/3403789298 >> v1:172.21.15.154:6789/0 conn(0x7f7a4c0a1330 legacy=0x7f7a4c0a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:16.642 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.633+0000 7f7a4ad9d640 1 -- 172.21.15.42:0/3403789298 >> v1:172.21.15.42:6789/0 conn(0x7f7a4c06dac0 legacy=0x7f7a4c13a340 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:51:16.642 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.633+0000 7f7a4ad9d640 10 monclient: _finish_hunting 0 2024-09-16T03:51:16.642 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.633+0000 7f7a4ad9d640 1 monclient: found mon.noname-c 2024-09-16T03:51:16.642 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.633+0000 7f7a4ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:51:16.642 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.633+0000 7f7a4ad9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.154:3300/0 2024-09-16T03:51:16.642 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.633+0000 7f7a4ad9d640 1 -- 172.21.15.42:0/3403789298 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a4c09fbd0 con 0x7f7a4c0a1e20 2024-09-16T03:51:16.647 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.633+0000 7f7a4ad9d640 1 --2- 172.21.15.42:0/3403789298 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7a4c0a1e20 0x7f7a4c13da70 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f7a44009e50 tx=0x7f7a4401c940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=92fd091a78cf937a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:51:16.647 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.633+0000 7f7a4959a640 1 -- 172.21.15.42:0/3403789298 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a440a5600 con 0x7f7a4c0a1e20 2024-09-16T03:51:16.648 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.633+0000 7f7a4959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:51:16.648 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.633+0000 7f7a4959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:51:16.648 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.633+0000 7f7a4959a640 10 monclient: dump: 2024-09-16T03:51:16.648 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:51:16.648 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:16.648 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:16.648 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:16.648 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:51:16.648 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:16.648 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:51:16.648 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:51:16.648 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:51:16.649 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:16.649 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.633+0000 7f7a4959a640 10 monclient: _finish_auth 0 2024-09-16T03:51:16.649 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.633+0000 7f7a4959a640 10 monclient: _check_auth_tickets 2024-09-16T03:51:16.649 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.633+0000 7f7a4959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:51:16.649 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.633+0000 7f7a4959a640 1 -- 172.21.15.42:0/3403789298 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7a440a57a0 con 0x7f7a4c0a1e20 2024-09-16T03:51:16.649 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.633+0000 7f7a4959a640 10 monclient: handle_config config(0 keys) 2024-09-16T03:51:16.649 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.633+0000 7f7a4959a640 1 -- 172.21.15.42:0/3403789298 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a440a5b70 con 0x7f7a4c0a1e20 2024-09-16T03:51:16.649 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.633+0000 7f7a50e08640 5 monclient: authenticate success, global_id 794347 2024-09-16T03:51:16.649 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.633+0000 7f7a50e08640 10 monclient: get_monmap_and_config success 2024-09-16T03:51:16.649 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.633+0000 7f7a50e08640 4 set_mon_vals no callback set 2024-09-16T03:51:16.649 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.633+0000 7f7a4959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:51:16.649 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.633+0000 7f7a4959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:51:16.649 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.633+0000 7f7a4959a640 10 monclient: dump: 2024-09-16T03:51:16.649 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:51:16.650 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:16.650 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:16.650 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:16.650 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:51:16.650 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:16.650 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:51:16.650 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:51:16.650 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:51:16.650 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:16.650 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.637+0000 7f7a50e08640 10 monclient: shutdown 2024-09-16T03:51:16.650 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.637+0000 7f7a50e08640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:51:16.650 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.637+0000 7f7a50e08640 1 -- 172.21.15.42:0/3403789298 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7a4c0a1e20 msgr2=0x7f7a4c13da70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:16.650 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.637+0000 7f7a50e08640 1 --2- 172.21.15.42:0/3403789298 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7a4c0a1e20 0x7f7a4c13da70 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f7a44009e50 tx=0x7f7a4401c940 comp rx=0 tx=0).stop 2024-09-16T03:51:16.650 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.637+0000 7f7a50e08640 1 -- 172.21.15.42:0/3403789298 shutdown_connections 2024-09-16T03:51:16.651 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.637+0000 7f7a50e08640 1 --2- 172.21.15.42:0/3403789298 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7a4c0a1e20 0x7f7a4c13da70 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:16.651 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.637+0000 7f7a50e08640 1 -- 172.21.15.42:0/3403789298 >> 172.21.15.42:0/3403789298 conn(0x7f7a4c0a2420 msgr2=0x7f7a4c0a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:51:16.651 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.637+0000 7f7a50e08640 1 -- 172.21.15.42:0/3403789298 shutdown_connections 2024-09-16T03:51:16.651 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.637+0000 7f7a50e08640 1 -- 172.21.15.42:0/3403789298 wait complete. 2024-09-16T03:51:16.651 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.637+0000 7f7a50e08640 10 monclient: build_initial_monmap 2024-09-16T03:51:16.651 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.637+0000 7f7a50e08640 10 monclient: monmap: 2024-09-16T03:51:16.651 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:51:16.651 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:51:16.651 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:51:16.651 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:51:16.651 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:51:16.651 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:16.651 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:51:16.651 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:51:16.652 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:51:16.652 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:16.652 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.637+0000 7f7a50e08640 1 librados: starting msgr at 2024-09-16T03:51:16.652 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.637+0000 7f7a50e08640 1 librados: starting objecter 2024-09-16T03:51:16.652 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.637+0000 7f7a50e08640 1 Processor -- start 2024-09-16T03:51:16.652 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.637+0000 7f7a50e08640 1 -- start start 2024-09-16T03:51:16.652 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.637+0000 7f7a50e08640 1 librados: setting wanted keys 2024-09-16T03:51:16.652 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.637+0000 7f7a50e08640 1 librados: calling monclient init 2024-09-16T03:51:16.652 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.637+0000 7f7a50e08640 10 monclient: init 2024-09-16T03:51:16.652 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.637+0000 7f7a50e08640 10 monclient: _reopen_session rank -1 2024-09-16T03:51:16.652 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.637+0000 7f7a50e08640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T03:51:16.652 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.637+0000 7f7a50e08640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7a4c0a1330 0x7f7a4c134ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:16.652 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.637+0000 7f7a50e08640 10 monclient(hunting): picked mon.noname-b con 0x7f7a4c0a1330 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:51:16.653 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.637+0000 7f7a50e08640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7a4c0a1e20 0x7f7a4c135430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:16.653 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.637+0000 7f7a50e08640 10 monclient(hunting): picked mon.noname-c con 0x7f7a4c0a1e20 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:51:16.653 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.637+0000 7f7a50e08640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7a4c11bf70 0x7f7a4c11c420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:16.653 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.637+0000 7f7a50e08640 10 monclient(hunting): picked mon.noname-a con 0x7f7a4c11bf70 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:51:16.653 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.637+0000 7f7a50e08640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:16.653 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.637+0000 7f7a50e08640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f7a4c1409e0 con 0x7f7a4c11bf70 2024-09-16T03:51:16.653 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.637+0000 7f7a50e08640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:16.653 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.637+0000 7f7a50e08640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f7a4c09efb0 con 0x7f7a4c0a1330 2024-09-16T03:51:16.653 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.637+0000 7f7a50e08640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:16.653 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.637+0000 7f7a50e08640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f7a4c140b60 con 0x7f7a4c0a1e20 2024-09-16T03:51:16.653 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.637+0000 7f7a50e08640 10 monclient(hunting): _renew_subs 2024-09-16T03:51:16.653 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.637+0000 7f7a50e08640 10 monclient(hunting): authenticate will time out at 1523.609818s 2024-09-16T03:51:16.653 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.637+0000 7f7a4ad9d640 1 -- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7a4c11bf70 msgr2=0x7f7a4c11c420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.42:3300/0 2024-09-16T03:51:16.654 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.637+0000 7f7a4ad9d640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7a4c11bf70 0x7f7a4c11c420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:51:16.654 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.637+0000 7f7a49d9b640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7a4c0a1e20 0x7f7a4c135430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:16.654 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.637+0000 7f7a49d9b640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7a4c0a1e20 0x7f7a4c135430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:53586/0 (socket says 172.21.15.42:53586) 2024-09-16T03:51:16.654 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.637+0000 7f7a49d9b640 1 -- 172.21.15.42:0/1906186865 learned_addr learned my addr 172.21.15.42:0/1906186865 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:51:16.654 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.637+0000 7f7a49d9b640 10 monclient(hunting): get_auth_request con 0x7f7a4c0a1e20 auth_method 0 2024-09-16T03:51:16.654 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.637+0000 7f7a49d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:16.654 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.637+0000 7f7a49d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:16.654 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.637+0000 7f7a49d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:51:16.654 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.637+0000 7f7a49d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:51:16.654 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.637+0000 7f7a49d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:51:16.654 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.637+0000 7f7a49d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:51:16.654 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.637+0000 7f7a4a59c640 1 --2- 172.21.15.42:0/1906186865 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7a4c0a1330 0x7f7a4c134ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:16.654 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.637+0000 7f7a4a59c640 10 monclient(hunting): get_auth_request con 0x7f7a4c0a1330 auth_method 0 2024-09-16T03:51:16.655 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.637+0000 7f7a4a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:16.655 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.637+0000 7f7a4a59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:16.655 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.637+0000 7f7a4a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:51:16.655 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.637+0000 7f7a49d9b640 10 monclient(hunting): handle_auth_done global_id 814160 payload 824 2024-09-16T03:51:16.655 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.637+0000 7f7a49d9b640 1 -- 172.21.15.42:0/1906186865 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7a4c0a1330 msgr2=0x7f7a4c134ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:16.655 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.637+0000 7f7a49d9b640 1 --2- 172.21.15.42:0/1906186865 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7a4c0a1330 0x7f7a4c134ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:16.655 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.637+0000 7f7a49d9b640 1 -- 172.21.15.42:0/1906186865 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7a4c11bf70 msgr2=0x7f7a4c11c420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:51:16.655 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.637+0000 7f7a49d9b640 1 --2- 172.21.15.42:0/1906186865 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7a4c11bf70 0x7f7a4c11c420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:16.655 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.637+0000 7f7a49d9b640 10 monclient: _finish_hunting 0 2024-09-16T03:51:16.655 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.637+0000 7f7a49d9b640 1 monclient: found mon.noname-c 2024-09-16T03:51:16.655 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.637+0000 7f7a49d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:51:16.655 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.637+0000 7f7a49d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:51:16.655 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.637+0000 7f7a49d9b640 1 -- 172.21.15.42:0/1906186865 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a4c11e680 con 0x7f7a4c0a1e20 2024-09-16T03:51:16.656 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.637+0000 7f7a49d9b640 1 --2- 172.21.15.42:0/1906186865 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7a4c0a1e20 0x7f7a4c135430 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f7a4000b530 tx=0x7f7a4000b9f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:51:16.656 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.637+0000 7f7a3affd640 10 client.?.objecter ms_handle_connect 0x7f7a4c0a1e20 2024-09-16T03:51:16.656 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.637+0000 7f7a3affd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:51:16.656 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.637+0000 7f7a3affd640 1 -- 172.21.15.42:0/1906186865 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a40013020 con 0x7f7a4c0a1e20 2024-09-16T03:51:16.656 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.637+0000 7f7a3affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:51:16.656 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.637+0000 7f7a3affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:51:16.656 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.637+0000 7f7a3affd640 10 monclient: dump: 2024-09-16T03:51:16.656 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:51:16.656 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:16.656 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:16.656 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:16.656 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:51:16.657 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:16.657 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:51:16.657 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:51:16.657 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:51:16.657 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:16.657 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.637+0000 7f7a3affd640 10 monclient: _finish_auth 0 2024-09-16T03:51:16.657 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.637+0000 7f7a3affd640 10 monclient: _check_auth_tickets 2024-09-16T03:51:16.657 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.637+0000 7f7a3affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:51:16.657 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.637+0000 7f7a50e08640 5 monclient: authenticate success, global_id 814160 2024-09-16T03:51:16.657 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.637+0000 7f7a50e08640 10 monclient: _renew_subs 2024-09-16T03:51:16.657 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.637+0000 7f7a3affd640 1 -- 172.21.15.42:0/1906186865 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7a40003df0 con 0x7f7a4c0a1e20 2024-09-16T03:51:16.657 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.637+0000 7f7a50e08640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:51:16.657 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.637+0000 7f7a50e08640 1 -- 172.21.15.42:0/1906186865 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a4c11d6d0 con 0x7f7a4c0a1e20 2024-09-16T03:51:16.658 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.637+0000 7f7a3affd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:51:16.658 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.637+0000 7f7a3affd640 1 -- 172.21.15.42:0/1906186865 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a400052b0 con 0x7f7a4c0a1e20 2024-09-16T03:51:16.658 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.637+0000 7f7a3affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:51:16.658 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.637+0000 7f7a3affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:51:16.658 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.637+0000 7f7a3affd640 10 monclient: dump: 2024-09-16T03:51:16.658 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:51:16.658 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:16.658 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:16.658 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:16.658 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:51:16.658 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:16.658 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:51:16.659 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:51:16.659 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:51:16.659 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:16.659 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.637+0000 7f7a48d99640 4 set_mon_vals no callback set 2024-09-16T03:51:16.659 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.637+0000 7f7a50e08640 10 client.814160.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:51:16.659 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.637+0000 7f7a50e08640 10 monclient: _renew_subs 2024-09-16T03:51:16.659 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.637+0000 7f7a50e08640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:51:16.659 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.637+0000 7f7a50e08640 1 -- 172.21.15.42:0/1906186865 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a4c06e590 con 0x7f7a4c0a1e20 2024-09-16T03:51:16.659 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.637+0000 7f7a50e08640 1 librados: init done 2024-09-16T03:51:16.659 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.637+0000 7f7a3affd640 1 -- 172.21.15.42:0/1906186865 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f7a40016020 con 0x7f7a4c0a1e20 2024-09-16T03:51:16.659 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.637+0000 7f7a50e08640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:51:16.659 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.641+0000 7f7a50e08640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:51:16.659 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.641+0000 7f7a50e08640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:51:16.659 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.641+0000 7f7a50e08640 1 -- 172.21.15.42:0/1906186865 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7a4c0a1710 con 0x7f7a4c0a1e20 2024-09-16T03:51:16.660 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.641+0000 7f7a3affd640 1 --2- 172.21.15.42:0/1906186865 >> v2:172.21.15.154:6812/254845342 conn(0x7f7a2c0377f0 0x7f7a2c039cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:16.660 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.641+0000 7f7a3affd640 1 -- 172.21.15.42:0/1906186865 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(309..309 src has 1..309) ==== 5731+0+0 (secure 0 0 0) 0x7f7a40048d10 con 0x7f7a4c0a1e20 2024-09-16T03:51:16.660 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.641+0000 7f7a3affd640 10 client.814160.objecter ms_dispatch 0x7f7a4c000ba0 osd_map(309..309 src has 1..309) 2024-09-16T03:51:16.660 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.641+0000 7f7a3affd640 3 client.814160.objecter handle_osd_map got epochs [309,309] > 0 2024-09-16T03:51:16.660 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.641+0000 7f7a3affd640 3 client.814160.objecter handle_osd_map decoding full epoch 309 2024-09-16T03:51:16.660 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.641+0000 7f7a3affd640 20 client.814160.objecter dump_active .. 0 homeless 2024-09-16T03:51:16.660 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.641+0000 7f7a4a59c640 1 --2- 172.21.15.42:0/1906186865 >> v2:172.21.15.154:6812/254845342 conn(0x7f7a2c0377f0 0x7f7a2c039cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:16.660 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.641+0000 7f7a4a59c640 10 monclient: get_auth_request con 0x7f7a2c0377f0 auth_method 0 2024-09-16T03:51:16.660 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.641+0000 7f7a4a59c640 1 --2- 172.21.15.42:0/1906186865 >> v2:172.21.15.154:6812/254845342 conn(0x7f7a2c0377f0 0x7f7a2c039cb0 secure :-1 s=READY pgs=3560 cs=0 l=1 rev1=1 crypto rx=0x7f7a3c001650 tx=0x7f7a3c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:51:16.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.641+0000 7f7a3affd640 10 client.814160.objecter ms_handle_connect 0x7f7a2c0377f0 2024-09-16T03:51:16.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.645+0000 7f7a3affd640 1 -- 172.21.15.42:0/1906186865 <== mon.2 v2:172.21.15.42:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f7a4001d020 con 0x7f7a4c0a1e20 2024-09-16T03:51:16.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.645+0000 7f7a3affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:51:16.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.645+0000 7f7a3affd640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:51:16.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.873+0000 7f7a50e08640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:51:16.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.873+0000 7f7a50e08640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:51:16.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.873+0000 7f7a50e08640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:51:16.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.873+0000 7f7a50e08640 1 -- 172.21.15.42:0/1906186865 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f7a4c135970 con 0x7f7a4c0a1e20 2024-09-16T03:51:16.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.873+0000 7f7a3affd640 1 -- 172.21.15.42:0/1906186865 <== mon.2 v2:172.21.15.42:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7f7a40017a00 con 0x7f7a4c0a1e20 2024-09-16T03:51:16.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.873+0000 7f7a3affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:51:16.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.873+0000 7f7a3affd640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:51:16.880 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":80,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:51:16.883 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.877+0000 7f7a50e08640 10 librados: watch_flush enter 2024-09-16T03:51:16.883 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.877+0000 7f7a50e08640 10 librados: watch_flush exit 2024-09-16T03:51:16.883 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.877+0000 7f7a50e08640 20 client.814160.objecter shutdown clearing up homeless session... 2024-09-16T03:51:16.883 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.877+0000 7f7a50e08640 10 client.814160.objecter successfully canceled tick 2024-09-16T03:51:16.883 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.877+0000 7f7a50e08640 1 -- 172.21.15.42:0/1906186865 >> v2:172.21.15.154:6812/254845342 conn(0x7f7a2c0377f0 msgr2=0x7f7a2c039cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:16.883 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.877+0000 7f7a50e08640 1 --2- 172.21.15.42:0/1906186865 >> v2:172.21.15.154:6812/254845342 conn(0x7f7a2c0377f0 0x7f7a2c039cb0 secure :-1 s=READY pgs=3560 cs=0 l=1 rev1=1 crypto rx=0x7f7a3c001650 tx=0x7f7a3c009040 comp rx=0 tx=0).stop 2024-09-16T03:51:16.883 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.877+0000 7f7a50e08640 10 monclient: shutdown 2024-09-16T03:51:16.883 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.877+0000 7f7a50e08640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:51:16.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.877+0000 7f7a50e08640 1 -- 172.21.15.42:0/1906186865 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7a4c0a1e20 msgr2=0x7f7a4c135430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:16.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.877+0000 7f7a50e08640 1 --2- 172.21.15.42:0/1906186865 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7a4c0a1e20 0x7f7a4c135430 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f7a4000b530 tx=0x7f7a4000b9f0 comp rx=0 tx=0).stop 2024-09-16T03:51:16.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.877+0000 7f7a50e08640 1 -- 172.21.15.42:0/1906186865 shutdown_connections 2024-09-16T03:51:16.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.877+0000 7f7a50e08640 1 --2- 172.21.15.42:0/1906186865 >> v2:172.21.15.154:6812/254845342 conn(0x7f7a2c0377f0 0x7f7a2c039cb0 unknown :-1 s=CLOSED pgs=3560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:16.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.877+0000 7f7a50e08640 1 --2- 172.21.15.42:0/1906186865 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7a4c11bf70 0x7f7a4c11c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:16.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.877+0000 7f7a50e08640 1 --2- 172.21.15.42:0/1906186865 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7a4c0a1e20 0x7f7a4c135430 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:16.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.877+0000 7f7a50e08640 1 --2- 172.21.15.42:0/1906186865 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7a4c0a1330 0x7f7a4c134ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:16.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.877+0000 7f7a50e08640 1 -- 172.21.15.42:0/1906186865 >> 172.21.15.42:0/1906186865 conn(0x7f7a4c0a2420 msgr2=0x7f7a4c0a46b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:51:16.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.877+0000 7f7a50e08640 1 -- 172.21.15.42:0/1906186865 shutdown_connections 2024-09-16T03:51:16.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.877+0000 7f7a50e08640 1 -- 172.21.15.42:0/1906186865 wait complete. 2024-09-16T03:51:16.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:16.877+0000 7f7a50e08640 1 librados: shutdown 2024-09-16T03:51:19.901 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:51:20.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.025+0000 7f78a908c640 10 monclient: get_monmap_and_config 2024-09-16T03:51:20.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.025+0000 7f78a908c640 10 monclient: build_initial_monmap 2024-09-16T03:51:20.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.025+0000 7f78a908c640 10 monclient: monmap: 2024-09-16T03:51:20.031 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:51:20.032 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:20.032 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:51:20.030309+0000 2024-09-16T03:51:20.032 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:51:20.030309+0000 2024-09-16T03:51:20.032 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:51:20.032 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:20.032 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:51:20.032 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:51:20.032 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:51:20.032 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:51:20.032 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:51:20.032 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:20.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.025+0000 7f78a908c640 1 Processor -- start 2024-09-16T03:51:20.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.025+0000 7f78a908c640 1 -- start start 2024-09-16T03:51:20.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.025+0000 7f78a908c640 10 monclient: init 2024-09-16T03:51:20.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.025+0000 7f78a908c640 10 monclient: _reopen_session rank -1 2024-09-16T03:51:20.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.025+0000 7f78a908c640 10 monclient: _add_conns ranks=[3,0,2,4,1] 2024-09-16T03:51:20.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.025+0000 7f78a908c640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f78a40a4720 0x7f78a41374e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:20.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.025+0000 7f78a908c640 10 monclient(hunting): picked mon.noname-a con 0x7f78a40a4720 addr v2:172.21.15.42:3300/0 2024-09-16T03:51:20.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.025+0000 7f78a908c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f78a4137a20 addr v1:172.21.15.42:6789/0 2024-09-16T03:51:20.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.025+0000 7f78a908c640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f78a4137e30 0x7f78a40a5210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:20.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.025+0000 7f78a908c640 10 monclient(hunting): picked mon.noname-b con 0x7f78a4137e30 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:51:20.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.025+0000 7f78a908c640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f78a40a1470 con 0x7f78a4137a20 2024-09-16T03:51:20.036 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.025+0000 7f78a908c640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:20.036 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.025+0000 7f78a908c640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f78a406e8f0 con 0x7f78a40a4720 2024-09-16T03:51:20.036 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.025+0000 7f78a908c640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:20.036 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.025+0000 7f78a908c640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f78a40a5f60 con 0x7f78a4137e30 2024-09-16T03:51:20.036 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.025+0000 7f78a908c640 10 monclient(hunting): _renew_subs 2024-09-16T03:51:20.036 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.025+0000 7f78a908c640 10 monclient(hunting): authenticate will time out at 1526.999769s 2024-09-16T03:51:20.036 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.025+0000 7f78a2d9d640 1 -- >> v2:172.21.15.42:3300/0 conn(0x7f78a40a4720 msgr2=0x7f78a41374e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3300/0 2024-09-16T03:51:20.036 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.025+0000 7f78a2d9d640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f78a40a4720 0x7f78a41374e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:51:20.036 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.025+0000 7f78a259c640 1 -- >> v1:172.21.15.42:6789/0 conn(0x7f78a4137a20 legacy=0x7f78a40a4b00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.42:6789/0 2024-09-16T03:51:20.036 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.025+0000 7f78a359e640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f78a4137e30 0x7f78a40a5210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:20.036 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.025+0000 7f78a359e640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f78a4137e30 0x7f78a40a5210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:53598/0 (socket says 172.21.15.42:53598) 2024-09-16T03:51:20.036 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.029+0000 7f78a359e640 1 -- 172.21.15.42:0/4017075663 learned_addr learned my addr 172.21.15.42:0/4017075663 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:51:20.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.029+0000 7f78a359e640 10 monclient(hunting): get_auth_request con 0x7f78a4137e30 auth_method 0 2024-09-16T03:51:20.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.029+0000 7f78a359e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:20.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.029+0000 7f78a359e640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:20.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.029+0000 7f78a359e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:51:20.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.029+0000 7f78a359e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:51:20.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.029+0000 7f78a359e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:51:20.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.029+0000 7f78a359e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:51:20.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.029+0000 7f78a359e640 10 monclient(hunting): handle_auth_done global_id 814163 payload 274 2024-09-16T03:51:20.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.029+0000 7f78a359e640 1 -- 172.21.15.42:0/4017075663 >> v2:172.21.15.42:3300/0 conn(0x7f78a40a4720 msgr2=0x7f78a41374e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:51:20.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.029+0000 7f78a359e640 1 --2- 172.21.15.42:0/4017075663 >> v2:172.21.15.42:3300/0 conn(0x7f78a40a4720 0x7f78a41374e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:20.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.029+0000 7f78a359e640 1 -- 172.21.15.42:0/4017075663 >> v1:172.21.15.42:6789/0 conn(0x7f78a4137a20 legacy=0x7f78a40a4b00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:51:20.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.029+0000 7f78a359e640 10 monclient: _finish_hunting 0 2024-09-16T03:51:20.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.029+0000 7f78a359e640 1 monclient: found mon.noname-b 2024-09-16T03:51:20.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.029+0000 7f78a359e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:51:20.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.029+0000 7f78a359e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T03:51:20.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.029+0000 7f78a359e640 1 -- 172.21.15.42:0/4017075663 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78a40a1750 con 0x7f78a4137e30 2024-09-16T03:51:20.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.029+0000 7f78a359e640 1 --2- 172.21.15.42:0/4017075663 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f78a4137e30 0x7f78a40a5210 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f789c009870 tx=0x7f789c01c400 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d3f095f7e04cab93 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:51:20.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.029+0000 7f78a1d9b640 1 -- 172.21.15.42:0/4017075663 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f789c0b0070 con 0x7f78a4137e30 2024-09-16T03:51:20.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.029+0000 7f78a1d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:51:20.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.029+0000 7f78a1d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:51:20.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.029+0000 7f78a1d9b640 10 monclient: dump: 2024-09-16T03:51:20.043 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:51:20.043 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:20.044 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:20.044 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:20.044 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:51:20.044 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:20.044 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:51:20.044 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:51:20.044 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:51:20.044 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:20.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.029+0000 7f78a1d9b640 10 monclient: _finish_auth 0 2024-09-16T03:51:20.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.029+0000 7f78a1d9b640 10 monclient: _check_auth_tickets 2024-09-16T03:51:20.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.029+0000 7f78a1d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:51:20.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.029+0000 7f78a1d9b640 1 -- 172.21.15.42:0/4017075663 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f789c0ab600 con 0x7f78a4137e30 2024-09-16T03:51:20.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.029+0000 7f78a1d9b640 10 monclient: handle_config config(0 keys) 2024-09-16T03:51:20.045 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.029+0000 7f78a1d9b640 1 -- 172.21.15.42:0/4017075663 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f789c0ab9d0 con 0x7f78a4137e30 2024-09-16T03:51:20.045 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.029+0000 7f78a908c640 5 monclient: authenticate success, global_id 814163 2024-09-16T03:51:20.045 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.029+0000 7f78a908c640 10 monclient: get_monmap_and_config success 2024-09-16T03:51:20.045 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.029+0000 7f78a908c640 4 set_mon_vals no callback set 2024-09-16T03:51:20.045 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.029+0000 7f78a1d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:51:20.045 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.029+0000 7f78a1d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:51:20.045 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.029+0000 7f78a1d9b640 10 monclient: dump: 2024-09-16T03:51:20.045 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:51:20.045 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:20.045 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:20.045 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:20.045 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:51:20.045 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:20.046 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:51:20.046 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:51:20.046 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:51:20.046 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:20.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.029+0000 7f78a908c640 10 monclient: shutdown 2024-09-16T03:51:20.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.029+0000 7f78a908c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:51:20.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.029+0000 7f78a908c640 1 -- 172.21.15.42:0/4017075663 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f78a4137e30 msgr2=0x7f78a40a5210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:20.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.029+0000 7f78a908c640 1 --2- 172.21.15.42:0/4017075663 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f78a4137e30 0x7f78a40a5210 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f789c009870 tx=0x7f789c01c400 comp rx=0 tx=0).stop 2024-09-16T03:51:20.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.029+0000 7f78a908c640 1 -- 172.21.15.42:0/4017075663 shutdown_connections 2024-09-16T03:51:20.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.029+0000 7f78a908c640 1 --2- 172.21.15.42:0/4017075663 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f78a4137e30 0x7f78a40a5210 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:20.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.029+0000 7f78a908c640 1 --2- 172.21.15.42:0/4017075663 >> v2:172.21.15.42:3300/0 conn(0x7f78a40a4720 0x7f78a41374e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:20.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.029+0000 7f78a908c640 1 -- 172.21.15.42:0/4017075663 >> 172.21.15.42:0/4017075663 conn(0x7f78a40a6820 msgr2=0x7f78a40a6c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:51:20.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.029+0000 7f78a908c640 1 -- 172.21.15.42:0/4017075663 shutdown_connections 2024-09-16T03:51:20.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.029+0000 7f78a908c640 1 -- 172.21.15.42:0/4017075663 wait complete. 2024-09-16T03:51:20.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.029+0000 7f78a908c640 10 monclient: build_initial_monmap 2024-09-16T03:51:20.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.029+0000 7f78a908c640 10 monclient: monmap: 2024-09-16T03:51:20.047 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:51:20.047 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:51:20.047 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:51:20.047 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:51:20.047 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:51:20.047 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:20.047 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:51:20.047 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:51:20.048 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:51:20.048 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:20.048 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.033+0000 7f78a908c640 1 librados: starting msgr at 2024-09-16T03:51:20.048 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.033+0000 7f78a908c640 1 librados: starting objecter 2024-09-16T03:51:20.048 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.033+0000 7f78a908c640 1 Processor -- start 2024-09-16T03:51:20.048 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.033+0000 7f78a908c640 1 -- start start 2024-09-16T03:51:20.048 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.033+0000 7f78a908c640 1 librados: setting wanted keys 2024-09-16T03:51:20.048 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.033+0000 7f78a908c640 1 librados: calling monclient init 2024-09-16T03:51:20.048 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.033+0000 7f78a908c640 10 monclient: init 2024-09-16T03:51:20.048 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.033+0000 7f78a908c640 10 monclient: _reopen_session rank -1 2024-09-16T03:51:20.048 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.033+0000 7f78a908c640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:51:20.048 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.033+0000 7f78a908c640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f78a40a4720 0x7f78a41cf610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:20.048 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.033+0000 7f78a908c640 10 monclient(hunting): picked mon.noname-c con 0x7f78a40a4720 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:51:20.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.033+0000 7f78a908c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f78a4137a20 0x7f78a4136900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:20.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.033+0000 7f78a908c640 10 monclient(hunting): picked mon.noname-a con 0x7f78a4137a20 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:51:20.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.033+0000 7f78a908c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f78a4137e30 0x7f78a413b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:20.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.033+0000 7f78a908c640 10 monclient(hunting): picked mon.noname-b con 0x7f78a4137e30 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:51:20.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.033+0000 7f78a908c640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:20.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.033+0000 7f78a908c640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f78a41420e0 con 0x7f78a4137a20 2024-09-16T03:51:20.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.033+0000 7f78a908c640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:20.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.033+0000 7f78a908c640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f78a4141f60 con 0x7f78a4137e30 2024-09-16T03:51:20.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.033+0000 7f78a908c640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:20.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.033+0000 7f78a908c640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f78a4142260 con 0x7f78a40a4720 2024-09-16T03:51:20.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.033+0000 7f78a908c640 10 monclient(hunting): _renew_subs 2024-09-16T03:51:20.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.033+0000 7f78a908c640 10 monclient(hunting): authenticate will time out at 1527.004789s 2024-09-16T03:51:20.050 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.033+0000 7f78a259c640 1 -- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f78a4137a20 msgr2=0x7f78a4136900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.42:3300/0 2024-09-16T03:51:20.050 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.033+0000 7f78a259c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f78a4137a20 0x7f78a4136900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:51:20.050 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.033+0000 7f78a2d9d640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f78a40a4720 0x7f78a41cf610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:20.050 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.033+0000 7f78a2d9d640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f78a40a4720 0x7f78a41cf610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:53612/0 (socket says 172.21.15.42:53612) 2024-09-16T03:51:20.050 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.033+0000 7f78a2d9d640 1 -- 172.21.15.42:0/3388942110 learned_addr learned my addr 172.21.15.42:0/3388942110 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:51:20.050 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.033+0000 7f78a2d9d640 10 monclient(hunting): get_auth_request con 0x7f78a40a4720 auth_method 0 2024-09-16T03:51:20.050 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.033+0000 7f78a2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:20.050 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.033+0000 7f78a2d9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:20.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.033+0000 7f78a2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:51:20.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.033+0000 7f78a2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:51:20.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.033+0000 7f78a2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:51:20.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.033+0000 7f78a2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:51:20.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.033+0000 7f78a2d9d640 10 monclient(hunting): handle_auth_done global_id 814166 payload 824 2024-09-16T03:51:20.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.033+0000 7f78a359e640 1 --2- 172.21.15.42:0/3388942110 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f78a4137e30 0x7f78a413b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:20.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.033+0000 7f78a2d9d640 1 -- 172.21.15.42:0/3388942110 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f78a4137e30 msgr2=0x7f78a413b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:20.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.033+0000 7f78a2d9d640 1 --2- 172.21.15.42:0/3388942110 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f78a4137e30 0x7f78a413b7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:20.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.033+0000 7f78a2d9d640 1 -- 172.21.15.42:0/3388942110 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f78a4137a20 msgr2=0x7f78a4136900 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:51:20.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.033+0000 7f78a2d9d640 1 --2- 172.21.15.42:0/3388942110 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f78a4137a20 0x7f78a4136900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:20.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.033+0000 7f78a2d9d640 10 monclient: _finish_hunting 0 2024-09-16T03:51:20.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.033+0000 7f78a2d9d640 1 monclient: found mon.noname-c 2024-09-16T03:51:20.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.033+0000 7f78a2d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:51:20.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.033+0000 7f78a2d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:51:20.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.033+0000 7f78a2d9d640 1 -- 172.21.15.42:0/3388942110 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78a4120cd0 con 0x7f78a40a4720 2024-09-16T03:51:20.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.033+0000 7f78a2d9d640 1 --2- 172.21.15.42:0/3388942110 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f78a40a4720 0x7f78a41cf610 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f788c00b530 tx=0x7f788c00b9f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:51:20.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.033+0000 7f789b7fe640 10 client.?.objecter ms_handle_connect 0x7f78a40a4720 2024-09-16T03:51:20.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.033+0000 7f789b7fe640 10 client.?.objecter resend_mon_ops 2024-09-16T03:51:20.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.033+0000 7f789b7fe640 1 -- 172.21.15.42:0/3388942110 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f788c013020 con 0x7f78a40a4720 2024-09-16T03:51:20.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.033+0000 7f789b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:51:20.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.033+0000 7f789b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:51:20.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.033+0000 7f789b7fe640 10 monclient: dump: 2024-09-16T03:51:20.052 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:51:20.053 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:20.053 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:20.053 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:20.053 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:51:20.053 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:20.053 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:51:20.053 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:51:20.053 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:51:20.053 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:20.053 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.033+0000 7f789b7fe640 10 monclient: _finish_auth 0 2024-09-16T03:51:20.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.033+0000 7f789b7fe640 10 monclient: _check_auth_tickets 2024-09-16T03:51:20.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.033+0000 7f789b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:51:20.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.033+0000 7f789b7fe640 1 -- 172.21.15.42:0/3388942110 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f788c003df0 con 0x7f78a40a4720 2024-09-16T03:51:20.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.033+0000 7f78a908c640 5 monclient: authenticate success, global_id 814166 2024-09-16T03:51:20.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.033+0000 7f78a908c640 10 monclient: _renew_subs 2024-09-16T03:51:20.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.033+0000 7f78a908c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:51:20.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.033+0000 7f78a908c640 1 -- 172.21.15.42:0/3388942110 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78a4120fb0 con 0x7f78a40a4720 2024-09-16T03:51:20.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.033+0000 7f789b7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:51:20.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.033+0000 7f789b7fe640 1 -- 172.21.15.42:0/3388942110 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f788c0052b0 con 0x7f78a40a4720 2024-09-16T03:51:20.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.033+0000 7f789b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:51:20.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.033+0000 7f789b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:51:20.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.033+0000 7f789b7fe640 10 monclient: dump: 2024-09-16T03:51:20.054 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:51:20.055 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:20.055 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:20.055 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:20.055 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:51:20.055 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:20.055 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:51:20.055 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:51:20.055 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:51:20.055 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:20.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.033+0000 7f78a908c640 10 client.814166.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:51:20.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.033+0000 7f78a908c640 10 monclient: _renew_subs 2024-09-16T03:51:20.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.033+0000 7f78a159a640 4 set_mon_vals no callback set 2024-09-16T03:51:20.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.033+0000 7f78a908c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:51:20.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.033+0000 7f78a908c640 1 -- 172.21.15.42:0/3388942110 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f78a40a8600 con 0x7f78a40a4720 2024-09-16T03:51:20.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.033+0000 7f78a908c640 1 librados: init done 2024-09-16T03:51:20.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.033+0000 7f78a908c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:51:20.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.033+0000 7f78a908c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:51:20.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.033+0000 7f78a908c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:51:20.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.033+0000 7f78a908c640 1 -- 172.21.15.42:0/3388942110 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f78a40a8240 con 0x7f78a40a4720 2024-09-16T03:51:20.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.033+0000 7f789b7fe640 1 -- 172.21.15.42:0/3388942110 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f788c016020 con 0x7f78a40a4720 2024-09-16T03:51:20.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.033+0000 7f789b7fe640 1 --2- 172.21.15.42:0/3388942110 >> v2:172.21.15.154:6812/254845342 conn(0x7f78840377c0 0x7f7884039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:20.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.033+0000 7f789b7fe640 1 -- 172.21.15.42:0/3388942110 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(309..309 src has 1..309) ==== 5731+0+0 (secure 0 0 0) 0x7f788c048bb0 con 0x7f78a40a4720 2024-09-16T03:51:20.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.033+0000 7f789b7fe640 10 client.814166.objecter ms_dispatch 0x7f78a4000ba0 osd_map(309..309 src has 1..309) 2024-09-16T03:51:20.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.033+0000 7f789b7fe640 3 client.814166.objecter handle_osd_map got epochs [309,309] > 0 2024-09-16T03:51:20.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.033+0000 7f789b7fe640 3 client.814166.objecter handle_osd_map decoding full epoch 309 2024-09-16T03:51:20.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.033+0000 7f789b7fe640 20 client.814166.objecter dump_active .. 0 homeless 2024-09-16T03:51:20.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.033+0000 7f78a259c640 1 --2- 172.21.15.42:0/3388942110 >> v2:172.21.15.154:6812/254845342 conn(0x7f78840377c0 0x7f7884039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:20.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.033+0000 7f78a259c640 10 monclient: get_auth_request con 0x7f78840377c0 auth_method 0 2024-09-16T03:51:20.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.037+0000 7f78a259c640 1 --2- 172.21.15.42:0/3388942110 >> v2:172.21.15.154:6812/254845342 conn(0x7f78840377c0 0x7f7884039c80 secure :-1 s=READY pgs=3561 cs=0 l=1 rev1=1 crypto rx=0x7f78a41211c0 tx=0x7f7894009270 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:51:20.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.037+0000 7f789b7fe640 10 client.814166.objecter ms_handle_connect 0x7f78840377c0 2024-09-16T03:51:20.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.037+0000 7f789b7fe640 1 -- 172.21.15.42:0/3388942110 <== mon.2 v2:172.21.15.42:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f788c01d020 con 0x7f78a40a4720 2024-09-16T03:51:20.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.037+0000 7f789b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:51:20.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.037+0000 7f789b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:51:20.268 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.261+0000 7f78a908c640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:51:20.268 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.261+0000 7f78a908c640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:51:20.268 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.261+0000 7f78a908c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:51:20.268 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.261+0000 7f78a908c640 1 -- 172.21.15.42:0/3388942110 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f78a41374e0 con 0x7f78a40a4720 2024-09-16T03:51:20.269 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.261+0000 7f789b7fe640 1 -- 172.21.15.42:0/3388942110 <== mon.2 v2:172.21.15.42:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7f788c048910 con 0x7f78a40a4720 2024-09-16T03:51:20.269 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.261+0000 7f789b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:51:20.269 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.261+0000 7f789b7fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:51:20.269 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":80,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:51:20.272 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.265+0000 7f78a908c640 10 librados: watch_flush enter 2024-09-16T03:51:20.272 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.265+0000 7f78a908c640 10 librados: watch_flush exit 2024-09-16T03:51:20.272 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.265+0000 7f78a908c640 20 client.814166.objecter shutdown clearing up homeless session... 2024-09-16T03:51:20.272 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.265+0000 7f78a908c640 10 client.814166.objecter successfully canceled tick 2024-09-16T03:51:20.272 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.265+0000 7f78a908c640 1 -- 172.21.15.42:0/3388942110 >> v2:172.21.15.154:6812/254845342 conn(0x7f78840377c0 msgr2=0x7f7884039c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:20.272 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.265+0000 7f78a908c640 1 --2- 172.21.15.42:0/3388942110 >> v2:172.21.15.154:6812/254845342 conn(0x7f78840377c0 0x7f7884039c80 secure :-1 s=READY pgs=3561 cs=0 l=1 rev1=1 crypto rx=0x7f78a41211c0 tx=0x7f7894009270 comp rx=0 tx=0).stop 2024-09-16T03:51:20.272 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.265+0000 7f78a908c640 10 monclient: shutdown 2024-09-16T03:51:20.272 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.265+0000 7f78a908c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:51:20.272 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.265+0000 7f78a908c640 1 -- 172.21.15.42:0/3388942110 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f78a40a4720 msgr2=0x7f78a41cf610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:20.272 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.265+0000 7f78a908c640 1 --2- 172.21.15.42:0/3388942110 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f78a40a4720 0x7f78a41cf610 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f788c00b530 tx=0x7f788c00b9f0 comp rx=0 tx=0).stop 2024-09-16T03:51:20.272 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.265+0000 7f78a908c640 1 -- 172.21.15.42:0/3388942110 shutdown_connections 2024-09-16T03:51:20.272 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.265+0000 7f78a908c640 1 --2- 172.21.15.42:0/3388942110 >> v2:172.21.15.154:6812/254845342 conn(0x7f78840377c0 0x7f7884039c80 unknown :-1 s=CLOSED pgs=3561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:20.273 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.265+0000 7f78a908c640 1 --2- 172.21.15.42:0/3388942110 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f78a4137e30 0x7f78a413b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:20.273 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.265+0000 7f78a908c640 1 --2- 172.21.15.42:0/3388942110 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f78a4137a20 0x7f78a4136900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:20.273 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.265+0000 7f78a908c640 1 --2- 172.21.15.42:0/3388942110 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f78a40a4720 0x7f78a41cf610 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:20.273 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.265+0000 7f78a908c640 1 -- 172.21.15.42:0/3388942110 >> 172.21.15.42:0/3388942110 conn(0x7f78a40a6820 msgr2=0x7f78a406dde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:51:20.273 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.265+0000 7f78a908c640 1 -- 172.21.15.42:0/3388942110 shutdown_connections 2024-09-16T03:51:20.273 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.265+0000 7f78a908c640 1 -- 172.21.15.42:0/3388942110 wait complete. 2024-09-16T03:51:20.273 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:20.265+0000 7f78a908c640 1 librados: shutdown 2024-09-16T03:51:23.290 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:51:23.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:23.417+0000 7f2eaff5d640 10 monclient: get_monmap_and_config 2024-09-16T03:51:23.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:23.417+0000 7f2eaff5d640 10 monclient: build_initial_monmap 2024-09-16T03:51:23.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:23.417+0000 7f2eaff5d640 10 monclient: monmap: 2024-09-16T03:51:23.423 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:51:23.423 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:23.423 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:51:23.421804+0000 2024-09-16T03:51:23.423 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:51:23.421804+0000 2024-09-16T03:51:23.424 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:51:23.424 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:23.424 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:51:23.424 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:51:23.424 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:51:23.424 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:51:23.424 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:51:23.424 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:23.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:23.417+0000 7f2eaff5d640 1 Processor -- start 2024-09-16T03:51:23.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:23.417+0000 7f2eaff5d640 1 -- start start 2024-09-16T03:51:23.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:23.417+0000 7f2eaff5d640 10 monclient: init 2024-09-16T03:51:23.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:23.417+0000 7f2eaff5d640 10 monclient: _reopen_session rank -1 2024-09-16T03:51:23.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:23.417+0000 7f2eaff5d640 10 monclient: _add_conns ranks=[4,1,2,0,3] 2024-09-16T03:51:23.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:23.417+0000 7f2eaff5d640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f2ea80a1330 0x7f2ea80a1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:23.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:23.417+0000 7f2eaff5d640 10 monclient(hunting): picked mon.noname-c con 0x7f2ea80a1330 addr v2:172.21.15.154:3300/0 2024-09-16T03:51:23.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:23.417+0000 7f2eaff5d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2ea806dac0 addr v1:172.21.15.154:6789/0 2024-09-16T03:51:23.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:23.417+0000 7f2eaff5d640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2ea80a1c50 0x7f2ea8135810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:23.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:23.417+0000 7f2eaff5d640 10 monclient(hunting): picked mon.noname-b con 0x7f2ea80a1c50 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:51:23.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:23.417+0000 7f2eaff5d640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2ea809c090 con 0x7f2ea806dac0 2024-09-16T03:51:23.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:23.417+0000 7f2eaff5d640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:23.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:23.417+0000 7f2eaff5d640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f2ea809f610 con 0x7f2ea80a1330 2024-09-16T03:51:23.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:23.417+0000 7f2eaff5d640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:23.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:23.417+0000 7f2eaff5d640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f2ea809f8f0 con 0x7f2ea80a1c50 2024-09-16T03:51:23.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:23.417+0000 7f2eaff5d640 10 monclient(hunting): _renew_subs 2024-09-16T03:51:23.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:23.417+0000 7f2eaff5d640 10 monclient(hunting): authenticate will time out at 1530.391249s 2024-09-16T03:51:23.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:23.417+0000 7f2eae4fa640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2ea80a1c50 0x7f2ea8135810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:23.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:23.417+0000 7f2eae4fa640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2ea80a1c50 0x7f2ea8135810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:53622/0 (socket says 172.21.15.42:53622) 2024-09-16T03:51:23.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:23.417+0000 7f2eae4fa640 1 -- 172.21.15.42:0/1791778748 learned_addr learned my addr 172.21.15.42:0/1791778748 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:51:23.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:23.417+0000 7f2eae4fa640 10 monclient(hunting): get_auth_request con 0x7f2ea80a1c50 auth_method 0 2024-09-16T03:51:23.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:23.417+0000 7f2eae4fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:23.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:23.417+0000 7f2eae4fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:23.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:23.417+0000 7f2eae4fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:51:23.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:23.417+0000 7f2eae4fa640 1 -- 172.21.15.42:0/1791778748 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2ea80a1c50 msgr2=0x7f2ea8135810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T03:51:23.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:23.417+0000 7f2eae4fa640 1 -- 172.21.15.42:0/1791778748 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2ea80a1c50 msgr2=0x7f2ea8135810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:51:23.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:23.417+0000 7f2eae4fa640 1 --2- 172.21.15.42:0/1791778748 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2ea80a1c50 0x7f2ea8135810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:51:23.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:23.417+0000 7f2eadcf9640 1 --2- 172.21.15.42:0/1791778748 >> v2:172.21.15.154:3300/0 conn(0x7f2ea80a1330 0x7f2ea80a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:23.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:23.417+0000 7f2eae4fa640 1 --2- 172.21.15.42:0/1791778748 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2ea80a1c50 0x7f2ea8135810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:51:23.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:23.417+0000 7f2eadcf9640 10 monclient(hunting): get_auth_request con 0x7f2ea80a1330 auth_method 0 2024-09-16T03:51:23.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:23.417+0000 7f2eadcf9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:23.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:23.417+0000 7f2eadcf9640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:23.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:23.417+0000 7f2eadcf9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:51:23.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:23.417+0000 7f2eadcf9640 1 -- 172.21.15.42:0/1791778748 >> v2:172.21.15.154:3300/0 conn(0x7f2ea80a1330 msgr2=0x7f2ea80a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T03:51:23.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:23.417+0000 7f2eadcf9640 1 -- 172.21.15.42:0/1791778748 >> v2:172.21.15.154:3300/0 conn(0x7f2ea80a1330 msgr2=0x7f2ea80a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:51:23.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:23.417+0000 7f2eadcf9640 1 --2- 172.21.15.42:0/1791778748 >> v2:172.21.15.154:3300/0 conn(0x7f2ea80a1330 0x7f2ea80a1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:51:23.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:23.421+0000 7f2eadcf9640 1 --2- 172.21.15.42:0/1791778748 >> v2:172.21.15.154:3300/0 conn(0x7f2ea80a1330 0x7f2ea80a1710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:51:23.625 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:23.621+0000 7f2eae4fa640 1 --2- 172.21.15.42:0/1791778748 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2ea80a1c50 0x7f2ea8135810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:23.626 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:23.621+0000 7f2eae4fa640 10 monclient(hunting): get_auth_request con 0x7f2ea80a1c50 auth_method 0 2024-09-16T03:51:23.626 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:23.621+0000 7f2eae4fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:23.626 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:23.621+0000 7f2eae4fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:23.626 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:23.621+0000 7f2eae4fa640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:51:23.626 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:23.621+0000 7f2eae4fa640 1 -- 172.21.15.42:0/1791778748 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2ea80a1c50 msgr2=0x7f2ea8135810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T03:51:23.626 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:23.621+0000 7f2eae4fa640 1 -- 172.21.15.42:0/1791778748 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2ea80a1c50 msgr2=0x7f2ea8135810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:51:23.626 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:23.621+0000 7f2eae4fa640 1 --2- 172.21.15.42:0/1791778748 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2ea80a1c50 0x7f2ea8135810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:51:23.626 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:23.621+0000 7f2eae4fa640 1 --2- 172.21.15.42:0/1791778748 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2ea80a1c50 0x7f2ea8135810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T03:51:23.630 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:23.625+0000 7f2eadcf9640 1 --2- 172.21.15.42:0/1791778748 >> v2:172.21.15.154:3300/0 conn(0x7f2ea80a1330 0x7f2ea80a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:23.630 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:23.625+0000 7f2eadcf9640 10 monclient(hunting): get_auth_request con 0x7f2ea80a1330 auth_method 0 2024-09-16T03:51:23.630 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:23.625+0000 7f2eadcf9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:23.630 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:23.625+0000 7f2eadcf9640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:23.630 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:23.625+0000 7f2eadcf9640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:51:23.630 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:23.625+0000 7f2eadcf9640 1 -- 172.21.15.42:0/1791778748 >> v2:172.21.15.154:3300/0 conn(0x7f2ea80a1330 msgr2=0x7f2ea80a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T03:51:23.630 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:23.625+0000 7f2eadcf9640 1 -- 172.21.15.42:0/1791778748 >> v2:172.21.15.154:3300/0 conn(0x7f2ea80a1330 msgr2=0x7f2ea80a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:51:23.630 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:23.625+0000 7f2eadcf9640 1 --2- 172.21.15.42:0/1791778748 >> v2:172.21.15.154:3300/0 conn(0x7f2ea80a1330 0x7f2ea80a1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:51:23.630 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:23.625+0000 7f2eadcf9640 1 --2- 172.21.15.42:0/1791778748 >> v2:172.21.15.154:3300/0 conn(0x7f2ea80a1330 0x7f2ea80a1710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T03:51:24.029 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:24.025+0000 7f2eae4fa640 1 --2- 172.21.15.42:0/1791778748 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2ea80a1c50 0x7f2ea8135810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:24.029 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:24.025+0000 7f2eae4fa640 10 monclient(hunting): get_auth_request con 0x7f2ea80a1c50 auth_method 0 2024-09-16T03:51:24.030 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:24.025+0000 7f2eae4fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:24.030 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:24.025+0000 7f2eae4fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:24.030 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:24.025+0000 7f2eae4fa640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:51:24.030 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:24.025+0000 7f2eae4fa640 1 -- 172.21.15.42:0/1791778748 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2ea80a1c50 msgr2=0x7f2ea8135810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T03:51:24.030 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:24.025+0000 7f2eae4fa640 1 -- 172.21.15.42:0/1791778748 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2ea80a1c50 msgr2=0x7f2ea8135810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:51:24.030 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:24.025+0000 7f2eae4fa640 1 --2- 172.21.15.42:0/1791778748 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2ea80a1c50 0x7f2ea8135810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:51:24.030 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:24.025+0000 7f2eae4fa640 1 --2- 172.21.15.42:0/1791778748 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2ea80a1c50 0x7f2ea8135810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T03:51:24.034 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:24.029+0000 7f2eadcf9640 1 --2- 172.21.15.42:0/1791778748 >> v2:172.21.15.154:3300/0 conn(0x7f2ea80a1330 0x7f2ea80a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:24.034 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:24.029+0000 7f2eadcf9640 10 monclient(hunting): get_auth_request con 0x7f2ea80a1330 auth_method 0 2024-09-16T03:51:24.034 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:24.029+0000 7f2eadcf9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:24.034 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:24.029+0000 7f2eadcf9640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:24.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:24.029+0000 7f2eadcf9640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:51:24.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:24.029+0000 7f2eadcf9640 1 -- 172.21.15.42:0/1791778748 >> v2:172.21.15.154:3300/0 conn(0x7f2ea80a1330 msgr2=0x7f2ea80a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T03:51:24.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:24.029+0000 7f2eadcf9640 1 -- 172.21.15.42:0/1791778748 >> v2:172.21.15.154:3300/0 conn(0x7f2ea80a1330 msgr2=0x7f2ea80a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:51:24.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:24.029+0000 7f2eadcf9640 1 --2- 172.21.15.42:0/1791778748 >> v2:172.21.15.154:3300/0 conn(0x7f2ea80a1330 0x7f2ea80a1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:51:24.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:24.029+0000 7f2eadcf9640 1 --2- 172.21.15.42:0/1791778748 >> v2:172.21.15.154:3300/0 conn(0x7f2ea80a1330 0x7f2ea80a1710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T03:51:24.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:24.829+0000 7f2eae4fa640 1 --2- 172.21.15.42:0/1791778748 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2ea80a1c50 0x7f2ea8135810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:24.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:24.829+0000 7f2eae4fa640 10 monclient(hunting): get_auth_request con 0x7f2ea80a1c50 auth_method 0 2024-09-16T03:51:24.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:24.829+0000 7f2eae4fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:24.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:24.829+0000 7f2eae4fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:24.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:24.829+0000 7f2eae4fa640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:51:24.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:24.829+0000 7f2eae4fa640 1 -- 172.21.15.42:0/1791778748 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2ea80a1c50 msgr2=0x7f2ea8135810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T03:51:24.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:24.829+0000 7f2eae4fa640 1 -- 172.21.15.42:0/1791778748 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2ea80a1c50 msgr2=0x7f2ea8135810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:51:24.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:24.829+0000 7f2eae4fa640 1 --2- 172.21.15.42:0/1791778748 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2ea80a1c50 0x7f2ea8135810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:51:24.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:24.829+0000 7f2eae4fa640 1 --2- 172.21.15.42:0/1791778748 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2ea80a1c50 0x7f2ea8135810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T03:51:24.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:24.833+0000 7f2eadcf9640 1 --2- 172.21.15.42:0/1791778748 >> v2:172.21.15.154:3300/0 conn(0x7f2ea80a1330 0x7f2ea80a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:24.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:24.833+0000 7f2eadcf9640 10 monclient(hunting): get_auth_request con 0x7f2ea80a1330 auth_method 0 2024-09-16T03:51:24.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:24.833+0000 7f2eadcf9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:24.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:24.833+0000 7f2eadcf9640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:24.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:24.833+0000 7f2eadcf9640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:51:24.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:24.833+0000 7f2eadcf9640 1 -- 172.21.15.42:0/1791778748 >> v2:172.21.15.154:3300/0 conn(0x7f2ea80a1330 msgr2=0x7f2ea80a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T03:51:24.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:24.833+0000 7f2eadcf9640 1 -- 172.21.15.42:0/1791778748 >> v2:172.21.15.154:3300/0 conn(0x7f2ea80a1330 msgr2=0x7f2ea80a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:51:24.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:24.833+0000 7f2eadcf9640 1 --2- 172.21.15.42:0/1791778748 >> v2:172.21.15.154:3300/0 conn(0x7f2ea80a1330 0x7f2ea80a1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:51:24.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:24.833+0000 7f2eadcf9640 1 --2- 172.21.15.42:0/1791778748 >> v2:172.21.15.154:3300/0 conn(0x7f2ea80a1330 0x7f2ea80a1710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T03:51:26.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:26.417+0000 7f2e9f7fe640 10 monclient(hunting): tick 2024-09-16T03:51:26.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:26.417+0000 7f2e9f7fe640 10 monclient(hunting): _check_auth_tickets 2024-09-16T03:51:26.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:26.417+0000 7f2e9f7fe640 1 monclient(hunting): continuing hunt 2024-09-16T03:51:26.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:26.417+0000 7f2e9f7fe640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T03:51:26.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:26.417+0000 7f2e9f7fe640 1 -- 172.21.15.42:0/1791778748 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2ea80a1c50 msgr2=0x7f2ea8135810 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:51:26.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:26.417+0000 7f2e9f7fe640 1 --2- 172.21.15.42:0/1791778748 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2ea80a1c50 0x7f2ea8135810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:26.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:26.417+0000 7f2e9f7fe640 1 -- 172.21.15.42:0/1791778748 >> v2:172.21.15.154:3300/0 conn(0x7f2ea80a1330 msgr2=0x7f2ea80a1710 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:51:26.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:26.417+0000 7f2e9f7fe640 1 --2- 172.21.15.42:0/1791778748 >> v2:172.21.15.154:3300/0 conn(0x7f2ea80a1330 0x7f2ea80a1710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:26.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:26.417+0000 7f2e9f7fe640 1 -- 172.21.15.42:0/1791778748 >> v1:172.21.15.154:6789/0 conn(0x7f2ea806dac0 legacy=0x7f2ea8135100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:26.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:26.417+0000 7f2e9f7fe640 10 monclient: _add_conns ranks=[0,3] 2024-09-16T03:51:26.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:26.417+0000 7f2e9f7fe640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2e8c0036d0 addr v1:172.21.15.42:6789/0 2024-09-16T03:51:26.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:26.417+0000 7f2e9f7fe640 1 --2- 172.21.15.42:0/1791778748 >> v2:172.21.15.42:3300/0 conn(0x7f2e8c007380 0x7f2e8c009820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:26.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:26.417+0000 7f2e9f7fe640 10 monclient(hunting): picked mon.noname-a con 0x7f2e8c007380 addr v2:172.21.15.42:3300/0 2024-09-16T03:51:26.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:26.417+0000 7f2e9f7fe640 1 -- 172.21.15.42:0/1791778748 --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2ea809fbd0 con 0x7f2e8c0036d0 2024-09-16T03:51:26.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:26.417+0000 7f2e9f7fe640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:26.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:26.417+0000 7f2e9f7fe640 1 -- 172.21.15.42:0/1791778748 --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f2ea809f610 con 0x7f2e8c007380 2024-09-16T03:51:26.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:26.417+0000 7f2e9f7fe640 10 monclient(hunting): _renew_subs 2024-09-16T03:51:26.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:26.417+0000 7f2ead4f8640 1 -- 172.21.15.42:0/1791778748 >> v2:172.21.15.42:3300/0 conn(0x7f2e8c007380 msgr2=0x7f2e8c009820 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3300/0 2024-09-16T03:51:26.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:26.417+0000 7f2eadcf9640 1 -- 172.21.15.42:0/1791778748 >> v1:172.21.15.42:6789/0 conn(0x7f2e8c0036d0 legacy=0x7f2e8c005b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.42:6789/0 2024-09-16T03:51:26.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:26.417+0000 7f2ead4f8640 1 --2- 172.21.15.42:0/1791778748 >> v2:172.21.15.42:3300/0 conn(0x7f2e8c007380 0x7f2e8c009820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:51:26.625 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:26.621+0000 7f2eadcf9640 1 -- 172.21.15.42:0/1791778748 >> v1:172.21.15.42:6789/0 conn(0x7f2e8c0036d0 legacy=0x7f2e8c005b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.42:6789/0 2024-09-16T03:51:26.625 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:26.621+0000 7f2ead4f8640 1 -- 172.21.15.42:0/1791778748 >> v2:172.21.15.42:3300/0 conn(0x7f2e8c007380 msgr2=0x7f2e8c009820 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3300/0 2024-09-16T03:51:26.625 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:26.621+0000 7f2ead4f8640 1 --2- 172.21.15.42:0/1791778748 >> v2:172.21.15.42:3300/0 conn(0x7f2e8c007380 0x7f2e8c009820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T03:51:27.029 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:27.025+0000 7f2ead4f8640 1 -- 172.21.15.42:0/1791778748 >> v2:172.21.15.42:3300/0 conn(0x7f2e8c007380 msgr2=0x7f2e8c009820 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3300/0 2024-09-16T03:51:27.029 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:27.025+0000 7f2eadcf9640 1 -- 172.21.15.42:0/1791778748 >> v1:172.21.15.42:6789/0 conn(0x7f2e8c0036d0 legacy=0x7f2e8c005b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.42:6789/0 2024-09-16T03:51:27.029 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:27.025+0000 7f2ead4f8640 1 --2- 172.21.15.42:0/1791778748 >> v2:172.21.15.42:3300/0 conn(0x7f2e8c007380 0x7f2e8c009820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T03:51:27.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:27.829+0000 7f2eadcf9640 1 -- 172.21.15.42:0/1791778748 >> v1:172.21.15.42:6789/0 conn(0x7f2e8c0036d0 legacy=0x7f2e8c005b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.42:6789/0 2024-09-16T03:51:27.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:27.829+0000 7f2ead4f8640 1 -- 172.21.15.42:0/1791778748 >> v2:172.21.15.42:3300/0 conn(0x7f2e8c007380 msgr2=0x7f2e8c009820 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3300/0 2024-09-16T03:51:27.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:27.829+0000 7f2ead4f8640 1 --2- 172.21.15.42:0/1791778748 >> v2:172.21.15.42:3300/0 conn(0x7f2e8c007380 0x7f2e8c009820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T03:51:29.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.417+0000 7f2e9f7fe640 10 monclient(hunting): tick 2024-09-16T03:51:29.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.417+0000 7f2e9f7fe640 10 monclient(hunting): _check_auth_tickets 2024-09-16T03:51:29.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.417+0000 7f2e9f7fe640 1 monclient(hunting): continuing hunt 2024-09-16T03:51:29.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.417+0000 7f2e9f7fe640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T03:51:29.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.417+0000 7f2e9f7fe640 1 -- 172.21.15.42:0/1791778748 >> v2:172.21.15.42:3300/0 conn(0x7f2e8c007380 msgr2=0x7f2e8c009820 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:51:29.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.417+0000 7f2e9f7fe640 1 --2- 172.21.15.42:0/1791778748 >> v2:172.21.15.42:3300/0 conn(0x7f2e8c007380 0x7f2e8c009820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:29.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.417+0000 7f2e9f7fe640 1 -- 172.21.15.42:0/1791778748 >> v1:172.21.15.42:6789/0 conn(0x7f2e8c0036d0 legacy=0x7f2e8c005b50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:51:29.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.417+0000 7f2e9f7fe640 10 monclient: _add_conns ranks=[2,1,3,0,4] 2024-09-16T03:51:29.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.417+0000 7f2eae4fa640 1 -- 172.21.15.42:0/1791778748 reap_dead start 2024-09-16T03:51:29.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.417+0000 7f2e9f7fe640 1 --2- 172.21.15.42:0/1791778748 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2ea80a1c50 0x7f2e8c009f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:29.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.417+0000 7f2e9f7fe640 10 monclient(hunting): picked mon.noname-b con 0x7f2ea80a1c50 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:51:29.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.417+0000 7f2e9f7fe640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2e8c00a460 addr v1:172.21.15.154:6789/0 2024-09-16T03:51:29.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.417+0000 7f2e9f7fe640 1 --2- 172.21.15.42:0/1791778748 >> v2:172.21.15.42:3300/0 conn(0x7f2e8c007380 0x7f2e8c009820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:29.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.417+0000 7f2e9f7fe640 10 monclient(hunting): picked mon.noname-a con 0x7f2e8c007380 addr v2:172.21.15.42:3300/0 2024-09-16T03:51:29.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.417+0000 7f2e9f7fe640 1 -- 172.21.15.42:0/1791778748 --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2e8c00af90 con 0x7f2e8c00a460 2024-09-16T03:51:29.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.417+0000 7f2e9f7fe640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:29.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.417+0000 7f2e9f7fe640 1 -- 172.21.15.42:0/1791778748 --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f2ea809f610 con 0x7f2e8c007380 2024-09-16T03:51:29.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.417+0000 7f2e9f7fe640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:29.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.417+0000 7f2e9f7fe640 1 -- 172.21.15.42:0/1791778748 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f2ea809f8f0 con 0x7f2ea80a1c50 2024-09-16T03:51:29.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.417+0000 7f2e9f7fe640 10 monclient(hunting): _renew_subs 2024-09-16T03:51:29.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.421+0000 7f2eae4fa640 1 -- 172.21.15.42:0/1791778748 >> v2:172.21.15.42:3300/0 conn(0x7f2e8c007380 msgr2=0x7f2e8c009820 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3300/0 2024-09-16T03:51:29.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.421+0000 7f2eae4fa640 1 --2- 172.21.15.42:0/1791778748 >> v2:172.21.15.42:3300/0 conn(0x7f2e8c007380 0x7f2e8c009820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:51:29.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.421+0000 7f2eadcf9640 1 --2- 172.21.15.42:0/1791778748 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2ea80a1c50 0x7f2e8c009f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:29.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.421+0000 7f2eadcf9640 10 monclient(hunting): get_auth_request con 0x7f2ea80a1c50 auth_method 0 2024-09-16T03:51:29.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.421+0000 7f2eadcf9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:29.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.421+0000 7f2eadcf9640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:29.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.421+0000 7f2eadcf9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:51:29.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.421+0000 7f2eadcf9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:51:29.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.421+0000 7f2eadcf9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:51:29.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.421+0000 7f2eadcf9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:51:29.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.421+0000 7f2eadcf9640 10 monclient(hunting): handle_auth_done global_id 814169 payload 274 2024-09-16T03:51:29.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.421+0000 7f2eadcf9640 1 -- 172.21.15.42:0/1791778748 >> v2:172.21.15.42:3300/0 conn(0x7f2e8c007380 msgr2=0x7f2e8c009820 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:51:29.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.421+0000 7f2eadcf9640 1 --2- 172.21.15.42:0/1791778748 >> v2:172.21.15.42:3300/0 conn(0x7f2e8c007380 0x7f2e8c009820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:29.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.421+0000 7f2eadcf9640 1 -- 172.21.15.42:0/1791778748 >> v1:172.21.15.154:6789/0 conn(0x7f2e8c00a460 legacy=0x7f2e8c00f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:29.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.421+0000 7f2eaccf7640 1 -- 172.21.15.42:0/1791778748 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 694454652 0 0) 0x7f2e8c00af90 con 0x7f2e8c00a460 2024-09-16T03:51:29.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.421+0000 7f2eadcf9640 10 monclient: _finish_hunting 0 2024-09-16T03:51:29.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.421+0000 7f2eadcf9640 1 monclient: found mon.noname-b 2024-09-16T03:51:29.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.421+0000 7f2eadcf9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:51:29.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.421+0000 7f2eadcf9640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T03:51:29.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.421+0000 7f2eadcf9640 1 -- 172.21.15.42:0/1791778748 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ea809fbd0 con 0x7f2ea80a1c50 2024-09-16T03:51:29.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.421+0000 7f2eaccf7640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T03:51:29.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.425+0000 7f2eadcf9640 1 --2- 172.21.15.42:0/1791778748 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2ea80a1c50 0x7f2e8c009f20 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f2ea4000ee0 tx=0x7f2ea401d2f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=efc7160fafda1711 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:51:29.432 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.425+0000 7f2eaccf7640 1 -- 172.21.15.42:0/1791778748 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ea400f040 con 0x7f2ea80a1c50 2024-09-16T03:51:29.432 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.425+0000 7f2eaccf7640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:51:29.432 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.425+0000 7f2eaccf7640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:51:29.432 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.425+0000 7f2eaccf7640 10 monclient: dump: 2024-09-16T03:51:29.433 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:51:29.433 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:29.433 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:29.433 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:29.433 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:51:29.433 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:29.433 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:51:29.434 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:51:29.434 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:51:29.434 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:29.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.425+0000 7f2eaccf7640 10 monclient: _finish_auth 0 2024-09-16T03:51:29.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.425+0000 7f2eaccf7640 10 monclient: _check_auth_tickets 2024-09-16T03:51:29.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.425+0000 7f2eaccf7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:51:29.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.425+0000 7f2eaff5d640 5 monclient: authenticate success, global_id 814169 2024-09-16T03:51:29.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.425+0000 7f2eaff5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:51:29.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.425+0000 7f2eaccf7640 1 -- 172.21.15.42:0/1791778748 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ea40af420 con 0x7f2ea80a1c50 2024-09-16T03:51:29.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.425+0000 7f2eaccf7640 10 monclient: handle_config config(0 keys) 2024-09-16T03:51:29.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.425+0000 7f2eaff5d640 10 monclient: get_monmap_and_config success 2024-09-16T03:51:29.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.425+0000 7f2eaff5d640 4 set_mon_vals no callback set 2024-09-16T03:51:29.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.425+0000 7f2eaccf7640 1 -- 172.21.15.42:0/1791778748 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ea40af810 con 0x7f2ea80a1c50 2024-09-16T03:51:29.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.425+0000 7f2eaccf7640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:51:29.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.425+0000 7f2eaccf7640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:51:29.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.425+0000 7f2eaccf7640 10 monclient: dump: 2024-09-16T03:51:29.435 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:51:29.435 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:29.435 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:29.435 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:29.435 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:51:29.435 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:29.435 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:51:29.436 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:51:29.436 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:51:29.436 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:29.442 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.429+0000 7f2eaff5d640 10 monclient: shutdown 2024-09-16T03:51:29.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.429+0000 7f2eaff5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:51:29.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.429+0000 7f2eaff5d640 1 -- 172.21.15.42:0/1791778748 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2ea80a1c50 msgr2=0x7f2e8c009f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:29.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.429+0000 7f2eaff5d640 1 --2- 172.21.15.42:0/1791778748 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2ea80a1c50 0x7f2e8c009f20 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f2ea4000ee0 tx=0x7f2ea401d2f0 comp rx=0 tx=0).stop 2024-09-16T03:51:29.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.429+0000 7f2eaff5d640 1 -- 172.21.15.42:0/1791778748 shutdown_connections 2024-09-16T03:51:29.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.429+0000 7f2eaff5d640 1 --2- 172.21.15.42:0/1791778748 >> v2:172.21.15.42:3300/0 conn(0x7f2e8c007380 0x7f2e8c009820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:29.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.429+0000 7f2eaff5d640 1 --2- 172.21.15.42:0/1791778748 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2ea80a1c50 0x7f2e8c009f20 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:29.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.429+0000 7f2eaff5d640 1 -- 172.21.15.42:0/1791778748 >> 172.21.15.42:0/1791778748 conn(0x7f2ea80a2420 msgr2=0x7f2ea80a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:51:29.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.429+0000 7f2eaff5d640 1 -- 172.21.15.42:0/1791778748 shutdown_connections 2024-09-16T03:51:29.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.429+0000 7f2eaff5d640 1 -- 172.21.15.42:0/1791778748 wait complete. 2024-09-16T03:51:29.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.429+0000 7f2eaff5d640 10 monclient: build_initial_monmap 2024-09-16T03:51:29.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.429+0000 7f2eaff5d640 10 monclient: monmap: 2024-09-16T03:51:29.443 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:51:29.443 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:51:29.444 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:51:29.444 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:51:29.444 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:51:29.444 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:29.444 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:51:29.444 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:51:29.444 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:51:29.444 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:29.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.429+0000 7f2eaff5d640 1 librados: starting msgr at 2024-09-16T03:51:29.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.429+0000 7f2eaff5d640 1 librados: starting objecter 2024-09-16T03:51:29.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.429+0000 7f2eaff5d640 1 Processor -- start 2024-09-16T03:51:29.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.429+0000 7f2eaff5d640 1 -- start start 2024-09-16T03:51:29.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.429+0000 7f2eaff5d640 1 librados: setting wanted keys 2024-09-16T03:51:29.445 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.429+0000 7f2eaff5d640 1 librados: calling monclient init 2024-09-16T03:51:29.445 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.429+0000 7f2eaff5d640 10 monclient: init 2024-09-16T03:51:29.445 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.433+0000 7f2eaff5d640 10 monclient: _reopen_session rank -1 2024-09-16T03:51:29.445 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.433+0000 7f2eaff5d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:51:29.445 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.433+0000 7f2eaff5d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2e8c007380 0x7f2ea81cf3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:29.445 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.433+0000 7f2eaff5d640 10 monclient(hunting): picked mon.noname-b con 0x7f2e8c007380 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:51:29.445 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.433+0000 7f2eaff5d640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2e8c00a460 0x7f2ea81cf900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:29.445 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.433+0000 7f2eaff5d640 10 monclient(hunting): picked mon.noname-a con 0x7f2e8c00a460 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:51:29.445 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.433+0000 7f2eaff5d640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2ea80a1c50 0x7f2ea8123ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:29.445 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.433+0000 7f2eaff5d640 10 monclient(hunting): picked mon.noname-c con 0x7f2ea80a1c50 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:51:29.445 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.433+0000 7f2eaff5d640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:29.445 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.433+0000 7f2eaff5d640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f2ea8135910 con 0x7f2e8c00a460 2024-09-16T03:51:29.445 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.433+0000 7f2eaff5d640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:29.445 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.433+0000 7f2eaff5d640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f2ea809ef80 con 0x7f2e8c007380 2024-09-16T03:51:29.446 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.433+0000 7f2eaff5d640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:29.446 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.433+0000 7f2eaff5d640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f2ea8135a90 con 0x7f2ea80a1c50 2024-09-16T03:51:29.446 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.433+0000 7f2eaff5d640 10 monclient(hunting): _renew_subs 2024-09-16T03:51:29.446 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.433+0000 7f2eaff5d640 10 monclient(hunting): authenticate will time out at 1536.404287s 2024-09-16T03:51:29.446 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.433+0000 7f2ead4f8640 1 -- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2e8c00a460 msgr2=0x7f2ea81cf900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.42:3300/0 2024-09-16T03:51:29.446 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.433+0000 7f2ead4f8640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2e8c00a460 0x7f2ea81cf900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:51:29.446 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.433+0000 7f2eae4fa640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2ea80a1c50 0x7f2ea8123ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:29.446 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.433+0000 7f2eae4fa640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2ea80a1c50 0x7f2ea8123ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:41760/0 (socket says 172.21.15.42:41760) 2024-09-16T03:51:29.446 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.433+0000 7f2eae4fa640 1 -- 172.21.15.42:0/3545756026 learned_addr learned my addr 172.21.15.42:0/3545756026 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:51:29.446 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.433+0000 7f2eae4fa640 10 monclient(hunting): get_auth_request con 0x7f2ea80a1c50 auth_method 0 2024-09-16T03:51:29.446 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.433+0000 7f2eae4fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:29.446 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.433+0000 7f2eae4fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:29.446 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.433+0000 7f2eae4fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:51:29.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.433+0000 7f2eae4fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:51:29.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.433+0000 7f2eae4fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:51:29.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.433+0000 7f2eadcf9640 1 --2- 172.21.15.42:0/3545756026 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2e8c007380 0x7f2ea81cf3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:29.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.433+0000 7f2eae4fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:51:29.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.433+0000 7f2eadcf9640 10 monclient(hunting): get_auth_request con 0x7f2e8c007380 auth_method 0 2024-09-16T03:51:29.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.433+0000 7f2eadcf9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:29.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.433+0000 7f2eadcf9640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:29.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.433+0000 7f2eadcf9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:51:29.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.433+0000 7f2eae4fa640 10 monclient(hunting): handle_auth_done global_id 814172 payload 824 2024-09-16T03:51:29.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.433+0000 7f2eae4fa640 1 -- 172.21.15.42:0/3545756026 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2e8c007380 msgr2=0x7f2ea81cf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:29.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.433+0000 7f2eae4fa640 1 --2- 172.21.15.42:0/3545756026 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2e8c007380 0x7f2ea81cf3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:29.448 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.433+0000 7f2eae4fa640 1 -- 172.21.15.42:0/3545756026 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2e8c00a460 msgr2=0x7f2ea81cf900 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:51:29.448 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.433+0000 7f2eae4fa640 1 --2- 172.21.15.42:0/3545756026 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2e8c00a460 0x7f2ea81cf900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:29.448 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.433+0000 7f2eae4fa640 10 monclient: _finish_hunting 0 2024-09-16T03:51:29.448 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.433+0000 7f2eae4fa640 1 monclient: found mon.noname-c 2024-09-16T03:51:29.448 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.433+0000 7f2eae4fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:51:29.448 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.433+0000 7f2eae4fa640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:51:29.448 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.433+0000 7f2eae4fa640 1 -- 172.21.15.42:0/3545756026 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ea8127980 con 0x7f2ea80a1c50 2024-09-16T03:51:29.448 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.433+0000 7f2eadcf9640 1 --2- 172.21.15.42:0/3545756026 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2e8c007380 0x7f2ea81cf3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T03:51:29.448 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.433+0000 7f2eae4fa640 1 --2- 172.21.15.42:0/3545756026 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2ea80a1c50 0x7f2ea8123ad0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f2ea8068f60 tx=0x7f2ea0007e60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:51:29.448 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.433+0000 7f2e9e7fc640 10 client.?.objecter ms_handle_connect 0x7f2ea80a1c50 2024-09-16T03:51:29.448 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.433+0000 7f2e9e7fc640 10 client.?.objecter resend_mon_ops 2024-09-16T03:51:29.448 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.433+0000 7f2e9e7fc640 1 -- 172.21.15.42:0/3545756026 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ea001a070 con 0x7f2ea80a1c50 2024-09-16T03:51:29.448 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.433+0000 7f2e9e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:51:29.449 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.433+0000 7f2e9e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:51:29.449 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.433+0000 7f2e9e7fc640 10 monclient: dump: 2024-09-16T03:51:29.449 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:51:29.449 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:29.449 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:29.449 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:29.449 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:51:29.449 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:29.450 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:51:29.450 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:51:29.450 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:51:29.450 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:29.450 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.433+0000 7f2e9e7fc640 10 monclient: _finish_auth 0 2024-09-16T03:51:29.450 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.433+0000 7f2e9e7fc640 10 monclient: _check_auth_tickets 2024-09-16T03:51:29.450 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.433+0000 7f2e9e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:51:29.450 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.433+0000 7f2e9e7fc640 1 -- 172.21.15.42:0/3545756026 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ea000b500 con 0x7f2ea80a1c50 2024-09-16T03:51:29.450 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.433+0000 7f2eaff5d640 5 monclient: authenticate success, global_id 814172 2024-09-16T03:51:29.450 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.433+0000 7f2eaff5d640 10 monclient: _renew_subs 2024-09-16T03:51:29.450 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.433+0000 7f2eaff5d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:51:29.450 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.433+0000 7f2eaff5d640 1 -- 172.21.15.42:0/3545756026 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ea809d360 con 0x7f2ea80a1c50 2024-09-16T03:51:29.450 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.433+0000 7f2e9e7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:51:29.450 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.433+0000 7f2e9e7fc640 1 -- 172.21.15.42:0/3545756026 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ea0012a90 con 0x7f2ea80a1c50 2024-09-16T03:51:29.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.433+0000 7f2e9e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:51:29.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.433+0000 7f2e9e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:51:29.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.433+0000 7f2e9e7fc640 10 monclient: dump: 2024-09-16T03:51:29.451 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:51:29.451 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:29.451 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:29.451 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:29.451 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:51:29.451 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:29.451 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:51:29.451 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:51:29.451 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:51:29.451 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:29.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.433+0000 7f2e9ffff640 4 set_mon_vals no callback set 2024-09-16T03:51:29.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.433+0000 7f2eaff5d640 10 client.814172.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:51:29.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.433+0000 7f2eaff5d640 10 monclient: _renew_subs 2024-09-16T03:51:29.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.433+0000 7f2eaff5d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:51:29.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.433+0000 7f2eaff5d640 1 -- 172.21.15.42:0/3545756026 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ea8134210 con 0x7f2ea80a1c50 2024-09-16T03:51:29.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.433+0000 7f2eaff5d640 1 librados: init done 2024-09-16T03:51:29.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.433+0000 7f2eaff5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:51:29.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.433+0000 7f2eaff5d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:51:29.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.433+0000 7f2eaff5d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:51:29.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.433+0000 7f2eaff5d640 1 -- 172.21.15.42:0/3545756026 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ea809ce90 con 0x7f2ea80a1c50 2024-09-16T03:51:29.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.433+0000 7f2e9e7fc640 1 -- 172.21.15.42:0/3545756026 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f2ea001d070 con 0x7f2ea80a1c50 2024-09-16T03:51:29.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.433+0000 7f2e9e7fc640 1 --2- 172.21.15.42:0/3545756026 >> v2:172.21.15.154:6812/254845342 conn(0x7f2e8c038730 0x7f2e8c03abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:29.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.433+0000 7f2e9e7fc640 1 -- 172.21.15.42:0/3545756026 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(311..311 src has 1..311) ==== 5731+0+0 (secure 0 0 0) 0x7f2ea0049870 con 0x7f2ea80a1c50 2024-09-16T03:51:29.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.433+0000 7f2e9e7fc640 10 client.814172.objecter ms_dispatch 0x7f2ea8000ba0 osd_map(311..311 src has 1..311) 2024-09-16T03:51:29.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.433+0000 7f2e9e7fc640 3 client.814172.objecter handle_osd_map got epochs [311,311] > 0 2024-09-16T03:51:29.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.433+0000 7f2e9e7fc640 3 client.814172.objecter handle_osd_map decoding full epoch 311 2024-09-16T03:51:29.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.433+0000 7f2e9e7fc640 20 client.814172.objecter dump_active .. 0 homeless 2024-09-16T03:51:29.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.433+0000 7f2eadcf9640 1 --2- 172.21.15.42:0/3545756026 >> v2:172.21.15.154:6812/254845342 conn(0x7f2e8c038730 0x7f2e8c03abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:29.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.433+0000 7f2eadcf9640 10 monclient: get_auth_request con 0x7f2e8c038730 auth_method 0 2024-09-16T03:51:29.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.437+0000 7f2eadcf9640 1 --2- 172.21.15.42:0/3545756026 >> v2:172.21.15.154:6812/254845342 conn(0x7f2e8c038730 0x7f2e8c03abf0 secure :-1 s=READY pgs=3562 cs=0 l=1 rev1=1 crypto rx=0x7f2ea40af000 tx=0x7f2ea4010040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:51:29.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.437+0000 7f2e9e7fc640 10 client.814172.objecter ms_handle_connect 0x7f2e8c038730 2024-09-16T03:51:29.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.441+0000 7f2e9e7fc640 1 -- 172.21.15.42:0/3545756026 <== mon.2 v2:172.21.15.42:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f2ea0024030 con 0x7f2ea80a1c50 2024-09-16T03:51:29.454 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.441+0000 7f2e9e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:51:29.454 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.441+0000 7f2e9e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:51:29.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.669+0000 7f2eaff5d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:51:29.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.669+0000 7f2eaff5d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:51:29.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.669+0000 7f2eaff5d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:51:29.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.669+0000 7f2eaff5d640 1 -- 172.21.15.42:0/3545756026 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f2ea8135e10 con 0x7f2ea80a1c50 2024-09-16T03:51:29.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.669+0000 7f2e9e7fc640 1 -- 172.21.15.42:0/3545756026 <== mon.2 v2:172.21.15.42:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1554 (secure 0 0 0) 0x7f2ea0014320 con 0x7f2ea80a1c50 2024-09-16T03:51:29.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.669+0000 7f2e9e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:51:29.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.669+0000 7f2e9e7fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:51:29.676 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":82,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":1,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:51:29.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.673+0000 7f2eaff5d640 10 librados: watch_flush enter 2024-09-16T03:51:29.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.673+0000 7f2eaff5d640 10 librados: watch_flush exit 2024-09-16T03:51:29.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.673+0000 7f2eaff5d640 20 client.814172.objecter shutdown clearing up homeless session... 2024-09-16T03:51:29.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.673+0000 7f2eaff5d640 10 client.814172.objecter successfully canceled tick 2024-09-16T03:51:29.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.673+0000 7f2eaff5d640 1 -- 172.21.15.42:0/3545756026 >> v2:172.21.15.154:6812/254845342 conn(0x7f2e8c038730 msgr2=0x7f2e8c03abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:29.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.673+0000 7f2eaff5d640 1 --2- 172.21.15.42:0/3545756026 >> v2:172.21.15.154:6812/254845342 conn(0x7f2e8c038730 0x7f2e8c03abf0 secure :-1 s=READY pgs=3562 cs=0 l=1 rev1=1 crypto rx=0x7f2ea40af000 tx=0x7f2ea4010040 comp rx=0 tx=0).stop 2024-09-16T03:51:29.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.673+0000 7f2eaff5d640 10 monclient: shutdown 2024-09-16T03:51:29.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.673+0000 7f2eaff5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:51:29.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.673+0000 7f2eaff5d640 1 -- 172.21.15.42:0/3545756026 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2ea80a1c50 msgr2=0x7f2ea8123ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:29.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.673+0000 7f2eaff5d640 1 --2- 172.21.15.42:0/3545756026 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2ea80a1c50 0x7f2ea8123ad0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f2ea8068f60 tx=0x7f2ea0007e60 comp rx=0 tx=0).stop 2024-09-16T03:51:29.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.673+0000 7f2eaff5d640 1 -- 172.21.15.42:0/3545756026 shutdown_connections 2024-09-16T03:51:29.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.673+0000 7f2eaff5d640 1 --2- 172.21.15.42:0/3545756026 >> v2:172.21.15.154:6812/254845342 conn(0x7f2e8c038730 0x7f2e8c03abf0 unknown :-1 s=CLOSED pgs=3562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:29.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.673+0000 7f2eaff5d640 1 --2- 172.21.15.42:0/3545756026 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2ea80a1c50 0x7f2ea8123ad0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:29.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.673+0000 7f2eaff5d640 1 --2- 172.21.15.42:0/3545756026 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2e8c00a460 0x7f2ea81cf900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:29.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.673+0000 7f2eaff5d640 1 --2- 172.21.15.42:0/3545756026 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2e8c007380 0x7f2ea81cf3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:29.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.673+0000 7f2eaff5d640 1 -- 172.21.15.42:0/3545756026 >> 172.21.15.42:0/3545756026 conn(0x7f2ea80a2420 msgr2=0x7f2ea811df80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:51:29.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.673+0000 7f2eaff5d640 1 -- 172.21.15.42:0/3545756026 shutdown_connections 2024-09-16T03:51:29.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.673+0000 7f2eaff5d640 1 -- 172.21.15.42:0/3545756026 wait complete. 2024-09-16T03:51:29.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.673+0000 7f2eaff5d640 1 librados: shutdown 2024-09-16T03:51:29.696 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T03:51:29.696 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-16T03:51:29.696 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T03:51:29.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.817+0000 7f27083df640 10 monclient: get_monmap_and_config 2024-09-16T03:51:29.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.817+0000 7f27083df640 10 monclient: build_initial_monmap 2024-09-16T03:51:29.823 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.817+0000 7f27083df640 10 monclient: monmap: 2024-09-16T03:51:29.823 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:51:29.823 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:29.823 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:51:29.821471+0000 2024-09-16T03:51:29.823 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:51:29.821471+0000 2024-09-16T03:51:29.823 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:51:29.823 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:29.823 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:51:29.823 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:51:29.823 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:51:29.823 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:51:29.823 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:51:29.823 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:29.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.817+0000 7f27083df640 1 Processor -- start 2024-09-16T03:51:29.828 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.817+0000 7f27083df640 1 -- start start 2024-09-16T03:51:29.828 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.817+0000 7f27083df640 10 monclient: init 2024-09-16T03:51:29.828 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.817+0000 7f27083df640 10 monclient: _reopen_session rank -1 2024-09-16T03:51:29.828 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.817+0000 7f27083df640 10 monclient: _add_conns ranks=[1,0,3,4,2] 2024-09-16T03:51:29.828 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.817+0000 7f27083df640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f27000a1330 addr v1:172.21.15.154:6789/0 2024-09-16T03:51:29.828 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.817+0000 7f27083df640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f270006dac0 addr v1:172.21.15.42:6789/0 2024-09-16T03:51:29.828 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.817+0000 7f27083df640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f27000a1e20 0x7f270013da70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:29.828 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.817+0000 7f27083df640 10 monclient(hunting): picked mon.noname-a con 0x7f27000a1e20 addr v2:172.21.15.42:3300/0 2024-09-16T03:51:29.828 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.817+0000 7f27083df640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f270009c090 con 0x7f270006dac0 2024-09-16T03:51:29.828 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.817+0000 7f27083df640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f270009f610 con 0x7f27000a1330 2024-09-16T03:51:29.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.817+0000 7f27083df640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:29.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.817+0000 7f27083df640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f270009f8f0 con 0x7f27000a1e20 2024-09-16T03:51:29.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.817+0000 7f27083df640 10 monclient(hunting): _renew_subs 2024-09-16T03:51:29.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.817+0000 7f270597a640 1 -- >> v1:172.21.15.42:6789/0 conn(0x7f270006dac0 legacy=0x7f270013a340 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.42:6789/0 2024-09-16T03:51:29.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.817+0000 7f27083df640 10 monclient(hunting): authenticate will time out at 1536.791077s 2024-09-16T03:51:29.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.817+0000 7f270697c640 1 -- >> v2:172.21.15.42:3300/0 conn(0x7f27000a1e20 msgr2=0x7f270013da70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3300/0 2024-09-16T03:51:29.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.817+0000 7f270697c640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f27000a1e20 0x7f270013da70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:51:29.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.817+0000 7f270617b640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f27000a1330 0x7f27000a1710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.42:37914/0 (socket says 172.21.15.42:37914) 2024-09-16T03:51:29.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.817+0000 7f270617b640 1 -- 172.21.15.42:0/4013411074 learned_addr learned my addr 172.21.15.42:0/4013411074 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:51:29.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.817+0000 7f2705179640 1 -- 172.21.15.42:0/4013411074 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 202151743 0 0) 0x7f270009f610 con 0x7f27000a1330 2024-09-16T03:51:29.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.817+0000 7f2705179640 10 monclient(hunting): _negotiate 2024-09-16T03:51:29.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.817+0000 7f2705179640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:29.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.817+0000 7f2705179640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:51:29.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.817+0000 7f2705179640 10 monclient(hunting): my global_id is 794359 2024-09-16T03:51:29.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.817+0000 7f2705179640 1 -- 172.21.15.42:0/4013411074 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f26ec003530 con 0x7f27000a1330 2024-09-16T03:51:29.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.821+0000 7f2705179640 1 -- 172.21.15.42:0/4013411074 <== mon.1 v1:172.21.15.154:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3341758370 0 0) 0x7f26ec003530 con 0x7f27000a1330 2024-09-16T03:51:29.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.821+0000 7f2705179640 1 -- 172.21.15.42:0/4013411074 >> v2:172.21.15.42:3300/0 conn(0x7f27000a1e20 msgr2=0x7f270013da70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:51:29.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.821+0000 7f2705179640 1 --2- 172.21.15.42:0/4013411074 >> v2:172.21.15.42:3300/0 conn(0x7f27000a1e20 0x7f270013da70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:29.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.821+0000 7f2705179640 1 -- 172.21.15.42:0/4013411074 >> v1:172.21.15.42:6789/0 conn(0x7f270006dac0 legacy=0x7f270013a340 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:51:29.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.821+0000 7f2705179640 10 monclient: _finish_hunting 0 2024-09-16T03:51:29.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.821+0000 7f2705179640 1 monclient: found mon.noname-c-legacy 2024-09-16T03:51:29.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.821+0000 7f2705179640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:51:29.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.821+0000 7f2705179640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.154:6789/0 2024-09-16T03:51:29.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.821+0000 7f2705179640 1 -- 172.21.15.42:0/4013411074 --> v1:172.21.15.154:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f270009fbd0 con 0x7f27000a1330 2024-09-16T03:51:29.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.821+0000 7f2705179640 10 monclient: _finish_auth 0 2024-09-16T03:51:29.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.821+0000 7f2705179640 10 monclient: _check_auth_tickets 2024-09-16T03:51:29.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.821+0000 7f2705179640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:51:29.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.821+0000 7f2705179640 1 -- 172.21.15.42:0/4013411074 <== mon.1 v1:172.21.15.154:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2177413734 0 0) 0x7f26fc001a00 con 0x7f27000a1330 2024-09-16T03:51:29.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.821+0000 7f2705179640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:51:29.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.821+0000 7f2705179640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-16T03:51:29.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.821+0000 7f2705179640 10 monclient: dump: 2024-09-16T03:51:29.831 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:51:29.831 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:29.831 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:29.831 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:29.832 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:51:29.832 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:29.832 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:51:29.832 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:51:29.832 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:51:29.832 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:29.832 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.821+0000 7f2705179640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] but i'm connected to v1:172.21.15.154:6789/0, reconnecting 2024-09-16T03:51:29.832 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.821+0000 7f2705179640 10 monclient: _reopen_session rank -1 2024-09-16T03:51:29.832 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.821+0000 7f2705179640 1 -- 172.21.15.42:0/4013411074 >> v1:172.21.15.154:6789/0 conn(0x7f27000a1330 legacy=0x7f27000a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:29.832 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.821+0000 7f2705179640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:51:29.832 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.821+0000 7f2705179640 1 --2- 172.21.15.42:0/4013411074 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f26ec0039c0 0x7f26ec003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:29.832 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.821+0000 7f2705179640 10 monclient(hunting): picked mon.a con 0x7f26ec0039c0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:51:29.832 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.821+0000 7f2705179640 1 --2- 172.21.15.42:0/4013411074 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f26ec004810 0x7f26ec004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:29.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.821+0000 7f2705179640 10 monclient(hunting): picked mon.c con 0x7f26ec004810 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:51:29.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.821+0000 7f2705179640 1 --2- 172.21.15.42:0/4013411074 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f26ec0052f0 0x7f26ec00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:29.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.821+0000 7f270617b640 1 -- 172.21.15.42:0/4013411074 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f26ec0039c0 msgr2=0x7f26ec003dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3300/0 2024-09-16T03:51:29.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.821+0000 7f2705179640 10 monclient(hunting): picked mon.b con 0x7f26ec0052f0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:51:29.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.821+0000 7f2705179640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:29.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.821+0000 7f2705179640 1 -- 172.21.15.42:0/4013411074 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f270009f8f0 con 0x7f26ec0039c0 2024-09-16T03:51:29.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.821+0000 7f270617b640 1 --2- 172.21.15.42:0/4013411074 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f26ec0039c0 0x7f26ec003dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:51:29.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.821+0000 7f2705179640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:29.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.821+0000 7f2705179640 1 -- 172.21.15.42:0/4013411074 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f26ec00c240 con 0x7f26ec0052f0 2024-09-16T03:51:29.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.821+0000 7f2705179640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:29.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.821+0000 7f2705179640 1 -- 172.21.15.42:0/4013411074 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f26ec00c440 con 0x7f26ec004810 2024-09-16T03:51:29.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.821+0000 7f2705179640 10 monclient(hunting): _renew_subs 2024-09-16T03:51:29.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.821+0000 7f2705179640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:51:29.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.821+0000 7f270597a640 1 --2- 172.21.15.42:0/4013411074 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f26ec004810 0x7f26ec004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:29.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.821+0000 7f270597a640 10 monclient(hunting): get_auth_request con 0x7f26ec004810 auth_method 0 2024-09-16T03:51:29.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.821+0000 7f270597a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:29.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.821+0000 7f270597a640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:29.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.821+0000 7f270597a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:51:29.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.821+0000 7f270597a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:51:29.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.821+0000 7f270597a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:51:29.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.821+0000 7f270597a640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T03:51:29.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.821+0000 7f270697c640 1 --2- 172.21.15.42:0/4013411074 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f26ec0052f0 0x7f26ec00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:29.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.821+0000 7f270697c640 10 monclient(hunting): get_auth_request con 0x7f26ec0052f0 auth_method 0 2024-09-16T03:51:29.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.821+0000 7f270697c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:29.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.821+0000 7f270697c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:29.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.821+0000 7f270697c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:51:29.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.821+0000 7f270597a640 10 monclient(hunting): handle_auth_done global_id 794359 payload 293 2024-09-16T03:51:29.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.821+0000 7f270597a640 1 -- 172.21.15.42:0/4013411074 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f26ec0052f0 msgr2=0x7f26ec00b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:51:29.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.821+0000 7f270597a640 1 --2- 172.21.15.42:0/4013411074 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f26ec0052f0 0x7f26ec00b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:29.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.821+0000 7f270597a640 1 -- 172.21.15.42:0/4013411074 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f26ec0039c0 msgr2=0x7f26ec003dc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:51:29.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.821+0000 7f270597a640 1 --2- 172.21.15.42:0/4013411074 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f26ec0039c0 0x7f26ec003dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:29.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.821+0000 7f270597a640 10 monclient: _finish_hunting 0 2024-09-16T03:51:29.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.821+0000 7f270597a640 1 monclient: found mon.c 2024-09-16T03:51:29.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.821+0000 7f270597a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:51:29.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.821+0000 7f270597a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:51:29.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.821+0000 7f270597a640 1 -- 172.21.15.42:0/4013411074 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26ec003530 con 0x7f26ec004810 2024-09-16T03:51:29.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.821+0000 7f270597a640 10 monclient: _finish_auth 0 2024-09-16T03:51:29.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.821+0000 7f270597a640 10 monclient: _check_auth_tickets 2024-09-16T03:51:29.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.821+0000 7f270597a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:51:29.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.821+0000 7f27083df640 5 monclient: authenticate success, global_id 794359 2024-09-16T03:51:29.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.821+0000 7f27083df640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:51:29.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.821+0000 7f270697c640 1 -- 172.21.15.42:0/4013411074 reap_dead start 2024-09-16T03:51:29.844 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.821+0000 7f270597a640 1 --2- 172.21.15.42:0/4013411074 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f26ec004810 0x7f26ec004c90 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f26f0006fd0 tx=0x7f26f001c3c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=38e9a27681ac576e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:51:29.845 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.821+0000 7f2705179640 1 -- 172.21.15.42:0/4013411074 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26f0009520 con 0x7f26ec004810 2024-09-16T03:51:29.845 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.821+0000 7f2705179640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:51:29.845 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.821+0000 7f2705179640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:51:29.845 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.821+0000 7f2705179640 10 monclient: dump: 2024-09-16T03:51:29.845 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:51:29.845 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:29.845 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:29.845 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:29.845 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:51:29.845 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:29.846 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:51:29.846 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:51:29.846 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:51:29.846 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:29.846 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.821+0000 7f2705179640 1 -- 172.21.15.42:0/4013411074 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f26f0003bc0 con 0x7f26ec004810 2024-09-16T03:51:29.846 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.821+0000 7f27083df640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:51:29.847 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.821+0000 7f2705179640 10 monclient: handle_config config(0 keys) 2024-09-16T03:51:29.847 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.821+0000 7f27083df640 10 monclient: get_monmap_and_config success 2024-09-16T03:51:29.847 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.821+0000 7f2705179640 1 -- 172.21.15.42:0/4013411074 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26f0005070 con 0x7f26ec004810 2024-09-16T03:51:29.847 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.821+0000 7f27083df640 4 set_mon_vals no callback set 2024-09-16T03:51:29.847 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.821+0000 7f2705179640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:51:29.847 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.821+0000 7f2705179640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:51:29.848 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.821+0000 7f2705179640 10 monclient: dump: 2024-09-16T03:51:29.848 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:51:29.848 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:29.848 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:29.848 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:29.848 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:51:29.848 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:29.848 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:51:29.848 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:51:29.848 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:51:29.848 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:29.848 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.825+0000 7f27083df640 10 monclient: shutdown 2024-09-16T03:51:29.849 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.825+0000 7f27083df640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:51:29.849 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.825+0000 7f27083df640 1 -- 172.21.15.42:0/4013411074 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f26ec004810 msgr2=0x7f26ec004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:29.849 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.825+0000 7f27083df640 1 --2- 172.21.15.42:0/4013411074 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f26ec004810 0x7f26ec004c90 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f26f0006fd0 tx=0x7f26f001c3c0 comp rx=0 tx=0).stop 2024-09-16T03:51:29.849 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.825+0000 7f27083df640 1 -- 172.21.15.42:0/4013411074 shutdown_connections 2024-09-16T03:51:29.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.825+0000 7f27083df640 1 --2- 172.21.15.42:0/4013411074 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f26ec004810 0x7f26ec004c90 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:29.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.825+0000 7f27083df640 1 -- 172.21.15.42:0/4013411074 >> 172.21.15.42:0/4013411074 conn(0x7f27000a2420 msgr2=0x7f27000a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:51:29.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.825+0000 7f27083df640 1 -- 172.21.15.42:0/4013411074 shutdown_connections 2024-09-16T03:51:29.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.825+0000 7f27083df640 1 -- 172.21.15.42:0/4013411074 wait complete. 2024-09-16T03:51:29.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.825+0000 7f27083df640 10 monclient: build_initial_monmap 2024-09-16T03:51:29.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.825+0000 7f27083df640 10 monclient: monmap: 2024-09-16T03:51:29.851 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:51:29.851 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:51:29.851 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:51:29.851 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:51:29.851 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:51:29.852 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:29.852 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:51:29.852 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:51:29.852 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:51:29.852 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:29.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.825+0000 7f27083df640 1 librados: starting msgr at 2024-09-16T03:51:29.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.825+0000 7f27083df640 1 librados: starting objecter 2024-09-16T03:51:29.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.825+0000 7f27083df640 1 Processor -- start 2024-09-16T03:51:29.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.825+0000 7f27083df640 1 -- start start 2024-09-16T03:51:29.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.825+0000 7f27083df640 1 librados: setting wanted keys 2024-09-16T03:51:29.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.825+0000 7f27083df640 1 librados: calling monclient init 2024-09-16T03:51:29.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.825+0000 7f27083df640 10 monclient: init 2024-09-16T03:51:29.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.825+0000 7f27083df640 10 monclient: _reopen_session rank -1 2024-09-16T03:51:29.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.825+0000 7f27083df640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:51:29.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.825+0000 7f27083df640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f26ec004810 0x7f27001cf3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:29.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.825+0000 7f27083df640 10 monclient(hunting): picked mon.noname-c con 0x7f26ec004810 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:51:29.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.825+0000 7f27083df640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f27001cf900 0x7f270009cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:29.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.825+0000 7f27083df640 10 monclient(hunting): picked mon.noname-b con 0x7f27001cf900 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:51:29.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.825+0000 7f27083df640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f27001cfd20 0x7f270009d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:29.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.825+0000 7f27083df640 10 monclient(hunting): picked mon.noname-a con 0x7f27001cfd20 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:51:29.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.825+0000 7f27083df640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:29.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.825+0000 7f27083df640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f2700135910 con 0x7f27001cfd20 2024-09-16T03:51:29.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.825+0000 7f27083df640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:29.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.825+0000 7f27083df640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f270009ef80 con 0x7f27001cf900 2024-09-16T03:51:29.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.825+0000 7f27083df640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:29.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.825+0000 7f27083df640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f2700135a90 con 0x7f26ec004810 2024-09-16T03:51:29.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.825+0000 7f27083df640 10 monclient(hunting): _renew_subs 2024-09-16T03:51:29.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.825+0000 7f27083df640 10 monclient(hunting): authenticate will time out at 1536.797977s 2024-09-16T03:51:29.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.825+0000 7f270697c640 1 -- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f27001cfd20 msgr2=0x7f270009d0d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.42:3300/0 2024-09-16T03:51:29.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.825+0000 7f270697c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f27001cfd20 0x7f270009d0d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:51:29.857 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.825+0000 7f270617b640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f26ec004810 0x7f27001cf3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:29.857 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.825+0000 7f270617b640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f26ec004810 0x7f27001cf3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:41780/0 (socket says 172.21.15.42:41780) 2024-09-16T03:51:29.857 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.825+0000 7f270617b640 1 -- 172.21.15.42:0/867730230 learned_addr learned my addr 172.21.15.42:0/867730230 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:51:29.857 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.825+0000 7f270617b640 10 monclient(hunting): get_auth_request con 0x7f26ec004810 auth_method 0 2024-09-16T03:51:29.857 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.825+0000 7f270617b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:29.857 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.825+0000 7f270617b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:29.857 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.825+0000 7f270617b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:51:29.857 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.825+0000 7f270617b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:51:29.857 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.825+0000 7f270617b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:51:29.857 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.825+0000 7f270617b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:51:29.857 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.825+0000 7f270597a640 1 --2- 172.21.15.42:0/867730230 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f27001cf900 0x7f270009cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:29.857 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.825+0000 7f270597a640 10 monclient(hunting): get_auth_request con 0x7f27001cf900 auth_method 0 2024-09-16T03:51:29.857 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.825+0000 7f270597a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:29.858 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.825+0000 7f270597a640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:29.858 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.825+0000 7f270597a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:51:29.858 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.825+0000 7f270617b640 10 monclient(hunting): handle_auth_done global_id 814175 payload 824 2024-09-16T03:51:29.858 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.825+0000 7f270617b640 1 -- 172.21.15.42:0/867730230 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f27001cf900 msgr2=0x7f270009cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:29.858 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.825+0000 7f270617b640 1 --2- 172.21.15.42:0/867730230 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f27001cf900 0x7f270009cb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:29.859 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.825+0000 7f270617b640 1 -- 172.21.15.42:0/867730230 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f27001cfd20 msgr2=0x7f270009d0d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:51:29.859 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.825+0000 7f270617b640 1 --2- 172.21.15.42:0/867730230 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f27001cfd20 0x7f270009d0d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:29.859 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.825+0000 7f270617b640 10 monclient: _finish_hunting 0 2024-09-16T03:51:29.859 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.825+0000 7f270617b640 1 monclient: found mon.noname-c 2024-09-16T03:51:29.859 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.825+0000 7f270617b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:51:29.859 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.825+0000 7f270617b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:51:29.859 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.825+0000 7f270617b640 1 -- 172.21.15.42:0/867730230 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27001288e0 con 0x7f26ec004810 2024-09-16T03:51:29.859 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.825+0000 7f270597a640 1 --2- 172.21.15.42:0/867730230 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f27001cf900 0x7f270009cb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T03:51:29.860 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.825+0000 7f270617b640 1 --2- 172.21.15.42:0/867730230 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f26ec004810 0x7f27001cf3c0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f26fc00ca80 tx=0x7f26fc00cf40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:51:29.860 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.825+0000 7f26f6ffd640 10 client.?.objecter ms_handle_connect 0x7f26ec004810 2024-09-16T03:51:29.860 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.825+0000 7f26f6ffd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:51:29.860 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.825+0000 7f26f6ffd640 1 -- 172.21.15.42:0/867730230 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26fc018070 con 0x7f26ec004810 2024-09-16T03:51:29.860 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.825+0000 7f26f6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:51:29.860 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.825+0000 7f26f6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:51:29.860 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.825+0000 7f26f6ffd640 10 monclient: dump: 2024-09-16T03:51:29.860 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:51:29.860 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:29.860 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:29.860 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:29.860 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:51:29.861 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:29.861 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:51:29.861 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:51:29.861 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:51:29.861 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:29.861 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.825+0000 7f26f6ffd640 10 monclient: _finish_auth 0 2024-09-16T03:51:29.861 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.825+0000 7f26f6ffd640 10 monclient: _check_auth_tickets 2024-09-16T03:51:29.861 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.825+0000 7f26f6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:51:29.861 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.825+0000 7f27083df640 5 monclient: authenticate success, global_id 814175 2024-09-16T03:51:29.861 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.825+0000 7f27083df640 10 monclient: _renew_subs 2024-09-16T03:51:29.861 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.825+0000 7f26f6ffd640 1 -- 172.21.15.42:0/867730230 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f26fc00a820 con 0x7f26ec004810 2024-09-16T03:51:29.861 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.825+0000 7f27083df640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:51:29.861 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.825+0000 7f27083df640 1 -- 172.21.15.42:0/867730230 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f270006e640 con 0x7f26ec004810 2024-09-16T03:51:29.862 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.825+0000 7f26f6ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:51:29.862 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.825+0000 7f26f6ffd640 1 -- 172.21.15.42:0/867730230 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26fc00ac50 con 0x7f26ec004810 2024-09-16T03:51:29.862 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.825+0000 7f26f6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:51:29.862 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.825+0000 7f26f6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:51:29.862 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.825+0000 7f26f6ffd640 10 monclient: dump: 2024-09-16T03:51:29.862 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:51:29.862 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:29.862 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:29.862 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:29.863 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:51:29.863 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:29.863 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:51:29.863 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:51:29.863 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:51:29.863 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:29.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.825+0000 7f27083df640 10 client.814175.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:51:29.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.825+0000 7f27083df640 10 monclient: _renew_subs 2024-09-16T03:51:29.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.825+0000 7f27083df640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:51:29.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.825+0000 7f26f7fff640 4 set_mon_vals no callback set 2024-09-16T03:51:29.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.825+0000 7f27083df640 1 -- 172.21.15.42:0/867730230 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f2700128600 con 0x7f26ec004810 2024-09-16T03:51:29.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.825+0000 7f27083df640 1 librados: init done 2024-09-16T03:51:29.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.825+0000 7f27083df640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:51:29.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.825+0000 7f27083df640 1 --2- 172.21.15.42:0/867730230 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f270002c650 0x7f270002ca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:29.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.825+0000 7f27083df640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:29.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.825+0000 7f27083df640 1 -- 172.21.15.42:0/867730230 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f2700127d30 con 0x7f270002c650 2024-09-16T03:51:29.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.825+0000 7f270597a640 1 --2- 172.21.15.42:0/867730230 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f270002c650 0x7f270002ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:29.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.829+0000 7f26f6ffd640 1 -- 172.21.15.42:0/867730230 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f26fc01b070 con 0x7f26ec004810 2024-09-16T03:51:29.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.829+0000 7f270597a640 10 monclient: get_auth_request con 0x7f270002c650 auth_method 0 2024-09-16T03:51:29.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.829+0000 7f270597a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:29.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.829+0000 7f270597a640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:29.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.829+0000 7f270597a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:51:29.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.829+0000 7f270597a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:51:29.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.829+0000 7f270597a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:51:29.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.829+0000 7f270597a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:51:29.865 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.829+0000 7f26f6ffd640 1 --2- 172.21.15.42:0/867730230 >> v2:172.21.15.154:6812/254845342 conn(0x7f26e4037820 0x7f26e4039ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:29.865 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.829+0000 7f270597a640 10 monclient(hunting): handle_auth_done global_id 814178 payload 824 2024-09-16T03:51:29.865 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.829+0000 7f270597a640 1 -- 172.21.15.42:0/867730230 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f27000710c0 con 0x7f270002c650 2024-09-16T03:51:29.865 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.829+0000 7f26f6ffd640 1 -- 172.21.15.42:0/867730230 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(311..311 src has 1..311) ==== 5731+0+0 (secure 0 0 0) 0x7f26fc048760 con 0x7f26ec004810 2024-09-16T03:51:29.865 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.829+0000 7f26f6ffd640 10 client.814175.objecter ms_dispatch 0x7f2700000ba0 osd_map(311..311 src has 1..311) 2024-09-16T03:51:29.865 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.829+0000 7f26f6ffd640 3 client.814175.objecter handle_osd_map got epochs [311,311] > 0 2024-09-16T03:51:29.865 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.829+0000 7f26f6ffd640 3 client.814175.objecter handle_osd_map decoding full epoch 311 2024-09-16T03:51:29.865 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.829+0000 7f26f6ffd640 20 client.814175.objecter dump_active .. 0 homeless 2024-09-16T03:51:29.865 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.829+0000 7f26f6ffd640 10 client.814175.objecter ms_handle_connect 0x7f270002c650 2024-09-16T03:51:29.865 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.829+0000 7f26f6ffd640 10 client.814175.objecter resend_mon_ops 2024-09-16T03:51:29.865 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.829+0000 7f270597a640 1 --2- 172.21.15.42:0/867730230 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f270002c650 0x7f270002ca30 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f26ec0057f0 tx=0x7f26f0019160 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:51:29.865 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.829+0000 7f26f6ffd640 1 -- 172.21.15.42:0/867730230 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26f00acc00 con 0x7f270002c650 2024-09-16T03:51:29.865 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.829+0000 7f26f6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:51:29.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.829+0000 7f26f6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:51:29.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.829+0000 7f26f6ffd640 10 monclient: dump: 2024-09-16T03:51:29.866 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:51:29.866 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:29.866 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:29.866 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:29.866 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:51:29.866 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:29.866 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:51:29.866 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:51:29.866 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:51:29.867 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:29.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.829+0000 7f270697c640 1 --2- 172.21.15.42:0/867730230 >> v2:172.21.15.154:6812/254845342 conn(0x7f26e4037820 0x7f26e4039ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:29.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.829+0000 7f270697c640 10 monclient: get_auth_request con 0x7f26e4037820 auth_method 0 2024-09-16T03:51:29.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.829+0000 7f26f6ffd640 1 -- 172.21.15.42:0/867730230 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f26f00095c0 con 0x7f270002c650 2024-09-16T03:51:29.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.829+0000 7f26f6ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:51:29.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.829+0000 7f26f6ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:51:29.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.829+0000 7f26f6ffd640 1 -- 172.21.15.42:0/867730230 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f270002c650 msgr2=0x7f270002ca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:29.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.829+0000 7f26f6ffd640 1 --2- 172.21.15.42:0/867730230 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f270002c650 0x7f270002ca30 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f26ec0057f0 tx=0x7f26f0019160 comp rx=0 tx=0).stop 2024-09-16T03:51:29.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.829+0000 7f26f6ffd640 1 -- 172.21.15.42:0/867730230 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f270002c650 msgr2=0x7f270002ca30 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:51:29.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.829+0000 7f26f6ffd640 1 --2- 172.21.15.42:0/867730230 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f270002c650 0x7f270002ca30 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:29.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.829+0000 7f26f6ffd640 10 client.814175.objecter ms_handle_connect 0x7f26e4037820 2024-09-16T03:51:29.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.829+0000 7f270697c640 1 --2- 172.21.15.42:0/867730230 >> v2:172.21.15.154:6812/254845342 conn(0x7f26e4037820 0x7f26e4039ce0 secure :-1 s=READY pgs=3563 cs=0 l=1 rev1=1 crypto rx=0x7f27000a6900 tx=0x7f26f8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:51:29.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.833+0000 7f27083df640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:51:29.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.833+0000 7f27083df640 1 --2- 172.21.15.42:0/867730230 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f270002cf70 0x7f270002d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:29.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.833+0000 7f27083df640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:29.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.833+0000 7f27083df640 1 -- 172.21.15.42:0/867730230 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f270011f690 con 0x7f270002cf70 2024-09-16T03:51:29.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.833+0000 7f270597a640 1 --2- 172.21.15.42:0/867730230 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f270002cf70 0x7f270002d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:29.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.833+0000 7f270597a640 10 monclient: get_auth_request con 0x7f270002cf70 auth_method 0 2024-09-16T03:51:29.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.833+0000 7f270597a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:29.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.833+0000 7f270597a640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:29.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.833+0000 7f270597a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:51:29.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.833+0000 7f270597a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:51:29.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.833+0000 7f270597a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:51:29.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.833+0000 7f270597a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:51:29.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.833+0000 7f270597a640 10 monclient(hunting): handle_auth_done global_id 814181 payload 824 2024-09-16T03:51:29.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.833+0000 7f270597a640 1 -- 172.21.15.42:0/867730230 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f2700126e90 con 0x7f270002cf70 2024-09-16T03:51:29.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.833+0000 7f26f6ffd640 10 client.814175.objecter ms_handle_connect 0x7f270002cf70 2024-09-16T03:51:29.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.833+0000 7f26f6ffd640 10 client.814175.objecter resend_mon_ops 2024-09-16T03:51:29.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.833+0000 7f270597a640 1 --2- 172.21.15.42:0/867730230 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f270002cf70 0x7f270002d380 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f26f001c400 tx=0x7f26ec005840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:51:29.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.833+0000 7f26f6ffd640 1 -- 172.21.15.42:0/867730230 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26f0005640 con 0x7f270002cf70 2024-09-16T03:51:29.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.833+0000 7f26f6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:51:29.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.833+0000 7f26f6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:51:29.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.833+0000 7f26f6ffd640 10 monclient: dump: 2024-09-16T03:51:29.869 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:51:29.869 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:29.869 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:29.869 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:29.870 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:51:29.870 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:29.870 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:51:29.870 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:51:29.870 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:51:29.870 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:29.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.837+0000 7f26f6ffd640 1 -- 172.21.15.42:0/867730230 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4867 (secure 0 0 0) 0x7f26f0005640 con 0x7f270002cf70 2024-09-16T03:51:29.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.837+0000 7f26f6ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:51:29.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.837+0000 7f26f6ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:51:29.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.837+0000 7f26f6ffd640 1 -- 172.21.15.42:0/867730230 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f270002cf70 msgr2=0x7f270002d380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:29.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.837+0000 7f26f6ffd640 1 --2- 172.21.15.42:0/867730230 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f270002cf70 0x7f270002d380 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f26f001c400 tx=0x7f26ec005840 comp rx=0 tx=0).stop 2024-09-16T03:51:29.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.837+0000 7f26f6ffd640 1 -- 172.21.15.42:0/867730230 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f270002cf70 msgr2=0x7f270002d380 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:51:29.871 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.837+0000 7f26f6ffd640 1 --2- 172.21.15.42:0/867730230 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f270002cf70 0x7f270002d380 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:29.871 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:51:29.871 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:51:29.871 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:51:29.871 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T03:51:29.871 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 82, 2024-09-16T03:51:29.871 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 34907, 2024-09-16T03:51:29.871 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:51:29.871 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T03:51:29.872 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T03:51:29.872 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:51:29.872 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 1, 2024-09-16T03:51:29.872 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:51:29.872 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:51:29.872 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:51:29.872 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:51:29.872 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:51:29.872 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:51:29.872 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:51:29.872 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:51:29.872 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:51:29.873 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:51:29.873 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:51:29.873 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:51:29.873 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:51:29.873 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:51:29.873 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:51:29.873 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:51:29.873 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:51:29.874 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:51:29.874 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:51:29.874 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:51:29.874 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:51:29.874 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:51:29.874 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:51:29.874 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:51:29.874 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:51:29.875 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:51:29.875 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:51:29.875 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:51:29.875 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:51:29.875 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:29.875 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:51:29.875 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:51:29.875 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:51:29.875 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:51:29.875 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:51:29.875 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:51:29.875 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:51:29.876 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:51:29.876 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:51:29.876 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:51:29.876 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:51:29.876 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:51:29.876 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:51:29.876 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:51:29.876 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:51:29.876 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:51:29.876 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:51:29.876 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:51:29.876 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:51:29.876 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:51:29.876 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:51:29.877 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:51:29.877 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:51:29.877 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:51:29.877 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:51:29.877 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:51:29.877 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:51:29.877 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:51:29.877 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:51:29.877 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:51:29.877 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:29.877 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:51:29.877 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:29.877 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:51:29.877 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:51:29.878 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:51:29.878 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:51:29.878 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:29.878 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:51:29.878 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:51:29.878 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:51:29.878 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:29.878 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:29.878 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:51:29.878 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:51:29.878 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:51:29.878 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:51:29.878 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:51:29.878 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:29.879 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:51:29.879 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:51:29.879 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:51:29.879 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:51:29.879 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:51:29.879 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:29.879 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:29.879 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:51:29.879 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:51:29.879 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:51:29.879 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:51:29.879 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:29.879 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:51:29.879 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:51:29.879 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:51:29.880 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:29.880 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:29.880 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:51:29.880 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:51:29.880 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:51:29.880 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:51:29.880 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:51:29.880 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:29.880 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:51:29.880 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:51:29.880 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:51:29.880 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:51:29.880 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:51:29.880 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:29.881 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:29.881 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:51:29.881 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:51:29.881 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:51:29.881 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:51:29.881 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:29.881 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:51:29.881 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:51:29.881 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:51:29.881 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:29.881 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:29.881 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:51:29.881 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:51:29.882 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:51:29.882 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:51:29.882 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:51:29.882 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:29.882 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:51:29.882 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:51:29.882 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:51:29.882 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:51:29.882 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:51:29.882 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:51:29.882 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:51:29.882 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:29.882 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:51:29.882 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:51:29.882 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:29.883 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:51:29.883 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:51:29.883 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:51:29.883 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:51:29.883 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:51:29.883 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T03:51:29.883 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:29.883 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:51:29.883 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:51:29.883 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 5 2024-09-16T03:51:29.883 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:51:29.883 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:51:29.883 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:51:29.883 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:29.884 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:51:29.884 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:51:29.884 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 7 2024-09-16T03:51:29.884 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:51:29.884 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:51:29.884 INFO:teuthology.orchestra.run.smithi042.stdout: "mgr": [ 2024-09-16T03:51:29.884 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:29.884 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:51:29.884 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:51:29.884 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:51:29.884 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:51:29.884 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:51:29.884 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:29.884 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:51:29.885 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:51:29.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.837+0000 7f27083df640 10 librados: watch_flush enter 2024-09-16T03:51:29.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.837+0000 7f27083df640 10 librados: watch_flush exit 2024-09-16T03:51:29.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.837+0000 7f27083df640 20 client.814175.objecter shutdown clearing up homeless session... 2024-09-16T03:51:29.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.837+0000 7f27083df640 10 client.814175.objecter successfully canceled tick 2024-09-16T03:51:29.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.837+0000 7f27083df640 1 -- 172.21.15.42:0/867730230 >> v2:172.21.15.154:6812/254845342 conn(0x7f26e4037820 msgr2=0x7f26e4039ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:29.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.837+0000 7f27083df640 1 --2- 172.21.15.42:0/867730230 >> v2:172.21.15.154:6812/254845342 conn(0x7f26e4037820 0x7f26e4039ce0 secure :-1 s=READY pgs=3563 cs=0 l=1 rev1=1 crypto rx=0x7f27000a6900 tx=0x7f26f8008040 comp rx=0 tx=0).stop 2024-09-16T03:51:29.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.837+0000 7f27083df640 10 monclient: shutdown 2024-09-16T03:51:29.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.837+0000 7f27083df640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:51:29.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.837+0000 7f27083df640 1 -- 172.21.15.42:0/867730230 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f26ec004810 msgr2=0x7f27001cf3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:29.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.837+0000 7f27083df640 1 --2- 172.21.15.42:0/867730230 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f26ec004810 0x7f27001cf3c0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f26fc00ca80 tx=0x7f26fc00cf40 comp rx=0 tx=0).stop 2024-09-16T03:51:29.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.837+0000 7f270697c640 1 -- 172.21.15.42:0/867730230 reap_dead start 2024-09-16T03:51:29.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.837+0000 7f270697c640 1 -- 172.21.15.42:0/867730230 reap_dead start 2024-09-16T03:51:29.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.837+0000 7f27083df640 1 -- 172.21.15.42:0/867730230 shutdown_connections 2024-09-16T03:51:29.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.837+0000 7f27083df640 1 -- 172.21.15.42:0/867730230 >> 172.21.15.42:0/867730230 conn(0x7f27000a2420 msgr2=0x7f2700071850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:51:29.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.837+0000 7f27083df640 1 -- 172.21.15.42:0/867730230 shutdown_connections 2024-09-16T03:51:29.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.837+0000 7f27083df640 1 -- 172.21.15.42:0/867730230 wait complete. 2024-09-16T03:51:29.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:29.837+0000 7f27083df640 1 librados: shutdown 2024-09-16T03:51:29.888 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T03:51:29.888 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-16T03:51:29.888 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T03:51:30.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.005+0000 7f34c8bf1640 10 monclient: get_monmap_and_config 2024-09-16T03:51:30.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.005+0000 7f34c8bf1640 10 monclient: build_initial_monmap 2024-09-16T03:51:30.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.005+0000 7f34c8bf1640 10 monclient: monmap: 2024-09-16T03:51:30.010 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:51:30.010 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:30.010 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:51:30.009110+0000 2024-09-16T03:51:30.010 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:51:30.009110+0000 2024-09-16T03:51:30.011 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:51:30.011 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:30.011 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:51:30.011 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:51:30.011 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:51:30.011 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:51:30.011 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:51:30.011 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:30.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.005+0000 7f34c8bf1640 1 Processor -- start 2024-09-16T03:51:30.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.005+0000 7f34c8bf1640 1 -- start start 2024-09-16T03:51:30.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.005+0000 7f34c8bf1640 10 monclient: init 2024-09-16T03:51:30.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.005+0000 7f34c8bf1640 10 monclient: _reopen_session rank -1 2024-09-16T03:51:30.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.005+0000 7f34c8bf1640 10 monclient: _add_conns ranks=[3,2,4,1,0] 2024-09-16T03:51:30.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.005+0000 7f34c8bf1640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f34c40a1330 0x7f34c409f180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:30.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.005+0000 7f34c8bf1640 10 monclient(hunting): picked mon.noname-a con 0x7f34c40a1330 addr v2:172.21.15.42:3300/0 2024-09-16T03:51:30.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.005+0000 7f34c8bf1640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f34c40a1710 0x7f34c40a1b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:30.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.005+0000 7f34c8bf1640 10 monclient(hunting): picked mon.noname-b con 0x7f34c40a1710 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:51:30.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.005+0000 7f34c8bf1640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f34c41340f0 0x7f34c41345a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:30.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.005+0000 7f34c8bf1640 10 monclient(hunting): picked mon.noname-c con 0x7f34c41340f0 addr v2:172.21.15.154:3300/0 2024-09-16T03:51:30.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.005+0000 7f34c8bf1640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:30.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.005+0000 7f34c8bf1640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f34c409edf0 con 0x7f34c40a1330 2024-09-16T03:51:30.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.005+0000 7f34c8bf1640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:30.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.005+0000 7f34c8bf1640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f34c409cd60 con 0x7f34c41340f0 2024-09-16T03:51:30.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.005+0000 7f34c8bf1640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:30.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.005+0000 7f34c8bf1640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f34c409e4c0 con 0x7f34c40a1710 2024-09-16T03:51:30.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.005+0000 7f34c8bf1640 10 monclient(hunting): _renew_subs 2024-09-16T03:51:30.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.005+0000 7f34c8bf1640 10 monclient(hunting): authenticate will time out at 1536.978574s 2024-09-16T03:51:30.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.005+0000 7f34c259c640 1 -- >> v2:172.21.15.42:3300/0 conn(0x7f34c40a1330 msgr2=0x7f34c409f180 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3300/0 2024-09-16T03:51:30.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.005+0000 7f34c259c640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f34c40a1330 0x7f34c409f180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:51:30.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.005+0000 7f34c1d9b640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f34c40a1710 0x7f34c40a1b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:30.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.005+0000 7f34c1d9b640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f34c40a1710 0x7f34c40a1b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:41806/0 (socket says 172.21.15.42:41806) 2024-09-16T03:51:30.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.005+0000 7f34c1d9b640 1 -- 172.21.15.42:0/2471267119 learned_addr learned my addr 172.21.15.42:0/2471267119 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:51:30.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.005+0000 7f34c1d9b640 10 monclient(hunting): get_auth_request con 0x7f34c40a1710 auth_method 0 2024-09-16T03:51:30.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.005+0000 7f34c1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:30.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.005+0000 7f34c1d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:30.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.005+0000 7f34c1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:51:30.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.005+0000 7f34c1d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:51:30.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.005+0000 7f34c1d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:51:30.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.005+0000 7f34c1d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:51:30.016 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.005+0000 7f34c2d9d640 1 --2- 172.21.15.42:0/2471267119 >> v2:172.21.15.154:3300/0 conn(0x7f34c41340f0 0x7f34c41345a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:30.016 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.005+0000 7f34c2d9d640 10 monclient(hunting): get_auth_request con 0x7f34c41340f0 auth_method 0 2024-09-16T03:51:30.016 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.005+0000 7f34c2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:30.016 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.005+0000 7f34c2d9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:30.016 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.005+0000 7f34c2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:51:30.016 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.005+0000 7f34c1d9b640 10 monclient(hunting): handle_auth_done global_id 814184 payload 274 2024-09-16T03:51:30.016 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.005+0000 7f34c1d9b640 1 -- 172.21.15.42:0/2471267119 >> v2:172.21.15.154:3300/0 conn(0x7f34c41340f0 msgr2=0x7f34c41345a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:51:30.016 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.005+0000 7f34c1d9b640 1 --2- 172.21.15.42:0/2471267119 >> v2:172.21.15.154:3300/0 conn(0x7f34c41340f0 0x7f34c41345a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:30.016 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.005+0000 7f34c1d9b640 1 -- 172.21.15.42:0/2471267119 >> v2:172.21.15.42:3300/0 conn(0x7f34c40a1330 msgr2=0x7f34c409f180 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:51:30.016 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.005+0000 7f34c1d9b640 1 --2- 172.21.15.42:0/2471267119 >> v2:172.21.15.42:3300/0 conn(0x7f34c40a1330 0x7f34c409f180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:30.016 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.005+0000 7f34c1d9b640 10 monclient: _finish_hunting 0 2024-09-16T03:51:30.016 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.005+0000 7f34c1d9b640 1 monclient: found mon.noname-b 2024-09-16T03:51:30.016 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.005+0000 7f34c1d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:51:30.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.005+0000 7f34c1d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T03:51:30.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.005+0000 7f34c1d9b640 1 -- 172.21.15.42:0/2471267119 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34c40a2050 con 0x7f34c40a1710 2024-09-16T03:51:30.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.009+0000 7f34c1d9b640 1 --2- 172.21.15.42:0/2471267119 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f34c40a1710 0x7f34c40a1b10 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f34ac009e50 tx=0x7f34ac01c8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=23edb51c866497e4 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:51:30.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.009+0000 7f34c159a640 1 -- 172.21.15.42:0/2471267119 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34ac0a5660 con 0x7f34c40a1710 2024-09-16T03:51:30.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.009+0000 7f34c159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:51:30.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.009+0000 7f34c159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:51:30.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.009+0000 7f34c159a640 10 monclient: dump: 2024-09-16T03:51:30.024 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:51:30.024 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:30.025 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:30.025 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:30.025 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:51:30.025 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:30.025 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:51:30.025 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:51:30.025 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:51:30.025 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:30.025 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.009+0000 7f34c159a640 10 monclient: _finish_auth 0 2024-09-16T03:51:30.025 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.009+0000 7f34c159a640 10 monclient: _check_auth_tickets 2024-09-16T03:51:30.025 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.009+0000 7f34c159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:51:30.025 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.009+0000 7f34c159a640 1 -- 172.21.15.42:0/2471267119 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f34ac0a5800 con 0x7f34c40a1710 2024-09-16T03:51:30.025 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.009+0000 7f34c159a640 10 monclient: handle_config config(0 keys) 2024-09-16T03:51:30.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.009+0000 7f34c159a640 1 -- 172.21.15.42:0/2471267119 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34ac0a5bd0 con 0x7f34c40a1710 2024-09-16T03:51:30.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.009+0000 7f34c159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:51:30.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.009+0000 7f34c159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:51:30.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.009+0000 7f34c159a640 10 monclient: dump: 2024-09-16T03:51:30.026 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:51:30.026 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:30.026 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:30.026 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:30.026 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:51:30.026 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:30.026 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:51:30.026 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:51:30.026 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:51:30.026 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:30.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.009+0000 7f34c8bf1640 5 monclient: authenticate success, global_id 814184 2024-09-16T03:51:30.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.009+0000 7f34c8bf1640 10 monclient: get_monmap_and_config success 2024-09-16T03:51:30.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.009+0000 7f34c8bf1640 4 set_mon_vals no callback set 2024-09-16T03:51:30.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.009+0000 7f34c8bf1640 10 monclient: shutdown 2024-09-16T03:51:30.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.009+0000 7f34c8bf1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:51:30.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.009+0000 7f34c8bf1640 1 -- 172.21.15.42:0/2471267119 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f34c40a1710 msgr2=0x7f34c40a1b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:30.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.009+0000 7f34c8bf1640 1 --2- 172.21.15.42:0/2471267119 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f34c40a1710 0x7f34c40a1b10 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f34ac009e50 tx=0x7f34ac01c8d0 comp rx=0 tx=0).stop 2024-09-16T03:51:30.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.009+0000 7f34c8bf1640 1 -- 172.21.15.42:0/2471267119 shutdown_connections 2024-09-16T03:51:30.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.009+0000 7f34c8bf1640 1 --2- 172.21.15.42:0/2471267119 >> v2:172.21.15.154:3300/0 conn(0x7f34c41340f0 0x7f34c41345a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:30.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.009+0000 7f34c8bf1640 1 --2- 172.21.15.42:0/2471267119 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f34c40a1710 0x7f34c40a1b10 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:30.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.009+0000 7f34c8bf1640 1 --2- 172.21.15.42:0/2471267119 >> v2:172.21.15.42:3300/0 conn(0x7f34c40a1330 0x7f34c409f180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:30.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.009+0000 7f34c8bf1640 1 -- 172.21.15.42:0/2471267119 >> 172.21.15.42:0/2471267119 conn(0x7f34c40a2420 msgr2=0x7f34c40a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:51:30.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.009+0000 7f34c8bf1640 1 -- 172.21.15.42:0/2471267119 shutdown_connections 2024-09-16T03:51:30.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.009+0000 7f34c8bf1640 1 -- 172.21.15.42:0/2471267119 wait complete. 2024-09-16T03:51:30.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.009+0000 7f34c8bf1640 10 monclient: build_initial_monmap 2024-09-16T03:51:30.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.009+0000 7f34c8bf1640 10 monclient: monmap: 2024-09-16T03:51:30.028 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:51:30.028 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:51:30.028 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:51:30.028 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:51:30.028 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:51:30.028 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:30.028 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:51:30.028 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:51:30.028 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:51:30.028 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:30.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.009+0000 7f34c8bf1640 1 librados: starting msgr at 2024-09-16T03:51:30.029 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.009+0000 7f34c8bf1640 1 librados: starting objecter 2024-09-16T03:51:30.029 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.009+0000 7f34c8bf1640 1 Processor -- start 2024-09-16T03:51:30.029 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.009+0000 7f34c8bf1640 1 -- start start 2024-09-16T03:51:30.029 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.009+0000 7f34c8bf1640 1 librados: setting wanted keys 2024-09-16T03:51:30.029 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.009+0000 7f34c8bf1640 1 librados: calling monclient init 2024-09-16T03:51:30.029 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.009+0000 7f34c8bf1640 10 monclient: init 2024-09-16T03:51:30.029 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.009+0000 7f34c8bf1640 10 monclient: _reopen_session rank -1 2024-09-16T03:51:30.029 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.009+0000 7f34c8bf1640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T03:51:30.029 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.009+0000 7f34c8bf1640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f34c40a1330 0x7f34c41cf1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:30.029 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.009+0000 7f34c8bf1640 10 monclient(hunting): picked mon.noname-b con 0x7f34c40a1330 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:51:30.029 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.009+0000 7f34c8bf1640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f34c40a1710 0x7f34c4132470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:30.029 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.009+0000 7f34c8bf1640 10 monclient(hunting): picked mon.noname-c con 0x7f34c40a1710 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:51:30.029 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.009+0000 7f34c8bf1640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f34c41340f0 0x7f34c41329b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:30.029 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.009+0000 7f34c8bf1640 10 monclient(hunting): picked mon.noname-a con 0x7f34c41340f0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:51:30.030 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.009+0000 7f34c8bf1640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:30.030 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.009+0000 7f34c8bf1640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f34c409d1b0 con 0x7f34c41340f0 2024-09-16T03:51:30.030 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.009+0000 7f34c8bf1640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:30.030 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.009+0000 7f34c8bf1640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f34c409d030 con 0x7f34c40a1330 2024-09-16T03:51:30.030 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.009+0000 7f34c8bf1640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:30.030 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.009+0000 7f34c8bf1640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f34c413dbd0 con 0x7f34c40a1710 2024-09-16T03:51:30.030 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.009+0000 7f34c8bf1640 10 monclient(hunting): _renew_subs 2024-09-16T03:51:30.030 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.009+0000 7f34c8bf1640 10 monclient(hunting): authenticate will time out at 1536.983723s 2024-09-16T03:51:30.030 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.009+0000 7f34c2d9d640 1 -- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f34c41340f0 msgr2=0x7f34c41329b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.42:3300/0 2024-09-16T03:51:30.030 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.009+0000 7f34c2d9d640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f34c41340f0 0x7f34c41329b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:51:30.030 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.009+0000 7f34c1d9b640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f34c40a1710 0x7f34c4132470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:30.030 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.009+0000 7f34c1d9b640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f34c40a1710 0x7f34c4132470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:41822/0 (socket says 172.21.15.42:41822) 2024-09-16T03:51:30.030 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.009+0000 7f34c1d9b640 1 -- 172.21.15.42:0/3563889197 learned_addr learned my addr 172.21.15.42:0/3563889197 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:51:30.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.009+0000 7f34c1d9b640 10 monclient(hunting): get_auth_request con 0x7f34c40a1710 auth_method 0 2024-09-16T03:51:30.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.009+0000 7f34c1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:30.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.009+0000 7f34c1d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:30.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.009+0000 7f34c1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:51:30.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.009+0000 7f34c1d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:51:30.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.009+0000 7f34c1d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:51:30.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.009+0000 7f34c1d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:51:30.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.013+0000 7f34c1d9b640 10 monclient(hunting): handle_auth_done global_id 814187 payload 824 2024-09-16T03:51:30.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.013+0000 7f34c259c640 1 --2- 172.21.15.42:0/3563889197 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f34c40a1330 0x7f34c41cf1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:30.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.013+0000 7f34c1d9b640 1 -- 172.21.15.42:0/3563889197 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f34c40a1330 msgr2=0x7f34c41cf1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:30.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.013+0000 7f34c1d9b640 1 --2- 172.21.15.42:0/3563889197 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f34c40a1330 0x7f34c41cf1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:30.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.013+0000 7f34c1d9b640 1 -- 172.21.15.42:0/3563889197 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f34c41340f0 msgr2=0x7f34c41329b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:51:30.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.013+0000 7f34c1d9b640 1 --2- 172.21.15.42:0/3563889197 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f34c41340f0 0x7f34c41329b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:30.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.013+0000 7f34c1d9b640 10 monclient: _finish_hunting 0 2024-09-16T03:51:30.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.013+0000 7f34c1d9b640 1 monclient: found mon.noname-c 2024-09-16T03:51:30.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.013+0000 7f34c1d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:51:30.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.013+0000 7f34c1d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:51:30.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.013+0000 7f34c1d9b640 1 -- 172.21.15.42:0/3563889197 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34c41cf6f0 con 0x7f34c40a1710 2024-09-16T03:51:30.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.013+0000 7f34c1d9b640 1 --2- 172.21.15.42:0/3563889197 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f34c40a1710 0x7f34c4132470 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f34ac01af80 tx=0x7f34ac019670 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:51:30.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.013+0000 7f34baffd640 10 client.?.objecter ms_handle_connect 0x7f34c40a1710 2024-09-16T03:51:30.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.013+0000 7f34baffd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:51:30.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.013+0000 7f34baffd640 1 -- 172.21.15.42:0/3563889197 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34ac0bc020 con 0x7f34c40a1710 2024-09-16T03:51:30.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.013+0000 7f34baffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:51:30.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.013+0000 7f34baffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:51:30.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.013+0000 7f34baffd640 10 monclient: dump: 2024-09-16T03:51:30.032 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:51:30.032 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:30.033 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:30.033 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:30.033 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:51:30.033 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:30.033 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:51:30.033 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:51:30.033 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:51:30.033 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:30.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.013+0000 7f34baffd640 10 monclient: _finish_auth 0 2024-09-16T03:51:30.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.013+0000 7f34baffd640 10 monclient: _check_auth_tickets 2024-09-16T03:51:30.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.013+0000 7f34baffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:51:30.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.013+0000 7f34baffd640 1 -- 172.21.15.42:0/3563889197 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f34ac0b8670 con 0x7f34c40a1710 2024-09-16T03:51:30.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.013+0000 7f34baffd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:51:30.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.013+0000 7f34baffd640 1 -- 172.21.15.42:0/3563889197 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34ac0b8a40 con 0x7f34c40a1710 2024-09-16T03:51:30.034 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.013+0000 7f34baffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:51:30.034 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.013+0000 7f34baffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:51:30.034 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.013+0000 7f34baffd640 10 monclient: dump: 2024-09-16T03:51:30.034 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:51:30.034 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:30.034 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:30.034 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:30.034 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:51:30.034 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:30.034 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:51:30.034 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:51:30.034 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:51:30.034 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:30.034 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.013+0000 7f34c8bf1640 5 monclient: authenticate success, global_id 814187 2024-09-16T03:51:30.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.013+0000 7f34c0d99640 4 set_mon_vals no callback set 2024-09-16T03:51:30.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.013+0000 7f34c8bf1640 10 monclient: _renew_subs 2024-09-16T03:51:30.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.013+0000 7f34c8bf1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:51:30.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.013+0000 7f34c8bf1640 1 -- 172.21.15.42:0/3563889197 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34c409f360 con 0x7f34c40a1710 2024-09-16T03:51:30.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.013+0000 7f34c8bf1640 10 client.814187.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:51:30.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.013+0000 7f34c8bf1640 10 monclient: _renew_subs 2024-09-16T03:51:30.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.013+0000 7f34c8bf1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:51:30.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.013+0000 7f34c8bf1640 1 -- 172.21.15.42:0/3563889197 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f34c411f870 con 0x7f34c40a1710 2024-09-16T03:51:30.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.013+0000 7f34c8bf1640 1 librados: init done 2024-09-16T03:51:30.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.013+0000 7f34c8bf1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:51:30.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.013+0000 7f34c8bf1640 1 --2- 172.21.15.42:0/3563889197 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f34c411fab0 0x7f34c411bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:30.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.013+0000 7f34c8bf1640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:30.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.013+0000 7f34c8bf1640 1 -- 172.21.15.42:0/3563889197 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f34c40a1b10 con 0x7f34c411fab0 2024-09-16T03:51:30.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.013+0000 7f34baffd640 1 -- 172.21.15.42:0/3563889197 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f34ac0bf020 con 0x7f34c40a1710 2024-09-16T03:51:30.036 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.013+0000 7f34baffd640 1 --2- 172.21.15.42:0/3563889197 >> v2:172.21.15.154:6812/254845342 conn(0x7f34a40377c0 0x7f34a4039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:30.036 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.013+0000 7f34c259c640 1 --2- 172.21.15.42:0/3563889197 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f34c411fab0 0x7f34c411bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:30.036 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.013+0000 7f34baffd640 1 -- 172.21.15.42:0/3563889197 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(311..311 src has 1..311) ==== 5731+0+0 (secure 0 0 0) 0x7f34ac0e3e90 con 0x7f34c40a1710 2024-09-16T03:51:30.036 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.013+0000 7f34baffd640 10 client.814187.objecter ms_dispatch 0x7f34c4000ba0 osd_map(311..311 src has 1..311) 2024-09-16T03:51:30.036 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.013+0000 7f34baffd640 3 client.814187.objecter handle_osd_map got epochs [311,311] > 0 2024-09-16T03:51:30.036 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.013+0000 7f34baffd640 3 client.814187.objecter handle_osd_map decoding full epoch 311 2024-09-16T03:51:30.036 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.013+0000 7f34c259c640 10 monclient: get_auth_request con 0x7f34c411fab0 auth_method 0 2024-09-16T03:51:30.036 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.013+0000 7f34c259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:30.036 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.013+0000 7f34c259c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:30.036 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.013+0000 7f34c259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:51:30.036 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.013+0000 7f34baffd640 20 client.814187.objecter dump_active .. 0 homeless 2024-09-16T03:51:30.036 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.013+0000 7f34c259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:51:30.036 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.013+0000 7f34c259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:51:30.036 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.013+0000 7f34c259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:51:30.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.013+0000 7f34c2d9d640 1 --2- 172.21.15.42:0/3563889197 >> v2:172.21.15.154:6812/254845342 conn(0x7f34a40377c0 0x7f34a4039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:30.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.013+0000 7f34c2d9d640 10 monclient: get_auth_request con 0x7f34a40377c0 auth_method 0 2024-09-16T03:51:30.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.013+0000 7f34c259c640 10 monclient(hunting): handle_auth_done global_id 794368 payload 824 2024-09-16T03:51:30.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.013+0000 7f34c259c640 1 -- 172.21.15.42:0/3563889197 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f34c411e420 con 0x7f34c411fab0 2024-09-16T03:51:30.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.013+0000 7f34c259c640 1 --2- 172.21.15.42:0/3563889197 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f34c411fab0 0x7f34c411bf70 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f34bc00c4c0 tx=0x7f34bc00c980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:51:30.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.013+0000 7f34baffd640 10 client.814187.objecter ms_handle_connect 0x7f34c411fab0 2024-09-16T03:51:30.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.013+0000 7f34baffd640 10 client.814187.objecter resend_mon_ops 2024-09-16T03:51:30.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.013+0000 7f34baffd640 1 -- 172.21.15.42:0/3563889197 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34bc003dd0 con 0x7f34c411fab0 2024-09-16T03:51:30.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.013+0000 7f34baffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:51:30.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.013+0000 7f34baffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:51:30.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.013+0000 7f34baffd640 10 monclient: dump: 2024-09-16T03:51:30.037 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:51:30.037 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:30.038 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:30.038 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:30.038 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:51:30.038 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:30.038 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:51:30.038 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:51:30.038 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:51:30.038 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:30.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.013+0000 7f34baffd640 10 client.814187.objecter ms_handle_connect 0x7f34a40377c0 2024-09-16T03:51:30.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.013+0000 7f34c2d9d640 1 --2- 172.21.15.42:0/3563889197 >> v2:172.21.15.154:6812/254845342 conn(0x7f34a40377c0 0x7f34a4039c80 secure :-1 s=READY pgs=3564 cs=0 l=1 rev1=1 crypto rx=0x7f34b4006e90 tx=0x7f34b4006e20 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:51:30.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.013+0000 7f34baffd640 1 -- 172.21.15.42:0/3563889197 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f34bc016070 con 0x7f34c411fab0 2024-09-16T03:51:30.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.013+0000 7f34baffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:51:30.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.013+0000 7f34baffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:51:30.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.013+0000 7f34baffd640 1 -- 172.21.15.42:0/3563889197 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f34c411fab0 msgr2=0x7f34c411bf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:30.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.013+0000 7f34baffd640 1 --2- 172.21.15.42:0/3563889197 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f34c411fab0 0x7f34c411bf70 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f34bc00c4c0 tx=0x7f34bc00c980 comp rx=0 tx=0).stop 2024-09-16T03:51:30.039 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.013+0000 7f34baffd640 1 -- 172.21.15.42:0/3563889197 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f34c411fab0 msgr2=0x7f34c411bf70 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:51:30.039 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.013+0000 7f34baffd640 1 --2- 172.21.15.42:0/3563889197 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f34c411fab0 0x7f34c411bf70 secure :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f34bc00c4c0 tx=0x7f34bc00c980 comp rx=0 tx=0).stop 2024-09-16T03:51:30.040 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.021+0000 7f34c8bf1640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:51:30.040 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.021+0000 7f34c8bf1640 1 --2- 172.21.15.42:0/3563889197 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f34c411c4b0 0x7f34c411c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:30.040 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.021+0000 7f34c8bf1640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:30.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.021+0000 7f34c8bf1640 1 -- 172.21.15.42:0/3563889197 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f34c411ce00 con 0x7f34c411c4b0 2024-09-16T03:51:30.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.021+0000 7f34c259c640 1 --2- 172.21.15.42:0/3563889197 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f34c411c4b0 0x7f34c411c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:30.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.021+0000 7f34c259c640 10 monclient: get_auth_request con 0x7f34c411c4b0 auth_method 0 2024-09-16T03:51:30.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.021+0000 7f34c259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:30.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.021+0000 7f34c259c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:30.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.021+0000 7f34c259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:51:30.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.021+0000 7f34c259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:51:30.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.021+0000 7f34c259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:51:30.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.021+0000 7f34c259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:51:30.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.021+0000 7f34c259c640 10 monclient(hunting): handle_auth_done global_id 794371 payload 824 2024-09-16T03:51:30.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.021+0000 7f34c259c640 1 -- 172.21.15.42:0/3563889197 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f34c411fe90 con 0x7f34c411c4b0 2024-09-16T03:51:30.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.021+0000 7f34c259c640 1 --2- 172.21.15.42:0/3563889197 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f34c411c4b0 0x7f34c411c8c0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f34bc00c4c0 tx=0x7f34bc0059e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:51:30.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.021+0000 7f34baffd640 10 client.814187.objecter ms_handle_connect 0x7f34c411c4b0 2024-09-16T03:51:30.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.021+0000 7f34baffd640 10 client.814187.objecter resend_mon_ops 2024-09-16T03:51:30.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.021+0000 7f34baffd640 1 -- 172.21.15.42:0/3563889197 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34bc005460 con 0x7f34c411c4b0 2024-09-16T03:51:30.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.021+0000 7f34baffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:51:30.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.021+0000 7f34baffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:51:30.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.021+0000 7f34baffd640 10 monclient: dump: 2024-09-16T03:51:30.042 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:51:30.042 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:30.042 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:30.042 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:30.042 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:51:30.042 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:30.042 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:51:30.042 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:51:30.042 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:51:30.043 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:30.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.025+0000 7f34baffd640 1 -- 172.21.15.42:0/3563889197 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4703 (secure 0 0 0) 0x7f34bc005460 con 0x7f34c411c4b0 2024-09-16T03:51:30.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.025+0000 7f34baffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:51:30.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.025+0000 7f34baffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:51:30.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.025+0000 7f34baffd640 1 -- 172.21.15.42:0/3563889197 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f34c411c4b0 msgr2=0x7f34c411c8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:30.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.025+0000 7f34baffd640 1 --2- 172.21.15.42:0/3563889197 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f34c411c4b0 0x7f34c411c8c0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f34bc00c4c0 tx=0x7f34bc0059e0 comp rx=0 tx=0).stop 2024-09-16T03:51:30.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.025+0000 7f34baffd640 1 -- 172.21.15.42:0/3563889197 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f34c411c4b0 msgr2=0x7f34c411c8c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:51:30.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.025+0000 7f34baffd640 1 --2- 172.21.15.42:0/3563889197 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f34c411c4b0 0x7f34c411c8c0 secure :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f34bc00c4c0 tx=0x7f34bc0059e0 comp rx=0 tx=0).stop 2024-09-16T03:51:30.043 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:51:30.043 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:51:30.044 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:51:30.044 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "leader", 2024-09-16T03:51:30.044 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 82, 2024-09-16T03:51:30.044 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 131861, 2024-09-16T03:51:30.044 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:51:30.044 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T03:51:30.044 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T03:51:30.044 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:51:30.044 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 2, 2024-09-16T03:51:30.044 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:51:30.044 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:51:30.044 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:51:30.044 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:51:30.044 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:51:30.045 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:51:30.045 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:51:30.045 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:51:30.045 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:51:30.045 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:51:30.045 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:51:30.045 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:51:30.045 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:51:30.045 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:51:30.045 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:51:30.045 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:51:30.045 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:51:30.045 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:51:30.046 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:51:30.046 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:51:30.046 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:51:30.046 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:51:30.046 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:51:30.046 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:51:30.046 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:51:30.046 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:51:30.046 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:51:30.046 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:51:30.046 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:51:30.046 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:30.046 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:51:30.046 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:51:30.047 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:51:30.047 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:51:30.047 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:51:30.047 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:51:30.047 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:51:30.047 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:51:30.047 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:51:30.047 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:51:30.047 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:51:30.047 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:51:30.047 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:51:30.047 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:51:30.047 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:51:30.048 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:51:30.048 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:51:30.048 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:51:30.048 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:51:30.048 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:51:30.048 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:51:30.048 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:51:30.048 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:51:30.048 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:51:30.048 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:51:30.048 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:51:30.048 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:51:30.048 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:51:30.048 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:51:30.049 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:51:30.049 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:30.049 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:51:30.049 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:30.049 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:51:30.049 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:51:30.049 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:51:30.049 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:51:30.049 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:30.049 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:51:30.049 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:51:30.049 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:51:30.049 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:30.049 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:30.050 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:51:30.050 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:51:30.050 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:51:30.050 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:51:30.050 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:51:30.050 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:30.050 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:51:30.050 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:51:30.050 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:51:30.050 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:51:30.050 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:51:30.050 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:30.050 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:30.050 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:51:30.051 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:51:30.051 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:51:30.051 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:51:30.051 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:30.051 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:51:30.051 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:51:30.051 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:51:30.051 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:30.051 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:30.051 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:51:30.051 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:51:30.051 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:51:30.051 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:51:30.051 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:51:30.052 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:30.052 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:51:30.052 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:51:30.052 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:51:30.052 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:51:30.052 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:51:30.052 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:30.052 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:30.052 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:51:30.052 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:51:30.052 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:51:30.052 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:51:30.052 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:30.052 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:51:30.053 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:51:30.053 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:51:30.053 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:30.053 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:30.053 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:51:30.053 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:51:30.053 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:51:30.053 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:51:30.053 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:51:30.053 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:30.053 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:51:30.053 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:51:30.053 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:51:30.053 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:51:30.054 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:51:30.054 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:51:30.054 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:51:30.054 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:30.054 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:51:30.054 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:51:30.054 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:30.054 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:51:30.054 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:51:30.054 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:51:30.054 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:51:30.054 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:51:30.054 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T03:51:30.054 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:30.055 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:51:30.055 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:51:30.055 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:51:30.055 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:51:30.055 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:51:30.055 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:51:30.055 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:30.055 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:51:30.055 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:51:30.055 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 2 2024-09-16T03:51:30.055 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:51:30.055 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:51:30.055 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:30.055 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:51:30.056 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:51:30.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.025+0000 7f34c8bf1640 10 librados: watch_flush enter 2024-09-16T03:51:30.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.025+0000 7f34c8bf1640 10 librados: watch_flush exit 2024-09-16T03:51:30.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.025+0000 7f34c8bf1640 20 client.814187.objecter shutdown clearing up homeless session... 2024-09-16T03:51:30.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.025+0000 7f34c8bf1640 10 client.814187.objecter successfully canceled tick 2024-09-16T03:51:30.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.025+0000 7f34c8bf1640 1 -- 172.21.15.42:0/3563889197 >> v2:172.21.15.154:6812/254845342 conn(0x7f34a40377c0 msgr2=0x7f34a4039c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:30.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.025+0000 7f34c8bf1640 1 --2- 172.21.15.42:0/3563889197 >> v2:172.21.15.154:6812/254845342 conn(0x7f34a40377c0 0x7f34a4039c80 secure :-1 s=READY pgs=3564 cs=0 l=1 rev1=1 crypto rx=0x7f34b4006e90 tx=0x7f34b4006e20 comp rx=0 tx=0).stop 2024-09-16T03:51:30.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.025+0000 7f34c8bf1640 10 monclient: shutdown 2024-09-16T03:51:30.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.025+0000 7f34c8bf1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:51:30.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.025+0000 7f34c8bf1640 1 -- 172.21.15.42:0/3563889197 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f34c40a1710 msgr2=0x7f34c4132470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:30.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.025+0000 7f34c8bf1640 1 --2- 172.21.15.42:0/3563889197 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f34c40a1710 0x7f34c4132470 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f34ac01af80 tx=0x7f34ac019670 comp rx=0 tx=0).stop 2024-09-16T03:51:30.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.025+0000 7f34c2d9d640 1 -- 172.21.15.42:0/3563889197 reap_dead start 2024-09-16T03:51:30.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.025+0000 7f34c2d9d640 1 -- 172.21.15.42:0/3563889197 reap_dead start 2024-09-16T03:51:30.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.025+0000 7f34c8bf1640 1 -- 172.21.15.42:0/3563889197 shutdown_connections 2024-09-16T03:51:30.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.025+0000 7f34c8bf1640 1 -- 172.21.15.42:0/3563889197 >> 172.21.15.42:0/3563889197 conn(0x7f34c40a2420 msgr2=0x7f34c409ba50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:51:30.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.025+0000 7f34c8bf1640 1 -- 172.21.15.42:0/3563889197 shutdown_connections 2024-09-16T03:51:30.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.025+0000 7f34c8bf1640 1 -- 172.21.15.42:0/3563889197 wait complete. 2024-09-16T03:51:30.059 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:30.025+0000 7f34c8bf1640 1 librados: shutdown 2024-09-16T03:51:30.059 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T03:51:30.059 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-16T03:51:50.060 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-16T03:51:50.061 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-09-16T03:51:50.061 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-16T03:51:50.061 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-16T03:51:50.065 INFO:tasks.ceph.mon.a:Started 2024-09-16T03:51:50.066 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T03:51:50.066 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:51:50.192 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.185+0000 7f9ad0400640 10 monclient: get_monmap_and_config 2024-09-16T03:51:50.193 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.185+0000 7f9ad0400640 10 monclient: build_initial_monmap 2024-09-16T03:51:50.193 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.185+0000 7f9ad0400640 10 monclient: monmap: 2024-09-16T03:51:50.193 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:51:50.193 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:50.193 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:51:50.191753+0000 2024-09-16T03:51:50.193 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:51:50.191753+0000 2024-09-16T03:51:50.193 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:51:50.193 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:50.193 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:51:50.193 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:51:50.194 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:51:50.194 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:51:50.194 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:51:50.194 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:50.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.189+0000 7f9ad0400640 1 Processor -- start 2024-09-16T03:51:50.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.189+0000 7f9ad0400640 1 -- start start 2024-09-16T03:51:50.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.189+0000 7f9ad0400640 10 monclient: init 2024-09-16T03:51:50.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.189+0000 7f9ad0400640 10 monclient: _reopen_session rank -1 2024-09-16T03:51:50.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.189+0000 7f9ad0400640 10 monclient: _add_conns ranks=[0,4,2,1,3] 2024-09-16T03:51:50.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.189+0000 7f9ad0400640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9ac81373e0 addr v1:172.21.15.42:6789/0 2024-09-16T03:51:50.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.189+0000 7f9ad0400640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f9ac806c810 0x7f9ac813c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:50.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.189+0000 7f9ad0400640 10 monclient(hunting): picked mon.noname-c con 0x7f9ac806c810 addr v2:172.21.15.154:3300/0 2024-09-16T03:51:50.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.189+0000 7f9ad0400640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9ac813c8e0 0x7f9ac813ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:50.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.189+0000 7f9ad0400640 10 monclient(hunting): picked mon.noname-b con 0x7f9ac813c8e0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:51:50.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.189+0000 7f9ad0400640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9ac80a0850 con 0x7f9ac81373e0 2024-09-16T03:51:50.197 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.189+0000 7f9ad0400640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:50.197 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.189+0000 7f9ad0400640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f9ac806c0c0 con 0x7f9ac806c810 2024-09-16T03:51:50.197 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.189+0000 7f9ad0400640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:50.197 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.189+0000 7f9ad0400640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f9ac806c440 con 0x7f9ac813c8e0 2024-09-16T03:51:50.197 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.189+0000 7f9ad0400640 10 monclient(hunting): _renew_subs 2024-09-16T03:51:50.197 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.189+0000 7f9ad0400640 10 monclient(hunting): authenticate will time out at 1557.160938s 2024-09-16T03:51:50.197 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.189+0000 7f9ace99d640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9ac813c8e0 0x7f9ac813ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:50.197 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.189+0000 7f9ace19c640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f9ac81373e0 0x7f9ac81377c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:52302/0 (socket says 172.21.15.42:52302) 2024-09-16T03:51:50.197 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.189+0000 7f9ace19c640 1 -- 172.21.15.42:0/148362826 learned_addr learned my addr 172.21.15.42:0/148362826 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:51:50.197 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.189+0000 7f9ace99d640 10 monclient(hunting): get_auth_request con 0x7f9ac813c8e0 auth_method 0 2024-09-16T03:51:50.197 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.189+0000 7f9ace99d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:50.197 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.189+0000 7f9ace99d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:50.198 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.189+0000 7f9ace99d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:51:50.198 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.189+0000 7f9ace99d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:51:50.198 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.189+0000 7f9ace99d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:51:50.198 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.189+0000 7f9ace99d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:51:50.198 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.189+0000 7f9ace99d640 10 monclient(hunting): handle_auth_done global_id 814190 payload 274 2024-09-16T03:51:50.198 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.189+0000 7f9acd99b640 1 --2- 172.21.15.42:0/148362826 >> v2:172.21.15.154:3300/0 conn(0x7f9ac806c810 0x7f9ac813c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:50.198 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.189+0000 7f9ace99d640 1 -- 172.21.15.42:0/148362826 >> v2:172.21.15.154:3300/0 conn(0x7f9ac806c810 msgr2=0x7f9ac813c3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:51:50.198 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.189+0000 7f9ace99d640 1 --2- 172.21.15.42:0/148362826 >> v2:172.21.15.154:3300/0 conn(0x7f9ac806c810 0x7f9ac813c3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:50.198 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.189+0000 7f9ace99d640 1 -- 172.21.15.42:0/148362826 >> v1:172.21.15.42:6789/0 conn(0x7f9ac81373e0 legacy=0x7f9ac81377c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:50.198 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.189+0000 7f9ace99d640 10 monclient: _finish_hunting 0 2024-09-16T03:51:50.198 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.189+0000 7f9ace99d640 1 monclient: found mon.noname-b 2024-09-16T03:51:50.198 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.189+0000 7f9ace99d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:51:50.198 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.189+0000 7f9ace99d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T03:51:50.199 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.189+0000 7f9ace99d640 1 -- 172.21.15.42:0/148362826 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ac8137ed0 con 0x7f9ac813c8e0 2024-09-16T03:51:50.199 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.189+0000 7f9acd99b640 10 monclient: get_auth_request con 0x7f9ac806c810 auth_method 0 2024-09-16T03:51:50.199 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.189+0000 7f9acd99b640 1 --2- 172.21.15.42:0/148362826 >> v2:172.21.15.154:3300/0 conn(0x7f9ac806c810 0x7f9ac813c3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T03:51:50.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.189+0000 7f9ace99d640 1 --2- 172.21.15.42:0/148362826 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9ac813c8e0 0x7f9ac813ecd0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f9ac00097c0 tx=0x7f9ac001c360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=679bd8bcb2f6ce82 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:51:50.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.189+0000 7f9acd19a640 1 -- 172.21.15.42:0/148362826 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ac00b0070 con 0x7f9ac813c8e0 2024-09-16T03:51:50.205 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.189+0000 7f9acd19a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:51:50.205 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.189+0000 7f9acd19a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:51:50.205 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.189+0000 7f9acd19a640 10 monclient: dump: 2024-09-16T03:51:50.205 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:51:50.205 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:50.205 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:50.205 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:50.205 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:51:50.205 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:50.205 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:51:50.205 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:51:50.205 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:51:50.205 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:50.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.189+0000 7f9acd19a640 10 monclient: _finish_auth 0 2024-09-16T03:51:50.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.189+0000 7f9acd19a640 10 monclient: _check_auth_tickets 2024-09-16T03:51:50.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.189+0000 7f9acd19a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:51:50.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.189+0000 7f9acd19a640 1 -- 172.21.15.42:0/148362826 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9ac00ab600 con 0x7f9ac813c8e0 2024-09-16T03:51:50.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.189+0000 7f9ad0400640 5 monclient: authenticate success, global_id 814190 2024-09-16T03:51:50.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.189+0000 7f9ad0400640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:51:50.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.189+0000 7f9acd19a640 10 monclient: handle_config config(0 keys) 2024-09-16T03:51:50.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.189+0000 7f9acd19a640 1 -- 172.21.15.42:0/148362826 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ac00ab9d0 con 0x7f9ac813c8e0 2024-09-16T03:51:50.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.189+0000 7f9acd19a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:51:50.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.189+0000 7f9acd19a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:51:50.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.189+0000 7f9acd19a640 10 monclient: dump: 2024-09-16T03:51:50.206 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:51:50.206 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:50.207 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:50.207 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:50.207 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:51:50.207 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:50.207 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:51:50.207 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:51:50.207 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:51:50.207 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:50.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.189+0000 7f9ad0400640 10 monclient: get_monmap_and_config success 2024-09-16T03:51:50.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.189+0000 7f9ad0400640 4 set_mon_vals no callback set 2024-09-16T03:51:50.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.193+0000 7f9ad0400640 10 monclient: shutdown 2024-09-16T03:51:50.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.193+0000 7f9ad0400640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:51:50.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.193+0000 7f9ad0400640 1 -- 172.21.15.42:0/148362826 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9ac813c8e0 msgr2=0x7f9ac813ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:50.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.193+0000 7f9ad0400640 1 --2- 172.21.15.42:0/148362826 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9ac813c8e0 0x7f9ac813ecd0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f9ac00097c0 tx=0x7f9ac001c360 comp rx=0 tx=0).stop 2024-09-16T03:51:50.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.193+0000 7f9ad0400640 1 -- 172.21.15.42:0/148362826 shutdown_connections 2024-09-16T03:51:50.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.193+0000 7f9ad0400640 1 --2- 172.21.15.42:0/148362826 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9ac813c8e0 0x7f9ac813ecd0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:50.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.193+0000 7f9ad0400640 1 --2- 172.21.15.42:0/148362826 >> v2:172.21.15.154:3300/0 conn(0x7f9ac806c810 0x7f9ac813c3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:50.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.193+0000 7f9ad0400640 1 -- 172.21.15.42:0/148362826 >> 172.21.15.42:0/148362826 conn(0x7f9ac80a6e80 msgr2=0x7f9ac80a3320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:51:50.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.193+0000 7f9ad0400640 1 -- 172.21.15.42:0/148362826 shutdown_connections 2024-09-16T03:51:50.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.193+0000 7f9ad0400640 1 -- 172.21.15.42:0/148362826 wait complete. 2024-09-16T03:51:50.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.193+0000 7f9ad0400640 10 monclient: build_initial_monmap 2024-09-16T03:51:50.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.193+0000 7f9ad0400640 10 monclient: monmap: 2024-09-16T03:51:50.208 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:51:50.208 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:51:50.208 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:51:50.208 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:51:50.208 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:51:50.209 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:50.209 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:51:50.209 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:51:50.209 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:51:50.209 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:50.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.193+0000 7f9ad0400640 1 librados: starting msgr at 2024-09-16T03:51:50.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.193+0000 7f9ad0400640 1 librados: starting objecter 2024-09-16T03:51:50.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.193+0000 7f9ad0400640 1 Processor -- start 2024-09-16T03:51:50.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.193+0000 7f9ad0400640 1 -- start start 2024-09-16T03:51:50.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.193+0000 7f9ad0400640 1 librados: setting wanted keys 2024-09-16T03:51:50.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.193+0000 7f9ad0400640 1 librados: calling monclient init 2024-09-16T03:51:50.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.193+0000 7f9ad0400640 10 monclient: init 2024-09-16T03:51:50.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.193+0000 7f9ad0400640 10 monclient: _reopen_session rank -1 2024-09-16T03:51:50.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.193+0000 7f9ad0400640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:51:50.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.193+0000 7f9ad0400640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9ac806c810 0x7f9ac8135f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:50.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.193+0000 7f9ad0400640 10 monclient(hunting): picked mon.noname-c con 0x7f9ac806c810 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:51:50.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.193+0000 7f9ad0400640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9ac81373e0 0x7f9ac811ce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:50.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.193+0000 7f9ad0400640 10 monclient(hunting): picked mon.noname-a con 0x7f9ac81373e0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:51:50.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.193+0000 7f9ad0400640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9ac813c8e0 0x7f9ac811d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:50.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.193+0000 7f9ad0400640 10 monclient(hunting): picked mon.noname-b con 0x7f9ac813c8e0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:51:50.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.193+0000 7f9ad0400640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:50.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.193+0000 7f9ad0400640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f9ac8141dc0 con 0x7f9ac81373e0 2024-09-16T03:51:50.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.193+0000 7f9ad0400640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:50.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.193+0000 7f9ad0400640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f9ac8141c40 con 0x7f9ac813c8e0 2024-09-16T03:51:50.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.193+0000 7f9ad0400640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:50.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.193+0000 7f9ad0400640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f9ac8141f40 con 0x7f9ac806c810 2024-09-16T03:51:50.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.193+0000 7f9ad0400640 10 monclient(hunting): _renew_subs 2024-09-16T03:51:50.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.193+0000 7f9ad0400640 10 monclient(hunting): authenticate will time out at 1557.165942s 2024-09-16T03:51:50.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.193+0000 7f9acd99b640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9ac81373e0 0x7f9ac811ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:50.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.193+0000 7f9acd99b640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9ac81373e0 0x7f9ac811ce70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:40592/0 (socket says 172.21.15.42:40592) 2024-09-16T03:51:50.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.193+0000 7f9acd99b640 1 -- 172.21.15.42:0/2975214975 learned_addr learned my addr 172.21.15.42:0/2975214975 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:51:50.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.193+0000 7f9acd99b640 10 monclient(hunting): get_auth_request con 0x7f9ac81373e0 auth_method 0 2024-09-16T03:51:50.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.193+0000 7f9acd99b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:50.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.193+0000 7f9acd99b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:50.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.193+0000 7f9acd99b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:51:50.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.193+0000 7f9ace19c640 1 --2- 172.21.15.42:0/2975214975 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9ac806c810 0x7f9ac8135f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:50.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.193+0000 7f9acd99b640 1 -- 172.21.15.42:0/2975214975 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9ac81373e0 msgr2=0x7f9ac811ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-16T03:51:50.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.193+0000 7f9acd99b640 1 -- 172.21.15.42:0/2975214975 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9ac81373e0 msgr2=0x7f9ac811ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T03:51:50.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.193+0000 7f9acd99b640 1 --2- 172.21.15.42:0/2975214975 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9ac81373e0 0x7f9ac811ce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:51:50.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.193+0000 7f9acd99b640 1 --2- 172.21.15.42:0/2975214975 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9ac81373e0 0x7f9ac811ce70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:51:50.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.193+0000 7f9ace19c640 10 monclient(hunting): get_auth_request con 0x7f9ac806c810 auth_method 0 2024-09-16T03:51:50.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.193+0000 7f9ace19c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:50.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.193+0000 7f9ace19c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:50.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.193+0000 7f9ace19c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:51:50.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.193+0000 7f9ace19c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:51:50.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.193+0000 7f9ace19c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:51:50.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.193+0000 7f9ace19c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:51:50.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.193+0000 7f9ace99d640 1 --2- 172.21.15.42:0/2975214975 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9ac813c8e0 0x7f9ac811d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:50.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.193+0000 7f9ace99d640 10 monclient(hunting): get_auth_request con 0x7f9ac813c8e0 auth_method 0 2024-09-16T03:51:50.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.193+0000 7f9ace99d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:50.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.193+0000 7f9ace99d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:50.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.193+0000 7f9ace99d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:51:50.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.193+0000 7f9ace19c640 10 monclient(hunting): handle_auth_done global_id 814193 payload 824 2024-09-16T03:51:50.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.193+0000 7f9ace19c640 1 -- 172.21.15.42:0/2975214975 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9ac813c8e0 msgr2=0x7f9ac811d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:50.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.193+0000 7f9ace19c640 1 --2- 172.21.15.42:0/2975214975 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9ac813c8e0 0x7f9ac811d3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:50.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.193+0000 7f9ace19c640 1 -- 172.21.15.42:0/2975214975 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9ac81373e0 msgr2=0x7f9ac811ce70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:51:50.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.193+0000 7f9ace19c640 1 --2- 172.21.15.42:0/2975214975 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9ac81373e0 0x7f9ac811ce70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:50.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.193+0000 7f9ace19c640 10 monclient: _finish_hunting 0 2024-09-16T03:51:50.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.193+0000 7f9ace19c640 1 monclient: found mon.noname-c 2024-09-16T03:51:50.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.193+0000 7f9ace19c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:51:50.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.193+0000 7f9ace19c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:51:50.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.193+0000 7f9ace19c640 1 -- 172.21.15.42:0/2975214975 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ac8120960 con 0x7f9ac806c810 2024-09-16T03:51:50.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.193+0000 7f9ace99d640 1 --2- 172.21.15.42:0/2975214975 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9ac813c8e0 0x7f9ac811d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T03:51:50.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.193+0000 7f9ace19c640 1 --2- 172.21.15.42:0/2975214975 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9ac806c810 0x7f9ac8135f80 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f9ac400ca80 tx=0x7f9ac400cf40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:51:50.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.193+0000 7f9abeffd640 10 client.?.objecter ms_handle_connect 0x7f9ac806c810 2024-09-16T03:51:50.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.193+0000 7f9abeffd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:51:50.214 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.193+0000 7f9abeffd640 1 -- 172.21.15.42:0/2975214975 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ac4017070 con 0x7f9ac806c810 2024-09-16T03:51:50.214 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.193+0000 7f9abeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:51:50.214 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.193+0000 7f9abeffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:51:50.214 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.193+0000 7f9abeffd640 10 monclient: dump: 2024-09-16T03:51:50.214 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:51:50.214 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:50.214 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:50.214 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:50.214 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:51:50.214 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:50.214 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:51:50.214 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:51:50.214 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:51:50.214 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:50.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.193+0000 7f9abeffd640 10 monclient: _finish_auth 0 2024-09-16T03:51:50.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.193+0000 7f9abeffd640 10 monclient: _check_auth_tickets 2024-09-16T03:51:50.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.193+0000 7f9abeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:51:50.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.193+0000 7f9abeffd640 1 -- 172.21.15.42:0/2975214975 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9ac400abd0 con 0x7f9ac806c810 2024-09-16T03:51:50.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.193+0000 7f9abeffd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:51:50.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.193+0000 7f9abeffd640 1 -- 172.21.15.42:0/2975214975 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ac4012400 con 0x7f9ac806c810 2024-09-16T03:51:50.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.193+0000 7f9abeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:51:50.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.193+0000 7f9abeffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:51:50.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.193+0000 7f9abeffd640 10 monclient: dump: 2024-09-16T03:51:50.215 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:51:50.215 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:50.215 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:50.215 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:50.215 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:51:50.216 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:50.216 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:51:50.216 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:51:50.216 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:51:50.216 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:50.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.193+0000 7f9ad0400640 5 monclient: authenticate success, global_id 814193 2024-09-16T03:51:50.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.193+0000 7f9ad0400640 10 monclient: _renew_subs 2024-09-16T03:51:50.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.193+0000 7f9ad0400640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:51:50.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.193+0000 7f9ad0400640 1 -- 172.21.15.42:0/2975214975 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ac813c490 con 0x7f9ac806c810 2024-09-16T03:51:50.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.193+0000 7f9acc999640 4 set_mon_vals no callback set 2024-09-16T03:51:50.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.193+0000 7f9ad0400640 10 client.814193.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:51:50.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.193+0000 7f9ad0400640 10 monclient: _renew_subs 2024-09-16T03:51:50.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.193+0000 7f9ad0400640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:51:50.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.193+0000 7f9ad0400640 1 -- 172.21.15.42:0/2975214975 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ac81203a0 con 0x7f9ac806c810 2024-09-16T03:51:50.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.193+0000 7f9ad0400640 1 librados: init done 2024-09-16T03:51:50.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.193+0000 7f9abeffd640 1 -- 172.21.15.42:0/2975214975 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f9ac401a070 con 0x7f9ac806c810 2024-09-16T03:51:50.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.193+0000 7f9ad0400640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:51:50.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.193+0000 7f9ad0400640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:51:50.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.193+0000 7f9ad0400640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:51:50.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.193+0000 7f9ad0400640 1 -- 172.21.15.42:0/2975214975 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ac81377c0 con 0x7f9ac806c810 2024-09-16T03:51:50.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.197+0000 7f9abeffd640 1 --2- 172.21.15.42:0/2975214975 >> v2:172.21.15.154:6812/254845342 conn(0x7f9aac037810 0x7f9aac039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:50.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.197+0000 7f9abeffd640 1 -- 172.21.15.42:0/2975214975 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(325..325 src has 286..325) ==== 5312+0+0 (secure 0 0 0) 0x7f9ac4048a20 con 0x7f9ac806c810 2024-09-16T03:51:50.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.197+0000 7f9abeffd640 10 client.814193.objecter ms_dispatch 0x7f9ac8000ba0 osd_map(325..325 src has 286..325) 2024-09-16T03:51:50.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.197+0000 7f9abeffd640 3 client.814193.objecter handle_osd_map got epochs [325,325] > 0 2024-09-16T03:51:50.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.197+0000 7f9abeffd640 3 client.814193.objecter handle_osd_map decoding full epoch 325 2024-09-16T03:51:50.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.197+0000 7f9abeffd640 20 client.814193.objecter dump_active .. 0 homeless 2024-09-16T03:51:50.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.197+0000 7f9acd99b640 1 --2- 172.21.15.42:0/2975214975 >> v2:172.21.15.154:6812/254845342 conn(0x7f9aac037810 0x7f9aac039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:50.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.197+0000 7f9acd99b640 10 monclient: get_auth_request con 0x7f9aac037810 auth_method 0 2024-09-16T03:51:50.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.197+0000 7f9acd99b640 1 --2- 172.21.15.42:0/2975214975 >> v2:172.21.15.154:6812/254845342 conn(0x7f9aac037810 0x7f9aac039cd0 secure :-1 s=READY pgs=3566 cs=0 l=1 rev1=1 crypto rx=0x7f9ac813bfc0 tx=0x7f9ab8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:51:50.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.197+0000 7f9abeffd640 10 client.814193.objecter ms_handle_connect 0x7f9aac037810 2024-09-16T03:51:50.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.201+0000 7f9abeffd640 1 -- 172.21.15.42:0/2975214975 <== mon.2 v2:172.21.15.42:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f9ac401a520 con 0x7f9ac806c810 2024-09-16T03:51:50.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.201+0000 7f9abeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:51:50.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.201+0000 7f9abeffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:51:50.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.429+0000 7f9ad0400640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:51:50.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.429+0000 7f9ad0400640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:51:50.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.429+0000 7f9ad0400640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:51:50.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.429+0000 7f9ad0400640 1 -- 172.21.15.42:0/2975214975 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f9ac811ef50 con 0x7f9ac806c810 2024-09-16T03:51:50.438 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.433+0000 7f9abeffd640 1 -- 172.21.15.42:0/2975214975 <== mon.2 v2:172.21.15.42:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1555 (secure 0 0 0) 0x7f9ac4007250 con 0x7f9ac806c810 2024-09-16T03:51:50.438 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.433+0000 7f9abeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:51:50.438 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.433+0000 7f9abeffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:51:50.438 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":82,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":22,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:51:50.441 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.433+0000 7f9ad0400640 10 librados: watch_flush enter 2024-09-16T03:51:50.441 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.433+0000 7f9ad0400640 10 librados: watch_flush exit 2024-09-16T03:51:50.441 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.433+0000 7f9ad0400640 20 client.814193.objecter shutdown clearing up homeless session... 2024-09-16T03:51:50.441 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.433+0000 7f9ad0400640 10 client.814193.objecter successfully canceled tick 2024-09-16T03:51:50.441 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.433+0000 7f9ad0400640 1 -- 172.21.15.42:0/2975214975 >> v2:172.21.15.154:6812/254845342 conn(0x7f9aac037810 msgr2=0x7f9aac039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:50.441 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.433+0000 7f9ad0400640 1 --2- 172.21.15.42:0/2975214975 >> v2:172.21.15.154:6812/254845342 conn(0x7f9aac037810 0x7f9aac039cd0 secure :-1 s=READY pgs=3566 cs=0 l=1 rev1=1 crypto rx=0x7f9ac813bfc0 tx=0x7f9ab8008040 comp rx=0 tx=0).stop 2024-09-16T03:51:50.441 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.433+0000 7f9ad0400640 10 monclient: shutdown 2024-09-16T03:51:50.441 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.433+0000 7f9ad0400640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:51:50.441 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.433+0000 7f9ad0400640 1 -- 172.21.15.42:0/2975214975 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9ac806c810 msgr2=0x7f9ac8135f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:50.441 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.433+0000 7f9ad0400640 1 --2- 172.21.15.42:0/2975214975 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9ac806c810 0x7f9ac8135f80 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f9ac400ca80 tx=0x7f9ac400cf40 comp rx=0 tx=0).stop 2024-09-16T03:51:50.441 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.433+0000 7f9ad0400640 1 -- 172.21.15.42:0/2975214975 shutdown_connections 2024-09-16T03:51:50.441 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.433+0000 7f9ad0400640 1 --2- 172.21.15.42:0/2975214975 >> v2:172.21.15.154:6812/254845342 conn(0x7f9aac037810 0x7f9aac039cd0 unknown :-1 s=CLOSED pgs=3566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:50.442 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.433+0000 7f9ad0400640 1 --2- 172.21.15.42:0/2975214975 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9ac813c8e0 0x7f9ac811d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:50.442 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.433+0000 7f9ad0400640 1 --2- 172.21.15.42:0/2975214975 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9ac81373e0 0x7f9ac811ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:50.442 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.433+0000 7f9ad0400640 1 --2- 172.21.15.42:0/2975214975 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9ac806c810 0x7f9ac8135f80 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:50.442 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.433+0000 7f9ad0400640 1 -- 172.21.15.42:0/2975214975 >> 172.21.15.42:0/2975214975 conn(0x7f9ac80a6e80 msgr2=0x7f9ac813dd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:51:50.442 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.433+0000 7f9ad0400640 1 -- 172.21.15.42:0/2975214975 shutdown_connections 2024-09-16T03:51:50.442 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.433+0000 7f9ad0400640 1 -- 172.21.15.42:0/2975214975 wait complete. 2024-09-16T03:51:50.442 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:50.433+0000 7f9ad0400640 1 librados: shutdown 2024-09-16T03:51:53.462 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:51:53.598 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:53.593+0000 7f00ff00c640 10 monclient: get_monmap_and_config 2024-09-16T03:51:53.599 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:53.593+0000 7f00ff00c640 10 monclient: build_initial_monmap 2024-09-16T03:51:53.599 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:53.593+0000 7f00ff00c640 10 monclient: monmap: 2024-09-16T03:51:53.599 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:51:53.599 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:53.599 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:51:53.597680+0000 2024-09-16T03:51:53.599 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:51:53.597680+0000 2024-09-16T03:51:53.599 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:51:53.599 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:53.599 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:51:53.599 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:51:53.599 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:51:53.599 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:51:53.600 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:51:53.600 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:53.601 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:53.593+0000 7f00ff00c640 1 Processor -- start 2024-09-16T03:51:53.601 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:53.593+0000 7f00ff00c640 1 -- start start 2024-09-16T03:51:53.601 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:53.593+0000 7f00ff00c640 10 monclient: init 2024-09-16T03:51:53.601 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:53.593+0000 7f00ff00c640 10 monclient: _reopen_session rank -1 2024-09-16T03:51:53.601 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:53.593+0000 7f00ff00c640 10 monclient: _add_conns ranks=[1,0,4,2,3] 2024-09-16T03:51:53.602 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:53.593+0000 7f00ff00c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f00f8130b20 addr v1:172.21.15.154:6789/0 2024-09-16T03:51:53.602 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:53.593+0000 7f00ff00c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f00f809fdb0 addr v1:172.21.15.42:6789/0 2024-09-16T03:51:53.602 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:53.593+0000 7f00ff00c640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f00f80a22e0 0x7f00f813d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:53.602 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:53.593+0000 7f00ff00c640 10 monclient(hunting): picked mon.noname-c con 0x7f00f80a22e0 addr v2:172.21.15.154:3300/0 2024-09-16T03:51:53.602 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:53.593+0000 7f00ff00c640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f00f8131610 con 0x7f00f809fdb0 2024-09-16T03:51:53.602 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:53.593+0000 7f00ff00c640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f00f80a1900 con 0x7f00f8130b20 2024-09-16T03:51:53.602 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:53.593+0000 7f00ff00c640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:53.602 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:53.593+0000 7f00ff00c640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f00f80a1340 con 0x7f00f80a22e0 2024-09-16T03:51:53.602 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:53.593+0000 7f00ff00c640 10 monclient(hunting): _renew_subs 2024-09-16T03:51:53.602 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:53.593+0000 7f00ff00c640 10 monclient(hunting): authenticate will time out at 1560.567282s 2024-09-16T03:51:53.602 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:53.593+0000 7f00f7fff640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f00f809fdb0 0x7f00f813a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:52306/0 (socket says 172.21.15.42:52306) 2024-09-16T03:51:53.602 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:53.593+0000 7f00f7fff640 1 -- 172.21.15.42:0/2565810496 learned_addr learned my addr 172.21.15.42:0/2565810496 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:51:53.602 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:53.593+0000 7f00fd5a9640 1 --2- 172.21.15.42:0/2565810496 >> v2:172.21.15.154:3300/0 conn(0x7f00f80a22e0 0x7f00f813d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:53.603 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:53.593+0000 7f00fd5a9640 10 monclient(hunting): get_auth_request con 0x7f00f80a22e0 auth_method 0 2024-09-16T03:51:53.603 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:53.593+0000 7f00fd5a9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:53.603 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:53.593+0000 7f00fd5a9640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:53.603 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:53.593+0000 7f00fd5a9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:51:53.603 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:53.593+0000 7f00fd5a9640 1 -- 172.21.15.42:0/2565810496 >> v2:172.21.15.154:3300/0 conn(0x7f00f80a22e0 msgr2=0x7f00f813d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T03:51:53.603 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:53.597+0000 7f00fd5a9640 1 -- 172.21.15.42:0/2565810496 >> v2:172.21.15.154:3300/0 conn(0x7f00f80a22e0 msgr2=0x7f00f813d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:51:53.603 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:53.597+0000 7f00fd5a9640 1 --2- 172.21.15.42:0/2565810496 >> v2:172.21.15.154:3300/0 conn(0x7f00f80a22e0 0x7f00f813d7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:51:53.603 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:53.597+0000 7f00fd5a9640 1 --2- 172.21.15.42:0/2565810496 >> v2:172.21.15.154:3300/0 conn(0x7f00f80a22e0 0x7f00f813d7c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:51:53.806 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:53.801+0000 7f00fd5a9640 1 --2- 172.21.15.42:0/2565810496 >> v2:172.21.15.154:3300/0 conn(0x7f00f80a22e0 0x7f00f813d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:53.806 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:53.801+0000 7f00fd5a9640 10 monclient(hunting): get_auth_request con 0x7f00f80a22e0 auth_method 0 2024-09-16T03:51:53.806 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:53.801+0000 7f00fd5a9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:53.806 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:53.801+0000 7f00fd5a9640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:53.806 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:53.801+0000 7f00fd5a9640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:51:53.806 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:53.801+0000 7f00fd5a9640 1 -- 172.21.15.42:0/2565810496 >> v2:172.21.15.154:3300/0 conn(0x7f00f80a22e0 msgr2=0x7f00f813d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T03:51:53.806 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:53.801+0000 7f00fd5a9640 1 -- 172.21.15.42:0/2565810496 >> v2:172.21.15.154:3300/0 conn(0x7f00f80a22e0 msgr2=0x7f00f813d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:51:53.807 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:53.801+0000 7f00fd5a9640 1 --2- 172.21.15.42:0/2565810496 >> v2:172.21.15.154:3300/0 conn(0x7f00f80a22e0 0x7f00f813d7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:51:53.807 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:53.801+0000 7f00fd5a9640 1 --2- 172.21.15.42:0/2565810496 >> v2:172.21.15.154:3300/0 conn(0x7f00f80a22e0 0x7f00f813d7c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T03:51:54.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:54.205+0000 7f00fd5a9640 1 --2- 172.21.15.42:0/2565810496 >> v2:172.21.15.154:3300/0 conn(0x7f00f80a22e0 0x7f00f813d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:54.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:54.205+0000 7f00fd5a9640 10 monclient(hunting): get_auth_request con 0x7f00f80a22e0 auth_method 0 2024-09-16T03:51:54.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:54.205+0000 7f00fd5a9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:54.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:54.205+0000 7f00fd5a9640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:54.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:54.205+0000 7f00fd5a9640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:51:54.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:54.205+0000 7f00fd5a9640 1 -- 172.21.15.42:0/2565810496 >> v2:172.21.15.154:3300/0 conn(0x7f00f80a22e0 msgr2=0x7f00f813d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T03:51:54.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:54.205+0000 7f00fd5a9640 1 -- 172.21.15.42:0/2565810496 >> v2:172.21.15.154:3300/0 conn(0x7f00f80a22e0 msgr2=0x7f00f813d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:51:54.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:54.205+0000 7f00fd5a9640 1 --2- 172.21.15.42:0/2565810496 >> v2:172.21.15.154:3300/0 conn(0x7f00f80a22e0 0x7f00f813d7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:51:54.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:54.205+0000 7f00fd5a9640 1 --2- 172.21.15.42:0/2565810496 >> v2:172.21.15.154:3300/0 conn(0x7f00f80a22e0 0x7f00f813d7c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T03:51:55.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:55.009+0000 7f00fd5a9640 1 --2- 172.21.15.42:0/2565810496 >> v2:172.21.15.154:3300/0 conn(0x7f00f80a22e0 0x7f00f813d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:55.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:55.009+0000 7f00fd5a9640 10 monclient(hunting): get_auth_request con 0x7f00f80a22e0 auth_method 0 2024-09-16T03:51:55.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:55.009+0000 7f00fd5a9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:55.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:55.009+0000 7f00fd5a9640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:55.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:55.009+0000 7f00fd5a9640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:51:55.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:55.009+0000 7f00fd5a9640 1 -- 172.21.15.42:0/2565810496 >> v2:172.21.15.154:3300/0 conn(0x7f00f80a22e0 msgr2=0x7f00f813d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T03:51:55.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:55.009+0000 7f00fd5a9640 1 -- 172.21.15.42:0/2565810496 >> v2:172.21.15.154:3300/0 conn(0x7f00f80a22e0 msgr2=0x7f00f813d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:51:55.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:55.009+0000 7f00fd5a9640 1 --2- 172.21.15.42:0/2565810496 >> v2:172.21.15.154:3300/0 conn(0x7f00f80a22e0 0x7f00f813d7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:51:55.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:55.009+0000 7f00fd5a9640 1 --2- 172.21.15.42:0/2565810496 >> v2:172.21.15.154:3300/0 conn(0x7f00f80a22e0 0x7f00f813d7c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T03:51:56.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.425+0000 7f00f77fe640 1 -- 172.21.15.42:0/2565810496 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4273701618 0 0) 0x7f00e8001120 con 0x7f00f8130b20 2024-09-16T03:51:56.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.425+0000 7f00f77fe640 10 monclient(hunting): _negotiate 2024-09-16T03:51:56.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.425+0000 7f00f77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:56.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.425+0000 7f00f77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:51:56.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.425+0000 7f00f77fe640 10 monclient(hunting): my global_id is 794377 2024-09-16T03:51:56.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.425+0000 7f00f77fe640 1 -- 172.21.15.42:0/2565810496 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f00e0003530 con 0x7f00f8130b20 2024-09-16T03:51:56.432 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.425+0000 7f00f77fe640 1 -- 172.21.15.42:0/2565810496 <== mon.1 v1:172.21.15.154:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2088325370 0 0) 0x7f00e0003530 con 0x7f00f8130b20 2024-09-16T03:51:56.432 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.425+0000 7f00f77fe640 1 -- 172.21.15.42:0/2565810496 >> v2:172.21.15.154:3300/0 conn(0x7f00f80a22e0 msgr2=0x7f00f813d7c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:51:56.432 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.425+0000 7f00f77fe640 1 --2- 172.21.15.42:0/2565810496 >> v2:172.21.15.154:3300/0 conn(0x7f00f80a22e0 0x7f00f813d7c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:56.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.425+0000 7f00f77fe640 1 -- 172.21.15.42:0/2565810496 >> v1:172.21.15.42:6789/0 conn(0x7f00f809fdb0 legacy=0x7f00f813a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:56.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.425+0000 7f00f77fe640 10 monclient: _finish_hunting 0 2024-09-16T03:51:56.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.425+0000 7f00f77fe640 1 monclient: found mon.noname-c-legacy 2024-09-16T03:51:56.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.425+0000 7f00f77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:51:56.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.425+0000 7f00f77fe640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.154:6789/0 2024-09-16T03:51:56.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.425+0000 7f00f77fe640 1 -- 172.21.15.42:0/2565810496 --> v1:172.21.15.154:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00f80a1620 con 0x7f00f8130b20 2024-09-16T03:51:56.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.425+0000 7f00f77fe640 10 monclient: _finish_auth 0 2024-09-16T03:51:56.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.425+0000 7f00f77fe640 10 monclient: _check_auth_tickets 2024-09-16T03:51:56.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.425+0000 7f00f77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:51:56.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.425+0000 7f00f77fe640 1 -- 172.21.15.42:0/2565810496 <== mon.1 v1:172.21.15.154:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2177413734 0 0) 0x7f00e8001850 con 0x7f00f8130b20 2024-09-16T03:51:56.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.425+0000 7f00ff00c640 5 monclient: authenticate success, global_id 794377 2024-09-16T03:51:56.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.425+0000 7f00ff00c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:51:56.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.425+0000 7f00f77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:51:56.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.425+0000 7f00f77fe640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-16T03:51:56.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.425+0000 7f00f77fe640 10 monclient: dump: 2024-09-16T03:51:56.434 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:51:56.434 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:56.434 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:56.434 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:56.434 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:51:56.434 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:56.434 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:51:56.434 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:51:56.434 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:51:56.434 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:56.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.425+0000 7f00f77fe640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] but i'm connected to v1:172.21.15.154:6789/0, reconnecting 2024-09-16T03:51:56.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.425+0000 7f00f77fe640 10 monclient: _reopen_session rank -1 2024-09-16T03:51:56.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.425+0000 7f00f77fe640 1 -- 172.21.15.42:0/2565810496 >> v1:172.21.15.154:6789/0 conn(0x7f00f8130b20 legacy=0x7f00f8130f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:56.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.425+0000 7f00f77fe640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T03:51:56.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.425+0000 7f00f77fe640 1 --2- 172.21.15.42:0/2565810496 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f00e0003a10 0x7f00e0003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:56.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.425+0000 7f00f77fe640 10 monclient(hunting): picked mon.b con 0x7f00e0003a10 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:51:56.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.425+0000 7f00f77fe640 1 --2- 172.21.15.42:0/2565810496 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f00e0004810 0x7f00e0004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:56.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.425+0000 7f00f77fe640 10 monclient(hunting): picked mon.c con 0x7f00e0004810 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:51:56.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.425+0000 7f00f77fe640 1 --2- 172.21.15.42:0/2565810496 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f00e00053d0 0x7f00e000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:56.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.425+0000 7f00f77fe640 10 monclient(hunting): picked mon.a con 0x7f00e00053d0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:51:56.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.425+0000 7f00f77fe640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:56.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.425+0000 7f00f77fe640 1 -- 172.21.15.42:0/2565810496 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f00f80a1340 con 0x7f00e00053d0 2024-09-16T03:51:56.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.425+0000 7f00f77fe640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:56.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.425+0000 7f00f77fe640 1 -- 172.21.15.42:0/2565810496 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f00e000c2b0 con 0x7f00e0003a10 2024-09-16T03:51:56.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.425+0000 7f00f77fe640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:56.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.425+0000 7f00f77fe640 1 -- 172.21.15.42:0/2565810496 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f00e000c4b0 con 0x7f00e0004810 2024-09-16T03:51:56.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.425+0000 7f00f77fe640 10 monclient(hunting): _renew_subs 2024-09-16T03:51:56.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.425+0000 7f00f77fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:51:56.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.425+0000 7f00ff00c640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-16T03:51:56.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.425+0000 7f00f7fff640 1 --2- 172.21.15.42:0/2565810496 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f00e0004810 0x7f00e0004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:56.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.425+0000 7f00fd5a9640 1 --2- 172.21.15.42:0/2565810496 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f00e00053d0 0x7f00e000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:56.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.425+0000 7f00f7fff640 10 monclient(hunting): get_auth_request con 0x7f00e0004810 auth_method 0 2024-09-16T03:51:56.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.425+0000 7f00f7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:56.437 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.425+0000 7f00f7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:56.437 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.425+0000 7f00f7fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:51:56.437 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.425+0000 7f00fd5a9640 10 monclient(hunting): get_auth_request con 0x7f00e00053d0 auth_method 0 2024-09-16T03:51:56.437 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.425+0000 7f00fcda8640 1 --2- 172.21.15.42:0/2565810496 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f00e0003a10 0x7f00e0003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:56.437 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.425+0000 7f00fd5a9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:56.437 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.425+0000 7f00fd5a9640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:56.437 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.425+0000 7f00fd5a9640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:51:56.437 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.425+0000 7f00fcda8640 10 monclient(hunting): get_auth_request con 0x7f00e0003a10 auth_method 0 2024-09-16T03:51:56.437 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.425+0000 7f00fcda8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:56.437 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.425+0000 7f00fcda8640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:56.437 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.425+0000 7f00fcda8640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:51:56.437 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.425+0000 7f00f7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:51:56.437 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.425+0000 7f00f7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:51:56.438 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.425+0000 7f00f7fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T03:51:56.438 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.425+0000 7f00fd5a9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:51:56.438 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.425+0000 7f00fd5a9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:51:56.438 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.425+0000 7f00fd5a9640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T03:51:56.438 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.425+0000 7f00fcda8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:51:56.438 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.425+0000 7f00fcda8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:51:56.438 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.425+0000 7f00fcda8640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T03:51:56.438 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.425+0000 7f00fd5a9640 10 monclient(hunting): handle_auth_done global_id 794377 payload 293 2024-09-16T03:51:56.438 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.425+0000 7f00fd5a9640 1 -- 172.21.15.42:0/2565810496 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f00e0004810 msgr2=0x7f00e0004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:51:56.438 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.425+0000 7f00fd5a9640 1 --2- 172.21.15.42:0/2565810496 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f00e0004810 0x7f00e0004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:56.438 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.425+0000 7f00fd5a9640 1 -- 172.21.15.42:0/2565810496 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f00e0003a10 msgr2=0x7f00e0003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:51:56.438 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.425+0000 7f00fd5a9640 1 --2- 172.21.15.42:0/2565810496 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f00e0003a10 0x7f00e0003e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:56.439 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.425+0000 7f00fd5a9640 10 monclient: _finish_hunting 0 2024-09-16T03:51:56.439 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.425+0000 7f00fd5a9640 1 monclient: found mon.a 2024-09-16T03:51:56.439 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.425+0000 7f00fd5a9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:51:56.439 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.425+0000 7f00fd5a9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:51:56.439 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.425+0000 7f00fd5a9640 1 -- 172.21.15.42:0/2565810496 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00e0003530 con 0x7f00e00053d0 2024-09-16T03:51:56.440 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.425+0000 7f00fd5a9640 10 monclient: _finish_auth 0 2024-09-16T03:51:56.440 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.425+0000 7f00fd5a9640 10 monclient: _check_auth_tickets 2024-09-16T03:51:56.440 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.425+0000 7f00fd5a9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:51:56.440 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.433+0000 7f00fd5a9640 1 -- 172.21.15.42:0/2565810496 reap_dead start 2024-09-16T03:51:56.440 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.433+0000 7f00fd5a9640 1 --2- 172.21.15.42:0/2565810496 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f00e00053d0 0x7f00e000b9d0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f00f00099c0 tx=0x7f00f00213f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6dcf5366d5d949c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:51:56.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.481+0000 7f00f77fe640 1 -- 172.21.15.42:0/2565810496 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00f00b7070 con 0x7f00e00053d0 2024-09-16T03:51:56.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.481+0000 7f00f77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:51:56.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.481+0000 7f00f77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:51:56.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.481+0000 7f00f77fe640 10 monclient: dump: 2024-09-16T03:51:56.487 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:51:56.487 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:56.487 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:56.487 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:56.488 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:51:56.488 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:56.488 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:51:56.488 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:51:56.488 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:51:56.488 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:56.488 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.481+0000 7f00ff00c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:51:56.488 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.481+0000 7f00f77fe640 1 -- 172.21.15.42:0/2565810496 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f00f00b2410 con 0x7f00e00053d0 2024-09-16T03:51:56.488 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.481+0000 7f00f77fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:51:56.488 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.481+0000 7f00f77fe640 1 -- 172.21.15.42:0/2565810496 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00f0009640 con 0x7f00e00053d0 2024-09-16T03:51:56.488 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.481+0000 7f00f77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:51:56.488 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.481+0000 7f00f77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:51:56.488 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.481+0000 7f00f77fe640 10 monclient: dump: 2024-09-16T03:51:56.489 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:51:56.489 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:56.489 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:56.489 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:56.489 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:51:56.489 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:56.489 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:51:56.489 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:51:56.489 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:51:56.489 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:56.489 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.481+0000 7f00ff00c640 10 monclient: get_monmap_and_config success 2024-09-16T03:51:56.489 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.481+0000 7f00ff00c640 4 set_mon_vals no callback set 2024-09-16T03:51:56.491 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.481+0000 7f00ff00c640 10 monclient: shutdown 2024-09-16T03:51:56.491 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.481+0000 7f00ff00c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:51:56.491 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.481+0000 7f00ff00c640 1 -- 172.21.15.42:0/2565810496 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f00e00053d0 msgr2=0x7f00e000b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:56.491 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.481+0000 7f00ff00c640 1 --2- 172.21.15.42:0/2565810496 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f00e00053d0 0x7f00e000b9d0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f00f00099c0 tx=0x7f00f00213f0 comp rx=0 tx=0).stop 2024-09-16T03:51:56.491 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.481+0000 7f00ff00c640 1 -- 172.21.15.42:0/2565810496 shutdown_connections 2024-09-16T03:51:56.491 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.481+0000 7f00ff00c640 1 --2- 172.21.15.42:0/2565810496 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f00e00053d0 0x7f00e000b9d0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:56.491 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.481+0000 7f00ff00c640 1 -- 172.21.15.42:0/2565810496 >> 172.21.15.42:0/2565810496 conn(0x7f00f80a6210 msgr2=0x7f00f80a6620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:51:56.491 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.481+0000 7f00ff00c640 1 -- 172.21.15.42:0/2565810496 shutdown_connections 2024-09-16T03:51:56.491 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.481+0000 7f00ff00c640 1 -- 172.21.15.42:0/2565810496 wait complete. 2024-09-16T03:51:56.491 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.485+0000 7f00ff00c640 10 monclient: build_initial_monmap 2024-09-16T03:51:56.491 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.485+0000 7f00ff00c640 10 monclient: monmap: 2024-09-16T03:51:56.492 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:51:56.492 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:51:56.492 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:51:56.492 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:51:56.492 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:51:56.492 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:56.492 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:51:56.492 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:51:56.492 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:51:56.492 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:56.492 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.485+0000 7f00ff00c640 1 librados: starting msgr at 2024-09-16T03:51:56.492 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.485+0000 7f00ff00c640 1 librados: starting objecter 2024-09-16T03:51:56.493 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.485+0000 7f00ff00c640 1 Processor -- start 2024-09-16T03:51:56.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.485+0000 7f00ff00c640 1 -- start start 2024-09-16T03:51:56.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.485+0000 7f00ff00c640 1 librados: setting wanted keys 2024-09-16T03:51:56.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.485+0000 7f00ff00c640 1 librados: calling monclient init 2024-09-16T03:51:56.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.485+0000 7f00ff00c640 10 monclient: init 2024-09-16T03:51:56.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.485+0000 7f00ff00c640 10 monclient: _reopen_session rank -1 2024-09-16T03:51:56.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.485+0000 7f00ff00c640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:51:56.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.489+0000 7f00ff00c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f00e00053d0 0x7f00f80701a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:56.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.489+0000 7f00ff00c640 10 monclient(hunting): picked mon.noname-b con 0x7f00e00053d0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:51:56.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.489+0000 7f00ff00c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f00f8072740 0x7f00f80706e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:56.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.489+0000 7f00ff00c640 10 monclient(hunting): picked mon.noname-a con 0x7f00f8072740 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:51:56.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.489+0000 7f00ff00c640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f00f8070c20 0x7f00f81ca760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:56.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.489+0000 7f00ff00c640 10 monclient(hunting): picked mon.noname-c con 0x7f00f8070c20 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:51:56.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.489+0000 7f00ff00c640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:56.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.489+0000 7f00ff00c640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f00f8131290 con 0x7f00f8072740 2024-09-16T03:51:56.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.489+0000 7f00ff00c640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:56.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.489+0000 7f00ff00c640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f00f8131110 con 0x7f00e00053d0 2024-09-16T03:51:56.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.489+0000 7f00ff00c640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:56.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.489+0000 7f00ff00c640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f00f8131410 con 0x7f00f8070c20 2024-09-16T03:51:56.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.489+0000 7f00ff00c640 10 monclient(hunting): _renew_subs 2024-09-16T03:51:56.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.489+0000 7f00ff00c640 10 monclient(hunting): authenticate will time out at 1563.460247s 2024-09-16T03:51:56.498 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.489+0000 7f00fd5a9640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f00f8070c20 0x7f00f81ca760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:56.498 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.489+0000 7f00fd5a9640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f00f8070c20 0x7f00f81ca760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:41428/0 (socket says 172.21.15.42:41428) 2024-09-16T03:51:56.498 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.489+0000 7f00fd5a9640 1 -- 172.21.15.42:0/2025084604 learned_addr learned my addr 172.21.15.42:0/2025084604 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:51:56.498 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.489+0000 7f00fd5a9640 10 monclient(hunting): get_auth_request con 0x7f00f8070c20 auth_method 0 2024-09-16T03:51:56.498 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.489+0000 7f00fd5a9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:56.498 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.489+0000 7f00fd5a9640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:56.498 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.489+0000 7f00fd5a9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:51:56.498 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.489+0000 7f00fcda8640 1 --2- 172.21.15.42:0/2025084604 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f00e00053d0 0x7f00f80701a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:56.498 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.489+0000 7f00f7fff640 1 --2- 172.21.15.42:0/2025084604 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f00f8072740 0x7f00f80706e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:56.498 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.489+0000 7f00fcda8640 10 monclient(hunting): get_auth_request con 0x7f00e00053d0 auth_method 0 2024-09-16T03:51:56.498 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.489+0000 7f00fcda8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:56.498 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.489+0000 7f00fcda8640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:56.498 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.489+0000 7f00fcda8640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:51:56.499 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.489+0000 7f00f7fff640 10 monclient(hunting): get_auth_request con 0x7f00f8072740 auth_method 0 2024-09-16T03:51:56.499 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.489+0000 7f00f7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:56.499 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.489+0000 7f00f7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:56.499 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.489+0000 7f00f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:51:56.499 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.489+0000 7f00fd5a9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:51:56.499 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.489+0000 7f00fd5a9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:51:56.499 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.489+0000 7f00fd5a9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:51:56.499 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.489+0000 7f00fcda8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:51:56.499 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.489+0000 7f00fcda8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:51:56.499 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.489+0000 7f00fcda8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:51:56.499 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.489+0000 7f00f7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:51:56.499 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.489+0000 7f00f7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:51:56.500 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.489+0000 7f00f7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:51:56.500 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.489+0000 7f00fd5a9640 10 monclient(hunting): handle_auth_done global_id 814196 payload 824 2024-09-16T03:51:56.500 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.489+0000 7f00fd5a9640 1 -- 172.21.15.42:0/2025084604 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f00e00053d0 msgr2=0x7f00f80701a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:56.500 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.489+0000 7f00fd5a9640 1 --2- 172.21.15.42:0/2025084604 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f00e00053d0 0x7f00f80701a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:56.500 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.489+0000 7f00fd5a9640 1 -- 172.21.15.42:0/2025084604 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f00f8072740 msgr2=0x7f00f80706e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:56.500 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.489+0000 7f00fd5a9640 1 --2- 172.21.15.42:0/2025084604 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f00f8072740 0x7f00f80706e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:56.500 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.489+0000 7f00fd5a9640 10 monclient: _finish_hunting 0 2024-09-16T03:51:56.500 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.489+0000 7f00fd5a9640 1 monclient: found mon.noname-c 2024-09-16T03:51:56.500 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.489+0000 7f00fd5a9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:51:56.500 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.489+0000 7f00fd5a9640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:51:56.500 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.489+0000 7f00fd5a9640 1 -- 172.21.15.42:0/2025084604 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00f80a4c70 con 0x7f00f8070c20 2024-09-16T03:51:56.500 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.489+0000 7f00fd5a9640 1 --2- 172.21.15.42:0/2025084604 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f00f8070c20 0x7f00f81ca760 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f00f0021220 tx=0x7f00f001e190 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:51:56.500 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.489+0000 7f00f57fa640 10 client.?.objecter ms_handle_connect 0x7f00f8070c20 2024-09-16T03:51:56.501 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.489+0000 7f00f57fa640 10 client.?.objecter resend_mon_ops 2024-09-16T03:51:56.501 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.489+0000 7f00f57fa640 1 -- 172.21.15.42:0/2025084604 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00f00b7040 con 0x7f00f8070c20 2024-09-16T03:51:56.501 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.489+0000 7f00f57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:51:56.501 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.489+0000 7f00f57fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:51:56.501 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.489+0000 7f00f57fa640 10 monclient: dump: 2024-09-16T03:51:56.501 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:51:56.501 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:56.501 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:56.501 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:56.501 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:51:56.501 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:56.501 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:51:56.501 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:51:56.502 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:51:56.502 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:56.502 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.489+0000 7f00f57fa640 10 monclient: _finish_auth 0 2024-09-16T03:51:56.502 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.489+0000 7f00f57fa640 10 monclient: _check_auth_tickets 2024-09-16T03:51:56.502 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.489+0000 7f00f57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:51:56.502 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.489+0000 7f00ff00c640 5 monclient: authenticate success, global_id 814196 2024-09-16T03:51:56.502 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.489+0000 7f00ff00c640 10 monclient: _renew_subs 2024-09-16T03:51:56.502 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.489+0000 7f00ff00c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:51:56.502 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.489+0000 7f00ff00c640 1 -- 172.21.15.42:0/2025084604 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00f8073630 con 0x7f00f8070c20 2024-09-16T03:51:56.502 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.489+0000 7f00f57fa640 1 -- 172.21.15.42:0/2025084604 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f00f00c2070 con 0x7f00f8070c20 2024-09-16T03:51:56.502 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.489+0000 7f00f57fa640 10 monclient: handle_config config(0 keys) 2024-09-16T03:51:56.502 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.489+0000 7f00ff00c640 10 client.814196.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:51:56.502 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.489+0000 7f00ff00c640 10 monclient: _renew_subs 2024-09-16T03:51:56.502 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.489+0000 7f00ff00c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:51:56.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.489+0000 7f00ff00c640 1 -- 172.21.15.42:0/2025084604 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f00f80720c0 con 0x7f00f8070c20 2024-09-16T03:51:56.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.489+0000 7f00ff00c640 1 librados: init done 2024-09-16T03:51:56.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.489+0000 7f00f67fc640 4 set_mon_vals no callback set 2024-09-16T03:51:56.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.489+0000 7f00f57fa640 1 -- 172.21.15.42:0/2025084604 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00f00b2e20 con 0x7f00f8070c20 2024-09-16T03:51:56.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.489+0000 7f00f57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:51:56.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.489+0000 7f00f57fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:51:56.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.489+0000 7f00f57fa640 10 monclient: dump: 2024-09-16T03:51:56.503 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:51:56.503 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:56.503 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:56.503 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:56.503 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:51:56.503 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:56.503 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:51:56.504 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:51:56.504 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:51:56.504 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:56.505 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.489+0000 7f00ff00c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:51:56.505 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.489+0000 7f00ff00c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:51:56.505 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.489+0000 7f00ff00c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:51:56.505 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.489+0000 7f00ff00c640 1 -- 172.21.15.42:0/2025084604 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f00f809ba50 con 0x7f00f8070c20 2024-09-16T03:51:56.505 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.497+0000 7f00f57fa640 1 -- 172.21.15.42:0/2025084604 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f00f00bd450 con 0x7f00f8070c20 2024-09-16T03:51:56.505 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.497+0000 7f00f57fa640 1 --2- 172.21.15.42:0/2025084604 >> v2:172.21.15.154:6812/254845342 conn(0x7f00e4037810 0x7f00e4039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:56.505 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.497+0000 7f00f57fa640 1 -- 172.21.15.42:0/2025084604 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(328..328 src has 286..328) ==== 5312+0+0 (secure 0 0 0) 0x7f00f00e8ec0 con 0x7f00f8070c20 2024-09-16T03:51:56.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.497+0000 7f00f57fa640 10 client.814196.objecter ms_dispatch 0x7f00f8000ba0 osd_map(328..328 src has 286..328) 2024-09-16T03:51:56.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.497+0000 7f00f57fa640 3 client.814196.objecter handle_osd_map got epochs [328,328] > 0 2024-09-16T03:51:56.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.497+0000 7f00f57fa640 3 client.814196.objecter handle_osd_map decoding full epoch 328 2024-09-16T03:51:56.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.497+0000 7f00f57fa640 20 client.814196.objecter dump_active .. 0 homeless 2024-09-16T03:51:56.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.497+0000 7f00fcda8640 1 --2- 172.21.15.42:0/2025084604 >> v2:172.21.15.154:6812/254845342 conn(0x7f00e4037810 0x7f00e4039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:56.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.497+0000 7f00fcda8640 10 monclient: get_auth_request con 0x7f00e4037810 auth_method 0 2024-09-16T03:51:56.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.497+0000 7f00f57fa640 10 client.814196.objecter ms_handle_connect 0x7f00e4037810 2024-09-16T03:51:56.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.497+0000 7f00fcda8640 1 --2- 172.21.15.42:0/2025084604 >> v2:172.21.15.154:6812/254845342 conn(0x7f00e4037810 0x7f00e4039cd0 secure :-1 s=READY pgs=3567 cs=0 l=1 rev1=1 crypto rx=0x7f00e8001850 tx=0x7f00e8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:51:56.513 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.509+0000 7f00f57fa640 1 -- 172.21.15.42:0/2025084604 <== mon.2 v2:172.21.15.42:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f00f001b140 con 0x7f00f8070c20 2024-09-16T03:51:56.513 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.509+0000 7f00f57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:51:56.513 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.509+0000 7f00f57fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:51:56.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.741+0000 7f00ff00c640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:51:56.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.741+0000 7f00ff00c640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:51:56.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.741+0000 7f00ff00c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:51:56.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.741+0000 7f00ff00c640 1 -- 172.21.15.42:0/2025084604 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f00f806b490 con 0x7f00f8070c20 2024-09-16T03:51:56.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.745+0000 7f00f57fa640 1 -- 172.21.15.42:0/2025084604 <== mon.2 v2:172.21.15.42:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7f00f00bc3b0 con 0x7f00f8070c20 2024-09-16T03:51:56.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.745+0000 7f00f57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:51:56.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.745+0000 7f00f57fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:51:56.749 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":86,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:51:56.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.745+0000 7f00ff00c640 10 librados: watch_flush enter 2024-09-16T03:51:56.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.745+0000 7f00ff00c640 10 librados: watch_flush exit 2024-09-16T03:51:56.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.745+0000 7f00ff00c640 20 client.814196.objecter shutdown clearing up homeless session... 2024-09-16T03:51:56.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.745+0000 7f00ff00c640 10 client.814196.objecter successfully canceled tick 2024-09-16T03:51:56.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.745+0000 7f00ff00c640 1 -- 172.21.15.42:0/2025084604 >> v2:172.21.15.154:6812/254845342 conn(0x7f00e4037810 msgr2=0x7f00e4039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:56.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.745+0000 7f00ff00c640 1 --2- 172.21.15.42:0/2025084604 >> v2:172.21.15.154:6812/254845342 conn(0x7f00e4037810 0x7f00e4039cd0 secure :-1 s=READY pgs=3567 cs=0 l=1 rev1=1 crypto rx=0x7f00e8001850 tx=0x7f00e8009040 comp rx=0 tx=0).stop 2024-09-16T03:51:56.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.745+0000 7f00ff00c640 10 monclient: shutdown 2024-09-16T03:51:56.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.745+0000 7f00ff00c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:51:56.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.745+0000 7f00ff00c640 1 -- 172.21.15.42:0/2025084604 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f00f8070c20 msgr2=0x7f00f81ca760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:56.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.745+0000 7f00ff00c640 1 --2- 172.21.15.42:0/2025084604 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f00f8070c20 0x7f00f81ca760 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f00f0021220 tx=0x7f00f001e190 comp rx=0 tx=0).stop 2024-09-16T03:51:56.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.745+0000 7f00ff00c640 1 -- 172.21.15.42:0/2025084604 shutdown_connections 2024-09-16T03:51:56.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.745+0000 7f00ff00c640 1 --2- 172.21.15.42:0/2025084604 >> v2:172.21.15.154:6812/254845342 conn(0x7f00e4037810 0x7f00e4039cd0 unknown :-1 s=CLOSED pgs=3567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:56.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.745+0000 7f00ff00c640 1 --2- 172.21.15.42:0/2025084604 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f00f8070c20 0x7f00f81ca760 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:56.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.745+0000 7f00ff00c640 1 --2- 172.21.15.42:0/2025084604 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f00f8072740 0x7f00f80706e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:56.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.745+0000 7f00ff00c640 1 --2- 172.21.15.42:0/2025084604 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f00e00053d0 0x7f00f80701a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:56.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.745+0000 7f00ff00c640 1 -- 172.21.15.42:0/2025084604 >> 172.21.15.42:0/2025084604 conn(0x7f00f80a6210 msgr2=0x7f00f81317e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:51:56.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.745+0000 7f00ff00c640 1 -- 172.21.15.42:0/2025084604 shutdown_connections 2024-09-16T03:51:56.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.745+0000 7f00ff00c640 1 -- 172.21.15.42:0/2025084604 wait complete. 2024-09-16T03:51:56.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.745+0000 7f00ff00c640 1 librados: shutdown 2024-09-16T03:51:56.767 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T03:51:56.768 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T03:51:56.768 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T03:51:56.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.877+0000 7f48f4d26640 10 monclient: get_monmap_and_config 2024-09-16T03:51:56.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.877+0000 7f48f4d26640 10 monclient: build_initial_monmap 2024-09-16T03:51:56.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.877+0000 7f48f4d26640 10 monclient: monmap: 2024-09-16T03:51:56.885 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:51:56.885 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:56.885 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:51:56.883955+0000 2024-09-16T03:51:56.885 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:51:56.883955+0000 2024-09-16T03:51:56.885 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:51:56.885 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:56.885 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:51:56.886 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:51:56.886 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:51:56.886 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:51:56.886 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:51:56.886 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:56.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.881+0000 7f48f4d26640 1 Processor -- start 2024-09-16T03:51:56.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.881+0000 7f48f4d26640 1 -- start start 2024-09-16T03:51:56.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.881+0000 7f48f4d26640 10 monclient: init 2024-09-16T03:51:56.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.881+0000 7f48f4d26640 10 monclient: _reopen_session rank -1 2024-09-16T03:51:56.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.881+0000 7f48f4d26640 10 monclient: _add_conns ranks=[1,0,4,2,3] 2024-09-16T03:51:56.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.881+0000 7f48f4d26640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f48f0130b20 addr v1:172.21.15.154:6789/0 2024-09-16T03:51:56.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.881+0000 7f48f4d26640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f48f00a2250 addr v1:172.21.15.42:6789/0 2024-09-16T03:51:56.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.881+0000 7f48f4d26640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f48f013d940 0x7f48f013fd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:56.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.881+0000 7f48f4d26640 10 monclient(hunting): picked mon.noname-c con 0x7f48f013d940 addr v2:172.21.15.154:3300/0 2024-09-16T03:51:56.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.881+0000 7f48f4d26640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f48f0131610 con 0x7f48f00a2250 2024-09-16T03:51:56.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.881+0000 7f48f4d26640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f48f00a12b0 con 0x7f48f0130b20 2024-09-16T03:51:56.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.881+0000 7f48f4d26640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:56.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.881+0000 7f48f4d26640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f48f00a1590 con 0x7f48f013d940 2024-09-16T03:51:56.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.881+0000 7f48f4d26640 10 monclient(hunting): _renew_subs 2024-09-16T03:51:56.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.881+0000 7f48f4d26640 10 monclient(hunting): authenticate will time out at 1563.853154s 2024-09-16T03:51:56.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.881+0000 7f48edd9b640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f48f00a2250 0x7f48f013c220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:56964/0 (socket says 172.21.15.42:56964) 2024-09-16T03:51:56.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.881+0000 7f48edd9b640 1 -- 172.21.15.42:0/3161054744 learned_addr learned my addr 172.21.15.42:0/3161054744 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:51:56.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.881+0000 7f48eed9d640 1 --2- 172.21.15.42:0/3161054744 >> v2:172.21.15.154:3300/0 conn(0x7f48f013d940 0x7f48f013fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:56.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.881+0000 7f48eed9d640 10 monclient(hunting): get_auth_request con 0x7f48f013d940 auth_method 0 2024-09-16T03:51:56.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.881+0000 7f48eed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:56.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.881+0000 7f48eed9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:56.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.881+0000 7f48eed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:51:56.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.881+0000 7f48ed59a640 1 -- 172.21.15.42:0/3161054744 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1438121499 0 0) 0x7f48f0131610 con 0x7f48f00a2250 2024-09-16T03:51:56.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.881+0000 7f48ed59a640 10 monclient(hunting): _negotiate 2024-09-16T03:51:56.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.881+0000 7f48ed59a640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:56.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.881+0000 7f48ed59a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:51:56.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.881+0000 7f48ed59a640 10 monclient(hunting): my global_id is 824100 2024-09-16T03:51:56.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.881+0000 7f48ed59a640 1 -- 172.21.15.42:0/3161054744 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f48d4003530 con 0x7f48f00a2250 2024-09-16T03:51:56.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.881+0000 7f48eed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:51:56.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.881+0000 7f48eed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:51:56.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.881+0000 7f48eed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:51:56.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.881+0000 7f48ed59a640 1 -- 172.21.15.42:0/3161054744 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1340829468 0 0) 0x7f48d4003530 con 0x7f48f00a2250 2024-09-16T03:51:56.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.881+0000 7f48ed59a640 1 -- 172.21.15.42:0/3161054744 >> v2:172.21.15.154:3300/0 conn(0x7f48f013d940 msgr2=0x7f48f013fd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:51:56.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.881+0000 7f48ed59a640 1 --2- 172.21.15.42:0/3161054744 >> v2:172.21.15.154:3300/0 conn(0x7f48f013d940 0x7f48f013fd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:56.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.881+0000 7f48ed59a640 1 -- 172.21.15.42:0/3161054744 >> v1:172.21.15.154:6789/0 conn(0x7f48f0130b20 legacy=0x7f48f0130f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:56.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.881+0000 7f48ed59a640 10 monclient: _finish_hunting 0 2024-09-16T03:51:56.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.881+0000 7f48ed59a640 1 monclient: found mon.noname-a-legacy 2024-09-16T03:51:56.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.881+0000 7f48ed59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:51:56.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.881+0000 7f48ed59a640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.42:6789/0 2024-09-16T03:51:56.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.881+0000 7f48ed59a640 1 -- 172.21.15.42:0/3161054744 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48f00a1870 con 0x7f48f00a2250 2024-09-16T03:51:56.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.881+0000 7f48ed59a640 10 monclient: _finish_auth 0 2024-09-16T03:51:56.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.881+0000 7f48ed59a640 10 monclient: _check_auth_tickets 2024-09-16T03:51:56.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.881+0000 7f48ed59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:51:56.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.881+0000 7f48f4d26640 5 monclient: authenticate success, global_id 824100 2024-09-16T03:51:56.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.881+0000 7f48ed59a640 1 -- 172.21.15.42:0/3161054744 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2177413734 0 0) 0x7f48dc001a00 con 0x7f48f00a2250 2024-09-16T03:51:56.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.881+0000 7f48f4d26640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:51:56.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.881+0000 7f48ed59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:51:56.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.881+0000 7f48ed59a640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T03:51:56.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.881+0000 7f48ed59a640 10 monclient: dump: 2024-09-16T03:51:56.905 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:51:56.905 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:56.905 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:56.905 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:56.905 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:51:56.905 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:56.905 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:51:56.905 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:51:56.905 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:51:56.906 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:56.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.881+0000 7f48ed59a640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] but i'm connected to v1:172.21.15.42:6789/0, reconnecting 2024-09-16T03:51:56.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.881+0000 7f48ed59a640 10 monclient: _reopen_session rank -1 2024-09-16T03:51:56.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.881+0000 7f48ed59a640 1 -- 172.21.15.42:0/3161054744 >> v1:172.21.15.42:6789/0 conn(0x7f48f00a2250 legacy=0x7f48f013c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:56.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.881+0000 7f48ed59a640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:51:56.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.881+0000 7f48ed59a640 1 --2- 172.21.15.42:0/3161054744 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f48d4003a10 0x7f48d4003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:56.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.881+0000 7f48ed59a640 10 monclient(hunting): picked mon.c con 0x7f48d4003a10 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:51:56.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.881+0000 7f48ed59a640 1 --2- 172.21.15.42:0/3161054744 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f48d4004810 0x7f48d4004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:56.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.881+0000 7f48ed59a640 10 monclient(hunting): picked mon.a con 0x7f48d4004810 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:51:56.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.881+0000 7f48ed59a640 1 --2- 172.21.15.42:0/3161054744 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f48d40052f0 0x7f48d400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:56.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.881+0000 7f48ed59a640 10 monclient(hunting): picked mon.b con 0x7f48d40052f0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:51:56.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.881+0000 7f48ed59a640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:56.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.881+0000 7f48ed59a640 1 -- 172.21.15.42:0/3161054744 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f48f00a1590 con 0x7f48d4004810 2024-09-16T03:51:56.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.881+0000 7f48ed59a640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:56.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.881+0000 7f48ed59a640 1 -- 172.21.15.42:0/3161054744 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f48d400c1f0 con 0x7f48d40052f0 2024-09-16T03:51:56.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.881+0000 7f48ed59a640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:56.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.881+0000 7f48ed59a640 1 -- 172.21.15.42:0/3161054744 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f48d400c3f0 con 0x7f48d4003a10 2024-09-16T03:51:56.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.881+0000 7f48ed59a640 10 monclient(hunting): _renew_subs 2024-09-16T03:51:56.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.881+0000 7f48ed59a640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:51:56.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.881+0000 7f48f4d26640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-16T03:51:56.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.881+0000 7f48edd9b640 1 --2- 172.21.15.42:0/3161054744 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f48d4004810 0x7f48d4004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:56.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.881+0000 7f48ee59c640 1 --2- 172.21.15.42:0/3161054744 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f48d4003a10 0x7f48d4003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:56.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.881+0000 7f48edd9b640 10 monclient(hunting): get_auth_request con 0x7f48d4004810 auth_method 0 2024-09-16T03:51:56.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.881+0000 7f48edd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:56.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.881+0000 7f48edd9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:56.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.881+0000 7f48edd9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:51:56.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.881+0000 7f48ee59c640 10 monclient(hunting): get_auth_request con 0x7f48d4003a10 auth_method 0 2024-09-16T03:51:56.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.881+0000 7f48ee59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:56.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.881+0000 7f48ee59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:56.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.881+0000 7f48ee59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:51:56.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.881+0000 7f48edd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:51:56.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.881+0000 7f48edd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:51:56.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.881+0000 7f48edd9b640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T03:51:56.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.881+0000 7f48ee59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:51:56.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.881+0000 7f48ee59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:51:56.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.881+0000 7f48ee59c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T03:51:56.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.881+0000 7f48eed9d640 1 --2- 172.21.15.42:0/3161054744 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f48d40052f0 0x7f48d400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:56.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.881+0000 7f48eed9d640 10 monclient(hunting): get_auth_request con 0x7f48d40052f0 auth_method 0 2024-09-16T03:51:56.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.881+0000 7f48eed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:56.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.881+0000 7f48eed9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:56.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.881+0000 7f48eed9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:51:56.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.881+0000 7f48edd9b640 10 monclient(hunting): handle_auth_done global_id 824100 payload 293 2024-09-16T03:51:56.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.881+0000 7f48edd9b640 1 -- 172.21.15.42:0/3161054744 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f48d4003a10 msgr2=0x7f48d4003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:51:56.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.881+0000 7f48edd9b640 1 --2- 172.21.15.42:0/3161054744 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f48d4003a10 0x7f48d4003e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:56.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.881+0000 7f48edd9b640 1 -- 172.21.15.42:0/3161054744 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f48d40052f0 msgr2=0x7f48d400b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:51:56.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.881+0000 7f48edd9b640 1 --2- 172.21.15.42:0/3161054744 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f48d40052f0 0x7f48d400b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:56.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.881+0000 7f48edd9b640 10 monclient: _finish_hunting 0 2024-09-16T03:51:56.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.881+0000 7f48edd9b640 1 monclient: found mon.a 2024-09-16T03:51:56.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.881+0000 7f48edd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:51:56.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.881+0000 7f48edd9b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:51:56.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.881+0000 7f48edd9b640 1 -- 172.21.15.42:0/3161054744 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48d4003530 con 0x7f48d4004810 2024-09-16T03:51:56.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.881+0000 7f48eed9d640 1 -- 172.21.15.42:0/3161054744 reap_dead start 2024-09-16T03:51:56.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.881+0000 7f48edd9b640 10 monclient: _finish_auth 0 2024-09-16T03:51:56.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.881+0000 7f48edd9b640 10 monclient: _check_auth_tickets 2024-09-16T03:51:56.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.881+0000 7f48edd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:51:56.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.885+0000 7f48edd9b640 1 --2- 172.21.15.42:0/3161054744 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f48d4004810 0x7f48d4004c90 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f48f00a2660 tx=0x7f48dc01d1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16d6c5fe8167538 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:51:56.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.885+0000 7f48ed59a640 1 -- 172.21.15.42:0/3161054744 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48dc00a3d0 con 0x7f48d4004810 2024-09-16T03:51:56.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.885+0000 7f48ed59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:51:56.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.885+0000 7f48ed59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:51:56.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.885+0000 7f48ed59a640 10 monclient: dump: 2024-09-16T03:51:56.910 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:51:56.910 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:56.911 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:56.911 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:56.911 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:51:56.911 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:56.911 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:51:56.911 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:51:56.911 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:51:56.911 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:56.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.885+0000 7f48ed59a640 1 -- 172.21.15.42:0/3161054744 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f48dc0ad630 con 0x7f48d4004810 2024-09-16T03:51:56.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.885+0000 7f48ed59a640 10 monclient: handle_config config(0 keys) 2024-09-16T03:51:56.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.885+0000 7f48ed59a640 1 -- 172.21.15.42:0/3161054744 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48dc0ada40 con 0x7f48d4004810 2024-09-16T03:51:56.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.885+0000 7f48ed59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:51:56.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.885+0000 7f48ed59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:51:56.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.885+0000 7f48ed59a640 10 monclient: dump: 2024-09-16T03:51:56.912 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:51:56.912 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:56.912 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:56.912 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:56.912 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:51:56.912 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:56.912 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:51:56.912 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:51:56.912 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:51:56.912 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:56.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.885+0000 7f48f4d26640 10 monclient: get_monmap_and_config success 2024-09-16T03:51:56.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.885+0000 7f48f4d26640 4 set_mon_vals no callback set 2024-09-16T03:51:56.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.885+0000 7f48f4d26640 10 monclient: shutdown 2024-09-16T03:51:56.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.885+0000 7f48f4d26640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:51:56.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.885+0000 7f48f4d26640 1 -- 172.21.15.42:0/3161054744 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f48d4004810 msgr2=0x7f48d4004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:56.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.885+0000 7f48f4d26640 1 --2- 172.21.15.42:0/3161054744 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f48d4004810 0x7f48d4004c90 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f48f00a2660 tx=0x7f48dc01d1e0 comp rx=0 tx=0).stop 2024-09-16T03:51:56.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.885+0000 7f48f4d26640 1 -- 172.21.15.42:0/3161054744 shutdown_connections 2024-09-16T03:51:56.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.885+0000 7f48f4d26640 1 --2- 172.21.15.42:0/3161054744 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f48d4004810 0x7f48d4004c90 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:56.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.885+0000 7f48f4d26640 1 -- 172.21.15.42:0/3161054744 >> 172.21.15.42:0/3161054744 conn(0x7f48f00a6210 msgr2=0x7f48f00a6620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:51:56.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.885+0000 7f48f4d26640 1 -- 172.21.15.42:0/3161054744 shutdown_connections 2024-09-16T03:51:56.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.885+0000 7f48f4d26640 1 -- 172.21.15.42:0/3161054744 wait complete. 2024-09-16T03:51:56.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.885+0000 7f48f4d26640 10 monclient: build_initial_monmap 2024-09-16T03:51:56.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.885+0000 7f48f4d26640 10 monclient: monmap: 2024-09-16T03:51:56.914 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:51:56.914 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:51:56.914 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:51:56.914 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:51:56.914 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:51:56.914 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:56.914 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:51:56.914 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:51:56.915 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:51:56.915 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:56.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.885+0000 7f48f4d26640 1 librados: starting msgr at 2024-09-16T03:51:56.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.885+0000 7f48f4d26640 1 librados: starting objecter 2024-09-16T03:51:56.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.885+0000 7f48f4d26640 1 Processor -- start 2024-09-16T03:51:56.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.885+0000 7f48f4d26640 1 -- start start 2024-09-16T03:51:56.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.885+0000 7f48f4d26640 1 librados: setting wanted keys 2024-09-16T03:51:56.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.885+0000 7f48f4d26640 1 librados: calling monclient init 2024-09-16T03:51:56.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.885+0000 7f48f4d26640 10 monclient: init 2024-09-16T03:51:56.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.885+0000 7f48f4d26640 10 monclient: _reopen_session rank -1 2024-09-16T03:51:56.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.885+0000 7f48f4d26640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:51:56.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.885+0000 7f48f4d26640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f48d4004810 0x7f48f01cf140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:56.916 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.885+0000 7f48f4d26640 10 monclient(hunting): picked mon.noname-b con 0x7f48d4004810 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:51:56.916 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.885+0000 7f48f4d26640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f48f012fbd0 0x7f48f0130030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:56.916 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.885+0000 7f48f4d26640 10 monclient(hunting): picked mon.noname-a con 0x7f48f012fbd0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:51:56.916 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.885+0000 7f48f4d26640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f48f0131b60 0x7f48f0132010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:56.916 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.885+0000 7f48f4d26640 10 monclient(hunting): picked mon.noname-c con 0x7f48f0131b60 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:51:56.916 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.885+0000 7f48f4d26640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:56.916 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.885+0000 7f48f4d26640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f48f0075ac0 con 0x7f48f012fbd0 2024-09-16T03:51:56.916 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.885+0000 7f48f4d26640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:56.916 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.885+0000 7f48f4d26640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f48f0075940 con 0x7f48d4004810 2024-09-16T03:51:56.916 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.885+0000 7f48f4d26640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:56.916 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.885+0000 7f48f4d26640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f48f0073520 con 0x7f48f0131b60 2024-09-16T03:51:56.916 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.885+0000 7f48f4d26640 10 monclient(hunting): _renew_subs 2024-09-16T03:51:56.916 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.885+0000 7f48f4d26640 10 monclient(hunting): authenticate will time out at 1563.859836s 2024-09-16T03:51:56.917 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.885+0000 7f48eed9d640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f48f0131b60 0x7f48f0132010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:56.917 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.889+0000 7f48eed9d640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f48f0131b60 0x7f48f0132010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:41448/0 (socket says 172.21.15.42:41448) 2024-09-16T03:51:56.917 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.889+0000 7f48eed9d640 1 -- 172.21.15.42:0/2205510857 learned_addr learned my addr 172.21.15.42:0/2205510857 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:51:56.917 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.889+0000 7f48eed9d640 10 monclient(hunting): get_auth_request con 0x7f48f0131b60 auth_method 0 2024-09-16T03:51:56.917 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.889+0000 7f48eed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:56.917 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.889+0000 7f48eed9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:56.917 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.889+0000 7f48eed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:51:56.917 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.889+0000 7f48ee59c640 1 --2- 172.21.15.42:0/2205510857 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f48d4004810 0x7f48f01cf140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:56.917 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.889+0000 7f48ee59c640 10 monclient(hunting): get_auth_request con 0x7f48d4004810 auth_method 0 2024-09-16T03:51:56.917 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.889+0000 7f48ee59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:56.917 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.889+0000 7f48ee59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:56.917 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.889+0000 7f48ee59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:51:56.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.889+0000 7f48eed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:51:56.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.889+0000 7f48eed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:51:56.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.889+0000 7f48eed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:51:56.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.889+0000 7f48ee59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:51:56.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.889+0000 7f48ee59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:51:56.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.889+0000 7f48ee59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:51:56.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.889+0000 7f48edd9b640 1 --2- 172.21.15.42:0/2205510857 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f48f012fbd0 0x7f48f0130030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:56.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.889+0000 7f48edd9b640 10 monclient(hunting): get_auth_request con 0x7f48f012fbd0 auth_method 0 2024-09-16T03:51:56.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.889+0000 7f48edd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:56.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.889+0000 7f48edd9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:56.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.889+0000 7f48edd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:51:56.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.889+0000 7f48eed9d640 10 monclient(hunting): handle_auth_done global_id 814199 payload 824 2024-09-16T03:51:56.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.889+0000 7f48eed9d640 1 -- 172.21.15.42:0/2205510857 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f48d4004810 msgr2=0x7f48f01cf140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:56.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.889+0000 7f48eed9d640 1 --2- 172.21.15.42:0/2205510857 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f48d4004810 0x7f48f01cf140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:56.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.889+0000 7f48eed9d640 1 -- 172.21.15.42:0/2205510857 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f48f012fbd0 msgr2=0x7f48f0130030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:56.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.889+0000 7f48eed9d640 1 --2- 172.21.15.42:0/2205510857 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f48f012fbd0 0x7f48f0130030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:56.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.889+0000 7f48eed9d640 10 monclient: _finish_hunting 0 2024-09-16T03:51:56.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.889+0000 7f48eed9d640 1 monclient: found mon.noname-c 2024-09-16T03:51:56.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.889+0000 7f48eed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:51:56.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.889+0000 7f48eed9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:51:56.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.889+0000 7f48eed9d640 1 -- 172.21.15.42:0/2205510857 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48f0130570 con 0x7f48f0131b60 2024-09-16T03:51:56.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.889+0000 7f48edd9b640 1 --2- 172.21.15.42:0/2205510857 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f48f012fbd0 0x7f48f0130030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T03:51:56.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.889+0000 7f48eed9d640 1 --2- 172.21.15.42:0/2205510857 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f48f0131b60 0x7f48f0132010 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f48f009c270 tx=0x7f48e0011920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:51:56.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.889+0000 7f48e6ffd640 10 client.?.objecter ms_handle_connect 0x7f48f0131b60 2024-09-16T03:51:56.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.889+0000 7f48e6ffd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:51:56.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.889+0000 7f48e6ffd640 1 -- 172.21.15.42:0/2205510857 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48e0019070 con 0x7f48f0131b60 2024-09-16T03:51:56.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.889+0000 7f48e6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:51:56.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.889+0000 7f48e6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:51:56.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.889+0000 7f48e6ffd640 10 monclient: dump: 2024-09-16T03:51:56.920 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:51:56.920 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:56.920 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:56.920 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:56.920 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:51:56.921 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:56.921 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:51:56.921 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:51:56.921 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:51:56.921 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:56.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.889+0000 7f48e6ffd640 10 monclient: _finish_auth 0 2024-09-16T03:51:56.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.889+0000 7f48e6ffd640 10 monclient: _check_auth_tickets 2024-09-16T03:51:56.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.889+0000 7f48e6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:51:56.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.889+0000 7f48e6ffd640 1 -- 172.21.15.42:0/2205510857 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f48e00159f0 con 0x7f48f0131b60 2024-09-16T03:51:56.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.889+0000 7f48e6ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:51:56.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.889+0000 7f48e6ffd640 1 -- 172.21.15.42:0/2205510857 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48e0015de0 con 0x7f48f0131b60 2024-09-16T03:51:56.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.889+0000 7f48f4d26640 5 monclient: authenticate success, global_id 814199 2024-09-16T03:51:56.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.889+0000 7f48f4d26640 10 monclient: _renew_subs 2024-09-16T03:51:56.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.889+0000 7f48f4d26640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:51:56.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.889+0000 7f48f4d26640 1 -- 172.21.15.42:0/2205510857 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48f011d2b0 con 0x7f48f0131b60 2024-09-16T03:51:56.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.889+0000 7f48ecd99640 4 set_mon_vals no callback set 2024-09-16T03:51:56.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.889+0000 7f48e6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:51:56.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.889+0000 7f48e6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:51:56.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.889+0000 7f48e6ffd640 10 monclient: dump: 2024-09-16T03:51:56.922 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:51:56.922 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:56.922 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:56.922 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:56.922 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:51:56.922 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:56.922 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:51:56.923 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:51:56.923 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:51:56.923 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:56.923 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.889+0000 7f48f4d26640 10 client.814199.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:51:56.923 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.889+0000 7f48f4d26640 10 monclient: _renew_subs 2024-09-16T03:51:56.923 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.889+0000 7f48f4d26640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:51:56.923 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.889+0000 7f48f4d26640 1 -- 172.21.15.42:0/2205510857 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f48f013d440 con 0x7f48f0131b60 2024-09-16T03:51:56.923 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.889+0000 7f48f4d26640 1 librados: init done 2024-09-16T03:51:56.923 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.889+0000 7f48f4d26640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:51:56.923 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.889+0000 7f48e6ffd640 1 -- 172.21.15.42:0/2205510857 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f48e001c070 con 0x7f48f0131b60 2024-09-16T03:51:56.923 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.889+0000 7f48f4d26640 1 --2- 172.21.15.42:0/2205510857 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f48f0132550 0x7f48f009d230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:56.923 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.889+0000 7f48f4d26640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:56.923 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.889+0000 7f48f4d26640 1 -- 172.21.15.42:0/2205510857 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f48f0131520 con 0x7f48f0132550 2024-09-16T03:51:56.924 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.889+0000 7f48ee59c640 1 --2- 172.21.15.42:0/2205510857 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f48f0132550 0x7f48f009d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:56.924 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.889+0000 7f48ee59c640 10 monclient: get_auth_request con 0x7f48f0132550 auth_method 0 2024-09-16T03:51:56.924 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.889+0000 7f48ee59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:56.924 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.889+0000 7f48ee59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:56.924 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.889+0000 7f48ee59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:51:56.924 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.889+0000 7f48e6ffd640 1 --2- 172.21.15.42:0/2205510857 >> v2:172.21.15.154:6812/254845342 conn(0x7f48d8037840 0x7f48d8039d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:56.924 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.889+0000 7f48ee59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:51:56.924 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.889+0000 7f48ee59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:51:56.924 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.889+0000 7f48ee59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:51:56.924 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.889+0000 7f48e6ffd640 1 -- 172.21.15.42:0/2205510857 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(329..329 src has 286..329) ==== 5684+0+0 (secure 0 0 0) 0x7f48e0048fb0 con 0x7f48f0131b60 2024-09-16T03:51:56.924 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.889+0000 7f48e6ffd640 10 client.814199.objecter ms_dispatch 0x7f48f0000ba0 osd_map(329..329 src has 286..329) 2024-09-16T03:51:56.924 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.889+0000 7f48e6ffd640 3 client.814199.objecter handle_osd_map got epochs [329,329] > 0 2024-09-16T03:51:56.924 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.889+0000 7f48e6ffd640 3 client.814199.objecter handle_osd_map decoding full epoch 329 2024-09-16T03:51:56.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.889+0000 7f48e6ffd640 20 client.814199.objecter dump_active .. 0 homeless 2024-09-16T03:51:56.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.889+0000 7f48ee59c640 10 monclient(hunting): handle_auth_done global_id 824106 payload 824 2024-09-16T03:51:56.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.889+0000 7f48ee59c640 1 -- 172.21.15.42:0/2205510857 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f48f0133f70 con 0x7f48f0132550 2024-09-16T03:51:56.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.889+0000 7f48e6ffd640 10 client.814199.objecter ms_handle_connect 0x7f48f0132550 2024-09-16T03:51:56.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.889+0000 7f48ee59c640 1 --2- 172.21.15.42:0/2205510857 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f48f0132550 0x7f48f009d230 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f48e8007fd0 tx=0x7f48e8012cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:51:56.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.889+0000 7f48e6ffd640 10 client.814199.objecter resend_mon_ops 2024-09-16T03:51:56.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.889+0000 7f48ee59c640 1 -- 172.21.15.42:0/2205510857 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f48f0132550 msgr2=0x7f48f009d230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-16T03:51:56.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.889+0000 7f48ee59c640 1 -- 172.21.15.42:0/2205510857 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f48f0132550 msgr2=0x7f48f009d230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T03:51:56.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.889+0000 7f48ee59c640 1 --2- 172.21.15.42:0/2205510857 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f48f0132550 0x7f48f009d230 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f48e8007fd0 tx=0x7f48e8012cf0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:51:56.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.889+0000 7f48ee59c640 1 --2- 172.21.15.42:0/2205510857 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f48f0132550 0x7f48f009d230 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f48e8007fd0 tx=0x7f48e8012cf0 comp rx=0 tx=0).stop 2024-09-16T03:51:56.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.889+0000 7f48e6ffd640 1 -- 172.21.15.42:0/2205510857 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f48f0132550 msgr2=0x7f48f009d230 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:51:56.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.889+0000 7f48e6ffd640 1 --2- 172.21.15.42:0/2205510857 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f48f0132550 0x7f48f009d230 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:56.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.889+0000 7f48e6ffd640 1 --2- 172.21.15.42:0/2205510857 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f48d8040ba0 0x7f48d8042fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:56.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.889+0000 7f48e6ffd640 1 -- 172.21.15.42:0/2205510857 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f48f0132550 msgr2=0x7f48f009d230 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:51:56.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.889+0000 7f48e6ffd640 1 --2- 172.21.15.42:0/2205510857 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f48f0132550 0x7f48f009d230 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:56.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.889+0000 7f48e6ffd640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:56.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.889+0000 7f48e6ffd640 1 -- 172.21.15.42:0/2205510857 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f48d8043640 con 0x7f48d8040ba0 2024-09-16T03:51:56.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.889+0000 7f48edd9b640 1 --2- 172.21.15.42:0/2205510857 >> v2:172.21.15.154:6812/254845342 conn(0x7f48d8037840 0x7f48d8039d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:56.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.889+0000 7f48edd9b640 10 monclient: get_auth_request con 0x7f48d8037840 auth_method 0 2024-09-16T03:51:56.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.889+0000 7f48ee59c640 1 --2- 172.21.15.42:0/2205510857 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f48d8040ba0 0x7f48d8042fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:56.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.889+0000 7f48ee59c640 10 monclient: get_auth_request con 0x7f48d8040ba0 auth_method 0 2024-09-16T03:51:56.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.889+0000 7f48ee59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:56.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.889+0000 7f48ee59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:56.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.889+0000 7f48ee59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:51:56.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.889+0000 7f48ee59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:51:56.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.889+0000 7f48ee59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:51:56.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.889+0000 7f48ee59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:51:56.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.889+0000 7f48ee59c640 10 monclient(hunting): handle_auth_done global_id 824109 payload 824 2024-09-16T03:51:56.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.889+0000 7f48ee59c640 1 -- 172.21.15.42:0/2205510857 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f48d80437f0 con 0x7f48d8040ba0 2024-09-16T03:51:56.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.889+0000 7f48e6ffd640 10 client.814199.objecter ms_handle_connect 0x7f48d8040ba0 2024-09-16T03:51:56.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.889+0000 7f48ee59c640 1 --2- 172.21.15.42:0/2205510857 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f48d8040ba0 0x7f48d8042fe0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f48e8007fd0 tx=0x7f48e8001130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:51:56.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.889+0000 7f48e6ffd640 10 client.814199.objecter resend_mon_ops 2024-09-16T03:51:56.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.889+0000 7f48e6ffd640 1 -- 172.21.15.42:0/2205510857 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48e8007570 con 0x7f48d8040ba0 2024-09-16T03:51:56.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.889+0000 7f48e6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:51:56.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.889+0000 7f48e6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:51:56.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.889+0000 7f48e6ffd640 10 monclient: dump: 2024-09-16T03:51:56.928 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:51:56.928 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:56.928 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:56.928 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:56.928 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:51:56.928 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:56.929 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:51:56.929 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:51:56.929 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:51:56.929 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:56.929 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.889+0000 7f48edd9b640 1 --2- 172.21.15.42:0/2205510857 >> v2:172.21.15.154:6812/254845342 conn(0x7f48d8037840 0x7f48d8039d00 secure :-1 s=READY pgs=3568 cs=0 l=1 rev1=1 crypto rx=0x7f48dc0ad000 tx=0x7f48dc009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:51:56.929 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.889+0000 7f48e6ffd640 10 client.814199.objecter ms_handle_connect 0x7f48d8037840 2024-09-16T03:51:56.929 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.889+0000 7f48e6ffd640 1 -- 172.21.15.42:0/2205510857 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f48e8007570 con 0x7f48d8040ba0 2024-09-16T03:51:56.929 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.889+0000 7f48e6ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:51:56.929 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.889+0000 7f48e6ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:51:56.929 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.889+0000 7f48e6ffd640 1 -- 172.21.15.42:0/2205510857 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f48d8040ba0 msgr2=0x7f48d8042fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:56.929 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.889+0000 7f48e6ffd640 1 --2- 172.21.15.42:0/2205510857 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f48d8040ba0 0x7f48d8042fe0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f48e8007fd0 tx=0x7f48e8001130 comp rx=0 tx=0).stop 2024-09-16T03:51:56.929 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.889+0000 7f48e6ffd640 1 -- 172.21.15.42:0/2205510857 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f48d8040ba0 msgr2=0x7f48d8042fe0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:51:56.929 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.889+0000 7f48e6ffd640 1 --2- 172.21.15.42:0/2205510857 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f48d8040ba0 0x7f48d8042fe0 secure :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f48e8007fd0 tx=0x7f48e8001130 comp rx=0 tx=0).stop 2024-09-16T03:51:56.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.897+0000 7f48f4d26640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:51:56.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.897+0000 7f48f4d26640 1 --2- 172.21.15.42:0/2205510857 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f48f009d770 0x7f48f009db50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:56.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.897+0000 7f48f4d26640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:56.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.897+0000 7f48f4d26640 1 -- 172.21.15.42:0/2205510857 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f48f0132930 con 0x7f48f009d770 2024-09-16T03:51:56.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.897+0000 7f48ee59c640 1 --2- 172.21.15.42:0/2205510857 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f48f009d770 0x7f48f009db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:56.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.897+0000 7f48ee59c640 10 monclient: get_auth_request con 0x7f48f009d770 auth_method 0 2024-09-16T03:51:56.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.897+0000 7f48ee59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:56.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.897+0000 7f48ee59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:56.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.897+0000 7f48ee59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:51:56.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.897+0000 7f48ee59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:51:56.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.897+0000 7f48ee59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:51:56.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.897+0000 7f48ee59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:51:56.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.897+0000 7f48ee59c640 10 monclient(hunting): handle_auth_done global_id 824112 payload 824 2024-09-16T03:51:56.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.897+0000 7f48ee59c640 1 -- 172.21.15.42:0/2205510857 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f48f006b490 con 0x7f48f009d770 2024-09-16T03:51:56.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.897+0000 7f48e6ffd640 10 client.814199.objecter ms_handle_connect 0x7f48f009d770 2024-09-16T03:51:56.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.897+0000 7f48ee59c640 1 --2- 172.21.15.42:0/2205510857 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f48f009d770 0x7f48f009db50 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f48e8007fd0 tx=0x7f48e8012cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:51:56.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.897+0000 7f48e6ffd640 10 client.814199.objecter resend_mon_ops 2024-09-16T03:51:56.933 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.897+0000 7f48e6ffd640 1 -- 172.21.15.42:0/2205510857 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48e8028040 con 0x7f48f009d770 2024-09-16T03:51:56.933 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.897+0000 7f48e6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:51:56.933 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.897+0000 7f48e6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:51:56.933 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.897+0000 7f48e6ffd640 10 monclient: dump: 2024-09-16T03:51:56.933 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:51:56.933 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:56.933 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:56.933 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:56.933 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:51:56.933 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:56.933 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:51:56.933 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:51:56.933 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:51:56.934 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:56.934 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.897+0000 7f48e6ffd640 1 -- 172.21.15.42:0/2205510857 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4545 (secure 0 0 0) 0x7f48e8003600 con 0x7f48f009d770 2024-09-16T03:51:56.934 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.897+0000 7f48e6ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:51:56.934 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.897+0000 7f48e6ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:51:56.934 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.897+0000 7f48e6ffd640 1 -- 172.21.15.42:0/2205510857 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f48f009d770 msgr2=0x7f48f009db50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:56.934 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.897+0000 7f48e6ffd640 1 --2- 172.21.15.42:0/2205510857 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f48f009d770 0x7f48f009db50 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f48e8007fd0 tx=0x7f48e8012cf0 comp rx=0 tx=0).stop 2024-09-16T03:51:56.934 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.897+0000 7f48e6ffd640 1 -- 172.21.15.42:0/2205510857 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f48f009d770 msgr2=0x7f48f009db50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:51:56.934 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.897+0000 7f48e6ffd640 1 --2- 172.21.15.42:0/2205510857 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f48f009d770 0x7f48f009db50 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:56.934 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.897+0000 7f48eed9d640 1 -- 172.21.15.42:0/2205510857 reap_dead start 2024-09-16T03:51:56.934 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:51:56.935 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:51:56.935 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:51:56.935 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "leader", 2024-09-16T03:51:56.935 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 86, 2024-09-16T03:51:56.935 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 6751, 2024-09-16T03:51:56.935 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:51:56.935 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:51:56.935 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T03:51:56.935 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T03:51:56.935 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:51:56.935 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 0, 2024-09-16T03:51:56.935 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:51:56.935 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:51:56.936 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:51:56.936 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:51:56.936 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:51:56.936 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:51:56.936 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:51:56.936 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:51:56.936 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:51:56.936 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:51:56.936 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:51:56.936 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:51:56.936 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:51:56.936 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:51:56.936 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:51:56.937 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:51:56.937 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:51:56.937 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:51:56.937 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:51:56.937 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:51:56.937 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:51:56.937 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:51:56.937 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:51:56.937 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:51:56.937 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:51:56.937 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:51:56.937 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:51:56.938 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:51:56.938 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:51:56.938 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:56.938 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:51:56.938 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:51:56.938 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:51:56.938 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:51:56.938 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:51:56.938 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:51:56.938 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:51:56.938 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:51:56.938 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:51:56.938 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:51:56.938 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:51:56.939 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:51:56.939 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:51:56.939 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:51:56.939 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:51:56.939 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:51:56.939 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:51:56.939 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:51:56.939 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:51:56.940 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:51:56.940 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:51:56.940 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:51:56.940 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:51:56.940 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:51:56.940 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:51:56.940 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:51:56.940 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:51:56.940 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:51:56.940 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:51:56.940 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:51:56.940 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:56.940 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:51:56.941 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:56.941 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:51:56.941 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:51:56.941 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:51:56.941 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:51:56.941 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:56.941 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:51:56.941 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:51:56.941 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:51:56.941 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:56.941 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:56.941 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:51:56.941 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:51:56.942 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:51:56.942 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:51:56.942 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:51:56.942 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:56.942 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:51:56.942 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:51:56.942 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:51:56.942 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:51:56.942 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:51:56.942 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:56.942 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:56.942 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:51:56.942 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:51:56.942 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:51:56.943 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:51:56.943 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:56.943 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:51:56.943 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:51:56.943 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:51:56.943 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:56.943 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:56.943 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:51:56.943 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:51:56.943 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:51:56.943 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:51:56.943 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:51:56.943 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:56.944 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:51:56.944 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:51:56.944 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:51:56.944 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:51:56.944 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:51:56.944 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:56.944 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:56.944 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:51:56.944 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:51:56.944 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:51:56.944 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:51:56.944 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:56.944 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:51:56.944 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:51:56.945 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:51:56.945 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:56.945 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:56.945 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:51:56.945 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:51:56.945 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:51:56.945 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:51:56.945 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:51:56.945 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:56.945 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:51:56.945 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:51:56.945 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:51:56.946 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:51:56.946 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:51:56.946 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:51:56.946 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:51:56.946 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:56.946 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:51:56.946 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:51:56.946 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:56.946 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:51:56.946 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:51:56.946 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:51:56.946 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:51:56.946 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:51:56.947 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:51:56.947 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:56.947 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:51:56.947 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:51:56.947 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:51:56.947 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:51:56.947 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:51:56.947 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:56.947 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:51:56.947 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:51:56.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.897+0000 7f48f4d26640 10 librados: watch_flush enter 2024-09-16T03:51:56.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.897+0000 7f48f4d26640 10 librados: watch_flush exit 2024-09-16T03:51:56.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.897+0000 7f48f4d26640 20 client.814199.objecter shutdown clearing up homeless session... 2024-09-16T03:51:56.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.897+0000 7f48f4d26640 10 client.814199.objecter successfully canceled tick 2024-09-16T03:51:56.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.897+0000 7f48f4d26640 1 -- 172.21.15.42:0/2205510857 >> v2:172.21.15.154:6812/254845342 conn(0x7f48d8037840 msgr2=0x7f48d8039d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:56.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.897+0000 7f48f4d26640 1 --2- 172.21.15.42:0/2205510857 >> v2:172.21.15.154:6812/254845342 conn(0x7f48d8037840 0x7f48d8039d00 secure :-1 s=READY pgs=3568 cs=0 l=1 rev1=1 crypto rx=0x7f48dc0ad000 tx=0x7f48dc009040 comp rx=0 tx=0).stop 2024-09-16T03:51:56.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.897+0000 7f48f4d26640 10 monclient: shutdown 2024-09-16T03:51:56.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.897+0000 7f48f4d26640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:51:56.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.897+0000 7f48f4d26640 1 -- 172.21.15.42:0/2205510857 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f48f0131b60 msgr2=0x7f48f0132010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:56.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.897+0000 7f48f4d26640 1 --2- 172.21.15.42:0/2205510857 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f48f0131b60 0x7f48f0132010 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f48f009c270 tx=0x7f48e0011920 comp rx=0 tx=0).stop 2024-09-16T03:51:56.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.897+0000 7f48f4d26640 1 -- 172.21.15.42:0/2205510857 shutdown_connections 2024-09-16T03:51:56.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.897+0000 7f48f4d26640 1 --2- 172.21.15.42:0/2205510857 >> v2:172.21.15.154:6812/254845342 conn(0x7f48d8037840 0x7f48d8039d00 unknown :-1 s=CLOSED pgs=3568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:56.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.897+0000 7f48f4d26640 1 --2- 172.21.15.42:0/2205510857 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f48f0131b60 0x7f48f0132010 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:56.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.897+0000 7f48f4d26640 1 -- 172.21.15.42:0/2205510857 >> 172.21.15.42:0/2205510857 conn(0x7f48f00a6210 msgr2=0x7f48f01353d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:51:56.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.897+0000 7f48f4d26640 1 -- 172.21.15.42:0/2205510857 shutdown_connections 2024-09-16T03:51:56.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.897+0000 7f48f4d26640 1 -- 172.21.15.42:0/2205510857 wait complete. 2024-09-16T03:51:56.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:56.897+0000 7f48f4d26640 1 librados: shutdown 2024-09-16T03:51:56.950 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:51:56.950 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T03:51:56.951 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T03:51:57.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.053+0000 7fe292927640 10 monclient: get_monmap_and_config 2024-09-16T03:51:57.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.053+0000 7fe292927640 10 monclient: build_initial_monmap 2024-09-16T03:51:57.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.053+0000 7fe292927640 10 monclient: monmap: 2024-09-16T03:51:57.058 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:51:57.058 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:57.058 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:51:57.057066+0000 2024-09-16T03:51:57.058 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:51:57.057066+0000 2024-09-16T03:51:57.058 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:51:57.059 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:57.059 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:51:57.059 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:51:57.059 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:51:57.059 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:51:57.059 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:51:57.059 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:57.059 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.053+0000 7fe292927640 1 Processor -- start 2024-09-16T03:51:57.060 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.053+0000 7fe292927640 1 -- start start 2024-09-16T03:51:57.060 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.053+0000 7fe292927640 10 monclient: init 2024-09-16T03:51:57.070 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.053+0000 7fe292927640 10 monclient: _reopen_session rank -1 2024-09-16T03:51:57.070 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.053+0000 7fe292927640 10 monclient: _add_conns ranks=[4,1,0,2,3] 2024-09-16T03:51:57.070 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.053+0000 7fe292927640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fe28c06db90 0x7fe28c1340c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:57.070 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.053+0000 7fe292927640 10 monclient(hunting): picked mon.noname-c con 0x7fe28c06db90 addr v2:172.21.15.154:3300/0 2024-09-16T03:51:57.070 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.053+0000 7fe292927640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe28c0a37f0 addr v1:172.21.15.154:6789/0 2024-09-16T03:51:57.071 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.053+0000 7fe292927640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe28c0a3c00 addr v1:172.21.15.42:6789/0 2024-09-16T03:51:57.071 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.053+0000 7fe292927640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe28c06ece0 con 0x7fe28c0a3c00 2024-09-16T03:51:57.071 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.053+0000 7fe292927640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe28c09bf80 con 0x7fe28c0a37f0 2024-09-16T03:51:57.071 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.053+0000 7fe292927640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:57.071 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.053+0000 7fe292927640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fe28c06df70 con 0x7fe28c06db90 2024-09-16T03:51:57.071 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.053+0000 7fe292927640 10 monclient(hunting): _renew_subs 2024-09-16T03:51:57.071 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.053+0000 7fe292927640 10 monclient(hunting): authenticate will time out at 1564.027256s 2024-09-16T03:51:57.071 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.053+0000 7fe290ec4640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fe28c0a3c00 0x7fe28c137520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:56978/0 (socket says 172.21.15.42:56978) 2024-09-16T03:51:57.071 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.053+0000 7fe290ec4640 1 -- 172.21.15.42:0/661152372 learned_addr learned my addr 172.21.15.42:0/661152372 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:51:57.071 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.053+0000 7fe28affd640 1 -- 172.21.15.42:0/661152372 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1399550029 0 0) 0x7fe28c06ece0 con 0x7fe28c0a3c00 2024-09-16T03:51:57.071 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.053+0000 7fe28affd640 10 monclient(hunting): _negotiate 2024-09-16T03:51:57.071 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.053+0000 7fe28affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:57.072 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.053+0000 7fe28affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:51:57.072 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.053+0000 7fe28bfff640 1 --2- 172.21.15.42:0/661152372 >> v2:172.21.15.154:3300/0 conn(0x7fe28c06db90 0x7fe28c1340c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:57.072 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.053+0000 7fe28affd640 10 monclient(hunting): my global_id is 824115 2024-09-16T03:51:57.072 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.053+0000 7fe28affd640 1 -- 172.21.15.42:0/661152372 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe274003530 con 0x7fe28c0a3c00 2024-09-16T03:51:57.072 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.053+0000 7fe28bfff640 10 monclient(hunting): get_auth_request con 0x7fe28c06db90 auth_method 0 2024-09-16T03:51:57.072 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.053+0000 7fe28bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:57.072 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.053+0000 7fe28bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:57.072 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.053+0000 7fe28bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:51:57.072 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.053+0000 7fe28affd640 1 -- 172.21.15.42:0/661152372 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2503397040 0 0) 0x7fe274003530 con 0x7fe28c0a3c00 2024-09-16T03:51:57.072 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.057+0000 7fe28affd640 1 -- 172.21.15.42:0/661152372 >> v2:172.21.15.154:3300/0 conn(0x7fe28c06db90 msgr2=0x7fe28c1340c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:51:57.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.057+0000 7fe28affd640 1 --2- 172.21.15.42:0/661152372 >> v2:172.21.15.154:3300/0 conn(0x7fe28c06db90 0x7fe28c1340c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:57.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.057+0000 7fe28affd640 1 -- 172.21.15.42:0/661152372 >> v1:172.21.15.154:6789/0 conn(0x7fe28c0a37f0 legacy=0x7fe28c136e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:57.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.057+0000 7fe28affd640 10 monclient: _finish_hunting 0 2024-09-16T03:51:57.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.057+0000 7fe28affd640 1 monclient: found mon.noname-a-legacy 2024-09-16T03:51:57.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.057+0000 7fe28affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:51:57.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.057+0000 7fe28affd640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.42:6789/0 2024-09-16T03:51:57.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.057+0000 7fe28affd640 1 -- 172.21.15.42:0/661152372 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe28c137c30 con 0x7fe28c0a3c00 2024-09-16T03:51:57.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.057+0000 7fe28affd640 10 monclient: _finish_auth 0 2024-09-16T03:51:57.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.057+0000 7fe28affd640 10 monclient: _check_auth_tickets 2024-09-16T03:51:57.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.057+0000 7fe28affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:51:57.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.057+0000 7fe292927640 5 monclient: authenticate success, global_id 824115 2024-09-16T03:51:57.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.057+0000 7fe292927640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:51:57.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.057+0000 7fe28affd640 1 -- 172.21.15.42:0/661152372 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2177413734 0 0) 0x7fe284001680 con 0x7fe28c0a3c00 2024-09-16T03:51:57.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.057+0000 7fe28affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:51:57.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.057+0000 7fe28bfff640 1 --2- 172.21.15.42:0/661152372 >> v2:172.21.15.154:3300/0 conn(0x7fe28c06db90 0x7fe28c1340c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T03:51:57.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.057+0000 7fe28affd640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T03:51:57.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.057+0000 7fe28affd640 10 monclient: dump: 2024-09-16T03:51:57.074 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:51:57.074 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:57.074 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:57.074 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:57.074 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:51:57.074 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:57.074 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:51:57.074 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:51:57.074 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:51:57.074 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:57.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.057+0000 7fe28affd640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] but i'm connected to v1:172.21.15.42:6789/0, reconnecting 2024-09-16T03:51:57.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.057+0000 7fe28affd640 10 monclient: _reopen_session rank -1 2024-09-16T03:51:57.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.057+0000 7fe28affd640 1 -- 172.21.15.42:0/661152372 >> v1:172.21.15.42:6789/0 conn(0x7fe28c0a3c00 legacy=0x7fe28c137520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:57.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.057+0000 7fe28affd640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:51:57.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.057+0000 7fe28affd640 1 --2- 172.21.15.42:0/661152372 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe2740039c0 0x7fe274003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:57.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.057+0000 7fe28affd640 10 monclient(hunting): picked mon.a con 0x7fe2740039c0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:51:57.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.057+0000 7fe28affd640 1 --2- 172.21.15.42:0/661152372 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe274004810 0x7fe274004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:57.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.057+0000 7fe28affd640 10 monclient(hunting): picked mon.c con 0x7fe274004810 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:51:57.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.057+0000 7fe28affd640 1 --2- 172.21.15.42:0/661152372 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe2740053d0 0x7fe27400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:57.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.057+0000 7fe28affd640 10 monclient(hunting): picked mon.b con 0x7fe2740053d0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:51:57.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.057+0000 7fe28affd640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:57.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.057+0000 7fe28affd640 1 -- 172.21.15.42:0/661152372 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fe28c06df70 con 0x7fe2740039c0 2024-09-16T03:51:57.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.057+0000 7fe28affd640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:57.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.057+0000 7fe28affd640 1 -- 172.21.15.42:0/661152372 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe27400c260 con 0x7fe2740053d0 2024-09-16T03:51:57.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.057+0000 7fe28affd640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:57.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.057+0000 7fe28affd640 1 -- 172.21.15.42:0/661152372 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fe27400c460 con 0x7fe274004810 2024-09-16T03:51:57.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.057+0000 7fe28affd640 10 monclient(hunting): _renew_subs 2024-09-16T03:51:57.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.057+0000 7fe28affd640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:51:57.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.057+0000 7fe292927640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-16T03:51:57.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.057+0000 7fe28bfff640 1 --2- 172.21.15.42:0/661152372 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe2740039c0 0x7fe274003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:57.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.057+0000 7fe28b7fe640 1 --2- 172.21.15.42:0/661152372 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe274004810 0x7fe274004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:57.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.057+0000 7fe28bfff640 10 monclient(hunting): get_auth_request con 0x7fe2740039c0 auth_method 0 2024-09-16T03:51:57.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.057+0000 7fe28bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:57.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.057+0000 7fe28bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:57.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.057+0000 7fe28bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:51:57.077 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.057+0000 7fe28b7fe640 10 monclient(hunting): get_auth_request con 0x7fe274004810 auth_method 0 2024-09-16T03:51:57.077 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.057+0000 7fe28b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:57.077 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.057+0000 7fe28b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:57.077 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.057+0000 7fe28b7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:51:57.077 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.057+0000 7fe28b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:51:57.077 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.057+0000 7fe28b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:51:57.077 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.057+0000 7fe28b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T03:51:57.077 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.057+0000 7fe28bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:51:57.077 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.057+0000 7fe28bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:51:57.077 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.057+0000 7fe28bfff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T03:51:57.077 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.057+0000 7fe290ec4640 1 --2- 172.21.15.42:0/661152372 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe2740053d0 0x7fe27400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:57.077 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.057+0000 7fe290ec4640 10 monclient(hunting): get_auth_request con 0x7fe2740053d0 auth_method 0 2024-09-16T03:51:57.077 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.057+0000 7fe290ec4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:57.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.057+0000 7fe290ec4640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:57.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.057+0000 7fe290ec4640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:51:57.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.057+0000 7fe28b7fe640 10 monclient(hunting): handle_auth_done global_id 824115 payload 293 2024-09-16T03:51:57.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.057+0000 7fe28b7fe640 1 -- 172.21.15.42:0/661152372 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe2740053d0 msgr2=0x7fe27400b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:51:57.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.057+0000 7fe28b7fe640 1 --2- 172.21.15.42:0/661152372 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe2740053d0 0x7fe27400b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:57.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.057+0000 7fe28b7fe640 1 -- 172.21.15.42:0/661152372 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe2740039c0 msgr2=0x7fe274003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:51:57.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.057+0000 7fe28b7fe640 1 --2- 172.21.15.42:0/661152372 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe2740039c0 0x7fe274003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:57.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.057+0000 7fe28b7fe640 10 monclient: _finish_hunting 0 2024-09-16T03:51:57.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.057+0000 7fe28b7fe640 1 monclient: found mon.c 2024-09-16T03:51:57.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.057+0000 7fe28b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:51:57.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.057+0000 7fe28b7fe640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:51:57.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.057+0000 7fe28b7fe640 1 -- 172.21.15.42:0/661152372 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe274003530 con 0x7fe274004810 2024-09-16T03:51:57.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.057+0000 7fe290ec4640 1 -- 172.21.15.42:0/661152372 reap_dead start 2024-09-16T03:51:57.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.057+0000 7fe28b7fe640 10 monclient: _finish_auth 0 2024-09-16T03:51:57.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.057+0000 7fe28b7fe640 10 monclient: _check_auth_tickets 2024-09-16T03:51:57.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.057+0000 7fe28b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:51:57.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.057+0000 7fe28b7fe640 1 --2- 172.21.15.42:0/661152372 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe274004810 0x7fe274004c90 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fe280001780 tx=0x7fe28001c200 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e2bab8af167eaa4a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:51:57.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.057+0000 7fe28affd640 1 -- 172.21.15.42:0/661152372 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2800013a0 con 0x7fe274004810 2024-09-16T03:51:57.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.057+0000 7fe28affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:51:57.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.057+0000 7fe28affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:51:57.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.057+0000 7fe28affd640 10 monclient: dump: 2024-09-16T03:51:57.079 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:51:57.079 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:57.080 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:57.080 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:57.080 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:51:57.080 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:57.080 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:51:57.080 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:51:57.080 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:51:57.080 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:57.080 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.057+0000 7fe292927640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:51:57.080 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.057+0000 7fe28affd640 1 -- 172.21.15.42:0/661152372 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe2800052f0 con 0x7fe274004810 2024-09-16T03:51:57.080 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.057+0000 7fe28affd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:51:57.080 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.057+0000 7fe28affd640 1 -- 172.21.15.42:0/661152372 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe280003840 con 0x7fe274004810 2024-09-16T03:51:57.080 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.057+0000 7fe292927640 10 monclient: get_monmap_and_config success 2024-09-16T03:51:57.081 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.057+0000 7fe292927640 4 set_mon_vals no callback set 2024-09-16T03:51:57.081 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.057+0000 7fe28affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:51:57.081 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.057+0000 7fe28affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:51:57.081 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.057+0000 7fe28affd640 10 monclient: dump: 2024-09-16T03:51:57.081 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:51:57.081 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:57.081 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:57.081 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:57.081 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:51:57.081 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:57.081 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:51:57.081 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:51:57.081 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:51:57.082 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:57.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.057+0000 7fe292927640 10 monclient: shutdown 2024-09-16T03:51:57.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.057+0000 7fe292927640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:51:57.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.057+0000 7fe292927640 1 -- 172.21.15.42:0/661152372 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe274004810 msgr2=0x7fe274004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:57.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.057+0000 7fe292927640 1 --2- 172.21.15.42:0/661152372 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe274004810 0x7fe274004c90 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fe280001780 tx=0x7fe28001c200 comp rx=0 tx=0).stop 2024-09-16T03:51:57.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.057+0000 7fe292927640 1 -- 172.21.15.42:0/661152372 shutdown_connections 2024-09-16T03:51:57.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.057+0000 7fe292927640 1 --2- 172.21.15.42:0/661152372 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe274004810 0x7fe274004c90 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:57.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.057+0000 7fe292927640 1 -- 172.21.15.42:0/661152372 >> 172.21.15.42:0/661152372 conn(0x7fe28c0a9fb0 msgr2=0x7fe28c0aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:51:57.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.057+0000 7fe292927640 1 -- 172.21.15.42:0/661152372 shutdown_connections 2024-09-16T03:51:57.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.057+0000 7fe292927640 1 -- 172.21.15.42:0/661152372 wait complete. 2024-09-16T03:51:57.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.061+0000 7fe292927640 10 monclient: build_initial_monmap 2024-09-16T03:51:57.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.061+0000 7fe292927640 10 monclient: monmap: 2024-09-16T03:51:57.082 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:51:57.083 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:51:57.083 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:51:57.083 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:51:57.083 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:51:57.083 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:57.083 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:51:57.083 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:51:57.083 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:51:57.083 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:57.083 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.061+0000 7fe292927640 1 librados: starting msgr at 2024-09-16T03:51:57.083 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.061+0000 7fe292927640 1 librados: starting objecter 2024-09-16T03:51:57.083 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.061+0000 7fe292927640 1 Processor -- start 2024-09-16T03:51:57.083 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.061+0000 7fe292927640 1 -- start start 2024-09-16T03:51:57.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.061+0000 7fe292927640 1 librados: setting wanted keys 2024-09-16T03:51:57.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.061+0000 7fe292927640 1 librados: calling monclient init 2024-09-16T03:51:57.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.061+0000 7fe292927640 10 monclient: init 2024-09-16T03:51:57.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.061+0000 7fe292927640 10 monclient: _reopen_session rank -1 2024-09-16T03:51:57.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.061+0000 7fe292927640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:51:57.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.061+0000 7fe292927640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe274004810 0x7fe28c120200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:57.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.061+0000 7fe292927640 10 monclient(hunting): picked mon.noname-c con 0x7fe274004810 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:51:57.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.061+0000 7fe292927640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe28c120740 0x7fe28c120b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:57.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.061+0000 7fe292927640 10 monclient(hunting): picked mon.noname-b con 0x7fe28c120740 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:51:57.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.061+0000 7fe292927640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe28c1cf1a0 0x7fe28c1cf610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:57.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.061+0000 7fe292927640 10 monclient(hunting): picked mon.noname-a con 0x7fe28c1cf1a0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:51:57.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.061+0000 7fe292927640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:57.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.061+0000 7fe292927640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fe28c137620 con 0x7fe28c1cf1a0 2024-09-16T03:51:57.085 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.061+0000 7fe292927640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:57.085 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.061+0000 7fe292927640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe28c0a4010 con 0x7fe28c120740 2024-09-16T03:51:57.085 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.061+0000 7fe292927640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:57.085 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.061+0000 7fe292927640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fe28c1377a0 con 0x7fe274004810 2024-09-16T03:51:57.085 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.061+0000 7fe292927640 10 monclient(hunting): _renew_subs 2024-09-16T03:51:57.085 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.061+0000 7fe292927640 10 monclient(hunting): authenticate will time out at 1564.033005s 2024-09-16T03:51:57.085 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.061+0000 7fe28bfff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe274004810 0x7fe28c120200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:57.085 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.061+0000 7fe290ec4640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe28c1cf1a0 0x7fe28c1cf610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:57.085 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.061+0000 7fe290ec4640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe28c1cf1a0 0x7fe28c1cf610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:37246/0 (socket says 172.21.15.42:37246) 2024-09-16T03:51:57.085 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.061+0000 7fe290ec4640 1 -- 172.21.15.42:0/3192440508 learned_addr learned my addr 172.21.15.42:0/3192440508 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:51:57.085 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.061+0000 7fe290ec4640 10 monclient(hunting): get_auth_request con 0x7fe28c1cf1a0 auth_method 0 2024-09-16T03:51:57.085 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.061+0000 7fe290ec4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:57.085 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.061+0000 7fe290ec4640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:57.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.061+0000 7fe290ec4640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:51:57.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.061+0000 7fe28bfff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe274004810 0x7fe28c120200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:41456/0 (socket says 172.21.15.42:41456) 2024-09-16T03:51:57.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.061+0000 7fe28bfff640 10 monclient(hunting): get_auth_request con 0x7fe274004810 auth_method 0 2024-09-16T03:51:57.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.061+0000 7fe28bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:57.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.061+0000 7fe28bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:57.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.061+0000 7fe28bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:51:57.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.061+0000 7fe28bfff640 1 -- 172.21.15.42:0/3192440508 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe274004810 msgr2=0x7fe28c120200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-16T03:51:57.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.061+0000 7fe28bfff640 1 -- 172.21.15.42:0/3192440508 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe274004810 msgr2=0x7fe28c120200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T03:51:57.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.061+0000 7fe28bfff640 1 --2- 172.21.15.42:0/3192440508 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe274004810 0x7fe28c120200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:51:57.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.061+0000 7fe28bfff640 1 --2- 172.21.15.42:0/3192440508 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe274004810 0x7fe28c120200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:51:57.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.061+0000 7fe290ec4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:51:57.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.061+0000 7fe290ec4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:51:57.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.061+0000 7fe290ec4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:51:57.087 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.061+0000 7fe28b7fe640 1 --2- 172.21.15.42:0/3192440508 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe28c120740 0x7fe28c120b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:57.087 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.061+0000 7fe28b7fe640 10 monclient(hunting): get_auth_request con 0x7fe28c120740 auth_method 0 2024-09-16T03:51:57.087 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.061+0000 7fe28b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:57.087 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.061+0000 7fe28b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:57.087 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.061+0000 7fe28b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:51:57.087 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.061+0000 7fe290ec4640 10 monclient(hunting): handle_auth_done global_id 824118 payload 824 2024-09-16T03:51:57.087 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.061+0000 7fe290ec4640 1 -- 172.21.15.42:0/3192440508 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe274004810 msgr2=0x7fe28c120200 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:51:57.087 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.061+0000 7fe290ec4640 1 --2- 172.21.15.42:0/3192440508 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe274004810 0x7fe28c120200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:57.087 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.061+0000 7fe290ec4640 1 -- 172.21.15.42:0/3192440508 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe28c120740 msgr2=0x7fe28c120b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:57.087 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.061+0000 7fe290ec4640 1 --2- 172.21.15.42:0/3192440508 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe28c120740 0x7fe28c120b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:57.087 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.061+0000 7fe290ec4640 10 monclient: _finish_hunting 0 2024-09-16T03:51:57.087 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.061+0000 7fe290ec4640 1 monclient: found mon.noname-a 2024-09-16T03:51:57.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.061+0000 7fe290ec4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:51:57.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.061+0000 7fe290ec4640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:51:57.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.061+0000 7fe290ec4640 1 -- 172.21.15.42:0/3192440508 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe28c1229b0 con 0x7fe28c1cf1a0 2024-09-16T03:51:57.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.061+0000 7fe290ec4640 1 --2- 172.21.15.42:0/3192440508 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe28c1cf1a0 0x7fe28c1cf610 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fe28c069180 tx=0x7fe28400e960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:51:57.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.061+0000 7fe288ff9640 10 client.?.objecter ms_handle_connect 0x7fe28c1cf1a0 2024-09-16T03:51:57.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.061+0000 7fe288ff9640 10 client.?.objecter resend_mon_ops 2024-09-16T03:51:57.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.061+0000 7fe288ff9640 1 -- 172.21.15.42:0/3192440508 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe284010510 con 0x7fe28c1cf1a0 2024-09-16T03:51:57.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.061+0000 7fe288ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:51:57.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.061+0000 7fe288ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:51:57.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.061+0000 7fe288ff9640 10 monclient: dump: 2024-09-16T03:51:57.088 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:51:57.088 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:57.088 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:57.089 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:57.089 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:51:57.089 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:57.089 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:51:57.089 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:51:57.089 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:51:57.089 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:57.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.061+0000 7fe288ff9640 10 monclient: _finish_auth 0 2024-09-16T03:51:57.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.061+0000 7fe288ff9640 10 monclient: _check_auth_tickets 2024-09-16T03:51:57.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.061+0000 7fe288ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:51:57.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.061+0000 7fe288ff9640 1 -- 172.21.15.42:0/3192440508 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe2840106b0 con 0x7fe28c1cf1a0 2024-09-16T03:51:57.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.061+0000 7fe292927640 5 monclient: authenticate success, global_id 824118 2024-09-16T03:51:57.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.061+0000 7fe292927640 10 monclient: _renew_subs 2024-09-16T03:51:57.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.061+0000 7fe292927640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:51:57.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.061+0000 7fe292927640 1 -- 172.21.15.42:0/3192440508 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe28c122c90 con 0x7fe28c1cf1a0 2024-09-16T03:51:57.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.061+0000 7fe288ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T03:51:57.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.061+0000 7fe288ff9640 1 -- 172.21.15.42:0/3192440508 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe284010aa0 con 0x7fe28c1cf1a0 2024-09-16T03:51:57.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.061+0000 7fe288ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:51:57.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.061+0000 7fe288ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:51:57.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.061+0000 7fe288ff9640 10 monclient: dump: 2024-09-16T03:51:57.090 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:51:57.090 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:57.090 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:57.090 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:57.090 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:51:57.090 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:57.090 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:51:57.091 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:51:57.091 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:51:57.091 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:57.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.061+0000 7fe292927640 10 client.824118.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:51:57.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.061+0000 7fe292927640 10 monclient: _renew_subs 2024-09-16T03:51:57.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.061+0000 7fe292927640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:51:57.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.061+0000 7fe292927640 1 -- 172.21.15.42:0/3192440508 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe28c133200 con 0x7fe28c1cf1a0 2024-09-16T03:51:57.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.061+0000 7fe28a7fc640 4 set_mon_vals no callback set 2024-09-16T03:51:57.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.061+0000 7fe292927640 1 librados: init done 2024-09-16T03:51:57.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.061+0000 7fe292927640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:51:57.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.061+0000 7fe292927640 1 --2- 172.21.15.42:0/3192440508 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe28c1cfb50 0x7fe28c09e720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:57.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.061+0000 7fe292927640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:57.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.061+0000 7fe292927640 1 -- 172.21.15.42:0/3192440508 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fe28c06f6a0 con 0x7fe28c1cfb50 2024-09-16T03:51:57.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.061+0000 7fe28bfff640 1 --2- 172.21.15.42:0/3192440508 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe28c1cfb50 0x7fe28c09e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:57.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.061+0000 7fe28bfff640 10 monclient: get_auth_request con 0x7fe28c1cfb50 auth_method 0 2024-09-16T03:51:57.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.061+0000 7fe28bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:57.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.061+0000 7fe28bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:57.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.061+0000 7fe28bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:51:57.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.061+0000 7fe28bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:51:57.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.061+0000 7fe288ff9640 1 -- 172.21.15.42:0/3192440508 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fe284010c40 con 0x7fe28c1cf1a0 2024-09-16T03:51:57.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.061+0000 7fe28bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:51:57.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.061+0000 7fe28bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:51:57.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.061+0000 7fe28bfff640 10 monclient(hunting): handle_auth_done global_id 814202 payload 824 2024-09-16T03:51:57.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.061+0000 7fe288ff9640 1 --2- 172.21.15.42:0/3192440508 >> v2:172.21.15.154:6812/254845342 conn(0x7fe278037810 0x7fe278039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:57.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.061+0000 7fe28bfff640 1 -- 172.21.15.42:0/3192440508 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe28c13a180 con 0x7fe28c1cfb50 2024-09-16T03:51:57.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.061+0000 7fe288ff9640 1 -- 172.21.15.42:0/3192440508 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(329..329 src has 286..329) ==== 5684+0+0 (secure 0 0 0) 0x7fe2840485e0 con 0x7fe28c1cf1a0 2024-09-16T03:51:57.093 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.061+0000 7fe288ff9640 10 client.824118.objecter ms_dispatch 0x7fe28c000ba0 osd_map(329..329 src has 286..329) 2024-09-16T03:51:57.093 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.061+0000 7fe288ff9640 3 client.824118.objecter handle_osd_map got epochs [329,329] > 0 2024-09-16T03:51:57.093 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.061+0000 7fe288ff9640 3 client.824118.objecter handle_osd_map decoding full epoch 329 2024-09-16T03:51:57.093 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.061+0000 7fe288ff9640 20 client.824118.objecter dump_active .. 0 homeless 2024-09-16T03:51:57.093 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.061+0000 7fe288ff9640 10 client.824118.objecter ms_handle_connect 0x7fe28c1cfb50 2024-09-16T03:51:57.093 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.061+0000 7fe288ff9640 10 client.824118.objecter resend_mon_ops 2024-09-16T03:51:57.093 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.061+0000 7fe28bfff640 1 --2- 172.21.15.42:0/3192440508 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe28c1cfb50 0x7fe28c09e720 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fe27c007f20 tx=0x7fe27c0125b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:51:57.093 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.061+0000 7fe288ff9640 1 -- 172.21.15.42:0/3192440508 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe27c01c020 con 0x7fe28c1cfb50 2024-09-16T03:51:57.093 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.061+0000 7fe288ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:51:57.093 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.061+0000 7fe288ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:51:57.093 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.061+0000 7fe288ff9640 10 monclient: dump: 2024-09-16T03:51:57.093 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:51:57.093 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:57.094 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:57.094 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:57.094 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:51:57.094 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:57.094 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:51:57.094 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:51:57.094 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:51:57.094 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:57.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.061+0000 7fe28b7fe640 1 --2- 172.21.15.42:0/3192440508 >> v2:172.21.15.154:6812/254845342 conn(0x7fe278037810 0x7fe278039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:57.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.061+0000 7fe28b7fe640 10 monclient: get_auth_request con 0x7fe278037810 auth_method 0 2024-09-16T03:51:57.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.061+0000 7fe288ff9640 1 -- 172.21.15.42:0/3192440508 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fe27c017660 con 0x7fe28c1cfb50 2024-09-16T03:51:57.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.061+0000 7fe288ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:51:57.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.061+0000 7fe288ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:51:57.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.061+0000 7fe288ff9640 1 -- 172.21.15.42:0/3192440508 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe28c1cfb50 msgr2=0x7fe28c09e720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:57.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.061+0000 7fe288ff9640 1 --2- 172.21.15.42:0/3192440508 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe28c1cfb50 0x7fe28c09e720 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fe27c007f20 tx=0x7fe27c0125b0 comp rx=0 tx=0).stop 2024-09-16T03:51:57.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.061+0000 7fe288ff9640 1 -- 172.21.15.42:0/3192440508 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe28c1cfb50 msgr2=0x7fe28c09e720 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:51:57.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.061+0000 7fe288ff9640 1 --2- 172.21.15.42:0/3192440508 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe28c1cfb50 0x7fe28c09e720 secure :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fe27c007f20 tx=0x7fe27c0125b0 comp rx=0 tx=0).stop 2024-09-16T03:51:57.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.065+0000 7fe28b7fe640 1 --2- 172.21.15.42:0/3192440508 >> v2:172.21.15.154:6812/254845342 conn(0x7fe278037810 0x7fe278039cd0 secure :-1 s=READY pgs=3569 cs=0 l=1 rev1=1 crypto rx=0x7fe280001750 tx=0x7fe2800011a0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:51:57.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.065+0000 7fe288ff9640 10 client.824118.objecter ms_handle_connect 0x7fe278037810 2024-09-16T03:51:57.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.069+0000 7fe292927640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:51:57.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.069+0000 7fe292927640 1 --2- 172.21.15.42:0/3192440508 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe28c09ec60 0x7fe28c09f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:57.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.069+0000 7fe292927640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:57.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.069+0000 7fe292927640 1 -- 172.21.15.42:0/3192440508 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fe28c130bf0 con 0x7fe28c09ec60 2024-09-16T03:51:57.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.069+0000 7fe28bfff640 1 --2- 172.21.15.42:0/3192440508 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe28c09ec60 0x7fe28c09f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:57.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.069+0000 7fe28bfff640 10 monclient: get_auth_request con 0x7fe28c09ec60 auth_method 0 2024-09-16T03:51:57.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.069+0000 7fe28bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:57.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.069+0000 7fe28bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:57.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.069+0000 7fe28bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:51:57.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.069+0000 7fe28bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:51:57.098 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.069+0000 7fe28bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:51:57.098 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.069+0000 7fe28bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:51:57.098 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.069+0000 7fe28bfff640 10 monclient(hunting): handle_auth_done global_id 814205 payload 824 2024-09-16T03:51:57.098 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.069+0000 7fe28bfff640 1 -- 172.21.15.42:0/3192440508 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fe28c1cff30 con 0x7fe28c09ec60 2024-09-16T03:51:57.098 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.069+0000 7fe28bfff640 1 --2- 172.21.15.42:0/3192440508 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe28c09ec60 0x7fe28c09f070 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fe27c01d000 tx=0x7fe27c004ac0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:51:57.098 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.069+0000 7fe288ff9640 10 client.824118.objecter ms_handle_connect 0x7fe28c09ec60 2024-09-16T03:51:57.098 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.069+0000 7fe288ff9640 10 client.824118.objecter resend_mon_ops 2024-09-16T03:51:57.098 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.069+0000 7fe288ff9640 1 -- 172.21.15.42:0/3192440508 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe27c01c040 con 0x7fe28c09ec60 2024-09-16T03:51:57.098 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.069+0000 7fe288ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:51:57.098 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.069+0000 7fe288ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:51:57.098 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.069+0000 7fe288ff9640 10 monclient: dump: 2024-09-16T03:51:57.098 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:51:57.098 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:57.099 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:57.099 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:57.099 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:51:57.099 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:57.099 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:51:57.099 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:51:57.099 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:51:57.099 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:57.099 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.069+0000 7fe288ff9640 1 -- 172.21.15.42:0/3192440508 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4878 (secure 0 0 0) 0x7fe27c023220 con 0x7fe28c09ec60 2024-09-16T03:51:57.099 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.069+0000 7fe288ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:51:57.099 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.069+0000 7fe288ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:51:57.099 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.069+0000 7fe288ff9640 1 -- 172.21.15.42:0/3192440508 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe28c09ec60 msgr2=0x7fe28c09f070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:57.100 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.069+0000 7fe288ff9640 1 --2- 172.21.15.42:0/3192440508 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe28c09ec60 0x7fe28c09f070 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fe27c01d000 tx=0x7fe27c004ac0 comp rx=0 tx=0).stop 2024-09-16T03:51:57.100 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.069+0000 7fe288ff9640 1 -- 172.21.15.42:0/3192440508 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe28c09ec60 msgr2=0x7fe28c09f070 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:51:57.100 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.069+0000 7fe288ff9640 1 --2- 172.21.15.42:0/3192440508 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe28c09ec60 0x7fe28c09f070 secure :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fe27c01d000 tx=0x7fe27c004ac0 comp rx=0 tx=0).stop 2024-09-16T03:51:57.100 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:51:57.100 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:51:57.100 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:51:57.100 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T03:51:57.100 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 86, 2024-09-16T03:51:57.100 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 62143, 2024-09-16T03:51:57.100 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:51:57.101 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:51:57.101 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T03:51:57.101 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T03:51:57.101 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:51:57.101 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 0, 2024-09-16T03:51:57.101 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:51:57.101 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:51:57.101 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:51:57.101 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:51:57.101 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:51:57.101 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:51:57.101 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:51:57.101 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:51:57.102 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:51:57.102 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:51:57.102 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:51:57.102 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:51:57.102 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:51:57.102 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:51:57.102 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:51:57.102 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:51:57.102 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:51:57.102 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:51:57.102 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:51:57.102 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:51:57.102 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:51:57.103 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:51:57.103 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:51:57.103 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:51:57.103 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:51:57.103 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:51:57.103 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:51:57.103 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:51:57.103 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:51:57.103 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:57.103 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:51:57.103 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:51:57.103 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:51:57.103 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:51:57.104 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:51:57.104 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:51:57.104 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:51:57.104 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:51:57.104 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:51:57.104 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:51:57.104 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:51:57.104 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:51:57.104 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:51:57.104 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:51:57.104 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:51:57.104 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:51:57.104 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:51:57.105 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:51:57.105 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:51:57.105 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:51:57.105 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:51:57.105 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:51:57.105 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:51:57.105 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:51:57.105 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:51:57.105 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:51:57.105 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:51:57.105 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:51:57.105 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:51:57.105 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:51:57.106 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:57.106 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:51:57.106 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:57.106 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:51:57.106 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:51:57.106 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:51:57.106 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:51:57.106 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:57.106 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:51:57.106 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:51:57.106 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:51:57.106 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:57.106 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:57.106 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:51:57.107 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:51:57.107 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:51:57.107 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:51:57.107 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:51:57.107 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:57.107 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:51:57.107 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:51:57.107 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:51:57.107 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:51:57.107 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:51:57.107 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:57.107 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:57.107 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:51:57.108 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:51:57.108 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:51:57.108 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:51:57.108 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:57.108 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:51:57.108 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:51:57.108 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:51:57.108 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:57.108 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:57.108 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:51:57.108 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:51:57.108 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:51:57.108 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:51:57.108 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:51:57.109 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:57.109 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:51:57.109 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:51:57.109 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:51:57.109 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:51:57.109 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:51:57.109 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:57.109 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:57.109 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:51:57.109 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:51:57.109 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:51:57.109 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:51:57.109 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:57.109 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:51:57.110 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:51:57.110 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:51:57.110 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:57.110 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:57.110 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:51:57.110 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:51:57.110 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:51:57.110 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:51:57.110 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:51:57.110 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:57.110 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:51:57.110 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:51:57.110 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:51:57.111 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:51:57.111 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:51:57.111 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:51:57.111 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:51:57.111 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:57.111 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:51:57.111 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:51:57.111 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:57.111 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:51:57.111 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:51:57.111 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:51:57.111 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:51:57.111 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:51:57.111 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T03:51:57.111 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:57.112 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:51:57.112 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:51:57.112 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 5 2024-09-16T03:51:57.112 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:51:57.112 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:51:57.112 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:51:57.112 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:57.112 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:51:57.112 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:51:57.112 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 6 2024-09-16T03:51:57.112 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:51:57.112 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:51:57.112 INFO:teuthology.orchestra.run.smithi042.stdout: "mgr": [ 2024-09-16T03:51:57.113 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:57.113 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:51:57.113 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:51:57.113 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:51:57.113 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:51:57.113 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:51:57.113 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:57.113 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:51:57.113 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:51:57.115 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.069+0000 7fe292927640 10 librados: watch_flush enter 2024-09-16T03:51:57.115 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.069+0000 7fe292927640 10 librados: watch_flush exit 2024-09-16T03:51:57.115 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.069+0000 7fe292927640 20 client.824118.objecter shutdown clearing up homeless session... 2024-09-16T03:51:57.115 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.069+0000 7fe292927640 10 client.824118.objecter successfully canceled tick 2024-09-16T03:51:57.115 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.069+0000 7fe292927640 1 -- 172.21.15.42:0/3192440508 >> v2:172.21.15.154:6812/254845342 conn(0x7fe278037810 msgr2=0x7fe278039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:57.115 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.069+0000 7fe292927640 1 --2- 172.21.15.42:0/3192440508 >> v2:172.21.15.154:6812/254845342 conn(0x7fe278037810 0x7fe278039cd0 secure :-1 s=READY pgs=3569 cs=0 l=1 rev1=1 crypto rx=0x7fe280001750 tx=0x7fe2800011a0 comp rx=0 tx=0).stop 2024-09-16T03:51:57.115 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.069+0000 7fe292927640 10 monclient: shutdown 2024-09-16T03:51:57.115 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.069+0000 7fe292927640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:51:57.115 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.069+0000 7fe292927640 1 -- 172.21.15.42:0/3192440508 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe28c1cf1a0 msgr2=0x7fe28c1cf610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:57.115 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.069+0000 7fe292927640 1 --2- 172.21.15.42:0/3192440508 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe28c1cf1a0 0x7fe28c1cf610 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fe28c069180 tx=0x7fe28400e960 comp rx=0 tx=0).stop 2024-09-16T03:51:57.116 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.069+0000 7fe290ec4640 1 -- 172.21.15.42:0/3192440508 reap_dead start 2024-09-16T03:51:57.116 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.069+0000 7fe290ec4640 1 -- 172.21.15.42:0/3192440508 reap_dead start 2024-09-16T03:51:57.116 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.069+0000 7fe292927640 1 -- 172.21.15.42:0/3192440508 shutdown_connections 2024-09-16T03:51:57.116 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.069+0000 7fe292927640 1 -- 172.21.15.42:0/3192440508 >> 172.21.15.42:0/3192440508 conn(0x7fe28c0a9fb0 msgr2=0x7fe28c0a7ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:51:57.116 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.069+0000 7fe292927640 1 -- 172.21.15.42:0/3192440508 shutdown_connections 2024-09-16T03:51:57.116 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.069+0000 7fe292927640 1 -- 172.21.15.42:0/3192440508 wait complete. 2024-09-16T03:51:57.116 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.069+0000 7fe292927640 1 librados: shutdown 2024-09-16T03:51:57.116 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:51:57.116 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T03:51:57.117 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T03:51:57.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.221+0000 7f10eaab1640 10 monclient: get_monmap_and_config 2024-09-16T03:51:57.227 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.221+0000 7f10eaab1640 10 monclient: build_initial_monmap 2024-09-16T03:51:57.227 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.221+0000 7f10eaab1640 10 monclient: monmap: 2024-09-16T03:51:57.227 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:51:57.227 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:57.227 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:51:57.225383+0000 2024-09-16T03:51:57.227 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:51:57.225383+0000 2024-09-16T03:51:57.227 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:51:57.227 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:57.227 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:51:57.227 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:51:57.227 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:51:57.227 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:51:57.228 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:51:57.228 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:57.230 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.221+0000 7f10eaab1640 1 Processor -- start 2024-09-16T03:51:57.230 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.221+0000 7f10eaab1640 1 -- start start 2024-09-16T03:51:57.230 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.221+0000 7f10eaab1640 10 monclient: init 2024-09-16T03:51:57.230 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.221+0000 7f10eaab1640 10 monclient: _reopen_session rank -1 2024-09-16T03:51:57.230 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.221+0000 7f10eaab1640 10 monclient: _add_conns ranks=[2,0,3,1,4] 2024-09-16T03:51:57.230 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.221+0000 7f10eaab1640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f10e40a4720 0x7f10e41374e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:57.230 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.221+0000 7f10eaab1640 10 monclient(hunting): picked mon.noname-b con 0x7f10e40a4720 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:51:57.230 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.221+0000 7f10eaab1640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f10e4137a20 addr v1:172.21.15.42:6789/0 2024-09-16T03:51:57.230 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.221+0000 7f10eaab1640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f10e4137e00 0x7f10e40a5210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:57.230 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.221+0000 7f10eaab1640 10 monclient(hunting): picked mon.noname-a con 0x7f10e4137e00 addr v2:172.21.15.42:3300/0 2024-09-16T03:51:57.230 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.221+0000 7f10eaab1640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f10e40a1470 con 0x7f10e4137a20 2024-09-16T03:51:57.230 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.221+0000 7f10eaab1640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:57.230 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.221+0000 7f10eaab1640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f10e406e8f0 con 0x7f10e4137e00 2024-09-16T03:51:57.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.221+0000 7f10eaab1640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:57.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.221+0000 7f10eaab1640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f10e40a5f60 con 0x7f10e40a4720 2024-09-16T03:51:57.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.221+0000 7f10eaab1640 10 monclient(hunting): _renew_subs 2024-09-16T03:51:57.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.221+0000 7f10eaab1640 10 monclient(hunting): authenticate will time out at 1564.194838s 2024-09-16T03:51:57.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.221+0000 7f10e884d640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f10e40a4720 0x7f10e41374e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:57.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.221+0000 7f10e904e640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f10e4137e00 0x7f10e40a5210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:57.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.221+0000 7f10e884d640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f10e40a4720 0x7f10e41374e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:41490/0 (socket says 172.21.15.42:41490) 2024-09-16T03:51:57.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.221+0000 7f10e884d640 1 -- 172.21.15.42:0/302957881 learned_addr learned my addr 172.21.15.42:0/302957881 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:51:57.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.221+0000 7f10e884d640 10 monclient(hunting): get_auth_request con 0x7f10e40a4720 auth_method 0 2024-09-16T03:51:57.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.221+0000 7f10e884d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:57.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.221+0000 7f10e884d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:57.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.221+0000 7f10e884d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:51:57.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.221+0000 7f10e3fff640 1 --1- 172.21.15.42:0/302957881 >> v1:172.21.15.42:6789/0 conn(0x7f10e4137a20 0x7f10e40a4b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:56994/0 (socket says 172.21.15.42:56994) 2024-09-16T03:51:57.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.221+0000 7f10e904e640 10 monclient(hunting): get_auth_request con 0x7f10e4137e00 auth_method 0 2024-09-16T03:51:57.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.221+0000 7f10e904e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:57.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.221+0000 7f10e904e640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:57.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.221+0000 7f10e904e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:51:57.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.221+0000 7f10e904e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:51:57.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.221+0000 7f10e904e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:51:57.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.221+0000 7f10e904e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:51:57.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.221+0000 7f10e884d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:51:57.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.221+0000 7f10e884d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:51:57.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.221+0000 7f10e884d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:51:57.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.221+0000 7f10e904e640 10 monclient(hunting): handle_auth_done global_id 824121 payload 274 2024-09-16T03:51:57.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.221+0000 7f10e904e640 1 -- 172.21.15.42:0/302957881 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f10e40a4720 msgr2=0x7f10e41374e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:51:57.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.221+0000 7f10e904e640 1 --2- 172.21.15.42:0/302957881 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f10e40a4720 0x7f10e41374e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:57.233 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.221+0000 7f10e904e640 1 -- 172.21.15.42:0/302957881 >> v1:172.21.15.42:6789/0 conn(0x7f10e4137a20 legacy=0x7f10e40a4b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:57.233 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.221+0000 7f10e904e640 10 monclient: _finish_hunting 0 2024-09-16T03:51:57.233 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.221+0000 7f10e904e640 1 monclient: found mon.noname-a 2024-09-16T03:51:57.233 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.221+0000 7f10e904e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:51:57.233 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.221+0000 7f10e904e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:51:57.233 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.221+0000 7f10e904e640 1 -- 172.21.15.42:0/302957881 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10e40a1750 con 0x7f10e4137e00 2024-09-16T03:51:57.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.225+0000 7f10e904e640 1 --2- 172.21.15.42:0/302957881 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f10e4137e00 0x7f10e40a5210 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f10d40097c0 tx=0x7f10d401c360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c168381abcd91e09 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:51:57.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.225+0000 7f10e37fe640 1 -- 172.21.15.42:0/302957881 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10d40b0070 con 0x7f10e4137e00 2024-09-16T03:51:57.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.225+0000 7f10e37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:51:57.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.225+0000 7f10e37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:51:57.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.225+0000 7f10e37fe640 10 monclient: dump: 2024-09-16T03:51:57.240 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:51:57.240 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:57.240 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:57.240 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:57.240 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:51:57.240 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:57.240 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:51:57.240 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:51:57.240 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:51:57.240 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:57.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.225+0000 7f10e37fe640 10 monclient: _finish_auth 0 2024-09-16T03:51:57.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.225+0000 7f10e37fe640 10 monclient: _check_auth_tickets 2024-09-16T03:51:57.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.225+0000 7f10e37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:51:57.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.225+0000 7f10e37fe640 1 -- 172.21.15.42:0/302957881 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f10d40ab600 con 0x7f10e4137e00 2024-09-16T03:51:57.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.225+0000 7f10eaab1640 5 monclient: authenticate success, global_id 824121 2024-09-16T03:51:57.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.225+0000 7f10eaab1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:51:57.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.225+0000 7f10e37fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:51:57.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.225+0000 7f10eaab1640 10 monclient: get_monmap_and_config success 2024-09-16T03:51:57.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.225+0000 7f10e37fe640 1 -- 172.21.15.42:0/302957881 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10d40ab9d0 con 0x7f10e4137e00 2024-09-16T03:51:57.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.225+0000 7f10e37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:51:57.242 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.225+0000 7f10eaab1640 4 set_mon_vals no callback set 2024-09-16T03:51:57.242 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.225+0000 7f10e37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:51:57.242 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.225+0000 7f10e37fe640 10 monclient: dump: 2024-09-16T03:51:57.242 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:51:57.242 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:57.242 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:57.242 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:57.242 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:51:57.242 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:57.242 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:51:57.242 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:51:57.242 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:51:57.242 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:57.243 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.225+0000 7f10eaab1640 10 monclient: shutdown 2024-09-16T03:51:57.243 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.225+0000 7f10eaab1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:51:57.243 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.225+0000 7f10eaab1640 1 -- 172.21.15.42:0/302957881 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f10e4137e00 msgr2=0x7f10e40a5210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:57.243 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.225+0000 7f10eaab1640 1 --2- 172.21.15.42:0/302957881 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f10e4137e00 0x7f10e40a5210 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f10d40097c0 tx=0x7f10d401c360 comp rx=0 tx=0).stop 2024-09-16T03:51:57.243 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.225+0000 7f10eaab1640 1 -- 172.21.15.42:0/302957881 shutdown_connections 2024-09-16T03:51:57.243 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.225+0000 7f10eaab1640 1 --2- 172.21.15.42:0/302957881 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f10e4137e00 0x7f10e40a5210 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:57.243 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.225+0000 7f10eaab1640 1 --2- 172.21.15.42:0/302957881 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f10e40a4720 0x7f10e41374e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:57.243 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.225+0000 7f10eaab1640 1 -- 172.21.15.42:0/302957881 >> 172.21.15.42:0/302957881 conn(0x7f10e40a6820 msgr2=0x7f10e40a6c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:51:57.243 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.225+0000 7f10eaab1640 1 -- 172.21.15.42:0/302957881 shutdown_connections 2024-09-16T03:51:57.243 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.225+0000 7f10eaab1640 1 -- 172.21.15.42:0/302957881 wait complete. 2024-09-16T03:51:57.243 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.225+0000 7f10eaab1640 10 monclient: build_initial_monmap 2024-09-16T03:51:57.243 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.225+0000 7f10eaab1640 10 monclient: monmap: 2024-09-16T03:51:57.243 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:51:57.244 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:51:57.244 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:51:57.244 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:51:57.244 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:51:57.244 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:57.244 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:51:57.244 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:51:57.244 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:51:57.244 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:57.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.225+0000 7f10eaab1640 1 librados: starting msgr at 2024-09-16T03:51:57.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.225+0000 7f10eaab1640 1 librados: starting objecter 2024-09-16T03:51:57.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.225+0000 7f10eaab1640 1 Processor -- start 2024-09-16T03:51:57.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.225+0000 7f10eaab1640 1 -- start start 2024-09-16T03:51:57.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.225+0000 7f10eaab1640 1 librados: setting wanted keys 2024-09-16T03:51:57.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.225+0000 7f10eaab1640 1 librados: calling monclient init 2024-09-16T03:51:57.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.225+0000 7f10eaab1640 10 monclient: init 2024-09-16T03:51:57.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.225+0000 7f10eaab1640 10 monclient: _reopen_session rank -1 2024-09-16T03:51:57.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.225+0000 7f10eaab1640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:51:57.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.225+0000 7f10eaab1640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f10e40a4720 0x7f10e41cf300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:57.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.225+0000 7f10eaab1640 10 monclient(hunting): picked mon.noname-b con 0x7f10e40a4720 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:51:57.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.225+0000 7f10eaab1640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f10e4137a20 0x7f10e41365f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:57.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.225+0000 7f10eaab1640 10 monclient(hunting): picked mon.noname-a con 0x7f10e4137a20 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:51:57.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.225+0000 7f10eaab1640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f10e4137e00 0x7f10e413c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:57.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.225+0000 7f10eaab1640 10 monclient(hunting): picked mon.noname-c con 0x7f10e4137e00 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:51:57.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.225+0000 7f10eaab1640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:57.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.225+0000 7f10eaab1640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f10e41420e0 con 0x7f10e4137a20 2024-09-16T03:51:57.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.225+0000 7f10eaab1640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:57.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.225+0000 7f10eaab1640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f10e4141f60 con 0x7f10e40a4720 2024-09-16T03:51:57.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.225+0000 7f10eaab1640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:57.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.225+0000 7f10eaab1640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f10e4142260 con 0x7f10e4137e00 2024-09-16T03:51:57.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.225+0000 7f10eaab1640 10 monclient(hunting): _renew_subs 2024-09-16T03:51:57.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.225+0000 7f10eaab1640 10 monclient(hunting): authenticate will time out at 1564.199595s 2024-09-16T03:51:57.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.225+0000 7f10e3fff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f10e4137a20 0x7f10e41365f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:57.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.225+0000 7f10e904e640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f10e4137e00 0x7f10e413c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:57.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.225+0000 7f10e3fff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f10e4137a20 0x7f10e41365f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:37278/0 (socket says 172.21.15.42:37278) 2024-09-16T03:51:57.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.225+0000 7f10e3fff640 1 -- 172.21.15.42:0/1039557411 learned_addr learned my addr 172.21.15.42:0/1039557411 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:51:57.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.225+0000 7f10e3fff640 10 monclient(hunting): get_auth_request con 0x7f10e4137a20 auth_method 0 2024-09-16T03:51:57.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.225+0000 7f10e3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:57.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.225+0000 7f10e3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:57.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.225+0000 7f10e3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:51:57.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.225+0000 7f10e904e640 10 monclient(hunting): get_auth_request con 0x7f10e4137e00 auth_method 0 2024-09-16T03:51:57.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.225+0000 7f10e904e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:57.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.225+0000 7f10e904e640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:57.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.225+0000 7f10e904e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:51:57.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.225+0000 7f10e904e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:51:57.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.225+0000 7f10e904e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:51:57.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.225+0000 7f10e904e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:51:57.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.225+0000 7f10e3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:51:57.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.225+0000 7f10e3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:51:57.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.225+0000 7f10e3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:51:57.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.225+0000 7f10e884d640 1 --2- 172.21.15.42:0/1039557411 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f10e40a4720 0x7f10e41cf300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:57.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.225+0000 7f10e884d640 10 monclient(hunting): get_auth_request con 0x7f10e40a4720 auth_method 0 2024-09-16T03:51:57.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.225+0000 7f10e884d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:57.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.225+0000 7f10e884d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:57.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.225+0000 7f10e884d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:51:57.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.225+0000 7f10e904e640 10 monclient(hunting): handle_auth_done global_id 814211 payload 824 2024-09-16T03:51:57.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.229+0000 7f10e904e640 1 -- 172.21.15.42:0/1039557411 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f10e40a4720 msgr2=0x7f10e41cf300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:57.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.229+0000 7f10e904e640 1 --2- 172.21.15.42:0/1039557411 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f10e40a4720 0x7f10e41cf300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:57.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.229+0000 7f10e904e640 1 -- 172.21.15.42:0/1039557411 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f10e4137a20 msgr2=0x7f10e41365f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:57.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.229+0000 7f10e904e640 1 --2- 172.21.15.42:0/1039557411 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f10e4137a20 0x7f10e41365f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:57.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.229+0000 7f10e904e640 10 monclient: _finish_hunting 0 2024-09-16T03:51:57.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.229+0000 7f10e904e640 1 monclient: found mon.noname-c 2024-09-16T03:51:57.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.229+0000 7f10e904e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:51:57.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.229+0000 7f10e904e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:51:57.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.229+0000 7f10e904e640 1 -- 172.21.15.42:0/1039557411 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10e4120ca0 con 0x7f10e4137e00 2024-09-16T03:51:57.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.229+0000 7f10e3fff640 1 --2- 172.21.15.42:0/1039557411 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f10e4137a20 0x7f10e41365f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:51:57.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.229+0000 7f10e904e640 1 --2- 172.21.15.42:0/1039557411 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f10e4137e00 0x7f10e413c370 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f10e406ef90 tx=0x7f10d4019100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:51:57.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.229+0000 7f10e17fa640 10 client.?.objecter ms_handle_connect 0x7f10e4137e00 2024-09-16T03:51:57.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.229+0000 7f10e17fa640 10 client.?.objecter resend_mon_ops 2024-09-16T03:51:57.249 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.229+0000 7f10e17fa640 1 -- 172.21.15.42:0/1039557411 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10d40b0040 con 0x7f10e4137e00 2024-09-16T03:51:57.249 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.229+0000 7f10e17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:51:57.249 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.229+0000 7f10e17fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:51:57.249 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.229+0000 7f10e17fa640 10 monclient: dump: 2024-09-16T03:51:57.249 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:51:57.249 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:57.249 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:57.249 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:57.249 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:51:57.249 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:57.249 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:51:57.249 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:51:57.249 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:51:57.249 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:57.250 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.229+0000 7f10e17fa640 10 monclient: _finish_auth 0 2024-09-16T03:51:57.250 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.229+0000 7f10e17fa640 10 monclient: _check_auth_tickets 2024-09-16T03:51:57.250 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.229+0000 7f10e17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:51:57.250 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.229+0000 7f10e17fa640 1 -- 172.21.15.42:0/1039557411 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f10d40b9070 con 0x7f10e4137e00 2024-09-16T03:51:57.250 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.229+0000 7f10eaab1640 5 monclient: authenticate success, global_id 814211 2024-09-16T03:51:57.250 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.229+0000 7f10eaab1640 10 monclient: _renew_subs 2024-09-16T03:51:57.250 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.229+0000 7f10eaab1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:51:57.250 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.229+0000 7f10eaab1640 1 -- 172.21.15.42:0/1039557411 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10e4120f80 con 0x7f10e4137e00 2024-09-16T03:51:57.250 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.229+0000 7f10e17fa640 10 monclient: handle_config config(0 keys) 2024-09-16T03:51:57.250 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.229+0000 7f10e17fa640 1 -- 172.21.15.42:0/1039557411 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10d40b4690 con 0x7f10e4137e00 2024-09-16T03:51:57.250 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.229+0000 7f10e17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:51:57.250 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.229+0000 7f10e17fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:51:57.250 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.229+0000 7f10e17fa640 10 monclient: dump: 2024-09-16T03:51:57.250 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:51:57.250 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:57.251 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:57.251 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:57.251 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:51:57.251 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:57.251 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:51:57.251 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:51:57.251 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:51:57.251 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:57.251 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.229+0000 7f10e2ffd640 4 set_mon_vals no callback set 2024-09-16T03:51:57.251 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.229+0000 7f10eaab1640 10 client.814211.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:51:57.251 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.229+0000 7f10eaab1640 10 monclient: _renew_subs 2024-09-16T03:51:57.251 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.229+0000 7f10eaab1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:51:57.251 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.229+0000 7f10eaab1640 1 -- 172.21.15.42:0/1039557411 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f10e40a8470 con 0x7f10e4137e00 2024-09-16T03:51:57.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.229+0000 7f10eaab1640 1 librados: init done 2024-09-16T03:51:57.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.229+0000 7f10eaab1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:51:57.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.229+0000 7f10eaab1640 1 --2- 172.21.15.42:0/1039557411 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f10e4136b30 0x7f10e413c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:57.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.229+0000 7f10eaab1640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:57.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.229+0000 7f10eaab1640 1 -- 172.21.15.42:0/1039557411 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f10e406f8a0 con 0x7f10e4136b30 2024-09-16T03:51:57.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.229+0000 7f10e17fa640 1 -- 172.21.15.42:0/1039557411 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f10d40b4830 con 0x7f10e4137e00 2024-09-16T03:51:57.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.229+0000 7f10e17fa640 1 --2- 172.21.15.42:0/1039557411 >> v2:172.21.15.154:6812/254845342 conn(0x7f10c8037810 0x7f10c8039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:57.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.229+0000 7f10e17fa640 1 -- 172.21.15.42:0/1039557411 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(329..329 src has 286..329) ==== 5684+0+0 (secure 0 0 0) 0x7f10d40e29f0 con 0x7f10e4137e00 2024-09-16T03:51:57.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.229+0000 7f10e17fa640 10 client.814211.objecter ms_dispatch 0x7f10e4000ba0 osd_map(329..329 src has 286..329) 2024-09-16T03:51:57.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.229+0000 7f10e17fa640 3 client.814211.objecter handle_osd_map got epochs [329,329] > 0 2024-09-16T03:51:57.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.229+0000 7f10e17fa640 3 client.814211.objecter handle_osd_map decoding full epoch 329 2024-09-16T03:51:57.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.229+0000 7f10e884d640 1 --2- 172.21.15.42:0/1039557411 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f10e4136b30 0x7f10e413c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:57.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.229+0000 7f10e884d640 10 monclient: get_auth_request con 0x7f10e4136b30 auth_method 0 2024-09-16T03:51:57.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.229+0000 7f10e884d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:57.253 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.229+0000 7f10e884d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:57.253 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.229+0000 7f10e884d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:51:57.253 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.229+0000 7f10e17fa640 20 client.814211.objecter dump_active .. 0 homeless 2024-09-16T03:51:57.253 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.229+0000 7f10e884d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:51:57.253 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.229+0000 7f10e884d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:51:57.253 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.229+0000 7f10e884d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:51:57.253 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.229+0000 7f10e3fff640 1 --2- 172.21.15.42:0/1039557411 >> v2:172.21.15.154:6812/254845342 conn(0x7f10c8037810 0x7f10c8039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:57.253 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.229+0000 7f10e3fff640 10 monclient: get_auth_request con 0x7f10c8037810 auth_method 0 2024-09-16T03:51:57.253 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.229+0000 7f10e884d640 10 monclient(hunting): handle_auth_done global_id 794404 payload 824 2024-09-16T03:51:57.253 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.229+0000 7f10e884d640 1 -- 172.21.15.42:0/1039557411 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f10e41374e0 con 0x7f10e4136b30 2024-09-16T03:51:57.253 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.229+0000 7f10e17fa640 10 client.814211.objecter ms_handle_connect 0x7f10e4136b30 2024-09-16T03:51:57.253 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.229+0000 7f10e17fa640 10 client.814211.objecter resend_mon_ops 2024-09-16T03:51:57.253 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.229+0000 7f10e884d640 1 --2- 172.21.15.42:0/1039557411 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f10e4136b30 0x7f10e413c8b0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f10dc007d30 tx=0x7f10dc010590 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:51:57.253 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.229+0000 7f10e17fa640 10 client.814211.objecter ms_handle_connect 0x7f10c8037810 2024-09-16T03:51:57.254 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.229+0000 7f10e3fff640 1 --2- 172.21.15.42:0/1039557411 >> v2:172.21.15.154:6812/254845342 conn(0x7f10c8037810 0x7f10c8039cd0 secure :-1 s=READY pgs=3570 cs=0 l=1 rev1=1 crypto rx=0x7f10e4120eb0 tx=0x7f10cc007f20 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:51:57.254 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.229+0000 7f10e17fa640 1 -- 172.21.15.42:0/1039557411 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10dc015440 con 0x7f10e4136b30 2024-09-16T03:51:57.254 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.229+0000 7f10e17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:51:57.254 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.229+0000 7f10e17fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:51:57.254 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.229+0000 7f10e17fa640 10 monclient: dump: 2024-09-16T03:51:57.254 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:51:57.254 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:57.254 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:57.254 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:57.254 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:51:57.254 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:57.254 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:51:57.254 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:51:57.255 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:51:57.255 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:57.255 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.229+0000 7f10e17fa640 1 -- 172.21.15.42:0/1039557411 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f10dc015440 con 0x7f10e4136b30 2024-09-16T03:51:57.255 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.229+0000 7f10e17fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:51:57.255 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.229+0000 7f10e17fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:51:57.255 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.229+0000 7f10e17fa640 1 -- 172.21.15.42:0/1039557411 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f10e4136b30 msgr2=0x7f10e413c8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:57.255 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.229+0000 7f10e17fa640 1 --2- 172.21.15.42:0/1039557411 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f10e4136b30 0x7f10e413c8b0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f10dc007d30 tx=0x7f10dc010590 comp rx=0 tx=0).stop 2024-09-16T03:51:57.255 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.229+0000 7f10e17fa640 1 -- 172.21.15.42:0/1039557411 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f10e4136b30 msgr2=0x7f10e413c8b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:51:57.255 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.229+0000 7f10e17fa640 1 --2- 172.21.15.42:0/1039557411 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f10e4136b30 0x7f10e413c8b0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:57.257 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.237+0000 7f10eaab1640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:51:57.257 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.237+0000 7f10eaab1640 1 --2- 172.21.15.42:0/1039557411 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f10e4120880 0x7f10e413cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:57.257 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.237+0000 7f10eaab1640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:57.257 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.237+0000 7f10eaab1640 1 -- 172.21.15.42:0/1039557411 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f10e41d0180 con 0x7f10e4120880 2024-09-16T03:51:57.257 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.237+0000 7f10e884d640 1 --2- 172.21.15.42:0/1039557411 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f10e4120880 0x7f10e413cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:57.257 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.237+0000 7f10e884d640 10 monclient: get_auth_request con 0x7f10e4120880 auth_method 0 2024-09-16T03:51:57.257 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.237+0000 7f10e884d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:57.257 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.237+0000 7f10e884d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:57.257 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.237+0000 7f10e884d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:51:57.257 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.237+0000 7f10e884d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:51:57.258 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.237+0000 7f10e884d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:51:57.258 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.237+0000 7f10e884d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:51:57.258 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.237+0000 7f10e884d640 10 monclient(hunting): handle_auth_done global_id 794407 payload 824 2024-09-16T03:51:57.258 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.237+0000 7f10e884d640 1 -- 172.21.15.42:0/1039557411 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f10e4068e40 con 0x7f10e4120880 2024-09-16T03:51:57.258 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.237+0000 7f10e884d640 1 --2- 172.21.15.42:0/1039557411 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f10e4120880 0x7f10e413cdf0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f10dc007d30 tx=0x7f10dc004a50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:51:57.258 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.237+0000 7f10e17fa640 10 client.814211.objecter ms_handle_connect 0x7f10e4120880 2024-09-16T03:51:57.258 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.237+0000 7f10e17fa640 10 client.814211.objecter resend_mon_ops 2024-09-16T03:51:57.258 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.237+0000 7f10e17fa640 1 -- 172.21.15.42:0/1039557411 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10dc020800 con 0x7f10e4120880 2024-09-16T03:51:57.258 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.237+0000 7f10e17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:51:57.258 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.237+0000 7f10e17fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:51:57.258 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.237+0000 7f10e17fa640 10 monclient: dump: 2024-09-16T03:51:57.258 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:51:57.258 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:57.258 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:57.259 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:57.259 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:51:57.259 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:57.259 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:51:57.259 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:51:57.259 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:51:57.259 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:57.259 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.237+0000 7f10e17fa640 1 -- 172.21.15.42:0/1039557411 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4712 (secure 0 0 0) 0x7f10dc01f440 con 0x7f10e4120880 2024-09-16T03:51:57.259 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.237+0000 7f10e17fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:51:57.259 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.237+0000 7f10e17fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:51:57.259 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.237+0000 7f10e17fa640 1 -- 172.21.15.42:0/1039557411 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f10e4120880 msgr2=0x7f10e413cdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:57.259 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.237+0000 7f10e17fa640 1 --2- 172.21.15.42:0/1039557411 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f10e4120880 0x7f10e413cdf0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f10dc007d30 tx=0x7f10dc004a50 comp rx=0 tx=0).stop 2024-09-16T03:51:57.259 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.237+0000 7f10e17fa640 1 -- 172.21.15.42:0/1039557411 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f10e4120880 msgr2=0x7f10e413cdf0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:51:57.259 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.237+0000 7f10e17fa640 1 --2- 172.21.15.42:0/1039557411 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f10e4120880 0x7f10e413cdf0 secure :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f10dc007d30 tx=0x7f10dc004a50 comp rx=0 tx=0).stop 2024-09-16T03:51:57.260 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:51:57.260 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:51:57.260 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:51:57.260 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T03:51:57.260 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 86, 2024-09-16T03:51:57.260 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 159081, 2024-09-16T03:51:57.260 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:51:57.260 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:51:57.260 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T03:51:57.260 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T03:51:57.261 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:51:57.261 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 0, 2024-09-16T03:51:57.261 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:51:57.261 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:51:57.261 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:51:57.261 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:51:57.261 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:51:57.261 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:51:57.261 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:51:57.261 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:51:57.261 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:51:57.261 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:51:57.261 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:51:57.261 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:51:57.261 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:51:57.262 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:51:57.262 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:51:57.262 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:51:57.262 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:51:57.262 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:51:57.262 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:51:57.262 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:51:57.262 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:51:57.262 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:51:57.262 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:51:57.262 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:51:57.262 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:51:57.262 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:51:57.262 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:51:57.263 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:51:57.263 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:51:57.263 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:57.263 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:51:57.263 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:51:57.263 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:51:57.263 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:51:57.263 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:51:57.263 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:51:57.263 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:51:57.263 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:51:57.263 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:51:57.263 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:51:57.264 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:51:57.264 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:51:57.264 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:51:57.264 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:51:57.264 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:51:57.264 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:51:57.264 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:51:57.264 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:51:57.264 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:51:57.264 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:51:57.264 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:51:57.264 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:51:57.264 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:51:57.264 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:51:57.265 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:51:57.265 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:51:57.265 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:51:57.265 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:51:57.265 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:51:57.265 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:51:57.265 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:57.265 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:51:57.265 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:57.265 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:51:57.265 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:51:57.265 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:51:57.265 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:51:57.266 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:57.266 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:51:57.266 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:51:57.266 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:51:57.266 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:57.266 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:57.266 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:51:57.266 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:51:57.266 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:51:57.266 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:51:57.266 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:51:57.266 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:57.266 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:51:57.267 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:51:57.267 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:51:57.267 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:51:57.267 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:51:57.267 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:57.267 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:57.267 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:51:57.267 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:51:57.267 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:51:57.267 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:51:57.267 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:57.267 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:51:57.267 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:51:57.267 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:51:57.268 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:57.268 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:57.268 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:51:57.268 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:51:57.268 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:51:57.268 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:51:57.268 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:51:57.268 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:57.268 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:51:57.268 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:51:57.268 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:51:57.268 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:51:57.268 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:51:57.269 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:57.269 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:57.269 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:51:57.269 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:51:57.269 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:51:57.269 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:51:57.269 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:57.269 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:51:57.269 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:51:57.269 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:51:57.269 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:57.269 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:57.269 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:51:57.269 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:51:57.270 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:51:57.270 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:51:57.270 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:51:57.270 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:57.270 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:51:57.270 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:51:57.270 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:51:57.270 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:51:57.270 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:51:57.270 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:51:57.270 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:51:57.270 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:57.270 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:51:57.270 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:51:57.271 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:57.271 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:51:57.271 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:51:57.271 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:51:57.271 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:51:57.271 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:51:57.271 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T03:51:57.271 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:57.271 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:51:57.271 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:51:57.271 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:51:57.271 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:51:57.271 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:51:57.271 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:51:57.271 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:57.272 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:51:57.272 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:51:57.272 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 2 2024-09-16T03:51:57.272 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:51:57.272 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:51:57.272 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:57.272 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:51:57.272 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:51:57.274 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.241+0000 7f10eaab1640 10 librados: watch_flush enter 2024-09-16T03:51:57.274 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.241+0000 7f10eaab1640 10 librados: watch_flush exit 2024-09-16T03:51:57.274 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.241+0000 7f10eaab1640 20 client.814211.objecter shutdown clearing up homeless session... 2024-09-16T03:51:57.274 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.241+0000 7f10eaab1640 10 client.814211.objecter successfully canceled tick 2024-09-16T03:51:57.274 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.241+0000 7f10eaab1640 1 -- 172.21.15.42:0/1039557411 >> v2:172.21.15.154:6812/254845342 conn(0x7f10c8037810 msgr2=0x7f10c8039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:57.274 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.241+0000 7f10eaab1640 1 --2- 172.21.15.42:0/1039557411 >> v2:172.21.15.154:6812/254845342 conn(0x7f10c8037810 0x7f10c8039cd0 secure :-1 s=READY pgs=3570 cs=0 l=1 rev1=1 crypto rx=0x7f10e4120eb0 tx=0x7f10cc007f20 comp rx=0 tx=0).stop 2024-09-16T03:51:57.274 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.241+0000 7f10eaab1640 10 monclient: shutdown 2024-09-16T03:51:57.274 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.241+0000 7f10eaab1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:51:57.274 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.241+0000 7f10eaab1640 1 -- 172.21.15.42:0/1039557411 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f10e4137e00 msgr2=0x7f10e413c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:57.274 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.241+0000 7f10eaab1640 1 --2- 172.21.15.42:0/1039557411 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f10e4137e00 0x7f10e413c370 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f10e406ef90 tx=0x7f10d4019100 comp rx=0 tx=0).stop 2024-09-16T03:51:57.274 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.241+0000 7f10e904e640 1 -- 172.21.15.42:0/1039557411 reap_dead start 2024-09-16T03:51:57.275 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.241+0000 7f10e904e640 1 -- 172.21.15.42:0/1039557411 reap_dead start 2024-09-16T03:51:57.275 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.241+0000 7f10eaab1640 1 -- 172.21.15.42:0/1039557411 shutdown_connections 2024-09-16T03:51:57.275 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.241+0000 7f10eaab1640 1 -- 172.21.15.42:0/1039557411 >> 172.21.15.42:0/1039557411 conn(0x7f10e40a6820 msgr2=0x7f10e406d5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:51:57.275 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.241+0000 7f10eaab1640 1 -- 172.21.15.42:0/1039557411 shutdown_connections 2024-09-16T03:51:57.275 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.241+0000 7f10eaab1640 1 -- 172.21.15.42:0/1039557411 wait complete. 2024-09-16T03:51:57.275 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.241+0000 7f10eaab1640 1 librados: shutdown 2024-09-16T03:51:57.275 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:51:57.275 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T03:51:57.275 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T03:51:57.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.381+0000 7faeaa501640 10 monclient: get_monmap_and_config 2024-09-16T03:51:57.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.381+0000 7faeaa501640 10 monclient: build_initial_monmap 2024-09-16T03:51:57.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.381+0000 7faeaa501640 10 monclient: monmap: 2024-09-16T03:51:57.391 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:51:57.391 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:57.391 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:51:57.386864+0000 2024-09-16T03:51:57.391 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:51:57.386864+0000 2024-09-16T03:51:57.391 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:51:57.391 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:57.391 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:51:57.391 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:51:57.391 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:51:57.391 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:51:57.391 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:51:57.392 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:57.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.381+0000 7faeaa501640 1 Processor -- start 2024-09-16T03:51:57.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.381+0000 7faeaa501640 1 -- start start 2024-09-16T03:51:57.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.381+0000 7faeaa501640 10 monclient: init 2024-09-16T03:51:57.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.385+0000 7faeaa501640 10 monclient: _reopen_session rank -1 2024-09-16T03:51:57.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.385+0000 7faeaa501640 10 monclient: _add_conns ranks=[3,1,0,4,2] 2024-09-16T03:51:57.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.385+0000 7faeaa501640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7faea40a1330 0x7faea409f180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:57.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.385+0000 7faeaa501640 10 monclient(hunting): picked mon.noname-a con 0x7faea40a1330 addr v2:172.21.15.42:3300/0 2024-09-16T03:51:57.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.385+0000 7faeaa501640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7faea40a1710 addr v1:172.21.15.154:6789/0 2024-09-16T03:51:57.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.385+0000 7faeaa501640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7faea41393a0 addr v1:172.21.15.42:6789/0 2024-09-16T03:51:57.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.385+0000 7faeaa501640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faea409e4c0 con 0x7faea41393a0 2024-09-16T03:51:57.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.385+0000 7faeaa501640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faea409f6c0 con 0x7faea40a1710 2024-09-16T03:51:57.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.385+0000 7faeaa501640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:57.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.385+0000 7faeaa501640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7faea409edf0 con 0x7faea40a1330 2024-09-16T03:51:57.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.385+0000 7faeaa501640 10 monclient(hunting): _renew_subs 2024-09-16T03:51:57.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.385+0000 7faeaa501640 10 monclient(hunting): authenticate will time out at 1564.356482s 2024-09-16T03:51:57.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.385+0000 7faea8a9e640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7faea41393a0 0x7faea413b790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:57006/0 (socket says 172.21.15.42:57006) 2024-09-16T03:51:57.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.385+0000 7faea8a9e640 1 -- 172.21.15.42:0/1031771838 learned_addr learned my addr 172.21.15.42:0/1031771838 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:51:57.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.385+0000 7faea3fff640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7faea40a1330 0x7faea409f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:57.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.385+0000 7faea3fff640 10 monclient(hunting): get_auth_request con 0x7faea40a1330 auth_method 0 2024-09-16T03:51:57.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.385+0000 7faea3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:57.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.385+0000 7faea3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:57.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.385+0000 7faea3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:51:57.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.385+0000 7faea3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:51:57.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.385+0000 7faea3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:51:57.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.385+0000 7faea3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:51:57.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.385+0000 7faea3fff640 10 monclient(hunting): handle_auth_done global_id 824130 payload 274 2024-09-16T03:51:57.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.385+0000 7faea3fff640 1 -- 172.21.15.42:0/1031771838 >> v1:172.21.15.154:6789/0 conn(0x7faea40a1710 legacy=0x7faea40a1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:57.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.385+0000 7faea3fff640 1 -- 172.21.15.42:0/1031771838 >> v1:172.21.15.42:6789/0 conn(0x7faea41393a0 legacy=0x7faea413b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:57.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.385+0000 7faea3fff640 10 monclient: _finish_hunting 0 2024-09-16T03:51:57.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.385+0000 7faea3fff640 1 monclient: found mon.noname-a 2024-09-16T03:51:57.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.385+0000 7faea3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:51:57.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.385+0000 7faea3fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:51:57.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.385+0000 7faea3fff640 1 -- 172.21.15.42:0/1031771838 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faea40a3170 con 0x7faea40a1330 2024-09-16T03:51:57.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.385+0000 7faea2ffd640 1 -- 172.21.15.42:0/1031771838 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 911449583 0 0) 0x7faea409e4c0 con 0x7faea41393a0 2024-09-16T03:51:57.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.385+0000 7faea2ffd640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T03:51:57.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.385+0000 7faea3fff640 1 --2- 172.21.15.42:0/1031771838 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faea40a1330 0x7faea409f180 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fae940097c0 tx=0x7fae9401c360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=637323fbaab3c774 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:51:57.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.389+0000 7faea2ffd640 1 -- 172.21.15.42:0/1031771838 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae940b0070 con 0x7faea40a1330 2024-09-16T03:51:57.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.389+0000 7faea2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:51:57.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.389+0000 7faea2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:51:57.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.389+0000 7faea2ffd640 10 monclient: dump: 2024-09-16T03:51:57.401 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:51:57.401 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:57.401 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:57.401 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:57.401 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:51:57.401 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:57.401 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:51:57.401 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:51:57.401 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:51:57.401 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:57.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.389+0000 7faea2ffd640 10 monclient: _finish_auth 0 2024-09-16T03:51:57.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.389+0000 7faea2ffd640 10 monclient: _check_auth_tickets 2024-09-16T03:51:57.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.389+0000 7faea2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:51:57.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.389+0000 7faeaa501640 5 monclient: authenticate success, global_id 824130 2024-09-16T03:51:57.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.389+0000 7faea2ffd640 1 -- 172.21.15.42:0/1031771838 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fae940ab600 con 0x7faea40a1330 2024-09-16T03:51:57.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.389+0000 7faeaa501640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:51:57.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.389+0000 7faea2ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:51:57.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.389+0000 7faeaa501640 10 monclient: get_monmap_and_config success 2024-09-16T03:51:57.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.389+0000 7faeaa501640 4 set_mon_vals no callback set 2024-09-16T03:51:57.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.389+0000 7faea2ffd640 1 -- 172.21.15.42:0/1031771838 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae940ab9d0 con 0x7faea40a1330 2024-09-16T03:51:57.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.389+0000 7faea2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:51:57.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.389+0000 7faea2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:51:57.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.389+0000 7faea2ffd640 10 monclient: dump: 2024-09-16T03:51:57.402 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:51:57.402 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:57.403 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:57.403 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:57.403 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:51:57.403 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:57.403 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:51:57.403 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:51:57.403 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:51:57.403 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:57.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.389+0000 7faeaa501640 10 monclient: shutdown 2024-09-16T03:51:57.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.389+0000 7faeaa501640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:51:57.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.389+0000 7faeaa501640 1 -- 172.21.15.42:0/1031771838 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faea40a1330 msgr2=0x7faea409f180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:57.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.389+0000 7faeaa501640 1 --2- 172.21.15.42:0/1031771838 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faea40a1330 0x7faea409f180 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fae940097c0 tx=0x7fae9401c360 comp rx=0 tx=0).stop 2024-09-16T03:51:57.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.389+0000 7faeaa501640 1 -- 172.21.15.42:0/1031771838 shutdown_connections 2024-09-16T03:51:57.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.389+0000 7faeaa501640 1 --2- 172.21.15.42:0/1031771838 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faea40a1330 0x7faea409f180 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:57.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.389+0000 7faeaa501640 1 -- 172.21.15.42:0/1031771838 >> 172.21.15.42:0/1031771838 conn(0x7faea40a2420 msgr2=0x7faea40a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:51:57.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.389+0000 7faeaa501640 1 -- 172.21.15.42:0/1031771838 shutdown_connections 2024-09-16T03:51:57.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.389+0000 7faeaa501640 1 -- 172.21.15.42:0/1031771838 wait complete. 2024-09-16T03:51:57.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.389+0000 7faeaa501640 10 monclient: build_initial_monmap 2024-09-16T03:51:57.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.389+0000 7faeaa501640 10 monclient: monmap: 2024-09-16T03:51:57.404 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:51:57.404 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:51:57.404 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:51:57.404 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:51:57.404 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:51:57.404 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:57.405 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:51:57.405 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:51:57.405 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:51:57.405 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:57.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.389+0000 7faeaa501640 1 librados: starting msgr at 2024-09-16T03:51:57.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.389+0000 7faeaa501640 1 librados: starting objecter 2024-09-16T03:51:57.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.389+0000 7faeaa501640 1 Processor -- start 2024-09-16T03:51:57.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.389+0000 7faeaa501640 1 -- start start 2024-09-16T03:51:57.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.389+0000 7faeaa501640 1 librados: setting wanted keys 2024-09-16T03:51:57.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.389+0000 7faeaa501640 1 librados: calling monclient init 2024-09-16T03:51:57.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.389+0000 7faeaa501640 10 monclient: init 2024-09-16T03:51:57.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.389+0000 7faeaa501640 10 monclient: _reopen_session rank -1 2024-09-16T03:51:57.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.389+0000 7faeaa501640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:51:57.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.389+0000 7faeaa501640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7faea40a1330 0x7faea411bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:57.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.389+0000 7faeaa501640 10 monclient(hunting): picked mon.noname-c con 0x7faea40a1330 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:51:57.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.389+0000 7faeaa501640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faea40a1710 0x7faea411c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:57.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.389+0000 7faeaa501640 10 monclient(hunting): picked mon.noname-a con 0x7faea40a1710 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:51:57.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.389+0000 7faeaa501640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faea41393a0 0x7faea411c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:57.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.389+0000 7faeaa501640 10 monclient(hunting): picked mon.noname-b con 0x7faea41393a0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:51:57.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.389+0000 7faeaa501640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:57.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.389+0000 7faeaa501640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7faea413f8e0 con 0x7faea40a1710 2024-09-16T03:51:57.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.389+0000 7faeaa501640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:57.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.389+0000 7faeaa501640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7faea409cd60 con 0x7faea41393a0 2024-09-16T03:51:57.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.389+0000 7faeaa501640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:57.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.389+0000 7faeaa501640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7faea413fa60 con 0x7faea40a1330 2024-09-16T03:51:57.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.389+0000 7faeaa501640 10 monclient(hunting): _renew_subs 2024-09-16T03:51:57.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.389+0000 7faeaa501640 10 monclient(hunting): authenticate will time out at 1564.362316s 2024-09-16T03:51:57.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.389+0000 7faea37fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faea40a1710 0x7faea411c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:57.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.389+0000 7faea37fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faea40a1710 0x7faea411c4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:37310/0 (socket says 172.21.15.42:37310) 2024-09-16T03:51:57.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.389+0000 7faea37fe640 1 -- 172.21.15.42:0/2755873776 learned_addr learned my addr 172.21.15.42:0/2755873776 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:51:57.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.389+0000 7faea37fe640 10 monclient(hunting): get_auth_request con 0x7faea40a1710 auth_method 0 2024-09-16T03:51:57.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.389+0000 7faea37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:57.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.389+0000 7faea37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:57.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.389+0000 7faea37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:51:57.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.389+0000 7faea3fff640 1 --2- 172.21.15.42:0/2755873776 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7faea40a1330 0x7faea411bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:57.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.389+0000 7faea3fff640 10 monclient(hunting): get_auth_request con 0x7faea40a1330 auth_method 0 2024-09-16T03:51:57.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.389+0000 7faea3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:57.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.389+0000 7faea3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:57.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.389+0000 7faea3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:51:57.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.389+0000 7faea37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:51:57.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.389+0000 7faea37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:51:57.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.389+0000 7faea37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:51:57.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.389+0000 7faea3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:51:57.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.389+0000 7faea3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:51:57.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.389+0000 7faea3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:51:57.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.389+0000 7faea8a9e640 1 --2- 172.21.15.42:0/2755873776 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faea41393a0 0x7faea411c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:57.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.389+0000 7faea8a9e640 10 monclient(hunting): get_auth_request con 0x7faea41393a0 auth_method 0 2024-09-16T03:51:57.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.389+0000 7faea8a9e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:57.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.389+0000 7faea8a9e640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:57.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.389+0000 7faea8a9e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:51:57.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.389+0000 7faea37fe640 10 monclient(hunting): handle_auth_done global_id 824136 payload 824 2024-09-16T03:51:57.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.389+0000 7faea37fe640 1 -- 172.21.15.42:0/2755873776 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7faea40a1330 msgr2=0x7faea411bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:57.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.389+0000 7faea37fe640 1 --2- 172.21.15.42:0/2755873776 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7faea40a1330 0x7faea411bf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:57.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.389+0000 7faea37fe640 1 -- 172.21.15.42:0/2755873776 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faea41393a0 msgr2=0x7faea411c9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:57.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.389+0000 7faea37fe640 1 --2- 172.21.15.42:0/2755873776 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faea41393a0 0x7faea411c9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:57.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.389+0000 7faea37fe640 10 monclient: _finish_hunting 0 2024-09-16T03:51:57.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.389+0000 7faea37fe640 1 monclient: found mon.noname-a 2024-09-16T03:51:57.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.389+0000 7faea37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:51:57.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.389+0000 7faea37fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:51:57.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.389+0000 7faea37fe640 1 -- 172.21.15.42:0/2755873776 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faea411eac0 con 0x7faea40a1710 2024-09-16T03:51:57.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.389+0000 7faea37fe640 1 --2- 172.21.15.42:0/2755873776 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faea40a1710 0x7faea411c4b0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fae8c00c7f0 tx=0x7fae8c00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:51:57.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.389+0000 7faea0ff9640 10 client.?.objecter ms_handle_connect 0x7faea40a1710 2024-09-16T03:51:57.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.389+0000 7faea0ff9640 10 client.?.objecter resend_mon_ops 2024-09-16T03:51:57.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.389+0000 7faea0ff9640 1 -- 172.21.15.42:0/2755873776 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae8c003b80 con 0x7faea40a1710 2024-09-16T03:51:57.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.389+0000 7faea0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:51:57.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.389+0000 7faea0ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:51:57.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.389+0000 7faea0ff9640 10 monclient: dump: 2024-09-16T03:51:57.409 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:51:57.409 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:57.409 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:57.409 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:57.409 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:51:57.410 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:57.410 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:51:57.410 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:51:57.410 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:51:57.410 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:57.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.389+0000 7faea0ff9640 10 monclient: _finish_auth 0 2024-09-16T03:51:57.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.389+0000 7faea0ff9640 10 monclient: _check_auth_tickets 2024-09-16T03:51:57.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.389+0000 7faea0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:51:57.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.389+0000 7faea0ff9640 1 -- 172.21.15.42:0/2755873776 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fae8c003d20 con 0x7faea40a1710 2024-09-16T03:51:57.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.389+0000 7faeaa501640 5 monclient: authenticate success, global_id 824136 2024-09-16T03:51:57.411 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.389+0000 7faeaa501640 10 monclient: _renew_subs 2024-09-16T03:51:57.411 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.389+0000 7faeaa501640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:51:57.411 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.389+0000 7faeaa501640 1 -- 172.21.15.42:0/2755873776 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faea413cc00 con 0x7faea40a1710 2024-09-16T03:51:57.411 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.389+0000 7faea0ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T03:51:57.411 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.389+0000 7faea0ff9640 1 -- 172.21.15.42:0/2755873776 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae8c00aa90 con 0x7faea40a1710 2024-09-16T03:51:57.411 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.389+0000 7faea0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:51:57.411 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.389+0000 7faea0ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:51:57.411 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.389+0000 7faea0ff9640 10 monclient: dump: 2024-09-16T03:51:57.411 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:51:57.411 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:57.411 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:57.411 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:57.411 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:51:57.412 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:57.412 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:51:57.412 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:51:57.412 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:51:57.412 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:57.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.389+0000 7faea27fc640 4 set_mon_vals no callback set 2024-09-16T03:51:57.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.389+0000 7faeaa501640 10 client.824136.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:51:57.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.389+0000 7faeaa501640 10 monclient: _renew_subs 2024-09-16T03:51:57.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.389+0000 7faeaa501640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:51:57.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.389+0000 7faeaa501640 1 -- 172.21.15.42:0/2755873776 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faea411e7e0 con 0x7faea40a1710 2024-09-16T03:51:57.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.389+0000 7faeaa501640 1 librados: init done 2024-09-16T03:51:57.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.393+0000 7faeaa501640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:51:57.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.393+0000 7faeaa501640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:51:57.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.393+0000 7faeaa501640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:51:57.413 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.393+0000 7faea0ff9640 1 -- 172.21.15.42:0/2755873776 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fae8c00ac30 con 0x7faea40a1710 2024-09-16T03:51:57.413 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.393+0000 7faeaa501640 1 -- 172.21.15.42:0/2755873776 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faea409f180 con 0x7faea40a1710 2024-09-16T03:51:57.413 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.393+0000 7faea0ff9640 1 --2- 172.21.15.42:0/2755873776 >> v2:172.21.15.154:6812/254845342 conn(0x7fae88037810 0x7fae88039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:57.413 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.393+0000 7faea0ff9640 1 -- 172.21.15.42:0/2755873776 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(329..329 src has 286..329) ==== 5684+0+0 (secure 0 0 0) 0x7fae8c048e70 con 0x7faea40a1710 2024-09-16T03:51:57.413 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.393+0000 7faea0ff9640 10 client.824136.objecter ms_dispatch 0x7faea4000ba0 osd_map(329..329 src has 286..329) 2024-09-16T03:51:57.413 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.393+0000 7faea0ff9640 3 client.824136.objecter handle_osd_map got epochs [329,329] > 0 2024-09-16T03:51:57.413 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.393+0000 7faea0ff9640 3 client.824136.objecter handle_osd_map decoding full epoch 329 2024-09-16T03:51:57.413 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.393+0000 7faea0ff9640 20 client.824136.objecter dump_active .. 0 homeless 2024-09-16T03:51:57.413 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.393+0000 7faea3fff640 1 --2- 172.21.15.42:0/2755873776 >> v2:172.21.15.154:6812/254845342 conn(0x7fae88037810 0x7fae88039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:57.413 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.393+0000 7faea3fff640 10 monclient: get_auth_request con 0x7fae88037810 auth_method 0 2024-09-16T03:51:57.413 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.393+0000 7faea3fff640 1 --2- 172.21.15.42:0/2755873776 >> v2:172.21.15.154:6812/254845342 conn(0x7fae88037810 0x7fae88039cd0 secure :-1 s=READY pgs=3571 cs=0 l=1 rev1=1 crypto rx=0x7fae940ad000 tx=0x7fae940a6000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:51:57.413 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.393+0000 7faea0ff9640 10 client.824136.objecter ms_handle_connect 0x7fae88037810 2024-09-16T03:51:57.414 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.397+0000 7faea0ff9640 1 -- 172.21.15.42:0/2755873776 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7fae8c012420 con 0x7faea40a1710 2024-09-16T03:51:57.414 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.397+0000 7faea0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:51:57.414 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.397+0000 7faea0ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:51:57.646 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.641+0000 7faeaa501640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T03:51:57.646 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.641+0000 7faeaa501640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T03:51:57.646 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.641+0000 7faeaa501640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:51:57.646 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.641+0000 7faeaa501640 1 -- 172.21.15.42:0/2755873776 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7faea4069040 con 0x7faea40a1710 2024-09-16T03:51:57.648 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.641+0000 7faea0ff9640 1 -- 172.21.15.42:0/2755873776 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7fae8c005080 con 0x7faea40a1710 2024-09-16T03:51:57.648 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.641+0000 7faea0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T03:51:57.648 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.641+0000 7faea0ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:51:57.649 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.641+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:57.649 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.641+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=32,config=2,health=12,kv=9,logm=45} crc {auth=907436828,config=692619927,health=1712667942,kv=950665868,logm=446935378}) 2024-09-16T03:51:57.649 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.641+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=31,config=2,health=11,kv=9,logm=47} crc {auth=1577747026,config=692619927,health=2699014984,kv=867007345,logm=2275123746}) 2024-09-16T03:51:57.649 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.641+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:57.649 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.641+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=32,config=2,health=12,kv=9,logm=45} crc {auth=907436828,config=692619927,health=1712667942,kv=950665868,logm=446935378}) 2024-09-16T03:51:57.649 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.641+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=30,config=1,health=10,kv=10,logm=49} crc {auth=2120777298,config=3678701417,health=3000590527,kv=3446188498,logm=2109900461}) 2024-09-16T03:51:57.650 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.645+0000 7faeaa501640 10 librados: watch_flush enter 2024-09-16T03:51:57.650 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.645+0000 7faeaa501640 10 librados: watch_flush exit 2024-09-16T03:51:57.650 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.645+0000 7faeaa501640 20 client.824136.objecter shutdown clearing up homeless session... 2024-09-16T03:51:57.650 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.645+0000 7faeaa501640 10 client.824136.objecter successfully canceled tick 2024-09-16T03:51:57.651 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.645+0000 7faeaa501640 1 -- 172.21.15.42:0/2755873776 >> v2:172.21.15.154:6812/254845342 conn(0x7fae88037810 msgr2=0x7fae88039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:57.651 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.645+0000 7faeaa501640 1 --2- 172.21.15.42:0/2755873776 >> v2:172.21.15.154:6812/254845342 conn(0x7fae88037810 0x7fae88039cd0 secure :-1 s=READY pgs=3571 cs=0 l=1 rev1=1 crypto rx=0x7fae940ad000 tx=0x7fae940a6000 comp rx=0 tx=0).stop 2024-09-16T03:51:57.651 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.645+0000 7faeaa501640 10 monclient: shutdown 2024-09-16T03:51:57.651 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.645+0000 7faeaa501640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:51:57.651 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.645+0000 7faeaa501640 1 -- 172.21.15.42:0/2755873776 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faea40a1710 msgr2=0x7faea411c4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:57.651 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.645+0000 7faeaa501640 1 --2- 172.21.15.42:0/2755873776 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faea40a1710 0x7faea411c4b0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fae8c00c7f0 tx=0x7fae8c00ccb0 comp rx=0 tx=0).stop 2024-09-16T03:51:57.651 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.645+0000 7faeaa501640 1 -- 172.21.15.42:0/2755873776 shutdown_connections 2024-09-16T03:51:57.651 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.645+0000 7faeaa501640 1 --2- 172.21.15.42:0/2755873776 >> v2:172.21.15.154:6812/254845342 conn(0x7fae88037810 0x7fae88039cd0 unknown :-1 s=CLOSED pgs=3571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:57.651 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.645+0000 7faeaa501640 1 --2- 172.21.15.42:0/2755873776 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faea41393a0 0x7faea411c9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:57.651 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.645+0000 7faeaa501640 1 --2- 172.21.15.42:0/2755873776 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faea40a1710 0x7faea411c4b0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:57.651 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.645+0000 7faeaa501640 1 --2- 172.21.15.42:0/2755873776 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7faea40a1330 0x7faea411bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:57.651 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.645+0000 7faeaa501640 1 -- 172.21.15.42:0/2755873776 >> 172.21.15.42:0/2755873776 conn(0x7faea40a2420 msgr2=0x7faea409ba50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:51:57.652 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.645+0000 7faeaa501640 1 -- 172.21.15.42:0/2755873776 shutdown_connections 2024-09-16T03:51:57.652 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.645+0000 7faeaa501640 1 -- 172.21.15.42:0/2755873776 wait complete. 2024-09-16T03:51:57.652 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:57.645+0000 7faeaa501640 1 librados: shutdown 2024-09-16T03:51:57.652 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.645+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:57.653 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.645+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2794504422}) 2024-09-16T03:51:57.653 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.645+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=571912093}) 2024-09-16T03:51:57.653 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.645+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:57.653 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.645+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2794504422}) 2024-09-16T03:51:57.653 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.645+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2708715080}) 2024-09-16T03:51:57.655 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.649+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:57.655 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.649+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3882058928}) 2024-09-16T03:51:57.655 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.649+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2862684115}) 2024-09-16T03:51:57.655 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.649+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:57.656 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.649+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3882058928}) 2024-09-16T03:51:57.656 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.649+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1815051710}) 2024-09-16T03:51:57.658 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.653+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:57.658 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.653+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=403915050}) 2024-09-16T03:51:57.658 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.653+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2901828219}) 2024-09-16T03:51:57.658 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.653+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:57.658 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.653+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=403915050}) 2024-09-16T03:51:57.658 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.653+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=133173771}) 2024-09-16T03:51:57.660 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.653+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:57.660 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.653+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=49970404}) 2024-09-16T03:51:57.661 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.653+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1901815272}) 2024-09-16T03:51:57.661 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.653+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:57.661 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.653+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=49970404}) 2024-09-16T03:51:57.661 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.653+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2696101800}) 2024-09-16T03:51:57.663 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.657+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:57.663 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.657+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2615793812}) 2024-09-16T03:51:57.663 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.657+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=207906362}) 2024-09-16T03:51:57.663 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.657+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:57.663 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.657+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2615793812}) 2024-09-16T03:51:57.663 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.657+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2953582664}) 2024-09-16T03:51:57.665 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.657+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:57.665 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.657+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4225293364}) 2024-09-16T03:51:57.665 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.657+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2264597843}) 2024-09-16T03:51:57.665 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.657+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:57.665 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.661+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4225293364}) 2024-09-16T03:51:57.665 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.661+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3307621433}) 2024-09-16T03:51:57.668 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T03:51:57.668 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.661+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:57.668 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.661+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1127236461}) 2024-09-16T03:51:57.668 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.661+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1578540226}) 2024-09-16T03:51:57.668 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.661+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:57.669 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.661+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1127236461}) 2024-09-16T03:51:57.669 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.661+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=956415996}) 2024-09-16T03:51:57.670 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.665+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:57.670 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.665+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1595261746}) 2024-09-16T03:51:57.670 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.665+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2441569169}) 2024-09-16T03:51:57.670 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.665+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:57.670 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.665+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1595261746}) 2024-09-16T03:51:57.670 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.665+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3853193927}) 2024-09-16T03:51:57.672 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.665+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:57.672 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.665+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2071327883}) 2024-09-16T03:51:57.672 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.665+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1677997529}) 2024-09-16T03:51:57.673 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.665+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:57.673 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.665+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2071327883}) 2024-09-16T03:51:57.673 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.665+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=735189261}) 2024-09-16T03:51:57.674 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.669+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:57.675 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.669+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2320991286}) 2024-09-16T03:51:57.675 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.669+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3432749441}) 2024-09-16T03:51:57.675 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.669+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:57.675 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.669+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2320991286}) 2024-09-16T03:51:57.675 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.669+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=129666118}) 2024-09-16T03:51:57.677 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.669+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:57.677 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.669+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3071546008}) 2024-09-16T03:51:57.677 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.669+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=909829068}) 2024-09-16T03:51:57.677 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.669+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:57.677 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.669+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3071546008}) 2024-09-16T03:51:57.677 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.669+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2248328840}) 2024-09-16T03:51:57.679 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.673+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:57.679 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.673+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2903425558}) 2024-09-16T03:51:57.679 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.673+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2946336024}) 2024-09-16T03:51:57.679 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.673+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:57.679 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.673+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2903425558}) 2024-09-16T03:51:57.679 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.673+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=958539785}) 2024-09-16T03:51:57.681 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.677+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:57.681 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.677+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4183510347}) 2024-09-16T03:51:57.681 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.677+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=243634163}) 2024-09-16T03:51:57.681 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.677+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:57.681 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.677+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4183510347}) 2024-09-16T03:51:57.681 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.677+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3332821772}) 2024-09-16T03:51:57.683 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.677+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:57.683 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.677+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2633047432}) 2024-09-16T03:51:57.684 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.677+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3044766618}) 2024-09-16T03:51:57.684 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.677+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:57.684 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.677+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2633047432}) 2024-09-16T03:51:57.684 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.677+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2907294953}) 2024-09-16T03:51:57.686 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.681+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:57.686 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.681+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=435745392}) 2024-09-16T03:51:57.686 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.681+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2006794620}) 2024-09-16T03:51:57.686 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.681+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:57.686 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.681+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=435745392}) 2024-09-16T03:51:57.686 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.681+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4140527029}) 2024-09-16T03:51:57.688 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.681+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:57.688 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.681+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1049275263}) 2024-09-16T03:51:57.688 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.681+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=89970532}) 2024-09-16T03:51:57.688 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.681+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:57.688 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.681+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1049275263}) 2024-09-16T03:51:57.688 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.681+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=166279070}) 2024-09-16T03:51:57.690 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.685+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:57.690 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.685+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3289072575}) 2024-09-16T03:51:57.690 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.685+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2538885031}) 2024-09-16T03:51:57.690 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.685+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:57.690 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.685+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3289072575}) 2024-09-16T03:51:57.690 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.685+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3458577243}) 2024-09-16T03:51:57.692 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.685+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:57.692 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.685+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=92936348}) 2024-09-16T03:51:57.692 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.685+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3473525326}) 2024-09-16T03:51:57.693 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.685+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:57.693 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.685+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=92936348}) 2024-09-16T03:51:57.693 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.685+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4253642293}) 2024-09-16T03:51:57.694 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.689+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:57.694 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.689+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3143327086}) 2024-09-16T03:51:57.694 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.689+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2747149457}) 2024-09-16T03:51:57.694 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.689+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:57.694 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.689+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3143327086}) 2024-09-16T03:51:57.694 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.689+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=777268177}) 2024-09-16T03:51:57.697 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.689+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:57.697 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.689+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=430750180}) 2024-09-16T03:51:57.697 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.689+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2319699101}) 2024-09-16T03:51:57.697 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.689+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:57.697 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.689+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=430750180}) 2024-09-16T03:51:57.697 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.689+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=264893790}) 2024-09-16T03:51:57.698 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.693+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:57.698 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.693+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1424928465}) 2024-09-16T03:51:57.698 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.693+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4199958638}) 2024-09-16T03:51:57.699 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.693+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:57.699 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.693+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1424928465}) 2024-09-16T03:51:57.699 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.693+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2776937613}) 2024-09-16T03:51:57.700 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.693+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:57.700 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.693+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3338679829}) 2024-09-16T03:51:57.701 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.693+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2798151222}) 2024-09-16T03:51:57.701 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.693+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:57.701 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.693+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3338679829}) 2024-09-16T03:51:57.701 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.693+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4207813002}) 2024-09-16T03:51:57.702 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.697+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:57.702 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.697+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=314919623}) 2024-09-16T03:51:57.702 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.697+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2686765777}) 2024-09-16T03:51:57.703 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.697+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:57.703 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.697+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=314919623}) 2024-09-16T03:51:57.703 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.697+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1096169083}) 2024-09-16T03:51:57.705 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.697+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:57.705 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.697+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=62847006}) 2024-09-16T03:51:57.705 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.697+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1370997810}) 2024-09-16T03:51:57.705 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.697+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:57.705 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.697+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=62847006}) 2024-09-16T03:51:57.705 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.697+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3927228659}) 2024-09-16T03:51:57.706 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.701+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:57.706 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.701+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=708622391}) 2024-09-16T03:51:57.707 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.701+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=398677540}) 2024-09-16T03:51:57.707 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.701+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:57.707 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.701+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=708622391}) 2024-09-16T03:51:57.707 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.701+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1057840405}) 2024-09-16T03:51:57.708 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.701+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:57.708 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.701+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1836956877}) 2024-09-16T03:51:57.708 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.701+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2835937264}) 2024-09-16T03:51:57.708 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.701+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:57.709 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.701+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1836956877}) 2024-09-16T03:51:57.709 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.701+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=230356742}) 2024-09-16T03:51:57.710 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.705+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:57.710 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.705+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1258900487}) 2024-09-16T03:51:57.710 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.705+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3832385843}) 2024-09-16T03:51:57.710 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.705+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:57.710 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.705+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1258900487}) 2024-09-16T03:51:57.711 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.705+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4275696522}) 2024-09-16T03:51:57.712 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.705+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:57.712 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.705+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=49179486}) 2024-09-16T03:51:57.713 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.705+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4066942967}) 2024-09-16T03:51:57.713 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.705+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:57.713 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.705+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=49179486}) 2024-09-16T03:51:57.713 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.705+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2121162559}) 2024-09-16T03:51:57.715 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.709+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:57.715 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.709+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1983052887}) 2024-09-16T03:51:57.715 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.709+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3731023373}) 2024-09-16T03:51:57.715 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.709+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:57.715 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.709+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1983052887}) 2024-09-16T03:51:57.715 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.709+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3004715013}) 2024-09-16T03:51:57.717 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.709+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:57.717 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.709+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3277248894}) 2024-09-16T03:51:57.717 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.709+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1735141651}) 2024-09-16T03:51:57.717 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.709+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:57.717 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.709+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3277248894}) 2024-09-16T03:51:57.718 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.709+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1252017267}) 2024-09-16T03:51:57.718 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.713+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:57.718 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.713+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4041180052}) 2024-09-16T03:51:57.718 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.713+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2963389784}) 2024-09-16T03:51:57.718 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.713+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:57.718 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.713+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4041180052}) 2024-09-16T03:51:57.719 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.713+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1731559226}) 2024-09-16T03:51:57.720 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.713+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:57.720 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.713+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2090211540}) 2024-09-16T03:51:57.720 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.713+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=598186863}) 2024-09-16T03:51:57.720 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.713+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:57.720 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.713+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2090211540}) 2024-09-16T03:51:57.721 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.713+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2942511147}) 2024-09-16T03:51:57.722 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.717+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:57.722 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.717+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2679918476}) 2024-09-16T03:51:57.722 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.717+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3409334018}) 2024-09-16T03:51:57.722 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.717+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:57.723 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.717+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2679918476}) 2024-09-16T03:51:57.723 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.717+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=655790392}) 2024-09-16T03:51:57.724 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.717+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:57.724 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.717+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1658412093}) 2024-09-16T03:51:57.724 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.717+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1185689576}) 2024-09-16T03:51:57.724 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.717+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:57.724 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.717+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1658412093}) 2024-09-16T03:51:57.724 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.717+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4027150859}) 2024-09-16T03:51:57.726 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.721+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:57.726 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.721+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3262683224}) 2024-09-16T03:51:57.726 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.721+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3096823875}) 2024-09-16T03:51:57.726 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.721+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:57.726 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.721+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3262683224}) 2024-09-16T03:51:57.726 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.721+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3650532049}) 2024-09-16T03:51:57.728 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.721+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:57.728 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.721+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3144226547}) 2024-09-16T03:51:57.728 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.721+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3161969320}) 2024-09-16T03:51:57.728 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.721+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:57.728 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.721+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3144226547}) 2024-09-16T03:51:57.728 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.721+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2189697154}) 2024-09-16T03:51:57.730 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.725+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:57.730 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.725+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4119117551}) 2024-09-16T03:51:57.730 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.725+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=970140792}) 2024-09-16T03:51:57.730 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.725+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:57.730 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.725+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4119117551}) 2024-09-16T03:51:57.730 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.725+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1849495511}) 2024-09-16T03:51:57.732 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.725+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:57.732 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.725+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2489903452}) 2024-09-16T03:51:57.732 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.725+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2127403954}) 2024-09-16T03:51:57.732 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.725+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:57.732 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.725+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2489903452}) 2024-09-16T03:51:57.732 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.725+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1267991859}) 2024-09-16T03:51:57.734 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.729+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:57.734 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.729+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4090274232}) 2024-09-16T03:51:57.735 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.729+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=210022046}) 2024-09-16T03:51:57.735 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.729+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:57.735 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.729+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4090274232}) 2024-09-16T03:51:57.735 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.729+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1307646852}) 2024-09-16T03:51:57.736 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.729+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:57.736 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.729+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2097570971}) 2024-09-16T03:51:57.737 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.729+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3930650929}) 2024-09-16T03:51:57.737 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.729+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:57.737 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.729+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2097570971}) 2024-09-16T03:51:57.737 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.729+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3092461444}) 2024-09-16T03:51:57.738 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.733+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:57.738 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.733+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2489944554}) 2024-09-16T03:51:57.738 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.733+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2451009959}) 2024-09-16T03:51:57.739 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.733+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:57.739 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.733+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2489944554}) 2024-09-16T03:51:57.739 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.733+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=515451074}) 2024-09-16T03:51:57.740 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.733+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:57.741 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.733+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4054015434}) 2024-09-16T03:51:57.741 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.733+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3064739108}) 2024-09-16T03:51:57.741 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.733+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:57.741 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.733+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4054015434}) 2024-09-16T03:51:57.741 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.733+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3798557484}) 2024-09-16T03:51:57.742 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.737+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:57.742 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.737+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2340706731}) 2024-09-16T03:51:57.743 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.737+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2754731607}) 2024-09-16T03:51:57.743 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.737+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:57.743 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.737+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2340706731}) 2024-09-16T03:51:57.743 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.737+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1417849595}) 2024-09-16T03:51:57.744 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.737+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:57.744 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.737+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1918865491}) 2024-09-16T03:51:57.744 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.737+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3792443010}) 2024-09-16T03:51:57.745 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.737+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:57.745 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.737+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1918865491}) 2024-09-16T03:51:57.745 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.737+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4181714399}) 2024-09-16T03:51:57.746 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.741+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:57.746 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.741+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=140342958}) 2024-09-16T03:51:57.746 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.741+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1572588745}) 2024-09-16T03:51:57.746 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.741+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:57.746 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.741+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=140342958}) 2024-09-16T03:51:57.746 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.741+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3737902995}) 2024-09-16T03:51:57.748 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.741+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:57.748 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.741+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1473261062}) 2024-09-16T03:51:57.748 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.741+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4230829218}) 2024-09-16T03:51:57.748 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.741+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:57.748 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.741+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1473261062}) 2024-09-16T03:51:57.749 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.741+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2833472582}) 2024-09-16T03:51:57.750 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.745+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:57.750 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.745+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3013646574}) 2024-09-16T03:51:57.750 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.745+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3432789296}) 2024-09-16T03:51:57.750 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.745+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:57.750 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.745+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3013646574}) 2024-09-16T03:51:57.751 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.745+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1898672868}) 2024-09-16T03:51:57.752 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.745+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:57.752 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.745+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2316660784}) 2024-09-16T03:51:57.752 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.745+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=689427878}) 2024-09-16T03:51:57.752 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.745+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:57.752 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.745+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2316660784}) 2024-09-16T03:51:57.752 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.745+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2335268600}) 2024-09-16T03:51:57.754 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.749+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:57.754 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.749+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2458852027}) 2024-09-16T03:51:57.754 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.749+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1043871537}) 2024-09-16T03:51:57.754 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.749+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:57.754 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.749+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2458852027}) 2024-09-16T03:51:57.754 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.749+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=588995050}) 2024-09-16T03:51:57.756 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.749+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:57.756 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.749+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=423311306}) 2024-09-16T03:51:57.756 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.749+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=585292223}) 2024-09-16T03:51:57.756 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.749+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:57.756 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.749+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=423311306}) 2024-09-16T03:51:57.756 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.749+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=577318817}) 2024-09-16T03:51:57.758 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.753+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:57.758 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.753+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1519657581}) 2024-09-16T03:51:57.758 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.753+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1743800526}) 2024-09-16T03:51:57.758 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.753+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:57.758 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.753+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1519657581}) 2024-09-16T03:51:57.758 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.753+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1260688117}) 2024-09-16T03:51:57.760 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.753+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:57.760 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.753+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1466658473}) 2024-09-16T03:51:57.760 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.753+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2297025209}) 2024-09-16T03:51:57.760 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.753+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:57.760 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.753+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1466658473}) 2024-09-16T03:51:57.760 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.753+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2609746676}) 2024-09-16T03:51:57.762 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.757+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:57.762 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.757+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2484885686}) 2024-09-16T03:51:57.762 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.757+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1954870417}) 2024-09-16T03:51:57.762 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.757+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:57.762 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.757+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2484885686}) 2024-09-16T03:51:57.762 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.757+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1280486800}) 2024-09-16T03:51:57.764 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.757+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:57.764 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.757+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=546417086}) 2024-09-16T03:51:57.764 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.757+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=922318821}) 2024-09-16T03:51:57.764 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.757+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:57.764 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.757+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=546417086}) 2024-09-16T03:51:57.764 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.757+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2311087906}) 2024-09-16T03:51:57.766 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.761+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:57.766 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.761+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=63,mdsmap=1,mgr=8,mgr_metadata=1,mgrstat=14,mon_config_key=10,monmap=2,osd_metadata=1} crc {logm=831581911,mdsmap=1228700967,mgr=460109776,mgr_metadata=1367272202,mgrstat=1952476194,mon_config_key=1530417952,monmap=268092033,osd_metadata=3633039108}) 2024-09-16T03:51:57.766 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.761+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=67,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=14,mon_config_key=7} crc {logm=3657780548,mdsmap=210808725,mgr=460109776,mgr_command_descs=2107673600,mgr_metadata=1367272202,mgrstat=1952476194,mon_config_key=1273075600}) 2024-09-16T03:51:57.766 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.761+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:57.766 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.761+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=63,mdsmap=1,mgr=8,mgr_metadata=1,mgrstat=14,mon_config_key=10,monmap=2,osd_metadata=1} crc {logm=831581911,mdsmap=1228700967,mgr=460109776,mgr_metadata=1367272202,mgrstat=1952476194,mon_config_key=1530417952,monmap=268092033,osd_metadata=3633039108}) 2024-09-16T03:51:57.766 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.761+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=63,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=14,mon_config_key=11} crc {logm=125626397,mdsmap=4088524775,mgr=1406439472,mgr_command_descs=2107673600,mgr_metadata=1367272202,mgrstat=1952476194,mon_config_key=85057112}) 2024-09-16T03:51:57.769 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.761+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:57.769 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.761+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_metadata=5,osd_pg_creating=1,osd_snap=29,osdmap=65} crc {osd_metadata=951161650,osd_pg_creating=818015075,osd_snap=906330396,osdmap=3763807820}) 2024-09-16T03:51:57.769 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.761+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_metadata=5,osd_pg_creating=1,osd_snap=29,osdmap=65} crc {osd_metadata=951161650,osd_pg_creating=818015075,osd_snap=1605731699,osdmap=3990866312}) 2024-09-16T03:51:57.769 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.761+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:57.769 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.765+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_metadata=5,osd_pg_creating=1,osd_snap=29,osdmap=65} crc {osd_metadata=951161650,osd_pg_creating=818015075,osd_snap=906330396,osdmap=3763807820}) 2024-09-16T03:51:57.769 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.765+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_metadata=5,osd_pg_creating=1,osd_snap=30,osdmap=64} crc {osd_metadata=951161650,osd_pg_creating=818015075,osd_snap=1047714956,osdmap=2254326089}) 2024-09-16T03:51:57.786 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.781+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:57.786 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.781+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=17} crc {osdmap=68036985}) 2024-09-16T03:51:57.787 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.781+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=17} crc {osdmap=3926993825}) 2024-09-16T03:51:57.787 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.781+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:51:57.787 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.781+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=17} crc {osdmap=68036985}) 2024-09-16T03:51:57.787 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:51:57.781+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=15} crc {osdmap=1726134970}) 2024-09-16T03:51:58.668 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T03:51:58.669 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:51:58.802 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.797+0000 7f2d0c47f640 10 monclient: get_monmap_and_config 2024-09-16T03:51:58.802 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.797+0000 7f2d0c47f640 10 monclient: build_initial_monmap 2024-09-16T03:51:58.802 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.797+0000 7f2d0c47f640 10 monclient: monmap: 2024-09-16T03:51:58.802 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:51:58.802 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:58.803 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:51:58.801085+0000 2024-09-16T03:51:58.803 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:51:58.801085+0000 2024-09-16T03:51:58.803 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:51:58.803 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:58.803 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:51:58.803 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:51:58.803 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:51:58.803 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:51:58.803 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:51:58.803 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:58.819 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.797+0000 7f2d0c47f640 1 Processor -- start 2024-09-16T03:51:58.819 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.797+0000 7f2d0c47f640 1 -- start start 2024-09-16T03:51:58.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.797+0000 7f2d0c47f640 10 monclient: init 2024-09-16T03:51:58.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.797+0000 7f2d0c47f640 10 monclient: _reopen_session rank -1 2024-09-16T03:51:58.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.797+0000 7f2d0c47f640 10 monclient: _add_conns ranks=[1,0,4,3,2] 2024-09-16T03:51:58.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.797+0000 7f2d0c47f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2d040a1330 addr v1:172.21.15.154:6789/0 2024-09-16T03:51:58.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.797+0000 7f2d0c47f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2d0409c920 addr v1:172.21.15.42:6789/0 2024-09-16T03:51:58.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.797+0000 7f2d0c47f640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f2d04072d40 0x7f2d0413fca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:58.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.797+0000 7f2d0c47f640 10 monclient(hunting): picked mon.noname-c con 0x7f2d04072d40 addr v2:172.21.15.154:3300/0 2024-09-16T03:51:58.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.797+0000 7f2d0c47f640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2d04072350 con 0x7f2d0409c920 2024-09-16T03:51:58.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.797+0000 7f2d0c47f640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2d0409bc60 con 0x7f2d040a1330 2024-09-16T03:51:58.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.797+0000 7f2d0c47f640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:58.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.797+0000 7f2d0c47f640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f2d04072b50 con 0x7f2d04072d40 2024-09-16T03:51:58.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.797+0000 7f2d0c47f640 10 monclient(hunting): _renew_subs 2024-09-16T03:51:58.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.797+0000 7f2d0c47f640 10 monclient(hunting): authenticate will time out at 1565.770508s 2024-09-16T03:51:58.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.797+0000 7f2d09a1a640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f2d0409c920 0x7f2d0413c570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:57016/0 (socket says 172.21.15.42:57016) 2024-09-16T03:51:58.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.797+0000 7f2d09a1a640 1 -- 172.21.15.42:0/880179703 learned_addr learned my addr 172.21.15.42:0/880179703 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:51:58.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.797+0000 7f2d0aa1c640 1 --2- 172.21.15.42:0/880179703 >> v2:172.21.15.154:3300/0 conn(0x7f2d04072d40 0x7f2d0413fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:58.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.797+0000 7f2d0aa1c640 10 monclient(hunting): get_auth_request con 0x7f2d04072d40 auth_method 0 2024-09-16T03:51:58.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.797+0000 7f2d0aa1c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:58.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.797+0000 7f2d0aa1c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:58.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.797+0000 7f2d0aa1c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:51:58.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.797+0000 7f2d09219640 1 -- 172.21.15.42:0/880179703 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1941590620 0 0) 0x7f2d04072350 con 0x7f2d0409c920 2024-09-16T03:51:58.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.797+0000 7f2d09219640 10 monclient(hunting): _negotiate 2024-09-16T03:51:58.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.797+0000 7f2d09219640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:58.823 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.797+0000 7f2d09219640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:51:58.823 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.797+0000 7f2d09219640 10 monclient(hunting): my global_id is 824139 2024-09-16T03:51:58.823 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.797+0000 7f2d09219640 1 -- 172.21.15.42:0/880179703 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2cf0003530 con 0x7f2d0409c920 2024-09-16T03:51:58.823 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.797+0000 7f2d0aa1c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:51:58.823 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.797+0000 7f2d0aa1c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:51:58.823 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.797+0000 7f2d0aa1c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:51:58.823 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.797+0000 7f2d09219640 1 -- 172.21.15.42:0/880179703 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2193101535 0 0) 0x7f2cf0003530 con 0x7f2d0409c920 2024-09-16T03:51:58.823 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.797+0000 7f2d09219640 1 -- 172.21.15.42:0/880179703 >> v2:172.21.15.154:3300/0 conn(0x7f2d04072d40 msgr2=0x7f2d0413fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:51:58.823 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.797+0000 7f2d09219640 1 --2- 172.21.15.42:0/880179703 >> v2:172.21.15.154:3300/0 conn(0x7f2d04072d40 0x7f2d0413fca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:58.823 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.797+0000 7f2d09219640 1 -- 172.21.15.42:0/880179703 >> v1:172.21.15.154:6789/0 conn(0x7f2d040a1330 legacy=0x7f2d040a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:58.823 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.797+0000 7f2d09219640 10 monclient: _finish_hunting 0 2024-09-16T03:51:58.824 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.797+0000 7f2d09219640 1 monclient: found mon.noname-a-legacy 2024-09-16T03:51:58.824 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.797+0000 7f2d09219640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:51:58.824 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.797+0000 7f2d09219640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.42:6789/0 2024-09-16T03:51:58.824 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.797+0000 7f2d09219640 1 -- 172.21.15.42:0/880179703 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d0409bf40 con 0x7f2d0409c920 2024-09-16T03:51:58.824 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.797+0000 7f2d09219640 10 monclient: _finish_auth 0 2024-09-16T03:51:58.824 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.797+0000 7f2d09219640 10 monclient: _check_auth_tickets 2024-09-16T03:51:58.824 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.797+0000 7f2d09219640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:51:58.824 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.797+0000 7f2d0c47f640 5 monclient: authenticate success, global_id 824139 2024-09-16T03:51:58.824 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.797+0000 7f2d09219640 1 -- 172.21.15.42:0/880179703 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2177413734 0 0) 0x7f2d00001680 con 0x7f2d0409c920 2024-09-16T03:51:58.824 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.797+0000 7f2d0c47f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:51:58.824 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.797+0000 7f2d09219640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:51:58.825 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.797+0000 7f2d09219640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T03:51:58.825 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.797+0000 7f2d09219640 10 monclient: dump: 2024-09-16T03:51:58.825 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:51:58.825 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:58.825 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:58.825 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:58.825 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:51:58.825 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:58.825 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:51:58.825 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:51:58.825 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:51:58.826 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:58.826 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.797+0000 7f2d09219640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] but i'm connected to v1:172.21.15.42:6789/0, reconnecting 2024-09-16T03:51:58.826 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.797+0000 7f2d09219640 10 monclient: _reopen_session rank -1 2024-09-16T03:51:58.826 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.797+0000 7f2d09219640 1 -- 172.21.15.42:0/880179703 >> v1:172.21.15.42:6789/0 conn(0x7f2d0409c920 legacy=0x7f2d0413c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:58.826 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.797+0000 7f2d09219640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T03:51:58.826 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.797+0000 7f2d09219640 1 --2- 172.21.15.42:0/880179703 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2cf00039c0 0x7f2cf0003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:58.826 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.797+0000 7f2d09219640 10 monclient(hunting): picked mon.a con 0x7f2cf00039c0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:51:58.826 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.797+0000 7f2d09219640 1 --2- 172.21.15.42:0/880179703 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2cf0004810 0x7f2cf0004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:58.826 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.797+0000 7f2d09219640 10 monclient(hunting): picked mon.b con 0x7f2cf0004810 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:51:58.826 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.797+0000 7f2d09219640 1 --2- 172.21.15.42:0/880179703 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2cf00052f0 0x7f2cf000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:58.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.797+0000 7f2d09219640 10 monclient(hunting): picked mon.c con 0x7f2cf00052f0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:51:58.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.797+0000 7f2d09219640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:58.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.797+0000 7f2d09219640 1 -- 172.21.15.42:0/880179703 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f2d04072b50 con 0x7f2cf00039c0 2024-09-16T03:51:58.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.797+0000 7f2d09219640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:58.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.797+0000 7f2d09219640 1 -- 172.21.15.42:0/880179703 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f2cf000c1c0 con 0x7f2cf0004810 2024-09-16T03:51:58.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.797+0000 7f2d09219640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:58.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.797+0000 7f2d09219640 1 -- 172.21.15.42:0/880179703 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f2cf000c3c0 con 0x7f2cf00052f0 2024-09-16T03:51:58.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.797+0000 7f2d09219640 10 monclient(hunting): _renew_subs 2024-09-16T03:51:58.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.797+0000 7f2d0a21b640 1 --2- 172.21.15.42:0/880179703 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2cf00039c0 0x7f2cf0003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:58.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.797+0000 7f2d09219640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:51:58.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.797+0000 7f2d0a21b640 10 monclient(hunting): get_auth_request con 0x7f2cf00039c0 auth_method 0 2024-09-16T03:51:58.828 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.797+0000 7f2d0a21b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:58.828 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.797+0000 7f2d0a21b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:58.828 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.797+0000 7f2d0a21b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:51:58.828 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.797+0000 7f2d0c47f640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-16T03:51:58.828 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.797+0000 7f2d0aa1c640 1 --2- 172.21.15.42:0/880179703 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2cf00052f0 0x7f2cf000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:58.828 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.797+0000 7f2d0aa1c640 10 monclient(hunting): get_auth_request con 0x7f2cf00052f0 auth_method 0 2024-09-16T03:51:58.828 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.797+0000 7f2d0aa1c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:58.828 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.797+0000 7f2d0aa1c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:58.828 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.797+0000 7f2d0aa1c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:51:58.828 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.797+0000 7f2d0a21b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:51:58.828 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.797+0000 7f2d0a21b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:51:58.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.797+0000 7f2d09a1a640 1 --2- 172.21.15.42:0/880179703 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2cf0004810 0x7f2cf0004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:58.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.797+0000 7f2d0a21b640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T03:51:58.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.797+0000 7f2d09a1a640 10 monclient(hunting): get_auth_request con 0x7f2cf0004810 auth_method 0 2024-09-16T03:51:58.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.797+0000 7f2d09a1a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:58.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.797+0000 7f2d09a1a640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:58.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.797+0000 7f2d09a1a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:51:58.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.797+0000 7f2d0aa1c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:51:58.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.797+0000 7f2d0aa1c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:51:58.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.797+0000 7f2d0aa1c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T03:51:58.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.801+0000 7f2d0a21b640 10 monclient(hunting): handle_auth_done global_id 824139 payload 293 2024-09-16T03:51:58.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.801+0000 7f2d0a21b640 1 -- 172.21.15.42:0/880179703 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2cf00052f0 msgr2=0x7f2cf000b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:51:58.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.801+0000 7f2d0a21b640 1 --2- 172.21.15.42:0/880179703 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2cf00052f0 0x7f2cf000b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:58.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.801+0000 7f2d0a21b640 1 -- 172.21.15.42:0/880179703 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2cf0004810 msgr2=0x7f2cf0004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:51:58.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.801+0000 7f2d0a21b640 1 --2- 172.21.15.42:0/880179703 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2cf0004810 0x7f2cf0004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:58.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.801+0000 7f2d0a21b640 10 monclient: _finish_hunting 0 2024-09-16T03:51:58.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.801+0000 7f2d0a21b640 1 monclient: found mon.a 2024-09-16T03:51:58.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.801+0000 7f2d0a21b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:51:58.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.801+0000 7f2d0a21b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:51:58.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.801+0000 7f2d0a21b640 1 -- 172.21.15.42:0/880179703 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2cf0003530 con 0x7f2cf00039c0 2024-09-16T03:51:58.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.801+0000 7f2d0a21b640 10 monclient: _finish_auth 0 2024-09-16T03:51:58.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.801+0000 7f2d0a21b640 10 monclient: _check_auth_tickets 2024-09-16T03:51:58.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.801+0000 7f2d0a21b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:51:58.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.801+0000 7f2d0aa1c640 1 -- 172.21.15.42:0/880179703 reap_dead start 2024-09-16T03:51:58.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.801+0000 7f2d09a1a640 1 --2- 172.21.15.42:0/880179703 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2cf0004810 0x7f2cf0004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T03:51:58.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.801+0000 7f2d0a21b640 1 --2- 172.21.15.42:0/880179703 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2cf00039c0 0x7f2cf0003de0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f2cf4001750 tx=0x7f2cf401c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=150c3700d5adc088 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:51:58.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.801+0000 7f2d09219640 1 -- 172.21.15.42:0/880179703 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cf40013a0 con 0x7f2cf00039c0 2024-09-16T03:51:58.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.801+0000 7f2d09219640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:51:58.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.801+0000 7f2d09219640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:51:58.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.801+0000 7f2d09219640 10 monclient: dump: 2024-09-16T03:51:58.831 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:51:58.831 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:58.831 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:58.831 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:58.832 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:51:58.832 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:58.832 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:51:58.832 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:51:58.832 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:51:58.832 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:58.832 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.801+0000 7f2d09219640 1 -- 172.21.15.42:0/880179703 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2cf40052f0 con 0x7f2cf00039c0 2024-09-16T03:51:58.832 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.801+0000 7f2d09219640 10 monclient: handle_config config(0 keys) 2024-09-16T03:51:58.832 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.801+0000 7f2d09219640 1 -- 172.21.15.42:0/880179703 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cf4003840 con 0x7f2cf00039c0 2024-09-16T03:51:58.832 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.801+0000 7f2d0c47f640 10 monclient: get_monmap_and_config success 2024-09-16T03:51:58.832 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.801+0000 7f2d0c47f640 4 set_mon_vals no callback set 2024-09-16T03:51:58.832 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.801+0000 7f2d09219640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:51:58.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.801+0000 7f2d09219640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:51:58.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.801+0000 7f2d09219640 10 monclient: dump: 2024-09-16T03:51:58.833 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:51:58.833 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:58.833 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:58.833 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:58.833 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:51:58.833 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:58.833 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:51:58.833 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:51:58.833 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:51:58.834 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:58.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.801+0000 7f2d0c47f640 10 monclient: shutdown 2024-09-16T03:51:58.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.801+0000 7f2d0c47f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:51:58.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.801+0000 7f2d0c47f640 1 -- 172.21.15.42:0/880179703 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2cf00039c0 msgr2=0x7f2cf0003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:58.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.801+0000 7f2d0c47f640 1 --2- 172.21.15.42:0/880179703 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2cf00039c0 0x7f2cf0003de0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f2cf4001750 tx=0x7f2cf401c3a0 comp rx=0 tx=0).stop 2024-09-16T03:51:58.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.801+0000 7f2d0c47f640 1 -- 172.21.15.42:0/880179703 shutdown_connections 2024-09-16T03:51:58.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.801+0000 7f2d0c47f640 1 --2- 172.21.15.42:0/880179703 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2cf00039c0 0x7f2cf0003de0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:58.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.801+0000 7f2d0c47f640 1 -- 172.21.15.42:0/880179703 >> 172.21.15.42:0/880179703 conn(0x7f2d040a2420 msgr2=0x7f2d040a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:51:58.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.801+0000 7f2d0c47f640 1 -- 172.21.15.42:0/880179703 shutdown_connections 2024-09-16T03:51:58.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.801+0000 7f2d0c47f640 1 -- 172.21.15.42:0/880179703 wait complete. 2024-09-16T03:51:58.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.801+0000 7f2d0c47f640 10 monclient: build_initial_monmap 2024-09-16T03:51:58.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.801+0000 7f2d0c47f640 10 monclient: monmap: 2024-09-16T03:51:58.835 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:51:58.835 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:51:58.835 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:51:58.835 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:51:58.835 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:51:58.835 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:58.835 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:51:58.835 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:51:58.835 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:51:58.836 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:58.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.801+0000 7f2d0c47f640 1 librados: starting msgr at 2024-09-16T03:51:58.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.801+0000 7f2d0c47f640 1 librados: starting objecter 2024-09-16T03:51:58.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.801+0000 7f2d0c47f640 1 Processor -- start 2024-09-16T03:51:58.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.805+0000 7f2d0c47f640 1 -- start start 2024-09-16T03:51:58.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.805+0000 7f2d0c47f640 1 librados: setting wanted keys 2024-09-16T03:51:58.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.805+0000 7f2d0c47f640 1 librados: calling monclient init 2024-09-16T03:51:58.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.805+0000 7f2d0c47f640 10 monclient: init 2024-09-16T03:51:58.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.805+0000 7f2d0c47f640 10 monclient: _reopen_session rank -1 2024-09-16T03:51:58.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.805+0000 7f2d0c47f640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:51:58.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.805+0000 7f2d0c47f640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2cf00039c0 0x7f2d041cebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:58.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.805+0000 7f2d0c47f640 10 monclient(hunting): picked mon.noname-c con 0x7f2cf00039c0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:51:58.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.805+0000 7f2d0c47f640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2d04136520 0x7f2d04136980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:58.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.805+0000 7f2d0c47f640 10 monclient(hunting): picked mon.noname-a con 0x7f2d04136520 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:51:58.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.805+0000 7f2d0c47f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d04120f60 0x7f2d041213e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:58.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.805+0000 7f2d0c47f640 10 monclient(hunting): picked mon.noname-b con 0x7f2d04120f60 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:51:58.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.805+0000 7f2d0c47f640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:58.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.805+0000 7f2d0c47f640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f2d04137120 con 0x7f2d04136520 2024-09-16T03:51:58.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.805+0000 7f2d0c47f640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:58.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.805+0000 7f2d0c47f640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f2d040a21c0 con 0x7f2d04120f60 2024-09-16T03:51:58.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.805+0000 7f2d0c47f640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:58.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.805+0000 7f2d0c47f640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f2d041372a0 con 0x7f2cf00039c0 2024-09-16T03:51:58.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.805+0000 7f2d0c47f640 10 monclient(hunting): _renew_subs 2024-09-16T03:51:58.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.805+0000 7f2d0c47f640 10 monclient(hunting): authenticate will time out at 1565.776492s 2024-09-16T03:51:58.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.805+0000 7f2d0a21b640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2cf00039c0 0x7f2d041cebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:58.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.805+0000 7f2d09a1a640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2d04136520 0x7f2d04136980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:58.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.805+0000 7f2d0a21b640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2cf00039c0 0x7f2d041cebe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:41536/0 (socket says 172.21.15.42:41536) 2024-09-16T03:51:58.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.805+0000 7f2d0a21b640 1 -- 172.21.15.42:0/637409761 learned_addr learned my addr 172.21.15.42:0/637409761 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:51:58.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.805+0000 7f2d09a1a640 10 monclient(hunting): get_auth_request con 0x7f2d04136520 auth_method 0 2024-09-16T03:51:58.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.805+0000 7f2d09a1a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:58.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.805+0000 7f2d09a1a640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:58.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.805+0000 7f2d09a1a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:51:58.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.805+0000 7f2d0a21b640 10 monclient(hunting): get_auth_request con 0x7f2cf00039c0 auth_method 0 2024-09-16T03:51:58.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.805+0000 7f2d0a21b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:58.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.805+0000 7f2d0a21b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:58.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.805+0000 7f2d0a21b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:51:58.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.805+0000 7f2d09a1a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:51:58.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.805+0000 7f2d09a1a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:51:58.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.805+0000 7f2d09a1a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:51:58.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.805+0000 7f2d0a21b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:51:58.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.805+0000 7f2d0a21b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:51:58.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.805+0000 7f2d0a21b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:51:58.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.805+0000 7f2d0aa1c640 1 --2- 172.21.15.42:0/637409761 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d04120f60 0x7f2d041213e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:58.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.805+0000 7f2d09a1a640 10 monclient(hunting): handle_auth_done global_id 824142 payload 824 2024-09-16T03:51:58.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.805+0000 7f2d09a1a640 1 -- 172.21.15.42:0/637409761 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2cf00039c0 msgr2=0x7f2d041cebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:58.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.805+0000 7f2d09a1a640 1 --2- 172.21.15.42:0/637409761 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2cf00039c0 0x7f2d041cebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:58.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.805+0000 7f2d09a1a640 1 -- 172.21.15.42:0/637409761 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d04120f60 msgr2=0x7f2d041213e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:58.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.805+0000 7f2d09a1a640 1 --2- 172.21.15.42:0/637409761 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d04120f60 0x7f2d041213e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:58.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.805+0000 7f2d09a1a640 10 monclient: _finish_hunting 0 2024-09-16T03:51:58.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.805+0000 7f2d09a1a640 1 monclient: found mon.noname-a 2024-09-16T03:51:58.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.805+0000 7f2d09a1a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:51:58.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.805+0000 7f2d09a1a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:51:58.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.805+0000 7f2d09a1a640 1 -- 172.21.15.42:0/637409761 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d040a3d70 con 0x7f2d04136520 2024-09-16T03:51:58.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.805+0000 7f2d0aa1c640 10 monclient: get_auth_request con 0x7f2d04120f60 auth_method 0 2024-09-16T03:51:58.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.805+0000 7f2d0a21b640 1 --2- 172.21.15.42:0/637409761 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2cf00039c0 0x7f2d041cebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:51:58.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.805+0000 7f2d0aa1c640 1 --2- 172.21.15.42:0/637409761 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d04120f60 0x7f2d041213e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T03:51:58.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.805+0000 7f2d09a1a640 1 --2- 172.21.15.42:0/637409761 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2d04136520 0x7f2d04136980 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f2d00000c10 tx=0x7f2d0000e9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:51:58.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.805+0000 7f2cfaffd640 10 client.?.objecter ms_handle_connect 0x7f2d04136520 2024-09-16T03:51:58.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.805+0000 7f2cfaffd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:51:58.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.805+0000 7f2cfaffd640 1 -- 172.21.15.42:0/637409761 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d00012590 con 0x7f2d04136520 2024-09-16T03:51:58.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.805+0000 7f2cfaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:51:58.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.805+0000 7f2cfaffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:51:58.841 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.805+0000 7f2cfaffd640 10 monclient: dump: 2024-09-16T03:51:58.841 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:51:58.841 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:58.841 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:58.841 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:58.841 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:51:58.841 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:58.841 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:51:58.841 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:51:58.841 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:51:58.841 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:58.841 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.805+0000 7f2cfaffd640 10 monclient: _finish_auth 0 2024-09-16T03:51:58.841 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.805+0000 7f2cfaffd640 10 monclient: _check_auth_tickets 2024-09-16T03:51:58.841 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.805+0000 7f2cfaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:51:58.842 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.805+0000 7f2cfaffd640 1 -- 172.21.15.42:0/637409761 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d00018070 con 0x7f2d04136520 2024-09-16T03:51:58.842 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.805+0000 7f2cfaffd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:51:58.842 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.805+0000 7f2cfaffd640 1 -- 172.21.15.42:0/637409761 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d00012950 con 0x7f2d04136520 2024-09-16T03:51:58.842 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.805+0000 7f2d0c47f640 5 monclient: authenticate success, global_id 824142 2024-09-16T03:51:58.842 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.805+0000 7f2d0c47f640 10 monclient: _renew_subs 2024-09-16T03:51:58.842 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.805+0000 7f2d0c47f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:51:58.842 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.805+0000 7f2d0c47f640 1 -- 172.21.15.42:0/637409761 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d040a3840 con 0x7f2d04136520 2024-09-16T03:51:58.842 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.805+0000 7f2d08a18640 4 set_mon_vals no callback set 2024-09-16T03:51:58.842 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.805+0000 7f2cfaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:51:58.842 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.805+0000 7f2cfaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:51:58.842 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.805+0000 7f2cfaffd640 10 monclient: dump: 2024-09-16T03:51:58.842 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:51:58.842 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:58.843 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:58.843 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:58.843 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:51:58.843 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:58.843 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:51:58.843 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:51:58.843 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:51:58.843 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:58.843 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.805+0000 7f2d0c47f640 10 client.824142.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:51:58.843 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.805+0000 7f2d0c47f640 10 monclient: _renew_subs 2024-09-16T03:51:58.843 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.805+0000 7f2d0c47f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:51:58.843 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.805+0000 7f2d0c47f640 1 -- 172.21.15.42:0/637409761 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d04124930 con 0x7f2d04136520 2024-09-16T03:51:58.843 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.805+0000 7f2d0c47f640 1 librados: init done 2024-09-16T03:51:58.844 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.805+0000 7f2cfaffd640 1 -- 172.21.15.42:0/637409761 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f2d00012af0 con 0x7f2d04136520 2024-09-16T03:51:58.844 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.805+0000 7f2cf8ff9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:51:58.844 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.805+0000 7f2cf8ff9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:51:58.844 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.805+0000 7f2cf8ff9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:51:58.844 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.805+0000 7f2cf8ff9640 1 -- 172.21.15.42:0/637409761 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ce0005090 con 0x7f2d04136520 2024-09-16T03:51:58.844 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.805+0000 7f2cfaffd640 1 --2- 172.21.15.42:0/637409761 >> v2:172.21.15.154:6812/254845342 conn(0x7f2ce8037840 0x7f2ce8039d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:58.844 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.805+0000 7f2cfaffd640 1 -- 172.21.15.42:0/637409761 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(331..331 src has 286..331) ==== 5713+0+0 (secure 0 0 0) 0x7f2d00048ac0 con 0x7f2d04136520 2024-09-16T03:51:58.844 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.805+0000 7f2cfaffd640 10 client.824142.objecter ms_dispatch 0x7f2d04000ba0 osd_map(331..331 src has 286..331) 2024-09-16T03:51:58.844 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.805+0000 7f2cfaffd640 3 client.824142.objecter handle_osd_map got epochs [331,331] > 0 2024-09-16T03:51:58.844 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.805+0000 7f2cfaffd640 3 client.824142.objecter handle_osd_map decoding full epoch 331 2024-09-16T03:51:58.844 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.805+0000 7f2cfaffd640 20 client.824142.objecter dump_active .. 0 homeless 2024-09-16T03:51:58.844 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.805+0000 7f2d0a21b640 1 --2- 172.21.15.42:0/637409761 >> v2:172.21.15.154:6812/254845342 conn(0x7f2ce8037840 0x7f2ce8039d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:58.844 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.805+0000 7f2d0a21b640 10 monclient: get_auth_request con 0x7f2ce8037840 auth_method 0 2024-09-16T03:51:58.844 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.805+0000 7f2cfaffd640 10 client.824142.objecter ms_handle_connect 0x7f2ce8037840 2024-09-16T03:51:58.845 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.805+0000 7f2d0a21b640 1 --2- 172.21.15.42:0/637409761 >> v2:172.21.15.154:6812/254845342 conn(0x7f2ce8037840 0x7f2ce8039d00 secure :-1 s=READY pgs=3572 cs=0 l=1 rev1=1 crypto rx=0x7f2cf4000c10 tx=0x7f2cf40011a0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:51:58.845 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.809+0000 7f2cfaffd640 1 -- 172.21.15.42:0/637409761 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f2d040a3d70 con 0x7f2d04136520 2024-09-16T03:51:58.845 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.809+0000 7f2cfaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:51:58.845 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:58.809+0000 7f2cfaffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:51:59.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.037+0000 7f2cf8ff9640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:51:59.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.037+0000 7f2cf8ff9640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:51:59.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.037+0000 7f2cf8ff9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:51:59.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.037+0000 7f2cf8ff9640 1 -- 172.21.15.42:0/637409761 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f2ce0005380 con 0x7f2d04136520 2024-09-16T03:51:59.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.037+0000 7f2cfaffd640 1 -- 172.21.15.42:0/637409761 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7f2d0001be40 con 0x7f2d04136520 2024-09-16T03:51:59.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.037+0000 7f2cfaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:51:59.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.037+0000 7f2cfaffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:51:59.043 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":86,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:51:59.045 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.037+0000 7f2cf8ff9640 10 librados: watch_flush enter 2024-09-16T03:51:59.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.041+0000 7f2cf8ff9640 10 librados: watch_flush exit 2024-09-16T03:51:59.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.041+0000 7f2cf8ff9640 20 client.824142.objecter shutdown clearing up homeless session... 2024-09-16T03:51:59.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.041+0000 7f2cf8ff9640 10 client.824142.objecter successfully canceled tick 2024-09-16T03:51:59.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.041+0000 7f2cf8ff9640 1 -- 172.21.15.42:0/637409761 >> v2:172.21.15.154:6812/254845342 conn(0x7f2ce8037840 msgr2=0x7f2ce8039d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:59.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.041+0000 7f2cf8ff9640 1 --2- 172.21.15.42:0/637409761 >> v2:172.21.15.154:6812/254845342 conn(0x7f2ce8037840 0x7f2ce8039d00 secure :-1 s=READY pgs=3572 cs=0 l=1 rev1=1 crypto rx=0x7f2cf4000c10 tx=0x7f2cf40011a0 comp rx=0 tx=0).stop 2024-09-16T03:51:59.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.041+0000 7f2cf8ff9640 10 monclient: shutdown 2024-09-16T03:51:59.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.041+0000 7f2cf8ff9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:51:59.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.041+0000 7f2cf8ff9640 1 -- 172.21.15.42:0/637409761 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2d04136520 msgr2=0x7f2d04136980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:59.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.041+0000 7f2cf8ff9640 1 --2- 172.21.15.42:0/637409761 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2d04136520 0x7f2d04136980 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f2d00000c10 tx=0x7f2d0000e9c0 comp rx=0 tx=0).stop 2024-09-16T03:51:59.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.041+0000 7f2cf8ff9640 1 -- 172.21.15.42:0/637409761 shutdown_connections 2024-09-16T03:51:59.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.041+0000 7f2cf8ff9640 1 --2- 172.21.15.42:0/637409761 >> v2:172.21.15.154:6812/254845342 conn(0x7f2ce8037840 0x7f2ce8039d00 unknown :-1 s=CLOSED pgs=3572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:59.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.041+0000 7f2cf8ff9640 1 --2- 172.21.15.42:0/637409761 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d04120f60 0x7f2d041213e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:59.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.041+0000 7f2cf8ff9640 1 --2- 172.21.15.42:0/637409761 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2d04136520 0x7f2d04136980 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:59.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.041+0000 7f2cf8ff9640 1 --2- 172.21.15.42:0/637409761 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2cf00039c0 0x7f2d041cebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:59.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.041+0000 7f2cf8ff9640 1 -- 172.21.15.42:0/637409761 >> 172.21.15.42:0/637409761 conn(0x7f2d040a2420 msgr2=0x7f2d040a4c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:51:59.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.041+0000 7f2cf8ff9640 1 -- 172.21.15.42:0/637409761 shutdown_connections 2024-09-16T03:51:59.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.041+0000 7f2cf8ff9640 1 -- 172.21.15.42:0/637409761 wait complete. 2024-09-16T03:51:59.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.041+0000 7f2cf8ff9640 1 librados: shutdown 2024-09-16T03:51:59.062 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T03:51:59.062 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T03:51:59.062 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T03:51:59.062 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T03:51:59.190 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.185+0000 7f3c14781640 10 monclient: get_monmap_and_config 2024-09-16T03:51:59.190 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.185+0000 7f3c14781640 10 monclient: build_initial_monmap 2024-09-16T03:51:59.190 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.185+0000 7f3c14781640 10 monclient: monmap: 2024-09-16T03:51:59.190 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:51:59.191 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:59.191 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:51:59.189182+0000 2024-09-16T03:51:59.191 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:51:59.189182+0000 2024-09-16T03:51:59.191 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:51:59.191 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:59.191 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:51:59.191 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:51:59.191 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:51:59.191 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:51:59.191 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:51:59.191 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:59.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.185+0000 7f3c14781640 1 Processor -- start 2024-09-16T03:51:59.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.185+0000 7f3c14781640 1 -- start start 2024-09-16T03:51:59.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.185+0000 7f3c14781640 10 monclient: init 2024-09-16T03:51:59.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.185+0000 7f3c14781640 10 monclient: _reopen_session rank -1 2024-09-16T03:51:59.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.185+0000 7f3c14781640 10 monclient: _add_conns ranks=[3,1,2,0,4] 2024-09-16T03:51:59.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.185+0000 7f3c14781640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f3c0c0a39b0 0x7f3c0c133b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:59.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.185+0000 7f3c14781640 10 monclient(hunting): picked mon.noname-a con 0x7f3c0c0a39b0 addr v2:172.21.15.42:3300/0 2024-09-16T03:51:59.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.185+0000 7f3c14781640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3c0c0a3d90 addr v1:172.21.15.154:6789/0 2024-09-16T03:51:59.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.185+0000 7f3c14781640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3c0c137710 0x7f3c0c13f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:59.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.185+0000 7f3c14781640 10 monclient(hunting): picked mon.noname-b con 0x7f3c0c137710 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:51:59.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.185+0000 7f3c14781640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3c0c09c140 con 0x7f3c0c0a3d90 2024-09-16T03:51:59.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.185+0000 7f3c14781640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:59.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.185+0000 7f3c14781640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f3c0c09c420 con 0x7f3c0c0a39b0 2024-09-16T03:51:59.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.185+0000 7f3c14781640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:59.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.185+0000 7f3c14781640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f3c0c06dd60 con 0x7f3c0c137710 2024-09-16T03:51:59.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.185+0000 7f3c14781640 10 monclient(hunting): _renew_subs 2024-09-16T03:51:59.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.185+0000 7f3c14781640 10 monclient(hunting): authenticate will time out at 1566.159278s 2024-09-16T03:51:59.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.185+0000 7f3c12d1e640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3c0c137710 0x7f3c0c13f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:59.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.185+0000 7f3c1251d640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f3c0c0a39b0 0x7f3c0c133b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:59.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.185+0000 7f3c12d1e640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3c0c137710 0x7f3c0c13f3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:41548/0 (socket says 172.21.15.42:41548) 2024-09-16T03:51:59.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.185+0000 7f3c12d1e640 1 -- 172.21.15.42:0/92013167 learned_addr learned my addr 172.21.15.42:0/92013167 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:51:59.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.185+0000 7f3c12d1e640 10 monclient(hunting): get_auth_request con 0x7f3c0c137710 auth_method 0 2024-09-16T03:51:59.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.185+0000 7f3c12d1e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:59.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.185+0000 7f3c12d1e640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:59.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.185+0000 7f3c12d1e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:51:59.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.185+0000 7f3c1251d640 10 monclient(hunting): get_auth_request con 0x7f3c0c0a39b0 auth_method 0 2024-09-16T03:51:59.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.185+0000 7f3c1251d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:59.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.185+0000 7f3c1251d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:59.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.185+0000 7f3c1251d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:51:59.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.185+0000 7f3c12d1e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:51:59.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.185+0000 7f3c12d1e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:51:59.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.185+0000 7f3c12d1e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:51:59.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.185+0000 7f3c1251d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:51:59.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.185+0000 7f3c1251d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:51:59.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.185+0000 7f3c1251d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:51:59.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.185+0000 7f3c1251d640 10 monclient(hunting): handle_auth_done global_id 824145 payload 274 2024-09-16T03:51:59.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.185+0000 7f3c1251d640 1 -- 172.21.15.42:0/92013167 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3c0c137710 msgr2=0x7f3c0c13f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:51:59.197 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.185+0000 7f3c1251d640 1 --2- 172.21.15.42:0/92013167 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3c0c137710 0x7f3c0c13f3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:59.197 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.185+0000 7f3c1251d640 1 -- 172.21.15.42:0/92013167 >> v1:172.21.15.154:6789/0 conn(0x7f3c0c0a3d90 legacy=0x7f3c0c137000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:59.197 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.185+0000 7f3c1251d640 10 monclient: _finish_hunting 0 2024-09-16T03:51:59.197 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.185+0000 7f3c1251d640 1 monclient: found mon.noname-a 2024-09-16T03:51:59.197 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.185+0000 7f3c1251d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:51:59.197 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.185+0000 7f3c1251d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:51:59.197 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.185+0000 7f3c1251d640 1 -- 172.21.15.42:0/92013167 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c0c0a5e20 con 0x7f3c0c0a39b0 2024-09-16T03:51:59.197 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.185+0000 7f3c12d1e640 1 --2- 172.21.15.42:0/92013167 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3c0c137710 0x7f3c0c13f3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:51:59.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.189+0000 7f3c1251d640 1 --2- 172.21.15.42:0/92013167 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3c0c0a39b0 0x7f3c0c133b80 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f3bfc009d90 tx=0x7f3bfc01c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e93eb76c840b8e3 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:51:59.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.189+0000 7f3c1151b640 1 -- 172.21.15.42:0/92013167 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bfc0a6700 con 0x7f3c0c0a39b0 2024-09-16T03:51:59.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.189+0000 7f3c1151b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:51:59.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.189+0000 7f3c1151b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:51:59.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.189+0000 7f3c1151b640 10 monclient: dump: 2024-09-16T03:51:59.204 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:51:59.204 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:59.204 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:59.205 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:59.205 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:51:59.205 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:59.205 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:51:59.205 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:51:59.205 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:51:59.205 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:59.205 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.189+0000 7f3c1151b640 10 monclient: _finish_auth 0 2024-09-16T03:51:59.205 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.189+0000 7f3c1151b640 10 monclient: _check_auth_tickets 2024-09-16T03:51:59.205 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.189+0000 7f3c1151b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:51:59.205 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.189+0000 7f3c1151b640 1 -- 172.21.15.42:0/92013167 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3bfc0a68a0 con 0x7f3c0c0a39b0 2024-09-16T03:51:59.205 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.189+0000 7f3c14781640 5 monclient: authenticate success, global_id 824145 2024-09-16T03:51:59.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.189+0000 7f3c14781640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:51:59.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.189+0000 7f3c1151b640 10 monclient: handle_config config(0 keys) 2024-09-16T03:51:59.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.189+0000 7f3c14781640 10 monclient: get_monmap_and_config success 2024-09-16T03:51:59.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.189+0000 7f3c14781640 4 set_mon_vals no callback set 2024-09-16T03:51:59.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.189+0000 7f3c1151b640 1 -- 172.21.15.42:0/92013167 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bfc0a6c70 con 0x7f3c0c0a39b0 2024-09-16T03:51:59.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.189+0000 7f3c1151b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:51:59.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.189+0000 7f3c1151b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:51:59.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.189+0000 7f3c1151b640 10 monclient: dump: 2024-09-16T03:51:59.206 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:51:59.206 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:59.206 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:59.206 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:59.206 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:51:59.207 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:59.207 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:51:59.207 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:51:59.207 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:51:59.207 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:59.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.189+0000 7f3c14781640 10 monclient: shutdown 2024-09-16T03:51:59.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.189+0000 7f3c14781640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:51:59.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.189+0000 7f3c14781640 1 -- 172.21.15.42:0/92013167 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3c0c0a39b0 msgr2=0x7f3c0c133b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:59.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.189+0000 7f3c14781640 1 --2- 172.21.15.42:0/92013167 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3c0c0a39b0 0x7f3c0c133b80 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f3bfc009d90 tx=0x7f3bfc01c830 comp rx=0 tx=0).stop 2024-09-16T03:51:59.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.189+0000 7f3c14781640 1 -- 172.21.15.42:0/92013167 shutdown_connections 2024-09-16T03:51:59.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.189+0000 7f3c14781640 1 --2- 172.21.15.42:0/92013167 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3c0c137710 0x7f3c0c13f3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:59.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.189+0000 7f3c14781640 1 --2- 172.21.15.42:0/92013167 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3c0c0a39b0 0x7f3c0c133b80 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:59.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.189+0000 7f3c14781640 1 -- 172.21.15.42:0/92013167 >> 172.21.15.42:0/92013167 conn(0x7f3c0c0a9fb0 msgr2=0x7f3c0c0aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:51:59.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.189+0000 7f3c14781640 1 -- 172.21.15.42:0/92013167 shutdown_connections 2024-09-16T03:51:59.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.189+0000 7f3c14781640 1 -- 172.21.15.42:0/92013167 wait complete. 2024-09-16T03:51:59.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.189+0000 7f3c14781640 10 monclient: build_initial_monmap 2024-09-16T03:51:59.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.189+0000 7f3c14781640 10 monclient: monmap: 2024-09-16T03:51:59.208 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:51:59.208 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:51:59.208 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:51:59.208 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:51:59.208 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:51:59.208 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:59.208 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:51:59.208 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:51:59.208 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:51:59.208 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:59.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.189+0000 7f3c14781640 1 librados: starting msgr at 2024-09-16T03:51:59.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.189+0000 7f3c14781640 1 librados: starting objecter 2024-09-16T03:51:59.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.189+0000 7f3c14781640 1 Processor -- start 2024-09-16T03:51:59.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.189+0000 7f3c14781640 1 -- start start 2024-09-16T03:51:59.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.189+0000 7f3c14781640 1 librados: setting wanted keys 2024-09-16T03:51:59.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.189+0000 7f3c14781640 1 librados: calling monclient init 2024-09-16T03:51:59.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.189+0000 7f3c14781640 10 monclient: init 2024-09-16T03:51:59.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.193+0000 7f3c14781640 10 monclient: _reopen_session rank -1 2024-09-16T03:51:59.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.193+0000 7f3c14781640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:51:59.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.193+0000 7f3c14781640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3c0c0a39b0 0x7f3c0c12d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:59.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.193+0000 7f3c14781640 10 monclient(hunting): picked mon.noname-c con 0x7f3c0c0a39b0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:51:59.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.193+0000 7f3c14781640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3c0c0a3d90 0x7f3c0c12d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:59.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.193+0000 7f3c14781640 10 monclient(hunting): picked mon.noname-a con 0x7f3c0c0a3d90 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:51:59.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.193+0000 7f3c14781640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3c0c137710 0x7f3c0c12dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:59.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.193+0000 7f3c14781640 10 monclient(hunting): picked mon.noname-b con 0x7f3c0c137710 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:51:59.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.193+0000 7f3c14781640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:59.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.193+0000 7f3c14781640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f3c0c142310 con 0x7f3c0c0a3d90 2024-09-16T03:51:59.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.193+0000 7f3c14781640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:59.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.193+0000 7f3c14781640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f3c0c142490 con 0x7f3c0c137710 2024-09-16T03:51:59.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.193+0000 7f3c14781640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:59.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.193+0000 7f3c14781640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f3c0c142610 con 0x7f3c0c0a39b0 2024-09-16T03:51:59.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.193+0000 7f3c14781640 10 monclient(hunting): _renew_subs 2024-09-16T03:51:59.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.193+0000 7f3c14781640 10 monclient(hunting): authenticate will time out at 1566.164349s 2024-09-16T03:51:59.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.193+0000 7f3c1251d640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3c0c0a39b0 0x7f3c0c12d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:59.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.193+0000 7f3c11d1c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3c0c0a3d90 0x7f3c0c12d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:59.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.193+0000 7f3c1251d640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3c0c0a39b0 0x7f3c0c12d2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:41550/0 (socket says 172.21.15.42:41550) 2024-09-16T03:51:59.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.193+0000 7f3c1251d640 1 -- 172.21.15.42:0/1925804287 learned_addr learned my addr 172.21.15.42:0/1925804287 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:51:59.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.193+0000 7f3c11d1c640 10 monclient(hunting): get_auth_request con 0x7f3c0c0a3d90 auth_method 0 2024-09-16T03:51:59.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.193+0000 7f3c11d1c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:59.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.193+0000 7f3c11d1c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:59.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.193+0000 7f3c11d1c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:51:59.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.193+0000 7f3c1251d640 10 monclient(hunting): get_auth_request con 0x7f3c0c0a39b0 auth_method 0 2024-09-16T03:51:59.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.193+0000 7f3c1251d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:59.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.193+0000 7f3c1251d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:59.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.193+0000 7f3c1251d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:51:59.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.193+0000 7f3c11d1c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:51:59.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.193+0000 7f3c11d1c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:51:59.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.193+0000 7f3c11d1c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:51:59.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.193+0000 7f3c1251d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:51:59.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.193+0000 7f3c1251d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:51:59.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.193+0000 7f3c1251d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:51:59.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.193+0000 7f3c12d1e640 1 --2- 172.21.15.42:0/1925804287 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3c0c137710 0x7f3c0c12dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:59.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.193+0000 7f3c12d1e640 10 monclient(hunting): get_auth_request con 0x7f3c0c137710 auth_method 0 2024-09-16T03:51:59.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.193+0000 7f3c12d1e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:59.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.193+0000 7f3c12d1e640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:59.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.193+0000 7f3c12d1e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:51:59.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.193+0000 7f3c11d1c640 10 monclient(hunting): handle_auth_done global_id 824148 payload 824 2024-09-16T03:51:59.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.193+0000 7f3c11d1c640 1 -- 172.21.15.42:0/1925804287 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3c0c0a39b0 msgr2=0x7f3c0c12d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:59.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.193+0000 7f3c11d1c640 1 --2- 172.21.15.42:0/1925804287 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3c0c0a39b0 0x7f3c0c12d2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:59.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.193+0000 7f3c11d1c640 1 -- 172.21.15.42:0/1925804287 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3c0c137710 msgr2=0x7f3c0c12dd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:59.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.193+0000 7f3c11d1c640 1 --2- 172.21.15.42:0/1925804287 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3c0c137710 0x7f3c0c12dd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:59.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.193+0000 7f3c11d1c640 10 monclient: _finish_hunting 0 2024-09-16T03:51:59.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.193+0000 7f3c11d1c640 1 monclient: found mon.noname-a 2024-09-16T03:51:59.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.193+0000 7f3c11d1c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:51:59.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.193+0000 7f3c11d1c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:51:59.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.193+0000 7f3c11d1c640 1 -- 172.21.15.42:0/1925804287 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c0c06f680 con 0x7f3c0c0a3d90 2024-09-16T03:51:59.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.193+0000 7f3c11d1c640 1 --2- 172.21.15.42:0/1925804287 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3c0c0a3d90 0x7f3c0c12d810 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f3c0000c3f0 tx=0x7f3c0000c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:51:59.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.193+0000 7f3c0affd640 10 client.?.objecter ms_handle_connect 0x7f3c0c0a3d90 2024-09-16T03:51:59.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.193+0000 7f3c0affd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:51:59.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.193+0000 7f3c0affd640 1 -- 172.21.15.42:0/1925804287 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c000052d0 con 0x7f3c0c0a3d90 2024-09-16T03:51:59.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.193+0000 7f3c0affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:51:59.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.193+0000 7f3c0affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:51:59.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.193+0000 7f3c0affd640 10 monclient: dump: 2024-09-16T03:51:59.213 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:51:59.213 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:59.214 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:59.214 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:59.214 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:51:59.214 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:59.214 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:51:59.214 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:51:59.214 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:51:59.214 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:59.214 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.193+0000 7f3c0affd640 10 monclient: _finish_auth 0 2024-09-16T03:51:59.214 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.193+0000 7f3c0affd640 10 monclient: _check_auth_tickets 2024-09-16T03:51:59.214 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.193+0000 7f3c0affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:51:59.214 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.193+0000 7f3c0affd640 1 -- 172.21.15.42:0/1925804287 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3c00005470 con 0x7f3c0c0a3d90 2024-09-16T03:51:59.214 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.193+0000 7f3c14781640 5 monclient: authenticate success, global_id 824148 2024-09-16T03:51:59.214 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.193+0000 7f3c14781640 10 monclient: _renew_subs 2024-09-16T03:51:59.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.193+0000 7f3c14781640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:51:59.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.193+0000 7f3c14781640 1 -- 172.21.15.42:0/1925804287 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c0c13ba30 con 0x7f3c0c0a3d90 2024-09-16T03:51:59.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.193+0000 7f3c0affd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:51:59.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.193+0000 7f3c0affd640 1 -- 172.21.15.42:0/1925804287 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c00007260 con 0x7f3c0c0a3d90 2024-09-16T03:51:59.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.193+0000 7f3c0affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:51:59.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.193+0000 7f3c0affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:51:59.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.193+0000 7f3c0affd640 10 monclient: dump: 2024-09-16T03:51:59.215 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:51:59.215 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:59.215 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:59.215 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:59.215 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:51:59.215 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:59.215 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:51:59.216 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:51:59.216 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:51:59.216 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:59.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.193+0000 7f3c14781640 10 client.824148.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:51:59.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.193+0000 7f3c10d1a640 4 set_mon_vals no callback set 2024-09-16T03:51:59.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.193+0000 7f3c14781640 10 monclient: _renew_subs 2024-09-16T03:51:59.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.193+0000 7f3c14781640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:51:59.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.193+0000 7f3c14781640 1 -- 172.21.15.42:0/1925804287 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c0c13f070 con 0x7f3c0c0a3d90 2024-09-16T03:51:59.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.193+0000 7f3c14781640 1 librados: init done 2024-09-16T03:51:59.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.193+0000 7f3c14781640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:51:59.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.193+0000 7f3c14781640 1 --2- 172.21.15.42:0/1925804287 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3c0c139ef0 0x7f3c0c13a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:59.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.193+0000 7f3c14781640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:59.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.193+0000 7f3c14781640 1 -- 172.21.15.42:0/1925804287 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f3c0c1342d0 con 0x7f3c0c139ef0 2024-09-16T03:51:59.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.193+0000 7f3c1251d640 1 --2- 172.21.15.42:0/1925804287 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3c0c139ef0 0x7f3c0c13a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:59.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.193+0000 7f3c1251d640 10 monclient: get_auth_request con 0x7f3c0c139ef0 auth_method 0 2024-09-16T03:51:59.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.193+0000 7f3c1251d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:59.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.193+0000 7f3c1251d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:59.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.193+0000 7f3c1251d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:51:59.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.193+0000 7f3c0affd640 1 -- 172.21.15.42:0/1925804287 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f3c00007400 con 0x7f3c0c0a3d90 2024-09-16T03:51:59.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.193+0000 7f3c1251d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:51:59.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.193+0000 7f3c1251d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:51:59.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.193+0000 7f3c1251d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:51:59.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.193+0000 7f3c1251d640 10 monclient(hunting): handle_auth_done global_id 824151 payload 824 2024-09-16T03:51:59.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.193+0000 7f3c0affd640 1 --2- 172.21.15.42:0/1925804287 >> v2:172.21.15.154:6812/254845342 conn(0x7f3bf8037810 0x7f3bf8039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:59.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.193+0000 7f3c1251d640 1 -- 172.21.15.42:0/1925804287 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3c0c133b80 con 0x7f3c0c139ef0 2024-09-16T03:51:59.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.193+0000 7f3c0affd640 1 -- 172.21.15.42:0/1925804287 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(331..331 src has 286..331) ==== 5713+0+0 (secure 0 0 0) 0x7f3c00048470 con 0x7f3c0c0a3d90 2024-09-16T03:51:59.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.193+0000 7f3c0affd640 10 client.824148.objecter ms_dispatch 0x7f3c0c000ba0 osd_map(331..331 src has 286..331) 2024-09-16T03:51:59.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.193+0000 7f3c0affd640 3 client.824148.objecter handle_osd_map got epochs [331,331] > 0 2024-09-16T03:51:59.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.193+0000 7f3c0affd640 3 client.824148.objecter handle_osd_map decoding full epoch 331 2024-09-16T03:51:59.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.193+0000 7f3c0affd640 20 client.824148.objecter dump_active .. 0 homeless 2024-09-16T03:51:59.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.193+0000 7f3c0affd640 10 client.824148.objecter ms_handle_connect 0x7f3c0c139ef0 2024-09-16T03:51:59.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.193+0000 7f3c0affd640 10 client.824148.objecter resend_mon_ops 2024-09-16T03:51:59.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.193+0000 7f3c1251d640 1 --2- 172.21.15.42:0/1925804287 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3c0c139ef0 0x7f3c0c13a2d0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f3bfc01a460 tx=0x7f3bfc0195d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:51:59.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.193+0000 7f3c12d1e640 1 --2- 172.21.15.42:0/1925804287 >> v2:172.21.15.154:6812/254845342 conn(0x7f3bf8037810 0x7f3bf8039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:59.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.193+0000 7f3c12d1e640 10 monclient: get_auth_request con 0x7f3bf8037810 auth_method 0 2024-09-16T03:51:59.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.193+0000 7f3c0affd640 1 -- 172.21.15.42:0/1925804287 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bfc009450 con 0x7f3c0c139ef0 2024-09-16T03:51:59.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.193+0000 7f3c0affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:51:59.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.193+0000 7f3c0affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:51:59.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.193+0000 7f3c0affd640 10 monclient: dump: 2024-09-16T03:51:59.218 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:51:59.219 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:59.219 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:59.219 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:59.219 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:51:59.219 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:59.219 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:51:59.219 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:51:59.219 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:51:59.219 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:59.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.193+0000 7f3c0affd640 1 -- 172.21.15.42:0/1925804287 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f3bfc0ae830 con 0x7f3c0c139ef0 2024-09-16T03:51:59.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.193+0000 7f3c0affd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:51:59.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.193+0000 7f3c0affd640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:51:59.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.193+0000 7f3c0affd640 1 -- 172.21.15.42:0/1925804287 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3c0c139ef0 msgr2=0x7f3c0c13a2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:59.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.193+0000 7f3c0affd640 1 --2- 172.21.15.42:0/1925804287 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3c0c139ef0 0x7f3c0c13a2d0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f3bfc01a460 tx=0x7f3bfc0195d0 comp rx=0 tx=0).stop 2024-09-16T03:51:59.220 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.193+0000 7f3c0affd640 1 -- 172.21.15.42:0/1925804287 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3c0c139ef0 msgr2=0x7f3c0c13a2d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:51:59.220 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.193+0000 7f3c0affd640 1 --2- 172.21.15.42:0/1925804287 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3c0c139ef0 0x7f3c0c13a2d0 secure :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f3bfc01a460 tx=0x7f3bfc0195d0 comp rx=0 tx=0).stop 2024-09-16T03:51:59.220 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.193+0000 7f3c0affd640 10 client.824148.objecter ms_handle_connect 0x7f3bf8037810 2024-09-16T03:51:59.220 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.193+0000 7f3c12d1e640 1 --2- 172.21.15.42:0/1925804287 >> v2:172.21.15.154:6812/254845342 conn(0x7f3bf8037810 0x7f3bf8039cd0 secure :-1 s=READY pgs=3573 cs=0 l=1 rev1=1 crypto rx=0x7f3c04009630 tx=0x7f3c04008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:51:59.225 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.201+0000 7f3c14781640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:51:59.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.201+0000 7f3c14781640 1 --2- 172.21.15.42:0/1925804287 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3c0c13a810 0x7f3c0c12f500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:59.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.201+0000 7f3c14781640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:59.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.201+0000 7f3c14781640 1 -- 172.21.15.42:0/1925804287 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f3c0c13d860 con 0x7f3c0c13a810 2024-09-16T03:51:59.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.201+0000 7f3c1251d640 1 --2- 172.21.15.42:0/1925804287 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3c0c13a810 0x7f3c0c12f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:59.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.201+0000 7f3c1251d640 10 monclient: get_auth_request con 0x7f3c0c13a810 auth_method 0 2024-09-16T03:51:59.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.201+0000 7f3c1251d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:59.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.201+0000 7f3c1251d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:59.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.201+0000 7f3c1251d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:51:59.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.201+0000 7f3c1251d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:51:59.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.201+0000 7f3c1251d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:51:59.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.201+0000 7f3c1251d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:51:59.227 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.201+0000 7f3c1251d640 10 monclient(hunting): handle_auth_done global_id 824154 payload 824 2024-09-16T03:51:59.227 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.201+0000 7f3c1251d640 1 -- 172.21.15.42:0/1925804287 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f3c0c068e40 con 0x7f3c0c13a810 2024-09-16T03:51:59.227 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.201+0000 7f3c1251d640 1 --2- 172.21.15.42:0/1925804287 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3c0c13a810 0x7f3c0c12f500 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f3bfc009bd0 tx=0x7f3bfc0a9000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:51:59.227 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.201+0000 7f3c0affd640 10 client.824148.objecter ms_handle_connect 0x7f3c0c13a810 2024-09-16T03:51:59.227 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.201+0000 7f3c0affd640 10 client.824148.objecter resend_mon_ops 2024-09-16T03:51:59.227 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.201+0000 7f3c0affd640 1 -- 172.21.15.42:0/1925804287 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bfc003e50 con 0x7f3c0c13a810 2024-09-16T03:51:59.227 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.201+0000 7f3c0affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:51:59.227 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.201+0000 7f3c0affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:51:59.227 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.201+0000 7f3c0affd640 10 monclient: dump: 2024-09-16T03:51:59.227 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:51:59.227 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:59.227 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:59.227 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:59.228 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:51:59.228 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:59.228 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:51:59.228 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:51:59.228 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:51:59.228 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:59.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.201+0000 7f3c0affd640 1 -- 172.21.15.42:0/1925804287 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4545 (secure 0 0 0) 0x7f3bfc004000 con 0x7f3c0c13a810 2024-09-16T03:51:59.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.201+0000 7f3c0affd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:51:59.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.201+0000 7f3c0affd640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:51:59.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.201+0000 7f3c0affd640 1 -- 172.21.15.42:0/1925804287 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3c0c13a810 msgr2=0x7f3c0c12f500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:59.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.201+0000 7f3c0affd640 1 --2- 172.21.15.42:0/1925804287 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3c0c13a810 0x7f3c0c12f500 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f3bfc009bd0 tx=0x7f3bfc0a9000 comp rx=0 tx=0).stop 2024-09-16T03:51:59.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.201+0000 7f3c0affd640 1 -- 172.21.15.42:0/1925804287 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3c0c13a810 msgr2=0x7f3c0c12f500 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:51:59.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.201+0000 7f3c0affd640 1 --2- 172.21.15.42:0/1925804287 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3c0c13a810 0x7f3c0c12f500 secure :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f3bfc009bd0 tx=0x7f3bfc0a9000 comp rx=0 tx=0).stop 2024-09-16T03:51:59.229 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:51:59.229 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:51:59.229 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:51:59.229 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "leader", 2024-09-16T03:51:59.229 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 86, 2024-09-16T03:51:59.229 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 9055, 2024-09-16T03:51:59.229 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:51:59.230 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:51:59.230 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T03:51:59.230 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T03:51:59.230 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:51:59.230 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 2, 2024-09-16T03:51:59.230 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:51:59.230 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:51:59.230 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:51:59.230 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:51:59.230 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:51:59.230 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:51:59.230 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:51:59.231 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:51:59.231 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:51:59.231 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:51:59.231 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:51:59.231 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:51:59.231 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:51:59.231 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:51:59.231 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:51:59.231 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:51:59.231 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:51:59.231 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:51:59.231 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:51:59.232 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:51:59.232 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:51:59.232 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:51:59.232 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:51:59.232 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:51:59.232 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:51:59.232 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:51:59.232 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:51:59.232 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:51:59.232 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:51:59.232 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:59.232 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:51:59.232 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:51:59.233 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:51:59.233 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:51:59.233 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:51:59.233 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:51:59.233 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:51:59.233 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:51:59.233 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:51:59.233 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:51:59.233 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:51:59.233 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:51:59.233 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:51:59.233 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:51:59.234 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:51:59.234 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:51:59.234 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:51:59.234 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:51:59.234 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:51:59.234 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:51:59.234 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:51:59.234 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:51:59.234 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:51:59.234 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:51:59.234 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:51:59.234 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:51:59.234 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:51:59.235 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:51:59.235 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:51:59.235 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:51:59.235 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:59.235 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:51:59.235 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:59.235 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:51:59.235 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:51:59.235 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:51:59.235 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:51:59.235 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:59.235 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:51:59.235 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:51:59.235 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:51:59.236 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:59.236 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:59.236 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:51:59.236 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:51:59.236 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:51:59.236 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:51:59.236 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:51:59.236 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:59.236 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:51:59.236 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:51:59.236 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:51:59.237 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:51:59.237 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:51:59.237 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:59.237 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:59.237 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:51:59.237 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:51:59.237 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:51:59.237 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:51:59.237 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:59.237 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:51:59.237 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:51:59.237 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:51:59.237 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:59.237 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:59.237 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:51:59.238 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:51:59.238 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:51:59.238 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:51:59.238 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:51:59.238 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:59.238 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:51:59.238 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:51:59.238 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:51:59.238 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:51:59.238 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:51:59.238 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:59.238 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:59.238 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:51:59.238 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:51:59.239 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:51:59.239 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:51:59.239 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:59.239 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:51:59.239 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:51:59.239 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:51:59.239 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:59.239 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:59.239 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:51:59.239 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:51:59.239 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:51:59.239 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:51:59.239 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:51:59.239 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:59.240 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:51:59.240 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:51:59.240 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:51:59.240 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:51:59.240 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:51:59.240 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:51:59.240 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:51:59.240 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:59.240 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:51:59.240 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:51:59.240 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:59.240 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:51:59.240 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:51:59.241 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:51:59.241 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:51:59.241 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:51:59.241 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:51:59.241 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:59.241 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:51:59.241 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:51:59.241 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 2 2024-09-16T03:51:59.241 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:51:59.241 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:51:59.241 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:59.241 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:51:59.241 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:51:59.243 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.201+0000 7f3c14781640 10 librados: watch_flush enter 2024-09-16T03:51:59.243 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.201+0000 7f3c14781640 10 librados: watch_flush exit 2024-09-16T03:51:59.243 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.201+0000 7f3c14781640 20 client.824148.objecter shutdown clearing up homeless session... 2024-09-16T03:51:59.243 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.201+0000 7f3c14781640 10 client.824148.objecter successfully canceled tick 2024-09-16T03:51:59.243 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.201+0000 7f3c14781640 1 -- 172.21.15.42:0/1925804287 >> v2:172.21.15.154:6812/254845342 conn(0x7f3bf8037810 msgr2=0x7f3bf8039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:59.243 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.201+0000 7f3c14781640 1 --2- 172.21.15.42:0/1925804287 >> v2:172.21.15.154:6812/254845342 conn(0x7f3bf8037810 0x7f3bf8039cd0 secure :-1 s=READY pgs=3573 cs=0 l=1 rev1=1 crypto rx=0x7f3c04009630 tx=0x7f3c04008040 comp rx=0 tx=0).stop 2024-09-16T03:51:59.243 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.201+0000 7f3c14781640 10 monclient: shutdown 2024-09-16T03:51:59.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.201+0000 7f3c14781640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:51:59.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.201+0000 7f3c14781640 1 -- 172.21.15.42:0/1925804287 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3c0c0a3d90 msgr2=0x7f3c0c12d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:59.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.201+0000 7f3c14781640 1 --2- 172.21.15.42:0/1925804287 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3c0c0a3d90 0x7f3c0c12d810 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f3c0000c3f0 tx=0x7f3c0000c8b0 comp rx=0 tx=0).stop 2024-09-16T03:51:59.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.201+0000 7f3c12d1e640 1 -- 172.21.15.42:0/1925804287 reap_dead start 2024-09-16T03:51:59.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.201+0000 7f3c12d1e640 1 -- 172.21.15.42:0/1925804287 reap_dead start 2024-09-16T03:51:59.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.201+0000 7f3c14781640 1 -- 172.21.15.42:0/1925804287 shutdown_connections 2024-09-16T03:51:59.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.201+0000 7f3c14781640 1 -- 172.21.15.42:0/1925804287 >> 172.21.15.42:0/1925804287 conn(0x7f3c0c0a9fb0 msgr2=0x7f3c0c0a8ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:51:59.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.201+0000 7f3c14781640 1 -- 172.21.15.42:0/1925804287 shutdown_connections 2024-09-16T03:51:59.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.201+0000 7f3c14781640 1 -- 172.21.15.42:0/1925804287 wait complete. 2024-09-16T03:51:59.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.201+0000 7f3c14781640 1 librados: shutdown 2024-09-16T03:51:59.245 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:51:59.245 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T03:51:59.245 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T03:51:59.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.357+0000 7f7c1381b640 10 monclient: get_monmap_and_config 2024-09-16T03:51:59.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.357+0000 7f7c1381b640 10 monclient: build_initial_monmap 2024-09-16T03:51:59.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.357+0000 7f7c1381b640 10 monclient: monmap: 2024-09-16T03:51:59.362 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:51:59.362 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:59.362 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:51:59.360682+0000 2024-09-16T03:51:59.362 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:51:59.360682+0000 2024-09-16T03:51:59.362 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:51:59.363 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:59.363 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:51:59.363 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:51:59.363 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:51:59.363 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:51:59.363 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:51:59.363 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:59.367 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.357+0000 7f7c1381b640 1 Processor -- start 2024-09-16T03:51:59.367 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.357+0000 7f7c1381b640 1 -- start start 2024-09-16T03:51:59.368 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.357+0000 7f7c1381b640 10 monclient: init 2024-09-16T03:51:59.368 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.357+0000 7f7c1381b640 10 monclient: _reopen_session rank -1 2024-09-16T03:51:59.368 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.357+0000 7f7c1381b640 10 monclient: _add_conns ranks=[1,0,3,4,2] 2024-09-16T03:51:59.368 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.357+0000 7f7c1381b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7c0c0a1330 addr v1:172.21.15.154:6789/0 2024-09-16T03:51:59.368 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.357+0000 7f7c1381b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7c0c0a1710 addr v1:172.21.15.42:6789/0 2024-09-16T03:51:59.368 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.357+0000 7f7c1381b640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f7c0c13a3b0 0x7f7c0c13c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:59.368 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.357+0000 7f7c1381b640 10 monclient(hunting): picked mon.noname-a con 0x7f7c0c13a3b0 addr v2:172.21.15.42:3300/0 2024-09-16T03:51:59.368 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.357+0000 7f7c1381b640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7c0c09e4c0 con 0x7f7c0c0a1710 2024-09-16T03:51:59.368 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.357+0000 7f7c1381b640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7c0c0a3170 con 0x7f7c0c0a1330 2024-09-16T03:51:59.369 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.357+0000 7f7c1381b640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:59.369 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.357+0000 7f7c1381b640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f7c0c09edf0 con 0x7f7c0c13a3b0 2024-09-16T03:51:59.369 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.357+0000 7f7c1381b640 10 monclient(hunting): _renew_subs 2024-09-16T03:51:59.369 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.357+0000 7f7c1381b640 10 monclient(hunting): authenticate will time out at 1566.330431s 2024-09-16T03:51:59.369 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.357+0000 7f7c11db8640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f7c0c13a3b0 0x7f7c0c13c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:59.369 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.357+0000 7f7c10db6640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f7c0c0a1710 0x7f7c0c0a1b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:57032/0 (socket says 172.21.15.42:57032) 2024-09-16T03:51:59.369 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.357+0000 7f7c11db8640 10 monclient(hunting): get_auth_request con 0x7f7c0c13a3b0 auth_method 0 2024-09-16T03:51:59.369 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.357+0000 7f7c10db6640 1 -- 172.21.15.42:0/19801814 learned_addr learned my addr 172.21.15.42:0/19801814 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:51:59.369 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.357+0000 7f7c11db8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:59.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.357+0000 7f7c11db8640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:59.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.357+0000 7f7c11db8640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:51:59.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.357+0000 7f7c11db8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:51:59.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.357+0000 7f7c11db8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:51:59.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.357+0000 7f7c11db8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:51:59.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.357+0000 7f7c11db8640 10 monclient(hunting): handle_auth_done global_id 824157 payload 274 2024-09-16T03:51:59.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.357+0000 7f7c11db8640 1 -- 172.21.15.42:0/19801814 >> v1:172.21.15.154:6789/0 conn(0x7f7c0c0a1330 legacy=0x7f7c0c09f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:59.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.357+0000 7f7c11db8640 1 -- 172.21.15.42:0/19801814 >> v1:172.21.15.42:6789/0 conn(0x7f7c0c0a1710 legacy=0x7f7c0c0a1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:59.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.357+0000 7f7c11db8640 10 monclient: _finish_hunting 0 2024-09-16T03:51:59.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.357+0000 7f7c11db8640 1 monclient: found mon.noname-a 2024-09-16T03:51:59.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.357+0000 7f7bfffff640 1 -- 172.21.15.42:0/19801814 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3834099357 0 0) 0x7f7c0c09e4c0 con 0x7f7c0c0a1710 2024-09-16T03:51:59.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.357+0000 7f7c11db8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:51:59.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.357+0000 7f7c11db8640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:51:59.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.357+0000 7f7c11db8640 1 -- 172.21.15.42:0/19801814 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c0c09d030 con 0x7f7c0c13a3b0 2024-09-16T03:51:59.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.357+0000 7f7bfffff640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T03:51:59.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.361+0000 7f7c11db8640 1 --2- 172.21.15.42:0/19801814 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7c0c13a3b0 0x7f7c0c13c7a0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f7c04009d90 tx=0x7f7c0401c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3bbcadacd1a1dfb server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:51:59.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.361+0000 7f7bfffff640 1 -- 172.21.15.42:0/19801814 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c040b2070 con 0x7f7c0c13a3b0 2024-09-16T03:51:59.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.361+0000 7f7bfffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:51:59.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.361+0000 7f7bfffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:51:59.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.361+0000 7f7bfffff640 10 monclient: dump: 2024-09-16T03:51:59.372 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:51:59.372 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:59.372 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:59.372 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:59.372 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:51:59.372 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:59.372 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:51:59.372 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:51:59.372 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:51:59.372 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:59.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.361+0000 7f7bfffff640 10 monclient: _finish_auth 0 2024-09-16T03:51:59.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.361+0000 7f7bfffff640 10 monclient: _check_auth_tickets 2024-09-16T03:51:59.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.361+0000 7f7bfffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:51:59.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.361+0000 7f7bfffff640 1 -- 172.21.15.42:0/19801814 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7c040a5800 con 0x7f7c0c13a3b0 2024-09-16T03:51:59.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.361+0000 7f7bfffff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:51:59.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.361+0000 7f7bfffff640 1 -- 172.21.15.42:0/19801814 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c040a5bd0 con 0x7f7c0c13a3b0 2024-09-16T03:51:59.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.361+0000 7f7c1381b640 5 monclient: authenticate success, global_id 824157 2024-09-16T03:51:59.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.361+0000 7f7c1381b640 10 monclient: get_monmap_and_config success 2024-09-16T03:51:59.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.361+0000 7f7c1381b640 4 set_mon_vals no callback set 2024-09-16T03:51:59.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.361+0000 7f7bfffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:51:59.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.361+0000 7f7bfffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:51:59.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.361+0000 7f7bfffff640 10 monclient: dump: 2024-09-16T03:51:59.374 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:51:59.374 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:59.374 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:59.374 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:59.374 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:51:59.374 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:59.374 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:51:59.375 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:51:59.375 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:51:59.375 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:59.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.361+0000 7f7c1381b640 10 monclient: shutdown 2024-09-16T03:51:59.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.361+0000 7f7c1381b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:51:59.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.361+0000 7f7c1381b640 1 -- 172.21.15.42:0/19801814 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7c0c13a3b0 msgr2=0x7f7c0c13c7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:59.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.361+0000 7f7c1381b640 1 --2- 172.21.15.42:0/19801814 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7c0c13a3b0 0x7f7c0c13c7a0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f7c04009d90 tx=0x7f7c0401c830 comp rx=0 tx=0).stop 2024-09-16T03:51:59.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.361+0000 7f7c1381b640 1 -- 172.21.15.42:0/19801814 shutdown_connections 2024-09-16T03:51:59.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.361+0000 7f7c1381b640 1 --2- 172.21.15.42:0/19801814 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7c0c13a3b0 0x7f7c0c13c7a0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:59.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.361+0000 7f7c1381b640 1 -- 172.21.15.42:0/19801814 >> 172.21.15.42:0/19801814 conn(0x7f7c0c0a2420 msgr2=0x7f7c0c0a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:51:59.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.361+0000 7f7c1381b640 1 -- 172.21.15.42:0/19801814 shutdown_connections 2024-09-16T03:51:59.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.361+0000 7f7c1381b640 1 -- 172.21.15.42:0/19801814 wait complete. 2024-09-16T03:51:59.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.361+0000 7f7c1381b640 10 monclient: build_initial_monmap 2024-09-16T03:51:59.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.361+0000 7f7c1381b640 10 monclient: monmap: 2024-09-16T03:51:59.376 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:51:59.376 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:51:59.376 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:51:59.376 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:51:59.376 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:51:59.377 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:59.377 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:51:59.377 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:51:59.377 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:51:59.377 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:59.388 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.361+0000 7f7c1381b640 1 librados: starting msgr at 2024-09-16T03:51:59.388 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.361+0000 7f7c1381b640 1 librados: starting objecter 2024-09-16T03:51:59.388 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.361+0000 7f7c1381b640 1 Processor -- start 2024-09-16T03:51:59.388 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.361+0000 7f7c1381b640 1 -- start start 2024-09-16T03:51:59.388 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.361+0000 7f7c1381b640 1 librados: setting wanted keys 2024-09-16T03:51:59.389 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.361+0000 7f7c1381b640 1 librados: calling monclient init 2024-09-16T03:51:59.389 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.361+0000 7f7c1381b640 10 monclient: init 2024-09-16T03:51:59.389 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.361+0000 7f7c1381b640 10 monclient: _reopen_session rank -1 2024-09-16T03:51:59.389 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.361+0000 7f7c1381b640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:51:59.389 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.361+0000 7f7c1381b640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7c0c0a1330 0x7f7c0c071d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:59.389 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.361+0000 7f7c1381b640 10 monclient(hunting): picked mon.noname-c con 0x7f7c0c0a1330 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:51:59.389 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.361+0000 7f7c1381b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7c0c0a1710 0x7f7c0c132290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:59.389 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.361+0000 7f7c1381b640 10 monclient(hunting): picked mon.noname-b con 0x7f7c0c0a1710 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:51:59.389 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.361+0000 7f7c1381b640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7c0c13a3b0 0x7f7c0c1327d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:59.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.361+0000 7f7c1381b640 10 monclient(hunting): picked mon.noname-a con 0x7f7c0c13a3b0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:51:59.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.361+0000 7f7c1381b640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:59.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.361+0000 7f7c1381b640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f7c0c09bea0 con 0x7f7c0c13a3b0 2024-09-16T03:51:59.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.361+0000 7f7c1381b640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:59.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.361+0000 7f7c1381b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f7c0c09bd20 con 0x7f7c0c0a1710 2024-09-16T03:51:59.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.361+0000 7f7c1381b640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:59.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.361+0000 7f7c1381b640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f7c0c09c020 con 0x7f7c0c0a1330 2024-09-16T03:51:59.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.361+0000 7f7c1381b640 10 monclient(hunting): _renew_subs 2024-09-16T03:51:59.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.361+0000 7f7c1381b640 10 monclient(hunting): authenticate will time out at 1566.335802s 2024-09-16T03:51:59.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.361+0000 7f7c115b7640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7c0c0a1330 0x7f7c0c071d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:59.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.361+0000 7f7c11db8640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7c0c13a3b0 0x7f7c0c1327d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:59.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.365+0000 7f7c115b7640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7c0c0a1330 0x7f7c0c071d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:41552/0 (socket says 172.21.15.42:41552) 2024-09-16T03:51:59.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.365+0000 7f7c115b7640 1 -- 172.21.15.42:0/3154742846 learned_addr learned my addr 172.21.15.42:0/3154742846 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:51:59.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.365+0000 7f7c11db8640 10 monclient(hunting): get_auth_request con 0x7f7c0c13a3b0 auth_method 0 2024-09-16T03:51:59.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.365+0000 7f7c11db8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:59.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.365+0000 7f7c11db8640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:59.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.365+0000 7f7c11db8640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:51:59.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.365+0000 7f7c115b7640 10 monclient(hunting): get_auth_request con 0x7f7c0c0a1330 auth_method 0 2024-09-16T03:51:59.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.365+0000 7f7c115b7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:59.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.365+0000 7f7c115b7640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:59.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.365+0000 7f7c115b7640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:51:59.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.365+0000 7f7c10db6640 1 --2- 172.21.15.42:0/3154742846 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7c0c0a1710 0x7f7c0c132290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:59.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.365+0000 7f7c10db6640 10 monclient(hunting): get_auth_request con 0x7f7c0c0a1710 auth_method 0 2024-09-16T03:51:59.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.365+0000 7f7c10db6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:59.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.365+0000 7f7c10db6640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:59.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.365+0000 7f7c10db6640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:51:59.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.365+0000 7f7c11db8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:51:59.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.365+0000 7f7c11db8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:51:59.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.365+0000 7f7c11db8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:51:59.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.365+0000 7f7c115b7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:51:59.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.365+0000 7f7c115b7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:51:59.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.365+0000 7f7c115b7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:51:59.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.365+0000 7f7c10db6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:51:59.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.365+0000 7f7c10db6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:51:59.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.365+0000 7f7c10db6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:51:59.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.365+0000 7f7c11db8640 10 monclient(hunting): handle_auth_done global_id 824163 payload 824 2024-09-16T03:51:59.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.365+0000 7f7c11db8640 1 -- 172.21.15.42:0/3154742846 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7c0c0a1330 msgr2=0x7f7c0c071d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:59.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.365+0000 7f7c11db8640 1 --2- 172.21.15.42:0/3154742846 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7c0c0a1330 0x7f7c0c071d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:59.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.365+0000 7f7c11db8640 1 -- 172.21.15.42:0/3154742846 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7c0c0a1710 msgr2=0x7f7c0c132290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:59.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.365+0000 7f7c11db8640 1 --2- 172.21.15.42:0/3154742846 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7c0c0a1710 0x7f7c0c132290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:59.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.365+0000 7f7c11db8640 10 monclient: _finish_hunting 0 2024-09-16T03:51:59.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.365+0000 7f7c11db8640 1 monclient: found mon.noname-a 2024-09-16T03:51:59.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.365+0000 7f7c11db8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:51:59.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.365+0000 7f7c11db8640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:51:59.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.365+0000 7f7c11db8640 1 -- 172.21.15.42:0/3154742846 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c0c11c0f0 con 0x7f7c0c13a3b0 2024-09-16T03:51:59.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.365+0000 7f7c115b7640 1 --2- 172.21.15.42:0/3154742846 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7c0c0a1330 0x7f7c0c071d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:51:59.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.365+0000 7f7c11db8640 1 --2- 172.21.15.42:0/3154742846 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7c0c13a3b0 0x7f7c0c1327d0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f7c040ac000 tx=0x7f7c040195d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:51:59.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.365+0000 7f7bfdffb640 10 client.?.objecter ms_handle_connect 0x7f7c0c13a3b0 2024-09-16T03:51:59.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.365+0000 7f7bfdffb640 10 client.?.objecter resend_mon_ops 2024-09-16T03:51:59.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.365+0000 7f7bfdffb640 1 -- 172.21.15.42:0/3154742846 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c040bd020 con 0x7f7c0c13a3b0 2024-09-16T03:51:59.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.365+0000 7f7bfdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:51:59.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.365+0000 7f7bfdffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:51:59.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.365+0000 7f7bfdffb640 10 monclient: dump: 2024-09-16T03:51:59.395 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:51:59.395 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:59.396 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:59.396 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:59.396 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:51:59.396 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:59.396 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:51:59.396 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:51:59.396 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:51:59.396 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:59.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.365+0000 7f7bfdffb640 10 monclient: _finish_auth 0 2024-09-16T03:51:59.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.365+0000 7f7bfdffb640 10 monclient: _check_auth_tickets 2024-09-16T03:51:59.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.365+0000 7f7bfdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:51:59.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.365+0000 7f7bfdffb640 1 -- 172.21.15.42:0/3154742846 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7c040b3040 con 0x7f7c0c13a3b0 2024-09-16T03:51:59.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.365+0000 7f7c1381b640 5 monclient: authenticate success, global_id 824163 2024-09-16T03:51:59.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.365+0000 7f7c1381b640 10 monclient: _renew_subs 2024-09-16T03:51:59.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.365+0000 7f7c1381b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:51:59.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.365+0000 7f7c1381b640 1 -- 172.21.15.42:0/3154742846 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c0c133c10 con 0x7f7c0c13a3b0 2024-09-16T03:51:59.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.365+0000 7f7bfdffb640 10 monclient: handle_config config(0 keys) 2024-09-16T03:51:59.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.365+0000 7f7bfdffb640 1 -- 172.21.15.42:0/3154742846 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c040b2050 con 0x7f7c0c13a3b0 2024-09-16T03:51:59.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.365+0000 7f7bfdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:51:59.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.365+0000 7f7bfdffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:51:59.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.365+0000 7f7bfdffb640 10 monclient: dump: 2024-09-16T03:51:59.398 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:51:59.398 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:59.398 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:59.398 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:59.398 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:51:59.398 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:59.398 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:51:59.399 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:51:59.399 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:51:59.399 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:59.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.365+0000 7f7bfeffd640 4 set_mon_vals no callback set 2024-09-16T03:51:59.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.365+0000 7f7c1381b640 10 client.824163.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:51:59.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.365+0000 7f7c1381b640 10 monclient: _renew_subs 2024-09-16T03:51:59.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.365+0000 7f7c1381b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:51:59.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.365+0000 7f7c1381b640 1 -- 172.21.15.42:0/3154742846 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c0c0a4950 con 0x7f7c0c13a3b0 2024-09-16T03:51:59.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.365+0000 7f7c1381b640 1 librados: init done 2024-09-16T03:51:59.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.365+0000 7f7c1381b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:51:59.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.365+0000 7f7c1381b640 1 --2- 172.21.15.42:0/3154742846 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7c0c132d10 0x7f7c0c137390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:59.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.365+0000 7f7c1381b640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:59.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.365+0000 7f7c1381b640 1 -- 172.21.15.42:0/3154742846 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f7c0c1340b0 con 0x7f7c0c132d10 2024-09-16T03:51:59.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.365+0000 7f7c115b7640 1 --2- 172.21.15.42:0/3154742846 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7c0c132d10 0x7f7c0c137390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:59.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.365+0000 7f7c115b7640 10 monclient: get_auth_request con 0x7f7c0c132d10 auth_method 0 2024-09-16T03:51:59.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.365+0000 7f7c115b7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:59.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.365+0000 7f7c115b7640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:59.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.365+0000 7f7c115b7640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:51:59.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.365+0000 7f7bfdffb640 1 -- 172.21.15.42:0/3154742846 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f7c040c0070 con 0x7f7c0c13a3b0 2024-09-16T03:51:59.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.365+0000 7f7c115b7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:51:59.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.365+0000 7f7c115b7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:51:59.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.365+0000 7f7c115b7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:51:59.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.365+0000 7f7bfdffb640 1 --2- 172.21.15.42:0/3154742846 >> v2:172.21.15.154:6812/254845342 conn(0x7f7bf0037810 0x7f7bf0039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:59.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.365+0000 7f7c115b7640 10 monclient(hunting): handle_auth_done global_id 814229 payload 824 2024-09-16T03:51:59.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.365+0000 7f7bfdffb640 1 -- 172.21.15.42:0/3154742846 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(331..331 src has 286..331) ==== 5713+0+0 (secure 0 0 0) 0x7f7c040e3460 con 0x7f7c0c13a3b0 2024-09-16T03:51:59.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.365+0000 7f7bfdffb640 10 client.824163.objecter ms_dispatch 0x7f7c0c000ba0 osd_map(331..331 src has 286..331) 2024-09-16T03:51:59.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.365+0000 7f7bfdffb640 3 client.824163.objecter handle_osd_map got epochs [331,331] > 0 2024-09-16T03:51:59.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.365+0000 7f7bfdffb640 3 client.824163.objecter handle_osd_map decoding full epoch 331 2024-09-16T03:51:59.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.365+0000 7f7c115b7640 1 -- 172.21.15.42:0/3154742846 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7c0c133e90 con 0x7f7c0c132d10 2024-09-16T03:51:59.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.365+0000 7f7bfdffb640 20 client.824163.objecter dump_active .. 0 homeless 2024-09-16T03:51:59.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.365+0000 7f7bfdffb640 10 client.824163.objecter ms_handle_connect 0x7f7c0c132d10 2024-09-16T03:51:59.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.365+0000 7f7c115b7640 1 --2- 172.21.15.42:0/3154742846 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7c0c132d10 0x7f7c0c137390 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f7c0000fb10 tx=0x7f7c00001e20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:51:59.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.365+0000 7f7bfdffb640 10 client.824163.objecter resend_mon_ops 2024-09-16T03:51:59.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.365+0000 7f7bfdffb640 1 -- 172.21.15.42:0/3154742846 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c000052e0 con 0x7f7c0c132d10 2024-09-16T03:51:59.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.365+0000 7f7bfdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:51:59.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.365+0000 7f7bfdffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:51:59.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.365+0000 7f7bfdffb640 10 monclient: dump: 2024-09-16T03:51:59.402 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:51:59.402 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:59.402 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:59.402 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:59.402 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:51:59.402 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:59.402 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:51:59.402 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:51:59.402 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:51:59.402 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:59.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.365+0000 7f7c10db6640 1 --2- 172.21.15.42:0/3154742846 >> v2:172.21.15.154:6812/254845342 conn(0x7f7bf0037810 0x7f7bf0039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:59.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.365+0000 7f7c10db6640 10 monclient: get_auth_request con 0x7f7bf0037810 auth_method 0 2024-09-16T03:51:59.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.365+0000 7f7bfdffb640 1 -- 172.21.15.42:0/3154742846 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f7c000052e0 con 0x7f7c0c132d10 2024-09-16T03:51:59.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.365+0000 7f7bfdffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:51:59.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.365+0000 7f7bfdffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:51:59.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.365+0000 7f7bfdffb640 1 -- 172.21.15.42:0/3154742846 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7c0c132d10 msgr2=0x7f7c0c137390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:59.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.365+0000 7f7bfdffb640 1 --2- 172.21.15.42:0/3154742846 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7c0c132d10 0x7f7c0c137390 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f7c0000fb10 tx=0x7f7c00001e20 comp rx=0 tx=0).stop 2024-09-16T03:51:59.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.365+0000 7f7bfdffb640 1 -- 172.21.15.42:0/3154742846 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7c0c132d10 msgr2=0x7f7c0c137390 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:51:59.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.365+0000 7f7bfdffb640 1 --2- 172.21.15.42:0/3154742846 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7c0c132d10 0x7f7c0c137390 secure :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f7c0000fb10 tx=0x7f7c00001e20 comp rx=0 tx=0).stop 2024-09-16T03:51:59.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.365+0000 7f7c10db6640 1 --2- 172.21.15.42:0/3154742846 >> v2:172.21.15.154:6812/254845342 conn(0x7f7bf0037810 0x7f7bf0039cd0 secure :-1 s=READY pgs=3574 cs=0 l=1 rev1=1 crypto rx=0x7f7c08007a90 tx=0x7f7c080079e0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:51:59.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.365+0000 7f7bfdffb640 10 client.824163.objecter ms_handle_connect 0x7f7bf0037810 2024-09-16T03:51:59.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.373+0000 7f7c1381b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:51:59.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.373+0000 7f7c1381b640 1 --2- 172.21.15.42:0/3154742846 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7c0c1378d0 0x7f7c0c137cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:59.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.373+0000 7f7c1381b640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:59.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.373+0000 7f7c1381b640 1 -- 172.21.15.42:0/3154742846 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f7c0c1330f0 con 0x7f7c0c1378d0 2024-09-16T03:51:59.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.373+0000 7f7c115b7640 1 --2- 172.21.15.42:0/3154742846 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7c0c1378d0 0x7f7c0c137cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:59.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.373+0000 7f7c115b7640 10 monclient: get_auth_request con 0x7f7c0c1378d0 auth_method 0 2024-09-16T03:51:59.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.373+0000 7f7c115b7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:59.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.373+0000 7f7c115b7640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:59.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.373+0000 7f7c115b7640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:51:59.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.373+0000 7f7c115b7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:51:59.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.373+0000 7f7c115b7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:51:59.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.373+0000 7f7c115b7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:51:59.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.373+0000 7f7c115b7640 10 monclient(hunting): handle_auth_done global_id 814232 payload 824 2024-09-16T03:51:59.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.373+0000 7f7c115b7640 1 -- 172.21.15.42:0/3154742846 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f7c0c09f180 con 0x7f7c0c1378d0 2024-09-16T03:51:59.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.373+0000 7f7c115b7640 1 --2- 172.21.15.42:0/3154742846 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7c0c1378d0 0x7f7c0c137cb0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f7c0000fb10 tx=0x7f7c00001130 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:51:59.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.373+0000 7f7bfdffb640 10 client.824163.objecter ms_handle_connect 0x7f7c0c1378d0 2024-09-16T03:51:59.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.373+0000 7f7bfdffb640 10 client.824163.objecter resend_mon_ops 2024-09-16T03:51:59.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.373+0000 7f7bfdffb640 1 -- 172.21.15.42:0/3154742846 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c00007670 con 0x7f7c0c1378d0 2024-09-16T03:51:59.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.373+0000 7f7bfdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:51:59.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.373+0000 7f7bfdffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:51:59.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.373+0000 7f7bfdffb640 10 monclient: dump: 2024-09-16T03:51:59.405 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:51:59.405 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:59.405 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:59.405 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:59.405 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:51:59.405 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:59.405 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:51:59.405 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:51:59.406 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:51:59.406 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:59.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.373+0000 7f7bfdffb640 1 -- 172.21.15.42:0/3154742846 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4878 (secure 0 0 0) 0x7f7c00007670 con 0x7f7c0c1378d0 2024-09-16T03:51:59.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.373+0000 7f7bfdffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:51:59.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.373+0000 7f7bfdffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:51:59.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.373+0000 7f7bfdffb640 1 -- 172.21.15.42:0/3154742846 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7c0c1378d0 msgr2=0x7f7c0c137cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:59.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.373+0000 7f7bfdffb640 1 --2- 172.21.15.42:0/3154742846 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7c0c1378d0 0x7f7c0c137cb0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f7c0000fb10 tx=0x7f7c00001130 comp rx=0 tx=0).stop 2024-09-16T03:51:59.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.373+0000 7f7bfdffb640 1 -- 172.21.15.42:0/3154742846 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7c0c1378d0 msgr2=0x7f7c0c137cb0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:51:59.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.373+0000 7f7bfdffb640 1 --2- 172.21.15.42:0/3154742846 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7c0c1378d0 0x7f7c0c137cb0 secure :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f7c0000fb10 tx=0x7f7c00001130 comp rx=0 tx=0).stop 2024-09-16T03:51:59.407 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:51:59.407 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:51:59.407 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:51:59.407 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T03:51:59.407 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 86, 2024-09-16T03:51:59.407 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 64447, 2024-09-16T03:51:59.407 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:51:59.407 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:51:59.407 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T03:51:59.408 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T03:51:59.408 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:51:59.408 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 2, 2024-09-16T03:51:59.408 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:51:59.408 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:51:59.408 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:51:59.408 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:51:59.408 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:51:59.408 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:51:59.408 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:51:59.408 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:51:59.408 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:51:59.408 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:51:59.409 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:51:59.409 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:51:59.409 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:51:59.409 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:51:59.409 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:51:59.409 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:51:59.409 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:51:59.409 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:51:59.409 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:51:59.409 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:51:59.409 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:51:59.410 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:51:59.410 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:51:59.410 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:51:59.410 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:51:59.410 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:51:59.410 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:51:59.410 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:51:59.410 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:51:59.410 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:59.410 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:51:59.410 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:51:59.410 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:51:59.410 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:51:59.411 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:51:59.411 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:51:59.411 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:51:59.411 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:51:59.411 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:51:59.411 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:51:59.411 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:51:59.411 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:51:59.411 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:51:59.411 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:51:59.411 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:51:59.411 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:51:59.412 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:51:59.412 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:51:59.412 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:51:59.412 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:51:59.412 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:51:59.412 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:51:59.412 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:51:59.412 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:51:59.412 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:51:59.412 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:51:59.412 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:51:59.412 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:51:59.413 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:51:59.413 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:51:59.413 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:59.413 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:51:59.413 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:59.413 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:51:59.413 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:51:59.413 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:51:59.413 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:51:59.413 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:59.413 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:51:59.413 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:51:59.413 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:51:59.414 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:59.414 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:59.414 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:51:59.414 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:51:59.414 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:51:59.414 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:51:59.414 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:51:59.414 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:59.414 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:51:59.414 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:51:59.414 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:51:59.414 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:51:59.414 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:51:59.415 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:59.415 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:59.415 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:51:59.415 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:51:59.415 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:51:59.415 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:51:59.415 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:59.415 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:51:59.415 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:51:59.415 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:51:59.415 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:59.415 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:59.415 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:51:59.416 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:51:59.416 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:51:59.416 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:51:59.416 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:51:59.416 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:59.416 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:51:59.416 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:51:59.416 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:51:59.416 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:51:59.416 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:51:59.416 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:59.417 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:59.417 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:51:59.417 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:51:59.417 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:51:59.417 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:51:59.417 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:59.417 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:51:59.417 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:51:59.417 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:51:59.417 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:59.418 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:59.418 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:51:59.418 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:51:59.418 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:51:59.418 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:51:59.418 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:51:59.418 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:59.418 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:51:59.418 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:51:59.418 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:51:59.418 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:51:59.419 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:51:59.419 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:51:59.419 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:51:59.419 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:59.419 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:51:59.419 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:51:59.419 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:59.419 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:51:59.419 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:51:59.419 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:51:59.419 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:51:59.419 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:51:59.420 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T03:51:59.420 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:59.420 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:51:59.420 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:51:59.420 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 5 2024-09-16T03:51:59.420 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:51:59.420 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:51:59.420 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:51:59.420 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:59.420 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:51:59.420 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:51:59.420 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 6 2024-09-16T03:51:59.420 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:51:59.421 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:51:59.421 INFO:teuthology.orchestra.run.smithi042.stdout: "mgr": [ 2024-09-16T03:51:59.421 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:59.421 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:51:59.421 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:51:59.421 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:51:59.421 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:51:59.421 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:51:59.421 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:59.421 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:51:59.421 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:51:59.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.373+0000 7f7c1381b640 10 librados: watch_flush enter 2024-09-16T03:51:59.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.373+0000 7f7c1381b640 10 librados: watch_flush exit 2024-09-16T03:51:59.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.373+0000 7f7c1381b640 20 client.824163.objecter shutdown clearing up homeless session... 2024-09-16T03:51:59.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.373+0000 7f7c1381b640 10 client.824163.objecter successfully canceled tick 2024-09-16T03:51:59.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.373+0000 7f7c1381b640 1 -- 172.21.15.42:0/3154742846 >> v2:172.21.15.154:6812/254845342 conn(0x7f7bf0037810 msgr2=0x7f7bf0039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:59.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.373+0000 7f7c1381b640 1 --2- 172.21.15.42:0/3154742846 >> v2:172.21.15.154:6812/254845342 conn(0x7f7bf0037810 0x7f7bf0039cd0 secure :-1 s=READY pgs=3574 cs=0 l=1 rev1=1 crypto rx=0x7f7c08007a90 tx=0x7f7c080079e0 comp rx=0 tx=0).stop 2024-09-16T03:51:59.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.373+0000 7f7c1381b640 10 monclient: shutdown 2024-09-16T03:51:59.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.373+0000 7f7c1381b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:51:59.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.373+0000 7f7c1381b640 1 -- 172.21.15.42:0/3154742846 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7c0c13a3b0 msgr2=0x7f7c0c1327d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:59.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.373+0000 7f7c1381b640 1 --2- 172.21.15.42:0/3154742846 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7c0c13a3b0 0x7f7c0c1327d0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f7c040ac000 tx=0x7f7c040195d0 comp rx=0 tx=0).stop 2024-09-16T03:51:59.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.373+0000 7f7c11db8640 1 -- 172.21.15.42:0/3154742846 reap_dead start 2024-09-16T03:51:59.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.373+0000 7f7c11db8640 1 -- 172.21.15.42:0/3154742846 reap_dead start 2024-09-16T03:51:59.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.373+0000 7f7c1381b640 1 -- 172.21.15.42:0/3154742846 shutdown_connections 2024-09-16T03:51:59.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.373+0000 7f7c1381b640 1 -- 172.21.15.42:0/3154742846 >> 172.21.15.42:0/3154742846 conn(0x7f7c0c0a2420 msgr2=0x7f7c0c139c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:51:59.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.373+0000 7f7c1381b640 1 -- 172.21.15.42:0/3154742846 shutdown_connections 2024-09-16T03:51:59.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.373+0000 7f7c1381b640 1 -- 172.21.15.42:0/3154742846 wait complete. 2024-09-16T03:51:59.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.373+0000 7f7c1381b640 1 librados: shutdown 2024-09-16T03:51:59.424 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:51:59.425 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T03:51:59.425 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T03:51:59.542 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.537+0000 7fbf4e671640 10 monclient: get_monmap_and_config 2024-09-16T03:51:59.542 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.537+0000 7fbf4e671640 10 monclient: build_initial_monmap 2024-09-16T03:51:59.542 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.537+0000 7fbf4e671640 10 monclient: monmap: 2024-09-16T03:51:59.543 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:51:59.543 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:59.543 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:51:59.541381+0000 2024-09-16T03:51:59.543 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:51:59.541381+0000 2024-09-16T03:51:59.543 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:51:59.543 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:59.543 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:51:59.543 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:51:59.543 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:51:59.543 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:51:59.543 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:51:59.543 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:59.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.537+0000 7fbf4e671640 1 Processor -- start 2024-09-16T03:51:59.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.537+0000 7fbf4e671640 1 -- start start 2024-09-16T03:51:59.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.537+0000 7fbf4e671640 10 monclient: init 2024-09-16T03:51:59.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.537+0000 7fbf4e671640 10 monclient: _reopen_session rank -1 2024-09-16T03:51:59.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.537+0000 7fbf4e671640 10 monclient: _add_conns ranks=[1,2,4,0,3] 2024-09-16T03:51:59.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.537+0000 7fbf4e671640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbf480a1330 addr v1:172.21.15.154:6789/0 2024-09-16T03:51:59.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.537+0000 7fbf4e671640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbf480a1710 0x7fbf480a1b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:59.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.537+0000 7fbf4e671640 10 monclient(hunting): picked mon.noname-b con 0x7fbf480a1710 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:51:59.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.537+0000 7fbf4e671640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fbf481393a0 0x7fbf4813b790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:59.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.537+0000 7fbf4e671640 10 monclient(hunting): picked mon.noname-c con 0x7fbf481393a0 addr v2:172.21.15.154:3300/0 2024-09-16T03:51:59.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.537+0000 7fbf4e671640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbf4809e4c0 con 0x7fbf480a1330 2024-09-16T03:51:59.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.537+0000 7fbf4e671640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:59.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.537+0000 7fbf4e671640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fbf4809edf0 con 0x7fbf481393a0 2024-09-16T03:51:59.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.537+0000 7fbf4e671640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:59.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.537+0000 7fbf4e671640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fbf4809cd60 con 0x7fbf480a1710 2024-09-16T03:51:59.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.537+0000 7fbf4e671640 10 monclient(hunting): _renew_subs 2024-09-16T03:51:59.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.537+0000 7fbf4e671640 10 monclient(hunting): authenticate will time out at 1566.510911s 2024-09-16T03:51:59.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.537+0000 7fbf477fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbf480a1710 0x7fbf480a1b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:59.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.537+0000 7fbf477fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbf480a1710 0x7fbf480a1b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:41564/0 (socket says 172.21.15.42:41564) 2024-09-16T03:51:59.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.537+0000 7fbf477fe640 1 -- 172.21.15.42:0/4146534929 learned_addr learned my addr 172.21.15.42:0/4146534929 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:51:59.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.537+0000 7fbf477fe640 10 monclient(hunting): get_auth_request con 0x7fbf480a1710 auth_method 0 2024-09-16T03:51:59.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.537+0000 7fbf477fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:59.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.537+0000 7fbf477fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:59.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.537+0000 7fbf477fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:51:59.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.537+0000 7fbf477fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:51:59.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.537+0000 7fbf477fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:51:59.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.537+0000 7fbf477fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:51:59.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.537+0000 7fbf4cc0e640 1 --2- 172.21.15.42:0/4146534929 >> v2:172.21.15.154:3300/0 conn(0x7fbf481393a0 0x7fbf4813b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:59.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.537+0000 7fbf4cc0e640 10 monclient(hunting): get_auth_request con 0x7fbf481393a0 auth_method 0 2024-09-16T03:51:59.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.537+0000 7fbf4cc0e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:59.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.537+0000 7fbf4cc0e640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:59.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.537+0000 7fbf4cc0e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:51:59.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.537+0000 7fbf477fe640 10 monclient(hunting): handle_auth_done global_id 814235 payload 274 2024-09-16T03:51:59.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.537+0000 7fbf477fe640 1 -- 172.21.15.42:0/4146534929 >> v2:172.21.15.154:3300/0 conn(0x7fbf481393a0 msgr2=0x7fbf4813b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:51:59.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.537+0000 7fbf477fe640 1 --2- 172.21.15.42:0/4146534929 >> v2:172.21.15.154:3300/0 conn(0x7fbf481393a0 0x7fbf4813b790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:59.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.537+0000 7fbf477fe640 1 -- 172.21.15.42:0/4146534929 >> v1:172.21.15.154:6789/0 conn(0x7fbf480a1330 legacy=0x7fbf4809f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:51:59.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.537+0000 7fbf477fe640 10 monclient: _finish_hunting 0 2024-09-16T03:51:59.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.537+0000 7fbf477fe640 1 monclient: found mon.noname-b 2024-09-16T03:51:59.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.537+0000 7fbf477fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:51:59.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.537+0000 7fbf477fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T03:51:59.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.537+0000 7fbf477fe640 1 -- 172.21.15.42:0/4146534929 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf480a2050 con 0x7fbf480a1710 2024-09-16T03:51:59.552 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.541+0000 7fbf477fe640 1 --2- 172.21.15.42:0/4146534929 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbf480a1710 0x7fbf480a1b10 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fbf34009870 tx=0x7fbf3401c420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8ae2b5f9a8a9a0a0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:51:59.552 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.541+0000 7fbf46ffd640 1 -- 172.21.15.42:0/4146534929 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf340b0070 con 0x7fbf480a1710 2024-09-16T03:51:59.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.541+0000 7fbf46ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:51:59.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.541+0000 7fbf46ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:51:59.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.541+0000 7fbf46ffd640 10 monclient: dump: 2024-09-16T03:51:59.553 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:51:59.553 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:59.553 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:59.553 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:59.553 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:51:59.553 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:59.553 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:51:59.553 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:51:59.553 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:51:59.554 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:59.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.541+0000 7fbf46ffd640 10 monclient: _finish_auth 0 2024-09-16T03:51:59.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.541+0000 7fbf46ffd640 10 monclient: _check_auth_tickets 2024-09-16T03:51:59.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.541+0000 7fbf46ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:51:59.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.541+0000 7fbf46ffd640 1 -- 172.21.15.42:0/4146534929 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbf340ab600 con 0x7fbf480a1710 2024-09-16T03:51:59.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.541+0000 7fbf4e671640 5 monclient: authenticate success, global_id 814235 2024-09-16T03:51:59.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.541+0000 7fbf4e671640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:51:59.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.541+0000 7fbf46ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:51:59.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.541+0000 7fbf46ffd640 1 -- 172.21.15.42:0/4146534929 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf340ab9d0 con 0x7fbf480a1710 2024-09-16T03:51:59.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.541+0000 7fbf4e671640 10 monclient: get_monmap_and_config success 2024-09-16T03:51:59.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.541+0000 7fbf4e671640 4 set_mon_vals no callback set 2024-09-16T03:51:59.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.541+0000 7fbf46ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:51:59.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.541+0000 7fbf46ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:51:59.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.541+0000 7fbf46ffd640 10 monclient: dump: 2024-09-16T03:51:59.555 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:51:59.555 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:59.555 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:59.555 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:59.555 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:51:59.555 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:59.555 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:51:59.555 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:51:59.555 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:51:59.555 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:59.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.541+0000 7fbf4e671640 10 monclient: shutdown 2024-09-16T03:51:59.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.541+0000 7fbf4e671640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:51:59.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.541+0000 7fbf4e671640 1 -- 172.21.15.42:0/4146534929 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbf480a1710 msgr2=0x7fbf480a1b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:59.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.541+0000 7fbf4e671640 1 --2- 172.21.15.42:0/4146534929 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbf480a1710 0x7fbf480a1b10 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fbf34009870 tx=0x7fbf3401c420 comp rx=0 tx=0).stop 2024-09-16T03:51:59.556 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.541+0000 7fbf4e671640 1 -- 172.21.15.42:0/4146534929 shutdown_connections 2024-09-16T03:51:59.556 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.541+0000 7fbf4e671640 1 --2- 172.21.15.42:0/4146534929 >> v2:172.21.15.154:3300/0 conn(0x7fbf481393a0 0x7fbf4813b790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:59.556 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.541+0000 7fbf4e671640 1 --2- 172.21.15.42:0/4146534929 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbf480a1710 0x7fbf480a1b10 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:59.556 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.541+0000 7fbf4e671640 1 -- 172.21.15.42:0/4146534929 >> 172.21.15.42:0/4146534929 conn(0x7fbf480a2420 msgr2=0x7fbf480a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:51:59.556 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.541+0000 7fbf4e671640 1 -- 172.21.15.42:0/4146534929 shutdown_connections 2024-09-16T03:51:59.556 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.541+0000 7fbf4e671640 1 -- 172.21.15.42:0/4146534929 wait complete. 2024-09-16T03:51:59.556 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.545+0000 7fbf4e671640 10 monclient: build_initial_monmap 2024-09-16T03:51:59.556 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.545+0000 7fbf4e671640 10 monclient: monmap: 2024-09-16T03:51:59.556 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:51:59.556 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:51:59.556 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:51:59.556 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:51:59.557 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:51:59.557 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:59.557 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:51:59.557 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:51:59.557 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:51:59.557 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:59.557 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.545+0000 7fbf4e671640 1 librados: starting msgr at 2024-09-16T03:51:59.557 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.545+0000 7fbf4e671640 1 librados: starting objecter 2024-09-16T03:51:59.557 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.545+0000 7fbf4e671640 1 Processor -- start 2024-09-16T03:51:59.557 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.545+0000 7fbf4e671640 1 -- start start 2024-09-16T03:51:59.557 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.545+0000 7fbf4e671640 1 librados: setting wanted keys 2024-09-16T03:51:59.557 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.545+0000 7fbf4e671640 1 librados: calling monclient init 2024-09-16T03:51:59.557 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.545+0000 7fbf4e671640 10 monclient: init 2024-09-16T03:51:59.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.545+0000 7fbf4e671640 10 monclient: _reopen_session rank -1 2024-09-16T03:51:59.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.545+0000 7fbf4e671640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:51:59.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.545+0000 7fbf4e671640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbf480a1330 0x7fbf481cee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:59.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.545+0000 7fbf4e671640 10 monclient(hunting): picked mon.noname-c con 0x7fbf480a1330 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:51:59.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.545+0000 7fbf4e671640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbf480a1710 0x7fbf481342a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:59.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.545+0000 7fbf4e671640 10 monclient(hunting): picked mon.noname-b con 0x7fbf480a1710 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:51:59.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.545+0000 7fbf4e671640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbf481393a0 0x7fbf481347e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:59.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.545+0000 7fbf4e671640 10 monclient(hunting): picked mon.noname-a con 0x7fbf481393a0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:51:59.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.545+0000 7fbf4e671640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:59.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.545+0000 7fbf4e671640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fbf4813e880 con 0x7fbf481393a0 2024-09-16T03:51:59.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.545+0000 7fbf4e671640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:59.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.545+0000 7fbf4e671640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fbf4813e700 con 0x7fbf480a1710 2024-09-16T03:51:59.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.545+0000 7fbf4e671640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:59.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.545+0000 7fbf4e671640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fbf4813ea00 con 0x7fbf480a1330 2024-09-16T03:51:59.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.545+0000 7fbf4e671640 10 monclient(hunting): _renew_subs 2024-09-16T03:51:59.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.545+0000 7fbf4e671640 10 monclient(hunting): authenticate will time out at 1566.517205s 2024-09-16T03:51:59.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.545+0000 7fbf47fff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbf480a1330 0x7fbf481cee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:59.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.545+0000 7fbf4cc0e640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbf481393a0 0x7fbf481347e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:59.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.545+0000 7fbf47fff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbf480a1330 0x7fbf481cee30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:41574/0 (socket says 172.21.15.42:41574) 2024-09-16T03:51:59.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.545+0000 7fbf47fff640 1 -- 172.21.15.42:0/2142293049 learned_addr learned my addr 172.21.15.42:0/2142293049 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:51:59.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.545+0000 7fbf47fff640 10 monclient(hunting): get_auth_request con 0x7fbf480a1330 auth_method 0 2024-09-16T03:51:59.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.545+0000 7fbf47fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:59.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.545+0000 7fbf47fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:59.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.545+0000 7fbf47fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:51:59.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.545+0000 7fbf47fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:51:59.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.545+0000 7fbf47fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:51:59.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.545+0000 7fbf47fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:51:59.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.545+0000 7fbf4cc0e640 10 monclient(hunting): get_auth_request con 0x7fbf481393a0 auth_method 0 2024-09-16T03:51:59.560 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.545+0000 7fbf4cc0e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:59.560 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.545+0000 7fbf4cc0e640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:59.560 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.545+0000 7fbf4cc0e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:51:59.560 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.545+0000 7fbf477fe640 1 --2- 172.21.15.42:0/2142293049 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbf480a1710 0x7fbf481342a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:59.560 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.545+0000 7fbf477fe640 10 monclient(hunting): get_auth_request con 0x7fbf480a1710 auth_method 0 2024-09-16T03:51:59.560 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.545+0000 7fbf477fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:59.560 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.545+0000 7fbf477fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:59.560 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.545+0000 7fbf477fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:51:59.560 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.545+0000 7fbf47fff640 10 monclient(hunting): handle_auth_done global_id 814238 payload 824 2024-09-16T03:51:59.560 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.545+0000 7fbf47fff640 1 -- 172.21.15.42:0/2142293049 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbf480a1710 msgr2=0x7fbf481342a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:59.560 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.545+0000 7fbf47fff640 1 --2- 172.21.15.42:0/2142293049 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbf480a1710 0x7fbf481342a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:59.560 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.545+0000 7fbf47fff640 1 -- 172.21.15.42:0/2142293049 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbf481393a0 msgr2=0x7fbf481347e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:59.560 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.545+0000 7fbf47fff640 1 --2- 172.21.15.42:0/2142293049 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbf481393a0 0x7fbf481347e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:59.560 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.545+0000 7fbf47fff640 10 monclient: _finish_hunting 0 2024-09-16T03:51:59.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.545+0000 7fbf47fff640 1 monclient: found mon.noname-c 2024-09-16T03:51:59.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.545+0000 7fbf47fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:51:59.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.545+0000 7fbf47fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:51:59.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.545+0000 7fbf47fff640 1 -- 172.21.15.42:0/2142293049 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf48073a30 con 0x7fbf480a1330 2024-09-16T03:51:59.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.545+0000 7fbf47fff640 1 --2- 172.21.15.42:0/2142293049 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbf480a1330 0x7fbf481cee30 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fbf3800c500 tx=0x7fbf3800c9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:51:59.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.545+0000 7fbf44ff9640 10 client.?.objecter ms_handle_connect 0x7fbf480a1330 2024-09-16T03:51:59.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.545+0000 7fbf44ff9640 10 client.?.objecter resend_mon_ops 2024-09-16T03:51:59.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.549+0000 7fbf44ff9640 1 -- 172.21.15.42:0/2142293049 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf38016020 con 0x7fbf480a1330 2024-09-16T03:51:59.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.549+0000 7fbf44ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:51:59.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.549+0000 7fbf44ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:51:59.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.549+0000 7fbf44ff9640 10 monclient: dump: 2024-09-16T03:51:59.564 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:51:59.564 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:59.564 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:59.564 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:59.564 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:51:59.564 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:59.564 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:51:59.564 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:51:59.564 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:51:59.564 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:59.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.549+0000 7fbf44ff9640 10 monclient: _finish_auth 0 2024-09-16T03:51:59.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.549+0000 7fbf44ff9640 10 monclient: _check_auth_tickets 2024-09-16T03:51:59.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.549+0000 7fbf44ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:51:59.565 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.549+0000 7fbf44ff9640 1 -- 172.21.15.42:0/2142293049 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbf38005220 con 0x7fbf480a1330 2024-09-16T03:51:59.565 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.549+0000 7fbf4e671640 5 monclient: authenticate success, global_id 814238 2024-09-16T03:51:59.565 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.549+0000 7fbf44ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T03:51:59.565 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.549+0000 7fbf44ff9640 1 -- 172.21.15.42:0/2142293049 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf38007030 con 0x7fbf480a1330 2024-09-16T03:51:59.565 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.549+0000 7fbf44ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:51:59.565 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.549+0000 7fbf44ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:51:59.565 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.549+0000 7fbf44ff9640 10 monclient: dump: 2024-09-16T03:51:59.565 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:51:59.565 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:59.565 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:59.565 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:59.565 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:51:59.566 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:59.566 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:51:59.566 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:51:59.566 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:51:59.566 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:59.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.549+0000 7fbf467fc640 4 set_mon_vals no callback set 2024-09-16T03:51:59.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.549+0000 7fbf4e671640 10 monclient: _renew_subs 2024-09-16T03:51:59.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.549+0000 7fbf4e671640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:51:59.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.549+0000 7fbf4e671640 1 -- 172.21.15.42:0/2142293049 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf48121980 con 0x7fbf480a1330 2024-09-16T03:51:59.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.549+0000 7fbf4e671640 10 client.814238.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:51:59.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.549+0000 7fbf4e671640 10 monclient: _renew_subs 2024-09-16T03:51:59.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.549+0000 7fbf4e671640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:51:59.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.549+0000 7fbf4e671640 1 -- 172.21.15.42:0/2142293049 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf480a3820 con 0x7fbf480a1330 2024-09-16T03:51:59.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.549+0000 7fbf4e671640 1 librados: init done 2024-09-16T03:51:59.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.549+0000 7fbf4e671640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:51:59.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.549+0000 7fbf4e671640 1 --2- 172.21.15.42:0/2142293049 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbf48121550 0x7fbf4811e490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:59.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.549+0000 7fbf4e671640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:59.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.549+0000 7fbf4e671640 1 -- 172.21.15.42:0/2142293049 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fbf480a1b10 con 0x7fbf48121550 2024-09-16T03:51:59.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.549+0000 7fbf44ff9640 1 -- 172.21.15.42:0/2142293049 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fbf380071f0 con 0x7fbf480a1330 2024-09-16T03:51:59.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.549+0000 7fbf44ff9640 1 --2- 172.21.15.42:0/2142293049 >> v2:172.21.15.154:6812/254845342 conn(0x7fbf30037840 0x7fbf30039d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:59.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.549+0000 7fbf44ff9640 1 -- 172.21.15.42:0/2142293049 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(332..332 src has 286..332) ==== 5328+0+0 (secure 0 0 0) 0x7fbf380486e0 con 0x7fbf480a1330 2024-09-16T03:51:59.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.549+0000 7fbf44ff9640 10 client.814238.objecter ms_dispatch 0x7fbf48000ba0 osd_map(332..332 src has 286..332) 2024-09-16T03:51:59.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.549+0000 7fbf44ff9640 3 client.814238.objecter handle_osd_map got epochs [332,332] > 0 2024-09-16T03:51:59.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.549+0000 7fbf44ff9640 3 client.814238.objecter handle_osd_map decoding full epoch 332 2024-09-16T03:51:59.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.549+0000 7fbf477fe640 1 --2- 172.21.15.42:0/2142293049 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbf48121550 0x7fbf4811e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:59.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.549+0000 7fbf44ff9640 20 client.814238.objecter dump_active .. 0 homeless 2024-09-16T03:51:59.568 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.549+0000 7fbf477fe640 10 monclient: get_auth_request con 0x7fbf48121550 auth_method 0 2024-09-16T03:51:59.568 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.549+0000 7fbf477fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:59.568 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.549+0000 7fbf477fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:59.568 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.549+0000 7fbf477fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:51:59.568 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.549+0000 7fbf477fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:51:59.568 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.549+0000 7fbf477fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:51:59.568 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.549+0000 7fbf477fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:51:59.568 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.553+0000 7fbf477fe640 10 monclient(hunting): handle_auth_done global_id 794434 payload 824 2024-09-16T03:51:59.568 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.553+0000 7fbf477fe640 1 -- 172.21.15.42:0/2142293049 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbf48120250 con 0x7fbf48121550 2024-09-16T03:51:59.568 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.553+0000 7fbf4cc0e640 1 --2- 172.21.15.42:0/2142293049 >> v2:172.21.15.154:6812/254845342 conn(0x7fbf30037840 0x7fbf30039d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:59.568 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.553+0000 7fbf4cc0e640 10 monclient: get_auth_request con 0x7fbf30037840 auth_method 0 2024-09-16T03:51:59.568 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.553+0000 7fbf44ff9640 10 client.814238.objecter ms_handle_connect 0x7fbf48121550 2024-09-16T03:51:59.569 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.553+0000 7fbf44ff9640 10 client.814238.objecter resend_mon_ops 2024-09-16T03:51:59.569 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.553+0000 7fbf477fe640 1 --2- 172.21.15.42:0/2142293049 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbf48121550 0x7fbf4811e490 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fbf4806add0 tx=0x7fbf3401ac60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:51:59.569 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.553+0000 7fbf4cc0e640 1 --2- 172.21.15.42:0/2142293049 >> v2:172.21.15.154:6812/254845342 conn(0x7fbf30037840 0x7fbf30039d00 secure :-1 s=READY pgs=3575 cs=0 l=1 rev1=1 crypto rx=0x7fbf4813ba20 tx=0x7fbf4000b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:51:59.569 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.553+0000 7fbf44ff9640 10 client.814238.objecter ms_handle_connect 0x7fbf30037840 2024-09-16T03:51:59.569 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.561+0000 7fbf44ff9640 1 -- 172.21.15.42:0/2142293049 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf340b0040 con 0x7fbf48121550 2024-09-16T03:51:59.570 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.561+0000 7fbf44ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:51:59.570 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.561+0000 7fbf44ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:51:59.570 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.561+0000 7fbf44ff9640 10 monclient: dump: 2024-09-16T03:51:59.570 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:51:59.570 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:59.570 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:59.570 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:59.570 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:51:59.570 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:59.570 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:51:59.570 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:51:59.570 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:51:59.570 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:59.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.561+0000 7fbf44ff9640 1 -- 172.21.15.42:0/2142293049 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fbf34009490 con 0x7fbf48121550 2024-09-16T03:51:59.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.561+0000 7fbf44ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:51:59.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.561+0000 7fbf44ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:51:59.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.561+0000 7fbf44ff9640 1 -- 172.21.15.42:0/2142293049 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbf48121550 msgr2=0x7fbf4811e490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:59.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.561+0000 7fbf44ff9640 1 --2- 172.21.15.42:0/2142293049 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbf48121550 0x7fbf4811e490 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fbf4806add0 tx=0x7fbf3401ac60 comp rx=0 tx=0).stop 2024-09-16T03:51:59.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.561+0000 7fbf44ff9640 1 -- 172.21.15.42:0/2142293049 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbf48121550 msgr2=0x7fbf4811e490 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:51:59.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.561+0000 7fbf44ff9640 1 --2- 172.21.15.42:0/2142293049 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbf48121550 0x7fbf4811e490 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:59.573 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.569+0000 7fbf4e671640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:51:59.573 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.569+0000 7fbf4e671640 1 --2- 172.21.15.42:0/2142293049 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbf4802c650 0x7fbf4802ca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:51:59.573 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.569+0000 7fbf4e671640 10 monclient(hunting): start opening mon connection 2024-09-16T03:51:59.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.569+0000 7fbf4e671640 1 -- 172.21.15.42:0/2142293049 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fbf48136a90 con 0x7fbf4802c650 2024-09-16T03:51:59.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.569+0000 7fbf477fe640 1 --2- 172.21.15.42:0/2142293049 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbf4802c650 0x7fbf4802ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:51:59.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.569+0000 7fbf477fe640 10 monclient: get_auth_request con 0x7fbf4802c650 auth_method 0 2024-09-16T03:51:59.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.569+0000 7fbf477fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:51:59.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.569+0000 7fbf477fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:51:59.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.569+0000 7fbf477fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:51:59.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.569+0000 7fbf477fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:51:59.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.569+0000 7fbf477fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:51:59.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.569+0000 7fbf477fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:51:59.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.569+0000 7fbf477fe640 10 monclient(hunting): handle_auth_done global_id 794437 payload 824 2024-09-16T03:51:59.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.569+0000 7fbf477fe640 1 -- 172.21.15.42:0/2142293049 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fbf481206f0 con 0x7fbf4802c650 2024-09-16T03:51:59.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.569+0000 7fbf44ff9640 10 client.814238.objecter ms_handle_connect 0x7fbf4802c650 2024-09-16T03:51:59.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.569+0000 7fbf44ff9640 10 client.814238.objecter resend_mon_ops 2024-09-16T03:51:59.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.569+0000 7fbf477fe640 1 --2- 172.21.15.42:0/2142293049 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbf4802c650 0x7fbf4802ca30 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fbf3401c460 tx=0x7fbf340191c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:51:59.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.569+0000 7fbf44ff9640 1 -- 172.21.15.42:0/2142293049 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf340b0070 con 0x7fbf4802c650 2024-09-16T03:51:59.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.569+0000 7fbf44ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:51:59.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.569+0000 7fbf44ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:51:59.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.569+0000 7fbf44ff9640 10 monclient: dump: 2024-09-16T03:51:59.577 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:51:59.577 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:51:59.577 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:59.577 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:51:59.577 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:51:59.577 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:51:59.577 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:51:59.577 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:51:59.577 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:51:59.577 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:51:59.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.569+0000 7fbf44ff9640 1 -- 172.21.15.42:0/2142293049 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4712 (secure 0 0 0) 0x7fbf340c0e50 con 0x7fbf4802c650 2024-09-16T03:51:59.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.569+0000 7fbf44ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:51:59.578 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.569+0000 7fbf44ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:51:59.578 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.569+0000 7fbf44ff9640 1 -- 172.21.15.42:0/2142293049 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbf4802c650 msgr2=0x7fbf4802ca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:59.578 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.569+0000 7fbf44ff9640 1 --2- 172.21.15.42:0/2142293049 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbf4802c650 0x7fbf4802ca30 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fbf3401c460 tx=0x7fbf340191c0 comp rx=0 tx=0).stop 2024-09-16T03:51:59.578 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.569+0000 7fbf44ff9640 1 -- 172.21.15.42:0/2142293049 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbf4802c650 msgr2=0x7fbf4802ca30 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:51:59.578 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.569+0000 7fbf44ff9640 1 --2- 172.21.15.42:0/2142293049 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbf4802c650 0x7fbf4802ca30 secure :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fbf3401c460 tx=0x7fbf340191c0 comp rx=0 tx=0).stop 2024-09-16T03:51:59.578 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:51:59.578 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:51:59.578 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:51:59.578 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T03:51:59.579 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 86, 2024-09-16T03:51:59.579 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 161413, 2024-09-16T03:51:59.579 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:51:59.579 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:51:59.579 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T03:51:59.579 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T03:51:59.579 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:51:59.579 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 3, 2024-09-16T03:51:59.579 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:51:59.579 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:51:59.579 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:51:59.579 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:51:59.579 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:51:59.580 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:51:59.580 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:51:59.580 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:51:59.580 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:51:59.580 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:51:59.580 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:51:59.580 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:51:59.580 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:51:59.580 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:51:59.580 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:51:59.580 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:51:59.580 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:51:59.580 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:51:59.581 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:51:59.581 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:51:59.581 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:51:59.581 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:51:59.581 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:51:59.581 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:51:59.581 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:51:59.581 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:51:59.581 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:51:59.581 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:51:59.581 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:51:59.581 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:59.581 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:51:59.581 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:51:59.582 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:51:59.582 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:51:59.582 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:51:59.582 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:51:59.582 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:51:59.582 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:51:59.582 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:51:59.582 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:51:59.582 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:51:59.582 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:51:59.582 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:51:59.582 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:51:59.582 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:51:59.583 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:51:59.583 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:51:59.583 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:51:59.583 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:51:59.583 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:51:59.583 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:51:59.583 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:51:59.583 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:51:59.583 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:51:59.583 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:51:59.583 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:51:59.583 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:51:59.584 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:51:59.584 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:51:59.584 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:51:59.584 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:59.584 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:51:59.584 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:59.584 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:51:59.584 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:51:59.584 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:51:59.584 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:51:59.584 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:59.584 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:51:59.584 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:51:59.584 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:51:59.585 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:59.585 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:59.585 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:51:59.585 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:51:59.585 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:51:59.585 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:51:59.585 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:51:59.585 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:59.585 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:51:59.585 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:51:59.585 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:51:59.585 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:51:59.586 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:51:59.586 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:59.586 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:59.586 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:51:59.586 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:51:59.586 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:51:59.586 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:51:59.586 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:59.586 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:51:59.586 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:51:59.586 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:51:59.586 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:59.586 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:59.587 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:51:59.587 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:51:59.587 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:51:59.587 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:51:59.587 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:51:59.587 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:59.587 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:51:59.587 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:51:59.587 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:51:59.587 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:51:59.587 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:51:59.587 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:59.587 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:59.588 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:51:59.588 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:51:59.588 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:51:59.588 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:51:59.588 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:59.588 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:51:59.588 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:51:59.588 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:51:59.588 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:59.588 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:59.588 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:51:59.588 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:51:59.588 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:51:59.589 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:51:59.589 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:51:59.589 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:59.589 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:51:59.589 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:51:59.589 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:51:59.589 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:51:59.589 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:51:59.589 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:51:59.589 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:51:59.589 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:59.589 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:51:59.590 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:51:59.590 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:59.590 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:51:59.590 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:51:59.590 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:51:59.590 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:51:59.590 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:51:59.590 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T03:51:59.590 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:59.590 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:51:59.590 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:51:59.590 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:51:59.590 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:51:59.591 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:51:59.591 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:51:59.591 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:51:59.591 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:51:59.591 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:51:59.591 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 2 2024-09-16T03:51:59.591 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:51:59.591 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:51:59.591 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:51:59.591 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:51:59.591 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:51:59.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.569+0000 7fbf4e671640 10 librados: watch_flush enter 2024-09-16T03:51:59.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.569+0000 7fbf4e671640 10 librados: watch_flush exit 2024-09-16T03:51:59.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.569+0000 7fbf4e671640 20 client.814238.objecter shutdown clearing up homeless session... 2024-09-16T03:51:59.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.569+0000 7fbf4e671640 10 client.814238.objecter successfully canceled tick 2024-09-16T03:51:59.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.569+0000 7fbf4e671640 1 -- 172.21.15.42:0/2142293049 >> v2:172.21.15.154:6812/254845342 conn(0x7fbf30037840 msgr2=0x7fbf30039d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:59.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.569+0000 7fbf4e671640 1 --2- 172.21.15.42:0/2142293049 >> v2:172.21.15.154:6812/254845342 conn(0x7fbf30037840 0x7fbf30039d00 secure :-1 s=READY pgs=3575 cs=0 l=1 rev1=1 crypto rx=0x7fbf4813ba20 tx=0x7fbf4000b040 comp rx=0 tx=0).stop 2024-09-16T03:51:59.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.569+0000 7fbf4e671640 10 monclient: shutdown 2024-09-16T03:51:59.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.569+0000 7fbf4e671640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:51:59.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.569+0000 7fbf4e671640 1 -- 172.21.15.42:0/2142293049 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbf480a1330 msgr2=0x7fbf481cee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:51:59.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.569+0000 7fbf4e671640 1 --2- 172.21.15.42:0/2142293049 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbf480a1330 0x7fbf481cee30 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fbf3800c500 tx=0x7fbf3800c9c0 comp rx=0 tx=0).stop 2024-09-16T03:51:59.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.569+0000 7fbf4cc0e640 1 -- 172.21.15.42:0/2142293049 reap_dead start 2024-09-16T03:51:59.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.569+0000 7fbf4e671640 1 -- 172.21.15.42:0/2142293049 shutdown_connections 2024-09-16T03:51:59.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.569+0000 7fbf4e671640 1 --2- 172.21.15.42:0/2142293049 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbf480a1330 0x7fbf481cee30 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:51:59.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.569+0000 7fbf4e671640 1 -- 172.21.15.42:0/2142293049 >> 172.21.15.42:0/2142293049 conn(0x7fbf480a2420 msgr2=0x7fbf481386a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:51:59.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.569+0000 7fbf4e671640 1 -- 172.21.15.42:0/2142293049 shutdown_connections 2024-09-16T03:51:59.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.569+0000 7fbf4e671640 1 -- 172.21.15.42:0/2142293049 wait complete. 2024-09-16T03:51:59.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:51:59.569+0000 7fbf4e671640 1 librados: shutdown 2024-09-16T03:51:59.594 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:51:59.595 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-09-16T03:51:59.595 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['a'] 2024-09-16T03:51:59.595 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-09-16T03:51:59.595 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-16T03:51:59.595 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-16T03:51:59.595 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T03:51:59.609 INFO:tasks.ceph.mon.c:Stopped 2024-09-16T03:51:59.609 INFO:tasks.mon_thrash:Sending STOP to mon a 2024-09-16T03:51:59.609 INFO:tasks.ceph.mon.a:Sent signal 19 2024-09-16T03:51:59.609 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-16T03:52:14.610 INFO:tasks.mon_thrash:Sending CONT to mon a 2024-09-16T03:52:14.611 INFO:tasks.ceph.mon.a:Sent signal 18 2024-09-16T03:52:14.611 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T03:52:14.611 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:52:14.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:14.737+0000 7f7efc6de640 10 monclient: get_monmap_and_config 2024-09-16T03:52:14.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:14.737+0000 7f7efc6de640 10 monclient: build_initial_monmap 2024-09-16T03:52:14.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:14.737+0000 7f7efc6de640 10 monclient: monmap: 2024-09-16T03:52:14.743 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:52:14.743 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:52:14.743 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:52:14.741794+0000 2024-09-16T03:52:14.743 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:52:14.741794+0000 2024-09-16T03:52:14.744 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:52:14.744 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:52:14.744 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:52:14.744 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:52:14.744 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:52:14.744 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:52:14.744 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:52:14.744 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:52:14.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:14.737+0000 7f7efc6de640 1 Processor -- start 2024-09-16T03:52:14.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:14.737+0000 7f7efc6de640 1 -- start start 2024-09-16T03:52:14.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:14.737+0000 7f7efc6de640 10 monclient: init 2024-09-16T03:52:14.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:14.737+0000 7f7efc6de640 10 monclient: _reopen_session rank -1 2024-09-16T03:52:14.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:14.737+0000 7f7efc6de640 10 monclient: _add_conns ranks=[4,0,3,1,2] 2024-09-16T03:52:14.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:14.737+0000 7f7efc6de640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f7ef40a1330 0x7f7ef40a1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:52:14.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:14.737+0000 7f7efc6de640 10 monclient(hunting): picked mon.noname-c con 0x7f7ef40a1330 addr v2:172.21.15.154:3300/0 2024-09-16T03:52:14.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:14.737+0000 7f7efc6de640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7ef406dac0 addr v1:172.21.15.42:6789/0 2024-09-16T03:52:14.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:14.737+0000 7f7efc6de640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f7ef40a1c50 0x7f7ef4135810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:52:14.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:14.737+0000 7f7efc6de640 10 monclient(hunting): picked mon.noname-a con 0x7f7ef40a1c50 addr v2:172.21.15.42:3300/0 2024-09-16T03:52:14.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:14.737+0000 7f7efc6de640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7ef409c090 con 0x7f7ef406dac0 2024-09-16T03:52:14.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:14.737+0000 7f7efc6de640 10 monclient(hunting): start opening mon connection 2024-09-16T03:52:14.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:14.737+0000 7f7efc6de640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f7ef409f610 con 0x7f7ef40a1c50 2024-09-16T03:52:14.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:14.737+0000 7f7efc6de640 10 monclient(hunting): start opening mon connection 2024-09-16T03:52:14.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:14.737+0000 7f7efc6de640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f7ef409f8f0 con 0x7f7ef40a1330 2024-09-16T03:52:14.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:14.737+0000 7f7efc6de640 10 monclient(hunting): _renew_subs 2024-09-16T03:52:14.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:14.737+0000 7f7efc6de640 10 monclient(hunting): authenticate will time out at 1581.711252s 2024-09-16T03:52:14.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:14.737+0000 7f7efac7b640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f7ef40a1c50 0x7f7ef4135810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:52:14.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:14.737+0000 7f7ef9c79640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f7ef406dac0 0x7f7ef4135100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:42732/0 (socket says 172.21.15.42:42732) 2024-09-16T03:52:14.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:14.737+0000 7f7efac7b640 10 monclient(hunting): get_auth_request con 0x7f7ef40a1c50 auth_method 0 2024-09-16T03:52:14.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:14.737+0000 7f7efac7b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:52:14.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:14.737+0000 7f7efac7b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:52:14.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:14.737+0000 7f7efac7b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:52:14.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:14.737+0000 7f7ef9c79640 1 -- 172.21.15.42:0/285916117 learned_addr learned my addr 172.21.15.42:0/285916117 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:52:14.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:14.737+0000 7f7efac7b640 1 -- 172.21.15.42:0/285916117 >> v2:172.21.15.42:3300/0 conn(0x7f7ef40a1c50 msgr2=0x7f7ef4135810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T03:52:14.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:14.737+0000 7f7efac7b640 1 -- 172.21.15.42:0/285916117 >> v2:172.21.15.42:3300/0 conn(0x7f7ef40a1c50 msgr2=0x7f7ef4135810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:52:14.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:14.737+0000 7f7efac7b640 1 --2- 172.21.15.42:0/285916117 >> v2:172.21.15.42:3300/0 conn(0x7f7ef40a1c50 0x7f7ef4135810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:52:14.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:14.737+0000 7f7efa47a640 1 --2- 172.21.15.42:0/285916117 >> v2:172.21.15.154:3300/0 conn(0x7f7ef40a1330 0x7f7ef40a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:52:14.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:14.737+0000 7f7efac7b640 1 --2- 172.21.15.42:0/285916117 >> v2:172.21.15.42:3300/0 conn(0x7f7ef40a1c50 0x7f7ef4135810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:52:14.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:14.737+0000 7f7efa47a640 10 monclient(hunting): get_auth_request con 0x7f7ef40a1330 auth_method 0 2024-09-16T03:52:14.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:14.737+0000 7f7efa47a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:52:14.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:14.737+0000 7f7efa47a640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:52:14.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:14.737+0000 7f7efa47a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:52:14.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:14.737+0000 7f7efa47a640 1 -- 172.21.15.42:0/285916117 >> v2:172.21.15.154:3300/0 conn(0x7f7ef40a1330 msgr2=0x7f7ef40a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T03:52:14.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:14.737+0000 7f7efa47a640 1 -- 172.21.15.42:0/285916117 >> v2:172.21.15.154:3300/0 conn(0x7f7ef40a1330 msgr2=0x7f7ef40a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:52:14.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:14.737+0000 7f7efa47a640 1 --2- 172.21.15.42:0/285916117 >> v2:172.21.15.154:3300/0 conn(0x7f7ef40a1330 0x7f7ef40a1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:52:14.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:14.737+0000 7f7efa47a640 1 --2- 172.21.15.42:0/285916117 >> v2:172.21.15.154:3300/0 conn(0x7f7ef40a1330 0x7f7ef40a1710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:52:14.946 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:14.941+0000 7f7efac7b640 1 --2- 172.21.15.42:0/285916117 >> v2:172.21.15.42:3300/0 conn(0x7f7ef40a1c50 0x7f7ef4135810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:52:14.946 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:14.941+0000 7f7efac7b640 10 monclient(hunting): get_auth_request con 0x7f7ef40a1c50 auth_method 0 2024-09-16T03:52:14.946 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:14.941+0000 7f7efac7b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:52:14.946 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:14.941+0000 7f7efac7b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:52:14.946 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:14.941+0000 7f7efac7b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:52:14.946 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:14.941+0000 7f7efa47a640 1 --2- 172.21.15.42:0/285916117 >> v2:172.21.15.154:3300/0 conn(0x7f7ef40a1330 0x7f7ef40a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:52:14.947 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:14.941+0000 7f7efa47a640 10 monclient(hunting): get_auth_request con 0x7f7ef40a1330 auth_method 0 2024-09-16T03:52:14.947 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:14.941+0000 7f7efa47a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:52:14.947 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:14.941+0000 7f7efa47a640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:52:14.947 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:14.941+0000 7f7efa47a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:52:14.947 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:14.941+0000 7f7efac7b640 1 -- 172.21.15.42:0/285916117 >> v2:172.21.15.42:3300/0 conn(0x7f7ef40a1c50 msgr2=0x7f7ef4135810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T03:52:14.947 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:14.941+0000 7f7efac7b640 1 -- 172.21.15.42:0/285916117 >> v2:172.21.15.42:3300/0 conn(0x7f7ef40a1c50 msgr2=0x7f7ef4135810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:52:14.947 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:14.941+0000 7f7efac7b640 1 --2- 172.21.15.42:0/285916117 >> v2:172.21.15.42:3300/0 conn(0x7f7ef40a1c50 0x7f7ef4135810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:52:14.947 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:14.941+0000 7f7efac7b640 1 --2- 172.21.15.42:0/285916117 >> v2:172.21.15.42:3300/0 conn(0x7f7ef40a1c50 0x7f7ef4135810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T03:52:14.947 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:14.941+0000 7f7efa47a640 1 -- 172.21.15.42:0/285916117 >> v2:172.21.15.154:3300/0 conn(0x7f7ef40a1330 msgr2=0x7f7ef40a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T03:52:14.947 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:14.941+0000 7f7efa47a640 1 -- 172.21.15.42:0/285916117 >> v2:172.21.15.154:3300/0 conn(0x7f7ef40a1330 msgr2=0x7f7ef40a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:52:14.947 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:14.941+0000 7f7efa47a640 1 --2- 172.21.15.42:0/285916117 >> v2:172.21.15.154:3300/0 conn(0x7f7ef40a1330 0x7f7ef40a1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:52:14.947 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:14.941+0000 7f7efa47a640 1 --2- 172.21.15.42:0/285916117 >> v2:172.21.15.154:3300/0 conn(0x7f7ef40a1330 0x7f7ef40a1710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T03:52:15.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:15.345+0000 7f7efac7b640 1 --2- 172.21.15.42:0/285916117 >> v2:172.21.15.42:3300/0 conn(0x7f7ef40a1c50 0x7f7ef4135810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:52:15.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:15.345+0000 7f7efac7b640 10 monclient(hunting): get_auth_request con 0x7f7ef40a1c50 auth_method 0 2024-09-16T03:52:15.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:15.345+0000 7f7efac7b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:52:15.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:15.345+0000 7f7efac7b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:52:15.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:15.345+0000 7f7efac7b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:52:15.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:15.345+0000 7f7efa47a640 1 --2- 172.21.15.42:0/285916117 >> v2:172.21.15.154:3300/0 conn(0x7f7ef40a1330 0x7f7ef40a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:52:15.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:15.345+0000 7f7efa47a640 10 monclient(hunting): get_auth_request con 0x7f7ef40a1330 auth_method 0 2024-09-16T03:52:15.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:15.345+0000 7f7efa47a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:52:15.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:15.345+0000 7f7efa47a640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:52:15.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:15.345+0000 7f7efa47a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:52:15.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:15.345+0000 7f7efac7b640 1 -- 172.21.15.42:0/285916117 >> v2:172.21.15.42:3300/0 conn(0x7f7ef40a1c50 msgr2=0x7f7ef4135810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T03:52:15.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:15.345+0000 7f7efac7b640 1 -- 172.21.15.42:0/285916117 >> v2:172.21.15.42:3300/0 conn(0x7f7ef40a1c50 msgr2=0x7f7ef4135810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:52:15.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:15.345+0000 7f7efac7b640 1 --2- 172.21.15.42:0/285916117 >> v2:172.21.15.42:3300/0 conn(0x7f7ef40a1c50 0x7f7ef4135810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:52:15.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:15.345+0000 7f7efac7b640 1 --2- 172.21.15.42:0/285916117 >> v2:172.21.15.42:3300/0 conn(0x7f7ef40a1c50 0x7f7ef4135810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T03:52:15.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:15.345+0000 7f7efa47a640 1 -- 172.21.15.42:0/285916117 >> v2:172.21.15.154:3300/0 conn(0x7f7ef40a1330 msgr2=0x7f7ef40a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T03:52:15.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:15.345+0000 7f7efa47a640 1 -- 172.21.15.42:0/285916117 >> v2:172.21.15.154:3300/0 conn(0x7f7ef40a1330 msgr2=0x7f7ef40a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:52:15.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:15.345+0000 7f7efa47a640 1 --2- 172.21.15.42:0/285916117 >> v2:172.21.15.154:3300/0 conn(0x7f7ef40a1330 0x7f7ef40a1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:52:15.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:15.345+0000 7f7efa47a640 1 --2- 172.21.15.42:0/285916117 >> v2:172.21.15.154:3300/0 conn(0x7f7ef40a1330 0x7f7ef40a1710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T03:52:16.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:16.149+0000 7f7efac7b640 1 --2- 172.21.15.42:0/285916117 >> v2:172.21.15.42:3300/0 conn(0x7f7ef40a1c50 0x7f7ef4135810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:52:16.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:16.149+0000 7f7efac7b640 10 monclient(hunting): get_auth_request con 0x7f7ef40a1c50 auth_method 0 2024-09-16T03:52:16.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:16.149+0000 7f7efac7b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:52:16.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:16.149+0000 7f7efac7b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:52:16.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:16.149+0000 7f7efac7b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:52:16.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:16.149+0000 7f7efa47a640 1 --2- 172.21.15.42:0/285916117 >> v2:172.21.15.154:3300/0 conn(0x7f7ef40a1330 0x7f7ef40a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:52:16.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:16.149+0000 7f7efac7b640 1 -- 172.21.15.42:0/285916117 >> v2:172.21.15.42:3300/0 conn(0x7f7ef40a1c50 msgr2=0x7f7ef4135810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T03:52:16.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:16.149+0000 7f7efac7b640 1 -- 172.21.15.42:0/285916117 >> v2:172.21.15.42:3300/0 conn(0x7f7ef40a1c50 msgr2=0x7f7ef4135810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:52:16.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:16.149+0000 7f7efac7b640 1 --2- 172.21.15.42:0/285916117 >> v2:172.21.15.42:3300/0 conn(0x7f7ef40a1c50 0x7f7ef4135810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:52:16.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:16.149+0000 7f7efa47a640 10 monclient(hunting): get_auth_request con 0x7f7ef40a1330 auth_method 0 2024-09-16T03:52:16.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:16.149+0000 7f7efa47a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:52:16.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:16.149+0000 7f7efa47a640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:52:16.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:16.149+0000 7f7efa47a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:52:16.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:16.149+0000 7f7efac7b640 1 --2- 172.21.15.42:0/285916117 >> v2:172.21.15.42:3300/0 conn(0x7f7ef40a1c50 0x7f7ef4135810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T03:52:16.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:16.149+0000 7f7efa47a640 1 -- 172.21.15.42:0/285916117 >> v2:172.21.15.154:3300/0 conn(0x7f7ef40a1330 msgr2=0x7f7ef40a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T03:52:16.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:16.149+0000 7f7efa47a640 1 -- 172.21.15.42:0/285916117 >> v2:172.21.15.154:3300/0 conn(0x7f7ef40a1330 msgr2=0x7f7ef40a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:52:16.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:16.149+0000 7f7efa47a640 1 --2- 172.21.15.42:0/285916117 >> v2:172.21.15.154:3300/0 conn(0x7f7ef40a1330 0x7f7ef40a1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:52:16.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:16.149+0000 7f7efa47a640 1 --2- 172.21.15.42:0/285916117 >> v2:172.21.15.154:3300/0 conn(0x7f7ef40a1330 0x7f7ef40a1710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T03:52:17.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:17.737+0000 7f7ee7fff640 10 monclient(hunting): tick 2024-09-16T03:52:17.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:17.737+0000 7f7ee7fff640 10 monclient(hunting): _check_auth_tickets 2024-09-16T03:52:17.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:17.737+0000 7f7ee7fff640 1 monclient(hunting): continuing hunt 2024-09-16T03:52:17.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:17.737+0000 7f7ee7fff640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T03:52:17.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:17.737+0000 7f7ee7fff640 1 -- 172.21.15.42:0/285916117 >> v2:172.21.15.154:3300/0 conn(0x7f7ef40a1330 msgr2=0x7f7ef40a1710 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:52:17.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:17.737+0000 7f7ee7fff640 1 --2- 172.21.15.42:0/285916117 >> v2:172.21.15.154:3300/0 conn(0x7f7ef40a1330 0x7f7ef40a1710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:52:17.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:17.737+0000 7f7ee7fff640 1 -- 172.21.15.42:0/285916117 >> v2:172.21.15.42:3300/0 conn(0x7f7ef40a1c50 msgr2=0x7f7ef4135810 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:52:17.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:17.737+0000 7f7ee7fff640 1 --2- 172.21.15.42:0/285916117 >> v2:172.21.15.42:3300/0 conn(0x7f7ef40a1c50 0x7f7ef4135810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:52:17.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:17.737+0000 7f7ee7fff640 1 -- 172.21.15.42:0/285916117 >> v1:172.21.15.42:6789/0 conn(0x7f7ef406dac0 legacy=0x7f7ef4135100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:52:17.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:17.737+0000 7f7ee7fff640 10 monclient: _add_conns ranks=[2,1] 2024-09-16T03:52:17.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:17.737+0000 7f7ee7fff640 1 --2- 172.21.15.42:0/285916117 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7ed80036c0 0x7f7ed8005af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:52:17.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:17.737+0000 7f7ee7fff640 10 monclient(hunting): picked mon.noname-b con 0x7f7ed80036c0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:52:17.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:17.737+0000 7f7ee7fff640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7ed8006180 addr v1:172.21.15.154:6789/0 2024-09-16T03:52:17.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:17.737+0000 7f7ee7fff640 1 -- 172.21.15.42:0/285916117 --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7ef409fbd0 con 0x7f7ed8006180 2024-09-16T03:52:17.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:17.737+0000 7f7ee7fff640 10 monclient(hunting): start opening mon connection 2024-09-16T03:52:17.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:17.737+0000 7f7ee7fff640 1 -- 172.21.15.42:0/285916117 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f7ef409f610 con 0x7f7ed80036c0 2024-09-16T03:52:17.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:17.737+0000 7f7ee7fff640 10 monclient(hunting): _renew_subs 2024-09-16T03:52:17.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:17.737+0000 7f7efa47a640 1 -- 172.21.15.42:0/285916117 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7ed80036c0 msgr2=0x7f7ed8005af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T03:52:17.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:17.737+0000 7f7efa47a640 1 --2- 172.21.15.42:0/285916117 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7ed80036c0 0x7f7ed8005af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:52:17.945 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:17.941+0000 7f7efa47a640 1 -- 172.21.15.42:0/285916117 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7ed80036c0 msgr2=0x7f7ed8005af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T03:52:17.945 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:17.941+0000 7f7efa47a640 1 --2- 172.21.15.42:0/285916117 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7ed80036c0 0x7f7ed8005af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T03:52:18.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:18.345+0000 7f7efa47a640 1 -- 172.21.15.42:0/285916117 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7ed80036c0 msgr2=0x7f7ed8005af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T03:52:18.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:18.345+0000 7f7efa47a640 1 --2- 172.21.15.42:0/285916117 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7ed80036c0 0x7f7ed8005af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T03:52:19.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.149+0000 7f7efa47a640 1 -- 172.21.15.42:0/285916117 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7ed80036c0 msgr2=0x7f7ed8005af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T03:52:19.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.149+0000 7f7efa47a640 1 --2- 172.21.15.42:0/285916117 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7ed80036c0 0x7f7ed8005af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T03:52:19.670 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.665+0000 7f7ef9478640 1 -- 172.21.15.42:0/285916117 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 249404343 0 0) 0x7f7ef409fbd0 con 0x7f7ed8006180 2024-09-16T03:52:19.670 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.665+0000 7f7ef9478640 10 monclient(hunting): _negotiate 2024-09-16T03:52:19.670 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.665+0000 7f7ef9478640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:52:19.670 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.665+0000 7f7ef9478640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:52:19.670 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.665+0000 7f7ef9478640 10 monclient(hunting): my global_id is 794440 2024-09-16T03:52:19.670 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.665+0000 7f7ef9478640 1 -- 172.21.15.42:0/285916117 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7ee0003530 con 0x7f7ed8006180 2024-09-16T03:52:19.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.665+0000 7f7ef9478640 1 -- 172.21.15.42:0/285916117 <== mon.1 v1:172.21.15.154:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3726327432 0 0) 0x7f7ee0003530 con 0x7f7ed8006180 2024-09-16T03:52:19.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.665+0000 7f7ef9478640 1 -- 172.21.15.42:0/285916117 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7ed80036c0 msgr2=0x7f7ed8005af0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:52:19.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.665+0000 7f7ef9478640 1 --2- 172.21.15.42:0/285916117 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7ed80036c0 0x7f7ed8005af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:52:19.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.665+0000 7f7ef9478640 10 monclient: _finish_hunting 0 2024-09-16T03:52:19.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.665+0000 7f7ef9478640 1 monclient: found mon.noname-c-legacy 2024-09-16T03:52:19.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.665+0000 7f7ef9478640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:52:19.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.665+0000 7f7ef9478640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.154:6789/0 2024-09-16T03:52:19.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.665+0000 7f7ef9478640 1 -- 172.21.15.42:0/285916117 --> v1:172.21.15.154:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ed800afc0 con 0x7f7ed8006180 2024-09-16T03:52:19.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.665+0000 7f7ef9478640 10 monclient: _finish_auth 0 2024-09-16T03:52:19.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.665+0000 7f7ef9478640 10 monclient: _check_auth_tickets 2024-09-16T03:52:19.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.665+0000 7f7ef9478640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:52:19.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.665+0000 7f7ef9478640 1 -- 172.21.15.42:0/285916117 <== mon.1 v1:172.21.15.154:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2177413734 0 0) 0x7f7ee80016c0 con 0x7f7ed8006180 2024-09-16T03:52:19.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.665+0000 7f7efc6de640 5 monclient: authenticate success, global_id 794440 2024-09-16T03:52:19.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.665+0000 7f7efc6de640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:52:19.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.665+0000 7f7ef9478640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:52:19.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.665+0000 7f7ef9478640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-16T03:52:19.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.665+0000 7f7ef9478640 10 monclient: dump: 2024-09-16T03:52:19.674 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:52:19.675 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:52:19.675 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:52:19.675 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:52:19.675 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:52:19.675 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:52:19.675 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:52:19.675 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:52:19.675 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:52:19.675 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:52:19.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.665+0000 7f7ef9478640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] but i'm connected to v1:172.21.15.154:6789/0, reconnecting 2024-09-16T03:52:19.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.665+0000 7f7ef9478640 10 monclient: _reopen_session rank -1 2024-09-16T03:52:19.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.665+0000 7f7ef9478640 1 -- 172.21.15.42:0/285916117 >> v1:172.21.15.154:6789/0 conn(0x7f7ed8006180 legacy=0x7f7ed8008620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:52:19.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.665+0000 7f7efac7b640 1 -- 172.21.15.42:0/285916117 reap_dead start 2024-09-16T03:52:19.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.665+0000 7f7ef9478640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:52:19.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.665+0000 7f7ef9478640 1 --2- 172.21.15.42:0/285916117 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7ee00039c0 0x7f7ee0003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:52:19.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.665+0000 7f7ef9478640 10 monclient(hunting): picked mon.a con 0x7f7ee00039c0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:52:19.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.665+0000 7f7ef9478640 1 --2- 172.21.15.42:0/285916117 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7ee0004810 0x7f7ee0004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:52:19.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.665+0000 7f7ef9478640 10 monclient(hunting): picked mon.c con 0x7f7ee0004810 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:52:19.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.665+0000 7f7ef9478640 1 --2- 172.21.15.42:0/285916117 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7ee00052f0 0x7f7ee000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:52:19.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.665+0000 7f7ef9478640 10 monclient(hunting): picked mon.b con 0x7f7ee00052f0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:52:19.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.665+0000 7f7ef9478640 10 monclient(hunting): start opening mon connection 2024-09-16T03:52:19.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.665+0000 7f7ef9478640 1 -- 172.21.15.42:0/285916117 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f7ef409f610 con 0x7f7ee00039c0 2024-09-16T03:52:19.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.665+0000 7f7ef9478640 10 monclient(hunting): start opening mon connection 2024-09-16T03:52:19.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.665+0000 7f7ef9478640 1 -- 172.21.15.42:0/285916117 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f7ee000c240 con 0x7f7ee00052f0 2024-09-16T03:52:19.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.665+0000 7f7ef9478640 10 monclient(hunting): start opening mon connection 2024-09-16T03:52:19.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.665+0000 7f7ef9478640 1 -- 172.21.15.42:0/285916117 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f7ee000c440 con 0x7f7ee0004810 2024-09-16T03:52:19.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.665+0000 7f7ef9478640 10 monclient(hunting): _renew_subs 2024-09-16T03:52:19.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.665+0000 7f7ef9478640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:52:19.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.665+0000 7f7efc6de640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-16T03:52:19.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.665+0000 7f7ef9c79640 1 -- 172.21.15.42:0/285916117 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7ee0004810 msgr2=0x7f7ee0004c90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T03:52:19.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.665+0000 7f7ef9c79640 1 --2- 172.21.15.42:0/285916117 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7ee0004810 0x7f7ee0004c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:52:19.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.665+0000 7f7efa47a640 1 --2- 172.21.15.42:0/285916117 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7ee00039c0 0x7f7ee0003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:52:19.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.665+0000 7f7efa47a640 10 monclient(hunting): get_auth_request con 0x7f7ee00039c0 auth_method 0 2024-09-16T03:52:19.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.665+0000 7f7efa47a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:52:19.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.665+0000 7f7efa47a640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:52:19.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.665+0000 7f7efa47a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:52:19.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.665+0000 7f7efac7b640 1 --2- 172.21.15.42:0/285916117 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7ee00052f0 0x7f7ee000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:52:19.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.665+0000 7f7efa47a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:52:19.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.665+0000 7f7efa47a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:52:19.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.665+0000 7f7efa47a640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T03:52:19.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.665+0000 7f7efac7b640 10 monclient(hunting): get_auth_request con 0x7f7ee00052f0 auth_method 0 2024-09-16T03:52:19.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.665+0000 7f7efac7b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:52:19.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.665+0000 7f7efac7b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:52:19.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.665+0000 7f7efac7b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:52:19.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.665+0000 7f7efac7b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:52:19.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.665+0000 7f7efac7b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:52:19.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.665+0000 7f7efac7b640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T03:52:19.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.665+0000 7f7efa47a640 10 monclient(hunting): handle_auth_done global_id 794440 payload 293 2024-09-16T03:52:19.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.665+0000 7f7efa47a640 1 -- 172.21.15.42:0/285916117 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7ee0004810 msgr2=0x7f7ee0004c90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:52:19.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.665+0000 7f7efa47a640 1 --2- 172.21.15.42:0/285916117 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7ee0004810 0x7f7ee0004c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:52:19.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.665+0000 7f7efa47a640 1 -- 172.21.15.42:0/285916117 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7ee00052f0 msgr2=0x7f7ee000b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:52:19.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.665+0000 7f7efa47a640 1 --2- 172.21.15.42:0/285916117 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7ee00052f0 0x7f7ee000b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:52:19.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.665+0000 7f7efa47a640 10 monclient: _finish_hunting 0 2024-09-16T03:52:19.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.665+0000 7f7efa47a640 1 monclient: found mon.a 2024-09-16T03:52:19.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.665+0000 7f7efa47a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:52:19.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.665+0000 7f7efa47a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:52:19.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.665+0000 7f7efa47a640 1 -- 172.21.15.42:0/285916117 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ee0003530 con 0x7f7ee00039c0 2024-09-16T03:52:19.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.665+0000 7f7efa47a640 10 monclient: _finish_auth 0 2024-09-16T03:52:19.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.665+0000 7f7efa47a640 10 monclient: _check_auth_tickets 2024-09-16T03:52:19.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.665+0000 7f7efa47a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:52:19.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.669+0000 7f7efa47a640 1 --2- 172.21.15.42:0/285916117 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7ee00039c0 0x7f7ee0003dc0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f7ef0009610 tx=0x7f7ef001d7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=199573a25f5ff05e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:52:19.725 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.717+0000 7f7ef9478640 1 -- 172.21.15.42:0/285916117 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ef0009900 con 0x7f7ee00039c0 2024-09-16T03:52:19.726 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.717+0000 7f7ef9478640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:52:19.726 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.717+0000 7f7ef9478640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:52:19.726 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.717+0000 7f7ef9478640 10 monclient: dump: 2024-09-16T03:52:19.726 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:52:19.726 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:52:19.726 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:52:19.726 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:52:19.726 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:52:19.727 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:52:19.727 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:52:19.727 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:52:19.727 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:52:19.727 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:52:19.727 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.717+0000 7f7efc6de640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:52:19.727 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.721+0000 7f7ef9478640 1 -- 172.21.15.42:0/285916117 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ef00b1070 con 0x7f7ee00039c0 2024-09-16T03:52:19.728 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.721+0000 7f7ef9478640 10 monclient: handle_config config(0 keys) 2024-09-16T03:52:19.728 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.721+0000 7f7ef9478640 1 -- 172.21.15.42:0/285916117 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ef00ac890 con 0x7f7ee00039c0 2024-09-16T03:52:19.728 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.721+0000 7f7ef9478640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:52:19.728 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.721+0000 7f7ef9478640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:52:19.728 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.721+0000 7f7ef9478640 10 monclient: dump: 2024-09-16T03:52:19.729 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:52:19.729 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:52:19.729 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:52:19.729 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:52:19.729 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:52:19.729 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:52:19.729 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:52:19.730 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:52:19.730 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:52:19.730 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:52:19.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.721+0000 7f7efc6de640 10 monclient: get_monmap_and_config success 2024-09-16T03:52:19.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.721+0000 7f7efc6de640 4 set_mon_vals no callback set 2024-09-16T03:52:19.736 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.721+0000 7f7efc6de640 10 monclient: shutdown 2024-09-16T03:52:19.736 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.721+0000 7f7efc6de640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:52:19.736 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.721+0000 7f7efc6de640 1 -- 172.21.15.42:0/285916117 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7ee00039c0 msgr2=0x7f7ee0003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:52:19.736 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.721+0000 7f7efc6de640 1 --2- 172.21.15.42:0/285916117 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7ee00039c0 0x7f7ee0003dc0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f7ef0009610 tx=0x7f7ef001d7f0 comp rx=0 tx=0).stop 2024-09-16T03:52:19.736 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.721+0000 7f7efc6de640 1 -- 172.21.15.42:0/285916117 shutdown_connections 2024-09-16T03:52:19.736 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.721+0000 7f7efc6de640 1 --2- 172.21.15.42:0/285916117 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7ee00052f0 0x7f7ee000b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:52:19.737 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.721+0000 7f7efc6de640 1 --2- 172.21.15.42:0/285916117 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7ee0004810 0x7f7ee0004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:52:19.737 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.721+0000 7f7efc6de640 1 --2- 172.21.15.42:0/285916117 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7ee00039c0 0x7f7ee0003dc0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:52:19.737 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.721+0000 7f7efc6de640 1 -- 172.21.15.42:0/285916117 >> 172.21.15.42:0/285916117 conn(0x7f7ef40a2420 msgr2=0x7f7ef40a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:52:19.737 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.721+0000 7f7efc6de640 1 -- 172.21.15.42:0/285916117 shutdown_connections 2024-09-16T03:52:19.737 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.721+0000 7f7efc6de640 1 -- 172.21.15.42:0/285916117 wait complete. 2024-09-16T03:52:19.737 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.721+0000 7f7efc6de640 10 monclient: build_initial_monmap 2024-09-16T03:52:19.737 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.721+0000 7f7efc6de640 10 monclient: monmap: 2024-09-16T03:52:19.737 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:52:19.737 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:52:19.737 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:52:19.737 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:52:19.738 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:52:19.738 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:52:19.738 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:52:19.738 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:52:19.738 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:52:19.738 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:52:19.738 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.721+0000 7f7efc6de640 1 librados: starting msgr at 2024-09-16T03:52:19.738 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.721+0000 7f7efc6de640 1 librados: starting objecter 2024-09-16T03:52:19.738 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.725+0000 7f7efc6de640 1 Processor -- start 2024-09-16T03:52:19.738 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.725+0000 7f7efc6de640 1 -- start start 2024-09-16T03:52:19.738 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.725+0000 7f7efc6de640 1 librados: setting wanted keys 2024-09-16T03:52:19.738 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.725+0000 7f7efc6de640 1 librados: calling monclient init 2024-09-16T03:52:19.738 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.725+0000 7f7efc6de640 10 monclient: init 2024-09-16T03:52:19.738 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.725+0000 7f7efc6de640 10 monclient: _reopen_session rank -1 2024-09-16T03:52:19.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.725+0000 7f7efc6de640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:52:19.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.725+0000 7f7efc6de640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7ee00039c0 0x7f7ef41cc390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:52:19.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.725+0000 7f7efc6de640 10 monclient(hunting): picked mon.noname-c con 0x7f7ee00039c0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:52:19.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.725+0000 7f7efc6de640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7ee0004810 0x7f7ef41cc8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:52:19.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.725+0000 7f7efc6de640 10 monclient(hunting): picked mon.noname-a con 0x7f7ee0004810 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:52:19.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.725+0000 7f7efc6de640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7ee00052f0 0x7f7ef41217c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:52:19.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.725+0000 7f7efc6de640 10 monclient(hunting): picked mon.noname-b con 0x7f7ee00052f0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:52:19.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.725+0000 7f7efc6de640 10 monclient(hunting): start opening mon connection 2024-09-16T03:52:19.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.725+0000 7f7efc6de640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f7ef409eef0 con 0x7f7ee0004810 2024-09-16T03:52:19.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.725+0000 7f7efc6de640 10 monclient(hunting): start opening mon connection 2024-09-16T03:52:19.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.725+0000 7f7efc6de640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f7ef409ed70 con 0x7f7ee00052f0 2024-09-16T03:52:19.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.725+0000 7f7efc6de640 10 monclient(hunting): start opening mon connection 2024-09-16T03:52:19.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.725+0000 7f7efc6de640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f7ef4135100 con 0x7f7ee00039c0 2024-09-16T03:52:19.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.725+0000 7f7efc6de640 10 monclient(hunting): _renew_subs 2024-09-16T03:52:19.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.725+0000 7f7efc6de640 10 monclient(hunting): authenticate will time out at 1586.697075s 2024-09-16T03:52:19.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.725+0000 7f7efa47a640 1 -- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7ee00039c0 msgr2=0x7f7ef41cc390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T03:52:19.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.725+0000 7f7efa47a640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7ee00039c0 0x7f7ef41cc390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:52:19.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.725+0000 7f7ef9c79640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7ee0004810 0x7f7ef41cc8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:52:19.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.725+0000 7f7ef9c79640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7ee0004810 0x7f7ef41cc8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:47594/0 (socket says 172.21.15.42:47594) 2024-09-16T03:52:19.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.725+0000 7f7ef9c79640 1 -- 172.21.15.42:0/4045147578 learned_addr learned my addr 172.21.15.42:0/4045147578 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:52:19.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.725+0000 7f7ef9c79640 10 monclient(hunting): get_auth_request con 0x7f7ee0004810 auth_method 0 2024-09-16T03:52:19.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.725+0000 7f7ef9c79640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:52:19.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.725+0000 7f7ef9c79640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:52:19.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.725+0000 7f7ef9c79640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:52:19.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.725+0000 7f7ef9c79640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:52:19.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.725+0000 7f7ef9c79640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:52:19.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.725+0000 7f7ef9c79640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:52:19.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.725+0000 7f7ef9c79640 10 monclient(hunting): handle_auth_done global_id 824169 payload 824 2024-09-16T03:52:19.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.725+0000 7f7ef9c79640 1 -- 172.21.15.42:0/4045147578 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7ee00039c0 msgr2=0x7f7ef41cc390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:52:19.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.725+0000 7f7ef9c79640 1 --2- 172.21.15.42:0/4045147578 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7ee00039c0 0x7f7ef41cc390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:52:19.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.725+0000 7f7ef9c79640 1 -- 172.21.15.42:0/4045147578 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7ee00052f0 msgr2=0x7f7ef41217c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:52:19.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.725+0000 7f7ef9c79640 1 --2- 172.21.15.42:0/4045147578 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7ee00052f0 0x7f7ef41217c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:52:19.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.725+0000 7f7ef9c79640 10 monclient: _finish_hunting 0 2024-09-16T03:52:19.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.725+0000 7f7ef9c79640 1 monclient: found mon.noname-a 2024-09-16T03:52:19.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.725+0000 7f7ef9c79640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:52:19.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.725+0000 7f7ef9c79640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:52:19.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.725+0000 7f7ef9c79640 1 -- 172.21.15.42:0/4045147578 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ef4125190 con 0x7f7ee0004810 2024-09-16T03:52:19.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.725+0000 7f7ef9c79640 1 --2- 172.21.15.42:0/4045147578 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7ee0004810 0x7f7ef41cc8d0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f7ee800d460 tx=0x7f7ee800d920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:52:19.742 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.725+0000 7f7ee6ffd640 10 client.?.objecter ms_handle_connect 0x7f7ee0004810 2024-09-16T03:52:19.742 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.725+0000 7f7ee6ffd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:52:19.742 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.725+0000 7f7ee6ffd640 1 -- 172.21.15.42:0/4045147578 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ee8019070 con 0x7f7ee0004810 2024-09-16T03:52:19.742 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.725+0000 7f7ee6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:52:19.742 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.725+0000 7f7ee6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:52:19.742 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.725+0000 7f7ee6ffd640 10 monclient: dump: 2024-09-16T03:52:19.742 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:52:19.742 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:52:19.742 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:52:19.742 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:52:19.742 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:52:19.742 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:52:19.742 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:52:19.743 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:52:19.743 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:52:19.743 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:52:19.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.725+0000 7f7ee6ffd640 10 monclient: _finish_auth 0 2024-09-16T03:52:19.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.725+0000 7f7ee6ffd640 10 monclient: _check_auth_tickets 2024-09-16T03:52:19.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.725+0000 7f7ee6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:52:19.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.725+0000 7f7ee6ffd640 1 -- 172.21.15.42:0/4045147578 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ee8005cd0 con 0x7f7ee0004810 2024-09-16T03:52:19.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.725+0000 7f7efc6de640 5 monclient: authenticate success, global_id 824169 2024-09-16T03:52:19.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.725+0000 7f7efc6de640 10 monclient: _renew_subs 2024-09-16T03:52:19.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.725+0000 7f7efc6de640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:52:19.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.725+0000 7f7efc6de640 1 -- 172.21.15.42:0/4045147578 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ef4071160 con 0x7f7ee0004810 2024-09-16T03:52:19.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.725+0000 7f7ee6ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:52:19.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.725+0000 7f7ee6ffd640 1 -- 172.21.15.42:0/4045147578 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ee8008310 con 0x7f7ee0004810 2024-09-16T03:52:19.744 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.725+0000 7f7ee6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:52:19.744 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.725+0000 7f7ee6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:52:19.744 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.725+0000 7f7ee6ffd640 10 monclient: dump: 2024-09-16T03:52:19.744 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:52:19.744 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:52:19.744 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:52:19.744 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:52:19.744 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:52:19.744 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:52:19.744 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:52:19.744 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:52:19.744 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:52:19.744 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:52:19.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.725+0000 7f7efc6de640 10 client.824169.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:52:19.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.725+0000 7f7efc6de640 10 monclient: _renew_subs 2024-09-16T03:52:19.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.725+0000 7f7efc6de640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:52:19.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.725+0000 7f7efc6de640 1 -- 172.21.15.42:0/4045147578 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ef4123fc0 con 0x7f7ee0004810 2024-09-16T03:52:19.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.725+0000 7f7efc6de640 1 librados: init done 2024-09-16T03:52:19.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.725+0000 7f7ef8c77640 4 set_mon_vals no callback set 2024-09-16T03:52:19.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.725+0000 7f7efc6de640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:52:19.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.725+0000 7f7efc6de640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:52:19.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.725+0000 7f7efc6de640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:52:19.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.725+0000 7f7efc6de640 1 -- 172.21.15.42:0/4045147578 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ef40a5050 con 0x7f7ee0004810 2024-09-16T03:52:19.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.725+0000 7f7ee6ffd640 1 -- 172.21.15.42:0/4045147578 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f7ee801c070 con 0x7f7ee0004810 2024-09-16T03:52:19.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.725+0000 7f7ee6ffd640 1 --2- 172.21.15.42:0/4045147578 >> v2:172.21.15.154:6812/254845342 conn(0x7f7ed8037ff0 0x7f7ed803a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:52:19.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.725+0000 7f7ee6ffd640 1 -- 172.21.15.42:0/4045147578 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(332..332 src has 286..332) ==== 5328+0+0 (secure 0 0 0) 0x7f7ee8049120 con 0x7f7ee0004810 2024-09-16T03:52:19.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.725+0000 7f7ee6ffd640 10 client.824169.objecter ms_dispatch 0x7f7ef4000ba0 osd_map(332..332 src has 286..332) 2024-09-16T03:52:19.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.725+0000 7f7ee6ffd640 3 client.824169.objecter handle_osd_map got epochs [332,332] > 0 2024-09-16T03:52:19.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.725+0000 7f7ee6ffd640 3 client.824169.objecter handle_osd_map decoding full epoch 332 2024-09-16T03:52:19.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.725+0000 7f7ee6ffd640 20 client.824169.objecter dump_active .. 0 homeless 2024-09-16T03:52:19.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.729+0000 7f7efa47a640 1 --2- 172.21.15.42:0/4045147578 >> v2:172.21.15.154:6812/254845342 conn(0x7f7ed8037ff0 0x7f7ed803a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:52:19.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.729+0000 7f7efa47a640 10 monclient: get_auth_request con 0x7f7ed8037ff0 auth_method 0 2024-09-16T03:52:19.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.729+0000 7f7ee6ffd640 10 client.824169.objecter ms_handle_connect 0x7f7ed8037ff0 2024-09-16T03:52:19.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.729+0000 7f7efa47a640 1 --2- 172.21.15.42:0/4045147578 >> v2:172.21.15.154:6812/254845342 conn(0x7f7ed8037ff0 0x7f7ed803a4b0 secure :-1 s=READY pgs=3576 cs=0 l=1 rev1=1 crypto rx=0x7f7ef00b0000 tx=0x7f7ef0004ac0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:52:19.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.733+0000 7f7ee6ffd640 1 -- 172.21.15.42:0/4045147578 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f7ee8012980 con 0x7f7ee0004810 2024-09-16T03:52:19.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.733+0000 7f7ee6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:52:19.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.733+0000 7f7ee6ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:52:19.967 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.961+0000 7f7efc6de640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:52:19.967 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.961+0000 7f7efc6de640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:52:19.967 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.961+0000 7f7efc6de640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:52:19.967 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.961+0000 7f7efc6de640 1 -- 172.21.15.42:0/4045147578 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f7ef409f8f0 con 0x7f7ee0004810 2024-09-16T03:52:19.967 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.961+0000 7f7ee6ffd640 1 -- 172.21.15.42:0/4045147578 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1554 (secure 0 0 0) 0x7f7ee8012be0 con 0x7f7ee0004810 2024-09-16T03:52:19.967 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.961+0000 7f7ee6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:52:19.968 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.961+0000 7f7ee6ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:52:19.968 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":90,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:52:19.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.965+0000 7f7efc6de640 10 librados: watch_flush enter 2024-09-16T03:52:19.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.965+0000 7f7efc6de640 10 librados: watch_flush exit 2024-09-16T03:52:19.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.965+0000 7f7efc6de640 20 client.824169.objecter shutdown clearing up homeless session... 2024-09-16T03:52:19.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.965+0000 7f7efc6de640 10 client.824169.objecter successfully canceled tick 2024-09-16T03:52:19.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.965+0000 7f7efc6de640 1 -- 172.21.15.42:0/4045147578 >> v2:172.21.15.154:6812/254845342 conn(0x7f7ed8037ff0 msgr2=0x7f7ed803a4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:52:19.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.965+0000 7f7efc6de640 1 --2- 172.21.15.42:0/4045147578 >> v2:172.21.15.154:6812/254845342 conn(0x7f7ed8037ff0 0x7f7ed803a4b0 secure :-1 s=READY pgs=3576 cs=0 l=1 rev1=1 crypto rx=0x7f7ef00b0000 tx=0x7f7ef0004ac0 comp rx=0 tx=0).stop 2024-09-16T03:52:19.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.965+0000 7f7efc6de640 10 monclient: shutdown 2024-09-16T03:52:19.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.965+0000 7f7efc6de640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:52:19.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.965+0000 7f7efc6de640 1 -- 172.21.15.42:0/4045147578 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7ee0004810 msgr2=0x7f7ef41cc8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:52:19.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.965+0000 7f7efc6de640 1 --2- 172.21.15.42:0/4045147578 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7ee0004810 0x7f7ef41cc8d0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f7ee800d460 tx=0x7f7ee800d920 comp rx=0 tx=0).stop 2024-09-16T03:52:19.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.965+0000 7f7efc6de640 1 -- 172.21.15.42:0/4045147578 shutdown_connections 2024-09-16T03:52:19.971 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.965+0000 7f7efc6de640 1 --2- 172.21.15.42:0/4045147578 >> v2:172.21.15.154:6812/254845342 conn(0x7f7ed8037ff0 0x7f7ed803a4b0 unknown :-1 s=CLOSED pgs=3576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:52:19.971 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.965+0000 7f7efc6de640 1 --2- 172.21.15.42:0/4045147578 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7ee00052f0 0x7f7ef41217c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:52:19.971 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.965+0000 7f7efc6de640 1 --2- 172.21.15.42:0/4045147578 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7ee0004810 0x7f7ef41cc8d0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:52:19.971 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.965+0000 7f7efc6de640 1 --2- 172.21.15.42:0/4045147578 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7ee00039c0 0x7f7ef41cc390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:52:19.971 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.965+0000 7f7efc6de640 1 -- 172.21.15.42:0/4045147578 >> 172.21.15.42:0/4045147578 conn(0x7f7ef40a2420 msgr2=0x7f7ef40a4040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:52:19.971 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.965+0000 7f7efc6de640 1 -- 172.21.15.42:0/4045147578 shutdown_connections 2024-09-16T03:52:19.971 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.965+0000 7f7efc6de640 1 -- 172.21.15.42:0/4045147578 wait complete. 2024-09-16T03:52:19.971 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:19.965+0000 7f7efc6de640 1 librados: shutdown 2024-09-16T03:52:19.987 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T03:52:19.987 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-16T03:52:19.988 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T03:52:20.114 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.109+0000 7f65762bd640 10 monclient: get_monmap_and_config 2024-09-16T03:52:20.114 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.109+0000 7f65762bd640 10 monclient: build_initial_monmap 2024-09-16T03:52:20.115 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.109+0000 7f65762bd640 10 monclient: monmap: 2024-09-16T03:52:20.115 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:52:20.115 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:52:20.115 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:52:20.113312+0000 2024-09-16T03:52:20.115 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:52:20.113312+0000 2024-09-16T03:52:20.115 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:52:20.115 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:52:20.115 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:52:20.115 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:52:20.115 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:52:20.115 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:52:20.115 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:52:20.115 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:52:20.118 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.109+0000 7f65762bd640 1 Processor -- start 2024-09-16T03:52:20.118 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.109+0000 7f65762bd640 1 -- start start 2024-09-16T03:52:20.118 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.109+0000 7f65762bd640 10 monclient: init 2024-09-16T03:52:20.118 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.109+0000 7f65762bd640 10 monclient: _reopen_session rank -1 2024-09-16T03:52:20.118 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.109+0000 7f65762bd640 10 monclient: _add_conns ranks=[3,4,1,2,0] 2024-09-16T03:52:20.118 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.109+0000 7f65762bd640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f657009fb00 0x7f657009ba50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:52:20.119 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.109+0000 7f65762bd640 10 monclient(hunting): picked mon.noname-a con 0x7f657009fb00 addr v2:172.21.15.42:3300/0 2024-09-16T03:52:20.119 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.109+0000 7f65762bd640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f657009bf90 0x7f657009c400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:52:20.119 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.109+0000 7f65762bd640 10 monclient(hunting): picked mon.noname-c con 0x7f657009bf90 addr v2:172.21.15.154:3300/0 2024-09-16T03:52:20.119 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.109+0000 7f65762bd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6570138110 addr v1:172.21.15.154:6789/0 2024-09-16T03:52:20.119 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.109+0000 7f65762bd640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f65700a1e20 con 0x7f6570138110 2024-09-16T03:52:20.119 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.109+0000 7f65762bd640 10 monclient(hunting): start opening mon connection 2024-09-16T03:52:20.119 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.109+0000 7f65762bd640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f657009e250 con 0x7f657009fb00 2024-09-16T03:52:20.119 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.109+0000 7f65762bd640 10 monclient(hunting): start opening mon connection 2024-09-16T03:52:20.119 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.109+0000 7f65762bd640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f657009ea30 con 0x7f657009bf90 2024-09-16T03:52:20.119 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.109+0000 7f65762bd640 10 monclient(hunting): _renew_subs 2024-09-16T03:52:20.119 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.109+0000 7f65762bd640 10 monclient(hunting): authenticate will time out at 1587.082963s 2024-09-16T03:52:20.119 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.109+0000 7f656ffff640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f657009fb00 0x7f657009ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:52:20.120 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.109+0000 7f656ffff640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f657009fb00 0x7f657009ba50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:47598/0 (socket says 172.21.15.42:47598) 2024-09-16T03:52:20.120 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.109+0000 7f656ffff640 1 -- 172.21.15.42:0/3900144926 learned_addr learned my addr 172.21.15.42:0/3900144926 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:52:20.120 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.109+0000 7f656ffff640 10 monclient(hunting): get_auth_request con 0x7f657009fb00 auth_method 0 2024-09-16T03:52:20.120 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.109+0000 7f656ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:52:20.120 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.109+0000 7f656ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:52:20.120 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.109+0000 7f656ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:52:20.120 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.109+0000 7f656ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:52:20.120 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.109+0000 7f656ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:52:20.120 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.109+0000 7f656ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:52:20.120 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.109+0000 7f656f7fe640 1 --2- 172.21.15.42:0/3900144926 >> v2:172.21.15.154:3300/0 conn(0x7f657009bf90 0x7f657009c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:52:20.120 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.109+0000 7f656f7fe640 10 monclient(hunting): get_auth_request con 0x7f657009bf90 auth_method 0 2024-09-16T03:52:20.120 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.109+0000 7f656f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:52:20.120 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.109+0000 7f656f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:52:20.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.109+0000 7f656f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:52:20.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.109+0000 7f656ffff640 10 monclient(hunting): handle_auth_done global_id 824172 payload 274 2024-09-16T03:52:20.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.109+0000 7f656ffff640 1 -- 172.21.15.42:0/3900144926 >> v2:172.21.15.154:3300/0 conn(0x7f657009bf90 msgr2=0x7f657009c400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:52:20.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.109+0000 7f656ffff640 1 --2- 172.21.15.42:0/3900144926 >> v2:172.21.15.154:3300/0 conn(0x7f657009bf90 0x7f657009c400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:52:20.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.109+0000 7f656ffff640 1 -- 172.21.15.42:0/3900144926 >> v1:172.21.15.154:6789/0 conn(0x7f6570138110 legacy=0x7f657013a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:52:20.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.109+0000 7f656ffff640 10 monclient: _finish_hunting 0 2024-09-16T03:52:20.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.109+0000 7f656ffff640 1 monclient: found mon.noname-a 2024-09-16T03:52:20.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.109+0000 7f656ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:52:20.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.109+0000 7f656ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:52:20.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.109+0000 7f656ffff640 1 -- 172.21.15.42:0/3900144926 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f657009fee0 con 0x7f657009fb00 2024-09-16T03:52:20.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.113+0000 7f656ffff640 1 --2- 172.21.15.42:0/3900144926 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f657009fb00 0x7f657009ba50 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f657009e3d0 tx=0x7f656001e870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de8c670bad54623d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:52:20.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.113+0000 7f656effd640 1 -- 172.21.15.42:0/3900144926 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65600ad070 con 0x7f657009fb00 2024-09-16T03:52:20.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.113+0000 7f656effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:52:20.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.113+0000 7f656effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:52:20.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.113+0000 7f656effd640 10 monclient: dump: 2024-09-16T03:52:20.129 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:52:20.129 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:52:20.129 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:52:20.129 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:52:20.129 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:52:20.129 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:52:20.129 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:52:20.129 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:52:20.130 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:52:20.130 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:52:20.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.113+0000 7f656effd640 10 monclient: _finish_auth 0 2024-09-16T03:52:20.131 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.113+0000 7f656effd640 10 monclient: _check_auth_tickets 2024-09-16T03:52:20.131 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.113+0000 7f656effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:52:20.132 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.113+0000 7f656effd640 1 -- 172.21.15.42:0/3900144926 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6560004230 con 0x7f657009fb00 2024-09-16T03:52:20.132 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.113+0000 7f656effd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:52:20.132 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.113+0000 7f656effd640 1 -- 172.21.15.42:0/3900144926 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65600a76e0 con 0x7f657009fb00 2024-09-16T03:52:20.132 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.113+0000 7f65762bd640 5 monclient: authenticate success, global_id 824172 2024-09-16T03:52:20.132 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.113+0000 7f65762bd640 10 monclient: get_monmap_and_config success 2024-09-16T03:52:20.132 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.113+0000 7f65762bd640 4 set_mon_vals no callback set 2024-09-16T03:52:20.132 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.113+0000 7f656effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:52:20.132 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.113+0000 7f656effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:52:20.132 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.113+0000 7f656effd640 10 monclient: dump: 2024-09-16T03:52:20.132 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:52:20.132 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:52:20.132 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:52:20.132 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:52:20.133 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:52:20.133 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:52:20.133 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:52:20.133 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:52:20.133 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:52:20.133 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:52:20.133 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.113+0000 7f65762bd640 10 monclient: shutdown 2024-09-16T03:52:20.133 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.113+0000 7f65762bd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:52:20.133 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.113+0000 7f65762bd640 1 -- 172.21.15.42:0/3900144926 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f657009fb00 msgr2=0x7f657009ba50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:52:20.133 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.113+0000 7f65762bd640 1 --2- 172.21.15.42:0/3900144926 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f657009fb00 0x7f657009ba50 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f657009e3d0 tx=0x7f656001e870 comp rx=0 tx=0).stop 2024-09-16T03:52:20.133 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.113+0000 7f65762bd640 1 -- 172.21.15.42:0/3900144926 shutdown_connections 2024-09-16T03:52:20.133 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.113+0000 7f65762bd640 1 --2- 172.21.15.42:0/3900144926 >> v2:172.21.15.154:3300/0 conn(0x7f657009bf90 0x7f657009c400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:52:20.133 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.113+0000 7f65762bd640 1 --2- 172.21.15.42:0/3900144926 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f657009fb00 0x7f657009ba50 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:52:20.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.113+0000 7f65762bd640 1 -- 172.21.15.42:0/3900144926 >> 172.21.15.42:0/3900144926 conn(0x7f65700a2420 msgr2=0x7f65700a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:52:20.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.113+0000 7f65762bd640 1 -- 172.21.15.42:0/3900144926 shutdown_connections 2024-09-16T03:52:20.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.113+0000 7f65762bd640 1 -- 172.21.15.42:0/3900144926 wait complete. 2024-09-16T03:52:20.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.117+0000 7f65762bd640 10 monclient: build_initial_monmap 2024-09-16T03:52:20.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.117+0000 7f65762bd640 10 monclient: monmap: 2024-09-16T03:52:20.134 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:52:20.134 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:52:20.134 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:52:20.134 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:52:20.134 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:52:20.134 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:52:20.134 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:52:20.134 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:52:20.135 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:52:20.135 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:52:20.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.117+0000 7f65762bd640 1 librados: starting msgr at 2024-09-16T03:52:20.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.117+0000 7f65762bd640 1 librados: starting objecter 2024-09-16T03:52:20.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.117+0000 7f65762bd640 1 Processor -- start 2024-09-16T03:52:20.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.117+0000 7f65762bd640 1 -- start start 2024-09-16T03:52:20.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.117+0000 7f65762bd640 1 librados: setting wanted keys 2024-09-16T03:52:20.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.117+0000 7f65762bd640 1 librados: calling monclient init 2024-09-16T03:52:20.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.117+0000 7f65762bd640 10 monclient: init 2024-09-16T03:52:20.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.117+0000 7f65762bd640 10 monclient: _reopen_session rank -1 2024-09-16T03:52:20.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.117+0000 7f65762bd640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T03:52:20.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.117+0000 7f65762bd640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f657009bf90 0x7f6570137590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:52:20.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.117+0000 7f65762bd640 10 monclient(hunting): picked mon.noname-a con 0x7f657009bf90 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:52:20.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.117+0000 7f65762bd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f657009fb00 0x7f6570137ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:52:20.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.117+0000 7f65762bd640 10 monclient(hunting): picked mon.noname-b con 0x7f657009fb00 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:52:20.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.117+0000 7f65762bd640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f6570138110 0x7f6570122010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:52:20.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.117+0000 7f65762bd640 10 monclient(hunting): picked mon.noname-c con 0x7f6570138110 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:52:20.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.117+0000 7f65762bd640 10 monclient(hunting): start opening mon connection 2024-09-16T03:52:20.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.117+0000 7f65762bd640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f657009d380 con 0x7f657009bf90 2024-09-16T03:52:20.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.117+0000 7f65762bd640 10 monclient(hunting): start opening mon connection 2024-09-16T03:52:20.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.117+0000 7f65762bd640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f657009d140 con 0x7f657009fb00 2024-09-16T03:52:20.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.117+0000 7f65762bd640 10 monclient(hunting): start opening mon connection 2024-09-16T03:52:20.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.117+0000 7f65762bd640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f657009d500 con 0x7f6570138110 2024-09-16T03:52:20.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.117+0000 7f65762bd640 10 monclient(hunting): _renew_subs 2024-09-16T03:52:20.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.117+0000 7f65762bd640 10 monclient(hunting): authenticate will time out at 1587.089137s 2024-09-16T03:52:20.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.117+0000 7f657485a640 1 -- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f6570138110 msgr2=0x7f6570122010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T03:52:20.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.117+0000 7f657485a640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f6570138110 0x7f6570122010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:52:20.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.117+0000 7f656ffff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f657009bf90 0x7f6570137590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:52:20.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.117+0000 7f656ffff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f657009bf90 0x7f6570137590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:47608/0 (socket says 172.21.15.42:47608) 2024-09-16T03:52:20.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.117+0000 7f656ffff640 1 -- 172.21.15.42:0/704093038 learned_addr learned my addr 172.21.15.42:0/704093038 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:52:20.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.117+0000 7f656ffff640 10 monclient(hunting): get_auth_request con 0x7f657009bf90 auth_method 0 2024-09-16T03:52:20.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.117+0000 7f656ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:52:20.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.117+0000 7f656ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:52:20.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.117+0000 7f656ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:52:20.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.117+0000 7f656ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:52:20.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.117+0000 7f656ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:52:20.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.117+0000 7f656ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:52:20.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.117+0000 7f656f7fe640 1 --2- 172.21.15.42:0/704093038 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f657009fb00 0x7f6570137ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:52:20.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.117+0000 7f656f7fe640 10 monclient(hunting): get_auth_request con 0x7f657009fb00 auth_method 0 2024-09-16T03:52:20.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.117+0000 7f656f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:52:20.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.117+0000 7f656f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:52:20.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.117+0000 7f656f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:52:20.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.117+0000 7f656ffff640 10 monclient(hunting): handle_auth_done global_id 824175 payload 824 2024-09-16T03:52:20.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.117+0000 7f656ffff640 1 -- 172.21.15.42:0/704093038 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f6570138110 msgr2=0x7f6570122010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:52:20.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.117+0000 7f656ffff640 1 --2- 172.21.15.42:0/704093038 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f6570138110 0x7f6570122010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:52:20.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.117+0000 7f656ffff640 1 -- 172.21.15.42:0/704093038 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f657009fb00 msgr2=0x7f6570137ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:52:20.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.117+0000 7f656ffff640 1 --2- 172.21.15.42:0/704093038 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f657009fb00 0x7f6570137ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:52:20.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.117+0000 7f656ffff640 10 monclient: _finish_hunting 0 2024-09-16T03:52:20.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.117+0000 7f656ffff640 1 monclient: found mon.noname-a 2024-09-16T03:52:20.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.117+0000 7f656ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:52:20.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.117+0000 7f656ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:52:20.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.117+0000 7f656ffff640 1 -- 172.21.15.42:0/704093038 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6570125f60 con 0x7f657009bf90 2024-09-16T03:52:20.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.117+0000 7f656ffff640 1 --2- 172.21.15.42:0/704093038 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f657009bf90 0x7f6570137590 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f6560024e80 tx=0x7f656001b610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:52:20.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.117+0000 7f656cff9640 10 client.?.objecter ms_handle_connect 0x7f657009bf90 2024-09-16T03:52:20.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.117+0000 7f656cff9640 10 client.?.objecter resend_mon_ops 2024-09-16T03:52:20.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.117+0000 7f656cff9640 1 -- 172.21.15.42:0/704093038 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65600be070 con 0x7f657009bf90 2024-09-16T03:52:20.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.117+0000 7f656cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:52:20.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.117+0000 7f656cff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:52:20.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.117+0000 7f656cff9640 10 monclient: dump: 2024-09-16T03:52:20.139 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:52:20.140 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:52:20.140 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:52:20.140 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:52:20.140 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:52:20.140 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:52:20.140 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:52:20.140 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:52:20.140 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:52:20.140 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:52:20.140 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.117+0000 7f656cff9640 10 monclient: _finish_auth 0 2024-09-16T03:52:20.140 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.117+0000 7f656cff9640 10 monclient: _check_auth_tickets 2024-09-16T03:52:20.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.117+0000 7f656cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:52:20.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.117+0000 7f656cff9640 1 -- 172.21.15.42:0/704093038 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6560004030 con 0x7f657009bf90 2024-09-16T03:52:20.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.117+0000 7f656cff9640 10 monclient: handle_config config(0 keys) 2024-09-16T03:52:20.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.117+0000 7f656cff9640 1 -- 172.21.15.42:0/704093038 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65600ad040 con 0x7f657009bf90 2024-09-16T03:52:20.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.117+0000 7f65762bd640 5 monclient: authenticate success, global_id 824175 2024-09-16T03:52:20.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.117+0000 7f65762bd640 10 monclient: _renew_subs 2024-09-16T03:52:20.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.117+0000 7f65762bd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:52:20.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.117+0000 7f65762bd640 1 -- 172.21.15.42:0/704093038 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65701388a0 con 0x7f657009bf90 2024-09-16T03:52:20.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.117+0000 7f656e7fc640 4 set_mon_vals no callback set 2024-09-16T03:52:20.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.117+0000 7f656cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:52:20.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.117+0000 7f656cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:52:20.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.117+0000 7f656cff9640 10 monclient: dump: 2024-09-16T03:52:20.142 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:52:20.142 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:52:20.142 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:52:20.142 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:52:20.142 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:52:20.142 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:52:20.142 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:52:20.142 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:52:20.142 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:52:20.142 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:52:20.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.117+0000 7f65762bd640 10 client.824175.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:52:20.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.117+0000 7f65762bd640 10 monclient: _renew_subs 2024-09-16T03:52:20.143 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.117+0000 7f65762bd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:52:20.143 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.117+0000 7f65762bd640 1 -- 172.21.15.42:0/704093038 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f657013b2c0 con 0x7f657009bf90 2024-09-16T03:52:20.143 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.117+0000 7f65762bd640 1 librados: init done 2024-09-16T03:52:20.143 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.117+0000 7f65762bd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:52:20.143 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.117+0000 7f65762bd640 1 --2- 172.21.15.42:0/704093038 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f65701cf170 0x7f65701258c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:52:20.143 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.117+0000 7f65762bd640 10 monclient(hunting): start opening mon connection 2024-09-16T03:52:20.143 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.117+0000 7f65762bd640 1 -- 172.21.15.42:0/704093038 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f6570123c70 con 0x7f65701cf170 2024-09-16T03:52:20.143 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.117+0000 7f656f7fe640 1 --2- 172.21.15.42:0/704093038 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f65701cf170 0x7f65701258c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:52:20.143 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.117+0000 7f656f7fe640 10 monclient: get_auth_request con 0x7f65701cf170 auth_method 0 2024-09-16T03:52:20.143 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.117+0000 7f656f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:52:20.143 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.117+0000 7f656f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:52:20.143 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.117+0000 7f656f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:52:20.143 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.117+0000 7f656f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:52:20.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.117+0000 7f656f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:52:20.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.117+0000 7f656cff9640 1 -- 172.21.15.42:0/704093038 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f65600c1070 con 0x7f657009bf90 2024-09-16T03:52:20.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.117+0000 7f656f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:52:20.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.117+0000 7f656f7fe640 10 monclient(hunting): handle_auth_done global_id 824178 payload 824 2024-09-16T03:52:20.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.117+0000 7f656f7fe640 1 -- 172.21.15.42:0/704093038 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f657009c400 con 0x7f65701cf170 2024-09-16T03:52:20.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.117+0000 7f656cff9640 1 --2- 172.21.15.42:0/704093038 >> v2:172.21.15.154:6812/254845342 conn(0x7f655c0377c0 0x7f655c039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:52:20.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.117+0000 7f656cff9640 1 -- 172.21.15.42:0/704093038 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(332..332 src has 286..332) ==== 5328+0+0 (secure 0 0 0) 0x7f65600e2c90 con 0x7f657009bf90 2024-09-16T03:52:20.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.117+0000 7f656cff9640 10 client.824175.objecter ms_dispatch 0x7f6570000ba0 osd_map(332..332 src has 286..332) 2024-09-16T03:52:20.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.117+0000 7f656cff9640 3 client.824175.objecter handle_osd_map got epochs [332,332] > 0 2024-09-16T03:52:20.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.117+0000 7f656cff9640 3 client.824175.objecter handle_osd_map decoding full epoch 332 2024-09-16T03:52:20.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.117+0000 7f656cff9640 20 client.824175.objecter dump_active .. 0 homeless 2024-09-16T03:52:20.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.117+0000 7f656cff9640 10 client.824175.objecter ms_handle_connect 0x7f65701cf170 2024-09-16T03:52:20.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.117+0000 7f656cff9640 10 client.824175.objecter resend_mon_ops 2024-09-16T03:52:20.145 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.117+0000 7f656f7fe640 1 --2- 172.21.15.42:0/704093038 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f65701cf170 0x7f65701258c0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f656400bfd0 tx=0x7f656400f8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:52:20.145 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.117+0000 7f656cff9640 1 -- 172.21.15.42:0/704093038 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6564019020 con 0x7f65701cf170 2024-09-16T03:52:20.145 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.117+0000 7f656cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:52:20.145 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.117+0000 7f656cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:52:20.145 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.117+0000 7f656cff9640 10 monclient: dump: 2024-09-16T03:52:20.145 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:52:20.145 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:52:20.145 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:52:20.145 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:52:20.145 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:52:20.145 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:52:20.145 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:52:20.145 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:52:20.146 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:52:20.146 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:52:20.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.117+0000 7f657485a640 1 --2- 172.21.15.42:0/704093038 >> v2:172.21.15.154:6812/254845342 conn(0x7f655c0377c0 0x7f655c039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:52:20.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.117+0000 7f657485a640 10 monclient: get_auth_request con 0x7f655c0377c0 auth_method 0 2024-09-16T03:52:20.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.121+0000 7f656cff9640 1 -- 172.21.15.42:0/704093038 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f6564014870 con 0x7f65701cf170 2024-09-16T03:52:20.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.121+0000 7f656cff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:52:20.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.121+0000 7f656cff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:52:20.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.121+0000 7f656cff9640 1 -- 172.21.15.42:0/704093038 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f65701cf170 msgr2=0x7f65701258c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:52:20.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.121+0000 7f656cff9640 1 --2- 172.21.15.42:0/704093038 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f65701cf170 0x7f65701258c0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f656400bfd0 tx=0x7f656400f8a0 comp rx=0 tx=0).stop 2024-09-16T03:52:20.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.121+0000 7f656cff9640 1 -- 172.21.15.42:0/704093038 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f65701cf170 msgr2=0x7f65701258c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:52:20.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.121+0000 7f656cff9640 1 --2- 172.21.15.42:0/704093038 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f65701cf170 0x7f65701258c0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:52:20.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.121+0000 7f656cff9640 10 client.824175.objecter ms_handle_connect 0x7f655c0377c0 2024-09-16T03:52:20.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.121+0000 7f657485a640 1 --2- 172.21.15.42:0/704093038 >> v2:172.21.15.154:6812/254845342 conn(0x7f655c0377c0 0x7f655c039c80 secure :-1 s=READY pgs=3577 cs=0 l=1 rev1=1 crypto rx=0x7f6568007fd0 tx=0x7f6568007ce0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:52:20.148 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.125+0000 7f65762bd640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:52:20.148 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.125+0000 7f65762bd640 1 --2- 172.21.15.42:0/704093038 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f657002c650 0x7f657002ca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:52:20.148 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.125+0000 7f65762bd640 10 monclient(hunting): start opening mon connection 2024-09-16T03:52:20.148 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.125+0000 7f65762bd640 1 -- 172.21.15.42:0/704093038 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f6570071020 con 0x7f657002c650 2024-09-16T03:52:20.148 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.125+0000 7f656f7fe640 1 --2- 172.21.15.42:0/704093038 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f657002c650 0x7f657002ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:52:20.148 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.125+0000 7f656f7fe640 10 monclient: get_auth_request con 0x7f657002c650 auth_method 0 2024-09-16T03:52:20.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.125+0000 7f656f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:52:20.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.125+0000 7f656f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:52:20.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.125+0000 7f656f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:52:20.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.125+0000 7f656f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:52:20.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.125+0000 7f656f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:52:20.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.125+0000 7f656f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:52:20.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.125+0000 7f656f7fe640 10 monclient(hunting): handle_auth_done global_id 824181 payload 824 2024-09-16T03:52:20.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.125+0000 7f656f7fe640 1 -- 172.21.15.42:0/704093038 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f6570124550 con 0x7f657002c650 2024-09-16T03:52:20.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.125+0000 7f656f7fe640 1 --2- 172.21.15.42:0/704093038 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f657002c650 0x7f657002ca60 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f656400bfd0 tx=0x7f65640046d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:52:20.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.125+0000 7f656cff9640 10 client.824175.objecter ms_handle_connect 0x7f657002c650 2024-09-16T03:52:20.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.125+0000 7f656cff9640 10 client.824175.objecter resend_mon_ops 2024-09-16T03:52:20.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.125+0000 7f656cff9640 1 -- 172.21.15.42:0/704093038 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6564019030 con 0x7f657002c650 2024-09-16T03:52:20.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.125+0000 7f656cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:52:20.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.125+0000 7f656cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:52:20.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.125+0000 7f656cff9640 10 monclient: dump: 2024-09-16T03:52:20.150 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:52:20.150 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:52:20.150 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:52:20.150 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:52:20.150 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:52:20.150 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:52:20.150 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:52:20.150 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:52:20.150 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:52:20.150 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:52:20.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.125+0000 7f656cff9640 1 -- 172.21.15.42:0/704093038 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4869 (secure 0 0 0) 0x7f6564004220 con 0x7f657002c650 2024-09-16T03:52:20.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.125+0000 7f656cff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:52:20.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.125+0000 7f656cff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:52:20.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.125+0000 7f656cff9640 1 -- 172.21.15.42:0/704093038 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f657002c650 msgr2=0x7f657002ca60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:52:20.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.125+0000 7f656cff9640 1 --2- 172.21.15.42:0/704093038 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f657002c650 0x7f657002ca60 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f656400bfd0 tx=0x7f65640046d0 comp rx=0 tx=0).stop 2024-09-16T03:52:20.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.125+0000 7f656cff9640 1 -- 172.21.15.42:0/704093038 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f657002c650 msgr2=0x7f657002ca60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:52:20.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.125+0000 7f656cff9640 1 --2- 172.21.15.42:0/704093038 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f657002c650 0x7f657002ca60 secure :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f656400bfd0 tx=0x7f65640046d0 comp rx=0 tx=0).stop 2024-09-16T03:52:20.151 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:52:20.151 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:52:20.151 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:52:20.151 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "leader", 2024-09-16T03:52:20.151 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 90, 2024-09-16T03:52:20.151 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 29979, 2024-09-16T03:52:20.151 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:52:20.152 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:52:20.152 INFO:teuthology.orchestra.run.smithi042.stdout: 1 2024-09-16T03:52:20.152 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:52:20.152 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 0, 2024-09-16T03:52:20.152 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:52:20.152 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:52:20.152 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:52:20.152 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:52:20.152 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:52:20.152 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:52:20.152 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:52:20.152 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:52:20.152 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:52:20.153 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:52:20.153 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:52:20.153 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:52:20.153 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:52:20.153 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:52:20.153 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:52:20.153 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:52:20.153 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:52:20.153 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:52:20.153 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:52:20.153 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:52:20.153 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:52:20.153 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:52:20.153 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:52:20.154 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:52:20.154 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:52:20.154 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:52:20.154 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:52:20.154 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:52:20.154 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:52:20.154 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:52:20.154 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:52:20.154 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:52:20.154 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:52:20.154 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:52:20.154 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:52:20.154 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:52:20.154 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:52:20.154 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:52:20.155 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:52:20.155 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:52:20.155 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:52:20.155 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:52:20.155 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:52:20.155 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:52:20.155 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:52:20.155 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:52:20.155 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:52:20.155 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:52:20.155 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:52:20.155 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:52:20.155 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:52:20.155 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:52:20.156 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:52:20.156 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:52:20.156 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:52:20.156 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:52:20.156 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:52:20.156 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:52:20.156 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:52:20.156 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:52:20.156 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:52:20.156 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:52:20.156 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:52:20.156 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:52:20.156 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:52:20.156 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:52:20.157 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:52:20.157 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:52:20.157 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:52:20.157 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:52:20.157 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:52:20.157 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:52:20.157 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:52:20.157 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:52:20.157 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:52:20.157 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:52:20.157 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:52:20.157 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:52:20.157 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:52:20.157 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:52:20.158 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:52:20.158 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:52:20.158 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:52:20.158 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:52:20.158 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:52:20.158 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:52:20.158 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:52:20.158 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:52:20.158 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:52:20.158 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:52:20.158 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:52:20.158 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:52:20.158 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:52:20.158 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:52:20.159 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:52:20.159 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:52:20.159 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:52:20.159 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:52:20.159 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:52:20.159 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:52:20.159 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:52:20.159 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:52:20.159 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:52:20.159 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:52:20.159 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:52:20.159 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:52:20.159 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:52:20.159 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:52:20.160 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:52:20.160 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:52:20.160 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:52:20.160 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:52:20.160 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:52:20.160 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:52:20.160 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:52:20.160 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:52:20.160 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:52:20.160 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:52:20.160 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:52:20.160 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:52:20.160 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:52:20.160 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:52:20.161 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:52:20.161 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:52:20.161 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:52:20.161 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:52:20.161 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:52:20.161 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:52:20.161 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:52:20.161 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:52:20.161 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:52:20.161 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:52:20.161 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:52:20.161 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:52:20.161 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:52:20.162 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:52:20.162 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:52:20.162 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:52:20.162 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:52:20.162 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:52:20.162 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:52:20.162 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T03:52:20.162 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:52:20.162 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:52:20.162 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:52:20.162 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 5 2024-09-16T03:52:20.162 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:52:20.162 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:52:20.162 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:52:20.163 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:52:20.163 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:52:20.163 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:52:20.163 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 7 2024-09-16T03:52:20.163 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:52:20.163 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:52:20.163 INFO:teuthology.orchestra.run.smithi042.stdout: "mgr": [ 2024-09-16T03:52:20.163 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:52:20.163 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:52:20.163 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:52:20.163 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:52:20.163 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:52:20.163 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:52:20.164 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:52:20.164 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:52:20.164 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:52:20.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.125+0000 7f65762bd640 10 librados: watch_flush enter 2024-09-16T03:52:20.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.125+0000 7f65762bd640 10 librados: watch_flush exit 2024-09-16T03:52:20.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.125+0000 7f65762bd640 20 client.824175.objecter shutdown clearing up homeless session... 2024-09-16T03:52:20.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.125+0000 7f65762bd640 10 client.824175.objecter successfully canceled tick 2024-09-16T03:52:20.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.125+0000 7f65762bd640 1 -- 172.21.15.42:0/704093038 >> v2:172.21.15.154:6812/254845342 conn(0x7f655c0377c0 msgr2=0x7f655c039c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:52:20.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.125+0000 7f65762bd640 1 --2- 172.21.15.42:0/704093038 >> v2:172.21.15.154:6812/254845342 conn(0x7f655c0377c0 0x7f655c039c80 secure :-1 s=READY pgs=3577 cs=0 l=1 rev1=1 crypto rx=0x7f6568007fd0 tx=0x7f6568007ce0 comp rx=0 tx=0).stop 2024-09-16T03:52:20.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.125+0000 7f65762bd640 10 monclient: shutdown 2024-09-16T03:52:20.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.125+0000 7f65762bd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:52:20.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.125+0000 7f65762bd640 1 -- 172.21.15.42:0/704093038 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f657009bf90 msgr2=0x7f6570137590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:52:20.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.125+0000 7f65762bd640 1 --2- 172.21.15.42:0/704093038 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f657009bf90 0x7f6570137590 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f6560024e80 tx=0x7f656001b610 comp rx=0 tx=0).stop 2024-09-16T03:52:20.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.125+0000 7f657485a640 1 -- 172.21.15.42:0/704093038 reap_dead start 2024-09-16T03:52:20.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.125+0000 7f657485a640 1 -- 172.21.15.42:0/704093038 reap_dead start 2024-09-16T03:52:20.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.125+0000 7f65762bd640 1 -- 172.21.15.42:0/704093038 shutdown_connections 2024-09-16T03:52:20.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.125+0000 7f65762bd640 1 -- 172.21.15.42:0/704093038 >> 172.21.15.42:0/704093038 conn(0x7f65700a2420 msgr2=0x7f65700a4e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:52:20.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.125+0000 7f65762bd640 1 -- 172.21.15.42:0/704093038 shutdown_connections 2024-09-16T03:52:20.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.125+0000 7f65762bd640 1 -- 172.21.15.42:0/704093038 wait complete. 2024-09-16T03:52:20.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.129+0000 7f65762bd640 1 librados: shutdown 2024-09-16T03:52:20.167 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T03:52:20.167 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-16T03:52:20.167 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T03:52:20.293 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.289+0000 7f879db5c640 10 monclient: get_monmap_and_config 2024-09-16T03:52:20.293 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.289+0000 7f879db5c640 10 monclient: build_initial_monmap 2024-09-16T03:52:20.293 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.289+0000 7f879db5c640 10 monclient: monmap: 2024-09-16T03:52:20.294 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:52:20.294 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:52:20.294 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:52:20.292391+0000 2024-09-16T03:52:20.294 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:52:20.292391+0000 2024-09-16T03:52:20.294 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:52:20.294 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:52:20.294 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:52:20.294 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:52:20.294 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:52:20.294 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:52:20.294 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:52:20.294 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:52:20.296 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.289+0000 7f879db5c640 1 Processor -- start 2024-09-16T03:52:20.296 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.289+0000 7f879db5c640 1 -- start start 2024-09-16T03:52:20.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.289+0000 7f879db5c640 10 monclient: init 2024-09-16T03:52:20.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.289+0000 7f879db5c640 10 monclient: _reopen_session rank -1 2024-09-16T03:52:20.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.289+0000 7f879db5c640 10 monclient: _add_conns ranks=[3,2,4,0,1] 2024-09-16T03:52:20.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.289+0000 7f879db5c640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f87980a3450 0x7f87980a3830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:52:20.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.289+0000 7f879db5c640 10 monclient(hunting): picked mon.noname-a con 0x7f87980a3450 addr v2:172.21.15.42:3300/0 2024-09-16T03:52:20.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.289+0000 7f879db5c640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f87980a0ea0 0x7f87980a3d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:52:20.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.289+0000 7f879db5c640 10 monclient(hunting): picked mon.noname-b con 0x7f87980a0ea0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:52:20.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.289+0000 7f879db5c640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f8798137220 0x7f8798137630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:52:20.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.289+0000 7f879db5c640 10 monclient(hunting): picked mon.noname-c con 0x7f8798137220 addr v2:172.21.15.154:3300/0 2024-09-16T03:52:20.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.289+0000 7f879db5c640 10 monclient(hunting): start opening mon connection 2024-09-16T03:52:20.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.289+0000 7f879db5c640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f87980a1b40 con 0x7f87980a3450 2024-09-16T03:52:20.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.289+0000 7f879db5c640 10 monclient(hunting): start opening mon connection 2024-09-16T03:52:20.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.289+0000 7f879db5c640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f87980a1d90 con 0x7f8798137220 2024-09-16T03:52:20.298 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.289+0000 7f879db5c640 10 monclient(hunting): start opening mon connection 2024-09-16T03:52:20.298 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.289+0000 7f879db5c640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f87980a07a0 con 0x7f87980a0ea0 2024-09-16T03:52:20.298 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.289+0000 7f879db5c640 10 monclient(hunting): _renew_subs 2024-09-16T03:52:20.298 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.289+0000 7f879db5c640 10 monclient(hunting): authenticate will time out at 1587.261848s 2024-09-16T03:52:20.298 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.289+0000 7f8796ffd640 1 -- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f87980a0ea0 msgr2=0x7f87980a3d70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T03:52:20.298 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.289+0000 7f8796ffd640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f87980a0ea0 0x7f87980a3d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:52:20.298 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.289+0000 7f87977fe640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f87980a3450 0x7f87980a3830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:52:20.298 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.289+0000 7f87977fe640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f87980a3450 0x7f87980a3830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:47626/0 (socket says 172.21.15.42:47626) 2024-09-16T03:52:20.298 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.289+0000 7f87977fe640 1 -- 172.21.15.42:0/340117571 learned_addr learned my addr 172.21.15.42:0/340117571 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:52:20.298 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.289+0000 7f87977fe640 10 monclient(hunting): get_auth_request con 0x7f87980a3450 auth_method 0 2024-09-16T03:52:20.298 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.289+0000 7f87977fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:52:20.298 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.289+0000 7f87977fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:52:20.298 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.289+0000 7f87977fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:52:20.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.289+0000 7f87977fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:52:20.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.289+0000 7f87977fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:52:20.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.289+0000 7f87977fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:52:20.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.289+0000 7f8797fff640 1 --2- 172.21.15.42:0/340117571 >> v2:172.21.15.154:3300/0 conn(0x7f8798137220 0x7f8798137630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:52:20.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.289+0000 7f87977fe640 10 monclient(hunting): handle_auth_done global_id 824184 payload 274 2024-09-16T03:52:20.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.289+0000 7f87977fe640 1 -- 172.21.15.42:0/340117571 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f87980a0ea0 msgr2=0x7f87980a3d70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:52:20.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.289+0000 7f87977fe640 1 --2- 172.21.15.42:0/340117571 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f87980a0ea0 0x7f87980a3d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:52:20.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.289+0000 7f87977fe640 1 -- 172.21.15.42:0/340117571 >> v2:172.21.15.154:3300/0 conn(0x7f8798137220 msgr2=0x7f8798137630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:52:20.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.289+0000 7f87977fe640 1 --2- 172.21.15.42:0/340117571 >> v2:172.21.15.154:3300/0 conn(0x7f8798137220 0x7f8798137630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:52:20.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.289+0000 7f87977fe640 10 monclient: _finish_hunting 0 2024-09-16T03:52:20.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.289+0000 7f87977fe640 1 monclient: found mon.noname-a 2024-09-16T03:52:20.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.289+0000 7f87977fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:52:20.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.289+0000 7f87977fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:52:20.300 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.289+0000 7f87977fe640 1 -- 172.21.15.42:0/340117571 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87980a00e0 con 0x7f87980a3450 2024-09-16T03:52:20.300 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.289+0000 7f8797fff640 10 monclient: get_auth_request con 0x7f8798137220 auth_method 0 2024-09-16T03:52:20.300 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.289+0000 7f8797fff640 1 --2- 172.21.15.42:0/340117571 >> v2:172.21.15.154:3300/0 conn(0x7f8798137220 0x7f8798137630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T03:52:20.306 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.293+0000 7f87977fe640 1 --2- 172.21.15.42:0/340117571 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f87980a3450 0x7f87980a3830 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f879806b650 tx=0x7f878801c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aaca135d4654ec42 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:52:20.306 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.293+0000 7f87967fc640 1 -- 172.21.15.42:0/340117571 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87880b0070 con 0x7f87980a3450 2024-09-16T03:52:20.306 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.293+0000 7f87967fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:52:20.306 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.293+0000 7f87967fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:52:20.306 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.293+0000 7f87967fc640 10 monclient: dump: 2024-09-16T03:52:20.307 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:52:20.307 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:52:20.307 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:52:20.307 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:52:20.307 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:52:20.307 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:52:20.307 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:52:20.307 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:52:20.307 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:52:20.307 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:52:20.307 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.293+0000 7f87967fc640 10 monclient: _finish_auth 0 2024-09-16T03:52:20.307 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.293+0000 7f87967fc640 10 monclient: _check_auth_tickets 2024-09-16T03:52:20.307 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.293+0000 7f87967fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:52:20.308 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.293+0000 7f879db5c640 5 monclient: authenticate success, global_id 824184 2024-09-16T03:52:20.308 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.293+0000 7f879db5c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:52:20.308 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.293+0000 7f87967fc640 1 -- 172.21.15.42:0/340117571 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f87880a4e50 con 0x7f87980a3450 2024-09-16T03:52:20.308 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.293+0000 7f87967fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:52:20.308 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.293+0000 7f879db5c640 10 monclient: get_monmap_and_config success 2024-09-16T03:52:20.308 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.293+0000 7f879db5c640 4 set_mon_vals no callback set 2024-09-16T03:52:20.308 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.293+0000 7f87967fc640 1 -- 172.21.15.42:0/340117571 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87880ab820 con 0x7f87980a3450 2024-09-16T03:52:20.308 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.293+0000 7f87967fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:52:20.308 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.293+0000 7f87967fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:52:20.308 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.293+0000 7f87967fc640 10 monclient: dump: 2024-09-16T03:52:20.308 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:52:20.308 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:52:20.308 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:52:20.308 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:52:20.309 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:52:20.309 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:52:20.309 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:52:20.309 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:52:20.309 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:52:20.309 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:52:20.309 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.293+0000 7f879db5c640 10 monclient: shutdown 2024-09-16T03:52:20.309 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.293+0000 7f879db5c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:52:20.309 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.293+0000 7f879db5c640 1 -- 172.21.15.42:0/340117571 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f87980a3450 msgr2=0x7f87980a3830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:52:20.309 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.293+0000 7f879db5c640 1 --2- 172.21.15.42:0/340117571 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f87980a3450 0x7f87980a3830 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f879806b650 tx=0x7f878801c400 comp rx=0 tx=0).stop 2024-09-16T03:52:20.309 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.293+0000 7f879db5c640 1 -- 172.21.15.42:0/340117571 shutdown_connections 2024-09-16T03:52:20.309 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.293+0000 7f879db5c640 1 --2- 172.21.15.42:0/340117571 >> v2:172.21.15.154:3300/0 conn(0x7f8798137220 0x7f8798137630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:52:20.309 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.293+0000 7f879db5c640 1 --2- 172.21.15.42:0/340117571 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f87980a0ea0 0x7f87980a3d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:52:20.309 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.293+0000 7f879db5c640 1 --2- 172.21.15.42:0/340117571 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f87980a3450 0x7f87980a3830 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:52:20.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.293+0000 7f879db5c640 1 -- 172.21.15.42:0/340117571 >> 172.21.15.42:0/340117571 conn(0x7f87980756e0 msgr2=0x7f8798075af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:52:20.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.293+0000 7f879db5c640 1 -- 172.21.15.42:0/340117571 shutdown_connections 2024-09-16T03:52:20.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.293+0000 7f879db5c640 1 -- 172.21.15.42:0/340117571 wait complete. 2024-09-16T03:52:20.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.293+0000 7f879db5c640 10 monclient: build_initial_monmap 2024-09-16T03:52:20.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.293+0000 7f879db5c640 10 monclient: monmap: 2024-09-16T03:52:20.310 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:52:20.310 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:52:20.310 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:52:20.310 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:52:20.310 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:52:20.310 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:52:20.310 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:52:20.310 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:52:20.311 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:52:20.311 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:52:20.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.293+0000 7f879db5c640 1 librados: starting msgr at 2024-09-16T03:52:20.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.293+0000 7f879db5c640 1 librados: starting objecter 2024-09-16T03:52:20.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.293+0000 7f879db5c640 1 Processor -- start 2024-09-16T03:52:20.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.293+0000 7f879db5c640 1 -- start start 2024-09-16T03:52:20.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.293+0000 7f879db5c640 1 librados: setting wanted keys 2024-09-16T03:52:20.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.293+0000 7f879db5c640 1 librados: calling monclient init 2024-09-16T03:52:20.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.293+0000 7f879db5c640 10 monclient: init 2024-09-16T03:52:20.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.293+0000 7f879db5c640 10 monclient: _reopen_session rank -1 2024-09-16T03:52:20.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.293+0000 7f879db5c640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T03:52:20.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.293+0000 7f879db5c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f87980a0ea0 0x7f87981cf410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:52:20.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.293+0000 7f879db5c640 10 monclient(hunting): picked mon.noname-b con 0x7f87980a0ea0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:52:20.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.293+0000 7f879db5c640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f87980a3450 0x7f87981366b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:52:20.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.293+0000 7f879db5c640 10 monclient(hunting): picked mon.noname-c con 0x7f87980a3450 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:52:20.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.293+0000 7f879db5c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8798137220 0x7f879811e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:52:20.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.293+0000 7f879db5c640 10 monclient(hunting): picked mon.noname-a con 0x7f8798137220 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:52:20.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.293+0000 7f879db5c640 10 monclient(hunting): start opening mon connection 2024-09-16T03:52:20.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.293+0000 7f879db5c640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f8798140e10 con 0x7f8798137220 2024-09-16T03:52:20.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.293+0000 7f879db5c640 10 monclient(hunting): start opening mon connection 2024-09-16T03:52:20.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.293+0000 7f879db5c640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f8798140c90 con 0x7f87980a0ea0 2024-09-16T03:52:20.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.293+0000 7f879db5c640 10 monclient(hunting): start opening mon connection 2024-09-16T03:52:20.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.293+0000 7f879db5c640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f8798140f90 con 0x7f87980a3450 2024-09-16T03:52:20.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.293+0000 7f879db5c640 10 monclient(hunting): _renew_subs 2024-09-16T03:52:20.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.293+0000 7f879db5c640 10 monclient(hunting): authenticate will time out at 1587.266883s 2024-09-16T03:52:20.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.293+0000 7f8796ffd640 1 -- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f87980a3450 msgr2=0x7f87981366b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T03:52:20.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.293+0000 7f8796ffd640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f87980a3450 0x7f87981366b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:52:20.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.293+0000 7f8797fff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8798137220 0x7f879811e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:52:20.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.293+0000 7f8797fff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8798137220 0x7f879811e0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:47634/0 (socket says 172.21.15.42:47634) 2024-09-16T03:52:20.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.293+0000 7f8797fff640 1 -- 172.21.15.42:0/4260431961 learned_addr learned my addr 172.21.15.42:0/4260431961 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:52:20.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.293+0000 7f8797fff640 10 monclient(hunting): get_auth_request con 0x7f8798137220 auth_method 0 2024-09-16T03:52:20.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.293+0000 7f8797fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:52:20.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.293+0000 7f8797fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:52:20.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.293+0000 7f8797fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:52:20.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.293+0000 7f8797fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:52:20.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.293+0000 7f8797fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:52:20.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.293+0000 7f8797fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:52:20.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.293+0000 7f8797fff640 10 monclient(hunting): handle_auth_done global_id 824187 payload 824 2024-09-16T03:52:20.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.293+0000 7f87977fe640 1 --2- 172.21.15.42:0/4260431961 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f87980a0ea0 0x7f87981cf410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:52:20.314 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.293+0000 7f8797fff640 1 -- 172.21.15.42:0/4260431961 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f87980a3450 msgr2=0x7f87981366b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:52:20.314 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.293+0000 7f8797fff640 1 --2- 172.21.15.42:0/4260431961 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f87980a3450 0x7f87981366b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:52:20.314 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.293+0000 7f8797fff640 1 -- 172.21.15.42:0/4260431961 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f87980a0ea0 msgr2=0x7f87981cf410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:52:20.314 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.293+0000 7f8797fff640 1 --2- 172.21.15.42:0/4260431961 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f87980a0ea0 0x7f87981cf410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:52:20.314 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.293+0000 7f8797fff640 10 monclient: _finish_hunting 0 2024-09-16T03:52:20.314 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.293+0000 7f8797fff640 1 monclient: found mon.noname-a 2024-09-16T03:52:20.314 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.293+0000 7f8797fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:52:20.314 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.293+0000 7f8797fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:52:20.314 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.293+0000 7f8797fff640 1 -- 172.21.15.42:0/4260431961 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8798121a90 con 0x7f8798137220 2024-09-16T03:52:20.314 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.293+0000 7f8797fff640 1 --2- 172.21.15.42:0/4260431961 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8798137220 0x7f879811e0c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f879000cef0 tx=0x7f879000a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:52:20.314 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.293+0000 7f877ffff640 10 client.?.objecter ms_handle_connect 0x7f8798137220 2024-09-16T03:52:20.314 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.293+0000 7f877ffff640 10 client.?.objecter resend_mon_ops 2024-09-16T03:52:20.314 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.293+0000 7f877ffff640 1 -- 172.21.15.42:0/4260431961 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f879001a070 con 0x7f8798137220 2024-09-16T03:52:20.314 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.293+0000 7f877ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:52:20.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.293+0000 7f877ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:52:20.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.293+0000 7f877ffff640 10 monclient: dump: 2024-09-16T03:52:20.315 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:52:20.315 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:52:20.315 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:52:20.315 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:52:20.315 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:52:20.315 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:52:20.315 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:52:20.315 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:52:20.315 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:52:20.315 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:52:20.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.293+0000 7f877ffff640 10 monclient: _finish_auth 0 2024-09-16T03:52:20.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.293+0000 7f877ffff640 10 monclient: _check_auth_tickets 2024-09-16T03:52:20.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.293+0000 7f877ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:52:20.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.293+0000 7f877ffff640 1 -- 172.21.15.42:0/4260431961 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f879000d860 con 0x7f8798137220 2024-09-16T03:52:20.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.293+0000 7f879db5c640 5 monclient: authenticate success, global_id 824187 2024-09-16T03:52:20.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.293+0000 7f877ffff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:52:20.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.293+0000 7f877ffff640 1 -- 172.21.15.42:0/4260431961 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f879000dc50 con 0x7f8798137220 2024-09-16T03:52:20.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.293+0000 7f879db5c640 10 monclient: _renew_subs 2024-09-16T03:52:20.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.293+0000 7f879db5c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:52:20.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.293+0000 7f879db5c640 1 -- 172.21.15.42:0/4260431961 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8798122050 con 0x7f8798137220 2024-09-16T03:52:20.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.293+0000 7f8795ffb640 4 set_mon_vals no callback set 2024-09-16T03:52:20.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.293+0000 7f877ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:52:20.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.293+0000 7f877ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:52:20.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.293+0000 7f877ffff640 10 monclient: dump: 2024-09-16T03:52:20.316 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:52:20.317 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:52:20.317 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:52:20.317 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:52:20.317 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:52:20.317 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:52:20.317 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:52:20.317 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:52:20.317 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:52:20.317 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:52:20.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.293+0000 7f879db5c640 10 client.824187.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:52:20.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.293+0000 7f879db5c640 10 monclient: _renew_subs 2024-09-16T03:52:20.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.293+0000 7f879db5c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:52:20.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.293+0000 7f879db5c640 1 -- 172.21.15.42:0/4260431961 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8798071a20 con 0x7f8798137220 2024-09-16T03:52:20.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.293+0000 7f879db5c640 1 librados: init done 2024-09-16T03:52:20.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.297+0000 7f879db5c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:52:20.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.297+0000 7f879db5c640 1 --2- 172.21.15.42:0/4260431961 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8798136bf0 0x7f879802c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:52:20.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.297+0000 7f879db5c640 10 monclient(hunting): start opening mon connection 2024-09-16T03:52:20.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.297+0000 7f879db5c640 1 -- 172.21.15.42:0/4260431961 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f87981208c0 con 0x7f8798136bf0 2024-09-16T03:52:20.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.297+0000 7f877ffff640 1 -- 172.21.15.42:0/4260431961 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f879001d070 con 0x7f8798137220 2024-09-16T03:52:20.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.297+0000 7f877ffff640 1 --2- 172.21.15.42:0/4260431961 >> v2:172.21.15.154:6812/254845342 conn(0x7f87840377c0 0x7f8784039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:52:20.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.297+0000 7f87977fe640 1 --2- 172.21.15.42:0/4260431961 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8798136bf0 0x7f879802c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:52:20.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.297+0000 7f877ffff640 1 -- 172.21.15.42:0/4260431961 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(332..332 src has 286..332) ==== 5328+0+0 (secure 0 0 0) 0x7f8790049670 con 0x7f8798137220 2024-09-16T03:52:20.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.297+0000 7f877ffff640 10 client.824187.objecter ms_dispatch 0x7f8798000ba0 osd_map(332..332 src has 286..332) 2024-09-16T03:52:20.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.297+0000 7f877ffff640 3 client.824187.objecter handle_osd_map got epochs [332,332] > 0 2024-09-16T03:52:20.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.297+0000 7f877ffff640 3 client.824187.objecter handle_osd_map decoding full epoch 332 2024-09-16T03:52:20.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.297+0000 7f87977fe640 10 monclient: get_auth_request con 0x7f8798136bf0 auth_method 0 2024-09-16T03:52:20.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.297+0000 7f87977fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:52:20.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.297+0000 7f87977fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:52:20.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.297+0000 7f87977fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:52:20.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.297+0000 7f877ffff640 20 client.824187.objecter dump_active .. 0 homeless 2024-09-16T03:52:20.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.297+0000 7f87977fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:52:20.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.297+0000 7f87977fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:52:20.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.297+0000 7f87977fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:52:20.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.297+0000 7f8796ffd640 1 --2- 172.21.15.42:0/4260431961 >> v2:172.21.15.154:6812/254845342 conn(0x7f87840377c0 0x7f8784039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:52:20.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.297+0000 7f8796ffd640 10 monclient: get_auth_request con 0x7f87840377c0 auth_method 0 2024-09-16T03:52:20.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.297+0000 7f87977fe640 10 monclient(hunting): handle_auth_done global_id 794452 payload 824 2024-09-16T03:52:20.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.297+0000 7f87977fe640 1 -- 172.21.15.42:0/4260431961 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8798137630 con 0x7f8798136bf0 2024-09-16T03:52:20.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.297+0000 7f877ffff640 10 client.824187.objecter ms_handle_connect 0x7f8798136bf0 2024-09-16T03:52:20.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.297+0000 7f877ffff640 10 client.824187.objecter resend_mon_ops 2024-09-16T03:52:20.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.297+0000 7f87977fe640 1 --2- 172.21.15.42:0/4260431961 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8798136bf0 0x7f879802c650 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f87980a2100 tx=0x7f878801ac40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:52:20.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.297+0000 7f877ffff640 1 -- 172.21.15.42:0/4260431961 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87880b0040 con 0x7f8798136bf0 2024-09-16T03:52:20.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.297+0000 7f877ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:52:20.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.297+0000 7f877ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:52:20.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.297+0000 7f877ffff640 10 monclient: dump: 2024-09-16T03:52:20.320 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:52:20.320 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:52:20.320 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:52:20.320 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:52:20.320 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:52:20.320 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:52:20.320 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:52:20.320 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:52:20.320 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:52:20.320 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:52:20.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.297+0000 7f877ffff640 10 client.824187.objecter ms_handle_connect 0x7f87840377c0 2024-09-16T03:52:20.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.297+0000 7f8796ffd640 1 --2- 172.21.15.42:0/4260431961 >> v2:172.21.15.154:6812/254845342 conn(0x7f87840377c0 0x7f8784039c80 secure :-1 s=READY pgs=3578 cs=0 l=1 rev1=1 crypto rx=0x7f878c006fd0 tx=0x7f878c006d40 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:52:20.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.297+0000 7f877ffff640 1 -- 172.21.15.42:0/4260431961 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f87880a4e50 con 0x7f8798136bf0 2024-09-16T03:52:20.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.297+0000 7f877ffff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:52:20.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.297+0000 7f877ffff640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:52:20.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.297+0000 7f877ffff640 1 -- 172.21.15.42:0/4260431961 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8798136bf0 msgr2=0x7f879802c650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:52:20.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.297+0000 7f877ffff640 1 --2- 172.21.15.42:0/4260431961 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8798136bf0 0x7f879802c650 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f87980a2100 tx=0x7f878801ac40 comp rx=0 tx=0).stop 2024-09-16T03:52:20.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.297+0000 7f877ffff640 1 -- 172.21.15.42:0/4260431961 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8798136bf0 msgr2=0x7f879802c650 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:52:20.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.297+0000 7f877ffff640 1 --2- 172.21.15.42:0/4260431961 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8798136bf0 0x7f879802c650 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:52:20.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.305+0000 7f879db5c640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:52:20.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.305+0000 7f879db5c640 1 --2- 172.21.15.42:0/4260431961 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f879802cb90 0x7f879802cf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:52:20.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.305+0000 7f879db5c640 10 monclient(hunting): start opening mon connection 2024-09-16T03:52:20.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.305+0000 7f879db5c640 1 -- 172.21.15.42:0/4260431961 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f8798121870 con 0x7f879802cb90 2024-09-16T03:52:20.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.305+0000 7f87977fe640 1 --2- 172.21.15.42:0/4260431961 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f879802cb90 0x7f879802cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:52:20.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.305+0000 7f87977fe640 10 monclient: get_auth_request con 0x7f879802cb90 auth_method 0 2024-09-16T03:52:20.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.305+0000 7f87977fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:52:20.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.305+0000 7f87977fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:52:20.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.305+0000 7f87977fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:52:20.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.305+0000 7f87977fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:52:20.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.305+0000 7f87977fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:52:20.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.305+0000 7f87977fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:52:20.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.305+0000 7f87977fe640 10 monclient(hunting): handle_auth_done global_id 794455 payload 824 2024-09-16T03:52:20.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.305+0000 7f87977fe640 1 -- 172.21.15.42:0/4260431961 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f879813b040 con 0x7f879802cb90 2024-09-16T03:52:20.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.305+0000 7f87977fe640 1 --2- 172.21.15.42:0/4260431961 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f879802cb90 0x7f879802cf70 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f8788000c10 tx=0x7f87880191a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:52:20.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.305+0000 7f877ffff640 10 client.824187.objecter ms_handle_connect 0x7f879802cb90 2024-09-16T03:52:20.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.305+0000 7f877ffff640 10 client.824187.objecter resend_mon_ops 2024-09-16T03:52:20.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.305+0000 7f877ffff640 1 -- 172.21.15.42:0/4260431961 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87880b0040 con 0x7f879802cb90 2024-09-16T03:52:20.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.305+0000 7f877ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:52:20.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.305+0000 7f877ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:52:20.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.305+0000 7f877ffff640 10 monclient: dump: 2024-09-16T03:52:20.324 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:52:20.324 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:52:20.324 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:52:20.324 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:52:20.324 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:52:20.325 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:52:20.325 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:52:20.325 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:52:20.325 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:52:20.325 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:52:20.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.305+0000 7f877ffff640 1 -- 172.21.15.42:0/4260431961 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4701 (secure 0 0 0) 0x7f8788003da0 con 0x7f879802cb90 2024-09-16T03:52:20.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.305+0000 7f877ffff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:52:20.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.305+0000 7f877ffff640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:52:20.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.305+0000 7f877ffff640 1 -- 172.21.15.42:0/4260431961 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f879802cb90 msgr2=0x7f879802cf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:52:20.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.305+0000 7f877ffff640 1 --2- 172.21.15.42:0/4260431961 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f879802cb90 0x7f879802cf70 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f8788000c10 tx=0x7f87880191a0 comp rx=0 tx=0).stop 2024-09-16T03:52:20.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.305+0000 7f877ffff640 1 -- 172.21.15.42:0/4260431961 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f879802cb90 msgr2=0x7f879802cf70 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:52:20.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.305+0000 7f877ffff640 1 --2- 172.21.15.42:0/4260431961 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f879802cb90 0x7f879802cf70 secure :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f8788000c10 tx=0x7f87880191a0 comp rx=0 tx=0).stop 2024-09-16T03:52:20.326 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:52:20.326 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:52:20.326 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:52:20.326 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T03:52:20.326 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 90, 2024-09-16T03:52:20.326 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 182149, 2024-09-16T03:52:20.326 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:52:20.326 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:52:20.326 INFO:teuthology.orchestra.run.smithi042.stdout: 1 2024-09-16T03:52:20.326 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:52:20.326 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 0, 2024-09-16T03:52:20.326 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:52:20.327 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:52:20.327 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:52:20.327 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:52:20.327 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:52:20.327 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:52:20.327 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:52:20.327 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:52:20.327 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:52:20.327 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:52:20.327 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:52:20.327 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:52:20.327 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:52:20.327 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:52:20.328 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:52:20.328 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:52:20.328 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:52:20.328 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:52:20.328 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:52:20.328 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:52:20.328 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:52:20.328 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:52:20.328 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:52:20.328 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:52:20.328 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:52:20.328 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:52:20.328 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:52:20.328 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:52:20.329 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:52:20.329 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:52:20.329 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:52:20.329 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:52:20.329 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:52:20.329 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:52:20.329 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:52:20.329 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:52:20.329 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:52:20.329 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:52:20.329 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:52:20.329 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:52:20.329 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:52:20.330 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:52:20.330 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:52:20.330 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:52:20.330 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:52:20.330 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:52:20.330 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:52:20.330 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:52:20.330 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:52:20.330 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:52:20.330 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:52:20.330 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:52:20.330 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:52:20.330 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:52:20.330 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:52:20.331 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:52:20.331 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:52:20.331 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:52:20.331 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:52:20.331 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:52:20.331 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:52:20.331 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:52:20.331 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:52:20.331 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:52:20.331 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:52:20.331 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:52:20.331 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:52:20.331 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:52:20.332 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:52:20.332 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:52:20.332 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:52:20.332 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:52:20.332 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:52:20.332 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:52:20.332 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:52:20.332 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:52:20.332 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:52:20.332 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:52:20.332 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:52:20.332 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:52:20.332 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:52:20.332 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:52:20.332 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:52:20.333 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:52:20.333 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:52:20.333 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:52:20.333 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:52:20.333 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:52:20.333 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:52:20.333 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:52:20.333 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:52:20.333 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:52:20.333 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:52:20.333 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:52:20.333 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:52:20.333 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:52:20.333 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:52:20.334 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:52:20.334 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:52:20.334 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:52:20.334 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:52:20.334 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:52:20.334 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:52:20.334 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:52:20.334 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:52:20.334 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:52:20.334 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:52:20.334 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:52:20.334 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:52:20.334 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:52:20.335 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:52:20.335 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:52:20.335 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:52:20.335 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:52:20.335 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:52:20.335 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:52:20.335 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:52:20.335 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:52:20.335 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:52:20.335 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:52:20.335 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:52:20.335 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:52:20.335 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:52:20.336 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:52:20.336 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:52:20.336 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:52:20.336 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:52:20.336 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:52:20.336 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:52:20.336 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:52:20.336 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:52:20.336 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:52:20.336 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:52:20.336 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:52:20.336 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:52:20.336 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:52:20.337 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:52:20.337 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:52:20.337 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:52:20.337 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:52:20.337 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:52:20.337 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T03:52:20.337 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:52:20.337 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:52:20.337 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:52:20.337 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:52:20.337 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:52:20.337 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:52:20.337 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:52:20.337 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:52:20.338 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:52:20.338 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:52:20.338 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 2 2024-09-16T03:52:20.338 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:52:20.338 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:52:20.338 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:52:20.338 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:52:20.338 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:52:20.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.305+0000 7f879db5c640 10 librados: watch_flush enter 2024-09-16T03:52:20.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.305+0000 7f879db5c640 10 librados: watch_flush exit 2024-09-16T03:52:20.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.305+0000 7f879db5c640 20 client.824187.objecter shutdown clearing up homeless session... 2024-09-16T03:52:20.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.305+0000 7f879db5c640 10 client.824187.objecter successfully canceled tick 2024-09-16T03:52:20.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.305+0000 7f879db5c640 1 -- 172.21.15.42:0/4260431961 >> v2:172.21.15.154:6812/254845342 conn(0x7f87840377c0 msgr2=0x7f8784039c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:52:20.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.305+0000 7f879db5c640 1 --2- 172.21.15.42:0/4260431961 >> v2:172.21.15.154:6812/254845342 conn(0x7f87840377c0 0x7f8784039c80 secure :-1 s=READY pgs=3578 cs=0 l=1 rev1=1 crypto rx=0x7f878c006fd0 tx=0x7f878c006d40 comp rx=0 tx=0).stop 2024-09-16T03:52:20.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.305+0000 7f879db5c640 10 monclient: shutdown 2024-09-16T03:52:20.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.305+0000 7f879db5c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:52:20.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.305+0000 7f879db5c640 1 -- 172.21.15.42:0/4260431961 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8798137220 msgr2=0x7f879811e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:52:20.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.305+0000 7f879db5c640 1 --2- 172.21.15.42:0/4260431961 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8798137220 0x7f879811e0c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f879000cef0 tx=0x7f879000a710 comp rx=0 tx=0).stop 2024-09-16T03:52:20.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.305+0000 7f8797fff640 1 -- 172.21.15.42:0/4260431961 reap_dead start 2024-09-16T03:52:20.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.305+0000 7f8797fff640 1 -- 172.21.15.42:0/4260431961 reap_dead start 2024-09-16T03:52:20.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.305+0000 7f879db5c640 1 -- 172.21.15.42:0/4260431961 shutdown_connections 2024-09-16T03:52:20.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.305+0000 7f879db5c640 1 -- 172.21.15.42:0/4260431961 >> 172.21.15.42:0/4260431961 conn(0x7f87980756e0 msgr2=0x7f8798138210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:52:20.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.305+0000 7f879db5c640 1 -- 172.21.15.42:0/4260431961 shutdown_connections 2024-09-16T03:52:20.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.305+0000 7f879db5c640 1 -- 172.21.15.42:0/4260431961 wait complete. 2024-09-16T03:52:20.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:20.305+0000 7f879db5c640 1 librados: shutdown 2024-09-16T03:52:20.341 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T03:52:20.341 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-16T03:52:40.342 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-16T03:52:40.342 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-09-16T03:52:40.342 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-16T03:52:40.343 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-16T03:52:40.346 INFO:tasks.ceph.mon.c:Started 2024-09-16T03:52:40.346 INFO:tasks.mon_thrash:Sending STOP to mon a 2024-09-16T03:52:40.346 INFO:tasks.ceph.mon.a:Sent signal 19 2024-09-16T03:52:40.346 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-16T03:52:55.347 INFO:tasks.mon_thrash:Sending CONT to mon a 2024-09-16T03:52:55.348 INFO:tasks.ceph.mon.a:Sent signal 18 2024-09-16T03:52:55.348 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T03:52:55.349 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:52:55.486 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.480+0000 7f44db9f7640 10 monclient: get_monmap_and_config 2024-09-16T03:52:55.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.480+0000 7f44db9f7640 10 monclient: build_initial_monmap 2024-09-16T03:52:55.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.480+0000 7f44db9f7640 10 monclient: monmap: 2024-09-16T03:52:55.487 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:52:55.487 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:52:55.487 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:52:55.485513+0000 2024-09-16T03:52:55.487 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:52:55.485513+0000 2024-09-16T03:52:55.487 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:52:55.487 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:52:55.487 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:52:55.487 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:52:55.487 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:52:55.487 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:52:55.488 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:52:55.488 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:52:55.491 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.480+0000 7f44db9f7640 1 Processor -- start 2024-09-16T03:52:55.491 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.480+0000 7f44db9f7640 1 -- start start 2024-09-16T03:52:55.491 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.480+0000 7f44db9f7640 10 monclient: init 2024-09-16T03:52:55.491 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.480+0000 7f44db9f7640 10 monclient: _reopen_session rank -1 2024-09-16T03:52:55.491 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.480+0000 7f44db9f7640 10 monclient: _add_conns ranks=[3,4,0,2,1] 2024-09-16T03:52:55.491 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.480+0000 7f44db9f7640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f44d40a1330 0x7f44d40a1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:52:55.491 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.480+0000 7f44db9f7640 10 monclient(hunting): picked mon.noname-a con 0x7f44d40a1330 addr v2:172.21.15.42:3300/0 2024-09-16T03:52:55.491 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.480+0000 7f44db9f7640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f44d406dac0 0x7f44d40a1c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:52:55.491 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.480+0000 7f44db9f7640 10 monclient(hunting): picked mon.noname-c con 0x7f44d406dac0 addr v2:172.21.15.154:3300/0 2024-09-16T03:52:55.491 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.480+0000 7f44db9f7640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f44d4135100 addr v1:172.21.15.42:6789/0 2024-09-16T03:52:55.492 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.480+0000 7f44db9f7640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f44d409c7d0 con 0x7f44d4135100 2024-09-16T03:52:55.492 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.480+0000 7f44db9f7640 10 monclient(hunting): start opening mon connection 2024-09-16T03:52:55.492 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.480+0000 7f44db9f7640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f44d409c090 con 0x7f44d40a1330 2024-09-16T03:52:55.492 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.480+0000 7f44db9f7640 10 monclient(hunting): start opening mon connection 2024-09-16T03:52:55.492 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.480+0000 7f44db9f7640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f44d409f610 con 0x7f44d406dac0 2024-09-16T03:52:55.492 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.480+0000 7f44db9f7640 10 monclient(hunting): _renew_subs 2024-09-16T03:52:55.492 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.480+0000 7f44db9f7640 10 monclient(hunting): authenticate will time out at 1622.454988s 2024-09-16T03:52:55.492 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.480+0000 7f44d9793640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f44d40a1330 0x7f44d40a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:52:55.492 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.480+0000 7f44d9793640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f44d40a1330 0x7f44d40a1710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:38436/0 (socket says 172.21.15.42:38436) 2024-09-16T03:52:55.492 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.480+0000 7f44d9f94640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f44d4135100 0x7f44d4135550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:41148/0 (socket says 172.21.15.42:41148) 2024-09-16T03:52:55.492 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.480+0000 7f44d9793640 1 -- 172.21.15.42:0/1336512271 learned_addr learned my addr 172.21.15.42:0/1336512271 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:52:55.492 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.480+0000 7f44d9793640 10 monclient(hunting): get_auth_request con 0x7f44d40a1330 auth_method 0 2024-09-16T03:52:55.493 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.480+0000 7f44d9793640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:52:55.493 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.480+0000 7f44d9793640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:52:55.493 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.480+0000 7f44d9793640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:52:55.493 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.480+0000 7f44d9793640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:52:55.493 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.480+0000 7f44d9793640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:52:55.493 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.480+0000 7f44d9793640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:52:55.493 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.480+0000 7f44d8f92640 1 --2- 172.21.15.42:0/1336512271 >> v2:172.21.15.154:3300/0 conn(0x7f44d406dac0 0x7f44d40a1c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:52:55.493 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.480+0000 7f44d8f92640 10 monclient(hunting): get_auth_request con 0x7f44d406dac0 auth_method 0 2024-09-16T03:52:55.493 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.480+0000 7f44d8f92640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:52:55.493 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.480+0000 7f44d8f92640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:52:55.493 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.480+0000 7f44d8f92640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:52:55.493 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.480+0000 7f44d9793640 10 monclient(hunting): handle_auth_done global_id 824190 payload 274 2024-09-16T03:52:55.493 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.480+0000 7f44d9793640 1 -- 172.21.15.42:0/1336512271 >> v2:172.21.15.154:3300/0 conn(0x7f44d406dac0 msgr2=0x7f44d40a1c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:52:55.494 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.480+0000 7f44d9793640 1 --2- 172.21.15.42:0/1336512271 >> v2:172.21.15.154:3300/0 conn(0x7f44d406dac0 0x7f44d40a1c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:52:55.494 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.480+0000 7f44d9793640 1 -- 172.21.15.42:0/1336512271 >> v1:172.21.15.42:6789/0 conn(0x7f44d4135100 legacy=0x7f44d4135550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:52:55.494 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.480+0000 7f44d9793640 10 monclient: _finish_hunting 0 2024-09-16T03:52:55.494 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.480+0000 7f44d9793640 1 monclient: found mon.noname-a 2024-09-16T03:52:55.494 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.480+0000 7f44d9793640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:52:55.494 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.480+0000 7f44d9793640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:52:55.494 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.480+0000 7f44d9793640 1 -- 172.21.15.42:0/1336512271 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44d409f8f0 con 0x7f44d40a1330 2024-09-16T03:52:55.494 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.480+0000 7f44cbfff640 1 -- 172.21.15.42:0/1336512271 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1445690678 0 0) 0x7f44d409c7d0 con 0x7f44d4135100 2024-09-16T03:52:55.494 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.480+0000 7f44cbfff640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T03:52:55.494 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.484+0000 7f44d9793640 1 --2- 172.21.15.42:0/1336512271 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f44d40a1330 0x7f44d40a1710 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f44d406da10 tx=0x7f44d001c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=544ef2e3c51e3efc server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:52:55.494 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.484+0000 7f44cbfff640 1 -- 172.21.15.42:0/1336512271 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44d00aa650 con 0x7f44d40a1330 2024-09-16T03:52:55.494 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.484+0000 7f44cbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:52:55.494 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.484+0000 7f44cbfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:52:55.494 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.484+0000 7f44cbfff640 10 monclient: dump: 2024-09-16T03:52:55.495 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:52:55.495 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:52:55.495 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:52:55.495 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:52:55.495 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:52:55.495 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:52:55.495 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:52:55.495 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:52:55.495 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:52:55.495 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:52:55.495 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.484+0000 7f44cbfff640 10 monclient: _finish_auth 0 2024-09-16T03:52:55.495 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.484+0000 7f44cbfff640 10 monclient: _check_auth_tickets 2024-09-16T03:52:55.495 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.484+0000 7f44cbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:52:55.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.484+0000 7f44cbfff640 1 -- 172.21.15.42:0/1336512271 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f44d00af070 con 0x7f44d40a1330 2024-09-16T03:52:55.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.484+0000 7f44cbfff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:52:55.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.484+0000 7f44cbfff640 1 -- 172.21.15.42:0/1336512271 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44d00aaa10 con 0x7f44d40a1330 2024-09-16T03:52:55.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.484+0000 7f44db9f7640 5 monclient: authenticate success, global_id 824190 2024-09-16T03:52:55.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.484+0000 7f44db9f7640 10 monclient: get_monmap_and_config success 2024-09-16T03:52:55.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.484+0000 7f44db9f7640 4 set_mon_vals no callback set 2024-09-16T03:52:55.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.484+0000 7f44cbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:52:55.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.484+0000 7f44cbfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:52:55.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.484+0000 7f44cbfff640 10 monclient: dump: 2024-09-16T03:52:55.496 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:52:55.496 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:52:55.496 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:52:55.496 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:52:55.496 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:52:55.497 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:52:55.497 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:52:55.497 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:52:55.497 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:52:55.497 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:52:55.502 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.484+0000 7f44db9f7640 10 monclient: shutdown 2024-09-16T03:52:55.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.484+0000 7f44db9f7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:52:55.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.484+0000 7f44db9f7640 1 -- 172.21.15.42:0/1336512271 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f44d40a1330 msgr2=0x7f44d40a1710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:52:55.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.484+0000 7f44db9f7640 1 --2- 172.21.15.42:0/1336512271 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f44d40a1330 0x7f44d40a1710 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f44d406da10 tx=0x7f44d001c4d0 comp rx=0 tx=0).stop 2024-09-16T03:52:55.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.484+0000 7f44db9f7640 1 -- 172.21.15.42:0/1336512271 shutdown_connections 2024-09-16T03:52:55.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.484+0000 7f44db9f7640 1 --2- 172.21.15.42:0/1336512271 >> v2:172.21.15.154:3300/0 conn(0x7f44d406dac0 0x7f44d40a1c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:52:55.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.484+0000 7f44db9f7640 1 --2- 172.21.15.42:0/1336512271 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f44d40a1330 0x7f44d40a1710 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:52:55.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.484+0000 7f44db9f7640 1 -- 172.21.15.42:0/1336512271 >> 172.21.15.42:0/1336512271 conn(0x7f44d40a2420 msgr2=0x7f44d40a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:52:55.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.484+0000 7f44db9f7640 1 -- 172.21.15.42:0/1336512271 shutdown_connections 2024-09-16T03:52:55.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.484+0000 7f44db9f7640 1 -- 172.21.15.42:0/1336512271 wait complete. 2024-09-16T03:52:55.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.484+0000 7f44db9f7640 10 monclient: build_initial_monmap 2024-09-16T03:52:55.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.484+0000 7f44db9f7640 10 monclient: monmap: 2024-09-16T03:52:55.503 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:52:55.503 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:52:55.503 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:52:55.504 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:52:55.504 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:52:55.504 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:52:55.504 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:52:55.504 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:52:55.504 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:52:55.504 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:52:55.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.484+0000 7f44db9f7640 1 librados: starting msgr at 2024-09-16T03:52:55.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.484+0000 7f44db9f7640 1 librados: starting objecter 2024-09-16T03:52:55.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.488+0000 7f44db9f7640 1 Processor -- start 2024-09-16T03:52:55.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.488+0000 7f44db9f7640 1 -- start start 2024-09-16T03:52:55.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.488+0000 7f44db9f7640 1 librados: setting wanted keys 2024-09-16T03:52:55.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.488+0000 7f44db9f7640 1 librados: calling monclient init 2024-09-16T03:52:55.505 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.488+0000 7f44db9f7640 10 monclient: init 2024-09-16T03:52:55.505 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.488+0000 7f44db9f7640 10 monclient: _reopen_session rank -1 2024-09-16T03:52:55.505 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.488+0000 7f44db9f7640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:52:55.505 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.488+0000 7f44db9f7640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f44d40a1330 0x7f44d413c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:52:55.505 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.488+0000 7f44db9f7640 10 monclient(hunting): picked mon.noname-b con 0x7f44d40a1330 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:52:55.505 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.488+0000 7f44db9f7640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f44d4135100 0x7f44d41345e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:52:55.505 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.488+0000 7f44db9f7640 10 monclient(hunting): picked mon.noname-a con 0x7f44d4135100 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:52:55.505 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.488+0000 7f44db9f7640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f44d41cf1d0 0x7f44d41cf660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:52:55.505 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.488+0000 7f44db9f7640 10 monclient(hunting): picked mon.noname-c con 0x7f44d41cf1d0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:52:55.505 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.488+0000 7f44db9f7640 10 monclient(hunting): start opening mon connection 2024-09-16T03:52:55.505 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.488+0000 7f44db9f7640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f44d413fb80 con 0x7f44d4135100 2024-09-16T03:52:55.505 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.488+0000 7f44db9f7640 10 monclient(hunting): start opening mon connection 2024-09-16T03:52:55.505 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.488+0000 7f44db9f7640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f44d409ef80 con 0x7f44d40a1330 2024-09-16T03:52:55.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.488+0000 7f44db9f7640 10 monclient(hunting): start opening mon connection 2024-09-16T03:52:55.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.488+0000 7f44db9f7640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f44d413fd00 con 0x7f44d41cf1d0 2024-09-16T03:52:55.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.488+0000 7f44db9f7640 10 monclient(hunting): _renew_subs 2024-09-16T03:52:55.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.488+0000 7f44db9f7640 10 monclient(hunting): authenticate will time out at 1622.461023s 2024-09-16T03:52:55.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.488+0000 7f44d8f92640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f44d4135100 0x7f44d41345e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:52:55.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.488+0000 7f44d8f92640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f44d4135100 0x7f44d41345e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:38440/0 (socket says 172.21.15.42:38440) 2024-09-16T03:52:55.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.488+0000 7f44d8f92640 1 -- 172.21.15.42:0/411309746 learned_addr learned my addr 172.21.15.42:0/411309746 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:52:55.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.488+0000 7f44d8f92640 10 monclient(hunting): get_auth_request con 0x7f44d4135100 auth_method 0 2024-09-16T03:52:55.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.488+0000 7f44d8f92640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:52:55.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.488+0000 7f44d8f92640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:52:55.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.488+0000 7f44d8f92640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:52:55.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.488+0000 7f44d9f94640 1 --2- 172.21.15.42:0/411309746 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f44d41cf1d0 0x7f44d41cf660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:52:55.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.488+0000 7f44d9f94640 10 monclient(hunting): get_auth_request con 0x7f44d41cf1d0 auth_method 0 2024-09-16T03:52:55.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.488+0000 7f44d9f94640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:52:55.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.488+0000 7f44d9f94640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:52:55.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.488+0000 7f44d9f94640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:52:55.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.488+0000 7f44d8f92640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:52:55.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.488+0000 7f44d8f92640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:52:55.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.488+0000 7f44d8f92640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:52:55.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.488+0000 7f44d9793640 1 --2- 172.21.15.42:0/411309746 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f44d40a1330 0x7f44d413c190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:52:55.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.488+0000 7f44d9f94640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:52:55.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.488+0000 7f44d9f94640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:52:55.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.488+0000 7f44d9f94640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:52:55.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.488+0000 7f44d9793640 10 monclient(hunting): get_auth_request con 0x7f44d40a1330 auth_method 0 2024-09-16T03:52:55.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.488+0000 7f44d9793640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:52:55.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.488+0000 7f44d9793640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:52:55.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.488+0000 7f44d9793640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:52:55.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.488+0000 7f44d8f92640 10 monclient(hunting): handle_auth_done global_id 824196 payload 824 2024-09-16T03:52:55.508 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.488+0000 7f44d8f92640 1 -- 172.21.15.42:0/411309746 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f44d41cf1d0 msgr2=0x7f44d41cf660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:52:55.508 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.488+0000 7f44d8f92640 1 --2- 172.21.15.42:0/411309746 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f44d41cf1d0 0x7f44d41cf660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:52:55.508 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.488+0000 7f44d8f92640 1 -- 172.21.15.42:0/411309746 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f44d40a1330 msgr2=0x7f44d413c190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:52:55.508 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.488+0000 7f44d8f92640 1 --2- 172.21.15.42:0/411309746 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f44d40a1330 0x7f44d413c190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:52:55.508 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.488+0000 7f44d8f92640 10 monclient: _finish_hunting 0 2024-09-16T03:52:55.508 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.488+0000 7f44d8f92640 1 monclient: found mon.noname-a 2024-09-16T03:52:55.508 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.488+0000 7f44d8f92640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:52:55.508 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.488+0000 7f44d8f92640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:52:55.508 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.488+0000 7f44d8f92640 1 -- 172.21.15.42:0/411309746 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44d409dc10 con 0x7f44d4135100 2024-09-16T03:52:55.508 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.488+0000 7f44d9793640 1 --2- 172.21.15.42:0/411309746 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f44d40a1330 0x7f44d413c190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T03:52:55.508 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.488+0000 7f44d8f92640 1 --2- 172.21.15.42:0/411309746 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f44d4135100 0x7f44d41345e0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f44c400d800 tx=0x7f44c400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:52:55.508 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.488+0000 7f44c9ffb640 10 client.?.objecter ms_handle_connect 0x7f44d4135100 2024-09-16T03:52:55.508 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.488+0000 7f44c9ffb640 10 client.?.objecter resend_mon_ops 2024-09-16T03:52:55.509 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.488+0000 7f44c9ffb640 1 -- 172.21.15.42:0/411309746 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44c400b6c0 con 0x7f44d4135100 2024-09-16T03:52:55.509 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.488+0000 7f44c9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:52:55.509 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.488+0000 7f44c9ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:52:55.509 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.488+0000 7f44c9ffb640 10 monclient: dump: 2024-09-16T03:52:55.509 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:52:55.509 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:52:55.509 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:52:55.509 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:52:55.509 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:52:55.509 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:52:55.509 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:52:55.509 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:52:55.509 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:52:55.510 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:52:55.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.488+0000 7f44c9ffb640 10 monclient: _finish_auth 0 2024-09-16T03:52:55.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.488+0000 7f44c9ffb640 10 monclient: _check_auth_tickets 2024-09-16T03:52:55.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.488+0000 7f44c9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:52:55.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.488+0000 7f44db9f7640 5 monclient: authenticate success, global_id 824196 2024-09-16T03:52:55.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.488+0000 7f44db9f7640 10 monclient: _renew_subs 2024-09-16T03:52:55.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.488+0000 7f44c9ffb640 1 -- 172.21.15.42:0/411309746 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f44c4019070 con 0x7f44d4135100 2024-09-16T03:52:55.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.488+0000 7f44db9f7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:52:55.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.488+0000 7f44db9f7640 1 -- 172.21.15.42:0/411309746 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44d413ce20 con 0x7f44d4135100 2024-09-16T03:52:55.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.488+0000 7f44c9ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T03:52:55.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.488+0000 7f44c9ffb640 1 -- 172.21.15.42:0/411309746 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44c400ba60 con 0x7f44d4135100 2024-09-16T03:52:55.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.488+0000 7f44c9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:52:55.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.488+0000 7f44c9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:52:55.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.488+0000 7f44c9ffb640 10 monclient: dump: 2024-09-16T03:52:55.511 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:52:55.511 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:52:55.511 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:52:55.511 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:52:55.511 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:52:55.511 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:52:55.511 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:52:55.511 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:52:55.511 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:52:55.511 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:52:55.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.488+0000 7f44cb7fe640 4 set_mon_vals no callback set 2024-09-16T03:52:55.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.488+0000 7f44db9f7640 10 client.824196.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:52:55.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.488+0000 7f44db9f7640 10 monclient: _renew_subs 2024-09-16T03:52:55.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.488+0000 7f44db9f7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:52:55.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.488+0000 7f44db9f7640 1 -- 172.21.15.42:0/411309746 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44d4120380 con 0x7f44d4135100 2024-09-16T03:52:55.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.488+0000 7f44db9f7640 1 librados: init done 2024-09-16T03:52:55.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.488+0000 7f44db9f7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:52:55.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.488+0000 7f44db9f7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:52:55.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.488+0000 7f44db9f7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:52:55.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.488+0000 7f44db9f7640 1 -- 172.21.15.42:0/411309746 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f44d4120800 con 0x7f44d4135100 2024-09-16T03:52:55.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.488+0000 7f44c9ffb640 1 -- 172.21.15.42:0/411309746 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f44c400bc00 con 0x7f44d4135100 2024-09-16T03:52:55.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.488+0000 7f44c9ffb640 1 --2- 172.21.15.42:0/411309746 >> v2:172.21.15.154:6812/254845342 conn(0x7f44b8037810 0x7f44b8039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:52:55.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.488+0000 7f44c9ffb640 1 -- 172.21.15.42:0/411309746 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(345..345 src has 286..345) ==== 5769+0+0 (secure 0 0 0) 0x7f44c4049e30 con 0x7f44d4135100 2024-09-16T03:52:55.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.488+0000 7f44c9ffb640 10 client.824196.objecter ms_dispatch 0x7f44d4000ba0 osd_map(345..345 src has 286..345) 2024-09-16T03:52:55.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.488+0000 7f44c9ffb640 3 client.824196.objecter handle_osd_map got epochs [345,345] > 0 2024-09-16T03:52:55.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.488+0000 7f44c9ffb640 3 client.824196.objecter handle_osd_map decoding full epoch 345 2024-09-16T03:52:55.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.492+0000 7f44c9ffb640 20 client.824196.objecter dump_active .. 0 homeless 2024-09-16T03:52:55.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.492+0000 7f44d9793640 1 --2- 172.21.15.42:0/411309746 >> v2:172.21.15.154:6812/254845342 conn(0x7f44b8037810 0x7f44b8039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:52:55.513 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.492+0000 7f44d9793640 10 monclient: get_auth_request con 0x7f44b8037810 auth_method 0 2024-09-16T03:52:55.513 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.492+0000 7f44d9793640 1 --2- 172.21.15.42:0/411309746 >> v2:172.21.15.154:6812/254845342 conn(0x7f44b8037810 0x7f44b8039cd0 secure :-1 s=READY pgs=3581 cs=0 l=1 rev1=1 crypto rx=0x7f44d00ae000 tx=0x7f44d00a7000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:52:55.513 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.492+0000 7f44c9ffb640 10 client.824196.objecter ms_handle_connect 0x7f44b8037810 2024-09-16T03:52:55.513 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.496+0000 7f44c9ffb640 1 -- 172.21.15.42:0/411309746 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f44c401ed10 con 0x7f44d4135100 2024-09-16T03:52:55.513 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.496+0000 7f44c9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:52:55.513 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.496+0000 7f44c9ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:52:55.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.724+0000 7f44db9f7640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:52:55.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.724+0000 7f44db9f7640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:52:55.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.724+0000 7f44db9f7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:52:55.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.724+0000 7f44db9f7640 1 -- 172.21.15.42:0/411309746 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f44d4135550 con 0x7f44d4135100 2024-09-16T03:52:55.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.724+0000 7f44c9ffb640 1 -- 172.21.15.42:0/411309746 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7f44c4047560 con 0x7f44d4135100 2024-09-16T03:52:55.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.724+0000 7f44c9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:52:55.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.724+0000 7f44c9ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:52:55.732 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":94,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:52:55.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.728+0000 7f44db9f7640 10 librados: watch_flush enter 2024-09-16T03:52:55.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.728+0000 7f44db9f7640 10 librados: watch_flush exit 2024-09-16T03:52:55.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.728+0000 7f44db9f7640 20 client.824196.objecter shutdown clearing up homeless session... 2024-09-16T03:52:55.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.728+0000 7f44db9f7640 10 client.824196.objecter successfully canceled tick 2024-09-16T03:52:55.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.728+0000 7f44db9f7640 1 -- 172.21.15.42:0/411309746 >> v2:172.21.15.154:6812/254845342 conn(0x7f44b8037810 msgr2=0x7f44b8039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:52:55.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.728+0000 7f44db9f7640 1 --2- 172.21.15.42:0/411309746 >> v2:172.21.15.154:6812/254845342 conn(0x7f44b8037810 0x7f44b8039cd0 secure :-1 s=READY pgs=3581 cs=0 l=1 rev1=1 crypto rx=0x7f44d00ae000 tx=0x7f44d00a7000 comp rx=0 tx=0).stop 2024-09-16T03:52:55.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.728+0000 7f44db9f7640 10 monclient: shutdown 2024-09-16T03:52:55.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.728+0000 7f44db9f7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:52:55.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.728+0000 7f44db9f7640 1 -- 172.21.15.42:0/411309746 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f44d4135100 msgr2=0x7f44d41345e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:52:55.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.728+0000 7f44db9f7640 1 --2- 172.21.15.42:0/411309746 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f44d4135100 0x7f44d41345e0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f44c400d800 tx=0x7f44c400dcc0 comp rx=0 tx=0).stop 2024-09-16T03:52:55.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.728+0000 7f44db9f7640 1 -- 172.21.15.42:0/411309746 shutdown_connections 2024-09-16T03:52:55.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.728+0000 7f44db9f7640 1 --2- 172.21.15.42:0/411309746 >> v2:172.21.15.154:6812/254845342 conn(0x7f44b8037810 0x7f44b8039cd0 unknown :-1 s=CLOSED pgs=3581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:52:55.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.728+0000 7f44db9f7640 1 --2- 172.21.15.42:0/411309746 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f44d41cf1d0 0x7f44d41cf660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:52:55.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.728+0000 7f44db9f7640 1 --2- 172.21.15.42:0/411309746 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f44d4135100 0x7f44d41345e0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:52:55.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.728+0000 7f44db9f7640 1 --2- 172.21.15.42:0/411309746 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f44d40a1330 0x7f44d413c190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:52:55.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.728+0000 7f44db9f7640 1 -- 172.21.15.42:0/411309746 >> 172.21.15.42:0/411309746 conn(0x7f44d40a2420 msgr2=0x7f44d40718d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:52:55.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.728+0000 7f44db9f7640 1 -- 172.21.15.42:0/411309746 shutdown_connections 2024-09-16T03:52:55.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.728+0000 7f44db9f7640 1 -- 172.21.15.42:0/411309746 wait complete. 2024-09-16T03:52:55.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.728+0000 7f44db9f7640 1 librados: shutdown 2024-09-16T03:52:55.752 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T03:52:55.752 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T03:52:55.752 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T03:52:55.873 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.868+0000 7ff00ec3b640 10 monclient: get_monmap_and_config 2024-09-16T03:52:55.874 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.868+0000 7ff00ec3b640 10 monclient: build_initial_monmap 2024-09-16T03:52:55.874 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.868+0000 7ff00ec3b640 10 monclient: monmap: 2024-09-16T03:52:55.874 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:52:55.874 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:52:55.874 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:52:55.872563+0000 2024-09-16T03:52:55.874 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:52:55.872563+0000 2024-09-16T03:52:55.874 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:52:55.874 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:52:55.874 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:52:55.874 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:52:55.875 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:52:55.875 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:52:55.875 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:52:55.875 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:52:55.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.868+0000 7ff00ec3b640 1 Processor -- start 2024-09-16T03:52:55.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.868+0000 7ff00ec3b640 1 -- start start 2024-09-16T03:52:55.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.868+0000 7ff00ec3b640 10 monclient: init 2024-09-16T03:52:55.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.868+0000 7ff00ec3b640 10 monclient: _reopen_session rank -1 2024-09-16T03:52:55.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.868+0000 7ff00ec3b640 10 monclient: _add_conns ranks=[4,3,2,0,1] 2024-09-16T03:52:55.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.868+0000 7ff00ec3b640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7ff0080a1330 0x7ff0080a1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:52:55.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.868+0000 7ff00ec3b640 10 monclient(hunting): picked mon.noname-c con 0x7ff0080a1330 addr v2:172.21.15.154:3300/0 2024-09-16T03:52:55.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.868+0000 7ff00ec3b640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff00806dac0 0x7ff0080a1c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:52:55.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.868+0000 7ff00ec3b640 10 monclient(hunting): picked mon.noname-a con 0x7ff00806dac0 addr v2:172.21.15.42:3300/0 2024-09-16T03:52:55.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.868+0000 7ff00ec3b640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff008135100 0x7ff008135550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:52:55.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.868+0000 7ff00ec3b640 10 monclient(hunting): picked mon.noname-b con 0x7ff008135100 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:52:55.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.868+0000 7ff00ec3b640 10 monclient(hunting): start opening mon connection 2024-09-16T03:52:55.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.868+0000 7ff00ec3b640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7ff00809c7d0 con 0x7ff00806dac0 2024-09-16T03:52:55.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.868+0000 7ff00ec3b640 10 monclient(hunting): start opening mon connection 2024-09-16T03:52:55.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.868+0000 7ff00ec3b640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7ff00809c090 con 0x7ff0080a1330 2024-09-16T03:52:55.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.868+0000 7ff00ec3b640 10 monclient(hunting): start opening mon connection 2024-09-16T03:52:55.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.868+0000 7ff00ec3b640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7ff00809f610 con 0x7ff008135100 2024-09-16T03:52:55.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.868+0000 7ff00ec3b640 10 monclient(hunting): _renew_subs 2024-09-16T03:52:55.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.868+0000 7ff00ec3b640 10 monclient(hunting): authenticate will time out at 1622.842136s 2024-09-16T03:52:55.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.868+0000 7ff007fff640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff00806dac0 0x7ff0080a1c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:52:55.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.868+0000 7ff00d1d8640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff008135100 0x7ff008135550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:52:55.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.868+0000 7ff00d1d8640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff008135100 0x7ff008135550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:42886/0 (socket says 172.21.15.42:42886) 2024-09-16T03:52:55.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.868+0000 7ff00d1d8640 1 -- 172.21.15.42:0/3318011507 learned_addr learned my addr 172.21.15.42:0/3318011507 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:52:55.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.868+0000 7ff00d1d8640 10 monclient(hunting): get_auth_request con 0x7ff008135100 auth_method 0 2024-09-16T03:52:55.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.868+0000 7ff00d1d8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:52:55.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.868+0000 7ff00d1d8640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:52:55.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.868+0000 7ff00d1d8640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:52:55.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.868+0000 7ff007fff640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff00806dac0 0x7ff0080a1c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:38448/0 (socket says 172.21.15.42:38448) 2024-09-16T03:52:55.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.868+0000 7ff007fff640 10 monclient(hunting): get_auth_request con 0x7ff00806dac0 auth_method 0 2024-09-16T03:52:55.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.868+0000 7ff007fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:52:55.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.868+0000 7ff007fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:52:55.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.868+0000 7ff007fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:52:55.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.868+0000 7ff00d1d8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:52:55.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.868+0000 7ff00d1d8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:52:55.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.868+0000 7ff00d1d8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:52:55.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.868+0000 7ff007fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:52:55.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.868+0000 7ff007fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:52:55.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.868+0000 7ff00c9d7640 1 --2- 172.21.15.42:0/3318011507 >> v2:172.21.15.154:3300/0 conn(0x7ff0080a1330 0x7ff0080a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:52:55.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.868+0000 7ff007fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:52:55.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.868+0000 7ff00c9d7640 10 monclient(hunting): get_auth_request con 0x7ff0080a1330 auth_method 0 2024-09-16T03:52:55.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.868+0000 7ff00c9d7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:52:55.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.868+0000 7ff00c9d7640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:52:55.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.868+0000 7ff00c9d7640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:52:55.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.868+0000 7ff00d1d8640 10 monclient(hunting): handle_auth_done global_id 834104 payload 274 2024-09-16T03:52:55.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.868+0000 7ff00d1d8640 1 -- 172.21.15.42:0/3318011507 >> v2:172.21.15.154:3300/0 conn(0x7ff0080a1330 msgr2=0x7ff0080a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:52:55.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.868+0000 7ff00d1d8640 1 --2- 172.21.15.42:0/3318011507 >> v2:172.21.15.154:3300/0 conn(0x7ff0080a1330 0x7ff0080a1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:52:55.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.868+0000 7ff00d1d8640 1 -- 172.21.15.42:0/3318011507 >> v2:172.21.15.42:3300/0 conn(0x7ff00806dac0 msgr2=0x7ff0080a1c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:52:55.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.868+0000 7ff00d1d8640 1 --2- 172.21.15.42:0/3318011507 >> v2:172.21.15.42:3300/0 conn(0x7ff00806dac0 0x7ff0080a1c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:52:55.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.868+0000 7ff00d1d8640 10 monclient: _finish_hunting 0 2024-09-16T03:52:55.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.868+0000 7ff00d1d8640 1 monclient: found mon.noname-b 2024-09-16T03:52:55.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.868+0000 7ff00d1d8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:52:55.883 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.868+0000 7ff00d1d8640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T03:52:55.883 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.868+0000 7ff00d1d8640 1 -- 172.21.15.42:0/3318011507 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff00809f8f0 con 0x7ff008135100 2024-09-16T03:52:55.883 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.868+0000 7ff007fff640 1 --2- 172.21.15.42:0/3318011507 >> v2:172.21.15.42:3300/0 conn(0x7ff00806dac0 0x7ff0080a1c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:52:55.883 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.868+0000 7ff00c9d7640 1 --2- 172.21.15.42:0/3318011507 >> v2:172.21.15.154:3300/0 conn(0x7ff0080a1330 0x7ff0080a1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T03:52:55.883 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.872+0000 7ff00d1d8640 1 --2- 172.21.15.42:0/3318011507 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff008135100 0x7ff008135550 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7ff00806add0 tx=0x7ff00001c910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=480cdbda6839ad2a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:52:55.883 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.872+0000 7ff0077fe640 1 -- 172.21.15.42:0/3318011507 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0000a5750 con 0x7ff008135100 2024-09-16T03:52:55.883 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.872+0000 7ff0077fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:52:55.883 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.872+0000 7ff0077fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:52:55.883 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.872+0000 7ff0077fe640 10 monclient: dump: 2024-09-16T03:52:55.883 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:52:55.883 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:52:55.884 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:52:55.884 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:52:55.884 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:52:55.884 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:52:55.884 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:52:55.884 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:52:55.884 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:52:55.884 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:52:55.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.872+0000 7ff0077fe640 10 monclient: _finish_auth 0 2024-09-16T03:52:55.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.872+0000 7ff0077fe640 10 monclient: _check_auth_tickets 2024-09-16T03:52:55.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.872+0000 7ff0077fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:52:55.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.872+0000 7ff0077fe640 1 -- 172.21.15.42:0/3318011507 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff0000a58f0 con 0x7ff008135100 2024-09-16T03:52:55.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.872+0000 7ff00ec3b640 5 monclient: authenticate success, global_id 834104 2024-09-16T03:52:55.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.872+0000 7ff00ec3b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:52:55.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.872+0000 7ff0077fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:52:55.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.872+0000 7ff00ec3b640 10 monclient: get_monmap_and_config success 2024-09-16T03:52:55.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.872+0000 7ff0077fe640 1 -- 172.21.15.42:0/3318011507 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0000a5cc0 con 0x7ff008135100 2024-09-16T03:52:55.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.872+0000 7ff0077fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:52:55.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.872+0000 7ff00ec3b640 4 set_mon_vals no callback set 2024-09-16T03:52:55.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.872+0000 7ff0077fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:52:55.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.872+0000 7ff0077fe640 10 monclient: dump: 2024-09-16T03:52:55.885 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:52:55.886 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:52:55.886 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:52:55.886 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:52:55.886 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:52:55.886 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:52:55.886 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:52:55.886 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:52:55.886 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:52:55.886 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:52:55.894 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.872+0000 7ff00ec3b640 10 monclient: shutdown 2024-09-16T03:52:55.894 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.872+0000 7ff00ec3b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:52:55.894 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.872+0000 7ff00ec3b640 1 -- 172.21.15.42:0/3318011507 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff008135100 msgr2=0x7ff008135550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:52:55.894 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.872+0000 7ff00ec3b640 1 --2- 172.21.15.42:0/3318011507 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff008135100 0x7ff008135550 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7ff00806add0 tx=0x7ff00001c910 comp rx=0 tx=0).stop 2024-09-16T03:52:55.895 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.872+0000 7ff00ec3b640 1 -- 172.21.15.42:0/3318011507 shutdown_connections 2024-09-16T03:52:55.895 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.872+0000 7ff00ec3b640 1 --2- 172.21.15.42:0/3318011507 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff008135100 0x7ff008135550 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:52:55.895 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.872+0000 7ff00ec3b640 1 --2- 172.21.15.42:0/3318011507 >> v2:172.21.15.42:3300/0 conn(0x7ff00806dac0 0x7ff0080a1c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:52:55.895 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.872+0000 7ff00ec3b640 1 --2- 172.21.15.42:0/3318011507 >> v2:172.21.15.154:3300/0 conn(0x7ff0080a1330 0x7ff0080a1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:52:55.895 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.872+0000 7ff00ec3b640 1 -- 172.21.15.42:0/3318011507 >> 172.21.15.42:0/3318011507 conn(0x7ff0080a2420 msgr2=0x7ff0080a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:52:55.895 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.872+0000 7ff00ec3b640 1 -- 172.21.15.42:0/3318011507 shutdown_connections 2024-09-16T03:52:55.895 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.872+0000 7ff00ec3b640 1 -- 172.21.15.42:0/3318011507 wait complete. 2024-09-16T03:52:55.895 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.872+0000 7ff00ec3b640 10 monclient: build_initial_monmap 2024-09-16T03:52:55.895 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.872+0000 7ff00ec3b640 10 monclient: monmap: 2024-09-16T03:52:55.895 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:52:55.895 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:52:55.895 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:52:55.896 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:52:55.896 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:52:55.896 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:52:55.896 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:52:55.896 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:52:55.896 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:52:55.896 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:52:55.896 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.872+0000 7ff00ec3b640 1 librados: starting msgr at 2024-09-16T03:52:55.896 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.872+0000 7ff00ec3b640 1 librados: starting objecter 2024-09-16T03:52:55.896 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.872+0000 7ff00ec3b640 1 Processor -- start 2024-09-16T03:52:55.896 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.872+0000 7ff00ec3b640 1 -- start start 2024-09-16T03:52:55.897 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.872+0000 7ff00ec3b640 1 librados: setting wanted keys 2024-09-16T03:52:55.897 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.872+0000 7ff00ec3b640 1 librados: calling monclient init 2024-09-16T03:52:55.897 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.872+0000 7ff00ec3b640 10 monclient: init 2024-09-16T03:52:55.897 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.872+0000 7ff00ec3b640 10 monclient: _reopen_session rank -1 2024-09-16T03:52:55.897 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.872+0000 7ff00ec3b640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:52:55.897 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.872+0000 7ff00ec3b640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff0080a1330 0x7ff0081cee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:52:55.897 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.872+0000 7ff00ec3b640 10 monclient(hunting): picked mon.noname-c con 0x7ff0080a1330 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:52:55.897 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.872+0000 7ff00ec3b640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff008135100 0x7ff0081343c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:52:55.897 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.872+0000 7ff00ec3b640 10 monclient(hunting): picked mon.noname-a con 0x7ff008135100 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:52:55.897 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.872+0000 7ff00ec3b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff008134900 0x7ff00809cb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:52:55.897 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.872+0000 7ff00ec3b640 10 monclient(hunting): picked mon.noname-b con 0x7ff008134900 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:52:55.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.872+0000 7ff00ec3b640 10 monclient(hunting): start opening mon connection 2024-09-16T03:52:55.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.872+0000 7ff00ec3b640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7ff00809eef0 con 0x7ff008135100 2024-09-16T03:52:55.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.872+0000 7ff00ec3b640 10 monclient(hunting): start opening mon connection 2024-09-16T03:52:55.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.872+0000 7ff00ec3b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff00809ed70 con 0x7ff008134900 2024-09-16T03:52:55.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.872+0000 7ff00ec3b640 10 monclient(hunting): start opening mon connection 2024-09-16T03:52:55.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.872+0000 7ff00ec3b640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7ff00813eb70 con 0x7ff0080a1330 2024-09-16T03:52:55.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.872+0000 7ff00ec3b640 10 monclient(hunting): _renew_subs 2024-09-16T03:52:55.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.872+0000 7ff00ec3b640 10 monclient(hunting): authenticate will time out at 1622.847662s 2024-09-16T03:52:55.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.872+0000 7ff00c9d7640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff0080a1330 0x7ff0081cee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:52:55.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.872+0000 7ff00c9d7640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff0080a1330 0x7ff0081cee50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:42894/0 (socket says 172.21.15.42:42894) 2024-09-16T03:52:55.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.872+0000 7ff00c9d7640 1 -- 172.21.15.42:0/1656013676 learned_addr learned my addr 172.21.15.42:0/1656013676 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:52:55.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.872+0000 7ff00c9d7640 10 monclient(hunting): get_auth_request con 0x7ff0080a1330 auth_method 0 2024-09-16T03:52:55.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.872+0000 7ff00c9d7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:52:55.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.872+0000 7ff00c9d7640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:52:55.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.872+0000 7ff007fff640 1 --2- 172.21.15.42:0/1656013676 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff008135100 0x7ff0081343c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:52:55.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.872+0000 7ff00c9d7640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:52:55.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.872+0000 7ff007fff640 10 monclient(hunting): get_auth_request con 0x7ff008135100 auth_method 0 2024-09-16T03:52:55.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.872+0000 7ff007fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:52:55.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.872+0000 7ff007fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:52:55.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.872+0000 7ff007fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:52:55.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.872+0000 7ff00c9d7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:52:55.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.872+0000 7ff00c9d7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:52:55.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.872+0000 7ff00c9d7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:52:55.900 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.872+0000 7ff007fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:52:55.900 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.872+0000 7ff007fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:52:55.900 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.872+0000 7ff007fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:52:55.900 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.876+0000 7ff00d1d8640 1 --2- 172.21.15.42:0/1656013676 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff008134900 0x7ff00809cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:52:55.900 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.876+0000 7ff00d1d8640 10 monclient(hunting): get_auth_request con 0x7ff008134900 auth_method 0 2024-09-16T03:52:55.900 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.876+0000 7ff00d1d8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:52:55.900 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.876+0000 7ff00d1d8640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:52:55.900 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.876+0000 7ff00d1d8640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:52:55.900 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.876+0000 7ff00c9d7640 10 monclient(hunting): handle_auth_done global_id 834107 payload 824 2024-09-16T03:52:55.900 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.876+0000 7ff00c9d7640 1 -- 172.21.15.42:0/1656013676 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff008134900 msgr2=0x7ff00809cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:52:55.900 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.876+0000 7ff00c9d7640 1 --2- 172.21.15.42:0/1656013676 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff008134900 0x7ff00809cb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:52:55.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.876+0000 7ff00c9d7640 1 -- 172.21.15.42:0/1656013676 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff008135100 msgr2=0x7ff0081343c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:52:55.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.876+0000 7ff00c9d7640 1 --2- 172.21.15.42:0/1656013676 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff008135100 0x7ff0081343c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:52:55.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.876+0000 7ff00c9d7640 10 monclient: _finish_hunting 0 2024-09-16T03:52:55.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.876+0000 7ff00c9d7640 1 monclient: found mon.noname-c 2024-09-16T03:52:55.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.876+0000 7ff00c9d7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:52:55.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.876+0000 7ff00c9d7640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:52:55.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.876+0000 7ff00c9d7640 1 -- 172.21.15.42:0/1656013676 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff008121d80 con 0x7ff0080a1330 2024-09-16T03:52:55.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.876+0000 7ff007fff640 1 --2- 172.21.15.42:0/1656013676 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff008135100 0x7ff0081343c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:52:55.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.876+0000 7ff00c9d7640 1 --2- 172.21.15.42:0/1656013676 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff0080a1330 0x7ff0081cee50 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7feff800e810 tx=0x7feff800ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:52:55.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.876+0000 7ff0057fa640 10 client.?.objecter ms_handle_connect 0x7ff0080a1330 2024-09-16T03:52:55.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.876+0000 7ff0057fa640 10 client.?.objecter resend_mon_ops 2024-09-16T03:52:55.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.876+0000 7ff0057fa640 1 -- 172.21.15.42:0/1656013676 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feff8017070 con 0x7ff0080a1330 2024-09-16T03:52:55.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.876+0000 7ff0057fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:52:55.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.876+0000 7ff0057fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:52:55.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.876+0000 7ff0057fa640 10 monclient: dump: 2024-09-16T03:52:55.902 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:52:55.902 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:52:55.902 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:52:55.902 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:52:55.902 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:52:55.902 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:52:55.902 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:52:55.902 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:52:55.903 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:52:55.903 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:52:55.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.876+0000 7ff0057fa640 10 monclient: _finish_auth 0 2024-09-16T03:52:55.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.876+0000 7ff0057fa640 10 monclient: _check_auth_tickets 2024-09-16T03:52:55.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.876+0000 7ff0057fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:52:55.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.876+0000 7ff00ec3b640 5 monclient: authenticate success, global_id 834107 2024-09-16T03:52:55.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.876+0000 7ff0057fa640 1 -- 172.21.15.42:0/1656013676 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feff800c8d0 con 0x7ff0080a1330 2024-09-16T03:52:55.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.876+0000 7ff00ec3b640 10 monclient: _renew_subs 2024-09-16T03:52:55.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.876+0000 7ff00ec3b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:52:55.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.876+0000 7ff00ec3b640 1 -- 172.21.15.42:0/1656013676 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff008071aa0 con 0x7ff0080a1330 2024-09-16T03:52:55.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.876+0000 7ff0057fa640 10 monclient: handle_config config(0 keys) 2024-09-16T03:52:55.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.876+0000 7ff0057fa640 1 -- 172.21.15.42:0/1656013676 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feff800f040 con 0x7ff0080a1330 2024-09-16T03:52:55.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.876+0000 7ff0057fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:52:55.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.876+0000 7ff0057fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:52:55.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.876+0000 7ff0057fa640 10 monclient: dump: 2024-09-16T03:52:55.904 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:52:55.904 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:52:55.904 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:52:55.904 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:52:55.904 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:52:55.904 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:52:55.904 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:52:55.904 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:52:55.905 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:52:55.905 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:52:55.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.876+0000 7ff00ec3b640 10 client.834107.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:52:55.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.876+0000 7ff0067fc640 4 set_mon_vals no callback set 2024-09-16T03:52:55.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.876+0000 7ff00ec3b640 10 monclient: _renew_subs 2024-09-16T03:52:55.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.876+0000 7ff00ec3b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:52:55.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.876+0000 7ff00ec3b640 1 -- 172.21.15.42:0/1656013676 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff008121aa0 con 0x7ff0080a1330 2024-09-16T03:52:55.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.876+0000 7ff00ec3b640 1 librados: init done 2024-09-16T03:52:55.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.876+0000 7ff0057fa640 1 -- 172.21.15.42:0/1656013676 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7feff801a070 con 0x7ff0080a1330 2024-09-16T03:52:55.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.876+0000 7ff00ec3b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:52:55.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.876+0000 7ff00ec3b640 1 --2- 172.21.15.42:0/1656013676 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff00806dac0 0x7ff00802c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:52:55.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.876+0000 7ff00ec3b640 10 monclient(hunting): start opening mon connection 2024-09-16T03:52:55.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.876+0000 7ff00ec3b640 1 -- 172.21.15.42:0/1656013676 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7ff0080a50b0 con 0x7ff00806dac0 2024-09-16T03:52:55.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.876+0000 7ff007fff640 1 --2- 172.21.15.42:0/1656013676 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff00806dac0 0x7ff00802c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:52:55.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.876+0000 7ff0057fa640 1 --2- 172.21.15.42:0/1656013676 >> v2:172.21.15.154:6812/254845342 conn(0x7fefe8037810 0x7fefe8039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:52:55.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.876+0000 7ff007fff640 10 monclient: get_auth_request con 0x7ff00806dac0 auth_method 0 2024-09-16T03:52:55.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.876+0000 7ff007fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:52:55.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.876+0000 7ff007fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:52:55.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.876+0000 7ff007fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:52:55.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.876+0000 7ff0057fa640 1 -- 172.21.15.42:0/1656013676 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(345..345 src has 286..345) ==== 5769+0+0 (secure 0 0 0) 0x7feff8048b30 con 0x7ff0080a1330 2024-09-16T03:52:55.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.876+0000 7ff0057fa640 10 client.834107.objecter ms_dispatch 0x7ff008000ba0 osd_map(345..345 src has 286..345) 2024-09-16T03:52:55.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.876+0000 7ff0057fa640 3 client.834107.objecter handle_osd_map got epochs [345,345] > 0 2024-09-16T03:52:55.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.876+0000 7ff0057fa640 3 client.834107.objecter handle_osd_map decoding full epoch 345 2024-09-16T03:52:55.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.876+0000 7ff007fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:52:55.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.876+0000 7ff007fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:52:55.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.876+0000 7ff007fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:52:55.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.876+0000 7ff0057fa640 20 client.834107.objecter dump_active .. 0 homeless 2024-09-16T03:52:55.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.876+0000 7ff007fff640 10 monclient(hunting): handle_auth_done global_id 824205 payload 824 2024-09-16T03:52:55.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.876+0000 7ff007fff640 1 -- 172.21.15.42:0/1656013676 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff008138a40 con 0x7ff00806dac0 2024-09-16T03:52:55.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.876+0000 7ff00d1d8640 1 --2- 172.21.15.42:0/1656013676 >> v2:172.21.15.154:6812/254845342 conn(0x7fefe8037810 0x7fefe8039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:52:55.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.876+0000 7ff00d1d8640 10 monclient: get_auth_request con 0x7fefe8037810 auth_method 0 2024-09-16T03:52:55.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.876+0000 7ff0057fa640 10 client.834107.objecter ms_handle_connect 0x7ff00806dac0 2024-09-16T03:52:55.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.876+0000 7ff0057fa640 10 client.834107.objecter resend_mon_ops 2024-09-16T03:52:55.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.876+0000 7ff007fff640 1 --2- 172.21.15.42:0/1656013676 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff00806dac0 0x7ff00802c650 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7ff00813c270 tx=0x7feff000f610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:52:55.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.876+0000 7ff0057fa640 1 -- 172.21.15.42:0/1656013676 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feff0019070 con 0x7ff00806dac0 2024-09-16T03:52:55.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.876+0000 7ff0057fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:52:55.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.876+0000 7ff0057fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:52:55.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.876+0000 7ff0057fa640 10 monclient: dump: 2024-09-16T03:52:55.908 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:52:55.908 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:52:55.908 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:52:55.908 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:52:55.908 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:52:55.909 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:52:55.909 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:52:55.909 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:52:55.909 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:52:55.909 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:52:55.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.876+0000 7ff0057fa640 1 -- 172.21.15.42:0/1656013676 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7feff0004030 con 0x7ff00806dac0 2024-09-16T03:52:55.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.876+0000 7ff0057fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:52:55.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.876+0000 7ff0057fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:52:55.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.876+0000 7ff0057fa640 1 -- 172.21.15.42:0/1656013676 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff00806dac0 msgr2=0x7ff00802c650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:52:55.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.876+0000 7ff0057fa640 1 --2- 172.21.15.42:0/1656013676 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff00806dac0 0x7ff00802c650 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7ff00813c270 tx=0x7feff000f610 comp rx=0 tx=0).stop 2024-09-16T03:52:55.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.876+0000 7ff0057fa640 1 -- 172.21.15.42:0/1656013676 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff00806dac0 msgr2=0x7ff00802c650 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:52:55.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.876+0000 7ff0057fa640 1 --2- 172.21.15.42:0/1656013676 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff00806dac0 0x7ff00802c650 secure :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7ff00813c270 tx=0x7feff000f610 comp rx=0 tx=0).stop 2024-09-16T03:52:55.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.876+0000 7ff0057fa640 10 client.834107.objecter ms_handle_connect 0x7fefe8037810 2024-09-16T03:52:55.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.876+0000 7ff00d1d8640 1 --2- 172.21.15.42:0/1656013676 >> v2:172.21.15.154:6812/254845342 conn(0x7fefe8037810 0x7fefe8039cd0 secure :-1 s=READY pgs=3582 cs=0 l=1 rev1=1 crypto rx=0x7ff0000ae000 tx=0x7ff0000a7070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:52:55.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.884+0000 7ff00ec3b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:52:55.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.884+0000 7ff00ec3b640 1 --2- 172.21.15.42:0/1656013676 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff00802cb90 0x7ff00802cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:52:55.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.884+0000 7ff00ec3b640 10 monclient(hunting): start opening mon connection 2024-09-16T03:52:55.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.884+0000 7ff00ec3b640 1 -- 172.21.15.42:0/1656013676 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7ff008135550 con 0x7ff00802cb90 2024-09-16T03:52:55.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.884+0000 7ff007fff640 1 --2- 172.21.15.42:0/1656013676 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff00802cb90 0x7ff00802cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:52:55.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.884+0000 7ff007fff640 10 monclient: get_auth_request con 0x7ff00802cb90 auth_method 0 2024-09-16T03:52:55.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.884+0000 7ff007fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:52:55.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.884+0000 7ff007fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:52:55.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.884+0000 7ff007fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:52:55.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.884+0000 7ff007fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:52:55.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.884+0000 7ff007fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:52:55.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.884+0000 7ff007fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:52:55.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.884+0000 7ff007fff640 10 monclient(hunting): handle_auth_done global_id 824208 payload 824 2024-09-16T03:52:55.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.884+0000 7ff007fff640 1 -- 172.21.15.42:0/1656013676 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ff0081cfc60 con 0x7ff00802cb90 2024-09-16T03:52:55.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.884+0000 7ff007fff640 1 --2- 172.21.15.42:0/1656013676 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff00802cb90 0x7ff00802cfa0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7ff00813c270 tx=0x7feff00046d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:52:55.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.884+0000 7ff0057fa640 10 client.834107.objecter ms_handle_connect 0x7ff00802cb90 2024-09-16T03:52:55.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.884+0000 7ff0057fa640 10 client.834107.objecter resend_mon_ops 2024-09-16T03:52:55.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.884+0000 7ff0057fa640 1 -- 172.21.15.42:0/1656013676 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feff0019040 con 0x7ff00802cb90 2024-09-16T03:52:55.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.884+0000 7ff0057fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:52:55.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.884+0000 7ff0057fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:52:55.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.884+0000 7ff0057fa640 10 monclient: dump: 2024-09-16T03:52:55.912 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:52:55.912 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:52:55.912 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:52:55.912 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:52:55.912 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:52:55.912 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:52:55.912 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:52:55.912 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:52:55.912 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:52:55.912 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:52:55.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.884+0000 7ff0057fa640 1 -- 172.21.15.42:0/1656013676 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4880 (secure 0 0 0) 0x7feff001e660 con 0x7ff00802cb90 2024-09-16T03:52:55.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.884+0000 7ff0057fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:52:55.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.884+0000 7ff0057fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:52:55.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.884+0000 7ff0057fa640 1 -- 172.21.15.42:0/1656013676 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff00802cb90 msgr2=0x7ff00802cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:52:55.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.884+0000 7ff0057fa640 1 --2- 172.21.15.42:0/1656013676 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff00802cb90 0x7ff00802cfa0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7ff00813c270 tx=0x7feff00046d0 comp rx=0 tx=0).stop 2024-09-16T03:52:55.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.884+0000 7ff0057fa640 1 -- 172.21.15.42:0/1656013676 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff00802cb90 msgr2=0x7ff00802cfa0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:52:55.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.884+0000 7ff0057fa640 1 --2- 172.21.15.42:0/1656013676 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff00802cb90 0x7ff00802cfa0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:52:55.913 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:52:55.913 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:52:55.914 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:52:55.914 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "leader", 2024-09-16T03:52:55.914 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 94, 2024-09-16T03:52:55.914 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 65739, 2024-09-16T03:52:55.914 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:52:55.914 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:52:55.914 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T03:52:55.914 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T03:52:55.914 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:52:55.914 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 0, 2024-09-16T03:52:55.914 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:52:55.915 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:52:55.915 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:52:55.915 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:52:55.915 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:52:55.915 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:52:55.915 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:52:55.915 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:52:55.915 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:52:55.915 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:52:55.915 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:52:55.915 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:52:55.915 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:52:55.916 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:52:55.916 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:52:55.916 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:52:55.916 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:52:55.916 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:52:55.916 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:52:55.916 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:52:55.916 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:52:55.916 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:52:55.916 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:52:55.916 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:52:55.917 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:52:55.917 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:52:55.917 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:52:55.917 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:52:55.917 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:52:55.917 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:52:55.917 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:52:55.917 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:52:55.917 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:52:55.917 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:52:55.917 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:52:55.918 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:52:55.918 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:52:55.918 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:52:55.918 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:52:55.918 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:52:55.918 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:52:55.918 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:52:55.918 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:52:55.918 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:52:55.918 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:52:55.918 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:52:55.919 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:52:55.919 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:52:55.919 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:52:55.919 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:52:55.919 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:52:55.919 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:52:55.919 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:52:55.919 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:52:55.919 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:52:55.919 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:52:55.919 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:52:55.919 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:52:55.920 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:52:55.920 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:52:55.920 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:52:55.920 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:52:55.920 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:52:55.920 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:52:55.920 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:52:55.920 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:52:55.920 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:52:55.920 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:52:55.920 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:52:55.921 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:52:55.921 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:52:55.921 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:52:55.921 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:52:55.921 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:52:55.921 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:52:55.921 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:52:55.921 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:52:55.921 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:52:55.921 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:52:55.921 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:52:55.922 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:52:55.922 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:52:55.922 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:52:55.922 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:52:55.922 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:52:55.922 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:52:55.922 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:52:55.922 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:52:55.922 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:52:55.922 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:52:55.922 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:52:55.922 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:52:55.923 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:52:55.923 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:52:55.923 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:52:55.923 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:52:55.923 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:52:55.923 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:52:55.923 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:52:55.923 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:52:55.923 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:52:55.923 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:52:55.923 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:52:55.924 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:52:55.924 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:52:55.924 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:52:55.924 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:52:55.924 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:52:55.924 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:52:55.924 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:52:55.924 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:52:55.924 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:52:55.924 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:52:55.924 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:52:55.925 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:52:55.925 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:52:55.925 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:52:55.925 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:52:55.925 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:52:55.925 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:52:55.925 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:52:55.925 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:52:55.925 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:52:55.925 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:52:55.925 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:52:55.925 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:52:55.926 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:52:55.926 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:52:55.926 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:52:55.926 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:52:55.926 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:52:55.926 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:52:55.926 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:52:55.926 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:52:55.926 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:52:55.926 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:52:55.926 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:52:55.926 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:52:55.927 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:52:55.927 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:52:55.927 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:52:55.927 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T03:52:55.927 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:52:55.927 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:52:55.927 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:52:55.927 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 5 2024-09-16T03:52:55.927 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:52:55.927 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:52:55.927 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:52:55.927 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:52:55.928 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:52:55.928 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:52:55.928 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 6 2024-09-16T03:52:55.928 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:52:55.928 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:52:55.928 INFO:teuthology.orchestra.run.smithi042.stdout: "mgr": [ 2024-09-16T03:52:55.928 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:52:55.928 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:52:55.928 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:52:55.928 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:52:55.928 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:52:55.929 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:52:55.929 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:52:55.929 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:52:55.929 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:52:55.930 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.884+0000 7ff00ec3b640 10 librados: watch_flush enter 2024-09-16T03:52:55.930 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.884+0000 7ff00ec3b640 10 librados: watch_flush exit 2024-09-16T03:52:55.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.884+0000 7ff00ec3b640 20 client.834107.objecter shutdown clearing up homeless session... 2024-09-16T03:52:55.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.884+0000 7ff00ec3b640 10 client.834107.objecter successfully canceled tick 2024-09-16T03:52:55.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.884+0000 7ff00ec3b640 1 -- 172.21.15.42:0/1656013676 >> v2:172.21.15.154:6812/254845342 conn(0x7fefe8037810 msgr2=0x7fefe8039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:52:55.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.884+0000 7ff00ec3b640 1 --2- 172.21.15.42:0/1656013676 >> v2:172.21.15.154:6812/254845342 conn(0x7fefe8037810 0x7fefe8039cd0 secure :-1 s=READY pgs=3582 cs=0 l=1 rev1=1 crypto rx=0x7ff0000ae000 tx=0x7ff0000a7070 comp rx=0 tx=0).stop 2024-09-16T03:52:55.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.884+0000 7ff00ec3b640 10 monclient: shutdown 2024-09-16T03:52:55.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.884+0000 7ff00ec3b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:52:55.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.884+0000 7ff00ec3b640 1 -- 172.21.15.42:0/1656013676 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff0080a1330 msgr2=0x7ff0081cee50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:52:55.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.884+0000 7ff00ec3b640 1 --2- 172.21.15.42:0/1656013676 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff0080a1330 0x7ff0081cee50 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7feff800e810 tx=0x7feff800ecd0 comp rx=0 tx=0).stop 2024-09-16T03:52:55.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.884+0000 7ff00d1d8640 1 -- 172.21.15.42:0/1656013676 reap_dead start 2024-09-16T03:52:55.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.884+0000 7ff00ec3b640 1 -- 172.21.15.42:0/1656013676 shutdown_connections 2024-09-16T03:52:55.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.884+0000 7ff00ec3b640 1 --2- 172.21.15.42:0/1656013676 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff0080a1330 0x7ff0081cee50 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:52:55.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.884+0000 7ff00ec3b640 1 -- 172.21.15.42:0/1656013676 >> 172.21.15.42:0/1656013676 conn(0x7ff0080a2420 msgr2=0x7ff0080a41f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:52:55.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.884+0000 7ff00ec3b640 1 -- 172.21.15.42:0/1656013676 shutdown_connections 2024-09-16T03:52:55.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.884+0000 7ff00ec3b640 1 -- 172.21.15.42:0/1656013676 wait complete. 2024-09-16T03:52:55.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:55.884+0000 7ff00ec3b640 1 librados: shutdown 2024-09-16T03:52:55.932 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:52:55.932 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T03:52:55.932 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T03:52:56.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.048+0000 7f636c890640 10 monclient: get_monmap_and_config 2024-09-16T03:52:56.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.048+0000 7f636c890640 10 monclient: build_initial_monmap 2024-09-16T03:52:56.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.048+0000 7f636c890640 10 monclient: monmap: 2024-09-16T03:52:56.055 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:52:56.055 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:52:56.055 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:52:56.052868+0000 2024-09-16T03:52:56.055 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:52:56.052868+0000 2024-09-16T03:52:56.055 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:52:56.055 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:52:56.055 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:52:56.055 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:52:56.055 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:52:56.056 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:52:56.056 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:52:56.056 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:52:56.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.048+0000 7f636c890640 1 Processor -- start 2024-09-16T03:52:56.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.048+0000 7f636c890640 1 -- start start 2024-09-16T03:52:56.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.048+0000 7f636c890640 10 monclient: init 2024-09-16T03:52:56.059 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.048+0000 7f636c890640 10 monclient: _reopen_session rank -1 2024-09-16T03:52:56.059 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.048+0000 7f636c890640 10 monclient: _add_conns ranks=[4,1,2,0,3] 2024-09-16T03:52:56.059 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.048+0000 7f636c890640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f63680a1330 0x7f636809f180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:52:56.059 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.048+0000 7f636c890640 10 monclient(hunting): picked mon.noname-c con 0x7f63680a1330 addr v2:172.21.15.154:3300/0 2024-09-16T03:52:56.059 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.048+0000 7f636c890640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f63680a1710 addr v1:172.21.15.154:6789/0 2024-09-16T03:52:56.060 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.048+0000 7f636c890640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f63681393a0 0x7f636813b790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:52:56.060 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.048+0000 7f636c890640 10 monclient(hunting): picked mon.noname-b con 0x7f63681393a0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:52:56.060 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.048+0000 7f636c890640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f636809e4c0 con 0x7f63680a1710 2024-09-16T03:52:56.060 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.048+0000 7f636c890640 10 monclient(hunting): start opening mon connection 2024-09-16T03:52:56.060 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.048+0000 7f636c890640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f636809edf0 con 0x7f63680a1330 2024-09-16T03:52:56.060 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.048+0000 7f636c890640 10 monclient(hunting): start opening mon connection 2024-09-16T03:52:56.060 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.048+0000 7f636c890640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f636809cd60 con 0x7f63681393a0 2024-09-16T03:52:56.060 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.048+0000 7f636c890640 10 monclient(hunting): _renew_subs 2024-09-16T03:52:56.060 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.048+0000 7f636c890640 10 monclient(hunting): authenticate will time out at 1623.022326s 2024-09-16T03:52:56.060 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.048+0000 7f6366d9d640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f63681393a0 0x7f636813b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:52:56.060 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.048+0000 7f6366d9d640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f63681393a0 0x7f636813b790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:42908/0 (socket says 172.21.15.42:42908) 2024-09-16T03:52:56.061 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.048+0000 7f6366d9d640 1 -- 172.21.15.42:0/1269873737 learned_addr learned my addr 172.21.15.42:0/1269873737 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:52:56.061 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.048+0000 7f6366d9d640 10 monclient(hunting): get_auth_request con 0x7f63681393a0 auth_method 0 2024-09-16T03:52:56.061 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.048+0000 7f6366d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:52:56.061 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.048+0000 7f6366d9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:52:56.061 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.048+0000 7f6366d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:52:56.061 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.048+0000 7f6366d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:52:56.061 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.048+0000 7f6366d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:52:56.061 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.048+0000 7f636659c640 1 --2- 172.21.15.42:0/1269873737 >> v2:172.21.15.154:3300/0 conn(0x7f63680a1330 0x7f636809f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:52:56.061 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.048+0000 7f6366d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:52:56.061 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.048+0000 7f636659c640 10 monclient(hunting): get_auth_request con 0x7f63680a1330 auth_method 0 2024-09-16T03:52:56.061 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.048+0000 7f636659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:52:56.061 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.048+0000 7f636659c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:52:56.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.048+0000 7f636659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:52:56.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.048+0000 7f6366d9d640 10 monclient(hunting): handle_auth_done global_id 834110 payload 274 2024-09-16T03:52:56.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.048+0000 7f6366d9d640 1 -- 172.21.15.42:0/1269873737 >> v2:172.21.15.154:3300/0 conn(0x7f63680a1330 msgr2=0x7f636809f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:52:56.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.048+0000 7f6366d9d640 1 --2- 172.21.15.42:0/1269873737 >> v2:172.21.15.154:3300/0 conn(0x7f63680a1330 0x7f636809f180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:52:56.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.048+0000 7f6366d9d640 1 -- 172.21.15.42:0/1269873737 >> v1:172.21.15.154:6789/0 conn(0x7f63680a1710 legacy=0x7f63680a1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:52:56.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.048+0000 7f6366d9d640 10 monclient: _finish_hunting 0 2024-09-16T03:52:56.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.048+0000 7f6366d9d640 1 monclient: found mon.noname-b 2024-09-16T03:52:56.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.048+0000 7f6366d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:52:56.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.048+0000 7f6366d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T03:52:56.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.048+0000 7f6366d9d640 1 -- 172.21.15.42:0/1269873737 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f636809f6c0 con 0x7f63681393a0 2024-09-16T03:52:56.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.052+0000 7f6366d9d640 1 -- 172.21.15.42:0/1269873737 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f63681393a0 msgr2=0x7f636813b790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T03:52:56.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.052+0000 7f6366d9d640 1 -- 172.21.15.42:0/1269873737 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f63681393a0 msgr2=0x7f636813b790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:52:56.063 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.052+0000 7f6366d9d640 1 --2- 172.21.15.42:0/1269873737 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f63681393a0 0x7f636813b790 secure :-1 s=COMPRESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0x7f636806add0 tx=0x7f636001c3a0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:52:56.063 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.052+0000 7f6366d9d640 1 --2- 172.21.15.42:0/1269873737 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f63681393a0 0x7f636813b790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:52:56.260 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.256+0000 7f6366d9d640 1 --2- 172.21.15.42:0/1269873737 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f63681393a0 0x7f636813b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:52:56.261 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.256+0000 7f6366d9d640 10 monclient: get_auth_request con 0x7f63681393a0 auth_method 0 2024-09-16T03:52:56.261 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.256+0000 7f6366d9d640 0 --2- 172.21.15.42:0/1269873737 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f63681393a0 0x7f636813b790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-09-16T03:52:56.261 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.256+0000 7f6366d9d640 1 --2- 172.21.15.42:0/1269873737 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f63681393a0 0x7f636813b790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:52:56.261 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.256+0000 7f636559a640 10 monclient: ms_handle_reset current mon [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:52:56.261 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.256+0000 7f636559a640 10 monclient: _reopen_session rank -1 2024-09-16T03:52:56.261 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.256+0000 7f636559a640 1 -- 172.21.15.42:0/1269873737 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f63681393a0 msgr2=0x7f636813b790 unknown :-1 s=STATE_CLOSED l=0).mark_down 2024-09-16T03:52:56.261 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.256+0000 7f636559a640 1 --2- 172.21.15.42:0/1269873737 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f63681393a0 0x7f636813b790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:52:56.261 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.256+0000 7f636559a640 10 monclient: _add_conns ranks=[3,0] 2024-09-16T03:52:56.261 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.256+0000 7f636559a640 1 --2- 172.21.15.42:0/1269873737 >> v2:172.21.15.42:3300/0 conn(0x7f6354003510 0x7f6354005990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:52:56.261 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.256+0000 7f636559a640 10 monclient(hunting): picked mon.noname-a con 0x7f6354003510 addr v2:172.21.15.42:3300/0 2024-09-16T03:52:56.261 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.256+0000 7f636559a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f63540062c0 addr v1:172.21.15.42:6789/0 2024-09-16T03:52:56.261 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.256+0000 7f636559a640 1 -- 172.21.15.42:0/1269873737 --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f635400a270 con 0x7f63540062c0 2024-09-16T03:52:56.262 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.256+0000 7f636559a640 10 monclient(hunting): start opening mon connection 2024-09-16T03:52:56.262 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.256+0000 7f636559a640 1 -- 172.21.15.42:0/1269873737 --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f635400b4b0 con 0x7f6354003510 2024-09-16T03:52:56.262 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.256+0000 7f636559a640 10 monclient(hunting): _renew_subs 2024-09-16T03:52:56.262 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.256+0000 7f636559a640 10 monclient(hunting): ms_handle_reset stray mon [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:52:56.262 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.256+0000 7f636659c640 1 --2- 172.21.15.42:0/1269873737 >> v2:172.21.15.42:3300/0 conn(0x7f6354003510 0x7f6354005990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:52:56.262 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.256+0000 7f636659c640 10 monclient(hunting): get_auth_request con 0x7f6354003510 auth_method 0 2024-09-16T03:52:56.262 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.256+0000 7f636659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:52:56.262 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.256+0000 7f636659c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:52:56.262 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.256+0000 7f636659c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:52:56.262 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.256+0000 7f636659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:52:56.262 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.256+0000 7f636659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:52:56.262 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.256+0000 7f636659c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T03:52:56.262 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.256+0000 7f636559a640 1 -- 172.21.15.42:0/1269873737 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3169488082 0 0) 0x7f635400a270 con 0x7f63540062c0 2024-09-16T03:52:56.262 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.256+0000 7f636559a640 10 monclient(hunting): _negotiate 2024-09-16T03:52:56.263 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.256+0000 7f636559a640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:52:56.263 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.256+0000 7f636559a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:52:56.263 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.256+0000 7f636559a640 1 -- 172.21.15.42:0/1269873737 --> v1:172.21.15.42:6789/0 -- auth(proto 2 132 bytes epoch 0) -- 0x7f635400a440 con 0x7f63540062c0 2024-09-16T03:52:56.263 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.256+0000 7f636659c640 10 monclient(hunting): handle_auth_done global_id 834110 payload 293 2024-09-16T03:52:56.263 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.256+0000 7f636659c640 1 -- 172.21.15.42:0/1269873737 >> v1:172.21.15.42:6789/0 conn(0x7f63540062c0 legacy=0x7f6354008780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:52:56.263 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.256+0000 7f636659c640 10 monclient: _finish_hunting 0 2024-09-16T03:52:56.263 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.256+0000 7f636659c640 1 monclient: found mon.noname-a 2024-09-16T03:52:56.263 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.256+0000 7f636659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:52:56.263 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.256+0000 7f636659c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:52:56.263 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.256+0000 7f636659c640 1 -- 172.21.15.42:0/1269873737 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f635400b720 con 0x7f6354003510 2024-09-16T03:52:56.267 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.256+0000 7f636659c640 1 --2- 172.21.15.42:0/1269873737 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6354003510 0x7f6354005990 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f635800b980 tx=0x7f6360009240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa57e89959739bcc server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:52:56.267 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.256+0000 7f636559a640 1 -- 172.21.15.42:0/1269873737 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f635800b600 con 0x7f6354003510 2024-09-16T03:52:56.268 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.256+0000 7f636559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:52:56.268 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.256+0000 7f636559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:52:56.268 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.256+0000 7f636559a640 10 monclient: dump: 2024-09-16T03:52:56.268 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:52:56.268 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:52:56.268 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:52:56.268 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:52:56.268 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:52:56.268 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:52:56.268 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:52:56.268 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:52:56.268 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:52:56.269 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:52:56.269 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.256+0000 7f636559a640 10 monclient: _finish_auth 0 2024-09-16T03:52:56.269 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.256+0000 7f636559a640 10 monclient: _check_auth_tickets 2024-09-16T03:52:56.269 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.256+0000 7f636559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:52:56.269 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.256+0000 7f636559a640 1 -- 172.21.15.42:0/1269873737 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6358019070 con 0x7f6354003510 2024-09-16T03:52:56.269 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.256+0000 7f636559a640 10 monclient: handle_config config(0 keys) 2024-09-16T03:52:56.269 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.256+0000 7f636559a640 1 -- 172.21.15.42:0/1269873737 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f635800db30 con 0x7f6354003510 2024-09-16T03:52:56.269 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.256+0000 7f636c890640 5 monclient: authenticate success, global_id 834110 2024-09-16T03:52:56.269 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.256+0000 7f636c890640 10 monclient: get_monmap_and_config success 2024-09-16T03:52:56.269 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.256+0000 7f636c890640 4 set_mon_vals no callback set 2024-09-16T03:52:56.269 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.256+0000 7f636559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:52:56.269 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.256+0000 7f636559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:52:56.269 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.256+0000 7f636559a640 10 monclient: dump: 2024-09-16T03:52:56.269 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:52:56.270 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:52:56.270 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:52:56.270 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:52:56.270 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:52:56.270 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:52:56.270 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:52:56.270 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:52:56.270 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:52:56.270 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:52:56.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.256+0000 7f636c890640 10 monclient: shutdown 2024-09-16T03:52:56.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.256+0000 7f636c890640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:52:56.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.256+0000 7f636c890640 1 -- 172.21.15.42:0/1269873737 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6354003510 msgr2=0x7f6354005990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:52:56.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.256+0000 7f636c890640 1 --2- 172.21.15.42:0/1269873737 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6354003510 0x7f6354005990 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f635800b980 tx=0x7f6360009240 comp rx=0 tx=0).stop 2024-09-16T03:52:56.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.256+0000 7f6366d9d640 1 -- 172.21.15.42:0/1269873737 reap_dead start 2024-09-16T03:52:56.271 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.256+0000 7f636c890640 1 -- 172.21.15.42:0/1269873737 shutdown_connections 2024-09-16T03:52:56.271 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.256+0000 7f636c890640 1 --2- 172.21.15.42:0/1269873737 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6354003510 0x7f6354005990 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:52:56.271 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.256+0000 7f636c890640 1 -- 172.21.15.42:0/1269873737 >> 172.21.15.42:0/1269873737 conn(0x7f63680a2420 msgr2=0x7f63680a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:52:56.271 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.256+0000 7f636c890640 1 -- 172.21.15.42:0/1269873737 shutdown_connections 2024-09-16T03:52:56.271 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.256+0000 7f636c890640 1 -- 172.21.15.42:0/1269873737 wait complete. 2024-09-16T03:52:56.271 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.256+0000 7f636c890640 10 monclient: build_initial_monmap 2024-09-16T03:52:56.271 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.256+0000 7f636c890640 10 monclient: monmap: 2024-09-16T03:52:56.271 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:52:56.271 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:52:56.271 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:52:56.271 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:52:56.271 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:52:56.271 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:52:56.271 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:52:56.272 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:52:56.272 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:52:56.272 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:52:56.272 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.256+0000 7f636c890640 1 librados: starting msgr at 2024-09-16T03:52:56.272 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.256+0000 7f636c890640 1 librados: starting objecter 2024-09-16T03:52:56.272 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.256+0000 7f636c890640 1 Processor -- start 2024-09-16T03:52:56.272 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.256+0000 7f636c890640 1 -- start start 2024-09-16T03:52:56.272 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.256+0000 7f636c890640 1 librados: setting wanted keys 2024-09-16T03:52:56.272 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.256+0000 7f636c890640 1 librados: calling monclient init 2024-09-16T03:52:56.272 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.256+0000 7f636c890640 10 monclient: init 2024-09-16T03:52:56.272 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.260+0000 7f636c890640 10 monclient: _reopen_session rank -1 2024-09-16T03:52:56.272 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.260+0000 7f636c890640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:52:56.272 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.260+0000 7f636c890640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f6354003510 0x7f63681cee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:52:56.273 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.260+0000 7f636c890640 10 monclient(hunting): picked mon.noname-c con 0x7f6354003510 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:52:56.273 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.260+0000 7f636c890640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f63681344c0 0x7f636813cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:52:56.273 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.260+0000 7f636c890640 10 monclient(hunting): picked mon.noname-b con 0x7f63681344c0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:52:56.273 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.260+0000 7f636c890640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6368134940 0x7f636813d250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:52:56.273 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.260+0000 7f636c890640 10 monclient(hunting): picked mon.noname-a con 0x7f6368134940 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:52:56.273 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.260+0000 7f636c890640 10 monclient(hunting): start opening mon connection 2024-09-16T03:52:56.273 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.260+0000 7f636c890640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f636813c8f0 con 0x7f6368134940 2024-09-16T03:52:56.273 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.260+0000 7f636c890640 10 monclient(hunting): start opening mon connection 2024-09-16T03:52:56.273 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.260+0000 7f636c890640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f636813c770 con 0x7f63681344c0 2024-09-16T03:52:56.273 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.260+0000 7f636c890640 10 monclient(hunting): start opening mon connection 2024-09-16T03:52:56.273 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.260+0000 7f636c890640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f636813ca70 con 0x7f6354003510 2024-09-16T03:52:56.273 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.260+0000 7f636c890640 10 monclient(hunting): _renew_subs 2024-09-16T03:52:56.273 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.260+0000 7f636c890640 10 monclient(hunting): authenticate will time out at 1623.232689s 2024-09-16T03:52:56.273 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.260+0000 7f636659c640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f6354003510 0x7f63681cee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:52:56.274 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.260+0000 7f636659c640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f6354003510 0x7f63681cee10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:42930/0 (socket says 172.21.15.42:42930) 2024-09-16T03:52:56.274 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.260+0000 7f6366d9d640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6368134940 0x7f636813d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:52:56.274 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.260+0000 7f636659c640 1 -- 172.21.15.42:0/3611391561 learned_addr learned my addr 172.21.15.42:0/3611391561 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:52:56.274 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.260+0000 7f636659c640 10 monclient(hunting): get_auth_request con 0x7f6354003510 auth_method 0 2024-09-16T03:52:56.274 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.260+0000 7f636659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:52:56.274 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.260+0000 7f636659c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:52:56.274 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.260+0000 7f636659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:52:56.274 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.260+0000 7f6366d9d640 10 monclient(hunting): get_auth_request con 0x7f6368134940 auth_method 0 2024-09-16T03:52:56.274 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.260+0000 7f6366d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:52:56.274 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.260+0000 7f6366d9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:52:56.274 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.260+0000 7f6366d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:52:56.274 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.260+0000 7f636659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:52:56.274 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.260+0000 7f636659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:52:56.275 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.260+0000 7f636659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:52:56.275 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.260+0000 7f6366d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:52:56.275 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.260+0000 7f6366d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:52:56.275 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.260+0000 7f6366d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:52:56.275 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.260+0000 7f6365d9b640 1 --2- 172.21.15.42:0/3611391561 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f63681344c0 0x7f636813cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:52:56.275 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.260+0000 7f6365d9b640 10 monclient(hunting): get_auth_request con 0x7f63681344c0 auth_method 0 2024-09-16T03:52:56.275 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.260+0000 7f6365d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:52:56.275 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.260+0000 7f6365d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:52:56.275 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.260+0000 7f6365d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:52:56.275 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.260+0000 7f636659c640 10 monclient(hunting): handle_auth_done global_id 834113 payload 824 2024-09-16T03:52:56.275 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.260+0000 7f636659c640 1 -- 172.21.15.42:0/3611391561 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f63681344c0 msgr2=0x7f636813cd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:52:56.275 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.260+0000 7f636659c640 1 --2- 172.21.15.42:0/3611391561 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f63681344c0 0x7f636813cd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:52:56.275 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.260+0000 7f636659c640 1 -- 172.21.15.42:0/3611391561 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6368134940 msgr2=0x7f636813d250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:52:56.275 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.260+0000 7f636659c640 1 --2- 172.21.15.42:0/3611391561 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6368134940 0x7f636813d250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:52:56.276 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.260+0000 7f636659c640 10 monclient: _finish_hunting 0 2024-09-16T03:52:56.276 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.260+0000 7f636659c640 1 monclient: found mon.noname-c 2024-09-16T03:52:56.276 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.260+0000 7f636659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:52:56.276 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.260+0000 7f636659c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:52:56.276 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.260+0000 7f636659c640 1 -- 172.21.15.42:0/3611391561 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6368073a30 con 0x7f6354003510 2024-09-16T03:52:56.276 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.260+0000 7f636659c640 1 --2- 172.21.15.42:0/3611391561 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f6354003510 0x7f63681cee10 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f6358008010 tx=0x7f6358004740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:52:56.276 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.260+0000 7f635effd640 10 client.?.objecter ms_handle_connect 0x7f6354003510 2024-09-16T03:52:56.276 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.260+0000 7f635effd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:52:56.276 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.260+0000 7f635effd640 1 -- 172.21.15.42:0/3611391561 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f635800dcc0 con 0x7f6354003510 2024-09-16T03:52:56.276 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.260+0000 7f635effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:52:56.276 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.260+0000 7f635effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:52:56.276 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.260+0000 7f635effd640 10 monclient: dump: 2024-09-16T03:52:56.276 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:52:56.277 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:52:56.277 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:52:56.277 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:52:56.277 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:52:56.277 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:52:56.277 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:52:56.277 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:52:56.277 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:52:56.277 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:52:56.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.260+0000 7f635effd640 10 monclient: _finish_auth 0 2024-09-16T03:52:56.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.260+0000 7f635effd640 10 monclient: _check_auth_tickets 2024-09-16T03:52:56.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.260+0000 7f635effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:52:56.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.260+0000 7f636c890640 5 monclient: authenticate success, global_id 834113 2024-09-16T03:52:56.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.260+0000 7f636c890640 10 monclient: _renew_subs 2024-09-16T03:52:56.278 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.260+0000 7f636c890640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:52:56.278 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.260+0000 7f636c890640 1 -- 172.21.15.42:0/3611391561 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63681cfd00 con 0x7f6354003510 2024-09-16T03:52:56.278 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.260+0000 7f636c890640 10 client.834113.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:52:56.278 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.260+0000 7f636c890640 10 monclient: _renew_subs 2024-09-16T03:52:56.278 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.260+0000 7f636c890640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:52:56.278 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.260+0000 7f636c890640 1 -- 172.21.15.42:0/3611391561 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f63680a1db0 con 0x7f6354003510 2024-09-16T03:52:56.278 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.260+0000 7f636c890640 1 librados: init done 2024-09-16T03:52:56.278 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.260+0000 7f635effd640 1 -- 172.21.15.42:0/3611391561 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f635800de60 con 0x7f6354003510 2024-09-16T03:52:56.278 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.260+0000 7f635effd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:52:56.278 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.260+0000 7f635effd640 1 -- 172.21.15.42:0/3611391561 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6358003860 con 0x7f6354003510 2024-09-16T03:52:56.278 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.260+0000 7f635effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:52:56.278 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.260+0000 7f635effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:52:56.278 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.260+0000 7f635effd640 10 monclient: dump: 2024-09-16T03:52:56.278 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:52:56.279 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:52:56.279 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:52:56.279 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:52:56.279 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:52:56.279 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:52:56.279 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:52:56.279 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:52:56.279 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:52:56.279 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:52:56.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.260+0000 7f6364d99640 4 set_mon_vals no callback set 2024-09-16T03:52:56.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.260+0000 7f636c890640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:52:56.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.260+0000 7f635effd640 1 -- 172.21.15.42:0/3611391561 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f6358025020 con 0x7f6354003510 2024-09-16T03:52:56.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.260+0000 7f635effd640 1 --2- 172.21.15.42:0/3611391561 >> v2:172.21.15.154:6812/254845342 conn(0x7f6348037840 0x7f6348039d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:52:56.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.260+0000 7f635effd640 1 -- 172.21.15.42:0/3611391561 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(345..345 src has 286..345) ==== 5769+0+0 (secure 0 0 0) 0x7f635804a7e0 con 0x7f6354003510 2024-09-16T03:52:56.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.260+0000 7f635effd640 10 client.834113.objecter ms_dispatch 0x7f6368000ba0 osd_map(345..345 src has 286..345) 2024-09-16T03:52:56.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.260+0000 7f635effd640 3 client.834113.objecter handle_osd_map got epochs [345,345] > 0 2024-09-16T03:52:56.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.260+0000 7f635effd640 3 client.834113.objecter handle_osd_map decoding full epoch 345 2024-09-16T03:52:56.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.260+0000 7f635effd640 20 client.834113.objecter dump_active .. 0 homeless 2024-09-16T03:52:56.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.260+0000 7f6365d9b640 1 --2- 172.21.15.42:0/3611391561 >> v2:172.21.15.154:6812/254845342 conn(0x7f6348037840 0x7f6348039d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:52:56.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.264+0000 7f636c890640 1 --2- 172.21.15.42:0/3611391561 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f636802c650 0x7f636802ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:52:56.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.264+0000 7f636c890640 10 monclient(hunting): start opening mon connection 2024-09-16T03:52:56.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.264+0000 7f636c890640 1 -- 172.21.15.42:0/3611391561 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f636809f180 con 0x7f636802c650 2024-09-16T03:52:56.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.264+0000 7f6365d9b640 10 monclient: get_auth_request con 0x7f6348037840 auth_method 0 2024-09-16T03:52:56.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.264+0000 7f6366d9d640 1 --2- 172.21.15.42:0/3611391561 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f636802c650 0x7f636802ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:52:56.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.264+0000 7f6366d9d640 10 monclient: get_auth_request con 0x7f636802c650 auth_method 0 2024-09-16T03:52:56.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.264+0000 7f6366d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:52:56.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.264+0000 7f6366d9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:52:56.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.264+0000 7f6366d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:52:56.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.264+0000 7f6366d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:52:56.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.264+0000 7f6366d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:52:56.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.264+0000 7f6366d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:52:56.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.264+0000 7f6366d9d640 10 monclient(hunting): handle_auth_done global_id 834116 payload 824 2024-09-16T03:52:56.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.264+0000 7f6366d9d640 1 -- 172.21.15.42:0/3611391561 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6368138aa0 con 0x7f636802c650 2024-09-16T03:52:56.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.264+0000 7f6365d9b640 1 --2- 172.21.15.42:0/3611391561 >> v2:172.21.15.154:6812/254845342 conn(0x7f6348037840 0x7f6348039d00 secure :-1 s=READY pgs=3583 cs=0 l=1 rev1=1 crypto rx=0x7f636813eec0 tx=0x7f6350009440 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:52:56.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.264+0000 7f635effd640 10 client.834113.objecter ms_handle_connect 0x7f6348037840 2024-09-16T03:52:56.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.264+0000 7f6366d9d640 1 --2- 172.21.15.42:0/3611391561 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f636802c650 0x7f636802ca50 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f636809df70 tx=0x7f6360019140 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:52:56.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.264+0000 7f635effd640 10 client.834113.objecter ms_handle_connect 0x7f636802c650 2024-09-16T03:52:56.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.264+0000 7f635effd640 10 client.834113.objecter resend_mon_ops 2024-09-16T03:52:56.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.264+0000 7f635effd640 1 -- 172.21.15.42:0/3611391561 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63600ac8c0 con 0x7f636802c650 2024-09-16T03:52:56.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.264+0000 7f635effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:52:56.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.264+0000 7f635effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:52:56.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.264+0000 7f635effd640 10 monclient: dump: 2024-09-16T03:52:56.286 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:52:56.286 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:52:56.286 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:52:56.286 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:52:56.286 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:52:56.286 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:52:56.286 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:52:56.286 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:52:56.286 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:52:56.286 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:52:56.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.264+0000 7f635effd640 1 -- 172.21.15.42:0/3611391561 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f6360000e50 con 0x7f636802c650 2024-09-16T03:52:56.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.264+0000 7f635effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:52:56.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.264+0000 7f635effd640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:52:56.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.264+0000 7f635effd640 1 -- 172.21.15.42:0/3611391561 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f636802c650 msgr2=0x7f636802ca50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:52:56.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.264+0000 7f635effd640 1 --2- 172.21.15.42:0/3611391561 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f636802c650 0x7f636802ca50 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f636809df70 tx=0x7f6360019140 comp rx=0 tx=0).stop 2024-09-16T03:52:56.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.264+0000 7f635effd640 1 -- 172.21.15.42:0/3611391561 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f636802c650 msgr2=0x7f636802ca50 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:52:56.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.264+0000 7f635effd640 1 --2- 172.21.15.42:0/3611391561 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f636802c650 0x7f636802ca50 secure :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f636809df70 tx=0x7f6360019140 comp rx=0 tx=0).stop 2024-09-16T03:52:56.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.272+0000 7f636c890640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:52:56.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.272+0000 7f636c890640 1 --2- 172.21.15.42:0/3611391561 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f636802cf90 0x7f636802d370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:52:56.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.272+0000 7f636c890640 10 monclient(hunting): start opening mon connection 2024-09-16T03:52:56.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.272+0000 7f636c890640 1 -- 172.21.15.42:0/3611391561 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f6368069040 con 0x7f636802cf90 2024-09-16T03:52:56.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.272+0000 7f6366d9d640 1 --2- 172.21.15.42:0/3611391561 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f636802cf90 0x7f636802d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:52:56.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.272+0000 7f6366d9d640 10 monclient: get_auth_request con 0x7f636802cf90 auth_method 0 2024-09-16T03:52:56.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.272+0000 7f6366d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:52:56.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.272+0000 7f6366d9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:52:56.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.272+0000 7f6366d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:52:56.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.272+0000 7f6366d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:52:56.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.272+0000 7f6366d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:52:56.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.272+0000 7f6366d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:52:56.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.272+0000 7f6366d9d640 10 monclient(hunting): handle_auth_done global_id 834119 payload 824 2024-09-16T03:52:56.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.272+0000 7f6366d9d640 1 -- 172.21.15.42:0/3611391561 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f636813df50 con 0x7f636802cf90 2024-09-16T03:52:56.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.272+0000 7f6366d9d640 1 --2- 172.21.15.42:0/3611391561 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f636802cf90 0x7f636802d370 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f636001aa50 tx=0x7f6360008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:52:56.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.272+0000 7f635effd640 10 client.834113.objecter ms_handle_connect 0x7f636802cf90 2024-09-16T03:52:56.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.272+0000 7f635effd640 10 client.834113.objecter resend_mon_ops 2024-09-16T03:52:56.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.272+0000 7f635effd640 1 -- 172.21.15.42:0/3611391561 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63600becb0 con 0x7f636802cf90 2024-09-16T03:52:56.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.272+0000 7f635effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:52:56.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.272+0000 7f635effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:52:56.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.272+0000 7f635effd640 10 monclient: dump: 2024-09-16T03:52:56.288 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:52:56.288 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:52:56.289 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:52:56.289 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:52:56.289 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:52:56.289 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:52:56.289 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:52:56.289 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:52:56.289 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:52:56.289 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:52:56.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.272+0000 7f635effd640 1 -- 172.21.15.42:0/3611391561 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4544 (secure 0 0 0) 0x7f63600b8440 con 0x7f636802cf90 2024-09-16T03:52:56.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.272+0000 7f635effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:52:56.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.272+0000 7f635effd640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:52:56.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.272+0000 7f635effd640 1 -- 172.21.15.42:0/3611391561 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f636802cf90 msgr2=0x7f636802d370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:52:56.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.272+0000 7f635effd640 1 --2- 172.21.15.42:0/3611391561 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f636802cf90 0x7f636802d370 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f636001aa50 tx=0x7f6360008040 comp rx=0 tx=0).stop 2024-09-16T03:52:56.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.272+0000 7f635effd640 1 -- 172.21.15.42:0/3611391561 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f636802cf90 msgr2=0x7f636802d370 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:52:56.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.272+0000 7f635effd640 1 --2- 172.21.15.42:0/3611391561 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f636802cf90 0x7f636802d370 secure :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f636001aa50 tx=0x7f6360008040 comp rx=0 tx=0).stop 2024-09-16T03:52:56.290 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:52:56.290 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:52:56.290 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:52:56.290 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T03:52:56.290 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 94, 2024-09-16T03:52:56.290 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 15843, 2024-09-16T03:52:56.290 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:52:56.290 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:52:56.290 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T03:52:56.290 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T03:52:56.291 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:52:56.291 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 0, 2024-09-16T03:52:56.291 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:52:56.291 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:52:56.291 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:52:56.291 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:52:56.291 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:52:56.291 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:52:56.291 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:52:56.291 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:52:56.291 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:52:56.291 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:52:56.291 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:52:56.292 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:52:56.292 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:52:56.292 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:52:56.292 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:52:56.292 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:52:56.292 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:52:56.292 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:52:56.292 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:52:56.292 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:52:56.292 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:52:56.292 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:52:56.292 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:52:56.292 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:52:56.292 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:52:56.293 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:52:56.293 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:52:56.293 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:52:56.293 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:52:56.293 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:52:56.293 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:52:56.293 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:52:56.293 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:52:56.293 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:52:56.293 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:52:56.293 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:52:56.293 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:52:56.293 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:52:56.293 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:52:56.294 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:52:56.294 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:52:56.294 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:52:56.294 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:52:56.294 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:52:56.294 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:52:56.294 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:52:56.294 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:52:56.294 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:52:56.294 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:52:56.294 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:52:56.294 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:52:56.294 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:52:56.294 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:52:56.295 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:52:56.295 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:52:56.295 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:52:56.295 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:52:56.295 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:52:56.295 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:52:56.295 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:52:56.295 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:52:56.295 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:52:56.295 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:52:56.295 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:52:56.295 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:52:56.295 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:52:56.295 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:52:56.296 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:52:56.296 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:52:56.296 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:52:56.296 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:52:56.296 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:52:56.296 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:52:56.296 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:52:56.296 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:52:56.296 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:52:56.296 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:52:56.296 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:52:56.296 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:52:56.296 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:52:56.296 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:52:56.297 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:52:56.297 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:52:56.297 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:52:56.297 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:52:56.297 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:52:56.297 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:52:56.297 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:52:56.297 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:52:56.297 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:52:56.297 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:52:56.297 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:52:56.297 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:52:56.297 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:52:56.297 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:52:56.298 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:52:56.298 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:52:56.298 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:52:56.298 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:52:56.298 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:52:56.298 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:52:56.298 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:52:56.298 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:52:56.298 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:52:56.298 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:52:56.298 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:52:56.298 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:52:56.298 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:52:56.298 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:52:56.299 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:52:56.299 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:52:56.299 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:52:56.299 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:52:56.299 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:52:56.299 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:52:56.299 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:52:56.299 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:52:56.299 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:52:56.299 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:52:56.299 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:52:56.299 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:52:56.299 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:52:56.299 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:52:56.300 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:52:56.300 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:52:56.300 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:52:56.300 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:52:56.300 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:52:56.300 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:52:56.300 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:52:56.300 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:52:56.300 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:52:56.300 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:52:56.300 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:52:56.300 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:52:56.300 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:52:56.301 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:52:56.301 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:52:56.301 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:52:56.301 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:52:56.301 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:52:56.301 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:52:56.301 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:52:56.301 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:52:56.301 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:52:56.301 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 2 2024-09-16T03:52:56.301 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:52:56.301 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:52:56.301 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:52:56.301 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:52:56.302 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:52:56.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.272+0000 7f636c890640 10 librados: watch_flush enter 2024-09-16T03:52:56.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.272+0000 7f636c890640 10 librados: watch_flush exit 2024-09-16T03:52:56.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.272+0000 7f636c890640 20 client.834113.objecter shutdown clearing up homeless session... 2024-09-16T03:52:56.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.272+0000 7f636c890640 10 client.834113.objecter successfully canceled tick 2024-09-16T03:52:56.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.272+0000 7f636c890640 1 -- 172.21.15.42:0/3611391561 >> v2:172.21.15.154:6812/254845342 conn(0x7f6348037840 msgr2=0x7f6348039d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:52:56.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.272+0000 7f636c890640 1 --2- 172.21.15.42:0/3611391561 >> v2:172.21.15.154:6812/254845342 conn(0x7f6348037840 0x7f6348039d00 secure :-1 s=READY pgs=3583 cs=0 l=1 rev1=1 crypto rx=0x7f636813eec0 tx=0x7f6350009440 comp rx=0 tx=0).stop 2024-09-16T03:52:56.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.272+0000 7f636c890640 10 monclient: shutdown 2024-09-16T03:52:56.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.272+0000 7f636c890640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:52:56.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.272+0000 7f636c890640 1 -- 172.21.15.42:0/3611391561 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f6354003510 msgr2=0x7f63681cee10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:52:56.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.272+0000 7f6366d9d640 1 -- 172.21.15.42:0/3611391561 reap_dead start 2024-09-16T03:52:56.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.272+0000 7f636c890640 1 --2- 172.21.15.42:0/3611391561 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f6354003510 0x7f63681cee10 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f6358008010 tx=0x7f6358004740 comp rx=0 tx=0).stop 2024-09-16T03:52:56.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.272+0000 7f636c890640 1 -- 172.21.15.42:0/3611391561 shutdown_connections 2024-09-16T03:52:56.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.272+0000 7f636c890640 1 --2- 172.21.15.42:0/3611391561 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f6354003510 0x7f63681cee10 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:52:56.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.272+0000 7f636c890640 1 -- 172.21.15.42:0/3611391561 >> 172.21.15.42:0/3611391561 conn(0x7f63680a2420 msgr2=0x7f636813a0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:52:56.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.272+0000 7f636c890640 1 -- 172.21.15.42:0/3611391561 shutdown_connections 2024-09-16T03:52:56.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.272+0000 7f636c890640 1 -- 172.21.15.42:0/3611391561 wait complete. 2024-09-16T03:52:56.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.272+0000 7f636c890640 1 librados: shutdown 2024-09-16T03:52:56.305 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:52:56.305 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T03:52:56.305 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T03:52:56.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.420+0000 7faddf8d9640 10 monclient: get_monmap_and_config 2024-09-16T03:52:56.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.420+0000 7faddf8d9640 10 monclient: build_initial_monmap 2024-09-16T03:52:56.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.420+0000 7faddf8d9640 10 monclient: monmap: 2024-09-16T03:52:56.426 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:52:56.426 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:52:56.426 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:52:56.424301+0000 2024-09-16T03:52:56.426 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:52:56.424301+0000 2024-09-16T03:52:56.426 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:52:56.426 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:52:56.426 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:52:56.426 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:52:56.426 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:52:56.426 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:52:56.426 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:52:56.427 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:52:56.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.420+0000 7faddf8d9640 1 Processor -- start 2024-09-16T03:52:56.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.420+0000 7faddf8d9640 1 -- start start 2024-09-16T03:52:56.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.420+0000 7faddf8d9640 10 monclient: init 2024-09-16T03:52:56.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.420+0000 7faddf8d9640 10 monclient: _reopen_session rank -1 2024-09-16T03:52:56.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.420+0000 7faddf8d9640 10 monclient: _add_conns ranks=[0,4,3,2,1] 2024-09-16T03:52:56.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.420+0000 7faddf8d9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fadd80a2340 addr v1:172.21.15.42:6789/0 2024-09-16T03:52:56.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.420+0000 7faddf8d9640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fadd80a2720 0x7fadd80a2b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:52:56.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.420+0000 7faddf8d9640 10 monclient(hunting): picked mon.noname-c con 0x7fadd80a2720 addr v2:172.21.15.154:3300/0 2024-09-16T03:52:56.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.420+0000 7faddf8d9640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fadd813a340 0x7fadd813c730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:52:56.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.420+0000 7faddf8d9640 10 monclient(hunting): picked mon.noname-a con 0x7fadd813a340 addr v2:172.21.15.42:3300/0 2024-09-16T03:52:56.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.420+0000 7faddf8d9640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fadd806cb00 con 0x7fadd80a2340 2024-09-16T03:52:56.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.420+0000 7faddf8d9640 10 monclient(hunting): start opening mon connection 2024-09-16T03:52:56.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.420+0000 7faddf8d9640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7fadd806c450 con 0x7fadd813a340 2024-09-16T03:52:56.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.420+0000 7faddf8d9640 10 monclient(hunting): start opening mon connection 2024-09-16T03:52:56.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.420+0000 7faddf8d9640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fadd8070b90 con 0x7fadd80a2720 2024-09-16T03:52:56.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.420+0000 7faddf8d9640 10 monclient(hunting): _renew_subs 2024-09-16T03:52:56.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.420+0000 7faddf8d9640 10 monclient(hunting): authenticate will time out at 1623.393830s 2024-09-16T03:52:56.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.420+0000 7faddd675640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fadd80a2340 0x7fadd8072180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:41164/0 (socket says 172.21.15.42:41164) 2024-09-16T03:52:56.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.420+0000 7faddde76640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fadd813a340 0x7fadd813c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:52:56.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.420+0000 7faddd675640 1 -- 172.21.15.42:0/632202848 learned_addr learned my addr 172.21.15.42:0/632202848 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:52:56.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.420+0000 7faddde76640 10 monclient(hunting): get_auth_request con 0x7fadd813a340 auth_method 0 2024-09-16T03:52:56.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.420+0000 7faddde76640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:52:56.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.420+0000 7faddde76640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:52:56.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.420+0000 7faddde76640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:52:56.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.420+0000 7faddde76640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:52:56.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.420+0000 7faddde76640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:52:56.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.420+0000 7faddde76640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:52:56.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.420+0000 7faddde76640 10 monclient(hunting): handle_auth_done global_id 824214 payload 274 2024-09-16T03:52:56.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.420+0000 7faddce74640 1 --2- 172.21.15.42:0/632202848 >> v2:172.21.15.154:3300/0 conn(0x7fadd80a2720 0x7fadd80a2b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:52:56.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.420+0000 7faddde76640 1 -- 172.21.15.42:0/632202848 >> v2:172.21.15.154:3300/0 conn(0x7fadd80a2720 msgr2=0x7fadd80a2b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:52:56.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.420+0000 7faddde76640 1 --2- 172.21.15.42:0/632202848 >> v2:172.21.15.154:3300/0 conn(0x7fadd80a2720 0x7fadd80a2b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:52:56.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.420+0000 7faddde76640 1 -- 172.21.15.42:0/632202848 >> v1:172.21.15.42:6789/0 conn(0x7fadd80a2340 legacy=0x7fadd8072180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:52:56.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.420+0000 7faddde76640 10 monclient: _finish_hunting 0 2024-09-16T03:52:56.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.420+0000 7faddde76640 1 monclient: found mon.noname-a 2024-09-16T03:52:56.432 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.420+0000 7faddde76640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:52:56.432 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.420+0000 7faddde76640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:52:56.432 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.420+0000 7faddde76640 1 -- 172.21.15.42:0/632202848 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fadd80a3070 con 0x7fadd813a340 2024-09-16T03:52:56.432 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.420+0000 7fadcbfff640 1 -- 172.21.15.42:0/632202848 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3200127711 0 0) 0x7fadd806cb00 con 0x7fadd80a2340 2024-09-16T03:52:56.432 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.420+0000 7fadcbfff640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T03:52:56.432 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.420+0000 7faddce74640 10 monclient: get_auth_request con 0x7fadd80a2720 auth_method 0 2024-09-16T03:52:56.432 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.420+0000 7faddce74640 1 --2- 172.21.15.42:0/632202848 >> v2:172.21.15.154:3300/0 conn(0x7fadd80a2720 0x7fadd80a2b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T03:52:56.439 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.424+0000 7faddde76640 1 --2- 172.21.15.42:0/632202848 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fadd813a340 0x7fadd813c730 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fadd8072ab0 tx=0x7fadd001c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19633c1a9e718c80 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:52:56.439 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.424+0000 7fadcbfff640 1 -- 172.21.15.42:0/632202848 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadd00a66c0 con 0x7fadd813a340 2024-09-16T03:52:56.439 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.424+0000 7fadcbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:52:56.439 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.424+0000 7fadcbfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:52:56.439 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.424+0000 7fadcbfff640 10 monclient: dump: 2024-09-16T03:52:56.439 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:52:56.440 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:52:56.440 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:52:56.440 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:52:56.440 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:52:56.440 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:52:56.440 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:52:56.440 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:52:56.440 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:52:56.440 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:52:56.440 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.424+0000 7fadcbfff640 10 monclient: _finish_auth 0 2024-09-16T03:52:56.440 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.424+0000 7fadcbfff640 10 monclient: _check_auth_tickets 2024-09-16T03:52:56.440 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.424+0000 7fadcbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:52:56.441 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.424+0000 7faddf8d9640 5 monclient: authenticate success, global_id 824214 2024-09-16T03:52:56.441 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.424+0000 7fadcbfff640 1 -- 172.21.15.42:0/632202848 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fadd00a6860 con 0x7fadd813a340 2024-09-16T03:52:56.441 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.424+0000 7faddf8d9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:52:56.441 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.424+0000 7fadcbfff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:52:56.441 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.424+0000 7faddf8d9640 10 monclient: get_monmap_and_config success 2024-09-16T03:52:56.441 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.424+0000 7faddf8d9640 4 set_mon_vals no callback set 2024-09-16T03:52:56.441 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.424+0000 7fadcbfff640 1 -- 172.21.15.42:0/632202848 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadd00a6c30 con 0x7fadd813a340 2024-09-16T03:52:56.441 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.424+0000 7fadcbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:52:56.441 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.424+0000 7fadcbfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:52:56.441 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.424+0000 7fadcbfff640 10 monclient: dump: 2024-09-16T03:52:56.441 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:52:56.442 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:52:56.442 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:52:56.442 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:52:56.442 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:52:56.442 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:52:56.442 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:52:56.442 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:52:56.442 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:52:56.442 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:52:56.442 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.424+0000 7faddf8d9640 10 monclient: shutdown 2024-09-16T03:52:56.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.424+0000 7faddf8d9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:52:56.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.424+0000 7faddf8d9640 1 -- 172.21.15.42:0/632202848 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fadd813a340 msgr2=0x7fadd813c730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:52:56.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.424+0000 7faddf8d9640 1 --2- 172.21.15.42:0/632202848 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fadd813a340 0x7fadd813c730 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fadd8072ab0 tx=0x7fadd001c7f0 comp rx=0 tx=0).stop 2024-09-16T03:52:56.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.424+0000 7faddf8d9640 1 -- 172.21.15.42:0/632202848 shutdown_connections 2024-09-16T03:52:56.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.424+0000 7faddf8d9640 1 --2- 172.21.15.42:0/632202848 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fadd813a340 0x7fadd813c730 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:52:56.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.424+0000 7faddf8d9640 1 --2- 172.21.15.42:0/632202848 >> v2:172.21.15.154:3300/0 conn(0x7fadd80a2720 0x7fadd80a2b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:52:56.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.424+0000 7faddf8d9640 1 -- 172.21.15.42:0/632202848 >> 172.21.15.42:0/632202848 conn(0x7fadd80a3430 msgr2=0x7fadd80a3810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:52:56.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.424+0000 7faddf8d9640 1 -- 172.21.15.42:0/632202848 shutdown_connections 2024-09-16T03:52:56.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.424+0000 7faddf8d9640 1 -- 172.21.15.42:0/632202848 wait complete. 2024-09-16T03:52:56.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.424+0000 7faddf8d9640 10 monclient: build_initial_monmap 2024-09-16T03:52:56.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.424+0000 7faddf8d9640 10 monclient: monmap: 2024-09-16T03:52:56.444 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:52:56.444 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:52:56.444 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:52:56.444 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:52:56.444 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:52:56.444 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:52:56.444 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:52:56.444 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:52:56.444 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:52:56.445 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:52:56.445 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.424+0000 7faddf8d9640 1 librados: starting msgr at 2024-09-16T03:52:56.445 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.424+0000 7faddf8d9640 1 librados: starting objecter 2024-09-16T03:52:56.445 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.424+0000 7faddf8d9640 1 Processor -- start 2024-09-16T03:52:56.445 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.424+0000 7faddf8d9640 1 -- start start 2024-09-16T03:52:56.445 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.424+0000 7faddf8d9640 1 librados: setting wanted keys 2024-09-16T03:52:56.445 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.424+0000 7faddf8d9640 1 librados: calling monclient init 2024-09-16T03:52:56.445 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.424+0000 7faddf8d9640 10 monclient: init 2024-09-16T03:52:56.445 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.424+0000 7faddf8d9640 10 monclient: _reopen_session rank -1 2024-09-16T03:52:56.446 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.424+0000 7faddf8d9640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:52:56.446 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.424+0000 7faddf8d9640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fadd80a2340 0x7fadd809f850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:52:56.446 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.424+0000 7faddf8d9640 10 monclient(hunting): picked mon.noname-c con 0x7fadd80a2340 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:52:56.446 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.424+0000 7faddf8d9640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fadd80a2720 0x7fadd8135320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:52:56.446 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.424+0000 7faddf8d9640 10 monclient(hunting): picked mon.noname-a con 0x7fadd80a2720 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:52:56.446 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.424+0000 7faddf8d9640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fadd813a340 0x7fadd8135860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:52:56.446 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.424+0000 7faddf8d9640 10 monclient(hunting): picked mon.noname-b con 0x7fadd813a340 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:52:56.446 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.424+0000 7faddf8d9640 10 monclient(hunting): start opening mon connection 2024-09-16T03:52:56.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.424+0000 7faddf8d9640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fadd813f820 con 0x7fadd80a2720 2024-09-16T03:52:56.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.424+0000 7faddf8d9640 10 monclient(hunting): start opening mon connection 2024-09-16T03:52:56.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.424+0000 7faddf8d9640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fadd813f6a0 con 0x7fadd813a340 2024-09-16T03:52:56.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.424+0000 7faddf8d9640 10 monclient(hunting): start opening mon connection 2024-09-16T03:52:56.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.424+0000 7faddf8d9640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fadd813f9a0 con 0x7fadd80a2340 2024-09-16T03:52:56.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.424+0000 7faddf8d9640 10 monclient(hunting): _renew_subs 2024-09-16T03:52:56.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.424+0000 7faddf8d9640 10 monclient(hunting): authenticate will time out at 1623.398760s 2024-09-16T03:52:56.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.424+0000 7faddce74640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fadd80a2720 0x7fadd8135320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:52:56.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.424+0000 7faddd675640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fadd80a2340 0x7fadd809f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:52:56.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.424+0000 7faddce74640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fadd80a2720 0x7fadd8135320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:38512/0 (socket says 172.21.15.42:38512) 2024-09-16T03:52:56.448 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.424+0000 7faddce74640 1 -- 172.21.15.42:0/601936016 learned_addr learned my addr 172.21.15.42:0/601936016 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:52:56.448 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.424+0000 7faddce74640 10 monclient(hunting): get_auth_request con 0x7fadd80a2720 auth_method 0 2024-09-16T03:52:56.448 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.424+0000 7faddce74640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:52:56.448 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.424+0000 7faddce74640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:52:56.448 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.424+0000 7faddce74640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:52:56.448 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.424+0000 7faddd675640 10 monclient(hunting): get_auth_request con 0x7fadd80a2340 auth_method 0 2024-09-16T03:52:56.448 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.424+0000 7faddd675640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:52:56.448 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.424+0000 7faddd675640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:52:56.449 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.424+0000 7faddd675640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:52:56.449 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.424+0000 7faddce74640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:52:56.449 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.424+0000 7faddce74640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:52:56.449 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.424+0000 7faddce74640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:52:56.449 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.424+0000 7faddd675640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:52:56.449 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.424+0000 7faddd675640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:52:56.449 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.424+0000 7faddd675640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:52:56.449 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.424+0000 7faddd675640 1 -- 172.21.15.42:0/601936016 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fadd80a2340 msgr2=0x7fadd809f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-16T03:52:56.449 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.424+0000 7faddd675640 1 -- 172.21.15.42:0/601936016 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fadd80a2340 msgr2=0x7fadd809f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T03:52:56.449 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.424+0000 7faddde76640 1 --2- 172.21.15.42:0/601936016 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fadd813a340 0x7fadd8135860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:52:56.449 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.424+0000 7faddd675640 1 --2- 172.21.15.42:0/601936016 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fadd80a2340 0x7fadd809f850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:52:56.450 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.424+0000 7faddd675640 1 --2- 172.21.15.42:0/601936016 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fadd80a2340 0x7fadd809f850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:52:56.450 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.424+0000 7faddde76640 10 monclient(hunting): get_auth_request con 0x7fadd813a340 auth_method 0 2024-09-16T03:52:56.450 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.424+0000 7faddde76640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:52:56.450 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.424+0000 7faddde76640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:52:56.450 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.424+0000 7faddde76640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:52:56.450 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.424+0000 7faddce74640 10 monclient(hunting): handle_auth_done global_id 824220 payload 824 2024-09-16T03:52:56.450 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.424+0000 7faddce74640 1 -- 172.21.15.42:0/601936016 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fadd80a2340 msgr2=0x7fadd809f850 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:52:56.450 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.424+0000 7faddce74640 1 --2- 172.21.15.42:0/601936016 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fadd80a2340 0x7fadd809f850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:52:56.450 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.424+0000 7faddce74640 1 -- 172.21.15.42:0/601936016 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fadd813a340 msgr2=0x7fadd8135860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:52:56.450 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.424+0000 7faddce74640 1 --2- 172.21.15.42:0/601936016 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fadd813a340 0x7fadd8135860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:52:56.450 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.424+0000 7faddce74640 10 monclient: _finish_hunting 0 2024-09-16T03:52:56.450 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.424+0000 7faddce74640 1 monclient: found mon.noname-a 2024-09-16T03:52:56.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.424+0000 7faddce74640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:52:56.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.424+0000 7faddce74640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:52:56.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.424+0000 7faddce74640 1 -- 172.21.15.42:0/601936016 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fadd80a04b0 con 0x7fadd80a2720 2024-09-16T03:52:56.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.424+0000 7faddce74640 1 --2- 172.21.15.42:0/601936016 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fadd80a2720 0x7fadd8135320 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fadcc00c7f0 tx=0x7fadcc00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:52:56.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.424+0000 7fadc9ffb640 10 client.?.objecter ms_handle_connect 0x7fadd80a2720 2024-09-16T03:52:56.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.424+0000 7fadc9ffb640 10 client.?.objecter resend_mon_ops 2024-09-16T03:52:56.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.424+0000 7fadc9ffb640 1 -- 172.21.15.42:0/601936016 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadcc0079d0 con 0x7fadd80a2720 2024-09-16T03:52:56.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.424+0000 7fadc9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:52:56.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.424+0000 7fadc9ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:52:56.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.424+0000 7fadc9ffb640 10 monclient: dump: 2024-09-16T03:52:56.451 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:52:56.451 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:52:56.452 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:52:56.452 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:52:56.452 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:52:56.452 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:52:56.452 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:52:56.452 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:52:56.452 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:52:56.452 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:52:56.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.424+0000 7fadc9ffb640 10 monclient: _finish_auth 0 2024-09-16T03:52:56.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.424+0000 7fadc9ffb640 10 monclient: _check_auth_tickets 2024-09-16T03:52:56.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.424+0000 7fadc9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:52:56.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.424+0000 7faddf8d9640 5 monclient: authenticate success, global_id 824220 2024-09-16T03:52:56.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.424+0000 7fadc9ffb640 1 -- 172.21.15.42:0/601936016 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fadcc007b70 con 0x7fadd80a2720 2024-09-16T03:52:56.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.424+0000 7faddf8d9640 10 monclient: _renew_subs 2024-09-16T03:52:56.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.424+0000 7faddf8d9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:52:56.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.424+0000 7faddf8d9640 1 -- 172.21.15.42:0/601936016 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fadd813c0d0 con 0x7fadd80a2720 2024-09-16T03:52:56.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.424+0000 7fadc9ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T03:52:56.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.424+0000 7fadc9ffb640 1 -- 172.21.15.42:0/601936016 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadcc0113f0 con 0x7fadd80a2720 2024-09-16T03:52:56.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.424+0000 7fadc9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:52:56.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.424+0000 7fadc9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:52:56.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.424+0000 7fadc9ffb640 10 monclient: dump: 2024-09-16T03:52:56.453 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:52:56.453 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:52:56.453 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:52:56.454 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:52:56.454 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:52:56.454 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:52:56.454 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:52:56.454 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:52:56.454 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:52:56.454 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:52:56.454 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.424+0000 7fadcaffd640 4 set_mon_vals no callback set 2024-09-16T03:52:56.454 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.424+0000 7faddf8d9640 10 client.824220.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:52:56.454 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.424+0000 7faddf8d9640 10 monclient: _renew_subs 2024-09-16T03:52:56.454 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.424+0000 7faddf8d9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:52:56.455 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.424+0000 7faddf8d9640 1 -- 172.21.15.42:0/601936016 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fadd806fbf0 con 0x7fadd80a2720 2024-09-16T03:52:56.455 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.424+0000 7faddf8d9640 1 librados: init done 2024-09-16T03:52:56.455 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.428+0000 7faddf8d9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:52:56.455 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.428+0000 7faddf8d9640 1 --2- 172.21.15.42:0/601936016 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fadd806e880 0x7fadd806ec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:52:56.455 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.428+0000 7faddf8d9640 10 monclient(hunting): start opening mon connection 2024-09-16T03:52:56.455 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.428+0000 7faddf8d9640 1 -- 172.21.15.42:0/601936016 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fadd80a6360 con 0x7fadd806e880 2024-09-16T03:52:56.455 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.428+0000 7fadc9ffb640 1 -- 172.21.15.42:0/601936016 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fadcc011590 con 0x7fadd80a2720 2024-09-16T03:52:56.455 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.428+0000 7fadc9ffb640 1 --2- 172.21.15.42:0/601936016 >> v2:172.21.15.154:6812/254845342 conn(0x7fadbc037810 0x7fadbc039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:52:56.455 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.428+0000 7fadc9ffb640 1 -- 172.21.15.42:0/601936016 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(345..345 src has 286..345) ==== 5769+0+0 (secure 0 0 0) 0x7fadcc049130 con 0x7fadd80a2720 2024-09-16T03:52:56.455 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.428+0000 7fadc9ffb640 10 client.824220.objecter ms_dispatch 0x7fadd8000ba0 osd_map(345..345 src has 286..345) 2024-09-16T03:52:56.455 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.428+0000 7fadc9ffb640 3 client.824220.objecter handle_osd_map got epochs [345,345] > 0 2024-09-16T03:52:56.455 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.428+0000 7fadc9ffb640 3 client.824220.objecter handle_osd_map decoding full epoch 345 2024-09-16T03:52:56.456 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.428+0000 7faddd675640 1 --2- 172.21.15.42:0/601936016 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fadd806e880 0x7fadd806ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:52:56.456 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.428+0000 7faddd675640 10 monclient: get_auth_request con 0x7fadd806e880 auth_method 0 2024-09-16T03:52:56.456 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.428+0000 7faddd675640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:52:56.456 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.428+0000 7fadc9ffb640 20 client.824220.objecter dump_active .. 0 homeless 2024-09-16T03:52:56.456 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.428+0000 7faddd675640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:52:56.456 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.428+0000 7faddd675640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:52:56.456 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.428+0000 7faddd675640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:52:56.456 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.428+0000 7faddd675640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:52:56.456 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.428+0000 7faddd675640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:52:56.456 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.428+0000 7faddde76640 1 --2- 172.21.15.42:0/601936016 >> v2:172.21.15.154:6812/254845342 conn(0x7fadbc037810 0x7fadbc039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:52:56.456 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.428+0000 7faddde76640 10 monclient: get_auth_request con 0x7fadbc037810 auth_method 0 2024-09-16T03:52:56.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.428+0000 7faddd675640 10 monclient(hunting): handle_auth_done global_id 794479 payload 824 2024-09-16T03:52:56.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.428+0000 7faddd675640 1 -- 172.21.15.42:0/601936016 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fadd80739f0 con 0x7fadd806e880 2024-09-16T03:52:56.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.428+0000 7faddd675640 1 --2- 172.21.15.42:0/601936016 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fadd806e880 0x7fadd806ec60 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fadd4000f70 tx=0x7fadd40019a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:52:56.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.428+0000 7fadc9ffb640 10 client.824220.objecter ms_handle_connect 0x7fadd806e880 2024-09-16T03:52:56.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.428+0000 7fadc9ffb640 10 client.824220.objecter resend_mon_ops 2024-09-16T03:52:56.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.428+0000 7fadc9ffb640 10 client.824220.objecter ms_handle_connect 0x7fadbc037810 2024-09-16T03:52:56.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.428+0000 7faddde76640 1 --2- 172.21.15.42:0/601936016 >> v2:172.21.15.154:6812/254845342 conn(0x7fadbc037810 0x7fadbc039cd0 secure :-1 s=READY pgs=3584 cs=0 l=1 rev1=1 crypto rx=0x7fadd00af000 tx=0x7fadd00a7000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:52:56.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.428+0000 7fadc9ffb640 1 -- 172.21.15.42:0/601936016 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadd400f5e0 con 0x7fadd806e880 2024-09-16T03:52:56.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.428+0000 7fadc9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:52:56.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.428+0000 7fadc9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:52:56.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.428+0000 7fadc9ffb640 10 monclient: dump: 2024-09-16T03:52:56.458 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:52:56.458 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:52:56.458 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:52:56.458 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:52:56.458 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:52:56.458 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:52:56.458 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:52:56.458 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:52:56.458 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:52:56.458 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:52:56.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.428+0000 7fadc9ffb640 1 -- 172.21.15.42:0/601936016 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fadd4018070 con 0x7fadd806e880 2024-09-16T03:52:56.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.428+0000 7fadc9ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:52:56.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.428+0000 7fadc9ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:52:56.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.428+0000 7fadc9ffb640 1 -- 172.21.15.42:0/601936016 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fadd806e880 msgr2=0x7fadd806ec60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:52:56.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.428+0000 7fadc9ffb640 1 --2- 172.21.15.42:0/601936016 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fadd806e880 0x7fadd806ec60 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fadd4000f70 tx=0x7fadd40019a0 comp rx=0 tx=0).stop 2024-09-16T03:52:56.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.428+0000 7fadc9ffb640 1 -- 172.21.15.42:0/601936016 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fadd806e880 msgr2=0x7fadd806ec60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:52:56.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.428+0000 7fadc9ffb640 1 --2- 172.21.15.42:0/601936016 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fadd806e880 0x7fadd806ec60 secure :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fadd4000f70 tx=0x7fadd40019a0 comp rx=0 tx=0).stop 2024-09-16T03:52:56.461 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.436+0000 7faddf8d9640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:52:56.461 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.436+0000 7faddf8d9640 1 --2- 172.21.15.42:0/601936016 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fadd806f1a0 0x7fadd809ba50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:52:56.461 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.436+0000 7faddf8d9640 10 monclient(hunting): start opening mon connection 2024-09-16T03:52:56.461 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.436+0000 7faddf8d9640 1 -- 172.21.15.42:0/601936016 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fadd80a2b30 con 0x7fadd806f1a0 2024-09-16T03:52:56.461 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.436+0000 7faddd675640 1 --2- 172.21.15.42:0/601936016 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fadd806f1a0 0x7fadd809ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:52:56.461 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.436+0000 7faddd675640 10 monclient: get_auth_request con 0x7fadd806f1a0 auth_method 0 2024-09-16T03:52:56.462 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.436+0000 7faddd675640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:52:56.462 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.436+0000 7faddd675640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:52:56.462 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.436+0000 7faddd675640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:52:56.462 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.436+0000 7faddd675640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:52:56.462 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.436+0000 7faddd675640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:52:56.462 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.436+0000 7faddd675640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:52:56.462 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.436+0000 7faddd675640 10 monclient(hunting): handle_auth_done global_id 794482 payload 824 2024-09-16T03:52:56.462 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.436+0000 7faddd675640 1 -- 172.21.15.42:0/601936016 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fadd8121510 con 0x7fadd806f1a0 2024-09-16T03:52:56.462 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.436+0000 7faddd675640 1 --2- 172.21.15.42:0/601936016 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fadd806f1a0 0x7fadd809ba50 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fadd4000f70 tx=0x7fadd4001090 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:52:56.462 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.436+0000 7fadc9ffb640 10 client.824220.objecter ms_handle_connect 0x7fadd806f1a0 2024-09-16T03:52:56.462 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.436+0000 7fadc9ffb640 10 client.824220.objecter resend_mon_ops 2024-09-16T03:52:56.462 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.436+0000 7fadc9ffb640 1 -- 172.21.15.42:0/601936016 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadd401e050 con 0x7fadd806f1a0 2024-09-16T03:52:56.463 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.436+0000 7fadc9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:52:56.463 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.436+0000 7fadc9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:52:56.463 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.436+0000 7fadc9ffb640 10 monclient: dump: 2024-09-16T03:52:56.463 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:52:56.463 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:52:56.463 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:52:56.463 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:52:56.463 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:52:56.463 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:52:56.463 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:52:56.463 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:52:56.463 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:52:56.464 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:52:56.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.436+0000 7fadc9ffb640 1 -- 172.21.15.42:0/601936016 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4712 (secure 0 0 0) 0x7fadd401e050 con 0x7fadd806f1a0 2024-09-16T03:52:56.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.436+0000 7fadc9ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:52:56.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.436+0000 7fadc9ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:52:56.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.436+0000 7fadc9ffb640 1 -- 172.21.15.42:0/601936016 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fadd806f1a0 msgr2=0x7fadd809ba50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:52:56.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.436+0000 7fadc9ffb640 1 --2- 172.21.15.42:0/601936016 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fadd806f1a0 0x7fadd809ba50 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fadd4000f70 tx=0x7fadd4001090 comp rx=0 tx=0).stop 2024-09-16T03:52:56.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.436+0000 7fadc9ffb640 1 -- 172.21.15.42:0/601936016 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fadd806f1a0 msgr2=0x7fadd809ba50 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:52:56.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.436+0000 7fadc9ffb640 1 --2- 172.21.15.42:0/601936016 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fadd806f1a0 0x7fadd809ba50 secure :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fadd4000f70 tx=0x7fadd4001090 comp rx=0 tx=0).stop 2024-09-16T03:52:56.464 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:52:56.465 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:52:56.465 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:52:56.465 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T03:52:56.465 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 94, 2024-09-16T03:52:56.465 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 218284, 2024-09-16T03:52:56.465 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:52:56.465 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:52:56.465 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T03:52:56.465 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T03:52:56.465 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:52:56.465 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 1, 2024-09-16T03:52:56.466 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:52:56.466 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:52:56.466 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:52:56.466 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:52:56.466 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:52:56.466 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:52:56.466 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:52:56.466 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:52:56.466 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:52:56.466 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:52:56.466 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:52:56.466 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:52:56.467 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:52:56.467 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:52:56.467 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:52:56.467 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:52:56.467 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:52:56.467 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:52:56.467 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:52:56.467 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:52:56.467 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:52:56.467 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:52:56.467 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:52:56.468 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:52:56.468 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:52:56.468 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:52:56.468 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:52:56.468 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:52:56.468 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:52:56.468 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:52:56.468 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:52:56.468 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:52:56.468 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:52:56.468 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:52:56.468 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:52:56.469 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:52:56.469 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:52:56.469 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:52:56.469 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:52:56.469 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:52:56.469 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:52:56.469 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:52:56.469 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:52:56.469 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:52:56.469 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:52:56.469 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:52:56.470 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:52:56.470 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:52:56.470 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:52:56.470 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:52:56.470 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:52:56.470 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:52:56.470 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:52:56.470 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:52:56.470 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:52:56.470 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:52:56.470 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:52:56.470 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:52:56.471 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:52:56.471 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:52:56.471 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:52:56.471 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:52:56.471 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:52:56.471 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:52:56.471 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:52:56.471 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:52:56.471 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:52:56.471 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:52:56.471 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:52:56.471 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:52:56.471 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:52:56.472 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:52:56.472 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:52:56.472 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:52:56.472 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:52:56.472 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:52:56.472 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:52:56.472 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:52:56.472 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:52:56.472 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:52:56.472 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:52:56.472 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:52:56.472 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:52:56.473 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:52:56.473 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:52:56.473 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:52:56.473 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:52:56.473 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:52:56.473 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:52:56.473 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:52:56.473 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:52:56.473 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:52:56.473 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:52:56.473 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:52:56.473 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:52:56.474 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:52:56.474 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:52:56.474 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:52:56.474 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:52:56.474 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:52:56.474 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:52:56.474 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:52:56.474 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:52:56.474 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:52:56.474 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:52:56.474 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:52:56.474 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:52:56.475 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:52:56.475 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:52:56.475 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:52:56.475 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:52:56.475 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:52:56.475 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:52:56.475 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:52:56.475 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:52:56.475 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:52:56.475 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:52:56.475 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:52:56.476 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:52:56.476 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:52:56.476 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:52:56.476 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:52:56.476 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:52:56.476 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:52:56.476 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:52:56.476 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:52:56.476 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:52:56.476 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:52:56.476 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:52:56.476 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:52:56.477 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:52:56.477 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:52:56.477 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:52:56.477 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:52:56.477 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:52:56.477 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:52:56.477 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:52:56.477 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:52:56.477 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:52:56.477 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:52:56.477 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:52:56.477 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T03:52:56.478 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:52:56.478 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:52:56.478 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:52:56.478 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:52:56.478 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:52:56.478 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:52:56.478 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:52:56.478 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:52:56.478 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:52:56.478 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:52:56.478 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 2 2024-09-16T03:52:56.478 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:52:56.479 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:52:56.479 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:52:56.479 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:52:56.479 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:52:56.481 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.440+0000 7faddf8d9640 10 librados: watch_flush enter 2024-09-16T03:52:56.481 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.440+0000 7faddf8d9640 10 librados: watch_flush exit 2024-09-16T03:52:56.481 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.440+0000 7faddf8d9640 20 client.824220.objecter shutdown clearing up homeless session... 2024-09-16T03:52:56.481 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.440+0000 7faddf8d9640 10 client.824220.objecter successfully canceled tick 2024-09-16T03:52:56.481 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.440+0000 7faddf8d9640 1 -- 172.21.15.42:0/601936016 >> v2:172.21.15.154:6812/254845342 conn(0x7fadbc037810 msgr2=0x7fadbc039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:52:56.481 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.440+0000 7faddf8d9640 1 --2- 172.21.15.42:0/601936016 >> v2:172.21.15.154:6812/254845342 conn(0x7fadbc037810 0x7fadbc039cd0 secure :-1 s=READY pgs=3584 cs=0 l=1 rev1=1 crypto rx=0x7fadd00af000 tx=0x7fadd00a7000 comp rx=0 tx=0).stop 2024-09-16T03:52:56.481 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.440+0000 7faddf8d9640 10 monclient: shutdown 2024-09-16T03:52:56.481 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.440+0000 7faddf8d9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:52:56.481 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.440+0000 7faddf8d9640 1 -- 172.21.15.42:0/601936016 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fadd80a2720 msgr2=0x7fadd8135320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:52:56.481 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.440+0000 7faddf8d9640 1 --2- 172.21.15.42:0/601936016 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fadd80a2720 0x7fadd8135320 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fadcc00c7f0 tx=0x7fadcc00ccb0 comp rx=0 tx=0).stop 2024-09-16T03:52:56.481 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.440+0000 7faddde76640 1 -- 172.21.15.42:0/601936016 reap_dead start 2024-09-16T03:52:56.482 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.440+0000 7faddde76640 1 -- 172.21.15.42:0/601936016 reap_dead start 2024-09-16T03:52:56.482 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.440+0000 7faddf8d9640 1 -- 172.21.15.42:0/601936016 shutdown_connections 2024-09-16T03:52:56.482 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.440+0000 7faddf8d9640 1 -- 172.21.15.42:0/601936016 >> 172.21.15.42:0/601936016 conn(0x7fadd80a3430 msgr2=0x7fadd80a5940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:52:56.482 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.440+0000 7faddf8d9640 1 -- 172.21.15.42:0/601936016 shutdown_connections 2024-09-16T03:52:56.482 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.440+0000 7faddf8d9640 1 -- 172.21.15.42:0/601936016 wait complete. 2024-09-16T03:52:56.482 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.440+0000 7faddf8d9640 1 librados: shutdown 2024-09-16T03:52:56.482 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:52:56.482 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T03:52:56.483 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T03:52:56.596 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.588+0000 7efd5d7d9640 10 monclient: get_monmap_and_config 2024-09-16T03:52:56.596 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.588+0000 7efd5d7d9640 10 monclient: build_initial_monmap 2024-09-16T03:52:56.596 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.588+0000 7efd5d7d9640 10 monclient: monmap: 2024-09-16T03:52:56.597 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:52:56.597 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:52:56.597 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:52:56.595161+0000 2024-09-16T03:52:56.597 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:52:56.595161+0000 2024-09-16T03:52:56.597 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:52:56.597 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:52:56.597 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:52:56.597 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:52:56.597 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:52:56.597 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:52:56.597 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:52:56.597 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:52:56.600 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.588+0000 7efd5d7d9640 1 Processor -- start 2024-09-16T03:52:56.600 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.588+0000 7efd5d7d9640 1 -- start start 2024-09-16T03:52:56.600 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.588+0000 7efd5d7d9640 10 monclient: init 2024-09-16T03:52:56.600 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.592+0000 7efd5d7d9640 10 monclient: _reopen_session rank -1 2024-09-16T03:52:56.600 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.592+0000 7efd5d7d9640 10 monclient: _add_conns ranks=[2,3,0,1,4] 2024-09-16T03:52:56.600 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.592+0000 7efd5d7d9640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7efd580a1330 0x7efd580a1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:52:56.600 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.592+0000 7efd5d7d9640 10 monclient(hunting): picked mon.noname-b con 0x7efd580a1330 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:52:56.600 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.592+0000 7efd5d7d9640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7efd5806dac0 0x7efd580a1c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:52:56.601 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.592+0000 7efd5d7d9640 10 monclient(hunting): picked mon.noname-a con 0x7efd5806dac0 addr v2:172.21.15.42:3300/0 2024-09-16T03:52:56.601 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.592+0000 7efd5d7d9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7efd58135100 addr v1:172.21.15.42:6789/0 2024-09-16T03:52:56.601 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.592+0000 7efd5d7d9640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efd5809c7d0 con 0x7efd58135100 2024-09-16T03:52:56.601 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.592+0000 7efd5d7d9640 10 monclient(hunting): start opening mon connection 2024-09-16T03:52:56.601 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.592+0000 7efd5d7d9640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7efd5809c090 con 0x7efd5806dac0 2024-09-16T03:52:56.601 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.592+0000 7efd5d7d9640 10 monclient(hunting): start opening mon connection 2024-09-16T03:52:56.601 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.592+0000 7efd5d7d9640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7efd5809f610 con 0x7efd580a1330 2024-09-16T03:52:56.601 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.592+0000 7efd5d7d9640 10 monclient(hunting): _renew_subs 2024-09-16T03:52:56.601 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.592+0000 7efd5d7d9640 10 monclient(hunting): authenticate will time out at 1623.564648s 2024-09-16T03:52:56.601 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.592+0000 7efd56ffd640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7efd580a1330 0x7efd580a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:52:56.601 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.592+0000 7efd577fe640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7efd58135100 0x7efd58135550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:41180/0 (socket says 172.21.15.42:41180) 2024-09-16T03:52:56.602 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.592+0000 7efd577fe640 1 -- 172.21.15.42:0/1822334738 learned_addr learned my addr 172.21.15.42:0/1822334738 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:52:56.602 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.592+0000 7efd56ffd640 10 monclient(hunting): get_auth_request con 0x7efd580a1330 auth_method 0 2024-09-16T03:52:56.602 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.592+0000 7efd567fc640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7efd5806dac0 0x7efd580a1c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:52:56.602 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.592+0000 7efd56ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:52:56.602 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.592+0000 7efd56ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:52:56.602 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.592+0000 7efd56ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:52:56.602 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.592+0000 7efd567fc640 10 monclient(hunting): get_auth_request con 0x7efd5806dac0 auth_method 0 2024-09-16T03:52:56.602 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.592+0000 7efd567fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:52:56.602 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.592+0000 7efd567fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:52:56.602 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.592+0000 7efd567fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:52:56.602 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.592+0000 7efd567fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:52:56.602 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.592+0000 7efd567fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:52:56.603 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.592+0000 7efd567fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:52:56.603 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.592+0000 7efd56ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:52:56.603 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.592+0000 7efd56ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:52:56.603 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.592+0000 7efd56ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:52:56.603 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.592+0000 7efd567fc640 10 monclient(hunting): handle_auth_done global_id 824223 payload 274 2024-09-16T03:52:56.603 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.592+0000 7efd567fc640 1 -- 172.21.15.42:0/1822334738 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7efd580a1330 msgr2=0x7efd580a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:52:56.603 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.592+0000 7efd567fc640 1 --2- 172.21.15.42:0/1822334738 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7efd580a1330 0x7efd580a1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:52:56.603 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.592+0000 7efd567fc640 1 -- 172.21.15.42:0/1822334738 >> v1:172.21.15.42:6789/0 conn(0x7efd58135100 legacy=0x7efd58135550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:52:56.603 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.592+0000 7efd567fc640 10 monclient: _finish_hunting 0 2024-09-16T03:52:56.603 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.592+0000 7efd567fc640 1 monclient: found mon.noname-a 2024-09-16T03:52:56.603 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.592+0000 7efd567fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:52:56.603 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.592+0000 7efd567fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:52:56.604 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.592+0000 7efd567fc640 1 -- 172.21.15.42:0/1822334738 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd5809f8f0 con 0x7efd5806dac0 2024-09-16T03:52:56.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.592+0000 7efd567fc640 1 --2- 172.21.15.42:0/1822334738 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efd5806dac0 0x7efd580a1c50 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7efd40009d90 tx=0x7efd4001c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=127bfb10495058e2 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:52:56.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.592+0000 7efd55ffb640 1 -- 172.21.15.42:0/1822334738 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd400b2070 con 0x7efd5806dac0 2024-09-16T03:52:56.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.592+0000 7efd55ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:52:56.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.592+0000 7efd55ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:52:56.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.592+0000 7efd55ffb640 10 monclient: dump: 2024-09-16T03:52:56.610 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:52:56.610 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:52:56.611 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:52:56.611 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:52:56.611 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:52:56.611 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:52:56.611 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:52:56.611 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:52:56.611 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:52:56.611 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:52:56.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.592+0000 7efd55ffb640 10 monclient: _finish_auth 0 2024-09-16T03:52:56.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.592+0000 7efd55ffb640 10 monclient: _check_auth_tickets 2024-09-16T03:52:56.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.592+0000 7efd55ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:52:56.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.592+0000 7efd55ffb640 1 -- 172.21.15.42:0/1822334738 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efd400a5920 con 0x7efd5806dac0 2024-09-16T03:52:56.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.592+0000 7efd5d7d9640 5 monclient: authenticate success, global_id 824223 2024-09-16T03:52:56.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.592+0000 7efd5d7d9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:52:56.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.592+0000 7efd55ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T03:52:56.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.592+0000 7efd5d7d9640 10 monclient: get_monmap_and_config success 2024-09-16T03:52:56.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.592+0000 7efd55ffb640 1 -- 172.21.15.42:0/1822334738 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd400a5cf0 con 0x7efd5806dac0 2024-09-16T03:52:56.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.592+0000 7efd55ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:52:56.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.592+0000 7efd5d7d9640 4 set_mon_vals no callback set 2024-09-16T03:52:56.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.592+0000 7efd55ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:52:56.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.592+0000 7efd55ffb640 10 monclient: dump: 2024-09-16T03:52:56.612 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:52:56.612 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:52:56.612 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:52:56.612 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:52:56.612 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:52:56.612 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:52:56.613 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:52:56.613 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:52:56.613 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:52:56.613 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:52:56.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.596+0000 7efd5d7d9640 10 monclient: shutdown 2024-09-16T03:52:56.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.596+0000 7efd5d7d9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:52:56.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.596+0000 7efd5d7d9640 1 -- 172.21.15.42:0/1822334738 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efd5806dac0 msgr2=0x7efd580a1c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:52:56.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.596+0000 7efd5d7d9640 1 --2- 172.21.15.42:0/1822334738 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efd5806dac0 0x7efd580a1c50 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7efd40009d90 tx=0x7efd4001c950 comp rx=0 tx=0).stop 2024-09-16T03:52:56.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.596+0000 7efd5d7d9640 1 -- 172.21.15.42:0/1822334738 shutdown_connections 2024-09-16T03:52:56.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.596+0000 7efd5d7d9640 1 --2- 172.21.15.42:0/1822334738 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efd5806dac0 0x7efd580a1c50 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:52:56.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.596+0000 7efd5d7d9640 1 --2- 172.21.15.42:0/1822334738 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7efd580a1330 0x7efd580a1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:52:56.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.596+0000 7efd5d7d9640 1 -- 172.21.15.42:0/1822334738 >> 172.21.15.42:0/1822334738 conn(0x7efd580a2420 msgr2=0x7efd580a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:52:56.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.596+0000 7efd5d7d9640 1 -- 172.21.15.42:0/1822334738 shutdown_connections 2024-09-16T03:52:56.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.596+0000 7efd5d7d9640 1 -- 172.21.15.42:0/1822334738 wait complete. 2024-09-16T03:52:56.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.596+0000 7efd5d7d9640 10 monclient: build_initial_monmap 2024-09-16T03:52:56.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.596+0000 7efd5d7d9640 10 monclient: monmap: 2024-09-16T03:52:56.614 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:52:56.614 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:52:56.614 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:52:56.614 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:52:56.614 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:52:56.614 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:52:56.614 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:52:56.614 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:52:56.614 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:52:56.614 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:52:56.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.596+0000 7efd5d7d9640 1 librados: starting msgr at 2024-09-16T03:52:56.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.596+0000 7efd5d7d9640 1 librados: starting objecter 2024-09-16T03:52:56.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.596+0000 7efd5d7d9640 1 Processor -- start 2024-09-16T03:52:56.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.596+0000 7efd5d7d9640 1 -- start start 2024-09-16T03:52:56.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.596+0000 7efd5d7d9640 1 librados: setting wanted keys 2024-09-16T03:52:56.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.596+0000 7efd5d7d9640 1 librados: calling monclient init 2024-09-16T03:52:56.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.596+0000 7efd5d7d9640 10 monclient: init 2024-09-16T03:52:56.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.596+0000 7efd5d7d9640 10 monclient: _reopen_session rank -1 2024-09-16T03:52:56.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.596+0000 7efd5d7d9640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T03:52:56.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.596+0000 7efd5d7d9640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efd580a1330 0x7efd581cd020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:52:56.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.596+0000 7efd5d7d9640 10 monclient(hunting): picked mon.noname-a con 0x7efd580a1330 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:52:56.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.596+0000 7efd5d7d9640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efd58135100 0x7efd5813b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:52:56.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.596+0000 7efd5d7d9640 10 monclient(hunting): picked mon.noname-b con 0x7efd58135100 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:52:56.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.596+0000 7efd5d7d9640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7efd58134320 0x7efd5811bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:52:56.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.596+0000 7efd5d7d9640 10 monclient(hunting): picked mon.noname-c con 0x7efd58134320 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:52:56.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.596+0000 7efd5d7d9640 10 monclient(hunting): start opening mon connection 2024-09-16T03:52:56.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.596+0000 7efd5d7d9640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7efd5813fd00 con 0x7efd580a1330 2024-09-16T03:52:56.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.596+0000 7efd5d7d9640 10 monclient(hunting): start opening mon connection 2024-09-16T03:52:56.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.596+0000 7efd5d7d9640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7efd5813fb80 con 0x7efd58135100 2024-09-16T03:52:56.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.596+0000 7efd5d7d9640 10 monclient(hunting): start opening mon connection 2024-09-16T03:52:56.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.596+0000 7efd5d7d9640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7efd5813fe80 con 0x7efd58134320 2024-09-16T03:52:56.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.596+0000 7efd5d7d9640 10 monclient(hunting): _renew_subs 2024-09-16T03:52:56.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.596+0000 7efd5d7d9640 10 monclient(hunting): authenticate will time out at 1623.570025s 2024-09-16T03:52:56.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.596+0000 7efd56ffd640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efd580a1330 0x7efd581cd020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:52:56.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.596+0000 7efd56ffd640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efd580a1330 0x7efd581cd020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:38534/0 (socket says 172.21.15.42:38534) 2024-09-16T03:52:56.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.596+0000 7efd577fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7efd58134320 0x7efd5811bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:52:56.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.596+0000 7efd56ffd640 1 -- 172.21.15.42:0/3359639834 learned_addr learned my addr 172.21.15.42:0/3359639834 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:52:56.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.596+0000 7efd56ffd640 10 monclient(hunting): get_auth_request con 0x7efd580a1330 auth_method 0 2024-09-16T03:52:56.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.596+0000 7efd56ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:52:56.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.596+0000 7efd56ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:52:56.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.596+0000 7efd56ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:52:56.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.596+0000 7efd577fe640 10 monclient(hunting): get_auth_request con 0x7efd58134320 auth_method 0 2024-09-16T03:52:56.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.596+0000 7efd577fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:52:56.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.596+0000 7efd577fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:52:56.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.596+0000 7efd577fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:52:56.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.596+0000 7efd577fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:52:56.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.596+0000 7efd577fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:52:56.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.596+0000 7efd577fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:52:56.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.596+0000 7efd56ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:52:56.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.596+0000 7efd56ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:52:56.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.596+0000 7efd56ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:52:56.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.596+0000 7efd567fc640 1 --2- 172.21.15.42:0/3359639834 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efd58135100 0x7efd5813b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:52:56.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.596+0000 7efd577fe640 10 monclient(hunting): handle_auth_done global_id 834128 payload 824 2024-09-16T03:52:56.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.596+0000 7efd577fe640 1 -- 172.21.15.42:0/3359639834 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efd58135100 msgr2=0x7efd5813b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:52:56.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.596+0000 7efd577fe640 1 --2- 172.21.15.42:0/3359639834 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efd58135100 0x7efd5813b2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:52:56.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.596+0000 7efd577fe640 1 -- 172.21.15.42:0/3359639834 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efd580a1330 msgr2=0x7efd581cd020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:52:56.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.596+0000 7efd577fe640 1 --2- 172.21.15.42:0/3359639834 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efd580a1330 0x7efd581cd020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:52:56.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.596+0000 7efd577fe640 10 monclient: _finish_hunting 0 2024-09-16T03:52:56.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.596+0000 7efd577fe640 1 monclient: found mon.noname-c 2024-09-16T03:52:56.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.596+0000 7efd577fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:52:56.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.596+0000 7efd577fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:52:56.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.596+0000 7efd577fe640 1 -- 172.21.15.42:0/3359639834 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd58071cb0 con 0x7efd58134320 2024-09-16T03:52:56.619 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.596+0000 7efd567fc640 10 monclient: get_auth_request con 0x7efd58135100 auth_method 0 2024-09-16T03:52:56.619 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.596+0000 7efd567fc640 1 --2- 172.21.15.42:0/3359639834 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efd58135100 0x7efd5813b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T03:52:56.619 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.596+0000 7efd577fe640 1 --2- 172.21.15.42:0/3359639834 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7efd58134320 0x7efd5811bf70 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7efd5000c3f0 tx=0x7efd5000c8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:52:56.619 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.596+0000 7efd4f7fe640 10 client.?.objecter ms_handle_connect 0x7efd58134320 2024-09-16T03:52:56.619 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.596+0000 7efd4f7fe640 10 client.?.objecter resend_mon_ops 2024-09-16T03:52:56.619 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.596+0000 7efd4f7fe640 1 -- 172.21.15.42:0/3359639834 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd50017070 con 0x7efd58134320 2024-09-16T03:52:56.619 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.596+0000 7efd4f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:52:56.619 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.596+0000 7efd4f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:52:56.619 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.596+0000 7efd4f7fe640 10 monclient: dump: 2024-09-16T03:52:56.619 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:52:56.619 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:52:56.619 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:52:56.619 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:52:56.619 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:52:56.620 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:52:56.620 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:52:56.620 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:52:56.620 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:52:56.620 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:52:56.620 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.596+0000 7efd4f7fe640 10 monclient: _finish_auth 0 2024-09-16T03:52:56.620 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.596+0000 7efd4f7fe640 10 monclient: _check_auth_tickets 2024-09-16T03:52:56.620 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.596+0000 7efd4f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:52:56.620 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.596+0000 7efd4f7fe640 1 -- 172.21.15.42:0/3359639834 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efd50005cf0 con 0x7efd58134320 2024-09-16T03:52:56.620 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.596+0000 7efd4f7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:52:56.620 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.596+0000 7efd4f7fe640 1 -- 172.21.15.42:0/3359639834 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd500072d0 con 0x7efd58134320 2024-09-16T03:52:56.620 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.596+0000 7efd5d7d9640 5 monclient: authenticate success, global_id 834128 2024-09-16T03:52:56.620 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.596+0000 7efd5d7d9640 10 monclient: _renew_subs 2024-09-16T03:52:56.621 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.596+0000 7efd5d7d9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:52:56.621 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.596+0000 7efd5d7d9640 1 -- 172.21.15.42:0/3359639834 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd580a5210 con 0x7efd58134320 2024-09-16T03:52:56.621 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.596+0000 7efd557fa640 4 set_mon_vals no callback set 2024-09-16T03:52:56.621 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.596+0000 7efd4f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:52:56.621 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.596+0000 7efd4f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:52:56.621 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.596+0000 7efd4f7fe640 10 monclient: dump: 2024-09-16T03:52:56.621 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:52:56.621 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:52:56.621 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:52:56.621 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:52:56.621 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:52:56.621 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:52:56.621 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:52:56.621 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:52:56.621 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:52:56.622 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:52:56.622 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.596+0000 7efd5d7d9640 10 client.834128.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:52:56.622 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.596+0000 7efd5d7d9640 10 monclient: _renew_subs 2024-09-16T03:52:56.622 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.596+0000 7efd5d7d9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:52:56.622 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.596+0000 7efd5d7d9640 1 -- 172.21.15.42:0/3359639834 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7efd5813bf80 con 0x7efd58134320 2024-09-16T03:52:56.622 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.596+0000 7efd5d7d9640 1 librados: init done 2024-09-16T03:52:56.622 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.596+0000 7efd5d7d9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:52:56.622 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.596+0000 7efd5d7d9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:52:56.622 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.596+0000 7efd5d7d9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:52:56.622 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.596+0000 7efd5d7d9640 1 -- 172.21.15.42:0/3359639834 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd58135510 con 0x7efd58134320 2024-09-16T03:52:56.622 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.596+0000 7efd4f7fe640 1 -- 172.21.15.42:0/3359639834 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7efd5001a070 con 0x7efd58134320 2024-09-16T03:52:56.622 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.600+0000 7efd4f7fe640 1 --2- 172.21.15.42:0/3359639834 >> v2:172.21.15.154:6812/254845342 conn(0x7efd3c037810 0x7efd3c039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:52:56.622 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.600+0000 7efd4f7fe640 1 -- 172.21.15.42:0/3359639834 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(345..345 src has 286..345) ==== 5769+0+0 (secure 0 0 0) 0x7efd500481f0 con 0x7efd58134320 2024-09-16T03:52:56.622 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.600+0000 7efd4f7fe640 10 client.834128.objecter ms_dispatch 0x7efd58000ba0 osd_map(345..345 src has 286..345) 2024-09-16T03:52:56.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.600+0000 7efd4f7fe640 3 client.834128.objecter handle_osd_map got epochs [345,345] > 0 2024-09-16T03:52:56.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.600+0000 7efd4f7fe640 3 client.834128.objecter handle_osd_map decoding full epoch 345 2024-09-16T03:52:56.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.600+0000 7efd4f7fe640 20 client.834128.objecter dump_active .. 0 homeless 2024-09-16T03:52:56.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.600+0000 7efd56ffd640 1 --2- 172.21.15.42:0/3359639834 >> v2:172.21.15.154:6812/254845342 conn(0x7efd3c037810 0x7efd3c039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:52:56.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.600+0000 7efd56ffd640 10 monclient: get_auth_request con 0x7efd3c037810 auth_method 0 2024-09-16T03:52:56.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.600+0000 7efd56ffd640 1 --2- 172.21.15.42:0/3359639834 >> v2:172.21.15.154:6812/254845342 conn(0x7efd3c037810 0x7efd3c039cd0 secure :-1 s=READY pgs=3585 cs=0 l=1 rev1=1 crypto rx=0x7efd48009c20 tx=0x7efd48009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:52:56.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.600+0000 7efd4f7fe640 10 client.834128.objecter ms_handle_connect 0x7efd3c037810 2024-09-16T03:52:56.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.604+0000 7efd4f7fe640 1 -- 172.21.15.42:0/3359639834 <== mon.2 v2:172.21.15.42:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7efd500076b0 con 0x7efd58134320 2024-09-16T03:52:56.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.604+0000 7efd4f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:52:56.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.604+0000 7efd4f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:52:56.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.824+0000 7efd5d7d9640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T03:52:56.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.824+0000 7efd5d7d9640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T03:52:56.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.824+0000 7efd5d7d9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:52:56.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.824+0000 7efd5d7d9640 1 -- 172.21.15.42:0/3359639834 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7efd58069040 con 0x7efd58134320 2024-09-16T03:52:56.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.828+0000 7efd4f7fe640 1 -- 172.21.15.42:0/3359639834 <== mon.2 v2:172.21.15.42:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7efd50007470 con 0x7efd58134320 2024-09-16T03:52:56.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.828+0000 7efd4f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T03:52:56.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.828+0000 7efd4f7fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:52:56.834 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.828+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:52:56.834 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.828+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=26,config=2,health=12,kv=10,logm=50} crc {auth=3414414147,config=692619927,health=8952337,kv=3446188498,logm=341796072}) 2024-09-16T03:52:56.835 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.828+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=27,config=2,health=12,kv=10,logm=49} crc {auth=1325781461,config=692619927,health=8952337,kv=3446188498,logm=777288939}) 2024-09-16T03:52:56.835 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.828+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:52:56.835 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.828+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=26,config=2,health=12,kv=10,logm=50} crc {auth=3414414147,config=692619927,health=8952337,kv=3446188498,logm=341796072}) 2024-09-16T03:52:56.835 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.828+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=25,config=2,health=12,kv=9,logm=52} crc {auth=3438050621,config=692619927,health=8952337,kv=4174212571,logm=223246066}) 2024-09-16T03:52:56.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.828+0000 7efd5d7d9640 10 librados: watch_flush enter 2024-09-16T03:52:56.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.828+0000 7efd5d7d9640 10 librados: watch_flush exit 2024-09-16T03:52:56.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.828+0000 7efd5d7d9640 20 client.834128.objecter shutdown clearing up homeless session... 2024-09-16T03:52:56.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.828+0000 7efd5d7d9640 10 client.834128.objecter successfully canceled tick 2024-09-16T03:52:56.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.828+0000 7efd5d7d9640 1 -- 172.21.15.42:0/3359639834 >> v2:172.21.15.154:6812/254845342 conn(0x7efd3c037810 msgr2=0x7efd3c039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:52:56.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.828+0000 7efd5d7d9640 1 --2- 172.21.15.42:0/3359639834 >> v2:172.21.15.154:6812/254845342 conn(0x7efd3c037810 0x7efd3c039cd0 secure :-1 s=READY pgs=3585 cs=0 l=1 rev1=1 crypto rx=0x7efd48009c20 tx=0x7efd48009190 comp rx=0 tx=0).stop 2024-09-16T03:52:56.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.828+0000 7efd5d7d9640 10 monclient: shutdown 2024-09-16T03:52:56.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.828+0000 7efd5d7d9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:52:56.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.828+0000 7efd5d7d9640 1 -- 172.21.15.42:0/3359639834 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7efd58134320 msgr2=0x7efd5811bf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:52:56.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.828+0000 7efd5d7d9640 1 --2- 172.21.15.42:0/3359639834 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7efd58134320 0x7efd5811bf70 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7efd5000c3f0 tx=0x7efd5000c8b0 comp rx=0 tx=0).stop 2024-09-16T03:52:56.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.828+0000 7efd5d7d9640 1 -- 172.21.15.42:0/3359639834 shutdown_connections 2024-09-16T03:52:56.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.828+0000 7efd5d7d9640 1 --2- 172.21.15.42:0/3359639834 >> v2:172.21.15.154:6812/254845342 conn(0x7efd3c037810 0x7efd3c039cd0 unknown :-1 s=CLOSED pgs=3585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:52:56.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.828+0000 7efd5d7d9640 1 --2- 172.21.15.42:0/3359639834 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7efd58134320 0x7efd5811bf70 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:52:56.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.828+0000 7efd5d7d9640 1 --2- 172.21.15.42:0/3359639834 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efd58135100 0x7efd5813b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:52:56.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.828+0000 7efd5d7d9640 1 --2- 172.21.15.42:0/3359639834 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efd580a1330 0x7efd581cd020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:52:56.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.828+0000 7efd5d7d9640 1 -- 172.21.15.42:0/3359639834 >> 172.21.15.42:0/3359639834 conn(0x7efd580a2420 msgr2=0x7efd5806e000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:52:56.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.828+0000 7efd5d7d9640 1 -- 172.21.15.42:0/3359639834 shutdown_connections 2024-09-16T03:52:56.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.828+0000 7efd5d7d9640 1 -- 172.21.15.42:0/3359639834 wait complete. 2024-09-16T03:52:56.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:56.828+0000 7efd5d7d9640 1 librados: shutdown 2024-09-16T03:52:56.839 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.832+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:52:56.840 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.832+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=390217368}) 2024-09-16T03:52:56.840 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.832+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2089349945}) 2024-09-16T03:52:56.840 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.832+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:52:56.840 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.832+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=390217368}) 2024-09-16T03:52:56.840 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.832+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3954428643}) 2024-09-16T03:52:56.841 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.836+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:52:56.842 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.836+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1404990394}) 2024-09-16T03:52:56.842 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.836+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3185698984}) 2024-09-16T03:52:56.842 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.836+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:52:56.842 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.836+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1404990394}) 2024-09-16T03:52:56.842 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.836+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=895073503}) 2024-09-16T03:52:56.845 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.836+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:52:56.845 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.836+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=425809192}) 2024-09-16T03:52:56.845 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.836+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1426773935}) 2024-09-16T03:52:56.845 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.836+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:52:56.845 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.836+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=425809192}) 2024-09-16T03:52:56.845 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.836+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=316342622}) 2024-09-16T03:52:56.847 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.840+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:52:56.847 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.840+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3279365301}) 2024-09-16T03:52:56.847 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.840+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2483697527}) 2024-09-16T03:52:56.847 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.840+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:52:56.847 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.840+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3279365301}) 2024-09-16T03:52:56.847 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.840+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2888533936}) 2024-09-16T03:52:56.849 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.840+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:52:56.849 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.840+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=374511549}) 2024-09-16T03:52:56.849 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.844+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1317693507}) 2024-09-16T03:52:56.849 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.844+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:52:56.849 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.844+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=374511549}) 2024-09-16T03:52:56.849 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.844+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4189676540}) 2024-09-16T03:52:56.852 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.844+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:52:56.852 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.844+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=516081820}) 2024-09-16T03:52:56.852 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.844+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1065235369}) 2024-09-16T03:52:56.852 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.844+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:52:56.852 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.844+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=516081820}) 2024-09-16T03:52:56.852 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.844+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1961550993}) 2024-09-16T03:52:56.854 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.848+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:52:56.854 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.848+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2805877698}) 2024-09-16T03:52:56.854 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.848+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=604112567}) 2024-09-16T03:52:56.854 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.848+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:52:56.854 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.848+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2805877698}) 2024-09-16T03:52:56.854 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.848+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1899589475}) 2024-09-16T03:52:56.855 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T03:52:56.856 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.848+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:52:56.856 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.848+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1657579581}) 2024-09-16T03:52:56.856 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.848+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3299061763}) 2024-09-16T03:52:56.856 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.848+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:52:56.856 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.848+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1657579581}) 2024-09-16T03:52:56.856 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.848+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1034853907}) 2024-09-16T03:52:56.858 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.852+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:52:56.858 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.852+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1399147561}) 2024-09-16T03:52:56.858 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.852+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2727178195}) 2024-09-16T03:52:56.858 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.852+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:52:56.858 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.852+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1399147561}) 2024-09-16T03:52:56.859 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.852+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2448415806}) 2024-09-16T03:52:56.860 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.852+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:52:56.860 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.852+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3648588058}) 2024-09-16T03:52:56.860 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.852+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=406677316}) 2024-09-16T03:52:56.860 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.852+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:52:56.860 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.852+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3648588058}) 2024-09-16T03:52:56.861 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.852+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3174337267}) 2024-09-16T03:52:56.862 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.856+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:52:56.862 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.856+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1422066075}) 2024-09-16T03:52:56.862 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.856+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4055379107}) 2024-09-16T03:52:56.862 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.856+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:52:56.862 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.856+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1422066075}) 2024-09-16T03:52:56.862 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.856+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=160718859}) 2024-09-16T03:52:56.864 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.856+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:52:56.864 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.856+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3014139}) 2024-09-16T03:52:56.864 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.856+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=263237000}) 2024-09-16T03:52:56.865 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.856+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:52:56.865 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.856+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3014139}) 2024-09-16T03:52:56.865 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.856+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2643115680}) 2024-09-16T03:52:56.866 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.860+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:52:56.866 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.860+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=953120788}) 2024-09-16T03:52:56.867 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.860+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4184700072}) 2024-09-16T03:52:56.867 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.860+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:52:56.867 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.860+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=953120788}) 2024-09-16T03:52:56.867 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.860+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=120567883}) 2024-09-16T03:52:56.869 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.860+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:52:56.869 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.860+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3117072180}) 2024-09-16T03:52:56.869 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.860+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2917014787}) 2024-09-16T03:52:56.869 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.860+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:52:56.869 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.860+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3117072180}) 2024-09-16T03:52:56.869 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.860+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3307523973}) 2024-09-16T03:52:56.871 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.864+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:52:56.871 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.864+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=685186811}) 2024-09-16T03:52:56.871 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.864+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1794874198}) 2024-09-16T03:52:56.871 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.864+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:52:56.871 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.864+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=685186811}) 2024-09-16T03:52:56.871 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.864+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2257090576}) 2024-09-16T03:52:56.873 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.864+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:52:56.873 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.864+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1017792791}) 2024-09-16T03:52:56.873 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.864+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3228225924}) 2024-09-16T03:52:56.873 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.864+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:52:56.873 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.868+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1017792791}) 2024-09-16T03:52:56.873 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.868+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4199459770}) 2024-09-16T03:52:56.875 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.868+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:52:56.875 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.868+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1792706091}) 2024-09-16T03:52:56.875 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.868+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1685544129}) 2024-09-16T03:52:56.875 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.868+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:52:56.875 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.868+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1792706091}) 2024-09-16T03:52:56.875 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.868+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=461821450}) 2024-09-16T03:52:56.877 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.872+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:52:56.877 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.872+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1922901119}) 2024-09-16T03:52:56.877 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.872+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1569778855}) 2024-09-16T03:52:56.877 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.872+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:52:56.877 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.872+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1922901119}) 2024-09-16T03:52:56.878 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.872+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1944658079}) 2024-09-16T03:52:56.879 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.872+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:52:56.879 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.872+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3553025462}) 2024-09-16T03:52:56.880 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.872+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=405566260}) 2024-09-16T03:52:56.880 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.872+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:52:56.880 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.872+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3553025462}) 2024-09-16T03:52:56.880 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.872+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1748384148}) 2024-09-16T03:52:56.882 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.876+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:52:56.882 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.876+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2373796062}) 2024-09-16T03:52:56.882 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.876+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1530360520}) 2024-09-16T03:52:56.882 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.876+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:52:56.882 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.876+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2373796062}) 2024-09-16T03:52:56.882 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.876+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1434757811}) 2024-09-16T03:52:56.884 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.876+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:52:56.884 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.876+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=543294092}) 2024-09-16T03:52:56.884 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.876+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1396953860}) 2024-09-16T03:52:56.884 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.876+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:52:56.884 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.876+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=543294092}) 2024-09-16T03:52:56.884 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.876+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3560286339}) 2024-09-16T03:52:56.886 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.880+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:52:56.886 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.880+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2005018458}) 2024-09-16T03:52:56.886 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.880+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2984189915}) 2024-09-16T03:52:56.886 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.880+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:52:56.886 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.880+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2005018458}) 2024-09-16T03:52:56.886 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.880+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3611764164}) 2024-09-16T03:52:56.888 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.880+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:52:56.888 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.880+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1887427862}) 2024-09-16T03:52:56.888 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.880+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3006919763}) 2024-09-16T03:52:56.888 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.880+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:52:56.888 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.880+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1887427862}) 2024-09-16T03:52:56.889 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.880+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=43337359}) 2024-09-16T03:52:56.890 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.884+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:52:56.890 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.884+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4060821286}) 2024-09-16T03:52:56.890 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.884+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2929208243}) 2024-09-16T03:52:56.891 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.884+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:52:56.891 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.884+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4060821286}) 2024-09-16T03:52:56.891 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.884+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3176223613}) 2024-09-16T03:52:56.892 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.884+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:52:56.892 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.884+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3829748199}) 2024-09-16T03:52:56.893 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.884+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3824469389}) 2024-09-16T03:52:56.893 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.884+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:52:56.893 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.884+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3829748199}) 2024-09-16T03:52:56.894 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.884+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1868430175}) 2024-09-16T03:52:56.894 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.888+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:52:56.894 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.888+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2056252230}) 2024-09-16T03:52:56.895 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.888+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3812056041}) 2024-09-16T03:52:56.895 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.888+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:52:56.895 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.888+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2056252230}) 2024-09-16T03:52:56.895 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.888+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2278895017}) 2024-09-16T03:52:56.896 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.888+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:52:56.897 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.888+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=988180532}) 2024-09-16T03:52:56.897 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.888+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1058673376}) 2024-09-16T03:52:56.897 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.888+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:52:56.897 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.888+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=988180532}) 2024-09-16T03:52:56.897 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.888+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3511626464}) 2024-09-16T03:52:56.899 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.892+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:52:56.899 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.892+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=965296917}) 2024-09-16T03:52:56.899 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.892+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=667475592}) 2024-09-16T03:52:56.899 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.892+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:52:56.899 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.892+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=965296917}) 2024-09-16T03:52:56.899 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.892+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2815920979}) 2024-09-16T03:52:56.901 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.892+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:52:56.901 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.892+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=237768299}) 2024-09-16T03:52:56.901 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.892+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4218766936}) 2024-09-16T03:52:56.901 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.892+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:52:56.901 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.892+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=237768299}) 2024-09-16T03:52:56.901 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.892+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1377439329}) 2024-09-16T03:52:56.903 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.896+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:52:56.903 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.896+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3229002512}) 2024-09-16T03:52:56.903 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.896+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3419255569}) 2024-09-16T03:52:56.903 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.896+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:52:56.903 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.896+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3229002512}) 2024-09-16T03:52:56.903 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.896+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3334077736}) 2024-09-16T03:52:56.905 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.896+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:52:56.905 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.900+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1831009275}) 2024-09-16T03:52:56.905 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.900+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3755594811}) 2024-09-16T03:52:56.905 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.900+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:52:56.905 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.900+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1831009275}) 2024-09-16T03:52:56.905 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.900+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4131885578}) 2024-09-16T03:52:56.907 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.900+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:52:56.907 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.900+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1666623999}) 2024-09-16T03:52:56.907 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.900+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3670040354}) 2024-09-16T03:52:56.907 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.900+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:52:56.907 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.900+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1666623999}) 2024-09-16T03:52:56.908 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.900+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3184378171}) 2024-09-16T03:52:56.909 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.904+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:52:56.909 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.904+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1913375116}) 2024-09-16T03:52:56.909 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.904+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=349841212}) 2024-09-16T03:52:56.909 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.904+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:52:56.910 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.904+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1913375116}) 2024-09-16T03:52:56.910 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.904+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4105651162}) 2024-09-16T03:52:56.912 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.904+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:52:56.912 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.904+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2349790473}) 2024-09-16T03:52:56.912 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.904+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2069704903}) 2024-09-16T03:52:56.912 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.904+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:52:56.912 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.904+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2349790473}) 2024-09-16T03:52:56.912 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.904+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1474270971}) 2024-09-16T03:52:56.914 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.908+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:52:56.914 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.908+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2461755235}) 2024-09-16T03:52:56.914 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.908+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1813261685}) 2024-09-16T03:52:56.914 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.908+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:52:56.914 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.908+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2461755235}) 2024-09-16T03:52:56.914 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.908+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=511238895}) 2024-09-16T03:52:56.916 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.908+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:52:56.916 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.908+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3905773662}) 2024-09-16T03:52:56.916 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.908+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2353207189}) 2024-09-16T03:52:56.916 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.908+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:52:56.916 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.908+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3905773662}) 2024-09-16T03:52:56.916 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.908+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=757025353}) 2024-09-16T03:52:56.918 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.912+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:52:56.918 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.912+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=683809595}) 2024-09-16T03:52:56.918 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.912+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2628597046}) 2024-09-16T03:52:56.918 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.912+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:52:56.918 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.912+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=683809595}) 2024-09-16T03:52:56.918 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.912+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=247102395}) 2024-09-16T03:52:56.920 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.912+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:52:56.920 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.912+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3267832202}) 2024-09-16T03:52:56.920 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.912+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=675864164}) 2024-09-16T03:52:56.920 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.912+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:52:56.920 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.912+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3267832202}) 2024-09-16T03:52:56.920 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.912+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3454156321}) 2024-09-16T03:52:56.922 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.916+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:52:56.922 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.916+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3457574959}) 2024-09-16T03:52:56.922 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.916+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2064036053}) 2024-09-16T03:52:56.922 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.916+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:52:56.922 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.916+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3457574959}) 2024-09-16T03:52:56.922 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.916+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3594178518}) 2024-09-16T03:52:56.924 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.916+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:52:56.924 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.916+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2149788018}) 2024-09-16T03:52:56.925 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.916+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=203259703}) 2024-09-16T03:52:56.925 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.916+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:52:56.925 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.916+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2149788018}) 2024-09-16T03:52:56.925 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.916+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=294144563}) 2024-09-16T03:52:56.927 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.920+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:52:56.927 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.920+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=288358333}) 2024-09-16T03:52:56.927 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.920+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=109310475}) 2024-09-16T03:52:56.927 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.920+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:52:56.927 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.920+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=288358333}) 2024-09-16T03:52:56.927 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.920+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3453200143}) 2024-09-16T03:52:56.929 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.920+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:52:56.929 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.924+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=514975887}) 2024-09-16T03:52:56.929 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.924+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1613698165}) 2024-09-16T03:52:56.929 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.924+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:52:56.929 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.924+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=514975887}) 2024-09-16T03:52:56.929 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.924+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2693289653}) 2024-09-16T03:52:56.932 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.924+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:52:56.933 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.924+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=584404284}) 2024-09-16T03:52:56.933 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.924+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1635228478}) 2024-09-16T03:52:56.933 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.924+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:52:56.933 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.924+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=584404284}) 2024-09-16T03:52:56.933 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.924+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3401442730}) 2024-09-16T03:52:56.934 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.928+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:52:56.934 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.928+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2494522273}) 2024-09-16T03:52:56.934 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.928+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=749102216}) 2024-09-16T03:52:56.934 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.928+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:52:56.934 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.928+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2494522273}) 2024-09-16T03:52:56.934 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.928+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2782455901}) 2024-09-16T03:52:56.936 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.928+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:52:56.936 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.928+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3203304765}) 2024-09-16T03:52:56.936 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.928+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=237043174}) 2024-09-16T03:52:56.936 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.928+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:52:56.936 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.928+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3203304765}) 2024-09-16T03:52:56.936 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.928+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=113203216}) 2024-09-16T03:52:56.938 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.932+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:52:56.938 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.932+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3049503112}) 2024-09-16T03:52:56.938 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.932+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3227523224}) 2024-09-16T03:52:56.938 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.932+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:52:56.938 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.932+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3049503112}) 2024-09-16T03:52:56.938 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.932+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1631041706}) 2024-09-16T03:52:56.940 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.932+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:52:56.941 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.932+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2142126203}) 2024-09-16T03:52:56.941 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.932+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2583367591}) 2024-09-16T03:52:56.941 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.932+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:52:56.941 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.932+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2142126203}) 2024-09-16T03:52:56.941 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.932+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=22569268}) 2024-09-16T03:52:56.943 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.936+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:52:56.943 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.936+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3073083882}) 2024-09-16T03:52:56.943 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.936+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3047831280}) 2024-09-16T03:52:56.943 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.936+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:52:56.943 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.936+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3073083882}) 2024-09-16T03:52:56.943 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.936+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=610633432}) 2024-09-16T03:52:56.945 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.936+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:52:56.945 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.936+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=347982334}) 2024-09-16T03:52:56.945 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.936+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3825027547}) 2024-09-16T03:52:56.945 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.936+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:52:56.945 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.936+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=347982334}) 2024-09-16T03:52:56.945 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.936+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2674269653}) 2024-09-16T03:52:56.947 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.940+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:52:56.947 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.940+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=998303899}) 2024-09-16T03:52:56.947 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.940+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=678536887}) 2024-09-16T03:52:56.947 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.940+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:52:56.947 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.940+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=998303899}) 2024-09-16T03:52:56.948 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.940+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3521090759}) 2024-09-16T03:52:56.949 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.944+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:52:56.949 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.944+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=789966123}) 2024-09-16T03:52:56.949 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.944+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=909221724}) 2024-09-16T03:52:56.949 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.944+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:52:56.949 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.944+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=789966123}) 2024-09-16T03:52:56.950 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.944+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=457656598}) 2024-09-16T03:52:56.951 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.944+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:52:56.951 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.944+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3003040012}) 2024-09-16T03:52:56.951 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.944+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=889456928}) 2024-09-16T03:52:56.951 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.944+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:52:56.952 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.944+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3003040012}) 2024-09-16T03:52:56.952 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.944+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3316462791}) 2024-09-16T03:52:56.953 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.948+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:52:56.953 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.948+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3863552783}) 2024-09-16T03:52:56.953 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.948+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2670574845}) 2024-09-16T03:52:56.954 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.948+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:52:56.954 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.948+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3863552783}) 2024-09-16T03:52:56.954 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.948+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1764908079}) 2024-09-16T03:52:56.955 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.948+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:52:56.955 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.948+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=346745826}) 2024-09-16T03:52:56.956 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.948+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1294246202}) 2024-09-16T03:52:56.956 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.948+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:52:56.956 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.948+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=346745826}) 2024-09-16T03:52:56.956 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.948+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2606274470}) 2024-09-16T03:52:56.957 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.952+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:52:56.957 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.952+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1344637119}) 2024-09-16T03:52:56.958 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.952+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4265003079}) 2024-09-16T03:52:56.958 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.952+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:52:56.958 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.952+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1344637119}) 2024-09-16T03:52:56.958 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.952+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3201613513}) 2024-09-16T03:52:56.960 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.952+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:52:56.960 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.952+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1217277468}) 2024-09-16T03:52:56.960 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.952+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=185599644}) 2024-09-16T03:52:56.960 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.952+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:52:56.960 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.952+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1217277468}) 2024-09-16T03:52:56.960 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.952+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3016261838}) 2024-09-16T03:52:56.962 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.956+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:52:56.962 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.956+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=906843100}) 2024-09-16T03:52:56.962 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.956+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=777424973}) 2024-09-16T03:52:56.962 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.956+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:52:56.962 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.956+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=906843100}) 2024-09-16T03:52:56.962 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.956+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3689417210}) 2024-09-16T03:52:56.964 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.956+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:52:56.964 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.956+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1783042179}) 2024-09-16T03:52:56.964 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.956+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1811941712}) 2024-09-16T03:52:56.964 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.956+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:52:56.964 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.956+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1783042179}) 2024-09-16T03:52:56.964 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.956+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2106530753}) 2024-09-16T03:52:56.965 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.960+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:52:56.966 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.960+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=783350371}) 2024-09-16T03:52:56.966 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.960+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=257835930}) 2024-09-16T03:52:56.966 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.960+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:52:56.966 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.960+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=783350371}) 2024-09-16T03:52:56.966 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.960+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4201380116}) 2024-09-16T03:52:56.969 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.960+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:52:56.969 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.960+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=10,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=11,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=33,osdmap=11} crc {logm=1687618428,mdsmap=4088524776,mgr=460109776,mgr_command_descs=2107673600,mgr_metadata=1367272202,mgrstat=285150040,mon_config_key=85057112,monmap=3937513981,osd_metadata=191738038,osd_pg_creating=953588495,osd_snap=4192190372,osdmap=668656351}) 2024-09-16T03:52:56.969 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.960+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=10,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=11,monmap=3,osd_metadata=4,osd_pg_creating=1,osd_snap=31,osdmap=15} crc {logm=1687618428,mdsmap=4088524775,mgr=1669371261,mgr_command_descs=2107673600,mgr_metadata=1367272202,mgrstat=285150040,mon_config_key=85057112,monmap=3937513981,osd_metadata=2399266538,osd_pg_creating=953588495,osd_snap=4003192486,osdmap=941683773}) 2024-09-16T03:52:56.969 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.964+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:52:56.969 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.964+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=10,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=11,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=33,osdmap=11} crc {logm=1687618428,mdsmap=4088524776,mgr=460109776,mgr_command_descs=2107673600,mgr_metadata=1367272202,mgrstat=285150040,mon_config_key=85057112,monmap=3937513981,osd_metadata=191738038,osd_pg_creating=953588495,osd_snap=4192190372,osdmap=668656351}) 2024-09-16T03:52:56.969 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.964+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=10,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=9,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=30,osdmap=16} crc {logm=1687618428,mdsmap=4088524775,mgr=460109776,mgr_command_descs=2107673600,mgr_metadata=1367272202,mgrstat=285150040,mon_config_key=2291258920,monmap=3937513981,osd_metadata=3450683029,osd_pg_creating=953588495,osd_snap=2102637450,osdmap=2032366287}) 2024-09-16T03:52:56.989 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.980+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:52:56.989 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.980+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=93} crc {osdmap=2379458945}) 2024-09-16T03:52:56.990 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.984+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=93} crc {osdmap=1693757738}) 2024-09-16T03:52:56.990 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.984+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:52:56.990 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.984+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=93} crc {osdmap=2379458945}) 2024-09-16T03:52:56.990 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:52:56.984+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=92} crc {osdmap=3456089996}) 2024-09-16T03:52:57.856 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T03:52:57.857 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:52:57.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.984+0000 7f4b6b094640 10 monclient: get_monmap_and_config 2024-09-16T03:52:57.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.984+0000 7f4b6b094640 10 monclient: build_initial_monmap 2024-09-16T03:52:57.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.984+0000 7f4b6b094640 10 monclient: monmap: 2024-09-16T03:52:57.991 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:52:57.991 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:52:57.991 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:52:57.989415+0000 2024-09-16T03:52:57.991 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:52:57.989415+0000 2024-09-16T03:52:57.991 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:52:57.991 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:52:57.991 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:52:57.991 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:52:57.991 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:52:57.991 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:52:57.992 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:52:57.992 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:52:57.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.984+0000 7f4b6b094640 1 Processor -- start 2024-09-16T03:52:57.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.984+0000 7f4b6b094640 1 -- start start 2024-09-16T03:52:57.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.984+0000 7f4b6b094640 10 monclient: init 2024-09-16T03:52:57.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.984+0000 7f4b6b094640 10 monclient: _reopen_session rank -1 2024-09-16T03:52:57.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.984+0000 7f4b6b094640 10 monclient: _add_conns ranks=[4,2,0,1,3] 2024-09-16T03:52:57.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.984+0000 7f4b6b094640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f4b6406db90 0x7f4b641340c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:52:57.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.984+0000 7f4b6b094640 10 monclient(hunting): picked mon.noname-c con 0x7f4b6406db90 addr v2:172.21.15.154:3300/0 2024-09-16T03:52:57.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.984+0000 7f4b6b094640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4b640a37f0 0x7f4b640a3c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:52:57.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.984+0000 7f4b6b094640 10 monclient(hunting): picked mon.noname-b con 0x7f4b640a37f0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:52:57.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.984+0000 7f4b6b094640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4b64136e10 addr v1:172.21.15.42:6789/0 2024-09-16T03:52:57.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.984+0000 7f4b6b094640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4b6406ece0 con 0x7f4b64136e10 2024-09-16T03:52:57.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.984+0000 7f4b6b094640 10 monclient(hunting): start opening mon connection 2024-09-16T03:52:57.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.984+0000 7f4b6b094640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f4b6406df70 con 0x7f4b6406db90 2024-09-16T03:52:57.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.984+0000 7f4b6b094640 10 monclient(hunting): start opening mon connection 2024-09-16T03:52:57.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.984+0000 7f4b6b094640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f4b6409bf80 con 0x7f4b640a37f0 2024-09-16T03:52:57.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.984+0000 7f4b6b094640 10 monclient(hunting): _renew_subs 2024-09-16T03:52:57.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.984+0000 7f4b6b094640 10 monclient(hunting): authenticate will time out at 1624.958859s 2024-09-16T03:52:57.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.984+0000 7f4b63fff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4b640a37f0 0x7f4b640a3c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:52:57.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.984+0000 7f4b69631640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f4b64136e10 0x7f4b64137240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:41192/0 (socket says 172.21.15.42:41192) 2024-09-16T03:52:57.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.984+0000 7f4b69631640 1 -- 172.21.15.42:0/56324905 learned_addr learned my addr 172.21.15.42:0/56324905 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:52:57.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.984+0000 7f4b63fff640 10 monclient(hunting): get_auth_request con 0x7f4b640a37f0 auth_method 0 2024-09-16T03:52:57.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.984+0000 7f4b63fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:52:57.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.984+0000 7f4b63fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:52:57.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.984+0000 7f4b63fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:52:57.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.984+0000 7f4b63fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:52:57.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.984+0000 7f4b63fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:52:57.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.984+0000 7f4b63fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:52:57.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.984+0000 7f4b68e30640 1 --2- 172.21.15.42:0/56324905 >> v2:172.21.15.154:3300/0 conn(0x7f4b6406db90 0x7f4b641340c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:52:57.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.984+0000 7f4b63fff640 10 monclient(hunting): handle_auth_done global_id 834131 payload 274 2024-09-16T03:52:57.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.984+0000 7f4b63fff640 1 -- 172.21.15.42:0/56324905 >> v2:172.21.15.154:3300/0 conn(0x7f4b6406db90 msgr2=0x7f4b641340c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:52:57.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.984+0000 7f4b63fff640 1 --2- 172.21.15.42:0/56324905 >> v2:172.21.15.154:3300/0 conn(0x7f4b6406db90 0x7f4b641340c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:52:57.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.984+0000 7f4b63fff640 1 -- 172.21.15.42:0/56324905 >> v1:172.21.15.42:6789/0 conn(0x7f4b64136e10 legacy=0x7f4b64137240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:52:57.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.984+0000 7f4b637fe640 1 -- 172.21.15.42:0/56324905 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2500210476 0 0) 0x7f4b6406ece0 con 0x7f4b64136e10 2024-09-16T03:52:57.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.984+0000 7f4b63fff640 10 monclient: _finish_hunting 0 2024-09-16T03:52:57.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.984+0000 7f4b63fff640 1 monclient: found mon.noname-b 2024-09-16T03:52:57.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.984+0000 7f4b63fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:52:57.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.984+0000 7f4b63fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T03:52:57.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.984+0000 7f4b63fff640 1 -- 172.21.15.42:0/56324905 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b64134780 con 0x7f4b640a37f0 2024-09-16T03:52:57.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.984+0000 7f4b68e30640 10 monclient: get_auth_request con 0x7f4b6406db90 auth_method 0 2024-09-16T03:52:57.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.984+0000 7f4b68e30640 1 --2- 172.21.15.42:0/56324905 >> v2:172.21.15.154:3300/0 conn(0x7f4b6406db90 0x7f4b641340c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T03:52:57.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.984+0000 7f4b637fe640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T03:52:57.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.988+0000 7f4b63fff640 1 --2- 172.21.15.42:0/56324905 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4b640a37f0 0x7f4b640a3c00 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f4b4c0097c0 tx=0x7f4b4c01c360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5ae06f7f1defdd2c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:52:57.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.988+0000 7f4b637fe640 1 -- 172.21.15.42:0/56324905 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b4c0b0070 con 0x7f4b640a37f0 2024-09-16T03:52:57.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.988+0000 7f4b637fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:52:57.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.988+0000 7f4b637fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:52:57.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.988+0000 7f4b637fe640 10 monclient: dump: 2024-09-16T03:52:57.999 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:52:57.999 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:52:57.999 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:52:57.999 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:52:57.999 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:52:57.999 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:52:57.999 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:52:58.000 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:52:58.000 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:52:58.000 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:52:58.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.988+0000 7f4b637fe640 10 monclient: _finish_auth 0 2024-09-16T03:52:58.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.988+0000 7f4b637fe640 10 monclient: _check_auth_tickets 2024-09-16T03:52:58.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.988+0000 7f4b637fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:52:58.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.988+0000 7f4b6b094640 5 monclient: authenticate success, global_id 834131 2024-09-16T03:52:58.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.988+0000 7f4b6b094640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:52:58.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.988+0000 7f4b637fe640 1 -- 172.21.15.42:0/56324905 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4b4c0ab600 con 0x7f4b640a37f0 2024-09-16T03:52:58.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.988+0000 7f4b637fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:52:58.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.988+0000 7f4b6b094640 10 monclient: get_monmap_and_config success 2024-09-16T03:52:58.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.988+0000 7f4b6b094640 4 set_mon_vals no callback set 2024-09-16T03:52:58.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.988+0000 7f4b637fe640 1 -- 172.21.15.42:0/56324905 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b4c0ab9d0 con 0x7f4b640a37f0 2024-09-16T03:52:58.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.988+0000 7f4b637fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:52:58.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.988+0000 7f4b637fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:52:58.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.988+0000 7f4b637fe640 10 monclient: dump: 2024-09-16T03:52:58.001 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:52:58.001 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:52:58.001 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:52:58.001 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:52:58.001 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:52:58.001 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:52:58.001 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:52:58.001 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:52:58.001 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:52:58.001 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:52:58.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.988+0000 7f4b6b094640 10 monclient: shutdown 2024-09-16T03:52:58.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.988+0000 7f4b6b094640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:52:58.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.988+0000 7f4b6b094640 1 -- 172.21.15.42:0/56324905 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4b640a37f0 msgr2=0x7f4b640a3c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:52:58.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.988+0000 7f4b6b094640 1 --2- 172.21.15.42:0/56324905 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4b640a37f0 0x7f4b640a3c00 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f4b4c0097c0 tx=0x7f4b4c01c360 comp rx=0 tx=0).stop 2024-09-16T03:52:58.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.988+0000 7f4b6b094640 1 -- 172.21.15.42:0/56324905 shutdown_connections 2024-09-16T03:52:58.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.988+0000 7f4b6b094640 1 --2- 172.21.15.42:0/56324905 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4b640a37f0 0x7f4b640a3c00 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:52:58.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.988+0000 7f4b6b094640 1 --2- 172.21.15.42:0/56324905 >> v2:172.21.15.154:3300/0 conn(0x7f4b6406db90 0x7f4b641340c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:52:58.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.988+0000 7f4b6b094640 1 -- 172.21.15.42:0/56324905 >> 172.21.15.42:0/56324905 conn(0x7f4b640a9fb0 msgr2=0x7f4b640aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:52:58.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.988+0000 7f4b6b094640 1 -- 172.21.15.42:0/56324905 shutdown_connections 2024-09-16T03:52:58.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.988+0000 7f4b6b094640 1 -- 172.21.15.42:0/56324905 wait complete. 2024-09-16T03:52:58.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.988+0000 7f4b6b094640 10 monclient: build_initial_monmap 2024-09-16T03:52:58.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.988+0000 7f4b6b094640 10 monclient: monmap: 2024-09-16T03:52:58.007 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:52:58.008 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:52:58.008 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:52:58.008 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:52:58.008 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:52:58.008 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:52:58.008 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:52:58.008 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:52:58.008 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:52:58.008 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:52:58.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.988+0000 7f4b6b094640 1 librados: starting msgr at 2024-09-16T03:52:58.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.988+0000 7f4b6b094640 1 librados: starting objecter 2024-09-16T03:52:58.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.988+0000 7f4b6b094640 1 Processor -- start 2024-09-16T03:52:58.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.988+0000 7f4b6b094640 1 -- start start 2024-09-16T03:52:58.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.988+0000 7f4b6b094640 1 librados: setting wanted keys 2024-09-16T03:52:58.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.988+0000 7f4b6b094640 1 librados: calling monclient init 2024-09-16T03:52:58.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.988+0000 7f4b6b094640 10 monclient: init 2024-09-16T03:52:58.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.988+0000 7f4b6b094640 10 monclient: _reopen_session rank -1 2024-09-16T03:52:58.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.988+0000 7f4b6b094640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:52:58.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.988+0000 7f4b6b094640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4b6406db90 0x7f4b641cf1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:52:58.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.988+0000 7f4b6b094640 10 monclient(hunting): picked mon.noname-c con 0x7f4b6406db90 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:52:58.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.988+0000 7f4b6b094640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4b640a37f0 0x7f4b6413e1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:52:58.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.988+0000 7f4b6b094640 10 monclient(hunting): picked mon.noname-a con 0x7f4b640a37f0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:52:58.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.988+0000 7f4b6b094640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4b64136e10 0x7f4b6413e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:52:58.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.988+0000 7f4b6b094640 10 monclient(hunting): picked mon.noname-b con 0x7f4b64136e10 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:52:58.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.988+0000 7f4b6b094640 10 monclient(hunting): start opening mon connection 2024-09-16T03:52:58.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.988+0000 7f4b6b094640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f4b64141e40 con 0x7f4b640a37f0 2024-09-16T03:52:58.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.988+0000 7f4b6b094640 10 monclient(hunting): start opening mon connection 2024-09-16T03:52:58.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.988+0000 7f4b6b094640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f4b64141cc0 con 0x7f4b64136e10 2024-09-16T03:52:58.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.988+0000 7f4b6b094640 10 monclient(hunting): start opening mon connection 2024-09-16T03:52:58.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.988+0000 7f4b6b094640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f4b64141fc0 con 0x7f4b6406db90 2024-09-16T03:52:58.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.988+0000 7f4b6b094640 10 monclient(hunting): _renew_subs 2024-09-16T03:52:58.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.988+0000 7f4b6b094640 10 monclient(hunting): authenticate will time out at 1624.963713s 2024-09-16T03:52:58.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.988+0000 7f4b68e30640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4b6406db90 0x7f4b641cf1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:52:58.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.988+0000 7f4b63fff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4b640a37f0 0x7f4b6413e1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:52:58.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.988+0000 7f4b68e30640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4b6406db90 0x7f4b641cf1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:42994/0 (socket says 172.21.15.42:42994) 2024-09-16T03:52:58.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.988+0000 7f4b68e30640 1 -- 172.21.15.42:0/980321359 learned_addr learned my addr 172.21.15.42:0/980321359 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:52:58.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.988+0000 7f4b68e30640 10 monclient(hunting): get_auth_request con 0x7f4b6406db90 auth_method 0 2024-09-16T03:52:58.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.988+0000 7f4b68e30640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:52:58.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.988+0000 7f4b68e30640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:52:58.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.988+0000 7f4b68e30640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:52:58.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.988+0000 7f4b63fff640 10 monclient(hunting): get_auth_request con 0x7f4b640a37f0 auth_method 0 2024-09-16T03:52:58.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.988+0000 7f4b63fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:52:58.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.988+0000 7f4b63fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:52:58.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.988+0000 7f4b63fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:52:58.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.988+0000 7f4b68e30640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:52:58.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.988+0000 7f4b68e30640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:52:58.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.988+0000 7f4b68e30640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:52:58.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.988+0000 7f4b63fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:52:58.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.988+0000 7f4b63fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:52:58.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.988+0000 7f4b63fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:52:58.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.992+0000 7f4b69631640 1 --2- 172.21.15.42:0/980321359 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4b64136e10 0x7f4b6413e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:52:58.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.992+0000 7f4b68e30640 10 monclient(hunting): handle_auth_done global_id 834134 payload 824 2024-09-16T03:52:58.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.992+0000 7f4b68e30640 1 -- 172.21.15.42:0/980321359 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4b64136e10 msgr2=0x7f4b6413e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:52:58.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.992+0000 7f4b68e30640 1 --2- 172.21.15.42:0/980321359 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4b64136e10 0x7f4b6413e710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:52:58.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.992+0000 7f4b68e30640 1 -- 172.21.15.42:0/980321359 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4b640a37f0 msgr2=0x7f4b6413e1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:52:58.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.992+0000 7f4b68e30640 1 --2- 172.21.15.42:0/980321359 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4b640a37f0 0x7f4b6413e1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:52:58.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.992+0000 7f4b68e30640 10 monclient: _finish_hunting 0 2024-09-16T03:52:58.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.992+0000 7f4b68e30640 1 monclient: found mon.noname-c 2024-09-16T03:52:58.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.992+0000 7f4b68e30640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:52:58.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.992+0000 7f4b68e30640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:52:58.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.992+0000 7f4b68e30640 1 -- 172.21.15.42:0/980321359 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b640a8d90 con 0x7f4b6406db90 2024-09-16T03:52:58.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.992+0000 7f4b69631640 10 monclient: get_auth_request con 0x7f4b64136e10 auth_method 0 2024-09-16T03:52:58.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.992+0000 7f4b69631640 1 --2- 172.21.15.42:0/980321359 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4b64136e10 0x7f4b6413e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T03:52:58.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.992+0000 7f4b68e30640 1 --2- 172.21.15.42:0/980321359 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4b6406db90 0x7f4b641cf1d0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f4b5c00ce30 tx=0x7f4b5c00a650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:52:58.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.992+0000 7f4b617fa640 10 client.?.objecter ms_handle_connect 0x7f4b6406db90 2024-09-16T03:52:58.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.992+0000 7f4b617fa640 10 client.?.objecter resend_mon_ops 2024-09-16T03:52:58.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.992+0000 7f4b617fa640 1 -- 172.21.15.42:0/980321359 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b5c00f890 con 0x7f4b6406db90 2024-09-16T03:52:58.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.992+0000 7f4b617fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:52:58.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.992+0000 7f4b617fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:52:58.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.992+0000 7f4b617fa640 10 monclient: dump: 2024-09-16T03:52:58.013 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:52:58.013 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:52:58.013 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:52:58.013 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:52:58.013 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:52:58.013 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:52:58.013 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:52:58.013 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:52:58.013 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:52:58.014 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:52:58.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.992+0000 7f4b617fa640 10 monclient: _finish_auth 0 2024-09-16T03:52:58.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.992+0000 7f4b617fa640 10 monclient: _check_auth_tickets 2024-09-16T03:52:58.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.992+0000 7f4b617fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:52:58.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.992+0000 7f4b617fa640 1 -- 172.21.15.42:0/980321359 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4b5c00fa30 con 0x7f4b6406db90 2024-09-16T03:52:58.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.992+0000 7f4b617fa640 10 monclient: handle_config config(0 keys) 2024-09-16T03:52:58.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.992+0000 7f4b617fa640 1 -- 172.21.15.42:0/980321359 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b5c00fe00 con 0x7f4b6406db90 2024-09-16T03:52:58.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.992+0000 7f4b617fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:52:58.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.992+0000 7f4b617fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:52:58.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.992+0000 7f4b617fa640 10 monclient: dump: 2024-09-16T03:52:58.014 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:52:58.014 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:52:58.014 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:52:58.014 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:52:58.015 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:52:58.015 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:52:58.015 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:52:58.015 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:52:58.015 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:52:58.015 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:52:58.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.992+0000 7f4b6b094640 5 monclient: authenticate success, global_id 834134 2024-09-16T03:52:58.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.992+0000 7f4b6b094640 10 monclient: _renew_subs 2024-09-16T03:52:58.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.992+0000 7f4b62ffd640 4 set_mon_vals no callback set 2024-09-16T03:52:58.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.992+0000 7f4b6b094640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:52:58.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.992+0000 7f4b6b094640 1 -- 172.21.15.42:0/980321359 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b640a5290 con 0x7f4b6406db90 2024-09-16T03:52:58.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.992+0000 7f4b6b094640 10 client.834134.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:52:58.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.992+0000 7f4b6b094640 10 monclient: _renew_subs 2024-09-16T03:52:58.016 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.992+0000 7f4b6b094640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:52:58.016 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.992+0000 7f4b6b094640 1 -- 172.21.15.42:0/980321359 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b64139990 con 0x7f4b6406db90 2024-09-16T03:52:58.016 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.992+0000 7f4b6b094640 1 librados: init done 2024-09-16T03:52:58.016 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.992+0000 7f4b6b094640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:52:58.016 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.992+0000 7f4b6b094640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:52:58.016 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.992+0000 7f4b6b094640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:52:58.016 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.992+0000 7f4b617fa640 1 -- 172.21.15.42:0/980321359 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f4b5c018400 con 0x7f4b6406db90 2024-09-16T03:52:58.016 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.992+0000 7f4b6b094640 1 -- 172.21.15.42:0/980321359 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4b641340c0 con 0x7f4b6406db90 2024-09-16T03:52:58.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.992+0000 7f4b617fa640 1 --2- 172.21.15.42:0/980321359 >> v2:172.21.15.154:6812/254845342 conn(0x7f4b44037810 0x7f4b44039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:52:58.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.992+0000 7f4b617fa640 1 -- 172.21.15.42:0/980321359 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(345..345 src has 286..345) ==== 5769+0+0 (secure 0 0 0) 0x7f4b5c049c00 con 0x7f4b6406db90 2024-09-16T03:52:58.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.992+0000 7f4b617fa640 10 client.834134.objecter ms_dispatch 0x7f4b64000ba0 osd_map(345..345 src has 286..345) 2024-09-16T03:52:58.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.992+0000 7f4b617fa640 3 client.834134.objecter handle_osd_map got epochs [345,345] > 0 2024-09-16T03:52:58.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.992+0000 7f4b617fa640 3 client.834134.objecter handle_osd_map decoding full epoch 345 2024-09-16T03:52:58.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.992+0000 7f4b617fa640 20 client.834134.objecter dump_active .. 0 homeless 2024-09-16T03:52:58.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.992+0000 7f4b63fff640 1 --2- 172.21.15.42:0/980321359 >> v2:172.21.15.154:6812/254845342 conn(0x7f4b44037810 0x7f4b44039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:52:58.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.992+0000 7f4b63fff640 10 monclient: get_auth_request con 0x7f4b44037810 auth_method 0 2024-09-16T03:52:58.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.992+0000 7f4b617fa640 10 client.834134.objecter ms_handle_connect 0x7f4b44037810 2024-09-16T03:52:58.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.992+0000 7f4b63fff640 1 --2- 172.21.15.42:0/980321359 >> v2:172.21.15.154:6812/254845342 conn(0x7f4b44037810 0x7f4b44039cd0 secure :-1 s=READY pgs=3586 cs=0 l=1 rev1=1 crypto rx=0x7f4b4c0ad000 tx=0x7f4b4c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:52:58.018 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.996+0000 7f4b617fa640 1 -- 172.21.15.42:0/980321359 <== mon.2 v2:172.21.15.42:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f4b5c014030 con 0x7f4b6406db90 2024-09-16T03:52:58.018 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.996+0000 7f4b617fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:52:58.018 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:57.996+0000 7f4b617fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:52:58.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.228+0000 7f4b6b094640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:52:58.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.228+0000 7f4b6b094640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:52:58.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.228+0000 7f4b6b094640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:52:58.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.228+0000 7f4b6b094640 1 -- 172.21.15.42:0/980321359 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f4b64068e40 con 0x7f4b6406db90 2024-09-16T03:52:58.235 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.228+0000 7f4b617fa640 1 -- 172.21.15.42:0/980321359 <== mon.2 v2:172.21.15.42:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7f4b5c0213a0 con 0x7f4b6406db90 2024-09-16T03:52:58.235 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.228+0000 7f4b617fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:52:58.235 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.228+0000 7f4b617fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:52:58.235 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":94,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:52:58.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.232+0000 7f4b6b094640 10 librados: watch_flush enter 2024-09-16T03:52:58.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.232+0000 7f4b6b094640 10 librados: watch_flush exit 2024-09-16T03:52:58.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.232+0000 7f4b6b094640 20 client.834134.objecter shutdown clearing up homeless session... 2024-09-16T03:52:58.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.232+0000 7f4b6b094640 10 client.834134.objecter successfully canceled tick 2024-09-16T03:52:58.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.232+0000 7f4b6b094640 1 -- 172.21.15.42:0/980321359 >> v2:172.21.15.154:6812/254845342 conn(0x7f4b44037810 msgr2=0x7f4b44039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:52:58.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.232+0000 7f4b6b094640 1 --2- 172.21.15.42:0/980321359 >> v2:172.21.15.154:6812/254845342 conn(0x7f4b44037810 0x7f4b44039cd0 secure :-1 s=READY pgs=3586 cs=0 l=1 rev1=1 crypto rx=0x7f4b4c0ad000 tx=0x7f4b4c008040 comp rx=0 tx=0).stop 2024-09-16T03:52:58.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.232+0000 7f4b6b094640 10 monclient: shutdown 2024-09-16T03:52:58.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.232+0000 7f4b6b094640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:52:58.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.232+0000 7f4b6b094640 1 -- 172.21.15.42:0/980321359 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4b6406db90 msgr2=0x7f4b641cf1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:52:58.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.232+0000 7f4b6b094640 1 --2- 172.21.15.42:0/980321359 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4b6406db90 0x7f4b641cf1d0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f4b5c00ce30 tx=0x7f4b5c00a650 comp rx=0 tx=0).stop 2024-09-16T03:52:58.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.232+0000 7f4b6b094640 1 -- 172.21.15.42:0/980321359 shutdown_connections 2024-09-16T03:52:58.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.232+0000 7f4b6b094640 1 --2- 172.21.15.42:0/980321359 >> v2:172.21.15.154:6812/254845342 conn(0x7f4b44037810 0x7f4b44039cd0 unknown :-1 s=CLOSED pgs=3586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:52:58.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.232+0000 7f4b6b094640 1 --2- 172.21.15.42:0/980321359 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4b64136e10 0x7f4b6413e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:52:58.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.232+0000 7f4b6b094640 1 --2- 172.21.15.42:0/980321359 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4b640a37f0 0x7f4b6413e1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:52:58.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.232+0000 7f4b6b094640 1 --2- 172.21.15.42:0/980321359 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4b6406db90 0x7f4b641cf1d0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:52:58.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.232+0000 7f4b6b094640 1 -- 172.21.15.42:0/980321359 >> 172.21.15.42:0/980321359 conn(0x7f4b640a9fb0 msgr2=0x7f4b64131940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:52:58.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.232+0000 7f4b6b094640 1 -- 172.21.15.42:0/980321359 shutdown_connections 2024-09-16T03:52:58.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.232+0000 7f4b6b094640 1 -- 172.21.15.42:0/980321359 wait complete. 2024-09-16T03:52:58.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.232+0000 7f4b6b094640 1 librados: shutdown 2024-09-16T03:52:58.254 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T03:52:58.254 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T03:52:58.254 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T03:52:58.255 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T03:52:58.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.376+0000 7f7811480640 10 monclient: get_monmap_and_config 2024-09-16T03:52:58.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.376+0000 7f7811480640 10 monclient: build_initial_monmap 2024-09-16T03:52:58.382 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.376+0000 7f7811480640 10 monclient: monmap: 2024-09-16T03:52:58.382 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:52:58.382 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:52:58.382 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:52:58.380455+0000 2024-09-16T03:52:58.382 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:52:58.380455+0000 2024-09-16T03:52:58.382 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:52:58.382 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:52:58.382 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:52:58.383 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:52:58.383 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:52:58.383 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:52:58.383 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:52:58.383 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:52:58.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.376+0000 7f7811480640 1 Processor -- start 2024-09-16T03:52:58.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.376+0000 7f7811480640 1 -- start start 2024-09-16T03:52:58.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.376+0000 7f7811480640 10 monclient: init 2024-09-16T03:52:58.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.376+0000 7f7811480640 10 monclient: _reopen_session rank -1 2024-09-16T03:52:58.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.376+0000 7f7811480640 10 monclient: _add_conns ranks=[4,2,0,1,3] 2024-09-16T03:52:58.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.376+0000 7f7811480640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f780c06dc10 0x7f780c133970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:52:58.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.376+0000 7f7811480640 10 monclient(hunting): picked mon.noname-c con 0x7f780c06dc10 addr v2:172.21.15.154:3300/0 2024-09-16T03:52:58.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.376+0000 7f7811480640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f780c0a38e0 0x7f780c0a3d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:52:58.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.376+0000 7f7811480640 10 monclient(hunting): picked mon.noname-b con 0x7f780c0a38e0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:52:58.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.376+0000 7f7811480640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f780c0a45b0 addr v1:172.21.15.42:6789/0 2024-09-16T03:52:58.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.376+0000 7f7811480640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f780c09be00 con 0x7f780c0a45b0 2024-09-16T03:52:58.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.376+0000 7f7811480640 10 monclient(hunting): start opening mon connection 2024-09-16T03:52:58.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.376+0000 7f7811480640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f780c09c350 con 0x7f780c06dc10 2024-09-16T03:52:58.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.376+0000 7f7811480640 10 monclient(hunting): start opening mon connection 2024-09-16T03:52:58.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.376+0000 7f7811480640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f780c134870 con 0x7f780c0a38e0 2024-09-16T03:52:58.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.376+0000 7f7811480640 10 monclient(hunting): _renew_subs 2024-09-16T03:52:58.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.376+0000 7f7811480640 10 monclient(hunting): authenticate will time out at 1625.349895s 2024-09-16T03:52:58.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.376+0000 7f780b7fe640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f780c0a45b0 0x7f780c0a4a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:41206/0 (socket says 172.21.15.42:41206) 2024-09-16T03:52:58.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.376+0000 7f780b7fe640 1 -- 172.21.15.42:0/1043116433 learned_addr learned my addr 172.21.15.42:0/1043116433 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:52:58.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.376+0000 7f780a7fc640 1 --2- 172.21.15.42:0/1043116433 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f780c0a38e0 0x7f780c0a3d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:52:58.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.376+0000 7f780a7fc640 10 monclient(hunting): get_auth_request con 0x7f780c0a38e0 auth_method 0 2024-09-16T03:52:58.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.376+0000 7f780a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:52:58.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.376+0000 7f780a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:52:58.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.376+0000 7f780a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:52:58.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.376+0000 7f780a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:52:58.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.376+0000 7f780a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:52:58.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.376+0000 7f780a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:52:58.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.376+0000 7f780affd640 1 --2- 172.21.15.42:0/1043116433 >> v2:172.21.15.154:3300/0 conn(0x7f780c06dc10 0x7f780c133970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:52:58.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.376+0000 7f780affd640 10 monclient(hunting): get_auth_request con 0x7f780c06dc10 auth_method 0 2024-09-16T03:52:58.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.376+0000 7f780affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:52:58.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.376+0000 7f780affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:52:58.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.376+0000 7f780affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:52:58.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.376+0000 7f780a7fc640 10 monclient(hunting): handle_auth_done global_id 834137 payload 274 2024-09-16T03:52:58.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.376+0000 7f780a7fc640 1 -- 172.21.15.42:0/1043116433 >> v2:172.21.15.154:3300/0 conn(0x7f780c06dc10 msgr2=0x7f780c133970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:52:58.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.376+0000 7f780a7fc640 1 --2- 172.21.15.42:0/1043116433 >> v2:172.21.15.154:3300/0 conn(0x7f780c06dc10 0x7f780c133970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:52:58.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.376+0000 7f7809ffb640 1 -- 172.21.15.42:0/1043116433 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 570298009 0 0) 0x7f780c09be00 con 0x7f780c0a45b0 2024-09-16T03:52:58.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.376+0000 7f780a7fc640 1 -- 172.21.15.42:0/1043116433 >> v1:172.21.15.42:6789/0 conn(0x7f780c0a45b0 legacy=0x7f780c0a4a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:52:58.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.376+0000 7f780a7fc640 10 monclient: _finish_hunting 0 2024-09-16T03:52:58.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.376+0000 7f780a7fc640 1 monclient: found mon.noname-b 2024-09-16T03:52:58.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.376+0000 7f780a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:52:58.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.376+0000 7f780a7fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T03:52:58.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.376+0000 7f780a7fc640 1 -- 172.21.15.42:0/1043116433 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f780c09c070 con 0x7f780c0a38e0 2024-09-16T03:52:58.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.376+0000 7f7809ffb640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T03:52:58.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.380+0000 7f780a7fc640 1 --2- 172.21.15.42:0/1043116433 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f780c0a38e0 0x7f780c0a3d00 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f77f40097c0 tx=0x7f77f401c3f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a63e774305187a15 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:52:58.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.380+0000 7f7809ffb640 1 -- 172.21.15.42:0/1043116433 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77f40b0070 con 0x7f780c0a38e0 2024-09-16T03:52:58.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.380+0000 7f7809ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:52:58.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.380+0000 7f7809ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:52:58.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.380+0000 7f7809ffb640 10 monclient: dump: 2024-09-16T03:52:58.396 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:52:58.396 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:52:58.396 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:52:58.397 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:52:58.397 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:52:58.397 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:52:58.397 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:52:58.397 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:52:58.397 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:52:58.397 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:52:58.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.380+0000 7f7809ffb640 10 monclient: _finish_auth 0 2024-09-16T03:52:58.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.380+0000 7f7809ffb640 10 monclient: _check_auth_tickets 2024-09-16T03:52:58.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.380+0000 7f7809ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:52:58.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.380+0000 7f7809ffb640 1 -- 172.21.15.42:0/1043116433 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f77f40ab600 con 0x7f780c0a38e0 2024-09-16T03:52:58.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.380+0000 7f7809ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T03:52:58.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.380+0000 7f7809ffb640 1 -- 172.21.15.42:0/1043116433 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77f40ab9d0 con 0x7f780c0a38e0 2024-09-16T03:52:58.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.380+0000 7f7811480640 5 monclient: authenticate success, global_id 834137 2024-09-16T03:52:58.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.380+0000 7f7811480640 10 monclient: get_monmap_and_config success 2024-09-16T03:52:58.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.380+0000 7f7811480640 4 set_mon_vals no callback set 2024-09-16T03:52:58.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.380+0000 7f7809ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:52:58.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.380+0000 7f7809ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:52:58.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.380+0000 7f7809ffb640 10 monclient: dump: 2024-09-16T03:52:58.398 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:52:58.398 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:52:58.398 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:52:58.398 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:52:58.398 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:52:58.398 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:52:58.399 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:52:58.399 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:52:58.399 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:52:58.399 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:52:58.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.380+0000 7f7811480640 10 monclient: shutdown 2024-09-16T03:52:58.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.380+0000 7f7811480640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:52:58.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.380+0000 7f7811480640 1 -- 172.21.15.42:0/1043116433 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f780c0a38e0 msgr2=0x7f780c0a3d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:52:58.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.380+0000 7f7811480640 1 --2- 172.21.15.42:0/1043116433 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f780c0a38e0 0x7f780c0a3d00 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f77f40097c0 tx=0x7f77f401c3f0 comp rx=0 tx=0).stop 2024-09-16T03:52:58.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.380+0000 7f7811480640 1 -- 172.21.15.42:0/1043116433 shutdown_connections 2024-09-16T03:52:58.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.380+0000 7f7811480640 1 --2- 172.21.15.42:0/1043116433 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f780c0a38e0 0x7f780c0a3d00 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:52:58.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.380+0000 7f7811480640 1 --2- 172.21.15.42:0/1043116433 >> v2:172.21.15.154:3300/0 conn(0x7f780c06dc10 0x7f780c133970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:52:58.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.380+0000 7f7811480640 1 -- 172.21.15.42:0/1043116433 >> 172.21.15.42:0/1043116433 conn(0x7f780c0a9fb0 msgr2=0x7f780c0aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:52:58.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.380+0000 7f7811480640 1 -- 172.21.15.42:0/1043116433 shutdown_connections 2024-09-16T03:52:58.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.380+0000 7f7811480640 1 -- 172.21.15.42:0/1043116433 wait complete. 2024-09-16T03:52:58.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.380+0000 7f7811480640 10 monclient: build_initial_monmap 2024-09-16T03:52:58.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.380+0000 7f7811480640 10 monclient: monmap: 2024-09-16T03:52:58.400 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:52:58.400 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:52:58.400 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:52:58.400 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:52:58.400 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:52:58.400 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:52:58.400 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:52:58.400 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:52:58.400 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:52:58.401 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:52:58.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.380+0000 7f7811480640 1 librados: starting msgr at 2024-09-16T03:52:58.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.380+0000 7f7811480640 1 librados: starting objecter 2024-09-16T03:52:58.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.380+0000 7f7811480640 1 Processor -- start 2024-09-16T03:52:58.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.380+0000 7f7811480640 1 -- start start 2024-09-16T03:52:58.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.380+0000 7f7811480640 1 librados: setting wanted keys 2024-09-16T03:52:58.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.380+0000 7f7811480640 1 librados: calling monclient init 2024-09-16T03:52:58.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.380+0000 7f7811480640 10 monclient: init 2024-09-16T03:52:58.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.380+0000 7f7811480640 10 monclient: _reopen_session rank -1 2024-09-16T03:52:58.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.380+0000 7f7811480640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:52:58.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.380+0000 7f7811480640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f780c06dc10 0x7f780c130940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:52:58.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.380+0000 7f7811480640 10 monclient(hunting): picked mon.noname-c con 0x7f780c06dc10 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:52:58.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.380+0000 7f7811480640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f780c0a38e0 0x7f780c1357b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:52:58.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.380+0000 7f7811480640 10 monclient(hunting): picked mon.noname-b con 0x7f780c0a38e0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:52:58.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.380+0000 7f7811480640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f780c0a45b0 0x7f780c135cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:52:58.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.380+0000 7f7811480640 10 monclient(hunting): picked mon.noname-a con 0x7f780c0a45b0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:52:58.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.380+0000 7f7811480640 10 monclient(hunting): start opening mon connection 2024-09-16T03:52:58.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.380+0000 7f7811480640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f780c141dc0 con 0x7f780c0a45b0 2024-09-16T03:52:58.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.380+0000 7f7811480640 10 monclient(hunting): start opening mon connection 2024-09-16T03:52:58.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.380+0000 7f7811480640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f780c141c40 con 0x7f780c0a38e0 2024-09-16T03:52:58.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.380+0000 7f7811480640 10 monclient(hunting): start opening mon connection 2024-09-16T03:52:58.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.380+0000 7f7811480640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f780c141f40 con 0x7f780c06dc10 2024-09-16T03:52:58.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.380+0000 7f7811480640 10 monclient(hunting): _renew_subs 2024-09-16T03:52:58.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.380+0000 7f7811480640 10 monclient(hunting): authenticate will time out at 1625.354947s 2024-09-16T03:52:58.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.380+0000 7f780b7fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f780c0a45b0 0x7f780c135cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:52:58.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.380+0000 7f780b7fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f780c0a45b0 0x7f780c135cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:38554/0 (socket says 172.21.15.42:38554) 2024-09-16T03:52:58.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.380+0000 7f780affd640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f780c06dc10 0x7f780c130940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:52:58.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.380+0000 7f780b7fe640 1 -- 172.21.15.42:0/2002932548 learned_addr learned my addr 172.21.15.42:0/2002932548 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:52:58.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.380+0000 7f780b7fe640 10 monclient(hunting): get_auth_request con 0x7f780c0a45b0 auth_method 0 2024-09-16T03:52:58.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.380+0000 7f780b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:52:58.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.380+0000 7f780b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:52:58.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.380+0000 7f780b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:52:58.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.380+0000 7f780affd640 10 monclient(hunting): get_auth_request con 0x7f780c06dc10 auth_method 0 2024-09-16T03:52:58.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.380+0000 7f780affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:52:58.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.380+0000 7f780affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:52:58.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.380+0000 7f780affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:52:58.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.380+0000 7f780b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:52:58.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.380+0000 7f780b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:52:58.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.380+0000 7f780b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:52:58.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.380+0000 7f780affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:52:58.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.380+0000 7f780affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:52:58.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.380+0000 7f780affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:52:58.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.380+0000 7f780a7fc640 1 --2- 172.21.15.42:0/2002932548 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f780c0a38e0 0x7f780c1357b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:52:58.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.380+0000 7f780a7fc640 10 monclient(hunting): get_auth_request con 0x7f780c0a38e0 auth_method 0 2024-09-16T03:52:58.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.380+0000 7f780a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:52:58.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.380+0000 7f780a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:52:58.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.380+0000 7f780a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:52:58.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.380+0000 7f780b7fe640 10 monclient(hunting): handle_auth_done global_id 824241 payload 824 2024-09-16T03:52:58.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.380+0000 7f780b7fe640 1 -- 172.21.15.42:0/2002932548 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f780c06dc10 msgr2=0x7f780c130940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:52:58.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.380+0000 7f780b7fe640 1 --2- 172.21.15.42:0/2002932548 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f780c06dc10 0x7f780c130940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:52:58.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.380+0000 7f780b7fe640 1 -- 172.21.15.42:0/2002932548 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f780c0a38e0 msgr2=0x7f780c1357b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:52:58.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.380+0000 7f780b7fe640 1 --2- 172.21.15.42:0/2002932548 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f780c0a38e0 0x7f780c1357b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:52:58.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.380+0000 7f780b7fe640 10 monclient: _finish_hunting 0 2024-09-16T03:52:58.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.380+0000 7f780b7fe640 1 monclient: found mon.noname-a 2024-09-16T03:52:58.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.380+0000 7f780b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:52:58.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.380+0000 7f780b7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:52:58.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.380+0000 7f780b7fe640 1 -- 172.21.15.42:0/2002932548 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f780c139420 con 0x7f780c0a45b0 2024-09-16T03:52:58.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.380+0000 7f780affd640 1 --2- 172.21.15.42:0/2002932548 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f780c06dc10 0x7f780c130940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:52:58.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.380+0000 7f780b7fe640 1 --2- 172.21.15.42:0/2002932548 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f780c0a45b0 0x7f780c135cf0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f77fc00c480 tx=0x7f77fc00c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:52:58.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.380+0000 7f78037fe640 10 client.?.objecter ms_handle_connect 0x7f780c0a45b0 2024-09-16T03:52:58.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.380+0000 7f78037fe640 10 client.?.objecter resend_mon_ops 2024-09-16T03:52:58.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.380+0000 7f78037fe640 1 -- 172.21.15.42:0/2002932548 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77fc018070 con 0x7f780c0a45b0 2024-09-16T03:52:58.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.380+0000 7f78037fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:52:58.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.380+0000 7f78037fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:52:58.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.380+0000 7f78037fe640 10 monclient: dump: 2024-09-16T03:52:58.406 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:52:58.406 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:52:58.406 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:52:58.406 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:52:58.406 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:52:58.406 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:52:58.406 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:52:58.406 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:52:58.406 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:52:58.406 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:52:58.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.380+0000 7f78037fe640 10 monclient: _finish_auth 0 2024-09-16T03:52:58.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.380+0000 7f78037fe640 10 monclient: _check_auth_tickets 2024-09-16T03:52:58.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.380+0000 7f78037fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:52:58.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.380+0000 7f78037fe640 1 -- 172.21.15.42:0/2002932548 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f77fc0074e0 con 0x7f780c0a45b0 2024-09-16T03:52:58.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.380+0000 7f7811480640 5 monclient: authenticate success, global_id 824241 2024-09-16T03:52:58.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.380+0000 7f7811480640 10 monclient: _renew_subs 2024-09-16T03:52:58.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.380+0000 7f7811480640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:52:58.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.380+0000 7f7811480640 1 -- 172.21.15.42:0/2002932548 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f780c139700 con 0x7f780c0a45b0 2024-09-16T03:52:58.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.380+0000 7f78037fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:52:58.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.380+0000 7f78037fe640 1 -- 172.21.15.42:0/2002932548 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77fc004a30 con 0x7f780c0a45b0 2024-09-16T03:52:58.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.380+0000 7f78037fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:52:58.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.380+0000 7f78037fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:52:58.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.380+0000 7f78037fe640 10 monclient: dump: 2024-09-16T03:52:58.407 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:52:58.408 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:52:58.408 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:52:58.408 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:52:58.408 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:52:58.408 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:52:58.408 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:52:58.408 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:52:58.408 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:52:58.408 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:52:58.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.380+0000 7f78097fa640 4 set_mon_vals no callback set 2024-09-16T03:52:58.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.380+0000 7f7811480640 10 client.824241.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:52:58.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.380+0000 7f7811480640 10 monclient: _renew_subs 2024-09-16T03:52:58.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.380+0000 7f7811480640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:52:58.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.380+0000 7f7811480640 1 -- 172.21.15.42:0/2002932548 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f780c06f680 con 0x7f780c0a45b0 2024-09-16T03:52:58.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.380+0000 7f7811480640 1 librados: init done 2024-09-16T03:52:58.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.384+0000 7f7811480640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:52:58.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.384+0000 7f7811480640 1 --2- 172.21.15.42:0/2002932548 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f780c130e80 0x7f780c136230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:52:58.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.384+0000 7f7811480640 10 monclient(hunting): start opening mon connection 2024-09-16T03:52:58.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.384+0000 7f7811480640 1 -- 172.21.15.42:0/2002932548 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f780c133970 con 0x7f780c130e80 2024-09-16T03:52:58.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.384+0000 7f780affd640 1 --2- 172.21.15.42:0/2002932548 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f780c130e80 0x7f780c136230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:52:58.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.384+0000 7f780affd640 10 monclient: get_auth_request con 0x7f780c130e80 auth_method 0 2024-09-16T03:52:58.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.384+0000 7f780affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:52:58.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.384+0000 7f780affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:52:58.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.384+0000 7f780affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:52:58.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.384+0000 7f780affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:52:58.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.384+0000 7f780affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:52:58.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.384+0000 7f780affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:52:58.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.384+0000 7f78037fe640 1 -- 172.21.15.42:0/2002932548 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f77fc01b070 con 0x7f780c0a45b0 2024-09-16T03:52:58.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.384+0000 7f780affd640 10 monclient(hunting): handle_auth_done global_id 824244 payload 824 2024-09-16T03:52:58.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.384+0000 7f780affd640 1 -- 172.21.15.42:0/2002932548 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f780c137cf0 con 0x7f780c130e80 2024-09-16T03:52:58.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.384+0000 7f78037fe640 1 --2- 172.21.15.42:0/2002932548 >> v2:172.21.15.154:6812/254845342 conn(0x7f77ec037810 0x7f77ec039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:52:58.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.384+0000 7f78037fe640 1 -- 172.21.15.42:0/2002932548 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(345..345 src has 286..345) ==== 5769+0+0 (secure 0 0 0) 0x7f77fc048250 con 0x7f780c0a45b0 2024-09-16T03:52:58.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.384+0000 7f78037fe640 10 client.824241.objecter ms_dispatch 0x7f780c000ba0 osd_map(345..345 src has 286..345) 2024-09-16T03:52:58.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.384+0000 7f78037fe640 3 client.824241.objecter handle_osd_map got epochs [345,345] > 0 2024-09-16T03:52:58.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.384+0000 7f78037fe640 3 client.824241.objecter handle_osd_map decoding full epoch 345 2024-09-16T03:52:58.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.384+0000 7f780affd640 1 --2- 172.21.15.42:0/2002932548 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f780c130e80 0x7f780c136230 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f7804007590 tx=0x7f7804007a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:52:58.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.384+0000 7f78037fe640 20 client.824241.objecter dump_active .. 0 homeless 2024-09-16T03:52:58.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.384+0000 7f78037fe640 10 client.824241.objecter ms_handle_connect 0x7f780c130e80 2024-09-16T03:52:58.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.384+0000 7f78037fe640 10 client.824241.objecter resend_mon_ops 2024-09-16T03:52:58.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.384+0000 7f78037fe640 1 -- 172.21.15.42:0/2002932548 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7804014590 con 0x7f780c130e80 2024-09-16T03:52:58.411 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.384+0000 7f78037fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:52:58.411 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.384+0000 7f78037fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:52:58.411 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.384+0000 7f78037fe640 10 monclient: dump: 2024-09-16T03:52:58.411 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:52:58.411 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:52:58.411 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:52:58.411 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:52:58.411 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:52:58.411 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:52:58.411 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:52:58.411 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:52:58.411 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:52:58.412 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:52:58.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.384+0000 7f780a7fc640 1 --2- 172.21.15.42:0/2002932548 >> v2:172.21.15.154:6812/254845342 conn(0x7f77ec037810 0x7f77ec039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:52:58.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.384+0000 7f780a7fc640 10 monclient: get_auth_request con 0x7f77ec037810 auth_method 0 2024-09-16T03:52:58.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.384+0000 7f78037fe640 1 -- 172.21.15.42:0/2002932548 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f780401a070 con 0x7f780c130e80 2024-09-16T03:52:58.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.384+0000 7f78037fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:52:58.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.384+0000 7f78037fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:52:58.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.384+0000 7f78037fe640 1 -- 172.21.15.42:0/2002932548 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f780c130e80 msgr2=0x7f780c136230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:52:58.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.384+0000 7f78037fe640 1 --2- 172.21.15.42:0/2002932548 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f780c130e80 0x7f780c136230 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f7804007590 tx=0x7f7804007a50 comp rx=0 tx=0).stop 2024-09-16T03:52:58.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.384+0000 7f78037fe640 1 -- 172.21.15.42:0/2002932548 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f780c130e80 msgr2=0x7f780c136230 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:52:58.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.384+0000 7f78037fe640 1 --2- 172.21.15.42:0/2002932548 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f780c130e80 0x7f780c136230 secure :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f7804007590 tx=0x7f7804007a50 comp rx=0 tx=0).stop 2024-09-16T03:52:58.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.384+0000 7f780a7fc640 1 --2- 172.21.15.42:0/2002932548 >> v2:172.21.15.154:6812/254845342 conn(0x7f77ec037810 0x7f77ec039cd0 secure :-1 s=READY pgs=3587 cs=0 l=1 rev1=1 crypto rx=0x7f77f40ad000 tx=0x7f77f4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:52:58.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.384+0000 7f78037fe640 10 client.824241.objecter ms_handle_connect 0x7f77ec037810 2024-09-16T03:52:58.414 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.388+0000 7f7811480640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:52:58.414 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.388+0000 7f7811480640 1 --2- 172.21.15.42:0/2002932548 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f780c136770 0x7f780c09fa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:52:58.415 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.388+0000 7f7811480640 10 monclient(hunting): start opening mon connection 2024-09-16T03:52:58.415 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.388+0000 7f7811480640 1 -- 172.21.15.42:0/2002932548 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f780c136b80 con 0x7f780c136770 2024-09-16T03:52:58.415 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.388+0000 7f780affd640 1 --2- 172.21.15.42:0/2002932548 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f780c136770 0x7f780c09fa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:52:58.415 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.388+0000 7f780affd640 10 monclient: get_auth_request con 0x7f780c136770 auth_method 0 2024-09-16T03:52:58.415 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.388+0000 7f780affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:52:58.415 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.388+0000 7f780affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:52:58.415 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.388+0000 7f780affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:52:58.415 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.388+0000 7f780affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:52:58.415 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.388+0000 7f780affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:52:58.416 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.388+0000 7f780affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:52:58.416 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.388+0000 7f780affd640 10 monclient(hunting): handle_auth_done global_id 824247 payload 824 2024-09-16T03:52:58.416 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.388+0000 7f780affd640 1 -- 172.21.15.42:0/2002932548 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f780c068e40 con 0x7f780c136770 2024-09-16T03:52:58.416 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.392+0000 7f780affd640 1 --2- 172.21.15.42:0/2002932548 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f780c136770 0x7f780c09fa00 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f7804007fd0 tx=0x7f7804004a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:52:58.416 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.392+0000 7f78037fe640 10 client.824241.objecter ms_handle_connect 0x7f780c136770 2024-09-16T03:52:58.416 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.392+0000 7f78037fe640 10 client.824241.objecter resend_mon_ops 2024-09-16T03:52:58.416 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.392+0000 7f78037fe640 1 -- 172.21.15.42:0/2002932548 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78040148f0 con 0x7f780c136770 2024-09-16T03:52:58.416 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.392+0000 7f78037fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:52:58.416 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.392+0000 7f78037fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:52:58.416 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.392+0000 7f78037fe640 10 monclient: dump: 2024-09-16T03:52:58.416 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:52:58.416 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:52:58.416 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:52:58.417 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:52:58.417 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:52:58.417 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:52:58.417 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:52:58.417 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:52:58.417 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:52:58.417 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:52:58.417 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.392+0000 7f78037fe640 1 -- 172.21.15.42:0/2002932548 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4880 (secure 0 0 0) 0x7f78040148f0 con 0x7f780c136770 2024-09-16T03:52:58.417 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.392+0000 7f78037fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:52:58.417 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.392+0000 7f78037fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:52:58.417 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.392+0000 7f78037fe640 1 -- 172.21.15.42:0/2002932548 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f780c136770 msgr2=0x7f780c09fa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:52:58.417 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.392+0000 7f78037fe640 1 --2- 172.21.15.42:0/2002932548 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f780c136770 0x7f780c09fa00 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f7804007fd0 tx=0x7f7804004a50 comp rx=0 tx=0).stop 2024-09-16T03:52:58.418 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.392+0000 7f78037fe640 1 -- 172.21.15.42:0/2002932548 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f780c136770 msgr2=0x7f780c09fa00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:52:58.418 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.392+0000 7f78037fe640 1 --2- 172.21.15.42:0/2002932548 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f780c136770 0x7f780c09fa00 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:52:58.418 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:52:58.418 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:52:58.418 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:52:58.418 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "leader", 2024-09-16T03:52:58.418 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 94, 2024-09-16T03:52:58.418 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 68247, 2024-09-16T03:52:58.418 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:52:58.419 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:52:58.419 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T03:52:58.419 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T03:52:58.419 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:52:58.419 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 3, 2024-09-16T03:52:58.419 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:52:58.419 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:52:58.419 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:52:58.419 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:52:58.419 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:52:58.419 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:52:58.419 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:52:58.419 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:52:58.419 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:52:58.420 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:52:58.420 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:52:58.420 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:52:58.420 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:52:58.420 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:52:58.420 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:52:58.420 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:52:58.420 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:52:58.420 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:52:58.420 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:52:58.420 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:52:58.420 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:52:58.420 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:52:58.421 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:52:58.421 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:52:58.421 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:52:58.421 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:52:58.421 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:52:58.421 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:52:58.421 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:52:58.421 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:52:58.421 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:52:58.421 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:52:58.421 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:52:58.421 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:52:58.422 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:52:58.422 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:52:58.422 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:52:58.422 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:52:58.422 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:52:58.422 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:52:58.422 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:52:58.422 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:52:58.422 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:52:58.422 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:52:58.422 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:52:58.422 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:52:58.422 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:52:58.423 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:52:58.423 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:52:58.423 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:52:58.423 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:52:58.423 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:52:58.423 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:52:58.423 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:52:58.423 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:52:58.423 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:52:58.423 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:52:58.423 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:52:58.423 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:52:58.424 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:52:58.424 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:52:58.424 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:52:58.424 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:52:58.424 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:52:58.424 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:52:58.424 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:52:58.424 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:52:58.424 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:52:58.424 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:52:58.424 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:52:58.424 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:52:58.425 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:52:58.425 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:52:58.425 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:52:58.425 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:52:58.425 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:52:58.425 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:52:58.425 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:52:58.425 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:52:58.425 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:52:58.425 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:52:58.425 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:52:58.425 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:52:58.426 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:52:58.426 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:52:58.426 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:52:58.426 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:52:58.426 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:52:58.426 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:52:58.426 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:52:58.426 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:52:58.426 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:52:58.426 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:52:58.426 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:52:58.426 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:52:58.427 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:52:58.427 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:52:58.427 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:52:58.427 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:52:58.427 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:52:58.427 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:52:58.427 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:52:58.427 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:52:58.427 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:52:58.427 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:52:58.427 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:52:58.428 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:52:58.428 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:52:58.428 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:52:58.428 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:52:58.428 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:52:58.428 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:52:58.428 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:52:58.428 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:52:58.428 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:52:58.428 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:52:58.428 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:52:58.428 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:52:58.428 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:52:58.429 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:52:58.429 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:52:58.429 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:52:58.429 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:52:58.429 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:52:58.429 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:52:58.429 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:52:58.429 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:52:58.429 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:52:58.429 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:52:58.429 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:52:58.430 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:52:58.430 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:52:58.430 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:52:58.430 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:52:58.430 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:52:58.430 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:52:58.430 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:52:58.430 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:52:58.430 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:52:58.430 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:52:58.430 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:52:58.430 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T03:52:58.430 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:52:58.431 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:52:58.431 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:52:58.431 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 5 2024-09-16T03:52:58.431 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:52:58.431 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:52:58.431 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:52:58.431 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:52:58.431 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:52:58.431 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:52:58.431 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 7 2024-09-16T03:52:58.431 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:52:58.431 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:52:58.431 INFO:teuthology.orchestra.run.smithi042.stdout: "mgr": [ 2024-09-16T03:52:58.431 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:52:58.432 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:52:58.432 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:52:58.432 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:52:58.432 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:52:58.432 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:52:58.432 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:52:58.432 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:52:58.432 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:52:58.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.392+0000 7f7811480640 10 librados: watch_flush enter 2024-09-16T03:52:58.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.392+0000 7f7811480640 10 librados: watch_flush exit 2024-09-16T03:52:58.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.392+0000 7f7811480640 20 client.824241.objecter shutdown clearing up homeless session... 2024-09-16T03:52:58.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.392+0000 7f7811480640 10 client.824241.objecter successfully canceled tick 2024-09-16T03:52:58.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.392+0000 7f7811480640 1 -- 172.21.15.42:0/2002932548 >> v2:172.21.15.154:6812/254845342 conn(0x7f77ec037810 msgr2=0x7f77ec039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:52:58.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.392+0000 7f7811480640 1 --2- 172.21.15.42:0/2002932548 >> v2:172.21.15.154:6812/254845342 conn(0x7f77ec037810 0x7f77ec039cd0 secure :-1 s=READY pgs=3587 cs=0 l=1 rev1=1 crypto rx=0x7f77f40ad000 tx=0x7f77f4008040 comp rx=0 tx=0).stop 2024-09-16T03:52:58.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.392+0000 7f7811480640 10 monclient: shutdown 2024-09-16T03:52:58.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.392+0000 7f7811480640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:52:58.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.392+0000 7f7811480640 1 -- 172.21.15.42:0/2002932548 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f780c0a45b0 msgr2=0x7f780c135cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:52:58.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.392+0000 7f7811480640 1 --2- 172.21.15.42:0/2002932548 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f780c0a45b0 0x7f780c135cf0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f77fc00c480 tx=0x7f77fc00c940 comp rx=0 tx=0).stop 2024-09-16T03:52:58.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.392+0000 7f780b7fe640 1 -- 172.21.15.42:0/2002932548 reap_dead start 2024-09-16T03:52:58.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.392+0000 7f780b7fe640 1 -- 172.21.15.42:0/2002932548 reap_dead start 2024-09-16T03:52:58.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.392+0000 7f7811480640 1 -- 172.21.15.42:0/2002932548 shutdown_connections 2024-09-16T03:52:58.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.392+0000 7f7811480640 1 -- 172.21.15.42:0/2002932548 >> 172.21.15.42:0/2002932548 conn(0x7f780c0a9fb0 msgr2=0x7f780c132700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:52:58.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.392+0000 7f7811480640 1 -- 172.21.15.42:0/2002932548 shutdown_connections 2024-09-16T03:52:58.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.392+0000 7f7811480640 1 -- 172.21.15.42:0/2002932548 wait complete. 2024-09-16T03:52:58.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.392+0000 7f7811480640 1 librados: shutdown 2024-09-16T03:52:58.435 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:52:58.435 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T03:52:58.436 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T03:52:58.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.548+0000 7f69ddbd3640 10 monclient: get_monmap_and_config 2024-09-16T03:52:58.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.548+0000 7f69ddbd3640 10 monclient: build_initial_monmap 2024-09-16T03:52:58.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.548+0000 7f69ddbd3640 10 monclient: monmap: 2024-09-16T03:52:58.553 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:52:58.554 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:52:58.554 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:52:58.552339+0000 2024-09-16T03:52:58.554 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:52:58.552339+0000 2024-09-16T03:52:58.554 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:52:58.554 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:52:58.554 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:52:58.554 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:52:58.554 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:52:58.554 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:52:58.554 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:52:58.554 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:52:58.556 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.548+0000 7f69ddbd3640 1 Processor -- start 2024-09-16T03:52:58.556 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.548+0000 7f69ddbd3640 1 -- start start 2024-09-16T03:52:58.556 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.548+0000 7f69ddbd3640 10 monclient: init 2024-09-16T03:52:58.556 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.548+0000 7f69ddbd3640 10 monclient: _reopen_session rank -1 2024-09-16T03:52:58.556 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.548+0000 7f69ddbd3640 10 monclient: _add_conns ranks=[0,2,1,3,4] 2024-09-16T03:52:58.556 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.548+0000 7f69ddbd3640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f69d80a3860 addr v1:172.21.15.42:6789/0 2024-09-16T03:52:58.556 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.548+0000 7f69ddbd3640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f69d80a3c40 0x7f69d813c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:52:58.556 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.548+0000 7f69ddbd3640 10 monclient(hunting): picked mon.noname-b con 0x7f69d80a3c40 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:52:58.557 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.548+0000 7f69ddbd3640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f69d813c710 addr v1:172.21.15.154:6789/0 2024-09-16T03:52:58.557 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.548+0000 7f69ddbd3640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f69d8137730 con 0x7f69d80a3860 2024-09-16T03:52:58.557 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.548+0000 7f69ddbd3640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f69d809c010 con 0x7f69d813c710 2024-09-16T03:52:58.557 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.548+0000 7f69ddbd3640 10 monclient(hunting): start opening mon connection 2024-09-16T03:52:58.557 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.548+0000 7f69ddbd3640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f69d80a5e80 con 0x7f69d80a3c40 2024-09-16T03:52:58.557 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.548+0000 7f69ddbd3640 10 monclient(hunting): _renew_subs 2024-09-16T03:52:58.557 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.548+0000 7f69ddbd3640 10 monclient(hunting): authenticate will time out at 1625.521756s 2024-09-16T03:52:58.557 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.548+0000 7f69d6ffd640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f69d80a3c40 0x7f69d813c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:52:58.557 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.548+0000 7f69d77fe640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f69d80a3860 0x7f69d8137020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:41214/0 (socket says 172.21.15.42:41214) 2024-09-16T03:52:58.557 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.548+0000 7f69d6ffd640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f69d80a3c40 0x7f69d813c1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:43020/0 (socket says 172.21.15.42:43020) 2024-09-16T03:52:58.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.548+0000 7f69d77fe640 1 -- 172.21.15.42:0/1379891853 learned_addr learned my addr 172.21.15.42:0/1379891853 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:52:58.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.548+0000 7f69d6ffd640 10 monclient(hunting): get_auth_request con 0x7f69d80a3c40 auth_method 0 2024-09-16T03:52:58.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.548+0000 7f69d6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:52:58.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.548+0000 7f69d6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:52:58.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.548+0000 7f69d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:52:58.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.548+0000 7f69d6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:52:58.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.548+0000 7f69d6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:52:58.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.548+0000 7f69d6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:52:58.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.548+0000 7f69d67fc640 1 -- 172.21.15.42:0/1379891853 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 203403409 0 0) 0x7f69d8137730 con 0x7f69d80a3860 2024-09-16T03:52:58.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.548+0000 7f69d6ffd640 10 monclient(hunting): handle_auth_done global_id 834143 payload 274 2024-09-16T03:52:58.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.548+0000 7f69d6ffd640 1 -- 172.21.15.42:0/1379891853 >> v1:172.21.15.154:6789/0 conn(0x7f69d813c710 legacy=0x7f69d813eb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:52:58.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.548+0000 7f69d6ffd640 1 -- 172.21.15.42:0/1379891853 >> v1:172.21.15.42:6789/0 conn(0x7f69d80a3860 legacy=0x7f69d8137020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:52:58.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.548+0000 7f69d6ffd640 10 monclient: _finish_hunting 0 2024-09-16T03:52:58.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.548+0000 7f69d6ffd640 1 monclient: found mon.noname-b 2024-09-16T03:52:58.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.548+0000 7f69d6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:52:58.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.548+0000 7f69d6ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T03:52:58.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.548+0000 7f69d6ffd640 1 -- 172.21.15.42:0/1379891853 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69d809c2f0 con 0x7f69d80a3c40 2024-09-16T03:52:58.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.548+0000 7f69d67fc640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T03:52:58.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.552+0000 7f69d6ffd640 1 --2- 172.21.15.42:0/1379891853 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f69d80a3c40 0x7f69d813c1d0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f69d80a6000 tx=0x7f69cc01c340 comp rx=0 tx=0).ready entity=mon.2 client_cookie=54697cf76172233a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:52:58.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.552+0000 7f69d67fc640 1 -- 172.21.15.42:0/1379891853 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69cc0b0070 con 0x7f69d80a3c40 2024-09-16T03:52:58.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.552+0000 7f69d67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:52:58.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.552+0000 7f69d67fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:52:58.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.552+0000 7f69d67fc640 10 monclient: dump: 2024-09-16T03:52:58.566 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:52:58.566 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:52:58.566 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:52:58.566 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:52:58.567 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:52:58.567 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:52:58.567 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:52:58.567 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:52:58.567 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:52:58.567 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:52:58.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.552+0000 7f69d67fc640 10 monclient: _finish_auth 0 2024-09-16T03:52:58.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.552+0000 7f69d67fc640 10 monclient: _check_auth_tickets 2024-09-16T03:52:58.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.552+0000 7f69d67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:52:58.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.552+0000 7f69ddbd3640 5 monclient: authenticate success, global_id 834143 2024-09-16T03:52:58.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.552+0000 7f69ddbd3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:52:58.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.552+0000 7f69d67fc640 1 -- 172.21.15.42:0/1379891853 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f69cc0ab600 con 0x7f69d80a3c40 2024-09-16T03:52:58.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.552+0000 7f69d67fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:52:58.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.552+0000 7f69ddbd3640 10 monclient: get_monmap_and_config success 2024-09-16T03:52:58.568 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.552+0000 7f69ddbd3640 4 set_mon_vals no callback set 2024-09-16T03:52:58.568 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.552+0000 7f69d67fc640 1 -- 172.21.15.42:0/1379891853 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69cc0ab9d0 con 0x7f69d80a3c40 2024-09-16T03:52:58.568 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.552+0000 7f69d67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:52:58.568 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.552+0000 7f69d67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:52:58.568 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.552+0000 7f69d67fc640 10 monclient: dump: 2024-09-16T03:52:58.568 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:52:58.569 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:52:58.569 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:52:58.569 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:52:58.569 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:52:58.569 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:52:58.569 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:52:58.569 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:52:58.569 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:52:58.569 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:52:58.569 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.552+0000 7f69ddbd3640 10 monclient: shutdown 2024-09-16T03:52:58.569 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.552+0000 7f69ddbd3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:52:58.569 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.552+0000 7f69ddbd3640 1 -- 172.21.15.42:0/1379891853 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f69d80a3c40 msgr2=0x7f69d813c1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:52:58.570 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.552+0000 7f69ddbd3640 1 --2- 172.21.15.42:0/1379891853 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f69d80a3c40 0x7f69d813c1d0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f69d80a6000 tx=0x7f69cc01c340 comp rx=0 tx=0).stop 2024-09-16T03:52:58.570 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.552+0000 7f69ddbd3640 1 -- 172.21.15.42:0/1379891853 shutdown_connections 2024-09-16T03:52:58.570 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.552+0000 7f69ddbd3640 1 --2- 172.21.15.42:0/1379891853 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f69d80a3c40 0x7f69d813c1d0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:52:58.570 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.552+0000 7f69ddbd3640 1 -- 172.21.15.42:0/1379891853 >> 172.21.15.42:0/1379891853 conn(0x7f69d80a9f40 msgr2=0x7f69d80aa350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:52:58.570 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.552+0000 7f69ddbd3640 1 -- 172.21.15.42:0/1379891853 shutdown_connections 2024-09-16T03:52:58.570 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.552+0000 7f69ddbd3640 1 -- 172.21.15.42:0/1379891853 wait complete. 2024-09-16T03:52:58.570 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.552+0000 7f69ddbd3640 10 monclient: build_initial_monmap 2024-09-16T03:52:58.570 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.552+0000 7f69ddbd3640 10 monclient: monmap: 2024-09-16T03:52:58.570 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:52:58.570 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:52:58.570 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:52:58.570 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:52:58.570 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:52:58.570 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:52:58.571 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:52:58.571 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:52:58.571 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:52:58.571 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:52:58.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.552+0000 7f69ddbd3640 1 librados: starting msgr at 2024-09-16T03:52:58.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.552+0000 7f69ddbd3640 1 librados: starting objecter 2024-09-16T03:52:58.572 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.552+0000 7f69ddbd3640 1 Processor -- start 2024-09-16T03:52:58.572 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.552+0000 7f69ddbd3640 1 -- start start 2024-09-16T03:52:58.572 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.552+0000 7f69ddbd3640 1 librados: setting wanted keys 2024-09-16T03:52:58.572 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.552+0000 7f69ddbd3640 1 librados: calling monclient init 2024-09-16T03:52:58.572 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.552+0000 7f69ddbd3640 10 monclient: init 2024-09-16T03:52:58.572 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.552+0000 7f69ddbd3640 10 monclient: _reopen_session rank -1 2024-09-16T03:52:58.572 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.552+0000 7f69ddbd3640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:52:58.572 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.552+0000 7f69ddbd3640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f69d80a3860 0x7f69d81cf460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:52:58.572 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.552+0000 7f69ddbd3640 10 monclient(hunting): picked mon.noname-a con 0x7f69d80a3860 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:52:58.572 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.552+0000 7f69ddbd3640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f69d80a3c40 0x7f69d8130cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:52:58.572 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.552+0000 7f69ddbd3640 10 monclient(hunting): picked mon.noname-c con 0x7f69d80a3c40 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:52:58.572 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.552+0000 7f69ddbd3640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f69d813c710 0x7f69d81311f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:52:58.572 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.552+0000 7f69ddbd3640 10 monclient(hunting): picked mon.noname-b con 0x7f69d813c710 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:52:58.573 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.552+0000 7f69ddbd3640 10 monclient(hunting): start opening mon connection 2024-09-16T03:52:58.573 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.552+0000 7f69ddbd3640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f69d8142dd0 con 0x7f69d80a3860 2024-09-16T03:52:58.573 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.552+0000 7f69ddbd3640 10 monclient(hunting): start opening mon connection 2024-09-16T03:52:58.573 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.552+0000 7f69ddbd3640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f69d8142c50 con 0x7f69d813c710 2024-09-16T03:52:58.573 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.552+0000 7f69ddbd3640 10 monclient(hunting): start opening mon connection 2024-09-16T03:52:58.573 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.552+0000 7f69ddbd3640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f69d8142f50 con 0x7f69d80a3c40 2024-09-16T03:52:58.573 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.552+0000 7f69ddbd3640 10 monclient(hunting): _renew_subs 2024-09-16T03:52:58.573 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.552+0000 7f69ddbd3640 10 monclient(hunting): authenticate will time out at 1625.526672s 2024-09-16T03:52:58.573 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.552+0000 7f69d6ffd640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f69d80a3c40 0x7f69d8130cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:52:58.573 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.552+0000 7f69d77fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f69d80a3860 0x7f69d81cf460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:52:58.573 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.552+0000 7f69d6ffd640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f69d80a3c40 0x7f69d8130cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:43022/0 (socket says 172.21.15.42:43022) 2024-09-16T03:52:58.573 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.552+0000 7f69d6ffd640 1 -- 172.21.15.42:0/1913685331 learned_addr learned my addr 172.21.15.42:0/1913685331 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:52:58.573 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.552+0000 7f69d6ffd640 10 monclient(hunting): get_auth_request con 0x7f69d80a3c40 auth_method 0 2024-09-16T03:52:58.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.552+0000 7f69d6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:52:58.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.552+0000 7f69d6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:52:58.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.552+0000 7f69d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:52:58.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.552+0000 7f69d77fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f69d80a3860 0x7f69d81cf460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:38594/0 (socket says 172.21.15.42:38594) 2024-09-16T03:52:58.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.552+0000 7f69d77fe640 10 monclient(hunting): get_auth_request con 0x7f69d80a3860 auth_method 0 2024-09-16T03:52:58.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.552+0000 7f69d77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:52:58.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.552+0000 7f69d77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:52:58.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.552+0000 7f69d77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:52:58.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.552+0000 7f69d77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:52:58.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.552+0000 7f69d77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:52:58.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.552+0000 7f69d77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:52:58.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.552+0000 7f69d6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:52:58.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.552+0000 7f69d6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:52:58.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.552+0000 7f69d6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:52:58.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.552+0000 7f69d7fff640 1 --2- 172.21.15.42:0/1913685331 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f69d813c710 0x7f69d81311f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:52:58.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.552+0000 7f69d7fff640 10 monclient(hunting): get_auth_request con 0x7f69d813c710 auth_method 0 2024-09-16T03:52:58.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.552+0000 7f69d7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:52:58.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.552+0000 7f69d7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:52:58.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.552+0000 7f69d7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:52:58.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.552+0000 7f69d6ffd640 10 monclient(hunting): handle_auth_done global_id 834146 payload 824 2024-09-16T03:52:58.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.552+0000 7f69d6ffd640 1 -- 172.21.15.42:0/1913685331 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f69d813c710 msgr2=0x7f69d81311f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:52:58.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.552+0000 7f69d6ffd640 1 --2- 172.21.15.42:0/1913685331 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f69d813c710 0x7f69d81311f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:52:58.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.552+0000 7f69d6ffd640 1 -- 172.21.15.42:0/1913685331 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f69d80a3860 msgr2=0x7f69d81cf460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:52:58.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.552+0000 7f69d6ffd640 1 --2- 172.21.15.42:0/1913685331 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f69d80a3860 0x7f69d81cf460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:52:58.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.552+0000 7f69d6ffd640 10 monclient: _finish_hunting 0 2024-09-16T03:52:58.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.552+0000 7f69d6ffd640 1 monclient: found mon.noname-c 2024-09-16T03:52:58.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.552+0000 7f69d6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:52:58.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.552+0000 7f69d6ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:52:58.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.552+0000 7f69d6ffd640 1 -- 172.21.15.42:0/1913685331 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69d813b150 con 0x7f69d80a3c40 2024-09-16T03:52:58.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.552+0000 7f69d77fe640 1 --2- 172.21.15.42:0/1913685331 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f69d80a3860 0x7f69d81cf460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:52:58.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.552+0000 7f69d6ffd640 1 --2- 172.21.15.42:0/1913685331 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f69d80a3c40 0x7f69d8130cb0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f69cc0ae040 tx=0x7f69cc01ab80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:52:58.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.552+0000 7f69bffff640 10 client.?.objecter ms_handle_connect 0x7f69d80a3c40 2024-09-16T03:52:58.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.552+0000 7f69bffff640 10 client.?.objecter resend_mon_ops 2024-09-16T03:52:58.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.552+0000 7f69bffff640 1 -- 172.21.15.42:0/1913685331 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69cc0b0040 con 0x7f69d80a3c40 2024-09-16T03:52:58.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.552+0000 7f69bffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:52:58.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.552+0000 7f69bffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:52:58.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.552+0000 7f69bffff640 10 monclient: dump: 2024-09-16T03:52:58.576 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:52:58.576 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:52:58.576 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:52:58.577 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:52:58.577 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:52:58.577 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:52:58.577 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:52:58.577 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:52:58.577 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:52:58.577 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:52:58.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.552+0000 7f69bffff640 10 monclient: _finish_auth 0 2024-09-16T03:52:58.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.552+0000 7f69bffff640 10 monclient: _check_auth_tickets 2024-09-16T03:52:58.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.552+0000 7f69bffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:52:58.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.552+0000 7f69bffff640 1 -- 172.21.15.42:0/1913685331 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f69cc004230 con 0x7f69d80a3c40 2024-09-16T03:52:58.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.552+0000 7f69ddbd3640 5 monclient: authenticate success, global_id 834146 2024-09-16T03:52:58.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.552+0000 7f69ddbd3640 10 monclient: _renew_subs 2024-09-16T03:52:58.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.552+0000 7f69ddbd3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:52:58.578 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.552+0000 7f69ddbd3640 1 -- 172.21.15.42:0/1913685331 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69d806f610 con 0x7f69d80a3c40 2024-09-16T03:52:58.578 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.552+0000 7f69bffff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:52:58.578 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.552+0000 7f69bffff640 1 -- 172.21.15.42:0/1913685331 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69cc003620 con 0x7f69d80a3c40 2024-09-16T03:52:58.578 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.552+0000 7f69bffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:52:58.578 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.552+0000 7f69bffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:52:58.578 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.552+0000 7f69bffff640 10 monclient: dump: 2024-09-16T03:52:58.578 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:52:58.578 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:52:58.578 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:52:58.578 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:52:58.578 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:52:58.578 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:52:58.578 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:52:58.578 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:52:58.579 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:52:58.579 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:52:58.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.552+0000 7f69ddbd3640 10 client.834146.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:52:58.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.552+0000 7f69ddbd3640 10 monclient: _renew_subs 2024-09-16T03:52:58.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.552+0000 7f69ddbd3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:52:58.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.552+0000 7f69ddbd3640 1 -- 172.21.15.42:0/1913685331 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f69d812e790 con 0x7f69d80a3c40 2024-09-16T03:52:58.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.552+0000 7f69ddbd3640 1 librados: init done 2024-09-16T03:52:58.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.552+0000 7f69d5ffb640 4 set_mon_vals no callback set 2024-09-16T03:52:58.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.556+0000 7f69bffff640 1 -- 172.21.15.42:0/1913685331 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f69cc0ab4c0 con 0x7f69d80a3c40 2024-09-16T03:52:58.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.556+0000 7f69ddbd3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:52:58.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.556+0000 7f69ddbd3640 1 --2- 172.21.15.42:0/1913685331 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f69d8139570 0x7f69d8139950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:52:58.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.556+0000 7f69ddbd3640 10 monclient(hunting): start opening mon connection 2024-09-16T03:52:58.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.556+0000 7f69ddbd3640 1 -- 172.21.15.42:0/1913685331 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f69d813c4f0 con 0x7f69d8139570 2024-09-16T03:52:58.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.556+0000 7f69d77fe640 1 --2- 172.21.15.42:0/1913685331 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f69d8139570 0x7f69d8139950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:52:58.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.556+0000 7f69d77fe640 10 monclient: get_auth_request con 0x7f69d8139570 auth_method 0 2024-09-16T03:52:58.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.556+0000 7f69d77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:52:58.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.556+0000 7f69d77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:52:58.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.556+0000 7f69d77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:52:58.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.556+0000 7f69bffff640 1 --2- 172.21.15.42:0/1913685331 >> v2:172.21.15.154:6812/254845342 conn(0x7f69c4037810 0x7f69c4039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:52:58.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.556+0000 7f69d77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:52:58.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.556+0000 7f69d77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:52:58.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.556+0000 7f69bffff640 1 -- 172.21.15.42:0/1913685331 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(345..345 src has 286..345) ==== 5769+0+0 (secure 0 0 0) 0x7f69cc0e3dc0 con 0x7f69d80a3c40 2024-09-16T03:52:58.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.556+0000 7f69bffff640 10 client.834146.objecter ms_dispatch 0x7f69d8000ba0 osd_map(345..345 src has 286..345) 2024-09-16T03:52:58.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.556+0000 7f69bffff640 3 client.834146.objecter handle_osd_map got epochs [345,345] > 0 2024-09-16T03:52:58.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.556+0000 7f69bffff640 3 client.834146.objecter handle_osd_map decoding full epoch 345 2024-09-16T03:52:58.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.556+0000 7f69d77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:52:58.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.556+0000 7f69bffff640 20 client.834146.objecter dump_active .. 0 homeless 2024-09-16T03:52:58.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.556+0000 7f69d77fe640 10 monclient(hunting): handle_auth_done global_id 834149 payload 824 2024-09-16T03:52:58.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.556+0000 7f69d77fe640 1 -- 172.21.15.42:0/1913685331 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f69d8137000 con 0x7f69d8139570 2024-09-16T03:52:58.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.556+0000 7f69d7fff640 1 --2- 172.21.15.42:0/1913685331 >> v2:172.21.15.154:6812/254845342 conn(0x7f69c4037810 0x7f69c4039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:52:58.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.556+0000 7f69d7fff640 10 monclient: get_auth_request con 0x7f69c4037810 auth_method 0 2024-09-16T03:52:58.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.556+0000 7f69bffff640 10 client.834146.objecter ms_handle_connect 0x7f69d8139570 2024-09-16T03:52:58.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.556+0000 7f69bffff640 10 client.834146.objecter resend_mon_ops 2024-09-16T03:52:58.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.556+0000 7f69d77fe640 1 --2- 172.21.15.42:0/1913685331 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f69d8139570 0x7f69d8139950 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f69c8000f70 tx=0x7f69c8001cb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:52:58.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.556+0000 7f69bffff640 1 -- 172.21.15.42:0/1913685331 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69c800fac0 con 0x7f69d8139570 2024-09-16T03:52:58.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.556+0000 7f69bffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:52:58.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.556+0000 7f69bffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:52:58.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.556+0000 7f69bffff640 10 monclient: dump: 2024-09-16T03:52:58.581 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:52:58.581 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:52:58.582 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:52:58.582 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:52:58.582 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:52:58.582 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:52:58.582 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:52:58.582 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:52:58.582 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:52:58.582 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:52:58.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.556+0000 7f69bffff640 1 -- 172.21.15.42:0/1913685331 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f69c8018070 con 0x7f69d8139570 2024-09-16T03:52:58.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.556+0000 7f69bffff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:52:58.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.556+0000 7f69bffff640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:52:58.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.556+0000 7f69bffff640 1 -- 172.21.15.42:0/1913685331 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f69d8139570 msgr2=0x7f69d8139950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:52:58.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.556+0000 7f69bffff640 1 --2- 172.21.15.42:0/1913685331 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f69d8139570 0x7f69d8139950 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f69c8000f70 tx=0x7f69c8001cb0 comp rx=0 tx=0).stop 2024-09-16T03:52:58.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.556+0000 7f69bffff640 1 -- 172.21.15.42:0/1913685331 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f69d8139570 msgr2=0x7f69d8139950 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:52:58.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.556+0000 7f69bffff640 1 --2- 172.21.15.42:0/1913685331 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f69d8139570 0x7f69d8139950 secure :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f69c8000f70 tx=0x7f69c8001cb0 comp rx=0 tx=0).stop 2024-09-16T03:52:58.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.556+0000 7f69bffff640 10 client.834146.objecter ms_handle_connect 0x7f69c4037810 2024-09-16T03:52:58.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.556+0000 7f69d7fff640 1 --2- 172.21.15.42:0/1913685331 >> v2:172.21.15.154:6812/254845342 conn(0x7f69c4037810 0x7f69c4039cd0 secure :-1 s=READY pgs=3588 cs=0 l=1 rev1=1 crypto rx=0x7f69d806cf30 tx=0x7f69d0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:52:58.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.560+0000 7f69ddbd3640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:52:58.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.560+0000 7f69ddbd3640 1 --2- 172.21.15.42:0/1913685331 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f69d8139e90 0x7f69d813caf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:52:58.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.560+0000 7f69ddbd3640 10 monclient(hunting): start opening mon connection 2024-09-16T03:52:58.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.560+0000 7f69ddbd3640 1 -- 172.21.15.42:0/1913685331 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f69d812e2d0 con 0x7f69d8139e90 2024-09-16T03:52:58.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.560+0000 7f69d77fe640 1 --2- 172.21.15.42:0/1913685331 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f69d8139e90 0x7f69d813caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:52:58.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.560+0000 7f69d77fe640 10 monclient: get_auth_request con 0x7f69d8139e90 auth_method 0 2024-09-16T03:52:58.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.560+0000 7f69d77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:52:58.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.560+0000 7f69d77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:52:58.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.560+0000 7f69d77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:52:58.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.560+0000 7f69d77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:52:58.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.560+0000 7f69d77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:52:58.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.560+0000 7f69d77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:52:58.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.564+0000 7f69d77fe640 10 monclient(hunting): handle_auth_done global_id 834152 payload 824 2024-09-16T03:52:58.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.564+0000 7f69d77fe640 1 -- 172.21.15.42:0/1913685331 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f69d8068e40 con 0x7f69d8139e90 2024-09-16T03:52:58.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.564+0000 7f69bffff640 10 client.834146.objecter ms_handle_connect 0x7f69d8139e90 2024-09-16T03:52:58.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.564+0000 7f69bffff640 10 client.834146.objecter resend_mon_ops 2024-09-16T03:52:58.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.564+0000 7f69d77fe640 1 --2- 172.21.15.42:0/1913685331 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f69d8139e90 0x7f69d813caf0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f69c8000f70 tx=0x7f69c8001090 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:52:58.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.564+0000 7f69bffff640 1 -- 172.21.15.42:0/1913685331 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69c80136c0 con 0x7f69d8139e90 2024-09-16T03:52:58.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.564+0000 7f69bffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:52:58.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.564+0000 7f69bffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:52:58.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.564+0000 7f69bffff640 10 monclient: dump: 2024-09-16T03:52:58.586 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:52:58.586 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:52:58.586 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:52:58.586 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:52:58.586 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:52:58.586 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:52:58.586 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:52:58.586 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:52:58.586 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:52:58.587 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:52:58.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.564+0000 7f69bffff640 1 -- 172.21.15.42:0/1913685331 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4544 (secure 0 0 0) 0x7f69c80136c0 con 0x7f69d8139e90 2024-09-16T03:52:58.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.564+0000 7f69bffff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:52:58.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.564+0000 7f69bffff640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:52:58.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.564+0000 7f69bffff640 1 -- 172.21.15.42:0/1913685331 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f69d8139e90 msgr2=0x7f69d813caf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:52:58.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.564+0000 7f69bffff640 1 --2- 172.21.15.42:0/1913685331 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f69d8139e90 0x7f69d813caf0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f69c8000f70 tx=0x7f69c8001090 comp rx=0 tx=0).stop 2024-09-16T03:52:58.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.564+0000 7f69bffff640 1 -- 172.21.15.42:0/1913685331 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f69d8139e90 msgr2=0x7f69d813caf0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:52:58.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.564+0000 7f69bffff640 1 --2- 172.21.15.42:0/1913685331 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f69d8139e90 0x7f69d813caf0 secure :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f69c8000f70 tx=0x7f69c8001090 comp rx=0 tx=0).stop 2024-09-16T03:52:58.587 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:52:58.587 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:52:58.588 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:52:58.588 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T03:52:58.588 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 94, 2024-09-16T03:52:58.588 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 18135, 2024-09-16T03:52:58.588 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:52:58.588 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:52:58.588 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T03:52:58.588 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T03:52:58.588 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:52:58.588 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 3, 2024-09-16T03:52:58.588 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:52:58.588 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:52:58.588 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:52:58.589 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:52:58.589 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:52:58.589 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:52:58.589 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:52:58.589 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:52:58.589 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:52:58.589 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:52:58.589 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:52:58.589 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:52:58.589 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:52:58.589 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:52:58.589 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:52:58.589 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:52:58.589 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:52:58.590 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:52:58.590 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:52:58.590 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:52:58.590 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:52:58.590 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:52:58.590 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:52:58.590 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:52:58.590 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:52:58.590 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:52:58.590 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:52:58.590 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:52:58.590 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:52:58.591 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:52:58.591 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:52:58.591 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:52:58.591 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:52:58.591 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:52:58.591 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:52:58.591 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:52:58.591 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:52:58.591 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:52:58.591 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:52:58.592 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:52:58.592 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:52:58.592 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:52:58.592 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:52:58.592 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:52:58.592 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:52:58.592 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:52:58.592 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:52:58.592 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:52:58.592 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:52:58.592 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:52:58.592 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:52:58.592 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:52:58.592 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:52:58.593 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:52:58.593 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:52:58.593 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:52:58.593 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:52:58.593 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:52:58.593 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:52:58.593 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:52:58.593 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:52:58.593 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:52:58.593 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:52:58.593 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:52:58.593 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:52:58.593 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:52:58.594 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:52:58.594 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:52:58.594 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:52:58.594 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:52:58.594 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:52:58.594 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:52:58.594 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:52:58.594 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:52:58.594 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:52:58.594 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:52:58.594 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:52:58.594 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:52:58.594 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:52:58.594 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:52:58.595 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:52:58.595 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:52:58.595 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:52:58.595 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:52:58.595 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:52:58.595 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:52:58.595 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:52:58.595 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:52:58.595 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:52:58.595 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:52:58.595 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:52:58.595 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:52:58.595 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:52:58.595 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:52:58.596 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:52:58.596 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:52:58.596 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:52:58.596 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:52:58.596 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:52:58.596 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:52:58.596 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:52:58.596 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:52:58.596 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:52:58.596 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:52:58.596 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:52:58.596 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:52:58.596 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:52:58.596 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:52:58.597 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:52:58.597 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:52:58.597 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:52:58.597 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:52:58.597 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:52:58.597 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:52:58.597 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:52:58.597 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:52:58.597 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:52:58.597 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:52:58.597 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:52:58.597 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:52:58.597 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:52:58.597 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:52:58.598 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:52:58.598 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:52:58.598 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:52:58.598 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:52:58.598 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:52:58.598 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:52:58.598 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:52:58.598 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:52:58.598 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:52:58.598 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:52:58.598 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:52:58.598 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:52:58.598 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:52:58.599 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:52:58.599 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:52:58.599 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:52:58.599 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:52:58.599 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:52:58.599 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:52:58.599 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:52:58.599 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:52:58.599 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:52:58.599 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:52:58.599 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 2 2024-09-16T03:52:58.599 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:52:58.599 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:52:58.599 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:52:58.600 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:52:58.600 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:52:58.601 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.564+0000 7f69ddbd3640 10 librados: watch_flush enter 2024-09-16T03:52:58.602 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.564+0000 7f69ddbd3640 10 librados: watch_flush exit 2024-09-16T03:52:58.602 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.564+0000 7f69ddbd3640 20 client.834146.objecter shutdown clearing up homeless session... 2024-09-16T03:52:58.602 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.564+0000 7f69ddbd3640 10 client.834146.objecter successfully canceled tick 2024-09-16T03:52:58.602 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.564+0000 7f69ddbd3640 1 -- 172.21.15.42:0/1913685331 >> v2:172.21.15.154:6812/254845342 conn(0x7f69c4037810 msgr2=0x7f69c4039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:52:58.602 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.564+0000 7f69ddbd3640 1 --2- 172.21.15.42:0/1913685331 >> v2:172.21.15.154:6812/254845342 conn(0x7f69c4037810 0x7f69c4039cd0 secure :-1 s=READY pgs=3588 cs=0 l=1 rev1=1 crypto rx=0x7f69d806cf30 tx=0x7f69d0009040 comp rx=0 tx=0).stop 2024-09-16T03:52:58.602 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.564+0000 7f69ddbd3640 10 monclient: shutdown 2024-09-16T03:52:58.602 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.564+0000 7f69ddbd3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:52:58.602 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.564+0000 7f69ddbd3640 1 -- 172.21.15.42:0/1913685331 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f69d80a3c40 msgr2=0x7f69d8130cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:52:58.602 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.564+0000 7f69ddbd3640 1 --2- 172.21.15.42:0/1913685331 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f69d80a3c40 0x7f69d8130cb0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f69cc0ae040 tx=0x7f69cc01ab80 comp rx=0 tx=0).stop 2024-09-16T03:52:58.602 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.564+0000 7f69d7fff640 1 -- 172.21.15.42:0/1913685331 reap_dead start 2024-09-16T03:52:58.602 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.564+0000 7f69d7fff640 1 -- 172.21.15.42:0/1913685331 reap_dead start 2024-09-16T03:52:58.602 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.564+0000 7f69ddbd3640 1 -- 172.21.15.42:0/1913685331 shutdown_connections 2024-09-16T03:52:58.602 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.564+0000 7f69ddbd3640 1 -- 172.21.15.42:0/1913685331 >> 172.21.15.42:0/1913685331 conn(0x7f69d80a9f40 msgr2=0x7f69d8133880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:52:58.603 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.564+0000 7f69ddbd3640 1 -- 172.21.15.42:0/1913685331 shutdown_connections 2024-09-16T03:52:58.603 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.564+0000 7f69ddbd3640 1 -- 172.21.15.42:0/1913685331 wait complete. 2024-09-16T03:52:58.603 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.564+0000 7f69ddbd3640 1 librados: shutdown 2024-09-16T03:52:58.603 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:52:58.603 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T03:52:58.603 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T03:52:58.725 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.720+0000 7f4a128f8640 10 monclient: get_monmap_and_config 2024-09-16T03:52:58.725 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.720+0000 7f4a128f8640 10 monclient: build_initial_monmap 2024-09-16T03:52:58.725 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.720+0000 7f4a128f8640 10 monclient: monmap: 2024-09-16T03:52:58.726 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:52:58.726 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:52:58.726 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:52:58.724382+0000 2024-09-16T03:52:58.726 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:52:58.724382+0000 2024-09-16T03:52:58.726 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:52:58.726 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:52:58.726 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:52:58.726 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:52:58.726 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:52:58.726 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:52:58.726 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:52:58.726 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:52:58.729 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.720+0000 7f4a128f8640 1 Processor -- start 2024-09-16T03:52:58.729 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.720+0000 7f4a128f8640 1 -- start start 2024-09-16T03:52:58.729 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.720+0000 7f4a128f8640 10 monclient: init 2024-09-16T03:52:58.729 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.720+0000 7f4a128f8640 10 monclient: _reopen_session rank -1 2024-09-16T03:52:58.729 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.720+0000 7f4a128f8640 10 monclient: _add_conns ranks=[2,3,1,4,0] 2024-09-16T03:52:58.729 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.720+0000 7f4a128f8640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4a0c09fe60 0x7f4a0c0a0240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:52:58.729 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.720+0000 7f4a128f8640 10 monclient(hunting): picked mon.noname-b con 0x7f4a0c09fe60 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:52:58.729 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.720+0000 7f4a128f8640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f4a0c0a1fb0 0x7f4a0c0a0780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:52:58.729 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.720+0000 7f4a128f8640 10 monclient(hunting): picked mon.noname-a con 0x7f4a0c0a1fb0 addr v2:172.21.15.42:3300/0 2024-09-16T03:52:58.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.720+0000 7f4a128f8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4a0c137120 addr v1:172.21.15.154:6789/0 2024-09-16T03:52:58.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.720+0000 7f4a128f8640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4a0c06caa0 con 0x7f4a0c137120 2024-09-16T03:52:58.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.720+0000 7f4a128f8640 10 monclient(hunting): start opening mon connection 2024-09-16T03:52:58.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.720+0000 7f4a128f8640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f4a0c0a0cc0 con 0x7f4a0c0a1fb0 2024-09-16T03:52:58.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.720+0000 7f4a128f8640 10 monclient(hunting): start opening mon connection 2024-09-16T03:52:58.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.720+0000 7f4a128f8640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f4a0c06c7f0 con 0x7f4a0c09fe60 2024-09-16T03:52:58.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.720+0000 7f4a128f8640 10 monclient(hunting): _renew_subs 2024-09-16T03:52:58.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.720+0000 7f4a128f8640 10 monclient(hunting): authenticate will time out at 1625.693903s 2024-09-16T03:52:58.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.720+0000 7f4a0bfff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4a0c09fe60 0x7f4a0c0a0240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:52:58.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.720+0000 7f4a03fff640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f4a0c0a1fb0 0x7f4a0c0a0780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:52:58.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.720+0000 7f4a0bfff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4a0c09fe60 0x7f4a0c0a0240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:43054/0 (socket says 172.21.15.42:43054) 2024-09-16T03:52:58.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.720+0000 7f4a0bfff640 1 -- 172.21.15.42:0/133700246 learned_addr learned my addr 172.21.15.42:0/133700246 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:52:58.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.720+0000 7f4a0bfff640 10 monclient(hunting): get_auth_request con 0x7f4a0c09fe60 auth_method 0 2024-09-16T03:52:58.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.720+0000 7f4a0bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:52:58.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.720+0000 7f4a0bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:52:58.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.720+0000 7f4a0bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:52:58.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.720+0000 7f4a03fff640 10 monclient(hunting): get_auth_request con 0x7f4a0c0a1fb0 auth_method 0 2024-09-16T03:52:58.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.720+0000 7f4a03fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:52:58.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.720+0000 7f4a03fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:52:58.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.720+0000 7f4a03fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:52:58.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.720+0000 7f4a0bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:52:58.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.720+0000 7f4a0bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:52:58.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.720+0000 7f4a0bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:52:58.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.720+0000 7f4a03fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:52:58.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.720+0000 7f4a03fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:52:58.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.720+0000 7f4a03fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:52:58.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.720+0000 7f4a0bfff640 10 monclient(hunting): handle_auth_done global_id 834155 payload 274 2024-09-16T03:52:58.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.720+0000 7f4a0bfff640 1 -- 172.21.15.42:0/133700246 >> v2:172.21.15.42:3300/0 conn(0x7f4a0c0a1fb0 msgr2=0x7f4a0c0a0780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:52:58.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.720+0000 7f4a0bfff640 1 --2- 172.21.15.42:0/133700246 >> v2:172.21.15.42:3300/0 conn(0x7f4a0c0a1fb0 0x7f4a0c0a0780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:52:58.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.720+0000 7f4a0bfff640 1 -- 172.21.15.42:0/133700246 >> v1:172.21.15.154:6789/0 conn(0x7f4a0c137120 legacy=0x7f4a0c137570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:52:58.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.720+0000 7f4a0bfff640 10 monclient: _finish_hunting 0 2024-09-16T03:52:58.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.720+0000 7f4a0bfff640 1 monclient: found mon.noname-b 2024-09-16T03:52:58.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.720+0000 7f4a0bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:52:58.733 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.720+0000 7f4a0bfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T03:52:58.733 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.720+0000 7f4a0bfff640 1 -- 172.21.15.42:0/133700246 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a0c0a1d80 con 0x7f4a0c09fe60 2024-09-16T03:52:58.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.724+0000 7f4a0bfff640 1 --2- 172.21.15.42:0/133700246 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4a0c09fe60 0x7f4a0c0a0240 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f49fc009870 tx=0x7f49fc01c430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=135fc1d099dc0d4b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:52:58.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.724+0000 7f4a0b7fe640 1 -- 172.21.15.42:0/133700246 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49fc0b0070 con 0x7f4a0c09fe60 2024-09-16T03:52:58.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.724+0000 7f4a0b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:52:58.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.724+0000 7f4a0b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:52:58.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.724+0000 7f4a0b7fe640 10 monclient: dump: 2024-09-16T03:52:58.740 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:52:58.740 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:52:58.740 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:52:58.740 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:52:58.740 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:52:58.740 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:52:58.740 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:52:58.740 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:52:58.740 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:52:58.740 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:52:58.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.724+0000 7f4a0b7fe640 10 monclient: _finish_auth 0 2024-09-16T03:52:58.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.724+0000 7f4a0b7fe640 10 monclient: _check_auth_tickets 2024-09-16T03:52:58.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.724+0000 7f4a0b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:52:58.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.724+0000 7f4a0b7fe640 1 -- 172.21.15.42:0/133700246 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f49fc0ab600 con 0x7f4a0c09fe60 2024-09-16T03:52:58.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.724+0000 7f4a0b7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:52:58.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.724+0000 7f4a0b7fe640 1 -- 172.21.15.42:0/133700246 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49fc0ab9d0 con 0x7f4a0c09fe60 2024-09-16T03:52:58.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.724+0000 7f4a128f8640 5 monclient: authenticate success, global_id 834155 2024-09-16T03:52:58.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.724+0000 7f4a128f8640 10 monclient: get_monmap_and_config success 2024-09-16T03:52:58.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.724+0000 7f4a128f8640 4 set_mon_vals no callback set 2024-09-16T03:52:58.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.724+0000 7f4a0b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:52:58.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.724+0000 7f4a0b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:52:58.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.724+0000 7f4a0b7fe640 10 monclient: dump: 2024-09-16T03:52:58.741 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:52:58.741 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:52:58.741 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:52:58.742 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:52:58.742 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:52:58.742 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:52:58.742 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:52:58.742 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:52:58.742 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:52:58.742 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:52:58.742 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.724+0000 7f4a128f8640 10 monclient: shutdown 2024-09-16T03:52:58.742 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.724+0000 7f4a128f8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:52:58.742 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.724+0000 7f4a128f8640 1 -- 172.21.15.42:0/133700246 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4a0c09fe60 msgr2=0x7f4a0c0a0240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:52:58.742 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.724+0000 7f4a128f8640 1 --2- 172.21.15.42:0/133700246 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4a0c09fe60 0x7f4a0c0a0240 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f49fc009870 tx=0x7f49fc01c430 comp rx=0 tx=0).stop 2024-09-16T03:52:58.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.724+0000 7f4a128f8640 1 -- 172.21.15.42:0/133700246 shutdown_connections 2024-09-16T03:52:58.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.724+0000 7f4a128f8640 1 --2- 172.21.15.42:0/133700246 >> v2:172.21.15.42:3300/0 conn(0x7f4a0c0a1fb0 0x7f4a0c0a0780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:52:58.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.724+0000 7f4a128f8640 1 --2- 172.21.15.42:0/133700246 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4a0c09fe60 0x7f4a0c0a0240 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:52:58.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.724+0000 7f4a128f8640 1 -- 172.21.15.42:0/133700246 >> 172.21.15.42:0/133700246 conn(0x7f4a0c075ae0 msgr2=0x7f4a0c075ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:52:58.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.724+0000 7f4a128f8640 1 -- 172.21.15.42:0/133700246 shutdown_connections 2024-09-16T03:52:58.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.724+0000 7f4a128f8640 1 -- 172.21.15.42:0/133700246 wait complete. 2024-09-16T03:52:58.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.724+0000 7f4a128f8640 10 monclient: build_initial_monmap 2024-09-16T03:52:58.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.724+0000 7f4a128f8640 10 monclient: monmap: 2024-09-16T03:52:58.743 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:52:58.743 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:52:58.743 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:52:58.743 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:52:58.743 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:52:58.743 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:52:58.744 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:52:58.744 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:52:58.744 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:52:58.744 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:52:58.744 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.724+0000 7f4a128f8640 1 librados: starting msgr at 2024-09-16T03:52:58.744 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.724+0000 7f4a128f8640 1 librados: starting objecter 2024-09-16T03:52:58.744 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.724+0000 7f4a128f8640 1 Processor -- start 2024-09-16T03:52:58.744 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.724+0000 7f4a128f8640 1 -- start start 2024-09-16T03:52:58.744 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.724+0000 7f4a128f8640 1 librados: setting wanted keys 2024-09-16T03:52:58.744 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.724+0000 7f4a128f8640 1 librados: calling monclient init 2024-09-16T03:52:58.744 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.724+0000 7f4a128f8640 10 monclient: init 2024-09-16T03:52:58.744 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.724+0000 7f4a128f8640 10 monclient: _reopen_session rank -1 2024-09-16T03:52:58.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.724+0000 7f4a128f8640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T03:52:58.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.724+0000 7f4a128f8640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4a0c09fe60 0x7f4a0c1cf5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:52:58.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.724+0000 7f4a128f8640 10 monclient(hunting): picked mon.noname-a con 0x7f4a0c09fe60 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:52:58.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.724+0000 7f4a128f8640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4a0c0a1fb0 0x7f4a0c1365e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:52:58.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.724+0000 7f4a128f8640 10 monclient(hunting): picked mon.noname-b con 0x7f4a0c0a1fb0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:52:58.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.724+0000 7f4a128f8640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4a0c137120 0x7f4a0c11d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:52:58.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.724+0000 7f4a128f8640 10 monclient(hunting): picked mon.noname-c con 0x7f4a0c137120 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:52:58.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.724+0000 7f4a128f8640 10 monclient(hunting): start opening mon connection 2024-09-16T03:52:58.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.724+0000 7f4a128f8640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f4a0c141f30 con 0x7f4a0c09fe60 2024-09-16T03:52:58.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.724+0000 7f4a128f8640 10 monclient(hunting): start opening mon connection 2024-09-16T03:52:58.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.724+0000 7f4a128f8640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f4a0c141db0 con 0x7f4a0c0a1fb0 2024-09-16T03:52:58.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.724+0000 7f4a128f8640 10 monclient(hunting): start opening mon connection 2024-09-16T03:52:58.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.724+0000 7f4a128f8640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f4a0c1420b0 con 0x7f4a0c137120 2024-09-16T03:52:58.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.724+0000 7f4a128f8640 10 monclient(hunting): _renew_subs 2024-09-16T03:52:58.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.724+0000 7f4a128f8640 10 monclient(hunting): authenticate will time out at 1625.698964s 2024-09-16T03:52:58.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.724+0000 7f4a10e95640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4a0c137120 0x7f4a0c11d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:52:58.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.724+0000 7f4a10e95640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4a0c137120 0x7f4a0c11d1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:43062/0 (socket says 172.21.15.42:43062) 2024-09-16T03:52:58.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.724+0000 7f4a10e95640 1 -- 172.21.15.42:0/1885298868 learned_addr learned my addr 172.21.15.42:0/1885298868 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:52:58.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.724+0000 7f4a10e95640 10 monclient(hunting): get_auth_request con 0x7f4a0c137120 auth_method 0 2024-09-16T03:52:58.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.724+0000 7f4a10e95640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:52:58.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.724+0000 7f4a10e95640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:52:58.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.724+0000 7f4a10e95640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:52:58.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.724+0000 7f4a0bfff640 1 --2- 172.21.15.42:0/1885298868 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4a0c09fe60 0x7f4a0c1cf5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:52:58.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.724+0000 7f4a0bfff640 10 monclient(hunting): get_auth_request con 0x7f4a0c09fe60 auth_method 0 2024-09-16T03:52:58.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.724+0000 7f4a0bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:52:58.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.724+0000 7f4a0bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:52:58.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.724+0000 7f4a0bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:52:58.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.724+0000 7f4a10e95640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:52:58.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.724+0000 7f4a10e95640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:52:58.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.724+0000 7f4a10e95640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:52:58.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.724+0000 7f4a0bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:52:58.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.724+0000 7f4a0bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:52:58.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.724+0000 7f4a0bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:52:58.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.724+0000 7f4a03fff640 1 --2- 172.21.15.42:0/1885298868 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4a0c0a1fb0 0x7f4a0c1365e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:52:58.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.724+0000 7f4a03fff640 10 monclient(hunting): get_auth_request con 0x7f4a0c0a1fb0 auth_method 0 2024-09-16T03:52:58.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.724+0000 7f4a03fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:52:58.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.724+0000 7f4a03fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:52:58.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.724+0000 7f4a03fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:52:58.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.724+0000 7f4a10e95640 10 monclient(hunting): handle_auth_done global_id 834158 payload 824 2024-09-16T03:52:58.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.724+0000 7f4a10e95640 1 -- 172.21.15.42:0/1885298868 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4a0c0a1fb0 msgr2=0x7f4a0c1365e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:52:58.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.724+0000 7f4a10e95640 1 --2- 172.21.15.42:0/1885298868 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4a0c0a1fb0 0x7f4a0c1365e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:52:58.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.724+0000 7f4a10e95640 1 -- 172.21.15.42:0/1885298868 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4a0c09fe60 msgr2=0x7f4a0c1cf5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:52:58.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.724+0000 7f4a10e95640 1 --2- 172.21.15.42:0/1885298868 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4a0c09fe60 0x7f4a0c1cf5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:52:58.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.724+0000 7f4a10e95640 10 monclient: _finish_hunting 0 2024-09-16T03:52:58.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.724+0000 7f4a10e95640 1 monclient: found mon.noname-c 2024-09-16T03:52:58.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.724+0000 7f4a10e95640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:52:58.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.724+0000 7f4a10e95640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:52:58.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.724+0000 7f4a10e95640 1 -- 172.21.15.42:0/1885298868 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a0c0a5fc0 con 0x7f4a0c137120 2024-09-16T03:52:58.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.724+0000 7f4a0bfff640 1 --2- 172.21.15.42:0/1885298868 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4a0c09fe60 0x7f4a0c1cf5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:52:58.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.724+0000 7f4a10e95640 1 --2- 172.21.15.42:0/1885298868 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4a0c137120 0x7f4a0c11d1b0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f4a0400c780 tx=0x7f4a0400cc40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:52:58.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.724+0000 7f4a097fa640 10 client.?.objecter ms_handle_connect 0x7f4a0c137120 2024-09-16T03:52:58.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.724+0000 7f4a097fa640 10 client.?.objecter resend_mon_ops 2024-09-16T03:52:58.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.724+0000 7f4a097fa640 1 -- 172.21.15.42:0/1885298868 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a04004b70 con 0x7f4a0c137120 2024-09-16T03:52:58.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.724+0000 7f4a097fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:52:58.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.724+0000 7f4a097fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:52:58.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.724+0000 7f4a097fa640 10 monclient: dump: 2024-09-16T03:52:58.749 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:52:58.749 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:52:58.749 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:52:58.749 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:52:58.749 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:52:58.749 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:52:58.749 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:52:58.749 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:52:58.749 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:52:58.749 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:52:58.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.724+0000 7f4a097fa640 10 monclient: _finish_auth 0 2024-09-16T03:52:58.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.724+0000 7f4a097fa640 10 monclient: _check_auth_tickets 2024-09-16T03:52:58.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.724+0000 7f4a097fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:52:58.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.724+0000 7f4a128f8640 5 monclient: authenticate success, global_id 834158 2024-09-16T03:52:58.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.724+0000 7f4a097fa640 1 -- 172.21.15.42:0/1885298868 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a04004d10 con 0x7f4a0c137120 2024-09-16T03:52:58.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.724+0000 7f4a097fa640 10 monclient: handle_config config(0 keys) 2024-09-16T03:52:58.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.724+0000 7f4a128f8640 10 monclient: _renew_subs 2024-09-16T03:52:58.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.724+0000 7f4a097fa640 1 -- 172.21.15.42:0/1885298868 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a0400aa90 con 0x7f4a0c137120 2024-09-16T03:52:58.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.724+0000 7f4a128f8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:52:58.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.724+0000 7f4a128f8640 1 -- 172.21.15.42:0/1885298868 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a0c071850 con 0x7f4a0c137120 2024-09-16T03:52:58.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.724+0000 7f4a097fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:52:58.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.724+0000 7f4a097fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:52:58.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.724+0000 7f4a097fa640 10 monclient: dump: 2024-09-16T03:52:58.750 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:52:58.750 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:52:58.751 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:52:58.751 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:52:58.751 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:52:58.751 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:52:58.751 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:52:58.751 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:52:58.751 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:52:58.751 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:52:58.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.724+0000 7f4a0a7fc640 4 set_mon_vals no callback set 2024-09-16T03:52:58.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.724+0000 7f4a128f8640 10 client.834158.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:52:58.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.724+0000 7f4a128f8640 10 monclient: _renew_subs 2024-09-16T03:52:58.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.724+0000 7f4a128f8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:52:58.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.724+0000 7f4a128f8640 1 -- 172.21.15.42:0/1885298868 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a0c120f70 con 0x7f4a0c137120 2024-09-16T03:52:58.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.724+0000 7f4a128f8640 1 librados: init done 2024-09-16T03:52:58.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.728+0000 7f4a128f8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:52:58.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.728+0000 7f4a128f8640 1 --2- 172.21.15.42:0/1885298868 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4a0c136b20 0x7f4a0c11d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:52:58.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.728+0000 7f4a128f8640 10 monclient(hunting): start opening mon connection 2024-09-16T03:52:58.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.728+0000 7f4a128f8640 1 -- 172.21.15.42:0/1885298868 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f4a0c137570 con 0x7f4a0c136b20 2024-09-16T03:52:58.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.728+0000 7f4a097fa640 1 -- 172.21.15.42:0/1885298868 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f4a0400ac30 con 0x7f4a0c137120 2024-09-16T03:52:58.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.728+0000 7f4a097fa640 1 --2- 172.21.15.42:0/1885298868 >> v2:172.21.15.154:6812/254845342 conn(0x7f49ec037810 0x7f49ec039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:52:58.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.728+0000 7f4a0bfff640 1 --2- 172.21.15.42:0/1885298868 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4a0c136b20 0x7f4a0c11d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:52:58.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.728+0000 7f4a097fa640 1 -- 172.21.15.42:0/1885298868 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(345..345 src has 286..345) ==== 5769+0+0 (secure 0 0 0) 0x7f4a04049170 con 0x7f4a0c137120 2024-09-16T03:52:58.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.728+0000 7f4a097fa640 10 client.834158.objecter ms_dispatch 0x7f4a0c000ba0 osd_map(345..345 src has 286..345) 2024-09-16T03:52:58.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.728+0000 7f4a097fa640 3 client.834158.objecter handle_osd_map got epochs [345,345] > 0 2024-09-16T03:52:58.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.728+0000 7f4a097fa640 3 client.834158.objecter handle_osd_map decoding full epoch 345 2024-09-16T03:52:58.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.728+0000 7f4a0bfff640 10 monclient: get_auth_request con 0x7f4a0c136b20 auth_method 0 2024-09-16T03:52:58.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.728+0000 7f4a0bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:52:58.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.728+0000 7f4a0bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:52:58.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.728+0000 7f4a0bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:52:58.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.728+0000 7f4a097fa640 20 client.834158.objecter dump_active .. 0 homeless 2024-09-16T03:52:58.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.728+0000 7f4a0bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:52:58.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.728+0000 7f4a0bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:52:58.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.728+0000 7f4a0bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:52:58.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.728+0000 7f4a03fff640 1 --2- 172.21.15.42:0/1885298868 >> v2:172.21.15.154:6812/254845342 conn(0x7f49ec037810 0x7f49ec039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:52:58.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.728+0000 7f4a03fff640 10 monclient: get_auth_request con 0x7f49ec037810 auth_method 0 2024-09-16T03:52:58.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.728+0000 7f4a0bfff640 10 monclient(hunting): handle_auth_done global_id 794497 payload 824 2024-09-16T03:52:58.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.728+0000 7f4a0bfff640 1 -- 172.21.15.42:0/1885298868 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4a0c071de0 con 0x7f4a0c136b20 2024-09-16T03:52:58.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.728+0000 7f4a097fa640 10 client.834158.objecter ms_handle_connect 0x7f4a0c136b20 2024-09-16T03:52:58.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.728+0000 7f4a097fa640 10 client.834158.objecter resend_mon_ops 2024-09-16T03:52:58.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.728+0000 7f4a0bfff640 1 --2- 172.21.15.42:0/1885298868 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4a0c136b20 0x7f4a0c11d6f0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f49fc0ab000 tx=0x7f49fc01ac70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:52:58.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.728+0000 7f4a097fa640 1 -- 172.21.15.42:0/1885298868 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49fc0b0050 con 0x7f4a0c136b20 2024-09-16T03:52:58.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.728+0000 7f4a097fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:52:58.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.728+0000 7f4a097fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:52:58.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.728+0000 7f4a03fff640 1 --2- 172.21.15.42:0/1885298868 >> v2:172.21.15.154:6812/254845342 conn(0x7f49ec037810 0x7f49ec039cd0 secure :-1 s=READY pgs=3589 cs=0 l=1 rev1=1 crypto rx=0x7f4a0c11e980 tx=0x7f49f4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:52:58.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.728+0000 7f4a097fa640 10 monclient: dump: 2024-09-16T03:52:58.754 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:52:58.754 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:52:58.754 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:52:58.754 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:52:58.755 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:52:58.755 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:52:58.755 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:52:58.755 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:52:58.755 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:52:58.755 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:52:58.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.728+0000 7f4a097fa640 10 client.834158.objecter ms_handle_connect 0x7f49ec037810 2024-09-16T03:52:58.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.728+0000 7f4a097fa640 1 -- 172.21.15.42:0/1885298868 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f49fc0b1040 con 0x7f4a0c136b20 2024-09-16T03:52:58.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.728+0000 7f4a097fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:52:58.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.728+0000 7f4a097fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:52:58.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.728+0000 7f4a097fa640 1 -- 172.21.15.42:0/1885298868 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4a0c136b20 msgr2=0x7f4a0c11d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:52:58.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.728+0000 7f4a097fa640 1 --2- 172.21.15.42:0/1885298868 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4a0c136b20 0x7f4a0c11d6f0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f49fc0ab000 tx=0x7f49fc01ac70 comp rx=0 tx=0).stop 2024-09-16T03:52:58.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.728+0000 7f4a097fa640 1 -- 172.21.15.42:0/1885298868 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4a0c136b20 msgr2=0x7f4a0c11d6f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:52:58.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.728+0000 7f4a097fa640 1 --2- 172.21.15.42:0/1885298868 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4a0c136b20 0x7f4a0c11d6f0 secure :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f49fc0ab000 tx=0x7f49fc01ac70 comp rx=0 tx=0).stop 2024-09-16T03:52:58.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.736+0000 7f4a128f8640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:52:58.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.736+0000 7f4a128f8640 1 --2- 172.21.15.42:0/1885298868 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4a0c11dc30 0x7f4a0c13a560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:52:58.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.736+0000 7f4a128f8640 10 monclient(hunting): start opening mon connection 2024-09-16T03:52:58.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.736+0000 7f4a128f8640 1 -- 172.21.15.42:0/1885298868 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f4a0c11e040 con 0x7f4a0c11dc30 2024-09-16T03:52:58.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.736+0000 7f4a0bfff640 1 --2- 172.21.15.42:0/1885298868 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4a0c11dc30 0x7f4a0c13a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:52:58.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.736+0000 7f4a0bfff640 10 monclient: get_auth_request con 0x7f4a0c11dc30 auth_method 0 2024-09-16T03:52:58.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.736+0000 7f4a0bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:52:58.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.736+0000 7f4a0bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:52:58.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.736+0000 7f4a0bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:52:58.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.736+0000 7f4a0bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:52:58.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.736+0000 7f4a0bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:52:58.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.736+0000 7f4a0bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:52:58.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.736+0000 7f4a0bfff640 10 monclient(hunting): handle_auth_done global_id 794500 payload 824 2024-09-16T03:52:58.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.736+0000 7f4a0bfff640 1 -- 172.21.15.42:0/1885298868 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f4a0c11fa00 con 0x7f4a0c11dc30 2024-09-16T03:52:58.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.736+0000 7f4a097fa640 10 client.834158.objecter ms_handle_connect 0x7f4a0c11dc30 2024-09-16T03:52:58.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.736+0000 7f4a097fa640 10 client.834158.objecter resend_mon_ops 2024-09-16T03:52:58.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.736+0000 7f4a0bfff640 1 --2- 172.21.15.42:0/1885298868 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4a0c11dc30 0x7f4a0c13a560 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f49fc009870 tx=0x7f49fc0191d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:52:58.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.736+0000 7f4a097fa640 1 -- 172.21.15.42:0/1885298868 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49fc003820 con 0x7f4a0c11dc30 2024-09-16T03:52:58.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.736+0000 7f4a097fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:52:58.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.736+0000 7f4a097fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:52:58.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.736+0000 7f4a097fa640 10 monclient: dump: 2024-09-16T03:52:58.759 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:52:58.759 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:52:58.759 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:52:58.759 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:52:58.759 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:52:58.759 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:52:58.759 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:52:58.760 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:52:58.760 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:52:58.760 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:52:58.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.736+0000 7f4a097fa640 1 -- 172.21.15.42:0/1885298868 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4712 (secure 0 0 0) 0x7f49fc003820 con 0x7f4a0c11dc30 2024-09-16T03:52:58.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.736+0000 7f4a097fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:52:58.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.736+0000 7f4a097fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:52:58.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.736+0000 7f4a097fa640 1 -- 172.21.15.42:0/1885298868 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4a0c11dc30 msgr2=0x7f4a0c13a560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:52:58.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.736+0000 7f4a097fa640 1 --2- 172.21.15.42:0/1885298868 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4a0c11dc30 0x7f4a0c13a560 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f49fc009870 tx=0x7f49fc0191d0 comp rx=0 tx=0).stop 2024-09-16T03:52:58.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.736+0000 7f4a097fa640 1 -- 172.21.15.42:0/1885298868 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4a0c11dc30 msgr2=0x7f4a0c13a560 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:52:58.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.736+0000 7f4a097fa640 1 --2- 172.21.15.42:0/1885298868 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4a0c11dc30 0x7f4a0c13a560 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:52:58.760 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:52:58.761 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:52:58.761 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:52:58.761 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T03:52:58.761 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 94, 2024-09-16T03:52:58.761 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 220580, 2024-09-16T03:52:58.761 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:52:58.761 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:52:58.761 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T03:52:58.761 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T03:52:58.761 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:52:58.761 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 3, 2024-09-16T03:52:58.761 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:52:58.761 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:52:58.762 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:52:58.762 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:52:58.762 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:52:58.762 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:52:58.762 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:52:58.762 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:52:58.762 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:52:58.762 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:52:58.762 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:52:58.762 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:52:58.762 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:52:58.762 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:52:58.762 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:52:58.762 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:52:58.763 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:52:58.763 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:52:58.763 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:52:58.763 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:52:58.763 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:52:58.763 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:52:58.763 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:52:58.763 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:52:58.763 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:52:58.763 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:52:58.763 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:52:58.763 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:52:58.763 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:52:58.764 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:52:58.764 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:52:58.764 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:52:58.764 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:52:58.764 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:52:58.764 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:52:58.764 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:52:58.764 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:52:58.764 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:52:58.764 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:52:58.764 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:52:58.764 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:52:58.764 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:52:58.765 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:52:58.765 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:52:58.765 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:52:58.765 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:52:58.765 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:52:58.765 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:52:58.765 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:52:58.765 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:52:58.765 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:52:58.765 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:52:58.765 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:52:58.765 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:52:58.765 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:52:58.766 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:52:58.766 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:52:58.766 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:52:58.766 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:52:58.766 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:52:58.766 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:52:58.766 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:52:58.766 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:52:58.766 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:52:58.766 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:52:58.766 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:52:58.766 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:52:58.766 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:52:58.767 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:52:58.767 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:52:58.767 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:52:58.767 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:52:58.767 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:52:58.767 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:52:58.767 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:52:58.767 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:52:58.767 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:52:58.767 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:52:58.767 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:52:58.767 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:52:58.768 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:52:58.768 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:52:58.768 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:52:58.768 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:52:58.768 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:52:58.768 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:52:58.768 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:52:58.768 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:52:58.768 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:52:58.768 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:52:58.768 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:52:58.768 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:52:58.769 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:52:58.769 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:52:58.769 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:52:58.769 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:52:58.769 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:52:58.769 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:52:58.769 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:52:58.769 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:52:58.769 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:52:58.769 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:52:58.769 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:52:58.769 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:52:58.770 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:52:58.770 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:52:58.770 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:52:58.770 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:52:58.770 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:52:58.770 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:52:58.770 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:52:58.770 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:52:58.770 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:52:58.770 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:52:58.770 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:52:58.770 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:52:58.770 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:52:58.771 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:52:58.771 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:52:58.771 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:52:58.771 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:52:58.771 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:52:58.771 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:52:58.771 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:52:58.771 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:52:58.771 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:52:58.771 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:52:58.771 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:52:58.771 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:52:58.771 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:52:58.771 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:52:58.772 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:52:58.772 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:52:58.772 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:52:58.772 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:52:58.772 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:52:58.772 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:52:58.772 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:52:58.772 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:52:58.772 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:52:58.773 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:52:58.773 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T03:52:58.773 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:52:58.773 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:52:58.773 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:52:58.773 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:52:58.773 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:52:58.773 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:52:58.773 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:52:58.773 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:52:58.773 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:52:58.773 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:52:58.773 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 2 2024-09-16T03:52:58.774 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:52:58.774 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:52:58.774 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:52:58.774 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:52:58.774 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:52:58.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.736+0000 7f4a128f8640 10 librados: watch_flush enter 2024-09-16T03:52:58.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.736+0000 7f4a128f8640 10 librados: watch_flush exit 2024-09-16T03:52:58.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.736+0000 7f4a128f8640 20 client.834158.objecter shutdown clearing up homeless session... 2024-09-16T03:52:58.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.736+0000 7f4a128f8640 10 client.834158.objecter successfully canceled tick 2024-09-16T03:52:58.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.736+0000 7f4a128f8640 1 -- 172.21.15.42:0/1885298868 >> v2:172.21.15.154:6812/254845342 conn(0x7f49ec037810 msgr2=0x7f49ec039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:52:58.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.736+0000 7f4a128f8640 1 --2- 172.21.15.42:0/1885298868 >> v2:172.21.15.154:6812/254845342 conn(0x7f49ec037810 0x7f49ec039cd0 secure :-1 s=READY pgs=3589 cs=0 l=1 rev1=1 crypto rx=0x7f4a0c11e980 tx=0x7f49f4008040 comp rx=0 tx=0).stop 2024-09-16T03:52:58.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.736+0000 7f4a128f8640 10 monclient: shutdown 2024-09-16T03:52:58.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.736+0000 7f4a128f8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:52:58.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.736+0000 7f4a128f8640 1 -- 172.21.15.42:0/1885298868 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4a0c137120 msgr2=0x7f4a0c11d1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:52:58.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.736+0000 7f4a10e95640 1 -- 172.21.15.42:0/1885298868 reap_dead start 2024-09-16T03:52:58.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.736+0000 7f4a128f8640 1 --2- 172.21.15.42:0/1885298868 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4a0c137120 0x7f4a0c11d1b0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f4a0400c780 tx=0x7f4a0400cc40 comp rx=0 tx=0).stop 2024-09-16T03:52:58.777 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.736+0000 7f4a128f8640 1 -- 172.21.15.42:0/1885298868 shutdown_connections 2024-09-16T03:52:58.777 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.736+0000 7f4a128f8640 1 --2- 172.21.15.42:0/1885298868 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4a0c137120 0x7f4a0c11d1b0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:52:58.777 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.736+0000 7f4a128f8640 1 -- 172.21.15.42:0/1885298868 >> 172.21.15.42:0/1885298868 conn(0x7f4a0c075ae0 msgr2=0x7f4a0c0702e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:52:58.777 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.736+0000 7f4a128f8640 1 -- 172.21.15.42:0/1885298868 shutdown_connections 2024-09-16T03:52:58.777 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.736+0000 7f4a128f8640 1 -- 172.21.15.42:0/1885298868 wait complete. 2024-09-16T03:52:58.777 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.736+0000 7f4a128f8640 1 librados: shutdown 2024-09-16T03:52:58.777 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:52:58.777 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-09-16T03:52:58.778 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-16T03:52:58.778 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-09-16T03:52:58.778 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-16T03:52:58.778 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-16T03:52:58.778 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T03:52:58.803 INFO:tasks.ceph.mon.b:Stopped 2024-09-16T03:52:58.803 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T03:52:58.803 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:52:58.904 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~0s 2024-09-16T03:52:58.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.908+0000 7fa6529d6640 10 monclient: get_monmap_and_config 2024-09-16T03:52:58.916 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.908+0000 7fa6529d6640 10 monclient: build_initial_monmap 2024-09-16T03:52:58.916 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.908+0000 7fa6529d6640 10 monclient: monmap: 2024-09-16T03:52:58.916 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:52:58.916 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:52:58.916 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:52:58.914796+0000 2024-09-16T03:52:58.916 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:52:58.914796+0000 2024-09-16T03:52:58.916 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:52:58.916 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:52:58.916 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:52:58.916 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:52:58.916 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:52:58.916 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:52:58.917 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:52:58.917 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:52:58.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.908+0000 7fa6529d6640 1 Processor -- start 2024-09-16T03:52:58.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.908+0000 7fa6529d6640 1 -- start start 2024-09-16T03:52:58.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.908+0000 7fa6529d6640 10 monclient: init 2024-09-16T03:52:58.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.912+0000 7fa6529d6640 10 monclient: _reopen_session rank -1 2024-09-16T03:52:58.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.912+0000 7fa6529d6640 10 monclient: _add_conns ranks=[0,3,4,1,2] 2024-09-16T03:52:58.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.912+0000 7fa6529d6640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa64c0a39b0 addr v1:172.21.15.42:6789/0 2024-09-16T03:52:58.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.912+0000 7fa6529d6640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fa64c0a3d90 0x7fa64c13c380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:52:58.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.912+0000 7fa6529d6640 10 monclient(hunting): picked mon.noname-a con 0x7fa64c0a3d90 addr v2:172.21.15.42:3300/0 2024-09-16T03:52:58.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.912+0000 7fa6529d6640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fa64c13c8c0 0x7fa64c13ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:52:58.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.912+0000 7fa6529d6640 10 monclient(hunting): picked mon.noname-c con 0x7fa64c13c8c0 addr v2:172.21.15.154:3300/0 2024-09-16T03:52:58.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.912+0000 7fa6529d6640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa64c134290 con 0x7fa64c0a39b0 2024-09-16T03:52:58.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.912+0000 7fa6529d6640 10 monclient(hunting): start opening mon connection 2024-09-16T03:52:58.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.912+0000 7fa6529d6640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7fa64c09c140 con 0x7fa64c0a3d90 2024-09-16T03:52:58.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.912+0000 7fa6529d6640 10 monclient(hunting): start opening mon connection 2024-09-16T03:52:58.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.912+0000 7fa6529d6640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fa64c09c420 con 0x7fa64c13c8c0 2024-09-16T03:52:58.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.912+0000 7fa6529d6640 10 monclient(hunting): _renew_subs 2024-09-16T03:52:58.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.912+0000 7fa6529d6640 10 monclient(hunting): authenticate will time out at 1625.884190s 2024-09-16T03:52:58.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.912+0000 7fa64b7fe640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fa64c0a3d90 0x7fa64c13c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:52:58.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.912+0000 7fa64bfff640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fa64c0a39b0 0x7fa64c133b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:41220/0 (socket says 172.21.15.42:41220) 2024-09-16T03:52:58.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.912+0000 7fa64bfff640 1 -- 172.21.15.42:0/4040392282 learned_addr learned my addr 172.21.15.42:0/4040392282 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:52:58.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.912+0000 7fa64b7fe640 10 monclient(hunting): get_auth_request con 0x7fa64c0a3d90 auth_method 0 2024-09-16T03:52:58.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.912+0000 7fa64b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:52:58.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.912+0000 7fa64b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:52:58.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.912+0000 7fa650f73640 1 -- 172.21.15.42:0/4040392282 >> v2:172.21.15.154:3300/0 conn(0x7fa64c13c8c0 msgr2=0x7fa64c13ecb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T03:52:58.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.912+0000 7fa64b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:52:58.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.912+0000 7fa650f73640 1 --2- 172.21.15.42:0/4040392282 >> v2:172.21.15.154:3300/0 conn(0x7fa64c13c8c0 0x7fa64c13ecb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:52:58.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.912+0000 7fa64b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:52:58.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.912+0000 7fa64b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:52:58.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.912+0000 7fa64b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:52:58.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.912+0000 7fa64b7fe640 10 monclient(hunting): handle_auth_done global_id 824262 payload 274 2024-09-16T03:52:58.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.912+0000 7fa64b7fe640 1 -- 172.21.15.42:0/4040392282 >> v2:172.21.15.154:3300/0 conn(0x7fa64c13c8c0 msgr2=0x7fa64c13ecb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:52:58.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.912+0000 7fa64b7fe640 1 --2- 172.21.15.42:0/4040392282 >> v2:172.21.15.154:3300/0 conn(0x7fa64c13c8c0 0x7fa64c13ecb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:52:58.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.912+0000 7fa64b7fe640 1 -- 172.21.15.42:0/4040392282 >> v1:172.21.15.42:6789/0 conn(0x7fa64c0a39b0 legacy=0x7fa64c133b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:52:58.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.912+0000 7fa64b7fe640 10 monclient: _finish_hunting 0 2024-09-16T03:52:58.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.912+0000 7fa64b7fe640 1 monclient: found mon.noname-a 2024-09-16T03:52:58.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.912+0000 7fa64b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:52:58.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.912+0000 7fa64b7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:52:58.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.912+0000 7fa64b7fe640 1 -- 172.21.15.42:0/4040392282 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa64c06dd60 con 0x7fa64c0a3d90 2024-09-16T03:52:58.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.912+0000 7fa64affd640 1 -- 172.21.15.42:0/4040392282 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2588184112 0 0) 0x7fa64c134290 con 0x7fa64c0a39b0 2024-09-16T03:52:58.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.912+0000 7fa64affd640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T03:52:58.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.912+0000 7fa64b7fe640 1 --2- 172.21.15.42:0/4040392282 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa64c0a3d90 0x7fa64c13c380 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fa64c06b540 tx=0x7fa64001c360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b2168625013a5fd server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:52:58.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.912+0000 7fa64affd640 1 -- 172.21.15.42:0/4040392282 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6400b0070 con 0x7fa64c0a3d90 2024-09-16T03:52:58.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.912+0000 7fa64affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:52:58.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.912+0000 7fa64affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:52:58.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.912+0000 7fa64affd640 10 monclient: dump: 2024-09-16T03:52:58.927 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:52:58.927 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:52:58.927 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:52:58.927 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:52:58.927 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:52:58.927 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:52:58.927 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:52:58.927 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:52:58.927 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:52:58.927 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:52:58.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.912+0000 7fa64affd640 10 monclient: _finish_auth 0 2024-09-16T03:52:58.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.912+0000 7fa64affd640 10 monclient: _check_auth_tickets 2024-09-16T03:52:58.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.912+0000 7fa64affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:52:58.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.912+0000 7fa64affd640 1 -- 172.21.15.42:0/4040392282 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa6400ab600 con 0x7fa64c0a3d90 2024-09-16T03:52:58.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.912+0000 7fa64affd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:52:58.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.912+0000 7fa64affd640 1 -- 172.21.15.42:0/4040392282 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6400ab9d0 con 0x7fa64c0a3d90 2024-09-16T03:52:58.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.912+0000 7fa6529d6640 5 monclient: authenticate success, global_id 824262 2024-09-16T03:52:58.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.912+0000 7fa6529d6640 10 monclient: get_monmap_and_config success 2024-09-16T03:52:58.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.912+0000 7fa6529d6640 4 set_mon_vals no callback set 2024-09-16T03:52:58.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.912+0000 7fa64affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:52:58.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.912+0000 7fa64affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:52:58.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.912+0000 7fa64affd640 10 monclient: dump: 2024-09-16T03:52:58.928 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:52:58.928 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:52:58.928 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:52:58.928 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:52:58.929 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:52:58.929 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:52:58.929 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:52:58.929 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:52:58.929 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:52:58.929 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:52:58.929 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.912+0000 7fa6529d6640 10 monclient: shutdown 2024-09-16T03:52:58.929 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.912+0000 7fa6529d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:52:58.929 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.912+0000 7fa6529d6640 1 -- 172.21.15.42:0/4040392282 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa64c0a3d90 msgr2=0x7fa64c13c380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:52:58.929 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.912+0000 7fa6529d6640 1 --2- 172.21.15.42:0/4040392282 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa64c0a3d90 0x7fa64c13c380 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fa64c06b540 tx=0x7fa64001c360 comp rx=0 tx=0).stop 2024-09-16T03:52:58.929 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.912+0000 7fa6529d6640 1 -- 172.21.15.42:0/4040392282 shutdown_connections 2024-09-16T03:52:58.929 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.912+0000 7fa6529d6640 1 --2- 172.21.15.42:0/4040392282 >> v2:172.21.15.154:3300/0 conn(0x7fa64c13c8c0 0x7fa64c13ecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:52:58.929 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.912+0000 7fa6529d6640 1 --2- 172.21.15.42:0/4040392282 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa64c0a3d90 0x7fa64c13c380 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:52:58.929 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.912+0000 7fa6529d6640 1 -- 172.21.15.42:0/4040392282 >> 172.21.15.42:0/4040392282 conn(0x7fa64c0a9fb0 msgr2=0x7fa64c0aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:52:58.930 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.912+0000 7fa6529d6640 1 -- 172.21.15.42:0/4040392282 shutdown_connections 2024-09-16T03:52:58.930 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.912+0000 7fa6529d6640 1 -- 172.21.15.42:0/4040392282 wait complete. 2024-09-16T03:52:58.930 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.916+0000 7fa6529d6640 10 monclient: build_initial_monmap 2024-09-16T03:52:58.930 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.916+0000 7fa6529d6640 10 monclient: monmap: 2024-09-16T03:52:58.930 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:52:58.930 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:52:58.930 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:52:58.930 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:52:58.930 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:52:58.930 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:52:58.930 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:52:58.930 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:52:58.930 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:52:58.930 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:52:58.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.916+0000 7fa6529d6640 1 librados: starting msgr at 2024-09-16T03:52:58.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.916+0000 7fa6529d6640 1 librados: starting objecter 2024-09-16T03:52:58.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.916+0000 7fa6529d6640 1 Processor -- start 2024-09-16T03:52:58.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.916+0000 7fa6529d6640 1 -- start start 2024-09-16T03:52:58.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.916+0000 7fa6529d6640 1 librados: setting wanted keys 2024-09-16T03:52:58.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.916+0000 7fa6529d6640 1 librados: calling monclient init 2024-09-16T03:52:58.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.916+0000 7fa6529d6640 10 monclient: init 2024-09-16T03:52:58.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.916+0000 7fa6529d6640 10 monclient: _reopen_session rank -1 2024-09-16T03:52:58.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.916+0000 7fa6529d6640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:52:58.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.916+0000 7fa6529d6640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa64c0a39b0 0x7fa64c138130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:52:58.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.916+0000 7fa6529d6640 10 monclient(hunting): picked mon.noname-a con 0x7fa64c0a39b0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:52:58.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.916+0000 7fa6529d6640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa64c0a3d90 0x7fa64c138670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:52:58.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.916+0000 7fa6529d6640 10 monclient(hunting): picked mon.noname-c con 0x7fa64c0a3d90 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:52:58.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.916+0000 7fa6529d6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa64c13c8c0 0x7fa64c138bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:52:58.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.916+0000 7fa6529d6640 10 monclient(hunting): picked mon.noname-b con 0x7fa64c13c8c0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:52:58.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.916+0000 7fa6529d6640 10 monclient(hunting): start opening mon connection 2024-09-16T03:52:58.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.916+0000 7fa6529d6640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fa64c141da0 con 0x7fa64c0a39b0 2024-09-16T03:52:58.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.916+0000 7fa6529d6640 10 monclient(hunting): start opening mon connection 2024-09-16T03:52:58.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.916+0000 7fa6529d6640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa64c141c20 con 0x7fa64c13c8c0 2024-09-16T03:52:58.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.916+0000 7fa6529d6640 10 monclient(hunting): start opening mon connection 2024-09-16T03:52:58.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.916+0000 7fa6529d6640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fa64c141f20 con 0x7fa64c0a3d90 2024-09-16T03:52:58.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.916+0000 7fa6529d6640 10 monclient(hunting): _renew_subs 2024-09-16T03:52:58.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.916+0000 7fa6529d6640 10 monclient(hunting): authenticate will time out at 1625.889295s 2024-09-16T03:52:58.933 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.916+0000 7fa64b7fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa64c0a3d90 0x7fa64c138670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:52:58.933 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.916+0000 7fa64bfff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa64c0a39b0 0x7fa64c138130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:52:58.933 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.916+0000 7fa64b7fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa64c0a3d90 0x7fa64c138670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:43072/0 (socket says 172.21.15.42:43072) 2024-09-16T03:52:58.933 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.916+0000 7fa64b7fe640 1 -- 172.21.15.42:0/2126208478 learned_addr learned my addr 172.21.15.42:0/2126208478 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:52:58.933 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.916+0000 7fa64bfff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa64c0a39b0 0x7fa64c138130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:38640/0 (socket says 172.21.15.42:38640) 2024-09-16T03:52:58.933 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.916+0000 7fa650f73640 1 -- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa64c13c8c0 msgr2=0x7fa64c138bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T03:52:58.933 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.916+0000 7fa64b7fe640 10 monclient(hunting): get_auth_request con 0x7fa64c0a3d90 auth_method 0 2024-09-16T03:52:58.933 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.916+0000 7fa64b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:52:58.933 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.916+0000 7fa64b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:52:58.933 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.916+0000 7fa64b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:52:58.933 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.916+0000 7fa64bfff640 10 monclient(hunting): get_auth_request con 0x7fa64c0a39b0 auth_method 0 2024-09-16T03:52:58.933 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.916+0000 7fa64bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:52:58.933 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.916+0000 7fa64bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:52:58.934 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.916+0000 7fa64bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:52:58.934 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.916+0000 7fa650f73640 1 --2- 172.21.15.42:0/2126208478 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa64c13c8c0 0x7fa64c138bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:52:58.934 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.916+0000 7fa64b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:52:58.934 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.916+0000 7fa64b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:52:58.934 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.916+0000 7fa64b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:52:58.934 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.916+0000 7fa64bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:52:58.934 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.916+0000 7fa64bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:52:58.934 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.916+0000 7fa64bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:52:58.934 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.916+0000 7fa64bfff640 10 monclient(hunting): handle_auth_done global_id 824268 payload 824 2024-09-16T03:52:58.934 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.916+0000 7fa64bfff640 1 -- 172.21.15.42:0/2126208478 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa64c0a3d90 msgr2=0x7fa64c138670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:52:58.934 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.916+0000 7fa64bfff640 1 --2- 172.21.15.42:0/2126208478 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa64c0a3d90 0x7fa64c138670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:52:58.934 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.916+0000 7fa64bfff640 1 -- 172.21.15.42:0/2126208478 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa64c13c8c0 msgr2=0x7fa64c138bb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:52:58.934 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.916+0000 7fa64bfff640 1 --2- 172.21.15.42:0/2126208478 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa64c13c8c0 0x7fa64c138bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:52:58.934 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.916+0000 7fa64bfff640 10 monclient: _finish_hunting 0 2024-09-16T03:52:58.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.916+0000 7fa64bfff640 1 monclient: found mon.noname-a 2024-09-16T03:52:58.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.916+0000 7fa64bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:52:58.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.916+0000 7fa64bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:52:58.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.916+0000 7fa64bfff640 1 -- 172.21.15.42:0/2126208478 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa64c13ee50 con 0x7fa64c0a39b0 2024-09-16T03:52:58.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.916+0000 7fa64b7fe640 1 --2- 172.21.15.42:0/2126208478 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa64c0a3d90 0x7fa64c138670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:52:58.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.916+0000 7fa64bfff640 1 --2- 172.21.15.42:0/2126208478 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa64c0a39b0 0x7fa64c138130 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fa63c00c460 tx=0x7fa63c00c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:52:58.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.916+0000 7fa648ff9640 10 client.?.objecter ms_handle_connect 0x7fa64c0a39b0 2024-09-16T03:52:58.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.916+0000 7fa648ff9640 10 client.?.objecter resend_mon_ops 2024-09-16T03:52:58.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.916+0000 7fa648ff9640 1 -- 172.21.15.42:0/2126208478 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa63c018070 con 0x7fa64c0a39b0 2024-09-16T03:52:58.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.916+0000 7fa648ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:52:58.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.916+0000 7fa648ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:52:58.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.916+0000 7fa648ff9640 10 monclient: dump: 2024-09-16T03:52:58.935 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:52:58.935 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:52:58.936 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:52:58.936 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:52:58.936 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:52:58.936 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:52:58.936 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:52:58.936 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:52:58.936 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:52:58.936 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:52:58.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.916+0000 7fa648ff9640 10 monclient: _finish_auth 0 2024-09-16T03:52:58.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.916+0000 7fa648ff9640 10 monclient: _check_auth_tickets 2024-09-16T03:52:58.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.916+0000 7fa648ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:52:58.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.916+0000 7fa648ff9640 1 -- 172.21.15.42:0/2126208478 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa63c007500 con 0x7fa64c0a39b0 2024-09-16T03:52:58.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.916+0000 7fa648ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T03:52:58.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.916+0000 7fa648ff9640 1 -- 172.21.15.42:0/2126208478 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa63c004a30 con 0x7fa64c0a39b0 2024-09-16T03:52:58.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.916+0000 7fa648ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:52:58.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.916+0000 7fa648ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:52:58.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.916+0000 7fa648ff9640 10 monclient: dump: 2024-09-16T03:52:58.937 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:52:58.937 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:52:58.937 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:52:58.937 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:52:58.937 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:52:58.938 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:52:58.938 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:52:58.938 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:52:58.939 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:52:58.939 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:52:58.939 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.916+0000 7fa64a7fc640 4 set_mon_vals no callback set 2024-09-16T03:52:58.939 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.916+0000 7fa6529d6640 5 monclient: authenticate success, global_id 824268 2024-09-16T03:52:58.939 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.916+0000 7fa6529d6640 10 monclient: _renew_subs 2024-09-16T03:52:58.939 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.916+0000 7fa6529d6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:52:58.939 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.916+0000 7fa6529d6640 1 -- 172.21.15.42:0/2126208478 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa64c13c320 con 0x7fa64c0a39b0 2024-09-16T03:52:58.939 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.916+0000 7fa6529d6640 10 client.824268.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:52:58.939 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.916+0000 7fa6529d6640 10 monclient: _renew_subs 2024-09-16T03:52:58.939 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.916+0000 7fa6529d6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:52:58.939 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.916+0000 7fa6529d6640 1 -- 172.21.15.42:0/2126208478 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa64c13a9c0 con 0x7fa64c0a39b0 2024-09-16T03:52:58.939 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.916+0000 7fa6529d6640 1 librados: init done 2024-09-16T03:52:58.939 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.916+0000 7fa6529d6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:52:58.940 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.916+0000 7fa6529d6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:52:58.940 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.916+0000 7fa6529d6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:52:58.940 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.916+0000 7fa6529d6640 1 -- 172.21.15.42:0/2126208478 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa64c133b80 con 0x7fa64c0a39b0 2024-09-16T03:52:58.940 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.916+0000 7fa648ff9640 1 -- 172.21.15.42:0/2126208478 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fa63c004bd0 con 0x7fa64c0a39b0 2024-09-16T03:52:58.940 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.916+0000 7fa648ff9640 1 --2- 172.21.15.42:0/2126208478 >> v2:172.21.15.154:6812/254845342 conn(0x7fa6380377c0 0x7fa638039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:52:58.940 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.916+0000 7fa648ff9640 1 -- 172.21.15.42:0/2126208478 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(345..345 src has 286..345) ==== 5769+0+0 (secure 0 0 0) 0x7fa63c0483d0 con 0x7fa64c0a39b0 2024-09-16T03:52:58.940 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.916+0000 7fa648ff9640 10 client.824268.objecter ms_dispatch 0x7fa64c000ba0 osd_map(345..345 src has 286..345) 2024-09-16T03:52:58.940 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.916+0000 7fa648ff9640 3 client.824268.objecter handle_osd_map got epochs [345,345] > 0 2024-09-16T03:52:58.940 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.916+0000 7fa648ff9640 3 client.824268.objecter handle_osd_map decoding full epoch 345 2024-09-16T03:52:58.940 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.920+0000 7fa648ff9640 20 client.824268.objecter dump_active .. 0 homeless 2024-09-16T03:52:58.940 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.920+0000 7fa64b7fe640 1 --2- 172.21.15.42:0/2126208478 >> v2:172.21.15.154:6812/254845342 conn(0x7fa6380377c0 0x7fa638039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:52:58.940 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.920+0000 7fa64b7fe640 10 monclient: get_auth_request con 0x7fa6380377c0 auth_method 0 2024-09-16T03:52:58.940 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.920+0000 7fa64b7fe640 1 --2- 172.21.15.42:0/2126208478 >> v2:172.21.15.154:6812/254845342 conn(0x7fa6380377c0 0x7fa638039c80 secure :-1 s=READY pgs=3590 cs=0 l=1 rev1=1 crypto rx=0x7fa64001c3a0 tx=0x7fa6400a6000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:52:58.941 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.920+0000 7fa648ff9640 10 client.824268.objecter ms_handle_connect 0x7fa6380377c0 2024-09-16T03:52:58.941 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.924+0000 7fa648ff9640 1 -- 172.21.15.42:0/2126208478 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7fa64c13ee50 con 0x7fa64c0a39b0 2024-09-16T03:52:58.941 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.924+0000 7fa648ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:52:58.941 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:58.924+0000 7fa648ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:52:59.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:59.140+0000 7fa6529d6640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:52:59.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:59.140+0000 7fa6529d6640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:52:59.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:59.140+0000 7fa6529d6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:52:59.148 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:59.140+0000 7fa6529d6640 1 -- 172.21.15.42:0/2126208478 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fa64c0a5750 con 0x7fa64c0a39b0 2024-09-16T03:52:59.148 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:59.140+0000 7fa648ff9640 1 -- 172.21.15.42:0/2126208478 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7fa63c011360 con 0x7fa64c0a39b0 2024-09-16T03:52:59.148 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:59.140+0000 7fa648ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:52:59.148 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:59.140+0000 7fa648ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:52:59.148 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":94,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:52:59.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:59.144+0000 7fa6529d6640 10 librados: watch_flush enter 2024-09-16T03:52:59.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:59.144+0000 7fa6529d6640 10 librados: watch_flush exit 2024-09-16T03:52:59.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:59.144+0000 7fa6529d6640 20 client.824268.objecter shutdown clearing up homeless session... 2024-09-16T03:52:59.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:59.144+0000 7fa6529d6640 10 client.824268.objecter successfully canceled tick 2024-09-16T03:52:59.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:59.144+0000 7fa6529d6640 1 -- 172.21.15.42:0/2126208478 >> v2:172.21.15.154:6812/254845342 conn(0x7fa6380377c0 msgr2=0x7fa638039c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:52:59.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:59.144+0000 7fa6529d6640 1 --2- 172.21.15.42:0/2126208478 >> v2:172.21.15.154:6812/254845342 conn(0x7fa6380377c0 0x7fa638039c80 secure :-1 s=READY pgs=3590 cs=0 l=1 rev1=1 crypto rx=0x7fa64001c3a0 tx=0x7fa6400a6000 comp rx=0 tx=0).stop 2024-09-16T03:52:59.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:59.144+0000 7fa6529d6640 10 monclient: shutdown 2024-09-16T03:52:59.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:59.144+0000 7fa6529d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:52:59.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:59.144+0000 7fa6529d6640 1 -- 172.21.15.42:0/2126208478 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa64c0a39b0 msgr2=0x7fa64c138130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:52:59.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:59.144+0000 7fa6529d6640 1 --2- 172.21.15.42:0/2126208478 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa64c0a39b0 0x7fa64c138130 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fa63c00c460 tx=0x7fa63c00c920 comp rx=0 tx=0).stop 2024-09-16T03:52:59.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:59.144+0000 7fa6529d6640 1 -- 172.21.15.42:0/2126208478 shutdown_connections 2024-09-16T03:52:59.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:59.144+0000 7fa6529d6640 1 --2- 172.21.15.42:0/2126208478 >> v2:172.21.15.154:6812/254845342 conn(0x7fa6380377c0 0x7fa638039c80 unknown :-1 s=CLOSED pgs=3590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:52:59.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:59.144+0000 7fa6529d6640 1 --2- 172.21.15.42:0/2126208478 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa64c13c8c0 0x7fa64c138bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:52:59.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:59.144+0000 7fa6529d6640 1 --2- 172.21.15.42:0/2126208478 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa64c0a3d90 0x7fa64c138670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:52:59.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:59.144+0000 7fa6529d6640 1 --2- 172.21.15.42:0/2126208478 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa64c0a39b0 0x7fa64c138130 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:52:59.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:59.144+0000 7fa6529d6640 1 -- 172.21.15.42:0/2126208478 >> 172.21.15.42:0/2126208478 conn(0x7fa64c0a9fb0 msgr2=0x7fa64c131980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:52:59.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:59.144+0000 7fa6529d6640 1 -- 172.21.15.42:0/2126208478 shutdown_connections 2024-09-16T03:52:59.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:59.144+0000 7fa6529d6640 1 -- 172.21.15.42:0/2126208478 wait complete. 2024-09-16T03:52:59.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:52:59.144+0000 7fa6529d6640 1 librados: shutdown 2024-09-16T03:53:02.166 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:53:02.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.292+0000 7fd734cdc640 10 monclient: get_monmap_and_config 2024-09-16T03:53:02.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.292+0000 7fd734cdc640 10 monclient: build_initial_monmap 2024-09-16T03:53:02.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.292+0000 7fd734cdc640 10 monclient: monmap: 2024-09-16T03:53:02.299 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:53:02.299 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:53:02.299 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:53:02.297715+0000 2024-09-16T03:53:02.299 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:53:02.297715+0000 2024-09-16T03:53:02.299 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:53:02.299 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:53:02.299 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:53:02.300 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:53:02.300 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:53:02.300 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:53:02.300 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:53:02.300 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:53:02.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.292+0000 7fd734cdc640 1 Processor -- start 2024-09-16T03:53:02.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.292+0000 7fd734cdc640 1 -- start start 2024-09-16T03:53:02.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.292+0000 7fd734cdc640 10 monclient: init 2024-09-16T03:53:02.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.292+0000 7fd734cdc640 10 monclient: _reopen_session rank -1 2024-09-16T03:53:02.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.292+0000 7fd734cdc640 10 monclient: _add_conns ranks=[2,1,0,4,3] 2024-09-16T03:53:02.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.292+0000 7fd734cdc640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd7300a1330 0x7fd7300a1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:53:02.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.292+0000 7fd734cdc640 10 monclient(hunting): picked mon.noname-b con 0x7fd7300a1330 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:53:02.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.292+0000 7fd734cdc640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd73009c920 addr v1:172.21.15.154:6789/0 2024-09-16T03:53:02.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.292+0000 7fd734cdc640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd730072d40 addr v1:172.21.15.42:6789/0 2024-09-16T03:53:02.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.292+0000 7fd734cdc640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd730137f40 con 0x7fd730072d40 2024-09-16T03:53:02.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.292+0000 7fd734cdc640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd730072350 con 0x7fd73009c920 2024-09-16T03:53:02.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.292+0000 7fd734cdc640 10 monclient(hunting): start opening mon connection 2024-09-16T03:53:02.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.292+0000 7fd734cdc640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fd730072b50 con 0x7fd7300a1330 2024-09-16T03:53:02.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.292+0000 7fd734cdc640 10 monclient(hunting): _renew_subs 2024-09-16T03:53:02.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.292+0000 7fd734cdc640 10 monclient(hunting): authenticate will time out at 1629.267235s 2024-09-16T03:53:02.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.292+0000 7fd72dd9b640 1 -- >> v1:172.21.15.154:6789/0 conn(0x7fd73009c920 legacy=0x7fd730137120 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.154:6789/0 2024-09-16T03:53:02.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.292+0000 7fd72e59c640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd7300a1330 0x7fd7300a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:53:02.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.292+0000 7fd72ed9d640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fd730072d40 0x7fd730137830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:41222/0 (socket says 172.21.15.42:41222) 2024-09-16T03:53:02.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.292+0000 7fd72ed9d640 1 -- 172.21.15.42:0/1662529010 learned_addr learned my addr 172.21.15.42:0/1662529010 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:53:02.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.292+0000 7fd72e59c640 10 monclient(hunting): get_auth_request con 0x7fd7300a1330 auth_method 0 2024-09-16T03:53:02.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.292+0000 7fd72e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:53:02.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.292+0000 7fd72e59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:53:02.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.292+0000 7fd72e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:53:02.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.292+0000 7fd72e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:53:02.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.292+0000 7fd72e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:53:02.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.292+0000 7fd72e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:53:02.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.292+0000 7fd72e59c640 10 monclient(hunting): handle_auth_done global_id 834164 payload 274 2024-09-16T03:53:02.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.292+0000 7fd72e59c640 1 -- 172.21.15.42:0/1662529010 >> v1:172.21.15.154:6789/0 conn(0x7fd73009c920 legacy=0x7fd730137120 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:53:02.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.292+0000 7fd72e59c640 1 -- 172.21.15.42:0/1662529010 >> v1:172.21.15.42:6789/0 conn(0x7fd730072d40 legacy=0x7fd730137830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:53:02.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.292+0000 7fd72e59c640 10 monclient: _finish_hunting 0 2024-09-16T03:53:02.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.292+0000 7fd72e59c640 1 monclient: found mon.noname-b 2024-09-16T03:53:02.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.292+0000 7fd72e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:53:02.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.292+0000 7fd72e59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T03:53:02.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.292+0000 7fd72e59c640 1 -- 172.21.15.42:0/1662529010 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd73009bc60 con 0x7fd7300a1330 2024-09-16T03:53:02.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.292+0000 7fd72d59a640 1 -- 172.21.15.42:0/1662529010 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4066492579 0 0) 0x7fd730137f40 con 0x7fd730072d40 2024-09-16T03:53:02.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.292+0000 7fd72d59a640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T03:53:02.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.296+0000 7fd72e59c640 1 --2- 172.21.15.42:0/1662529010 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd7300a1330 0x7fd7300a1710 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fd720009dc0 tx=0x7fd72001c8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=17329e221b8d7546 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:53:02.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.296+0000 7fd72d59a640 1 -- 172.21.15.42:0/1662529010 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7200b2070 con 0x7fd7300a1330 2024-09-16T03:53:02.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.296+0000 7fd72d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:53:02.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.296+0000 7fd72d59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:53:02.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.296+0000 7fd72d59a640 10 monclient: dump: 2024-09-16T03:53:02.312 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:53:02.312 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:53:02.312 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:02.312 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:02.312 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:53:02.312 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:53:02.312 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:53:02.312 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:53:02.312 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:53:02.312 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:53:02.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.296+0000 7fd72d59a640 10 monclient: _finish_auth 0 2024-09-16T03:53:02.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.296+0000 7fd72d59a640 10 monclient: _check_auth_tickets 2024-09-16T03:53:02.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.296+0000 7fd72d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:53:02.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.296+0000 7fd734cdc640 5 monclient: authenticate success, global_id 834164 2024-09-16T03:53:02.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.296+0000 7fd734cdc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:53:02.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.296+0000 7fd72d59a640 1 -- 172.21.15.42:0/1662529010 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7200a5980 con 0x7fd7300a1330 2024-09-16T03:53:02.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.296+0000 7fd72d59a640 10 monclient: handle_config config(0 keys) 2024-09-16T03:53:02.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.296+0000 7fd734cdc640 10 monclient: get_monmap_and_config success 2024-09-16T03:53:02.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.296+0000 7fd734cdc640 4 set_mon_vals no callback set 2024-09-16T03:53:02.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.296+0000 7fd72d59a640 1 -- 172.21.15.42:0/1662529010 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7200a5d50 con 0x7fd7300a1330 2024-09-16T03:53:02.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.296+0000 7fd72d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:53:02.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.296+0000 7fd72d59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:53:02.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.296+0000 7fd72d59a640 10 monclient: dump: 2024-09-16T03:53:02.313 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:53:02.313 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:53:02.314 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:02.314 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:02.314 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:53:02.314 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:53:02.314 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:53:02.314 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:53:02.314 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:53:02.314 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:53:02.314 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.296+0000 7fd734cdc640 10 monclient: shutdown 2024-09-16T03:53:02.314 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.296+0000 7fd734cdc640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:53:02.314 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.296+0000 7fd734cdc640 1 -- 172.21.15.42:0/1662529010 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd7300a1330 msgr2=0x7fd7300a1710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:53:02.314 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.296+0000 7fd734cdc640 1 --2- 172.21.15.42:0/1662529010 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd7300a1330 0x7fd7300a1710 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fd720009dc0 tx=0x7fd72001c8a0 comp rx=0 tx=0).stop 2024-09-16T03:53:02.314 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.296+0000 7fd734cdc640 1 -- 172.21.15.42:0/1662529010 shutdown_connections 2024-09-16T03:53:02.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.296+0000 7fd734cdc640 1 --2- 172.21.15.42:0/1662529010 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd7300a1330 0x7fd7300a1710 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:53:02.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.296+0000 7fd734cdc640 1 -- 172.21.15.42:0/1662529010 >> 172.21.15.42:0/1662529010 conn(0x7fd7300a2420 msgr2=0x7fd7300a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:53:02.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.296+0000 7fd734cdc640 1 -- 172.21.15.42:0/1662529010 shutdown_connections 2024-09-16T03:53:02.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.296+0000 7fd734cdc640 1 -- 172.21.15.42:0/1662529010 wait complete. 2024-09-16T03:53:02.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.296+0000 7fd734cdc640 10 monclient: build_initial_monmap 2024-09-16T03:53:02.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.296+0000 7fd734cdc640 10 monclient: monmap: 2024-09-16T03:53:02.315 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:53:02.315 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:53:02.315 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:53:02.315 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:53:02.315 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:53:02.315 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:53:02.315 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:53:02.315 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:53:02.316 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:53:02.316 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:53:02.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.296+0000 7fd734cdc640 1 librados: starting msgr at 2024-09-16T03:53:02.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.296+0000 7fd734cdc640 1 librados: starting objecter 2024-09-16T03:53:02.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.296+0000 7fd734cdc640 1 Processor -- start 2024-09-16T03:53:02.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.296+0000 7fd734cdc640 1 -- start start 2024-09-16T03:53:02.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.296+0000 7fd734cdc640 1 librados: setting wanted keys 2024-09-16T03:53:02.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.296+0000 7fd734cdc640 1 librados: calling monclient init 2024-09-16T03:53:02.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.296+0000 7fd734cdc640 10 monclient: init 2024-09-16T03:53:02.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.300+0000 7fd734cdc640 10 monclient: _reopen_session rank -1 2024-09-16T03:53:02.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.300+0000 7fd734cdc640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T03:53:02.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.300+0000 7fd734cdc640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd730072d40 0x7fd730136f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:53:02.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.300+0000 7fd734cdc640 10 monclient(hunting): picked mon.noname-b con 0x7fd730072d40 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:53:02.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.300+0000 7fd734cdc640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd73009c920 0x7fd730137440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:53:02.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.300+0000 7fd734cdc640 10 monclient(hunting): picked mon.noname-c con 0x7fd73009c920 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:53:02.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.300+0000 7fd734cdc640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd7300a1330 0x7fd73011caf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:53:02.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.300+0000 7fd734cdc640 10 monclient(hunting): picked mon.noname-a con 0x7fd7300a1330 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:53:02.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.300+0000 7fd734cdc640 10 monclient(hunting): start opening mon connection 2024-09-16T03:53:02.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.300+0000 7fd734cdc640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fd730143150 con 0x7fd7300a1330 2024-09-16T03:53:02.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.300+0000 7fd734cdc640 10 monclient(hunting): start opening mon connection 2024-09-16T03:53:02.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.300+0000 7fd734cdc640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd730142fd0 con 0x7fd730072d40 2024-09-16T03:53:02.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.300+0000 7fd734cdc640 10 monclient(hunting): start opening mon connection 2024-09-16T03:53:02.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.300+0000 7fd734cdc640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fd7301432d0 con 0x7fd73009c920 2024-09-16T03:53:02.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.300+0000 7fd734cdc640 10 monclient(hunting): _renew_subs 2024-09-16T03:53:02.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.300+0000 7fd734cdc640 10 monclient(hunting): authenticate will time out at 1629.272303s 2024-09-16T03:53:02.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.300+0000 7fd72dd9b640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd73009c920 0x7fd730137440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:53:02.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.300+0000 7fd72ed9d640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd7300a1330 0x7fd73011caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:53:02.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.300+0000 7fd72e59c640 1 -- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd730072d40 msgr2=0x7fd730136f00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T03:53:02.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.300+0000 7fd72dd9b640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd73009c920 0x7fd730137440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:43096/0 (socket says 172.21.15.42:43096) 2024-09-16T03:53:02.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.300+0000 7fd72dd9b640 1 -- 172.21.15.42:0/3229889061 learned_addr learned my addr 172.21.15.42:0/3229889061 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:53:02.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.300+0000 7fd72dd9b640 10 monclient(hunting): get_auth_request con 0x7fd73009c920 auth_method 0 2024-09-16T03:53:02.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.300+0000 7fd72ed9d640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd7300a1330 0x7fd73011caf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:38654/0 (socket says 172.21.15.42:38654) 2024-09-16T03:53:02.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.300+0000 7fd72dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:53:02.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.300+0000 7fd72dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:53:02.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.300+0000 7fd72dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:53:02.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.300+0000 7fd72e59c640 1 --2- 172.21.15.42:0/3229889061 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd730072d40 0x7fd730136f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:53:02.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.300+0000 7fd72ed9d640 10 monclient(hunting): get_auth_request con 0x7fd7300a1330 auth_method 0 2024-09-16T03:53:02.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.300+0000 7fd72ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:53:02.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.300+0000 7fd72ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:53:02.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.300+0000 7fd72ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:53:02.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.300+0000 7fd72dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:53:02.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.300+0000 7fd72dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:53:02.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.300+0000 7fd72dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:53:02.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.300+0000 7fd72ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:53:02.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.300+0000 7fd72ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:53:02.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.300+0000 7fd72ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:53:02.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.300+0000 7fd72dd9b640 10 monclient(hunting): handle_auth_done global_id 834167 payload 824 2024-09-16T03:53:02.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.300+0000 7fd72dd9b640 1 -- 172.21.15.42:0/3229889061 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd730072d40 msgr2=0x7fd730136f00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:53:02.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.300+0000 7fd72dd9b640 1 --2- 172.21.15.42:0/3229889061 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd730072d40 0x7fd730136f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:53:02.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.300+0000 7fd72dd9b640 1 -- 172.21.15.42:0/3229889061 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd7300a1330 msgr2=0x7fd73011caf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:53:02.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.300+0000 7fd72dd9b640 1 --2- 172.21.15.42:0/3229889061 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd7300a1330 0x7fd73011caf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:53:02.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.300+0000 7fd72dd9b640 10 monclient: _finish_hunting 0 2024-09-16T03:53:02.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.300+0000 7fd72dd9b640 1 monclient: found mon.noname-c 2024-09-16T03:53:02.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.300+0000 7fd72dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:53:02.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.300+0000 7fd72dd9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:53:02.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.300+0000 7fd72dd9b640 1 -- 172.21.15.42:0/3229889061 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7301207a0 con 0x7fd73009c920 2024-09-16T03:53:02.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.300+0000 7fd72dd9b640 1 --2- 172.21.15.42:0/3229889061 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd73009c920 0x7fd730137440 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fd71800b580 tx=0x7fd71800ba40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:53:02.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.300+0000 7fd726ffd640 10 client.?.objecter ms_handle_connect 0x7fd73009c920 2024-09-16T03:53:02.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.300+0000 7fd726ffd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:53:02.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.300+0000 7fd726ffd640 1 -- 172.21.15.42:0/3229889061 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd718016070 con 0x7fd73009c920 2024-09-16T03:53:02.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.300+0000 7fd726ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:53:02.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.300+0000 7fd726ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:53:02.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.300+0000 7fd726ffd640 10 monclient: dump: 2024-09-16T03:53:02.320 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:53:02.320 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:53:02.320 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:02.320 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:02.320 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:53:02.321 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:53:02.321 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:53:02.321 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:53:02.321 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:53:02.321 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:53:02.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.300+0000 7fd726ffd640 10 monclient: _finish_auth 0 2024-09-16T03:53:02.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.300+0000 7fd726ffd640 10 monclient: _check_auth_tickets 2024-09-16T03:53:02.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.300+0000 7fd726ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:53:02.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.300+0000 7fd726ffd640 1 -- 172.21.15.42:0/3229889061 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd718003d80 con 0x7fd73009c920 2024-09-16T03:53:02.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.300+0000 7fd726ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:53:02.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.300+0000 7fd726ffd640 1 -- 172.21.15.42:0/3229889061 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7180052f0 con 0x7fd73009c920 2024-09-16T03:53:02.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.300+0000 7fd726ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:53:02.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.300+0000 7fd726ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:53:02.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.300+0000 7fd726ffd640 10 monclient: dump: 2024-09-16T03:53:02.322 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:53:02.322 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:53:02.322 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:02.322 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:02.322 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:53:02.322 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:53:02.322 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:53:02.322 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:53:02.322 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:53:02.322 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:53:02.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.300+0000 7fd727fff640 4 set_mon_vals no callback set 2024-09-16T03:53:02.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.300+0000 7fd734cdc640 5 monclient: authenticate success, global_id 834167 2024-09-16T03:53:02.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.300+0000 7fd734cdc640 10 monclient: _renew_subs 2024-09-16T03:53:02.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.300+0000 7fd734cdc640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:53:02.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.300+0000 7fd734cdc640 1 -- 172.21.15.42:0/3229889061 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd730120a80 con 0x7fd73009c920 2024-09-16T03:53:02.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.300+0000 7fd734cdc640 10 client.834167.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:53:02.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.300+0000 7fd734cdc640 10 monclient: _renew_subs 2024-09-16T03:53:02.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.300+0000 7fd734cdc640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:53:02.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.300+0000 7fd734cdc640 1 -- 172.21.15.42:0/3229889061 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd730073a30 con 0x7fd73009c920 2024-09-16T03:53:02.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.300+0000 7fd734cdc640 1 librados: init done 2024-09-16T03:53:02.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.300+0000 7fd734cdc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:53:02.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.300+0000 7fd734cdc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:53:02.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.300+0000 7fd734cdc640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:53:02.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.300+0000 7fd734cdc640 1 -- 172.21.15.42:0/3229889061 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd7300a1710 con 0x7fd73009c920 2024-09-16T03:53:02.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.300+0000 7fd726ffd640 1 -- 172.21.15.42:0/3229889061 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fd718005490 con 0x7fd73009c920 2024-09-16T03:53:02.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.300+0000 7fd726ffd640 1 --2- 172.21.15.42:0/3229889061 >> v2:172.21.15.154:6812/254845342 conn(0x7fd7100377c0 0x7fd710039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:53:02.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.300+0000 7fd726ffd640 1 -- 172.21.15.42:0/3229889061 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(345..345 src has 286..345) ==== 5769+0+0 (secure 0 0 0) 0x7fd718048200 con 0x7fd73009c920 2024-09-16T03:53:02.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.300+0000 7fd726ffd640 10 client.834167.objecter ms_dispatch 0x7fd730000ba0 osd_map(345..345 src has 286..345) 2024-09-16T03:53:02.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.300+0000 7fd726ffd640 3 client.834167.objecter handle_osd_map got epochs [345,345] > 0 2024-09-16T03:53:02.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.300+0000 7fd726ffd640 3 client.834167.objecter handle_osd_map decoding full epoch 345 2024-09-16T03:53:02.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.300+0000 7fd726ffd640 20 client.834167.objecter dump_active .. 0 homeless 2024-09-16T03:53:02.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.300+0000 7fd72e59c640 1 --2- 172.21.15.42:0/3229889061 >> v2:172.21.15.154:6812/254845342 conn(0x7fd7100377c0 0x7fd710039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:53:02.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.300+0000 7fd72e59c640 10 monclient: get_auth_request con 0x7fd7100377c0 auth_method 0 2024-09-16T03:53:02.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.300+0000 7fd72e59c640 1 --2- 172.21.15.42:0/3229889061 >> v2:172.21.15.154:6812/254845342 conn(0x7fd7100377c0 0x7fd710039c80 secure :-1 s=READY pgs=3591 cs=0 l=1 rev1=1 crypto rx=0x7fd7200af000 tx=0x7fd7200a7070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:53:02.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.300+0000 7fd726ffd640 10 client.834167.objecter ms_handle_connect 0x7fd7100377c0 2024-09-16T03:53:02.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.304+0000 7fd726ffd640 1 -- 172.21.15.42:0/3229889061 <== mon.2 v2:172.21.15.42:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7fd71800e340 con 0x7fd73009c920 2024-09-16T03:53:02.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.304+0000 7fd726ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:53:02.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.304+0000 7fd726ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:53:02.543 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.536+0000 7fd734cdc640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:53:02.543 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.536+0000 7fd734cdc640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:53:02.543 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.536+0000 7fd734cdc640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:53:02.543 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.536+0000 7fd734cdc640 1 -- 172.21.15.42:0/3229889061 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fd73011f070 con 0x7fd73009c920 2024-09-16T03:53:02.543 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.536+0000 7fd726ffd640 1 -- 172.21.15.42:0/3229889061 <== mon.2 v2:172.21.15.42:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7fd718005780 con 0x7fd73009c920 2024-09-16T03:53:02.543 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.536+0000 7fd726ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:53:02.544 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.536+0000 7fd726ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:53:02.544 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":94,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:53:02.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.540+0000 7fd734cdc640 10 librados: watch_flush enter 2024-09-16T03:53:02.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.540+0000 7fd734cdc640 10 librados: watch_flush exit 2024-09-16T03:53:02.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.540+0000 7fd734cdc640 20 client.834167.objecter shutdown clearing up homeless session... 2024-09-16T03:53:02.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.540+0000 7fd734cdc640 10 client.834167.objecter successfully canceled tick 2024-09-16T03:53:02.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.540+0000 7fd734cdc640 1 -- 172.21.15.42:0/3229889061 >> v2:172.21.15.154:6812/254845342 conn(0x7fd7100377c0 msgr2=0x7fd710039c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:53:02.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.540+0000 7fd734cdc640 1 --2- 172.21.15.42:0/3229889061 >> v2:172.21.15.154:6812/254845342 conn(0x7fd7100377c0 0x7fd710039c80 secure :-1 s=READY pgs=3591 cs=0 l=1 rev1=1 crypto rx=0x7fd7200af000 tx=0x7fd7200a7070 comp rx=0 tx=0).stop 2024-09-16T03:53:02.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.540+0000 7fd734cdc640 10 monclient: shutdown 2024-09-16T03:53:02.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.540+0000 7fd734cdc640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:53:02.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.540+0000 7fd734cdc640 1 -- 172.21.15.42:0/3229889061 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd73009c920 msgr2=0x7fd730137440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:53:02.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.540+0000 7fd734cdc640 1 --2- 172.21.15.42:0/3229889061 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd73009c920 0x7fd730137440 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fd71800b580 tx=0x7fd71800ba40 comp rx=0 tx=0).stop 2024-09-16T03:53:02.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.540+0000 7fd734cdc640 1 -- 172.21.15.42:0/3229889061 shutdown_connections 2024-09-16T03:53:02.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.540+0000 7fd734cdc640 1 --2- 172.21.15.42:0/3229889061 >> v2:172.21.15.154:6812/254845342 conn(0x7fd7100377c0 0x7fd710039c80 unknown :-1 s=CLOSED pgs=3591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:53:02.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.540+0000 7fd734cdc640 1 --2- 172.21.15.42:0/3229889061 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd7300a1330 0x7fd73011caf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:53:02.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.540+0000 7fd734cdc640 1 --2- 172.21.15.42:0/3229889061 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd73009c920 0x7fd730137440 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:53:02.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.540+0000 7fd734cdc640 1 --2- 172.21.15.42:0/3229889061 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd730072d40 0x7fd730136f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:53:02.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.540+0000 7fd734cdc640 1 -- 172.21.15.42:0/3229889061 >> 172.21.15.42:0/3229889061 conn(0x7fd7300a2420 msgr2=0x7fd7300a0a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:53:02.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.540+0000 7fd734cdc640 1 -- 172.21.15.42:0/3229889061 shutdown_connections 2024-09-16T03:53:02.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.540+0000 7fd734cdc640 1 -- 172.21.15.42:0/3229889061 wait complete. 2024-09-16T03:53:02.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:02.540+0000 7fd734cdc640 1 librados: shutdown 2024-09-16T03:53:04.811 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b has been restored 2024-09-16T03:53:05.563 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:53:05.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.692+0000 7f2fd84a0640 10 monclient: get_monmap_and_config 2024-09-16T03:53:05.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.692+0000 7f2fd84a0640 10 monclient: build_initial_monmap 2024-09-16T03:53:05.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.692+0000 7f2fd84a0640 10 monclient: monmap: 2024-09-16T03:53:05.698 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:53:05.698 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:53:05.698 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:53:05.696782+0000 2024-09-16T03:53:05.698 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:53:05.696782+0000 2024-09-16T03:53:05.698 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:53:05.699 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:53:05.699 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:53:05.699 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:53:05.699 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:53:05.699 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:53:05.699 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:53:05.699 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:53:05.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.692+0000 7f2fd84a0640 1 Processor -- start 2024-09-16T03:53:05.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.692+0000 7f2fd84a0640 1 -- start start 2024-09-16T03:53:05.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.692+0000 7f2fd84a0640 10 monclient: init 2024-09-16T03:53:05.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.692+0000 7f2fd84a0640 10 monclient: _reopen_session rank -1 2024-09-16T03:53:05.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.692+0000 7f2fd84a0640 10 monclient: _add_conns ranks=[4,0,3,2,1] 2024-09-16T03:53:05.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.692+0000 7f2fd84a0640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f2fd00a39b0 0x7f2fd0133b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:53:05.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.692+0000 7f2fd84a0640 10 monclient(hunting): picked mon.noname-c con 0x7f2fd00a39b0 addr v2:172.21.15.154:3300/0 2024-09-16T03:53:05.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.692+0000 7f2fd84a0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2fd00a3d90 addr v1:172.21.15.42:6789/0 2024-09-16T03:53:05.702 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.692+0000 7f2fd84a0640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f2fd0137710 0x7f2fd013f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:53:05.702 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.692+0000 7f2fd84a0640 10 monclient(hunting): picked mon.noname-a con 0x7f2fd0137710 addr v2:172.21.15.42:3300/0 2024-09-16T03:53:05.702 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.692+0000 7f2fd84a0640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2fd009c140 con 0x7f2fd00a3d90 2024-09-16T03:53:05.702 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.692+0000 7f2fd84a0640 10 monclient(hunting): start opening mon connection 2024-09-16T03:53:05.702 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.692+0000 7f2fd84a0640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f2fd009c420 con 0x7f2fd0137710 2024-09-16T03:53:05.702 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.692+0000 7f2fd84a0640 10 monclient(hunting): start opening mon connection 2024-09-16T03:53:05.702 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.692+0000 7f2fd84a0640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f2fd006dd60 con 0x7f2fd00a39b0 2024-09-16T03:53:05.702 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.692+0000 7f2fd84a0640 10 monclient(hunting): _renew_subs 2024-09-16T03:53:05.702 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.692+0000 7f2fd84a0640 10 monclient(hunting): authenticate will time out at 1632.666704s 2024-09-16T03:53:05.702 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.692+0000 7f2fd623c640 1 -- >> v2:172.21.15.154:3300/0 conn(0x7f2fd00a39b0 msgr2=0x7f2fd0133b80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T03:53:05.702 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.692+0000 7f2fd623c640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f2fd00a39b0 0x7f2fd0133b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:53:05.702 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.692+0000 7f2fd6a3d640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f2fd0137710 0x7f2fd013f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:53:05.702 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.692+0000 7f2fd5a3b640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f2fd00a3d90 0x7f2fd0137000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:50976/0 (socket says 172.21.15.42:50976) 2024-09-16T03:53:05.702 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.692+0000 7f2fd5a3b640 1 -- 172.21.15.42:0/3786204166 learned_addr learned my addr 172.21.15.42:0/3786204166 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:53:05.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.692+0000 7f2fd6a3d640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f2fd0137710 0x7f2fd013f3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:37178/0 (socket says 172.21.15.42:37178) 2024-09-16T03:53:05.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.692+0000 7f2fd6a3d640 10 monclient(hunting): get_auth_request con 0x7f2fd0137710 auth_method 0 2024-09-16T03:53:05.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.692+0000 7f2fd6a3d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:53:05.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.692+0000 7f2fd6a3d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:53:05.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.692+0000 7f2fd6a3d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:53:05.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.692+0000 7f2fd6a3d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:53:05.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.692+0000 7f2fd6a3d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:53:05.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.692+0000 7f2fd6a3d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:53:05.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.692+0000 7f2fd6a3d640 10 monclient(hunting): handle_auth_done global_id 824277 payload 274 2024-09-16T03:53:05.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.692+0000 7f2fd6a3d640 1 -- 172.21.15.42:0/3786204166 >> v2:172.21.15.154:3300/0 conn(0x7f2fd00a39b0 msgr2=0x7f2fd0133b80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:53:05.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.692+0000 7f2fd6a3d640 1 --2- 172.21.15.42:0/3786204166 >> v2:172.21.15.154:3300/0 conn(0x7f2fd00a39b0 0x7f2fd0133b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:53:05.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.692+0000 7f2fd6a3d640 1 -- 172.21.15.42:0/3786204166 >> v1:172.21.15.42:6789/0 conn(0x7f2fd00a3d90 legacy=0x7f2fd0137000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:53:05.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.692+0000 7f2fd6a3d640 10 monclient: _finish_hunting 0 2024-09-16T03:53:05.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.692+0000 7f2fd6a3d640 1 monclient: found mon.noname-a 2024-09-16T03:53:05.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.692+0000 7f2fd6a3d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:53:05.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.692+0000 7f2fd6a3d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:53:05.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.692+0000 7f2fd6a3d640 1 -- 172.21.15.42:0/3786204166 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2fd00a5e20 con 0x7f2fd0137710 2024-09-16T03:53:05.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.696+0000 7f2fd6a3d640 1 --2- 172.21.15.42:0/3786204166 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2fd0137710 0x7f2fd013f3a0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f2fd00a7750 tx=0x7f2fc801c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=185da3a175cd4bd4 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:53:05.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.696+0000 7f2fd523a640 1 -- 172.21.15.42:0/3786204166 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fc80a66e0 con 0x7f2fd0137710 2024-09-16T03:53:05.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.696+0000 7f2fd523a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:53:05.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.696+0000 7f2fd523a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:53:05.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.696+0000 7f2fd523a640 10 monclient: dump: 2024-09-16T03:53:05.710 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:53:05.710 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:53:05.710 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:05.710 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:05.710 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:53:05.710 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:53:05.710 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:53:05.710 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:53:05.710 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:53:05.710 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:53:05.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.696+0000 7f2fd523a640 10 monclient: _finish_auth 0 2024-09-16T03:53:05.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.696+0000 7f2fd523a640 10 monclient: _check_auth_tickets 2024-09-16T03:53:05.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.696+0000 7f2fd523a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:53:05.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.696+0000 7f2fd523a640 1 -- 172.21.15.42:0/3786204166 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2fc80a6880 con 0x7f2fd0137710 2024-09-16T03:53:05.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.696+0000 7f2fd84a0640 5 monclient: authenticate success, global_id 824277 2024-09-16T03:53:05.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.696+0000 7f2fd84a0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:53:05.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.696+0000 7f2fd523a640 10 monclient: handle_config config(0 keys) 2024-09-16T03:53:05.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.696+0000 7f2fd84a0640 10 monclient: get_monmap_and_config success 2024-09-16T03:53:05.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.696+0000 7f2fd523a640 1 -- 172.21.15.42:0/3786204166 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fc80a6c50 con 0x7f2fd0137710 2024-09-16T03:53:05.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.696+0000 7f2fd523a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:53:05.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.696+0000 7f2fd84a0640 4 set_mon_vals no callback set 2024-09-16T03:53:05.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.696+0000 7f2fd523a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:53:05.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.696+0000 7f2fd523a640 10 monclient: dump: 2024-09-16T03:53:05.711 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:53:05.711 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:53:05.711 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:05.711 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:05.712 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:53:05.712 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:53:05.712 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:53:05.712 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:53:05.712 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:53:05.712 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:53:05.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.696+0000 7f2fd84a0640 10 monclient: shutdown 2024-09-16T03:53:05.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.696+0000 7f2fd84a0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:53:05.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.696+0000 7f2fd84a0640 1 -- 172.21.15.42:0/3786204166 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2fd0137710 msgr2=0x7f2fd013f3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:53:05.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.696+0000 7f2fd84a0640 1 --2- 172.21.15.42:0/3786204166 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2fd0137710 0x7f2fd013f3a0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f2fd00a7750 tx=0x7f2fc801c810 comp rx=0 tx=0).stop 2024-09-16T03:53:05.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.696+0000 7f2fd84a0640 1 -- 172.21.15.42:0/3786204166 shutdown_connections 2024-09-16T03:53:05.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.696+0000 7f2fd84a0640 1 --2- 172.21.15.42:0/3786204166 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2fd0137710 0x7f2fd013f3a0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:53:05.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.696+0000 7f2fd84a0640 1 --2- 172.21.15.42:0/3786204166 >> v2:172.21.15.154:3300/0 conn(0x7f2fd00a39b0 0x7f2fd0133b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:53:05.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.696+0000 7f2fd84a0640 1 -- 172.21.15.42:0/3786204166 >> 172.21.15.42:0/3786204166 conn(0x7f2fd00a9fb0 msgr2=0x7f2fd00aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:53:05.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.696+0000 7f2fd84a0640 1 -- 172.21.15.42:0/3786204166 shutdown_connections 2024-09-16T03:53:05.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.696+0000 7f2fd84a0640 1 -- 172.21.15.42:0/3786204166 wait complete. 2024-09-16T03:53:05.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.696+0000 7f2fd84a0640 10 monclient: build_initial_monmap 2024-09-16T03:53:05.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.696+0000 7f2fd84a0640 10 monclient: monmap: 2024-09-16T03:53:05.713 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:53:05.713 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:53:05.713 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:53:05.713 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:53:05.713 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:53:05.713 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:53:05.713 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:53:05.713 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:53:05.714 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:53:05.714 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:53:05.714 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.696+0000 7f2fd84a0640 1 librados: starting msgr at 2024-09-16T03:53:05.714 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.696+0000 7f2fd84a0640 1 librados: starting objecter 2024-09-16T03:53:05.714 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.696+0000 7f2fd84a0640 1 Processor -- start 2024-09-16T03:53:05.714 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.696+0000 7f2fd84a0640 1 -- start start 2024-09-16T03:53:05.714 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.696+0000 7f2fd84a0640 1 librados: setting wanted keys 2024-09-16T03:53:05.714 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.696+0000 7f2fd84a0640 1 librados: calling monclient init 2024-09-16T03:53:05.714 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.696+0000 7f2fd84a0640 10 monclient: init 2024-09-16T03:53:05.714 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.696+0000 7f2fd84a0640 10 monclient: _reopen_session rank -1 2024-09-16T03:53:05.714 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.696+0000 7f2fd84a0640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:53:05.714 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.696+0000 7f2fd84a0640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2fd00a39b0 0x7f2fd0139eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:53:05.714 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.696+0000 7f2fd84a0640 10 monclient(hunting): picked mon.noname-a con 0x7f2fd00a39b0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:53:05.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.696+0000 7f2fd84a0640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2fd00a3d90 0x7f2fd013a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:53:05.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.696+0000 7f2fd84a0640 10 monclient(hunting): picked mon.noname-c con 0x7f2fd00a3d90 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:53:05.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.696+0000 7f2fd84a0640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2fd0137710 0x7f2fd013a930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:53:05.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.696+0000 7f2fd84a0640 10 monclient(hunting): picked mon.noname-b con 0x7f2fd0137710 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:53:05.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.696+0000 7f2fd84a0640 10 monclient(hunting): start opening mon connection 2024-09-16T03:53:05.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.696+0000 7f2fd84a0640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f2fd0142310 con 0x7f2fd00a39b0 2024-09-16T03:53:05.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.696+0000 7f2fd84a0640 10 monclient(hunting): start opening mon connection 2024-09-16T03:53:05.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.696+0000 7f2fd84a0640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f2fd0142490 con 0x7f2fd0137710 2024-09-16T03:53:05.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.696+0000 7f2fd84a0640 10 monclient(hunting): start opening mon connection 2024-09-16T03:53:05.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.696+0000 7f2fd84a0640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f2fd0142610 con 0x7f2fd00a3d90 2024-09-16T03:53:05.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.696+0000 7f2fd84a0640 10 monclient(hunting): _renew_subs 2024-09-16T03:53:05.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.696+0000 7f2fd84a0640 10 monclient(hunting): authenticate will time out at 1632.671840s 2024-09-16T03:53:05.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.696+0000 7f2fd623c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2fd00a39b0 0x7f2fd0139eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:53:05.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.696+0000 7f2fd5a3b640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2fd00a3d90 0x7f2fd013a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:53:05.716 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.700+0000 7f2fd5a3b640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2fd00a3d90 0x7f2fd013a3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:55102/0 (socket says 172.21.15.42:55102) 2024-09-16T03:53:05.716 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.700+0000 7f2fd5a3b640 1 -- 172.21.15.42:0/1151326679 learned_addr learned my addr 172.21.15.42:0/1151326679 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:53:05.716 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.700+0000 7f2fd5a3b640 10 monclient(hunting): get_auth_request con 0x7f2fd00a3d90 auth_method 0 2024-09-16T03:53:05.716 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.700+0000 7f2fd5a3b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:53:05.716 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.700+0000 7f2fd5a3b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:53:05.716 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.700+0000 7f2fd5a3b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:53:05.716 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.700+0000 7f2fd623c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2fd00a39b0 0x7f2fd0139eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:37188/0 (socket says 172.21.15.42:37188) 2024-09-16T03:53:05.716 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.700+0000 7f2fd623c640 10 monclient(hunting): get_auth_request con 0x7f2fd00a39b0 auth_method 0 2024-09-16T03:53:05.716 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.700+0000 7f2fd623c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:53:05.716 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.700+0000 7f2fd6a3d640 1 -- 172.21.15.42:0/1151326679 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2fd0137710 msgr2=0x7f2fd013a930 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T03:53:05.716 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.700+0000 7f2fd623c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:53:05.716 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.700+0000 7f2fd623c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:53:05.717 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.700+0000 7f2fd6a3d640 1 --2- 172.21.15.42:0/1151326679 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2fd0137710 0x7f2fd013a930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:53:05.717 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.700+0000 7f2fd5a3b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:53:05.717 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.700+0000 7f2fd5a3b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:53:05.717 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.700+0000 7f2fd5a3b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:53:05.717 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.700+0000 7f2fd623c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:53:05.717 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.700+0000 7f2fd623c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:53:05.717 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.700+0000 7f2fd623c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:53:05.717 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.700+0000 7f2fd5a3b640 10 monclient(hunting): handle_auth_done global_id 834170 payload 824 2024-09-16T03:53:05.717 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.700+0000 7f2fd5a3b640 1 -- 172.21.15.42:0/1151326679 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2fd0137710 msgr2=0x7f2fd013a930 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:53:05.717 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.700+0000 7f2fd5a3b640 1 --2- 172.21.15.42:0/1151326679 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2fd0137710 0x7f2fd013a930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:53:05.717 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.700+0000 7f2fd5a3b640 1 -- 172.21.15.42:0/1151326679 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2fd00a39b0 msgr2=0x7f2fd0139eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:53:05.717 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.700+0000 7f2fd5a3b640 1 --2- 172.21.15.42:0/1151326679 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2fd00a39b0 0x7f2fd0139eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:53:05.717 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.700+0000 7f2fd5a3b640 10 monclient: _finish_hunting 0 2024-09-16T03:53:05.718 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.700+0000 7f2fd5a3b640 1 monclient: found mon.noname-c 2024-09-16T03:53:05.718 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.700+0000 7f2fd5a3b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:53:05.718 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.700+0000 7f2fd5a3b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:53:05.718 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.700+0000 7f2fd5a3b640 1 -- 172.21.15.42:0/1151326679 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2fd013ba30 con 0x7f2fd00a3d90 2024-09-16T03:53:05.718 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.700+0000 7f2fd623c640 1 --2- 172.21.15.42:0/1151326679 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2fd00a39b0 0x7f2fd0139eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:53:05.718 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.700+0000 7f2fd5a3b640 1 --2- 172.21.15.42:0/1151326679 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2fd00a3d90 0x7f2fd013a3f0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f2fc000c450 tx=0x7f2fc000c910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:53:05.718 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.700+0000 7f2fc6ffd640 10 client.?.objecter ms_handle_connect 0x7f2fd00a3d90 2024-09-16T03:53:05.718 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.700+0000 7f2fc6ffd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:53:05.718 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.700+0000 7f2fc6ffd640 1 -- 172.21.15.42:0/1151326679 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fc0017070 con 0x7f2fd00a3d90 2024-09-16T03:53:05.718 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.700+0000 7f2fc6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:53:05.718 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.700+0000 7f2fc6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:53:05.718 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.700+0000 7f2fc6ffd640 10 monclient: dump: 2024-09-16T03:53:05.718 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:53:05.718 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:53:05.719 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:05.719 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:05.719 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:53:05.719 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:53:05.719 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:53:05.719 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:53:05.719 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:53:05.719 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:53:05.719 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.700+0000 7f2fc6ffd640 10 monclient: _finish_auth 0 2024-09-16T03:53:05.719 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.700+0000 7f2fc6ffd640 10 monclient: _check_auth_tickets 2024-09-16T03:53:05.719 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.700+0000 7f2fc6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:53:05.719 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.700+0000 7f2fd84a0640 5 monclient: authenticate success, global_id 834170 2024-09-16T03:53:05.719 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.700+0000 7f2fc6ffd640 1 -- 172.21.15.42:0/1151326679 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2fc0005cf0 con 0x7f2fd00a3d90 2024-09-16T03:53:05.720 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.700+0000 7f2fd84a0640 10 monclient: _renew_subs 2024-09-16T03:53:05.720 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.700+0000 7f2fd84a0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:53:05.720 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.700+0000 7f2fd84a0640 1 -- 172.21.15.42:0/1151326679 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2fd013c590 con 0x7f2fd00a3d90 2024-09-16T03:53:05.720 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.700+0000 7f2fc6ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:53:05.720 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.700+0000 7f2fc6ffd640 1 -- 172.21.15.42:0/1151326679 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fc0007290 con 0x7f2fd00a3d90 2024-09-16T03:53:05.720 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.700+0000 7f2fc6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:53:05.720 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.700+0000 7f2fc6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:53:05.720 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.700+0000 7f2fc6ffd640 10 monclient: dump: 2024-09-16T03:53:05.720 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:53:05.720 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:53:05.720 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:05.720 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:05.721 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:53:05.721 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:53:05.721 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:53:05.721 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:53:05.721 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:53:05.721 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:53:05.721 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.700+0000 7f2fd84a0640 10 client.834170.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:53:05.721 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.700+0000 7f2fd84a0640 10 monclient: _renew_subs 2024-09-16T03:53:05.721 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.700+0000 7f2fc7fff640 4 set_mon_vals no callback set 2024-09-16T03:53:05.721 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.700+0000 7f2fd84a0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:53:05.721 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.700+0000 7f2fd84a0640 1 -- 172.21.15.42:0/1151326679 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f2fd013f1b0 con 0x7f2fd00a3d90 2024-09-16T03:53:05.721 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.700+0000 7f2fd84a0640 1 librados: init done 2024-09-16T03:53:05.721 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.700+0000 7f2fd84a0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:53:05.722 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.700+0000 7f2fd84a0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:53:05.722 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.700+0000 7f2fd84a0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:53:05.722 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.700+0000 7f2fd84a0640 1 -- 172.21.15.42:0/1151326679 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2fd006f660 con 0x7f2fd00a3d90 2024-09-16T03:53:05.722 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.700+0000 7f2fc6ffd640 1 -- 172.21.15.42:0/1151326679 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f2fc001a070 con 0x7f2fd00a3d90 2024-09-16T03:53:05.722 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.700+0000 7f2fc6ffd640 1 --2- 172.21.15.42:0/1151326679 >> v2:172.21.15.154:6812/254845342 conn(0x7f2fb40377c0 0x7f2fb4039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:53:05.722 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.700+0000 7f2fc6ffd640 1 -- 172.21.15.42:0/1151326679 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(345..345 src has 286..345) ==== 5769+0+0 (secure 0 0 0) 0x7f2fc00482b0 con 0x7f2fd00a3d90 2024-09-16T03:53:05.722 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.700+0000 7f2fc6ffd640 10 client.834170.objecter ms_dispatch 0x7f2fd0000ba0 osd_map(345..345 src has 286..345) 2024-09-16T03:53:05.722 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.700+0000 7f2fc6ffd640 3 client.834170.objecter handle_osd_map got epochs [345,345] > 0 2024-09-16T03:53:05.722 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.700+0000 7f2fc6ffd640 3 client.834170.objecter handle_osd_map decoding full epoch 345 2024-09-16T03:53:05.722 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.700+0000 7f2fc6ffd640 20 client.834170.objecter dump_active .. 0 homeless 2024-09-16T03:53:05.722 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.700+0000 7f2fd623c640 1 --2- 172.21.15.42:0/1151326679 >> v2:172.21.15.154:6812/254845342 conn(0x7f2fb40377c0 0x7f2fb4039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:53:05.722 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.700+0000 7f2fd623c640 10 monclient: get_auth_request con 0x7f2fb40377c0 auth_method 0 2024-09-16T03:53:05.722 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.700+0000 7f2fc6ffd640 10 client.834170.objecter ms_handle_connect 0x7f2fb40377c0 2024-09-16T03:53:05.722 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.700+0000 7f2fd623c640 1 --2- 172.21.15.42:0/1151326679 >> v2:172.21.15.154:6812/254845342 conn(0x7f2fb40377c0 0x7f2fb4039c80 secure :-1 s=READY pgs=3592 cs=0 l=1 rev1=1 crypto rx=0x7f2fcc003940 tx=0x7f2fcc003ba0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:53:05.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.704+0000 7f2fc6ffd640 1 -- 172.21.15.42:0/1151326679 <== mon.2 v2:172.21.15.42:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f2fc0007610 con 0x7f2fd00a3d90 2024-09-16T03:53:05.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.704+0000 7f2fc6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:53:05.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.704+0000 7f2fc6ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:53:05.943 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.936+0000 7f2fd84a0640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:53:05.943 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.936+0000 7f2fd84a0640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:53:05.944 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.936+0000 7f2fd84a0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:53:05.944 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.936+0000 7f2fd84a0640 1 -- 172.21.15.42:0/1151326679 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f2fd0133b80 con 0x7f2fd00a3d90 2024-09-16T03:53:05.944 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.936+0000 7f2fc6ffd640 1 -- 172.21.15.42:0/1151326679 <== mon.2 v2:172.21.15.42:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7f2fc001b8d0 con 0x7f2fd00a3d90 2024-09-16T03:53:05.944 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.936+0000 7f2fc6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:53:05.944 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.936+0000 7f2fc6ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:53:05.945 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":94,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:53:05.946 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.940+0000 7f2fd84a0640 10 librados: watch_flush enter 2024-09-16T03:53:05.947 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.940+0000 7f2fd84a0640 10 librados: watch_flush exit 2024-09-16T03:53:05.947 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.940+0000 7f2fd84a0640 20 client.834170.objecter shutdown clearing up homeless session... 2024-09-16T03:53:05.947 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.940+0000 7f2fd84a0640 10 client.834170.objecter successfully canceled tick 2024-09-16T03:53:05.947 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.940+0000 7f2fd84a0640 1 -- 172.21.15.42:0/1151326679 >> v2:172.21.15.154:6812/254845342 conn(0x7f2fb40377c0 msgr2=0x7f2fb4039c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:53:05.947 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.940+0000 7f2fd84a0640 1 --2- 172.21.15.42:0/1151326679 >> v2:172.21.15.154:6812/254845342 conn(0x7f2fb40377c0 0x7f2fb4039c80 secure :-1 s=READY pgs=3592 cs=0 l=1 rev1=1 crypto rx=0x7f2fcc003940 tx=0x7f2fcc003ba0 comp rx=0 tx=0).stop 2024-09-16T03:53:05.947 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.940+0000 7f2fd84a0640 10 monclient: shutdown 2024-09-16T03:53:05.947 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.940+0000 7f2fd84a0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:53:05.947 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.940+0000 7f2fd84a0640 1 -- 172.21.15.42:0/1151326679 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2fd00a3d90 msgr2=0x7f2fd013a3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:53:05.947 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.940+0000 7f2fd84a0640 1 --2- 172.21.15.42:0/1151326679 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2fd00a3d90 0x7f2fd013a3f0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f2fc000c450 tx=0x7f2fc000c910 comp rx=0 tx=0).stop 2024-09-16T03:53:05.947 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.940+0000 7f2fd84a0640 1 -- 172.21.15.42:0/1151326679 shutdown_connections 2024-09-16T03:53:05.947 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.940+0000 7f2fd84a0640 1 --2- 172.21.15.42:0/1151326679 >> v2:172.21.15.154:6812/254845342 conn(0x7f2fb40377c0 0x7f2fb4039c80 unknown :-1 s=CLOSED pgs=3592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:53:05.947 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.940+0000 7f2fd84a0640 1 --2- 172.21.15.42:0/1151326679 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2fd0137710 0x7f2fd013a930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:53:05.948 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.940+0000 7f2fd84a0640 1 --2- 172.21.15.42:0/1151326679 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2fd00a3d90 0x7f2fd013a3f0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:53:05.948 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.940+0000 7f2fd84a0640 1 --2- 172.21.15.42:0/1151326679 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2fd00a39b0 0x7f2fd0139eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:53:05.948 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.940+0000 7f2fd84a0640 1 -- 172.21.15.42:0/1151326679 >> 172.21.15.42:0/1151326679 conn(0x7f2fd00a9fb0 msgr2=0x7f2fd0131980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:53:05.948 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.940+0000 7f2fd84a0640 1 -- 172.21.15.42:0/1151326679 shutdown_connections 2024-09-16T03:53:05.948 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.940+0000 7f2fd84a0640 1 -- 172.21.15.42:0/1151326679 wait complete. 2024-09-16T03:53:05.948 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:05.940+0000 7f2fd84a0640 1 librados: shutdown 2024-09-16T03:53:08.963 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:53:09.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:09.088+0000 7f5bb627b640 10 monclient: get_monmap_and_config 2024-09-16T03:53:09.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:09.088+0000 7f5bb627b640 10 monclient: build_initial_monmap 2024-09-16T03:53:09.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:09.088+0000 7f5bb627b640 10 monclient: monmap: 2024-09-16T03:53:09.095 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:53:09.095 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:53:09.095 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:53:09.093855+0000 2024-09-16T03:53:09.095 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:53:09.093855+0000 2024-09-16T03:53:09.096 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:53:09.096 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:53:09.096 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:53:09.096 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:53:09.096 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:53:09.096 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:53:09.096 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:53:09.096 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:53:09.098 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:09.088+0000 7f5bb627b640 1 Processor -- start 2024-09-16T03:53:09.098 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:09.088+0000 7f5bb627b640 1 -- start start 2024-09-16T03:53:09.098 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:09.088+0000 7f5bb627b640 10 monclient: init 2024-09-16T03:53:09.098 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:09.088+0000 7f5bb627b640 10 monclient: _reopen_session rank -1 2024-09-16T03:53:09.098 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:09.088+0000 7f5bb627b640 10 monclient: _add_conns ranks=[0,4,3,1,2] 2024-09-16T03:53:09.098 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:09.088+0000 7f5bb627b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5bb00a1330 addr v1:172.21.15.42:6789/0 2024-09-16T03:53:09.098 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:09.088+0000 7f5bb627b640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f5bb006dac0 0x7f5bb013a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:53:09.098 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:09.092+0000 7f5bb627b640 10 monclient(hunting): picked mon.noname-c con 0x7f5bb006dac0 addr v2:172.21.15.154:3300/0 2024-09-16T03:53:09.099 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:09.092+0000 7f5bb627b640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f5bb00a1e20 0x7f5bb013c890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:53:09.099 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:09.092+0000 7f5bb627b640 10 monclient(hunting): picked mon.noname-a con 0x7f5bb00a1e20 addr v2:172.21.15.42:3300/0 2024-09-16T03:53:09.099 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:09.092+0000 7f5bb627b640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5bb009c090 con 0x7f5bb00a1330 2024-09-16T03:53:09.099 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:09.092+0000 7f5bb627b640 10 monclient(hunting): start opening mon connection 2024-09-16T03:53:09.099 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:09.092+0000 7f5bb627b640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f5bb009f610 con 0x7f5bb00a1e20 2024-09-16T03:53:09.099 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:09.092+0000 7f5bb627b640 10 monclient(hunting): start opening mon connection 2024-09-16T03:53:09.099 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:09.092+0000 7f5bb627b640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f5bb009f8f0 con 0x7f5bb006dac0 2024-09-16T03:53:09.099 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:09.092+0000 7f5bb627b640 10 monclient(hunting): _renew_subs 2024-09-16T03:53:09.099 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:09.092+0000 7f5bb627b640 10 monclient(hunting): authenticate will time out at 1636.064090s 2024-09-16T03:53:09.099 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:09.092+0000 7f5bb4818640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f5bb00a1e20 0x7f5bb013c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:53:09.099 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:09.092+0000 7f5baffff640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f5bb00a1330 0x7f5bb00a1710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:50980/0 (socket says 172.21.15.42:50980) 2024-09-16T03:53:09.099 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:09.092+0000 7f5ba77fe640 1 -- >> v2:172.21.15.154:3300/0 conn(0x7f5bb006dac0 msgr2=0x7f5bb013a340 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T03:53:09.100 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:09.092+0000 7f5baffff640 1 -- 172.21.15.42:0/3118286035 learned_addr learned my addr 172.21.15.42:0/3118286035 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:53:09.100 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:09.092+0000 7f5bb4818640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f5bb00a1e20 0x7f5bb013c890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:37194/0 (socket says 172.21.15.42:37194) 2024-09-16T03:53:09.100 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:09.092+0000 7f5bb4818640 10 monclient(hunting): get_auth_request con 0x7f5bb00a1e20 auth_method 0 2024-09-16T03:53:09.100 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:09.092+0000 7f5bb4818640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:53:09.100 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:09.092+0000 7f5bb4818640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:53:09.100 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:09.092+0000 7f5bb4818640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:53:09.100 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:09.092+0000 7f5ba77fe640 1 --2- 172.21.15.42:0/3118286035 >> v2:172.21.15.154:3300/0 conn(0x7f5bb006dac0 0x7f5bb013a340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:53:09.100 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:09.092+0000 7f5bb4818640 1 -- 172.21.15.42:0/3118286035 >> v2:172.21.15.42:3300/0 conn(0x7f5bb00a1e20 msgr2=0x7f5bb013c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T03:53:09.100 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:09.092+0000 7f5bb4818640 1 -- 172.21.15.42:0/3118286035 >> v2:172.21.15.42:3300/0 conn(0x7f5bb00a1e20 msgr2=0x7f5bb013c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:53:09.100 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:09.092+0000 7f5bb4818640 1 --2- 172.21.15.42:0/3118286035 >> v2:172.21.15.42:3300/0 conn(0x7f5bb00a1e20 0x7f5bb013c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:53:09.100 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:09.092+0000 7f5bb4818640 1 --2- 172.21.15.42:0/3118286035 >> v2:172.21.15.42:3300/0 conn(0x7f5bb00a1e20 0x7f5bb013c890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:53:09.298 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:09.296+0000 7f5ba77fe640 1 -- 172.21.15.42:0/3118286035 >> v2:172.21.15.154:3300/0 conn(0x7f5bb006dac0 msgr2=0x7f5bb013a340 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T03:53:09.298 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:09.296+0000 7f5ba77fe640 1 --2- 172.21.15.42:0/3118286035 >> v2:172.21.15.154:3300/0 conn(0x7f5bb006dac0 0x7f5bb013a340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T03:53:09.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:09.296+0000 7f5bb4818640 1 --2- 172.21.15.42:0/3118286035 >> v2:172.21.15.42:3300/0 conn(0x7f5bb00a1e20 0x7f5bb013c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:53:09.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:09.296+0000 7f5bb4818640 10 monclient(hunting): get_auth_request con 0x7f5bb00a1e20 auth_method 0 2024-09-16T03:53:09.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:09.296+0000 7f5bb4818640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:53:09.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:09.296+0000 7f5bb4818640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:53:09.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:09.296+0000 7f5bb4818640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:53:09.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:09.296+0000 7f5bb4818640 1 -- 172.21.15.42:0/3118286035 >> v2:172.21.15.42:3300/0 conn(0x7f5bb00a1e20 msgr2=0x7f5bb013c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T03:53:09.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:09.296+0000 7f5bb4818640 1 -- 172.21.15.42:0/3118286035 >> v2:172.21.15.42:3300/0 conn(0x7f5bb00a1e20 msgr2=0x7f5bb013c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:53:09.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:09.296+0000 7f5bb4818640 1 --2- 172.21.15.42:0/3118286035 >> v2:172.21.15.42:3300/0 conn(0x7f5bb00a1e20 0x7f5bb013c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:53:09.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:09.296+0000 7f5bb4818640 1 --2- 172.21.15.42:0/3118286035 >> v2:172.21.15.42:3300/0 conn(0x7f5bb00a1e20 0x7f5bb013c890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T03:53:09.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:09.700+0000 7f5ba77fe640 1 -- 172.21.15.42:0/3118286035 >> v2:172.21.15.154:3300/0 conn(0x7f5bb006dac0 msgr2=0x7f5bb013a340 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T03:53:09.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:09.700+0000 7f5ba77fe640 1 --2- 172.21.15.42:0/3118286035 >> v2:172.21.15.154:3300/0 conn(0x7f5bb006dac0 0x7f5bb013a340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T03:53:09.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:09.700+0000 7f5bb4818640 1 --2- 172.21.15.42:0/3118286035 >> v2:172.21.15.42:3300/0 conn(0x7f5bb00a1e20 0x7f5bb013c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:53:09.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:09.700+0000 7f5bb4818640 10 monclient(hunting): get_auth_request con 0x7f5bb00a1e20 auth_method 0 2024-09-16T03:53:09.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:09.700+0000 7f5bb4818640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:53:09.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:09.700+0000 7f5bb4818640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:53:09.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:09.700+0000 7f5bb4818640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:53:09.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:09.700+0000 7f5bb4818640 1 -- 172.21.15.42:0/3118286035 >> v2:172.21.15.42:3300/0 conn(0x7f5bb00a1e20 msgr2=0x7f5bb013c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T03:53:09.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:09.700+0000 7f5bb4818640 1 -- 172.21.15.42:0/3118286035 >> v2:172.21.15.42:3300/0 conn(0x7f5bb00a1e20 msgr2=0x7f5bb013c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:53:09.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:09.700+0000 7f5bb4818640 1 --2- 172.21.15.42:0/3118286035 >> v2:172.21.15.42:3300/0 conn(0x7f5bb00a1e20 0x7f5bb013c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:53:09.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:09.700+0000 7f5bb4818640 1 --2- 172.21.15.42:0/3118286035 >> v2:172.21.15.42:3300/0 conn(0x7f5bb00a1e20 0x7f5bb013c890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T03:53:10.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:10.504+0000 7f5ba77fe640 1 -- 172.21.15.42:0/3118286035 >> v2:172.21.15.154:3300/0 conn(0x7f5bb006dac0 msgr2=0x7f5bb013a340 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T03:53:10.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:10.504+0000 7f5ba77fe640 1 --2- 172.21.15.42:0/3118286035 >> v2:172.21.15.154:3300/0 conn(0x7f5bb006dac0 0x7f5bb013a340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T03:53:10.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:10.504+0000 7f5bb4818640 1 --2- 172.21.15.42:0/3118286035 >> v2:172.21.15.42:3300/0 conn(0x7f5bb00a1e20 0x7f5bb013c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:53:10.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:10.504+0000 7f5bb4818640 10 monclient(hunting): get_auth_request con 0x7f5bb00a1e20 auth_method 0 2024-09-16T03:53:10.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:10.504+0000 7f5bb4818640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:53:10.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:10.504+0000 7f5bb4818640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:53:10.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:10.504+0000 7f5bb4818640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:53:10.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:10.504+0000 7f5bb4818640 1 -- 172.21.15.42:0/3118286035 >> v2:172.21.15.42:3300/0 conn(0x7f5bb00a1e20 msgr2=0x7f5bb013c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T03:53:10.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:10.504+0000 7f5bb4818640 1 -- 172.21.15.42:0/3118286035 >> v2:172.21.15.42:3300/0 conn(0x7f5bb00a1e20 msgr2=0x7f5bb013c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:53:10.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:10.504+0000 7f5bb4818640 1 --2- 172.21.15.42:0/3118286035 >> v2:172.21.15.42:3300/0 conn(0x7f5bb00a1e20 0x7f5bb013c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:53:10.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:10.504+0000 7f5bb4818640 1 --2- 172.21.15.42:0/3118286035 >> v2:172.21.15.42:3300/0 conn(0x7f5bb00a1e20 0x7f5bb013c890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T03:53:12.099 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:12.092+0000 7f5bae7fc640 10 monclient(hunting): tick 2024-09-16T03:53:12.099 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:12.092+0000 7f5bae7fc640 10 monclient(hunting): _check_auth_tickets 2024-09-16T03:53:12.099 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:12.092+0000 7f5bae7fc640 1 monclient(hunting): continuing hunt 2024-09-16T03:53:12.099 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:12.092+0000 7f5bae7fc640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T03:53:12.099 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:12.092+0000 7f5bae7fc640 1 -- 172.21.15.42:0/3118286035 >> v2:172.21.15.154:3300/0 conn(0x7f5bb006dac0 msgr2=0x7f5bb013a340 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:53:12.099 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:12.092+0000 7f5bae7fc640 1 --2- 172.21.15.42:0/3118286035 >> v2:172.21.15.154:3300/0 conn(0x7f5bb006dac0 0x7f5bb013a340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:53:12.100 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:12.092+0000 7f5bae7fc640 1 -- 172.21.15.42:0/3118286035 >> v2:172.21.15.42:3300/0 conn(0x7f5bb00a1e20 msgr2=0x7f5bb013c890 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:53:12.100 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:12.092+0000 7f5bae7fc640 1 --2- 172.21.15.42:0/3118286035 >> v2:172.21.15.42:3300/0 conn(0x7f5bb00a1e20 0x7f5bb013c890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:53:12.100 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:12.092+0000 7f5bae7fc640 1 -- 172.21.15.42:0/3118286035 >> v1:172.21.15.42:6789/0 conn(0x7f5bb00a1330 legacy=0x7f5bb00a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:53:12.100 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:12.092+0000 7f5bae7fc640 10 monclient: _add_conns ranks=[2,1] 2024-09-16T03:53:12.100 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:12.092+0000 7f5bae7fc640 1 --2- 172.21.15.42:0/3118286035 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5b900036a0 0x7f5b90005af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:53:12.100 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:12.092+0000 7f5bae7fc640 10 monclient(hunting): picked mon.noname-b con 0x7f5b900036a0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:53:12.100 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:12.092+0000 7f5bae7fc640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5b90006180 addr v1:172.21.15.154:6789/0 2024-09-16T03:53:12.100 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:12.092+0000 7f5bae7fc640 1 -- 172.21.15.42:0/3118286035 --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5bb009fbd0 con 0x7f5b90006180 2024-09-16T03:53:12.100 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:12.092+0000 7f5bae7fc640 10 monclient(hunting): start opening mon connection 2024-09-16T03:53:12.100 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:12.092+0000 7f5bae7fc640 1 -- 172.21.15.42:0/3118286035 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f5bb009f610 con 0x7f5b900036a0 2024-09-16T03:53:12.100 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:12.092+0000 7f5bae7fc640 10 monclient(hunting): _renew_subs 2024-09-16T03:53:12.101 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:12.092+0000 7f5ba77fe640 1 -- 172.21.15.42:0/3118286035 >> v1:172.21.15.154:6789/0 conn(0x7f5b90006180 legacy=0x7f5b90008620 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.154:6789/0 2024-09-16T03:53:12.101 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:12.092+0000 7f5baffff640 1 --2- 172.21.15.42:0/3118286035 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5b900036a0 0x7f5b90005af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:53:12.101 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:12.092+0000 7f5baffff640 10 monclient(hunting): get_auth_request con 0x7f5b900036a0 auth_method 0 2024-09-16T03:53:12.101 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:12.092+0000 7f5baffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:53:12.101 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:12.092+0000 7f5baffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:53:12.101 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:12.092+0000 7f5baffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:53:12.101 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:12.092+0000 7f5baffff640 1 -- 172.21.15.42:0/3118286035 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5b900036a0 msgr2=0x7f5b90005af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T03:53:12.101 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:12.092+0000 7f5baffff640 1 -- 172.21.15.42:0/3118286035 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5b900036a0 msgr2=0x7f5b90005af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:53:12.101 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:12.092+0000 7f5baffff640 1 --2- 172.21.15.42:0/3118286035 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5b900036a0 0x7f5b90005af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:53:12.101 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:12.092+0000 7f5baffff640 1 --2- 172.21.15.42:0/3118286035 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5b900036a0 0x7f5b90005af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:53:12.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:12.296+0000 7f5ba77fe640 1 -- 172.21.15.42:0/3118286035 >> v1:172.21.15.154:6789/0 conn(0x7f5b90006180 legacy=0x7f5b90008620 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.154:6789/0 2024-09-16T03:53:12.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:12.296+0000 7f5baffff640 1 --2- 172.21.15.42:0/3118286035 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5b900036a0 0x7f5b90005af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:53:12.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:12.296+0000 7f5baffff640 10 monclient(hunting): get_auth_request con 0x7f5b900036a0 auth_method 0 2024-09-16T03:53:12.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:12.296+0000 7f5baffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:53:12.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:12.296+0000 7f5baffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:53:12.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:12.296+0000 7f5baffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:53:12.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:12.296+0000 7f5baffff640 1 -- 172.21.15.42:0/3118286035 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5b900036a0 msgr2=0x7f5b90005af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T03:53:12.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:12.296+0000 7f5baffff640 1 -- 172.21.15.42:0/3118286035 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5b900036a0 msgr2=0x7f5b90005af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:53:12.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:12.296+0000 7f5baffff640 1 --2- 172.21.15.42:0/3118286035 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5b900036a0 0x7f5b90005af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:53:12.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:12.296+0000 7f5baffff640 1 --2- 172.21.15.42:0/3118286035 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5b900036a0 0x7f5b90005af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T03:53:12.702 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:12.700+0000 7f5ba77fe640 1 -- 172.21.15.42:0/3118286035 >> v1:172.21.15.154:6789/0 conn(0x7f5b90006180 legacy=0x7f5b90008620 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.154:6789/0 2024-09-16T03:53:12.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:12.700+0000 7f5baffff640 1 --2- 172.21.15.42:0/3118286035 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5b900036a0 0x7f5b90005af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:53:12.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:12.700+0000 7f5baffff640 10 monclient(hunting): get_auth_request con 0x7f5b900036a0 auth_method 0 2024-09-16T03:53:12.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:12.700+0000 7f5baffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:53:12.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:12.700+0000 7f5baffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:53:12.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:12.700+0000 7f5baffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:53:12.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:12.700+0000 7f5baffff640 1 -- 172.21.15.42:0/3118286035 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5b900036a0 msgr2=0x7f5b90005af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T03:53:12.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:12.700+0000 7f5baffff640 1 -- 172.21.15.42:0/3118286035 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5b900036a0 msgr2=0x7f5b90005af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:53:12.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:12.700+0000 7f5baffff640 1 --2- 172.21.15.42:0/3118286035 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5b900036a0 0x7f5b90005af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:53:12.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:12.700+0000 7f5baffff640 1 --2- 172.21.15.42:0/3118286035 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5b900036a0 0x7f5b90005af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T03:53:13.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:13.504+0000 7f5ba77fe640 1 -- 172.21.15.42:0/3118286035 >> v1:172.21.15.154:6789/0 conn(0x7f5b90006180 legacy=0x7f5b90008620 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.154:6789/0 2024-09-16T03:53:13.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:13.504+0000 7f5baffff640 1 --2- 172.21.15.42:0/3118286035 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5b900036a0 0x7f5b90005af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:53:13.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:13.504+0000 7f5baffff640 10 monclient(hunting): get_auth_request con 0x7f5b900036a0 auth_method 0 2024-09-16T03:53:13.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:13.504+0000 7f5baffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:53:13.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:13.504+0000 7f5baffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:53:13.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:13.504+0000 7f5baffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:53:13.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:13.504+0000 7f5baffff640 1 -- 172.21.15.42:0/3118286035 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5b900036a0 msgr2=0x7f5b90005af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T03:53:13.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:13.504+0000 7f5baffff640 1 -- 172.21.15.42:0/3118286035 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5b900036a0 msgr2=0x7f5b90005af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:53:13.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:13.504+0000 7f5baffff640 1 --2- 172.21.15.42:0/3118286035 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5b900036a0 0x7f5b90005af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:53:13.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:13.504+0000 7f5baffff640 1 --2- 172.21.15.42:0/3118286035 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5b900036a0 0x7f5b90005af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T03:53:15.099 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.092+0000 7f5bae7fc640 10 monclient(hunting): tick 2024-09-16T03:53:15.100 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.092+0000 7f5bae7fc640 10 monclient(hunting): _check_auth_tickets 2024-09-16T03:53:15.100 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.092+0000 7f5bae7fc640 1 monclient(hunting): continuing hunt 2024-09-16T03:53:15.100 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.092+0000 7f5bae7fc640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T03:53:15.100 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.092+0000 7f5bae7fc640 1 -- 172.21.15.42:0/3118286035 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5b900036a0 msgr2=0x7f5b90005af0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:53:15.100 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.092+0000 7f5bae7fc640 1 --2- 172.21.15.42:0/3118286035 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5b900036a0 0x7f5b90005af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:53:15.100 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.092+0000 7f5bae7fc640 1 -- 172.21.15.42:0/3118286035 >> v1:172.21.15.154:6789/0 conn(0x7f5b90006180 legacy=0x7f5b90008620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:53:15.100 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.092+0000 7f5bae7fc640 10 monclient: _add_conns ranks=[1,0,3,4,2] 2024-09-16T03:53:15.100 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.092+0000 7f5bb4818640 1 -- 172.21.15.42:0/3118286035 reap_dead start 2024-09-16T03:53:15.100 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.092+0000 7f5bae7fc640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5b90009f10 addr v1:172.21.15.154:6789/0 2024-09-16T03:53:15.100 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.092+0000 7f5bae7fc640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5bb00a1330 addr v1:172.21.15.42:6789/0 2024-09-16T03:53:15.100 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.092+0000 7f5bae7fc640 1 --2- 172.21.15.42:0/3118286035 >> v2:172.21.15.42:3300/0 conn(0x7f5bb00a1e20 0x7f5b90010420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:53:15.100 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.092+0000 7f5bae7fc640 10 monclient(hunting): picked mon.noname-a con 0x7f5bb00a1e20 addr v2:172.21.15.42:3300/0 2024-09-16T03:53:15.101 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.092+0000 7f5bae7fc640 1 -- 172.21.15.42:0/3118286035 --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5b9000afc0 con 0x7f5bb00a1330 2024-09-16T03:53:15.101 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.092+0000 7f5bae7fc640 1 -- 172.21.15.42:0/3118286035 --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5bb009fbd0 con 0x7f5b90009f10 2024-09-16T03:53:15.101 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.092+0000 7f5bae7fc640 10 monclient(hunting): start opening mon connection 2024-09-16T03:53:15.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.092+0000 7f5bae7fc640 1 -- 172.21.15.42:0/3118286035 --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f5bb009f610 con 0x7f5bb00a1e20 2024-09-16T03:53:15.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.092+0000 7f5bae7fc640 10 monclient(hunting): _renew_subs 2024-09-16T03:53:15.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.092+0000 7f5baffff640 1 -- 172.21.15.42:0/3118286035 >> v1:172.21.15.154:6789/0 conn(0x7f5b90009f10 legacy=0x7f5b9000a2f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.154:6789/0 2024-09-16T03:53:15.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.092+0000 7f5bb4818640 1 --2- 172.21.15.42:0/3118286035 >> v2:172.21.15.42:3300/0 conn(0x7f5bb00a1e20 0x7f5b90010420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:53:15.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.092+0000 7f5bb4818640 10 monclient(hunting): get_auth_request con 0x7f5bb00a1e20 auth_method 0 2024-09-16T03:53:15.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.092+0000 7f5bb4818640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:53:15.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.092+0000 7f5bb4818640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:53:15.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.092+0000 7f5bb4818640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:53:15.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.092+0000 7f5bb4818640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:53:15.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.092+0000 7f5bb4818640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:53:15.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.092+0000 7f5bb4818640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:53:15.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.092+0000 7f5bb4818640 10 monclient(hunting): handle_auth_done global_id 824283 payload 274 2024-09-16T03:53:15.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.092+0000 7f5bb4818640 1 -- 172.21.15.42:0/3118286035 >> v1:172.21.15.154:6789/0 conn(0x7f5b90009f10 legacy=0x7f5b9000a2f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:53:15.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.092+0000 7f5bb4818640 1 -- 172.21.15.42:0/3118286035 >> v1:172.21.15.42:6789/0 conn(0x7f5bb00a1330 legacy=0x7f5b90008620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:53:15.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.092+0000 7f5bb4818640 10 monclient: _finish_hunting 0 2024-09-16T03:53:15.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.092+0000 7f5bb4818640 1 monclient: found mon.noname-a 2024-09-16T03:53:15.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.092+0000 7f5baf7fe640 1 -- 172.21.15.42:0/3118286035 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4184668783 0 0) 0x7f5b9000afc0 con 0x7f5bb00a1330 2024-09-16T03:53:15.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.092+0000 7f5bb4818640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:53:15.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.092+0000 7f5bb4818640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:53:15.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.092+0000 7f5bb4818640 1 -- 172.21.15.42:0/3118286035 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b9000ac30 con 0x7f5bb00a1e20 2024-09-16T03:53:15.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.092+0000 7f5baf7fe640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T03:53:15.105 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.100+0000 7f5bb4818640 1 --2- 172.21.15.42:0/3118286035 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5bb00a1e20 0x7f5b90010420 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f5bb006add0 tx=0x7f5ba801d330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec08f24fce987dba server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:53:15.106 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.100+0000 7f5baf7fe640 1 -- 172.21.15.42:0/3118286035 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ba80b3070 con 0x7f5bb00a1e20 2024-09-16T03:53:15.106 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.100+0000 7f5baf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:53:15.106 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.100+0000 7f5baf7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:53:15.106 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.100+0000 7f5baf7fe640 10 monclient: dump: 2024-09-16T03:53:15.106 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:53:15.106 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:53:15.106 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:15.106 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:15.106 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:53:15.107 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:53:15.107 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:53:15.107 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:53:15.107 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:53:15.107 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:53:15.107 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.100+0000 7f5baf7fe640 10 monclient: _finish_auth 0 2024-09-16T03:53:15.107 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.100+0000 7f5baf7fe640 10 monclient: _check_auth_tickets 2024-09-16T03:53:15.107 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.100+0000 7f5baf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:53:15.107 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.100+0000 7f5baf7fe640 1 -- 172.21.15.42:0/3118286035 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ba80ae410 con 0x7f5bb00a1e20 2024-09-16T03:53:15.107 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.100+0000 7f5baf7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:53:15.107 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.100+0000 7f5baf7fe640 1 -- 172.21.15.42:0/3118286035 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ba80ae600 con 0x7f5bb00a1e20 2024-09-16T03:53:15.107 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.100+0000 7f5bb627b640 5 monclient: authenticate success, global_id 824283 2024-09-16T03:53:15.108 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.100+0000 7f5bb627b640 10 monclient: get_monmap_and_config success 2024-09-16T03:53:15.108 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.100+0000 7f5bb627b640 4 set_mon_vals no callback set 2024-09-16T03:53:15.108 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.100+0000 7f5baf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:53:15.108 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.100+0000 7f5baf7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:53:15.108 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.100+0000 7f5baf7fe640 10 monclient: dump: 2024-09-16T03:53:15.108 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:53:15.108 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:53:15.108 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:15.108 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:15.108 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:53:15.109 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:53:15.109 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:53:15.109 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:53:15.109 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:53:15.109 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:53:15.115 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.100+0000 7f5bb627b640 10 monclient: shutdown 2024-09-16T03:53:15.115 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.100+0000 7f5bb627b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:53:15.115 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.100+0000 7f5bb627b640 1 -- 172.21.15.42:0/3118286035 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5bb00a1e20 msgr2=0x7f5b90010420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:53:15.115 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.100+0000 7f5bb627b640 1 --2- 172.21.15.42:0/3118286035 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5bb00a1e20 0x7f5b90010420 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f5bb006add0 tx=0x7f5ba801d330 comp rx=0 tx=0).stop 2024-09-16T03:53:15.115 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.100+0000 7f5bb627b640 1 -- 172.21.15.42:0/3118286035 shutdown_connections 2024-09-16T03:53:15.115 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.100+0000 7f5bb627b640 1 --2- 172.21.15.42:0/3118286035 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5bb00a1e20 0x7f5b90010420 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:53:15.115 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.100+0000 7f5bb627b640 1 -- 172.21.15.42:0/3118286035 >> 172.21.15.42:0/3118286035 conn(0x7f5bb00a2420 msgr2=0x7f5bb00a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:53:15.116 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.100+0000 7f5bb627b640 1 -- 172.21.15.42:0/3118286035 shutdown_connections 2024-09-16T03:53:15.116 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.100+0000 7f5bb627b640 1 -- 172.21.15.42:0/3118286035 wait complete. 2024-09-16T03:53:15.116 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.100+0000 7f5bb627b640 10 monclient: build_initial_monmap 2024-09-16T03:53:15.116 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.100+0000 7f5bb627b640 10 monclient: monmap: 2024-09-16T03:53:15.116 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:53:15.116 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:53:15.116 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:53:15.116 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:53:15.116 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:53:15.116 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:53:15.116 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:53:15.116 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:53:15.116 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:53:15.117 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:53:15.117 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.100+0000 7f5bb627b640 1 librados: starting msgr at 2024-09-16T03:53:15.117 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.100+0000 7f5bb627b640 1 librados: starting objecter 2024-09-16T03:53:15.117 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.104+0000 7f5bb627b640 1 Processor -- start 2024-09-16T03:53:15.117 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.104+0000 7f5bb627b640 1 -- start start 2024-09-16T03:53:15.117 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.104+0000 7f5bb627b640 1 librados: setting wanted keys 2024-09-16T03:53:15.117 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.104+0000 7f5bb627b640 1 librados: calling monclient init 2024-09-16T03:53:15.117 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.104+0000 7f5bb627b640 10 monclient: init 2024-09-16T03:53:15.117 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.104+0000 7f5bb627b640 10 monclient: _reopen_session rank -1 2024-09-16T03:53:15.117 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.104+0000 7f5bb627b640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:53:15.117 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.104+0000 7f5bb627b640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5b90009f10 0x7f5bb01cf450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:53:15.118 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.104+0000 7f5bb627b640 10 monclient(hunting): picked mon.noname-c con 0x7f5b90009f10 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:53:15.118 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.104+0000 7f5bb627b640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5bb00a1330 0x7f5bb01cf990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:53:15.118 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.104+0000 7f5bb627b640 10 monclient(hunting): picked mon.noname-a con 0x7f5bb00a1330 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:53:15.118 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.104+0000 7f5bb627b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5bb00a1e20 0x7f5bb009cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:53:15.118 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.104+0000 7f5bb627b640 10 monclient(hunting): picked mon.noname-b con 0x7f5bb00a1e20 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:53:15.118 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.104+0000 7f5bb627b640 10 monclient(hunting): start opening mon connection 2024-09-16T03:53:15.118 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.104+0000 7f5bb627b640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f5bb0135910 con 0x7f5bb00a1330 2024-09-16T03:53:15.118 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.104+0000 7f5bb627b640 10 monclient(hunting): start opening mon connection 2024-09-16T03:53:15.118 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.104+0000 7f5bb627b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f5bb009ef80 con 0x7f5bb00a1e20 2024-09-16T03:53:15.118 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.104+0000 7f5bb627b640 10 monclient(hunting): start opening mon connection 2024-09-16T03:53:15.118 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.104+0000 7f5bb627b640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f5bb0135a90 con 0x7f5b90009f10 2024-09-16T03:53:15.118 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.104+0000 7f5bb627b640 10 monclient(hunting): _renew_subs 2024-09-16T03:53:15.118 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.104+0000 7f5bb627b640 10 monclient(hunting): authenticate will time out at 1642.077020s 2024-09-16T03:53:15.119 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.104+0000 7f5ba77fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5bb00a1330 0x7f5bb01cf990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:53:15.119 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.104+0000 7f5ba77fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5bb00a1330 0x7f5bb01cf990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:45696/0 (socket says 172.21.15.42:45696) 2024-09-16T03:53:15.119 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.104+0000 7f5ba77fe640 1 -- 172.21.15.42:0/225880481 learned_addr learned my addr 172.21.15.42:0/225880481 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:53:15.119 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.104+0000 7f5ba77fe640 10 monclient(hunting): get_auth_request con 0x7f5bb00a1330 auth_method 0 2024-09-16T03:53:15.119 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.104+0000 7f5ba77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:53:15.119 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.104+0000 7f5ba77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:53:15.119 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.104+0000 7f5ba77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:53:15.119 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.104+0000 7f5baffff640 1 --2- 172.21.15.42:0/225880481 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5b90009f10 0x7f5bb01cf450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:53:15.119 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.104+0000 7f5bb4818640 1 -- 172.21.15.42:0/225880481 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5bb00a1e20 msgr2=0x7f5bb009cb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T03:53:15.119 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.104+0000 7f5bb4818640 1 --2- 172.21.15.42:0/225880481 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5bb00a1e20 0x7f5bb009cb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:53:15.119 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.104+0000 7f5baffff640 10 monclient(hunting): get_auth_request con 0x7f5b90009f10 auth_method 0 2024-09-16T03:53:15.120 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.104+0000 7f5baffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:53:15.120 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.104+0000 7f5baffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:53:15.120 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.104+0000 7f5baffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:53:15.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.104+0000 7f5ba77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:53:15.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.104+0000 7f5ba77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:53:15.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.104+0000 7f5ba77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:53:15.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.104+0000 7f5baffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:53:15.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.104+0000 7f5baffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:53:15.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.104+0000 7f5baffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:53:15.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.104+0000 7f5baffff640 10 monclient(hunting): handle_auth_done global_id 834173 payload 824 2024-09-16T03:53:15.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.104+0000 7f5baffff640 1 -- 172.21.15.42:0/225880481 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5bb00a1e20 msgr2=0x7f5bb009cb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:53:15.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.104+0000 7f5baffff640 1 --2- 172.21.15.42:0/225880481 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5bb00a1e20 0x7f5bb009cb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:53:15.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.104+0000 7f5baffff640 1 -- 172.21.15.42:0/225880481 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5bb00a1330 msgr2=0x7f5bb01cf990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:53:15.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.104+0000 7f5baffff640 1 --2- 172.21.15.42:0/225880481 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5bb00a1330 0x7f5bb01cf990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:53:15.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.104+0000 7f5baffff640 10 monclient: _finish_hunting 0 2024-09-16T03:53:15.125 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.104+0000 7f5baffff640 1 monclient: found mon.noname-c 2024-09-16T03:53:15.125 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.104+0000 7f5baffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:53:15.125 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.104+0000 7f5baffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:53:15.125 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.104+0000 7f5baffff640 1 -- 172.21.15.42:0/225880481 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5bb01288f0 con 0x7f5b90009f10 2024-09-16T03:53:15.125 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.104+0000 7f5ba77fe640 1 --2- 172.21.15.42:0/225880481 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5bb00a1330 0x7f5bb01cf990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:53:15.125 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.104+0000 7f5baffff640 1 --2- 172.21.15.42:0/225880481 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5b90009f10 0x7f5bb01cf450 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f5ba0007d30 tx=0x7f5ba0001640 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:53:15.125 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.104+0000 7f5bad7fa640 10 client.?.objecter ms_handle_connect 0x7f5b90009f10 2024-09-16T03:53:15.125 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.104+0000 7f5bad7fa640 10 client.?.objecter resend_mon_ops 2024-09-16T03:53:15.125 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.104+0000 7f5bad7fa640 1 -- 172.21.15.42:0/225880481 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ba0007780 con 0x7f5b90009f10 2024-09-16T03:53:15.125 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.104+0000 7f5bad7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:53:15.126 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.104+0000 7f5bad7fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:53:15.126 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.104+0000 7f5bad7fa640 10 monclient: dump: 2024-09-16T03:53:15.126 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:53:15.126 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:53:15.126 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:15.126 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:15.126 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:53:15.126 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:53:15.126 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:53:15.126 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:53:15.127 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:53:15.127 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:53:15.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.104+0000 7f5bad7fa640 10 monclient: _finish_auth 0 2024-09-16T03:53:15.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.104+0000 7f5bad7fa640 10 monclient: _check_auth_tickets 2024-09-16T03:53:15.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.104+0000 7f5bad7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:53:15.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.104+0000 7f5bad7fa640 1 -- 172.21.15.42:0/225880481 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ba0004a00 con 0x7f5b90009f10 2024-09-16T03:53:15.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.104+0000 7f5bad7fa640 10 monclient: handle_config config(0 keys) 2024-09-16T03:53:15.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.104+0000 7f5bad7fa640 1 -- 172.21.15.42:0/225880481 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ba0004dd0 con 0x7f5b90009f10 2024-09-16T03:53:15.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.104+0000 7f5bb627b640 5 monclient: authenticate success, global_id 834173 2024-09-16T03:53:15.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.104+0000 7f5bb627b640 10 monclient: _renew_subs 2024-09-16T03:53:15.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.104+0000 7f5bb627b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:53:15.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.104+0000 7f5bb627b640 1 -- 172.21.15.42:0/225880481 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5bb0071140 con 0x7f5b90009f10 2024-09-16T03:53:15.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.104+0000 7f5bad7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:53:15.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.104+0000 7f5baeffd640 4 set_mon_vals no callback set 2024-09-16T03:53:15.128 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.104+0000 7f5bad7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:53:15.128 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.104+0000 7f5bad7fa640 10 monclient: dump: 2024-09-16T03:53:15.128 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:53:15.128 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:53:15.128 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:15.128 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:15.128 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:53:15.128 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:53:15.128 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:53:15.128 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:53:15.128 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:53:15.128 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:53:15.128 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.104+0000 7f5bb627b640 10 client.834173.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:53:15.128 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.104+0000 7f5bb627b640 10 monclient: _renew_subs 2024-09-16T03:53:15.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.104+0000 7f5bb627b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:53:15.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.104+0000 7f5bb627b640 1 -- 172.21.15.42:0/225880481 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f5bb0128610 con 0x7f5b90009f10 2024-09-16T03:53:15.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.104+0000 7f5bb627b640 1 librados: init done 2024-09-16T03:53:15.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.104+0000 7f5bb627b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:53:15.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.104+0000 7f5bb627b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:53:15.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.104+0000 7f5bb627b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:53:15.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.104+0000 7f5bb627b640 1 -- 172.21.15.42:0/225880481 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5bb0135e20 con 0x7f5b90009f10 2024-09-16T03:53:15.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.104+0000 7f5bad7fa640 1 -- 172.21.15.42:0/225880481 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f5ba0005050 con 0x7f5b90009f10 2024-09-16T03:53:15.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.104+0000 7f5bad7fa640 1 --2- 172.21.15.42:0/225880481 >> v2:172.21.15.154:6812/254845342 conn(0x7f5b90038ae0 0x7f5b9003afa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:53:15.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.104+0000 7f5bad7fa640 1 -- 172.21.15.42:0/225880481 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(346..346 src has 286..346) ==== 5356+0+0 (secure 0 0 0) 0x7f5ba0049760 con 0x7f5b90009f10 2024-09-16T03:53:15.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.104+0000 7f5bad7fa640 10 client.834173.objecter ms_dispatch 0x7f5bb0000ba0 osd_map(346..346 src has 286..346) 2024-09-16T03:53:15.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.104+0000 7f5bad7fa640 3 client.834173.objecter handle_osd_map got epochs [346,346] > 0 2024-09-16T03:53:15.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.104+0000 7f5bad7fa640 3 client.834173.objecter handle_osd_map decoding full epoch 346 2024-09-16T03:53:15.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.108+0000 7f5bad7fa640 20 client.834173.objecter dump_active .. 0 homeless 2024-09-16T03:53:15.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.108+0000 7f5ba77fe640 1 --2- 172.21.15.42:0/225880481 >> v2:172.21.15.154:6812/254845342 conn(0x7f5b90038ae0 0x7f5b9003afa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:53:15.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.108+0000 7f5ba77fe640 10 monclient: get_auth_request con 0x7f5b90038ae0 auth_method 0 2024-09-16T03:53:15.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.108+0000 7f5ba77fe640 1 --2- 172.21.15.42:0/225880481 >> v2:172.21.15.154:6812/254845342 conn(0x7f5b90038ae0 0x7f5b9003afa0 secure :-1 s=READY pgs=3593 cs=0 l=1 rev1=1 crypto rx=0x7f5b98003850 tx=0x7f5b98009290 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:53:15.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.108+0000 7f5bad7fa640 10 client.834173.objecter ms_handle_connect 0x7f5b90038ae0 2024-09-16T03:53:15.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.112+0000 7f5bad7fa640 1 -- 172.21.15.42:0/225880481 <== mon.2 v2:172.21.15.42:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f5ba00163c0 con 0x7f5b90009f10 2024-09-16T03:53:15.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.112+0000 7f5bad7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:53:15.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.112+0000 7f5bad7fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:53:15.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.340+0000 7f5bb627b640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:53:15.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.340+0000 7f5bb627b640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:53:15.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.340+0000 7f5bb627b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:53:15.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.340+0000 7f5bb627b640 1 -- 172.21.15.42:0/225880481 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f5bb0069040 con 0x7f5b90009f10 2024-09-16T03:53:15.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.340+0000 7f5bad7fa640 1 -- 172.21.15.42:0/225880481 <== mon.2 v2:172.21.15.42:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1554 (secure 0 0 0) 0x7f5ba0005520 con 0x7f5b90009f10 2024-09-16T03:53:15.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.340+0000 7f5bad7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:53:15.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.340+0000 7f5bad7fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:53:15.347 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":98,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:53:15.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.340+0000 7f5bb627b640 10 librados: watch_flush enter 2024-09-16T03:53:15.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.340+0000 7f5bb627b640 10 librados: watch_flush exit 2024-09-16T03:53:15.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.340+0000 7f5bb627b640 20 client.834173.objecter shutdown clearing up homeless session... 2024-09-16T03:53:15.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.340+0000 7f5bb627b640 10 client.834173.objecter successfully canceled tick 2024-09-16T03:53:15.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.340+0000 7f5bb627b640 1 -- 172.21.15.42:0/225880481 >> v2:172.21.15.154:6812/254845342 conn(0x7f5b90038ae0 msgr2=0x7f5b9003afa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:53:15.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.340+0000 7f5bb627b640 1 --2- 172.21.15.42:0/225880481 >> v2:172.21.15.154:6812/254845342 conn(0x7f5b90038ae0 0x7f5b9003afa0 secure :-1 s=READY pgs=3593 cs=0 l=1 rev1=1 crypto rx=0x7f5b98003850 tx=0x7f5b98009290 comp rx=0 tx=0).stop 2024-09-16T03:53:15.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.340+0000 7f5bb627b640 10 monclient: shutdown 2024-09-16T03:53:15.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.340+0000 7f5bb627b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:53:15.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.340+0000 7f5bb627b640 1 -- 172.21.15.42:0/225880481 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5b90009f10 msgr2=0x7f5bb01cf450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:53:15.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.340+0000 7f5bb627b640 1 --2- 172.21.15.42:0/225880481 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5b90009f10 0x7f5bb01cf450 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f5ba0007d30 tx=0x7f5ba0001640 comp rx=0 tx=0).stop 2024-09-16T03:53:15.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.340+0000 7f5bb627b640 1 -- 172.21.15.42:0/225880481 shutdown_connections 2024-09-16T03:53:15.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.340+0000 7f5bb627b640 1 --2- 172.21.15.42:0/225880481 >> v2:172.21.15.154:6812/254845342 conn(0x7f5b90038ae0 0x7f5b9003afa0 unknown :-1 s=CLOSED pgs=3593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:53:15.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.340+0000 7f5bb627b640 1 --2- 172.21.15.42:0/225880481 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5bb00a1e20 0x7f5bb009cb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:53:15.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.340+0000 7f5bb627b640 1 --2- 172.21.15.42:0/225880481 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5bb00a1330 0x7f5bb01cf990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:53:15.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.340+0000 7f5bb627b640 1 --2- 172.21.15.42:0/225880481 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5b90009f10 0x7f5bb01cf450 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:53:15.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.340+0000 7f5bb627b640 1 -- 172.21.15.42:0/225880481 >> 172.21.15.42:0/225880481 conn(0x7f5bb00a2420 msgr2=0x7f5bb0071690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:53:15.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.340+0000 7f5bb627b640 1 -- 172.21.15.42:0/225880481 shutdown_connections 2024-09-16T03:53:15.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.340+0000 7f5bb627b640 1 -- 172.21.15.42:0/225880481 wait complete. 2024-09-16T03:53:15.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.344+0000 7f5bb627b640 1 librados: shutdown 2024-09-16T03:53:15.372 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T03:53:15.372 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-16T03:53:15.372 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T03:53:15.494 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.488+0000 7f9cc8794640 10 monclient: get_monmap_and_config 2024-09-16T03:53:15.494 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.488+0000 7f9cc8794640 10 monclient: build_initial_monmap 2024-09-16T03:53:15.494 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.488+0000 7f9cc8794640 10 monclient: monmap: 2024-09-16T03:53:15.494 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:53:15.494 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:53:15.495 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:53:15.493138+0000 2024-09-16T03:53:15.495 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:53:15.493138+0000 2024-09-16T03:53:15.495 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:53:15.495 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:53:15.495 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:53:15.495 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:53:15.495 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:53:15.495 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:53:15.495 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:53:15.495 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:53:15.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.488+0000 7f9cc8794640 1 Processor -- start 2024-09-16T03:53:15.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.488+0000 7f9cc8794640 1 -- start start 2024-09-16T03:53:15.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.488+0000 7f9cc8794640 10 monclient: init 2024-09-16T03:53:15.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.488+0000 7f9cc8794640 10 monclient: _reopen_session rank -1 2024-09-16T03:53:15.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.488+0000 7f9cc8794640 10 monclient: _add_conns ranks=[0,4,2,3,1] 2024-09-16T03:53:15.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.488+0000 7f9cc8794640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9cc00a39b0 addr v1:172.21.15.42:6789/0 2024-09-16T03:53:15.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.488+0000 7f9cc8794640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f9cc00a3d90 0x7f9cc013c380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:53:15.498 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.488+0000 7f9cc8794640 10 monclient(hunting): picked mon.noname-c con 0x7f9cc00a3d90 addr v2:172.21.15.154:3300/0 2024-09-16T03:53:15.498 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.488+0000 7f9cc8794640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9cc013c8c0 0x7f9cc013ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:53:15.498 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.488+0000 7f9cc8794640 10 monclient(hunting): picked mon.noname-b con 0x7f9cc013c8c0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:53:15.498 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.488+0000 7f9cc8794640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9cc0134290 con 0x7f9cc00a39b0 2024-09-16T03:53:15.498 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.488+0000 7f9cc8794640 10 monclient(hunting): start opening mon connection 2024-09-16T03:53:15.498 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.488+0000 7f9cc8794640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f9cc009c140 con 0x7f9cc00a3d90 2024-09-16T03:53:15.498 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.488+0000 7f9cc8794640 10 monclient(hunting): start opening mon connection 2024-09-16T03:53:15.498 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.488+0000 7f9cc8794640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f9cc009c420 con 0x7f9cc013c8c0 2024-09-16T03:53:15.498 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.488+0000 7f9cc8794640 10 monclient(hunting): _renew_subs 2024-09-16T03:53:15.498 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.488+0000 7f9cc8794640 10 monclient(hunting): authenticate will time out at 1642.463309s 2024-09-16T03:53:15.498 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.488+0000 7f9cc6530640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f9cc00a39b0 0x7f9cc0133b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:45100/0 (socket says 172.21.15.42:45100) 2024-09-16T03:53:15.498 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.488+0000 7f9cc6530640 1 -- 172.21.15.42:0/4207016423 learned_addr learned my addr 172.21.15.42:0/4207016423 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:53:15.498 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.488+0000 7f9cc5d2f640 1 -- 172.21.15.42:0/4207016423 >> v2:172.21.15.154:3300/0 conn(0x7f9cc00a3d90 msgr2=0x7f9cc013c380 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T03:53:15.498 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.488+0000 7f9cc6d31640 1 --2- 172.21.15.42:0/4207016423 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9cc013c8c0 0x7f9cc013ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:53:15.499 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.488+0000 7f9cc5d2f640 1 --2- 172.21.15.42:0/4207016423 >> v2:172.21.15.154:3300/0 conn(0x7f9cc00a3d90 0x7f9cc013c380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:53:15.499 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.488+0000 7f9cc6d31640 10 monclient(hunting): get_auth_request con 0x7f9cc013c8c0 auth_method 0 2024-09-16T03:53:15.499 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.488+0000 7f9cc6d31640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:53:15.499 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.488+0000 7f9cc6d31640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:53:15.499 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.488+0000 7f9cc6d31640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:53:15.499 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.488+0000 7f9cc6d31640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:53:15.499 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.488+0000 7f9cc6d31640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:53:15.499 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.488+0000 7f9cc6d31640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:53:15.499 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.488+0000 7f9cc6d31640 10 monclient(hunting): handle_auth_done global_id 834176 payload 274 2024-09-16T03:53:15.499 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.488+0000 7f9cc6d31640 1 -- 172.21.15.42:0/4207016423 >> v2:172.21.15.154:3300/0 conn(0x7f9cc00a3d90 msgr2=0x7f9cc013c380 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:53:15.499 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.488+0000 7f9cc6d31640 1 --2- 172.21.15.42:0/4207016423 >> v2:172.21.15.154:3300/0 conn(0x7f9cc00a3d90 0x7f9cc013c380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:53:15.499 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.488+0000 7f9cc6d31640 1 -- 172.21.15.42:0/4207016423 >> v1:172.21.15.42:6789/0 conn(0x7f9cc00a39b0 legacy=0x7f9cc0133b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:53:15.499 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.488+0000 7f9cc6d31640 10 monclient: _finish_hunting 0 2024-09-16T03:53:15.499 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.488+0000 7f9cc6d31640 1 monclient: found mon.noname-b 2024-09-16T03:53:15.500 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.488+0000 7f9cc6d31640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:53:15.500 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.488+0000 7f9cc6d31640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T03:53:15.500 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.488+0000 7f9cc6d31640 1 -- 172.21.15.42:0/4207016423 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9cc006dd60 con 0x7f9cc013c8c0 2024-09-16T03:53:15.500 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.488+0000 7f9cc552e640 1 -- 172.21.15.42:0/4207016423 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2514451552 0 0) 0x7f9cc0134290 con 0x7f9cc00a39b0 2024-09-16T03:53:15.500 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.488+0000 7f9cc552e640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T03:53:15.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.492+0000 7f9cc6d31640 1 --2- 172.21.15.42:0/4207016423 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9cc013c8c0 0x7f9cc013ecb0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f9cb8009d90 tx=0x7f9cb801c850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=429a004a8eab7ff3 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:53:15.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.492+0000 7f9cc552e640 1 -- 172.21.15.42:0/4207016423 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cb80b2070 con 0x7f9cc013c8c0 2024-09-16T03:53:15.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.492+0000 7f9cc552e640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:53:15.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.492+0000 7f9cc552e640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:53:15.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.492+0000 7f9cc552e640 10 monclient: dump: 2024-09-16T03:53:15.507 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:53:15.507 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:53:15.507 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:15.507 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:15.507 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:53:15.507 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:53:15.507 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:53:15.507 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:53:15.507 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:53:15.508 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:53:15.508 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.492+0000 7f9cc552e640 10 monclient: _finish_auth 0 2024-09-16T03:53:15.508 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.492+0000 7f9cc552e640 10 monclient: _check_auth_tickets 2024-09-16T03:53:15.508 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.492+0000 7f9cc552e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:53:15.508 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.492+0000 7f9cc552e640 1 -- 172.21.15.42:0/4207016423 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9cb80a5710 con 0x7f9cc013c8c0 2024-09-16T03:53:15.508 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.492+0000 7f9cc552e640 10 monclient: handle_config config(0 keys) 2024-09-16T03:53:15.508 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.492+0000 7f9cc552e640 1 -- 172.21.15.42:0/4207016423 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cb80a5ae0 con 0x7f9cc013c8c0 2024-09-16T03:53:15.508 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.492+0000 7f9cc8794640 5 monclient: authenticate success, global_id 834176 2024-09-16T03:53:15.508 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.492+0000 7f9cc8794640 10 monclient: get_monmap_and_config success 2024-09-16T03:53:15.508 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.492+0000 7f9cc8794640 4 set_mon_vals no callback set 2024-09-16T03:53:15.508 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.492+0000 7f9cc552e640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:53:15.508 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.492+0000 7f9cc552e640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:53:15.508 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.492+0000 7f9cc552e640 10 monclient: dump: 2024-09-16T03:53:15.508 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:53:15.509 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:53:15.509 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:15.509 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:15.509 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:53:15.509 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:53:15.509 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:53:15.509 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:53:15.509 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:53:15.509 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:53:15.509 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.492+0000 7f9cc8794640 10 monclient: shutdown 2024-09-16T03:53:15.509 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.492+0000 7f9cc8794640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:53:15.509 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.492+0000 7f9cc8794640 1 -- 172.21.15.42:0/4207016423 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9cc013c8c0 msgr2=0x7f9cc013ecb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:53:15.509 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.492+0000 7f9cc8794640 1 --2- 172.21.15.42:0/4207016423 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9cc013c8c0 0x7f9cc013ecb0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f9cb8009d90 tx=0x7f9cb801c850 comp rx=0 tx=0).stop 2024-09-16T03:53:15.509 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.492+0000 7f9cc8794640 1 -- 172.21.15.42:0/4207016423 shutdown_connections 2024-09-16T03:53:15.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.492+0000 7f9cc8794640 1 --2- 172.21.15.42:0/4207016423 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9cc013c8c0 0x7f9cc013ecb0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:53:15.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.492+0000 7f9cc8794640 1 --2- 172.21.15.42:0/4207016423 >> v2:172.21.15.154:3300/0 conn(0x7f9cc00a3d90 0x7f9cc013c380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:53:15.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.492+0000 7f9cc8794640 1 -- 172.21.15.42:0/4207016423 >> 172.21.15.42:0/4207016423 conn(0x7f9cc00a9fb0 msgr2=0x7f9cc00aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:53:15.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.492+0000 7f9cc8794640 1 -- 172.21.15.42:0/4207016423 shutdown_connections 2024-09-16T03:53:15.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.492+0000 7f9cc8794640 1 -- 172.21.15.42:0/4207016423 wait complete. 2024-09-16T03:53:15.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.492+0000 7f9cc8794640 10 monclient: build_initial_monmap 2024-09-16T03:53:15.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.492+0000 7f9cc8794640 10 monclient: monmap: 2024-09-16T03:53:15.510 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:53:15.510 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:53:15.510 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:53:15.511 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:53:15.511 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:53:15.511 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:53:15.511 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:53:15.511 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:53:15.511 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:53:15.511 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:53:15.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.492+0000 7f9cc8794640 1 librados: starting msgr at 2024-09-16T03:53:15.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.492+0000 7f9cc8794640 1 librados: starting objecter 2024-09-16T03:53:15.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.492+0000 7f9cc8794640 1 Processor -- start 2024-09-16T03:53:15.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.492+0000 7f9cc8794640 1 -- start start 2024-09-16T03:53:15.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.492+0000 7f9cc8794640 1 librados: setting wanted keys 2024-09-16T03:53:15.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.492+0000 7f9cc8794640 1 librados: calling monclient init 2024-09-16T03:53:15.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.492+0000 7f9cc8794640 10 monclient: init 2024-09-16T03:53:15.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.496+0000 7f9cc8794640 10 monclient: _reopen_session rank -1 2024-09-16T03:53:15.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.496+0000 7f9cc8794640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T03:53:15.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.496+0000 7f9cc8794640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9cc00a39b0 0x7f9cc0130920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:53:15.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.496+0000 7f9cc8794640 10 monclient(hunting): picked mon.noname-a con 0x7f9cc00a39b0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:53:15.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.496+0000 7f9cc8794640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9cc00a3d90 0x7f9cc0130e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:53:15.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.496+0000 7f9cc8794640 10 monclient(hunting): picked mon.noname-b con 0x7f9cc00a3d90 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:53:15.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.496+0000 7f9cc8794640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9cc013c8c0 0x7f9cc0139ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:53:15.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.496+0000 7f9cc8794640 10 monclient(hunting): picked mon.noname-c con 0x7f9cc013c8c0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:53:15.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.496+0000 7f9cc8794640 10 monclient(hunting): start opening mon connection 2024-09-16T03:53:15.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.496+0000 7f9cc8794640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f9cc0141da0 con 0x7f9cc00a39b0 2024-09-16T03:53:15.513 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.496+0000 7f9cc8794640 10 monclient(hunting): start opening mon connection 2024-09-16T03:53:15.513 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.496+0000 7f9cc8794640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f9cc0141c20 con 0x7f9cc00a3d90 2024-09-16T03:53:15.513 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.496+0000 7f9cc8794640 10 monclient(hunting): start opening mon connection 2024-09-16T03:53:15.513 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.496+0000 7f9cc8794640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f9cc0141f20 con 0x7f9cc013c8c0 2024-09-16T03:53:15.513 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.496+0000 7f9cc8794640 10 monclient(hunting): _renew_subs 2024-09-16T03:53:15.513 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.496+0000 7f9cc8794640 10 monclient(hunting): authenticate will time out at 1642.468493s 2024-09-16T03:53:15.513 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.496+0000 7f9cc6530640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9cc00a39b0 0x7f9cc0130920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:53:15.513 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.496+0000 7f9cc6530640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9cc00a39b0 0x7f9cc0130920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:45698/0 (socket says 172.21.15.42:45698) 2024-09-16T03:53:15.513 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.496+0000 7f9cc6530640 1 -- 172.21.15.42:0/3679600131 learned_addr learned my addr 172.21.15.42:0/3679600131 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:53:15.513 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.496+0000 7f9cc6530640 10 monclient(hunting): get_auth_request con 0x7f9cc00a39b0 auth_method 0 2024-09-16T03:53:15.513 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.496+0000 7f9cc6530640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:53:15.513 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.496+0000 7f9cc6530640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:53:15.513 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.496+0000 7f9cc6530640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:53:15.513 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.496+0000 7f9cc5d2f640 1 -- 172.21.15.42:0/3679600131 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9cc00a3d90 msgr2=0x7f9cc0130e60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T03:53:15.514 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.496+0000 7f9cc5d2f640 1 --2- 172.21.15.42:0/3679600131 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9cc00a3d90 0x7f9cc0130e60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:53:15.514 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.496+0000 7f9cc6d31640 1 --2- 172.21.15.42:0/3679600131 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9cc013c8c0 0x7f9cc0139ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:53:15.514 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.496+0000 7f9cc6530640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:53:15.514 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.496+0000 7f9cc6530640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:53:15.514 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.496+0000 7f9cc6530640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:53:15.514 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.496+0000 7f9cc6d31640 10 monclient(hunting): get_auth_request con 0x7f9cc013c8c0 auth_method 0 2024-09-16T03:53:15.514 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.496+0000 7f9cc6d31640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:53:15.514 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.496+0000 7f9cc6d31640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:53:15.514 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.496+0000 7f9cc6d31640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:53:15.514 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.496+0000 7f9cc6d31640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:53:15.514 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.496+0000 7f9cc6d31640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:53:15.514 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.496+0000 7f9cc6d31640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:53:15.514 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.496+0000 7f9cc6530640 10 monclient(hunting): handle_auth_done global_id 824295 payload 824 2024-09-16T03:53:15.514 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.496+0000 7f9cc6530640 1 -- 172.21.15.42:0/3679600131 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9cc013c8c0 msgr2=0x7f9cc0139ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:53:15.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.496+0000 7f9cc6530640 1 --2- 172.21.15.42:0/3679600131 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9cc013c8c0 0x7f9cc0139ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:53:15.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.496+0000 7f9cc6530640 1 -- 172.21.15.42:0/3679600131 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9cc00a3d90 msgr2=0x7f9cc0130e60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:53:15.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.496+0000 7f9cc6530640 1 --2- 172.21.15.42:0/3679600131 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9cc00a3d90 0x7f9cc0130e60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:53:15.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.496+0000 7f9cc6530640 10 monclient: _finish_hunting 0 2024-09-16T03:53:15.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.496+0000 7f9cc6530640 1 monclient: found mon.noname-a 2024-09-16T03:53:15.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.496+0000 7f9cc6530640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:53:15.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.496+0000 7f9cc6530640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:53:15.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.496+0000 7f9cc6530640 1 -- 172.21.15.42:0/3679600131 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9cc009e000 con 0x7f9cc00a39b0 2024-09-16T03:53:15.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.496+0000 7f9cc6530640 1 --2- 172.21.15.42:0/3679600131 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9cc00a39b0 0x7f9cc0130920 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f9cbc00c520 tx=0x7f9cbc00c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:53:15.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.496+0000 7f9cb6ffd640 10 client.?.objecter ms_handle_connect 0x7f9cc00a39b0 2024-09-16T03:53:15.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.496+0000 7f9cb6ffd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:53:15.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.496+0000 7f9cb6ffd640 1 -- 172.21.15.42:0/3679600131 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cbc018070 con 0x7f9cc00a39b0 2024-09-16T03:53:15.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.496+0000 7f9cb6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:53:15.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.496+0000 7f9cb6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:53:15.516 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.496+0000 7f9cb6ffd640 10 monclient: dump: 2024-09-16T03:53:15.516 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:53:15.516 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:53:15.516 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:15.516 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:15.516 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:53:15.516 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:53:15.516 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:53:15.516 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:53:15.516 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:53:15.516 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:53:15.516 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.496+0000 7f9cb6ffd640 10 monclient: _finish_auth 0 2024-09-16T03:53:15.516 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.496+0000 7f9cb6ffd640 10 monclient: _check_auth_tickets 2024-09-16T03:53:15.516 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.496+0000 7f9cb6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:53:15.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.496+0000 7f9cc8794640 5 monclient: authenticate success, global_id 824295 2024-09-16T03:53:15.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.496+0000 7f9cb6ffd640 1 -- 172.21.15.42:0/3679600131 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9cbc0074e0 con 0x7f9cc00a39b0 2024-09-16T03:53:15.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.496+0000 7f9cb6ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:53:15.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.496+0000 7f9cb6ffd640 1 -- 172.21.15.42:0/3679600131 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cbc0049f0 con 0x7f9cc00a39b0 2024-09-16T03:53:15.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.496+0000 7f9cc8794640 10 monclient: _renew_subs 2024-09-16T03:53:15.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.496+0000 7f9cc8794640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:53:15.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.496+0000 7f9cc8794640 1 -- 172.21.15.42:0/3679600131 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9cc009ce30 con 0x7f9cc00a39b0 2024-09-16T03:53:15.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.496+0000 7f9cb6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:53:15.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.496+0000 7f9cb6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:53:15.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.496+0000 7f9cb6ffd640 10 monclient: dump: 2024-09-16T03:53:15.517 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:53:15.517 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:53:15.517 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:15.517 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:15.518 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:53:15.518 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:53:15.518 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:53:15.518 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:53:15.518 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:53:15.518 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:53:15.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.496+0000 7f9cc4d2d640 4 set_mon_vals no callback set 2024-09-16T03:53:15.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.496+0000 7f9cc8794640 10 client.824295.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:53:15.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.496+0000 7f9cc8794640 10 monclient: _renew_subs 2024-09-16T03:53:15.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.496+0000 7f9cc8794640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:53:15.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.496+0000 7f9cc8794640 1 -- 172.21.15.42:0/3679600131 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9cc013ef50 con 0x7f9cc00a39b0 2024-09-16T03:53:15.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.496+0000 7f9cc8794640 1 librados: init done 2024-09-16T03:53:15.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.496+0000 7f9cc8794640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:53:15.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.496+0000 7f9cc8794640 1 --2- 172.21.15.42:0/3679600131 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9cc013abc0 0x7f9cc002c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:53:15.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.496+0000 7f9cc8794640 10 monclient(hunting): start opening mon connection 2024-09-16T03:53:15.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.496+0000 7f9cc8794640 1 -- 172.21.15.42:0/3679600131 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f9cc0133b80 con 0x7f9cc013abc0 2024-09-16T03:53:15.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.496+0000 7f9cc5d2f640 1 --2- 172.21.15.42:0/3679600131 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9cc013abc0 0x7f9cc002c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:53:15.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.496+0000 7f9cc5d2f640 10 monclient: get_auth_request con 0x7f9cc013abc0 auth_method 0 2024-09-16T03:53:15.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.496+0000 7f9cc5d2f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:53:15.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.496+0000 7f9cc5d2f640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:53:15.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.496+0000 7f9cc5d2f640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:53:15.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.496+0000 7f9cc5d2f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:53:15.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.496+0000 7f9cc5d2f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:53:15.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.496+0000 7f9cc5d2f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:53:15.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.496+0000 7f9cb6ffd640 1 -- 172.21.15.42:0/3679600131 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f9cbc01b070 con 0x7f9cc00a39b0 2024-09-16T03:53:15.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.496+0000 7f9cc5d2f640 10 monclient(hunting): handle_auth_done global_id 824298 payload 824 2024-09-16T03:53:15.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.496+0000 7f9cc5d2f640 1 -- 172.21.15.42:0/3679600131 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9cc00a5750 con 0x7f9cc013abc0 2024-09-16T03:53:15.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.496+0000 7f9cb6ffd640 1 --2- 172.21.15.42:0/3679600131 >> v2:172.21.15.154:6812/254845342 conn(0x7f9ca40377c0 0x7f9ca4039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:53:15.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.496+0000 7f9cb6ffd640 1 -- 172.21.15.42:0/3679600131 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(347..347 src has 286..347) ==== 5728+0+0 (secure 0 0 0) 0x7f9cbc048350 con 0x7f9cc00a39b0 2024-09-16T03:53:15.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.496+0000 7f9cb6ffd640 10 client.824295.objecter ms_dispatch 0x7f9cc0000ba0 osd_map(347..347 src has 286..347) 2024-09-16T03:53:15.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.496+0000 7f9cb6ffd640 3 client.824295.objecter handle_osd_map got epochs [347,347] > 0 2024-09-16T03:53:15.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.496+0000 7f9cb6ffd640 3 client.824295.objecter handle_osd_map decoding full epoch 347 2024-09-16T03:53:15.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.496+0000 7f9cb6ffd640 20 client.824295.objecter dump_active .. 0 homeless 2024-09-16T03:53:15.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.496+0000 7f9cb6ffd640 10 client.824295.objecter ms_handle_connect 0x7f9cc013abc0 2024-09-16T03:53:15.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.496+0000 7f9cb6ffd640 10 client.824295.objecter resend_mon_ops 2024-09-16T03:53:15.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.496+0000 7f9cc5d2f640 1 --2- 172.21.15.42:0/3679600131 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9cc013abc0 0x7f9cc002c650 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f9cc00a35d0 tx=0x7f9cb000b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:53:15.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.496+0000 7f9cb6ffd640 1 -- 172.21.15.42:0/3679600131 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cb0003e20 con 0x7f9cc013abc0 2024-09-16T03:53:15.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.496+0000 7f9cb6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:53:15.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.496+0000 7f9cb6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:53:15.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.496+0000 7f9cb6ffd640 10 monclient: dump: 2024-09-16T03:53:15.520 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:53:15.520 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:53:15.521 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:15.521 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:15.521 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:53:15.521 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:53:15.521 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:53:15.522 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:53:15.522 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:53:15.522 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:53:15.522 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.496+0000 7f9cc6d31640 1 --2- 172.21.15.42:0/3679600131 >> v2:172.21.15.154:6812/254845342 conn(0x7f9ca40377c0 0x7f9ca4039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:53:15.522 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.496+0000 7f9cc6d31640 10 monclient: get_auth_request con 0x7f9ca40377c0 auth_method 0 2024-09-16T03:53:15.522 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.496+0000 7f9cb6ffd640 1 -- 172.21.15.42:0/3679600131 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f9cb0015070 con 0x7f9cc013abc0 2024-09-16T03:53:15.522 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.496+0000 7f9cb6ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:53:15.522 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.496+0000 7f9cb6ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:53:15.522 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.496+0000 7f9cb6ffd640 1 -- 172.21.15.42:0/3679600131 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9cc013abc0 msgr2=0x7f9cc002c650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:53:15.522 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.496+0000 7f9cb6ffd640 1 --2- 172.21.15.42:0/3679600131 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9cc013abc0 0x7f9cc002c650 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f9cc00a35d0 tx=0x7f9cb000b8e0 comp rx=0 tx=0).stop 2024-09-16T03:53:15.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.496+0000 7f9cb6ffd640 1 -- 172.21.15.42:0/3679600131 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9cc013abc0 msgr2=0x7f9cc002c650 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:53:15.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.496+0000 7f9cb6ffd640 1 --2- 172.21.15.42:0/3679600131 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9cc013abc0 0x7f9cc002c650 secure :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f9cc00a35d0 tx=0x7f9cb000b8e0 comp rx=0 tx=0).stop 2024-09-16T03:53:15.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.496+0000 7f9cc6d31640 1 --2- 172.21.15.42:0/3679600131 >> v2:172.21.15.154:6812/254845342 conn(0x7f9ca40377c0 0x7f9ca4039c80 secure :-1 s=READY pgs=3594 cs=0 l=1 rev1=1 crypto rx=0x7f9cb80af000 tx=0x7f9cb80a7070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:53:15.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.496+0000 7f9cb6ffd640 10 client.824295.objecter ms_handle_connect 0x7f9ca40377c0 2024-09-16T03:53:15.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.504+0000 7f9cc8794640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:53:15.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.504+0000 7f9cc8794640 1 --2- 172.21.15.42:0/3679600131 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9cc013afa0 0x7f9cc002cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:53:15.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.504+0000 7f9cc8794640 10 monclient(hunting): start opening mon connection 2024-09-16T03:53:15.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.504+0000 7f9cc8794640 1 -- 172.21.15.42:0/3679600131 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f9cc01362e0 con 0x7f9cc013afa0 2024-09-16T03:53:15.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.504+0000 7f9cc5d2f640 1 --2- 172.21.15.42:0/3679600131 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9cc013afa0 0x7f9cc002cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:53:15.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.504+0000 7f9cc5d2f640 10 monclient: get_auth_request con 0x7f9cc013afa0 auth_method 0 2024-09-16T03:53:15.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.504+0000 7f9cc5d2f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:53:15.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.504+0000 7f9cc5d2f640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:53:15.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.504+0000 7f9cc5d2f640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:53:15.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.504+0000 7f9cc5d2f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:53:15.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.504+0000 7f9cc5d2f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:53:15.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.504+0000 7f9cc5d2f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:53:15.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.504+0000 7f9cc5d2f640 10 monclient(hunting): handle_auth_done global_id 824301 payload 824 2024-09-16T03:53:15.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.504+0000 7f9cc5d2f640 1 -- 172.21.15.42:0/3679600131 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f9cc0068e40 con 0x7f9cc013afa0 2024-09-16T03:53:15.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.504+0000 7f9cc5d2f640 1 --2- 172.21.15.42:0/3679600131 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9cc013afa0 0x7f9cc002cb90 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f9cc00a35d0 tx=0x7f9cb0005a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:53:15.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.504+0000 7f9cb6ffd640 10 client.824295.objecter ms_handle_connect 0x7f9cc013afa0 2024-09-16T03:53:15.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.504+0000 7f9cb6ffd640 10 client.824295.objecter resend_mon_ops 2024-09-16T03:53:15.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.504+0000 7f9cb6ffd640 1 -- 172.21.15.42:0/3679600131 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cb0005690 con 0x7f9cc013afa0 2024-09-16T03:53:15.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.504+0000 7f9cb6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:53:15.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.504+0000 7f9cb6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:53:15.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.504+0000 7f9cb6ffd640 10 monclient: dump: 2024-09-16T03:53:15.526 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:53:15.526 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:53:15.526 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:15.526 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:15.527 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:53:15.527 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:53:15.527 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:53:15.527 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:53:15.527 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:53:15.527 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:53:15.527 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.504+0000 7f9cb6ffd640 1 -- 172.21.15.42:0/3679600131 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4869 (secure 0 0 0) 0x7f9cb0005690 con 0x7f9cc013afa0 2024-09-16T03:53:15.527 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.504+0000 7f9cb6ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:53:15.527 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.504+0000 7f9cb6ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:53:15.527 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.504+0000 7f9cb6ffd640 1 -- 172.21.15.42:0/3679600131 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9cc013afa0 msgr2=0x7f9cc002cb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:53:15.527 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.504+0000 7f9cb6ffd640 1 --2- 172.21.15.42:0/3679600131 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9cc013afa0 0x7f9cc002cb90 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f9cc00a35d0 tx=0x7f9cb0005a30 comp rx=0 tx=0).stop 2024-09-16T03:53:15.527 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.504+0000 7f9cb6ffd640 1 -- 172.21.15.42:0/3679600131 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9cc013afa0 msgr2=0x7f9cc002cb90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:53:15.527 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.504+0000 7f9cb6ffd640 1 --2- 172.21.15.42:0/3679600131 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9cc013afa0 0x7f9cc002cb90 secure :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f9cc00a35d0 tx=0x7f9cb0005a30 comp rx=0 tx=0).stop 2024-09-16T03:53:15.528 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:53:15.528 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:53:15.528 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:53:15.528 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "leader", 2024-09-16T03:53:15.528 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 98, 2024-09-16T03:53:15.528 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 85359, 2024-09-16T03:53:15.528 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:53:15.528 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:53:15.528 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T03:53:15.529 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:53:15.529 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 1, 2024-09-16T03:53:15.529 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:53:15.529 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:53:15.529 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:53:15.529 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:53:15.529 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:53:15.529 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:53:15.529 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:53:15.529 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:53:15.529 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:53:15.529 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:53:15.529 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:53:15.529 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:53:15.530 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:53:15.530 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:53:15.530 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:53:15.530 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:53:15.530 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:53:15.530 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:53:15.530 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:53:15.530 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:53:15.530 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:53:15.530 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:53:15.530 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:53:15.530 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:53:15.530 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:53:15.531 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:53:15.531 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:53:15.531 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:53:15.531 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:53:15.531 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:53:15.531 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:53:15.531 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:53:15.531 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:53:15.531 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:53:15.531 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:53:15.531 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:53:15.531 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:53:15.531 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:53:15.531 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:53:15.532 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:53:15.532 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:53:15.532 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:53:15.532 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:53:15.532 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:53:15.532 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:53:15.532 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:53:15.532 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:53:15.532 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:53:15.532 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:53:15.532 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:53:15.532 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:53:15.532 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:53:15.533 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:53:15.533 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:53:15.533 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:53:15.533 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:53:15.533 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:53:15.533 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:53:15.533 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:53:15.533 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:53:15.533 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:53:15.533 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:53:15.533 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:53:15.533 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:53:15.533 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:53:15.533 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:53:15.534 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:53:15.534 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:53:15.534 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:53:15.534 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:53:15.534 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:53:15.534 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:53:15.534 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:53:15.534 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:53:15.534 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:53:15.534 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:53:15.534 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:53:15.534 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:53:15.534 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:53:15.535 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:53:15.535 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:53:15.535 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:53:15.535 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:53:15.535 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:53:15.535 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:53:15.535 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:53:15.535 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:53:15.535 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:53:15.535 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:53:15.535 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:53:15.535 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:53:15.535 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:53:15.535 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:53:15.535 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:53:15.536 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:53:15.536 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:53:15.536 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:53:15.536 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:53:15.536 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:53:15.536 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:53:15.536 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:53:15.536 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:53:15.536 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:53:15.536 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:53:15.536 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:53:15.536 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:53:15.536 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:53:15.537 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:53:15.537 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:53:15.537 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:53:15.537 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:53:15.537 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:53:15.537 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:53:15.537 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:53:15.537 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:53:15.537 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:53:15.537 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:53:15.537 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:53:15.537 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:53:15.537 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:53:15.537 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:53:15.538 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:53:15.538 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:53:15.538 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:53:15.538 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:53:15.538 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:53:15.538 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:53:15.538 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:53:15.538 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:53:15.538 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:53:15.538 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:53:15.538 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:53:15.538 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:53:15.538 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:53:15.539 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:53:15.539 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:53:15.539 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:53:15.539 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:53:15.539 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:53:15.539 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:53:15.539 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:53:15.539 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T03:53:15.539 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:53:15.539 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:53:15.539 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:53:15.539 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 5 2024-09-16T03:53:15.539 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:53:15.539 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:53:15.540 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:53:15.540 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:53:15.540 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:53:15.540 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:53:15.540 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 7 2024-09-16T03:53:15.540 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:53:15.540 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:53:15.540 INFO:teuthology.orchestra.run.smithi042.stdout: "mgr": [ 2024-09-16T03:53:15.540 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:53:15.540 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:53:15.540 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:53:15.540 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:53:15.540 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:53:15.540 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:53:15.541 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:53:15.541 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:53:15.541 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:53:15.542 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.504+0000 7f9cc8794640 10 librados: watch_flush enter 2024-09-16T03:53:15.542 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.504+0000 7f9cc8794640 10 librados: watch_flush exit 2024-09-16T03:53:15.543 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.504+0000 7f9cc8794640 20 client.824295.objecter shutdown clearing up homeless session... 2024-09-16T03:53:15.543 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.504+0000 7f9cc8794640 10 client.824295.objecter successfully canceled tick 2024-09-16T03:53:15.543 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.504+0000 7f9cc8794640 1 -- 172.21.15.42:0/3679600131 >> v2:172.21.15.154:6812/254845342 conn(0x7f9ca40377c0 msgr2=0x7f9ca4039c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:53:15.543 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.504+0000 7f9cc8794640 1 --2- 172.21.15.42:0/3679600131 >> v2:172.21.15.154:6812/254845342 conn(0x7f9ca40377c0 0x7f9ca4039c80 secure :-1 s=READY pgs=3594 cs=0 l=1 rev1=1 crypto rx=0x7f9cb80af000 tx=0x7f9cb80a7070 comp rx=0 tx=0).stop 2024-09-16T03:53:15.543 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.504+0000 7f9cc8794640 10 monclient: shutdown 2024-09-16T03:53:15.543 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.504+0000 7f9cc8794640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:53:15.543 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.504+0000 7f9cc8794640 1 -- 172.21.15.42:0/3679600131 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9cc00a39b0 msgr2=0x7f9cc0130920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:53:15.543 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.504+0000 7f9cc8794640 1 --2- 172.21.15.42:0/3679600131 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9cc00a39b0 0x7f9cc0130920 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f9cbc00c520 tx=0x7f9cbc00c9e0 comp rx=0 tx=0).stop 2024-09-16T03:53:15.543 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.504+0000 7f9cc6d31640 1 -- 172.21.15.42:0/3679600131 reap_dead start 2024-09-16T03:53:15.543 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.504+0000 7f9cc6d31640 1 -- 172.21.15.42:0/3679600131 reap_dead start 2024-09-16T03:53:15.543 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.504+0000 7f9cc8794640 1 -- 172.21.15.42:0/3679600131 shutdown_connections 2024-09-16T03:53:15.543 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.504+0000 7f9cc8794640 1 -- 172.21.15.42:0/3679600131 >> 172.21.15.42:0/3679600131 conn(0x7f9cc00a9fb0 msgr2=0x7f9cc0132d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:53:15.544 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.504+0000 7f9cc8794640 1 -- 172.21.15.42:0/3679600131 shutdown_connections 2024-09-16T03:53:15.544 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.504+0000 7f9cc8794640 1 -- 172.21.15.42:0/3679600131 wait complete. 2024-09-16T03:53:15.544 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.504+0000 7f9cc8794640 1 librados: shutdown 2024-09-16T03:53:15.544 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T03:53:15.544 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-16T03:53:15.544 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T03:53:15.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.656+0000 7f66fc7e2640 10 monclient: get_monmap_and_config 2024-09-16T03:53:15.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.656+0000 7f66fc7e2640 10 monclient: build_initial_monmap 2024-09-16T03:53:15.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.656+0000 7f66fc7e2640 10 monclient: monmap: 2024-09-16T03:53:15.663 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:53:15.663 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:53:15.663 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:53:15.661824+0000 2024-09-16T03:53:15.663 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:53:15.661824+0000 2024-09-16T03:53:15.664 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:53:15.664 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:53:15.664 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:53:15.664 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:53:15.664 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:53:15.664 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:53:15.664 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:53:15.664 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:53:15.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.656+0000 7f66fc7e2640 1 Processor -- start 2024-09-16T03:53:15.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.656+0000 7f66fc7e2640 1 -- start start 2024-09-16T03:53:15.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.656+0000 7f66fc7e2640 10 monclient: init 2024-09-16T03:53:15.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.656+0000 7f66fc7e2640 10 monclient: _reopen_session rank -1 2024-09-16T03:53:15.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.656+0000 7f66fc7e2640 10 monclient: _add_conns ranks=[3,0,2,1,4] 2024-09-16T03:53:15.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.656+0000 7f66fc7e2640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f66f40a1330 0x7f66f409f180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:53:15.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.656+0000 7f66fc7e2640 10 monclient(hunting): picked mon.noname-a con 0x7f66f40a1330 addr v2:172.21.15.42:3300/0 2024-09-16T03:53:15.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.656+0000 7f66fc7e2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f66f40a1710 addr v1:172.21.15.42:6789/0 2024-09-16T03:53:15.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.656+0000 7f66fc7e2640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f66f41393a0 0x7f66f413b790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:53:15.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.656+0000 7f66fc7e2640 10 monclient(hunting): picked mon.noname-b con 0x7f66f41393a0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:53:15.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.656+0000 7f66fc7e2640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f66f409e4c0 con 0x7f66f40a1710 2024-09-16T03:53:15.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.656+0000 7f66fc7e2640 10 monclient(hunting): start opening mon connection 2024-09-16T03:53:15.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.656+0000 7f66fc7e2640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f66f409edf0 con 0x7f66f40a1330 2024-09-16T03:53:15.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.656+0000 7f66fc7e2640 10 monclient(hunting): start opening mon connection 2024-09-16T03:53:15.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.656+0000 7f66fc7e2640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f66f409cd60 con 0x7f66f41393a0 2024-09-16T03:53:15.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.656+0000 7f66fc7e2640 10 monclient(hunting): _renew_subs 2024-09-16T03:53:15.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.656+0000 7f66fc7e2640 10 monclient(hunting): authenticate will time out at 1642.631237s 2024-09-16T03:53:15.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.656+0000 7f66f9d7d640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f66f40a1710 0x7f66f40a1b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:45108/0 (socket says 172.21.15.42:45108) 2024-09-16T03:53:15.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.656+0000 7f66f9d7d640 1 -- 172.21.15.42:0/3271836883 learned_addr learned my addr 172.21.15.42:0/3271836883 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:53:15.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.656+0000 7f66fa57e640 1 --2- 172.21.15.42:0/3271836883 >> v2:172.21.15.42:3300/0 conn(0x7f66f40a1330 0x7f66f409f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:53:15.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.656+0000 7f66fa57e640 10 monclient(hunting): get_auth_request con 0x7f66f40a1330 auth_method 0 2024-09-16T03:53:15.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.656+0000 7f66fa57e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:53:15.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.656+0000 7f66fa57e640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:53:15.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.656+0000 7f66fa57e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:53:15.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.656+0000 7f66fad7f640 1 --2- 172.21.15.42:0/3271836883 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f66f41393a0 0x7f66f413b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:53:15.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.656+0000 7f66fad7f640 10 monclient(hunting): get_auth_request con 0x7f66f41393a0 auth_method 0 2024-09-16T03:53:15.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.656+0000 7f66fad7f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:53:15.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.656+0000 7f66fad7f640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:53:15.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.656+0000 7f66fad7f640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:53:15.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.656+0000 7f66fa57e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:53:15.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.656+0000 7f66fa57e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:53:15.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.656+0000 7f66fa57e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:53:15.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.656+0000 7f66fad7f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:53:15.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.656+0000 7f66fad7f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:53:15.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.656+0000 7f66fad7f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:53:15.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.656+0000 7f66fa57e640 10 monclient(hunting): handle_auth_done global_id 824304 payload 274 2024-09-16T03:53:15.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.656+0000 7f66fa57e640 1 -- 172.21.15.42:0/3271836883 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f66f41393a0 msgr2=0x7f66f413b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:53:15.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.656+0000 7f66fa57e640 1 --2- 172.21.15.42:0/3271836883 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f66f41393a0 0x7f66f413b790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:53:15.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.656+0000 7f66fa57e640 1 -- 172.21.15.42:0/3271836883 >> v1:172.21.15.42:6789/0 conn(0x7f66f40a1710 legacy=0x7f66f40a1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:53:15.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.656+0000 7f66fa57e640 10 monclient: _finish_hunting 0 2024-09-16T03:53:15.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.656+0000 7f66fa57e640 1 monclient: found mon.noname-a 2024-09-16T03:53:15.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.656+0000 7f66fa57e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:53:15.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.656+0000 7f66fa57e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:53:15.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.656+0000 7f66fa57e640 1 -- 172.21.15.42:0/3271836883 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66f409f6c0 con 0x7f66f40a1330 2024-09-16T03:53:15.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.660+0000 7f66fa57e640 1 --2- 172.21.15.42:0/3271836883 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f66f40a1330 0x7f66f409f180 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f66f00097c0 tx=0x7f66f001c360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5278f9f9bee48b76 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:53:15.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.660+0000 7f66f957c640 1 -- 172.21.15.42:0/3271836883 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66f00b0070 con 0x7f66f40a1330 2024-09-16T03:53:15.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.660+0000 7f66f957c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:53:15.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.660+0000 7f66f957c640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:53:15.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.660+0000 7f66f957c640 10 monclient: dump: 2024-09-16T03:53:15.676 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:53:15.676 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:53:15.676 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:15.677 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:15.677 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:53:15.677 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:53:15.677 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:53:15.677 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:53:15.677 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:53:15.677 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:53:15.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.660+0000 7f66f957c640 10 monclient: _finish_auth 0 2024-09-16T03:53:15.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.660+0000 7f66f957c640 10 monclient: _check_auth_tickets 2024-09-16T03:53:15.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.660+0000 7f66f957c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:53:15.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.660+0000 7f66f957c640 1 -- 172.21.15.42:0/3271836883 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f66f00ab600 con 0x7f66f40a1330 2024-09-16T03:53:15.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.660+0000 7f66fc7e2640 5 monclient: authenticate success, global_id 824304 2024-09-16T03:53:15.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.660+0000 7f66f957c640 10 monclient: handle_config config(0 keys) 2024-09-16T03:53:15.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.660+0000 7f66fc7e2640 10 monclient: get_monmap_and_config success 2024-09-16T03:53:15.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.660+0000 7f66fc7e2640 4 set_mon_vals no callback set 2024-09-16T03:53:15.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.660+0000 7f66f957c640 1 -- 172.21.15.42:0/3271836883 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66f00ab9d0 con 0x7f66f40a1330 2024-09-16T03:53:15.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.660+0000 7f66f957c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:53:15.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.660+0000 7f66f957c640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:53:15.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.660+0000 7f66f957c640 10 monclient: dump: 2024-09-16T03:53:15.678 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:53:15.678 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:53:15.678 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:15.678 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:15.678 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:53:15.678 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:53:15.679 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:53:15.679 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:53:15.679 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:53:15.679 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:53:15.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.660+0000 7f66fc7e2640 10 monclient: shutdown 2024-09-16T03:53:15.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.660+0000 7f66fc7e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:53:15.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.660+0000 7f66fc7e2640 1 -- 172.21.15.42:0/3271836883 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f66f40a1330 msgr2=0x7f66f409f180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:53:15.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.660+0000 7f66fc7e2640 1 --2- 172.21.15.42:0/3271836883 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f66f40a1330 0x7f66f409f180 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f66f00097c0 tx=0x7f66f001c360 comp rx=0 tx=0).stop 2024-09-16T03:53:15.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.660+0000 7f66fc7e2640 1 -- 172.21.15.42:0/3271836883 shutdown_connections 2024-09-16T03:53:15.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.660+0000 7f66fc7e2640 1 --2- 172.21.15.42:0/3271836883 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f66f41393a0 0x7f66f413b790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:53:15.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.660+0000 7f66fc7e2640 1 --2- 172.21.15.42:0/3271836883 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f66f40a1330 0x7f66f409f180 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:53:15.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.660+0000 7f66fc7e2640 1 -- 172.21.15.42:0/3271836883 >> 172.21.15.42:0/3271836883 conn(0x7f66f40a2420 msgr2=0x7f66f40a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:53:15.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.660+0000 7f66fc7e2640 1 -- 172.21.15.42:0/3271836883 shutdown_connections 2024-09-16T03:53:15.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.660+0000 7f66fc7e2640 1 -- 172.21.15.42:0/3271836883 wait complete. 2024-09-16T03:53:15.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.660+0000 7f66fc7e2640 10 monclient: build_initial_monmap 2024-09-16T03:53:15.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.660+0000 7f66fc7e2640 10 monclient: monmap: 2024-09-16T03:53:15.681 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:53:15.681 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:53:15.681 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:53:15.681 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:53:15.681 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:53:15.681 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:53:15.681 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:53:15.681 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:53:15.681 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:53:15.681 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:53:15.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.660+0000 7f66fc7e2640 1 librados: starting msgr at 2024-09-16T03:53:15.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.660+0000 7f66fc7e2640 1 librados: starting objecter 2024-09-16T03:53:15.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.660+0000 7f66fc7e2640 1 Processor -- start 2024-09-16T03:53:15.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.660+0000 7f66fc7e2640 1 -- start start 2024-09-16T03:53:15.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.660+0000 7f66fc7e2640 1 librados: setting wanted keys 2024-09-16T03:53:15.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.660+0000 7f66fc7e2640 1 librados: calling monclient init 2024-09-16T03:53:15.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.660+0000 7f66fc7e2640 10 monclient: init 2024-09-16T03:53:15.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.664+0000 7f66fc7e2640 10 monclient: _reopen_session rank -1 2024-09-16T03:53:15.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.664+0000 7f66fc7e2640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:53:15.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.664+0000 7f66fc7e2640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f66f40a1330 0x7f66f413e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:53:15.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.664+0000 7f66fc7e2640 10 monclient(hunting): picked mon.noname-c con 0x7f66f40a1330 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:53:15.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.664+0000 7f66fc7e2640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f66f40a1710 0x7f66f413ea30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:53:15.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.664+0000 7f66fc7e2640 10 monclient(hunting): picked mon.noname-b con 0x7f66f40a1710 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:53:15.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.664+0000 7f66fc7e2640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f66f41393a0 0x7f66f413ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:53:15.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.664+0000 7f66fc7e2640 10 monclient(hunting): picked mon.noname-a con 0x7f66f41393a0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:53:15.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.664+0000 7f66fc7e2640 10 monclient(hunting): start opening mon connection 2024-09-16T03:53:15.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.664+0000 7f66fc7e2640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f66f409be60 con 0x7f66f41393a0 2024-09-16T03:53:15.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.664+0000 7f66fc7e2640 10 monclient(hunting): start opening mon connection 2024-09-16T03:53:15.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.664+0000 7f66fc7e2640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f66f409bce0 con 0x7f66f40a1710 2024-09-16T03:53:15.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.664+0000 7f66fc7e2640 10 monclient(hunting): start opening mon connection 2024-09-16T03:53:15.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.664+0000 7f66fc7e2640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f66f409bfe0 con 0x7f66f40a1330 2024-09-16T03:53:15.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.664+0000 7f66fc7e2640 10 monclient(hunting): _renew_subs 2024-09-16T03:53:15.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.664+0000 7f66fc7e2640 10 monclient(hunting): authenticate will time out at 1642.636385s 2024-09-16T03:53:15.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.664+0000 7f66fa57e640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f66f40a1330 0x7f66f413e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:53:15.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.664+0000 7f66fad7f640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f66f41393a0 0x7f66f413ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:53:15.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.664+0000 7f66fa57e640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f66f40a1330 0x7f66f413e4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:46636/0 (socket says 172.21.15.42:46636) 2024-09-16T03:53:15.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.664+0000 7f66fa57e640 1 -- 172.21.15.42:0/1369230403 learned_addr learned my addr 172.21.15.42:0/1369230403 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:53:15.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.664+0000 7f66fa57e640 10 monclient(hunting): get_auth_request con 0x7f66f40a1330 auth_method 0 2024-09-16T03:53:15.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.664+0000 7f66fa57e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:53:15.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.664+0000 7f66fa57e640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:53:15.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.664+0000 7f66fa57e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:53:15.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.664+0000 7f66fad7f640 10 monclient(hunting): get_auth_request con 0x7f66f41393a0 auth_method 0 2024-09-16T03:53:15.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.664+0000 7f66fad7f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:53:15.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.664+0000 7f66fad7f640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:53:15.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.664+0000 7f66fad7f640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:53:15.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.664+0000 7f66f9d7d640 1 -- 172.21.15.42:0/1369230403 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f66f40a1710 msgr2=0x7f66f413ea30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T03:53:15.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.664+0000 7f66f9d7d640 1 --2- 172.21.15.42:0/1369230403 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f66f40a1710 0x7f66f413ea30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:53:15.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.664+0000 7f66fa57e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:53:15.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.664+0000 7f66fa57e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:53:15.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.664+0000 7f66fa57e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:53:15.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.664+0000 7f66fad7f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:53:15.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.664+0000 7f66fad7f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:53:15.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.664+0000 7f66fad7f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:53:15.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.664+0000 7f66fa57e640 10 monclient(hunting): handle_auth_done global_id 834185 payload 824 2024-09-16T03:53:15.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.664+0000 7f66fa57e640 1 -- 172.21.15.42:0/1369230403 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f66f40a1710 msgr2=0x7f66f413ea30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:53:15.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.664+0000 7f66fa57e640 1 --2- 172.21.15.42:0/1369230403 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f66f40a1710 0x7f66f413ea30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:53:15.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.664+0000 7f66fa57e640 1 -- 172.21.15.42:0/1369230403 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f66f41393a0 msgr2=0x7f66f413ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:53:15.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.664+0000 7f66fa57e640 1 --2- 172.21.15.42:0/1369230403 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f66f41393a0 0x7f66f413ef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:53:15.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.664+0000 7f66fa57e640 10 monclient: _finish_hunting 0 2024-09-16T03:53:15.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.664+0000 7f66fa57e640 1 monclient: found mon.noname-c 2024-09-16T03:53:15.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.664+0000 7f66fa57e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:53:15.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.664+0000 7f66fa57e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:53:15.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.664+0000 7f66fa57e640 1 -- 172.21.15.42:0/1369230403 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66f40a5050 con 0x7f66f40a1330 2024-09-16T03:53:15.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.664+0000 7f66fad7f640 1 --2- 172.21.15.42:0/1369230403 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f66f41393a0 0x7f66f413ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:53:15.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.664+0000 7f66fa57e640 1 --2- 172.21.15.42:0/1369230403 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f66f40a1330 0x7f66f413e4f0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f66f0019f90 tx=0x7f66f0019100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:53:15.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.664+0000 7f66eaffd640 10 client.?.objecter ms_handle_connect 0x7f66f40a1330 2024-09-16T03:53:15.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.664+0000 7f66eaffd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:53:15.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.664+0000 7f66eaffd640 1 -- 172.21.15.42:0/1369230403 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66f00b0040 con 0x7f66f40a1330 2024-09-16T03:53:15.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.664+0000 7f66eaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:53:15.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.664+0000 7f66eaffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:53:15.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.664+0000 7f66eaffd640 10 monclient: dump: 2024-09-16T03:53:15.687 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:53:15.687 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:53:15.687 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:15.687 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:15.687 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:53:15.687 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:53:15.687 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:53:15.688 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:53:15.688 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:53:15.688 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:53:15.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.664+0000 7f66eaffd640 10 monclient: _finish_auth 0 2024-09-16T03:53:15.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.664+0000 7f66eaffd640 10 monclient: _check_auth_tickets 2024-09-16T03:53:15.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.664+0000 7f66eaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:53:15.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.664+0000 7f66eaffd640 1 -- 172.21.15.42:0/1369230403 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f66f00b43f0 con 0x7f66f40a1330 2024-09-16T03:53:15.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.664+0000 7f66fc7e2640 5 monclient: authenticate success, global_id 834185 2024-09-16T03:53:15.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.664+0000 7f66fc7e2640 10 monclient: _renew_subs 2024-09-16T03:53:15.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.664+0000 7f66fc7e2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:53:15.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.664+0000 7f66fc7e2640 1 -- 172.21.15.42:0/1369230403 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66f413ff70 con 0x7f66f40a1330 2024-09-16T03:53:15.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.664+0000 7f66eaffd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:53:15.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.664+0000 7f66eaffd640 1 -- 172.21.15.42:0/1369230403 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66f00b47c0 con 0x7f66f40a1330 2024-09-16T03:53:15.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.664+0000 7f66eaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:53:15.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.664+0000 7f66eaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:53:15.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.664+0000 7f66eaffd640 10 monclient: dump: 2024-09-16T03:53:15.689 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:53:15.689 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:53:15.689 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:15.689 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:15.689 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:53:15.690 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:53:15.690 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:53:15.690 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:53:15.690 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:53:15.690 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:53:15.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.664+0000 7f66fc7e2640 10 client.834185.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:53:15.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.664+0000 7f66fc7e2640 10 monclient: _renew_subs 2024-09-16T03:53:15.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.664+0000 7f66f8d7b640 4 set_mon_vals no callback set 2024-09-16T03:53:15.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.664+0000 7f66fc7e2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:53:15.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.664+0000 7f66fc7e2640 1 -- 172.21.15.42:0/1369230403 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f66f4132180 con 0x7f66f40a1330 2024-09-16T03:53:15.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.664+0000 7f66fc7e2640 1 librados: init done 2024-09-16T03:53:15.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.664+0000 7f66fc7e2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:53:15.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.664+0000 7f66fc7e2640 1 --2- 172.21.15.42:0/1369230403 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f66f412d080 0x7f66f412d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:53:15.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.664+0000 7f66fc7e2640 10 monclient(hunting): start opening mon connection 2024-09-16T03:53:15.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.664+0000 7f66fc7e2640 1 -- 172.21.15.42:0/1369230403 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f66f41327d0 con 0x7f66f412d080 2024-09-16T03:53:15.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.664+0000 7f66eaffd640 1 -- 172.21.15.42:0/1369230403 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f66f00b4960 con 0x7f66f40a1330 2024-09-16T03:53:15.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.664+0000 7f66f9d7d640 1 --2- 172.21.15.42:0/1369230403 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f66f412d080 0x7f66f412d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:53:15.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.664+0000 7f66f9d7d640 10 monclient: get_auth_request con 0x7f66f412d080 auth_method 0 2024-09-16T03:53:15.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.664+0000 7f66f9d7d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:53:15.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.664+0000 7f66f9d7d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:53:15.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.664+0000 7f66f9d7d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:53:15.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.664+0000 7f66f9d7d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:53:15.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.664+0000 7f66f9d7d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:53:15.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.664+0000 7f66f9d7d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:53:15.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.664+0000 7f66eaffd640 1 --2- 172.21.15.42:0/1369230403 >> v2:172.21.15.154:6812/254845342 conn(0x7f66d80377c0 0x7f66d8039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:53:15.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.664+0000 7f66eaffd640 1 -- 172.21.15.42:0/1369230403 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(347..347 src has 286..347) ==== 5728+0+0 (secure 0 0 0) 0x7f66f00e2950 con 0x7f66f40a1330 2024-09-16T03:53:15.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.664+0000 7f66eaffd640 10 client.834185.objecter ms_dispatch 0x7f66f4000ba0 osd_map(347..347 src has 286..347) 2024-09-16T03:53:15.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.664+0000 7f66eaffd640 3 client.834185.objecter handle_osd_map got epochs [347,347] > 0 2024-09-16T03:53:15.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.664+0000 7f66eaffd640 3 client.834185.objecter handle_osd_map decoding full epoch 347 2024-09-16T03:53:15.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.664+0000 7f66eaffd640 20 client.834185.objecter dump_active .. 0 homeless 2024-09-16T03:53:15.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.664+0000 7f66f9d7d640 10 monclient(hunting): handle_auth_done global_id 834188 payload 824 2024-09-16T03:53:15.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.664+0000 7f66f9d7d640 1 -- 172.21.15.42:0/1369230403 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f66f4134680 con 0x7f66f412d080 2024-09-16T03:53:15.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.664+0000 7f66eaffd640 10 client.834185.objecter ms_handle_connect 0x7f66f412d080 2024-09-16T03:53:15.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.664+0000 7f66eaffd640 10 client.834185.objecter resend_mon_ops 2024-09-16T03:53:15.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.664+0000 7f66f9d7d640 1 --2- 172.21.15.42:0/1369230403 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f66f412d080 0x7f66f412d460 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f66f4072460 tx=0x7f66e400c7a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:53:15.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.664+0000 7f66fad7f640 1 --2- 172.21.15.42:0/1369230403 >> v2:172.21.15.154:6812/254845342 conn(0x7f66d80377c0 0x7f66d8039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:53:15.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.664+0000 7f66fad7f640 10 monclient: get_auth_request con 0x7f66d80377c0 auth_method 0 2024-09-16T03:53:15.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.664+0000 7f66eaffd640 1 -- 172.21.15.42:0/1369230403 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66e4017070 con 0x7f66f412d080 2024-09-16T03:53:15.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.664+0000 7f66eaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:53:15.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.664+0000 7f66eaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:53:15.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.664+0000 7f66eaffd640 10 monclient: dump: 2024-09-16T03:53:15.693 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:53:15.694 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:53:15.694 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:15.694 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:15.694 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:53:15.694 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:53:15.694 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:53:15.694 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:53:15.694 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:53:15.694 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:53:15.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.664+0000 7f66eaffd640 1 -- 172.21.15.42:0/1369230403 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f66e4005cf0 con 0x7f66f412d080 2024-09-16T03:53:15.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.664+0000 7f66eaffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:53:15.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.664+0000 7f66eaffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:53:15.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.664+0000 7f66eaffd640 1 -- 172.21.15.42:0/1369230403 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f66f412d080 msgr2=0x7f66f412d460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:53:15.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.664+0000 7f66eaffd640 1 --2- 172.21.15.42:0/1369230403 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f66f412d080 0x7f66f412d460 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f66f4072460 tx=0x7f66e400c7a0 comp rx=0 tx=0).stop 2024-09-16T03:53:15.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.664+0000 7f66eaffd640 1 -- 172.21.15.42:0/1369230403 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f66f412d080 msgr2=0x7f66f412d460 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:53:15.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.664+0000 7f66eaffd640 1 --2- 172.21.15.42:0/1369230403 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f66f412d080 0x7f66f412d460 secure :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f66f4072460 tx=0x7f66e400c7a0 comp rx=0 tx=0).stop 2024-09-16T03:53:15.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.664+0000 7f66fad7f640 1 --2- 172.21.15.42:0/1369230403 >> v2:172.21.15.154:6812/254845342 conn(0x7f66d80377c0 0x7f66d8039c80 secure :-1 s=READY pgs=3595 cs=0 l=1 rev1=1 crypto rx=0x7f66f409df70 tx=0x7f66ec00b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:53:15.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.664+0000 7f66eaffd640 10 client.834185.objecter ms_handle_connect 0x7f66d80377c0 2024-09-16T03:53:15.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.672+0000 7f66fc7e2640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:53:15.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.672+0000 7f66fc7e2640 1 --2- 172.21.15.42:0/1369230403 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f66f412d9a0 0x7f66f413a350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:53:15.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.672+0000 7f66fc7e2640 10 monclient(hunting): start opening mon connection 2024-09-16T03:53:15.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.672+0000 7f66fc7e2640 1 -- 172.21.15.42:0/1369230403 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f66f4073a30 con 0x7f66f412d9a0 2024-09-16T03:53:15.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.672+0000 7f66f9d7d640 1 --2- 172.21.15.42:0/1369230403 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f66f412d9a0 0x7f66f413a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:53:15.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.672+0000 7f66f9d7d640 10 monclient: get_auth_request con 0x7f66f412d9a0 auth_method 0 2024-09-16T03:53:15.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.672+0000 7f66f9d7d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:53:15.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.672+0000 7f66f9d7d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:53:15.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.672+0000 7f66f9d7d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:53:15.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.672+0000 7f66f9d7d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:53:15.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.672+0000 7f66f9d7d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:53:15.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.672+0000 7f66f9d7d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:53:15.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.672+0000 7f66f9d7d640 10 monclient(hunting): handle_auth_done global_id 834191 payload 824 2024-09-16T03:53:15.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.672+0000 7f66f9d7d640 1 -- 172.21.15.42:0/1369230403 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f66f409f180 con 0x7f66f412d9a0 2024-09-16T03:53:15.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.672+0000 7f66f9d7d640 1 --2- 172.21.15.42:0/1369230403 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f66f412d9a0 0x7f66f413a350 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f66e4000fc0 tx=0x7f66e4000bd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:53:15.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.672+0000 7f66eaffd640 10 client.834185.objecter ms_handle_connect 0x7f66f412d9a0 2024-09-16T03:53:15.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.672+0000 7f66eaffd640 10 client.834185.objecter resend_mon_ops 2024-09-16T03:53:15.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.672+0000 7f66eaffd640 1 -- 172.21.15.42:0/1369230403 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66e4017030 con 0x7f66f412d9a0 2024-09-16T03:53:15.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.672+0000 7f66eaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:53:15.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.672+0000 7f66eaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:53:15.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.672+0000 7f66eaffd640 10 monclient: dump: 2024-09-16T03:53:15.699 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:53:15.699 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:53:15.699 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:15.699 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:15.699 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:53:15.699 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:53:15.699 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:53:15.699 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:53:15.699 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:53:15.700 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:53:15.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.672+0000 7f66eaffd640 1 -- 172.21.15.42:0/1369230403 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4700 (secure 0 0 0) 0x7f66e40213b0 con 0x7f66f412d9a0 2024-09-16T03:53:15.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.672+0000 7f66eaffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:53:15.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.672+0000 7f66eaffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:53:15.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.672+0000 7f66eaffd640 1 -- 172.21.15.42:0/1369230403 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f66f412d9a0 msgr2=0x7f66f413a350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:53:15.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.672+0000 7f66eaffd640 1 --2- 172.21.15.42:0/1369230403 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f66f412d9a0 0x7f66f413a350 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f66e4000fc0 tx=0x7f66e4000bd0 comp rx=0 tx=0).stop 2024-09-16T03:53:15.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.672+0000 7f66eaffd640 1 -- 172.21.15.42:0/1369230403 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f66f412d9a0 msgr2=0x7f66f413a350 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:53:15.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.672+0000 7f66eaffd640 1 --2- 172.21.15.42:0/1369230403 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f66f412d9a0 0x7f66f413a350 secure :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f66e4000fc0 tx=0x7f66e4000bd0 comp rx=0 tx=0).stop 2024-09-16T03:53:15.700 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:53:15.701 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:53:15.701 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:53:15.701 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T03:53:15.701 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 98, 2024-09-16T03:53:15.701 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 35243, 2024-09-16T03:53:15.701 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:53:15.701 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:53:15.701 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T03:53:15.701 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:53:15.701 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 2, 2024-09-16T03:53:15.701 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:53:15.702 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:53:15.702 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:53:15.702 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:53:15.702 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:53:15.702 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:53:15.702 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:53:15.702 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:53:15.702 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:53:15.702 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:53:15.702 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:53:15.702 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:53:15.702 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:53:15.703 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:53:15.703 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:53:15.703 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:53:15.703 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:53:15.703 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:53:15.703 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:53:15.703 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:53:15.703 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:53:15.703 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:53:15.703 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:53:15.703 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:53:15.703 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:53:15.704 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:53:15.704 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:53:15.704 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:53:15.704 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:53:15.704 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:53:15.704 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:53:15.704 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:53:15.704 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:53:15.704 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:53:15.704 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:53:15.705 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:53:15.705 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:53:15.705 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:53:15.705 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:53:15.705 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:53:15.705 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:53:15.705 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:53:15.705 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:53:15.705 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:53:15.705 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:53:15.705 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:53:15.705 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:53:15.706 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:53:15.706 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:53:15.706 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:53:15.706 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:53:15.706 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:53:15.706 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:53:15.706 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:53:15.706 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:53:15.706 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:53:15.706 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:53:15.706 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:53:15.707 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:53:15.707 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:53:15.707 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:53:15.707 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:53:15.707 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:53:15.707 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:53:15.707 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:53:15.707 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:53:15.707 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:53:15.707 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:53:15.707 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:53:15.708 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:53:15.708 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:53:15.708 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:53:15.708 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:53:15.708 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:53:15.708 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:53:15.708 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:53:15.708 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:53:15.708 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:53:15.708 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:53:15.709 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:53:15.709 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:53:15.709 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:53:15.709 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:53:15.709 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:53:15.709 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:53:15.709 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:53:15.709 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:53:15.709 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:53:15.709 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:53:15.709 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:53:15.709 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:53:15.710 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:53:15.710 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:53:15.710 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:53:15.710 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:53:15.710 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:53:15.710 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:53:15.710 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:53:15.710 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:53:15.710 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:53:15.710 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:53:15.710 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:53:15.711 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:53:15.711 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:53:15.711 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:53:15.711 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:53:15.711 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:53:15.711 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:53:15.711 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:53:15.711 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:53:15.711 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:53:15.711 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:53:15.711 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:53:15.711 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:53:15.712 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:53:15.712 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:53:15.712 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:53:15.712 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:53:15.712 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:53:15.712 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:53:15.712 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:53:15.712 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:53:15.712 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:53:15.712 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:53:15.712 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:53:15.713 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:53:15.713 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:53:15.713 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:53:15.713 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:53:15.713 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:53:15.713 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:53:15.713 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:53:15.713 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:53:15.713 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:53:15.713 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:53:15.713 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:53:15.713 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:53:15.714 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:53:15.714 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:53:15.714 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:53:15.714 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:53:15.714 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T03:53:15.714 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:53:15.714 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:53:15.714 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:53:15.714 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:53:15.714 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:53:15.714 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:53:15.715 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:53:15.715 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:53:15.715 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:53:15.715 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:53:15.715 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 3 2024-09-16T03:53:15.715 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:53:15.715 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:53:15.715 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:53:15.715 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:53:15.715 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:53:15.717 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.672+0000 7f66fc7e2640 10 librados: watch_flush enter 2024-09-16T03:53:15.717 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.672+0000 7f66fc7e2640 10 librados: watch_flush exit 2024-09-16T03:53:15.717 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.672+0000 7f66fc7e2640 20 client.834185.objecter shutdown clearing up homeless session... 2024-09-16T03:53:15.717 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.672+0000 7f66fc7e2640 10 client.834185.objecter successfully canceled tick 2024-09-16T03:53:15.717 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.672+0000 7f66fc7e2640 1 -- 172.21.15.42:0/1369230403 >> v2:172.21.15.154:6812/254845342 conn(0x7f66d80377c0 msgr2=0x7f66d8039c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:53:15.718 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.672+0000 7f66fc7e2640 1 --2- 172.21.15.42:0/1369230403 >> v2:172.21.15.154:6812/254845342 conn(0x7f66d80377c0 0x7f66d8039c80 secure :-1 s=READY pgs=3595 cs=0 l=1 rev1=1 crypto rx=0x7f66f409df70 tx=0x7f66ec00b040 comp rx=0 tx=0).stop 2024-09-16T03:53:15.718 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.672+0000 7f66fc7e2640 10 monclient: shutdown 2024-09-16T03:53:15.718 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.672+0000 7f66fc7e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:53:15.718 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.672+0000 7f66fc7e2640 1 -- 172.21.15.42:0/1369230403 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f66f40a1330 msgr2=0x7f66f413e4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:53:15.718 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.672+0000 7f66fc7e2640 1 --2- 172.21.15.42:0/1369230403 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f66f40a1330 0x7f66f413e4f0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f66f0019f90 tx=0x7f66f0019100 comp rx=0 tx=0).stop 2024-09-16T03:53:15.718 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.672+0000 7f66fad7f640 1 -- 172.21.15.42:0/1369230403 reap_dead start 2024-09-16T03:53:15.718 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.672+0000 7f66fad7f640 1 -- 172.21.15.42:0/1369230403 reap_dead start 2024-09-16T03:53:15.718 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.672+0000 7f66fc7e2640 1 -- 172.21.15.42:0/1369230403 shutdown_connections 2024-09-16T03:53:15.718 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.672+0000 7f66fc7e2640 1 -- 172.21.15.42:0/1369230403 >> 172.21.15.42:0/1369230403 conn(0x7f66f40a2420 msgr2=0x7f66f41304e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:53:15.718 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.672+0000 7f66fc7e2640 1 -- 172.21.15.42:0/1369230403 shutdown_connections 2024-09-16T03:53:15.719 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.672+0000 7f66fc7e2640 1 -- 172.21.15.42:0/1369230403 wait complete. 2024-09-16T03:53:15.719 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:15.672+0000 7f66fc7e2640 1 librados: shutdown 2024-09-16T03:53:15.719 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T03:53:15.719 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-16T03:53:35.720 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-16T03:53:35.720 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-09-16T03:53:35.720 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-16T03:53:35.721 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-16T03:53:35.724 INFO:tasks.ceph.mon.b:Started 2024-09-16T03:53:35.724 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T03:53:35.725 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:53:35.858 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.852+0000 7fe62b97f640 10 monclient: get_monmap_and_config 2024-09-16T03:53:35.858 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.852+0000 7fe62b97f640 10 monclient: build_initial_monmap 2024-09-16T03:53:35.859 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.852+0000 7fe62b97f640 10 monclient: monmap: 2024-09-16T03:53:35.859 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:53:35.859 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:53:35.859 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:53:35.857290+0000 2024-09-16T03:53:35.859 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:53:35.857290+0000 2024-09-16T03:53:35.859 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:53:35.859 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:53:35.859 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:53:35.859 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:53:35.859 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:53:35.859 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:53:35.859 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:53:35.859 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:53:35.862 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.852+0000 7fe62b97f640 1 Processor -- start 2024-09-16T03:53:35.862 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.852+0000 7fe62b97f640 1 -- start start 2024-09-16T03:53:35.862 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.852+0000 7fe62b97f640 10 monclient: init 2024-09-16T03:53:35.862 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.852+0000 7fe62b97f640 10 monclient: _reopen_session rank -1 2024-09-16T03:53:35.862 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.852+0000 7fe62b97f640 10 monclient: _add_conns ranks=[3,4,0,2,1] 2024-09-16T03:53:35.862 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.852+0000 7fe62b97f640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe6240a1330 0x7fe6240a1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:53:35.862 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.852+0000 7fe62b97f640 10 monclient(hunting): picked mon.noname-a con 0x7fe6240a1330 addr v2:172.21.15.42:3300/0 2024-09-16T03:53:35.862 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.852+0000 7fe62b97f640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fe62406dac0 0x7fe6240a1c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:53:35.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.852+0000 7fe62b97f640 10 monclient(hunting): picked mon.noname-c con 0x7fe62406dac0 addr v2:172.21.15.154:3300/0 2024-09-16T03:53:35.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.852+0000 7fe62b97f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe624135100 addr v1:172.21.15.42:6789/0 2024-09-16T03:53:35.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.852+0000 7fe62b97f640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe62409c7d0 con 0x7fe624135100 2024-09-16T03:53:35.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.852+0000 7fe62b97f640 10 monclient(hunting): start opening mon connection 2024-09-16T03:53:35.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.852+0000 7fe62b97f640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7fe62409c090 con 0x7fe6240a1330 2024-09-16T03:53:35.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.852+0000 7fe62b97f640 10 monclient(hunting): start opening mon connection 2024-09-16T03:53:35.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.852+0000 7fe62b97f640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fe62409f610 con 0x7fe62406dac0 2024-09-16T03:53:35.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.852+0000 7fe62b97f640 10 monclient(hunting): _renew_subs 2024-09-16T03:53:35.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.852+0000 7fe62b97f640 10 monclient(hunting): authenticate will time out at 1662.826830s 2024-09-16T03:53:35.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.852+0000 7fe62971b640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe6240a1330 0x7fe6240a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:53:35.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.852+0000 7fe629f1c640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fe624135100 0x7fe624135550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:50152/0 (socket says 172.21.15.42:50152) 2024-09-16T03:53:35.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.852+0000 7fe62971b640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe6240a1330 0x7fe6240a1710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:44470/0 (socket says 172.21.15.42:44470) 2024-09-16T03:53:35.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.852+0000 7fe629f1c640 1 -- 172.21.15.42:0/2982986155 learned_addr learned my addr 172.21.15.42:0/2982986155 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:53:35.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.852+0000 7fe62971b640 10 monclient(hunting): get_auth_request con 0x7fe6240a1330 auth_method 0 2024-09-16T03:53:35.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.852+0000 7fe62971b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:53:35.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.852+0000 7fe62971b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:53:35.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.852+0000 7fe62971b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:53:35.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.852+0000 7fe62971b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:53:35.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.852+0000 7fe62971b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:53:35.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.852+0000 7fe62971b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:53:35.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.852+0000 7fe628f1a640 1 --2- 172.21.15.42:0/2982986155 >> v2:172.21.15.154:3300/0 conn(0x7fe62406dac0 0x7fe6240a1c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:53:35.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.852+0000 7fe628f1a640 10 monclient(hunting): get_auth_request con 0x7fe62406dac0 auth_method 0 2024-09-16T03:53:35.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.852+0000 7fe628f1a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:53:35.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.852+0000 7fe628f1a640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:53:35.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.852+0000 7fe628f1a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:53:35.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.852+0000 7fe62971b640 10 monclient(hunting): handle_auth_done global_id 824313 payload 274 2024-09-16T03:53:35.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.852+0000 7fe628f1a640 1 -- 172.21.15.42:0/2982986155 >> v2:172.21.15.154:3300/0 conn(0x7fe62406dac0 msgr2=0x7fe6240a1c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T03:53:35.865 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.852+0000 7fe628f1a640 1 -- 172.21.15.42:0/2982986155 >> v2:172.21.15.154:3300/0 conn(0x7fe62406dac0 msgr2=0x7fe6240a1c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:53:35.865 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.852+0000 7fe628f1a640 1 --2- 172.21.15.42:0/2982986155 >> v2:172.21.15.154:3300/0 conn(0x7fe62406dac0 0x7fe6240a1c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:53:35.865 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.852+0000 7fe62971b640 1 -- 172.21.15.42:0/2982986155 >> v2:172.21.15.154:3300/0 conn(0x7fe62406dac0 msgr2=0x7fe6240a1c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:53:35.865 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.852+0000 7fe628f1a640 1 --2- 172.21.15.42:0/2982986155 >> v2:172.21.15.154:3300/0 conn(0x7fe62406dac0 0x7fe6240a1c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:53:35.865 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.852+0000 7fe62971b640 1 --2- 172.21.15.42:0/2982986155 >> v2:172.21.15.154:3300/0 conn(0x7fe62406dac0 0x7fe6240a1c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:53:35.865 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.852+0000 7fe62971b640 1 -- 172.21.15.42:0/2982986155 >> v1:172.21.15.42:6789/0 conn(0x7fe624135100 legacy=0x7fe624135550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:53:35.865 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.852+0000 7fe62971b640 10 monclient: _finish_hunting 0 2024-09-16T03:53:35.865 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.852+0000 7fe62971b640 1 monclient: found mon.noname-a 2024-09-16T03:53:35.865 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.852+0000 7fe62971b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:53:35.865 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.852+0000 7fe62971b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:53:35.865 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.852+0000 7fe62971b640 1 -- 172.21.15.42:0/2982986155 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe62409f8f0 con 0x7fe6240a1330 2024-09-16T03:53:35.865 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.852+0000 7fe623fff640 1 -- 172.21.15.42:0/2982986155 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2329478769 0 0) 0x7fe62409c7d0 con 0x7fe624135100 2024-09-16T03:53:35.865 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.852+0000 7fe623fff640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T03:53:35.871 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.856+0000 7fe62971b640 1 --2- 172.21.15.42:0/2982986155 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe6240a1330 0x7fe6240a1710 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fe62406da10 tx=0x7fe61401c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53403c7aef85c88c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:53:35.871 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.856+0000 7fe623fff640 1 -- 172.21.15.42:0/2982986155 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6140b2070 con 0x7fe6240a1330 2024-09-16T03:53:35.871 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.856+0000 7fe623fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:53:35.871 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.856+0000 7fe623fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:53:35.871 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.856+0000 7fe623fff640 10 monclient: dump: 2024-09-16T03:53:35.871 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:53:35.871 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:53:35.871 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:35.871 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:35.871 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:53:35.872 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:53:35.872 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:53:35.872 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:53:35.872 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:53:35.872 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:53:35.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.856+0000 7fe623fff640 10 monclient: _finish_auth 0 2024-09-16T03:53:35.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.856+0000 7fe623fff640 10 monclient: _check_auth_tickets 2024-09-16T03:53:35.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.856+0000 7fe623fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:53:35.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.856+0000 7fe623fff640 1 -- 172.21.15.42:0/2982986155 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe6140a59c0 con 0x7fe6240a1330 2024-09-16T03:53:35.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.856+0000 7fe62b97f640 5 monclient: authenticate success, global_id 824313 2024-09-16T03:53:35.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.856+0000 7fe62b97f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:53:35.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.856+0000 7fe623fff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:53:35.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.856+0000 7fe62b97f640 10 monclient: get_monmap_and_config success 2024-09-16T03:53:35.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.856+0000 7fe62b97f640 4 set_mon_vals no callback set 2024-09-16T03:53:35.873 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.856+0000 7fe623fff640 1 -- 172.21.15.42:0/2982986155 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6140a5d90 con 0x7fe6240a1330 2024-09-16T03:53:35.873 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.856+0000 7fe623fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:53:35.873 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.856+0000 7fe623fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:53:35.873 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.856+0000 7fe623fff640 10 monclient: dump: 2024-09-16T03:53:35.873 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:53:35.873 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:53:35.873 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:35.873 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:35.873 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:53:35.873 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:53:35.873 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:53:35.873 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:53:35.873 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:53:35.873 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:53:35.874 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.856+0000 7fe62b97f640 10 monclient: shutdown 2024-09-16T03:53:35.874 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.856+0000 7fe62b97f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:53:35.874 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.856+0000 7fe62b97f640 1 -- 172.21.15.42:0/2982986155 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe6240a1330 msgr2=0x7fe6240a1710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:53:35.874 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.856+0000 7fe62b97f640 1 --2- 172.21.15.42:0/2982986155 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe6240a1330 0x7fe6240a1710 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fe62406da10 tx=0x7fe61401c8e0 comp rx=0 tx=0).stop 2024-09-16T03:53:35.874 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.856+0000 7fe62b97f640 1 -- 172.21.15.42:0/2982986155 shutdown_connections 2024-09-16T03:53:35.874 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.856+0000 7fe62b97f640 1 --2- 172.21.15.42:0/2982986155 >> v2:172.21.15.154:3300/0 conn(0x7fe62406dac0 0x7fe6240a1c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:53:35.874 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.856+0000 7fe62b97f640 1 --2- 172.21.15.42:0/2982986155 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe6240a1330 0x7fe6240a1710 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:53:35.874 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.856+0000 7fe62b97f640 1 -- 172.21.15.42:0/2982986155 >> 172.21.15.42:0/2982986155 conn(0x7fe6240a2420 msgr2=0x7fe6240a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:53:35.874 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.856+0000 7fe62b97f640 1 -- 172.21.15.42:0/2982986155 shutdown_connections 2024-09-16T03:53:35.874 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.856+0000 7fe62b97f640 1 -- 172.21.15.42:0/2982986155 wait complete. 2024-09-16T03:53:35.874 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.856+0000 7fe62b97f640 10 monclient: build_initial_monmap 2024-09-16T03:53:35.874 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.856+0000 7fe62b97f640 10 monclient: monmap: 2024-09-16T03:53:35.874 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:53:35.875 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:53:35.875 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:53:35.875 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:53:35.875 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:53:35.875 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:53:35.875 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:53:35.875 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:53:35.875 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:53:35.875 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:53:35.875 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.860+0000 7fe62b97f640 1 librados: starting msgr at 2024-09-16T03:53:35.876 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.860+0000 7fe62b97f640 1 librados: starting objecter 2024-09-16T03:53:35.876 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.860+0000 7fe62b97f640 1 Processor -- start 2024-09-16T03:53:35.876 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.860+0000 7fe62b97f640 1 -- start start 2024-09-16T03:53:35.876 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.860+0000 7fe62b97f640 1 librados: setting wanted keys 2024-09-16T03:53:35.876 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.860+0000 7fe62b97f640 1 librados: calling monclient init 2024-09-16T03:53:35.876 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.860+0000 7fe62b97f640 10 monclient: init 2024-09-16T03:53:35.876 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.860+0000 7fe62b97f640 10 monclient: _reopen_session rank -1 2024-09-16T03:53:35.876 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.860+0000 7fe62b97f640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:53:35.876 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.860+0000 7fe62b97f640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe6240a1330 0x7fe62413bef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:53:35.876 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.860+0000 7fe62b97f640 10 monclient(hunting): picked mon.noname-c con 0x7fe6240a1330 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:53:35.876 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.860+0000 7fe62b97f640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe624135100 0x7fe624134310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:53:35.876 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.860+0000 7fe62b97f640 10 monclient(hunting): picked mon.noname-a con 0x7fe624135100 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:53:35.876 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.860+0000 7fe62b97f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe624134850 0x7fe62409cb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:53:35.876 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.860+0000 7fe62b97f640 10 monclient(hunting): picked mon.noname-b con 0x7fe624134850 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:53:35.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.860+0000 7fe62b97f640 10 monclient(hunting): start opening mon connection 2024-09-16T03:53:35.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.860+0000 7fe62b97f640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fe62413fb80 con 0x7fe624135100 2024-09-16T03:53:35.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.860+0000 7fe62b97f640 10 monclient(hunting): start opening mon connection 2024-09-16T03:53:35.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.860+0000 7fe62b97f640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe62409ef80 con 0x7fe624134850 2024-09-16T03:53:35.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.860+0000 7fe62b97f640 10 monclient(hunting): start opening mon connection 2024-09-16T03:53:35.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.860+0000 7fe62b97f640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fe62413fd00 con 0x7fe6240a1330 2024-09-16T03:53:35.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.860+0000 7fe62b97f640 10 monclient(hunting): _renew_subs 2024-09-16T03:53:35.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.860+0000 7fe62b97f640 10 monclient(hunting): authenticate will time out at 1662.832766s 2024-09-16T03:53:35.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.860+0000 7fe62971b640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe6240a1330 0x7fe62413bef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:53:35.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.860+0000 7fe62971b640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe6240a1330 0x7fe62413bef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:39802/0 (socket says 172.21.15.42:39802) 2024-09-16T03:53:35.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.860+0000 7fe62971b640 1 -- 172.21.15.42:0/783089505 learned_addr learned my addr 172.21.15.42:0/783089505 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:53:35.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.860+0000 7fe62971b640 10 monclient(hunting): get_auth_request con 0x7fe6240a1330 auth_method 0 2024-09-16T03:53:35.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.860+0000 7fe62971b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:53:35.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.860+0000 7fe62971b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:53:35.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.860+0000 7fe62971b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:53:35.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.860+0000 7fe628f1a640 1 --2- 172.21.15.42:0/783089505 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe624135100 0x7fe624134310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:53:35.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.860+0000 7fe628f1a640 10 monclient(hunting): get_auth_request con 0x7fe624135100 auth_method 0 2024-09-16T03:53:35.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.860+0000 7fe628f1a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:53:35.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.860+0000 7fe628f1a640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:53:35.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.860+0000 7fe628f1a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:53:35.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.860+0000 7fe62971b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:53:35.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.860+0000 7fe62971b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:53:35.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.860+0000 7fe62971b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:53:35.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.860+0000 7fe628f1a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:53:35.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.860+0000 7fe628f1a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:53:35.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.860+0000 7fe628f1a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:53:35.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.860+0000 7fe629f1c640 1 --2- 172.21.15.42:0/783089505 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe624134850 0x7fe62409cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:53:35.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.860+0000 7fe62971b640 10 monclient(hunting): handle_auth_done global_id 834194 payload 824 2024-09-16T03:53:35.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.860+0000 7fe62971b640 1 -- 172.21.15.42:0/783089505 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe624134850 msgr2=0x7fe62409cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:53:35.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.860+0000 7fe62971b640 1 --2- 172.21.15.42:0/783089505 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe624134850 0x7fe62409cb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:53:35.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.860+0000 7fe62971b640 1 -- 172.21.15.42:0/783089505 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe624135100 msgr2=0x7fe624134310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:53:35.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.860+0000 7fe62971b640 1 --2- 172.21.15.42:0/783089505 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe624135100 0x7fe624134310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:53:35.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.860+0000 7fe62971b640 10 monclient: _finish_hunting 0 2024-09-16T03:53:35.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.860+0000 7fe62971b640 1 monclient: found mon.noname-c 2024-09-16T03:53:35.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.860+0000 7fe62971b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:53:35.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.860+0000 7fe62971b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:53:35.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.860+0000 7fe62971b640 1 -- 172.21.15.42:0/783089505 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe624120cc0 con 0x7fe6240a1330 2024-09-16T03:53:35.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.860+0000 7fe629f1c640 10 monclient: get_auth_request con 0x7fe624134850 auth_method 0 2024-09-16T03:53:35.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.860+0000 7fe629f1c640 1 --2- 172.21.15.42:0/783089505 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe624134850 0x7fe62409cb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T03:53:35.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.860+0000 7fe628f1a640 1 --2- 172.21.15.42:0/783089505 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe624135100 0x7fe624134310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:53:35.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.860+0000 7fe62971b640 1 --2- 172.21.15.42:0/783089505 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe6240a1330 0x7fe62413bef0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fe6140194f0 tx=0x7fe614019680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:53:35.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.860+0000 7fe621ffb640 10 client.?.objecter ms_handle_connect 0x7fe6240a1330 2024-09-16T03:53:35.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.860+0000 7fe621ffb640 10 client.?.objecter resend_mon_ops 2024-09-16T03:53:35.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.860+0000 7fe621ffb640 1 -- 172.21.15.42:0/783089505 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6140bd020 con 0x7fe6240a1330 2024-09-16T03:53:35.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.860+0000 7fe621ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:53:35.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.860+0000 7fe621ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:53:35.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.860+0000 7fe621ffb640 10 monclient: dump: 2024-09-16T03:53:35.880 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:53:35.880 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:53:35.880 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:35.880 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:35.880 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:53:35.880 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:53:35.880 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:53:35.880 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:53:35.880 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:53:35.881 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:53:35.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.860+0000 7fe621ffb640 10 monclient: _finish_auth 0 2024-09-16T03:53:35.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.860+0000 7fe621ffb640 10 monclient: _check_auth_tickets 2024-09-16T03:53:35.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.860+0000 7fe621ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:53:35.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.860+0000 7fe621ffb640 1 -- 172.21.15.42:0/783089505 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe6140ad740 con 0x7fe6240a1330 2024-09-16T03:53:35.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.860+0000 7fe621ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T03:53:35.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.860+0000 7fe621ffb640 1 -- 172.21.15.42:0/783089505 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6140b2040 con 0x7fe6240a1330 2024-09-16T03:53:35.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.860+0000 7fe621ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:53:35.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.860+0000 7fe621ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:53:35.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.860+0000 7fe621ffb640 10 monclient: dump: 2024-09-16T03:53:35.881 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:53:35.881 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:53:35.881 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:35.881 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:35.882 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:53:35.882 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:53:35.882 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:53:35.882 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:53:35.882 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:53:35.882 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:53:35.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.860+0000 7fe6237fe640 4 set_mon_vals no callback set 2024-09-16T03:53:35.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.860+0000 7fe62b97f640 5 monclient: authenticate success, global_id 834194 2024-09-16T03:53:35.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.860+0000 7fe62b97f640 10 monclient: _renew_subs 2024-09-16T03:53:35.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.860+0000 7fe62b97f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:53:35.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.860+0000 7fe62b97f640 1 -- 172.21.15.42:0/783089505 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe624071a20 con 0x7fe6240a1330 2024-09-16T03:53:35.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.860+0000 7fe62b97f640 10 client.834194.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:53:35.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.860+0000 7fe62b97f640 10 monclient: _renew_subs 2024-09-16T03:53:35.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.860+0000 7fe62b97f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:53:35.883 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.860+0000 7fe62b97f640 1 -- 172.21.15.42:0/783089505 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fe62413cb30 con 0x7fe6240a1330 2024-09-16T03:53:35.883 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.860+0000 7fe62b97f640 1 librados: init done 2024-09-16T03:53:35.883 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.860+0000 7fe62b97f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:53:35.883 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.860+0000 7fe62b97f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:53:35.883 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.860+0000 7fe62b97f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:53:35.883 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.860+0000 7fe621ffb640 1 -- 172.21.15.42:0/783089505 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fe6140c0020 con 0x7fe6240a1330 2024-09-16T03:53:35.883 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.860+0000 7fe62b97f640 1 -- 172.21.15.42:0/783089505 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe624135550 con 0x7fe6240a1330 2024-09-16T03:53:35.883 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.860+0000 7fe621ffb640 1 --2- 172.21.15.42:0/783089505 >> v2:172.21.15.154:6812/254845342 conn(0x7fe610037810 0x7fe610039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:53:35.883 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.860+0000 7fe621ffb640 1 -- 172.21.15.42:0/783089505 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(359..359 src has 286..359) ==== 5328+0+0 (secure 0 0 0) 0x7fe6140e3ee0 con 0x7fe6240a1330 2024-09-16T03:53:35.883 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.860+0000 7fe621ffb640 10 client.834194.objecter ms_dispatch 0x7fe624000ba0 osd_map(359..359 src has 286..359) 2024-09-16T03:53:35.883 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.860+0000 7fe621ffb640 3 client.834194.objecter handle_osd_map got epochs [359,359] > 0 2024-09-16T03:53:35.883 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.860+0000 7fe621ffb640 3 client.834194.objecter handle_osd_map decoding full epoch 359 2024-09-16T03:53:35.883 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.860+0000 7fe621ffb640 20 client.834194.objecter dump_active .. 0 homeless 2024-09-16T03:53:35.883 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.860+0000 7fe628f1a640 1 --2- 172.21.15.42:0/783089505 >> v2:172.21.15.154:6812/254845342 conn(0x7fe610037810 0x7fe610039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:53:35.883 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.860+0000 7fe628f1a640 10 monclient: get_auth_request con 0x7fe610037810 auth_method 0 2024-09-16T03:53:35.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.864+0000 7fe621ffb640 10 client.834194.objecter ms_handle_connect 0x7fe610037810 2024-09-16T03:53:35.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.864+0000 7fe628f1a640 1 --2- 172.21.15.42:0/783089505 >> v2:172.21.15.154:6812/254845342 conn(0x7fe610037810 0x7fe610039cd0 secure :-1 s=READY pgs=3597 cs=0 l=1 rev1=1 crypto rx=0x7fe6241350d0 tx=0x7fe618008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:53:35.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.868+0000 7fe621ffb640 1 -- 172.21.15.42:0/783089505 <== mon.2 v2:172.21.15.42:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7fe6140c7030 con 0x7fe6240a1330 2024-09-16T03:53:35.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.868+0000 7fe621ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:53:35.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:35.868+0000 7fe621ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:53:36.099 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:36.092+0000 7fe62b97f640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:53:36.099 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:36.092+0000 7fe62b97f640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:53:36.099 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:36.092+0000 7fe62b97f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:53:36.099 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:36.092+0000 7fe62b97f640 1 -- 172.21.15.42:0/783089505 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fe624069040 con 0x7fe6240a1330 2024-09-16T03:53:36.099 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:36.092+0000 7fe621ffb640 1 -- 172.21.15.42:0/783089505 <== mon.2 v2:172.21.15.42:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1555 (secure 0 0 0) 0x7fe614016500 con 0x7fe6240a1330 2024-09-16T03:53:36.099 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:36.092+0000 7fe621ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:53:36.100 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:36.092+0000 7fe621ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:53:36.100 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":98,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":22,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:53:36.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:36.096+0000 7fe62b97f640 10 librados: watch_flush enter 2024-09-16T03:53:36.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:36.096+0000 7fe62b97f640 10 librados: watch_flush exit 2024-09-16T03:53:36.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:36.096+0000 7fe62b97f640 20 client.834194.objecter shutdown clearing up homeless session... 2024-09-16T03:53:36.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:36.096+0000 7fe62b97f640 10 client.834194.objecter successfully canceled tick 2024-09-16T03:53:36.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:36.096+0000 7fe62b97f640 1 -- 172.21.15.42:0/783089505 >> v2:172.21.15.154:6812/254845342 conn(0x7fe610037810 msgr2=0x7fe610039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:53:36.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:36.096+0000 7fe62b97f640 1 --2- 172.21.15.42:0/783089505 >> v2:172.21.15.154:6812/254845342 conn(0x7fe610037810 0x7fe610039cd0 secure :-1 s=READY pgs=3597 cs=0 l=1 rev1=1 crypto rx=0x7fe6241350d0 tx=0x7fe618008040 comp rx=0 tx=0).stop 2024-09-16T03:53:36.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:36.096+0000 7fe62b97f640 10 monclient: shutdown 2024-09-16T03:53:36.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:36.096+0000 7fe62b97f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:53:36.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:36.096+0000 7fe62b97f640 1 -- 172.21.15.42:0/783089505 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe6240a1330 msgr2=0x7fe62413bef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:53:36.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:36.096+0000 7fe62b97f640 1 --2- 172.21.15.42:0/783089505 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe6240a1330 0x7fe62413bef0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fe6140194f0 tx=0x7fe614019680 comp rx=0 tx=0).stop 2024-09-16T03:53:36.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:36.096+0000 7fe62b97f640 1 -- 172.21.15.42:0/783089505 shutdown_connections 2024-09-16T03:53:36.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:36.096+0000 7fe62b97f640 1 --2- 172.21.15.42:0/783089505 >> v2:172.21.15.154:6812/254845342 conn(0x7fe610037810 0x7fe610039cd0 unknown :-1 s=CLOSED pgs=3597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:53:36.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:36.096+0000 7fe62b97f640 1 --2- 172.21.15.42:0/783089505 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe624134850 0x7fe62409cb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:53:36.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:36.096+0000 7fe62b97f640 1 --2- 172.21.15.42:0/783089505 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe624135100 0x7fe624134310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:53:36.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:36.096+0000 7fe62b97f640 1 --2- 172.21.15.42:0/783089505 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe6240a1330 0x7fe62413bef0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:53:36.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:36.096+0000 7fe62b97f640 1 -- 172.21.15.42:0/783089505 >> 172.21.15.42:0/783089505 conn(0x7fe6240a2420 msgr2=0x7fe6240a40d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:53:36.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:36.096+0000 7fe62b97f640 1 -- 172.21.15.42:0/783089505 shutdown_connections 2024-09-16T03:53:36.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:36.096+0000 7fe62b97f640 1 -- 172.21.15.42:0/783089505 wait complete. 2024-09-16T03:53:36.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:36.096+0000 7fe62b97f640 1 librados: shutdown 2024-09-16T03:53:39.120 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:53:39.254 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.248+0000 7f2ed9821640 10 monclient: get_monmap_and_config 2024-09-16T03:53:39.255 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.248+0000 7f2ed9821640 10 monclient: build_initial_monmap 2024-09-16T03:53:39.255 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.248+0000 7f2ed9821640 10 monclient: monmap: 2024-09-16T03:53:39.255 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:53:39.255 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:53:39.255 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:53:39.253453+0000 2024-09-16T03:53:39.255 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:53:39.253453+0000 2024-09-16T03:53:39.255 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:53:39.255 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:53:39.255 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:53:39.255 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:53:39.255 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:53:39.255 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:53:39.255 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:53:39.256 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:53:39.257 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.248+0000 7f2ed9821640 1 Processor -- start 2024-09-16T03:53:39.257 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.248+0000 7f2ed9821640 1 -- start start 2024-09-16T03:53:39.257 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.248+0000 7f2ed9821640 10 monclient: init 2024-09-16T03:53:39.258 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.248+0000 7f2ed9821640 10 monclient: _reopen_session rank -1 2024-09-16T03:53:39.258 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.248+0000 7f2ed9821640 10 monclient: _add_conns ranks=[0,3,1,2,4] 2024-09-16T03:53:39.258 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.248+0000 7f2ed9821640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2ed40a39b0 addr v1:172.21.15.42:6789/0 2024-09-16T03:53:39.258 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.248+0000 7f2ed9821640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f2ed40a3d90 0x7f2ed413c380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:53:39.258 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.248+0000 7f2ed9821640 10 monclient(hunting): picked mon.noname-a con 0x7f2ed40a3d90 addr v2:172.21.15.42:3300/0 2024-09-16T03:53:39.258 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.248+0000 7f2ed9821640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2ed413c8c0 addr v1:172.21.15.154:6789/0 2024-09-16T03:53:39.258 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.248+0000 7f2ed9821640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2ed4134290 con 0x7f2ed40a39b0 2024-09-16T03:53:39.258 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.248+0000 7f2ed9821640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2ed409c140 con 0x7f2ed413c8c0 2024-09-16T03:53:39.258 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.248+0000 7f2ed9821640 10 monclient(hunting): start opening mon connection 2024-09-16T03:53:39.258 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.248+0000 7f2ed9821640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f2ed409c420 con 0x7f2ed40a3d90 2024-09-16T03:53:39.258 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.248+0000 7f2ed9821640 10 monclient(hunting): _renew_subs 2024-09-16T03:53:39.258 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.248+0000 7f2ed9821640 10 monclient(hunting): authenticate will time out at 1666.223698s 2024-09-16T03:53:39.258 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.248+0000 7f2ed27fc640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f2ed40a3d90 0x7f2ed413c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:53:39.259 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.248+0000 7f2ed27fc640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f2ed40a3d90 0x7f2ed413c380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:44482/0 (socket says 172.21.15.42:44482) 2024-09-16T03:53:39.259 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.248+0000 7f2ed27fc640 1 -- 172.21.15.42:0/606774055 learned_addr learned my addr 172.21.15.42:0/606774055 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:53:39.259 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.248+0000 7f2ed27fc640 10 monclient(hunting): get_auth_request con 0x7f2ed40a3d90 auth_method 0 2024-09-16T03:53:39.259 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.248+0000 7f2ed27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:53:39.259 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.248+0000 7f2ed27fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:53:39.259 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.248+0000 7f2ed27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:53:39.259 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.248+0000 7f2ed2ffd640 1 --1- 172.21.15.42:0/606774055 >> v1:172.21.15.42:6789/0 conn(0x7f2ed40a39b0 0x7f2ed4133b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:50166/0 (socket says 172.21.15.42:50166) 2024-09-16T03:53:39.259 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.252+0000 7f2ed27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:53:39.259 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.252+0000 7f2ed27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:53:39.259 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.252+0000 7f2ed27fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:53:39.259 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.252+0000 7f2ed27fc640 10 monclient(hunting): handle_auth_done global_id 824322 payload 274 2024-09-16T03:53:39.259 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.252+0000 7f2ed27fc640 1 -- 172.21.15.42:0/606774055 >> v1:172.21.15.154:6789/0 conn(0x7f2ed413c8c0 legacy=0x7f2ed413ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:53:39.259 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.252+0000 7f2ed27fc640 1 -- 172.21.15.42:0/606774055 >> v1:172.21.15.42:6789/0 conn(0x7f2ed40a39b0 legacy=0x7f2ed4133b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:53:39.259 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.252+0000 7f2ed27fc640 10 monclient: _finish_hunting 0 2024-09-16T03:53:39.260 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.252+0000 7f2ed27fc640 1 monclient: found mon.noname-a 2024-09-16T03:53:39.260 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.252+0000 7f2ed27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:53:39.260 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.252+0000 7f2ed27fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:53:39.260 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.252+0000 7f2ed1ffb640 1 -- 172.21.15.42:0/606774055 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2228696218 0 0) 0x7f2ed4134290 con 0x7f2ed40a39b0 2024-09-16T03:53:39.260 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.252+0000 7f2ed27fc640 1 -- 172.21.15.42:0/606774055 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ed406dd60 con 0x7f2ed40a3d90 2024-09-16T03:53:39.260 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.252+0000 7f2ed1ffb640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T03:53:39.265 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.252+0000 7f2ed27fc640 1 --2- 172.21.15.42:0/606774055 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2ed40a3d90 0x7f2ed413c380 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f2ec8009870 tx=0x7f2ec801c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce4d126312b35db8 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:53:39.265 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.252+0000 7f2ed1ffb640 1 -- 172.21.15.42:0/606774055 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ec80b0070 con 0x7f2ed40a3d90 2024-09-16T03:53:39.265 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.252+0000 7f2ed1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:53:39.265 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.252+0000 7f2ed1ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:53:39.266 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.252+0000 7f2ed1ffb640 10 monclient: dump: 2024-09-16T03:53:39.266 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:53:39.266 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:53:39.266 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:39.266 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:39.266 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:53:39.266 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:53:39.266 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:53:39.267 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:53:39.267 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:53:39.267 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:53:39.267 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.252+0000 7f2ed1ffb640 10 monclient: _finish_auth 0 2024-09-16T03:53:39.267 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.252+0000 7f2ed1ffb640 10 monclient: _check_auth_tickets 2024-09-16T03:53:39.267 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.252+0000 7f2ed1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:53:39.267 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.252+0000 7f2ed1ffb640 1 -- 172.21.15.42:0/606774055 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ec80ab600 con 0x7f2ed40a3d90 2024-09-16T03:53:39.267 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.252+0000 7f2ed1ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T03:53:39.267 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.252+0000 7f2ed1ffb640 1 -- 172.21.15.42:0/606774055 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ec80ab9d0 con 0x7f2ed40a3d90 2024-09-16T03:53:39.267 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.252+0000 7f2ed9821640 5 monclient: authenticate success, global_id 824322 2024-09-16T03:53:39.267 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.252+0000 7f2ed9821640 10 monclient: get_monmap_and_config success 2024-09-16T03:53:39.267 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.252+0000 7f2ed9821640 4 set_mon_vals no callback set 2024-09-16T03:53:39.267 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.252+0000 7f2ed1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:53:39.267 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.252+0000 7f2ed1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:53:39.268 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.252+0000 7f2ed1ffb640 10 monclient: dump: 2024-09-16T03:53:39.268 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:53:39.268 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:53:39.268 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:39.268 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:39.268 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:53:39.268 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:53:39.268 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:53:39.268 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:53:39.268 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:53:39.268 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:53:39.268 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.252+0000 7f2ed9821640 10 monclient: shutdown 2024-09-16T03:53:39.269 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.252+0000 7f2ed9821640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:53:39.269 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.252+0000 7f2ed9821640 1 -- 172.21.15.42:0/606774055 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2ed40a3d90 msgr2=0x7f2ed413c380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:53:39.269 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.252+0000 7f2ed9821640 1 --2- 172.21.15.42:0/606774055 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2ed40a3d90 0x7f2ed413c380 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f2ec8009870 tx=0x7f2ec801c3e0 comp rx=0 tx=0).stop 2024-09-16T03:53:39.269 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.252+0000 7f2ed9821640 1 -- 172.21.15.42:0/606774055 shutdown_connections 2024-09-16T03:53:39.269 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.252+0000 7f2ed9821640 1 --2- 172.21.15.42:0/606774055 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2ed40a3d90 0x7f2ed413c380 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:53:39.269 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.252+0000 7f2ed9821640 1 -- 172.21.15.42:0/606774055 >> 172.21.15.42:0/606774055 conn(0x7f2ed40a9fb0 msgr2=0x7f2ed40aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:53:39.269 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.252+0000 7f2ed9821640 1 -- 172.21.15.42:0/606774055 shutdown_connections 2024-09-16T03:53:39.269 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.252+0000 7f2ed9821640 1 -- 172.21.15.42:0/606774055 wait complete. 2024-09-16T03:53:39.269 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.252+0000 7f2ed9821640 10 monclient: build_initial_monmap 2024-09-16T03:53:39.269 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.252+0000 7f2ed9821640 10 monclient: monmap: 2024-09-16T03:53:39.269 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:53:39.269 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:53:39.269 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:53:39.269 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:53:39.269 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:53:39.270 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:53:39.270 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:53:39.270 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:53:39.270 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:53:39.270 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:53:39.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.256+0000 7f2ed9821640 1 librados: starting msgr at 2024-09-16T03:53:39.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.256+0000 7f2ed9821640 1 librados: starting objecter 2024-09-16T03:53:39.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.256+0000 7f2ed9821640 1 Processor -- start 2024-09-16T03:53:39.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.256+0000 7f2ed9821640 1 -- start start 2024-09-16T03:53:39.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.256+0000 7f2ed9821640 1 librados: setting wanted keys 2024-09-16T03:53:39.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.256+0000 7f2ed9821640 1 librados: calling monclient init 2024-09-16T03:53:39.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.256+0000 7f2ed9821640 10 monclient: init 2024-09-16T03:53:39.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.256+0000 7f2ed9821640 10 monclient: _reopen_session rank -1 2024-09-16T03:53:39.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.256+0000 7f2ed9821640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T03:53:39.271 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.256+0000 7f2ed9821640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2ed40a39b0 0x7f2ed41308b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:53:39.271 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.256+0000 7f2ed9821640 10 monclient(hunting): picked mon.noname-b con 0x7f2ed40a39b0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:53:39.271 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.256+0000 7f2ed9821640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2ed40a3d90 0x7f2ed4139070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:53:39.271 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.256+0000 7f2ed9821640 10 monclient(hunting): picked mon.noname-c con 0x7f2ed40a3d90 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:53:39.271 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.256+0000 7f2ed9821640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2ed413c8c0 0x7f2ed41395b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:53:39.271 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.256+0000 7f2ed9821640 10 monclient(hunting): picked mon.noname-a con 0x7f2ed413c8c0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:53:39.271 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.256+0000 7f2ed9821640 10 monclient(hunting): start opening mon connection 2024-09-16T03:53:39.271 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.256+0000 7f2ed9821640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f2ed4142f80 con 0x7f2ed413c8c0 2024-09-16T03:53:39.271 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.256+0000 7f2ed9821640 10 monclient(hunting): start opening mon connection 2024-09-16T03:53:39.271 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.256+0000 7f2ed9821640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f2ed4142e00 con 0x7f2ed40a39b0 2024-09-16T03:53:39.271 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.256+0000 7f2ed9821640 10 monclient(hunting): start opening mon connection 2024-09-16T03:53:39.271 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.256+0000 7f2ed9821640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f2ed4143100 con 0x7f2ed40a3d90 2024-09-16T03:53:39.271 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.256+0000 7f2ed9821640 10 monclient(hunting): _renew_subs 2024-09-16T03:53:39.271 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.256+0000 7f2ed9821640 10 monclient(hunting): authenticate will time out at 1666.228978s 2024-09-16T03:53:39.272 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.256+0000 7f2ed37fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2ed413c8c0 0x7f2ed41395b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:53:39.272 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.256+0000 7f2ed27fc640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2ed40a3d90 0x7f2ed4139070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:53:39.272 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.256+0000 7f2ed37fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2ed413c8c0 0x7f2ed41395b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:44488/0 (socket says 172.21.15.42:44488) 2024-09-16T03:53:39.272 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.256+0000 7f2ed37fe640 1 -- 172.21.15.42:0/1132612758 learned_addr learned my addr 172.21.15.42:0/1132612758 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:53:39.272 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.256+0000 7f2ed27fc640 10 monclient(hunting): get_auth_request con 0x7f2ed40a3d90 auth_method 0 2024-09-16T03:53:39.272 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.256+0000 7f2ed27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:53:39.272 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.256+0000 7f2ed27fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:53:39.272 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.256+0000 7f2ed27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:53:39.272 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.256+0000 7f2ed37fe640 10 monclient(hunting): get_auth_request con 0x7f2ed413c8c0 auth_method 0 2024-09-16T03:53:39.272 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.256+0000 7f2ed37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:53:39.272 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.256+0000 7f2ed37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:53:39.272 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.256+0000 7f2ed37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:53:39.272 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.256+0000 7f2ed27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:53:39.273 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.256+0000 7f2ed27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:53:39.273 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.256+0000 7f2ed27fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:53:39.273 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.256+0000 7f2ed37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:53:39.273 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.256+0000 7f2ed37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:53:39.273 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.256+0000 7f2ed37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:53:39.273 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.256+0000 7f2ed2ffd640 1 --2- 172.21.15.42:0/1132612758 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2ed40a39b0 0x7f2ed41308b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:53:39.273 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.256+0000 7f2ed2ffd640 10 monclient(hunting): get_auth_request con 0x7f2ed40a39b0 auth_method 0 2024-09-16T03:53:39.273 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.256+0000 7f2ed2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:53:39.273 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.256+0000 7f2ed2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:53:39.273 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.256+0000 7f2ed2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:53:39.273 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.256+0000 7f2ed27fc640 10 monclient(hunting): handle_auth_done global_id 834197 payload 824 2024-09-16T03:53:39.273 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.256+0000 7f2ed27fc640 1 -- 172.21.15.42:0/1132612758 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2ed40a39b0 msgr2=0x7f2ed41308b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:53:39.273 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.256+0000 7f2ed27fc640 1 --2- 172.21.15.42:0/1132612758 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2ed40a39b0 0x7f2ed41308b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:53:39.273 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.256+0000 7f2ed27fc640 1 -- 172.21.15.42:0/1132612758 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2ed413c8c0 msgr2=0x7f2ed41395b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:53:39.274 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.256+0000 7f2ed27fc640 1 --2- 172.21.15.42:0/1132612758 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2ed413c8c0 0x7f2ed41395b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:53:39.274 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.256+0000 7f2ed27fc640 10 monclient: _finish_hunting 0 2024-09-16T03:53:39.274 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.256+0000 7f2ed27fc640 1 monclient: found mon.noname-c 2024-09-16T03:53:39.274 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.256+0000 7f2ed27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:53:39.274 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.256+0000 7f2ed27fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:53:39.274 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.256+0000 7f2ed27fc640 1 -- 172.21.15.42:0/1132612758 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ed409d410 con 0x7f2ed40a3d90 2024-09-16T03:53:39.274 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.256+0000 7f2ed37fe640 1 --2- 172.21.15.42:0/1132612758 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2ed413c8c0 0x7f2ed41395b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:53:39.274 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.256+0000 7f2ed27fc640 1 --2- 172.21.15.42:0/1132612758 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2ed40a3d90 0x7f2ed4139070 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f2ec80ab000 tx=0x7f2ec801ac20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:53:39.274 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.256+0000 7f2ebbfff640 10 client.?.objecter ms_handle_connect 0x7f2ed40a3d90 2024-09-16T03:53:39.274 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.256+0000 7f2ebbfff640 10 client.?.objecter resend_mon_ops 2024-09-16T03:53:39.274 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.256+0000 7f2ebbfff640 1 -- 172.21.15.42:0/1132612758 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ec80ba020 con 0x7f2ed40a3d90 2024-09-16T03:53:39.274 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.256+0000 7f2ebbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:53:39.274 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.256+0000 7f2ebbfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:53:39.274 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.256+0000 7f2ebbfff640 10 monclient: dump: 2024-09-16T03:53:39.275 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:53:39.275 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:53:39.275 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:39.275 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:39.275 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:53:39.275 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:53:39.275 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:53:39.275 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:53:39.275 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:53:39.275 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:53:39.275 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.256+0000 7f2ebbfff640 10 monclient: _finish_auth 0 2024-09-16T03:53:39.275 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.256+0000 7f2ebbfff640 10 monclient: _check_auth_tickets 2024-09-16T03:53:39.275 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.256+0000 7f2ebbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:53:39.275 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.256+0000 7f2ebbfff640 1 -- 172.21.15.42:0/1132612758 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ec80abe50 con 0x7f2ed40a3d90 2024-09-16T03:53:39.275 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.256+0000 7f2ed9821640 5 monclient: authenticate success, global_id 834197 2024-09-16T03:53:39.276 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.256+0000 7f2ed9821640 10 monclient: _renew_subs 2024-09-16T03:53:39.276 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.256+0000 7f2ed9821640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:53:39.276 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.256+0000 7f2ed9821640 1 -- 172.21.15.42:0/1132612758 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ed413c200 con 0x7f2ed40a3d90 2024-09-16T03:53:39.276 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.256+0000 7f2ebbfff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:53:39.276 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.256+0000 7f2ebbfff640 1 -- 172.21.15.42:0/1132612758 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ec80b0040 con 0x7f2ed40a3d90 2024-09-16T03:53:39.276 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.256+0000 7f2ebbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:53:39.276 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.256+0000 7f2ebbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:53:39.276 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.256+0000 7f2ebbfff640 10 monclient: dump: 2024-09-16T03:53:39.276 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:53:39.276 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:53:39.276 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:39.276 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:39.276 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:53:39.277 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:53:39.277 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:53:39.277 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:53:39.277 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:53:39.277 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:53:39.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.256+0000 7f2ed9821640 10 client.834197.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:53:39.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.256+0000 7f2ed9821640 10 monclient: _renew_subs 2024-09-16T03:53:39.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.256+0000 7f2ed9821640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:53:39.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.256+0000 7f2ed9821640 1 -- 172.21.15.42:0/1132612758 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ed409d130 con 0x7f2ed40a3d90 2024-09-16T03:53:39.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.256+0000 7f2ed9821640 1 librados: init done 2024-09-16T03:53:39.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.256+0000 7f2ed17fa640 4 set_mon_vals no callback set 2024-09-16T03:53:39.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.256+0000 7f2ebbfff640 1 -- 172.21.15.42:0/1132612758 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f2ec80bd020 con 0x7f2ed40a3d90 2024-09-16T03:53:39.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.256+0000 7f2ed9821640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:53:39.278 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.256+0000 7f2ed9821640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:53:39.278 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.256+0000 7f2ed9821640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:53:39.278 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.256+0000 7f2ed9821640 1 -- 172.21.15.42:0/1132612758 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ed4133b80 con 0x7f2ed40a3d90 2024-09-16T03:53:39.278 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.256+0000 7f2ebbfff640 1 --2- 172.21.15.42:0/1132612758 >> v2:172.21.15.154:6812/254845342 conn(0x7f2ec0037810 0x7f2ec0039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:53:39.278 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.256+0000 7f2ebbfff640 1 -- 172.21.15.42:0/1132612758 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(362..362 src has 286..362) ==== 5328+0+0 (secure 0 0 0) 0x7f2ec80e3260 con 0x7f2ed40a3d90 2024-09-16T03:53:39.278 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.256+0000 7f2ebbfff640 10 client.834197.objecter ms_dispatch 0x7f2ed4000ba0 osd_map(362..362 src has 286..362) 2024-09-16T03:53:39.278 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.256+0000 7f2ebbfff640 3 client.834197.objecter handle_osd_map got epochs [362,362] > 0 2024-09-16T03:53:39.278 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.256+0000 7f2ebbfff640 3 client.834197.objecter handle_osd_map decoding full epoch 362 2024-09-16T03:53:39.278 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.256+0000 7f2ebbfff640 20 client.834197.objecter dump_active .. 0 homeless 2024-09-16T03:53:39.278 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.256+0000 7f2ed2ffd640 1 --2- 172.21.15.42:0/1132612758 >> v2:172.21.15.154:6812/254845342 conn(0x7f2ec0037810 0x7f2ec0039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:53:39.278 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.256+0000 7f2ed2ffd640 10 monclient: get_auth_request con 0x7f2ec0037810 auth_method 0 2024-09-16T03:53:39.278 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.260+0000 7f2ed2ffd640 1 --2- 172.21.15.42:0/1132612758 >> v2:172.21.15.154:6812/254845342 conn(0x7f2ec0037810 0x7f2ec0039cd0 secure :-1 s=READY pgs=3598 cs=0 l=1 rev1=1 crypto rx=0x7f2ec4000f70 tx=0x7f2ec4007c10 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:53:39.278 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.260+0000 7f2ebbfff640 10 client.834197.objecter ms_handle_connect 0x7f2ec0037810 2024-09-16T03:53:39.279 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.264+0000 7f2ebbfff640 1 -- 172.21.15.42:0/1132612758 <== mon.2 v2:172.21.15.42:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f2ec80c4030 con 0x7f2ed40a3d90 2024-09-16T03:53:39.279 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.264+0000 7f2ebbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:53:39.279 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.264+0000 7f2ebbfff640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:53:39.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.504+0000 7f2ed9821640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:53:39.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.504+0000 7f2ed9821640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:53:39.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.504+0000 7f2ed9821640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:53:39.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.504+0000 7f2ed9821640 1 -- 172.21.15.42:0/1132612758 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f2ed40a5750 con 0x7f2ed40a3d90 2024-09-16T03:53:39.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.504+0000 7f2ebbfff640 1 -- 172.21.15.42:0/1132612758 <== mon.2 v2:172.21.15.42:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1555 (secure 0 0 0) 0x7f2ec80160f0 con 0x7f2ed40a3d90 2024-09-16T03:53:39.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.504+0000 7f2ebbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:53:39.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.504+0000 7f2ebbfff640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:53:39.511 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":98,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":25,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:53:39.514 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.508+0000 7f2ed9821640 10 librados: watch_flush enter 2024-09-16T03:53:39.514 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.508+0000 7f2ed9821640 10 librados: watch_flush exit 2024-09-16T03:53:39.514 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.508+0000 7f2ed9821640 20 client.834197.objecter shutdown clearing up homeless session... 2024-09-16T03:53:39.514 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.508+0000 7f2ed9821640 10 client.834197.objecter successfully canceled tick 2024-09-16T03:53:39.514 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.508+0000 7f2ed9821640 1 -- 172.21.15.42:0/1132612758 >> v2:172.21.15.154:6812/254845342 conn(0x7f2ec0037810 msgr2=0x7f2ec0039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:53:39.514 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.508+0000 7f2ed9821640 1 --2- 172.21.15.42:0/1132612758 >> v2:172.21.15.154:6812/254845342 conn(0x7f2ec0037810 0x7f2ec0039cd0 secure :-1 s=READY pgs=3598 cs=0 l=1 rev1=1 crypto rx=0x7f2ec4000f70 tx=0x7f2ec4007c10 comp rx=0 tx=0).stop 2024-09-16T03:53:39.514 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.508+0000 7f2ed9821640 10 monclient: shutdown 2024-09-16T03:53:39.514 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.508+0000 7f2ed9821640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:53:39.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.508+0000 7f2ed9821640 1 -- 172.21.15.42:0/1132612758 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2ed40a3d90 msgr2=0x7f2ed4139070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:53:39.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.508+0000 7f2ed9821640 1 --2- 172.21.15.42:0/1132612758 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2ed40a3d90 0x7f2ed4139070 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f2ec80ab000 tx=0x7f2ec801ac20 comp rx=0 tx=0).stop 2024-09-16T03:53:39.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.508+0000 7f2ed9821640 1 -- 172.21.15.42:0/1132612758 shutdown_connections 2024-09-16T03:53:39.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.508+0000 7f2ed9821640 1 --2- 172.21.15.42:0/1132612758 >> v2:172.21.15.154:6812/254845342 conn(0x7f2ec0037810 0x7f2ec0039cd0 unknown :-1 s=CLOSED pgs=3598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:53:39.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.508+0000 7f2ed9821640 1 --2- 172.21.15.42:0/1132612758 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2ed413c8c0 0x7f2ed41395b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:53:39.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.508+0000 7f2ed9821640 1 --2- 172.21.15.42:0/1132612758 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2ed40a3d90 0x7f2ed4139070 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:53:39.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.508+0000 7f2ed9821640 1 --2- 172.21.15.42:0/1132612758 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2ed40a39b0 0x7f2ed41308b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:53:39.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.508+0000 7f2ed9821640 1 -- 172.21.15.42:0/1132612758 >> 172.21.15.42:0/1132612758 conn(0x7f2ed40a9fb0 msgr2=0x7f2ed40a8830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:53:39.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.508+0000 7f2ed9821640 1 -- 172.21.15.42:0/1132612758 shutdown_connections 2024-09-16T03:53:39.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.508+0000 7f2ed9821640 1 -- 172.21.15.42:0/1132612758 wait complete. 2024-09-16T03:53:39.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:39.508+0000 7f2ed9821640 1 librados: shutdown 2024-09-16T03:53:42.530 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:53:42.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.656+0000 7f8198f8e640 10 monclient: get_monmap_and_config 2024-09-16T03:53:42.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.656+0000 7f8198f8e640 10 monclient: build_initial_monmap 2024-09-16T03:53:42.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.656+0000 7f8198f8e640 10 monclient: monmap: 2024-09-16T03:53:42.664 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:53:42.664 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:53:42.664 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:53:42.662733+0000 2024-09-16T03:53:42.664 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:53:42.662733+0000 2024-09-16T03:53:42.664 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:53:42.664 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:53:42.664 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:53:42.665 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:53:42.665 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:53:42.665 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:53:42.665 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:53:42.665 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:53:42.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.656+0000 7f8198f8e640 1 Processor -- start 2024-09-16T03:53:42.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.656+0000 7f8198f8e640 1 -- start start 2024-09-16T03:53:42.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.656+0000 7f8198f8e640 10 monclient: init 2024-09-16T03:53:42.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.660+0000 7f8198f8e640 10 monclient: _reopen_session rank -1 2024-09-16T03:53:42.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.660+0000 7f8198f8e640 10 monclient: _add_conns ranks=[4,3,1,0,2] 2024-09-16T03:53:42.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.660+0000 7f8198f8e640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f81940a1330 0x7f81940a1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:53:42.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.660+0000 7f8198f8e640 10 monclient(hunting): picked mon.noname-c con 0x7f81940a1330 addr v2:172.21.15.154:3300/0 2024-09-16T03:53:42.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.660+0000 7f8198f8e640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f819406dac0 0x7f81940a1c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:53:42.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.660+0000 7f8198f8e640 10 monclient(hunting): picked mon.noname-a con 0x7f819406dac0 addr v2:172.21.15.42:3300/0 2024-09-16T03:53:42.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.660+0000 7f8198f8e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8194135100 addr v1:172.21.15.154:6789/0 2024-09-16T03:53:42.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.660+0000 7f8198f8e640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f819409c7d0 con 0x7f8194135100 2024-09-16T03:53:42.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.660+0000 7f8198f8e640 10 monclient(hunting): start opening mon connection 2024-09-16T03:53:42.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.660+0000 7f8198f8e640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f819409c090 con 0x7f819406dac0 2024-09-16T03:53:42.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.660+0000 7f8198f8e640 10 monclient(hunting): start opening mon connection 2024-09-16T03:53:42.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.660+0000 7f8198f8e640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f819409f610 con 0x7f81940a1330 2024-09-16T03:53:42.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.660+0000 7f8198f8e640 10 monclient(hunting): _renew_subs 2024-09-16T03:53:42.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.660+0000 7f8198f8e640 10 monclient(hunting): authenticate will time out at 1669.632181s 2024-09-16T03:53:42.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.660+0000 7f8191d9b640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f819406dac0 0x7f81940a1c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:53:42.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.660+0000 7f8191d9b640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f819406dac0 0x7f81940a1c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:44492/0 (socket says 172.21.15.42:44492) 2024-09-16T03:53:42.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.660+0000 7f8191d9b640 1 -- 172.21.15.42:0/1514849873 learned_addr learned my addr 172.21.15.42:0/1514849873 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:53:42.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.660+0000 7f8191d9b640 10 monclient(hunting): get_auth_request con 0x7f819406dac0 auth_method 0 2024-09-16T03:53:42.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.660+0000 7f8191d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:53:42.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.660+0000 7f8191d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:53:42.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.660+0000 7f8191d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:53:42.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.660+0000 7f8191d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:53:42.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.660+0000 7f8191d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:53:42.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.660+0000 7f8191d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:53:42.670 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.660+0000 7f8191d9b640 10 monclient(hunting): handle_auth_done global_id 824331 payload 274 2024-09-16T03:53:42.670 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.660+0000 7f8191d9b640 1 -- 172.21.15.42:0/1514849873 >> v2:172.21.15.154:3300/0 conn(0x7f81940a1330 msgr2=0x7f81940a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:53:42.670 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.660+0000 7f819259c640 1 --2- 172.21.15.42:0/1514849873 >> v2:172.21.15.154:3300/0 conn(0x7f81940a1330 0x7f81940a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:53:42.670 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.660+0000 7f8191d9b640 1 --2- 172.21.15.42:0/1514849873 >> v2:172.21.15.154:3300/0 conn(0x7f81940a1330 0x7f81940a1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:53:42.670 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.660+0000 7f8191d9b640 1 -- 172.21.15.42:0/1514849873 >> v1:172.21.15.154:6789/0 conn(0x7f8194135100 legacy=0x7f8194135550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:53:42.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.660+0000 7f8191d9b640 10 monclient: _finish_hunting 0 2024-09-16T03:53:42.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.660+0000 7f8191d9b640 1 monclient: found mon.noname-a 2024-09-16T03:53:42.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.660+0000 7f8191d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:53:42.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.660+0000 7f8191d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:53:42.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.660+0000 7f8191d9b640 1 -- 172.21.15.42:0/1514849873 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f819409f8f0 con 0x7f819406dac0 2024-09-16T03:53:42.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.660+0000 7f819259c640 10 monclient: get_auth_request con 0x7f81940a1330 auth_method 0 2024-09-16T03:53:42.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.660+0000 7f819259c640 1 --2- 172.21.15.42:0/1514849873 >> v2:172.21.15.154:3300/0 conn(0x7f81940a1330 0x7f81940a1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T03:53:42.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.660+0000 7f8191d9b640 1 --2- 172.21.15.42:0/1514849873 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f819406dac0 0x7f81940a1c50 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f819406add0 tx=0x7f818801c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e130f3d924f240da server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:53:42.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.660+0000 7f819159a640 1 -- 172.21.15.42:0/1514849873 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81880aa610 con 0x7f819406dac0 2024-09-16T03:53:42.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.660+0000 7f819159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:53:42.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.660+0000 7f819159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:53:42.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.660+0000 7f819159a640 10 monclient: dump: 2024-09-16T03:53:42.679 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:53:42.680 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:53:42.680 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:42.680 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:42.680 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:53:42.680 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:53:42.680 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:53:42.680 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:53:42.680 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:53:42.680 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:53:42.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.660+0000 7f819159a640 10 monclient: _finish_auth 0 2024-09-16T03:53:42.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.660+0000 7f819159a640 10 monclient: _check_auth_tickets 2024-09-16T03:53:42.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.660+0000 7f819159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:53:42.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.660+0000 7f819159a640 1 -- 172.21.15.42:0/1514849873 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f81880af070 con 0x7f819406dac0 2024-09-16T03:53:42.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.660+0000 7f8198f8e640 5 monclient: authenticate success, global_id 824331 2024-09-16T03:53:42.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.660+0000 7f8198f8e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:53:42.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.660+0000 7f819159a640 10 monclient: handle_config config(0 keys) 2024-09-16T03:53:42.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.660+0000 7f8198f8e640 10 monclient: get_monmap_and_config success 2024-09-16T03:53:42.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.660+0000 7f819159a640 1 -- 172.21.15.42:0/1514849873 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81880aa9d0 con 0x7f819406dac0 2024-09-16T03:53:42.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.660+0000 7f819159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:53:42.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.660+0000 7f8198f8e640 4 set_mon_vals no callback set 2024-09-16T03:53:42.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.660+0000 7f819159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:53:42.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.660+0000 7f819159a640 10 monclient: dump: 2024-09-16T03:53:42.681 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:53:42.681 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:53:42.682 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:42.682 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:42.682 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:53:42.682 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:53:42.682 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:53:42.682 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:53:42.682 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:53:42.682 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:53:42.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.660+0000 7f8198f8e640 10 monclient: shutdown 2024-09-16T03:53:42.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.660+0000 7f8198f8e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:53:42.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.660+0000 7f8198f8e640 1 -- 172.21.15.42:0/1514849873 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f819406dac0 msgr2=0x7f81940a1c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:53:42.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.660+0000 7f8198f8e640 1 --2- 172.21.15.42:0/1514849873 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f819406dac0 0x7f81940a1c50 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f819406add0 tx=0x7f818801c490 comp rx=0 tx=0).stop 2024-09-16T03:53:42.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.660+0000 7f8198f8e640 1 -- 172.21.15.42:0/1514849873 shutdown_connections 2024-09-16T03:53:42.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.660+0000 7f8198f8e640 1 --2- 172.21.15.42:0/1514849873 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f819406dac0 0x7f81940a1c50 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:53:42.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.660+0000 7f8198f8e640 1 --2- 172.21.15.42:0/1514849873 >> v2:172.21.15.154:3300/0 conn(0x7f81940a1330 0x7f81940a1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:53:42.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.660+0000 7f8198f8e640 1 -- 172.21.15.42:0/1514849873 >> 172.21.15.42:0/1514849873 conn(0x7f81940a2420 msgr2=0x7f81940a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:53:42.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.660+0000 7f8198f8e640 1 -- 172.21.15.42:0/1514849873 shutdown_connections 2024-09-16T03:53:42.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.660+0000 7f8198f8e640 1 -- 172.21.15.42:0/1514849873 wait complete. 2024-09-16T03:53:42.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.664+0000 7f8198f8e640 10 monclient: build_initial_monmap 2024-09-16T03:53:42.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.664+0000 7f8198f8e640 10 monclient: monmap: 2024-09-16T03:53:42.683 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:53:42.683 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:53:42.683 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:53:42.683 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:53:42.683 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:53:42.684 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:53:42.684 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:53:42.684 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:53:42.684 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:53:42.684 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:53:42.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.664+0000 7f8198f8e640 1 librados: starting msgr at 2024-09-16T03:53:42.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.664+0000 7f8198f8e640 1 librados: starting objecter 2024-09-16T03:53:42.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.664+0000 7f8198f8e640 1 Processor -- start 2024-09-16T03:53:42.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.664+0000 7f8198f8e640 1 -- start start 2024-09-16T03:53:42.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.664+0000 7f8198f8e640 1 librados: setting wanted keys 2024-09-16T03:53:42.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.664+0000 7f8198f8e640 1 librados: calling monclient init 2024-09-16T03:53:42.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.664+0000 7f8198f8e640 10 monclient: init 2024-09-16T03:53:42.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.664+0000 7f8198f8e640 10 monclient: _reopen_session rank -1 2024-09-16T03:53:42.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.664+0000 7f8198f8e640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T03:53:42.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.664+0000 7f8198f8e640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f81940a1330 0x7f819413c210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:53:42.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.664+0000 7f8198f8e640 10 monclient(hunting): picked mon.noname-a con 0x7f81940a1330 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:53:42.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.664+0000 7f8198f8e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8194135100 0x7f819413c750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:53:42.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.664+0000 7f8198f8e640 10 monclient(hunting): picked mon.noname-b con 0x7f8194135100 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:53:42.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.664+0000 7f8198f8e640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f81941cad50 0x7f81941cb1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:53:42.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.664+0000 7f8198f8e640 10 monclient(hunting): picked mon.noname-c con 0x7f81941cad50 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:53:42.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.664+0000 7f8198f8e640 10 monclient(hunting): start opening mon connection 2024-09-16T03:53:42.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.664+0000 7f8198f8e640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f819409cef0 con 0x7f81940a1330 2024-09-16T03:53:42.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.664+0000 7f8198f8e640 10 monclient(hunting): start opening mon connection 2024-09-16T03:53:42.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.664+0000 7f8198f8e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f819409cd70 con 0x7f8194135100 2024-09-16T03:53:42.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.664+0000 7f8198f8e640 10 monclient(hunting): start opening mon connection 2024-09-16T03:53:42.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.664+0000 7f8198f8e640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f819409d070 con 0x7f81941cad50 2024-09-16T03:53:42.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.664+0000 7f8198f8e640 10 monclient(hunting): _renew_subs 2024-09-16T03:53:42.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.664+0000 7f8198f8e640 10 monclient(hunting): authenticate will time out at 1669.637806s 2024-09-16T03:53:42.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.664+0000 7f819259c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f81940a1330 0x7f819413c210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:53:42.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.664+0000 7f819259c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f81940a1330 0x7f819413c210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:44496/0 (socket says 172.21.15.42:44496) 2024-09-16T03:53:42.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.664+0000 7f819259c640 1 -- 172.21.15.42:0/2020517612 learned_addr learned my addr 172.21.15.42:0/2020517612 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:53:42.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.664+0000 7f819259c640 10 monclient(hunting): get_auth_request con 0x7f81940a1330 auth_method 0 2024-09-16T03:53:42.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.664+0000 7f819259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:53:42.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.664+0000 7f819259c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:53:42.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.664+0000 7f819259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:53:42.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.664+0000 7f8192d9d640 1 --2- 172.21.15.42:0/2020517612 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f81941cad50 0x7f81941cb1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:53:42.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.664+0000 7f8192d9d640 10 monclient(hunting): get_auth_request con 0x7f81941cad50 auth_method 0 2024-09-16T03:53:42.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.664+0000 7f8192d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:53:42.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.664+0000 7f8192d9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:53:42.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.664+0000 7f8192d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:53:42.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.664+0000 7f819259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:53:42.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.664+0000 7f819259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:53:42.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.664+0000 7f819259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:53:42.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.664+0000 7f8192d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:53:42.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.664+0000 7f8192d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:53:42.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.664+0000 7f8192d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:53:42.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.664+0000 7f8191d9b640 1 --2- 172.21.15.42:0/2020517612 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8194135100 0x7f819413c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:53:42.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.664+0000 7f8191d9b640 10 monclient(hunting): get_auth_request con 0x7f8194135100 auth_method 0 2024-09-16T03:53:42.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.664+0000 7f8191d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:53:42.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.664+0000 7f8191d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:53:42.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.664+0000 7f8191d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:53:42.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.664+0000 7f819259c640 10 monclient(hunting): handle_auth_done global_id 824334 payload 824 2024-09-16T03:53:42.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.664+0000 7f819259c640 1 -- 172.21.15.42:0/2020517612 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f81941cad50 msgr2=0x7f81941cb1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:53:42.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.664+0000 7f819259c640 1 --2- 172.21.15.42:0/2020517612 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f81941cad50 0x7f81941cb1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:53:42.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.664+0000 7f819259c640 1 -- 172.21.15.42:0/2020517612 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8194135100 msgr2=0x7f819413c750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:53:42.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.664+0000 7f819259c640 1 --2- 172.21.15.42:0/2020517612 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8194135100 0x7f819413c750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:53:42.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.664+0000 7f819259c640 10 monclient: _finish_hunting 0 2024-09-16T03:53:42.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.664+0000 7f819259c640 1 monclient: found mon.noname-a 2024-09-16T03:53:42.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.664+0000 7f819259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:53:42.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.664+0000 7f819259c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:53:42.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.664+0000 7f819259c640 1 -- 172.21.15.42:0/2020517612 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81940a3840 con 0x7f81940a1330 2024-09-16T03:53:42.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.664+0000 7f8192d9d640 1 --2- 172.21.15.42:0/2020517612 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f81941cad50 0x7f81941cb1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:53:42.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.664+0000 7f819259c640 1 --2- 172.21.15.42:0/2020517612 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f81940a1330 0x7f819413c210 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f818400c7c0 tx=0x7f818400cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:53:42.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.664+0000 7f8182ffd640 10 client.?.objecter ms_handle_connect 0x7f81940a1330 2024-09-16T03:53:42.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.664+0000 7f8182ffd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:53:42.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.664+0000 7f8182ffd640 1 -- 172.21.15.42:0/2020517612 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8184007a30 con 0x7f81940a1330 2024-09-16T03:53:42.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.664+0000 7f8182ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:53:42.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.664+0000 7f8182ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:53:42.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.664+0000 7f8182ffd640 10 monclient: dump: 2024-09-16T03:53:42.689 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:53:42.689 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:53:42.689 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:42.689 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:42.689 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:53:42.689 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:53:42.689 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:53:42.689 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:53:42.689 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:53:42.689 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:53:42.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.664+0000 7f8182ffd640 10 monclient: _finish_auth 0 2024-09-16T03:53:42.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.664+0000 7f8182ffd640 10 monclient: _check_auth_tickets 2024-09-16T03:53:42.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.664+0000 7f8182ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:53:42.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.664+0000 7f8198f8e640 5 monclient: authenticate success, global_id 824334 2024-09-16T03:53:42.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.664+0000 7f8182ffd640 1 -- 172.21.15.42:0/2020517612 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8184007bd0 con 0x7f81940a1330 2024-09-16T03:53:42.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.664+0000 7f8198f8e640 10 monclient: _renew_subs 2024-09-16T03:53:42.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.664+0000 7f8198f8e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:53:42.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.664+0000 7f8198f8e640 1 -- 172.21.15.42:0/2020517612 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8194073a30 con 0x7f81940a1330 2024-09-16T03:53:42.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.664+0000 7f8182ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:53:42.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.664+0000 7f8182ffd640 1 -- 172.21.15.42:0/2020517612 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81840113f0 con 0x7f81940a1330 2024-09-16T03:53:42.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.664+0000 7f8182ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:53:42.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.664+0000 7f8182ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:53:42.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.664+0000 7f8182ffd640 10 monclient: dump: 2024-09-16T03:53:42.690 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:53:42.690 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:53:42.690 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:42.691 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:42.691 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:53:42.691 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:53:42.691 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:53:42.691 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:53:42.691 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:53:42.691 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:53:42.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.664+0000 7f8183fff640 4 set_mon_vals no callback set 2024-09-16T03:53:42.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.664+0000 7f8198f8e640 10 client.824334.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:53:42.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.664+0000 7f8198f8e640 10 monclient: _renew_subs 2024-09-16T03:53:42.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.664+0000 7f8198f8e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:53:42.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.664+0000 7f8198f8e640 1 -- 172.21.15.42:0/2020517612 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81941389e0 con 0x7f81940a1330 2024-09-16T03:53:42.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.664+0000 7f8198f8e640 1 librados: init done 2024-09-16T03:53:42.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.664+0000 7f8198f8e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:53:42.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.664+0000 7f8198f8e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:53:42.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.664+0000 7f8198f8e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:53:42.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.664+0000 7f8198f8e640 1 -- 172.21.15.42:0/2020517612 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8194135550 con 0x7f81940a1330 2024-09-16T03:53:42.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.664+0000 7f8182ffd640 1 -- 172.21.15.42:0/2020517612 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f8184011590 con 0x7f81940a1330 2024-09-16T03:53:42.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.664+0000 7f8182ffd640 1 --2- 172.21.15.42:0/2020517612 >> v2:172.21.15.154:6812/254845342 conn(0x7f8174037810 0x7f8174039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:53:42.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.664+0000 7f8182ffd640 1 -- 172.21.15.42:0/2020517612 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(366..366 src has 286..366) ==== 5713+0+0 (secure 0 0 0) 0x7f8184049110 con 0x7f81940a1330 2024-09-16T03:53:42.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.664+0000 7f8182ffd640 10 client.824334.objecter ms_dispatch 0x7f8194000ba0 osd_map(366..366 src has 286..366) 2024-09-16T03:53:42.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.664+0000 7f8182ffd640 3 client.824334.objecter handle_osd_map got epochs [366,366] > 0 2024-09-16T03:53:42.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.664+0000 7f8182ffd640 3 client.824334.objecter handle_osd_map decoding full epoch 366 2024-09-16T03:53:42.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.668+0000 7f8182ffd640 20 client.824334.objecter dump_active .. 0 homeless 2024-09-16T03:53:42.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.668+0000 7f8191d9b640 1 --2- 172.21.15.42:0/2020517612 >> v2:172.21.15.154:6812/254845342 conn(0x7f8174037810 0x7f8174039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:53:42.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.668+0000 7f8191d9b640 10 monclient: get_auth_request con 0x7f8174037810 auth_method 0 2024-09-16T03:53:42.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.668+0000 7f8182ffd640 10 client.824334.objecter ms_handle_connect 0x7f8174037810 2024-09-16T03:53:42.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.668+0000 7f8191d9b640 1 --2- 172.21.15.42:0/2020517612 >> v2:172.21.15.154:6812/254845342 conn(0x7f8174037810 0x7f8174039cd0 secure :-1 s=READY pgs=3599 cs=0 l=1 rev1=1 crypto rx=0x7f81880ae000 tx=0x7f81880a7000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:53:42.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.672+0000 7f8182ffd640 1 -- 172.21.15.42:0/2020517612 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f8184015020 con 0x7f81940a1330 2024-09-16T03:53:42.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.672+0000 7f8182ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:53:42.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.672+0000 7f8182ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:53:42.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.900+0000 7f8198f8e640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:53:42.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.900+0000 7f8198f8e640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:53:42.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.900+0000 7f8198f8e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:53:42.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.900+0000 7f8198f8e640 1 -- 172.21.15.42:0/2020517612 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f8194137650 con 0x7f81940a1330 2024-09-16T03:53:42.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.900+0000 7f8182ffd640 1 -- 172.21.15.42:0/2020517612 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7f8184020700 con 0x7f81940a1330 2024-09-16T03:53:42.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.900+0000 7f8182ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:53:42.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.900+0000 7f8182ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:53:42.906 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":100,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:53:42.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.900+0000 7f8198f8e640 10 librados: watch_flush enter 2024-09-16T03:53:42.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.900+0000 7f8198f8e640 10 librados: watch_flush exit 2024-09-16T03:53:42.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.900+0000 7f8198f8e640 20 client.824334.objecter shutdown clearing up homeless session... 2024-09-16T03:53:42.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.900+0000 7f8198f8e640 10 client.824334.objecter successfully canceled tick 2024-09-16T03:53:42.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.900+0000 7f8198f8e640 1 -- 172.21.15.42:0/2020517612 >> v2:172.21.15.154:6812/254845342 conn(0x7f8174037810 msgr2=0x7f8174039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:53:42.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.900+0000 7f8198f8e640 1 --2- 172.21.15.42:0/2020517612 >> v2:172.21.15.154:6812/254845342 conn(0x7f8174037810 0x7f8174039cd0 secure :-1 s=READY pgs=3599 cs=0 l=1 rev1=1 crypto rx=0x7f81880ae000 tx=0x7f81880a7000 comp rx=0 tx=0).stop 2024-09-16T03:53:42.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.900+0000 7f8198f8e640 10 monclient: shutdown 2024-09-16T03:53:42.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.900+0000 7f8198f8e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:53:42.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.900+0000 7f8198f8e640 1 -- 172.21.15.42:0/2020517612 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f81940a1330 msgr2=0x7f819413c210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:53:42.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.900+0000 7f8198f8e640 1 --2- 172.21.15.42:0/2020517612 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f81940a1330 0x7f819413c210 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f818400c7c0 tx=0x7f818400cc80 comp rx=0 tx=0).stop 2024-09-16T03:53:42.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.900+0000 7f8198f8e640 1 -- 172.21.15.42:0/2020517612 shutdown_connections 2024-09-16T03:53:42.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.900+0000 7f8198f8e640 1 --2- 172.21.15.42:0/2020517612 >> v2:172.21.15.154:6812/254845342 conn(0x7f8174037810 0x7f8174039cd0 unknown :-1 s=CLOSED pgs=3599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:53:42.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.900+0000 7f8198f8e640 1 --2- 172.21.15.42:0/2020517612 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f81941cad50 0x7f81941cb1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:53:42.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.900+0000 7f8198f8e640 1 --2- 172.21.15.42:0/2020517612 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8194135100 0x7f819413c750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:53:42.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.900+0000 7f8198f8e640 1 --2- 172.21.15.42:0/2020517612 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f81940a1330 0x7f819413c210 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:53:42.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.900+0000 7f8198f8e640 1 -- 172.21.15.42:0/2020517612 >> 172.21.15.42:0/2020517612 conn(0x7f81940a2420 msgr2=0x7f819409d3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:53:42.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.900+0000 7f8198f8e640 1 -- 172.21.15.42:0/2020517612 shutdown_connections 2024-09-16T03:53:42.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.900+0000 7f8198f8e640 1 -- 172.21.15.42:0/2020517612 wait complete. 2024-09-16T03:53:42.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:42.900+0000 7f8198f8e640 1 librados: shutdown 2024-09-16T03:53:42.925 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T03:53:42.925 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T03:53:42.925 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T03:53:43.045 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.040+0000 7f7cfe000640 10 monclient: get_monmap_and_config 2024-09-16T03:53:43.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.040+0000 7f7cfe000640 10 monclient: build_initial_monmap 2024-09-16T03:53:43.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.040+0000 7f7cfe000640 10 monclient: monmap: 2024-09-16T03:53:43.046 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:53:43.046 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:53:43.046 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:53:43.044385+0000 2024-09-16T03:53:43.047 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:53:43.044385+0000 2024-09-16T03:53:43.047 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:53:43.047 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:53:43.047 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:53:43.047 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:53:43.047 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:53:43.048 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:53:43.048 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:53:43.048 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:53:43.072 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.040+0000 7f7cfe000640 1 Processor -- start 2024-09-16T03:53:43.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.040+0000 7f7cfe000640 1 -- start start 2024-09-16T03:53:43.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.040+0000 7f7cfe000640 10 monclient: init 2024-09-16T03:53:43.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.040+0000 7f7cfe000640 10 monclient: _reopen_session rank -1 2024-09-16T03:53:43.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.040+0000 7f7cfe000640 10 monclient: _add_conns ranks=[3,0,4,1,2] 2024-09-16T03:53:43.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.040+0000 7f7cfe000640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f7cf80a1330 0x7f7cf809f180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:53:43.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.040+0000 7f7cfe000640 10 monclient(hunting): picked mon.noname-a con 0x7f7cf80a1330 addr v2:172.21.15.42:3300/0 2024-09-16T03:53:43.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.040+0000 7f7cfe000640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7cf80a1710 addr v1:172.21.15.42:6789/0 2024-09-16T03:53:43.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.040+0000 7f7cfe000640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f7cf81393a0 0x7f7cf813b790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:53:43.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.040+0000 7f7cfe000640 10 monclient(hunting): picked mon.noname-c con 0x7f7cf81393a0 addr v2:172.21.15.154:3300/0 2024-09-16T03:53:43.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.040+0000 7f7cfe000640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7cf809e4c0 con 0x7f7cf80a1710 2024-09-16T03:53:43.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.040+0000 7f7cfe000640 10 monclient(hunting): start opening mon connection 2024-09-16T03:53:43.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.040+0000 7f7cfe000640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f7cf809edf0 con 0x7f7cf80a1330 2024-09-16T03:53:43.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.040+0000 7f7cfe000640 10 monclient(hunting): start opening mon connection 2024-09-16T03:53:43.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.040+0000 7f7cfe000640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f7cf809cd60 con 0x7f7cf81393a0 2024-09-16T03:53:43.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.040+0000 7f7cfe000640 10 monclient(hunting): _renew_subs 2024-09-16T03:53:43.077 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.040+0000 7f7cfe000640 10 monclient(hunting): authenticate will time out at 1670.013884s 2024-09-16T03:53:43.077 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.040+0000 7f7cf77fe640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f7cf80a1330 0x7f7cf809f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:53:43.077 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.040+0000 7f7cf77fe640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f7cf80a1330 0x7f7cf809f180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:44508/0 (socket says 172.21.15.42:44508) 2024-09-16T03:53:43.077 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.040+0000 7f7cf77fe640 1 -- 172.21.15.42:0/1193120232 learned_addr learned my addr 172.21.15.42:0/1193120232 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:53:43.077 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.040+0000 7f7cf77fe640 10 monclient(hunting): get_auth_request con 0x7f7cf80a1330 auth_method 0 2024-09-16T03:53:43.077 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.040+0000 7f7cf77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:53:43.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.040+0000 7f7cf77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:53:43.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.040+0000 7f7cf77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:53:43.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.040+0000 7f7cf77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:53:43.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.040+0000 7f7cf77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:53:43.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.040+0000 7f7cf77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:53:43.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.040+0000 7f7cf7fff640 1 --2- 172.21.15.42:0/1193120232 >> v2:172.21.15.154:3300/0 conn(0x7f7cf81393a0 0x7f7cf813b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:53:43.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.040+0000 7f7cf7fff640 10 monclient(hunting): get_auth_request con 0x7f7cf81393a0 auth_method 0 2024-09-16T03:53:43.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.040+0000 7f7cf7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:53:43.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.040+0000 7f7cf7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:53:43.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.040+0000 7f7cf7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:53:43.080 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.040+0000 7f7cf77fe640 10 monclient(hunting): handle_auth_done global_id 824337 payload 274 2024-09-16T03:53:43.080 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.040+0000 7f7cf77fe640 1 -- 172.21.15.42:0/1193120232 >> v2:172.21.15.154:3300/0 conn(0x7f7cf81393a0 msgr2=0x7f7cf813b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:53:43.080 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.040+0000 7f7cf77fe640 1 --2- 172.21.15.42:0/1193120232 >> v2:172.21.15.154:3300/0 conn(0x7f7cf81393a0 0x7f7cf813b790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:53:43.080 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.040+0000 7f7cf77fe640 1 -- 172.21.15.42:0/1193120232 >> v1:172.21.15.42:6789/0 conn(0x7f7cf80a1710 legacy=0x7f7cf80a1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:53:43.080 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.040+0000 7f7cf77fe640 10 monclient: _finish_hunting 0 2024-09-16T03:53:43.081 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.040+0000 7f7cf77fe640 1 monclient: found mon.noname-a 2024-09-16T03:53:43.081 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.040+0000 7f7cf67fc640 1 -- 172.21.15.42:0/1193120232 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1745623587 0 0) 0x7f7cf809e4c0 con 0x7f7cf80a1710 2024-09-16T03:53:43.081 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.040+0000 7f7cf77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:53:43.081 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.040+0000 7f7cf77fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:53:43.081 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.040+0000 7f7cf77fe640 1 -- 172.21.15.42:0/1193120232 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7cf809f6c0 con 0x7f7cf80a1330 2024-09-16T03:53:43.081 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.040+0000 7f7cf67fc640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T03:53:43.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.044+0000 7f7cf77fe640 1 --2- 172.21.15.42:0/1193120232 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7cf80a1330 0x7f7cf809f180 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f7cf809df50 tx=0x7f7ce801c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c55e345ac208cc99 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:53:43.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.044+0000 7f7cf67fc640 1 -- 172.21.15.42:0/1193120232 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ce80b0070 con 0x7f7cf80a1330 2024-09-16T03:53:43.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.044+0000 7f7cf67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:53:43.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.044+0000 7f7cf67fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:53:43.083 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.044+0000 7f7cf67fc640 10 monclient: dump: 2024-09-16T03:53:43.083 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:53:43.083 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:53:43.084 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:43.084 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:43.084 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:53:43.084 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:53:43.084 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:53:43.084 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:53:43.085 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:53:43.085 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:53:43.085 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.044+0000 7f7cf67fc640 10 monclient: _finish_auth 0 2024-09-16T03:53:43.085 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.044+0000 7f7cf67fc640 10 monclient: _check_auth_tickets 2024-09-16T03:53:43.085 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.044+0000 7f7cf67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:53:43.085 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.044+0000 7f7cf67fc640 1 -- 172.21.15.42:0/1193120232 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ce80ab600 con 0x7f7cf80a1330 2024-09-16T03:53:43.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.044+0000 7f7cf67fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:53:43.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.044+0000 7f7cf67fc640 1 -- 172.21.15.42:0/1193120232 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ce80ab9d0 con 0x7f7cf80a1330 2024-09-16T03:53:43.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.044+0000 7f7cfe000640 5 monclient: authenticate success, global_id 824337 2024-09-16T03:53:43.087 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.044+0000 7f7cfe000640 10 monclient: get_monmap_and_config success 2024-09-16T03:53:43.087 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.044+0000 7f7cfe000640 4 set_mon_vals no callback set 2024-09-16T03:53:43.087 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.044+0000 7f7cf67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:53:43.087 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.044+0000 7f7cf67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:53:43.087 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.044+0000 7f7cf67fc640 10 monclient: dump: 2024-09-16T03:53:43.087 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:53:43.088 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:53:43.088 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:43.088 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:43.088 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:53:43.088 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:53:43.089 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:53:43.089 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:53:43.089 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:53:43.089 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:53:43.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.044+0000 7f7cfe000640 10 monclient: shutdown 2024-09-16T03:53:43.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.044+0000 7f7cfe000640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:53:43.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.044+0000 7f7cfe000640 1 -- 172.21.15.42:0/1193120232 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7cf80a1330 msgr2=0x7f7cf809f180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:53:43.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.044+0000 7f7cfe000640 1 --2- 172.21.15.42:0/1193120232 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7cf80a1330 0x7f7cf809f180 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f7cf809df50 tx=0x7f7ce801c3e0 comp rx=0 tx=0).stop 2024-09-16T03:53:43.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.044+0000 7f7cfe000640 1 -- 172.21.15.42:0/1193120232 shutdown_connections 2024-09-16T03:53:43.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.044+0000 7f7cfe000640 1 --2- 172.21.15.42:0/1193120232 >> v2:172.21.15.154:3300/0 conn(0x7f7cf81393a0 0x7f7cf813b790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:53:43.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.044+0000 7f7cfe000640 1 --2- 172.21.15.42:0/1193120232 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7cf80a1330 0x7f7cf809f180 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:53:43.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.044+0000 7f7cfe000640 1 -- 172.21.15.42:0/1193120232 >> 172.21.15.42:0/1193120232 conn(0x7f7cf80a2420 msgr2=0x7f7cf80a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:53:43.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.044+0000 7f7cfe000640 1 -- 172.21.15.42:0/1193120232 shutdown_connections 2024-09-16T03:53:43.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.044+0000 7f7cfe000640 1 -- 172.21.15.42:0/1193120232 wait complete. 2024-09-16T03:53:43.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.044+0000 7f7cfe000640 10 monclient: build_initial_monmap 2024-09-16T03:53:43.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.044+0000 7f7cfe000640 10 monclient: monmap: 2024-09-16T03:53:43.092 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:53:43.092 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:53:43.092 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:53:43.093 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:53:43.093 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:53:43.093 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:53:43.093 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:53:43.093 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:53:43.094 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:53:43.094 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:53:43.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.044+0000 7f7cfe000640 1 librados: starting msgr at 2024-09-16T03:53:43.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.044+0000 7f7cfe000640 1 librados: starting objecter 2024-09-16T03:53:43.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.044+0000 7f7cfe000640 1 Processor -- start 2024-09-16T03:53:43.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.044+0000 7f7cfe000640 1 -- start start 2024-09-16T03:53:43.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.044+0000 7f7cfe000640 1 librados: setting wanted keys 2024-09-16T03:53:43.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.044+0000 7f7cfe000640 1 librados: calling monclient init 2024-09-16T03:53:43.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.044+0000 7f7cfe000640 10 monclient: init 2024-09-16T03:53:43.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.044+0000 7f7cfe000640 10 monclient: _reopen_session rank -1 2024-09-16T03:53:43.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.044+0000 7f7cfe000640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:53:43.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.044+0000 7f7cfe000640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7cf80a1330 0x7f7cf811bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:53:43.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.044+0000 7f7cfe000640 10 monclient(hunting): picked mon.noname-c con 0x7f7cf80a1330 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:53:43.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.044+0000 7f7cfe000640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7cf80a1710 0x7f7cf811c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:53:43.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.044+0000 7f7cfe000640 10 monclient(hunting): picked mon.noname-b con 0x7f7cf80a1710 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:53:43.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.044+0000 7f7cfe000640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7cf81393a0 0x7f7cf811c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:53:43.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.044+0000 7f7cfe000640 10 monclient(hunting): picked mon.noname-a con 0x7f7cf81393a0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:53:43.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.044+0000 7f7cfe000640 10 monclient(hunting): start opening mon connection 2024-09-16T03:53:43.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.044+0000 7f7cfe000640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f7cf809bbd0 con 0x7f7cf81393a0 2024-09-16T03:53:43.098 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.044+0000 7f7cfe000640 10 monclient(hunting): start opening mon connection 2024-09-16T03:53:43.098 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.044+0000 7f7cfe000640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f7cf809ba50 con 0x7f7cf80a1710 2024-09-16T03:53:43.098 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.044+0000 7f7cfe000640 10 monclient(hunting): start opening mon connection 2024-09-16T03:53:43.098 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.044+0000 7f7cfe000640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f7cf809bd50 con 0x7f7cf80a1330 2024-09-16T03:53:43.098 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.044+0000 7f7cfe000640 10 monclient(hunting): _renew_subs 2024-09-16T03:53:43.098 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.044+0000 7f7cfe000640 10 monclient(hunting): authenticate will time out at 1670.018848s 2024-09-16T03:53:43.099 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.044+0000 7f7cf77fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7cf80a1330 0x7f7cf811bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:53:43.099 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.044+0000 7f7cf7fff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7cf81393a0 0x7f7cf811c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:53:43.099 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.044+0000 7f7cf7fff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7cf81393a0 0x7f7cf811c9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:44520/0 (socket says 172.21.15.42:44520) 2024-09-16T03:53:43.099 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.044+0000 7f7cf77fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7cf80a1330 0x7f7cf811bf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:39832/0 (socket says 172.21.15.42:39832) 2024-09-16T03:53:43.099 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.044+0000 7f7cf7fff640 1 -- 172.21.15.42:0/4105339893 learned_addr learned my addr 172.21.15.42:0/4105339893 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:53:43.100 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.044+0000 7f7cf7fff640 10 monclient(hunting): get_auth_request con 0x7f7cf81393a0 auth_method 0 2024-09-16T03:53:43.100 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.044+0000 7f7cf7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:53:43.100 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.044+0000 7f7cf7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:53:43.101 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.044+0000 7f7cf7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:53:43.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.044+0000 7f7cf77fe640 10 monclient(hunting): get_auth_request con 0x7f7cf80a1330 auth_method 0 2024-09-16T03:53:43.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.044+0000 7f7cf77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:53:43.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.044+0000 7f7cf77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:53:43.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.044+0000 7f7cf77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:53:43.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.044+0000 7f7cf77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:53:43.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.044+0000 7f7cf77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:53:43.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.044+0000 7f7cf77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:53:43.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.044+0000 7f7cf7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:53:43.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.044+0000 7f7cf7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:53:43.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.044+0000 7f7cf7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:53:43.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.044+0000 7f7cf6ffd640 1 --2- 172.21.15.42:0/4105339893 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7cf80a1710 0x7f7cf811c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:53:43.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.044+0000 7f7cf77fe640 10 monclient(hunting): handle_auth_done global_id 834203 payload 824 2024-09-16T03:53:43.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.044+0000 7f7cf77fe640 1 -- 172.21.15.42:0/4105339893 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7cf80a1710 msgr2=0x7f7cf811c4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:53:43.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.044+0000 7f7cf77fe640 1 --2- 172.21.15.42:0/4105339893 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7cf80a1710 0x7f7cf811c4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:53:43.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.044+0000 7f7cf77fe640 1 -- 172.21.15.42:0/4105339893 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7cf81393a0 msgr2=0x7f7cf811c9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:53:43.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.044+0000 7f7cf77fe640 1 --2- 172.21.15.42:0/4105339893 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7cf81393a0 0x7f7cf811c9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:53:43.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.044+0000 7f7cf77fe640 10 monclient: _finish_hunting 0 2024-09-16T03:53:43.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.044+0000 7f7cf77fe640 1 monclient: found mon.noname-c 2024-09-16T03:53:43.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.044+0000 7f7cf77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:53:43.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.044+0000 7f7cf77fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:53:43.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.044+0000 7f7cf77fe640 1 -- 172.21.15.42:0/4105339893 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7cf8071fa0 con 0x7f7cf80a1330 2024-09-16T03:53:43.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.044+0000 7f7cf6ffd640 10 monclient: get_auth_request con 0x7f7cf80a1710 auth_method 0 2024-09-16T03:53:43.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.044+0000 7f7cf6ffd640 1 --2- 172.21.15.42:0/4105339893 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7cf80a1710 0x7f7cf811c4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T03:53:43.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.044+0000 7f7cf7fff640 1 --2- 172.21.15.42:0/4105339893 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7cf81393a0 0x7f7cf811c9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:53:43.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.044+0000 7f7cf77fe640 1 --2- 172.21.15.42:0/4105339893 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7cf80a1330 0x7f7cf811bf70 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f7ce80ab000 tx=0x7f7ce8019180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:53:43.105 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.044+0000 7f7cdffff640 10 client.?.objecter ms_handle_connect 0x7f7cf80a1330 2024-09-16T03:53:43.105 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.044+0000 7f7cdffff640 10 client.?.objecter resend_mon_ops 2024-09-16T03:53:43.105 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.044+0000 7f7cdffff640 1 -- 172.21.15.42:0/4105339893 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ce80b0040 con 0x7f7cf80a1330 2024-09-16T03:53:43.105 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.044+0000 7f7cdffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:53:43.105 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.044+0000 7f7cdffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:53:43.105 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.044+0000 7f7cdffff640 10 monclient: dump: 2024-09-16T03:53:43.105 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:53:43.106 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:53:43.106 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:43.106 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:43.106 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:53:43.106 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:53:43.106 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:53:43.106 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:53:43.106 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:53:43.106 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:53:43.106 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.044+0000 7f7cdffff640 10 monclient: _finish_auth 0 2024-09-16T03:53:43.106 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.044+0000 7f7cdffff640 10 monclient: _check_auth_tickets 2024-09-16T03:53:43.106 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.044+0000 7f7cdffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:53:43.106 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.044+0000 7f7cdffff640 1 -- 172.21.15.42:0/4105339893 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ce80ab400 con 0x7f7cf80a1330 2024-09-16T03:53:43.106 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.044+0000 7f7cdffff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:53:43.107 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.044+0000 7f7cdffff640 1 -- 172.21.15.42:0/4105339893 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ce80b4660 con 0x7f7cf80a1330 2024-09-16T03:53:43.107 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.044+0000 7f7cfe000640 5 monclient: authenticate success, global_id 834203 2024-09-16T03:53:43.107 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.044+0000 7f7cfe000640 10 monclient: _renew_subs 2024-09-16T03:53:43.107 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.044+0000 7f7cfe000640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:53:43.108 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.044+0000 7f7cfe000640 1 -- 172.21.15.42:0/4105339893 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7cf8073a30 con 0x7f7cf80a1330 2024-09-16T03:53:43.108 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.044+0000 7f7cdffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:53:43.108 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.044+0000 7f7cdffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:53:43.108 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.044+0000 7f7cdffff640 10 monclient: dump: 2024-09-16T03:53:43.108 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:53:43.108 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:53:43.108 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:43.108 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:43.108 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:53:43.108 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:53:43.108 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:53:43.109 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:53:43.109 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:53:43.109 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:53:43.109 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.044+0000 7f7cf5ffb640 4 set_mon_vals no callback set 2024-09-16T03:53:43.109 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.044+0000 7f7cfe000640 10 client.834203.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:53:43.109 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.044+0000 7f7cfe000640 10 monclient: _renew_subs 2024-09-16T03:53:43.109 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.044+0000 7f7cfe000640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:53:43.109 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.044+0000 7f7cfe000640 1 -- 172.21.15.42:0/4105339893 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f7cf811f460 con 0x7f7cf80a1330 2024-09-16T03:53:43.109 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.044+0000 7f7cfe000640 1 librados: init done 2024-09-16T03:53:43.109 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.048+0000 7f7cfe000640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:53:43.109 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.048+0000 7f7cfe000640 1 --2- 172.21.15.42:0/4105339893 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7cf802d7d0 0x7f7cf8062ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:53:43.110 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.048+0000 7f7cfe000640 10 monclient(hunting): start opening mon connection 2024-09-16T03:53:43.110 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.048+0000 7f7cfe000640 1 -- 172.21.15.42:0/4105339893 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f7cf811de80 con 0x7f7cf802d7d0 2024-09-16T03:53:43.110 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.048+0000 7f7cdffff640 1 -- 172.21.15.42:0/4105339893 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f7ce80bb020 con 0x7f7cf80a1330 2024-09-16T03:53:43.110 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.048+0000 7f7cf6ffd640 1 --2- 172.21.15.42:0/4105339893 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7cf802d7d0 0x7f7cf8062ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:53:43.110 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.048+0000 7f7cf6ffd640 10 monclient: get_auth_request con 0x7f7cf802d7d0 auth_method 0 2024-09-16T03:53:43.110 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.048+0000 7f7cf6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:53:43.110 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.048+0000 7f7cf6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:53:43.110 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.048+0000 7f7cf6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:53:43.110 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.048+0000 7f7cf6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:53:43.110 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.048+0000 7f7cf6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:53:43.110 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.048+0000 7f7cf6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:53:43.110 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.048+0000 7f7cdffff640 1 --2- 172.21.15.42:0/4105339893 >> v2:172.21.15.154:6812/254845342 conn(0x7f7ce4037810 0x7f7ce4039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:53:43.110 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.048+0000 7f7cdffff640 1 -- 172.21.15.42:0/4105339893 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(366..366 src has 286..366) ==== 5713+0+0 (secure 0 0 0) 0x7f7ce80e22d0 con 0x7f7cf80a1330 2024-09-16T03:53:43.111 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.048+0000 7f7cdffff640 10 client.834203.objecter ms_dispatch 0x7f7cf8000ba0 osd_map(366..366 src has 286..366) 2024-09-16T03:53:43.111 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.048+0000 7f7cdffff640 3 client.834203.objecter handle_osd_map got epochs [366,366] > 0 2024-09-16T03:53:43.111 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.048+0000 7f7cdffff640 3 client.834203.objecter handle_osd_map decoding full epoch 366 2024-09-16T03:53:43.111 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.048+0000 7f7cf6ffd640 10 monclient(hunting): handle_auth_done global_id 824346 payload 824 2024-09-16T03:53:43.111 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.048+0000 7f7cf6ffd640 1 -- 172.21.15.42:0/4105339893 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7cf802dbb0 con 0x7f7cf802d7d0 2024-09-16T03:53:43.111 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.048+0000 7f7cdffff640 20 client.834203.objecter dump_active .. 0 homeless 2024-09-16T03:53:43.111 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.048+0000 7f7cf6ffd640 1 --2- 172.21.15.42:0/4105339893 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7cf802d7d0 0x7f7cf8062ea0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f7cf809db40 tx=0x7f7cec00db80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:53:43.111 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.048+0000 7f7cdffff640 10 client.834203.objecter ms_handle_connect 0x7f7cf802d7d0 2024-09-16T03:53:43.111 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.048+0000 7f7cdffff640 10 client.834203.objecter resend_mon_ops 2024-09-16T03:53:43.111 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.048+0000 7f7cf7fff640 1 --2- 172.21.15.42:0/4105339893 >> v2:172.21.15.154:6812/254845342 conn(0x7f7ce4037810 0x7f7ce4039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:53:43.111 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.048+0000 7f7cf7fff640 10 monclient: get_auth_request con 0x7f7ce4037810 auth_method 0 2024-09-16T03:53:43.111 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.048+0000 7f7cdffff640 1 -- 172.21.15.42:0/4105339893 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cec005270 con 0x7f7cf802d7d0 2024-09-16T03:53:43.111 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.048+0000 7f7cdffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:53:43.111 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.048+0000 7f7cdffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:53:43.112 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.048+0000 7f7cdffff640 10 monclient: dump: 2024-09-16T03:53:43.112 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:53:43.112 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:53:43.112 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:43.112 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:43.112 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:53:43.112 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:53:43.112 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:53:43.112 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:53:43.112 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:53:43.112 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:53:43.112 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.048+0000 7f7cdffff640 1 -- 172.21.15.42:0/4105339893 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f7cec017070 con 0x7f7cf802d7d0 2024-09-16T03:53:43.112 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.048+0000 7f7cdffff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:53:43.113 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.048+0000 7f7cdffff640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:53:43.113 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.048+0000 7f7cdffff640 1 -- 172.21.15.42:0/4105339893 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7cf802d7d0 msgr2=0x7f7cf8062ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:53:43.113 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.048+0000 7f7cdffff640 1 --2- 172.21.15.42:0/4105339893 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7cf802d7d0 0x7f7cf8062ea0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f7cf809db40 tx=0x7f7cec00db80 comp rx=0 tx=0).stop 2024-09-16T03:53:43.113 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.048+0000 7f7cdffff640 1 -- 172.21.15.42:0/4105339893 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7cf802d7d0 msgr2=0x7f7cf8062ea0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:53:43.113 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.048+0000 7f7cdffff640 1 --2- 172.21.15.42:0/4105339893 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7cf802d7d0 0x7f7cf8062ea0 secure :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f7cf809db40 tx=0x7f7cec00db80 comp rx=0 tx=0).stop 2024-09-16T03:53:43.113 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.048+0000 7f7cdffff640 10 client.834203.objecter ms_handle_connect 0x7f7ce4037810 2024-09-16T03:53:43.113 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.048+0000 7f7cf7fff640 1 --2- 172.21.15.42:0/4105339893 >> v2:172.21.15.154:6812/254845342 conn(0x7f7ce4037810 0x7f7ce4039cd0 secure :-1 s=READY pgs=3600 cs=0 l=1 rev1=1 crypto rx=0x7f7cf0007880 tx=0x7f7cf00098d0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:53:43.113 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.056+0000 7f7cfe000640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:53:43.113 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.056+0000 7f7cfe000640 1 --2- 172.21.15.42:0/4105339893 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7cf80633e0 0x7f7cf81d9520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:53:43.113 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.056+0000 7f7cfe000640 10 monclient(hunting): start opening mon connection 2024-09-16T03:53:43.113 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.056+0000 7f7cfe000640 1 -- 172.21.15.42:0/4105339893 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f7cf811dcd0 con 0x7f7cf80633e0 2024-09-16T03:53:43.113 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.056+0000 7f7cf6ffd640 1 --2- 172.21.15.42:0/4105339893 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7cf80633e0 0x7f7cf81d9520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:53:43.113 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.056+0000 7f7cf6ffd640 10 monclient: get_auth_request con 0x7f7cf80633e0 auth_method 0 2024-09-16T03:53:43.114 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.056+0000 7f7cf6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:53:43.114 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.056+0000 7f7cf6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:53:43.114 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.056+0000 7f7cf6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:53:43.114 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.056+0000 7f7cf6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:53:43.114 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.056+0000 7f7cf6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:53:43.114 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.056+0000 7f7cf6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:53:43.114 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.056+0000 7f7cf6ffd640 10 monclient(hunting): handle_auth_done global_id 824349 payload 824 2024-09-16T03:53:43.114 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.056+0000 7f7cf6ffd640 1 -- 172.21.15.42:0/4105339893 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f7cf809f180 con 0x7f7cf80633e0 2024-09-16T03:53:43.114 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.056+0000 7f7cf6ffd640 1 --2- 172.21.15.42:0/4105339893 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7cf80633e0 0x7f7cf81d9520 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f7cec016000 tx=0x7f7cec001090 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:53:43.114 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.056+0000 7f7cdffff640 10 client.834203.objecter ms_handle_connect 0x7f7cf80633e0 2024-09-16T03:53:43.114 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.056+0000 7f7cdffff640 10 client.834203.objecter resend_mon_ops 2024-09-16T03:53:43.114 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.056+0000 7f7cdffff640 1 -- 172.21.15.42:0/4105339893 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cec00b900 con 0x7f7cf80633e0 2024-09-16T03:53:43.114 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.056+0000 7f7cdffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:53:43.114 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.056+0000 7f7cdffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:53:43.114 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.056+0000 7f7cdffff640 10 monclient: dump: 2024-09-16T03:53:43.115 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:53:43.115 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:53:43.115 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:43.115 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:43.115 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:53:43.115 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:53:43.115 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:53:43.115 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:53:43.115 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:53:43.115 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:53:43.115 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.056+0000 7f7cdffff640 1 -- 172.21.15.42:0/4105339893 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4882 (secure 0 0 0) 0x7f7cec00b900 con 0x7f7cf80633e0 2024-09-16T03:53:43.115 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.056+0000 7f7cdffff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:53:43.115 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.056+0000 7f7cdffff640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:53:43.115 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.056+0000 7f7cdffff640 1 -- 172.21.15.42:0/4105339893 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7cf80633e0 msgr2=0x7f7cf81d9520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:53:43.116 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.056+0000 7f7cdffff640 1 --2- 172.21.15.42:0/4105339893 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7cf80633e0 0x7f7cf81d9520 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f7cec016000 tx=0x7f7cec001090 comp rx=0 tx=0).stop 2024-09-16T03:53:43.116 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.056+0000 7f7cdffff640 1 -- 172.21.15.42:0/4105339893 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7cf80633e0 msgr2=0x7f7cf81d9520 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:53:43.116 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.056+0000 7f7cdffff640 1 --2- 172.21.15.42:0/4105339893 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7cf80633e0 0x7f7cf81d9520 secure :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f7cec016000 tx=0x7f7cec001090 comp rx=0 tx=0).stop 2024-09-16T03:53:43.116 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:53:43.116 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:53:43.116 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:53:43.116 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "leader", 2024-09-16T03:53:43.116 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 100, 2024-09-16T03:53:43.117 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 112910, 2024-09-16T03:53:43.117 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:53:43.117 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:53:43.117 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T03:53:43.117 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T03:53:43.117 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:53:43.117 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 3, 2024-09-16T03:53:43.117 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:53:43.117 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:53:43.117 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:53:43.117 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:53:43.117 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:53:43.117 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:53:43.117 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:53:43.118 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:53:43.118 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:53:43.118 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:53:43.118 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:53:43.118 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:53:43.118 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:53:43.118 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:53:43.118 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:53:43.118 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:53:43.118 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:53:43.118 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:53:43.118 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:53:43.118 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:53:43.118 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:53:43.119 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:53:43.119 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:53:43.119 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:53:43.119 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:53:43.119 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:53:43.119 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:53:43.119 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:53:43.119 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:53:43.119 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:53:43.119 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:53:43.119 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:53:43.119 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:53:43.119 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:53:43.120 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:53:43.120 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:53:43.120 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:53:43.120 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:53:43.120 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:53:43.120 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:53:43.120 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:53:43.120 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:53:43.120 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:53:43.120 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:53:43.120 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:53:43.120 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:53:43.120 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:53:43.120 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:53:43.121 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:53:43.121 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:53:43.121 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:53:43.121 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:53:43.121 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:53:43.121 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:53:43.121 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:53:43.121 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:53:43.121 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:53:43.121 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:53:43.121 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:53:43.121 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:53:43.121 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:53:43.121 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:53:43.121 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:53:43.122 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:53:43.122 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:53:43.122 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:53:43.122 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:53:43.122 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:53:43.122 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:53:43.122 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:53:43.122 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:53:43.122 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:53:43.122 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:53:43.122 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:53:43.122 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:53:43.122 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:53:43.122 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:53:43.123 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:53:43.123 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:53:43.123 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:53:43.123 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:53:43.123 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:53:43.123 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:53:43.123 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:53:43.123 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:53:43.123 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:53:43.123 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:53:43.123 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:53:43.123 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:53:43.123 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:53:43.123 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:53:43.123 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:53:43.124 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:53:43.124 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:53:43.124 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:53:43.124 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:53:43.124 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:53:43.124 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:53:43.124 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:53:43.124 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:53:43.124 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:53:43.124 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:53:43.124 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:53:43.124 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:53:43.124 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:53:43.125 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:53:43.125 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:53:43.125 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:53:43.125 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:53:43.125 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:53:43.125 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:53:43.125 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:53:43.125 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:53:43.125 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:53:43.125 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:53:43.125 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:53:43.125 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:53:43.125 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:53:43.125 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:53:43.126 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:53:43.126 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:53:43.126 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:53:43.126 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:53:43.126 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:53:43.126 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:53:43.126 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:53:43.126 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:53:43.126 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:53:43.126 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:53:43.126 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:53:43.126 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:53:43.126 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:53:43.126 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:53:43.126 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:53:43.127 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:53:43.127 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:53:43.127 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:53:43.127 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:53:43.127 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:53:43.127 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:53:43.127 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:53:43.127 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T03:53:43.127 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:53:43.127 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:53:43.127 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:53:43.127 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 5 2024-09-16T03:53:43.127 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:53:43.127 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:53:43.128 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:53:43.128 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:53:43.128 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:53:43.128 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:53:43.128 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 6 2024-09-16T03:53:43.128 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:53:43.128 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:53:43.128 INFO:teuthology.orchestra.run.smithi042.stdout: "mgr": [ 2024-09-16T03:53:43.128 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:53:43.128 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:53:43.128 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:53:43.128 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:53:43.128 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:53:43.128 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:53:43.129 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:53:43.129 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:53:43.129 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:53:43.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.056+0000 7f7cfe000640 10 librados: watch_flush enter 2024-09-16T03:53:43.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.056+0000 7f7cfe000640 10 librados: watch_flush exit 2024-09-16T03:53:43.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.056+0000 7f7cfe000640 20 client.834203.objecter shutdown clearing up homeless session... 2024-09-16T03:53:43.131 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.056+0000 7f7cfe000640 10 client.834203.objecter successfully canceled tick 2024-09-16T03:53:43.131 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.056+0000 7f7cfe000640 1 -- 172.21.15.42:0/4105339893 >> v2:172.21.15.154:6812/254845342 conn(0x7f7ce4037810 msgr2=0x7f7ce4039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:53:43.131 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.056+0000 7f7cfe000640 1 --2- 172.21.15.42:0/4105339893 >> v2:172.21.15.154:6812/254845342 conn(0x7f7ce4037810 0x7f7ce4039cd0 secure :-1 s=READY pgs=3600 cs=0 l=1 rev1=1 crypto rx=0x7f7cf0007880 tx=0x7f7cf00098d0 comp rx=0 tx=0).stop 2024-09-16T03:53:43.131 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.056+0000 7f7cfe000640 10 monclient: shutdown 2024-09-16T03:53:43.131 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.056+0000 7f7cfe000640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:53:43.131 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.056+0000 7f7cfe000640 1 -- 172.21.15.42:0/4105339893 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7cf80a1330 msgr2=0x7f7cf811bf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:53:43.131 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.056+0000 7f7cfe000640 1 --2- 172.21.15.42:0/4105339893 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7cf80a1330 0x7f7cf811bf70 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f7ce80ab000 tx=0x7f7ce8019180 comp rx=0 tx=0).stop 2024-09-16T03:53:43.131 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.056+0000 7f7cf7fff640 1 -- 172.21.15.42:0/4105339893 reap_dead start 2024-09-16T03:53:43.131 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.056+0000 7f7cf7fff640 1 -- 172.21.15.42:0/4105339893 reap_dead start 2024-09-16T03:53:43.131 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.056+0000 7f7cfe000640 1 -- 172.21.15.42:0/4105339893 shutdown_connections 2024-09-16T03:53:43.131 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.056+0000 7f7cfe000640 1 -- 172.21.15.42:0/4105339893 >> 172.21.15.42:0/4105339893 conn(0x7f7cf80a2420 msgr2=0x7f7cf81348e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:53:43.131 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.056+0000 7f7cfe000640 1 -- 172.21.15.42:0/4105339893 shutdown_connections 2024-09-16T03:53:43.131 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.056+0000 7f7cfe000640 1 -- 172.21.15.42:0/4105339893 wait complete. 2024-09-16T03:53:43.132 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.056+0000 7f7cfe000640 1 librados: shutdown 2024-09-16T03:53:43.132 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:53:43.132 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T03:53:43.132 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T03:53:43.258 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.252+0000 7f9a858b3640 10 monclient: get_monmap_and_config 2024-09-16T03:53:43.258 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.252+0000 7f9a858b3640 10 monclient: build_initial_monmap 2024-09-16T03:53:43.258 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.252+0000 7f9a858b3640 10 monclient: monmap: 2024-09-16T03:53:43.258 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:53:43.258 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:53:43.258 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:53:43.256657+0000 2024-09-16T03:53:43.258 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:53:43.256657+0000 2024-09-16T03:53:43.258 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:53:43.258 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:53:43.258 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:53:43.258 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:53:43.259 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:53:43.259 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:53:43.259 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:53:43.259 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:53:43.261 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.252+0000 7f9a858b3640 1 Processor -- start 2024-09-16T03:53:43.261 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.252+0000 7f9a858b3640 1 -- start start 2024-09-16T03:53:43.261 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.252+0000 7f9a858b3640 10 monclient: init 2024-09-16T03:53:43.261 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.252+0000 7f9a858b3640 10 monclient: _reopen_session rank -1 2024-09-16T03:53:43.261 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.252+0000 7f9a858b3640 10 monclient: _add_conns ranks=[0,3,4,2,1] 2024-09-16T03:53:43.261 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.252+0000 7f9a858b3640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9a800a1330 addr v1:172.21.15.42:6789/0 2024-09-16T03:53:43.261 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.252+0000 7f9a858b3640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f9a8006dac0 0x7f9a8013a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:53:43.261 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.252+0000 7f9a858b3640 10 monclient(hunting): picked mon.noname-a con 0x7f9a8006dac0 addr v2:172.21.15.42:3300/0 2024-09-16T03:53:43.261 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.252+0000 7f9a858b3640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f9a800a1e20 0x7f9a8013c890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:53:43.262 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.252+0000 7f9a858b3640 10 monclient(hunting): picked mon.noname-c con 0x7f9a800a1e20 addr v2:172.21.15.154:3300/0 2024-09-16T03:53:43.262 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.252+0000 7f9a858b3640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9a8009c090 con 0x7f9a800a1330 2024-09-16T03:53:43.262 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.252+0000 7f9a858b3640 10 monclient(hunting): start opening mon connection 2024-09-16T03:53:43.262 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.252+0000 7f9a858b3640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f9a8009f610 con 0x7f9a8006dac0 2024-09-16T03:53:43.262 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.252+0000 7f9a858b3640 10 monclient(hunting): start opening mon connection 2024-09-16T03:53:43.262 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.252+0000 7f9a858b3640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f9a8009f8f0 con 0x7f9a800a1e20 2024-09-16T03:53:43.262 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.252+0000 7f9a858b3640 10 monclient(hunting): _renew_subs 2024-09-16T03:53:43.262 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.252+0000 7f9a858b3640 10 monclient(hunting): authenticate will time out at 1670.226716s 2024-09-16T03:53:43.262 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.252+0000 7f9a7effd640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f9a800a1330 0x7f9a800a1710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:50180/0 (socket says 172.21.15.42:50180) 2024-09-16T03:53:43.262 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.252+0000 7f9a767fc640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f9a8006dac0 0x7f9a8013a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:53:43.262 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.252+0000 7f9a7effd640 1 -- 172.21.15.42:0/1060666782 learned_addr learned my addr 172.21.15.42:0/1060666782 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:53:43.262 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.252+0000 7f9a767fc640 10 monclient(hunting): get_auth_request con 0x7f9a8006dac0 auth_method 0 2024-09-16T03:53:43.262 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.252+0000 7f9a767fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:53:43.262 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.252+0000 7f9a767fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:53:43.263 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.252+0000 7f9a767fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:53:43.263 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.252+0000 7f9a767fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:53:43.263 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.252+0000 7f9a767fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:53:43.263 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.252+0000 7f9a767fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:53:43.263 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.252+0000 7f9a7f7fe640 1 --2- 172.21.15.42:0/1060666782 >> v2:172.21.15.154:3300/0 conn(0x7f9a800a1e20 0x7f9a8013c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:53:43.263 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.252+0000 7f9a7f7fe640 10 monclient(hunting): get_auth_request con 0x7f9a800a1e20 auth_method 0 2024-09-16T03:53:43.263 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.252+0000 7f9a7f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:53:43.263 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.252+0000 7f9a7f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:53:43.263 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.252+0000 7f9a7f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:53:43.263 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.252+0000 7f9a767fc640 10 monclient(hunting): handle_auth_done global_id 824352 payload 274 2024-09-16T03:53:43.263 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.252+0000 7f9a767fc640 1 -- 172.21.15.42:0/1060666782 >> v2:172.21.15.154:3300/0 conn(0x7f9a800a1e20 msgr2=0x7f9a8013c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:53:43.263 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.252+0000 7f9a767fc640 1 --2- 172.21.15.42:0/1060666782 >> v2:172.21.15.154:3300/0 conn(0x7f9a800a1e20 0x7f9a8013c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:53:43.263 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.252+0000 7f9a767fc640 1 -- 172.21.15.42:0/1060666782 >> v1:172.21.15.42:6789/0 conn(0x7f9a800a1330 legacy=0x7f9a800a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:53:43.263 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.252+0000 7f9a767fc640 10 monclient: _finish_hunting 0 2024-09-16T03:53:43.264 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.252+0000 7f9a767fc640 1 monclient: found mon.noname-a 2024-09-16T03:53:43.264 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.252+0000 7f9a767fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:53:43.264 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.252+0000 7f9a767fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:53:43.264 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.252+0000 7f9a767fc640 1 -- 172.21.15.42:0/1060666782 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a8009fbd0 con 0x7f9a8006dac0 2024-09-16T03:53:43.264 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.252+0000 7f9a7e7fc640 1 -- 172.21.15.42:0/1060666782 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 900512072 0 0) 0x7f9a8009c090 con 0x7f9a800a1330 2024-09-16T03:53:43.264 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.252+0000 7f9a7e7fc640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T03:53:43.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.256+0000 7f9a767fc640 1 --2- 172.21.15.42:0/1060666782 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9a8006dac0 0x7f9a8013a340 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f9a8009c800 tx=0x7f9a6801c3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8764402a6fd2a235 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:53:43.271 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.256+0000 7f9a7e7fc640 1 -- 172.21.15.42:0/1060666782 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a680b0070 con 0x7f9a8006dac0 2024-09-16T03:53:43.271 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.256+0000 7f9a7e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:53:43.271 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.256+0000 7f9a7e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:53:43.271 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.256+0000 7f9a7e7fc640 10 monclient: dump: 2024-09-16T03:53:43.271 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:53:43.271 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:53:43.271 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:43.271 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:43.271 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:53:43.271 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:53:43.271 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:53:43.271 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:53:43.271 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:53:43.272 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:53:43.272 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.256+0000 7f9a7e7fc640 10 monclient: _finish_auth 0 2024-09-16T03:53:43.272 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.256+0000 7f9a7e7fc640 10 monclient: _check_auth_tickets 2024-09-16T03:53:43.272 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.256+0000 7f9a7e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:53:43.272 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.256+0000 7f9a7e7fc640 1 -- 172.21.15.42:0/1060666782 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a680ab600 con 0x7f9a8006dac0 2024-09-16T03:53:43.272 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.256+0000 7f9a7e7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:53:43.272 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.256+0000 7f9a858b3640 5 monclient: authenticate success, global_id 824352 2024-09-16T03:53:43.272 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.256+0000 7f9a858b3640 10 monclient: get_monmap_and_config success 2024-09-16T03:53:43.272 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.256+0000 7f9a7e7fc640 1 -- 172.21.15.42:0/1060666782 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a680a4e60 con 0x7f9a8006dac0 2024-09-16T03:53:43.273 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.256+0000 7f9a858b3640 4 set_mon_vals no callback set 2024-09-16T03:53:43.273 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.256+0000 7f9a7e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:53:43.273 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.256+0000 7f9a7e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:53:43.273 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.256+0000 7f9a7e7fc640 10 monclient: dump: 2024-09-16T03:53:43.273 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:53:43.273 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:53:43.273 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:43.273 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:43.273 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:53:43.273 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:53:43.273 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:53:43.273 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:53:43.273 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:53:43.274 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:53:43.274 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.256+0000 7f9a858b3640 10 monclient: shutdown 2024-09-16T03:53:43.274 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.256+0000 7f9a858b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:53:43.274 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.256+0000 7f9a858b3640 1 -- 172.21.15.42:0/1060666782 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9a8006dac0 msgr2=0x7f9a8013a340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:53:43.274 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.256+0000 7f9a858b3640 1 --2- 172.21.15.42:0/1060666782 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9a8006dac0 0x7f9a8013a340 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f9a8009c800 tx=0x7f9a6801c3d0 comp rx=0 tx=0).stop 2024-09-16T03:53:43.274 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.256+0000 7f9a858b3640 1 -- 172.21.15.42:0/1060666782 shutdown_connections 2024-09-16T03:53:43.274 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.256+0000 7f9a858b3640 1 --2- 172.21.15.42:0/1060666782 >> v2:172.21.15.154:3300/0 conn(0x7f9a800a1e20 0x7f9a8013c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:53:43.274 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.256+0000 7f9a858b3640 1 --2- 172.21.15.42:0/1060666782 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9a8006dac0 0x7f9a8013a340 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:53:43.274 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.256+0000 7f9a858b3640 1 -- 172.21.15.42:0/1060666782 >> 172.21.15.42:0/1060666782 conn(0x7f9a800a2420 msgr2=0x7f9a800a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:53:43.274 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.256+0000 7f9a858b3640 1 -- 172.21.15.42:0/1060666782 shutdown_connections 2024-09-16T03:53:43.274 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.256+0000 7f9a858b3640 1 -- 172.21.15.42:0/1060666782 wait complete. 2024-09-16T03:53:43.274 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.256+0000 7f9a858b3640 10 monclient: build_initial_monmap 2024-09-16T03:53:43.274 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.256+0000 7f9a858b3640 10 monclient: monmap: 2024-09-16T03:53:43.274 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:53:43.275 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:53:43.275 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:53:43.275 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:53:43.275 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:53:43.275 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:53:43.275 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:53:43.275 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:53:43.275 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:53:43.275 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:53:43.276 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.256+0000 7f9a858b3640 1 librados: starting msgr at 2024-09-16T03:53:43.276 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.256+0000 7f9a858b3640 1 librados: starting objecter 2024-09-16T03:53:43.276 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.256+0000 7f9a858b3640 1 Processor -- start 2024-09-16T03:53:43.276 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.256+0000 7f9a858b3640 1 -- start start 2024-09-16T03:53:43.276 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.256+0000 7f9a858b3640 1 librados: setting wanted keys 2024-09-16T03:53:43.276 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.256+0000 7f9a858b3640 1 librados: calling monclient init 2024-09-16T03:53:43.276 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.256+0000 7f9a858b3640 10 monclient: init 2024-09-16T03:53:43.276 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.256+0000 7f9a858b3640 10 monclient: _reopen_session rank -1 2024-09-16T03:53:43.276 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.256+0000 7f9a858b3640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:53:43.276 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.256+0000 7f9a858b3640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9a800a1330 0x7f9a801ced40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:53:43.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.256+0000 7f9a858b3640 10 monclient(hunting): picked mon.noname-b con 0x7f9a800a1330 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:53:43.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.256+0000 7f9a858b3640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9a800a1e20 0x7f9a801353e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:53:43.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.256+0000 7f9a858b3640 10 monclient(hunting): picked mon.noname-a con 0x7f9a800a1e20 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:53:43.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.256+0000 7f9a858b3640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9a80135920 0x7f9a8009cb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:53:43.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.256+0000 7f9a858b3640 10 monclient(hunting): picked mon.noname-c con 0x7f9a80135920 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:53:43.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.256+0000 7f9a858b3640 10 monclient(hunting): start opening mon connection 2024-09-16T03:53:43.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.256+0000 7f9a858b3640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f9a8013f800 con 0x7f9a800a1e20 2024-09-16T03:53:43.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.256+0000 7f9a858b3640 10 monclient(hunting): start opening mon connection 2024-09-16T03:53:43.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.256+0000 7f9a858b3640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f9a8009ef80 con 0x7f9a800a1330 2024-09-16T03:53:43.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.256+0000 7f9a858b3640 10 monclient(hunting): start opening mon connection 2024-09-16T03:53:43.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.256+0000 7f9a858b3640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f9a8013f980 con 0x7f9a80135920 2024-09-16T03:53:43.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.256+0000 7f9a858b3640 10 monclient(hunting): _renew_subs 2024-09-16T03:53:43.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.256+0000 7f9a858b3640 10 monclient(hunting): authenticate will time out at 1670.231848s 2024-09-16T03:53:43.278 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.256+0000 7f9a767fc640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9a800a1e20 0x7f9a801353e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:53:43.278 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.260+0000 7f9a767fc640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9a800a1e20 0x7f9a801353e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:44560/0 (socket says 172.21.15.42:44560) 2024-09-16T03:53:43.278 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.260+0000 7f9a767fc640 1 -- 172.21.15.42:0/93347741 learned_addr learned my addr 172.21.15.42:0/93347741 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:53:43.278 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.260+0000 7f9a7f7fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9a80135920 0x7f9a8009cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:53:43.278 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.260+0000 7f9a767fc640 10 monclient(hunting): get_auth_request con 0x7f9a800a1e20 auth_method 0 2024-09-16T03:53:43.278 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.260+0000 7f9a767fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:53:43.278 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.260+0000 7f9a767fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:53:43.278 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.260+0000 7f9a767fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:53:43.278 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.260+0000 7f9a7f7fe640 10 monclient(hunting): get_auth_request con 0x7f9a80135920 auth_method 0 2024-09-16T03:53:43.278 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.260+0000 7f9a7f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:53:43.278 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.260+0000 7f9a7f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:53:43.278 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.260+0000 7f9a7f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:53:43.278 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.260+0000 7f9a767fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:53:43.278 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.260+0000 7f9a767fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:53:43.279 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.260+0000 7f9a767fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:53:43.279 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.260+0000 7f9a7effd640 1 --2- 172.21.15.42:0/93347741 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9a800a1330 0x7f9a801ced40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:53:43.279 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.260+0000 7f9a7f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:53:43.279 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.260+0000 7f9a7f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:53:43.279 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.260+0000 7f9a7f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:53:43.279 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.260+0000 7f9a7effd640 10 monclient(hunting): get_auth_request con 0x7f9a800a1330 auth_method 0 2024-09-16T03:53:43.279 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.260+0000 7f9a7effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:53:43.279 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.260+0000 7f9a7effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:53:43.279 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.260+0000 7f9a7effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:53:43.279 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.260+0000 7f9a767fc640 10 monclient(hunting): handle_auth_done global_id 824358 payload 824 2024-09-16T03:53:43.279 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.260+0000 7f9a767fc640 1 -- 172.21.15.42:0/93347741 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9a80135920 msgr2=0x7f9a8009cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:53:43.279 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.260+0000 7f9a767fc640 1 --2- 172.21.15.42:0/93347741 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9a80135920 0x7f9a8009cb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:53:43.279 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.260+0000 7f9a767fc640 1 -- 172.21.15.42:0/93347741 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9a800a1330 msgr2=0x7f9a801ced40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:53:43.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.260+0000 7f9a767fc640 1 --2- 172.21.15.42:0/93347741 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9a800a1330 0x7f9a801ced40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:53:43.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.260+0000 7f9a767fc640 10 monclient: _finish_hunting 0 2024-09-16T03:53:43.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.260+0000 7f9a767fc640 1 monclient: found mon.noname-a 2024-09-16T03:53:43.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.260+0000 7f9a767fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:53:43.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.260+0000 7f9a767fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:53:43.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.260+0000 7f9a767fc640 1 -- 172.21.15.42:0/93347741 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a8009eb40 con 0x7f9a800a1e20 2024-09-16T03:53:43.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.260+0000 7f9a767fc640 1 --2- 172.21.15.42:0/93347741 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9a800a1e20 0x7f9a801353e0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f9a68018fe0 tx=0x7f9a68019170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:53:43.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.260+0000 7f9a7cff9640 10 client.?.objecter ms_handle_connect 0x7f9a800a1e20 2024-09-16T03:53:43.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.260+0000 7f9a7cff9640 10 client.?.objecter resend_mon_ops 2024-09-16T03:53:43.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.260+0000 7f9a7cff9640 1 -- 172.21.15.42:0/93347741 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a680b8020 con 0x7f9a800a1e20 2024-09-16T03:53:43.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.260+0000 7f9a7cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:53:43.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.260+0000 7f9a7cff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:53:43.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.260+0000 7f9a7cff9640 10 monclient: dump: 2024-09-16T03:53:43.281 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:53:43.281 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:53:43.281 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:43.281 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:43.281 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:53:43.281 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:53:43.281 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:53:43.281 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:53:43.281 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:53:43.281 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:53:43.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.260+0000 7f9a7cff9640 10 monclient: _finish_auth 0 2024-09-16T03:53:43.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.260+0000 7f9a7cff9640 10 monclient: _check_auth_tickets 2024-09-16T03:53:43.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.260+0000 7f9a7cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:53:43.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.260+0000 7f9a7cff9640 1 -- 172.21.15.42:0/93347741 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a680a4e60 con 0x7f9a800a1e20 2024-09-16T03:53:43.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.260+0000 7f9a7cff9640 10 monclient: handle_config config(0 keys) 2024-09-16T03:53:43.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.260+0000 7f9a7cff9640 1 -- 172.21.15.42:0/93347741 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a680b0040 con 0x7f9a800a1e20 2024-09-16T03:53:43.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.260+0000 7f9a7cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:53:43.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.260+0000 7f9a7cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:53:43.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.260+0000 7f9a7cff9640 10 monclient: dump: 2024-09-16T03:53:43.282 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:53:43.282 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:53:43.282 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:43.282 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:43.283 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:53:43.283 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:53:43.283 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:53:43.283 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:53:43.283 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:53:43.283 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:53:43.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.260+0000 7f9a7dffb640 4 set_mon_vals no callback set 2024-09-16T03:53:43.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.260+0000 7f9a858b3640 5 monclient: authenticate success, global_id 824358 2024-09-16T03:53:43.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.260+0000 7f9a858b3640 10 monclient: _renew_subs 2024-09-16T03:53:43.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.260+0000 7f9a858b3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:53:43.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.260+0000 7f9a858b3640 1 -- 172.21.15.42:0/93347741 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a80073a30 con 0x7f9a800a1e20 2024-09-16T03:53:43.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.260+0000 7f9a858b3640 10 client.824358.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:53:43.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.260+0000 7f9a858b3640 10 monclient: _renew_subs 2024-09-16T03:53:43.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.260+0000 7f9a858b3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:53:43.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.260+0000 7f9a858b3640 1 -- 172.21.15.42:0/93347741 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a8009e460 con 0x7f9a800a1e20 2024-09-16T03:53:43.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.260+0000 7f9a858b3640 1 librados: init done 2024-09-16T03:53:43.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.260+0000 7f9a858b3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:53:43.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.260+0000 7f9a858b3640 1 --2- 172.21.15.42:0/93347741 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9a8006dac0 0x7f9a80121a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:53:43.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.260+0000 7f9a858b3640 10 monclient(hunting): start opening mon connection 2024-09-16T03:53:43.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.260+0000 7f9a858b3640 1 -- 172.21.15.42:0/93347741 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f9a80120780 con 0x7f9a8006dac0 2024-09-16T03:53:43.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.260+0000 7f9a7effd640 1 --2- 172.21.15.42:0/93347741 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9a8006dac0 0x7f9a80121a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:53:43.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.260+0000 7f9a7cff9640 1 -- 172.21.15.42:0/93347741 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f9a680bb020 con 0x7f9a800a1e20 2024-09-16T03:53:43.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.260+0000 7f9a7effd640 10 monclient: get_auth_request con 0x7f9a8006dac0 auth_method 0 2024-09-16T03:53:43.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.260+0000 7f9a7effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:53:43.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.260+0000 7f9a7effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:53:43.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.260+0000 7f9a7effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:53:43.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.260+0000 7f9a7effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:53:43.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.260+0000 7f9a7effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:53:43.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.260+0000 7f9a7effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:53:43.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.260+0000 7f9a7cff9640 1 --2- 172.21.15.42:0/93347741 >> v2:172.21.15.154:6812/254845342 conn(0x7f9a60037810 0x7f9a60039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:53:43.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.260+0000 7f9a7cff9640 1 -- 172.21.15.42:0/93347741 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(366..366 src has 286..366) ==== 5713+0+0 (secure 0 0 0) 0x7f9a680e29d0 con 0x7f9a800a1e20 2024-09-16T03:53:43.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.260+0000 7f9a7effd640 10 monclient(hunting): handle_auth_done global_id 834209 payload 824 2024-09-16T03:53:43.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.260+0000 7f9a7cff9640 10 client.824358.objecter ms_dispatch 0x7f9a80000ba0 osd_map(366..366 src has 286..366) 2024-09-16T03:53:43.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.260+0000 7f9a7cff9640 3 client.824358.objecter handle_osd_map got epochs [366,366] > 0 2024-09-16T03:53:43.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.260+0000 7f9a7cff9640 3 client.824358.objecter handle_osd_map decoding full epoch 366 2024-09-16T03:53:43.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.260+0000 7f9a7effd640 1 -- 172.21.15.42:0/93347741 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9a80135d30 con 0x7f9a8006dac0 2024-09-16T03:53:43.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.260+0000 7f9a7cff9640 20 client.824358.objecter dump_active .. 0 homeless 2024-09-16T03:53:43.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.260+0000 7f9a7cff9640 10 client.824358.objecter ms_handle_connect 0x7f9a8006dac0 2024-09-16T03:53:43.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.260+0000 7f9a7cff9640 10 client.824358.objecter resend_mon_ops 2024-09-16T03:53:43.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.260+0000 7f9a7effd640 1 --2- 172.21.15.42:0/93347741 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9a8006dac0 0x7f9a80121a10 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f9a700011f0 tx=0x7f9a70001980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:53:43.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.260+0000 7f9a7f7fe640 1 --2- 172.21.15.42:0/93347741 >> v2:172.21.15.154:6812/254845342 conn(0x7f9a60037810 0x7f9a60039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:53:43.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.260+0000 7f9a7f7fe640 10 monclient: get_auth_request con 0x7f9a60037810 auth_method 0 2024-09-16T03:53:43.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.260+0000 7f9a7cff9640 1 -- 172.21.15.42:0/93347741 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a70012590 con 0x7f9a8006dac0 2024-09-16T03:53:43.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.260+0000 7f9a7cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:53:43.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.260+0000 7f9a7cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:53:43.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.260+0000 7f9a7cff9640 10 monclient: dump: 2024-09-16T03:53:43.286 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:53:43.286 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:53:43.287 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:43.287 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:43.287 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:53:43.287 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:53:43.287 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:53:43.287 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:53:43.287 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:53:43.287 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:53:43.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.260+0000 7f9a7cff9640 1 -- 172.21.15.42:0/93347741 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f9a70018070 con 0x7f9a8006dac0 2024-09-16T03:53:43.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.260+0000 7f9a7cff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:53:43.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.260+0000 7f9a7cff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:53:43.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.260+0000 7f9a7cff9640 1 -- 172.21.15.42:0/93347741 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9a8006dac0 msgr2=0x7f9a80121a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:53:43.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.260+0000 7f9a7cff9640 1 --2- 172.21.15.42:0/93347741 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9a8006dac0 0x7f9a80121a10 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f9a700011f0 tx=0x7f9a70001980 comp rx=0 tx=0).stop 2024-09-16T03:53:43.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.260+0000 7f9a7cff9640 1 -- 172.21.15.42:0/93347741 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9a8006dac0 msgr2=0x7f9a80121a10 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:53:43.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.260+0000 7f9a7cff9640 1 --2- 172.21.15.42:0/93347741 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9a8006dac0 0x7f9a80121a10 secure :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f9a700011f0 tx=0x7f9a70001980 comp rx=0 tx=0).stop 2024-09-16T03:53:43.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.260+0000 7f9a7cff9640 10 client.824358.objecter ms_handle_connect 0x7f9a60037810 2024-09-16T03:53:43.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.260+0000 7f9a7f7fe640 1 --2- 172.21.15.42:0/93347741 >> v2:172.21.15.154:6812/254845342 conn(0x7f9a60037810 0x7f9a60039cd0 secure :-1 s=READY pgs=3601 cs=0 l=1 rev1=1 crypto rx=0x7f9a78003ab0 tx=0x7f9a78009820 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:53:43.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.268+0000 7f9a858b3640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:53:43.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.268+0000 7f9a858b3640 1 --2- 172.21.15.42:0/93347741 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9a8002c650 0x7f9a8002ca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:53:43.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.268+0000 7f9a858b3640 10 monclient(hunting): start opening mon connection 2024-09-16T03:53:43.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.268+0000 7f9a858b3640 1 -- 172.21.15.42:0/93347741 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f9a80121550 con 0x7f9a8002c650 2024-09-16T03:53:43.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.268+0000 7f9a7effd640 1 --2- 172.21.15.42:0/93347741 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9a8002c650 0x7f9a8002ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:53:43.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.268+0000 7f9a7effd640 10 monclient: get_auth_request con 0x7f9a8002c650 auth_method 0 2024-09-16T03:53:43.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.268+0000 7f9a7effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:53:43.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.268+0000 7f9a7effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:53:43.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.268+0000 7f9a7effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:53:43.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.268+0000 7f9a7effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:53:43.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.268+0000 7f9a7effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:53:43.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.268+0000 7f9a7effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:53:43.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.268+0000 7f9a7effd640 10 monclient(hunting): handle_auth_done global_id 834212 payload 824 2024-09-16T03:53:43.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.268+0000 7f9a7effd640 1 -- 172.21.15.42:0/93347741 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f9a800a1710 con 0x7f9a8002c650 2024-09-16T03:53:43.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.268+0000 7f9a7cff9640 10 client.824358.objecter ms_handle_connect 0x7f9a8002c650 2024-09-16T03:53:43.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.268+0000 7f9a7cff9640 10 client.824358.objecter resend_mon_ops 2024-09-16T03:53:43.291 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.268+0000 7f9a7effd640 1 --2- 172.21.15.42:0/93347741 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9a8002c650 0x7f9a8002ca60 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f9a700011f0 tx=0x7f9a70001090 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:53:43.291 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.268+0000 7f9a7cff9640 1 -- 172.21.15.42:0/93347741 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a70012c10 con 0x7f9a8002c650 2024-09-16T03:53:43.291 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.268+0000 7f9a7cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:53:43.291 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.268+0000 7f9a7cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:53:43.291 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.268+0000 7f9a7cff9640 10 monclient: dump: 2024-09-16T03:53:43.291 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:53:43.291 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:53:43.291 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:43.291 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:43.291 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:53:43.291 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:53:43.291 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:53:43.291 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:53:43.291 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:53:43.292 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:53:43.292 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.268+0000 7f9a7cff9640 1 -- 172.21.15.42:0/93347741 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4712 (secure 0 0 0) 0x7f9a70012c10 con 0x7f9a8002c650 2024-09-16T03:53:43.292 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.268+0000 7f9a7cff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:53:43.292 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.268+0000 7f9a7cff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:53:43.292 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.268+0000 7f9a7cff9640 1 -- 172.21.15.42:0/93347741 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9a8002c650 msgr2=0x7f9a8002ca60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:53:43.292 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.268+0000 7f9a7cff9640 1 --2- 172.21.15.42:0/93347741 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9a8002c650 0x7f9a8002ca60 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f9a700011f0 tx=0x7f9a70001090 comp rx=0 tx=0).stop 2024-09-16T03:53:43.292 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.268+0000 7f9a7cff9640 1 -- 172.21.15.42:0/93347741 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9a8002c650 msgr2=0x7f9a8002ca60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:53:43.292 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.268+0000 7f9a7cff9640 1 --2- 172.21.15.42:0/93347741 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9a8002c650 0x7f9a8002ca60 secure :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f9a700011f0 tx=0x7f9a70001090 comp rx=0 tx=0).stop 2024-09-16T03:53:43.292 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:53:43.293 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:53:43.293 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:53:43.293 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T03:53:43.293 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 100, 2024-09-16T03:53:43.293 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 62839, 2024-09-16T03:53:43.293 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:53:43.293 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:53:43.293 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T03:53:43.293 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T03:53:43.293 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:53:43.293 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 3, 2024-09-16T03:53:43.293 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:53:43.293 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:53:43.294 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:53:43.294 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:53:43.294 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:53:43.294 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:53:43.294 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:53:43.294 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:53:43.294 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:53:43.294 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:53:43.294 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:53:43.294 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:53:43.294 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:53:43.294 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:53:43.294 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:53:43.294 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:53:43.295 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:53:43.295 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:53:43.295 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:53:43.295 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:53:43.295 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:53:43.295 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:53:43.295 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:53:43.295 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:53:43.295 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:53:43.295 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:53:43.295 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:53:43.295 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:53:43.295 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:53:43.295 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:53:43.296 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:53:43.296 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:53:43.296 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:53:43.296 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:53:43.296 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:53:43.296 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:53:43.296 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:53:43.296 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:53:43.296 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:53:43.296 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:53:43.296 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:53:43.296 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:53:43.296 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:53:43.297 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:53:43.297 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:53:43.297 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:53:43.297 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:53:43.297 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:53:43.297 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:53:43.297 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:53:43.297 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:53:43.297 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:53:43.297 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:53:43.297 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:53:43.297 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:53:43.297 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:53:43.298 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:53:43.298 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:53:43.298 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:53:43.298 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:53:43.298 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:53:43.298 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:53:43.298 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:53:43.298 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:53:43.298 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:53:43.298 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:53:43.298 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:53:43.298 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:53:43.298 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:53:43.298 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:53:43.299 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:53:43.299 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:53:43.299 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:53:43.299 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:53:43.299 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:53:43.299 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:53:43.299 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:53:43.299 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:53:43.299 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:53:43.299 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:53:43.299 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:53:43.299 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:53:43.299 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:53:43.299 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:53:43.299 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:53:43.300 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:53:43.300 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:53:43.300 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:53:43.300 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:53:43.300 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:53:43.300 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:53:43.300 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:53:43.300 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:53:43.300 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:53:43.300 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:53:43.300 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:53:43.300 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:53:43.301 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:53:43.301 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:53:43.301 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:53:43.301 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:53:43.301 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:53:43.301 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:53:43.301 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:53:43.301 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:53:43.301 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:53:43.301 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:53:43.301 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:53:43.301 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:53:43.301 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:53:43.301 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:53:43.302 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:53:43.302 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:53:43.302 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:53:43.302 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:53:43.302 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:53:43.302 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:53:43.302 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:53:43.302 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:53:43.302 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:53:43.302 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:53:43.302 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:53:43.302 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:53:43.302 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:53:43.302 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:53:43.303 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:53:43.303 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:53:43.303 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:53:43.303 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:53:43.303 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:53:43.303 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:53:43.303 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:53:43.303 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:53:43.303 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:53:43.303 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:53:43.303 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:53:43.303 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:53:43.303 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:53:43.303 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:53:43.304 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:53:43.304 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:53:43.304 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T03:53:43.304 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:53:43.304 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:53:43.304 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:53:43.304 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:53:43.304 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:53:43.304 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:53:43.304 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:53:43.304 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:53:43.304 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:53:43.304 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:53:43.304 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 2 2024-09-16T03:53:43.305 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:53:43.305 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:53:43.305 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:53:43.305 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:53:43.305 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:53:43.307 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.268+0000 7f9a858b3640 10 librados: watch_flush enter 2024-09-16T03:53:43.307 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.268+0000 7f9a858b3640 10 librados: watch_flush exit 2024-09-16T03:53:43.308 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.268+0000 7f9a858b3640 20 client.824358.objecter shutdown clearing up homeless session... 2024-09-16T03:53:43.308 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.268+0000 7f9a858b3640 10 client.824358.objecter successfully canceled tick 2024-09-16T03:53:43.308 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.268+0000 7f9a858b3640 1 -- 172.21.15.42:0/93347741 >> v2:172.21.15.154:6812/254845342 conn(0x7f9a60037810 msgr2=0x7f9a60039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:53:43.308 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.268+0000 7f9a858b3640 1 --2- 172.21.15.42:0/93347741 >> v2:172.21.15.154:6812/254845342 conn(0x7f9a60037810 0x7f9a60039cd0 secure :-1 s=READY pgs=3601 cs=0 l=1 rev1=1 crypto rx=0x7f9a78003ab0 tx=0x7f9a78009820 comp rx=0 tx=0).stop 2024-09-16T03:53:43.308 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.268+0000 7f9a858b3640 10 monclient: shutdown 2024-09-16T03:53:43.308 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.268+0000 7f9a858b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:53:43.308 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.268+0000 7f9a858b3640 1 -- 172.21.15.42:0/93347741 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9a800a1e20 msgr2=0x7f9a801353e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:53:43.308 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.268+0000 7f9a858b3640 1 --2- 172.21.15.42:0/93347741 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9a800a1e20 0x7f9a801353e0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f9a68018fe0 tx=0x7f9a68019170 comp rx=0 tx=0).stop 2024-09-16T03:53:43.308 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.268+0000 7f9a7f7fe640 1 -- 172.21.15.42:0/93347741 reap_dead start 2024-09-16T03:53:43.308 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.268+0000 7f9a7f7fe640 1 -- 172.21.15.42:0/93347741 reap_dead start 2024-09-16T03:53:43.308 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.268+0000 7f9a858b3640 1 -- 172.21.15.42:0/93347741 shutdown_connections 2024-09-16T03:53:43.308 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.268+0000 7f9a858b3640 1 -- 172.21.15.42:0/93347741 >> 172.21.15.42:0/93347741 conn(0x7f9a800a2420 msgr2=0x7f9a800a4dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:53:43.308 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.268+0000 7f9a858b3640 1 -- 172.21.15.42:0/93347741 shutdown_connections 2024-09-16T03:53:43.309 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.268+0000 7f9a858b3640 1 -- 172.21.15.42:0/93347741 wait complete. 2024-09-16T03:53:43.309 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.268+0000 7f9a858b3640 1 librados: shutdown 2024-09-16T03:53:43.309 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:53:43.309 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T03:53:43.309 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T03:53:43.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.420+0000 7fa0a45b7640 10 monclient: get_monmap_and_config 2024-09-16T03:53:43.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.420+0000 7fa0a45b7640 10 monclient: build_initial_monmap 2024-09-16T03:53:43.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.420+0000 7fa0a45b7640 10 monclient: monmap: 2024-09-16T03:53:43.426 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:53:43.426 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:53:43.427 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:53:43.425183+0000 2024-09-16T03:53:43.427 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:53:43.425183+0000 2024-09-16T03:53:43.427 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:53:43.427 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:53:43.427 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:53:43.427 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:53:43.427 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:53:43.427 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:53:43.427 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:53:43.427 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:53:43.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.420+0000 7fa0a45b7640 1 Processor -- start 2024-09-16T03:53:43.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.420+0000 7fa0a45b7640 1 -- start start 2024-09-16T03:53:43.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.420+0000 7fa0a45b7640 10 monclient: init 2024-09-16T03:53:43.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.420+0000 7fa0a45b7640 10 monclient: _reopen_session rank -1 2024-09-16T03:53:43.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.420+0000 7fa0a45b7640 10 monclient: _add_conns ranks=[2,4,1,3,0] 2024-09-16T03:53:43.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.420+0000 7fa0a45b7640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa09c0a1330 0x7fa09c0a1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:53:43.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.420+0000 7fa0a45b7640 10 monclient(hunting): picked mon.noname-b con 0x7fa09c0a1330 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:53:43.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.420+0000 7fa0a45b7640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fa09c06dac0 0x7fa09c0a1c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:53:43.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.420+0000 7fa0a45b7640 10 monclient(hunting): picked mon.noname-c con 0x7fa09c06dac0 addr v2:172.21.15.154:3300/0 2024-09-16T03:53:43.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.420+0000 7fa0a45b7640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa09c135100 addr v1:172.21.15.154:6789/0 2024-09-16T03:53:43.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.420+0000 7fa0a45b7640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa09c09c7d0 con 0x7fa09c135100 2024-09-16T03:53:43.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.420+0000 7fa0a45b7640 10 monclient(hunting): start opening mon connection 2024-09-16T03:53:43.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.420+0000 7fa0a45b7640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fa09c09c090 con 0x7fa09c06dac0 2024-09-16T03:53:43.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.420+0000 7fa0a45b7640 10 monclient(hunting): start opening mon connection 2024-09-16T03:53:43.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.420+0000 7fa0a45b7640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fa09c09f610 con 0x7fa09c0a1330 2024-09-16T03:53:43.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.420+0000 7fa0a45b7640 10 monclient(hunting): _renew_subs 2024-09-16T03:53:43.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.420+0000 7fa0a45b7640 10 monclient(hunting): authenticate will time out at 1670.394608s 2024-09-16T03:53:43.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.420+0000 7fa0a2353640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa09c0a1330 0x7fa09c0a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:53:43.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.420+0000 7fa0a2353640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa09c0a1330 0x7fa09c0a1710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:39864/0 (socket says 172.21.15.42:39864) 2024-09-16T03:53:43.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.420+0000 7fa0a2353640 1 -- 172.21.15.42:0/1138001439 learned_addr learned my addr 172.21.15.42:0/1138001439 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:53:43.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.420+0000 7fa0a2353640 10 monclient(hunting): get_auth_request con 0x7fa09c0a1330 auth_method 0 2024-09-16T03:53:43.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.420+0000 7fa0a2353640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:53:43.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.420+0000 7fa0a2353640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:53:43.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.420+0000 7fa0a2353640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:53:43.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.420+0000 7fa0a2353640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:53:43.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.420+0000 7fa0a2353640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:53:43.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.420+0000 7fa0a2353640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:53:43.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.420+0000 7fa0a2353640 10 monclient(hunting): handle_auth_done global_id 834215 payload 274 2024-09-16T03:53:43.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.420+0000 7fa0a2353640 1 -- 172.21.15.42:0/1138001439 >> v2:172.21.15.154:3300/0 conn(0x7fa09c06dac0 msgr2=0x7fa09c0a1c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:53:43.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.420+0000 7fa0a1b52640 1 --2- 172.21.15.42:0/1138001439 >> v2:172.21.15.154:3300/0 conn(0x7fa09c06dac0 0x7fa09c0a1c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:53:43.432 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.420+0000 7fa0a2353640 1 --2- 172.21.15.42:0/1138001439 >> v2:172.21.15.154:3300/0 conn(0x7fa09c06dac0 0x7fa09c0a1c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:53:43.432 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.420+0000 7fa0a2353640 1 -- 172.21.15.42:0/1138001439 >> v1:172.21.15.154:6789/0 conn(0x7fa09c135100 legacy=0x7fa09c135550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:53:43.432 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.420+0000 7fa0a2353640 10 monclient: _finish_hunting 0 2024-09-16T03:53:43.432 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.420+0000 7fa0a2353640 1 monclient: found mon.noname-b 2024-09-16T03:53:43.432 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.420+0000 7fa0a2353640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:53:43.432 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.420+0000 7fa0a2353640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T03:53:43.432 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.420+0000 7fa0a2353640 1 -- 172.21.15.42:0/1138001439 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa09c09f8f0 con 0x7fa09c0a1330 2024-09-16T03:53:43.439 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.424+0000 7fa0a2353640 1 --2- 172.21.15.42:0/1138001439 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa09c0a1330 0x7fa09c0a1710 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa090009870 tx=0x7fa09001c500 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ccac7233e84d0177 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:53:43.439 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.424+0000 7fa0a1351640 1 -- 172.21.15.42:0/1138001439 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0900b1070 con 0x7fa09c0a1330 2024-09-16T03:53:43.439 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.424+0000 7fa0a1351640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:53:43.439 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.424+0000 7fa0a1351640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:53:43.439 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.424+0000 7fa0a1351640 10 monclient: dump: 2024-09-16T03:53:43.440 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:53:43.440 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:53:43.440 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:43.440 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:43.440 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:53:43.440 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:53:43.440 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:53:43.440 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:53:43.440 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:53:43.440 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:53:43.440 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.424+0000 7fa0a1351640 10 monclient: _finish_auth 0 2024-09-16T03:53:43.440 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.424+0000 7fa0a1351640 10 monclient: _check_auth_tickets 2024-09-16T03:53:43.440 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.424+0000 7fa0a1351640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:53:43.440 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.424+0000 7fa0a45b7640 5 monclient: authenticate success, global_id 834215 2024-09-16T03:53:43.441 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.424+0000 7fa0a45b7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:53:43.441 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.424+0000 7fa0a1351640 1 -- 172.21.15.42:0/1138001439 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa0900ac630 con 0x7fa09c0a1330 2024-09-16T03:53:43.441 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.424+0000 7fa0a1351640 10 monclient: handle_config config(0 keys) 2024-09-16T03:53:43.441 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.424+0000 7fa0a45b7640 10 monclient: get_monmap_and_config success 2024-09-16T03:53:43.441 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.424+0000 7fa0a45b7640 4 set_mon_vals no callback set 2024-09-16T03:53:43.441 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.424+0000 7fa0a1351640 1 -- 172.21.15.42:0/1138001439 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0900a8050 con 0x7fa09c0a1330 2024-09-16T03:53:43.441 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.424+0000 7fa0a1351640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:53:43.441 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.424+0000 7fa0a1351640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:53:43.441 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.424+0000 7fa0a1351640 10 monclient: dump: 2024-09-16T03:53:43.441 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:53:43.441 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:53:43.441 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:43.441 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:43.441 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:53:43.442 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:53:43.442 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:53:43.442 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:53:43.442 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:53:43.442 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:53:43.442 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.424+0000 7fa0a45b7640 10 monclient: shutdown 2024-09-16T03:53:43.442 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.424+0000 7fa0a45b7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:53:43.442 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.424+0000 7fa0a45b7640 1 -- 172.21.15.42:0/1138001439 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa09c0a1330 msgr2=0x7fa09c0a1710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:53:43.442 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.424+0000 7fa0a45b7640 1 --2- 172.21.15.42:0/1138001439 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa09c0a1330 0x7fa09c0a1710 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa090009870 tx=0x7fa09001c500 comp rx=0 tx=0).stop 2024-09-16T03:53:43.442 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.424+0000 7fa0a45b7640 1 -- 172.21.15.42:0/1138001439 shutdown_connections 2024-09-16T03:53:43.442 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.424+0000 7fa0a45b7640 1 --2- 172.21.15.42:0/1138001439 >> v2:172.21.15.154:3300/0 conn(0x7fa09c06dac0 0x7fa09c0a1c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:53:43.442 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.424+0000 7fa0a45b7640 1 --2- 172.21.15.42:0/1138001439 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa09c0a1330 0x7fa09c0a1710 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:53:43.442 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.424+0000 7fa0a45b7640 1 -- 172.21.15.42:0/1138001439 >> 172.21.15.42:0/1138001439 conn(0x7fa09c0a2420 msgr2=0x7fa09c0a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:53:43.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.424+0000 7fa0a45b7640 1 -- 172.21.15.42:0/1138001439 shutdown_connections 2024-09-16T03:53:43.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.424+0000 7fa0a45b7640 1 -- 172.21.15.42:0/1138001439 wait complete. 2024-09-16T03:53:43.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.424+0000 7fa0a45b7640 10 monclient: build_initial_monmap 2024-09-16T03:53:43.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.424+0000 7fa0a45b7640 10 monclient: monmap: 2024-09-16T03:53:43.443 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:53:43.443 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:53:43.443 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:53:43.443 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:53:43.443 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:53:43.443 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:53:43.443 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:53:43.443 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:53:43.443 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:53:43.443 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:53:43.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.424+0000 7fa0a45b7640 1 librados: starting msgr at 2024-09-16T03:53:43.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.424+0000 7fa0a45b7640 1 librados: starting objecter 2024-09-16T03:53:43.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.424+0000 7fa0a45b7640 1 Processor -- start 2024-09-16T03:53:43.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.424+0000 7fa0a45b7640 1 -- start start 2024-09-16T03:53:43.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.424+0000 7fa0a45b7640 1 librados: setting wanted keys 2024-09-16T03:53:43.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.424+0000 7fa0a45b7640 1 librados: calling monclient init 2024-09-16T03:53:43.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.424+0000 7fa0a45b7640 10 monclient: init 2024-09-16T03:53:43.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.424+0000 7fa0a45b7640 10 monclient: _reopen_session rank -1 2024-09-16T03:53:43.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.424+0000 7fa0a45b7640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T03:53:43.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.424+0000 7fa0a45b7640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa09c0a1330 0x7fa09c13b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:53:43.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.424+0000 7fa0a45b7640 10 monclient(hunting): picked mon.noname-a con 0x7fa09c0a1330 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:53:43.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.424+0000 7fa0a45b7640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa09c135100 0x7fa09c134820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:53:43.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.424+0000 7fa0a45b7640 10 monclient(hunting): picked mon.noname-b con 0x7fa09c135100 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:53:43.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.424+0000 7fa0a45b7640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa09c09cb60 0x7fa09c09cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:53:43.445 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.424+0000 7fa0a45b7640 10 monclient(hunting): picked mon.noname-c con 0x7fa09c09cb60 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:53:43.445 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.424+0000 7fa0a45b7640 10 monclient(hunting): start opening mon connection 2024-09-16T03:53:43.445 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.424+0000 7fa0a45b7640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fa09c13fd00 con 0x7fa09c0a1330 2024-09-16T03:53:43.445 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.424+0000 7fa0a45b7640 10 monclient(hunting): start opening mon connection 2024-09-16T03:53:43.445 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.424+0000 7fa0a45b7640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa09c13fb80 con 0x7fa09c135100 2024-09-16T03:53:43.445 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.424+0000 7fa0a45b7640 10 monclient(hunting): start opening mon connection 2024-09-16T03:53:43.445 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.424+0000 7fa0a45b7640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fa09c13fe80 con 0x7fa09c09cb60 2024-09-16T03:53:43.445 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.424+0000 7fa0a45b7640 10 monclient(hunting): _renew_subs 2024-09-16T03:53:43.445 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.424+0000 7fa0a45b7640 10 monclient(hunting): authenticate will time out at 1670.399671s 2024-09-16T03:53:43.445 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.424+0000 7fa0a2353640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa09c0a1330 0x7fa09c13b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:53:43.445 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.424+0000 7fa0a2b54640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa09c09cb60 0x7fa09c09cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:53:43.445 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.424+0000 7fa0a2b54640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa09c09cb60 0x7fa09c09cfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:39868/0 (socket says 172.21.15.42:39868) 2024-09-16T03:53:43.445 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.424+0000 7fa0a2b54640 1 -- 172.21.15.42:0/420863515 learned_addr learned my addr 172.21.15.42:0/420863515 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:53:43.445 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.424+0000 7fa0a2b54640 10 monclient(hunting): get_auth_request con 0x7fa09c09cb60 auth_method 0 2024-09-16T03:53:43.446 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.424+0000 7fa0a2353640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa09c0a1330 0x7fa09c13b510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:44570/0 (socket says 172.21.15.42:44570) 2024-09-16T03:53:43.446 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.424+0000 7fa0a2b54640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:53:43.446 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.424+0000 7fa0a2b54640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:53:43.446 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.424+0000 7fa0a2b54640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:53:43.446 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.424+0000 7fa0a2353640 10 monclient(hunting): get_auth_request con 0x7fa09c0a1330 auth_method 0 2024-09-16T03:53:43.446 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.424+0000 7fa0a2353640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:53:43.446 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.424+0000 7fa0a2353640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:53:43.446 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.424+0000 7fa0a2353640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:53:43.446 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.424+0000 7fa0a2b54640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:53:43.446 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.424+0000 7fa0a2b54640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:53:43.446 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.424+0000 7fa0a2b54640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:53:43.446 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.424+0000 7fa0a2353640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:53:43.446 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.424+0000 7fa0a2353640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:53:43.446 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.428+0000 7fa0a2353640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:53:43.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.428+0000 7fa0a1b52640 1 --2- 172.21.15.42:0/420863515 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa09c135100 0x7fa09c134820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:53:43.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.428+0000 7fa0a1b52640 10 monclient(hunting): get_auth_request con 0x7fa09c135100 auth_method 0 2024-09-16T03:53:43.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.428+0000 7fa0a1b52640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:53:43.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.428+0000 7fa0a1b52640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:53:43.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.428+0000 7fa0a1b52640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:53:43.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.428+0000 7fa0a2b54640 10 monclient(hunting): handle_auth_done global_id 834218 payload 824 2024-09-16T03:53:43.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.428+0000 7fa0a2b54640 1 -- 172.21.15.42:0/420863515 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa09c135100 msgr2=0x7fa09c134820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:53:43.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.428+0000 7fa0a2b54640 1 --2- 172.21.15.42:0/420863515 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa09c135100 0x7fa09c134820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:53:43.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.428+0000 7fa0a2b54640 1 -- 172.21.15.42:0/420863515 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa09c0a1330 msgr2=0x7fa09c13b510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:53:43.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.428+0000 7fa0a2b54640 1 --2- 172.21.15.42:0/420863515 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa09c0a1330 0x7fa09c13b510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:53:43.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.428+0000 7fa0a2b54640 10 monclient: _finish_hunting 0 2024-09-16T03:53:43.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.428+0000 7fa0a2b54640 1 monclient: found mon.noname-c 2024-09-16T03:53:43.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.428+0000 7fa0a2b54640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:53:43.448 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.428+0000 7fa0a2b54640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:53:43.448 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.428+0000 7fa0a2b54640 1 -- 172.21.15.42:0/420863515 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa09c0a51e0 con 0x7fa09c09cb60 2024-09-16T03:53:43.448 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.428+0000 7fa0a2353640 1 --2- 172.21.15.42:0/420863515 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa09c0a1330 0x7fa09c13b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:53:43.448 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.428+0000 7fa0a2b54640 1 --2- 172.21.15.42:0/420863515 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa09c09cb60 0x7fa09c09cfc0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fa09400c780 tx=0x7fa09400cc40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:53:43.448 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.428+0000 7fa08effd640 10 client.?.objecter ms_handle_connect 0x7fa09c09cb60 2024-09-16T03:53:43.448 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.428+0000 7fa08effd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:53:43.448 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.428+0000 7fa08effd640 1 -- 172.21.15.42:0/420863515 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa094017070 con 0x7fa09c09cb60 2024-09-16T03:53:43.448 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.428+0000 7fa08effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:53:43.448 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.428+0000 7fa08effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:53:43.448 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.428+0000 7fa08effd640 10 monclient: dump: 2024-09-16T03:53:43.448 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:53:43.448 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:53:43.448 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:43.449 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:43.449 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:53:43.449 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:53:43.449 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:53:43.449 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:53:43.449 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:53:43.449 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:53:43.449 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.428+0000 7fa08effd640 10 monclient: _finish_auth 0 2024-09-16T03:53:43.449 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.428+0000 7fa08effd640 10 monclient: _check_auth_tickets 2024-09-16T03:53:43.449 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.428+0000 7fa08effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:53:43.449 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.428+0000 7fa08effd640 1 -- 172.21.15.42:0/420863515 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa094003db0 con 0x7fa09c09cb60 2024-09-16T03:53:43.449 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.428+0000 7fa0a45b7640 5 monclient: authenticate success, global_id 834218 2024-09-16T03:53:43.450 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.428+0000 7fa0a45b7640 10 monclient: _renew_subs 2024-09-16T03:53:43.450 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.428+0000 7fa0a45b7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:53:43.450 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.428+0000 7fa0a45b7640 1 -- 172.21.15.42:0/420863515 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa09c073a30 con 0x7fa09c09cb60 2024-09-16T03:53:43.450 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.428+0000 7fa08effd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:53:43.450 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.428+0000 7fa08effd640 1 -- 172.21.15.42:0/420863515 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0940052b0 con 0x7fa09c09cb60 2024-09-16T03:53:43.450 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.428+0000 7fa08effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:53:43.450 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.428+0000 7fa08effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:53:43.450 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.428+0000 7fa08effd640 10 monclient: dump: 2024-09-16T03:53:43.450 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:53:43.450 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:53:43.450 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:43.450 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:43.450 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:53:43.450 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:53:43.451 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:53:43.451 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:53:43.451 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:53:43.451 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:53:43.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.428+0000 7fa0a45b7640 10 client.834218.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:53:43.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.428+0000 7fa0a45b7640 10 monclient: _renew_subs 2024-09-16T03:53:43.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.428+0000 7fa0a0b50640 4 set_mon_vals no callback set 2024-09-16T03:53:43.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.428+0000 7fa0a45b7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:53:43.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.428+0000 7fa0a45b7640 1 -- 172.21.15.42:0/420863515 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa09c1cce50 con 0x7fa09c09cb60 2024-09-16T03:53:43.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.428+0000 7fa0a45b7640 1 librados: init done 2024-09-16T03:53:43.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.428+0000 7fa0a45b7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:53:43.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.428+0000 7fa0a45b7640 1 --2- 172.21.15.42:0/420863515 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa09c06dac0 0x7fa09c120e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:53:43.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.428+0000 7fa0a45b7640 10 monclient(hunting): start opening mon connection 2024-09-16T03:53:43.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.428+0000 7fa0a45b7640 1 -- 172.21.15.42:0/420863515 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa09c120b70 con 0x7fa09c06dac0 2024-09-16T03:53:43.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.428+0000 7fa08effd640 1 -- 172.21.15.42:0/420863515 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fa09401a070 con 0x7fa09c09cb60 2024-09-16T03:53:43.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.428+0000 7fa08effd640 1 --2- 172.21.15.42:0/420863515 >> v2:172.21.15.154:6812/254845342 conn(0x7fa080037840 0x7fa080039d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:53:43.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.428+0000 7fa08effd640 1 -- 172.21.15.42:0/420863515 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(366..366 src has 286..366) ==== 5713+0+0 (secure 0 0 0) 0x7fa0940485a0 con 0x7fa09c09cb60 2024-09-16T03:53:43.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.428+0000 7fa08effd640 10 client.834218.objecter ms_dispatch 0x7fa09c000ba0 osd_map(366..366 src has 286..366) 2024-09-16T03:53:43.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.428+0000 7fa08effd640 3 client.834218.objecter handle_osd_map got epochs [366,366] > 0 2024-09-16T03:53:43.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.428+0000 7fa08effd640 3 client.834218.objecter handle_osd_map decoding full epoch 366 2024-09-16T03:53:43.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.428+0000 7fa08effd640 20 client.834218.objecter dump_active .. 0 homeless 2024-09-16T03:53:43.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.428+0000 7fa0a2353640 1 --2- 172.21.15.42:0/420863515 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa09c06dac0 0x7fa09c120e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:53:43.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.428+0000 7fa0a2353640 10 monclient: get_auth_request con 0x7fa09c06dac0 auth_method 0 2024-09-16T03:53:43.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.428+0000 7fa0a2353640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:53:43.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.428+0000 7fa0a2353640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:53:43.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.428+0000 7fa0a2353640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:53:43.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.428+0000 7fa0a1b52640 1 --2- 172.21.15.42:0/420863515 >> v2:172.21.15.154:6812/254845342 conn(0x7fa080037840 0x7fa080039d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:53:43.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.428+0000 7fa0a1b52640 10 monclient: get_auth_request con 0x7fa080037840 auth_method 0 2024-09-16T03:53:43.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.428+0000 7fa0a2353640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:53:43.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.428+0000 7fa0a2353640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:53:43.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.428+0000 7fa0a2353640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:53:43.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.428+0000 7fa0a2353640 10 monclient(hunting): handle_auth_done global_id 844114 payload 824 2024-09-16T03:53:43.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.428+0000 7fa0a2353640 1 -- 172.21.15.42:0/420863515 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa09c135550 con 0x7fa09c06dac0 2024-09-16T03:53:43.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.428+0000 7fa08effd640 10 client.834218.objecter ms_handle_connect 0x7fa080037840 2024-09-16T03:53:43.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.428+0000 7fa0a1b52640 1 --2- 172.21.15.42:0/420863515 >> v2:172.21.15.154:6812/254845342 conn(0x7fa080037840 0x7fa080039d00 secure :-1 s=READY pgs=3602 cs=0 l=1 rev1=1 crypto rx=0x7fa09c13d330 tx=0x7fa098008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:53:43.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.428+0000 7fa08effd640 10 client.834218.objecter ms_handle_connect 0x7fa09c06dac0 2024-09-16T03:53:43.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.428+0000 7fa08effd640 10 client.834218.objecter resend_mon_ops 2024-09-16T03:53:43.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.428+0000 7fa0a2353640 1 --2- 172.21.15.42:0/420863515 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa09c06dac0 0x7fa09c120e50 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fa09c09bb00 tx=0x7fa0900192a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:53:43.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.428+0000 7fa08effd640 1 -- 172.21.15.42:0/420863515 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0900bb070 con 0x7fa09c06dac0 2024-09-16T03:53:43.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.428+0000 7fa08effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:53:43.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.428+0000 7fa08effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:53:43.454 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.428+0000 7fa08effd640 10 monclient: dump: 2024-09-16T03:53:43.454 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:53:43.454 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:53:43.454 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:43.454 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:43.454 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:53:43.454 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:53:43.454 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:53:43.454 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:53:43.454 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:53:43.454 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:53:43.454 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.432+0000 7fa08effd640 1 -- 172.21.15.42:0/420863515 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fa0900b6720 con 0x7fa09c06dac0 2024-09-16T03:53:43.454 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.432+0000 7fa08effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:53:43.454 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.432+0000 7fa08effd640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:53:43.455 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.432+0000 7fa08effd640 1 -- 172.21.15.42:0/420863515 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa09c06dac0 msgr2=0x7fa09c120e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:53:43.455 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.432+0000 7fa08effd640 1 --2- 172.21.15.42:0/420863515 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa09c06dac0 0x7fa09c120e50 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fa09c09bb00 tx=0x7fa0900192a0 comp rx=0 tx=0).stop 2024-09-16T03:53:43.455 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.432+0000 7fa08effd640 1 -- 172.21.15.42:0/420863515 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa09c06dac0 msgr2=0x7fa09c120e50 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:53:43.455 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.432+0000 7fa08effd640 1 --2- 172.21.15.42:0/420863515 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa09c06dac0 0x7fa09c120e50 secure :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fa09c09bb00 tx=0x7fa0900192a0 comp rx=0 tx=0).stop 2024-09-16T03:53:43.456 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.436+0000 7fa0a45b7640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:53:43.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.436+0000 7fa0a45b7640 1 --2- 172.21.15.42:0/420863515 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa09c09d500 0x7fa09c138330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:53:43.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.436+0000 7fa0a45b7640 10 monclient(hunting): start opening mon connection 2024-09-16T03:53:43.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.436+0000 7fa0a45b7640 1 -- 172.21.15.42:0/420863515 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa09c13c070 con 0x7fa09c09d500 2024-09-16T03:53:43.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.436+0000 7fa0a2353640 1 --2- 172.21.15.42:0/420863515 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa09c09d500 0x7fa09c138330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:53:43.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.436+0000 7fa0a2353640 10 monclient: get_auth_request con 0x7fa09c09d500 auth_method 0 2024-09-16T03:53:43.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.436+0000 7fa0a2353640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:53:43.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.436+0000 7fa0a2353640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:53:43.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.436+0000 7fa0a2353640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:53:43.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.436+0000 7fa0a2353640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:53:43.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.436+0000 7fa0a2353640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:53:43.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.436+0000 7fa0a2353640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:53:43.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.436+0000 7fa0a2353640 10 monclient(hunting): handle_auth_done global_id 844117 payload 824 2024-09-16T03:53:43.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.436+0000 7fa0a2353640 1 -- 172.21.15.42:0/420863515 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fa09c071ec0 con 0x7fa09c09d500 2024-09-16T03:53:43.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.436+0000 7fa0a2353640 1 --2- 172.21.15.42:0/420863515 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa09c09d500 0x7fa09c138330 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fa090009870 tx=0x7fa09c0a22d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:53:43.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.436+0000 7fa08effd640 10 client.834218.objecter ms_handle_connect 0x7fa09c09d500 2024-09-16T03:53:43.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.436+0000 7fa08effd640 10 client.834218.objecter resend_mon_ops 2024-09-16T03:53:43.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.436+0000 7fa08effd640 1 -- 172.21.15.42:0/420863515 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0900bb040 con 0x7fa09c09d500 2024-09-16T03:53:43.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.436+0000 7fa08effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:53:43.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.440+0000 7fa08effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:53:43.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.440+0000 7fa08effd640 10 monclient: dump: 2024-09-16T03:53:43.458 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:53:43.458 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:53:43.458 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:43.458 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:43.458 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:53:43.458 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:53:43.458 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:53:43.459 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:53:43.459 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:53:43.459 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:53:43.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.440+0000 7fa08effd640 1 -- 172.21.15.42:0/420863515 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4544 (secure 0 0 0) 0x7fa0900ca420 con 0x7fa09c09d500 2024-09-16T03:53:43.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.440+0000 7fa08effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:53:43.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.440+0000 7fa08effd640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:53:43.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.440+0000 7fa08effd640 1 -- 172.21.15.42:0/420863515 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa09c09d500 msgr2=0x7fa09c138330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:53:43.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.440+0000 7fa08effd640 1 --2- 172.21.15.42:0/420863515 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa09c09d500 0x7fa09c138330 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fa090009870 tx=0x7fa09c0a22d0 comp rx=0 tx=0).stop 2024-09-16T03:53:43.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.440+0000 7fa08effd640 1 -- 172.21.15.42:0/420863515 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa09c09d500 msgr2=0x7fa09c138330 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:53:43.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.440+0000 7fa08effd640 1 --2- 172.21.15.42:0/420863515 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa09c09d500 0x7fa09c138330 secure :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fa090009870 tx=0x7fa09c0a22d0 comp rx=0 tx=0).stop 2024-09-16T03:53:43.459 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:53:43.460 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:53:43.460 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:53:43.460 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T03:53:43.460 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 100, 2024-09-16T03:53:43.460 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 7639, 2024-09-16T03:53:43.460 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:53:43.460 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:53:43.460 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T03:53:43.460 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T03:53:43.460 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:53:43.460 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 3, 2024-09-16T03:53:43.460 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:53:43.460 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:53:43.461 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:53:43.461 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:53:43.461 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:53:43.461 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:53:43.461 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:53:43.461 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:53:43.461 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:53:43.461 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:53:43.461 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:53:43.461 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:53:43.461 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:53:43.461 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:53:43.461 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:53:43.461 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:53:43.462 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:53:43.464 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:53:43.464 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:53:43.464 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:53:43.464 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:53:43.464 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:53:43.464 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:53:43.464 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:53:43.464 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:53:43.465 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:53:43.465 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:53:43.465 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:53:43.465 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:53:43.465 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:53:43.465 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:53:43.465 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:53:43.465 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:53:43.465 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:53:43.465 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:53:43.465 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:53:43.465 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:53:43.465 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:53:43.465 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:53:43.466 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:53:43.466 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:53:43.466 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:53:43.466 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:53:43.466 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:53:43.466 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:53:43.466 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:53:43.466 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:53:43.466 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:53:43.466 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:53:43.466 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:53:43.466 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:53:43.466 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:53:43.466 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:53:43.467 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:53:43.467 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:53:43.467 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:53:43.467 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:53:43.467 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:53:43.467 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:53:43.467 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:53:43.467 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:53:43.467 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:53:43.467 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:53:43.467 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:53:43.467 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:53:43.467 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:53:43.467 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:53:43.468 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:53:43.468 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:53:43.468 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:53:43.468 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:53:43.468 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:53:43.468 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:53:43.468 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:53:43.468 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:53:43.468 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:53:43.468 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:53:43.468 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:53:43.468 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:53:43.468 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:53:43.468 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:53:43.469 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:53:43.469 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:53:43.469 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:53:43.469 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:53:43.469 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:53:43.469 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:53:43.469 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:53:43.469 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:53:43.469 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:53:43.469 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:53:43.469 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:53:43.469 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:53:43.469 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:53:43.469 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:53:43.470 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:53:43.470 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:53:43.470 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:53:43.470 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:53:43.470 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:53:43.470 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:53:43.470 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:53:43.470 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:53:43.470 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:53:43.470 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:53:43.470 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:53:43.470 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:53:43.470 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:53:43.470 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:53:43.471 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:53:43.471 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:53:43.471 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:53:43.471 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:53:43.471 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:53:43.471 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:53:43.471 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:53:43.471 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:53:43.471 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:53:43.471 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:53:43.471 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:53:43.471 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:53:43.471 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:53:43.471 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:53:43.472 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:53:43.472 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:53:43.472 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:53:43.472 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:53:43.472 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:53:43.472 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:53:43.472 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:53:43.472 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:53:43.472 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:53:43.472 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:53:43.472 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:53:43.472 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:53:43.472 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:53:43.472 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:53:43.473 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:53:43.473 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:53:43.473 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:53:43.473 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:53:43.473 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:53:43.473 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:53:43.473 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:53:43.473 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:53:43.473 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:53:43.473 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:53:43.473 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:53:43.473 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:53:43.473 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:53:43.473 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:53:43.475 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.440+0000 7fa0a45b7640 10 librados: watch_flush enter 2024-09-16T03:53:43.475 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.440+0000 7fa0a45b7640 10 librados: watch_flush exit 2024-09-16T03:53:43.475 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.440+0000 7fa0a45b7640 20 client.834218.objecter shutdown clearing up homeless session... 2024-09-16T03:53:43.475 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.440+0000 7fa0a45b7640 10 client.834218.objecter successfully canceled tick 2024-09-16T03:53:43.475 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.440+0000 7fa0a45b7640 1 -- 172.21.15.42:0/420863515 >> v2:172.21.15.154:6812/254845342 conn(0x7fa080037840 msgr2=0x7fa080039d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:53:43.475 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.440+0000 7fa0a45b7640 1 --2- 172.21.15.42:0/420863515 >> v2:172.21.15.154:6812/254845342 conn(0x7fa080037840 0x7fa080039d00 secure :-1 s=READY pgs=3602 cs=0 l=1 rev1=1 crypto rx=0x7fa09c13d330 tx=0x7fa098008040 comp rx=0 tx=0).stop 2024-09-16T03:53:43.476 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.440+0000 7fa0a45b7640 10 monclient: shutdown 2024-09-16T03:53:43.476 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.440+0000 7fa0a45b7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:53:43.476 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.440+0000 7fa0a45b7640 1 -- 172.21.15.42:0/420863515 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa09c09cb60 msgr2=0x7fa09c09cfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:53:43.476 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.440+0000 7fa0a45b7640 1 --2- 172.21.15.42:0/420863515 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa09c09cb60 0x7fa09c09cfc0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fa09400c780 tx=0x7fa09400cc40 comp rx=0 tx=0).stop 2024-09-16T03:53:43.476 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.440+0000 7fa0a2b54640 1 -- 172.21.15.42:0/420863515 reap_dead start 2024-09-16T03:53:43.476 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.440+0000 7fa0a45b7640 1 -- 172.21.15.42:0/420863515 shutdown_connections 2024-09-16T03:53:43.476 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.440+0000 7fa0a45b7640 1 --2- 172.21.15.42:0/420863515 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa09c09cb60 0x7fa09c09cfc0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:53:43.476 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.440+0000 7fa0a45b7640 1 -- 172.21.15.42:0/420863515 >> 172.21.15.42:0/420863515 conn(0x7fa09c0a2420 msgr2=0x7fa09c13c590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:53:43.476 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.440+0000 7fa0a45b7640 1 -- 172.21.15.42:0/420863515 shutdown_connections 2024-09-16T03:53:43.476 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.440+0000 7fa0a45b7640 1 -- 172.21.15.42:0/420863515 wait complete. 2024-09-16T03:53:43.476 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.440+0000 7fa0a45b7640 1 librados: shutdown 2024-09-16T03:53:43.477 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:53:43.477 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T03:53:43.477 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T03:53:43.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.584+0000 7f6426373640 10 monclient: get_monmap_and_config 2024-09-16T03:53:43.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.584+0000 7f6426373640 10 monclient: build_initial_monmap 2024-09-16T03:53:43.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.584+0000 7f6426373640 10 monclient: monmap: 2024-09-16T03:53:43.591 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:53:43.591 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:53:43.591 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:53:43.589385+0000 2024-09-16T03:53:43.591 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:53:43.589385+0000 2024-09-16T03:53:43.591 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:53:43.591 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:53:43.591 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:53:43.591 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:53:43.591 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:53:43.592 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:53:43.592 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:53:43.592 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:53:43.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.584+0000 7f6426373640 1 Processor -- start 2024-09-16T03:53:43.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.584+0000 7f6426373640 1 -- start start 2024-09-16T03:53:43.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.584+0000 7f6426373640 10 monclient: init 2024-09-16T03:53:43.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.584+0000 7f6426373640 10 monclient: _reopen_session rank -1 2024-09-16T03:53:43.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.584+0000 7f6426373640 10 monclient: _add_conns ranks=[2,3,1,0,4] 2024-09-16T03:53:43.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.584+0000 7f6426373640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f642006db90 0x7f64201340c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:53:43.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.584+0000 7f6426373640 10 monclient(hunting): picked mon.noname-b con 0x7f642006db90 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:53:43.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.584+0000 7f6426373640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f64200a37f0 0x7f64200a3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:53:43.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.584+0000 7f6426373640 10 monclient(hunting): picked mon.noname-a con 0x7f64200a37f0 addr v2:172.21.15.42:3300/0 2024-09-16T03:53:43.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.584+0000 7f6426373640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6420136e10 addr v1:172.21.15.154:6789/0 2024-09-16T03:53:43.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.584+0000 7f6426373640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f642006ece0 con 0x7f6420136e10 2024-09-16T03:53:43.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.584+0000 7f6426373640 10 monclient(hunting): start opening mon connection 2024-09-16T03:53:43.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.584+0000 7f6426373640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f64200a4110 con 0x7f64200a37f0 2024-09-16T03:53:43.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.584+0000 7f6426373640 10 monclient(hunting): start opening mon connection 2024-09-16T03:53:43.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.584+0000 7f6426373640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f642006df70 con 0x7f642006db90 2024-09-16T03:53:43.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.584+0000 7f6426373640 10 monclient(hunting): _renew_subs 2024-09-16T03:53:43.596 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.584+0000 7f6426373640 10 monclient(hunting): authenticate will time out at 1670.558930s 2024-09-16T03:53:43.596 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.584+0000 7f641ffff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f642006db90 0x7f64201340c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:53:43.596 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.584+0000 7f641f7fe640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f64200a37f0 0x7f64200a3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:53:43.596 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.584+0000 7f641ffff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f642006db90 0x7f64201340c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:39882/0 (socket says 172.21.15.42:39882) 2024-09-16T03:53:43.596 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.584+0000 7f641ffff640 1 -- 172.21.15.42:0/4291178797 learned_addr learned my addr 172.21.15.42:0/4291178797 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:53:43.596 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.584+0000 7f641ffff640 10 monclient(hunting): get_auth_request con 0x7f642006db90 auth_method 0 2024-09-16T03:53:43.596 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.584+0000 7f641ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:53:43.596 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.584+0000 7f641ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:53:43.596 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.584+0000 7f641ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:53:43.596 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.584+0000 7f641f7fe640 10 monclient(hunting): get_auth_request con 0x7f64200a37f0 auth_method 0 2024-09-16T03:53:43.596 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.584+0000 7f641f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:53:43.596 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.584+0000 7f641f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:53:43.596 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.584+0000 7f641f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:53:43.596 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.584+0000 7f641ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:53:43.597 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.584+0000 7f641ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:53:43.597 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.584+0000 7f641ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:53:43.597 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.584+0000 7f641f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:53:43.597 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.584+0000 7f641f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:53:43.597 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.584+0000 7f641f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:53:43.597 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.584+0000 7f641ffff640 10 monclient(hunting): handle_auth_done global_id 834221 payload 274 2024-09-16T03:53:43.597 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.584+0000 7f641ffff640 1 -- 172.21.15.42:0/4291178797 >> v2:172.21.15.42:3300/0 conn(0x7f64200a37f0 msgr2=0x7f64200a3bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:53:43.597 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.584+0000 7f641ffff640 1 --2- 172.21.15.42:0/4291178797 >> v2:172.21.15.42:3300/0 conn(0x7f64200a37f0 0x7f64200a3bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:53:43.597 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.584+0000 7f641ffff640 1 -- 172.21.15.42:0/4291178797 >> v1:172.21.15.154:6789/0 conn(0x7f6420136e10 legacy=0x7f6420137220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:53:43.597 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.584+0000 7f641ffff640 10 monclient: _finish_hunting 0 2024-09-16T03:53:43.597 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.584+0000 7f641ffff640 1 monclient: found mon.noname-b 2024-09-16T03:53:43.597 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.584+0000 7f641ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:53:43.597 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.584+0000 7f641ffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T03:53:43.598 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.584+0000 7f641ffff640 1 -- 172.21.15.42:0/4291178797 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f642009bf80 con 0x7f642006db90 2024-09-16T03:53:43.598 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.584+0000 7f641f7fe640 1 --2- 172.21.15.42:0/4291178797 >> v2:172.21.15.42:3300/0 conn(0x7f64200a37f0 0x7f64200a3bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:53:43.598 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.588+0000 7f641ffff640 1 --2- 172.21.15.42:0/4291178797 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f642006db90 0x7f64201340c0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f6418009e50 tx=0x7f641801c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=212c87dae064b9fc server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:53:43.603 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.588+0000 7f641effd640 1 -- 172.21.15.42:0/4291178797 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64180a56a0 con 0x7f642006db90 2024-09-16T03:53:43.603 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.588+0000 7f641effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:53:43.604 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.588+0000 7f641effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:53:43.604 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.588+0000 7f641effd640 10 monclient: dump: 2024-09-16T03:53:43.604 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:53:43.604 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:53:43.604 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:43.604 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:43.604 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:53:43.604 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:53:43.604 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:53:43.604 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:53:43.604 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:53:43.604 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:53:43.605 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.588+0000 7f641effd640 10 monclient: _finish_auth 0 2024-09-16T03:53:43.605 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.588+0000 7f641effd640 10 monclient: _check_auth_tickets 2024-09-16T03:53:43.605 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.588+0000 7f641effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:53:43.605 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.588+0000 7f641effd640 1 -- 172.21.15.42:0/4291178797 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f64180a5840 con 0x7f642006db90 2024-09-16T03:53:43.605 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.588+0000 7f6426373640 5 monclient: authenticate success, global_id 834221 2024-09-16T03:53:43.605 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.588+0000 7f6426373640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:53:43.605 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.588+0000 7f641effd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:53:43.605 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.588+0000 7f641effd640 1 -- 172.21.15.42:0/4291178797 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64180a5c10 con 0x7f642006db90 2024-09-16T03:53:43.605 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.588+0000 7f6426373640 10 monclient: get_monmap_and_config success 2024-09-16T03:53:43.605 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.588+0000 7f6426373640 4 set_mon_vals no callback set 2024-09-16T03:53:43.605 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.588+0000 7f641effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:53:43.605 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.588+0000 7f641effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:53:43.605 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.588+0000 7f641effd640 10 monclient: dump: 2024-09-16T03:53:43.605 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:53:43.606 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:53:43.606 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:43.606 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:43.606 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:53:43.606 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:53:43.606 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:53:43.607 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:53:43.607 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:53:43.607 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:53:43.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.588+0000 7f6426373640 10 monclient: shutdown 2024-09-16T03:53:43.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.588+0000 7f6426373640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:53:43.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.588+0000 7f6426373640 1 -- 172.21.15.42:0/4291178797 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f642006db90 msgr2=0x7f64201340c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:53:43.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.588+0000 7f6426373640 1 --2- 172.21.15.42:0/4291178797 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f642006db90 0x7f64201340c0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f6418009e50 tx=0x7f641801c8f0 comp rx=0 tx=0).stop 2024-09-16T03:53:43.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.588+0000 7f6426373640 1 -- 172.21.15.42:0/4291178797 shutdown_connections 2024-09-16T03:53:43.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.588+0000 7f6426373640 1 --2- 172.21.15.42:0/4291178797 >> v2:172.21.15.42:3300/0 conn(0x7f64200a37f0 0x7f64200a3bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:53:43.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.588+0000 7f6426373640 1 --2- 172.21.15.42:0/4291178797 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f642006db90 0x7f64201340c0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:53:43.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.588+0000 7f6426373640 1 -- 172.21.15.42:0/4291178797 >> 172.21.15.42:0/4291178797 conn(0x7f64200a9fb0 msgr2=0x7f64200aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:53:43.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.588+0000 7f6426373640 1 -- 172.21.15.42:0/4291178797 shutdown_connections 2024-09-16T03:53:43.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.588+0000 7f6426373640 1 -- 172.21.15.42:0/4291178797 wait complete. 2024-09-16T03:53:43.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.588+0000 7f6426373640 10 monclient: build_initial_monmap 2024-09-16T03:53:43.609 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.588+0000 7f6426373640 10 monclient: monmap: 2024-09-16T03:53:43.609 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:53:43.609 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:53:43.609 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:53:43.609 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:53:43.609 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:53:43.609 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:53:43.609 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:53:43.609 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:53:43.610 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:53:43.610 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:53:43.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.588+0000 7f6426373640 1 librados: starting msgr at 2024-09-16T03:53:43.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.588+0000 7f6426373640 1 librados: starting objecter 2024-09-16T03:53:43.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.588+0000 7f6426373640 1 Processor -- start 2024-09-16T03:53:43.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.588+0000 7f6426373640 1 -- start start 2024-09-16T03:53:43.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.588+0000 7f6426373640 1 librados: setting wanted keys 2024-09-16T03:53:43.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.588+0000 7f6426373640 1 librados: calling monclient init 2024-09-16T03:53:43.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.588+0000 7f6426373640 10 monclient: init 2024-09-16T03:53:43.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.588+0000 7f6426373640 10 monclient: _reopen_session rank -1 2024-09-16T03:53:43.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.588+0000 7f6426373640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T03:53:43.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.588+0000 7f6426373640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f642006db90 0x7f64201cf250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:53:43.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.588+0000 7f6426373640 10 monclient(hunting): picked mon.noname-b con 0x7f642006db90 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:53:43.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.588+0000 7f6426373640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f64200a37f0 0x7f642013e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:53:43.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.588+0000 7f6426373640 10 monclient(hunting): picked mon.noname-c con 0x7f64200a37f0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:53:43.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.588+0000 7f6426373640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6420136e10 0x7f642013e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:53:43.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.588+0000 7f6426373640 10 monclient(hunting): picked mon.noname-a con 0x7f6420136e10 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:53:43.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.588+0000 7f6426373640 10 monclient(hunting): start opening mon connection 2024-09-16T03:53:43.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.588+0000 7f6426373640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f6420141d60 con 0x7f6420136e10 2024-09-16T03:53:43.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.588+0000 7f6426373640 10 monclient(hunting): start opening mon connection 2024-09-16T03:53:43.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.588+0000 7f6426373640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f6420141be0 con 0x7f642006db90 2024-09-16T03:53:43.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.588+0000 7f6426373640 10 monclient(hunting): start opening mon connection 2024-09-16T03:53:43.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.592+0000 7f6426373640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f6420141ee0 con 0x7f64200a37f0 2024-09-16T03:53:43.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.592+0000 7f6426373640 10 monclient(hunting): _renew_subs 2024-09-16T03:53:43.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.592+0000 7f6426373640 10 monclient(hunting): authenticate will time out at 1670.564018s 2024-09-16T03:53:43.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.592+0000 7f641f7fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f64200a37f0 0x7f642013e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:53:43.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.592+0000 7f641f7fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f64200a37f0 0x7f642013e320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:39890/0 (socket says 172.21.15.42:39890) 2024-09-16T03:53:43.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.592+0000 7f641f7fe640 1 -- 172.21.15.42:0/4140041703 learned_addr learned my addr 172.21.15.42:0/4140041703 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:53:43.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.592+0000 7f641f7fe640 10 monclient(hunting): get_auth_request con 0x7f64200a37f0 auth_method 0 2024-09-16T03:53:43.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.592+0000 7f641f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:53:43.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.592+0000 7f641f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:53:43.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.592+0000 7f641f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:53:43.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.592+0000 7f6424910640 1 --2- 172.21.15.42:0/4140041703 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6420136e10 0x7f642013e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:53:43.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.592+0000 7f641f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:53:43.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.592+0000 7f641f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:53:43.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.592+0000 7f641f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:53:43.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.592+0000 7f6424910640 10 monclient(hunting): get_auth_request con 0x7f6420136e10 auth_method 0 2024-09-16T03:53:43.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.592+0000 7f6424910640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:53:43.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.592+0000 7f6424910640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:53:43.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.592+0000 7f6424910640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:53:43.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.592+0000 7f6424910640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:53:43.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.592+0000 7f6424910640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:53:43.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.592+0000 7f641ffff640 1 --2- 172.21.15.42:0/4140041703 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f642006db90 0x7f64201cf250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:53:43.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.592+0000 7f6424910640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:53:43.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.592+0000 7f641f7fe640 10 monclient(hunting): handle_auth_done global_id 834224 payload 824 2024-09-16T03:53:43.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.592+0000 7f641f7fe640 1 -- 172.21.15.42:0/4140041703 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f642006db90 msgr2=0x7f64201cf250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:53:43.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.592+0000 7f641f7fe640 1 --2- 172.21.15.42:0/4140041703 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f642006db90 0x7f64201cf250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:53:43.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.592+0000 7f641f7fe640 1 -- 172.21.15.42:0/4140041703 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6420136e10 msgr2=0x7f642013e860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:53:43.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.592+0000 7f641f7fe640 1 --2- 172.21.15.42:0/4140041703 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6420136e10 0x7f642013e860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:53:43.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.592+0000 7f641f7fe640 10 monclient: _finish_hunting 0 2024-09-16T03:53:43.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.592+0000 7f641f7fe640 1 monclient: found mon.noname-c 2024-09-16T03:53:43.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.592+0000 7f641f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:53:43.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.592+0000 7f641f7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:53:43.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.592+0000 7f641f7fe640 1 -- 172.21.15.42:0/4140041703 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64201323e0 con 0x7f64200a37f0 2024-09-16T03:53:43.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.592+0000 7f641ffff640 10 monclient: get_auth_request con 0x7f642006db90 auth_method 0 2024-09-16T03:53:43.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.592+0000 7f641ffff640 1 --2- 172.21.15.42:0/4140041703 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f642006db90 0x7f64201cf250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T03:53:43.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.592+0000 7f641f7fe640 1 --2- 172.21.15.42:0/4140041703 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f64200a37f0 0x7f642013e320 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f640800ebe0 tx=0x7f640800c520 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:53:43.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.592+0000 7f641cff9640 10 client.?.objecter ms_handle_connect 0x7f64200a37f0 2024-09-16T03:53:43.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.592+0000 7f641cff9640 10 client.?.objecter resend_mon_ops 2024-09-16T03:53:43.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.592+0000 7f641cff9640 1 -- 172.21.15.42:0/4140041703 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6408017070 con 0x7f64200a37f0 2024-09-16T03:53:43.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.592+0000 7f641cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:53:43.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.592+0000 7f641cff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:53:43.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.592+0000 7f641cff9640 10 monclient: dump: 2024-09-16T03:53:43.616 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:53:43.616 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:53:43.616 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:43.616 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:43.616 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:53:43.616 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:53:43.616 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:53:43.616 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:53:43.616 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:53:43.617 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:53:43.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.592+0000 7f641cff9640 10 monclient: _finish_auth 0 2024-09-16T03:53:43.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.592+0000 7f641cff9640 10 monclient: _check_auth_tickets 2024-09-16T03:53:43.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.592+0000 7f641cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:53:43.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.592+0000 7f641cff9640 1 -- 172.21.15.42:0/4140041703 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f640800cc30 con 0x7f64200a37f0 2024-09-16T03:53:43.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.592+0000 7f6426373640 5 monclient: authenticate success, global_id 834224 2024-09-16T03:53:43.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.592+0000 7f6426373640 10 monclient: _renew_subs 2024-09-16T03:53:43.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.592+0000 7f6426373640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:53:43.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.592+0000 7f6426373640 1 -- 172.21.15.42:0/4140041703 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64201cf790 con 0x7f64200a37f0 2024-09-16T03:53:43.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.592+0000 7f641cff9640 10 monclient: handle_config config(0 keys) 2024-09-16T03:53:43.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.592+0000 7f641cff9640 1 -- 172.21.15.42:0/4140041703 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6408013470 con 0x7f64200a37f0 2024-09-16T03:53:43.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.592+0000 7f641cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:53:43.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.592+0000 7f641cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:53:43.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.592+0000 7f641cff9640 10 monclient: dump: 2024-09-16T03:53:43.618 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:53:43.618 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:53:43.618 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:43.618 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:43.618 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:53:43.618 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:53:43.618 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:53:43.618 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:53:43.618 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:53:43.618 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:53:43.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.592+0000 7f641e7fc640 4 set_mon_vals no callback set 2024-09-16T03:53:43.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.592+0000 7f6426373640 10 client.834224.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:53:43.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.592+0000 7f6426373640 10 monclient: _renew_subs 2024-09-16T03:53:43.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.592+0000 7f6426373640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:53:43.619 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.592+0000 7f6426373640 1 -- 172.21.15.42:0/4140041703 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f642006f6a0 con 0x7f64200a37f0 2024-09-16T03:53:43.619 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.592+0000 7f6426373640 1 librados: init done 2024-09-16T03:53:43.619 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.592+0000 7f641cff9640 1 -- 172.21.15.42:0/4140041703 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f640801a070 con 0x7f64200a37f0 2024-09-16T03:53:43.619 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.592+0000 7f6426373640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:53:43.619 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.592+0000 7f6426373640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:53:43.619 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.592+0000 7f641cff9640 1 --2- 172.21.15.42:0/4140041703 >> v2:172.21.15.154:6812/254845342 conn(0x7f6404037810 0x7f6404039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:53:43.619 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.592+0000 7f6426373640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:53:43.619 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.592+0000 7f6426373640 1 -- 172.21.15.42:0/4140041703 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f64201340c0 con 0x7f64200a37f0 2024-09-16T03:53:43.619 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.592+0000 7f641cff9640 1 -- 172.21.15.42:0/4140041703 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(366..366 src has 286..366) ==== 5713+0+0 (secure 0 0 0) 0x7f6408049280 con 0x7f64200a37f0 2024-09-16T03:53:43.620 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.592+0000 7f641cff9640 10 client.834224.objecter ms_dispatch 0x7f6420000ba0 osd_map(366..366 src has 286..366) 2024-09-16T03:53:43.620 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.592+0000 7f641cff9640 3 client.834224.objecter handle_osd_map got epochs [366,366] > 0 2024-09-16T03:53:43.620 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.592+0000 7f641cff9640 3 client.834224.objecter handle_osd_map decoding full epoch 366 2024-09-16T03:53:43.620 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.592+0000 7f641cff9640 20 client.834224.objecter dump_active .. 0 homeless 2024-09-16T03:53:43.620 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.592+0000 7f641ffff640 1 --2- 172.21.15.42:0/4140041703 >> v2:172.21.15.154:6812/254845342 conn(0x7f6404037810 0x7f6404039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:53:43.620 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.592+0000 7f641ffff640 10 monclient: get_auth_request con 0x7f6404037810 auth_method 0 2024-09-16T03:53:43.620 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.592+0000 7f641cff9640 10 client.834224.objecter ms_handle_connect 0x7f6404037810 2024-09-16T03:53:43.620 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.592+0000 7f641ffff640 1 --2- 172.21.15.42:0/4140041703 >> v2:172.21.15.154:6812/254845342 conn(0x7f6404037810 0x7f6404039cd0 secure :-1 s=READY pgs=3603 cs=0 l=1 rev1=1 crypto rx=0x7f64180ae000 tx=0x7f64180a7070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:53:43.620 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.596+0000 7f641cff9640 1 -- 172.21.15.42:0/4140041703 <== mon.2 v2:172.21.15.42:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f6408013ae0 con 0x7f64200a37f0 2024-09-16T03:53:43.620 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.596+0000 7f641cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:53:43.620 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.596+0000 7f641cff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:53:43.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.824+0000 7f6426373640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T03:53:43.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.824+0000 7f6426373640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T03:53:43.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.824+0000 7f6426373640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:53:43.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.824+0000 7f6426373640 1 -- 172.21.15.42:0/4140041703 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f64201383b0 con 0x7f64200a37f0 2024-09-16T03:53:43.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.828+0000 7f641cff9640 1 -- 172.21.15.42:0/4140041703 <== mon.2 v2:172.21.15.42:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f6408013610 con 0x7f64200a37f0 2024-09-16T03:53:43.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.828+0000 7f641cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T03:53:43.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.828+0000 7f641cff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:53:43.836 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.828+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:53:43.837 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.828+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=27,config=2,health=11,kv=10,logm=50} crc {auth=3189390035,config=692619927,health=1477407780,kv=3446188498,logm=2900279878}) 2024-09-16T03:53:43.837 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.828+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=28,config=2,health=12,kv=10,logm=48} crc {auth=1213469563,config=2435484489,health=4222336806,kv=3446188498,logm=1543680878}) 2024-09-16T03:53:43.837 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.828+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:53:43.837 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.828+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=27,config=2,health=11,kv=10,logm=50} crc {auth=3189390035,config=692619927,health=1477407780,kv=3446188498,logm=2900279878}) 2024-09-16T03:53:43.837 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.828+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=28,config=2,health=11,kv=10,logm=49} crc {auth=1213469563,config=692619927,health=4217736590,kv=3446188498,logm=995569800}) 2024-09-16T03:53:43.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.828+0000 7f6426373640 10 librados: watch_flush enter 2024-09-16T03:53:43.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.832+0000 7f6426373640 10 librados: watch_flush exit 2024-09-16T03:53:43.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.832+0000 7f6426373640 20 client.834224.objecter shutdown clearing up homeless session... 2024-09-16T03:53:43.841 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.832+0000 7f6426373640 10 client.834224.objecter successfully canceled tick 2024-09-16T03:53:43.841 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.832+0000 7f6426373640 1 -- 172.21.15.42:0/4140041703 >> v2:172.21.15.154:6812/254845342 conn(0x7f6404037810 msgr2=0x7f6404039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:53:43.841 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.832+0000 7f6426373640 1 --2- 172.21.15.42:0/4140041703 >> v2:172.21.15.154:6812/254845342 conn(0x7f6404037810 0x7f6404039cd0 secure :-1 s=READY pgs=3603 cs=0 l=1 rev1=1 crypto rx=0x7f64180ae000 tx=0x7f64180a7070 comp rx=0 tx=0).stop 2024-09-16T03:53:43.841 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.832+0000 7f6426373640 10 monclient: shutdown 2024-09-16T03:53:43.841 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.832+0000 7f6426373640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:53:43.841 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.832+0000 7f6426373640 1 -- 172.21.15.42:0/4140041703 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f64200a37f0 msgr2=0x7f642013e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:53:43.842 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.832+0000 7f6426373640 1 --2- 172.21.15.42:0/4140041703 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f64200a37f0 0x7f642013e320 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f640800ebe0 tx=0x7f640800c520 comp rx=0 tx=0).stop 2024-09-16T03:53:43.842 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.832+0000 7f6426373640 1 -- 172.21.15.42:0/4140041703 shutdown_connections 2024-09-16T03:53:43.842 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.832+0000 7f6426373640 1 --2- 172.21.15.42:0/4140041703 >> v2:172.21.15.154:6812/254845342 conn(0x7f6404037810 0x7f6404039cd0 unknown :-1 s=CLOSED pgs=3603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:53:43.842 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.832+0000 7f6426373640 1 --2- 172.21.15.42:0/4140041703 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6420136e10 0x7f642013e860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:53:43.842 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.832+0000 7f6426373640 1 --2- 172.21.15.42:0/4140041703 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f64200a37f0 0x7f642013e320 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:53:43.843 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.832+0000 7f6426373640 1 --2- 172.21.15.42:0/4140041703 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f642006db90 0x7f64201cf250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:53:43.843 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.832+0000 7f6426373640 1 -- 172.21.15.42:0/4140041703 >> 172.21.15.42:0/4140041703 conn(0x7f64200a9fb0 msgr2=0x7f64201327f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:53:43.843 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.832+0000 7f6426373640 1 -- 172.21.15.42:0/4140041703 shutdown_connections 2024-09-16T03:53:43.843 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.832+0000 7f6426373640 1 -- 172.21.15.42:0/4140041703 wait complete. 2024-09-16T03:53:43.843 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:43.832+0000 7f6426373640 1 librados: shutdown 2024-09-16T03:53:43.845 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.832+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:53:43.846 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.832+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3516555275}) 2024-09-16T03:53:43.846 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.832+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1885368040}) 2024-09-16T03:53:43.846 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.832+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:53:43.847 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.832+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3516555275}) 2024-09-16T03:53:43.847 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.832+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2867313389}) 2024-09-16T03:53:43.847 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.836+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:53:43.847 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.836+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3796732905}) 2024-09-16T03:53:43.847 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.836+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=312333722}) 2024-09-16T03:53:43.848 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.836+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:53:43.848 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.836+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3796732905}) 2024-09-16T03:53:43.848 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.836+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3160728080}) 2024-09-16T03:53:43.848 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.840+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:53:43.848 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.840+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2835689561}) 2024-09-16T03:53:43.848 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.840+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=95268145}) 2024-09-16T03:53:43.849 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.840+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:53:43.849 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.840+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2835689561}) 2024-09-16T03:53:43.849 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.840+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2520802906}) 2024-09-16T03:53:43.851 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.840+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:53:43.852 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.840+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=121629006}) 2024-09-16T03:53:43.852 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.840+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4146261325}) 2024-09-16T03:53:43.852 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.840+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:53:43.852 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.840+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=121629006}) 2024-09-16T03:53:43.852 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.840+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=951174849}) 2024-09-16T03:53:43.853 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.840+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:53:43.853 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.840+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2647296458}) 2024-09-16T03:53:43.853 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.840+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2500042979}) 2024-09-16T03:53:43.853 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.840+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:53:43.853 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.840+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2647296458}) 2024-09-16T03:53:43.853 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.840+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1600835066}) 2024-09-16T03:53:43.854 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.844+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:53:43.854 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.844+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=49403054}) 2024-09-16T03:53:43.854 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.844+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3044753063}) 2024-09-16T03:53:43.854 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.844+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:53:43.854 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.844+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=49403054}) 2024-09-16T03:53:43.854 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.844+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1740745853}) 2024-09-16T03:53:43.857 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.844+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:53:43.857 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.844+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=701128002}) 2024-09-16T03:53:43.857 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.844+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3500696226}) 2024-09-16T03:53:43.857 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.844+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:53:43.857 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.844+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=701128002}) 2024-09-16T03:53:43.857 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.844+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=926441235}) 2024-09-16T03:53:43.857 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.848+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:53:43.858 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.848+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=936838987}) 2024-09-16T03:53:43.858 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.848+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3740824683}) 2024-09-16T03:53:43.858 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.848+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:53:43.858 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.848+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=936838987}) 2024-09-16T03:53:43.858 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.848+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=423387641}) 2024-09-16T03:53:43.858 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.848+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:53:43.858 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.848+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=711764689}) 2024-09-16T03:53:43.858 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.848+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3030390667}) 2024-09-16T03:53:43.858 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.848+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:53:43.858 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.848+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=711764689}) 2024-09-16T03:53:43.859 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.848+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2635019184}) 2024-09-16T03:53:43.859 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.852+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:53:43.860 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.852+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4221907447}) 2024-09-16T03:53:43.860 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.852+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2946668680}) 2024-09-16T03:53:43.860 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.852+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:53:43.860 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.852+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4221907447}) 2024-09-16T03:53:43.860 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.852+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1095326469}) 2024-09-16T03:53:43.861 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T03:53:43.862 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.852+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:53:43.863 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.852+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=395215258}) 2024-09-16T03:53:43.863 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.852+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2007148903}) 2024-09-16T03:53:43.863 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.852+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:53:43.863 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.852+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=395215258}) 2024-09-16T03:53:43.863 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.852+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3290988636}) 2024-09-16T03:53:43.863 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.856+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:53:43.863 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.856+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2208842580}) 2024-09-16T03:53:43.863 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.856+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=818505791}) 2024-09-16T03:53:43.863 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.856+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:53:43.863 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.856+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2208842580}) 2024-09-16T03:53:43.863 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.856+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2554271342}) 2024-09-16T03:53:43.864 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.856+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:53:43.864 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.856+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3931438624}) 2024-09-16T03:53:43.864 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.856+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2861598531}) 2024-09-16T03:53:43.864 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.856+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:53:43.864 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.856+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3931438624}) 2024-09-16T03:53:43.864 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.856+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4011435560}) 2024-09-16T03:53:43.865 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.856+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:53:43.865 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.860+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2309386257}) 2024-09-16T03:53:43.865 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.860+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2746465667}) 2024-09-16T03:53:43.865 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.860+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:53:43.865 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.860+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2309386257}) 2024-09-16T03:53:43.865 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.860+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2058126554}) 2024-09-16T03:53:43.866 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.860+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:53:43.867 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.860+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2275543312}) 2024-09-16T03:53:43.867 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.860+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2627972972}) 2024-09-16T03:53:43.867 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.860+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:53:43.867 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.860+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2275543312}) 2024-09-16T03:53:43.867 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.860+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3332558829}) 2024-09-16T03:53:43.868 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.860+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:53:43.868 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.860+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1511972933}) 2024-09-16T03:53:43.869 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.860+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4214979698}) 2024-09-16T03:53:43.869 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.860+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:53:43.869 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.860+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1511972933}) 2024-09-16T03:53:43.869 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.860+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=280770654}) 2024-09-16T03:53:43.870 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.864+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:53:43.870 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.864+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=116995029}) 2024-09-16T03:53:43.870 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.864+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=284486496}) 2024-09-16T03:53:43.870 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.864+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:53:43.870 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.864+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=116995029}) 2024-09-16T03:53:43.871 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.864+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=724940210}) 2024-09-16T03:53:43.872 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.864+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:53:43.872 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.864+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1381599281}) 2024-09-16T03:53:43.872 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.864+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2172343416}) 2024-09-16T03:53:43.872 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.864+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:53:43.872 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.864+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1381599281}) 2024-09-16T03:53:43.872 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.864+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=499428318}) 2024-09-16T03:53:43.874 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.868+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:53:43.874 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.868+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1328309576}) 2024-09-16T03:53:43.874 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.868+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=122589851}) 2024-09-16T03:53:43.874 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.868+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:53:43.874 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.868+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1328309576}) 2024-09-16T03:53:43.874 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.868+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=537935039}) 2024-09-16T03:53:43.875 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.868+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:53:43.876 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.868+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1719710139}) 2024-09-16T03:53:43.876 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.868+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1147287391}) 2024-09-16T03:53:43.876 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.868+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:53:43.876 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.868+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1719710139}) 2024-09-16T03:53:43.876 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.868+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3974159995}) 2024-09-16T03:53:43.877 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.872+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:53:43.877 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.872+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=296951907}) 2024-09-16T03:53:43.877 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.872+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=604031357}) 2024-09-16T03:53:43.877 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.872+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:53:43.878 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.872+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=296951907}) 2024-09-16T03:53:43.878 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.872+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1732487086}) 2024-09-16T03:53:43.879 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.872+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:53:43.879 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.872+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1672366711}) 2024-09-16T03:53:43.879 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.872+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=875567124}) 2024-09-16T03:53:43.879 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.872+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:53:43.879 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.872+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1672366711}) 2024-09-16T03:53:43.880 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.872+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3247697583}) 2024-09-16T03:53:43.881 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.876+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:53:43.882 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.876+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=589958513}) 2024-09-16T03:53:43.882 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.876+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3018226378}) 2024-09-16T03:53:43.882 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.876+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:53:43.882 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.876+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=589958513}) 2024-09-16T03:53:43.882 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.876+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4094817695}) 2024-09-16T03:53:43.883 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.876+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:53:43.883 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.876+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=840736096}) 2024-09-16T03:53:43.883 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.876+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3640031542}) 2024-09-16T03:53:43.883 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.876+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:53:43.883 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.876+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=840736096}) 2024-09-16T03:53:43.884 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.876+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1182263384}) 2024-09-16T03:53:43.885 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.876+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:53:43.885 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.876+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2928956211}) 2024-09-16T03:53:43.885 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.876+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1724677626}) 2024-09-16T03:53:43.885 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.876+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:53:43.885 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.876+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2928956211}) 2024-09-16T03:53:43.885 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.880+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=578781644}) 2024-09-16T03:53:43.887 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.880+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:53:43.887 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.880+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4027745344}) 2024-09-16T03:53:43.887 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.880+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3059065257}) 2024-09-16T03:53:43.887 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.880+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:53:43.887 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.880+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4027745344}) 2024-09-16T03:53:43.888 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.880+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=981948821}) 2024-09-16T03:53:43.888 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.880+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:53:43.888 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.880+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3528853844}) 2024-09-16T03:53:43.888 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.880+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2713000826}) 2024-09-16T03:53:43.889 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.880+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:53:43.889 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.880+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3528853844}) 2024-09-16T03:53:43.889 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.880+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=214334835}) 2024-09-16T03:53:43.890 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.884+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:53:43.890 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.884+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3283564803}) 2024-09-16T03:53:43.890 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.884+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2300591243}) 2024-09-16T03:53:43.890 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.884+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:53:43.890 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.884+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3283564803}) 2024-09-16T03:53:43.891 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.884+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1857639826}) 2024-09-16T03:53:43.892 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.884+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:53:43.893 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.884+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2319468738}) 2024-09-16T03:53:43.893 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.884+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3216319156}) 2024-09-16T03:53:43.893 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.884+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:53:43.893 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.884+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2319468738}) 2024-09-16T03:53:43.893 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.884+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=139034600}) 2024-09-16T03:53:43.894 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.888+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:53:43.894 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.888+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4175195743}) 2024-09-16T03:53:43.894 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.888+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2046114129}) 2024-09-16T03:53:43.894 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.888+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:53:43.895 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.888+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4175195743}) 2024-09-16T03:53:43.895 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.888+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2327507489}) 2024-09-16T03:53:43.896 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.888+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:53:43.896 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.888+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3278881926}) 2024-09-16T03:53:43.896 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.888+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1440105974}) 2024-09-16T03:53:43.896 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.888+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:53:43.896 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.888+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3278881926}) 2024-09-16T03:53:43.896 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.888+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=768183478}) 2024-09-16T03:53:43.897 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.888+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:53:43.897 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.888+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=21929910}) 2024-09-16T03:53:43.898 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.888+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3009496636}) 2024-09-16T03:53:43.898 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.888+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:53:43.898 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.888+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=21929910}) 2024-09-16T03:53:43.898 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.888+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3304027505}) 2024-09-16T03:53:43.899 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.892+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:53:43.899 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.892+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3354308873}) 2024-09-16T03:53:43.899 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.892+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2414085330}) 2024-09-16T03:53:43.899 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.892+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:53:43.899 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.892+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3354308873}) 2024-09-16T03:53:43.899 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.892+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2496115765}) 2024-09-16T03:53:43.900 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.892+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:53:43.900 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.892+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1968961447}) 2024-09-16T03:53:43.900 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.892+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4125365381}) 2024-09-16T03:53:43.901 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.892+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:53:43.901 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.892+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1968961447}) 2024-09-16T03:53:43.901 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.892+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2402089651}) 2024-09-16T03:53:43.902 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.896+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:53:43.902 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.896+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2246488391}) 2024-09-16T03:53:43.902 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.896+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2044996413}) 2024-09-16T03:53:43.902 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.896+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:53:43.903 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.896+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2246488391}) 2024-09-16T03:53:43.903 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.896+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3218086098}) 2024-09-16T03:53:43.904 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.896+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:53:43.904 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.896+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=61725892}) 2024-09-16T03:53:43.904 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.896+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3995643235}) 2024-09-16T03:53:43.904 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.896+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:53:43.904 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.896+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=61725892}) 2024-09-16T03:53:43.905 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.896+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3121493173}) 2024-09-16T03:53:43.906 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.900+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:53:43.906 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.900+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4178600622}) 2024-09-16T03:53:43.906 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.900+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3546036660}) 2024-09-16T03:53:43.906 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.900+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:53:43.906 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.900+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4178600622}) 2024-09-16T03:53:43.906 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.900+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1419400926}) 2024-09-16T03:53:43.907 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.900+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:53:43.907 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.900+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=963382070}) 2024-09-16T03:53:43.907 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.900+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=253649689}) 2024-09-16T03:53:43.908 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.900+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:53:43.908 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.900+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=963382070}) 2024-09-16T03:53:43.908 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.900+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3616946965}) 2024-09-16T03:53:43.909 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.904+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:53:43.909 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.904+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4195601803}) 2024-09-16T03:53:43.909 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.904+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3872280325}) 2024-09-16T03:53:43.909 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.904+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:53:43.909 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.904+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4195601803}) 2024-09-16T03:53:43.909 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.904+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3939865728}) 2024-09-16T03:53:43.911 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.904+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:53:43.911 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.904+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2709991237}) 2024-09-16T03:53:43.911 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.904+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3772580678}) 2024-09-16T03:53:43.911 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.904+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:53:43.911 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.904+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2709991237}) 2024-09-16T03:53:43.911 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.904+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3207380754}) 2024-09-16T03:53:43.912 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.904+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:53:43.912 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.904+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=924518272}) 2024-09-16T03:53:43.913 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.904+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1235030285}) 2024-09-16T03:53:43.913 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.904+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:53:43.913 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.904+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=924518272}) 2024-09-16T03:53:43.913 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.904+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1909147198}) 2024-09-16T03:53:43.914 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.908+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:53:43.914 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.908+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1914083905}) 2024-09-16T03:53:43.915 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.908+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1828779418}) 2024-09-16T03:53:43.915 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.908+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:53:43.915 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.908+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1914083905}) 2024-09-16T03:53:43.915 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.908+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1833671201}) 2024-09-16T03:53:43.917 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.908+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:53:43.917 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.908+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1836990520}) 2024-09-16T03:53:43.917 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.908+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2116820087}) 2024-09-16T03:53:43.917 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.908+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:53:43.917 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.908+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1836990520}) 2024-09-16T03:53:43.918 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.908+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3149572343}) 2024-09-16T03:53:43.918 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.912+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:53:43.918 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.912+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=432953803}) 2024-09-16T03:53:43.918 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.912+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2054095146}) 2024-09-16T03:53:43.918 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.912+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:53:43.919 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.912+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=432953803}) 2024-09-16T03:53:43.919 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.912+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2598805970}) 2024-09-16T03:53:43.920 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.912+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:53:43.920 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.912+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=275366175}) 2024-09-16T03:53:43.920 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.912+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2048805441}) 2024-09-16T03:53:43.920 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.912+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:53:43.920 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.912+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=275366175}) 2024-09-16T03:53:43.920 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.912+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1782621688}) 2024-09-16T03:53:43.922 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.916+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:53:43.922 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.916+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1285894757}) 2024-09-16T03:53:43.922 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.916+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1013402618}) 2024-09-16T03:53:43.922 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.916+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:53:43.922 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.916+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1285894757}) 2024-09-16T03:53:43.922 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.916+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2582471248}) 2024-09-16T03:53:43.924 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.916+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:53:43.924 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.916+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4104471651}) 2024-09-16T03:53:43.924 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.916+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3273470855}) 2024-09-16T03:53:43.924 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.916+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:53:43.924 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.916+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4104471651}) 2024-09-16T03:53:43.924 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.916+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3747142117}) 2024-09-16T03:53:43.926 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.920+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:53:43.926 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.920+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=352781446}) 2024-09-16T03:53:43.926 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.920+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3727095435}) 2024-09-16T03:53:43.926 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.920+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:53:43.926 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.920+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=352781446}) 2024-09-16T03:53:43.926 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.920+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1946549465}) 2024-09-16T03:53:43.928 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.920+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:53:43.928 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.920+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=511786366}) 2024-09-16T03:53:43.928 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.920+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3363881902}) 2024-09-16T03:53:43.928 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.920+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:53:43.928 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.920+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=511786366}) 2024-09-16T03:53:43.928 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.920+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=371597938}) 2024-09-16T03:53:43.929 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.924+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:53:43.929 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.924+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=676498790}) 2024-09-16T03:53:43.929 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.924+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=746953693}) 2024-09-16T03:53:43.929 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.924+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:53:43.929 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.924+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=676498790}) 2024-09-16T03:53:43.929 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.924+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3408717894}) 2024-09-16T03:53:43.931 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.924+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:53:43.931 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.924+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1798990613}) 2024-09-16T03:53:43.931 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.924+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3243120536}) 2024-09-16T03:53:43.931 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.924+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:53:43.932 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.924+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1798990613}) 2024-09-16T03:53:43.932 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.924+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3295042039}) 2024-09-16T03:53:43.933 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.924+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:53:43.933 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.924+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3654361053}) 2024-09-16T03:53:43.933 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.924+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3110822094}) 2024-09-16T03:53:43.933 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.924+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:53:43.933 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.924+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3654361053}) 2024-09-16T03:53:43.933 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.924+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3404619054}) 2024-09-16T03:53:43.936 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.928+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:53:43.936 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.928+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2486562591}) 2024-09-16T03:53:43.936 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.928+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1866874231}) 2024-09-16T03:53:43.936 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.928+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:53:43.936 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.928+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2486562591}) 2024-09-16T03:53:43.936 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.928+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=904906945}) 2024-09-16T03:53:43.937 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.928+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:53:43.937 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.928+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2670721430}) 2024-09-16T03:53:43.937 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.928+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=661379357}) 2024-09-16T03:53:43.937 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.928+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:53:43.937 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.928+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2670721430}) 2024-09-16T03:53:43.937 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.928+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3212492046}) 2024-09-16T03:53:43.938 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.932+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:53:43.939 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.932+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=98700271}) 2024-09-16T03:53:43.939 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.932+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=970035602}) 2024-09-16T03:53:43.939 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.932+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:53:43.939 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.932+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=98700271}) 2024-09-16T03:53:43.939 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.932+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=385441160}) 2024-09-16T03:53:43.940 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.932+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:53:43.940 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.932+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3999164395}) 2024-09-16T03:53:43.941 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.932+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=651276796}) 2024-09-16T03:53:43.941 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.932+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:53:43.941 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.932+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3999164395}) 2024-09-16T03:53:43.941 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.932+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2912911363}) 2024-09-16T03:53:43.942 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.936+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:53:43.942 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.936+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3790412485}) 2024-09-16T03:53:43.942 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.936+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1754854534}) 2024-09-16T03:53:43.942 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.936+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:53:43.942 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.936+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3790412485}) 2024-09-16T03:53:43.942 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.936+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=680474841}) 2024-09-16T03:53:43.945 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.936+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:53:43.945 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.936+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=417564981}) 2024-09-16T03:53:43.945 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.936+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=725824138}) 2024-09-16T03:53:43.945 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.936+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:53:43.945 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.936+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=417564981}) 2024-09-16T03:53:43.945 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.936+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3251313785}) 2024-09-16T03:53:43.946 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.940+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:53:43.946 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.940+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1856261866}) 2024-09-16T03:53:43.946 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.940+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2686655830}) 2024-09-16T03:53:43.946 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.940+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:53:43.946 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.940+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1856261866}) 2024-09-16T03:53:43.946 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.940+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=878884444}) 2024-09-16T03:53:43.948 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.940+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:53:43.948 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.940+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3744783920}) 2024-09-16T03:53:43.948 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.940+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2583462653}) 2024-09-16T03:53:43.948 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.940+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:53:43.948 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.940+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3744783920}) 2024-09-16T03:53:43.948 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.940+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=58232632}) 2024-09-16T03:53:43.949 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.944+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:53:43.949 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.944+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2963123463}) 2024-09-16T03:53:43.949 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.944+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3868610455}) 2024-09-16T03:53:43.949 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.944+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:53:43.949 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.944+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2963123463}) 2024-09-16T03:53:43.950 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.944+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3132613021}) 2024-09-16T03:53:43.951 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.944+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:53:43.951 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.944+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2878460914}) 2024-09-16T03:53:43.951 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.944+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3520467922}) 2024-09-16T03:53:43.951 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.944+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:53:43.951 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.944+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2878460914}) 2024-09-16T03:53:43.951 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.944+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1092461844}) 2024-09-16T03:53:43.953 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.944+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:53:43.953 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.944+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3667840105}) 2024-09-16T03:53:43.953 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.944+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2260298592}) 2024-09-16T03:53:43.953 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.944+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:53:43.953 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.944+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3667840105}) 2024-09-16T03:53:43.953 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.944+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2327310012}) 2024-09-16T03:53:43.954 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.948+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:53:43.955 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.948+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2411463687}) 2024-09-16T03:53:43.955 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.948+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=670599259}) 2024-09-16T03:53:43.955 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.948+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:53:43.955 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.948+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2411463687}) 2024-09-16T03:53:43.955 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.948+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2929914348}) 2024-09-16T03:53:43.957 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.952+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:53:43.957 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.952+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=33,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=23} crc {logm=3790477782,mdsmap=4088524775,mgr=460109776,mgr_command_descs=2107673600,mgr_metadata=1367272202,mgrstat=395533111,mon_config_key=85057112,monmap=3937513981,osd_metadata=4182959540,osd_pg_creating=2871298393,osd_snap=469974799}) 2024-09-16T03:53:43.957 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.952+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=32,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=23} crc {logm=3561626187,mdsmap=4088524775,mgr=460109776,mgr_command_descs=2107673600,mgr_metadata=1367272202,mgrstat=745823495,mon_config_key=85057112,monmap=3937513981,osd_metadata=4182959540,osd_pg_creating=2871298393,osd_snap=4197385620}) 2024-09-16T03:53:43.957 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.952+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:53:43.958 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.952+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=33,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=23} crc {logm=3790477782,mdsmap=4088524775,mgr=460109776,mgr_command_descs=2107673600,mgr_metadata=1367272202,mgrstat=395533111,mon_config_key=85057112,monmap=3937513981,osd_metadata=4182959540,osd_pg_creating=2871298393,osd_snap=469974799}) 2024-09-16T03:53:43.958 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.952+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=30,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=25} crc {logm=4161462996,mdsmap=4088524775,mgr=460109776,mgr_command_descs=2107673600,mgr_metadata=1367272202,mgrstat=745823495,mon_config_key=85057112,monmap=3937513981,osd_metadata=4182959540,osd_pg_creating=2871298393,osd_snap=3772797981}) 2024-09-16T03:53:43.959 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.952+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:53:43.959 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.952+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=10,osdmap=90} crc {osd_snap=3872841081,osdmap=1271877580}) 2024-09-16T03:53:43.959 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.952+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=12,osdmap=88} crc {osd_snap=1594004101,osdmap=2853727514}) 2024-09-16T03:53:43.960 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.952+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:53:43.960 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.952+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=10,osdmap=90} crc {osd_snap=3872841081,osdmap=1271877580}) 2024-09-16T03:53:43.960 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.952+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=11,osdmap=89} crc {osd_snap=1064299228,osdmap=3524834779}) 2024-09-16T03:53:43.972 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.964+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:53:43.972 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.964+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=44} crc {osdmap=1678583552}) 2024-09-16T03:53:43.972 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.964+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=42} crc {osdmap=3236955249}) 2024-09-16T03:53:43.972 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.964+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:53:43.973 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.964+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=44} crc {osdmap=1678583552}) 2024-09-16T03:53:43.973 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:53:43.964+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=46} crc {osdmap=982347763}) 2024-09-16T03:53:44.862 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T03:53:44.863 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:53:44.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.984+0000 7f26ada6a640 10 monclient: get_monmap_and_config 2024-09-16T03:53:44.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.984+0000 7f26ada6a640 10 monclient: build_initial_monmap 2024-09-16T03:53:44.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.984+0000 7f26ada6a640 10 monclient: monmap: 2024-09-16T03:53:44.992 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:53:44.992 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:53:44.992 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:53:44.990321+0000 2024-09-16T03:53:44.992 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:53:44.990321+0000 2024-09-16T03:53:44.992 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:53:44.992 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:53:44.992 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:53:44.992 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:53:44.992 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:53:44.992 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:53:44.992 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:53:44.992 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:53:44.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.984+0000 7f26ada6a640 1 Processor -- start 2024-09-16T03:53:44.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.984+0000 7f26ada6a640 1 -- start start 2024-09-16T03:53:44.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.984+0000 7f26ada6a640 10 monclient: init 2024-09-16T03:53:44.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.984+0000 7f26ada6a640 10 monclient: _reopen_session rank -1 2024-09-16T03:53:44.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.984+0000 7f26ada6a640 10 monclient: _add_conns ranks=[0,2,1,4,3] 2024-09-16T03:53:44.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.984+0000 7f26ada6a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f26a80a1330 addr v1:172.21.15.42:6789/0 2024-09-16T03:53:44.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.984+0000 7f26ada6a640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f26a80a1710 0x7f26a80a1b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:53:44.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.984+0000 7f26ada6a640 10 monclient(hunting): picked mon.noname-b con 0x7f26a80a1710 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:53:44.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.984+0000 7f26ada6a640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f26a81393a0 addr v1:172.21.15.154:6789/0 2024-09-16T03:53:44.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.984+0000 7f26ada6a640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f26a809e4c0 con 0x7f26a80a1330 2024-09-16T03:53:44.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.984+0000 7f26ada6a640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f26a80a2050 con 0x7f26a81393a0 2024-09-16T03:53:44.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.984+0000 7f26ada6a640 10 monclient(hunting): start opening mon connection 2024-09-16T03:53:44.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.984+0000 7f26ada6a640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f26a809edf0 con 0x7f26a80a1710 2024-09-16T03:53:44.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.984+0000 7f26ada6a640 10 monclient(hunting): _renew_subs 2024-09-16T03:53:44.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.984+0000 7f26ada6a640 10 monclient(hunting): authenticate will time out at 1671.959981s 2024-09-16T03:53:44.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.988+0000 7f26a77fe640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f26a80a1330 0x7f26a809f180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:40530/0 (socket says 172.21.15.42:40530) 2024-09-16T03:53:44.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.988+0000 7f26a77fe640 1 -- 172.21.15.42:0/2335517613 learned_addr learned my addr 172.21.15.42:0/2335517613 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:53:44.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.988+0000 7f26a6ffd640 1 --2- 172.21.15.42:0/2335517613 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f26a80a1710 0x7f26a80a1b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:53:44.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.988+0000 7f26a6ffd640 10 monclient(hunting): get_auth_request con 0x7f26a80a1710 auth_method 0 2024-09-16T03:53:44.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.988+0000 7f26a6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:53:44.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.988+0000 7f26a6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:53:44.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.988+0000 7f26a6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:53:44.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.988+0000 7f26a6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:53:44.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.988+0000 7f26a6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:53:44.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.988+0000 7f26a6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:53:44.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.988+0000 7f26a67fc640 1 -- 172.21.15.42:0/2335517613 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4262606048 0 0) 0x7f26a809e4c0 con 0x7f26a80a1330 2024-09-16T03:53:44.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.988+0000 7f26a67fc640 10 monclient(hunting): _negotiate 2024-09-16T03:53:44.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.988+0000 7f26a67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:53:44.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.988+0000 7f26a67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:53:44.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.988+0000 7f26a67fc640 10 monclient(hunting): my global_id is 824370 2024-09-16T03:53:44.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.988+0000 7f26a67fc640 1 -- 172.21.15.42:0/2335517613 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2694005740 con 0x7f26a80a1330 2024-09-16T03:53:44.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.988+0000 7f26a6ffd640 10 monclient(hunting): handle_auth_done global_id 834227 payload 274 2024-09-16T03:53:44.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.988+0000 7f26a6ffd640 1 -- 172.21.15.42:0/2335517613 >> v1:172.21.15.154:6789/0 conn(0x7f26a81393a0 legacy=0x7f26a813b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:53:44.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.988+0000 7f26a6ffd640 1 -- 172.21.15.42:0/2335517613 >> v1:172.21.15.42:6789/0 conn(0x7f26a80a1330 legacy=0x7f26a809f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:53:44.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.988+0000 7f26a6ffd640 10 monclient: _finish_hunting 0 2024-09-16T03:53:44.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.988+0000 7f26a6ffd640 1 monclient: found mon.noname-b 2024-09-16T03:53:44.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.988+0000 7f26a6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:53:44.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.988+0000 7f26a6ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T03:53:44.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.988+0000 7f26a6ffd640 1 -- 172.21.15.42:0/2335517613 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26a80a3170 con 0x7f26a80a1710 2024-09-16T03:53:45.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.988+0000 7f26a6ffd640 1 --2- 172.21.15.42:0/2335517613 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f26a80a1710 0x7f26a80a1b10 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f26900097f0 tx=0x7f269001c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5375c7a6e537877b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:53:45.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.988+0000 7f26a67fc640 1 -- 172.21.15.42:0/2335517613 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26900b0070 con 0x7f26a80a1710 2024-09-16T03:53:45.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.988+0000 7f26a67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:53:45.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.988+0000 7f26a67fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:53:45.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.988+0000 7f26a67fc640 10 monclient: dump: 2024-09-16T03:53:45.004 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:53:45.004 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:53:45.004 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:45.005 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:45.005 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:53:45.005 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:53:45.005 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:53:45.005 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:53:45.005 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:53:45.005 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:53:45.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.988+0000 7f26a67fc640 10 monclient: _finish_auth 0 2024-09-16T03:53:45.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.988+0000 7f26a67fc640 10 monclient: _check_auth_tickets 2024-09-16T03:53:45.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.988+0000 7f26a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:53:45.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.988+0000 7f26a67fc640 1 -- 172.21.15.42:0/2335517613 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f26900ab600 con 0x7f26a80a1710 2024-09-16T03:53:45.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.988+0000 7f26a67fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:53:45.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.988+0000 7f26a67fc640 1 -- 172.21.15.42:0/2335517613 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26900ab9d0 con 0x7f26a80a1710 2024-09-16T03:53:45.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.988+0000 7f26ada6a640 5 monclient: authenticate success, global_id 834227 2024-09-16T03:53:45.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.988+0000 7f26ada6a640 10 monclient: get_monmap_and_config success 2024-09-16T03:53:45.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.988+0000 7f26ada6a640 4 set_mon_vals no callback set 2024-09-16T03:53:45.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.988+0000 7f26a67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:53:45.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.988+0000 7f26a67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:53:45.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.988+0000 7f26a67fc640 10 monclient: dump: 2024-09-16T03:53:45.006 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:53:45.006 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:53:45.006 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:45.007 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:45.007 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:53:45.007 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:53:45.007 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:53:45.007 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:53:45.007 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:53:45.007 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:53:45.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.988+0000 7f26ada6a640 10 monclient: shutdown 2024-09-16T03:53:45.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.988+0000 7f26ada6a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:53:45.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.988+0000 7f26ada6a640 1 -- 172.21.15.42:0/2335517613 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f26a80a1710 msgr2=0x7f26a80a1b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:53:45.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.988+0000 7f26ada6a640 1 --2- 172.21.15.42:0/2335517613 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f26a80a1710 0x7f26a80a1b10 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f26900097f0 tx=0x7f269001c3a0 comp rx=0 tx=0).stop 2024-09-16T03:53:45.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.988+0000 7f26ada6a640 1 -- 172.21.15.42:0/2335517613 shutdown_connections 2024-09-16T03:53:45.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.988+0000 7f26ada6a640 1 --2- 172.21.15.42:0/2335517613 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f26a80a1710 0x7f26a80a1b10 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:53:45.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.988+0000 7f26ada6a640 1 -- 172.21.15.42:0/2335517613 >> 172.21.15.42:0/2335517613 conn(0x7f26a80a2420 msgr2=0x7f26a80a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:53:45.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.988+0000 7f26ada6a640 1 -- 172.21.15.42:0/2335517613 shutdown_connections 2024-09-16T03:53:45.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.988+0000 7f26ada6a640 1 -- 172.21.15.42:0/2335517613 wait complete. 2024-09-16T03:53:45.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.992+0000 7f26ada6a640 10 monclient: build_initial_monmap 2024-09-16T03:53:45.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.992+0000 7f26ada6a640 10 monclient: monmap: 2024-09-16T03:53:45.008 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:53:45.008 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:53:45.008 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:53:45.009 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:53:45.009 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:53:45.009 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:53:45.009 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:53:45.009 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:53:45.009 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:53:45.009 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:53:45.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.992+0000 7f26ada6a640 1 librados: starting msgr at 2024-09-16T03:53:45.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.992+0000 7f26ada6a640 1 librados: starting objecter 2024-09-16T03:53:45.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.992+0000 7f26ada6a640 1 Processor -- start 2024-09-16T03:53:45.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.992+0000 7f26ada6a640 1 -- start start 2024-09-16T03:53:45.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.992+0000 7f26ada6a640 1 librados: setting wanted keys 2024-09-16T03:53:45.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.992+0000 7f26ada6a640 1 librados: calling monclient init 2024-09-16T03:53:45.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.992+0000 7f26ada6a640 10 monclient: init 2024-09-16T03:53:45.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.992+0000 7f26ada6a640 10 monclient: _reopen_session rank -1 2024-09-16T03:53:45.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.992+0000 7f26ada6a640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:53:45.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.992+0000 7f26ada6a640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f26a80a1330 0x7f26a81cf070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:53:45.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.992+0000 7f26ada6a640 10 monclient(hunting): picked mon.noname-a con 0x7f26a80a1330 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:53:45.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.992+0000 7f26ada6a640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f26a80a1710 0x7f26a8134560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:53:45.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.992+0000 7f26ada6a640 10 monclient(hunting): picked mon.noname-c con 0x7f26a80a1710 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:53:45.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.992+0000 7f26ada6a640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f26a81393a0 0x7f26a811dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:53:45.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.992+0000 7f26ada6a640 10 monclient(hunting): picked mon.noname-b con 0x7f26a81393a0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:53:45.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.992+0000 7f26ada6a640 10 monclient(hunting): start opening mon connection 2024-09-16T03:53:45.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.992+0000 7f26ada6a640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f26a809c5f0 con 0x7f26a80a1330 2024-09-16T03:53:45.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.992+0000 7f26ada6a640 10 monclient(hunting): start opening mon connection 2024-09-16T03:53:45.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.992+0000 7f26ada6a640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f26a809c470 con 0x7f26a81393a0 2024-09-16T03:53:45.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.992+0000 7f26ada6a640 10 monclient(hunting): start opening mon connection 2024-09-16T03:53:45.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.992+0000 7f26ada6a640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f26a809c770 con 0x7f26a80a1710 2024-09-16T03:53:45.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.992+0000 7f26ada6a640 10 monclient(hunting): _renew_subs 2024-09-16T03:53:45.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.992+0000 7f26ada6a640 10 monclient(hunting): authenticate will time out at 1671.965026s 2024-09-16T03:53:45.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.992+0000 7f26a6ffd640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f26a80a1710 0x7f26a8134560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:53:45.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.992+0000 7f26a6ffd640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f26a80a1710 0x7f26a8134560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:34632/0 (socket says 172.21.15.42:34632) 2024-09-16T03:53:45.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.992+0000 7f26a6ffd640 1 -- 172.21.15.42:0/2520722036 learned_addr learned my addr 172.21.15.42:0/2520722036 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:53:45.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.992+0000 7f26a6ffd640 10 monclient(hunting): get_auth_request con 0x7f26a80a1710 auth_method 0 2024-09-16T03:53:45.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.992+0000 7f26a6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:53:45.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.992+0000 7f26a6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:53:45.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.992+0000 7f26a6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:53:45.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.992+0000 7f26a77fe640 1 --2- 172.21.15.42:0/2520722036 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f26a80a1330 0x7f26a81cf070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:53:45.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.992+0000 7f26a77fe640 10 monclient(hunting): get_auth_request con 0x7f26a80a1330 auth_method 0 2024-09-16T03:53:45.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.992+0000 7f26a77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:53:45.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.992+0000 7f26a77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:53:45.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.992+0000 7f26a77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:53:45.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.992+0000 7f26a6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:53:45.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.992+0000 7f26a6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:53:45.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.992+0000 7f26a6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:53:45.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.992+0000 7f26a77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:53:45.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.992+0000 7f26a77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:53:45.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.992+0000 7f26a77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:53:45.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.992+0000 7f26a7fff640 1 --2- 172.21.15.42:0/2520722036 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f26a81393a0 0x7f26a811dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:53:45.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.992+0000 7f26a6ffd640 10 monclient(hunting): handle_auth_done global_id 834230 payload 824 2024-09-16T03:53:45.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.992+0000 7f26a6ffd640 1 -- 172.21.15.42:0/2520722036 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f26a81393a0 msgr2=0x7f26a811dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:53:45.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.992+0000 7f26a6ffd640 1 --2- 172.21.15.42:0/2520722036 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f26a81393a0 0x7f26a811dfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:53:45.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.992+0000 7f26a6ffd640 1 -- 172.21.15.42:0/2520722036 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f26a80a1330 msgr2=0x7f26a81cf070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:53:45.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.992+0000 7f26a6ffd640 1 --2- 172.21.15.42:0/2520722036 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f26a80a1330 0x7f26a81cf070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:53:45.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.992+0000 7f26a6ffd640 10 monclient: _finish_hunting 0 2024-09-16T03:53:45.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.992+0000 7f26a6ffd640 1 monclient: found mon.noname-c 2024-09-16T03:53:45.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.992+0000 7f26a6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:53:45.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.992+0000 7f26a6ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:53:45.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.992+0000 7f26a6ffd640 1 -- 172.21.15.42:0/2520722036 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26a81cff80 con 0x7f26a80a1710 2024-09-16T03:53:45.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.992+0000 7f26a7fff640 10 monclient: get_auth_request con 0x7f26a81393a0 auth_method 0 2024-09-16T03:53:45.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.992+0000 7f26a7fff640 1 --2- 172.21.15.42:0/2520722036 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f26a81393a0 0x7f26a811dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T03:53:45.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.992+0000 7f26a6ffd640 1 --2- 172.21.15.42:0/2520722036 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f26a80a1710 0x7f26a8134560 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f26900097f0 tx=0x7f269001abe0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:53:45.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.992+0000 7f269ffff640 10 client.?.objecter ms_handle_connect 0x7f26a80a1710 2024-09-16T03:53:45.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.992+0000 7f269ffff640 10 client.?.objecter resend_mon_ops 2024-09-16T03:53:45.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.992+0000 7f269ffff640 1 -- 172.21.15.42:0/2520722036 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26900b0050 con 0x7f26a80a1710 2024-09-16T03:53:45.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.992+0000 7f269ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:53:45.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.992+0000 7f269ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:53:45.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.992+0000 7f269ffff640 10 monclient: dump: 2024-09-16T03:53:45.014 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:53:45.014 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:53:45.014 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:45.014 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:45.014 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:53:45.014 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:53:45.014 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:53:45.015 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:53:45.015 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:53:45.015 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:53:45.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.992+0000 7f269ffff640 10 monclient: _finish_auth 0 2024-09-16T03:53:45.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.992+0000 7f269ffff640 10 monclient: _check_auth_tickets 2024-09-16T03:53:45.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.992+0000 7f269ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:53:45.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.992+0000 7f269ffff640 1 -- 172.21.15.42:0/2520722036 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f26900bc070 con 0x7f26a80a1710 2024-09-16T03:53:45.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.992+0000 7f26ada6a640 5 monclient: authenticate success, global_id 834230 2024-09-16T03:53:45.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.992+0000 7f26ada6a640 10 monclient: _renew_subs 2024-09-16T03:53:45.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.992+0000 7f26ada6a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:53:45.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.992+0000 7f26ada6a640 1 -- 172.21.15.42:0/2520722036 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26a8121f20 con 0x7f26a80a1710 2024-09-16T03:53:45.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.992+0000 7f269ffff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:53:45.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.992+0000 7f269ffff640 1 -- 172.21.15.42:0/2520722036 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26900b5a70 con 0x7f26a80a1710 2024-09-16T03:53:45.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.992+0000 7f269ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:53:45.016 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.992+0000 7f269ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:53:45.016 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.992+0000 7f269ffff640 10 monclient: dump: 2024-09-16T03:53:45.016 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:53:45.016 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:53:45.016 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:45.016 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:45.016 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:53:45.016 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:53:45.016 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:53:45.017 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:53:45.017 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:53:45.017 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:53:45.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.992+0000 7f26ada6a640 10 client.834230.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:53:45.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.992+0000 7f26ada6a640 10 monclient: _renew_subs 2024-09-16T03:53:45.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.992+0000 7f26a5ffb640 4 set_mon_vals no callback set 2024-09-16T03:53:45.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.992+0000 7f26ada6a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:53:45.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.992+0000 7f26ada6a640 1 -- 172.21.15.42:0/2520722036 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f26a80a0ee0 con 0x7f26a80a1710 2024-09-16T03:53:45.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.992+0000 7f26ada6a640 1 librados: init done 2024-09-16T03:53:45.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.992+0000 7f26ada6a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:53:45.018 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.992+0000 7f26ada6a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:53:45.018 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.992+0000 7f26ada6a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:53:45.018 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.992+0000 7f26ada6a640 1 -- 172.21.15.42:0/2520722036 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f26a80a1b10 con 0x7f26a80a1710 2024-09-16T03:53:45.018 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.992+0000 7f269ffff640 1 -- 172.21.15.42:0/2520722036 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f26900b5c10 con 0x7f26a80a1710 2024-09-16T03:53:45.018 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.992+0000 7f269ffff640 1 --2- 172.21.15.42:0/2520722036 >> v2:172.21.15.154:6812/254845342 conn(0x7f2688037810 0x7f2688039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:53:45.018 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.992+0000 7f269ffff640 1 -- 172.21.15.42:0/2520722036 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(366..366 src has 286..366) ==== 5713+0+0 (secure 0 0 0) 0x7f26900e3700 con 0x7f26a80a1710 2024-09-16T03:53:45.018 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.992+0000 7f269ffff640 10 client.834230.objecter ms_dispatch 0x7f26a8000ba0 osd_map(366..366 src has 286..366) 2024-09-16T03:53:45.018 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.992+0000 7f269ffff640 3 client.834230.objecter handle_osd_map got epochs [366,366] > 0 2024-09-16T03:53:45.018 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.992+0000 7f269ffff640 3 client.834230.objecter handle_osd_map decoding full epoch 366 2024-09-16T03:53:45.018 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.992+0000 7f269ffff640 20 client.834230.objecter dump_active .. 0 homeless 2024-09-16T03:53:45.019 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.992+0000 7f26a77fe640 1 --2- 172.21.15.42:0/2520722036 >> v2:172.21.15.154:6812/254845342 conn(0x7f2688037810 0x7f2688039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:53:45.019 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.992+0000 7f26a77fe640 10 monclient: get_auth_request con 0x7f2688037810 auth_method 0 2024-09-16T03:53:45.019 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.996+0000 7f269ffff640 10 client.834230.objecter ms_handle_connect 0x7f2688037810 2024-09-16T03:53:45.019 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:44.996+0000 7f26a77fe640 1 --2- 172.21.15.42:0/2520722036 >> v2:172.21.15.154:6812/254845342 conn(0x7f2688037810 0x7f2688039cd0 secure :-1 s=READY pgs=3604 cs=0 l=1 rev1=1 crypto rx=0x7f26980011f0 tx=0x7f2698007d40 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:53:45.019 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.000+0000 7f269ffff640 1 -- 172.21.15.42:0/2520722036 <== mon.2 v2:172.21.15.42:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f26a81cff80 con 0x7f26a80a1710 2024-09-16T03:53:45.019 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.000+0000 7f269ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:53:45.020 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.000+0000 7f269ffff640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:53:45.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.220+0000 7f26ada6a640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:53:45.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.220+0000 7f26ada6a640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:53:45.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.220+0000 7f26ada6a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:53:45.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.220+0000 7f26ada6a640 1 -- 172.21.15.42:0/2520722036 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f26a813cc90 con 0x7f26a80a1710 2024-09-16T03:53:45.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.220+0000 7f269ffff640 1 -- 172.21.15.42:0/2520722036 <== mon.2 v2:172.21.15.42:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7f26900160b0 con 0x7f26a80a1710 2024-09-16T03:53:45.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.220+0000 7f269ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:53:45.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.220+0000 7f269ffff640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:53:45.229 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":100,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:53:45.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.224+0000 7f26ada6a640 10 librados: watch_flush enter 2024-09-16T03:53:45.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.224+0000 7f26ada6a640 10 librados: watch_flush exit 2024-09-16T03:53:45.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.224+0000 7f26ada6a640 20 client.834230.objecter shutdown clearing up homeless session... 2024-09-16T03:53:45.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.224+0000 7f26ada6a640 10 client.834230.objecter successfully canceled tick 2024-09-16T03:53:45.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.224+0000 7f26ada6a640 1 -- 172.21.15.42:0/2520722036 >> v2:172.21.15.154:6812/254845342 conn(0x7f2688037810 msgr2=0x7f2688039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:53:45.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.224+0000 7f26ada6a640 1 --2- 172.21.15.42:0/2520722036 >> v2:172.21.15.154:6812/254845342 conn(0x7f2688037810 0x7f2688039cd0 secure :-1 s=READY pgs=3604 cs=0 l=1 rev1=1 crypto rx=0x7f26980011f0 tx=0x7f2698007d40 comp rx=0 tx=0).stop 2024-09-16T03:53:45.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.224+0000 7f26ada6a640 10 monclient: shutdown 2024-09-16T03:53:45.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.224+0000 7f26ada6a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:53:45.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.224+0000 7f26ada6a640 1 -- 172.21.15.42:0/2520722036 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f26a80a1710 msgr2=0x7f26a8134560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:53:45.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.224+0000 7f26ada6a640 1 --2- 172.21.15.42:0/2520722036 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f26a80a1710 0x7f26a8134560 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f26900097f0 tx=0x7f269001abe0 comp rx=0 tx=0).stop 2024-09-16T03:53:45.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.224+0000 7f26ada6a640 1 -- 172.21.15.42:0/2520722036 shutdown_connections 2024-09-16T03:53:45.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.224+0000 7f26ada6a640 1 --2- 172.21.15.42:0/2520722036 >> v2:172.21.15.154:6812/254845342 conn(0x7f2688037810 0x7f2688039cd0 unknown :-1 s=CLOSED pgs=3604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:53:45.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.224+0000 7f26ada6a640 1 --2- 172.21.15.42:0/2520722036 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f26a81393a0 0x7f26a811dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:53:45.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.224+0000 7f26ada6a640 1 --2- 172.21.15.42:0/2520722036 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f26a80a1710 0x7f26a8134560 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:53:45.233 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.224+0000 7f26ada6a640 1 --2- 172.21.15.42:0/2520722036 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f26a80a1330 0x7f26a81cf070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:53:45.233 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.224+0000 7f26ada6a640 1 -- 172.21.15.42:0/2520722036 >> 172.21.15.42:0/2520722036 conn(0x7f26a80a2420 msgr2=0x7f26a80a4fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:53:45.233 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.224+0000 7f26ada6a640 1 -- 172.21.15.42:0/2520722036 shutdown_connections 2024-09-16T03:53:45.233 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.224+0000 7f26ada6a640 1 -- 172.21.15.42:0/2520722036 wait complete. 2024-09-16T03:53:45.233 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.224+0000 7f26ada6a640 1 librados: shutdown 2024-09-16T03:53:45.248 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T03:53:45.248 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T03:53:45.248 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T03:53:45.248 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T03:53:45.368 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.360+0000 7f145aa3e640 10 monclient: get_monmap_and_config 2024-09-16T03:53:45.368 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.360+0000 7f145aa3e640 10 monclient: build_initial_monmap 2024-09-16T03:53:45.368 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.360+0000 7f145aa3e640 10 monclient: monmap: 2024-09-16T03:53:45.368 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:53:45.368 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:53:45.369 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:53:45.367102+0000 2024-09-16T03:53:45.369 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:53:45.367102+0000 2024-09-16T03:53:45.369 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:53:45.369 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:53:45.369 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:53:45.369 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:53:45.369 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:53:45.369 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:53:45.369 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:53:45.369 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:53:45.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.360+0000 7f145aa3e640 1 Processor -- start 2024-09-16T03:53:45.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.360+0000 7f145aa3e640 1 -- start start 2024-09-16T03:53:45.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.360+0000 7f145aa3e640 10 monclient: init 2024-09-16T03:53:45.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.364+0000 7f145aa3e640 10 monclient: _reopen_session rank -1 2024-09-16T03:53:45.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.364+0000 7f145aa3e640 10 monclient: _add_conns ranks=[0,1,4,2,3] 2024-09-16T03:53:45.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.364+0000 7f145aa3e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f14540a1330 addr v1:172.21.15.42:6789/0 2024-09-16T03:53:45.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.364+0000 7f145aa3e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f14540a1710 addr v1:172.21.15.154:6789/0 2024-09-16T03:53:45.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.364+0000 7f145aa3e640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f145413a3b0 0x7f145413c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:53:45.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.364+0000 7f145aa3e640 10 monclient(hunting): picked mon.noname-c con 0x7f145413a3b0 addr v2:172.21.15.154:3300/0 2024-09-16T03:53:45.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.364+0000 7f145aa3e640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f145409e4c0 con 0x7f14540a1330 2024-09-16T03:53:45.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.364+0000 7f145aa3e640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f14540a3170 con 0x7f14540a1710 2024-09-16T03:53:45.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.364+0000 7f145aa3e640 10 monclient(hunting): start opening mon connection 2024-09-16T03:53:45.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.364+0000 7f145aa3e640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f145409edf0 con 0x7f145413a3b0 2024-09-16T03:53:45.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.364+0000 7f145aa3e640 10 monclient(hunting): _renew_subs 2024-09-16T03:53:45.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.364+0000 7f145aa3e640 10 monclient(hunting): authenticate will time out at 1672.336728s 2024-09-16T03:53:45.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.364+0000 7f1453fff640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f14540a1330 0x7f145409f180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:40542/0 (socket says 172.21.15.42:40542) 2024-09-16T03:53:45.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.364+0000 7f1453fff640 1 -- 172.21.15.42:0/743026471 learned_addr learned my addr 172.21.15.42:0/743026471 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:53:45.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.364+0000 7f1458fdb640 1 --2- 172.21.15.42:0/743026471 >> v2:172.21.15.154:3300/0 conn(0x7f145413a3b0 0x7f145413c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:53:45.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.364+0000 7f1458fdb640 10 monclient(hunting): get_auth_request con 0x7f145413a3b0 auth_method 0 2024-09-16T03:53:45.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.364+0000 7f1458fdb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:53:45.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.364+0000 7f1458fdb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:53:45.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.364+0000 7f1458fdb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:53:45.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.364+0000 7f14537fe640 1 -- 172.21.15.42:0/743026471 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1406794133 0 0) 0x7f145409e4c0 con 0x7f14540a1330 2024-09-16T03:53:45.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.364+0000 7f14537fe640 10 monclient(hunting): _negotiate 2024-09-16T03:53:45.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.364+0000 7f14537fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:53:45.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.364+0000 7f14537fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:53:45.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.364+0000 7f14537fe640 10 monclient(hunting): my global_id is 824376 2024-09-16T03:53:45.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.364+0000 7f14537fe640 1 -- 172.21.15.42:0/743026471 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1440003530 con 0x7f14540a1330 2024-09-16T03:53:45.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.364+0000 7f1458fdb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:53:45.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.364+0000 7f1458fdb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:53:45.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.364+0000 7f1458fdb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:53:45.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.364+0000 7f14537fe640 1 -- 172.21.15.42:0/743026471 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 942317379 0 0) 0x7f1440003530 con 0x7f14540a1330 2024-09-16T03:53:45.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.364+0000 7f14537fe640 1 -- 172.21.15.42:0/743026471 >> v2:172.21.15.154:3300/0 conn(0x7f145413a3b0 msgr2=0x7f145413c7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:53:45.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.364+0000 7f14537fe640 1 --2- 172.21.15.42:0/743026471 >> v2:172.21.15.154:3300/0 conn(0x7f145413a3b0 0x7f145413c7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:53:45.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.364+0000 7f14537fe640 1 -- 172.21.15.42:0/743026471 >> v1:172.21.15.154:6789/0 conn(0x7f14540a1710 legacy=0x7f14540a1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:53:45.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.364+0000 7f14537fe640 10 monclient: _finish_hunting 0 2024-09-16T03:53:45.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.364+0000 7f14537fe640 1 monclient: found mon.noname-a-legacy 2024-09-16T03:53:45.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.364+0000 7f14537fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:53:45.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.364+0000 7f14537fe640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.42:6789/0 2024-09-16T03:53:45.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.364+0000 7f14537fe640 1 -- 172.21.15.42:0/743026471 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f145409d030 con 0x7f14540a1330 2024-09-16T03:53:45.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.364+0000 7f14537fe640 10 monclient: _finish_auth 0 2024-09-16T03:53:45.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.364+0000 7f14537fe640 10 monclient: _check_auth_tickets 2024-09-16T03:53:45.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.364+0000 7f14537fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:53:45.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.364+0000 7f14537fe640 1 -- 172.21.15.42:0/743026471 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2177413734 0 0) 0x7f1444001680 con 0x7f14540a1330 2024-09-16T03:53:45.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.364+0000 7f14537fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:53:45.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.364+0000 7f14537fe640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T03:53:45.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.364+0000 7f14537fe640 10 monclient: dump: 2024-09-16T03:53:45.377 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:53:45.377 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:53:45.377 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:45.377 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:45.377 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:53:45.377 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:53:45.377 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:53:45.377 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:53:45.377 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:53:45.378 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:53:45.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.364+0000 7f14537fe640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] but i'm connected to v1:172.21.15.42:6789/0, reconnecting 2024-09-16T03:53:45.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.364+0000 7f14537fe640 10 monclient: _reopen_session rank -1 2024-09-16T03:53:45.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.364+0000 7f14537fe640 1 -- 172.21.15.42:0/743026471 >> v1:172.21.15.42:6789/0 conn(0x7f14540a1330 legacy=0x7f145409f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:53:45.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.364+0000 7f14537fe640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T03:53:45.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.364+0000 7f14537fe640 1 --2- 172.21.15.42:0/743026471 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f14400039c0 0x7f1440003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:53:45.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.364+0000 7f14537fe640 10 monclient(hunting): picked mon.a con 0x7f14400039c0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:53:45.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.364+0000 7f14537fe640 1 --2- 172.21.15.42:0/743026471 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1440004810 0x7f1440004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:53:45.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.364+0000 7f14537fe640 10 monclient(hunting): picked mon.b con 0x7f1440004810 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:53:45.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.364+0000 7f14537fe640 1 --2- 172.21.15.42:0/743026471 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f14400052f0 0x7f144000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:53:45.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.364+0000 7f14537fe640 10 monclient(hunting): picked mon.c con 0x7f14400052f0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:53:45.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.364+0000 7f14537fe640 10 monclient(hunting): start opening mon connection 2024-09-16T03:53:45.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.364+0000 7f14537fe640 1 -- 172.21.15.42:0/743026471 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f145409edf0 con 0x7f14400039c0 2024-09-16T03:53:45.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.364+0000 7f14537fe640 10 monclient(hunting): start opening mon connection 2024-09-16T03:53:45.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.364+0000 7f14537fe640 1 -- 172.21.15.42:0/743026471 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f144000c1c0 con 0x7f1440004810 2024-09-16T03:53:45.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.364+0000 7f14537fe640 10 monclient(hunting): start opening mon connection 2024-09-16T03:53:45.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.364+0000 7f14537fe640 1 -- 172.21.15.42:0/743026471 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f144000c3c0 con 0x7f14400052f0 2024-09-16T03:53:45.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.364+0000 7f14537fe640 10 monclient(hunting): _renew_subs 2024-09-16T03:53:45.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.364+0000 7f14537fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:53:45.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.364+0000 7f1453fff640 1 --2- 172.21.15.42:0/743026471 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f14400039c0 0x7f1440003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:53:45.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.364+0000 7f1453fff640 10 monclient(hunting): get_auth_request con 0x7f14400039c0 auth_method 0 2024-09-16T03:53:45.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.364+0000 7f1453fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:53:45.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.364+0000 7f1453fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:53:45.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.364+0000 7f1453fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:53:45.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.364+0000 7f1458fdb640 1 --2- 172.21.15.42:0/743026471 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f14400052f0 0x7f144000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:53:45.380 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.364+0000 7f1453fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:53:45.380 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.364+0000 7f1453fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:53:45.380 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.364+0000 7f1453fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T03:53:45.380 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.364+0000 7f1458fdb640 10 monclient(hunting): get_auth_request con 0x7f14400052f0 auth_method 0 2024-09-16T03:53:45.380 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.364+0000 7f1458fdb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:53:45.380 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.364+0000 7f1458fdb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:53:45.380 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.364+0000 7f1458fdb640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:53:45.380 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.364+0000 7f1458fdb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:53:45.380 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.364+0000 7f1458fdb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:53:45.380 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.364+0000 7f1458fdb640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T03:53:45.380 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.364+0000 7f144b7fe640 1 --2- 172.21.15.42:0/743026471 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1440004810 0x7f1440004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:53:45.380 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.364+0000 7f1453fff640 10 monclient(hunting): handle_auth_done global_id 824376 payload 293 2024-09-16T03:53:45.380 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.364+0000 7f1453fff640 1 -- 172.21.15.42:0/743026471 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f14400052f0 msgr2=0x7f144000b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:53:45.380 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.364+0000 7f1453fff640 1 --2- 172.21.15.42:0/743026471 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f14400052f0 0x7f144000b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:53:45.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.364+0000 7f1453fff640 1 -- 172.21.15.42:0/743026471 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1440004810 msgr2=0x7f1440004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:53:45.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.364+0000 7f1453fff640 1 --2- 172.21.15.42:0/743026471 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1440004810 0x7f1440004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:53:45.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.364+0000 7f1453fff640 10 monclient: _finish_hunting 0 2024-09-16T03:53:45.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.364+0000 7f1453fff640 1 monclient: found mon.a 2024-09-16T03:53:45.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.364+0000 7f1453fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:53:45.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.364+0000 7f1453fff640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:53:45.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.364+0000 7f1458fdb640 1 -- 172.21.15.42:0/743026471 reap_dead start 2024-09-16T03:53:45.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.364+0000 7f1453fff640 1 -- 172.21.15.42:0/743026471 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1440003530 con 0x7f14400039c0 2024-09-16T03:53:45.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.364+0000 7f1453fff640 10 monclient: _finish_auth 0 2024-09-16T03:53:45.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.364+0000 7f1453fff640 10 monclient: _check_auth_tickets 2024-09-16T03:53:45.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.364+0000 7f1453fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:53:45.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.364+0000 7f145aa3e640 5 monclient: authenticate success, global_id 824376 2024-09-16T03:53:45.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.364+0000 7f145aa3e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:53:45.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.364+0000 7f144b7fe640 10 monclient: get_auth_request con 0x7f1440004810 auth_method 0 2024-09-16T03:53:45.382 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.364+0000 7f144b7fe640 1 --2- 172.21.15.42:0/743026471 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1440004810 0x7f1440004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T03:53:45.382 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.368+0000 7f1453fff640 1 --2- 172.21.15.42:0/743026471 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f14400039c0 0x7f1440003de0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f1444001ec0 tx=0x7f144401cba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30ad10f6a5514 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:53:45.382 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.368+0000 7f14537fe640 1 -- 172.21.15.42:0/743026471 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1444007d40 con 0x7f14400039c0 2024-09-16T03:53:45.382 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.368+0000 7f14537fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:53:45.382 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.368+0000 7f14537fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:53:45.382 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.368+0000 7f14537fe640 10 monclient: dump: 2024-09-16T03:53:45.382 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:53:45.382 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:53:45.382 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:45.382 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:45.382 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:53:45.382 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:53:45.382 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:53:45.382 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:53:45.383 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:53:45.383 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:53:45.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.368+0000 7f145aa3e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:53:45.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.368+0000 7f14537fe640 1 -- 172.21.15.42:0/743026471 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f14440a9070 con 0x7f14400039c0 2024-09-16T03:53:45.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.368+0000 7f14537fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:53:45.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.368+0000 7f145aa3e640 10 monclient: get_monmap_and_config success 2024-09-16T03:53:45.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.368+0000 7f14537fe640 1 -- 172.21.15.42:0/743026471 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14440a5dd0 con 0x7f14400039c0 2024-09-16T03:53:45.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.368+0000 7f14537fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:53:45.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.368+0000 7f145aa3e640 4 set_mon_vals no callback set 2024-09-16T03:53:45.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.368+0000 7f14537fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:53:45.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.368+0000 7f14537fe640 10 monclient: dump: 2024-09-16T03:53:45.383 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:53:45.383 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:53:45.383 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:45.384 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:45.384 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:53:45.384 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:53:45.384 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:53:45.384 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:53:45.384 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:53:45.384 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:53:45.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.368+0000 7f145aa3e640 10 monclient: shutdown 2024-09-16T03:53:45.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.368+0000 7f145aa3e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:53:45.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.368+0000 7f145aa3e640 1 -- 172.21.15.42:0/743026471 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f14400039c0 msgr2=0x7f1440003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:53:45.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.368+0000 7f145aa3e640 1 --2- 172.21.15.42:0/743026471 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f14400039c0 0x7f1440003de0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f1444001ec0 tx=0x7f144401cba0 comp rx=0 tx=0).stop 2024-09-16T03:53:45.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.368+0000 7f145aa3e640 1 -- 172.21.15.42:0/743026471 shutdown_connections 2024-09-16T03:53:45.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.368+0000 7f145aa3e640 1 --2- 172.21.15.42:0/743026471 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f14400039c0 0x7f1440003de0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:53:45.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.368+0000 7f145aa3e640 1 -- 172.21.15.42:0/743026471 >> 172.21.15.42:0/743026471 conn(0x7f14540a2420 msgr2=0x7f14540a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:53:45.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.368+0000 7f145aa3e640 1 -- 172.21.15.42:0/743026471 shutdown_connections 2024-09-16T03:53:45.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.368+0000 7f145aa3e640 1 -- 172.21.15.42:0/743026471 wait complete. 2024-09-16T03:53:45.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.368+0000 7f145aa3e640 10 monclient: build_initial_monmap 2024-09-16T03:53:45.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.368+0000 7f145aa3e640 10 monclient: monmap: 2024-09-16T03:53:45.394 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:53:45.394 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:53:45.394 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:53:45.394 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:53:45.394 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:53:45.395 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:53:45.395 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:53:45.395 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:53:45.395 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:53:45.395 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:53:45.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.368+0000 7f145aa3e640 1 librados: starting msgr at 2024-09-16T03:53:45.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.368+0000 7f145aa3e640 1 librados: starting objecter 2024-09-16T03:53:45.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.368+0000 7f145aa3e640 1 Processor -- start 2024-09-16T03:53:45.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.368+0000 7f145aa3e640 1 -- start start 2024-09-16T03:53:45.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.368+0000 7f145aa3e640 1 librados: setting wanted keys 2024-09-16T03:53:45.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.368+0000 7f145aa3e640 1 librados: calling monclient init 2024-09-16T03:53:45.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.368+0000 7f145aa3e640 10 monclient: init 2024-09-16T03:53:45.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.368+0000 7f145aa3e640 10 monclient: _reopen_session rank -1 2024-09-16T03:53:45.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.368+0000 7f145aa3e640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T03:53:45.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.368+0000 7f145aa3e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f14400039c0 0x7f1454070bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:53:45.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.368+0000 7f145aa3e640 10 monclient(hunting): picked mon.noname-b con 0x7f14400039c0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:53:45.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.368+0000 7f145aa3e640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1454071100 0x7f1454071580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:53:45.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.368+0000 7f145aa3e640 10 monclient(hunting): picked mon.noname-c con 0x7f1454071100 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:53:45.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.368+0000 7f145aa3e640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f14541cf5a0 0x7f14541cfa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:53:45.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.368+0000 7f145aa3e640 10 monclient(hunting): picked mon.noname-a con 0x7f14541cf5a0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:53:45.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.368+0000 7f145aa3e640 10 monclient(hunting): start opening mon connection 2024-09-16T03:53:45.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.368+0000 7f145aa3e640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f1454134480 con 0x7f14541cf5a0 2024-09-16T03:53:45.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.368+0000 7f145aa3e640 10 monclient(hunting): start opening mon connection 2024-09-16T03:53:45.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.368+0000 7f145aa3e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f1454134300 con 0x7f14400039c0 2024-09-16T03:53:45.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.368+0000 7f145aa3e640 10 monclient(hunting): start opening mon connection 2024-09-16T03:53:45.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.368+0000 7f145aa3e640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f1454134600 con 0x7f1454071100 2024-09-16T03:53:45.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.368+0000 7f145aa3e640 10 monclient(hunting): _renew_subs 2024-09-16T03:53:45.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.368+0000 7f145aa3e640 10 monclient(hunting): authenticate will time out at 1672.342868s 2024-09-16T03:53:45.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.368+0000 7f144b7fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1454071100 0x7f1454071580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:53:45.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.368+0000 7f1458fdb640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f14541cf5a0 0x7f14541cfa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:53:45.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.368+0000 7f144b7fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1454071100 0x7f1454071580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:34642/0 (socket says 172.21.15.42:34642) 2024-09-16T03:53:45.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.368+0000 7f144b7fe640 1 -- 172.21.15.42:0/2569516426 learned_addr learned my addr 172.21.15.42:0/2569516426 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:53:45.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.368+0000 7f144b7fe640 10 monclient(hunting): get_auth_request con 0x7f1454071100 auth_method 0 2024-09-16T03:53:45.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.368+0000 7f144b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:53:45.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.368+0000 7f144b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:53:45.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.368+0000 7f144b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:53:45.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.368+0000 7f1458fdb640 10 monclient(hunting): get_auth_request con 0x7f14541cf5a0 auth_method 0 2024-09-16T03:53:45.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.368+0000 7f1458fdb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:53:45.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.368+0000 7f1458fdb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:53:45.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.368+0000 7f1458fdb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:53:45.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.368+0000 7f1458fdb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:53:45.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.368+0000 7f1458fdb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:53:45.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.368+0000 7f1458fdb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:53:45.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.368+0000 7f144b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:53:45.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.368+0000 7f144b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:53:45.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.368+0000 7f144b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:53:45.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.368+0000 7f1453fff640 1 --2- 172.21.15.42:0/2569516426 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f14400039c0 0x7f1454070bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:53:45.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.368+0000 7f1453fff640 10 monclient(hunting): get_auth_request con 0x7f14400039c0 auth_method 0 2024-09-16T03:53:45.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.368+0000 7f1453fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:53:45.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.368+0000 7f1453fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:53:45.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.368+0000 7f1453fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:53:45.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.368+0000 7f1458fdb640 10 monclient(hunting): handle_auth_done global_id 824379 payload 824 2024-09-16T03:53:45.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.368+0000 7f1458fdb640 1 -- 172.21.15.42:0/2569516426 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1454071100 msgr2=0x7f1454071580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:53:45.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.368+0000 7f1458fdb640 1 --2- 172.21.15.42:0/2569516426 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1454071100 0x7f1454071580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:53:45.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.368+0000 7f1458fdb640 1 -- 172.21.15.42:0/2569516426 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f14400039c0 msgr2=0x7f1454070bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:53:45.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.368+0000 7f1458fdb640 1 --2- 172.21.15.42:0/2569516426 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f14400039c0 0x7f1454070bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:53:45.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.368+0000 7f1458fdb640 10 monclient: _finish_hunting 0 2024-09-16T03:53:45.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.368+0000 7f1458fdb640 1 monclient: found mon.noname-a 2024-09-16T03:53:45.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.368+0000 7f1458fdb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:53:45.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.368+0000 7f1458fdb640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:53:45.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.368+0000 7f1458fdb640 1 -- 172.21.15.42:0/2569516426 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1454071cf0 con 0x7f14541cf5a0 2024-09-16T03:53:45.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.368+0000 7f1458fdb640 1 --2- 172.21.15.42:0/2569516426 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f14541cf5a0 0x7f14541cfa20 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f14540a2270 tx=0x7f144c00f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:53:45.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.368+0000 7f14517fa640 10 client.?.objecter ms_handle_connect 0x7f14541cf5a0 2024-09-16T03:53:45.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.368+0000 7f14517fa640 10 client.?.objecter resend_mon_ops 2024-09-16T03:53:45.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.368+0000 7f14517fa640 1 -- 172.21.15.42:0/2569516426 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f144c00fd80 con 0x7f14541cf5a0 2024-09-16T03:53:45.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.368+0000 7f14517fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:53:45.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.368+0000 7f14517fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:53:45.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.368+0000 7f14517fa640 10 monclient: dump: 2024-09-16T03:53:45.401 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:53:45.401 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:53:45.401 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:45.401 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:45.401 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:53:45.401 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:53:45.401 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:53:45.401 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:53:45.401 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:53:45.401 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:53:45.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.368+0000 7f14517fa640 10 monclient: _finish_auth 0 2024-09-16T03:53:45.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.368+0000 7f14517fa640 10 monclient: _check_auth_tickets 2024-09-16T03:53:45.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.368+0000 7f14517fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:53:45.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.368+0000 7f14517fa640 1 -- 172.21.15.42:0/2569516426 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f144c015430 con 0x7f14541cf5a0 2024-09-16T03:53:45.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.368+0000 7f145aa3e640 5 monclient: authenticate success, global_id 824379 2024-09-16T03:53:45.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.368+0000 7f145aa3e640 10 monclient: _renew_subs 2024-09-16T03:53:45.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.368+0000 7f145aa3e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:53:45.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.368+0000 7f145aa3e640 1 -- 172.21.15.42:0/2569516426 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14541377c0 con 0x7f14541cf5a0 2024-09-16T03:53:45.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.368+0000 7f14517fa640 10 monclient: handle_config config(0 keys) 2024-09-16T03:53:45.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.368+0000 7f14517fa640 1 -- 172.21.15.42:0/2569516426 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f144c015820 con 0x7f14541cf5a0 2024-09-16T03:53:45.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.368+0000 7f14517fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:53:45.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.368+0000 7f14517fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:53:45.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.368+0000 7f14517fa640 10 monclient: dump: 2024-09-16T03:53:45.402 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:53:45.403 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:53:45.403 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:45.403 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:45.403 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:53:45.403 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:53:45.403 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:53:45.403 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:53:45.403 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:53:45.403 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:53:45.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.368+0000 7f14527fc640 4 set_mon_vals no callback set 2024-09-16T03:53:45.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.368+0000 7f145aa3e640 10 client.824379.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:53:45.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.368+0000 7f145aa3e640 10 monclient: _renew_subs 2024-09-16T03:53:45.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.368+0000 7f145aa3e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:53:45.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.368+0000 7f145aa3e640 1 -- 172.21.15.42:0/2569516426 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f145411cb70 con 0x7f14541cf5a0 2024-09-16T03:53:45.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.368+0000 7f145aa3e640 1 librados: init done 2024-09-16T03:53:45.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.372+0000 7f145aa3e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:53:45.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.372+0000 7f145aa3e640 1 --2- 172.21.15.42:0/2569516426 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f145402d7d0 0x7f14541cff60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:53:45.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.372+0000 7f145aa3e640 10 monclient(hunting): start opening mon connection 2024-09-16T03:53:45.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.372+0000 7f145aa3e640 1 -- 172.21.15.42:0/2569516426 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f145412fee0 con 0x7f145402d7d0 2024-09-16T03:53:45.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.372+0000 7f14517fa640 1 -- 172.21.15.42:0/2569516426 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f144c0159c0 con 0x7f14541cf5a0 2024-09-16T03:53:45.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.372+0000 7f1453fff640 1 --2- 172.21.15.42:0/2569516426 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f145402d7d0 0x7f14541cff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:53:45.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.372+0000 7f1453fff640 10 monclient: get_auth_request con 0x7f145402d7d0 auth_method 0 2024-09-16T03:53:45.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.372+0000 7f1453fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:53:45.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.372+0000 7f1453fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:53:45.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.372+0000 7f1453fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:53:45.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.372+0000 7f1453fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:53:45.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.372+0000 7f1453fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:53:45.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.372+0000 7f1453fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:53:45.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.372+0000 7f14517fa640 1 --2- 172.21.15.42:0/2569516426 >> v2:172.21.15.154:6812/254845342 conn(0x7f1434037810 0x7f1434039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:53:45.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.372+0000 7f1453fff640 10 monclient(hunting): handle_auth_done global_id 824382 payload 824 2024-09-16T03:53:45.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.372+0000 7f1453fff640 1 -- 172.21.15.42:0/2569516426 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f145402dbb0 con 0x7f145402d7d0 2024-09-16T03:53:45.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.372+0000 7f14517fa640 1 -- 172.21.15.42:0/2569516426 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(366..366 src has 286..366) ==== 5713+0+0 (secure 0 0 0) 0x7f144c049ca0 con 0x7f14541cf5a0 2024-09-16T03:53:45.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.372+0000 7f14517fa640 10 client.824379.objecter ms_dispatch 0x7f1454000ba0 osd_map(366..366 src has 286..366) 2024-09-16T03:53:45.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.372+0000 7f14517fa640 3 client.824379.objecter handle_osd_map got epochs [366,366] > 0 2024-09-16T03:53:45.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.372+0000 7f14517fa640 3 client.824379.objecter handle_osd_map decoding full epoch 366 2024-09-16T03:53:45.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.372+0000 7f14517fa640 20 client.824379.objecter dump_active .. 0 homeless 2024-09-16T03:53:45.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.372+0000 7f14517fa640 10 client.824379.objecter ms_handle_connect 0x7f145402d7d0 2024-09-16T03:53:45.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.372+0000 7f14517fa640 10 client.824379.objecter resend_mon_ops 2024-09-16T03:53:45.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.372+0000 7f1453fff640 1 --2- 172.21.15.42:0/2569516426 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f145402d7d0 0x7f14541cff60 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f14400057f0 tx=0x7f1444019940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:53:45.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.372+0000 7f144b7fe640 1 --2- 172.21.15.42:0/2569516426 >> v2:172.21.15.154:6812/254845342 conn(0x7f1434037810 0x7f1434039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:53:45.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.372+0000 7f144b7fe640 10 monclient: get_auth_request con 0x7f1434037810 auth_method 0 2024-09-16T03:53:45.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.372+0000 7f14517fa640 1 -- 172.21.15.42:0/2569516426 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14440a5d40 con 0x7f145402d7d0 2024-09-16T03:53:45.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.372+0000 7f14517fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:53:45.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.372+0000 7f14517fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:53:45.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.372+0000 7f14517fa640 10 monclient: dump: 2024-09-16T03:53:45.406 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:53:45.406 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:53:45.406 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:45.407 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:45.407 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:53:45.407 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:53:45.408 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:53:45.408 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:53:45.408 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:53:45.408 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:53:45.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.372+0000 7f14517fa640 1 -- 172.21.15.42:0/2569516426 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f14440a9030 con 0x7f145402d7d0 2024-09-16T03:53:45.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.372+0000 7f14517fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:53:45.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.372+0000 7f14517fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:53:45.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.372+0000 7f14517fa640 1 -- 172.21.15.42:0/2569516426 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f145402d7d0 msgr2=0x7f14541cff60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:53:45.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.372+0000 7f14517fa640 1 --2- 172.21.15.42:0/2569516426 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f145402d7d0 0x7f14541cff60 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f14400057f0 tx=0x7f1444019940 comp rx=0 tx=0).stop 2024-09-16T03:53:45.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.372+0000 7f14517fa640 1 -- 172.21.15.42:0/2569516426 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f145402d7d0 msgr2=0x7f14541cff60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:53:45.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.372+0000 7f14517fa640 1 --2- 172.21.15.42:0/2569516426 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f145402d7d0 0x7f14541cff60 secure :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f14400057f0 tx=0x7f1444019940 comp rx=0 tx=0).stop 2024-09-16T03:53:45.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.372+0000 7f14517fa640 10 client.824379.objecter ms_handle_connect 0x7f1434037810 2024-09-16T03:53:45.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.372+0000 7f144b7fe640 1 --2- 172.21.15.42:0/2569516426 >> v2:172.21.15.154:6812/254845342 conn(0x7f1434037810 0x7f1434039cd0 secure :-1 s=READY pgs=3605 cs=0 l=1 rev1=1 crypto rx=0x7f143c007fd0 tx=0x7f143c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:53:45.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.380+0000 7f145aa3e640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:53:45.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.380+0000 7f145aa3e640 1 --2- 172.21.15.42:0/2569516426 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f145411cdb0 0x7f1454062ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:53:45.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.380+0000 7f145aa3e640 10 monclient(hunting): start opening mon connection 2024-09-16T03:53:45.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.380+0000 7f145aa3e640 1 -- 172.21.15.42:0/2569516426 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f145411de10 con 0x7f145411cdb0 2024-09-16T03:53:45.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.380+0000 7f1453fff640 1 --2- 172.21.15.42:0/2569516426 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f145411cdb0 0x7f1454062ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:53:45.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.380+0000 7f1453fff640 10 monclient: get_auth_request con 0x7f145411cdb0 auth_method 0 2024-09-16T03:53:45.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.380+0000 7f1453fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:53:45.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.380+0000 7f1453fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:53:45.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.380+0000 7f1453fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:53:45.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.380+0000 7f1453fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:53:45.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.380+0000 7f1453fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:53:45.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.380+0000 7f1453fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:53:45.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.380+0000 7f1453fff640 10 monclient(hunting): handle_auth_done global_id 824385 payload 824 2024-09-16T03:53:45.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.380+0000 7f1453fff640 1 -- 172.21.15.42:0/2569516426 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f145411c970 con 0x7f145411cdb0 2024-09-16T03:53:45.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.380+0000 7f1453fff640 1 --2- 172.21.15.42:0/2569516426 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f145411cdb0 0x7f1454062ea0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f1444000c10 tx=0x7f1440005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:53:45.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.380+0000 7f14517fa640 10 client.824379.objecter ms_handle_connect 0x7f145411cdb0 2024-09-16T03:53:45.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.380+0000 7f14517fa640 10 client.824379.objecter resend_mon_ops 2024-09-16T03:53:45.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.380+0000 7f14517fa640 1 -- 172.21.15.42:0/2569516426 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14440074b0 con 0x7f145411cdb0 2024-09-16T03:53:45.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.380+0000 7f14517fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:53:45.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.380+0000 7f14517fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:53:45.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.380+0000 7f14517fa640 10 monclient: dump: 2024-09-16T03:53:45.410 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:53:45.410 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:53:45.410 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:45.410 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:45.410 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:53:45.410 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:53:45.410 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:53:45.411 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:53:45.411 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:53:45.411 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:53:45.411 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.380+0000 7f14517fa640 1 -- 172.21.15.42:0/2569516426 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4882 (secure 0 0 0) 0x7f14440c4070 con 0x7f145411cdb0 2024-09-16T03:53:45.411 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.380+0000 7f14517fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:53:45.411 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.380+0000 7f14517fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:53:45.411 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.380+0000 7f14517fa640 1 -- 172.21.15.42:0/2569516426 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f145411cdb0 msgr2=0x7f1454062ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:53:45.411 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.380+0000 7f14517fa640 1 --2- 172.21.15.42:0/2569516426 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f145411cdb0 0x7f1454062ea0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f1444000c10 tx=0x7f1440005840 comp rx=0 tx=0).stop 2024-09-16T03:53:45.411 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.380+0000 7f14517fa640 1 -- 172.21.15.42:0/2569516426 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f145411cdb0 msgr2=0x7f1454062ea0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:53:45.411 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.380+0000 7f14517fa640 1 --2- 172.21.15.42:0/2569516426 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f145411cdb0 0x7f1454062ea0 secure :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f1444000c10 tx=0x7f1440005840 comp rx=0 tx=0).stop 2024-09-16T03:53:45.412 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:53:45.412 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:53:45.412 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:53:45.412 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "leader", 2024-09-16T03:53:45.412 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 100, 2024-09-16T03:53:45.412 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 115234, 2024-09-16T03:53:45.412 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:53:45.412 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:53:45.412 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T03:53:45.412 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T03:53:45.412 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:53:45.413 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 5, 2024-09-16T03:53:45.413 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:53:45.413 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:53:45.413 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:53:45.413 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:53:45.413 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:53:45.413 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:53:45.413 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:53:45.413 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:53:45.413 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:53:45.413 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:53:45.413 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:53:45.413 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:53:45.414 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:53:45.414 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:53:45.414 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:53:45.414 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:53:45.414 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:53:45.414 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:53:45.415 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:53:45.415 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:53:45.415 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:53:45.415 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:53:45.415 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:53:45.415 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:53:45.415 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:53:45.415 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:53:45.415 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:53:45.415 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:53:45.415 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:53:45.415 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:53:45.415 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:53:45.416 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:53:45.416 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:53:45.416 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:53:45.416 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:53:45.416 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:53:45.416 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:53:45.416 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:53:45.416 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:53:45.416 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:53:45.416 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:53:45.416 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:53:45.416 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:53:45.416 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:53:45.416 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:53:45.417 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:53:45.417 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:53:45.417 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:53:45.418 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:53:45.418 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:53:45.418 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:53:45.418 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:53:45.418 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:53:45.418 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:53:45.418 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:53:45.419 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:53:45.419 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:53:45.419 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:53:45.419 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:53:45.419 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:53:45.419 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:53:45.419 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:53:45.419 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:53:45.420 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:53:45.420 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:53:45.420 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:53:45.420 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:53:45.420 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:53:45.420 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:53:45.420 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:53:45.420 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:53:45.420 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:53:45.420 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:53:45.420 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:53:45.421 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:53:45.421 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:53:45.421 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:53:45.421 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:53:45.422 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:53:45.422 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:53:45.422 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:53:45.422 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:53:45.422 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:53:45.422 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:53:45.422 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:53:45.422 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:53:45.422 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:53:45.422 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:53:45.422 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:53:45.422 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:53:45.423 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:53:45.423 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:53:45.423 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:53:45.423 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:53:45.423 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:53:45.423 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:53:45.423 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:53:45.423 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:53:45.423 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:53:45.423 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:53:45.423 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:53:45.423 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:53:45.423 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:53:45.424 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:53:45.424 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:53:45.425 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:53:45.425 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:53:45.425 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:53:45.425 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:53:45.425 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:53:45.425 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:53:45.425 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:53:45.425 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:53:45.425 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:53:45.426 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:53:45.426 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:53:45.426 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:53:45.426 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:53:45.426 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:53:45.426 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:53:45.427 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:53:45.427 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:53:45.427 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:53:45.427 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:53:45.427 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:53:45.427 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:53:45.427 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:53:45.427 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:53:45.427 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:53:45.427 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:53:45.427 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:53:45.427 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:53:45.427 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:53:45.427 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:53:45.428 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:53:45.428 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:53:45.428 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:53:45.428 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:53:45.428 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:53:45.428 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:53:45.428 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:53:45.428 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T03:53:45.428 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:53:45.428 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:53:45.428 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:53:45.428 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 5 2024-09-16T03:53:45.428 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:53:45.428 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:53:45.429 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:53:45.429 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:53:45.429 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:53:45.429 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:53:45.429 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 7 2024-09-16T03:53:45.429 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:53:45.429 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:53:45.429 INFO:teuthology.orchestra.run.smithi042.stdout: "mgr": [ 2024-09-16T03:53:45.429 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:53:45.429 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:53:45.429 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:53:45.429 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:53:45.429 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:53:45.429 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:53:45.430 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:53:45.430 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:53:45.430 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:53:45.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.380+0000 7f145aa3e640 10 librados: watch_flush enter 2024-09-16T03:53:45.432 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.380+0000 7f145aa3e640 10 librados: watch_flush exit 2024-09-16T03:53:45.432 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.380+0000 7f145aa3e640 20 client.824379.objecter shutdown clearing up homeless session... 2024-09-16T03:53:45.432 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.380+0000 7f145aa3e640 10 client.824379.objecter successfully canceled tick 2024-09-16T03:53:45.432 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.380+0000 7f145aa3e640 1 -- 172.21.15.42:0/2569516426 >> v2:172.21.15.154:6812/254845342 conn(0x7f1434037810 msgr2=0x7f1434039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:53:45.432 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.380+0000 7f145aa3e640 1 --2- 172.21.15.42:0/2569516426 >> v2:172.21.15.154:6812/254845342 conn(0x7f1434037810 0x7f1434039cd0 secure :-1 s=READY pgs=3605 cs=0 l=1 rev1=1 crypto rx=0x7f143c007fd0 tx=0x7f143c009040 comp rx=0 tx=0).stop 2024-09-16T03:53:45.432 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.380+0000 7f145aa3e640 10 monclient: shutdown 2024-09-16T03:53:45.432 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.380+0000 7f145aa3e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:53:45.432 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.380+0000 7f145aa3e640 1 -- 172.21.15.42:0/2569516426 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f14541cf5a0 msgr2=0x7f14541cfa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:53:45.432 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.380+0000 7f1458fdb640 1 -- 172.21.15.42:0/2569516426 reap_dead start 2024-09-16T03:53:45.432 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.380+0000 7f145aa3e640 1 --2- 172.21.15.42:0/2569516426 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f14541cf5a0 0x7f14541cfa20 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f14540a2270 tx=0x7f144c00f580 comp rx=0 tx=0).stop 2024-09-16T03:53:45.432 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.380+0000 7f145aa3e640 1 -- 172.21.15.42:0/2569516426 shutdown_connections 2024-09-16T03:53:45.432 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.380+0000 7f145aa3e640 1 --2- 172.21.15.42:0/2569516426 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f14541cf5a0 0x7f14541cfa20 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:53:45.432 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.380+0000 7f145aa3e640 1 -- 172.21.15.42:0/2569516426 >> 172.21.15.42:0/2569516426 conn(0x7f14540a2420 msgr2=0x7f14541324f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:53:45.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.380+0000 7f145aa3e640 1 -- 172.21.15.42:0/2569516426 shutdown_connections 2024-09-16T03:53:45.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.380+0000 7f145aa3e640 1 -- 172.21.15.42:0/2569516426 wait complete. 2024-09-16T03:53:45.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.380+0000 7f145aa3e640 1 librados: shutdown 2024-09-16T03:53:45.433 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:53:45.433 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T03:53:45.434 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T03:53:45.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.548+0000 7f137ba4c640 10 monclient: get_monmap_and_config 2024-09-16T03:53:45.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.548+0000 7f137ba4c640 10 monclient: build_initial_monmap 2024-09-16T03:53:45.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.548+0000 7f137ba4c640 10 monclient: monmap: 2024-09-16T03:53:45.554 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:53:45.554 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:53:45.554 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:53:45.552306+0000 2024-09-16T03:53:45.554 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:53:45.552306+0000 2024-09-16T03:53:45.554 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:53:45.554 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:53:45.554 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:53:45.554 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:53:45.554 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:53:45.554 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:53:45.554 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:53:45.554 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:53:45.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.548+0000 7f137ba4c640 1 Processor -- start 2024-09-16T03:53:45.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.548+0000 7f137ba4c640 1 -- start start 2024-09-16T03:53:45.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.548+0000 7f137ba4c640 10 monclient: init 2024-09-16T03:53:45.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.548+0000 7f137ba4c640 10 monclient: _reopen_session rank -1 2024-09-16T03:53:45.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.548+0000 7f137ba4c640 10 monclient: _add_conns ranks=[4,1,0,2,3] 2024-09-16T03:53:45.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.548+0000 7f137ba4c640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f13740a39b0 0x7f1374133b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:53:45.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.548+0000 7f137ba4c640 10 monclient(hunting): picked mon.noname-c con 0x7f13740a39b0 addr v2:172.21.15.154:3300/0 2024-09-16T03:53:45.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.548+0000 7f137ba4c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f13740a3d90 addr v1:172.21.15.154:6789/0 2024-09-16T03:53:45.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.548+0000 7f137ba4c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1374137710 addr v1:172.21.15.42:6789/0 2024-09-16T03:53:45.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.548+0000 7f137ba4c640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f137409c140 con 0x7f1374137710 2024-09-16T03:53:45.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.548+0000 7f137ba4c640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f137409c420 con 0x7f13740a3d90 2024-09-16T03:53:45.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.548+0000 7f137ba4c640 10 monclient(hunting): start opening mon connection 2024-09-16T03:53:45.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.548+0000 7f137ba4c640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f137406dd60 con 0x7f13740a39b0 2024-09-16T03:53:45.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.548+0000 7f137ba4c640 10 monclient(hunting): _renew_subs 2024-09-16T03:53:45.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.548+0000 7f137ba4c640 10 monclient(hunting): authenticate will time out at 1672.521820s 2024-09-16T03:53:45.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.548+0000 7f1379fe9640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f1374137710 0x7f137413f3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:40544/0 (socket says 172.21.15.42:40544) 2024-09-16T03:53:45.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.548+0000 7f1379fe9640 1 -- 172.21.15.42:0/29484937 learned_addr learned my addr 172.21.15.42:0/29484937 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:53:45.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.548+0000 7f13797e8640 1 --2- 172.21.15.42:0/29484937 >> v2:172.21.15.154:3300/0 conn(0x7f13740a39b0 0x7f1374133b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:53:45.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.548+0000 7f13797e8640 10 monclient(hunting): get_auth_request con 0x7f13740a39b0 auth_method 0 2024-09-16T03:53:45.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.548+0000 7f13797e8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:53:45.568 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.548+0000 7f13797e8640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:53:45.568 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.548+0000 7f13797e8640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:53:45.568 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.548+0000 7f136bfff640 1 -- 172.21.15.42:0/29484937 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 280857898 0 0) 0x7f137409c140 con 0x7f1374137710 2024-09-16T03:53:45.568 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.548+0000 7f136bfff640 10 monclient(hunting): _negotiate 2024-09-16T03:53:45.568 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.548+0000 7f136bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:53:45.568 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.548+0000 7f136bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:53:45.568 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.548+0000 7f136bfff640 10 monclient(hunting): my global_id is 824388 2024-09-16T03:53:45.568 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.548+0000 7f136bfff640 1 -- 172.21.15.42:0/29484937 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1360003530 con 0x7f1374137710 2024-09-16T03:53:45.568 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.548+0000 7f13797e8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:53:45.568 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.548+0000 7f13797e8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:53:45.568 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.548+0000 7f13797e8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:53:45.568 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.548+0000 7f136bfff640 1 -- 172.21.15.42:0/29484937 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 745229777 0 0) 0x7f1360003530 con 0x7f1374137710 2024-09-16T03:53:45.568 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.548+0000 7f136bfff640 1 -- 172.21.15.42:0/29484937 >> v2:172.21.15.154:3300/0 conn(0x7f13740a39b0 msgr2=0x7f1374133b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:53:45.569 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.548+0000 7f136bfff640 1 --2- 172.21.15.42:0/29484937 >> v2:172.21.15.154:3300/0 conn(0x7f13740a39b0 0x7f1374133b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:53:45.569 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.548+0000 7f136bfff640 1 -- 172.21.15.42:0/29484937 >> v1:172.21.15.154:6789/0 conn(0x7f13740a3d90 legacy=0x7f1374137000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:53:45.569 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.548+0000 7f136bfff640 10 monclient: _finish_hunting 0 2024-09-16T03:53:45.569 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.548+0000 7f136bfff640 1 monclient: found mon.noname-a-legacy 2024-09-16T03:53:45.569 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.548+0000 7f136bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:53:45.569 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.548+0000 7f136bfff640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.42:6789/0 2024-09-16T03:53:45.569 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.548+0000 7f136bfff640 1 -- 172.21.15.42:0/29484937 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13740a5e20 con 0x7f1374137710 2024-09-16T03:53:45.569 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.548+0000 7f136bfff640 10 monclient: _finish_auth 0 2024-09-16T03:53:45.569 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.548+0000 7f136bfff640 10 monclient: _check_auth_tickets 2024-09-16T03:53:45.569 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.548+0000 7f136bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:53:45.569 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.548+0000 7f136bfff640 1 -- 172.21.15.42:0/29484937 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2177413734 0 0) 0x7f136c001a00 con 0x7f1374137710 2024-09-16T03:53:45.569 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.548+0000 7f136bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:53:45.569 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.548+0000 7f136bfff640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T03:53:45.569 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.548+0000 7f136bfff640 10 monclient: dump: 2024-09-16T03:53:45.570 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:53:45.570 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:53:45.570 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:45.570 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:45.570 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:53:45.570 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:53:45.570 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:53:45.570 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:53:45.570 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:53:45.570 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:53:45.570 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.548+0000 7f136bfff640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] but i'm connected to v1:172.21.15.42:6789/0, reconnecting 2024-09-16T03:53:45.570 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.548+0000 7f136bfff640 10 monclient: _reopen_session rank -1 2024-09-16T03:53:45.570 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.548+0000 7f136bfff640 1 -- 172.21.15.42:0/29484937 >> v1:172.21.15.42:6789/0 conn(0x7f1374137710 legacy=0x7f137413f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:53:45.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.548+0000 7f136bfff640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T03:53:45.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.548+0000 7f136bfff640 1 --2- 172.21.15.42:0/29484937 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f13600039c0 0x7f1360003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:53:45.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.548+0000 7f136bfff640 10 monclient(hunting): picked mon.a con 0x7f13600039c0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:53:45.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.548+0000 7f136bfff640 1 --2- 172.21.15.42:0/29484937 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1360004810 0x7f1360004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:53:45.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.548+0000 7f136bfff640 10 monclient(hunting): picked mon.b con 0x7f1360004810 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:53:45.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.548+0000 7f136bfff640 1 --2- 172.21.15.42:0/29484937 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f13600052f0 0x7f136000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:53:45.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.548+0000 7f136bfff640 10 monclient(hunting): picked mon.c con 0x7f13600052f0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:53:45.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.548+0000 7f136bfff640 10 monclient(hunting): start opening mon connection 2024-09-16T03:53:45.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.548+0000 7f136bfff640 1 -- 172.21.15.42:0/29484937 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f137406dd60 con 0x7f13600039c0 2024-09-16T03:53:45.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.548+0000 7f136bfff640 10 monclient(hunting): start opening mon connection 2024-09-16T03:53:45.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.548+0000 7f136bfff640 1 -- 172.21.15.42:0/29484937 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f136000c1c0 con 0x7f1360004810 2024-09-16T03:53:45.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.548+0000 7f136bfff640 10 monclient(hunting): start opening mon connection 2024-09-16T03:53:45.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.548+0000 7f136bfff640 1 -- 172.21.15.42:0/29484937 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f136000c3c0 con 0x7f13600052f0 2024-09-16T03:53:45.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.548+0000 7f136bfff640 10 monclient(hunting): _renew_subs 2024-09-16T03:53:45.572 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.548+0000 7f136bfff640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:53:45.572 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.548+0000 7f13797e8640 1 --2- 172.21.15.42:0/29484937 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f13600039c0 0x7f1360003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:53:45.572 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.548+0000 7f13797e8640 10 monclient(hunting): get_auth_request con 0x7f13600039c0 auth_method 0 2024-09-16T03:53:45.572 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.548+0000 7f1379fe9640 1 --2- 172.21.15.42:0/29484937 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f13600052f0 0x7f136000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:53:45.572 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.548+0000 7f13797e8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:53:45.572 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.548+0000 7f13797e8640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:53:45.572 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.548+0000 7f13797e8640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:53:45.572 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.548+0000 7f1379fe9640 10 monclient(hunting): get_auth_request con 0x7f13600052f0 auth_method 0 2024-09-16T03:53:45.573 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.548+0000 7f1379fe9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:53:45.573 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.548+0000 7f1379fe9640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:53:45.573 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.548+0000 7f1379fe9640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:53:45.573 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.548+0000 7f13797e8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:53:45.573 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.548+0000 7f13797e8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:53:45.573 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.548+0000 7f13797e8640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T03:53:45.573 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.548+0000 7f1379fe9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:53:45.573 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.548+0000 7f1379fe9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:53:45.573 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.548+0000 7f1379fe9640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T03:53:45.573 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.548+0000 7f1378fe7640 1 --2- 172.21.15.42:0/29484937 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1360004810 0x7f1360004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:53:45.573 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.548+0000 7f1378fe7640 10 monclient(hunting): get_auth_request con 0x7f1360004810 auth_method 0 2024-09-16T03:53:45.573 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.548+0000 7f1378fe7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:53:45.573 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.548+0000 7f1378fe7640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:53:45.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.548+0000 7f1378fe7640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:53:45.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.548+0000 7f13797e8640 10 monclient(hunting): handle_auth_done global_id 824388 payload 293 2024-09-16T03:53:45.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.548+0000 7f13797e8640 1 -- 172.21.15.42:0/29484937 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f13600052f0 msgr2=0x7f136000b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:53:45.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.548+0000 7f13797e8640 1 --2- 172.21.15.42:0/29484937 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f13600052f0 0x7f136000b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:53:45.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.548+0000 7f13797e8640 1 -- 172.21.15.42:0/29484937 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1360004810 msgr2=0x7f1360004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:53:45.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.548+0000 7f13797e8640 1 --2- 172.21.15.42:0/29484937 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1360004810 0x7f1360004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:53:45.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.548+0000 7f13797e8640 10 monclient: _finish_hunting 0 2024-09-16T03:53:45.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.548+0000 7f13797e8640 1 monclient: found mon.a 2024-09-16T03:53:45.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.548+0000 7f13797e8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:53:45.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.548+0000 7f13797e8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:53:45.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.548+0000 7f13797e8640 1 -- 172.21.15.42:0/29484937 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1360003530 con 0x7f13600039c0 2024-09-16T03:53:45.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.548+0000 7f13797e8640 10 monclient: _finish_auth 0 2024-09-16T03:53:45.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.548+0000 7f13797e8640 10 monclient: _check_auth_tickets 2024-09-16T03:53:45.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.548+0000 7f13797e8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:53:45.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.548+0000 7f137ba4c640 5 monclient: authenticate success, global_id 824388 2024-09-16T03:53:45.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.548+0000 7f137ba4c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:53:45.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.548+0000 7f1379fe9640 1 -- 172.21.15.42:0/29484937 reap_dead start 2024-09-16T03:53:45.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.552+0000 7f13797e8640 1 --2- 172.21.15.42:0/29484937 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f13600039c0 0x7f1360003de0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f1364009ab0 tx=0x7f136401c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d15f76095f026280 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:53:45.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.552+0000 7f136bfff640 1 -- 172.21.15.42:0/29484937 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13640096b0 con 0x7f13600039c0 2024-09-16T03:53:45.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.552+0000 7f136bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:53:45.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.552+0000 7f136bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:53:45.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.552+0000 7f136bfff640 10 monclient: dump: 2024-09-16T03:53:45.575 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:53:45.575 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:53:45.575 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:45.575 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:45.575 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:53:45.576 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:53:45.576 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:53:45.576 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:53:45.576 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:53:45.576 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:53:45.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.552+0000 7f137ba4c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:53:45.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.552+0000 7f136bfff640 1 -- 172.21.15.42:0/29484937 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f13640a9430 con 0x7f13600039c0 2024-09-16T03:53:45.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.552+0000 7f136bfff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:53:45.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.552+0000 7f136bfff640 1 -- 172.21.15.42:0/29484937 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13640a9820 con 0x7f13600039c0 2024-09-16T03:53:45.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.552+0000 7f136bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:53:45.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.552+0000 7f136bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:53:45.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.552+0000 7f136bfff640 10 monclient: dump: 2024-09-16T03:53:45.576 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:53:45.576 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:53:45.577 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:45.577 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:45.577 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:53:45.577 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:53:45.577 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:53:45.577 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:53:45.577 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:53:45.577 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:53:45.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.552+0000 7f137ba4c640 10 monclient: get_monmap_and_config success 2024-09-16T03:53:45.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.552+0000 7f137ba4c640 4 set_mon_vals no callback set 2024-09-16T03:53:45.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.552+0000 7f137ba4c640 10 monclient: shutdown 2024-09-16T03:53:45.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.552+0000 7f137ba4c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:53:45.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.552+0000 7f137ba4c640 1 -- 172.21.15.42:0/29484937 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f13600039c0 msgr2=0x7f1360003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:53:45.578 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.552+0000 7f137ba4c640 1 --2- 172.21.15.42:0/29484937 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f13600039c0 0x7f1360003de0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f1364009ab0 tx=0x7f136401c950 comp rx=0 tx=0).stop 2024-09-16T03:53:45.578 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.552+0000 7f137ba4c640 1 -- 172.21.15.42:0/29484937 shutdown_connections 2024-09-16T03:53:45.578 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.552+0000 7f137ba4c640 1 --2- 172.21.15.42:0/29484937 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f13600039c0 0x7f1360003de0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:53:45.578 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.552+0000 7f137ba4c640 1 -- 172.21.15.42:0/29484937 >> 172.21.15.42:0/29484937 conn(0x7f13740a9fb0 msgr2=0x7f13740aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:53:45.578 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.552+0000 7f137ba4c640 1 -- 172.21.15.42:0/29484937 shutdown_connections 2024-09-16T03:53:45.578 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.552+0000 7f137ba4c640 1 -- 172.21.15.42:0/29484937 wait complete. 2024-09-16T03:53:45.578 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.552+0000 7f137ba4c640 10 monclient: build_initial_monmap 2024-09-16T03:53:45.578 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.552+0000 7f137ba4c640 10 monclient: monmap: 2024-09-16T03:53:45.578 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:53:45.578 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:53:45.578 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:53:45.578 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:53:45.578 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:53:45.578 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:53:45.579 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:53:45.579 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:53:45.579 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:53:45.579 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:53:45.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.552+0000 7f137ba4c640 1 librados: starting msgr at 2024-09-16T03:53:45.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.552+0000 7f137ba4c640 1 librados: starting objecter 2024-09-16T03:53:45.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.552+0000 7f137ba4c640 1 Processor -- start 2024-09-16T03:53:45.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.552+0000 7f137ba4c640 1 -- start start 2024-09-16T03:53:45.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.552+0000 7f137ba4c640 1 librados: setting wanted keys 2024-09-16T03:53:45.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.552+0000 7f137ba4c640 1 librados: calling monclient init 2024-09-16T03:53:45.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.552+0000 7f137ba4c640 10 monclient: init 2024-09-16T03:53:45.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.552+0000 7f137ba4c640 10 monclient: _reopen_session rank -1 2024-09-16T03:53:45.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.552+0000 7f137ba4c640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:53:45.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.552+0000 7f137ba4c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f13600039c0 0x7f13741cfc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:53:45.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.552+0000 7f137ba4c640 10 monclient(hunting): picked mon.noname-a con 0x7f13600039c0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:53:45.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.552+0000 7f137ba4c640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f13741d0160 0x7f13740a22e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:53:45.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.552+0000 7f137ba4c640 10 monclient(hunting): picked mon.noname-c con 0x7f13741d0160 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:53:45.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.552+0000 7f137ba4c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f13741d0560 0x7f13740a2820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:53:45.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.552+0000 7f137ba4c640 10 monclient(hunting): picked mon.noname-b con 0x7f13741d0560 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:53:45.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.552+0000 7f137ba4c640 10 monclient(hunting): start opening mon connection 2024-09-16T03:53:45.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.552+0000 7f137ba4c640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f137413a580 con 0x7f13600039c0 2024-09-16T03:53:45.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.552+0000 7f137ba4c640 10 monclient(hunting): start opening mon connection 2024-09-16T03:53:45.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.552+0000 7f137ba4c640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f137406dee0 con 0x7f13741d0560 2024-09-16T03:53:45.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.552+0000 7f137ba4c640 10 monclient(hunting): start opening mon connection 2024-09-16T03:53:45.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.552+0000 7f137ba4c640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f137413a700 con 0x7f13741d0160 2024-09-16T03:53:45.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.552+0000 7f137ba4c640 10 monclient(hunting): _renew_subs 2024-09-16T03:53:45.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.552+0000 7f137ba4c640 10 monclient(hunting): authenticate will time out at 1672.527844s 2024-09-16T03:53:45.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.552+0000 7f13797e8640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f13600039c0 0x7f13741cfc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:53:45.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.552+0000 7f1378fe7640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f13741d0160 0x7f13740a22e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:53:45.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.552+0000 7f13797e8640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f13600039c0 0x7f13741cfc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:53786/0 (socket says 172.21.15.42:53786) 2024-09-16T03:53:45.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.552+0000 7f13797e8640 1 -- 172.21.15.42:0/21746962 learned_addr learned my addr 172.21.15.42:0/21746962 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:53:45.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.556+0000 7f13797e8640 10 monclient(hunting): get_auth_request con 0x7f13600039c0 auth_method 0 2024-09-16T03:53:45.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.556+0000 7f13797e8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:53:45.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.556+0000 7f13797e8640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:53:45.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.556+0000 7f13797e8640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:53:45.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.556+0000 7f1378fe7640 10 monclient(hunting): get_auth_request con 0x7f13741d0160 auth_method 0 2024-09-16T03:53:45.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.556+0000 7f1378fe7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:53:45.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.556+0000 7f1378fe7640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:53:45.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.556+0000 7f1378fe7640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:53:45.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.556+0000 7f13797e8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:53:45.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.556+0000 7f13797e8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:53:45.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.556+0000 7f13797e8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:53:45.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.556+0000 7f1378fe7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:53:45.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.556+0000 7f1378fe7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:53:45.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.556+0000 7f1378fe7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:53:45.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.556+0000 7f1379fe9640 1 --2- 172.21.15.42:0/21746962 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f13741d0560 0x7f13740a2820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:53:45.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.556+0000 7f13797e8640 10 monclient(hunting): handle_auth_done global_id 824391 payload 824 2024-09-16T03:53:45.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.556+0000 7f13797e8640 1 -- 172.21.15.42:0/21746962 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f13741d0160 msgr2=0x7f13740a22e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:53:45.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.556+0000 7f13797e8640 1 --2- 172.21.15.42:0/21746962 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f13741d0160 0x7f13740a22e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:53:45.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.556+0000 7f13797e8640 1 -- 172.21.15.42:0/21746962 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f13741d0560 msgr2=0x7f13740a2820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:53:45.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.556+0000 7f13797e8640 1 --2- 172.21.15.42:0/21746962 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f13741d0560 0x7f13740a2820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:53:45.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.556+0000 7f13797e8640 10 monclient: _finish_hunting 0 2024-09-16T03:53:45.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.556+0000 7f13797e8640 1 monclient: found mon.noname-a 2024-09-16T03:53:45.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.556+0000 7f13797e8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:53:45.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.556+0000 7f13797e8640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:53:45.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.556+0000 7f13797e8640 1 -- 172.21.15.42:0/21746962 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13741227e0 con 0x7f13600039c0 2024-09-16T03:53:45.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.556+0000 7f1379fe9640 10 monclient: get_auth_request con 0x7f13741d0560 auth_method 0 2024-09-16T03:53:45.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.556+0000 7f1379fe9640 1 --2- 172.21.15.42:0/21746962 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f13741d0560 0x7f13740a2820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T03:53:45.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.556+0000 7f13797e8640 1 --2- 172.21.15.42:0/21746962 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f13600039c0 0x7f13741cfc20 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f136401c780 tx=0x7f13640196f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:53:45.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.556+0000 7f1369ffb640 10 client.?.objecter ms_handle_connect 0x7f13600039c0 2024-09-16T03:53:45.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.556+0000 7f1369ffb640 10 client.?.objecter resend_mon_ops 2024-09-16T03:53:45.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.556+0000 7f1369ffb640 1 -- 172.21.15.42:0/21746962 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13640c1070 con 0x7f13600039c0 2024-09-16T03:53:45.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.556+0000 7f1369ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:53:45.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.556+0000 7f1369ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:53:45.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.556+0000 7f1369ffb640 10 monclient: dump: 2024-09-16T03:53:45.583 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:53:45.583 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:53:45.583 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:45.584 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:45.584 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:53:45.584 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:53:45.584 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:53:45.584 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:53:45.584 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:53:45.584 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:53:45.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.556+0000 7f1369ffb640 10 monclient: _finish_auth 0 2024-09-16T03:53:45.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.556+0000 7f1369ffb640 10 monclient: _check_auth_tickets 2024-09-16T03:53:45.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.556+0000 7f1369ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:53:45.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.556+0000 7f1369ffb640 1 -- 172.21.15.42:0/21746962 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f13640a9e10 con 0x7f13600039c0 2024-09-16T03:53:45.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.556+0000 7f1369ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T03:53:45.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.556+0000 7f1369ffb640 1 -- 172.21.15.42:0/21746962 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13640bc420 con 0x7f13600039c0 2024-09-16T03:53:45.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.556+0000 7f1369ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:53:45.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.556+0000 7f1369ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:53:45.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.556+0000 7f1369ffb640 10 monclient: dump: 2024-09-16T03:53:45.585 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:53:45.585 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:53:45.585 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:45.586 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:45.586 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:53:45.586 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:53:45.586 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:53:45.586 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:53:45.586 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:53:45.586 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:53:45.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.556+0000 7f136b7fe640 4 set_mon_vals no callback set 2024-09-16T03:53:45.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.556+0000 7f137ba4c640 5 monclient: authenticate success, global_id 824391 2024-09-16T03:53:45.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.556+0000 7f137ba4c640 10 monclient: _renew_subs 2024-09-16T03:53:45.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.556+0000 7f137ba4c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:53:45.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.556+0000 7f137ba4c640 1 -- 172.21.15.42:0/21746962 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1374122ac0 con 0x7f13600039c0 2024-09-16T03:53:45.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.556+0000 7f137ba4c640 10 client.824391.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:53:45.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.556+0000 7f137ba4c640 10 monclient: _renew_subs 2024-09-16T03:53:45.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.556+0000 7f137ba4c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:53:45.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.556+0000 7f137ba4c640 1 -- 172.21.15.42:0/21746962 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1374120c30 con 0x7f13600039c0 2024-09-16T03:53:45.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.556+0000 7f137ba4c640 1 librados: init done 2024-09-16T03:53:45.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.556+0000 7f137ba4c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:53:45.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.556+0000 7f137ba4c640 1 --2- 172.21.15.42:0/21746962 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f13740a2d60 0x7f13740a3140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:53:45.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.556+0000 7f137ba4c640 10 monclient(hunting): start opening mon connection 2024-09-16T03:53:45.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.556+0000 7f137ba4c640 1 -- 172.21.15.42:0/21746962 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f13741337c0 con 0x7f13740a2d60 2024-09-16T03:53:45.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.556+0000 7f1369ffb640 1 -- 172.21.15.42:0/21746962 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f13640bc5c0 con 0x7f13600039c0 2024-09-16T03:53:45.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.556+0000 7f1378fe7640 1 --2- 172.21.15.42:0/21746962 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f13740a2d60 0x7f13740a3140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:53:45.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.556+0000 7f1378fe7640 10 monclient: get_auth_request con 0x7f13740a2d60 auth_method 0 2024-09-16T03:53:45.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.556+0000 7f1378fe7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:53:45.588 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.556+0000 7f1378fe7640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:53:45.588 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.556+0000 7f1378fe7640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:53:45.588 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.556+0000 7f1378fe7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:53:45.588 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.556+0000 7f1378fe7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:53:45.588 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.556+0000 7f1378fe7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:53:45.588 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.556+0000 7f1369ffb640 1 --2- 172.21.15.42:0/21746962 >> v2:172.21.15.154:6812/254845342 conn(0x7f1358037810 0x7f1358039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:53:45.588 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.556+0000 7f1369ffb640 1 -- 172.21.15.42:0/21746962 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(366..366 src has 286..366) ==== 5713+0+0 (secure 0 0 0) 0x7f13640e32a0 con 0x7f13600039c0 2024-09-16T03:53:45.588 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.556+0000 7f1369ffb640 10 client.824391.objecter ms_dispatch 0x7f1374000ba0 osd_map(366..366 src has 286..366) 2024-09-16T03:53:45.588 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.556+0000 7f1378fe7640 10 monclient(hunting): handle_auth_done global_id 834239 payload 824 2024-09-16T03:53:45.588 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.556+0000 7f1369ffb640 3 client.824391.objecter handle_osd_map got epochs [366,366] > 0 2024-09-16T03:53:45.588 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.556+0000 7f1369ffb640 3 client.824391.objecter handle_osd_map decoding full epoch 366 2024-09-16T03:53:45.588 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.556+0000 7f1378fe7640 1 -- 172.21.15.42:0/21746962 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1374133290 con 0x7f13740a2d60 2024-09-16T03:53:45.589 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.556+0000 7f1369ffb640 20 client.824391.objecter dump_active .. 0 homeless 2024-09-16T03:53:45.589 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.556+0000 7f1369ffb640 10 client.824391.objecter ms_handle_connect 0x7f13740a2d60 2024-09-16T03:53:45.589 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.556+0000 7f1369ffb640 10 client.824391.objecter resend_mon_ops 2024-09-16T03:53:45.589 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.556+0000 7f1378fe7640 1 --2- 172.21.15.42:0/21746962 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f13740a2d60 0x7f13740a3140 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f1370000c10 tx=0x7f1370011a50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:53:45.589 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.556+0000 7f1379fe9640 1 --2- 172.21.15.42:0/21746962 >> v2:172.21.15.154:6812/254845342 conn(0x7f1358037810 0x7f1358039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:53:45.589 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.556+0000 7f1379fe9640 10 monclient: get_auth_request con 0x7f1358037810 auth_method 0 2024-09-16T03:53:45.589 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.556+0000 7f1369ffb640 1 -- 172.21.15.42:0/21746962 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13700053a0 con 0x7f13740a2d60 2024-09-16T03:53:45.589 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.556+0000 7f1369ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:53:45.589 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.556+0000 7f1369ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:53:45.589 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.556+0000 7f1369ffb640 10 monclient: dump: 2024-09-16T03:53:45.589 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:53:45.590 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:53:45.590 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:45.590 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:45.590 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:53:45.590 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:53:45.590 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:53:45.590 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:53:45.590 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:53:45.590 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:53:45.590 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.556+0000 7f1369ffb640 1 -- 172.21.15.42:0/21746962 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f137001a070 con 0x7f13740a2d60 2024-09-16T03:53:45.590 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.556+0000 7f1369ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:53:45.590 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.556+0000 7f1369ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:53:45.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.556+0000 7f1369ffb640 1 -- 172.21.15.42:0/21746962 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f13740a2d60 msgr2=0x7f13740a3140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:53:45.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.556+0000 7f1369ffb640 1 --2- 172.21.15.42:0/21746962 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f13740a2d60 0x7f13740a3140 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f1370000c10 tx=0x7f1370011a50 comp rx=0 tx=0).stop 2024-09-16T03:53:45.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.556+0000 7f1369ffb640 1 -- 172.21.15.42:0/21746962 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f13740a2d60 msgr2=0x7f13740a3140 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:53:45.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.556+0000 7f1369ffb640 1 --2- 172.21.15.42:0/21746962 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f13740a2d60 0x7f13740a3140 secure :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f1370000c10 tx=0x7f1370011a50 comp rx=0 tx=0).stop 2024-09-16T03:53:45.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.556+0000 7f1369ffb640 10 client.824391.objecter ms_handle_connect 0x7f1358037810 2024-09-16T03:53:45.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.556+0000 7f1379fe9640 1 --2- 172.21.15.42:0/21746962 >> v2:172.21.15.154:6812/254845342 conn(0x7f1358037810 0x7f1358039cd0 secure :-1 s=READY pgs=3606 cs=0 l=1 rev1=1 crypto rx=0x7f13740a4270 tx=0x7f136c006040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:53:45.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.564+0000 7f137ba4c640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:53:45.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.564+0000 7f137ba4c640 1 --2- 172.21.15.42:0/21746962 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f137409e950 0x7f137409ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:53:45.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.564+0000 7f137ba4c640 10 monclient(hunting): start opening mon connection 2024-09-16T03:53:45.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.564+0000 7f137ba4c640 1 -- 172.21.15.42:0/21746962 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f137413a9a0 con 0x7f137409e950 2024-09-16T03:53:45.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.564+0000 7f1378fe7640 1 --2- 172.21.15.42:0/21746962 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f137409e950 0x7f137409ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:53:45.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.564+0000 7f1378fe7640 10 monclient: get_auth_request con 0x7f137409e950 auth_method 0 2024-09-16T03:53:45.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.564+0000 7f1378fe7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:53:45.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.564+0000 7f1378fe7640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:53:45.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.564+0000 7f1378fe7640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:53:45.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.564+0000 7f1378fe7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:53:45.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.564+0000 7f1378fe7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:53:45.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.564+0000 7f1378fe7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:53:45.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.564+0000 7f1378fe7640 10 monclient(hunting): handle_auth_done global_id 834242 payload 824 2024-09-16T03:53:45.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.564+0000 7f1378fe7640 1 -- 172.21.15.42:0/21746962 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f13740a5750 con 0x7f137409e950 2024-09-16T03:53:45.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.564+0000 7f1378fe7640 1 --2- 172.21.15.42:0/21746962 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f137409e950 0x7f137409ed60 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f1370000c10 tx=0x7f1370001130 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:53:45.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.564+0000 7f1369ffb640 10 client.824391.objecter ms_handle_connect 0x7f137409e950 2024-09-16T03:53:45.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.564+0000 7f1369ffb640 10 client.824391.objecter resend_mon_ops 2024-09-16T03:53:45.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.564+0000 7f1369ffb640 1 -- 172.21.15.42:0/21746962 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1370005470 con 0x7f137409e950 2024-09-16T03:53:45.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.564+0000 7f1369ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:53:45.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.564+0000 7f1369ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:53:45.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.564+0000 7f1369ffb640 10 monclient: dump: 2024-09-16T03:53:45.594 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:53:45.594 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:53:45.595 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:45.595 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:45.595 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:53:45.595 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:53:45.595 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:53:45.595 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:53:45.595 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:53:45.595 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:53:45.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.564+0000 7f1369ffb640 1 -- 172.21.15.42:0/21746962 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4712 (secure 0 0 0) 0x7f1370005470 con 0x7f137409e950 2024-09-16T03:53:45.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.564+0000 7f1369ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:53:45.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.564+0000 7f1369ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:53:45.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.564+0000 7f1369ffb640 1 -- 172.21.15.42:0/21746962 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f137409e950 msgr2=0x7f137409ed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:53:45.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.564+0000 7f1369ffb640 1 --2- 172.21.15.42:0/21746962 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f137409e950 0x7f137409ed60 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f1370000c10 tx=0x7f1370001130 comp rx=0 tx=0).stop 2024-09-16T03:53:45.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.564+0000 7f1369ffb640 1 -- 172.21.15.42:0/21746962 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f137409e950 msgr2=0x7f137409ed60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:53:45.596 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.564+0000 7f1369ffb640 1 --2- 172.21.15.42:0/21746962 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f137409e950 0x7f137409ed60 secure :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f1370000c10 tx=0x7f1370001130 comp rx=0 tx=0).stop 2024-09-16T03:53:45.596 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:53:45.596 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:53:45.596 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:53:45.596 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T03:53:45.596 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 100, 2024-09-16T03:53:45.596 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 65135, 2024-09-16T03:53:45.596 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:53:45.596 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:53:45.596 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T03:53:45.597 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T03:53:45.597 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:53:45.597 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 5, 2024-09-16T03:53:45.597 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:53:45.597 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:53:45.597 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:53:45.597 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:53:45.597 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:53:45.597 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:53:45.597 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:53:45.597 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:53:45.597 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:53:45.597 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:53:45.597 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:53:45.598 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:53:45.598 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:53:45.598 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:53:45.598 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:53:45.598 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:53:45.598 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:53:45.598 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:53:45.598 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:53:45.598 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:53:45.598 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:53:45.598 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:53:45.598 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:53:45.598 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:53:45.599 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:53:45.599 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:53:45.599 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:53:45.599 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:53:45.599 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:53:45.599 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:53:45.599 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:53:45.599 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:53:45.599 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:53:45.599 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:53:45.599 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:53:45.599 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:53:45.599 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:53:45.599 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:53:45.600 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:53:45.600 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:53:45.600 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:53:45.600 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:53:45.600 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:53:45.600 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:53:45.600 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:53:45.600 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:53:45.600 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:53:45.600 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:53:45.600 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:53:45.600 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:53:45.600 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:53:45.600 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:53:45.601 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:53:45.601 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:53:45.601 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:53:45.601 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:53:45.601 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:53:45.601 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:53:45.601 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:53:45.601 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:53:45.601 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:53:45.601 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:53:45.601 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:53:45.601 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:53:45.601 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:53:45.601 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:53:45.602 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:53:45.602 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:53:45.602 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:53:45.602 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:53:45.602 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:53:45.602 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:53:45.602 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:53:45.602 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:53:45.602 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:53:45.602 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:53:45.602 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:53:45.602 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:53:45.602 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:53:45.602 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:53:45.603 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:53:45.603 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:53:45.603 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:53:45.603 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:53:45.603 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:53:45.603 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:53:45.603 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:53:45.603 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:53:45.603 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:53:45.603 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:53:45.603 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:53:45.603 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:53:45.603 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:53:45.603 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:53:45.603 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:53:45.604 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:53:45.604 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:53:45.604 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:53:45.604 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:53:45.604 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:53:45.604 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:53:45.604 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:53:45.604 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:53:45.604 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:53:45.604 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:53:45.604 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:53:45.604 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:53:45.604 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:53:45.605 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:53:45.605 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:53:45.605 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:53:45.605 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:53:45.605 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:53:45.605 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:53:45.605 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:53:45.605 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:53:45.605 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:53:45.605 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:53:45.605 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:53:45.605 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:53:45.605 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:53:45.605 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:53:45.606 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:53:45.606 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:53:45.606 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:53:45.606 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:53:45.606 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:53:45.606 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:53:45.606 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:53:45.606 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:53:45.606 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:53:45.606 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:53:45.606 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:53:45.606 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:53:45.606 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:53:45.606 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:53:45.607 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:53:45.607 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:53:45.607 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:53:45.607 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:53:45.607 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:53:45.607 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T03:53:45.607 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:53:45.607 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:53:45.607 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:53:45.607 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:53:45.607 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:53:45.607 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:53:45.607 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:53:45.607 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:53:45.607 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:53:45.608 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:53:45.608 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 2 2024-09-16T03:53:45.608 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:53:45.608 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:53:45.608 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:53:45.608 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:53:45.608 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:53:45.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.564+0000 7f137ba4c640 10 librados: watch_flush enter 2024-09-16T03:53:45.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.564+0000 7f137ba4c640 10 librados: watch_flush exit 2024-09-16T03:53:45.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.564+0000 7f137ba4c640 20 client.824391.objecter shutdown clearing up homeless session... 2024-09-16T03:53:45.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.564+0000 7f137ba4c640 10 client.824391.objecter successfully canceled tick 2024-09-16T03:53:45.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.564+0000 7f137ba4c640 1 -- 172.21.15.42:0/21746962 >> v2:172.21.15.154:6812/254845342 conn(0x7f1358037810 msgr2=0x7f1358039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:53:45.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.564+0000 7f137ba4c640 1 --2- 172.21.15.42:0/21746962 >> v2:172.21.15.154:6812/254845342 conn(0x7f1358037810 0x7f1358039cd0 secure :-1 s=READY pgs=3606 cs=0 l=1 rev1=1 crypto rx=0x7f13740a4270 tx=0x7f136c006040 comp rx=0 tx=0).stop 2024-09-16T03:53:45.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.564+0000 7f137ba4c640 10 monclient: shutdown 2024-09-16T03:53:45.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.564+0000 7f137ba4c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:53:45.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.564+0000 7f137ba4c640 1 -- 172.21.15.42:0/21746962 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f13600039c0 msgr2=0x7f13741cfc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:53:45.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.564+0000 7f137ba4c640 1 --2- 172.21.15.42:0/21746962 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f13600039c0 0x7f13741cfc20 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f136401c780 tx=0x7f13640196f0 comp rx=0 tx=0).stop 2024-09-16T03:53:45.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.564+0000 7f1379fe9640 1 -- 172.21.15.42:0/21746962 reap_dead start 2024-09-16T03:53:45.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.564+0000 7f1379fe9640 1 -- 172.21.15.42:0/21746962 reap_dead start 2024-09-16T03:53:45.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.564+0000 7f137ba4c640 1 -- 172.21.15.42:0/21746962 shutdown_connections 2024-09-16T03:53:45.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.564+0000 7f137ba4c640 1 -- 172.21.15.42:0/21746962 >> 172.21.15.42:0/21746962 conn(0x7f13740a9fb0 msgr2=0x7f13740a7ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:53:45.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.564+0000 7f137ba4c640 1 -- 172.21.15.42:0/21746962 shutdown_connections 2024-09-16T03:53:45.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.564+0000 7f137ba4c640 1 -- 172.21.15.42:0/21746962 wait complete. 2024-09-16T03:53:45.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.564+0000 7f137ba4c640 1 librados: shutdown 2024-09-16T03:53:45.611 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:53:45.611 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T03:53:45.612 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T03:53:45.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.724+0000 7f74e3d65640 10 monclient: get_monmap_and_config 2024-09-16T03:53:45.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.724+0000 7f74e3d65640 10 monclient: build_initial_monmap 2024-09-16T03:53:45.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.724+0000 7f74e3d65640 10 monclient: monmap: 2024-09-16T03:53:45.731 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:53:45.731 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:53:45.731 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:53:45.729176+0000 2024-09-16T03:53:45.731 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:53:45.729176+0000 2024-09-16T03:53:45.731 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:53:45.731 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:53:45.731 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:53:45.731 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:53:45.731 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:53:45.731 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:53:45.731 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:53:45.731 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:53:45.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.724+0000 7f74e3d65640 1 Processor -- start 2024-09-16T03:53:45.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.724+0000 7f74e3d65640 1 -- start start 2024-09-16T03:53:45.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.724+0000 7f74e3d65640 10 monclient: init 2024-09-16T03:53:45.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.724+0000 7f74e3d65640 10 monclient: _reopen_session rank -1 2024-09-16T03:53:45.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.724+0000 7f74e3d65640 10 monclient: _add_conns ranks=[4,1,3,2,0] 2024-09-16T03:53:45.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.724+0000 7f74e3d65640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f74dc0a1330 0x7f74dc0a1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:53:45.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.724+0000 7f74e3d65640 10 monclient(hunting): picked mon.noname-c con 0x7f74dc0a1330 addr v2:172.21.15.154:3300/0 2024-09-16T03:53:45.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.724+0000 7f74e3d65640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f74dc06dac0 addr v1:172.21.15.154:6789/0 2024-09-16T03:53:45.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.724+0000 7f74e3d65640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f74dc0a1c50 0x7f74dc135810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:53:45.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.724+0000 7f74e3d65640 10 monclient(hunting): picked mon.noname-a con 0x7f74dc0a1c50 addr v2:172.21.15.42:3300/0 2024-09-16T03:53:45.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.724+0000 7f74e3d65640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f74dc09c090 con 0x7f74dc06dac0 2024-09-16T03:53:45.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.724+0000 7f74e3d65640 10 monclient(hunting): start opening mon connection 2024-09-16T03:53:45.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.724+0000 7f74e3d65640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f74dc09f610 con 0x7f74dc0a1c50 2024-09-16T03:53:45.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.724+0000 7f74e3d65640 10 monclient(hunting): start opening mon connection 2024-09-16T03:53:45.736 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.724+0000 7f74e3d65640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f74dc09f8f0 con 0x7f74dc0a1330 2024-09-16T03:53:45.736 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.724+0000 7f74e3d65640 10 monclient(hunting): _renew_subs 2024-09-16T03:53:45.736 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.724+0000 7f74e3d65640 10 monclient(hunting): authenticate will time out at 1672.698619s 2024-09-16T03:53:45.736 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.724+0000 7f74e2302640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f74dc0a1c50 0x7f74dc135810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:53:45.736 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.724+0000 7f74e2302640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f74dc0a1c50 0x7f74dc135810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:53794/0 (socket says 172.21.15.42:53794) 2024-09-16T03:53:45.736 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.724+0000 7f74e2302640 1 -- 172.21.15.42:0/1506549582 learned_addr learned my addr 172.21.15.42:0/1506549582 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:53:45.736 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.724+0000 7f74e2302640 10 monclient(hunting): get_auth_request con 0x7f74dc0a1c50 auth_method 0 2024-09-16T03:53:45.736 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.724+0000 7f74e2302640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:53:45.736 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.724+0000 7f74e2302640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:53:45.736 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.724+0000 7f74e2302640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:53:45.736 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.724+0000 7f74e2302640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:53:45.736 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.724+0000 7f74e2302640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:53:45.736 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.724+0000 7f74e2302640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:53:45.737 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.724+0000 7f74e1b01640 1 --2- 172.21.15.42:0/1506549582 >> v2:172.21.15.154:3300/0 conn(0x7f74dc0a1330 0x7f74dc0a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:53:45.737 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.724+0000 7f74e1b01640 10 monclient(hunting): get_auth_request con 0x7f74dc0a1330 auth_method 0 2024-09-16T03:53:45.737 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.724+0000 7f74e1b01640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:53:45.737 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.724+0000 7f74e1b01640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:53:45.737 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.724+0000 7f74e1b01640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:53:45.737 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.724+0000 7f74e2302640 10 monclient(hunting): handle_auth_done global_id 824394 payload 274 2024-09-16T03:53:45.737 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.724+0000 7f74e2302640 1 -- 172.21.15.42:0/1506549582 >> v2:172.21.15.154:3300/0 conn(0x7f74dc0a1330 msgr2=0x7f74dc0a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:53:45.737 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.724+0000 7f74e2302640 1 --2- 172.21.15.42:0/1506549582 >> v2:172.21.15.154:3300/0 conn(0x7f74dc0a1330 0x7f74dc0a1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:53:45.737 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.724+0000 7f74e2302640 1 -- 172.21.15.42:0/1506549582 >> v1:172.21.15.154:6789/0 conn(0x7f74dc06dac0 legacy=0x7f74dc135100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:53:45.737 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.724+0000 7f74e2302640 10 monclient: _finish_hunting 0 2024-09-16T03:53:45.737 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.724+0000 7f74e2302640 1 monclient: found mon.noname-a 2024-09-16T03:53:45.737 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.724+0000 7f74e2302640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:53:45.737 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.724+0000 7f74e2302640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:53:45.738 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.724+0000 7f74e2302640 1 -- 172.21.15.42:0/1506549582 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74dc09fbd0 con 0x7f74dc0a1c50 2024-09-16T03:53:45.738 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.728+0000 7f74e2302640 1 --2- 172.21.15.42:0/1506549582 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f74dc0a1c50 0x7f74dc135810 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f74dc06add0 tx=0x7f74d401c870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=137bb8edf166597 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:53:45.738 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.728+0000 7f74e0aff640 1 -- 172.21.15.42:0/1506549582 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74d40a5750 con 0x7f74dc0a1c50 2024-09-16T03:53:45.738 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.728+0000 7f74e0aff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:53:45.738 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.728+0000 7f74e0aff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:53:45.738 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.728+0000 7f74e0aff640 10 monclient: dump: 2024-09-16T03:53:45.738 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:53:45.738 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:53:45.738 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:45.738 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:45.738 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:53:45.738 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:53:45.738 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:53:45.739 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:53:45.739 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:53:45.739 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:53:45.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.728+0000 7f74e0aff640 10 monclient: _finish_auth 0 2024-09-16T03:53:45.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.728+0000 7f74e0aff640 10 monclient: _check_auth_tickets 2024-09-16T03:53:45.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.728+0000 7f74e0aff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:53:45.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.728+0000 7f74e0aff640 1 -- 172.21.15.42:0/1506549582 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f74d40a58f0 con 0x7f74dc0a1c50 2024-09-16T03:53:45.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.728+0000 7f74e3d65640 5 monclient: authenticate success, global_id 824394 2024-09-16T03:53:45.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.728+0000 7f74e3d65640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:53:45.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.728+0000 7f74e0aff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:53:45.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.728+0000 7f74e0aff640 1 -- 172.21.15.42:0/1506549582 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74d40a5cc0 con 0x7f74dc0a1c50 2024-09-16T03:53:45.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.728+0000 7f74e3d65640 10 monclient: get_monmap_and_config success 2024-09-16T03:53:45.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.728+0000 7f74e3d65640 4 set_mon_vals no callback set 2024-09-16T03:53:45.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.728+0000 7f74e0aff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:53:45.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.728+0000 7f74e0aff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:53:45.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.728+0000 7f74e0aff640 10 monclient: dump: 2024-09-16T03:53:45.740 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:53:45.740 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:53:45.740 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:45.740 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:45.740 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:53:45.740 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:53:45.741 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:53:45.741 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:53:45.741 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:53:45.741 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:53:45.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.728+0000 7f74e3d65640 10 monclient: shutdown 2024-09-16T03:53:45.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.728+0000 7f74e3d65640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:53:45.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.728+0000 7f74e3d65640 1 -- 172.21.15.42:0/1506549582 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f74dc0a1c50 msgr2=0x7f74dc135810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:53:45.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.728+0000 7f74e3d65640 1 --2- 172.21.15.42:0/1506549582 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f74dc0a1c50 0x7f74dc135810 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f74dc06add0 tx=0x7f74d401c870 comp rx=0 tx=0).stop 2024-09-16T03:53:45.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.728+0000 7f74e3d65640 1 -- 172.21.15.42:0/1506549582 shutdown_connections 2024-09-16T03:53:45.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.728+0000 7f74e3d65640 1 --2- 172.21.15.42:0/1506549582 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f74dc0a1c50 0x7f74dc135810 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:53:45.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.728+0000 7f74e3d65640 1 --2- 172.21.15.42:0/1506549582 >> v2:172.21.15.154:3300/0 conn(0x7f74dc0a1330 0x7f74dc0a1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:53:45.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.728+0000 7f74e3d65640 1 -- 172.21.15.42:0/1506549582 >> 172.21.15.42:0/1506549582 conn(0x7f74dc0a2420 msgr2=0x7f74dc0a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:53:45.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.728+0000 7f74e3d65640 1 -- 172.21.15.42:0/1506549582 shutdown_connections 2024-09-16T03:53:45.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.728+0000 7f74e3d65640 1 -- 172.21.15.42:0/1506549582 wait complete. 2024-09-16T03:53:45.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.728+0000 7f74e3d65640 10 monclient: build_initial_monmap 2024-09-16T03:53:45.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.728+0000 7f74e3d65640 10 monclient: monmap: 2024-09-16T03:53:45.748 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:53:45.748 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:53:45.748 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:53:45.748 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:53:45.748 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:53:45.748 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:53:45.748 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:53:45.748 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:53:45.748 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:53:45.748 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:53:45.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.728+0000 7f74e3d65640 1 librados: starting msgr at 2024-09-16T03:53:45.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.728+0000 7f74e3d65640 1 librados: starting objecter 2024-09-16T03:53:45.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.728+0000 7f74e3d65640 1 Processor -- start 2024-09-16T03:53:45.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.728+0000 7f74e3d65640 1 -- start start 2024-09-16T03:53:45.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.728+0000 7f74e3d65640 1 librados: setting wanted keys 2024-09-16T03:53:45.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.728+0000 7f74e3d65640 1 librados: calling monclient init 2024-09-16T03:53:45.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.728+0000 7f74e3d65640 10 monclient: init 2024-09-16T03:53:45.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.728+0000 7f74e3d65640 10 monclient: _reopen_session rank -1 2024-09-16T03:53:45.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.728+0000 7f74e3d65640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:53:45.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.728+0000 7f74e3d65640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f74dc0a1330 0x7f74dc1cecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:53:45.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.728+0000 7f74e3d65640 10 monclient(hunting): picked mon.noname-c con 0x7f74dc0a1330 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:53:45.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.728+0000 7f74e3d65640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f74dc0a1c50 0x7f74dc135080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:53:45.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.728+0000 7f74e3d65640 10 monclient(hunting): picked mon.noname-a con 0x7f74dc0a1c50 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:53:45.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.728+0000 7f74e3d65640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f74dc1355c0 0x7f74dc11daa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:53:45.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.728+0000 7f74e3d65640 10 monclient(hunting): picked mon.noname-b con 0x7f74dc1355c0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:53:45.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.728+0000 7f74e3d65640 10 monclient(hunting): start opening mon connection 2024-09-16T03:53:45.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.728+0000 7f74e3d65640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f74dc135d50 con 0x7f74dc0a1c50 2024-09-16T03:53:45.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.728+0000 7f74e3d65640 10 monclient(hunting): start opening mon connection 2024-09-16T03:53:45.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.728+0000 7f74e3d65640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f74dc09ef80 con 0x7f74dc1355c0 2024-09-16T03:53:45.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.728+0000 7f74e3d65640 10 monclient(hunting): start opening mon connection 2024-09-16T03:53:45.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.728+0000 7f74e3d65640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f74dc135ed0 con 0x7f74dc0a1330 2024-09-16T03:53:45.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.728+0000 7f74e3d65640 10 monclient(hunting): _renew_subs 2024-09-16T03:53:45.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.728+0000 7f74e3d65640 10 monclient(hunting): authenticate will time out at 1672.703881s 2024-09-16T03:53:45.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.732+0000 7f74e1300640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f74dc0a1c50 0x7f74dc135080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:53:45.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.732+0000 7f74e1b01640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f74dc0a1330 0x7f74dc1cecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:53:45.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.732+0000 7f74e1300640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f74dc0a1c50 0x7f74dc135080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:53802/0 (socket says 172.21.15.42:53802) 2024-09-16T03:53:45.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.732+0000 7f74e1300640 1 -- 172.21.15.42:0/3766362936 learned_addr learned my addr 172.21.15.42:0/3766362936 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:53:45.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.732+0000 7f74e1300640 10 monclient(hunting): get_auth_request con 0x7f74dc0a1c50 auth_method 0 2024-09-16T03:53:45.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.732+0000 7f74e1300640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:53:45.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.732+0000 7f74e1300640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:53:45.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.732+0000 7f74e1300640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:53:45.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.732+0000 7f74e1b01640 10 monclient(hunting): get_auth_request con 0x7f74dc0a1330 auth_method 0 2024-09-16T03:53:45.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.732+0000 7f74e1b01640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:53:45.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.732+0000 7f74e1b01640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:53:45.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.732+0000 7f74e1b01640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:53:45.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.732+0000 7f74e2302640 1 --2- 172.21.15.42:0/3766362936 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f74dc1355c0 0x7f74dc11daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:53:45.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.732+0000 7f74e1b01640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:53:45.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.732+0000 7f74e1b01640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:53:45.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.732+0000 7f74e1b01640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:53:45.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.732+0000 7f74e1300640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:53:45.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.732+0000 7f74e1300640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:53:45.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.732+0000 7f74e1300640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:53:45.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.732+0000 7f74e2302640 10 monclient(hunting): get_auth_request con 0x7f74dc1355c0 auth_method 0 2024-09-16T03:53:45.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.732+0000 7f74e2302640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:53:45.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.732+0000 7f74e2302640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:53:45.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.732+0000 7f74e2302640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:53:45.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.732+0000 7f74e1b01640 10 monclient(hunting): handle_auth_done global_id 834245 payload 824 2024-09-16T03:53:45.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.732+0000 7f74e1b01640 1 -- 172.21.15.42:0/3766362936 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f74dc1355c0 msgr2=0x7f74dc11daa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:53:45.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.732+0000 7f74e1b01640 1 --2- 172.21.15.42:0/3766362936 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f74dc1355c0 0x7f74dc11daa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:53:45.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.732+0000 7f74e1b01640 1 -- 172.21.15.42:0/3766362936 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f74dc0a1c50 msgr2=0x7f74dc135080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:53:45.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.732+0000 7f74e1b01640 1 --2- 172.21.15.42:0/3766362936 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f74dc0a1c50 0x7f74dc135080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:53:45.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.732+0000 7f74e1b01640 10 monclient: _finish_hunting 0 2024-09-16T03:53:45.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.732+0000 7f74e1b01640 1 monclient: found mon.noname-c 2024-09-16T03:53:45.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.732+0000 7f74e1b01640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:53:45.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.732+0000 7f74e1b01640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:53:45.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.732+0000 7f74e1b01640 1 -- 172.21.15.42:0/3766362936 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74dc11fbe0 con 0x7f74dc0a1330 2024-09-16T03:53:45.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.732+0000 7f74e1300640 1 --2- 172.21.15.42:0/3766362936 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f74dc0a1c50 0x7f74dc135080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:53:45.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.732+0000 7f74e2302640 1 --2- 172.21.15.42:0/3766362936 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f74dc1355c0 0x7f74dc11daa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T03:53:45.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.732+0000 7f74e1b01640 1 --2- 172.21.15.42:0/3766362936 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f74dc0a1330 0x7f74dc1cecd0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f74d800d5f0 tx=0x7f74d800dab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:53:45.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.732+0000 7f74d27fc640 10 client.?.objecter ms_handle_connect 0x7f74dc0a1330 2024-09-16T03:53:45.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.732+0000 7f74d27fc640 10 client.?.objecter resend_mon_ops 2024-09-16T03:53:45.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.732+0000 7f74d27fc640 1 -- 172.21.15.42:0/3766362936 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74d80125f0 con 0x7f74dc0a1330 2024-09-16T03:53:45.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.732+0000 7f74d27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:53:45.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.732+0000 7f74d27fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:53:45.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.732+0000 7f74d27fc640 10 monclient: dump: 2024-09-16T03:53:45.753 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:53:45.753 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:53:45.753 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:45.753 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:45.754 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:53:45.754 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:53:45.754 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:53:45.754 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:53:45.754 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:53:45.754 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:53:45.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.732+0000 7f74d27fc640 10 monclient: _finish_auth 0 2024-09-16T03:53:45.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.732+0000 7f74d27fc640 10 monclient: _check_auth_tickets 2024-09-16T03:53:45.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.732+0000 7f74d27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:53:45.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.732+0000 7f74d27fc640 1 -- 172.21.15.42:0/3766362936 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f74d8019070 con 0x7f74dc0a1330 2024-09-16T03:53:45.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.732+0000 7f74e3d65640 5 monclient: authenticate success, global_id 834245 2024-09-16T03:53:45.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.732+0000 7f74e3d65640 10 monclient: _renew_subs 2024-09-16T03:53:45.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.732+0000 7f74e3d65640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:53:45.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.732+0000 7f74e3d65640 1 -- 172.21.15.42:0/3766362936 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74dc0712c0 con 0x7f74dc0a1330 2024-09-16T03:53:45.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.732+0000 7f74d27fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:53:45.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.732+0000 7f74d27fc640 1 -- 172.21.15.42:0/3766362936 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74d80129d0 con 0x7f74dc0a1330 2024-09-16T03:53:45.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.732+0000 7f74d27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:53:45.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.732+0000 7f74d27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:53:45.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.732+0000 7f74d27fc640 10 monclient: dump: 2024-09-16T03:53:45.755 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:53:45.755 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:53:45.755 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:45.755 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:45.755 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:53:45.755 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:53:45.755 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:53:45.755 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:53:45.756 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:53:45.756 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:53:45.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.732+0000 7f74d3fff640 4 set_mon_vals no callback set 2024-09-16T03:53:45.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.732+0000 7f74e3d65640 10 client.834245.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:53:45.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.732+0000 7f74e3d65640 10 monclient: _renew_subs 2024-09-16T03:53:45.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.732+0000 7f74e3d65640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:53:45.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.732+0000 7f74e3d65640 1 -- 172.21.15.42:0/3766362936 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f74dc120020 con 0x7f74dc0a1330 2024-09-16T03:53:45.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.732+0000 7f74e3d65640 1 librados: init done 2024-09-16T03:53:45.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.732+0000 7f74e3d65640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:53:45.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.732+0000 7f74e3d65640 1 --2- 172.21.15.42:0/3766362936 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f74dc06dac0 0x7f74dc121190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:53:45.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.732+0000 7f74e3d65640 10 monclient(hunting): start opening mon connection 2024-09-16T03:53:45.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.732+0000 7f74e3d65640 1 -- 172.21.15.42:0/3766362936 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f74dc06e7a0 con 0x7f74dc06dac0 2024-09-16T03:53:45.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.732+0000 7f74d27fc640 1 -- 172.21.15.42:0/3766362936 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f74d8012b70 con 0x7f74dc0a1330 2024-09-16T03:53:45.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.732+0000 7f74e1300640 1 --2- 172.21.15.42:0/3766362936 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f74dc06dac0 0x7f74dc121190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:53:45.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.732+0000 7f74e1300640 10 monclient: get_auth_request con 0x7f74dc06dac0 auth_method 0 2024-09-16T03:53:45.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.732+0000 7f74e1300640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:53:45.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.732+0000 7f74e1300640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:53:45.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.732+0000 7f74e1300640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:53:45.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.732+0000 7f74d27fc640 1 --2- 172.21.15.42:0/3766362936 >> v2:172.21.15.154:6812/254845342 conn(0x7f74c0037870 0x7f74c0039d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:53:45.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.732+0000 7f74d27fc640 1 -- 172.21.15.42:0/3766362936 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(366..366 src has 286..366) ==== 5713+0+0 (secure 0 0 0) 0x7f74d8049270 con 0x7f74dc0a1330 2024-09-16T03:53:45.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.732+0000 7f74d27fc640 10 client.834245.objecter ms_dispatch 0x7f74dc000ba0 osd_map(366..366 src has 286..366) 2024-09-16T03:53:45.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.732+0000 7f74d27fc640 3 client.834245.objecter handle_osd_map got epochs [366,366] > 0 2024-09-16T03:53:45.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.732+0000 7f74d27fc640 3 client.834245.objecter handle_osd_map decoding full epoch 366 2024-09-16T03:53:45.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.732+0000 7f74e1300640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:53:45.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.732+0000 7f74e1300640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:53:45.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.732+0000 7f74e1300640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:53:45.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.732+0000 7f74d27fc640 20 client.834245.objecter dump_active .. 0 homeless 2024-09-16T03:53:45.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.732+0000 7f74e2302640 1 --2- 172.21.15.42:0/3766362936 >> v2:172.21.15.154:6812/254845342 conn(0x7f74c0037870 0x7f74c0039d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:53:45.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.732+0000 7f74e2302640 10 monclient: get_auth_request con 0x7f74c0037870 auth_method 0 2024-09-16T03:53:45.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.732+0000 7f74e1300640 10 monclient(hunting): handle_auth_done global_id 844141 payload 824 2024-09-16T03:53:45.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.732+0000 7f74e1300640 1 -- 172.21.15.42:0/3766362936 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f74dc0a1710 con 0x7f74dc06dac0 2024-09-16T03:53:45.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.732+0000 7f74e1300640 1 --2- 172.21.15.42:0/3766362936 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f74dc06dac0 0x7f74dc121190 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f74dc1210c0 tx=0x7f74cc0045a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:53:45.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.732+0000 7f74d27fc640 10 client.834245.objecter ms_handle_connect 0x7f74dc06dac0 2024-09-16T03:53:45.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.732+0000 7f74d27fc640 10 client.834245.objecter resend_mon_ops 2024-09-16T03:53:45.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.732+0000 7f74d27fc640 10 client.834245.objecter ms_handle_connect 0x7f74c0037870 2024-09-16T03:53:45.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.732+0000 7f74e2302640 1 --2- 172.21.15.42:0/3766362936 >> v2:172.21.15.154:6812/254845342 conn(0x7f74c0037870 0x7f74c0039d30 secure :-1 s=READY pgs=3607 cs=0 l=1 rev1=1 crypto rx=0x7f74d40ae000 tx=0x7f74d40a7070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:53:45.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.732+0000 7f74d27fc640 1 -- 172.21.15.42:0/3766362936 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74cc018030 con 0x7f74dc06dac0 2024-09-16T03:53:45.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.732+0000 7f74d27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:53:45.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.732+0000 7f74d27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:53:45.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.732+0000 7f74d27fc640 10 monclient: dump: 2024-09-16T03:53:45.759 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:53:45.759 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:53:45.759 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:45.759 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:45.759 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:53:45.759 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:53:45.759 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:53:45.759 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:53:45.759 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:53:45.759 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:53:45.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.736+0000 7f74d27fc640 1 -- 172.21.15.42:0/3766362936 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f74cc00db30 con 0x7f74dc06dac0 2024-09-16T03:53:45.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.736+0000 7f74d27fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:53:45.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.736+0000 7f74d27fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:53:45.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.736+0000 7f74d27fc640 1 -- 172.21.15.42:0/3766362936 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f74dc06dac0 msgr2=0x7f74dc121190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:53:45.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.736+0000 7f74d27fc640 1 --2- 172.21.15.42:0/3766362936 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f74dc06dac0 0x7f74dc121190 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f74dc1210c0 tx=0x7f74cc0045a0 comp rx=0 tx=0).stop 2024-09-16T03:53:45.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.736+0000 7f74d27fc640 1 -- 172.21.15.42:0/3766362936 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f74dc06dac0 msgr2=0x7f74dc121190 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:53:45.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.736+0000 7f74d27fc640 1 --2- 172.21.15.42:0/3766362936 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f74dc06dac0 0x7f74dc121190 secure :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f74dc1210c0 tx=0x7f74cc0045a0 comp rx=0 tx=0).stop 2024-09-16T03:53:45.761 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.740+0000 7f74e3d65640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:53:45.761 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.740+0000 7f74e3d65640 1 --2- 172.21.15.42:0/3766362936 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f74dc11dfe0 0x7f74dc11e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:53:45.761 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.740+0000 7f74e3d65640 10 monclient(hunting): start opening mon connection 2024-09-16T03:53:45.761 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.740+0000 7f74e3d65640 1 -- 172.21.15.42:0/3766362936 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f74dc11e930 con 0x7f74dc11dfe0 2024-09-16T03:53:45.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.740+0000 7f74e1300640 1 --2- 172.21.15.42:0/3766362936 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f74dc11dfe0 0x7f74dc11e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:53:45.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.740+0000 7f74e1300640 10 monclient: get_auth_request con 0x7f74dc11dfe0 auth_method 0 2024-09-16T03:53:45.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.740+0000 7f74e1300640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:53:45.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.740+0000 7f74e1300640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:53:45.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.740+0000 7f74e1300640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:53:45.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.740+0000 7f74e1300640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:53:45.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.740+0000 7f74e1300640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:53:45.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.740+0000 7f74e1300640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:53:45.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.740+0000 7f74e1300640 10 monclient(hunting): handle_auth_done global_id 844144 payload 824 2024-09-16T03:53:45.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.740+0000 7f74e1300640 1 -- 172.21.15.42:0/3766362936 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f74dc069040 con 0x7f74dc11dfe0 2024-09-16T03:53:45.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.740+0000 7f74e1300640 1 --2- 172.21.15.42:0/3766362936 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f74dc11dfe0 0x7f74dc11e3f0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f74cc018000 tx=0x7f74cc007f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:53:45.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.740+0000 7f74d27fc640 10 client.834245.objecter ms_handle_connect 0x7f74dc11dfe0 2024-09-16T03:53:45.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.740+0000 7f74d27fc640 10 client.834245.objecter resend_mon_ops 2024-09-16T03:53:45.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.744+0000 7f74d27fc640 1 -- 172.21.15.42:0/3766362936 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74cc00de70 con 0x7f74dc11dfe0 2024-09-16T03:53:45.763 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.744+0000 7f74d27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:53:45.763 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.744+0000 7f74d27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:53:45.763 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.744+0000 7f74d27fc640 10 monclient: dump: 2024-09-16T03:53:45.763 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:53:45.763 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:53:45.763 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:45.763 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:53:45.763 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:53:45.763 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:53:45.763 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:53:45.763 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:53:45.763 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:53:45.763 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:53:45.763 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.744+0000 7f74d27fc640 1 -- 172.21.15.42:0/3766362936 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4544 (secure 0 0 0) 0x7f74cc00de70 con 0x7f74dc11dfe0 2024-09-16T03:53:45.764 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.744+0000 7f74d27fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:53:45.764 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.744+0000 7f74d27fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:53:45.764 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.744+0000 7f74d27fc640 1 -- 172.21.15.42:0/3766362936 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f74dc11dfe0 msgr2=0x7f74dc11e3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:53:45.764 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.744+0000 7f74d27fc640 1 --2- 172.21.15.42:0/3766362936 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f74dc11dfe0 0x7f74dc11e3f0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f74cc018000 tx=0x7f74cc007f90 comp rx=0 tx=0).stop 2024-09-16T03:53:45.764 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.744+0000 7f74d27fc640 1 -- 172.21.15.42:0/3766362936 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f74dc11dfe0 msgr2=0x7f74dc11e3f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:53:45.764 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.744+0000 7f74d27fc640 1 --2- 172.21.15.42:0/3766362936 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f74dc11dfe0 0x7f74dc11e3f0 secure :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f74cc018000 tx=0x7f74cc007f90 comp rx=0 tx=0).stop 2024-09-16T03:53:45.764 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:53:45.764 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:53:45.764 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:53:45.764 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T03:53:45.765 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 100, 2024-09-16T03:53:45.765 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 9943, 2024-09-16T03:53:45.765 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:53:45.765 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:53:45.765 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T03:53:45.765 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T03:53:45.765 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:53:45.765 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 5, 2024-09-16T03:53:45.765 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:53:45.765 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:53:45.765 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:53:45.765 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:53:45.765 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:53:45.766 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:53:45.766 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:53:45.766 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:53:45.766 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:53:45.766 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:53:45.766 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:53:45.766 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:53:45.766 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:53:45.766 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:53:45.766 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:53:45.766 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:53:45.766 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:53:45.766 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:53:45.767 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:53:45.767 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:53:45.767 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:53:45.767 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:53:45.767 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:53:45.767 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:53:45.767 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:53:45.767 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:53:45.767 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:53:45.768 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:53:45.768 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:53:45.768 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:53:45.768 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:53:45.768 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:53:45.769 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:53:45.769 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:53:45.769 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:53:45.769 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:53:45.769 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:53:45.769 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:53:45.769 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:53:45.769 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:53:45.769 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:53:45.769 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:53:45.769 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:53:45.769 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:53:45.769 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:53:45.769 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:53:45.770 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:53:45.770 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:53:45.770 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:53:45.770 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:53:45.770 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:53:45.770 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:53:45.770 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:53:45.770 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:53:45.770 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:53:45.770 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:53:45.770 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:53:45.770 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:53:45.770 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:53:45.771 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:53:45.771 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:53:45.771 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:53:45.771 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:53:45.771 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:53:45.771 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:53:45.771 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:53:45.771 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:53:45.771 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:53:45.771 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:53:45.771 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:53:45.771 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:53:45.771 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:53:45.771 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:53:45.772 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:53:45.772 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:53:45.772 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:53:45.772 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:53:45.772 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:53:45.772 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:53:45.772 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:53:45.772 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:53:45.772 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:53:45.772 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:53:45.772 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:53:45.772 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:53:45.772 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:53:45.773 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:53:45.773 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:53:45.773 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:53:45.773 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:53:45.773 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:53:45.773 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:53:45.773 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:53:45.773 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:53:45.773 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:53:45.773 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:53:45.773 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:53:45.773 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:53:45.773 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:53:45.774 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:53:45.774 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:53:45.774 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:53:45.774 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:53:45.774 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:53:45.774 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:53:45.774 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:53:45.774 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:53:45.774 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:53:45.774 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:53:45.774 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:53:45.774 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:53:45.775 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:53:45.775 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:53:45.775 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:53:45.775 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:53:45.775 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:53:45.775 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:53:45.775 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:53:45.775 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:53:45.775 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:53:45.776 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:53:45.776 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:53:45.776 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:53:45.776 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:53:45.776 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:53:45.776 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:53:45.776 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:53:45.776 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:53:45.776 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:53:45.776 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:53:45.776 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:53:45.776 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:53:45.777 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:53:45.777 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:53:45.777 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:53:45.777 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:53:45.777 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:53:45.777 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:53:45.777 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:53:45.777 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:53:45.777 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:53:45.777 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:53:45.777 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:53:45.777 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:53:45.777 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:53:45.777 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:53:45.778 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:53:45.778 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:53:45.778 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:53:45.778 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:53:45.778 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:53:45.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.744+0000 7f74e3d65640 10 librados: watch_flush enter 2024-09-16T03:53:45.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.744+0000 7f74e3d65640 10 librados: watch_flush exit 2024-09-16T03:53:45.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.744+0000 7f74e3d65640 20 client.834245.objecter shutdown clearing up homeless session... 2024-09-16T03:53:45.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.744+0000 7f74e3d65640 10 client.834245.objecter successfully canceled tick 2024-09-16T03:53:45.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.744+0000 7f74e3d65640 1 -- 172.21.15.42:0/3766362936 >> v2:172.21.15.154:6812/254845342 conn(0x7f74c0037870 msgr2=0x7f74c0039d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:53:45.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.744+0000 7f74e3d65640 1 --2- 172.21.15.42:0/3766362936 >> v2:172.21.15.154:6812/254845342 conn(0x7f74c0037870 0x7f74c0039d30 secure :-1 s=READY pgs=3607 cs=0 l=1 rev1=1 crypto rx=0x7f74d40ae000 tx=0x7f74d40a7070 comp rx=0 tx=0).stop 2024-09-16T03:53:45.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.744+0000 7f74e3d65640 10 monclient: shutdown 2024-09-16T03:53:45.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.744+0000 7f74e3d65640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:53:45.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.744+0000 7f74e3d65640 1 -- 172.21.15.42:0/3766362936 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f74dc0a1330 msgr2=0x7f74dc1cecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:53:45.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.744+0000 7f74e2302640 1 -- 172.21.15.42:0/3766362936 reap_dead start 2024-09-16T03:53:45.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.744+0000 7f74e3d65640 1 --2- 172.21.15.42:0/3766362936 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f74dc0a1330 0x7f74dc1cecd0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f74d800d5f0 tx=0x7f74d800dab0 comp rx=0 tx=0).stop 2024-09-16T03:53:45.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.744+0000 7f74e3d65640 1 -- 172.21.15.42:0/3766362936 shutdown_connections 2024-09-16T03:53:45.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.744+0000 7f74e3d65640 1 --2- 172.21.15.42:0/3766362936 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f74dc0a1330 0x7f74dc1cecd0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:53:45.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.744+0000 7f74e3d65640 1 -- 172.21.15.42:0/3766362936 >> 172.21.15.42:0/3766362936 conn(0x7f74dc0a2420 msgr2=0x7f74dc06f920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:53:45.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.744+0000 7f74e3d65640 1 -- 172.21.15.42:0/3766362936 shutdown_connections 2024-09-16T03:53:45.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.744+0000 7f74e3d65640 1 -- 172.21.15.42:0/3766362936 wait complete. 2024-09-16T03:53:45.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:53:45.744+0000 7f74e3d65640 1 librados: shutdown 2024-09-16T03:53:45.782 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:53:45.782 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-09-16T03:53:45.782 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['b'] 2024-09-16T03:53:45.782 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-09-16T03:53:45.782 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-16T03:53:45.782 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-16T03:53:45.783 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T03:53:45.796 INFO:tasks.ceph.mon.c:Stopped 2024-09-16T03:53:45.796 INFO:tasks.mon_thrash:Sending STOP to mon b 2024-09-16T03:53:45.796 INFO:tasks.ceph.mon.b:Sent signal 19 2024-09-16T03:53:45.797 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-16T03:54:00.797 INFO:tasks.mon_thrash:Sending CONT to mon b 2024-09-16T03:54:00.798 INFO:tasks.ceph.mon.b:Sent signal 18 2024-09-16T03:54:00.798 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T03:54:00.798 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:54:00.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:00.924+0000 7f7781361640 10 monclient: get_monmap_and_config 2024-09-16T03:54:00.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:00.924+0000 7f7781361640 10 monclient: build_initial_monmap 2024-09-16T03:54:00.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:00.924+0000 7f7781361640 10 monclient: monmap: 2024-09-16T03:54:00.932 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:54:00.932 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:54:00.933 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:54:00.931010+0000 2024-09-16T03:54:00.933 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:54:00.931010+0000 2024-09-16T03:54:00.933 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:54:00.933 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:54:00.933 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:54:00.933 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:54:00.933 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:54:00.933 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:54:00.933 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:54:00.933 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:54:00.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:00.924+0000 7f7781361640 1 Processor -- start 2024-09-16T03:54:00.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:00.924+0000 7f7781361640 1 -- start start 2024-09-16T03:54:00.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:00.924+0000 7f7781361640 10 monclient: init 2024-09-16T03:54:00.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:00.928+0000 7f7781361640 10 monclient: _reopen_session rank -1 2024-09-16T03:54:00.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:00.928+0000 7f7781361640 10 monclient: _add_conns ranks=[4,2,0,1,3] 2024-09-16T03:54:00.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:00.928+0000 7f7781361640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f777c0a1330 0x7f777c09f180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:54:00.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:00.928+0000 7f7781361640 10 monclient(hunting): picked mon.noname-c con 0x7f777c0a1330 addr v2:172.21.15.154:3300/0 2024-09-16T03:54:00.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:00.928+0000 7f7781361640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f777c0a1710 0x7f777c0a1b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:54:00.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:00.928+0000 7f7781361640 10 monclient(hunting): picked mon.noname-b con 0x7f777c0a1710 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:54:00.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:00.928+0000 7f7781361640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f777c1340f0 addr v1:172.21.15.42:6789/0 2024-09-16T03:54:00.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:00.928+0000 7f7781361640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f777c09e4c0 con 0x7f777c1340f0 2024-09-16T03:54:00.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:00.928+0000 7f7781361640 10 monclient(hunting): start opening mon connection 2024-09-16T03:54:00.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:00.928+0000 7f7781361640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f777c09edf0 con 0x7f777c0a1330 2024-09-16T03:54:00.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:00.928+0000 7f7781361640 10 monclient(hunting): start opening mon connection 2024-09-16T03:54:00.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:00.928+0000 7f7781361640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f777c09cd60 con 0x7f777c0a1710 2024-09-16T03:54:00.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:00.928+0000 7f7781361640 10 monclient(hunting): _renew_subs 2024-09-16T03:54:00.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:00.928+0000 7f7781361640 10 monclient(hunting): authenticate will time out at 1687.900508s 2024-09-16T03:54:00.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:00.928+0000 7f777a7fc640 1 -- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f777c0a1710 msgr2=0x7f777c0a1b10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T03:54:00.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:00.928+0000 7f777a7fc640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f777c0a1710 0x7f777c0a1b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:54:00.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:00.928+0000 7f777b7fe640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f777c1340f0 0x7f777c1345a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:32972/0 (socket says 172.21.15.42:32972) 2024-09-16T03:54:00.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:00.928+0000 7f777b7fe640 1 -- 172.21.15.42:0/86996660 learned_addr learned my addr 172.21.15.42:0/86996660 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:54:00.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:00.928+0000 7f777affd640 1 --2- 172.21.15.42:0/86996660 >> v2:172.21.15.154:3300/0 conn(0x7f777c0a1330 0x7f777c09f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:54:00.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:00.928+0000 7f777affd640 10 monclient(hunting): get_auth_request con 0x7f777c0a1330 auth_method 0 2024-09-16T03:54:00.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:00.928+0000 7f777affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:54:00.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:00.928+0000 7f777affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:54:00.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:00.928+0000 7f777affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:54:00.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:00.928+0000 7f777affd640 1 -- 172.21.15.42:0/86996660 >> v2:172.21.15.154:3300/0 conn(0x7f777c0a1330 msgr2=0x7f777c09f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T03:54:00.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:00.928+0000 7f777affd640 1 -- 172.21.15.42:0/86996660 >> v2:172.21.15.154:3300/0 conn(0x7f777c0a1330 msgr2=0x7f777c09f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:54:00.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:00.928+0000 7f777affd640 1 --2- 172.21.15.42:0/86996660 >> v2:172.21.15.154:3300/0 conn(0x7f777c0a1330 0x7f777c09f180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:54:00.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:00.928+0000 7f777affd640 1 --2- 172.21.15.42:0/86996660 >> v2:172.21.15.154:3300/0 conn(0x7f777c0a1330 0x7f777c09f180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:54:01.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:01.132+0000 7f777a7fc640 1 -- 172.21.15.42:0/86996660 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f777c0a1710 msgr2=0x7f777c0a1b10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T03:54:01.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:01.132+0000 7f777a7fc640 1 --2- 172.21.15.42:0/86996660 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f777c0a1710 0x7f777c0a1b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T03:54:01.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:01.132+0000 7f777affd640 1 --2- 172.21.15.42:0/86996660 >> v2:172.21.15.154:3300/0 conn(0x7f777c0a1330 0x7f777c09f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:54:01.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:01.132+0000 7f777affd640 10 monclient(hunting): get_auth_request con 0x7f777c0a1330 auth_method 0 2024-09-16T03:54:01.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:01.132+0000 7f777affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:54:01.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:01.132+0000 7f777affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:54:01.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:01.132+0000 7f777affd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:54:01.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:01.132+0000 7f777affd640 1 -- 172.21.15.42:0/86996660 >> v2:172.21.15.154:3300/0 conn(0x7f777c0a1330 msgr2=0x7f777c09f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T03:54:01.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:01.132+0000 7f777affd640 1 -- 172.21.15.42:0/86996660 >> v2:172.21.15.154:3300/0 conn(0x7f777c0a1330 msgr2=0x7f777c09f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:54:01.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:01.132+0000 7f777affd640 1 --2- 172.21.15.42:0/86996660 >> v2:172.21.15.154:3300/0 conn(0x7f777c0a1330 0x7f777c09f180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:54:01.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:01.132+0000 7f777affd640 1 --2- 172.21.15.42:0/86996660 >> v2:172.21.15.154:3300/0 conn(0x7f777c0a1330 0x7f777c09f180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T03:54:01.541 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:01.536+0000 7f777a7fc640 1 -- 172.21.15.42:0/86996660 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f777c0a1710 msgr2=0x7f777c0a1b10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T03:54:01.541 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:01.536+0000 7f777a7fc640 1 --2- 172.21.15.42:0/86996660 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f777c0a1710 0x7f777c0a1b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T03:54:01.542 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:01.536+0000 7f777affd640 1 --2- 172.21.15.42:0/86996660 >> v2:172.21.15.154:3300/0 conn(0x7f777c0a1330 0x7f777c09f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:54:01.542 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:01.536+0000 7f777affd640 10 monclient(hunting): get_auth_request con 0x7f777c0a1330 auth_method 0 2024-09-16T03:54:01.542 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:01.536+0000 7f777affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:54:01.543 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:01.536+0000 7f777affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:54:01.543 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:01.536+0000 7f777affd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:54:01.543 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:01.536+0000 7f777affd640 1 -- 172.21.15.42:0/86996660 >> v2:172.21.15.154:3300/0 conn(0x7f777c0a1330 msgr2=0x7f777c09f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T03:54:01.543 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:01.536+0000 7f777affd640 1 -- 172.21.15.42:0/86996660 >> v2:172.21.15.154:3300/0 conn(0x7f777c0a1330 msgr2=0x7f777c09f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:54:01.543 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:01.536+0000 7f777affd640 1 --2- 172.21.15.42:0/86996660 >> v2:172.21.15.154:3300/0 conn(0x7f777c0a1330 0x7f777c09f180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:54:01.543 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:01.536+0000 7f777affd640 1 --2- 172.21.15.42:0/86996660 >> v2:172.21.15.154:3300/0 conn(0x7f777c0a1330 0x7f777c09f180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T03:54:02.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:02.340+0000 7f777a7fc640 1 -- 172.21.15.42:0/86996660 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f777c0a1710 msgr2=0x7f777c0a1b10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T03:54:02.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:02.340+0000 7f777a7fc640 1 --2- 172.21.15.42:0/86996660 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f777c0a1710 0x7f777c0a1b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T03:54:02.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:02.340+0000 7f777affd640 1 --2- 172.21.15.42:0/86996660 >> v2:172.21.15.154:3300/0 conn(0x7f777c0a1330 0x7f777c09f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:54:02.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:02.340+0000 7f777affd640 10 monclient(hunting): get_auth_request con 0x7f777c0a1330 auth_method 0 2024-09-16T03:54:02.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:02.340+0000 7f777affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:54:02.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:02.340+0000 7f777affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:54:02.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:02.340+0000 7f777affd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:54:02.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:02.340+0000 7f777affd640 1 -- 172.21.15.42:0/86996660 >> v2:172.21.15.154:3300/0 conn(0x7f777c0a1330 msgr2=0x7f777c09f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T03:54:02.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:02.340+0000 7f777affd640 1 -- 172.21.15.42:0/86996660 >> v2:172.21.15.154:3300/0 conn(0x7f777c0a1330 msgr2=0x7f777c09f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:54:02.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:02.340+0000 7f777affd640 1 --2- 172.21.15.42:0/86996660 >> v2:172.21.15.154:3300/0 conn(0x7f777c0a1330 0x7f777c09f180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:54:02.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:02.340+0000 7f777affd640 1 --2- 172.21.15.42:0/86996660 >> v2:172.21.15.154:3300/0 conn(0x7f777c0a1330 0x7f777c09f180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T03:54:03.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:03.932+0000 7f7778ff9640 10 monclient(hunting): tick 2024-09-16T03:54:03.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:03.932+0000 7f7778ff9640 10 monclient(hunting): _check_auth_tickets 2024-09-16T03:54:03.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:03.932+0000 7f7778ff9640 1 monclient(hunting): continuing hunt 2024-09-16T03:54:03.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:03.932+0000 7f7778ff9640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T03:54:03.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:03.932+0000 7f7778ff9640 1 -- 172.21.15.42:0/86996660 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f777c0a1710 msgr2=0x7f777c0a1b10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:54:03.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:03.932+0000 7f7778ff9640 1 --2- 172.21.15.42:0/86996660 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f777c0a1710 0x7f777c0a1b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:54:03.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:03.932+0000 7f7778ff9640 1 -- 172.21.15.42:0/86996660 >> v2:172.21.15.154:3300/0 conn(0x7f777c0a1330 msgr2=0x7f777c09f180 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:54:03.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:03.932+0000 7f7778ff9640 1 --2- 172.21.15.42:0/86996660 >> v2:172.21.15.154:3300/0 conn(0x7f777c0a1330 0x7f777c09f180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:54:03.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:03.932+0000 7f7778ff9640 1 -- 172.21.15.42:0/86996660 >> v1:172.21.15.42:6789/0 conn(0x7f777c1340f0 legacy=0x7f777c1345a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:54:03.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:03.932+0000 7f7778ff9640 10 monclient: _add_conns ranks=[3,1] 2024-09-16T03:54:03.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:03.932+0000 7f7778ff9640 1 --2- 172.21.15.42:0/86996660 >> v2:172.21.15.42:3300/0 conn(0x7f7760003720 0x7f7760005b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:54:03.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:03.932+0000 7f7778ff9640 10 monclient(hunting): picked mon.noname-a con 0x7f7760003720 addr v2:172.21.15.42:3300/0 2024-09-16T03:54:03.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:03.932+0000 7f7778ff9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7760006270 addr v1:172.21.15.154:6789/0 2024-09-16T03:54:03.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:03.932+0000 7f7778ff9640 1 -- 172.21.15.42:0/86996660 --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f777c0a2050 con 0x7f7760006270 2024-09-16T03:54:03.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:03.932+0000 7f7778ff9640 10 monclient(hunting): start opening mon connection 2024-09-16T03:54:03.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:03.932+0000 7f7778ff9640 1 -- 172.21.15.42:0/86996660 --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f777c09edf0 con 0x7f7760003720 2024-09-16T03:54:03.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:03.932+0000 7f7778ff9640 10 monclient(hunting): _renew_subs 2024-09-16T03:54:03.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:03.932+0000 7f777affd640 1 --2- 172.21.15.42:0/86996660 >> v2:172.21.15.42:3300/0 conn(0x7f7760003720 0x7f7760005b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:54:03.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:03.932+0000 7f777affd640 10 monclient(hunting): get_auth_request con 0x7f7760003720 auth_method 0 2024-09-16T03:54:03.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:03.932+0000 7f777affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:54:03.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:03.932+0000 7f777affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:54:03.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:03.932+0000 7f777affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:54:03.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:03.932+0000 7f777affd640 1 -- 172.21.15.42:0/86996660 >> v2:172.21.15.42:3300/0 conn(0x7f7760003720 msgr2=0x7f7760005b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T03:54:03.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:03.932+0000 7f777affd640 1 -- 172.21.15.42:0/86996660 >> v2:172.21.15.42:3300/0 conn(0x7f7760003720 msgr2=0x7f7760005b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:54:03.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:03.932+0000 7f777affd640 1 --2- 172.21.15.42:0/86996660 >> v2:172.21.15.42:3300/0 conn(0x7f7760003720 0x7f7760005b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:54:03.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:03.932+0000 7f777affd640 1 --2- 172.21.15.42:0/86996660 >> v2:172.21.15.42:3300/0 conn(0x7f7760003720 0x7f7760005b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:54:04.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:04.136+0000 7f777affd640 1 --2- 172.21.15.42:0/86996660 >> v2:172.21.15.42:3300/0 conn(0x7f7760003720 0x7f7760005b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:54:04.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:04.136+0000 7f777affd640 10 monclient(hunting): get_auth_request con 0x7f7760003720 auth_method 0 2024-09-16T03:54:04.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:04.136+0000 7f777affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:54:04.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:04.136+0000 7f777affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:54:04.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:04.136+0000 7f777affd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:54:04.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:04.136+0000 7f777affd640 1 -- 172.21.15.42:0/86996660 >> v2:172.21.15.42:3300/0 conn(0x7f7760003720 msgr2=0x7f7760005b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T03:54:04.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:04.136+0000 7f777affd640 1 -- 172.21.15.42:0/86996660 >> v2:172.21.15.42:3300/0 conn(0x7f7760003720 msgr2=0x7f7760005b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:54:04.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:04.136+0000 7f777affd640 1 --2- 172.21.15.42:0/86996660 >> v2:172.21.15.42:3300/0 conn(0x7f7760003720 0x7f7760005b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:54:04.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:04.136+0000 7f777affd640 1 --2- 172.21.15.42:0/86996660 >> v2:172.21.15.42:3300/0 conn(0x7f7760003720 0x7f7760005b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T03:54:04.543 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:04.540+0000 7f777affd640 1 --2- 172.21.15.42:0/86996660 >> v2:172.21.15.42:3300/0 conn(0x7f7760003720 0x7f7760005b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:54:04.543 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:04.540+0000 7f777affd640 10 monclient(hunting): get_auth_request con 0x7f7760003720 auth_method 0 2024-09-16T03:54:04.543 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:04.540+0000 7f777affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:54:04.543 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:04.540+0000 7f777affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:54:04.543 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:04.540+0000 7f777affd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:54:04.543 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:04.540+0000 7f777affd640 1 -- 172.21.15.42:0/86996660 >> v2:172.21.15.42:3300/0 conn(0x7f7760003720 msgr2=0x7f7760005b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T03:54:04.543 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:04.540+0000 7f777affd640 1 -- 172.21.15.42:0/86996660 >> v2:172.21.15.42:3300/0 conn(0x7f7760003720 msgr2=0x7f7760005b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:54:04.544 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:04.540+0000 7f777affd640 1 --2- 172.21.15.42:0/86996660 >> v2:172.21.15.42:3300/0 conn(0x7f7760003720 0x7f7760005b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:54:04.544 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:04.540+0000 7f777affd640 1 --2- 172.21.15.42:0/86996660 >> v2:172.21.15.42:3300/0 conn(0x7f7760003720 0x7f7760005b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T03:54:05.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.344+0000 7f777affd640 1 --2- 172.21.15.42:0/86996660 >> v2:172.21.15.42:3300/0 conn(0x7f7760003720 0x7f7760005b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:54:05.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.344+0000 7f777affd640 10 monclient(hunting): get_auth_request con 0x7f7760003720 auth_method 0 2024-09-16T03:54:05.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.344+0000 7f777affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:54:05.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.344+0000 7f777affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:54:05.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.344+0000 7f777affd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:54:05.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.344+0000 7f777affd640 1 -- 172.21.15.42:0/86996660 >> v2:172.21.15.42:3300/0 conn(0x7f7760003720 msgr2=0x7f7760005b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T03:54:05.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.344+0000 7f777affd640 1 -- 172.21.15.42:0/86996660 >> v2:172.21.15.42:3300/0 conn(0x7f7760003720 msgr2=0x7f7760005b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:54:05.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.344+0000 7f777affd640 1 --2- 172.21.15.42:0/86996660 >> v2:172.21.15.42:3300/0 conn(0x7f7760003720 0x7f7760005b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:54:05.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.344+0000 7f777affd640 1 --2- 172.21.15.42:0/86996660 >> v2:172.21.15.42:3300/0 conn(0x7f7760003720 0x7f7760005b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T03:54:05.848 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.840+0000 7f7779ffb640 1 -- 172.21.15.42:0/86996660 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3520218137 0 0) 0x7f777c0a2050 con 0x7f7760006270 2024-09-16T03:54:05.848 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.840+0000 7f7779ffb640 10 monclient(hunting): _negotiate 2024-09-16T03:54:05.849 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.840+0000 7f7779ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:54:05.849 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.840+0000 7f7779ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:54:05.849 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.840+0000 7f7779ffb640 10 monclient(hunting): my global_id is 844147 2024-09-16T03:54:05.849 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.840+0000 7f7779ffb640 1 -- 172.21.15.42:0/86996660 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f775c003530 con 0x7f7760006270 2024-09-16T03:54:05.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.840+0000 7f7779ffb640 1 -- 172.21.15.42:0/86996660 <== mon.1 v1:172.21.15.154:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1551562101 0 0) 0x7f775c003530 con 0x7f7760006270 2024-09-16T03:54:05.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.844+0000 7f7779ffb640 1 -- 172.21.15.42:0/86996660 >> v2:172.21.15.42:3300/0 conn(0x7f7760003720 msgr2=0x7f7760005b80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:54:05.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.844+0000 7f7779ffb640 1 --2- 172.21.15.42:0/86996660 >> v2:172.21.15.42:3300/0 conn(0x7f7760003720 0x7f7760005b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:54:05.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.844+0000 7f7779ffb640 10 monclient: _finish_hunting 0 2024-09-16T03:54:05.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.844+0000 7f7779ffb640 1 monclient: found mon.noname-c-legacy 2024-09-16T03:54:05.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.844+0000 7f7779ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:54:05.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.844+0000 7f7779ffb640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.154:6789/0 2024-09-16T03:54:05.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.844+0000 7f7779ffb640 1 -- 172.21.15.42:0/86996660 --> v1:172.21.15.154:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f776000afc0 con 0x7f7760006270 2024-09-16T03:54:05.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.844+0000 7f7779ffb640 10 monclient: _finish_auth 0 2024-09-16T03:54:05.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.844+0000 7f7779ffb640 10 monclient: _check_auth_tickets 2024-09-16T03:54:05.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.844+0000 7f7779ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:54:05.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.844+0000 7f7779ffb640 1 -- 172.21.15.42:0/86996660 <== mon.1 v1:172.21.15.154:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2177413734 0 0) 0x7f77640046c0 con 0x7f7760006270 2024-09-16T03:54:05.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.844+0000 7f7779ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:54:05.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.844+0000 7f7779ffb640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-16T03:54:05.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.844+0000 7f7779ffb640 10 monclient: dump: 2024-09-16T03:54:05.852 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:54:05.852 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:54:05.852 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:05.853 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:05.853 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:54:05.853 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:54:05.853 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:54:05.853 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:54:05.853 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:54:05.853 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:54:05.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.844+0000 7f7779ffb640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] but i'm connected to v1:172.21.15.154:6789/0, reconnecting 2024-09-16T03:54:05.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.844+0000 7f7779ffb640 10 monclient: _reopen_session rank -1 2024-09-16T03:54:05.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.844+0000 7f7779ffb640 1 -- 172.21.15.42:0/86996660 >> v1:172.21.15.154:6789/0 conn(0x7f7760006270 legacy=0x7f7760008710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:54:05.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.844+0000 7f777b7fe640 1 -- 172.21.15.42:0/86996660 reap_dead start 2024-09-16T03:54:05.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.844+0000 7f7779ffb640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T03:54:05.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.844+0000 7f7779ffb640 1 --2- 172.21.15.42:0/86996660 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f775c0039c0 0x7f775c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:54:05.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.844+0000 7f7779ffb640 10 monclient(hunting): picked mon.a con 0x7f775c0039c0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:54:05.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.844+0000 7f7779ffb640 1 --2- 172.21.15.42:0/86996660 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f775c004810 0x7f775c004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:54:05.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.844+0000 7f7779ffb640 10 monclient(hunting): picked mon.b con 0x7f775c004810 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:54:05.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.844+0000 7f7779ffb640 1 --2- 172.21.15.42:0/86996660 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f775c005220 0x7f775c00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:54:05.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.844+0000 7f7779ffb640 10 monclient(hunting): picked mon.c con 0x7f775c005220 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:54:05.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.844+0000 7f7779ffb640 10 monclient(hunting): start opening mon connection 2024-09-16T03:54:05.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.844+0000 7f7779ffb640 1 -- 172.21.15.42:0/86996660 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f777c09edf0 con 0x7f775c0039c0 2024-09-16T03:54:05.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.844+0000 7f7779ffb640 10 monclient(hunting): start opening mon connection 2024-09-16T03:54:05.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.844+0000 7f7779ffb640 1 -- 172.21.15.42:0/86996660 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f775c00c150 con 0x7f775c004810 2024-09-16T03:54:05.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.844+0000 7f7779ffb640 10 monclient(hunting): start opening mon connection 2024-09-16T03:54:05.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.844+0000 7f7779ffb640 1 -- 172.21.15.42:0/86996660 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f775c00c350 con 0x7f775c005220 2024-09-16T03:54:05.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.844+0000 7f7779ffb640 10 monclient(hunting): _renew_subs 2024-09-16T03:54:05.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.844+0000 7f7779ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:54:05.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.844+0000 7f777b7fe640 1 -- 172.21.15.42:0/86996660 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f775c005220 msgr2=0x7f775c00b9d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T03:54:05.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.844+0000 7f777b7fe640 1 --2- 172.21.15.42:0/86996660 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f775c005220 0x7f775c00b9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:54:05.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.844+0000 7f777affd640 1 --2- 172.21.15.42:0/86996660 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f775c0039c0 0x7f775c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:54:05.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.844+0000 7f777affd640 10 monclient(hunting): get_auth_request con 0x7f775c0039c0 auth_method 0 2024-09-16T03:54:05.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.844+0000 7f777affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:54:05.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.844+0000 7f777affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:54:05.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.844+0000 7f777affd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:54:05.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.844+0000 7f777a7fc640 1 --2- 172.21.15.42:0/86996660 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f775c004810 0x7f775c004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:54:05.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.844+0000 7f777a7fc640 10 monclient(hunting): get_auth_request con 0x7f775c004810 auth_method 0 2024-09-16T03:54:05.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.844+0000 7f777a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:54:05.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.844+0000 7f777a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:54:05.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.844+0000 7f777a7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:54:05.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.844+0000 7f777affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:54:05.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.844+0000 7f777affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:54:05.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.844+0000 7f777affd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T03:54:05.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.844+0000 7f777a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:54:05.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.844+0000 7f777a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:54:05.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.844+0000 7f777a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T03:54:05.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.844+0000 7f777affd640 10 monclient(hunting): handle_auth_done global_id 844147 payload 293 2024-09-16T03:54:05.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.844+0000 7f777affd640 1 -- 172.21.15.42:0/86996660 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f775c005220 msgr2=0x7f775c00b9d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:54:05.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.844+0000 7f777affd640 1 --2- 172.21.15.42:0/86996660 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f775c005220 0x7f775c00b9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:54:05.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.844+0000 7f777affd640 1 -- 172.21.15.42:0/86996660 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f775c004810 msgr2=0x7f775c004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:54:05.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.844+0000 7f777affd640 1 --2- 172.21.15.42:0/86996660 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f775c004810 0x7f775c004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:54:05.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.844+0000 7f777affd640 10 monclient: _finish_hunting 0 2024-09-16T03:54:05.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.844+0000 7f777affd640 1 monclient: found mon.a 2024-09-16T03:54:05.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.844+0000 7f777affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:54:05.857 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.844+0000 7f777affd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:54:05.857 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.844+0000 7f777affd640 1 -- 172.21.15.42:0/86996660 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f775c003530 con 0x7f775c0039c0 2024-09-16T03:54:05.857 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.844+0000 7f777affd640 10 monclient: _finish_auth 0 2024-09-16T03:54:05.857 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.844+0000 7f777affd640 10 monclient: _check_auth_tickets 2024-09-16T03:54:05.857 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.844+0000 7f777affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:54:05.857 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.844+0000 7f7781361640 5 monclient: authenticate success, global_id 844147 2024-09-16T03:54:05.857 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.844+0000 7f7781361640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:54:05.857 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.848+0000 7f777affd640 1 --2- 172.21.15.42:0/86996660 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f775c0039c0 0x7f775c003dc0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f776c000ee0 tx=0x7f776c01dd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aca2f6db4b702a93 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:54:05.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.864+0000 7f7779ffb640 1 -- 172.21.15.42:0/86996660 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f776c00f040 con 0x7f775c0039c0 2024-09-16T03:54:05.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.864+0000 7f7779ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:54:05.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.864+0000 7f7779ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:54:05.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.864+0000 7f7779ffb640 10 monclient: dump: 2024-09-16T03:54:05.870 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:54:05.871 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:54:05.871 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:05.871 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:05.871 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:54:05.871 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:54:05.871 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:54:05.871 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:54:05.871 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:54:05.871 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:54:05.871 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.864+0000 7f7781361640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:54:05.871 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.864+0000 7f7779ffb640 1 -- 172.21.15.42:0/86996660 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f776c019250 con 0x7f775c0039c0 2024-09-16T03:54:05.871 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.864+0000 7f7779ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T03:54:05.871 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.864+0000 7f7779ffb640 1 -- 172.21.15.42:0/86996660 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f776c004260 con 0x7f775c0039c0 2024-09-16T03:54:05.871 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.864+0000 7f7781361640 10 monclient: get_monmap_and_config success 2024-09-16T03:54:05.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.864+0000 7f7779ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:54:05.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.864+0000 7f7781361640 4 set_mon_vals no callback set 2024-09-16T03:54:05.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.864+0000 7f7779ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:54:05.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.864+0000 7f7779ffb640 10 monclient: dump: 2024-09-16T03:54:05.872 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:54:05.872 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:54:05.873 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:05.873 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:05.873 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:54:05.873 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:54:05.873 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:54:05.873 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:54:05.873 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:54:05.873 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:54:05.874 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.864+0000 7f7781361640 10 monclient: shutdown 2024-09-16T03:54:05.874 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.864+0000 7f7781361640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:54:05.874 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.864+0000 7f7781361640 1 -- 172.21.15.42:0/86996660 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f775c0039c0 msgr2=0x7f775c003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:54:05.875 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.864+0000 7f7781361640 1 --2- 172.21.15.42:0/86996660 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f775c0039c0 0x7f775c003dc0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f776c000ee0 tx=0x7f776c01dd30 comp rx=0 tx=0).stop 2024-09-16T03:54:05.875 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.864+0000 7f7781361640 1 -- 172.21.15.42:0/86996660 shutdown_connections 2024-09-16T03:54:05.875 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.864+0000 7f7781361640 1 --2- 172.21.15.42:0/86996660 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f775c005220 0x7f775c00b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:54:05.875 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.864+0000 7f7781361640 1 --2- 172.21.15.42:0/86996660 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f775c004810 0x7f775c004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:54:05.875 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.864+0000 7f7781361640 1 --2- 172.21.15.42:0/86996660 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f775c0039c0 0x7f775c003dc0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:54:05.875 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.864+0000 7f7781361640 1 -- 172.21.15.42:0/86996660 >> 172.21.15.42:0/86996660 conn(0x7f777c0a2420 msgr2=0x7f777c0a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:54:05.875 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.864+0000 7f7781361640 1 -- 172.21.15.42:0/86996660 shutdown_connections 2024-09-16T03:54:05.875 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.864+0000 7f7781361640 1 -- 172.21.15.42:0/86996660 wait complete. 2024-09-16T03:54:05.875 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.868+0000 7f7781361640 10 monclient: build_initial_monmap 2024-09-16T03:54:05.875 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.868+0000 7f7781361640 10 monclient: monmap: 2024-09-16T03:54:05.875 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:54:05.875 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:54:05.875 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:54:05.876 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:54:05.876 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:54:05.876 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:54:05.876 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:54:05.876 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:54:05.876 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:54:05.876 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:54:05.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.868+0000 7f7781361640 1 librados: starting msgr at 2024-09-16T03:54:05.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.868+0000 7f7781361640 1 librados: starting objecter 2024-09-16T03:54:05.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.868+0000 7f7781361640 1 Processor -- start 2024-09-16T03:54:05.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.868+0000 7f7781361640 1 -- start start 2024-09-16T03:54:05.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.868+0000 7f7781361640 1 librados: setting wanted keys 2024-09-16T03:54:05.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.868+0000 7f7781361640 1 librados: calling monclient init 2024-09-16T03:54:05.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.868+0000 7f7781361640 10 monclient: init 2024-09-16T03:54:05.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.868+0000 7f7781361640 10 monclient: _reopen_session rank -1 2024-09-16T03:54:05.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.868+0000 7f7781361640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:54:05.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.868+0000 7f7781361640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f775c0039c0 0x7f777c1ca970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:54:05.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.868+0000 7f7781361640 10 monclient(hunting): picked mon.noname-b con 0x7f775c0039c0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:54:05.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.868+0000 7f7781361640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f775c004810 0x7f777c1325b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:54:05.883 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.868+0000 7f7781361640 10 monclient(hunting): picked mon.noname-a con 0x7f775c004810 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:54:05.883 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.868+0000 7f7781361640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f775c005220 0x7f777c132af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:54:05.883 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.868+0000 7f7781361640 10 monclient(hunting): picked mon.noname-c con 0x7f775c005220 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:54:05.883 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.868+0000 7f7781361640 10 monclient(hunting): start opening mon connection 2024-09-16T03:54:05.883 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.868+0000 7f7781361640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f777c134890 con 0x7f775c004810 2024-09-16T03:54:05.883 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.868+0000 7f7781361640 10 monclient(hunting): start opening mon connection 2024-09-16T03:54:05.883 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.868+0000 7f7781361640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f777c134710 con 0x7f775c0039c0 2024-09-16T03:54:05.883 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.868+0000 7f7781361640 10 monclient(hunting): start opening mon connection 2024-09-16T03:54:05.883 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.868+0000 7f7781361640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f777c134a10 con 0x7f775c005220 2024-09-16T03:54:05.883 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.868+0000 7f7781361640 10 monclient(hunting): _renew_subs 2024-09-16T03:54:05.883 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.868+0000 7f7781361640 10 monclient(hunting): authenticate will time out at 1692.842182s 2024-09-16T03:54:05.883 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.868+0000 7f777b7fe640 1 -- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f775c005220 msgr2=0x7f777c132af0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T03:54:05.883 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.868+0000 7f777b7fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f775c005220 0x7f777c132af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:54:05.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.868+0000 7f777a7fc640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f775c004810 0x7f777c1325b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:54:05.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.868+0000 7f777a7fc640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f775c004810 0x7f777c1325b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:53060/0 (socket says 172.21.15.42:53060) 2024-09-16T03:54:05.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.868+0000 7f777a7fc640 1 -- 172.21.15.42:0/1617352707 learned_addr learned my addr 172.21.15.42:0/1617352707 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:54:05.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.868+0000 7f777a7fc640 10 monclient(hunting): get_auth_request con 0x7f775c004810 auth_method 0 2024-09-16T03:54:05.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.868+0000 7f777a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:54:05.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.868+0000 7f777a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:54:05.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.868+0000 7f777a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:54:05.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.868+0000 7f777affd640 1 --2- 172.21.15.42:0/1617352707 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f775c0039c0 0x7f777c1ca970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:54:05.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.868+0000 7f777affd640 10 monclient(hunting): get_auth_request con 0x7f775c0039c0 auth_method 0 2024-09-16T03:54:05.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.868+0000 7f777affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:54:05.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.868+0000 7f777affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:54:05.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.868+0000 7f777affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:54:05.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.868+0000 7f777a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:54:05.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.868+0000 7f777a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:54:05.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.868+0000 7f777a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:54:05.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.868+0000 7f777affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:54:05.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.868+0000 7f777affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:54:05.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.868+0000 7f777affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:54:05.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.868+0000 7f777a7fc640 10 monclient(hunting): handle_auth_done global_id 824400 payload 824 2024-09-16T03:54:05.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.868+0000 7f777a7fc640 1 -- 172.21.15.42:0/1617352707 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f775c005220 msgr2=0x7f777c132af0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:54:05.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.868+0000 7f777a7fc640 1 --2- 172.21.15.42:0/1617352707 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f775c005220 0x7f777c132af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:54:05.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.868+0000 7f777a7fc640 1 -- 172.21.15.42:0/1617352707 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f775c0039c0 msgr2=0x7f777c1ca970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:54:05.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.868+0000 7f777a7fc640 1 --2- 172.21.15.42:0/1617352707 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f775c0039c0 0x7f777c1ca970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:54:05.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.868+0000 7f777a7fc640 10 monclient: _finish_hunting 0 2024-09-16T03:54:05.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.868+0000 7f777a7fc640 1 monclient: found mon.noname-a 2024-09-16T03:54:05.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.868+0000 7f777a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:54:05.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.868+0000 7f777a7fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:54:05.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.868+0000 7f777a7fc640 1 -- 172.21.15.42:0/1617352707 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f777c11fc90 con 0x7f775c004810 2024-09-16T03:54:05.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.868+0000 7f777a7fc640 1 --2- 172.21.15.42:0/1617352707 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f775c004810 0x7f777c1325b0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f7764003c20 tx=0x7f776400f5a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:54:05.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.868+0000 7f77737fe640 10 client.?.objecter ms_handle_connect 0x7f775c004810 2024-09-16T03:54:05.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.868+0000 7f77737fe640 10 client.?.objecter resend_mon_ops 2024-09-16T03:54:05.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.868+0000 7f77737fe640 1 -- 172.21.15.42:0/1617352707 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7764003cf0 con 0x7f775c004810 2024-09-16T03:54:05.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.868+0000 7f77737fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:54:05.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.868+0000 7f77737fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:54:05.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.868+0000 7f77737fe640 10 monclient: dump: 2024-09-16T03:54:05.886 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:54:05.886 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:54:05.886 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:05.886 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:05.886 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:54:05.887 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:54:05.887 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:54:05.887 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:54:05.887 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:54:05.887 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:54:05.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.868+0000 7f77737fe640 10 monclient: _finish_auth 0 2024-09-16T03:54:05.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.868+0000 7f77737fe640 10 monclient: _check_auth_tickets 2024-09-16T03:54:05.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.868+0000 7f77737fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:54:05.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.868+0000 7f77737fe640 1 -- 172.21.15.42:0/1617352707 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f776400fc80 con 0x7f775c004810 2024-09-16T03:54:05.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.868+0000 7f7781361640 5 monclient: authenticate success, global_id 824400 2024-09-16T03:54:05.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.868+0000 7f7781361640 10 monclient: _renew_subs 2024-09-16T03:54:05.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.868+0000 7f7781361640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:54:05.888 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.868+0000 7f7781361640 1 -- 172.21.15.42:0/1617352707 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f777c1caeb0 con 0x7f775c004810 2024-09-16T03:54:05.888 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.868+0000 7f77737fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:54:05.888 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.868+0000 7f77737fe640 1 -- 172.21.15.42:0/1617352707 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7764014470 con 0x7f775c004810 2024-09-16T03:54:05.888 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.868+0000 7f77737fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:54:05.888 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.868+0000 7f77737fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:54:05.888 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.868+0000 7f77737fe640 10 monclient: dump: 2024-09-16T03:54:05.888 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:54:05.888 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:54:05.888 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:05.888 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:05.888 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:54:05.888 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:54:05.888 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:54:05.888 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:54:05.889 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:54:05.889 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:54:05.889 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.868+0000 7f7781361640 10 client.824400.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:54:05.889 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.868+0000 7f7781361640 10 monclient: _renew_subs 2024-09-16T03:54:05.889 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.868+0000 7f7781361640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:54:05.889 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.868+0000 7f7781361640 1 -- 172.21.15.42:0/1617352707 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f777c0a4830 con 0x7f775c004810 2024-09-16T03:54:05.889 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.868+0000 7f7781361640 1 librados: init done 2024-09-16T03:54:05.889 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.868+0000 7f77797fa640 4 set_mon_vals no callback set 2024-09-16T03:54:05.889 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.872+0000 7f77737fe640 1 -- 172.21.15.42:0/1617352707 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f7764014610 con 0x7f775c004810 2024-09-16T03:54:05.889 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.872+0000 7f77737fe640 1 --2- 172.21.15.42:0/1617352707 >> v2:172.21.15.154:6812/254845342 conn(0x7f7760038050 0x7f776003a510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:54:05.889 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.872+0000 7f7781361640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:54:05.889 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.872+0000 7f7781361640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:54:05.889 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.872+0000 7f7781361640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:54:05.890 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.872+0000 7f77737fe640 1 -- 172.21.15.42:0/1617352707 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(366..366 src has 286..366) ==== 5713+0+0 (secure 0 0 0) 0x7f7764049750 con 0x7f775c004810 2024-09-16T03:54:05.890 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.872+0000 7f77737fe640 10 client.824400.objecter ms_dispatch 0x7f777c000ba0 osd_map(366..366 src has 286..366) 2024-09-16T03:54:05.890 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.872+0000 7f77737fe640 3 client.824400.objecter handle_osd_map got epochs [366,366] > 0 2024-09-16T03:54:05.890 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.872+0000 7f77737fe640 3 client.824400.objecter handle_osd_map decoding full epoch 366 2024-09-16T03:54:05.890 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.872+0000 7f7781361640 1 -- 172.21.15.42:0/1617352707 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f777c0a1f20 con 0x7f775c004810 2024-09-16T03:54:05.890 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.872+0000 7f77737fe640 20 client.824400.objecter dump_active .. 0 homeless 2024-09-16T03:54:05.890 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.872+0000 7f777affd640 1 --2- 172.21.15.42:0/1617352707 >> v2:172.21.15.154:6812/254845342 conn(0x7f7760038050 0x7f776003a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:54:05.890 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.872+0000 7f777affd640 10 monclient: get_auth_request con 0x7f7760038050 auth_method 0 2024-09-16T03:54:05.890 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.872+0000 7f777affd640 1 --2- 172.21.15.42:0/1617352707 >> v2:172.21.15.154:6812/254845342 conn(0x7f7760038050 0x7f776003a510 secure :-1 s=READY pgs=3608 cs=0 l=1 rev1=1 crypto rx=0x7f776c0ae000 tx=0x7f776c004740 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:54:05.890 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.872+0000 7f77737fe640 10 client.824400.objecter ms_handle_connect 0x7f7760038050 2024-09-16T03:54:05.891 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.880+0000 7f77737fe640 1 -- 172.21.15.42:0/1617352707 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f7764018020 con 0x7f775c004810 2024-09-16T03:54:05.891 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.880+0000 7f77737fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:54:05.891 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:05.880+0000 7f77737fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:54:06.115 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.108+0000 7f7781361640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:54:06.115 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.108+0000 7f7781361640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:54:06.115 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.108+0000 7f7781361640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:54:06.115 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.108+0000 7f7781361640 1 -- 172.21.15.42:0/1617352707 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f777c11e560 con 0x7f775c004810 2024-09-16T03:54:06.116 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.108+0000 7f77737fe640 1 -- 172.21.15.42:0/1617352707 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1555 (secure 0 0 0) 0x7f7764014900 con 0x7f775c004810 2024-09-16T03:54:06.116 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.108+0000 7f77737fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:54:06.116 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.108+0000 7f77737fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:54:06.116 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":102,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:54:06.118 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.112+0000 7f7781361640 10 librados: watch_flush enter 2024-09-16T03:54:06.118 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.112+0000 7f7781361640 10 librados: watch_flush exit 2024-09-16T03:54:06.118 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.112+0000 7f7781361640 20 client.824400.objecter shutdown clearing up homeless session... 2024-09-16T03:54:06.118 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.112+0000 7f7781361640 10 client.824400.objecter successfully canceled tick 2024-09-16T03:54:06.118 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.112+0000 7f7781361640 1 -- 172.21.15.42:0/1617352707 >> v2:172.21.15.154:6812/254845342 conn(0x7f7760038050 msgr2=0x7f776003a510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:54:06.118 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.112+0000 7f7781361640 1 --2- 172.21.15.42:0/1617352707 >> v2:172.21.15.154:6812/254845342 conn(0x7f7760038050 0x7f776003a510 secure :-1 s=READY pgs=3608 cs=0 l=1 rev1=1 crypto rx=0x7f776c0ae000 tx=0x7f776c004740 comp rx=0 tx=0).stop 2024-09-16T03:54:06.119 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.112+0000 7f7781361640 10 monclient: shutdown 2024-09-16T03:54:06.119 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.112+0000 7f7781361640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:54:06.119 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.112+0000 7f7781361640 1 -- 172.21.15.42:0/1617352707 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f775c004810 msgr2=0x7f777c1325b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:54:06.119 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.112+0000 7f7781361640 1 --2- 172.21.15.42:0/1617352707 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f775c004810 0x7f777c1325b0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f7764003c20 tx=0x7f776400f5a0 comp rx=0 tx=0).stop 2024-09-16T03:54:06.119 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.112+0000 7f7781361640 1 -- 172.21.15.42:0/1617352707 shutdown_connections 2024-09-16T03:54:06.119 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.112+0000 7f7781361640 1 --2- 172.21.15.42:0/1617352707 >> v2:172.21.15.154:6812/254845342 conn(0x7f7760038050 0x7f776003a510 unknown :-1 s=CLOSED pgs=3608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:54:06.119 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.112+0000 7f7781361640 1 --2- 172.21.15.42:0/1617352707 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f775c005220 0x7f777c132af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:54:06.119 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.112+0000 7f7781361640 1 --2- 172.21.15.42:0/1617352707 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f775c004810 0x7f777c1325b0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:54:06.119 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.112+0000 7f7781361640 1 --2- 172.21.15.42:0/1617352707 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f775c0039c0 0x7f777c1ca970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:54:06.119 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.112+0000 7f7781361640 1 -- 172.21.15.42:0/1617352707 >> 172.21.15.42:0/1617352707 conn(0x7f777c0a2420 msgr2=0x7f777c0a3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:54:06.119 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.112+0000 7f7781361640 1 -- 172.21.15.42:0/1617352707 shutdown_connections 2024-09-16T03:54:06.119 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.112+0000 7f7781361640 1 -- 172.21.15.42:0/1617352707 wait complete. 2024-09-16T03:54:06.119 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.112+0000 7f7781361640 1 librados: shutdown 2024-09-16T03:54:06.133 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T03:54:06.134 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-16T03:54:06.134 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T03:54:06.249 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.240+0000 7faa6e54b640 10 monclient: get_monmap_and_config 2024-09-16T03:54:06.249 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.240+0000 7faa6e54b640 10 monclient: build_initial_monmap 2024-09-16T03:54:06.249 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.240+0000 7faa6e54b640 10 monclient: monmap: 2024-09-16T03:54:06.249 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:54:06.249 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:54:06.250 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:54:06.247622+0000 2024-09-16T03:54:06.250 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:54:06.247622+0000 2024-09-16T03:54:06.250 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:54:06.250 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:54:06.250 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:54:06.250 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:54:06.250 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:54:06.250 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:54:06.250 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:54:06.250 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:54:06.251 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.244+0000 7faa6e54b640 1 Processor -- start 2024-09-16T03:54:06.251 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.244+0000 7faa6e54b640 1 -- start start 2024-09-16T03:54:06.251 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.244+0000 7faa6e54b640 10 monclient: init 2024-09-16T03:54:06.264 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.244+0000 7faa6e54b640 10 monclient: _reopen_session rank -1 2024-09-16T03:54:06.264 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.244+0000 7faa6e54b640 10 monclient: _add_conns ranks=[0,2,1,3,4] 2024-09-16T03:54:06.264 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.244+0000 7faa6e54b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7faa680a1330 addr v1:172.21.15.42:6789/0 2024-09-16T03:54:06.264 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.244+0000 7faa6e54b640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7faa680a1710 0x7faa680a1b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:54:06.264 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.244+0000 7faa6e54b640 10 monclient(hunting): picked mon.noname-b con 0x7faa680a1710 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:54:06.264 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.244+0000 7faa6e54b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7faa681393a0 addr v1:172.21.15.154:6789/0 2024-09-16T03:54:06.264 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.244+0000 7faa6e54b640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa6809e4c0 con 0x7faa680a1330 2024-09-16T03:54:06.265 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.244+0000 7faa6e54b640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa680a2050 con 0x7faa681393a0 2024-09-16T03:54:06.265 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.244+0000 7faa6e54b640 10 monclient(hunting): start opening mon connection 2024-09-16T03:54:06.265 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.244+0000 7faa6e54b640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7faa6809edf0 con 0x7faa680a1710 2024-09-16T03:54:06.265 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.244+0000 7faa6e54b640 10 monclient(hunting): _renew_subs 2024-09-16T03:54:06.265 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.244+0000 7faa6e54b640 10 monclient(hunting): authenticate will time out at 1693.217059s 2024-09-16T03:54:06.265 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.244+0000 7faa5f7fe640 1 -- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7faa680a1710 msgr2=0x7faa680a1b10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T03:54:06.265 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.244+0000 7faa5f7fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7faa680a1710 0x7faa680a1b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:54:06.265 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.244+0000 7faa67fff640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7faa680a1330 0x7faa6809f180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:53816/0 (socket says 172.21.15.42:53816) 2024-09-16T03:54:06.265 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.244+0000 7faa67fff640 1 -- 172.21.15.42:0/607439995 learned_addr learned my addr 172.21.15.42:0/607439995 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:54:06.265 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.244+0000 7faa677fe640 1 -- 172.21.15.42:0/607439995 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3778295909 0 0) 0x7faa6809e4c0 con 0x7faa680a1330 2024-09-16T03:54:06.265 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.244+0000 7faa677fe640 10 monclient(hunting): _negotiate 2024-09-16T03:54:06.265 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.244+0000 7faa677fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:54:06.265 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.244+0000 7faa677fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:54:06.266 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.244+0000 7faa677fe640 10 monclient(hunting): my global_id is 824403 2024-09-16T03:54:06.266 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.244+0000 7faa677fe640 1 -- 172.21.15.42:0/607439995 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faa54003530 con 0x7faa680a1330 2024-09-16T03:54:06.266 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.244+0000 7faa677fe640 1 -- 172.21.15.42:0/607439995 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3689920765 0 0) 0x7faa54003530 con 0x7faa680a1330 2024-09-16T03:54:06.266 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.244+0000 7faa677fe640 1 -- 172.21.15.42:0/607439995 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7faa680a1710 msgr2=0x7faa680a1b10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:54:06.266 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.244+0000 7faa677fe640 1 --2- 172.21.15.42:0/607439995 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7faa680a1710 0x7faa680a1b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:54:06.266 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.244+0000 7faa677fe640 1 -- 172.21.15.42:0/607439995 >> v1:172.21.15.154:6789/0 conn(0x7faa681393a0 legacy=0x7faa6813b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:54:06.266 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.244+0000 7faa677fe640 10 monclient: _finish_hunting 0 2024-09-16T03:54:06.266 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.244+0000 7faa677fe640 1 monclient: found mon.noname-a-legacy 2024-09-16T03:54:06.266 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.244+0000 7faa677fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:54:06.266 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.244+0000 7faa677fe640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.42:6789/0 2024-09-16T03:54:06.266 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.244+0000 7faa677fe640 1 -- 172.21.15.42:0/607439995 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa680a3170 con 0x7faa680a1330 2024-09-16T03:54:06.266 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.244+0000 7faa677fe640 10 monclient: _finish_auth 0 2024-09-16T03:54:06.266 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.244+0000 7faa677fe640 10 monclient: _check_auth_tickets 2024-09-16T03:54:06.267 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.244+0000 7faa677fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:54:06.267 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.244+0000 7faa677fe640 1 -- 172.21.15.42:0/607439995 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2177413734 0 0) 0x7faa58001680 con 0x7faa680a1330 2024-09-16T03:54:06.267 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.244+0000 7faa6e54b640 5 monclient: authenticate success, global_id 824403 2024-09-16T03:54:06.267 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.244+0000 7faa6e54b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:54:06.267 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.244+0000 7faa677fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:54:06.267 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.244+0000 7faa677fe640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T03:54:06.267 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.244+0000 7faa677fe640 10 monclient: dump: 2024-09-16T03:54:06.267 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:54:06.268 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:54:06.268 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:06.268 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:06.268 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:54:06.268 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:54:06.268 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:54:06.268 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:54:06.268 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:54:06.268 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:54:06.269 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.244+0000 7faa677fe640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] but i'm connected to v1:172.21.15.42:6789/0, reconnecting 2024-09-16T03:54:06.269 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.244+0000 7faa677fe640 10 monclient: _reopen_session rank -1 2024-09-16T03:54:06.269 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.244+0000 7faa677fe640 1 -- 172.21.15.42:0/607439995 >> v1:172.21.15.42:6789/0 conn(0x7faa680a1330 legacy=0x7faa6809f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:54:06.269 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.244+0000 7faa677fe640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:54:06.269 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.244+0000 7faa677fe640 1 --2- 172.21.15.42:0/607439995 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faa540039f0 0x7faa54003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:54:06.269 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.244+0000 7faa677fe640 10 monclient(hunting): picked mon.a con 0x7faa540039f0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:54:06.269 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.244+0000 7faa677fe640 1 --2- 172.21.15.42:0/607439995 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7faa680a1710 0x7faa54004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:54:06.269 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.244+0000 7faa677fe640 10 monclient(hunting): picked mon.c con 0x7faa680a1710 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:54:06.269 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.244+0000 7faa677fe640 1 --2- 172.21.15.42:0/607439995 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faa54004e70 0x7faa5400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:54:06.269 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.244+0000 7faa677fe640 10 monclient(hunting): picked mon.b con 0x7faa54004e70 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:54:06.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.244+0000 7faa677fe640 10 monclient(hunting): start opening mon connection 2024-09-16T03:54:06.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.244+0000 7faa677fe640 1 -- 172.21.15.42:0/607439995 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7faa6809edf0 con 0x7faa540039f0 2024-09-16T03:54:06.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.244+0000 7faa677fe640 10 monclient(hunting): start opening mon connection 2024-09-16T03:54:06.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.244+0000 7faa677fe640 1 -- 172.21.15.42:0/607439995 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7faa54005630 con 0x7faa54004e70 2024-09-16T03:54:06.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.244+0000 7faa677fe640 10 monclient(hunting): start opening mon connection 2024-09-16T03:54:06.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.244+0000 7faa677fe640 1 -- 172.21.15.42:0/607439995 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7faa5400bf40 con 0x7faa680a1710 2024-09-16T03:54:06.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.244+0000 7faa677fe640 10 monclient(hunting): _renew_subs 2024-09-16T03:54:06.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.244+0000 7faa677fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:54:06.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.244+0000 7faa5f7fe640 1 -- 172.21.15.42:0/607439995 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7faa680a1710 msgr2=0x7faa54004810 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T03:54:06.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.244+0000 7faa5f7fe640 1 --2- 172.21.15.42:0/607439995 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7faa680a1710 0x7faa54004810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:54:06.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.244+0000 7faa6e54b640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-16T03:54:06.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.244+0000 7faa67fff640 1 --2- 172.21.15.42:0/607439995 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faa540039f0 0x7faa54003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:54:06.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.244+0000 7faa67fff640 10 monclient(hunting): get_auth_request con 0x7faa540039f0 auth_method 0 2024-09-16T03:54:06.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.244+0000 7faa67fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:54:06.271 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.244+0000 7faa67fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:54:06.271 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.244+0000 7faa67fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:54:06.271 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.244+0000 7faa67fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:54:06.271 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.244+0000 7faa67fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:54:06.271 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.244+0000 7faa67fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T03:54:06.271 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.244+0000 7faa6cae8640 1 --2- 172.21.15.42:0/607439995 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faa54004e70 0x7faa5400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:54:06.271 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.244+0000 7faa6cae8640 10 monclient(hunting): get_auth_request con 0x7faa54004e70 auth_method 0 2024-09-16T03:54:06.271 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.244+0000 7faa6cae8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:54:06.271 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.244+0000 7faa6cae8640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:54:06.271 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.244+0000 7faa6cae8640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:54:06.271 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.244+0000 7faa67fff640 10 monclient(hunting): handle_auth_done global_id 824403 payload 293 2024-09-16T03:54:06.271 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.244+0000 7faa67fff640 1 -- 172.21.15.42:0/607439995 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7faa680a1710 msgr2=0x7faa54004810 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:54:06.271 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.244+0000 7faa67fff640 1 --2- 172.21.15.42:0/607439995 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7faa680a1710 0x7faa54004810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:54:06.271 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.244+0000 7faa67fff640 1 -- 172.21.15.42:0/607439995 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faa54004e70 msgr2=0x7faa5400b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:54:06.272 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.244+0000 7faa67fff640 1 --2- 172.21.15.42:0/607439995 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faa54004e70 0x7faa5400b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:54:06.272 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.244+0000 7faa67fff640 10 monclient: _finish_hunting 0 2024-09-16T03:54:06.272 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.244+0000 7faa67fff640 1 monclient: found mon.a 2024-09-16T03:54:06.272 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.244+0000 7faa67fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:54:06.272 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.244+0000 7faa67fff640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:54:06.272 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.244+0000 7faa67fff640 1 -- 172.21.15.42:0/607439995 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa54003530 con 0x7faa540039f0 2024-09-16T03:54:06.272 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.244+0000 7faa67fff640 10 monclient: _finish_auth 0 2024-09-16T03:54:06.272 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.244+0000 7faa67fff640 10 monclient: _check_auth_tickets 2024-09-16T03:54:06.272 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.244+0000 7faa67fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:54:06.273 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.248+0000 7faa67fff640 1 --2- 172.21.15.42:0/607439995 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faa540039f0 0x7faa54003e10 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7faa5800c1c0 tx=0x7faa5801cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a26e48f7e33a71d8 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:54:06.273 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.248+0000 7faa677fe640 1 -- 172.21.15.42:0/607439995 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa58007d00 con 0x7faa540039f0 2024-09-16T03:54:06.273 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.248+0000 7faa677fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:54:06.273 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.248+0000 7faa677fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:54:06.273 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.248+0000 7faa677fe640 10 monclient: dump: 2024-09-16T03:54:06.273 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:54:06.273 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:54:06.273 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:06.273 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:06.273 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:54:06.273 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:54:06.273 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:54:06.273 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:54:06.273 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:54:06.274 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:54:06.274 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.248+0000 7faa677fe640 1 -- 172.21.15.42:0/607439995 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa580a9020 con 0x7faa540039f0 2024-09-16T03:54:06.274 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.248+0000 7faa6e54b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:54:06.274 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.248+0000 7faa677fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:54:06.274 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.248+0000 7faa6e54b640 10 monclient: get_monmap_and_config success 2024-09-16T03:54:06.274 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.248+0000 7faa6e54b640 4 set_mon_vals no callback set 2024-09-16T03:54:06.274 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.248+0000 7faa677fe640 1 -- 172.21.15.42:0/607439995 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa58005b00 con 0x7faa540039f0 2024-09-16T03:54:06.274 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.248+0000 7faa677fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:54:06.274 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.248+0000 7faa677fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:54:06.275 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.248+0000 7faa677fe640 10 monclient: dump: 2024-09-16T03:54:06.275 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:54:06.275 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:54:06.275 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:06.275 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:06.275 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:54:06.275 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:54:06.275 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:54:06.275 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:54:06.275 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:54:06.275 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:54:06.275 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.248+0000 7faa6e54b640 10 monclient: shutdown 2024-09-16T03:54:06.275 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.248+0000 7faa6e54b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:54:06.275 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.248+0000 7faa6e54b640 1 -- 172.21.15.42:0/607439995 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faa540039f0 msgr2=0x7faa54003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:54:06.276 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.248+0000 7faa6e54b640 1 --2- 172.21.15.42:0/607439995 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faa540039f0 0x7faa54003e10 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7faa5800c1c0 tx=0x7faa5801cc20 comp rx=0 tx=0).stop 2024-09-16T03:54:06.276 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.248+0000 7faa6cae8640 1 -- 172.21.15.42:0/607439995 reap_dead start 2024-09-16T03:54:06.276 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.248+0000 7faa6e54b640 1 -- 172.21.15.42:0/607439995 shutdown_connections 2024-09-16T03:54:06.276 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.248+0000 7faa6e54b640 1 -- 172.21.15.42:0/607439995 >> 172.21.15.42:0/607439995 conn(0x7faa680a2420 msgr2=0x7faa680a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:54:06.276 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.248+0000 7faa6e54b640 1 -- 172.21.15.42:0/607439995 shutdown_connections 2024-09-16T03:54:06.276 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.248+0000 7faa6e54b640 1 -- 172.21.15.42:0/607439995 wait complete. 2024-09-16T03:54:06.276 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.248+0000 7faa6e54b640 10 monclient: build_initial_monmap 2024-09-16T03:54:06.276 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.248+0000 7faa6e54b640 10 monclient: monmap: 2024-09-16T03:54:06.276 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:54:06.276 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:54:06.277 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:54:06.278 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:54:06.278 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:54:06.278 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:54:06.278 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:54:06.278 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:54:06.278 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:54:06.278 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:54:06.278 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.248+0000 7faa6e54b640 1 librados: starting msgr at 2024-09-16T03:54:06.278 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.248+0000 7faa6e54b640 1 librados: starting objecter 2024-09-16T03:54:06.278 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.248+0000 7faa6e54b640 1 Processor -- start 2024-09-16T03:54:06.278 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.248+0000 7faa6e54b640 1 -- start start 2024-09-16T03:54:06.278 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.248+0000 7faa6e54b640 1 librados: setting wanted keys 2024-09-16T03:54:06.279 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.248+0000 7faa6e54b640 1 librados: calling monclient init 2024-09-16T03:54:06.279 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.248+0000 7faa6e54b640 10 monclient: init 2024-09-16T03:54:06.279 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.248+0000 7faa6e54b640 10 monclient: _reopen_session rank -1 2024-09-16T03:54:06.279 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.248+0000 7faa6e54b640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:54:06.279 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.248+0000 7faa6e54b640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7faa540039f0 0x7faa680713a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:54:06.279 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.248+0000 7faa6e54b640 10 monclient(hunting): picked mon.noname-c con 0x7faa540039f0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:54:06.279 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.248+0000 7faa6e54b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faa68131f80 0x7faa68132380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:54:06.279 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.248+0000 7faa6e54b640 10 monclient(hunting): picked mon.noname-b con 0x7faa68131f80 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:54:06.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.248+0000 7faa6e54b640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faa681cce40 0x7faa681cd2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:54:06.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.248+0000 7faa6e54b640 10 monclient(hunting): picked mon.noname-a con 0x7faa681cce40 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:54:06.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.248+0000 7faa6e54b640 10 monclient(hunting): start opening mon connection 2024-09-16T03:54:06.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.248+0000 7faa6e54b640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7faa6813d8d0 con 0x7faa681cce40 2024-09-16T03:54:06.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.248+0000 7faa6e54b640 10 monclient(hunting): start opening mon connection 2024-09-16T03:54:06.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.248+0000 7faa6e54b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7faa6809cd60 con 0x7faa68131f80 2024-09-16T03:54:06.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.248+0000 7faa6e54b640 10 monclient(hunting): start opening mon connection 2024-09-16T03:54:06.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.248+0000 7faa6e54b640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7faa6813da50 con 0x7faa540039f0 2024-09-16T03:54:06.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.248+0000 7faa6e54b640 10 monclient(hunting): _renew_subs 2024-09-16T03:54:06.281 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.248+0000 7faa6e54b640 10 monclient(hunting): authenticate will time out at 1693.222734s 2024-09-16T03:54:06.281 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.248+0000 7faa67fff640 1 -- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7faa540039f0 msgr2=0x7faa680713a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T03:54:06.281 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.248+0000 7faa6cae8640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faa681cce40 0x7faa681cd2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:54:06.281 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.248+0000 7faa67fff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7faa540039f0 0x7faa680713a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:54:06.281 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.248+0000 7faa6cae8640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faa681cce40 0x7faa681cd2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:53074/0 (socket says 172.21.15.42:53074) 2024-09-16T03:54:06.281 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.248+0000 7faa6cae8640 1 -- 172.21.15.42:0/1197543974 learned_addr learned my addr 172.21.15.42:0/1197543974 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:54:06.281 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.248+0000 7faa6cae8640 10 monclient(hunting): get_auth_request con 0x7faa681cce40 auth_method 0 2024-09-16T03:54:06.281 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.248+0000 7faa6cae8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:54:06.281 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.248+0000 7faa6cae8640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:54:06.281 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.248+0000 7faa6cae8640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:54:06.281 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.248+0000 7faa6cae8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:54:06.281 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.248+0000 7faa6cae8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:54:06.281 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.248+0000 7faa6cae8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:54:06.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.248+0000 7faa6cae8640 10 monclient(hunting): handle_auth_done global_id 824406 payload 824 2024-09-16T03:54:06.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.248+0000 7faa6cae8640 1 -- 172.21.15.42:0/1197543974 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7faa540039f0 msgr2=0x7faa680713a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:54:06.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.248+0000 7faa6cae8640 1 --2- 172.21.15.42:0/1197543974 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7faa540039f0 0x7faa680713a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:54:06.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.248+0000 7faa5f7fe640 1 --2- 172.21.15.42:0/1197543974 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faa68131f80 0x7faa68132380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:54:06.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.248+0000 7faa6cae8640 1 -- 172.21.15.42:0/1197543974 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faa68131f80 msgr2=0x7faa68132380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:54:06.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.248+0000 7faa6cae8640 1 --2- 172.21.15.42:0/1197543974 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faa68131f80 0x7faa68132380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:54:06.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.248+0000 7faa6cae8640 10 monclient: _finish_hunting 0 2024-09-16T03:54:06.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.248+0000 7faa6cae8640 1 monclient: found mon.noname-a 2024-09-16T03:54:06.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.248+0000 7faa6cae8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:54:06.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.248+0000 7faa6cae8640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:54:06.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.248+0000 7faa6cae8640 1 -- 172.21.15.42:0/1197543974 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa6811d1c0 con 0x7faa681cce40 2024-09-16T03:54:06.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.248+0000 7faa5f7fe640 10 monclient: get_auth_request con 0x7faa68131f80 auth_method 0 2024-09-16T03:54:06.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.248+0000 7faa5f7fe640 1 --2- 172.21.15.42:0/1197543974 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faa68131f80 0x7faa68132380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T03:54:06.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.248+0000 7faa6cae8640 1 --2- 172.21.15.42:0/1197543974 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faa681cce40 0x7faa681cd2c0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7faa6809cba0 tx=0x7faa6000ec10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:54:06.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.248+0000 7faa657fa640 10 client.?.objecter ms_handle_connect 0x7faa681cce40 2024-09-16T03:54:06.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.248+0000 7faa657fa640 10 client.?.objecter resend_mon_ops 2024-09-16T03:54:06.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.248+0000 7faa657fa640 1 -- 172.21.15.42:0/1197543974 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa60006b60 con 0x7faa681cce40 2024-09-16T03:54:06.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.248+0000 7faa657fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:54:06.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.248+0000 7faa657fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:54:06.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.248+0000 7faa657fa640 10 monclient: dump: 2024-09-16T03:54:06.283 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:54:06.283 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:54:06.283 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:06.283 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:06.283 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:54:06.283 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:54:06.283 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:54:06.284 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:54:06.284 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:54:06.284 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:54:06.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.248+0000 7faa657fa640 10 monclient: _finish_auth 0 2024-09-16T03:54:06.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.248+0000 7faa657fa640 10 monclient: _check_auth_tickets 2024-09-16T03:54:06.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.248+0000 7faa657fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:54:06.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.248+0000 7faa6e54b640 5 monclient: authenticate success, global_id 824406 2024-09-16T03:54:06.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.248+0000 7faa657fa640 1 -- 172.21.15.42:0/1197543974 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa60006d20 con 0x7faa681cce40 2024-09-16T03:54:06.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.248+0000 7faa657fa640 10 monclient: handle_config config(0 keys) 2024-09-16T03:54:06.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.248+0000 7faa6e54b640 10 monclient: _renew_subs 2024-09-16T03:54:06.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.248+0000 7faa657fa640 1 -- 172.21.15.42:0/1197543974 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa60003980 con 0x7faa681cce40 2024-09-16T03:54:06.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.248+0000 7faa6e54b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:54:06.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.248+0000 7faa6e54b640 1 -- 172.21.15.42:0/1197543974 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa681365b0 con 0x7faa681cce40 2024-09-16T03:54:06.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.248+0000 7faa657fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:54:06.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.248+0000 7faa657fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:54:06.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.248+0000 7faa657fa640 10 monclient: dump: 2024-09-16T03:54:06.285 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:54:06.285 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:54:06.285 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:06.285 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:06.285 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:54:06.285 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:54:06.285 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:54:06.285 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:54:06.285 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:54:06.286 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:54:06.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.248+0000 7faa66ffd640 4 set_mon_vals no callback set 2024-09-16T03:54:06.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.248+0000 7faa6e54b640 10 client.824406.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:54:06.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.248+0000 7faa6e54b640 10 monclient: _renew_subs 2024-09-16T03:54:06.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.248+0000 7faa6e54b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:54:06.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.248+0000 7faa6e54b640 1 -- 172.21.15.42:0/1197543974 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa6811e950 con 0x7faa681cce40 2024-09-16T03:54:06.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.248+0000 7faa6e54b640 1 librados: init done 2024-09-16T03:54:06.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.252+0000 7faa6e54b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:54:06.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.252+0000 7faa6e54b640 1 --2- 172.21.15.42:0/1197543974 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faa681cd800 0x7faa6806d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:54:06.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.252+0000 7faa6e54b640 10 monclient(hunting): start opening mon connection 2024-09-16T03:54:06.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.252+0000 7faa6e54b640 1 -- 172.21.15.42:0/1197543974 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7faa68073a30 con 0x7faa681cd800 2024-09-16T03:54:06.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.252+0000 7faa657fa640 1 -- 172.21.15.42:0/1197543974 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7faa6001a020 con 0x7faa681cce40 2024-09-16T03:54:06.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.252+0000 7faa67fff640 1 --2- 172.21.15.42:0/1197543974 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faa681cd800 0x7faa6806d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:54:06.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.252+0000 7faa67fff640 10 monclient: get_auth_request con 0x7faa681cd800 auth_method 0 2024-09-16T03:54:06.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.252+0000 7faa67fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:54:06.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.252+0000 7faa67fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:54:06.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.252+0000 7faa67fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:54:06.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.252+0000 7faa67fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:54:06.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.252+0000 7faa67fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:54:06.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.252+0000 7faa67fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:54:06.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.252+0000 7faa657fa640 1 --2- 172.21.15.42:0/1197543974 >> v2:172.21.15.154:6812/254845342 conn(0x7faa480377f0 0x7faa48039cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:54:06.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.252+0000 7faa657fa640 1 -- 172.21.15.42:0/1197543974 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(366..366 src has 286..366) ==== 5713+0+0 (secure 0 0 0) 0x7faa60048700 con 0x7faa681cce40 2024-09-16T03:54:06.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.252+0000 7faa657fa640 10 client.824406.objecter ms_dispatch 0x7faa68000ba0 osd_map(366..366 src has 286..366) 2024-09-16T03:54:06.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.252+0000 7faa657fa640 3 client.824406.objecter handle_osd_map got epochs [366,366] > 0 2024-09-16T03:54:06.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.252+0000 7faa657fa640 3 client.824406.objecter handle_osd_map decoding full epoch 366 2024-09-16T03:54:06.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.252+0000 7faa67fff640 10 monclient(hunting): handle_auth_done global_id 824409 payload 824 2024-09-16T03:54:06.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.252+0000 7faa67fff640 1 -- 172.21.15.42:0/1197543974 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7faa68136ae0 con 0x7faa681cd800 2024-09-16T03:54:06.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.252+0000 7faa657fa640 20 client.824406.objecter dump_active .. 0 homeless 2024-09-16T03:54:06.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.252+0000 7faa657fa640 10 client.824406.objecter ms_handle_connect 0x7faa681cd800 2024-09-16T03:54:06.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.252+0000 7faa657fa640 10 client.824406.objecter resend_mon_ops 2024-09-16T03:54:06.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.252+0000 7faa67fff640 1 --2- 172.21.15.42:0/1197543974 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faa681cd800 0x7faa6806d900 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7faa5800c190 tx=0x7faa54005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:54:06.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.252+0000 7faa5f7fe640 1 --2- 172.21.15.42:0/1197543974 >> v2:172.21.15.154:6812/254845342 conn(0x7faa480377f0 0x7faa48039cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:54:06.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.252+0000 7faa5f7fe640 10 monclient: get_auth_request con 0x7faa480377f0 auth_method 0 2024-09-16T03:54:06.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.252+0000 7faa657fa640 1 -- 172.21.15.42:0/1197543974 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa580a5810 con 0x7faa681cd800 2024-09-16T03:54:06.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.252+0000 7faa657fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:54:06.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.252+0000 7faa657fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:54:06.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.252+0000 7faa657fa640 10 monclient: dump: 2024-09-16T03:54:06.288 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:54:06.288 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:54:06.288 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:06.289 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:06.289 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:54:06.289 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:54:06.289 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:54:06.289 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:54:06.289 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:54:06.289 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:54:06.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.252+0000 7faa657fa640 1 -- 172.21.15.42:0/1197543974 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7faa580a9050 con 0x7faa681cd800 2024-09-16T03:54:06.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.252+0000 7faa657fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:54:06.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.252+0000 7faa657fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:54:06.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.252+0000 7faa657fa640 1 -- 172.21.15.42:0/1197543974 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faa681cd800 msgr2=0x7faa6806d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:54:06.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.252+0000 7faa657fa640 1 --2- 172.21.15.42:0/1197543974 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faa681cd800 0x7faa6806d900 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7faa5800c190 tx=0x7faa54005840 comp rx=0 tx=0).stop 2024-09-16T03:54:06.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.252+0000 7faa657fa640 1 -- 172.21.15.42:0/1197543974 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faa681cd800 msgr2=0x7faa6806d900 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:54:06.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.252+0000 7faa657fa640 1 --2- 172.21.15.42:0/1197543974 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faa681cd800 0x7faa6806d900 secure :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7faa5800c190 tx=0x7faa54005840 comp rx=0 tx=0).stop 2024-09-16T03:54:06.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.252+0000 7faa657fa640 10 client.824406.objecter ms_handle_connect 0x7faa480377f0 2024-09-16T03:54:06.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.252+0000 7faa5f7fe640 1 --2- 172.21.15.42:0/1197543974 >> v2:172.21.15.154:6812/254845342 conn(0x7faa480377f0 0x7faa48039cb0 secure :-1 s=READY pgs=3609 cs=0 l=1 rev1=1 crypto rx=0x7faa680728b0 tx=0x7faa50008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:54:06.291 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.256+0000 7faa6e54b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:54:06.292 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.256+0000 7faa6e54b640 1 --2- 172.21.15.42:0/1197543974 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faa6806de40 0x7faa6806e220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:54:06.292 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.256+0000 7faa6e54b640 10 monclient(hunting): start opening mon connection 2024-09-16T03:54:06.292 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.256+0000 7faa6e54b640 1 -- 172.21.15.42:0/1197543974 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7faa6809f180 con 0x7faa6806de40 2024-09-16T03:54:06.292 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.256+0000 7faa67fff640 1 --2- 172.21.15.42:0/1197543974 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faa6806de40 0x7faa6806e220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:54:06.292 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.256+0000 7faa67fff640 10 monclient: get_auth_request con 0x7faa6806de40 auth_method 0 2024-09-16T03:54:06.292 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.256+0000 7faa67fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:54:06.292 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.256+0000 7faa67fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:54:06.292 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.256+0000 7faa67fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:54:06.293 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.256+0000 7faa67fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:54:06.293 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.256+0000 7faa67fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:54:06.293 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.256+0000 7faa67fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:54:06.293 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.256+0000 7faa67fff640 10 monclient(hunting): handle_auth_done global_id 824412 payload 824 2024-09-16T03:54:06.293 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.256+0000 7faa67fff640 1 -- 172.21.15.42:0/1197543974 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7faa6811c440 con 0x7faa6806de40 2024-09-16T03:54:06.293 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.256+0000 7faa67fff640 1 --2- 172.21.15.42:0/1197543974 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faa6806de40 0x7faa6806e220 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7faa58000fc0 tx=0x7faa58007900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:54:06.293 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.256+0000 7faa657fa640 10 client.824406.objecter ms_handle_connect 0x7faa6806de40 2024-09-16T03:54:06.293 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.256+0000 7faa657fa640 10 client.824406.objecter resend_mon_ops 2024-09-16T03:54:06.293 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.260+0000 7faa657fa640 1 -- 172.21.15.42:0/1197543974 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa58004e40 con 0x7faa6806de40 2024-09-16T03:54:06.293 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.260+0000 7faa657fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:54:06.294 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.260+0000 7faa657fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:54:06.294 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.260+0000 7faa657fa640 10 monclient: dump: 2024-09-16T03:54:06.294 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:54:06.294 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:54:06.294 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:06.294 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:06.294 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:54:06.294 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:54:06.294 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:54:06.294 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:54:06.294 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:54:06.294 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:54:06.294 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.260+0000 7faa657fa640 1 -- 172.21.15.42:0/1197543974 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4871 (secure 0 0 0) 0x7faa580072e0 con 0x7faa6806de40 2024-09-16T03:54:06.294 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.260+0000 7faa657fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:54:06.295 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.260+0000 7faa657fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:54:06.295 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.260+0000 7faa657fa640 1 -- 172.21.15.42:0/1197543974 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faa6806de40 msgr2=0x7faa6806e220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:54:06.295 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.260+0000 7faa657fa640 1 --2- 172.21.15.42:0/1197543974 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faa6806de40 0x7faa6806e220 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7faa58000fc0 tx=0x7faa58007900 comp rx=0 tx=0).stop 2024-09-16T03:54:06.295 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.260+0000 7faa657fa640 1 -- 172.21.15.42:0/1197543974 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faa6806de40 msgr2=0x7faa6806e220 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:54:06.295 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.260+0000 7faa657fa640 1 --2- 172.21.15.42:0/1197543974 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faa6806de40 0x7faa6806e220 secure :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7faa58000fc0 tx=0x7faa58007900 comp rx=0 tx=0).stop 2024-09-16T03:54:06.295 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:54:06.295 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:54:06.295 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:54:06.295 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "leader", 2024-09-16T03:54:06.296 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 102, 2024-09-16T03:54:06.296 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 136114, 2024-09-16T03:54:06.296 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:54:06.296 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:54:06.296 INFO:teuthology.orchestra.run.smithi042.stdout: 1 2024-09-16T03:54:06.296 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:54:06.296 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 0, 2024-09-16T03:54:06.296 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:54:06.296 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:54:06.296 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:54:06.296 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:54:06.296 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:54:06.296 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:54:06.296 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:54:06.297 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:54:06.297 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:54:06.297 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:54:06.297 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:54:06.297 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:54:06.297 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:54:06.297 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:54:06.297 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:54:06.297 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:54:06.297 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:54:06.297 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:54:06.297 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:54:06.297 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:54:06.298 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:54:06.298 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:54:06.298 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:54:06.298 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:54:06.298 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:54:06.298 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:54:06.298 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:54:06.298 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:54:06.298 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:54:06.298 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:54:06.298 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:54:06.298 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:54:06.298 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:54:06.298 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:54:06.299 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:54:06.299 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:54:06.299 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:54:06.299 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:54:06.299 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:54:06.299 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:54:06.299 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:54:06.299 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:54:06.299 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:54:06.299 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:54:06.299 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:54:06.299 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:54:06.299 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:54:06.300 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:54:06.300 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:54:06.300 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:54:06.300 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:54:06.300 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:54:06.300 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:54:06.300 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:54:06.300 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:54:06.300 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:54:06.300 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:54:06.300 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:54:06.300 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:54:06.300 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:54:06.301 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:54:06.301 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:54:06.301 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:54:06.301 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:54:06.301 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:54:06.301 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:54:06.301 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:54:06.301 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:54:06.301 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:54:06.301 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:54:06.301 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:54:06.301 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:54:06.301 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:54:06.302 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:54:06.302 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:54:06.302 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:54:06.302 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:54:06.302 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:54:06.302 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:54:06.302 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:54:06.302 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:54:06.302 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:54:06.302 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:54:06.302 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:54:06.302 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:54:06.302 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:54:06.302 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:54:06.303 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:54:06.303 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:54:06.303 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:54:06.303 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:54:06.303 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:54:06.303 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:54:06.303 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:54:06.303 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:54:06.303 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:54:06.303 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:54:06.303 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:54:06.303 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:54:06.303 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:54:06.303 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:54:06.304 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:54:06.304 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:54:06.304 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:54:06.304 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:54:06.304 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:54:06.304 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:54:06.304 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:54:06.304 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:54:06.304 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:54:06.304 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:54:06.304 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:54:06.304 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:54:06.304 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:54:06.305 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:54:06.305 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:54:06.305 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:54:06.305 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:54:06.305 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:54:06.305 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:54:06.305 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:54:06.305 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:54:06.305 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:54:06.305 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:54:06.305 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:54:06.305 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:54:06.305 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:54:06.306 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:54:06.306 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:54:06.306 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:54:06.306 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:54:06.306 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:54:06.306 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:54:06.306 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:54:06.306 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:54:06.306 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:54:06.306 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:54:06.306 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:54:06.306 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:54:06.306 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:54:06.307 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:54:06.307 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T03:54:06.307 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:54:06.307 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:54:06.307 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:54:06.307 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 6 2024-09-16T03:54:06.307 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:54:06.307 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:54:06.307 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:54:06.307 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:54:06.307 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:54:06.307 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:54:06.307 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 7 2024-09-16T03:54:06.307 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:54:06.308 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:54:06.308 INFO:teuthology.orchestra.run.smithi042.stdout: "mgr": [ 2024-09-16T03:54:06.308 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:54:06.308 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:54:06.308 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:54:06.308 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:54:06.308 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:54:06.308 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:54:06.308 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:54:06.308 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:54:06.308 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:54:06.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.260+0000 7faa6e54b640 10 librados: watch_flush enter 2024-09-16T03:54:06.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.260+0000 7faa6e54b640 10 librados: watch_flush exit 2024-09-16T03:54:06.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.260+0000 7faa6e54b640 20 client.824406.objecter shutdown clearing up homeless session... 2024-09-16T03:54:06.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.260+0000 7faa6e54b640 10 client.824406.objecter successfully canceled tick 2024-09-16T03:54:06.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.260+0000 7faa6e54b640 1 -- 172.21.15.42:0/1197543974 >> v2:172.21.15.154:6812/254845342 conn(0x7faa480377f0 msgr2=0x7faa48039cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:54:06.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.260+0000 7faa6e54b640 1 --2- 172.21.15.42:0/1197543974 >> v2:172.21.15.154:6812/254845342 conn(0x7faa480377f0 0x7faa48039cb0 secure :-1 s=READY pgs=3609 cs=0 l=1 rev1=1 crypto rx=0x7faa680728b0 tx=0x7faa50008040 comp rx=0 tx=0).stop 2024-09-16T03:54:06.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.260+0000 7faa6e54b640 10 monclient: shutdown 2024-09-16T03:54:06.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.260+0000 7faa6e54b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:54:06.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.260+0000 7faa6e54b640 1 -- 172.21.15.42:0/1197543974 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faa681cce40 msgr2=0x7faa681cd2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:54:06.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.260+0000 7faa6e54b640 1 --2- 172.21.15.42:0/1197543974 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faa681cce40 0x7faa681cd2c0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7faa6809cba0 tx=0x7faa6000ec10 comp rx=0 tx=0).stop 2024-09-16T03:54:06.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.260+0000 7faa6cae8640 1 -- 172.21.15.42:0/1197543974 reap_dead start 2024-09-16T03:54:06.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.260+0000 7faa6cae8640 1 -- 172.21.15.42:0/1197543974 reap_dead start 2024-09-16T03:54:06.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.260+0000 7faa6e54b640 1 -- 172.21.15.42:0/1197543974 shutdown_connections 2024-09-16T03:54:06.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.260+0000 7faa6e54b640 1 -- 172.21.15.42:0/1197543974 >> 172.21.15.42:0/1197543974 conn(0x7faa680a2420 msgr2=0x7faa6809ba50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:54:06.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.260+0000 7faa6e54b640 1 -- 172.21.15.42:0/1197543974 shutdown_connections 2024-09-16T03:54:06.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.260+0000 7faa6e54b640 1 -- 172.21.15.42:0/1197543974 wait complete. 2024-09-16T03:54:06.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.260+0000 7faa6e54b640 1 librados: shutdown 2024-09-16T03:54:06.311 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T03:54:06.312 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-16T03:54:06.312 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T03:54:06.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.416+0000 7f642afa3640 10 monclient: get_monmap_and_config 2024-09-16T03:54:06.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.416+0000 7f642afa3640 10 monclient: build_initial_monmap 2024-09-16T03:54:06.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.416+0000 7f642afa3640 10 monclient: monmap: 2024-09-16T03:54:06.423 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:54:06.423 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:54:06.423 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:54:06.420991+0000 2024-09-16T03:54:06.423 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:54:06.420991+0000 2024-09-16T03:54:06.423 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:54:06.423 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:54:06.423 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:54:06.423 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:54:06.423 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:54:06.424 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:54:06.424 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:54:06.424 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:54:06.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.416+0000 7f642afa3640 1 Processor -- start 2024-09-16T03:54:06.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.416+0000 7f642afa3640 1 -- start start 2024-09-16T03:54:06.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.416+0000 7f642afa3640 10 monclient: init 2024-09-16T03:54:06.432 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.416+0000 7f642afa3640 10 monclient: _reopen_session rank -1 2024-09-16T03:54:06.432 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.416+0000 7f642afa3640 10 monclient: _add_conns ranks=[3,4,1,2,0] 2024-09-16T03:54:06.432 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.416+0000 7f642afa3640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f64240a39b0 0x7f6424133b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:54:06.432 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.416+0000 7f642afa3640 10 monclient(hunting): picked mon.noname-a con 0x7f64240a39b0 addr v2:172.21.15.42:3300/0 2024-09-16T03:54:06.432 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.416+0000 7f642afa3640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f64240a3d90 0x7f6424137000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:54:06.432 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.416+0000 7f642afa3640 10 monclient(hunting): picked mon.noname-c con 0x7f64240a3d90 addr v2:172.21.15.154:3300/0 2024-09-16T03:54:06.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.416+0000 7f642afa3640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6424137540 addr v1:172.21.15.154:6789/0 2024-09-16T03:54:06.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.416+0000 7f642afa3640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f642409c140 con 0x7f6424137540 2024-09-16T03:54:06.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.416+0000 7f642afa3640 10 monclient(hunting): start opening mon connection 2024-09-16T03:54:06.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.416+0000 7f642afa3640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f642409c420 con 0x7f64240a39b0 2024-09-16T03:54:06.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.416+0000 7f642afa3640 10 monclient(hunting): start opening mon connection 2024-09-16T03:54:06.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.416+0000 7f642afa3640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f642406dd60 con 0x7f64240a3d90 2024-09-16T03:54:06.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.416+0000 7f642afa3640 10 monclient(hunting): _renew_subs 2024-09-16T03:54:06.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.416+0000 7f642afa3640 10 monclient(hunting): authenticate will time out at 1693.390031s 2024-09-16T03:54:06.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.416+0000 7f6428d3f640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f64240a39b0 0x7f6424133b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:54:06.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.416+0000 7f6428d3f640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f64240a39b0 0x7f6424133b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:53112/0 (socket says 172.21.15.42:53112) 2024-09-16T03:54:06.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.416+0000 7f6428d3f640 1 -- 172.21.15.42:0/2266908755 learned_addr learned my addr 172.21.15.42:0/2266908755 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:54:06.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.416+0000 7f6428d3f640 10 monclient(hunting): get_auth_request con 0x7f64240a39b0 auth_method 0 2024-09-16T03:54:06.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.416+0000 7f6428d3f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:54:06.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.416+0000 7f6428d3f640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:54:06.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.416+0000 7f6428d3f640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:54:06.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.416+0000 7f6428d3f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:54:06.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.416+0000 7f6428d3f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:54:06.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.416+0000 7f6428d3f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:54:06.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.416+0000 7f6423fff640 1 --2- 172.21.15.42:0/2266908755 >> v2:172.21.15.154:3300/0 conn(0x7f64240a3d90 0x7f6424137000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:54:06.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.416+0000 7f6423fff640 10 monclient(hunting): get_auth_request con 0x7f64240a3d90 auth_method 0 2024-09-16T03:54:06.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.416+0000 7f6423fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:54:06.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.416+0000 7f6423fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:54:06.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.416+0000 7f6423fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:54:06.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.416+0000 7f6428d3f640 10 monclient(hunting): handle_auth_done global_id 824415 payload 274 2024-09-16T03:54:06.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.416+0000 7f6428d3f640 1 -- 172.21.15.42:0/2266908755 >> v2:172.21.15.154:3300/0 conn(0x7f64240a3d90 msgr2=0x7f6424137000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:54:06.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.416+0000 7f6428d3f640 1 --2- 172.21.15.42:0/2266908755 >> v2:172.21.15.154:3300/0 conn(0x7f64240a3d90 0x7f6424137000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:54:06.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.416+0000 7f6428d3f640 1 -- 172.21.15.42:0/2266908755 >> v1:172.21.15.154:6789/0 conn(0x7f6424137540 legacy=0x7f642413e390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:54:06.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.416+0000 7f6428d3f640 10 monclient: _finish_hunting 0 2024-09-16T03:54:06.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.416+0000 7f6428d3f640 1 monclient: found mon.noname-a 2024-09-16T03:54:06.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.416+0000 7f6428d3f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:54:06.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.416+0000 7f6428d3f640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:54:06.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.416+0000 7f6428d3f640 1 -- 172.21.15.42:0/2266908755 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64240a5e20 con 0x7f64240a39b0 2024-09-16T03:54:06.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.420+0000 7f6428d3f640 1 --2- 172.21.15.42:0/2266908755 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f64240a39b0 0x7f6424133b80 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f64240a56c0 tx=0x7f641401c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=968b1a54b2b18189 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:54:06.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.420+0000 7f64237fe640 1 -- 172.21.15.42:0/2266908755 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64140b0070 con 0x7f64240a39b0 2024-09-16T03:54:06.437 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.420+0000 7f64237fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:54:06.437 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.420+0000 7f64237fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:54:06.437 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.420+0000 7f64237fe640 10 monclient: dump: 2024-09-16T03:54:06.437 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:54:06.437 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:54:06.437 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:06.437 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:06.437 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:54:06.437 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:54:06.437 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:54:06.438 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:54:06.438 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:54:06.438 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:54:06.438 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.420+0000 7f64237fe640 10 monclient: _finish_auth 0 2024-09-16T03:54:06.438 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.420+0000 7f64237fe640 10 monclient: _check_auth_tickets 2024-09-16T03:54:06.438 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.420+0000 7f64237fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:54:06.439 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.420+0000 7f642afa3640 5 monclient: authenticate success, global_id 824415 2024-09-16T03:54:06.439 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.420+0000 7f642afa3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:54:06.439 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.420+0000 7f64237fe640 1 -- 172.21.15.42:0/2266908755 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f64140ab600 con 0x7f64240a39b0 2024-09-16T03:54:06.439 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.420+0000 7f64237fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:54:06.439 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.420+0000 7f642afa3640 10 monclient: get_monmap_and_config success 2024-09-16T03:54:06.440 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.420+0000 7f642afa3640 4 set_mon_vals no callback set 2024-09-16T03:54:06.440 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.420+0000 7f64237fe640 1 -- 172.21.15.42:0/2266908755 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64140ab9d0 con 0x7f64240a39b0 2024-09-16T03:54:06.440 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.420+0000 7f64237fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:54:06.440 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.420+0000 7f64237fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:54:06.440 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.420+0000 7f64237fe640 10 monclient: dump: 2024-09-16T03:54:06.440 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:54:06.440 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:54:06.440 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:06.441 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:06.441 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:54:06.441 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:54:06.441 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:54:06.441 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:54:06.441 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:54:06.441 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:54:06.441 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.420+0000 7f642afa3640 10 monclient: shutdown 2024-09-16T03:54:06.441 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.420+0000 7f642afa3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:54:06.441 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.420+0000 7f642afa3640 1 -- 172.21.15.42:0/2266908755 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f64240a39b0 msgr2=0x7f6424133b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:54:06.441 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.420+0000 7f642afa3640 1 --2- 172.21.15.42:0/2266908755 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f64240a39b0 0x7f6424133b80 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f64240a56c0 tx=0x7f641401c3a0 comp rx=0 tx=0).stop 2024-09-16T03:54:06.441 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.420+0000 7f642afa3640 1 -- 172.21.15.42:0/2266908755 shutdown_connections 2024-09-16T03:54:06.441 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.420+0000 7f642afa3640 1 --2- 172.21.15.42:0/2266908755 >> v2:172.21.15.154:3300/0 conn(0x7f64240a3d90 0x7f6424137000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:54:06.442 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.420+0000 7f642afa3640 1 --2- 172.21.15.42:0/2266908755 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f64240a39b0 0x7f6424133b80 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:54:06.442 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.420+0000 7f642afa3640 1 -- 172.21.15.42:0/2266908755 >> 172.21.15.42:0/2266908755 conn(0x7f64240a9fb0 msgr2=0x7f64240aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:54:06.442 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.420+0000 7f642afa3640 1 -- 172.21.15.42:0/2266908755 shutdown_connections 2024-09-16T03:54:06.442 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.420+0000 7f642afa3640 1 -- 172.21.15.42:0/2266908755 wait complete. 2024-09-16T03:54:06.442 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.420+0000 7f642afa3640 10 monclient: build_initial_monmap 2024-09-16T03:54:06.442 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.420+0000 7f642afa3640 10 monclient: monmap: 2024-09-16T03:54:06.442 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:54:06.442 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:54:06.442 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:54:06.442 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:54:06.442 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:54:06.442 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:54:06.442 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:54:06.442 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:54:06.443 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:54:06.443 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:54:06.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.420+0000 7f642afa3640 1 librados: starting msgr at 2024-09-16T03:54:06.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.420+0000 7f642afa3640 1 librados: starting objecter 2024-09-16T03:54:06.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.420+0000 7f642afa3640 1 Processor -- start 2024-09-16T03:54:06.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.420+0000 7f642afa3640 1 -- start start 2024-09-16T03:54:06.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.420+0000 7f642afa3640 1 librados: setting wanted keys 2024-09-16T03:54:06.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.420+0000 7f642afa3640 1 librados: calling monclient init 2024-09-16T03:54:06.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.420+0000 7f642afa3640 10 monclient: init 2024-09-16T03:54:06.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.420+0000 7f642afa3640 10 monclient: _reopen_session rank -1 2024-09-16T03:54:06.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.420+0000 7f642afa3640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T03:54:06.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.420+0000 7f642afa3640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f64240a39b0 0x7f64241cf960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:54:06.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.420+0000 7f642afa3640 10 monclient(hunting): picked mon.noname-a con 0x7f64240a39b0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:54:06.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.420+0000 7f642afa3640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f64240a3d90 0x7f642413e870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:54:06.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.420+0000 7f642afa3640 10 monclient(hunting): picked mon.noname-b con 0x7f64240a3d90 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:54:06.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.420+0000 7f642afa3640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f6424137540 0x7f64240a1d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:54:06.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.420+0000 7f642afa3640 10 monclient(hunting): picked mon.noname-c con 0x7f6424137540 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:54:06.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.420+0000 7f642afa3640 10 monclient(hunting): start opening mon connection 2024-09-16T03:54:06.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.420+0000 7f642afa3640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f64241424e0 con 0x7f64240a39b0 2024-09-16T03:54:06.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.420+0000 7f642afa3640 10 monclient(hunting): start opening mon connection 2024-09-16T03:54:06.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.420+0000 7f642afa3640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f6424142660 con 0x7f64240a3d90 2024-09-16T03:54:06.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.420+0000 7f642afa3640 10 monclient(hunting): start opening mon connection 2024-09-16T03:54:06.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.420+0000 7f642afa3640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f64241427e0 con 0x7f6424137540 2024-09-16T03:54:06.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.420+0000 7f642afa3640 10 monclient(hunting): _renew_subs 2024-09-16T03:54:06.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.420+0000 7f642afa3640 10 monclient(hunting): authenticate will time out at 1693.394870s 2024-09-16T03:54:06.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.420+0000 7f6429540640 1 -- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f6424137540 msgr2=0x7f64240a1d10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T03:54:06.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.420+0000 7f6429540640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f6424137540 0x7f64240a1d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:54:06.445 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.420+0000 7f6428d3f640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f64240a39b0 0x7f64241cf960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:54:06.445 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.420+0000 7f6428d3f640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f64240a39b0 0x7f64241cf960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:53120/0 (socket says 172.21.15.42:53120) 2024-09-16T03:54:06.445 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.420+0000 7f6428d3f640 1 -- 172.21.15.42:0/876320715 learned_addr learned my addr 172.21.15.42:0/876320715 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:54:06.445 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.420+0000 7f6428d3f640 10 monclient(hunting): get_auth_request con 0x7f64240a39b0 auth_method 0 2024-09-16T03:54:06.445 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.420+0000 7f6428d3f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:54:06.445 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.420+0000 7f6428d3f640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:54:06.445 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.420+0000 7f6428d3f640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:54:06.445 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.420+0000 7f6428d3f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:54:06.445 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.420+0000 7f6428d3f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:54:06.445 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.420+0000 7f6428d3f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:54:06.446 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.420+0000 7f6428d3f640 10 monclient(hunting): handle_auth_done global_id 824418 payload 824 2024-09-16T03:54:06.446 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.420+0000 7f6428d3f640 1 -- 172.21.15.42:0/876320715 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f6424137540 msgr2=0x7f64240a1d10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:54:06.446 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.420+0000 7f6428d3f640 1 --2- 172.21.15.42:0/876320715 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f6424137540 0x7f64240a1d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:54:06.446 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.420+0000 7f6428d3f640 1 -- 172.21.15.42:0/876320715 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f64240a3d90 msgr2=0x7f642413e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:54:06.446 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.420+0000 7f6428d3f640 1 --2- 172.21.15.42:0/876320715 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f64240a3d90 0x7f642413e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:54:06.446 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.420+0000 7f6428d3f640 10 monclient: _finish_hunting 0 2024-09-16T03:54:06.446 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.420+0000 7f6428d3f640 1 monclient: found mon.noname-a 2024-09-16T03:54:06.446 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.420+0000 7f6428d3f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:54:06.446 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.420+0000 7f6428d3f640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:54:06.446 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.420+0000 7f6428d3f640 1 -- 172.21.15.42:0/876320715 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64241cfea0 con 0x7f64240a39b0 2024-09-16T03:54:06.446 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.420+0000 7f6428d3f640 1 --2- 172.21.15.42:0/876320715 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f64240a39b0 0x7f64241cf960 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f6414000c10 tx=0x7f6414019140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:54:06.446 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.420+0000 7f64217fa640 10 client.?.objecter ms_handle_connect 0x7f64240a39b0 2024-09-16T03:54:06.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.420+0000 7f64217fa640 10 client.?.objecter resend_mon_ops 2024-09-16T03:54:06.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.420+0000 7f64217fa640 1 -- 172.21.15.42:0/876320715 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64140b0040 con 0x7f64240a39b0 2024-09-16T03:54:06.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.420+0000 7f64217fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:54:06.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.420+0000 7f64217fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:54:06.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.420+0000 7f64217fa640 10 monclient: dump: 2024-09-16T03:54:06.447 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:54:06.447 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:54:06.447 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:06.447 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:06.447 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:54:06.447 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:54:06.447 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:54:06.447 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:54:06.448 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:54:06.448 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:54:06.448 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.420+0000 7f64217fa640 10 monclient: _finish_auth 0 2024-09-16T03:54:06.448 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.420+0000 7f64217fa640 10 monclient: _check_auth_tickets 2024-09-16T03:54:06.448 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.420+0000 7f64217fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:54:06.448 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.420+0000 7f64217fa640 1 -- 172.21.15.42:0/876320715 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f64140b9070 con 0x7f64240a39b0 2024-09-16T03:54:06.448 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.420+0000 7f64217fa640 10 monclient: handle_config config(0 keys) 2024-09-16T03:54:06.448 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.420+0000 7f64217fa640 1 -- 172.21.15.42:0/876320715 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64140b4830 con 0x7f64240a39b0 2024-09-16T03:54:06.448 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.420+0000 7f64217fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:54:06.448 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.420+0000 7f64217fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:54:06.448 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.420+0000 7f64217fa640 10 monclient: dump: 2024-09-16T03:54:06.448 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:54:06.448 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:54:06.448 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:06.449 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:06.449 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:54:06.449 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:54:06.449 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:54:06.449 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:54:06.449 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:54:06.449 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:54:06.449 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.420+0000 7f642afa3640 5 monclient: authenticate success, global_id 824418 2024-09-16T03:54:06.449 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.420+0000 7f642afa3640 10 monclient: _renew_subs 2024-09-16T03:54:06.449 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.420+0000 7f64227fc640 4 set_mon_vals no callback set 2024-09-16T03:54:06.449 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.420+0000 7f642afa3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:54:06.449 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.420+0000 7f642afa3640 1 -- 172.21.15.42:0/876320715 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f642406f680 con 0x7f64240a39b0 2024-09-16T03:54:06.449 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.420+0000 7f642afa3640 10 client.824418.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:54:06.450 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.420+0000 7f642afa3640 10 monclient: _renew_subs 2024-09-16T03:54:06.450 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.420+0000 7f642afa3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:54:06.450 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.420+0000 7f642afa3640 1 -- 172.21.15.42:0/876320715 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6424139d50 con 0x7f64240a39b0 2024-09-16T03:54:06.450 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.420+0000 7f642afa3640 1 librados: init done 2024-09-16T03:54:06.450 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.424+0000 7f642afa3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:54:06.450 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.424+0000 7f642afa3640 1 --2- 172.21.15.42:0/876320715 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6424139f20 0x7f642402c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:54:06.450 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.424+0000 7f642afa3640 10 monclient(hunting): start opening mon connection 2024-09-16T03:54:06.450 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.424+0000 7f642afa3640 1 -- 172.21.15.42:0/876320715 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f64241342d0 con 0x7f6424139f20 2024-09-16T03:54:06.450 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.424+0000 7f64217fa640 1 -- 172.21.15.42:0/876320715 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f64140ab480 con 0x7f64240a39b0 2024-09-16T03:54:06.450 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.424+0000 7f64217fa640 1 --2- 172.21.15.42:0/876320715 >> v2:172.21.15.154:6812/254845342 conn(0x7f64100377c0 0x7f6410039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:54:06.450 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.424+0000 7f64217fa640 1 -- 172.21.15.42:0/876320715 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(366..366 src has 286..366) ==== 5713+0+0 (secure 0 0 0) 0x7f64140e2c50 con 0x7f64240a39b0 2024-09-16T03:54:06.450 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.424+0000 7f64217fa640 10 client.824418.objecter ms_dispatch 0x7f6424000ba0 osd_map(366..366 src has 286..366) 2024-09-16T03:54:06.450 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.424+0000 7f64217fa640 3 client.824418.objecter handle_osd_map got epochs [366,366] > 0 2024-09-16T03:54:06.450 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.424+0000 7f64217fa640 3 client.824418.objecter handle_osd_map decoding full epoch 366 2024-09-16T03:54:06.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.424+0000 7f6423fff640 1 --2- 172.21.15.42:0/876320715 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6424139f20 0x7f642402c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:54:06.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.424+0000 7f64217fa640 20 client.824418.objecter dump_active .. 0 homeless 2024-09-16T03:54:06.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.424+0000 7f6423fff640 10 monclient: get_auth_request con 0x7f6424139f20 auth_method 0 2024-09-16T03:54:06.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.424+0000 7f6423fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:54:06.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.424+0000 7f6423fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:54:06.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.424+0000 7f6423fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:54:06.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.424+0000 7f6423fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:54:06.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.424+0000 7f6423fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:54:06.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.424+0000 7f6423fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:54:06.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.424+0000 7f6429540640 1 --2- 172.21.15.42:0/876320715 >> v2:172.21.15.154:6812/254845342 conn(0x7f64100377c0 0x7f6410039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:54:06.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.424+0000 7f6429540640 10 monclient: get_auth_request con 0x7f64100377c0 auth_method 0 2024-09-16T03:54:06.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.424+0000 7f6423fff640 10 monclient(hunting): handle_auth_done global_id 844159 payload 824 2024-09-16T03:54:06.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.424+0000 7f6423fff640 1 -- 172.21.15.42:0/876320715 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6424136e50 con 0x7f6424139f20 2024-09-16T03:54:06.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.424+0000 7f6429540640 1 --2- 172.21.15.42:0/876320715 >> v2:172.21.15.154:6812/254845342 conn(0x7f64100377c0 0x7f6410039c80 secure :-1 s=READY pgs=3610 cs=0 l=1 rev1=1 crypto rx=0x7f641c003b00 tx=0x7f641c007d20 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:54:06.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.424+0000 7f64217fa640 10 client.824418.objecter ms_handle_connect 0x7f64100377c0 2024-09-16T03:54:06.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.424+0000 7f64217fa640 10 client.824418.objecter ms_handle_connect 0x7f6424139f20 2024-09-16T03:54:06.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.424+0000 7f64217fa640 10 client.824418.objecter resend_mon_ops 2024-09-16T03:54:06.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.424+0000 7f6423fff640 1 --2- 172.21.15.42:0/876320715 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6424139f20 0x7f642402c650 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f641800d4b0 tx=0x7f641800d970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:54:06.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.424+0000 7f64217fa640 1 -- 172.21.15.42:0/876320715 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6418004020 con 0x7f6424139f20 2024-09-16T03:54:06.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.424+0000 7f64217fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:54:06.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.424+0000 7f64217fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:54:06.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.424+0000 7f64217fa640 10 monclient: dump: 2024-09-16T03:54:06.452 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:54:06.452 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:54:06.452 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:06.452 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:06.452 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:54:06.453 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:54:06.453 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:54:06.453 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:54:06.453 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:54:06.453 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:54:06.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.424+0000 7f64217fa640 1 -- 172.21.15.42:0/876320715 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f6418004020 con 0x7f6424139f20 2024-09-16T03:54:06.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.424+0000 7f64217fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:54:06.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.424+0000 7f64217fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:54:06.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.424+0000 7f64217fa640 1 -- 172.21.15.42:0/876320715 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6424139f20 msgr2=0x7f642402c650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:54:06.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.424+0000 7f64217fa640 1 --2- 172.21.15.42:0/876320715 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6424139f20 0x7f642402c650 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f641800d4b0 tx=0x7f641800d970 comp rx=0 tx=0).stop 2024-09-16T03:54:06.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.424+0000 7f64217fa640 1 -- 172.21.15.42:0/876320715 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6424139f20 msgr2=0x7f642402c650 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:54:06.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.424+0000 7f64217fa640 1 --2- 172.21.15.42:0/876320715 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6424139f20 0x7f642402c650 secure :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f641800d4b0 tx=0x7f641800d970 comp rx=0 tx=0).stop 2024-09-16T03:54:06.455 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.432+0000 7f642afa3640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:54:06.455 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.432+0000 7f642afa3640 1 --2- 172.21.15.42:0/876320715 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f642413edb0 0x7f642402cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:54:06.455 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.432+0000 7f642afa3640 10 monclient(hunting): start opening mon connection 2024-09-16T03:54:06.455 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.432+0000 7f642afa3640 1 -- 172.21.15.42:0/876320715 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f6424138b80 con 0x7f642413edb0 2024-09-16T03:54:06.455 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.432+0000 7f6423fff640 1 --2- 172.21.15.42:0/876320715 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f642413edb0 0x7f642402cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:54:06.455 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.432+0000 7f6423fff640 10 monclient: get_auth_request con 0x7f642413edb0 auth_method 0 2024-09-16T03:54:06.455 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.432+0000 7f6423fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:54:06.456 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.432+0000 7f6423fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:54:06.456 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.432+0000 7f6423fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:54:06.456 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.432+0000 7f6423fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:54:06.456 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.432+0000 7f6423fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:54:06.456 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.432+0000 7f6423fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:54:06.456 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.432+0000 7f6423fff640 10 monclient(hunting): handle_auth_done global_id 844162 payload 824 2024-09-16T03:54:06.456 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.432+0000 7f6423fff640 1 -- 172.21.15.42:0/876320715 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f6424133b80 con 0x7f642413edb0 2024-09-16T03:54:06.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.432+0000 7f6423fff640 1 --2- 172.21.15.42:0/876320715 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f642413edb0 0x7f642402cb90 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f6418017000 tx=0x7f64180046d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:54:06.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.432+0000 7f64217fa640 10 client.824418.objecter ms_handle_connect 0x7f642413edb0 2024-09-16T03:54:06.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.432+0000 7f64217fa640 10 client.824418.objecter resend_mon_ops 2024-09-16T03:54:06.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.432+0000 7f64217fa640 1 -- 172.21.15.42:0/876320715 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6418003740 con 0x7f642413edb0 2024-09-16T03:54:06.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.432+0000 7f64217fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:54:06.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.432+0000 7f64217fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:54:06.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.432+0000 7f64217fa640 10 monclient: dump: 2024-09-16T03:54:06.457 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:54:06.457 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:54:06.457 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:06.457 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:06.457 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:54:06.457 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:54:06.457 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:54:06.458 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:54:06.458 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:54:06.458 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:54:06.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.432+0000 7f64217fa640 1 -- 172.21.15.42:0/876320715 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4534 (secure 0 0 0) 0x7f6418003740 con 0x7f642413edb0 2024-09-16T03:54:06.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.432+0000 7f64217fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:54:06.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.432+0000 7f64217fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:54:06.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.432+0000 7f64217fa640 1 -- 172.21.15.42:0/876320715 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f642413edb0 msgr2=0x7f642402cb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:54:06.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.432+0000 7f64217fa640 1 --2- 172.21.15.42:0/876320715 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f642413edb0 0x7f642402cb90 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f6418017000 tx=0x7f64180046d0 comp rx=0 tx=0).stop 2024-09-16T03:54:06.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.432+0000 7f64217fa640 1 -- 172.21.15.42:0/876320715 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f642413edb0 msgr2=0x7f642402cb90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:54:06.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.432+0000 7f64217fa640 1 --2- 172.21.15.42:0/876320715 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f642413edb0 0x7f642402cb90 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:54:06.459 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:54:06.459 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:54:06.459 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:54:06.459 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T03:54:06.459 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 102, 2024-09-16T03:54:06.459 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 30635, 2024-09-16T03:54:06.459 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:54:06.459 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:54:06.459 INFO:teuthology.orchestra.run.smithi042.stdout: 1 2024-09-16T03:54:06.459 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:54:06.459 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 0, 2024-09-16T03:54:06.459 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:54:06.459 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:54:06.460 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:54:06.460 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:54:06.460 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:54:06.460 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:54:06.460 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:54:06.460 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:54:06.460 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:54:06.460 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:54:06.460 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:54:06.460 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:54:06.460 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:54:06.460 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:54:06.460 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:54:06.460 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:54:06.461 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:54:06.461 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:54:06.461 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:54:06.461 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:54:06.461 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:54:06.461 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:54:06.461 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:54:06.461 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:54:06.461 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:54:06.461 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:54:06.461 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:54:06.461 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:54:06.461 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:54:06.461 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:54:06.462 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:54:06.462 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:54:06.462 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:54:06.462 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:54:06.462 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:54:06.462 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:54:06.462 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:54:06.462 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:54:06.462 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:54:06.462 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:54:06.462 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:54:06.462 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:54:06.462 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:54:06.462 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:54:06.463 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:54:06.463 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:54:06.463 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:54:06.463 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:54:06.463 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:54:06.463 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:54:06.463 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:54:06.463 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:54:06.463 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:54:06.463 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:54:06.463 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:54:06.463 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:54:06.463 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:54:06.463 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:54:06.463 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:54:06.464 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:54:06.464 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:54:06.464 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:54:06.464 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:54:06.464 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:54:06.464 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:54:06.464 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:54:06.464 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:54:06.464 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:54:06.464 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:54:06.464 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:54:06.464 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:54:06.464 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:54:06.465 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:54:06.465 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:54:06.465 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:54:06.465 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:54:06.465 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:54:06.465 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:54:06.465 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:54:06.465 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:54:06.465 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:54:06.465 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:54:06.465 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:54:06.465 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:54:06.465 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:54:06.466 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:54:06.466 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:54:06.466 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:54:06.466 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:54:06.466 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:54:06.466 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:54:06.466 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:54:06.466 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:54:06.466 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:54:06.466 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:54:06.466 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:54:06.466 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:54:06.466 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:54:06.466 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:54:06.467 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:54:06.467 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:54:06.467 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:54:06.467 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:54:06.467 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:54:06.467 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:54:06.467 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:54:06.467 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:54:06.467 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:54:06.467 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:54:06.467 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:54:06.467 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:54:06.467 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:54:06.467 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:54:06.468 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:54:06.468 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:54:06.468 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:54:06.468 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:54:06.468 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:54:06.468 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:54:06.468 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:54:06.468 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:54:06.468 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:54:06.468 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:54:06.468 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:54:06.468 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:54:06.468 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:54:06.468 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:54:06.469 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:54:06.469 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:54:06.469 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:54:06.469 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:54:06.469 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:54:06.469 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:54:06.469 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:54:06.469 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:54:06.469 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:54:06.469 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:54:06.469 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:54:06.469 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:54:06.469 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:54:06.469 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:54:06.470 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:54:06.470 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:54:06.470 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:54:06.470 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:54:06.470 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 2 2024-09-16T03:54:06.470 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:54:06.470 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:54:06.470 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:54:06.470 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:54:06.470 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:54:06.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.432+0000 7f642afa3640 10 librados: watch_flush enter 2024-09-16T03:54:06.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.432+0000 7f642afa3640 10 librados: watch_flush exit 2024-09-16T03:54:06.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.432+0000 7f642afa3640 20 client.824418.objecter shutdown clearing up homeless session... 2024-09-16T03:54:06.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.432+0000 7f642afa3640 10 client.824418.objecter successfully canceled tick 2024-09-16T03:54:06.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.432+0000 7f642afa3640 1 -- 172.21.15.42:0/876320715 >> v2:172.21.15.154:6812/254845342 conn(0x7f64100377c0 msgr2=0x7f6410039c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:54:06.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.432+0000 7f642afa3640 1 --2- 172.21.15.42:0/876320715 >> v2:172.21.15.154:6812/254845342 conn(0x7f64100377c0 0x7f6410039c80 secure :-1 s=READY pgs=3610 cs=0 l=1 rev1=1 crypto rx=0x7f641c003b00 tx=0x7f641c007d20 comp rx=0 tx=0).stop 2024-09-16T03:54:06.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.432+0000 7f642afa3640 10 monclient: shutdown 2024-09-16T03:54:06.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.432+0000 7f642afa3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:54:06.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.432+0000 7f642afa3640 1 -- 172.21.15.42:0/876320715 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f64240a39b0 msgr2=0x7f64241cf960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:54:06.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.432+0000 7f642afa3640 1 --2- 172.21.15.42:0/876320715 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f64240a39b0 0x7f64241cf960 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f6414000c10 tx=0x7f6414019140 comp rx=0 tx=0).stop 2024-09-16T03:54:06.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.432+0000 7f6429540640 1 -- 172.21.15.42:0/876320715 reap_dead start 2024-09-16T03:54:06.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.432+0000 7f6429540640 1 -- 172.21.15.42:0/876320715 reap_dead start 2024-09-16T03:54:06.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.432+0000 7f642afa3640 1 -- 172.21.15.42:0/876320715 shutdown_connections 2024-09-16T03:54:06.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.432+0000 7f642afa3640 1 -- 172.21.15.42:0/876320715 >> 172.21.15.42:0/876320715 conn(0x7f64240a9fb0 msgr2=0x7f6424131e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:54:06.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.432+0000 7f642afa3640 1 -- 172.21.15.42:0/876320715 shutdown_connections 2024-09-16T03:54:06.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.432+0000 7f642afa3640 1 -- 172.21.15.42:0/876320715 wait complete. 2024-09-16T03:54:06.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:06.432+0000 7f642afa3640 1 librados: shutdown 2024-09-16T03:54:06.473 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T03:54:06.474 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-16T03:54:26.475 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-16T03:54:26.475 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-09-16T03:54:26.476 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-16T03:54:26.476 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-16T03:54:26.479 INFO:tasks.ceph.mon.c:Started 2024-09-16T03:54:26.479 INFO:tasks.mon_thrash:Sending STOP to mon b 2024-09-16T03:54:26.479 INFO:tasks.ceph.mon.b:Sent signal 19 2024-09-16T03:54:26.479 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-16T03:54:41.480 INFO:tasks.mon_thrash:Sending CONT to mon b 2024-09-16T03:54:41.481 INFO:tasks.ceph.mon.b:Sent signal 18 2024-09-16T03:54:41.481 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T03:54:41.481 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:54:41.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.611+0000 7fbb406be640 10 monclient: get_monmap_and_config 2024-09-16T03:54:41.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.611+0000 7fbb406be640 10 monclient: build_initial_monmap 2024-09-16T03:54:41.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.611+0000 7fbb406be640 10 monclient: monmap: 2024-09-16T03:54:41.615 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:54:41.616 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:54:41.616 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:54:41.613906+0000 2024-09-16T03:54:41.616 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:54:41.613906+0000 2024-09-16T03:54:41.616 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:54:41.616 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:54:41.616 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:54:41.616 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:54:41.616 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:54:41.616 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:54:41.616 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:54:41.616 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:54:41.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.611+0000 7fbb406be640 1 Processor -- start 2024-09-16T03:54:41.619 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.611+0000 7fbb406be640 1 -- start start 2024-09-16T03:54:41.619 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.611+0000 7fbb406be640 10 monclient: init 2024-09-16T03:54:41.619 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.611+0000 7fbb406be640 10 monclient: _reopen_session rank -1 2024-09-16T03:54:41.619 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.611+0000 7fbb406be640 10 monclient: _add_conns ranks=[3,1,4,2,0] 2024-09-16T03:54:41.619 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.611+0000 7fbb406be640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fbb380a1330 0x7fbb3809f180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:54:41.619 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.611+0000 7fbb406be640 10 monclient(hunting): picked mon.noname-a con 0x7fbb380a1330 addr v2:172.21.15.42:3300/0 2024-09-16T03:54:41.619 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.611+0000 7fbb406be640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbb380a1710 addr v1:172.21.15.154:6789/0 2024-09-16T03:54:41.619 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.611+0000 7fbb406be640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fbb381393a0 0x7fbb3813b790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:54:41.619 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.611+0000 7fbb406be640 10 monclient(hunting): picked mon.noname-c con 0x7fbb381393a0 addr v2:172.21.15.154:3300/0 2024-09-16T03:54:41.619 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.611+0000 7fbb406be640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbb3809e4c0 con 0x7fbb380a1710 2024-09-16T03:54:41.619 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.611+0000 7fbb406be640 10 monclient(hunting): start opening mon connection 2024-09-16T03:54:41.620 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.611+0000 7fbb406be640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7fbb3809edf0 con 0x7fbb380a1330 2024-09-16T03:54:41.620 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.611+0000 7fbb406be640 10 monclient(hunting): start opening mon connection 2024-09-16T03:54:41.620 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.611+0000 7fbb406be640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fbb3809cd60 con 0x7fbb381393a0 2024-09-16T03:54:41.620 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.611+0000 7fbb406be640 10 monclient(hunting): _renew_subs 2024-09-16T03:54:41.620 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.611+0000 7fbb406be640 10 monclient(hunting): authenticate will time out at 1728.583357s 2024-09-16T03:54:41.620 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.611+0000 7fbb3e45a640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fbb380a1330 0x7fbb3809f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:54:41.620 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.611+0000 7fbb3e45a640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fbb380a1330 0x7fbb3809f180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:33618/0 (socket says 172.21.15.42:33618) 2024-09-16T03:54:41.620 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.611+0000 7fbb3e45a640 1 -- 172.21.15.42:0/2297392311 learned_addr learned my addr 172.21.15.42:0/2297392311 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:54:41.620 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.611+0000 7fbb3e45a640 10 monclient(hunting): get_auth_request con 0x7fbb380a1330 auth_method 0 2024-09-16T03:54:41.620 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.611+0000 7fbb3e45a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:54:41.620 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.611+0000 7fbb3e45a640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:54:41.620 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.611+0000 7fbb3e45a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:54:41.620 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.611+0000 7fbb3e45a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:54:41.621 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.611+0000 7fbb3e45a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:54:41.621 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.611+0000 7fbb3e45a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:54:41.621 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.611+0000 7fbb3ec5b640 1 --2- 172.21.15.42:0/2297392311 >> v2:172.21.15.154:3300/0 conn(0x7fbb381393a0 0x7fbb3813b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:54:41.621 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.611+0000 7fbb3ec5b640 10 monclient(hunting): get_auth_request con 0x7fbb381393a0 auth_method 0 2024-09-16T03:54:41.621 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.611+0000 7fbb3ec5b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:54:41.621 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.611+0000 7fbb3ec5b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:54:41.621 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.611+0000 7fbb3ec5b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:54:41.621 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.611+0000 7fbb3e45a640 10 monclient(hunting): handle_auth_done global_id 824421 payload 274 2024-09-16T03:54:41.621 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.611+0000 7fbb3e45a640 1 -- 172.21.15.42:0/2297392311 >> v2:172.21.15.154:3300/0 conn(0x7fbb381393a0 msgr2=0x7fbb3813b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:54:41.621 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.611+0000 7fbb3e45a640 1 --2- 172.21.15.42:0/2297392311 >> v2:172.21.15.154:3300/0 conn(0x7fbb381393a0 0x7fbb3813b790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:54:41.622 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.611+0000 7fbb3e45a640 1 -- 172.21.15.42:0/2297392311 >> v1:172.21.15.154:6789/0 conn(0x7fbb380a1710 legacy=0x7fbb380a1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:54:41.622 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.615+0000 7fbb3e45a640 10 monclient: _finish_hunting 0 2024-09-16T03:54:41.622 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.615+0000 7fbb3e45a640 1 monclient: found mon.noname-a 2024-09-16T03:54:41.622 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.615+0000 7fbb3e45a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:54:41.622 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.615+0000 7fbb3e45a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:54:41.622 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.615+0000 7fbb3e45a640 1 -- 172.21.15.42:0/2297392311 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb3809f6c0 con 0x7fbb380a1330 2024-09-16T03:54:41.622 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.615+0000 7fbb3ec5b640 1 --2- 172.21.15.42:0/2297392311 >> v2:172.21.15.154:3300/0 conn(0x7fbb381393a0 0x7fbb3813b790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T03:54:41.627 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.615+0000 7fbb3e45a640 1 --2- 172.21.15.42:0/2297392311 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbb380a1330 0x7fbb3809f180 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fbb3809df50 tx=0x7fbb2801c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b31c0ab31f97f90 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:54:41.627 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.615+0000 7fbb3d458640 1 -- 172.21.15.42:0/2297392311 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb280b0070 con 0x7fbb380a1330 2024-09-16T03:54:41.627 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.615+0000 7fbb3d458640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:54:41.628 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.615+0000 7fbb3d458640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:54:41.628 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.615+0000 7fbb3d458640 10 monclient: dump: 2024-09-16T03:54:41.628 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:54:41.628 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:54:41.628 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:41.628 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:41.628 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:54:41.628 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:54:41.628 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:54:41.628 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:54:41.628 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:54:41.628 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:54:41.628 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.615+0000 7fbb3d458640 10 monclient: _finish_auth 0 2024-09-16T03:54:41.629 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.615+0000 7fbb3d458640 10 monclient: _check_auth_tickets 2024-09-16T03:54:41.629 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.615+0000 7fbb3d458640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:54:41.629 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.615+0000 7fbb3d458640 1 -- 172.21.15.42:0/2297392311 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb280ab600 con 0x7fbb380a1330 2024-09-16T03:54:41.629 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.615+0000 7fbb406be640 5 monclient: authenticate success, global_id 824421 2024-09-16T03:54:41.629 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.615+0000 7fbb406be640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:54:41.629 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.615+0000 7fbb3d458640 10 monclient: handle_config config(0 keys) 2024-09-16T03:54:41.629 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.615+0000 7fbb3d458640 1 -- 172.21.15.42:0/2297392311 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb280ab9d0 con 0x7fbb380a1330 2024-09-16T03:54:41.629 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.615+0000 7fbb406be640 10 monclient: get_monmap_and_config success 2024-09-16T03:54:41.629 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.615+0000 7fbb406be640 4 set_mon_vals no callback set 2024-09-16T03:54:41.629 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.615+0000 7fbb3d458640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:54:41.629 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.615+0000 7fbb3d458640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:54:41.629 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.615+0000 7fbb3d458640 10 monclient: dump: 2024-09-16T03:54:41.629 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:54:41.629 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:54:41.630 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:41.630 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:41.630 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:54:41.630 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:54:41.630 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:54:41.630 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:54:41.630 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:54:41.630 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:54:41.630 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.615+0000 7fbb406be640 10 monclient: shutdown 2024-09-16T03:54:41.630 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.615+0000 7fbb406be640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:54:41.630 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.615+0000 7fbb406be640 1 -- 172.21.15.42:0/2297392311 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbb380a1330 msgr2=0x7fbb3809f180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:54:41.630 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.615+0000 7fbb406be640 1 --2- 172.21.15.42:0/2297392311 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbb380a1330 0x7fbb3809f180 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fbb3809df50 tx=0x7fbb2801c3e0 comp rx=0 tx=0).stop 2024-09-16T03:54:41.630 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.615+0000 7fbb406be640 1 -- 172.21.15.42:0/2297392311 shutdown_connections 2024-09-16T03:54:41.630 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.615+0000 7fbb406be640 1 --2- 172.21.15.42:0/2297392311 >> v2:172.21.15.154:3300/0 conn(0x7fbb381393a0 0x7fbb3813b790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:54:41.631 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.615+0000 7fbb406be640 1 --2- 172.21.15.42:0/2297392311 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbb380a1330 0x7fbb3809f180 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:54:41.631 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.615+0000 7fbb406be640 1 -- 172.21.15.42:0/2297392311 >> 172.21.15.42:0/2297392311 conn(0x7fbb380a2420 msgr2=0x7fbb380a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:54:41.631 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.615+0000 7fbb406be640 1 -- 172.21.15.42:0/2297392311 shutdown_connections 2024-09-16T03:54:41.631 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.615+0000 7fbb406be640 1 -- 172.21.15.42:0/2297392311 wait complete. 2024-09-16T03:54:41.631 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.619+0000 7fbb406be640 10 monclient: build_initial_monmap 2024-09-16T03:54:41.631 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.619+0000 7fbb406be640 10 monclient: monmap: 2024-09-16T03:54:41.631 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:54:41.631 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:54:41.631 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:54:41.631 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:54:41.631 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:54:41.631 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:54:41.632 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:54:41.632 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:54:41.632 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:54:41.632 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:54:41.632 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.619+0000 7fbb406be640 1 librados: starting msgr at 2024-09-16T03:54:41.632 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.619+0000 7fbb406be640 1 librados: starting objecter 2024-09-16T03:54:41.632 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.619+0000 7fbb406be640 1 Processor -- start 2024-09-16T03:54:41.632 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.619+0000 7fbb406be640 1 -- start start 2024-09-16T03:54:41.632 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.619+0000 7fbb406be640 1 librados: setting wanted keys 2024-09-16T03:54:41.632 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.619+0000 7fbb406be640 1 librados: calling monclient init 2024-09-16T03:54:41.632 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.619+0000 7fbb406be640 10 monclient: init 2024-09-16T03:54:41.632 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.619+0000 7fbb406be640 10 monclient: _reopen_session rank -1 2024-09-16T03:54:41.632 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.619+0000 7fbb406be640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:54:41.633 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.619+0000 7fbb406be640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbb380a1330 0x7fbb381cede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:54:41.633 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.619+0000 7fbb406be640 10 monclient(hunting): picked mon.noname-c con 0x7fbb380a1330 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:54:41.633 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.619+0000 7fbb406be640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbb380a1710 0x7fbb381342f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:54:41.633 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.619+0000 7fbb406be640 10 monclient(hunting): picked mon.noname-b con 0x7fbb380a1710 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:54:41.633 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.619+0000 7fbb406be640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbb381393a0 0x7fbb38134830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:54:41.633 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.619+0000 7fbb406be640 10 monclient(hunting): picked mon.noname-a con 0x7fbb381393a0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:54:41.633 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.619+0000 7fbb406be640 10 monclient(hunting): start opening mon connection 2024-09-16T03:54:41.633 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.619+0000 7fbb406be640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fbb3813e880 con 0x7fbb381393a0 2024-09-16T03:54:41.633 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.619+0000 7fbb406be640 10 monclient(hunting): start opening mon connection 2024-09-16T03:54:41.633 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.619+0000 7fbb406be640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fbb3813e700 con 0x7fbb380a1710 2024-09-16T03:54:41.633 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.619+0000 7fbb406be640 10 monclient(hunting): start opening mon connection 2024-09-16T03:54:41.633 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.619+0000 7fbb406be640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fbb3813ea00 con 0x7fbb380a1330 2024-09-16T03:54:41.633 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.619+0000 7fbb406be640 10 monclient(hunting): _renew_subs 2024-09-16T03:54:41.633 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.619+0000 7fbb406be640 10 monclient(hunting): authenticate will time out at 1728.589413s 2024-09-16T03:54:41.634 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.619+0000 7fbb3ec5b640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbb381393a0 0x7fbb38134830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:54:41.634 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.619+0000 7fbb3ec5b640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbb381393a0 0x7fbb38134830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:33630/0 (socket says 172.21.15.42:33630) 2024-09-16T03:54:41.634 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.619+0000 7fbb3ec5b640 1 -- 172.21.15.42:0/203433301 learned_addr learned my addr 172.21.15.42:0/203433301 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:54:41.634 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.619+0000 7fbb3e45a640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbb380a1330 0x7fbb381cede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:54:41.634 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.619+0000 7fbb3ec5b640 10 monclient(hunting): get_auth_request con 0x7fbb381393a0 auth_method 0 2024-09-16T03:54:41.634 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.619+0000 7fbb3ec5b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:54:41.634 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.619+0000 7fbb3ec5b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:54:41.634 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.619+0000 7fbb3ec5b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:54:41.634 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.619+0000 7fbb3e45a640 10 monclient(hunting): get_auth_request con 0x7fbb380a1330 auth_method 0 2024-09-16T03:54:41.634 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.619+0000 7fbb3e45a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:54:41.634 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.619+0000 7fbb3e45a640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:54:41.634 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.619+0000 7fbb3e45a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:54:41.634 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.619+0000 7fbb3ec5b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:54:41.635 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.619+0000 7fbb3ec5b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:54:41.635 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.619+0000 7fbb3dc59640 1 --2- 172.21.15.42:0/203433301 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbb380a1710 0x7fbb381342f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:54:41.635 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.619+0000 7fbb3ec5b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:54:41.635 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.619+0000 7fbb3dc59640 10 monclient(hunting): get_auth_request con 0x7fbb380a1710 auth_method 0 2024-09-16T03:54:41.635 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.619+0000 7fbb3dc59640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:54:41.635 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.619+0000 7fbb3dc59640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:54:41.635 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.619+0000 7fbb3dc59640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:54:41.635 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.619+0000 7fbb3e45a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:54:41.635 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.619+0000 7fbb3e45a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:54:41.635 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.619+0000 7fbb3e45a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:54:41.636 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.619+0000 7fbb3dc59640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:54:41.636 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.619+0000 7fbb3dc59640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:54:41.636 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.619+0000 7fbb3dc59640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:54:41.636 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.619+0000 7fbb3ec5b640 10 monclient(hunting): handle_auth_done global_id 824424 payload 824 2024-09-16T03:54:41.636 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.619+0000 7fbb3ec5b640 1 -- 172.21.15.42:0/203433301 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbb380a1330 msgr2=0x7fbb381cede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:54:41.636 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.619+0000 7fbb3ec5b640 1 --2- 172.21.15.42:0/203433301 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbb380a1330 0x7fbb381cede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:54:41.636 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.619+0000 7fbb3ec5b640 1 -- 172.21.15.42:0/203433301 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbb380a1710 msgr2=0x7fbb381342f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:54:41.636 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.619+0000 7fbb3ec5b640 1 --2- 172.21.15.42:0/203433301 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbb380a1710 0x7fbb381342f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:54:41.636 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.619+0000 7fbb3ec5b640 10 monclient: _finish_hunting 0 2024-09-16T03:54:41.636 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.619+0000 7fbb3ec5b640 1 monclient: found mon.noname-a 2024-09-16T03:54:41.636 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.619+0000 7fbb3ec5b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:54:41.636 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.619+0000 7fbb3ec5b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:54:41.637 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.619+0000 7fbb3ec5b640 1 -- 172.21.15.42:0/203433301 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb381219d0 con 0x7fbb381393a0 2024-09-16T03:54:41.637 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.619+0000 7fbb3ec5b640 1 --2- 172.21.15.42:0/203433301 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbb381393a0 0x7fbb38134830 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fbb30007ef0 tx=0x7fbb3000d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:54:41.637 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.619+0000 7fbb2effd640 10 client.?.objecter ms_handle_connect 0x7fbb381393a0 2024-09-16T03:54:41.637 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.619+0000 7fbb2effd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:54:41.637 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.619+0000 7fbb2effd640 1 -- 172.21.15.42:0/203433301 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb3001a070 con 0x7fbb381393a0 2024-09-16T03:54:41.637 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.619+0000 7fbb2effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:54:41.637 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.619+0000 7fbb2effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:54:41.637 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.619+0000 7fbb2effd640 10 monclient: dump: 2024-09-16T03:54:41.637 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:54:41.637 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:54:41.637 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:41.637 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:41.638 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:54:41.638 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:54:41.638 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:54:41.638 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:54:41.638 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:54:41.638 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:54:41.638 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.619+0000 7fbb2effd640 10 monclient: _finish_auth 0 2024-09-16T03:54:41.639 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.619+0000 7fbb2effd640 10 monclient: _check_auth_tickets 2024-09-16T03:54:41.639 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.619+0000 7fbb2effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:54:41.639 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.619+0000 7fbb2effd640 1 -- 172.21.15.42:0/203433301 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb30003ba0 con 0x7fbb381393a0 2024-09-16T03:54:41.639 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.619+0000 7fbb2effd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:54:41.639 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.619+0000 7fbb2effd640 1 -- 172.21.15.42:0/203433301 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb30004030 con 0x7fbb381393a0 2024-09-16T03:54:41.639 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.619+0000 7fbb2effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:54:41.639 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.619+0000 7fbb2effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:54:41.639 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.619+0000 7fbb2effd640 10 monclient: dump: 2024-09-16T03:54:41.639 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:54:41.639 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:54:41.639 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:41.639 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:41.639 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:54:41.639 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:54:41.640 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:54:41.640 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:54:41.640 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:54:41.640 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:54:41.640 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.619+0000 7fbb406be640 5 monclient: authenticate success, global_id 824424 2024-09-16T03:54:41.640 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.619+0000 7fbb406be640 10 monclient: _renew_subs 2024-09-16T03:54:41.640 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.619+0000 7fbb406be640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:54:41.640 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.619+0000 7fbb406be640 1 -- 172.21.15.42:0/203433301 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb381cf320 con 0x7fbb381393a0 2024-09-16T03:54:41.640 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.619+0000 7fbb3cc57640 4 set_mon_vals no callback set 2024-09-16T03:54:41.640 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.619+0000 7fbb406be640 10 client.824424.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:54:41.640 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.619+0000 7fbb406be640 10 monclient: _renew_subs 2024-09-16T03:54:41.640 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.619+0000 7fbb406be640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:54:41.640 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.619+0000 7fbb406be640 1 -- 172.21.15.42:0/203433301 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb38071ff0 con 0x7fbb381393a0 2024-09-16T03:54:41.641 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.619+0000 7fbb406be640 1 librados: init done 2024-09-16T03:54:41.641 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.619+0000 7fbb406be640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:54:41.641 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.619+0000 7fbb406be640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:54:41.641 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.619+0000 7fbb406be640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:54:41.641 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.619+0000 7fbb2effd640 1 -- 172.21.15.42:0/203433301 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fbb3001d070 con 0x7fbb381393a0 2024-09-16T03:54:41.641 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.619+0000 7fbb406be640 1 -- 172.21.15.42:0/203433301 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb3809f180 con 0x7fbb381393a0 2024-09-16T03:54:41.641 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.619+0000 7fbb2effd640 1 --2- 172.21.15.42:0/203433301 >> v2:172.21.15.154:6812/254845342 conn(0x7fbb1c037840 0x7fbb1c039d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:54:41.641 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.619+0000 7fbb2effd640 1 -- 172.21.15.42:0/203433301 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(386..386 src has 286..386) ==== 5713+0+0 (secure 0 0 0) 0x7fbb30049120 con 0x7fbb381393a0 2024-09-16T03:54:41.641 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.619+0000 7fbb2effd640 10 client.824424.objecter ms_dispatch 0x7fbb38000ba0 osd_map(386..386 src has 286..386) 2024-09-16T03:54:41.641 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.619+0000 7fbb2effd640 3 client.824424.objecter handle_osd_map got epochs [386,386] > 0 2024-09-16T03:54:41.641 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.619+0000 7fbb2effd640 3 client.824424.objecter handle_osd_map decoding full epoch 386 2024-09-16T03:54:41.641 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.619+0000 7fbb2effd640 20 client.824424.objecter dump_active .. 0 homeless 2024-09-16T03:54:41.641 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.619+0000 7fbb3e45a640 1 --2- 172.21.15.42:0/203433301 >> v2:172.21.15.154:6812/254845342 conn(0x7fbb1c037840 0x7fbb1c039d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:54:41.642 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.619+0000 7fbb3e45a640 10 monclient: get_auth_request con 0x7fbb1c037840 auth_method 0 2024-09-16T03:54:41.642 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.623+0000 7fbb2effd640 10 client.824424.objecter ms_handle_connect 0x7fbb1c037840 2024-09-16T03:54:41.642 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.623+0000 7fbb3e45a640 1 --2- 172.21.15.42:0/203433301 >> v2:172.21.15.154:6812/254845342 conn(0x7fbb1c037840 0x7fbb1c039d00 secure :-1 s=READY pgs=3613 cs=0 l=1 rev1=1 crypto rx=0x7fbb280ad000 tx=0x7fbb28008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:54:41.642 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.627+0000 7fbb2effd640 1 -- 172.21.15.42:0/203433301 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7fbb30010070 con 0x7fbb381393a0 2024-09-16T03:54:41.642 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.627+0000 7fbb2effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:54:41.642 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.627+0000 7fbb2effd640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:54:41.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.847+0000 7fbb406be640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:54:41.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.851+0000 7fbb406be640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:54:41.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.851+0000 7fbb406be640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:54:41.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.851+0000 7fbb406be640 1 -- 172.21.15.42:0/203433301 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fbb381202a0 con 0x7fbb381393a0 2024-09-16T03:54:41.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.851+0000 7fbb2effd640 1 -- 172.21.15.42:0/203433301 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7fbb300041d0 con 0x7fbb381393a0 2024-09-16T03:54:41.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.851+0000 7fbb2effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:54:41.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.851+0000 7fbb2effd640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:54:41.855 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":110,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:54:41.857 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.851+0000 7fbb406be640 10 librados: watch_flush enter 2024-09-16T03:54:41.857 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.851+0000 7fbb406be640 10 librados: watch_flush exit 2024-09-16T03:54:41.857 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.851+0000 7fbb406be640 20 client.824424.objecter shutdown clearing up homeless session... 2024-09-16T03:54:41.857 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.851+0000 7fbb406be640 10 client.824424.objecter successfully canceled tick 2024-09-16T03:54:41.857 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.851+0000 7fbb406be640 1 -- 172.21.15.42:0/203433301 >> v2:172.21.15.154:6812/254845342 conn(0x7fbb1c037840 msgr2=0x7fbb1c039d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:54:41.857 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.851+0000 7fbb406be640 1 --2- 172.21.15.42:0/203433301 >> v2:172.21.15.154:6812/254845342 conn(0x7fbb1c037840 0x7fbb1c039d00 secure :-1 s=READY pgs=3613 cs=0 l=1 rev1=1 crypto rx=0x7fbb280ad000 tx=0x7fbb28008040 comp rx=0 tx=0).stop 2024-09-16T03:54:41.857 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.851+0000 7fbb406be640 10 monclient: shutdown 2024-09-16T03:54:41.858 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.851+0000 7fbb406be640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:54:41.858 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.851+0000 7fbb406be640 1 -- 172.21.15.42:0/203433301 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbb381393a0 msgr2=0x7fbb38134830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:54:41.858 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.851+0000 7fbb406be640 1 --2- 172.21.15.42:0/203433301 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbb381393a0 0x7fbb38134830 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fbb30007ef0 tx=0x7fbb3000d4a0 comp rx=0 tx=0).stop 2024-09-16T03:54:41.858 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.851+0000 7fbb406be640 1 -- 172.21.15.42:0/203433301 shutdown_connections 2024-09-16T03:54:41.858 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.851+0000 7fbb406be640 1 --2- 172.21.15.42:0/203433301 >> v2:172.21.15.154:6812/254845342 conn(0x7fbb1c037840 0x7fbb1c039d00 unknown :-1 s=CLOSED pgs=3613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:54:41.858 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.851+0000 7fbb406be640 1 --2- 172.21.15.42:0/203433301 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbb381393a0 0x7fbb38134830 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:54:41.858 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.851+0000 7fbb406be640 1 --2- 172.21.15.42:0/203433301 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbb380a1710 0x7fbb381342f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:54:41.858 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.851+0000 7fbb406be640 1 --2- 172.21.15.42:0/203433301 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbb380a1330 0x7fbb381cede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:54:41.859 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.851+0000 7fbb406be640 1 -- 172.21.15.42:0/203433301 >> 172.21.15.42:0/203433301 conn(0x7fbb380a2420 msgr2=0x7fbb3809ba50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:54:41.859 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.851+0000 7fbb406be640 1 -- 172.21.15.42:0/203433301 shutdown_connections 2024-09-16T03:54:41.859 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.851+0000 7fbb406be640 1 -- 172.21.15.42:0/203433301 wait complete. 2024-09-16T03:54:41.859 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.851+0000 7fbb406be640 1 librados: shutdown 2024-09-16T03:54:41.872 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T03:54:41.872 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T03:54:41.873 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T03:54:41.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.991+0000 7f3950817640 10 monclient: get_monmap_and_config 2024-09-16T03:54:41.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.991+0000 7f3950817640 10 monclient: build_initial_monmap 2024-09-16T03:54:41.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.991+0000 7f3950817640 10 monclient: monmap: 2024-09-16T03:54:41.994 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:54:41.994 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:54:41.994 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:54:41.992295+0000 2024-09-16T03:54:41.994 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:54:41.992295+0000 2024-09-16T03:54:41.994 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:54:41.994 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:54:41.994 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:54:41.994 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:54:41.994 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:54:41.994 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:54:41.994 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:54:41.994 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:54:41.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.991+0000 7f3950817640 1 Processor -- start 2024-09-16T03:54:41.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.991+0000 7f3950817640 1 -- start start 2024-09-16T03:54:41.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.991+0000 7f3950817640 10 monclient: init 2024-09-16T03:54:41.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.991+0000 7f3950817640 10 monclient: _reopen_session rank -1 2024-09-16T03:54:41.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.991+0000 7f3950817640 10 monclient: _add_conns ranks=[4,1,2,3,0] 2024-09-16T03:54:41.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.991+0000 7f3950817640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f394c130b20 0x7f394c130f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:54:41.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.991+0000 7f3950817640 10 monclient(hunting): picked mon.noname-c con 0x7f394c130b20 addr v2:172.21.15.154:3300/0 2024-09-16T03:54:41.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.991+0000 7f3950817640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f394c0a2250 addr v1:172.21.15.154:6789/0 2024-09-16T03:54:41.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.991+0000 7f3950817640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f394c073420 0x7f394c13f240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:54:41.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.991+0000 7f3950817640 10 monclient(hunting): picked mon.noname-b con 0x7f394c073420 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:54:41.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.991+0000 7f3950817640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f394c0a1040 con 0x7f394c0a2250 2024-09-16T03:54:41.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.991+0000 7f3950817640 10 monclient(hunting): start opening mon connection 2024-09-16T03:54:41.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.991+0000 7f3950817640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f394c0a12b0 con 0x7f394c130b20 2024-09-16T03:54:41.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.991+0000 7f3950817640 10 monclient(hunting): start opening mon connection 2024-09-16T03:54:41.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.991+0000 7f3950817640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f394c0a1590 con 0x7f394c073420 2024-09-16T03:54:41.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.991+0000 7f3950817640 10 monclient(hunting): _renew_subs 2024-09-16T03:54:41.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.991+0000 7f3950817640 10 monclient(hunting): authenticate will time out at 1728.961715s 2024-09-16T03:54:41.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.991+0000 7f394ad9d640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f394c073420 0x7f394c13f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:54:41.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.991+0000 7f394ad9d640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f394c073420 0x7f394c13f240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:58252/0 (socket says 172.21.15.42:58252) 2024-09-16T03:54:41.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.991+0000 7f394ad9d640 1 -- 172.21.15.42:0/29158328 learned_addr learned my addr 172.21.15.42:0/29158328 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:54:41.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.991+0000 7f394ad9d640 10 monclient(hunting): get_auth_request con 0x7f394c073420 auth_method 0 2024-09-16T03:54:41.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.991+0000 7f394ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:54:41.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.991+0000 7f394ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:54:41.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.991+0000 7f394ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:54:42.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.991+0000 7f394ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:54:42.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.991+0000 7f394ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:54:42.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.991+0000 7f394a59c640 1 --2- 172.21.15.42:0/29158328 >> v2:172.21.15.154:3300/0 conn(0x7f394c130b20 0x7f394c130f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:54:42.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.991+0000 7f394ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:54:42.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.991+0000 7f394a59c640 10 monclient(hunting): get_auth_request con 0x7f394c130b20 auth_method 0 2024-09-16T03:54:42.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.991+0000 7f394a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:54:42.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.991+0000 7f394a59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:54:42.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.991+0000 7f394a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:54:42.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.991+0000 7f394ad9d640 10 monclient(hunting): handle_auth_done global_id 854102 payload 274 2024-09-16T03:54:42.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.991+0000 7f394ad9d640 1 -- 172.21.15.42:0/29158328 >> v2:172.21.15.154:3300/0 conn(0x7f394c130b20 msgr2=0x7f394c130f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:54:42.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.991+0000 7f394ad9d640 1 --2- 172.21.15.42:0/29158328 >> v2:172.21.15.154:3300/0 conn(0x7f394c130b20 0x7f394c130f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:54:42.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.991+0000 7f394ad9d640 1 -- 172.21.15.42:0/29158328 >> v1:172.21.15.154:6789/0 conn(0x7f394c0a2250 legacy=0x7f394c072d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:54:42.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.991+0000 7f394ad9d640 10 monclient: _finish_hunting 0 2024-09-16T03:54:42.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.991+0000 7f394ad9d640 1 monclient: found mon.noname-b 2024-09-16T03:54:42.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.991+0000 7f394ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:54:42.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.991+0000 7f394ad9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T03:54:42.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.991+0000 7f394ad9d640 1 -- 172.21.15.42:0/29158328 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f394c0a1870 con 0x7f394c073420 2024-09-16T03:54:42.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.991+0000 7f394a59c640 1 --2- 172.21.15.42:0/29158328 >> v2:172.21.15.154:3300/0 conn(0x7f394c130b20 0x7f394c130f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T03:54:42.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.995+0000 7f394ad9d640 1 --2- 172.21.15.42:0/29158328 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f394c073420 0x7f394c13f240 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f3944009e50 tx=0x7f394401c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d21ad86df88641cd server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:54:42.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.995+0000 7f394959a640 1 -- 172.21.15.42:0/29158328 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39440a57d0 con 0x7f394c073420 2024-09-16T03:54:42.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.995+0000 7f394959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:54:42.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.995+0000 7f394959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:54:42.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.995+0000 7f394959a640 10 monclient: dump: 2024-09-16T03:54:42.011 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:54:42.011 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:54:42.011 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:42.011 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:42.011 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:54:42.011 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:54:42.011 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:54:42.011 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:54:42.011 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:54:42.011 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:54:42.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.995+0000 7f394959a640 10 monclient: _finish_auth 0 2024-09-16T03:54:42.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.995+0000 7f394959a640 10 monclient: _check_auth_tickets 2024-09-16T03:54:42.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.995+0000 7f394959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:54:42.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.995+0000 7f394959a640 1 -- 172.21.15.42:0/29158328 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f39440a5970 con 0x7f394c073420 2024-09-16T03:54:42.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.995+0000 7f394959a640 10 monclient: handle_config config(0 keys) 2024-09-16T03:54:42.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.995+0000 7f394959a640 1 -- 172.21.15.42:0/29158328 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39440a5d40 con 0x7f394c073420 2024-09-16T03:54:42.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.995+0000 7f3950817640 5 monclient: authenticate success, global_id 854102 2024-09-16T03:54:42.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.995+0000 7f3950817640 10 monclient: get_monmap_and_config success 2024-09-16T03:54:42.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.995+0000 7f3950817640 4 set_mon_vals no callback set 2024-09-16T03:54:42.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.995+0000 7f394959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:54:42.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.995+0000 7f394959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:54:42.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.995+0000 7f394959a640 10 monclient: dump: 2024-09-16T03:54:42.012 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:54:42.012 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:54:42.012 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:42.012 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:42.013 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:54:42.013 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:54:42.013 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:54:42.013 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:54:42.013 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:54:42.013 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:54:42.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.995+0000 7f3950817640 10 monclient: shutdown 2024-09-16T03:54:42.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.995+0000 7f3950817640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:54:42.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.995+0000 7f3950817640 1 -- 172.21.15.42:0/29158328 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f394c073420 msgr2=0x7f394c13f240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:54:42.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.995+0000 7f3950817640 1 --2- 172.21.15.42:0/29158328 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f394c073420 0x7f394c13f240 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f3944009e50 tx=0x7f394401c8f0 comp rx=0 tx=0).stop 2024-09-16T03:54:42.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.995+0000 7f3950817640 1 -- 172.21.15.42:0/29158328 shutdown_connections 2024-09-16T03:54:42.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.995+0000 7f3950817640 1 --2- 172.21.15.42:0/29158328 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f394c073420 0x7f394c13f240 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:54:42.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.995+0000 7f3950817640 1 --2- 172.21.15.42:0/29158328 >> v2:172.21.15.154:3300/0 conn(0x7f394c130b20 0x7f394c130f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:54:42.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.995+0000 7f3950817640 1 -- 172.21.15.42:0/29158328 >> 172.21.15.42:0/29158328 conn(0x7f394c0a6210 msgr2=0x7f394c0a6620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:54:42.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.995+0000 7f3950817640 1 -- 172.21.15.42:0/29158328 shutdown_connections 2024-09-16T03:54:42.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.995+0000 7f3950817640 1 -- 172.21.15.42:0/29158328 wait complete. 2024-09-16T03:54:42.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.995+0000 7f3950817640 10 monclient: build_initial_monmap 2024-09-16T03:54:42.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.995+0000 7f3950817640 10 monclient: monmap: 2024-09-16T03:54:42.014 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:54:42.014 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:54:42.014 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:54:42.014 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:54:42.014 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:54:42.014 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:54:42.014 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:54:42.014 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:54:42.015 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:54:42.015 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:54:42.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.995+0000 7f3950817640 1 librados: starting msgr at 2024-09-16T03:54:42.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.995+0000 7f3950817640 1 librados: starting objecter 2024-09-16T03:54:42.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.995+0000 7f3950817640 1 Processor -- start 2024-09-16T03:54:42.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.995+0000 7f3950817640 1 -- start start 2024-09-16T03:54:42.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.995+0000 7f3950817640 1 librados: setting wanted keys 2024-09-16T03:54:42.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.995+0000 7f3950817640 1 librados: calling monclient init 2024-09-16T03:54:42.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.995+0000 7f3950817640 10 monclient: init 2024-09-16T03:54:42.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.995+0000 7f3950817640 10 monclient: _reopen_session rank -1 2024-09-16T03:54:42.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.995+0000 7f3950817640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:54:42.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.995+0000 7f3950817640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f394c073420 0x7f394c1cf1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:54:42.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.995+0000 7f3950817640 10 monclient(hunting): picked mon.noname-c con 0x7f394c073420 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:54:42.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.995+0000 7f3950817640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f394c0a2250 0x7f394c13e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:54:42.016 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.995+0000 7f3950817640 10 monclient(hunting): picked mon.noname-b con 0x7f394c0a2250 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:54:42.016 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.995+0000 7f3950817640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f394c130b20 0x7f394c13e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:54:42.016 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.995+0000 7f3950817640 10 monclient(hunting): picked mon.noname-a con 0x7f394c130b20 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:54:42.016 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.995+0000 7f3950817640 10 monclient(hunting): start opening mon connection 2024-09-16T03:54:42.016 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.995+0000 7f3950817640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f394c1315c0 con 0x7f394c130b20 2024-09-16T03:54:42.016 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.995+0000 7f3950817640 10 monclient(hunting): start opening mon connection 2024-09-16T03:54:42.016 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.995+0000 7f3950817640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f394c131440 con 0x7f394c0a2250 2024-09-16T03:54:42.016 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.995+0000 7f3950817640 10 monclient(hunting): start opening mon connection 2024-09-16T03:54:42.016 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.995+0000 7f3950817640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f394c141fa0 con 0x7f394c073420 2024-09-16T03:54:42.016 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.995+0000 7f3950817640 10 monclient(hunting): _renew_subs 2024-09-16T03:54:42.016 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.995+0000 7f3950817640 10 monclient(hunting): authenticate will time out at 1728.967377s 2024-09-16T03:54:42.016 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.995+0000 7f394a59c640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f394c073420 0x7f394c1cf1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:54:42.016 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.995+0000 7f394ad9d640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f394c130b20 0x7f394c13e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:54:42.016 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.995+0000 7f394a59c640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f394c073420 0x7f394c1cf1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:58256/0 (socket says 172.21.15.42:58256) 2024-09-16T03:54:42.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.995+0000 7f394a59c640 1 -- 172.21.15.42:0/2196146829 learned_addr learned my addr 172.21.15.42:0/2196146829 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:54:42.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.995+0000 7f394a59c640 10 monclient(hunting): get_auth_request con 0x7f394c073420 auth_method 0 2024-09-16T03:54:42.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.995+0000 7f394a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:54:42.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.995+0000 7f394a59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:54:42.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.995+0000 7f394a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:54:42.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.995+0000 7f394ad9d640 10 monclient(hunting): get_auth_request con 0x7f394c130b20 auth_method 0 2024-09-16T03:54:42.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.995+0000 7f394ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:54:42.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.995+0000 7f394ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:54:42.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.995+0000 7f394ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:54:42.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.995+0000 7f394a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:54:42.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.995+0000 7f394a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:54:42.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.995+0000 7f394a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:54:42.018 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.995+0000 7f394ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:54:42.018 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.995+0000 7f394ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:54:42.018 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.995+0000 7f3949d9b640 1 --2- 172.21.15.42:0/2196146829 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f394c0a2250 0x7f394c13e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:54:42.018 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.995+0000 7f394ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:54:42.018 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.995+0000 7f3949d9b640 10 monclient(hunting): get_auth_request con 0x7f394c0a2250 auth_method 0 2024-09-16T03:54:42.018 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.995+0000 7f3949d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:54:42.018 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.995+0000 7f3949d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:54:42.018 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.995+0000 7f3949d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:54:42.018 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.995+0000 7f394a59c640 10 monclient(hunting): handle_auth_done global_id 854105 payload 824 2024-09-16T03:54:42.018 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.995+0000 7f394a59c640 1 -- 172.21.15.42:0/2196146829 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f394c0a2250 msgr2=0x7f394c13e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:54:42.018 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.995+0000 7f394a59c640 1 --2- 172.21.15.42:0/2196146829 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f394c0a2250 0x7f394c13e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:54:42.018 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.995+0000 7f394a59c640 1 -- 172.21.15.42:0/2196146829 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f394c130b20 msgr2=0x7f394c13e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:54:42.019 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.995+0000 7f394a59c640 1 --2- 172.21.15.42:0/2196146829 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f394c130b20 0x7f394c13e970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:54:42.019 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.995+0000 7f394a59c640 10 monclient: _finish_hunting 0 2024-09-16T03:54:42.019 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.995+0000 7f394a59c640 1 monclient: found mon.noname-c 2024-09-16T03:54:42.019 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.995+0000 7f394a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:54:42.019 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.995+0000 7f394a59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:54:42.019 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.995+0000 7f394a59c640 1 -- 172.21.15.42:0/2196146829 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f394c139bf0 con 0x7f394c073420 2024-09-16T03:54:42.019 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.995+0000 7f3949d9b640 1 --2- 172.21.15.42:0/2196146829 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f394c0a2250 0x7f394c13e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T03:54:42.019 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.999+0000 7f394a59c640 1 --2- 172.21.15.42:0/2196146829 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f394c073420 0x7f394c1cf1b0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f393c00e810 tx=0x7f393c00ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:54:42.019 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.999+0000 7f393affd640 10 client.?.objecter ms_handle_connect 0x7f394c073420 2024-09-16T03:54:42.019 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.999+0000 7f393affd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:54:42.019 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.999+0000 7f393affd640 1 -- 172.21.15.42:0/2196146829 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f393c017070 con 0x7f394c073420 2024-09-16T03:54:42.019 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.999+0000 7f393affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:54:42.020 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.999+0000 7f393affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:54:42.020 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.999+0000 7f393affd640 10 monclient: dump: 2024-09-16T03:54:42.020 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:54:42.020 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:54:42.020 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:42.020 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:42.020 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:54:42.020 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:54:42.020 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:54:42.020 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:54:42.020 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:54:42.020 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:54:42.020 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.999+0000 7f393affd640 10 monclient: _finish_auth 0 2024-09-16T03:54:42.020 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.999+0000 7f393affd640 10 monclient: _check_auth_tickets 2024-09-16T03:54:42.021 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.999+0000 7f393affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:54:42.021 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.999+0000 7f393affd640 1 -- 172.21.15.42:0/2196146829 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f393c00c8d0 con 0x7f394c073420 2024-09-16T03:54:42.021 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.999+0000 7f3950817640 5 monclient: authenticate success, global_id 854105 2024-09-16T03:54:42.021 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.999+0000 7f3950817640 10 monclient: _renew_subs 2024-09-16T03:54:42.021 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.999+0000 7f3950817640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:54:42.021 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.999+0000 7f3950817640 1 -- 172.21.15.42:0/2196146829 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f394c1cf6f0 con 0x7f394c073420 2024-09-16T03:54:42.021 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.999+0000 7f393affd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:54:42.021 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.999+0000 7f393affd640 1 -- 172.21.15.42:0/2196146829 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f393c00f040 con 0x7f394c073420 2024-09-16T03:54:42.021 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.999+0000 7f393affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:54:42.021 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.999+0000 7f393affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:54:42.021 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.999+0000 7f393affd640 10 monclient: dump: 2024-09-16T03:54:42.021 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:54:42.021 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:54:42.021 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:42.022 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:42.022 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:54:42.022 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:54:42.022 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:54:42.022 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:54:42.022 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:54:42.022 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:54:42.022 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.999+0000 7f3950817640 10 client.854105.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:54:42.022 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.999+0000 7f3948d99640 4 set_mon_vals no callback set 2024-09-16T03:54:42.022 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.999+0000 7f3950817640 10 monclient: _renew_subs 2024-09-16T03:54:42.022 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.999+0000 7f3950817640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:54:42.022 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.999+0000 7f3950817640 1 -- 172.21.15.42:0/2196146829 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f394c13f510 con 0x7f394c073420 2024-09-16T03:54:42.022 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.999+0000 7f3950817640 1 librados: init done 2024-09-16T03:54:42.023 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.999+0000 7f3950817640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:54:42.023 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.999+0000 7f3950817640 1 --2- 172.21.15.42:0/2196146829 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f394c09edf0 0x7f394c02c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:54:42.023 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.999+0000 7f3950817640 10 monclient(hunting): start opening mon connection 2024-09-16T03:54:42.023 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.999+0000 7f3950817640 1 -- 172.21.15.42:0/2196146829 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f394c139a30 con 0x7f394c09edf0 2024-09-16T03:54:42.023 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.999+0000 7f3949d9b640 1 --2- 172.21.15.42:0/2196146829 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f394c09edf0 0x7f394c02c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:54:42.023 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.999+0000 7f3949d9b640 10 monclient: get_auth_request con 0x7f394c09edf0 auth_method 0 2024-09-16T03:54:42.023 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.999+0000 7f3949d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:54:42.023 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.999+0000 7f3949d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:54:42.023 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.999+0000 7f3949d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:54:42.023 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.999+0000 7f3949d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:54:42.023 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.999+0000 7f3949d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:54:42.023 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.999+0000 7f3949d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:54:42.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.999+0000 7f393affd640 1 -- 172.21.15.42:0/2196146829 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f393c01a070 con 0x7f394c073420 2024-09-16T03:54:42.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.999+0000 7f3949d9b640 10 monclient(hunting): handle_auth_done global_id 824430 payload 824 2024-09-16T03:54:42.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.999+0000 7f393affd640 1 --2- 172.21.15.42:0/2196146829 >> v2:172.21.15.154:6812/254845342 conn(0x7f392c037810 0x7f392c039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:54:42.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.999+0000 7f3949d9b640 1 -- 172.21.15.42:0/2196146829 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f394c130f00 con 0x7f394c09edf0 2024-09-16T03:54:42.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.999+0000 7f393affd640 1 -- 172.21.15.42:0/2196146829 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(386..386 src has 286..386) ==== 5713+0+0 (secure 0 0 0) 0x7f393c048cc0 con 0x7f394c073420 2024-09-16T03:54:42.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.999+0000 7f393affd640 10 client.854105.objecter ms_dispatch 0x7f394c000ba0 osd_map(386..386 src has 286..386) 2024-09-16T03:54:42.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.999+0000 7f393affd640 3 client.854105.objecter handle_osd_map got epochs [386,386] > 0 2024-09-16T03:54:42.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.999+0000 7f393affd640 3 client.854105.objecter handle_osd_map decoding full epoch 386 2024-09-16T03:54:42.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.999+0000 7f393affd640 20 client.854105.objecter dump_active .. 0 homeless 2024-09-16T03:54:42.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.999+0000 7f393affd640 10 client.854105.objecter ms_handle_connect 0x7f394c09edf0 2024-09-16T03:54:42.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.999+0000 7f3949d9b640 1 --2- 172.21.15.42:0/2196146829 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f394c09edf0 0x7f394c02c650 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f394c139e00 tx=0x7f394000e940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:54:42.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.999+0000 7f393affd640 10 client.854105.objecter resend_mon_ops 2024-09-16T03:54:42.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.999+0000 7f393affd640 1 -- 172.21.15.42:0/2196146829 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39400052e0 con 0x7f394c09edf0 2024-09-16T03:54:42.025 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.999+0000 7f393affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:54:42.025 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.999+0000 7f393affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:54:42.025 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.999+0000 7f393affd640 10 monclient: dump: 2024-09-16T03:54:42.025 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:54:42.025 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:54:42.025 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:42.025 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:42.025 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:54:42.025 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:54:42.025 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:54:42.025 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:54:42.025 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:54:42.025 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:54:42.025 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.999+0000 7f394ad9d640 1 --2- 172.21.15.42:0/2196146829 >> v2:172.21.15.154:6812/254845342 conn(0x7f392c037810 0x7f392c039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:54:42.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.999+0000 7f394ad9d640 10 monclient: get_auth_request con 0x7f392c037810 auth_method 0 2024-09-16T03:54:42.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.999+0000 7f393affd640 1 -- 172.21.15.42:0/2196146829 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f39400052e0 con 0x7f394c09edf0 2024-09-16T03:54:42.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.999+0000 7f393affd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:54:42.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.999+0000 7f393affd640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:54:42.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.999+0000 7f393affd640 1 -- 172.21.15.42:0/2196146829 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f394c09edf0 msgr2=0x7f394c02c650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:54:42.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.999+0000 7f393affd640 1 --2- 172.21.15.42:0/2196146829 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f394c09edf0 0x7f394c02c650 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f394c139e00 tx=0x7f394000e940 comp rx=0 tx=0).stop 2024-09-16T03:54:42.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.999+0000 7f393affd640 1 -- 172.21.15.42:0/2196146829 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f394c09edf0 msgr2=0x7f394c02c650 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:54:42.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.999+0000 7f393affd640 1 --2- 172.21.15.42:0/2196146829 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f394c09edf0 0x7f394c02c650 secure :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f394c139e00 tx=0x7f394000e940 comp rx=0 tx=0).stop 2024-09-16T03:54:42.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.999+0000 7f393affd640 10 client.854105.objecter ms_handle_connect 0x7f392c037810 2024-09-16T03:54:42.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:41.999+0000 7f394ad9d640 1 --2- 172.21.15.42:0/2196146829 >> v2:172.21.15.154:6812/254845342 conn(0x7f392c037810 0x7f392c039cd0 secure :-1 s=READY pgs=3614 cs=0 l=1 rev1=1 crypto rx=0x7f39440ae000 tx=0x7f39440a7070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:54:42.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.003+0000 7f3950817640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:54:42.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.007+0000 7f3950817640 1 --2- 172.21.15.42:0/2196146829 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f394c09f1d0 0x7f394c02cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:54:42.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.007+0000 7f3950817640 10 monclient(hunting): start opening mon connection 2024-09-16T03:54:42.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.007+0000 7f3950817640 1 -- 172.21.15.42:0/2196146829 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f394c132ff0 con 0x7f394c09f1d0 2024-09-16T03:54:42.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.007+0000 7f3949d9b640 1 --2- 172.21.15.42:0/2196146829 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f394c09f1d0 0x7f394c02cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:54:42.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.007+0000 7f3949d9b640 10 monclient: get_auth_request con 0x7f394c09f1d0 auth_method 0 2024-09-16T03:54:42.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.007+0000 7f3949d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:54:42.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.007+0000 7f3949d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:54:42.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.007+0000 7f3949d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:54:42.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.007+0000 7f3949d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:54:42.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.007+0000 7f3949d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:54:42.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.007+0000 7f3949d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:54:42.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.007+0000 7f3949d9b640 10 monclient(hunting): handle_auth_done global_id 824433 payload 824 2024-09-16T03:54:42.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.007+0000 7f3949d9b640 1 -- 172.21.15.42:0/2196146829 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f394c1384c0 con 0x7f394c09f1d0 2024-09-16T03:54:42.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.007+0000 7f393affd640 10 client.854105.objecter ms_handle_connect 0x7f394c09f1d0 2024-09-16T03:54:42.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.007+0000 7f3949d9b640 1 --2- 172.21.15.42:0/2196146829 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f394c09f1d0 0x7f394c02cb90 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f3940014000 tx=0x7f3940000ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:54:42.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.007+0000 7f393affd640 10 client.854105.objecter resend_mon_ops 2024-09-16T03:54:42.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.007+0000 7f393affd640 1 -- 172.21.15.42:0/2196146829 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39400069f0 con 0x7f394c09f1d0 2024-09-16T03:54:42.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.007+0000 7f393affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:54:42.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.007+0000 7f393affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:54:42.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.007+0000 7f393affd640 10 monclient: dump: 2024-09-16T03:54:42.028 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:54:42.028 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:54:42.028 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:42.028 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:42.029 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:54:42.029 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:54:42.029 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:54:42.029 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:54:42.029 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:54:42.029 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:54:42.029 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.007+0000 7f393affd640 1 -- 172.21.15.42:0/2196146829 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4882 (secure 0 0 0) 0x7f39400069f0 con 0x7f394c09f1d0 2024-09-16T03:54:42.029 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.007+0000 7f393affd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:54:42.029 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.007+0000 7f393affd640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:54:42.029 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.007+0000 7f393affd640 1 -- 172.21.15.42:0/2196146829 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f394c09f1d0 msgr2=0x7f394c02cb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:54:42.029 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.007+0000 7f393affd640 1 --2- 172.21.15.42:0/2196146829 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f394c09f1d0 0x7f394c02cb90 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f3940014000 tx=0x7f3940000ee0 comp rx=0 tx=0).stop 2024-09-16T03:54:42.029 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.007+0000 7f393affd640 1 -- 172.21.15.42:0/2196146829 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f394c09f1d0 msgr2=0x7f394c02cb90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:54:42.029 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.007+0000 7f393affd640 1 --2- 172.21.15.42:0/2196146829 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f394c09f1d0 0x7f394c02cb90 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:54:42.030 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:54:42.030 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:54:42.030 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:54:42.030 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "leader", 2024-09-16T03:54:42.030 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 110, 2024-09-16T03:54:42.030 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 171862, 2024-09-16T03:54:42.030 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:54:42.030 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:54:42.031 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T03:54:42.031 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T03:54:42.031 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:54:42.031 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 0, 2024-09-16T03:54:42.031 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:54:42.031 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:54:42.031 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:54:42.031 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:54:42.031 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:54:42.031 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:54:42.031 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:54:42.031 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:54:42.031 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:54:42.031 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:54:42.032 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:54:42.032 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:54:42.032 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:54:42.032 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:54:42.032 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:54:42.032 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:54:42.032 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:54:42.032 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:54:42.033 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:54:42.033 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:54:42.033 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:54:42.033 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:54:42.033 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:54:42.033 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:54:42.033 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:54:42.033 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:54:42.033 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:54:42.033 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:54:42.033 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:54:42.034 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:54:42.034 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:54:42.034 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:54:42.034 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:54:42.034 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:54:42.034 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:54:42.034 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:54:42.034 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:54:42.034 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:54:42.034 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:54:42.035 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:54:42.035 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:54:42.035 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:54:42.035 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:54:42.035 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:54:42.035 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:54:42.035 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:54:42.035 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:54:42.035 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:54:42.035 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:54:42.035 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:54:42.035 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:54:42.035 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:54:42.036 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:54:42.036 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:54:42.036 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:54:42.036 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:54:42.036 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:54:42.036 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:54:42.036 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:54:42.036 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:54:42.036 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:54:42.036 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:54:42.036 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:54:42.036 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:54:42.037 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:54:42.037 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:54:42.037 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:54:42.037 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:54:42.037 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:54:42.037 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:54:42.037 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:54:42.037 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:54:42.037 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:54:42.037 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:54:42.037 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:54:42.037 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:54:42.037 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:54:42.038 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:54:42.038 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:54:42.038 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:54:42.038 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:54:42.038 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:54:42.038 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:54:42.038 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:54:42.038 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:54:42.038 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:54:42.038 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:54:42.038 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:54:42.038 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:54:42.038 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:54:42.039 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:54:42.039 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:54:42.039 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:54:42.039 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:54:42.039 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:54:42.039 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:54:42.039 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:54:42.039 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:54:42.039 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:54:42.039 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:54:42.039 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:54:42.039 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:54:42.039 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:54:42.039 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:54:42.040 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:54:42.040 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:54:42.040 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:54:42.040 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:54:42.040 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:54:42.040 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:54:42.040 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:54:42.040 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:54:42.040 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:54:42.040 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:54:42.040 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:54:42.040 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:54:42.040 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:54:42.040 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:54:42.041 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:54:42.041 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:54:42.041 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:54:42.041 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:54:42.041 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:54:42.041 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:54:42.041 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:54:42.041 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:54:42.041 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:54:42.041 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:54:42.041 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:54:42.041 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:54:42.041 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:54:42.041 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:54:42.042 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:54:42.042 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:54:42.042 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:54:42.042 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:54:42.042 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:54:42.042 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:54:42.042 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:54:42.042 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:54:42.042 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:54:42.042 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T03:54:42.042 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:54:42.042 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:54:42.042 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:54:42.042 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 6 2024-09-16T03:54:42.043 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:54:42.043 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:54:42.043 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:54:42.043 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:54:42.043 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:54:42.043 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:54:42.043 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 6 2024-09-16T03:54:42.043 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:54:42.043 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:54:42.043 INFO:teuthology.orchestra.run.smithi042.stdout: "mgr": [ 2024-09-16T03:54:42.043 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:54:42.043 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:54:42.043 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:54:42.043 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:54:42.044 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:54:42.044 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:54:42.044 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:54:42.044 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:54:42.044 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:54:42.045 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.007+0000 7f3950817640 10 librados: watch_flush enter 2024-09-16T03:54:42.045 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.007+0000 7f3950817640 10 librados: watch_flush exit 2024-09-16T03:54:42.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.007+0000 7f3950817640 20 client.854105.objecter shutdown clearing up homeless session... 2024-09-16T03:54:42.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.007+0000 7f3950817640 10 client.854105.objecter successfully canceled tick 2024-09-16T03:54:42.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.007+0000 7f3950817640 1 -- 172.21.15.42:0/2196146829 >> v2:172.21.15.154:6812/254845342 conn(0x7f392c037810 msgr2=0x7f392c039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:54:42.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.007+0000 7f3950817640 1 --2- 172.21.15.42:0/2196146829 >> v2:172.21.15.154:6812/254845342 conn(0x7f392c037810 0x7f392c039cd0 secure :-1 s=READY pgs=3614 cs=0 l=1 rev1=1 crypto rx=0x7f39440ae000 tx=0x7f39440a7070 comp rx=0 tx=0).stop 2024-09-16T03:54:42.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.007+0000 7f3950817640 10 monclient: shutdown 2024-09-16T03:54:42.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.007+0000 7f3950817640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:54:42.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.007+0000 7f3950817640 1 -- 172.21.15.42:0/2196146829 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f394c073420 msgr2=0x7f394c1cf1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:54:42.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.007+0000 7f3950817640 1 --2- 172.21.15.42:0/2196146829 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f394c073420 0x7f394c1cf1b0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f393c00e810 tx=0x7f393c00ecd0 comp rx=0 tx=0).stop 2024-09-16T03:54:42.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.007+0000 7f394ad9d640 1 -- 172.21.15.42:0/2196146829 reap_dead start 2024-09-16T03:54:42.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.007+0000 7f394ad9d640 1 -- 172.21.15.42:0/2196146829 reap_dead start 2024-09-16T03:54:42.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.007+0000 7f3950817640 1 -- 172.21.15.42:0/2196146829 shutdown_connections 2024-09-16T03:54:42.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.007+0000 7f3950817640 1 -- 172.21.15.42:0/2196146829 >> 172.21.15.42:0/2196146829 conn(0x7f394c0a6210 msgr2=0x7f394c12fa50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:54:42.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.007+0000 7f3950817640 1 -- 172.21.15.42:0/2196146829 shutdown_connections 2024-09-16T03:54:42.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.007+0000 7f3950817640 1 -- 172.21.15.42:0/2196146829 wait complete. 2024-09-16T03:54:42.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.007+0000 7f3950817640 1 librados: shutdown 2024-09-16T03:54:42.047 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:54:42.047 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T03:54:42.047 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T03:54:42.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.155+0000 7f5980bff640 10 monclient: get_monmap_and_config 2024-09-16T03:54:42.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.155+0000 7f5980bff640 10 monclient: build_initial_monmap 2024-09-16T03:54:42.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.155+0000 7f5980bff640 10 monclient: monmap: 2024-09-16T03:54:42.158 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:54:42.158 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:54:42.158 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:54:42.157185+0000 2024-09-16T03:54:42.159 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:54:42.157185+0000 2024-09-16T03:54:42.159 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:54:42.159 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:54:42.159 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:54:42.159 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:54:42.159 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:54:42.159 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:54:42.159 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:54:42.159 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:54:42.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.155+0000 7f5980bff640 1 Processor -- start 2024-09-16T03:54:42.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.155+0000 7f5980bff640 1 -- start start 2024-09-16T03:54:42.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.155+0000 7f5980bff640 10 monclient: init 2024-09-16T03:54:42.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.155+0000 7f5980bff640 10 monclient: _reopen_session rank -1 2024-09-16T03:54:42.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.155+0000 7f5980bff640 10 monclient: _add_conns ranks=[2,3,0,1,4] 2024-09-16T03:54:42.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.155+0000 7f5980bff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f597c0a1330 0x7f597c09f180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:54:42.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.155+0000 7f5980bff640 10 monclient(hunting): picked mon.noname-b con 0x7f597c0a1330 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:54:42.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.155+0000 7f5980bff640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f597c0a1710 0x7f597c0a1b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:54:42.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.155+0000 7f5980bff640 10 monclient(hunting): picked mon.noname-a con 0x7f597c0a1710 addr v2:172.21.15.42:3300/0 2024-09-16T03:54:42.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.155+0000 7f5980bff640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f597c1340f0 addr v1:172.21.15.42:6789/0 2024-09-16T03:54:42.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.155+0000 7f5980bff640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f597c09e4c0 con 0x7f597c1340f0 2024-09-16T03:54:42.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.155+0000 7f5980bff640 10 monclient(hunting): start opening mon connection 2024-09-16T03:54:42.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.155+0000 7f5980bff640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f597c09edf0 con 0x7f597c0a1710 2024-09-16T03:54:42.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.155+0000 7f5980bff640 10 monclient(hunting): start opening mon connection 2024-09-16T03:54:42.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.155+0000 7f5980bff640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f597c09cd60 con 0x7f597c0a1330 2024-09-16T03:54:42.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.155+0000 7f5980bff640 10 monclient(hunting): _renew_subs 2024-09-16T03:54:42.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.155+0000 7f5980bff640 10 monclient(hunting): authenticate will time out at 1729.126225s 2024-09-16T03:54:42.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.155+0000 7f597ad9d640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f597c1340f0 0x7f597c1345a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:57206/0 (socket says 172.21.15.42:57206) 2024-09-16T03:54:42.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.155+0000 7f597ad9d640 1 -- 172.21.15.42:0/2850996697 learned_addr learned my addr 172.21.15.42:0/2850996697 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:54:42.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.155+0000 7f5979d9b640 1 --2- 172.21.15.42:0/2850996697 >> v2:172.21.15.42:3300/0 conn(0x7f597c0a1710 0x7f597c0a1b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:54:42.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.155+0000 7f597a59c640 1 --2- 172.21.15.42:0/2850996697 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f597c0a1330 0x7f597c09f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:54:42.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.155+0000 7f5979d9b640 10 monclient(hunting): get_auth_request con 0x7f597c0a1710 auth_method 0 2024-09-16T03:54:42.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.155+0000 7f5979d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:54:42.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.155+0000 7f5979d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:54:42.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.155+0000 7f5979d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:54:42.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.155+0000 7f597a59c640 10 monclient(hunting): get_auth_request con 0x7f597c0a1330 auth_method 0 2024-09-16T03:54:42.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.155+0000 7f597a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:54:42.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.155+0000 7f597a59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:54:42.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.155+0000 7f597a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:54:42.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.155+0000 7f5979d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:54:42.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.155+0000 7f5979d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:54:42.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.155+0000 7f5979d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:54:42.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.155+0000 7f597a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:54:42.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.155+0000 7f597a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:54:42.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.155+0000 7f597a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:54:42.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.155+0000 7f5979d9b640 10 monclient(hunting): handle_auth_done global_id 824436 payload 274 2024-09-16T03:54:42.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.155+0000 7f5979d9b640 1 -- 172.21.15.42:0/2850996697 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f597c0a1330 msgr2=0x7f597c09f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:54:42.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.155+0000 7f5979d9b640 1 --2- 172.21.15.42:0/2850996697 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f597c0a1330 0x7f597c09f180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:54:42.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.155+0000 7f5979d9b640 1 -- 172.21.15.42:0/2850996697 >> v1:172.21.15.42:6789/0 conn(0x7f597c1340f0 legacy=0x7f597c1345a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:54:42.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.155+0000 7f5979d9b640 10 monclient: _finish_hunting 0 2024-09-16T03:54:42.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.155+0000 7f5979d9b640 1 monclient: found mon.noname-a 2024-09-16T03:54:42.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.155+0000 7f5979d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:54:42.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.155+0000 7f5979d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:54:42.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.155+0000 7f5979d9b640 1 -- 172.21.15.42:0/2850996697 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f597c0a2050 con 0x7f597c0a1710 2024-09-16T03:54:42.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.159+0000 7f5979d9b640 1 --2- 172.21.15.42:0/2850996697 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f597c0a1710 0x7f597c0a1b10 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f59700097f0 tx=0x7f597001c360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c67d7df997c072b6 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:54:42.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.159+0000 7f597959a640 1 -- 172.21.15.42:0/2850996697 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59700b0070 con 0x7f597c0a1710 2024-09-16T03:54:42.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.159+0000 7f597959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:54:42.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.159+0000 7f597959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:54:42.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.159+0000 7f597959a640 10 monclient: dump: 2024-09-16T03:54:42.171 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:54:42.171 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:54:42.171 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:42.171 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:42.171 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:54:42.171 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:54:42.171 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:54:42.171 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:54:42.171 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:54:42.172 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:54:42.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.159+0000 7f597959a640 10 monclient: _finish_auth 0 2024-09-16T03:54:42.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.159+0000 7f597959a640 10 monclient: _check_auth_tickets 2024-09-16T03:54:42.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.159+0000 7f597959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:54:42.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.159+0000 7f597959a640 1 -- 172.21.15.42:0/2850996697 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f59700ab600 con 0x7f597c0a1710 2024-09-16T03:54:42.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.159+0000 7f5980bff640 5 monclient: authenticate success, global_id 824436 2024-09-16T03:54:42.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.159+0000 7f5980bff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:54:42.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.159+0000 7f597959a640 10 monclient: handle_config config(0 keys) 2024-09-16T03:54:42.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.159+0000 7f5980bff640 10 monclient: get_monmap_and_config success 2024-09-16T03:54:42.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.159+0000 7f5980bff640 4 set_mon_vals no callback set 2024-09-16T03:54:42.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.159+0000 7f597959a640 1 -- 172.21.15.42:0/2850996697 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59700ab9d0 con 0x7f597c0a1710 2024-09-16T03:54:42.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.159+0000 7f597959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:54:42.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.159+0000 7f597959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:54:42.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.159+0000 7f597959a640 10 monclient: dump: 2024-09-16T03:54:42.173 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:54:42.173 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:54:42.173 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:42.173 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:42.173 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:54:42.173 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:54:42.173 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:54:42.173 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:54:42.173 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:54:42.173 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:54:42.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.159+0000 7f5980bff640 10 monclient: shutdown 2024-09-16T03:54:42.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.159+0000 7f5980bff640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:54:42.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.159+0000 7f5980bff640 1 -- 172.21.15.42:0/2850996697 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f597c0a1710 msgr2=0x7f597c0a1b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:54:42.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.159+0000 7f5980bff640 1 --2- 172.21.15.42:0/2850996697 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f597c0a1710 0x7f597c0a1b10 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f59700097f0 tx=0x7f597001c360 comp rx=0 tx=0).stop 2024-09-16T03:54:42.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.159+0000 7f5980bff640 1 -- 172.21.15.42:0/2850996697 shutdown_connections 2024-09-16T03:54:42.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.159+0000 7f5980bff640 1 --2- 172.21.15.42:0/2850996697 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f597c0a1710 0x7f597c0a1b10 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:54:42.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.159+0000 7f5980bff640 1 --2- 172.21.15.42:0/2850996697 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f597c0a1330 0x7f597c09f180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:54:42.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.159+0000 7f5980bff640 1 -- 172.21.15.42:0/2850996697 >> 172.21.15.42:0/2850996697 conn(0x7f597c0a2420 msgr2=0x7f597c0a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:54:42.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.159+0000 7f5980bff640 1 -- 172.21.15.42:0/2850996697 shutdown_connections 2024-09-16T03:54:42.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.159+0000 7f5980bff640 1 -- 172.21.15.42:0/2850996697 wait complete. 2024-09-16T03:54:42.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.159+0000 7f5980bff640 10 monclient: build_initial_monmap 2024-09-16T03:54:42.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.159+0000 7f5980bff640 10 monclient: monmap: 2024-09-16T03:54:42.174 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:54:42.174 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:54:42.174 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:54:42.174 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:54:42.175 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:54:42.175 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:54:42.175 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:54:42.175 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:54:42.175 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:54:42.175 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:54:42.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.159+0000 7f5980bff640 1 librados: starting msgr at 2024-09-16T03:54:42.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.159+0000 7f5980bff640 1 librados: starting objecter 2024-09-16T03:54:42.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.159+0000 7f5980bff640 1 Processor -- start 2024-09-16T03:54:42.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.159+0000 7f5980bff640 1 -- start start 2024-09-16T03:54:42.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.159+0000 7f5980bff640 1 librados: setting wanted keys 2024-09-16T03:54:42.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.159+0000 7f5980bff640 1 librados: calling monclient init 2024-09-16T03:54:42.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.159+0000 7f5980bff640 10 monclient: init 2024-09-16T03:54:42.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.159+0000 7f5980bff640 10 monclient: _reopen_session rank -1 2024-09-16T03:54:42.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.159+0000 7f5980bff640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:54:42.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.159+0000 7f5980bff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f597c0a1330 0x7f597c071c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:54:42.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.159+0000 7f5980bff640 10 monclient(hunting): picked mon.noname-c con 0x7f597c0a1330 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:54:42.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.159+0000 7f5980bff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f597c0a1710 0x7f597c1315d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:54:42.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.159+0000 7f5980bff640 10 monclient(hunting): picked mon.noname-a con 0x7f597c0a1710 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:54:42.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.159+0000 7f5980bff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f597c1340f0 0x7f597c131b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:54:42.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.159+0000 7f5980bff640 10 monclient(hunting): picked mon.noname-b con 0x7f597c1340f0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:54:42.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.159+0000 7f5980bff640 10 monclient(hunting): start opening mon connection 2024-09-16T03:54:42.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.159+0000 7f5980bff640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f597c13ed60 con 0x7f597c0a1710 2024-09-16T03:54:42.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.159+0000 7f5980bff640 10 monclient(hunting): start opening mon connection 2024-09-16T03:54:42.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.159+0000 7f5980bff640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f597c13ebe0 con 0x7f597c1340f0 2024-09-16T03:54:42.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.159+0000 7f5980bff640 10 monclient(hunting): start opening mon connection 2024-09-16T03:54:42.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.159+0000 7f5980bff640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f597c13eee0 con 0x7f597c0a1330 2024-09-16T03:54:42.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.159+0000 7f5980bff640 10 monclient(hunting): _renew_subs 2024-09-16T03:54:42.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.159+0000 7f5980bff640 10 monclient(hunting): authenticate will time out at 1729.131244s 2024-09-16T03:54:42.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.159+0000 7f597a59c640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f597c0a1330 0x7f597c071c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:54:42.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.159+0000 7f597a59c640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f597c0a1330 0x7f597c071c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:58270/0 (socket says 172.21.15.42:58270) 2024-09-16T03:54:42.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.159+0000 7f597a59c640 1 -- 172.21.15.42:0/864445096 learned_addr learned my addr 172.21.15.42:0/864445096 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:54:42.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.159+0000 7f597a59c640 10 monclient(hunting): get_auth_request con 0x7f597c0a1330 auth_method 0 2024-09-16T03:54:42.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.159+0000 7f597a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:54:42.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.159+0000 7f597a59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:54:42.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.159+0000 7f597a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:54:42.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.159+0000 7f5979d9b640 1 --2- 172.21.15.42:0/864445096 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f597c0a1710 0x7f597c1315d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:54:42.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.159+0000 7f597a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:54:42.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.159+0000 7f597a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:54:42.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.159+0000 7f597a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:54:42.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.159+0000 7f5979d9b640 10 monclient(hunting): get_auth_request con 0x7f597c0a1710 auth_method 0 2024-09-16T03:54:42.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.159+0000 7f5979d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:54:42.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.159+0000 7f5979d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:54:42.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.159+0000 7f5979d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:54:42.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.159+0000 7f597ad9d640 1 --2- 172.21.15.42:0/864445096 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f597c1340f0 0x7f597c131b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:54:42.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.159+0000 7f597ad9d640 10 monclient(hunting): get_auth_request con 0x7f597c1340f0 auth_method 0 2024-09-16T03:54:42.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.159+0000 7f597ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:54:42.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.159+0000 7f597ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:54:42.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.159+0000 7f597ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:54:42.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.159+0000 7f5979d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:54:42.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.159+0000 7f5979d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:54:42.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.159+0000 7f5979d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:54:42.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.159+0000 7f597a59c640 10 monclient(hunting): handle_auth_done global_id 854111 payload 824 2024-09-16T03:54:42.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.159+0000 7f597a59c640 1 -- 172.21.15.42:0/864445096 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f597c1340f0 msgr2=0x7f597c131b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:54:42.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.159+0000 7f597a59c640 1 --2- 172.21.15.42:0/864445096 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f597c1340f0 0x7f597c131b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:54:42.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.159+0000 7f597a59c640 1 -- 172.21.15.42:0/864445096 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f597c0a1710 msgr2=0x7f597c1315d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:54:42.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.159+0000 7f597a59c640 1 --2- 172.21.15.42:0/864445096 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f597c0a1710 0x7f597c1315d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:54:42.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.159+0000 7f597a59c640 10 monclient: _finish_hunting 0 2024-09-16T03:54:42.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.159+0000 7f597a59c640 1 monclient: found mon.noname-c 2024-09-16T03:54:42.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.159+0000 7f597a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:54:42.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.159+0000 7f597a59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:54:42.180 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.159+0000 7f597a59c640 1 -- 172.21.15.42:0/864445096 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f597c11c720 con 0x7f597c0a1330 2024-09-16T03:54:42.180 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.163+0000 7f597a59c640 1 --2- 172.21.15.42:0/864445096 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f597c0a1330 0x7f597c071c20 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f596c00e960 tx=0x7f596c00ee20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:54:42.180 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.163+0000 7f596affd640 10 client.?.objecter ms_handle_connect 0x7f597c0a1330 2024-09-16T03:54:42.180 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.163+0000 7f596affd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:54:42.180 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.163+0000 7f596affd640 1 -- 172.21.15.42:0/864445096 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f596c017070 con 0x7f597c0a1330 2024-09-16T03:54:42.180 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.163+0000 7f596affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:54:42.180 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.163+0000 7f596affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:54:42.180 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.163+0000 7f596affd640 10 monclient: dump: 2024-09-16T03:54:42.180 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:54:42.180 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:54:42.180 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:42.180 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:42.180 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:54:42.180 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:54:42.181 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:54:42.181 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:54:42.181 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:54:42.181 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:54:42.181 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.163+0000 7f596affd640 10 monclient: _finish_auth 0 2024-09-16T03:54:42.181 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.163+0000 7f596affd640 10 monclient: _check_auth_tickets 2024-09-16T03:54:42.181 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.163+0000 7f596affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:54:42.181 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.163+0000 7f596affd640 1 -- 172.21.15.42:0/864445096 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f596c00ca00 con 0x7f597c0a1330 2024-09-16T03:54:42.181 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.163+0000 7f5980bff640 5 monclient: authenticate success, global_id 854111 2024-09-16T03:54:42.181 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.163+0000 7f5980bff640 10 monclient: _renew_subs 2024-09-16T03:54:42.181 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.163+0000 7f5980bff640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:54:42.181 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.163+0000 7f5980bff640 1 -- 172.21.15.42:0/864445096 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f597c073a30 con 0x7f597c0a1330 2024-09-16T03:54:42.181 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.163+0000 7f596affd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:54:42.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.163+0000 7f596affd640 1 -- 172.21.15.42:0/864445096 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f596c00cdb0 con 0x7f597c0a1330 2024-09-16T03:54:42.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.163+0000 7f596affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:54:42.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.163+0000 7f596affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:54:42.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.163+0000 7f596affd640 10 monclient: dump: 2024-09-16T03:54:42.182 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:54:42.182 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:54:42.182 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:42.182 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:42.182 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:54:42.182 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:54:42.183 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:54:42.183 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:54:42.183 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:54:42.183 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:54:42.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.163+0000 7f5978d99640 4 set_mon_vals no callback set 2024-09-16T03:54:42.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.163+0000 7f5980bff640 10 client.854111.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:54:42.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.163+0000 7f5980bff640 10 monclient: _renew_subs 2024-09-16T03:54:42.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.163+0000 7f5980bff640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:54:42.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.163+0000 7f5980bff640 1 -- 172.21.15.42:0/864445096 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f597c11e9d0 con 0x7f597c0a1330 2024-09-16T03:54:42.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.163+0000 7f5980bff640 1 librados: init done 2024-09-16T03:54:42.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.163+0000 7f5980bff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:54:42.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.163+0000 7f5980bff640 1 --2- 172.21.15.42:0/864445096 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f597c1cf230 0x7f597c1cf610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:54:42.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.163+0000 7f5980bff640 10 monclient(hunting): start opening mon connection 2024-09-16T03:54:42.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.163+0000 7f5980bff640 1 -- 172.21.15.42:0/864445096 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f597c11eef0 con 0x7f597c1cf230 2024-09-16T03:54:42.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.163+0000 7f596affd640 1 -- 172.21.15.42:0/864445096 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f596c01a070 con 0x7f597c0a1330 2024-09-16T03:54:42.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.163+0000 7f5979d9b640 1 --2- 172.21.15.42:0/864445096 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f597c1cf230 0x7f597c1cf610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:54:42.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.163+0000 7f5979d9b640 10 monclient: get_auth_request con 0x7f597c1cf230 auth_method 0 2024-09-16T03:54:42.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.163+0000 7f5979d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:54:42.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.163+0000 7f5979d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:54:42.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.163+0000 7f5979d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:54:42.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.163+0000 7f5979d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:54:42.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.163+0000 7f5979d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:54:42.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.163+0000 7f5979d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:54:42.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.163+0000 7f596affd640 1 --2- 172.21.15.42:0/864445096 >> v2:172.21.15.154:6812/254845342 conn(0x7f595c037840 0x7f595c039d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:54:42.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.163+0000 7f5979d9b640 10 monclient(hunting): handle_auth_done global_id 854114 payload 824 2024-09-16T03:54:42.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.163+0000 7f596affd640 1 -- 172.21.15.42:0/864445096 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(386..386 src has 286..386) ==== 5713+0+0 (secure 0 0 0) 0x7f596c048fb0 con 0x7f597c0a1330 2024-09-16T03:54:42.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.163+0000 7f596affd640 10 client.854111.objecter ms_dispatch 0x7f597c000ba0 osd_map(386..386 src has 286..386) 2024-09-16T03:54:42.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.163+0000 7f596affd640 3 client.854111.objecter handle_osd_map got epochs [386,386] > 0 2024-09-16T03:54:42.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.163+0000 7f596affd640 3 client.854111.objecter handle_osd_map decoding full epoch 386 2024-09-16T03:54:42.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.163+0000 7f5979d9b640 1 -- 172.21.15.42:0/864445096 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f597c09f180 con 0x7f597c1cf230 2024-09-16T03:54:42.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.163+0000 7f596affd640 20 client.854111.objecter dump_active .. 0 homeless 2024-09-16T03:54:42.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.163+0000 7f597ad9d640 1 --2- 172.21.15.42:0/864445096 >> v2:172.21.15.154:6812/254845342 conn(0x7f595c037840 0x7f595c039d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:54:42.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.163+0000 7f596affd640 10 client.854111.objecter ms_handle_connect 0x7f597c1cf230 2024-09-16T03:54:42.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.163+0000 7f596affd640 10 client.854111.objecter resend_mon_ops 2024-09-16T03:54:42.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.163+0000 7f5979d9b640 1 --2- 172.21.15.42:0/864445096 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f597c1cf230 0x7f597c1cf610 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f59700097c0 tx=0x7f597001aba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:54:42.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.163+0000 7f597ad9d640 10 monclient: get_auth_request con 0x7f595c037840 auth_method 0 2024-09-16T03:54:42.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.163+0000 7f596affd640 1 -- 172.21.15.42:0/864445096 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59700bb030 con 0x7f597c1cf230 2024-09-16T03:54:42.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.163+0000 7f596affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:54:42.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.163+0000 7f596affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:54:42.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.163+0000 7f596affd640 10 monclient: dump: 2024-09-16T03:54:42.186 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:54:42.186 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:54:42.186 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:42.186 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:42.187 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:54:42.187 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:54:42.187 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:54:42.187 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:54:42.187 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:54:42.188 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:54:42.188 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.163+0000 7f596affd640 1 -- 172.21.15.42:0/864445096 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f59700093e0 con 0x7f597c1cf230 2024-09-16T03:54:42.188 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.163+0000 7f596affd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:54:42.188 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.163+0000 7f596affd640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:54:42.188 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.163+0000 7f596affd640 1 -- 172.21.15.42:0/864445096 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f597c1cf230 msgr2=0x7f597c1cf610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:54:42.188 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.163+0000 7f596affd640 1 --2- 172.21.15.42:0/864445096 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f597c1cf230 0x7f597c1cf610 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f59700097c0 tx=0x7f597001aba0 comp rx=0 tx=0).stop 2024-09-16T03:54:42.188 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.163+0000 7f596affd640 1 -- 172.21.15.42:0/864445096 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f597c1cf230 msgr2=0x7f597c1cf610 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:54:42.188 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.163+0000 7f596affd640 1 --2- 172.21.15.42:0/864445096 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f597c1cf230 0x7f597c1cf610 secure :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f59700097c0 tx=0x7f597001aba0 comp rx=0 tx=0).stop 2024-09-16T03:54:42.188 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.163+0000 7f596affd640 10 client.854111.objecter ms_handle_connect 0x7f595c037840 2024-09-16T03:54:42.188 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.163+0000 7f597ad9d640 1 --2- 172.21.15.42:0/864445096 >> v2:172.21.15.154:6812/254845342 conn(0x7f595c037840 0x7f595c039d00 secure :-1 s=READY pgs=3615 cs=0 l=1 rev1=1 crypto rx=0x7f59740012f0 tx=0x7f5974009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:54:42.190 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.171+0000 7f5980bff640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:54:42.190 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.171+0000 7f5980bff640 1 --2- 172.21.15.42:0/864445096 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f597c1cfb50 0x7f597c06d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:54:42.190 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.171+0000 7f5980bff640 10 monclient(hunting): start opening mon connection 2024-09-16T03:54:42.190 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.171+0000 7f5980bff640 1 -- 172.21.15.42:0/864445096 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f597c11c430 con 0x7f597c1cfb50 2024-09-16T03:54:42.190 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.171+0000 7f5979d9b640 1 --2- 172.21.15.42:0/864445096 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f597c1cfb50 0x7f597c06d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:54:42.190 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.171+0000 7f5979d9b640 10 monclient: get_auth_request con 0x7f597c1cfb50 auth_method 0 2024-09-16T03:54:42.190 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.171+0000 7f5979d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:54:42.191 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.171+0000 7f5979d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:54:42.191 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.171+0000 7f5979d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:54:42.191 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.171+0000 7f5979d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:54:42.191 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.171+0000 7f5979d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:54:42.191 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.171+0000 7f5979d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:54:42.191 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.171+0000 7f5979d9b640 10 monclient(hunting): handle_auth_done global_id 854117 payload 824 2024-09-16T03:54:42.191 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.171+0000 7f5979d9b640 1 -- 172.21.15.42:0/864445096 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f597c069040 con 0x7f597c1cfb50 2024-09-16T03:54:42.191 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.171+0000 7f596affd640 10 client.854111.objecter ms_handle_connect 0x7f597c1cfb50 2024-09-16T03:54:42.191 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.171+0000 7f596affd640 10 client.854111.objecter resend_mon_ops 2024-09-16T03:54:42.191 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.171+0000 7f5979d9b640 1 --2- 172.21.15.42:0/864445096 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f597c1cfb50 0x7f597c06d8e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f597001aa10 tx=0x7f5970019100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:54:42.191 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.171+0000 7f596affd640 1 -- 172.21.15.42:0/864445096 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59700bb070 con 0x7f597c1cfb50 2024-09-16T03:54:42.191 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.171+0000 7f596affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:54:42.191 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.171+0000 7f596affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:54:42.191 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.171+0000 7f596affd640 10 monclient: dump: 2024-09-16T03:54:42.192 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:54:42.192 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:54:42.192 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:42.192 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:42.192 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:54:42.192 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:54:42.192 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:54:42.192 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:54:42.192 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:54:42.192 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:54:42.192 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.171+0000 7f596affd640 1 -- 172.21.15.42:0/864445096 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4545 (secure 0 0 0) 0x7f5970003620 con 0x7f597c1cfb50 2024-09-16T03:54:42.192 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.171+0000 7f596affd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:54:42.192 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.171+0000 7f596affd640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:54:42.192 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.171+0000 7f596affd640 1 -- 172.21.15.42:0/864445096 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f597c1cfb50 msgr2=0x7f597c06d8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:54:42.193 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.171+0000 7f596affd640 1 --2- 172.21.15.42:0/864445096 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f597c1cfb50 0x7f597c06d8e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f597001aa10 tx=0x7f5970019100 comp rx=0 tx=0).stop 2024-09-16T03:54:42.193 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.171+0000 7f596affd640 1 -- 172.21.15.42:0/864445096 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f597c1cfb50 msgr2=0x7f597c06d8e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:54:42.193 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.171+0000 7f596affd640 1 --2- 172.21.15.42:0/864445096 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f597c1cfb50 0x7f597c06d8e0 secure :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f597001aa10 tx=0x7f5970019100 comp rx=0 tx=0).stop 2024-09-16T03:54:42.193 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:54:42.193 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:54:42.193 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:54:42.193 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T03:54:42.193 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 110, 2024-09-16T03:54:42.193 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 15607, 2024-09-16T03:54:42.193 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:54:42.194 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:54:42.194 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T03:54:42.194 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T03:54:42.194 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:54:42.194 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 0, 2024-09-16T03:54:42.194 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:54:42.194 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:54:42.194 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:54:42.194 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:54:42.194 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:54:42.194 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:54:42.194 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:54:42.194 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:54:42.194 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:54:42.194 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:54:42.195 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:54:42.195 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:54:42.195 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:54:42.195 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:54:42.195 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:54:42.195 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:54:42.195 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:54:42.195 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:54:42.195 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:54:42.195 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:54:42.195 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:54:42.195 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:54:42.195 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:54:42.196 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:54:42.196 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:54:42.196 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:54:42.196 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:54:42.196 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:54:42.196 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:54:42.196 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:54:42.196 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:54:42.196 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:54:42.196 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:54:42.196 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:54:42.196 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:54:42.196 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:54:42.196 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:54:42.197 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:54:42.197 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:54:42.197 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:54:42.197 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:54:42.197 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:54:42.197 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:54:42.197 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:54:42.197 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:54:42.197 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:54:42.197 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:54:42.197 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:54:42.197 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:54:42.197 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:54:42.198 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:54:42.198 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:54:42.198 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:54:42.198 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:54:42.198 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:54:42.198 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:54:42.198 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:54:42.198 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:54:42.198 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:54:42.198 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:54:42.198 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:54:42.198 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:54:42.198 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:54:42.198 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:54:42.199 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:54:42.199 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:54:42.199 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:54:42.199 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:54:42.199 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:54:42.199 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:54:42.199 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:54:42.199 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:54:42.199 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:54:42.199 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:54:42.199 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:54:42.199 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:54:42.199 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:54:42.199 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:54:42.200 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:54:42.200 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:54:42.200 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:54:42.200 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:54:42.200 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:54:42.200 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:54:42.200 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:54:42.200 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:54:42.200 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:54:42.200 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:54:42.200 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:54:42.200 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:54:42.200 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:54:42.201 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:54:42.201 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:54:42.201 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:54:42.201 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:54:42.201 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:54:42.201 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:54:42.201 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:54:42.201 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:54:42.201 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:54:42.201 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:54:42.201 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:54:42.201 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:54:42.201 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:54:42.201 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:54:42.202 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:54:42.202 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:54:42.202 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:54:42.202 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:54:42.202 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:54:42.202 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:54:42.202 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:54:42.202 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:54:42.202 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:54:42.202 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:54:42.202 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:54:42.202 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:54:42.202 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:54:42.202 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:54:42.203 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:54:42.203 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:54:42.203 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:54:42.203 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:54:42.203 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:54:42.203 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:54:42.203 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:54:42.203 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:54:42.203 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:54:42.203 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:54:42.203 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:54:42.203 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:54:42.203 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:54:42.203 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:54:42.204 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:54:42.204 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:54:42.204 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:54:42.204 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:54:42.204 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:54:42.204 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:54:42.204 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:54:42.204 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:54:42.204 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:54:42.204 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:54:42.204 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:54:42.204 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:54:42.204 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 2 2024-09-16T03:54:42.205 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:54:42.205 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:54:42.205 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:54:42.205 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:54:42.205 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:54:42.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.171+0000 7f5980bff640 10 librados: watch_flush enter 2024-09-16T03:54:42.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.171+0000 7f5980bff640 10 librados: watch_flush exit 2024-09-16T03:54:42.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.171+0000 7f5980bff640 20 client.854111.objecter shutdown clearing up homeless session... 2024-09-16T03:54:42.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.171+0000 7f5980bff640 10 client.854111.objecter successfully canceled tick 2024-09-16T03:54:42.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.171+0000 7f5980bff640 1 -- 172.21.15.42:0/864445096 >> v2:172.21.15.154:6812/254845342 conn(0x7f595c037840 msgr2=0x7f595c039d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:54:42.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.171+0000 7f5980bff640 1 --2- 172.21.15.42:0/864445096 >> v2:172.21.15.154:6812/254845342 conn(0x7f595c037840 0x7f595c039d00 secure :-1 s=READY pgs=3615 cs=0 l=1 rev1=1 crypto rx=0x7f59740012f0 tx=0x7f5974009040 comp rx=0 tx=0).stop 2024-09-16T03:54:42.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.171+0000 7f5980bff640 10 monclient: shutdown 2024-09-16T03:54:42.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.171+0000 7f5980bff640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:54:42.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.171+0000 7f5980bff640 1 -- 172.21.15.42:0/864445096 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f597c0a1330 msgr2=0x7f597c071c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:54:42.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.171+0000 7f5980bff640 1 --2- 172.21.15.42:0/864445096 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f597c0a1330 0x7f597c071c20 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f596c00e960 tx=0x7f596c00ee20 comp rx=0 tx=0).stop 2024-09-16T03:54:42.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.171+0000 7f597ad9d640 1 -- 172.21.15.42:0/864445096 reap_dead start 2024-09-16T03:54:42.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.171+0000 7f597ad9d640 1 -- 172.21.15.42:0/864445096 reap_dead start 2024-09-16T03:54:42.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.171+0000 7f5980bff640 1 -- 172.21.15.42:0/864445096 shutdown_connections 2024-09-16T03:54:42.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.171+0000 7f5980bff640 1 -- 172.21.15.42:0/864445096 >> 172.21.15.42:0/864445096 conn(0x7f597c0a2420 msgr2=0x7f597c133900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:54:42.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.171+0000 7f5980bff640 1 -- 172.21.15.42:0/864445096 shutdown_connections 2024-09-16T03:54:42.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.171+0000 7f5980bff640 1 -- 172.21.15.42:0/864445096 wait complete. 2024-09-16T03:54:42.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.171+0000 7f5980bff640 1 librados: shutdown 2024-09-16T03:54:42.208 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:54:42.209 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T03:54:42.209 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T03:54:42.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.319+0000 7f2d5a1e3640 10 monclient: get_monmap_and_config 2024-09-16T03:54:42.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.319+0000 7f2d5a1e3640 10 monclient: build_initial_monmap 2024-09-16T03:54:42.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.319+0000 7f2d5a1e3640 10 monclient: monmap: 2024-09-16T03:54:42.322 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:54:42.322 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:54:42.322 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:54:42.320635+0000 2024-09-16T03:54:42.322 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:54:42.320635+0000 2024-09-16T03:54:42.322 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:54:42.322 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:54:42.323 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:54:42.323 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:54:42.323 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:54:42.323 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:54:42.323 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:54:42.323 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:54:42.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.319+0000 7f2d5a1e3640 1 Processor -- start 2024-09-16T03:54:42.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.319+0000 7f2d5a1e3640 1 -- start start 2024-09-16T03:54:42.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.319+0000 7f2d5a1e3640 10 monclient: init 2024-09-16T03:54:42.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.319+0000 7f2d5a1e3640 10 monclient: _reopen_session rank -1 2024-09-16T03:54:42.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.319+0000 7f2d5a1e3640 10 monclient: _add_conns ranks=[1,3,0,2,4] 2024-09-16T03:54:42.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.319+0000 7f2d5a1e3640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2d540a3450 addr v1:172.21.15.154:6789/0 2024-09-16T03:54:42.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.319+0000 7f2d5a1e3640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f2d540a0ea0 0x7f2d5413c460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:54:42.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.319+0000 7f2d5a1e3640 10 monclient(hunting): picked mon.noname-a con 0x7f2d540a0ea0 addr v2:172.21.15.42:3300/0 2024-09-16T03:54:42.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.319+0000 7f2d5a1e3640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2d540a3f40 addr v1:172.21.15.42:6789/0 2024-09-16T03:54:42.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.319+0000 7f2d5a1e3640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2d540a00e0 con 0x7f2d540a3f40 2024-09-16T03:54:42.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.319+0000 7f2d5a1e3640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2d5409fea0 con 0x7f2d540a3450 2024-09-16T03:54:42.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.319+0000 7f2d5a1e3640 10 monclient(hunting): start opening mon connection 2024-09-16T03:54:42.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.319+0000 7f2d5a1e3640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f2d540a1b40 con 0x7f2d540a0ea0 2024-09-16T03:54:42.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.319+0000 7f2d5a1e3640 10 monclient(hunting): _renew_subs 2024-09-16T03:54:42.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.319+0000 7f2d5a1e3640 10 monclient(hunting): authenticate will time out at 1729.290012s 2024-09-16T03:54:42.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.319+0000 7f2d52ffd640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f2d540a0ea0 0x7f2d5413c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:54:42.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.319+0000 7f2d53fff640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f2d540a3f40 0x7f2d5413e9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:57210/0 (socket says 172.21.15.42:57210) 2024-09-16T03:54:42.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.319+0000 7f2d53fff640 1 -- 172.21.15.42:0/1445637545 learned_addr learned my addr 172.21.15.42:0/1445637545 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:54:42.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.319+0000 7f2d52ffd640 10 monclient(hunting): get_auth_request con 0x7f2d540a0ea0 auth_method 0 2024-09-16T03:54:42.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.319+0000 7f2d52ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:54:42.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.319+0000 7f2d52ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:54:42.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.319+0000 7f2d52ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:54:42.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.319+0000 7f2d52ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:54:42.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.319+0000 7f2d52ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:54:42.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.319+0000 7f2d52ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:54:42.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.319+0000 7f2d527fc640 1 -- 172.21.15.42:0/1445637545 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3851228358 0 0) 0x7f2d540a00e0 con 0x7f2d540a3f40 2024-09-16T03:54:42.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.319+0000 7f2d527fc640 10 monclient(hunting): _negotiate 2024-09-16T03:54:42.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.319+0000 7f2d527fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:54:42.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.319+0000 7f2d527fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:54:42.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.319+0000 7f2d527fc640 10 monclient(hunting): my global_id is 824448 2024-09-16T03:54:42.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.319+0000 7f2d527fc640 1 -- 172.21.15.42:0/1445637545 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2d40003530 con 0x7f2d540a3f40 2024-09-16T03:54:42.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.319+0000 7f2d52ffd640 10 monclient(hunting): handle_auth_done global_id 824445 payload 274 2024-09-16T03:54:42.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.319+0000 7f2d52ffd640 1 -- 172.21.15.42:0/1445637545 >> v1:172.21.15.154:6789/0 conn(0x7f2d540a3450 legacy=0x7f2d540a3830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:54:42.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.319+0000 7f2d52ffd640 1 -- 172.21.15.42:0/1445637545 >> v1:172.21.15.42:6789/0 conn(0x7f2d540a3f40 legacy=0x7f2d5413e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:54:42.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.319+0000 7f2d52ffd640 10 monclient: _finish_hunting 0 2024-09-16T03:54:42.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.319+0000 7f2d52ffd640 1 monclient: found mon.noname-a 2024-09-16T03:54:42.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.319+0000 7f2d52ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:54:42.329 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.319+0000 7f2d52ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:54:42.329 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.319+0000 7f2d52ffd640 1 -- 172.21.15.42:0/1445637545 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d540a04e0 con 0x7f2d540a0ea0 2024-09-16T03:54:42.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.323+0000 7f2d52ffd640 1 --2- 172.21.15.42:0/1445637545 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2d540a0ea0 0x7f2d5413c460 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f2d3c0097c0 tx=0x7f2d3c01c360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e628126d090fd27 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:54:42.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.323+0000 7f2d527fc640 1 -- 172.21.15.42:0/1445637545 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d3c0b0070 con 0x7f2d540a0ea0 2024-09-16T03:54:42.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.323+0000 7f2d527fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:54:42.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.323+0000 7f2d527fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:54:42.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.323+0000 7f2d527fc640 10 monclient: dump: 2024-09-16T03:54:42.337 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:54:42.337 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:54:42.337 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:42.337 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:42.337 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:54:42.337 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:54:42.337 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:54:42.337 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:54:42.337 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:54:42.337 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:54:42.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.323+0000 7f2d527fc640 10 monclient: _finish_auth 0 2024-09-16T03:54:42.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.323+0000 7f2d527fc640 10 monclient: _check_auth_tickets 2024-09-16T03:54:42.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.323+0000 7f2d527fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:54:42.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.323+0000 7f2d527fc640 1 -- 172.21.15.42:0/1445637545 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d3c0ab600 con 0x7f2d540a0ea0 2024-09-16T03:54:42.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.323+0000 7f2d5a1e3640 5 monclient: authenticate success, global_id 824445 2024-09-16T03:54:42.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.323+0000 7f2d5a1e3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:54:42.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.323+0000 7f2d527fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:54:42.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.323+0000 7f2d5a1e3640 10 monclient: get_monmap_and_config success 2024-09-16T03:54:42.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.323+0000 7f2d5a1e3640 4 set_mon_vals no callback set 2024-09-16T03:54:42.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.323+0000 7f2d527fc640 1 -- 172.21.15.42:0/1445637545 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d3c0ab9d0 con 0x7f2d540a0ea0 2024-09-16T03:54:42.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.323+0000 7f2d527fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:54:42.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.323+0000 7f2d527fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:54:42.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.323+0000 7f2d527fc640 10 monclient: dump: 2024-09-16T03:54:42.339 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:54:42.339 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:54:42.339 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:42.339 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:42.339 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:54:42.339 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:54:42.339 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:54:42.339 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:54:42.339 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:54:42.339 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:54:42.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.323+0000 7f2d5a1e3640 10 monclient: shutdown 2024-09-16T03:54:42.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.323+0000 7f2d5a1e3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:54:42.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.323+0000 7f2d5a1e3640 1 -- 172.21.15.42:0/1445637545 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2d540a0ea0 msgr2=0x7f2d5413c460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:54:42.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.323+0000 7f2d5a1e3640 1 --2- 172.21.15.42:0/1445637545 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2d540a0ea0 0x7f2d5413c460 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f2d3c0097c0 tx=0x7f2d3c01c360 comp rx=0 tx=0).stop 2024-09-16T03:54:42.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.323+0000 7f2d5a1e3640 1 -- 172.21.15.42:0/1445637545 shutdown_connections 2024-09-16T03:54:42.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.323+0000 7f2d5a1e3640 1 --2- 172.21.15.42:0/1445637545 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2d540a0ea0 0x7f2d5413c460 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:54:42.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.323+0000 7f2d5a1e3640 1 -- 172.21.15.42:0/1445637545 >> 172.21.15.42:0/1445637545 conn(0x7f2d540756e0 msgr2=0x7f2d54075af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:54:42.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.323+0000 7f2d5a1e3640 1 -- 172.21.15.42:0/1445637545 shutdown_connections 2024-09-16T03:54:42.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.323+0000 7f2d5a1e3640 1 -- 172.21.15.42:0/1445637545 wait complete. 2024-09-16T03:54:42.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.323+0000 7f2d5a1e3640 10 monclient: build_initial_monmap 2024-09-16T03:54:42.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.323+0000 7f2d5a1e3640 10 monclient: monmap: 2024-09-16T03:54:42.340 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:54:42.340 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:54:42.341 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:54:42.341 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:54:42.341 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:54:42.341 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:54:42.341 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:54:42.341 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:54:42.341 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:54:42.341 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:54:42.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.323+0000 7f2d5a1e3640 1 librados: starting msgr at 2024-09-16T03:54:42.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.323+0000 7f2d5a1e3640 1 librados: starting objecter 2024-09-16T03:54:42.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.323+0000 7f2d5a1e3640 1 Processor -- start 2024-09-16T03:54:42.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.323+0000 7f2d5a1e3640 1 -- start start 2024-09-16T03:54:42.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.323+0000 7f2d5a1e3640 1 librados: setting wanted keys 2024-09-16T03:54:42.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.323+0000 7f2d5a1e3640 1 librados: calling monclient init 2024-09-16T03:54:42.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.323+0000 7f2d5a1e3640 10 monclient: init 2024-09-16T03:54:42.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.323+0000 7f2d5a1e3640 10 monclient: _reopen_session rank -1 2024-09-16T03:54:42.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.323+0000 7f2d5a1e3640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T03:54:42.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.323+0000 7f2d5a1e3640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d540a0ea0 0x7f2d5413ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:54:42.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.323+0000 7f2d5a1e3640 10 monclient(hunting): picked mon.noname-b con 0x7f2d540a0ea0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:54:42.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.323+0000 7f2d5a1e3640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2d540a3450 0x7f2d5413f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:54:42.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.323+0000 7f2d5a1e3640 10 monclient(hunting): picked mon.noname-c con 0x7f2d540a3450 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:54:42.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.323+0000 7f2d5a1e3640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2d540a3f40 0x7f2d541249d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:54:42.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.323+0000 7f2d5a1e3640 10 monclient(hunting): picked mon.noname-a con 0x7f2d540a3f40 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:54:42.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.323+0000 7f2d5a1e3640 10 monclient(hunting): start opening mon connection 2024-09-16T03:54:42.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.323+0000 7f2d5a1e3640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f2d54142b00 con 0x7f2d540a3f40 2024-09-16T03:54:42.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.323+0000 7f2d5a1e3640 10 monclient(hunting): start opening mon connection 2024-09-16T03:54:42.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.323+0000 7f2d5a1e3640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f2d54142c80 con 0x7f2d540a0ea0 2024-09-16T03:54:42.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.323+0000 7f2d5a1e3640 10 monclient(hunting): start opening mon connection 2024-09-16T03:54:42.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.323+0000 7f2d5a1e3640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f2d54142e00 con 0x7f2d540a3450 2024-09-16T03:54:42.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.323+0000 7f2d5a1e3640 10 monclient(hunting): _renew_subs 2024-09-16T03:54:42.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.323+0000 7f2d5a1e3640 10 monclient(hunting): authenticate will time out at 1729.295039s 2024-09-16T03:54:42.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.323+0000 7f2d52ffd640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2d540a3450 0x7f2d5413f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:54:42.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.323+0000 7f2d53fff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2d540a3f40 0x7f2d541249d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:54:42.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.323+0000 7f2d52ffd640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2d540a3450 0x7f2d5413f010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:58300/0 (socket says 172.21.15.42:58300) 2024-09-16T03:54:42.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.323+0000 7f2d52ffd640 1 -- 172.21.15.42:0/3029903204 learned_addr learned my addr 172.21.15.42:0/3029903204 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:54:42.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.323+0000 7f2d52ffd640 10 monclient(hunting): get_auth_request con 0x7f2d540a3450 auth_method 0 2024-09-16T03:54:42.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.323+0000 7f2d52ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:54:42.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.323+0000 7f2d52ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:54:42.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.323+0000 7f2d52ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:54:42.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.323+0000 7f2d53fff640 10 monclient(hunting): get_auth_request con 0x7f2d540a3f40 auth_method 0 2024-09-16T03:54:42.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.323+0000 7f2d53fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:54:42.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.323+0000 7f2d53fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:54:42.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.323+0000 7f2d53fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:54:42.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.323+0000 7f2d52ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:54:42.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.323+0000 7f2d52ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:54:42.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.323+0000 7f2d52ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:54:42.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.323+0000 7f2d53fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:54:42.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.323+0000 7f2d53fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:54:42.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.323+0000 7f2d53fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:54:42.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.323+0000 7f2d537fe640 1 --2- 172.21.15.42:0/3029903204 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d540a0ea0 0x7f2d5413ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:54:42.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.323+0000 7f2d537fe640 10 monclient(hunting): get_auth_request con 0x7f2d540a0ea0 auth_method 0 2024-09-16T03:54:42.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.323+0000 7f2d537fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:54:42.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.323+0000 7f2d537fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:54:42.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.323+0000 7f2d537fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:54:42.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.323+0000 7f2d52ffd640 10 monclient(hunting): handle_auth_done global_id 854120 payload 824 2024-09-16T03:54:42.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.323+0000 7f2d52ffd640 1 -- 172.21.15.42:0/3029903204 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d540a0ea0 msgr2=0x7f2d5413ead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:54:42.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.323+0000 7f2d52ffd640 1 --2- 172.21.15.42:0/3029903204 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d540a0ea0 0x7f2d5413ead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:54:42.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.323+0000 7f2d52ffd640 1 -- 172.21.15.42:0/3029903204 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2d540a3f40 msgr2=0x7f2d541249d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:54:42.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.323+0000 7f2d52ffd640 1 --2- 172.21.15.42:0/3029903204 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2d540a3f40 0x7f2d541249d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:54:42.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.323+0000 7f2d52ffd640 10 monclient: _finish_hunting 0 2024-09-16T03:54:42.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.323+0000 7f2d52ffd640 1 monclient: found mon.noname-c 2024-09-16T03:54:42.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.323+0000 7f2d52ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:54:42.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.323+0000 7f2d52ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:54:42.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.323+0000 7f2d52ffd640 1 -- 172.21.15.42:0/3029903204 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d54126f50 con 0x7f2d540a3450 2024-09-16T03:54:42.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.323+0000 7f2d52ffd640 1 --2- 172.21.15.42:0/3029903204 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2d540a3450 0x7f2d5413f010 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f2d3c018f70 tx=0x7f2d3c019100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:54:42.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.323+0000 7f2d4bfff640 10 client.?.objecter ms_handle_connect 0x7f2d540a3450 2024-09-16T03:54:42.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.323+0000 7f2d4bfff640 10 client.?.objecter resend_mon_ops 2024-09-16T03:54:42.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.323+0000 7f2d4bfff640 1 -- 172.21.15.42:0/3029903204 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d3c0b0040 con 0x7f2d540a3450 2024-09-16T03:54:42.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.323+0000 7f2d4bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:54:42.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.323+0000 7f2d4bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:54:42.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.323+0000 7f2d4bfff640 10 monclient: dump: 2024-09-16T03:54:42.347 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:54:42.347 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:54:42.347 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:42.348 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:42.348 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:54:42.348 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:54:42.348 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:54:42.348 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:54:42.348 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:54:42.348 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:54:42.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.323+0000 7f2d4bfff640 10 monclient: _finish_auth 0 2024-09-16T03:54:42.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.323+0000 7f2d4bfff640 10 monclient: _check_auth_tickets 2024-09-16T03:54:42.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.323+0000 7f2d4bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:54:42.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.323+0000 7f2d4bfff640 1 -- 172.21.15.42:0/3029903204 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d3c0b43f0 con 0x7f2d540a3450 2024-09-16T03:54:42.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.323+0000 7f2d4bfff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:54:42.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.323+0000 7f2d4bfff640 1 -- 172.21.15.42:0/3029903204 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d3c0b47c0 con 0x7f2d540a3450 2024-09-16T03:54:42.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.323+0000 7f2d5a1e3640 5 monclient: authenticate success, global_id 854120 2024-09-16T03:54:42.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.323+0000 7f2d5a1e3640 10 monclient: _renew_subs 2024-09-16T03:54:42.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.323+0000 7f2d5a1e3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:54:42.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.323+0000 7f2d5a1e3640 1 -- 172.21.15.42:0/3029903204 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d54128960 con 0x7f2d540a3450 2024-09-16T03:54:42.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.323+0000 7f2d517fa640 4 set_mon_vals no callback set 2024-09-16T03:54:42.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.323+0000 7f2d4bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:54:42.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.323+0000 7f2d4bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:54:42.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.323+0000 7f2d4bfff640 10 monclient: dump: 2024-09-16T03:54:42.350 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:54:42.350 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:54:42.350 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:42.350 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:42.350 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:54:42.350 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:54:42.350 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:54:42.350 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:54:42.351 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:54:42.351 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:54:42.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.323+0000 7f2d5a1e3640 10 client.854120.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:54:42.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.323+0000 7f2d5a1e3640 10 monclient: _renew_subs 2024-09-16T03:54:42.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.323+0000 7f2d5a1e3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:54:42.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.323+0000 7f2d5a1e3640 1 -- 172.21.15.42:0/3029903204 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d5406e540 con 0x7f2d540a3450 2024-09-16T03:54:42.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.323+0000 7f2d5a1e3640 1 librados: init done 2024-09-16T03:54:42.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.327+0000 7f2d5a1e3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:54:42.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.327+0000 7f2d5a1e3640 1 --2- 172.21.15.42:0/3029903204 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d54128300 0x7f2d541256d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:54:42.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.327+0000 7f2d5a1e3640 10 monclient(hunting): start opening mon connection 2024-09-16T03:54:42.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.327+0000 7f2d5a1e3640 1 -- 172.21.15.42:0/3029903204 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f2d541273f0 con 0x7f2d54128300 2024-09-16T03:54:42.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.327+0000 7f2d4bfff640 1 -- 172.21.15.42:0/3029903204 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f2d3c0b4960 con 0x7f2d540a3450 2024-09-16T03:54:42.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.327+0000 7f2d537fe640 1 --2- 172.21.15.42:0/3029903204 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d54128300 0x7f2d541256d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:54:42.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.327+0000 7f2d4bfff640 1 --2- 172.21.15.42:0/3029903204 >> v2:172.21.15.154:6812/254845342 conn(0x7f2d34037810 0x7f2d34039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:54:42.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.327+0000 7f2d537fe640 10 monclient: get_auth_request con 0x7f2d54128300 auth_method 0 2024-09-16T03:54:42.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.327+0000 7f2d537fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:54:42.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.327+0000 7f2d537fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:54:42.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.327+0000 7f2d537fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:54:42.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.327+0000 7f2d4bfff640 1 -- 172.21.15.42:0/3029903204 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(386..386 src has 286..386) ==== 5713+0+0 (secure 0 0 0) 0x7f2d3c0e2c50 con 0x7f2d540a3450 2024-09-16T03:54:42.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.327+0000 7f2d4bfff640 10 client.854120.objecter ms_dispatch 0x7f2d54000ba0 osd_map(386..386 src has 286..386) 2024-09-16T03:54:42.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.327+0000 7f2d4bfff640 3 client.854120.objecter handle_osd_map got epochs [386,386] > 0 2024-09-16T03:54:42.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.327+0000 7f2d4bfff640 3 client.854120.objecter handle_osd_map decoding full epoch 386 2024-09-16T03:54:42.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.327+0000 7f2d537fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:54:42.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.327+0000 7f2d537fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:54:42.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.327+0000 7f2d537fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:54:42.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.327+0000 7f2d4bfff640 20 client.854120.objecter dump_active .. 0 homeless 2024-09-16T03:54:42.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.327+0000 7f2d53fff640 1 --2- 172.21.15.42:0/3029903204 >> v2:172.21.15.154:6812/254845342 conn(0x7f2d34037810 0x7f2d34039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:54:42.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.327+0000 7f2d53fff640 10 monclient: get_auth_request con 0x7f2d34037810 auth_method 0 2024-09-16T03:54:42.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.327+0000 7f2d537fe640 10 monclient(hunting): handle_auth_done global_id 844186 payload 824 2024-09-16T03:54:42.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.327+0000 7f2d537fe640 1 -- 172.21.15.42:0/3029903204 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2d540a3830 con 0x7f2d54128300 2024-09-16T03:54:42.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.327+0000 7f2d53fff640 1 --2- 172.21.15.42:0/3029903204 >> v2:172.21.15.154:6812/254845342 conn(0x7f2d34037810 0x7f2d34039cd0 secure :-1 s=READY pgs=3616 cs=0 l=1 rev1=1 crypto rx=0x7f2d54128e50 tx=0x7f2d4c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:54:42.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.327+0000 7f2d4bfff640 10 client.854120.objecter ms_handle_connect 0x7f2d34037810 2024-09-16T03:54:42.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.327+0000 7f2d4bfff640 10 client.854120.objecter ms_handle_connect 0x7f2d54128300 2024-09-16T03:54:42.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.327+0000 7f2d4bfff640 10 client.854120.objecter resend_mon_ops 2024-09-16T03:54:42.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.327+0000 7f2d537fe640 1 --2- 172.21.15.42:0/3029903204 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d54128300 0x7f2d541256d0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f2d540a0070 tx=0x7f2d4400e960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:54:42.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.327+0000 7f2d4bfff640 1 -- 172.21.15.42:0/3029903204 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d440052e0 con 0x7f2d54128300 2024-09-16T03:54:42.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.327+0000 7f2d4bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:54:42.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.327+0000 7f2d4bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:54:42.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.327+0000 7f2d4bfff640 10 monclient: dump: 2024-09-16T03:54:42.354 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:54:42.355 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:54:42.355 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:42.355 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:42.355 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:54:42.355 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:54:42.355 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:54:42.355 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:54:42.355 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:54:42.355 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:54:42.355 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.327+0000 7f2d4bfff640 1 -- 172.21.15.42:0/3029903204 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f2d440052e0 con 0x7f2d54128300 2024-09-16T03:54:42.355 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.327+0000 7f2d4bfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:54:42.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.327+0000 7f2d4bfff640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:54:42.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.327+0000 7f2d4bfff640 1 -- 172.21.15.42:0/3029903204 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d54128300 msgr2=0x7f2d541256d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:54:42.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.327+0000 7f2d4bfff640 1 --2- 172.21.15.42:0/3029903204 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d54128300 0x7f2d541256d0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f2d540a0070 tx=0x7f2d4400e960 comp rx=0 tx=0).stop 2024-09-16T03:54:42.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.327+0000 7f2d4bfff640 1 -- 172.21.15.42:0/3029903204 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d54128300 msgr2=0x7f2d541256d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:54:42.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.327+0000 7f2d4bfff640 1 --2- 172.21.15.42:0/3029903204 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d54128300 0x7f2d541256d0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:54:42.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.335+0000 7f2d5a1e3640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:54:42.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.335+0000 7f2d5a1e3640 1 --2- 172.21.15.42:0/3029903204 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d5402c650 0x7f2d5402ca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:54:42.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.335+0000 7f2d5a1e3640 10 monclient(hunting): start opening mon connection 2024-09-16T03:54:42.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.335+0000 7f2d5a1e3640 1 -- 172.21.15.42:0/3029903204 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f2d540a07a0 con 0x7f2d5402c650 2024-09-16T03:54:42.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.335+0000 7f2d537fe640 1 --2- 172.21.15.42:0/3029903204 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d5402c650 0x7f2d5402ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:54:42.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.335+0000 7f2d537fe640 10 monclient: get_auth_request con 0x7f2d5402c650 auth_method 0 2024-09-16T03:54:42.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.335+0000 7f2d537fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:54:42.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.335+0000 7f2d537fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:54:42.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.335+0000 7f2d537fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:54:42.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.335+0000 7f2d537fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:54:42.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.335+0000 7f2d537fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:54:42.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.335+0000 7f2d537fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:54:42.360 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.335+0000 7f2d537fe640 10 monclient(hunting): handle_auth_done global_id 844189 payload 824 2024-09-16T03:54:42.360 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.335+0000 7f2d537fe640 1 -- 172.21.15.42:0/3029903204 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f2d541217b0 con 0x7f2d5402c650 2024-09-16T03:54:42.360 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.335+0000 7f2d537fe640 1 --2- 172.21.15.42:0/3029903204 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d5402c650 0x7f2d5402ca60 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f2d540a0070 tx=0x7f2d44000ee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:54:42.360 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.335+0000 7f2d4bfff640 10 client.854120.objecter ms_handle_connect 0x7f2d5402c650 2024-09-16T03:54:42.360 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.335+0000 7f2d4bfff640 10 client.854120.objecter resend_mon_ops 2024-09-16T03:54:42.360 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.335+0000 7f2d4bfff640 1 -- 172.21.15.42:0/3029903204 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d440073d0 con 0x7f2d5402c650 2024-09-16T03:54:42.360 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.335+0000 7f2d4bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:54:42.360 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.335+0000 7f2d4bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:54:42.360 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.335+0000 7f2d4bfff640 10 monclient: dump: 2024-09-16T03:54:42.360 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:54:42.361 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:54:42.361 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:42.361 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:42.361 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:54:42.361 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:54:42.361 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:54:42.361 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:54:42.361 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:54:42.361 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:54:42.361 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.335+0000 7f2d4bfff640 1 -- 172.21.15.42:0/3029903204 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4545 (secure 0 0 0) 0x7f2d440073d0 con 0x7f2d5402c650 2024-09-16T03:54:42.361 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.335+0000 7f2d4bfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:54:42.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.335+0000 7f2d4bfff640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:54:42.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.335+0000 7f2d4bfff640 1 -- 172.21.15.42:0/3029903204 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d5402c650 msgr2=0x7f2d5402ca60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:54:42.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.335+0000 7f2d4bfff640 1 --2- 172.21.15.42:0/3029903204 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d5402c650 0x7f2d5402ca60 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f2d540a0070 tx=0x7f2d44000ee0 comp rx=0 tx=0).stop 2024-09-16T03:54:42.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.335+0000 7f2d4bfff640 1 -- 172.21.15.42:0/3029903204 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d5402c650 msgr2=0x7f2d5402ca60 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:54:42.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.335+0000 7f2d4bfff640 1 --2- 172.21.15.42:0/3029903204 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d5402c650 0x7f2d5402ca60 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:54:42.362 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:54:42.363 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:54:42.363 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:54:42.363 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T03:54:42.363 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 110, 2024-09-16T03:54:42.363 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 66535, 2024-09-16T03:54:42.363 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:54:42.363 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:54:42.363 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T03:54:42.363 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T03:54:42.363 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:54:42.364 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 0, 2024-09-16T03:54:42.364 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:54:42.364 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:54:42.364 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:54:42.364 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:54:42.364 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:54:42.364 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:54:42.364 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:54:42.364 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:54:42.364 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:54:42.364 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:54:42.365 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:54:42.365 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:54:42.365 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:54:42.365 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:54:42.365 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:54:42.365 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:54:42.365 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:54:42.365 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:54:42.365 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:54:42.365 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:54:42.365 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:54:42.366 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:54:42.366 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:54:42.366 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:54:42.366 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:54:42.366 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:54:42.366 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:54:42.366 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:54:42.366 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:54:42.366 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:54:42.366 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:54:42.367 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:54:42.367 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:54:42.367 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:54:42.367 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:54:42.367 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:54:42.367 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:54:42.367 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:54:42.367 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:54:42.367 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:54:42.367 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:54:42.367 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:54:42.367 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:54:42.368 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:54:42.368 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:54:42.368 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:54:42.368 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:54:42.368 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:54:42.368 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:54:42.368 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:54:42.368 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:54:42.368 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:54:42.368 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:54:42.368 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:54:42.369 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:54:42.369 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:54:42.369 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:54:42.369 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:54:42.369 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:54:42.369 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:54:42.369 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:54:42.369 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:54:42.369 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:54:42.369 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:54:42.369 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:54:42.370 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:54:42.370 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:54:42.370 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:54:42.370 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:54:42.370 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:54:42.370 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:54:42.370 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:54:42.370 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:54:42.370 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:54:42.370 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:54:42.370 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:54:42.370 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:54:42.371 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:54:42.371 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:54:42.371 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:54:42.371 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:54:42.371 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:54:42.371 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:54:42.371 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:54:42.371 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:54:42.371 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:54:42.371 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:54:42.371 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:54:42.371 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:54:42.372 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:54:42.372 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:54:42.372 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:54:42.372 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:54:42.372 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:54:42.372 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:54:42.372 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:54:42.372 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:54:42.372 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:54:42.373 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:54:42.373 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:54:42.373 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:54:42.373 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:54:42.373 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:54:42.373 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:54:42.373 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:54:42.373 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:54:42.373 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:54:42.373 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:54:42.374 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:54:42.374 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:54:42.374 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:54:42.374 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:54:42.374 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:54:42.374 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:54:42.374 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:54:42.374 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:54:42.374 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:54:42.374 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:54:42.374 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:54:42.375 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:54:42.375 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:54:42.375 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:54:42.375 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:54:42.375 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:54:42.375 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:54:42.375 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:54:42.375 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:54:42.375 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:54:42.375 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:54:42.375 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:54:42.375 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:54:42.375 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:54:42.376 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:54:42.376 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:54:42.376 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:54:42.376 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:54:42.376 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:54:42.376 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:54:42.376 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:54:42.376 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:54:42.376 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:54:42.377 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:54:42.377 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:54:42.377 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:54:42.377 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:54:42.377 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 2 2024-09-16T03:54:42.377 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:54:42.377 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:54:42.377 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:54:42.377 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:54:42.377 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:54:42.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.335+0000 7f2d5a1e3640 10 librados: watch_flush enter 2024-09-16T03:54:42.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.335+0000 7f2d5a1e3640 10 librados: watch_flush exit 2024-09-16T03:54:42.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.335+0000 7f2d5a1e3640 20 client.854120.objecter shutdown clearing up homeless session... 2024-09-16T03:54:42.380 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.335+0000 7f2d5a1e3640 10 client.854120.objecter successfully canceled tick 2024-09-16T03:54:42.380 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.335+0000 7f2d5a1e3640 1 -- 172.21.15.42:0/3029903204 >> v2:172.21.15.154:6812/254845342 conn(0x7f2d34037810 msgr2=0x7f2d34039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:54:42.380 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.335+0000 7f2d5a1e3640 1 --2- 172.21.15.42:0/3029903204 >> v2:172.21.15.154:6812/254845342 conn(0x7f2d34037810 0x7f2d34039cd0 secure :-1 s=READY pgs=3616 cs=0 l=1 rev1=1 crypto rx=0x7f2d54128e50 tx=0x7f2d4c009040 comp rx=0 tx=0).stop 2024-09-16T03:54:42.380 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.335+0000 7f2d5a1e3640 10 monclient: shutdown 2024-09-16T03:54:42.380 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.335+0000 7f2d5a1e3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:54:42.380 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.335+0000 7f2d5a1e3640 1 -- 172.21.15.42:0/3029903204 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2d540a3450 msgr2=0x7f2d5413f010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:54:42.380 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.335+0000 7f2d5a1e3640 1 --2- 172.21.15.42:0/3029903204 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2d540a3450 0x7f2d5413f010 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f2d3c018f70 tx=0x7f2d3c019100 comp rx=0 tx=0).stop 2024-09-16T03:54:42.380 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.335+0000 7f2d53fff640 1 -- 172.21.15.42:0/3029903204 reap_dead start 2024-09-16T03:54:42.380 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.335+0000 7f2d5a1e3640 1 -- 172.21.15.42:0/3029903204 shutdown_connections 2024-09-16T03:54:42.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.335+0000 7f2d5a1e3640 1 --2- 172.21.15.42:0/3029903204 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2d540a3450 0x7f2d5413f010 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:54:42.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.335+0000 7f2d5a1e3640 1 -- 172.21.15.42:0/3029903204 >> 172.21.15.42:0/3029903204 conn(0x7f2d540756e0 msgr2=0x7f2d54071540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:54:42.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.335+0000 7f2d5a1e3640 1 -- 172.21.15.42:0/3029903204 shutdown_connections 2024-09-16T03:54:42.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.335+0000 7f2d5a1e3640 1 -- 172.21.15.42:0/3029903204 wait complete. 2024-09-16T03:54:42.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.335+0000 7f2d5a1e3640 1 librados: shutdown 2024-09-16T03:54:42.381 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:54:42.381 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T03:54:42.382 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T03:54:42.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.499+0000 7fcefde70640 10 monclient: get_monmap_and_config 2024-09-16T03:54:42.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.499+0000 7fcefde70640 10 monclient: build_initial_monmap 2024-09-16T03:54:42.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.499+0000 7fcefde70640 10 monclient: monmap: 2024-09-16T03:54:42.504 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:54:42.504 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:54:42.504 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:54:42.502963+0000 2024-09-16T03:54:42.505 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:54:42.502963+0000 2024-09-16T03:54:42.505 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:54:42.505 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:54:42.505 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:54:42.505 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:54:42.505 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:54:42.505 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:54:42.505 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:54:42.505 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:54:42.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.499+0000 7fcefde70640 1 Processor -- start 2024-09-16T03:54:42.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.499+0000 7fcefde70640 1 -- start start 2024-09-16T03:54:42.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.499+0000 7fcefde70640 10 monclient: init 2024-09-16T03:54:42.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.503+0000 7fcefde70640 10 monclient: _reopen_session rank -1 2024-09-16T03:54:42.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.503+0000 7fcefde70640 10 monclient: _add_conns ranks=[0,1,4,3,2] 2024-09-16T03:54:42.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.503+0000 7fcefde70640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fcef80a1330 addr v1:172.21.15.42:6789/0 2024-09-16T03:54:42.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.503+0000 7fcefde70640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fcef806dac0 addr v1:172.21.15.154:6789/0 2024-09-16T03:54:42.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.503+0000 7fcefde70640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fcef80a1e20 0x7fcef813da70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:54:42.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.503+0000 7fcefde70640 10 monclient(hunting): picked mon.noname-c con 0x7fcef80a1e20 addr v2:172.21.15.154:3300/0 2024-09-16T03:54:42.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.503+0000 7fcefde70640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcef809c090 con 0x7fcef80a1330 2024-09-16T03:54:42.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.503+0000 7fcefde70640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcef809f610 con 0x7fcef806dac0 2024-09-16T03:54:42.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.503+0000 7fcefde70640 10 monclient(hunting): start opening mon connection 2024-09-16T03:54:42.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.503+0000 7fcefde70640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fcef809f8f0 con 0x7fcef80a1e20 2024-09-16T03:54:42.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.503+0000 7fcefde70640 10 monclient(hunting): _renew_subs 2024-09-16T03:54:42.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.503+0000 7fcefde70640 10 monclient(hunting): authenticate will time out at 1729.472678s 2024-09-16T03:54:42.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.503+0000 7fcef77fe640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fcef80a1330 0x7fcef80a1710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:57216/0 (socket says 172.21.15.42:57216) 2024-09-16T03:54:42.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.503+0000 7fcef77fe640 1 -- 172.21.15.42:0/2212700763 learned_addr learned my addr 172.21.15.42:0/2212700763 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:54:42.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.503+0000 7fcef7fff640 1 --2- 172.21.15.42:0/2212700763 >> v2:172.21.15.154:3300/0 conn(0x7fcef80a1e20 0x7fcef813da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:54:42.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.503+0000 7fcef7fff640 10 monclient(hunting): get_auth_request con 0x7fcef80a1e20 auth_method 0 2024-09-16T03:54:42.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.503+0000 7fcef7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:54:42.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.503+0000 7fcef7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:54:42.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.503+0000 7fcef7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:54:42.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.503+0000 7fcef67fc640 1 -- 172.21.15.42:0/2212700763 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2983029277 0 0) 0x7fcef809c090 con 0x7fcef80a1330 2024-09-16T03:54:42.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.503+0000 7fcef67fc640 10 monclient(hunting): _negotiate 2024-09-16T03:54:42.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.503+0000 7fcef67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:54:42.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.503+0000 7fcef67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:54:42.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.503+0000 7fcef67fc640 10 monclient(hunting): my global_id is 824454 2024-09-16T03:54:42.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.503+0000 7fcef67fc640 1 -- 172.21.15.42:0/2212700763 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcee4003530 con 0x7fcef80a1330 2024-09-16T03:54:42.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.503+0000 7fcef7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:54:42.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.503+0000 7fcef7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:54:42.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.503+0000 7fcef7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:54:42.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.503+0000 7fcef67fc640 1 -- 172.21.15.42:0/2212700763 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3295233405 0 0) 0x7fcee4003530 con 0x7fcef80a1330 2024-09-16T03:54:42.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.503+0000 7fcef67fc640 1 -- 172.21.15.42:0/2212700763 >> v2:172.21.15.154:3300/0 conn(0x7fcef80a1e20 msgr2=0x7fcef813da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:54:42.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.503+0000 7fcef67fc640 1 --2- 172.21.15.42:0/2212700763 >> v2:172.21.15.154:3300/0 conn(0x7fcef80a1e20 0x7fcef813da70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:54:42.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.503+0000 7fcef67fc640 1 -- 172.21.15.42:0/2212700763 >> v1:172.21.15.154:6789/0 conn(0x7fcef806dac0 legacy=0x7fcef813a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:54:42.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.503+0000 7fcef67fc640 10 monclient: _finish_hunting 0 2024-09-16T03:54:42.513 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.503+0000 7fcef67fc640 1 monclient: found mon.noname-a-legacy 2024-09-16T03:54:42.513 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.503+0000 7fcef67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:54:42.513 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.503+0000 7fcef67fc640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.42:6789/0 2024-09-16T03:54:42.513 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.503+0000 7fcef67fc640 1 -- 172.21.15.42:0/2212700763 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcef809fbd0 con 0x7fcef80a1330 2024-09-16T03:54:42.513 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.503+0000 7fcef67fc640 10 monclient: _finish_auth 0 2024-09-16T03:54:42.513 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.503+0000 7fcef67fc640 10 monclient: _check_auth_tickets 2024-09-16T03:54:42.513 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.503+0000 7fcef67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:54:42.513 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.503+0000 7fcef67fc640 1 -- 172.21.15.42:0/2212700763 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2177413734 0 0) 0x7fcee0001680 con 0x7fcef80a1330 2024-09-16T03:54:42.513 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.503+0000 7fcef67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:54:42.513 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.503+0000 7fcef67fc640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T03:54:42.513 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.503+0000 7fcef67fc640 10 monclient: dump: 2024-09-16T03:54:42.513 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:54:42.513 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:54:42.513 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:42.514 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:42.514 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:54:42.514 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:54:42.514 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:54:42.514 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:54:42.514 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:54:42.514 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:54:42.514 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.503+0000 7fcef67fc640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] but i'm connected to v1:172.21.15.42:6789/0, reconnecting 2024-09-16T03:54:42.514 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.503+0000 7fcef67fc640 10 monclient: _reopen_session rank -1 2024-09-16T03:54:42.514 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.503+0000 7fcef67fc640 1 -- 172.21.15.42:0/2212700763 >> v1:172.21.15.42:6789/0 conn(0x7fcef80a1330 legacy=0x7fcef80a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:54:42.514 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.503+0000 7fcef67fc640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:54:42.514 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.503+0000 7fcef67fc640 1 --2- 172.21.15.42:0/2212700763 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcee40039c0 0x7fcee4003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:54:42.514 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.503+0000 7fcef67fc640 10 monclient(hunting): picked mon.a con 0x7fcee40039c0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:54:42.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.503+0000 7fcef67fc640 1 --2- 172.21.15.42:0/2212700763 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fcee4004810 0x7fcee4004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:54:42.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.503+0000 7fcef67fc640 10 monclient(hunting): picked mon.c con 0x7fcee4004810 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:54:42.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.503+0000 7fcef67fc640 1 --2- 172.21.15.42:0/2212700763 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcee40051d0 0x7fcee400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:54:42.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.503+0000 7fcef67fc640 10 monclient(hunting): picked mon.b con 0x7fcee40051d0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:54:42.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.503+0000 7fcef67fc640 10 monclient(hunting): start opening mon connection 2024-09-16T03:54:42.516 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.503+0000 7fcef67fc640 1 -- 172.21.15.42:0/2212700763 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fcef809f8f0 con 0x7fcee40039c0 2024-09-16T03:54:42.516 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.503+0000 7fcef67fc640 10 monclient(hunting): start opening mon connection 2024-09-16T03:54:42.516 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.503+0000 7fcef67fc640 1 -- 172.21.15.42:0/2212700763 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fcee400c170 con 0x7fcee40051d0 2024-09-16T03:54:42.516 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.503+0000 7fcef67fc640 10 monclient(hunting): start opening mon connection 2024-09-16T03:54:42.516 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.503+0000 7fcef67fc640 1 -- 172.21.15.42:0/2212700763 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fcee400c370 con 0x7fcee4004810 2024-09-16T03:54:42.516 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.503+0000 7fcef67fc640 10 monclient(hunting): _renew_subs 2024-09-16T03:54:42.516 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.503+0000 7fcef67fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:54:42.516 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.503+0000 7fcef77fe640 1 --2- 172.21.15.42:0/2212700763 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcee40039c0 0x7fcee4003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:54:42.516 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.503+0000 7fcef77fe640 10 monclient(hunting): get_auth_request con 0x7fcee40039c0 auth_method 0 2024-09-16T03:54:42.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.503+0000 7fcef77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:54:42.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.503+0000 7fcef77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:54:42.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.503+0000 7fcef77fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:54:42.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.503+0000 7fcef6ffd640 1 --2- 172.21.15.42:0/2212700763 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fcee4004810 0x7fcee4004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:54:42.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.503+0000 7fcef6ffd640 10 monclient(hunting): get_auth_request con 0x7fcee4004810 auth_method 0 2024-09-16T03:54:42.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.503+0000 7fcef6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:54:42.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.503+0000 7fcef6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:54:42.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.503+0000 7fcef6ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:54:42.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.503+0000 7fcef77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:54:42.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.503+0000 7fcef77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:54:42.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.503+0000 7fcef77fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T03:54:42.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.503+0000 7fcef6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:54:42.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.503+0000 7fcef6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:54:42.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.503+0000 7fcef7fff640 1 --2- 172.21.15.42:0/2212700763 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcee40051d0 0x7fcee400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:54:42.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.503+0000 7fcef6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T03:54:42.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.503+0000 7fcef7fff640 10 monclient(hunting): get_auth_request con 0x7fcee40051d0 auth_method 0 2024-09-16T03:54:42.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.503+0000 7fcef7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:54:42.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.503+0000 7fcef7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:54:42.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.503+0000 7fcef7fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:54:42.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.503+0000 7fcef77fe640 10 monclient(hunting): handle_auth_done global_id 824454 payload 293 2024-09-16T03:54:42.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.503+0000 7fcef77fe640 1 -- 172.21.15.42:0/2212700763 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fcee4004810 msgr2=0x7fcee4004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:54:42.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.503+0000 7fcef77fe640 1 --2- 172.21.15.42:0/2212700763 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fcee4004810 0x7fcee4004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:54:42.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.503+0000 7fcef77fe640 1 -- 172.21.15.42:0/2212700763 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcee40051d0 msgr2=0x7fcee400b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:54:42.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.503+0000 7fcef77fe640 1 --2- 172.21.15.42:0/2212700763 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcee40051d0 0x7fcee400b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:54:42.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.503+0000 7fcef77fe640 10 monclient: _finish_hunting 0 2024-09-16T03:54:42.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.503+0000 7fcef77fe640 1 monclient: found mon.a 2024-09-16T03:54:42.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.503+0000 7fcef77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:54:42.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.503+0000 7fcef77fe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:54:42.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.503+0000 7fcef77fe640 1 -- 172.21.15.42:0/2212700763 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcee4003530 con 0x7fcee40039c0 2024-09-16T03:54:42.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.503+0000 7fcef77fe640 10 monclient: _finish_auth 0 2024-09-16T03:54:42.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.503+0000 7fcef77fe640 10 monclient: _check_auth_tickets 2024-09-16T03:54:42.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.503+0000 7fcef77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:54:42.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.503+0000 7fcef7fff640 1 -- 172.21.15.42:0/2212700763 reap_dead start 2024-09-16T03:54:42.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.503+0000 7fcef6ffd640 1 --2- 172.21.15.42:0/2212700763 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fcee4004810 0x7fcee4004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:54:42.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.503+0000 7fcefde70640 5 monclient: authenticate success, global_id 824454 2024-09-16T03:54:42.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.503+0000 7fcefde70640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:54:42.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.507+0000 7fcef77fe640 1 --2- 172.21.15.42:0/2212700763 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcee40039c0 0x7fcee4003de0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fcee000c240 tx=0x7fcee001cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f42bf124111ced00 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:54:42.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.507+0000 7fcef67fc640 1 -- 172.21.15.42:0/2212700763 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcee0007d20 con 0x7fcee40039c0 2024-09-16T03:54:42.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.507+0000 7fcef67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:54:42.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.507+0000 7fcef67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:54:42.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.507+0000 7fcef67fc640 10 monclient: dump: 2024-09-16T03:54:42.524 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:54:42.524 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:54:42.524 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:42.524 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:42.525 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:54:42.525 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:54:42.525 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:54:42.525 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:54:42.525 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:54:42.525 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:54:42.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.507+0000 7fcef67fc640 1 -- 172.21.15.42:0/2212700763 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcee00a9070 con 0x7fcee40039c0 2024-09-16T03:54:42.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.507+0000 7fcefde70640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:54:42.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.507+0000 7fcef67fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:54:42.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.507+0000 7fcefde70640 10 monclient: get_monmap_and_config success 2024-09-16T03:54:42.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.507+0000 7fcefde70640 4 set_mon_vals no callback set 2024-09-16T03:54:42.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.507+0000 7fcef67fc640 1 -- 172.21.15.42:0/2212700763 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcee0005da0 con 0x7fcee40039c0 2024-09-16T03:54:42.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.507+0000 7fcef67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:54:42.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.507+0000 7fcef67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:54:42.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.507+0000 7fcef67fc640 10 monclient: dump: 2024-09-16T03:54:42.526 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:54:42.526 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:54:42.526 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:42.526 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:42.526 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:54:42.526 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:54:42.526 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:54:42.526 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:54:42.526 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:54:42.526 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:54:42.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.507+0000 7fcefde70640 10 monclient: shutdown 2024-09-16T03:54:42.527 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.507+0000 7fcefde70640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:54:42.527 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.507+0000 7fcefde70640 1 -- 172.21.15.42:0/2212700763 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcee40039c0 msgr2=0x7fcee4003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:54:42.527 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.507+0000 7fcefde70640 1 --2- 172.21.15.42:0/2212700763 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcee40039c0 0x7fcee4003de0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fcee000c240 tx=0x7fcee001cc20 comp rx=0 tx=0).stop 2024-09-16T03:54:42.527 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.507+0000 7fcefde70640 1 -- 172.21.15.42:0/2212700763 shutdown_connections 2024-09-16T03:54:42.527 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.507+0000 7fcefde70640 1 --2- 172.21.15.42:0/2212700763 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcee40039c0 0x7fcee4003de0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:54:42.527 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.507+0000 7fcefde70640 1 -- 172.21.15.42:0/2212700763 >> 172.21.15.42:0/2212700763 conn(0x7fcef80a2420 msgr2=0x7fcef80a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:54:42.529 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.507+0000 7fcefde70640 1 -- 172.21.15.42:0/2212700763 shutdown_connections 2024-09-16T03:54:42.529 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.507+0000 7fcefde70640 1 -- 172.21.15.42:0/2212700763 wait complete. 2024-09-16T03:54:42.530 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.507+0000 7fcefde70640 10 monclient: build_initial_monmap 2024-09-16T03:54:42.530 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.507+0000 7fcefde70640 10 monclient: monmap: 2024-09-16T03:54:42.530 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:54:42.530 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:54:42.530 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:54:42.531 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:54:42.531 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:54:42.531 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:54:42.531 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:54:42.531 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:54:42.531 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:54:42.531 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:54:42.531 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.507+0000 7fcefde70640 1 librados: starting msgr at 2024-09-16T03:54:42.531 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.507+0000 7fcefde70640 1 librados: starting objecter 2024-09-16T03:54:42.531 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.507+0000 7fcefde70640 1 Processor -- start 2024-09-16T03:54:42.531 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.507+0000 7fcefde70640 1 -- start start 2024-09-16T03:54:42.532 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.507+0000 7fcefde70640 1 librados: setting wanted keys 2024-09-16T03:54:42.532 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.507+0000 7fcefde70640 1 librados: calling monclient init 2024-09-16T03:54:42.532 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.507+0000 7fcefde70640 10 monclient: init 2024-09-16T03:54:42.532 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.511+0000 7fcefde70640 10 monclient: _reopen_session rank -1 2024-09-16T03:54:42.532 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.511+0000 7fcefde70640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:54:42.532 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.511+0000 7fcefde70640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fcee40039c0 0x7fcef81cd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:54:42.532 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.511+0000 7fcefde70640 10 monclient(hunting): picked mon.noname-c con 0x7fcee40039c0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:54:42.532 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.511+0000 7fcefde70640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcef81cd8e0 0x7fcef8122950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:54:42.532 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.511+0000 7fcefde70640 10 monclient(hunting): picked mon.noname-b con 0x7fcef81cd8e0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:54:42.532 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.511+0000 7fcefde70640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcef81cdcc0 0x7fcef8122e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:54:42.532 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.511+0000 7fcefde70640 10 monclient(hunting): picked mon.noname-a con 0x7fcef81cdcc0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:54:42.532 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.511+0000 7fcefde70640 10 monclient(hunting): start opening mon connection 2024-09-16T03:54:42.532 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.511+0000 7fcefde70640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fcef8135910 con 0x7fcef81cdcc0 2024-09-16T03:54:42.533 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.511+0000 7fcefde70640 10 monclient(hunting): start opening mon connection 2024-09-16T03:54:42.533 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.511+0000 7fcefde70640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fcef809ef80 con 0x7fcef81cd8e0 2024-09-16T03:54:42.533 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.511+0000 7fcefde70640 10 monclient(hunting): start opening mon connection 2024-09-16T03:54:42.533 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.511+0000 7fcefde70640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fcef8135a90 con 0x7fcee40039c0 2024-09-16T03:54:42.533 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.511+0000 7fcefde70640 10 monclient(hunting): _renew_subs 2024-09-16T03:54:42.533 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.511+0000 7fcefde70640 10 monclient(hunting): authenticate will time out at 1729.480721s 2024-09-16T03:54:42.533 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.511+0000 7fcef7fff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcef81cdcc0 0x7fcef8122e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:54:42.533 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.511+0000 7fcef77fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fcee40039c0 0x7fcef81cd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:54:42.533 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.511+0000 7fcef7fff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcef81cdcc0 0x7fcef8122e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:33704/0 (socket says 172.21.15.42:33704) 2024-09-16T03:54:42.533 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.511+0000 7fcef7fff640 1 -- 172.21.15.42:0/3465460995 learned_addr learned my addr 172.21.15.42:0/3465460995 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:54:42.533 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.511+0000 7fcef7fff640 10 monclient(hunting): get_auth_request con 0x7fcef81cdcc0 auth_method 0 2024-09-16T03:54:42.533 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.511+0000 7fcef7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:54:42.533 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.511+0000 7fcef7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:54:42.534 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.511+0000 7fcef7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:54:42.534 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.511+0000 7fcef77fe640 10 monclient(hunting): get_auth_request con 0x7fcee40039c0 auth_method 0 2024-09-16T03:54:42.534 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.511+0000 7fcef77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:54:42.534 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.511+0000 7fcef77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:54:42.534 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.511+0000 7fcef77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:54:42.534 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.511+0000 7fcef7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:54:42.534 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.511+0000 7fcef7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:54:42.534 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.511+0000 7fcef7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:54:42.534 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.511+0000 7fcef77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:54:42.534 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.511+0000 7fcef77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:54:42.534 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.511+0000 7fcef77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:54:42.534 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.511+0000 7fcef6ffd640 1 --2- 172.21.15.42:0/3465460995 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcef81cd8e0 0x7fcef8122950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:54:42.534 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.511+0000 7fcef6ffd640 10 monclient(hunting): get_auth_request con 0x7fcef81cd8e0 auth_method 0 2024-09-16T03:54:42.534 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.511+0000 7fcef6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:54:42.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.511+0000 7fcef6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:54:42.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.511+0000 7fcef6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:54:42.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.511+0000 7fcef7fff640 10 monclient(hunting): handle_auth_done global_id 824457 payload 824 2024-09-16T03:54:42.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.511+0000 7fcef7fff640 1 -- 172.21.15.42:0/3465460995 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fcee40039c0 msgr2=0x7fcef81cd3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:54:42.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.511+0000 7fcef7fff640 1 --2- 172.21.15.42:0/3465460995 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fcee40039c0 0x7fcef81cd3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:54:42.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.511+0000 7fcef7fff640 1 -- 172.21.15.42:0/3465460995 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcef81cd8e0 msgr2=0x7fcef8122950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:54:42.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.511+0000 7fcef7fff640 1 --2- 172.21.15.42:0/3465460995 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcef81cd8e0 0x7fcef8122950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:54:42.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.511+0000 7fcef7fff640 10 monclient: _finish_hunting 0 2024-09-16T03:54:42.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.511+0000 7fcef7fff640 1 monclient: found mon.noname-a 2024-09-16T03:54:42.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.511+0000 7fcef7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:54:42.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.511+0000 7fcef7fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:54:42.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.511+0000 7fcef7fff640 1 -- 172.21.15.42:0/3465460995 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcef81268e0 con 0x7fcef81cdcc0 2024-09-16T03:54:42.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.511+0000 7fcef7fff640 1 --2- 172.21.15.42:0/3465460995 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcef81cdcc0 0x7fcef8122e90 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fcef80a22f0 tx=0x7fcef00115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:54:42.536 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.511+0000 7fceeffff640 10 client.?.objecter ms_handle_connect 0x7fcef81cdcc0 2024-09-16T03:54:42.536 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.511+0000 7fceeffff640 10 client.?.objecter resend_mon_ops 2024-09-16T03:54:42.536 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.511+0000 7fceeffff640 1 -- 172.21.15.42:0/3465460995 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcef0011db0 con 0x7fcef81cdcc0 2024-09-16T03:54:42.536 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.511+0000 7fceeffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:54:42.536 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.511+0000 7fceeffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:54:42.536 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.511+0000 7fceeffff640 10 monclient: dump: 2024-09-16T03:54:42.536 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:54:42.536 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:54:42.536 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:42.536 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:42.536 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:54:42.536 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:54:42.537 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:54:42.537 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:54:42.537 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:54:42.537 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:54:42.537 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.511+0000 7fceeffff640 10 monclient: _finish_auth 0 2024-09-16T03:54:42.537 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.511+0000 7fceeffff640 10 monclient: _check_auth_tickets 2024-09-16T03:54:42.537 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.511+0000 7fceeffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:54:42.537 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.511+0000 7fcefde70640 5 monclient: authenticate success, global_id 824457 2024-09-16T03:54:42.537 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.511+0000 7fceeffff640 1 -- 172.21.15.42:0/3465460995 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcef001b070 con 0x7fcef81cdcc0 2024-09-16T03:54:42.537 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.511+0000 7fcefde70640 10 monclient: _renew_subs 2024-09-16T03:54:42.537 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.511+0000 7fcefde70640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:54:42.537 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.511+0000 7fcefde70640 1 -- 172.21.15.42:0/3465460995 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcef8071260 con 0x7fcef81cdcc0 2024-09-16T03:54:42.537 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.511+0000 7fceeffff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:54:42.538 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.511+0000 7fceeffff640 1 -- 172.21.15.42:0/3465460995 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcef00039a0 con 0x7fcef81cdcc0 2024-09-16T03:54:42.538 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.511+0000 7fceeffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:54:42.538 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.511+0000 7fceeffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:54:42.538 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.511+0000 7fcef5ffb640 4 set_mon_vals no callback set 2024-09-16T03:54:42.538 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.511+0000 7fcefde70640 10 client.824457.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:54:42.538 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.511+0000 7fceeffff640 10 monclient: dump: 2024-09-16T03:54:42.538 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:54:42.538 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:54:42.538 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:42.538 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:42.538 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:54:42.538 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:54:42.538 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:54:42.539 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:54:42.539 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:54:42.539 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:54:42.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.511+0000 7fcefde70640 10 monclient: _renew_subs 2024-09-16T03:54:42.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.511+0000 7fcefde70640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:54:42.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.511+0000 7fcefde70640 1 -- 172.21.15.42:0/3465460995 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcef8126600 con 0x7fcef81cdcc0 2024-09-16T03:54:42.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.511+0000 7fcefde70640 1 librados: init done 2024-09-16T03:54:42.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.511+0000 7fceeffff640 1 -- 172.21.15.42:0/3465460995 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fcef0003b40 con 0x7fcef81cdcc0 2024-09-16T03:54:42.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.511+0000 7fcefde70640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:54:42.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.511+0000 7fcefde70640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:54:42.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.511+0000 7fcefde70640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:54:42.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.511+0000 7fcefde70640 1 -- 172.21.15.42:0/3465460995 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcef8135e20 con 0x7fcef81cdcc0 2024-09-16T03:54:42.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.511+0000 7fceeffff640 1 --2- 172.21.15.42:0/3465460995 >> v2:172.21.15.154:6812/254845342 conn(0x7fced8037810 0x7fced8039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:54:42.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.511+0000 7fceeffff640 1 -- 172.21.15.42:0/3465460995 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(386..386 src has 286..386) ==== 5713+0+0 (secure 0 0 0) 0x7fcef00496a0 con 0x7fcef81cdcc0 2024-09-16T03:54:42.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.511+0000 7fceeffff640 10 client.824457.objecter ms_dispatch 0x7fcef8000ba0 osd_map(386..386 src has 286..386) 2024-09-16T03:54:42.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.511+0000 7fceeffff640 3 client.824457.objecter handle_osd_map got epochs [386,386] > 0 2024-09-16T03:54:42.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.511+0000 7fceeffff640 3 client.824457.objecter handle_osd_map decoding full epoch 386 2024-09-16T03:54:42.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.511+0000 7fceeffff640 20 client.824457.objecter dump_active .. 0 homeless 2024-09-16T03:54:42.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.511+0000 7fcef77fe640 1 --2- 172.21.15.42:0/3465460995 >> v2:172.21.15.154:6812/254845342 conn(0x7fced8037810 0x7fced8039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:54:42.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.511+0000 7fcef77fe640 10 monclient: get_auth_request con 0x7fced8037810 auth_method 0 2024-09-16T03:54:42.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.515+0000 7fcef77fe640 1 --2- 172.21.15.42:0/3465460995 >> v2:172.21.15.154:6812/254845342 conn(0x7fced8037810 0x7fced8039cd0 secure :-1 s=READY pgs=3617 cs=0 l=1 rev1=1 crypto rx=0x7fcee00ab000 tx=0x7fcee00a7070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:54:42.541 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.515+0000 7fceeffff640 10 client.824457.objecter ms_handle_connect 0x7fced8037810 2024-09-16T03:54:42.541 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.515+0000 7fceeffff640 1 -- 172.21.15.42:0/3465460995 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7fcef0004030 con 0x7fcef81cdcc0 2024-09-16T03:54:42.541 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.515+0000 7fceeffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:54:42.541 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.515+0000 7fceeffff640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:54:42.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.743+0000 7fcefde70640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T03:54:42.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.743+0000 7fcefde70640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T03:54:42.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.743+0000 7fcefde70640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:54:42.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.743+0000 7fcefde70640 1 -- 172.21.15.42:0/3465460995 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7fcef8124a40 con 0x7fcef81cdcc0 2024-09-16T03:54:42.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.747+0000 7fceeffff640 1 -- 172.21.15.42:0/3465460995 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7fcef0004790 con 0x7fcef81cdcc0 2024-09-16T03:54:42.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.747+0000 7fceeffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T03:54:42.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.747+0000 7fceeffff640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:54:42.751 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.747+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:54:42.752 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.747+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=28,config=2,health=11,kv=9,logm=50} crc {auth=2088017422,config=692619927,health=3043910703,kv=950665868,logm=3135884550}) 2024-09-16T03:54:42.752 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.747+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=27,config=2,health=11,kv=10,logm=50} crc {auth=3277017804,config=2435484489,health=1811047944,kv=3446188498,logm=3115639175}) 2024-09-16T03:54:42.752 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.747+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:54:42.752 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.747+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=28,config=2,health=11,kv=9,logm=50} crc {auth=2088017422,config=692619927,health=3043910703,kv=950665868,logm=3135884550}) 2024-09-16T03:54:42.752 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.747+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=27,config=2,health=12,kv=10,logm=49} crc {auth=4058761148,config=692619927,health=2588199157,kv=3446188498,logm=3787155348}) 2024-09-16T03:54:42.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.747+0000 7fcefde70640 10 librados: watch_flush enter 2024-09-16T03:54:42.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.747+0000 7fcefde70640 10 librados: watch_flush exit 2024-09-16T03:54:42.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.747+0000 7fcefde70640 20 client.824457.objecter shutdown clearing up homeless session... 2024-09-16T03:54:42.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.747+0000 7fcefde70640 10 client.824457.objecter successfully canceled tick 2024-09-16T03:54:42.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.747+0000 7fcefde70640 1 -- 172.21.15.42:0/3465460995 >> v2:172.21.15.154:6812/254845342 conn(0x7fced8037810 msgr2=0x7fced8039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:54:42.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.747+0000 7fcefde70640 1 --2- 172.21.15.42:0/3465460995 >> v2:172.21.15.154:6812/254845342 conn(0x7fced8037810 0x7fced8039cd0 secure :-1 s=READY pgs=3617 cs=0 l=1 rev1=1 crypto rx=0x7fcee00ab000 tx=0x7fcee00a7070 comp rx=0 tx=0).stop 2024-09-16T03:54:42.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.747+0000 7fcefde70640 10 monclient: shutdown 2024-09-16T03:54:42.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.747+0000 7fcefde70640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:54:42.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.747+0000 7fcefde70640 1 -- 172.21.15.42:0/3465460995 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcef81cdcc0 msgr2=0x7fcef8122e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:54:42.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.747+0000 7fcefde70640 1 --2- 172.21.15.42:0/3465460995 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcef81cdcc0 0x7fcef8122e90 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fcef80a22f0 tx=0x7fcef00115a0 comp rx=0 tx=0).stop 2024-09-16T03:54:42.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.747+0000 7fcefde70640 1 -- 172.21.15.42:0/3465460995 shutdown_connections 2024-09-16T03:54:42.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.747+0000 7fcefde70640 1 --2- 172.21.15.42:0/3465460995 >> v2:172.21.15.154:6812/254845342 conn(0x7fced8037810 0x7fced8039cd0 unknown :-1 s=CLOSED pgs=3617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:54:42.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.747+0000 7fcefde70640 1 --2- 172.21.15.42:0/3465460995 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcef81cdcc0 0x7fcef8122e90 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:54:42.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.747+0000 7fcefde70640 1 --2- 172.21.15.42:0/3465460995 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcef81cd8e0 0x7fcef8122950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:54:42.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.747+0000 7fcefde70640 1 --2- 172.21.15.42:0/3465460995 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fcee40039c0 0x7fcef81cd3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:54:42.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.747+0000 7fcefde70640 1 -- 172.21.15.42:0/3465460995 >> 172.21.15.42:0/3465460995 conn(0x7fcef80a2420 msgr2=0x7fcef806ebf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:54:42.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.747+0000 7fcefde70640 1 -- 172.21.15.42:0/3465460995 shutdown_connections 2024-09-16T03:54:42.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.747+0000 7fcefde70640 1 -- 172.21.15.42:0/3465460995 wait complete. 2024-09-16T03:54:42.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:42.751+0000 7fcefde70640 1 librados: shutdown 2024-09-16T03:54:42.756 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.751+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:54:42.756 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.751+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=249771054}) 2024-09-16T03:54:42.756 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.751+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1339266565}) 2024-09-16T03:54:42.756 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.751+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:54:42.756 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.751+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=249771054}) 2024-09-16T03:54:42.756 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.751+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4132452678}) 2024-09-16T03:54:42.757 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.751+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:54:42.757 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.751+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1461443900}) 2024-09-16T03:54:42.757 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.751+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3004941980}) 2024-09-16T03:54:42.757 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.751+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:54:42.758 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.751+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1461443900}) 2024-09-16T03:54:42.758 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.751+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=611323926}) 2024-09-16T03:54:42.759 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.755+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:54:42.759 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.755+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3690842013}) 2024-09-16T03:54:42.759 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.755+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1865013286}) 2024-09-16T03:54:42.760 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.755+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:54:42.760 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.755+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3690842013}) 2024-09-16T03:54:42.760 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.755+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3173821514}) 2024-09-16T03:54:42.761 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.759+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:54:42.762 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.759+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3264203336}) 2024-09-16T03:54:42.762 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.759+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=458121677}) 2024-09-16T03:54:42.762 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.759+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:54:42.762 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.759+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3264203336}) 2024-09-16T03:54:42.762 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.759+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3351490181}) 2024-09-16T03:54:42.763 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.759+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:54:42.763 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.759+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=414222557}) 2024-09-16T03:54:42.764 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.759+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=774211932}) 2024-09-16T03:54:42.764 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.759+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:54:42.764 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.759+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=414222557}) 2024-09-16T03:54:42.764 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.759+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2384142019}) 2024-09-16T03:54:42.765 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.763+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:54:42.765 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.763+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1758674482}) 2024-09-16T03:54:42.765 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.763+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1107563628}) 2024-09-16T03:54:42.765 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.763+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:54:42.766 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.763+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1758674482}) 2024-09-16T03:54:42.766 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.763+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=177705246}) 2024-09-16T03:54:42.767 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.763+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:54:42.767 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.763+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=722339356}) 2024-09-16T03:54:42.767 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.763+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1182653614}) 2024-09-16T03:54:42.767 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.763+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:54:42.768 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.763+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=722339356}) 2024-09-16T03:54:42.768 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.763+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2149140178}) 2024-09-16T03:54:42.769 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.767+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:54:42.769 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.767+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4207840857}) 2024-09-16T03:54:42.769 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.767+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2411660829}) 2024-09-16T03:54:42.769 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.767+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:54:42.769 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.767+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4207840857}) 2024-09-16T03:54:42.769 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.767+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=714164230}) 2024-09-16T03:54:42.771 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T03:54:42.771 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.767+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:54:42.771 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.767+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=391062}) 2024-09-16T03:54:42.771 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.767+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3536812598}) 2024-09-16T03:54:42.771 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.767+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:54:42.772 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.767+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=391062}) 2024-09-16T03:54:42.772 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.767+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3387503774}) 2024-09-16T03:54:42.773 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.767+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:54:42.773 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.767+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2005264530}) 2024-09-16T03:54:42.773 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.767+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=595191264}) 2024-09-16T03:54:42.773 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.767+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:54:42.773 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.767+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2005264530}) 2024-09-16T03:54:42.773 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.767+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1869733824}) 2024-09-16T03:54:42.775 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.771+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:54:42.775 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.771+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4096314041}) 2024-09-16T03:54:42.775 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.771+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=398190468}) 2024-09-16T03:54:42.775 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.771+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:54:42.775 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.771+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4096314041}) 2024-09-16T03:54:42.775 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.771+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2860324779}) 2024-09-16T03:54:42.776 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.771+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:54:42.777 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.771+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3474716600}) 2024-09-16T03:54:42.777 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.771+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4110409313}) 2024-09-16T03:54:42.777 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.771+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:54:42.777 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.771+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3474716600}) 2024-09-16T03:54:42.777 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.771+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3235440740}) 2024-09-16T03:54:42.778 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.775+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:54:42.778 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.775+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2671883350}) 2024-09-16T03:54:42.778 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.775+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2674653939}) 2024-09-16T03:54:42.778 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.775+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:54:42.779 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.775+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2671883350}) 2024-09-16T03:54:42.779 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.775+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2862259551}) 2024-09-16T03:54:42.780 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.775+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:54:42.780 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.775+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2745851788}) 2024-09-16T03:54:42.780 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.775+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3773091115}) 2024-09-16T03:54:42.780 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.775+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:54:42.780 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.775+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2745851788}) 2024-09-16T03:54:42.780 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.775+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2729511657}) 2024-09-16T03:54:42.782 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.779+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:54:42.782 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.779+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4150213767}) 2024-09-16T03:54:42.782 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.779+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3256853525}) 2024-09-16T03:54:42.782 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.779+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:54:42.782 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.779+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4150213767}) 2024-09-16T03:54:42.782 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.779+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=521475330}) 2024-09-16T03:54:42.784 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.779+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:54:42.784 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.779+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1190972534}) 2024-09-16T03:54:42.784 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.779+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2252200217}) 2024-09-16T03:54:42.784 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.779+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:54:42.784 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.779+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1190972534}) 2024-09-16T03:54:42.784 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.779+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1354546738}) 2024-09-16T03:54:42.785 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.783+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:54:42.785 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.783+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3821492376}) 2024-09-16T03:54:42.785 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.783+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3848720359}) 2024-09-16T03:54:42.785 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.783+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:54:42.786 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.783+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3821492376}) 2024-09-16T03:54:42.786 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.783+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3942069648}) 2024-09-16T03:54:42.787 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.783+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:54:42.787 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.783+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1323011521}) 2024-09-16T03:54:42.787 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.783+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1854494468}) 2024-09-16T03:54:42.787 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.783+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:54:42.788 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.783+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1323011521}) 2024-09-16T03:54:42.788 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.783+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2538819786}) 2024-09-16T03:54:42.789 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.787+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:54:42.789 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.787+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1490771646}) 2024-09-16T03:54:42.789 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.787+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1826178890}) 2024-09-16T03:54:42.789 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.787+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:54:42.790 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.787+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1490771646}) 2024-09-16T03:54:42.790 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.787+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3884982898}) 2024-09-16T03:54:42.791 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.787+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:54:42.791 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.787+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1369550476}) 2024-09-16T03:54:42.791 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.787+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4089385843}) 2024-09-16T03:54:42.792 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.787+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:54:42.792 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.787+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1369550476}) 2024-09-16T03:54:42.792 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.787+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=525030563}) 2024-09-16T03:54:42.793 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.791+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:54:42.794 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.791+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2242764457}) 2024-09-16T03:54:42.794 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.791+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2058605943}) 2024-09-16T03:54:42.794 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.791+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:54:42.794 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.791+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2242764457}) 2024-09-16T03:54:42.794 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.791+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=125185260}) 2024-09-16T03:54:42.796 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.791+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:54:42.796 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.791+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=544585248}) 2024-09-16T03:54:42.796 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.791+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3948268948}) 2024-09-16T03:54:42.796 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.791+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:54:42.796 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.791+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=544585248}) 2024-09-16T03:54:42.796 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.791+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3209272334}) 2024-09-16T03:54:42.798 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.795+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:54:42.798 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.795+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=300455475}) 2024-09-16T03:54:42.798 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.795+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4169203880}) 2024-09-16T03:54:42.798 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.795+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:54:42.798 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.795+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=300455475}) 2024-09-16T03:54:42.798 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.795+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3740042984}) 2024-09-16T03:54:42.800 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.795+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:54:42.800 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.795+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3239615216}) 2024-09-16T03:54:42.800 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.795+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3991994912}) 2024-09-16T03:54:42.800 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.795+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:54:42.800 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.795+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3239615216}) 2024-09-16T03:54:42.800 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.795+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2899259640}) 2024-09-16T03:54:42.802 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.799+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:54:42.802 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.799+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3925927075}) 2024-09-16T03:54:42.802 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.799+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1546242064}) 2024-09-16T03:54:42.802 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.799+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:54:42.802 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.799+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3925927075}) 2024-09-16T03:54:42.802 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.799+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3112608080}) 2024-09-16T03:54:42.804 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.799+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:54:42.804 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.799+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1884837098}) 2024-09-16T03:54:42.804 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.799+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1851671935}) 2024-09-16T03:54:42.804 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.799+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:54:42.804 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.799+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1884837098}) 2024-09-16T03:54:42.804 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.799+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2170075742}) 2024-09-16T03:54:42.806 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.803+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:54:42.806 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.803+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=501168339}) 2024-09-16T03:54:42.806 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.803+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2600585576}) 2024-09-16T03:54:42.806 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.803+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:54:42.806 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.803+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=501168339}) 2024-09-16T03:54:42.806 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.803+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3191503573}) 2024-09-16T03:54:42.808 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.803+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:54:42.808 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.803+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2456751655}) 2024-09-16T03:54:42.808 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.803+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2405308531}) 2024-09-16T03:54:42.808 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.803+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:54:42.808 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.803+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2456751655}) 2024-09-16T03:54:42.808 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.803+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=265708274}) 2024-09-16T03:54:42.810 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.807+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:54:42.810 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.807+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=650862822}) 2024-09-16T03:54:42.810 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.807+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1915180148}) 2024-09-16T03:54:42.810 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.807+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:54:42.810 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.807+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=650862822}) 2024-09-16T03:54:42.810 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.807+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1422707122}) 2024-09-16T03:54:42.812 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.807+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:54:42.812 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.807+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1212504688}) 2024-09-16T03:54:42.812 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.807+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2760734073}) 2024-09-16T03:54:42.812 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.807+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:54:42.812 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.807+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1212504688}) 2024-09-16T03:54:42.812 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.807+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=684317122}) 2024-09-16T03:54:42.814 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.811+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:54:42.814 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.811+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=400576216}) 2024-09-16T03:54:42.814 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.811+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1090421226}) 2024-09-16T03:54:42.814 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.811+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:54:42.814 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.811+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=400576216}) 2024-09-16T03:54:42.814 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.811+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3681663387}) 2024-09-16T03:54:42.816 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.811+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:54:42.816 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.811+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2610914608}) 2024-09-16T03:54:42.816 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.811+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=371052830}) 2024-09-16T03:54:42.817 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.811+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:54:42.817 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.811+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2610914608}) 2024-09-16T03:54:42.817 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.811+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3405960298}) 2024-09-16T03:54:42.818 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.815+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:54:42.818 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.815+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2945652189}) 2024-09-16T03:54:42.818 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.815+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1099642057}) 2024-09-16T03:54:42.818 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.815+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:54:42.818 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.815+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2945652189}) 2024-09-16T03:54:42.818 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.815+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2689757699}) 2024-09-16T03:54:42.820 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.815+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:54:42.820 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.815+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=687153371}) 2024-09-16T03:54:42.820 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.815+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3238081284}) 2024-09-16T03:54:42.820 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.815+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:54:42.820 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.815+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=687153371}) 2024-09-16T03:54:42.820 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.815+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1075470741}) 2024-09-16T03:54:42.822 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.819+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:54:42.822 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.819+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3846640522}) 2024-09-16T03:54:42.822 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.819+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2830003693}) 2024-09-16T03:54:42.822 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.819+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:54:42.822 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.819+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3846640522}) 2024-09-16T03:54:42.822 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.819+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=590602993}) 2024-09-16T03:54:42.824 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.819+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:54:42.824 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.819+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=581349517}) 2024-09-16T03:54:42.824 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.819+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2119761660}) 2024-09-16T03:54:42.824 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.819+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:54:42.824 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.819+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=581349517}) 2024-09-16T03:54:42.825 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.819+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3885307771}) 2024-09-16T03:54:42.826 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.823+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:54:42.826 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.823+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3948516062}) 2024-09-16T03:54:42.826 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.823+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4044080063}) 2024-09-16T03:54:42.827 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.823+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:54:42.827 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.823+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3948516062}) 2024-09-16T03:54:42.827 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.823+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4269560275}) 2024-09-16T03:54:42.828 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.823+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:54:42.829 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.823+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2189639010}) 2024-09-16T03:54:42.829 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.823+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=737235732}) 2024-09-16T03:54:42.829 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.823+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:54:42.829 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.823+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2189639010}) 2024-09-16T03:54:42.829 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.823+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3815746136}) 2024-09-16T03:54:42.832 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.827+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:54:42.832 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.827+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=361515148}) 2024-09-16T03:54:42.832 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.827+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1983032890}) 2024-09-16T03:54:42.832 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.827+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:54:42.832 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.827+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=361515148}) 2024-09-16T03:54:42.833 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.827+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=683121111}) 2024-09-16T03:54:42.833 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.827+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:54:42.833 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.827+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1559155500}) 2024-09-16T03:54:42.833 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.827+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1063337081}) 2024-09-16T03:54:42.834 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.827+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:54:42.834 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.827+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1559155500}) 2024-09-16T03:54:42.834 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.827+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1424778873}) 2024-09-16T03:54:42.835 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.831+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:54:42.835 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.831+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4108104486}) 2024-09-16T03:54:42.835 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.831+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2892176240}) 2024-09-16T03:54:42.835 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.831+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:54:42.835 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.831+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4108104486}) 2024-09-16T03:54:42.835 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.831+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1487879844}) 2024-09-16T03:54:42.837 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.835+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:54:42.838 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.835+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2264499950}) 2024-09-16T03:54:42.838 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.835+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=100374297}) 2024-09-16T03:54:42.838 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.835+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:54:42.838 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.835+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2264499950}) 2024-09-16T03:54:42.838 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.835+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=727419398}) 2024-09-16T03:54:42.839 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.835+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:54:42.839 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.835+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=952745299}) 2024-09-16T03:54:42.840 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.835+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=39384394}) 2024-09-16T03:54:42.840 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.835+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:54:42.840 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.835+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=952745299}) 2024-09-16T03:54:42.840 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.835+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3035005544}) 2024-09-16T03:54:42.841 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.839+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:54:42.841 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.839+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4220893163}) 2024-09-16T03:54:42.842 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.839+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2866199682}) 2024-09-16T03:54:42.842 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.839+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:54:42.842 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.839+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4220893163}) 2024-09-16T03:54:42.842 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.839+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1585727428}) 2024-09-16T03:54:42.844 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.839+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:54:42.844 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.839+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1446874481}) 2024-09-16T03:54:42.844 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.839+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3932126344}) 2024-09-16T03:54:42.844 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.839+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:54:42.844 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.839+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1446874481}) 2024-09-16T03:54:42.844 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.839+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3378880651}) 2024-09-16T03:54:42.846 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.843+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:54:42.846 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.843+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2332700486}) 2024-09-16T03:54:42.846 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.843+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=73209579}) 2024-09-16T03:54:42.846 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.843+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:54:42.846 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.843+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2332700486}) 2024-09-16T03:54:42.846 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.843+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1292149893}) 2024-09-16T03:54:42.848 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.843+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:54:42.848 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.843+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3651144144}) 2024-09-16T03:54:42.848 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.843+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1504419327}) 2024-09-16T03:54:42.848 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.843+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:54:42.848 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.843+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3651144144}) 2024-09-16T03:54:42.848 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.843+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=828906019}) 2024-09-16T03:54:42.850 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.847+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:54:42.850 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.847+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=500955308}) 2024-09-16T03:54:42.850 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.847+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=854703673}) 2024-09-16T03:54:42.850 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.847+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:54:42.850 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.847+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=500955308}) 2024-09-16T03:54:42.850 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.847+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2601076495}) 2024-09-16T03:54:42.852 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.847+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:54:42.852 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.847+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=133231788}) 2024-09-16T03:54:42.852 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.847+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3194237126}) 2024-09-16T03:54:42.852 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.847+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:54:42.852 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.847+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=133231788}) 2024-09-16T03:54:42.852 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.847+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1793553513}) 2024-09-16T03:54:42.853 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.851+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:54:42.854 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.851+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=131777624}) 2024-09-16T03:54:42.854 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.851+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2172262643}) 2024-09-16T03:54:42.854 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.851+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:54:42.854 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.851+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=131777624}) 2024-09-16T03:54:42.854 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.851+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3302704623}) 2024-09-16T03:54:42.856 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.851+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:54:42.856 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.851+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=870484919}) 2024-09-16T03:54:42.856 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.851+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3291112320}) 2024-09-16T03:54:42.856 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.851+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:54:42.856 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.851+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=870484919}) 2024-09-16T03:54:42.856 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.851+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2596826716}) 2024-09-16T03:54:42.858 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.855+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:54:42.858 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.855+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3027873635}) 2024-09-16T03:54:42.858 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.855+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3855324793}) 2024-09-16T03:54:42.858 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.855+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:54:42.858 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.855+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3027873635}) 2024-09-16T03:54:42.858 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.855+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3515555881}) 2024-09-16T03:54:42.860 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.855+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:54:42.860 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.855+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2691370621}) 2024-09-16T03:54:42.860 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.855+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3601785820}) 2024-09-16T03:54:42.860 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.855+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:54:42.860 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.855+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2691370621}) 2024-09-16T03:54:42.860 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.855+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4180762365}) 2024-09-16T03:54:42.862 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.859+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:54:42.862 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.859+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1208237265}) 2024-09-16T03:54:42.862 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.859+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3299794329}) 2024-09-16T03:54:42.862 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.859+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:54:42.862 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.859+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1208237265}) 2024-09-16T03:54:42.862 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.859+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1559440493}) 2024-09-16T03:54:42.864 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.859+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:54:42.864 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.859+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3168092490}) 2024-09-16T03:54:42.864 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.859+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3898259438}) 2024-09-16T03:54:42.864 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.859+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:54:42.864 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.859+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3168092490}) 2024-09-16T03:54:42.864 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.859+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3948158888}) 2024-09-16T03:54:42.866 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.863+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:54:42.866 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.863+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3475355784}) 2024-09-16T03:54:42.866 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.863+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3851150082}) 2024-09-16T03:54:42.866 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.863+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:54:42.866 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.863+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3475355784}) 2024-09-16T03:54:42.866 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.863+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=925522273}) 2024-09-16T03:54:42.868 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.863+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:54:42.868 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.863+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=195180037}) 2024-09-16T03:54:42.868 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.863+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3591065479}) 2024-09-16T03:54:42.868 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.863+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:54:42.869 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.863+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=195180037}) 2024-09-16T03:54:42.869 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.863+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=597835060}) 2024-09-16T03:54:42.870 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.867+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:54:42.870 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.867+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1726951173}) 2024-09-16T03:54:42.870 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.867+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=9386643}) 2024-09-16T03:54:42.870 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.867+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:54:42.870 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.867+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1726951173}) 2024-09-16T03:54:42.870 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.867+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=72442723}) 2024-09-16T03:54:42.872 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.867+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:54:42.872 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.867+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2505582910}) 2024-09-16T03:54:42.872 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.867+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1854332295}) 2024-09-16T03:54:42.873 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.867+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:54:42.873 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.867+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2505582910}) 2024-09-16T03:54:42.873 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.867+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1979458872}) 2024-09-16T03:54:42.874 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.871+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:54:42.874 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.871+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=171849214}) 2024-09-16T03:54:42.874 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.871+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3490618552}) 2024-09-16T03:54:42.874 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.871+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:54:42.874 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.871+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=171849214}) 2024-09-16T03:54:42.874 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.871+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3318681676}) 2024-09-16T03:54:42.876 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.871+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:54:42.876 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.871+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3938530728}) 2024-09-16T03:54:42.876 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.871+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4157466532}) 2024-09-16T03:54:42.876 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.871+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:54:42.876 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.871+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3938530728}) 2024-09-16T03:54:42.877 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.871+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2374286278}) 2024-09-16T03:54:42.878 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.875+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:54:42.878 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.875+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2739948961}) 2024-09-16T03:54:42.878 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.875+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3072460028}) 2024-09-16T03:54:42.878 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.875+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:54:42.879 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.875+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2739948961}) 2024-09-16T03:54:42.879 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.875+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3524481819}) 2024-09-16T03:54:42.880 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.875+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:54:42.880 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.875+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1905257735}) 2024-09-16T03:54:42.880 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.875+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1278615047}) 2024-09-16T03:54:42.881 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.875+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:54:42.881 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.875+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1905257735}) 2024-09-16T03:54:42.881 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.875+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1606395139}) 2024-09-16T03:54:42.882 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.879+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:54:42.882 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.879+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4208311082}) 2024-09-16T03:54:42.882 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.879+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=346479923}) 2024-09-16T03:54:42.883 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.879+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:54:42.883 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.879+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4208311082}) 2024-09-16T03:54:42.883 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.879+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3723557421}) 2024-09-16T03:54:42.884 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.879+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:54:42.884 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.879+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=453417317}) 2024-09-16T03:54:42.885 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.879+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3686126379}) 2024-09-16T03:54:42.885 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.879+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:54:42.885 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.879+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=453417317}) 2024-09-16T03:54:42.885 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.879+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2506143323}) 2024-09-16T03:54:42.887 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.883+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:54:42.887 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.883+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2036447382}) 2024-09-16T03:54:42.887 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.883+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3644401678}) 2024-09-16T03:54:42.887 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.883+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:54:42.887 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.883+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2036447382}) 2024-09-16T03:54:42.887 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.883+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4104932813}) 2024-09-16T03:54:42.889 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.883+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:54:42.889 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.883+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2438892784}) 2024-09-16T03:54:42.889 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.883+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=688194863}) 2024-09-16T03:54:42.889 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.883+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:54:42.889 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.883+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2438892784}) 2024-09-16T03:54:42.889 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.883+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3439398917}) 2024-09-16T03:54:42.891 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.887+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:54:42.891 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.887+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=460200129}) 2024-09-16T03:54:42.891 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.887+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2069442748}) 2024-09-16T03:54:42.891 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.887+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:54:42.891 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.887+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=460200129}) 2024-09-16T03:54:42.891 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.887+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1784308662}) 2024-09-16T03:54:42.893 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.887+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:54:42.893 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.887+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2951399503}) 2024-09-16T03:54:42.893 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.887+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=392931898}) 2024-09-16T03:54:42.893 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.887+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:54:42.893 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.891+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2951399503}) 2024-09-16T03:54:42.893 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.891+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=330697828}) 2024-09-16T03:54:42.895 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.891+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:54:42.895 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.891+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=96,mdsmap=3,mgr=1} crc {logm=2777787407,mdsmap=4088524775,mgr=3189685207}) 2024-09-16T03:54:42.895 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.891+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2877393319}) 2024-09-16T03:54:42.895 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.891+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:54:42.895 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.891+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=96,mdsmap=3,mgr=1} crc {logm=2777787407,mdsmap=4088524775,mgr=3189685207}) 2024-09-16T03:54:42.895 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.891+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=94,mdsmap=3,mgr=3} crc {logm=1435829036,mdsmap=4088524775,mgr=2623675823}) 2024-09-16T03:54:42.898 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.895+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:54:42.898 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.895+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgr=5,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=10,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=33,osdmap=29} crc {mgr=4222087715,mgr_command_descs=2107673600,mgr_metadata=1367272202,mgrstat=1717573563,mon_config_key=3148957187,monmap=3937513981,osd_metadata=4182959540,osd_pg_creating=2799207921,osd_snap=3745056784,osdmap=2956713124}) 2024-09-16T03:54:42.898 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.895+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=9,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=35,osdmap=25} crc {mgr=366503433,mgr_command_descs=2107673600,mgr_metadata=1367272202,mgrstat=444851345,mon_config_key=454780364,monmap=3937513981,osd_metadata=4182959540,osd_pg_creating=2799207921,osd_snap=1008964830,osdmap=601220492}) 2024-09-16T03:54:42.898 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.895+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:54:42.899 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.895+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgr=5,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=10,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=33,osdmap=29} crc {mgr=4222087715,mgr_command_descs=2107673600,mgr_metadata=1367272202,mgrstat=1717573563,mon_config_key=3148957187,monmap=3937513981,osd_metadata=4182959540,osd_pg_creating=2799207921,osd_snap=3745056784,osdmap=2956713124}) 2024-09-16T03:54:42.899 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.895+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=10,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=33,osdmap=26} crc {mgr=366503433,mgr_command_descs=2107673600,mgr_metadata=1367272202,mgrstat=444851345,mon_config_key=3148957187,monmap=3937513981,osd_metadata=4182959540,osd_pg_creating=2799207921,osd_snap=1394403960,osdmap=988508647}) 2024-09-16T03:54:42.901 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.895+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:54:42.901 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.895+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3837369636}) 2024-09-16T03:54:42.901 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.895+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3867475268}) 2024-09-16T03:54:42.901 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.895+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:54:42.901 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.895+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3837369636}) 2024-09-16T03:54:42.902 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.895+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=866200626}) 2024-09-16T03:54:42.915 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.911+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:54:42.915 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.911+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=33} crc {osdmap=3177634949}) 2024-09-16T03:54:42.915 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.911+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=32} crc {osdmap=1100863919}) 2024-09-16T03:54:42.915 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.911+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:54:42.915 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.911+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=33} crc {osdmap=3177634949}) 2024-09-16T03:54:42.916 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:54:42.911+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=29} crc {osdmap=2253911274}) 2024-09-16T03:54:43.772 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T03:54:43.772 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:54:43.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.899+0000 7fbdab190640 10 monclient: get_monmap_and_config 2024-09-16T03:54:43.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.899+0000 7fbdab190640 10 monclient: build_initial_monmap 2024-09-16T03:54:43.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.899+0000 7fbdab190640 10 monclient: monmap: 2024-09-16T03:54:43.903 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:54:43.903 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:54:43.903 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:54:43.901646+0000 2024-09-16T03:54:43.903 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:54:43.901646+0000 2024-09-16T03:54:43.903 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:54:43.903 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:54:43.903 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:54:43.903 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:54:43.903 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:54:43.904 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:54:43.904 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:54:43.904 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:54:43.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.899+0000 7fbdab190640 1 Processor -- start 2024-09-16T03:54:43.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.899+0000 7fbdab190640 1 -- start start 2024-09-16T03:54:43.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.899+0000 7fbdab190640 10 monclient: init 2024-09-16T03:54:43.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.899+0000 7fbdab190640 10 monclient: _reopen_session rank -1 2024-09-16T03:54:43.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.899+0000 7fbdab190640 10 monclient: _add_conns ranks=[1,0,3,4,2] 2024-09-16T03:54:43.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.899+0000 7fbdab190640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbda4130b20 addr v1:172.21.15.154:6789/0 2024-09-16T03:54:43.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.899+0000 7fbdab190640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbda40a2250 addr v1:172.21.15.42:6789/0 2024-09-16T03:54:43.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.899+0000 7fbdab190640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fbda413d940 0x7fbda413fd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:54:43.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.899+0000 7fbdab190640 10 monclient(hunting): picked mon.noname-a con 0x7fbda413d940 addr v2:172.21.15.42:3300/0 2024-09-16T03:54:43.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.899+0000 7fbdab190640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbda4131610 con 0x7fbda40a2250 2024-09-16T03:54:43.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.899+0000 7fbdab190640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbda40a12b0 con 0x7fbda4130b20 2024-09-16T03:54:43.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.899+0000 7fbdab190640 10 monclient(hunting): start opening mon connection 2024-09-16T03:54:43.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.899+0000 7fbdab190640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7fbda40a1590 con 0x7fbda413d940 2024-09-16T03:54:43.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.899+0000 7fbdab190640 10 monclient(hunting): _renew_subs 2024-09-16T03:54:43.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.899+0000 7fbdab190640 10 monclient(hunting): authenticate will time out at 1730.871632s 2024-09-16T03:54:43.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.899+0000 7fbda972d640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fbda413d940 0x7fbda413fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:54:43.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.899+0000 7fbd9bfff640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fbda40a2250 0x7fbda413c220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:57220/0 (socket says 172.21.15.42:57220) 2024-09-16T03:54:43.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.899+0000 7fbd9bfff640 1 -- 172.21.15.42:0/4277427183 learned_addr learned my addr 172.21.15.42:0/4277427183 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:54:43.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.899+0000 7fbda972d640 10 monclient(hunting): get_auth_request con 0x7fbda413d940 auth_method 0 2024-09-16T03:54:43.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.899+0000 7fbda972d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:54:43.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.899+0000 7fbda972d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:54:43.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.899+0000 7fbda972d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:54:43.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.903+0000 7fbda972d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:54:43.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.903+0000 7fbda972d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:54:43.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.903+0000 7fbda972d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:54:43.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.903+0000 7fbda972d640 10 monclient(hunting): handle_auth_done global_id 824460 payload 274 2024-09-16T03:54:43.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.903+0000 7fbda972d640 1 -- 172.21.15.42:0/4277427183 >> v1:172.21.15.154:6789/0 conn(0x7fbda4130b20 legacy=0x7fbda4130f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:54:43.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.903+0000 7fbda972d640 1 -- 172.21.15.42:0/4277427183 >> v1:172.21.15.42:6789/0 conn(0x7fbda40a2250 legacy=0x7fbda413c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:54:43.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.903+0000 7fbda972d640 10 monclient: _finish_hunting 0 2024-09-16T03:54:43.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.903+0000 7fbda972d640 1 monclient: found mon.noname-a 2024-09-16T03:54:43.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.903+0000 7fbda972d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:54:43.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.903+0000 7fbda972d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:54:43.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.903+0000 7fbda972d640 1 -- 172.21.15.42:0/4277427183 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbda40a1870 con 0x7fbda413d940 2024-09-16T03:54:43.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.903+0000 7fbd9b7fe640 1 -- 172.21.15.42:0/4277427183 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3254560988 0 0) 0x7fbda4131610 con 0x7fbda40a2250 2024-09-16T03:54:43.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.903+0000 7fbd9b7fe640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T03:54:43.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.903+0000 7fbda972d640 1 --2- 172.21.15.42:0/4277427183 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbda413d940 0x7fbda413fd30 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fbd9c009d90 tx=0x7fbd9c01c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc149782677e6156 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:54:43.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.903+0000 7fbd9b7fe640 1 -- 172.21.15.42:0/4277427183 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd9c0b2070 con 0x7fbda413d940 2024-09-16T03:54:43.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.903+0000 7fbd9b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:54:43.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.903+0000 7fbd9b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:54:43.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.903+0000 7fbd9b7fe640 10 monclient: dump: 2024-09-16T03:54:43.915 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:54:43.915 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:54:43.915 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:43.915 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:43.915 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:54:43.915 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:54:43.915 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:54:43.915 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:54:43.916 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:54:43.916 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:54:43.916 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.903+0000 7fbd9b7fe640 10 monclient: _finish_auth 0 2024-09-16T03:54:43.916 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.903+0000 7fbd9b7fe640 10 monclient: _check_auth_tickets 2024-09-16T03:54:43.916 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.903+0000 7fbd9b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:54:43.916 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.903+0000 7fbdab190640 5 monclient: authenticate success, global_id 824460 2024-09-16T03:54:43.916 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.903+0000 7fbdab190640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:54:43.916 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.903+0000 7fbd9b7fe640 1 -- 172.21.15.42:0/4277427183 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbd9c0a5930 con 0x7fbda413d940 2024-09-16T03:54:43.916 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.903+0000 7fbd9b7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:54:43.916 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.903+0000 7fbdab190640 10 monclient: get_monmap_and_config success 2024-09-16T03:54:43.916 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.903+0000 7fbdab190640 4 set_mon_vals no callback set 2024-09-16T03:54:43.917 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.903+0000 7fbd9b7fe640 1 -- 172.21.15.42:0/4277427183 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd9c0a5d00 con 0x7fbda413d940 2024-09-16T03:54:43.917 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.903+0000 7fbd9b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:54:43.917 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.903+0000 7fbd9b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:54:43.917 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.903+0000 7fbd9b7fe640 10 monclient: dump: 2024-09-16T03:54:43.917 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:54:43.917 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:54:43.917 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:43.917 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:43.917 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:54:43.917 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:54:43.917 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:54:43.918 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:54:43.918 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:54:43.918 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:54:43.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.903+0000 7fbdab190640 10 monclient: shutdown 2024-09-16T03:54:43.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.903+0000 7fbdab190640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:54:43.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.903+0000 7fbdab190640 1 -- 172.21.15.42:0/4277427183 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbda413d940 msgr2=0x7fbda413fd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:54:43.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.903+0000 7fbdab190640 1 --2- 172.21.15.42:0/4277427183 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbda413d940 0x7fbda413fd30 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fbd9c009d90 tx=0x7fbd9c01c850 comp rx=0 tx=0).stop 2024-09-16T03:54:43.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.907+0000 7fbdab190640 1 -- 172.21.15.42:0/4277427183 shutdown_connections 2024-09-16T03:54:43.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.907+0000 7fbdab190640 1 --2- 172.21.15.42:0/4277427183 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbda413d940 0x7fbda413fd30 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:54:43.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.907+0000 7fbdab190640 1 -- 172.21.15.42:0/4277427183 >> 172.21.15.42:0/4277427183 conn(0x7fbda40a6210 msgr2=0x7fbda40a6620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:54:43.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.907+0000 7fbdab190640 1 -- 172.21.15.42:0/4277427183 shutdown_connections 2024-09-16T03:54:43.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.907+0000 7fbdab190640 1 -- 172.21.15.42:0/4277427183 wait complete. 2024-09-16T03:54:43.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.907+0000 7fbdab190640 10 monclient: build_initial_monmap 2024-09-16T03:54:43.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.907+0000 7fbdab190640 10 monclient: monmap: 2024-09-16T03:54:43.919 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:54:43.919 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:54:43.919 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:54:43.919 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:54:43.919 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:54:43.919 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:54:43.919 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:54:43.919 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:54:43.919 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:54:43.919 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:54:43.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.907+0000 7fbdab190640 1 librados: starting msgr at 2024-09-16T03:54:43.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.907+0000 7fbdab190640 1 librados: starting objecter 2024-09-16T03:54:43.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.907+0000 7fbdab190640 1 Processor -- start 2024-09-16T03:54:43.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.907+0000 7fbdab190640 1 -- start start 2024-09-16T03:54:43.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.907+0000 7fbdab190640 1 librados: setting wanted keys 2024-09-16T03:54:43.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.907+0000 7fbdab190640 1 librados: calling monclient init 2024-09-16T03:54:43.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.907+0000 7fbdab190640 10 monclient: init 2024-09-16T03:54:43.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.907+0000 7fbdab190640 10 monclient: _reopen_session rank -1 2024-09-16T03:54:43.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.907+0000 7fbdab190640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:54:43.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.907+0000 7fbdab190640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbda40a2250 0x7fbda412fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:54:43.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.907+0000 7fbdab190640 10 monclient(hunting): picked mon.noname-b con 0x7fbda40a2250 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:54:43.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.907+0000 7fbdab190640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbda4130b20 0x7fbda4130060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:54:43.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.907+0000 7fbdab190640 10 monclient(hunting): picked mon.noname-a con 0x7fbda4130b20 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:54:43.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.907+0000 7fbdab190640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbda413d940 0x7fbda409e270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:54:43.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.907+0000 7fbdab190640 10 monclient(hunting): picked mon.noname-c con 0x7fbda413d940 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:54:43.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.907+0000 7fbdab190640 10 monclient(hunting): start opening mon connection 2024-09-16T03:54:43.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.907+0000 7fbdab190640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fbda4142c10 con 0x7fbda4130b20 2024-09-16T03:54:43.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.907+0000 7fbdab190640 10 monclient(hunting): start opening mon connection 2024-09-16T03:54:43.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.907+0000 7fbdab190640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fbda4142a90 con 0x7fbda40a2250 2024-09-16T03:54:43.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.907+0000 7fbdab190640 10 monclient(hunting): start opening mon connection 2024-09-16T03:54:43.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.907+0000 7fbdab190640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fbda4142d90 con 0x7fbda413d940 2024-09-16T03:54:43.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.907+0000 7fbdab190640 10 monclient(hunting): _renew_subs 2024-09-16T03:54:43.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.907+0000 7fbdab190640 10 monclient(hunting): authenticate will time out at 1730.877971s 2024-09-16T03:54:43.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.907+0000 7fbd9bfff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbda4130b20 0x7fbda4130060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:54:43.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.907+0000 7fbda972d640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbda413d940 0x7fbda409e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:54:43.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.907+0000 7fbd9bfff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbda4130b20 0x7fbda4130060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:33716/0 (socket says 172.21.15.42:33716) 2024-09-16T03:54:43.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.907+0000 7fbd9bfff640 1 -- 172.21.15.42:0/1046406728 learned_addr learned my addr 172.21.15.42:0/1046406728 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:54:43.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.907+0000 7fbd9bfff640 10 monclient(hunting): get_auth_request con 0x7fbda4130b20 auth_method 0 2024-09-16T03:54:43.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.907+0000 7fbd9bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:54:43.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.907+0000 7fbd9bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:54:43.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.907+0000 7fbd9bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:54:43.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.907+0000 7fbda972d640 10 monclient(hunting): get_auth_request con 0x7fbda413d940 auth_method 0 2024-09-16T03:54:43.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.907+0000 7fbda972d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:54:43.923 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.907+0000 7fbda972d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:54:43.923 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.907+0000 7fbda972d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:54:43.923 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.907+0000 7fbd9bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:54:43.923 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.907+0000 7fbd9bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:54:43.923 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.907+0000 7fbd9bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:54:43.923 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.907+0000 7fbda972d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:54:43.923 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.907+0000 7fbda972d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:54:43.923 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.907+0000 7fbda972d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:54:43.923 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.907+0000 7fbda8f2c640 1 --2- 172.21.15.42:0/1046406728 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbda40a2250 0x7fbda412fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:54:43.923 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.907+0000 7fbda8f2c640 10 monclient(hunting): get_auth_request con 0x7fbda40a2250 auth_method 0 2024-09-16T03:54:43.923 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.907+0000 7fbda8f2c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:54:43.923 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.907+0000 7fbda8f2c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:54:43.923 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.907+0000 7fbda8f2c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:54:43.923 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.907+0000 7fbd9bfff640 10 monclient(hunting): handle_auth_done global_id 824466 payload 824 2024-09-16T03:54:43.924 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.907+0000 7fbd9bfff640 1 -- 172.21.15.42:0/1046406728 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbda413d940 msgr2=0x7fbda409e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:54:43.924 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.907+0000 7fbd9bfff640 1 --2- 172.21.15.42:0/1046406728 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbda413d940 0x7fbda409e270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:54:43.924 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.907+0000 7fbd9bfff640 1 -- 172.21.15.42:0/1046406728 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbda40a2250 msgr2=0x7fbda412fb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:54:43.924 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.907+0000 7fbd9bfff640 1 --2- 172.21.15.42:0/1046406728 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbda40a2250 0x7fbda412fb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:54:43.924 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.907+0000 7fbd9bfff640 10 monclient: _finish_hunting 0 2024-09-16T03:54:43.924 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.907+0000 7fbd9bfff640 1 monclient: found mon.noname-a 2024-09-16T03:54:43.924 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.907+0000 7fbd9bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:54:43.924 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.907+0000 7fbd9bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:54:43.924 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.907+0000 7fbd9bfff640 1 -- 172.21.15.42:0/1046406728 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbda409d230 con 0x7fbda4130b20 2024-09-16T03:54:43.924 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.907+0000 7fbd9bfff640 1 --2- 172.21.15.42:0/1046406728 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbda4130b20 0x7fbda4130060 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fbd9400c450 tx=0x7fbd9400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:54:43.924 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.907+0000 7fbd997fa640 10 client.?.objecter ms_handle_connect 0x7fbda4130b20 2024-09-16T03:54:43.924 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.907+0000 7fbd997fa640 10 client.?.objecter resend_mon_ops 2024-09-16T03:54:43.924 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.907+0000 7fbd997fa640 1 -- 172.21.15.42:0/1046406728 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd94018070 con 0x7fbda4130b20 2024-09-16T03:54:43.924 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.907+0000 7fbd997fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:54:43.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.907+0000 7fbd997fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:54:43.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.907+0000 7fbd997fa640 10 monclient: dump: 2024-09-16T03:54:43.925 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:54:43.925 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:54:43.925 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:43.925 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:43.925 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:54:43.925 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:54:43.925 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:54:43.925 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:54:43.925 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:54:43.925 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:54:43.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.907+0000 7fbd997fa640 10 monclient: _finish_auth 0 2024-09-16T03:54:43.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.907+0000 7fbd997fa640 10 monclient: _check_auth_tickets 2024-09-16T03:54:43.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.907+0000 7fbd997fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:54:43.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.907+0000 7fbdab190640 5 monclient: authenticate success, global_id 824466 2024-09-16T03:54:43.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.907+0000 7fbd997fa640 1 -- 172.21.15.42:0/1046406728 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbd940074e0 con 0x7fbda4130b20 2024-09-16T03:54:43.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.907+0000 7fbd997fa640 10 monclient: handle_config config(0 keys) 2024-09-16T03:54:43.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.907+0000 7fbd997fa640 1 -- 172.21.15.42:0/1046406728 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd94004a30 con 0x7fbda4130b20 2024-09-16T03:54:43.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.907+0000 7fbd997fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:54:43.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.907+0000 7fbd997fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:54:43.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.907+0000 7fbd997fa640 10 monclient: dump: 2024-09-16T03:54:43.926 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:54:43.926 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:54:43.926 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:43.926 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:43.926 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:54:43.926 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:54:43.927 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:54:43.927 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:54:43.927 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:54:43.927 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:54:43.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.907+0000 7fbd9affd640 4 set_mon_vals no callback set 2024-09-16T03:54:43.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.907+0000 7fbdab190640 10 monclient: _renew_subs 2024-09-16T03:54:43.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.907+0000 7fbdab190640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:54:43.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.907+0000 7fbdab190640 1 -- 172.21.15.42:0/1046406728 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbda409fe70 con 0x7fbda4130b20 2024-09-16T03:54:43.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.907+0000 7fbdab190640 10 client.824466.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:54:43.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.907+0000 7fbdab190640 10 monclient: _renew_subs 2024-09-16T03:54:43.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.907+0000 7fbdab190640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:54:43.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.907+0000 7fbdab190640 1 -- 172.21.15.42:0/1046406728 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbda4139220 con 0x7fbda4130b20 2024-09-16T03:54:43.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.907+0000 7fbdab190640 1 librados: init done 2024-09-16T03:54:43.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.911+0000 7fbdab190640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:54:43.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.911+0000 7fbdab190640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:54:43.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.911+0000 7fbdab190640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:54:43.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.911+0000 7fbdab190640 1 -- 172.21.15.42:0/1046406728 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbda4130f00 con 0x7fbda4130b20 2024-09-16T03:54:43.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.911+0000 7fbd997fa640 1 -- 172.21.15.42:0/1046406728 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fbd94004bd0 con 0x7fbda4130b20 2024-09-16T03:54:43.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.911+0000 7fbd997fa640 1 --2- 172.21.15.42:0/1046406728 >> v2:172.21.15.154:6812/254845342 conn(0x7fbd90037840 0x7fbd90039d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:54:43.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.911+0000 7fbd997fa640 1 -- 172.21.15.42:0/1046406728 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(386..386 src has 286..386) ==== 5713+0+0 (secure 0 0 0) 0x7fbd940480a0 con 0x7fbda4130b20 2024-09-16T03:54:43.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.911+0000 7fbd997fa640 10 client.824466.objecter ms_dispatch 0x7fbda4000ba0 osd_map(386..386 src has 286..386) 2024-09-16T03:54:43.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.911+0000 7fbd997fa640 3 client.824466.objecter handle_osd_map got epochs [386,386] > 0 2024-09-16T03:54:43.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.911+0000 7fbd997fa640 3 client.824466.objecter handle_osd_map decoding full epoch 386 2024-09-16T03:54:43.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.911+0000 7fbd997fa640 20 client.824466.objecter dump_active .. 0 homeless 2024-09-16T03:54:43.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.911+0000 7fbda8f2c640 1 --2- 172.21.15.42:0/1046406728 >> v2:172.21.15.154:6812/254845342 conn(0x7fbd90037840 0x7fbd90039d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:54:43.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.911+0000 7fbda8f2c640 10 monclient: get_auth_request con 0x7fbd90037840 auth_method 0 2024-09-16T03:54:43.929 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.911+0000 7fbda8f2c640 1 --2- 172.21.15.42:0/1046406728 >> v2:172.21.15.154:6812/254845342 conn(0x7fbd90037840 0x7fbd90039d00 secure :-1 s=READY pgs=3618 cs=0 l=1 rev1=1 crypto rx=0x7fbda0000f70 tx=0x7fbda0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:54:43.929 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.911+0000 7fbd997fa640 10 client.824466.objecter ms_handle_connect 0x7fbd90037840 2024-09-16T03:54:43.929 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.915+0000 7fbd997fa640 1 -- 172.21.15.42:0/1046406728 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7fbd94010300 con 0x7fbda4130b20 2024-09-16T03:54:43.929 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.915+0000 7fbd997fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:54:43.929 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:43.915+0000 7fbd997fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:54:44.143 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.139+0000 7fbdab190640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:54:44.143 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.139+0000 7fbdab190640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:54:44.143 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.139+0000 7fbdab190640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:54:44.143 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.139+0000 7fbdab190640 1 -- 172.21.15.42:0/1046406728 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fbda413cfe0 con 0x7fbda4130b20 2024-09-16T03:54:44.143 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.139+0000 7fbd997fa640 1 -- 172.21.15.42:0/1046406728 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7fbd9401b360 con 0x7fbda4130b20 2024-09-16T03:54:44.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.139+0000 7fbd997fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:54:44.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.139+0000 7fbd997fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:54:44.144 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":110,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:54:44.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.143+0000 7fbdab190640 10 librados: watch_flush enter 2024-09-16T03:54:44.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.143+0000 7fbdab190640 10 librados: watch_flush exit 2024-09-16T03:54:44.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.143+0000 7fbdab190640 20 client.824466.objecter shutdown clearing up homeless session... 2024-09-16T03:54:44.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.143+0000 7fbdab190640 10 client.824466.objecter successfully canceled tick 2024-09-16T03:54:44.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.143+0000 7fbdab190640 1 -- 172.21.15.42:0/1046406728 >> v2:172.21.15.154:6812/254845342 conn(0x7fbd90037840 msgr2=0x7fbd90039d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:54:44.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.143+0000 7fbdab190640 1 --2- 172.21.15.42:0/1046406728 >> v2:172.21.15.154:6812/254845342 conn(0x7fbd90037840 0x7fbd90039d00 secure :-1 s=READY pgs=3618 cs=0 l=1 rev1=1 crypto rx=0x7fbda0000f70 tx=0x7fbda0009040 comp rx=0 tx=0).stop 2024-09-16T03:54:44.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.143+0000 7fbdab190640 10 monclient: shutdown 2024-09-16T03:54:44.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.143+0000 7fbdab190640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:54:44.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.143+0000 7fbdab190640 1 -- 172.21.15.42:0/1046406728 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbda4130b20 msgr2=0x7fbda4130060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:54:44.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.143+0000 7fbdab190640 1 --2- 172.21.15.42:0/1046406728 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbda4130b20 0x7fbda4130060 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fbd9400c450 tx=0x7fbd9400c910 comp rx=0 tx=0).stop 2024-09-16T03:54:44.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.143+0000 7fbdab190640 1 -- 172.21.15.42:0/1046406728 shutdown_connections 2024-09-16T03:54:44.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.143+0000 7fbdab190640 1 --2- 172.21.15.42:0/1046406728 >> v2:172.21.15.154:6812/254845342 conn(0x7fbd90037840 0x7fbd90039d00 unknown :-1 s=CLOSED pgs=3618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:54:44.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.143+0000 7fbdab190640 1 --2- 172.21.15.42:0/1046406728 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbda413d940 0x7fbda409e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:54:44.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.143+0000 7fbdab190640 1 --2- 172.21.15.42:0/1046406728 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbda4130b20 0x7fbda4130060 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:54:44.148 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.143+0000 7fbdab190640 1 --2- 172.21.15.42:0/1046406728 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbda40a2250 0x7fbda412fb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:54:44.148 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.143+0000 7fbdab190640 1 -- 172.21.15.42:0/1046406728 >> 172.21.15.42:0/1046406728 conn(0x7fbda40a6210 msgr2=0x7fbda40a3e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:54:44.148 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.143+0000 7fbdab190640 1 -- 172.21.15.42:0/1046406728 shutdown_connections 2024-09-16T03:54:44.148 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.143+0000 7fbdab190640 1 -- 172.21.15.42:0/1046406728 wait complete. 2024-09-16T03:54:44.148 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.143+0000 7fbdab190640 1 librados: shutdown 2024-09-16T03:54:44.164 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T03:54:44.164 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T03:54:44.164 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T03:54:44.164 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T03:54:44.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.279+0000 7f8267bc7640 10 monclient: get_monmap_and_config 2024-09-16T03:54:44.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.279+0000 7f8267bc7640 10 monclient: build_initial_monmap 2024-09-16T03:54:44.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.279+0000 7f8267bc7640 10 monclient: monmap: 2024-09-16T03:54:44.282 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:54:44.283 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:54:44.283 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:54:44.281175+0000 2024-09-16T03:54:44.283 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:54:44.281175+0000 2024-09-16T03:54:44.283 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:54:44.283 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:54:44.283 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:54:44.283 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:54:44.283 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:54:44.283 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:54:44.283 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:54:44.283 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:54:44.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.279+0000 7f8267bc7640 1 Processor -- start 2024-09-16T03:54:44.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.279+0000 7f8267bc7640 1 -- start start 2024-09-16T03:54:44.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.279+0000 7f8267bc7640 10 monclient: init 2024-09-16T03:54:44.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.279+0000 7f8267bc7640 10 monclient: _reopen_session rank -1 2024-09-16T03:54:44.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.279+0000 7f8267bc7640 10 monclient: _add_conns ranks=[1,0,2,4,3] 2024-09-16T03:54:44.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.279+0000 7f8267bc7640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f82600a1330 addr v1:172.21.15.154:6789/0 2024-09-16T03:54:44.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.279+0000 7f8267bc7640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f826006dac0 addr v1:172.21.15.42:6789/0 2024-09-16T03:54:44.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.279+0000 7f8267bc7640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f82600a1e20 0x7f826013da70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:54:44.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.279+0000 7f8267bc7640 10 monclient(hunting): picked mon.noname-b con 0x7f82600a1e20 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:54:44.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.279+0000 7f8267bc7640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f826009c090 con 0x7f826006dac0 2024-09-16T03:54:44.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.279+0000 7f8267bc7640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f826009f610 con 0x7f82600a1330 2024-09-16T03:54:44.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.279+0000 7f8267bc7640 10 monclient(hunting): start opening mon connection 2024-09-16T03:54:44.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.279+0000 7f8267bc7640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f826009c890 con 0x7f82600a1e20 2024-09-16T03:54:44.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.279+0000 7f8267bc7640 10 monclient(hunting): _renew_subs 2024-09-16T03:54:44.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.279+0000 7f8267bc7640 10 monclient(hunting): authenticate will time out at 1731.250621s 2024-09-16T03:54:44.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.279+0000 7f8266164640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f82600a1e20 0x7f826013da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:54:44.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.279+0000 7f8265162640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f826006dac0 0x7f826013a340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:57026/0 (socket says 172.21.15.42:57026) 2024-09-16T03:54:44.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.279+0000 7f8265162640 1 -- 172.21.15.42:0/2112948082 learned_addr learned my addr 172.21.15.42:0/2112948082 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:54:44.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.279+0000 7f8266164640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f82600a1e20 0x7f826013da70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:60998/0 (socket says 172.21.15.42:60998) 2024-09-16T03:54:44.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.279+0000 7f8266164640 10 monclient(hunting): get_auth_request con 0x7f82600a1e20 auth_method 0 2024-09-16T03:54:44.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.279+0000 7f8266164640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:54:44.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.279+0000 7f8266164640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:54:44.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.279+0000 7f8266164640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:54:44.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.279+0000 7f8266164640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:54:44.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.279+0000 7f8266164640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:54:44.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.279+0000 7f8266164640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:54:44.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.279+0000 7f8264961640 1 -- 172.21.15.42:0/2112948082 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3305850998 0 0) 0x7f826009c090 con 0x7f826006dac0 2024-09-16T03:54:44.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.279+0000 7f8264961640 10 monclient(hunting): _negotiate 2024-09-16T03:54:44.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.279+0000 7f8264961640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:54:44.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.279+0000 7f8264961640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:54:44.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.279+0000 7f8264961640 10 monclient(hunting): my global_id is 824469 2024-09-16T03:54:44.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.279+0000 7f8264961640 1 -- 172.21.15.42:0/2112948082 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f824c003530 con 0x7f826006dac0 2024-09-16T03:54:44.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.279+0000 7f8266164640 10 monclient(hunting): handle_auth_done global_id 854129 payload 274 2024-09-16T03:54:44.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.279+0000 7f8266164640 1 -- 172.21.15.42:0/2112948082 >> v1:172.21.15.154:6789/0 conn(0x7f82600a1330 legacy=0x7f82600a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:54:44.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.279+0000 7f8266164640 1 -- 172.21.15.42:0/2112948082 >> v1:172.21.15.42:6789/0 conn(0x7f826006dac0 legacy=0x7f826013a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:54:44.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.279+0000 7f8266164640 10 monclient: _finish_hunting 0 2024-09-16T03:54:44.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.279+0000 7f8266164640 1 monclient: found mon.noname-b 2024-09-16T03:54:44.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.279+0000 7f8266164640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:54:44.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.279+0000 7f8266164640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T03:54:44.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.279+0000 7f8266164640 1 -- 172.21.15.42:0/2112948082 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f826009f8f0 con 0x7f82600a1e20 2024-09-16T03:54:44.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.283+0000 7f8266164640 1 --2- 172.21.15.42:0/2112948082 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f82600a1e20 0x7f826013da70 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f8258009dc0 tx=0x7f825801c870 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e12f75cfe104f599 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:54:44.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.283+0000 7f8264961640 1 -- 172.21.15.42:0/2112948082 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82580b2070 con 0x7f82600a1e20 2024-09-16T03:54:44.298 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.283+0000 7f8264961640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:54:44.298 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.283+0000 7f8264961640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:54:44.298 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.283+0000 7f8264961640 10 monclient: dump: 2024-09-16T03:54:44.298 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:54:44.298 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:54:44.298 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:44.298 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:44.298 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:54:44.298 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:54:44.298 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:54:44.298 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:54:44.298 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:54:44.299 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:54:44.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.283+0000 7f8264961640 10 monclient: _finish_auth 0 2024-09-16T03:54:44.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.283+0000 7f8264961640 10 monclient: _check_auth_tickets 2024-09-16T03:54:44.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.283+0000 7f8264961640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:54:44.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.283+0000 7f8264961640 1 -- 172.21.15.42:0/2112948082 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f82580a5950 con 0x7f82600a1e20 2024-09-16T03:54:44.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.283+0000 7f8267bc7640 5 monclient: authenticate success, global_id 854129 2024-09-16T03:54:44.300 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.283+0000 7f8267bc7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:54:44.300 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.283+0000 7f8264961640 10 monclient: handle_config config(0 keys) 2024-09-16T03:54:44.300 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.283+0000 7f8267bc7640 10 monclient: get_monmap_and_config success 2024-09-16T03:54:44.300 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.283+0000 7f8267bc7640 4 set_mon_vals no callback set 2024-09-16T03:54:44.300 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.283+0000 7f8264961640 1 -- 172.21.15.42:0/2112948082 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82580a5d20 con 0x7f82600a1e20 2024-09-16T03:54:44.300 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.283+0000 7f8264961640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:54:44.300 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.283+0000 7f8264961640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:54:44.300 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.283+0000 7f8264961640 10 monclient: dump: 2024-09-16T03:54:44.300 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:54:44.300 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:54:44.301 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:44.301 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:44.301 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:54:44.301 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:54:44.301 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:54:44.301 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:54:44.301 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:54:44.301 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:54:44.301 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.283+0000 7f8267bc7640 10 monclient: shutdown 2024-09-16T03:54:44.301 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.283+0000 7f8267bc7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:54:44.301 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.283+0000 7f8267bc7640 1 -- 172.21.15.42:0/2112948082 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f82600a1e20 msgr2=0x7f826013da70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:54:44.301 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.283+0000 7f8267bc7640 1 --2- 172.21.15.42:0/2112948082 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f82600a1e20 0x7f826013da70 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f8258009dc0 tx=0x7f825801c870 comp rx=0 tx=0).stop 2024-09-16T03:54:44.301 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.283+0000 7f8267bc7640 1 -- 172.21.15.42:0/2112948082 shutdown_connections 2024-09-16T03:54:44.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.283+0000 7f8267bc7640 1 --2- 172.21.15.42:0/2112948082 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f82600a1e20 0x7f826013da70 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:54:44.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.283+0000 7f8267bc7640 1 -- 172.21.15.42:0/2112948082 >> 172.21.15.42:0/2112948082 conn(0x7f82600a2420 msgr2=0x7f82600a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:54:44.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.283+0000 7f8267bc7640 1 -- 172.21.15.42:0/2112948082 shutdown_connections 2024-09-16T03:54:44.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.283+0000 7f8267bc7640 1 -- 172.21.15.42:0/2112948082 wait complete. 2024-09-16T03:54:44.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.283+0000 7f8267bc7640 10 monclient: build_initial_monmap 2024-09-16T03:54:44.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.283+0000 7f8267bc7640 10 monclient: monmap: 2024-09-16T03:54:44.302 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:54:44.302 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:54:44.302 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:54:44.302 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:54:44.302 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:54:44.302 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:54:44.302 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:54:44.302 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:54:44.303 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:54:44.303 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:54:44.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.283+0000 7f8267bc7640 1 librados: starting msgr at 2024-09-16T03:54:44.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.283+0000 7f8267bc7640 1 librados: starting objecter 2024-09-16T03:54:44.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.283+0000 7f8267bc7640 1 Processor -- start 2024-09-16T03:54:44.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.283+0000 7f8267bc7640 1 -- start start 2024-09-16T03:54:44.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.283+0000 7f8267bc7640 1 librados: setting wanted keys 2024-09-16T03:54:44.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.283+0000 7f8267bc7640 1 librados: calling monclient init 2024-09-16T03:54:44.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.283+0000 7f8267bc7640 10 monclient: init 2024-09-16T03:54:44.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.283+0000 7f8267bc7640 10 monclient: _reopen_session rank -1 2024-09-16T03:54:44.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.283+0000 7f8267bc7640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:54:44.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.283+0000 7f8267bc7640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f82600a1330 0x7f826011bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:54:44.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.283+0000 7f8267bc7640 10 monclient(hunting): picked mon.noname-c con 0x7f82600a1330 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:54:44.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.283+0000 7f8267bc7640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f82600a1e20 0x7f826011c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:54:44.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.283+0000 7f8267bc7640 10 monclient(hunting): picked mon.noname-b con 0x7f82600a1e20 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:54:44.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.283+0000 7f8267bc7640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f826011c9f0 0x7f82601349f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:54:44.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.283+0000 7f8267bc7640 10 monclient(hunting): picked mon.noname-a con 0x7f826011c9f0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:54:44.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.283+0000 7f8267bc7640 10 monclient(hunting): start opening mon connection 2024-09-16T03:54:44.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.283+0000 7f8267bc7640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f8260140950 con 0x7f826011c9f0 2024-09-16T03:54:44.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.283+0000 7f8267bc7640 10 monclient(hunting): start opening mon connection 2024-09-16T03:54:44.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.283+0000 7f8267bc7640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f82601407d0 con 0x7f82600a1e20 2024-09-16T03:54:44.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.283+0000 7f8267bc7640 10 monclient(hunting): start opening mon connection 2024-09-16T03:54:44.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.283+0000 7f8267bc7640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f8260140ad0 con 0x7f82600a1330 2024-09-16T03:54:44.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.283+0000 7f8267bc7640 10 monclient(hunting): _renew_subs 2024-09-16T03:54:44.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.283+0000 7f8267bc7640 10 monclient(hunting): authenticate will time out at 1731.255752s 2024-09-16T03:54:44.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.283+0000 7f8265963640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f82600a1330 0x7f826011bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:54:44.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.283+0000 7f8266164640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f826011c9f0 0x7f82601349f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:54:44.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.283+0000 7f8265963640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f82600a1330 0x7f826011bf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:32770/0 (socket says 172.21.15.42:32770) 2024-09-16T03:54:44.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.283+0000 7f8265963640 1 -- 172.21.15.42:0/1653083196 learned_addr learned my addr 172.21.15.42:0/1653083196 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:54:44.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.283+0000 7f8266164640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f826011c9f0 0x7f82601349f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:41534/0 (socket says 172.21.15.42:41534) 2024-09-16T03:54:44.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.283+0000 7f8265963640 10 monclient(hunting): get_auth_request con 0x7f82600a1330 auth_method 0 2024-09-16T03:54:44.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.283+0000 7f8265963640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:54:44.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.283+0000 7f8265963640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:54:44.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.283+0000 7f8265963640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:54:44.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.283+0000 7f8266164640 10 monclient(hunting): get_auth_request con 0x7f826011c9f0 auth_method 0 2024-09-16T03:54:44.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.283+0000 7f8266164640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:54:44.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.283+0000 7f8266164640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:54:44.306 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.283+0000 7f8266164640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:54:44.306 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.287+0000 7f8265963640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:54:44.306 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.287+0000 7f8265963640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:54:44.306 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.287+0000 7f8265963640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:54:44.306 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.287+0000 7f8266164640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:54:44.306 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.287+0000 7f8266164640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:54:44.306 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.287+0000 7f8266164640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:54:44.306 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.287+0000 7f8265162640 1 --2- 172.21.15.42:0/1653083196 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f82600a1e20 0x7f826011c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:54:44.306 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.287+0000 7f8265162640 10 monclient(hunting): get_auth_request con 0x7f82600a1e20 auth_method 0 2024-09-16T03:54:44.306 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.287+0000 7f8265162640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:54:44.306 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.287+0000 7f8265162640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:54:44.306 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.287+0000 7f8265162640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:54:44.306 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.287+0000 7f8265963640 10 monclient(hunting): handle_auth_done global_id 854132 payload 824 2024-09-16T03:54:44.306 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.287+0000 7f8265963640 1 -- 172.21.15.42:0/1653083196 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f82600a1e20 msgr2=0x7f826011c4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:54:44.307 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.287+0000 7f8265963640 1 --2- 172.21.15.42:0/1653083196 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f82600a1e20 0x7f826011c4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:54:44.307 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.287+0000 7f8265963640 1 -- 172.21.15.42:0/1653083196 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f826011c9f0 msgr2=0x7f82601349f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:54:44.307 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.287+0000 7f8265963640 1 --2- 172.21.15.42:0/1653083196 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f826011c9f0 0x7f82601349f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:54:44.307 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.287+0000 7f8265963640 10 monclient: _finish_hunting 0 2024-09-16T03:54:44.308 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.287+0000 7f8265963640 1 monclient: found mon.noname-c 2024-09-16T03:54:44.308 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.287+0000 7f8265963640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:54:44.308 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.287+0000 7f8265963640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:54:44.308 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.287+0000 7f8265963640 1 -- 172.21.15.42:0/1653083196 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f826013d420 con 0x7f82600a1330 2024-09-16T03:54:44.308 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.287+0000 7f8265963640 1 --2- 172.21.15.42:0/1653083196 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f82600a1330 0x7f826011bf70 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f825000c420 tx=0x7f825000c8e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:54:44.308 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.287+0000 7f82567fc640 10 client.?.objecter ms_handle_connect 0x7f82600a1330 2024-09-16T03:54:44.308 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.287+0000 7f82567fc640 10 client.?.objecter resend_mon_ops 2024-09-16T03:54:44.308 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.287+0000 7f82567fc640 1 -- 172.21.15.42:0/1653083196 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82500052d0 con 0x7f82600a1330 2024-09-16T03:54:44.308 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.287+0000 7f82567fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:54:44.308 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.287+0000 7f82567fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:54:44.308 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.287+0000 7f82567fc640 10 monclient: dump: 2024-09-16T03:54:44.309 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:54:44.309 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:54:44.309 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:44.309 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:44.309 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:54:44.309 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:54:44.309 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:54:44.309 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:54:44.309 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:54:44.309 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:54:44.309 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.287+0000 7f82567fc640 10 monclient: _finish_auth 0 2024-09-16T03:54:44.309 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.287+0000 7f82567fc640 10 monclient: _check_auth_tickets 2024-09-16T03:54:44.309 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.287+0000 7f82567fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:54:44.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.287+0000 7f82567fc640 1 -- 172.21.15.42:0/1653083196 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8250005470 con 0x7f82600a1330 2024-09-16T03:54:44.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.287+0000 7f82567fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:54:44.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.287+0000 7f82567fc640 1 -- 172.21.15.42:0/1653083196 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8250007260 con 0x7f82600a1330 2024-09-16T03:54:44.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.287+0000 7f82567fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:54:44.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.287+0000 7f82567fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:54:44.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.287+0000 7f82567fc640 10 monclient: dump: 2024-09-16T03:54:44.310 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:54:44.310 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:54:44.310 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:44.310 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:44.310 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:54:44.310 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:54:44.310 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:54:44.311 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:54:44.311 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:54:44.311 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:54:44.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.287+0000 7f8267bc7640 5 monclient: authenticate success, global_id 854132 2024-09-16T03:54:44.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.287+0000 7f8267bc7640 10 monclient: _renew_subs 2024-09-16T03:54:44.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.287+0000 7f8267bc7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:54:44.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.287+0000 7f8267bc7640 1 -- 172.21.15.42:0/1653083196 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f826006e3b0 con 0x7f82600a1330 2024-09-16T03:54:44.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.287+0000 7f8257fff640 4 set_mon_vals no callback set 2024-09-16T03:54:44.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.287+0000 7f8267bc7640 10 client.854132.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:54:44.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.287+0000 7f8267bc7640 10 monclient: _renew_subs 2024-09-16T03:54:44.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.287+0000 7f8267bc7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:54:44.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.287+0000 7f8267bc7640 1 -- 172.21.15.42:0/1653083196 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f826011e790 con 0x7f82600a1330 2024-09-16T03:54:44.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.287+0000 7f8267bc7640 1 librados: init done 2024-09-16T03:54:44.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.287+0000 7f8267bc7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:54:44.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.287+0000 7f82567fc640 1 -- 172.21.15.42:0/1653083196 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f8250007400 con 0x7f82600a1330 2024-09-16T03:54:44.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.287+0000 7f8267bc7640 1 --2- 172.21.15.42:0/1653083196 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f826006dac0 0x7f826002c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:54:44.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.287+0000 7f8267bc7640 10 monclient(hunting): start opening mon connection 2024-09-16T03:54:44.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.287+0000 7f8267bc7640 1 -- 172.21.15.42:0/1653083196 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f8260073a30 con 0x7f826006dac0 2024-09-16T03:54:44.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.287+0000 7f8265162640 1 --2- 172.21.15.42:0/1653083196 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f826006dac0 0x7f826002c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:54:44.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.287+0000 7f8265162640 10 monclient: get_auth_request con 0x7f826006dac0 auth_method 0 2024-09-16T03:54:44.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.287+0000 7f8265162640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:54:44.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.287+0000 7f8265162640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:54:44.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.287+0000 7f8265162640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:54:44.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.287+0000 7f82567fc640 1 --2- 172.21.15.42:0/1653083196 >> v2:172.21.15.154:6812/254845342 conn(0x7f8244037810 0x7f8244039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:54:44.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.287+0000 7f8265162640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:54:44.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.287+0000 7f8265162640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:54:44.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.287+0000 7f82567fc640 1 -- 172.21.15.42:0/1653083196 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(386..386 src has 286..386) ==== 5713+0+0 (secure 0 0 0) 0x7f8250048470 con 0x7f82600a1330 2024-09-16T03:54:44.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.287+0000 7f82567fc640 10 client.854132.objecter ms_dispatch 0x7f8260000ba0 osd_map(386..386 src has 286..386) 2024-09-16T03:54:44.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.287+0000 7f82567fc640 3 client.854132.objecter handle_osd_map got epochs [386,386] > 0 2024-09-16T03:54:44.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.287+0000 7f82567fc640 3 client.854132.objecter handle_osd_map decoding full epoch 386 2024-09-16T03:54:44.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.287+0000 7f8265162640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:54:44.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.287+0000 7f82567fc640 20 client.854132.objecter dump_active .. 0 homeless 2024-09-16T03:54:44.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.287+0000 7f8265162640 10 monclient(hunting): handle_auth_done global_id 824475 payload 824 2024-09-16T03:54:44.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.287+0000 7f8265162640 1 -- 172.21.15.42:0/1653083196 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f82600a1710 con 0x7f826006dac0 2024-09-16T03:54:44.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.287+0000 7f8266164640 1 --2- 172.21.15.42:0/1653083196 >> v2:172.21.15.154:6812/254845342 conn(0x7f8244037810 0x7f8244039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:54:44.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.287+0000 7f8266164640 10 monclient: get_auth_request con 0x7f8244037810 auth_method 0 2024-09-16T03:54:44.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.287+0000 7f82567fc640 10 client.854132.objecter ms_handle_connect 0x7f826006dac0 2024-09-16T03:54:44.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.287+0000 7f8265162640 1 --2- 172.21.15.42:0/1653083196 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f826006dac0 0x7f826002c650 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f826013c680 tx=0x7f825c001980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:54:44.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.287+0000 7f82567fc640 10 client.854132.objecter resend_mon_ops 2024-09-16T03:54:44.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.287+0000 7f82567fc640 1 -- 172.21.15.42:0/1653083196 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f825c012590 con 0x7f826006dac0 2024-09-16T03:54:44.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.287+0000 7f82567fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:54:44.314 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.287+0000 7f82567fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:54:44.314 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.287+0000 7f82567fc640 10 monclient: dump: 2024-09-16T03:54:44.314 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:54:44.314 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:54:44.314 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:44.314 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:44.314 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:54:44.314 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:54:44.314 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:54:44.314 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:54:44.314 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:54:44.314 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:54:44.314 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.287+0000 7f82567fc640 1 -- 172.21.15.42:0/1653083196 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f825c018070 con 0x7f826006dac0 2024-09-16T03:54:44.314 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.287+0000 7f82567fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:54:44.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.287+0000 7f82567fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:54:44.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.287+0000 7f82567fc640 1 -- 172.21.15.42:0/1653083196 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f826006dac0 msgr2=0x7f826002c650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:54:44.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.287+0000 7f82567fc640 1 --2- 172.21.15.42:0/1653083196 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f826006dac0 0x7f826002c650 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f826013c680 tx=0x7f825c001980 comp rx=0 tx=0).stop 2024-09-16T03:54:44.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.287+0000 7f82567fc640 1 -- 172.21.15.42:0/1653083196 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f826006dac0 msgr2=0x7f826002c650 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:54:44.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.287+0000 7f82567fc640 1 --2- 172.21.15.42:0/1653083196 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f826006dac0 0x7f826002c650 secure :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f826013c680 tx=0x7f825c001980 comp rx=0 tx=0).stop 2024-09-16T03:54:44.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.287+0000 7f82567fc640 10 client.854132.objecter ms_handle_connect 0x7f8244037810 2024-09-16T03:54:44.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.287+0000 7f8266164640 1 --2- 172.21.15.42:0/1653083196 >> v2:172.21.15.154:6812/254845342 conn(0x7f8244037810 0x7f8244039cd0 secure :-1 s=READY pgs=3619 cs=0 l=1 rev1=1 crypto rx=0x7f82580af000 tx=0x7f82580a7070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:54:44.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.295+0000 7f8267bc7640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:54:44.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.295+0000 7f8267bc7640 1 --2- 172.21.15.42:0/1653083196 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f826002cb90 0x7f826002cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:54:44.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.295+0000 7f8267bc7640 10 monclient(hunting): start opening mon connection 2024-09-16T03:54:44.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.295+0000 7f8267bc7640 1 -- 172.21.15.42:0/1653083196 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f8260139150 con 0x7f826002cb90 2024-09-16T03:54:44.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.295+0000 7f8265162640 1 --2- 172.21.15.42:0/1653083196 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f826002cb90 0x7f826002cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:54:44.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.295+0000 7f8265162640 10 monclient: get_auth_request con 0x7f826002cb90 auth_method 0 2024-09-16T03:54:44.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.295+0000 7f8265162640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:54:44.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.295+0000 7f8265162640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:54:44.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.295+0000 7f8265162640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:54:44.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.295+0000 7f8265162640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:54:44.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.295+0000 7f8265162640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:54:44.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.295+0000 7f8265162640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:54:44.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.295+0000 7f8265162640 10 monclient(hunting): handle_auth_done global_id 824478 payload 824 2024-09-16T03:54:44.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.295+0000 7f8265162640 1 -- 172.21.15.42:0/1653083196 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f8260069040 con 0x7f826002cb90 2024-09-16T03:54:44.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.295+0000 7f82567fc640 10 client.854132.objecter ms_handle_connect 0x7f826002cb90 2024-09-16T03:54:44.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.295+0000 7f82567fc640 10 client.854132.objecter resend_mon_ops 2024-09-16T03:54:44.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.295+0000 7f8265162640 1 --2- 172.21.15.42:0/1653083196 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f826002cb90 0x7f826002cfa0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f825c0077f0 tx=0x7f825c000ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:54:44.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.295+0000 7f82567fc640 1 -- 172.21.15.42:0/1653083196 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f825c012bc0 con 0x7f826002cb90 2024-09-16T03:54:44.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.295+0000 7f82567fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:54:44.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.295+0000 7f82567fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:54:44.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.295+0000 7f82567fc640 10 monclient: dump: 2024-09-16T03:54:44.319 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:54:44.319 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:54:44.319 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:44.319 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:44.319 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:54:44.319 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:54:44.319 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:54:44.319 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:54:44.319 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:54:44.320 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:54:44.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.295+0000 7f82567fc640 1 -- 172.21.15.42:0/1653083196 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4882 (secure 0 0 0) 0x7f825c012bc0 con 0x7f826002cb90 2024-09-16T03:54:44.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.295+0000 7f82567fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:54:44.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.295+0000 7f82567fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:54:44.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.295+0000 7f82567fc640 1 -- 172.21.15.42:0/1653083196 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f826002cb90 msgr2=0x7f826002cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:54:44.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.295+0000 7f82567fc640 1 --2- 172.21.15.42:0/1653083196 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f826002cb90 0x7f826002cfa0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f825c0077f0 tx=0x7f825c000ee0 comp rx=0 tx=0).stop 2024-09-16T03:54:44.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.295+0000 7f82567fc640 1 -- 172.21.15.42:0/1653083196 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f826002cb90 msgr2=0x7f826002cfa0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:54:44.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.295+0000 7f82567fc640 1 --2- 172.21.15.42:0/1653083196 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f826002cb90 0x7f826002cfa0 secure :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f825c0077f0 tx=0x7f825c000ee0 comp rx=0 tx=0).stop 2024-09-16T03:54:44.320 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:54:44.321 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:54:44.321 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:54:44.321 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "leader", 2024-09-16T03:54:44.321 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 110, 2024-09-16T03:54:44.321 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 174150, 2024-09-16T03:54:44.321 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:54:44.321 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:54:44.321 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T03:54:44.321 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T03:54:44.321 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:54:44.321 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 2, 2024-09-16T03:54:44.321 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:54:44.321 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:54:44.321 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:54:44.322 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:54:44.322 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:54:44.322 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:54:44.322 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:54:44.322 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:54:44.322 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:54:44.322 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:54:44.322 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:54:44.322 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:54:44.322 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:54:44.322 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:54:44.322 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:54:44.322 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:54:44.322 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:54:44.323 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:54:44.323 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:54:44.323 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:54:44.323 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:54:44.323 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:54:44.323 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:54:44.323 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:54:44.323 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:54:44.323 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:54:44.323 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:54:44.323 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:54:44.323 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:54:44.323 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:54:44.323 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:54:44.324 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:54:44.324 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:54:44.324 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:54:44.324 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:54:44.324 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:54:44.324 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:54:44.324 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:54:44.324 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:54:44.324 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:54:44.324 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:54:44.324 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:54:44.324 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:54:44.324 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:54:44.324 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:54:44.325 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:54:44.325 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:54:44.325 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:54:44.325 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:54:44.325 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:54:44.325 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:54:44.325 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:54:44.325 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:54:44.325 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:54:44.325 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:54:44.325 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:54:44.325 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:54:44.325 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:54:44.325 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:54:44.326 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:54:44.326 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:54:44.326 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:54:44.326 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:54:44.326 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:54:44.326 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:54:44.326 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:54:44.326 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:54:44.326 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:54:44.326 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:54:44.326 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:54:44.326 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:54:44.326 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:54:44.326 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:54:44.327 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:54:44.327 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:54:44.327 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:54:44.327 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:54:44.327 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:54:44.327 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:54:44.327 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:54:44.327 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:54:44.327 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:54:44.327 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:54:44.327 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:54:44.327 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:54:44.327 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:54:44.327 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:54:44.327 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:54:44.328 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:54:44.328 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:54:44.328 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:54:44.328 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:54:44.328 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:54:44.328 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:54:44.328 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:54:44.328 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:54:44.328 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:54:44.328 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:54:44.328 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:54:44.328 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:54:44.328 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:54:44.328 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:54:44.329 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:54:44.329 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:54:44.329 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:54:44.329 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:54:44.329 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:54:44.329 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:54:44.329 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:54:44.329 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:54:44.329 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:54:44.329 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:54:44.329 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:54:44.329 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:54:44.329 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:54:44.329 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:54:44.329 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:54:44.330 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:54:44.330 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:54:44.330 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:54:44.330 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:54:44.330 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:54:44.330 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:54:44.330 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:54:44.330 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:54:44.330 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:54:44.330 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:54:44.330 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:54:44.330 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:54:44.330 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:54:44.330 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:54:44.331 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:54:44.331 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:54:44.331 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:54:44.331 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:54:44.331 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:54:44.331 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:54:44.331 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:54:44.331 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:54:44.331 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:54:44.331 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:54:44.331 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T03:54:44.331 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:54:44.331 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:54:44.331 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:54:44.332 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 6 2024-09-16T03:54:44.332 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:54:44.332 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:54:44.332 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:54:44.332 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:54:44.332 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:54:44.332 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:54:44.332 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 6 2024-09-16T03:54:44.332 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:54:44.332 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:54:44.332 INFO:teuthology.orchestra.run.smithi042.stdout: "mgr": [ 2024-09-16T03:54:44.332 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:54:44.332 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:54:44.332 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:54:44.332 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:54:44.333 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:54:44.333 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:54:44.333 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:54:44.333 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:54:44.333 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:54:44.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.295+0000 7f8267bc7640 10 librados: watch_flush enter 2024-09-16T03:54:44.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.295+0000 7f8267bc7640 10 librados: watch_flush exit 2024-09-16T03:54:44.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.295+0000 7f8267bc7640 20 client.854132.objecter shutdown clearing up homeless session... 2024-09-16T03:54:44.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.295+0000 7f8267bc7640 10 client.854132.objecter successfully canceled tick 2024-09-16T03:54:44.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.295+0000 7f8267bc7640 1 -- 172.21.15.42:0/1653083196 >> v2:172.21.15.154:6812/254845342 conn(0x7f8244037810 msgr2=0x7f8244039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:54:44.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.295+0000 7f8267bc7640 1 --2- 172.21.15.42:0/1653083196 >> v2:172.21.15.154:6812/254845342 conn(0x7f8244037810 0x7f8244039cd0 secure :-1 s=READY pgs=3619 cs=0 l=1 rev1=1 crypto rx=0x7f82580af000 tx=0x7f82580a7070 comp rx=0 tx=0).stop 2024-09-16T03:54:44.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.295+0000 7f8267bc7640 10 monclient: shutdown 2024-09-16T03:54:44.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.295+0000 7f8267bc7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:54:44.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.295+0000 7f8267bc7640 1 -- 172.21.15.42:0/1653083196 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f82600a1330 msgr2=0x7f826011bf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:54:44.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.295+0000 7f8267bc7640 1 --2- 172.21.15.42:0/1653083196 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f82600a1330 0x7f826011bf70 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f825000c420 tx=0x7f825000c8e0 comp rx=0 tx=0).stop 2024-09-16T03:54:44.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.295+0000 7f8266164640 1 -- 172.21.15.42:0/1653083196 reap_dead start 2024-09-16T03:54:44.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.295+0000 7f8267bc7640 1 -- 172.21.15.42:0/1653083196 shutdown_connections 2024-09-16T03:54:44.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.295+0000 7f8267bc7640 1 --2- 172.21.15.42:0/1653083196 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f82600a1330 0x7f826011bf70 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:54:44.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.295+0000 7f8267bc7640 1 -- 172.21.15.42:0/1653083196 >> 172.21.15.42:0/1653083196 conn(0x7f82600a2420 msgr2=0x7f8260136320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:54:44.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.295+0000 7f8267bc7640 1 -- 172.21.15.42:0/1653083196 shutdown_connections 2024-09-16T03:54:44.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.295+0000 7f8267bc7640 1 -- 172.21.15.42:0/1653083196 wait complete. 2024-09-16T03:54:44.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.295+0000 7f8267bc7640 1 librados: shutdown 2024-09-16T03:54:44.336 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:54:44.336 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T03:54:44.336 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T03:54:44.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.451+0000 7fa07d699640 10 monclient: get_monmap_and_config 2024-09-16T03:54:44.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.451+0000 7fa07d699640 10 monclient: build_initial_monmap 2024-09-16T03:54:44.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.451+0000 7fa07d699640 10 monclient: monmap: 2024-09-16T03:54:44.457 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:54:44.457 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:54:44.457 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:54:44.455799+0000 2024-09-16T03:54:44.457 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:54:44.455799+0000 2024-09-16T03:54:44.457 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:54:44.457 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:54:44.457 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:54:44.458 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:54:44.458 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:54:44.458 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:54:44.458 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:54:44.458 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:54:44.460 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.455+0000 7fa07d699640 1 Processor -- start 2024-09-16T03:54:44.460 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.455+0000 7fa07d699640 1 -- start start 2024-09-16T03:54:44.460 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.455+0000 7fa07d699640 10 monclient: init 2024-09-16T03:54:44.461 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.455+0000 7fa07d699640 10 monclient: _reopen_session rank -1 2024-09-16T03:54:44.461 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.455+0000 7fa07d699640 10 monclient: _add_conns ranks=[4,0,3,2,1] 2024-09-16T03:54:44.462 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.455+0000 7fa07d699640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fa07806db90 0x7fa0781340c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:54:44.462 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.455+0000 7fa07d699640 10 monclient(hunting): picked mon.noname-c con 0x7fa07806db90 addr v2:172.21.15.154:3300/0 2024-09-16T03:54:44.462 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.455+0000 7fa07d699640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa0780a37f0 addr v1:172.21.15.42:6789/0 2024-09-16T03:54:44.463 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.455+0000 7fa07d699640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fa0780a3c00 0x7fa078137520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:54:44.463 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.455+0000 7fa07d699640 10 monclient(hunting): picked mon.noname-a con 0x7fa0780a3c00 addr v2:172.21.15.42:3300/0 2024-09-16T03:54:44.463 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.455+0000 7fa07d699640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa07806ece0 con 0x7fa0780a37f0 2024-09-16T03:54:44.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.455+0000 7fa07d699640 10 monclient(hunting): start opening mon connection 2024-09-16T03:54:44.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.455+0000 7fa07d699640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7fa07806df70 con 0x7fa0780a3c00 2024-09-16T03:54:44.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.455+0000 7fa07d699640 10 monclient(hunting): start opening mon connection 2024-09-16T03:54:44.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.455+0000 7fa07d699640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fa07809bf80 con 0x7fa07806db90 2024-09-16T03:54:44.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.455+0000 7fa07d699640 10 monclient(hunting): _renew_subs 2024-09-16T03:54:44.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.455+0000 7fa07d699640 10 monclient(hunting): authenticate will time out at 1731.425181s 2024-09-16T03:54:44.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.455+0000 7fa0777fe640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fa0780a3c00 0x7fa078137520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:54:44.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.455+0000 7fa0767fc640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fa0780a37f0 0x7fa078136e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:57028/0 (socket says 172.21.15.42:57028) 2024-09-16T03:54:44.465 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.455+0000 7fa0767fc640 1 -- 172.21.15.42:0/256290708 learned_addr learned my addr 172.21.15.42:0/256290708 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:54:44.465 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.455+0000 7fa0777fe640 10 monclient(hunting): get_auth_request con 0x7fa0780a3c00 auth_method 0 2024-09-16T03:54:44.465 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.455+0000 7fa0777fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:54:44.465 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.455+0000 7fa0777fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:54:44.465 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.455+0000 7fa0777fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:54:44.465 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.455+0000 7fa076ffd640 1 --2- 172.21.15.42:0/256290708 >> v2:172.21.15.154:3300/0 conn(0x7fa07806db90 0x7fa0781340c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:54:44.465 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.455+0000 7fa076ffd640 10 monclient(hunting): get_auth_request con 0x7fa07806db90 auth_method 0 2024-09-16T03:54:44.465 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.455+0000 7fa076ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:54:44.465 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.455+0000 7fa076ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:54:44.465 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.455+0000 7fa076ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:54:44.466 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.455+0000 7fa0777fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:54:44.466 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.455+0000 7fa0777fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:54:44.466 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.455+0000 7fa0777fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:54:44.466 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.455+0000 7fa076ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:54:44.466 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.455+0000 7fa076ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:54:44.466 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.455+0000 7fa076ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:54:44.467 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.455+0000 7fa075ffb640 1 -- 172.21.15.42:0/256290708 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2712493596 0 0) 0x7fa07806ece0 con 0x7fa0780a37f0 2024-09-16T03:54:44.468 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.455+0000 7fa075ffb640 10 monclient(hunting): _negotiate 2024-09-16T03:54:44.468 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.455+0000 7fa075ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:54:44.468 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.455+0000 7fa075ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:54:44.468 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.455+0000 7fa075ffb640 10 monclient(hunting): my global_id is 824484 2024-09-16T03:54:44.468 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.455+0000 7fa075ffb640 1 -- 172.21.15.42:0/256290708 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa060003530 con 0x7fa0780a37f0 2024-09-16T03:54:44.468 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.455+0000 7fa0777fe640 10 monclient(hunting): handle_auth_done global_id 824481 payload 274 2024-09-16T03:54:44.468 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.455+0000 7fa0777fe640 1 -- 172.21.15.42:0/256290708 >> v2:172.21.15.154:3300/0 conn(0x7fa07806db90 msgr2=0x7fa0781340c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:54:44.469 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.455+0000 7fa0777fe640 1 --2- 172.21.15.42:0/256290708 >> v2:172.21.15.154:3300/0 conn(0x7fa07806db90 0x7fa0781340c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:54:44.469 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.455+0000 7fa0777fe640 1 -- 172.21.15.42:0/256290708 >> v1:172.21.15.42:6789/0 conn(0x7fa0780a37f0 legacy=0x7fa078136e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:54:44.469 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.455+0000 7fa0777fe640 10 monclient: _finish_hunting 0 2024-09-16T03:54:44.469 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.455+0000 7fa0777fe640 1 monclient: found mon.noname-a 2024-09-16T03:54:44.469 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.455+0000 7fa0777fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:54:44.469 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.455+0000 7fa0777fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:54:44.469 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.455+0000 7fa0777fe640 1 -- 172.21.15.42:0/256290708 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa078134780 con 0x7fa0780a3c00 2024-09-16T03:54:44.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.459+0000 7fa0777fe640 1 --2- 172.21.15.42:0/256290708 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa0780a3c00 0x7fa078137520 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fa0780a76f0 tx=0x7fa07001d220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5eec3ec6546b20d8 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:54:44.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.459+0000 7fa075ffb640 1 -- 172.21.15.42:0/256290708 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0700b2070 con 0x7fa0780a3c00 2024-09-16T03:54:44.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.459+0000 7fa075ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:54:44.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.459+0000 7fa075ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:54:44.479 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.459+0000 7fa075ffb640 10 monclient: dump: 2024-09-16T03:54:44.479 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:54:44.479 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:54:44.479 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:44.479 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:44.479 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:54:44.479 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:54:44.479 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:54:44.479 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:54:44.479 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:54:44.479 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:54:44.479 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.459+0000 7fa075ffb640 10 monclient: _finish_auth 0 2024-09-16T03:54:44.479 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.459+0000 7fa075ffb640 10 monclient: _check_auth_tickets 2024-09-16T03:54:44.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.459+0000 7fa075ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:54:44.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.459+0000 7fa075ffb640 1 -- 172.21.15.42:0/256290708 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa0700a58a0 con 0x7fa0780a3c00 2024-09-16T03:54:44.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.459+0000 7fa075ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T03:54:44.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.459+0000 7fa075ffb640 1 -- 172.21.15.42:0/256290708 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0700a5c70 con 0x7fa0780a3c00 2024-09-16T03:54:44.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.459+0000 7fa07d699640 5 monclient: authenticate success, global_id 824481 2024-09-16T03:54:44.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.459+0000 7fa07d699640 10 monclient: get_monmap_and_config success 2024-09-16T03:54:44.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.459+0000 7fa07d699640 4 set_mon_vals no callback set 2024-09-16T03:54:44.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.459+0000 7fa075ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:54:44.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.459+0000 7fa075ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:54:44.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.459+0000 7fa075ffb640 10 monclient: dump: 2024-09-16T03:54:44.480 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:54:44.481 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:54:44.481 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:44.481 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:44.481 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:54:44.481 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:54:44.481 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:54:44.481 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:54:44.481 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:54:44.481 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:54:44.482 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.459+0000 7fa07d699640 10 monclient: shutdown 2024-09-16T03:54:44.482 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.459+0000 7fa07d699640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:54:44.482 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.459+0000 7fa07d699640 1 -- 172.21.15.42:0/256290708 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa0780a3c00 msgr2=0x7fa078137520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:54:44.482 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.459+0000 7fa07d699640 1 --2- 172.21.15.42:0/256290708 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa0780a3c00 0x7fa078137520 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fa0780a76f0 tx=0x7fa07001d220 comp rx=0 tx=0).stop 2024-09-16T03:54:44.482 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.459+0000 7fa07d699640 1 -- 172.21.15.42:0/256290708 shutdown_connections 2024-09-16T03:54:44.483 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.459+0000 7fa07d699640 1 --2- 172.21.15.42:0/256290708 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa0780a3c00 0x7fa078137520 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:54:44.483 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.459+0000 7fa07d699640 1 --2- 172.21.15.42:0/256290708 >> v2:172.21.15.154:3300/0 conn(0x7fa07806db90 0x7fa0781340c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:54:44.483 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.459+0000 7fa07d699640 1 -- 172.21.15.42:0/256290708 >> 172.21.15.42:0/256290708 conn(0x7fa0780a9fb0 msgr2=0x7fa0780aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:54:44.483 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.459+0000 7fa07d699640 1 -- 172.21.15.42:0/256290708 shutdown_connections 2024-09-16T03:54:44.483 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.459+0000 7fa07d699640 1 -- 172.21.15.42:0/256290708 wait complete. 2024-09-16T03:54:44.483 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.459+0000 7fa07d699640 10 monclient: build_initial_monmap 2024-09-16T03:54:44.483 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.459+0000 7fa07d699640 10 monclient: monmap: 2024-09-16T03:54:44.483 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:54:44.483 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:54:44.484 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:54:44.484 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:54:44.484 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:54:44.484 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:54:44.484 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:54:44.484 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:54:44.484 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:54:44.484 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:54:44.484 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.459+0000 7fa07d699640 1 librados: starting msgr at 2024-09-16T03:54:44.484 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.459+0000 7fa07d699640 1 librados: starting objecter 2024-09-16T03:54:44.485 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.459+0000 7fa07d699640 1 Processor -- start 2024-09-16T03:54:44.485 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.459+0000 7fa07d699640 1 -- start start 2024-09-16T03:54:44.485 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.459+0000 7fa07d699640 1 librados: setting wanted keys 2024-09-16T03:54:44.485 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.459+0000 7fa07d699640 1 librados: calling monclient init 2024-09-16T03:54:44.485 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.459+0000 7fa07d699640 10 monclient: init 2024-09-16T03:54:44.485 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.459+0000 7fa07d699640 10 monclient: _reopen_session rank -1 2024-09-16T03:54:44.485 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.459+0000 7fa07d699640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:54:44.485 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.459+0000 7fa07d699640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa07806db90 0x7fa0781cebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:54:44.485 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.459+0000 7fa07d699640 10 monclient(hunting): picked mon.noname-c con 0x7fa07806db90 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:54:44.485 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.459+0000 7fa07d699640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa0780a37f0 0x7fa07813e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:54:44.485 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.459+0000 7fa07d699640 10 monclient(hunting): picked mon.noname-b con 0x7fa0780a37f0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:54:44.485 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.459+0000 7fa07d699640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa0780a3c00 0x7fa07813e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:54:44.485 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.459+0000 7fa07d699640 10 monclient(hunting): picked mon.noname-a con 0x7fa0780a3c00 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:54:44.485 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.459+0000 7fa07d699640 10 monclient(hunting): start opening mon connection 2024-09-16T03:54:44.486 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.459+0000 7fa07d699640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fa0780a40d0 con 0x7fa0780a3c00 2024-09-16T03:54:44.486 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.459+0000 7fa07d699640 10 monclient(hunting): start opening mon connection 2024-09-16T03:54:44.486 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.459+0000 7fa07d699640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa078137c70 con 0x7fa0780a37f0 2024-09-16T03:54:44.486 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.459+0000 7fa07d699640 10 monclient(hunting): start opening mon connection 2024-09-16T03:54:44.486 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.459+0000 7fa07d699640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fa0781419d0 con 0x7fa07806db90 2024-09-16T03:54:44.486 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.459+0000 7fa07d699640 10 monclient(hunting): _renew_subs 2024-09-16T03:54:44.486 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.459+0000 7fa07d699640 10 monclient(hunting): authenticate will time out at 1731.430934s 2024-09-16T03:54:44.486 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.459+0000 7fa076ffd640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa07806db90 0x7fa0781cebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:54:44.486 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.459+0000 7fa0777fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa0780a3c00 0x7fa07813e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:54:44.486 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.459+0000 7fa0777fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa0780a3c00 0x7fa07813e6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:41556/0 (socket says 172.21.15.42:41556) 2024-09-16T03:54:44.486 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.459+0000 7fa0777fe640 1 -- 172.21.15.42:0/3087823702 learned_addr learned my addr 172.21.15.42:0/3087823702 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:54:44.486 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.459+0000 7fa076ffd640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa07806db90 0x7fa0781cebd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:32780/0 (socket says 172.21.15.42:32780) 2024-09-16T03:54:44.486 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.459+0000 7fa0777fe640 10 monclient(hunting): get_auth_request con 0x7fa0780a3c00 auth_method 0 2024-09-16T03:54:44.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.459+0000 7fa0777fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:54:44.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.459+0000 7fa0777fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:54:44.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.459+0000 7fa0777fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:54:44.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.459+0000 7fa076ffd640 10 monclient(hunting): get_auth_request con 0x7fa07806db90 auth_method 0 2024-09-16T03:54:44.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.459+0000 7fa076ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:54:44.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.459+0000 7fa076ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:54:44.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.459+0000 7fa076ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:54:44.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.459+0000 7fa0777fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:54:44.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.459+0000 7fa0777fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:54:44.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.459+0000 7fa0777fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:54:44.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.459+0000 7fa076ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:54:44.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.459+0000 7fa076ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:54:44.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.459+0000 7fa076ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:54:44.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.459+0000 7fa0767fc640 1 --2- 172.21.15.42:0/3087823702 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa0780a37f0 0x7fa07813e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:54:44.488 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.459+0000 7fa0767fc640 10 monclient(hunting): get_auth_request con 0x7fa0780a37f0 auth_method 0 2024-09-16T03:54:44.488 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.459+0000 7fa0767fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:54:44.488 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.459+0000 7fa0767fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:54:44.488 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.459+0000 7fa0767fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:54:44.488 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.459+0000 7fa0777fe640 10 monclient(hunting): handle_auth_done global_id 824487 payload 824 2024-09-16T03:54:44.488 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.459+0000 7fa0777fe640 1 -- 172.21.15.42:0/3087823702 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa07806db90 msgr2=0x7fa0781cebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:54:44.488 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.459+0000 7fa0777fe640 1 --2- 172.21.15.42:0/3087823702 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa07806db90 0x7fa0781cebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:54:44.488 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.459+0000 7fa0777fe640 1 -- 172.21.15.42:0/3087823702 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa0780a37f0 msgr2=0x7fa07813e1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:54:44.488 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.459+0000 7fa0777fe640 1 --2- 172.21.15.42:0/3087823702 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa0780a37f0 0x7fa07813e1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:54:44.488 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.459+0000 7fa0777fe640 10 monclient: _finish_hunting 0 2024-09-16T03:54:44.488 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.459+0000 7fa0777fe640 1 monclient: found mon.noname-a 2024-09-16T03:54:44.489 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.459+0000 7fa0777fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:54:44.489 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.459+0000 7fa0777fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:54:44.489 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.459+0000 7fa0777fe640 1 -- 172.21.15.42:0/3087823702 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0781cfac0 con 0x7fa0780a3c00 2024-09-16T03:54:44.489 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.459+0000 7fa0777fe640 1 --2- 172.21.15.42:0/3087823702 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa0780a3c00 0x7fa07813e6f0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fa0700b0040 tx=0x7fa070019560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:54:44.489 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.459+0000 7fa05f7fe640 10 client.?.objecter ms_handle_connect 0x7fa0780a3c00 2024-09-16T03:54:44.489 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.459+0000 7fa05f7fe640 10 client.?.objecter resend_mon_ops 2024-09-16T03:54:44.489 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.459+0000 7fa05f7fe640 1 -- 172.21.15.42:0/3087823702 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0700bc030 con 0x7fa0780a3c00 2024-09-16T03:54:44.489 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.459+0000 7fa05f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:54:44.489 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.459+0000 7fa05f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:54:44.489 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.459+0000 7fa05f7fe640 10 monclient: dump: 2024-09-16T03:54:44.489 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:54:44.489 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:54:44.489 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:44.490 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:44.490 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:54:44.490 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:54:44.490 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:54:44.490 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:54:44.490 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:54:44.490 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:54:44.490 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.459+0000 7fa05f7fe640 10 monclient: _finish_auth 0 2024-09-16T03:54:44.490 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.459+0000 7fa05f7fe640 10 monclient: _check_auth_tickets 2024-09-16T03:54:44.490 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.459+0000 7fa05f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:54:44.490 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.459+0000 7fa05f7fe640 1 -- 172.21.15.42:0/3087823702 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa0700a56a0 con 0x7fa0780a3c00 2024-09-16T03:54:44.490 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.459+0000 7fa07d699640 5 monclient: authenticate success, global_id 824487 2024-09-16T03:54:44.490 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.459+0000 7fa07d699640 10 monclient: _renew_subs 2024-09-16T03:54:44.490 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.459+0000 7fa07d699640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:54:44.490 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.459+0000 7fa07d699640 1 -- 172.21.15.42:0/3087823702 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa078139c50 con 0x7fa0780a3c00 2024-09-16T03:54:44.491 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.459+0000 7fa05f7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:54:44.491 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.459+0000 7fa05f7fe640 1 -- 172.21.15.42:0/3087823702 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0700b2040 con 0x7fa0780a3c00 2024-09-16T03:54:44.491 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.459+0000 7fa05f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:54:44.491 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.459+0000 7fa05f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:54:44.491 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.459+0000 7fa05f7fe640 10 monclient: dump: 2024-09-16T03:54:44.491 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:54:44.491 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:54:44.491 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:44.491 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:44.491 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:54:44.491 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:54:44.491 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:54:44.491 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:54:44.492 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:54:44.492 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:54:44.492 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.463+0000 7fa07d699640 10 client.824487.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:54:44.492 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.463+0000 7fa07d699640 10 monclient: _renew_subs 2024-09-16T03:54:44.492 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.463+0000 7fa07d699640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:54:44.492 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.459+0000 7fa0757fa640 4 set_mon_vals no callback set 2024-09-16T03:54:44.493 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.463+0000 7fa07d699640 1 -- 172.21.15.42:0/3087823702 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0780a3210 con 0x7fa0780a3c00 2024-09-16T03:54:44.493 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.463+0000 7fa07d699640 1 librados: init done 2024-09-16T03:54:44.493 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.463+0000 7fa07d699640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:54:44.493 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.463+0000 7fa07d699640 1 --2- 172.21.15.42:0/3087823702 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa0781384c0 0x7fa0780a1d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:54:44.493 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.463+0000 7fa07d699640 10 monclient(hunting): start opening mon connection 2024-09-16T03:54:44.493 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.463+0000 7fa07d699640 1 -- 172.21.15.42:0/3087823702 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fa0781340c0 con 0x7fa0781384c0 2024-09-16T03:54:44.493 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.463+0000 7fa076ffd640 1 --2- 172.21.15.42:0/3087823702 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa0781384c0 0x7fa0780a1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:54:44.493 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.463+0000 7fa076ffd640 10 monclient: get_auth_request con 0x7fa0781384c0 auth_method 0 2024-09-16T03:54:44.493 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.463+0000 7fa076ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:54:44.493 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.463+0000 7fa076ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:54:44.493 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.463+0000 7fa076ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:54:44.493 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.463+0000 7fa076ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:54:44.494 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.463+0000 7fa076ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:54:44.494 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.463+0000 7fa076ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:54:44.494 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.463+0000 7fa05f7fe640 1 -- 172.21.15.42:0/3087823702 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fa0700bf070 con 0x7fa0780a3c00 2024-09-16T03:54:44.494 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.463+0000 7fa076ffd640 10 monclient(hunting): handle_auth_done global_id 854138 payload 824 2024-09-16T03:54:44.494 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.463+0000 7fa076ffd640 1 -- 172.21.15.42:0/3087823702 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa078139470 con 0x7fa0781384c0 2024-09-16T03:54:44.494 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.463+0000 7fa05f7fe640 1 --2- 172.21.15.42:0/3087823702 >> v2:172.21.15.154:6812/254845342 conn(0x7fa064037810 0x7fa064039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:54:44.494 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.463+0000 7fa05f7fe640 1 -- 172.21.15.42:0/3087823702 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(386..386 src has 286..386) ==== 5713+0+0 (secure 0 0 0) 0x7fa0700e36e0 con 0x7fa0780a3c00 2024-09-16T03:54:44.494 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.463+0000 7fa05f7fe640 10 client.824487.objecter ms_dispatch 0x7fa078000ba0 osd_map(386..386 src has 286..386) 2024-09-16T03:54:44.494 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.463+0000 7fa076ffd640 1 --2- 172.21.15.42:0/3087823702 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa0781384c0 0x7fa0780a1d70 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fa068011900 tx=0x7fa068011dc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:54:44.494 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.463+0000 7fa05f7fe640 3 client.824487.objecter handle_osd_map got epochs [386,386] > 0 2024-09-16T03:54:44.494 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.463+0000 7fa05f7fe640 3 client.824487.objecter handle_osd_map decoding full epoch 386 2024-09-16T03:54:44.494 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.463+0000 7fa05f7fe640 20 client.824487.objecter dump_active .. 0 homeless 2024-09-16T03:54:44.495 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.463+0000 7fa05f7fe640 10 client.824487.objecter ms_handle_connect 0x7fa0781384c0 2024-09-16T03:54:44.495 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.463+0000 7fa05f7fe640 10 client.824487.objecter resend_mon_ops 2024-09-16T03:54:44.495 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.463+0000 7fa05f7fe640 1 -- 172.21.15.42:0/3087823702 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa068019070 con 0x7fa0781384c0 2024-09-16T03:54:44.495 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.463+0000 7fa05f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:54:44.495 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.463+0000 7fa05f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:54:44.495 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.463+0000 7fa05f7fe640 10 monclient: dump: 2024-09-16T03:54:44.495 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:54:44.495 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:54:44.495 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:44.495 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:44.495 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:54:44.495 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:54:44.495 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:54:44.496 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:54:44.496 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:54:44.496 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:54:44.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.463+0000 7fa0767fc640 1 --2- 172.21.15.42:0/3087823702 >> v2:172.21.15.154:6812/254845342 conn(0x7fa064037810 0x7fa064039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:54:44.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.463+0000 7fa0767fc640 10 monclient: get_auth_request con 0x7fa064037810 auth_method 0 2024-09-16T03:54:44.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.463+0000 7fa05f7fe640 1 -- 172.21.15.42:0/3087823702 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fa068004300 con 0x7fa0781384c0 2024-09-16T03:54:44.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.463+0000 7fa05f7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:54:44.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.463+0000 7fa05f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:54:44.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.463+0000 7fa05f7fe640 1 -- 172.21.15.42:0/3087823702 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa0781384c0 msgr2=0x7fa0780a1d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:54:44.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.463+0000 7fa05f7fe640 1 --2- 172.21.15.42:0/3087823702 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa0781384c0 0x7fa0780a1d70 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fa068011900 tx=0x7fa068011dc0 comp rx=0 tx=0).stop 2024-09-16T03:54:44.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.463+0000 7fa05f7fe640 1 -- 172.21.15.42:0/3087823702 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa0781384c0 msgr2=0x7fa0780a1d70 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:54:44.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.463+0000 7fa05f7fe640 1 --2- 172.21.15.42:0/3087823702 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa0781384c0 0x7fa0780a1d70 secure :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fa068011900 tx=0x7fa068011dc0 comp rx=0 tx=0).stop 2024-09-16T03:54:44.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.463+0000 7fa0767fc640 1 --2- 172.21.15.42:0/3087823702 >> v2:172.21.15.154:6812/254845342 conn(0x7fa064037810 0x7fa064039cd0 secure :-1 s=READY pgs=3620 cs=0 l=1 rev1=1 crypto rx=0x7fa06c0011f0 tx=0x7fa06c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:54:44.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.463+0000 7fa05f7fe640 10 client.824487.objecter ms_handle_connect 0x7fa064037810 2024-09-16T03:54:44.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.467+0000 7fa07d699640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:54:44.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.467+0000 7fa07d699640 1 --2- 172.21.15.42:0/3087823702 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa07802c650 0x7fa07802ca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:54:44.498 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.467+0000 7fa07d699640 10 monclient(hunting): start opening mon connection 2024-09-16T03:54:44.498 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.467+0000 7fa07d699640 1 -- 172.21.15.42:0/3087823702 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fa07809de20 con 0x7fa07802c650 2024-09-16T03:54:44.498 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.471+0000 7fa076ffd640 1 --2- 172.21.15.42:0/3087823702 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa07802c650 0x7fa07802ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:54:44.498 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.471+0000 7fa076ffd640 10 monclient: get_auth_request con 0x7fa07802c650 auth_method 0 2024-09-16T03:54:44.498 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.471+0000 7fa076ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:54:44.498 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.471+0000 7fa076ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:54:44.498 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.471+0000 7fa076ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:54:44.498 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.471+0000 7fa076ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:54:44.498 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.471+0000 7fa076ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:54:44.498 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.471+0000 7fa076ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:54:44.498 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.471+0000 7fa076ffd640 10 monclient(hunting): handle_auth_done global_id 854141 payload 824 2024-09-16T03:54:44.498 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.471+0000 7fa076ffd640 1 -- 172.21.15.42:0/3087823702 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fa078068e40 con 0x7fa07802c650 2024-09-16T03:54:44.498 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.471+0000 7fa076ffd640 1 --2- 172.21.15.42:0/3087823702 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa07802c650 0x7fa07802ca30 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fa068011900 tx=0x7fa0680046d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:54:44.499 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.471+0000 7fa05f7fe640 10 client.824487.objecter ms_handle_connect 0x7fa07802c650 2024-09-16T03:54:44.499 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.471+0000 7fa05f7fe640 10 client.824487.objecter resend_mon_ops 2024-09-16T03:54:44.499 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.471+0000 7fa05f7fe640 1 -- 172.21.15.42:0/3087823702 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa068019040 con 0x7fa07802c650 2024-09-16T03:54:44.499 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.471+0000 7fa05f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:54:44.499 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.471+0000 7fa05f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:54:44.499 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.471+0000 7fa05f7fe640 10 monclient: dump: 2024-09-16T03:54:44.499 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:54:44.499 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:54:44.499 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:44.499 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:44.499 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:54:44.499 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:54:44.499 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:54:44.500 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:54:44.500 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:54:44.500 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:54:44.500 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.471+0000 7fa05f7fe640 1 -- 172.21.15.42:0/3087823702 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4545 (secure 0 0 0) 0x7fa06801e450 con 0x7fa07802c650 2024-09-16T03:54:44.500 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.471+0000 7fa05f7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:54:44.500 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.471+0000 7fa05f7fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:54:44.500 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.471+0000 7fa05f7fe640 1 -- 172.21.15.42:0/3087823702 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa07802c650 msgr2=0x7fa07802ca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:54:44.500 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.471+0000 7fa05f7fe640 1 --2- 172.21.15.42:0/3087823702 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa07802c650 0x7fa07802ca30 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fa068011900 tx=0x7fa0680046d0 comp rx=0 tx=0).stop 2024-09-16T03:54:44.500 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.471+0000 7fa05f7fe640 1 -- 172.21.15.42:0/3087823702 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa07802c650 msgr2=0x7fa07802ca30 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:54:44.500 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.471+0000 7fa05f7fe640 1 --2- 172.21.15.42:0/3087823702 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa07802c650 0x7fa07802ca30 secure :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fa068011900 tx=0x7fa0680046d0 comp rx=0 tx=0).stop 2024-09-16T03:54:44.501 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:54:44.501 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:54:44.501 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:54:44.501 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T03:54:44.501 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 110, 2024-09-16T03:54:44.501 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 17907, 2024-09-16T03:54:44.501 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:54:44.501 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:54:44.501 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T03:54:44.501 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T03:54:44.501 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:54:44.501 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 2, 2024-09-16T03:54:44.502 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:54:44.502 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:54:44.502 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:54:44.502 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:54:44.503 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:54:44.503 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:54:44.503 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:54:44.503 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:54:44.503 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:54:44.503 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:54:44.503 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:54:44.503 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:54:44.504 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:54:44.504 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:54:44.504 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:54:44.504 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:54:44.504 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:54:44.504 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:54:44.504 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:54:44.504 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:54:44.504 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:54:44.504 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:54:44.504 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:54:44.504 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:54:44.504 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:54:44.505 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:54:44.505 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:54:44.505 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:54:44.505 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:54:44.505 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:54:44.505 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:54:44.505 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:54:44.505 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:54:44.505 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:54:44.505 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:54:44.505 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:54:44.505 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:54:44.505 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:54:44.505 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:54:44.506 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:54:44.506 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:54:44.506 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:54:44.506 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:54:44.506 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:54:44.506 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:54:44.506 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:54:44.506 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:54:44.506 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:54:44.506 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:54:44.506 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:54:44.506 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:54:44.506 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:54:44.507 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:54:44.507 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:54:44.507 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:54:44.507 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:54:44.507 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:54:44.507 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:54:44.507 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:54:44.507 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:54:44.507 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:54:44.507 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:54:44.507 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:54:44.507 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:54:44.507 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:54:44.508 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:54:44.508 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:54:44.508 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:54:44.508 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:54:44.508 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:54:44.508 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:54:44.508 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:54:44.508 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:54:44.508 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:54:44.508 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:54:44.508 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:54:44.508 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:54:44.509 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:54:44.509 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:54:44.509 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:54:44.509 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:54:44.509 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:54:44.509 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:54:44.509 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:54:44.509 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:54:44.509 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:54:44.509 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:54:44.509 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:54:44.509 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:54:44.509 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:54:44.509 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:54:44.510 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:54:44.510 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:54:44.510 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:54:44.510 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:54:44.510 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:54:44.510 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:54:44.510 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:54:44.510 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:54:44.510 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:54:44.510 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:54:44.510 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:54:44.510 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:54:44.510 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:54:44.510 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:54:44.511 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:54:44.511 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:54:44.511 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:54:44.511 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:54:44.511 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:54:44.511 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:54:44.511 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:54:44.511 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:54:44.511 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:54:44.511 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:54:44.511 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:54:44.511 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:54:44.511 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:54:44.512 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:54:44.512 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:54:44.512 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:54:44.512 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:54:44.512 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:54:44.512 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:54:44.512 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:54:44.512 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:54:44.512 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:54:44.512 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:54:44.513 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:54:44.513 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:54:44.513 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:54:44.513 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:54:44.513 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:54:44.513 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:54:44.513 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:54:44.513 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:54:44.514 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:54:44.514 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:54:44.514 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:54:44.514 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:54:44.514 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:54:44.514 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:54:44.514 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:54:44.514 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:54:44.514 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:54:44.514 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:54:44.515 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:54:44.515 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:54:44.515 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:54:44.515 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:54:44.515 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:54:44.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.471+0000 7fa07d699640 10 librados: watch_flush enter 2024-09-16T03:54:44.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.471+0000 7fa07d699640 10 librados: watch_flush exit 2024-09-16T03:54:44.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.471+0000 7fa07d699640 20 client.824487.objecter shutdown clearing up homeless session... 2024-09-16T03:54:44.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.471+0000 7fa07d699640 10 client.824487.objecter successfully canceled tick 2024-09-16T03:54:44.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.471+0000 7fa07d699640 1 -- 172.21.15.42:0/3087823702 >> v2:172.21.15.154:6812/254845342 conn(0x7fa064037810 msgr2=0x7fa064039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:54:44.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.471+0000 7fa07d699640 1 --2- 172.21.15.42:0/3087823702 >> v2:172.21.15.154:6812/254845342 conn(0x7fa064037810 0x7fa064039cd0 secure :-1 s=READY pgs=3620 cs=0 l=1 rev1=1 crypto rx=0x7fa06c0011f0 tx=0x7fa06c009040 comp rx=0 tx=0).stop 2024-09-16T03:54:44.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.471+0000 7fa07d699640 10 monclient: shutdown 2024-09-16T03:54:44.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.471+0000 7fa07d699640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:54:44.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.471+0000 7fa07d699640 1 -- 172.21.15.42:0/3087823702 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa0780a3c00 msgr2=0x7fa07813e6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:54:44.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.471+0000 7fa07d699640 1 --2- 172.21.15.42:0/3087823702 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa0780a3c00 0x7fa07813e6f0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fa0700b0040 tx=0x7fa070019560 comp rx=0 tx=0).stop 2024-09-16T03:54:44.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.471+0000 7fa0777fe640 1 -- 172.21.15.42:0/3087823702 reap_dead start 2024-09-16T03:54:44.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.471+0000 7fa07d699640 1 -- 172.21.15.42:0/3087823702 shutdown_connections 2024-09-16T03:54:44.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.471+0000 7fa07d699640 1 --2- 172.21.15.42:0/3087823702 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa0780a3c00 0x7fa07813e6f0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:54:44.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.471+0000 7fa07d699640 1 -- 172.21.15.42:0/3087823702 >> 172.21.15.42:0/3087823702 conn(0x7fa0780a9fb0 msgr2=0x7fa0781369f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:54:44.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.471+0000 7fa07d699640 1 -- 172.21.15.42:0/3087823702 shutdown_connections 2024-09-16T03:54:44.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.471+0000 7fa07d699640 1 -- 172.21.15.42:0/3087823702 wait complete. 2024-09-16T03:54:44.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.471+0000 7fa07d699640 1 librados: shutdown 2024-09-16T03:54:44.518 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:54:44.519 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T03:54:44.519 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T03:54:44.640 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.635+0000 7f4797983640 10 monclient: get_monmap_and_config 2024-09-16T03:54:44.640 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.635+0000 7f4797983640 10 monclient: build_initial_monmap 2024-09-16T03:54:44.640 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.635+0000 7f4797983640 10 monclient: monmap: 2024-09-16T03:54:44.640 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:54:44.640 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:54:44.640 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:54:44.638813+0000 2024-09-16T03:54:44.640 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:54:44.638813+0000 2024-09-16T03:54:44.640 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:54:44.641 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:54:44.641 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:54:44.641 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:54:44.641 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:54:44.641 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:54:44.641 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:54:44.641 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:54:44.643 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.635+0000 7f4797983640 1 Processor -- start 2024-09-16T03:54:44.643 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.635+0000 7f4797983640 1 -- start start 2024-09-16T03:54:44.643 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.635+0000 7f4797983640 10 monclient: init 2024-09-16T03:54:44.643 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.639+0000 7f4797983640 10 monclient: _reopen_session rank -1 2024-09-16T03:54:44.643 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.639+0000 7f4797983640 10 monclient: _add_conns ranks=[3,0,1,4,2] 2024-09-16T03:54:44.643 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.639+0000 7f4797983640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f47900a1330 0x7f47900a1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:54:44.644 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.639+0000 7f4797983640 10 monclient(hunting): picked mon.noname-a con 0x7f47900a1330 addr v2:172.21.15.42:3300/0 2024-09-16T03:54:44.644 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.639+0000 7f4797983640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f479006dac0 addr v1:172.21.15.42:6789/0 2024-09-16T03:54:44.644 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.639+0000 7f4797983640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f47900a1c50 addr v1:172.21.15.154:6789/0 2024-09-16T03:54:44.644 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.639+0000 7f4797983640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4790135f20 con 0x7f479006dac0 2024-09-16T03:54:44.644 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.639+0000 7f4797983640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f479009c090 con 0x7f47900a1c50 2024-09-16T03:54:44.644 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.639+0000 7f4797983640 10 monclient(hunting): start opening mon connection 2024-09-16T03:54:44.644 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.639+0000 7f4797983640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f479009f610 con 0x7f47900a1330 2024-09-16T03:54:44.644 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.639+0000 7f4797983640 10 monclient(hunting): _renew_subs 2024-09-16T03:54:44.644 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.639+0000 7f4797983640 10 monclient(hunting): authenticate will time out at 1731.608281s 2024-09-16T03:54:44.644 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.639+0000 7f479571f640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f47900a1330 0x7f47900a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:54:44.644 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.639+0000 7f479571f640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f47900a1330 0x7f47900a1710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:41566/0 (socket says 172.21.15.42:41566) 2024-09-16T03:54:44.644 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.639+0000 7f479571f640 1 -- 172.21.15.42:0/1060638877 learned_addr learned my addr 172.21.15.42:0/1060638877 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:54:44.644 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.639+0000 7f479571f640 10 monclient(hunting): get_auth_request con 0x7f47900a1330 auth_method 0 2024-09-16T03:54:44.644 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.639+0000 7f479571f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:54:44.645 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.639+0000 7f479571f640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:54:44.645 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.639+0000 7f479571f640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:54:44.645 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.639+0000 7f479571f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:54:44.645 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.639+0000 7f479571f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:54:44.645 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.639+0000 7f479571f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:54:44.646 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.639+0000 7f479571f640 10 monclient(hunting): handle_auth_done global_id 824490 payload 274 2024-09-16T03:54:44.646 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.639+0000 7f479571f640 1 -- 172.21.15.42:0/1060638877 >> v1:172.21.15.154:6789/0 conn(0x7f47900a1c50 legacy=0x7f4790135810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:54:44.646 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.639+0000 7f479571f640 1 -- 172.21.15.42:0/1060638877 >> v1:172.21.15.42:6789/0 conn(0x7f479006dac0 legacy=0x7f4790135100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:54:44.646 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.639+0000 7f479571f640 10 monclient: _finish_hunting 0 2024-09-16T03:54:44.646 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.639+0000 7f479571f640 1 monclient: found mon.noname-a 2024-09-16T03:54:44.646 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.639+0000 7f479571f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:54:44.646 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.639+0000 7f479571f640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:54:44.646 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.639+0000 7f479571f640 1 -- 172.21.15.42:0/1060638877 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f479009f8f0 con 0x7f47900a1330 2024-09-16T03:54:44.646 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.639+0000 7f4787fff640 1 -- 172.21.15.42:0/1060638877 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 16679703 0 0) 0x7f4790135f20 con 0x7f479006dac0 2024-09-16T03:54:44.646 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.639+0000 7f4787fff640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T03:54:44.653 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.639+0000 7f479571f640 1 --2- 172.21.15.42:0/1060638877 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f47900a1330 0x7f47900a1710 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f478c009870 tx=0x7f478c01c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac14ca8461a71581 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:54:44.653 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.639+0000 7f4787fff640 1 -- 172.21.15.42:0/1060638877 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f478c0b1070 con 0x7f47900a1330 2024-09-16T03:54:44.653 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.639+0000 7f4787fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:54:44.653 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.639+0000 7f4787fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:54:44.653 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.639+0000 7f4787fff640 10 monclient: dump: 2024-09-16T03:54:44.654 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:54:44.654 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:54:44.654 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:44.654 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:44.654 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:54:44.654 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:54:44.654 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:54:44.654 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:54:44.654 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:54:44.654 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:54:44.654 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.639+0000 7f4787fff640 10 monclient: _finish_auth 0 2024-09-16T03:54:44.654 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.639+0000 7f4787fff640 10 monclient: _check_auth_tickets 2024-09-16T03:54:44.654 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.639+0000 7f4787fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:54:44.654 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.639+0000 7f4797983640 5 monclient: authenticate success, global_id 824490 2024-09-16T03:54:44.655 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.639+0000 7f4787fff640 1 -- 172.21.15.42:0/1060638877 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f478c0ac630 con 0x7f47900a1330 2024-09-16T03:54:44.655 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.639+0000 7f4787fff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:54:44.655 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.639+0000 7f4797983640 10 monclient: get_monmap_and_config success 2024-09-16T03:54:44.655 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.639+0000 7f4797983640 4 set_mon_vals no callback set 2024-09-16T03:54:44.655 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.639+0000 7f4787fff640 1 -- 172.21.15.42:0/1060638877 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f478c0a8050 con 0x7f47900a1330 2024-09-16T03:54:44.655 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.639+0000 7f4787fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:54:44.655 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.639+0000 7f4787fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:54:44.655 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.639+0000 7f4787fff640 10 monclient: dump: 2024-09-16T03:54:44.655 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:54:44.655 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:54:44.655 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:44.655 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:44.655 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:54:44.656 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:54:44.656 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:54:44.656 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:54:44.656 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:54:44.656 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:54:44.656 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.643+0000 7f4797983640 10 monclient: shutdown 2024-09-16T03:54:44.656 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.643+0000 7f4797983640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:54:44.656 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.643+0000 7f4797983640 1 -- 172.21.15.42:0/1060638877 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f47900a1330 msgr2=0x7f47900a1710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:54:44.656 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.643+0000 7f4797983640 1 --2- 172.21.15.42:0/1060638877 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f47900a1330 0x7f47900a1710 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f478c009870 tx=0x7f478c01c440 comp rx=0 tx=0).stop 2024-09-16T03:54:44.656 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.643+0000 7f4797983640 1 -- 172.21.15.42:0/1060638877 shutdown_connections 2024-09-16T03:54:44.656 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.643+0000 7f4797983640 1 --2- 172.21.15.42:0/1060638877 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f47900a1330 0x7f47900a1710 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:54:44.656 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.643+0000 7f4797983640 1 -- 172.21.15.42:0/1060638877 >> 172.21.15.42:0/1060638877 conn(0x7f47900a2420 msgr2=0x7f47900a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:54:44.656 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.643+0000 7f4797983640 1 -- 172.21.15.42:0/1060638877 shutdown_connections 2024-09-16T03:54:44.656 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.643+0000 7f4797983640 1 -- 172.21.15.42:0/1060638877 wait complete. 2024-09-16T03:54:44.657 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.643+0000 7f4797983640 10 monclient: build_initial_monmap 2024-09-16T03:54:44.657 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.643+0000 7f4797983640 10 monclient: monmap: 2024-09-16T03:54:44.657 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:54:44.657 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:54:44.657 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:54:44.657 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:54:44.657 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:54:44.657 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:54:44.657 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:54:44.657 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:54:44.657 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:54:44.657 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:54:44.657 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.643+0000 7f4797983640 1 librados: starting msgr at 2024-09-16T03:54:44.658 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.643+0000 7f4797983640 1 librados: starting objecter 2024-09-16T03:54:44.658 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.643+0000 7f4797983640 1 Processor -- start 2024-09-16T03:54:44.658 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.643+0000 7f4797983640 1 -- start start 2024-09-16T03:54:44.658 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.643+0000 7f4797983640 1 librados: setting wanted keys 2024-09-16T03:54:44.658 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.643+0000 7f4797983640 1 librados: calling monclient init 2024-09-16T03:54:44.658 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.643+0000 7f4797983640 10 monclient: init 2024-09-16T03:54:44.658 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.643+0000 7f4797983640 10 monclient: _reopen_session rank -1 2024-09-16T03:54:44.658 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.643+0000 7f4797983640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T03:54:44.658 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.643+0000 7f4797983640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f47900a1330 0x7f479009cb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:54:44.658 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.643+0000 7f4797983640 10 monclient(hunting): picked mon.noname-b con 0x7f47900a1330 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:54:44.658 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.643+0000 7f4797983640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f47900a1c50 0x7f479013cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:54:44.658 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.643+0000 7f4797983640 10 monclient(hunting): picked mon.noname-c con 0x7f47900a1c50 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:54:44.659 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.643+0000 7f4797983640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f47901351f0 0x7f47901356a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:54:44.659 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.643+0000 7f4797983640 10 monclient(hunting): picked mon.noname-a con 0x7f47901351f0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:54:44.659 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.643+0000 7f4797983640 10 monclient(hunting): start opening mon connection 2024-09-16T03:54:44.659 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.643+0000 7f4797983640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f479009eef0 con 0x7f47901351f0 2024-09-16T03:54:44.659 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.643+0000 7f4797983640 10 monclient(hunting): start opening mon connection 2024-09-16T03:54:44.659 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.643+0000 7f4797983640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f479009ed70 con 0x7f47900a1330 2024-09-16T03:54:44.659 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.643+0000 7f4797983640 10 monclient(hunting): start opening mon connection 2024-09-16T03:54:44.659 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.643+0000 7f4797983640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f4790140b90 con 0x7f47900a1c50 2024-09-16T03:54:44.659 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.643+0000 7f4797983640 10 monclient(hunting): _renew_subs 2024-09-16T03:54:44.659 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.643+0000 7f4797983640 10 monclient(hunting): authenticate will time out at 1731.613774s 2024-09-16T03:54:44.659 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.643+0000 7f4795f20640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f47901351f0 0x7f47901356a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:54:44.659 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.643+0000 7f4794f1e640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f47900a1c50 0x7f479013cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:54:44.659 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.643+0000 7f4795f20640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f47901351f0 0x7f47901356a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:41582/0 (socket says 172.21.15.42:41582) 2024-09-16T03:54:44.660 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.643+0000 7f4795f20640 1 -- 172.21.15.42:0/1358407748 learned_addr learned my addr 172.21.15.42:0/1358407748 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:54:44.660 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.643+0000 7f4795f20640 10 monclient(hunting): get_auth_request con 0x7f47901351f0 auth_method 0 2024-09-16T03:54:44.660 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.643+0000 7f4794f1e640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f47900a1c50 0x7f479013cef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:32820/0 (socket says 172.21.15.42:32820) 2024-09-16T03:54:44.660 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.643+0000 7f4795f20640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:54:44.660 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.643+0000 7f4795f20640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:54:44.660 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.643+0000 7f4795f20640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:54:44.660 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.643+0000 7f4794f1e640 10 monclient(hunting): get_auth_request con 0x7f47900a1c50 auth_method 0 2024-09-16T03:54:44.660 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.643+0000 7f4794f1e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:54:44.660 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.643+0000 7f4794f1e640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:54:44.660 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.643+0000 7f4794f1e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:54:44.660 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.643+0000 7f4795f20640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:54:44.660 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.643+0000 7f4795f20640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:54:44.660 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.643+0000 7f4795f20640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:54:44.660 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.643+0000 7f4794f1e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:54:44.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.643+0000 7f4794f1e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:54:44.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.643+0000 7f4794f1e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:54:44.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.643+0000 7f479571f640 1 --2- 172.21.15.42:0/1358407748 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f47900a1330 0x7f479009cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:54:44.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.643+0000 7f479571f640 10 monclient(hunting): get_auth_request con 0x7f47900a1330 auth_method 0 2024-09-16T03:54:44.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.643+0000 7f479571f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:54:44.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.643+0000 7f479571f640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:54:44.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.643+0000 7f479571f640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:54:44.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.643+0000 7f4795f20640 10 monclient(hunting): handle_auth_done global_id 824496 payload 824 2024-09-16T03:54:44.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.643+0000 7f4795f20640 1 -- 172.21.15.42:0/1358407748 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f47900a1c50 msgr2=0x7f479013cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:54:44.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.643+0000 7f4795f20640 1 --2- 172.21.15.42:0/1358407748 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f47900a1c50 0x7f479013cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:54:44.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.643+0000 7f4795f20640 1 -- 172.21.15.42:0/1358407748 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f47900a1330 msgr2=0x7f479009cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:54:44.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.643+0000 7f4795f20640 1 --2- 172.21.15.42:0/1358407748 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f47900a1330 0x7f479009cb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:54:44.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.643+0000 7f4795f20640 10 monclient: _finish_hunting 0 2024-09-16T03:54:44.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.643+0000 7f4795f20640 1 monclient: found mon.noname-a 2024-09-16T03:54:44.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.643+0000 7f4795f20640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:54:44.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.643+0000 7f4795f20640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:54:44.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.643+0000 7f4795f20640 1 -- 172.21.15.42:0/1358407748 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47900712c0 con 0x7f47901351f0 2024-09-16T03:54:44.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.643+0000 7f479571f640 1 --2- 172.21.15.42:0/1358407748 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f47900a1330 0x7f479009cb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T03:54:44.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.643+0000 7f4794f1e640 1 --2- 172.21.15.42:0/1358407748 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f47900a1c50 0x7f479013cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:54:44.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.643+0000 7f4795f20640 1 --2- 172.21.15.42:0/1358407748 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f47901351f0 0x7f47901356a0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f478800c7b0 tx=0x7f478800cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:54:44.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.643+0000 7f4785ffb640 10 client.?.objecter ms_handle_connect 0x7f47901351f0 2024-09-16T03:54:44.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.643+0000 7f4785ffb640 10 client.?.objecter resend_mon_ops 2024-09-16T03:54:44.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.643+0000 7f4785ffb640 1 -- 172.21.15.42:0/1358407748 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4788004b70 con 0x7f47901351f0 2024-09-16T03:54:44.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.643+0000 7f4785ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:54:44.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.643+0000 7f4785ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:54:44.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.643+0000 7f4785ffb640 10 monclient: dump: 2024-09-16T03:54:44.663 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:54:44.663 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:54:44.663 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:44.663 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:44.663 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:54:44.663 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:54:44.663 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:54:44.663 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:54:44.663 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:54:44.664 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:54:44.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.643+0000 7f4785ffb640 10 monclient: _finish_auth 0 2024-09-16T03:54:44.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.643+0000 7f4785ffb640 10 monclient: _check_auth_tickets 2024-09-16T03:54:44.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.643+0000 7f4785ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:54:44.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.643+0000 7f4785ffb640 1 -- 172.21.15.42:0/1358407748 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4788004d10 con 0x7f47901351f0 2024-09-16T03:54:44.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.643+0000 7f4797983640 5 monclient: authenticate success, global_id 824496 2024-09-16T03:54:44.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.643+0000 7f4797983640 10 monclient: _renew_subs 2024-09-16T03:54:44.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.643+0000 7f4797983640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:54:44.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.643+0000 7f4797983640 1 -- 172.21.15.42:0/1358407748 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f479006e6d0 con 0x7f47901351f0 2024-09-16T03:54:44.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.643+0000 7f4785ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T03:54:44.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.643+0000 7f4785ffb640 1 -- 172.21.15.42:0/1358407748 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f478800aa30 con 0x7f47901351f0 2024-09-16T03:54:44.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.643+0000 7f4785ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:54:44.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.643+0000 7f4785ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:54:44.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.643+0000 7f4785ffb640 10 monclient: dump: 2024-09-16T03:54:44.665 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:54:44.665 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:54:44.665 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:44.665 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:44.665 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:54:44.665 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:54:44.665 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:54:44.665 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:54:44.665 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:54:44.665 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:54:44.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.643+0000 7f4797983640 10 client.824496.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:54:44.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.643+0000 7f4797983640 10 monclient: _renew_subs 2024-09-16T03:54:44.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.643+0000 7f4797983640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:54:44.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.643+0000 7f4797983640 1 -- 172.21.15.42:0/1358407748 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4790120b90 con 0x7f47901351f0 2024-09-16T03:54:44.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.643+0000 7f4797983640 1 librados: init done 2024-09-16T03:54:44.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.643+0000 7f47877fe640 4 set_mon_vals no callback set 2024-09-16T03:54:44.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.643+0000 7f4797983640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:54:44.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.643+0000 7f4797983640 1 --2- 172.21.15.42:0/1358407748 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f479006dac0 0x7f47901ced40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:54:44.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.643+0000 7f4797983640 10 monclient(hunting): start opening mon connection 2024-09-16T03:54:44.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.643+0000 7f4797983640 1 -- 172.21.15.42:0/1358407748 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f4790120810 con 0x7f479006dac0 2024-09-16T03:54:44.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.643+0000 7f479571f640 1 --2- 172.21.15.42:0/1358407748 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f479006dac0 0x7f47901ced40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:54:44.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.643+0000 7f479571f640 10 monclient: get_auth_request con 0x7f479006dac0 auth_method 0 2024-09-16T03:54:44.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.643+0000 7f479571f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:54:44.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.643+0000 7f479571f640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:54:44.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.643+0000 7f479571f640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:54:44.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.647+0000 7f479571f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:54:44.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.647+0000 7f479571f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:54:44.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.647+0000 7f479571f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:54:44.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.647+0000 7f4785ffb640 1 -- 172.21.15.42:0/1358407748 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f478800abd0 con 0x7f47901351f0 2024-09-16T03:54:44.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.647+0000 7f4785ffb640 1 --2- 172.21.15.42:0/1358407748 >> v2:172.21.15.154:6812/254845342 conn(0x7f4774037810 0x7f4774039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:54:44.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.647+0000 7f479571f640 10 monclient(hunting): handle_auth_done global_id 844222 payload 824 2024-09-16T03:54:44.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.647+0000 7f479571f640 1 -- 172.21.15.42:0/1358407748 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f47900a1710 con 0x7f479006dac0 2024-09-16T03:54:44.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.647+0000 7f4785ffb640 1 -- 172.21.15.42:0/1358407748 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(386..386 src has 286..386) ==== 5713+0+0 (secure 0 0 0) 0x7f4788049400 con 0x7f47901351f0 2024-09-16T03:54:44.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.647+0000 7f4785ffb640 10 client.824496.objecter ms_dispatch 0x7f4790000ba0 osd_map(386..386 src has 286..386) 2024-09-16T03:54:44.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.647+0000 7f4785ffb640 3 client.824496.objecter handle_osd_map got epochs [386,386] > 0 2024-09-16T03:54:44.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.647+0000 7f4785ffb640 3 client.824496.objecter handle_osd_map decoding full epoch 386 2024-09-16T03:54:44.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.647+0000 7f4785ffb640 20 client.824496.objecter dump_active .. 0 homeless 2024-09-16T03:54:44.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.647+0000 7f4794f1e640 1 --2- 172.21.15.42:0/1358407748 >> v2:172.21.15.154:6812/254845342 conn(0x7f4774037810 0x7f4774039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:54:44.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.647+0000 7f4794f1e640 10 monclient: get_auth_request con 0x7f4774037810 auth_method 0 2024-09-16T03:54:44.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.647+0000 7f4785ffb640 10 client.824496.objecter ms_handle_connect 0x7f479006dac0 2024-09-16T03:54:44.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.647+0000 7f4785ffb640 10 client.824496.objecter resend_mon_ops 2024-09-16T03:54:44.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.647+0000 7f479571f640 1 --2- 172.21.15.42:0/1358407748 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f479006dac0 0x7f47901ced40 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f479006da10 tx=0x7f478c0191e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:54:44.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.647+0000 7f4785ffb640 1 -- 172.21.15.42:0/1358407748 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f478c0bc070 con 0x7f479006dac0 2024-09-16T03:54:44.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.647+0000 7f4785ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:54:44.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.647+0000 7f4785ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:54:44.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.647+0000 7f4785ffb640 10 monclient: dump: 2024-09-16T03:54:44.668 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:54:44.668 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:54:44.668 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:44.668 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:44.668 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:54:44.668 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:54:44.669 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:54:44.669 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:54:44.669 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:54:44.669 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:54:44.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.647+0000 7f4785ffb640 10 client.824496.objecter ms_handle_connect 0x7f4774037810 2024-09-16T03:54:44.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.647+0000 7f4794f1e640 1 --2- 172.21.15.42:0/1358407748 >> v2:172.21.15.154:6812/254845342 conn(0x7f4774037810 0x7f4774039cd0 secure :-1 s=READY pgs=3621 cs=0 l=1 rev1=1 crypto rx=0x7f479013d840 tx=0x7f4780009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:54:44.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.647+0000 7f4785ffb640 1 -- 172.21.15.42:0/1358407748 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f478c0acde0 con 0x7f479006dac0 2024-09-16T03:54:44.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.647+0000 7f4785ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:54:44.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.647+0000 7f4785ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:54:44.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.647+0000 7f4785ffb640 1 -- 172.21.15.42:0/1358407748 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f479006dac0 msgr2=0x7f47901ced40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:54:44.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.647+0000 7f4785ffb640 1 --2- 172.21.15.42:0/1358407748 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f479006dac0 0x7f47901ced40 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f479006da10 tx=0x7f478c0191e0 comp rx=0 tx=0).stop 2024-09-16T03:54:44.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.647+0000 7f4785ffb640 1 -- 172.21.15.42:0/1358407748 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f479006dac0 msgr2=0x7f47901ced40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:54:44.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.647+0000 7f4785ffb640 1 --2- 172.21.15.42:0/1358407748 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f479006dac0 0x7f47901ced40 secure :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f479006da10 tx=0x7f478c0191e0 comp rx=0 tx=0).stop 2024-09-16T03:54:44.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.655+0000 7f4797983640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:54:44.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.655+0000 7f4797983640 1 --2- 172.21.15.42:0/1358407748 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f47901cf280 0x7f47901cf690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:54:44.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.655+0000 7f4797983640 10 monclient(hunting): start opening mon connection 2024-09-16T03:54:44.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.655+0000 7f4797983640 1 -- 172.21.15.42:0/1358407748 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f47901cfbd0 con 0x7f47901cf280 2024-09-16T03:54:44.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.655+0000 7f479571f640 1 --2- 172.21.15.42:0/1358407748 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f47901cf280 0x7f47901cf690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:54:44.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.655+0000 7f479571f640 10 monclient: get_auth_request con 0x7f47901cf280 auth_method 0 2024-09-16T03:54:44.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.655+0000 7f479571f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:54:44.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.655+0000 7f479571f640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:54:44.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.655+0000 7f479571f640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:54:44.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.655+0000 7f479571f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:54:44.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.655+0000 7f479571f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:54:44.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.655+0000 7f479571f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:54:44.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.655+0000 7f479571f640 10 monclient(hunting): handle_auth_done global_id 844225 payload 824 2024-09-16T03:54:44.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.655+0000 7f479571f640 1 -- 172.21.15.42:0/1358407748 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f4790069040 con 0x7f47901cf280 2024-09-16T03:54:44.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.655+0000 7f479571f640 1 --2- 172.21.15.42:0/1358407748 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f47901cf280 0x7f47901cf690 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f478c009870 tx=0x7f478c009240 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:54:44.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.655+0000 7f4785ffb640 10 client.824496.objecter ms_handle_connect 0x7f47901cf280 2024-09-16T03:54:44.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.655+0000 7f4785ffb640 10 client.824496.objecter resend_mon_ops 2024-09-16T03:54:44.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.655+0000 7f4785ffb640 1 -- 172.21.15.42:0/1358407748 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f478c0a8040 con 0x7f47901cf280 2024-09-16T03:54:44.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.655+0000 7f4785ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:54:44.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.655+0000 7f4785ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:54:44.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.655+0000 7f4785ffb640 10 monclient: dump: 2024-09-16T03:54:44.674 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:54:44.674 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:54:44.674 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:44.675 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:44.675 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:54:44.675 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:54:44.675 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:54:44.675 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:54:44.675 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:54:44.675 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:54:44.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.655+0000 7f4785ffb640 1 -- 172.21.15.42:0/1358407748 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4545 (secure 0 0 0) 0x7f478c0b8660 con 0x7f47901cf280 2024-09-16T03:54:44.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.655+0000 7f4785ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:54:44.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.655+0000 7f4785ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:54:44.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.655+0000 7f4785ffb640 1 -- 172.21.15.42:0/1358407748 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f47901cf280 msgr2=0x7f47901cf690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:54:44.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.655+0000 7f4785ffb640 1 --2- 172.21.15.42:0/1358407748 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f47901cf280 0x7f47901cf690 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f478c009870 tx=0x7f478c009240 comp rx=0 tx=0).stop 2024-09-16T03:54:44.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.655+0000 7f4785ffb640 1 -- 172.21.15.42:0/1358407748 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f47901cf280 msgr2=0x7f47901cf690 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:54:44.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.655+0000 7f4785ffb640 1 --2- 172.21.15.42:0/1358407748 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f47901cf280 0x7f47901cf690 secure :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f478c009870 tx=0x7f478c009240 comp rx=0 tx=0).stop 2024-09-16T03:54:44.676 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:54:44.676 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:54:44.676 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:54:44.676 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T03:54:44.676 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 110, 2024-09-16T03:54:44.676 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 68855, 2024-09-16T03:54:44.676 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:54:44.676 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:54:44.677 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T03:54:44.677 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T03:54:44.677 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:54:44.677 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 3, 2024-09-16T03:54:44.677 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:54:44.677 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:54:44.677 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:54:44.677 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:54:44.677 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:54:44.678 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:54:44.678 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:54:44.678 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:54:44.678 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:54:44.678 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:54:44.678 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:54:44.678 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:54:44.679 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:54:44.679 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:54:44.679 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:54:44.679 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:54:44.679 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:54:44.679 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:54:44.679 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:54:44.679 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:54:44.679 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:54:44.679 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:54:44.679 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:54:44.679 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:54:44.679 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:54:44.679 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:54:44.680 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:54:44.680 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:54:44.680 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:54:44.680 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:54:44.680 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:54:44.680 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:54:44.680 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:54:44.680 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:54:44.680 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:54:44.680 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:54:44.680 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:54:44.680 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:54:44.680 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:54:44.681 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:54:44.681 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:54:44.681 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:54:44.681 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:54:44.681 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:54:44.681 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:54:44.681 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:54:44.681 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:54:44.681 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:54:44.681 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:54:44.681 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:54:44.681 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:54:44.681 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:54:44.681 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:54:44.682 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:54:44.682 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:54:44.682 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:54:44.682 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:54:44.682 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:54:44.682 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:54:44.682 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:54:44.682 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:54:44.682 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:54:44.682 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:54:44.682 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:54:44.682 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:54:44.682 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:54:44.682 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:54:44.682 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:54:44.683 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:54:44.683 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:54:44.683 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:54:44.683 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:54:44.683 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:54:44.683 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:54:44.683 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:54:44.683 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:54:44.683 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:54:44.683 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:54:44.683 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:54:44.683 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:54:44.684 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:54:44.684 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:54:44.684 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:54:44.684 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:54:44.684 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:54:44.684 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:54:44.684 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:54:44.684 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:54:44.684 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:54:44.684 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:54:44.684 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:54:44.685 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:54:44.685 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:54:44.685 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:54:44.685 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:54:44.685 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:54:44.685 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:54:44.686 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:54:44.686 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:54:44.686 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:54:44.686 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:54:44.686 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:54:44.686 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:54:44.686 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:54:44.686 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:54:44.686 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:54:44.686 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:54:44.686 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:54:44.686 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:54:44.686 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:54:44.686 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:54:44.687 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:54:44.687 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:54:44.687 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:54:44.687 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:54:44.687 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:54:44.687 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:54:44.687 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:54:44.687 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:54:44.687 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:54:44.687 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:54:44.687 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:54:44.687 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:54:44.687 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:54:44.688 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:54:44.688 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:54:44.688 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:54:44.688 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:54:44.688 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:54:44.688 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:54:44.688 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:54:44.688 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:54:44.688 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:54:44.688 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:54:44.688 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:54:44.688 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:54:44.688 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:54:44.688 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:54:44.689 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:54:44.689 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:54:44.689 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:54:44.689 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:54:44.689 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:54:44.689 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:54:44.689 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:54:44.689 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 2 2024-09-16T03:54:44.689 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:54:44.689 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:54:44.689 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:54:44.689 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:54:44.689 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:54:44.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.655+0000 7f4797983640 10 librados: watch_flush enter 2024-09-16T03:54:44.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.655+0000 7f4797983640 10 librados: watch_flush exit 2024-09-16T03:54:44.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.655+0000 7f4797983640 20 client.824496.objecter shutdown clearing up homeless session... 2024-09-16T03:54:44.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.655+0000 7f4797983640 10 client.824496.objecter successfully canceled tick 2024-09-16T03:54:44.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.655+0000 7f4797983640 1 -- 172.21.15.42:0/1358407748 >> v2:172.21.15.154:6812/254845342 conn(0x7f4774037810 msgr2=0x7f4774039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:54:44.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.655+0000 7f4797983640 1 --2- 172.21.15.42:0/1358407748 >> v2:172.21.15.154:6812/254845342 conn(0x7f4774037810 0x7f4774039cd0 secure :-1 s=READY pgs=3621 cs=0 l=1 rev1=1 crypto rx=0x7f479013d840 tx=0x7f4780009040 comp rx=0 tx=0).stop 2024-09-16T03:54:44.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.655+0000 7f4797983640 10 monclient: shutdown 2024-09-16T03:54:44.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.655+0000 7f4797983640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:54:44.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.655+0000 7f4797983640 1 -- 172.21.15.42:0/1358407748 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f47901351f0 msgr2=0x7f47901356a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:54:44.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.655+0000 7f4797983640 1 --2- 172.21.15.42:0/1358407748 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f47901351f0 0x7f47901356a0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f478800c7b0 tx=0x7f478800cc70 comp rx=0 tx=0).stop 2024-09-16T03:54:44.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.655+0000 7f4795f20640 1 -- 172.21.15.42:0/1358407748 reap_dead start 2024-09-16T03:54:44.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.655+0000 7f4795f20640 1 -- 172.21.15.42:0/1358407748 reap_dead start 2024-09-16T03:54:44.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.655+0000 7f4797983640 1 -- 172.21.15.42:0/1358407748 shutdown_connections 2024-09-16T03:54:44.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.655+0000 7f4797983640 1 -- 172.21.15.42:0/1358407748 >> 172.21.15.42:0/1358407748 conn(0x7f47900a2420 msgr2=0x7f47900a0a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:54:44.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.655+0000 7f4797983640 1 -- 172.21.15.42:0/1358407748 shutdown_connections 2024-09-16T03:54:44.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.655+0000 7f4797983640 1 -- 172.21.15.42:0/1358407748 wait complete. 2024-09-16T03:54:44.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.655+0000 7f4797983640 1 librados: shutdown 2024-09-16T03:54:44.693 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:54:44.693 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-09-16T03:54:44.693 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-16T03:54:44.693 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-09-16T03:54:44.693 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-16T03:54:44.693 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-16T03:54:44.694 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T03:54:44.716 INFO:tasks.ceph.mon.b:Stopped 2024-09-16T03:54:44.716 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T03:54:44.716 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:54:44.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.827+0000 7f72da79d640 10 monclient: get_monmap_and_config 2024-09-16T03:54:44.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.827+0000 7f72da79d640 10 monclient: build_initial_monmap 2024-09-16T03:54:44.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.827+0000 7f72da79d640 10 monclient: monmap: 2024-09-16T03:54:44.832 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:54:44.832 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:54:44.832 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:54:44.830268+0000 2024-09-16T03:54:44.832 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:54:44.830268+0000 2024-09-16T03:54:44.832 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:54:44.832 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:54:44.832 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:54:44.832 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:54:44.832 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:54:44.832 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:54:44.832 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:54:44.832 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:54:44.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.827+0000 7f72da79d640 1 Processor -- start 2024-09-16T03:54:44.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.827+0000 7f72da79d640 1 -- start start 2024-09-16T03:54:44.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.827+0000 7f72da79d640 10 monclient: init 2024-09-16T03:54:44.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.827+0000 7f72da79d640 10 monclient: _reopen_session rank -1 2024-09-16T03:54:44.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.827+0000 7f72da79d640 10 monclient: _add_conns ranks=[2,1,3,4,0] 2024-09-16T03:54:44.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.827+0000 7f72da79d640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f72d40a1330 0x7f72d40a1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:54:44.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.827+0000 7f72da79d640 10 monclient(hunting): picked mon.noname-b con 0x7f72d40a1330 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:54:44.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.827+0000 7f72da79d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f72d406dac0 addr v1:172.21.15.154:6789/0 2024-09-16T03:54:44.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.827+0000 7f72da79d640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f72d40a1c50 0x7f72d4135810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:54:44.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.827+0000 7f72da79d640 10 monclient(hunting): picked mon.noname-a con 0x7f72d40a1c50 addr v2:172.21.15.42:3300/0 2024-09-16T03:54:44.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.827+0000 7f72da79d640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f72d409c090 con 0x7f72d406dac0 2024-09-16T03:54:44.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.827+0000 7f72da79d640 10 monclient(hunting): start opening mon connection 2024-09-16T03:54:44.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.827+0000 7f72da79d640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f72d409c890 con 0x7f72d40a1c50 2024-09-16T03:54:44.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.827+0000 7f72da79d640 10 monclient(hunting): start opening mon connection 2024-09-16T03:54:44.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.827+0000 7f72da79d640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f72d409f610 con 0x7f72d40a1330 2024-09-16T03:54:44.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.827+0000 7f72da79d640 10 monclient(hunting): _renew_subs 2024-09-16T03:54:44.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.827+0000 7f72da79d640 10 monclient(hunting): authenticate will time out at 1731.799968s 2024-09-16T03:54:44.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.831+0000 7f72d37fe640 1 -- >> v1:172.21.15.154:6789/0 conn(0x7f72d406dac0 legacy=0x7f72d4135100 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.154:6789/0 2024-09-16T03:54:44.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.831+0000 7f72d3fff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f72d40a1330 0x7f72d40a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:54:44.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.831+0000 7f72d3fff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f72d40a1330 0x7f72d40a1710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:32824/0 (socket says 172.21.15.42:32824) 2024-09-16T03:54:44.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.831+0000 7f72d3fff640 1 -- 172.21.15.42:0/4229433668 learned_addr learned my addr 172.21.15.42:0/4229433668 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:54:44.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.831+0000 7f72d3fff640 10 monclient(hunting): get_auth_request con 0x7f72d40a1330 auth_method 0 2024-09-16T03:54:44.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.831+0000 7f72d3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:54:44.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.831+0000 7f72d3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:54:44.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.831+0000 7f72d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:54:44.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.831+0000 7f72d8d3a640 1 --2- 172.21.15.42:0/4229433668 >> v2:172.21.15.42:3300/0 conn(0x7f72d40a1c50 0x7f72d4135810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:54:44.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.831+0000 7f72d3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:54:44.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.831+0000 7f72d3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:54:44.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.831+0000 7f72d3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:54:44.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.831+0000 7f72d8d3a640 10 monclient(hunting): get_auth_request con 0x7f72d40a1c50 auth_method 0 2024-09-16T03:54:44.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.831+0000 7f72d8d3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:54:44.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.831+0000 7f72d8d3a640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:54:44.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.831+0000 7f72d8d3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:54:44.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.831+0000 7f72d8d3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:54:44.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.831+0000 7f72d8d3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:54:44.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.831+0000 7f72d8d3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:54:44.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.831+0000 7f72d3fff640 10 monclient(hunting): handle_auth_done global_id 854147 payload 274 2024-09-16T03:54:44.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.831+0000 7f72d3fff640 1 -- 172.21.15.42:0/4229433668 >> v2:172.21.15.42:3300/0 conn(0x7f72d40a1c50 msgr2=0x7f72d4135810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:54:44.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.831+0000 7f72d3fff640 1 --2- 172.21.15.42:0/4229433668 >> v2:172.21.15.42:3300/0 conn(0x7f72d40a1c50 0x7f72d4135810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:54:44.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.831+0000 7f72d3fff640 1 -- 172.21.15.42:0/4229433668 >> v1:172.21.15.154:6789/0 conn(0x7f72d406dac0 legacy=0x7f72d4135100 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:54:44.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.831+0000 7f72d3fff640 10 monclient: _finish_hunting 0 2024-09-16T03:54:44.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.831+0000 7f72d3fff640 1 monclient: found mon.noname-b 2024-09-16T03:54:44.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.831+0000 7f72d3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:54:44.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.831+0000 7f72d3fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T03:54:44.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.831+0000 7f72d3fff640 1 -- 172.21.15.42:0/4229433668 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72d409f8f0 con 0x7f72d40a1330 2024-09-16T03:54:44.843 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.831+0000 7f72d3fff640 1 --2- 172.21.15.42:0/4229433668 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f72d40a1330 0x7f72d40a1710 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f72cc009e50 tx=0x7f72cc01c910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=504a24c7a642111f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:54:44.844 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.831+0000 7f72d2ffd640 1 -- 172.21.15.42:0/4229433668 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72cc0a57b0 con 0x7f72d40a1330 2024-09-16T03:54:44.844 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.831+0000 7f72d2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:54:44.844 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.831+0000 7f72d2ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:54:44.844 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.831+0000 7f72d2ffd640 10 monclient: dump: 2024-09-16T03:54:44.844 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:54:44.844 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:54:44.844 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:44.844 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:44.844 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:54:44.844 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:54:44.844 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:54:44.844 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:54:44.844 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:54:44.845 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:54:44.845 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.831+0000 7f72d2ffd640 10 monclient: _finish_auth 0 2024-09-16T03:54:44.845 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.831+0000 7f72d2ffd640 10 monclient: _check_auth_tickets 2024-09-16T03:54:44.845 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.831+0000 7f72d2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:54:44.845 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.831+0000 7f72d2ffd640 1 -- 172.21.15.42:0/4229433668 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f72cc0a5950 con 0x7f72d40a1330 2024-09-16T03:54:44.845 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.831+0000 7f72d2ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:54:44.845 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.831+0000 7f72d2ffd640 1 -- 172.21.15.42:0/4229433668 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72cc0a5d20 con 0x7f72d40a1330 2024-09-16T03:54:44.845 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.831+0000 7f72da79d640 5 monclient: authenticate success, global_id 854147 2024-09-16T03:54:44.845 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.831+0000 7f72da79d640 10 monclient: get_monmap_and_config success 2024-09-16T03:54:44.845 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.831+0000 7f72da79d640 4 set_mon_vals no callback set 2024-09-16T03:54:44.845 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.831+0000 7f72d2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:54:44.845 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.831+0000 7f72d2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:54:44.845 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.831+0000 7f72d2ffd640 10 monclient: dump: 2024-09-16T03:54:44.846 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:54:44.846 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:54:44.846 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:44.846 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:44.846 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:54:44.846 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:54:44.846 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:54:44.846 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:54:44.846 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:54:44.846 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:54:44.846 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.831+0000 7f72da79d640 10 monclient: shutdown 2024-09-16T03:54:44.846 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.831+0000 7f72da79d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:54:44.846 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.831+0000 7f72da79d640 1 -- 172.21.15.42:0/4229433668 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f72d40a1330 msgr2=0x7f72d40a1710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:54:44.846 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.831+0000 7f72da79d640 1 --2- 172.21.15.42:0/4229433668 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f72d40a1330 0x7f72d40a1710 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f72cc009e50 tx=0x7f72cc01c910 comp rx=0 tx=0).stop 2024-09-16T03:54:44.847 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.831+0000 7f72da79d640 1 -- 172.21.15.42:0/4229433668 shutdown_connections 2024-09-16T03:54:44.847 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.831+0000 7f72da79d640 1 --2- 172.21.15.42:0/4229433668 >> v2:172.21.15.42:3300/0 conn(0x7f72d40a1c50 0x7f72d4135810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:54:44.847 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.831+0000 7f72da79d640 1 --2- 172.21.15.42:0/4229433668 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f72d40a1330 0x7f72d40a1710 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:54:44.847 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.831+0000 7f72da79d640 1 -- 172.21.15.42:0/4229433668 >> 172.21.15.42:0/4229433668 conn(0x7f72d40a2420 msgr2=0x7f72d40a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:54:44.847 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.831+0000 7f72da79d640 1 -- 172.21.15.42:0/4229433668 shutdown_connections 2024-09-16T03:54:44.847 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.831+0000 7f72da79d640 1 -- 172.21.15.42:0/4229433668 wait complete. 2024-09-16T03:54:44.847 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.835+0000 7f72da79d640 10 monclient: build_initial_monmap 2024-09-16T03:54:44.847 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.835+0000 7f72da79d640 10 monclient: monmap: 2024-09-16T03:54:44.847 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:54:44.847 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:54:44.847 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:54:44.847 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:54:44.847 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:54:44.848 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:54:44.848 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:54:44.848 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:54:44.848 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:54:44.848 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:54:44.848 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.835+0000 7f72da79d640 1 librados: starting msgr at 2024-09-16T03:54:44.848 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.835+0000 7f72da79d640 1 librados: starting objecter 2024-09-16T03:54:44.848 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.835+0000 7f72da79d640 1 Processor -- start 2024-09-16T03:54:44.848 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.835+0000 7f72da79d640 1 -- start start 2024-09-16T03:54:44.848 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.835+0000 7f72da79d640 1 librados: setting wanted keys 2024-09-16T03:54:44.848 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.835+0000 7f72da79d640 1 librados: calling monclient init 2024-09-16T03:54:44.848 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.835+0000 7f72da79d640 10 monclient: init 2024-09-16T03:54:44.848 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.835+0000 7f72da79d640 10 monclient: _reopen_session rank -1 2024-09-16T03:54:44.848 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.835+0000 7f72da79d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:54:44.848 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.835+0000 7f72da79d640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f72d40a1330 0x7f72d41ccb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:54:44.849 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.835+0000 7f72da79d640 10 monclient(hunting): picked mon.noname-c con 0x7f72d40a1330 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:54:44.849 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.835+0000 7f72da79d640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f72d40a1c50 0x7f72d4134f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:54:44.849 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.835+0000 7f72da79d640 10 monclient(hunting): picked mon.noname-a con 0x7f72d40a1c50 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:54:44.849 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.835+0000 7f72da79d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f72d4135450 0x7f72d411bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:54:44.849 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.835+0000 7f72da79d640 10 monclient(hunting): picked mon.noname-b con 0x7f72d4135450 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:54:44.849 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.835+0000 7f72da79d640 10 monclient(hunting): start opening mon connection 2024-09-16T03:54:44.849 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.835+0000 7f72da79d640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f72d409ef30 con 0x7f72d40a1c50 2024-09-16T03:54:44.849 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.835+0000 7f72da79d640 10 monclient(hunting): start opening mon connection 2024-09-16T03:54:44.849 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.835+0000 7f72da79d640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f72d409edb0 con 0x7f72d4135450 2024-09-16T03:54:44.849 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.835+0000 7f72da79d640 10 monclient(hunting): start opening mon connection 2024-09-16T03:54:44.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.835+0000 7f72da79d640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f72d4135d50 con 0x7f72d40a1330 2024-09-16T03:54:44.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.835+0000 7f72da79d640 10 monclient(hunting): _renew_subs 2024-09-16T03:54:44.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.835+0000 7f72da79d640 10 monclient(hunting): authenticate will time out at 1731.804977s 2024-09-16T03:54:44.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.835+0000 7f72d3fff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f72d40a1330 0x7f72d41ccb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:54:44.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.835+0000 7f72d37fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f72d40a1c50 0x7f72d4134f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:54:44.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.835+0000 7f72d37fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f72d40a1c50 0x7f72d4134f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:41606/0 (socket says 172.21.15.42:41606) 2024-09-16T03:54:44.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.835+0000 7f72d3fff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f72d40a1330 0x7f72d41ccb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:32840/0 (socket says 172.21.15.42:32840) 2024-09-16T03:54:44.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.835+0000 7f72d8d3a640 1 -- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f72d4135450 msgr2=0x7f72d411bf70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T03:54:44.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.835+0000 7f72d37fe640 1 -- 172.21.15.42:0/262903418 learned_addr learned my addr 172.21.15.42:0/262903418 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:54:44.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.835+0000 7f72d37fe640 10 monclient(hunting): get_auth_request con 0x7f72d40a1c50 auth_method 0 2024-09-16T03:54:44.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.835+0000 7f72d37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:54:44.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.835+0000 7f72d37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:54:44.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.835+0000 7f72d37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:54:44.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.835+0000 7f72d3fff640 10 monclient(hunting): get_auth_request con 0x7f72d40a1330 auth_method 0 2024-09-16T03:54:44.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.835+0000 7f72d8d3a640 1 --2- 172.21.15.42:0/262903418 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f72d4135450 0x7f72d411bf70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:54:44.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.835+0000 7f72d3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:54:44.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.835+0000 7f72d3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:54:44.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.835+0000 7f72d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:54:44.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.835+0000 7f72d37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:54:44.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.835+0000 7f72d37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:54:44.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.835+0000 7f72d37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:54:44.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.835+0000 7f72d3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:54:44.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.835+0000 7f72d3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:54:44.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.835+0000 7f72d3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:54:44.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.835+0000 7f72d3fff640 10 monclient(hunting): handle_auth_done global_id 854150 payload 824 2024-09-16T03:54:44.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.835+0000 7f72d3fff640 1 -- 172.21.15.42:0/262903418 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f72d4135450 msgr2=0x7f72d411bf70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:54:44.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.835+0000 7f72d3fff640 1 --2- 172.21.15.42:0/262903418 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f72d4135450 0x7f72d411bf70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:54:44.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.835+0000 7f72d3fff640 1 -- 172.21.15.42:0/262903418 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f72d40a1c50 msgr2=0x7f72d4134f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:54:44.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.835+0000 7f72d3fff640 1 --2- 172.21.15.42:0/262903418 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f72d40a1c50 0x7f72d4134f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:54:44.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.835+0000 7f72d3fff640 10 monclient: _finish_hunting 0 2024-09-16T03:54:44.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.835+0000 7f72d3fff640 1 monclient: found mon.noname-c 2024-09-16T03:54:44.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.835+0000 7f72d3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:54:44.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.835+0000 7f72d3fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:54:44.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.835+0000 7f72d3fff640 1 -- 172.21.15.42:0/262903418 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72d4071270 con 0x7f72d40a1330 2024-09-16T03:54:44.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.835+0000 7f72d37fe640 1 --2- 172.21.15.42:0/262903418 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f72d40a1c50 0x7f72d4134f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:54:44.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.835+0000 7f72d3fff640 1 --2- 172.21.15.42:0/262903418 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f72d40a1330 0x7f72d41ccb10 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f72cc0ab000 tx=0x7f72cc0196b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:54:44.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.835+0000 7f72d0ff9640 10 client.?.objecter ms_handle_connect 0x7f72d40a1330 2024-09-16T03:54:44.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.835+0000 7f72d0ff9640 10 client.?.objecter resend_mon_ops 2024-09-16T03:54:44.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.835+0000 7f72d0ff9640 1 -- 172.21.15.42:0/262903418 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72cc0c4070 con 0x7f72d40a1330 2024-09-16T03:54:44.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.835+0000 7f72d0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:54:44.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.835+0000 7f72d0ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:54:44.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.835+0000 7f72d0ff9640 10 monclient: dump: 2024-09-16T03:54:44.853 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:54:44.853 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:54:44.853 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:44.853 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:44.853 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:54:44.853 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:54:44.853 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:54:44.853 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:54:44.853 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:54:44.853 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:54:44.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.835+0000 7f72d0ff9640 10 monclient: _finish_auth 0 2024-09-16T03:54:44.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.835+0000 7f72d0ff9640 10 monclient: _check_auth_tickets 2024-09-16T03:54:44.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.835+0000 7f72d0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:54:44.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.835+0000 7f72d0ff9640 1 -- 172.21.15.42:0/262903418 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f72cc0b55d0 con 0x7f72d40a1330 2024-09-16T03:54:44.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.835+0000 7f72da79d640 5 monclient: authenticate success, global_id 854150 2024-09-16T03:54:44.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.835+0000 7f72da79d640 10 monclient: _renew_subs 2024-09-16T03:54:44.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.835+0000 7f72da79d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:54:44.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.835+0000 7f72da79d640 1 -- 172.21.15.42:0/262903418 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72d411f5c0 con 0x7f72d40a1330 2024-09-16T03:54:44.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.835+0000 7f72d0ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T03:54:44.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.835+0000 7f72d0ff9640 1 -- 172.21.15.42:0/262903418 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72cc0b59c0 con 0x7f72d40a1330 2024-09-16T03:54:44.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.835+0000 7f72d0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:54:44.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.835+0000 7f72d0ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:54:44.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.835+0000 7f72d0ff9640 10 monclient: dump: 2024-09-16T03:54:44.854 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:54:44.854 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:54:44.854 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:44.854 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:44.854 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:54:44.854 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:54:44.855 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:54:44.855 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:54:44.855 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:54:44.855 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:54:44.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.835+0000 7f72da79d640 10 client.854150.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:54:44.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.835+0000 7f72da79d640 10 monclient: _renew_subs 2024-09-16T03:54:44.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.835+0000 7f72da79d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:54:44.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.835+0000 7f72d27fc640 4 set_mon_vals no callback set 2024-09-16T03:54:44.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.835+0000 7f72da79d640 1 -- 172.21.15.42:0/262903418 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f72d411f2e0 con 0x7f72d40a1330 2024-09-16T03:54:44.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.835+0000 7f72da79d640 1 librados: init done 2024-09-16T03:54:44.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.835+0000 7f72d0ff9640 1 -- 172.21.15.42:0/262903418 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f72cc0b5b60 con 0x7f72d40a1330 2024-09-16T03:54:44.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.835+0000 7f72da79d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:54:44.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.835+0000 7f72da79d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:54:44.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.835+0000 7f72da79d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:54:44.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.835+0000 7f72da79d640 1 -- 172.21.15.42:0/262903418 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f72d4073a30 con 0x7f72d40a1330 2024-09-16T03:54:44.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.835+0000 7f72d0ff9640 1 --2- 172.21.15.42:0/262903418 >> v2:172.21.15.154:6812/254845342 conn(0x7f72c00377f0 0x7f72c0039cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:54:44.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.835+0000 7f72d0ff9640 1 -- 172.21.15.42:0/262903418 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(386..386 src has 286..386) ==== 5713+0+0 (secure 0 0 0) 0x7f72cc0e7310 con 0x7f72d40a1330 2024-09-16T03:54:44.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.835+0000 7f72d0ff9640 10 client.854150.objecter ms_dispatch 0x7f72d4000ba0 osd_map(386..386 src has 286..386) 2024-09-16T03:54:44.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.835+0000 7f72d0ff9640 3 client.854150.objecter handle_osd_map got epochs [386,386] > 0 2024-09-16T03:54:44.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.835+0000 7f72d0ff9640 3 client.854150.objecter handle_osd_map decoding full epoch 386 2024-09-16T03:54:44.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.835+0000 7f72d0ff9640 20 client.854150.objecter dump_active .. 0 homeless 2024-09-16T03:54:44.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.835+0000 7f72d37fe640 1 --2- 172.21.15.42:0/262903418 >> v2:172.21.15.154:6812/254845342 conn(0x7f72c00377f0 0x7f72c0039cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:54:44.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.835+0000 7f72d37fe640 10 monclient: get_auth_request con 0x7f72c00377f0 auth_method 0 2024-09-16T03:54:44.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.839+0000 7f72d37fe640 1 --2- 172.21.15.42:0/262903418 >> v2:172.21.15.154:6812/254845342 conn(0x7f72c00377f0 0x7f72c0039cb0 secure :-1 s=READY pgs=3622 cs=0 l=1 rev1=1 crypto rx=0x7f72d413cff0 tx=0x7f72c8003ba0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:54:44.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.839+0000 7f72d0ff9640 10 client.854150.objecter ms_handle_connect 0x7f72c00377f0 2024-09-16T03:54:44.857 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.839+0000 7f72d0ff9640 1 -- 172.21.15.42:0/262903418 <== mon.2 v2:172.21.15.42:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f72d4073a30 con 0x7f72d40a1330 2024-09-16T03:54:44.857 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.839+0000 7f72d0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:54:44.857 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:44.839+0000 7f72d0ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:54:45.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:45.063+0000 7f72da79d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:54:45.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:45.063+0000 7f72da79d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:54:45.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:45.063+0000 7f72da79d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:54:45.067 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:45.063+0000 7f72da79d640 1 -- 172.21.15.42:0/262903418 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f72d40a1710 con 0x7f72d40a1330 2024-09-16T03:54:45.067 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:45.063+0000 7f72d0ff9640 1 -- 172.21.15.42:0/262903418 <== mon.2 v2:172.21.15.42:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7f72d4071270 con 0x7f72d40a1330 2024-09-16T03:54:45.067 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:45.063+0000 7f72d0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:54:45.067 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:45.063+0000 7f72d0ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:54:45.067 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":110,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:54:45.070 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:45.063+0000 7f72da79d640 10 librados: watch_flush enter 2024-09-16T03:54:45.070 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:45.063+0000 7f72da79d640 10 librados: watch_flush exit 2024-09-16T03:54:45.070 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:45.063+0000 7f72da79d640 20 client.854150.objecter shutdown clearing up homeless session... 2024-09-16T03:54:45.070 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:45.067+0000 7f72da79d640 10 client.854150.objecter successfully canceled tick 2024-09-16T03:54:45.070 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:45.067+0000 7f72da79d640 1 -- 172.21.15.42:0/262903418 >> v2:172.21.15.154:6812/254845342 conn(0x7f72c00377f0 msgr2=0x7f72c0039cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:54:45.070 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:45.067+0000 7f72da79d640 1 --2- 172.21.15.42:0/262903418 >> v2:172.21.15.154:6812/254845342 conn(0x7f72c00377f0 0x7f72c0039cb0 secure :-1 s=READY pgs=3622 cs=0 l=1 rev1=1 crypto rx=0x7f72d413cff0 tx=0x7f72c8003ba0 comp rx=0 tx=0).stop 2024-09-16T03:54:45.070 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:45.067+0000 7f72da79d640 10 monclient: shutdown 2024-09-16T03:54:45.070 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:45.067+0000 7f72da79d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:54:45.070 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:45.067+0000 7f72da79d640 1 -- 172.21.15.42:0/262903418 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f72d40a1330 msgr2=0x7f72d41ccb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:54:45.070 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:45.067+0000 7f72da79d640 1 --2- 172.21.15.42:0/262903418 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f72d40a1330 0x7f72d41ccb10 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f72cc0ab000 tx=0x7f72cc0196b0 comp rx=0 tx=0).stop 2024-09-16T03:54:45.071 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:45.067+0000 7f72da79d640 1 -- 172.21.15.42:0/262903418 shutdown_connections 2024-09-16T03:54:45.071 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:45.067+0000 7f72da79d640 1 --2- 172.21.15.42:0/262903418 >> v2:172.21.15.154:6812/254845342 conn(0x7f72c00377f0 0x7f72c0039cb0 unknown :-1 s=CLOSED pgs=3622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:54:45.071 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:45.067+0000 7f72da79d640 1 --2- 172.21.15.42:0/262903418 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f72d4135450 0x7f72d411bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:54:45.071 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:45.067+0000 7f72da79d640 1 --2- 172.21.15.42:0/262903418 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f72d40a1c50 0x7f72d4134f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:54:45.071 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:45.067+0000 7f72da79d640 1 --2- 172.21.15.42:0/262903418 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f72d40a1330 0x7f72d41ccb10 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:54:45.071 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:45.067+0000 7f72da79d640 1 -- 172.21.15.42:0/262903418 >> 172.21.15.42:0/262903418 conn(0x7f72d40a2420 msgr2=0x7f72d406f860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:54:45.071 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:45.067+0000 7f72da79d640 1 -- 172.21.15.42:0/262903418 shutdown_connections 2024-09-16T03:54:45.071 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:45.067+0000 7f72da79d640 1 -- 172.21.15.42:0/262903418 wait complete. 2024-09-16T03:54:45.071 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:45.067+0000 7f72da79d640 1 librados: shutdown 2024-09-16T03:54:48.085 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:54:48.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.215+0000 7fad21b16640 10 monclient: get_monmap_and_config 2024-09-16T03:54:48.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.215+0000 7fad21b16640 10 monclient: build_initial_monmap 2024-09-16T03:54:48.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.215+0000 7fad21b16640 10 monclient: monmap: 2024-09-16T03:54:48.219 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:54:48.219 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:54:48.219 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:54:48.217491+0000 2024-09-16T03:54:48.220 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:54:48.217491+0000 2024-09-16T03:54:48.220 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:54:48.220 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:54:48.220 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:54:48.220 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:54:48.221 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:54:48.221 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:54:48.221 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:54:48.221 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:54:48.235 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.215+0000 7fad21b16640 1 Processor -- start 2024-09-16T03:54:48.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.215+0000 7fad21b16640 1 -- start start 2024-09-16T03:54:48.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.215+0000 7fad21b16640 10 monclient: init 2024-09-16T03:54:48.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.215+0000 7fad21b16640 10 monclient: _reopen_session rank -1 2024-09-16T03:54:48.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.215+0000 7fad21b16640 10 monclient: _add_conns ranks=[3,2,0,4,1] 2024-09-16T03:54:48.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.215+0000 7fad21b16640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fad1c0a1330 0x7fad1c0a1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:54:48.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.215+0000 7fad21b16640 10 monclient(hunting): picked mon.noname-a con 0x7fad1c0a1330 addr v2:172.21.15.42:3300/0 2024-09-16T03:54:48.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.215+0000 7fad21b16640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fad1c06dac0 0x7fad1c0a1c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:54:48.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.215+0000 7fad21b16640 10 monclient(hunting): picked mon.noname-b con 0x7fad1c06dac0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:54:48.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.215+0000 7fad21b16640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fad1c135100 addr v1:172.21.15.42:6789/0 2024-09-16T03:54:48.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.215+0000 7fad21b16640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fad1c09c7d0 con 0x7fad1c135100 2024-09-16T03:54:48.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.215+0000 7fad21b16640 10 monclient(hunting): start opening mon connection 2024-09-16T03:54:48.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.215+0000 7fad21b16640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7fad1c09c090 con 0x7fad1c0a1330 2024-09-16T03:54:48.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.215+0000 7fad21b16640 10 monclient(hunting): start opening mon connection 2024-09-16T03:54:48.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.215+0000 7fad21b16640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fad1c09f610 con 0x7fad1c06dac0 2024-09-16T03:54:48.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.215+0000 7fad21b16640 10 monclient(hunting): _renew_subs 2024-09-16T03:54:48.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.215+0000 7fad21b16640 10 monclient(hunting): authenticate will time out at 1735.186932s 2024-09-16T03:54:48.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.215+0000 7fad1affd640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fad1c06dac0 0x7fad1c0a1c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:54:48.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.215+0000 7fad1b7fe640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fad1c0a1330 0x7fad1c0a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:54:48.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.215+0000 7fad1bfff640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fad1c135100 0x7fad1c135550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:57040/0 (socket says 172.21.15.42:57040) 2024-09-16T03:54:48.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.215+0000 7fad1b7fe640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fad1c0a1330 0x7fad1c0a1710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:41612/0 (socket says 172.21.15.42:41612) 2024-09-16T03:54:48.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.215+0000 7fad1affd640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fad1c06dac0 0x7fad1c0a1c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:32842/0 (socket says 172.21.15.42:32842) 2024-09-16T03:54:48.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.215+0000 7fad1bfff640 1 -- 172.21.15.42:0/3232120384 learned_addr learned my addr 172.21.15.42:0/3232120384 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:54:48.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.215+0000 7fad1b7fe640 10 monclient(hunting): get_auth_request con 0x7fad1c0a1330 auth_method 0 2024-09-16T03:54:48.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.215+0000 7fad1b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:54:48.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.215+0000 7fad1b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:54:48.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.215+0000 7fad1b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:54:48.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.215+0000 7fad1affd640 10 monclient(hunting): get_auth_request con 0x7fad1c06dac0 auth_method 0 2024-09-16T03:54:48.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.215+0000 7fad1affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:54:48.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.215+0000 7fad1affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:54:48.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.215+0000 7fad1affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:54:48.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.215+0000 7fad1b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:54:48.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.215+0000 7fad1b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:54:48.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.215+0000 7fad1b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:54:48.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.215+0000 7fad1affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:54:48.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.215+0000 7fad1affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:54:48.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.215+0000 7fad1affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:54:48.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.215+0000 7fad1affd640 10 monclient(hunting): handle_auth_done global_id 854153 payload 274 2024-09-16T03:54:48.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.215+0000 7fad1affd640 1 -- 172.21.15.42:0/3232120384 >> v2:172.21.15.42:3300/0 conn(0x7fad1c0a1330 msgr2=0x7fad1c0a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:54:48.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.215+0000 7fad1a7fc640 1 -- 172.21.15.42:0/3232120384 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2981595395 0 0) 0x7fad1c09c7d0 con 0x7fad1c135100 2024-09-16T03:54:48.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.215+0000 7fad1affd640 1 --2- 172.21.15.42:0/3232120384 >> v2:172.21.15.42:3300/0 conn(0x7fad1c0a1330 0x7fad1c0a1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:54:48.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.215+0000 7fad1affd640 1 -- 172.21.15.42:0/3232120384 >> v1:172.21.15.42:6789/0 conn(0x7fad1c135100 legacy=0x7fad1c135550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:54:48.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.215+0000 7fad1affd640 10 monclient: _finish_hunting 0 2024-09-16T03:54:48.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.215+0000 7fad1affd640 1 monclient: found mon.noname-b 2024-09-16T03:54:48.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.215+0000 7fad1affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:54:48.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.215+0000 7fad1affd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T03:54:48.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.215+0000 7fad1affd640 1 -- 172.21.15.42:0/3232120384 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad1c09f8f0 con 0x7fad1c06dac0 2024-09-16T03:54:48.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.215+0000 7fad1a7fc640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T03:54:48.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.215+0000 7fad1b7fe640 1 --2- 172.21.15.42:0/3232120384 >> v2:172.21.15.42:3300/0 conn(0x7fad1c0a1330 0x7fad1c0a1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:54:48.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.219+0000 7fad1affd640 1 --2- 172.21.15.42:0/3232120384 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fad1c06dac0 0x7fad1c0a1c50 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fad040097c0 tx=0x7fad0401c480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6fe7d53f7b24dde3 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:54:48.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.219+0000 7fad1a7fc640 1 -- 172.21.15.42:0/3232120384 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad040b2070 con 0x7fad1c06dac0 2024-09-16T03:54:48.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.219+0000 7fad1a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:54:48.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.219+0000 7fad1a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:54:48.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.219+0000 7fad1a7fc640 10 monclient: dump: 2024-09-16T03:54:48.241 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:54:48.241 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:54:48.241 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:48.241 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:48.242 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:54:48.242 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:54:48.242 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:54:48.242 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:54:48.242 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:54:48.242 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:54:48.243 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.219+0000 7fad1a7fc640 10 monclient: _finish_auth 0 2024-09-16T03:54:48.243 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.219+0000 7fad1a7fc640 10 monclient: _check_auth_tickets 2024-09-16T03:54:48.243 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.219+0000 7fad1a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:54:48.243 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.219+0000 7fad21b16640 5 monclient: authenticate success, global_id 854153 2024-09-16T03:54:48.243 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.219+0000 7fad21b16640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:54:48.243 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.219+0000 7fad1a7fc640 1 -- 172.21.15.42:0/3232120384 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fad040ad640 con 0x7fad1c06dac0 2024-09-16T03:54:48.243 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.219+0000 7fad1a7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:54:48.243 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.219+0000 7fad21b16640 10 monclient: get_monmap_and_config success 2024-09-16T03:54:48.243 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.219+0000 7fad21b16640 4 set_mon_vals no callback set 2024-09-16T03:54:48.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.219+0000 7fad1a7fc640 1 -- 172.21.15.42:0/3232120384 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad040ada10 con 0x7fad1c06dac0 2024-09-16T03:54:48.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.219+0000 7fad1a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:54:48.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.219+0000 7fad1a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:54:48.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.219+0000 7fad1a7fc640 10 monclient: dump: 2024-09-16T03:54:48.244 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:54:48.244 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:54:48.244 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:48.244 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:48.244 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:54:48.244 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:54:48.244 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:54:48.244 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:54:48.245 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:54:48.245 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:54:48.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.219+0000 7fad21b16640 10 monclient: shutdown 2024-09-16T03:54:48.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.219+0000 7fad21b16640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:54:48.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.219+0000 7fad21b16640 1 -- 172.21.15.42:0/3232120384 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fad1c06dac0 msgr2=0x7fad1c0a1c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:54:48.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.219+0000 7fad21b16640 1 --2- 172.21.15.42:0/3232120384 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fad1c06dac0 0x7fad1c0a1c50 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fad040097c0 tx=0x7fad0401c480 comp rx=0 tx=0).stop 2024-09-16T03:54:48.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.219+0000 7fad21b16640 1 -- 172.21.15.42:0/3232120384 shutdown_connections 2024-09-16T03:54:48.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.219+0000 7fad21b16640 1 --2- 172.21.15.42:0/3232120384 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fad1c06dac0 0x7fad1c0a1c50 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:54:48.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.219+0000 7fad21b16640 1 --2- 172.21.15.42:0/3232120384 >> v2:172.21.15.42:3300/0 conn(0x7fad1c0a1330 0x7fad1c0a1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:54:48.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.219+0000 7fad21b16640 1 -- 172.21.15.42:0/3232120384 >> 172.21.15.42:0/3232120384 conn(0x7fad1c0a2420 msgr2=0x7fad1c0a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:54:48.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.219+0000 7fad21b16640 1 -- 172.21.15.42:0/3232120384 shutdown_connections 2024-09-16T03:54:48.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.219+0000 7fad21b16640 1 -- 172.21.15.42:0/3232120384 wait complete. 2024-09-16T03:54:48.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.219+0000 7fad21b16640 10 monclient: build_initial_monmap 2024-09-16T03:54:48.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.219+0000 7fad21b16640 10 monclient: monmap: 2024-09-16T03:54:48.246 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:54:48.246 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:54:48.246 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:54:48.246 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:54:48.246 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:54:48.246 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:54:48.246 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:54:48.247 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:54:48.247 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:54:48.247 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:54:48.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.219+0000 7fad21b16640 1 librados: starting msgr at 2024-09-16T03:54:48.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.219+0000 7fad21b16640 1 librados: starting objecter 2024-09-16T03:54:48.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.219+0000 7fad21b16640 1 Processor -- start 2024-09-16T03:54:48.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.219+0000 7fad21b16640 1 -- start start 2024-09-16T03:54:48.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.219+0000 7fad21b16640 1 librados: setting wanted keys 2024-09-16T03:54:48.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.219+0000 7fad21b16640 1 librados: calling monclient init 2024-09-16T03:54:48.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.219+0000 7fad21b16640 10 monclient: init 2024-09-16T03:54:48.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.219+0000 7fad21b16640 10 monclient: _reopen_session rank -1 2024-09-16T03:54:48.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.223+0000 7fad21b16640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:54:48.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.223+0000 7fad21b16640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fad1c0a1330 0x7fad1c1cf020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:54:48.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.223+0000 7fad21b16640 10 monclient(hunting): picked mon.noname-c con 0x7fad1c0a1330 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:54:48.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.223+0000 7fad21b16640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fad1c135100 0x7fad1c134350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:54:48.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.223+0000 7fad21b16640 10 monclient(hunting): picked mon.noname-b con 0x7fad1c135100 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:54:48.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.223+0000 7fad21b16640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fad1c134890 0x7fad1c11ce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:54:48.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.223+0000 7fad21b16640 10 monclient(hunting): picked mon.noname-a con 0x7fad1c134890 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:54:48.249 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.223+0000 7fad21b16640 10 monclient(hunting): start opening mon connection 2024-09-16T03:54:48.249 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.223+0000 7fad21b16640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fad1c13fd00 con 0x7fad1c134890 2024-09-16T03:54:48.249 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.223+0000 7fad21b16640 10 monclient(hunting): start opening mon connection 2024-09-16T03:54:48.249 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.223+0000 7fad21b16640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fad1c13fb80 con 0x7fad1c135100 2024-09-16T03:54:48.249 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.223+0000 7fad21b16640 10 monclient(hunting): start opening mon connection 2024-09-16T03:54:48.249 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.223+0000 7fad21b16640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fad1c13fe80 con 0x7fad1c0a1330 2024-09-16T03:54:48.249 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.223+0000 7fad21b16640 10 monclient(hunting): _renew_subs 2024-09-16T03:54:48.249 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.223+0000 7fad21b16640 10 monclient(hunting): authenticate will time out at 1735.192111s 2024-09-16T03:54:48.249 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.223+0000 7fad1b7fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fad1c0a1330 0x7fad1c1cf020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:54:48.250 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.223+0000 7fad1bfff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fad1c134890 0x7fad1c11ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:54:48.250 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.223+0000 7fad1b7fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fad1c0a1330 0x7fad1c1cf020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:32848/0 (socket says 172.21.15.42:32848) 2024-09-16T03:54:48.250 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.223+0000 7fad1b7fe640 1 -- 172.21.15.42:0/1391441843 learned_addr learned my addr 172.21.15.42:0/1391441843 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:54:48.250 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.223+0000 7fad1b7fe640 10 monclient(hunting): get_auth_request con 0x7fad1c0a1330 auth_method 0 2024-09-16T03:54:48.250 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.223+0000 7fad1b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:54:48.250 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.223+0000 7fad1b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:54:48.250 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.223+0000 7fad1b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:54:48.250 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.223+0000 7fad1bfff640 10 monclient(hunting): get_auth_request con 0x7fad1c134890 auth_method 0 2024-09-16T03:54:48.250 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.223+0000 7fad1bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:54:48.250 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.223+0000 7fad1bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:54:48.250 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.223+0000 7fad1bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:54:48.251 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.223+0000 7fad1affd640 1 -- 172.21.15.42:0/1391441843 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fad1c135100 msgr2=0x7fad1c134350 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T03:54:48.251 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.223+0000 7fad1affd640 1 --2- 172.21.15.42:0/1391441843 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fad1c135100 0x7fad1c134350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:54:48.251 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.223+0000 7fad1b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:54:48.251 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.223+0000 7fad1b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:54:48.251 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.223+0000 7fad1b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:54:48.251 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.223+0000 7fad1bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:54:48.251 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.223+0000 7fad1bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:54:48.251 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.223+0000 7fad1bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:54:48.251 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.223+0000 7fad1b7fe640 10 monclient(hunting): handle_auth_done global_id 854156 payload 824 2024-09-16T03:54:48.251 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.223+0000 7fad1b7fe640 1 -- 172.21.15.42:0/1391441843 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fad1c135100 msgr2=0x7fad1c134350 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:54:48.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.223+0000 7fad1b7fe640 1 --2- 172.21.15.42:0/1391441843 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fad1c135100 0x7fad1c134350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:54:48.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.223+0000 7fad1b7fe640 1 -- 172.21.15.42:0/1391441843 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fad1c134890 msgr2=0x7fad1c11ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:54:48.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.223+0000 7fad1b7fe640 1 --2- 172.21.15.42:0/1391441843 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fad1c134890 0x7fad1c11ce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:54:48.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.223+0000 7fad1b7fe640 10 monclient: _finish_hunting 0 2024-09-16T03:54:48.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.223+0000 7fad1b7fe640 1 monclient: found mon.noname-c 2024-09-16T03:54:48.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.223+0000 7fad1b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:54:48.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.223+0000 7fad1b7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:54:48.253 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.223+0000 7fad1b7fe640 1 -- 172.21.15.42:0/1391441843 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad1c13c750 con 0x7fad1c0a1330 2024-09-16T03:54:48.253 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.223+0000 7fad1bfff640 1 --2- 172.21.15.42:0/1391441843 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fad1c134890 0x7fad1c11ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:54:48.253 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.223+0000 7fad1b7fe640 1 --2- 172.21.15.42:0/1391441843 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fad1c0a1330 0x7fad1c1cf020 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fad14009f60 tx=0x7fad14009f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:54:48.253 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.223+0000 7fad13fff640 10 client.?.objecter ms_handle_connect 0x7fad1c0a1330 2024-09-16T03:54:48.253 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.223+0000 7fad13fff640 10 client.?.objecter resend_mon_ops 2024-09-16T03:54:48.253 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.223+0000 7fad13fff640 1 -- 172.21.15.42:0/1391441843 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad14018020 con 0x7fad1c0a1330 2024-09-16T03:54:48.253 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.223+0000 7fad13fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:54:48.253 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.223+0000 7fad13fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:54:48.253 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.223+0000 7fad13fff640 10 monclient: dump: 2024-09-16T03:54:48.253 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:54:48.254 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:54:48.254 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:48.254 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:48.254 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:54:48.254 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:54:48.254 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:54:48.254 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:54:48.254 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:54:48.254 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:54:48.254 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.223+0000 7fad13fff640 10 monclient: _finish_auth 0 2024-09-16T03:54:48.254 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.223+0000 7fad13fff640 10 monclient: _check_auth_tickets 2024-09-16T03:54:48.254 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.223+0000 7fad13fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:54:48.255 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.223+0000 7fad13fff640 1 -- 172.21.15.42:0/1391441843 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fad14014680 con 0x7fad1c0a1330 2024-09-16T03:54:48.255 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.223+0000 7fad21b16640 5 monclient: authenticate success, global_id 854156 2024-09-16T03:54:48.255 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.223+0000 7fad21b16640 10 monclient: _renew_subs 2024-09-16T03:54:48.255 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.223+0000 7fad21b16640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:54:48.255 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.223+0000 7fad21b16640 1 -- 172.21.15.42:0/1391441843 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad1c120a20 con 0x7fad1c0a1330 2024-09-16T03:54:48.255 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.223+0000 7fad13fff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:54:48.255 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.223+0000 7fad13fff640 1 -- 172.21.15.42:0/1391441843 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad14014a50 con 0x7fad1c0a1330 2024-09-16T03:54:48.255 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.223+0000 7fad13fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:54:48.255 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.223+0000 7fad13fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:54:48.255 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.223+0000 7fad13fff640 10 monclient: dump: 2024-09-16T03:54:48.255 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:54:48.255 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:54:48.255 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:48.256 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:48.256 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:54:48.256 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:54:48.256 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:54:48.256 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:54:48.256 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:54:48.256 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:54:48.256 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.223+0000 7fad19ffb640 4 set_mon_vals no callback set 2024-09-16T03:54:48.257 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.223+0000 7fad21b16640 10 client.854156.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:54:48.257 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.223+0000 7fad21b16640 10 monclient: _renew_subs 2024-09-16T03:54:48.257 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.223+0000 7fad21b16640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:54:48.257 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.223+0000 7fad21b16640 1 -- 172.21.15.42:0/1391441843 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fad1c0a5410 con 0x7fad1c0a1330 2024-09-16T03:54:48.257 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.223+0000 7fad21b16640 1 librados: init done 2024-09-16T03:54:48.258 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.223+0000 7fad21b16640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:54:48.258 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.223+0000 7fad21b16640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:54:48.258 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.223+0000 7fad21b16640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:54:48.259 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.223+0000 7fad21b16640 1 -- 172.21.15.42:0/1391441843 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fad1c11f2f0 con 0x7fad1c0a1330 2024-09-16T03:54:48.259 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.223+0000 7fad13fff640 1 -- 172.21.15.42:0/1391441843 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fad1401b020 con 0x7fad1c0a1330 2024-09-16T03:54:48.259 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.223+0000 7fad13fff640 1 --2- 172.21.15.42:0/1391441843 >> v2:172.21.15.154:6812/254845342 conn(0x7facfc0377c0 0x7facfc039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:54:48.259 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.223+0000 7fad13fff640 1 -- 172.21.15.42:0/1391441843 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(386..386 src has 286..386) ==== 5713+0+0 (secure 0 0 0) 0x7fad14049c90 con 0x7fad1c0a1330 2024-09-16T03:54:48.259 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.223+0000 7fad13fff640 10 client.854156.objecter ms_dispatch 0x7fad1c000ba0 osd_map(386..386 src has 286..386) 2024-09-16T03:54:48.259 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.223+0000 7fad13fff640 3 client.854156.objecter handle_osd_map got epochs [386,386] > 0 2024-09-16T03:54:48.260 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.223+0000 7fad13fff640 3 client.854156.objecter handle_osd_map decoding full epoch 386 2024-09-16T03:54:48.260 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.223+0000 7fad13fff640 20 client.854156.objecter dump_active .. 0 homeless 2024-09-16T03:54:48.260 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.223+0000 7fad1affd640 1 --2- 172.21.15.42:0/1391441843 >> v2:172.21.15.154:6812/254845342 conn(0x7facfc0377c0 0x7facfc039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:54:48.260 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.223+0000 7fad1affd640 10 monclient: get_auth_request con 0x7facfc0377c0 auth_method 0 2024-09-16T03:54:48.260 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.223+0000 7fad13fff640 10 client.854156.objecter ms_handle_connect 0x7facfc0377c0 2024-09-16T03:54:48.260 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.223+0000 7fad1affd640 1 --2- 172.21.15.42:0/1391441843 >> v2:172.21.15.154:6812/254845342 conn(0x7facfc0377c0 0x7facfc039c80 secure :-1 s=READY pgs=3624 cs=0 l=1 rev1=1 crypto rx=0x7fad040af000 tx=0x7fad04004740 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:54:48.260 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.227+0000 7fad13fff640 1 -- 172.21.15.42:0/1391441843 <== mon.2 v2:172.21.15.42:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7fad14022030 con 0x7fad1c0a1330 2024-09-16T03:54:48.260 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.227+0000 7fad13fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:54:48.261 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.227+0000 7fad13fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:54:48.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.447+0000 7fad21b16640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:54:48.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.447+0000 7fad21b16640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:54:48.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.447+0000 7fad21b16640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:54:48.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.447+0000 7fad21b16640 1 -- 172.21.15.42:0/1391441843 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fad1c135510 con 0x7fad1c0a1330 2024-09-16T03:54:48.454 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.451+0000 7fad13fff640 1 -- 172.21.15.42:0/1391441843 <== mon.2 v2:172.21.15.42:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7fad1c13c750 con 0x7fad1c0a1330 2024-09-16T03:54:48.454 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.451+0000 7fad13fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:54:48.454 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.451+0000 7fad13fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:54:48.454 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":110,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:54:48.456 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.451+0000 7fad21b16640 10 librados: watch_flush enter 2024-09-16T03:54:48.456 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.451+0000 7fad21b16640 10 librados: watch_flush exit 2024-09-16T03:54:48.456 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.451+0000 7fad21b16640 20 client.854156.objecter shutdown clearing up homeless session... 2024-09-16T03:54:48.456 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.451+0000 7fad21b16640 10 client.854156.objecter successfully canceled tick 2024-09-16T03:54:48.456 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.451+0000 7fad21b16640 1 -- 172.21.15.42:0/1391441843 >> v2:172.21.15.154:6812/254845342 conn(0x7facfc0377c0 msgr2=0x7facfc039c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:54:48.456 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.451+0000 7fad21b16640 1 --2- 172.21.15.42:0/1391441843 >> v2:172.21.15.154:6812/254845342 conn(0x7facfc0377c0 0x7facfc039c80 secure :-1 s=READY pgs=3624 cs=0 l=1 rev1=1 crypto rx=0x7fad040af000 tx=0x7fad04004740 comp rx=0 tx=0).stop 2024-09-16T03:54:48.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.451+0000 7fad21b16640 10 monclient: shutdown 2024-09-16T03:54:48.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.451+0000 7fad21b16640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:54:48.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.451+0000 7fad21b16640 1 -- 172.21.15.42:0/1391441843 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fad1c0a1330 msgr2=0x7fad1c1cf020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:54:48.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.451+0000 7fad21b16640 1 --2- 172.21.15.42:0/1391441843 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fad1c0a1330 0x7fad1c1cf020 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fad14009f60 tx=0x7fad14009f90 comp rx=0 tx=0).stop 2024-09-16T03:54:48.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.451+0000 7fad21b16640 1 -- 172.21.15.42:0/1391441843 shutdown_connections 2024-09-16T03:54:48.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.451+0000 7fad21b16640 1 --2- 172.21.15.42:0/1391441843 >> v2:172.21.15.154:6812/254845342 conn(0x7facfc0377c0 0x7facfc039c80 unknown :-1 s=CLOSED pgs=3624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:54:48.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.451+0000 7fad21b16640 1 --2- 172.21.15.42:0/1391441843 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fad1c134890 0x7fad1c11ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:54:48.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.451+0000 7fad21b16640 1 --2- 172.21.15.42:0/1391441843 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fad1c135100 0x7fad1c134350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:54:48.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.451+0000 7fad21b16640 1 --2- 172.21.15.42:0/1391441843 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fad1c0a1330 0x7fad1c1cf020 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:54:48.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.451+0000 7fad21b16640 1 -- 172.21.15.42:0/1391441843 >> 172.21.15.42:0/1391441843 conn(0x7fad1c0a2420 msgr2=0x7fad1c09cb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:54:48.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.451+0000 7fad21b16640 1 -- 172.21.15.42:0/1391441843 shutdown_connections 2024-09-16T03:54:48.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.451+0000 7fad21b16640 1 -- 172.21.15.42:0/1391441843 wait complete. 2024-09-16T03:54:48.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:48.451+0000 7fad21b16640 1 librados: shutdown 2024-09-16T03:54:51.472 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:54:51.601 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.595+0000 7f7b06585640 10 monclient: get_monmap_and_config 2024-09-16T03:54:51.601 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.595+0000 7f7b06585640 10 monclient: build_initial_monmap 2024-09-16T03:54:51.601 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.595+0000 7f7b06585640 10 monclient: monmap: 2024-09-16T03:54:51.601 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:54:51.601 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:54:51.601 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:54:51.599925+0000 2024-09-16T03:54:51.601 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:54:51.599925+0000 2024-09-16T03:54:51.602 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:54:51.602 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:54:51.602 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:54:51.602 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:54:51.602 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:54:51.602 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:54:51.602 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:54:51.602 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:54:51.604 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.599+0000 7f7b06585640 1 Processor -- start 2024-09-16T03:54:51.604 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.599+0000 7f7b06585640 1 -- start start 2024-09-16T03:54:51.604 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.599+0000 7f7b06585640 10 monclient: init 2024-09-16T03:54:51.604 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.599+0000 7f7b06585640 10 monclient: _reopen_session rank -1 2024-09-16T03:54:51.604 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.599+0000 7f7b06585640 10 monclient: _add_conns ranks=[3,4,0,1,2] 2024-09-16T03:54:51.604 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.599+0000 7f7b06585640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f7b0009fb00 0x7f7b0009ba50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:54:51.604 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.599+0000 7f7b06585640 10 monclient(hunting): picked mon.noname-a con 0x7f7b0009fb00 addr v2:172.21.15.42:3300/0 2024-09-16T03:54:51.604 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.599+0000 7f7b06585640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f7b0009bf90 0x7f7b0009c400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:54:51.604 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.599+0000 7f7b06585640 10 monclient(hunting): picked mon.noname-c con 0x7f7b0009bf90 addr v2:172.21.15.154:3300/0 2024-09-16T03:54:51.605 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.599+0000 7f7b06585640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7b000a1330 addr v1:172.21.15.42:6789/0 2024-09-16T03:54:51.605 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.599+0000 7f7b06585640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7b000a1e20 con 0x7f7b000a1330 2024-09-16T03:54:51.605 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.599+0000 7f7b06585640 10 monclient(hunting): start opening mon connection 2024-09-16T03:54:51.605 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.599+0000 7f7b06585640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f7b0009e250 con 0x7f7b0009fb00 2024-09-16T03:54:51.605 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.599+0000 7f7b06585640 10 monclient(hunting): start opening mon connection 2024-09-16T03:54:51.605 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.599+0000 7f7b06585640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f7b0009ea30 con 0x7f7b0009bf90 2024-09-16T03:54:51.605 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.599+0000 7f7b06585640 10 monclient(hunting): _renew_subs 2024-09-16T03:54:51.605 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.599+0000 7f7b06585640 10 monclient(hunting): authenticate will time out at 1738.569507s 2024-09-16T03:54:51.605 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.599+0000 7f7afffff640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f7b0009fb00 0x7f7b0009ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:54:51.605 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.599+0000 7f7b04b22640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f7b000a1330 0x7f7b0013b230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:57044/0 (socket says 172.21.15.42:57044) 2024-09-16T03:54:51.605 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.599+0000 7f7b04b22640 1 -- 172.21.15.42:0/919935278 learned_addr learned my addr 172.21.15.42:0/919935278 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:54:51.605 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.599+0000 7f7afffff640 10 monclient(hunting): get_auth_request con 0x7f7b0009fb00 auth_method 0 2024-09-16T03:54:51.605 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.599+0000 7f7afffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:54:51.606 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.599+0000 7f7afffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:54:51.606 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.599+0000 7f7afffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:54:51.606 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.599+0000 7f7aff7fe640 1 -- 172.21.15.42:0/919935278 >> v2:172.21.15.154:3300/0 conn(0x7f7b0009bf90 msgr2=0x7f7b0009c400 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T03:54:51.606 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.599+0000 7f7aff7fe640 1 --2- 172.21.15.42:0/919935278 >> v2:172.21.15.154:3300/0 conn(0x7f7b0009bf90 0x7f7b0009c400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:54:51.606 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.599+0000 7f7afffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:54:51.606 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.599+0000 7f7afffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:54:51.606 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.599+0000 7f7afffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:54:51.606 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.599+0000 7f7afffff640 10 monclient(hunting): handle_auth_done global_id 824514 payload 274 2024-09-16T03:54:51.606 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.599+0000 7f7afffff640 1 -- 172.21.15.42:0/919935278 >> v2:172.21.15.154:3300/0 conn(0x7f7b0009bf90 msgr2=0x7f7b0009c400 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:54:51.606 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.599+0000 7f7afffff640 1 --2- 172.21.15.42:0/919935278 >> v2:172.21.15.154:3300/0 conn(0x7f7b0009bf90 0x7f7b0009c400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:54:51.606 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.599+0000 7f7afffff640 1 -- 172.21.15.42:0/919935278 >> v1:172.21.15.42:6789/0 conn(0x7f7b000a1330 legacy=0x7f7b0013b230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:54:51.606 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.599+0000 7f7afffff640 10 monclient: _finish_hunting 0 2024-09-16T03:54:51.606 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.599+0000 7f7afffff640 1 monclient: found mon.noname-a 2024-09-16T03:54:51.606 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.599+0000 7f7afffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:54:51.606 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.599+0000 7f7afffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:54:51.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.599+0000 7f7afffff640 1 -- 172.21.15.42:0/919935278 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b0009fee0 con 0x7f7b0009fb00 2024-09-16T03:54:51.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.603+0000 7f7afffff640 1 --2- 172.21.15.42:0/919935278 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7b0009fb00 0x7f7b0009ba50 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f7b0006c650 tx=0x7f7af001e570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a9dd100b393b34f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:54:51.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.603+0000 7f7afeffd640 1 -- 172.21.15.42:0/919935278 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7af00ac600 con 0x7f7b0009fb00 2024-09-16T03:54:51.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.603+0000 7f7afeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:54:51.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.603+0000 7f7afeffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:54:51.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.603+0000 7f7afeffd640 10 monclient: dump: 2024-09-16T03:54:51.613 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:54:51.613 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:54:51.613 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:51.613 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:51.613 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:54:51.613 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:54:51.614 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:54:51.614 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:54:51.614 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:54:51.614 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:54:51.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.603+0000 7f7afeffd640 10 monclient: _finish_auth 0 2024-09-16T03:54:51.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.603+0000 7f7afeffd640 10 monclient: _check_auth_tickets 2024-09-16T03:54:51.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.603+0000 7f7afeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:54:51.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.603+0000 7f7afeffd640 1 -- 172.21.15.42:0/919935278 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7af00b1070 con 0x7f7b0009fb00 2024-09-16T03:54:51.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.603+0000 7f7afeffd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:54:51.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.603+0000 7f7afeffd640 1 -- 172.21.15.42:0/919935278 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7af00ac9c0 con 0x7f7b0009fb00 2024-09-16T03:54:51.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.603+0000 7f7b06585640 5 monclient: authenticate success, global_id 824514 2024-09-16T03:54:51.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.603+0000 7f7b06585640 10 monclient: get_monmap_and_config success 2024-09-16T03:54:51.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.603+0000 7f7b06585640 4 set_mon_vals no callback set 2024-09-16T03:54:51.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.603+0000 7f7afeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:54:51.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.603+0000 7f7afeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:54:51.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.603+0000 7f7afeffd640 10 monclient: dump: 2024-09-16T03:54:51.615 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:54:51.615 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:54:51.615 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:51.615 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:51.615 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:54:51.615 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:54:51.615 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:54:51.615 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:54:51.615 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:54:51.615 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:54:51.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.603+0000 7f7b06585640 10 monclient: shutdown 2024-09-16T03:54:51.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.603+0000 7f7b06585640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:54:51.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.603+0000 7f7b06585640 1 -- 172.21.15.42:0/919935278 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7b0009fb00 msgr2=0x7f7b0009ba50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:54:51.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.603+0000 7f7b06585640 1 --2- 172.21.15.42:0/919935278 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7b0009fb00 0x7f7b0009ba50 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f7b0006c650 tx=0x7f7af001e570 comp rx=0 tx=0).stop 2024-09-16T03:54:51.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.603+0000 7f7b06585640 1 -- 172.21.15.42:0/919935278 shutdown_connections 2024-09-16T03:54:51.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.603+0000 7f7b06585640 1 --2- 172.21.15.42:0/919935278 >> v2:172.21.15.154:3300/0 conn(0x7f7b0009bf90 0x7f7b0009c400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:54:51.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.603+0000 7f7b06585640 1 --2- 172.21.15.42:0/919935278 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7b0009fb00 0x7f7b0009ba50 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:54:51.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.603+0000 7f7b06585640 1 -- 172.21.15.42:0/919935278 >> 172.21.15.42:0/919935278 conn(0x7f7b000a2420 msgr2=0x7f7b000a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:54:51.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.603+0000 7f7b06585640 1 -- 172.21.15.42:0/919935278 shutdown_connections 2024-09-16T03:54:51.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.603+0000 7f7b06585640 1 -- 172.21.15.42:0/919935278 wait complete. 2024-09-16T03:54:51.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.603+0000 7f7b06585640 10 monclient: build_initial_monmap 2024-09-16T03:54:51.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.603+0000 7f7b06585640 10 monclient: monmap: 2024-09-16T03:54:51.617 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:54:51.617 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:54:51.617 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:54:51.617 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:54:51.617 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:54:51.617 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:54:51.617 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:54:51.617 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:54:51.617 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:54:51.617 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:54:51.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.603+0000 7f7b06585640 1 librados: starting msgr at 2024-09-16T03:54:51.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.603+0000 7f7b06585640 1 librados: starting objecter 2024-09-16T03:54:51.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.603+0000 7f7b06585640 1 Processor -- start 2024-09-16T03:54:51.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.603+0000 7f7b06585640 1 -- start start 2024-09-16T03:54:51.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.603+0000 7f7b06585640 1 librados: setting wanted keys 2024-09-16T03:54:51.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.603+0000 7f7b06585640 1 librados: calling monclient init 2024-09-16T03:54:51.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.603+0000 7f7b06585640 10 monclient: init 2024-09-16T03:54:51.620 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.603+0000 7f7b06585640 10 monclient: _reopen_session rank -1 2024-09-16T03:54:51.620 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.603+0000 7f7b06585640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T03:54:51.620 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.603+0000 7f7b06585640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7b0009bf90 0x7f7b00135e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:54:51.620 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.603+0000 7f7b06585640 10 monclient(hunting): picked mon.noname-a con 0x7f7b0009bf90 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:54:51.620 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.603+0000 7f7b06585640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7b0009fb00 0x7f7b001363c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:54:51.620 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.603+0000 7f7b06585640 10 monclient(hunting): picked mon.noname-b con 0x7f7b0009fb00 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:54:51.621 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.603+0000 7f7b06585640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7b000a1330 0x7f7b00136900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:54:51.621 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.603+0000 7f7b06585640 10 monclient(hunting): picked mon.noname-c con 0x7f7b000a1330 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:54:51.621 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.603+0000 7f7b06585640 10 monclient(hunting): start opening mon connection 2024-09-16T03:54:51.621 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.603+0000 7f7b06585640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f7b0009d380 con 0x7f7b0009bf90 2024-09-16T03:54:51.621 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.603+0000 7f7b06585640 10 monclient(hunting): start opening mon connection 2024-09-16T03:54:51.621 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.603+0000 7f7b06585640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f7b0009d140 con 0x7f7b0009fb00 2024-09-16T03:54:51.621 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.603+0000 7f7b06585640 10 monclient(hunting): start opening mon connection 2024-09-16T03:54:51.621 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.603+0000 7f7b06585640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f7b0009d500 con 0x7f7b000a1330 2024-09-16T03:54:51.621 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.603+0000 7f7b06585640 10 monclient(hunting): _renew_subs 2024-09-16T03:54:51.621 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.603+0000 7f7b06585640 10 monclient(hunting): authenticate will time out at 1738.575090s 2024-09-16T03:54:51.621 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.603+0000 7f7afffff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7b0009bf90 0x7f7b00135e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:54:51.622 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.603+0000 7f7afffff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7b0009bf90 0x7f7b00135e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:41622/0 (socket says 172.21.15.42:41622) 2024-09-16T03:54:51.622 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.603+0000 7f7afffff640 1 -- 172.21.15.42:0/711503345 learned_addr learned my addr 172.21.15.42:0/711503345 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:54:51.622 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.603+0000 7f7afffff640 10 monclient(hunting): get_auth_request con 0x7f7b0009bf90 auth_method 0 2024-09-16T03:54:51.622 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.603+0000 7f7afffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:54:51.622 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.603+0000 7f7afffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:54:51.622 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.603+0000 7f7afffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:54:51.622 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.603+0000 7f7b04b22640 1 --2- 172.21.15.42:0/711503345 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7b000a1330 0x7f7b00136900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:54:51.622 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.603+0000 7f7aff7fe640 1 -- 172.21.15.42:0/711503345 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7b0009fb00 msgr2=0x7f7b001363c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T03:54:51.622 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.603+0000 7f7b04b22640 10 monclient(hunting): get_auth_request con 0x7f7b000a1330 auth_method 0 2024-09-16T03:54:51.622 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.603+0000 7f7b04b22640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:54:51.622 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.603+0000 7f7b04b22640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:54:51.622 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.603+0000 7f7b04b22640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:54:51.622 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.603+0000 7f7afffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:54:51.622 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.603+0000 7f7afffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:54:51.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.603+0000 7f7aff7fe640 1 --2- 172.21.15.42:0/711503345 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7b0009fb00 0x7f7b001363c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:54:51.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.603+0000 7f7afffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:54:51.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.603+0000 7f7b04b22640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:54:51.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.603+0000 7f7b04b22640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:54:51.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.603+0000 7f7b04b22640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:54:51.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.603+0000 7f7afffff640 10 monclient(hunting): handle_auth_done global_id 824517 payload 824 2024-09-16T03:54:51.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.603+0000 7f7afffff640 1 -- 172.21.15.42:0/711503345 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7b000a1330 msgr2=0x7f7b00136900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:54:51.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.603+0000 7f7afffff640 1 --2- 172.21.15.42:0/711503345 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7b000a1330 0x7f7b00136900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:54:51.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.603+0000 7f7afffff640 1 -- 172.21.15.42:0/711503345 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7b0009fb00 msgr2=0x7f7b001363c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:54:51.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.603+0000 7f7afffff640 1 --2- 172.21.15.42:0/711503345 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7b0009fb00 0x7f7b001363c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:54:51.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.603+0000 7f7afffff640 10 monclient: _finish_hunting 0 2024-09-16T03:54:51.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.603+0000 7f7afffff640 1 monclient: found mon.noname-a 2024-09-16T03:54:51.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.603+0000 7f7afffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:54:51.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.603+0000 7f7afffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:54:51.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.603+0000 7f7afffff640 1 -- 172.21.15.42:0/711503345 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b0006fd90 con 0x7f7b0009bf90 2024-09-16T03:54:51.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.603+0000 7f7afffff640 1 --2- 172.21.15.42:0/711503345 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7b0009bf90 0x7f7b00135e80 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f7af00ae040 tx=0x7f7af001cdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:54:51.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.603+0000 7f7afcff9640 10 client.?.objecter ms_handle_connect 0x7f7b0009bf90 2024-09-16T03:54:51.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.603+0000 7f7afcff9640 10 client.?.objecter resend_mon_ops 2024-09-16T03:54:51.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.603+0000 7f7afcff9640 1 -- 172.21.15.42:0/711503345 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7af00c0610 con 0x7f7b0009bf90 2024-09-16T03:54:51.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.607+0000 7f7afcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:54:51.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.607+0000 7f7afcff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:54:51.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.607+0000 7f7afcff9640 10 monclient: dump: 2024-09-16T03:54:51.624 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:54:51.624 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:54:51.624 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:51.624 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:51.624 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:54:51.625 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:54:51.625 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:54:51.625 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:54:51.625 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:54:51.625 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:54:51.625 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.607+0000 7f7afcff9640 10 monclient: _finish_auth 0 2024-09-16T03:54:51.625 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.607+0000 7f7afcff9640 10 monclient: _check_auth_tickets 2024-09-16T03:54:51.625 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.607+0000 7f7afcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:54:51.625 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.607+0000 7f7afcff9640 1 -- 172.21.15.42:0/711503345 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7af00c07d0 con 0x7f7b0009bf90 2024-09-16T03:54:51.625 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.607+0000 7f7afcff9640 10 monclient: handle_config config(0 keys) 2024-09-16T03:54:51.625 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.607+0000 7f7afcff9640 1 -- 172.21.15.42:0/711503345 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7af00c0bc0 con 0x7f7b0009bf90 2024-09-16T03:54:51.625 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.607+0000 7f7afcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:54:51.625 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.607+0000 7f7afcff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:54:51.625 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.607+0000 7f7afcff9640 10 monclient: dump: 2024-09-16T03:54:51.626 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:54:51.626 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:54:51.626 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:51.626 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:54:51.626 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:54:51.626 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:54:51.626 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:54:51.626 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:54:51.626 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:54:51.626 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:54:51.626 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.607+0000 7f7b06585640 5 monclient: authenticate success, global_id 824517 2024-09-16T03:54:51.626 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.607+0000 7f7b06585640 10 monclient: _renew_subs 2024-09-16T03:54:51.626 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.607+0000 7f7afdffb640 4 set_mon_vals no callback set 2024-09-16T03:54:51.627 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.607+0000 7f7b06585640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:54:51.627 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.607+0000 7f7b06585640 1 -- 172.21.15.42:0/711503345 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b0006e380 con 0x7f7b0009bf90 2024-09-16T03:54:51.627 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.607+0000 7f7b06585640 10 client.824517.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:54:51.627 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.607+0000 7f7b06585640 10 monclient: _renew_subs 2024-09-16T03:54:51.627 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.607+0000 7f7b06585640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:54:51.627 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.607+0000 7f7b06585640 1 -- 172.21.15.42:0/711503345 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b00073a30 con 0x7f7b0009bf90 2024-09-16T03:54:51.627 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.607+0000 7f7b06585640 1 librados: init done 2024-09-16T03:54:51.627 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.607+0000 7f7afcff9640 1 -- 172.21.15.42:0/711503345 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f7af00be020 con 0x7f7b0009bf90 2024-09-16T03:54:51.627 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.607+0000 7f7b06585640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:54:51.627 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.607+0000 7f7b06585640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:54:51.628 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.607+0000 7f7b06585640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:54:51.628 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.607+0000 7f7b06585640 1 -- 172.21.15.42:0/711503345 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7b0009c400 con 0x7f7b0009bf90 2024-09-16T03:54:51.628 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.607+0000 7f7afcff9640 1 --2- 172.21.15.42:0/711503345 >> v2:172.21.15.154:6812/254845342 conn(0x7f7ae00377c0 0x7f7ae0039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:54:51.628 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.607+0000 7f7afcff9640 1 -- 172.21.15.42:0/711503345 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(386..386 src has 286..386) ==== 5713+0+0 (secure 0 0 0) 0x7f7af00e5770 con 0x7f7b0009bf90 2024-09-16T03:54:51.628 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.607+0000 7f7afcff9640 10 client.824517.objecter ms_dispatch 0x7f7b00000ba0 osd_map(386..386 src has 286..386) 2024-09-16T03:54:51.628 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.607+0000 7f7afcff9640 3 client.824517.objecter handle_osd_map got epochs [386,386] > 0 2024-09-16T03:54:51.628 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.607+0000 7f7afcff9640 3 client.824517.objecter handle_osd_map decoding full epoch 386 2024-09-16T03:54:51.628 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.607+0000 7f7afcff9640 20 client.824517.objecter dump_active .. 0 homeless 2024-09-16T03:54:51.628 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.607+0000 7f7aff7fe640 1 --2- 172.21.15.42:0/711503345 >> v2:172.21.15.154:6812/254845342 conn(0x7f7ae00377c0 0x7f7ae0039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:54:51.628 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.607+0000 7f7aff7fe640 10 monclient: get_auth_request con 0x7f7ae00377c0 auth_method 0 2024-09-16T03:54:51.628 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.607+0000 7f7afcff9640 10 client.824517.objecter ms_handle_connect 0x7f7ae00377c0 2024-09-16T03:54:51.628 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.607+0000 7f7aff7fe640 1 --2- 172.21.15.42:0/711503345 >> v2:172.21.15.154:6812/254845342 conn(0x7f7ae00377c0 0x7f7ae0039c80 secure :-1 s=READY pgs=3625 cs=0 l=1 rev1=1 crypto rx=0x7f7b0006e590 tx=0x7f7ae8009270 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:54:51.629 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.611+0000 7f7afcff9640 1 -- 172.21.15.42:0/711503345 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f7af0018280 con 0x7f7b0009bf90 2024-09-16T03:54:51.629 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.611+0000 7f7afcff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:54:51.629 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.611+0000 7f7afcff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:54:51.845 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.839+0000 7f7b06585640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:54:51.845 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.839+0000 7f7b06585640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:54:51.845 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.839+0000 7f7b06585640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:54:51.845 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.839+0000 7f7b06585640 1 -- 172.21.15.42:0/711503345 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f7b001329e0 con 0x7f7b0009bf90 2024-09-16T03:54:51.846 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.843+0000 7f7afcff9640 1 -- 172.21.15.42:0/711503345 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1562 (secure 0 0 0) 0x7f7af00b6400 con 0x7f7b0009bf90 2024-09-16T03:54:51.846 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.843+0000 7f7afcff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:54:51.846 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.843+0000 7f7afcff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:54:51.846 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":110,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:54:51.848 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.843+0000 7f7b06585640 10 librados: watch_flush enter 2024-09-16T03:54:51.848 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.843+0000 7f7b06585640 10 librados: watch_flush exit 2024-09-16T03:54:51.849 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.843+0000 7f7b06585640 20 client.824517.objecter shutdown clearing up homeless session... 2024-09-16T03:54:51.849 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.843+0000 7f7b06585640 10 client.824517.objecter successfully canceled tick 2024-09-16T03:54:51.849 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.843+0000 7f7b06585640 1 -- 172.21.15.42:0/711503345 >> v2:172.21.15.154:6812/254845342 conn(0x7f7ae00377c0 msgr2=0x7f7ae0039c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:54:51.849 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.843+0000 7f7b06585640 1 --2- 172.21.15.42:0/711503345 >> v2:172.21.15.154:6812/254845342 conn(0x7f7ae00377c0 0x7f7ae0039c80 secure :-1 s=READY pgs=3625 cs=0 l=1 rev1=1 crypto rx=0x7f7b0006e590 tx=0x7f7ae8009270 comp rx=0 tx=0).stop 2024-09-16T03:54:51.849 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.843+0000 7f7b06585640 10 monclient: shutdown 2024-09-16T03:54:51.849 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.843+0000 7f7b06585640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:54:51.849 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.843+0000 7f7b06585640 1 -- 172.21.15.42:0/711503345 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7b0009bf90 msgr2=0x7f7b00135e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:54:51.849 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.843+0000 7f7b06585640 1 --2- 172.21.15.42:0/711503345 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7b0009bf90 0x7f7b00135e80 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f7af00ae040 tx=0x7f7af001cdb0 comp rx=0 tx=0).stop 2024-09-16T03:54:51.849 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.843+0000 7f7b06585640 1 -- 172.21.15.42:0/711503345 shutdown_connections 2024-09-16T03:54:51.849 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.843+0000 7f7b06585640 1 --2- 172.21.15.42:0/711503345 >> v2:172.21.15.154:6812/254845342 conn(0x7f7ae00377c0 0x7f7ae0039c80 unknown :-1 s=CLOSED pgs=3625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:54:51.849 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.843+0000 7f7b06585640 1 --2- 172.21.15.42:0/711503345 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7b000a1330 0x7f7b00136900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:54:51.849 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.843+0000 7f7b06585640 1 --2- 172.21.15.42:0/711503345 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7b0009fb00 0x7f7b001363c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:54:51.849 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.843+0000 7f7b06585640 1 --2- 172.21.15.42:0/711503345 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7b0009bf90 0x7f7b00135e80 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:54:51.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.843+0000 7f7b06585640 1 -- 172.21.15.42:0/711503345 >> 172.21.15.42:0/711503345 conn(0x7f7b000a2420 msgr2=0x7f7b0013be20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:54:51.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.843+0000 7f7b06585640 1 -- 172.21.15.42:0/711503345 shutdown_connections 2024-09-16T03:54:51.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.843+0000 7f7b06585640 1 -- 172.21.15.42:0/711503345 wait complete. 2024-09-16T03:54:51.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:51.843+0000 7f7b06585640 1 librados: shutdown 2024-09-16T03:54:54.866 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:54:55.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:54.999+0000 7f4e44f0e640 10 monclient: get_monmap_and_config 2024-09-16T03:54:55.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:54.999+0000 7f4e44f0e640 10 monclient: build_initial_monmap 2024-09-16T03:54:55.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:54.999+0000 7f4e44f0e640 10 monclient: monmap: 2024-09-16T03:54:55.003 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:54:55.003 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:54:55.003 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:54:55.001193+0000 2024-09-16T03:54:55.004 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:54:55.001193+0000 2024-09-16T03:54:55.004 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:54:55.004 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:54:55.004 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:54:55.004 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:54:55.004 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:54:55.004 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:54:55.004 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:54:55.004 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:54:55.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:54.999+0000 7f4e44f0e640 1 Processor -- start 2024-09-16T03:54:55.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:54.999+0000 7f4e44f0e640 1 -- start start 2024-09-16T03:54:55.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:54.999+0000 7f4e44f0e640 10 monclient: init 2024-09-16T03:54:55.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:54.999+0000 7f4e44f0e640 10 monclient: _reopen_session rank -1 2024-09-16T03:54:55.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:54.999+0000 7f4e44f0e640 10 monclient: _add_conns ranks=[2,0,4,1,3] 2024-09-16T03:54:55.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:54.999+0000 7f4e44f0e640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4e400a1330 0x7f4e400a1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:54:55.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:54.999+0000 7f4e44f0e640 10 monclient(hunting): picked mon.noname-b con 0x7f4e400a1330 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:54:55.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:54.999+0000 7f4e44f0e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4e4006dac0 addr v1:172.21.15.42:6789/0 2024-09-16T03:54:55.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:54.999+0000 7f4e44f0e640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f4e400a1c50 0x7f4e40135810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:54:55.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:54.999+0000 7f4e44f0e640 10 monclient(hunting): picked mon.noname-c con 0x7f4e400a1c50 addr v2:172.21.15.154:3300/0 2024-09-16T03:54:55.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:54.999+0000 7f4e44f0e640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4e4009c090 con 0x7f4e4006dac0 2024-09-16T03:54:55.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:54.999+0000 7f4e44f0e640 10 monclient(hunting): start opening mon connection 2024-09-16T03:54:55.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:54.999+0000 7f4e44f0e640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f4e4009c890 con 0x7f4e400a1c50 2024-09-16T03:54:55.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:54.999+0000 7f4e44f0e640 10 monclient(hunting): start opening mon connection 2024-09-16T03:54:55.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:54.999+0000 7f4e44f0e640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f4e4009f610 con 0x7f4e400a1330 2024-09-16T03:54:55.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:54.999+0000 7f4e44f0e640 10 monclient(hunting): _renew_subs 2024-09-16T03:54:55.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:54.999+0000 7f4e44f0e640 10 monclient(hunting): authenticate will time out at 1741.971171s 2024-09-16T03:54:55.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:54.999+0000 7f4e3e59c640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4e400a1330 0x7f4e400a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:54:55.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:54.999+0000 7f4e3dd9b640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f4e4006dac0 0x7f4e40135100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:36122/0 (socket says 172.21.15.42:36122) 2024-09-16T03:54:55.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:54.999+0000 7f4e3e59c640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4e400a1330 0x7f4e400a1710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:45668/0 (socket says 172.21.15.42:45668) 2024-09-16T03:54:55.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:54.999+0000 7f4e3dd9b640 1 -- 172.21.15.42:0/3529888599 learned_addr learned my addr 172.21.15.42:0/3529888599 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:54:55.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:54.999+0000 7f4e3ed9d640 1 -- 172.21.15.42:0/3529888599 >> v2:172.21.15.154:3300/0 conn(0x7f4e400a1c50 msgr2=0x7f4e40135810 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T03:54:55.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:54.999+0000 7f4e3e59c640 10 monclient(hunting): get_auth_request con 0x7f4e400a1330 auth_method 0 2024-09-16T03:54:55.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:54.999+0000 7f4e3e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:54:55.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:54.999+0000 7f4e3e59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:54:55.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:54.999+0000 7f4e3e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:54:55.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:54.999+0000 7f4e3ed9d640 1 --2- 172.21.15.42:0/3529888599 >> v2:172.21.15.154:3300/0 conn(0x7f4e400a1c50 0x7f4e40135810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:54:55.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:54.999+0000 7f4e3e59c640 1 -- 172.21.15.42:0/3529888599 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4e400a1330 msgr2=0x7f4e400a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T03:54:55.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:54.999+0000 7f4e3e59c640 1 -- 172.21.15.42:0/3529888599 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4e400a1330 msgr2=0x7f4e400a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:54:55.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:54.999+0000 7f4e3e59c640 1 --2- 172.21.15.42:0/3529888599 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4e400a1330 0x7f4e400a1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:54:55.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:54.999+0000 7f4e3e59c640 1 --2- 172.21.15.42:0/3529888599 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4e400a1330 0x7f4e400a1710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:54:55.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:55.203+0000 7f4e3ed9d640 1 -- 172.21.15.42:0/3529888599 >> v2:172.21.15.154:3300/0 conn(0x7f4e400a1c50 msgr2=0x7f4e40135810 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T03:54:55.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:55.203+0000 7f4e3ed9d640 1 --2- 172.21.15.42:0/3529888599 >> v2:172.21.15.154:3300/0 conn(0x7f4e400a1c50 0x7f4e40135810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T03:54:55.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:55.203+0000 7f4e3e59c640 1 --2- 172.21.15.42:0/3529888599 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4e400a1330 0x7f4e400a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:54:55.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:55.203+0000 7f4e3e59c640 10 monclient(hunting): get_auth_request con 0x7f4e400a1330 auth_method 0 2024-09-16T03:54:55.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:55.203+0000 7f4e3e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:54:55.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:55.203+0000 7f4e3e59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:54:55.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:55.203+0000 7f4e3e59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:54:55.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:55.203+0000 7f4e3e59c640 1 -- 172.21.15.42:0/3529888599 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4e400a1330 msgr2=0x7f4e400a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T03:54:55.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:55.203+0000 7f4e3e59c640 1 -- 172.21.15.42:0/3529888599 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4e400a1330 msgr2=0x7f4e400a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:54:55.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:55.203+0000 7f4e3e59c640 1 --2- 172.21.15.42:0/3529888599 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4e400a1330 0x7f4e400a1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:54:55.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:55.203+0000 7f4e3e59c640 1 --2- 172.21.15.42:0/3529888599 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4e400a1330 0x7f4e400a1710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T03:54:55.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:55.607+0000 7f4e3ed9d640 1 -- 172.21.15.42:0/3529888599 >> v2:172.21.15.154:3300/0 conn(0x7f4e400a1c50 msgr2=0x7f4e40135810 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T03:54:55.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:55.607+0000 7f4e3ed9d640 1 --2- 172.21.15.42:0/3529888599 >> v2:172.21.15.154:3300/0 conn(0x7f4e400a1c50 0x7f4e40135810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T03:54:55.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:55.607+0000 7f4e3e59c640 1 --2- 172.21.15.42:0/3529888599 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4e400a1330 0x7f4e400a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:54:55.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:55.607+0000 7f4e3e59c640 10 monclient(hunting): get_auth_request con 0x7f4e400a1330 auth_method 0 2024-09-16T03:54:55.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:55.607+0000 7f4e3e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:54:55.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:55.607+0000 7f4e3e59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:54:55.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:55.607+0000 7f4e3e59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:54:55.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:55.607+0000 7f4e3e59c640 1 -- 172.21.15.42:0/3529888599 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4e400a1330 msgr2=0x7f4e400a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T03:54:55.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:55.607+0000 7f4e3e59c640 1 -- 172.21.15.42:0/3529888599 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4e400a1330 msgr2=0x7f4e400a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:54:55.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:55.607+0000 7f4e3e59c640 1 --2- 172.21.15.42:0/3529888599 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4e400a1330 0x7f4e400a1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:54:55.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:55.607+0000 7f4e3e59c640 1 --2- 172.21.15.42:0/3529888599 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4e400a1330 0x7f4e400a1710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T03:54:56.414 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:56.411+0000 7f4e3ed9d640 1 -- 172.21.15.42:0/3529888599 >> v2:172.21.15.154:3300/0 conn(0x7f4e400a1c50 msgr2=0x7f4e40135810 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T03:54:56.414 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:56.411+0000 7f4e3ed9d640 1 --2- 172.21.15.42:0/3529888599 >> v2:172.21.15.154:3300/0 conn(0x7f4e400a1c50 0x7f4e40135810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T03:54:56.414 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:56.411+0000 7f4e3e59c640 1 --2- 172.21.15.42:0/3529888599 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4e400a1330 0x7f4e400a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:54:56.414 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:56.411+0000 7f4e3e59c640 10 monclient(hunting): get_auth_request con 0x7f4e400a1330 auth_method 0 2024-09-16T03:54:56.414 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:56.411+0000 7f4e3e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:54:56.414 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:56.411+0000 7f4e3e59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:54:56.414 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:56.411+0000 7f4e3e59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:54:56.414 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:56.411+0000 7f4e3e59c640 1 -- 172.21.15.42:0/3529888599 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4e400a1330 msgr2=0x7f4e400a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T03:54:56.415 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:56.411+0000 7f4e3e59c640 1 -- 172.21.15.42:0/3529888599 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4e400a1330 msgr2=0x7f4e400a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:54:56.415 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:56.411+0000 7f4e3e59c640 1 --2- 172.21.15.42:0/3529888599 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4e400a1330 0x7f4e400a1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:54:56.415 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:56.411+0000 7f4e3e59c640 1 --2- 172.21.15.42:0/3529888599 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4e400a1330 0x7f4e400a1710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T03:54:58.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:57.999+0000 7f4e2ffff640 10 monclient(hunting): tick 2024-09-16T03:54:58.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:57.999+0000 7f4e2ffff640 10 monclient(hunting): _check_auth_tickets 2024-09-16T03:54:58.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:57.999+0000 7f4e2ffff640 1 monclient(hunting): continuing hunt 2024-09-16T03:54:58.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:57.999+0000 7f4e2ffff640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T03:54:58.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:57.999+0000 7f4e2ffff640 1 -- 172.21.15.42:0/3529888599 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4e400a1330 msgr2=0x7f4e400a1710 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:54:58.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:57.999+0000 7f4e2ffff640 1 --2- 172.21.15.42:0/3529888599 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4e400a1330 0x7f4e400a1710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:54:58.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:57.999+0000 7f4e2ffff640 1 -- 172.21.15.42:0/3529888599 >> v2:172.21.15.154:3300/0 conn(0x7f4e400a1c50 msgr2=0x7f4e40135810 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:54:58.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:57.999+0000 7f4e2ffff640 1 --2- 172.21.15.42:0/3529888599 >> v2:172.21.15.154:3300/0 conn(0x7f4e400a1c50 0x7f4e40135810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:54:58.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:57.999+0000 7f4e2ffff640 1 -- 172.21.15.42:0/3529888599 >> v1:172.21.15.42:6789/0 conn(0x7f4e4006dac0 legacy=0x7f4e40135100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:54:58.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:57.999+0000 7f4e2ffff640 10 monclient: _add_conns ranks=[3,1] 2024-09-16T03:54:58.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:57.999+0000 7f4e2ffff640 1 --2- 172.21.15.42:0/3529888599 >> v2:172.21.15.42:3300/0 conn(0x7f4e200036d0 0x7f4e20005b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:54:58.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:57.999+0000 7f4e2ffff640 10 monclient(hunting): picked mon.noname-a con 0x7f4e200036d0 addr v2:172.21.15.42:3300/0 2024-09-16T03:54:58.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:57.999+0000 7f4e2ffff640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4e200061a0 addr v1:172.21.15.154:6789/0 2024-09-16T03:54:58.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:57.999+0000 7f4e2ffff640 1 -- 172.21.15.42:0/3529888599 --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4e4009f8f0 con 0x7f4e200061a0 2024-09-16T03:54:58.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:57.999+0000 7f4e2ffff640 10 monclient(hunting): start opening mon connection 2024-09-16T03:54:58.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:57.999+0000 7f4e2ffff640 1 -- 172.21.15.42:0/3529888599 --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f4e4009c890 con 0x7f4e200036d0 2024-09-16T03:54:58.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:57.999+0000 7f4e2ffff640 10 monclient(hunting): _renew_subs 2024-09-16T03:54:58.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:57.999+0000 7f4e3dd9b640 1 -- 172.21.15.42:0/3529888599 >> v1:172.21.15.154:6789/0 conn(0x7f4e200061a0 legacy=0x7f4e20008640 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.154:6789/0 2024-09-16T03:54:58.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:57.999+0000 7f4e3e59c640 1 --2- 172.21.15.42:0/3529888599 >> v2:172.21.15.42:3300/0 conn(0x7f4e200036d0 0x7f4e20005b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:54:58.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:57.999+0000 7f4e3e59c640 10 monclient(hunting): get_auth_request con 0x7f4e200036d0 auth_method 0 2024-09-16T03:54:58.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:57.999+0000 7f4e3e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:54:58.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:57.999+0000 7f4e3e59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:54:58.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:57.999+0000 7f4e3e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:54:58.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:58.003+0000 7f4e3e59c640 1 -- 172.21.15.42:0/3529888599 >> v2:172.21.15.42:3300/0 conn(0x7f4e200036d0 msgr2=0x7f4e20005b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T03:54:58.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:58.003+0000 7f4e3e59c640 1 -- 172.21.15.42:0/3529888599 >> v2:172.21.15.42:3300/0 conn(0x7f4e200036d0 msgr2=0x7f4e20005b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:54:58.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:58.003+0000 7f4e3e59c640 1 --2- 172.21.15.42:0/3529888599 >> v2:172.21.15.42:3300/0 conn(0x7f4e200036d0 0x7f4e20005b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:54:58.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:58.003+0000 7f4e3e59c640 1 --2- 172.21.15.42:0/3529888599 >> v2:172.21.15.42:3300/0 conn(0x7f4e200036d0 0x7f4e20005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:54:58.205 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:58.203+0000 7f4e3dd9b640 1 -- 172.21.15.42:0/3529888599 >> v1:172.21.15.154:6789/0 conn(0x7f4e200061a0 legacy=0x7f4e20008640 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.154:6789/0 2024-09-16T03:54:58.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:58.207+0000 7f4e3e59c640 1 --2- 172.21.15.42:0/3529888599 >> v2:172.21.15.42:3300/0 conn(0x7f4e200036d0 0x7f4e20005b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:54:58.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:58.207+0000 7f4e3e59c640 10 monclient(hunting): get_auth_request con 0x7f4e200036d0 auth_method 0 2024-09-16T03:54:58.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:58.207+0000 7f4e3e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:54:58.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:58.207+0000 7f4e3e59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:54:58.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:58.207+0000 7f4e3e59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:54:58.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:58.207+0000 7f4e3e59c640 1 -- 172.21.15.42:0/3529888599 >> v2:172.21.15.42:3300/0 conn(0x7f4e200036d0 msgr2=0x7f4e20005b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T03:54:58.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:58.207+0000 7f4e3e59c640 1 -- 172.21.15.42:0/3529888599 >> v2:172.21.15.42:3300/0 conn(0x7f4e200036d0 msgr2=0x7f4e20005b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:54:58.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:58.207+0000 7f4e3e59c640 1 --2- 172.21.15.42:0/3529888599 >> v2:172.21.15.42:3300/0 conn(0x7f4e200036d0 0x7f4e20005b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:54:58.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:58.207+0000 7f4e3e59c640 1 --2- 172.21.15.42:0/3529888599 >> v2:172.21.15.42:3300/0 conn(0x7f4e200036d0 0x7f4e20005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T03:54:58.609 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:58.607+0000 7f4e3dd9b640 1 -- 172.21.15.42:0/3529888599 >> v1:172.21.15.154:6789/0 conn(0x7f4e200061a0 legacy=0x7f4e20008640 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.154:6789/0 2024-09-16T03:54:58.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:58.611+0000 7f4e3e59c640 1 --2- 172.21.15.42:0/3529888599 >> v2:172.21.15.42:3300/0 conn(0x7f4e200036d0 0x7f4e20005b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:54:58.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:58.611+0000 7f4e3e59c640 10 monclient(hunting): get_auth_request con 0x7f4e200036d0 auth_method 0 2024-09-16T03:54:58.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:58.611+0000 7f4e3e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:54:58.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:58.611+0000 7f4e3e59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:54:58.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:58.611+0000 7f4e3e59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:54:58.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:58.611+0000 7f4e3e59c640 1 -- 172.21.15.42:0/3529888599 >> v2:172.21.15.42:3300/0 conn(0x7f4e200036d0 msgr2=0x7f4e20005b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T03:54:58.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:58.611+0000 7f4e3e59c640 1 -- 172.21.15.42:0/3529888599 >> v2:172.21.15.42:3300/0 conn(0x7f4e200036d0 msgr2=0x7f4e20005b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:54:58.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:58.611+0000 7f4e3e59c640 1 --2- 172.21.15.42:0/3529888599 >> v2:172.21.15.42:3300/0 conn(0x7f4e200036d0 0x7f4e20005b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:54:58.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:58.611+0000 7f4e3e59c640 1 --2- 172.21.15.42:0/3529888599 >> v2:172.21.15.42:3300/0 conn(0x7f4e200036d0 0x7f4e20005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T03:54:59.413 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:59.411+0000 7f4e3dd9b640 1 -- 172.21.15.42:0/3529888599 >> v1:172.21.15.154:6789/0 conn(0x7f4e200061a0 legacy=0x7f4e20008640 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.154:6789/0 2024-09-16T03:54:59.418 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:59.415+0000 7f4e3e59c640 1 --2- 172.21.15.42:0/3529888599 >> v2:172.21.15.42:3300/0 conn(0x7f4e200036d0 0x7f4e20005b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:54:59.418 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:59.415+0000 7f4e3e59c640 10 monclient(hunting): get_auth_request con 0x7f4e200036d0 auth_method 0 2024-09-16T03:54:59.418 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:59.415+0000 7f4e3e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:54:59.418 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:59.415+0000 7f4e3e59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:54:59.418 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:59.415+0000 7f4e3e59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:54:59.418 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:59.415+0000 7f4e3e59c640 1 -- 172.21.15.42:0/3529888599 >> v2:172.21.15.42:3300/0 conn(0x7f4e200036d0 msgr2=0x7f4e20005b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T03:54:59.418 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:59.415+0000 7f4e3e59c640 1 -- 172.21.15.42:0/3529888599 >> v2:172.21.15.42:3300/0 conn(0x7f4e200036d0 msgr2=0x7f4e20005b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:54:59.418 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:59.415+0000 7f4e3e59c640 1 --2- 172.21.15.42:0/3529888599 >> v2:172.21.15.42:3300/0 conn(0x7f4e200036d0 0x7f4e20005b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:54:59.418 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:54:59.415+0000 7f4e3e59c640 1 --2- 172.21.15.42:0/3529888599 >> v2:172.21.15.42:3300/0 conn(0x7f4e200036d0 0x7f4e20005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T03:54:59.485 INFO:tasks.ceph.osd.1.smithi042.stderr:2024-09-16T03:54:59.479+0000 7f4561b97640 -1 snap_mapper.add_oid found existing snaps mapped on 300:68c55315:test-rados-api-smithi154-28216-80::foo:2, removing 2024-09-16T03:54:59.485 INFO:tasks.ceph.osd.4.smithi154.stderr:2024-09-16T03:54:59.486+0000 7fdda1af2640 -1 snap_mapper.add_oid found existing snaps mapped on 300:68c55315:test-rados-api-smithi154-28216-80::foo:2, removing 2024-09-16T03:55:01.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:00.999+0000 7f4e2ffff640 10 monclient(hunting): tick 2024-09-16T03:55:01.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:00.999+0000 7f4e2ffff640 10 monclient(hunting): _check_auth_tickets 2024-09-16T03:55:01.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:00.999+0000 7f4e2ffff640 1 monclient(hunting): continuing hunt 2024-09-16T03:55:01.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:00.999+0000 7f4e2ffff640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T03:55:01.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:00.999+0000 7f4e2ffff640 1 -- 172.21.15.42:0/3529888599 >> v2:172.21.15.42:3300/0 conn(0x7f4e200036d0 msgr2=0x7f4e20005b50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:55:01.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:00.999+0000 7f4e2ffff640 1 --2- 172.21.15.42:0/3529888599 >> v2:172.21.15.42:3300/0 conn(0x7f4e200036d0 0x7f4e20005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:55:01.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:00.999+0000 7f4e2ffff640 1 -- 172.21.15.42:0/3529888599 >> v1:172.21.15.154:6789/0 conn(0x7f4e200061a0 legacy=0x7f4e20008640 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:55:01.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:00.999+0000 7f4e3ed9d640 1 -- 172.21.15.42:0/3529888599 reap_dead start 2024-09-16T03:55:01.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:00.999+0000 7f4e2ffff640 10 monclient: _add_conns ranks=[2,3,0,1,4] 2024-09-16T03:55:01.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.003+0000 7f4e2ffff640 1 --2- 172.21.15.42:0/3529888599 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4e20009ed0 0x7f4e20005b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:55:01.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.003+0000 7f4e2ffff640 10 monclient(hunting): picked mon.noname-b con 0x7f4e20009ed0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:55:01.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.003+0000 7f4e2ffff640 1 --2- 172.21.15.42:0/3529888599 >> v2:172.21.15.42:3300/0 conn(0x7f4e2000a350 0x7f4e20008640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:55:01.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.003+0000 7f4e2ffff640 10 monclient(hunting): picked mon.noname-a con 0x7f4e2000a350 addr v2:172.21.15.42:3300/0 2024-09-16T03:55:01.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.003+0000 7f4e2ffff640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4e2000a750 addr v1:172.21.15.42:6789/0 2024-09-16T03:55:01.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.003+0000 7f4e2ffff640 1 -- 172.21.15.42:0/3529888599 --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4e2000af90 con 0x7f4e2000a750 2024-09-16T03:55:01.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.003+0000 7f4e2ffff640 10 monclient(hunting): start opening mon connection 2024-09-16T03:55:01.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.003+0000 7f4e2ffff640 1 -- 172.21.15.42:0/3529888599 --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f4e4009c890 con 0x7f4e2000a350 2024-09-16T03:55:01.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.003+0000 7f4e2ffff640 10 monclient(hunting): start opening mon connection 2024-09-16T03:55:01.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.003+0000 7f4e2ffff640 1 -- 172.21.15.42:0/3529888599 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f4e4009f610 con 0x7f4e20009ed0 2024-09-16T03:55:01.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.003+0000 7f4e2ffff640 10 monclient(hunting): _renew_subs 2024-09-16T03:55:01.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.003+0000 7f4e3e59c640 1 --2- 172.21.15.42:0/3529888599 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4e20009ed0 0x7f4e20005b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:55:01.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.003+0000 7f4e3e59c640 10 monclient(hunting): get_auth_request con 0x7f4e20009ed0 auth_method 0 2024-09-16T03:55:01.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.003+0000 7f4e3e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:55:01.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.003+0000 7f4e3e59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:55:01.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.003+0000 7f4e3e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:55:01.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.003+0000 7f4e3dd9b640 1 --2- 172.21.15.42:0/3529888599 >> v2:172.21.15.42:3300/0 conn(0x7f4e2000a350 0x7f4e20008640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:55:01.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.003+0000 7f4e3dd9b640 10 monclient(hunting): get_auth_request con 0x7f4e2000a350 auth_method 0 2024-09-16T03:55:01.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.003+0000 7f4e3dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:55:01.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.003+0000 7f4e3dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:55:01.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.003+0000 7f4e3dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:55:01.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.003+0000 7f4e3e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:55:01.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.003+0000 7f4e3e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:55:01.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.003+0000 7f4e3e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:55:01.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.003+0000 7f4e3dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:55:01.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.003+0000 7f4e3dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:55:01.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.003+0000 7f4e3dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:55:01.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.003+0000 7f4e3e59c640 10 monclient(hunting): handle_auth_done global_id 854162 payload 274 2024-09-16T03:55:01.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.003+0000 7f4e3e59c640 1 -- 172.21.15.42:0/3529888599 >> v2:172.21.15.42:3300/0 conn(0x7f4e2000a350 msgr2=0x7f4e20008640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:55:01.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.003+0000 7f4e3d59a640 1 -- 172.21.15.42:0/3529888599 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1696289453 0 0) 0x7f4e2000af90 con 0x7f4e2000a750 2024-09-16T03:55:01.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.003+0000 7f4e3e59c640 1 --2- 172.21.15.42:0/3529888599 >> v2:172.21.15.42:3300/0 conn(0x7f4e2000a350 0x7f4e20008640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:55:01.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.003+0000 7f4e3e59c640 1 -- 172.21.15.42:0/3529888599 >> v1:172.21.15.42:6789/0 conn(0x7f4e2000a750 legacy=0x7f4e20008e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:55:01.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.003+0000 7f4e3e59c640 10 monclient: _finish_hunting 0 2024-09-16T03:55:01.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.003+0000 7f4e3e59c640 1 monclient: found mon.noname-b 2024-09-16T03:55:01.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.003+0000 7f4e3e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:55:01.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.003+0000 7f4e3e59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T03:55:01.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.003+0000 7f4e3e59c640 1 -- 172.21.15.42:0/3529888599 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e4009f8f0 con 0x7f4e20009ed0 2024-09-16T03:55:01.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.003+0000 7f4e3d59a640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T03:55:01.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.003+0000 7f4e3dd9b640 1 --2- 172.21.15.42:0/3529888599 >> v2:172.21.15.42:3300/0 conn(0x7f4e2000a350 0x7f4e20008640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:55:01.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.007+0000 7f4e3e59c640 1 --2- 172.21.15.42:0/3529888599 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4e20009ed0 0x7f4e20005b50 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f4e30000ee0 tx=0x7f4e3001dcc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=818f46c144a1873a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:55:01.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.007+0000 7f4e3d59a640 1 -- 172.21.15.42:0/3529888599 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e3000f040 con 0x7f4e20009ed0 2024-09-16T03:55:01.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.011+0000 7f4e3d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:55:01.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.011+0000 7f4e3d59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:55:01.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.011+0000 7f4e3d59a640 10 monclient: dump: 2024-09-16T03:55:01.013 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:55:01.013 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:55:01.013 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:01.014 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:01.014 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:55:01.014 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:55:01.014 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:55:01.014 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:55:01.014 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:55:01.014 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:55:01.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.011+0000 7f4e3d59a640 10 monclient: _finish_auth 0 2024-09-16T03:55:01.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.011+0000 7f4e3d59a640 10 monclient: _check_auth_tickets 2024-09-16T03:55:01.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.011+0000 7f4e3d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:55:01.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.011+0000 7f4e3d59a640 1 -- 172.21.15.42:0/3529888599 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e30004230 con 0x7f4e20009ed0 2024-09-16T03:55:01.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.011+0000 7f4e3d59a640 10 monclient: handle_config config(0 keys) 2024-09-16T03:55:01.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.011+0000 7f4e3d59a640 1 -- 172.21.15.42:0/3529888599 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e300a6950 con 0x7f4e20009ed0 2024-09-16T03:55:01.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.011+0000 7f4e44f0e640 5 monclient: authenticate success, global_id 854162 2024-09-16T03:55:01.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.011+0000 7f4e44f0e640 10 monclient: get_monmap_and_config success 2024-09-16T03:55:01.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.011+0000 7f4e44f0e640 4 set_mon_vals no callback set 2024-09-16T03:55:01.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.011+0000 7f4e3d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:55:01.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.011+0000 7f4e3d59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:55:01.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.011+0000 7f4e3d59a640 10 monclient: dump: 2024-09-16T03:55:01.015 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:55:01.015 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:55:01.015 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:01.015 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:01.015 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:55:01.015 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:55:01.015 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:55:01.016 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:55:01.016 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:55:01.016 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:55:01.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.011+0000 7f4e44f0e640 10 monclient: shutdown 2024-09-16T03:55:01.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.011+0000 7f4e44f0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:55:01.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.011+0000 7f4e44f0e640 1 -- 172.21.15.42:0/3529888599 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4e20009ed0 msgr2=0x7f4e20005b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:55:01.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.011+0000 7f4e44f0e640 1 --2- 172.21.15.42:0/3529888599 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4e20009ed0 0x7f4e20005b50 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f4e30000ee0 tx=0x7f4e3001dcc0 comp rx=0 tx=0).stop 2024-09-16T03:55:01.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.011+0000 7f4e44f0e640 1 -- 172.21.15.42:0/3529888599 shutdown_connections 2024-09-16T03:55:01.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.011+0000 7f4e44f0e640 1 --2- 172.21.15.42:0/3529888599 >> v2:172.21.15.42:3300/0 conn(0x7f4e2000a350 0x7f4e20008640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:55:01.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.011+0000 7f4e44f0e640 1 --2- 172.21.15.42:0/3529888599 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4e20009ed0 0x7f4e20005b50 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:55:01.018 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.011+0000 7f4e44f0e640 1 -- 172.21.15.42:0/3529888599 >> 172.21.15.42:0/3529888599 conn(0x7f4e400a2420 msgr2=0x7f4e400a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:55:01.018 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.011+0000 7f4e44f0e640 1 -- 172.21.15.42:0/3529888599 shutdown_connections 2024-09-16T03:55:01.018 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.011+0000 7f4e44f0e640 1 -- 172.21.15.42:0/3529888599 wait complete. 2024-09-16T03:55:01.018 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.011+0000 7f4e44f0e640 10 monclient: build_initial_monmap 2024-09-16T03:55:01.018 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.011+0000 7f4e44f0e640 10 monclient: monmap: 2024-09-16T03:55:01.018 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:55:01.018 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:55:01.018 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:55:01.018 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:55:01.018 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:55:01.018 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:55:01.018 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:55:01.018 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:55:01.019 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:55:01.019 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:55:01.019 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.011+0000 7f4e44f0e640 1 librados: starting msgr at 2024-09-16T03:55:01.019 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.011+0000 7f4e44f0e640 1 librados: starting objecter 2024-09-16T03:55:01.019 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.015+0000 7f4e44f0e640 1 Processor -- start 2024-09-16T03:55:01.019 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.015+0000 7f4e44f0e640 1 -- start start 2024-09-16T03:55:01.019 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.015+0000 7f4e44f0e640 1 librados: setting wanted keys 2024-09-16T03:55:01.019 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.015+0000 7f4e44f0e640 1 librados: calling monclient init 2024-09-16T03:55:01.019 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.015+0000 7f4e44f0e640 10 monclient: init 2024-09-16T03:55:01.022 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.015+0000 7f4e44f0e640 10 monclient: _reopen_session rank -1 2024-09-16T03:55:01.022 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.015+0000 7f4e44f0e640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:55:01.022 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.015+0000 7f4e44f0e640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4e20009ed0 0x7f4e401cc3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:55:01.022 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.015+0000 7f4e44f0e640 10 monclient(hunting): picked mon.noname-c con 0x7f4e20009ed0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:55:01.022 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.015+0000 7f4e44f0e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4e2000a350 0x7f4e401cc8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:55:01.022 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.015+0000 7f4e44f0e640 10 monclient(hunting): picked mon.noname-b con 0x7f4e2000a350 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:55:01.023 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.015+0000 7f4e44f0e640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4e2000a750 0x7f4e40123e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:55:01.023 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.015+0000 7f4e44f0e640 10 monclient(hunting): picked mon.noname-a con 0x7f4e2000a750 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:55:01.023 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.015+0000 7f4e44f0e640 10 monclient(hunting): start opening mon connection 2024-09-16T03:55:01.023 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.015+0000 7f4e44f0e640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f4e400a3170 con 0x7f4e2000a750 2024-09-16T03:55:01.023 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.015+0000 7f4e44f0e640 10 monclient(hunting): start opening mon connection 2024-09-16T03:55:01.023 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.015+0000 7f4e44f0e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f4e4009fc30 con 0x7f4e2000a350 2024-09-16T03:55:01.023 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.015+0000 7f4e44f0e640 10 monclient(hunting): start opening mon connection 2024-09-16T03:55:01.023 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.015+0000 7f4e44f0e640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f4e400a32f0 con 0x7f4e20009ed0 2024-09-16T03:55:01.023 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.015+0000 7f4e44f0e640 10 monclient(hunting): _renew_subs 2024-09-16T03:55:01.023 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.015+0000 7f4e44f0e640 10 monclient(hunting): authenticate will time out at 1747.984871s 2024-09-16T03:55:01.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.015+0000 7f4e3dd9b640 1 -- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4e2000a350 msgr2=0x7f4e401cc8e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T03:55:01.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.015+0000 7f4e3dd9b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4e2000a350 0x7f4e401cc8e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:55:01.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.015+0000 7f4e3ed9d640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4e2000a750 0x7f4e40123e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:55:01.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.015+0000 7f4e3e59c640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4e20009ed0 0x7f4e401cc3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:55:01.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.015+0000 7f4e3ed9d640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4e2000a750 0x7f4e40123e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:45298/0 (socket says 172.21.15.42:45298) 2024-09-16T03:55:01.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.015+0000 7f4e3ed9d640 1 -- 172.21.15.42:0/3921068086 learned_addr learned my addr 172.21.15.42:0/3921068086 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:55:01.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.015+0000 7f4e3ed9d640 10 monclient(hunting): get_auth_request con 0x7f4e2000a750 auth_method 0 2024-09-16T03:55:01.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.015+0000 7f4e3ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:55:01.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.015+0000 7f4e3ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:55:01.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.015+0000 7f4e3ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:55:01.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.015+0000 7f4e3e59c640 10 monclient(hunting): get_auth_request con 0x7f4e20009ed0 auth_method 0 2024-09-16T03:55:01.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.015+0000 7f4e3e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:55:01.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.015+0000 7f4e3e59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:55:01.025 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.015+0000 7f4e3e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:55:01.025 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.015+0000 7f4e3ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:55:01.025 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.015+0000 7f4e3ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:55:01.025 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.015+0000 7f4e3ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:55:01.025 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.015+0000 7f4e3e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:55:01.025 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.015+0000 7f4e3e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:55:01.025 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.015+0000 7f4e3e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:55:01.025 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.015+0000 7f4e3ed9d640 10 monclient(hunting): handle_auth_done global_id 824526 payload 824 2024-09-16T03:55:01.025 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.015+0000 7f4e3ed9d640 1 -- 172.21.15.42:0/3921068086 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4e20009ed0 msgr2=0x7f4e401cc3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:55:01.025 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.015+0000 7f4e3ed9d640 1 --2- 172.21.15.42:0/3921068086 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4e20009ed0 0x7f4e401cc3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:55:01.025 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.015+0000 7f4e3ed9d640 1 -- 172.21.15.42:0/3921068086 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4e2000a350 msgr2=0x7f4e401cc8e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:55:01.025 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.015+0000 7f4e3ed9d640 1 --2- 172.21.15.42:0/3921068086 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4e2000a350 0x7f4e401cc8e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:55:01.025 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.015+0000 7f4e3ed9d640 10 monclient: _finish_hunting 0 2024-09-16T03:55:01.025 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.015+0000 7f4e3ed9d640 1 monclient: found mon.noname-a 2024-09-16T03:55:01.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.015+0000 7f4e3ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:55:01.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.015+0000 7f4e3ed9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:55:01.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.015+0000 7f4e3ed9d640 1 -- 172.21.15.42:0/3921068086 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e40127810 con 0x7f4e2000a750 2024-09-16T03:55:01.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.015+0000 7f4e3e59c640 1 --2- 172.21.15.42:0/3921068086 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4e20009ed0 0x7f4e401cc3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:55:01.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.015+0000 7f4e3ed9d640 1 --2- 172.21.15.42:0/3921068086 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4e2000a750 0x7f4e40123e40 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f4e20009e80 tx=0x7f4e38009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:55:01.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.015+0000 7f4e2effd640 10 client.?.objecter ms_handle_connect 0x7f4e2000a750 2024-09-16T03:55:01.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.015+0000 7f4e2effd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:55:01.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.015+0000 7f4e2effd640 1 -- 172.21.15.42:0/3921068086 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e38017070 con 0x7f4e2000a750 2024-09-16T03:55:01.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.015+0000 7f4e2effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:55:01.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.015+0000 7f4e2effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:55:01.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.015+0000 7f4e2effd640 10 monclient: dump: 2024-09-16T03:55:01.026 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:55:01.026 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:55:01.027 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:01.027 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:01.027 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:55:01.027 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:55:01.027 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:55:01.027 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:55:01.027 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:55:01.027 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:55:01.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.015+0000 7f4e2effd640 10 monclient: _finish_auth 0 2024-09-16T03:55:01.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.015+0000 7f4e2effd640 10 monclient: _check_auth_tickets 2024-09-16T03:55:01.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.015+0000 7f4e2effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:55:01.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.015+0000 7f4e2effd640 1 -- 172.21.15.42:0/3921068086 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e380060d0 con 0x7f4e2000a750 2024-09-16T03:55:01.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.015+0000 7f4e2effd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:55:01.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.015+0000 7f4e2effd640 1 -- 172.21.15.42:0/3921068086 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e380064a0 con 0x7f4e2000a750 2024-09-16T03:55:01.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.015+0000 7f4e2effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:55:01.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.015+0000 7f4e2effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:55:01.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.015+0000 7f4e2effd640 10 monclient: dump: 2024-09-16T03:55:01.028 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:55:01.028 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:55:01.028 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:01.028 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:01.028 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:55:01.028 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:55:01.028 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:55:01.029 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:55:01.029 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:55:01.029 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:55:01.029 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.015+0000 7f4e3cd99640 4 set_mon_vals no callback set 2024-09-16T03:55:01.029 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.015+0000 7f4e44f0e640 5 monclient: authenticate success, global_id 824526 2024-09-16T03:55:01.029 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.015+0000 7f4e44f0e640 10 monclient: _renew_subs 2024-09-16T03:55:01.029 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.015+0000 7f4e44f0e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:55:01.029 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.015+0000 7f4e44f0e640 1 -- 172.21.15.42:0/3921068086 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e40126640 con 0x7f4e2000a750 2024-09-16T03:55:01.030 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.015+0000 7f4e44f0e640 10 client.824526.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:55:01.030 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.015+0000 7f4e44f0e640 10 monclient: _renew_subs 2024-09-16T03:55:01.030 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.015+0000 7f4e44f0e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:55:01.030 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.015+0000 7f4e44f0e640 1 -- 172.21.15.42:0/3921068086 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e40135560 con 0x7f4e2000a750 2024-09-16T03:55:01.030 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.015+0000 7f4e44f0e640 1 librados: init done 2024-09-16T03:55:01.030 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.019+0000 7f4e44f0e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:55:01.030 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.019+0000 7f4e44f0e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:55:01.030 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.019+0000 7f4e44f0e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:55:01.030 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.019+0000 7f4e44f0e640 1 -- 172.21.15.42:0/3921068086 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e4009d2d0 con 0x7f4e2000a750 2024-09-16T03:55:01.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.019+0000 7f4e2effd640 1 -- 172.21.15.42:0/3921068086 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f4e38006640 con 0x7f4e2000a750 2024-09-16T03:55:01.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.019+0000 7f4e2effd640 1 --2- 172.21.15.42:0/3921068086 >> v2:172.21.15.154:6812/254845342 conn(0x7f4e2003c290 0x7f4e2003e750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:55:01.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.019+0000 7f4e2effd640 1 -- 172.21.15.42:0/3921068086 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(387..387 src has 286..387) ==== 5328+0+0 (secure 0 0 0) 0x7f4e38047ee0 con 0x7f4e2000a750 2024-09-16T03:55:01.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.019+0000 7f4e2effd640 10 client.824526.objecter ms_dispatch 0x7f4e40000ba0 osd_map(387..387 src has 286..387) 2024-09-16T03:55:01.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.019+0000 7f4e2effd640 3 client.824526.objecter handle_osd_map got epochs [387,387] > 0 2024-09-16T03:55:01.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.019+0000 7f4e2effd640 3 client.824526.objecter handle_osd_map decoding full epoch 387 2024-09-16T03:55:01.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.019+0000 7f4e2effd640 20 client.824526.objecter dump_active .. 0 homeless 2024-09-16T03:55:01.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.019+0000 7f4e3e59c640 1 --2- 172.21.15.42:0/3921068086 >> v2:172.21.15.154:6812/254845342 conn(0x7f4e2003c290 0x7f4e2003e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:55:01.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.019+0000 7f4e3e59c640 10 monclient: get_auth_request con 0x7f4e2003c290 auth_method 0 2024-09-16T03:55:01.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.019+0000 7f4e2effd640 10 client.824526.objecter ms_handle_connect 0x7f4e2003c290 2024-09-16T03:55:01.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.019+0000 7f4e3e59c640 1 --2- 172.21.15.42:0/3921068086 >> v2:172.21.15.154:6812/254845342 conn(0x7f4e2003c290 0x7f4e2003e750 secure :-1 s=READY pgs=3626 cs=0 l=1 rev1=1 crypto rx=0x7f4e300af000 tx=0x7f4e300a8070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:55:01.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.023+0000 7f4e2effd640 1 -- 172.21.15.42:0/3921068086 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f4e38010360 con 0x7f4e2000a750 2024-09-16T03:55:01.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.023+0000 7f4e2effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:55:01.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.023+0000 7f4e2effd640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:55:01.261 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.255+0000 7f4e44f0e640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:55:01.261 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.255+0000 7f4e44f0e640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:55:01.261 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.255+0000 7f4e44f0e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:55:01.262 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.255+0000 7f4e44f0e640 1 -- 172.21.15.42:0/3921068086 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f4e4009eee0 con 0x7f4e2000a750 2024-09-16T03:55:01.262 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.255+0000 7f4e2effd640 1 -- 172.21.15.42:0/3921068086 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1555 (secure 0 0 0) 0x7f4e38010e80 con 0x7f4e2000a750 2024-09-16T03:55:01.262 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.255+0000 7f4e2effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:55:01.262 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.255+0000 7f4e2effd640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:55:01.262 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":114,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:55:01.263 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.259+0000 7f4e44f0e640 10 librados: watch_flush enter 2024-09-16T03:55:01.263 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.259+0000 7f4e44f0e640 10 librados: watch_flush exit 2024-09-16T03:55:01.263 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.259+0000 7f4e44f0e640 20 client.824526.objecter shutdown clearing up homeless session... 2024-09-16T03:55:01.263 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.259+0000 7f4e44f0e640 10 client.824526.objecter successfully canceled tick 2024-09-16T03:55:01.263 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.259+0000 7f4e44f0e640 1 -- 172.21.15.42:0/3921068086 >> v2:172.21.15.154:6812/254845342 conn(0x7f4e2003c290 msgr2=0x7f4e2003e750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:55:01.264 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.259+0000 7f4e44f0e640 1 --2- 172.21.15.42:0/3921068086 >> v2:172.21.15.154:6812/254845342 conn(0x7f4e2003c290 0x7f4e2003e750 secure :-1 s=READY pgs=3626 cs=0 l=1 rev1=1 crypto rx=0x7f4e300af000 tx=0x7f4e300a8070 comp rx=0 tx=0).stop 2024-09-16T03:55:01.264 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.259+0000 7f4e44f0e640 10 monclient: shutdown 2024-09-16T03:55:01.264 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.259+0000 7f4e44f0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:55:01.264 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.259+0000 7f4e44f0e640 1 -- 172.21.15.42:0/3921068086 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4e2000a750 msgr2=0x7f4e40123e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:55:01.264 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.259+0000 7f4e44f0e640 1 --2- 172.21.15.42:0/3921068086 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4e2000a750 0x7f4e40123e40 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f4e20009e80 tx=0x7f4e38009f90 comp rx=0 tx=0).stop 2024-09-16T03:55:01.264 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.259+0000 7f4e44f0e640 1 -- 172.21.15.42:0/3921068086 shutdown_connections 2024-09-16T03:55:01.264 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.259+0000 7f4e44f0e640 1 --2- 172.21.15.42:0/3921068086 >> v2:172.21.15.154:6812/254845342 conn(0x7f4e2003c290 0x7f4e2003e750 unknown :-1 s=CLOSED pgs=3626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:55:01.264 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.259+0000 7f4e44f0e640 1 --2- 172.21.15.42:0/3921068086 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4e2000a750 0x7f4e40123e40 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:55:01.264 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.259+0000 7f4e44f0e640 1 --2- 172.21.15.42:0/3921068086 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4e2000a350 0x7f4e401cc8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:55:01.264 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.259+0000 7f4e44f0e640 1 --2- 172.21.15.42:0/3921068086 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4e20009ed0 0x7f4e401cc3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:55:01.264 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.259+0000 7f4e44f0e640 1 -- 172.21.15.42:0/3921068086 >> 172.21.15.42:0/3921068086 conn(0x7f4e400a2420 msgr2=0x7f4e400a45b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:55:01.264 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.259+0000 7f4e44f0e640 1 -- 172.21.15.42:0/3921068086 shutdown_connections 2024-09-16T03:55:01.264 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.259+0000 7f4e44f0e640 1 -- 172.21.15.42:0/3921068086 wait complete. 2024-09-16T03:55:01.265 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.259+0000 7f4e44f0e640 1 librados: shutdown 2024-09-16T03:55:01.279 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T03:55:01.279 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-16T03:55:01.280 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T03:55:01.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.395+0000 7f12877f2640 10 monclient: get_monmap_and_config 2024-09-16T03:55:01.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.395+0000 7f12877f2640 10 monclient: build_initial_monmap 2024-09-16T03:55:01.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.395+0000 7f12877f2640 10 monclient: monmap: 2024-09-16T03:55:01.398 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:55:01.398 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:55:01.398 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:55:01.396477+0000 2024-09-16T03:55:01.398 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:55:01.396477+0000 2024-09-16T03:55:01.398 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:55:01.398 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:55:01.398 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:55:01.398 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:55:01.398 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:55:01.398 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:55:01.398 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:55:01.398 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:55:01.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.395+0000 7f12877f2640 1 Processor -- start 2024-09-16T03:55:01.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.395+0000 7f12877f2640 1 -- start start 2024-09-16T03:55:01.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.395+0000 7f12877f2640 10 monclient: init 2024-09-16T03:55:01.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.395+0000 7f12877f2640 10 monclient: _reopen_session rank -1 2024-09-16T03:55:01.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.395+0000 7f12877f2640 10 monclient: _add_conns ranks=[1,4,0,3,2] 2024-09-16T03:55:01.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.395+0000 7f12877f2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f12800a39b0 addr v1:172.21.15.154:6789/0 2024-09-16T03:55:01.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.395+0000 7f12877f2640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f12800a3d90 0x7f128013c380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:55:01.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.395+0000 7f12877f2640 10 monclient(hunting): picked mon.noname-c con 0x7f12800a3d90 addr v2:172.21.15.154:3300/0 2024-09-16T03:55:01.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.395+0000 7f12877f2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f128013c8c0 addr v1:172.21.15.42:6789/0 2024-09-16T03:55:01.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.395+0000 7f12877f2640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1280134290 con 0x7f128013c8c0 2024-09-16T03:55:01.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.395+0000 7f12877f2640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f128009c140 con 0x7f12800a39b0 2024-09-16T03:55:01.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.395+0000 7f12877f2640 10 monclient(hunting): start opening mon connection 2024-09-16T03:55:01.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.395+0000 7f12877f2640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f128009c420 con 0x7f12800a3d90 2024-09-16T03:55:01.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.395+0000 7f12877f2640 10 monclient(hunting): _renew_subs 2024-09-16T03:55:01.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.395+0000 7f12877f2640 10 monclient(hunting): authenticate will time out at 1748.365817s 2024-09-16T03:55:01.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.395+0000 7f1285d8f640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f128013c8c0 0x7f128013ecb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:36136/0 (socket says 172.21.15.42:36136) 2024-09-16T03:55:01.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.395+0000 7f1285d8f640 1 -- 172.21.15.42:0/3172937560 learned_addr learned my addr 172.21.15.42:0/3172937560 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:55:01.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.395+0000 7f128558e640 1 -- 172.21.15.42:0/3172937560 >> v1:172.21.15.154:6789/0 conn(0x7f12800a39b0 legacy=0x7f1280133b80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.154:6789/0 2024-09-16T03:55:01.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.395+0000 7f1284d8d640 1 -- 172.21.15.42:0/3172937560 >> v2:172.21.15.154:3300/0 conn(0x7f12800a3d90 msgr2=0x7f128013c380 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T03:55:01.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.395+0000 7f1284d8d640 1 --2- 172.21.15.42:0/3172937560 >> v2:172.21.15.154:3300/0 conn(0x7f12800a3d90 0x7f128013c380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:55:01.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.395+0000 7f1277fff640 1 -- 172.21.15.42:0/3172937560 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2371966229 0 0) 0x7f1280134290 con 0x7f128013c8c0 2024-09-16T03:55:01.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.395+0000 7f1277fff640 10 monclient(hunting): _negotiate 2024-09-16T03:55:01.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.395+0000 7f1277fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:55:01.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.395+0000 7f1277fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:55:01.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.395+0000 7f1277fff640 10 monclient(hunting): my global_id is 824529 2024-09-16T03:55:01.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.395+0000 7f1277fff640 1 -- 172.21.15.42:0/3172937560 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f126c003530 con 0x7f128013c8c0 2024-09-16T03:55:01.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.395+0000 7f1277fff640 1 -- 172.21.15.42:0/3172937560 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2014296154 0 0) 0x7f126c003530 con 0x7f128013c8c0 2024-09-16T03:55:01.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.395+0000 7f1277fff640 1 -- 172.21.15.42:0/3172937560 >> v2:172.21.15.154:3300/0 conn(0x7f12800a3d90 msgr2=0x7f128013c380 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:55:01.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.395+0000 7f1277fff640 1 --2- 172.21.15.42:0/3172937560 >> v2:172.21.15.154:3300/0 conn(0x7f12800a3d90 0x7f128013c380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:55:01.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.395+0000 7f1277fff640 1 -- 172.21.15.42:0/3172937560 >> v1:172.21.15.154:6789/0 conn(0x7f12800a39b0 legacy=0x7f1280133b80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:55:01.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.395+0000 7f1277fff640 10 monclient: _finish_hunting 0 2024-09-16T03:55:01.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.395+0000 7f1277fff640 1 monclient: found mon.noname-a-legacy 2024-09-16T03:55:01.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.395+0000 7f1277fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:55:01.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.395+0000 7f1277fff640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.42:6789/0 2024-09-16T03:55:01.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.395+0000 7f1277fff640 1 -- 172.21.15.42:0/3172937560 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f128006dd60 con 0x7f128013c8c0 2024-09-16T03:55:01.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.395+0000 7f1277fff640 10 monclient: _finish_auth 0 2024-09-16T03:55:01.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.395+0000 7f1277fff640 10 monclient: _check_auth_tickets 2024-09-16T03:55:01.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.395+0000 7f1277fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:55:01.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.395+0000 7f1277fff640 1 -- 172.21.15.42:0/3172937560 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2177413734 0 0) 0x7f1278001a00 con 0x7f128013c8c0 2024-09-16T03:55:01.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.395+0000 7f12877f2640 5 monclient: authenticate success, global_id 824529 2024-09-16T03:55:01.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.395+0000 7f12877f2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:55:01.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.395+0000 7f1277fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:55:01.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.395+0000 7f1277fff640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T03:55:01.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.395+0000 7f1277fff640 10 monclient: dump: 2024-09-16T03:55:01.406 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:55:01.406 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:55:01.406 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:01.406 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:01.406 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:55:01.406 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:55:01.406 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:55:01.406 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:55:01.406 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:55:01.407 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:55:01.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.395+0000 7f1277fff640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] but i'm connected to v1:172.21.15.42:6789/0, reconnecting 2024-09-16T03:55:01.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.395+0000 7f1277fff640 10 monclient: _reopen_session rank -1 2024-09-16T03:55:01.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.395+0000 7f1277fff640 1 -- 172.21.15.42:0/3172937560 >> v1:172.21.15.42:6789/0 conn(0x7f128013c8c0 legacy=0x7f128013ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:55:01.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.395+0000 7f1277fff640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:55:01.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.395+0000 7f1277fff640 1 --2- 172.21.15.42:0/3172937560 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f126c0039c0 0x7f126c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:55:01.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.395+0000 7f1277fff640 10 monclient(hunting): picked mon.b con 0x7f126c0039c0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:55:01.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.395+0000 7f1277fff640 1 --2- 172.21.15.42:0/3172937560 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f126c004810 0x7f126c004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:55:01.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.395+0000 7f1277fff640 10 monclient(hunting): picked mon.a con 0x7f126c004810 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:55:01.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.395+0000 7f1277fff640 1 --2- 172.21.15.42:0/3172937560 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f126c0053d0 0x7f126c00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:55:01.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.395+0000 7f1277fff640 10 monclient(hunting): picked mon.c con 0x7f126c0053d0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:55:01.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.395+0000 7f1277fff640 10 monclient(hunting): start opening mon connection 2024-09-16T03:55:01.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.395+0000 7f1277fff640 1 -- 172.21.15.42:0/3172937560 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f128009c420 con 0x7f126c004810 2024-09-16T03:55:01.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.395+0000 7f1277fff640 10 monclient(hunting): start opening mon connection 2024-09-16T03:55:01.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.395+0000 7f1277fff640 1 -- 172.21.15.42:0/3172937560 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f126c00c270 con 0x7f126c0039c0 2024-09-16T03:55:01.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.395+0000 7f1277fff640 10 monclient(hunting): start opening mon connection 2024-09-16T03:55:01.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.395+0000 7f1277fff640 1 -- 172.21.15.42:0/3172937560 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f126c00c470 con 0x7f126c0053d0 2024-09-16T03:55:01.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.395+0000 7f1277fff640 10 monclient(hunting): _renew_subs 2024-09-16T03:55:01.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.395+0000 7f1277fff640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:55:01.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.395+0000 7f12877f2640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-16T03:55:01.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.395+0000 7f1284d8d640 1 --2- 172.21.15.42:0/3172937560 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f126c004810 0x7f126c004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:55:01.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.395+0000 7f128558e640 1 -- 172.21.15.42:0/3172937560 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f126c0039c0 msgr2=0x7f126c003dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T03:55:01.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.395+0000 7f1284d8d640 10 monclient(hunting): get_auth_request con 0x7f126c004810 auth_method 0 2024-09-16T03:55:01.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.395+0000 7f1284d8d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:55:01.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.395+0000 7f1284d8d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:55:01.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.395+0000 7f1284d8d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:55:01.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.395+0000 7f1285d8f640 1 --2- 172.21.15.42:0/3172937560 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f126c0053d0 0x7f126c00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:55:01.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.395+0000 7f128558e640 1 --2- 172.21.15.42:0/3172937560 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f126c0039c0 0x7f126c003dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:55:01.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.395+0000 7f1285d8f640 10 monclient(hunting): get_auth_request con 0x7f126c0053d0 auth_method 0 2024-09-16T03:55:01.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.395+0000 7f1285d8f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:55:01.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.395+0000 7f1285d8f640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:55:01.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.395+0000 7f1285d8f640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:55:01.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.395+0000 7f1284d8d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:55:01.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.395+0000 7f1284d8d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:55:01.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.395+0000 7f1284d8d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T03:55:01.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.395+0000 7f1285d8f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:55:01.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.395+0000 7f1285d8f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:55:01.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.395+0000 7f1285d8f640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T03:55:01.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.395+0000 7f1284d8d640 10 monclient(hunting): handle_auth_done global_id 824529 payload 293 2024-09-16T03:55:01.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.395+0000 7f1284d8d640 1 -- 172.21.15.42:0/3172937560 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f126c0053d0 msgr2=0x7f126c00b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:55:01.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.395+0000 7f1284d8d640 1 --2- 172.21.15.42:0/3172937560 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f126c0053d0 0x7f126c00b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:55:01.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.395+0000 7f1284d8d640 1 -- 172.21.15.42:0/3172937560 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f126c0039c0 msgr2=0x7f126c003dc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:55:01.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.395+0000 7f1284d8d640 1 --2- 172.21.15.42:0/3172937560 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f126c0039c0 0x7f126c003dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:55:01.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.395+0000 7f1284d8d640 10 monclient: _finish_hunting 0 2024-09-16T03:55:01.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.395+0000 7f1284d8d640 1 monclient: found mon.a 2024-09-16T03:55:01.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.395+0000 7f1284d8d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:55:01.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.395+0000 7f1284d8d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:55:01.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.395+0000 7f1284d8d640 1 -- 172.21.15.42:0/3172937560 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f126c003530 con 0x7f126c004810 2024-09-16T03:55:01.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.395+0000 7f1284d8d640 10 monclient: _finish_auth 0 2024-09-16T03:55:01.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.395+0000 7f1284d8d640 10 monclient: _check_auth_tickets 2024-09-16T03:55:01.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.395+0000 7f1284d8d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:55:01.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.395+0000 7f1285d8f640 1 --2- 172.21.15.42:0/3172937560 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f126c0053d0 0x7f126c00b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:55:01.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.395+0000 7f1285d8f640 1 -- 172.21.15.42:0/3172937560 reap_dead start 2024-09-16T03:55:01.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.399+0000 7f1284d8d640 1 --2- 172.21.15.42:0/3172937560 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f126c004810 0x7f126c004c90 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f1270003940 tx=0x7f127001c420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2cc156155ef093ec server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:55:01.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.399+0000 7f1277fff640 1 -- 172.21.15.42:0/3172937560 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1270009520 con 0x7f126c004810 2024-09-16T03:55:01.411 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.399+0000 7f1277fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:55:01.411 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.399+0000 7f1277fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:55:01.411 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.399+0000 7f1277fff640 10 monclient: dump: 2024-09-16T03:55:01.411 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:55:01.411 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:55:01.411 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:01.411 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:01.411 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:55:01.411 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:55:01.411 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:55:01.411 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:55:01.411 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:55:01.411 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:55:01.411 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.399+0000 7f1277fff640 1 -- 172.21.15.42:0/3172937560 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1270003bc0 con 0x7f126c004810 2024-09-16T03:55:01.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.399+0000 7f1277fff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:55:01.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.399+0000 7f1277fff640 1 -- 172.21.15.42:0/3172937560 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1270005070 con 0x7f126c004810 2024-09-16T03:55:01.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.399+0000 7f12877f2640 10 monclient: get_monmap_and_config success 2024-09-16T03:55:01.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.399+0000 7f12877f2640 4 set_mon_vals no callback set 2024-09-16T03:55:01.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.399+0000 7f1277fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:55:01.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.399+0000 7f1277fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:55:01.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.399+0000 7f1277fff640 10 monclient: dump: 2024-09-16T03:55:01.412 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:55:01.412 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:55:01.412 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:01.412 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:01.412 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:55:01.412 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:55:01.413 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:55:01.413 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:55:01.413 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:55:01.413 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:55:01.420 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.399+0000 7f12877f2640 10 monclient: shutdown 2024-09-16T03:55:01.420 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.399+0000 7f12877f2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:55:01.420 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.399+0000 7f12877f2640 1 -- 172.21.15.42:0/3172937560 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f126c004810 msgr2=0x7f126c004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:55:01.420 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.399+0000 7f12877f2640 1 --2- 172.21.15.42:0/3172937560 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f126c004810 0x7f126c004c90 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f1270003940 tx=0x7f127001c420 comp rx=0 tx=0).stop 2024-09-16T03:55:01.420 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.399+0000 7f12877f2640 1 -- 172.21.15.42:0/3172937560 shutdown_connections 2024-09-16T03:55:01.420 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.399+0000 7f12877f2640 1 --2- 172.21.15.42:0/3172937560 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f126c004810 0x7f126c004c90 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:55:01.420 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.399+0000 7f12877f2640 1 -- 172.21.15.42:0/3172937560 >> 172.21.15.42:0/3172937560 conn(0x7f12800a9fb0 msgr2=0x7f12800aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:55:01.420 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.399+0000 7f12877f2640 1 -- 172.21.15.42:0/3172937560 shutdown_connections 2024-09-16T03:55:01.420 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.399+0000 7f12877f2640 1 -- 172.21.15.42:0/3172937560 wait complete. 2024-09-16T03:55:01.420 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.399+0000 7f12877f2640 10 monclient: build_initial_monmap 2024-09-16T03:55:01.420 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.399+0000 7f12877f2640 10 monclient: monmap: 2024-09-16T03:55:01.420 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:55:01.421 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:55:01.421 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:55:01.421 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:55:01.421 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:55:01.421 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:55:01.421 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:55:01.421 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:55:01.421 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:55:01.421 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:55:01.421 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.399+0000 7f12877f2640 1 librados: starting msgr at 2024-09-16T03:55:01.421 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.399+0000 7f12877f2640 1 librados: starting objecter 2024-09-16T03:55:01.421 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.399+0000 7f12877f2640 1 Processor -- start 2024-09-16T03:55:01.421 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.399+0000 7f12877f2640 1 -- start start 2024-09-16T03:55:01.421 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.399+0000 7f12877f2640 1 librados: setting wanted keys 2024-09-16T03:55:01.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.399+0000 7f12877f2640 1 librados: calling monclient init 2024-09-16T03:55:01.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.399+0000 7f12877f2640 10 monclient: init 2024-09-16T03:55:01.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.399+0000 7f12877f2640 10 monclient: _reopen_session rank -1 2024-09-16T03:55:01.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.399+0000 7f12877f2640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:55:01.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.399+0000 7f12877f2640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f126c004810 0x7f12800a01e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:55:01.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.399+0000 7f12877f2640 10 monclient(hunting): picked mon.noname-b con 0x7f126c004810 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:55:01.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.399+0000 7f12877f2640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f12800a0720 0x7f12800a0b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:55:01.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.399+0000 7f12877f2640 10 monclient(hunting): picked mon.noname-a con 0x7f12800a0720 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:55:01.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.399+0000 7f12877f2640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f12800a10a0 0x7f12801d5ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:55:01.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.399+0000 7f12877f2640 10 monclient(hunting): picked mon.noname-c con 0x7f12800a10a0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:55:01.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.399+0000 7f12877f2640 10 monclient(hunting): start opening mon connection 2024-09-16T03:55:01.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.399+0000 7f12877f2640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f1280137ba0 con 0x7f12800a0720 2024-09-16T03:55:01.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.399+0000 7f12877f2640 10 monclient(hunting): start opening mon connection 2024-09-16T03:55:01.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.399+0000 7f12877f2640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f1280137a20 con 0x7f126c004810 2024-09-16T03:55:01.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.399+0000 7f12877f2640 10 monclient(hunting): start opening mon connection 2024-09-16T03:55:01.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.399+0000 7f12877f2640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f1280137d20 con 0x7f12800a10a0 2024-09-16T03:55:01.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.399+0000 7f12877f2640 10 monclient(hunting): _renew_subs 2024-09-16T03:55:01.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.399+0000 7f12877f2640 10 monclient(hunting): authenticate will time out at 1748.371719s 2024-09-16T03:55:01.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.399+0000 7f1284d8d640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f12800a0720 0x7f12800a0b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:55:01.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.399+0000 7f1285d8f640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f12800a10a0 0x7f12801d5ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:55:01.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.399+0000 7f1284d8d640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f12800a0720 0x7f12800a0b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:45320/0 (socket says 172.21.15.42:45320) 2024-09-16T03:55:01.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.399+0000 7f1285d8f640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f12800a10a0 0x7f12801d5ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:45728/0 (socket says 172.21.15.42:45728) 2024-09-16T03:55:01.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.399+0000 7f128558e640 1 -- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f126c004810 msgr2=0x7f12800a01e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T03:55:01.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.399+0000 7f1284d8d640 1 -- 172.21.15.42:0/84364580 learned_addr learned my addr 172.21.15.42:0/84364580 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:55:01.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.399+0000 7f1284d8d640 10 monclient(hunting): get_auth_request con 0x7f12800a0720 auth_method 0 2024-09-16T03:55:01.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.399+0000 7f1284d8d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:55:01.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.399+0000 7f1284d8d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:55:01.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.399+0000 7f1284d8d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:55:01.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.399+0000 7f1285d8f640 10 monclient(hunting): get_auth_request con 0x7f12800a10a0 auth_method 0 2024-09-16T03:55:01.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.399+0000 7f1285d8f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:55:01.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.399+0000 7f1285d8f640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:55:01.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.399+0000 7f1285d8f640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:55:01.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.399+0000 7f128558e640 1 --2- 172.21.15.42:0/84364580 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f126c004810 0x7f12800a01e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:55:01.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.399+0000 7f1285d8f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:55:01.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.399+0000 7f1285d8f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:55:01.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.399+0000 7f1285d8f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:55:01.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.399+0000 7f1284d8d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:55:01.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.399+0000 7f1284d8d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:55:01.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.403+0000 7f1284d8d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:55:01.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.403+0000 7f1285d8f640 10 monclient(hunting): handle_auth_done global_id 854168 payload 824 2024-09-16T03:55:01.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.403+0000 7f1285d8f640 1 -- 172.21.15.42:0/84364580 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f126c004810 msgr2=0x7f12800a01e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:55:01.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.403+0000 7f1285d8f640 1 --2- 172.21.15.42:0/84364580 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f126c004810 0x7f12800a01e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:55:01.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.403+0000 7f1285d8f640 1 -- 172.21.15.42:0/84364580 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f12800a0720 msgr2=0x7f12800a0b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:55:01.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.403+0000 7f1285d8f640 1 --2- 172.21.15.42:0/84364580 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f12800a0720 0x7f12800a0b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:55:01.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.403+0000 7f1285d8f640 10 monclient: _finish_hunting 0 2024-09-16T03:55:01.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.403+0000 7f1285d8f640 1 monclient: found mon.noname-c 2024-09-16T03:55:01.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.403+0000 7f1285d8f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:55:01.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.403+0000 7f1285d8f640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:55:01.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.403+0000 7f1285d8f640 1 -- 172.21.15.42:0/84364580 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1280120450 con 0x7f12800a10a0 2024-09-16T03:55:01.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.403+0000 7f1285d8f640 1 --2- 172.21.15.42:0/84364580 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f12800a10a0 0x7f12801d5ac0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f12800a7a80 tx=0x7f1278007430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:55:01.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.403+0000 7f1275ffb640 10 client.?.objecter ms_handle_connect 0x7f12800a10a0 2024-09-16T03:55:01.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.403+0000 7f1275ffb640 10 client.?.objecter resend_mon_ops 2024-09-16T03:55:01.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.403+0000 7f1275ffb640 1 -- 172.21.15.42:0/84364580 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f127800a480 con 0x7f12800a10a0 2024-09-16T03:55:01.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.403+0000 7f1275ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:55:01.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.403+0000 7f1275ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:55:01.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.403+0000 7f1275ffb640 10 monclient: dump: 2024-09-16T03:55:01.426 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:55:01.426 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:55:01.426 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:01.426 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:01.426 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:55:01.426 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:55:01.426 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:55:01.427 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:55:01.427 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:55:01.427 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:55:01.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.403+0000 7f1275ffb640 10 monclient: _finish_auth 0 2024-09-16T03:55:01.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.403+0000 7f1275ffb640 10 monclient: _check_auth_tickets 2024-09-16T03:55:01.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.403+0000 7f1275ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:55:01.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.403+0000 7f1275ffb640 1 -- 172.21.15.42:0/84364580 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1278018070 con 0x7f12800a10a0 2024-09-16T03:55:01.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.403+0000 7f12877f2640 5 monclient: authenticate success, global_id 854168 2024-09-16T03:55:01.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.403+0000 7f12877f2640 10 monclient: _renew_subs 2024-09-16T03:55:01.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.403+0000 7f12877f2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:55:01.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.403+0000 7f12877f2640 1 -- 172.21.15.42:0/84364580 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12800a2740 con 0x7f12800a10a0 2024-09-16T03:55:01.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.403+0000 7f1275ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T03:55:01.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.403+0000 7f1275ffb640 1 -- 172.21.15.42:0/84364580 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1278013650 con 0x7f12800a10a0 2024-09-16T03:55:01.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.403+0000 7f1275ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:55:01.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.403+0000 7f1275ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:55:01.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.403+0000 7f1275ffb640 10 monclient: dump: 2024-09-16T03:55:01.428 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:55:01.428 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:55:01.428 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:01.428 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:01.428 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:55:01.428 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:55:01.428 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:55:01.428 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:55:01.428 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:55:01.428 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:55:01.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.403+0000 7f12777fe640 4 set_mon_vals no callback set 2024-09-16T03:55:01.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.403+0000 7f12877f2640 10 client.854168.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:55:01.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.403+0000 7f12877f2640 10 monclient: _renew_subs 2024-09-16T03:55:01.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.403+0000 7f12877f2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:55:01.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.403+0000 7f12877f2640 1 -- 172.21.15.42:0/84364580 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f1280133300 con 0x7f12800a10a0 2024-09-16T03:55:01.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.403+0000 7f12877f2640 1 librados: init done 2024-09-16T03:55:01.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.403+0000 7f12877f2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:55:01.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.403+0000 7f12877f2640 1 --2- 172.21.15.42:0/84364580 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f128013a370 0x7f128013a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:55:01.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.403+0000 7f12877f2640 10 monclient(hunting): start opening mon connection 2024-09-16T03:55:01.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.403+0000 7f12877f2640 1 -- 172.21.15.42:0/84364580 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f1280131010 con 0x7f128013a370 2024-09-16T03:55:01.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.403+0000 7f1275ffb640 1 -- 172.21.15.42:0/84364580 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f12780137f0 con 0x7f12800a10a0 2024-09-16T03:55:01.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.403+0000 7f128558e640 1 --2- 172.21.15.42:0/84364580 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f128013a370 0x7f128013a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:55:01.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.403+0000 7f128558e640 10 monclient: get_auth_request con 0x7f128013a370 auth_method 0 2024-09-16T03:55:01.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.403+0000 7f128558e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:55:01.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.403+0000 7f128558e640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:55:01.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.403+0000 7f128558e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:55:01.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.403+0000 7f128558e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:55:01.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.403+0000 7f128558e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:55:01.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.403+0000 7f128558e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:55:01.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.403+0000 7f1275ffb640 1 --2- 172.21.15.42:0/84364580 >> v2:172.21.15.154:6812/254845342 conn(0x7f12640377c0 0x7f1264039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:55:01.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.403+0000 7f1275ffb640 1 -- 172.21.15.42:0/84364580 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(387..387 src has 286..387) ==== 5328+0+0 (secure 0 0 0) 0x7f1278049310 con 0x7f12800a10a0 2024-09-16T03:55:01.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.403+0000 7f1275ffb640 10 client.854168.objecter ms_dispatch 0x7f1280000ba0 osd_map(387..387 src has 286..387) 2024-09-16T03:55:01.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.403+0000 7f1275ffb640 3 client.854168.objecter handle_osd_map got epochs [387,387] > 0 2024-09-16T03:55:01.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.403+0000 7f1275ffb640 3 client.854168.objecter handle_osd_map decoding full epoch 387 2024-09-16T03:55:01.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.403+0000 7f128558e640 10 monclient(hunting): handle_auth_done global_id 824535 payload 824 2024-09-16T03:55:01.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.403+0000 7f128558e640 1 -- 172.21.15.42:0/84364580 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1280134230 con 0x7f128013a370 2024-09-16T03:55:01.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.403+0000 7f1275ffb640 20 client.854168.objecter dump_active .. 0 homeless 2024-09-16T03:55:01.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.403+0000 7f128558e640 1 --2- 172.21.15.42:0/84364580 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f128013a370 0x7f128013a750 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f127c00b430 tx=0x7f127c00b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:55:01.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.403+0000 7f1275ffb640 10 client.854168.objecter ms_handle_connect 0x7f128013a370 2024-09-16T03:55:01.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.403+0000 7f1275ffb640 10 client.854168.objecter resend_mon_ops 2024-09-16T03:55:01.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.403+0000 7f1275ffb640 1 -- 172.21.15.42:0/84364580 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f127c003e60 con 0x7f128013a370 2024-09-16T03:55:01.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.403+0000 7f1275ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:55:01.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.403+0000 7f1275ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:55:01.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.403+0000 7f1275ffb640 10 monclient: dump: 2024-09-16T03:55:01.431 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:55:01.431 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:55:01.431 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:01.431 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:01.431 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:55:01.432 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:55:01.432 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:55:01.432 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:55:01.432 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:55:01.432 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:55:01.432 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.403+0000 7f1284d8d640 1 --2- 172.21.15.42:0/84364580 >> v2:172.21.15.154:6812/254845342 conn(0x7f12640377c0 0x7f1264039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:55:01.432 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.403+0000 7f1284d8d640 10 monclient: get_auth_request con 0x7f12640377c0 auth_method 0 2024-09-16T03:55:01.432 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.403+0000 7f1275ffb640 1 -- 172.21.15.42:0/84364580 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f127c015070 con 0x7f128013a370 2024-09-16T03:55:01.432 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.403+0000 7f1275ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:55:01.432 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.403+0000 7f1275ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:55:01.432 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.403+0000 7f1275ffb640 1 -- 172.21.15.42:0/84364580 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f128013a370 msgr2=0x7f128013a750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:55:01.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.403+0000 7f1275ffb640 1 --2- 172.21.15.42:0/84364580 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f128013a370 0x7f128013a750 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f127c00b430 tx=0x7f127c00b8f0 comp rx=0 tx=0).stop 2024-09-16T03:55:01.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.403+0000 7f1275ffb640 1 -- 172.21.15.42:0/84364580 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f128013a370 msgr2=0x7f128013a750 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:55:01.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.403+0000 7f1275ffb640 1 --2- 172.21.15.42:0/84364580 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f128013a370 0x7f128013a750 secure :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f127c00b430 tx=0x7f127c00b8f0 comp rx=0 tx=0).stop 2024-09-16T03:55:01.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.403+0000 7f1284d8d640 1 --2- 172.21.15.42:0/84364580 >> v2:172.21.15.154:6812/254845342 conn(0x7f12640377c0 0x7f1264039c80 secure :-1 s=READY pgs=3627 cs=0 l=1 rev1=1 crypto rx=0x7f1280120660 tx=0x7f1270005040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:55:01.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.403+0000 7f1275ffb640 10 client.854168.objecter ms_handle_connect 0x7f12640377c0 2024-09-16T03:55:01.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.411+0000 7f12877f2640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:55:01.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.411+0000 7f12877f2640 1 --2- 172.21.15.42:0/84364580 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f128013ac90 0x7f12801da020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:55:01.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.411+0000 7f12877f2640 10 monclient(hunting): start opening mon connection 2024-09-16T03:55:01.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.411+0000 7f12877f2640 1 -- 172.21.15.42:0/84364580 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f12800a5750 con 0x7f128013ac90 2024-09-16T03:55:01.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.411+0000 7f128558e640 1 --2- 172.21.15.42:0/84364580 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f128013ac90 0x7f12801da020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:55:01.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.411+0000 7f128558e640 10 monclient: get_auth_request con 0x7f128013ac90 auth_method 0 2024-09-16T03:55:01.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.411+0000 7f128558e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:55:01.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.411+0000 7f128558e640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:55:01.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.411+0000 7f128558e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:55:01.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.411+0000 7f128558e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:55:01.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.411+0000 7f128558e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:55:01.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.411+0000 7f128558e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:55:01.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.411+0000 7f128558e640 10 monclient(hunting): handle_auth_done global_id 824538 payload 824 2024-09-16T03:55:01.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.411+0000 7f128558e640 1 -- 172.21.15.42:0/84364580 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f12800a22c0 con 0x7f128013ac90 2024-09-16T03:55:01.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.411+0000 7f128558e640 1 --2- 172.21.15.42:0/84364580 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f128013ac90 0x7f12801da020 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f127c00b430 tx=0x7f127c005a70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:55:01.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.411+0000 7f1275ffb640 10 client.854168.objecter ms_handle_connect 0x7f128013ac90 2024-09-16T03:55:01.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.411+0000 7f1275ffb640 10 client.854168.objecter resend_mon_ops 2024-09-16T03:55:01.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.411+0000 7f1275ffb640 1 -- 172.21.15.42:0/84364580 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f127c005690 con 0x7f128013ac90 2024-09-16T03:55:01.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.411+0000 7f1275ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:55:01.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.411+0000 7f1275ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:55:01.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.411+0000 7f1275ffb640 10 monclient: dump: 2024-09-16T03:55:01.435 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:55:01.435 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:55:01.435 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:01.435 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:01.435 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:55:01.435 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:55:01.435 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:55:01.435 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:55:01.435 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:55:01.435 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:55:01.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.411+0000 7f1275ffb640 1 -- 172.21.15.42:0/84364580 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4871 (secure 0 0 0) 0x7f127c005690 con 0x7f128013ac90 2024-09-16T03:55:01.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.411+0000 7f1275ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:55:01.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.411+0000 7f1275ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:55:01.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.411+0000 7f1275ffb640 1 -- 172.21.15.42:0/84364580 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f128013ac90 msgr2=0x7f12801da020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:55:01.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.411+0000 7f1275ffb640 1 --2- 172.21.15.42:0/84364580 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f128013ac90 0x7f12801da020 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f127c00b430 tx=0x7f127c005a70 comp rx=0 tx=0).stop 2024-09-16T03:55:01.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.411+0000 7f1275ffb640 1 -- 172.21.15.42:0/84364580 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f128013ac90 msgr2=0x7f12801da020 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:55:01.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.411+0000 7f1275ffb640 1 --2- 172.21.15.42:0/84364580 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f128013ac90 0x7f12801da020 secure :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f127c00b430 tx=0x7f127c005a70 comp rx=0 tx=0).stop 2024-09-16T03:55:01.436 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:55:01.436 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:55:01.436 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:55:01.436 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "leader", 2024-09-16T03:55:01.437 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 114, 2024-09-16T03:55:01.437 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 191266, 2024-09-16T03:55:01.437 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:55:01.437 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:55:01.437 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T03:55:01.437 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:55:01.437 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 1, 2024-09-16T03:55:01.437 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:55:01.437 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:55:01.437 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:55:01.437 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:55:01.437 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:55:01.437 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:55:01.437 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:55:01.438 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:55:01.438 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:55:01.438 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:55:01.438 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:55:01.438 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:55:01.438 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:55:01.438 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:55:01.438 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:55:01.438 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:55:01.438 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:55:01.438 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:55:01.438 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:55:01.438 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:55:01.438 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:55:01.439 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:55:01.439 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:55:01.439 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:55:01.439 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:55:01.439 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:55:01.439 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:55:01.439 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:55:01.439 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:55:01.439 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:55:01.439 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:55:01.439 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:55:01.439 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:55:01.439 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:55:01.439 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:55:01.440 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:55:01.440 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:55:01.440 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:55:01.440 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:55:01.440 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:55:01.440 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:55:01.440 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:55:01.440 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:55:01.440 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:55:01.440 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:55:01.440 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:55:01.440 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:55:01.440 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:55:01.441 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:55:01.441 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:55:01.441 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:55:01.441 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:55:01.441 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:55:01.441 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:55:01.441 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:55:01.441 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:55:01.441 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:55:01.441 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:55:01.441 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:55:01.441 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:55:01.441 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:55:01.441 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:55:01.442 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:55:01.442 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:55:01.442 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:55:01.442 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:55:01.442 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:55:01.442 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:55:01.442 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:55:01.442 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:55:01.442 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:55:01.442 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:55:01.442 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:55:01.442 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:55:01.442 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:55:01.442 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:55:01.443 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:55:01.443 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:55:01.443 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:55:01.443 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:55:01.443 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:55:01.443 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:55:01.443 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:55:01.443 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:55:01.443 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:55:01.443 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:55:01.443 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:55:01.443 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:55:01.443 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:55:01.443 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:55:01.444 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:55:01.444 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:55:01.444 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:55:01.444 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:55:01.444 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:55:01.444 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:55:01.444 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:55:01.444 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:55:01.444 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:55:01.444 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:55:01.444 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:55:01.444 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:55:01.444 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:55:01.444 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:55:01.445 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:55:01.445 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:55:01.445 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:55:01.445 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:55:01.445 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:55:01.445 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:55:01.445 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:55:01.445 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:55:01.445 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:55:01.445 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:55:01.445 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:55:01.445 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:55:01.445 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:55:01.445 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:55:01.446 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:55:01.446 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:55:01.446 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:55:01.446 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:55:01.446 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:55:01.446 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:55:01.446 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:55:01.446 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:55:01.446 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:55:01.446 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:55:01.446 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:55:01.446 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:55:01.446 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:55:01.446 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:55:01.447 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:55:01.447 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:55:01.447 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:55:01.447 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:55:01.447 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:55:01.447 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:55:01.447 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:55:01.447 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:55:01.447 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:55:01.447 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T03:55:01.447 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:55:01.447 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:55:01.447 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:55:01.447 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 6 2024-09-16T03:55:01.447 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:55:01.448 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:55:01.448 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:55:01.448 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:55:01.448 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:55:01.448 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:55:01.448 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 7 2024-09-16T03:55:01.448 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:55:01.448 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:55:01.448 INFO:teuthology.orchestra.run.smithi042.stdout: "mgr": [ 2024-09-16T03:55:01.448 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:55:01.448 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:55:01.448 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:55:01.448 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:55:01.448 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:55:01.449 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:55:01.449 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:55:01.449 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:55:01.449 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:55:01.450 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.411+0000 7f12877f2640 10 librados: watch_flush enter 2024-09-16T03:55:01.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.411+0000 7f12877f2640 10 librados: watch_flush exit 2024-09-16T03:55:01.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.411+0000 7f12877f2640 20 client.854168.objecter shutdown clearing up homeless session... 2024-09-16T03:55:01.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.411+0000 7f12877f2640 10 client.854168.objecter successfully canceled tick 2024-09-16T03:55:01.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.411+0000 7f12877f2640 1 -- 172.21.15.42:0/84364580 >> v2:172.21.15.154:6812/254845342 conn(0x7f12640377c0 msgr2=0x7f1264039c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:55:01.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.411+0000 7f12877f2640 1 --2- 172.21.15.42:0/84364580 >> v2:172.21.15.154:6812/254845342 conn(0x7f12640377c0 0x7f1264039c80 secure :-1 s=READY pgs=3627 cs=0 l=1 rev1=1 crypto rx=0x7f1280120660 tx=0x7f1270005040 comp rx=0 tx=0).stop 2024-09-16T03:55:01.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.411+0000 7f12877f2640 10 monclient: shutdown 2024-09-16T03:55:01.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.411+0000 7f12877f2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:55:01.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.411+0000 7f12877f2640 1 -- 172.21.15.42:0/84364580 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f12800a10a0 msgr2=0x7f12801d5ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:55:01.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.411+0000 7f12877f2640 1 --2- 172.21.15.42:0/84364580 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f12800a10a0 0x7f12801d5ac0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f12800a7a80 tx=0x7f1278007430 comp rx=0 tx=0).stop 2024-09-16T03:55:01.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.411+0000 7f1285d8f640 1 -- 172.21.15.42:0/84364580 reap_dead start 2024-09-16T03:55:01.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.411+0000 7f1285d8f640 1 -- 172.21.15.42:0/84364580 reap_dead start 2024-09-16T03:55:01.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.411+0000 7f12877f2640 1 -- 172.21.15.42:0/84364580 shutdown_connections 2024-09-16T03:55:01.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.411+0000 7f12877f2640 1 -- 172.21.15.42:0/84364580 >> 172.21.15.42:0/84364580 conn(0x7f12800a9fb0 msgr2=0x7f12800a90d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:55:01.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.411+0000 7f12877f2640 1 -- 172.21.15.42:0/84364580 shutdown_connections 2024-09-16T03:55:01.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.411+0000 7f12877f2640 1 -- 172.21.15.42:0/84364580 wait complete. 2024-09-16T03:55:01.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.411+0000 7f12877f2640 1 librados: shutdown 2024-09-16T03:55:01.452 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T03:55:01.452 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-16T03:55:01.452 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T03:55:01.572 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.567+0000 7f8bb3c64640 10 monclient: get_monmap_and_config 2024-09-16T03:55:01.573 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.567+0000 7f8bb3c64640 10 monclient: build_initial_monmap 2024-09-16T03:55:01.573 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.567+0000 7f8bb3c64640 10 monclient: monmap: 2024-09-16T03:55:01.573 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:55:01.573 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:55:01.573 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:55:01.570945+0000 2024-09-16T03:55:01.573 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:55:01.570945+0000 2024-09-16T03:55:01.573 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:55:01.573 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:55:01.573 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:55:01.573 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:55:01.573 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:55:01.573 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:55:01.574 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:55:01.574 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:55:01.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.567+0000 7f8bb3c64640 1 Processor -- start 2024-09-16T03:55:01.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.567+0000 7f8bb3c64640 1 -- start start 2024-09-16T03:55:01.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.567+0000 7f8bb3c64640 10 monclient: init 2024-09-16T03:55:01.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.571+0000 7f8bb3c64640 10 monclient: _reopen_session rank -1 2024-09-16T03:55:01.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.571+0000 7f8bb3c64640 10 monclient: _add_conns ranks=[3,1,0,2,4] 2024-09-16T03:55:01.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.571+0000 7f8bb3c64640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f8bac0a1330 0x7f8bac0a1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:55:01.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.571+0000 7f8bb3c64640 10 monclient(hunting): picked mon.noname-a con 0x7f8bac0a1330 addr v2:172.21.15.42:3300/0 2024-09-16T03:55:01.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.571+0000 7f8bb3c64640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8bac06dac0 addr v1:172.21.15.154:6789/0 2024-09-16T03:55:01.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.571+0000 7f8bb3c64640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8bac0a1c50 addr v1:172.21.15.42:6789/0 2024-09-16T03:55:01.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.571+0000 7f8bb3c64640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8bac135f20 con 0x7f8bac0a1c50 2024-09-16T03:55:01.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.571+0000 7f8bb3c64640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8bac09c090 con 0x7f8bac06dac0 2024-09-16T03:55:01.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.571+0000 7f8bb3c64640 10 monclient(hunting): start opening mon connection 2024-09-16T03:55:01.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.571+0000 7f8bb3c64640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f8bac09f610 con 0x7f8bac0a1330 2024-09-16T03:55:01.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.571+0000 7f8bb3c64640 10 monclient(hunting): _renew_subs 2024-09-16T03:55:01.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.571+0000 7f8bb3c64640 10 monclient(hunting): authenticate will time out at 1748.540388s 2024-09-16T03:55:01.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.571+0000 7f8bb1a00640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f8bac0a1330 0x7f8bac0a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:55:01.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.571+0000 7f8bb2201640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f8bac0a1c50 0x7f8bac135810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:36144/0 (socket says 172.21.15.42:36144) 2024-09-16T03:55:01.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.571+0000 7f8bb11ff640 1 -- >> v1:172.21.15.154:6789/0 conn(0x7f8bac06dac0 legacy=0x7f8bac135100 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.154:6789/0 2024-09-16T03:55:01.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.571+0000 7f8bb2201640 1 -- 172.21.15.42:0/1271604894 learned_addr learned my addr 172.21.15.42:0/1271604894 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:55:01.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.571+0000 7f8bb1a00640 10 monclient(hunting): get_auth_request con 0x7f8bac0a1330 auth_method 0 2024-09-16T03:55:01.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.571+0000 7f8bb1a00640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:55:01.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.571+0000 7f8bb1a00640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:55:01.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.571+0000 7f8bb1a00640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:55:01.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.571+0000 7f8bb1a00640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:55:01.578 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.571+0000 7f8bb1a00640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:55:01.578 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.571+0000 7f8bb1a00640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:55:01.578 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.571+0000 7f8bb1a00640 10 monclient(hunting): handle_auth_done global_id 824541 payload 274 2024-09-16T03:55:01.578 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.571+0000 7f8bb1a00640 1 -- 172.21.15.42:0/1271604894 >> v1:172.21.15.154:6789/0 conn(0x7f8bac06dac0 legacy=0x7f8bac135100 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:55:01.578 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.571+0000 7f8bb1a00640 1 -- 172.21.15.42:0/1271604894 >> v1:172.21.15.42:6789/0 conn(0x7f8bac0a1c50 legacy=0x7f8bac135810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:55:01.578 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.571+0000 7f8bb1a00640 10 monclient: _finish_hunting 0 2024-09-16T03:55:01.578 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.571+0000 7f8bb1a00640 1 monclient: found mon.noname-a 2024-09-16T03:55:01.578 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.571+0000 7f8bb09fe640 1 -- 172.21.15.42:0/1271604894 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 587722907 0 0) 0x7f8bac135f20 con 0x7f8bac0a1c50 2024-09-16T03:55:01.578 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.571+0000 7f8bb1a00640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:55:01.578 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.571+0000 7f8bb1a00640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:55:01.578 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.571+0000 7f8bb1a00640 1 -- 172.21.15.42:0/1271604894 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8bac09f8f0 con 0x7f8bac0a1330 2024-09-16T03:55:01.578 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.571+0000 7f8bb09fe640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T03:55:01.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.575+0000 7f8bb1a00640 1 --2- 172.21.15.42:0/1271604894 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8bac0a1330 0x7f8bac0a1710 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f8ba40097c0 tx=0x7f8ba401c380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61e0c3264ed687a0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:55:01.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.575+0000 7f8bb09fe640 1 -- 172.21.15.42:0/1271604894 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ba40b1070 con 0x7f8bac0a1330 2024-09-16T03:55:01.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.575+0000 7f8bb09fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:55:01.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.575+0000 7f8bb09fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:55:01.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.575+0000 7f8bb09fe640 10 monclient: dump: 2024-09-16T03:55:01.585 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:55:01.585 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:55:01.586 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:01.586 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:01.586 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:55:01.586 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:55:01.586 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:55:01.586 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:55:01.586 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:55:01.586 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:55:01.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.575+0000 7f8bb09fe640 10 monclient: _finish_auth 0 2024-09-16T03:55:01.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.575+0000 7f8bb09fe640 10 monclient: _check_auth_tickets 2024-09-16T03:55:01.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.575+0000 7f8bb09fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:55:01.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.575+0000 7f8bb09fe640 1 -- 172.21.15.42:0/1271604894 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ba40ac600 con 0x7f8bac0a1330 2024-09-16T03:55:01.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.575+0000 7f8bb3c64640 5 monclient: authenticate success, global_id 824541 2024-09-16T03:55:01.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.575+0000 7f8bb3c64640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:55:01.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.575+0000 7f8bb09fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:55:01.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.575+0000 7f8bb09fe640 1 -- 172.21.15.42:0/1271604894 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ba40ac9d0 con 0x7f8bac0a1330 2024-09-16T03:55:01.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.575+0000 7f8bb3c64640 10 monclient: get_monmap_and_config success 2024-09-16T03:55:01.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.575+0000 7f8bb3c64640 4 set_mon_vals no callback set 2024-09-16T03:55:01.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.575+0000 7f8bb09fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:55:01.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.575+0000 7f8bb09fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:55:01.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.575+0000 7f8bb09fe640 10 monclient: dump: 2024-09-16T03:55:01.587 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:55:01.587 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:55:01.587 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:01.587 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:01.587 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:55:01.587 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:55:01.587 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:55:01.588 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:55:01.588 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:55:01.588 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:55:01.588 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.575+0000 7f8bb3c64640 10 monclient: shutdown 2024-09-16T03:55:01.588 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.575+0000 7f8bb3c64640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:55:01.588 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.575+0000 7f8bb3c64640 1 -- 172.21.15.42:0/1271604894 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8bac0a1330 msgr2=0x7f8bac0a1710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:55:01.588 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.575+0000 7f8bb3c64640 1 --2- 172.21.15.42:0/1271604894 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8bac0a1330 0x7f8bac0a1710 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f8ba40097c0 tx=0x7f8ba401c380 comp rx=0 tx=0).stop 2024-09-16T03:55:01.588 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.575+0000 7f8bb3c64640 1 -- 172.21.15.42:0/1271604894 shutdown_connections 2024-09-16T03:55:01.588 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.575+0000 7f8bb3c64640 1 --2- 172.21.15.42:0/1271604894 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8bac0a1330 0x7f8bac0a1710 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:55:01.588 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.575+0000 7f8bb3c64640 1 -- 172.21.15.42:0/1271604894 >> 172.21.15.42:0/1271604894 conn(0x7f8bac0a2420 msgr2=0x7f8bac0a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:55:01.588 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.575+0000 7f8bb3c64640 1 -- 172.21.15.42:0/1271604894 shutdown_connections 2024-09-16T03:55:01.588 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.575+0000 7f8bb3c64640 1 -- 172.21.15.42:0/1271604894 wait complete. 2024-09-16T03:55:01.588 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.575+0000 7f8bb3c64640 10 monclient: build_initial_monmap 2024-09-16T03:55:01.589 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.575+0000 7f8bb3c64640 10 monclient: monmap: 2024-09-16T03:55:01.589 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:55:01.589 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:55:01.589 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:55:01.589 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:55:01.589 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:55:01.589 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:55:01.589 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:55:01.589 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:55:01.589 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:55:01.589 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:55:01.589 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.575+0000 7f8bb3c64640 1 librados: starting msgr at 2024-09-16T03:55:01.589 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.575+0000 7f8bb3c64640 1 librados: starting objecter 2024-09-16T03:55:01.589 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.575+0000 7f8bb3c64640 1 Processor -- start 2024-09-16T03:55:01.590 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.575+0000 7f8bb3c64640 1 -- start start 2024-09-16T03:55:01.590 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.575+0000 7f8bb3c64640 1 librados: setting wanted keys 2024-09-16T03:55:01.590 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.575+0000 7f8bb3c64640 1 librados: calling monclient init 2024-09-16T03:55:01.590 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.575+0000 7f8bb3c64640 10 monclient: init 2024-09-16T03:55:01.590 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.575+0000 7f8bb3c64640 10 monclient: _reopen_session rank -1 2024-09-16T03:55:01.590 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.575+0000 7f8bb3c64640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:55:01.590 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.575+0000 7f8bb3c64640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8bac0a1330 0x7f8bac13ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:55:01.590 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.575+0000 7f8bb3c64640 10 monclient(hunting): picked mon.noname-a con 0x7f8bac0a1330 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:55:01.590 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.575+0000 7f8bb3c64640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f8bac0a1c50 0x7f8bac135120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:55:01.590 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.575+0000 7f8bb3c64640 10 monclient(hunting): picked mon.noname-c con 0x7f8bac0a1c50 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:55:01.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.575+0000 7f8bb3c64640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8bac135660 0x7f8bac09cb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:55:01.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.575+0000 7f8bb3c64640 10 monclient(hunting): picked mon.noname-b con 0x7f8bac135660 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:55:01.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.575+0000 7f8bb3c64640 10 monclient(hunting): start opening mon connection 2024-09-16T03:55:01.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.575+0000 7f8bb3c64640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f8bac09eef0 con 0x7f8bac0a1330 2024-09-16T03:55:01.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.575+0000 7f8bb3c64640 10 monclient(hunting): start opening mon connection 2024-09-16T03:55:01.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.575+0000 7f8bb3c64640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f8bac09ed70 con 0x7f8bac135660 2024-09-16T03:55:01.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.575+0000 7f8bb3c64640 10 monclient(hunting): start opening mon connection 2024-09-16T03:55:01.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.575+0000 7f8bb3c64640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f8bac140b90 con 0x7f8bac0a1c50 2024-09-16T03:55:01.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.575+0000 7f8bb3c64640 10 monclient(hunting): _renew_subs 2024-09-16T03:55:01.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.575+0000 7f8bb3c64640 10 monclient(hunting): authenticate will time out at 1748.547709s 2024-09-16T03:55:01.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.575+0000 7f8bb11ff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f8bac0a1c50 0x7f8bac135120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:55:01.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.575+0000 7f8bb1a00640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8bac0a1330 0x7f8bac13ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:55:01.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.575+0000 7f8bb11ff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f8bac0a1c50 0x7f8bac135120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:45738/0 (socket says 172.21.15.42:45738) 2024-09-16T03:55:01.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.575+0000 7f8bb11ff640 1 -- 172.21.15.42:0/122785749 learned_addr learned my addr 172.21.15.42:0/122785749 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:55:01.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.575+0000 7f8bb11ff640 10 monclient(hunting): get_auth_request con 0x7f8bac0a1c50 auth_method 0 2024-09-16T03:55:01.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.575+0000 7f8bb1a00640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8bac0a1330 0x7f8bac13ccf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:45356/0 (socket says 172.21.15.42:45356) 2024-09-16T03:55:01.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.575+0000 7f8bb11ff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:55:01.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.575+0000 7f8bb11ff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:55:01.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.575+0000 7f8bb11ff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:55:01.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.575+0000 7f8bb2201640 1 -- 172.21.15.42:0/122785749 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8bac135660 msgr2=0x7f8bac09cb60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T03:55:01.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.575+0000 7f8bb1a00640 10 monclient(hunting): get_auth_request con 0x7f8bac0a1330 auth_method 0 2024-09-16T03:55:01.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.575+0000 7f8bb1a00640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:55:01.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.575+0000 7f8bb1a00640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:55:01.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.575+0000 7f8bb1a00640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:55:01.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.575+0000 7f8bb2201640 1 --2- 172.21.15.42:0/122785749 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8bac135660 0x7f8bac09cb60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:55:01.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.575+0000 7f8bb11ff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:55:01.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.579+0000 7f8bb11ff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:55:01.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.579+0000 7f8bb11ff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:55:01.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.579+0000 7f8bb1a00640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:55:01.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.579+0000 7f8bb1a00640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:55:01.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.579+0000 7f8bb1a00640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:55:01.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.579+0000 7f8bb11ff640 10 monclient(hunting): handle_auth_done global_id 854171 payload 824 2024-09-16T03:55:01.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.579+0000 7f8bb11ff640 1 -- 172.21.15.42:0/122785749 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8bac135660 msgr2=0x7f8bac09cb60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:55:01.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.579+0000 7f8bb11ff640 1 --2- 172.21.15.42:0/122785749 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8bac135660 0x7f8bac09cb60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:55:01.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.579+0000 7f8bb11ff640 1 -- 172.21.15.42:0/122785749 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8bac0a1330 msgr2=0x7f8bac13ccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:55:01.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.579+0000 7f8bb11ff640 1 --2- 172.21.15.42:0/122785749 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8bac0a1330 0x7f8bac13ccf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:55:01.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.579+0000 7f8bb11ff640 10 monclient: _finish_hunting 0 2024-09-16T03:55:01.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.579+0000 7f8bb11ff640 1 monclient: found mon.noname-c 2024-09-16T03:55:01.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.579+0000 7f8bb11ff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:55:01.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.579+0000 7f8bb11ff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:55:01.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.579+0000 7f8bb11ff640 1 -- 172.21.15.42:0/122785749 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8bac13de20 con 0x7f8bac0a1c50 2024-09-16T03:55:01.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.579+0000 7f8bb1a00640 1 --2- 172.21.15.42:0/122785749 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8bac0a1330 0x7f8bac13ccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:55:01.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.579+0000 7f8bb11ff640 1 --2- 172.21.15.42:0/122785749 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f8bac0a1c50 0x7f8bac135120 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f8ba000b9e0 tx=0x7f8ba000bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:55:01.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.579+0000 7f8baa7fc640 10 client.?.objecter ms_handle_connect 0x7f8bac0a1c50 2024-09-16T03:55:01.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.579+0000 7f8baa7fc640 10 client.?.objecter resend_mon_ops 2024-09-16T03:55:01.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.579+0000 7f8baa7fc640 1 -- 172.21.15.42:0/122785749 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ba000cba0 con 0x7f8bac0a1c50 2024-09-16T03:55:01.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.579+0000 7f8baa7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:55:01.596 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.579+0000 7f8baa7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:55:01.596 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.579+0000 7f8baa7fc640 10 monclient: dump: 2024-09-16T03:55:01.596 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:55:01.596 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:55:01.596 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:01.596 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:01.596 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:55:01.596 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:55:01.596 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:55:01.596 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:55:01.596 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:55:01.596 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:55:01.596 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.579+0000 7f8baa7fc640 10 monclient: _finish_auth 0 2024-09-16T03:55:01.597 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.579+0000 7f8baa7fc640 10 monclient: _check_auth_tickets 2024-09-16T03:55:01.597 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.579+0000 7f8baa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:55:01.597 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.579+0000 7f8baa7fc640 1 -- 172.21.15.42:0/122785749 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ba0010070 con 0x7f8bac0a1c50 2024-09-16T03:55:01.597 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.579+0000 7f8baa7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:55:01.597 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.579+0000 7f8baa7fc640 1 -- 172.21.15.42:0/122785749 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ba0015470 con 0x7f8bac0a1c50 2024-09-16T03:55:01.597 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.579+0000 7f8baa7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:55:01.597 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.579+0000 7f8baa7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:55:01.597 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.579+0000 7f8baa7fc640 10 monclient: dump: 2024-09-16T03:55:01.597 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:55:01.597 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:55:01.597 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:01.597 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:01.597 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:55:01.597 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:55:01.598 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:55:01.598 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:55:01.598 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:55:01.598 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:55:01.598 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.579+0000 7f8bb3c64640 5 monclient: authenticate success, global_id 854171 2024-09-16T03:55:01.598 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.579+0000 7f8bb3c64640 10 monclient: _renew_subs 2024-09-16T03:55:01.598 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.579+0000 7f8bb3c64640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:55:01.598 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.579+0000 7f8bb3c64640 1 -- 172.21.15.42:0/122785749 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8bac071210 con 0x7f8bac0a1c50 2024-09-16T03:55:01.598 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.579+0000 7f8babfff640 4 set_mon_vals no callback set 2024-09-16T03:55:01.598 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.579+0000 7f8bb3c64640 10 client.854171.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:55:01.598 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.579+0000 7f8bb3c64640 10 monclient: _renew_subs 2024-09-16T03:55:01.598 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.579+0000 7f8bb3c64640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:55:01.598 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.579+0000 7f8bb3c64640 1 -- 172.21.15.42:0/122785749 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f8bac120800 con 0x7f8bac0a1c50 2024-09-16T03:55:01.598 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.579+0000 7f8bb3c64640 1 librados: init done 2024-09-16T03:55:01.599 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.579+0000 7f8bb3c64640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:55:01.599 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.579+0000 7f8bb3c64640 1 --2- 172.21.15.42:0/122785749 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f8bac06da40 0x7f8bac02c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:55:01.599 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.579+0000 7f8bb3c64640 10 monclient(hunting): start opening mon connection 2024-09-16T03:55:01.599 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.579+0000 7f8bb3c64640 1 -- 172.21.15.42:0/122785749 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f8bac0a1710 con 0x7f8bac06da40 2024-09-16T03:55:01.599 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.579+0000 7f8bb1a00640 1 --2- 172.21.15.42:0/122785749 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f8bac06da40 0x7f8bac02c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:55:01.599 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.579+0000 7f8bb1a00640 10 monclient: get_auth_request con 0x7f8bac06da40 auth_method 0 2024-09-16T03:55:01.599 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.579+0000 7f8bb1a00640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:55:01.599 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.579+0000 7f8bb1a00640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:55:01.599 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.579+0000 7f8bb1a00640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:55:01.599 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.579+0000 7f8baa7fc640 1 -- 172.21.15.42:0/122785749 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f8ba001e3a0 con 0x7f8bac0a1c50 2024-09-16T03:55:01.599 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.579+0000 7f8bb1a00640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:55:01.599 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.579+0000 7f8bb1a00640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:55:01.599 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.579+0000 7f8bb1a00640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:55:01.599 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.579+0000 7f8baa7fc640 1 --2- 172.21.15.42:0/122785749 >> v2:172.21.15.154:6812/254845342 conn(0x7f8b980377f0 0x7f8b98039cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:55:01.600 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.579+0000 7f8baa7fc640 1 -- 172.21.15.42:0/122785749 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(387..387 src has 286..387) ==== 5328+0+0 (secure 0 0 0) 0x7f8ba0048e90 con 0x7f8bac0a1c50 2024-09-16T03:55:01.600 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.579+0000 7f8baa7fc640 10 client.854171.objecter ms_dispatch 0x7f8bac000ba0 osd_map(387..387 src has 286..387) 2024-09-16T03:55:01.600 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.579+0000 7f8baa7fc640 3 client.854171.objecter handle_osd_map got epochs [387,387] > 0 2024-09-16T03:55:01.600 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.579+0000 7f8baa7fc640 3 client.854171.objecter handle_osd_map decoding full epoch 387 2024-09-16T03:55:01.600 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.579+0000 7f8bb1a00640 10 monclient(hunting): handle_auth_done global_id 854174 payload 824 2024-09-16T03:55:01.600 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.579+0000 7f8bb1a00640 1 -- 172.21.15.42:0/122785749 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8bac139a40 con 0x7f8bac06da40 2024-09-16T03:55:01.600 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.579+0000 7f8baa7fc640 20 client.854171.objecter dump_active .. 0 homeless 2024-09-16T03:55:01.600 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.579+0000 7f8baa7fc640 10 client.854171.objecter ms_handle_connect 0x7f8bac06da40 2024-09-16T03:55:01.600 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.579+0000 7f8baa7fc640 10 client.854171.objecter resend_mon_ops 2024-09-16T03:55:01.600 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.579+0000 7f8bb1a00640 1 --2- 172.21.15.42:0/122785749 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f8bac06da40 0x7f8bac02c650 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f8bac09c800 tx=0x7f8ba4019120 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:55:01.600 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.579+0000 7f8bb2201640 1 --2- 172.21.15.42:0/122785749 >> v2:172.21.15.154:6812/254845342 conn(0x7f8b980377f0 0x7f8b98039cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:55:01.600 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.579+0000 7f8baa7fc640 1 -- 172.21.15.42:0/122785749 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ba40b1040 con 0x7f8bac06da40 2024-09-16T03:55:01.600 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.579+0000 7f8baa7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:55:01.601 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.579+0000 7f8baa7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:55:01.601 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.579+0000 7f8baa7fc640 10 monclient: dump: 2024-09-16T03:55:01.601 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:55:01.601 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:55:01.601 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:01.601 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:01.601 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:55:01.601 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:55:01.601 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:55:01.601 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:55:01.601 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:55:01.601 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:55:01.601 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.579+0000 7f8bb2201640 10 monclient: get_auth_request con 0x7f8b980377f0 auth_method 0 2024-09-16T03:55:01.601 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.579+0000 7f8baa7fc640 1 -- 172.21.15.42:0/122785749 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f8ba4004220 con 0x7f8bac06da40 2024-09-16T03:55:01.602 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.579+0000 7f8baa7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:55:01.602 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.579+0000 7f8baa7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:55:01.602 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.579+0000 7f8baa7fc640 1 -- 172.21.15.42:0/122785749 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f8bac06da40 msgr2=0x7f8bac02c650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:55:01.602 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.579+0000 7f8baa7fc640 1 --2- 172.21.15.42:0/122785749 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f8bac06da40 0x7f8bac02c650 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f8bac09c800 tx=0x7f8ba4019120 comp rx=0 tx=0).stop 2024-09-16T03:55:01.602 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.579+0000 7f8baa7fc640 1 -- 172.21.15.42:0/122785749 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f8bac06da40 msgr2=0x7f8bac02c650 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:55:01.602 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.579+0000 7f8baa7fc640 1 --2- 172.21.15.42:0/122785749 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f8bac06da40 0x7f8bac02c650 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:55:01.602 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.579+0000 7f8baa7fc640 10 client.854171.objecter ms_handle_connect 0x7f8b980377f0 2024-09-16T03:55:01.602 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.579+0000 7f8bb2201640 1 --2- 172.21.15.42:0/122785749 >> v2:172.21.15.154:6812/254845342 conn(0x7f8b980377f0 0x7f8b98039cb0 secure :-1 s=READY pgs=3628 cs=0 l=1 rev1=1 crypto rx=0x7f8b9c004820 tx=0x7f8b9c007a50 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:55:01.604 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.587+0000 7f8bb3c64640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:55:01.604 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.587+0000 7f8bb3c64640 1 --2- 172.21.15.42:0/122785749 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f8bac02cb90 0x7f8bac02cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:55:01.604 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.587+0000 7f8bb3c64640 10 monclient(hunting): start opening mon connection 2024-09-16T03:55:01.604 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.587+0000 7f8bb3c64640 1 -- 172.21.15.42:0/122785749 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f8bac1cfa30 con 0x7f8bac02cb90 2024-09-16T03:55:01.604 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.587+0000 7f8bb1a00640 1 --2- 172.21.15.42:0/122785749 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f8bac02cb90 0x7f8bac02cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:55:01.604 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.587+0000 7f8bb1a00640 10 monclient: get_auth_request con 0x7f8bac02cb90 auth_method 0 2024-09-16T03:55:01.604 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.587+0000 7f8bb1a00640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:55:01.604 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.587+0000 7f8bb1a00640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:55:01.605 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.587+0000 7f8bb1a00640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:55:01.605 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.587+0000 7f8bb1a00640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:55:01.605 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.587+0000 7f8bb1a00640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:55:01.605 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.587+0000 7f8bb1a00640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:55:01.605 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.587+0000 7f8bb1a00640 10 monclient(hunting): handle_auth_done global_id 854177 payload 824 2024-09-16T03:55:01.605 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.587+0000 7f8bb1a00640 1 -- 172.21.15.42:0/122785749 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f8bac09dd00 con 0x7f8bac02cb90 2024-09-16T03:55:01.605 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.587+0000 7f8bb1a00640 1 --2- 172.21.15.42:0/122785749 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f8bac02cb90 0x7f8bac02cfa0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f8ba4000c10 tx=0x7f8ba4009190 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:55:01.605 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.587+0000 7f8baa7fc640 10 client.854171.objecter ms_handle_connect 0x7f8bac02cb90 2024-09-16T03:55:01.605 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.587+0000 7f8baa7fc640 10 client.854171.objecter resend_mon_ops 2024-09-16T03:55:01.605 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.587+0000 7f8baa7fc640 1 -- 172.21.15.42:0/122785749 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ba40b1040 con 0x7f8bac02cb90 2024-09-16T03:55:01.605 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.587+0000 7f8baa7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:55:01.605 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.587+0000 7f8baa7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:55:01.605 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.587+0000 7f8baa7fc640 10 monclient: dump: 2024-09-16T03:55:01.606 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:55:01.606 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:55:01.606 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:01.606 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:01.606 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:55:01.606 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:55:01.606 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:55:01.606 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:55:01.606 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:55:01.606 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:55:01.606 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.587+0000 7f8baa7fc640 1 -- 172.21.15.42:0/122785749 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4534 (secure 0 0 0) 0x7f8ba40b7b90 con 0x7f8bac02cb90 2024-09-16T03:55:01.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.587+0000 7f8baa7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:55:01.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.587+0000 7f8baa7fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:55:01.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.587+0000 7f8baa7fc640 1 -- 172.21.15.42:0/122785749 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f8bac02cb90 msgr2=0x7f8bac02cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:55:01.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.587+0000 7f8baa7fc640 1 --2- 172.21.15.42:0/122785749 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f8bac02cb90 0x7f8bac02cfa0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f8ba4000c10 tx=0x7f8ba4009190 comp rx=0 tx=0).stop 2024-09-16T03:55:01.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.587+0000 7f8baa7fc640 1 -- 172.21.15.42:0/122785749 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f8bac02cb90 msgr2=0x7f8bac02cfa0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:55:01.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.587+0000 7f8baa7fc640 1 --2- 172.21.15.42:0/122785749 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f8bac02cb90 0x7f8bac02cfa0 secure :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f8ba4000c10 tx=0x7f8ba4009190 comp rx=0 tx=0).stop 2024-09-16T03:55:01.607 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:55:01.607 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:55:01.607 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:55:01.607 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T03:55:01.608 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 114, 2024-09-16T03:55:01.608 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 35023, 2024-09-16T03:55:01.608 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:55:01.608 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:55:01.608 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T03:55:01.608 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:55:01.608 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 1, 2024-09-16T03:55:01.608 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:55:01.608 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:55:01.608 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:55:01.608 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:55:01.608 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:55:01.609 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:55:01.609 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:55:01.609 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:55:01.609 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:55:01.609 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:55:01.609 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:55:01.609 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:55:01.609 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:55:01.609 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:55:01.609 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:55:01.609 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:55:01.609 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:55:01.610 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:55:01.610 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:55:01.610 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:55:01.610 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:55:01.610 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:55:01.610 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:55:01.610 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:55:01.610 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:55:01.610 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:55:01.610 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:55:01.610 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:55:01.610 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:55:01.610 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:55:01.610 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:55:01.611 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:55:01.611 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:55:01.611 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:55:01.611 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:55:01.611 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:55:01.611 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:55:01.611 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:55:01.611 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:55:01.611 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:55:01.611 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:55:01.611 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:55:01.611 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:55:01.611 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:55:01.611 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:55:01.612 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:55:01.612 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:55:01.612 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:55:01.612 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:55:01.612 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:55:01.612 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:55:01.612 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:55:01.612 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:55:01.612 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:55:01.612 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:55:01.612 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:55:01.612 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:55:01.612 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:55:01.612 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:55:01.613 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:55:01.613 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:55:01.613 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:55:01.613 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:55:01.613 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:55:01.613 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:55:01.613 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:55:01.613 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:55:01.613 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:55:01.613 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:55:01.614 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:55:01.614 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:55:01.614 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:55:01.614 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:55:01.614 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:55:01.614 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:55:01.614 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:55:01.614 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:55:01.614 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:55:01.614 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:55:01.614 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:55:01.614 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:55:01.614 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:55:01.614 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:55:01.615 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:55:01.615 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:55:01.615 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:55:01.615 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:55:01.615 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:55:01.615 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:55:01.615 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:55:01.615 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:55:01.615 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:55:01.615 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:55:01.615 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:55:01.615 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:55:01.615 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:55:01.615 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:55:01.615 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:55:01.616 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:55:01.616 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:55:01.616 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:55:01.616 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:55:01.616 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:55:01.616 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:55:01.616 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:55:01.616 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:55:01.616 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:55:01.616 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:55:01.616 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:55:01.616 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:55:01.616 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:55:01.617 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:55:01.617 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:55:01.617 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:55:01.617 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:55:01.617 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:55:01.617 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:55:01.617 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:55:01.617 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:55:01.617 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:55:01.617 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:55:01.617 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:55:01.617 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:55:01.617 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:55:01.617 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:55:01.618 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:55:01.618 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:55:01.618 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:55:01.618 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:55:01.618 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:55:01.618 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:55:01.618 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:55:01.618 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:55:01.618 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:55:01.618 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:55:01.618 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:55:01.618 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:55:01.618 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:55:01.618 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:55:01.619 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:55:01.619 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:55:01.619 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:55:01.619 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:55:01.619 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:55:01.620 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:55:01.620 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 2 2024-09-16T03:55:01.621 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:55:01.621 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:55:01.621 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:55:01.621 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:55:01.621 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:55:01.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.587+0000 7f8bb3c64640 10 librados: watch_flush enter 2024-09-16T03:55:01.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.587+0000 7f8bb3c64640 10 librados: watch_flush exit 2024-09-16T03:55:01.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.587+0000 7f8bb3c64640 20 client.854171.objecter shutdown clearing up homeless session... 2024-09-16T03:55:01.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.587+0000 7f8bb3c64640 10 client.854171.objecter successfully canceled tick 2024-09-16T03:55:01.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.587+0000 7f8bb3c64640 1 -- 172.21.15.42:0/122785749 >> v2:172.21.15.154:6812/254845342 conn(0x7f8b980377f0 msgr2=0x7f8b98039cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:55:01.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.587+0000 7f8bb3c64640 1 --2- 172.21.15.42:0/122785749 >> v2:172.21.15.154:6812/254845342 conn(0x7f8b980377f0 0x7f8b98039cb0 secure :-1 s=READY pgs=3628 cs=0 l=1 rev1=1 crypto rx=0x7f8b9c004820 tx=0x7f8b9c007a50 comp rx=0 tx=0).stop 2024-09-16T03:55:01.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.587+0000 7f8bb3c64640 10 monclient: shutdown 2024-09-16T03:55:01.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.587+0000 7f8bb3c64640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:55:01.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.587+0000 7f8bb3c64640 1 -- 172.21.15.42:0/122785749 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f8bac0a1c50 msgr2=0x7f8bac135120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:55:01.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.587+0000 7f8bb3c64640 1 --2- 172.21.15.42:0/122785749 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f8bac0a1c50 0x7f8bac135120 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f8ba000b9e0 tx=0x7f8ba000bea0 comp rx=0 tx=0).stop 2024-09-16T03:55:01.625 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.587+0000 7f8bb2201640 1 -- 172.21.15.42:0/122785749 reap_dead start 2024-09-16T03:55:01.625 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.587+0000 7f8bb2201640 1 -- 172.21.15.42:0/122785749 reap_dead start 2024-09-16T03:55:01.625 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.587+0000 7f8bb3c64640 1 -- 172.21.15.42:0/122785749 shutdown_connections 2024-09-16T03:55:01.625 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.587+0000 7f8bb3c64640 1 -- 172.21.15.42:0/122785749 >> 172.21.15.42:0/122785749 conn(0x7f8bac0a2420 msgr2=0x7f8bac071740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:55:01.625 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.587+0000 7f8bb3c64640 1 -- 172.21.15.42:0/122785749 shutdown_connections 2024-09-16T03:55:01.625 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.587+0000 7f8bb3c64640 1 -- 172.21.15.42:0/122785749 wait complete. 2024-09-16T03:55:01.625 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:01.587+0000 7f8bb3c64640 1 librados: shutdown 2024-09-16T03:55:01.625 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T03:55:01.626 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-16T03:55:21.627 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-16T03:55:21.627 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-09-16T03:55:21.627 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-16T03:55:21.628 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-16T03:55:21.631 INFO:tasks.ceph.mon.b:Started 2024-09-16T03:55:21.631 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T03:55:21.631 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:55:21.767 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.763+0000 7eff320f6640 10 monclient: get_monmap_and_config 2024-09-16T03:55:21.767 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.763+0000 7eff320f6640 10 monclient: build_initial_monmap 2024-09-16T03:55:21.767 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.763+0000 7eff320f6640 10 monclient: monmap: 2024-09-16T03:55:21.767 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:55:21.767 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:55:21.767 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:55:21.765638+0000 2024-09-16T03:55:21.767 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:55:21.765638+0000 2024-09-16T03:55:21.767 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:55:21.767 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:55:21.767 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:55:21.767 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:55:21.768 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:55:21.768 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:55:21.768 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:55:21.768 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:55:21.770 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.763+0000 7eff320f6640 1 Processor -- start 2024-09-16T03:55:21.770 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.763+0000 7eff320f6640 1 -- start start 2024-09-16T03:55:21.770 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.763+0000 7eff320f6640 10 monclient: init 2024-09-16T03:55:21.770 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.763+0000 7eff320f6640 10 monclient: _reopen_session rank -1 2024-09-16T03:55:21.770 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.763+0000 7eff320f6640 10 monclient: _add_conns ranks=[3,1,2,4,0] 2024-09-16T03:55:21.770 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.763+0000 7eff320f6640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7eff2c0a1330 0x7eff2c09f180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:55:21.770 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.763+0000 7eff320f6640 10 monclient(hunting): picked mon.noname-a con 0x7eff2c0a1330 addr v2:172.21.15.42:3300/0 2024-09-16T03:55:21.770 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.763+0000 7eff320f6640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7eff2c0a1710 addr v1:172.21.15.154:6789/0 2024-09-16T03:55:21.770 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.763+0000 7eff320f6640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7eff2c1393a0 0x7eff2c13b790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:55:21.770 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.763+0000 7eff320f6640 10 monclient(hunting): picked mon.noname-b con 0x7eff2c1393a0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:55:21.770 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.763+0000 7eff320f6640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7eff2c09e4c0 con 0x7eff2c0a1710 2024-09-16T03:55:21.771 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.763+0000 7eff320f6640 10 monclient(hunting): start opening mon connection 2024-09-16T03:55:21.771 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.763+0000 7eff320f6640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7eff2c09edf0 con 0x7eff2c0a1330 2024-09-16T03:55:21.771 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.763+0000 7eff320f6640 10 monclient(hunting): start opening mon connection 2024-09-16T03:55:21.771 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.763+0000 7eff320f6640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7eff2c09cd60 con 0x7eff2c1393a0 2024-09-16T03:55:21.771 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.763+0000 7eff320f6640 10 monclient(hunting): _renew_subs 2024-09-16T03:55:21.771 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.763+0000 7eff320f6640 10 monclient(hunting): authenticate will time out at 1768.735095s 2024-09-16T03:55:21.771 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.763+0000 7eff2bfff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7eff2c1393a0 0x7eff2c13b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:55:21.771 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.763+0000 7eff2b7fe640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7eff2c0a1330 0x7eff2c09f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:55:21.771 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.763+0000 7eff2bfff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7eff2c1393a0 0x7eff2c13b790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:38072/0 (socket says 172.21.15.42:38072) 2024-09-16T03:55:21.771 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.763+0000 7eff2bfff640 1 -- 172.21.15.42:0/2331303104 learned_addr learned my addr 172.21.15.42:0/2331303104 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:55:21.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.763+0000 7eff2bfff640 10 monclient(hunting): get_auth_request con 0x7eff2c1393a0 auth_method 0 2024-09-16T03:55:21.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.763+0000 7eff2bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:55:21.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.763+0000 7eff2bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:55:21.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.763+0000 7eff2bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:55:21.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.763+0000 7eff2b7fe640 10 monclient(hunting): get_auth_request con 0x7eff2c0a1330 auth_method 0 2024-09-16T03:55:21.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.763+0000 7eff2b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:55:21.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.763+0000 7eff2b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:55:21.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.763+0000 7eff2b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:55:21.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.763+0000 7eff2bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:55:21.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.763+0000 7eff2bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:55:21.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.763+0000 7eff2bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:55:21.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.763+0000 7eff2b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:55:21.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.763+0000 7eff2b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:55:21.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.763+0000 7eff2b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:55:21.773 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.763+0000 7eff2bfff640 10 monclient(hunting): handle_auth_done global_id 854180 payload 274 2024-09-16T03:55:21.773 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.763+0000 7eff2bfff640 1 -- 172.21.15.42:0/2331303104 >> v2:172.21.15.42:3300/0 conn(0x7eff2c0a1330 msgr2=0x7eff2c09f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:55:21.773 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.763+0000 7eff2bfff640 1 --2- 172.21.15.42:0/2331303104 >> v2:172.21.15.42:3300/0 conn(0x7eff2c0a1330 0x7eff2c09f180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:55:21.773 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.763+0000 7eff2bfff640 1 -- 172.21.15.42:0/2331303104 >> v1:172.21.15.154:6789/0 conn(0x7eff2c0a1710 legacy=0x7eff2c0a1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:55:21.773 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.763+0000 7eff2bfff640 10 monclient: _finish_hunting 0 2024-09-16T03:55:21.773 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.763+0000 7eff2bfff640 1 monclient: found mon.noname-b 2024-09-16T03:55:21.773 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.763+0000 7eff2bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:55:21.773 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.763+0000 7eff2bfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T03:55:21.773 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.763+0000 7eff2bfff640 1 -- 172.21.15.42:0/2331303104 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff2c09f6c0 con 0x7eff2c1393a0 2024-09-16T03:55:21.779 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.767+0000 7eff2bfff640 1 --2- 172.21.15.42:0/2331303104 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7eff2c1393a0 0x7eff2c13b790 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7eff24009870 tx=0x7eff2401c420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=befe343e4a39e6ab server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:55:21.779 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.767+0000 7eff2a7fc640 1 -- 172.21.15.42:0/2331303104 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff240b0070 con 0x7eff2c1393a0 2024-09-16T03:55:21.779 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.767+0000 7eff2a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:55:21.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.767+0000 7eff2a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:55:21.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.767+0000 7eff2a7fc640 10 monclient: dump: 2024-09-16T03:55:21.780 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:55:21.780 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:55:21.780 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:21.780 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:21.780 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:55:21.780 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:55:21.780 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:55:21.780 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:55:21.780 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:55:21.780 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:55:21.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.767+0000 7eff2a7fc640 10 monclient: _finish_auth 0 2024-09-16T03:55:21.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.767+0000 7eff2a7fc640 10 monclient: _check_auth_tickets 2024-09-16T03:55:21.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.767+0000 7eff2a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:55:21.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.767+0000 7eff2a7fc640 1 -- 172.21.15.42:0/2331303104 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff240ab600 con 0x7eff2c1393a0 2024-09-16T03:55:21.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.767+0000 7eff320f6640 5 monclient: authenticate success, global_id 854180 2024-09-16T03:55:21.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.767+0000 7eff320f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:55:21.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.767+0000 7eff2a7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:55:21.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.767+0000 7eff320f6640 10 monclient: get_monmap_and_config success 2024-09-16T03:55:21.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.767+0000 7eff320f6640 4 set_mon_vals no callback set 2024-09-16T03:55:21.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.767+0000 7eff2a7fc640 1 -- 172.21.15.42:0/2331303104 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff240ab9d0 con 0x7eff2c1393a0 2024-09-16T03:55:21.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.767+0000 7eff2a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:55:21.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.767+0000 7eff2a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:55:21.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.767+0000 7eff2a7fc640 10 monclient: dump: 2024-09-16T03:55:21.781 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:55:21.781 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:55:21.782 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:21.782 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:21.782 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:55:21.782 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:55:21.782 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:55:21.782 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:55:21.782 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:55:21.782 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:55:21.782 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.767+0000 7eff320f6640 10 monclient: shutdown 2024-09-16T03:55:21.782 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.767+0000 7eff320f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:55:21.783 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.767+0000 7eff320f6640 1 -- 172.21.15.42:0/2331303104 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7eff2c1393a0 msgr2=0x7eff2c13b790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:55:21.783 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.767+0000 7eff320f6640 1 --2- 172.21.15.42:0/2331303104 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7eff2c1393a0 0x7eff2c13b790 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7eff24009870 tx=0x7eff2401c420 comp rx=0 tx=0).stop 2024-09-16T03:55:21.783 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.767+0000 7eff320f6640 1 -- 172.21.15.42:0/2331303104 shutdown_connections 2024-09-16T03:55:21.783 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.767+0000 7eff320f6640 1 --2- 172.21.15.42:0/2331303104 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7eff2c1393a0 0x7eff2c13b790 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:55:21.783 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.767+0000 7eff320f6640 1 --2- 172.21.15.42:0/2331303104 >> v2:172.21.15.42:3300/0 conn(0x7eff2c0a1330 0x7eff2c09f180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:55:21.783 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.767+0000 7eff320f6640 1 -- 172.21.15.42:0/2331303104 >> 172.21.15.42:0/2331303104 conn(0x7eff2c0a2420 msgr2=0x7eff2c0a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:55:21.783 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.767+0000 7eff320f6640 1 -- 172.21.15.42:0/2331303104 shutdown_connections 2024-09-16T03:55:21.783 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.767+0000 7eff320f6640 1 -- 172.21.15.42:0/2331303104 wait complete. 2024-09-16T03:55:21.783 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.767+0000 7eff320f6640 10 monclient: build_initial_monmap 2024-09-16T03:55:21.783 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.767+0000 7eff320f6640 10 monclient: monmap: 2024-09-16T03:55:21.783 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:55:21.783 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:55:21.783 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:55:21.783 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:55:21.784 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:55:21.784 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:55:21.784 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:55:21.784 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:55:21.784 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:55:21.784 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:55:21.784 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.767+0000 7eff320f6640 1 librados: starting msgr at 2024-09-16T03:55:21.785 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.767+0000 7eff320f6640 1 librados: starting objecter 2024-09-16T03:55:21.785 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.767+0000 7eff320f6640 1 Processor -- start 2024-09-16T03:55:21.785 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.767+0000 7eff320f6640 1 -- start start 2024-09-16T03:55:21.785 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.767+0000 7eff320f6640 1 librados: setting wanted keys 2024-09-16T03:55:21.785 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.767+0000 7eff320f6640 1 librados: calling monclient init 2024-09-16T03:55:21.785 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.767+0000 7eff320f6640 10 monclient: init 2024-09-16T03:55:21.785 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.771+0000 7eff320f6640 10 monclient: _reopen_session rank -1 2024-09-16T03:55:21.785 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.771+0000 7eff320f6640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:55:21.785 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.771+0000 7eff320f6640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7eff2c0a1330 0x7eff2c1cee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:55:21.785 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.771+0000 7eff320f6640 10 monclient(hunting): picked mon.noname-c con 0x7eff2c0a1330 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:55:21.785 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.771+0000 7eff320f6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7eff2c0a1710 0x7eff2c1344c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:55:21.785 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.771+0000 7eff320f6640 10 monclient(hunting): picked mon.noname-b con 0x7eff2c0a1710 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:55:21.785 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.771+0000 7eff320f6640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7eff2c1393a0 0x7eff2c13cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:55:21.785 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.771+0000 7eff320f6640 10 monclient(hunting): picked mon.noname-a con 0x7eff2c1393a0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:55:21.786 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.771+0000 7eff320f6640 10 monclient(hunting): start opening mon connection 2024-09-16T03:55:21.786 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.771+0000 7eff320f6640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7eff2c13c8f0 con 0x7eff2c1393a0 2024-09-16T03:55:21.786 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.771+0000 7eff320f6640 10 monclient(hunting): start opening mon connection 2024-09-16T03:55:21.786 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.771+0000 7eff320f6640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7eff2c13c770 con 0x7eff2c0a1710 2024-09-16T03:55:21.786 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.771+0000 7eff320f6640 10 monclient(hunting): start opening mon connection 2024-09-16T03:55:21.786 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.771+0000 7eff320f6640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7eff2c13ca70 con 0x7eff2c0a1330 2024-09-16T03:55:21.786 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.771+0000 7eff320f6640 10 monclient(hunting): _renew_subs 2024-09-16T03:55:21.786 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.771+0000 7eff320f6640 10 monclient(hunting): authenticate will time out at 1768.740239s 2024-09-16T03:55:21.786 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.771+0000 7eff2b7fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7eff2c0a1330 0x7eff2c1cee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:55:21.786 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.771+0000 7eff2b7fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7eff2c0a1330 0x7eff2c1cee80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:38078/0 (socket says 172.21.15.42:38078) 2024-09-16T03:55:21.786 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.771+0000 7eff2b7fe640 1 -- 172.21.15.42:0/3694836924 learned_addr learned my addr 172.21.15.42:0/3694836924 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:55:21.786 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.771+0000 7eff2b7fe640 10 monclient(hunting): get_auth_request con 0x7eff2c0a1330 auth_method 0 2024-09-16T03:55:21.786 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.771+0000 7eff2b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:55:21.787 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.771+0000 7eff2b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:55:21.787 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.771+0000 7eff2b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:55:21.787 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.771+0000 7eff2bfff640 1 --2- 172.21.15.42:0/3694836924 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7eff2c1393a0 0x7eff2c13cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:55:21.787 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.771+0000 7eff2bfff640 10 monclient(hunting): get_auth_request con 0x7eff2c1393a0 auth_method 0 2024-09-16T03:55:21.787 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.771+0000 7eff2bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:55:21.787 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.771+0000 7eff2bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:55:21.787 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.771+0000 7eff2bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:55:21.787 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.771+0000 7eff2b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:55:21.787 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.771+0000 7eff2b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:55:21.787 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.771+0000 7eff2b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:55:21.787 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.771+0000 7eff2bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:55:21.787 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.771+0000 7eff2bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:55:21.787 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.771+0000 7eff2bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:55:21.787 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.771+0000 7eff2affd640 1 --2- 172.21.15.42:0/3694836924 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7eff2c0a1710 0x7eff2c1344c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:55:21.787 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.771+0000 7eff2affd640 10 monclient(hunting): get_auth_request con 0x7eff2c0a1710 auth_method 0 2024-09-16T03:55:21.788 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.771+0000 7eff2affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:55:21.788 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.771+0000 7eff2affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:55:21.788 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.771+0000 7eff2affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:55:21.788 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.771+0000 7eff2b7fe640 10 monclient(hunting): handle_auth_done global_id 854183 payload 824 2024-09-16T03:55:21.788 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.771+0000 7eff2b7fe640 1 -- 172.21.15.42:0/3694836924 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7eff2c0a1710 msgr2=0x7eff2c1344c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:55:21.788 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.771+0000 7eff2b7fe640 1 --2- 172.21.15.42:0/3694836924 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7eff2c0a1710 0x7eff2c1344c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:55:21.788 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.771+0000 7eff2b7fe640 1 -- 172.21.15.42:0/3694836924 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7eff2c1393a0 msgr2=0x7eff2c13cd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:55:21.788 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.771+0000 7eff2b7fe640 1 --2- 172.21.15.42:0/3694836924 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7eff2c1393a0 0x7eff2c13cd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:55:21.788 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.771+0000 7eff2b7fe640 10 monclient: _finish_hunting 0 2024-09-16T03:55:21.788 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.771+0000 7eff2b7fe640 1 monclient: found mon.noname-c 2024-09-16T03:55:21.788 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.771+0000 7eff2b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:55:21.788 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.771+0000 7eff2b7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:55:21.788 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.771+0000 7eff2b7fe640 1 -- 172.21.15.42:0/3694836924 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff2c0a49e0 con 0x7eff2c0a1330 2024-09-16T03:55:21.789 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.771+0000 7eff2b7fe640 1 --2- 172.21.15.42:0/3694836924 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7eff2c0a1330 0x7eff2c1cee80 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7eff1c00efc0 tx=0x7eff1c00c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:55:21.789 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.771+0000 7eff23fff640 10 client.?.objecter ms_handle_connect 0x7eff2c0a1330 2024-09-16T03:55:21.789 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.771+0000 7eff23fff640 10 client.?.objecter resend_mon_ops 2024-09-16T03:55:21.789 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.771+0000 7eff23fff640 1 -- 172.21.15.42:0/3694836924 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff1c018020 con 0x7eff2c0a1330 2024-09-16T03:55:21.789 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.771+0000 7eff23fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:55:21.789 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.771+0000 7eff23fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:55:21.789 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.771+0000 7eff23fff640 10 monclient: dump: 2024-09-16T03:55:21.789 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:55:21.789 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:55:21.789 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:21.789 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:21.789 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:55:21.789 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:55:21.789 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:55:21.789 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:55:21.790 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:55:21.790 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:55:21.790 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.771+0000 7eff23fff640 10 monclient: _finish_auth 0 2024-09-16T03:55:21.790 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.771+0000 7eff23fff640 10 monclient: _check_auth_tickets 2024-09-16T03:55:21.790 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.771+0000 7eff23fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:55:21.790 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.771+0000 7eff23fff640 1 -- 172.21.15.42:0/3694836924 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff1c014480 con 0x7eff2c0a1330 2024-09-16T03:55:21.790 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.771+0000 7eff23fff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:55:21.790 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.771+0000 7eff23fff640 1 -- 172.21.15.42:0/3694836924 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff1c014850 con 0x7eff2c0a1330 2024-09-16T03:55:21.790 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.771+0000 7eff23fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:55:21.790 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.771+0000 7eff23fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:55:21.790 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.771+0000 7eff23fff640 10 monclient: dump: 2024-09-16T03:55:21.790 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:55:21.790 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:55:21.790 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:21.791 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:21.791 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:55:21.791 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:55:21.791 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:55:21.791 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:55:21.791 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:55:21.791 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:55:21.791 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.771+0000 7eff29ffb640 4 set_mon_vals no callback set 2024-09-16T03:55:21.791 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.771+0000 7eff320f6640 5 monclient: authenticate success, global_id 854183 2024-09-16T03:55:21.791 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.771+0000 7eff320f6640 10 monclient: _renew_subs 2024-09-16T03:55:21.791 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.771+0000 7eff320f6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:55:21.792 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.771+0000 7eff320f6640 1 -- 172.21.15.42:0/3694836924 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff2c073a30 con 0x7eff2c0a1330 2024-09-16T03:55:21.792 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.771+0000 7eff320f6640 10 client.854183.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:55:21.792 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.771+0000 7eff320f6640 10 monclient: _renew_subs 2024-09-16T03:55:21.792 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.771+0000 7eff320f6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:55:21.792 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.771+0000 7eff320f6640 1 -- 172.21.15.42:0/3694836924 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7eff2c072060 con 0x7eff2c0a1330 2024-09-16T03:55:21.792 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.771+0000 7eff320f6640 1 librados: init done 2024-09-16T03:55:21.792 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.771+0000 7eff320f6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:55:21.792 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.771+0000 7eff320f6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:55:21.792 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.771+0000 7eff320f6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:55:21.792 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.771+0000 7eff320f6640 1 -- 172.21.15.42:0/3694836924 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7eff2c09f180 con 0x7eff2c0a1330 2024-09-16T03:55:21.792 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.771+0000 7eff23fff640 1 -- 172.21.15.42:0/3694836924 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7eff1c01b020 con 0x7eff2c0a1330 2024-09-16T03:55:21.792 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.771+0000 7eff23fff640 1 --2- 172.21.15.42:0/3694836924 >> v2:172.21.15.154:6812/254845342 conn(0x7eff0c037810 0x7eff0c039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:55:21.792 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.771+0000 7eff23fff640 1 -- 172.21.15.42:0/3694836924 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(408..408 src has 330..408) ==== 5713+0+0 (secure 0 0 0) 0x7eff1c049ca0 con 0x7eff2c0a1330 2024-09-16T03:55:21.793 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.771+0000 7eff23fff640 10 client.854183.objecter ms_dispatch 0x7eff2c000ba0 osd_map(408..408 src has 330..408) 2024-09-16T03:55:21.793 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.771+0000 7eff23fff640 3 client.854183.objecter handle_osd_map got epochs [408,408] > 0 2024-09-16T03:55:21.793 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.771+0000 7eff23fff640 3 client.854183.objecter handle_osd_map decoding full epoch 408 2024-09-16T03:55:21.793 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.771+0000 7eff23fff640 20 client.854183.objecter dump_active .. 0 homeless 2024-09-16T03:55:21.793 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.771+0000 7eff2affd640 1 --2- 172.21.15.42:0/3694836924 >> v2:172.21.15.154:6812/254845342 conn(0x7eff0c037810 0x7eff0c039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:55:21.794 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.771+0000 7eff2affd640 10 monclient: get_auth_request con 0x7eff0c037810 auth_method 0 2024-09-16T03:55:21.794 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.771+0000 7eff2affd640 1 --2- 172.21.15.42:0/3694836924 >> v2:172.21.15.154:6812/254845342 conn(0x7eff0c037810 0x7eff0c039cd0 secure :-1 s=READY pgs=3630 cs=0 l=1 rev1=1 crypto rx=0x7eff2c13e5b0 tx=0x7eff14009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:55:21.794 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.771+0000 7eff23fff640 10 client.854183.objecter ms_handle_connect 0x7eff0c037810 2024-09-16T03:55:21.794 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.775+0000 7eff23fff640 1 -- 172.21.15.42:0/3694836924 <== mon.2 v2:172.21.15.42:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7eff1c022030 con 0x7eff2c0a1330 2024-09-16T03:55:21.794 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.775+0000 7eff23fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:55:21.794 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:21.775+0000 7eff23fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:55:22.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:22.003+0000 7eff320f6640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:55:22.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:22.003+0000 7eff320f6640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:55:22.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:22.003+0000 7eff320f6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:55:22.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:22.003+0000 7eff320f6640 1 -- 172.21.15.42:0/3694836924 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7eff2c13e9b0 con 0x7eff2c0a1330 2024-09-16T03:55:22.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:22.003+0000 7eff23fff640 1 -- 172.21.15.42:0/3694836924 <== mon.2 v2:172.21.15.42:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7eff2c073a30 con 0x7eff2c0a1330 2024-09-16T03:55:22.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:22.003+0000 7eff23fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:55:22.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:22.003+0000 7eff23fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:55:22.008 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":114,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":22,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:55:22.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:22.007+0000 7eff320f6640 10 librados: watch_flush enter 2024-09-16T03:55:22.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:22.007+0000 7eff320f6640 10 librados: watch_flush exit 2024-09-16T03:55:22.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:22.007+0000 7eff320f6640 20 client.854183.objecter shutdown clearing up homeless session... 2024-09-16T03:55:22.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:22.007+0000 7eff320f6640 10 client.854183.objecter successfully canceled tick 2024-09-16T03:55:22.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:22.007+0000 7eff320f6640 1 -- 172.21.15.42:0/3694836924 >> v2:172.21.15.154:6812/254845342 conn(0x7eff0c037810 msgr2=0x7eff0c039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:55:22.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:22.007+0000 7eff320f6640 1 --2- 172.21.15.42:0/3694836924 >> v2:172.21.15.154:6812/254845342 conn(0x7eff0c037810 0x7eff0c039cd0 secure :-1 s=READY pgs=3630 cs=0 l=1 rev1=1 crypto rx=0x7eff2c13e5b0 tx=0x7eff14009040 comp rx=0 tx=0).stop 2024-09-16T03:55:22.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:22.007+0000 7eff320f6640 10 monclient: shutdown 2024-09-16T03:55:22.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:22.007+0000 7eff320f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:55:22.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:22.007+0000 7eff320f6640 1 -- 172.21.15.42:0/3694836924 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7eff2c0a1330 msgr2=0x7eff2c1cee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:55:22.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:22.007+0000 7eff320f6640 1 --2- 172.21.15.42:0/3694836924 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7eff2c0a1330 0x7eff2c1cee80 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7eff1c00efc0 tx=0x7eff1c00c490 comp rx=0 tx=0).stop 2024-09-16T03:55:22.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:22.007+0000 7eff320f6640 1 -- 172.21.15.42:0/3694836924 shutdown_connections 2024-09-16T03:55:22.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:22.007+0000 7eff320f6640 1 --2- 172.21.15.42:0/3694836924 >> v2:172.21.15.154:6812/254845342 conn(0x7eff0c037810 0x7eff0c039cd0 unknown :-1 s=CLOSED pgs=3630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:55:22.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:22.007+0000 7eff320f6640 1 --2- 172.21.15.42:0/3694836924 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7eff2c1393a0 0x7eff2c13cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:55:22.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:22.007+0000 7eff320f6640 1 --2- 172.21.15.42:0/3694836924 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7eff2c0a1710 0x7eff2c1344c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:55:22.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:22.007+0000 7eff320f6640 1 --2- 172.21.15.42:0/3694836924 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7eff2c0a1330 0x7eff2c1cee80 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:55:22.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:22.007+0000 7eff320f6640 1 -- 172.21.15.42:0/3694836924 >> 172.21.15.42:0/3694836924 conn(0x7eff2c0a2420 msgr2=0x7eff2c0a3f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:55:22.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:22.007+0000 7eff320f6640 1 -- 172.21.15.42:0/3694836924 shutdown_connections 2024-09-16T03:55:22.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:22.007+0000 7eff320f6640 1 -- 172.21.15.42:0/3694836924 wait complete. 2024-09-16T03:55:22.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:22.007+0000 7eff320f6640 1 librados: shutdown 2024-09-16T03:55:25.028 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:55:25.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.155+0000 7f07c0ad8640 10 monclient: get_monmap_and_config 2024-09-16T03:55:25.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.155+0000 7f07c0ad8640 10 monclient: build_initial_monmap 2024-09-16T03:55:25.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.155+0000 7f07c0ad8640 10 monclient: monmap: 2024-09-16T03:55:25.159 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:55:25.159 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:55:25.159 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:55:25.158014+0000 2024-09-16T03:55:25.160 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:55:25.158014+0000 2024-09-16T03:55:25.160 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:55:25.160 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:55:25.160 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:55:25.160 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:55:25.160 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:55:25.160 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:55:25.160 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:55:25.160 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:55:25.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.155+0000 7f07c0ad8640 1 Processor -- start 2024-09-16T03:55:25.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.155+0000 7f07c0ad8640 1 -- start start 2024-09-16T03:55:25.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.155+0000 7f07c0ad8640 10 monclient: init 2024-09-16T03:55:25.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.155+0000 7f07c0ad8640 10 monclient: _reopen_session rank -1 2024-09-16T03:55:25.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.155+0000 7f07c0ad8640 10 monclient: _add_conns ranks=[1,3,4,2,0] 2024-09-16T03:55:25.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.155+0000 7f07c0ad8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f07bc0a1330 addr v1:172.21.15.154:6789/0 2024-09-16T03:55:25.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.155+0000 7f07c0ad8640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f07bc06dac0 0x7f07bc13a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:55:25.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.155+0000 7f07c0ad8640 10 monclient(hunting): picked mon.noname-a con 0x7f07bc06dac0 addr v2:172.21.15.42:3300/0 2024-09-16T03:55:25.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.155+0000 7f07c0ad8640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f07bc0a1e20 0x7f07bc13c890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:55:25.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.155+0000 7f07c0ad8640 10 monclient(hunting): picked mon.noname-c con 0x7f07bc0a1e20 addr v2:172.21.15.154:3300/0 2024-09-16T03:55:25.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.155+0000 7f07c0ad8640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f07bc09c090 con 0x7f07bc0a1330 2024-09-16T03:55:25.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.155+0000 7f07c0ad8640 10 monclient(hunting): start opening mon connection 2024-09-16T03:55:25.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.155+0000 7f07c0ad8640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f07bc09f610 con 0x7f07bc06dac0 2024-09-16T03:55:25.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.155+0000 7f07c0ad8640 10 monclient(hunting): start opening mon connection 2024-09-16T03:55:25.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.155+0000 7f07c0ad8640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f07bc09f8f0 con 0x7f07bc0a1e20 2024-09-16T03:55:25.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.155+0000 7f07c0ad8640 10 monclient(hunting): _renew_subs 2024-09-16T03:55:25.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.155+0000 7f07c0ad8640 10 monclient(hunting): authenticate will time out at 1772.127477s 2024-09-16T03:55:25.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.155+0000 7f07b9d9b640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f07bc06dac0 0x7f07bc13a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:55:25.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.155+0000 7f07b9d9b640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f07bc06dac0 0x7f07bc13a340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:34730/0 (socket says 172.21.15.42:34730) 2024-09-16T03:55:25.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.155+0000 7f07b9d9b640 1 -- 172.21.15.42:0/3433360211 learned_addr learned my addr 172.21.15.42:0/3433360211 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:55:25.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.155+0000 7f07b9d9b640 10 monclient(hunting): get_auth_request con 0x7f07bc06dac0 auth_method 0 2024-09-16T03:55:25.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.155+0000 7f07b9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:55:25.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.155+0000 7f07b9d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:55:25.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.155+0000 7f07b9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:55:25.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.155+0000 7f07b9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:55:25.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.155+0000 7f07b9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:55:25.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.155+0000 7f07b9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:55:25.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.155+0000 7f07bad9d640 1 --2- 172.21.15.42:0/3433360211 >> v2:172.21.15.154:3300/0 conn(0x7f07bc0a1e20 0x7f07bc13c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:55:25.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.155+0000 7f07bad9d640 10 monclient(hunting): get_auth_request con 0x7f07bc0a1e20 auth_method 0 2024-09-16T03:55:25.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.155+0000 7f07bad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:55:25.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.155+0000 7f07bad9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:55:25.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.155+0000 7f07bad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:55:25.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.159+0000 7f07b9d9b640 10 monclient(hunting): handle_auth_done global_id 824556 payload 274 2024-09-16T03:55:25.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.159+0000 7f07b9d9b640 1 -- 172.21.15.42:0/3433360211 >> v2:172.21.15.154:3300/0 conn(0x7f07bc0a1e20 msgr2=0x7f07bc13c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:55:25.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.159+0000 7f07b9d9b640 1 --2- 172.21.15.42:0/3433360211 >> v2:172.21.15.154:3300/0 conn(0x7f07bc0a1e20 0x7f07bc13c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:55:25.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.159+0000 7f07b9d9b640 1 -- 172.21.15.42:0/3433360211 >> v1:172.21.15.154:6789/0 conn(0x7f07bc0a1330 legacy=0x7f07bc0a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:55:25.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.159+0000 7f07b9d9b640 10 monclient: _finish_hunting 0 2024-09-16T03:55:25.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.159+0000 7f07b9d9b640 1 monclient: found mon.noname-a 2024-09-16T03:55:25.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.159+0000 7f07b9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:55:25.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.159+0000 7f07b9d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:55:25.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.159+0000 7f07b9d9b640 1 -- 172.21.15.42:0/3433360211 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07bc09fbd0 con 0x7f07bc06dac0 2024-09-16T03:55:25.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.159+0000 7f07b9d9b640 1 --2- 172.21.15.42:0/3433360211 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f07bc06dac0 0x7f07bc13a340 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f07bc09c800 tx=0x7f07a401c430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f162c913539733a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:55:25.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.159+0000 7f07b959a640 1 -- 172.21.15.42:0/3433360211 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07a40b0070 con 0x7f07bc06dac0 2024-09-16T03:55:25.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.159+0000 7f07b959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:55:25.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.159+0000 7f07b959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:55:25.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.159+0000 7f07b959a640 10 monclient: dump: 2024-09-16T03:55:25.173 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:55:25.173 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:55:25.173 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:25.173 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:25.173 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:55:25.173 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:55:25.173 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:55:25.173 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:55:25.173 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:55:25.173 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:55:25.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.159+0000 7f07b959a640 10 monclient: _finish_auth 0 2024-09-16T03:55:25.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.159+0000 7f07b959a640 10 monclient: _check_auth_tickets 2024-09-16T03:55:25.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.159+0000 7f07b959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:55:25.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.159+0000 7f07b959a640 1 -- 172.21.15.42:0/3433360211 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f07a40ab600 con 0x7f07bc06dac0 2024-09-16T03:55:25.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.159+0000 7f07b959a640 10 monclient: handle_config config(0 keys) 2024-09-16T03:55:25.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.159+0000 7f07b959a640 1 -- 172.21.15.42:0/3433360211 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07a40ab9d0 con 0x7f07bc06dac0 2024-09-16T03:55:25.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.159+0000 7f07c0ad8640 5 monclient: authenticate success, global_id 824556 2024-09-16T03:55:25.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.159+0000 7f07c0ad8640 10 monclient: get_monmap_and_config success 2024-09-16T03:55:25.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.159+0000 7f07c0ad8640 4 set_mon_vals no callback set 2024-09-16T03:55:25.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.159+0000 7f07b959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:55:25.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.159+0000 7f07b959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:55:25.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.159+0000 7f07b959a640 10 monclient: dump: 2024-09-16T03:55:25.174 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:55:25.174 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:55:25.174 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:25.174 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:25.174 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:55:25.174 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:55:25.175 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:55:25.175 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:55:25.175 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:55:25.175 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:55:25.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.159+0000 7f07c0ad8640 10 monclient: shutdown 2024-09-16T03:55:25.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.159+0000 7f07c0ad8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:55:25.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.159+0000 7f07c0ad8640 1 -- 172.21.15.42:0/3433360211 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f07bc06dac0 msgr2=0x7f07bc13a340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:55:25.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.159+0000 7f07c0ad8640 1 --2- 172.21.15.42:0/3433360211 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f07bc06dac0 0x7f07bc13a340 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f07bc09c800 tx=0x7f07a401c430 comp rx=0 tx=0).stop 2024-09-16T03:55:25.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.159+0000 7f07c0ad8640 1 -- 172.21.15.42:0/3433360211 shutdown_connections 2024-09-16T03:55:25.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.159+0000 7f07c0ad8640 1 --2- 172.21.15.42:0/3433360211 >> v2:172.21.15.154:3300/0 conn(0x7f07bc0a1e20 0x7f07bc13c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:55:25.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.159+0000 7f07c0ad8640 1 --2- 172.21.15.42:0/3433360211 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f07bc06dac0 0x7f07bc13a340 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:55:25.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.159+0000 7f07c0ad8640 1 -- 172.21.15.42:0/3433360211 >> 172.21.15.42:0/3433360211 conn(0x7f07bc0a2420 msgr2=0x7f07bc0a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:55:25.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.159+0000 7f07c0ad8640 1 -- 172.21.15.42:0/3433360211 shutdown_connections 2024-09-16T03:55:25.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.159+0000 7f07c0ad8640 1 -- 172.21.15.42:0/3433360211 wait complete. 2024-09-16T03:55:25.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.159+0000 7f07c0ad8640 10 monclient: build_initial_monmap 2024-09-16T03:55:25.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.159+0000 7f07c0ad8640 10 monclient: monmap: 2024-09-16T03:55:25.176 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:55:25.176 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:55:25.176 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:55:25.176 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:55:25.176 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:55:25.176 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:55:25.176 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:55:25.176 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:55:25.176 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:55:25.176 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:55:25.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.159+0000 7f07c0ad8640 1 librados: starting msgr at 2024-09-16T03:55:25.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.159+0000 7f07c0ad8640 1 librados: starting objecter 2024-09-16T03:55:25.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.163+0000 7f07c0ad8640 1 Processor -- start 2024-09-16T03:55:25.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.163+0000 7f07c0ad8640 1 -- start start 2024-09-16T03:55:25.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.163+0000 7f07c0ad8640 1 librados: setting wanted keys 2024-09-16T03:55:25.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.163+0000 7f07c0ad8640 1 librados: calling monclient init 2024-09-16T03:55:25.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.163+0000 7f07c0ad8640 10 monclient: init 2024-09-16T03:55:25.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.163+0000 7f07c0ad8640 10 monclient: _reopen_session rank -1 2024-09-16T03:55:25.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.163+0000 7f07c0ad8640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:55:25.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.163+0000 7f07c0ad8640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f07bc0a1330 0x7f07bc1ceff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:55:25.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.163+0000 7f07c0ad8640 10 monclient(hunting): picked mon.noname-b con 0x7f07bc0a1330 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:55:25.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.163+0000 7f07c0ad8640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f07bc0a1e20 0x7f07bc135670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:55:25.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.163+0000 7f07c0ad8640 10 monclient(hunting): picked mon.noname-a con 0x7f07bc0a1e20 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:55:25.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.163+0000 7f07c0ad8640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f07bc135bb0 0x7f07bc09cb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:55:25.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.163+0000 7f07c0ad8640 10 monclient(hunting): picked mon.noname-c con 0x7f07bc135bb0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:55:25.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.163+0000 7f07c0ad8640 10 monclient(hunting): start opening mon connection 2024-09-16T03:55:25.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.163+0000 7f07c0ad8640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f07bc13f800 con 0x7f07bc0a1e20 2024-09-16T03:55:25.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.163+0000 7f07c0ad8640 10 monclient(hunting): start opening mon connection 2024-09-16T03:55:25.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.163+0000 7f07c0ad8640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f07bc09ef80 con 0x7f07bc0a1330 2024-09-16T03:55:25.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.163+0000 7f07c0ad8640 10 monclient(hunting): start opening mon connection 2024-09-16T03:55:25.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.163+0000 7f07c0ad8640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f07bc13f980 con 0x7f07bc135bb0 2024-09-16T03:55:25.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.163+0000 7f07c0ad8640 10 monclient(hunting): _renew_subs 2024-09-16T03:55:25.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.163+0000 7f07c0ad8640 10 monclient(hunting): authenticate will time out at 1772.132638s 2024-09-16T03:55:25.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.163+0000 7f07b9d9b640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f07bc0a1e20 0x7f07bc135670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:55:25.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.163+0000 7f07b9d9b640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f07bc0a1e20 0x7f07bc135670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:34732/0 (socket says 172.21.15.42:34732) 2024-09-16T03:55:25.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.163+0000 7f07b9d9b640 1 -- 172.21.15.42:0/857747251 learned_addr learned my addr 172.21.15.42:0/857747251 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:55:25.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.163+0000 7f07b9d9b640 10 monclient(hunting): get_auth_request con 0x7f07bc0a1e20 auth_method 0 2024-09-16T03:55:25.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.163+0000 7f07b9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:55:25.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.163+0000 7f07b9d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:55:25.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.163+0000 7f07b9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:55:25.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.163+0000 7f07bad9d640 1 --2- 172.21.15.42:0/857747251 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f07bc135bb0 0x7f07bc09cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:55:25.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.163+0000 7f07bad9d640 10 monclient(hunting): get_auth_request con 0x7f07bc135bb0 auth_method 0 2024-09-16T03:55:25.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.163+0000 7f07bad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:55:25.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.163+0000 7f07bad9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:55:25.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.163+0000 7f07bad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:55:25.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.163+0000 7f07b9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:55:25.180 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.163+0000 7f07b9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:55:25.180 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.163+0000 7f07b9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:55:25.180 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.163+0000 7f07ba59c640 1 --2- 172.21.15.42:0/857747251 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f07bc0a1330 0x7f07bc1ceff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:55:25.180 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.163+0000 7f07bad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:55:25.180 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.163+0000 7f07bad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:55:25.180 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.163+0000 7f07bad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:55:25.180 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.163+0000 7f07ba59c640 10 monclient(hunting): get_auth_request con 0x7f07bc0a1330 auth_method 0 2024-09-16T03:55:25.180 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.163+0000 7f07ba59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:55:25.180 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.163+0000 7f07ba59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:55:25.180 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.163+0000 7f07ba59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:55:25.180 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.163+0000 7f07b9d9b640 10 monclient(hunting): handle_auth_done global_id 824559 payload 824 2024-09-16T03:55:25.180 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.163+0000 7f07b9d9b640 1 -- 172.21.15.42:0/857747251 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f07bc135bb0 msgr2=0x7f07bc09cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:55:25.180 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.163+0000 7f07b9d9b640 1 --2- 172.21.15.42:0/857747251 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f07bc135bb0 0x7f07bc09cb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:55:25.181 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.163+0000 7f07b9d9b640 1 -- 172.21.15.42:0/857747251 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f07bc0a1330 msgr2=0x7f07bc1ceff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:55:25.181 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.163+0000 7f07b9d9b640 1 --2- 172.21.15.42:0/857747251 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f07bc0a1330 0x7f07bc1ceff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:55:25.181 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.163+0000 7f07b9d9b640 10 monclient: _finish_hunting 0 2024-09-16T03:55:25.181 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.163+0000 7f07b9d9b640 1 monclient: found mon.noname-a 2024-09-16T03:55:25.181 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.163+0000 7f07b9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:55:25.181 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.163+0000 7f07b9d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:55:25.181 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.163+0000 7f07b9d9b640 1 -- 172.21.15.42:0/857747251 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07bc121d60 con 0x7f07bc0a1e20 2024-09-16T03:55:25.181 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.163+0000 7f07b9d9b640 1 --2- 172.21.15.42:0/857747251 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f07bc0a1e20 0x7f07bc135670 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f07a40ae040 tx=0x7f07a401ac70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:55:25.181 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.163+0000 7f07b2ffd640 10 client.?.objecter ms_handle_connect 0x7f07bc0a1e20 2024-09-16T03:55:25.181 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.163+0000 7f07b2ffd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:55:25.181 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.163+0000 7f07b2ffd640 1 -- 172.21.15.42:0/857747251 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07a40ba020 con 0x7f07bc0a1e20 2024-09-16T03:55:25.181 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.163+0000 7f07b2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:55:25.181 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.163+0000 7f07b2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:55:25.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.163+0000 7f07b2ffd640 10 monclient: dump: 2024-09-16T03:55:25.182 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:55:25.182 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:55:25.182 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:25.182 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:25.182 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:55:25.182 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:55:25.182 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:55:25.182 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:55:25.182 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:55:25.182 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:55:25.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.163+0000 7f07b2ffd640 10 monclient: _finish_auth 0 2024-09-16T03:55:25.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.163+0000 7f07b2ffd640 10 monclient: _check_auth_tickets 2024-09-16T03:55:25.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.163+0000 7f07b2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:55:25.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.163+0000 7f07b2ffd640 1 -- 172.21.15.42:0/857747251 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f07a40b6620 con 0x7f07bc0a1e20 2024-09-16T03:55:25.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.163+0000 7f07b2ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:55:25.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.163+0000 7f07b2ffd640 1 -- 172.21.15.42:0/857747251 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07a40b0040 con 0x7f07bc0a1e20 2024-09-16T03:55:25.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.163+0000 7f07c0ad8640 5 monclient: authenticate success, global_id 824559 2024-09-16T03:55:25.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.163+0000 7f07c0ad8640 10 monclient: _renew_subs 2024-09-16T03:55:25.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.163+0000 7f07c0ad8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:55:25.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.163+0000 7f07c0ad8640 1 -- 172.21.15.42:0/857747251 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07bc06e180 con 0x7f07bc0a1e20 2024-09-16T03:55:25.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.163+0000 7f07b2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:55:25.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.163+0000 7f07b2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:55:25.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.163+0000 7f07b2ffd640 10 monclient: dump: 2024-09-16T03:55:25.183 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:55:25.183 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:55:25.183 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:25.183 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:25.184 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:55:25.184 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:55:25.184 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:55:25.184 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:55:25.184 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:55:25.184 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:55:25.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.163+0000 7f07b8d99640 4 set_mon_vals no callback set 2024-09-16T03:55:25.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.163+0000 7f07c0ad8640 10 client.824559.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:55:25.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.163+0000 7f07c0ad8640 10 monclient: _renew_subs 2024-09-16T03:55:25.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.163+0000 7f07c0ad8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:55:25.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.163+0000 7f07c0ad8640 1 -- 172.21.15.42:0/857747251 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07bc073a30 con 0x7f07bc0a1e20 2024-09-16T03:55:25.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.163+0000 7f07c0ad8640 1 librados: init done 2024-09-16T03:55:25.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.163+0000 7f07b2ffd640 1 -- 172.21.15.42:0/857747251 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f07a40bd020 con 0x7f07bc0a1e20 2024-09-16T03:55:25.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.163+0000 7f07c0ad8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:55:25.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.163+0000 7f07c0ad8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:55:25.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.163+0000 7f07c0ad8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:55:25.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.163+0000 7f07c0ad8640 1 -- 172.21.15.42:0/857747251 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f07bc0a1710 con 0x7f07bc0a1e20 2024-09-16T03:55:25.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.163+0000 7f07b2ffd640 1 --2- 172.21.15.42:0/857747251 >> v2:172.21.15.154:6812/254845342 conn(0x7f079c037810 0x7f079c039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:55:25.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.163+0000 7f07b2ffd640 1 -- 172.21.15.42:0/857747251 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(411..411 src has 330..411) ==== 5764+0+0 (secure 0 0 0) 0x7f07a40e3210 con 0x7f07bc0a1e20 2024-09-16T03:55:25.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.163+0000 7f07b2ffd640 10 client.824559.objecter ms_dispatch 0x7f07bc000ba0 osd_map(411..411 src has 330..411) 2024-09-16T03:55:25.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.163+0000 7f07b2ffd640 3 client.824559.objecter handle_osd_map got epochs [411,411] > 0 2024-09-16T03:55:25.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.163+0000 7f07b2ffd640 3 client.824559.objecter handle_osd_map decoding full epoch 411 2024-09-16T03:55:25.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.163+0000 7f07b2ffd640 20 client.824559.objecter dump_active .. 0 homeless 2024-09-16T03:55:25.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.163+0000 7f07ba59c640 1 --2- 172.21.15.42:0/857747251 >> v2:172.21.15.154:6812/254845342 conn(0x7f079c037810 0x7f079c039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:55:25.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.163+0000 7f07ba59c640 10 monclient: get_auth_request con 0x7f079c037810 auth_method 0 2024-09-16T03:55:25.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.167+0000 7f07ba59c640 1 --2- 172.21.15.42:0/857747251 >> v2:172.21.15.154:6812/254845342 conn(0x7f079c037810 0x7f079c039cd0 secure :-1 s=READY pgs=3631 cs=0 l=1 rev1=1 crypto rx=0x7f07ac005530 tx=0x7f07ac009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:55:25.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.167+0000 7f07b2ffd640 10 client.824559.objecter ms_handle_connect 0x7f079c037810 2024-09-16T03:55:25.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.167+0000 7f07b2ffd640 1 -- 172.21.15.42:0/857747251 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f07a40c4030 con 0x7f07bc0a1e20 2024-09-16T03:55:25.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.167+0000 7f07b2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:55:25.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.167+0000 7f07b2ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:55:25.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.391+0000 7f07c0ad8640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:55:25.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.391+0000 7f07c0ad8640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:55:25.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.391+0000 7f07c0ad8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:55:25.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.391+0000 7f07c0ad8640 1 -- 172.21.15.42:0/857747251 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f07bc069040 con 0x7f07bc0a1e20 2024-09-16T03:55:25.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.395+0000 7f07b2ffd640 1 -- 172.21.15.42:0/857747251 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7f07a40160b0 con 0x7f07bc0a1e20 2024-09-16T03:55:25.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.395+0000 7f07b2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:55:25.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.395+0000 7f07b2ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:55:25.398 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":114,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":25,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:55:25.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.395+0000 7f07c0ad8640 10 librados: watch_flush enter 2024-09-16T03:55:25.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.395+0000 7f07c0ad8640 10 librados: watch_flush exit 2024-09-16T03:55:25.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.395+0000 7f07c0ad8640 20 client.824559.objecter shutdown clearing up homeless session... 2024-09-16T03:55:25.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.395+0000 7f07c0ad8640 10 client.824559.objecter successfully canceled tick 2024-09-16T03:55:25.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.395+0000 7f07c0ad8640 1 -- 172.21.15.42:0/857747251 >> v2:172.21.15.154:6812/254845342 conn(0x7f079c037810 msgr2=0x7f079c039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:55:25.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.395+0000 7f07c0ad8640 1 --2- 172.21.15.42:0/857747251 >> v2:172.21.15.154:6812/254845342 conn(0x7f079c037810 0x7f079c039cd0 secure :-1 s=READY pgs=3631 cs=0 l=1 rev1=1 crypto rx=0x7f07ac005530 tx=0x7f07ac009040 comp rx=0 tx=0).stop 2024-09-16T03:55:25.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.395+0000 7f07c0ad8640 10 monclient: shutdown 2024-09-16T03:55:25.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.395+0000 7f07c0ad8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:55:25.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.395+0000 7f07c0ad8640 1 -- 172.21.15.42:0/857747251 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f07bc0a1e20 msgr2=0x7f07bc135670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:55:25.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.395+0000 7f07c0ad8640 1 --2- 172.21.15.42:0/857747251 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f07bc0a1e20 0x7f07bc135670 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f07a40ae040 tx=0x7f07a401ac70 comp rx=0 tx=0).stop 2024-09-16T03:55:25.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.395+0000 7f07c0ad8640 1 -- 172.21.15.42:0/857747251 shutdown_connections 2024-09-16T03:55:25.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.395+0000 7f07c0ad8640 1 --2- 172.21.15.42:0/857747251 >> v2:172.21.15.154:6812/254845342 conn(0x7f079c037810 0x7f079c039cd0 unknown :-1 s=CLOSED pgs=3631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:55:25.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.395+0000 7f07c0ad8640 1 --2- 172.21.15.42:0/857747251 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f07bc135bb0 0x7f07bc09cb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:55:25.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.395+0000 7f07c0ad8640 1 --2- 172.21.15.42:0/857747251 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f07bc0a1e20 0x7f07bc135670 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:55:25.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.395+0000 7f07c0ad8640 1 --2- 172.21.15.42:0/857747251 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f07bc0a1330 0x7f07bc1ceff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:55:25.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.395+0000 7f07c0ad8640 1 -- 172.21.15.42:0/857747251 >> 172.21.15.42:0/857747251 conn(0x7f07bc0a2420 msgr2=0x7f07bc0a45b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:55:25.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.395+0000 7f07c0ad8640 1 -- 172.21.15.42:0/857747251 shutdown_connections 2024-09-16T03:55:25.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.395+0000 7f07c0ad8640 1 -- 172.21.15.42:0/857747251 wait complete. 2024-09-16T03:55:25.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:25.395+0000 7f07c0ad8640 1 librados: shutdown 2024-09-16T03:55:28.417 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:55:28.544 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.539+0000 7fd985b66640 10 monclient: get_monmap_and_config 2024-09-16T03:55:28.544 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.539+0000 7fd985b66640 10 monclient: build_initial_monmap 2024-09-16T03:55:28.544 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.539+0000 7fd985b66640 10 monclient: monmap: 2024-09-16T03:55:28.544 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:55:28.544 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:55:28.544 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:55:28.542308+0000 2024-09-16T03:55:28.544 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:55:28.542308+0000 2024-09-16T03:55:28.544 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:55:28.544 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:55:28.545 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:55:28.545 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:55:28.545 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:55:28.545 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:55:28.545 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:55:28.545 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:55:28.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.539+0000 7fd985b66640 1 Processor -- start 2024-09-16T03:55:28.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.539+0000 7fd985b66640 1 -- start start 2024-09-16T03:55:28.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.539+0000 7fd985b66640 10 monclient: init 2024-09-16T03:55:28.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.539+0000 7fd985b66640 10 monclient: _reopen_session rank -1 2024-09-16T03:55:28.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.539+0000 7fd985b66640 10 monclient: _add_conns ranks=[1,3,4,0,2] 2024-09-16T03:55:28.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.539+0000 7fd985b66640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd9800a39b0 addr v1:172.21.15.154:6789/0 2024-09-16T03:55:28.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.539+0000 7fd985b66640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd9800a3d90 0x7fd98013c380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:55:28.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.539+0000 7fd985b66640 10 monclient(hunting): picked mon.noname-a con 0x7fd9800a3d90 addr v2:172.21.15.42:3300/0 2024-09-16T03:55:28.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.539+0000 7fd985b66640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fd98013c8c0 0x7fd98013ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:55:28.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.539+0000 7fd985b66640 10 monclient(hunting): picked mon.noname-c con 0x7fd98013c8c0 addr v2:172.21.15.154:3300/0 2024-09-16T03:55:28.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.539+0000 7fd985b66640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd980134290 con 0x7fd9800a39b0 2024-09-16T03:55:28.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.539+0000 7fd985b66640 10 monclient(hunting): start opening mon connection 2024-09-16T03:55:28.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.539+0000 7fd985b66640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7fd98009c140 con 0x7fd9800a3d90 2024-09-16T03:55:28.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.539+0000 7fd985b66640 10 monclient(hunting): start opening mon connection 2024-09-16T03:55:28.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.539+0000 7fd985b66640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fd98009c420 con 0x7fd98013c8c0 2024-09-16T03:55:28.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.539+0000 7fd985b66640 10 monclient(hunting): _renew_subs 2024-09-16T03:55:28.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.539+0000 7fd985b66640 10 monclient(hunting): authenticate will time out at 1775.511908s 2024-09-16T03:55:28.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.543+0000 7fd97effd640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd9800a3d90 0x7fd98013c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:55:28.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.543+0000 7fd97effd640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd9800a3d90 0x7fd98013c380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:34750/0 (socket says 172.21.15.42:34750) 2024-09-16T03:55:28.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.543+0000 7fd97effd640 1 -- 172.21.15.42:0/1489222299 learned_addr learned my addr 172.21.15.42:0/1489222299 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:55:28.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.543+0000 7fd97effd640 10 monclient(hunting): get_auth_request con 0x7fd9800a3d90 auth_method 0 2024-09-16T03:55:28.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.543+0000 7fd97effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:55:28.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.543+0000 7fd97effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:55:28.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.543+0000 7fd97effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:55:28.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.543+0000 7fd97effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:55:28.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.543+0000 7fd97effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:55:28.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.543+0000 7fd97effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:55:28.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.543+0000 7fd97ffff640 1 --2- 172.21.15.42:0/1489222299 >> v2:172.21.15.154:3300/0 conn(0x7fd98013c8c0 0x7fd98013ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:55:28.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.543+0000 7fd97ffff640 10 monclient(hunting): get_auth_request con 0x7fd98013c8c0 auth_method 0 2024-09-16T03:55:28.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.543+0000 7fd97ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:55:28.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.543+0000 7fd97ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:55:28.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.543+0000 7fd97ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:55:28.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.543+0000 7fd97effd640 10 monclient(hunting): handle_auth_done global_id 824562 payload 274 2024-09-16T03:55:28.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.543+0000 7fd97effd640 1 -- 172.21.15.42:0/1489222299 >> v2:172.21.15.154:3300/0 conn(0x7fd98013c8c0 msgr2=0x7fd98013ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:55:28.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.543+0000 7fd97effd640 1 --2- 172.21.15.42:0/1489222299 >> v2:172.21.15.154:3300/0 conn(0x7fd98013c8c0 0x7fd98013ecb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:55:28.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.543+0000 7fd97effd640 1 -- 172.21.15.42:0/1489222299 >> v1:172.21.15.154:6789/0 conn(0x7fd9800a39b0 legacy=0x7fd980133b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:55:28.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.543+0000 7fd97effd640 10 monclient: _finish_hunting 0 2024-09-16T03:55:28.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.543+0000 7fd97effd640 1 monclient: found mon.noname-a 2024-09-16T03:55:28.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.543+0000 7fd97effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:55:28.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.543+0000 7fd97effd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:55:28.551 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.543+0000 7fd97effd640 1 -- 172.21.15.42:0/1489222299 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd98006dd60 con 0x7fd9800a3d90 2024-09-16T03:55:28.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.543+0000 7fd97effd640 1 --2- 172.21.15.42:0/1489222299 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd9800a3d90 0x7fd98013c380 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fd98006b540 tx=0x7fd96801c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7fd3ba7a4bcae6c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:55:28.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.547+0000 7fd97e7fc640 1 -- 172.21.15.42:0/1489222299 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9680a5570 con 0x7fd9800a3d90 2024-09-16T03:55:28.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.547+0000 7fd97e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:55:28.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.547+0000 7fd97e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:55:28.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.547+0000 7fd97e7fc640 10 monclient: dump: 2024-09-16T03:55:28.558 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:55:28.558 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:55:28.558 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:28.558 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:28.559 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:55:28.559 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:55:28.559 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:55:28.559 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:55:28.559 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:55:28.559 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:55:28.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.547+0000 7fd97e7fc640 10 monclient: _finish_auth 0 2024-09-16T03:55:28.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.547+0000 7fd97e7fc640 10 monclient: _check_auth_tickets 2024-09-16T03:55:28.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.547+0000 7fd97e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:55:28.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.547+0000 7fd97e7fc640 1 -- 172.21.15.42:0/1489222299 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd9680a5710 con 0x7fd9800a3d90 2024-09-16T03:55:28.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.547+0000 7fd97e7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:55:28.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.547+0000 7fd97e7fc640 1 -- 172.21.15.42:0/1489222299 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9680a5ae0 con 0x7fd9800a3d90 2024-09-16T03:55:28.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.547+0000 7fd97e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:55:28.560 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.547+0000 7fd97e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:55:28.560 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.547+0000 7fd97e7fc640 10 monclient: dump: 2024-09-16T03:55:28.560 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:55:28.560 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:55:28.560 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:28.560 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:28.560 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:55:28.560 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:55:28.560 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:55:28.560 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:55:28.560 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:55:28.560 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:55:28.560 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.547+0000 7fd985b66640 5 monclient: authenticate success, global_id 824562 2024-09-16T03:55:28.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.547+0000 7fd985b66640 10 monclient: get_monmap_and_config success 2024-09-16T03:55:28.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.547+0000 7fd985b66640 4 set_mon_vals no callback set 2024-09-16T03:55:28.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.547+0000 7fd985b66640 10 monclient: shutdown 2024-09-16T03:55:28.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.547+0000 7fd985b66640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:55:28.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.547+0000 7fd985b66640 1 -- 172.21.15.42:0/1489222299 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd9800a3d90 msgr2=0x7fd98013c380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:55:28.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.547+0000 7fd985b66640 1 --2- 172.21.15.42:0/1489222299 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd9800a3d90 0x7fd98013c380 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fd98006b540 tx=0x7fd96801c8b0 comp rx=0 tx=0).stop 2024-09-16T03:55:28.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.547+0000 7fd985b66640 1 -- 172.21.15.42:0/1489222299 shutdown_connections 2024-09-16T03:55:28.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.547+0000 7fd985b66640 1 --2- 172.21.15.42:0/1489222299 >> v2:172.21.15.154:3300/0 conn(0x7fd98013c8c0 0x7fd98013ecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:55:28.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.547+0000 7fd985b66640 1 --2- 172.21.15.42:0/1489222299 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd9800a3d90 0x7fd98013c380 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:55:28.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.547+0000 7fd985b66640 1 -- 172.21.15.42:0/1489222299 >> 172.21.15.42:0/1489222299 conn(0x7fd9800a9fb0 msgr2=0x7fd9800aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:55:28.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.547+0000 7fd985b66640 1 -- 172.21.15.42:0/1489222299 shutdown_connections 2024-09-16T03:55:28.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.547+0000 7fd985b66640 1 -- 172.21.15.42:0/1489222299 wait complete. 2024-09-16T03:55:28.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.547+0000 7fd985b66640 10 monclient: build_initial_monmap 2024-09-16T03:55:28.562 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.547+0000 7fd985b66640 10 monclient: monmap: 2024-09-16T03:55:28.562 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:55:28.562 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:55:28.562 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:55:28.562 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:55:28.562 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:55:28.562 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:55:28.562 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:55:28.562 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:55:28.563 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:55:28.563 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:55:28.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.547+0000 7fd985b66640 1 librados: starting msgr at 2024-09-16T03:55:28.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.547+0000 7fd985b66640 1 librados: starting objecter 2024-09-16T03:55:28.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.547+0000 7fd985b66640 1 Processor -- start 2024-09-16T03:55:28.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.547+0000 7fd985b66640 1 -- start start 2024-09-16T03:55:28.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.547+0000 7fd985b66640 1 librados: setting wanted keys 2024-09-16T03:55:28.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.547+0000 7fd985b66640 1 librados: calling monclient init 2024-09-16T03:55:28.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.547+0000 7fd985b66640 10 monclient: init 2024-09-16T03:55:28.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.547+0000 7fd985b66640 10 monclient: _reopen_session rank -1 2024-09-16T03:55:28.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.547+0000 7fd985b66640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T03:55:28.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.547+0000 7fd985b66640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd9800a39b0 0x7fd980138130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:55:28.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.547+0000 7fd985b66640 10 monclient(hunting): picked mon.noname-a con 0x7fd9800a39b0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:55:28.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.547+0000 7fd985b66640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd9800a3d90 0x7fd980138670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:55:28.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.547+0000 7fd985b66640 10 monclient(hunting): picked mon.noname-b con 0x7fd9800a3d90 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:55:28.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.547+0000 7fd985b66640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd98013c8c0 0x7fd980138bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:55:28.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.547+0000 7fd985b66640 10 monclient(hunting): picked mon.noname-c con 0x7fd98013c8c0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:55:28.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.547+0000 7fd985b66640 10 monclient(hunting): start opening mon connection 2024-09-16T03:55:28.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.547+0000 7fd985b66640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fd980141da0 con 0x7fd9800a39b0 2024-09-16T03:55:28.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.547+0000 7fd985b66640 10 monclient(hunting): start opening mon connection 2024-09-16T03:55:28.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.547+0000 7fd985b66640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd980141c20 con 0x7fd9800a3d90 2024-09-16T03:55:28.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.547+0000 7fd985b66640 10 monclient(hunting): start opening mon connection 2024-09-16T03:55:28.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.547+0000 7fd985b66640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fd980141f20 con 0x7fd98013c8c0 2024-09-16T03:55:28.565 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.547+0000 7fd985b66640 10 monclient(hunting): _renew_subs 2024-09-16T03:55:28.565 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.547+0000 7fd985b66640 10 monclient(hunting): authenticate will time out at 1775.519380s 2024-09-16T03:55:28.565 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.547+0000 7fd97ffff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd98013c8c0 0x7fd980138bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:55:28.565 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.547+0000 7fd97f7fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd9800a39b0 0x7fd980138130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:55:28.565 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.547+0000 7fd97ffff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd98013c8c0 0x7fd980138bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:51664/0 (socket says 172.21.15.42:51664) 2024-09-16T03:55:28.565 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.547+0000 7fd97f7fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd9800a39b0 0x7fd980138130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:34758/0 (socket says 172.21.15.42:34758) 2024-09-16T03:55:28.565 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.547+0000 7fd97ffff640 1 -- 172.21.15.42:0/3745779644 learned_addr learned my addr 172.21.15.42:0/3745779644 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:55:28.565 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.547+0000 7fd97ffff640 10 monclient(hunting): get_auth_request con 0x7fd98013c8c0 auth_method 0 2024-09-16T03:55:28.565 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.547+0000 7fd97ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:55:28.565 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.547+0000 7fd97ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:55:28.565 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.547+0000 7fd97ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:55:28.565 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.547+0000 7fd97f7fe640 10 monclient(hunting): get_auth_request con 0x7fd9800a39b0 auth_method 0 2024-09-16T03:55:28.565 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.547+0000 7fd97f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:55:28.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.547+0000 7fd97f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:55:28.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.547+0000 7fd97f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:55:28.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.547+0000 7fd97effd640 1 --2- 172.21.15.42:0/3745779644 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd9800a3d90 0x7fd980138670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:55:28.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.547+0000 7fd97f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:55:28.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.547+0000 7fd97f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:55:28.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.547+0000 7fd97f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:55:28.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.547+0000 7fd97ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:55:28.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.547+0000 7fd97ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:55:28.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.547+0000 7fd97ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:55:28.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.547+0000 7fd97effd640 10 monclient(hunting): get_auth_request con 0x7fd9800a3d90 auth_method 0 2024-09-16T03:55:28.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.547+0000 7fd97effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:55:28.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.547+0000 7fd97effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:55:28.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.547+0000 7fd97effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:55:28.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.551+0000 7fd97ffff640 10 monclient(hunting): handle_auth_done global_id 854189 payload 824 2024-09-16T03:55:28.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.551+0000 7fd97ffff640 1 -- 172.21.15.42:0/3745779644 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd9800a3d90 msgr2=0x7fd980138670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:55:28.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.551+0000 7fd97ffff640 1 --2- 172.21.15.42:0/3745779644 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd9800a3d90 0x7fd980138670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:55:28.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.551+0000 7fd97ffff640 1 -- 172.21.15.42:0/3745779644 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd9800a39b0 msgr2=0x7fd980138130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:55:28.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.551+0000 7fd97ffff640 1 --2- 172.21.15.42:0/3745779644 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd9800a39b0 0x7fd980138130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:55:28.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.551+0000 7fd97ffff640 10 monclient: _finish_hunting 0 2024-09-16T03:55:28.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.551+0000 7fd97ffff640 1 monclient: found mon.noname-c 2024-09-16T03:55:28.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.551+0000 7fd97ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:55:28.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.551+0000 7fd97ffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:55:28.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.551+0000 7fd97ffff640 1 -- 172.21.15.42:0/3745779644 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd980139ce0 con 0x7fd98013c8c0 2024-09-16T03:55:28.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.551+0000 7fd97effd640 1 --2- 172.21.15.42:0/3745779644 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd9800a3d90 0x7fd980138670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T03:55:28.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.551+0000 7fd97f7fe640 1 --2- 172.21.15.42:0/3745779644 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd9800a39b0 0x7fd980138130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:55:28.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.551+0000 7fd97ffff640 1 --2- 172.21.15.42:0/3745779644 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd98013c8c0 0x7fd980138bb0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fd97800d800 tx=0x7fd97800dcc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:55:28.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.551+0000 7fd977fff640 10 client.?.objecter ms_handle_connect 0x7fd98013c8c0 2024-09-16T03:55:28.568 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.551+0000 7fd977fff640 10 client.?.objecter resend_mon_ops 2024-09-16T03:55:28.568 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.551+0000 7fd977fff640 1 -- 172.21.15.42:0/3745779644 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd97800b6c0 con 0x7fd98013c8c0 2024-09-16T03:55:28.568 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.551+0000 7fd977fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:55:28.568 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.551+0000 7fd977fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:55:28.568 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.551+0000 7fd977fff640 10 monclient: dump: 2024-09-16T03:55:28.568 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:55:28.568 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:55:28.568 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:28.568 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:28.568 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:55:28.568 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:55:28.568 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:55:28.568 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:55:28.568 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:55:28.569 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:55:28.569 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.551+0000 7fd977fff640 10 monclient: _finish_auth 0 2024-09-16T03:55:28.569 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.551+0000 7fd977fff640 10 monclient: _check_auth_tickets 2024-09-16T03:55:28.569 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.551+0000 7fd977fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:55:28.569 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.551+0000 7fd977fff640 1 -- 172.21.15.42:0/3745779644 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd978019070 con 0x7fd98013c8c0 2024-09-16T03:55:28.569 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.551+0000 7fd985b66640 5 monclient: authenticate success, global_id 854189 2024-09-16T03:55:28.569 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.551+0000 7fd977fff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:55:28.569 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.551+0000 7fd985b66640 10 monclient: _renew_subs 2024-09-16T03:55:28.569 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.551+0000 7fd977fff640 1 -- 172.21.15.42:0/3745779644 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd97800baa0 con 0x7fd98013c8c0 2024-09-16T03:55:28.569 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.551+0000 7fd985b66640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:55:28.569 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.551+0000 7fd985b66640 1 -- 172.21.15.42:0/3745779644 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd98013ad40 con 0x7fd98013c8c0 2024-09-16T03:55:28.569 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.551+0000 7fd977fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:55:28.569 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.551+0000 7fd977fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:55:28.569 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.551+0000 7fd977fff640 10 monclient: dump: 2024-09-16T03:55:28.570 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:55:28.570 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:55:28.570 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:28.570 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:28.570 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:55:28.570 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:55:28.570 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:55:28.570 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:55:28.570 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:55:28.570 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:55:28.570 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.551+0000 7fd97d7fa640 4 set_mon_vals no callback set 2024-09-16T03:55:28.570 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.551+0000 7fd985b66640 10 client.854189.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:55:28.570 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.551+0000 7fd985b66640 10 monclient: _renew_subs 2024-09-16T03:55:28.570 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.551+0000 7fd985b66640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:55:28.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.551+0000 7fd985b66640 1 -- 172.21.15.42:0/3745779644 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd98013aa60 con 0x7fd98013c8c0 2024-09-16T03:55:28.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.551+0000 7fd985b66640 1 librados: init done 2024-09-16T03:55:28.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.551+0000 7fd977fff640 1 -- 172.21.15.42:0/3745779644 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fd97800bc40 con 0x7fd98013c8c0 2024-09-16T03:55:28.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.551+0000 7fd985b66640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:55:28.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.551+0000 7fd985b66640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:55:28.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.551+0000 7fd985b66640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:55:28.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.551+0000 7fd985b66640 1 -- 172.21.15.42:0/3745779644 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd980133b80 con 0x7fd98013c8c0 2024-09-16T03:55:28.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.551+0000 7fd977fff640 1 --2- 172.21.15.42:0/3745779644 >> v2:172.21.15.154:6812/254845342 conn(0x7fd960037810 0x7fd960039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:55:28.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.551+0000 7fd977fff640 1 -- 172.21.15.42:0/3745779644 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(415..415 src has 330..415) ==== 5328+0+0 (secure 0 0 0) 0x7fd978049170 con 0x7fd98013c8c0 2024-09-16T03:55:28.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.551+0000 7fd977fff640 10 client.854189.objecter ms_dispatch 0x7fd980000ba0 osd_map(415..415 src has 330..415) 2024-09-16T03:55:28.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.551+0000 7fd977fff640 3 client.854189.objecter handle_osd_map got epochs [415,415] > 0 2024-09-16T03:55:28.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.551+0000 7fd977fff640 3 client.854189.objecter handle_osd_map decoding full epoch 415 2024-09-16T03:55:28.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.551+0000 7fd977fff640 20 client.854189.objecter dump_active .. 0 homeless 2024-09-16T03:55:28.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.551+0000 7fd97f7fe640 1 --2- 172.21.15.42:0/3745779644 >> v2:172.21.15.154:6812/254845342 conn(0x7fd960037810 0x7fd960039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:55:28.572 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.551+0000 7fd97f7fe640 10 monclient: get_auth_request con 0x7fd960037810 auth_method 0 2024-09-16T03:55:28.572 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.551+0000 7fd97f7fe640 1 --2- 172.21.15.42:0/3745779644 >> v2:172.21.15.154:6812/254845342 conn(0x7fd960037810 0x7fd960039cd0 secure :-1 s=READY pgs=3632 cs=0 l=1 rev1=1 crypto rx=0x7fd970001c00 tx=0x7fd970009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:55:28.572 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.551+0000 7fd977fff640 10 client.854189.objecter ms_handle_connect 0x7fd960037810 2024-09-16T03:55:28.573 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.555+0000 7fd977fff640 1 -- 172.21.15.42:0/3745779644 <== mon.2 v2:172.21.15.42:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7fd97801edb0 con 0x7fd98013c8c0 2024-09-16T03:55:28.573 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.555+0000 7fd977fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:55:28.573 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.555+0000 7fd977fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:55:28.786 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.783+0000 7fd985b66640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:55:28.786 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.783+0000 7fd985b66640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:55:28.786 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.783+0000 7fd985b66640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:55:28.786 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.783+0000 7fd985b66640 1 -- 172.21.15.42:0/3745779644 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fd9800a5750 con 0x7fd98013c8c0 2024-09-16T03:55:28.786 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.783+0000 7fd977fff640 1 -- 172.21.15.42:0/3745779644 <== mon.2 v2:172.21.15.42:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7fd97801e890 con 0x7fd98013c8c0 2024-09-16T03:55:28.786 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.783+0000 7fd977fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:55:28.786 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.783+0000 7fd977fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:55:28.787 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":116,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:55:28.789 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.783+0000 7fd985b66640 10 librados: watch_flush enter 2024-09-16T03:55:28.789 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.783+0000 7fd985b66640 10 librados: watch_flush exit 2024-09-16T03:55:28.789 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.783+0000 7fd985b66640 20 client.854189.objecter shutdown clearing up homeless session... 2024-09-16T03:55:28.789 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.783+0000 7fd985b66640 10 client.854189.objecter successfully canceled tick 2024-09-16T03:55:28.789 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.783+0000 7fd985b66640 1 -- 172.21.15.42:0/3745779644 >> v2:172.21.15.154:6812/254845342 conn(0x7fd960037810 msgr2=0x7fd960039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:55:28.789 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.783+0000 7fd985b66640 1 --2- 172.21.15.42:0/3745779644 >> v2:172.21.15.154:6812/254845342 conn(0x7fd960037810 0x7fd960039cd0 secure :-1 s=READY pgs=3632 cs=0 l=1 rev1=1 crypto rx=0x7fd970001c00 tx=0x7fd970009040 comp rx=0 tx=0).stop 2024-09-16T03:55:28.789 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.783+0000 7fd985b66640 10 monclient: shutdown 2024-09-16T03:55:28.789 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.783+0000 7fd985b66640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:55:28.789 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.783+0000 7fd985b66640 1 -- 172.21.15.42:0/3745779644 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd98013c8c0 msgr2=0x7fd980138bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:55:28.789 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.783+0000 7fd985b66640 1 --2- 172.21.15.42:0/3745779644 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd98013c8c0 0x7fd980138bb0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fd97800d800 tx=0x7fd97800dcc0 comp rx=0 tx=0).stop 2024-09-16T03:55:28.789 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.783+0000 7fd985b66640 1 -- 172.21.15.42:0/3745779644 shutdown_connections 2024-09-16T03:55:28.790 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.783+0000 7fd985b66640 1 --2- 172.21.15.42:0/3745779644 >> v2:172.21.15.154:6812/254845342 conn(0x7fd960037810 0x7fd960039cd0 unknown :-1 s=CLOSED pgs=3632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:55:28.790 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.783+0000 7fd985b66640 1 --2- 172.21.15.42:0/3745779644 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd98013c8c0 0x7fd980138bb0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:55:28.790 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.783+0000 7fd985b66640 1 --2- 172.21.15.42:0/3745779644 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd9800a3d90 0x7fd980138670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:55:28.790 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.783+0000 7fd985b66640 1 --2- 172.21.15.42:0/3745779644 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd9800a39b0 0x7fd980138130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:55:28.790 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.783+0000 7fd985b66640 1 -- 172.21.15.42:0/3745779644 >> 172.21.15.42:0/3745779644 conn(0x7fd9800a9fb0 msgr2=0x7fd980131980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:55:28.790 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.783+0000 7fd985b66640 1 -- 172.21.15.42:0/3745779644 shutdown_connections 2024-09-16T03:55:28.790 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.783+0000 7fd985b66640 1 -- 172.21.15.42:0/3745779644 wait complete. 2024-09-16T03:55:28.790 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.783+0000 7fd985b66640 1 librados: shutdown 2024-09-16T03:55:28.806 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T03:55:28.806 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T03:55:28.806 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T03:55:28.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.923+0000 7f59e628e640 10 monclient: get_monmap_and_config 2024-09-16T03:55:28.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.923+0000 7f59e628e640 10 monclient: build_initial_monmap 2024-09-16T03:55:28.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.923+0000 7f59e628e640 10 monclient: monmap: 2024-09-16T03:55:28.926 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:55:28.926 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:55:28.927 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:55:28.924638+0000 2024-09-16T03:55:28.927 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:55:28.924638+0000 2024-09-16T03:55:28.927 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:55:28.927 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:55:28.927 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:55:28.927 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:55:28.928 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:55:28.928 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:55:28.928 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:55:28.928 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:55:28.946 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.923+0000 7f59e628e640 1 Processor -- start 2024-09-16T03:55:28.946 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.923+0000 7f59e628e640 1 -- start start 2024-09-16T03:55:28.947 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.923+0000 7f59e628e640 10 monclient: init 2024-09-16T03:55:28.947 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.923+0000 7f59e628e640 10 monclient: _reopen_session rank -1 2024-09-16T03:55:28.947 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.923+0000 7f59e628e640 10 monclient: _add_conns ranks=[1,2,4,3,0] 2024-09-16T03:55:28.947 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.923+0000 7f59e628e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f59e00a39b0 addr v1:172.21.15.154:6789/0 2024-09-16T03:55:28.947 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.923+0000 7f59e628e640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f59e00a3d90 0x7f59e013c380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:55:28.947 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.923+0000 7f59e628e640 10 monclient(hunting): picked mon.noname-b con 0x7f59e00a3d90 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:55:28.947 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.923+0000 7f59e628e640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f59e013c8c0 0x7f59e013ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:55:28.948 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.923+0000 7f59e628e640 10 monclient(hunting): picked mon.noname-c con 0x7f59e013c8c0 addr v2:172.21.15.154:3300/0 2024-09-16T03:55:28.948 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.923+0000 7f59e628e640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f59e0134290 con 0x7f59e00a39b0 2024-09-16T03:55:28.948 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.923+0000 7f59e628e640 10 monclient(hunting): start opening mon connection 2024-09-16T03:55:28.948 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.923+0000 7f59e628e640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f59e009c140 con 0x7f59e013c8c0 2024-09-16T03:55:28.948 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.923+0000 7f59e628e640 10 monclient(hunting): start opening mon connection 2024-09-16T03:55:28.948 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.923+0000 7f59e628e640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f59e009c420 con 0x7f59e00a3d90 2024-09-16T03:55:28.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.923+0000 7f59e628e640 10 monclient(hunting): _renew_subs 2024-09-16T03:55:28.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.923+0000 7f59e628e640 10 monclient(hunting): authenticate will time out at 1775.894279s 2024-09-16T03:55:28.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.923+0000 7f59df7fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f59e00a3d90 0x7f59e013c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:55:28.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.923+0000 7f59df7fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f59e00a3d90 0x7f59e013c380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:51672/0 (socket says 172.21.15.42:51672) 2024-09-16T03:55:28.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.923+0000 7f59df7fe640 1 -- 172.21.15.42:0/2109052324 learned_addr learned my addr 172.21.15.42:0/2109052324 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:55:28.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.923+0000 7f59df7fe640 10 monclient(hunting): get_auth_request con 0x7f59e00a3d90 auth_method 0 2024-09-16T03:55:28.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.923+0000 7f59df7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:55:28.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.923+0000 7f59df7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:55:28.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.923+0000 7f59df7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:55:28.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.923+0000 7f59df7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:55:28.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.923+0000 7f59df7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:55:28.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.923+0000 7f59df7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:55:28.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.923+0000 7f59e482b640 1 --2- 172.21.15.42:0/2109052324 >> v2:172.21.15.154:3300/0 conn(0x7f59e013c8c0 0x7f59e013ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:55:28.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.923+0000 7f59df7fe640 10 monclient(hunting): handle_auth_done global_id 854192 payload 274 2024-09-16T03:55:28.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.923+0000 7f59df7fe640 1 -- 172.21.15.42:0/2109052324 >> v2:172.21.15.154:3300/0 conn(0x7f59e013c8c0 msgr2=0x7f59e013ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:55:28.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.923+0000 7f59df7fe640 1 --2- 172.21.15.42:0/2109052324 >> v2:172.21.15.154:3300/0 conn(0x7f59e013c8c0 0x7f59e013ecb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:55:28.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.923+0000 7f59df7fe640 1 -- 172.21.15.42:0/2109052324 >> v1:172.21.15.154:6789/0 conn(0x7f59e00a39b0 legacy=0x7f59e0133b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:55:28.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.923+0000 7f59df7fe640 10 monclient: _finish_hunting 0 2024-09-16T03:55:28.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.923+0000 7f59df7fe640 1 monclient: found mon.noname-b 2024-09-16T03:55:28.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.923+0000 7f59df7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:55:28.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.923+0000 7f59df7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T03:55:28.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.923+0000 7f59df7fe640 1 -- 172.21.15.42:0/2109052324 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59e006dd60 con 0x7f59e00a3d90 2024-09-16T03:55:28.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.923+0000 7f59e482b640 10 monclient: get_auth_request con 0x7f59e013c8c0 auth_method 0 2024-09-16T03:55:28.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.923+0000 7f59e482b640 1 --2- 172.21.15.42:0/2109052324 >> v2:172.21.15.154:3300/0 conn(0x7f59e013c8c0 0x7f59e013ecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T03:55:28.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.927+0000 7f59df7fe640 1 --2- 172.21.15.42:0/2109052324 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f59e00a3d90 0x7f59e013c380 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f59c8009870 tx=0x7f59c801c3e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=657627864a62af0d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:55:28.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.927+0000 7f59deffd640 1 -- 172.21.15.42:0/2109052324 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59c80b0070 con 0x7f59e00a3d90 2024-09-16T03:55:28.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.927+0000 7f59deffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:55:28.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.927+0000 7f59deffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:55:28.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.927+0000 7f59deffd640 10 monclient: dump: 2024-09-16T03:55:28.951 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:55:28.951 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:55:28.951 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:28.951 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:28.951 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:55:28.951 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:55:28.951 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:55:28.952 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:55:28.952 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:55:28.952 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:55:28.952 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.927+0000 7f59deffd640 10 monclient: _finish_auth 0 2024-09-16T03:55:28.952 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.927+0000 7f59deffd640 10 monclient: _check_auth_tickets 2024-09-16T03:55:28.952 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.927+0000 7f59deffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:55:28.952 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.927+0000 7f59deffd640 1 -- 172.21.15.42:0/2109052324 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f59c80ab600 con 0x7f59e00a3d90 2024-09-16T03:55:28.952 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.927+0000 7f59e628e640 5 monclient: authenticate success, global_id 854192 2024-09-16T03:55:28.952 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.927+0000 7f59e628e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:55:28.952 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.927+0000 7f59deffd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:55:28.952 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.927+0000 7f59e628e640 10 monclient: get_monmap_and_config success 2024-09-16T03:55:28.952 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.927+0000 7f59deffd640 1 -- 172.21.15.42:0/2109052324 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59c80ab9d0 con 0x7f59e00a3d90 2024-09-16T03:55:28.952 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.927+0000 7f59e628e640 4 set_mon_vals no callback set 2024-09-16T03:55:28.953 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.927+0000 7f59deffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:55:28.953 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.927+0000 7f59deffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:55:28.953 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.927+0000 7f59deffd640 10 monclient: dump: 2024-09-16T03:55:28.953 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:55:28.953 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:55:28.953 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:28.953 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:28.953 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:55:28.953 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:55:28.953 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:55:28.954 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:55:28.954 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:55:28.954 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:55:28.954 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.927+0000 7f59e628e640 10 monclient: shutdown 2024-09-16T03:55:28.954 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.927+0000 7f59e628e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:55:28.954 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.927+0000 7f59e628e640 1 -- 172.21.15.42:0/2109052324 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f59e00a3d90 msgr2=0x7f59e013c380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:55:28.954 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.927+0000 7f59e628e640 1 --2- 172.21.15.42:0/2109052324 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f59e00a3d90 0x7f59e013c380 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f59c8009870 tx=0x7f59c801c3e0 comp rx=0 tx=0).stop 2024-09-16T03:55:28.954 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.927+0000 7f59e628e640 1 -- 172.21.15.42:0/2109052324 shutdown_connections 2024-09-16T03:55:28.954 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.927+0000 7f59e628e640 1 --2- 172.21.15.42:0/2109052324 >> v2:172.21.15.154:3300/0 conn(0x7f59e013c8c0 0x7f59e013ecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:55:28.954 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.927+0000 7f59e628e640 1 --2- 172.21.15.42:0/2109052324 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f59e00a3d90 0x7f59e013c380 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:55:28.954 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.927+0000 7f59e628e640 1 -- 172.21.15.42:0/2109052324 >> 172.21.15.42:0/2109052324 conn(0x7f59e00a9fb0 msgr2=0x7f59e00aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:55:28.954 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.927+0000 7f59e628e640 1 -- 172.21.15.42:0/2109052324 shutdown_connections 2024-09-16T03:55:28.954 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.927+0000 7f59e628e640 1 -- 172.21.15.42:0/2109052324 wait complete. 2024-09-16T03:55:28.954 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.927+0000 7f59e628e640 10 monclient: build_initial_monmap 2024-09-16T03:55:28.955 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.927+0000 7f59e628e640 10 monclient: monmap: 2024-09-16T03:55:28.955 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:55:28.955 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:55:28.955 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:55:28.955 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:55:28.955 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:55:28.955 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:55:28.955 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:55:28.955 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:55:28.955 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:55:28.955 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:55:28.955 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.927+0000 7f59e628e640 1 librados: starting msgr at 2024-09-16T03:55:28.955 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.927+0000 7f59e628e640 1 librados: starting objecter 2024-09-16T03:55:28.955 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.927+0000 7f59e628e640 1 Processor -- start 2024-09-16T03:55:28.956 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.927+0000 7f59e628e640 1 -- start start 2024-09-16T03:55:28.956 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.927+0000 7f59e628e640 1 librados: setting wanted keys 2024-09-16T03:55:28.956 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.927+0000 7f59e628e640 1 librados: calling monclient init 2024-09-16T03:55:28.956 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.927+0000 7f59e628e640 10 monclient: init 2024-09-16T03:55:28.956 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.927+0000 7f59e628e640 10 monclient: _reopen_session rank -1 2024-09-16T03:55:28.956 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.927+0000 7f59e628e640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:55:28.956 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.927+0000 7f59e628e640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f59e00a39b0 0x7f59e013a410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:55:28.956 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.927+0000 7f59e628e640 10 monclient(hunting): picked mon.noname-c con 0x7f59e00a39b0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:55:28.956 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.927+0000 7f59e628e640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f59e013c8c0 0x7f59e013a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:55:28.956 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.927+0000 7f59e628e640 10 monclient(hunting): picked mon.noname-a con 0x7f59e013c8c0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:55:28.956 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.927+0000 7f59e628e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f59e009d360 0x7f59e013ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:55:28.956 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.927+0000 7f59e628e640 10 monclient(hunting): picked mon.noname-b con 0x7f59e009d360 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:55:28.956 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.927+0000 7f59e628e640 10 monclient(hunting): start opening mon connection 2024-09-16T03:55:28.957 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.927+0000 7f59e628e640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f59e0141da0 con 0x7f59e013c8c0 2024-09-16T03:55:28.957 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.927+0000 7f59e628e640 10 monclient(hunting): start opening mon connection 2024-09-16T03:55:28.957 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.927+0000 7f59e628e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f59e0141c20 con 0x7f59e009d360 2024-09-16T03:55:28.957 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.927+0000 7f59e628e640 10 monclient(hunting): start opening mon connection 2024-09-16T03:55:28.957 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.927+0000 7f59e628e640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f59e0141f20 con 0x7f59e00a39b0 2024-09-16T03:55:28.957 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.927+0000 7f59e628e640 10 monclient(hunting): _renew_subs 2024-09-16T03:55:28.957 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.927+0000 7f59e628e640 10 monclient(hunting): authenticate will time out at 1775.899270s 2024-09-16T03:55:28.957 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.927+0000 7f59dffff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f59e00a39b0 0x7f59e013a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:55:28.957 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.927+0000 7f59df7fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f59e013c8c0 0x7f59e013a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:55:28.958 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.927+0000 7f59dffff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f59e00a39b0 0x7f59e013a410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:51674/0 (socket says 172.21.15.42:51674) 2024-09-16T03:55:28.958 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.927+0000 7f59dffff640 1 -- 172.21.15.42:0/3472252414 learned_addr learned my addr 172.21.15.42:0/3472252414 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:55:28.958 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.927+0000 7f59dffff640 10 monclient(hunting): get_auth_request con 0x7f59e00a39b0 auth_method 0 2024-09-16T03:55:28.958 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.927+0000 7f59dffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:55:28.958 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.927+0000 7f59dffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:55:28.958 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.927+0000 7f59dffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:55:28.958 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.927+0000 7f59df7fe640 10 monclient(hunting): get_auth_request con 0x7f59e013c8c0 auth_method 0 2024-09-16T03:55:28.958 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.927+0000 7f59df7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:55:28.958 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.927+0000 7f59df7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:55:28.958 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.927+0000 7f59df7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:55:28.958 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.927+0000 7f59df7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:55:28.959 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.927+0000 7f59df7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:55:28.959 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.927+0000 7f59df7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:55:28.959 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.927+0000 7f59dffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:55:28.959 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.927+0000 7f59dffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:55:28.959 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.927+0000 7f59dffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:55:28.959 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.927+0000 7f59df7fe640 10 monclient(hunting): handle_auth_done global_id 824568 payload 824 2024-09-16T03:55:28.960 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.927+0000 7f59df7fe640 1 -- 172.21.15.42:0/3472252414 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f59e00a39b0 msgr2=0x7f59e013a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:55:28.960 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.927+0000 7f59e482b640 1 --2- 172.21.15.42:0/3472252414 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f59e009d360 0x7f59e013ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:55:28.960 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.927+0000 7f59df7fe640 1 --2- 172.21.15.42:0/3472252414 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f59e00a39b0 0x7f59e013a410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:55:28.960 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.927+0000 7f59df7fe640 1 -- 172.21.15.42:0/3472252414 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f59e009d360 msgr2=0x7f59e013ae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:55:28.960 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.927+0000 7f59df7fe640 1 --2- 172.21.15.42:0/3472252414 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f59e009d360 0x7f59e013ae90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:55:28.960 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.927+0000 7f59df7fe640 10 monclient: _finish_hunting 0 2024-09-16T03:55:28.960 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.927+0000 7f59df7fe640 1 monclient: found mon.noname-a 2024-09-16T03:55:28.960 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.927+0000 7f59df7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:55:28.961 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.927+0000 7f59df7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:55:28.961 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.927+0000 7f59df7fe640 1 -- 172.21.15.42:0/3472252414 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59e013efc0 con 0x7f59e013c8c0 2024-09-16T03:55:28.961 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.927+0000 7f59df7fe640 1 --2- 172.21.15.42:0/3472252414 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f59e013c8c0 0x7f59e013a950 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f59c80ab000 tx=0x7f59c8019180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:55:28.961 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.931+0000 7f59dcff9640 10 client.?.objecter ms_handle_connect 0x7f59e013c8c0 2024-09-16T03:55:28.961 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.931+0000 7f59dcff9640 10 client.?.objecter resend_mon_ops 2024-09-16T03:55:28.961 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.931+0000 7f59dcff9640 1 -- 172.21.15.42:0/3472252414 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59c80bb070 con 0x7f59e013c8c0 2024-09-16T03:55:28.961 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.931+0000 7f59dcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:55:28.961 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.931+0000 7f59dcff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:55:28.961 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.931+0000 7f59dcff9640 10 monclient: dump: 2024-09-16T03:55:28.962 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:55:28.962 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:55:28.962 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:28.962 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:28.962 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:55:28.962 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:55:28.962 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:55:28.962 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:55:28.962 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:55:28.962 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:55:28.962 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.931+0000 7f59dcff9640 10 monclient: _finish_auth 0 2024-09-16T03:55:28.962 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.931+0000 7f59dcff9640 10 monclient: _check_auth_tickets 2024-09-16T03:55:28.962 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.931+0000 7f59dcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:55:28.962 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.931+0000 7f59e628e640 5 monclient: authenticate success, global_id 824568 2024-09-16T03:55:28.963 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.931+0000 7f59dcff9640 1 -- 172.21.15.42:0/3472252414 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f59c80abe30 con 0x7f59e013c8c0 2024-09-16T03:55:28.963 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.931+0000 7f59dcff9640 10 monclient: handle_config config(0 keys) 2024-09-16T03:55:28.963 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.931+0000 7f59e628e640 10 monclient: _renew_subs 2024-09-16T03:55:28.963 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.931+0000 7f59dcff9640 1 -- 172.21.15.42:0/3472252414 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59c80b0040 con 0x7f59e013c8c0 2024-09-16T03:55:28.963 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.931+0000 7f59e628e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:55:28.963 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.931+0000 7f59e628e640 1 -- 172.21.15.42:0/3472252414 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59e009d0e0 con 0x7f59e013c8c0 2024-09-16T03:55:28.963 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.931+0000 7f59ddffb640 4 set_mon_vals no callback set 2024-09-16T03:55:28.964 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.931+0000 7f59dcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:55:28.964 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.931+0000 7f59dcff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:55:28.964 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.931+0000 7f59dcff9640 10 monclient: dump: 2024-09-16T03:55:28.964 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:55:28.964 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:55:28.964 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:28.964 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:28.964 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:55:28.964 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:55:28.964 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:55:28.965 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:55:28.965 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:55:28.965 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:55:28.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.931+0000 7f59e628e640 10 client.824568.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:55:28.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.931+0000 7f59e628e640 10 monclient: _renew_subs 2024-09-16T03:55:28.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.931+0000 7f59e628e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:55:28.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.931+0000 7f59e628e640 1 -- 172.21.15.42:0/3472252414 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59e01cf4f0 con 0x7f59e013c8c0 2024-09-16T03:55:28.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.931+0000 7f59e628e640 1 librados: init done 2024-09-16T03:55:28.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.931+0000 7f59e628e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:55:28.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.931+0000 7f59e628e640 1 --2- 172.21.15.42:0/3472252414 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f59e01cf7e0 0x7f59e01cfbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:55:28.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.931+0000 7f59e628e640 10 monclient(hunting): start opening mon connection 2024-09-16T03:55:28.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.931+0000 7f59e628e640 1 -- 172.21.15.42:0/3472252414 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f59e009e5c0 con 0x7f59e01cf7e0 2024-09-16T03:55:28.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.931+0000 7f59dffff640 1 --2- 172.21.15.42:0/3472252414 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f59e01cf7e0 0x7f59e01cfbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:55:28.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.931+0000 7f59dffff640 10 monclient: get_auth_request con 0x7f59e01cf7e0 auth_method 0 2024-09-16T03:55:28.967 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.931+0000 7f59dffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:55:28.968 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.931+0000 7f59dffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:55:28.968 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.931+0000 7f59dffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:55:28.969 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.931+0000 7f59dffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:55:28.969 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.931+0000 7f59dffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:55:28.969 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.931+0000 7f59dffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:55:28.969 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.931+0000 7f59dcff9640 1 -- 172.21.15.42:0/3472252414 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f59c80be070 con 0x7f59e013c8c0 2024-09-16T03:55:28.969 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.931+0000 7f59dffff640 10 monclient(hunting): handle_auth_done global_id 824571 payload 824 2024-09-16T03:55:28.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.931+0000 7f59dffff640 1 -- 172.21.15.42:0/3472252414 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f59e00a7cb0 con 0x7f59e01cf7e0 2024-09-16T03:55:28.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.931+0000 7f59dcff9640 1 --2- 172.21.15.42:0/3472252414 >> v2:172.21.15.154:6812/254845342 conn(0x7f59c0037810 0x7f59c0039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:55:28.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.931+0000 7f59dcff9640 1 -- 172.21.15.42:0/3472252414 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(415..415 src has 330..415) ==== 5328+0+0 (secure 0 0 0) 0x7f59c80e2780 con 0x7f59e013c8c0 2024-09-16T03:55:28.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.931+0000 7f59dcff9640 10 client.824568.objecter ms_dispatch 0x7f59e0000ba0 osd_map(415..415 src has 330..415) 2024-09-16T03:55:28.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.931+0000 7f59dcff9640 3 client.824568.objecter handle_osd_map got epochs [415,415] > 0 2024-09-16T03:55:28.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.931+0000 7f59dcff9640 3 client.824568.objecter handle_osd_map decoding full epoch 415 2024-09-16T03:55:28.971 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.931+0000 7f59dcff9640 20 client.824568.objecter dump_active .. 0 homeless 2024-09-16T03:55:28.971 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.931+0000 7f59dffff640 1 --2- 172.21.15.42:0/3472252414 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f59e01cf7e0 0x7f59e01cfbc0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f59d000fb10 tx=0x7f59d00058e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:55:28.971 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.931+0000 7f59dcff9640 10 client.824568.objecter ms_handle_connect 0x7f59e01cf7e0 2024-09-16T03:55:28.971 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.931+0000 7f59dcff9640 10 client.824568.objecter resend_mon_ops 2024-09-16T03:55:28.971 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.931+0000 7f59dcff9640 1 -- 172.21.15.42:0/3472252414 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59d0018030 con 0x7f59e01cf7e0 2024-09-16T03:55:28.971 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.931+0000 7f59dcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:55:28.971 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.931+0000 7f59dcff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:55:28.971 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.931+0000 7f59dcff9640 10 monclient: dump: 2024-09-16T03:55:28.972 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:55:28.972 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:55:28.972 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:28.972 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:28.972 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:55:28.973 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:55:28.973 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:55:28.973 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:55:28.973 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:55:28.973 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:55:28.973 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.931+0000 7f59e482b640 1 --2- 172.21.15.42:0/3472252414 >> v2:172.21.15.154:6812/254845342 conn(0x7f59c0037810 0x7f59c0039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:55:28.974 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.931+0000 7f59e482b640 10 monclient: get_auth_request con 0x7f59c0037810 auth_method 0 2024-09-16T03:55:28.974 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.931+0000 7f59dcff9640 1 -- 172.21.15.42:0/3472252414 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f59d0006b60 con 0x7f59e01cf7e0 2024-09-16T03:55:28.974 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.931+0000 7f59dcff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:55:28.974 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.931+0000 7f59dcff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:55:28.974 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.931+0000 7f59dcff9640 1 -- 172.21.15.42:0/3472252414 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f59e01cf7e0 msgr2=0x7f59e01cfbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:55:28.974 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.931+0000 7f59dcff9640 1 --2- 172.21.15.42:0/3472252414 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f59e01cf7e0 0x7f59e01cfbc0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f59d000fb10 tx=0x7f59d00058e0 comp rx=0 tx=0).stop 2024-09-16T03:55:28.975 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.931+0000 7f59dcff9640 1 -- 172.21.15.42:0/3472252414 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f59e01cf7e0 msgr2=0x7f59e01cfbc0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:55:28.975 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.931+0000 7f59dcff9640 1 --2- 172.21.15.42:0/3472252414 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f59e01cf7e0 0x7f59e01cfbc0 secure :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f59d000fb10 tx=0x7f59d00058e0 comp rx=0 tx=0).stop 2024-09-16T03:55:28.975 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.931+0000 7f59e482b640 1 --2- 172.21.15.42:0/3472252414 >> v2:172.21.15.154:6812/254845342 conn(0x7f59c0037810 0x7f59c0039cd0 secure :-1 s=READY pgs=3633 cs=0 l=1 rev1=1 crypto rx=0x7f59d8006fd0 tx=0x7f59d8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:55:28.975 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.931+0000 7f59dcff9640 10 client.824568.objecter ms_handle_connect 0x7f59c0037810 2024-09-16T03:55:28.975 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.939+0000 7f59e628e640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:55:28.976 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.939+0000 7f59e628e640 1 --2- 172.21.15.42:0/3472252414 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f59e0135bc0 0x7f59e0135fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:55:28.976 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.939+0000 7f59e628e640 10 monclient(hunting): start opening mon connection 2024-09-16T03:55:28.976 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.939+0000 7f59e628e640 1 -- 172.21.15.42:0/3472252414 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f59e006f680 con 0x7f59e0135bc0 2024-09-16T03:55:28.976 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.939+0000 7f59dffff640 1 --2- 172.21.15.42:0/3472252414 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f59e0135bc0 0x7f59e0135fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:55:28.976 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.939+0000 7f59dffff640 10 monclient: get_auth_request con 0x7f59e0135bc0 auth_method 0 2024-09-16T03:55:28.976 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.939+0000 7f59dffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:55:28.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.939+0000 7f59dffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:55:28.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.939+0000 7f59dffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:55:28.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.939+0000 7f59dffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:55:28.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.939+0000 7f59dffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:55:28.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.939+0000 7f59dffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:55:28.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.939+0000 7f59dffff640 10 monclient(hunting): handle_auth_done global_id 824574 payload 824 2024-09-16T03:55:28.978 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.939+0000 7f59dffff640 1 -- 172.21.15.42:0/3472252414 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f59e0133b80 con 0x7f59e0135bc0 2024-09-16T03:55:28.978 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.939+0000 7f59dffff640 1 --2- 172.21.15.42:0/3472252414 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f59e0135bc0 0x7f59e0135fd0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f59d00059b0 tx=0x7f59d0003af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:55:28.978 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.939+0000 7f59dcff9640 10 client.824568.objecter ms_handle_connect 0x7f59e0135bc0 2024-09-16T03:55:28.978 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.939+0000 7f59dcff9640 10 client.824568.objecter resend_mon_ops 2024-09-16T03:55:28.978 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.939+0000 7f59dcff9640 1 -- 172.21.15.42:0/3472252414 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59d001e020 con 0x7f59e0135bc0 2024-09-16T03:55:28.978 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.939+0000 7f59dcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:55:28.979 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.939+0000 7f59dcff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:55:28.979 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.939+0000 7f59dcff9640 10 monclient: dump: 2024-09-16T03:55:28.979 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:55:28.979 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:55:28.979 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:28.979 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:28.979 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:55:28.979 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:55:28.979 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:55:28.980 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:55:28.980 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:55:28.980 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:55:28.980 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.939+0000 7f59dcff9640 1 -- 172.21.15.42:0/3472252414 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4882 (secure 0 0 0) 0x7f59d001e020 con 0x7f59e0135bc0 2024-09-16T03:55:28.980 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.939+0000 7f59dcff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:55:28.980 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.939+0000 7f59dcff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:55:28.980 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.939+0000 7f59dcff9640 1 -- 172.21.15.42:0/3472252414 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f59e0135bc0 msgr2=0x7f59e0135fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:55:28.980 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.939+0000 7f59dcff9640 1 --2- 172.21.15.42:0/3472252414 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f59e0135bc0 0x7f59e0135fd0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f59d00059b0 tx=0x7f59d0003af0 comp rx=0 tx=0).stop 2024-09-16T03:55:28.980 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.939+0000 7f59dcff9640 1 -- 172.21.15.42:0/3472252414 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f59e0135bc0 msgr2=0x7f59e0135fd0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:55:28.980 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.939+0000 7f59dcff9640 1 --2- 172.21.15.42:0/3472252414 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f59e0135bc0 0x7f59e0135fd0 secure :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f59d00059b0 tx=0x7f59d0003af0 comp rx=0 tx=0).stop 2024-09-16T03:55:28.981 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:55:28.981 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:55:28.981 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:55:28.981 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "leader", 2024-09-16T03:55:28.981 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 116, 2024-09-16T03:55:28.981 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 218793, 2024-09-16T03:55:28.981 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:55:28.981 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:55:28.981 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T03:55:28.981 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T03:55:28.981 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:55:28.981 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 0, 2024-09-16T03:55:28.981 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:55:28.982 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:55:28.982 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:55:28.982 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:55:28.982 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:55:28.982 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:55:28.982 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:55:28.982 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:55:28.982 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:55:28.982 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:55:28.982 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:55:28.982 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:55:28.982 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:55:28.982 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:55:28.982 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:55:28.983 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:55:28.983 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:55:28.983 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:55:28.983 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:55:28.983 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:55:28.983 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:55:28.983 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:55:28.983 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:55:28.983 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:55:28.983 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:55:28.983 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:55:28.983 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:55:28.983 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:55:28.983 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:55:28.984 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:55:28.984 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:55:28.984 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:55:28.984 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:55:28.984 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:55:28.984 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:55:28.984 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:55:28.984 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:55:28.984 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:55:28.984 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:55:28.984 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:55:28.985 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:55:28.985 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:55:28.985 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:55:28.985 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:55:28.985 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:55:28.985 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:55:28.985 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:55:28.985 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:55:28.985 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:55:28.985 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:55:28.985 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:55:28.985 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:55:28.985 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:55:28.985 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:55:28.986 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:55:28.986 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:55:28.986 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:55:28.986 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:55:28.986 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:55:28.986 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:55:28.986 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:55:28.986 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:55:28.986 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:55:28.987 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:55:28.987 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:55:28.987 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:55:28.987 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:55:28.987 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:55:28.987 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:55:28.987 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:55:28.987 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:55:28.987 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:55:28.987 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:55:28.987 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:55:28.987 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:55:28.987 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:55:28.987 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:55:28.988 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:55:28.988 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:55:28.988 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:55:28.988 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:55:28.988 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:55:28.988 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:55:28.988 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:55:28.988 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:55:28.988 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:55:28.988 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:55:28.988 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:55:28.988 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:55:28.988 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:55:28.988 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:55:28.989 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:55:28.989 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:55:28.989 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:55:28.989 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:55:28.989 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:55:28.989 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:55:28.989 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:55:28.989 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:55:28.989 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:55:28.989 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:55:28.989 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:55:28.989 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:55:28.989 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:55:28.989 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:55:28.990 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:55:28.990 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:55:28.990 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:55:28.990 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:55:28.990 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:55:28.990 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:55:28.990 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:55:28.990 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:55:28.990 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:55:28.990 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:55:28.990 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:55:28.990 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:55:28.990 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:55:28.990 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:55:28.991 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:55:28.991 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:55:28.991 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:55:28.991 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:55:28.991 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:55:28.991 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:55:28.991 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:55:28.991 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:55:28.991 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:55:28.991 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:55:28.991 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:55:28.991 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:55:28.991 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:55:28.991 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:55:28.992 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:55:28.992 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:55:28.992 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:55:28.992 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:55:28.992 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:55:28.992 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:55:28.992 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:55:28.992 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:55:28.992 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T03:55:28.992 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:55:28.992 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:55:28.992 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:55:28.992 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 6 2024-09-16T03:55:28.992 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:55:28.993 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:55:28.993 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:55:28.993 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:55:28.993 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:55:28.993 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:55:28.993 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 8 2024-09-16T03:55:28.993 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:55:28.993 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:55:28.993 INFO:teuthology.orchestra.run.smithi042.stdout: "mgr": [ 2024-09-16T03:55:28.993 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:55:28.993 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:55:28.993 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:55:28.993 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:55:28.993 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:55:28.994 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:55:28.994 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:55:28.994 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:55:28.994 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:55:28.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.939+0000 7f59e628e640 10 librados: watch_flush enter 2024-09-16T03:55:28.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.939+0000 7f59e628e640 10 librados: watch_flush exit 2024-09-16T03:55:28.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.939+0000 7f59e628e640 20 client.824568.objecter shutdown clearing up homeless session... 2024-09-16T03:55:28.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.939+0000 7f59e628e640 10 client.824568.objecter successfully canceled tick 2024-09-16T03:55:28.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.939+0000 7f59e628e640 1 -- 172.21.15.42:0/3472252414 >> v2:172.21.15.154:6812/254845342 conn(0x7f59c0037810 msgr2=0x7f59c0039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:55:28.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.939+0000 7f59e628e640 1 --2- 172.21.15.42:0/3472252414 >> v2:172.21.15.154:6812/254845342 conn(0x7f59c0037810 0x7f59c0039cd0 secure :-1 s=READY pgs=3633 cs=0 l=1 rev1=1 crypto rx=0x7f59d8006fd0 tx=0x7f59d8008040 comp rx=0 tx=0).stop 2024-09-16T03:55:28.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.939+0000 7f59e628e640 10 monclient: shutdown 2024-09-16T03:55:28.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.939+0000 7f59e628e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:55:28.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.939+0000 7f59e628e640 1 -- 172.21.15.42:0/3472252414 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f59e013c8c0 msgr2=0x7f59e013a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:55:28.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.939+0000 7f59e482b640 1 -- 172.21.15.42:0/3472252414 reap_dead start 2024-09-16T03:55:28.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.939+0000 7f59e628e640 1 --2- 172.21.15.42:0/3472252414 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f59e013c8c0 0x7f59e013a950 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f59c80ab000 tx=0x7f59c8019180 comp rx=0 tx=0).stop 2024-09-16T03:55:28.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.939+0000 7f59e628e640 1 -- 172.21.15.42:0/3472252414 shutdown_connections 2024-09-16T03:55:28.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.939+0000 7f59e628e640 1 --2- 172.21.15.42:0/3472252414 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f59e013c8c0 0x7f59e013a950 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:55:28.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.939+0000 7f59e628e640 1 -- 172.21.15.42:0/3472252414 >> 172.21.15.42:0/3472252414 conn(0x7f59e00a9fb0 msgr2=0x7f59e0131980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:55:28.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.939+0000 7f59e628e640 1 -- 172.21.15.42:0/3472252414 shutdown_connections 2024-09-16T03:55:28.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.939+0000 7f59e628e640 1 -- 172.21.15.42:0/3472252414 wait complete. 2024-09-16T03:55:28.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:28.939+0000 7f59e628e640 1 librados: shutdown 2024-09-16T03:55:28.997 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:55:28.997 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T03:55:28.997 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T03:55:29.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.119+0000 7f473e281640 10 monclient: get_monmap_and_config 2024-09-16T03:55:29.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.119+0000 7f473e281640 10 monclient: build_initial_monmap 2024-09-16T03:55:29.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.119+0000 7f473e281640 10 monclient: monmap: 2024-09-16T03:55:29.122 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:55:29.122 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:55:29.122 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:55:29.120840+0000 2024-09-16T03:55:29.122 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:55:29.120840+0000 2024-09-16T03:55:29.122 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:55:29.122 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:55:29.122 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:55:29.123 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:55:29.123 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:55:29.123 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:55:29.123 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:55:29.123 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:55:29.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.119+0000 7f473e281640 1 Processor -- start 2024-09-16T03:55:29.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.119+0000 7f473e281640 1 -- start start 2024-09-16T03:55:29.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.119+0000 7f473e281640 10 monclient: init 2024-09-16T03:55:29.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.119+0000 7f473e281640 10 monclient: _reopen_session rank -1 2024-09-16T03:55:29.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.119+0000 7f473e281640 10 monclient: _add_conns ranks=[4,1,0,3,2] 2024-09-16T03:55:29.128 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.119+0000 7f473e281640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f47380a39b0 0x7f4738133b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:55:29.128 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.119+0000 7f473e281640 10 monclient(hunting): picked mon.noname-c con 0x7f47380a39b0 addr v2:172.21.15.154:3300/0 2024-09-16T03:55:29.128 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.119+0000 7f473e281640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f47380a3d90 addr v1:172.21.15.154:6789/0 2024-09-16T03:55:29.128 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.119+0000 7f473e281640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4738137710 addr v1:172.21.15.42:6789/0 2024-09-16T03:55:29.128 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.119+0000 7f473e281640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f473809c140 con 0x7f4738137710 2024-09-16T03:55:29.128 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.119+0000 7f473e281640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f473809c420 con 0x7f47380a3d90 2024-09-16T03:55:29.128 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.119+0000 7f473e281640 10 monclient(hunting): start opening mon connection 2024-09-16T03:55:29.128 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.119+0000 7f473e281640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f473806dd60 con 0x7f47380a39b0 2024-09-16T03:55:29.128 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.119+0000 7f473e281640 10 monclient(hunting): _renew_subs 2024-09-16T03:55:29.128 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.119+0000 7f473e281640 10 monclient(hunting): authenticate will time out at 1776.091003s 2024-09-16T03:55:29.128 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.119+0000 7f473c81e640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f4738137710 0x7f473813f3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:49432/0 (socket says 172.21.15.42:49432) 2024-09-16T03:55:29.128 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.119+0000 7f473c81e640 1 -- 172.21.15.42:0/847799240 learned_addr learned my addr 172.21.15.42:0/847799240 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:55:29.128 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.119+0000 7f4737fff640 1 --2- 172.21.15.42:0/847799240 >> v2:172.21.15.154:3300/0 conn(0x7f47380a39b0 0x7f4738133b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:55:29.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.119+0000 7f4737fff640 10 monclient(hunting): get_auth_request con 0x7f47380a39b0 auth_method 0 2024-09-16T03:55:29.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.119+0000 7f4737fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:55:29.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.119+0000 7f4737fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:55:29.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.119+0000 7f4737fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:55:29.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.119+0000 7f4737fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:55:29.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.119+0000 7f4737fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:55:29.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.119+0000 7f4737fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:55:29.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.119+0000 7f4736ffd640 1 -- 172.21.15.42:0/847799240 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3320624404 0 0) 0x7f473809c140 con 0x7f4738137710 2024-09-16T03:55:29.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.119+0000 7f4736ffd640 10 monclient(hunting): _negotiate 2024-09-16T03:55:29.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.119+0000 7f4736ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:55:29.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.119+0000 7f4736ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:55:29.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.119+0000 7f4736ffd640 10 monclient(hunting): my global_id is 824577 2024-09-16T03:55:29.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.119+0000 7f4736ffd640 1 -- 172.21.15.42:0/847799240 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4720003530 con 0x7f4738137710 2024-09-16T03:55:29.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.119+0000 7f4736ffd640 1 -- 172.21.15.42:0/847799240 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2390803213 0 0) 0x7f4720003530 con 0x7f4738137710 2024-09-16T03:55:29.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.119+0000 7f4736ffd640 1 -- 172.21.15.42:0/847799240 >> v2:172.21.15.154:3300/0 conn(0x7f47380a39b0 msgr2=0x7f4738133b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:55:29.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.119+0000 7f4736ffd640 1 --2- 172.21.15.42:0/847799240 >> v2:172.21.15.154:3300/0 conn(0x7f47380a39b0 0x7f4738133b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:55:29.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.119+0000 7f4736ffd640 1 -- 172.21.15.42:0/847799240 >> v1:172.21.15.154:6789/0 conn(0x7f47380a3d90 legacy=0x7f4738137000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:55:29.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.123+0000 7f4736ffd640 10 monclient: _finish_hunting 0 2024-09-16T03:55:29.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.123+0000 7f4736ffd640 1 monclient: found mon.noname-a-legacy 2024-09-16T03:55:29.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.123+0000 7f4736ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:55:29.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.123+0000 7f4736ffd640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.42:6789/0 2024-09-16T03:55:29.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.123+0000 7f4736ffd640 1 -- 172.21.15.42:0/847799240 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47380a5e20 con 0x7f4738137710 2024-09-16T03:55:29.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.123+0000 7f4736ffd640 10 monclient: _finish_auth 0 2024-09-16T03:55:29.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.123+0000 7f4736ffd640 10 monclient: _check_auth_tickets 2024-09-16T03:55:29.131 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.123+0000 7f4736ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:55:29.131 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.123+0000 7f473e281640 5 monclient: authenticate success, global_id 824577 2024-09-16T03:55:29.131 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.123+0000 7f473e281640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:55:29.131 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.123+0000 7f4736ffd640 1 -- 172.21.15.42:0/847799240 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2177413734 0 0) 0x7f4730001a00 con 0x7f4738137710 2024-09-16T03:55:29.131 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.123+0000 7f4736ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:55:29.131 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.123+0000 7f4736ffd640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T03:55:29.131 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.123+0000 7f4736ffd640 10 monclient: dump: 2024-09-16T03:55:29.131 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:55:29.131 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:55:29.131 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:29.131 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:29.131 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:55:29.131 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:55:29.131 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:55:29.131 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:55:29.132 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:55:29.132 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:55:29.132 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.123+0000 7f4736ffd640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] but i'm connected to v1:172.21.15.42:6789/0, reconnecting 2024-09-16T03:55:29.132 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.123+0000 7f4736ffd640 10 monclient: _reopen_session rank -1 2024-09-16T03:55:29.132 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.123+0000 7f4736ffd640 1 -- 172.21.15.42:0/847799240 >> v1:172.21.15.42:6789/0 conn(0x7f4738137710 legacy=0x7f473813f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:55:29.132 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.123+0000 7f4736ffd640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:55:29.132 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.123+0000 7f4736ffd640 1 --2- 172.21.15.42:0/847799240 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f47200039c0 0x7f4720003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:55:29.132 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.123+0000 7f4736ffd640 10 monclient(hunting): picked mon.b con 0x7f47200039c0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:55:29.132 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.123+0000 7f4736ffd640 1 --2- 172.21.15.42:0/847799240 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4720004810 0x7f4720004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:55:29.132 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.123+0000 7f4736ffd640 10 monclient(hunting): picked mon.a con 0x7f4720004810 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:55:29.132 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.123+0000 7f4736ffd640 1 --2- 172.21.15.42:0/847799240 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f47200052f0 0x7f472000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:55:29.132 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.123+0000 7f4736ffd640 10 monclient(hunting): picked mon.c con 0x7f47200052f0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:55:29.132 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.123+0000 7f4736ffd640 10 monclient(hunting): start opening mon connection 2024-09-16T03:55:29.133 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.123+0000 7f4736ffd640 1 -- 172.21.15.42:0/847799240 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f473806dd60 con 0x7f4720004810 2024-09-16T03:55:29.133 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.123+0000 7f4736ffd640 10 monclient(hunting): start opening mon connection 2024-09-16T03:55:29.133 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.123+0000 7f4736ffd640 1 -- 172.21.15.42:0/847799240 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f472000c1c0 con 0x7f47200039c0 2024-09-16T03:55:29.133 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.123+0000 7f4736ffd640 10 monclient(hunting): start opening mon connection 2024-09-16T03:55:29.133 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.123+0000 7f4736ffd640 1 -- 172.21.15.42:0/847799240 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f472000c3c0 con 0x7f47200052f0 2024-09-16T03:55:29.133 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.123+0000 7f4736ffd640 10 monclient(hunting): _renew_subs 2024-09-16T03:55:29.133 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.123+0000 7f4736ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:55:29.133 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.123+0000 7f473e281640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-16T03:55:29.133 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.123+0000 7f4737fff640 1 --2- 172.21.15.42:0/847799240 >> v2:172.21.15.154:3300/0 conn(0x7f47380a39b0 0x7f4738133b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:55:29.133 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.123+0000 7f47377fe640 1 --2- 172.21.15.42:0/847799240 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4720004810 0x7f4720004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:55:29.133 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.123+0000 7f473c81e640 1 --2- 172.21.15.42:0/847799240 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f47200052f0 0x7f472000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:55:29.133 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.123+0000 7f47377fe640 10 monclient(hunting): get_auth_request con 0x7f4720004810 auth_method 0 2024-09-16T03:55:29.133 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.123+0000 7f47377fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:55:29.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.123+0000 7f47377fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:55:29.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.123+0000 7f47377fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:55:29.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.123+0000 7f473c81e640 10 monclient(hunting): get_auth_request con 0x7f47200052f0 auth_method 0 2024-09-16T03:55:29.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.123+0000 7f473c81e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:55:29.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.123+0000 7f473c81e640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:55:29.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.123+0000 7f473c81e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:55:29.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.123+0000 7f47377fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:55:29.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.123+0000 7f47377fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:55:29.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.123+0000 7f47377fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T03:55:29.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.123+0000 7f473c81e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:55:29.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.123+0000 7f473c81e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:55:29.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.123+0000 7f473c81e640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T03:55:29.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.123+0000 7f4737fff640 1 --2- 172.21.15.42:0/847799240 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f47200039c0 0x7f4720003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:55:29.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.123+0000 7f4737fff640 10 monclient(hunting): get_auth_request con 0x7f47200039c0 auth_method 0 2024-09-16T03:55:29.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.123+0000 7f4737fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:55:29.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.123+0000 7f4737fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:55:29.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.123+0000 7f4737fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:55:29.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.123+0000 7f47377fe640 10 monclient(hunting): handle_auth_done global_id 824577 payload 293 2024-09-16T03:55:29.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.123+0000 7f47377fe640 1 -- 172.21.15.42:0/847799240 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f47200052f0 msgr2=0x7f472000b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:55:29.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.123+0000 7f47377fe640 1 --2- 172.21.15.42:0/847799240 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f47200052f0 0x7f472000b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:55:29.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.123+0000 7f47377fe640 1 -- 172.21.15.42:0/847799240 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f47200039c0 msgr2=0x7f4720003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:55:29.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.123+0000 7f47377fe640 1 --2- 172.21.15.42:0/847799240 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f47200039c0 0x7f4720003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:55:29.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.123+0000 7f47377fe640 10 monclient: _finish_hunting 0 2024-09-16T03:55:29.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.123+0000 7f47377fe640 1 monclient: found mon.a 2024-09-16T03:55:29.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.123+0000 7f47377fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:55:29.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.123+0000 7f47377fe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:55:29.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.123+0000 7f47377fe640 1 -- 172.21.15.42:0/847799240 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4720003530 con 0x7f4720004810 2024-09-16T03:55:29.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.123+0000 7f47377fe640 10 monclient: _finish_auth 0 2024-09-16T03:55:29.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.123+0000 7f47377fe640 10 monclient: _check_auth_tickets 2024-09-16T03:55:29.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.123+0000 7f47377fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:55:29.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.123+0000 7f473c81e640 1 --2- 172.21.15.42:0/847799240 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f47200052f0 0x7f472000b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:55:29.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.123+0000 7f473c81e640 1 -- 172.21.15.42:0/847799240 reap_dead start 2024-09-16T03:55:29.143 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.123+0000 7f47377fe640 1 --2- 172.21.15.42:0/847799240 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4720004810 0x7f4720004c90 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f472c001750 tx=0x7f472c01c2c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8ba6270a643c6e0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:55:29.143 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.123+0000 7f4736ffd640 1 -- 172.21.15.42:0/847799240 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f472c0013a0 con 0x7f4720004810 2024-09-16T03:55:29.143 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.123+0000 7f4736ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:55:29.143 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.123+0000 7f4736ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:55:29.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.123+0000 7f4736ffd640 10 monclient: dump: 2024-09-16T03:55:29.144 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:55:29.144 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:55:29.144 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:29.144 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:29.144 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:55:29.144 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:55:29.144 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:55:29.145 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:55:29.146 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:55:29.146 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:55:29.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.123+0000 7f4736ffd640 1 -- 172.21.15.42:0/847799240 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f472c0052f0 con 0x7f4720004810 2024-09-16T03:55:29.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.123+0000 7f4736ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:55:29.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.123+0000 7f4736ffd640 1 -- 172.21.15.42:0/847799240 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f472c003840 con 0x7f4720004810 2024-09-16T03:55:29.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.123+0000 7f473e281640 10 monclient: get_monmap_and_config success 2024-09-16T03:55:29.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.123+0000 7f473e281640 4 set_mon_vals no callback set 2024-09-16T03:55:29.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.123+0000 7f4736ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:55:29.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.123+0000 7f4736ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:55:29.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.123+0000 7f4736ffd640 10 monclient: dump: 2024-09-16T03:55:29.146 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:55:29.146 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:55:29.146 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:29.147 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:29.147 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:55:29.147 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:55:29.147 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:55:29.147 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:55:29.147 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:55:29.147 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:55:29.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.123+0000 7f473e281640 10 monclient: shutdown 2024-09-16T03:55:29.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.123+0000 7f473e281640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:55:29.148 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.123+0000 7f473e281640 1 -- 172.21.15.42:0/847799240 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4720004810 msgr2=0x7f4720004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:55:29.148 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.123+0000 7f473e281640 1 --2- 172.21.15.42:0/847799240 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4720004810 0x7f4720004c90 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f472c001750 tx=0x7f472c01c2c0 comp rx=0 tx=0).stop 2024-09-16T03:55:29.148 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.123+0000 7f473e281640 1 -- 172.21.15.42:0/847799240 shutdown_connections 2024-09-16T03:55:29.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.123+0000 7f473e281640 1 --2- 172.21.15.42:0/847799240 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4720004810 0x7f4720004c90 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:55:29.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.123+0000 7f473e281640 1 -- 172.21.15.42:0/847799240 >> 172.21.15.42:0/847799240 conn(0x7f47380a9fb0 msgr2=0x7f47380aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:55:29.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.123+0000 7f473e281640 1 -- 172.21.15.42:0/847799240 shutdown_connections 2024-09-16T03:55:29.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.123+0000 7f473e281640 1 -- 172.21.15.42:0/847799240 wait complete. 2024-09-16T03:55:29.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.127+0000 7f473e281640 10 monclient: build_initial_monmap 2024-09-16T03:55:29.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.127+0000 7f473e281640 10 monclient: monmap: 2024-09-16T03:55:29.149 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:55:29.149 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:55:29.149 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:55:29.149 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:55:29.149 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:55:29.149 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:55:29.150 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:55:29.150 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:55:29.150 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:55:29.150 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:55:29.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.127+0000 7f473e281640 1 librados: starting msgr at 2024-09-16T03:55:29.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.127+0000 7f473e281640 1 librados: starting objecter 2024-09-16T03:55:29.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.127+0000 7f473e281640 1 Processor -- start 2024-09-16T03:55:29.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.127+0000 7f473e281640 1 -- start start 2024-09-16T03:55:29.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.127+0000 7f473e281640 1 librados: setting wanted keys 2024-09-16T03:55:29.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.127+0000 7f473e281640 1 librados: calling monclient init 2024-09-16T03:55:29.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.127+0000 7f473e281640 10 monclient: init 2024-09-16T03:55:29.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.127+0000 7f473e281640 10 monclient: _reopen_session rank -1 2024-09-16T03:55:29.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.127+0000 7f473e281640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:55:29.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.127+0000 7f473e281640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4720004810 0x7f47381206e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:55:29.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.127+0000 7f473e281640 10 monclient(hunting): picked mon.noname-b con 0x7f4720004810 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:55:29.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.127+0000 7f473e281640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4738120c20 0x7f47381cfa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:55:29.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.127+0000 7f473e281640 10 monclient(hunting): picked mon.noname-a con 0x7f4738120c20 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:55:29.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.127+0000 7f473e281640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4738121060 0x7f47381cff50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:55:29.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.127+0000 7f473e281640 10 monclient(hunting): picked mon.noname-c con 0x7f4738121060 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:55:29.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.127+0000 7f473e281640 10 monclient(hunting): start opening mon connection 2024-09-16T03:55:29.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.127+0000 7f473e281640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f473813a580 con 0x7f4738120c20 2024-09-16T03:55:29.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.127+0000 7f473e281640 10 monclient(hunting): start opening mon connection 2024-09-16T03:55:29.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.127+0000 7f473e281640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f473806dee0 con 0x7f4720004810 2024-09-16T03:55:29.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.127+0000 7f473e281640 10 monclient(hunting): start opening mon connection 2024-09-16T03:55:29.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.127+0000 7f473e281640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f473813a700 con 0x7f4738121060 2024-09-16T03:55:29.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.127+0000 7f473e281640 10 monclient(hunting): _renew_subs 2024-09-16T03:55:29.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.127+0000 7f473e281640 10 monclient(hunting): authenticate will time out at 1776.097104s 2024-09-16T03:55:29.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.127+0000 7f473c81e640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4738121060 0x7f47381cff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:55:29.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.127+0000 7f47377fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4738120c20 0x7f47381cfa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:55:29.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.127+0000 7f47377fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4738120c20 0x7f47381cfa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:34784/0 (socket says 172.21.15.42:34784) 2024-09-16T03:55:29.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.127+0000 7f473c81e640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4738121060 0x7f47381cff50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:51686/0 (socket says 172.21.15.42:51686) 2024-09-16T03:55:29.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.127+0000 7f47377fe640 1 -- 172.21.15.42:0/3818941490 learned_addr learned my addr 172.21.15.42:0/3818941490 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:55:29.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.127+0000 7f47377fe640 10 monclient(hunting): get_auth_request con 0x7f4738120c20 auth_method 0 2024-09-16T03:55:29.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.127+0000 7f47377fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:55:29.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.127+0000 7f47377fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:55:29.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.127+0000 7f47377fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:55:29.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.127+0000 7f473c81e640 10 monclient(hunting): get_auth_request con 0x7f4738121060 auth_method 0 2024-09-16T03:55:29.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.127+0000 7f473c81e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:55:29.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.127+0000 7f473c81e640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:55:29.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.127+0000 7f473c81e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:55:29.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.127+0000 7f47377fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:55:29.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.127+0000 7f47377fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:55:29.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.127+0000 7f47377fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:55:29.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.127+0000 7f473c81e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:55:29.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.127+0000 7f473c81e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:55:29.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.127+0000 7f473c81e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:55:29.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.127+0000 7f4737fff640 1 --2- 172.21.15.42:0/3818941490 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4720004810 0x7f47381206e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:55:29.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.127+0000 7f4737fff640 10 monclient(hunting): get_auth_request con 0x7f4720004810 auth_method 0 2024-09-16T03:55:29.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.127+0000 7f4737fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:55:29.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.127+0000 7f4737fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:55:29.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.127+0000 7f4737fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:55:29.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.127+0000 7f47377fe640 10 monclient(hunting): handle_auth_done global_id 824580 payload 824 2024-09-16T03:55:29.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.127+0000 7f47377fe640 1 -- 172.21.15.42:0/3818941490 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4738121060 msgr2=0x7f47381cff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:55:29.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.127+0000 7f47377fe640 1 --2- 172.21.15.42:0/3818941490 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4738121060 0x7f47381cff50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:55:29.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.127+0000 7f47377fe640 1 -- 172.21.15.42:0/3818941490 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4720004810 msgr2=0x7f47381206e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:55:29.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.127+0000 7f47377fe640 1 --2- 172.21.15.42:0/3818941490 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4720004810 0x7f47381206e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:55:29.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.127+0000 7f47377fe640 10 monclient: _finish_hunting 0 2024-09-16T03:55:29.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.127+0000 7f47377fe640 1 monclient: found mon.noname-a 2024-09-16T03:55:29.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.127+0000 7f47377fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:55:29.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.127+0000 7f47377fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:55:29.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.127+0000 7f47377fe640 1 -- 172.21.15.42:0/3818941490 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47380a33c0 con 0x7f4738120c20 2024-09-16T03:55:29.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.127+0000 7f473c81e640 1 --2- 172.21.15.42:0/3818941490 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4738121060 0x7f47381cff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:55:29.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.127+0000 7f47377fe640 1 --2- 172.21.15.42:0/3818941490 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4738120c20 0x7f47381cfa10 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f472c01c0f0 tx=0x7f4720005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:55:29.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.127+0000 7f4734ff9640 10 client.?.objecter ms_handle_connect 0x7f4738120c20 2024-09-16T03:55:29.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.127+0000 7f4734ff9640 10 client.?.objecter resend_mon_ops 2024-09-16T03:55:29.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.127+0000 7f4734ff9640 1 -- 172.21.15.42:0/3818941490 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f472c0b8020 con 0x7f4738120c20 2024-09-16T03:55:29.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.127+0000 7f4734ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:55:29.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.127+0000 7f4734ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:55:29.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.127+0000 7f4734ff9640 10 monclient: dump: 2024-09-16T03:55:29.155 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:55:29.155 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:55:29.155 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:29.155 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:29.155 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:55:29.155 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:55:29.155 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:55:29.155 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:55:29.155 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:55:29.155 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:55:29.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.127+0000 7f4734ff9640 10 monclient: _finish_auth 0 2024-09-16T03:55:29.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.127+0000 7f4734ff9640 10 monclient: _check_auth_tickets 2024-09-16T03:55:29.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.127+0000 7f4734ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:55:29.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.127+0000 7f4734ff9640 1 -- 172.21.15.42:0/3818941490 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f472c007230 con 0x7f4738120c20 2024-09-16T03:55:29.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.127+0000 7f4734ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T03:55:29.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.127+0000 7f4734ff9640 1 -- 172.21.15.42:0/3818941490 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f472c0013a0 con 0x7f4738120c20 2024-09-16T03:55:29.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.127+0000 7f4734ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:55:29.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.127+0000 7f4734ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:55:29.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.127+0000 7f4734ff9640 10 monclient: dump: 2024-09-16T03:55:29.156 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:55:29.156 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:55:29.156 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:29.157 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:29.157 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:55:29.157 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:55:29.157 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:55:29.157 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:55:29.157 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:55:29.157 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:55:29.157 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.127+0000 7f47367fc640 4 set_mon_vals no callback set 2024-09-16T03:55:29.157 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.127+0000 7f473e281640 5 monclient: authenticate success, global_id 824580 2024-09-16T03:55:29.157 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.127+0000 7f473e281640 10 monclient: _renew_subs 2024-09-16T03:55:29.157 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.127+0000 7f473e281640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:55:29.157 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.127+0000 7f473e281640 1 -- 172.21.15.42:0/3818941490 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4738122f40 con 0x7f4738120c20 2024-09-16T03:55:29.157 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.127+0000 7f473e281640 10 client.824580.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:55:29.157 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.127+0000 7f473e281640 10 monclient: _renew_subs 2024-09-16T03:55:29.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.127+0000 7f473e281640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:55:29.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.127+0000 7f473e281640 1 -- 172.21.15.42:0/3818941490 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47380a98e0 con 0x7f4738120c20 2024-09-16T03:55:29.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.127+0000 7f473e281640 1 librados: init done 2024-09-16T03:55:29.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.127+0000 7f473e281640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:55:29.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.127+0000 7f473e281640 1 --2- 172.21.15.42:0/3818941490 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4738122b40 0x7f47381d0490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:55:29.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.127+0000 7f473e281640 10 monclient(hunting): start opening mon connection 2024-09-16T03:55:29.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.127+0000 7f473e281640 1 -- 172.21.15.42:0/3818941490 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f473806f680 con 0x7f4738122b40 2024-09-16T03:55:29.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.127+0000 7f4737fff640 1 --2- 172.21.15.42:0/3818941490 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4738122b40 0x7f47381d0490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:55:29.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.127+0000 7f4737fff640 10 monclient: get_auth_request con 0x7f4738122b40 auth_method 0 2024-09-16T03:55:29.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.127+0000 7f4734ff9640 1 -- 172.21.15.42:0/3818941490 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f472c0bb020 con 0x7f4738120c20 2024-09-16T03:55:29.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.127+0000 7f4737fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:55:29.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.127+0000 7f4737fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:55:29.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.127+0000 7f4737fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:55:29.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.127+0000 7f4737fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:55:29.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.127+0000 7f4737fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:55:29.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.127+0000 7f4737fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:55:29.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.127+0000 7f4734ff9640 1 --2- 172.21.15.42:0/3818941490 >> v2:172.21.15.154:6812/254845342 conn(0x7f4724037840 0x7f4724039d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:55:29.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.127+0000 7f4737fff640 10 monclient(hunting): handle_auth_done global_id 854201 payload 824 2024-09-16T03:55:29.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.127+0000 7f4734ff9640 1 -- 172.21.15.42:0/3818941490 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(415..415 src has 330..415) ==== 5328+0+0 (secure 0 0 0) 0x7f472c0e31a0 con 0x7f4738120c20 2024-09-16T03:55:29.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.127+0000 7f4734ff9640 10 client.824580.objecter ms_dispatch 0x7f4738000ba0 osd_map(415..415 src has 330..415) 2024-09-16T03:55:29.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.127+0000 7f4734ff9640 3 client.824580.objecter handle_osd_map got epochs [415,415] > 0 2024-09-16T03:55:29.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.127+0000 7f4734ff9640 3 client.824580.objecter handle_osd_map decoding full epoch 415 2024-09-16T03:55:29.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.127+0000 7f4737fff640 1 -- 172.21.15.42:0/3818941490 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f473813a910 con 0x7f4738122b40 2024-09-16T03:55:29.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.127+0000 7f4734ff9640 20 client.824580.objecter dump_active .. 0 homeless 2024-09-16T03:55:29.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.127+0000 7f4734ff9640 10 client.824580.objecter ms_handle_connect 0x7f4738122b40 2024-09-16T03:55:29.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.127+0000 7f4734ff9640 10 client.824580.objecter resend_mon_ops 2024-09-16T03:55:29.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.127+0000 7f4737fff640 1 --2- 172.21.15.42:0/3818941490 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4738122b40 0x7f47381d0490 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f47280138b0 tx=0x7f4728013d70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:55:29.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.131+0000 7f473c81e640 1 --2- 172.21.15.42:0/3818941490 >> v2:172.21.15.154:6812/254845342 conn(0x7f4724037840 0x7f4724039d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:55:29.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.131+0000 7f473c81e640 10 monclient: get_auth_request con 0x7f4724037840 auth_method 0 2024-09-16T03:55:29.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.131+0000 7f4734ff9640 1 -- 172.21.15.42:0/3818941490 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f472801c070 con 0x7f4738122b40 2024-09-16T03:55:29.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.131+0000 7f4734ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:55:29.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.131+0000 7f4734ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:55:29.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.131+0000 7f4734ff9640 10 monclient: dump: 2024-09-16T03:55:29.161 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:55:29.161 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:55:29.162 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:29.162 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:29.162 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:55:29.162 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:55:29.162 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:55:29.162 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:55:29.162 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:55:29.162 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:55:29.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.131+0000 7f4734ff9640 1 -- 172.21.15.42:0/3818941490 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f4728009e50 con 0x7f4738122b40 2024-09-16T03:55:29.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.131+0000 7f4734ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:55:29.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.131+0000 7f4734ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:55:29.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.131+0000 7f4734ff9640 1 -- 172.21.15.42:0/3818941490 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4738122b40 msgr2=0x7f47381d0490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:55:29.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.131+0000 7f4734ff9640 1 --2- 172.21.15.42:0/3818941490 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4738122b40 0x7f47381d0490 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f47280138b0 tx=0x7f4728013d70 comp rx=0 tx=0).stop 2024-09-16T03:55:29.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.131+0000 7f4734ff9640 1 -- 172.21.15.42:0/3818941490 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4738122b40 msgr2=0x7f47381d0490 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:55:29.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.131+0000 7f4734ff9640 1 --2- 172.21.15.42:0/3818941490 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4738122b40 0x7f47381d0490 secure :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f47280138b0 tx=0x7f4728013d70 comp rx=0 tx=0).stop 2024-09-16T03:55:29.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.131+0000 7f473c81e640 1 --2- 172.21.15.42:0/3818941490 >> v2:172.21.15.154:6812/254845342 conn(0x7f4724037840 0x7f4724039d00 secure :-1 s=READY pgs=3634 cs=0 l=1 rev1=1 crypto rx=0x7f47380a4270 tx=0x7f4730006040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:55:29.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.131+0000 7f4734ff9640 10 client.824580.objecter ms_handle_connect 0x7f4724037840 2024-09-16T03:55:29.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.135+0000 7f473e281640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:55:29.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.135+0000 7f473e281640 1 --2- 172.21.15.42:0/3818941490 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f473809e950 0x7f473809ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:55:29.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.135+0000 7f473e281640 10 monclient(hunting): start opening mon connection 2024-09-16T03:55:29.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.135+0000 7f473e281640 1 -- 172.21.15.42:0/3818941490 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f47380a5750 con 0x7f473809e950 2024-09-16T03:55:29.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.135+0000 7f4737fff640 1 --2- 172.21.15.42:0/3818941490 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f473809e950 0x7f473809ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:55:29.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.135+0000 7f4737fff640 10 monclient: get_auth_request con 0x7f473809e950 auth_method 0 2024-09-16T03:55:29.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.135+0000 7f4737fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:55:29.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.135+0000 7f4737fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:55:29.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.135+0000 7f4737fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:55:29.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.135+0000 7f4737fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:55:29.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.135+0000 7f4737fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:55:29.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.135+0000 7f4737fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:55:29.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.135+0000 7f4737fff640 10 monclient(hunting): handle_auth_done global_id 854204 payload 824 2024-09-16T03:55:29.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.135+0000 7f4737fff640 1 -- 172.21.15.42:0/3818941490 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f47380a9330 con 0x7f473809e950 2024-09-16T03:55:29.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.135+0000 7f4737fff640 1 --2- 172.21.15.42:0/3818941490 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f473809e950 0x7f473809ed60 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f47280138b0 tx=0x7f4728004740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:55:29.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.135+0000 7f4734ff9640 10 client.824580.objecter ms_handle_connect 0x7f473809e950 2024-09-16T03:55:29.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.135+0000 7f4734ff9640 10 client.824580.objecter resend_mon_ops 2024-09-16T03:55:29.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.135+0000 7f4734ff9640 1 -- 172.21.15.42:0/3818941490 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4728006610 con 0x7f473809e950 2024-09-16T03:55:29.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.135+0000 7f4734ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:55:29.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.135+0000 7f4734ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:55:29.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.135+0000 7f4734ff9640 10 monclient: dump: 2024-09-16T03:55:29.165 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:55:29.165 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:55:29.165 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:29.166 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:29.166 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:55:29.167 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:55:29.168 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:55:29.168 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:55:29.168 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:55:29.168 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:55:29.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.135+0000 7f4734ff9640 1 -- 172.21.15.42:0/3818941490 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4545 (secure 0 0 0) 0x7f4728006610 con 0x7f473809e950 2024-09-16T03:55:29.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.135+0000 7f4734ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:55:29.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.135+0000 7f4734ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:55:29.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.135+0000 7f4734ff9640 1 -- 172.21.15.42:0/3818941490 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f473809e950 msgr2=0x7f473809ed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:55:29.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.135+0000 7f4734ff9640 1 --2- 172.21.15.42:0/3818941490 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f473809e950 0x7f473809ed60 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f47280138b0 tx=0x7f4728004740 comp rx=0 tx=0).stop 2024-09-16T03:55:29.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.135+0000 7f4734ff9640 1 -- 172.21.15.42:0/3818941490 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f473809e950 msgr2=0x7f473809ed60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:55:29.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.135+0000 7f4734ff9640 1 --2- 172.21.15.42:0/3818941490 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f473809e950 0x7f473809ed60 secure :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f47280138b0 tx=0x7f4728004740 comp rx=0 tx=0).stop 2024-09-16T03:55:29.169 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:55:29.169 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:55:29.169 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:55:29.170 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T03:55:29.170 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 116, 2024-09-16T03:55:29.170 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 62571, 2024-09-16T03:55:29.170 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:55:29.170 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:55:29.170 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T03:55:29.170 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T03:55:29.170 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:55:29.170 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 1, 2024-09-16T03:55:29.170 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:55:29.170 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:55:29.170 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:55:29.170 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:55:29.170 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:55:29.171 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:55:29.171 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:55:29.171 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:55:29.171 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:55:29.171 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:55:29.171 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:55:29.171 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:55:29.171 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:55:29.172 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:55:29.172 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:55:29.172 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:55:29.172 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:55:29.172 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:55:29.172 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:55:29.172 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:55:29.172 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:55:29.172 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:55:29.173 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:55:29.173 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:55:29.173 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:55:29.173 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:55:29.173 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:55:29.173 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:55:29.173 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:55:29.173 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:55:29.173 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:55:29.173 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:55:29.173 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:55:29.173 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:55:29.173 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:55:29.174 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:55:29.174 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:55:29.174 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:55:29.174 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:55:29.174 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:55:29.174 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:55:29.174 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:55:29.174 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:55:29.174 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:55:29.175 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:55:29.175 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:55:29.175 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:55:29.175 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:55:29.175 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:55:29.175 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:55:29.175 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:55:29.175 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:55:29.175 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:55:29.175 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:55:29.176 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:55:29.176 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:55:29.176 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:55:29.176 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:55:29.176 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:55:29.176 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:55:29.176 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:55:29.176 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:55:29.176 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:55:29.177 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:55:29.177 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:55:29.177 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:55:29.177 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:55:29.177 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:55:29.177 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:55:29.177 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:55:29.177 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:55:29.177 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:55:29.177 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:55:29.177 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:55:29.177 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:55:29.177 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:55:29.177 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:55:29.178 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:55:29.178 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:55:29.178 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:55:29.178 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:55:29.178 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:55:29.178 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:55:29.178 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:55:29.178 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:55:29.178 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:55:29.178 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:55:29.178 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:55:29.178 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:55:29.178 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:55:29.178 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:55:29.179 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:55:29.179 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:55:29.179 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:55:29.179 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:55:29.179 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:55:29.179 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:55:29.179 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:55:29.179 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:55:29.179 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:55:29.180 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:55:29.180 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:55:29.180 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:55:29.180 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:55:29.180 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:55:29.180 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:55:29.180 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:55:29.180 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:55:29.180 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:55:29.180 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:55:29.180 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:55:29.180 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:55:29.180 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:55:29.181 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:55:29.181 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:55:29.181 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:55:29.181 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:55:29.181 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:55:29.181 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:55:29.181 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:55:29.181 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:55:29.181 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:55:29.182 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:55:29.182 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:55:29.182 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:55:29.182 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:55:29.182 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:55:29.182 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:55:29.182 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:55:29.182 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:55:29.182 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:55:29.183 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:55:29.183 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:55:29.183 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:55:29.183 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:55:29.183 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:55:29.183 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:55:29.183 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:55:29.183 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:55:29.183 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:55:29.183 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:55:29.183 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:55:29.183 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:55:29.183 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:55:29.184 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:55:29.184 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:55:29.184 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:55:29.184 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:55:29.184 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:55:29.184 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:55:29.184 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:55:29.187 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.135+0000 7f473e281640 10 librados: watch_flush enter 2024-09-16T03:55:29.187 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.135+0000 7f473e281640 10 librados: watch_flush exit 2024-09-16T03:55:29.187 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.135+0000 7f473e281640 20 client.824580.objecter shutdown clearing up homeless session... 2024-09-16T03:55:29.187 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.135+0000 7f473e281640 10 client.824580.objecter successfully canceled tick 2024-09-16T03:55:29.187 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.139+0000 7f473e281640 1 -- 172.21.15.42:0/3818941490 >> v2:172.21.15.154:6812/254845342 conn(0x7f4724037840 msgr2=0x7f4724039d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:55:29.187 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.139+0000 7f473e281640 1 --2- 172.21.15.42:0/3818941490 >> v2:172.21.15.154:6812/254845342 conn(0x7f4724037840 0x7f4724039d00 secure :-1 s=READY pgs=3634 cs=0 l=1 rev1=1 crypto rx=0x7f47380a4270 tx=0x7f4730006040 comp rx=0 tx=0).stop 2024-09-16T03:55:29.187 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.139+0000 7f473e281640 10 monclient: shutdown 2024-09-16T03:55:29.187 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.139+0000 7f473e281640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:55:29.187 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.139+0000 7f473e281640 1 -- 172.21.15.42:0/3818941490 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4738120c20 msgr2=0x7f47381cfa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:55:29.187 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.139+0000 7f473e281640 1 --2- 172.21.15.42:0/3818941490 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4738120c20 0x7f47381cfa10 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f472c01c0f0 tx=0x7f4720005840 comp rx=0 tx=0).stop 2024-09-16T03:55:29.187 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.139+0000 7f473c81e640 1 -- 172.21.15.42:0/3818941490 reap_dead start 2024-09-16T03:55:29.187 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.139+0000 7f473c81e640 1 -- 172.21.15.42:0/3818941490 reap_dead start 2024-09-16T03:55:29.188 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.139+0000 7f473e281640 1 -- 172.21.15.42:0/3818941490 shutdown_connections 2024-09-16T03:55:29.188 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.139+0000 7f473e281640 1 -- 172.21.15.42:0/3818941490 >> 172.21.15.42:0/3818941490 conn(0x7f47380a9fb0 msgr2=0x7f47380a7a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:55:29.188 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.139+0000 7f473e281640 1 -- 172.21.15.42:0/3818941490 shutdown_connections 2024-09-16T03:55:29.188 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.139+0000 7f473e281640 1 -- 172.21.15.42:0/3818941490 wait complete. 2024-09-16T03:55:29.188 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.139+0000 7f473e281640 1 librados: shutdown 2024-09-16T03:55:29.188 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:55:29.188 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T03:55:29.188 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T03:55:29.307 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.303+0000 7f0c635c2640 10 monclient: get_monmap_and_config 2024-09-16T03:55:29.307 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.303+0000 7f0c635c2640 10 monclient: build_initial_monmap 2024-09-16T03:55:29.307 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.303+0000 7f0c635c2640 10 monclient: monmap: 2024-09-16T03:55:29.307 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:55:29.307 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:55:29.307 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:55:29.305855+0000 2024-09-16T03:55:29.307 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:55:29.305855+0000 2024-09-16T03:55:29.307 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:55:29.308 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:55:29.308 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:55:29.308 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:55:29.308 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:55:29.308 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:55:29.308 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:55:29.308 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:55:29.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.303+0000 7f0c635c2640 1 Processor -- start 2024-09-16T03:55:29.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.303+0000 7f0c635c2640 1 -- start start 2024-09-16T03:55:29.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.303+0000 7f0c635c2640 10 monclient: init 2024-09-16T03:55:29.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.307+0000 7f0c635c2640 10 monclient: _reopen_session rank -1 2024-09-16T03:55:29.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.307+0000 7f0c635c2640 10 monclient: _add_conns ranks=[2,3,4,0,1] 2024-09-16T03:55:29.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.307+0000 7f0c635c2640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0c5c0a39b0 0x7f0c5c133b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:55:29.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.307+0000 7f0c635c2640 10 monclient(hunting): picked mon.noname-b con 0x7f0c5c0a39b0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:55:29.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.307+0000 7f0c635c2640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f0c5c0a3d90 0x7f0c5c137000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:55:29.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.307+0000 7f0c635c2640 10 monclient(hunting): picked mon.noname-a con 0x7f0c5c0a3d90 addr v2:172.21.15.42:3300/0 2024-09-16T03:55:29.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.307+0000 7f0c635c2640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f0c5c137540 0x7f0c5c137950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:55:29.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.307+0000 7f0c635c2640 10 monclient(hunting): picked mon.noname-c con 0x7f0c5c137540 addr v2:172.21.15.154:3300/0 2024-09-16T03:55:29.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.307+0000 7f0c635c2640 10 monclient(hunting): start opening mon connection 2024-09-16T03:55:29.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.307+0000 7f0c635c2640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f0c5c137e90 con 0x7f0c5c0a3d90 2024-09-16T03:55:29.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.307+0000 7f0c635c2640 10 monclient(hunting): start opening mon connection 2024-09-16T03:55:29.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.307+0000 7f0c635c2640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f0c5c09c140 con 0x7f0c5c137540 2024-09-16T03:55:29.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.307+0000 7f0c635c2640 10 monclient(hunting): start opening mon connection 2024-09-16T03:55:29.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.307+0000 7f0c635c2640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f0c5c09c420 con 0x7f0c5c0a39b0 2024-09-16T03:55:29.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.307+0000 7f0c635c2640 10 monclient(hunting): _renew_subs 2024-09-16T03:55:29.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.307+0000 7f0c635c2640 10 monclient(hunting): authenticate will time out at 1776.276355s 2024-09-16T03:55:29.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.307+0000 7f0c60b5d640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f0c5c0a3d90 0x7f0c5c137000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:55:29.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.307+0000 7f0c6135e640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0c5c0a39b0 0x7f0c5c133b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:55:29.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.307+0000 7f0c6135e640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0c5c0a39b0 0x7f0c5c133b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:51700/0 (socket says 172.21.15.42:51700) 2024-09-16T03:55:29.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.307+0000 7f0c60b5d640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f0c5c0a3d90 0x7f0c5c137000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:34788/0 (socket says 172.21.15.42:34788) 2024-09-16T03:55:29.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.307+0000 7f0c6135e640 1 -- 172.21.15.42:0/604970377 learned_addr learned my addr 172.21.15.42:0/604970377 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:55:29.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.307+0000 7f0c6135e640 10 monclient(hunting): get_auth_request con 0x7f0c5c0a39b0 auth_method 0 2024-09-16T03:55:29.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.307+0000 7f0c6135e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:55:29.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.307+0000 7f0c6135e640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:55:29.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.307+0000 7f0c6135e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:55:29.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.307+0000 7f0c60b5d640 10 monclient(hunting): get_auth_request con 0x7f0c5c0a3d90 auth_method 0 2024-09-16T03:55:29.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.307+0000 7f0c60b5d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:55:29.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.307+0000 7f0c60b5d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:55:29.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.307+0000 7f0c60b5d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:55:29.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.307+0000 7f0c6135e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:55:29.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.307+0000 7f0c6135e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:55:29.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.307+0000 7f0c6135e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:55:29.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.307+0000 7f0c60b5d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:55:29.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.307+0000 7f0c60b5d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:55:29.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.307+0000 7f0c60b5d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:55:29.314 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.307+0000 7f0c61b5f640 1 --2- 172.21.15.42:0/604970377 >> v2:172.21.15.154:3300/0 conn(0x7f0c5c137540 0x7f0c5c137950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:55:29.314 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.307+0000 7f0c6135e640 10 monclient(hunting): handle_auth_done global_id 854207 payload 274 2024-09-16T03:55:29.314 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.307+0000 7f0c6135e640 1 -- 172.21.15.42:0/604970377 >> v2:172.21.15.154:3300/0 conn(0x7f0c5c137540 msgr2=0x7f0c5c137950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:55:29.314 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.307+0000 7f0c6135e640 1 --2- 172.21.15.42:0/604970377 >> v2:172.21.15.154:3300/0 conn(0x7f0c5c137540 0x7f0c5c137950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:55:29.314 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.307+0000 7f0c6135e640 1 -- 172.21.15.42:0/604970377 >> v2:172.21.15.42:3300/0 conn(0x7f0c5c0a3d90 msgr2=0x7f0c5c137000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:55:29.314 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.307+0000 7f0c6135e640 1 --2- 172.21.15.42:0/604970377 >> v2:172.21.15.42:3300/0 conn(0x7f0c5c0a3d90 0x7f0c5c137000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:55:29.314 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.307+0000 7f0c6135e640 10 monclient: _finish_hunting 0 2024-09-16T03:55:29.314 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.307+0000 7f0c6135e640 1 monclient: found mon.noname-b 2024-09-16T03:55:29.314 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.307+0000 7f0c6135e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:55:29.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.307+0000 7f0c6135e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T03:55:29.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.307+0000 7f0c6135e640 1 -- 172.21.15.42:0/604970377 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c5c06dd60 con 0x7f0c5c0a39b0 2024-09-16T03:55:29.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.307+0000 7f0c61b5f640 10 monclient: get_auth_request con 0x7f0c5c137540 auth_method 0 2024-09-16T03:55:29.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.307+0000 7f0c61b5f640 1 --2- 172.21.15.42:0/604970377 >> v2:172.21.15.154:3300/0 conn(0x7f0c5c137540 0x7f0c5c137950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T03:55:29.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.307+0000 7f0c6135e640 1 --2- 172.21.15.42:0/604970377 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0c5c0a39b0 0x7f0c5c133b80 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f0c5c0a56c0 tx=0x7f0c5801c3e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4a7d122808730a91 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:55:29.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.307+0000 7f0c53fff640 1 -- 172.21.15.42:0/604970377 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c580b0070 con 0x7f0c5c0a39b0 2024-09-16T03:55:29.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.307+0000 7f0c53fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:55:29.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.307+0000 7f0c53fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:55:29.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.307+0000 7f0c53fff640 10 monclient: dump: 2024-09-16T03:55:29.322 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:55:29.322 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:55:29.322 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:29.322 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:29.322 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:55:29.322 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:55:29.322 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:55:29.322 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:55:29.322 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:55:29.322 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:55:29.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.307+0000 7f0c53fff640 10 monclient: _finish_auth 0 2024-09-16T03:55:29.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.307+0000 7f0c53fff640 10 monclient: _check_auth_tickets 2024-09-16T03:55:29.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.307+0000 7f0c53fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:55:29.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.307+0000 7f0c53fff640 1 -- 172.21.15.42:0/604970377 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c580ab600 con 0x7f0c5c0a39b0 2024-09-16T03:55:29.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.307+0000 7f0c635c2640 5 monclient: authenticate success, global_id 854207 2024-09-16T03:55:29.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.307+0000 7f0c635c2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:55:29.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.307+0000 7f0c53fff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:55:29.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.307+0000 7f0c635c2640 10 monclient: get_monmap_and_config success 2024-09-16T03:55:29.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.307+0000 7f0c635c2640 4 set_mon_vals no callback set 2024-09-16T03:55:29.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.307+0000 7f0c53fff640 1 -- 172.21.15.42:0/604970377 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c580ab9d0 con 0x7f0c5c0a39b0 2024-09-16T03:55:29.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.307+0000 7f0c53fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:55:29.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.307+0000 7f0c53fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:55:29.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.307+0000 7f0c53fff640 10 monclient: dump: 2024-09-16T03:55:29.323 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:55:29.324 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:55:29.324 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:29.324 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:29.324 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:55:29.324 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:55:29.324 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:55:29.324 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:55:29.324 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:55:29.324 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:55:29.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.307+0000 7f0c635c2640 10 monclient: shutdown 2024-09-16T03:55:29.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.307+0000 7f0c635c2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:55:29.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.307+0000 7f0c635c2640 1 -- 172.21.15.42:0/604970377 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0c5c0a39b0 msgr2=0x7f0c5c133b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:55:29.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.307+0000 7f0c635c2640 1 --2- 172.21.15.42:0/604970377 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0c5c0a39b0 0x7f0c5c133b80 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f0c5c0a56c0 tx=0x7f0c5801c3e0 comp rx=0 tx=0).stop 2024-09-16T03:55:29.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.307+0000 7f0c635c2640 1 -- 172.21.15.42:0/604970377 shutdown_connections 2024-09-16T03:55:29.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.307+0000 7f0c635c2640 1 --2- 172.21.15.42:0/604970377 >> v2:172.21.15.154:3300/0 conn(0x7f0c5c137540 0x7f0c5c137950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:55:29.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.307+0000 7f0c635c2640 1 --2- 172.21.15.42:0/604970377 >> v2:172.21.15.42:3300/0 conn(0x7f0c5c0a3d90 0x7f0c5c137000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:55:29.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.307+0000 7f0c635c2640 1 --2- 172.21.15.42:0/604970377 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0c5c0a39b0 0x7f0c5c133b80 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:55:29.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.307+0000 7f0c635c2640 1 -- 172.21.15.42:0/604970377 >> 172.21.15.42:0/604970377 conn(0x7f0c5c0a9fb0 msgr2=0x7f0c5c0aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:55:29.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.311+0000 7f0c635c2640 1 -- 172.21.15.42:0/604970377 shutdown_connections 2024-09-16T03:55:29.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.311+0000 7f0c635c2640 1 -- 172.21.15.42:0/604970377 wait complete. 2024-09-16T03:55:29.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.311+0000 7f0c635c2640 10 monclient: build_initial_monmap 2024-09-16T03:55:29.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.311+0000 7f0c635c2640 10 monclient: monmap: 2024-09-16T03:55:29.326 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:55:29.326 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:55:29.326 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:55:29.326 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:55:29.326 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:55:29.326 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:55:29.326 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:55:29.326 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:55:29.326 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:55:29.326 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:55:29.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.311+0000 7f0c635c2640 1 librados: starting msgr at 2024-09-16T03:55:29.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.311+0000 7f0c635c2640 1 librados: starting objecter 2024-09-16T03:55:29.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.311+0000 7f0c635c2640 1 Processor -- start 2024-09-16T03:55:29.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.311+0000 7f0c635c2640 1 -- start start 2024-09-16T03:55:29.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.311+0000 7f0c635c2640 1 librados: setting wanted keys 2024-09-16T03:55:29.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.311+0000 7f0c635c2640 1 librados: calling monclient init 2024-09-16T03:55:29.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.311+0000 7f0c635c2640 10 monclient: init 2024-09-16T03:55:29.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.311+0000 7f0c635c2640 10 monclient: _reopen_session rank -1 2024-09-16T03:55:29.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.311+0000 7f0c635c2640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T03:55:29.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.311+0000 7f0c635c2640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0c5c0a39b0 0x7f0c5c1cf2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:55:29.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.311+0000 7f0c635c2640 10 monclient(hunting): picked mon.noname-b con 0x7f0c5c0a39b0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:55:29.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.311+0000 7f0c635c2640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0c5c0a3d90 0x7f0c5c1cf820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:55:29.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.311+0000 7f0c635c2640 10 monclient(hunting): picked mon.noname-c con 0x7f0c5c0a3d90 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:55:29.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.311+0000 7f0c635c2640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0c5c137540 0x7f0c5c1cfd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:55:29.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.311+0000 7f0c635c2640 10 monclient(hunting): picked mon.noname-a con 0x7f0c5c137540 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:55:29.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.311+0000 7f0c635c2640 10 monclient(hunting): start opening mon connection 2024-09-16T03:55:29.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.311+0000 7f0c635c2640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f0c5c140dc0 con 0x7f0c5c137540 2024-09-16T03:55:29.329 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.311+0000 7f0c635c2640 10 monclient(hunting): start opening mon connection 2024-09-16T03:55:29.329 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.311+0000 7f0c635c2640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f0c5c140f40 con 0x7f0c5c0a39b0 2024-09-16T03:55:29.329 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.311+0000 7f0c635c2640 10 monclient(hunting): start opening mon connection 2024-09-16T03:55:29.329 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.311+0000 7f0c635c2640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f0c5c1410c0 con 0x7f0c5c0a3d90 2024-09-16T03:55:29.329 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.311+0000 7f0c635c2640 10 monclient(hunting): _renew_subs 2024-09-16T03:55:29.329 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.311+0000 7f0c635c2640 10 monclient(hunting): authenticate will time out at 1776.281633s 2024-09-16T03:55:29.329 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.311+0000 7f0c60b5d640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0c5c0a3d90 0x7f0c5c1cf820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:55:29.329 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.311+0000 7f0c61b5f640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0c5c137540 0x7f0c5c1cfd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:55:29.329 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.311+0000 7f0c60b5d640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0c5c0a3d90 0x7f0c5c1cf820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:51714/0 (socket says 172.21.15.42:51714) 2024-09-16T03:55:29.329 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.311+0000 7f0c60b5d640 1 -- 172.21.15.42:0/1885030380 learned_addr learned my addr 172.21.15.42:0/1885030380 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:55:29.329 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.311+0000 7f0c60b5d640 10 monclient(hunting): get_auth_request con 0x7f0c5c0a3d90 auth_method 0 2024-09-16T03:55:29.329 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.311+0000 7f0c60b5d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:55:29.329 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.311+0000 7f0c60b5d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:55:29.329 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.311+0000 7f0c60b5d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:55:29.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.311+0000 7f0c61b5f640 10 monclient(hunting): get_auth_request con 0x7f0c5c137540 auth_method 0 2024-09-16T03:55:29.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.311+0000 7f0c61b5f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:55:29.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.311+0000 7f0c61b5f640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:55:29.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.311+0000 7f0c61b5f640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:55:29.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.311+0000 7f0c60b5d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:55:29.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.311+0000 7f0c60b5d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:55:29.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.311+0000 7f0c60b5d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:55:29.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.311+0000 7f0c61b5f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:55:29.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.311+0000 7f0c61b5f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:55:29.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.311+0000 7f0c61b5f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:55:29.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.311+0000 7f0c6135e640 1 --2- 172.21.15.42:0/1885030380 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0c5c0a39b0 0x7f0c5c1cf2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:55:29.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.311+0000 7f0c61b5f640 10 monclient(hunting): handle_auth_done global_id 824586 payload 824 2024-09-16T03:55:29.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.311+0000 7f0c61b5f640 1 -- 172.21.15.42:0/1885030380 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0c5c0a3d90 msgr2=0x7f0c5c1cf820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:55:29.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.311+0000 7f0c61b5f640 1 --2- 172.21.15.42:0/1885030380 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0c5c0a3d90 0x7f0c5c1cf820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:55:29.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.311+0000 7f0c61b5f640 1 -- 172.21.15.42:0/1885030380 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0c5c0a39b0 msgr2=0x7f0c5c1cf2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:55:29.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.311+0000 7f0c61b5f640 1 --2- 172.21.15.42:0/1885030380 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0c5c0a39b0 0x7f0c5c1cf2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:55:29.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.311+0000 7f0c61b5f640 10 monclient: _finish_hunting 0 2024-09-16T03:55:29.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.311+0000 7f0c61b5f640 1 monclient: found mon.noname-a 2024-09-16T03:55:29.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.311+0000 7f0c61b5f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:55:29.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.311+0000 7f0c61b5f640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:55:29.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.311+0000 7f0c61b5f640 1 -- 172.21.15.42:0/1885030380 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c5c13de10 con 0x7f0c5c137540 2024-09-16T03:55:29.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.311+0000 7f0c6135e640 10 monclient: get_auth_request con 0x7f0c5c0a39b0 auth_method 0 2024-09-16T03:55:29.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.311+0000 7f0c60b5d640 1 --2- 172.21.15.42:0/1885030380 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0c5c0a3d90 0x7f0c5c1cf820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:55:29.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.311+0000 7f0c6135e640 1 --2- 172.21.15.42:0/1885030380 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0c5c0a39b0 0x7f0c5c1cf2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T03:55:29.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.311+0000 7f0c61b5f640 1 --2- 172.21.15.42:0/1885030380 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0c5c137540 0x7f0c5c1cfd60 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f0c5400cdf0 tx=0x7f0c5400a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:55:29.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.311+0000 7f0c51ffb640 10 client.?.objecter ms_handle_connect 0x7f0c5c137540 2024-09-16T03:55:29.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.311+0000 7f0c51ffb640 10 client.?.objecter resend_mon_ops 2024-09-16T03:55:29.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.311+0000 7f0c51ffb640 1 -- 172.21.15.42:0/1885030380 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c5401a070 con 0x7f0c5c137540 2024-09-16T03:55:29.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.311+0000 7f0c51ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:55:29.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.311+0000 7f0c51ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:55:29.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.311+0000 7f0c51ffb640 10 monclient: dump: 2024-09-16T03:55:29.332 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:55:29.332 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:55:29.332 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:29.332 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:29.332 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:55:29.332 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:55:29.332 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:55:29.332 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:55:29.332 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:55:29.332 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:55:29.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.311+0000 7f0c51ffb640 10 monclient: _finish_auth 0 2024-09-16T03:55:29.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.311+0000 7f0c51ffb640 10 monclient: _check_auth_tickets 2024-09-16T03:55:29.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.311+0000 7f0c51ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:55:29.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.311+0000 7f0c51ffb640 1 -- 172.21.15.42:0/1885030380 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c5400d840 con 0x7f0c5c137540 2024-09-16T03:55:29.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.311+0000 7f0c51ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T03:55:29.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.311+0000 7f0c51ffb640 1 -- 172.21.15.42:0/1885030380 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c5400dc30 con 0x7f0c5c137540 2024-09-16T03:55:29.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.311+0000 7f0c635c2640 5 monclient: authenticate success, global_id 824586 2024-09-16T03:55:29.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.311+0000 7f0c635c2640 10 monclient: _renew_subs 2024-09-16T03:55:29.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.311+0000 7f0c635c2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:55:29.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.311+0000 7f0c635c2640 1 -- 172.21.15.42:0/1885030380 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c5c09f910 con 0x7f0c5c137540 2024-09-16T03:55:29.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.311+0000 7f0c537fe640 4 set_mon_vals no callback set 2024-09-16T03:55:29.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.311+0000 7f0c51ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:55:29.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.311+0000 7f0c51ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:55:29.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.311+0000 7f0c51ffb640 10 monclient: dump: 2024-09-16T03:55:29.334 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:55:29.334 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:55:29.334 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:29.334 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:29.334 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:55:29.334 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:55:29.334 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:55:29.334 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:55:29.334 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:55:29.334 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:55:29.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.311+0000 7f0c635c2640 10 client.824586.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:55:29.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.311+0000 7f0c635c2640 10 monclient: _renew_subs 2024-09-16T03:55:29.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.311+0000 7f0c635c2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:55:29.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.311+0000 7f0c635c2640 1 -- 172.21.15.42:0/1885030380 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c5c0a4e20 con 0x7f0c5c137540 2024-09-16T03:55:29.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.311+0000 7f0c635c2640 1 librados: init done 2024-09-16T03:55:29.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.311+0000 7f0c635c2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:55:29.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.311+0000 7f0c635c2640 1 --2- 172.21.15.42:0/1885030380 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0c5c09d590 0x7f0c5c09d970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:55:29.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.311+0000 7f0c635c2640 10 monclient(hunting): start opening mon connection 2024-09-16T03:55:29.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.311+0000 7f0c635c2640 1 -- 172.21.15.42:0/1885030380 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f0c5c1342d0 con 0x7f0c5c09d590 2024-09-16T03:55:29.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.311+0000 7f0c51ffb640 1 -- 172.21.15.42:0/1885030380 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f0c5401d070 con 0x7f0c5c137540 2024-09-16T03:55:29.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.311+0000 7f0c51ffb640 1 --2- 172.21.15.42:0/1885030380 >> v2:172.21.15.154:6812/254845342 conn(0x7f0c40037810 0x7f0c40039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:55:29.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.311+0000 7f0c6135e640 1 --2- 172.21.15.42:0/1885030380 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0c5c09d590 0x7f0c5c09d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:55:29.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.311+0000 7f0c51ffb640 1 -- 172.21.15.42:0/1885030380 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(415..415 src has 330..415) ==== 5328+0+0 (secure 0 0 0) 0x7f0c54049480 con 0x7f0c5c137540 2024-09-16T03:55:29.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.311+0000 7f0c51ffb640 10 client.824586.objecter ms_dispatch 0x7f0c5c000ba0 osd_map(415..415 src has 330..415) 2024-09-16T03:55:29.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.311+0000 7f0c51ffb640 3 client.824586.objecter handle_osd_map got epochs [415,415] > 0 2024-09-16T03:55:29.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.311+0000 7f0c51ffb640 3 client.824586.objecter handle_osd_map decoding full epoch 415 2024-09-16T03:55:29.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.311+0000 7f0c6135e640 10 monclient: get_auth_request con 0x7f0c5c09d590 auth_method 0 2024-09-16T03:55:29.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.311+0000 7f0c6135e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:55:29.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.311+0000 7f0c6135e640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:55:29.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.311+0000 7f0c6135e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:55:29.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.311+0000 7f0c51ffb640 20 client.824586.objecter dump_active .. 0 homeless 2024-09-16T03:55:29.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.315+0000 7f0c6135e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:55:29.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.315+0000 7f0c6135e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:55:29.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.315+0000 7f0c6135e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:55:29.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.315+0000 7f0c60b5d640 1 --2- 172.21.15.42:0/1885030380 >> v2:172.21.15.154:6812/254845342 conn(0x7f0c40037810 0x7f0c40039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:55:29.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.315+0000 7f0c60b5d640 10 monclient: get_auth_request con 0x7f0c40037810 auth_method 0 2024-09-16T03:55:29.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.315+0000 7f0c6135e640 10 monclient(hunting): handle_auth_done global_id 864112 payload 824 2024-09-16T03:55:29.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.315+0000 7f0c6135e640 1 -- 172.21.15.42:0/1885030380 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0c5c137000 con 0x7f0c5c09d590 2024-09-16T03:55:29.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.315+0000 7f0c6135e640 1 --2- 172.21.15.42:0/1885030380 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0c5c09d590 0x7f0c5c09d970 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f0c5c06e3c0 tx=0x7f0c5801ac20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:55:29.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.315+0000 7f0c51ffb640 10 client.824586.objecter ms_handle_connect 0x7f0c5c09d590 2024-09-16T03:55:29.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.315+0000 7f0c51ffb640 10 client.824586.objecter resend_mon_ops 2024-09-16T03:55:29.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.315+0000 7f0c51ffb640 10 client.824586.objecter ms_handle_connect 0x7f0c40037810 2024-09-16T03:55:29.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.315+0000 7f0c60b5d640 1 --2- 172.21.15.42:0/1885030380 >> v2:172.21.15.154:6812/254845342 conn(0x7f0c40037810 0x7f0c40039cd0 secure :-1 s=READY pgs=3635 cs=0 l=1 rev1=1 crypto rx=0x7f0c5c1315f0 tx=0x7f0c4c009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:55:29.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.315+0000 7f0c51ffb640 1 -- 172.21.15.42:0/1885030380 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c580ba030 con 0x7f0c5c09d590 2024-09-16T03:55:29.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.315+0000 7f0c51ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:55:29.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.315+0000 7f0c51ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:55:29.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.315+0000 7f0c51ffb640 10 monclient: dump: 2024-09-16T03:55:29.337 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:55:29.337 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:55:29.337 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:29.337 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:29.337 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:55:29.337 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:55:29.337 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:55:29.337 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:55:29.337 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:55:29.338 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:55:29.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.315+0000 7f0c51ffb640 1 -- 172.21.15.42:0/1885030380 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f0c58009490 con 0x7f0c5c09d590 2024-09-16T03:55:29.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.315+0000 7f0c51ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:55:29.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.315+0000 7f0c51ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:55:29.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.315+0000 7f0c51ffb640 1 -- 172.21.15.42:0/1885030380 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0c5c09d590 msgr2=0x7f0c5c09d970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:55:29.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.315+0000 7f0c51ffb640 1 --2- 172.21.15.42:0/1885030380 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0c5c09d590 0x7f0c5c09d970 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f0c5c06e3c0 tx=0x7f0c5801ac20 comp rx=0 tx=0).stop 2024-09-16T03:55:29.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.315+0000 7f0c51ffb640 1 -- 172.21.15.42:0/1885030380 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0c5c09d590 msgr2=0x7f0c5c09d970 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:55:29.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.315+0000 7f0c51ffb640 1 --2- 172.21.15.42:0/1885030380 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0c5c09d590 0x7f0c5c09d970 secure :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f0c5c06e3c0 tx=0x7f0c5801ac20 comp rx=0 tx=0).stop 2024-09-16T03:55:29.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.319+0000 7f0c635c2640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:55:29.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.319+0000 7f0c635c2640 1 --2- 172.21.15.42:0/1885030380 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0c5c02c650 0x7f0c5c02ca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:55:29.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.319+0000 7f0c635c2640 10 monclient(hunting): start opening mon connection 2024-09-16T03:55:29.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.319+0000 7f0c635c2640 1 -- 172.21.15.42:0/1885030380 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f0c5c1397e0 con 0x7f0c5c02c650 2024-09-16T03:55:29.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.323+0000 7f0c6135e640 1 --2- 172.21.15.42:0/1885030380 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0c5c02c650 0x7f0c5c02ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:55:29.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.323+0000 7f0c6135e640 10 monclient: get_auth_request con 0x7f0c5c02c650 auth_method 0 2024-09-16T03:55:29.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.323+0000 7f0c6135e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:55:29.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.323+0000 7f0c6135e640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:55:29.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.323+0000 7f0c6135e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:55:29.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.323+0000 7f0c6135e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:55:29.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.323+0000 7f0c6135e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:55:29.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.323+0000 7f0c6135e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:55:29.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.323+0000 7f0c6135e640 10 monclient(hunting): handle_auth_done global_id 864115 payload 824 2024-09-16T03:55:29.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.323+0000 7f0c6135e640 1 -- 172.21.15.42:0/1885030380 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f0c5c068e40 con 0x7f0c5c02c650 2024-09-16T03:55:29.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.323+0000 7f0c51ffb640 10 client.824586.objecter ms_handle_connect 0x7f0c5c02c650 2024-09-16T03:55:29.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.323+0000 7f0c6135e640 1 --2- 172.21.15.42:0/1885030380 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0c5c02c650 0x7f0c5c02ca60 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f0c5801aa90 tx=0x7f0c58019180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:55:29.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.323+0000 7f0c51ffb640 10 client.824586.objecter resend_mon_ops 2024-09-16T03:55:29.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.323+0000 7f0c51ffb640 1 -- 172.21.15.42:0/1885030380 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c580c6070 con 0x7f0c5c02c650 2024-09-16T03:55:29.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.323+0000 7f0c51ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:55:29.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.323+0000 7f0c51ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:55:29.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.323+0000 7f0c51ffb640 10 monclient: dump: 2024-09-16T03:55:29.342 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:55:29.342 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:55:29.342 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:29.342 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:29.342 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:55:29.342 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:55:29.342 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:55:29.342 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:55:29.342 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:55:29.342 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:55:29.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.323+0000 7f0c51ffb640 1 -- 172.21.15.42:0/1885030380 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4544 (secure 0 0 0) 0x7f0c580b6de0 con 0x7f0c5c02c650 2024-09-16T03:55:29.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.323+0000 7f0c51ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:55:29.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.323+0000 7f0c51ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:55:29.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.323+0000 7f0c51ffb640 1 -- 172.21.15.42:0/1885030380 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0c5c02c650 msgr2=0x7f0c5c02ca60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:55:29.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.323+0000 7f0c51ffb640 1 --2- 172.21.15.42:0/1885030380 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0c5c02c650 0x7f0c5c02ca60 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f0c5801aa90 tx=0x7f0c58019180 comp rx=0 tx=0).stop 2024-09-16T03:55:29.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.323+0000 7f0c51ffb640 1 -- 172.21.15.42:0/1885030380 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0c5c02c650 msgr2=0x7f0c5c02ca60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:55:29.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.323+0000 7f0c51ffb640 1 --2- 172.21.15.42:0/1885030380 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0c5c02c650 0x7f0c5c02ca60 secure :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f0c5801aa90 tx=0x7f0c58019180 comp rx=0 tx=0).stop 2024-09-16T03:55:29.343 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:55:29.343 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:55:29.343 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:55:29.343 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T03:55:29.343 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 116, 2024-09-16T03:55:29.343 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 7615, 2024-09-16T03:55:29.344 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:55:29.344 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:55:29.344 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T03:55:29.344 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T03:55:29.344 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:55:29.344 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 1, 2024-09-16T03:55:29.344 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:55:29.344 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:55:29.344 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:55:29.344 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:55:29.344 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:55:29.344 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:55:29.344 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:55:29.344 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:55:29.345 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:55:29.345 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:55:29.345 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:55:29.345 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:55:29.345 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:55:29.345 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:55:29.345 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:55:29.345 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:55:29.345 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:55:29.345 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:55:29.345 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:55:29.345 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:55:29.345 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:55:29.345 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:55:29.346 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:55:29.346 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:55:29.346 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:55:29.346 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:55:29.346 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:55:29.346 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:55:29.346 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:55:29.346 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:55:29.346 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:55:29.346 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:55:29.346 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:55:29.346 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:55:29.346 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:55:29.346 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:55:29.347 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:55:29.347 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:55:29.347 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:55:29.347 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:55:29.347 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:55:29.347 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:55:29.347 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:55:29.347 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:55:29.347 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:55:29.347 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:55:29.347 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:55:29.347 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:55:29.347 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:55:29.347 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:55:29.347 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:55:29.348 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:55:29.348 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:55:29.348 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:55:29.348 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:55:29.348 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:55:29.348 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:55:29.348 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:55:29.348 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:55:29.348 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:55:29.348 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:55:29.348 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:55:29.348 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:55:29.348 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:55:29.349 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:55:29.349 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:55:29.349 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:55:29.349 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:55:29.349 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:55:29.349 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:55:29.349 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:55:29.349 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:55:29.349 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:55:29.349 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:55:29.349 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:55:29.349 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:55:29.349 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:55:29.349 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:55:29.350 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:55:29.350 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:55:29.350 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:55:29.350 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:55:29.350 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:55:29.350 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:55:29.350 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:55:29.350 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:55:29.350 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:55:29.350 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:55:29.350 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:55:29.350 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:55:29.350 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:55:29.350 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:55:29.350 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:55:29.351 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:55:29.351 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:55:29.351 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:55:29.351 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:55:29.351 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:55:29.351 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:55:29.351 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:55:29.351 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:55:29.351 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:55:29.351 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:55:29.351 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:55:29.351 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:55:29.351 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:55:29.351 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:55:29.352 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:55:29.352 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:55:29.352 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:55:29.352 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:55:29.352 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:55:29.352 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:55:29.352 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:55:29.352 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:55:29.352 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:55:29.352 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:55:29.352 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:55:29.352 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:55:29.352 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:55:29.352 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:55:29.353 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:55:29.353 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:55:29.353 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:55:29.353 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:55:29.353 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:55:29.353 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:55:29.353 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:55:29.353 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:55:29.353 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:55:29.353 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:55:29.353 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:55:29.353 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:55:29.353 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:55:29.353 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:55:29.353 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:55:29.354 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:55:29.354 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:55:29.354 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:55:29.354 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:55:29.354 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:55:29.354 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:55:29.354 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:55:29.354 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:55:29.354 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:55:29.354 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:55:29.354 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:55:29.354 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:55:29.354 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:55:29.354 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:55:29.355 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:55:29.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.323+0000 7f0c635c2640 10 librados: watch_flush enter 2024-09-16T03:55:29.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.323+0000 7f0c635c2640 10 librados: watch_flush exit 2024-09-16T03:55:29.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.323+0000 7f0c635c2640 20 client.824586.objecter shutdown clearing up homeless session... 2024-09-16T03:55:29.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.323+0000 7f0c635c2640 10 client.824586.objecter successfully canceled tick 2024-09-16T03:55:29.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.323+0000 7f0c635c2640 1 -- 172.21.15.42:0/1885030380 >> v2:172.21.15.154:6812/254845342 conn(0x7f0c40037810 msgr2=0x7f0c40039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:55:29.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.323+0000 7f0c635c2640 1 --2- 172.21.15.42:0/1885030380 >> v2:172.21.15.154:6812/254845342 conn(0x7f0c40037810 0x7f0c40039cd0 secure :-1 s=READY pgs=3635 cs=0 l=1 rev1=1 crypto rx=0x7f0c5c1315f0 tx=0x7f0c4c009190 comp rx=0 tx=0).stop 2024-09-16T03:55:29.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.323+0000 7f0c635c2640 10 monclient: shutdown 2024-09-16T03:55:29.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.323+0000 7f0c635c2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:55:29.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.323+0000 7f0c635c2640 1 -- 172.21.15.42:0/1885030380 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0c5c137540 msgr2=0x7f0c5c1cfd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:55:29.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.323+0000 7f0c635c2640 1 --2- 172.21.15.42:0/1885030380 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0c5c137540 0x7f0c5c1cfd60 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f0c5400cdf0 tx=0x7f0c5400a710 comp rx=0 tx=0).stop 2024-09-16T03:55:29.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.323+0000 7f0c61b5f640 1 -- 172.21.15.42:0/1885030380 reap_dead start 2024-09-16T03:55:29.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.323+0000 7f0c635c2640 1 -- 172.21.15.42:0/1885030380 shutdown_connections 2024-09-16T03:55:29.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.323+0000 7f0c635c2640 1 --2- 172.21.15.42:0/1885030380 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0c5c137540 0x7f0c5c1cfd60 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:55:29.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.323+0000 7f0c635c2640 1 -- 172.21.15.42:0/1885030380 >> 172.21.15.42:0/1885030380 conn(0x7f0c5c0a9fb0 msgr2=0x7f0c5c0a8ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:55:29.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.323+0000 7f0c635c2640 1 -- 172.21.15.42:0/1885030380 shutdown_connections 2024-09-16T03:55:29.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.323+0000 7f0c635c2640 1 -- 172.21.15.42:0/1885030380 wait complete. 2024-09-16T03:55:29.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.323+0000 7f0c635c2640 1 librados: shutdown 2024-09-16T03:55:29.357 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:55:29.358 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T03:55:29.358 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T03:55:29.481 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.475+0000 7faeb7f88640 10 monclient: get_monmap_and_config 2024-09-16T03:55:29.481 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.475+0000 7faeb7f88640 10 monclient: build_initial_monmap 2024-09-16T03:55:29.481 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.475+0000 7faeb7f88640 10 monclient: monmap: 2024-09-16T03:55:29.481 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:55:29.481 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:55:29.481 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:55:29.479668+0000 2024-09-16T03:55:29.481 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:55:29.479668+0000 2024-09-16T03:55:29.481 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:55:29.481 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:55:29.482 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:55:29.482 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:55:29.482 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:55:29.482 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:55:29.482 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:55:29.482 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:55:29.484 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.479+0000 7faeb7f88640 1 Processor -- start 2024-09-16T03:55:29.484 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.479+0000 7faeb7f88640 1 -- start start 2024-09-16T03:55:29.484 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.479+0000 7faeb7f88640 10 monclient: init 2024-09-16T03:55:29.484 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.479+0000 7faeb7f88640 10 monclient: _reopen_session rank -1 2024-09-16T03:55:29.485 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.479+0000 7faeb7f88640 10 monclient: _add_conns ranks=[1,3,4,0,2] 2024-09-16T03:55:29.485 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.479+0000 7faeb7f88640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7faeb00a2340 addr v1:172.21.15.154:6789/0 2024-09-16T03:55:29.485 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.479+0000 7faeb7f88640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7faeb009fe60 0x7faeb013c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:55:29.485 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.479+0000 7faeb7f88640 10 monclient(hunting): picked mon.noname-a con 0x7faeb009fe60 addr v2:172.21.15.42:3300/0 2024-09-16T03:55:29.485 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.479+0000 7faeb7f88640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7faeb00a2e30 0x7faeb013eac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:55:29.485 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.479+0000 7faeb7f88640 10 monclient(hunting): picked mon.noname-c con 0x7faeb00a2e30 addr v2:172.21.15.154:3300/0 2024-09-16T03:55:29.485 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.479+0000 7faeb7f88640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faeb00a0d70 con 0x7faeb00a2340 2024-09-16T03:55:29.485 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.479+0000 7faeb7f88640 10 monclient(hunting): start opening mon connection 2024-09-16T03:55:29.485 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.479+0000 7faeb7f88640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7faeb00726e0 con 0x7faeb009fe60 2024-09-16T03:55:29.485 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.479+0000 7faeb7f88640 10 monclient(hunting): start opening mon connection 2024-09-16T03:55:29.485 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.479+0000 7faeb7f88640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7faeb006cae0 con 0x7faeb00a2e30 2024-09-16T03:55:29.485 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.479+0000 7faeb7f88640 10 monclient(hunting): _renew_subs 2024-09-16T03:55:29.485 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.479+0000 7faeb7f88640 10 monclient(hunting): authenticate will time out at 1776.449439s 2024-09-16T03:55:29.486 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.479+0000 7faeb5523640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7faeb009fe60 0x7faeb013c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:55:29.486 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.479+0000 7faeb5523640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7faeb009fe60 0x7faeb013c570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:34808/0 (socket says 172.21.15.42:34808) 2024-09-16T03:55:29.486 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.479+0000 7faeb5523640 1 -- 172.21.15.42:0/2950490247 learned_addr learned my addr 172.21.15.42:0/2950490247 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:55:29.486 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.479+0000 7faeb5523640 10 monclient(hunting): get_auth_request con 0x7faeb009fe60 auth_method 0 2024-09-16T03:55:29.486 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.479+0000 7faeb5523640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:55:29.486 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.479+0000 7faeb5523640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:55:29.486 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.479+0000 7faeb5523640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:55:29.486 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.479+0000 7faeb5523640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:55:29.486 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.479+0000 7faeb5523640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:55:29.486 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.479+0000 7faeb5523640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:55:29.486 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.479+0000 7faeb5523640 10 monclient(hunting): handle_auth_done global_id 824589 payload 274 2024-09-16T03:55:29.486 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.479+0000 7faeb5523640 1 -- 172.21.15.42:0/2950490247 >> v2:172.21.15.154:3300/0 conn(0x7faeb00a2e30 msgr2=0x7faeb013eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:55:29.486 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.479+0000 7faeb5523640 1 --2- 172.21.15.42:0/2950490247 >> v2:172.21.15.154:3300/0 conn(0x7faeb00a2e30 0x7faeb013eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:55:29.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.479+0000 7faeb5523640 1 -- 172.21.15.42:0/2950490247 >> v1:172.21.15.154:6789/0 conn(0x7faeb00a2340 legacy=0x7faeb00a2720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:55:29.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.479+0000 7faeb5523640 10 monclient: _finish_hunting 0 2024-09-16T03:55:29.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.479+0000 7faeb5523640 1 monclient: found mon.noname-a 2024-09-16T03:55:29.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.479+0000 7faeb5523640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:55:29.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.479+0000 7faeb5523640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:55:29.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.479+0000 7faeb5523640 1 -- 172.21.15.42:0/2950490247 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faeb006c350 con 0x7faeb009fe60 2024-09-16T03:55:29.493 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.483+0000 7faeb5523640 1 --2- 172.21.15.42:0/2950490247 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faeb009fe60 0x7faeb013c570 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7faeb00a0840 tx=0x7faea001c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4ac5e216e0120b3 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:55:29.493 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.483+0000 7faeb4d22640 1 -- 172.21.15.42:0/2950490247 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faea00b0070 con 0x7faeb009fe60 2024-09-16T03:55:29.493 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.483+0000 7faeb4d22640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:55:29.493 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.483+0000 7faeb4d22640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:55:29.493 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.483+0000 7faeb4d22640 10 monclient: dump: 2024-09-16T03:55:29.493 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:55:29.493 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:55:29.493 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:29.493 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:29.494 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:55:29.494 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:55:29.494 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:55:29.494 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:55:29.494 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:55:29.494 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:55:29.494 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.483+0000 7faeb4d22640 10 monclient: _finish_auth 0 2024-09-16T03:55:29.494 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.483+0000 7faeb4d22640 10 monclient: _check_auth_tickets 2024-09-16T03:55:29.494 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.483+0000 7faeb4d22640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:55:29.494 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.483+0000 7faeb4d22640 1 -- 172.21.15.42:0/2950490247 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faea00ab600 con 0x7faeb009fe60 2024-09-16T03:55:29.494 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.483+0000 7faeb4d22640 10 monclient: handle_config config(0 keys) 2024-09-16T03:55:29.494 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.483+0000 7faeb4d22640 1 -- 172.21.15.42:0/2950490247 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faea00ab9d0 con 0x7faeb009fe60 2024-09-16T03:55:29.494 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.483+0000 7faeb4d22640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:55:29.495 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.483+0000 7faeb4d22640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:55:29.495 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.483+0000 7faeb4d22640 10 monclient: dump: 2024-09-16T03:55:29.495 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:55:29.495 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:55:29.495 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:29.495 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:29.495 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:55:29.495 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:55:29.495 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:55:29.496 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:55:29.496 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:55:29.496 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:55:29.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.483+0000 7faeb7f88640 5 monclient: authenticate success, global_id 824589 2024-09-16T03:55:29.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.483+0000 7faeb7f88640 10 monclient: get_monmap_and_config success 2024-09-16T03:55:29.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.483+0000 7faeb7f88640 4 set_mon_vals no callback set 2024-09-16T03:55:29.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.483+0000 7faeb7f88640 10 monclient: shutdown 2024-09-16T03:55:29.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.483+0000 7faeb7f88640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:55:29.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.483+0000 7faeb7f88640 1 -- 172.21.15.42:0/2950490247 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faeb009fe60 msgr2=0x7faeb013c570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:55:29.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.483+0000 7faeb7f88640 1 --2- 172.21.15.42:0/2950490247 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faeb009fe60 0x7faeb013c570 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7faeb00a0840 tx=0x7faea001c3a0 comp rx=0 tx=0).stop 2024-09-16T03:55:29.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.483+0000 7faeb7f88640 1 -- 172.21.15.42:0/2950490247 shutdown_connections 2024-09-16T03:55:29.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.483+0000 7faeb7f88640 1 --2- 172.21.15.42:0/2950490247 >> v2:172.21.15.154:3300/0 conn(0x7faeb00a2e30 0x7faeb013eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:55:29.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.483+0000 7faeb7f88640 1 --2- 172.21.15.42:0/2950490247 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faeb009fe60 0x7faeb013c570 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:55:29.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.483+0000 7faeb7f88640 1 -- 172.21.15.42:0/2950490247 >> 172.21.15.42:0/2950490247 conn(0x7faeb00a3430 msgr2=0x7faeb00a3810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:55:29.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.483+0000 7faeb7f88640 1 -- 172.21.15.42:0/2950490247 shutdown_connections 2024-09-16T03:55:29.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.483+0000 7faeb7f88640 1 -- 172.21.15.42:0/2950490247 wait complete. 2024-09-16T03:55:29.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.483+0000 7faeb7f88640 10 monclient: build_initial_monmap 2024-09-16T03:55:29.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.483+0000 7faeb7f88640 10 monclient: monmap: 2024-09-16T03:55:29.497 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:55:29.497 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:55:29.497 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:55:29.497 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:55:29.497 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:55:29.497 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:55:29.497 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:55:29.498 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:55:29.498 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:55:29.498 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:55:29.498 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.483+0000 7faeb7f88640 1 librados: starting msgr at 2024-09-16T03:55:29.498 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.483+0000 7faeb7f88640 1 librados: starting objecter 2024-09-16T03:55:29.498 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.483+0000 7faeb7f88640 1 Processor -- start 2024-09-16T03:55:29.498 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.483+0000 7faeb7f88640 1 -- start start 2024-09-16T03:55:29.498 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.483+0000 7faeb7f88640 1 librados: setting wanted keys 2024-09-16T03:55:29.498 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.483+0000 7faeb7f88640 1 librados: calling monclient init 2024-09-16T03:55:29.498 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.483+0000 7faeb7f88640 10 monclient: init 2024-09-16T03:55:29.498 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.483+0000 7faeb7f88640 10 monclient: _reopen_session rank -1 2024-09-16T03:55:29.498 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.483+0000 7faeb7f88640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T03:55:29.499 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.483+0000 7faeb7f88640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faeb00a2340 0x7faeb01cf080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:55:29.499 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.483+0000 7faeb7f88640 10 monclient(hunting): picked mon.noname-b con 0x7faeb00a2340 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:55:29.499 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.483+0000 7faeb7f88640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7faeb00a2e30 0x7faeb01344d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:55:29.499 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.483+0000 7faeb7f88640 10 monclient(hunting): picked mon.noname-c con 0x7faeb00a2e30 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:55:29.499 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.483+0000 7faeb7f88640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faeb0134a10 0x7faeb013c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:55:29.499 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.483+0000 7faeb7f88640 10 monclient(hunting): picked mon.noname-a con 0x7faeb0134a10 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:55:29.499 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.483+0000 7faeb7f88640 10 monclient(hunting): start opening mon connection 2024-09-16T03:55:29.499 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.483+0000 7faeb7f88640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7faeb0141a30 con 0x7faeb0134a10 2024-09-16T03:55:29.499 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.483+0000 7faeb7f88640 10 monclient(hunting): start opening mon connection 2024-09-16T03:55:29.500 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.483+0000 7faeb7f88640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7faeb0141bb0 con 0x7faeb00a2340 2024-09-16T03:55:29.500 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.483+0000 7faeb7f88640 10 monclient(hunting): start opening mon connection 2024-09-16T03:55:29.500 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.483+0000 7faeb7f88640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7faeb0141d30 con 0x7faeb00a2e30 2024-09-16T03:55:29.500 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.483+0000 7faeb7f88640 10 monclient(hunting): _renew_subs 2024-09-16T03:55:29.500 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.483+0000 7faeb7f88640 10 monclient(hunting): authenticate will time out at 1776.455088s 2024-09-16T03:55:29.500 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.483+0000 7faeb5523640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7faeb00a2e30 0x7faeb01344d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:55:29.500 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.483+0000 7faeb6525640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faeb0134a10 0x7faeb013c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:55:29.500 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.483+0000 7faeb6525640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faeb0134a10 0x7faeb013c500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:34816/0 (socket says 172.21.15.42:34816) 2024-09-16T03:55:29.500 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.483+0000 7faeb6525640 1 -- 172.21.15.42:0/1445032279 learned_addr learned my addr 172.21.15.42:0/1445032279 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:55:29.500 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.483+0000 7faeb5523640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7faeb00a2e30 0x7faeb01344d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:51720/0 (socket says 172.21.15.42:51720) 2024-09-16T03:55:29.501 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.483+0000 7faeb6525640 10 monclient(hunting): get_auth_request con 0x7faeb0134a10 auth_method 0 2024-09-16T03:55:29.501 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.483+0000 7faeb6525640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:55:29.501 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.483+0000 7faeb6525640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:55:29.501 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.483+0000 7faeb6525640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:55:29.501 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.483+0000 7faeb5523640 10 monclient(hunting): get_auth_request con 0x7faeb00a2e30 auth_method 0 2024-09-16T03:55:29.501 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.483+0000 7faeb5523640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:55:29.501 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.483+0000 7faeb5523640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:55:29.501 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.483+0000 7faeb5523640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:55:29.501 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.483+0000 7faeb6525640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:55:29.501 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.483+0000 7faeb6525640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:55:29.501 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.483+0000 7faeb6525640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:55:29.502 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.483+0000 7faeb5d24640 1 --2- 172.21.15.42:0/1445032279 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faeb00a2340 0x7faeb01cf080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:55:29.502 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.483+0000 7faeb5523640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:55:29.502 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.483+0000 7faeb5523640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:55:29.502 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.483+0000 7faeb5523640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:55:29.502 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.483+0000 7faeb5d24640 10 monclient(hunting): get_auth_request con 0x7faeb00a2340 auth_method 0 2024-09-16T03:55:29.502 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.483+0000 7faeb5d24640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:55:29.502 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.483+0000 7faeb5d24640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:55:29.502 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.483+0000 7faeb5d24640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:55:29.502 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.483+0000 7faeb6525640 10 monclient(hunting): handle_auth_done global_id 824592 payload 824 2024-09-16T03:55:29.502 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.483+0000 7faeb6525640 1 -- 172.21.15.42:0/1445032279 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7faeb00a2e30 msgr2=0x7faeb01344d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:55:29.502 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.483+0000 7faeb6525640 1 --2- 172.21.15.42:0/1445032279 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7faeb00a2e30 0x7faeb01344d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:55:29.502 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.483+0000 7faeb6525640 1 -- 172.21.15.42:0/1445032279 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faeb00a2340 msgr2=0x7faeb01cf080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:55:29.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.483+0000 7faeb6525640 1 --2- 172.21.15.42:0/1445032279 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faeb00a2340 0x7faeb01cf080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:55:29.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.483+0000 7faeb6525640 10 monclient: _finish_hunting 0 2024-09-16T03:55:29.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.483+0000 7faeb6525640 1 monclient: found mon.noname-a 2024-09-16T03:55:29.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.483+0000 7faeb6525640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:55:29.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.483+0000 7faeb6525640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:55:29.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.483+0000 7faeb6525640 1 -- 172.21.15.42:0/1445032279 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faeb011ee60 con 0x7faeb0134a10 2024-09-16T03:55:29.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.483+0000 7faeb5d24640 1 --2- 172.21.15.42:0/1445032279 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faeb00a2340 0x7faeb01cf080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T03:55:29.631 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.483+0000 7faeb6525640 1 --2- 172.21.15.42:0/1445032279 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faeb0134a10 0x7faeb013c500 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7faea800b460 tx=0x7faea800b920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:55:29.631 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.483+0000 7faea67fc640 10 client.?.objecter ms_handle_connect 0x7faeb0134a10 2024-09-16T03:55:29.631 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.483+0000 7faea67fc640 10 client.?.objecter resend_mon_ops 2024-09-16T03:55:29.631 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.487+0000 7faea67fc640 1 -- 172.21.15.42:0/1445032279 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faea8015020 con 0x7faeb0134a10 2024-09-16T03:55:29.631 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.487+0000 7faea67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:55:29.631 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.487+0000 7faea67fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:55:29.631 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.487+0000 7faea67fc640 10 monclient: dump: 2024-09-16T03:55:29.631 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:55:29.631 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:55:29.631 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:29.632 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:29.632 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:55:29.632 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:55:29.632 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:55:29.632 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:55:29.632 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:55:29.632 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:55:29.632 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.487+0000 7faea67fc640 10 monclient: _finish_auth 0 2024-09-16T03:55:29.632 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.487+0000 7faea67fc640 10 monclient: _check_auth_tickets 2024-09-16T03:55:29.632 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.487+0000 7faea67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:55:29.633 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.487+0000 7faea67fc640 1 -- 172.21.15.42:0/1445032279 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faea8011890 con 0x7faeb0134a10 2024-09-16T03:55:29.633 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.487+0000 7faea67fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:55:29.633 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.487+0000 7faea67fc640 1 -- 172.21.15.42:0/1445032279 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faea8011c80 con 0x7faeb0134a10 2024-09-16T03:55:29.633 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.487+0000 7faeb7f88640 5 monclient: authenticate success, global_id 824592 2024-09-16T03:55:29.633 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.487+0000 7faeb7f88640 10 monclient: _renew_subs 2024-09-16T03:55:29.633 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.487+0000 7faeb7f88640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:55:29.633 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.487+0000 7faeb7f88640 1 -- 172.21.15.42:0/1445032279 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faeb013ebc0 con 0x7faeb0134a10 2024-09-16T03:55:29.633 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.487+0000 7faea7fff640 4 set_mon_vals no callback set 2024-09-16T03:55:29.633 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.487+0000 7faea67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:55:29.633 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.487+0000 7faea67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:55:29.633 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.487+0000 7faea67fc640 10 monclient: dump: 2024-09-16T03:55:29.634 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:55:29.634 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:55:29.634 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:29.634 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:29.634 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:55:29.634 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:55:29.634 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:55:29.634 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:55:29.634 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:55:29.634 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:55:29.634 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.487+0000 7faeb7f88640 10 client.824592.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:55:29.635 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.487+0000 7faeb7f88640 10 monclient: _renew_subs 2024-09-16T03:55:29.635 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.487+0000 7faeb7f88640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:55:29.635 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.487+0000 7faeb7f88640 1 -- 172.21.15.42:0/1445032279 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faeb011eb80 con 0x7faeb0134a10 2024-09-16T03:55:29.635 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.487+0000 7faeb7f88640 1 librados: init done 2024-09-16T03:55:29.635 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.487+0000 7faea67fc640 1 -- 172.21.15.42:0/1445032279 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7faea8018020 con 0x7faeb0134a10 2024-09-16T03:55:29.635 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.487+0000 7faeb7f88640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:55:29.635 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.487+0000 7faeb7f88640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:55:29.635 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.487+0000 7faeb7f88640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:55:29.636 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.487+0000 7faeb7f88640 1 -- 172.21.15.42:0/1445032279 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faeb00a2720 con 0x7faeb0134a10 2024-09-16T03:55:29.636 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.487+0000 7faea67fc640 1 --2- 172.21.15.42:0/1445032279 >> v2:172.21.15.154:6812/254845342 conn(0x7fae94037810 0x7fae94039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:55:29.636 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.487+0000 7faea67fc640 1 -- 172.21.15.42:0/1445032279 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(416..416 src has 330..416) ==== 5700+0+0 (secure 0 0 0) 0x7faea8048fa0 con 0x7faeb0134a10 2024-09-16T03:55:29.636 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.487+0000 7faea67fc640 10 client.824592.objecter ms_dispatch 0x7faeb0000ba0 osd_map(416..416 src has 330..416) 2024-09-16T03:55:29.636 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.487+0000 7faea67fc640 3 client.824592.objecter handle_osd_map got epochs [416,416] > 0 2024-09-16T03:55:29.636 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.487+0000 7faea67fc640 3 client.824592.objecter handle_osd_map decoding full epoch 416 2024-09-16T03:55:29.636 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.487+0000 7faea67fc640 20 client.824592.objecter dump_active .. 0 homeless 2024-09-16T03:55:29.636 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.487+0000 7faeb5d24640 1 --2- 172.21.15.42:0/1445032279 >> v2:172.21.15.154:6812/254845342 conn(0x7fae94037810 0x7fae94039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:55:29.636 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.487+0000 7faeb5d24640 10 monclient: get_auth_request con 0x7fae94037810 auth_method 0 2024-09-16T03:55:29.636 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.487+0000 7faea67fc640 10 client.824592.objecter ms_handle_connect 0x7fae94037810 2024-09-16T03:55:29.637 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.487+0000 7faeb5d24640 1 --2- 172.21.15.42:0/1445032279 >> v2:172.21.15.154:6812/254845342 conn(0x7fae94037810 0x7fae94039cd0 secure :-1 s=READY pgs=3636 cs=0 l=1 rev1=1 crypto rx=0x7faeac0045c0 tx=0x7faeac009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:55:29.637 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.491+0000 7faea67fc640 1 -- 172.21.15.42:0/1445032279 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7faea801f020 con 0x7faeb0134a10 2024-09-16T03:55:29.637 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.491+0000 7faea67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:55:29.637 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.491+0000 7faea67fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:55:29.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.719+0000 7faeb7f88640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T03:55:29.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.719+0000 7faeb7f88640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T03:55:29.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.719+0000 7faeb7f88640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:55:29.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.719+0000 7faeb7f88640 1 -- 172.21.15.42:0/1445032279 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7faeb006b370 con 0x7faeb0134a10 2024-09-16T03:55:29.724 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.719+0000 7faea67fc640 1 -- 172.21.15.42:0/1445032279 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7faea800d040 con 0x7faeb0134a10 2024-09-16T03:55:29.724 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.719+0000 7faea67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T03:55:29.725 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.719+0000 7faea67fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:55:29.725 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.723+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.726 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.723+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=31,config=2,health=10,kv=9,logm=48} crc {auth=2790179879,config=692619927,health=2016492843,kv=2236749221,logm=3290173030}) 2024-09-16T03:55:29.726 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.723+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=30,config=1,health=12,kv=10,logm=47} crc {auth=2226017230,config=1228700967,health=4242108353,kv=3446188498,logm=1548059080}) 2024-09-16T03:55:29.726 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.723+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.726 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.723+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=31,config=2,health=10,kv=9,logm=48} crc {auth=2790179879,config=692619927,health=2016492843,kv=2236749221,logm=3290173030}) 2024-09-16T03:55:29.726 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.723+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=31,config=2,health=11,kv=10,logm=46} crc {auth=2790179879,config=692619927,health=1602069699,kv=3446188498,logm=1029622049}) 2024-09-16T03:55:29.729 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.723+0000 7faeb7f88640 10 librados: watch_flush enter 2024-09-16T03:55:29.729 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.723+0000 7faeb7f88640 10 librados: watch_flush exit 2024-09-16T03:55:29.729 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.723+0000 7faeb7f88640 20 client.824592.objecter shutdown clearing up homeless session... 2024-09-16T03:55:29.729 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.723+0000 7faeb7f88640 10 client.824592.objecter successfully canceled tick 2024-09-16T03:55:29.729 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.723+0000 7faeb7f88640 1 -- 172.21.15.42:0/1445032279 >> v2:172.21.15.154:6812/254845342 conn(0x7fae94037810 msgr2=0x7fae94039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:55:29.729 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.723+0000 7faeb7f88640 1 --2- 172.21.15.42:0/1445032279 >> v2:172.21.15.154:6812/254845342 conn(0x7fae94037810 0x7fae94039cd0 secure :-1 s=READY pgs=3636 cs=0 l=1 rev1=1 crypto rx=0x7faeac0045c0 tx=0x7faeac009040 comp rx=0 tx=0).stop 2024-09-16T03:55:29.729 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.723+0000 7faeb7f88640 10 monclient: shutdown 2024-09-16T03:55:29.729 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.723+0000 7faeb7f88640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:55:29.729 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.723+0000 7faeb7f88640 1 -- 172.21.15.42:0/1445032279 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faeb0134a10 msgr2=0x7faeb013c500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:55:29.729 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.723+0000 7faeb7f88640 1 --2- 172.21.15.42:0/1445032279 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faeb0134a10 0x7faeb013c500 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7faea800b460 tx=0x7faea800b920 comp rx=0 tx=0).stop 2024-09-16T03:55:29.729 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.723+0000 7faeb7f88640 1 -- 172.21.15.42:0/1445032279 shutdown_connections 2024-09-16T03:55:29.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.723+0000 7faeb7f88640 1 --2- 172.21.15.42:0/1445032279 >> v2:172.21.15.154:6812/254845342 conn(0x7fae94037810 0x7fae94039cd0 unknown :-1 s=CLOSED pgs=3636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:55:29.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.723+0000 7faeb7f88640 1 --2- 172.21.15.42:0/1445032279 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faeb0134a10 0x7faeb013c500 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:55:29.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.723+0000 7faeb7f88640 1 --2- 172.21.15.42:0/1445032279 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7faeb00a2e30 0x7faeb01344d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:55:29.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.723+0000 7faeb7f88640 1 --2- 172.21.15.42:0/1445032279 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faeb00a2340 0x7faeb01cf080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:55:29.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.723+0000 7faeb7f88640 1 -- 172.21.15.42:0/1445032279 >> 172.21.15.42:0/1445032279 conn(0x7faeb00a3430 msgr2=0x7faeb0071300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:55:29.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.723+0000 7faeb7f88640 1 -- 172.21.15.42:0/1445032279 shutdown_connections 2024-09-16T03:55:29.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.723+0000 7faeb7f88640 1 -- 172.21.15.42:0/1445032279 wait complete. 2024-09-16T03:55:29.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:29.723+0000 7faeb7f88640 1 librados: shutdown 2024-09-16T03:55:29.731 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.723+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.731 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.723+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=455545883}) 2024-09-16T03:55:29.731 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.723+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1526470375}) 2024-09-16T03:55:29.732 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.723+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.732 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.727+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=455545883}) 2024-09-16T03:55:29.732 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.727+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=865676200}) 2024-09-16T03:55:29.732 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.727+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.732 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.727+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1104400377}) 2024-09-16T03:55:29.732 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.727+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=184114866}) 2024-09-16T03:55:29.732 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.727+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.732 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.727+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1104400377}) 2024-09-16T03:55:29.732 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.727+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2509569199}) 2024-09-16T03:55:29.734 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.731+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.735 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.731+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3643458248}) 2024-09-16T03:55:29.735 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.731+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1388740195}) 2024-09-16T03:55:29.735 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.731+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.735 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.731+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3643458248}) 2024-09-16T03:55:29.735 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.731+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1851464190}) 2024-09-16T03:55:29.737 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.735+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.737 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.735+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3660904824}) 2024-09-16T03:55:29.737 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.735+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1728131894}) 2024-09-16T03:55:29.737 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.735+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.738 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.735+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3660904824}) 2024-09-16T03:55:29.738 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.735+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4157310064}) 2024-09-16T03:55:29.739 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.735+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.739 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.735+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=505848979}) 2024-09-16T03:55:29.739 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.735+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3627447232}) 2024-09-16T03:55:29.740 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.735+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.740 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.735+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=505848979}) 2024-09-16T03:55:29.740 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.735+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1013471743}) 2024-09-16T03:55:29.741 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.739+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.741 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.739+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3324914804}) 2024-09-16T03:55:29.742 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.739+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1494836423}) 2024-09-16T03:55:29.742 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.739+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.742 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.739+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3324914804}) 2024-09-16T03:55:29.742 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.739+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2451183442}) 2024-09-16T03:55:29.744 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.739+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.744 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.739+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1320745939}) 2024-09-16T03:55:29.744 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.739+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=683576012}) 2024-09-16T03:55:29.744 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.739+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.744 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.739+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1320745939}) 2024-09-16T03:55:29.745 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.739+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4192328277}) 2024-09-16T03:55:29.747 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T03:55:29.747 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.743+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.748 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.743+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=524933928}) 2024-09-16T03:55:29.748 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.743+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=540828014}) 2024-09-16T03:55:29.748 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.743+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.748 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.743+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=524933928}) 2024-09-16T03:55:29.748 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.743+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3114235866}) 2024-09-16T03:55:29.749 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.747+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.750 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.747+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2016013640}) 2024-09-16T03:55:29.750 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.747+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1393450785}) 2024-09-16T03:55:29.750 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.747+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.750 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.747+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2016013640}) 2024-09-16T03:55:29.750 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.747+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3240312113}) 2024-09-16T03:55:29.752 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.747+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.752 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.747+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2784431545}) 2024-09-16T03:55:29.752 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.747+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1015393094}) 2024-09-16T03:55:29.752 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.747+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.752 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.747+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2784431545}) 2024-09-16T03:55:29.752 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.747+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4234637099}) 2024-09-16T03:55:29.753 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.751+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.753 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.751+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=207482527}) 2024-09-16T03:55:29.754 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.751+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1278021430}) 2024-09-16T03:55:29.754 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.751+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.754 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.751+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=207482527}) 2024-09-16T03:55:29.754 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.751+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2392199556}) 2024-09-16T03:55:29.755 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.751+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.755 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.751+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=970571586}) 2024-09-16T03:55:29.755 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.751+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=343300435}) 2024-09-16T03:55:29.756 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.751+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.756 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.751+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=970571586}) 2024-09-16T03:55:29.756 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.751+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2975745200}) 2024-09-16T03:55:29.757 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.755+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.757 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.755+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=178211577}) 2024-09-16T03:55:29.757 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.755+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2244839100}) 2024-09-16T03:55:29.757 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.755+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.757 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.755+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=178211577}) 2024-09-16T03:55:29.758 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.755+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1304442442}) 2024-09-16T03:55:29.759 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.755+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.759 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.755+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3799289234}) 2024-09-16T03:55:29.759 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.755+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4173363603}) 2024-09-16T03:55:29.759 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.755+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.759 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.755+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3799289234}) 2024-09-16T03:55:29.759 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.755+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2522272681}) 2024-09-16T03:55:29.761 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.755+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.761 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.755+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2417682692}) 2024-09-16T03:55:29.761 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.759+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3174519289}) 2024-09-16T03:55:29.762 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.759+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.762 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.759+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2417682692}) 2024-09-16T03:55:29.762 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.759+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2446653619}) 2024-09-16T03:55:29.763 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.759+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.763 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.759+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1865151018}) 2024-09-16T03:55:29.763 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.759+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2919644813}) 2024-09-16T03:55:29.763 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.759+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.763 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.759+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1865151018}) 2024-09-16T03:55:29.763 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.759+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1017587881}) 2024-09-16T03:55:29.765 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.759+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.765 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.759+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=616693651}) 2024-09-16T03:55:29.765 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.759+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=167524021}) 2024-09-16T03:55:29.765 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.759+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.765 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.759+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=616693651}) 2024-09-16T03:55:29.765 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.759+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1076878985}) 2024-09-16T03:55:29.767 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.763+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.767 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.763+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=455855124}) 2024-09-16T03:55:29.767 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.763+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3358454694}) 2024-09-16T03:55:29.767 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.763+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.767 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.763+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=455855124}) 2024-09-16T03:55:29.767 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.763+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4126519350}) 2024-09-16T03:55:29.769 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.763+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.769 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.763+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=503466238}) 2024-09-16T03:55:29.769 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.763+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3347938515}) 2024-09-16T03:55:29.769 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.763+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.769 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.763+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=503466238}) 2024-09-16T03:55:29.769 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.763+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4242888291}) 2024-09-16T03:55:29.770 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.767+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.771 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.767+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1250859748}) 2024-09-16T03:55:29.771 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.767+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1548710387}) 2024-09-16T03:55:29.771 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.767+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.771 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.767+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1250859748}) 2024-09-16T03:55:29.771 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.767+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3218586407}) 2024-09-16T03:55:29.772 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.767+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.773 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.767+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2594187198}) 2024-09-16T03:55:29.773 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.767+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3863228544}) 2024-09-16T03:55:29.773 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.767+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.773 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.767+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2594187198}) 2024-09-16T03:55:29.773 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.767+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3506194521}) 2024-09-16T03:55:29.775 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.771+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.775 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.771+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=242446031}) 2024-09-16T03:55:29.775 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.771+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4289279159}) 2024-09-16T03:55:29.775 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.771+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.775 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.771+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=242446031}) 2024-09-16T03:55:29.775 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.771+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4262136467}) 2024-09-16T03:55:29.777 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.771+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.777 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.771+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3157170557}) 2024-09-16T03:55:29.777 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.771+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=663409805}) 2024-09-16T03:55:29.777 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.771+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.777 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.771+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3157170557}) 2024-09-16T03:55:29.777 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.771+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3327950480}) 2024-09-16T03:55:29.779 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.775+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.779 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.775+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2584154877}) 2024-09-16T03:55:29.779 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.775+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2031392762}) 2024-09-16T03:55:29.779 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.775+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.779 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.775+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2584154877}) 2024-09-16T03:55:29.779 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.775+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2384862000}) 2024-09-16T03:55:29.781 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.775+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.781 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.775+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3285320920}) 2024-09-16T03:55:29.781 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.775+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=244674883}) 2024-09-16T03:55:29.782 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.775+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.782 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.775+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3285320920}) 2024-09-16T03:55:29.782 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.775+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2976422458}) 2024-09-16T03:55:29.786 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.779+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.786 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.779+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1560314712}) 2024-09-16T03:55:29.786 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.779+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3552865123}) 2024-09-16T03:55:29.786 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.779+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.786 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.779+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1560314712}) 2024-09-16T03:55:29.786 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.779+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3964173048}) 2024-09-16T03:55:29.787 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.783+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.787 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.783+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=659996325}) 2024-09-16T03:55:29.787 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.783+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=88432492}) 2024-09-16T03:55:29.787 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.783+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.787 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.783+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=659996325}) 2024-09-16T03:55:29.787 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.783+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3356008690}) 2024-09-16T03:55:29.788 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.783+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.788 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.783+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3875477642}) 2024-09-16T03:55:29.788 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.783+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1460078773}) 2024-09-16T03:55:29.788 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.783+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.788 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.783+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3875477642}) 2024-09-16T03:55:29.788 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.783+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3788073621}) 2024-09-16T03:55:29.789 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.787+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.789 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.787+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4262594569}) 2024-09-16T03:55:29.789 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.787+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3815882677}) 2024-09-16T03:55:29.790 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.787+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.790 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.787+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4262594569}) 2024-09-16T03:55:29.790 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.787+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=971580889}) 2024-09-16T03:55:29.792 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.787+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.792 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.787+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2339039204}) 2024-09-16T03:55:29.792 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.787+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=211406446}) 2024-09-16T03:55:29.793 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.787+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.793 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.787+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2339039204}) 2024-09-16T03:55:29.793 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.787+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3157706538}) 2024-09-16T03:55:29.794 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.791+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.794 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.791+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1053629466}) 2024-09-16T03:55:29.795 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.791+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2124345141}) 2024-09-16T03:55:29.795 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.791+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.795 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.791+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1053629466}) 2024-09-16T03:55:29.795 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.791+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1235158524}) 2024-09-16T03:55:29.796 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.791+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.796 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.791+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=552569467}) 2024-09-16T03:55:29.797 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.791+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2995211265}) 2024-09-16T03:55:29.797 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.791+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.797 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.791+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=552569467}) 2024-09-16T03:55:29.797 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.791+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2082017601}) 2024-09-16T03:55:29.798 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.795+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.799 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.795+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3870302112}) 2024-09-16T03:55:29.799 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.795+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2744669436}) 2024-09-16T03:55:29.799 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.795+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.799 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.795+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3870302112}) 2024-09-16T03:55:29.799 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.795+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3059581524}) 2024-09-16T03:55:29.800 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.795+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.801 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.795+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1909976570}) 2024-09-16T03:55:29.801 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.795+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2149324806}) 2024-09-16T03:55:29.801 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.795+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.801 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.795+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1909976570}) 2024-09-16T03:55:29.801 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.795+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2778103174}) 2024-09-16T03:55:29.802 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.799+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.803 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.799+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4028565471}) 2024-09-16T03:55:29.803 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.799+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2959287811}) 2024-09-16T03:55:29.803 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.799+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.803 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.799+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4028565471}) 2024-09-16T03:55:29.803 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.799+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=241574506}) 2024-09-16T03:55:29.805 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.799+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.805 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.799+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2605132447}) 2024-09-16T03:55:29.805 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.799+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=649012784}) 2024-09-16T03:55:29.805 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.799+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.805 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.799+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2605132447}) 2024-09-16T03:55:29.805 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.799+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3541780673}) 2024-09-16T03:55:29.806 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.803+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.807 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.803+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=585074174}) 2024-09-16T03:55:29.807 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.803+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2204613167}) 2024-09-16T03:55:29.807 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.803+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.807 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.803+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=585074174}) 2024-09-16T03:55:29.807 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.803+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3017405625}) 2024-09-16T03:55:29.808 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.803+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.808 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.803+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1450666190}) 2024-09-16T03:55:29.809 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.803+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=162389448}) 2024-09-16T03:55:29.809 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.803+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.809 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.803+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1450666190}) 2024-09-16T03:55:29.809 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.803+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3456640514}) 2024-09-16T03:55:29.810 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.807+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.810 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.807+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2998520837}) 2024-09-16T03:55:29.810 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.807+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1905437552}) 2024-09-16T03:55:29.811 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.807+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.811 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.807+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2998520837}) 2024-09-16T03:55:29.811 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.807+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1005562445}) 2024-09-16T03:55:29.813 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.807+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.813 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.807+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2490005532}) 2024-09-16T03:55:29.813 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.807+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2534159894}) 2024-09-16T03:55:29.813 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.807+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.813 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.807+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2490005532}) 2024-09-16T03:55:29.813 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.807+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1578033950}) 2024-09-16T03:55:29.814 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.811+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.814 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.811+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2534076744}) 2024-09-16T03:55:29.814 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.811+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2777338144}) 2024-09-16T03:55:29.814 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.811+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.814 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.811+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2534076744}) 2024-09-16T03:55:29.815 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.811+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1737042983}) 2024-09-16T03:55:29.816 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.811+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.816 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.811+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2932542146}) 2024-09-16T03:55:29.817 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.811+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3063774863}) 2024-09-16T03:55:29.817 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.811+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.817 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.811+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2932542146}) 2024-09-16T03:55:29.817 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.811+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1410243396}) 2024-09-16T03:55:29.818 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.815+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.819 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.815+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=303410763}) 2024-09-16T03:55:29.819 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.815+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1614617696}) 2024-09-16T03:55:29.819 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.815+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.819 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.815+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=303410763}) 2024-09-16T03:55:29.819 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.815+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2186130509}) 2024-09-16T03:55:29.820 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.815+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.820 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.815+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1115205372}) 2024-09-16T03:55:29.820 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.815+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=612846529}) 2024-09-16T03:55:29.820 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.815+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.820 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.815+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1115205372}) 2024-09-16T03:55:29.820 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.815+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3675536135}) 2024-09-16T03:55:29.822 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.819+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.822 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.819+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2132787032}) 2024-09-16T03:55:29.822 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.819+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3057482559}) 2024-09-16T03:55:29.823 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.819+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.823 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.819+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2132787032}) 2024-09-16T03:55:29.823 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.819+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1479238882}) 2024-09-16T03:55:29.824 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.819+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.824 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.819+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2637561765}) 2024-09-16T03:55:29.824 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.819+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3664936896}) 2024-09-16T03:55:29.824 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.819+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.824 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.819+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2637561765}) 2024-09-16T03:55:29.824 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.819+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1425008668}) 2024-09-16T03:55:29.826 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.823+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.826 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.823+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1366121272}) 2024-09-16T03:55:29.826 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.823+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3909898768}) 2024-09-16T03:55:29.826 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.823+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.826 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.823+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1366121272}) 2024-09-16T03:55:29.826 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.823+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1846306854}) 2024-09-16T03:55:29.828 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.823+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.828 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.823+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1929885446}) 2024-09-16T03:55:29.828 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.823+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3559401776}) 2024-09-16T03:55:29.828 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.823+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.828 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.823+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1929885446}) 2024-09-16T03:55:29.828 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.823+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3608612726}) 2024-09-16T03:55:29.830 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.827+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.830 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.827+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3204401008}) 2024-09-16T03:55:29.831 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.827+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2545813542}) 2024-09-16T03:55:29.831 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.827+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.831 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.827+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3204401008}) 2024-09-16T03:55:29.831 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.827+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=216051701}) 2024-09-16T03:55:29.833 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.827+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.833 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.827+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1914054646}) 2024-09-16T03:55:29.833 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.827+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2480774124}) 2024-09-16T03:55:29.833 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.827+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.833 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.827+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1914054646}) 2024-09-16T03:55:29.833 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.827+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2735938019}) 2024-09-16T03:55:29.834 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.831+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.834 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.831+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=144032063}) 2024-09-16T03:55:29.834 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.831+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3922781214}) 2024-09-16T03:55:29.834 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.831+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.834 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.831+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=144032063}) 2024-09-16T03:55:29.834 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.831+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1150547467}) 2024-09-16T03:55:29.836 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.831+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.836 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.831+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3841453556}) 2024-09-16T03:55:29.836 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.831+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=360764344}) 2024-09-16T03:55:29.836 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.831+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.836 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.831+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3841453556}) 2024-09-16T03:55:29.836 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.831+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=231962430}) 2024-09-16T03:55:29.838 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.835+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.838 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.835+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2088012987}) 2024-09-16T03:55:29.838 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.835+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2017913418}) 2024-09-16T03:55:29.838 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.835+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.838 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.835+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2088012987}) 2024-09-16T03:55:29.838 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.835+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1556203986}) 2024-09-16T03:55:29.839 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.835+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.839 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.835+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=337554387}) 2024-09-16T03:55:29.840 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.835+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2388350687}) 2024-09-16T03:55:29.840 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.835+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.840 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.835+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=337554387}) 2024-09-16T03:55:29.840 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.835+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2664460733}) 2024-09-16T03:55:29.841 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.839+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.841 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.839+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=735836149}) 2024-09-16T03:55:29.841 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.839+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2939880925}) 2024-09-16T03:55:29.842 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.839+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.842 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.839+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=735836149}) 2024-09-16T03:55:29.842 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.839+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1100301089}) 2024-09-16T03:55:29.843 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.839+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.843 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.839+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2348754869}) 2024-09-16T03:55:29.843 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.839+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3727801159}) 2024-09-16T03:55:29.843 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.839+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.843 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.839+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2348754869}) 2024-09-16T03:55:29.843 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.839+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3687391186}) 2024-09-16T03:55:29.845 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.843+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.846 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.843+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3046263889}) 2024-09-16T03:55:29.846 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.843+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=960446466}) 2024-09-16T03:55:29.846 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.843+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.846 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.843+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3046263889}) 2024-09-16T03:55:29.846 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.843+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4263110261}) 2024-09-16T03:55:29.848 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.843+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.848 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.843+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2331538408}) 2024-09-16T03:55:29.848 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.843+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=483689606}) 2024-09-16T03:55:29.848 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.843+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.849 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.843+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2331538408}) 2024-09-16T03:55:29.849 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.843+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1281401643}) 2024-09-16T03:55:29.850 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.847+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.850 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.847+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=498792291}) 2024-09-16T03:55:29.850 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.847+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3631058126}) 2024-09-16T03:55:29.851 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.847+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.851 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.847+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=498792291}) 2024-09-16T03:55:29.851 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.847+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1577002306}) 2024-09-16T03:55:29.852 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.847+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.852 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.847+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1655179146}) 2024-09-16T03:55:29.853 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.847+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2922320436}) 2024-09-16T03:55:29.853 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.847+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.853 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.847+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1655179146}) 2024-09-16T03:55:29.853 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.847+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3993333913}) 2024-09-16T03:55:29.854 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.851+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.854 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.851+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1805123617}) 2024-09-16T03:55:29.855 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.851+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1334942956}) 2024-09-16T03:55:29.855 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.851+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.855 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.851+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1805123617}) 2024-09-16T03:55:29.855 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.851+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=337288428}) 2024-09-16T03:55:29.856 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.851+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.857 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.851+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2953055821}) 2024-09-16T03:55:29.857 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.851+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4161688933}) 2024-09-16T03:55:29.857 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.851+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.857 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.851+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2953055821}) 2024-09-16T03:55:29.858 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.851+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2742106810}) 2024-09-16T03:55:29.859 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.855+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.859 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.855+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4246942489}) 2024-09-16T03:55:29.859 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.855+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1609244322}) 2024-09-16T03:55:29.859 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.855+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.859 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.855+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4246942489}) 2024-09-16T03:55:29.859 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.855+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=730411411}) 2024-09-16T03:55:29.861 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.855+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.861 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.855+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3485220292}) 2024-09-16T03:55:29.862 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.855+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=972429066}) 2024-09-16T03:55:29.862 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.855+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.862 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.855+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3485220292}) 2024-09-16T03:55:29.862 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.855+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3753246396}) 2024-09-16T03:55:29.863 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.859+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.863 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.859+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3724173827}) 2024-09-16T03:55:29.863 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.859+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=257539029}) 2024-09-16T03:55:29.864 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.859+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.864 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.859+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3724173827}) 2024-09-16T03:55:29.864 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.859+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1607327297}) 2024-09-16T03:55:29.865 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.863+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.865 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.863+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=466470400}) 2024-09-16T03:55:29.866 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.863+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3456385891}) 2024-09-16T03:55:29.866 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.863+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.866 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.863+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=466470400}) 2024-09-16T03:55:29.866 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.863+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=90330697}) 2024-09-16T03:55:29.868 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.863+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.868 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.863+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1562690946}) 2024-09-16T03:55:29.868 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.863+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1475377263}) 2024-09-16T03:55:29.868 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.863+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.868 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.863+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1562690946}) 2024-09-16T03:55:29.869 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.863+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3258775145}) 2024-09-16T03:55:29.870 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.867+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.870 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.867+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3587554669}) 2024-09-16T03:55:29.870 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.867+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1811375917}) 2024-09-16T03:55:29.870 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.867+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.871 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.867+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3587554669}) 2024-09-16T03:55:29.871 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.867+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3342000931}) 2024-09-16T03:55:29.872 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.867+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.872 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.867+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2115707303}) 2024-09-16T03:55:29.872 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.867+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3691088427}) 2024-09-16T03:55:29.873 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.867+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.873 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.867+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2115707303}) 2024-09-16T03:55:29.873 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.867+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1525858789}) 2024-09-16T03:55:29.874 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.871+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.875 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.871+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2186569688}) 2024-09-16T03:55:29.875 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.871+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=113583279}) 2024-09-16T03:55:29.875 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.871+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.875 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.871+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2186569688}) 2024-09-16T03:55:29.875 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.871+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1972963887}) 2024-09-16T03:55:29.876 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.871+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.877 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.871+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1405342612}) 2024-09-16T03:55:29.877 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.871+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1538992203}) 2024-09-16T03:55:29.877 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.871+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.877 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.871+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1405342612}) 2024-09-16T03:55:29.877 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.871+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4111824037}) 2024-09-16T03:55:29.879 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.875+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.879 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.875+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2841668447}) 2024-09-16T03:55:29.879 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.875+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2949748294}) 2024-09-16T03:55:29.879 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.875+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.879 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.875+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2841668447}) 2024-09-16T03:55:29.879 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.875+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2715642743}) 2024-09-16T03:55:29.882 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.875+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.882 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.875+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=831419337}) 2024-09-16T03:55:29.882 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.875+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2017924539}) 2024-09-16T03:55:29.882 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.875+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.882 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.875+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=831419337}) 2024-09-16T03:55:29.882 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.875+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1567791946}) 2024-09-16T03:55:29.883 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.879+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.883 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.879+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=220510752}) 2024-09-16T03:55:29.883 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.879+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4277119477}) 2024-09-16T03:55:29.883 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.879+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.883 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.879+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=220510752}) 2024-09-16T03:55:29.883 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.879+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=903177647}) 2024-09-16T03:55:29.885 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.879+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.885 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.879+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2386393975}) 2024-09-16T03:55:29.885 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.879+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2708155928}) 2024-09-16T03:55:29.885 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.879+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.885 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.879+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2386393975}) 2024-09-16T03:55:29.885 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.879+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1591402749}) 2024-09-16T03:55:29.888 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.883+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.888 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.883+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=77,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10} crc {logm=3315080052,mdsmap=4088524775,mgr=460109776,mgr_command_descs=2107673600,mgr_metadata=1367272202,mgrstat=3886664394}) 2024-09-16T03:55:29.888 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.883+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=79,mdsmap=2,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=10} crc {logm=1838387224,mdsmap=3140514009,mgr=911650295,mgr_command_descs=2107673600,mgr_metadata=1367272202,mgrstat=2663896686}) 2024-09-16T03:55:29.888 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.883+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.888 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.883+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=77,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10} crc {logm=3315080052,mdsmap=4088524775,mgr=460109776,mgr_command_descs=2107673600,mgr_metadata=1367272202,mgrstat=3886664394}) 2024-09-16T03:55:29.888 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.883+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=81,mdsmap=3,mgr=8,mgr_metadata=1,mgrstat=7} crc {logm=2497416819,mdsmap=4088524775,mgr=2470928226,mgr_metadata=1367272202,mgrstat=1724244657}) 2024-09-16T03:55:29.890 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.887+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.890 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.887+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgrstat=1,mon_config_key=11,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=33,osdmap=46} crc {mgrstat=3529458742,mon_config_key=85057112,monmap=3937513981,osd_metadata=4196947761,osd_pg_creating=2728950537,osd_snap=4211304029,osdmap=1791926053}) 2024-09-16T03:55:29.891 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.887+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mgrstat=1,mon_config_key=11,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=32,osdmap=47} crc {mgrstat=3529458742,mon_config_key=85057112,monmap=3937513981,osd_metadata=951161650,osd_pg_creating=2728950537,osd_snap=2692937955,osdmap=661179235}) 2024-09-16T03:55:29.891 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.887+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.891 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.887+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgrstat=1,mon_config_key=11,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=33,osdmap=46} crc {mgrstat=3529458742,mon_config_key=85057112,monmap=3937513981,osd_metadata=4196947761,osd_pg_creating=2728950537,osd_snap=4211304029,osdmap=1791926053}) 2024-09-16T03:55:29.891 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.887+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mgrstat=1,mon_config_key=10,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=32,osdmap=48} crc {mgrstat=3529458742,mon_config_key=1481991774,monmap=3937513981,osd_metadata=951161650,osd_pg_creating=2728950537,osd_snap=284530533,osdmap=4204161999}) 2024-09-16T03:55:29.907 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.903+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.907 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.903+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=94} crc {osdmap=3882295975}) 2024-09-16T03:55:29.907 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.903+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=90} crc {osdmap=1315467707}) 2024-09-16T03:55:29.907 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.903+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:55:29.907 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.903+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=94} crc {osdmap=3882295975}) 2024-09-16T03:55:29.907 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:55:29.903+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=96} crc {osdmap=1225896290}) 2024-09-16T03:55:30.463 INFO:tasks.workunit.client.0.smithi154.stdout: [ OK ] LibRadosTwoPoolsPP.ProxyRead (38895 ms) 2024-09-16T03:55:30.463 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.CachePin 2024-09-16T03:55:30.463 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.CachePin (50166 ms) 2024-09-16T03:55:30.463 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.SetRedirectRead 2024-09-16T03:55:30.463 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.SetRedirectRead (3024 ms) 2024-09-16T03:55:30.464 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestPromoteRead 2024-09-16T03:55:30.464 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: require_osd_release = squid 2024-09-16T03:55:30.464 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestPromoteRead (3038 ms) 2024-09-16T03:55:30.464 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRefRead 2024-09-16T03:55:30.464 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRefRead (3022 ms) 2024-09-16T03:55:30.464 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestUnset 2024-09-16T03:55:30.464 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: require_osd_release = squid 2024-09-16T03:55:30.464 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestUnset (3033 ms) 2024-09-16T03:55:30.464 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestDedupRefRead 2024-09-16T03:55:30.464 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: require_osd_release = squid 2024-09-16T03:55:30.464 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestDedupRefRead (7085 ms) 2024-09-16T03:55:30.464 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapRefcount 2024-09-16T03:55:30.464 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: require_osd_release = squid 2024-09-16T03:55:30.465 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapRefcount (75015 ms) 2024-09-16T03:55:30.465 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapRefcount2 2024-09-16T03:55:30.465 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: require_osd_release = squid 2024-09-16T03:55:30.465 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapRefcount2 (16805 ms) 2024-09-16T03:55:30.465 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestTestSnapCreate 2024-09-16T03:55:30.465 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: require_osd_release = squid 2024-09-16T03:55:30.465 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestTestSnapCreate (4057 ms) 2024-09-16T03:55:30.465 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRedirectAfterPromote 2024-09-16T03:55:30.465 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: require_osd_release = squid 2024-09-16T03:55:30.465 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRedirectAfterPromote (3031 ms) 2024-09-16T03:55:30.465 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestCheckRefcountWhenModification 2024-09-16T03:55:30.465 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: require_osd_release = squid 2024-09-16T03:55:30.465 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestCheckRefcountWhenModification (28413 ms) 2024-09-16T03:55:30.465 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapIncCount 2024-09-16T03:55:30.465 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: require_osd_release = squid 2024-09-16T03:55:30.466 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapIncCount (14971 ms) 2024-09-16T03:55:30.466 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvict 2024-09-16T03:55:30.466 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: require_osd_release = squid 2024-09-16T03:55:30.466 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvict (10100 ms) 2024-09-16T03:55:30.466 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvictPromote 2024-09-16T03:55:30.466 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: require_osd_release = squid 2024-09-16T03:55:30.466 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvictPromote (3440 ms) 2024-09-16T03:55:30.466 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapSizeMismatch 2024-09-16T03:55:30.466 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: require_osd_release = squid 2024-09-16T03:55:30.466 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: waiting for scrubs... 2024-09-16T03:55:30.466 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: done waiting 2024-09-16T03:55:30.466 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapSizeMismatch (25276 ms) 2024-09-16T03:55:30.466 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.DedupFlushRead 2024-09-16T03:55:30.467 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: require_osd_release = squid 2024-09-16T03:55:30.467 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.DedupFlushRead (9769 ms) 2024-09-16T03:55:30.467 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestFlushSnap 2024-09-16T03:55:30.467 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: require_osd_release = squid 2024-09-16T03:55:30.467 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestFlushSnap (8987 ms) 2024-09-16T03:55:30.467 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestFlushDupCount 2024-09-16T03:55:30.467 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: require_osd_release = squid 2024-09-16T03:55:30.467 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestFlushDupCount (8998 ms) 2024-09-16T03:55:30.467 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TierFlushDuringFlush 2024-09-16T03:55:30.748 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T03:55:30.748 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:55:30.893 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.879+0000 7fd436a51640 10 monclient: get_monmap_and_config 2024-09-16T03:55:30.894 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.879+0000 7fd436a51640 10 monclient: build_initial_monmap 2024-09-16T03:55:30.894 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.879+0000 7fd436a51640 10 monclient: monmap: 2024-09-16T03:55:30.894 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:55:30.894 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:55:30.894 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:55:30.881562+0000 2024-09-16T03:55:30.894 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:55:30.881562+0000 2024-09-16T03:55:30.894 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:55:30.894 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:55:30.894 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:55:30.894 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:55:30.895 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:55:30.895 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:55:30.895 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:55:30.895 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:55:30.895 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.879+0000 7fd436a51640 1 Processor -- start 2024-09-16T03:55:30.895 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.879+0000 7fd436a51640 1 -- start start 2024-09-16T03:55:30.895 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.879+0000 7fd436a51640 10 monclient: init 2024-09-16T03:55:30.895 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.879+0000 7fd436a51640 10 monclient: _reopen_session rank -1 2024-09-16T03:55:30.895 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.879+0000 7fd436a51640 10 monclient: _add_conns ranks=[3,1,4,2,0] 2024-09-16T03:55:30.895 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.879+0000 7fd436a51640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd43009fb00 0x7fd43009ba50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:55:30.896 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.879+0000 7fd436a51640 10 monclient(hunting): picked mon.noname-a con 0x7fd43009fb00 addr v2:172.21.15.42:3300/0 2024-09-16T03:55:30.896 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.879+0000 7fd436a51640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd43009bf90 addr v1:172.21.15.154:6789/0 2024-09-16T03:55:30.896 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.879+0000 7fd436a51640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fd43009c400 0x7fd43013b840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:55:30.896 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.879+0000 7fd436a51640 10 monclient(hunting): picked mon.noname-c con 0x7fd43009c400 addr v2:172.21.15.154:3300/0 2024-09-16T03:55:30.896 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.879+0000 7fd436a51640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd4300a1e20 con 0x7fd43009bf90 2024-09-16T03:55:30.896 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.879+0000 7fd436a51640 10 monclient(hunting): start opening mon connection 2024-09-16T03:55:30.896 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.879+0000 7fd436a51640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7fd43009e250 con 0x7fd43009fb00 2024-09-16T03:55:30.896 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.879+0000 7fd436a51640 10 monclient(hunting): start opening mon connection 2024-09-16T03:55:30.896 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.879+0000 7fd436a51640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fd43009c850 con 0x7fd43009c400 2024-09-16T03:55:30.897 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.879+0000 7fd436a51640 10 monclient(hunting): _renew_subs 2024-09-16T03:55:30.897 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.879+0000 7fd436a51640 10 monclient(hunting): authenticate will time out at 1777.851131s 2024-09-16T03:55:30.897 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.879+0000 7fd42ffff640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd43009fb00 0x7fd43009ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:55:30.897 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.879+0000 7fd42ffff640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd43009fb00 0x7fd43009ba50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:34828/0 (socket says 172.21.15.42:34828) 2024-09-16T03:55:30.897 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.879+0000 7fd42ffff640 1 -- 172.21.15.42:0/3603766904 learned_addr learned my addr 172.21.15.42:0/3603766904 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:55:30.897 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.879+0000 7fd42ffff640 10 monclient(hunting): get_auth_request con 0x7fd43009fb00 auth_method 0 2024-09-16T03:55:30.897 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.879+0000 7fd42ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:55:30.897 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.879+0000 7fd42ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:55:30.897 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.879+0000 7fd42ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:55:30.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.879+0000 7fd42ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:55:30.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.879+0000 7fd42ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:55:30.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.879+0000 7fd42ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:55:30.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.879+0000 7fd434fee640 1 --2- 172.21.15.42:0/3603766904 >> v2:172.21.15.154:3300/0 conn(0x7fd43009c400 0x7fd43013b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:55:30.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.879+0000 7fd42ffff640 10 monclient(hunting): handle_auth_done global_id 824595 payload 274 2024-09-16T03:55:30.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.879+0000 7fd42ffff640 1 -- 172.21.15.42:0/3603766904 >> v2:172.21.15.154:3300/0 conn(0x7fd43009c400 msgr2=0x7fd43013b840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:55:30.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.879+0000 7fd42ffff640 1 --2- 172.21.15.42:0/3603766904 >> v2:172.21.15.154:3300/0 conn(0x7fd43009c400 0x7fd43013b840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:55:30.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.879+0000 7fd42ffff640 1 -- 172.21.15.42:0/3603766904 >> v1:172.21.15.154:6789/0 conn(0x7fd43009bf90 legacy=0x7fd430138110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:55:30.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.879+0000 7fd42ffff640 10 monclient: _finish_hunting 0 2024-09-16T03:55:30.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.879+0000 7fd42ffff640 1 monclient: found mon.noname-a 2024-09-16T03:55:30.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.879+0000 7fd42ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:55:30.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.879+0000 7fd42ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:55:30.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.879+0000 7fd42ffff640 1 -- 172.21.15.42:0/3603766904 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd43009ea30 con 0x7fd43009fb00 2024-09-16T03:55:30.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.879+0000 7fd434fee640 10 monclient: get_auth_request con 0x7fd43009c400 auth_method 0 2024-09-16T03:55:30.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.879+0000 7fd434fee640 1 --2- 172.21.15.42:0/3603766904 >> v2:172.21.15.154:3300/0 conn(0x7fd43009c400 0x7fd43013b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T03:55:30.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.883+0000 7fd42ffff640 1 --2- 172.21.15.42:0/3603766904 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd43009fb00 0x7fd43009ba50 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fd4300a37b0 tx=0x7fd42001ea70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b4e96e87bf1ec8d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:55:30.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.883+0000 7fd42effd640 1 -- 172.21.15.42:0/3603766904 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4200b4070 con 0x7fd43009fb00 2024-09-16T03:55:30.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.883+0000 7fd42effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:55:30.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.883+0000 7fd42effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:55:30.900 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.883+0000 7fd42effd640 10 monclient: dump: 2024-09-16T03:55:30.900 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:55:30.900 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:55:30.900 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:30.900 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:30.900 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:55:30.900 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:55:30.900 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:55:30.900 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:55:30.900 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:55:30.900 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:55:30.900 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.883+0000 7fd42effd640 10 monclient: _finish_auth 0 2024-09-16T03:55:30.900 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.883+0000 7fd42effd640 10 monclient: _check_auth_tickets 2024-09-16T03:55:30.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.883+0000 7fd42effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:55:30.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.883+0000 7fd42effd640 1 -- 172.21.15.42:0/3603766904 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd4200a7ac0 con 0x7fd43009fb00 2024-09-16T03:55:30.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.883+0000 7fd436a51640 5 monclient: authenticate success, global_id 824595 2024-09-16T03:55:30.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.883+0000 7fd436a51640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:55:30.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.883+0000 7fd42effd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:55:30.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.883+0000 7fd436a51640 10 monclient: get_monmap_and_config success 2024-09-16T03:55:30.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.883+0000 7fd42effd640 1 -- 172.21.15.42:0/3603766904 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4200af440 con 0x7fd43009fb00 2024-09-16T03:55:30.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.883+0000 7fd42effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:55:30.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.883+0000 7fd436a51640 4 set_mon_vals no callback set 2024-09-16T03:55:30.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.883+0000 7fd42effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:55:30.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.883+0000 7fd42effd640 10 monclient: dump: 2024-09-16T03:55:30.901 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:55:30.901 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:55:30.901 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:30.902 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:30.902 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:55:30.902 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:55:30.902 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:55:30.902 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:55:30.902 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:55:30.902 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:55:30.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.883+0000 7fd436a51640 10 monclient: shutdown 2024-09-16T03:55:30.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.883+0000 7fd436a51640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:55:30.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.883+0000 7fd436a51640 1 -- 172.21.15.42:0/3603766904 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd43009fb00 msgr2=0x7fd43009ba50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:55:30.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.883+0000 7fd436a51640 1 --2- 172.21.15.42:0/3603766904 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd43009fb00 0x7fd43009ba50 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fd4300a37b0 tx=0x7fd42001ea70 comp rx=0 tx=0).stop 2024-09-16T03:55:30.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.883+0000 7fd436a51640 1 -- 172.21.15.42:0/3603766904 shutdown_connections 2024-09-16T03:55:30.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.883+0000 7fd436a51640 1 --2- 172.21.15.42:0/3603766904 >> v2:172.21.15.154:3300/0 conn(0x7fd43009c400 0x7fd43013b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:55:30.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.883+0000 7fd436a51640 1 --2- 172.21.15.42:0/3603766904 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd43009fb00 0x7fd43009ba50 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:55:30.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.883+0000 7fd436a51640 1 -- 172.21.15.42:0/3603766904 >> 172.21.15.42:0/3603766904 conn(0x7fd4300a2420 msgr2=0x7fd4300a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:55:30.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.883+0000 7fd436a51640 1 -- 172.21.15.42:0/3603766904 shutdown_connections 2024-09-16T03:55:30.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.883+0000 7fd436a51640 1 -- 172.21.15.42:0/3603766904 wait complete. 2024-09-16T03:55:30.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.883+0000 7fd436a51640 10 monclient: build_initial_monmap 2024-09-16T03:55:30.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.883+0000 7fd436a51640 10 monclient: monmap: 2024-09-16T03:55:30.903 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:55:30.903 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:55:30.903 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:55:30.903 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:55:30.904 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:55:30.904 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:55:30.904 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:55:30.904 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:55:30.904 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:55:30.904 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:55:30.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.883+0000 7fd436a51640 1 librados: starting msgr at 2024-09-16T03:55:30.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.883+0000 7fd436a51640 1 librados: starting objecter 2024-09-16T03:55:30.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.883+0000 7fd436a51640 1 Processor -- start 2024-09-16T03:55:30.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.883+0000 7fd436a51640 1 -- start start 2024-09-16T03:55:30.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.883+0000 7fd436a51640 1 librados: setting wanted keys 2024-09-16T03:55:30.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.883+0000 7fd436a51640 1 librados: calling monclient init 2024-09-16T03:55:30.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.883+0000 7fd436a51640 10 monclient: init 2024-09-16T03:55:30.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.887+0000 7fd436a51640 10 monclient: _reopen_session rank -1 2024-09-16T03:55:30.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.887+0000 7fd436a51640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T03:55:30.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.887+0000 7fd436a51640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd43009bf90 0x7fd4301cf140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:55:30.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.887+0000 7fd436a51640 10 monclient(hunting): picked mon.noname-a con 0x7fd43009bf90 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:55:30.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.887+0000 7fd436a51640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd43009c400 0x7fd43013b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:55:30.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.887+0000 7fd436a51640 10 monclient(hunting): picked mon.noname-b con 0x7fd43009c400 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:55:30.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.887+0000 7fd436a51640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd43013b6b0 0x7fd430125bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:55:30.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.887+0000 7fd436a51640 10 monclient(hunting): picked mon.noname-c con 0x7fd43013b6b0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:55:30.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.887+0000 7fd436a51640 10 monclient(hunting): start opening mon connection 2024-09-16T03:55:30.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.887+0000 7fd436a51640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fd4300a32f0 con 0x7fd43009bf90 2024-09-16T03:55:30.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.887+0000 7fd436a51640 10 monclient(hunting): start opening mon connection 2024-09-16T03:55:30.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.887+0000 7fd436a51640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd4300a3170 con 0x7fd43009c400 2024-09-16T03:55:30.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.887+0000 7fd436a51640 10 monclient(hunting): start opening mon connection 2024-09-16T03:55:30.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.887+0000 7fd436a51640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fd43009d380 con 0x7fd43013b6b0 2024-09-16T03:55:30.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.887+0000 7fd436a51640 10 monclient(hunting): _renew_subs 2024-09-16T03:55:30.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.887+0000 7fd436a51640 10 monclient(hunting): authenticate will time out at 1777.856303s 2024-09-16T03:55:30.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.887+0000 7fd42ffff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd43009bf90 0x7fd4301cf140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:55:30.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.887+0000 7fd42ffff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd43009bf90 0x7fd4301cf140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:34834/0 (socket says 172.21.15.42:34834) 2024-09-16T03:55:30.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.887+0000 7fd42ffff640 1 -- 172.21.15.42:0/3133926291 learned_addr learned my addr 172.21.15.42:0/3133926291 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:55:30.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.887+0000 7fd42ffff640 10 monclient(hunting): get_auth_request con 0x7fd43009bf90 auth_method 0 2024-09-16T03:55:30.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.887+0000 7fd42ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:55:30.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.887+0000 7fd42ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:55:30.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.887+0000 7fd42ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:55:30.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.887+0000 7fd434fee640 1 --2- 172.21.15.42:0/3133926291 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd43013b6b0 0x7fd430125bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:55:30.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.887+0000 7fd434fee640 10 monclient(hunting): get_auth_request con 0x7fd43013b6b0 auth_method 0 2024-09-16T03:55:30.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.887+0000 7fd434fee640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:55:30.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.887+0000 7fd434fee640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:55:30.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.887+0000 7fd434fee640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:55:30.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.887+0000 7fd42ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:55:30.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.887+0000 7fd42ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:55:30.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.887+0000 7fd42ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:55:30.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.887+0000 7fd434fee640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:55:30.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.887+0000 7fd434fee640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:55:30.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.887+0000 7fd434fee640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:55:30.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.887+0000 7fd42f7fe640 1 --2- 172.21.15.42:0/3133926291 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd43009c400 0x7fd43013b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:55:30.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.887+0000 7fd42ffff640 10 monclient(hunting): handle_auth_done global_id 824598 payload 824 2024-09-16T03:55:30.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.887+0000 7fd42ffff640 1 -- 172.21.15.42:0/3133926291 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd43013b6b0 msgr2=0x7fd430125bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:55:30.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.887+0000 7fd42ffff640 1 --2- 172.21.15.42:0/3133926291 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd43013b6b0 0x7fd430125bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:55:30.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.887+0000 7fd42ffff640 1 -- 172.21.15.42:0/3133926291 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd43009c400 msgr2=0x7fd43013b170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:55:30.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.887+0000 7fd42ffff640 1 --2- 172.21.15.42:0/3133926291 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd43009c400 0x7fd43013b170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:55:30.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.887+0000 7fd42ffff640 10 monclient: _finish_hunting 0 2024-09-16T03:55:30.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.887+0000 7fd42ffff640 1 monclient: found mon.noname-a 2024-09-16T03:55:30.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.887+0000 7fd42ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:55:30.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.887+0000 7fd42ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:55:30.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.887+0000 7fd42ffff640 1 -- 172.21.15.42:0/3133926291 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd430073a30 con 0x7fd43009bf90 2024-09-16T03:55:30.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.887+0000 7fd42f7fe640 10 monclient: get_auth_request con 0x7fd43009c400 auth_method 0 2024-09-16T03:55:30.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.887+0000 7fd42f7fe640 1 --2- 172.21.15.42:0/3133926291 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd43009c400 0x7fd43013b170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T03:55:30.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.887+0000 7fd42ffff640 1 --2- 172.21.15.42:0/3133926291 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd43009bf90 0x7fd4301cf140 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fd42000b2c0 tx=0x7fd42001d2b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:55:30.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.887+0000 7fd42cff9640 10 client.?.objecter ms_handle_connect 0x7fd43009bf90 2024-09-16T03:55:30.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.887+0000 7fd42cff9640 10 client.?.objecter resend_mon_ops 2024-09-16T03:55:30.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.887+0000 7fd42cff9640 1 -- 172.21.15.42:0/3133926291 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4200bf070 con 0x7fd43009bf90 2024-09-16T03:55:30.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.887+0000 7fd42cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:55:30.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.887+0000 7fd42cff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:55:30.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.887+0000 7fd42cff9640 10 monclient: dump: 2024-09-16T03:55:30.909 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:55:30.909 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:55:30.909 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:30.909 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:30.909 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:55:30.909 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:55:30.909 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:55:30.909 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:55:30.909 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:55:30.910 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:55:30.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.887+0000 7fd42cff9640 10 monclient: _finish_auth 0 2024-09-16T03:55:30.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.887+0000 7fd42cff9640 10 monclient: _check_auth_tickets 2024-09-16T03:55:30.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.887+0000 7fd42cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:55:30.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.887+0000 7fd42cff9640 1 -- 172.21.15.42:0/3133926291 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd4200ba440 con 0x7fd43009bf90 2024-09-16T03:55:30.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.887+0000 7fd436a51640 5 monclient: authenticate success, global_id 824598 2024-09-16T03:55:30.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.887+0000 7fd436a51640 10 monclient: _renew_subs 2024-09-16T03:55:30.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.887+0000 7fd436a51640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:55:30.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.887+0000 7fd436a51640 1 -- 172.21.15.42:0/3133926291 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd430071b90 con 0x7fd43009bf90 2024-09-16T03:55:30.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.887+0000 7fd42cff9640 10 monclient: handle_config config(0 keys) 2024-09-16T03:55:30.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.887+0000 7fd42cff9640 1 -- 172.21.15.42:0/3133926291 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4200b4070 con 0x7fd43009bf90 2024-09-16T03:55:30.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.887+0000 7fd42cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:55:30.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.887+0000 7fd42cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:55:30.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.887+0000 7fd42cff9640 10 monclient: dump: 2024-09-16T03:55:30.911 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:55:30.911 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:55:30.911 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:30.911 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:30.911 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:55:30.911 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:55:30.911 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:55:30.911 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:55:30.911 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:55:30.911 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:55:30.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.887+0000 7fd42e7fc640 4 set_mon_vals no callback set 2024-09-16T03:55:30.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.887+0000 7fd436a51640 10 client.824598.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:55:30.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.887+0000 7fd436a51640 10 monclient: _renew_subs 2024-09-16T03:55:30.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.887+0000 7fd436a51640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:55:30.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.887+0000 7fd436a51640 1 -- 172.21.15.42:0/3133926291 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd430129580 con 0x7fd43009bf90 2024-09-16T03:55:30.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.887+0000 7fd436a51640 1 librados: init done 2024-09-16T03:55:30.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.887+0000 7fd436a51640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:55:30.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.887+0000 7fd436a51640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:55:30.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.887+0000 7fd436a51640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:55:30.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.887+0000 7fd436a51640 1 -- 172.21.15.42:0/3133926291 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd43009ba50 con 0x7fd43009bf90 2024-09-16T03:55:30.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.887+0000 7fd42cff9640 1 -- 172.21.15.42:0/3133926291 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fd4200c2070 con 0x7fd43009bf90 2024-09-16T03:55:30.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.887+0000 7fd42cff9640 1 --2- 172.21.15.42:0/3133926291 >> v2:172.21.15.154:6812/254845342 conn(0x7fd41c037840 0x7fd41c039d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:55:30.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.887+0000 7fd42cff9640 1 -- 172.21.15.42:0/3133926291 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(417..417 src has 330..417) ==== 5713+0+0 (secure 0 0 0) 0x7fd4200e3a70 con 0x7fd43009bf90 2024-09-16T03:55:30.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.887+0000 7fd42cff9640 10 client.824598.objecter ms_dispatch 0x7fd430000ba0 osd_map(417..417 src has 330..417) 2024-09-16T03:55:30.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.887+0000 7fd42cff9640 3 client.824598.objecter handle_osd_map got epochs [417,417] > 0 2024-09-16T03:55:30.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.887+0000 7fd42cff9640 3 client.824598.objecter handle_osd_map decoding full epoch 417 2024-09-16T03:55:30.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.887+0000 7fd42cff9640 20 client.824598.objecter dump_active .. 0 homeless 2024-09-16T03:55:30.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.887+0000 7fd42f7fe640 1 --2- 172.21.15.42:0/3133926291 >> v2:172.21.15.154:6812/254845342 conn(0x7fd41c037840 0x7fd41c039d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:55:30.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.887+0000 7fd42f7fe640 10 monclient: get_auth_request con 0x7fd41c037840 auth_method 0 2024-09-16T03:55:30.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.891+0000 7fd42f7fe640 1 --2- 172.21.15.42:0/3133926291 >> v2:172.21.15.154:6812/254845342 conn(0x7fd41c037840 0x7fd41c039d00 secure :-1 s=READY pgs=3637 cs=0 l=1 rev1=1 crypto rx=0x7fd430127450 tx=0x7fd424009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:55:30.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.891+0000 7fd42cff9640 10 client.824598.objecter ms_handle_connect 0x7fd41c037840 2024-09-16T03:55:30.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.891+0000 7fd42cff9640 1 -- 172.21.15.42:0/3133926291 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7fd430073a30 con 0x7fd43009bf90 2024-09-16T03:55:30.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.891+0000 7fd42cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:55:30.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:30.891+0000 7fd42cff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:55:31.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.119+0000 7fd436a51640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:55:31.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.119+0000 7fd436a51640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:55:31.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.119+0000 7fd436a51640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:55:31.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.119+0000 7fd436a51640 1 -- 172.21.15.42:0/3133926291 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fd430069040 con 0x7fd43009bf90 2024-09-16T03:55:31.125 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.119+0000 7fd42cff9640 1 -- 172.21.15.42:0/3133926291 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7fd420018780 con 0x7fd43009bf90 2024-09-16T03:55:31.125 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.119+0000 7fd42cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:55:31.125 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.119+0000 7fd42cff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:55:31.126 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":116,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:55:31.128 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.123+0000 7fd436a51640 10 librados: watch_flush enter 2024-09-16T03:55:31.128 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.123+0000 7fd436a51640 10 librados: watch_flush exit 2024-09-16T03:55:31.128 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.123+0000 7fd436a51640 20 client.824598.objecter shutdown clearing up homeless session... 2024-09-16T03:55:31.128 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.123+0000 7fd436a51640 10 client.824598.objecter successfully canceled tick 2024-09-16T03:55:31.128 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.123+0000 7fd436a51640 1 -- 172.21.15.42:0/3133926291 >> v2:172.21.15.154:6812/254845342 conn(0x7fd41c037840 msgr2=0x7fd41c039d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:55:31.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.123+0000 7fd436a51640 1 --2- 172.21.15.42:0/3133926291 >> v2:172.21.15.154:6812/254845342 conn(0x7fd41c037840 0x7fd41c039d00 secure :-1 s=READY pgs=3637 cs=0 l=1 rev1=1 crypto rx=0x7fd430127450 tx=0x7fd424009040 comp rx=0 tx=0).stop 2024-09-16T03:55:31.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.123+0000 7fd436a51640 10 monclient: shutdown 2024-09-16T03:55:31.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.123+0000 7fd436a51640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:55:31.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.123+0000 7fd436a51640 1 -- 172.21.15.42:0/3133926291 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd43009bf90 msgr2=0x7fd4301cf140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:55:31.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.123+0000 7fd436a51640 1 --2- 172.21.15.42:0/3133926291 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd43009bf90 0x7fd4301cf140 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fd42000b2c0 tx=0x7fd42001d2b0 comp rx=0 tx=0).stop 2024-09-16T03:55:31.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.123+0000 7fd436a51640 1 -- 172.21.15.42:0/3133926291 shutdown_connections 2024-09-16T03:55:31.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.123+0000 7fd436a51640 1 --2- 172.21.15.42:0/3133926291 >> v2:172.21.15.154:6812/254845342 conn(0x7fd41c037840 0x7fd41c039d00 unknown :-1 s=CLOSED pgs=3637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:55:31.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.123+0000 7fd436a51640 1 --2- 172.21.15.42:0/3133926291 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd43013b6b0 0x7fd430125bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:55:31.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.123+0000 7fd436a51640 1 --2- 172.21.15.42:0/3133926291 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd43009c400 0x7fd43013b170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:55:31.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.123+0000 7fd436a51640 1 --2- 172.21.15.42:0/3133926291 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd43009bf90 0x7fd4301cf140 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:55:31.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.123+0000 7fd436a51640 1 -- 172.21.15.42:0/3133926291 >> 172.21.15.42:0/3133926291 conn(0x7fd4300a2420 msgr2=0x7fd430072340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:55:31.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.123+0000 7fd436a51640 1 -- 172.21.15.42:0/3133926291 shutdown_connections 2024-09-16T03:55:31.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.123+0000 7fd436a51640 1 -- 172.21.15.42:0/3133926291 wait complete. 2024-09-16T03:55:31.131 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.123+0000 7fd436a51640 1 librados: shutdown 2024-09-16T03:55:31.143 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T03:55:31.143 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T03:55:31.143 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T03:55:31.143 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T03:55:31.262 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.259+0000 7ff133197640 10 monclient: get_monmap_and_config 2024-09-16T03:55:31.263 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.259+0000 7ff133197640 10 monclient: build_initial_monmap 2024-09-16T03:55:31.263 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.259+0000 7ff133197640 10 monclient: monmap: 2024-09-16T03:55:31.263 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:55:31.263 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:55:31.263 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:55:31.261396+0000 2024-09-16T03:55:31.263 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:55:31.261396+0000 2024-09-16T03:55:31.263 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:55:31.263 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:55:31.263 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:55:31.263 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:55:31.263 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:55:31.264 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:55:31.264 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:55:31.264 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:55:31.266 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.259+0000 7ff133197640 1 Processor -- start 2024-09-16T03:55:31.266 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.259+0000 7ff133197640 1 -- start start 2024-09-16T03:55:31.266 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.259+0000 7ff133197640 10 monclient: init 2024-09-16T03:55:31.266 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.259+0000 7ff133197640 10 monclient: _reopen_session rank -1 2024-09-16T03:55:31.266 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.259+0000 7ff133197640 10 monclient: _add_conns ranks=[4,3,0,2,1] 2024-09-16T03:55:31.266 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.259+0000 7ff133197640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7ff12c0a39b0 0x7ff12c133b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:55:31.266 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.259+0000 7ff133197640 10 monclient(hunting): picked mon.noname-c con 0x7ff12c0a39b0 addr v2:172.21.15.154:3300/0 2024-09-16T03:55:31.266 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.259+0000 7ff133197640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff12c0a3d90 0x7ff12c137000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:55:31.266 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.259+0000 7ff133197640 10 monclient(hunting): picked mon.noname-a con 0x7ff12c0a3d90 addr v2:172.21.15.42:3300/0 2024-09-16T03:55:31.266 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.259+0000 7ff133197640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff12c137540 addr v1:172.21.15.42:6789/0 2024-09-16T03:55:31.266 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.259+0000 7ff133197640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff12c09c140 con 0x7ff12c137540 2024-09-16T03:55:31.267 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.259+0000 7ff133197640 10 monclient(hunting): start opening mon connection 2024-09-16T03:55:31.267 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.259+0000 7ff133197640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7ff12c09c420 con 0x7ff12c0a3d90 2024-09-16T03:55:31.267 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.259+0000 7ff133197640 10 monclient(hunting): start opening mon connection 2024-09-16T03:55:31.267 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.259+0000 7ff133197640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7ff12c06dd60 con 0x7ff12c0a39b0 2024-09-16T03:55:31.267 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.259+0000 7ff133197640 10 monclient(hunting): _renew_subs 2024-09-16T03:55:31.267 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.259+0000 7ff133197640 10 monclient(hunting): authenticate will time out at 1778.231122s 2024-09-16T03:55:31.267 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.259+0000 7ff12bfff640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff12c0a3d90 0x7ff12c137000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:55:31.267 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.259+0000 7ff131734640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7ff12c137540 0x7ff12c13e390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:49436/0 (socket says 172.21.15.42:49436) 2024-09-16T03:55:31.267 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.259+0000 7ff131734640 1 -- 172.21.15.42:0/3040281461 learned_addr learned my addr 172.21.15.42:0/3040281461 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:55:31.267 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.259+0000 7ff12bfff640 10 monclient(hunting): get_auth_request con 0x7ff12c0a3d90 auth_method 0 2024-09-16T03:55:31.267 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.259+0000 7ff12bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:55:31.268 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.259+0000 7ff12bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:55:31.268 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.259+0000 7ff12bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:55:31.268 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.259+0000 7ff12bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:55:31.268 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.259+0000 7ff12bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:55:31.268 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.259+0000 7ff12bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:55:31.268 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.259+0000 7ff130f33640 1 --2- 172.21.15.42:0/3040281461 >> v2:172.21.15.154:3300/0 conn(0x7ff12c0a39b0 0x7ff12c133b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:55:31.268 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.259+0000 7ff130f33640 10 monclient(hunting): get_auth_request con 0x7ff12c0a39b0 auth_method 0 2024-09-16T03:55:31.268 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.259+0000 7ff130f33640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:55:31.268 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.259+0000 7ff130f33640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:55:31.268 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.259+0000 7ff130f33640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:55:31.268 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.259+0000 7ff12bfff640 10 monclient(hunting): handle_auth_done global_id 824601 payload 274 2024-09-16T03:55:31.268 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.259+0000 7ff12bfff640 1 -- 172.21.15.42:0/3040281461 >> v2:172.21.15.154:3300/0 conn(0x7ff12c0a39b0 msgr2=0x7ff12c133b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:55:31.268 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.259+0000 7ff12bfff640 1 --2- 172.21.15.42:0/3040281461 >> v2:172.21.15.154:3300/0 conn(0x7ff12c0a39b0 0x7ff12c133b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:55:31.269 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.259+0000 7ff12bfff640 1 -- 172.21.15.42:0/3040281461 >> v1:172.21.15.42:6789/0 conn(0x7ff12c137540 legacy=0x7ff12c13e390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:55:31.269 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.259+0000 7ff12bfff640 10 monclient: _finish_hunting 0 2024-09-16T03:55:31.269 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.259+0000 7ff12bfff640 1 monclient: found mon.noname-a 2024-09-16T03:55:31.269 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.259+0000 7ff12bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:55:31.269 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.259+0000 7ff12bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:55:31.269 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.259+0000 7ff12bfff640 1 -- 172.21.15.42:0/3040281461 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff12c0a5e20 con 0x7ff12c0a3d90 2024-09-16T03:55:31.274 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.263+0000 7ff12bfff640 1 --2- 172.21.15.42:0/3040281461 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff12c0a3d90 0x7ff12c137000 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7ff12c0a7750 tx=0x7ff11c01c340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c37a6d462751c13c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:55:31.274 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.263+0000 7ff12b7fe640 1 -- 172.21.15.42:0/3040281461 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff11c0b0070 con 0x7ff12c0a3d90 2024-09-16T03:55:31.275 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.263+0000 7ff12b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:55:31.275 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.263+0000 7ff12b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:55:31.275 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.263+0000 7ff12b7fe640 10 monclient: dump: 2024-09-16T03:55:31.275 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:55:31.275 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:55:31.275 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:31.275 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:31.275 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:55:31.275 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:55:31.275 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:55:31.275 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:55:31.275 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:55:31.275 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:55:31.275 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.263+0000 7ff12b7fe640 10 monclient: _finish_auth 0 2024-09-16T03:55:31.276 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.263+0000 7ff12b7fe640 10 monclient: _check_auth_tickets 2024-09-16T03:55:31.276 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.263+0000 7ff12b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:55:31.276 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.263+0000 7ff12b7fe640 1 -- 172.21.15.42:0/3040281461 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff11c0ab600 con 0x7ff12c0a3d90 2024-09-16T03:55:31.276 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.263+0000 7ff12b7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:55:31.276 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.263+0000 7ff12b7fe640 1 -- 172.21.15.42:0/3040281461 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff11c0ab9d0 con 0x7ff12c0a3d90 2024-09-16T03:55:31.276 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.263+0000 7ff133197640 5 monclient: authenticate success, global_id 824601 2024-09-16T03:55:31.276 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.263+0000 7ff133197640 10 monclient: get_monmap_and_config success 2024-09-16T03:55:31.276 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.263+0000 7ff133197640 4 set_mon_vals no callback set 2024-09-16T03:55:31.276 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.263+0000 7ff12b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:55:31.276 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.263+0000 7ff12b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:55:31.276 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.263+0000 7ff12b7fe640 10 monclient: dump: 2024-09-16T03:55:31.276 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:55:31.276 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:55:31.277 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:31.277 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:31.277 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:55:31.277 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:55:31.277 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:55:31.277 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:55:31.277 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:55:31.277 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:55:31.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.263+0000 7ff133197640 10 monclient: shutdown 2024-09-16T03:55:31.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.263+0000 7ff133197640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:55:31.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.263+0000 7ff133197640 1 -- 172.21.15.42:0/3040281461 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff12c0a3d90 msgr2=0x7ff12c137000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:55:31.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.263+0000 7ff133197640 1 --2- 172.21.15.42:0/3040281461 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff12c0a3d90 0x7ff12c137000 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7ff12c0a7750 tx=0x7ff11c01c340 comp rx=0 tx=0).stop 2024-09-16T03:55:31.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.263+0000 7ff133197640 1 -- 172.21.15.42:0/3040281461 shutdown_connections 2024-09-16T03:55:31.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.263+0000 7ff133197640 1 --2- 172.21.15.42:0/3040281461 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff12c0a3d90 0x7ff12c137000 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:55:31.278 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.263+0000 7ff133197640 1 --2- 172.21.15.42:0/3040281461 >> v2:172.21.15.154:3300/0 conn(0x7ff12c0a39b0 0x7ff12c133b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:55:31.278 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.263+0000 7ff133197640 1 -- 172.21.15.42:0/3040281461 >> 172.21.15.42:0/3040281461 conn(0x7ff12c0a9fb0 msgr2=0x7ff12c0aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:55:31.278 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.263+0000 7ff133197640 1 -- 172.21.15.42:0/3040281461 shutdown_connections 2024-09-16T03:55:31.278 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.263+0000 7ff133197640 1 -- 172.21.15.42:0/3040281461 wait complete. 2024-09-16T03:55:31.278 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.263+0000 7ff133197640 10 monclient: build_initial_monmap 2024-09-16T03:55:31.278 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.263+0000 7ff133197640 10 monclient: monmap: 2024-09-16T03:55:31.278 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:55:31.278 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:55:31.278 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:55:31.278 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:55:31.278 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:55:31.278 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:55:31.278 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:55:31.278 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:55:31.279 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:55:31.279 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:55:31.279 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.263+0000 7ff133197640 1 librados: starting msgr at 2024-09-16T03:55:31.279 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.263+0000 7ff133197640 1 librados: starting objecter 2024-09-16T03:55:31.279 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.263+0000 7ff133197640 1 Processor -- start 2024-09-16T03:55:31.279 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.263+0000 7ff133197640 1 -- start start 2024-09-16T03:55:31.279 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.263+0000 7ff133197640 1 librados: setting wanted keys 2024-09-16T03:55:31.279 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.263+0000 7ff133197640 1 librados: calling monclient init 2024-09-16T03:55:31.279 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.263+0000 7ff133197640 10 monclient: init 2024-09-16T03:55:31.279 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.267+0000 7ff133197640 10 monclient: _reopen_session rank -1 2024-09-16T03:55:31.279 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.267+0000 7ff133197640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T03:55:31.279 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.267+0000 7ff133197640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff12c0a39b0 0x7ff12c130a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:55:31.279 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.267+0000 7ff133197640 10 monclient(hunting): picked mon.noname-b con 0x7ff12c0a39b0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:55:31.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.267+0000 7ff133197640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff12c0a3d90 0x7ff12c13d240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:55:31.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.267+0000 7ff133197640 10 monclient(hunting): picked mon.noname-c con 0x7ff12c0a3d90 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:55:31.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.267+0000 7ff133197640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff12c137540 0x7ff12c13d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:55:31.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.267+0000 7ff133197640 10 monclient(hunting): picked mon.noname-a con 0x7ff12c137540 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:55:31.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.267+0000 7ff133197640 10 monclient(hunting): start opening mon connection 2024-09-16T03:55:31.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.267+0000 7ff133197640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7ff12c1424e0 con 0x7ff12c137540 2024-09-16T03:55:31.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.267+0000 7ff133197640 10 monclient(hunting): start opening mon connection 2024-09-16T03:55:31.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.267+0000 7ff133197640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff12c142660 con 0x7ff12c0a39b0 2024-09-16T03:55:31.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.267+0000 7ff133197640 10 monclient(hunting): start opening mon connection 2024-09-16T03:55:31.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.267+0000 7ff133197640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7ff12c1427e0 con 0x7ff12c0a3d90 2024-09-16T03:55:31.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.267+0000 7ff133197640 10 monclient(hunting): _renew_subs 2024-09-16T03:55:31.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.267+0000 7ff133197640 10 monclient(hunting): authenticate will time out at 1778.236092s 2024-09-16T03:55:31.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.267+0000 7ff12bfff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff12c0a3d90 0x7ff12c13d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:55:31.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.267+0000 7ff131734640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff12c137540 0x7ff12c13d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:55:31.281 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.267+0000 7ff12bfff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff12c0a3d90 0x7ff12c13d240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:51746/0 (socket says 172.21.15.42:51746) 2024-09-16T03:55:31.281 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.267+0000 7ff131734640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff12c137540 0x7ff12c13d780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:34848/0 (socket says 172.21.15.42:34848) 2024-09-16T03:55:31.281 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.267+0000 7ff12bfff640 1 -- 172.21.15.42:0/695820863 learned_addr learned my addr 172.21.15.42:0/695820863 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:55:31.281 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.267+0000 7ff12bfff640 10 monclient(hunting): get_auth_request con 0x7ff12c0a3d90 auth_method 0 2024-09-16T03:55:31.281 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.267+0000 7ff12bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:55:31.281 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.267+0000 7ff12bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:55:31.281 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.267+0000 7ff12bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:55:31.281 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.267+0000 7ff131734640 10 monclient(hunting): get_auth_request con 0x7ff12c137540 auth_method 0 2024-09-16T03:55:31.281 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.267+0000 7ff131734640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:55:31.281 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.267+0000 7ff131734640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:55:31.281 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.267+0000 7ff131734640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:55:31.281 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.267+0000 7ff12bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:55:31.281 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.267+0000 7ff12bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:55:31.281 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.267+0000 7ff12bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:55:31.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.267+0000 7ff131734640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:55:31.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.267+0000 7ff131734640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:55:31.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.267+0000 7ff131734640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:55:31.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.267+0000 7ff130f33640 1 --2- 172.21.15.42:0/695820863 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff12c0a39b0 0x7ff12c130a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:55:31.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.267+0000 7ff130f33640 10 monclient(hunting): get_auth_request con 0x7ff12c0a39b0 auth_method 0 2024-09-16T03:55:31.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.267+0000 7ff130f33640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:55:31.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.267+0000 7ff130f33640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:55:31.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.267+0000 7ff130f33640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:55:31.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.267+0000 7ff12bfff640 10 monclient(hunting): handle_auth_done global_id 854219 payload 824 2024-09-16T03:55:31.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.267+0000 7ff12bfff640 1 -- 172.21.15.42:0/695820863 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff12c0a39b0 msgr2=0x7ff12c130a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:55:31.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.267+0000 7ff12bfff640 1 --2- 172.21.15.42:0/695820863 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff12c0a39b0 0x7ff12c130a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:55:31.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.267+0000 7ff12bfff640 1 -- 172.21.15.42:0/695820863 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff12c137540 msgr2=0x7ff12c13d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:55:31.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.267+0000 7ff12bfff640 1 --2- 172.21.15.42:0/695820863 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff12c137540 0x7ff12c13d780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:55:31.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.267+0000 7ff12bfff640 10 monclient: _finish_hunting 0 2024-09-16T03:55:31.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.267+0000 7ff12bfff640 1 monclient: found mon.noname-c 2024-09-16T03:55:31.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.267+0000 7ff12bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:55:31.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.267+0000 7ff12bfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:55:31.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.267+0000 7ff12bfff640 1 -- 172.21.15.42:0/695820863 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff12c09e900 con 0x7ff12c0a3d90 2024-09-16T03:55:31.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.267+0000 7ff131734640 1 --2- 172.21.15.42:0/695820863 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff12c137540 0x7ff12c13d780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:55:31.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.267+0000 7ff12bfff640 1 --2- 172.21.15.42:0/695820863 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff12c0a3d90 0x7ff12c13d240 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7ff11c0ae040 tx=0x7ff11c0190e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:55:31.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.267+0000 7ff1297fa640 10 client.?.objecter ms_handle_connect 0x7ff12c0a3d90 2024-09-16T03:55:31.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.267+0000 7ff1297fa640 10 client.?.objecter resend_mon_ops 2024-09-16T03:55:31.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.267+0000 7ff1297fa640 1 -- 172.21.15.42:0/695820863 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff11c0b0040 con 0x7ff12c0a3d90 2024-09-16T03:55:31.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.267+0000 7ff1297fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:55:31.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.267+0000 7ff1297fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:55:31.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.267+0000 7ff1297fa640 10 monclient: dump: 2024-09-16T03:55:31.283 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:55:31.283 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:55:31.284 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:31.284 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:31.284 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:55:31.284 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:55:31.284 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:55:31.284 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:55:31.284 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:55:31.284 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:55:31.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.267+0000 7ff1297fa640 10 monclient: _finish_auth 0 2024-09-16T03:55:31.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.267+0000 7ff1297fa640 10 monclient: _check_auth_tickets 2024-09-16T03:55:31.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.267+0000 7ff1297fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:55:31.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.267+0000 7ff133197640 5 monclient: authenticate success, global_id 854219 2024-09-16T03:55:31.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.267+0000 7ff133197640 10 monclient: _renew_subs 2024-09-16T03:55:31.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.267+0000 7ff133197640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:55:31.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.267+0000 7ff133197640 1 -- 172.21.15.42:0/695820863 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff12c1332f0 con 0x7ff12c0a3d90 2024-09-16T03:55:31.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.267+0000 7ff133197640 10 client.854219.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:55:31.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.267+0000 7ff133197640 10 monclient: _renew_subs 2024-09-16T03:55:31.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.267+0000 7ff133197640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:55:31.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.267+0000 7ff133197640 1 -- 172.21.15.42:0/695820863 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff12c06f680 con 0x7ff12c0a3d90 2024-09-16T03:55:31.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.267+0000 7ff133197640 1 librados: init done 2024-09-16T03:55:31.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.267+0000 7ff1297fa640 1 -- 172.21.15.42:0/695820863 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff11c0b9070 con 0x7ff12c0a3d90 2024-09-16T03:55:31.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.267+0000 7ff1297fa640 10 monclient: handle_config config(0 keys) 2024-09-16T03:55:31.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.267+0000 7ff1297fa640 1 -- 172.21.15.42:0/695820863 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff11c0be660 con 0x7ff12c0a3d90 2024-09-16T03:55:31.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.267+0000 7ff1297fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:55:31.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.267+0000 7ff1297fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:55:31.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.267+0000 7ff1297fa640 10 monclient: dump: 2024-09-16T03:55:31.285 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:55:31.285 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:55:31.286 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:31.286 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:31.286 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:55:31.286 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:55:31.286 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:55:31.286 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:55:31.286 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:55:31.286 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:55:31.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.267+0000 7ff12affd640 4 set_mon_vals no callback set 2024-09-16T03:55:31.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.267+0000 7ff133197640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:55:31.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.267+0000 7ff133197640 1 --2- 172.21.15.42:0/695820863 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff110002890 0x7ff110004d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:55:31.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.267+0000 7ff133197640 10 monclient(hunting): start opening mon connection 2024-09-16T03:55:31.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.267+0000 7ff133197640 1 -- 172.21.15.42:0/695820863 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7ff110007cc0 con 0x7ff110002890 2024-09-16T03:55:31.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.267+0000 7ff130f33640 1 --2- 172.21.15.42:0/695820863 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff110002890 0x7ff110004d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:55:31.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.267+0000 7ff1297fa640 1 -- 172.21.15.42:0/695820863 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7ff11c0be800 con 0x7ff12c0a3d90 2024-09-16T03:55:31.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.267+0000 7ff130f33640 10 monclient: get_auth_request con 0x7ff110002890 auth_method 0 2024-09-16T03:55:31.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.267+0000 7ff130f33640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:55:31.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.267+0000 7ff130f33640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:55:31.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.267+0000 7ff130f33640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:55:31.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.267+0000 7ff130f33640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:55:31.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.267+0000 7ff130f33640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:55:31.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.267+0000 7ff130f33640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:55:31.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.267+0000 7ff1297fa640 1 --2- 172.21.15.42:0/695820863 >> v2:172.21.15.154:6812/254845342 conn(0x7ff10c037810 0x7ff10c039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:55:31.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.267+0000 7ff1297fa640 1 -- 172.21.15.42:0/695820863 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(417..417 src has 330..417) ==== 5713+0+0 (secure 0 0 0) 0x7ff11c0e2d10 con 0x7ff12c0a3d90 2024-09-16T03:55:31.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.267+0000 7ff1297fa640 10 client.854219.objecter ms_dispatch 0x7ff12c000ba0 osd_map(417..417 src has 330..417) 2024-09-16T03:55:31.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.267+0000 7ff1297fa640 3 client.854219.objecter handle_osd_map got epochs [417,417] > 0 2024-09-16T03:55:31.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.267+0000 7ff130f33640 10 monclient(hunting): handle_auth_done global_id 824610 payload 824 2024-09-16T03:55:31.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.267+0000 7ff1297fa640 3 client.854219.objecter handle_osd_map decoding full epoch 417 2024-09-16T03:55:31.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.267+0000 7ff130f33640 1 -- 172.21.15.42:0/695820863 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff110007ec0 con 0x7ff110002890 2024-09-16T03:55:31.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.267+0000 7ff1297fa640 20 client.854219.objecter dump_active .. 0 homeless 2024-09-16T03:55:31.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.267+0000 7ff1297fa640 10 client.854219.objecter ms_handle_connect 0x7ff110002890 2024-09-16T03:55:31.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.267+0000 7ff1297fa640 10 client.854219.objecter resend_mon_ops 2024-09-16T03:55:31.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.267+0000 7ff130f33640 1 --2- 172.21.15.42:0/695820863 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff110002890 0x7ff110004d30 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7ff1140045c0 tx=0x7ff114007700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:55:31.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.267+0000 7ff131734640 1 --2- 172.21.15.42:0/695820863 >> v2:172.21.15.154:6812/254845342 conn(0x7ff10c037810 0x7ff10c039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:55:31.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.267+0000 7ff131734640 10 monclient: get_auth_request con 0x7ff10c037810 auth_method 0 2024-09-16T03:55:31.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.267+0000 7ff1297fa640 1 -- 172.21.15.42:0/695820863 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff114014610 con 0x7ff110002890 2024-09-16T03:55:31.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.267+0000 7ff1297fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:55:31.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.267+0000 7ff1297fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:55:31.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.267+0000 7ff1297fa640 10 monclient: dump: 2024-09-16T03:55:31.288 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:55:31.289 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:55:31.289 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:31.289 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:31.289 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:55:31.289 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:55:31.289 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:55:31.289 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:55:31.289 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:55:31.289 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:55:31.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.267+0000 7ff1297fa640 1 -- 172.21.15.42:0/695820863 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7ff114019070 con 0x7ff110002890 2024-09-16T03:55:31.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.267+0000 7ff1297fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:55:31.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.267+0000 7ff1297fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:55:31.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.267+0000 7ff1297fa640 1 -- 172.21.15.42:0/695820863 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff110002890 msgr2=0x7ff110004d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:55:31.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.267+0000 7ff1297fa640 1 --2- 172.21.15.42:0/695820863 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff110002890 0x7ff110004d30 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7ff1140045c0 tx=0x7ff114007700 comp rx=0 tx=0).stop 2024-09-16T03:55:31.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.267+0000 7ff1297fa640 1 -- 172.21.15.42:0/695820863 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff110002890 msgr2=0x7ff110004d30 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:55:31.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.267+0000 7ff1297fa640 1 --2- 172.21.15.42:0/695820863 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff110002890 0x7ff110004d30 secure :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7ff1140045c0 tx=0x7ff114007700 comp rx=0 tx=0).stop 2024-09-16T03:55:31.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.267+0000 7ff1297fa640 10 client.854219.objecter ms_handle_connect 0x7ff10c037810 2024-09-16T03:55:31.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.267+0000 7ff131734640 1 --2- 172.21.15.42:0/695820863 >> v2:172.21.15.154:6812/254845342 conn(0x7ff10c037810 0x7ff10c039cd0 secure :-1 s=READY pgs=3638 cs=0 l=1 rev1=1 crypto rx=0x7ff124005890 tx=0x7ff124009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:55:31.292 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.275+0000 7ff133197640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:55:31.292 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.275+0000 7ff133197640 1 --2- 172.21.15.42:0/695820863 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff110006ce0 0x7ff110007170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:55:31.292 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.275+0000 7ff133197640 10 monclient(hunting): start opening mon connection 2024-09-16T03:55:31.292 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.275+0000 7ff133197640 1 -- 172.21.15.42:0/695820863 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7ff11000a1e0 con 0x7ff110006ce0 2024-09-16T03:55:31.292 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.275+0000 7ff130f33640 1 --2- 172.21.15.42:0/695820863 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff110006ce0 0x7ff110007170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:55:31.292 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.275+0000 7ff130f33640 10 monclient: get_auth_request con 0x7ff110006ce0 auth_method 0 2024-09-16T03:55:31.292 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.275+0000 7ff130f33640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:55:31.292 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.275+0000 7ff130f33640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:55:31.292 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.275+0000 7ff130f33640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:55:31.292 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.275+0000 7ff130f33640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:55:31.292 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.275+0000 7ff130f33640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:55:31.292 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.275+0000 7ff130f33640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:55:31.292 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.275+0000 7ff130f33640 10 monclient(hunting): handle_auth_done global_id 824613 payload 824 2024-09-16T03:55:31.293 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.275+0000 7ff130f33640 1 -- 172.21.15.42:0/695820863 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ff1100076b0 con 0x7ff110006ce0 2024-09-16T03:55:31.293 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.275+0000 7ff1297fa640 10 client.854219.objecter ms_handle_connect 0x7ff110006ce0 2024-09-16T03:55:31.293 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.275+0000 7ff1297fa640 10 client.854219.objecter resend_mon_ops 2024-09-16T03:55:31.293 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.275+0000 7ff130f33640 1 --2- 172.21.15.42:0/695820863 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff110006ce0 0x7ff110007170 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7ff1140045c0 tx=0x7ff114004a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:55:31.293 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.275+0000 7ff1297fa640 1 -- 172.21.15.42:0/695820863 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff114014710 con 0x7ff110006ce0 2024-09-16T03:55:31.293 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.275+0000 7ff1297fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:55:31.293 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.275+0000 7ff1297fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:55:31.293 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.275+0000 7ff1297fa640 10 monclient: dump: 2024-09-16T03:55:31.293 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:55:31.293 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:55:31.293 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:31.293 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:31.293 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:55:31.294 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:55:31.294 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:55:31.294 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:55:31.294 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:55:31.294 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:55:31.294 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.275+0000 7ff1297fa640 1 -- 172.21.15.42:0/695820863 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4882 (secure 0 0 0) 0x7ff114014710 con 0x7ff110006ce0 2024-09-16T03:55:31.294 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.275+0000 7ff1297fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:55:31.294 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.275+0000 7ff1297fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:55:31.294 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.275+0000 7ff1297fa640 1 -- 172.21.15.42:0/695820863 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff110006ce0 msgr2=0x7ff110007170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:55:31.294 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.275+0000 7ff1297fa640 1 --2- 172.21.15.42:0/695820863 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff110006ce0 0x7ff110007170 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7ff1140045c0 tx=0x7ff114004a50 comp rx=0 tx=0).stop 2024-09-16T03:55:31.294 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.275+0000 7ff1297fa640 1 -- 172.21.15.42:0/695820863 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff110006ce0 msgr2=0x7ff110007170 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:55:31.294 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.275+0000 7ff1297fa640 1 --2- 172.21.15.42:0/695820863 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff110006ce0 0x7ff110007170 secure :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7ff1140045c0 tx=0x7ff114004a50 comp rx=0 tx=0).stop 2024-09-16T03:55:31.295 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:55:31.295 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:55:31.295 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:55:31.295 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "leader", 2024-09-16T03:55:31.295 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 116, 2024-09-16T03:55:31.295 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 221129, 2024-09-16T03:55:31.295 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:55:31.295 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:55:31.295 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T03:55:31.295 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T03:55:31.295 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:55:31.295 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 3, 2024-09-16T03:55:31.295 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:55:31.296 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:55:31.296 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:55:31.296 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:55:31.296 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:55:31.296 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:55:31.296 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:55:31.296 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:55:31.296 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:55:31.296 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:55:31.296 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:55:31.296 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:55:31.296 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:55:31.296 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:55:31.296 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:55:31.297 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:55:31.297 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:55:31.297 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:55:31.297 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:55:31.297 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:55:31.297 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:55:31.297 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:55:31.297 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:55:31.297 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:55:31.297 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:55:31.297 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:55:31.297 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:55:31.297 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:55:31.297 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:55:31.298 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:55:31.298 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:55:31.298 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:55:31.298 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:55:31.298 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:55:31.298 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:55:31.298 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:55:31.298 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:55:31.298 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:55:31.298 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:55:31.298 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:55:31.298 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:55:31.298 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:55:31.298 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:55:31.299 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:55:31.299 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:55:31.299 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:55:31.299 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:55:31.299 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:55:31.299 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:55:31.299 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:55:31.299 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:55:31.299 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:55:31.299 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:55:31.299 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:55:31.299 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:55:31.299 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:55:31.299 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:55:31.300 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:55:31.300 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:55:31.300 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:55:31.300 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:55:31.300 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:55:31.300 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:55:31.300 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:55:31.300 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:55:31.300 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:55:31.300 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:55:31.300 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:55:31.300 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:55:31.300 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:55:31.300 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:55:31.301 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:55:31.301 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:55:31.301 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:55:31.301 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:55:31.301 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:55:31.301 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:55:31.301 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:55:31.301 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:55:31.301 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:55:31.301 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:55:31.301 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:55:31.301 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:55:31.301 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:55:31.301 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:55:31.302 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:55:31.302 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:55:31.302 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:55:31.302 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:55:31.302 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:55:31.302 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:55:31.302 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:55:31.302 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:55:31.302 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:55:31.302 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:55:31.302 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:55:31.302 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:55:31.302 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:55:31.302 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:55:31.303 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:55:31.303 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:55:31.303 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:55:31.303 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:55:31.303 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:55:31.303 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:55:31.303 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:55:31.303 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:55:31.303 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:55:31.303 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:55:31.303 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:55:31.304 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:55:31.304 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:55:31.304 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:55:31.304 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:55:31.304 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:55:31.304 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:55:31.304 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:55:31.304 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:55:31.304 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:55:31.304 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:55:31.304 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:55:31.304 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:55:31.304 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:55:31.305 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:55:31.305 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:55:31.305 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:55:31.305 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:55:31.305 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:55:31.305 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:55:31.305 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:55:31.305 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:55:31.305 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:55:31.305 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:55:31.305 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:55:31.305 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:55:31.305 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:55:31.305 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:55:31.306 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:55:31.306 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:55:31.306 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:55:31.306 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:55:31.306 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T03:55:31.306 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:55:31.306 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:55:31.306 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:55:31.306 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 6 2024-09-16T03:55:31.306 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:55:31.306 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:55:31.306 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:55:31.306 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:55:31.306 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:55:31.307 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:55:31.307 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 7 2024-09-16T03:55:31.307 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:55:31.307 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:55:31.307 INFO:teuthology.orchestra.run.smithi042.stdout: "mgr": [ 2024-09-16T03:55:31.307 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:55:31.307 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:55:31.307 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:55:31.307 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:55:31.307 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:55:31.307 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:55:31.307 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:55:31.307 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:55:31.307 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:55:31.309 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.275+0000 7ff133197640 10 librados: watch_flush enter 2024-09-16T03:55:31.309 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.275+0000 7ff133197640 10 librados: watch_flush exit 2024-09-16T03:55:31.309 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.275+0000 7ff133197640 20 client.854219.objecter shutdown clearing up homeless session... 2024-09-16T03:55:31.309 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.275+0000 7ff133197640 10 client.854219.objecter successfully canceled tick 2024-09-16T03:55:31.309 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.275+0000 7ff133197640 1 -- 172.21.15.42:0/695820863 >> v2:172.21.15.154:6812/254845342 conn(0x7ff10c037810 msgr2=0x7ff10c039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:55:31.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.275+0000 7ff133197640 1 --2- 172.21.15.42:0/695820863 >> v2:172.21.15.154:6812/254845342 conn(0x7ff10c037810 0x7ff10c039cd0 secure :-1 s=READY pgs=3638 cs=0 l=1 rev1=1 crypto rx=0x7ff124005890 tx=0x7ff124009040 comp rx=0 tx=0).stop 2024-09-16T03:55:31.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.275+0000 7ff133197640 10 monclient: shutdown 2024-09-16T03:55:31.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.275+0000 7ff133197640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:55:31.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.275+0000 7ff133197640 1 -- 172.21.15.42:0/695820863 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff12c0a3d90 msgr2=0x7ff12c13d240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:55:31.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.275+0000 7ff133197640 1 --2- 172.21.15.42:0/695820863 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff12c0a3d90 0x7ff12c13d240 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7ff11c0ae040 tx=0x7ff11c0190e0 comp rx=0 tx=0).stop 2024-09-16T03:55:31.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.275+0000 7ff131734640 1 -- 172.21.15.42:0/695820863 reap_dead start 2024-09-16T03:55:31.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.275+0000 7ff133197640 1 -- 172.21.15.42:0/695820863 shutdown_connections 2024-09-16T03:55:31.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.275+0000 7ff133197640 1 --2- 172.21.15.42:0/695820863 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff12c0a3d90 0x7ff12c13d240 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:55:31.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.275+0000 7ff133197640 1 -- 172.21.15.42:0/695820863 >> 172.21.15.42:0/695820863 conn(0x7ff12c0a9fb0 msgr2=0x7ff12c131e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:55:31.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.275+0000 7ff133197640 1 -- 172.21.15.42:0/695820863 shutdown_connections 2024-09-16T03:55:31.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.275+0000 7ff133197640 1 -- 172.21.15.42:0/695820863 wait complete. 2024-09-16T03:55:31.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.275+0000 7ff133197640 1 librados: shutdown 2024-09-16T03:55:31.311 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:55:31.311 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T03:55:31.311 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T03:55:31.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.419+0000 7f620a709640 10 monclient: get_monmap_and_config 2024-09-16T03:55:31.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.419+0000 7f620a709640 10 monclient: build_initial_monmap 2024-09-16T03:55:31.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.419+0000 7f620a709640 10 monclient: monmap: 2024-09-16T03:55:31.428 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:55:31.428 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:55:31.428 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:55:31.422979+0000 2024-09-16T03:55:31.429 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:55:31.422979+0000 2024-09-16T03:55:31.429 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:55:31.429 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:55:31.429 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:55:31.430 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:55:31.430 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:55:31.430 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:55:31.430 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:55:31.430 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:55:31.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.419+0000 7f620a709640 1 Processor -- start 2024-09-16T03:55:31.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.423+0000 7f620a709640 1 -- start start 2024-09-16T03:55:31.432 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.423+0000 7f620a709640 10 monclient: init 2024-09-16T03:55:31.432 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.423+0000 7f620a709640 10 monclient: _reopen_session rank -1 2024-09-16T03:55:31.432 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.423+0000 7f620a709640 10 monclient: _add_conns ranks=[1,0,2,3,4] 2024-09-16T03:55:31.432 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.423+0000 7f620a709640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f62040a39b0 addr v1:172.21.15.154:6789/0 2024-09-16T03:55:31.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.423+0000 7f620a709640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f62040a3d90 addr v1:172.21.15.42:6789/0 2024-09-16T03:55:31.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.423+0000 7f620a709640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f620413daa0 0x7f620413fe90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:55:31.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.423+0000 7f620a709640 10 monclient(hunting): picked mon.noname-b con 0x7f620413daa0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:55:31.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.423+0000 7f620a709640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6204134290 con 0x7f62040a3d90 2024-09-16T03:55:31.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.423+0000 7f620a709640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f620409c140 con 0x7f62040a39b0 2024-09-16T03:55:31.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.423+0000 7f620a709640 10 monclient(hunting): start opening mon connection 2024-09-16T03:55:31.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.423+0000 7f620a709640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f620409c420 con 0x7f620413daa0 2024-09-16T03:55:31.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.423+0000 7f620a709640 10 monclient(hunting): _renew_subs 2024-09-16T03:55:31.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.423+0000 7f620a709640 10 monclient(hunting): authenticate will time out at 1778.392706s 2024-09-16T03:55:31.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.423+0000 7f6208ca6640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f620413daa0 0x7f620413fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:55:31.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.423+0000 7f62037fe640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f62040a3d90 0x7f620413c380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:49442/0 (socket says 172.21.15.42:49442) 2024-09-16T03:55:31.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.423+0000 7f62037fe640 1 -- 172.21.15.42:0/3065140721 learned_addr learned my addr 172.21.15.42:0/3065140721 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:55:31.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.423+0000 7f6208ca6640 10 monclient(hunting): get_auth_request con 0x7f620413daa0 auth_method 0 2024-09-16T03:55:31.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.423+0000 7f6208ca6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:55:31.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.423+0000 7f6208ca6640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:55:31.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.423+0000 7f6208ca6640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:55:31.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.423+0000 7f6208ca6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:55:31.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.423+0000 7f6208ca6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:55:31.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.423+0000 7f6208ca6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:55:31.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.423+0000 7f6208ca6640 10 monclient(hunting): handle_auth_done global_id 854222 payload 274 2024-09-16T03:55:31.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.423+0000 7f6208ca6640 1 -- 172.21.15.42:0/3065140721 >> v1:172.21.15.154:6789/0 conn(0x7f62040a39b0 legacy=0x7f6204133b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:55:31.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.423+0000 7f6208ca6640 1 -- 172.21.15.42:0/3065140721 >> v1:172.21.15.42:6789/0 conn(0x7f62040a3d90 legacy=0x7f620413c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:55:31.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.423+0000 7f6208ca6640 10 monclient: _finish_hunting 0 2024-09-16T03:55:31.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.423+0000 7f6208ca6640 1 monclient: found mon.noname-b 2024-09-16T03:55:31.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.423+0000 7f6208ca6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:55:31.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.423+0000 7f6208ca6640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T03:55:31.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.423+0000 7f6208ca6640 1 -- 172.21.15.42:0/3065140721 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f620406dd60 con 0x7f620413daa0 2024-09-16T03:55:31.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.423+0000 7f6208ca6640 1 --2- 172.21.15.42:0/3065140721 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f620413daa0 0x7f620413fe90 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f61fc009dc0 tx=0x7f61fc01c850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=58e914cbf905c8c4 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:55:31.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.423+0000 7f6202ffd640 1 -- 172.21.15.42:0/3065140721 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61fc0b2070 con 0x7f620413daa0 2024-09-16T03:55:31.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.423+0000 7f6202ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:55:31.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.423+0000 7f6202ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:55:31.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.423+0000 7f6202ffd640 10 monclient: dump: 2024-09-16T03:55:31.452 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:55:31.452 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:55:31.452 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:31.452 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:31.452 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:55:31.452 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:55:31.452 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:55:31.453 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:55:31.453 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:55:31.453 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:55:31.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.423+0000 7f6202ffd640 10 monclient: _finish_auth 0 2024-09-16T03:55:31.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.423+0000 7f6202ffd640 10 monclient: _check_auth_tickets 2024-09-16T03:55:31.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.423+0000 7f6202ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:55:31.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.423+0000 7f6202ffd640 1 -- 172.21.15.42:0/3065140721 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f61fc0a5710 con 0x7f620413daa0 2024-09-16T03:55:31.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.423+0000 7f620a709640 5 monclient: authenticate success, global_id 854222 2024-09-16T03:55:31.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.423+0000 7f620a709640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:55:31.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.423+0000 7f6202ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:55:31.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.423+0000 7f620a709640 10 monclient: get_monmap_and_config success 2024-09-16T03:55:31.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.423+0000 7f6202ffd640 1 -- 172.21.15.42:0/3065140721 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61fc0a5ae0 con 0x7f620413daa0 2024-09-16T03:55:31.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.423+0000 7f6202ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:55:31.454 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.423+0000 7f620a709640 4 set_mon_vals no callback set 2024-09-16T03:55:31.454 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.423+0000 7f6202ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:55:31.454 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.423+0000 7f6202ffd640 10 monclient: dump: 2024-09-16T03:55:31.454 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:55:31.454 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:55:31.454 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:31.454 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:31.454 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:55:31.454 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:55:31.454 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:55:31.455 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:55:31.455 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:55:31.455 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:55:31.455 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.427+0000 7f620a709640 10 monclient: shutdown 2024-09-16T03:55:31.455 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.427+0000 7f620a709640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:55:31.455 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.427+0000 7f620a709640 1 -- 172.21.15.42:0/3065140721 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f620413daa0 msgr2=0x7f620413fe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:55:31.455 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.427+0000 7f620a709640 1 --2- 172.21.15.42:0/3065140721 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f620413daa0 0x7f620413fe90 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f61fc009dc0 tx=0x7f61fc01c850 comp rx=0 tx=0).stop 2024-09-16T03:55:31.455 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.427+0000 7f620a709640 1 -- 172.21.15.42:0/3065140721 shutdown_connections 2024-09-16T03:55:31.455 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.427+0000 7f620a709640 1 --2- 172.21.15.42:0/3065140721 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f620413daa0 0x7f620413fe90 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:55:31.455 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.427+0000 7f620a709640 1 -- 172.21.15.42:0/3065140721 >> 172.21.15.42:0/3065140721 conn(0x7f62040a9fb0 msgr2=0x7f62040aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:55:31.455 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.427+0000 7f620a709640 1 -- 172.21.15.42:0/3065140721 shutdown_connections 2024-09-16T03:55:31.455 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.427+0000 7f620a709640 1 -- 172.21.15.42:0/3065140721 wait complete. 2024-09-16T03:55:31.456 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.427+0000 7f620a709640 10 monclient: build_initial_monmap 2024-09-16T03:55:31.456 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.427+0000 7f620a709640 10 monclient: monmap: 2024-09-16T03:55:31.456 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:55:31.456 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:55:31.456 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:55:31.456 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:55:31.456 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:55:31.456 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:55:31.456 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:55:31.456 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:55:31.457 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:55:31.457 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:55:31.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.427+0000 7f620a709640 1 librados: starting msgr at 2024-09-16T03:55:31.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.427+0000 7f620a709640 1 librados: starting objecter 2024-09-16T03:55:31.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.427+0000 7f620a709640 1 Processor -- start 2024-09-16T03:55:31.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.427+0000 7f620a709640 1 -- start start 2024-09-16T03:55:31.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.427+0000 7f620a709640 1 librados: setting wanted keys 2024-09-16T03:55:31.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.427+0000 7f620a709640 1 librados: calling monclient init 2024-09-16T03:55:31.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.427+0000 7f620a709640 10 monclient: init 2024-09-16T03:55:31.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.431+0000 7f620a709640 10 monclient: _reopen_session rank -1 2024-09-16T03:55:31.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.431+0000 7f620a709640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:55:31.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.431+0000 7f620a709640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f62040a39b0 0x7f62041309b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:55:31.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.431+0000 7f620a709640 10 monclient(hunting): picked mon.noname-c con 0x7f62040a39b0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:55:31.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.431+0000 7f620a709640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f62040a3d90 0x7f620413ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:55:31.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.431+0000 7f620a709640 10 monclient(hunting): picked mon.noname-a con 0x7f62040a3d90 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:55:31.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.431+0000 7f620a709640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f620413daa0 0x7f620413b160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:55:31.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.431+0000 7f620a709640 10 monclient(hunting): picked mon.noname-b con 0x7f620413daa0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:55:31.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.431+0000 7f620a709640 10 monclient(hunting): start opening mon connection 2024-09-16T03:55:31.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.431+0000 7f620a709640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f6204142f80 con 0x7f62040a3d90 2024-09-16T03:55:31.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.431+0000 7f620a709640 10 monclient(hunting): start opening mon connection 2024-09-16T03:55:31.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.431+0000 7f620a709640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f6204142e00 con 0x7f620413daa0 2024-09-16T03:55:31.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.431+0000 7f620a709640 10 monclient(hunting): start opening mon connection 2024-09-16T03:55:31.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.431+0000 7f620a709640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f6204143100 con 0x7f62040a39b0 2024-09-16T03:55:31.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.431+0000 7f620a709640 10 monclient(hunting): _renew_subs 2024-09-16T03:55:31.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.431+0000 7f620a709640 10 monclient(hunting): authenticate will time out at 1778.400552s 2024-09-16T03:55:31.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.431+0000 7f6208ca6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f620413daa0 0x7f620413b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:55:31.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.431+0000 7f6208ca6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f620413daa0 0x7f620413b160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.42:47298/0 (socket says 172.21.15.42:47298) 2024-09-16T03:55:31.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.431+0000 7f6208ca6640 1 -- 172.21.15.42:0/3238981363 learned_addr learned my addr 172.21.15.42:0/3238981363 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:55:31.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.431+0000 7f6208ca6640 10 monclient(hunting): get_auth_request con 0x7f620413daa0 auth_method 0 2024-09-16T03:55:31.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.431+0000 7f6208ca6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:55:31.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.431+0000 7f6208ca6640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:55:31.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.431+0000 7f6208ca6640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:55:31.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.431+0000 7f6208ca6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:55:31.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.431+0000 7f6208ca6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:55:31.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.431+0000 7f6208ca6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:55:31.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.431+0000 7f62037fe640 1 --2- 172.21.15.42:0/3238981363 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f62040a3d90 0x7f620413ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:55:31.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.431+0000 7f62037fe640 10 monclient(hunting): get_auth_request con 0x7f62040a3d90 auth_method 0 2024-09-16T03:55:31.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.431+0000 7f62037fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:55:31.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.431+0000 7f62037fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:55:31.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.431+0000 7f62037fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:55:31.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.431+0000 7f62037fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:55:31.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.431+0000 7f62037fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:55:31.460 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.431+0000 7f62037fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:55:31.460 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.431+0000 7f6208ca6640 10 monclient(hunting): handle_auth_done global_id 864127 payload 824 2024-09-16T03:55:31.460 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.431+0000 7f6208ca6640 1 -- 172.21.15.42:0/3238981363 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f62040a39b0 msgr2=0x7f62041309b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:55:31.460 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.431+0000 7f6208ca6640 1 --2- 172.21.15.42:0/3238981363 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f62040a39b0 0x7f62041309b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:55:31.460 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.431+0000 7f6208ca6640 1 -- 172.21.15.42:0/3238981363 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f62040a3d90 msgr2=0x7f620413ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:55:31.460 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.431+0000 7f6208ca6640 1 --2- 172.21.15.42:0/3238981363 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f62040a3d90 0x7f620413ac20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:55:31.460 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.431+0000 7f6208ca6640 10 monclient: _finish_hunting 0 2024-09-16T03:55:31.460 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.431+0000 7f6208ca6640 1 monclient: found mon.noname-b 2024-09-16T03:55:31.460 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.431+0000 7f6208ca6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:55:31.460 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.431+0000 7f6208ca6640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.154:3300/0 2024-09-16T03:55:31.460 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.431+0000 7f6208ca6640 1 -- 172.21.15.42:0/3238981363 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f620409d990 con 0x7f620413daa0 2024-09-16T03:55:31.460 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.431+0000 7f6208ca6640 1 --2- 172.21.15.42:0/3238981363 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f620413daa0 0x7f620413b160 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f61fc098230 tx=0x7f61fc0195f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:55:31.460 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.431+0000 7f6200ff9640 10 client.?.objecter ms_handle_connect 0x7f620413daa0 2024-09-16T03:55:31.460 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.431+0000 7f6200ff9640 10 client.?.objecter resend_mon_ops 2024-09-16T03:55:31.461 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.435+0000 7f6200ff9640 1 -- 172.21.15.42:0/3238981363 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61fc0b2050 con 0x7f620413daa0 2024-09-16T03:55:31.461 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.435+0000 7f6200ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:55:31.461 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.435+0000 7f6200ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:55:31.461 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.435+0000 7f6200ff9640 10 monclient: dump: 2024-09-16T03:55:31.461 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:55:31.461 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:55:31.461 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:31.461 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:31.461 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:55:31.461 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:55:31.461 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:55:31.461 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:55:31.461 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:55:31.461 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:55:31.462 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.435+0000 7f6200ff9640 10 monclient: _finish_auth 0 2024-09-16T03:55:31.462 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.435+0000 7f6200ff9640 10 monclient: _check_auth_tickets 2024-09-16T03:55:31.462 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.435+0000 7f6200ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:55:31.462 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.435+0000 7f6200ff9640 1 -- 172.21.15.42:0/3238981363 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f61fc0be070 con 0x7f620413daa0 2024-09-16T03:55:31.462 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.435+0000 7f6200ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T03:55:31.462 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.435+0000 7f6200ff9640 1 -- 172.21.15.42:0/3238981363 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61fc0adc20 con 0x7f620413daa0 2024-09-16T03:55:31.462 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.435+0000 7f620a709640 5 monclient: authenticate success, global_id 864127 2024-09-16T03:55:31.462 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.435+0000 7f620a709640 10 monclient: _renew_subs 2024-09-16T03:55:31.462 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.435+0000 7f620a709640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2024-09-16T03:55:31.462 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.435+0000 7f620a709640 1 -- 172.21.15.42:0/3238981363 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62040a93a0 con 0x7f620413daa0 2024-09-16T03:55:31.462 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.435+0000 7f6200ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:55:31.462 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.435+0000 7f6200ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:55:31.462 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.435+0000 7f6200ff9640 10 monclient: dump: 2024-09-16T03:55:31.462 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:55:31.463 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:55:31.463 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:31.463 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:31.463 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:55:31.463 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:55:31.463 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:55:31.463 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:55:31.463 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:55:31.463 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:55:31.463 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.435+0000 7f62027fc640 4 set_mon_vals no callback set 2024-09-16T03:55:31.463 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.435+0000 7f620a709640 10 client.864127.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:55:31.463 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.435+0000 7f620a709640 10 monclient: _renew_subs 2024-09-16T03:55:31.463 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.435+0000 7f620a709640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2024-09-16T03:55:31.463 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.435+0000 7f620a709640 1 -- 172.21.15.42:0/3238981363 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f620409d6b0 con 0x7f620413daa0 2024-09-16T03:55:31.463 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.435+0000 7f620a709640 1 librados: init done 2024-09-16T03:55:31.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.435+0000 7f620a709640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:55:31.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.435+0000 7f620a709640 1 --2- 172.21.15.42:0/3238981363 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f6204130ef0 0x7f620413b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:55:31.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.435+0000 7f620a709640 10 monclient(hunting): start opening mon connection 2024-09-16T03:55:31.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.435+0000 7f620a709640 1 -- 172.21.15.42:0/3238981363 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f620413fea0 con 0x7f6204130ef0 2024-09-16T03:55:31.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.435+0000 7f6203fff640 1 --2- 172.21.15.42:0/3238981363 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f6204130ef0 0x7f620413b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:55:31.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.435+0000 7f6203fff640 10 monclient: get_auth_request con 0x7f6204130ef0 auth_method 0 2024-09-16T03:55:31.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.435+0000 7f6203fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:55:31.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.435+0000 7f6203fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:55:31.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.435+0000 7f6203fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:55:31.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.435+0000 7f6203fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:55:31.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.435+0000 7f6203fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:55:31.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.435+0000 7f6203fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:55:31.465 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.435+0000 7f6200ff9640 1 -- 172.21.15.42:0/3238981363 <== mon.1 v2:172.21.15.154:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f61fc0c3460 con 0x7f620413daa0 2024-09-16T03:55:31.465 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.435+0000 7f6203fff640 10 monclient(hunting): handle_auth_done global_id 854225 payload 824 2024-09-16T03:55:31.465 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.435+0000 7f6203fff640 1 -- 172.21.15.42:0/3238981363 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f620413cc70 con 0x7f6204130ef0 2024-09-16T03:55:31.465 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.435+0000 7f6200ff9640 1 --2- 172.21.15.42:0/3238981363 >> v2:172.21.15.154:6812/254845342 conn(0x7f61f0037890 0x7f61f0039d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:55:31.465 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.435+0000 7f6200ff9640 1 -- 172.21.15.42:0/3238981363 <== mon.1 v2:172.21.15.154:3300/0 5 ==== osd_map(418..418 src has 330..418) ==== 6096+0+0 (secure 0 0 0) 0x7f61fc0e4300 con 0x7f620413daa0 2024-09-16T03:55:31.465 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.435+0000 7f6200ff9640 10 client.864127.objecter ms_dispatch 0x7f6204000ba0 osd_map(418..418 src has 330..418) 2024-09-16T03:55:31.465 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.435+0000 7f6200ff9640 3 client.864127.objecter handle_osd_map got epochs [418,418] > 0 2024-09-16T03:55:31.465 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.435+0000 7f6200ff9640 3 client.864127.objecter handle_osd_map decoding full epoch 418 2024-09-16T03:55:31.465 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.435+0000 7f6203fff640 1 --2- 172.21.15.42:0/3238981363 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f6204130ef0 0x7f620413b6a0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f61f400c340 tx=0x7f61f400c800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:55:31.465 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.435+0000 7f6200ff9640 20 client.864127.objecter dump_active .. 0 homeless 2024-09-16T03:55:31.465 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.435+0000 7f6200ff9640 10 client.864127.objecter ms_handle_connect 0x7f6204130ef0 2024-09-16T03:55:31.465 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.435+0000 7f6200ff9640 10 client.864127.objecter resend_mon_ops 2024-09-16T03:55:31.465 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.435+0000 7f6200ff9640 1 -- 172.21.15.42:0/3238981363 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61f40052d0 con 0x7f6204130ef0 2024-09-16T03:55:31.466 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.435+0000 7f6200ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:55:31.466 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.435+0000 7f6200ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:55:31.466 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.435+0000 7f6200ff9640 10 monclient: dump: 2024-09-16T03:55:31.466 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:55:31.466 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:55:31.466 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:31.466 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:31.466 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:55:31.466 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:55:31.467 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:55:31.467 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:55:31.467 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:55:31.467 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:55:31.467 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.435+0000 7f6200ff9640 1 -- 172.21.15.42:0/3238981363 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f61f40052d0 con 0x7f6204130ef0 2024-09-16T03:55:31.467 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.435+0000 7f6200ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:55:31.467 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.435+0000 7f6200ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:55:31.467 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.435+0000 7f6200ff9640 1 -- 172.21.15.42:0/3238981363 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f6204130ef0 msgr2=0x7f620413b6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:55:31.467 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.435+0000 7f6200ff9640 1 --2- 172.21.15.42:0/3238981363 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f6204130ef0 0x7f620413b6a0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f61f400c340 tx=0x7f61f400c800 comp rx=0 tx=0).stop 2024-09-16T03:55:31.467 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.435+0000 7f6200ff9640 1 -- 172.21.15.42:0/3238981363 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f6204130ef0 msgr2=0x7f620413b6a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:55:31.467 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.435+0000 7f6200ff9640 1 --2- 172.21.15.42:0/3238981363 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f6204130ef0 0x7f620413b6a0 secure :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f61f400c340 tx=0x7f61f400c800 comp rx=0 tx=0).stop 2024-09-16T03:55:31.467 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.435+0000 7f62037fe640 1 --2- 172.21.15.42:0/3238981363 >> v2:172.21.15.154:6812/254845342 conn(0x7f61f0037890 0x7f61f0039d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:55:31.467 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.439+0000 7f62037fe640 10 monclient: get_auth_request con 0x7f61f0037890 auth_method 0 2024-09-16T03:55:31.467 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.439+0000 7f62037fe640 1 --2- 172.21.15.42:0/3238981363 >> v2:172.21.15.154:6812/254845342 conn(0x7f61f0037890 0x7f61f0039d50 secure :-1 s=READY pgs=3639 cs=0 l=1 rev1=1 crypto rx=0x7f620413c3d0 tx=0x7f61f8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:55:31.468 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.439+0000 7f6200ff9640 10 client.864127.objecter ms_handle_connect 0x7f61f0037890 2024-09-16T03:55:31.468 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.443+0000 7f620a709640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:55:31.468 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.443+0000 7f620a709640 1 --2- 172.21.15.42:0/3238981363 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f620413bbe0 0x7f620413de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:55:31.468 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.443+0000 7f620a709640 10 monclient(hunting): start opening mon connection 2024-09-16T03:55:31.468 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.443+0000 7f620a709640 1 -- 172.21.15.42:0/3238981363 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f620409e370 con 0x7f620413bbe0 2024-09-16T03:55:31.468 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.443+0000 7f6203fff640 1 --2- 172.21.15.42:0/3238981363 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f620413bbe0 0x7f620413de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:55:31.468 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.443+0000 7f6203fff640 10 monclient: get_auth_request con 0x7f620413bbe0 auth_method 0 2024-09-16T03:55:31.468 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.443+0000 7f6203fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:55:31.468 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.443+0000 7f6203fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:55:31.468 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.443+0000 7f6203fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:55:31.468 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.443+0000 7f6203fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:55:31.468 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.443+0000 7f6203fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:55:31.468 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.443+0000 7f6203fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:55:31.469 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.443+0000 7f6203fff640 10 monclient(hunting): handle_auth_done global_id 854228 payload 824 2024-09-16T03:55:31.469 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.443+0000 7f6203fff640 1 -- 172.21.15.42:0/3238981363 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f6204133b80 con 0x7f620413bbe0 2024-09-16T03:55:31.469 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.443+0000 7f6203fff640 1 --2- 172.21.15.42:0/3238981363 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f620413bbe0 0x7f620413de80 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f61f4013000 tx=0x7f61f4000bd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:55:31.469 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.443+0000 7f6200ff9640 10 client.864127.objecter ms_handle_connect 0x7f620413bbe0 2024-09-16T03:55:31.469 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.443+0000 7f6200ff9640 10 client.864127.objecter resend_mon_ops 2024-09-16T03:55:31.469 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.443+0000 7f6200ff9640 1 -- 172.21.15.42:0/3238981363 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61f4011650 con 0x7f620413bbe0 2024-09-16T03:55:31.469 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.443+0000 7f6200ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:55:31.469 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.443+0000 7f6200ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:55:31.469 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.443+0000 7f6200ff9640 10 monclient: dump: 2024-09-16T03:55:31.469 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:55:31.469 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:55:31.469 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:31.469 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:31.469 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:55:31.470 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:55:31.470 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:55:31.470 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:55:31.470 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:55:31.470 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:55:31.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.443+0000 7f6200ff9640 1 -- 172.21.15.42:0/3238981363 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4545 (secure 0 0 0) 0x7f61f4011650 con 0x7f620413bbe0 2024-09-16T03:55:31.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.443+0000 7f6200ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:55:31.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.443+0000 7f6200ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:55:31.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.443+0000 7f6200ff9640 1 -- 172.21.15.42:0/3238981363 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f620413bbe0 msgr2=0x7f620413de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:55:31.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.443+0000 7f6200ff9640 1 --2- 172.21.15.42:0/3238981363 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f620413bbe0 0x7f620413de80 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f61f4013000 tx=0x7f61f4000bd0 comp rx=0 tx=0).stop 2024-09-16T03:55:31.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.443+0000 7f6200ff9640 1 -- 172.21.15.42:0/3238981363 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f620413bbe0 msgr2=0x7f620413de80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:55:31.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.443+0000 7f6200ff9640 1 --2- 172.21.15.42:0/3238981363 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f620413bbe0 0x7f620413de80 secure :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f61f4013000 tx=0x7f61f4000bd0 comp rx=0 tx=0).stop 2024-09-16T03:55:31.471 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:55:31.471 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:55:31.471 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:55:31.471 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T03:55:31.471 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 116, 2024-09-16T03:55:31.471 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 64879, 2024-09-16T03:55:31.471 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:55:31.471 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:55:31.471 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T03:55:31.471 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T03:55:31.471 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:55:31.472 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 3, 2024-09-16T03:55:31.472 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:55:31.472 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:55:31.472 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:55:31.472 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:55:31.472 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:55:31.472 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:55:31.472 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:55:31.472 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:55:31.472 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:55:31.472 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:55:31.472 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:55:31.472 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:55:31.472 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:55:31.473 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:55:31.473 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:55:31.473 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:55:31.473 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:55:31.473 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:55:31.473 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:55:31.473 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:55:31.473 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:55:31.473 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:55:31.473 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:55:31.473 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:55:31.473 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:55:31.473 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:55:31.473 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:55:31.474 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:55:31.474 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:55:31.474 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:55:31.474 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:55:31.474 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:55:31.474 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:55:31.474 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:55:31.474 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:55:31.474 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:55:31.474 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:55:31.474 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:55:31.474 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:55:31.474 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:55:31.474 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:55:31.475 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:55:31.475 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:55:31.475 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:55:31.475 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:55:31.475 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:55:31.475 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:55:31.475 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:55:31.475 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:55:31.475 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:55:31.475 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:55:31.475 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:55:31.475 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:55:31.475 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:55:31.475 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:55:31.476 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:55:31.476 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:55:31.476 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:55:31.476 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:55:31.476 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:55:31.476 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:55:31.476 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:55:31.476 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:55:31.476 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:55:31.476 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:55:31.477 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:55:31.477 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:55:31.477 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:55:31.477 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:55:31.477 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:55:31.477 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:55:31.477 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:55:31.477 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:55:31.477 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:55:31.477 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:55:31.477 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:55:31.477 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:55:31.477 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:55:31.478 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:55:31.478 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:55:31.478 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:55:31.478 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:55:31.478 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:55:31.478 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:55:31.478 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:55:31.478 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:55:31.478 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:55:31.478 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:55:31.478 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:55:31.478 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:55:31.478 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:55:31.478 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:55:31.478 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:55:31.479 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:55:31.479 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:55:31.479 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:55:31.479 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:55:31.479 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:55:31.479 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:55:31.479 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:55:31.479 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:55:31.479 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:55:31.479 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:55:31.479 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:55:31.479 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:55:31.479 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:55:31.480 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:55:31.480 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:55:31.480 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:55:31.480 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:55:31.480 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:55:31.480 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:55:31.480 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:55:31.480 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:55:31.480 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:55:31.480 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:55:31.480 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:55:31.480 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:55:31.480 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:55:31.480 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:55:31.481 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:55:31.481 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:55:31.481 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:55:31.481 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:55:31.481 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:55:31.481 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:55:31.481 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:55:31.481 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:55:31.481 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:55:31.481 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:55:31.481 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:55:31.481 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:55:31.481 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:55:31.481 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:55:31.482 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:55:31.482 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:55:31.482 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:55:31.482 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:55:31.482 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:55:31.482 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:55:31.482 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:55:31.482 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:55:31.482 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:55:31.482 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:55:31.482 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:55:31.482 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:55:31.482 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:55:31.482 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:55:31.483 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:55:31.483 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:55:31.483 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:55:31.485 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.443+0000 7f620a709640 10 librados: watch_flush enter 2024-09-16T03:55:31.485 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.443+0000 7f620a709640 10 librados: watch_flush exit 2024-09-16T03:55:31.485 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.443+0000 7f620a709640 20 client.864127.objecter shutdown clearing up homeless session... 2024-09-16T03:55:31.485 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.443+0000 7f620a709640 10 client.864127.objecter successfully canceled tick 2024-09-16T03:55:31.485 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.443+0000 7f620a709640 1 -- 172.21.15.42:0/3238981363 >> v2:172.21.15.154:6812/254845342 conn(0x7f61f0037890 msgr2=0x7f61f0039d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:55:31.485 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.443+0000 7f620a709640 1 --2- 172.21.15.42:0/3238981363 >> v2:172.21.15.154:6812/254845342 conn(0x7f61f0037890 0x7f61f0039d50 secure :-1 s=READY pgs=3639 cs=0 l=1 rev1=1 crypto rx=0x7f620413c3d0 tx=0x7f61f8009040 comp rx=0 tx=0).stop 2024-09-16T03:55:31.485 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.443+0000 7f620a709640 10 monclient: shutdown 2024-09-16T03:55:31.485 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.443+0000 7f620a709640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:55:31.485 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.443+0000 7f620a709640 1 -- 172.21.15.42:0/3238981363 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f620413daa0 msgr2=0x7f620413b160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:55:31.485 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.443+0000 7f620a709640 1 --2- 172.21.15.42:0/3238981363 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f620413daa0 0x7f620413b160 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f61fc098230 tx=0x7f61fc0195f0 comp rx=0 tx=0).stop 2024-09-16T03:55:31.485 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.443+0000 7f6208ca6640 1 -- 172.21.15.42:0/3238981363 reap_dead start 2024-09-16T03:55:31.485 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.443+0000 7f6208ca6640 1 -- 172.21.15.42:0/3238981363 reap_dead start 2024-09-16T03:55:31.485 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.443+0000 7f620a709640 1 -- 172.21.15.42:0/3238981363 shutdown_connections 2024-09-16T03:55:31.486 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.443+0000 7f620a709640 1 -- 172.21.15.42:0/3238981363 >> 172.21.15.42:0/3238981363 conn(0x7f62040a9fb0 msgr2=0x7f6204132fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:55:31.486 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.443+0000 7f620a709640 1 -- 172.21.15.42:0/3238981363 shutdown_connections 2024-09-16T03:55:31.486 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.443+0000 7f620a709640 1 -- 172.21.15.42:0/3238981363 wait complete. 2024-09-16T03:55:31.486 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.443+0000 7f620a709640 1 librados: shutdown 2024-09-16T03:55:31.486 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:55:31.486 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T03:55:31.486 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T03:55:31.601 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.599+0000 7fe181526640 10 monclient: get_monmap_and_config 2024-09-16T03:55:31.601 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.599+0000 7fe181526640 10 monclient: build_initial_monmap 2024-09-16T03:55:31.601 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.599+0000 7fe181526640 10 monclient: monmap: 2024-09-16T03:55:31.602 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:55:31.602 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:55:31.602 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:55:31.600356+0000 2024-09-16T03:55:31.602 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:55:31.600356+0000 2024-09-16T03:55:31.602 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:55:31.602 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:55:31.602 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:55:31.602 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:55:31.602 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:55:31.602 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:55:31.602 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:55:31.602 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:55:31.605 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.599+0000 7fe181526640 1 Processor -- start 2024-09-16T03:55:31.605 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.599+0000 7fe181526640 1 -- start start 2024-09-16T03:55:31.605 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.599+0000 7fe181526640 10 monclient: init 2024-09-16T03:55:31.605 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.599+0000 7fe181526640 10 monclient: _reopen_session rank -1 2024-09-16T03:55:31.605 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.599+0000 7fe181526640 10 monclient: _add_conns ranks=[4,0,2,1,3] 2024-09-16T03:55:31.605 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.599+0000 7fe181526640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fe17c0a39b0 0x7fe17c133b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:55:31.605 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.599+0000 7fe181526640 10 monclient(hunting): picked mon.noname-c con 0x7fe17c0a39b0 addr v2:172.21.15.154:3300/0 2024-09-16T03:55:31.605 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.599+0000 7fe181526640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe17c0a3d90 addr v1:172.21.15.42:6789/0 2024-09-16T03:55:31.605 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.599+0000 7fe181526640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe17c137710 0x7fe17c13f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:55:31.606 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.599+0000 7fe181526640 10 monclient(hunting): picked mon.noname-b con 0x7fe17c137710 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:55:31.606 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.599+0000 7fe181526640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe17c09c140 con 0x7fe17c0a3d90 2024-09-16T03:55:31.606 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.599+0000 7fe181526640 10 monclient(hunting): start opening mon connection 2024-09-16T03:55:31.606 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.599+0000 7fe181526640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fe17c09c420 con 0x7fe17c0a39b0 2024-09-16T03:55:31.606 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.599+0000 7fe181526640 10 monclient(hunting): start opening mon connection 2024-09-16T03:55:31.606 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.599+0000 7fe181526640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fe17c06dd60 con 0x7fe17c137710 2024-09-16T03:55:31.606 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.599+0000 7fe181526640 10 monclient(hunting): _renew_subs 2024-09-16T03:55:31.606 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.599+0000 7fe181526640 10 monclient(hunting): authenticate will time out at 1778.569748s 2024-09-16T03:55:31.606 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.599+0000 7fe17a7fc640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fe17c0a3d90 0x7fe17c137000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:49454/0 (socket says 172.21.15.42:49454) 2024-09-16T03:55:31.606 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.599+0000 7fe17a7fc640 1 -- 172.21.15.42:0/3824631385 learned_addr learned my addr 172.21.15.42:0/3824631385 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:55:31.606 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.599+0000 7fe17b7fe640 1 --2- 172.21.15.42:0/3824631385 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe17c137710 0x7fe17c13f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:55:31.606 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.599+0000 7fe17b7fe640 10 monclient(hunting): get_auth_request con 0x7fe17c137710 auth_method 0 2024-09-16T03:55:31.606 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.599+0000 7fe17b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:55:31.606 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.599+0000 7fe17b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:55:31.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.599+0000 7fe17b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:55:31.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.599+0000 7fe17b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:55:31.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.599+0000 7fe17b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:55:31.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.599+0000 7fe17affd640 1 --2- 172.21.15.42:0/3824631385 >> v2:172.21.15.154:3300/0 conn(0x7fe17c0a39b0 0x7fe17c133b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:55:31.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.599+0000 7fe17b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:55:31.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.599+0000 7fe17affd640 10 monclient(hunting): get_auth_request con 0x7fe17c0a39b0 auth_method 0 2024-09-16T03:55:31.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.599+0000 7fe17affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:55:31.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.599+0000 7fe17affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:55:31.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.599+0000 7fe17affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:55:31.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.599+0000 7fe179ffb640 1 -- 172.21.15.42:0/3824631385 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 482937311 0 0) 0x7fe17c09c140 con 0x7fe17c0a3d90 2024-09-16T03:55:31.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.599+0000 7fe179ffb640 10 monclient(hunting): _negotiate 2024-09-16T03:55:31.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.599+0000 7fe179ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:55:31.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.599+0000 7fe179ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:55:31.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.599+0000 7fe179ffb640 10 monclient(hunting): my global_id is 824619 2024-09-16T03:55:31.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.599+0000 7fe179ffb640 1 -- 172.21.15.42:0/3824631385 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe164003530 con 0x7fe17c0a3d90 2024-09-16T03:55:31.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.599+0000 7fe17affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:55:31.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.599+0000 7fe17affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:55:31.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.599+0000 7fe17affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:55:31.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.599+0000 7fe17b7fe640 10 monclient(hunting): handle_auth_done global_id 854231 payload 274 2024-09-16T03:55:31.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.599+0000 7fe17b7fe640 1 -- 172.21.15.42:0/3824631385 >> v2:172.21.15.154:3300/0 conn(0x7fe17c0a39b0 msgr2=0x7fe17c133b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:55:31.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.599+0000 7fe17b7fe640 1 --2- 172.21.15.42:0/3824631385 >> v2:172.21.15.154:3300/0 conn(0x7fe17c0a39b0 0x7fe17c133b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:55:31.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.599+0000 7fe17b7fe640 1 -- 172.21.15.42:0/3824631385 >> v1:172.21.15.42:6789/0 conn(0x7fe17c0a3d90 legacy=0x7fe17c137000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:55:31.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.599+0000 7fe17b7fe640 10 monclient: _finish_hunting 0 2024-09-16T03:55:31.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.599+0000 7fe17b7fe640 1 monclient: found mon.noname-b 2024-09-16T03:55:31.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.599+0000 7fe17b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:55:31.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.599+0000 7fe17b7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T03:55:31.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.599+0000 7fe17b7fe640 1 -- 172.21.15.42:0/3824631385 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe17c0a5e20 con 0x7fe17c137710 2024-09-16T03:55:31.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.603+0000 7fe17b7fe640 1 --2- 172.21.15.42:0/3824631385 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe17c137710 0x7fe17c13f3a0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fe16c0097c0 tx=0x7fe16c01c360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ceffa4a8a72b3706 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:55:31.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.603+0000 7fe179ffb640 1 -- 172.21.15.42:0/3824631385 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe16c0b0070 con 0x7fe17c137710 2024-09-16T03:55:31.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.603+0000 7fe179ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:55:31.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.603+0000 7fe179ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:55:31.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.603+0000 7fe179ffb640 10 monclient: dump: 2024-09-16T03:55:31.615 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:55:31.615 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:55:31.615 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:31.615 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:31.615 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:55:31.615 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:55:31.616 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:55:31.616 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:55:31.616 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:55:31.616 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:55:31.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.603+0000 7fe179ffb640 10 monclient: _finish_auth 0 2024-09-16T03:55:31.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.603+0000 7fe179ffb640 10 monclient: _check_auth_tickets 2024-09-16T03:55:31.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.603+0000 7fe179ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:55:31.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.603+0000 7fe179ffb640 1 -- 172.21.15.42:0/3824631385 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe16c0ab600 con 0x7fe17c137710 2024-09-16T03:55:31.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.603+0000 7fe179ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T03:55:31.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.603+0000 7fe179ffb640 1 -- 172.21.15.42:0/3824631385 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe16c0ab9d0 con 0x7fe17c137710 2024-09-16T03:55:31.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.603+0000 7fe181526640 5 monclient: authenticate success, global_id 854231 2024-09-16T03:55:31.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.603+0000 7fe181526640 10 monclient: get_monmap_and_config success 2024-09-16T03:55:31.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.603+0000 7fe181526640 4 set_mon_vals no callback set 2024-09-16T03:55:31.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.603+0000 7fe179ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:55:31.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.603+0000 7fe179ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:55:31.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.603+0000 7fe179ffb640 10 monclient: dump: 2024-09-16T03:55:31.617 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:55:31.617 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:55:31.617 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:31.617 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:31.617 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:55:31.617 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:55:31.617 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:55:31.617 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:55:31.617 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:55:31.617 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:55:31.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.603+0000 7fe181526640 10 monclient: shutdown 2024-09-16T03:55:31.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.603+0000 7fe181526640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:55:31.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.603+0000 7fe181526640 1 -- 172.21.15.42:0/3824631385 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe17c137710 msgr2=0x7fe17c13f3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:55:31.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.603+0000 7fe181526640 1 --2- 172.21.15.42:0/3824631385 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe17c137710 0x7fe17c13f3a0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fe16c0097c0 tx=0x7fe16c01c360 comp rx=0 tx=0).stop 2024-09-16T03:55:31.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.603+0000 7fe181526640 1 -- 172.21.15.42:0/3824631385 shutdown_connections 2024-09-16T03:55:31.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.603+0000 7fe181526640 1 --2- 172.21.15.42:0/3824631385 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe17c137710 0x7fe17c13f3a0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:55:31.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.603+0000 7fe181526640 1 --2- 172.21.15.42:0/3824631385 >> v2:172.21.15.154:3300/0 conn(0x7fe17c0a39b0 0x7fe17c133b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:55:31.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.603+0000 7fe181526640 1 -- 172.21.15.42:0/3824631385 >> 172.21.15.42:0/3824631385 conn(0x7fe17c0a9fb0 msgr2=0x7fe17c0aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:55:31.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.603+0000 7fe181526640 1 -- 172.21.15.42:0/3824631385 shutdown_connections 2024-09-16T03:55:31.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.603+0000 7fe181526640 1 -- 172.21.15.42:0/3824631385 wait complete. 2024-09-16T03:55:31.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.603+0000 7fe181526640 10 monclient: build_initial_monmap 2024-09-16T03:55:31.619 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.603+0000 7fe181526640 10 monclient: monmap: 2024-09-16T03:55:31.619 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:55:31.619 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:55:31.619 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:55:31.619 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:55:31.619 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:55:31.619 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:55:31.619 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:55:31.619 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:55:31.619 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:55:31.619 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:55:31.619 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.603+0000 7fe181526640 1 librados: starting msgr at 2024-09-16T03:55:31.619 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.603+0000 7fe181526640 1 librados: starting objecter 2024-09-16T03:55:31.620 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.603+0000 7fe181526640 1 Processor -- start 2024-09-16T03:55:31.620 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.603+0000 7fe181526640 1 -- start start 2024-09-16T03:55:31.620 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.603+0000 7fe181526640 1 librados: setting wanted keys 2024-09-16T03:55:31.620 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.603+0000 7fe181526640 1 librados: calling monclient init 2024-09-16T03:55:31.620 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.603+0000 7fe181526640 10 monclient: init 2024-09-16T03:55:31.620 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.603+0000 7fe181526640 10 monclient: _reopen_session rank -1 2024-09-16T03:55:31.621 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.603+0000 7fe181526640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:55:31.621 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.603+0000 7fe181526640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe17c0a39b0 0x7fe17c130560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:55:31.621 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.603+0000 7fe181526640 10 monclient(hunting): picked mon.noname-b con 0x7fe17c0a39b0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:55:31.621 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.603+0000 7fe181526640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe17c0a3d90 0x7fe17c130aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:55:31.621 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.603+0000 7fe181526640 10 monclient(hunting): picked mon.noname-a con 0x7fe17c0a3d90 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:55:31.621 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.603+0000 7fe181526640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe17c137710 0x7fe17c0a17e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:55:31.621 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.603+0000 7fe181526640 10 monclient(hunting): picked mon.noname-c con 0x7fe17c137710 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:55:31.621 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.603+0000 7fe181526640 10 monclient(hunting): start opening mon connection 2024-09-16T03:55:31.621 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.603+0000 7fe181526640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fe17c142310 con 0x7fe17c0a3d90 2024-09-16T03:55:31.621 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.603+0000 7fe181526640 10 monclient(hunting): start opening mon connection 2024-09-16T03:55:31.621 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.603+0000 7fe181526640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe17c142490 con 0x7fe17c0a39b0 2024-09-16T03:55:31.621 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.603+0000 7fe181526640 10 monclient(hunting): start opening mon connection 2024-09-16T03:55:31.621 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.603+0000 7fe181526640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fe17c142610 con 0x7fe17c137710 2024-09-16T03:55:31.621 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.603+0000 7fe181526640 10 monclient(hunting): _renew_subs 2024-09-16T03:55:31.621 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.603+0000 7fe181526640 10 monclient(hunting): authenticate will time out at 1778.574960s 2024-09-16T03:55:31.622 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.603+0000 7fe17a7fc640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe17c0a3d90 0x7fe17c130aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:55:31.622 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.603+0000 7fe17b7fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe17c137710 0x7fe17c0a17e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:55:31.622 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.603+0000 7fe17a7fc640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe17c0a3d90 0x7fe17c130aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:34874/0 (socket says 172.21.15.42:34874) 2024-09-16T03:55:31.622 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.603+0000 7fe17a7fc640 1 -- 172.21.15.42:0/3819635124 learned_addr learned my addr 172.21.15.42:0/3819635124 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:55:31.622 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.603+0000 7fe17a7fc640 10 monclient(hunting): get_auth_request con 0x7fe17c0a3d90 auth_method 0 2024-09-16T03:55:31.622 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.603+0000 7fe17a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:55:31.622 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.603+0000 7fe17a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:55:31.622 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.603+0000 7fe17a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:55:31.622 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.603+0000 7fe17b7fe640 10 monclient(hunting): get_auth_request con 0x7fe17c137710 auth_method 0 2024-09-16T03:55:31.622 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.603+0000 7fe17b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:55:31.622 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.603+0000 7fe17b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:55:31.622 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.603+0000 7fe17b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:55:31.622 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.603+0000 7fe17a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:55:31.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.603+0000 7fe17a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:55:31.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.603+0000 7fe17a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:55:31.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.603+0000 7fe17b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:55:31.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.603+0000 7fe17b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:55:31.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.603+0000 7fe17b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:55:31.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.603+0000 7fe17affd640 1 --2- 172.21.15.42:0/3819635124 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe17c0a39b0 0x7fe17c130560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:55:31.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.603+0000 7fe17affd640 10 monclient(hunting): get_auth_request con 0x7fe17c0a39b0 auth_method 0 2024-09-16T03:55:31.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.603+0000 7fe17affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:55:31.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.603+0000 7fe17affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:55:31.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.603+0000 7fe17affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:55:31.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.603+0000 7fe17a7fc640 10 monclient(hunting): handle_auth_done global_id 824622 payload 824 2024-09-16T03:55:31.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.603+0000 7fe17a7fc640 1 -- 172.21.15.42:0/3819635124 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe17c137710 msgr2=0x7fe17c0a17e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:55:31.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.603+0000 7fe17a7fc640 1 --2- 172.21.15.42:0/3819635124 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe17c137710 0x7fe17c0a17e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:55:31.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.603+0000 7fe17a7fc640 1 -- 172.21.15.42:0/3819635124 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe17c0a39b0 msgr2=0x7fe17c130560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:55:31.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.603+0000 7fe17a7fc640 1 --2- 172.21.15.42:0/3819635124 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe17c0a39b0 0x7fe17c130560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:55:31.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.603+0000 7fe17a7fc640 10 monclient: _finish_hunting 0 2024-09-16T03:55:31.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.603+0000 7fe17a7fc640 1 monclient: found mon.noname-a 2024-09-16T03:55:31.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.603+0000 7fe17a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:55:31.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.603+0000 7fe17a7fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:55:31.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.603+0000 7fe17a7fc640 1 -- 172.21.15.42:0/3819635124 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe17c139b20 con 0x7fe17c0a3d90 2024-09-16T03:55:31.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.603+0000 7fe17a7fc640 1 --2- 172.21.15.42:0/3819635124 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe17c0a3d90 0x7fe17c130aa0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fe17000c450 tx=0x7fe17000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:55:31.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.603+0000 7fe1637fe640 10 client.?.objecter ms_handle_connect 0x7fe17c0a3d90 2024-09-16T03:55:31.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.603+0000 7fe1637fe640 10 client.?.objecter resend_mon_ops 2024-09-16T03:55:31.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.603+0000 7fe1637fe640 1 -- 172.21.15.42:0/3819635124 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe170017070 con 0x7fe17c0a3d90 2024-09-16T03:55:31.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.603+0000 7fe1637fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:55:31.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.603+0000 7fe1637fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:55:31.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.603+0000 7fe1637fe640 10 monclient: dump: 2024-09-16T03:55:31.625 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:55:31.625 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:55:31.625 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:31.625 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:31.625 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:55:31.625 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:55:31.626 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:55:31.626 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:55:31.626 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:55:31.626 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:55:31.626 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.603+0000 7fe1637fe640 10 monclient: _finish_auth 0 2024-09-16T03:55:31.626 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.603+0000 7fe1637fe640 10 monclient: _check_auth_tickets 2024-09-16T03:55:31.626 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.603+0000 7fe1637fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:55:31.626 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.603+0000 7fe181526640 5 monclient: authenticate success, global_id 824622 2024-09-16T03:55:31.626 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.603+0000 7fe1637fe640 1 -- 172.21.15.42:0/3819635124 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe170005cf0 con 0x7fe17c0a3d90 2024-09-16T03:55:31.626 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.603+0000 7fe181526640 10 monclient: _renew_subs 2024-09-16T03:55:31.627 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.603+0000 7fe181526640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:55:31.627 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.603+0000 7fe181526640 1 -- 172.21.15.42:0/3819635124 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe17c13f570 con 0x7fe17c0a3d90 2024-09-16T03:55:31.629 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.603+0000 7fe1637fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:55:31.629 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.603+0000 7fe1637fe640 1 -- 172.21.15.42:0/3819635124 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe170007290 con 0x7fe17c0a3d90 2024-09-16T03:55:31.629 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.603+0000 7fe1637fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:55:31.629 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.603+0000 7fe1637fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:55:31.629 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.603+0000 7fe1637fe640 10 monclient: dump: 2024-09-16T03:55:31.629 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:55:31.629 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:55:31.629 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:31.629 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:31.629 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:55:31.630 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:55:31.630 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:55:31.630 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:55:31.630 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:55:31.630 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:55:31.630 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.603+0000 7fe181526640 10 client.824622.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:55:31.630 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.603+0000 7fe181526640 10 monclient: _renew_subs 2024-09-16T03:55:31.630 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.603+0000 7fe181526640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:55:31.630 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.603+0000 7fe181526640 1 -- 172.21.15.42:0/3819635124 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe17c137020 con 0x7fe17c0a3d90 2024-09-16T03:55:31.631 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.603+0000 7fe1797fa640 4 set_mon_vals no callback set 2024-09-16T03:55:31.631 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.603+0000 7fe181526640 1 librados: init done 2024-09-16T03:55:31.631 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.607+0000 7fe181526640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:55:31.631 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.607+0000 7fe181526640 1 --2- 172.21.15.42:0/3819635124 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe17c139620 0x7fe17c0a1d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:55:31.631 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.607+0000 7fe181526640 10 monclient(hunting): start opening mon connection 2024-09-16T03:55:31.631 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.607+0000 7fe181526640 1 -- 172.21.15.42:0/3819635124 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe17c1342d0 con 0x7fe17c139620 2024-09-16T03:55:31.631 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.607+0000 7fe1637fe640 1 -- 172.21.15.42:0/3819635124 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fe17001a070 con 0x7fe17c0a3d90 2024-09-16T03:55:31.631 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.607+0000 7fe17affd640 1 --2- 172.21.15.42:0/3819635124 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe17c139620 0x7fe17c0a1d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:55:31.631 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.607+0000 7fe17affd640 10 monclient: get_auth_request con 0x7fe17c139620 auth_method 0 2024-09-16T03:55:31.632 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.607+0000 7fe17affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:55:31.632 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.607+0000 7fe17affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:55:31.632 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.607+0000 7fe17affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:55:31.632 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.607+0000 7fe1637fe640 1 --2- 172.21.15.42:0/3819635124 >> v2:172.21.15.154:6812/254845342 conn(0x7fe168037810 0x7fe168039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:55:31.632 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.607+0000 7fe1637fe640 1 -- 172.21.15.42:0/3819635124 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(418..418 src has 330..418) ==== 6096+0+0 (secure 0 0 0) 0x7fe170048350 con 0x7fe17c0a3d90 2024-09-16T03:55:31.632 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.607+0000 7fe1637fe640 10 client.824622.objecter ms_dispatch 0x7fe17c000ba0 osd_map(418..418 src has 330..418) 2024-09-16T03:55:31.632 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.607+0000 7fe1637fe640 3 client.824622.objecter handle_osd_map got epochs [418,418] > 0 2024-09-16T03:55:31.632 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.607+0000 7fe1637fe640 3 client.824622.objecter handle_osd_map decoding full epoch 418 2024-09-16T03:55:31.632 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.607+0000 7fe17affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:55:31.632 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.607+0000 7fe17affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:55:31.632 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.607+0000 7fe17affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:55:31.632 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.607+0000 7fe1637fe640 20 client.824622.objecter dump_active .. 0 homeless 2024-09-16T03:55:31.632 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.607+0000 7fe17b7fe640 1 --2- 172.21.15.42:0/3819635124 >> v2:172.21.15.154:6812/254845342 conn(0x7fe168037810 0x7fe168039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:55:31.633 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.607+0000 7fe17b7fe640 10 monclient: get_auth_request con 0x7fe168037810 auth_method 0 2024-09-16T03:55:31.633 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.607+0000 7fe17affd640 10 monclient(hunting): handle_auth_done global_id 864136 payload 824 2024-09-16T03:55:31.633 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.607+0000 7fe17affd640 1 -- 172.21.15.42:0/3819635124 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe17c133b80 con 0x7fe17c139620 2024-09-16T03:55:31.633 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.607+0000 7fe1637fe640 10 client.824622.objecter ms_handle_connect 0x7fe17c139620 2024-09-16T03:55:31.633 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.607+0000 7fe17affd640 1 --2- 172.21.15.42:0/3819635124 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe17c139620 0x7fe17c0a1d20 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fe174007f00 tx=0x7fe1740115a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:55:31.633 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.607+0000 7fe1637fe640 10 client.824622.objecter resend_mon_ops 2024-09-16T03:55:31.633 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.607+0000 7fe1637fe640 1 -- 172.21.15.42:0/3819635124 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe17401a020 con 0x7fe17c139620 2024-09-16T03:55:31.633 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.607+0000 7fe1637fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:55:31.633 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.607+0000 7fe1637fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:55:31.633 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.607+0000 7fe1637fe640 10 monclient: dump: 2024-09-16T03:55:31.633 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:55:31.633 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:55:31.633 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:31.634 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:31.634 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:55:31.634 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:55:31.634 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:55:31.634 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:55:31.634 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:55:31.634 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:55:31.634 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.607+0000 7fe1637fe640 10 client.824622.objecter ms_handle_connect 0x7fe168037810 2024-09-16T03:55:31.635 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.607+0000 7fe17b7fe640 1 --2- 172.21.15.42:0/3819635124 >> v2:172.21.15.154:6812/254845342 conn(0x7fe168037810 0x7fe168039cd0 secure :-1 s=READY pgs=3640 cs=0 l=1 rev1=1 crypto rx=0x7fe16c0ad000 tx=0x7fe16c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:55:31.635 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.607+0000 7fe1637fe640 1 -- 172.21.15.42:0/3819635124 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fe174016470 con 0x7fe17c139620 2024-09-16T03:55:31.635 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.607+0000 7fe1637fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:55:31.635 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.607+0000 7fe1637fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:55:31.635 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.607+0000 7fe1637fe640 1 -- 172.21.15.42:0/3819635124 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe17c139620 msgr2=0x7fe17c0a1d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:55:31.635 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.607+0000 7fe1637fe640 1 --2- 172.21.15.42:0/3819635124 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe17c139620 0x7fe17c0a1d20 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fe174007f00 tx=0x7fe1740115a0 comp rx=0 tx=0).stop 2024-09-16T03:55:31.635 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.607+0000 7fe1637fe640 1 -- 172.21.15.42:0/3819635124 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe17c139620 msgr2=0x7fe17c0a1d20 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:55:31.635 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.607+0000 7fe1637fe640 1 --2- 172.21.15.42:0/3819635124 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe17c139620 0x7fe17c0a1d20 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:55:31.637 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.615+0000 7fe181526640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:55:31.637 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.615+0000 7fe181526640 1 --2- 172.21.15.42:0/3819635124 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe17c0a2260 0x7fe17c0a2670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:55:31.637 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.615+0000 7fe181526640 10 monclient(hunting): start opening mon connection 2024-09-16T03:55:31.637 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.615+0000 7fe181526640 1 -- 172.21.15.42:0/3819635124 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe17c0a2bb0 con 0x7fe17c0a2260 2024-09-16T03:55:31.637 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.615+0000 7fe17affd640 1 --2- 172.21.15.42:0/3819635124 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe17c0a2260 0x7fe17c0a2670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:55:31.637 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.615+0000 7fe17affd640 10 monclient: get_auth_request con 0x7fe17c0a2260 auth_method 0 2024-09-16T03:55:31.637 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.615+0000 7fe17affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:55:31.637 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.615+0000 7fe17affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:55:31.638 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.615+0000 7fe17affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:55:31.638 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.615+0000 7fe17affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:55:31.638 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.615+0000 7fe17affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:55:31.638 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.615+0000 7fe17affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:55:31.638 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.615+0000 7fe17affd640 10 monclient(hunting): handle_auth_done global_id 864139 payload 824 2024-09-16T03:55:31.638 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.615+0000 7fe17affd640 1 -- 172.21.15.42:0/3819635124 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fe17c0a5750 con 0x7fe17c0a2260 2024-09-16T03:55:31.638 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.615+0000 7fe17affd640 1 --2- 172.21.15.42:0/3819635124 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe17c0a2260 0x7fe17c0a2670 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fe17401b000 tx=0x7fe174004a50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:55:31.638 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.615+0000 7fe1637fe640 10 client.824622.objecter ms_handle_connect 0x7fe17c0a2260 2024-09-16T03:55:31.638 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.615+0000 7fe1637fe640 10 client.824622.objecter resend_mon_ops 2024-09-16T03:55:31.638 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.615+0000 7fe1637fe640 1 -- 172.21.15.42:0/3819635124 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe17401a030 con 0x7fe17c0a2260 2024-09-16T03:55:31.638 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.615+0000 7fe1637fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:55:31.638 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.615+0000 7fe1637fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:55:31.638 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.615+0000 7fe1637fe640 10 monclient: dump: 2024-09-16T03:55:31.639 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:55:31.639 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:55:31.639 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:31.639 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:31.639 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:55:31.639 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:55:31.639 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:55:31.640 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:55:31.640 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:55:31.640 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:55:31.640 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.615+0000 7fe1637fe640 1 -- 172.21.15.42:0/3819635124 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4544 (secure 0 0 0) 0x7fe174021220 con 0x7fe17c0a2260 2024-09-16T03:55:31.640 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.615+0000 7fe1637fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:55:31.640 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.615+0000 7fe1637fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:55:31.640 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.615+0000 7fe1637fe640 1 -- 172.21.15.42:0/3819635124 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe17c0a2260 msgr2=0x7fe17c0a2670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:55:31.640 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.615+0000 7fe1637fe640 1 --2- 172.21.15.42:0/3819635124 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe17c0a2260 0x7fe17c0a2670 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fe17401b000 tx=0x7fe174004a50 comp rx=0 tx=0).stop 2024-09-16T03:55:31.640 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.615+0000 7fe1637fe640 1 -- 172.21.15.42:0/3819635124 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe17c0a2260 msgr2=0x7fe17c0a2670 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:55:31.640 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.615+0000 7fe1637fe640 1 --2- 172.21.15.42:0/3819635124 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe17c0a2260 0x7fe17c0a2670 secure :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fe17401b000 tx=0x7fe174004a50 comp rx=0 tx=0).stop 2024-09-16T03:55:31.641 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:55:31.641 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:55:31.641 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:55:31.641 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T03:55:31.641 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 116, 2024-09-16T03:55:31.641 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 9907, 2024-09-16T03:55:31.641 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:55:31.641 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:55:31.641 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T03:55:31.641 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T03:55:31.641 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:55:31.641 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 3, 2024-09-16T03:55:31.641 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:55:31.642 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:55:31.642 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:55:31.642 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:55:31.642 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:55:31.642 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:55:31.642 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:55:31.642 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:55:31.642 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:55:31.642 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:55:31.642 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:55:31.642 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:55:31.642 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:55:31.642 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:55:31.642 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:55:31.643 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:55:31.643 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:55:31.643 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:55:31.643 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:55:31.643 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:55:31.643 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:55:31.643 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:55:31.643 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:55:31.643 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:55:31.643 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:55:31.643 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:55:31.643 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:55:31.643 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:55:31.643 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:55:31.644 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:55:31.644 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:55:31.644 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:55:31.644 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:55:31.644 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:55:31.644 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:55:31.644 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:55:31.644 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:55:31.645 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:55:31.645 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:55:31.645 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:55:31.645 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:55:31.645 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:55:31.645 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:55:31.645 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:55:31.645 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:55:31.645 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:55:31.645 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:55:31.645 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:55:31.647 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:55:31.647 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:55:31.647 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:55:31.647 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:55:31.647 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:55:31.647 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:55:31.647 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:55:31.647 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:55:31.647 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:55:31.647 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:55:31.648 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:55:31.648 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:55:31.648 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:55:31.648 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:55:31.648 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:55:31.648 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:55:31.648 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:55:31.648 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:55:31.649 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:55:31.649 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:55:31.649 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:55:31.649 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:55:31.649 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:55:31.649 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:55:31.649 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:55:31.649 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:55:31.649 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:55:31.649 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:55:31.650 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:55:31.650 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:55:31.650 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:55:31.650 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:55:31.650 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:55:31.650 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:55:31.650 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:55:31.650 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:55:31.650 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:55:31.651 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:55:31.651 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:55:31.651 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:55:31.651 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:55:31.651 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:55:31.651 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:55:31.651 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:55:31.651 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:55:31.651 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:55:31.651 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:55:31.652 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:55:31.652 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:55:31.652 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:55:31.652 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:55:31.652 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:55:31.652 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:55:31.652 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:55:31.652 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:55:31.652 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:55:31.652 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:55:31.652 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:55:31.652 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:55:31.652 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:55:31.652 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:55:31.653 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:55:31.653 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:55:31.653 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:55:31.653 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:55:31.653 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:55:31.653 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:55:31.653 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:55:31.653 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:55:31.653 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:55:31.653 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:55:31.653 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:55:31.653 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:55:31.653 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:55:31.653 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:55:31.654 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:55:31.654 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:55:31.654 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:55:31.654 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:55:31.654 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:55:31.654 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:55:31.654 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:55:31.654 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:55:31.654 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:55:31.654 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:55:31.654 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:55:31.655 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:55:31.655 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:55:31.655 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:55:31.655 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:55:31.655 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:55:31.655 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:55:31.655 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:55:31.655 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:55:31.655 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:55:31.656 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:55:31.656 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:55:31.656 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:55:31.656 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:55:31.656 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:55:31.656 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:55:31.656 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:55:31.656 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:55:31.658 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.615+0000 7fe181526640 10 librados: watch_flush enter 2024-09-16T03:55:31.658 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.615+0000 7fe181526640 10 librados: watch_flush exit 2024-09-16T03:55:31.658 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.615+0000 7fe181526640 20 client.824622.objecter shutdown clearing up homeless session... 2024-09-16T03:55:31.658 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.615+0000 7fe181526640 10 client.824622.objecter successfully canceled tick 2024-09-16T03:55:31.658 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.615+0000 7fe181526640 1 -- 172.21.15.42:0/3819635124 >> v2:172.21.15.154:6812/254845342 conn(0x7fe168037810 msgr2=0x7fe168039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:55:31.658 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.615+0000 7fe181526640 1 --2- 172.21.15.42:0/3819635124 >> v2:172.21.15.154:6812/254845342 conn(0x7fe168037810 0x7fe168039cd0 secure :-1 s=READY pgs=3640 cs=0 l=1 rev1=1 crypto rx=0x7fe16c0ad000 tx=0x7fe16c008040 comp rx=0 tx=0).stop 2024-09-16T03:55:31.658 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.615+0000 7fe181526640 10 monclient: shutdown 2024-09-16T03:55:31.658 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.615+0000 7fe181526640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:55:31.658 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.615+0000 7fe181526640 1 -- 172.21.15.42:0/3819635124 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe17c0a3d90 msgr2=0x7fe17c130aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:55:31.658 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.615+0000 7fe181526640 1 --2- 172.21.15.42:0/3819635124 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe17c0a3d90 0x7fe17c130aa0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fe17000c450 tx=0x7fe17000c910 comp rx=0 tx=0).stop 2024-09-16T03:55:31.659 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.615+0000 7fe17b7fe640 1 -- 172.21.15.42:0/3819635124 reap_dead start 2024-09-16T03:55:31.659 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.615+0000 7fe17b7fe640 1 -- 172.21.15.42:0/3819635124 reap_dead start 2024-09-16T03:55:31.659 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.615+0000 7fe181526640 1 -- 172.21.15.42:0/3819635124 shutdown_connections 2024-09-16T03:55:31.659 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.615+0000 7fe181526640 1 -- 172.21.15.42:0/3819635124 >> 172.21.15.42:0/3819635124 conn(0x7fe17c0a9fb0 msgr2=0x7fe17c0a7a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:55:31.659 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.615+0000 7fe181526640 1 -- 172.21.15.42:0/3819635124 shutdown_connections 2024-09-16T03:55:31.659 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.615+0000 7fe181526640 1 -- 172.21.15.42:0/3819635124 wait complete. 2024-09-16T03:55:31.659 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.615+0000 7fe181526640 1 librados: shutdown 2024-09-16T03:55:31.659 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:55:31.659 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-09-16T03:55:31.660 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-16T03:55:31.660 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-09-16T03:55:31.660 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-16T03:55:31.660 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-16T03:55:31.660 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T03:55:31.685 INFO:tasks.ceph.mon.b:Stopped 2024-09-16T03:55:31.685 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T03:55:31.685 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:55:31.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.803+0000 7fdab2723640 10 monclient: get_monmap_and_config 2024-09-16T03:55:31.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.803+0000 7fdab2723640 10 monclient: build_initial_monmap 2024-09-16T03:55:31.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.807+0000 7fdab2723640 10 monclient: monmap: 2024-09-16T03:55:31.809 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:55:31.810 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:55:31.810 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:55:31.808134+0000 2024-09-16T03:55:31.810 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:55:31.808134+0000 2024-09-16T03:55:31.810 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:55:31.810 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:55:31.810 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:55:31.810 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:55:31.810 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:55:31.810 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:55:31.811 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:55:31.811 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:55:31.813 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.807+0000 7fdab2723640 1 Processor -- start 2024-09-16T03:55:31.813 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.807+0000 7fdab2723640 1 -- start start 2024-09-16T03:55:31.813 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.807+0000 7fdab2723640 10 monclient: init 2024-09-16T03:55:31.813 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.807+0000 7fdab2723640 10 monclient: _reopen_session rank -1 2024-09-16T03:55:31.813 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.807+0000 7fdab2723640 10 monclient: _add_conns ranks=[1,2,3,0,4] 2024-09-16T03:55:31.813 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.807+0000 7fdab2723640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fdaac0a39b0 addr v1:172.21.15.154:6789/0 2024-09-16T03:55:31.813 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.807+0000 7fdab2723640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fdaac0a3d90 0x7fdaac13c380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:55:31.813 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.807+0000 7fdab2723640 10 monclient(hunting): picked mon.noname-b con 0x7fdaac0a3d90 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:55:31.813 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.807+0000 7fdab2723640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fdaac13c8c0 0x7fdaac13ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:55:31.813 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.807+0000 7fdab2723640 10 monclient(hunting): picked mon.noname-a con 0x7fdaac13c8c0 addr v2:172.21.15.42:3300/0 2024-09-16T03:55:31.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.807+0000 7fdab2723640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdaac134290 con 0x7fdaac0a39b0 2024-09-16T03:55:31.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.807+0000 7fdab2723640 10 monclient(hunting): start opening mon connection 2024-09-16T03:55:31.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.807+0000 7fdab2723640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7fdaac09c140 con 0x7fdaac13c8c0 2024-09-16T03:55:31.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.807+0000 7fdab2723640 10 monclient(hunting): start opening mon connection 2024-09-16T03:55:31.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.807+0000 7fdab2723640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fdaac09c420 con 0x7fdaac0a3d90 2024-09-16T03:55:31.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.807+0000 7fdab2723640 10 monclient(hunting): _renew_subs 2024-09-16T03:55:31.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.807+0000 7fdab2723640 10 monclient(hunting): authenticate will time out at 1778.777577s 2024-09-16T03:55:31.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.807+0000 7fdaab7fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fdaac0a3d90 0x7fdaac13c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:55:31.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.807+0000 7fdab0cc0640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fdaac13c8c0 0x7fdaac13ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:55:31.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.807+0000 7fdaab7fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fdaac0a3d90 0x7fdaac13c380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:51804/0 (socket says 172.21.15.42:51804) 2024-09-16T03:55:31.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.807+0000 7fdaab7fe640 1 -- 172.21.15.42:0/694834049 learned_addr learned my addr 172.21.15.42:0/694834049 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:55:31.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.807+0000 7fdaab7fe640 10 monclient(hunting): get_auth_request con 0x7fdaac0a3d90 auth_method 0 2024-09-16T03:55:31.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.807+0000 7fdaab7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:55:31.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.807+0000 7fdaab7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:55:31.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.807+0000 7fdaab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:55:31.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.807+0000 7fdab0cc0640 10 monclient(hunting): get_auth_request con 0x7fdaac13c8c0 auth_method 0 2024-09-16T03:55:31.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.807+0000 7fdab0cc0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:55:31.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.807+0000 7fdab0cc0640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:55:31.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.807+0000 7fdab0cc0640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:55:31.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.807+0000 7fdaabfff640 1 -- 172.21.15.42:0/694834049 >> v1:172.21.15.154:6789/0 conn(0x7fdaac0a39b0 legacy=0x7fdaac133b80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.154:6789/0 2024-09-16T03:55:31.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.807+0000 7fdaab7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:55:31.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.807+0000 7fdaab7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:55:31.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.807+0000 7fdaab7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:55:31.816 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.807+0000 7fdab0cc0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:55:31.816 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.807+0000 7fdab0cc0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:55:31.816 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.807+0000 7fdab0cc0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:55:31.816 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.807+0000 7fdab0cc0640 10 monclient(hunting): handle_auth_done global_id 824625 payload 274 2024-09-16T03:55:31.816 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.807+0000 7fdab0cc0640 1 -- 172.21.15.42:0/694834049 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fdaac0a3d90 msgr2=0x7fdaac13c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:55:31.816 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.807+0000 7fdab0cc0640 1 --2- 172.21.15.42:0/694834049 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fdaac0a3d90 0x7fdaac13c380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:55:31.816 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.807+0000 7fdab0cc0640 1 -- 172.21.15.42:0/694834049 >> v1:172.21.15.154:6789/0 conn(0x7fdaac0a39b0 legacy=0x7fdaac133b80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:55:31.816 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.807+0000 7fdab0cc0640 10 monclient: _finish_hunting 0 2024-09-16T03:55:31.816 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.807+0000 7fdab0cc0640 1 monclient: found mon.noname-a 2024-09-16T03:55:31.816 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.807+0000 7fdab0cc0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:55:31.816 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.807+0000 7fdab0cc0640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:55:31.817 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.807+0000 7fdab0cc0640 1 -- 172.21.15.42:0/694834049 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdaac06dd60 con 0x7fdaac13c8c0 2024-09-16T03:55:31.817 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.807+0000 7fdaab7fe640 1 --2- 172.21.15.42:0/694834049 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fdaac0a3d90 0x7fdaac13c380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:55:31.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.811+0000 7fdab0cc0640 1 --2- 172.21.15.42:0/694834049 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdaac13c8c0 0x7fdaac13ecb0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fdaa4009d90 tx=0x7fdaa401c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e20dcfbe695000ee server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:55:31.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.811+0000 7fdaaaffd640 1 -- 172.21.15.42:0/694834049 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaa40ac070 con 0x7fdaac13c8c0 2024-09-16T03:55:31.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.811+0000 7fdaaaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:55:31.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.811+0000 7fdaaaffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:55:31.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.811+0000 7fdaaaffd640 10 monclient: dump: 2024-09-16T03:55:31.823 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:55:31.823 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:55:31.823 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:31.823 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:31.823 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:55:31.823 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:55:31.823 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:55:31.823 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:55:31.823 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:55:31.823 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:55:31.823 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.811+0000 7fdaaaffd640 10 monclient: _finish_auth 0 2024-09-16T03:55:31.823 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.811+0000 7fdaaaffd640 10 monclient: _check_auth_tickets 2024-09-16T03:55:31.823 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.811+0000 7fdaaaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:55:31.824 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.811+0000 7fdaaaffd640 1 -- 172.21.15.42:0/694834049 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdaa40a6680 con 0x7fdaac13c8c0 2024-09-16T03:55:31.824 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.811+0000 7fdab2723640 5 monclient: authenticate success, global_id 824625 2024-09-16T03:55:31.824 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.811+0000 7fdab2723640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:55:31.824 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.811+0000 7fdaaaffd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:55:31.824 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.811+0000 7fdaaaffd640 1 -- 172.21.15.42:0/694834049 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaa40a6a50 con 0x7fdaac13c8c0 2024-09-16T03:55:31.824 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.811+0000 7fdaaaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:55:31.824 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.811+0000 7fdaaaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:55:31.824 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.811+0000 7fdaaaffd640 10 monclient: dump: 2024-09-16T03:55:31.824 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:55:31.824 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:55:31.824 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:31.824 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:31.825 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:55:31.825 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:55:31.825 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:55:31.825 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:55:31.825 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:55:31.825 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:55:31.825 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.811+0000 7fdab2723640 10 monclient: get_monmap_and_config success 2024-09-16T03:55:31.825 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.811+0000 7fdab2723640 4 set_mon_vals no callback set 2024-09-16T03:55:31.825 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.811+0000 7fdab2723640 10 monclient: shutdown 2024-09-16T03:55:31.825 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.811+0000 7fdab2723640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:55:31.825 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.811+0000 7fdab2723640 1 -- 172.21.15.42:0/694834049 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdaac13c8c0 msgr2=0x7fdaac13ecb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:55:31.825 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.811+0000 7fdab2723640 1 --2- 172.21.15.42:0/694834049 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdaac13c8c0 0x7fdaac13ecb0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fdaa4009d90 tx=0x7fdaa401c7f0 comp rx=0 tx=0).stop 2024-09-16T03:55:31.825 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.811+0000 7fdab2723640 1 -- 172.21.15.42:0/694834049 shutdown_connections 2024-09-16T03:55:31.825 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.811+0000 7fdab2723640 1 --2- 172.21.15.42:0/694834049 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdaac13c8c0 0x7fdaac13ecb0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:55:31.826 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.811+0000 7fdab2723640 1 --2- 172.21.15.42:0/694834049 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fdaac0a3d90 0x7fdaac13c380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:55:31.826 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.811+0000 7fdab2723640 1 -- 172.21.15.42:0/694834049 >> 172.21.15.42:0/694834049 conn(0x7fdaac0a9fb0 msgr2=0x7fdaac0aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:55:31.826 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.811+0000 7fdab2723640 1 -- 172.21.15.42:0/694834049 shutdown_connections 2024-09-16T03:55:31.826 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.811+0000 7fdab2723640 1 -- 172.21.15.42:0/694834049 wait complete. 2024-09-16T03:55:31.826 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.811+0000 7fdab2723640 10 monclient: build_initial_monmap 2024-09-16T03:55:31.826 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.811+0000 7fdab2723640 10 monclient: monmap: 2024-09-16T03:55:31.826 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:55:31.826 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:55:31.826 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:55:31.826 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:55:31.826 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:55:31.826 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:55:31.826 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:55:31.827 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:55:31.827 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:55:31.827 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:55:31.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.811+0000 7fdab2723640 1 librados: starting msgr at 2024-09-16T03:55:31.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.811+0000 7fdab2723640 1 librados: starting objecter 2024-09-16T03:55:31.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.811+0000 7fdab2723640 1 Processor -- start 2024-09-16T03:55:31.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.811+0000 7fdab2723640 1 -- start start 2024-09-16T03:55:31.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.811+0000 7fdab2723640 1 librados: setting wanted keys 2024-09-16T03:55:31.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.811+0000 7fdab2723640 1 librados: calling monclient init 2024-09-16T03:55:31.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.811+0000 7fdab2723640 10 monclient: init 2024-09-16T03:55:31.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.811+0000 7fdab2723640 10 monclient: _reopen_session rank -1 2024-09-16T03:55:31.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.811+0000 7fdab2723640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T03:55:31.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.811+0000 7fdab2723640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdaac0a39b0 0x7fdaac0a24f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:55:31.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.811+0000 7fdab2723640 10 monclient(hunting): picked mon.noname-a con 0x7fdaac0a39b0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:55:31.828 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.811+0000 7fdab2723640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdaac0a3d90 0x7fdaac13a900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:55:31.828 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.811+0000 7fdab2723640 10 monclient(hunting): picked mon.noname-b con 0x7fdaac0a3d90 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:55:31.828 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.811+0000 7fdab2723640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fdaac13c8c0 0x7fdaac13ae40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:55:31.828 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.811+0000 7fdab2723640 10 monclient(hunting): picked mon.noname-c con 0x7fdaac13c8c0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:55:31.828 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.811+0000 7fdab2723640 10 monclient(hunting): start opening mon connection 2024-09-16T03:55:31.828 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.811+0000 7fdab2723640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fdaac141da0 con 0x7fdaac0a39b0 2024-09-16T03:55:31.828 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.811+0000 7fdab2723640 10 monclient(hunting): start opening mon connection 2024-09-16T03:55:31.828 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.811+0000 7fdab2723640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fdaac141c20 con 0x7fdaac0a3d90 2024-09-16T03:55:31.828 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.811+0000 7fdab2723640 10 monclient(hunting): start opening mon connection 2024-09-16T03:55:31.828 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.811+0000 7fdab2723640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fdaac141f20 con 0x7fdaac13c8c0 2024-09-16T03:55:31.828 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.811+0000 7fdab2723640 10 monclient(hunting): _renew_subs 2024-09-16T03:55:31.828 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.811+0000 7fdab2723640 10 monclient(hunting): authenticate will time out at 1778.783780s 2024-09-16T03:55:31.828 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.811+0000 7fdaabfff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdaac0a39b0 0x7fdaac0a24f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:55:31.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.811+0000 7fdab0cc0640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fdaac13c8c0 0x7fdaac13ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:55:31.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.811+0000 7fdaabfff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdaac0a39b0 0x7fdaac0a24f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:34880/0 (socket says 172.21.15.42:34880) 2024-09-16T03:55:31.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.811+0000 7fdaab7fe640 1 -- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdaac0a3d90 msgr2=0x7fdaac13a900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T03:55:31.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.811+0000 7fdaabfff640 1 -- 172.21.15.42:0/616114133 learned_addr learned my addr 172.21.15.42:0/616114133 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:55:31.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.811+0000 7fdaabfff640 10 monclient(hunting): get_auth_request con 0x7fdaac0a39b0 auth_method 0 2024-09-16T03:55:31.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.811+0000 7fdaabfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:55:31.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.811+0000 7fdaabfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:55:31.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.811+0000 7fdaabfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:55:31.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.811+0000 7fdab0cc0640 10 monclient(hunting): get_auth_request con 0x7fdaac13c8c0 auth_method 0 2024-09-16T03:55:31.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.811+0000 7fdaab7fe640 1 --2- 172.21.15.42:0/616114133 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdaac0a3d90 0x7fdaac13a900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:55:31.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.811+0000 7fdab0cc0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:55:31.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.811+0000 7fdab0cc0640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:55:31.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.811+0000 7fdab0cc0640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:55:31.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.815+0000 7fdaabfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:55:31.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.815+0000 7fdaabfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:55:31.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.815+0000 7fdaabfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:55:31.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.815+0000 7fdab0cc0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:55:31.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.815+0000 7fdab0cc0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:55:31.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.815+0000 7fdab0cc0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:55:31.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.815+0000 7fdaabfff640 10 monclient(hunting): handle_auth_done global_id 824628 payload 824 2024-09-16T03:55:31.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.815+0000 7fdaabfff640 1 -- 172.21.15.42:0/616114133 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fdaac13c8c0 msgr2=0x7fdaac13ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:55:31.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.815+0000 7fdaabfff640 1 --2- 172.21.15.42:0/616114133 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fdaac13c8c0 0x7fdaac13ae40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:55:31.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.815+0000 7fdaabfff640 1 -- 172.21.15.42:0/616114133 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdaac0a3d90 msgr2=0x7fdaac13a900 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:55:31.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.815+0000 7fdaabfff640 1 --2- 172.21.15.42:0/616114133 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdaac0a3d90 0x7fdaac13a900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:55:31.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.815+0000 7fdaabfff640 10 monclient: _finish_hunting 0 2024-09-16T03:55:31.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.815+0000 7fdaabfff640 1 monclient: found mon.noname-a 2024-09-16T03:55:31.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.815+0000 7fdaabfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:55:31.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.815+0000 7fdaabfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:55:31.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.815+0000 7fdaabfff640 1 -- 172.21.15.42:0/616114133 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdaac09ea90 con 0x7fdaac0a39b0 2024-09-16T03:55:31.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.815+0000 7fdaabfff640 1 --2- 172.21.15.42:0/616114133 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdaac0a39b0 0x7fdaac0a24f0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fda9c00b4e0 tx=0x7fda9c00b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:55:31.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.815+0000 7fdaa8ff9640 10 client.?.objecter ms_handle_connect 0x7fdaac0a39b0 2024-09-16T03:55:31.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.815+0000 7fdaa8ff9640 10 client.?.objecter resend_mon_ops 2024-09-16T03:55:31.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.815+0000 7fdaa8ff9640 1 -- 172.21.15.42:0/616114133 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda9c003d80 con 0x7fdaac0a39b0 2024-09-16T03:55:31.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.815+0000 7fdaa8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:55:31.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.815+0000 7fdaa8ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:55:31.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.815+0000 7fdaa8ff9640 10 monclient: dump: 2024-09-16T03:55:31.831 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:55:31.831 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:55:31.831 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:31.832 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:31.832 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:55:31.832 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:55:31.832 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:55:31.832 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:55:31.832 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:55:31.832 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:55:31.832 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.815+0000 7fdaa8ff9640 10 monclient: _finish_auth 0 2024-09-16T03:55:31.832 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.815+0000 7fdaa8ff9640 10 monclient: _check_auth_tickets 2024-09-16T03:55:31.832 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.815+0000 7fdaa8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:55:31.832 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.815+0000 7fdaa8ff9640 1 -- 172.21.15.42:0/616114133 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda9c015070 con 0x7fdaac0a39b0 2024-09-16T03:55:31.832 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.815+0000 7fdab2723640 5 monclient: authenticate success, global_id 824628 2024-09-16T03:55:31.832 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.815+0000 7fdab2723640 10 monclient: _renew_subs 2024-09-16T03:55:31.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.815+0000 7fdab2723640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:55:31.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.815+0000 7fdab2723640 1 -- 172.21.15.42:0/616114133 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdaac09ed70 con 0x7fdaac0a39b0 2024-09-16T03:55:31.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.815+0000 7fdaa8ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T03:55:31.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.815+0000 7fdaa8ff9640 1 -- 172.21.15.42:0/616114133 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda9c0052b0 con 0x7fdaac0a39b0 2024-09-16T03:55:31.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.815+0000 7fdaa8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:55:31.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.815+0000 7fdaa8ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:55:31.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.815+0000 7fdaa8ff9640 10 monclient: dump: 2024-09-16T03:55:31.833 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:55:31.833 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:55:31.833 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:31.833 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:31.833 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:55:31.833 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:55:31.833 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:55:31.834 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:55:31.834 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:55:31.834 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:55:31.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.815+0000 7fdab2723640 10 client.824628.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:55:31.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.815+0000 7fdab2723640 10 monclient: _renew_subs 2024-09-16T03:55:31.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.815+0000 7fdaaa7fc640 4 set_mon_vals no callback set 2024-09-16T03:55:31.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.815+0000 7fdab2723640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:55:31.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.815+0000 7fdab2723640 1 -- 172.21.15.42:0/616114133 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdaac13c370 con 0x7fdaac0a39b0 2024-09-16T03:55:31.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.815+0000 7fdab2723640 1 librados: init done 2024-09-16T03:55:31.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.815+0000 7fdaa8ff9640 1 -- 172.21.15.42:0/616114133 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fda9c005450 con 0x7fdaac0a39b0 2024-09-16T03:55:31.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.815+0000 7fdab2723640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:55:31.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.815+0000 7fdab2723640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:55:31.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.815+0000 7fdab2723640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:55:31.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.815+0000 7fdab2723640 1 -- 172.21.15.42:0/616114133 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdaac133b80 con 0x7fdaac0a39b0 2024-09-16T03:55:31.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.815+0000 7fdaa8ff9640 1 --2- 172.21.15.42:0/616114133 >> v2:172.21.15.154:6812/254845342 conn(0x7fda980377c0 0x7fda98039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:55:31.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.815+0000 7fdaa8ff9640 1 -- 172.21.15.42:0/616114133 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(418..418 src has 330..418) ==== 6096+0+0 (secure 0 0 0) 0x7fda9c048ea0 con 0x7fdaac0a39b0 2024-09-16T03:55:31.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.815+0000 7fdaa8ff9640 10 client.824628.objecter ms_dispatch 0x7fdaac000ba0 osd_map(418..418 src has 330..418) 2024-09-16T03:55:31.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.815+0000 7fdaa8ff9640 3 client.824628.objecter handle_osd_map got epochs [418,418] > 0 2024-09-16T03:55:31.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.815+0000 7fdaa8ff9640 3 client.824628.objecter handle_osd_map decoding full epoch 418 2024-09-16T03:55:31.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.815+0000 7fdaa8ff9640 20 client.824628.objecter dump_active .. 0 homeless 2024-09-16T03:55:31.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.815+0000 7fdaab7fe640 1 --2- 172.21.15.42:0/616114133 >> v2:172.21.15.154:6812/254845342 conn(0x7fda980377c0 0x7fda98039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:55:31.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.815+0000 7fdaab7fe640 10 monclient: get_auth_request con 0x7fda980377c0 auth_method 0 2024-09-16T03:55:31.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.815+0000 7fdaab7fe640 1 --2- 172.21.15.42:0/616114133 >> v2:172.21.15.154:6812/254845342 conn(0x7fda980377c0 0x7fda98039c80 secure :-1 s=READY pgs=3641 cs=0 l=1 rev1=1 crypto rx=0x7fdaac13c120 tx=0x7fdaa0009240 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:55:31.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.815+0000 7fdaa8ff9640 10 client.824628.objecter ms_handle_connect 0x7fda980377c0 2024-09-16T03:55:31.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.819+0000 7fdaa8ff9640 1 -- 172.21.15.42:0/616114133 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7fda9c0104e0 con 0x7fdaac0a39b0 2024-09-16T03:55:31.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.819+0000 7fdaa8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:55:31.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:31.819+0000 7fdaa8ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:55:32.048 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:32.043+0000 7fdab2723640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:55:32.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:32.043+0000 7fdab2723640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:55:32.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:32.043+0000 7fdab2723640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:55:32.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:32.043+0000 7fdab2723640 1 -- 172.21.15.42:0/616114133 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fdaac132fc0 con 0x7fdaac0a39b0 2024-09-16T03:55:32.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:32.047+0000 7fdaa8ff9640 1 -- 172.21.15.42:0/616114133 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7fda9c0473e0 con 0x7fdaac0a39b0 2024-09-16T03:55:32.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:32.047+0000 7fdaa8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:55:32.050 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:32.047+0000 7fdaa8ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:55:32.050 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":116,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":4,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:55:32.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:32.047+0000 7fdab2723640 10 librados: watch_flush enter 2024-09-16T03:55:32.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:32.047+0000 7fdab2723640 10 librados: watch_flush exit 2024-09-16T03:55:32.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:32.047+0000 7fdab2723640 20 client.824628.objecter shutdown clearing up homeless session... 2024-09-16T03:55:32.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:32.047+0000 7fdab2723640 10 client.824628.objecter successfully canceled tick 2024-09-16T03:55:32.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:32.047+0000 7fdab2723640 1 -- 172.21.15.42:0/616114133 >> v2:172.21.15.154:6812/254845342 conn(0x7fda980377c0 msgr2=0x7fda98039c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:55:32.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:32.047+0000 7fdab2723640 1 --2- 172.21.15.42:0/616114133 >> v2:172.21.15.154:6812/254845342 conn(0x7fda980377c0 0x7fda98039c80 secure :-1 s=READY pgs=3641 cs=0 l=1 rev1=1 crypto rx=0x7fdaac13c120 tx=0x7fdaa0009240 comp rx=0 tx=0).stop 2024-09-16T03:55:32.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:32.047+0000 7fdab2723640 10 monclient: shutdown 2024-09-16T03:55:32.053 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:32.047+0000 7fdab2723640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:55:32.053 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:32.047+0000 7fdab2723640 1 -- 172.21.15.42:0/616114133 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdaac0a39b0 msgr2=0x7fdaac0a24f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:55:32.053 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:32.047+0000 7fdab2723640 1 --2- 172.21.15.42:0/616114133 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdaac0a39b0 0x7fdaac0a24f0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fda9c00b4e0 tx=0x7fda9c00b9a0 comp rx=0 tx=0).stop 2024-09-16T03:55:32.053 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:32.047+0000 7fdab2723640 1 -- 172.21.15.42:0/616114133 shutdown_connections 2024-09-16T03:55:32.053 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:32.047+0000 7fdab2723640 1 --2- 172.21.15.42:0/616114133 >> v2:172.21.15.154:6812/254845342 conn(0x7fda980377c0 0x7fda98039c80 unknown :-1 s=CLOSED pgs=3641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:55:32.053 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:32.047+0000 7fdab2723640 1 --2- 172.21.15.42:0/616114133 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fdaac13c8c0 0x7fdaac13ae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:55:32.053 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:32.047+0000 7fdab2723640 1 --2- 172.21.15.42:0/616114133 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdaac0a3d90 0x7fdaac13a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:55:32.053 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:32.047+0000 7fdab2723640 1 --2- 172.21.15.42:0/616114133 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdaac0a39b0 0x7fdaac0a24f0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:55:32.053 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:32.047+0000 7fdab2723640 1 -- 172.21.15.42:0/616114133 >> 172.21.15.42:0/616114133 conn(0x7fdaac0a9fb0 msgr2=0x7fdaac0a8e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:55:32.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:32.047+0000 7fdab2723640 1 -- 172.21.15.42:0/616114133 shutdown_connections 2024-09-16T03:55:32.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:32.047+0000 7fdab2723640 1 -- 172.21.15.42:0/616114133 wait complete. 2024-09-16T03:55:32.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:32.047+0000 7fdab2723640 1 librados: shutdown 2024-09-16T03:55:35.068 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:55:35.199 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.195+0000 7f0c0fcea640 10 monclient: get_monmap_and_config 2024-09-16T03:55:35.199 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.195+0000 7f0c0fcea640 10 monclient: build_initial_monmap 2024-09-16T03:55:35.199 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.195+0000 7f0c0fcea640 10 monclient: monmap: 2024-09-16T03:55:35.199 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:55:35.200 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:55:35.200 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:55:35.197906+0000 2024-09-16T03:55:35.200 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:55:35.197906+0000 2024-09-16T03:55:35.200 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:55:35.200 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:55:35.201 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:55:35.201 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:55:35.201 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:55:35.201 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:55:35.201 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:55:35.201 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:55:35.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.195+0000 7f0c0fcea640 1 Processor -- start 2024-09-16T03:55:35.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.195+0000 7f0c0fcea640 1 -- start start 2024-09-16T03:55:35.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.195+0000 7f0c0fcea640 10 monclient: init 2024-09-16T03:55:35.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.195+0000 7f0c0fcea640 10 monclient: _reopen_session rank -1 2024-09-16T03:55:35.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.195+0000 7f0c0fcea640 10 monclient: _add_conns ranks=[2,1,3,0,4] 2024-09-16T03:55:35.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.195+0000 7f0c0fcea640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0c08130b20 0x7f0c08130f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:55:35.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.195+0000 7f0c0fcea640 10 monclient(hunting): picked mon.noname-b con 0x7f0c08130b20 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:55:35.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.195+0000 7f0c0fcea640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0c080a2250 addr v1:172.21.15.154:6789/0 2024-09-16T03:55:35.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.195+0000 7f0c0fcea640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f0c08073420 0x7f0c0813f240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:55:35.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.195+0000 7f0c0fcea640 10 monclient(hunting): picked mon.noname-a con 0x7f0c08073420 addr v2:172.21.15.42:3300/0 2024-09-16T03:55:35.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.195+0000 7f0c0fcea640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c080a1040 con 0x7f0c080a2250 2024-09-16T03:55:35.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.195+0000 7f0c0fcea640 10 monclient(hunting): start opening mon connection 2024-09-16T03:55:35.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.195+0000 7f0c0fcea640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f0c080a12b0 con 0x7f0c08073420 2024-09-16T03:55:35.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.195+0000 7f0c0fcea640 10 monclient(hunting): start opening mon connection 2024-09-16T03:55:35.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.195+0000 7f0c0fcea640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f0c080a1590 con 0x7f0c08130b20 2024-09-16T03:55:35.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.195+0000 7f0c0fcea640 10 monclient(hunting): _renew_subs 2024-09-16T03:55:35.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.195+0000 7f0c0fcea640 10 monclient(hunting): authenticate will time out at 1782.167327s 2024-09-16T03:55:35.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.195+0000 7f0c0d285640 1 -- >> v1:172.21.15.154:6789/0 conn(0x7f0c080a2250 legacy=0x7f0c08072d10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.154:6789/0 2024-09-16T03:55:35.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.195+0000 7f0c0da86640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0c08130b20 0x7f0c08130f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:55:35.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.195+0000 7f0c0e287640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f0c08073420 0x7f0c0813f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:55:35.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.195+0000 7f0c0e287640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f0c08073420 0x7f0c0813f240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:51946/0 (socket says 172.21.15.42:51946) 2024-09-16T03:55:35.214 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.195+0000 7f0c0da86640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0c08130b20 0x7f0c08130f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:59072/0 (socket says 172.21.15.42:59072) 2024-09-16T03:55:35.214 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.195+0000 7f0c0e287640 1 -- 172.21.15.42:0/3444890039 learned_addr learned my addr 172.21.15.42:0/3444890039 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:55:35.214 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.195+0000 7f0c0e287640 10 monclient(hunting): get_auth_request con 0x7f0c08073420 auth_method 0 2024-09-16T03:55:35.214 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.195+0000 7f0c0e287640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:55:35.214 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.195+0000 7f0c0e287640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:55:35.214 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.195+0000 7f0c0e287640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:55:35.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.195+0000 7f0c0da86640 10 monclient(hunting): get_auth_request con 0x7f0c08130b20 auth_method 0 2024-09-16T03:55:35.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.195+0000 7f0c0da86640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:55:35.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.195+0000 7f0c0da86640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:55:35.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.195+0000 7f0c0da86640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:55:35.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.195+0000 7f0c0da86640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:55:35.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.195+0000 7f0c0da86640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:55:35.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.195+0000 7f0c0da86640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:55:35.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.195+0000 7f0c0e287640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:55:35.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.195+0000 7f0c0e287640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:55:35.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.195+0000 7f0c0e287640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:55:35.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.195+0000 7f0c0e287640 10 monclient(hunting): handle_auth_done global_id 824631 payload 274 2024-09-16T03:55:35.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.195+0000 7f0c0e287640 1 -- 172.21.15.42:0/3444890039 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0c08130b20 msgr2=0x7f0c08130f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:55:35.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.195+0000 7f0c0e287640 1 --2- 172.21.15.42:0/3444890039 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0c08130b20 0x7f0c08130f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:55:35.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.195+0000 7f0c0e287640 1 -- 172.21.15.42:0/3444890039 >> v1:172.21.15.154:6789/0 conn(0x7f0c080a2250 legacy=0x7f0c08072d10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:55:35.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.195+0000 7f0c0e287640 10 monclient: _finish_hunting 0 2024-09-16T03:55:35.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.195+0000 7f0c0e287640 1 monclient: found mon.noname-a 2024-09-16T03:55:35.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.195+0000 7f0c0e287640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:55:35.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.195+0000 7f0c0e287640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:55:35.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.195+0000 7f0c0e287640 1 -- 172.21.15.42:0/3444890039 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c080a1870 con 0x7f0c08073420 2024-09-16T03:55:35.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.195+0000 7f0c0da86640 1 --2- 172.21.15.42:0/3444890039 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0c08130b20 0x7f0c08130f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:55:35.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.199+0000 7f0c0e287640 1 --2- 172.21.15.42:0/3444890039 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0c08073420 0x7f0c0813f240 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f0c00009e50 tx=0x7f0c0001c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4e8be36649fa603 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:55:35.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.199+0000 7f0c0ca84640 1 -- 172.21.15.42:0/3444890039 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c000a5750 con 0x7f0c08073420 2024-09-16T03:55:35.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.199+0000 7f0c0ca84640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:55:35.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.199+0000 7f0c0ca84640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:55:35.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.199+0000 7f0c0ca84640 10 monclient: dump: 2024-09-16T03:55:35.219 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:55:35.219 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:55:35.219 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:35.219 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:35.219 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:55:35.219 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:55:35.220 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:55:35.220 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:55:35.220 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:55:35.220 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:55:35.220 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.199+0000 7f0c0ca84640 10 monclient: _finish_auth 0 2024-09-16T03:55:35.220 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.199+0000 7f0c0ca84640 10 monclient: _check_auth_tickets 2024-09-16T03:55:35.221 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.199+0000 7f0c0ca84640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:55:35.221 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.199+0000 7f0c0ca84640 1 -- 172.21.15.42:0/3444890039 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c000a58f0 con 0x7f0c08073420 2024-09-16T03:55:35.221 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.199+0000 7f0c0fcea640 5 monclient: authenticate success, global_id 824631 2024-09-16T03:55:35.221 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.199+0000 7f0c0fcea640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:55:35.221 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.199+0000 7f0c0ca84640 10 monclient: handle_config config(0 keys) 2024-09-16T03:55:35.221 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.199+0000 7f0c0fcea640 10 monclient: get_monmap_and_config success 2024-09-16T03:55:35.221 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.199+0000 7f0c0fcea640 4 set_mon_vals no callback set 2024-09-16T03:55:35.221 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.199+0000 7f0c0ca84640 1 -- 172.21.15.42:0/3444890039 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c000a5cc0 con 0x7f0c08073420 2024-09-16T03:55:35.221 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.199+0000 7f0c0ca84640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:55:35.221 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.199+0000 7f0c0ca84640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:55:35.221 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.199+0000 7f0c0ca84640 10 monclient: dump: 2024-09-16T03:55:35.221 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:55:35.221 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:55:35.222 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:35.222 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:35.222 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:55:35.222 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:55:35.222 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:55:35.222 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:55:35.222 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:55:35.222 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:55:35.222 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.199+0000 7f0c0fcea640 10 monclient: shutdown 2024-09-16T03:55:35.222 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.199+0000 7f0c0fcea640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:55:35.223 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.199+0000 7f0c0fcea640 1 -- 172.21.15.42:0/3444890039 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0c08073420 msgr2=0x7f0c0813f240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:55:35.223 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.199+0000 7f0c0fcea640 1 --2- 172.21.15.42:0/3444890039 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0c08073420 0x7f0c0813f240 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f0c00009e50 tx=0x7f0c0001c8b0 comp rx=0 tx=0).stop 2024-09-16T03:55:35.223 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.199+0000 7f0c0fcea640 1 -- 172.21.15.42:0/3444890039 shutdown_connections 2024-09-16T03:55:35.223 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.199+0000 7f0c0fcea640 1 --2- 172.21.15.42:0/3444890039 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0c08073420 0x7f0c0813f240 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:55:35.223 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.199+0000 7f0c0fcea640 1 --2- 172.21.15.42:0/3444890039 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0c08130b20 0x7f0c08130f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:55:35.223 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.199+0000 7f0c0fcea640 1 -- 172.21.15.42:0/3444890039 >> 172.21.15.42:0/3444890039 conn(0x7f0c080a6210 msgr2=0x7f0c080a6620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:55:35.223 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.199+0000 7f0c0fcea640 1 -- 172.21.15.42:0/3444890039 shutdown_connections 2024-09-16T03:55:35.223 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.199+0000 7f0c0fcea640 1 -- 172.21.15.42:0/3444890039 wait complete. 2024-09-16T03:55:35.223 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.199+0000 7f0c0fcea640 10 monclient: build_initial_monmap 2024-09-16T03:55:35.223 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.199+0000 7f0c0fcea640 10 monclient: monmap: 2024-09-16T03:55:35.224 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:55:35.224 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:55:35.224 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:55:35.224 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:55:35.224 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:55:35.224 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:55:35.224 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:55:35.224 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:55:35.224 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:55:35.224 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:55:35.225 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.199+0000 7f0c0fcea640 1 librados: starting msgr at 2024-09-16T03:55:35.225 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.199+0000 7f0c0fcea640 1 librados: starting objecter 2024-09-16T03:55:35.225 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.203+0000 7f0c0fcea640 1 Processor -- start 2024-09-16T03:55:35.225 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.203+0000 7f0c0fcea640 1 -- start start 2024-09-16T03:55:35.225 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.203+0000 7f0c0fcea640 1 librados: setting wanted keys 2024-09-16T03:55:35.225 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.203+0000 7f0c0fcea640 1 librados: calling monclient init 2024-09-16T03:55:35.225 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.203+0000 7f0c0fcea640 10 monclient: init 2024-09-16T03:55:35.225 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.203+0000 7f0c0fcea640 10 monclient: _reopen_session rank -1 2024-09-16T03:55:35.225 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.203+0000 7f0c0fcea640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:55:35.225 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.203+0000 7f0c0fcea640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0c08073420 0x7f0c081cf0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:55:35.225 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.203+0000 7f0c0fcea640 10 monclient(hunting): picked mon.noname-a con 0x7f0c08073420 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:55:35.225 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.203+0000 7f0c0fcea640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0c080a2250 0x7f0c0813e2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:55:35.225 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.203+0000 7f0c0fcea640 10 monclient(hunting): picked mon.noname-c con 0x7f0c080a2250 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:55:35.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.203+0000 7f0c0fcea640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0c08130b20 0x7f0c0813e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:55:35.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.203+0000 7f0c0fcea640 10 monclient(hunting): picked mon.noname-b con 0x7f0c08130b20 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:55:35.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.203+0000 7f0c0fcea640 10 monclient(hunting): start opening mon connection 2024-09-16T03:55:35.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.203+0000 7f0c0fcea640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f0c081315c0 con 0x7f0c08073420 2024-09-16T03:55:35.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.203+0000 7f0c0fcea640 10 monclient(hunting): start opening mon connection 2024-09-16T03:55:35.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.203+0000 7f0c0fcea640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f0c08131440 con 0x7f0c08130b20 2024-09-16T03:55:35.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.203+0000 7f0c0fcea640 10 monclient(hunting): start opening mon connection 2024-09-16T03:55:35.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.203+0000 7f0c0fcea640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f0c08141fa0 con 0x7f0c080a2250 2024-09-16T03:55:35.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.203+0000 7f0c0fcea640 10 monclient(hunting): _renew_subs 2024-09-16T03:55:35.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.203+0000 7f0c0fcea640 10 monclient(hunting): authenticate will time out at 1782.172621s 2024-09-16T03:55:35.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.203+0000 7f0c0da86640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0c08073420 0x7f0c081cf0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:55:35.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.203+0000 7f0c0d285640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0c080a2250 0x7f0c0813e2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:55:35.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.203+0000 7f0c0da86640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0c08073420 0x7f0c081cf0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:51950/0 (socket says 172.21.15.42:51950) 2024-09-16T03:55:35.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.203+0000 7f0c0da86640 1 -- 172.21.15.42:0/3292945069 learned_addr learned my addr 172.21.15.42:0/3292945069 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:55:35.227 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.203+0000 7f0c0da86640 10 monclient(hunting): get_auth_request con 0x7f0c08073420 auth_method 0 2024-09-16T03:55:35.227 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.203+0000 7f0c0da86640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:55:35.227 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.203+0000 7f0c0da86640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:55:35.227 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.203+0000 7f0c0da86640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:55:35.227 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.203+0000 7f0c0d285640 10 monclient(hunting): get_auth_request con 0x7f0c080a2250 auth_method 0 2024-09-16T03:55:35.227 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.203+0000 7f0c0d285640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:55:35.227 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.203+0000 7f0c0d285640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:55:35.227 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.203+0000 7f0c0d285640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:55:35.227 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.203+0000 7f0c0e287640 1 -- 172.21.15.42:0/3292945069 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0c08130b20 msgr2=0x7f0c0813e7f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T03:55:35.227 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.203+0000 7f0c0e287640 1 --2- 172.21.15.42:0/3292945069 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0c08130b20 0x7f0c0813e7f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:55:35.227 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.203+0000 7f0c0da86640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:55:35.227 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.203+0000 7f0c0da86640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:55:35.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.203+0000 7f0c0da86640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:55:35.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.203+0000 7f0c0d285640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:55:35.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.203+0000 7f0c0d285640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:55:35.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.203+0000 7f0c0d285640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:55:35.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.203+0000 7f0c0d285640 10 monclient(hunting): handle_auth_done global_id 854246 payload 824 2024-09-16T03:55:35.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.203+0000 7f0c0d285640 1 -- 172.21.15.42:0/3292945069 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0c08130b20 msgr2=0x7f0c0813e7f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:55:35.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.203+0000 7f0c0d285640 1 --2- 172.21.15.42:0/3292945069 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0c08130b20 0x7f0c0813e7f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:55:35.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.203+0000 7f0c0d285640 1 -- 172.21.15.42:0/3292945069 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0c08073420 msgr2=0x7f0c081cf0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:55:35.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.203+0000 7f0c0d285640 1 --2- 172.21.15.42:0/3292945069 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0c08073420 0x7f0c081cf0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:55:35.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.203+0000 7f0c0d285640 10 monclient: _finish_hunting 0 2024-09-16T03:55:35.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.203+0000 7f0c0d285640 1 monclient: found mon.noname-c 2024-09-16T03:55:35.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.203+0000 7f0c0d285640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:55:35.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.203+0000 7f0c0d285640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:55:35.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.203+0000 7f0c0d285640 1 -- 172.21.15.42:0/3292945069 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c080a42a0 con 0x7f0c080a2250 2024-09-16T03:55:35.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.203+0000 7f0c0da86640 1 --2- 172.21.15.42:0/3292945069 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0c08073420 0x7f0c081cf0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:55:35.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.203+0000 7f0c0d285640 1 --2- 172.21.15.42:0/3292945069 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0c080a2250 0x7f0c0813e2b0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f0bf800b480 tx=0x7f0bf800b940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:55:35.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.203+0000 7f0bfe7fc640 10 client.?.objecter ms_handle_connect 0x7f0c080a2250 2024-09-16T03:55:35.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.203+0000 7f0bfe7fc640 10 client.?.objecter resend_mon_ops 2024-09-16T03:55:35.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.203+0000 7f0bfe7fc640 1 -- 172.21.15.42:0/3292945069 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bf8003d80 con 0x7f0c080a2250 2024-09-16T03:55:35.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.203+0000 7f0bfe7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:55:35.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.203+0000 7f0bfe7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:55:35.230 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.203+0000 7f0bfe7fc640 10 monclient: dump: 2024-09-16T03:55:35.230 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:55:35.230 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:55:35.230 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:35.230 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:35.230 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:55:35.230 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:55:35.230 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:55:35.230 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:55:35.230 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:55:35.230 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:55:35.230 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.203+0000 7f0bfe7fc640 10 monclient: _finish_auth 0 2024-09-16T03:55:35.230 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.203+0000 7f0bfe7fc640 10 monclient: _check_auth_tickets 2024-09-16T03:55:35.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.203+0000 7f0bfe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:55:35.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.203+0000 7f0bfe7fc640 1 -- 172.21.15.42:0/3292945069 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0bf8015070 con 0x7f0c080a2250 2024-09-16T03:55:35.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.203+0000 7f0c0fcea640 5 monclient: authenticate success, global_id 854246 2024-09-16T03:55:35.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.203+0000 7f0c0fcea640 10 monclient: _renew_subs 2024-09-16T03:55:35.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.203+0000 7f0c0fcea640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:55:35.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.203+0000 7f0c0fcea640 1 -- 172.21.15.42:0/3292945069 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c0813caa0 con 0x7f0c080a2250 2024-09-16T03:55:35.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.203+0000 7f0bfe7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:55:35.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.203+0000 7f0bfe7fc640 1 -- 172.21.15.42:0/3292945069 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bf80052b0 con 0x7f0c080a2250 2024-09-16T03:55:35.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.203+0000 7f0bfe7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:55:35.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.203+0000 7f0bfe7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:55:35.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.203+0000 7f0bfe7fc640 10 monclient: dump: 2024-09-16T03:55:35.231 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:55:35.231 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:55:35.232 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:35.232 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:35.232 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:55:35.232 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:55:35.232 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:55:35.232 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:55:35.232 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:55:35.232 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:55:35.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.203+0000 7f0bfffff640 4 set_mon_vals no callback set 2024-09-16T03:55:35.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.203+0000 7f0c0fcea640 10 client.854246.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:55:35.233 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.203+0000 7f0c0fcea640 10 monclient: _renew_subs 2024-09-16T03:55:35.233 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.203+0000 7f0c0fcea640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:55:35.233 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.203+0000 7f0c0fcea640 1 -- 172.21.15.42:0/3292945069 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c0809d230 con 0x7f0c080a2250 2024-09-16T03:55:35.233 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.203+0000 7f0c0fcea640 1 librados: init done 2024-09-16T03:55:35.233 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.203+0000 7f0c0fcea640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:55:35.233 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.203+0000 7f0c0fcea640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:55:35.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.203+0000 7f0c0fcea640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:55:35.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.203+0000 7f0bfe7fc640 1 -- 172.21.15.42:0/3292945069 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f0bf8005450 con 0x7f0c080a2250 2024-09-16T03:55:35.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.203+0000 7f0c0fcea640 1 -- 172.21.15.42:0/3292945069 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0c08130f00 con 0x7f0c080a2250 2024-09-16T03:55:35.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.203+0000 7f0bfe7fc640 1 --2- 172.21.15.42:0/3292945069 >> v2:172.21.15.154:6812/254845342 conn(0x7f0bec0377c0 0x7f0bec039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:55:35.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.203+0000 7f0bfe7fc640 1 -- 172.21.15.42:0/3292945069 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(418..418 src has 330..418) ==== 6096+0+0 (secure 0 0 0) 0x7f0bf8048e00 con 0x7f0c080a2250 2024-09-16T03:55:35.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.203+0000 7f0bfe7fc640 10 client.854246.objecter ms_dispatch 0x7f0c08000ba0 osd_map(418..418 src has 330..418) 2024-09-16T03:55:35.235 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.203+0000 7f0bfe7fc640 3 client.854246.objecter handle_osd_map got epochs [418,418] > 0 2024-09-16T03:55:35.235 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.203+0000 7f0bfe7fc640 3 client.854246.objecter handle_osd_map decoding full epoch 418 2024-09-16T03:55:35.235 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.203+0000 7f0bfe7fc640 20 client.854246.objecter dump_active .. 0 homeless 2024-09-16T03:55:35.235 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.203+0000 7f0c0da86640 1 --2- 172.21.15.42:0/3292945069 >> v2:172.21.15.154:6812/254845342 conn(0x7f0bec0377c0 0x7f0bec039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:55:35.235 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.203+0000 7f0c0da86640 10 monclient: get_auth_request con 0x7f0bec0377c0 auth_method 0 2024-09-16T03:55:35.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.203+0000 7f0c0da86640 1 --2- 172.21.15.42:0/3292945069 >> v2:172.21.15.154:6812/254845342 conn(0x7f0bec0377c0 0x7f0bec039c80 secure :-1 s=READY pgs=3642 cs=0 l=1 rev1=1 crypto rx=0x7f0c04009580 tx=0x7f0c04009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:55:35.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.203+0000 7f0bfe7fc640 10 client.854246.objecter ms_handle_connect 0x7f0bec0377c0 2024-09-16T03:55:35.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.207+0000 7f0bfe7fc640 1 -- 172.21.15.42:0/3292945069 <== mon.2 v2:172.21.15.42:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f0c080a42a0 con 0x7f0c080a2250 2024-09-16T03:55:35.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.207+0000 7f0bfe7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:55:35.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.207+0000 7f0bfe7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:55:35.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.431+0000 7f0c0fcea640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:55:35.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.431+0000 7f0c0fcea640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:55:35.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.431+0000 7f0c0fcea640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:55:35.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.431+0000 7f0c0fcea640 1 -- 172.21.15.42:0/3292945069 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f0c0806b490 con 0x7f0c080a2250 2024-09-16T03:55:35.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.431+0000 7f0bfe7fc640 1 -- 172.21.15.42:0/3292945069 <== mon.2 v2:172.21.15.42:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7f0bf8010480 con 0x7f0c080a2250 2024-09-16T03:55:35.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.431+0000 7f0bfe7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:55:35.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.431+0000 7f0bfe7fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:55:35.435 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":116,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:55:35.437 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.431+0000 7f0c0fcea640 10 librados: watch_flush enter 2024-09-16T03:55:35.438 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.431+0000 7f0c0fcea640 10 librados: watch_flush exit 2024-09-16T03:55:35.438 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.431+0000 7f0c0fcea640 20 client.854246.objecter shutdown clearing up homeless session... 2024-09-16T03:55:35.438 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.431+0000 7f0c0fcea640 10 client.854246.objecter successfully canceled tick 2024-09-16T03:55:35.438 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.435+0000 7f0c0fcea640 1 -- 172.21.15.42:0/3292945069 >> v2:172.21.15.154:6812/254845342 conn(0x7f0bec0377c0 msgr2=0x7f0bec039c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:55:35.438 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.435+0000 7f0c0fcea640 1 --2- 172.21.15.42:0/3292945069 >> v2:172.21.15.154:6812/254845342 conn(0x7f0bec0377c0 0x7f0bec039c80 secure :-1 s=READY pgs=3642 cs=0 l=1 rev1=1 crypto rx=0x7f0c04009580 tx=0x7f0c04009190 comp rx=0 tx=0).stop 2024-09-16T03:55:35.438 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.435+0000 7f0c0fcea640 10 monclient: shutdown 2024-09-16T03:55:35.438 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.435+0000 7f0c0fcea640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:55:35.438 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.435+0000 7f0c0fcea640 1 -- 172.21.15.42:0/3292945069 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0c080a2250 msgr2=0x7f0c0813e2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:55:35.438 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.435+0000 7f0c0fcea640 1 --2- 172.21.15.42:0/3292945069 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0c080a2250 0x7f0c0813e2b0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f0bf800b480 tx=0x7f0bf800b940 comp rx=0 tx=0).stop 2024-09-16T03:55:35.438 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.435+0000 7f0c0fcea640 1 -- 172.21.15.42:0/3292945069 shutdown_connections 2024-09-16T03:55:35.438 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.435+0000 7f0c0fcea640 1 --2- 172.21.15.42:0/3292945069 >> v2:172.21.15.154:6812/254845342 conn(0x7f0bec0377c0 0x7f0bec039c80 unknown :-1 s=CLOSED pgs=3642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:55:35.439 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.435+0000 7f0c0fcea640 1 --2- 172.21.15.42:0/3292945069 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0c08130b20 0x7f0c0813e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:55:35.439 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.435+0000 7f0c0fcea640 1 --2- 172.21.15.42:0/3292945069 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0c080a2250 0x7f0c0813e2b0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:55:35.439 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.435+0000 7f0c0fcea640 1 --2- 172.21.15.42:0/3292945069 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0c08073420 0x7f0c081cf0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:55:35.439 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.435+0000 7f0c0fcea640 1 -- 172.21.15.42:0/3292945069 >> 172.21.15.42:0/3292945069 conn(0x7f0c080a6210 msgr2=0x7f0c08133a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:55:35.439 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.435+0000 7f0c0fcea640 1 -- 172.21.15.42:0/3292945069 shutdown_connections 2024-09-16T03:55:35.439 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.435+0000 7f0c0fcea640 1 -- 172.21.15.42:0/3292945069 wait complete. 2024-09-16T03:55:35.439 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:35.435+0000 7f0c0fcea640 1 librados: shutdown 2024-09-16T03:55:38.454 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:55:38.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.583+0000 7f2dd5cd7640 10 monclient: get_monmap_and_config 2024-09-16T03:55:38.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.583+0000 7f2dd5cd7640 10 monclient: build_initial_monmap 2024-09-16T03:55:38.588 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.583+0000 7f2dd5cd7640 10 monclient: monmap: 2024-09-16T03:55:38.588 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:55:38.588 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:55:38.588 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:55:38.586367+0000 2024-09-16T03:55:38.588 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:55:38.586367+0000 2024-09-16T03:55:38.588 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:55:38.588 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:55:38.588 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:55:38.588 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:55:38.588 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:55:38.588 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:55:38.588 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:55:38.588 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:55:38.590 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.583+0000 7f2dd5cd7640 1 Processor -- start 2024-09-16T03:55:38.590 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.583+0000 7f2dd5cd7640 1 -- start start 2024-09-16T03:55:38.590 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.583+0000 7f2dd5cd7640 10 monclient: init 2024-09-16T03:55:38.590 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.583+0000 7f2dd5cd7640 10 monclient: _reopen_session rank -1 2024-09-16T03:55:38.590 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.583+0000 7f2dd5cd7640 10 monclient: _add_conns ranks=[0,1,3,4,2] 2024-09-16T03:55:38.590 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.583+0000 7f2dd5cd7640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2dd00a39b0 addr v1:172.21.15.42:6789/0 2024-09-16T03:55:38.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.583+0000 7f2dd5cd7640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2dd00a3d90 addr v1:172.21.15.154:6789/0 2024-09-16T03:55:38.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.583+0000 7f2dd5cd7640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f2dd013daa0 0x7f2dd013fe90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:55:38.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.583+0000 7f2dd5cd7640 10 monclient(hunting): picked mon.noname-a con 0x7f2dd013daa0 addr v2:172.21.15.42:3300/0 2024-09-16T03:55:38.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.583+0000 7f2dd5cd7640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2dd0134290 con 0x7f2dd00a39b0 2024-09-16T03:55:38.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.583+0000 7f2dd5cd7640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2dd009c140 con 0x7f2dd00a3d90 2024-09-16T03:55:38.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.583+0000 7f2dd5cd7640 10 monclient(hunting): start opening mon connection 2024-09-16T03:55:38.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.583+0000 7f2dd5cd7640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f2dd009c420 con 0x7f2dd013daa0 2024-09-16T03:55:38.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.583+0000 7f2dd5cd7640 10 monclient(hunting): _renew_subs 2024-09-16T03:55:38.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.583+0000 7f2dd5cd7640 10 monclient(hunting): authenticate will time out at 1785.555795s 2024-09-16T03:55:38.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.583+0000 7f2dcf7fe640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f2dd00a39b0 0x7f2dd0133b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:46124/0 (socket says 172.21.15.42:46124) 2024-09-16T03:55:38.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.583+0000 7f2dcf7fe640 1 -- 172.21.15.42:0/152006557 learned_addr learned my addr 172.21.15.42:0/152006557 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:55:38.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.583+0000 7f2dceffd640 1 -- 172.21.15.42:0/152006557 >> v1:172.21.15.154:6789/0 conn(0x7f2dd00a3d90 legacy=0x7f2dd013c380 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.154:6789/0 2024-09-16T03:55:38.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.587+0000 7f2dcffff640 1 --2- 172.21.15.42:0/152006557 >> v2:172.21.15.42:3300/0 conn(0x7f2dd013daa0 0x7f2dd013fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:55:38.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.587+0000 7f2dcffff640 10 monclient(hunting): get_auth_request con 0x7f2dd013daa0 auth_method 0 2024-09-16T03:55:38.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.587+0000 7f2dcffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:55:38.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.587+0000 7f2dcffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:55:38.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.587+0000 7f2dcffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:55:38.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.587+0000 7f2dcffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:55:38.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.587+0000 7f2dcffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:55:38.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.587+0000 7f2dcffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:55:38.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.587+0000 7f2dcffff640 10 monclient(hunting): handle_auth_done global_id 824637 payload 274 2024-09-16T03:55:38.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.587+0000 7f2dcffff640 1 -- 172.21.15.42:0/152006557 >> v1:172.21.15.154:6789/0 conn(0x7f2dd00a3d90 legacy=0x7f2dd013c380 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:55:38.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.587+0000 7f2dcffff640 1 -- 172.21.15.42:0/152006557 >> v1:172.21.15.42:6789/0 conn(0x7f2dd00a39b0 legacy=0x7f2dd0133b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:55:38.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.587+0000 7f2dcffff640 10 monclient: _finish_hunting 0 2024-09-16T03:55:38.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.587+0000 7f2dcffff640 1 monclient: found mon.noname-a 2024-09-16T03:55:38.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.587+0000 7f2dcffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:55:38.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.587+0000 7f2dcffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:55:38.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.587+0000 7f2dcffff640 1 -- 172.21.15.42:0/152006557 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2dd006dd60 con 0x7f2dd013daa0 2024-09-16T03:55:38.599 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.587+0000 7f2dcffff640 1 --2- 172.21.15.42:0/152006557 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2dd013daa0 0x7f2dd013fe90 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f2dc8009d90 tx=0x7f2dc801c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca5ff4be99fd9aa4 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:55:38.599 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.587+0000 7f2dce7fc640 1 -- 172.21.15.42:0/152006557 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dc80a66c0 con 0x7f2dd013daa0 2024-09-16T03:55:38.599 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.587+0000 7f2dce7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:55:38.599 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.587+0000 7f2dce7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:55:38.600 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.587+0000 7f2dce7fc640 10 monclient: dump: 2024-09-16T03:55:38.600 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:55:38.600 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:55:38.600 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:38.600 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:38.600 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:55:38.600 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:55:38.600 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:55:38.600 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:55:38.600 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:55:38.600 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:55:38.600 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.587+0000 7f2dce7fc640 10 monclient: _finish_auth 0 2024-09-16T03:55:38.600 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.587+0000 7f2dce7fc640 10 monclient: _check_auth_tickets 2024-09-16T03:55:38.601 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.587+0000 7f2dce7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:55:38.601 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.587+0000 7f2dce7fc640 1 -- 172.21.15.42:0/152006557 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2dc80a6860 con 0x7f2dd013daa0 2024-09-16T03:55:38.601 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.587+0000 7f2dce7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:55:38.601 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.587+0000 7f2dce7fc640 1 -- 172.21.15.42:0/152006557 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dc80a6c30 con 0x7f2dd013daa0 2024-09-16T03:55:38.601 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.587+0000 7f2dd5cd7640 5 monclient: authenticate success, global_id 824637 2024-09-16T03:55:38.601 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.587+0000 7f2dd5cd7640 10 monclient: get_monmap_and_config success 2024-09-16T03:55:38.601 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.587+0000 7f2dd5cd7640 4 set_mon_vals no callback set 2024-09-16T03:55:38.601 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.587+0000 7f2dce7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:55:38.601 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.587+0000 7f2dce7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:55:38.601 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.587+0000 7f2dce7fc640 10 monclient: dump: 2024-09-16T03:55:38.601 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:55:38.601 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:55:38.601 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:38.601 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:38.602 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:55:38.602 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:55:38.602 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:55:38.602 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:55:38.602 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:55:38.602 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:55:38.602 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.591+0000 7f2dd5cd7640 10 monclient: shutdown 2024-09-16T03:55:38.602 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.591+0000 7f2dd5cd7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:55:38.602 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.591+0000 7f2dd5cd7640 1 -- 172.21.15.42:0/152006557 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2dd013daa0 msgr2=0x7f2dd013fe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:55:38.602 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.591+0000 7f2dd5cd7640 1 --2- 172.21.15.42:0/152006557 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2dd013daa0 0x7f2dd013fe90 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f2dc8009d90 tx=0x7f2dc801c7f0 comp rx=0 tx=0).stop 2024-09-16T03:55:38.602 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.591+0000 7f2dd5cd7640 1 -- 172.21.15.42:0/152006557 shutdown_connections 2024-09-16T03:55:38.602 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.591+0000 7f2dd5cd7640 1 --2- 172.21.15.42:0/152006557 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2dd013daa0 0x7f2dd013fe90 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:55:38.602 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.591+0000 7f2dd5cd7640 1 -- 172.21.15.42:0/152006557 >> 172.21.15.42:0/152006557 conn(0x7f2dd00a9fb0 msgr2=0x7f2dd00aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:55:38.603 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.591+0000 7f2dd5cd7640 1 -- 172.21.15.42:0/152006557 shutdown_connections 2024-09-16T03:55:38.603 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.591+0000 7f2dd5cd7640 1 -- 172.21.15.42:0/152006557 wait complete. 2024-09-16T03:55:38.603 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.591+0000 7f2dd5cd7640 10 monclient: build_initial_monmap 2024-09-16T03:55:38.603 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.591+0000 7f2dd5cd7640 10 monclient: monmap: 2024-09-16T03:55:38.603 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:55:38.603 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:55:38.603 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:55:38.603 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:55:38.603 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:55:38.603 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:55:38.603 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:55:38.603 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:55:38.603 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:55:38.603 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:55:38.604 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.591+0000 7f2dd5cd7640 1 librados: starting msgr at 2024-09-16T03:55:38.604 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.591+0000 7f2dd5cd7640 1 librados: starting objecter 2024-09-16T03:55:38.604 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.591+0000 7f2dd5cd7640 1 Processor -- start 2024-09-16T03:55:38.604 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.591+0000 7f2dd5cd7640 1 -- start start 2024-09-16T03:55:38.604 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.591+0000 7f2dd5cd7640 1 librados: setting wanted keys 2024-09-16T03:55:38.604 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.591+0000 7f2dd5cd7640 1 librados: calling monclient init 2024-09-16T03:55:38.604 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.591+0000 7f2dd5cd7640 10 monclient: init 2024-09-16T03:55:38.604 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.591+0000 7f2dd5cd7640 10 monclient: _reopen_session rank -1 2024-09-16T03:55:38.604 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.591+0000 7f2dd5cd7640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:55:38.604 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.591+0000 7f2dd5cd7640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2dd00a39b0 0x7f2dd01305f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:55:38.604 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.591+0000 7f2dd5cd7640 10 monclient(hunting): picked mon.noname-c con 0x7f2dd00a39b0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:55:38.604 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.591+0000 7f2dd5cd7640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2dd00a3d90 0x7f2dd0130b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:55:38.604 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.591+0000 7f2dd5cd7640 10 monclient(hunting): picked mon.noname-b con 0x7f2dd00a3d90 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:55:38.605 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.591+0000 7f2dd5cd7640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2dd013daa0 0x7f2dd013ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:55:38.605 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.591+0000 7f2dd5cd7640 10 monclient(hunting): picked mon.noname-a con 0x7f2dd013daa0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:55:38.605 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.591+0000 7f2dd5cd7640 10 monclient(hunting): start opening mon connection 2024-09-16T03:55:38.605 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.591+0000 7f2dd5cd7640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f2dd0142f80 con 0x7f2dd013daa0 2024-09-16T03:55:38.605 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.591+0000 7f2dd5cd7640 10 monclient(hunting): start opening mon connection 2024-09-16T03:55:38.605 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.591+0000 7f2dd5cd7640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f2dd0142e00 con 0x7f2dd00a3d90 2024-09-16T03:55:38.605 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.591+0000 7f2dd5cd7640 10 monclient(hunting): start opening mon connection 2024-09-16T03:55:38.605 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.591+0000 7f2dd5cd7640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f2dd0143100 con 0x7f2dd00a39b0 2024-09-16T03:55:38.605 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.591+0000 7f2dd5cd7640 10 monclient(hunting): _renew_subs 2024-09-16T03:55:38.605 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.591+0000 7f2dd5cd7640 10 monclient(hunting): authenticate will time out at 1785.562530s 2024-09-16T03:55:38.605 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.591+0000 7f2dcffff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2dd013daa0 0x7f2dd013ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:55:38.605 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.591+0000 7f2dceffd640 1 -- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2dd00a3d90 msgr2=0x7f2dd0130b30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T03:55:38.605 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.591+0000 7f2dcf7fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2dd00a39b0 0x7f2dd01305f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:55:38.605 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.591+0000 7f2dceffd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2dd00a3d90 0x7f2dd0130b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:55:38.606 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.591+0000 7f2dcffff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2dd013daa0 0x7f2dd013ab80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:51974/0 (socket says 172.21.15.42:51974) 2024-09-16T03:55:38.606 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.591+0000 7f2dcffff640 1 -- 172.21.15.42:0/2075611758 learned_addr learned my addr 172.21.15.42:0/2075611758 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:55:38.606 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.591+0000 7f2dcf7fe640 10 monclient(hunting): get_auth_request con 0x7f2dd00a39b0 auth_method 0 2024-09-16T03:55:38.606 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.591+0000 7f2dcf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:55:38.606 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.591+0000 7f2dcf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:55:38.606 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.591+0000 7f2dcf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:55:38.606 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.591+0000 7f2dcffff640 10 monclient(hunting): get_auth_request con 0x7f2dd013daa0 auth_method 0 2024-09-16T03:55:38.606 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.591+0000 7f2dcffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:55:38.606 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.591+0000 7f2dcffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:55:38.606 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.591+0000 7f2dcffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:55:38.606 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.591+0000 7f2dcf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:55:38.606 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.591+0000 7f2dcf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:55:38.606 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.591+0000 7f2dcf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:55:38.606 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.591+0000 7f2dcffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:55:38.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.591+0000 7f2dcffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:55:38.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.591+0000 7f2dcffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:55:38.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.591+0000 7f2dcf7fe640 10 monclient(hunting): handle_auth_done global_id 854249 payload 824 2024-09-16T03:55:38.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.591+0000 7f2dcf7fe640 1 -- 172.21.15.42:0/2075611758 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2dd00a3d90 msgr2=0x7f2dd0130b30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:55:38.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.591+0000 7f2dcf7fe640 1 --2- 172.21.15.42:0/2075611758 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2dd00a3d90 0x7f2dd0130b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:55:38.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.591+0000 7f2dcf7fe640 1 -- 172.21.15.42:0/2075611758 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2dd013daa0 msgr2=0x7f2dd013ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:55:38.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.591+0000 7f2dcf7fe640 1 --2- 172.21.15.42:0/2075611758 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2dd013daa0 0x7f2dd013ab80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:55:38.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.591+0000 7f2dcf7fe640 10 monclient: _finish_hunting 0 2024-09-16T03:55:38.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.591+0000 7f2dcf7fe640 1 monclient: found mon.noname-c 2024-09-16T03:55:38.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.591+0000 7f2dcf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:55:38.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.591+0000 7f2dcf7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:55:38.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.591+0000 7f2dcf7fe640 1 -- 172.21.15.42:0/2075611758 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2dd009d940 con 0x7f2dd00a39b0 2024-09-16T03:55:38.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.591+0000 7f2dcf7fe640 1 --2- 172.21.15.42:0/2075611758 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2dd00a39b0 0x7f2dd01305f0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f2dc000c3c0 tx=0x7f2dc000c880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:55:38.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.591+0000 7f2db7fff640 10 client.?.objecter ms_handle_connect 0x7f2dd00a39b0 2024-09-16T03:55:38.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.591+0000 7f2db7fff640 10 client.?.objecter resend_mon_ops 2024-09-16T03:55:38.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.591+0000 7f2db7fff640 1 -- 172.21.15.42:0/2075611758 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dc0017070 con 0x7f2dd00a39b0 2024-09-16T03:55:38.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.591+0000 7f2db7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:55:38.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.591+0000 7f2db7fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:55:38.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.591+0000 7f2db7fff640 10 monclient: dump: 2024-09-16T03:55:38.608 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:55:38.608 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:55:38.608 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:38.608 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:38.608 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:55:38.608 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:55:38.608 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:55:38.608 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:55:38.609 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:55:38.609 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:55:38.609 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.591+0000 7f2db7fff640 10 monclient: _finish_auth 0 2024-09-16T03:55:38.609 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.591+0000 7f2db7fff640 10 monclient: _check_auth_tickets 2024-09-16T03:55:38.609 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.591+0000 7f2db7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:55:38.609 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.591+0000 7f2db7fff640 1 -- 172.21.15.42:0/2075611758 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2dc0005cf0 con 0x7f2dd00a39b0 2024-09-16T03:55:38.609 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.591+0000 7f2db7fff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:55:38.609 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.591+0000 7f2db7fff640 1 -- 172.21.15.42:0/2075611758 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dc00072f0 con 0x7f2dd00a39b0 2024-09-16T03:55:38.609 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.591+0000 7f2db7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:55:38.609 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.591+0000 7f2db7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:55:38.609 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.591+0000 7f2db7fff640 10 monclient: dump: 2024-09-16T03:55:38.609 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:55:38.609 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:55:38.609 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:38.610 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:38.610 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:55:38.610 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:55:38.610 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:55:38.610 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:55:38.610 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:55:38.610 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:55:38.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.591+0000 7f2dd5cd7640 5 monclient: authenticate success, global_id 854249 2024-09-16T03:55:38.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.591+0000 7f2dd5cd7640 10 monclient: _renew_subs 2024-09-16T03:55:38.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.591+0000 7f2dd5cd7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:55:38.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.591+0000 7f2dd5cd7640 1 -- 172.21.15.42:0/2075611758 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2dd006f680 con 0x7f2dd00a39b0 2024-09-16T03:55:38.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.591+0000 7f2dcdffb640 4 set_mon_vals no callback set 2024-09-16T03:55:38.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.591+0000 7f2dd5cd7640 10 client.854249.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:55:38.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.591+0000 7f2dd5cd7640 10 monclient: _renew_subs 2024-09-16T03:55:38.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.591+0000 7f2dd5cd7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:55:38.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.591+0000 7f2dd5cd7640 1 -- 172.21.15.42:0/2075611758 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f2dd0132a20 con 0x7f2dd00a39b0 2024-09-16T03:55:38.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.591+0000 7f2dd5cd7640 1 librados: init done 2024-09-16T03:55:38.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.595+0000 7f2db7fff640 1 -- 172.21.15.42:0/2075611758 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f2dc001a070 con 0x7f2dd00a39b0 2024-09-16T03:55:38.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.595+0000 7f2dd5cd7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:55:38.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.595+0000 7f2db7fff640 1 --2- 172.21.15.42:0/2075611758 >> v2:172.21.15.154:6812/254845342 conn(0x7f2dbc0377c0 0x7f2dbc039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:55:38.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.595+0000 7f2dd5cd7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:55:38.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.595+0000 7f2dd5cd7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:55:38.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.595+0000 7f2dd5cd7640 1 -- 172.21.15.42:0/2075611758 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2dd0133b80 con 0x7f2dd00a39b0 2024-09-16T03:55:38.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.595+0000 7f2db7fff640 1 -- 172.21.15.42:0/2075611758 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(418..418 src has 330..418) ==== 6096+0+0 (secure 0 0 0) 0x7f2dc0048310 con 0x7f2dd00a39b0 2024-09-16T03:55:38.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.595+0000 7f2db7fff640 10 client.854249.objecter ms_dispatch 0x7f2dd0000ba0 osd_map(418..418 src has 330..418) 2024-09-16T03:55:38.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.595+0000 7f2db7fff640 3 client.854249.objecter handle_osd_map got epochs [418,418] > 0 2024-09-16T03:55:38.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.595+0000 7f2db7fff640 3 client.854249.objecter handle_osd_map decoding full epoch 418 2024-09-16T03:55:38.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.595+0000 7f2db7fff640 20 client.854249.objecter dump_active .. 0 homeless 2024-09-16T03:55:38.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.595+0000 7f2dceffd640 1 --2- 172.21.15.42:0/2075611758 >> v2:172.21.15.154:6812/254845342 conn(0x7f2dbc0377c0 0x7f2dbc039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:55:38.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.595+0000 7f2dceffd640 10 monclient: get_auth_request con 0x7f2dbc0377c0 auth_method 0 2024-09-16T03:55:38.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.595+0000 7f2dceffd640 1 --2- 172.21.15.42:0/2075611758 >> v2:172.21.15.154:6812/254845342 conn(0x7f2dbc0377c0 0x7f2dbc039c80 secure :-1 s=READY pgs=3643 cs=0 l=1 rev1=1 crypto rx=0x7f2dd013d9f0 tx=0x7f2dc4006d90 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:55:38.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.595+0000 7f2db7fff640 10 client.854249.objecter ms_handle_connect 0x7f2dbc0377c0 2024-09-16T03:55:38.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.599+0000 7f2db7fff640 1 -- 172.21.15.42:0/2075611758 <== mon.2 v2:172.21.15.42:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f2dc0019e10 con 0x7f2dd00a39b0 2024-09-16T03:55:38.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.599+0000 7f2db7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:55:38.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.599+0000 7f2db7fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:55:38.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.831+0000 7f2dd5cd7640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:55:38.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.831+0000 7f2dd5cd7640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:55:38.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.831+0000 7f2dd5cd7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:55:38.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.831+0000 7f2dd5cd7640 1 -- 172.21.15.42:0/2075611758 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f2dd00a5750 con 0x7f2dd00a39b0 2024-09-16T03:55:38.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.831+0000 7f2db7fff640 1 -- 172.21.15.42:0/2075611758 <== mon.2 v2:172.21.15.42:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1562 (secure 0 0 0) 0x7f2dc0010900 con 0x7f2dd00a39b0 2024-09-16T03:55:38.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.831+0000 7f2db7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:55:38.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.831+0000 7f2db7fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:55:38.836 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":116,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:55:38.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.835+0000 7f2dd5cd7640 10 librados: watch_flush enter 2024-09-16T03:55:38.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.835+0000 7f2dd5cd7640 10 librados: watch_flush exit 2024-09-16T03:55:38.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.835+0000 7f2dd5cd7640 20 client.854249.objecter shutdown clearing up homeless session... 2024-09-16T03:55:38.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.835+0000 7f2dd5cd7640 10 client.854249.objecter successfully canceled tick 2024-09-16T03:55:38.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.835+0000 7f2dd5cd7640 1 -- 172.21.15.42:0/2075611758 >> v2:172.21.15.154:6812/254845342 conn(0x7f2dbc0377c0 msgr2=0x7f2dbc039c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:55:38.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.835+0000 7f2dd5cd7640 1 --2- 172.21.15.42:0/2075611758 >> v2:172.21.15.154:6812/254845342 conn(0x7f2dbc0377c0 0x7f2dbc039c80 secure :-1 s=READY pgs=3643 cs=0 l=1 rev1=1 crypto rx=0x7f2dd013d9f0 tx=0x7f2dc4006d90 comp rx=0 tx=0).stop 2024-09-16T03:55:38.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.835+0000 7f2dd5cd7640 10 monclient: shutdown 2024-09-16T03:55:38.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.835+0000 7f2dd5cd7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:55:38.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.835+0000 7f2dd5cd7640 1 -- 172.21.15.42:0/2075611758 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2dd00a39b0 msgr2=0x7f2dd01305f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:55:38.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.835+0000 7f2dd5cd7640 1 --2- 172.21.15.42:0/2075611758 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2dd00a39b0 0x7f2dd01305f0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f2dc000c3c0 tx=0x7f2dc000c880 comp rx=0 tx=0).stop 2024-09-16T03:55:38.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.835+0000 7f2dd5cd7640 1 -- 172.21.15.42:0/2075611758 shutdown_connections 2024-09-16T03:55:38.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.835+0000 7f2dd5cd7640 1 --2- 172.21.15.42:0/2075611758 >> v2:172.21.15.154:6812/254845342 conn(0x7f2dbc0377c0 0x7f2dbc039c80 unknown :-1 s=CLOSED pgs=3643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:55:38.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.835+0000 7f2dd5cd7640 1 --2- 172.21.15.42:0/2075611758 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2dd013daa0 0x7f2dd013ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:55:38.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.835+0000 7f2dd5cd7640 1 --2- 172.21.15.42:0/2075611758 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2dd00a3d90 0x7f2dd0130b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:55:38.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.835+0000 7f2dd5cd7640 1 --2- 172.21.15.42:0/2075611758 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2dd00a39b0 0x7f2dd01305f0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:55:38.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.835+0000 7f2dd5cd7640 1 -- 172.21.15.42:0/2075611758 >> 172.21.15.42:0/2075611758 conn(0x7f2dd00a9fb0 msgr2=0x7f2dd00a8c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:55:38.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.835+0000 7f2dd5cd7640 1 -- 172.21.15.42:0/2075611758 shutdown_connections 2024-09-16T03:55:38.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.835+0000 7f2dd5cd7640 1 -- 172.21.15.42:0/2075611758 wait complete. 2024-09-16T03:55:38.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:38.835+0000 7f2dd5cd7640 1 librados: shutdown 2024-09-16T03:55:41.855 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:55:41.987 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:41.983+0000 7f2964b83640 10 monclient: get_monmap_and_config 2024-09-16T03:55:41.987 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:41.983+0000 7f2964b83640 10 monclient: build_initial_monmap 2024-09-16T03:55:41.987 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:41.983+0000 7f2964b83640 10 monclient: monmap: 2024-09-16T03:55:41.987 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:55:41.987 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:55:41.987 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:55:41.985682+0000 2024-09-16T03:55:41.987 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:55:41.985682+0000 2024-09-16T03:55:41.987 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:55:41.987 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:55:41.987 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:55:41.988 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:55:41.988 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:55:41.988 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:55:41.988 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:55:41.988 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:55:41.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:41.983+0000 7f2964b83640 1 Processor -- start 2024-09-16T03:55:41.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:41.983+0000 7f2964b83640 1 -- start start 2024-09-16T03:55:41.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:41.983+0000 7f2964b83640 10 monclient: init 2024-09-16T03:55:41.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:41.983+0000 7f2964b83640 10 monclient: _reopen_session rank -1 2024-09-16T03:55:41.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:41.983+0000 7f2964b83640 10 monclient: _add_conns ranks=[0,1,3,2,4] 2024-09-16T03:55:41.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:41.983+0000 7f2964b83640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f29600a1330 addr v1:172.21.15.42:6789/0 2024-09-16T03:55:41.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:41.983+0000 7f2964b83640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f296009c920 addr v1:172.21.15.154:6789/0 2024-09-16T03:55:41.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:41.983+0000 7f2964b83640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f2960072d40 0x7f296013fca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:55:41.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:41.983+0000 7f2964b83640 10 monclient(hunting): picked mon.noname-a con 0x7f2960072d40 addr v2:172.21.15.42:3300/0 2024-09-16T03:55:41.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:41.983+0000 7f2964b83640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2960072350 con 0x7f29600a1330 2024-09-16T03:55:41.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:41.983+0000 7f2964b83640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f296009bc60 con 0x7f296009c920 2024-09-16T03:55:41.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:41.983+0000 7f2964b83640 10 monclient(hunting): start opening mon connection 2024-09-16T03:55:41.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:41.983+0000 7f2964b83640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f2960072b50 con 0x7f2960072d40 2024-09-16T03:55:41.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:41.983+0000 7f2964b83640 10 monclient(hunting): _renew_subs 2024-09-16T03:55:41.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:41.983+0000 7f2964b83640 10 monclient(hunting): authenticate will time out at 1788.955211s 2024-09-16T03:55:41.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:41.983+0000 7f295e59c640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f29600a1330 0x7f29600a1710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:46136/0 (socket says 172.21.15.42:46136) 2024-09-16T03:55:41.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:41.983+0000 7f295e59c640 1 -- 172.21.15.42:0/1038096286 learned_addr learned my addr 172.21.15.42:0/1038096286 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:55:41.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:41.983+0000 7f295dd9b640 1 -- 172.21.15.42:0/1038096286 >> v1:172.21.15.154:6789/0 conn(0x7f296009c920 legacy=0x7f296013c570 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.154:6789/0 2024-09-16T03:55:41.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:41.983+0000 7f295ed9d640 1 --2- 172.21.15.42:0/1038096286 >> v2:172.21.15.42:3300/0 conn(0x7f2960072d40 0x7f296013fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:55:41.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:41.983+0000 7f295ed9d640 10 monclient(hunting): get_auth_request con 0x7f2960072d40 auth_method 0 2024-09-16T03:55:41.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:41.983+0000 7f295ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:55:41.992 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:41.983+0000 7f295ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:55:41.992 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:41.983+0000 7f295ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:55:41.992 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:41.983+0000 7f295ed9d640 1 -- 172.21.15.42:0/1038096286 >> v2:172.21.15.42:3300/0 conn(0x7f2960072d40 msgr2=0x7f296013fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T03:55:41.992 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:41.983+0000 7f295ed9d640 1 -- 172.21.15.42:0/1038096286 >> v2:172.21.15.42:3300/0 conn(0x7f2960072d40 msgr2=0x7f296013fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:55:41.992 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:41.983+0000 7f295ed9d640 1 --2- 172.21.15.42:0/1038096286 >> v2:172.21.15.42:3300/0 conn(0x7f2960072d40 0x7f296013fca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:55:41.992 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:41.983+0000 7f295ed9d640 1 --2- 172.21.15.42:0/1038096286 >> v2:172.21.15.42:3300/0 conn(0x7f2960072d40 0x7f296013fca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:55:42.190 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:42.187+0000 7f295dd9b640 1 -- 172.21.15.42:0/1038096286 >> v1:172.21.15.154:6789/0 conn(0x7f296009c920 legacy=0x7f296013c570 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.154:6789/0 2024-09-16T03:55:42.190 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:42.187+0000 7f295ed9d640 1 --2- 172.21.15.42:0/1038096286 >> v2:172.21.15.42:3300/0 conn(0x7f2960072d40 0x7f296013fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:55:42.190 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:42.187+0000 7f295ed9d640 10 monclient(hunting): get_auth_request con 0x7f2960072d40 auth_method 0 2024-09-16T03:55:42.190 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:42.187+0000 7f295ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:55:42.190 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:42.187+0000 7f295ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:55:42.190 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:42.187+0000 7f295ed9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:55:42.190 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:42.187+0000 7f295ed9d640 1 -- 172.21.15.42:0/1038096286 >> v2:172.21.15.42:3300/0 conn(0x7f2960072d40 msgr2=0x7f296013fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T03:55:42.190 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:42.187+0000 7f295ed9d640 1 -- 172.21.15.42:0/1038096286 >> v2:172.21.15.42:3300/0 conn(0x7f2960072d40 msgr2=0x7f296013fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:55:42.190 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:42.187+0000 7f295ed9d640 1 --2- 172.21.15.42:0/1038096286 >> v2:172.21.15.42:3300/0 conn(0x7f2960072d40 0x7f296013fca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:55:42.190 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:42.187+0000 7f295ed9d640 1 --2- 172.21.15.42:0/1038096286 >> v2:172.21.15.42:3300/0 conn(0x7f2960072d40 0x7f296013fca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T03:55:42.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:42.591+0000 7f295dd9b640 1 -- 172.21.15.42:0/1038096286 >> v1:172.21.15.154:6789/0 conn(0x7f296009c920 legacy=0x7f296013c570 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.154:6789/0 2024-09-16T03:55:42.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:42.591+0000 7f295ed9d640 1 --2- 172.21.15.42:0/1038096286 >> v2:172.21.15.42:3300/0 conn(0x7f2960072d40 0x7f296013fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:55:42.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:42.591+0000 7f295ed9d640 10 monclient(hunting): get_auth_request con 0x7f2960072d40 auth_method 0 2024-09-16T03:55:42.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:42.591+0000 7f295ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:55:42.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:42.591+0000 7f295ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:55:42.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:42.591+0000 7f295ed9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:55:42.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:42.591+0000 7f295ed9d640 1 -- 172.21.15.42:0/1038096286 >> v2:172.21.15.42:3300/0 conn(0x7f2960072d40 msgr2=0x7f296013fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T03:55:42.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:42.591+0000 7f295ed9d640 1 -- 172.21.15.42:0/1038096286 >> v2:172.21.15.42:3300/0 conn(0x7f2960072d40 msgr2=0x7f296013fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:55:42.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:42.591+0000 7f295ed9d640 1 --2- 172.21.15.42:0/1038096286 >> v2:172.21.15.42:3300/0 conn(0x7f2960072d40 0x7f296013fca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:55:42.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:42.591+0000 7f295ed9d640 1 --2- 172.21.15.42:0/1038096286 >> v2:172.21.15.42:3300/0 conn(0x7f2960072d40 0x7f296013fca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T03:55:43.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:43.395+0000 7f295dd9b640 1 -- 172.21.15.42:0/1038096286 >> v1:172.21.15.154:6789/0 conn(0x7f296009c920 legacy=0x7f296013c570 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.154:6789/0 2024-09-16T03:55:43.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:43.395+0000 7f295ed9d640 1 --2- 172.21.15.42:0/1038096286 >> v2:172.21.15.42:3300/0 conn(0x7f2960072d40 0x7f296013fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:55:43.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:43.395+0000 7f295ed9d640 10 monclient(hunting): get_auth_request con 0x7f2960072d40 auth_method 0 2024-09-16T03:55:43.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:43.395+0000 7f295ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:55:43.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:43.395+0000 7f295ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:55:43.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:43.395+0000 7f295ed9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:55:43.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:43.395+0000 7f295ed9d640 1 -- 172.21.15.42:0/1038096286 >> v2:172.21.15.42:3300/0 conn(0x7f2960072d40 msgr2=0x7f296013fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T03:55:43.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:43.395+0000 7f295ed9d640 1 -- 172.21.15.42:0/1038096286 >> v2:172.21.15.42:3300/0 conn(0x7f2960072d40 msgr2=0x7f296013fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:55:43.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:43.395+0000 7f295ed9d640 1 --2- 172.21.15.42:0/1038096286 >> v2:172.21.15.42:3300/0 conn(0x7f2960072d40 0x7f296013fca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:55:43.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:43.395+0000 7f295ed9d640 1 --2- 172.21.15.42:0/1038096286 >> v2:172.21.15.42:3300/0 conn(0x7f2960072d40 0x7f296013fca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T03:55:44.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:44.983+0000 7f2957fff640 10 monclient(hunting): tick 2024-09-16T03:55:44.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:44.983+0000 7f2957fff640 10 monclient(hunting): _check_auth_tickets 2024-09-16T03:55:44.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:44.983+0000 7f2957fff640 1 monclient(hunting): continuing hunt 2024-09-16T03:55:44.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:44.983+0000 7f2957fff640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T03:55:44.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:44.983+0000 7f2957fff640 1 -- 172.21.15.42:0/1038096286 >> v2:172.21.15.42:3300/0 conn(0x7f2960072d40 msgr2=0x7f296013fca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:55:44.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:44.983+0000 7f2957fff640 1 --2- 172.21.15.42:0/1038096286 >> v2:172.21.15.42:3300/0 conn(0x7f2960072d40 0x7f296013fca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:55:44.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:44.983+0000 7f2957fff640 1 -- 172.21.15.42:0/1038096286 >> v1:172.21.15.154:6789/0 conn(0x7f296009c920 legacy=0x7f296013c570 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:55:44.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:44.983+0000 7f2957fff640 1 -- 172.21.15.42:0/1038096286 >> v1:172.21.15.42:6789/0 conn(0x7f29600a1330 legacy=0x7f29600a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:55:44.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:44.983+0000 7f2957fff640 10 monclient: _add_conns ranks=[4,2] 2024-09-16T03:55:44.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:44.983+0000 7f2957fff640 1 --2- 172.21.15.42:0/1038096286 >> v2:172.21.15.154:3300/0 conn(0x7f29400036d0 0x7f2940005b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:55:44.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:44.983+0000 7f2957fff640 10 monclient(hunting): picked mon.noname-c con 0x7f29400036d0 addr v2:172.21.15.154:3300/0 2024-09-16T03:55:44.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:44.983+0000 7f2957fff640 1 --2- 172.21.15.42:0/1038096286 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f29400061e0 0x7f29400086a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:55:44.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:44.983+0000 7f2957fff640 10 monclient(hunting): picked mon.noname-b con 0x7f29400061e0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:55:44.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:44.983+0000 7f2957fff640 10 monclient(hunting): start opening mon connection 2024-09-16T03:55:44.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:44.983+0000 7f2957fff640 1 -- 172.21.15.42:0/1038096286 --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f2960072b50 con 0x7f29400036d0 2024-09-16T03:55:44.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:44.983+0000 7f2957fff640 10 monclient(hunting): start opening mon connection 2024-09-16T03:55:44.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:44.983+0000 7f2957fff640 1 -- 172.21.15.42:0/1038096286 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f2940008dd0 con 0x7f29400061e0 2024-09-16T03:55:44.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:44.983+0000 7f2957fff640 10 monclient(hunting): _renew_subs 2024-09-16T03:55:44.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:44.983+0000 7f295e59c640 1 -- 172.21.15.42:0/1038096286 >> v2:172.21.15.154:3300/0 conn(0x7f29400036d0 msgr2=0x7f2940005b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T03:55:44.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:44.983+0000 7f295dd9b640 1 --2- 172.21.15.42:0/1038096286 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f29400061e0 0x7f29400086a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:55:44.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:44.983+0000 7f295e59c640 1 --2- 172.21.15.42:0/1038096286 >> v2:172.21.15.154:3300/0 conn(0x7f29400036d0 0x7f2940005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:55:44.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:44.983+0000 7f295dd9b640 10 monclient(hunting): get_auth_request con 0x7f29400061e0 auth_method 0 2024-09-16T03:55:44.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:44.983+0000 7f295dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:55:44.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:44.983+0000 7f295dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:55:44.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:44.983+0000 7f295dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:55:44.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:44.987+0000 7f295dd9b640 1 -- 172.21.15.42:0/1038096286 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f29400061e0 msgr2=0x7f29400086a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T03:55:44.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:44.987+0000 7f295dd9b640 1 -- 172.21.15.42:0/1038096286 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f29400061e0 msgr2=0x7f29400086a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:55:44.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:44.987+0000 7f295dd9b640 1 --2- 172.21.15.42:0/1038096286 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f29400061e0 0x7f29400086a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:55:44.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:44.987+0000 7f295dd9b640 1 --2- 172.21.15.42:0/1038096286 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f29400061e0 0x7f29400086a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:55:45.189 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:45.187+0000 7f295e59c640 1 -- 172.21.15.42:0/1038096286 >> v2:172.21.15.154:3300/0 conn(0x7f29400036d0 msgr2=0x7f2940005b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T03:55:45.190 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:45.187+0000 7f295e59c640 1 --2- 172.21.15.42:0/1038096286 >> v2:172.21.15.154:3300/0 conn(0x7f29400036d0 0x7f2940005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T03:55:45.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:45.191+0000 7f295dd9b640 1 --2- 172.21.15.42:0/1038096286 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f29400061e0 0x7f29400086a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:55:45.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:45.191+0000 7f295dd9b640 10 monclient(hunting): get_auth_request con 0x7f29400061e0 auth_method 0 2024-09-16T03:55:45.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:45.191+0000 7f295dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:55:45.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:45.191+0000 7f295dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:55:45.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:45.191+0000 7f295dd9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:55:45.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:45.191+0000 7f295dd9b640 1 -- 172.21.15.42:0/1038096286 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f29400061e0 msgr2=0x7f29400086a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T03:55:45.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:45.191+0000 7f295dd9b640 1 -- 172.21.15.42:0/1038096286 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f29400061e0 msgr2=0x7f29400086a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:55:45.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:45.191+0000 7f295dd9b640 1 --2- 172.21.15.42:0/1038096286 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f29400061e0 0x7f29400086a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:55:45.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:45.191+0000 7f295dd9b640 1 --2- 172.21.15.42:0/1038096286 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f29400061e0 0x7f29400086a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T03:55:45.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:45.591+0000 7f295e59c640 1 -- 172.21.15.42:0/1038096286 >> v2:172.21.15.154:3300/0 conn(0x7f29400036d0 msgr2=0x7f2940005b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T03:55:45.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:45.591+0000 7f295e59c640 1 --2- 172.21.15.42:0/1038096286 >> v2:172.21.15.154:3300/0 conn(0x7f29400036d0 0x7f2940005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T03:55:45.598 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:45.595+0000 7f295dd9b640 1 --2- 172.21.15.42:0/1038096286 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f29400061e0 0x7f29400086a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:55:45.598 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:45.595+0000 7f295dd9b640 10 monclient(hunting): get_auth_request con 0x7f29400061e0 auth_method 0 2024-09-16T03:55:45.599 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:45.595+0000 7f295dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:55:45.599 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:45.595+0000 7f295dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:55:45.599 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:45.595+0000 7f295dd9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:55:45.599 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:45.595+0000 7f295dd9b640 1 -- 172.21.15.42:0/1038096286 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f29400061e0 msgr2=0x7f29400086a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T03:55:45.599 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:45.595+0000 7f295dd9b640 1 -- 172.21.15.42:0/1038096286 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f29400061e0 msgr2=0x7f29400086a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:55:45.599 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:45.595+0000 7f295dd9b640 1 --2- 172.21.15.42:0/1038096286 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f29400061e0 0x7f29400086a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:55:45.600 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:45.595+0000 7f295dd9b640 1 --2- 172.21.15.42:0/1038096286 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f29400061e0 0x7f29400086a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T03:55:46.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:46.395+0000 7f295e59c640 1 -- 172.21.15.42:0/1038096286 >> v2:172.21.15.154:3300/0 conn(0x7f29400036d0 msgr2=0x7f2940005b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T03:55:46.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:46.395+0000 7f295e59c640 1 --2- 172.21.15.42:0/1038096286 >> v2:172.21.15.154:3300/0 conn(0x7f29400036d0 0x7f2940005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T03:55:46.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:46.399+0000 7f295dd9b640 1 --2- 172.21.15.42:0/1038096286 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f29400061e0 0x7f29400086a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:55:46.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:46.399+0000 7f295dd9b640 10 monclient(hunting): get_auth_request con 0x7f29400061e0 auth_method 0 2024-09-16T03:55:46.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:46.399+0000 7f295dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:55:46.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:46.399+0000 7f295dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:55:46.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:46.399+0000 7f295dd9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:55:46.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:46.399+0000 7f295dd9b640 1 -- 172.21.15.42:0/1038096286 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f29400061e0 msgr2=0x7f29400086a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T03:55:46.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:46.399+0000 7f295dd9b640 1 -- 172.21.15.42:0/1038096286 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f29400061e0 msgr2=0x7f29400086a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:55:46.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:46.399+0000 7f295dd9b640 1 --2- 172.21.15.42:0/1038096286 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f29400061e0 0x7f29400086a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:55:46.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:46.399+0000 7f295dd9b640 1 --2- 172.21.15.42:0/1038096286 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f29400061e0 0x7f29400086a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T03:55:47.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.983+0000 7f2957fff640 10 monclient(hunting): tick 2024-09-16T03:55:47.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.983+0000 7f2957fff640 10 monclient(hunting): _check_auth_tickets 2024-09-16T03:55:47.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.983+0000 7f2957fff640 1 monclient(hunting): continuing hunt 2024-09-16T03:55:47.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.983+0000 7f2957fff640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T03:55:47.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.983+0000 7f2957fff640 1 -- 172.21.15.42:0/1038096286 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f29400061e0 msgr2=0x7f29400086a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:55:47.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.983+0000 7f2957fff640 1 --2- 172.21.15.42:0/1038096286 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f29400061e0 0x7f29400086a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:55:47.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.983+0000 7f2957fff640 1 -- 172.21.15.42:0/1038096286 >> v2:172.21.15.154:3300/0 conn(0x7f29400036d0 msgr2=0x7f2940005b50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:55:47.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.983+0000 7f2957fff640 1 --2- 172.21.15.42:0/1038096286 >> v2:172.21.15.154:3300/0 conn(0x7f29400036d0 0x7f2940005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:55:47.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.983+0000 7f2957fff640 10 monclient: _add_conns ranks=[4,0,1,2,3] 2024-09-16T03:55:47.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.983+0000 7f2957fff640 1 --2- 172.21.15.42:0/1038096286 >> v2:172.21.15.154:3300/0 conn(0x7f29400091d0 0x7f294000b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:55:47.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.983+0000 7f295ed9d640 1 -- 172.21.15.42:0/1038096286 reap_dead start 2024-09-16T03:55:47.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.983+0000 7f2957fff640 10 monclient(hunting): picked mon.noname-c con 0x7f29400091d0 addr v2:172.21.15.154:3300/0 2024-09-16T03:55:47.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.983+0000 7f2957fff640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2940003b70 addr v1:172.21.15.42:6789/0 2024-09-16T03:55:47.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.983+0000 7f2957fff640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2940007de0 addr v1:172.21.15.154:6789/0 2024-09-16T03:55:47.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.983+0000 7f2957fff640 1 -- 172.21.15.42:0/1038096286 --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f296009bf40 con 0x7f2940003b70 2024-09-16T03:55:47.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.983+0000 7f2957fff640 1 -- 172.21.15.42:0/1038096286 --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f296009bc60 con 0x7f2940007de0 2024-09-16T03:55:47.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.983+0000 7f2957fff640 10 monclient(hunting): start opening mon connection 2024-09-16T03:55:47.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.983+0000 7f2957fff640 1 -- 172.21.15.42:0/1038096286 --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f2960072b50 con 0x7f29400091d0 2024-09-16T03:55:47.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.983+0000 7f2957fff640 10 monclient(hunting): _renew_subs 2024-09-16T03:55:47.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.987+0000 7f295ed9d640 1 -- 172.21.15.42:0/1038096286 >> v1:172.21.15.154:6789/0 conn(0x7f2940007de0 legacy=0x7f2940008290 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.154:6789/0 2024-09-16T03:55:47.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.987+0000 7f295e59c640 1 -- 172.21.15.42:0/1038096286 >> v2:172.21.15.154:3300/0 conn(0x7f29400091d0 msgr2=0x7f294000b5c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T03:55:47.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.987+0000 7f295e59c640 1 --2- 172.21.15.42:0/1038096286 >> v2:172.21.15.154:3300/0 conn(0x7f29400091d0 0x7f294000b5c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:55:47.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.987+0000 7f295d59a640 1 -- 172.21.15.42:0/1038096286 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3719549201 0 0) 0x7f296009bf40 con 0x7f2940003b70 2024-09-16T03:55:47.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.987+0000 7f295d59a640 10 monclient(hunting): _negotiate 2024-09-16T03:55:47.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.987+0000 7f295d59a640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:55:47.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.987+0000 7f295d59a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:55:47.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.987+0000 7f295d59a640 10 monclient(hunting): my global_id is 824643 2024-09-16T03:55:47.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.987+0000 7f295d59a640 1 -- 172.21.15.42:0/1038096286 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f294c003530 con 0x7f2940003b70 2024-09-16T03:55:47.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.987+0000 7f295d59a640 1 -- 172.21.15.42:0/1038096286 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1616252042 0 0) 0x7f294c003530 con 0x7f2940003b70 2024-09-16T03:55:47.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.987+0000 7f295d59a640 1 -- 172.21.15.42:0/1038096286 >> v2:172.21.15.154:3300/0 conn(0x7f29400091d0 msgr2=0x7f294000b5c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:55:47.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.987+0000 7f295d59a640 1 --2- 172.21.15.42:0/1038096286 >> v2:172.21.15.154:3300/0 conn(0x7f29400091d0 0x7f294000b5c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:55:47.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.987+0000 7f295d59a640 1 -- 172.21.15.42:0/1038096286 >> v1:172.21.15.154:6789/0 conn(0x7f2940007de0 legacy=0x7f2940008290 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:55:47.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.987+0000 7f295d59a640 10 monclient: _finish_hunting 0 2024-09-16T03:55:47.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.987+0000 7f295d59a640 1 monclient: found mon.noname-a-legacy 2024-09-16T03:55:47.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.987+0000 7f295d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:55:47.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.987+0000 7f295d59a640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.42:6789/0 2024-09-16T03:55:47.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.987+0000 7f295d59a640 1 -- 172.21.15.42:0/1038096286 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2940010be0 con 0x7f2940003b70 2024-09-16T03:55:47.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.987+0000 7f295d59a640 10 monclient: _finish_auth 0 2024-09-16T03:55:47.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.987+0000 7f295d59a640 10 monclient: _check_auth_tickets 2024-09-16T03:55:47.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.987+0000 7f295d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:55:47.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.987+0000 7f2964b83640 5 monclient: authenticate success, global_id 824643 2024-09-16T03:55:47.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.987+0000 7f295d59a640 1 -- 172.21.15.42:0/1038096286 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2177413734 0 0) 0x7f2948009950 con 0x7f2940003b70 2024-09-16T03:55:47.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.987+0000 7f2964b83640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:55:47.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.987+0000 7f295d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:55:47.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.987+0000 7f295d59a640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T03:55:47.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.987+0000 7f295d59a640 10 monclient: dump: 2024-09-16T03:55:47.996 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:55:47.996 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:55:47.996 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:47.997 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:47.997 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:55:47.997 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:55:47.997 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:55:47.997 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:55:47.997 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:55:47.997 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:55:47.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.987+0000 7f295d59a640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] but i'm connected to v1:172.21.15.42:6789/0, reconnecting 2024-09-16T03:55:47.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.987+0000 7f295d59a640 10 monclient: _reopen_session rank -1 2024-09-16T03:55:47.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.987+0000 7f295d59a640 1 -- 172.21.15.42:0/1038096286 >> v1:172.21.15.42:6789/0 conn(0x7f2940003b70 legacy=0x7f2940006690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:55:47.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.987+0000 7f295d59a640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:55:47.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.987+0000 7f295d59a640 1 --2- 172.21.15.42:0/1038096286 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f294c0039c0 0x7f294c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:55:47.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.987+0000 7f295d59a640 10 monclient(hunting): picked mon.b con 0x7f294c0039c0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:55:47.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.987+0000 7f295d59a640 1 --2- 172.21.15.42:0/1038096286 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f294c004810 0x7f294c004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:55:47.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.987+0000 7f295d59a640 10 monclient(hunting): picked mon.a con 0x7f294c004810 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:55:47.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.987+0000 7f295d59a640 1 --2- 172.21.15.42:0/1038096286 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f294c0052f0 0x7f294c00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:55:47.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.987+0000 7f295d59a640 10 monclient(hunting): picked mon.c con 0x7f294c0052f0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:55:47.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.987+0000 7f295d59a640 10 monclient(hunting): start opening mon connection 2024-09-16T03:55:47.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.987+0000 7f295d59a640 1 -- 172.21.15.42:0/1038096286 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f2960072b50 con 0x7f294c004810 2024-09-16T03:55:47.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.987+0000 7f295d59a640 10 monclient(hunting): start opening mon connection 2024-09-16T03:55:47.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.987+0000 7f295d59a640 1 -- 172.21.15.42:0/1038096286 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f294c00c240 con 0x7f294c0039c0 2024-09-16T03:55:47.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.987+0000 7f295d59a640 10 monclient(hunting): start opening mon connection 2024-09-16T03:55:47.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.987+0000 7f295d59a640 1 -- 172.21.15.42:0/1038096286 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f294c00c440 con 0x7f294c0052f0 2024-09-16T03:55:47.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.987+0000 7f295d59a640 10 monclient(hunting): _renew_subs 2024-09-16T03:55:47.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.987+0000 7f295d59a640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:55:47.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.987+0000 7f2964b83640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-16T03:55:47.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.987+0000 7f295e59c640 1 -- 172.21.15.42:0/1038096286 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f294c0039c0 msgr2=0x7f294c003dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T03:55:47.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.987+0000 7f295e59c640 1 --2- 172.21.15.42:0/1038096286 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f294c0039c0 0x7f294c003dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:55:47.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.987+0000 7f295dd9b640 1 --2- 172.21.15.42:0/1038096286 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f294c004810 0x7f294c004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:55:47.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.987+0000 7f295ed9d640 1 --2- 172.21.15.42:0/1038096286 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f294c0052f0 0x7f294c00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:55:47.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.987+0000 7f295dd9b640 10 monclient(hunting): get_auth_request con 0x7f294c004810 auth_method 0 2024-09-16T03:55:47.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.987+0000 7f295dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:55:47.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.987+0000 7f295dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:55:47.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.987+0000 7f295dd9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:55:47.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.987+0000 7f295ed9d640 10 monclient(hunting): get_auth_request con 0x7f294c0052f0 auth_method 0 2024-09-16T03:55:47.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.987+0000 7f295ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:55:47.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.987+0000 7f295ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:55:47.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.987+0000 7f295ed9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:55:47.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.987+0000 7f295dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:55:47.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.987+0000 7f295dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:55:47.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.987+0000 7f295dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T03:55:47.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.987+0000 7f295ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:55:48.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.987+0000 7f295ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:55:48.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.987+0000 7f295ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T03:55:48.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.987+0000 7f295ed9d640 10 monclient(hunting): handle_auth_done global_id 824643 payload 293 2024-09-16T03:55:48.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.987+0000 7f295ed9d640 1 -- 172.21.15.42:0/1038096286 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f294c0039c0 msgr2=0x7f294c003dc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:55:48.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.987+0000 7f295ed9d640 1 --2- 172.21.15.42:0/1038096286 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f294c0039c0 0x7f294c003dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:55:48.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.987+0000 7f295ed9d640 1 -- 172.21.15.42:0/1038096286 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f294c004810 msgr2=0x7f294c004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:55:48.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.987+0000 7f295ed9d640 1 --2- 172.21.15.42:0/1038096286 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f294c004810 0x7f294c004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:55:48.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.987+0000 7f295ed9d640 10 monclient: _finish_hunting 0 2024-09-16T03:55:48.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.987+0000 7f295ed9d640 1 monclient: found mon.c 2024-09-16T03:55:48.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.987+0000 7f295ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:55:48.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.987+0000 7f295ed9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:55:48.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.987+0000 7f295ed9d640 1 -- 172.21.15.42:0/1038096286 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f294c003530 con 0x7f294c0052f0 2024-09-16T03:55:48.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.987+0000 7f295ed9d640 10 monclient: _finish_auth 0 2024-09-16T03:55:48.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.987+0000 7f295ed9d640 10 monclient: _check_auth_tickets 2024-09-16T03:55:48.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.987+0000 7f295ed9d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:55:48.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.987+0000 7f295dd9b640 1 --2- 172.21.15.42:0/1038096286 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f294c004810 0x7f294c004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:55:48.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.995+0000 7f295ed9d640 1 -- 172.21.15.42:0/1038096286 reap_dead start 2024-09-16T03:55:48.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.995+0000 7f295ed9d640 1 --2- 172.21.15.42:0/1038096286 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f294c0052f0 0x7f294c00b9d0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f29600a6950 tx=0x7f295801cf20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b88fa9cb5d6d01d4 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:55:48.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.995+0000 7f295d59a640 1 -- 172.21.15.42:0/1038096286 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f294800b470 con 0x7f294c0052f0 2024-09-16T03:55:48.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.995+0000 7f295d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:55:48.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.995+0000 7f295d59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:55:48.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.995+0000 7f295d59a640 10 monclient: dump: 2024-09-16T03:55:48.003 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:55:48.003 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:55:48.003 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:48.003 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:48.003 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:55:48.003 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:55:48.003 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:55:48.003 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:55:48.004 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:55:48.004 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:55:48.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.995+0000 7f295d59a640 1 -- 172.21.15.42:0/1038096286 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f29580a9070 con 0x7f294c0052f0 2024-09-16T03:55:48.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.995+0000 7f2964b83640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:55:48.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.995+0000 7f295d59a640 10 monclient: handle_config config(0 keys) 2024-09-16T03:55:48.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.995+0000 7f2964b83640 10 monclient: get_monmap_and_config success 2024-09-16T03:55:48.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.995+0000 7f295d59a640 1 -- 172.21.15.42:0/1038096286 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29580ab070 con 0x7f294c0052f0 2024-09-16T03:55:48.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.995+0000 7f2964b83640 4 set_mon_vals no callback set 2024-09-16T03:55:48.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.995+0000 7f295d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:55:48.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.995+0000 7f295d59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:55:48.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.995+0000 7f295d59a640 10 monclient: dump: 2024-09-16T03:55:48.005 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:55:48.006 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:55:48.006 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:48.006 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:48.006 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:55:48.006 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:55:48.006 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:55:48.006 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:55:48.006 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:55:48.006 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:55:48.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.995+0000 7f2964b83640 10 monclient: shutdown 2024-09-16T03:55:48.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.995+0000 7f2964b83640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:55:48.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.995+0000 7f2964b83640 1 -- 172.21.15.42:0/1038096286 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f294c0052f0 msgr2=0x7f294c00b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:55:48.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.995+0000 7f2964b83640 1 --2- 172.21.15.42:0/1038096286 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f294c0052f0 0x7f294c00b9d0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f29600a6950 tx=0x7f295801cf20 comp rx=0 tx=0).stop 2024-09-16T03:55:48.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.995+0000 7f2964b83640 1 -- 172.21.15.42:0/1038096286 shutdown_connections 2024-09-16T03:55:48.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.995+0000 7f2964b83640 1 --2- 172.21.15.42:0/1038096286 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f294c0052f0 0x7f294c00b9d0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:55:48.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.995+0000 7f2964b83640 1 -- 172.21.15.42:0/1038096286 >> 172.21.15.42:0/1038096286 conn(0x7f29600a2420 msgr2=0x7f29600a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:55:48.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.995+0000 7f2964b83640 1 -- 172.21.15.42:0/1038096286 shutdown_connections 2024-09-16T03:55:48.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.995+0000 7f2964b83640 1 -- 172.21.15.42:0/1038096286 wait complete. 2024-09-16T03:55:48.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.999+0000 7f2964b83640 10 monclient: build_initial_monmap 2024-09-16T03:55:48.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.999+0000 7f2964b83640 10 monclient: monmap: 2024-09-16T03:55:48.007 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:55:48.007 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:55:48.007 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:55:48.007 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:55:48.008 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:55:48.008 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:55:48.008 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:55:48.008 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:55:48.008 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:55:48.008 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:55:48.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.999+0000 7f2964b83640 1 librados: starting msgr at 2024-09-16T03:55:48.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.999+0000 7f2964b83640 1 librados: starting objecter 2024-09-16T03:55:48.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.999+0000 7f2964b83640 1 Processor -- start 2024-09-16T03:55:48.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.999+0000 7f2964b83640 1 -- start start 2024-09-16T03:55:48.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.999+0000 7f2964b83640 1 librados: setting wanted keys 2024-09-16T03:55:48.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.999+0000 7f2964b83640 1 librados: calling monclient init 2024-09-16T03:55:48.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.999+0000 7f2964b83640 10 monclient: init 2024-09-16T03:55:48.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.999+0000 7f2964b83640 10 monclient: _reopen_session rank -1 2024-09-16T03:55:48.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.999+0000 7f2964b83640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:55:48.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.999+0000 7f2964b83640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f294c0052f0 0x7f29601ce820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:55:48.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.999+0000 7f2964b83640 10 monclient(hunting): picked mon.noname-b con 0x7f294c0052f0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:55:48.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.999+0000 7f2964b83640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f29601365f0 0x7f2960136a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:55:48.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.999+0000 7f2964b83640 10 monclient(hunting): picked mon.noname-a con 0x7f29601365f0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:55:48.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.999+0000 7f2964b83640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2960121140 0x7f29601215c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:55:48.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.999+0000 7f2964b83640 10 monclient(hunting): picked mon.noname-c con 0x7f2960121140 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:55:48.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.999+0000 7f2964b83640 10 monclient(hunting): start opening mon connection 2024-09-16T03:55:48.018 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.999+0000 7f2964b83640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f2960137120 con 0x7f29601365f0 2024-09-16T03:55:48.018 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.999+0000 7f2964b83640 10 monclient(hunting): start opening mon connection 2024-09-16T03:55:48.018 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.999+0000 7f2964b83640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f29600a21f0 con 0x7f294c0052f0 2024-09-16T03:55:48.018 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.999+0000 7f2964b83640 10 monclient(hunting): start opening mon connection 2024-09-16T03:55:48.018 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.999+0000 7f2964b83640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f29601372a0 con 0x7f2960121140 2024-09-16T03:55:48.018 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.999+0000 7f2964b83640 10 monclient(hunting): _renew_subs 2024-09-16T03:55:48.018 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.999+0000 7f2964b83640 10 monclient(hunting): authenticate will time out at 1794.970617s 2024-09-16T03:55:48.018 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.999+0000 7f295dd9b640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f29601365f0 0x7f2960136a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:55:48.018 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.999+0000 7f295e59c640 1 -- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f294c0052f0 msgr2=0x7f29601ce820 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T03:55:48.018 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.999+0000 7f295dd9b640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f29601365f0 0x7f2960136a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:36722/0 (socket says 172.21.15.42:36722) 2024-09-16T03:55:48.018 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.999+0000 7f295dd9b640 1 -- 172.21.15.42:0/255261564 learned_addr learned my addr 172.21.15.42:0/255261564 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:55:48.018 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.999+0000 7f295dd9b640 10 monclient(hunting): get_auth_request con 0x7f29601365f0 auth_method 0 2024-09-16T03:55:48.018 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.999+0000 7f295dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:55:48.019 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.999+0000 7f295dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:55:48.019 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.999+0000 7f295dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:55:48.019 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.999+0000 7f295e59c640 1 --2- 172.21.15.42:0/255261564 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f294c0052f0 0x7f29601ce820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:55:48.019 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.999+0000 7f295ed9d640 1 --2- 172.21.15.42:0/255261564 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2960121140 0x7f29601215c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:55:48.019 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.999+0000 7f295dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:55:48.019 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.999+0000 7f295dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:55:48.019 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.999+0000 7f295dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:55:48.019 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.999+0000 7f295ed9d640 10 monclient(hunting): get_auth_request con 0x7f2960121140 auth_method 0 2024-09-16T03:55:48.019 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.999+0000 7f295ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:55:48.019 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.999+0000 7f295ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:55:48.019 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.999+0000 7f295ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:55:48.019 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.999+0000 7f295ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:55:48.019 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.999+0000 7f295ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:55:48.019 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.999+0000 7f295ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:55:48.020 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.999+0000 7f295dd9b640 10 monclient(hunting): handle_auth_done global_id 824646 payload 824 2024-09-16T03:55:48.020 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.999+0000 7f295dd9b640 1 -- 172.21.15.42:0/255261564 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2960121140 msgr2=0x7f29601215c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:55:48.020 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.999+0000 7f295dd9b640 1 --2- 172.21.15.42:0/255261564 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2960121140 0x7f29601215c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:55:48.020 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.999+0000 7f295dd9b640 1 -- 172.21.15.42:0/255261564 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f294c0052f0 msgr2=0x7f29601ce820 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:55:48.020 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.999+0000 7f295dd9b640 1 --2- 172.21.15.42:0/255261564 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f294c0052f0 0x7f29601ce820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:55:48.020 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.999+0000 7f295dd9b640 10 monclient: _finish_hunting 0 2024-09-16T03:55:48.020 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.999+0000 7f295dd9b640 1 monclient: found mon.noname-a 2024-09-16T03:55:48.020 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.999+0000 7f295dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:55:48.020 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.999+0000 7f295dd9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:55:48.020 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.999+0000 7f295dd9b640 1 -- 172.21.15.42:0/255261564 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29600a5410 con 0x7f29601365f0 2024-09-16T03:55:48.020 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.999+0000 7f295dd9b640 1 --2- 172.21.15.42:0/255261564 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f29601365f0 0x7f2960136a70 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f2948009b00 tx=0x7f2948005480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:55:48.020 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.999+0000 7f2956ffd640 10 client.?.objecter ms_handle_connect 0x7f29601365f0 2024-09-16T03:55:48.020 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.999+0000 7f2956ffd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:55:48.021 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.999+0000 7f2956ffd640 1 -- 172.21.15.42:0/255261564 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f294801d070 con 0x7f29601365f0 2024-09-16T03:55:48.021 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.999+0000 7f2956ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:55:48.021 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.999+0000 7f2956ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:55:48.021 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.999+0000 7f2956ffd640 10 monclient: dump: 2024-09-16T03:55:48.021 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:55:48.021 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:55:48.021 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:48.021 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:48.021 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:55:48.021 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:55:48.021 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:55:48.021 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:55:48.021 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:55:48.022 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:55:48.022 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.999+0000 7f2956ffd640 10 monclient: _finish_auth 0 2024-09-16T03:55:48.022 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.999+0000 7f2956ffd640 10 monclient: _check_auth_tickets 2024-09-16T03:55:48.022 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.999+0000 7f2956ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:55:48.022 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.999+0000 7f2956ffd640 1 -- 172.21.15.42:0/255261564 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2948005b30 con 0x7f29601365f0 2024-09-16T03:55:48.022 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.999+0000 7f2956ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:55:48.022 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.999+0000 7f2956ffd640 1 -- 172.21.15.42:0/255261564 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2948019640 con 0x7f29601365f0 2024-09-16T03:55:48.022 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.999+0000 7f2964b83640 5 monclient: authenticate success, global_id 824646 2024-09-16T03:55:48.022 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.999+0000 7f2964b83640 10 monclient: _renew_subs 2024-09-16T03:55:48.022 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.999+0000 7f2964b83640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:55:48.023 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.999+0000 7f2964b83640 1 -- 172.21.15.42:0/255261564 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f296009e990 con 0x7f29601365f0 2024-09-16T03:55:48.023 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.999+0000 7f295cd99640 4 set_mon_vals no callback set 2024-09-16T03:55:48.023 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.999+0000 7f2956ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:55:48.023 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.999+0000 7f2956ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:55:48.023 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.999+0000 7f2956ffd640 10 monclient: dump: 2024-09-16T03:55:48.023 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:55:48.024 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:55:48.024 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:48.024 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:48.024 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:55:48.025 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:55:48.025 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:55:48.025 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:55:48.025 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:55:48.025 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:55:48.025 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.999+0000 7f2964b83640 10 client.824646.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:55:48.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.999+0000 7f2964b83640 10 monclient: _renew_subs 2024-09-16T03:55:48.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.999+0000 7f2964b83640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:55:48.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.999+0000 7f2964b83640 1 -- 172.21.15.42:0/255261564 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2960124a00 con 0x7f29601365f0 2024-09-16T03:55:48.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:47.999+0000 7f2964b83640 1 librados: init done 2024-09-16T03:55:48.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.003+0000 7f2964b83640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:55:48.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.003+0000 7f2964b83640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:55:48.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.003+0000 7f2964b83640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:55:48.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.003+0000 7f2964b83640 1 -- 172.21.15.42:0/255261564 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2960137630 con 0x7f29601365f0 2024-09-16T03:55:48.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.003+0000 7f2956ffd640 1 -- 172.21.15.42:0/255261564 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f2948020070 con 0x7f29601365f0 2024-09-16T03:55:48.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.003+0000 7f2956ffd640 1 --2- 172.21.15.42:0/255261564 >> v2:172.21.15.154:6812/254845342 conn(0x7f2940038910 0x7f294003add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:55:48.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.003+0000 7f2956ffd640 1 -- 172.21.15.42:0/255261564 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(420..420 src has 330..420) ==== 6112+0+0 (secure 0 0 0) 0x7f294804a430 con 0x7f29601365f0 2024-09-16T03:55:48.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.003+0000 7f2956ffd640 10 client.824646.objecter ms_dispatch 0x7f2960000ba0 osd_map(420..420 src has 330..420) 2024-09-16T03:55:48.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.003+0000 7f2956ffd640 3 client.824646.objecter handle_osd_map got epochs [420,420] > 0 2024-09-16T03:55:48.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.003+0000 7f2956ffd640 3 client.824646.objecter handle_osd_map decoding full epoch 420 2024-09-16T03:55:48.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.003+0000 7f2956ffd640 20 client.824646.objecter dump_active .. 0 homeless 2024-09-16T03:55:48.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.003+0000 7f295e59c640 1 --2- 172.21.15.42:0/255261564 >> v2:172.21.15.154:6812/254845342 conn(0x7f2940038910 0x7f294003add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:55:48.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.003+0000 7f295e59c640 10 monclient: get_auth_request con 0x7f2940038910 auth_method 0 2024-09-16T03:55:48.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.003+0000 7f295e59c640 1 --2- 172.21.15.42:0/255261564 >> v2:172.21.15.154:6812/254845342 conn(0x7f2940038910 0x7f294003add0 secure :-1 s=READY pgs=3644 cs=0 l=1 rev1=1 crypto rx=0x7f29500015f0 tx=0x7f2950001710 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:55:48.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.003+0000 7f2956ffd640 10 client.824646.objecter ms_handle_connect 0x7f2940038910 2024-09-16T03:55:48.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.007+0000 7f2956ffd640 1 -- 172.21.15.42:0/255261564 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f296009e990 con 0x7f29601365f0 2024-09-16T03:55:48.029 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.007+0000 7f2956ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:55:48.029 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.007+0000 7f2956ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:55:48.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.235+0000 7f2964b83640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:55:48.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.235+0000 7f2964b83640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:55:48.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.235+0000 7f2964b83640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:55:48.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.235+0000 7f2964b83640 1 -- 172.21.15.42:0/255261564 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f2960069040 con 0x7f29601365f0 2024-09-16T03:55:48.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.239+0000 7f2956ffd640 1 -- 172.21.15.42:0/255261564 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1555 (secure 0 0 0) 0x7f29600a5410 con 0x7f29601365f0 2024-09-16T03:55:48.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.239+0000 7f2956ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:55:48.242 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.239+0000 7f2956ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:55:48.242 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":120,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:55:48.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.239+0000 7f2964b83640 10 librados: watch_flush enter 2024-09-16T03:55:48.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.239+0000 7f2964b83640 10 librados: watch_flush exit 2024-09-16T03:55:48.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.239+0000 7f2964b83640 20 client.824646.objecter shutdown clearing up homeless session... 2024-09-16T03:55:48.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.239+0000 7f2964b83640 10 client.824646.objecter successfully canceled tick 2024-09-16T03:55:48.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.239+0000 7f2964b83640 1 -- 172.21.15.42:0/255261564 >> v2:172.21.15.154:6812/254845342 conn(0x7f2940038910 msgr2=0x7f294003add0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:55:48.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.239+0000 7f2964b83640 1 --2- 172.21.15.42:0/255261564 >> v2:172.21.15.154:6812/254845342 conn(0x7f2940038910 0x7f294003add0 secure :-1 s=READY pgs=3644 cs=0 l=1 rev1=1 crypto rx=0x7f29500015f0 tx=0x7f2950001710 comp rx=0 tx=0).stop 2024-09-16T03:55:48.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.239+0000 7f2964b83640 10 monclient: shutdown 2024-09-16T03:55:48.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.239+0000 7f2964b83640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:55:48.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.239+0000 7f2964b83640 1 -- 172.21.15.42:0/255261564 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f29601365f0 msgr2=0x7f2960136a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:55:48.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.239+0000 7f2964b83640 1 --2- 172.21.15.42:0/255261564 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f29601365f0 0x7f2960136a70 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f2948009b00 tx=0x7f2948005480 comp rx=0 tx=0).stop 2024-09-16T03:55:48.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.239+0000 7f2964b83640 1 -- 172.21.15.42:0/255261564 shutdown_connections 2024-09-16T03:55:48.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.239+0000 7f2964b83640 1 --2- 172.21.15.42:0/255261564 >> v2:172.21.15.154:6812/254845342 conn(0x7f2940038910 0x7f294003add0 unknown :-1 s=CLOSED pgs=3644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:55:48.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.239+0000 7f2964b83640 1 --2- 172.21.15.42:0/255261564 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2960121140 0x7f29601215c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:55:48.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.239+0000 7f2964b83640 1 --2- 172.21.15.42:0/255261564 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f29601365f0 0x7f2960136a70 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:55:48.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.239+0000 7f2964b83640 1 --2- 172.21.15.42:0/255261564 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f294c0052f0 0x7f29601ce820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:55:48.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.239+0000 7f2964b83640 1 -- 172.21.15.42:0/255261564 >> 172.21.15.42:0/255261564 conn(0x7f29600a2420 msgr2=0x7f29601c1280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:55:48.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.239+0000 7f2964b83640 1 -- 172.21.15.42:0/255261564 shutdown_connections 2024-09-16T03:55:48.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.239+0000 7f2964b83640 1 -- 172.21.15.42:0/255261564 wait complete. 2024-09-16T03:55:48.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.239+0000 7f2964b83640 1 librados: shutdown 2024-09-16T03:55:48.260 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T03:55:48.261 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-16T03:55:48.261 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T03:55:48.387 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.383+0000 7fb226ed3640 10 monclient: get_monmap_and_config 2024-09-16T03:55:48.387 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.383+0000 7fb226ed3640 10 monclient: build_initial_monmap 2024-09-16T03:55:48.387 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.383+0000 7fb226ed3640 10 monclient: monmap: 2024-09-16T03:55:48.387 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:55:48.387 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:55:48.387 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:55:48.385702+0000 2024-09-16T03:55:48.387 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:55:48.385702+0000 2024-09-16T03:55:48.387 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:55:48.387 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:55:48.387 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:55:48.387 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:55:48.387 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:55:48.388 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:55:48.388 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:55:48.388 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:55:48.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.383+0000 7fb226ed3640 1 Processor -- start 2024-09-16T03:55:48.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.383+0000 7fb226ed3640 1 -- start start 2024-09-16T03:55:48.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.383+0000 7fb226ed3640 10 monclient: init 2024-09-16T03:55:48.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.383+0000 7fb226ed3640 10 monclient: _reopen_session rank -1 2024-09-16T03:55:48.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.383+0000 7fb226ed3640 10 monclient: _add_conns ranks=[0,2,3,1,4] 2024-09-16T03:55:48.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.383+0000 7fb226ed3640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb22006db90 addr v1:172.21.15.42:6789/0 2024-09-16T03:55:48.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.383+0000 7fb226ed3640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb2201340c0 0x7fb22013c190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:55:48.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.383+0000 7fb226ed3640 10 monclient(hunting): picked mon.noname-b con 0x7fb2201340c0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:55:48.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.383+0000 7fb226ed3640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb22013c6d0 0x7fb22013eac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:55:48.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.383+0000 7fb226ed3640 10 monclient(hunting): picked mon.noname-a con 0x7fb22013c6d0 addr v2:172.21.15.42:3300/0 2024-09-16T03:55:48.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.383+0000 7fb226ed3640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb22006ece0 con 0x7fb22006db90 2024-09-16T03:55:48.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.383+0000 7fb226ed3640 10 monclient(hunting): start opening mon connection 2024-09-16T03:55:48.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.383+0000 7fb226ed3640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7fb22006df70 con 0x7fb22013c6d0 2024-09-16T03:55:48.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.383+0000 7fb226ed3640 10 monclient(hunting): start opening mon connection 2024-09-16T03:55:48.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.383+0000 7fb226ed3640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fb22009bf80 con 0x7fb2201340c0 2024-09-16T03:55:48.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.383+0000 7fb226ed3640 10 monclient(hunting): _renew_subs 2024-09-16T03:55:48.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.383+0000 7fb226ed3640 10 monclient(hunting): authenticate will time out at 1795.355413s 2024-09-16T03:55:48.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.383+0000 7fb224c6f640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fb22006db90 0x7fb2200a37f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:34640/0 (socket says 172.21.15.42:34640) 2024-09-16T03:55:48.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.383+0000 7fb224c6f640 1 -- 172.21.15.42:0/509746727 learned_addr learned my addr 172.21.15.42:0/509746727 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:55:48.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.383+0000 7fb225470640 1 --2- 172.21.15.42:0/509746727 >> v2:172.21.15.42:3300/0 conn(0x7fb22013c6d0 0x7fb22013eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:55:48.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.383+0000 7fb225470640 10 monclient(hunting): get_auth_request con 0x7fb22013c6d0 auth_method 0 2024-09-16T03:55:48.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.383+0000 7fb225470640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:55:48.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.383+0000 7fb225470640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:55:48.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.383+0000 7fb225470640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:55:48.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.383+0000 7fb21ffff640 1 --2- 172.21.15.42:0/509746727 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb2201340c0 0x7fb22013c190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:55:48.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.383+0000 7fb21ffff640 10 monclient(hunting): get_auth_request con 0x7fb2201340c0 auth_method 0 2024-09-16T03:55:48.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.383+0000 7fb21ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:55:48.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.383+0000 7fb21ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:55:48.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.383+0000 7fb21ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:55:48.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.383+0000 7fb225470640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:55:48.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.383+0000 7fb225470640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:55:48.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.383+0000 7fb225470640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:55:48.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.387+0000 7fb21f7fe640 1 -- 172.21.15.42:0/509746727 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4028838961 0 0) 0x7fb22006ece0 con 0x7fb22006db90 2024-09-16T03:55:48.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.387+0000 7fb21ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:55:48.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.387+0000 7fb21ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:55:48.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.387+0000 7fb21ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:55:48.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.387+0000 7fb21f7fe640 10 monclient(hunting): _negotiate 2024-09-16T03:55:48.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.387+0000 7fb21f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:55:48.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.387+0000 7fb21f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:55:48.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.387+0000 7fb21f7fe640 10 monclient(hunting): my global_id is 824652 2024-09-16T03:55:48.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.387+0000 7fb21f7fe640 1 -- 172.21.15.42:0/509746727 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb208003530 con 0x7fb22006db90 2024-09-16T03:55:48.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.387+0000 7fb225470640 10 monclient(hunting): handle_auth_done global_id 824649 payload 274 2024-09-16T03:55:48.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.387+0000 7fb225470640 1 -- 172.21.15.42:0/509746727 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb2201340c0 msgr2=0x7fb22013c190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:55:48.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.387+0000 7fb225470640 1 --2- 172.21.15.42:0/509746727 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb2201340c0 0x7fb22013c190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:55:48.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.387+0000 7fb225470640 1 -- 172.21.15.42:0/509746727 >> v1:172.21.15.42:6789/0 conn(0x7fb22006db90 legacy=0x7fb2200a37f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:55:48.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.387+0000 7fb225470640 10 monclient: _finish_hunting 0 2024-09-16T03:55:48.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.387+0000 7fb225470640 1 monclient: found mon.noname-a 2024-09-16T03:55:48.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.387+0000 7fb225470640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:55:48.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.387+0000 7fb225470640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:55:48.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.387+0000 7fb225470640 1 -- 172.21.15.42:0/509746727 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2201344d0 con 0x7fb22013c6d0 2024-09-16T03:55:48.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.387+0000 7fb225470640 1 --2- 172.21.15.42:0/509746727 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb22013c6d0 0x7fb22013eac0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fb218009d90 tx=0x7fb21801c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bef04d1c28ee6f3 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:55:48.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.387+0000 7fb21f7fe640 1 -- 172.21.15.42:0/509746727 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2180b2070 con 0x7fb22013c6d0 2024-09-16T03:55:48.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.387+0000 7fb21f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:55:48.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.387+0000 7fb21f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:55:48.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.387+0000 7fb21f7fe640 10 monclient: dump: 2024-09-16T03:55:48.401 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:55:48.401 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:55:48.401 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:48.401 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:48.401 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:55:48.402 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:55:48.402 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:55:48.402 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:55:48.402 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:55:48.402 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:55:48.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.387+0000 7fb21f7fe640 10 monclient: _finish_auth 0 2024-09-16T03:55:48.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.387+0000 7fb21f7fe640 10 monclient: _check_auth_tickets 2024-09-16T03:55:48.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.387+0000 7fb21f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:55:48.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.387+0000 7fb21f7fe640 1 -- 172.21.15.42:0/509746727 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb2180a58f0 con 0x7fb22013c6d0 2024-09-16T03:55:48.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.387+0000 7fb226ed3640 5 monclient: authenticate success, global_id 824649 2024-09-16T03:55:48.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.387+0000 7fb226ed3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:55:48.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.387+0000 7fb21f7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:55:48.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.387+0000 7fb226ed3640 10 monclient: get_monmap_and_config success 2024-09-16T03:55:48.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.387+0000 7fb21f7fe640 1 -- 172.21.15.42:0/509746727 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2180a5cc0 con 0x7fb22013c6d0 2024-09-16T03:55:48.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.387+0000 7fb21f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:55:48.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.387+0000 7fb226ed3640 4 set_mon_vals no callback set 2024-09-16T03:55:48.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.387+0000 7fb21f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:55:48.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.387+0000 7fb21f7fe640 10 monclient: dump: 2024-09-16T03:55:48.403 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:55:48.404 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:55:48.404 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:48.404 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:48.404 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:55:48.404 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:55:48.404 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:55:48.404 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:55:48.404 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:55:48.404 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:55:48.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.387+0000 7fb226ed3640 10 monclient: shutdown 2024-09-16T03:55:48.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.387+0000 7fb226ed3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:55:48.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.387+0000 7fb226ed3640 1 -- 172.21.15.42:0/509746727 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb22013c6d0 msgr2=0x7fb22013eac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:55:48.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.387+0000 7fb226ed3640 1 --2- 172.21.15.42:0/509746727 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb22013c6d0 0x7fb22013eac0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fb218009d90 tx=0x7fb21801c830 comp rx=0 tx=0).stop 2024-09-16T03:55:48.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.387+0000 7fb226ed3640 1 -- 172.21.15.42:0/509746727 shutdown_connections 2024-09-16T03:55:48.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.387+0000 7fb226ed3640 1 --2- 172.21.15.42:0/509746727 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb22013c6d0 0x7fb22013eac0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:55:48.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.387+0000 7fb226ed3640 1 --2- 172.21.15.42:0/509746727 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb2201340c0 0x7fb22013c190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:55:48.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.387+0000 7fb226ed3640 1 -- 172.21.15.42:0/509746727 >> 172.21.15.42:0/509746727 conn(0x7fb2200a9fb0 msgr2=0x7fb2200aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:55:48.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.387+0000 7fb226ed3640 1 -- 172.21.15.42:0/509746727 shutdown_connections 2024-09-16T03:55:48.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.387+0000 7fb226ed3640 1 -- 172.21.15.42:0/509746727 wait complete. 2024-09-16T03:55:48.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.387+0000 7fb226ed3640 10 monclient: build_initial_monmap 2024-09-16T03:55:48.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.387+0000 7fb226ed3640 10 monclient: monmap: 2024-09-16T03:55:48.405 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:55:48.405 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:55:48.405 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:55:48.405 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:55:48.405 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:55:48.405 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:55:48.406 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:55:48.406 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:55:48.406 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:55:48.406 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:55:48.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.387+0000 7fb226ed3640 1 librados: starting msgr at 2024-09-16T03:55:48.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.387+0000 7fb226ed3640 1 librados: starting objecter 2024-09-16T03:55:48.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.387+0000 7fb226ed3640 1 Processor -- start 2024-09-16T03:55:48.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.391+0000 7fb226ed3640 1 -- start start 2024-09-16T03:55:48.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.391+0000 7fb226ed3640 1 librados: setting wanted keys 2024-09-16T03:55:48.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.391+0000 7fb226ed3640 1 librados: calling monclient init 2024-09-16T03:55:48.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.391+0000 7fb226ed3640 10 monclient: init 2024-09-16T03:55:48.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.391+0000 7fb226ed3640 10 monclient: _reopen_session rank -1 2024-09-16T03:55:48.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.391+0000 7fb226ed3640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T03:55:48.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.391+0000 7fb226ed3640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb22006db90 0x7fb220139de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:55:48.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.391+0000 7fb226ed3640 10 monclient(hunting): picked mon.noname-a con 0x7fb22006db90 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:55:48.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.391+0000 7fb226ed3640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb2201340c0 0x7fb22013a320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:55:48.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.391+0000 7fb226ed3640 10 monclient(hunting): picked mon.noname-b con 0x7fb2201340c0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:55:48.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.391+0000 7fb226ed3640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb22013c6d0 0x7fb22013a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:55:48.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.391+0000 7fb226ed3640 10 monclient(hunting): picked mon.noname-c con 0x7fb22013c6d0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:55:48.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.391+0000 7fb226ed3640 10 monclient(hunting): start opening mon connection 2024-09-16T03:55:48.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.391+0000 7fb226ed3640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fb220141820 con 0x7fb22006db90 2024-09-16T03:55:48.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.391+0000 7fb226ed3640 10 monclient(hunting): start opening mon connection 2024-09-16T03:55:48.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.391+0000 7fb226ed3640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fb2201419a0 con 0x7fb2201340c0 2024-09-16T03:55:48.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.391+0000 7fb226ed3640 10 monclient(hunting): start opening mon connection 2024-09-16T03:55:48.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.391+0000 7fb226ed3640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fb220141b20 con 0x7fb22013c6d0 2024-09-16T03:55:48.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.391+0000 7fb226ed3640 10 monclient(hunting): _renew_subs 2024-09-16T03:55:48.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.391+0000 7fb226ed3640 10 monclient(hunting): authenticate will time out at 1795.360559s 2024-09-16T03:55:48.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.391+0000 7fb224c6f640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb22006db90 0x7fb220139de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:55:48.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.391+0000 7fb225470640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb22013c6d0 0x7fb22013a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:55:48.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.391+0000 7fb21ffff640 1 -- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb2201340c0 msgr2=0x7fb22013a320 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T03:55:48.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.391+0000 7fb21ffff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb2201340c0 0x7fb22013a320 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:55:48.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.391+0000 7fb225470640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb22013c6d0 0x7fb22013a860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:32830/0 (socket says 172.21.15.42:32830) 2024-09-16T03:55:48.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.391+0000 7fb225470640 1 -- 172.21.15.42:0/2345590612 learned_addr learned my addr 172.21.15.42:0/2345590612 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:55:48.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.391+0000 7fb225470640 10 monclient(hunting): get_auth_request con 0x7fb22013c6d0 auth_method 0 2024-09-16T03:55:48.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.391+0000 7fb224c6f640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb22006db90 0x7fb220139de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:36752/0 (socket says 172.21.15.42:36752) 2024-09-16T03:55:48.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.391+0000 7fb225470640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:55:48.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.391+0000 7fb225470640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:55:48.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.391+0000 7fb225470640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:55:48.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.391+0000 7fb224c6f640 10 monclient(hunting): get_auth_request con 0x7fb22006db90 auth_method 0 2024-09-16T03:55:48.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.391+0000 7fb224c6f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:55:48.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.391+0000 7fb224c6f640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:55:48.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.391+0000 7fb224c6f640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:55:48.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.391+0000 7fb225470640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:55:48.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.391+0000 7fb225470640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:55:48.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.391+0000 7fb225470640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:55:48.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.391+0000 7fb224c6f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:55:48.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.391+0000 7fb224c6f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:55:48.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.391+0000 7fb224c6f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:55:48.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.391+0000 7fb224c6f640 10 monclient(hunting): handle_auth_done global_id 824655 payload 824 2024-09-16T03:55:48.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.391+0000 7fb224c6f640 1 -- 172.21.15.42:0/2345590612 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb22013c6d0 msgr2=0x7fb22013a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:55:48.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.391+0000 7fb224c6f640 1 --2- 172.21.15.42:0/2345590612 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb22013c6d0 0x7fb22013a860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:55:48.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.391+0000 7fb224c6f640 1 -- 172.21.15.42:0/2345590612 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb2201340c0 msgr2=0x7fb22013a320 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:55:48.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.391+0000 7fb224c6f640 1 --2- 172.21.15.42:0/2345590612 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb2201340c0 0x7fb22013a320 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:55:48.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.391+0000 7fb224c6f640 10 monclient: _finish_hunting 0 2024-09-16T03:55:48.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.391+0000 7fb224c6f640 1 monclient: found mon.noname-a 2024-09-16T03:55:48.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.391+0000 7fb224c6f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:55:48.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.391+0000 7fb224c6f640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:55:48.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.391+0000 7fb224c6f640 1 -- 172.21.15.42:0/2345590612 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb22013c340 con 0x7fb22006db90 2024-09-16T03:55:48.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.391+0000 7fb225470640 1 --2- 172.21.15.42:0/2345590612 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb22013c6d0 0x7fb22013a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:55:48.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.391+0000 7fb224c6f640 1 --2- 172.21.15.42:0/2345590612 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb22006db90 0x7fb220139de0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fb210007fd0 tx=0x7fb210001e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:55:48.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.391+0000 7fb21d7fa640 10 client.?.objecter ms_handle_connect 0x7fb22006db90 2024-09-16T03:55:48.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.391+0000 7fb21d7fa640 10 client.?.objecter resend_mon_ops 2024-09-16T03:55:48.411 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.391+0000 7fb21d7fa640 1 -- 172.21.15.42:0/2345590612 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb210017070 con 0x7fb22006db90 2024-09-16T03:55:48.411 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.391+0000 7fb21d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:55:48.411 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.391+0000 7fb21d7fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:55:48.411 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.391+0000 7fb21d7fa640 10 monclient: dump: 2024-09-16T03:55:48.411 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:55:48.411 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:55:48.411 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:48.411 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:48.411 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:55:48.411 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:55:48.411 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:55:48.411 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:55:48.411 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:55:48.411 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:55:48.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.391+0000 7fb21d7fa640 10 monclient: _finish_auth 0 2024-09-16T03:55:48.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.391+0000 7fb21d7fa640 10 monclient: _check_auth_tickets 2024-09-16T03:55:48.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.391+0000 7fb21d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:55:48.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.391+0000 7fb21d7fa640 1 -- 172.21.15.42:0/2345590612 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb210005cf0 con 0x7fb22006db90 2024-09-16T03:55:48.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.391+0000 7fb226ed3640 5 monclient: authenticate success, global_id 824655 2024-09-16T03:55:48.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.391+0000 7fb226ed3640 10 monclient: _renew_subs 2024-09-16T03:55:48.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.391+0000 7fb226ed3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:55:48.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.391+0000 7fb226ed3640 1 -- 172.21.15.42:0/2345590612 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb22006f6a0 con 0x7fb22006db90 2024-09-16T03:55:48.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.391+0000 7fb21d7fa640 10 monclient: handle_config config(0 keys) 2024-09-16T03:55:48.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.391+0000 7fb21d7fa640 1 -- 172.21.15.42:0/2345590612 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb210007310 con 0x7fb22006db90 2024-09-16T03:55:48.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.391+0000 7fb21d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:55:48.413 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.391+0000 7fb21d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:55:48.413 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.391+0000 7fb21d7fa640 10 monclient: dump: 2024-09-16T03:55:48.413 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:55:48.413 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:55:48.413 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:48.413 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:48.413 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:55:48.413 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:55:48.413 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:55:48.414 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:55:48.414 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:55:48.414 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:55:48.414 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.391+0000 7fb21effd640 4 set_mon_vals no callback set 2024-09-16T03:55:48.414 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.391+0000 7fb226ed3640 10 client.824655.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:55:48.414 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.391+0000 7fb226ed3640 10 monclient: _renew_subs 2024-09-16T03:55:48.414 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.391+0000 7fb226ed3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:55:48.414 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.391+0000 7fb226ed3640 1 -- 172.21.15.42:0/2345590612 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb22009cd50 con 0x7fb22006db90 2024-09-16T03:55:48.414 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.391+0000 7fb226ed3640 1 librados: init done 2024-09-16T03:55:48.414 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.391+0000 7fb226ed3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:55:48.414 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.391+0000 7fb226ed3640 1 --2- 172.21.15.42:0/2345590612 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb2201cf560 0x7fb2201cf940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:55:48.414 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.391+0000 7fb226ed3640 10 monclient(hunting): start opening mon connection 2024-09-16T03:55:48.414 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.391+0000 7fb226ed3640 1 -- 172.21.15.42:0/2345590612 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fb22009cb80 con 0x7fb2201cf560 2024-09-16T03:55:48.415 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.391+0000 7fb21ffff640 1 --2- 172.21.15.42:0/2345590612 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb2201cf560 0x7fb2201cf940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:55:48.415 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.391+0000 7fb21ffff640 10 monclient: get_auth_request con 0x7fb2201cf560 auth_method 0 2024-09-16T03:55:48.415 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.391+0000 7fb21ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:55:48.415 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.391+0000 7fb21ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:55:48.415 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.391+0000 7fb21ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:55:48.415 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.391+0000 7fb21ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:55:48.415 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.391+0000 7fb21ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:55:48.415 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.391+0000 7fb21ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:55:48.415 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.391+0000 7fb21d7fa640 1 -- 172.21.15.42:0/2345590612 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fb21001a070 con 0x7fb22006db90 2024-09-16T03:55:48.415 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.391+0000 7fb21ffff640 10 monclient(hunting): handle_auth_done global_id 824658 payload 824 2024-09-16T03:55:48.415 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.391+0000 7fb21ffff640 1 -- 172.21.15.42:0/2345590612 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb2200a37f0 con 0x7fb2201cf560 2024-09-16T03:55:48.415 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.391+0000 7fb21d7fa640 1 --2- 172.21.15.42:0/2345590612 >> v2:172.21.15.154:6812/254845342 conn(0x7fb20c0377c0 0x7fb20c039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:55:48.415 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.391+0000 7fb21d7fa640 1 -- 172.21.15.42:0/2345590612 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(420..420 src has 330..420) ==== 6112+0+0 (secure 0 0 0) 0x7fb2100482c0 con 0x7fb22006db90 2024-09-16T03:55:48.415 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.391+0000 7fb21d7fa640 10 client.824655.objecter ms_dispatch 0x7fb220000ba0 osd_map(420..420 src has 330..420) 2024-09-16T03:55:48.415 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.391+0000 7fb21d7fa640 3 client.824655.objecter handle_osd_map got epochs [420,420] > 0 2024-09-16T03:55:48.416 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.391+0000 7fb21d7fa640 3 client.824655.objecter handle_osd_map decoding full epoch 420 2024-09-16T03:55:48.416 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.391+0000 7fb21ffff640 1 --2- 172.21.15.42:0/2345590612 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb2201cf560 0x7fb2201cf940 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fb21400e7e0 tx=0x7fb21400eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:55:48.416 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.391+0000 7fb21d7fa640 20 client.824655.objecter dump_active .. 0 homeless 2024-09-16T03:55:48.416 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.391+0000 7fb21d7fa640 10 client.824655.objecter ms_handle_connect 0x7fb2201cf560 2024-09-16T03:55:48.416 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.391+0000 7fb21d7fa640 10 client.824655.objecter resend_mon_ops 2024-09-16T03:55:48.416 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.391+0000 7fb21d7fa640 1 -- 172.21.15.42:0/2345590612 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb214017070 con 0x7fb2201cf560 2024-09-16T03:55:48.416 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.391+0000 7fb21d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:55:48.416 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.391+0000 7fb21d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:55:48.416 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.391+0000 7fb21d7fa640 10 monclient: dump: 2024-09-16T03:55:48.416 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:55:48.416 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:55:48.416 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:48.416 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:48.417 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:55:48.417 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:55:48.417 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:55:48.417 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:55:48.417 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:55:48.417 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:55:48.417 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.391+0000 7fb225470640 1 --2- 172.21.15.42:0/2345590612 >> v2:172.21.15.154:6812/254845342 conn(0x7fb20c0377c0 0x7fb20c039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:55:48.417 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.391+0000 7fb225470640 10 monclient: get_auth_request con 0x7fb20c0377c0 auth_method 0 2024-09-16T03:55:48.417 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.391+0000 7fb21d7fa640 1 -- 172.21.15.42:0/2345590612 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fb214004020 con 0x7fb2201cf560 2024-09-16T03:55:48.417 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.391+0000 7fb21d7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:55:48.417 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.391+0000 7fb21d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:55:48.417 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.391+0000 7fb21d7fa640 1 -- 172.21.15.42:0/2345590612 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb2201cf560 msgr2=0x7fb2201cf940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:55:48.417 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.391+0000 7fb21d7fa640 1 --2- 172.21.15.42:0/2345590612 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb2201cf560 0x7fb2201cf940 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fb21400e7e0 tx=0x7fb21400eca0 comp rx=0 tx=0).stop 2024-09-16T03:55:48.417 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.391+0000 7fb21d7fa640 1 -- 172.21.15.42:0/2345590612 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb2201cf560 msgr2=0x7fb2201cf940 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:55:48.418 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.391+0000 7fb21d7fa640 1 --2- 172.21.15.42:0/2345590612 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb2201cf560 0x7fb2201cf940 secure :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fb21400e7e0 tx=0x7fb21400eca0 comp rx=0 tx=0).stop 2024-09-16T03:55:48.418 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.395+0000 7fb21d7fa640 10 client.824655.objecter ms_handle_connect 0x7fb20c0377c0 2024-09-16T03:55:48.418 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.395+0000 7fb225470640 1 --2- 172.21.15.42:0/2345590612 >> v2:172.21.15.154:6812/254845342 conn(0x7fb20c0377c0 0x7fb20c039c80 secure :-1 s=READY pgs=3645 cs=0 l=1 rev1=1 crypto rx=0x7fb21801aee0 tx=0x7fb2180a7070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:55:48.419 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.399+0000 7fb226ed3640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:55:48.420 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.399+0000 7fb226ed3640 1 --2- 172.21.15.42:0/2345590612 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb22002c650 0x7fb22002ca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:55:48.420 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.399+0000 7fb226ed3640 10 monclient(hunting): start opening mon connection 2024-09-16T03:55:48.420 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.399+0000 7fb226ed3640 1 -- 172.21.15.42:0/2345590612 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fb2200a4110 con 0x7fb22002c650 2024-09-16T03:55:48.420 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.399+0000 7fb21ffff640 1 --2- 172.21.15.42:0/2345590612 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb22002c650 0x7fb22002ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:55:48.420 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.399+0000 7fb21ffff640 10 monclient: get_auth_request con 0x7fb22002c650 auth_method 0 2024-09-16T03:55:48.420 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.399+0000 7fb21ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:55:48.420 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.399+0000 7fb21ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:55:48.420 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.399+0000 7fb21ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:55:48.420 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.399+0000 7fb21ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:55:48.420 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.399+0000 7fb21ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:55:48.420 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.399+0000 7fb21ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:55:48.420 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.399+0000 7fb21ffff640 10 monclient(hunting): handle_auth_done global_id 824661 payload 824 2024-09-16T03:55:48.420 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.399+0000 7fb21ffff640 1 -- 172.21.15.42:0/2345590612 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fb2201360f0 con 0x7fb22002c650 2024-09-16T03:55:48.421 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.399+0000 7fb21ffff640 1 --2- 172.21.15.42:0/2345590612 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb22002c650 0x7fb22002ca60 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fb21400e7e0 tx=0x7fb214004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:55:48.421 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.399+0000 7fb21d7fa640 10 client.824655.objecter ms_handle_connect 0x7fb22002c650 2024-09-16T03:55:48.421 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.399+0000 7fb21d7fa640 10 client.824655.objecter resend_mon_ops 2024-09-16T03:55:48.421 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.399+0000 7fb21d7fa640 1 -- 172.21.15.42:0/2345590612 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb214017040 con 0x7fb22002c650 2024-09-16T03:55:48.421 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.399+0000 7fb21d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:55:48.421 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.399+0000 7fb21d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:55:48.421 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.399+0000 7fb21d7fa640 10 monclient: dump: 2024-09-16T03:55:48.421 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:55:48.421 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:55:48.422 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:48.422 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:48.422 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:55:48.422 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:55:48.422 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:55:48.422 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:55:48.422 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:55:48.422 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:55:48.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.399+0000 7fb21d7fa640 1 -- 172.21.15.42:0/2345590612 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4871 (secure 0 0 0) 0x7fb214005bb0 con 0x7fb22002c650 2024-09-16T03:55:48.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.399+0000 7fb21d7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:55:48.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.399+0000 7fb21d7fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:55:48.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.399+0000 7fb21d7fa640 1 -- 172.21.15.42:0/2345590612 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb22002c650 msgr2=0x7fb22002ca60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:55:48.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.399+0000 7fb21d7fa640 1 --2- 172.21.15.42:0/2345590612 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb22002c650 0x7fb22002ca60 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fb21400e7e0 tx=0x7fb214004740 comp rx=0 tx=0).stop 2024-09-16T03:55:48.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.399+0000 7fb21d7fa640 1 -- 172.21.15.42:0/2345590612 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb22002c650 msgr2=0x7fb22002ca60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:55:48.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.399+0000 7fb21d7fa640 1 --2- 172.21.15.42:0/2345590612 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb22002c650 0x7fb22002ca60 secure :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fb21400e7e0 tx=0x7fb214004740 comp rx=0 tx=0).stop 2024-09-16T03:55:48.423 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:55:48.423 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:55:48.423 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:55:48.423 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "leader", 2024-09-16T03:55:48.424 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 120, 2024-09-16T03:55:48.424 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 238253, 2024-09-16T03:55:48.424 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:55:48.424 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:55:48.424 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T03:55:48.424 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:55:48.424 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 1, 2024-09-16T03:55:48.424 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:55:48.424 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:55:48.424 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:55:48.424 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:55:48.424 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:55:48.424 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:55:48.425 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:55:48.425 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:55:48.425 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:55:48.425 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:55:48.425 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:55:48.425 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:55:48.425 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:55:48.425 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:55:48.425 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:55:48.425 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:55:48.425 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:55:48.425 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:55:48.425 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:55:48.425 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:55:48.426 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:55:48.426 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:55:48.426 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:55:48.426 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:55:48.426 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:55:48.426 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:55:48.426 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:55:48.426 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:55:48.426 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:55:48.426 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:55:48.426 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:55:48.426 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:55:48.426 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:55:48.426 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:55:48.427 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:55:48.427 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:55:48.427 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:55:48.427 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:55:48.427 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:55:48.427 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:55:48.427 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:55:48.427 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:55:48.427 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:55:48.427 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:55:48.427 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:55:48.427 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:55:48.427 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:55:48.427 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:55:48.428 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:55:48.428 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:55:48.428 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:55:48.428 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:55:48.428 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:55:48.428 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:55:48.428 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:55:48.428 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:55:48.428 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:55:48.428 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:55:48.428 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:55:48.428 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:55:48.428 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:55:48.428 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:55:48.429 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:55:48.429 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:55:48.429 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:55:48.429 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:55:48.429 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:55:48.429 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:55:48.429 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:55:48.429 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:55:48.429 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:55:48.429 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:55:48.429 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:55:48.429 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:55:48.429 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:55:48.429 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:55:48.430 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:55:48.430 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:55:48.430 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:55:48.430 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:55:48.430 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:55:48.430 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:55:48.430 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:55:48.430 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:55:48.430 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:55:48.430 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:55:48.430 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:55:48.430 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:55:48.430 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:55:48.430 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:55:48.431 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:55:48.431 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:55:48.431 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:55:48.431 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:55:48.431 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:55:48.431 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:55:48.431 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:55:48.431 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:55:48.431 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:55:48.431 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:55:48.431 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:55:48.431 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:55:48.431 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:55:48.431 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:55:48.431 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:55:48.432 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:55:48.432 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:55:48.432 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:55:48.432 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:55:48.432 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:55:48.432 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:55:48.432 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:55:48.432 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:55:48.432 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:55:48.432 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:55:48.432 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:55:48.433 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:55:48.433 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:55:48.433 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:55:48.433 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:55:48.433 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:55:48.433 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:55:48.434 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:55:48.434 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:55:48.434 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:55:48.434 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:55:48.434 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:55:48.434 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:55:48.434 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:55:48.434 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:55:48.434 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:55:48.434 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:55:48.434 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:55:48.434 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:55:48.434 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:55:48.434 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:55:48.434 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:55:48.435 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:55:48.435 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:55:48.435 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:55:48.435 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:55:48.435 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T03:55:48.435 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:55:48.435 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:55:48.435 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:55:48.435 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 6 2024-09-16T03:55:48.435 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:55:48.435 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:55:48.435 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:55:48.435 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:55:48.435 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:55:48.436 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:55:48.436 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 8 2024-09-16T03:55:48.436 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:55:48.436 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:55:48.436 INFO:teuthology.orchestra.run.smithi042.stdout: "mgr": [ 2024-09-16T03:55:48.436 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:55:48.436 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:55:48.436 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:55:48.436 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:55:48.436 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:55:48.436 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:55:48.436 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:55:48.436 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:55:48.436 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:55:48.438 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.399+0000 7fb226ed3640 10 librados: watch_flush enter 2024-09-16T03:55:48.438 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.399+0000 7fb226ed3640 10 librados: watch_flush exit 2024-09-16T03:55:48.438 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.399+0000 7fb226ed3640 20 client.824655.objecter shutdown clearing up homeless session... 2024-09-16T03:55:48.438 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.399+0000 7fb226ed3640 10 client.824655.objecter successfully canceled tick 2024-09-16T03:55:48.438 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.399+0000 7fb226ed3640 1 -- 172.21.15.42:0/2345590612 >> v2:172.21.15.154:6812/254845342 conn(0x7fb20c0377c0 msgr2=0x7fb20c039c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:55:48.439 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.399+0000 7fb226ed3640 1 --2- 172.21.15.42:0/2345590612 >> v2:172.21.15.154:6812/254845342 conn(0x7fb20c0377c0 0x7fb20c039c80 secure :-1 s=READY pgs=3645 cs=0 l=1 rev1=1 crypto rx=0x7fb21801aee0 tx=0x7fb2180a7070 comp rx=0 tx=0).stop 2024-09-16T03:55:48.439 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.399+0000 7fb226ed3640 10 monclient: shutdown 2024-09-16T03:55:48.439 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.399+0000 7fb226ed3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:55:48.439 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.399+0000 7fb226ed3640 1 -- 172.21.15.42:0/2345590612 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb22006db90 msgr2=0x7fb220139de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:55:48.439 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.399+0000 7fb226ed3640 1 --2- 172.21.15.42:0/2345590612 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb22006db90 0x7fb220139de0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fb210007fd0 tx=0x7fb210001e80 comp rx=0 tx=0).stop 2024-09-16T03:55:48.439 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.399+0000 7fb225470640 1 -- 172.21.15.42:0/2345590612 reap_dead start 2024-09-16T03:55:48.440 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.399+0000 7fb226ed3640 1 -- 172.21.15.42:0/2345590612 shutdown_connections 2024-09-16T03:55:48.440 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.399+0000 7fb226ed3640 1 --2- 172.21.15.42:0/2345590612 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb22006db90 0x7fb220139de0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:55:48.440 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.399+0000 7fb226ed3640 1 -- 172.21.15.42:0/2345590612 >> 172.21.15.42:0/2345590612 conn(0x7fb2200a9fb0 msgr2=0x7fb2200a8000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:55:48.440 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.399+0000 7fb226ed3640 1 -- 172.21.15.42:0/2345590612 shutdown_connections 2024-09-16T03:55:48.440 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.399+0000 7fb226ed3640 1 -- 172.21.15.42:0/2345590612 wait complete. 2024-09-16T03:55:48.440 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.399+0000 7fb226ed3640 1 librados: shutdown 2024-09-16T03:55:48.440 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T03:55:48.440 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-16T03:55:48.440 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T03:55:48.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.555+0000 7ff4d305b640 10 monclient: get_monmap_and_config 2024-09-16T03:55:48.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.555+0000 7ff4d305b640 10 monclient: build_initial_monmap 2024-09-16T03:55:48.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.555+0000 7ff4d305b640 10 monclient: monmap: 2024-09-16T03:55:48.558 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:55:48.559 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:55:48.559 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:55:48.557199+0000 2024-09-16T03:55:48.559 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:55:48.557199+0000 2024-09-16T03:55:48.559 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:55:48.559 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:55:48.559 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:55:48.559 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:55:48.559 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:55:48.559 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:55:48.559 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:55:48.559 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:55:48.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.555+0000 7ff4d305b640 1 Processor -- start 2024-09-16T03:55:48.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.555+0000 7ff4d305b640 1 -- start start 2024-09-16T03:55:48.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.555+0000 7ff4d305b640 10 monclient: init 2024-09-16T03:55:48.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.555+0000 7ff4d305b640 10 monclient: _reopen_session rank -1 2024-09-16T03:55:48.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.555+0000 7ff4d305b640 10 monclient: _add_conns ranks=[3,0,1,2,4] 2024-09-16T03:55:48.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.555+0000 7ff4d305b640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff4cc0a1330 0x7ff4cc0a1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:55:48.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.555+0000 7ff4d305b640 10 monclient(hunting): picked mon.noname-a con 0x7ff4cc0a1330 addr v2:172.21.15.42:3300/0 2024-09-16T03:55:48.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.555+0000 7ff4d305b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff4cc06dac0 addr v1:172.21.15.42:6789/0 2024-09-16T03:55:48.562 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.555+0000 7ff4d305b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff4cc0a1c50 addr v1:172.21.15.154:6789/0 2024-09-16T03:55:48.562 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.555+0000 7ff4d305b640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff4cc135f20 con 0x7ff4cc06dac0 2024-09-16T03:55:48.562 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.555+0000 7ff4d305b640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff4cc09c090 con 0x7ff4cc0a1c50 2024-09-16T03:55:48.562 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.555+0000 7ff4d305b640 10 monclient(hunting): start opening mon connection 2024-09-16T03:55:48.562 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.555+0000 7ff4d305b640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7ff4cc09f610 con 0x7ff4cc0a1330 2024-09-16T03:55:48.562 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.555+0000 7ff4d305b640 10 monclient(hunting): _renew_subs 2024-09-16T03:55:48.562 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.555+0000 7ff4d305b640 10 monclient(hunting): authenticate will time out at 1795.526553s 2024-09-16T03:55:48.562 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.555+0000 7ff4d0df7640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff4cc0a1330 0x7ff4cc0a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:55:48.562 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.555+0000 7ff4cbfff640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7ff4cc06dac0 0x7ff4cc135100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:34652/0 (socket says 172.21.15.42:34652) 2024-09-16T03:55:48.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.555+0000 7ff4cbfff640 1 -- 172.21.15.42:0/3946680218 learned_addr learned my addr 172.21.15.42:0/3946680218 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:55:48.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.555+0000 7ff4d0df7640 10 monclient(hunting): get_auth_request con 0x7ff4cc0a1330 auth_method 0 2024-09-16T03:55:48.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.555+0000 7ff4d0df7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:55:48.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.555+0000 7ff4d0df7640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:55:48.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.555+0000 7ff4d0df7640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:55:48.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.555+0000 7ff4d15f8640 1 -- 172.21.15.42:0/3946680218 >> v1:172.21.15.154:6789/0 conn(0x7ff4cc0a1c50 legacy=0x7ff4cc135810 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.154:6789/0 2024-09-16T03:55:48.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.555+0000 7ff4d0df7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:55:48.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.555+0000 7ff4d0df7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:55:48.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.555+0000 7ff4d0df7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:55:48.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.555+0000 7ff4d0df7640 10 monclient(hunting): handle_auth_done global_id 824664 payload 274 2024-09-16T03:55:48.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.555+0000 7ff4d0df7640 1 -- 172.21.15.42:0/3946680218 >> v1:172.21.15.154:6789/0 conn(0x7ff4cc0a1c50 legacy=0x7ff4cc135810 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:55:48.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.555+0000 7ff4d0df7640 1 -- 172.21.15.42:0/3946680218 >> v1:172.21.15.42:6789/0 conn(0x7ff4cc06dac0 legacy=0x7ff4cc135100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:55:48.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.555+0000 7ff4d0df7640 10 monclient: _finish_hunting 0 2024-09-16T03:55:48.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.555+0000 7ff4d0df7640 1 monclient: found mon.noname-a 2024-09-16T03:55:48.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.555+0000 7ff4d0df7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:55:48.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.555+0000 7ff4d0df7640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:55:48.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.555+0000 7ff4d0df7640 1 -- 172.21.15.42:0/3946680218 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4cc09f8f0 con 0x7ff4cc0a1330 2024-09-16T03:55:48.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.555+0000 7ff4cb7fe640 1 -- 172.21.15.42:0/3946680218 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1892408122 0 0) 0x7ff4cc135f20 con 0x7ff4cc06dac0 2024-09-16T03:55:48.565 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.555+0000 7ff4cb7fe640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T03:55:48.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.559+0000 7ff4d0df7640 1 --2- 172.21.15.42:0/3946680218 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff4cc0a1330 0x7ff4cc0a1710 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7ff4c4009d90 tx=0x7ff4c401c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3cb34b9ccb85f3f2 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:55:48.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.559+0000 7ff4cb7fe640 1 -- 172.21.15.42:0/3946680218 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4c40b2070 con 0x7ff4cc0a1330 2024-09-16T03:55:48.572 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.559+0000 7ff4cb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:55:48.572 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.559+0000 7ff4cb7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:55:48.572 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.559+0000 7ff4cb7fe640 10 monclient: dump: 2024-09-16T03:55:48.572 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:55:48.572 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:55:48.572 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:48.572 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:48.572 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:55:48.572 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:55:48.572 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:55:48.573 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:55:48.573 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:55:48.573 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:55:48.573 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.559+0000 7ff4cb7fe640 10 monclient: _finish_auth 0 2024-09-16T03:55:48.573 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.559+0000 7ff4cb7fe640 10 monclient: _check_auth_tickets 2024-09-16T03:55:48.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.559+0000 7ff4cb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:55:48.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.559+0000 7ff4cb7fe640 1 -- 172.21.15.42:0/3946680218 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4c40a58a0 con 0x7ff4cc0a1330 2024-09-16T03:55:48.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.559+0000 7ff4d305b640 5 monclient: authenticate success, global_id 824664 2024-09-16T03:55:48.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.559+0000 7ff4d305b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:55:48.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.559+0000 7ff4cb7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:55:48.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.559+0000 7ff4d305b640 10 monclient: get_monmap_and_config success 2024-09-16T03:55:48.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.559+0000 7ff4cb7fe640 1 -- 172.21.15.42:0/3946680218 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4c40a5c70 con 0x7ff4cc0a1330 2024-09-16T03:55:48.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.559+0000 7ff4d305b640 4 set_mon_vals no callback set 2024-09-16T03:55:48.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.559+0000 7ff4cb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:55:48.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.559+0000 7ff4cb7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:55:48.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.559+0000 7ff4cb7fe640 10 monclient: dump: 2024-09-16T03:55:48.576 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:55:48.576 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:55:48.576 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:48.576 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:48.576 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:55:48.577 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:55:48.577 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:55:48.577 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:55:48.577 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:55:48.577 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:55:48.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.559+0000 7ff4d305b640 10 monclient: shutdown 2024-09-16T03:55:48.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.559+0000 7ff4d305b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:55:48.578 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.559+0000 7ff4d305b640 1 -- 172.21.15.42:0/3946680218 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff4cc0a1330 msgr2=0x7ff4cc0a1710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:55:48.578 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.559+0000 7ff4d305b640 1 --2- 172.21.15.42:0/3946680218 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff4cc0a1330 0x7ff4cc0a1710 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7ff4c4009d90 tx=0x7ff4c401c850 comp rx=0 tx=0).stop 2024-09-16T03:55:48.578 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.559+0000 7ff4d305b640 1 -- 172.21.15.42:0/3946680218 shutdown_connections 2024-09-16T03:55:48.578 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.559+0000 7ff4d305b640 1 --2- 172.21.15.42:0/3946680218 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff4cc0a1330 0x7ff4cc0a1710 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:55:48.578 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.559+0000 7ff4d305b640 1 -- 172.21.15.42:0/3946680218 >> 172.21.15.42:0/3946680218 conn(0x7ff4cc0a2420 msgr2=0x7ff4cc0a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:55:48.578 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.559+0000 7ff4d305b640 1 -- 172.21.15.42:0/3946680218 shutdown_connections 2024-09-16T03:55:48.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.559+0000 7ff4d305b640 1 -- 172.21.15.42:0/3946680218 wait complete. 2024-09-16T03:55:48.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.559+0000 7ff4d305b640 10 monclient: build_initial_monmap 2024-09-16T03:55:48.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.559+0000 7ff4d305b640 10 monclient: monmap: 2024-09-16T03:55:48.579 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:55:48.579 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:55:48.579 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:55:48.579 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:55:48.580 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:55:48.580 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:55:48.580 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:55:48.580 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:55:48.580 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:55:48.580 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:55:48.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.559+0000 7ff4d305b640 1 librados: starting msgr at 2024-09-16T03:55:48.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.559+0000 7ff4d305b640 1 librados: starting objecter 2024-09-16T03:55:48.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.559+0000 7ff4d305b640 1 Processor -- start 2024-09-16T03:55:48.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.559+0000 7ff4d305b640 1 -- start start 2024-09-16T03:55:48.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.559+0000 7ff4d305b640 1 librados: setting wanted keys 2024-09-16T03:55:48.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.559+0000 7ff4d305b640 1 librados: calling monclient init 2024-09-16T03:55:48.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.559+0000 7ff4d305b640 10 monclient: init 2024-09-16T03:55:48.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.559+0000 7ff4d305b640 10 monclient: _reopen_session rank -1 2024-09-16T03:55:48.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.559+0000 7ff4d305b640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T03:55:48.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.559+0000 7ff4d305b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff4cc0a1330 0x7ff4cc13cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:55:48.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.559+0000 7ff4d305b640 10 monclient(hunting): picked mon.noname-b con 0x7ff4cc0a1330 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:55:48.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.559+0000 7ff4d305b640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff4cc0a1c50 0x7ff4cc1351f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:55:48.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.559+0000 7ff4d305b640 10 monclient(hunting): picked mon.noname-c con 0x7ff4cc0a1c50 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:55:48.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.559+0000 7ff4d305b640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff4cc09dd90 0x7ff4cc1cecb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:55:48.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.559+0000 7ff4d305b640 10 monclient(hunting): picked mon.noname-a con 0x7ff4cc09dd90 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:55:48.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.559+0000 7ff4d305b640 10 monclient(hunting): start opening mon connection 2024-09-16T03:55:48.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.559+0000 7ff4d305b640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7ff4cc09eef0 con 0x7ff4cc09dd90 2024-09-16T03:55:48.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.559+0000 7ff4d305b640 10 monclient(hunting): start opening mon connection 2024-09-16T03:55:48.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.559+0000 7ff4d305b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff4cc09ed70 con 0x7ff4cc0a1330 2024-09-16T03:55:48.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.559+0000 7ff4d305b640 10 monclient(hunting): start opening mon connection 2024-09-16T03:55:48.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.559+0000 7ff4d305b640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7ff4cc140b90 con 0x7ff4cc0a1c50 2024-09-16T03:55:48.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.559+0000 7ff4d305b640 10 monclient(hunting): _renew_subs 2024-09-16T03:55:48.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.559+0000 7ff4d305b640 10 monclient(hunting): authenticate will time out at 1795.531535s 2024-09-16T03:55:48.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.559+0000 7ff4d15f8640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff4cc09dd90 0x7ff4cc1cecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:55:48.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.559+0000 7ff4cbfff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff4cc0a1c50 0x7ff4cc1351f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:55:48.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.559+0000 7ff4d0df7640 1 -- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff4cc0a1330 msgr2=0x7ff4cc13cda0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T03:55:48.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.559+0000 7ff4d15f8640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff4cc09dd90 0x7ff4cc1cecb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:36806/0 (socket says 172.21.15.42:36806) 2024-09-16T03:55:48.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.559+0000 7ff4d15f8640 1 -- 172.21.15.42:0/3603072546 learned_addr learned my addr 172.21.15.42:0/3603072546 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:55:48.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.559+0000 7ff4cbfff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff4cc0a1c50 0x7ff4cc1351f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:32836/0 (socket says 172.21.15.42:32836) 2024-09-16T03:55:48.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.559+0000 7ff4d15f8640 10 monclient(hunting): get_auth_request con 0x7ff4cc09dd90 auth_method 0 2024-09-16T03:55:48.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.559+0000 7ff4d15f8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:55:48.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.559+0000 7ff4d15f8640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:55:48.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.559+0000 7ff4d15f8640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:55:48.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.559+0000 7ff4d0df7640 1 --2- 172.21.15.42:0/3603072546 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff4cc0a1330 0x7ff4cc13cda0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:55:48.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.559+0000 7ff4cbfff640 10 monclient(hunting): get_auth_request con 0x7ff4cc0a1c50 auth_method 0 2024-09-16T03:55:48.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.559+0000 7ff4cbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:55:48.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.559+0000 7ff4cbfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:55:48.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.559+0000 7ff4cbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:55:48.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.559+0000 7ff4cbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:55:48.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.559+0000 7ff4cbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:55:48.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.559+0000 7ff4cbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:55:48.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.559+0000 7ff4d15f8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:55:48.588 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.559+0000 7ff4d15f8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:55:48.588 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.559+0000 7ff4d15f8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:55:48.588 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.563+0000 7ff4cbfff640 10 monclient(hunting): handle_auth_done global_id 854261 payload 824 2024-09-16T03:55:48.588 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.563+0000 7ff4cbfff640 1 -- 172.21.15.42:0/3603072546 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff4cc0a1330 msgr2=0x7ff4cc13cda0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:55:48.588 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.563+0000 7ff4cbfff640 1 --2- 172.21.15.42:0/3603072546 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff4cc0a1330 0x7ff4cc13cda0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:55:48.588 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.563+0000 7ff4cbfff640 1 -- 172.21.15.42:0/3603072546 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff4cc09dd90 msgr2=0x7ff4cc1cecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:55:48.589 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.563+0000 7ff4cbfff640 1 --2- 172.21.15.42:0/3603072546 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff4cc09dd90 0x7ff4cc1cecb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:55:48.589 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.563+0000 7ff4cbfff640 10 monclient: _finish_hunting 0 2024-09-16T03:55:48.589 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.563+0000 7ff4cbfff640 1 monclient: found mon.noname-c 2024-09-16T03:55:48.589 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.563+0000 7ff4cbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:55:48.589 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.563+0000 7ff4cbfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:55:48.589 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.563+0000 7ff4cbfff640 1 -- 172.21.15.42:0/3603072546 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4cc09d070 con 0x7ff4cc0a1c50 2024-09-16T03:55:48.589 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.563+0000 7ff4d15f8640 1 --2- 172.21.15.42:0/3603072546 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff4cc09dd90 0x7ff4cc1cecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:55:48.590 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.563+0000 7ff4cbfff640 1 --2- 172.21.15.42:0/3603072546 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff4cc0a1c50 0x7ff4cc1351f0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7ff4c000c410 tx=0x7ff4c000c8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:55:48.590 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.563+0000 7ff4c97fa640 10 client.?.objecter ms_handle_connect 0x7ff4cc0a1c50 2024-09-16T03:55:48.590 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.563+0000 7ff4c97fa640 10 client.?.objecter resend_mon_ops 2024-09-16T03:55:48.590 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.563+0000 7ff4c97fa640 1 -- 172.21.15.42:0/3603072546 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4c00074e0 con 0x7ff4cc0a1c50 2024-09-16T03:55:48.590 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.563+0000 7ff4c97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:55:48.590 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.563+0000 7ff4c97fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:55:48.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.563+0000 7ff4c97fa640 10 monclient: dump: 2024-09-16T03:55:48.591 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:55:48.591 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:55:48.591 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:48.591 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:48.591 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:55:48.592 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:55:48.592 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:55:48.592 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:55:48.592 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:55:48.592 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:55:48.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.563+0000 7ff4c97fa640 10 monclient: _finish_auth 0 2024-09-16T03:55:48.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.563+0000 7ff4c97fa640 10 monclient: _check_auth_tickets 2024-09-16T03:55:48.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.563+0000 7ff4c97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:55:48.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.563+0000 7ff4d305b640 5 monclient: authenticate success, global_id 854261 2024-09-16T03:55:48.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.563+0000 7ff4d305b640 10 monclient: _renew_subs 2024-09-16T03:55:48.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.563+0000 7ff4c97fa640 1 -- 172.21.15.42:0/3603072546 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4c00047f0 con 0x7ff4cc0a1c50 2024-09-16T03:55:48.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.563+0000 7ff4d305b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:55:48.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.563+0000 7ff4d305b640 1 -- 172.21.15.42:0/3603072546 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4cc06e8f0 con 0x7ff4cc0a1c50 2024-09-16T03:55:48.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.563+0000 7ff4c97fa640 10 monclient: handle_config config(0 keys) 2024-09-16T03:55:48.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.563+0000 7ff4c97fa640 1 -- 172.21.15.42:0/3603072546 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4c0007680 con 0x7ff4cc0a1c50 2024-09-16T03:55:48.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.563+0000 7ff4c97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:55:48.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.563+0000 7ff4c97fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:55:48.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.563+0000 7ff4c97fa640 10 monclient: dump: 2024-09-16T03:55:48.594 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:55:48.594 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:55:48.595 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:48.595 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:48.595 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:55:48.595 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:55:48.595 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:55:48.595 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:55:48.596 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:55:48.596 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:55:48.596 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.563+0000 7ff4ca7fc640 4 set_mon_vals no callback set 2024-09-16T03:55:48.596 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.563+0000 7ff4d305b640 10 client.854261.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:55:48.596 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.563+0000 7ff4d305b640 10 monclient: _renew_subs 2024-09-16T03:55:48.596 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.563+0000 7ff4d305b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:55:48.596 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.563+0000 7ff4d305b640 1 -- 172.21.15.42:0/3603072546 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4cc1205d0 con 0x7ff4cc0a1c50 2024-09-16T03:55:48.597 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.563+0000 7ff4d305b640 1 librados: init done 2024-09-16T03:55:48.597 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.563+0000 7ff4d305b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:55:48.597 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.563+0000 7ff4d305b640 1 --2- 172.21.15.42:0/3603072546 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff4cc06dac0 0x7ff4cc02c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:55:48.597 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.563+0000 7ff4d305b640 10 monclient(hunting): start opening mon connection 2024-09-16T03:55:48.597 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.563+0000 7ff4d305b640 1 -- 172.21.15.42:0/3603072546 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7ff4cc1207d0 con 0x7ff4cc06dac0 2024-09-16T03:55:48.597 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.563+0000 7ff4c97fa640 1 -- 172.21.15.42:0/3603072546 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7ff4c0011070 con 0x7ff4cc0a1c50 2024-09-16T03:55:48.598 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.563+0000 7ff4d0df7640 1 --2- 172.21.15.42:0/3603072546 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff4cc06dac0 0x7ff4cc02c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:55:48.598 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.563+0000 7ff4d0df7640 10 monclient: get_auth_request con 0x7ff4cc06dac0 auth_method 0 2024-09-16T03:55:48.598 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.563+0000 7ff4d0df7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:55:48.598 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.563+0000 7ff4d0df7640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:55:48.598 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.563+0000 7ff4d0df7640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:55:48.598 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.563+0000 7ff4d0df7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:55:48.599 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.563+0000 7ff4d0df7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:55:48.599 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.563+0000 7ff4d0df7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:55:48.599 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.563+0000 7ff4c97fa640 1 --2- 172.21.15.42:0/3603072546 >> v2:172.21.15.154:6812/254845342 conn(0x7ff4b80377c0 0x7ff4b8039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:55:48.599 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.563+0000 7ff4c97fa640 1 -- 172.21.15.42:0/3603072546 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(421..421 src has 330..421) ==== 6128+0+0 (secure 0 0 0) 0x7ff4c0048c50 con 0x7ff4cc0a1c50 2024-09-16T03:55:48.599 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.563+0000 7ff4c97fa640 10 client.854261.objecter ms_dispatch 0x7ff4cc000ba0 osd_map(421..421 src has 330..421) 2024-09-16T03:55:48.599 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.563+0000 7ff4c97fa640 3 client.854261.objecter handle_osd_map got epochs [421,421] > 0 2024-09-16T03:55:48.600 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.563+0000 7ff4c97fa640 3 client.854261.objecter handle_osd_map decoding full epoch 421 2024-09-16T03:55:48.600 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.563+0000 7ff4c97fa640 20 client.854261.objecter dump_active .. 0 homeless 2024-09-16T03:55:48.600 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.563+0000 7ff4d0df7640 10 monclient(hunting): handle_auth_done global_id 854264 payload 824 2024-09-16T03:55:48.600 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.563+0000 7ff4d0df7640 1 -- 172.21.15.42:0/3603072546 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff4cc0a1710 con 0x7ff4cc06dac0 2024-09-16T03:55:48.600 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.563+0000 7ff4c97fa640 10 client.854261.objecter ms_handle_connect 0x7ff4cc06dac0 2024-09-16T03:55:48.600 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.563+0000 7ff4c97fa640 10 client.854261.objecter resend_mon_ops 2024-09-16T03:55:48.601 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.563+0000 7ff4d0df7640 1 --2- 172.21.15.42:0/3603072546 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff4cc06dac0 0x7ff4cc02c650 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7ff4cc06da10 tx=0x7ff4c40195f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:55:48.601 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.563+0000 7ff4d15f8640 1 --2- 172.21.15.42:0/3603072546 >> v2:172.21.15.154:6812/254845342 conn(0x7ff4b80377c0 0x7ff4b8039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:55:48.601 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.563+0000 7ff4c97fa640 1 -- 172.21.15.42:0/3603072546 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4c40bd070 con 0x7ff4cc06dac0 2024-09-16T03:55:48.601 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.563+0000 7ff4c97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:55:48.601 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.563+0000 7ff4c97fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:55:48.601 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.563+0000 7ff4c97fa640 10 monclient: dump: 2024-09-16T03:55:48.602 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:55:48.602 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:55:48.603 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:48.603 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:48.603 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:55:48.603 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:55:48.603 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:55:48.603 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:55:48.604 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:55:48.604 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:55:48.604 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.563+0000 7ff4d15f8640 10 monclient: get_auth_request con 0x7ff4b80377c0 auth_method 0 2024-09-16T03:55:48.604 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.563+0000 7ff4c97fa640 1 -- 172.21.15.42:0/3603072546 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7ff4c40a5dc0 con 0x7ff4cc06dac0 2024-09-16T03:55:48.604 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.563+0000 7ff4c97fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:55:48.604 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.563+0000 7ff4c97fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:55:48.604 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.563+0000 7ff4c97fa640 1 -- 172.21.15.42:0/3603072546 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff4cc06dac0 msgr2=0x7ff4cc02c650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:55:48.604 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.563+0000 7ff4c97fa640 1 --2- 172.21.15.42:0/3603072546 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff4cc06dac0 0x7ff4cc02c650 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7ff4cc06da10 tx=0x7ff4c40195f0 comp rx=0 tx=0).stop 2024-09-16T03:55:48.604 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.563+0000 7ff4c97fa640 1 -- 172.21.15.42:0/3603072546 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff4cc06dac0 msgr2=0x7ff4cc02c650 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:55:48.604 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.563+0000 7ff4c97fa640 1 --2- 172.21.15.42:0/3603072546 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff4cc06dac0 0x7ff4cc02c650 secure :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7ff4cc06da10 tx=0x7ff4c40195f0 comp rx=0 tx=0).stop 2024-09-16T03:55:48.604 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.563+0000 7ff4d15f8640 1 --2- 172.21.15.42:0/3603072546 >> v2:172.21.15.154:6812/254845342 conn(0x7ff4b80377c0 0x7ff4b8039c80 secure :-1 s=READY pgs=3646 cs=0 l=1 rev1=1 crypto rx=0x7ff4bc003fa0 tx=0x7ff4bc0090e0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:55:48.604 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.563+0000 7ff4c97fa640 10 client.854261.objecter ms_handle_connect 0x7ff4b80377c0 2024-09-16T03:55:48.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.571+0000 7ff4d305b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:55:48.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.571+0000 7ff4d305b640 1 --2- 172.21.15.42:0/3603072546 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff4cc135730 0x7ff4cc02cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:55:48.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.571+0000 7ff4d305b640 10 monclient(hunting): start opening mon connection 2024-09-16T03:55:48.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.571+0000 7ff4d305b640 1 -- 172.21.15.42:0/3603072546 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7ff4cc139a40 con 0x7ff4cc135730 2024-09-16T03:55:48.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.571+0000 7ff4d0df7640 1 --2- 172.21.15.42:0/3603072546 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff4cc135730 0x7ff4cc02cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:55:48.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.571+0000 7ff4d0df7640 10 monclient: get_auth_request con 0x7ff4cc135730 auth_method 0 2024-09-16T03:55:48.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.571+0000 7ff4d0df7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:55:48.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.571+0000 7ff4d0df7640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:55:48.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.571+0000 7ff4d0df7640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:55:48.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.571+0000 7ff4d0df7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:55:48.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.571+0000 7ff4d0df7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:55:48.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.571+0000 7ff4d0df7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:55:48.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.571+0000 7ff4d0df7640 10 monclient(hunting): handle_auth_done global_id 854267 payload 824 2024-09-16T03:55:48.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.571+0000 7ff4d0df7640 1 -- 172.21.15.42:0/3603072546 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ff4cc069040 con 0x7ff4cc135730 2024-09-16T03:55:48.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.571+0000 7ff4c97fa640 10 client.854261.objecter ms_handle_connect 0x7ff4cc135730 2024-09-16T03:55:48.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.571+0000 7ff4c97fa640 10 client.854261.objecter resend_mon_ops 2024-09-16T03:55:48.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.571+0000 7ff4d0df7640 1 --2- 172.21.15.42:0/3603072546 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff4cc135730 0x7ff4cc02cb90 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ff4c401a480 tx=0x7ff4c40a7000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:55:48.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.571+0000 7ff4c97fa640 1 -- 172.21.15.42:0/3603072546 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4c40b2040 con 0x7ff4cc135730 2024-09-16T03:55:48.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.571+0000 7ff4c97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:55:48.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.571+0000 7ff4c97fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:55:48.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.571+0000 7ff4c97fa640 10 monclient: dump: 2024-09-16T03:55:48.608 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:55:48.608 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:55:48.608 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:48.608 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:55:48.609 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:55:48.609 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:55:48.609 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:55:48.609 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:55:48.609 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:55:48.609 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:55:48.609 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.571+0000 7ff4c97fa640 1 -- 172.21.15.42:0/3603072546 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4534 (secure 0 0 0) 0x7ff4c40adc40 con 0x7ff4cc135730 2024-09-16T03:55:48.609 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.571+0000 7ff4c97fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:55:48.609 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.571+0000 7ff4c97fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:55:48.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.571+0000 7ff4c97fa640 1 -- 172.21.15.42:0/3603072546 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff4cc135730 msgr2=0x7ff4cc02cb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:55:48.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.571+0000 7ff4c97fa640 1 --2- 172.21.15.42:0/3603072546 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff4cc135730 0x7ff4cc02cb90 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ff4c401a480 tx=0x7ff4c40a7000 comp rx=0 tx=0).stop 2024-09-16T03:55:48.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.571+0000 7ff4c97fa640 1 -- 172.21.15.42:0/3603072546 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff4cc135730 msgr2=0x7ff4cc02cb90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:55:48.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.571+0000 7ff4c97fa640 1 --2- 172.21.15.42:0/3603072546 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff4cc135730 0x7ff4cc02cb90 secure :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ff4c401a480 tx=0x7ff4c40a7000 comp rx=0 tx=0).stop 2024-09-16T03:55:48.610 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:55:48.610 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:55:48.610 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:55:48.611 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T03:55:48.611 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 120, 2024-09-16T03:55:48.611 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 82007, 2024-09-16T03:55:48.611 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:55:48.611 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:55:48.611 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T03:55:48.611 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:55:48.611 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 2, 2024-09-16T03:55:48.611 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:55:48.611 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:55:48.611 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:55:48.611 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:55:48.611 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:55:48.611 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:55:48.612 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:55:48.612 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:55:48.612 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:55:48.612 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:55:48.612 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:55:48.612 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:55:48.612 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:55:48.613 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:55:48.613 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:55:48.613 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:55:48.613 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:55:48.613 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:55:48.613 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:55:48.613 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:55:48.613 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:55:48.613 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:55:48.613 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:55:48.613 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:55:48.613 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:55:48.613 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:55:48.613 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:55:48.614 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:55:48.614 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:55:48.614 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:55:48.614 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:55:48.614 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:55:48.614 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:55:48.614 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:55:48.614 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:55:48.614 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:55:48.614 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:55:48.614 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:55:48.614 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:55:48.614 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:55:48.614 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:55:48.615 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:55:48.615 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:55:48.615 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:55:48.615 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:55:48.615 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:55:48.615 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:55:48.615 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:55:48.615 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:55:48.615 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:55:48.615 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:55:48.615 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:55:48.615 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:55:48.616 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:55:48.616 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:55:48.616 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:55:48.616 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:55:48.616 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:55:48.616 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:55:48.616 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:55:48.616 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:55:48.616 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:55:48.616 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:55:48.616 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:55:48.616 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:55:48.617 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:55:48.617 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:55:48.617 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:55:48.617 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:55:48.617 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:55:48.617 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:55:48.617 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:55:48.617 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:55:48.617 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:55:48.617 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:55:48.617 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:55:48.618 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:55:48.618 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:55:48.618 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:55:48.618 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:55:48.618 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:55:48.618 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:55:48.618 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:55:48.618 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:55:48.618 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:55:48.618 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:55:48.618 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:55:48.618 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:55:48.619 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:55:48.619 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:55:48.619 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:55:48.619 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:55:48.619 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:55:48.619 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:55:48.619 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:55:48.619 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:55:48.619 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:55:48.619 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:55:48.619 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:55:48.619 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:55:48.619 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:55:48.619 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:55:48.620 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:55:48.620 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:55:48.620 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:55:48.620 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:55:48.620 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:55:48.620 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:55:48.620 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:55:48.620 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:55:48.620 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:55:48.620 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:55:48.620 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:55:48.620 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:55:48.620 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:55:48.620 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:55:48.620 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:55:48.621 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:55:48.621 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:55:48.621 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:55:48.621 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:55:48.621 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:55:48.621 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:55:48.621 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:55:48.621 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:55:48.621 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:55:48.621 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:55:48.621 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:55:48.621 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:55:48.621 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:55:48.621 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:55:48.622 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:55:48.622 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:55:48.622 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:55:48.622 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:55:48.622 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:55:48.622 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:55:48.622 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:55:48.622 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:55:48.622 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:55:48.622 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:55:48.622 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:55:48.622 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:55:48.622 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:55:48.622 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:55:48.623 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 2 2024-09-16T03:55:48.623 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:55:48.623 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:55:48.623 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:55:48.623 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:55:48.623 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:55:48.626 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.571+0000 7ff4d305b640 10 librados: watch_flush enter 2024-09-16T03:55:48.626 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.571+0000 7ff4d305b640 10 librados: watch_flush exit 2024-09-16T03:55:48.626 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.571+0000 7ff4d305b640 20 client.854261.objecter shutdown clearing up homeless session... 2024-09-16T03:55:48.626 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.571+0000 7ff4d305b640 10 client.854261.objecter successfully canceled tick 2024-09-16T03:55:48.626 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.571+0000 7ff4d305b640 1 -- 172.21.15.42:0/3603072546 >> v2:172.21.15.154:6812/254845342 conn(0x7ff4b80377c0 msgr2=0x7ff4b8039c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:55:48.626 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.571+0000 7ff4d305b640 1 --2- 172.21.15.42:0/3603072546 >> v2:172.21.15.154:6812/254845342 conn(0x7ff4b80377c0 0x7ff4b8039c80 secure :-1 s=READY pgs=3646 cs=0 l=1 rev1=1 crypto rx=0x7ff4bc003fa0 tx=0x7ff4bc0090e0 comp rx=0 tx=0).stop 2024-09-16T03:55:48.627 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.571+0000 7ff4d305b640 10 monclient: shutdown 2024-09-16T03:55:48.627 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.571+0000 7ff4d305b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:55:48.627 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.571+0000 7ff4d305b640 1 -- 172.21.15.42:0/3603072546 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff4cc0a1c50 msgr2=0x7ff4cc1351f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:55:48.627 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.571+0000 7ff4d305b640 1 --2- 172.21.15.42:0/3603072546 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff4cc0a1c50 0x7ff4cc1351f0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7ff4c000c410 tx=0x7ff4c000c8d0 comp rx=0 tx=0).stop 2024-09-16T03:55:48.627 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.571+0000 7ff4d15f8640 1 -- 172.21.15.42:0/3603072546 reap_dead start 2024-09-16T03:55:48.627 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.571+0000 7ff4d15f8640 1 -- 172.21.15.42:0/3603072546 reap_dead start 2024-09-16T03:55:48.628 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.571+0000 7ff4d305b640 1 -- 172.21.15.42:0/3603072546 shutdown_connections 2024-09-16T03:55:48.628 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.571+0000 7ff4d305b640 1 -- 172.21.15.42:0/3603072546 >> 172.21.15.42:0/3603072546 conn(0x7ff4cc0a2420 msgr2=0x7ff4cc0a4ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:55:48.628 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.571+0000 7ff4d305b640 1 -- 172.21.15.42:0/3603072546 shutdown_connections 2024-09-16T03:55:48.628 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.571+0000 7ff4d305b640 1 -- 172.21.15.42:0/3603072546 wait complete. 2024-09-16T03:55:48.628 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:55:48.571+0000 7ff4d305b640 1 librados: shutdown 2024-09-16T03:55:48.633 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T03:55:48.633 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-16T03:56:08.633 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-16T03:56:08.634 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-09-16T03:56:08.634 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-16T03:56:08.634 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-16T03:56:08.637 INFO:tasks.ceph.mon.b:Started 2024-09-16T03:56:08.637 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T03:56:08.638 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:56:08.767 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.763+0000 7f31f0366640 10 monclient: get_monmap_and_config 2024-09-16T03:56:08.767 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.763+0000 7f31f0366640 10 monclient: build_initial_monmap 2024-09-16T03:56:08.768 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.763+0000 7f31f0366640 10 monclient: monmap: 2024-09-16T03:56:08.768 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:56:08.768 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:56:08.768 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:56:08.766381+0000 2024-09-16T03:56:08.768 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:56:08.766381+0000 2024-09-16T03:56:08.768 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:56:08.768 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:56:08.768 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:56:08.768 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:56:08.768 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:56:08.768 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:56:08.768 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:56:08.769 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:56:08.773 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.763+0000 7f31f0366640 1 Processor -- start 2024-09-16T03:56:08.773 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.763+0000 7f31f0366640 1 -- start start 2024-09-16T03:56:08.773 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.763+0000 7f31f0366640 10 monclient: init 2024-09-16T03:56:08.773 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.763+0000 7f31f0366640 10 monclient: _reopen_session rank -1 2024-09-16T03:56:08.773 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.763+0000 7f31f0366640 10 monclient: _add_conns ranks=[0,4,1,2,3] 2024-09-16T03:56:08.773 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.763+0000 7f31f0366640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f31e80a2340 addr v1:172.21.15.42:6789/0 2024-09-16T03:56:08.773 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.763+0000 7f31f0366640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f31e80a2720 0x7f31e80a2b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:56:08.773 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.763+0000 7f31f0366640 10 monclient(hunting): picked mon.noname-c con 0x7f31e80a2720 addr v2:172.21.15.154:3300/0 2024-09-16T03:56:08.773 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.763+0000 7f31f0366640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f31e813a420 addr v1:172.21.15.154:6789/0 2024-09-16T03:56:08.773 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.763+0000 7f31f0366640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f31e8070b90 con 0x7f31e80a2340 2024-09-16T03:56:08.773 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.763+0000 7f31f0366640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f31e80713e0 con 0x7f31e813a420 2024-09-16T03:56:08.774 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.763+0000 7f31f0366640 10 monclient(hunting): start opening mon connection 2024-09-16T03:56:08.774 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.763+0000 7f31f0366640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f31e806cae0 con 0x7f31e80a2720 2024-09-16T03:56:08.774 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.763+0000 7f31f0366640 10 monclient(hunting): _renew_subs 2024-09-16T03:56:08.774 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.763+0000 7f31f0366640 10 monclient(hunting): authenticate will time out at 1815.735873s 2024-09-16T03:56:08.774 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.767+0000 7f31ee102640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f31e80a2340 0x7f31e8072180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:41838/0 (socket says 172.21.15.42:41838) 2024-09-16T03:56:08.774 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.767+0000 7f31ee102640 1 -- 172.21.15.42:0/1425914453 learned_addr learned my addr 172.21.15.42:0/1425914453 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:56:08.774 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.767+0000 7f31ed901640 1 --2- 172.21.15.42:0/1425914453 >> v2:172.21.15.154:3300/0 conn(0x7f31e80a2720 0x7f31e80a2b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:56:08.774 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.767+0000 7f31ed901640 10 monclient(hunting): get_auth_request con 0x7f31e80a2720 auth_method 0 2024-09-16T03:56:08.774 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.767+0000 7f31ed901640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:56:08.774 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.767+0000 7f31ed901640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:56:08.774 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.767+0000 7f31ed901640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:56:08.774 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.767+0000 7f31ed901640 1 -- 172.21.15.42:0/1425914453 >> v2:172.21.15.154:3300/0 conn(0x7f31e80a2720 msgr2=0x7f31e80a2b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T03:56:08.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.767+0000 7f31ed901640 1 -- 172.21.15.42:0/1425914453 >> v2:172.21.15.154:3300/0 conn(0x7f31e80a2720 msgr2=0x7f31e80a2b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:56:08.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.767+0000 7f31ed901640 1 --2- 172.21.15.42:0/1425914453 >> v2:172.21.15.154:3300/0 conn(0x7f31e80a2720 0x7f31e80a2b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:56:08.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.767+0000 7f31ed901640 1 --2- 172.21.15.42:0/1425914453 >> v2:172.21.15.154:3300/0 conn(0x7f31e80a2720 0x7f31e80a2b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:56:08.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.767+0000 7f31ed100640 1 -- 172.21.15.42:0/1425914453 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1089628211 0 0) 0x7f31e8070b90 con 0x7f31e80a2340 2024-09-16T03:56:08.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.767+0000 7f31ed100640 10 monclient(hunting): _negotiate 2024-09-16T03:56:08.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.767+0000 7f31ed100640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:56:08.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.767+0000 7f31ed100640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:56:08.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.767+0000 7f31ed100640 10 monclient(hunting): my global_id is 824673 2024-09-16T03:56:08.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.767+0000 7f31ed100640 1 -- 172.21.15.42:0/1425914453 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f31d4003530 con 0x7f31e80a2340 2024-09-16T03:56:08.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.767+0000 7f31ed100640 1 -- 172.21.15.42:0/1425914453 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4070630169 0 0) 0x7f31d4003530 con 0x7f31e80a2340 2024-09-16T03:56:08.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.767+0000 7f31ed100640 1 -- 172.21.15.42:0/1425914453 >> v2:172.21.15.154:3300/0 conn(0x7f31e80a2720 msgr2=0x7f31e80a2b30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:56:08.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.767+0000 7f31ed100640 1 --2- 172.21.15.42:0/1425914453 >> v2:172.21.15.154:3300/0 conn(0x7f31e80a2720 0x7f31e80a2b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:56:08.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.767+0000 7f31ed100640 1 -- 172.21.15.42:0/1425914453 >> v1:172.21.15.154:6789/0 conn(0x7f31e813a420 legacy=0x7f31e813c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:56:08.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.767+0000 7f31ed100640 10 monclient: _finish_hunting 0 2024-09-16T03:56:08.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.767+0000 7f31ed100640 1 monclient: found mon.noname-a-legacy 2024-09-16T03:56:08.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.767+0000 7f31ed100640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:56:08.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.767+0000 7f31ed100640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.42:6789/0 2024-09-16T03:56:08.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.767+0000 7f31ed100640 1 -- 172.21.15.42:0/1425914453 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31e80a3070 con 0x7f31e80a2340 2024-09-16T03:56:08.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.767+0000 7f31ed100640 10 monclient: _finish_auth 0 2024-09-16T03:56:08.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.767+0000 7f31ed100640 10 monclient: _check_auth_tickets 2024-09-16T03:56:08.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.767+0000 7f31ed100640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:56:08.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.767+0000 7f31ed100640 1 -- 172.21.15.42:0/1425914453 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2177413734 0 0) 0x7f31d8001680 con 0x7f31e80a2340 2024-09-16T03:56:08.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.767+0000 7f31ed100640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:56:08.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.767+0000 7f31ed100640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T03:56:08.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.767+0000 7f31ed100640 10 monclient: dump: 2024-09-16T03:56:08.777 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:56:08.777 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:56:08.777 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:08.777 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:08.777 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:56:08.777 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:56:08.777 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:56:08.777 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:56:08.777 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:56:08.777 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:56:08.777 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.767+0000 7f31ed100640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] but i'm connected to v1:172.21.15.42:6789/0, reconnecting 2024-09-16T03:56:08.777 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.767+0000 7f31ed100640 10 monclient: _reopen_session rank -1 2024-09-16T03:56:08.777 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.767+0000 7f31ed100640 1 -- 172.21.15.42:0/1425914453 >> v1:172.21.15.42:6789/0 conn(0x7f31e80a2340 legacy=0x7f31e8072180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:56:08.777 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.767+0000 7f31ed100640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:56:08.778 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.767+0000 7f31ed100640 1 --2- 172.21.15.42:0/1425914453 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f31d40039f0 0x7f31d4003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:56:08.778 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.767+0000 7f31ed100640 10 monclient(hunting): picked mon.b con 0x7f31d40039f0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:56:08.778 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.767+0000 7f31ed100640 1 --2- 172.21.15.42:0/1425914453 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f31d4004810 0x7f31d4004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:56:08.778 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.767+0000 7f31ed100640 10 monclient(hunting): picked mon.a con 0x7f31d4004810 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:56:08.778 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.767+0000 7f31ed100640 1 --2- 172.21.15.42:0/1425914453 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f31d40052a0 0x7f31d400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:56:08.778 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.767+0000 7f31ed100640 10 monclient(hunting): picked mon.c con 0x7f31d40052a0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:56:08.778 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.767+0000 7f31ed100640 10 monclient(hunting): start opening mon connection 2024-09-16T03:56:08.778 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.767+0000 7f31ed100640 1 -- 172.21.15.42:0/1425914453 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f31e806cae0 con 0x7f31d4004810 2024-09-16T03:56:08.778 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.767+0000 7f31ed100640 10 monclient(hunting): start opening mon connection 2024-09-16T03:56:08.778 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.767+0000 7f31ed100640 1 -- 172.21.15.42:0/1425914453 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f31d400c1f0 con 0x7f31d40039f0 2024-09-16T03:56:08.778 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.767+0000 7f31ed100640 10 monclient(hunting): start opening mon connection 2024-09-16T03:56:08.778 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.767+0000 7f31ed100640 1 -- 172.21.15.42:0/1425914453 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f31d400c3f0 con 0x7f31d40052a0 2024-09-16T03:56:08.778 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.767+0000 7f31ed100640 10 monclient(hunting): _renew_subs 2024-09-16T03:56:08.779 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.767+0000 7f31ed100640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:56:08.779 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.767+0000 7f31ed901640 1 --2- 172.21.15.42:0/1425914453 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f31d4004810 0x7f31d4004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:56:08.779 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.767+0000 7f31ed901640 10 monclient(hunting): get_auth_request con 0x7f31d4004810 auth_method 0 2024-09-16T03:56:08.779 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.767+0000 7f31ed901640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:56:08.779 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.767+0000 7f31ed901640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:56:08.779 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.767+0000 7f31ed901640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:56:08.779 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.767+0000 7f31ee102640 1 --2- 172.21.15.42:0/1425914453 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f31d40039f0 0x7f31d4003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:56:08.779 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.767+0000 7f31ee903640 1 --2- 172.21.15.42:0/1425914453 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f31d40052a0 0x7f31d400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:56:08.779 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.767+0000 7f31ee102640 10 monclient(hunting): get_auth_request con 0x7f31d40039f0 auth_method 0 2024-09-16T03:56:08.779 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.767+0000 7f31ee102640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:56:08.779 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.767+0000 7f31ee102640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:56:08.779 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.767+0000 7f31ee102640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:56:08.779 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.767+0000 7f31ee903640 10 monclient(hunting): get_auth_request con 0x7f31d40052a0 auth_method 0 2024-09-16T03:56:08.779 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.767+0000 7f31ee903640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:56:08.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.767+0000 7f31ee903640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:56:08.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.767+0000 7f31ee903640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:56:08.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.767+0000 7f31ed901640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:56:08.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.767+0000 7f31ed901640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:56:08.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.767+0000 7f31ed901640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T03:56:08.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.767+0000 7f31ee102640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:56:08.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.767+0000 7f31ee102640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:56:08.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.767+0000 7f31ee102640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T03:56:08.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.767+0000 7f31ee903640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:56:08.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.767+0000 7f31ee903640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:56:08.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.767+0000 7f31ee903640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T03:56:08.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.767+0000 7f31ed901640 10 monclient(hunting): handle_auth_done global_id 824673 payload 293 2024-09-16T03:56:08.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.767+0000 7f31ed901640 1 -- 172.21.15.42:0/1425914453 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f31d40052a0 msgr2=0x7f31d400b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:56:08.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.767+0000 7f31ed901640 1 --2- 172.21.15.42:0/1425914453 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f31d40052a0 0x7f31d400b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:56:08.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.767+0000 7f31ed901640 1 -- 172.21.15.42:0/1425914453 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f31d40039f0 msgr2=0x7f31d4003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:56:08.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.767+0000 7f31ed901640 1 --2- 172.21.15.42:0/1425914453 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f31d40039f0 0x7f31d4003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:56:08.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.767+0000 7f31ed901640 10 monclient: _finish_hunting 0 2024-09-16T03:56:08.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.767+0000 7f31ed901640 1 monclient: found mon.a 2024-09-16T03:56:08.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.767+0000 7f31ed901640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:56:08.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.767+0000 7f31ed901640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:56:08.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.767+0000 7f31ed901640 1 -- 172.21.15.42:0/1425914453 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31d4003530 con 0x7f31d4004810 2024-09-16T03:56:08.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.767+0000 7f31ed901640 10 monclient: _finish_auth 0 2024-09-16T03:56:08.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.767+0000 7f31ed901640 10 monclient: _check_auth_tickets 2024-09-16T03:56:08.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.767+0000 7f31ed901640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:56:08.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.767+0000 7f31ee903640 1 -- 172.21.15.42:0/1425914453 reap_dead start 2024-09-16T03:56:08.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.767+0000 7f31f0366640 5 monclient: authenticate success, global_id 824673 2024-09-16T03:56:08.782 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.767+0000 7f31f0366640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:56:08.782 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.767+0000 7f31ed901640 1 --2- 172.21.15.42:0/1425914453 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f31d4004810 0x7f31d4004c90 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f31e400c820 tx=0x7f31e401c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18e4de3c4fe18e2d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:56:08.782 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.767+0000 7f31ed100640 1 -- 172.21.15.42:0/1425914453 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31e40b0070 con 0x7f31d4004810 2024-09-16T03:56:08.782 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.767+0000 7f31ed100640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:56:08.782 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.767+0000 7f31ed100640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:56:08.782 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.767+0000 7f31ed100640 10 monclient: dump: 2024-09-16T03:56:08.782 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:56:08.782 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:56:08.782 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:08.782 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:08.782 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:56:08.783 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:56:08.783 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:56:08.783 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:56:08.783 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:56:08.783 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:56:08.783 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.767+0000 7f31f0366640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:56:08.783 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.767+0000 7f31ed100640 1 -- 172.21.15.42:0/1425914453 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f31e40ab600 con 0x7f31d4004810 2024-09-16T03:56:08.783 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.767+0000 7f31ed100640 10 monclient: handle_config config(0 keys) 2024-09-16T03:56:08.783 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.767+0000 7f31f0366640 10 monclient: get_monmap_and_config success 2024-09-16T03:56:08.783 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.767+0000 7f31ed100640 1 -- 172.21.15.42:0/1425914453 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31e400f040 con 0x7f31d4004810 2024-09-16T03:56:08.783 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.767+0000 7f31ed100640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:56:08.783 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.767+0000 7f31f0366640 4 set_mon_vals no callback set 2024-09-16T03:56:08.783 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.767+0000 7f31ed100640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:56:08.784 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.767+0000 7f31ed100640 10 monclient: dump: 2024-09-16T03:56:08.784 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:56:08.784 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:56:08.784 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:08.784 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:08.784 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:56:08.784 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:56:08.784 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:56:08.784 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:56:08.784 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:56:08.784 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:56:08.789 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.771+0000 7f31f0366640 10 monclient: shutdown 2024-09-16T03:56:08.789 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.771+0000 7f31f0366640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:56:08.789 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.771+0000 7f31f0366640 1 -- 172.21.15.42:0/1425914453 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f31d4004810 msgr2=0x7f31d4004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:56:08.790 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.771+0000 7f31f0366640 1 --2- 172.21.15.42:0/1425914453 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f31d4004810 0x7f31d4004c90 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f31e400c820 tx=0x7f31e401c4f0 comp rx=0 tx=0).stop 2024-09-16T03:56:08.790 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.771+0000 7f31f0366640 1 -- 172.21.15.42:0/1425914453 shutdown_connections 2024-09-16T03:56:08.790 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.771+0000 7f31f0366640 1 --2- 172.21.15.42:0/1425914453 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f31d4004810 0x7f31d4004c90 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:56:08.790 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.771+0000 7f31f0366640 1 -- 172.21.15.42:0/1425914453 >> 172.21.15.42:0/1425914453 conn(0x7f31e80a3430 msgr2=0x7f31e80a3810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:56:08.790 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.771+0000 7f31f0366640 1 -- 172.21.15.42:0/1425914453 shutdown_connections 2024-09-16T03:56:08.790 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.771+0000 7f31f0366640 1 -- 172.21.15.42:0/1425914453 wait complete. 2024-09-16T03:56:08.790 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.771+0000 7f31f0366640 10 monclient: build_initial_monmap 2024-09-16T03:56:08.790 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.771+0000 7f31f0366640 10 monclient: monmap: 2024-09-16T03:56:08.790 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:56:08.790 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:56:08.790 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:56:08.791 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:56:08.791 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:56:08.791 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:56:08.791 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:56:08.791 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:56:08.791 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:56:08.791 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:56:08.791 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.771+0000 7f31f0366640 1 librados: starting msgr at 2024-09-16T03:56:08.791 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.771+0000 7f31f0366640 1 librados: starting objecter 2024-09-16T03:56:08.791 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.771+0000 7f31f0366640 1 Processor -- start 2024-09-16T03:56:08.791 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.771+0000 7f31f0366640 1 -- start start 2024-09-16T03:56:08.791 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.771+0000 7f31f0366640 1 librados: setting wanted keys 2024-09-16T03:56:08.791 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.771+0000 7f31f0366640 1 librados: calling monclient init 2024-09-16T03:56:08.792 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.771+0000 7f31f0366640 10 monclient: init 2024-09-16T03:56:08.792 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.771+0000 7f31f0366640 10 monclient: _reopen_session rank -1 2024-09-16T03:56:08.792 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.771+0000 7f31f0366640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:56:08.792 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.771+0000 7f31f0366640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f31d4004810 0x7f31e81cf780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:56:08.792 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.771+0000 7f31f0366640 10 monclient(hunting): picked mon.noname-a con 0x7f31d4004810 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:56:08.792 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.771+0000 7f31f0366640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f31e81343d0 0x7f31e8134810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:56:08.792 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.771+0000 7f31f0366640 10 monclient(hunting): picked mon.noname-c con 0x7f31e81343d0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:56:08.792 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.771+0000 7f31f0366640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f31e806e8b0 0x7f31e806ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:56:08.792 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.771+0000 7f31f0366640 10 monclient(hunting): picked mon.noname-b con 0x7f31e806e8b0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:56:08.792 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.771+0000 7f31f0366640 10 monclient(hunting): start opening mon connection 2024-09-16T03:56:08.792 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.771+0000 7f31f0366640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f31e8135d10 con 0x7f31d4004810 2024-09-16T03:56:08.792 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.771+0000 7f31f0366640 10 monclient(hunting): start opening mon connection 2024-09-16T03:56:08.792 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.771+0000 7f31f0366640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f31e8135b90 con 0x7f31e806e8b0 2024-09-16T03:56:08.793 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.771+0000 7f31f0366640 10 monclient(hunting): start opening mon connection 2024-09-16T03:56:08.793 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.771+0000 7f31f0366640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f31e8135e90 con 0x7f31e81343d0 2024-09-16T03:56:08.793 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.771+0000 7f31f0366640 10 monclient(hunting): _renew_subs 2024-09-16T03:56:08.793 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.771+0000 7f31f0366640 10 monclient(hunting): authenticate will time out at 1815.741987s 2024-09-16T03:56:08.793 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.771+0000 7f31ee102640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f31d4004810 0x7f31e81cf780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:56:08.793 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.771+0000 7f31ed901640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f31e81343d0 0x7f31e8134810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:56:08.793 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.771+0000 7f31ee102640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f31d4004810 0x7f31e81cf780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:44980/0 (socket says 172.21.15.42:44980) 2024-09-16T03:56:08.793 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.771+0000 7f31ee102640 1 -- 172.21.15.42:0/365950380 learned_addr learned my addr 172.21.15.42:0/365950380 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:56:08.793 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.771+0000 7f31ee102640 10 monclient(hunting): get_auth_request con 0x7f31d4004810 auth_method 0 2024-09-16T03:56:08.793 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.771+0000 7f31ee102640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:56:08.793 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.771+0000 7f31ee102640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:56:08.793 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.771+0000 7f31ee102640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:56:08.793 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.771+0000 7f31ed901640 10 monclient(hunting): get_auth_request con 0x7f31e81343d0 auth_method 0 2024-09-16T03:56:08.794 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.771+0000 7f31ed901640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:56:08.794 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.771+0000 7f31ed901640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:56:08.794 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.771+0000 7f31ed901640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:56:08.794 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.771+0000 7f31ed901640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:56:08.794 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.771+0000 7f31ed901640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:56:08.794 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.771+0000 7f31ed901640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:56:08.794 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.771+0000 7f31ee102640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:56:08.794 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.771+0000 7f31ee102640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:56:08.794 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.771+0000 7f31ee102640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:56:08.794 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.771+0000 7f31ee903640 1 --2- 172.21.15.42:0/365950380 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f31e806e8b0 0x7f31e806ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:56:08.794 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.771+0000 7f31ee903640 10 monclient(hunting): get_auth_request con 0x7f31e806e8b0 auth_method 0 2024-09-16T03:56:08.794 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.771+0000 7f31ee903640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:56:08.794 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.771+0000 7f31ee903640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:56:08.794 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.771+0000 7f31ee903640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:56:08.795 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.771+0000 7f31ee102640 10 monclient(hunting): handle_auth_done global_id 824676 payload 824 2024-09-16T03:56:08.795 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.771+0000 7f31ee102640 1 -- 172.21.15.42:0/365950380 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f31e81343d0 msgr2=0x7f31e8134810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:56:08.795 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.771+0000 7f31ee102640 1 --2- 172.21.15.42:0/365950380 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f31e81343d0 0x7f31e8134810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:56:08.795 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.771+0000 7f31ee102640 1 -- 172.21.15.42:0/365950380 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f31e806e8b0 msgr2=0x7f31e806ed30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:56:08.795 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.771+0000 7f31ee102640 1 --2- 172.21.15.42:0/365950380 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f31e806e8b0 0x7f31e806ed30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:56:08.795 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.771+0000 7f31ee102640 10 monclient: _finish_hunting 0 2024-09-16T03:56:08.795 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.771+0000 7f31ee102640 1 monclient: found mon.noname-a 2024-09-16T03:56:08.795 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.771+0000 7f31ee102640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:56:08.795 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.771+0000 7f31ee102640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:56:08.795 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.771+0000 7f31ee102640 1 -- 172.21.15.42:0/365950380 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31e80a0750 con 0x7f31d4004810 2024-09-16T03:56:08.795 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.771+0000 7f31ed901640 1 --2- 172.21.15.42:0/365950380 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f31e81343d0 0x7f31e8134810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:56:08.795 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.771+0000 7f31ee102640 1 --2- 172.21.15.42:0/365950380 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f31d4004810 0x7f31e81cf780 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f31d8007290 tx=0x7f31d8007750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:56:08.795 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.771+0000 7f31deffd640 10 client.?.objecter ms_handle_connect 0x7f31d4004810 2024-09-16T03:56:08.796 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.771+0000 7f31deffd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:56:08.796 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.771+0000 7f31deffd640 1 -- 172.21.15.42:0/365950380 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31d800f940 con 0x7f31d4004810 2024-09-16T03:56:08.796 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.771+0000 7f31deffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:56:08.796 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.771+0000 7f31deffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:56:08.796 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.771+0000 7f31deffd640 10 monclient: dump: 2024-09-16T03:56:08.796 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:56:08.796 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:56:08.796 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:08.796 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:08.796 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:56:08.796 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:56:08.796 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:56:08.796 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:56:08.796 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:56:08.797 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:56:08.797 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.771+0000 7f31deffd640 10 monclient: _finish_auth 0 2024-09-16T03:56:08.797 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.771+0000 7f31deffd640 10 monclient: _check_auth_tickets 2024-09-16T03:56:08.797 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.771+0000 7f31deffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:56:08.797 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.771+0000 7f31deffd640 1 -- 172.21.15.42:0/365950380 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f31d8018070 con 0x7f31d4004810 2024-09-16T03:56:08.797 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.771+0000 7f31f0366640 5 monclient: authenticate success, global_id 824676 2024-09-16T03:56:08.797 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.771+0000 7f31f0366640 10 monclient: _renew_subs 2024-09-16T03:56:08.797 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.771+0000 7f31f0366640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:56:08.797 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.771+0000 7f31f0366640 1 -- 172.21.15.42:0/365950380 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31e8126820 con 0x7f31d4004810 2024-09-16T03:56:08.797 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.771+0000 7f31deffd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:56:08.797 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.771+0000 7f31deffd640 1 -- 172.21.15.42:0/365950380 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31d800fd40 con 0x7f31d4004810 2024-09-16T03:56:08.797 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.771+0000 7f31deffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:56:08.797 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.771+0000 7f31deffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:56:08.798 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.771+0000 7f31deffd640 10 monclient: dump: 2024-09-16T03:56:08.798 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:56:08.798 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:56:08.798 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:08.798 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:08.798 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:56:08.798 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:56:08.798 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:56:08.798 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:56:08.798 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:56:08.798 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:56:08.798 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.771+0000 7f31dffff640 4 set_mon_vals no callback set 2024-09-16T03:56:08.798 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.771+0000 7f31f0366640 10 client.824676.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:56:08.798 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.771+0000 7f31f0366640 10 monclient: _renew_subs 2024-09-16T03:56:08.799 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.771+0000 7f31f0366640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:56:08.799 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.771+0000 7f31f0366640 1 -- 172.21.15.42:0/365950380 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31e81253d0 con 0x7f31d4004810 2024-09-16T03:56:08.799 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.771+0000 7f31f0366640 1 librados: init done 2024-09-16T03:56:08.799 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.771+0000 7f31f0366640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:56:08.799 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.771+0000 7f31f0366640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:56:08.799 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.771+0000 7f31f0366640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:56:08.799 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.771+0000 7f31f0366640 1 -- 172.21.15.42:0/365950380 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f31e8070b90 con 0x7f31d4004810 2024-09-16T03:56:08.799 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.771+0000 7f31deffd640 1 -- 172.21.15.42:0/365950380 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f31d801d460 con 0x7f31d4004810 2024-09-16T03:56:08.799 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.775+0000 7f31deffd640 1 --2- 172.21.15.42:0/365950380 >> v2:172.21.15.154:6812/254845342 conn(0x7f31cc037810 0x7f31cc039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:56:08.799 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.775+0000 7f31deffd640 1 -- 172.21.15.42:0/365950380 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(440..440 src has 330..440) ==== 5713+0+0 (secure 0 0 0) 0x7f31d8048f60 con 0x7f31d4004810 2024-09-16T03:56:08.799 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.775+0000 7f31deffd640 10 client.824676.objecter ms_dispatch 0x7f31e8000ba0 osd_map(440..440 src has 330..440) 2024-09-16T03:56:08.799 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.775+0000 7f31deffd640 3 client.824676.objecter handle_osd_map got epochs [440,440] > 0 2024-09-16T03:56:08.799 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.775+0000 7f31deffd640 3 client.824676.objecter handle_osd_map decoding full epoch 440 2024-09-16T03:56:08.799 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.775+0000 7f31deffd640 20 client.824676.objecter dump_active .. 0 homeless 2024-09-16T03:56:08.800 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.775+0000 7f31ed901640 1 --2- 172.21.15.42:0/365950380 >> v2:172.21.15.154:6812/254845342 conn(0x7f31cc037810 0x7f31cc039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:56:08.800 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.775+0000 7f31ed901640 10 monclient: get_auth_request con 0x7f31cc037810 auth_method 0 2024-09-16T03:56:08.800 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.775+0000 7f31deffd640 10 client.824676.objecter ms_handle_connect 0x7f31cc037810 2024-09-16T03:56:08.800 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.775+0000 7f31ed901640 1 --2- 172.21.15.42:0/365950380 >> v2:172.21.15.154:6812/254845342 conn(0x7f31cc037810 0x7f31cc039cd0 secure :-1 s=READY pgs=3648 cs=0 l=1 rev1=1 crypto rx=0x7f31e40ad000 tx=0x7f31e40046d0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:56:08.800 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.779+0000 7f31deffd640 1 -- 172.21.15.42:0/365950380 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f31d801b520 con 0x7f31d4004810 2024-09-16T03:56:08.800 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.779+0000 7f31deffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:56:08.800 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:08.779+0000 7f31deffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:56:09.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:09.003+0000 7f31f0366640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:56:09.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:09.003+0000 7f31f0366640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:56:09.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:09.003+0000 7f31f0366640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:56:09.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:09.003+0000 7f31f0366640 1 -- 172.21.15.42:0/365950380 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f31e80739f0 con 0x7f31d4004810 2024-09-16T03:56:09.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:09.003+0000 7f31deffd640 1 -- 172.21.15.42:0/365950380 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7f31d801b780 con 0x7f31d4004810 2024-09-16T03:56:09.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:09.003+0000 7f31deffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:56:09.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:09.003+0000 7f31deffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:56:09.008 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":120,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":22,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:56:09.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:09.007+0000 7f31f0366640 10 librados: watch_flush enter 2024-09-16T03:56:09.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:09.007+0000 7f31f0366640 10 librados: watch_flush exit 2024-09-16T03:56:09.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:09.007+0000 7f31f0366640 20 client.824676.objecter shutdown clearing up homeless session... 2024-09-16T03:56:09.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:09.007+0000 7f31f0366640 10 client.824676.objecter successfully canceled tick 2024-09-16T03:56:09.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:09.007+0000 7f31f0366640 1 -- 172.21.15.42:0/365950380 >> v2:172.21.15.154:6812/254845342 conn(0x7f31cc037810 msgr2=0x7f31cc039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:56:09.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:09.007+0000 7f31f0366640 1 --2- 172.21.15.42:0/365950380 >> v2:172.21.15.154:6812/254845342 conn(0x7f31cc037810 0x7f31cc039cd0 secure :-1 s=READY pgs=3648 cs=0 l=1 rev1=1 crypto rx=0x7f31e40ad000 tx=0x7f31e40046d0 comp rx=0 tx=0).stop 2024-09-16T03:56:09.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:09.007+0000 7f31f0366640 10 monclient: shutdown 2024-09-16T03:56:09.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:09.007+0000 7f31f0366640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:56:09.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:09.007+0000 7f31f0366640 1 -- 172.21.15.42:0/365950380 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f31d4004810 msgr2=0x7f31e81cf780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:56:09.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:09.007+0000 7f31f0366640 1 --2- 172.21.15.42:0/365950380 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f31d4004810 0x7f31e81cf780 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f31d8007290 tx=0x7f31d8007750 comp rx=0 tx=0).stop 2024-09-16T03:56:09.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:09.007+0000 7f31f0366640 1 -- 172.21.15.42:0/365950380 shutdown_connections 2024-09-16T03:56:09.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:09.007+0000 7f31f0366640 1 --2- 172.21.15.42:0/365950380 >> v2:172.21.15.154:6812/254845342 conn(0x7f31cc037810 0x7f31cc039cd0 unknown :-1 s=CLOSED pgs=3648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:56:09.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:09.007+0000 7f31f0366640 1 --2- 172.21.15.42:0/365950380 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f31e806e8b0 0x7f31e806ed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:56:09.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:09.007+0000 7f31f0366640 1 --2- 172.21.15.42:0/365950380 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f31e81343d0 0x7f31e8134810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:56:09.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:09.007+0000 7f31f0366640 1 --2- 172.21.15.42:0/365950380 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f31d4004810 0x7f31e81cf780 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:56:09.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:09.007+0000 7f31f0366640 1 -- 172.21.15.42:0/365950380 >> 172.21.15.42:0/365950380 conn(0x7f31e80a3430 msgr2=0x7f31e80a56c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:56:09.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:09.007+0000 7f31f0366640 1 -- 172.21.15.42:0/365950380 shutdown_connections 2024-09-16T03:56:09.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:09.007+0000 7f31f0366640 1 -- 172.21.15.42:0/365950380 wait complete. 2024-09-16T03:56:09.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:09.007+0000 7f31f0366640 1 librados: shutdown 2024-09-16T03:56:12.028 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:56:12.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:12.155+0000 7f470ae77640 10 monclient: get_monmap_and_config 2024-09-16T03:56:12.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:12.155+0000 7f470ae77640 10 monclient: build_initial_monmap 2024-09-16T03:56:12.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:12.155+0000 7f470ae77640 10 monclient: monmap: 2024-09-16T03:56:12.159 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:56:12.159 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:56:12.159 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:56:12.157910+0000 2024-09-16T03:56:12.160 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:56:12.157910+0000 2024-09-16T03:56:12.160 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:56:12.160 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:56:12.160 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:56:12.160 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:56:12.160 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:56:12.160 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:56:12.160 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:56:12.160 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:56:12.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:12.155+0000 7f470ae77640 1 Processor -- start 2024-09-16T03:56:12.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:12.155+0000 7f470ae77640 1 -- start start 2024-09-16T03:56:12.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:12.155+0000 7f470ae77640 10 monclient: init 2024-09-16T03:56:12.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:12.155+0000 7f470ae77640 10 monclient: _reopen_session rank -1 2024-09-16T03:56:12.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:12.155+0000 7f470ae77640 10 monclient: _add_conns ranks=[2,1,0,3,4] 2024-09-16T03:56:12.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:12.155+0000 7f470ae77640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f470409fb00 0x7f470409ba50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:56:12.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:12.155+0000 7f470ae77640 10 monclient(hunting): picked mon.noname-b con 0x7f470409fb00 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:56:12.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:12.155+0000 7f470ae77640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f470409bf90 addr v1:172.21.15.154:6789/0 2024-09-16T03:56:12.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:12.155+0000 7f470ae77640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f470409c3d0 addr v1:172.21.15.42:6789/0 2024-09-16T03:56:12.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:12.155+0000 7f470ae77640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f47040a1e20 con 0x7f470409c3d0 2024-09-16T03:56:12.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:12.155+0000 7f470ae77640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f470409c860 con 0x7f470409bf90 2024-09-16T03:56:12.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:12.155+0000 7f470ae77640 10 monclient(hunting): start opening mon connection 2024-09-16T03:56:12.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:12.155+0000 7f470ae77640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f470409e250 con 0x7f470409fb00 2024-09-16T03:56:12.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:12.155+0000 7f470ae77640 10 monclient(hunting): _renew_subs 2024-09-16T03:56:12.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:12.155+0000 7f4708c13640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f470409fb00 0x7f470409ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:56:12.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:12.155+0000 7f470ae77640 10 monclient(hunting): authenticate will time out at 1819.127712s 2024-09-16T03:56:12.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:12.155+0000 7f4709414640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f470409c3d0 0x7f470413b840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:41852/0 (socket says 172.21.15.42:41852) 2024-09-16T03:56:12.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:12.155+0000 7f4709414640 1 -- 172.21.15.42:0/2821108027 learned_addr learned my addr 172.21.15.42:0/2821108027 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:56:12.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:12.155+0000 7f4708c13640 10 monclient(hunting): get_auth_request con 0x7f470409fb00 auth_method 0 2024-09-16T03:56:12.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:12.155+0000 7f4708c13640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:56:12.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:12.155+0000 7f4708c13640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:56:12.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:12.155+0000 7f4708c13640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:56:12.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:12.155+0000 7f4708c13640 1 -- 172.21.15.42:0/2821108027 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f470409fb00 msgr2=0x7f470409ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T03:56:12.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:12.155+0000 7f4708c13640 1 -- 172.21.15.42:0/2821108027 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f470409fb00 msgr2=0x7f470409ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:56:12.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:12.155+0000 7f4708c13640 1 --2- 172.21.15.42:0/2821108027 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f470409fb00 0x7f470409ba50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:56:12.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:12.155+0000 7f4708c13640 1 --2- 172.21.15.42:0/2821108027 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f470409fb00 0x7f470409ba50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:56:12.366 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:12.363+0000 7f4708c13640 1 --2- 172.21.15.42:0/2821108027 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f470409fb00 0x7f470409ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:56:12.366 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:12.363+0000 7f4708c13640 10 monclient(hunting): get_auth_request con 0x7f470409fb00 auth_method 0 2024-09-16T03:56:12.366 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:12.363+0000 7f4708c13640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:56:12.366 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:12.363+0000 7f4708c13640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:56:12.366 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:12.363+0000 7f4708c13640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:56:12.366 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:12.363+0000 7f4708c13640 1 -- 172.21.15.42:0/2821108027 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f470409fb00 msgr2=0x7f470409ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T03:56:12.366 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:12.363+0000 7f4708c13640 1 -- 172.21.15.42:0/2821108027 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f470409fb00 msgr2=0x7f470409ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:56:12.366 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:12.363+0000 7f4708c13640 1 --2- 172.21.15.42:0/2821108027 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f470409fb00 0x7f470409ba50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:56:12.366 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:12.363+0000 7f4708c13640 1 --2- 172.21.15.42:0/2821108027 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f470409fb00 0x7f470409ba50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T03:56:12.770 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:12.767+0000 7f4708c13640 1 --2- 172.21.15.42:0/2821108027 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f470409fb00 0x7f470409ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:56:12.770 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:12.767+0000 7f4708c13640 10 monclient(hunting): get_auth_request con 0x7f470409fb00 auth_method 0 2024-09-16T03:56:12.770 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:12.767+0000 7f4708c13640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:56:12.770 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:12.767+0000 7f4708c13640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:56:12.770 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:12.767+0000 7f4708c13640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:56:12.770 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:12.767+0000 7f4708c13640 1 -- 172.21.15.42:0/2821108027 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f470409fb00 msgr2=0x7f470409ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T03:56:12.770 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:12.767+0000 7f4708c13640 1 -- 172.21.15.42:0/2821108027 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f470409fb00 msgr2=0x7f470409ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:56:12.770 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:12.767+0000 7f4708c13640 1 --2- 172.21.15.42:0/2821108027 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f470409fb00 0x7f470409ba50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:56:12.771 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:12.767+0000 7f4708c13640 1 --2- 172.21.15.42:0/2821108027 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f470409fb00 0x7f470409ba50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T03:56:13.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:13.571+0000 7f4708c13640 1 --2- 172.21.15.42:0/2821108027 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f470409fb00 0x7f470409ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:56:13.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:13.571+0000 7f4708c13640 10 monclient(hunting): get_auth_request con 0x7f470409fb00 auth_method 0 2024-09-16T03:56:13.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:13.571+0000 7f4708c13640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:56:13.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:13.571+0000 7f4708c13640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:56:13.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:13.571+0000 7f4708c13640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:56:13.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:13.571+0000 7f4708c13640 1 -- 172.21.15.42:0/2821108027 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f470409fb00 msgr2=0x7f470409ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T03:56:13.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:13.571+0000 7f4708c13640 1 -- 172.21.15.42:0/2821108027 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f470409fb00 msgr2=0x7f470409ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:56:13.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:13.571+0000 7f4708c13640 1 --2- 172.21.15.42:0/2821108027 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f470409fb00 0x7f470409ba50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:56:13.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:13.571+0000 7f4708c13640 1 --2- 172.21.15.42:0/2821108027 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f470409fb00 0x7f470409ba50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T03:56:15.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:15.159+0000 7f47027fc640 10 monclient(hunting): tick 2024-09-16T03:56:15.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:15.159+0000 7f47027fc640 10 monclient(hunting): _check_auth_tickets 2024-09-16T03:56:15.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:15.159+0000 7f47027fc640 1 monclient(hunting): continuing hunt 2024-09-16T03:56:15.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:15.159+0000 7f47027fc640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T03:56:15.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:15.159+0000 7f47027fc640 1 -- 172.21.15.42:0/2821108027 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f470409fb00 msgr2=0x7f470409ba50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:56:15.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:15.159+0000 7f47027fc640 1 --2- 172.21.15.42:0/2821108027 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f470409fb00 0x7f470409ba50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:56:15.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:15.159+0000 7f47027fc640 1 -- 172.21.15.42:0/2821108027 >> v1:172.21.15.154:6789/0 conn(0x7f470409bf90 legacy=0x7f4704138110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:56:15.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:15.159+0000 7f47027fc640 1 -- 172.21.15.42:0/2821108027 >> v1:172.21.15.42:6789/0 conn(0x7f470409c3d0 legacy=0x7f470413b840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:56:15.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:15.159+0000 7f47027fc640 10 monclient: _add_conns ranks=[3,4] 2024-09-16T03:56:15.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:15.159+0000 7f47027fc640 1 --2- 172.21.15.42:0/2821108027 >> v2:172.21.15.42:3300/0 conn(0x7f46e4005930 0x7f46e4007d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:56:15.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:15.159+0000 7f47027fc640 10 monclient(hunting): picked mon.noname-a con 0x7f46e4005930 addr v2:172.21.15.42:3300/0 2024-09-16T03:56:15.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:15.159+0000 7f47027fc640 1 --2- 172.21.15.42:0/2821108027 >> v2:172.21.15.154:3300/0 conn(0x7f46e4008480 0x7f46e400a940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:56:15.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:15.159+0000 7f47027fc640 10 monclient(hunting): picked mon.noname-c con 0x7f46e4008480 addr v2:172.21.15.154:3300/0 2024-09-16T03:56:15.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:15.159+0000 7f47027fc640 10 monclient(hunting): start opening mon connection 2024-09-16T03:56:15.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:15.159+0000 7f47027fc640 1 -- 172.21.15.42:0/2821108027 --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f470409e250 con 0x7f46e4005930 2024-09-16T03:56:15.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:15.159+0000 7f47027fc640 10 monclient(hunting): start opening mon connection 2024-09-16T03:56:15.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:15.159+0000 7f47027fc640 1 -- 172.21.15.42:0/2821108027 --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f46e400b000 con 0x7f46e4008480 2024-09-16T03:56:15.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:15.159+0000 7f47027fc640 10 monclient(hunting): _renew_subs 2024-09-16T03:56:15.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:15.159+0000 7f4708c13640 1 --2- 172.21.15.42:0/2821108027 >> v2:172.21.15.42:3300/0 conn(0x7f46e4005930 0x7f46e4007d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:56:15.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:15.159+0000 7f4708c13640 10 monclient(hunting): get_auth_request con 0x7f46e4005930 auth_method 0 2024-09-16T03:56:15.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:15.159+0000 7f4708c13640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:56:15.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:15.159+0000 7f4708c13640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:56:15.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:15.159+0000 7f4708c13640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:56:15.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:15.159+0000 7f4703fff640 1 --2- 172.21.15.42:0/2821108027 >> v2:172.21.15.154:3300/0 conn(0x7f46e4008480 0x7f46e400a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:56:15.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:15.159+0000 7f4703fff640 10 monclient(hunting): get_auth_request con 0x7f46e4008480 auth_method 0 2024-09-16T03:56:15.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:15.159+0000 7f4703fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:56:15.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:15.159+0000 7f4703fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:56:15.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:15.159+0000 7f4703fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:56:15.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:15.159+0000 7f4708c13640 1 -- 172.21.15.42:0/2821108027 >> v2:172.21.15.42:3300/0 conn(0x7f46e4005930 msgr2=0x7f46e4007d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T03:56:15.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:15.159+0000 7f4708c13640 1 -- 172.21.15.42:0/2821108027 >> v2:172.21.15.42:3300/0 conn(0x7f46e4005930 msgr2=0x7f46e4007d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:56:15.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:15.159+0000 7f4708c13640 1 --2- 172.21.15.42:0/2821108027 >> v2:172.21.15.42:3300/0 conn(0x7f46e4005930 0x7f46e4007d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:56:15.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:15.159+0000 7f4703fff640 1 -- 172.21.15.42:0/2821108027 >> v2:172.21.15.154:3300/0 conn(0x7f46e4008480 msgr2=0x7f46e400a940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T03:56:15.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:15.159+0000 7f4703fff640 1 -- 172.21.15.42:0/2821108027 >> v2:172.21.15.154:3300/0 conn(0x7f46e4008480 msgr2=0x7f46e400a940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:56:15.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:15.159+0000 7f4703fff640 1 --2- 172.21.15.42:0/2821108027 >> v2:172.21.15.154:3300/0 conn(0x7f46e4008480 0x7f46e400a940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:56:15.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:15.159+0000 7f4708c13640 1 --2- 172.21.15.42:0/2821108027 >> v2:172.21.15.42:3300/0 conn(0x7f46e4005930 0x7f46e4007d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:56:15.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:15.159+0000 7f4703fff640 1 --2- 172.21.15.42:0/2821108027 >> v2:172.21.15.154:3300/0 conn(0x7f46e4008480 0x7f46e400a940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:56:15.366 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:15.363+0000 7f4708c13640 1 --2- 172.21.15.42:0/2821108027 >> v2:172.21.15.42:3300/0 conn(0x7f46e4005930 0x7f46e4007d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:56:15.366 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:15.363+0000 7f4708c13640 10 monclient(hunting): get_auth_request con 0x7f46e4005930 auth_method 0 2024-09-16T03:56:15.367 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:15.363+0000 7f4708c13640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:56:15.367 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:15.363+0000 7f4708c13640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:56:15.367 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:15.363+0000 7f4708c13640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:56:15.367 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:15.363+0000 7f4708c13640 1 -- 172.21.15.42:0/2821108027 >> v2:172.21.15.42:3300/0 conn(0x7f46e4005930 msgr2=0x7f46e4007d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T03:56:15.367 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:15.363+0000 7f4708c13640 1 -- 172.21.15.42:0/2821108027 >> v2:172.21.15.42:3300/0 conn(0x7f46e4005930 msgr2=0x7f46e4007d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:56:15.367 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:15.363+0000 7f4703fff640 1 --2- 172.21.15.42:0/2821108027 >> v2:172.21.15.154:3300/0 conn(0x7f46e4008480 0x7f46e400a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:56:15.367 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:15.363+0000 7f4708c13640 1 --2- 172.21.15.42:0/2821108027 >> v2:172.21.15.42:3300/0 conn(0x7f46e4005930 0x7f46e4007d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:56:15.367 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:15.363+0000 7f4703fff640 10 monclient(hunting): get_auth_request con 0x7f46e4008480 auth_method 0 2024-09-16T03:56:15.367 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:15.363+0000 7f4703fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:56:15.367 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:15.363+0000 7f4703fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:56:15.368 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:15.363+0000 7f4703fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:56:15.368 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:15.363+0000 7f4708c13640 1 --2- 172.21.15.42:0/2821108027 >> v2:172.21.15.42:3300/0 conn(0x7f46e4005930 0x7f46e4007d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T03:56:15.368 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:15.363+0000 7f4703fff640 1 -- 172.21.15.42:0/2821108027 >> v2:172.21.15.154:3300/0 conn(0x7f46e4008480 msgr2=0x7f46e400a940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T03:56:15.368 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:15.363+0000 7f4703fff640 1 -- 172.21.15.42:0/2821108027 >> v2:172.21.15.154:3300/0 conn(0x7f46e4008480 msgr2=0x7f46e400a940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:56:15.368 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:15.363+0000 7f4703fff640 1 --2- 172.21.15.42:0/2821108027 >> v2:172.21.15.154:3300/0 conn(0x7f46e4008480 0x7f46e400a940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:56:15.368 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:15.363+0000 7f4703fff640 1 --2- 172.21.15.42:0/2821108027 >> v2:172.21.15.154:3300/0 conn(0x7f46e4008480 0x7f46e400a940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T03:56:15.770 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:15.767+0000 7f4708c13640 1 --2- 172.21.15.42:0/2821108027 >> v2:172.21.15.42:3300/0 conn(0x7f46e4005930 0x7f46e4007d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:56:15.770 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:15.767+0000 7f4708c13640 10 monclient(hunting): get_auth_request con 0x7f46e4005930 auth_method 0 2024-09-16T03:56:15.770 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:15.767+0000 7f4708c13640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:56:15.770 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:15.767+0000 7f4708c13640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:56:15.770 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:15.767+0000 7f4708c13640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:56:15.770 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:15.767+0000 7f4703fff640 1 --2- 172.21.15.42:0/2821108027 >> v2:172.21.15.154:3300/0 conn(0x7f46e4008480 0x7f46e400a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:56:15.771 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:15.767+0000 7f4708c13640 1 -- 172.21.15.42:0/2821108027 >> v2:172.21.15.42:3300/0 conn(0x7f46e4005930 msgr2=0x7f46e4007d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T03:56:15.771 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:15.767+0000 7f4708c13640 1 -- 172.21.15.42:0/2821108027 >> v2:172.21.15.42:3300/0 conn(0x7f46e4005930 msgr2=0x7f46e4007d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:56:15.771 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:15.767+0000 7f4708c13640 1 --2- 172.21.15.42:0/2821108027 >> v2:172.21.15.42:3300/0 conn(0x7f46e4005930 0x7f46e4007d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:56:15.771 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:15.767+0000 7f4703fff640 10 monclient(hunting): get_auth_request con 0x7f46e4008480 auth_method 0 2024-09-16T03:56:15.771 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:15.767+0000 7f4703fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:56:15.771 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:15.767+0000 7f4703fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:56:15.771 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:15.767+0000 7f4703fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:56:15.771 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:15.767+0000 7f4708c13640 1 --2- 172.21.15.42:0/2821108027 >> v2:172.21.15.42:3300/0 conn(0x7f46e4005930 0x7f46e4007d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T03:56:15.771 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:15.767+0000 7f4703fff640 1 -- 172.21.15.42:0/2821108027 >> v2:172.21.15.154:3300/0 conn(0x7f46e4008480 msgr2=0x7f46e400a940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T03:56:15.771 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:15.767+0000 7f4703fff640 1 -- 172.21.15.42:0/2821108027 >> v2:172.21.15.154:3300/0 conn(0x7f46e4008480 msgr2=0x7f46e400a940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:56:15.771 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:15.767+0000 7f4703fff640 1 --2- 172.21.15.42:0/2821108027 >> v2:172.21.15.154:3300/0 conn(0x7f46e4008480 0x7f46e400a940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:56:15.771 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:15.767+0000 7f4703fff640 1 --2- 172.21.15.42:0/2821108027 >> v2:172.21.15.154:3300/0 conn(0x7f46e4008480 0x7f46e400a940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T03:56:16.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:16.570+0000 7f4708c13640 1 --2- 172.21.15.42:0/2821108027 >> v2:172.21.15.42:3300/0 conn(0x7f46e4005930 0x7f46e4007d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:56:16.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:16.570+0000 7f4708c13640 10 monclient(hunting): get_auth_request con 0x7f46e4005930 auth_method 0 2024-09-16T03:56:16.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:16.570+0000 7f4708c13640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:56:16.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:16.570+0000 7f4708c13640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:56:16.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:16.570+0000 7f4708c13640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:56:16.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:16.570+0000 7f4703fff640 1 --2- 172.21.15.42:0/2821108027 >> v2:172.21.15.154:3300/0 conn(0x7f46e4008480 0x7f46e400a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:56:16.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:16.570+0000 7f4708c13640 1 -- 172.21.15.42:0/2821108027 >> v2:172.21.15.42:3300/0 conn(0x7f46e4005930 msgr2=0x7f46e4007d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T03:56:16.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:16.570+0000 7f4708c13640 1 -- 172.21.15.42:0/2821108027 >> v2:172.21.15.42:3300/0 conn(0x7f46e4005930 msgr2=0x7f46e4007d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:56:16.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:16.570+0000 7f4708c13640 1 --2- 172.21.15.42:0/2821108027 >> v2:172.21.15.42:3300/0 conn(0x7f46e4005930 0x7f46e4007d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:56:16.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:16.570+0000 7f4703fff640 10 monclient(hunting): get_auth_request con 0x7f46e4008480 auth_method 0 2024-09-16T03:56:16.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:16.570+0000 7f4703fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:56:16.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:16.570+0000 7f4703fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:56:16.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:16.570+0000 7f4703fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:56:16.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:16.570+0000 7f4708c13640 1 --2- 172.21.15.42:0/2821108027 >> v2:172.21.15.42:3300/0 conn(0x7f46e4005930 0x7f46e4007d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T03:56:16.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:16.570+0000 7f4703fff640 1 -- 172.21.15.42:0/2821108027 >> v2:172.21.15.154:3300/0 conn(0x7f46e4008480 msgr2=0x7f46e400a940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T03:56:16.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:16.570+0000 7f4703fff640 1 -- 172.21.15.42:0/2821108027 >> v2:172.21.15.154:3300/0 conn(0x7f46e4008480 msgr2=0x7f46e400a940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:56:16.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:16.570+0000 7f4703fff640 1 --2- 172.21.15.42:0/2821108027 >> v2:172.21.15.154:3300/0 conn(0x7f46e4008480 0x7f46e400a940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:56:16.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:16.570+0000 7f4703fff640 1 --2- 172.21.15.42:0/2821108027 >> v2:172.21.15.154:3300/0 conn(0x7f46e4008480 0x7f46e400a940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T03:56:18.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.154+0000 7f47027fc640 10 monclient(hunting): tick 2024-09-16T03:56:18.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.154+0000 7f47027fc640 10 monclient(hunting): _check_auth_tickets 2024-09-16T03:56:18.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.154+0000 7f47027fc640 1 monclient(hunting): continuing hunt 2024-09-16T03:56:18.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.154+0000 7f47027fc640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T03:56:18.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.154+0000 7f47027fc640 1 -- 172.21.15.42:0/2821108027 >> v2:172.21.15.154:3300/0 conn(0x7f46e4008480 msgr2=0x7f46e400a940 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:56:18.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.158+0000 7f47027fc640 1 --2- 172.21.15.42:0/2821108027 >> v2:172.21.15.154:3300/0 conn(0x7f46e4008480 0x7f46e400a940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:56:18.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.158+0000 7f47027fc640 1 -- 172.21.15.42:0/2821108027 >> v2:172.21.15.42:3300/0 conn(0x7f46e4005930 msgr2=0x7f46e4007d90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:56:18.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.158+0000 7f47027fc640 1 --2- 172.21.15.42:0/2821108027 >> v2:172.21.15.42:3300/0 conn(0x7f46e4005930 0x7f46e4007d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:56:18.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.158+0000 7f4709414640 1 -- 172.21.15.42:0/2821108027 reap_dead start 2024-09-16T03:56:18.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.158+0000 7f47027fc640 10 monclient: _add_conns ranks=[0,1,3,4,2] 2024-09-16T03:56:18.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.158+0000 7f47027fc640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f46e4005d80 addr v1:172.21.15.42:6789/0 2024-09-16T03:56:18.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.158+0000 7f47027fc640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f46e400a030 addr v1:172.21.15.154:6789/0 2024-09-16T03:56:18.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.158+0000 7f47027fc640 1 --2- 172.21.15.42:0/2821108027 >> v2:172.21.15.42:3300/0 conn(0x7f46e400e4d0 0x7f46e40108c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:56:18.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.158+0000 7f47027fc640 10 monclient(hunting): picked mon.noname-a con 0x7f46e400e4d0 addr v2:172.21.15.42:3300/0 2024-09-16T03:56:18.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.158+0000 7f47027fc640 1 -- 172.21.15.42:0/2821108027 --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f470409ea30 con 0x7f46e4005d80 2024-09-16T03:56:18.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.158+0000 7f47027fc640 1 -- 172.21.15.42:0/2821108027 --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f46e4011e10 con 0x7f46e400a030 2024-09-16T03:56:18.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.158+0000 7f47027fc640 10 monclient(hunting): start opening mon connection 2024-09-16T03:56:18.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.158+0000 7f47027fc640 1 -- 172.21.15.42:0/2821108027 --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f470409e250 con 0x7f46e400e4d0 2024-09-16T03:56:18.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.158+0000 7f47027fc640 10 monclient(hunting): _renew_subs 2024-09-16T03:56:18.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.158+0000 7f4709414640 1 --2- 172.21.15.42:0/2821108027 >> v2:172.21.15.42:3300/0 conn(0x7f46e400e4d0 0x7f46e40108c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:56:18.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.158+0000 7f4709414640 10 monclient(hunting): get_auth_request con 0x7f46e400e4d0 auth_method 0 2024-09-16T03:56:18.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.158+0000 7f4709414640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:56:18.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.158+0000 7f4709414640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:56:18.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.158+0000 7f4709414640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:56:18.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.158+0000 7f4709414640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:56:18.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.158+0000 7f4709414640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:56:18.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.158+0000 7f4709414640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:56:18.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.158+0000 7f4709414640 10 monclient(hunting): handle_auth_done global_id 824679 payload 274 2024-09-16T03:56:18.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.158+0000 7f4709414640 1 -- 172.21.15.42:0/2821108027 >> v1:172.21.15.154:6789/0 conn(0x7f46e400a030 legacy=0x7f46e400a450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:56:18.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.158+0000 7f4709414640 1 -- 172.21.15.42:0/2821108027 >> v1:172.21.15.42:6789/0 conn(0x7f46e4005d80 legacy=0x7f46e4008910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:56:18.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.158+0000 7f47037fe640 1 -- 172.21.15.42:0/2821108027 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3386392004 0 0) 0x7f470409ea30 con 0x7f46e4005d80 2024-09-16T03:56:18.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.158+0000 7f4709414640 10 monclient: _finish_hunting 0 2024-09-16T03:56:18.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.158+0000 7f4709414640 1 monclient: found mon.noname-a 2024-09-16T03:56:18.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.158+0000 7f4709414640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:56:18.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.158+0000 7f4709414640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:56:18.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.158+0000 7f4709414640 1 -- 172.21.15.42:0/2821108027 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46e4012ff0 con 0x7f46e400e4d0 2024-09-16T03:56:18.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.158+0000 7f47037fe640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T03:56:18.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.162+0000 7f4709414640 1 --2- 172.21.15.42:0/2821108027 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f46e400e4d0 0x7f46e40108c0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f470409c7b0 tx=0x7f46fc01c0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1812e2e01f8e8f18 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:56:18.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.166+0000 7f47037fe640 1 -- 172.21.15.42:0/2821108027 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46fc0a4cb0 con 0x7f46e400e4d0 2024-09-16T03:56:18.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.166+0000 7f47037fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:56:18.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.166+0000 7f47037fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:56:18.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.166+0000 7f47037fe640 10 monclient: dump: 2024-09-16T03:56:18.171 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:56:18.171 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:56:18.171 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:18.171 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:18.171 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:56:18.171 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:56:18.171 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:56:18.172 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:56:18.172 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:56:18.172 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:56:18.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.166+0000 7f47037fe640 10 monclient: _finish_auth 0 2024-09-16T03:56:18.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.166+0000 7f47037fe640 10 monclient: _check_auth_tickets 2024-09-16T03:56:18.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.166+0000 7f47037fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:56:18.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.166+0000 7f47037fe640 1 -- 172.21.15.42:0/2821108027 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f46fc0a8020 con 0x7f46e400e4d0 2024-09-16T03:56:18.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.166+0000 7f470ae77640 5 monclient: authenticate success, global_id 824679 2024-09-16T03:56:18.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.166+0000 7f470ae77640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:56:18.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.166+0000 7f47037fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:56:18.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.166+0000 7f470ae77640 10 monclient: get_monmap_and_config success 2024-09-16T03:56:18.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.166+0000 7f470ae77640 4 set_mon_vals no callback set 2024-09-16T03:56:18.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.166+0000 7f47037fe640 1 -- 172.21.15.42:0/2821108027 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46fc004ad0 con 0x7f46e400e4d0 2024-09-16T03:56:18.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.166+0000 7f47037fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:56:18.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.166+0000 7f47037fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:56:18.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.166+0000 7f47037fe640 10 monclient: dump: 2024-09-16T03:56:18.173 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:56:18.173 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:56:18.173 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:18.173 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:18.173 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:56:18.173 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:56:18.173 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:56:18.173 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:56:18.173 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:56:18.173 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:56:18.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.166+0000 7f470ae77640 10 monclient: shutdown 2024-09-16T03:56:18.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.166+0000 7f470ae77640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:56:18.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.166+0000 7f470ae77640 1 -- 172.21.15.42:0/2821108027 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f46e400e4d0 msgr2=0x7f46e40108c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:56:18.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.166+0000 7f470ae77640 1 --2- 172.21.15.42:0/2821108027 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f46e400e4d0 0x7f46e40108c0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f470409c7b0 tx=0x7f46fc01c0f0 comp rx=0 tx=0).stop 2024-09-16T03:56:18.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.166+0000 7f470ae77640 1 -- 172.21.15.42:0/2821108027 shutdown_connections 2024-09-16T03:56:18.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.166+0000 7f470ae77640 1 --2- 172.21.15.42:0/2821108027 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f46e400e4d0 0x7f46e40108c0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:56:18.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.166+0000 7f470ae77640 1 -- 172.21.15.42:0/2821108027 >> 172.21.15.42:0/2821108027 conn(0x7f47040a2420 msgr2=0x7f47040a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:56:18.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.166+0000 7f470ae77640 1 -- 172.21.15.42:0/2821108027 shutdown_connections 2024-09-16T03:56:18.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.166+0000 7f470ae77640 1 -- 172.21.15.42:0/2821108027 wait complete. 2024-09-16T03:56:18.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.170+0000 7f470ae77640 10 monclient: build_initial_monmap 2024-09-16T03:56:18.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.170+0000 7f470ae77640 10 monclient: monmap: 2024-09-16T03:56:18.175 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:56:18.175 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:56:18.175 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:56:18.175 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:56:18.175 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:56:18.176 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:56:18.176 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:56:18.176 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:56:18.176 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:56:18.176 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:56:18.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.170+0000 7f470ae77640 1 librados: starting msgr at 2024-09-16T03:56:18.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.170+0000 7f470ae77640 1 librados: starting objecter 2024-09-16T03:56:18.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.170+0000 7f470ae77640 1 Processor -- start 2024-09-16T03:56:18.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.170+0000 7f470ae77640 1 -- start start 2024-09-16T03:56:18.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.170+0000 7f470ae77640 1 librados: setting wanted keys 2024-09-16T03:56:18.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.170+0000 7f470ae77640 1 librados: calling monclient init 2024-09-16T03:56:18.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.170+0000 7f470ae77640 10 monclient: init 2024-09-16T03:56:18.180 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.170+0000 7f470ae77640 10 monclient: _reopen_session rank -1 2024-09-16T03:56:18.181 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.170+0000 7f470ae77640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T03:56:18.181 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.170+0000 7f470ae77640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f46e4005d80 0x7f47041cf340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:56:18.181 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.170+0000 7f470ae77640 10 monclient(hunting): picked mon.noname-a con 0x7f46e4005d80 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:56:18.181 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.170+0000 7f470ae77640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f46e400a030 0x7f47041333a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:56:18.181 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.170+0000 7f470ae77640 10 monclient(hunting): picked mon.noname-b con 0x7f46e400a030 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:56:18.181 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.170+0000 7f470ae77640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f46e400e4d0 0x7f47041338e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:56:18.181 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.170+0000 7f470ae77640 10 monclient(hunting): picked mon.noname-c con 0x7f46e400e4d0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:56:18.181 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.170+0000 7f470ae77640 10 monclient(hunting): start opening mon connection 2024-09-16T03:56:18.181 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.170+0000 7f470ae77640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f47040a3170 con 0x7f46e4005d80 2024-09-16T03:56:18.181 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.170+0000 7f470ae77640 10 monclient(hunting): start opening mon connection 2024-09-16T03:56:18.181 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.170+0000 7f470ae77640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f470409d0b0 con 0x7f46e400a030 2024-09-16T03:56:18.181 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.170+0000 7f470ae77640 10 monclient(hunting): start opening mon connection 2024-09-16T03:56:18.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.170+0000 7f470ae77640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f47040a32f0 con 0x7f46e400e4d0 2024-09-16T03:56:18.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.170+0000 7f470ae77640 10 monclient(hunting): _renew_subs 2024-09-16T03:56:18.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.170+0000 7f470ae77640 10 monclient(hunting): authenticate will time out at 1825.142167s 2024-09-16T03:56:18.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.170+0000 7f4708c13640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f46e4005d80 0x7f47041cf340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:56:18.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.170+0000 7f4709414640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f46e400e4d0 0x7f47041338e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:56:18.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.170+0000 7f4708c13640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f46e4005d80 0x7f47041cf340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:36998/0 (socket says 172.21.15.42:36998) 2024-09-16T03:56:18.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.170+0000 7f4708c13640 1 -- 172.21.15.42:0/1132462571 learned_addr learned my addr 172.21.15.42:0/1132462571 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:56:18.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.170+0000 7f4708c13640 10 monclient(hunting): get_auth_request con 0x7f46e4005d80 auth_method 0 2024-09-16T03:56:18.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.170+0000 7f4708c13640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:56:18.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.170+0000 7f4708c13640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:56:18.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.170+0000 7f4708c13640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:56:18.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.170+0000 7f4709414640 10 monclient(hunting): get_auth_request con 0x7f46e400e4d0 auth_method 0 2024-09-16T03:56:18.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.170+0000 7f4709414640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:56:18.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.170+0000 7f4709414640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:56:18.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.170+0000 7f4709414640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:56:18.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.170+0000 7f4708c13640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:56:18.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.170+0000 7f4708c13640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:56:18.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.170+0000 7f4708c13640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:56:18.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.170+0000 7f4703fff640 1 --2- 172.21.15.42:0/1132462571 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f46e400a030 0x7f47041333a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:56:18.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.170+0000 7f4709414640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:56:18.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.170+0000 7f4709414640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:56:18.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.170+0000 7f4709414640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:56:18.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.170+0000 7f4703fff640 10 monclient(hunting): get_auth_request con 0x7f46e400a030 auth_method 0 2024-09-16T03:56:18.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.170+0000 7f4703fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:56:18.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.170+0000 7f4703fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:56:18.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.170+0000 7f4703fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:56:18.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.170+0000 7f4703fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:56:18.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.170+0000 7f4703fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:56:18.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.170+0000 7f4703fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:56:18.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.170+0000 7f4708c13640 10 monclient(hunting): handle_auth_done global_id 824685 payload 824 2024-09-16T03:56:18.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.170+0000 7f4708c13640 1 -- 172.21.15.42:0/1132462571 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f46e400e4d0 msgr2=0x7f47041338e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:56:18.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.170+0000 7f4708c13640 1 --2- 172.21.15.42:0/1132462571 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f46e400e4d0 0x7f47041338e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:56:18.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.170+0000 7f4708c13640 1 -- 172.21.15.42:0/1132462571 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f46e400a030 msgr2=0x7f47041333a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:56:18.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.170+0000 7f4708c13640 1 --2- 172.21.15.42:0/1132462571 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f46e400a030 0x7f47041333a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:56:18.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.170+0000 7f4708c13640 10 monclient: _finish_hunting 0 2024-09-16T03:56:18.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.170+0000 7f4708c13640 1 monclient: found mon.noname-a 2024-09-16T03:56:18.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.170+0000 7f4708c13640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:56:18.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.170+0000 7f4708c13640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:56:18.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.170+0000 7f4708c13640 1 -- 172.21.15.42:0/1132462571 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4704071d30 con 0x7f46e4005d80 2024-09-16T03:56:18.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.170+0000 7f4708c13640 1 --2- 172.21.15.42:0/1132462571 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f46e4005d80 0x7f47041cf340 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f46f40071c0 tx=0x7f46f4017780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:56:18.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.170+0000 7f47017fa640 10 client.?.objecter ms_handle_connect 0x7f46e4005d80 2024-09-16T03:56:18.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.170+0000 7f47017fa640 10 client.?.objecter resend_mon_ops 2024-09-16T03:56:18.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.170+0000 7f47017fa640 1 -- 172.21.15.42:0/1132462571 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46f400acc0 con 0x7f46e4005d80 2024-09-16T03:56:18.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.170+0000 7f47017fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:56:18.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.170+0000 7f47017fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:56:18.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.170+0000 7f47017fa640 10 monclient: dump: 2024-09-16T03:56:18.185 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:56:18.185 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:56:18.185 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:18.185 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:18.185 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:56:18.185 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:56:18.185 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:56:18.185 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:56:18.185 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:56:18.186 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:56:18.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.170+0000 7f47017fa640 10 monclient: _finish_auth 0 2024-09-16T03:56:18.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.170+0000 7f47017fa640 10 monclient: _check_auth_tickets 2024-09-16T03:56:18.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.170+0000 7f47017fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:56:18.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.170+0000 7f47017fa640 1 -- 172.21.15.42:0/1132462571 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f46f400ae80 con 0x7f46e4005d80 2024-09-16T03:56:18.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.170+0000 7f47017fa640 10 monclient: handle_config config(0 keys) 2024-09-16T03:56:18.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.170+0000 7f47017fa640 1 -- 172.21.15.42:0/1132462571 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46f4021670 con 0x7f46e4005d80 2024-09-16T03:56:18.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.170+0000 7f47017fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:56:18.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.170+0000 7f47017fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:56:18.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.170+0000 7f47017fa640 10 monclient: dump: 2024-09-16T03:56:18.186 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:56:18.186 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:56:18.186 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:18.186 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:18.187 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:56:18.187 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:56:18.187 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:56:18.187 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:56:18.187 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:56:18.187 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:56:18.187 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.170+0000 7f470ae77640 5 monclient: authenticate success, global_id 824685 2024-09-16T03:56:18.187 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.170+0000 7f4702ffd640 4 set_mon_vals no callback set 2024-09-16T03:56:18.187 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.170+0000 7f470ae77640 10 monclient: _renew_subs 2024-09-16T03:56:18.187 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.170+0000 7f470ae77640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:56:18.187 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.170+0000 7f470ae77640 1 -- 172.21.15.42:0/1132462571 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f470406e240 con 0x7f46e4005d80 2024-09-16T03:56:18.187 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.170+0000 7f470ae77640 10 client.824685.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:56:18.187 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.170+0000 7f470ae77640 10 monclient: _renew_subs 2024-09-16T03:56:18.187 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.170+0000 7f470ae77640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:56:18.188 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.170+0000 7f470ae77640 1 -- 172.21.15.42:0/1132462571 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4704129df0 con 0x7f46e4005d80 2024-09-16T03:56:18.188 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.170+0000 7f470ae77640 1 librados: init done 2024-09-16T03:56:18.188 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.174+0000 7f47017fa640 1 -- 172.21.15.42:0/1132462571 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f46f4015040 con 0x7f46e4005d80 2024-09-16T03:56:18.188 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.174+0000 7f470ae77640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:56:18.188 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.174+0000 7f470ae77640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:56:18.188 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.174+0000 7f470ae77640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:56:18.188 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.174+0000 7f47017fa640 1 --2- 172.21.15.42:0/1132462571 >> v2:172.21.15.154:6812/254845342 conn(0x7f46e4039470 0x7f46e403b930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:56:18.188 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.174+0000 7f470ae77640 1 -- 172.21.15.42:0/1132462571 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4704136100 con 0x7f46e4005d80 2024-09-16T03:56:18.188 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.174+0000 7f47017fa640 1 -- 172.21.15.42:0/1132462571 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(441..441 src has 330..441) ==== 5769+0+0 (secure 0 0 0) 0x7f46f404a5d0 con 0x7f46e4005d80 2024-09-16T03:56:18.188 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.174+0000 7f47017fa640 10 client.824685.objecter ms_dispatch 0x7f4704000ba0 osd_map(441..441 src has 330..441) 2024-09-16T03:56:18.188 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.174+0000 7f47017fa640 3 client.824685.objecter handle_osd_map got epochs [441,441] > 0 2024-09-16T03:56:18.188 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.174+0000 7f47017fa640 3 client.824685.objecter handle_osd_map decoding full epoch 441 2024-09-16T03:56:18.188 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.174+0000 7f47017fa640 20 client.824685.objecter dump_active .. 0 homeless 2024-09-16T03:56:18.188 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.174+0000 7f4703fff640 1 --2- 172.21.15.42:0/1132462571 >> v2:172.21.15.154:6812/254845342 conn(0x7f46e4039470 0x7f46e403b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:56:18.189 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.174+0000 7f4703fff640 10 monclient: get_auth_request con 0x7f46e4039470 auth_method 0 2024-09-16T03:56:18.189 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.174+0000 7f4703fff640 1 --2- 172.21.15.42:0/1132462571 >> v2:172.21.15.154:6812/254845342 conn(0x7f46e4039470 0x7f46e403b930 secure :-1 s=READY pgs=3649 cs=0 l=1 rev1=1 crypto rx=0x7f4704129d20 tx=0x7f46ec009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:56:18.189 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.174+0000 7f47017fa640 10 client.824685.objecter ms_handle_connect 0x7f46e4039470 2024-09-16T03:56:18.189 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.178+0000 7f47017fa640 1 -- 172.21.15.42:0/1132462571 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f46f4020e20 con 0x7f46e4005d80 2024-09-16T03:56:18.189 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.178+0000 7f47017fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:56:18.189 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.178+0000 7f47017fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:56:18.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.402+0000 7f470ae77640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:56:18.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.402+0000 7f470ae77640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:56:18.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.402+0000 7f470ae77640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:56:18.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.402+0000 7f470ae77640 1 -- 172.21.15.42:0/1132462571 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f47041283e0 con 0x7f46e4005d80 2024-09-16T03:56:18.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.406+0000 7f47017fa640 1 -- 172.21.15.42:0/1132462571 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7f46f402a440 con 0x7f46e4005d80 2024-09-16T03:56:18.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.406+0000 7f47017fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:56:18.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.406+0000 7f47017fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:56:18.410 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":128,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:56:18.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.406+0000 7f470ae77640 10 librados: watch_flush enter 2024-09-16T03:56:18.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.406+0000 7f470ae77640 10 librados: watch_flush exit 2024-09-16T03:56:18.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.406+0000 7f470ae77640 20 client.824685.objecter shutdown clearing up homeless session... 2024-09-16T03:56:18.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.406+0000 7f470ae77640 10 client.824685.objecter successfully canceled tick 2024-09-16T03:56:18.413 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.406+0000 7f470ae77640 1 -- 172.21.15.42:0/1132462571 >> v2:172.21.15.154:6812/254845342 conn(0x7f46e4039470 msgr2=0x7f46e403b930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:56:18.413 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.406+0000 7f470ae77640 1 --2- 172.21.15.42:0/1132462571 >> v2:172.21.15.154:6812/254845342 conn(0x7f46e4039470 0x7f46e403b930 secure :-1 s=READY pgs=3649 cs=0 l=1 rev1=1 crypto rx=0x7f4704129d20 tx=0x7f46ec009040 comp rx=0 tx=0).stop 2024-09-16T03:56:18.413 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.406+0000 7f470ae77640 10 monclient: shutdown 2024-09-16T03:56:18.413 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.406+0000 7f470ae77640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:56:18.413 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.406+0000 7f470ae77640 1 -- 172.21.15.42:0/1132462571 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f46e4005d80 msgr2=0x7f47041cf340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:56:18.413 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.406+0000 7f470ae77640 1 --2- 172.21.15.42:0/1132462571 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f46e4005d80 0x7f47041cf340 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f46f40071c0 tx=0x7f46f4017780 comp rx=0 tx=0).stop 2024-09-16T03:56:18.413 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.406+0000 7f470ae77640 1 -- 172.21.15.42:0/1132462571 shutdown_connections 2024-09-16T03:56:18.413 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.406+0000 7f470ae77640 1 --2- 172.21.15.42:0/1132462571 >> v2:172.21.15.154:6812/254845342 conn(0x7f46e4039470 0x7f46e403b930 unknown :-1 s=CLOSED pgs=3649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:56:18.413 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.406+0000 7f470ae77640 1 --2- 172.21.15.42:0/1132462571 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f46e400e4d0 0x7f47041338e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:56:18.413 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.406+0000 7f470ae77640 1 --2- 172.21.15.42:0/1132462571 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f46e400a030 0x7f47041333a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:56:18.413 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.406+0000 7f470ae77640 1 --2- 172.21.15.42:0/1132462571 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f46e4005d80 0x7f47041cf340 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:56:18.414 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.406+0000 7f470ae77640 1 -- 172.21.15.42:0/1132462571 >> 172.21.15.42:0/1132462571 conn(0x7f47040a2420 msgr2=0x7f470406e760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:56:18.414 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.406+0000 7f470ae77640 1 -- 172.21.15.42:0/1132462571 shutdown_connections 2024-09-16T03:56:18.414 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.406+0000 7f470ae77640 1 -- 172.21.15.42:0/1132462571 wait complete. 2024-09-16T03:56:18.414 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.406+0000 7f470ae77640 1 librados: shutdown 2024-09-16T03:56:18.430 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T03:56:18.430 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T03:56:18.431 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T03:56:18.557 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.554+0000 7f77c754d640 10 monclient: get_monmap_and_config 2024-09-16T03:56:18.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.554+0000 7f77c754d640 10 monclient: build_initial_monmap 2024-09-16T03:56:18.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.554+0000 7f77c754d640 10 monclient: monmap: 2024-09-16T03:56:18.558 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:56:18.558 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:56:18.558 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:56:18.556366+0000 2024-09-16T03:56:18.558 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:56:18.556366+0000 2024-09-16T03:56:18.558 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:56:18.558 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:56:18.558 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:56:18.558 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:56:18.558 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:56:18.558 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:56:18.558 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:56:18.559 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:56:18.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.554+0000 7f77c754d640 1 Processor -- start 2024-09-16T03:56:18.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.554+0000 7f77c754d640 1 -- start start 2024-09-16T03:56:18.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.554+0000 7f77c754d640 10 monclient: init 2024-09-16T03:56:18.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.554+0000 7f77c754d640 10 monclient: _reopen_session rank -1 2024-09-16T03:56:18.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.554+0000 7f77c754d640 10 monclient: _add_conns ranks=[4,3,2,0,1] 2024-09-16T03:56:18.562 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.554+0000 7f77c754d640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f77c00a39b0 0x7f77c0133b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:56:18.562 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.554+0000 7f77c754d640 10 monclient(hunting): picked mon.noname-c con 0x7f77c00a39b0 addr v2:172.21.15.154:3300/0 2024-09-16T03:56:18.562 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.554+0000 7f77c754d640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f77c00a3d90 0x7f77c0137000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:56:18.562 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.554+0000 7f77c754d640 10 monclient(hunting): picked mon.noname-a con 0x7f77c00a3d90 addr v2:172.21.15.42:3300/0 2024-09-16T03:56:18.562 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.554+0000 7f77c754d640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f77c0137540 0x7f77c0137990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:56:18.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.554+0000 7f77c754d640 10 monclient(hunting): picked mon.noname-b con 0x7f77c0137540 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:56:18.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.554+0000 7f77c754d640 10 monclient(hunting): start opening mon connection 2024-09-16T03:56:18.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.554+0000 7f77c754d640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f77c009c140 con 0x7f77c00a3d90 2024-09-16T03:56:18.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.554+0000 7f77c754d640 10 monclient(hunting): start opening mon connection 2024-09-16T03:56:18.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.554+0000 7f77c754d640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f77c009c420 con 0x7f77c00a39b0 2024-09-16T03:56:18.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.554+0000 7f77c754d640 10 monclient(hunting): start opening mon connection 2024-09-16T03:56:18.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.554+0000 7f77c754d640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f77c006dd60 con 0x7f77c0137540 2024-09-16T03:56:18.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.554+0000 7f77c754d640 10 monclient(hunting): _renew_subs 2024-09-16T03:56:18.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.554+0000 7f77c754d640 10 monclient(hunting): authenticate will time out at 1825.525842s 2024-09-16T03:56:18.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.554+0000 7f77c5aea640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f77c0137540 0x7f77c0137990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:56:18.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.554+0000 7f77c4ae8640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f77c00a3d90 0x7f77c0137000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:56:18.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.554+0000 7f77c4ae8640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f77c00a3d90 0x7f77c0137000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:37006/0 (socket says 172.21.15.42:37006) 2024-09-16T03:56:18.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.554+0000 7f77c5aea640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f77c0137540 0x7f77c0137990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:40766/0 (socket says 172.21.15.42:40766) 2024-09-16T03:56:18.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.554+0000 7f77c4ae8640 1 -- 172.21.15.42:0/1756899423 learned_addr learned my addr 172.21.15.42:0/1756899423 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:56:18.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.554+0000 7f77c4ae8640 10 monclient(hunting): get_auth_request con 0x7f77c00a3d90 auth_method 0 2024-09-16T03:56:18.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.554+0000 7f77c4ae8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:56:18.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.554+0000 7f77c4ae8640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:56:18.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.554+0000 7f77c4ae8640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:56:18.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.554+0000 7f77c5aea640 10 monclient(hunting): get_auth_request con 0x7f77c0137540 auth_method 0 2024-09-16T03:56:18.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.554+0000 7f77c5aea640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:56:18.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.554+0000 7f77c5aea640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:56:18.565 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.554+0000 7f77c5aea640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:56:18.565 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.554+0000 7f77c4ae8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:56:18.565 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.554+0000 7f77c4ae8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:56:18.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.554+0000 7f77c4ae8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:56:18.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.554+0000 7f77c5aea640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:56:18.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.554+0000 7f77c5aea640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:56:18.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.554+0000 7f77c5aea640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:56:18.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.554+0000 7f77c52e9640 1 --2- 172.21.15.42:0/1756899423 >> v2:172.21.15.154:3300/0 conn(0x7f77c00a39b0 0x7f77c0133b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:56:18.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.554+0000 7f77c52e9640 10 monclient(hunting): get_auth_request con 0x7f77c00a39b0 auth_method 0 2024-09-16T03:56:18.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.554+0000 7f77c52e9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:56:18.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.554+0000 7f77c52e9640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:56:18.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.554+0000 7f77c52e9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:56:18.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.554+0000 7f77c5aea640 10 monclient(hunting): handle_auth_done global_id 854276 payload 274 2024-09-16T03:56:18.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.554+0000 7f77c5aea640 1 -- 172.21.15.42:0/1756899423 >> v2:172.21.15.154:3300/0 conn(0x7f77c00a39b0 msgr2=0x7f77c0133b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:56:18.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.554+0000 7f77c5aea640 1 --2- 172.21.15.42:0/1756899423 >> v2:172.21.15.154:3300/0 conn(0x7f77c00a39b0 0x7f77c0133b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:56:18.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.554+0000 7f77c5aea640 1 -- 172.21.15.42:0/1756899423 >> v2:172.21.15.42:3300/0 conn(0x7f77c00a3d90 msgr2=0x7f77c0137000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:56:18.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.554+0000 7f77c5aea640 1 --2- 172.21.15.42:0/1756899423 >> v2:172.21.15.42:3300/0 conn(0x7f77c00a3d90 0x7f77c0137000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:56:18.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.554+0000 7f77c5aea640 10 monclient: _finish_hunting 0 2024-09-16T03:56:18.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.554+0000 7f77c5aea640 1 monclient: found mon.noname-b 2024-09-16T03:56:18.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.554+0000 7f77c5aea640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:56:18.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.554+0000 7f77c5aea640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T03:56:18.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.554+0000 7f77c5aea640 1 -- 172.21.15.42:0/1756899423 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77c00a5e20 con 0x7f77c0137540 2024-09-16T03:56:18.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.554+0000 7f77c4ae8640 1 --2- 172.21.15.42:0/1756899423 >> v2:172.21.15.42:3300/0 conn(0x7f77c00a3d90 0x7f77c0137000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:56:18.573 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.558+0000 7f77c5aea640 1 --2- 172.21.15.42:0/1756899423 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f77c0137540 0x7f77c0137990 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f77c00a7750 tx=0x7f77b801c7f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a0a7a7b7bb4c1386 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:56:18.573 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.558+0000 7f77b3fff640 1 -- 172.21.15.42:0/1756899423 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77b80ac070 con 0x7f77c0137540 2024-09-16T03:56:18.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.558+0000 7f77b3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:56:18.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.558+0000 7f77b3fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:56:18.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.558+0000 7f77b3fff640 10 monclient: dump: 2024-09-16T03:56:18.574 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:56:18.574 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:56:18.574 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:18.574 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:18.574 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:56:18.574 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:56:18.574 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:56:18.574 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:56:18.574 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:56:18.574 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:56:18.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.558+0000 7f77b3fff640 10 monclient: _finish_auth 0 2024-09-16T03:56:18.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.558+0000 7f77b3fff640 10 monclient: _check_auth_tickets 2024-09-16T03:56:18.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.558+0000 7f77b3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:56:18.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.558+0000 7f77b3fff640 1 -- 172.21.15.42:0/1756899423 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f77b80a6680 con 0x7f77c0137540 2024-09-16T03:56:18.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.558+0000 7f77b3fff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:56:18.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.558+0000 7f77b3fff640 1 -- 172.21.15.42:0/1756899423 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77b80a6a50 con 0x7f77c0137540 2024-09-16T03:56:18.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.558+0000 7f77c754d640 5 monclient: authenticate success, global_id 854276 2024-09-16T03:56:18.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.558+0000 7f77c754d640 10 monclient: get_monmap_and_config success 2024-09-16T03:56:18.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.558+0000 7f77c754d640 4 set_mon_vals no callback set 2024-09-16T03:56:18.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.558+0000 7f77b3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:56:18.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.558+0000 7f77b3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:56:18.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.558+0000 7f77b3fff640 10 monclient: dump: 2024-09-16T03:56:18.576 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:56:18.576 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:56:18.576 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:18.576 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:18.576 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:56:18.576 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:56:18.576 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:56:18.576 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:56:18.576 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:56:18.577 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:56:18.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.558+0000 7f77c754d640 10 monclient: shutdown 2024-09-16T03:56:18.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.558+0000 7f77c754d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:56:18.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.558+0000 7f77c754d640 1 -- 172.21.15.42:0/1756899423 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f77c0137540 msgr2=0x7f77c0137990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:56:18.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.558+0000 7f77c754d640 1 --2- 172.21.15.42:0/1756899423 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f77c0137540 0x7f77c0137990 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f77c00a7750 tx=0x7f77b801c7f0 comp rx=0 tx=0).stop 2024-09-16T03:56:18.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.558+0000 7f77c754d640 1 -- 172.21.15.42:0/1756899423 shutdown_connections 2024-09-16T03:56:18.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.558+0000 7f77c754d640 1 --2- 172.21.15.42:0/1756899423 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f77c0137540 0x7f77c0137990 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:56:18.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.558+0000 7f77c754d640 1 --2- 172.21.15.42:0/1756899423 >> v2:172.21.15.42:3300/0 conn(0x7f77c00a3d90 0x7f77c0137000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:56:18.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.558+0000 7f77c754d640 1 --2- 172.21.15.42:0/1756899423 >> v2:172.21.15.154:3300/0 conn(0x7f77c00a39b0 0x7f77c0133b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:56:18.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.558+0000 7f77c754d640 1 -- 172.21.15.42:0/1756899423 >> 172.21.15.42:0/1756899423 conn(0x7f77c00a9fb0 msgr2=0x7f77c00aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:56:18.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.558+0000 7f77c754d640 1 -- 172.21.15.42:0/1756899423 shutdown_connections 2024-09-16T03:56:18.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.558+0000 7f77c754d640 1 -- 172.21.15.42:0/1756899423 wait complete. 2024-09-16T03:56:18.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.558+0000 7f77c754d640 10 monclient: build_initial_monmap 2024-09-16T03:56:18.578 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.558+0000 7f77c754d640 10 monclient: monmap: 2024-09-16T03:56:18.578 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:56:18.578 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:56:18.578 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:56:18.578 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:56:18.578 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:56:18.578 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:56:18.578 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:56:18.578 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:56:18.578 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:56:18.578 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:56:18.578 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.558+0000 7f77c754d640 1 librados: starting msgr at 2024-09-16T03:56:18.578 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.558+0000 7f77c754d640 1 librados: starting objecter 2024-09-16T03:56:18.578 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.558+0000 7f77c754d640 1 Processor -- start 2024-09-16T03:56:18.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.558+0000 7f77c754d640 1 -- start start 2024-09-16T03:56:18.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.558+0000 7f77c754d640 1 librados: setting wanted keys 2024-09-16T03:56:18.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.558+0000 7f77c754d640 1 librados: calling monclient init 2024-09-16T03:56:18.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.558+0000 7f77c754d640 10 monclient: init 2024-09-16T03:56:18.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.558+0000 7f77c754d640 10 monclient: _reopen_session rank -1 2024-09-16T03:56:18.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.558+0000 7f77c754d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:56:18.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.558+0000 7f77c754d640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f77c00a39b0 0x7f77c013b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:56:18.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.558+0000 7f77c754d640 10 monclient(hunting): picked mon.noname-c con 0x7f77c00a39b0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:56:18.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.558+0000 7f77c754d640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f77c00a3d90 0x7f77c013b600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:56:18.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.558+0000 7f77c754d640 10 monclient(hunting): picked mon.noname-a con 0x7f77c00a3d90 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:56:18.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.558+0000 7f77c754d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f77c0137540 0x7f77c013bb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:56:18.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.558+0000 7f77c754d640 10 monclient(hunting): picked mon.noname-b con 0x7f77c0137540 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:56:18.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.558+0000 7f77c754d640 10 monclient(hunting): start opening mon connection 2024-09-16T03:56:18.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.558+0000 7f77c754d640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f77c0140dc0 con 0x7f77c00a3d90 2024-09-16T03:56:18.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.558+0000 7f77c754d640 10 monclient(hunting): start opening mon connection 2024-09-16T03:56:18.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.558+0000 7f77c754d640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f77c0140f40 con 0x7f77c0137540 2024-09-16T03:56:18.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.558+0000 7f77c754d640 10 monclient(hunting): start opening mon connection 2024-09-16T03:56:18.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.558+0000 7f77c754d640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f77c009c8d0 con 0x7f77c00a39b0 2024-09-16T03:56:18.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.558+0000 7f77c754d640 10 monclient(hunting): _renew_subs 2024-09-16T03:56:18.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.558+0000 7f77c754d640 10 monclient(hunting): authenticate will time out at 1825.531640s 2024-09-16T03:56:18.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.558+0000 7f77c52e9640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f77c00a39b0 0x7f77c013b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:56:18.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.558+0000 7f77c52e9640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f77c00a39b0 0x7f77c013b0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:40780/0 (socket says 172.21.15.42:40780) 2024-09-16T03:56:18.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.558+0000 7f77c4ae8640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f77c00a3d90 0x7f77c013b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:56:18.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.558+0000 7f77c52e9640 1 -- 172.21.15.42:0/624801176 learned_addr learned my addr 172.21.15.42:0/624801176 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:56:18.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.558+0000 7f77c52e9640 10 monclient(hunting): get_auth_request con 0x7f77c00a39b0 auth_method 0 2024-09-16T03:56:18.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.558+0000 7f77c52e9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:56:18.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.558+0000 7f77c52e9640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:56:18.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.558+0000 7f77c52e9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:56:18.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.558+0000 7f77c4ae8640 10 monclient(hunting): get_auth_request con 0x7f77c00a3d90 auth_method 0 2024-09-16T03:56:18.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.558+0000 7f77c4ae8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:56:18.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.558+0000 7f77c4ae8640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:56:18.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.558+0000 7f77c4ae8640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:56:18.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.558+0000 7f77c52e9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:56:18.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.558+0000 7f77c52e9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:56:18.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.558+0000 7f77c52e9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:56:18.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.558+0000 7f77c4ae8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:56:18.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.558+0000 7f77c4ae8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:56:18.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.558+0000 7f77c4ae8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:56:18.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.562+0000 7f77c5aea640 1 --2- 172.21.15.42:0/624801176 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f77c0137540 0x7f77c013bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:56:18.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.562+0000 7f77c52e9640 10 monclient(hunting): handle_auth_done global_id 854279 payload 824 2024-09-16T03:56:18.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.562+0000 7f77c52e9640 1 -- 172.21.15.42:0/624801176 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f77c0137540 msgr2=0x7f77c013bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:56:18.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.562+0000 7f77c52e9640 1 --2- 172.21.15.42:0/624801176 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f77c0137540 0x7f77c013bb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:56:18.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.562+0000 7f77c52e9640 1 -- 172.21.15.42:0/624801176 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f77c00a3d90 msgr2=0x7f77c013b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:56:18.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.562+0000 7f77c52e9640 1 --2- 172.21.15.42:0/624801176 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f77c00a3d90 0x7f77c013b600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:56:18.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.562+0000 7f77c52e9640 10 monclient: _finish_hunting 0 2024-09-16T03:56:18.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.562+0000 7f77c52e9640 1 monclient: found mon.noname-c 2024-09-16T03:56:18.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.562+0000 7f77c52e9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:56:18.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.562+0000 7f77c52e9640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:56:18.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.562+0000 7f77c52e9640 1 -- 172.21.15.42:0/624801176 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77c013d130 con 0x7f77c00a39b0 2024-09-16T03:56:18.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.562+0000 7f77c5aea640 10 monclient: get_auth_request con 0x7f77c0137540 auth_method 0 2024-09-16T03:56:18.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.562+0000 7f77c5aea640 1 --2- 172.21.15.42:0/624801176 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f77c0137540 0x7f77c013bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T03:56:18.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.562+0000 7f77c4ae8640 1 --2- 172.21.15.42:0/624801176 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f77c00a3d90 0x7f77c013b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:56:18.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.562+0000 7f77c52e9640 1 --2- 172.21.15.42:0/624801176 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f77c00a39b0 0x7f77c013b0c0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f77b400d830 tx=0x7f77b400dcf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:56:18.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.562+0000 7f77b1ffb640 10 client.?.objecter ms_handle_connect 0x7f77c00a39b0 2024-09-16T03:56:18.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.562+0000 7f77b1ffb640 10 client.?.objecter resend_mon_ops 2024-09-16T03:56:18.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.562+0000 7f77b1ffb640 1 -- 172.21.15.42:0/624801176 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77b400b6c0 con 0x7f77c00a39b0 2024-09-16T03:56:18.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.562+0000 7f77b1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:56:18.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.562+0000 7f77b1ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:56:18.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.562+0000 7f77b1ffb640 10 monclient: dump: 2024-09-16T03:56:18.583 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:56:18.583 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:56:18.583 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:18.583 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:18.583 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:56:18.583 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:56:18.583 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:56:18.583 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:56:18.583 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:56:18.583 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:56:18.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.562+0000 7f77b1ffb640 10 monclient: _finish_auth 0 2024-09-16T03:56:18.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.562+0000 7f77b1ffb640 10 monclient: _check_auth_tickets 2024-09-16T03:56:18.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.562+0000 7f77b1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:56:18.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.562+0000 7f77b1ffb640 1 -- 172.21.15.42:0/624801176 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f77b4019070 con 0x7f77c00a39b0 2024-09-16T03:56:18.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.562+0000 7f77b1ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T03:56:18.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.562+0000 7f77b1ffb640 1 -- 172.21.15.42:0/624801176 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77b400ba60 con 0x7f77c00a39b0 2024-09-16T03:56:18.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.562+0000 7f77b1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:56:18.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.562+0000 7f77b1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:56:18.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.562+0000 7f77b1ffb640 10 monclient: dump: 2024-09-16T03:56:18.584 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:56:18.584 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:56:18.584 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:18.584 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:18.584 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:56:18.585 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:56:18.585 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:56:18.585 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:56:18.585 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:56:18.585 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:56:18.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.562+0000 7f77c754d640 5 monclient: authenticate success, global_id 854279 2024-09-16T03:56:18.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.562+0000 7f77c754d640 10 monclient: _renew_subs 2024-09-16T03:56:18.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.562+0000 7f77b37fe640 4 set_mon_vals no callback set 2024-09-16T03:56:18.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.562+0000 7f77c754d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:56:18.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.562+0000 7f77c754d640 1 -- 172.21.15.42:0/624801176 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77c00a4d20 con 0x7f77c00a39b0 2024-09-16T03:56:18.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.562+0000 7f77c754d640 10 client.854279.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:56:18.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.562+0000 7f77c754d640 10 monclient: _renew_subs 2024-09-16T03:56:18.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.562+0000 7f77c754d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:56:18.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.562+0000 7f77c754d640 1 -- 172.21.15.42:0/624801176 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f77c0132a20 con 0x7f77c00a39b0 2024-09-16T03:56:18.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.562+0000 7f77c754d640 1 librados: init done 2024-09-16T03:56:18.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.562+0000 7f77c754d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:56:18.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.562+0000 7f77c754d640 1 --2- 172.21.15.42:0/624801176 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f77c009ceb0 0x7f77c01ceff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:56:18.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.562+0000 7f77c754d640 10 monclient(hunting): start opening mon connection 2024-09-16T03:56:18.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.562+0000 7f77c754d640 1 -- 172.21.15.42:0/624801176 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f77c01342d0 con 0x7f77c009ceb0 2024-09-16T03:56:18.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.562+0000 7f77b1ffb640 1 -- 172.21.15.42:0/624801176 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f77b401e460 con 0x7f77c00a39b0 2024-09-16T03:56:18.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.562+0000 7f77c4ae8640 1 --2- 172.21.15.42:0/624801176 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f77c009ceb0 0x7f77c01ceff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:56:18.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.562+0000 7f77c4ae8640 10 monclient: get_auth_request con 0x7f77c009ceb0 auth_method 0 2024-09-16T03:56:18.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.562+0000 7f77c4ae8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:56:18.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.562+0000 7f77c4ae8640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:56:18.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.562+0000 7f77c4ae8640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:56:18.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.562+0000 7f77c4ae8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:56:18.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.562+0000 7f77c4ae8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:56:18.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.562+0000 7f77c4ae8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:56:18.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.562+0000 7f77b1ffb640 1 --2- 172.21.15.42:0/624801176 >> v2:172.21.15.154:6812/254845342 conn(0x7f77a4037810 0x7f77a4039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:56:18.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.562+0000 7f77b1ffb640 1 -- 172.21.15.42:0/624801176 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(441..441 src has 330..441) ==== 5769+0+0 (secure 0 0 0) 0x7f77b40495a0 con 0x7f77c00a39b0 2024-09-16T03:56:18.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.562+0000 7f77b1ffb640 10 client.854279.objecter ms_dispatch 0x7f77c0000ba0 osd_map(441..441 src has 330..441) 2024-09-16T03:56:18.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.562+0000 7f77b1ffb640 3 client.854279.objecter handle_osd_map got epochs [441,441] > 0 2024-09-16T03:56:18.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.562+0000 7f77b1ffb640 3 client.854279.objecter handle_osd_map decoding full epoch 441 2024-09-16T03:56:18.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.562+0000 7f77c4ae8640 10 monclient(hunting): handle_auth_done global_id 824694 payload 824 2024-09-16T03:56:18.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.562+0000 7f77c4ae8640 1 -- 172.21.15.42:0/624801176 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f77c0137000 con 0x7f77c009ceb0 2024-09-16T03:56:18.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.562+0000 7f77b1ffb640 20 client.854279.objecter dump_active .. 0 homeless 2024-09-16T03:56:18.588 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.562+0000 7f77b1ffb640 10 client.854279.objecter ms_handle_connect 0x7f77c009ceb0 2024-09-16T03:56:18.588 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.562+0000 7f77c4ae8640 1 --2- 172.21.15.42:0/624801176 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f77c009ceb0 0x7f77c01ceff0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f77bc006cb0 tx=0x7f77bc00f610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:56:18.588 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.562+0000 7f77b1ffb640 10 client.854279.objecter resend_mon_ops 2024-09-16T03:56:18.588 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.562+0000 7f77c5aea640 1 --2- 172.21.15.42:0/624801176 >> v2:172.21.15.154:6812/254845342 conn(0x7f77a4037810 0x7f77a4039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:56:18.588 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.562+0000 7f77c5aea640 10 monclient: get_auth_request con 0x7f77a4037810 auth_method 0 2024-09-16T03:56:18.588 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.562+0000 7f77b1ffb640 1 -- 172.21.15.42:0/624801176 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77bc014440 con 0x7f77c009ceb0 2024-09-16T03:56:18.588 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.562+0000 7f77b1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:56:18.588 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.562+0000 7f77b1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:56:18.588 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.562+0000 7f77b1ffb640 10 monclient: dump: 2024-09-16T03:56:18.588 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:56:18.588 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:56:18.588 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:18.588 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:18.588 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:56:18.589 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:56:18.589 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:56:18.589 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:56:18.589 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:56:18.589 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:56:18.589 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.562+0000 7f77b1ffb640 1 -- 172.21.15.42:0/624801176 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f77bc014440 con 0x7f77c009ceb0 2024-09-16T03:56:18.589 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.562+0000 7f77b1ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:56:18.589 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.562+0000 7f77b1ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:56:18.589 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.562+0000 7f77b1ffb640 1 -- 172.21.15.42:0/624801176 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f77c009ceb0 msgr2=0x7f77c01ceff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:56:18.589 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.562+0000 7f77b1ffb640 1 --2- 172.21.15.42:0/624801176 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f77c009ceb0 0x7f77c01ceff0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f77bc006cb0 tx=0x7f77bc00f610 comp rx=0 tx=0).stop 2024-09-16T03:56:18.589 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.562+0000 7f77b1ffb640 1 -- 172.21.15.42:0/624801176 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f77c009ceb0 msgr2=0x7f77c01ceff0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:56:18.589 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.562+0000 7f77b1ffb640 1 --2- 172.21.15.42:0/624801176 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f77c009ceb0 0x7f77c01ceff0 secure :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f77bc006cb0 tx=0x7f77bc00f610 comp rx=0 tx=0).stop 2024-09-16T03:56:18.589 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.562+0000 7f77b1ffb640 10 client.854279.objecter ms_handle_connect 0x7f77a4037810 2024-09-16T03:56:18.589 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.562+0000 7f77c5aea640 1 --2- 172.21.15.42:0/624801176 >> v2:172.21.15.154:6812/254845342 conn(0x7f77a4037810 0x7f77a4039cd0 secure :-1 s=READY pgs=3650 cs=0 l=1 rev1=1 crypto rx=0x7f77b80ae000 tx=0x7f77b80a7000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:56:18.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.570+0000 7f77c754d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:56:18.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.570+0000 7f77c754d640 1 --2- 172.21.15.42:0/624801176 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f77c01cf530 0x7f77c01cf940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:56:18.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.570+0000 7f77c754d640 10 monclient(hunting): start opening mon connection 2024-09-16T03:56:18.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.570+0000 7f77c754d640 1 -- 172.21.15.42:0/624801176 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f77c01cfe80 con 0x7f77c01cf530 2024-09-16T03:56:18.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.570+0000 7f77c4ae8640 1 --2- 172.21.15.42:0/624801176 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f77c01cf530 0x7f77c01cf940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:56:18.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.570+0000 7f77c4ae8640 10 monclient: get_auth_request con 0x7f77c01cf530 auth_method 0 2024-09-16T03:56:18.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.570+0000 7f77c4ae8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:56:18.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.570+0000 7f77c4ae8640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:56:18.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.570+0000 7f77c4ae8640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:56:18.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.570+0000 7f77c4ae8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:56:18.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.570+0000 7f77c4ae8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:56:18.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.570+0000 7f77c4ae8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:56:18.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.570+0000 7f77c4ae8640 10 monclient(hunting): handle_auth_done global_id 824697 payload 824 2024-09-16T03:56:18.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.570+0000 7f77c4ae8640 1 -- 172.21.15.42:0/624801176 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f77c00a5750 con 0x7f77c01cf530 2024-09-16T03:56:18.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.570+0000 7f77b1ffb640 10 client.854279.objecter ms_handle_connect 0x7f77c01cf530 2024-09-16T03:56:18.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.570+0000 7f77b1ffb640 10 client.854279.objecter resend_mon_ops 2024-09-16T03:56:18.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.570+0000 7f77c4ae8640 1 --2- 172.21.15.42:0/624801176 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f77c01cf530 0x7f77c01cf940 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f77bc01c000 tx=0x7f77bc0046d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:56:18.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.570+0000 7f77b1ffb640 1 -- 172.21.15.42:0/624801176 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77bc004240 con 0x7f77c01cf530 2024-09-16T03:56:18.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.570+0000 7f77b1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:56:18.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.570+0000 7f77b1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:56:18.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.570+0000 7f77b1ffb640 10 monclient: dump: 2024-09-16T03:56:18.593 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:56:18.593 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:56:18.593 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:18.593 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:18.593 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:56:18.593 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:56:18.593 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:56:18.593 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:56:18.593 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:56:18.593 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:56:18.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.570+0000 7f77b1ffb640 1 -- 172.21.15.42:0/624801176 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4882 (secure 0 0 0) 0x7f77bc004220 con 0x7f77c01cf530 2024-09-16T03:56:18.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.570+0000 7f77b1ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:56:18.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.570+0000 7f77b1ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:56:18.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.570+0000 7f77b1ffb640 1 -- 172.21.15.42:0/624801176 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f77c01cf530 msgr2=0x7f77c01cf940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:56:18.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.570+0000 7f77b1ffb640 1 --2- 172.21.15.42:0/624801176 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f77c01cf530 0x7f77c01cf940 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f77bc01c000 tx=0x7f77bc0046d0 comp rx=0 tx=0).stop 2024-09-16T03:56:18.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.570+0000 7f77b1ffb640 1 -- 172.21.15.42:0/624801176 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f77c01cf530 msgr2=0x7f77c01cf940 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:56:18.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.570+0000 7f77b1ffb640 1 --2- 172.21.15.42:0/624801176 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f77c01cf530 0x7f77c01cf940 secure :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f77bc01c000 tx=0x7f77bc0046d0 comp rx=0 tx=0).stop 2024-09-16T03:56:18.594 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:56:18.594 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:56:18.595 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:56:18.595 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "leader", 2024-09-16T03:56:18.595 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 128, 2024-09-16T03:56:18.595 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 268425, 2024-09-16T03:56:18.595 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:56:18.595 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:56:18.595 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T03:56:18.595 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T03:56:18.595 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:56:18.595 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 1, 2024-09-16T03:56:18.595 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:56:18.595 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:56:18.595 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:56:18.595 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:56:18.596 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:56:18.596 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:56:18.596 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:56:18.596 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:56:18.596 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:56:18.596 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:56:18.596 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:56:18.596 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:56:18.596 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:56:18.596 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:56:18.596 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:56:18.596 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:56:18.596 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:56:18.597 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:56:18.597 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:56:18.597 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:56:18.597 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:56:18.597 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:56:18.597 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:56:18.597 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:56:18.597 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:56:18.597 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:56:18.597 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:56:18.597 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:56:18.597 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:56:18.597 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:56:18.597 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:56:18.598 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:56:18.598 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:56:18.598 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:56:18.598 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:56:18.598 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:56:18.598 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:56:18.598 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:56:18.598 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:56:18.598 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:56:18.598 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:56:18.598 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:56:18.598 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:56:18.598 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:56:18.598 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:56:18.599 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:56:18.599 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:56:18.599 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:56:18.599 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:56:18.599 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:56:18.599 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:56:18.599 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:56:18.599 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:56:18.599 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:56:18.599 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:56:18.599 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:56:18.599 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:56:18.599 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:56:18.599 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:56:18.600 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:56:18.600 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:56:18.600 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:56:18.600 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:56:18.600 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:56:18.600 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:56:18.600 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:56:18.600 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:56:18.600 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:56:18.600 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:56:18.600 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:56:18.600 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:56:18.600 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:56:18.601 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:56:18.601 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:56:18.601 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:56:18.601 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:56:18.601 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:56:18.601 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:56:18.601 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:56:18.601 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:56:18.601 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:56:18.601 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:56:18.601 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:56:18.601 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:56:18.601 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:56:18.601 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:56:18.602 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:56:18.602 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:56:18.602 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:56:18.602 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:56:18.602 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:56:18.602 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:56:18.602 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:56:18.602 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:56:18.602 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:56:18.602 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:56:18.602 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:56:18.602 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:56:18.602 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:56:18.602 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:56:18.602 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:56:18.603 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:56:18.603 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:56:18.603 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:56:18.603 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:56:18.603 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:56:18.603 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:56:18.603 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:56:18.603 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:56:18.603 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:56:18.603 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:56:18.603 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:56:18.603 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:56:18.603 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:56:18.603 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:56:18.604 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:56:18.604 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:56:18.604 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:56:18.604 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:56:18.604 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:56:18.604 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:56:18.604 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:56:18.604 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:56:18.604 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:56:18.604 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:56:18.604 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:56:18.604 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:56:18.604 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:56:18.604 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:56:18.605 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:56:18.605 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:56:18.605 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:56:18.605 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:56:18.605 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:56:18.605 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:56:18.605 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:56:18.605 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:56:18.605 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:56:18.605 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:56:18.605 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:56:18.605 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:56:18.605 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T03:56:18.605 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:56:18.606 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:56:18.606 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:56:18.606 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 6 2024-09-16T03:56:18.606 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:56:18.606 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:56:18.606 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:56:18.606 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:56:18.606 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:56:18.606 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:56:18.606 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 7 2024-09-16T03:56:18.606 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:56:18.606 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:56:18.606 INFO:teuthology.orchestra.run.smithi042.stdout: "mgr": [ 2024-09-16T03:56:18.606 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:56:18.607 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:56:18.607 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:56:18.607 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:56:18.607 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:56:18.607 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:56:18.607 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:56:18.607 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:56:18.607 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:56:18.609 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.570+0000 7f77c754d640 10 librados: watch_flush enter 2024-09-16T03:56:18.609 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.570+0000 7f77c754d640 10 librados: watch_flush exit 2024-09-16T03:56:18.609 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.570+0000 7f77c754d640 20 client.854279.objecter shutdown clearing up homeless session... 2024-09-16T03:56:18.609 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.570+0000 7f77c754d640 10 client.854279.objecter successfully canceled tick 2024-09-16T03:56:18.609 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.570+0000 7f77c754d640 1 -- 172.21.15.42:0/624801176 >> v2:172.21.15.154:6812/254845342 conn(0x7f77a4037810 msgr2=0x7f77a4039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:56:18.609 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.570+0000 7f77c754d640 1 --2- 172.21.15.42:0/624801176 >> v2:172.21.15.154:6812/254845342 conn(0x7f77a4037810 0x7f77a4039cd0 secure :-1 s=READY pgs=3650 cs=0 l=1 rev1=1 crypto rx=0x7f77b80ae000 tx=0x7f77b80a7000 comp rx=0 tx=0).stop 2024-09-16T03:56:18.609 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.570+0000 7f77c754d640 10 monclient: shutdown 2024-09-16T03:56:18.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.570+0000 7f77c754d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:56:18.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.570+0000 7f77c754d640 1 -- 172.21.15.42:0/624801176 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f77c00a39b0 msgr2=0x7f77c013b0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:56:18.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.570+0000 7f77c754d640 1 --2- 172.21.15.42:0/624801176 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f77c00a39b0 0x7f77c013b0c0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f77b400d830 tx=0x7f77b400dcf0 comp rx=0 tx=0).stop 2024-09-16T03:56:18.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.570+0000 7f77c5aea640 1 -- 172.21.15.42:0/624801176 reap_dead start 2024-09-16T03:56:18.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.570+0000 7f77c754d640 1 -- 172.21.15.42:0/624801176 shutdown_connections 2024-09-16T03:56:18.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.570+0000 7f77c754d640 1 --2- 172.21.15.42:0/624801176 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f77c00a39b0 0x7f77c013b0c0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:56:18.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.570+0000 7f77c754d640 1 -- 172.21.15.42:0/624801176 >> 172.21.15.42:0/624801176 conn(0x7f77c00a9fb0 msgr2=0x7f77c0131980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:56:18.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.570+0000 7f77c754d640 1 -- 172.21.15.42:0/624801176 shutdown_connections 2024-09-16T03:56:18.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.570+0000 7f77c754d640 1 -- 172.21.15.42:0/624801176 wait complete. 2024-09-16T03:56:18.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.570+0000 7f77c754d640 1 librados: shutdown 2024-09-16T03:56:18.610 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:56:18.611 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T03:56:18.611 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T03:56:18.727 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.722+0000 7f1d731e9640 10 monclient: get_monmap_and_config 2024-09-16T03:56:18.727 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.722+0000 7f1d731e9640 10 monclient: build_initial_monmap 2024-09-16T03:56:18.727 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.722+0000 7f1d731e9640 10 monclient: monmap: 2024-09-16T03:56:18.727 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:56:18.727 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:56:18.727 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:56:18.725761+0000 2024-09-16T03:56:18.727 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:56:18.725761+0000 2024-09-16T03:56:18.728 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:56:18.728 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:56:18.728 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:56:18.728 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:56:18.728 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:56:18.728 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:56:18.728 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:56:18.728 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:56:18.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.722+0000 7f1d731e9640 1 Processor -- start 2024-09-16T03:56:18.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.722+0000 7f1d731e9640 1 -- start start 2024-09-16T03:56:18.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.722+0000 7f1d731e9640 10 monclient: init 2024-09-16T03:56:18.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.722+0000 7f1d731e9640 10 monclient: _reopen_session rank -1 2024-09-16T03:56:18.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.722+0000 7f1d731e9640 10 monclient: _add_conns ranks=[3,2,4,0,1] 2024-09-16T03:56:18.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.722+0000 7f1d731e9640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f1d6c0a1330 0x7f1d6c0a1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:56:18.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.722+0000 7f1d731e9640 10 monclient(hunting): picked mon.noname-a con 0x7f1d6c0a1330 addr v2:172.21.15.42:3300/0 2024-09-16T03:56:18.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.722+0000 7f1d731e9640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1d6c06dac0 0x7f1d6c0a1c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:56:18.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.722+0000 7f1d731e9640 10 monclient(hunting): picked mon.noname-b con 0x7f1d6c06dac0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:56:18.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.722+0000 7f1d731e9640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f1d6c135100 0x7f1d6c135550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:56:18.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.722+0000 7f1d731e9640 10 monclient(hunting): picked mon.noname-c con 0x7f1d6c135100 addr v2:172.21.15.154:3300/0 2024-09-16T03:56:18.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.722+0000 7f1d731e9640 10 monclient(hunting): start opening mon connection 2024-09-16T03:56:18.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.722+0000 7f1d731e9640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f1d6c09c7d0 con 0x7f1d6c0a1330 2024-09-16T03:56:18.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.722+0000 7f1d731e9640 10 monclient(hunting): start opening mon connection 2024-09-16T03:56:18.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.722+0000 7f1d731e9640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f1d6c09c090 con 0x7f1d6c135100 2024-09-16T03:56:18.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.722+0000 7f1d731e9640 10 monclient(hunting): start opening mon connection 2024-09-16T03:56:18.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.722+0000 7f1d731e9640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f1d6c09f610 con 0x7f1d6c06dac0 2024-09-16T03:56:18.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.722+0000 7f1d731e9640 10 monclient(hunting): _renew_subs 2024-09-16T03:56:18.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.722+0000 7f1d731e9640 10 monclient(hunting): authenticate will time out at 1825.695199s 2024-09-16T03:56:18.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.722+0000 7f1d70f85640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f1d6c0a1330 0x7f1d6c0a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:56:18.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.722+0000 7f1d6bfff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1d6c06dac0 0x7f1d6c0a1c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:56:18.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.722+0000 7f1d6bfff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1d6c06dac0 0x7f1d6c0a1c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:40784/0 (socket says 172.21.15.42:40784) 2024-09-16T03:56:18.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.722+0000 7f1d6bfff640 1 -- 172.21.15.42:0/4181708998 learned_addr learned my addr 172.21.15.42:0/4181708998 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:56:18.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.722+0000 7f1d70f85640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f1d6c0a1330 0x7f1d6c0a1710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:37042/0 (socket says 172.21.15.42:37042) 2024-09-16T03:56:18.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.722+0000 7f1d6bfff640 10 monclient(hunting): get_auth_request con 0x7f1d6c06dac0 auth_method 0 2024-09-16T03:56:18.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.722+0000 7f1d6bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:56:18.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.722+0000 7f1d6bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:56:18.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.722+0000 7f1d6bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:56:18.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.722+0000 7f1d70f85640 10 monclient(hunting): get_auth_request con 0x7f1d6c0a1330 auth_method 0 2024-09-16T03:56:18.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.722+0000 7f1d70f85640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:56:18.742 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.722+0000 7f1d70f85640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:56:18.742 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.722+0000 7f1d70f85640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:56:18.742 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.722+0000 7f1d70f85640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:56:18.742 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.722+0000 7f1d70f85640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:56:18.742 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.722+0000 7f1d70f85640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:56:18.742 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.722+0000 7f1d6bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:56:18.742 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.722+0000 7f1d6bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:56:18.742 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.722+0000 7f1d71786640 1 --2- 172.21.15.42:0/4181708998 >> v2:172.21.15.154:3300/0 conn(0x7f1d6c135100 0x7f1d6c135550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:56:18.742 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.722+0000 7f1d6bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:56:18.742 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.722+0000 7f1d71786640 10 monclient(hunting): get_auth_request con 0x7f1d6c135100 auth_method 0 2024-09-16T03:56:18.742 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.722+0000 7f1d71786640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:56:18.742 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.722+0000 7f1d71786640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:56:18.742 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.722+0000 7f1d71786640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:56:18.742 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.722+0000 7f1d70f85640 10 monclient(hunting): handle_auth_done global_id 824700 payload 274 2024-09-16T03:56:18.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.722+0000 7f1d70f85640 1 -- 172.21.15.42:0/4181708998 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1d6c06dac0 msgr2=0x7f1d6c0a1c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:56:18.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.722+0000 7f1d70f85640 1 --2- 172.21.15.42:0/4181708998 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1d6c06dac0 0x7f1d6c0a1c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:56:18.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.722+0000 7f1d70f85640 1 -- 172.21.15.42:0/4181708998 >> v2:172.21.15.154:3300/0 conn(0x7f1d6c135100 msgr2=0x7f1d6c135550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:56:18.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.722+0000 7f1d70f85640 1 --2- 172.21.15.42:0/4181708998 >> v2:172.21.15.154:3300/0 conn(0x7f1d6c135100 0x7f1d6c135550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:56:18.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.722+0000 7f1d70f85640 10 monclient: _finish_hunting 0 2024-09-16T03:56:18.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.722+0000 7f1d70f85640 1 monclient: found mon.noname-a 2024-09-16T03:56:18.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.722+0000 7f1d70f85640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:56:18.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.722+0000 7f1d70f85640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:56:18.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.722+0000 7f1d70f85640 1 -- 172.21.15.42:0/4181708998 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d6c09f8f0 con 0x7f1d6c0a1330 2024-09-16T03:56:18.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.722+0000 7f1d6bfff640 1 --2- 172.21.15.42:0/4181708998 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1d6c06dac0 0x7f1d6c0a1c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:56:18.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.722+0000 7f1d71786640 1 --2- 172.21.15.42:0/4181708998 >> v2:172.21.15.154:3300/0 conn(0x7f1d6c135100 0x7f1d6c135550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T03:56:18.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.726+0000 7f1d70f85640 1 --2- 172.21.15.42:0/4181708998 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1d6c0a1330 0x7f1d6c0a1710 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f1d6c09bb00 tx=0x7f1d6401c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1281ce33c648dc4 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:56:18.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.726+0000 7f1d6b7fe640 1 -- 172.21.15.42:0/4181708998 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d640b2070 con 0x7f1d6c0a1330 2024-09-16T03:56:18.744 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.726+0000 7f1d6b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:56:18.744 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.726+0000 7f1d6b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:56:18.744 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.726+0000 7f1d6b7fe640 10 monclient: dump: 2024-09-16T03:56:18.744 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:56:18.744 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:56:18.744 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:18.744 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:18.744 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:56:18.744 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:56:18.744 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:56:18.744 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:56:18.744 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:56:18.744 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:56:18.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.726+0000 7f1d6b7fe640 10 monclient: _finish_auth 0 2024-09-16T03:56:18.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.726+0000 7f1d6b7fe640 10 monclient: _check_auth_tickets 2024-09-16T03:56:18.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.726+0000 7f1d6b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:56:18.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.726+0000 7f1d6b7fe640 1 -- 172.21.15.42:0/4181708998 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1d640a5840 con 0x7f1d6c0a1330 2024-09-16T03:56:18.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.726+0000 7f1d731e9640 5 monclient: authenticate success, global_id 824700 2024-09-16T03:56:18.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.726+0000 7f1d731e9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:56:18.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.726+0000 7f1d6b7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:56:18.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.726+0000 7f1d731e9640 10 monclient: get_monmap_and_config success 2024-09-16T03:56:18.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.726+0000 7f1d6b7fe640 1 -- 172.21.15.42:0/4181708998 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d640a5c10 con 0x7f1d6c0a1330 2024-09-16T03:56:18.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.726+0000 7f1d6b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:56:18.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.726+0000 7f1d731e9640 4 set_mon_vals no callback set 2024-09-16T03:56:18.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.726+0000 7f1d6b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:56:18.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.726+0000 7f1d6b7fe640 10 monclient: dump: 2024-09-16T03:56:18.746 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:56:18.746 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:56:18.746 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:18.746 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:18.746 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:56:18.746 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:56:18.746 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:56:18.746 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:56:18.746 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:56:18.746 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:56:18.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.726+0000 7f1d731e9640 10 monclient: shutdown 2024-09-16T03:56:18.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.726+0000 7f1d731e9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:56:18.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.726+0000 7f1d731e9640 1 -- 172.21.15.42:0/4181708998 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1d6c0a1330 msgr2=0x7f1d6c0a1710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:56:18.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.726+0000 7f1d731e9640 1 --2- 172.21.15.42:0/4181708998 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1d6c0a1330 0x7f1d6c0a1710 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f1d6c09bb00 tx=0x7f1d6401c910 comp rx=0 tx=0).stop 2024-09-16T03:56:18.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.726+0000 7f1d731e9640 1 -- 172.21.15.42:0/4181708998 shutdown_connections 2024-09-16T03:56:18.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.726+0000 7f1d731e9640 1 --2- 172.21.15.42:0/4181708998 >> v2:172.21.15.154:3300/0 conn(0x7f1d6c135100 0x7f1d6c135550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:56:18.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.726+0000 7f1d731e9640 1 --2- 172.21.15.42:0/4181708998 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1d6c06dac0 0x7f1d6c0a1c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:56:18.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.726+0000 7f1d731e9640 1 --2- 172.21.15.42:0/4181708998 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1d6c0a1330 0x7f1d6c0a1710 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:56:18.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.726+0000 7f1d731e9640 1 -- 172.21.15.42:0/4181708998 >> 172.21.15.42:0/4181708998 conn(0x7f1d6c0a2420 msgr2=0x7f1d6c0a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:56:18.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.726+0000 7f1d731e9640 1 -- 172.21.15.42:0/4181708998 shutdown_connections 2024-09-16T03:56:18.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.726+0000 7f1d731e9640 1 -- 172.21.15.42:0/4181708998 wait complete. 2024-09-16T03:56:18.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.726+0000 7f1d731e9640 10 monclient: build_initial_monmap 2024-09-16T03:56:18.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.726+0000 7f1d731e9640 10 monclient: monmap: 2024-09-16T03:56:18.747 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:56:18.747 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:56:18.747 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:56:18.747 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:56:18.747 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:56:18.748 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:56:18.748 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:56:18.748 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:56:18.748 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:56:18.748 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:56:18.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.726+0000 7f1d731e9640 1 librados: starting msgr at 2024-09-16T03:56:18.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.726+0000 7f1d731e9640 1 librados: starting objecter 2024-09-16T03:56:18.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.726+0000 7f1d731e9640 1 Processor -- start 2024-09-16T03:56:18.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.730+0000 7f1d731e9640 1 -- start start 2024-09-16T03:56:18.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.730+0000 7f1d731e9640 1 librados: setting wanted keys 2024-09-16T03:56:18.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.730+0000 7f1d731e9640 1 librados: calling monclient init 2024-09-16T03:56:18.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.730+0000 7f1d731e9640 10 monclient: init 2024-09-16T03:56:18.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.730+0000 7f1d731e9640 10 monclient: _reopen_session rank -1 2024-09-16T03:56:18.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.730+0000 7f1d731e9640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:56:18.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.730+0000 7f1d731e9640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1d6c0a1330 0x7f1d6c1ccd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:56:18.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.730+0000 7f1d731e9640 10 monclient(hunting): picked mon.noname-c con 0x7f1d6c0a1330 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:56:18.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.730+0000 7f1d731e9640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1d6c135100 0x7f1d6c132330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:56:18.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.730+0000 7f1d731e9640 10 monclient(hunting): picked mon.noname-b con 0x7f1d6c135100 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:56:18.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.730+0000 7f1d731e9640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1d6c132870 0x7f1d6c139350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:56:18.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.730+0000 7f1d731e9640 10 monclient(hunting): picked mon.noname-a con 0x7f1d6c132870 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:56:18.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.730+0000 7f1d731e9640 10 monclient(hunting): start opening mon connection 2024-09-16T03:56:18.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.730+0000 7f1d731e9640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f1d6c09eef0 con 0x7f1d6c132870 2024-09-16T03:56:18.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.730+0000 7f1d731e9640 10 monclient(hunting): start opening mon connection 2024-09-16T03:56:18.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.730+0000 7f1d731e9640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f1d6c09ed70 con 0x7f1d6c135100 2024-09-16T03:56:18.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.730+0000 7f1d731e9640 10 monclient(hunting): start opening mon connection 2024-09-16T03:56:18.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.730+0000 7f1d731e9640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f1d6c13eb70 con 0x7f1d6c0a1330 2024-09-16T03:56:18.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.730+0000 7f1d731e9640 10 monclient(hunting): _renew_subs 2024-09-16T03:56:18.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.730+0000 7f1d731e9640 10 monclient(hunting): authenticate will time out at 1825.700545s 2024-09-16T03:56:18.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.730+0000 7f1d70f85640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1d6c0a1330 0x7f1d6c1ccd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:56:18.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.730+0000 7f1d71786640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1d6c132870 0x7f1d6c139350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:56:18.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.730+0000 7f1d70f85640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1d6c0a1330 0x7f1d6c1ccd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:40786/0 (socket says 172.21.15.42:40786) 2024-09-16T03:56:18.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.730+0000 7f1d70f85640 1 -- 172.21.15.42:0/3634392074 learned_addr learned my addr 172.21.15.42:0/3634392074 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:56:18.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.730+0000 7f1d70f85640 10 monclient(hunting): get_auth_request con 0x7f1d6c0a1330 auth_method 0 2024-09-16T03:56:18.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.730+0000 7f1d70f85640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:56:18.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.730+0000 7f1d70f85640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:56:18.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.730+0000 7f1d70f85640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:56:18.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.730+0000 7f1d71786640 10 monclient(hunting): get_auth_request con 0x7f1d6c132870 auth_method 0 2024-09-16T03:56:18.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.730+0000 7f1d71786640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:56:18.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.730+0000 7f1d71786640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:56:18.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.730+0000 7f1d71786640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:56:18.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.730+0000 7f1d70f85640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:56:18.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.730+0000 7f1d70f85640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:56:18.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.730+0000 7f1d70f85640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:56:18.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.730+0000 7f1d71786640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:56:18.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.730+0000 7f1d71786640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:56:18.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.730+0000 7f1d71786640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:56:18.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.730+0000 7f1d6bfff640 1 --2- 172.21.15.42:0/3634392074 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1d6c135100 0x7f1d6c132330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:56:18.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.730+0000 7f1d71786640 10 monclient(hunting): handle_auth_done global_id 824703 payload 824 2024-09-16T03:56:18.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.730+0000 7f1d71786640 1 -- 172.21.15.42:0/3634392074 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1d6c0a1330 msgr2=0x7f1d6c1ccd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:56:18.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.730+0000 7f1d71786640 1 --2- 172.21.15.42:0/3634392074 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1d6c0a1330 0x7f1d6c1ccd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:56:18.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.730+0000 7f1d71786640 1 -- 172.21.15.42:0/3634392074 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1d6c135100 msgr2=0x7f1d6c132330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:56:18.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.730+0000 7f1d71786640 1 --2- 172.21.15.42:0/3634392074 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1d6c135100 0x7f1d6c132330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:56:18.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.730+0000 7f1d71786640 10 monclient: _finish_hunting 0 2024-09-16T03:56:18.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.730+0000 7f1d71786640 1 monclient: found mon.noname-a 2024-09-16T03:56:18.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.730+0000 7f1d71786640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:56:18.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.730+0000 7f1d71786640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:56:18.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.730+0000 7f1d71786640 1 -- 172.21.15.42:0/3634392074 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d6c13aff0 con 0x7f1d6c132870 2024-09-16T03:56:18.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.730+0000 7f1d6bfff640 10 monclient: get_auth_request con 0x7f1d6c135100 auth_method 0 2024-09-16T03:56:18.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.730+0000 7f1d70f85640 1 --2- 172.21.15.42:0/3634392074 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1d6c0a1330 0x7f1d6c1ccd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:56:18.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.730+0000 7f1d6bfff640 1 --2- 172.21.15.42:0/3634392074 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1d6c135100 0x7f1d6c132330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T03:56:18.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.730+0000 7f1d71786640 1 --2- 172.21.15.42:0/3634392074 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1d6c132870 0x7f1d6c139350 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f1d5c00e630 tx=0x7f1d5c00eaf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:56:18.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.730+0000 7f1d697fa640 10 client.?.objecter ms_handle_connect 0x7f1d6c132870 2024-09-16T03:56:18.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.730+0000 7f1d697fa640 10 client.?.objecter resend_mon_ops 2024-09-16T03:56:18.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.730+0000 7f1d697fa640 1 -- 172.21.15.42:0/3634392074 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d5c017070 con 0x7f1d6c132870 2024-09-16T03:56:18.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.730+0000 7f1d697fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:56:18.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.730+0000 7f1d697fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:56:18.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.730+0000 7f1d697fa640 10 monclient: dump: 2024-09-16T03:56:18.753 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:56:18.753 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:56:18.753 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:18.754 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:18.754 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:56:18.754 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:56:18.754 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:56:18.754 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:56:18.754 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:56:18.754 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:56:18.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.730+0000 7f1d697fa640 10 monclient: _finish_auth 0 2024-09-16T03:56:18.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.730+0000 7f1d697fa640 10 monclient: _check_auth_tickets 2024-09-16T03:56:18.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.730+0000 7f1d697fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:56:18.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.730+0000 7f1d697fa640 1 -- 172.21.15.42:0/3634392074 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1d5c00c620 con 0x7f1d6c132870 2024-09-16T03:56:18.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.730+0000 7f1d731e9640 5 monclient: authenticate success, global_id 824703 2024-09-16T03:56:18.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.730+0000 7f1d731e9640 10 monclient: _renew_subs 2024-09-16T03:56:18.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.730+0000 7f1d731e9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:56:18.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.730+0000 7f1d731e9640 1 -- 172.21.15.42:0/3634392074 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d6c071af0 con 0x7f1d6c132870 2024-09-16T03:56:18.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.730+0000 7f1d697fa640 10 monclient: handle_config config(0 keys) 2024-09-16T03:56:18.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.730+0000 7f1d697fa640 1 -- 172.21.15.42:0/3634392074 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d5c00f040 con 0x7f1d6c132870 2024-09-16T03:56:18.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.730+0000 7f1d697fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:56:18.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.730+0000 7f1d697fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:56:18.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.730+0000 7f1d697fa640 10 monclient: dump: 2024-09-16T03:56:18.755 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:56:18.755 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:56:18.755 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:18.755 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:18.755 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:56:18.755 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:56:18.755 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:56:18.755 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:56:18.756 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:56:18.756 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:56:18.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.730+0000 7f1d731e9640 10 client.824703.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:56:18.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.730+0000 7f1d731e9640 10 monclient: _renew_subs 2024-09-16T03:56:18.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.730+0000 7f1d731e9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:56:18.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.730+0000 7f1d731e9640 1 -- 172.21.15.42:0/3634392074 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d6c11d430 con 0x7f1d6c132870 2024-09-16T03:56:18.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.730+0000 7f1d731e9640 1 librados: init done 2024-09-16T03:56:18.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.730+0000 7f1d6affd640 4 set_mon_vals no callback set 2024-09-16T03:56:18.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.730+0000 7f1d731e9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:56:18.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.730+0000 7f1d731e9640 1 --2- 172.21.15.42:0/3634392074 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1d6c06dac0 0x7f1d6c02c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:56:18.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.730+0000 7f1d731e9640 10 monclient(hunting): start opening mon connection 2024-09-16T03:56:18.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.730+0000 7f1d731e9640 1 -- 172.21.15.42:0/3634392074 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f1d6c11d770 con 0x7f1d6c06dac0 2024-09-16T03:56:18.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.730+0000 7f1d697fa640 1 -- 172.21.15.42:0/3634392074 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f1d5c01a070 con 0x7f1d6c132870 2024-09-16T03:56:18.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.730+0000 7f1d70f85640 1 --2- 172.21.15.42:0/3634392074 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1d6c06dac0 0x7f1d6c02c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:56:18.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.730+0000 7f1d70f85640 10 monclient: get_auth_request con 0x7f1d6c06dac0 auth_method 0 2024-09-16T03:56:18.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.730+0000 7f1d70f85640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:56:18.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.730+0000 7f1d70f85640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:56:18.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.730+0000 7f1d70f85640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:56:18.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.730+0000 7f1d70f85640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:56:18.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.730+0000 7f1d70f85640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:56:18.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.730+0000 7f1d70f85640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:56:18.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.730+0000 7f1d697fa640 1 --2- 172.21.15.42:0/3634392074 >> v2:172.21.15.154:6812/254845342 conn(0x7f1d4c037810 0x7f1d4c039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:56:18.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.730+0000 7f1d70f85640 10 monclient(hunting): handle_auth_done global_id 854288 payload 824 2024-09-16T03:56:18.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.730+0000 7f1d70f85640 1 -- 172.21.15.42:0/3634392074 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1d6c135550 con 0x7f1d6c06dac0 2024-09-16T03:56:18.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.730+0000 7f1d697fa640 1 -- 172.21.15.42:0/3634392074 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(441..441 src has 330..441) ==== 5769+0+0 (secure 0 0 0) 0x7f1d5c048690 con 0x7f1d6c132870 2024-09-16T03:56:18.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.730+0000 7f1d697fa640 10 client.824703.objecter ms_dispatch 0x7f1d6c000ba0 osd_map(441..441 src has 330..441) 2024-09-16T03:56:18.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.730+0000 7f1d697fa640 3 client.824703.objecter handle_osd_map got epochs [441,441] > 0 2024-09-16T03:56:18.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.730+0000 7f1d697fa640 3 client.824703.objecter handle_osd_map decoding full epoch 441 2024-09-16T03:56:18.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.730+0000 7f1d697fa640 20 client.824703.objecter dump_active .. 0 homeless 2024-09-16T03:56:18.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.730+0000 7f1d70f85640 1 --2- 172.21.15.42:0/3634392074 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1d6c06dac0 0x7f1d6c02c650 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f1d6401afc0 tx=0x7f1d6c0a22d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:56:18.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.730+0000 7f1d697fa640 10 client.824703.objecter ms_handle_connect 0x7f1d6c06dac0 2024-09-16T03:56:18.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.730+0000 7f1d697fa640 10 client.824703.objecter resend_mon_ops 2024-09-16T03:56:18.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.730+0000 7f1d6bfff640 1 --2- 172.21.15.42:0/3634392074 >> v2:172.21.15.154:6812/254845342 conn(0x7f1d4c037810 0x7f1d4c039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:56:18.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.730+0000 7f1d697fa640 1 -- 172.21.15.42:0/3634392074 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d640be070 con 0x7f1d6c06dac0 2024-09-16T03:56:18.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.730+0000 7f1d697fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:56:18.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.730+0000 7f1d697fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:56:18.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.730+0000 7f1d697fa640 10 monclient: dump: 2024-09-16T03:56:18.758 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:56:18.758 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:56:18.758 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:18.758 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:18.758 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:56:18.759 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:56:18.759 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:56:18.759 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:56:18.759 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:56:18.759 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:56:18.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.730+0000 7f1d6bfff640 10 monclient: get_auth_request con 0x7f1d4c037810 auth_method 0 2024-09-16T03:56:18.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.730+0000 7f1d697fa640 1 -- 172.21.15.42:0/3634392074 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f1d640b7640 con 0x7f1d6c06dac0 2024-09-16T03:56:18.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.730+0000 7f1d697fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:56:18.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.730+0000 7f1d697fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:56:18.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.730+0000 7f1d697fa640 1 -- 172.21.15.42:0/3634392074 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1d6c06dac0 msgr2=0x7f1d6c02c650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:56:18.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.730+0000 7f1d697fa640 1 --2- 172.21.15.42:0/3634392074 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1d6c06dac0 0x7f1d6c02c650 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f1d6401afc0 tx=0x7f1d6c0a22d0 comp rx=0 tx=0).stop 2024-09-16T03:56:18.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.730+0000 7f1d697fa640 1 -- 172.21.15.42:0/3634392074 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1d6c06dac0 msgr2=0x7f1d6c02c650 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:56:18.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.730+0000 7f1d697fa640 1 --2- 172.21.15.42:0/3634392074 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1d6c06dac0 0x7f1d6c02c650 secure :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f1d6401afc0 tx=0x7f1d6c0a22d0 comp rx=0 tx=0).stop 2024-09-16T03:56:18.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.730+0000 7f1d697fa640 10 client.824703.objecter ms_handle_connect 0x7f1d4c037810 2024-09-16T03:56:18.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.730+0000 7f1d6bfff640 1 --2- 172.21.15.42:0/3634392074 >> v2:172.21.15.154:6812/254845342 conn(0x7f1d4c037810 0x7f1d4c039cd0 secure :-1 s=READY pgs=3651 cs=0 l=1 rev1=1 crypto rx=0x7f1d6c13abf0 tx=0x7f1d54008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:56:18.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.738+0000 7f1d731e9640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:56:18.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.738+0000 7f1d731e9640 1 --2- 172.21.15.42:0/3634392074 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1d6c02cb90 0x7f1d6c02cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:56:18.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.738+0000 7f1d731e9640 10 monclient(hunting): start opening mon connection 2024-09-16T03:56:18.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.738+0000 7f1d731e9640 1 -- 172.21.15.42:0/3634392074 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f1d6c134a00 con 0x7f1d6c02cb90 2024-09-16T03:56:18.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.738+0000 7f1d70f85640 1 --2- 172.21.15.42:0/3634392074 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1d6c02cb90 0x7f1d6c02cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:56:18.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.738+0000 7f1d70f85640 10 monclient: get_auth_request con 0x7f1d6c02cb90 auth_method 0 2024-09-16T03:56:18.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.738+0000 7f1d70f85640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:56:18.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.738+0000 7f1d70f85640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:56:18.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.738+0000 7f1d70f85640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:56:18.763 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.738+0000 7f1d70f85640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:56:18.763 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.738+0000 7f1d70f85640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:56:18.763 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.738+0000 7f1d70f85640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:56:18.763 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.738+0000 7f1d70f85640 10 monclient(hunting): handle_auth_done global_id 854291 payload 824 2024-09-16T03:56:18.763 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.738+0000 7f1d70f85640 1 -- 172.21.15.42:0/3634392074 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f1d6c1cd2d0 con 0x7f1d6c02cb90 2024-09-16T03:56:18.763 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.738+0000 7f1d697fa640 10 client.824703.objecter ms_handle_connect 0x7f1d6c02cb90 2024-09-16T03:56:18.763 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.738+0000 7f1d697fa640 10 client.824703.objecter resend_mon_ops 2024-09-16T03:56:18.763 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.738+0000 7f1d70f85640 1 --2- 172.21.15.42:0/3634392074 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1d6c02cb90 0x7f1d6c02cfa0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f1d6c0744e0 tx=0x7f1d640a7070 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:56:18.763 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.738+0000 7f1d697fa640 1 -- 172.21.15.42:0/3634392074 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d640b2040 con 0x7f1d6c02cb90 2024-09-16T03:56:18.763 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.738+0000 7f1d697fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:56:18.763 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.738+0000 7f1d697fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:56:18.763 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.738+0000 7f1d697fa640 10 monclient: dump: 2024-09-16T03:56:18.763 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:56:18.763 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:56:18.764 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:18.764 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:18.764 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:56:18.764 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:56:18.764 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:56:18.764 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:56:18.764 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:56:18.764 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:56:18.764 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.738+0000 7f1d697fa640 1 -- 172.21.15.42:0/3634392074 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4546 (secure 0 0 0) 0x7f1d64004450 con 0x7f1d6c02cb90 2024-09-16T03:56:18.764 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.738+0000 7f1d697fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:56:18.764 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.738+0000 7f1d697fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:56:18.764 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.738+0000 7f1d697fa640 1 -- 172.21.15.42:0/3634392074 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1d6c02cb90 msgr2=0x7f1d6c02cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:56:18.764 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.738+0000 7f1d697fa640 1 --2- 172.21.15.42:0/3634392074 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1d6c02cb90 0x7f1d6c02cfa0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f1d6c0744e0 tx=0x7f1d640a7070 comp rx=0 tx=0).stop 2024-09-16T03:56:18.765 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.738+0000 7f1d697fa640 1 -- 172.21.15.42:0/3634392074 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1d6c02cb90 msgr2=0x7f1d6c02cfa0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:56:18.765 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.738+0000 7f1d697fa640 1 --2- 172.21.15.42:0/3634392074 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1d6c02cb90 0x7f1d6c02cfa0 secure :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f1d6c0744e0 tx=0x7f1d640a7070 comp rx=0 tx=0).stop 2024-09-16T03:56:18.765 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:56:18.765 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:56:18.765 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:56:18.765 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T03:56:18.765 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 128, 2024-09-16T03:56:18.765 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 112174, 2024-09-16T03:56:18.765 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:56:18.765 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:56:18.766 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T03:56:18.766 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T03:56:18.766 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:56:18.766 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 1, 2024-09-16T03:56:18.766 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:56:18.766 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:56:18.766 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:56:18.766 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:56:18.766 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:56:18.766 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:56:18.766 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:56:18.766 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:56:18.766 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:56:18.766 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:56:18.767 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:56:18.767 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:56:18.767 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:56:18.767 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:56:18.767 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:56:18.767 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:56:18.767 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:56:18.767 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:56:18.767 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:56:18.767 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:56:18.767 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:56:18.767 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:56:18.767 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:56:18.767 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:56:18.768 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:56:18.768 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:56:18.768 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:56:18.768 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:56:18.768 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:56:18.768 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:56:18.768 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:56:18.768 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:56:18.768 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:56:18.768 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:56:18.768 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:56:18.768 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:56:18.768 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:56:18.769 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:56:18.769 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:56:18.769 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:56:18.769 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:56:18.769 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:56:18.769 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:56:18.769 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:56:18.769 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:56:18.769 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:56:18.769 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:56:18.769 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:56:18.769 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:56:18.769 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:56:18.770 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:56:18.770 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:56:18.770 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:56:18.770 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:56:18.770 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:56:18.770 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:56:18.770 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:56:18.770 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:56:18.770 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:56:18.770 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:56:18.770 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:56:18.770 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:56:18.771 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:56:18.771 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:56:18.771 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:56:18.771 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:56:18.771 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:56:18.771 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:56:18.771 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:56:18.771 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:56:18.771 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:56:18.771 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:56:18.771 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:56:18.771 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:56:18.771 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:56:18.771 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:56:18.772 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:56:18.772 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:56:18.772 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:56:18.772 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:56:18.772 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:56:18.772 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:56:18.772 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:56:18.772 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:56:18.772 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:56:18.772 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:56:18.773 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:56:18.773 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:56:18.773 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:56:18.773 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:56:18.773 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:56:18.773 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:56:18.773 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:56:18.773 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:56:18.773 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:56:18.773 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:56:18.773 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:56:18.773 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:56:18.773 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:56:18.773 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:56:18.774 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:56:18.774 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:56:18.774 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:56:18.774 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:56:18.774 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:56:18.774 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:56:18.774 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:56:18.774 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:56:18.774 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:56:18.774 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:56:18.774 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:56:18.774 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:56:18.774 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:56:18.774 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:56:18.775 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:56:18.775 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:56:18.775 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:56:18.775 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:56:18.775 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:56:18.775 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:56:18.775 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:56:18.775 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:56:18.775 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:56:18.775 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:56:18.775 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:56:18.775 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:56:18.775 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:56:18.775 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:56:18.776 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:56:18.776 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:56:18.776 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:56:18.776 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:56:18.776 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:56:18.776 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:56:18.776 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:56:18.776 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:56:18.776 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:56:18.776 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:56:18.776 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:56:18.776 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:56:18.776 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:56:18.777 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:56:18.777 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:56:18.777 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:56:18.777 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:56:18.777 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:56:18.777 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:56:18.777 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:56:18.777 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:56:18.777 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:56:18.777 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:56:18.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.738+0000 7f1d731e9640 10 librados: watch_flush enter 2024-09-16T03:56:18.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.738+0000 7f1d731e9640 10 librados: watch_flush exit 2024-09-16T03:56:18.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.738+0000 7f1d731e9640 20 client.824703.objecter shutdown clearing up homeless session... 2024-09-16T03:56:18.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.738+0000 7f1d731e9640 10 client.824703.objecter successfully canceled tick 2024-09-16T03:56:18.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.738+0000 7f1d731e9640 1 -- 172.21.15.42:0/3634392074 >> v2:172.21.15.154:6812/254845342 conn(0x7f1d4c037810 msgr2=0x7f1d4c039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:56:18.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.738+0000 7f1d731e9640 1 --2- 172.21.15.42:0/3634392074 >> v2:172.21.15.154:6812/254845342 conn(0x7f1d4c037810 0x7f1d4c039cd0 secure :-1 s=READY pgs=3651 cs=0 l=1 rev1=1 crypto rx=0x7f1d6c13abf0 tx=0x7f1d54008040 comp rx=0 tx=0).stop 2024-09-16T03:56:18.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.738+0000 7f1d731e9640 10 monclient: shutdown 2024-09-16T03:56:18.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.738+0000 7f1d731e9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:56:18.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.738+0000 7f1d731e9640 1 -- 172.21.15.42:0/3634392074 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1d6c132870 msgr2=0x7f1d6c139350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:56:18.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.738+0000 7f1d731e9640 1 --2- 172.21.15.42:0/3634392074 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1d6c132870 0x7f1d6c139350 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f1d5c00e630 tx=0x7f1d5c00eaf0 comp rx=0 tx=0).stop 2024-09-16T03:56:18.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.738+0000 7f1d71786640 1 -- 172.21.15.42:0/3634392074 reap_dead start 2024-09-16T03:56:18.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.738+0000 7f1d731e9640 1 -- 172.21.15.42:0/3634392074 shutdown_connections 2024-09-16T03:56:18.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.738+0000 7f1d731e9640 1 --2- 172.21.15.42:0/3634392074 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1d6c132870 0x7f1d6c139350 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:56:18.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.738+0000 7f1d731e9640 1 -- 172.21.15.42:0/3634392074 >> 172.21.15.42:0/3634392074 conn(0x7f1d6c0a2420 msgr2=0x7f1d6c13bc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:56:18.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.738+0000 7f1d731e9640 1 -- 172.21.15.42:0/3634392074 shutdown_connections 2024-09-16T03:56:18.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.738+0000 7f1d731e9640 1 -- 172.21.15.42:0/3634392074 wait complete. 2024-09-16T03:56:18.782 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.738+0000 7f1d731e9640 1 librados: shutdown 2024-09-16T03:56:18.782 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:56:18.782 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T03:56:18.782 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T03:56:18.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.898+0000 7f9aa560e640 10 monclient: get_monmap_and_config 2024-09-16T03:56:18.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.898+0000 7f9aa560e640 10 monclient: build_initial_monmap 2024-09-16T03:56:18.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.898+0000 7f9aa560e640 10 monclient: monmap: 2024-09-16T03:56:18.902 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:56:18.902 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:56:18.902 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:56:18.900492+0000 2024-09-16T03:56:18.902 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:56:18.900492+0000 2024-09-16T03:56:18.902 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:56:18.902 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:56:18.902 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:56:18.902 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:56:18.902 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:56:18.902 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:56:18.903 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:56:18.903 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:56:18.916 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.898+0000 7f9aa560e640 1 Processor -- start 2024-09-16T03:56:18.916 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.898+0000 7f9aa560e640 1 -- start start 2024-09-16T03:56:18.916 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.898+0000 7f9aa560e640 10 monclient: init 2024-09-16T03:56:18.916 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.898+0000 7f9aa560e640 10 monclient: _reopen_session rank -1 2024-09-16T03:56:18.916 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.898+0000 7f9aa560e640 10 monclient: _add_conns ranks=[2,3,4,1,0] 2024-09-16T03:56:18.917 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.898+0000 7f9aa560e640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9aa00a39b0 0x7f9aa0133b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:56:18.917 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.898+0000 7f9aa560e640 10 monclient(hunting): picked mon.noname-b con 0x7f9aa00a39b0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:56:18.917 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.898+0000 7f9aa560e640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f9aa00a3d90 0x7f9aa0137000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:56:18.917 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.898+0000 7f9aa560e640 10 monclient(hunting): picked mon.noname-a con 0x7f9aa00a3d90 addr v2:172.21.15.42:3300/0 2024-09-16T03:56:18.917 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.898+0000 7f9aa560e640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f9aa0137540 0x7f9aa0137950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:56:18.917 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.898+0000 7f9aa560e640 10 monclient(hunting): picked mon.noname-c con 0x7f9aa0137540 addr v2:172.21.15.154:3300/0 2024-09-16T03:56:18.917 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.898+0000 7f9aa560e640 10 monclient(hunting): start opening mon connection 2024-09-16T03:56:18.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.898+0000 7f9aa560e640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f9aa0137e90 con 0x7f9aa00a3d90 2024-09-16T03:56:18.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.898+0000 7f9aa560e640 10 monclient(hunting): start opening mon connection 2024-09-16T03:56:18.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.898+0000 7f9aa560e640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f9aa009c140 con 0x7f9aa0137540 2024-09-16T03:56:18.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.898+0000 7f9aa560e640 10 monclient(hunting): start opening mon connection 2024-09-16T03:56:18.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.898+0000 7f9aa560e640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f9aa009c420 con 0x7f9aa00a39b0 2024-09-16T03:56:18.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.898+0000 7f9aa560e640 10 monclient(hunting): _renew_subs 2024-09-16T03:56:18.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.898+0000 7f9aa560e640 10 monclient(hunting): authenticate will time out at 1825.869877s 2024-09-16T03:56:18.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.898+0000 7f9a9effd640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9aa00a39b0 0x7f9aa0133b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:56:18.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.898+0000 7f9a9effd640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9aa00a39b0 0x7f9aa0133b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:40798/0 (socket says 172.21.15.42:40798) 2024-09-16T03:56:18.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.898+0000 7f9a9effd640 1 -- 172.21.15.42:0/2162664421 learned_addr learned my addr 172.21.15.42:0/2162664421 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:56:18.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.898+0000 7f9a9effd640 10 monclient(hunting): get_auth_request con 0x7f9aa00a39b0 auth_method 0 2024-09-16T03:56:18.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.898+0000 7f9a9effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:56:18.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.898+0000 7f9a9effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:56:18.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.898+0000 7f9a9effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:56:18.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.898+0000 7f9a9effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:56:18.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.898+0000 7f9a9effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:56:18.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.898+0000 7f9a9effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:56:18.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.898+0000 7f9a9e7fc640 1 --2- 172.21.15.42:0/2162664421 >> v2:172.21.15.42:3300/0 conn(0x7f9aa00a3d90 0x7f9aa0137000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:56:18.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.898+0000 7f9a9e7fc640 10 monclient(hunting): get_auth_request con 0x7f9aa00a3d90 auth_method 0 2024-09-16T03:56:18.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.898+0000 7f9a9e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:56:18.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.898+0000 7f9a9e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:56:18.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.898+0000 7f9a9e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:56:18.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.898+0000 7f9a9effd640 10 monclient(hunting): handle_auth_done global_id 854294 payload 274 2024-09-16T03:56:18.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.898+0000 7f9a9f7fe640 1 --2- 172.21.15.42:0/2162664421 >> v2:172.21.15.154:3300/0 conn(0x7f9aa0137540 0x7f9aa0137950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:56:18.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.898+0000 7f9a9effd640 1 -- 172.21.15.42:0/2162664421 >> v2:172.21.15.154:3300/0 conn(0x7f9aa0137540 msgr2=0x7f9aa0137950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:56:18.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.898+0000 7f9a9effd640 1 --2- 172.21.15.42:0/2162664421 >> v2:172.21.15.154:3300/0 conn(0x7f9aa0137540 0x7f9aa0137950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:56:18.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.898+0000 7f9a9effd640 1 -- 172.21.15.42:0/2162664421 >> v2:172.21.15.42:3300/0 conn(0x7f9aa00a3d90 msgr2=0x7f9aa0137000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:56:18.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.898+0000 7f9a9effd640 1 --2- 172.21.15.42:0/2162664421 >> v2:172.21.15.42:3300/0 conn(0x7f9aa00a3d90 0x7f9aa0137000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:56:18.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.898+0000 7f9a9effd640 10 monclient: _finish_hunting 0 2024-09-16T03:56:18.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.898+0000 7f9a9effd640 1 monclient: found mon.noname-b 2024-09-16T03:56:18.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.898+0000 7f9a9effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:56:18.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.898+0000 7f9a9effd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T03:56:18.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.898+0000 7f9a9effd640 1 -- 172.21.15.42:0/2162664421 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9aa006dd60 con 0x7f9aa00a39b0 2024-09-16T03:56:18.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.898+0000 7f9a9e7fc640 1 --2- 172.21.15.42:0/2162664421 >> v2:172.21.15.42:3300/0 conn(0x7f9aa00a3d90 0x7f9aa0137000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T03:56:18.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.898+0000 7f9a9f7fe640 10 monclient: get_auth_request con 0x7f9aa0137540 auth_method 0 2024-09-16T03:56:18.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.898+0000 7f9a9f7fe640 1 --2- 172.21.15.42:0/2162664421 >> v2:172.21.15.154:3300/0 conn(0x7f9aa0137540 0x7f9aa0137950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T03:56:18.923 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.902+0000 7f9a9effd640 1 --2- 172.21.15.42:0/2162664421 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9aa00a39b0 0x7f9aa0133b80 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f9aa00a56c0 tx=0x7f9a9801c8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a7f3923ab03eede0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:56:18.923 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.902+0000 7f9a9dffb640 1 -- 172.21.15.42:0/2162664421 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a980a5530 con 0x7f9aa00a39b0 2024-09-16T03:56:18.923 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.902+0000 7f9a9dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:56:18.923 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.902+0000 7f9a9dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:56:18.923 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.902+0000 7f9a9dffb640 10 monclient: dump: 2024-09-16T03:56:18.923 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:56:18.923 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:56:18.923 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:18.923 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:18.923 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:56:18.923 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:56:18.923 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:56:18.923 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:56:18.923 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:56:18.924 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:56:18.924 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.902+0000 7f9a9dffb640 10 monclient: _finish_auth 0 2024-09-16T03:56:18.924 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.902+0000 7f9a9dffb640 10 monclient: _check_auth_tickets 2024-09-16T03:56:18.924 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.902+0000 7f9a9dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:56:18.924 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.902+0000 7f9a9dffb640 1 -- 172.21.15.42:0/2162664421 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a980a56d0 con 0x7f9aa00a39b0 2024-09-16T03:56:18.924 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.902+0000 7f9a9dffb640 10 monclient: handle_config config(0 keys) 2024-09-16T03:56:18.924 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.902+0000 7f9a9dffb640 1 -- 172.21.15.42:0/2162664421 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a980a5aa0 con 0x7f9aa00a39b0 2024-09-16T03:56:18.924 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.902+0000 7f9aa560e640 5 monclient: authenticate success, global_id 854294 2024-09-16T03:56:18.924 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.902+0000 7f9aa560e640 10 monclient: get_monmap_and_config success 2024-09-16T03:56:18.924 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.902+0000 7f9aa560e640 4 set_mon_vals no callback set 2024-09-16T03:56:18.924 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.902+0000 7f9a9dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:56:18.924 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.902+0000 7f9a9dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:56:18.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.902+0000 7f9a9dffb640 10 monclient: dump: 2024-09-16T03:56:18.925 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:56:18.925 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:56:18.925 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:18.925 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:18.925 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:56:18.925 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:56:18.925 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:56:18.925 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:56:18.925 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:56:18.925 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:56:18.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.902+0000 7f9aa560e640 10 monclient: shutdown 2024-09-16T03:56:18.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.902+0000 7f9aa560e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:56:18.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.902+0000 7f9aa560e640 1 -- 172.21.15.42:0/2162664421 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9aa00a39b0 msgr2=0x7f9aa0133b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:56:18.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.902+0000 7f9aa560e640 1 --2- 172.21.15.42:0/2162664421 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9aa00a39b0 0x7f9aa0133b80 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f9aa00a56c0 tx=0x7f9a9801c8b0 comp rx=0 tx=0).stop 2024-09-16T03:56:18.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.902+0000 7f9aa560e640 1 -- 172.21.15.42:0/2162664421 shutdown_connections 2024-09-16T03:56:18.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.902+0000 7f9aa560e640 1 --2- 172.21.15.42:0/2162664421 >> v2:172.21.15.154:3300/0 conn(0x7f9aa0137540 0x7f9aa0137950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:56:18.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.902+0000 7f9aa560e640 1 --2- 172.21.15.42:0/2162664421 >> v2:172.21.15.42:3300/0 conn(0x7f9aa00a3d90 0x7f9aa0137000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:56:18.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.902+0000 7f9aa560e640 1 --2- 172.21.15.42:0/2162664421 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9aa00a39b0 0x7f9aa0133b80 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:56:18.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.902+0000 7f9aa560e640 1 -- 172.21.15.42:0/2162664421 >> 172.21.15.42:0/2162664421 conn(0x7f9aa00a9fb0 msgr2=0x7f9aa00aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:56:18.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.902+0000 7f9aa560e640 1 -- 172.21.15.42:0/2162664421 shutdown_connections 2024-09-16T03:56:18.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.902+0000 7f9aa560e640 1 -- 172.21.15.42:0/2162664421 wait complete. 2024-09-16T03:56:18.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.902+0000 7f9aa560e640 10 monclient: build_initial_monmap 2024-09-16T03:56:18.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.902+0000 7f9aa560e640 10 monclient: monmap: 2024-09-16T03:56:18.926 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:56:18.926 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:56:18.926 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:56:18.927 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:56:18.927 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:56:18.927 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:56:18.927 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:56:18.927 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:56:18.927 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:56:18.927 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:56:18.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.902+0000 7f9aa560e640 1 librados: starting msgr at 2024-09-16T03:56:18.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.902+0000 7f9aa560e640 1 librados: starting objecter 2024-09-16T03:56:18.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.902+0000 7f9aa560e640 1 Processor -- start 2024-09-16T03:56:18.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.902+0000 7f9aa560e640 1 -- start start 2024-09-16T03:56:18.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.902+0000 7f9aa560e640 1 librados: setting wanted keys 2024-09-16T03:56:18.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.902+0000 7f9aa560e640 1 librados: calling monclient init 2024-09-16T03:56:18.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.902+0000 7f9aa560e640 10 monclient: init 2024-09-16T03:56:18.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.902+0000 7f9aa560e640 10 monclient: _reopen_session rank -1 2024-09-16T03:56:18.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.902+0000 7f9aa560e640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:56:18.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.902+0000 7f9aa560e640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9aa00a39b0 0x7f9aa01cf540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:56:18.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.902+0000 7f9aa560e640 10 monclient(hunting): picked mon.noname-c con 0x7f9aa00a39b0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:56:18.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.902+0000 7f9aa560e640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9aa00a3d90 0x7f9aa00a2ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:56:18.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.902+0000 7f9aa560e640 10 monclient(hunting): picked mon.noname-a con 0x7f9aa00a3d90 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:56:18.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.902+0000 7f9aa560e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9aa0137540 0x7f9aa00a3400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:56:18.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.902+0000 7f9aa560e640 10 monclient(hunting): picked mon.noname-b con 0x7f9aa0137540 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:56:18.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.902+0000 7f9aa560e640 10 monclient(hunting): start opening mon connection 2024-09-16T03:56:18.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.902+0000 7f9aa560e640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f9aa0140dc0 con 0x7f9aa00a3d90 2024-09-16T03:56:18.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.902+0000 7f9aa560e640 10 monclient(hunting): start opening mon connection 2024-09-16T03:56:18.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.902+0000 7f9aa560e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f9aa0140f40 con 0x7f9aa0137540 2024-09-16T03:56:18.929 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.902+0000 7f9aa560e640 10 monclient(hunting): start opening mon connection 2024-09-16T03:56:18.929 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.902+0000 7f9aa560e640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f9aa01410c0 con 0x7f9aa00a39b0 2024-09-16T03:56:18.929 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.902+0000 7f9aa560e640 10 monclient(hunting): _renew_subs 2024-09-16T03:56:18.929 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.902+0000 7f9aa560e640 10 monclient(hunting): authenticate will time out at 1825.875091s 2024-09-16T03:56:18.929 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.902+0000 7f9a9effd640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9aa00a39b0 0x7f9aa01cf540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:56:18.929 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.902+0000 7f9a9e7fc640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9aa00a3d90 0x7f9aa00a2ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:56:18.929 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.902+0000 7f9a9effd640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9aa00a39b0 0x7f9aa01cf540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:40800/0 (socket says 172.21.15.42:40800) 2024-09-16T03:56:18.929 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.902+0000 7f9a9effd640 1 -- 172.21.15.42:0/363307151 learned_addr learned my addr 172.21.15.42:0/363307151 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:56:18.929 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.902+0000 7f9a9effd640 10 monclient(hunting): get_auth_request con 0x7f9aa00a39b0 auth_method 0 2024-09-16T03:56:18.929 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.902+0000 7f9a9effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:56:18.929 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.902+0000 7f9a9effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:56:18.929 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.902+0000 7f9a9effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:56:18.929 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.902+0000 7f9a9e7fc640 10 monclient(hunting): get_auth_request con 0x7f9aa00a3d90 auth_method 0 2024-09-16T03:56:18.930 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.902+0000 7f9a9e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:56:18.930 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.902+0000 7f9a9e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:56:18.930 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.902+0000 7f9a9e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:56:18.930 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.902+0000 7f9a9effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:56:18.930 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.902+0000 7f9a9effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:56:18.930 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.902+0000 7f9a9effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:56:18.930 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.902+0000 7f9a9e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:56:18.930 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.902+0000 7f9a9e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:56:18.930 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.902+0000 7f9a9e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:56:18.930 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.902+0000 7f9a9effd640 10 monclient(hunting): handle_auth_done global_id 854297 payload 824 2024-09-16T03:56:18.930 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.902+0000 7f9a9f7fe640 1 --2- 172.21.15.42:0/363307151 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9aa0137540 0x7f9aa00a3400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:56:18.930 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.902+0000 7f9a9effd640 1 -- 172.21.15.42:0/363307151 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9aa0137540 msgr2=0x7f9aa00a3400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:56:18.930 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.902+0000 7f9a9effd640 1 --2- 172.21.15.42:0/363307151 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9aa0137540 0x7f9aa00a3400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:56:18.930 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.902+0000 7f9a9effd640 1 -- 172.21.15.42:0/363307151 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9aa00a3d90 msgr2=0x7f9aa00a2ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:56:18.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.902+0000 7f9a9effd640 1 --2- 172.21.15.42:0/363307151 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9aa00a3d90 0x7f9aa00a2ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:56:18.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.902+0000 7f9a9effd640 10 monclient: _finish_hunting 0 2024-09-16T03:56:18.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.902+0000 7f9a9effd640 1 monclient: found mon.noname-c 2024-09-16T03:56:18.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.902+0000 7f9a9effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:56:18.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.902+0000 7f9a9effd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:56:18.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.902+0000 7f9a9effd640 1 -- 172.21.15.42:0/363307151 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9aa009d440 con 0x7f9aa00a39b0 2024-09-16T03:56:18.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.902+0000 7f9a9f7fe640 10 monclient: get_auth_request con 0x7f9aa0137540 auth_method 0 2024-09-16T03:56:18.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.902+0000 7f9a9f7fe640 1 --2- 172.21.15.42:0/363307151 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9aa0137540 0x7f9aa00a3400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T03:56:18.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.902+0000 7f9a9e7fc640 1 --2- 172.21.15.42:0/363307151 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9aa00a3d90 0x7f9aa00a2ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:56:18.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.902+0000 7f9a9effd640 1 --2- 172.21.15.42:0/363307151 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9aa00a39b0 0x7f9aa01cf540 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f9a9801af60 tx=0x7f9a98019650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:56:18.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.902+0000 7f9a877fe640 10 client.?.objecter ms_handle_connect 0x7f9aa00a39b0 2024-09-16T03:56:18.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.902+0000 7f9a877fe640 10 client.?.objecter resend_mon_ops 2024-09-16T03:56:18.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.902+0000 7f9a877fe640 1 -- 172.21.15.42:0/363307151 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a980bc020 con 0x7f9aa00a39b0 2024-09-16T03:56:18.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.902+0000 7f9a877fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:56:18.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.902+0000 7f9a877fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:56:18.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.906+0000 7f9a877fe640 10 monclient: dump: 2024-09-16T03:56:18.932 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:56:18.932 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:56:18.932 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:18.932 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:18.932 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:56:18.932 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:56:18.932 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:56:18.933 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:56:18.933 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:56:18.933 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:56:18.933 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.906+0000 7f9a877fe640 10 monclient: _finish_auth 0 2024-09-16T03:56:18.933 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.906+0000 7f9a877fe640 10 monclient: _check_auth_tickets 2024-09-16T03:56:18.933 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.906+0000 7f9a877fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:56:18.934 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.906+0000 7f9aa560e640 5 monclient: authenticate success, global_id 854297 2024-09-16T03:56:18.934 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.906+0000 7f9a877fe640 1 -- 172.21.15.42:0/363307151 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a980b86b0 con 0x7f9aa00a39b0 2024-09-16T03:56:18.934 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.906+0000 7f9a877fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:56:18.934 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.906+0000 7f9a877fe640 1 -- 172.21.15.42:0/363307151 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a98009510 con 0x7f9aa00a39b0 2024-09-16T03:56:18.934 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.906+0000 7f9a877fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:56:18.934 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.906+0000 7f9a877fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:56:18.934 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.906+0000 7f9a877fe640 10 monclient: dump: 2024-09-16T03:56:18.935 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:56:18.935 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:56:18.935 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:18.935 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:18.935 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:56:18.935 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:56:18.935 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:56:18.935 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:56:18.935 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:56:18.935 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:56:18.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.906+0000 7f9a9cff9640 4 set_mon_vals no callback set 2024-09-16T03:56:18.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.906+0000 7f9aa560e640 10 monclient: _renew_subs 2024-09-16T03:56:18.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.906+0000 7f9aa560e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:56:18.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.906+0000 7f9aa560e640 1 -- 172.21.15.42:0/363307151 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9aa009d720 con 0x7f9aa00a39b0 2024-09-16T03:56:18.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.906+0000 7f9aa560e640 10 client.854297.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:56:18.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.906+0000 7f9aa560e640 10 monclient: _renew_subs 2024-09-16T03:56:18.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.906+0000 7f9aa560e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:56:18.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.906+0000 7f9aa560e640 1 -- 172.21.15.42:0/363307151 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f9aa009c940 con 0x7f9aa00a39b0 2024-09-16T03:56:18.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.906+0000 7f9aa560e640 1 librados: init done 2024-09-16T03:56:18.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.906+0000 7f9aa560e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:56:18.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.906+0000 7f9aa560e640 1 --2- 172.21.15.42:0/363307151 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9aa013ba10 0x7f9aa002c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:56:18.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.906+0000 7f9aa560e640 10 monclient(hunting): start opening mon connection 2024-09-16T03:56:18.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.906+0000 7f9aa560e640 1 -- 172.21.15.42:0/363307151 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f9aa01342d0 con 0x7f9aa013ba10 2024-09-16T03:56:18.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.906+0000 7f9a877fe640 1 -- 172.21.15.42:0/363307151 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f9a980bf020 con 0x7f9aa00a39b0 2024-09-16T03:56:18.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.906+0000 7f9a877fe640 1 --2- 172.21.15.42:0/363307151 >> v2:172.21.15.154:6812/254845342 conn(0x7f9a8c037810 0x7f9a8c039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:56:18.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.906+0000 7f9a877fe640 1 -- 172.21.15.42:0/363307151 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(441..441 src has 330..441) ==== 5769+0+0 (secure 0 0 0) 0x7f9a980e4390 con 0x7f9aa00a39b0 2024-09-16T03:56:18.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.906+0000 7f9a877fe640 10 client.854297.objecter ms_dispatch 0x7f9aa0000ba0 osd_map(441..441 src has 330..441) 2024-09-16T03:56:18.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.906+0000 7f9a877fe640 3 client.854297.objecter handle_osd_map got epochs [441,441] > 0 2024-09-16T03:56:18.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.906+0000 7f9a877fe640 3 client.854297.objecter handle_osd_map decoding full epoch 441 2024-09-16T03:56:18.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.906+0000 7f9a9e7fc640 1 --2- 172.21.15.42:0/363307151 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9aa013ba10 0x7f9aa002c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:56:18.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.906+0000 7f9a9e7fc640 10 monclient: get_auth_request con 0x7f9aa013ba10 auth_method 0 2024-09-16T03:56:18.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.906+0000 7f9a9e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:56:18.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.906+0000 7f9a9e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:56:18.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.906+0000 7f9a9e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:56:18.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.906+0000 7f9a877fe640 20 client.854297.objecter dump_active .. 0 homeless 2024-09-16T03:56:18.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.906+0000 7f9a9e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:56:18.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.906+0000 7f9a9e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:56:18.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.906+0000 7f9a9e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:56:18.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.906+0000 7f9a9f7fe640 1 --2- 172.21.15.42:0/363307151 >> v2:172.21.15.154:6812/254845342 conn(0x7f9a8c037810 0x7f9a8c039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:56:18.938 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.906+0000 7f9a9f7fe640 10 monclient: get_auth_request con 0x7f9a8c037810 auth_method 0 2024-09-16T03:56:18.938 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.906+0000 7f9a9e7fc640 10 monclient(hunting): handle_auth_done global_id 874108 payload 824 2024-09-16T03:56:18.938 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.906+0000 7f9a9e7fc640 1 -- 172.21.15.42:0/363307151 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9aa013c710 con 0x7f9aa013ba10 2024-09-16T03:56:18.938 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.906+0000 7f9a877fe640 10 client.854297.objecter ms_handle_connect 0x7f9a8c037810 2024-09-16T03:56:18.938 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.906+0000 7f9a9f7fe640 1 --2- 172.21.15.42:0/363307151 >> v2:172.21.15.154:6812/254845342 conn(0x7f9a8c037810 0x7f9a8c039cd0 secure :-1 s=READY pgs=3652 cs=0 l=1 rev1=1 crypto rx=0x7f9a90006fd0 tx=0x7f9a90008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:56:18.938 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.906+0000 7f9a877fe640 10 client.854297.objecter ms_handle_connect 0x7f9aa013ba10 2024-09-16T03:56:18.938 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.906+0000 7f9a877fe640 10 client.854297.objecter resend_mon_ops 2024-09-16T03:56:18.938 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.906+0000 7f9a9e7fc640 1 --2- 172.21.15.42:0/363307151 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9aa013ba10 0x7f9aa002c650 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f9aa0137b00 tx=0x7f9a94011e60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:56:18.938 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.906+0000 7f9a877fe640 1 -- 172.21.15.42:0/363307151 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a94019070 con 0x7f9aa013ba10 2024-09-16T03:56:18.938 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.906+0000 7f9a877fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:56:18.938 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.906+0000 7f9a877fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:56:18.938 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.906+0000 7f9a877fe640 10 monclient: dump: 2024-09-16T03:56:18.938 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:56:18.938 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:56:18.939 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:18.939 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:18.939 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:56:18.939 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:56:18.939 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:56:18.939 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:56:18.939 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:56:18.939 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:56:18.939 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.906+0000 7f9a877fe640 1 -- 172.21.15.42:0/363307151 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f9a9400fcd0 con 0x7f9aa013ba10 2024-09-16T03:56:18.939 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.906+0000 7f9a877fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:56:18.939 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.906+0000 7f9a877fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:56:18.939 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.906+0000 7f9a877fe640 1 -- 172.21.15.42:0/363307151 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9aa013ba10 msgr2=0x7f9aa002c650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:56:18.939 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.906+0000 7f9a877fe640 1 --2- 172.21.15.42:0/363307151 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9aa013ba10 0x7f9aa002c650 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f9aa0137b00 tx=0x7f9a94011e60 comp rx=0 tx=0).stop 2024-09-16T03:56:18.939 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.906+0000 7f9a877fe640 1 -- 172.21.15.42:0/363307151 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9aa013ba10 msgr2=0x7f9aa002c650 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:56:18.940 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.906+0000 7f9a877fe640 1 --2- 172.21.15.42:0/363307151 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9aa013ba10 0x7f9aa002c650 secure :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f9aa0137b00 tx=0x7f9a94011e60 comp rx=0 tx=0).stop 2024-09-16T03:56:18.940 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.914+0000 7f9aa560e640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:56:18.940 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.914+0000 7f9aa560e640 1 --2- 172.21.15.42:0/363307151 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9aa013bdf0 0x7f9aa002cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:56:18.941 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.914+0000 7f9aa560e640 10 monclient(hunting): start opening mon connection 2024-09-16T03:56:18.941 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.914+0000 7f9aa560e640 1 -- 172.21.15.42:0/363307151 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f9aa0137000 con 0x7f9aa013bdf0 2024-09-16T03:56:18.941 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.914+0000 7f9a9e7fc640 1 --2- 172.21.15.42:0/363307151 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9aa013bdf0 0x7f9aa002cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:56:18.941 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.914+0000 7f9a9e7fc640 1 -- 172.21.15.42:0/363307151 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9aa013bdf0 msgr2=0x7f9aa002cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-16T03:56:18.941 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.914+0000 7f9a9e7fc640 1 -- 172.21.15.42:0/363307151 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9aa013bdf0 msgr2=0x7f9aa002cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T03:56:18.941 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.914+0000 7f9a9e7fc640 1 --2- 172.21.15.42:0/363307151 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9aa013bdf0 0x7f9aa002cb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:56:18.941 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:18.914+0000 7f9a9e7fc640 1 --2- 172.21.15.42:0/363307151 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9aa013bdf0 0x7f9aa002cb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:56:19.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.118+0000 7f9a9e7fc640 1 --2- 172.21.15.42:0/363307151 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9aa013bdf0 0x7f9aa002cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:56:19.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.118+0000 7f9a9e7fc640 10 monclient: get_auth_request con 0x7f9aa013bdf0 auth_method 0 2024-09-16T03:56:19.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.118+0000 7f9a9e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:56:19.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.118+0000 7f9a9e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:56:19.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.118+0000 7f9a9e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:56:19.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.118+0000 7f9a9e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:56:19.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.118+0000 7f9a9e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:56:19.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.118+0000 7f9a9e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:56:19.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.118+0000 7f9a9e7fc640 10 monclient(hunting): handle_auth_done global_id 874111 payload 824 2024-09-16T03:56:19.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.118+0000 7f9a9e7fc640 1 -- 172.21.15.42:0/363307151 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f9aa013dda0 con 0x7f9aa013bdf0 2024-09-16T03:56:19.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.118+0000 7f9a9e7fc640 1 --2- 172.21.15.42:0/363307151 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9aa013bdf0 0x7f9aa002cb90 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f9aa0137b00 tx=0x7f9a94004740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:56:19.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.118+0000 7f9a877fe640 10 client.854297.objecter ms_handle_connect 0x7f9aa013bdf0 2024-09-16T03:56:19.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.118+0000 7f9a877fe640 10 client.854297.objecter resend_mon_ops 2024-09-16T03:56:19.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.118+0000 7f9a877fe640 1 -- 172.21.15.42:0/363307151 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a94019040 con 0x7f9aa013bdf0 2024-09-16T03:56:19.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.118+0000 7f9a877fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:56:19.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.118+0000 7f9a877fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:56:19.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.118+0000 7f9a877fe640 10 monclient: dump: 2024-09-16T03:56:19.124 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:56:19.124 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:56:19.124 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:19.124 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:19.124 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:56:19.124 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:56:19.125 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:56:19.125 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:56:19.125 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:56:19.125 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:56:19.125 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.118+0000 7f9a877fe640 1 -- 172.21.15.42:0/363307151 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4545 (secure 0 0 0) 0x7f9a940143e0 con 0x7f9aa013bdf0 2024-09-16T03:56:19.126 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.118+0000 7f9a877fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:56:19.126 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.118+0000 7f9a877fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:56:19.126 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.118+0000 7f9a877fe640 1 -- 172.21.15.42:0/363307151 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9aa013bdf0 msgr2=0x7f9aa002cb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:56:19.126 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.118+0000 7f9a877fe640 1 --2- 172.21.15.42:0/363307151 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9aa013bdf0 0x7f9aa002cb90 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f9aa0137b00 tx=0x7f9a94004740 comp rx=0 tx=0).stop 2024-09-16T03:56:19.126 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.118+0000 7f9a877fe640 1 -- 172.21.15.42:0/363307151 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9aa013bdf0 msgr2=0x7f9aa002cb90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:56:19.126 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.118+0000 7f9a877fe640 1 --2- 172.21.15.42:0/363307151 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9aa013bdf0 0x7f9aa002cb90 secure :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f9aa0137b00 tx=0x7f9a94004740 comp rx=0 tx=0).stop 2024-09-16T03:56:19.127 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:56:19.127 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:56:19.127 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:56:19.127 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T03:56:19.127 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 128, 2024-09-16T03:56:19.127 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 10399, 2024-09-16T03:56:19.127 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:56:19.127 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:56:19.127 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T03:56:19.127 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T03:56:19.127 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:56:19.128 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 2, 2024-09-16T03:56:19.128 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:56:19.128 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:56:19.128 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:56:19.128 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:56:19.128 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:56:19.128 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:56:19.128 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:56:19.128 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:56:19.128 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:56:19.128 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:56:19.128 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:56:19.129 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:56:19.129 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:56:19.129 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:56:19.129 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:56:19.129 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:56:19.129 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:56:19.129 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:56:19.129 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:56:19.129 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:56:19.130 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:56:19.130 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:56:19.130 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:56:19.130 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:56:19.130 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:56:19.130 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:56:19.130 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:56:19.130 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:56:19.130 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:56:19.130 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:56:19.130 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:56:19.130 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:56:19.130 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:56:19.130 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:56:19.131 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:56:19.131 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:56:19.131 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:56:19.131 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:56:19.131 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:56:19.131 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:56:19.131 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:56:19.131 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:56:19.131 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:56:19.131 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:56:19.131 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:56:19.131 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:56:19.131 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:56:19.132 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:56:19.132 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:56:19.132 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:56:19.132 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:56:19.132 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:56:19.132 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:56:19.132 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:56:19.132 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:56:19.132 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:56:19.132 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:56:19.132 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:56:19.132 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:56:19.132 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:56:19.132 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:56:19.133 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:56:19.133 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:56:19.133 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:56:19.133 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:56:19.133 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:56:19.133 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:56:19.133 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:56:19.133 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:56:19.133 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:56:19.133 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:56:19.133 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:56:19.133 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:56:19.133 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:56:19.133 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:56:19.134 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:56:19.134 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:56:19.134 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:56:19.134 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:56:19.134 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:56:19.134 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:56:19.134 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:56:19.134 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:56:19.134 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:56:19.134 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:56:19.134 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:56:19.134 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:56:19.134 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:56:19.134 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:56:19.134 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:56:19.135 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:56:19.135 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:56:19.135 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:56:19.135 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:56:19.135 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:56:19.135 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:56:19.135 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:56:19.135 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:56:19.135 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:56:19.135 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:56:19.135 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:56:19.135 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:56:19.135 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:56:19.135 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:56:19.136 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:56:19.136 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:56:19.136 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:56:19.136 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:56:19.136 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:56:19.136 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:56:19.136 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:56:19.136 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:56:19.136 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:56:19.136 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:56:19.136 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:56:19.136 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:56:19.136 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:56:19.137 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:56:19.137 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:56:19.137 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:56:19.137 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:56:19.137 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:56:19.137 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:56:19.137 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:56:19.137 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:56:19.137 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:56:19.137 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:56:19.137 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:56:19.137 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:56:19.137 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:56:19.138 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:56:19.138 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:56:19.138 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:56:19.138 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:56:19.138 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:56:19.138 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:56:19.138 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:56:19.138 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:56:19.138 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:56:19.138 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:56:19.138 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:56:19.138 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:56:19.139 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:56:19.139 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:56:19.139 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:56:19.139 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:56:19.139 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:56:19.139 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:56:19.139 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:56:19.139 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:56:19.139 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:56:19.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.118+0000 7f9aa560e640 10 librados: watch_flush enter 2024-09-16T03:56:19.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.118+0000 7f9aa560e640 10 librados: watch_flush exit 2024-09-16T03:56:19.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.118+0000 7f9aa560e640 20 client.854297.objecter shutdown clearing up homeless session... 2024-09-16T03:56:19.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.118+0000 7f9aa560e640 10 client.854297.objecter successfully canceled tick 2024-09-16T03:56:19.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.122+0000 7f9aa560e640 1 -- 172.21.15.42:0/363307151 >> v2:172.21.15.154:6812/254845342 conn(0x7f9a8c037810 msgr2=0x7f9a8c039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:56:19.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.122+0000 7f9aa560e640 1 --2- 172.21.15.42:0/363307151 >> v2:172.21.15.154:6812/254845342 conn(0x7f9a8c037810 0x7f9a8c039cd0 secure :-1 s=READY pgs=3652 cs=0 l=1 rev1=1 crypto rx=0x7f9a90006fd0 tx=0x7f9a90008040 comp rx=0 tx=0).stop 2024-09-16T03:56:19.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.122+0000 7f9aa560e640 10 monclient: shutdown 2024-09-16T03:56:19.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.122+0000 7f9aa560e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:56:19.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.122+0000 7f9aa560e640 1 -- 172.21.15.42:0/363307151 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9aa00a39b0 msgr2=0x7f9aa01cf540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:56:19.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.122+0000 7f9aa560e640 1 --2- 172.21.15.42:0/363307151 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9aa00a39b0 0x7f9aa01cf540 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f9a9801af60 tx=0x7f9a98019650 comp rx=0 tx=0).stop 2024-09-16T03:56:19.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.122+0000 7f9a9f7fe640 1 -- 172.21.15.42:0/363307151 reap_dead start 2024-09-16T03:56:19.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.122+0000 7f9a9f7fe640 1 -- 172.21.15.42:0/363307151 reap_dead start 2024-09-16T03:56:19.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.122+0000 7f9aa560e640 1 -- 172.21.15.42:0/363307151 shutdown_connections 2024-09-16T03:56:19.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.122+0000 7f9aa560e640 1 -- 172.21.15.42:0/363307151 >> 172.21.15.42:0/363307151 conn(0x7f9aa00a9fb0 msgr2=0x7f9aa0131980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:56:19.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.122+0000 7f9aa560e640 1 -- 172.21.15.42:0/363307151 shutdown_connections 2024-09-16T03:56:19.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.122+0000 7f9aa560e640 1 -- 172.21.15.42:0/363307151 wait complete. 2024-09-16T03:56:19.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.122+0000 7f9aa560e640 1 librados: shutdown 2024-09-16T03:56:19.150 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:56:19.150 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T03:56:19.150 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T03:56:19.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.270+0000 7f81367d9640 10 monclient: get_monmap_and_config 2024-09-16T03:56:19.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.270+0000 7f81367d9640 10 monclient: build_initial_monmap 2024-09-16T03:56:19.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.270+0000 7f81367d9640 10 monclient: monmap: 2024-09-16T03:56:19.277 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:56:19.278 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:56:19.278 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:56:19.275403+0000 2024-09-16T03:56:19.278 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:56:19.275403+0000 2024-09-16T03:56:19.278 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:56:19.278 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:56:19.278 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:56:19.279 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:56:19.279 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:56:19.279 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:56:19.279 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:56:19.279 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:56:19.291 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.274+0000 7f81367d9640 1 Processor -- start 2024-09-16T03:56:19.291 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.274+0000 7f81367d9640 1 -- start start 2024-09-16T03:56:19.291 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.274+0000 7f81367d9640 10 monclient: init 2024-09-16T03:56:19.292 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.274+0000 7f81367d9640 10 monclient: _reopen_session rank -1 2024-09-16T03:56:19.292 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.274+0000 7f81367d9640 10 monclient: _add_conns ranks=[4,2,0,3,1] 2024-09-16T03:56:19.292 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.274+0000 7f81367d9640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f81300a1330 0x7f81300a1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:56:19.292 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.274+0000 7f81367d9640 10 monclient(hunting): picked mon.noname-c con 0x7f81300a1330 addr v2:172.21.15.154:3300/0 2024-09-16T03:56:19.292 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.274+0000 7f81367d9640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f813006dac0 0x7f81300a1c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:56:19.292 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.274+0000 7f81367d9640 10 monclient(hunting): picked mon.noname-b con 0x7f813006dac0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:56:19.292 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.274+0000 7f81367d9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8130135100 addr v1:172.21.15.42:6789/0 2024-09-16T03:56:19.292 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.274+0000 7f81367d9640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f813009c7d0 con 0x7f8130135100 2024-09-16T03:56:19.292 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.274+0000 7f81367d9640 10 monclient(hunting): start opening mon connection 2024-09-16T03:56:19.292 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.274+0000 7f81367d9640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f813009c090 con 0x7f81300a1330 2024-09-16T03:56:19.293 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.274+0000 7f81367d9640 10 monclient(hunting): start opening mon connection 2024-09-16T03:56:19.293 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.274+0000 7f81367d9640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f813009f610 con 0x7f813006dac0 2024-09-16T03:56:19.293 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.274+0000 7f81367d9640 10 monclient(hunting): _renew_subs 2024-09-16T03:56:19.293 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.274+0000 7f81367d9640 10 monclient(hunting): authenticate will time out at 1826.245161s 2024-09-16T03:56:19.293 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.274+0000 7f81277fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f813006dac0 0x7f81300a1c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:56:19.294 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.274+0000 7f81277fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f813006dac0 0x7f81300a1c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:40804/0 (socket says 172.21.15.42:40804) 2024-09-16T03:56:19.294 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.274+0000 7f8134d76640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f8130135100 0x7f8130135550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:33360/0 (socket says 172.21.15.42:33360) 2024-09-16T03:56:19.294 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.274+0000 7f81277fe640 1 -- 172.21.15.42:0/1441895865 learned_addr learned my addr 172.21.15.42:0/1441895865 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:56:19.294 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.274+0000 7f81277fe640 10 monclient(hunting): get_auth_request con 0x7f813006dac0 auth_method 0 2024-09-16T03:56:19.294 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.274+0000 7f81277fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:56:19.294 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.274+0000 7f81277fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:56:19.294 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.274+0000 7f81277fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:56:19.294 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.274+0000 7f81277fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:56:19.295 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.274+0000 7f81277fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:56:19.295 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.274+0000 7f81277fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:56:19.295 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.274+0000 7f812ffff640 1 --2- 172.21.15.42:0/1441895865 >> v2:172.21.15.154:3300/0 conn(0x7f81300a1330 0x7f81300a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:56:19.295 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.274+0000 7f812ffff640 10 monclient(hunting): get_auth_request con 0x7f81300a1330 auth_method 0 2024-09-16T03:56:19.295 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.274+0000 7f812ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:56:19.295 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.274+0000 7f812ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:56:19.295 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.274+0000 7f812ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:56:19.295 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.274+0000 7f81277fe640 10 monclient(hunting): handle_auth_done global_id 854300 payload 274 2024-09-16T03:56:19.295 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.274+0000 7f81277fe640 1 -- 172.21.15.42:0/1441895865 >> v2:172.21.15.154:3300/0 conn(0x7f81300a1330 msgr2=0x7f81300a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:56:19.295 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.274+0000 7f81277fe640 1 --2- 172.21.15.42:0/1441895865 >> v2:172.21.15.154:3300/0 conn(0x7f81300a1330 0x7f81300a1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:56:19.295 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.274+0000 7f81277fe640 1 -- 172.21.15.42:0/1441895865 >> v1:172.21.15.42:6789/0 conn(0x7f8130135100 legacy=0x7f8130135550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:56:19.295 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.274+0000 7f81277fe640 10 monclient: _finish_hunting 0 2024-09-16T03:56:19.295 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.274+0000 7f81277fe640 1 monclient: found mon.noname-b 2024-09-16T03:56:19.296 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.274+0000 7f81277fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:56:19.296 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.274+0000 7f81277fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T03:56:19.296 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.274+0000 7f81277fe640 1 -- 172.21.15.42:0/1441895865 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f813009f8f0 con 0x7f813006dac0 2024-09-16T03:56:19.296 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.274+0000 7f81277fe640 1 --2- 172.21.15.42:0/1441895865 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f813006dac0 0x7f81300a1c50 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f8118009870 tx=0x7f811801c540 comp rx=0 tx=0).ready entity=mon.2 client_cookie=87864a9fe503c8cf server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:56:19.296 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.278+0000 7f812f7fe640 1 -- 172.21.15.42:0/1441895865 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81180ab650 con 0x7f813006dac0 2024-09-16T03:56:19.296 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.278+0000 7f812f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:56:19.296 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.278+0000 7f812f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:56:19.296 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.278+0000 7f812f7fe640 10 monclient: dump: 2024-09-16T03:56:19.296 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:56:19.296 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:56:19.296 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:19.296 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:19.297 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:56:19.297 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:56:19.297 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:56:19.297 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:56:19.297 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:56:19.297 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:56:19.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.278+0000 7f812f7fe640 10 monclient: _finish_auth 0 2024-09-16T03:56:19.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.278+0000 7f812f7fe640 10 monclient: _check_auth_tickets 2024-09-16T03:56:19.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.278+0000 7f812f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:56:19.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.278+0000 7f812f7fe640 1 -- 172.21.15.42:0/1441895865 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f81180b0070 con 0x7f813006dac0 2024-09-16T03:56:19.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.278+0000 7f81367d9640 5 monclient: authenticate success, global_id 854300 2024-09-16T03:56:19.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.278+0000 7f81367d9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:56:19.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.278+0000 7f812f7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:56:19.298 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.278+0000 7f81367d9640 10 monclient: get_monmap_and_config success 2024-09-16T03:56:19.298 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.278+0000 7f81367d9640 4 set_mon_vals no callback set 2024-09-16T03:56:19.298 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.278+0000 7f812f7fe640 1 -- 172.21.15.42:0/1441895865 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81180aba10 con 0x7f813006dac0 2024-09-16T03:56:19.298 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.278+0000 7f812f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:56:19.298 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.278+0000 7f812f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:56:19.298 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.278+0000 7f812f7fe640 10 monclient: dump: 2024-09-16T03:56:19.298 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:56:19.298 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:56:19.298 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:19.298 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:19.298 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:56:19.298 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:56:19.298 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:56:19.299 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:56:19.299 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:56:19.299 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:56:19.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.278+0000 7f81367d9640 10 monclient: shutdown 2024-09-16T03:56:19.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.278+0000 7f81367d9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:56:19.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.278+0000 7f81367d9640 1 -- 172.21.15.42:0/1441895865 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f813006dac0 msgr2=0x7f81300a1c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:56:19.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.278+0000 7f81367d9640 1 --2- 172.21.15.42:0/1441895865 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f813006dac0 0x7f81300a1c50 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f8118009870 tx=0x7f811801c540 comp rx=0 tx=0).stop 2024-09-16T03:56:19.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.278+0000 7f81367d9640 1 -- 172.21.15.42:0/1441895865 shutdown_connections 2024-09-16T03:56:19.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.278+0000 7f81367d9640 1 --2- 172.21.15.42:0/1441895865 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f813006dac0 0x7f81300a1c50 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:56:19.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.278+0000 7f81367d9640 1 --2- 172.21.15.42:0/1441895865 >> v2:172.21.15.154:3300/0 conn(0x7f81300a1330 0x7f81300a1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:56:19.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.278+0000 7f81367d9640 1 -- 172.21.15.42:0/1441895865 >> 172.21.15.42:0/1441895865 conn(0x7f81300a2420 msgr2=0x7f81300a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:56:19.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.278+0000 7f81367d9640 1 -- 172.21.15.42:0/1441895865 shutdown_connections 2024-09-16T03:56:19.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.278+0000 7f81367d9640 1 -- 172.21.15.42:0/1441895865 wait complete. 2024-09-16T03:56:19.300 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.278+0000 7f81367d9640 10 monclient: build_initial_monmap 2024-09-16T03:56:19.300 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.278+0000 7f81367d9640 10 monclient: monmap: 2024-09-16T03:56:19.300 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:56:19.300 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:56:19.300 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:56:19.300 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:56:19.300 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:56:19.300 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:56:19.301 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:56:19.301 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:56:19.301 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:56:19.301 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:56:19.301 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.278+0000 7f81367d9640 1 librados: starting msgr at 2024-09-16T03:56:19.301 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.278+0000 7f81367d9640 1 librados: starting objecter 2024-09-16T03:56:19.301 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.278+0000 7f81367d9640 1 Processor -- start 2024-09-16T03:56:19.301 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.278+0000 7f81367d9640 1 -- start start 2024-09-16T03:56:19.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.278+0000 7f81367d9640 1 librados: setting wanted keys 2024-09-16T03:56:19.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.278+0000 7f81367d9640 1 librados: calling monclient init 2024-09-16T03:56:19.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.278+0000 7f81367d9640 10 monclient: init 2024-09-16T03:56:19.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.278+0000 7f81367d9640 10 monclient: _reopen_session rank -1 2024-09-16T03:56:19.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.278+0000 7f81367d9640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:56:19.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.278+0000 7f81367d9640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f81300a1330 0x7f813013b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:56:19.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.278+0000 7f81367d9640 10 monclient(hunting): picked mon.noname-c con 0x7f81300a1330 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:56:19.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.278+0000 7f81367d9640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8130135100 0x7f8130134480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:56:19.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.278+0000 7f81367d9640 10 monclient(hunting): picked mon.noname-b con 0x7f8130135100 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:56:19.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.278+0000 7f81367d9640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f81301349c0 0x7f813011bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:56:19.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.278+0000 7f81367d9640 10 monclient(hunting): picked mon.noname-a con 0x7f81301349c0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:56:19.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.278+0000 7f81367d9640 10 monclient(hunting): start opening mon connection 2024-09-16T03:56:19.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.278+0000 7f81367d9640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f813013fd00 con 0x7f81301349c0 2024-09-16T03:56:19.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.278+0000 7f81367d9640 10 monclient(hunting): start opening mon connection 2024-09-16T03:56:19.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.278+0000 7f81367d9640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f813013fb80 con 0x7f8130135100 2024-09-16T03:56:19.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.278+0000 7f81367d9640 10 monclient(hunting): start opening mon connection 2024-09-16T03:56:19.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.278+0000 7f81367d9640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f813013fe80 con 0x7f81300a1330 2024-09-16T03:56:19.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.278+0000 7f81367d9640 10 monclient(hunting): _renew_subs 2024-09-16T03:56:19.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.278+0000 7f81367d9640 10 monclient(hunting): authenticate will time out at 1826.250149s 2024-09-16T03:56:19.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.278+0000 7f812ffff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f81300a1330 0x7f813013b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:56:19.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.278+0000 7f8134d76640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f81301349c0 0x7f813011bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:56:19.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.278+0000 7f812ffff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f81300a1330 0x7f813013b9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:40814/0 (socket says 172.21.15.42:40814) 2024-09-16T03:56:19.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.278+0000 7f812ffff640 1 -- 172.21.15.42:0/2519725849 learned_addr learned my addr 172.21.15.42:0/2519725849 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:56:19.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.278+0000 7f812ffff640 10 monclient(hunting): get_auth_request con 0x7f81300a1330 auth_method 0 2024-09-16T03:56:19.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.278+0000 7f812ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:56:19.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.278+0000 7f812ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:56:19.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.278+0000 7f812ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:56:19.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.278+0000 7f8134d76640 10 monclient(hunting): get_auth_request con 0x7f81301349c0 auth_method 0 2024-09-16T03:56:19.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.278+0000 7f8134d76640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:56:19.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.278+0000 7f8134d76640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:56:19.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.278+0000 7f8134d76640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:56:19.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.278+0000 7f812ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:56:19.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.278+0000 7f812ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:56:19.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.278+0000 7f812ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:56:19.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.278+0000 7f8134d76640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:56:19.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.278+0000 7f8134d76640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:56:19.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.278+0000 7f8134d76640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:56:19.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.278+0000 7f81277fe640 1 --2- 172.21.15.42:0/2519725849 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8130135100 0x7f8130134480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:56:19.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.278+0000 7f81277fe640 10 monclient(hunting): get_auth_request con 0x7f8130135100 auth_method 0 2024-09-16T03:56:19.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.278+0000 7f81277fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:56:19.306 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.278+0000 7f81277fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:56:19.306 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.278+0000 7f81277fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:56:19.306 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.278+0000 7f812ffff640 10 monclient(hunting): handle_auth_done global_id 854303 payload 824 2024-09-16T03:56:19.306 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.278+0000 7f812ffff640 1 -- 172.21.15.42:0/2519725849 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8130135100 msgr2=0x7f8130134480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:56:19.306 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.278+0000 7f812ffff640 1 --2- 172.21.15.42:0/2519725849 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8130135100 0x7f8130134480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:56:19.306 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.278+0000 7f812ffff640 1 -- 172.21.15.42:0/2519725849 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f81301349c0 msgr2=0x7f813011bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:56:19.306 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.278+0000 7f812ffff640 1 --2- 172.21.15.42:0/2519725849 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f81301349c0 0x7f813011bf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:56:19.306 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.278+0000 7f812ffff640 10 monclient: _finish_hunting 0 2024-09-16T03:56:19.306 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.278+0000 7f812ffff640 1 monclient: found mon.noname-c 2024-09-16T03:56:19.306 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.278+0000 7f812ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:56:19.306 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.278+0000 7f812ffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:56:19.307 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.278+0000 7f812ffff640 1 -- 172.21.15.42:0/2519725849 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8130073a30 con 0x7f81300a1330 2024-09-16T03:56:19.307 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.278+0000 7f8134d76640 1 --2- 172.21.15.42:0/2519725849 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f81301349c0 0x7f813011bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:56:19.307 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.278+0000 7f812ffff640 1 --2- 172.21.15.42:0/2519725849 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f81300a1330 0x7f813013b9b0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f812000d830 tx=0x7f812000dcf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:56:19.307 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.278+0000 7f812d7fa640 10 client.?.objecter ms_handle_connect 0x7f81300a1330 2024-09-16T03:56:19.307 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.278+0000 7f812d7fa640 10 client.?.objecter resend_mon_ops 2024-09-16T03:56:19.307 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.278+0000 7f812d7fa640 1 -- 172.21.15.42:0/2519725849 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f812000b6c0 con 0x7f81300a1330 2024-09-16T03:56:19.307 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.278+0000 7f812d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:56:19.307 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.278+0000 7f812d7fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:56:19.307 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.278+0000 7f812d7fa640 10 monclient: dump: 2024-09-16T03:56:19.307 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:56:19.307 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:56:19.307 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:19.307 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:19.308 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:56:19.308 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:56:19.308 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:56:19.308 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:56:19.308 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:56:19.308 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:56:19.308 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.278+0000 7f812d7fa640 10 monclient: _finish_auth 0 2024-09-16T03:56:19.308 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.278+0000 7f812d7fa640 10 monclient: _check_auth_tickets 2024-09-16T03:56:19.308 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.278+0000 7f812d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:56:19.308 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.278+0000 7f812d7fa640 1 -- 172.21.15.42:0/2519725849 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8120019070 con 0x7f81300a1330 2024-09-16T03:56:19.309 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.278+0000 7f812d7fa640 10 monclient: handle_config config(0 keys) 2024-09-16T03:56:19.309 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.278+0000 7f812d7fa640 1 -- 172.21.15.42:0/2519725849 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f812000ba60 con 0x7f81300a1330 2024-09-16T03:56:19.309 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.278+0000 7f812d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:56:19.309 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.278+0000 7f812d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:56:19.309 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.278+0000 7f812d7fa640 10 monclient: dump: 2024-09-16T03:56:19.309 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:56:19.309 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:56:19.309 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:19.309 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:19.310 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:56:19.310 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:56:19.310 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:56:19.310 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:56:19.310 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:56:19.310 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:56:19.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.278+0000 7f81367d9640 5 monclient: authenticate success, global_id 854303 2024-09-16T03:56:19.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.278+0000 7f812effd640 4 set_mon_vals no callback set 2024-09-16T03:56:19.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.278+0000 7f81367d9640 10 monclient: _renew_subs 2024-09-16T03:56:19.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.278+0000 7f81367d9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:56:19.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.278+0000 7f81367d9640 1 -- 172.21.15.42:0/2519725849 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f813013c850 con 0x7f81300a1330 2024-09-16T03:56:19.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.278+0000 7f81367d9640 10 client.854303.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:56:19.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.278+0000 7f81367d9640 10 monclient: _renew_subs 2024-09-16T03:56:19.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.278+0000 7f81367d9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:56:19.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.278+0000 7f81367d9640 1 -- 172.21.15.42:0/2519725849 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f813011eb30 con 0x7f81300a1330 2024-09-16T03:56:19.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.278+0000 7f81367d9640 1 librados: init done 2024-09-16T03:56:19.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.282+0000 7f812d7fa640 1 -- 172.21.15.42:0/2519725849 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f812001e460 con 0x7f81300a1330 2024-09-16T03:56:19.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.282+0000 7f81367d9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:56:19.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.282+0000 7f81367d9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:56:19.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.282+0000 7f81367d9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:56:19.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.282+0000 7f81367d9640 1 -- 172.21.15.42:0/2519725849 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f813011e6e0 con 0x7f81300a1330 2024-09-16T03:56:19.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.282+0000 7f812d7fa640 1 --2- 172.21.15.42:0/2519725849 >> v2:172.21.15.154:6812/254845342 conn(0x7f8110037810 0x7f8110039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:56:19.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.282+0000 7f812d7fa640 1 -- 172.21.15.42:0/2519725849 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(441..441 src has 330..441) ==== 5769+0+0 (secure 0 0 0) 0x7f81200495a0 con 0x7f81300a1330 2024-09-16T03:56:19.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.282+0000 7f812d7fa640 10 client.854303.objecter ms_dispatch 0x7f8130000ba0 osd_map(441..441 src has 330..441) 2024-09-16T03:56:19.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.282+0000 7f812d7fa640 3 client.854303.objecter handle_osd_map got epochs [441,441] > 0 2024-09-16T03:56:19.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.282+0000 7f812d7fa640 3 client.854303.objecter handle_osd_map decoding full epoch 441 2024-09-16T03:56:19.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.282+0000 7f812d7fa640 20 client.854303.objecter dump_active .. 0 homeless 2024-09-16T03:56:19.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.282+0000 7f81277fe640 1 --2- 172.21.15.42:0/2519725849 >> v2:172.21.15.154:6812/254845342 conn(0x7f8110037810 0x7f8110039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:56:19.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.282+0000 7f81277fe640 10 monclient: get_auth_request con 0x7f8110037810 auth_method 0 2024-09-16T03:56:19.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.282+0000 7f812d7fa640 10 client.854303.objecter ms_handle_connect 0x7f8110037810 2024-09-16T03:56:19.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.282+0000 7f81277fe640 1 --2- 172.21.15.42:0/2519725849 >> v2:172.21.15.154:6812/254845342 conn(0x7f8110037810 0x7f8110039cd0 secure :-1 s=READY pgs=3653 cs=0 l=1 rev1=1 crypto rx=0x7f81180af000 tx=0x7f8118004740 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:56:19.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.286+0000 7f812d7fa640 1 -- 172.21.15.42:0/2519725849 <== mon.2 v2:172.21.15.42:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f813011e6e0 con 0x7f81300a1330 2024-09-16T03:56:19.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.286+0000 7f812d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:56:19.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.286+0000 7f812d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:56:19.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.514+0000 7f81367d9640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T03:56:19.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.514+0000 7f81367d9640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T03:56:19.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.514+0000 7f81367d9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:56:19.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.514+0000 7f81367d9640 1 -- 172.21.15.42:0/2519725849 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f8130135510 con 0x7f81300a1330 2024-09-16T03:56:19.521 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.514+0000 7f812d7fa640 1 -- 172.21.15.42:0/2519725849 <== mon.2 v2:172.21.15.42:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f812000be10 con 0x7f81300a1330 2024-09-16T03:56:19.521 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.514+0000 7f812d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T03:56:19.521 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.514+0000 7f812d7fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:56:19.521 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.518+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.522 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.518+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=26,config=1,health=10,kv=9,logm=54} crc {auth=2919804621,config=3678701417,health=3591851382,kv=3446188498,logm=3197624518}) 2024-09-16T03:56:19.522 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.518+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=23,config=2,health=11,kv=9,logm=55} crc {auth=2121426087,config=692619928,health=1999565914,kv=867007345,logm=2532344706}) 2024-09-16T03:56:19.522 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.518+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.522 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.518+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=26,config=1,health=10,kv=9,logm=54} crc {auth=2919804621,config=3678701417,health=3591851382,kv=3446188498,logm=3197624518}) 2024-09-16T03:56:19.522 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.518+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=25,config=2,health=12,kv=8,logm=53} crc {auth=1779891625,config=692619927,health=1258189448,kv=1897997419,logm=2929955319}) 2024-09-16T03:56:19.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.518+0000 7f81367d9640 10 librados: watch_flush enter 2024-09-16T03:56:19.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.518+0000 7f81367d9640 10 librados: watch_flush exit 2024-09-16T03:56:19.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.518+0000 7f81367d9640 20 client.854303.objecter shutdown clearing up homeless session... 2024-09-16T03:56:19.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.518+0000 7f81367d9640 10 client.854303.objecter successfully canceled tick 2024-09-16T03:56:19.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.518+0000 7f81367d9640 1 -- 172.21.15.42:0/2519725849 >> v2:172.21.15.154:6812/254845342 conn(0x7f8110037810 msgr2=0x7f8110039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:56:19.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.518+0000 7f81367d9640 1 --2- 172.21.15.42:0/2519725849 >> v2:172.21.15.154:6812/254845342 conn(0x7f8110037810 0x7f8110039cd0 secure :-1 s=READY pgs=3653 cs=0 l=1 rev1=1 crypto rx=0x7f81180af000 tx=0x7f8118004740 comp rx=0 tx=0).stop 2024-09-16T03:56:19.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.518+0000 7f81367d9640 10 monclient: shutdown 2024-09-16T03:56:19.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.518+0000 7f81367d9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:56:19.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.518+0000 7f81367d9640 1 -- 172.21.15.42:0/2519725849 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f81300a1330 msgr2=0x7f813013b9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:56:19.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.518+0000 7f81367d9640 1 --2- 172.21.15.42:0/2519725849 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f81300a1330 0x7f813013b9b0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f812000d830 tx=0x7f812000dcf0 comp rx=0 tx=0).stop 2024-09-16T03:56:19.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.518+0000 7f81367d9640 1 -- 172.21.15.42:0/2519725849 shutdown_connections 2024-09-16T03:56:19.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.518+0000 7f81367d9640 1 --2- 172.21.15.42:0/2519725849 >> v2:172.21.15.154:6812/254845342 conn(0x7f8110037810 0x7f8110039cd0 secure :-1 s=CLOSED pgs=3653 cs=0 l=1 rev1=1 crypto rx=0x7f81180af000 tx=0x7f8118004740 comp rx=0 tx=0).stop 2024-09-16T03:56:19.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.518+0000 7f81367d9640 1 --2- 172.21.15.42:0/2519725849 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f81301349c0 0x7f813011bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:56:19.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.518+0000 7f81367d9640 1 --2- 172.21.15.42:0/2519725849 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8130135100 0x7f8130134480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:56:19.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.518+0000 7f81367d9640 1 --2- 172.21.15.42:0/2519725849 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f81300a1330 0x7f813013b9b0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:56:19.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.518+0000 7f81367d9640 1 -- 172.21.15.42:0/2519725849 >> 172.21.15.42:0/2519725849 conn(0x7f81300a2420 msgr2=0x7f81300a3720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:56:19.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.518+0000 7f81367d9640 1 -- 172.21.15.42:0/2519725849 shutdown_connections 2024-09-16T03:56:19.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.518+0000 7f81367d9640 1 -- 172.21.15.42:0/2519725849 wait complete. 2024-09-16T03:56:19.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:19.518+0000 7f81367d9640 1 librados: shutdown 2024-09-16T03:56:19.526 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.518+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.526 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.518+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2385063622}) 2024-09-16T03:56:19.526 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.518+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3087627290}) 2024-09-16T03:56:19.526 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.518+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.526 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.518+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2385063622}) 2024-09-16T03:56:19.526 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.518+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1372094356}) 2024-09-16T03:56:19.528 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.522+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.528 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.522+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1512076290}) 2024-09-16T03:56:19.528 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.522+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3257208074}) 2024-09-16T03:56:19.528 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.522+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.528 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.522+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1512076290}) 2024-09-16T03:56:19.528 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.522+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3275824101}) 2024-09-16T03:56:19.531 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.526+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.531 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.526+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=90825773}) 2024-09-16T03:56:19.531 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.526+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2805536100}) 2024-09-16T03:56:19.531 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.526+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.531 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.526+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=90825773}) 2024-09-16T03:56:19.531 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.526+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2118724689}) 2024-09-16T03:56:19.533 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.530+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.534 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.530+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=334098129}) 2024-09-16T03:56:19.534 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.530+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2476183819}) 2024-09-16T03:56:19.534 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.530+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.534 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.530+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=334098129}) 2024-09-16T03:56:19.534 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.530+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3668341323}) 2024-09-16T03:56:19.535 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.530+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.535 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.530+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3362449553}) 2024-09-16T03:56:19.535 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.530+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3487875686}) 2024-09-16T03:56:19.535 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.530+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.536 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.530+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3362449553}) 2024-09-16T03:56:19.536 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.530+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3954666336}) 2024-09-16T03:56:19.537 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.534+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.537 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.534+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2148417991}) 2024-09-16T03:56:19.537 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.534+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2736483503}) 2024-09-16T03:56:19.537 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.534+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.537 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.534+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2148417991}) 2024-09-16T03:56:19.537 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.534+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4112301030}) 2024-09-16T03:56:19.539 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.534+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.539 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.534+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=941679587}) 2024-09-16T03:56:19.539 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.534+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3605312440}) 2024-09-16T03:56:19.539 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.534+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.539 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.534+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=941679587}) 2024-09-16T03:56:19.539 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.534+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1161585358}) 2024-09-16T03:56:19.541 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.534+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.541 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.534+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=817388712}) 2024-09-16T03:56:19.541 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.534+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=135271578}) 2024-09-16T03:56:19.541 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.534+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.541 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.534+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=817388712}) 2024-09-16T03:56:19.541 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.534+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3532909937}) 2024-09-16T03:56:19.542 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.538+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.542 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.538+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3466205728}) 2024-09-16T03:56:19.542 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.538+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2455213712}) 2024-09-16T03:56:19.542 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.538+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.543 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.538+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3466205728}) 2024-09-16T03:56:19.543 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.538+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2090020409}) 2024-09-16T03:56:19.544 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.538+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.545 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.538+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1823869433}) 2024-09-16T03:56:19.545 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.538+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=135610299}) 2024-09-16T03:56:19.545 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.538+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.545 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.538+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1823869433}) 2024-09-16T03:56:19.545 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.538+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2723415741}) 2024-09-16T03:56:19.546 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T03:56:19.546 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.542+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.546 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.542+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3194415687}) 2024-09-16T03:56:19.547 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.542+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2184006964}) 2024-09-16T03:56:19.547 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.542+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.547 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.542+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3194415687}) 2024-09-16T03:56:19.547 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.542+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4164555444}) 2024-09-16T03:56:19.548 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.542+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.548 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.542+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3884840799}) 2024-09-16T03:56:19.549 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.542+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3373648183}) 2024-09-16T03:56:19.549 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.542+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.549 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.542+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3884840799}) 2024-09-16T03:56:19.549 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.542+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2542430414}) 2024-09-16T03:56:19.550 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.546+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.550 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.546+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=386943152}) 2024-09-16T03:56:19.550 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.546+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1560456565}) 2024-09-16T03:56:19.551 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.546+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.551 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.546+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=386943152}) 2024-09-16T03:56:19.551 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.546+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1547310382}) 2024-09-16T03:56:19.552 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.546+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.552 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.546+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=643526659}) 2024-09-16T03:56:19.552 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.546+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=44000158}) 2024-09-16T03:56:19.552 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.546+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.553 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.546+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=643526659}) 2024-09-16T03:56:19.553 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.546+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1554818384}) 2024-09-16T03:56:19.554 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.550+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.554 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.550+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2495446048}) 2024-09-16T03:56:19.554 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.550+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1952929998}) 2024-09-16T03:56:19.554 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.550+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.554 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.550+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2495446048}) 2024-09-16T03:56:19.554 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.550+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=767886087}) 2024-09-16T03:56:19.556 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.550+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.556 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.550+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=218985993}) 2024-09-16T03:56:19.556 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.550+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1512085471}) 2024-09-16T03:56:19.556 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.550+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.556 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.550+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=218985993}) 2024-09-16T03:56:19.557 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.550+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1655336955}) 2024-09-16T03:56:19.558 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.554+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.558 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.554+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1194638571}) 2024-09-16T03:56:19.558 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.554+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2736851414}) 2024-09-16T03:56:19.558 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.554+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.558 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.554+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1194638571}) 2024-09-16T03:56:19.558 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.554+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3342052702}) 2024-09-16T03:56:19.559 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.554+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.559 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.554+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=157738478}) 2024-09-16T03:56:19.560 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.554+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=482048681}) 2024-09-16T03:56:19.560 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.554+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.560 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.554+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=157738478}) 2024-09-16T03:56:19.560 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.554+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=375650826}) 2024-09-16T03:56:19.561 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.558+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.561 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.558+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3350963486}) 2024-09-16T03:56:19.562 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.558+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1604620207}) 2024-09-16T03:56:19.562 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.558+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.562 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.558+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3350963486}) 2024-09-16T03:56:19.562 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.558+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3400242452}) 2024-09-16T03:56:19.563 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.558+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.563 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.558+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3074173821}) 2024-09-16T03:56:19.563 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.558+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1107307885}) 2024-09-16T03:56:19.564 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.558+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.564 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.558+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3074173821}) 2024-09-16T03:56:19.564 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.558+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2835056185}) 2024-09-16T03:56:19.565 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.562+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.565 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.562+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1847528841}) 2024-09-16T03:56:19.565 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.562+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2634728464}) 2024-09-16T03:56:19.566 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.562+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.566 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.562+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1847528841}) 2024-09-16T03:56:19.566 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.562+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3623410817}) 2024-09-16T03:56:19.568 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.562+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.568 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.562+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2555871023}) 2024-09-16T03:56:19.568 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.562+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2941927339}) 2024-09-16T03:56:19.568 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.562+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.568 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.562+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2555871023}) 2024-09-16T03:56:19.569 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.562+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2854766166}) 2024-09-16T03:56:19.569 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.566+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.569 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.566+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1141176499}) 2024-09-16T03:56:19.569 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.566+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2950914721}) 2024-09-16T03:56:19.570 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.566+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.570 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.566+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1141176499}) 2024-09-16T03:56:19.570 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.566+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3459983462}) 2024-09-16T03:56:19.571 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.566+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.571 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.566+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2469568300}) 2024-09-16T03:56:19.572 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.566+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3875119527}) 2024-09-16T03:56:19.572 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.566+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.572 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.566+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2469568300}) 2024-09-16T03:56:19.572 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.566+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2891974547}) 2024-09-16T03:56:19.573 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.570+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.573 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.570+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1490097584}) 2024-09-16T03:56:19.573 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.570+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=135634875}) 2024-09-16T03:56:19.574 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.570+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.574 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.570+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1490097584}) 2024-09-16T03:56:19.574 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.570+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=470229490}) 2024-09-16T03:56:19.575 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.570+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.576 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.570+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=355482387}) 2024-09-16T03:56:19.576 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.570+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3104611308}) 2024-09-16T03:56:19.576 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.570+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.576 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.570+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=355482387}) 2024-09-16T03:56:19.576 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.570+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1341466583}) 2024-09-16T03:56:19.577 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.574+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.577 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.574+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3257551100}) 2024-09-16T03:56:19.577 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.574+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1247303337}) 2024-09-16T03:56:19.578 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.574+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.578 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.574+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3257551100}) 2024-09-16T03:56:19.578 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.574+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1247997102}) 2024-09-16T03:56:19.579 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.574+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.579 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.574+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=171908838}) 2024-09-16T03:56:19.579 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.574+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3304957921}) 2024-09-16T03:56:19.579 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.574+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.579 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.574+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=171908838}) 2024-09-16T03:56:19.579 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.574+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=574159163}) 2024-09-16T03:56:19.581 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.574+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.581 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.578+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=248056256}) 2024-09-16T03:56:19.581 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.578+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1221011131}) 2024-09-16T03:56:19.581 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.578+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.582 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.578+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=248056256}) 2024-09-16T03:56:19.582 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.578+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1914574893}) 2024-09-16T03:56:19.584 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.578+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.584 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.578+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=196886894}) 2024-09-16T03:56:19.584 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.578+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=164442049}) 2024-09-16T03:56:19.585 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.578+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.585 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.578+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=196886894}) 2024-09-16T03:56:19.585 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.578+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3074928245}) 2024-09-16T03:56:19.586 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.582+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.587 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.582+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=335646709}) 2024-09-16T03:56:19.587 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.582+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3754789428}) 2024-09-16T03:56:19.587 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.582+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.587 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.582+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=335646709}) 2024-09-16T03:56:19.587 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.582+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2346746616}) 2024-09-16T03:56:19.589 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.582+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.589 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.582+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3681959835}) 2024-09-16T03:56:19.589 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.582+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=306936423}) 2024-09-16T03:56:19.589 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.582+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.589 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.582+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3681959835}) 2024-09-16T03:56:19.589 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.582+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4276020459}) 2024-09-16T03:56:19.590 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.586+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.590 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.586+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2438278191}) 2024-09-16T03:56:19.591 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.586+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2367219304}) 2024-09-16T03:56:19.591 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.586+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.591 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.586+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2438278191}) 2024-09-16T03:56:19.591 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.586+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2840464890}) 2024-09-16T03:56:19.592 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.586+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.593 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.586+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2821344698}) 2024-09-16T03:56:19.593 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.586+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3921191535}) 2024-09-16T03:56:19.593 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.586+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.593 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.586+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2821344698}) 2024-09-16T03:56:19.593 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.586+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1195025581}) 2024-09-16T03:56:19.594 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.590+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.594 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.590+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2000882314}) 2024-09-16T03:56:19.594 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.590+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2647299446}) 2024-09-16T03:56:19.594 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.590+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.595 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.590+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2000882314}) 2024-09-16T03:56:19.595 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.590+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=539660499}) 2024-09-16T03:56:19.596 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.590+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.597 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.590+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1493825678}) 2024-09-16T03:56:19.597 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.590+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1763241462}) 2024-09-16T03:56:19.597 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.590+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.597 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.590+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1493825678}) 2024-09-16T03:56:19.597 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.590+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1082646945}) 2024-09-16T03:56:19.598 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.594+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.598 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.594+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4221958350}) 2024-09-16T03:56:19.598 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.594+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2429903776}) 2024-09-16T03:56:19.598 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.594+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.598 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.594+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4221958350}) 2024-09-16T03:56:19.598 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.594+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2365957075}) 2024-09-16T03:56:19.600 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.594+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.600 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.594+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=556574105}) 2024-09-16T03:56:19.600 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.594+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1537833617}) 2024-09-16T03:56:19.601 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.594+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.601 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.594+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=556574105}) 2024-09-16T03:56:19.601 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.594+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=182105497}) 2024-09-16T03:56:19.602 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.598+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.602 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.598+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2680247302}) 2024-09-16T03:56:19.602 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.598+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1954663445}) 2024-09-16T03:56:19.602 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.598+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.602 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.598+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2680247302}) 2024-09-16T03:56:19.602 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.598+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2809781493}) 2024-09-16T03:56:19.604 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.598+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.604 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.598+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1233097267}) 2024-09-16T03:56:19.604 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.598+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1549528956}) 2024-09-16T03:56:19.604 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.598+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.604 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.598+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1233097267}) 2024-09-16T03:56:19.604 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.598+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=149952240}) 2024-09-16T03:56:19.606 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.602+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.606 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.602+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1976942337}) 2024-09-16T03:56:19.606 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.602+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1852310917}) 2024-09-16T03:56:19.606 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.602+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.607 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.602+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1976942337}) 2024-09-16T03:56:19.607 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.602+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2823688559}) 2024-09-16T03:56:19.609 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.602+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.609 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.602+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3391753258}) 2024-09-16T03:56:19.609 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.602+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1543918215}) 2024-09-16T03:56:19.609 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.602+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.610 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.602+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3391753258}) 2024-09-16T03:56:19.610 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.602+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2701789729}) 2024-09-16T03:56:19.610 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.606+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.611 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.606+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1754936516}) 2024-09-16T03:56:19.611 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.606+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=514428400}) 2024-09-16T03:56:19.611 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.606+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.611 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.606+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1754936516}) 2024-09-16T03:56:19.611 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.606+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=574728561}) 2024-09-16T03:56:19.612 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.606+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.612 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.606+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1032143817}) 2024-09-16T03:56:19.613 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.606+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4064136839}) 2024-09-16T03:56:19.613 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.606+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.613 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.606+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1032143817}) 2024-09-16T03:56:19.613 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.606+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3245542513}) 2024-09-16T03:56:19.614 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.610+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.614 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.610+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4241946049}) 2024-09-16T03:56:19.614 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.610+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=472262711}) 2024-09-16T03:56:19.614 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.610+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.615 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.610+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4241946049}) 2024-09-16T03:56:19.615 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.610+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1669071253}) 2024-09-16T03:56:19.617 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.610+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.617 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.610+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=164378262}) 2024-09-16T03:56:19.617 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.610+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3506986014}) 2024-09-16T03:56:19.617 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.610+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.617 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.610+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=164378262}) 2024-09-16T03:56:19.617 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.610+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4038342154}) 2024-09-16T03:56:19.618 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.614+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.618 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.614+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1625404385}) 2024-09-16T03:56:19.618 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.614+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=301863260}) 2024-09-16T03:56:19.618 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.614+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.618 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.614+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1625404385}) 2024-09-16T03:56:19.619 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.614+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2647168323}) 2024-09-16T03:56:19.620 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.614+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.621 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.614+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3240024474}) 2024-09-16T03:56:19.621 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.614+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2235899097}) 2024-09-16T03:56:19.621 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.614+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.621 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.614+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3240024474}) 2024-09-16T03:56:19.621 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.614+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3616775703}) 2024-09-16T03:56:19.622 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.618+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.622 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.618+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=413662761}) 2024-09-16T03:56:19.622 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.618+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1864775046}) 2024-09-16T03:56:19.622 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.618+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.622 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.618+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=413662761}) 2024-09-16T03:56:19.623 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.618+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3247795222}) 2024-09-16T03:56:19.624 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.618+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.624 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.618+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4037040863}) 2024-09-16T03:56:19.624 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.618+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1736667179}) 2024-09-16T03:56:19.624 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.618+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.625 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.618+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4037040863}) 2024-09-16T03:56:19.625 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.618+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3645122335}) 2024-09-16T03:56:19.626 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.622+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.626 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.622+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2691154122}) 2024-09-16T03:56:19.626 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.622+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2796428032}) 2024-09-16T03:56:19.626 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.622+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.626 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.622+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2691154122}) 2024-09-16T03:56:19.626 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.622+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=926333324}) 2024-09-16T03:56:19.627 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.622+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.628 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.622+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3549841024}) 2024-09-16T03:56:19.628 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.622+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=224010636}) 2024-09-16T03:56:19.628 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.622+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.628 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.622+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3549841024}) 2024-09-16T03:56:19.628 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.622+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3426323943}) 2024-09-16T03:56:19.630 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.626+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.630 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.626+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2338314366}) 2024-09-16T03:56:19.631 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.626+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=818840692}) 2024-09-16T03:56:19.631 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.626+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.631 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.626+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2338314366}) 2024-09-16T03:56:19.631 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.626+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4084759871}) 2024-09-16T03:56:19.631 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.626+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.631 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.626+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1394468337}) 2024-09-16T03:56:19.631 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.626+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2078570818}) 2024-09-16T03:56:19.632 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.626+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.632 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.626+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1394468337}) 2024-09-16T03:56:19.632 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.626+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1346685684}) 2024-09-16T03:56:19.633 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.626+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.633 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.626+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3921980438}) 2024-09-16T03:56:19.633 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.626+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4036437678}) 2024-09-16T03:56:19.633 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.626+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.633 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.626+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3921980438}) 2024-09-16T03:56:19.633 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.626+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3861025759}) 2024-09-16T03:56:19.635 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.630+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.635 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.630+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=900079435}) 2024-09-16T03:56:19.635 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.630+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1682356023}) 2024-09-16T03:56:19.635 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.630+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.635 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.630+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=900079435}) 2024-09-16T03:56:19.635 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.630+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=613603944}) 2024-09-16T03:56:19.636 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.630+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.636 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.630+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2360568773}) 2024-09-16T03:56:19.636 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.630+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1208410344}) 2024-09-16T03:56:19.636 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.630+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.636 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.630+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2360568773}) 2024-09-16T03:56:19.637 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.630+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1014080163}) 2024-09-16T03:56:19.638 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.634+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.638 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.634+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1797023270}) 2024-09-16T03:56:19.638 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.634+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2665845930}) 2024-09-16T03:56:19.638 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.634+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.638 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.634+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1797023270}) 2024-09-16T03:56:19.638 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.634+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2501552973}) 2024-09-16T03:56:19.639 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.634+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.639 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.634+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3395657631}) 2024-09-16T03:56:19.639 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.634+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1116792402}) 2024-09-16T03:56:19.639 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.634+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.640 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.634+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3395657631}) 2024-09-16T03:56:19.640 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.634+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2670023100}) 2024-09-16T03:56:19.641 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.634+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.641 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.634+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=946928840}) 2024-09-16T03:56:19.641 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.634+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2391421505}) 2024-09-16T03:56:19.641 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.634+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.641 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.634+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=946928840}) 2024-09-16T03:56:19.642 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.634+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2329274105}) 2024-09-16T03:56:19.642 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.638+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.643 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.638+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2468103866}) 2024-09-16T03:56:19.643 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.638+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4139346096}) 2024-09-16T03:56:19.643 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.638+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.643 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.638+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2468103866}) 2024-09-16T03:56:19.643 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.638+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=641030672}) 2024-09-16T03:56:19.644 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.638+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.644 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.638+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=568617292}) 2024-09-16T03:56:19.644 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.638+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4285263224}) 2024-09-16T03:56:19.644 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.638+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.644 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.638+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=568617292}) 2024-09-16T03:56:19.645 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.638+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1594819813}) 2024-09-16T03:56:19.646 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.642+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.646 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.642+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=850581429}) 2024-09-16T03:56:19.646 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.642+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3997349653}) 2024-09-16T03:56:19.646 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.642+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.646 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.642+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=850581429}) 2024-09-16T03:56:19.646 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.642+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3449545780}) 2024-09-16T03:56:19.647 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.642+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.647 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.642+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2090471235}) 2024-09-16T03:56:19.647 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.642+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=459472505}) 2024-09-16T03:56:19.648 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.642+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.648 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.642+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2090471235}) 2024-09-16T03:56:19.648 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.642+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2272784103}) 2024-09-16T03:56:19.649 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.646+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.649 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.646+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3144927714}) 2024-09-16T03:56:19.650 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.646+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2133401307}) 2024-09-16T03:56:19.650 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.646+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.650 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.646+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3144927714}) 2024-09-16T03:56:19.650 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.646+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2057794091}) 2024-09-16T03:56:19.652 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.646+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.652 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.646+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=401944019}) 2024-09-16T03:56:19.652 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.646+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=841794441}) 2024-09-16T03:56:19.652 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.646+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.652 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.646+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=401944019}) 2024-09-16T03:56:19.652 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.646+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3860916975}) 2024-09-16T03:56:19.654 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.650+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.654 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.650+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3100743799}) 2024-09-16T03:56:19.654 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.650+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3789083231}) 2024-09-16T03:56:19.654 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.650+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.654 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.650+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3100743799}) 2024-09-16T03:56:19.654 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.650+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3868709655}) 2024-09-16T03:56:19.656 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.650+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.656 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.650+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3715649746}) 2024-09-16T03:56:19.656 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.650+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2845220229}) 2024-09-16T03:56:19.656 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.650+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.656 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.650+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3715649746}) 2024-09-16T03:56:19.656 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.650+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2046042883}) 2024-09-16T03:56:19.658 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.654+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.658 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.654+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=600212728}) 2024-09-16T03:56:19.658 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.654+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4067241204}) 2024-09-16T03:56:19.658 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.654+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.658 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.654+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=600212728}) 2024-09-16T03:56:19.658 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.654+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2336605159}) 2024-09-16T03:56:19.660 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.654+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.660 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.654+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3453702003}) 2024-09-16T03:56:19.660 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.654+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3125431804}) 2024-09-16T03:56:19.660 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.654+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.660 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.654+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3453702003}) 2024-09-16T03:56:19.660 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.654+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2586721045}) 2024-09-16T03:56:19.662 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.658+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.662 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.658+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1559259972}) 2024-09-16T03:56:19.662 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.658+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=643706717}) 2024-09-16T03:56:19.662 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.658+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.662 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.658+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1559259972}) 2024-09-16T03:56:19.662 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.658+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4254756714}) 2024-09-16T03:56:19.664 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.658+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.664 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.658+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2114539492}) 2024-09-16T03:56:19.664 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.658+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2639103300}) 2024-09-16T03:56:19.664 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.658+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.664 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.658+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2114539492}) 2024-09-16T03:56:19.664 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.658+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1091301664}) 2024-09-16T03:56:19.666 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.662+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.666 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.662+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4203752308}) 2024-09-16T03:56:19.666 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.662+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4081752718}) 2024-09-16T03:56:19.666 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.662+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.666 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.662+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4203752308}) 2024-09-16T03:56:19.666 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.662+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3005549263}) 2024-09-16T03:56:19.668 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.662+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.668 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.662+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=796823497}) 2024-09-16T03:56:19.668 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.662+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=314060785}) 2024-09-16T03:56:19.668 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.662+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.668 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.662+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=796823497}) 2024-09-16T03:56:19.668 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.662+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1179673463}) 2024-09-16T03:56:19.670 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.666+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.670 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.666+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=532066022}) 2024-09-16T03:56:19.670 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.666+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1746127292}) 2024-09-16T03:56:19.670 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.666+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.670 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.666+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=532066022}) 2024-09-16T03:56:19.670 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.666+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3691069634}) 2024-09-16T03:56:19.672 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.666+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.672 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.666+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3893571646}) 2024-09-16T03:56:19.672 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.666+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3733498871}) 2024-09-16T03:56:19.672 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.666+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.672 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.666+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3893571646}) 2024-09-16T03:56:19.673 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.666+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=694045465}) 2024-09-16T03:56:19.674 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.670+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.674 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.670+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2482618230}) 2024-09-16T03:56:19.674 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.670+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3153575203}) 2024-09-16T03:56:19.674 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.670+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.674 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.670+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2482618230}) 2024-09-16T03:56:19.674 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.670+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=516443115}) 2024-09-16T03:56:19.676 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.670+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.676 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.670+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3564045239}) 2024-09-16T03:56:19.676 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.670+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3931082655}) 2024-09-16T03:56:19.676 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.670+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.676 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.670+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3564045239}) 2024-09-16T03:56:19.676 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.670+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=56393508}) 2024-09-16T03:56:19.677 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.674+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.678 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.674+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3453310011}) 2024-09-16T03:56:19.678 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.674+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4168718416}) 2024-09-16T03:56:19.678 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.674+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.678 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.674+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3453310011}) 2024-09-16T03:56:19.678 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.674+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1189176986}) 2024-09-16T03:56:19.680 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.674+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.680 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.674+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=790234872}) 2024-09-16T03:56:19.680 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.674+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=81283339}) 2024-09-16T03:56:19.680 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.674+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.680 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.674+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=790234872}) 2024-09-16T03:56:19.680 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.674+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2440968468}) 2024-09-16T03:56:19.682 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.678+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.682 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.678+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=927049620}) 2024-09-16T03:56:19.682 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.678+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=880043036}) 2024-09-16T03:56:19.682 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.678+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.683 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.678+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=927049620}) 2024-09-16T03:56:19.683 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.678+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1961989152}) 2024-09-16T03:56:19.684 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.678+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.684 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.678+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=93,mdsmap=3,mgr=4} crc {logm=2655642049,mdsmap=4088524775,mgr=2296812847}) 2024-09-16T03:56:19.685 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.678+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=92,mdsmap=3,mgr=5} crc {logm=3291302503,mdsmap=4088524775,mgr=3544355776}) 2024-09-16T03:56:19.685 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.678+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.685 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.678+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=93,mdsmap=3,mgr=4} crc {logm=2655642049,mdsmap=4088524775,mgr=2296812847}) 2024-09-16T03:56:19.685 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.678+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=90,mdsmap=3,mgr=7} crc {logm=3565326438,mdsmap=4088524775,mgr=390803354}) 2024-09-16T03:56:19.687 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.682+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.687 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.682+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgr=4,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=34,osdmap=29} crc {mgr=2243171706,mgr_command_descs=2107673600,mgr_metadata=1367272202,mgrstat=1020246790,mon_config_key=85057112,monmap=3937513981,osd_metadata=4182959540,osd_pg_creating=465202155,osd_snap=3973512030,osdmap=2539669342}) 2024-09-16T03:56:19.688 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.682+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mgr=4,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=10,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=33,osdmap=31} crc {mgr=2243171706,mgr_command_descs=2107673600,mgr_metadata=1367272202,mgrstat=1020246790,mon_config_key=3148957187,monmap=3937513981,osd_metadata=4182959540,osd_pg_creating=465202155,osd_snap=1991302640,osdmap=1631877364}) 2024-09-16T03:56:19.688 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.682+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.688 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.682+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgr=4,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=34,osdmap=29} crc {mgr=2243171706,mgr_command_descs=2107673600,mgr_metadata=1367272202,mgrstat=1020246790,mon_config_key=85057112,monmap=3937513981,osd_metadata=4182959540,osd_pg_creating=465202155,osd_snap=3973512030,osdmap=2539669342}) 2024-09-16T03:56:19.688 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.682+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mgr=4,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=33,osdmap=30} crc {mgr=2243171706,mgr_command_descs=2107673600,mgr_metadata=1367272202,mgrstat=1020246790,mon_config_key=3228664050,monmap=3937513981,osd_metadata=4182959540,osd_pg_creating=465202155,osd_snap=431280670,osdmap=2913867659}) 2024-09-16T03:56:19.691 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.686+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.691 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.686+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2835960660}) 2024-09-16T03:56:19.692 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.686+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2963401391}) 2024-09-16T03:56:19.692 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.686+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.692 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.686+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2835960660}) 2024-09-16T03:56:19.692 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.686+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2891199540}) 2024-09-16T03:56:19.708 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.702+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.708 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.702+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=48} crc {osdmap=3436459710}) 2024-09-16T03:56:19.709 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.702+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=49} crc {osdmap=1784482605}) 2024-09-16T03:56:19.709 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.702+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:56:19.709 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.702+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=48} crc {osdmap=3436459710}) 2024-09-16T03:56:19.709 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:56:19.702+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=48} crc {osdmap=415346808}) 2024-09-16T03:56:20.547 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T03:56:20.547 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:56:20.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.674+0000 7fac55cb3640 10 monclient: get_monmap_and_config 2024-09-16T03:56:20.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.674+0000 7fac55cb3640 10 monclient: build_initial_monmap 2024-09-16T03:56:20.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.674+0000 7fac55cb3640 10 monclient: monmap: 2024-09-16T03:56:20.679 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:56:20.679 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:56:20.679 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:56:20.677879+0000 2024-09-16T03:56:20.680 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:56:20.677879+0000 2024-09-16T03:56:20.680 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:56:20.680 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:56:20.680 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:56:20.680 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:56:20.680 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:56:20.680 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:56:20.680 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:56:20.680 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:56:20.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.674+0000 7fac55cb3640 1 Processor -- start 2024-09-16T03:56:20.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.674+0000 7fac55cb3640 1 -- start start 2024-09-16T03:56:20.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.674+0000 7fac55cb3640 10 monclient: init 2024-09-16T03:56:20.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.674+0000 7fac55cb3640 10 monclient: _reopen_session rank -1 2024-09-16T03:56:20.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.674+0000 7fac55cb3640 10 monclient: _add_conns ranks=[2,1,0,4,3] 2024-09-16T03:56:20.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.674+0000 7fac55cb3640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fac500a39b0 0x7fac50133b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:56:20.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.674+0000 7fac55cb3640 10 monclient(hunting): picked mon.noname-b con 0x7fac500a39b0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:56:20.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.674+0000 7fac55cb3640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fac500a3d90 addr v1:172.21.15.154:6789/0 2024-09-16T03:56:20.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.674+0000 7fac55cb3640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fac50137710 addr v1:172.21.15.42:6789/0 2024-09-16T03:56:20.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.674+0000 7fac55cb3640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fac5009c140 con 0x7fac50137710 2024-09-16T03:56:20.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.674+0000 7fac55cb3640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fac5009c420 con 0x7fac500a3d90 2024-09-16T03:56:20.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.674+0000 7fac55cb3640 10 monclient(hunting): start opening mon connection 2024-09-16T03:56:20.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.674+0000 7fac55cb3640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fac5006dd60 con 0x7fac500a39b0 2024-09-16T03:56:20.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.674+0000 7fac55cb3640 10 monclient(hunting): _renew_subs 2024-09-16T03:56:20.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.674+0000 7fac55cb3640 10 monclient(hunting): authenticate will time out at 1827.647350s 2024-09-16T03:56:20.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.674+0000 7fac4f7fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fac500a39b0 0x7fac50133b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:56:20.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.674+0000 7fac4ffff640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fac50137710 0x7fac5013f3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:33370/0 (socket says 172.21.15.42:33370) 2024-09-16T03:56:20.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.674+0000 7fac4ffff640 1 -- 172.21.15.42:0/3979485492 learned_addr learned my addr 172.21.15.42:0/3979485492 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:56:20.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.674+0000 7fac4f7fe640 10 monclient(hunting): get_auth_request con 0x7fac500a39b0 auth_method 0 2024-09-16T03:56:20.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.674+0000 7fac4f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:56:20.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.674+0000 7fac4f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:56:20.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.674+0000 7fac4f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:56:20.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.674+0000 7fac4f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:56:20.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.674+0000 7fac4f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:56:20.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.674+0000 7fac4f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:56:20.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.674+0000 7fac4e7fc640 1 -- 172.21.15.42:0/3979485492 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3909678587 0 0) 0x7fac5009c140 con 0x7fac50137710 2024-09-16T03:56:20.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.674+0000 7fac4e7fc640 10 monclient(hunting): _negotiate 2024-09-16T03:56:20.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.674+0000 7fac4e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:56:20.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.674+0000 7fac4e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:56:20.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.674+0000 7fac4e7fc640 10 monclient(hunting): my global_id is 824718 2024-09-16T03:56:20.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.674+0000 7fac4e7fc640 1 -- 172.21.15.42:0/3979485492 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fac3c003530 con 0x7fac50137710 2024-09-16T03:56:20.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.674+0000 7fac4f7fe640 10 monclient(hunting): handle_auth_done global_id 854306 payload 274 2024-09-16T03:56:20.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.674+0000 7fac4f7fe640 1 -- 172.21.15.42:0/3979485492 >> v1:172.21.15.154:6789/0 conn(0x7fac500a3d90 legacy=0x7fac50137000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:56:20.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.678+0000 7fac4f7fe640 1 -- 172.21.15.42:0/3979485492 >> v1:172.21.15.42:6789/0 conn(0x7fac50137710 legacy=0x7fac5013f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:56:20.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.678+0000 7fac4f7fe640 10 monclient: _finish_hunting 0 2024-09-16T03:56:20.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.678+0000 7fac4f7fe640 1 monclient: found mon.noname-b 2024-09-16T03:56:20.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.678+0000 7fac4f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:56:20.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.678+0000 7fac4f7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T03:56:20.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.678+0000 7fac4f7fe640 1 -- 172.21.15.42:0/3979485492 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac500a5e20 con 0x7fac500a39b0 2024-09-16T03:56:20.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.678+0000 7fac4f7fe640 1 --2- 172.21.15.42:0/3979485492 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fac500a39b0 0x7fac50133b80 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fac40009dc0 tx=0x7fac4001c810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f9c0edd3eb7c0d9a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:56:20.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.678+0000 7fac4e7fc640 1 -- 172.21.15.42:0/3979485492 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac400a66e0 con 0x7fac500a39b0 2024-09-16T03:56:20.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.678+0000 7fac4e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:56:20.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.678+0000 7fac4e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:56:20.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.678+0000 7fac4e7fc640 10 monclient: dump: 2024-09-16T03:56:20.691 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:56:20.691 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:56:20.692 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:20.692 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:20.692 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:56:20.692 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:56:20.692 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:56:20.692 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:56:20.692 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:56:20.692 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:56:20.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.678+0000 7fac4e7fc640 10 monclient: _finish_auth 0 2024-09-16T03:56:20.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.678+0000 7fac4e7fc640 10 monclient: _check_auth_tickets 2024-09-16T03:56:20.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.678+0000 7fac4e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:56:20.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.678+0000 7fac4e7fc640 1 -- 172.21.15.42:0/3979485492 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac400a6880 con 0x7fac500a39b0 2024-09-16T03:56:20.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.678+0000 7fac55cb3640 5 monclient: authenticate success, global_id 854306 2024-09-16T03:56:20.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.678+0000 7fac55cb3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:56:20.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.678+0000 7fac4e7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:56:20.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.678+0000 7fac4e7fc640 1 -- 172.21.15.42:0/3979485492 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac400a6c50 con 0x7fac500a39b0 2024-09-16T03:56:20.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.678+0000 7fac55cb3640 10 monclient: get_monmap_and_config success 2024-09-16T03:56:20.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.678+0000 7fac55cb3640 4 set_mon_vals no callback set 2024-09-16T03:56:20.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.678+0000 7fac4e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:56:20.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.678+0000 7fac4e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:56:20.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.678+0000 7fac4e7fc640 10 monclient: dump: 2024-09-16T03:56:20.693 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:56:20.693 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:56:20.693 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:20.693 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:20.693 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:56:20.694 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:56:20.694 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:56:20.694 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:56:20.694 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:56:20.694 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:56:20.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.678+0000 7fac55cb3640 10 monclient: shutdown 2024-09-16T03:56:20.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.678+0000 7fac55cb3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:56:20.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.678+0000 7fac55cb3640 1 -- 172.21.15.42:0/3979485492 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fac500a39b0 msgr2=0x7fac50133b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:56:20.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.678+0000 7fac55cb3640 1 --2- 172.21.15.42:0/3979485492 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fac500a39b0 0x7fac50133b80 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fac40009dc0 tx=0x7fac4001c810 comp rx=0 tx=0).stop 2024-09-16T03:56:20.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.678+0000 7fac55cb3640 1 -- 172.21.15.42:0/3979485492 shutdown_connections 2024-09-16T03:56:20.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.678+0000 7fac55cb3640 1 --2- 172.21.15.42:0/3979485492 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fac500a39b0 0x7fac50133b80 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:56:20.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.678+0000 7fac55cb3640 1 -- 172.21.15.42:0/3979485492 >> 172.21.15.42:0/3979485492 conn(0x7fac500a9fb0 msgr2=0x7fac500aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:56:20.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.678+0000 7fac55cb3640 1 -- 172.21.15.42:0/3979485492 shutdown_connections 2024-09-16T03:56:20.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.678+0000 7fac55cb3640 1 -- 172.21.15.42:0/3979485492 wait complete. 2024-09-16T03:56:20.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.682+0000 7fac55cb3640 10 monclient: build_initial_monmap 2024-09-16T03:56:20.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.682+0000 7fac55cb3640 10 monclient: monmap: 2024-09-16T03:56:20.695 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:56:20.695 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:56:20.695 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:56:20.695 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:56:20.695 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:56:20.695 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:56:20.695 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:56:20.695 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:56:20.695 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:56:20.696 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:56:20.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.682+0000 7fac55cb3640 1 librados: starting msgr at 2024-09-16T03:56:20.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.682+0000 7fac55cb3640 1 librados: starting objecter 2024-09-16T03:56:20.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.682+0000 7fac55cb3640 1 Processor -- start 2024-09-16T03:56:20.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.682+0000 7fac55cb3640 1 -- start start 2024-09-16T03:56:20.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.682+0000 7fac55cb3640 1 librados: setting wanted keys 2024-09-16T03:56:20.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.682+0000 7fac55cb3640 1 librados: calling monclient init 2024-09-16T03:56:20.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.682+0000 7fac55cb3640 10 monclient: init 2024-09-16T03:56:20.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.682+0000 7fac55cb3640 10 monclient: _reopen_session rank -1 2024-09-16T03:56:20.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.682+0000 7fac55cb3640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:56:20.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.682+0000 7fac55cb3640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fac500a39b0 0x7fac501cfd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:56:20.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.682+0000 7fac55cb3640 10 monclient(hunting): picked mon.noname-a con 0x7fac500a39b0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:56:20.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.682+0000 7fac55cb3640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fac500a3d90 0x7fac501d02d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:56:20.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.682+0000 7fac55cb3640 10 monclient(hunting): picked mon.noname-c con 0x7fac500a3d90 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:56:20.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.682+0000 7fac55cb3640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fac50137710 0x7fac500a1330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:56:20.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.682+0000 7fac55cb3640 10 monclient(hunting): picked mon.noname-b con 0x7fac50137710 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:56:20.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.682+0000 7fac55cb3640 10 monclient(hunting): start opening mon connection 2024-09-16T03:56:20.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.682+0000 7fac55cb3640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fac50143900 con 0x7fac500a39b0 2024-09-16T03:56:20.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.682+0000 7fac55cb3640 10 monclient(hunting): start opening mon connection 2024-09-16T03:56:20.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.682+0000 7fac55cb3640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fac5006dee0 con 0x7fac50137710 2024-09-16T03:56:20.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.682+0000 7fac55cb3640 10 monclient(hunting): start opening mon connection 2024-09-16T03:56:20.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.682+0000 7fac55cb3640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fac50143a80 con 0x7fac500a3d90 2024-09-16T03:56:20.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.682+0000 7fac55cb3640 10 monclient(hunting): _renew_subs 2024-09-16T03:56:20.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.682+0000 7fac55cb3640 10 monclient(hunting): authenticate will time out at 1827.653201s 2024-09-16T03:56:20.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.682+0000 7fac4f7fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fac500a39b0 0x7fac501cfd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:56:20.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.682+0000 7fac4f7fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fac500a39b0 0x7fac501cfd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:37080/0 (socket says 172.21.15.42:37080) 2024-09-16T03:56:20.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.682+0000 7fac4f7fe640 1 -- 172.21.15.42:0/2332820841 learned_addr learned my addr 172.21.15.42:0/2332820841 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:56:20.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.682+0000 7fac4f7fe640 10 monclient(hunting): get_auth_request con 0x7fac500a39b0 auth_method 0 2024-09-16T03:56:20.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.682+0000 7fac4f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:56:20.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.682+0000 7fac4f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:56:20.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.682+0000 7fac4f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:56:20.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.682+0000 7fac4effd640 1 --2- 172.21.15.42:0/2332820841 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fac500a3d90 0x7fac501d02d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:56:20.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.682+0000 7fac4effd640 10 monclient(hunting): get_auth_request con 0x7fac500a3d90 auth_method 0 2024-09-16T03:56:20.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.682+0000 7fac4effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:56:20.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.682+0000 7fac4effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:56:20.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.682+0000 7fac4effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:56:20.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.682+0000 7fac4f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:56:20.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.682+0000 7fac4f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:56:20.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.682+0000 7fac4f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:56:20.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.682+0000 7fac4effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:56:20.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.682+0000 7fac4effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:56:20.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.682+0000 7fac4effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:56:20.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.682+0000 7fac4ffff640 1 --2- 172.21.15.42:0/2332820841 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fac50137710 0x7fac500a1330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:56:20.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.682+0000 7fac4ffff640 10 monclient(hunting): get_auth_request con 0x7fac50137710 auth_method 0 2024-09-16T03:56:20.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.682+0000 7fac4ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:56:20.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.682+0000 7fac4ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:56:20.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.682+0000 7fac4ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:56:20.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.682+0000 7fac4f7fe640 10 monclient(hunting): handle_auth_done global_id 824721 payload 824 2024-09-16T03:56:20.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.682+0000 7fac4f7fe640 1 -- 172.21.15.42:0/2332820841 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fac500a3d90 msgr2=0x7fac501d02d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:56:20.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.682+0000 7fac4f7fe640 1 --2- 172.21.15.42:0/2332820841 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fac500a3d90 0x7fac501d02d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:56:20.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.682+0000 7fac4f7fe640 1 -- 172.21.15.42:0/2332820841 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fac50137710 msgr2=0x7fac500a1330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:56:20.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.682+0000 7fac4f7fe640 1 --2- 172.21.15.42:0/2332820841 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fac50137710 0x7fac500a1330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:56:20.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.682+0000 7fac4f7fe640 10 monclient: _finish_hunting 0 2024-09-16T03:56:20.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.682+0000 7fac4f7fe640 1 monclient: found mon.noname-a 2024-09-16T03:56:20.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.682+0000 7fac4f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:56:20.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.682+0000 7fac4f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:56:20.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.682+0000 7fac4f7fe640 1 -- 172.21.15.42:0/2332820841 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac50139770 con 0x7fac500a39b0 2024-09-16T03:56:20.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.682+0000 7fac4effd640 1 --2- 172.21.15.42:0/2332820841 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fac500a3d90 0x7fac501d02d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:56:20.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.682+0000 7fac4ffff640 1 --2- 172.21.15.42:0/2332820841 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fac50137710 0x7fac500a1330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T03:56:20.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.682+0000 7fac4f7fe640 1 --2- 172.21.15.42:0/2332820841 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fac500a39b0 0x7fac501cfd90 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fac400b0040 tx=0x7fac400195b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:56:20.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.682+0000 7fac47fff640 10 client.?.objecter ms_handle_connect 0x7fac500a39b0 2024-09-16T03:56:20.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.682+0000 7fac47fff640 10 client.?.objecter resend_mon_ops 2024-09-16T03:56:20.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.682+0000 7fac47fff640 1 -- 172.21.15.42:0/2332820841 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac400be070 con 0x7fac500a39b0 2024-09-16T03:56:20.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.682+0000 7fac47fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:56:20.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.682+0000 7fac47fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:56:20.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.682+0000 7fac47fff640 10 monclient: dump: 2024-09-16T03:56:20.701 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:56:20.701 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:56:20.701 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:20.701 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:20.701 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:56:20.702 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:56:20.702 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:56:20.702 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:56:20.702 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:56:20.702 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:56:20.702 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.682+0000 7fac47fff640 10 monclient: _finish_auth 0 2024-09-16T03:56:20.702 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.682+0000 7fac47fff640 10 monclient: _check_auth_tickets 2024-09-16T03:56:20.702 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.682+0000 7fac47fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:56:20.702 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.682+0000 7fac47fff640 1 -- 172.21.15.42:0/2332820841 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac400b8880 con 0x7fac500a39b0 2024-09-16T03:56:20.702 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.682+0000 7fac47fff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:56:20.702 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.682+0000 7fac47fff640 1 -- 172.21.15.42:0/2332820841 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac400b8c70 con 0x7fac500a39b0 2024-09-16T03:56:20.702 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.682+0000 7fac47fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:56:20.702 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.682+0000 7fac47fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:56:20.702 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.682+0000 7fac47fff640 10 monclient: dump: 2024-09-16T03:56:20.703 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:56:20.703 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:56:20.703 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:20.703 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:20.703 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:56:20.703 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:56:20.703 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:56:20.703 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:56:20.703 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:56:20.703 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:56:20.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.682+0000 7fac55cb3640 5 monclient: authenticate success, global_id 824721 2024-09-16T03:56:20.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.682+0000 7fac55cb3640 10 monclient: _renew_subs 2024-09-16T03:56:20.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.682+0000 7fac4dffb640 4 set_mon_vals no callback set 2024-09-16T03:56:20.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.682+0000 7fac55cb3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:56:20.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.682+0000 7fac55cb3640 1 -- 172.21.15.42:0/2332820841 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac50139a50 con 0x7fac500a39b0 2024-09-16T03:56:20.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.682+0000 7fac55cb3640 10 client.824721.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:56:20.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.682+0000 7fac55cb3640 10 monclient: _renew_subs 2024-09-16T03:56:20.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.682+0000 7fac55cb3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:56:20.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.682+0000 7fac55cb3640 1 -- 172.21.15.42:0/2332820841 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac50137340 con 0x7fac500a39b0 2024-09-16T03:56:20.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.682+0000 7fac55cb3640 1 librados: init done 2024-09-16T03:56:20.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.682+0000 7fac47fff640 1 -- 172.21.15.42:0/2332820841 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fac400c1070 con 0x7fac500a39b0 2024-09-16T03:56:20.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.682+0000 7fac55cb3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:56:20.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.682+0000 7fac55cb3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:56:20.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.682+0000 7fac55cb3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:56:20.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.682+0000 7fac55cb3640 1 -- 172.21.15.42:0/2332820841 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fac5006f680 con 0x7fac500a39b0 2024-09-16T03:56:20.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.682+0000 7fac47fff640 1 --2- 172.21.15.42:0/2332820841 >> v2:172.21.15.154:6812/254845342 conn(0x7fac30037840 0x7fac30039d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:56:20.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.682+0000 7fac47fff640 1 -- 172.21.15.42:0/2332820841 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(442..442 src has 330..442) ==== 5741+0+0 (secure 0 0 0) 0x7fac400e3eb0 con 0x7fac500a39b0 2024-09-16T03:56:20.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.682+0000 7fac47fff640 10 client.824721.objecter ms_dispatch 0x7fac50000ba0 osd_map(442..442 src has 330..442) 2024-09-16T03:56:20.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.682+0000 7fac47fff640 3 client.824721.objecter handle_osd_map got epochs [442,442] > 0 2024-09-16T03:56:20.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.682+0000 7fac47fff640 3 client.824721.objecter handle_osd_map decoding full epoch 442 2024-09-16T03:56:20.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.682+0000 7fac47fff640 20 client.824721.objecter dump_active .. 0 homeless 2024-09-16T03:56:20.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.682+0000 7fac4effd640 1 --2- 172.21.15.42:0/2332820841 >> v2:172.21.15.154:6812/254845342 conn(0x7fac30037840 0x7fac30039d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:56:20.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.682+0000 7fac4effd640 10 monclient: get_auth_request con 0x7fac30037840 auth_method 0 2024-09-16T03:56:20.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.686+0000 7fac4effd640 1 --2- 172.21.15.42:0/2332820841 >> v2:172.21.15.154:6812/254845342 conn(0x7fac30037840 0x7fac30039d00 secure :-1 s=READY pgs=3654 cs=0 l=1 rev1=1 crypto rx=0x7fac50138250 tx=0x7fac38007e40 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:56:20.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.686+0000 7fac47fff640 10 client.824721.objecter ms_handle_connect 0x7fac30037840 2024-09-16T03:56:20.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.690+0000 7fac47fff640 1 -- 172.21.15.42:0/2332820841 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7fac400c8070 con 0x7fac500a39b0 2024-09-16T03:56:20.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.690+0000 7fac47fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:56:20.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.690+0000 7fac47fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:56:20.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.918+0000 7fac55cb3640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:56:20.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.918+0000 7fac55cb3640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:56:20.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.918+0000 7fac55cb3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:56:20.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.918+0000 7fac55cb3640 1 -- 172.21.15.42:0/2332820841 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fac50133b80 con 0x7fac500a39b0 2024-09-16T03:56:20.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.918+0000 7fac47fff640 1 -- 172.21.15.42:0/2332820841 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7fac40016520 con 0x7fac500a39b0 2024-09-16T03:56:20.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.918+0000 7fac47fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:56:20.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.918+0000 7fac47fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:56:20.923 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":128,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":4,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:56:20.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.918+0000 7fac55cb3640 10 librados: watch_flush enter 2024-09-16T03:56:20.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.918+0000 7fac55cb3640 10 librados: watch_flush exit 2024-09-16T03:56:20.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.918+0000 7fac55cb3640 20 client.824721.objecter shutdown clearing up homeless session... 2024-09-16T03:56:20.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.918+0000 7fac55cb3640 10 client.824721.objecter successfully canceled tick 2024-09-16T03:56:20.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.918+0000 7fac55cb3640 1 -- 172.21.15.42:0/2332820841 >> v2:172.21.15.154:6812/254845342 conn(0x7fac30037840 msgr2=0x7fac30039d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:56:20.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.918+0000 7fac55cb3640 1 --2- 172.21.15.42:0/2332820841 >> v2:172.21.15.154:6812/254845342 conn(0x7fac30037840 0x7fac30039d00 secure :-1 s=READY pgs=3654 cs=0 l=1 rev1=1 crypto rx=0x7fac50138250 tx=0x7fac38007e40 comp rx=0 tx=0).stop 2024-09-16T03:56:20.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.918+0000 7fac55cb3640 10 monclient: shutdown 2024-09-16T03:56:20.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.918+0000 7fac55cb3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:56:20.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.918+0000 7fac55cb3640 1 -- 172.21.15.42:0/2332820841 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fac500a39b0 msgr2=0x7fac501cfd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:56:20.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.918+0000 7fac55cb3640 1 --2- 172.21.15.42:0/2332820841 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fac500a39b0 0x7fac501cfd90 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fac400b0040 tx=0x7fac400195b0 comp rx=0 tx=0).stop 2024-09-16T03:56:20.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.918+0000 7fac55cb3640 1 -- 172.21.15.42:0/2332820841 shutdown_connections 2024-09-16T03:56:20.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.918+0000 7fac55cb3640 1 --2- 172.21.15.42:0/2332820841 >> v2:172.21.15.154:6812/254845342 conn(0x7fac30037840 0x7fac30039d00 unknown :-1 s=CLOSED pgs=3654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:56:20.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.918+0000 7fac55cb3640 1 --2- 172.21.15.42:0/2332820841 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fac50137710 0x7fac500a1330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:56:20.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.918+0000 7fac55cb3640 1 --2- 172.21.15.42:0/2332820841 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fac500a3d90 0x7fac501d02d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:56:20.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.918+0000 7fac55cb3640 1 --2- 172.21.15.42:0/2332820841 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fac500a39b0 0x7fac501cfd90 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:56:20.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.918+0000 7fac55cb3640 1 -- 172.21.15.42:0/2332820841 >> 172.21.15.42:0/2332820841 conn(0x7fac500a9fb0 msgr2=0x7fac500a79b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:56:20.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.918+0000 7fac55cb3640 1 -- 172.21.15.42:0/2332820841 shutdown_connections 2024-09-16T03:56:20.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.918+0000 7fac55cb3640 1 -- 172.21.15.42:0/2332820841 wait complete. 2024-09-16T03:56:20.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:20.918+0000 7fac55cb3640 1 librados: shutdown 2024-09-16T03:56:20.942 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T03:56:20.942 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T03:56:20.942 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T03:56:20.942 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T03:56:21.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.054+0000 7f3afae0f640 10 monclient: get_monmap_and_config 2024-09-16T03:56:21.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.054+0000 7f3afae0f640 10 monclient: build_initial_monmap 2024-09-16T03:56:21.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.054+0000 7f3afae0f640 10 monclient: monmap: 2024-09-16T03:56:21.057 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:56:21.057 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:56:21.058 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:56:21.056186+0000 2024-09-16T03:56:21.058 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:56:21.056186+0000 2024-09-16T03:56:21.058 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:56:21.058 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:56:21.058 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:56:21.058 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:56:21.058 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:56:21.058 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:56:21.058 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:56:21.058 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:56:21.060 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.054+0000 7f3afae0f640 1 Processor -- start 2024-09-16T03:56:21.060 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.054+0000 7f3afae0f640 1 -- start start 2024-09-16T03:56:21.060 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.054+0000 7f3afae0f640 10 monclient: init 2024-09-16T03:56:21.060 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.054+0000 7f3afae0f640 10 monclient: _reopen_session rank -1 2024-09-16T03:56:21.060 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.054+0000 7f3afae0f640 10 monclient: _add_conns ranks=[1,2,0,4,3] 2024-09-16T03:56:21.060 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.054+0000 7f3afae0f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3af40a39b0 addr v1:172.21.15.154:6789/0 2024-09-16T03:56:21.060 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.054+0000 7f3afae0f640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3af40a3d90 0x7f3af413c380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:56:21.060 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.054+0000 7f3afae0f640 10 monclient(hunting): picked mon.noname-b con 0x7f3af40a3d90 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:56:21.061 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.054+0000 7f3afae0f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3af413c8c0 addr v1:172.21.15.42:6789/0 2024-09-16T03:56:21.061 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.054+0000 7f3afae0f640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3af4134290 con 0x7f3af413c8c0 2024-09-16T03:56:21.061 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.054+0000 7f3afae0f640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3af409c140 con 0x7f3af40a39b0 2024-09-16T03:56:21.061 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.054+0000 7f3afae0f640 10 monclient(hunting): start opening mon connection 2024-09-16T03:56:21.061 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.054+0000 7f3afae0f640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f3af409c420 con 0x7f3af40a3d90 2024-09-16T03:56:21.061 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.054+0000 7f3afae0f640 10 monclient(hunting): _renew_subs 2024-09-16T03:56:21.061 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.054+0000 7f3afae0f640 10 monclient(hunting): authenticate will time out at 1828.025735s 2024-09-16T03:56:21.061 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.054+0000 7f3af93ac640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f3af413c8c0 0x7f3af413ecb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:33374/0 (socket says 172.21.15.42:33374) 2024-09-16T03:56:21.061 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.054+0000 7f3af93ac640 1 -- 172.21.15.42:0/2073250010 learned_addr learned my addr 172.21.15.42:0/2073250010 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:56:21.061 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.054+0000 7f3af3fff640 1 --2- 172.21.15.42:0/2073250010 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3af40a3d90 0x7f3af413c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:56:21.061 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.054+0000 7f3af3fff640 10 monclient(hunting): get_auth_request con 0x7f3af40a3d90 auth_method 0 2024-09-16T03:56:21.061 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.054+0000 7f3af3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:56:21.061 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.054+0000 7f3af3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:56:21.061 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.054+0000 7f3af3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:56:21.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.054+0000 7f3af3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:56:21.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.054+0000 7f3af3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:56:21.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.054+0000 7f3af3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:56:21.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.054+0000 7f3af3fff640 10 monclient(hunting): handle_auth_done global_id 854312 payload 274 2024-09-16T03:56:21.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.054+0000 7f3af37fe640 1 -- 172.21.15.42:0/2073250010 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1645327314 0 0) 0x7f3af4134290 con 0x7f3af413c8c0 2024-09-16T03:56:21.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.054+0000 7f3af3fff640 1 -- 172.21.15.42:0/2073250010 >> v1:172.21.15.154:6789/0 conn(0x7f3af40a39b0 legacy=0x7f3af4133b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:56:21.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.054+0000 7f3af3fff640 1 -- 172.21.15.42:0/2073250010 >> v1:172.21.15.42:6789/0 conn(0x7f3af413c8c0 legacy=0x7f3af413ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:56:21.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.054+0000 7f3af3fff640 10 monclient: _finish_hunting 0 2024-09-16T03:56:21.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.054+0000 7f3af3fff640 1 monclient: found mon.noname-b 2024-09-16T03:56:21.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.054+0000 7f3af3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:56:21.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.054+0000 7f3af3fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T03:56:21.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.054+0000 7f3af3fff640 1 -- 172.21.15.42:0/2073250010 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3af406dd60 con 0x7f3af40a3d90 2024-09-16T03:56:21.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.054+0000 7f3af37fe640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T03:56:21.068 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.058+0000 7f3af3fff640 1 --2- 172.21.15.42:0/2073250010 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3af40a3d90 0x7f3af413c380 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f3ae40097f0 tx=0x7f3ae401c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b7a3fca97fb48f0e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:56:21.068 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.058+0000 7f3af37fe640 1 -- 172.21.15.42:0/2073250010 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ae40b0070 con 0x7f3af40a3d90 2024-09-16T03:56:21.069 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.058+0000 7f3af37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:56:21.069 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.058+0000 7f3af37fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:56:21.069 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.058+0000 7f3af37fe640 10 monclient: dump: 2024-09-16T03:56:21.069 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:56:21.069 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:56:21.069 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:21.069 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:21.069 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:56:21.069 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:56:21.069 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:56:21.069 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:56:21.069 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:56:21.069 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:56:21.070 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.058+0000 7f3af37fe640 10 monclient: _finish_auth 0 2024-09-16T03:56:21.070 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.058+0000 7f3af37fe640 10 monclient: _check_auth_tickets 2024-09-16T03:56:21.070 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.058+0000 7f3af37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:56:21.070 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.058+0000 7f3af37fe640 1 -- 172.21.15.42:0/2073250010 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3ae40ab600 con 0x7f3af40a3d90 2024-09-16T03:56:21.070 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.058+0000 7f3af37fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:56:21.070 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.058+0000 7f3af37fe640 1 -- 172.21.15.42:0/2073250010 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ae40ab9d0 con 0x7f3af40a3d90 2024-09-16T03:56:21.070 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.058+0000 7f3afae0f640 5 monclient: authenticate success, global_id 854312 2024-09-16T03:56:21.070 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.058+0000 7f3afae0f640 10 monclient: get_monmap_and_config success 2024-09-16T03:56:21.070 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.058+0000 7f3afae0f640 4 set_mon_vals no callback set 2024-09-16T03:56:21.070 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.058+0000 7f3af37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:56:21.070 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.058+0000 7f3af37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:56:21.070 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.058+0000 7f3af37fe640 10 monclient: dump: 2024-09-16T03:56:21.070 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:56:21.070 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:56:21.071 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:21.071 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:21.071 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:56:21.071 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:56:21.071 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:56:21.071 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:56:21.071 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:56:21.071 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:56:21.071 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.058+0000 7f3afae0f640 10 monclient: shutdown 2024-09-16T03:56:21.071 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.058+0000 7f3afae0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:56:21.071 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.058+0000 7f3afae0f640 1 -- 172.21.15.42:0/2073250010 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3af40a3d90 msgr2=0x7f3af413c380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:56:21.071 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.058+0000 7f3afae0f640 1 --2- 172.21.15.42:0/2073250010 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3af40a3d90 0x7f3af413c380 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f3ae40097f0 tx=0x7f3ae401c3a0 comp rx=0 tx=0).stop 2024-09-16T03:56:21.071 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.058+0000 7f3afae0f640 1 -- 172.21.15.42:0/2073250010 shutdown_connections 2024-09-16T03:56:21.071 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.058+0000 7f3afae0f640 1 --2- 172.21.15.42:0/2073250010 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3af40a3d90 0x7f3af413c380 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:56:21.072 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.058+0000 7f3afae0f640 1 -- 172.21.15.42:0/2073250010 >> 172.21.15.42:0/2073250010 conn(0x7f3af40a9fb0 msgr2=0x7f3af40aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:56:21.072 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.058+0000 7f3afae0f640 1 -- 172.21.15.42:0/2073250010 shutdown_connections 2024-09-16T03:56:21.072 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.058+0000 7f3afae0f640 1 -- 172.21.15.42:0/2073250010 wait complete. 2024-09-16T03:56:21.072 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.058+0000 7f3afae0f640 10 monclient: build_initial_monmap 2024-09-16T03:56:21.072 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.058+0000 7f3afae0f640 10 monclient: monmap: 2024-09-16T03:56:21.072 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:56:21.072 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:56:21.072 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:56:21.072 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:56:21.072 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:56:21.072 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:56:21.072 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:56:21.072 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:56:21.072 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:56:21.073 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:56:21.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.058+0000 7f3afae0f640 1 librados: starting msgr at 2024-09-16T03:56:21.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.058+0000 7f3afae0f640 1 librados: starting objecter 2024-09-16T03:56:21.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.058+0000 7f3afae0f640 1 Processor -- start 2024-09-16T03:56:21.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.058+0000 7f3afae0f640 1 -- start start 2024-09-16T03:56:21.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.058+0000 7f3afae0f640 1 librados: setting wanted keys 2024-09-16T03:56:21.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.058+0000 7f3afae0f640 1 librados: calling monclient init 2024-09-16T03:56:21.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.058+0000 7f3afae0f640 10 monclient: init 2024-09-16T03:56:21.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.058+0000 7f3afae0f640 10 monclient: _reopen_session rank -1 2024-09-16T03:56:21.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.058+0000 7f3afae0f640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:56:21.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.058+0000 7f3afae0f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3af40a39b0 0x7f3af4130300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:56:21.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.058+0000 7f3afae0f640 10 monclient(hunting): picked mon.noname-b con 0x7f3af40a39b0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:56:21.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.058+0000 7f3afae0f640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3af40a3d90 0x7f3af4130840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:56:21.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.058+0000 7f3afae0f640 10 monclient(hunting): picked mon.noname-a con 0x7f3af40a3d90 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:56:21.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.058+0000 7f3afae0f640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3af413c8c0 0x7f3af4130d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:56:21.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.058+0000 7f3afae0f640 10 monclient(hunting): picked mon.noname-c con 0x7f3af413c8c0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:56:21.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.058+0000 7f3afae0f640 10 monclient(hunting): start opening mon connection 2024-09-16T03:56:21.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.058+0000 7f3afae0f640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f3af4142f80 con 0x7f3af40a3d90 2024-09-16T03:56:21.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.058+0000 7f3afae0f640 10 monclient(hunting): start opening mon connection 2024-09-16T03:56:21.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.058+0000 7f3afae0f640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f3af4142e00 con 0x7f3af40a39b0 2024-09-16T03:56:21.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.058+0000 7f3afae0f640 10 monclient(hunting): start opening mon connection 2024-09-16T03:56:21.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.058+0000 7f3afae0f640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f3af4143100 con 0x7f3af413c8c0 2024-09-16T03:56:21.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.058+0000 7f3afae0f640 10 monclient(hunting): _renew_subs 2024-09-16T03:56:21.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.058+0000 7f3afae0f640 10 monclient(hunting): authenticate will time out at 1828.030700s 2024-09-16T03:56:21.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.058+0000 7f3af93ac640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3af413c8c0 0x7f3af4130d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:56:21.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.058+0000 7f3af3fff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3af40a3d90 0x7f3af4130840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:56:21.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.058+0000 7f3af93ac640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3af413c8c0 0x7f3af4130d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:40848/0 (socket says 172.21.15.42:40848) 2024-09-16T03:56:21.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.058+0000 7f3af3fff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3af40a3d90 0x7f3af4130840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:37092/0 (socket says 172.21.15.42:37092) 2024-09-16T03:56:21.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.058+0000 7f3af93ac640 1 -- 172.21.15.42:0/2616561402 learned_addr learned my addr 172.21.15.42:0/2616561402 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:56:21.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.058+0000 7f3af93ac640 10 monclient(hunting): get_auth_request con 0x7f3af413c8c0 auth_method 0 2024-09-16T03:56:21.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.058+0000 7f3af93ac640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:56:21.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.058+0000 7f3af93ac640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:56:21.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.058+0000 7f3af93ac640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:56:21.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.058+0000 7f3af8bab640 1 --2- 172.21.15.42:0/2616561402 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3af40a39b0 0x7f3af4130300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:56:21.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.058+0000 7f3af3fff640 10 monclient(hunting): get_auth_request con 0x7f3af40a3d90 auth_method 0 2024-09-16T03:56:21.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.058+0000 7f3af3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:56:21.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.058+0000 7f3af3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:56:21.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.058+0000 7f3af3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:56:21.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.058+0000 7f3af8bab640 10 monclient(hunting): get_auth_request con 0x7f3af40a39b0 auth_method 0 2024-09-16T03:56:21.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.058+0000 7f3af8bab640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:56:21.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.058+0000 7f3af8bab640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:56:21.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.058+0000 7f3af8bab640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:56:21.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.058+0000 7f3af93ac640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:56:21.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.058+0000 7f3af93ac640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:56:21.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.058+0000 7f3af93ac640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:56:21.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.058+0000 7f3af3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:56:21.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.058+0000 7f3af3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:56:21.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.058+0000 7f3af3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:56:21.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.058+0000 7f3af8bab640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:56:21.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.058+0000 7f3af8bab640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:56:21.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.058+0000 7f3af8bab640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:56:21.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.058+0000 7f3af93ac640 10 monclient(hunting): handle_auth_done global_id 854315 payload 824 2024-09-16T03:56:21.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.058+0000 7f3af93ac640 1 -- 172.21.15.42:0/2616561402 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3af40a39b0 msgr2=0x7f3af4130300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:56:21.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.058+0000 7f3af93ac640 1 --2- 172.21.15.42:0/2616561402 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3af40a39b0 0x7f3af4130300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:56:21.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.058+0000 7f3af93ac640 1 -- 172.21.15.42:0/2616561402 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3af40a3d90 msgr2=0x7f3af4130840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:56:21.077 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.058+0000 7f3af93ac640 1 --2- 172.21.15.42:0/2616561402 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3af40a3d90 0x7f3af4130840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:56:21.077 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.058+0000 7f3af93ac640 10 monclient: _finish_hunting 0 2024-09-16T03:56:21.077 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.058+0000 7f3af93ac640 1 monclient: found mon.noname-c 2024-09-16T03:56:21.077 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.058+0000 7f3af93ac640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:56:21.077 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.058+0000 7f3af93ac640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:56:21.077 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.058+0000 7f3af93ac640 1 -- 172.21.15.42:0/2616561402 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3af406f680 con 0x7f3af413c8c0 2024-09-16T03:56:21.077 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.058+0000 7f3af93ac640 1 --2- 172.21.15.42:0/2616561402 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3af413c8c0 0x7f3af4130d80 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f3aec00c780 tx=0x7f3aec00cc40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:56:21.077 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.058+0000 7f3af17fa640 10 client.?.objecter ms_handle_connect 0x7f3af413c8c0 2024-09-16T03:56:21.077 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.058+0000 7f3af17fa640 10 client.?.objecter resend_mon_ops 2024-09-16T03:56:21.077 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.058+0000 7f3af17fa640 1 -- 172.21.15.42:0/2616561402 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3aec018070 con 0x7f3af413c8c0 2024-09-16T03:56:21.077 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.058+0000 7f3af17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:56:21.077 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.058+0000 7f3af17fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:56:21.077 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.058+0000 7f3af17fa640 10 monclient: dump: 2024-09-16T03:56:21.077 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:56:21.078 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:56:21.078 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:21.078 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:21.078 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:56:21.078 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:56:21.078 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:56:21.078 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:56:21.078 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:56:21.078 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:56:21.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.058+0000 7f3af17fa640 10 monclient: _finish_auth 0 2024-09-16T03:56:21.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.058+0000 7f3af17fa640 10 monclient: _check_auth_tickets 2024-09-16T03:56:21.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.058+0000 7f3af17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:56:21.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.058+0000 7f3afae0f640 5 monclient: authenticate success, global_id 854315 2024-09-16T03:56:21.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.058+0000 7f3af17fa640 1 -- 172.21.15.42:0/2616561402 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3aec00a640 con 0x7f3af413c8c0 2024-09-16T03:56:21.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.058+0000 7f3afae0f640 10 monclient: _renew_subs 2024-09-16T03:56:21.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.058+0000 7f3afae0f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:56:21.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.058+0000 7f3afae0f640 1 -- 172.21.15.42:0/2616561402 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3af413c0f0 con 0x7f3af413c8c0 2024-09-16T03:56:21.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.058+0000 7f3af17fa640 10 monclient: handle_config config(0 keys) 2024-09-16T03:56:21.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.058+0000 7f3af17fa640 1 -- 172.21.15.42:0/2616561402 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3aec00aab0 con 0x7f3af413c8c0 2024-09-16T03:56:21.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.058+0000 7f3af17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:56:21.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.058+0000 7f3af17fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:56:21.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.058+0000 7f3af17fa640 10 monclient: dump: 2024-09-16T03:56:21.080 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:56:21.080 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:56:21.080 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:21.080 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:21.080 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:56:21.080 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:56:21.080 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:56:21.080 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:56:21.080 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:56:21.080 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:56:21.080 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.058+0000 7f3afae0f640 10 client.854315.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:56:21.080 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.058+0000 7f3afae0f640 10 monclient: _renew_subs 2024-09-16T03:56:21.080 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.058+0000 7f3afae0f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:56:21.081 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.058+0000 7f3afae0f640 1 -- 172.21.15.42:0/2616561402 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f3af409ce40 con 0x7f3af413c8c0 2024-09-16T03:56:21.081 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.058+0000 7f3af2ffd640 4 set_mon_vals no callback set 2024-09-16T03:56:21.081 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.058+0000 7f3afae0f640 1 librados: init done 2024-09-16T03:56:21.081 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.062+0000 7f3afae0f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:56:21.081 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.062+0000 7f3afae0f640 1 --2- 172.21.15.42:0/2616561402 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3af413cca0 0x7f3af413d080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:56:21.081 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.062+0000 7f3afae0f640 10 monclient(hunting): start opening mon connection 2024-09-16T03:56:21.081 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.062+0000 7f3afae0f640 1 -- 172.21.15.42:0/2616561402 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f3af409c930 con 0x7f3af413cca0 2024-09-16T03:56:21.081 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.062+0000 7f3af8bab640 1 --2- 172.21.15.42:0/2616561402 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3af413cca0 0x7f3af413d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:56:21.081 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.062+0000 7f3af8bab640 10 monclient: get_auth_request con 0x7f3af413cca0 auth_method 0 2024-09-16T03:56:21.081 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.062+0000 7f3af8bab640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:56:21.081 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.062+0000 7f3af8bab640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:56:21.081 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.062+0000 7f3af8bab640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:56:21.081 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.062+0000 7f3af17fa640 1 -- 172.21.15.42:0/2616561402 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f3aec01b070 con 0x7f3af413c8c0 2024-09-16T03:56:21.081 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.062+0000 7f3af8bab640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:56:21.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.062+0000 7f3af8bab640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:56:21.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.062+0000 7f3af8bab640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:56:21.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.062+0000 7f3af8bab640 10 monclient(hunting): handle_auth_done global_id 824730 payload 824 2024-09-16T03:56:21.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.062+0000 7f3af8bab640 1 -- 172.21.15.42:0/2616561402 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3af409d4f0 con 0x7f3af413cca0 2024-09-16T03:56:21.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.062+0000 7f3af17fa640 1 --2- 172.21.15.42:0/2616561402 >> v2:172.21.15.154:6812/254845342 conn(0x7f3ae0037840 0x7f3ae0039d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:56:21.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.062+0000 7f3af17fa640 1 -- 172.21.15.42:0/2616561402 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(442..442 src has 330..442) ==== 5741+0+0 (secure 0 0 0) 0x7f3aec048500 con 0x7f3af413c8c0 2024-09-16T03:56:21.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.062+0000 7f3af17fa640 10 client.854315.objecter ms_dispatch 0x7f3af4000ba0 osd_map(442..442 src has 330..442) 2024-09-16T03:56:21.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.062+0000 7f3af17fa640 3 client.854315.objecter handle_osd_map got epochs [442,442] > 0 2024-09-16T03:56:21.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.062+0000 7f3af17fa640 3 client.854315.objecter handle_osd_map decoding full epoch 442 2024-09-16T03:56:21.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.062+0000 7f3af17fa640 20 client.854315.objecter dump_active .. 0 homeless 2024-09-16T03:56:21.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.062+0000 7f3af8bab640 1 --2- 172.21.15.42:0/2616561402 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3af413cca0 0x7f3af413d080 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f3ae8000f70 tx=0x7f3ae800fc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:56:21.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.062+0000 7f3af17fa640 10 client.854315.objecter ms_handle_connect 0x7f3af413cca0 2024-09-16T03:56:21.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.062+0000 7f3af17fa640 10 client.854315.objecter resend_mon_ops 2024-09-16T03:56:21.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.062+0000 7f3af17fa640 1 -- 172.21.15.42:0/2616561402 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ae80050d0 con 0x7f3af413cca0 2024-09-16T03:56:21.083 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.062+0000 7f3af17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:56:21.083 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.062+0000 7f3af17fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:56:21.083 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.062+0000 7f3af3fff640 1 --2- 172.21.15.42:0/2616561402 >> v2:172.21.15.154:6812/254845342 conn(0x7f3ae0037840 0x7f3ae0039d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:56:21.083 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.062+0000 7f3af17fa640 10 monclient: dump: 2024-09-16T03:56:21.083 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:56:21.083 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:56:21.083 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:21.083 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:21.083 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:56:21.083 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:56:21.083 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:56:21.083 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:56:21.083 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:56:21.083 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:56:21.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.062+0000 7f3af3fff640 10 monclient: get_auth_request con 0x7f3ae0037840 auth_method 0 2024-09-16T03:56:21.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.062+0000 7f3af17fa640 1 -- 172.21.15.42:0/2616561402 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f3ae8018070 con 0x7f3af413cca0 2024-09-16T03:56:21.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.062+0000 7f3af17fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:56:21.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.062+0000 7f3af17fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:56:21.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.062+0000 7f3af17fa640 1 -- 172.21.15.42:0/2616561402 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3af413cca0 msgr2=0x7f3af413d080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:56:21.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.062+0000 7f3af17fa640 1 --2- 172.21.15.42:0/2616561402 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3af413cca0 0x7f3af413d080 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f3ae8000f70 tx=0x7f3ae800fc00 comp rx=0 tx=0).stop 2024-09-16T03:56:21.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.062+0000 7f3af17fa640 1 -- 172.21.15.42:0/2616561402 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3af413cca0 msgr2=0x7f3af413d080 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:56:21.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.062+0000 7f3af17fa640 1 --2- 172.21.15.42:0/2616561402 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3af413cca0 0x7f3af413d080 secure :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f3ae8000f70 tx=0x7f3ae800fc00 comp rx=0 tx=0).stop 2024-09-16T03:56:21.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.062+0000 7f3af3fff640 1 --2- 172.21.15.42:0/2616561402 >> v2:172.21.15.154:6812/254845342 conn(0x7f3ae0037840 0x7f3ae0039d00 secure :-1 s=READY pgs=3655 cs=0 l=1 rev1=1 crypto rx=0x7f3ae40ad000 tx=0x7f3ae4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:56:21.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.062+0000 7f3af17fa640 10 client.854315.objecter ms_handle_connect 0x7f3ae0037840 2024-09-16T03:56:21.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.066+0000 7f3afae0f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:56:21.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.066+0000 7f3afae0f640 1 --2- 172.21.15.42:0/2616561402 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3af413d5c0 0x7f3af4138f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:56:21.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.066+0000 7f3afae0f640 10 monclient(hunting): start opening mon connection 2024-09-16T03:56:21.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.066+0000 7f3afae0f640 1 -- 172.21.15.42:0/2616561402 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f3af40a5750 con 0x7f3af413d5c0 2024-09-16T03:56:21.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.066+0000 7f3af8bab640 1 --2- 172.21.15.42:0/2616561402 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3af413d5c0 0x7f3af4138f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:56:21.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.066+0000 7f3af8bab640 10 monclient: get_auth_request con 0x7f3af413d5c0 auth_method 0 2024-09-16T03:56:21.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.066+0000 7f3af8bab640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:56:21.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.066+0000 7f3af8bab640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:56:21.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.066+0000 7f3af8bab640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:56:21.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.066+0000 7f3af8bab640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:56:21.087 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.066+0000 7f3af8bab640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:56:21.087 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.066+0000 7f3af8bab640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:56:21.087 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.070+0000 7f3af8bab640 10 monclient(hunting): handle_auth_done global_id 824733 payload 824 2024-09-16T03:56:21.087 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.070+0000 7f3af8bab640 1 -- 172.21.15.42:0/2616561402 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f3af413b350 con 0x7f3af413d5c0 2024-09-16T03:56:21.087 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.070+0000 7f3af17fa640 10 client.854315.objecter ms_handle_connect 0x7f3af413d5c0 2024-09-16T03:56:21.087 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.070+0000 7f3af17fa640 10 client.854315.objecter resend_mon_ops 2024-09-16T03:56:21.087 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.070+0000 7f3af8bab640 1 --2- 172.21.15.42:0/2616561402 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3af413d5c0 0x7f3af4138f20 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f3ae8000f70 tx=0x7f3ae8001130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:56:21.087 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.070+0000 7f3af17fa640 1 -- 172.21.15.42:0/2616561402 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ae801ee20 con 0x7f3af413d5c0 2024-09-16T03:56:21.087 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.070+0000 7f3af17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:56:21.087 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.070+0000 7f3af17fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:56:21.087 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.070+0000 7f3af17fa640 10 monclient: dump: 2024-09-16T03:56:21.087 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:56:21.087 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:56:21.087 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:21.088 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:21.088 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:56:21.088 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:56:21.088 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:56:21.088 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:56:21.088 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:56:21.088 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:56:21.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.070+0000 7f3af17fa640 1 -- 172.21.15.42:0/2616561402 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4882 (secure 0 0 0) 0x7f3ae8007240 con 0x7f3af413d5c0 2024-09-16T03:56:21.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.070+0000 7f3af17fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:56:21.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.070+0000 7f3af17fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:56:21.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.070+0000 7f3af17fa640 1 -- 172.21.15.42:0/2616561402 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3af413d5c0 msgr2=0x7f3af4138f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:56:21.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.070+0000 7f3af17fa640 1 --2- 172.21.15.42:0/2616561402 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3af413d5c0 0x7f3af4138f20 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f3ae8000f70 tx=0x7f3ae8001130 comp rx=0 tx=0).stop 2024-09-16T03:56:21.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.070+0000 7f3af17fa640 1 -- 172.21.15.42:0/2616561402 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3af413d5c0 msgr2=0x7f3af4138f20 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:56:21.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.070+0000 7f3af17fa640 1 --2- 172.21.15.42:0/2616561402 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3af413d5c0 0x7f3af4138f20 secure :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f3ae8000f70 tx=0x7f3ae8001130 comp rx=0 tx=0).stop 2024-09-16T03:56:21.089 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:56:21.089 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:56:21.089 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:56:21.089 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "leader", 2024-09-16T03:56:21.089 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 128, 2024-09-16T03:56:21.089 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 270925, 2024-09-16T03:56:21.089 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:56:21.089 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:56:21.090 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T03:56:21.090 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T03:56:21.090 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:56:21.090 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 4, 2024-09-16T03:56:21.090 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:56:21.090 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:56:21.090 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:56:21.090 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:56:21.090 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:56:21.090 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:56:21.090 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:56:21.090 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:56:21.091 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:56:21.091 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:56:21.091 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:56:21.091 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:56:21.091 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:56:21.091 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:56:21.091 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:56:21.091 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:56:21.091 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:56:21.091 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:56:21.091 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:56:21.091 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:56:21.091 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:56:21.091 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:56:21.092 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:56:21.092 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:56:21.092 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:56:21.092 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:56:21.092 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:56:21.092 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:56:21.092 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:56:21.092 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:56:21.092 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:56:21.092 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:56:21.092 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:56:21.092 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:56:21.093 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:56:21.093 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:56:21.093 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:56:21.093 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:56:21.093 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:56:21.093 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:56:21.093 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:56:21.093 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:56:21.093 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:56:21.093 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:56:21.093 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:56:21.093 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:56:21.093 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:56:21.094 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:56:21.094 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:56:21.094 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:56:21.094 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:56:21.094 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:56:21.094 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:56:21.094 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:56:21.094 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:56:21.094 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:56:21.094 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:56:21.094 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:56:21.094 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:56:21.094 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:56:21.094 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:56:21.095 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:56:21.095 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:56:21.095 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:56:21.095 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:56:21.095 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:56:21.095 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:56:21.095 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:56:21.095 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:56:21.095 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:56:21.095 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:56:21.095 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:56:21.095 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:56:21.095 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:56:21.095 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:56:21.096 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:56:21.096 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:56:21.096 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:56:21.096 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:56:21.096 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:56:21.096 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:56:21.096 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:56:21.096 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:56:21.096 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:56:21.096 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:56:21.096 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:56:21.097 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:56:21.097 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:56:21.097 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:56:21.097 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:56:21.097 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:56:21.097 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:56:21.097 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:56:21.097 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:56:21.097 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:56:21.097 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:56:21.097 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:56:21.097 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:56:21.097 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:56:21.097 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:56:21.098 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:56:21.098 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:56:21.098 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:56:21.098 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:56:21.098 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:56:21.098 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:56:21.098 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:56:21.098 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:56:21.098 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:56:21.098 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:56:21.098 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:56:21.098 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:56:21.098 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:56:21.098 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:56:21.099 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:56:21.099 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:56:21.099 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:56:21.099 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:56:21.099 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:56:21.099 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:56:21.099 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:56:21.099 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:56:21.099 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:56:21.099 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:56:21.099 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:56:21.099 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:56:21.099 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:56:21.099 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:56:21.099 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:56:21.100 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:56:21.100 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:56:21.100 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:56:21.100 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:56:21.100 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:56:21.100 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:56:21.100 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:56:21.100 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:56:21.100 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:56:21.100 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:56:21.100 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:56:21.100 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:56:21.100 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T03:56:21.100 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:56:21.101 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:56:21.101 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:56:21.101 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 6 2024-09-16T03:56:21.101 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:56:21.101 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:56:21.101 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:56:21.101 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:56:21.101 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:56:21.101 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:56:21.101 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 7 2024-09-16T03:56:21.101 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:56:21.101 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:56:21.101 INFO:teuthology.orchestra.run.smithi042.stdout: "mgr": [ 2024-09-16T03:56:21.101 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:56:21.102 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:56:21.102 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:56:21.102 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:56:21.102 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:56:21.102 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:56:21.102 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:56:21.102 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:56:21.102 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:56:21.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.070+0000 7f3afae0f640 10 librados: watch_flush enter 2024-09-16T03:56:21.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.070+0000 7f3afae0f640 10 librados: watch_flush exit 2024-09-16T03:56:21.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.070+0000 7f3afae0f640 20 client.854315.objecter shutdown clearing up homeless session... 2024-09-16T03:56:21.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.070+0000 7f3afae0f640 10 client.854315.objecter successfully canceled tick 2024-09-16T03:56:21.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.070+0000 7f3afae0f640 1 -- 172.21.15.42:0/2616561402 >> v2:172.21.15.154:6812/254845342 conn(0x7f3ae0037840 msgr2=0x7f3ae0039d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:56:21.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.070+0000 7f3afae0f640 1 --2- 172.21.15.42:0/2616561402 >> v2:172.21.15.154:6812/254845342 conn(0x7f3ae0037840 0x7f3ae0039d00 secure :-1 s=READY pgs=3655 cs=0 l=1 rev1=1 crypto rx=0x7f3ae40ad000 tx=0x7f3ae4008040 comp rx=0 tx=0).stop 2024-09-16T03:56:21.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.070+0000 7f3afae0f640 10 monclient: shutdown 2024-09-16T03:56:21.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.070+0000 7f3afae0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:56:21.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.070+0000 7f3afae0f640 1 -- 172.21.15.42:0/2616561402 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3af413c8c0 msgr2=0x7f3af4130d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:56:21.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.070+0000 7f3af93ac640 1 -- 172.21.15.42:0/2616561402 reap_dead start 2024-09-16T03:56:21.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.070+0000 7f3afae0f640 1 --2- 172.21.15.42:0/2616561402 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3af413c8c0 0x7f3af4130d80 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f3aec00c780 tx=0x7f3aec00cc40 comp rx=0 tx=0).stop 2024-09-16T03:56:21.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.070+0000 7f3afae0f640 1 -- 172.21.15.42:0/2616561402 shutdown_connections 2024-09-16T03:56:21.105 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.070+0000 7f3afae0f640 1 --2- 172.21.15.42:0/2616561402 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3af413c8c0 0x7f3af4130d80 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:56:21.105 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.070+0000 7f3afae0f640 1 -- 172.21.15.42:0/2616561402 >> 172.21.15.42:0/2616561402 conn(0x7f3af40a9fb0 msgr2=0x7f3af40a87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:56:21.105 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.070+0000 7f3afae0f640 1 -- 172.21.15.42:0/2616561402 shutdown_connections 2024-09-16T03:56:21.105 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.070+0000 7f3afae0f640 1 -- 172.21.15.42:0/2616561402 wait complete. 2024-09-16T03:56:21.105 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.070+0000 7f3afae0f640 1 librados: shutdown 2024-09-16T03:56:21.105 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:56:21.105 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T03:56:21.105 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T03:56:21.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.222+0000 7f603ff9a640 10 monclient: get_monmap_and_config 2024-09-16T03:56:21.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.222+0000 7f603ff9a640 10 monclient: build_initial_monmap 2024-09-16T03:56:21.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.222+0000 7f603ff9a640 10 monclient: monmap: 2024-09-16T03:56:21.226 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:56:21.226 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:56:21.226 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:56:21.224938+0000 2024-09-16T03:56:21.227 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:56:21.224938+0000 2024-09-16T03:56:21.227 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:56:21.227 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:56:21.227 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:56:21.227 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:56:21.227 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:56:21.227 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:56:21.227 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:56:21.227 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:56:21.230 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.222+0000 7f603ff9a640 1 Processor -- start 2024-09-16T03:56:21.230 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.222+0000 7f603ff9a640 1 -- start start 2024-09-16T03:56:21.230 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.222+0000 7f603ff9a640 10 monclient: init 2024-09-16T03:56:21.230 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.222+0000 7f603ff9a640 10 monclient: _reopen_session rank -1 2024-09-16T03:56:21.230 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.222+0000 7f603ff9a640 10 monclient: _add_conns ranks=[2,1,4,0,3] 2024-09-16T03:56:21.230 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.222+0000 7f603ff9a640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f60380a1330 0x7f60380a1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:56:21.230 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.222+0000 7f603ff9a640 10 monclient(hunting): picked mon.noname-b con 0x7f60380a1330 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:56:21.230 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.222+0000 7f603ff9a640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f603806dac0 addr v1:172.21.15.154:6789/0 2024-09-16T03:56:21.230 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.222+0000 7f603ff9a640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f60380a1c50 0x7f6038135810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:56:21.230 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.222+0000 7f603ff9a640 10 monclient(hunting): picked mon.noname-c con 0x7f60380a1c50 addr v2:172.21.15.154:3300/0 2024-09-16T03:56:21.230 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.222+0000 7f603ff9a640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f603809c090 con 0x7f603806dac0 2024-09-16T03:56:21.230 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.222+0000 7f603ff9a640 10 monclient(hunting): start opening mon connection 2024-09-16T03:56:21.230 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.222+0000 7f603ff9a640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f603809c890 con 0x7f60380a1c50 2024-09-16T03:56:21.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.222+0000 7f603ff9a640 10 monclient(hunting): start opening mon connection 2024-09-16T03:56:21.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.222+0000 7f603ff9a640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f603809f610 con 0x7f60380a1330 2024-09-16T03:56:21.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.222+0000 7f603ff9a640 10 monclient(hunting): _renew_subs 2024-09-16T03:56:21.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.222+0000 7f603ff9a640 10 monclient(hunting): authenticate will time out at 1828.194356s 2024-09-16T03:56:21.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.222+0000 7f603dd36640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f60380a1330 0x7f60380a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:56:21.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.222+0000 7f603dd36640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f60380a1330 0x7f60380a1710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:40852/0 (socket says 172.21.15.42:40852) 2024-09-16T03:56:21.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.222+0000 7f603dd36640 1 -- 172.21.15.42:0/1133003507 learned_addr learned my addr 172.21.15.42:0/1133003507 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:56:21.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.222+0000 7f603dd36640 10 monclient(hunting): get_auth_request con 0x7f60380a1330 auth_method 0 2024-09-16T03:56:21.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.222+0000 7f603dd36640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:56:21.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.222+0000 7f603dd36640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:56:21.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.222+0000 7f603dd36640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:56:21.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.222+0000 7f603dd36640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:56:21.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.222+0000 7f603dd36640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:56:21.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.222+0000 7f603dd36640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:56:21.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.222+0000 7f603e537640 1 --2- 172.21.15.42:0/1133003507 >> v2:172.21.15.154:3300/0 conn(0x7f60380a1c50 0x7f6038135810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:56:21.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.222+0000 7f603e537640 10 monclient(hunting): get_auth_request con 0x7f60380a1c50 auth_method 0 2024-09-16T03:56:21.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.222+0000 7f603e537640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:56:21.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.222+0000 7f603e537640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:56:21.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.222+0000 7f603e537640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:56:21.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.222+0000 7f603dd36640 10 monclient(hunting): handle_auth_done global_id 854318 payload 274 2024-09-16T03:56:21.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.222+0000 7f603dd36640 1 -- 172.21.15.42:0/1133003507 >> v2:172.21.15.154:3300/0 conn(0x7f60380a1c50 msgr2=0x7f6038135810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:56:21.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.222+0000 7f603dd36640 1 --2- 172.21.15.42:0/1133003507 >> v2:172.21.15.154:3300/0 conn(0x7f60380a1c50 0x7f6038135810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:56:21.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.222+0000 7f603dd36640 1 -- 172.21.15.42:0/1133003507 >> v1:172.21.15.154:6789/0 conn(0x7f603806dac0 legacy=0x7f6038135100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:56:21.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.222+0000 7f603dd36640 10 monclient: _finish_hunting 0 2024-09-16T03:56:21.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.222+0000 7f603dd36640 1 monclient: found mon.noname-b 2024-09-16T03:56:21.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.222+0000 7f603dd36640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:56:21.233 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.222+0000 7f603dd36640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T03:56:21.233 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.222+0000 7f603dd36640 1 -- 172.21.15.42:0/1133003507 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f603809f8f0 con 0x7f60380a1330 2024-09-16T03:56:21.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.226+0000 7f603dd36640 1 --2- 172.21.15.42:0/1133003507 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f60380a1330 0x7f60380a1710 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f6034009870 tx=0x7f603401c440 comp rx=0 tx=0).ready entity=mon.2 client_cookie=edd4b3181724d307 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:56:21.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.226+0000 7f603cd34640 1 -- 172.21.15.42:0/1133003507 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60340ab620 con 0x7f60380a1330 2024-09-16T03:56:21.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.226+0000 7f603cd34640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:56:21.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.226+0000 7f603cd34640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:56:21.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.226+0000 7f603cd34640 10 monclient: dump: 2024-09-16T03:56:21.239 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:56:21.239 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:56:21.239 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:21.240 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:21.240 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:56:21.240 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:56:21.240 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:56:21.240 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:56:21.240 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:56:21.240 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:56:21.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.226+0000 7f603cd34640 10 monclient: _finish_auth 0 2024-09-16T03:56:21.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.226+0000 7f603cd34640 10 monclient: _check_auth_tickets 2024-09-16T03:56:21.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.226+0000 7f603cd34640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:56:21.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.226+0000 7f603cd34640 1 -- 172.21.15.42:0/1133003507 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f60340b0070 con 0x7f60380a1330 2024-09-16T03:56:21.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.226+0000 7f603ff9a640 5 monclient: authenticate success, global_id 854318 2024-09-16T03:56:21.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.226+0000 7f603ff9a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:56:21.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.226+0000 7f603cd34640 10 monclient: handle_config config(0 keys) 2024-09-16T03:56:21.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.226+0000 7f603cd34640 1 -- 172.21.15.42:0/1133003507 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60340ab9e0 con 0x7f60380a1330 2024-09-16T03:56:21.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.226+0000 7f603ff9a640 10 monclient: get_monmap_and_config success 2024-09-16T03:56:21.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.226+0000 7f603ff9a640 4 set_mon_vals no callback set 2024-09-16T03:56:21.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.226+0000 7f603cd34640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:56:21.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.226+0000 7f603cd34640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:56:21.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.226+0000 7f603cd34640 10 monclient: dump: 2024-09-16T03:56:21.241 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:56:21.241 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:56:21.241 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:21.241 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:21.241 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:56:21.241 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:56:21.241 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:56:21.241 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:56:21.242 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:56:21.242 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:56:21.242 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.226+0000 7f603ff9a640 10 monclient: shutdown 2024-09-16T03:56:21.242 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.226+0000 7f603ff9a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:56:21.242 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.226+0000 7f603ff9a640 1 -- 172.21.15.42:0/1133003507 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f60380a1330 msgr2=0x7f60380a1710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:56:21.242 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.226+0000 7f603ff9a640 1 --2- 172.21.15.42:0/1133003507 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f60380a1330 0x7f60380a1710 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f6034009870 tx=0x7f603401c440 comp rx=0 tx=0).stop 2024-09-16T03:56:21.242 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.226+0000 7f603ff9a640 1 -- 172.21.15.42:0/1133003507 shutdown_connections 2024-09-16T03:56:21.242 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.226+0000 7f603ff9a640 1 --2- 172.21.15.42:0/1133003507 >> v2:172.21.15.154:3300/0 conn(0x7f60380a1c50 0x7f6038135810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:56:21.242 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.226+0000 7f603ff9a640 1 --2- 172.21.15.42:0/1133003507 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f60380a1330 0x7f60380a1710 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:56:21.242 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.226+0000 7f603ff9a640 1 -- 172.21.15.42:0/1133003507 >> 172.21.15.42:0/1133003507 conn(0x7f60380a2420 msgr2=0x7f60380a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:56:21.242 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.226+0000 7f603ff9a640 1 -- 172.21.15.42:0/1133003507 shutdown_connections 2024-09-16T03:56:21.242 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.226+0000 7f603ff9a640 1 -- 172.21.15.42:0/1133003507 wait complete. 2024-09-16T03:56:21.242 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.226+0000 7f603ff9a640 10 monclient: build_initial_monmap 2024-09-16T03:56:21.242 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.226+0000 7f603ff9a640 10 monclient: monmap: 2024-09-16T03:56:21.243 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:56:21.243 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:56:21.243 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:56:21.243 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:56:21.243 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:56:21.243 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:56:21.243 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:56:21.243 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:56:21.243 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:56:21.243 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:56:21.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.226+0000 7f603ff9a640 1 librados: starting msgr at 2024-09-16T03:56:21.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.226+0000 7f603ff9a640 1 librados: starting objecter 2024-09-16T03:56:21.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.226+0000 7f603ff9a640 1 Processor -- start 2024-09-16T03:56:21.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.226+0000 7f603ff9a640 1 -- start start 2024-09-16T03:56:21.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.226+0000 7f603ff9a640 1 librados: setting wanted keys 2024-09-16T03:56:21.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.226+0000 7f603ff9a640 1 librados: calling monclient init 2024-09-16T03:56:21.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.226+0000 7f603ff9a640 10 monclient: init 2024-09-16T03:56:21.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.226+0000 7f603ff9a640 10 monclient: _reopen_session rank -1 2024-09-16T03:56:21.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.226+0000 7f603ff9a640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T03:56:21.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.226+0000 7f603ff9a640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f60380a1330 0x7f6038134bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:56:21.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.226+0000 7f603ff9a640 10 monclient(hunting): picked mon.noname-b con 0x7f60380a1330 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:56:21.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.226+0000 7f603ff9a640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f60380a1c50 0x7f60381350f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:56:21.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.226+0000 7f603ff9a640 10 monclient(hunting): picked mon.noname-c con 0x7f60380a1c50 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:56:21.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.226+0000 7f603ff9a640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f603811e420 0x7f603811bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:56:21.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.226+0000 7f603ff9a640 10 monclient(hunting): picked mon.noname-a con 0x7f603811e420 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:56:21.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.226+0000 7f603ff9a640 10 monclient(hunting): start opening mon connection 2024-09-16T03:56:21.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.226+0000 7f603ff9a640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f603809ef30 con 0x7f603811e420 2024-09-16T03:56:21.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.226+0000 7f603ff9a640 10 monclient(hunting): start opening mon connection 2024-09-16T03:56:21.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.226+0000 7f603ff9a640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f603809edb0 con 0x7f60380a1330 2024-09-16T03:56:21.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.226+0000 7f603ff9a640 10 monclient(hunting): start opening mon connection 2024-09-16T03:56:21.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.226+0000 7f603ff9a640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f6038135d50 con 0x7f60380a1c50 2024-09-16T03:56:21.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.226+0000 7f603ff9a640 10 monclient(hunting): _renew_subs 2024-09-16T03:56:21.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.226+0000 7f603ff9a640 10 monclient(hunting): authenticate will time out at 1828.199975s 2024-09-16T03:56:21.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.230+0000 7f603e537640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f603811e420 0x7f603811bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:56:21.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.230+0000 7f603e537640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f603811e420 0x7f603811bf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:37108/0 (socket says 172.21.15.42:37108) 2024-09-16T03:56:21.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.230+0000 7f603e537640 1 -- 172.21.15.42:0/3052015911 learned_addr learned my addr 172.21.15.42:0/3052015911 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:56:21.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.230+0000 7f603e537640 10 monclient(hunting): get_auth_request con 0x7f603811e420 auth_method 0 2024-09-16T03:56:21.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.230+0000 7f603d535640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f60380a1c50 0x7f60381350f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:56:21.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.230+0000 7f603e537640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:56:21.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.230+0000 7f603e537640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:56:21.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.230+0000 7f603e537640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:56:21.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.230+0000 7f603d535640 10 monclient(hunting): get_auth_request con 0x7f60380a1c50 auth_method 0 2024-09-16T03:56:21.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.230+0000 7f603d535640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:56:21.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.230+0000 7f603d535640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:56:21.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.230+0000 7f603d535640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:56:21.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.230+0000 7f603dd36640 1 --2- 172.21.15.42:0/3052015911 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f60380a1330 0x7f6038134bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:56:21.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.230+0000 7f603dd36640 10 monclient(hunting): get_auth_request con 0x7f60380a1330 auth_method 0 2024-09-16T03:56:21.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.230+0000 7f603dd36640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:56:21.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.230+0000 7f603dd36640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:56:21.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.230+0000 7f603dd36640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:56:21.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.230+0000 7f603e537640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:56:21.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.230+0000 7f603e537640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:56:21.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.230+0000 7f603e537640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:56:21.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.230+0000 7f603d535640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:56:21.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.230+0000 7f603d535640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:56:21.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.230+0000 7f603d535640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:56:21.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.230+0000 7f603dd36640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:56:21.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.230+0000 7f603dd36640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:56:21.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.230+0000 7f603dd36640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:56:21.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.230+0000 7f603e537640 10 monclient(hunting): handle_auth_done global_id 824736 payload 824 2024-09-16T03:56:21.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.230+0000 7f603e537640 1 -- 172.21.15.42:0/3052015911 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f60380a1c50 msgr2=0x7f60381350f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:56:21.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.230+0000 7f603e537640 1 --2- 172.21.15.42:0/3052015911 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f60380a1c50 0x7f60381350f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:56:21.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.230+0000 7f603e537640 1 -- 172.21.15.42:0/3052015911 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f60380a1330 msgr2=0x7f6038134bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:56:21.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.230+0000 7f603e537640 1 --2- 172.21.15.42:0/3052015911 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f60380a1330 0x7f6038134bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:56:21.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.230+0000 7f603e537640 10 monclient: _finish_hunting 0 2024-09-16T03:56:21.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.230+0000 7f603e537640 1 monclient: found mon.noname-a 2024-09-16T03:56:21.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.230+0000 7f603e537640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:56:21.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.230+0000 7f603e537640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:56:21.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.230+0000 7f603e537640 1 -- 172.21.15.42:0/3052015911 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f603811f6b0 con 0x7f603811e420 2024-09-16T03:56:21.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.230+0000 7f603d535640 1 --2- 172.21.15.42:0/3052015911 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f60380a1c50 0x7f60381350f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:56:21.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.230+0000 7f603e537640 1 --2- 172.21.15.42:0/3052015911 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f603811e420 0x7f603811bf70 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f6030007e40 tx=0x7f603000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:56:21.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.230+0000 7f602e7fc640 10 client.?.objecter ms_handle_connect 0x7f603811e420 2024-09-16T03:56:21.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.230+0000 7f602e7fc640 10 client.?.objecter resend_mon_ops 2024-09-16T03:56:21.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.230+0000 7f602e7fc640 1 -- 172.21.15.42:0/3052015911 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6030014620 con 0x7f603811e420 2024-09-16T03:56:21.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.230+0000 7f602e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:56:21.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.230+0000 7f602e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:56:21.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.230+0000 7f602e7fc640 10 monclient: dump: 2024-09-16T03:56:21.249 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:56:21.249 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:56:21.249 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:21.249 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:21.249 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:56:21.249 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:56:21.249 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:56:21.249 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:56:21.249 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:56:21.249 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:56:21.249 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.230+0000 7f602e7fc640 10 monclient: _finish_auth 0 2024-09-16T03:56:21.249 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.230+0000 7f602e7fc640 10 monclient: _check_auth_tickets 2024-09-16T03:56:21.249 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.230+0000 7f602e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:56:21.249 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.230+0000 7f602e7fc640 1 -- 172.21.15.42:0/3052015911 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6030019070 con 0x7f603811e420 2024-09-16T03:56:21.250 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.230+0000 7f602e7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:56:21.250 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.230+0000 7f602e7fc640 1 -- 172.21.15.42:0/3052015911 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6030014a20 con 0x7f603811e420 2024-09-16T03:56:21.250 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.230+0000 7f602e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:56:21.250 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.230+0000 7f602e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:56:21.250 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.230+0000 7f602e7fc640 10 monclient: dump: 2024-09-16T03:56:21.250 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:56:21.250 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:56:21.250 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:21.250 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:21.250 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:56:21.250 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:56:21.250 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:56:21.250 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:56:21.250 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:56:21.250 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:56:21.251 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.230+0000 7f603ff9a640 5 monclient: authenticate success, global_id 824736 2024-09-16T03:56:21.251 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.230+0000 7f602ffff640 4 set_mon_vals no callback set 2024-09-16T03:56:21.251 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.230+0000 7f603ff9a640 10 monclient: _renew_subs 2024-09-16T03:56:21.251 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.230+0000 7f603ff9a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:56:21.251 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.230+0000 7f603ff9a640 1 -- 172.21.15.42:0/3052015911 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f603811df80 con 0x7f603811e420 2024-09-16T03:56:21.251 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.230+0000 7f603ff9a640 10 client.824736.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:56:21.251 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.230+0000 7f603ff9a640 10 monclient: _renew_subs 2024-09-16T03:56:21.251 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.230+0000 7f603ff9a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:56:21.251 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.230+0000 7f603ff9a640 1 -- 172.21.15.42:0/3052015911 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f603811f3d0 con 0x7f603811e420 2024-09-16T03:56:21.251 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.230+0000 7f603ff9a640 1 librados: init done 2024-09-16T03:56:21.251 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.230+0000 7f603ff9a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:56:21.251 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.230+0000 7f603ff9a640 1 --2- 172.21.15.42:0/3052015911 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f603806dac0 0x7f603811c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:56:21.251 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.230+0000 7f603ff9a640 10 monclient(hunting): start opening mon connection 2024-09-16T03:56:21.251 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.230+0000 7f603ff9a640 1 -- 172.21.15.42:0/3052015911 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f60380713e0 con 0x7f603806dac0 2024-09-16T03:56:21.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.230+0000 7f603dd36640 1 --2- 172.21.15.42:0/3052015911 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f603806dac0 0x7f603811c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:56:21.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.230+0000 7f603dd36640 10 monclient: get_auth_request con 0x7f603806dac0 auth_method 0 2024-09-16T03:56:21.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.230+0000 7f603dd36640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:56:21.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.230+0000 7f603dd36640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:56:21.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.230+0000 7f603dd36640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:56:21.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.230+0000 7f602e7fc640 1 -- 172.21.15.42:0/3052015911 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f6030014bc0 con 0x7f603811e420 2024-09-16T03:56:21.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.230+0000 7f603dd36640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:56:21.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.230+0000 7f603dd36640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:56:21.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.230+0000 7f603dd36640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:56:21.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.230+0000 7f603dd36640 10 monclient(hunting): handle_auth_done global_id 854324 payload 824 2024-09-16T03:56:21.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.230+0000 7f602e7fc640 1 --2- 172.21.15.42:0/3052015911 >> v2:172.21.15.154:6812/254845342 conn(0x7f601c037810 0x7f601c039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:56:21.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.230+0000 7f603dd36640 1 -- 172.21.15.42:0/3052015911 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f60380a1710 con 0x7f603806dac0 2024-09-16T03:56:21.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.230+0000 7f602e7fc640 1 -- 172.21.15.42:0/3052015911 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(442..442 src has 330..442) ==== 5741+0+0 (secure 0 0 0) 0x7f60300497b0 con 0x7f603811e420 2024-09-16T03:56:21.253 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.230+0000 7f602e7fc640 10 client.824736.objecter ms_dispatch 0x7f6038000ba0 osd_map(442..442 src has 330..442) 2024-09-16T03:56:21.253 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.230+0000 7f602e7fc640 3 client.824736.objecter handle_osd_map got epochs [442,442] > 0 2024-09-16T03:56:21.253 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.230+0000 7f602e7fc640 3 client.824736.objecter handle_osd_map decoding full epoch 442 2024-09-16T03:56:21.253 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.230+0000 7f602e7fc640 20 client.824736.objecter dump_active .. 0 homeless 2024-09-16T03:56:21.253 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.230+0000 7f603dd36640 1 --2- 172.21.15.42:0/3052015911 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f603806dac0 0x7f603811c4b0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f603806c630 tx=0x7f60340191e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:56:21.253 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.230+0000 7f602e7fc640 10 client.824736.objecter ms_handle_connect 0x7f603806dac0 2024-09-16T03:56:21.253 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.230+0000 7f602e7fc640 10 client.824736.objecter resend_mon_ops 2024-09-16T03:56:21.253 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.230+0000 7f603d535640 1 --2- 172.21.15.42:0/3052015911 >> v2:172.21.15.154:6812/254845342 conn(0x7f601c037810 0x7f601c039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:56:21.253 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.230+0000 7f602e7fc640 1 -- 172.21.15.42:0/3052015911 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60340abc00 con 0x7f603806dac0 2024-09-16T03:56:21.253 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.230+0000 7f602e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:56:21.253 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.230+0000 7f602e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:56:21.253 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.230+0000 7f602e7fc640 10 monclient: dump: 2024-09-16T03:56:21.253 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:56:21.254 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:56:21.254 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:21.254 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:21.254 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:56:21.254 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:56:21.254 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:56:21.254 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:56:21.254 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:56:21.254 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:56:21.254 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.230+0000 7f603d535640 10 monclient: get_auth_request con 0x7f601c037810 auth_method 0 2024-09-16T03:56:21.254 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.230+0000 7f602e7fc640 1 -- 172.21.15.42:0/3052015911 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f60340b0040 con 0x7f603806dac0 2024-09-16T03:56:21.254 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.230+0000 7f602e7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:56:21.254 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.230+0000 7f602e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:56:21.254 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.230+0000 7f602e7fc640 1 -- 172.21.15.42:0/3052015911 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f603806dac0 msgr2=0x7f603811c4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:56:21.255 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.230+0000 7f602e7fc640 1 --2- 172.21.15.42:0/3052015911 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f603806dac0 0x7f603811c4b0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f603806c630 tx=0x7f60340191e0 comp rx=0 tx=0).stop 2024-09-16T03:56:21.255 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.230+0000 7f602e7fc640 1 -- 172.21.15.42:0/3052015911 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f603806dac0 msgr2=0x7f603811c4b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:56:21.255 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.230+0000 7f602e7fc640 1 --2- 172.21.15.42:0/3052015911 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f603806dac0 0x7f603811c4b0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:56:21.255 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.230+0000 7f603d535640 1 --2- 172.21.15.42:0/3052015911 >> v2:172.21.15.154:6812/254845342 conn(0x7f601c037810 0x7f601c039cd0 secure :-1 s=READY pgs=3656 cs=0 l=1 rev1=1 crypto rx=0x7f6038070890 tx=0x7f6028007e40 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:56:21.255 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.230+0000 7f602e7fc640 10 client.824736.objecter ms_handle_connect 0x7f601c037810 2024-09-16T03:56:21.257 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.238+0000 7f603ff9a640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:56:21.257 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.238+0000 7f603ff9a640 1 --2- 172.21.15.42:0/3052015911 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f603811c9f0 0x7f603811cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:56:21.257 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.238+0000 7f603ff9a640 10 monclient(hunting): start opening mon connection 2024-09-16T03:56:21.257 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.238+0000 7f603ff9a640 1 -- 172.21.15.42:0/3052015911 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f603811d310 con 0x7f603811c9f0 2024-09-16T03:56:21.257 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.238+0000 7f603dd36640 1 --2- 172.21.15.42:0/3052015911 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f603811c9f0 0x7f603811cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:56:21.257 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.238+0000 7f603dd36640 10 monclient: get_auth_request con 0x7f603811c9f0 auth_method 0 2024-09-16T03:56:21.257 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.238+0000 7f603dd36640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:56:21.257 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.238+0000 7f603dd36640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:56:21.257 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.238+0000 7f603dd36640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:56:21.257 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.238+0000 7f603dd36640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:56:21.257 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.238+0000 7f603dd36640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:56:21.257 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.238+0000 7f603dd36640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:56:21.258 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.238+0000 7f603dd36640 10 monclient(hunting): handle_auth_done global_id 854327 payload 824 2024-09-16T03:56:21.258 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.238+0000 7f603dd36640 1 -- 172.21.15.42:0/3052015911 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f6038069040 con 0x7f603811c9f0 2024-09-16T03:56:21.258 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.238+0000 7f603dd36640 1 --2- 172.21.15.42:0/3052015911 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f603811c9f0 0x7f603811cdd0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f603401aaf0 tx=0x7f6034009240 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:56:21.258 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.238+0000 7f602e7fc640 10 client.824736.objecter ms_handle_connect 0x7f603811c9f0 2024-09-16T03:56:21.258 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.238+0000 7f602e7fc640 10 client.824736.objecter resend_mon_ops 2024-09-16T03:56:21.258 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.238+0000 7f602e7fc640 1 -- 172.21.15.42:0/3052015911 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6034003d60 con 0x7f603811c9f0 2024-09-16T03:56:21.258 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.238+0000 7f602e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:56:21.258 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.238+0000 7f602e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:56:21.258 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.238+0000 7f602e7fc640 10 monclient: dump: 2024-09-16T03:56:21.258 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:56:21.258 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:56:21.258 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:21.258 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:21.259 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:56:21.259 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:56:21.259 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:56:21.259 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:56:21.259 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:56:21.259 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:56:21.259 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.238+0000 7f602e7fc640 1 -- 172.21.15.42:0/3052015911 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4546 (secure 0 0 0) 0x7f60340abe50 con 0x7f603811c9f0 2024-09-16T03:56:21.259 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.238+0000 7f602e7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:56:21.259 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.238+0000 7f602e7fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:56:21.259 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.238+0000 7f602e7fc640 1 -- 172.21.15.42:0/3052015911 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f603811c9f0 msgr2=0x7f603811cdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:56:21.259 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.238+0000 7f602e7fc640 1 --2- 172.21.15.42:0/3052015911 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f603811c9f0 0x7f603811cdd0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f603401aaf0 tx=0x7f6034009240 comp rx=0 tx=0).stop 2024-09-16T03:56:21.259 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.238+0000 7f602e7fc640 1 -- 172.21.15.42:0/3052015911 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f603811c9f0 msgr2=0x7f603811cdd0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:56:21.259 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.238+0000 7f602e7fc640 1 --2- 172.21.15.42:0/3052015911 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f603811c9f0 0x7f603811cdd0 secure :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f603401aaf0 tx=0x7f6034009240 comp rx=0 tx=0).stop 2024-09-16T03:56:21.260 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:56:21.260 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:56:21.260 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:56:21.260 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T03:56:21.260 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 128, 2024-09-16T03:56:21.260 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 114674, 2024-09-16T03:56:21.260 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:56:21.260 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:56:21.260 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T03:56:21.260 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T03:56:21.260 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:56:21.261 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 4, 2024-09-16T03:56:21.261 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:56:21.261 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:56:21.261 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:56:21.261 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:56:21.261 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:56:21.261 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:56:21.261 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:56:21.261 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:56:21.261 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:56:21.261 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:56:21.261 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:56:21.261 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:56:21.261 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:56:21.262 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:56:21.262 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:56:21.262 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:56:21.262 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:56:21.262 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:56:21.262 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:56:21.262 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:56:21.262 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:56:21.262 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:56:21.262 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:56:21.262 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:56:21.262 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:56:21.262 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:56:21.262 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:56:21.263 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:56:21.263 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:56:21.263 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:56:21.263 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:56:21.263 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:56:21.263 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:56:21.263 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:56:21.263 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:56:21.263 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:56:21.263 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:56:21.263 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:56:21.263 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:56:21.263 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:56:21.263 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:56:21.264 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:56:21.264 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:56:21.264 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:56:21.264 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:56:21.264 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:56:21.264 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:56:21.264 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:56:21.264 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:56:21.264 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:56:21.264 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:56:21.264 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:56:21.264 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:56:21.264 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:56:21.264 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:56:21.264 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:56:21.265 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:56:21.265 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:56:21.265 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:56:21.265 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:56:21.265 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:56:21.265 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:56:21.265 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:56:21.265 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:56:21.265 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:56:21.265 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:56:21.265 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:56:21.265 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:56:21.265 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:56:21.265 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:56:21.266 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:56:21.266 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:56:21.266 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:56:21.266 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:56:21.266 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:56:21.266 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:56:21.266 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:56:21.266 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:56:21.266 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:56:21.266 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:56:21.266 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:56:21.266 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:56:21.266 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:56:21.266 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:56:21.267 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:56:21.267 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:56:21.267 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:56:21.267 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:56:21.267 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:56:21.267 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:56:21.267 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:56:21.267 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:56:21.267 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:56:21.267 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:56:21.267 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:56:21.267 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:56:21.267 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:56:21.267 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:56:21.268 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:56:21.268 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:56:21.268 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:56:21.268 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:56:21.268 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:56:21.268 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:56:21.268 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:56:21.268 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:56:21.268 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:56:21.268 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:56:21.268 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:56:21.268 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:56:21.268 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:56:21.268 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:56:21.269 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:56:21.269 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:56:21.269 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:56:21.269 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:56:21.269 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:56:21.269 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:56:21.269 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:56:21.269 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:56:21.269 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:56:21.269 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:56:21.269 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:56:21.269 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:56:21.269 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:56:21.270 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:56:21.270 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:56:21.270 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:56:21.270 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:56:21.270 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:56:21.270 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:56:21.270 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:56:21.270 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:56:21.270 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:56:21.270 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:56:21.270 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:56:21.270 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:56:21.270 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:56:21.270 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:56:21.270 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:56:21.271 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:56:21.271 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:56:21.271 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:56:21.271 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:56:21.271 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:56:21.271 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:56:21.271 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:56:21.271 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:56:21.271 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:56:21.271 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:56:21.271 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:56:21.273 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.238+0000 7f603ff9a640 10 librados: watch_flush enter 2024-09-16T03:56:21.273 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.238+0000 7f603ff9a640 10 librados: watch_flush exit 2024-09-16T03:56:21.273 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.238+0000 7f603ff9a640 20 client.824736.objecter shutdown clearing up homeless session... 2024-09-16T03:56:21.274 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.238+0000 7f603ff9a640 10 client.824736.objecter successfully canceled tick 2024-09-16T03:56:21.274 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.238+0000 7f603ff9a640 1 -- 172.21.15.42:0/3052015911 >> v2:172.21.15.154:6812/254845342 conn(0x7f601c037810 msgr2=0x7f601c039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:56:21.274 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.238+0000 7f603ff9a640 1 --2- 172.21.15.42:0/3052015911 >> v2:172.21.15.154:6812/254845342 conn(0x7f601c037810 0x7f601c039cd0 secure :-1 s=READY pgs=3656 cs=0 l=1 rev1=1 crypto rx=0x7f6038070890 tx=0x7f6028007e40 comp rx=0 tx=0).stop 2024-09-16T03:56:21.274 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.238+0000 7f603ff9a640 10 monclient: shutdown 2024-09-16T03:56:21.274 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.238+0000 7f603ff9a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:56:21.274 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.238+0000 7f603ff9a640 1 -- 172.21.15.42:0/3052015911 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f603811e420 msgr2=0x7f603811bf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:56:21.274 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.238+0000 7f603ff9a640 1 --2- 172.21.15.42:0/3052015911 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f603811e420 0x7f603811bf70 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f6030007e40 tx=0x7f603000e4b0 comp rx=0 tx=0).stop 2024-09-16T03:56:21.274 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.238+0000 7f603e537640 1 -- 172.21.15.42:0/3052015911 reap_dead start 2024-09-16T03:56:21.274 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.238+0000 7f603e537640 1 -- 172.21.15.42:0/3052015911 reap_dead start 2024-09-16T03:56:21.275 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.238+0000 7f603ff9a640 1 -- 172.21.15.42:0/3052015911 shutdown_connections 2024-09-16T03:56:21.275 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.238+0000 7f603ff9a640 1 -- 172.21.15.42:0/3052015911 >> 172.21.15.42:0/3052015911 conn(0x7f60380a2420 msgr2=0x7f60380a46b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:56:21.275 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.238+0000 7f603ff9a640 1 -- 172.21.15.42:0/3052015911 shutdown_connections 2024-09-16T03:56:21.275 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.238+0000 7f603ff9a640 1 -- 172.21.15.42:0/3052015911 wait complete. 2024-09-16T03:56:21.275 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.238+0000 7f603ff9a640 1 librados: shutdown 2024-09-16T03:56:21.275 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:56:21.275 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T03:56:21.275 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T03:56:21.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.386+0000 7f20d03c6640 10 monclient: get_monmap_and_config 2024-09-16T03:56:21.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.386+0000 7f20d03c6640 10 monclient: build_initial_monmap 2024-09-16T03:56:21.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.386+0000 7f20d03c6640 10 monclient: monmap: 2024-09-16T03:56:21.390 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:56:21.390 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:56:21.390 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:56:21.389030+0000 2024-09-16T03:56:21.390 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:56:21.389030+0000 2024-09-16T03:56:21.391 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:56:21.391 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:56:21.391 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:56:21.391 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:56:21.391 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:56:21.391 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:56:21.391 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:56:21.391 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:56:21.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.386+0000 7f20d03c6640 1 Processor -- start 2024-09-16T03:56:21.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.386+0000 7f20d03c6640 1 -- start start 2024-09-16T03:56:21.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.386+0000 7f20d03c6640 10 monclient: init 2024-09-16T03:56:21.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.386+0000 7f20d03c6640 10 monclient: _reopen_session rank -1 2024-09-16T03:56:21.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.386+0000 7f20d03c6640 10 monclient: _add_conns ranks=[0,3,4,1,2] 2024-09-16T03:56:21.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.386+0000 7f20d03c6640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f20c806db90 addr v1:172.21.15.42:6789/0 2024-09-16T03:56:21.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.386+0000 7f20d03c6640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f20c81340c0 0x7f20c813c190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:56:21.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.386+0000 7f20d03c6640 10 monclient(hunting): picked mon.noname-a con 0x7f20c81340c0 addr v2:172.21.15.42:3300/0 2024-09-16T03:56:21.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.386+0000 7f20d03c6640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f20c813c6d0 0x7f20c813eac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:56:21.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.386+0000 7f20d03c6640 10 monclient(hunting): picked mon.noname-c con 0x7f20c813c6d0 addr v2:172.21.15.154:3300/0 2024-09-16T03:56:21.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.386+0000 7f20d03c6640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f20c806ece0 con 0x7f20c806db90 2024-09-16T03:56:21.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.386+0000 7f20d03c6640 10 monclient(hunting): start opening mon connection 2024-09-16T03:56:21.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.386+0000 7f20d03c6640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f20c806df70 con 0x7f20c81340c0 2024-09-16T03:56:21.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.386+0000 7f20d03c6640 10 monclient(hunting): start opening mon connection 2024-09-16T03:56:21.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.386+0000 7f20d03c6640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f20c809bf80 con 0x7f20c813c6d0 2024-09-16T03:56:21.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.386+0000 7f20d03c6640 10 monclient(hunting): _renew_subs 2024-09-16T03:56:21.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.386+0000 7f20d03c6640 10 monclient(hunting): authenticate will time out at 1828.358442s 2024-09-16T03:56:21.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.386+0000 7f20ce162640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f20c806db90 0x7f20c80a37f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:33382/0 (socket says 172.21.15.42:33382) 2024-09-16T03:56:21.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.386+0000 7f20ce162640 1 -- 172.21.15.42:0/3898372855 learned_addr learned my addr 172.21.15.42:0/3898372855 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:56:21.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.386+0000 7f20cd961640 1 --2- 172.21.15.42:0/3898372855 >> v2:172.21.15.42:3300/0 conn(0x7f20c81340c0 0x7f20c813c190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:56:21.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.386+0000 7f20cd961640 10 monclient(hunting): get_auth_request con 0x7f20c81340c0 auth_method 0 2024-09-16T03:56:21.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.386+0000 7f20cd961640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:56:21.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.386+0000 7f20cd961640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:56:21.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.386+0000 7f20cd961640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:56:21.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.386+0000 7f20cd961640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:56:21.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.386+0000 7f20cd961640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:56:21.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.386+0000 7f20cd961640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:56:21.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.386+0000 7f20ce963640 1 --2- 172.21.15.42:0/3898372855 >> v2:172.21.15.154:3300/0 conn(0x7f20c813c6d0 0x7f20c813eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:56:21.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.386+0000 7f20ce963640 10 monclient(hunting): get_auth_request con 0x7f20c813c6d0 auth_method 0 2024-09-16T03:56:21.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.386+0000 7f20ce963640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:56:21.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.386+0000 7f20ce963640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:56:21.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.386+0000 7f20ce963640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:56:21.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.386+0000 7f20cd961640 10 monclient(hunting): handle_auth_done global_id 824739 payload 274 2024-09-16T03:56:21.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.386+0000 7f20cd160640 1 -- 172.21.15.42:0/3898372855 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3788181136 0 0) 0x7f20c806ece0 con 0x7f20c806db90 2024-09-16T03:56:21.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.386+0000 7f20cd961640 1 -- 172.21.15.42:0/3898372855 >> v2:172.21.15.154:3300/0 conn(0x7f20c813c6d0 msgr2=0x7f20c813eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:56:21.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.386+0000 7f20cd961640 1 --2- 172.21.15.42:0/3898372855 >> v2:172.21.15.154:3300/0 conn(0x7f20c813c6d0 0x7f20c813eac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:56:21.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.386+0000 7f20cd961640 1 -- 172.21.15.42:0/3898372855 >> v1:172.21.15.42:6789/0 conn(0x7f20c806db90 legacy=0x7f20c80a37f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:56:21.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.386+0000 7f20cd961640 10 monclient: _finish_hunting 0 2024-09-16T03:56:21.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.386+0000 7f20cd961640 1 monclient: found mon.noname-a 2024-09-16T03:56:21.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.386+0000 7f20cd961640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:56:21.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.386+0000 7f20cd961640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:56:21.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.386+0000 7f20cd961640 1 -- 172.21.15.42:0/3898372855 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20c81344d0 con 0x7f20c81340c0 2024-09-16T03:56:21.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.386+0000 7f20cd160640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T03:56:21.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.390+0000 7f20cd961640 1 --2- 172.21.15.42:0/3898372855 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f20c81340c0 0x7f20c813c190 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f20c809bba0 tx=0x7f20b801cda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f77b6a83b1322626 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:56:21.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.390+0000 7f20cd160640 1 -- 172.21.15.42:0/3898372855 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20b80b0070 con 0x7f20c81340c0 2024-09-16T03:56:21.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.390+0000 7f20cd160640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:56:21.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.390+0000 7f20cd160640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:56:21.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.390+0000 7f20cd160640 10 monclient: dump: 2024-09-16T03:56:21.405 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:56:21.405 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:56:21.405 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:21.405 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:21.405 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:56:21.405 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:56:21.405 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:56:21.405 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:56:21.405 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:56:21.406 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:56:21.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.390+0000 7f20cd160640 10 monclient: _finish_auth 0 2024-09-16T03:56:21.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.390+0000 7f20cd160640 10 monclient: _check_auth_tickets 2024-09-16T03:56:21.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.390+0000 7f20cd160640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:56:21.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.390+0000 7f20d03c6640 5 monclient: authenticate success, global_id 824739 2024-09-16T03:56:21.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.390+0000 7f20d03c6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:56:21.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.390+0000 7f20cd160640 1 -- 172.21.15.42:0/3898372855 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f20b80ab600 con 0x7f20c81340c0 2024-09-16T03:56:21.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.390+0000 7f20cd160640 10 monclient: handle_config config(0 keys) 2024-09-16T03:56:21.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.390+0000 7f20d03c6640 10 monclient: get_monmap_and_config success 2024-09-16T03:56:21.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.390+0000 7f20d03c6640 4 set_mon_vals no callback set 2024-09-16T03:56:21.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.390+0000 7f20cd160640 1 -- 172.21.15.42:0/3898372855 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20b80ab9d0 con 0x7f20c81340c0 2024-09-16T03:56:21.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.390+0000 7f20cd160640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:56:21.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.390+0000 7f20cd160640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:56:21.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.390+0000 7f20cd160640 10 monclient: dump: 2024-09-16T03:56:21.407 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:56:21.407 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:56:21.407 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:21.407 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:21.407 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:56:21.407 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:56:21.407 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:56:21.407 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:56:21.407 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:56:21.407 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:56:21.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.390+0000 7f20d03c6640 10 monclient: shutdown 2024-09-16T03:56:21.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.390+0000 7f20d03c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:56:21.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.390+0000 7f20d03c6640 1 -- 172.21.15.42:0/3898372855 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f20c81340c0 msgr2=0x7f20c813c190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:56:21.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.390+0000 7f20d03c6640 1 --2- 172.21.15.42:0/3898372855 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f20c81340c0 0x7f20c813c190 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f20c809bba0 tx=0x7f20b801cda0 comp rx=0 tx=0).stop 2024-09-16T03:56:21.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.390+0000 7f20d03c6640 1 -- 172.21.15.42:0/3898372855 shutdown_connections 2024-09-16T03:56:21.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.390+0000 7f20d03c6640 1 --2- 172.21.15.42:0/3898372855 >> v2:172.21.15.154:3300/0 conn(0x7f20c813c6d0 0x7f20c813eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:56:21.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.390+0000 7f20d03c6640 1 --2- 172.21.15.42:0/3898372855 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f20c81340c0 0x7f20c813c190 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:56:21.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.390+0000 7f20d03c6640 1 -- 172.21.15.42:0/3898372855 >> 172.21.15.42:0/3898372855 conn(0x7f20c80a9fb0 msgr2=0x7f20c80aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:56:21.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.390+0000 7f20d03c6640 1 -- 172.21.15.42:0/3898372855 shutdown_connections 2024-09-16T03:56:21.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.390+0000 7f20d03c6640 1 -- 172.21.15.42:0/3898372855 wait complete. 2024-09-16T03:56:21.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.390+0000 7f20d03c6640 10 monclient: build_initial_monmap 2024-09-16T03:56:21.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.390+0000 7f20d03c6640 10 monclient: monmap: 2024-09-16T03:56:21.408 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:56:21.408 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:56:21.409 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:56:21.409 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:56:21.409 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:56:21.409 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:56:21.409 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:56:21.409 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:56:21.409 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:56:21.409 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:56:21.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.390+0000 7f20d03c6640 1 librados: starting msgr at 2024-09-16T03:56:21.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.390+0000 7f20d03c6640 1 librados: starting objecter 2024-09-16T03:56:21.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.390+0000 7f20d03c6640 1 Processor -- start 2024-09-16T03:56:21.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.390+0000 7f20d03c6640 1 -- start start 2024-09-16T03:56:21.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.390+0000 7f20d03c6640 1 librados: setting wanted keys 2024-09-16T03:56:21.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.390+0000 7f20d03c6640 1 librados: calling monclient init 2024-09-16T03:56:21.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.390+0000 7f20d03c6640 10 monclient: init 2024-09-16T03:56:21.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.390+0000 7f20d03c6640 10 monclient: _reopen_session rank -1 2024-09-16T03:56:21.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.390+0000 7f20d03c6640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T03:56:21.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.390+0000 7f20d03c6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f20c806db90 0x7f20c81cecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:56:21.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.390+0000 7f20d03c6640 10 monclient(hunting): picked mon.noname-b con 0x7f20c806db90 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:56:21.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.390+0000 7f20d03c6640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f20c81340c0 0x7f20c813ace0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:56:21.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.390+0000 7f20d03c6640 10 monclient(hunting): picked mon.noname-c con 0x7f20c81340c0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:56:21.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.390+0000 7f20d03c6640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f20c813c6d0 0x7f20c813b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:56:21.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.390+0000 7f20d03c6640 10 monclient(hunting): picked mon.noname-a con 0x7f20c813c6d0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:56:21.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.390+0000 7f20d03c6640 10 monclient(hunting): start opening mon connection 2024-09-16T03:56:21.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.390+0000 7f20d03c6640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f20c81419a0 con 0x7f20c813c6d0 2024-09-16T03:56:21.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.390+0000 7f20d03c6640 10 monclient(hunting): start opening mon connection 2024-09-16T03:56:21.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.390+0000 7f20d03c6640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f20c8141820 con 0x7f20c806db90 2024-09-16T03:56:21.411 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.390+0000 7f20d03c6640 10 monclient(hunting): start opening mon connection 2024-09-16T03:56:21.411 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.390+0000 7f20d03c6640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f20c8141b20 con 0x7f20c81340c0 2024-09-16T03:56:21.411 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.390+0000 7f20d03c6640 10 monclient(hunting): _renew_subs 2024-09-16T03:56:21.411 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.390+0000 7f20d03c6640 10 monclient(hunting): authenticate will time out at 1828.363478s 2024-09-16T03:56:21.411 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.390+0000 7f20ce963640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f20c813c6d0 0x7f20c813b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:56:21.411 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.390+0000 7f20cd961640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f20c81340c0 0x7f20c813ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:56:21.411 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.390+0000 7f20ce963640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f20c813c6d0 0x7f20c813b220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:37132/0 (socket says 172.21.15.42:37132) 2024-09-16T03:56:21.411 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.390+0000 7f20ce963640 1 -- 172.21.15.42:0/153293145 learned_addr learned my addr 172.21.15.42:0/153293145 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:56:21.411 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.390+0000 7f20ce963640 10 monclient(hunting): get_auth_request con 0x7f20c813c6d0 auth_method 0 2024-09-16T03:56:21.411 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.390+0000 7f20ce963640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:56:21.411 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.390+0000 7f20ce963640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:56:21.411 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.390+0000 7f20ce963640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:56:21.411 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.390+0000 7f20cd961640 10 monclient(hunting): get_auth_request con 0x7f20c81340c0 auth_method 0 2024-09-16T03:56:21.411 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.390+0000 7f20cd961640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:56:21.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.390+0000 7f20cd961640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:56:21.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.390+0000 7f20cd961640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:56:21.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.390+0000 7f20cd961640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:56:21.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.390+0000 7f20cd961640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:56:21.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.390+0000 7f20cd961640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:56:21.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.390+0000 7f20ce963640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:56:21.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.390+0000 7f20ce963640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:56:21.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.390+0000 7f20ce963640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:56:21.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.390+0000 7f20ce162640 1 --2- 172.21.15.42:0/153293145 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f20c806db90 0x7f20c81cecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:56:21.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.390+0000 7f20ce162640 10 monclient(hunting): get_auth_request con 0x7f20c806db90 auth_method 0 2024-09-16T03:56:21.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.390+0000 7f20ce162640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:56:21.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.390+0000 7f20ce162640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:56:21.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.390+0000 7f20ce162640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:56:21.416 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.390+0000 7f20cd961640 10 monclient(hunting): handle_auth_done global_id 854330 payload 824 2024-09-16T03:56:21.416 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.390+0000 7f20cd961640 1 -- 172.21.15.42:0/153293145 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f20c806db90 msgr2=0x7f20c81cecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:56:21.417 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.390+0000 7f20cd961640 1 --2- 172.21.15.42:0/153293145 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f20c806db90 0x7f20c81cecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:56:21.417 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.390+0000 7f20cd961640 1 -- 172.21.15.42:0/153293145 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f20c813c6d0 msgr2=0x7f20c813b220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:56:21.417 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.394+0000 7f20cd961640 1 --2- 172.21.15.42:0/153293145 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f20c813c6d0 0x7f20c813b220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:56:21.417 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.394+0000 7f20cd961640 10 monclient: _finish_hunting 0 2024-09-16T03:56:21.417 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.394+0000 7f20cd961640 1 monclient: found mon.noname-c 2024-09-16T03:56:21.417 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.394+0000 7f20cd961640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:56:21.417 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.394+0000 7f20cd961640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:56:21.417 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.394+0000 7f20cd961640 1 -- 172.21.15.42:0/153293145 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20c81364a0 con 0x7f20c81340c0 2024-09-16T03:56:21.417 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.394+0000 7f20ce963640 1 --2- 172.21.15.42:0/153293145 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f20c813c6d0 0x7f20c813b220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:56:21.417 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.394+0000 7f20cd961640 1 --2- 172.21.15.42:0/153293145 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f20c81340c0 0x7f20c813ace0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f20b80ab000 tx=0x7f20b801ab80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:56:21.417 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.394+0000 7f20beffd640 10 client.?.objecter ms_handle_connect 0x7f20c81340c0 2024-09-16T03:56:21.417 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.394+0000 7f20beffd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:56:21.417 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.394+0000 7f20beffd640 1 -- 172.21.15.42:0/153293145 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20b80bc070 con 0x7f20c81340c0 2024-09-16T03:56:21.418 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.394+0000 7f20beffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:56:21.418 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.394+0000 7f20beffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:56:21.418 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.394+0000 7f20beffd640 10 monclient: dump: 2024-09-16T03:56:21.418 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:56:21.418 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:56:21.418 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:21.418 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:21.418 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:56:21.418 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:56:21.418 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:56:21.418 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:56:21.418 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:56:21.418 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:56:21.418 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.394+0000 7f20beffd640 10 monclient: _finish_auth 0 2024-09-16T03:56:21.419 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.394+0000 7f20beffd640 10 monclient: _check_auth_tickets 2024-09-16T03:56:21.419 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.394+0000 7f20beffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:56:21.419 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.394+0000 7f20d03c6640 5 monclient: authenticate success, global_id 854330 2024-09-16T03:56:21.419 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.394+0000 7f20beffd640 1 -- 172.21.15.42:0/153293145 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f20b80b1040 con 0x7f20c81340c0 2024-09-16T03:56:21.419 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.394+0000 7f20beffd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:56:21.419 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.394+0000 7f20beffd640 1 -- 172.21.15.42:0/153293145 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20b80b0050 con 0x7f20c81340c0 2024-09-16T03:56:21.419 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.394+0000 7f20beffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:56:21.419 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.394+0000 7f20beffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:56:21.419 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.394+0000 7f20beffd640 10 monclient: dump: 2024-09-16T03:56:21.419 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:56:21.419 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:56:21.419 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:21.419 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:21.419 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:56:21.420 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:56:21.420 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:56:21.420 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:56:21.420 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:56:21.420 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:56:21.420 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.394+0000 7f20cc95f640 4 set_mon_vals no callback set 2024-09-16T03:56:21.420 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.394+0000 7f20d03c6640 10 monclient: _renew_subs 2024-09-16T03:56:21.420 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.394+0000 7f20d03c6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:56:21.420 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.394+0000 7f20d03c6640 1 -- 172.21.15.42:0/153293145 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20c813bcf0 con 0x7f20c81340c0 2024-09-16T03:56:21.420 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.394+0000 7f20d03c6640 10 client.854330.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:56:21.420 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.394+0000 7f20d03c6640 10 monclient: _renew_subs 2024-09-16T03:56:21.420 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.394+0000 7f20d03c6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:56:21.420 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.394+0000 7f20d03c6640 1 -- 172.21.15.42:0/153293145 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f20c81cfb90 con 0x7f20c81340c0 2024-09-16T03:56:21.421 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.394+0000 7f20d03c6640 1 librados: init done 2024-09-16T03:56:21.421 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.394+0000 7f20d03c6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:56:21.421 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.394+0000 7f20d03c6640 1 --2- 172.21.15.42:0/153293145 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f20c8134ff0 0x7f20c80a1d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:56:21.421 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.394+0000 7f20d03c6640 10 monclient(hunting): start opening mon connection 2024-09-16T03:56:21.421 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.394+0000 7f20d03c6640 1 -- 172.21.15.42:0/153293145 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f20c8136090 con 0x7f20c8134ff0 2024-09-16T03:56:21.421 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.394+0000 7f20beffd640 1 -- 172.21.15.42:0/153293145 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f20b8004270 con 0x7f20c81340c0 2024-09-16T03:56:21.421 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.394+0000 7f20beffd640 1 --2- 172.21.15.42:0/153293145 >> v2:172.21.15.154:6812/254845342 conn(0x7f20ac037810 0x7f20ac039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:56:21.421 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.394+0000 7f20ce162640 1 --2- 172.21.15.42:0/153293145 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f20c8134ff0 0x7f20c80a1d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:56:21.421 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.394+0000 7f20beffd640 1 -- 172.21.15.42:0/153293145 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(442..442 src has 330..442) ==== 5741+0+0 (secure 0 0 0) 0x7f20b80e29a0 con 0x7f20c81340c0 2024-09-16T03:56:21.421 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.394+0000 7f20beffd640 10 client.854330.objecter ms_dispatch 0x7f20c8000ba0 osd_map(442..442 src has 330..442) 2024-09-16T03:56:21.421 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.394+0000 7f20beffd640 3 client.854330.objecter handle_osd_map got epochs [442,442] > 0 2024-09-16T03:56:21.421 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.394+0000 7f20beffd640 3 client.854330.objecter handle_osd_map decoding full epoch 442 2024-09-16T03:56:21.421 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.394+0000 7f20ce162640 10 monclient: get_auth_request con 0x7f20c8134ff0 auth_method 0 2024-09-16T03:56:21.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.394+0000 7f20ce162640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:56:21.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.394+0000 7f20ce162640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:56:21.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.394+0000 7f20ce162640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:56:21.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.394+0000 7f20beffd640 20 client.854330.objecter dump_active .. 0 homeless 2024-09-16T03:56:21.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.394+0000 7f20ce162640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:56:21.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.394+0000 7f20ce162640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:56:21.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.394+0000 7f20ce162640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:56:21.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.394+0000 7f20ce963640 1 --2- 172.21.15.42:0/153293145 >> v2:172.21.15.154:6812/254845342 conn(0x7f20ac037810 0x7f20ac039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:56:21.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.394+0000 7f20ce963640 10 monclient: get_auth_request con 0x7f20ac037810 auth_method 0 2024-09-16T03:56:21.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.394+0000 7f20ce162640 10 monclient(hunting): handle_auth_done global_id 874141 payload 824 2024-09-16T03:56:21.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.394+0000 7f20ce162640 1 -- 172.21.15.42:0/153293145 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f20c81cf200 con 0x7f20c8134ff0 2024-09-16T03:56:21.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.394+0000 7f20beffd640 10 client.854330.objecter ms_handle_connect 0x7f20c8134ff0 2024-09-16T03:56:21.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.394+0000 7f20beffd640 10 client.854330.objecter resend_mon_ops 2024-09-16T03:56:21.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.394+0000 7f20ce162640 1 --2- 172.21.15.42:0/153293145 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f20c8134ff0 0x7f20c80a1d20 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f20c40077f0 tx=0x7f20c4001820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:56:21.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.394+0000 7f20beffd640 10 client.854330.objecter ms_handle_connect 0x7f20ac037810 2024-09-16T03:56:21.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.394+0000 7f20ce963640 1 --2- 172.21.15.42:0/153293145 >> v2:172.21.15.154:6812/254845342 conn(0x7f20ac037810 0x7f20ac039cd0 secure :-1 s=READY pgs=3657 cs=0 l=1 rev1=1 crypto rx=0x7f20c0007880 tx=0x7f20c00098d0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:56:21.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.394+0000 7f20beffd640 1 -- 172.21.15.42:0/153293145 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20c4019070 con 0x7f20c8134ff0 2024-09-16T03:56:21.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.394+0000 7f20beffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:56:21.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.394+0000 7f20beffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:56:21.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.394+0000 7f20beffd640 10 monclient: dump: 2024-09-16T03:56:21.423 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:56:21.423 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:56:21.424 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:21.424 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:21.424 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:56:21.424 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:56:21.424 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:56:21.424 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:56:21.424 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:56:21.424 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:56:21.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.398+0000 7f20beffd640 1 -- 172.21.15.42:0/153293145 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f20c4010510 con 0x7f20c8134ff0 2024-09-16T03:56:21.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.398+0000 7f20beffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:56:21.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.398+0000 7f20beffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:56:21.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.398+0000 7f20beffd640 1 -- 172.21.15.42:0/153293145 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f20c8134ff0 msgr2=0x7f20c80a1d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:56:21.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.398+0000 7f20beffd640 1 --2- 172.21.15.42:0/153293145 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f20c8134ff0 0x7f20c80a1d20 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f20c40077f0 tx=0x7f20c4001820 comp rx=0 tx=0).stop 2024-09-16T03:56:21.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.398+0000 7f20beffd640 1 -- 172.21.15.42:0/153293145 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f20c8134ff0 msgr2=0x7f20c80a1d20 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:56:21.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.398+0000 7f20beffd640 1 --2- 172.21.15.42:0/153293145 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f20c8134ff0 0x7f20c80a1d20 secure :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f20c40077f0 tx=0x7f20c4001820 comp rx=0 tx=0).stop 2024-09-16T03:56:21.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.402+0000 7f20d03c6640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:56:21.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.402+0000 7f20d03c6640 1 --2- 172.21.15.42:0/153293145 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f20c802c650 0x7f20c802ca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:56:21.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.402+0000 7f20d03c6640 10 monclient(hunting): start opening mon connection 2024-09-16T03:56:21.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.402+0000 7f20d03c6640 1 -- 172.21.15.42:0/153293145 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f20c80a3200 con 0x7f20c802c650 2024-09-16T03:56:21.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.402+0000 7f20ce162640 1 --2- 172.21.15.42:0/153293145 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f20c802c650 0x7f20c802ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:56:21.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.402+0000 7f20ce162640 10 monclient: get_auth_request con 0x7f20c802c650 auth_method 0 2024-09-16T03:56:21.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.402+0000 7f20ce162640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:56:21.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.402+0000 7f20ce162640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:56:21.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.402+0000 7f20ce162640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:56:21.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.402+0000 7f20ce162640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:56:21.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.402+0000 7f20ce162640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:56:21.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.402+0000 7f20ce162640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:56:21.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.402+0000 7f20ce162640 10 monclient(hunting): handle_auth_done global_id 874144 payload 824 2024-09-16T03:56:21.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.402+0000 7f20ce162640 1 -- 172.21.15.42:0/153293145 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f20c809de50 con 0x7f20c802c650 2024-09-16T03:56:21.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.402+0000 7f20ce162640 1 --2- 172.21.15.42:0/153293145 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f20c802c650 0x7f20c802ca60 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f20c4001ec0 tx=0x7f20c4001090 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:56:21.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.402+0000 7f20beffd640 10 client.854330.objecter ms_handle_connect 0x7f20c802c650 2024-09-16T03:56:21.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.402+0000 7f20beffd640 10 client.854330.objecter resend_mon_ops 2024-09-16T03:56:21.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.410+0000 7f20beffd640 1 -- 172.21.15.42:0/153293145 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20c4019070 con 0x7f20c802c650 2024-09-16T03:56:21.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.410+0000 7f20beffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:56:21.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.410+0000 7f20beffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:56:21.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.410+0000 7f20beffd640 10 monclient: dump: 2024-09-16T03:56:21.428 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:56:21.428 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:56:21.428 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:21.428 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:21.428 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:56:21.428 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:56:21.429 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:56:21.429 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:56:21.429 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:56:21.429 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:56:21.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.410+0000 7f20beffd640 1 -- 172.21.15.42:0/153293145 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4545 (secure 0 0 0) 0x7f20c401a040 con 0x7f20c802c650 2024-09-16T03:56:21.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.410+0000 7f20beffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:56:21.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.410+0000 7f20beffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:56:21.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.410+0000 7f20beffd640 1 -- 172.21.15.42:0/153293145 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f20c802c650 msgr2=0x7f20c802ca60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:56:21.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.410+0000 7f20beffd640 1 --2- 172.21.15.42:0/153293145 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f20c802c650 0x7f20c802ca60 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f20c4001ec0 tx=0x7f20c4001090 comp rx=0 tx=0).stop 2024-09-16T03:56:21.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.410+0000 7f20beffd640 1 -- 172.21.15.42:0/153293145 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f20c802c650 msgr2=0x7f20c802ca60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:56:21.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.410+0000 7f20beffd640 1 --2- 172.21.15.42:0/153293145 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f20c802c650 0x7f20c802ca60 secure :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f20c4001ec0 tx=0x7f20c4001090 comp rx=0 tx=0).stop 2024-09-16T03:56:21.430 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:56:21.430 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:56:21.430 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:56:21.430 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T03:56:21.430 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 128, 2024-09-16T03:56:21.430 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 12691, 2024-09-16T03:56:21.430 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:56:21.430 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:56:21.430 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T03:56:21.430 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T03:56:21.430 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:56:21.430 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 4, 2024-09-16T03:56:21.431 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:56:21.431 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:56:21.431 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:56:21.431 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:56:21.431 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:56:21.431 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:56:21.431 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:56:21.431 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:56:21.431 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:56:21.431 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:56:21.431 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:56:21.431 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:56:21.431 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:56:21.431 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:56:21.432 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:56:21.432 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:56:21.432 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:56:21.432 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:56:21.432 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:56:21.432 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:56:21.432 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:56:21.432 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:56:21.432 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:56:21.432 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:56:21.432 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:56:21.432 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:56:21.432 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:56:21.433 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:56:21.433 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:56:21.433 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:56:21.433 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:56:21.433 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:56:21.433 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:56:21.433 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:56:21.433 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:56:21.433 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:56:21.433 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:56:21.433 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:56:21.433 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:56:21.433 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:56:21.434 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:56:21.434 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:56:21.434 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:56:21.434 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:56:21.434 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:56:21.434 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:56:21.434 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:56:21.434 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:56:21.434 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:56:21.434 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:56:21.434 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:56:21.434 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:56:21.434 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:56:21.435 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:56:21.435 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:56:21.435 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:56:21.435 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:56:21.435 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:56:21.435 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:56:21.435 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:56:21.435 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:56:21.435 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:56:21.435 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:56:21.435 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:56:21.435 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:56:21.435 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:56:21.435 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:56:21.436 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:56:21.436 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:56:21.436 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:56:21.436 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:56:21.436 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:56:21.436 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:56:21.436 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:56:21.436 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:56:21.436 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:56:21.436 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:56:21.436 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:56:21.436 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:56:21.436 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:56:21.437 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:56:21.437 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:56:21.437 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:56:21.437 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:56:21.437 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:56:21.437 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:56:21.437 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:56:21.437 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:56:21.437 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:56:21.437 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:56:21.437 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:56:21.437 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:56:21.437 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:56:21.437 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:56:21.438 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:56:21.438 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:56:21.438 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:56:21.438 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:56:21.438 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:56:21.438 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:56:21.438 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:56:21.438 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:56:21.438 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:56:21.438 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:56:21.438 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:56:21.438 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:56:21.438 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:56:21.438 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:56:21.439 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:56:21.439 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:56:21.439 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:56:21.439 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:56:21.439 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:56:21.439 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:56:21.439 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:56:21.439 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:56:21.439 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:56:21.439 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:56:21.439 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:56:21.439 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:56:21.439 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:56:21.439 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:56:21.440 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:56:21.440 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:56:21.440 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:56:21.440 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:56:21.440 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:56:21.440 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:56:21.440 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:56:21.440 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:56:21.440 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:56:21.440 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:56:21.440 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:56:21.440 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:56:21.440 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:56:21.441 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:56:21.441 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:56:21.441 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:56:21.441 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:56:21.441 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:56:21.441 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:56:21.441 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:56:21.441 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:56:21.441 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:56:21.441 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:56:21.441 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:56:21.441 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:56:21.441 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:56:21.441 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:56:21.442 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:56:21.442 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:56:21.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.410+0000 7f20d03c6640 10 librados: watch_flush enter 2024-09-16T03:56:21.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.410+0000 7f20d03c6640 10 librados: watch_flush exit 2024-09-16T03:56:21.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.410+0000 7f20d03c6640 20 client.854330.objecter shutdown clearing up homeless session... 2024-09-16T03:56:21.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.410+0000 7f20d03c6640 10 client.854330.objecter successfully canceled tick 2024-09-16T03:56:21.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.410+0000 7f20d03c6640 1 -- 172.21.15.42:0/153293145 >> v2:172.21.15.154:6812/254845342 conn(0x7f20ac037810 msgr2=0x7f20ac039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:56:21.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.410+0000 7f20d03c6640 1 --2- 172.21.15.42:0/153293145 >> v2:172.21.15.154:6812/254845342 conn(0x7f20ac037810 0x7f20ac039cd0 secure :-1 s=READY pgs=3657 cs=0 l=1 rev1=1 crypto rx=0x7f20c0007880 tx=0x7f20c00098d0 comp rx=0 tx=0).stop 2024-09-16T03:56:21.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.410+0000 7f20d03c6640 10 monclient: shutdown 2024-09-16T03:56:21.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.410+0000 7f20d03c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:56:21.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.410+0000 7f20d03c6640 1 -- 172.21.15.42:0/153293145 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f20c81340c0 msgr2=0x7f20c813ace0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:56:21.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.410+0000 7f20d03c6640 1 --2- 172.21.15.42:0/153293145 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f20c81340c0 0x7f20c813ace0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f20b80ab000 tx=0x7f20b801ab80 comp rx=0 tx=0).stop 2024-09-16T03:56:21.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.410+0000 7f20ce963640 1 -- 172.21.15.42:0/153293145 reap_dead start 2024-09-16T03:56:21.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.410+0000 7f20d03c6640 1 -- 172.21.15.42:0/153293145 shutdown_connections 2024-09-16T03:56:21.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.410+0000 7f20d03c6640 1 --2- 172.21.15.42:0/153293145 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f20c81340c0 0x7f20c813ace0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:56:21.445 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.410+0000 7f20d03c6640 1 -- 172.21.15.42:0/153293145 >> 172.21.15.42:0/153293145 conn(0x7f20c80a9fb0 msgr2=0x7f20c80a7ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:56:21.445 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.410+0000 7f20d03c6640 1 -- 172.21.15.42:0/153293145 shutdown_connections 2024-09-16T03:56:21.445 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.410+0000 7f20d03c6640 1 -- 172.21.15.42:0/153293145 wait complete. 2024-09-16T03:56:21.445 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.410+0000 7f20d03c6640 1 librados: shutdown 2024-09-16T03:56:21.445 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:56:21.445 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-09-16T03:56:21.445 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-16T03:56:21.445 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-09-16T03:56:21.446 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-16T03:56:21.446 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-16T03:56:21.446 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T03:56:21.461 INFO:tasks.ceph.mon.b:Stopped 2024-09-16T03:56:21.461 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T03:56:21.461 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:56:21.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.562+0000 7f30495b1640 10 monclient: get_monmap_and_config 2024-09-16T03:56:21.568 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.562+0000 7f30495b1640 10 monclient: build_initial_monmap 2024-09-16T03:56:21.568 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.562+0000 7f30495b1640 10 monclient: monmap: 2024-09-16T03:56:21.568 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:56:21.568 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:56:21.568 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:56:21.566255+0000 2024-09-16T03:56:21.568 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:56:21.566255+0000 2024-09-16T03:56:21.568 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:56:21.568 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:56:21.568 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:56:21.568 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:56:21.568 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:56:21.568 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:56:21.569 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:56:21.569 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:56:21.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.562+0000 7f30495b1640 1 Processor -- start 2024-09-16T03:56:21.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.562+0000 7f30495b1640 1 -- start start 2024-09-16T03:56:21.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.562+0000 7f30495b1640 10 monclient: init 2024-09-16T03:56:21.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.562+0000 7f30495b1640 10 monclient: _reopen_session rank -1 2024-09-16T03:56:21.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.562+0000 7f30495b1640 10 monclient: _add_conns ranks=[1,0,4,3,2] 2024-09-16T03:56:21.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.562+0000 7f30495b1640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f30440a1330 addr v1:172.21.15.154:6789/0 2024-09-16T03:56:21.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.562+0000 7f30495b1640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f304406dac0 addr v1:172.21.15.42:6789/0 2024-09-16T03:56:21.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.562+0000 7f30495b1640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f30440a1e20 0x7f304413da70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:56:21.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.562+0000 7f30495b1640 10 monclient(hunting): picked mon.noname-c con 0x7f30440a1e20 addr v2:172.21.15.154:3300/0 2024-09-16T03:56:21.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.562+0000 7f30495b1640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f304409c090 con 0x7f304406dac0 2024-09-16T03:56:21.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.562+0000 7f30495b1640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f304409f610 con 0x7f30440a1330 2024-09-16T03:56:21.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.562+0000 7f30495b1640 10 monclient(hunting): start opening mon connection 2024-09-16T03:56:21.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.562+0000 7f30495b1640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f304409f8f0 con 0x7f30440a1e20 2024-09-16T03:56:21.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.562+0000 7f30495b1640 10 monclient(hunting): _renew_subs 2024-09-16T03:56:21.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.562+0000 7f30495b1640 10 monclient(hunting): authenticate will time out at 1828.535714s 2024-09-16T03:56:21.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.562+0000 7f3042ffd640 1 -- >> v1:172.21.15.154:6789/0 conn(0x7f30440a1330 legacy=0x7f30440a1710 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.154:6789/0 2024-09-16T03:56:21.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.562+0000 7f30427fc640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f304406dac0 0x7f304413a340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:33392/0 (socket says 172.21.15.42:33392) 2024-09-16T03:56:21.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.562+0000 7f30437fe640 1 -- >> v2:172.21.15.154:3300/0 conn(0x7f30440a1e20 msgr2=0x7f304413da70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T03:56:21.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.562+0000 7f30427fc640 1 -- 172.21.15.42:0/1243028877 learned_addr learned my addr 172.21.15.42:0/1243028877 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:56:21.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.562+0000 7f30437fe640 1 --2- 172.21.15.42:0/1243028877 >> v2:172.21.15.154:3300/0 conn(0x7f30440a1e20 0x7f304413da70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:56:21.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.566+0000 7f3041ffb640 1 -- 172.21.15.42:0/1243028877 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2174652669 0 0) 0x7f304409c090 con 0x7f304406dac0 2024-09-16T03:56:21.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.566+0000 7f3041ffb640 10 monclient(hunting): _negotiate 2024-09-16T03:56:21.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.566+0000 7f3041ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:56:21.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.566+0000 7f3041ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:56:21.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.566+0000 7f3041ffb640 10 monclient(hunting): my global_id is 824748 2024-09-16T03:56:21.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.566+0000 7f3041ffb640 1 -- 172.21.15.42:0/1243028877 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f302c003530 con 0x7f304406dac0 2024-09-16T03:56:21.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.566+0000 7f3041ffb640 1 -- 172.21.15.42:0/1243028877 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1587472438 0 0) 0x7f302c003530 con 0x7f304406dac0 2024-09-16T03:56:21.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.566+0000 7f3041ffb640 1 -- 172.21.15.42:0/1243028877 >> v2:172.21.15.154:3300/0 conn(0x7f30440a1e20 msgr2=0x7f304413da70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:56:21.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.566+0000 7f3041ffb640 1 --2- 172.21.15.42:0/1243028877 >> v2:172.21.15.154:3300/0 conn(0x7f30440a1e20 0x7f304413da70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:56:21.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.566+0000 7f3041ffb640 1 -- 172.21.15.42:0/1243028877 >> v1:172.21.15.154:6789/0 conn(0x7f30440a1330 legacy=0x7f30440a1710 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:56:21.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.566+0000 7f3041ffb640 10 monclient: _finish_hunting 0 2024-09-16T03:56:21.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.566+0000 7f3041ffb640 1 monclient: found mon.noname-a-legacy 2024-09-16T03:56:21.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.566+0000 7f3041ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:56:21.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.566+0000 7f3041ffb640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.42:6789/0 2024-09-16T03:56:21.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.566+0000 7f3041ffb640 1 -- 172.21.15.42:0/1243028877 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f304409fbd0 con 0x7f304406dac0 2024-09-16T03:56:21.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.566+0000 7f3041ffb640 10 monclient: _finish_auth 0 2024-09-16T03:56:21.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.566+0000 7f3041ffb640 10 monclient: _check_auth_tickets 2024-09-16T03:56:21.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.566+0000 7f3041ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:56:21.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.566+0000 7f3041ffb640 1 -- 172.21.15.42:0/1243028877 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2177413734 0 0) 0x7f3038001680 con 0x7f304406dac0 2024-09-16T03:56:21.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.566+0000 7f30495b1640 5 monclient: authenticate success, global_id 824748 2024-09-16T03:56:21.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.566+0000 7f30495b1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:56:21.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.566+0000 7f3041ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:56:21.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.566+0000 7f3041ffb640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T03:56:21.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.566+0000 7f3041ffb640 10 monclient: dump: 2024-09-16T03:56:21.583 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:56:21.583 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:56:21.583 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:21.583 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:21.583 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:56:21.583 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:56:21.583 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:56:21.583 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:56:21.583 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:56:21.583 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:56:21.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.566+0000 7f3041ffb640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] but i'm connected to v1:172.21.15.42:6789/0, reconnecting 2024-09-16T03:56:21.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.566+0000 7f3041ffb640 10 monclient: _reopen_session rank -1 2024-09-16T03:56:21.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.566+0000 7f3041ffb640 1 -- 172.21.15.42:0/1243028877 >> v1:172.21.15.42:6789/0 conn(0x7f304406dac0 legacy=0x7f304413a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:56:21.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.566+0000 7f3041ffb640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:56:21.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.566+0000 7f3041ffb640 1 --2- 172.21.15.42:0/1243028877 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f302c0039c0 0x7f302c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:56:21.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.566+0000 7f3041ffb640 10 monclient(hunting): picked mon.a con 0x7f302c0039c0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:56:21.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.566+0000 7f3041ffb640 1 --2- 172.21.15.42:0/1243028877 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f302c004810 0x7f302c004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:56:21.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.566+0000 7f3041ffb640 10 monclient(hunting): picked mon.c con 0x7f302c004810 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:56:21.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.566+0000 7f3041ffb640 1 --2- 172.21.15.42:0/1243028877 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f302c0052f0 0x7f302c00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:56:21.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.566+0000 7f3041ffb640 10 monclient(hunting): picked mon.b con 0x7f302c0052f0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:56:21.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.566+0000 7f3041ffb640 10 monclient(hunting): start opening mon connection 2024-09-16T03:56:21.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.566+0000 7f3041ffb640 1 -- 172.21.15.42:0/1243028877 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f304409f8f0 con 0x7f302c0039c0 2024-09-16T03:56:21.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.566+0000 7f3041ffb640 10 monclient(hunting): start opening mon connection 2024-09-16T03:56:21.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.566+0000 7f3041ffb640 1 -- 172.21.15.42:0/1243028877 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f302c00c1f0 con 0x7f302c0052f0 2024-09-16T03:56:21.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.566+0000 7f3041ffb640 10 monclient(hunting): start opening mon connection 2024-09-16T03:56:21.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.566+0000 7f3041ffb640 1 -- 172.21.15.42:0/1243028877 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f302c00c3f0 con 0x7f302c004810 2024-09-16T03:56:21.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.566+0000 7f3041ffb640 10 monclient(hunting): _renew_subs 2024-09-16T03:56:21.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.566+0000 7f3041ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:56:21.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.566+0000 7f30495b1640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-16T03:56:21.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.566+0000 7f3042ffd640 1 --2- 172.21.15.42:0/1243028877 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f302c0039c0 0x7f302c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:56:21.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.566+0000 7f30427fc640 1 --2- 172.21.15.42:0/1243028877 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f302c004810 0x7f302c004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:56:21.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.566+0000 7f30437fe640 1 -- 172.21.15.42:0/1243028877 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f302c0052f0 msgr2=0x7f302c00b9d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T03:56:21.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.566+0000 7f30427fc640 10 monclient(hunting): get_auth_request con 0x7f302c004810 auth_method 0 2024-09-16T03:56:21.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.566+0000 7f30427fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:56:21.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.566+0000 7f30427fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:56:21.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.566+0000 7f30427fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:56:21.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.566+0000 7f30437fe640 1 --2- 172.21.15.42:0/1243028877 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f302c0052f0 0x7f302c00b9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:56:21.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.566+0000 7f3042ffd640 10 monclient(hunting): get_auth_request con 0x7f302c0039c0 auth_method 0 2024-09-16T03:56:21.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.566+0000 7f3042ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:56:21.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.566+0000 7f3042ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:56:21.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.566+0000 7f3042ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:56:21.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.566+0000 7f30427fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:56:21.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.566+0000 7f30427fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:56:21.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.566+0000 7f30427fc640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T03:56:21.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.566+0000 7f3042ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:56:21.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.566+0000 7f3042ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:56:21.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.566+0000 7f3042ffd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T03:56:21.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.566+0000 7f3042ffd640 10 monclient(hunting): handle_auth_done global_id 824748 payload 293 2024-09-16T03:56:21.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.566+0000 7f3042ffd640 1 -- 172.21.15.42:0/1243028877 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f302c004810 msgr2=0x7f302c004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:56:21.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.566+0000 7f3042ffd640 1 --2- 172.21.15.42:0/1243028877 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f302c004810 0x7f302c004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:56:21.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.566+0000 7f3042ffd640 1 -- 172.21.15.42:0/1243028877 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f302c0052f0 msgr2=0x7f302c00b9d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:56:21.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.566+0000 7f3042ffd640 1 --2- 172.21.15.42:0/1243028877 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f302c0052f0 0x7f302c00b9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:56:21.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.566+0000 7f3042ffd640 10 monclient: _finish_hunting 0 2024-09-16T03:56:21.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.566+0000 7f3042ffd640 1 monclient: found mon.a 2024-09-16T03:56:21.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.566+0000 7f3042ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:56:21.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.566+0000 7f3042ffd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:56:21.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.566+0000 7f30437fe640 1 -- 172.21.15.42:0/1243028877 reap_dead start 2024-09-16T03:56:21.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.566+0000 7f3042ffd640 1 -- 172.21.15.42:0/1243028877 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f302c003530 con 0x7f302c0039c0 2024-09-16T03:56:21.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.566+0000 7f3042ffd640 10 monclient: _finish_auth 0 2024-09-16T03:56:21.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.566+0000 7f3042ffd640 10 monclient: _check_auth_tickets 2024-09-16T03:56:21.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.566+0000 7f3042ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:56:21.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.566+0000 7f3042ffd640 1 --2- 172.21.15.42:0/1243028877 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f302c0039c0 0x7f302c003dc0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f3034006fd0 tx=0x7f303401c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df38560e5a55c126 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:56:21.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.566+0000 7f3041ffb640 1 -- 172.21.15.42:0/1243028877 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3034009520 con 0x7f302c0039c0 2024-09-16T03:56:21.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.566+0000 7f3041ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:56:21.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.566+0000 7f3041ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:56:21.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.566+0000 7f3041ffb640 10 monclient: dump: 2024-09-16T03:56:21.587 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:56:21.588 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:56:21.588 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:21.588 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:21.588 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:56:21.588 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:56:21.588 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:56:21.588 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:56:21.588 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:56:21.588 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:56:21.588 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.566+0000 7f30495b1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:56:21.588 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.566+0000 7f3041ffb640 1 -- 172.21.15.42:0/1243028877 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3034003bc0 con 0x7f302c0039c0 2024-09-16T03:56:21.588 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.566+0000 7f3041ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T03:56:21.588 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.566+0000 7f30495b1640 10 monclient: get_monmap_and_config success 2024-09-16T03:56:21.588 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.566+0000 7f30495b1640 4 set_mon_vals no callback set 2024-09-16T03:56:21.589 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.566+0000 7f3041ffb640 1 -- 172.21.15.42:0/1243028877 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3034005070 con 0x7f302c0039c0 2024-09-16T03:56:21.589 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.566+0000 7f3041ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:56:21.589 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.566+0000 7f3041ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:56:21.589 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.566+0000 7f3041ffb640 10 monclient: dump: 2024-09-16T03:56:21.589 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:56:21.589 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:56:21.589 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:21.589 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:21.589 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:56:21.589 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:56:21.589 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:56:21.589 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:56:21.589 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:56:21.589 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:56:21.590 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.566+0000 7f30495b1640 10 monclient: shutdown 2024-09-16T03:56:21.590 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.566+0000 7f30495b1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:56:21.590 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.566+0000 7f30495b1640 1 -- 172.21.15.42:0/1243028877 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f302c0039c0 msgr2=0x7f302c003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:56:21.590 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.566+0000 7f30495b1640 1 --2- 172.21.15.42:0/1243028877 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f302c0039c0 0x7f302c003dc0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f3034006fd0 tx=0x7f303401c3e0 comp rx=0 tx=0).stop 2024-09-16T03:56:21.590 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.566+0000 7f30495b1640 1 -- 172.21.15.42:0/1243028877 shutdown_connections 2024-09-16T03:56:21.590 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.566+0000 7f30495b1640 1 --2- 172.21.15.42:0/1243028877 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f302c0039c0 0x7f302c003dc0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:56:21.590 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.566+0000 7f30495b1640 1 -- 172.21.15.42:0/1243028877 >> 172.21.15.42:0/1243028877 conn(0x7f30440a2420 msgr2=0x7f30440a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:56:21.590 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.566+0000 7f30495b1640 1 -- 172.21.15.42:0/1243028877 shutdown_connections 2024-09-16T03:56:21.590 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.566+0000 7f30495b1640 1 -- 172.21.15.42:0/1243028877 wait complete. 2024-09-16T03:56:21.590 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.570+0000 7f30495b1640 10 monclient: build_initial_monmap 2024-09-16T03:56:21.590 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.570+0000 7f30495b1640 10 monclient: monmap: 2024-09-16T03:56:21.590 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:56:21.590 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:56:21.590 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:56:21.591 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:56:21.591 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:56:21.591 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:56:21.591 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:56:21.591 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:56:21.591 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:56:21.591 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:56:21.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.570+0000 7f30495b1640 1 librados: starting msgr at 2024-09-16T03:56:21.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.570+0000 7f30495b1640 1 librados: starting objecter 2024-09-16T03:56:21.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.570+0000 7f30495b1640 1 Processor -- start 2024-09-16T03:56:21.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.570+0000 7f30495b1640 1 -- start start 2024-09-16T03:56:21.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.570+0000 7f30495b1640 1 librados: setting wanted keys 2024-09-16T03:56:21.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.570+0000 7f30495b1640 1 librados: calling monclient init 2024-09-16T03:56:21.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.570+0000 7f30495b1640 10 monclient: init 2024-09-16T03:56:21.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.570+0000 7f30495b1640 10 monclient: _reopen_session rank -1 2024-09-16T03:56:21.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.570+0000 7f30495b1640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:56:21.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.570+0000 7f30495b1640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f302c0039c0 0x7f30441cf430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:56:21.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.570+0000 7f30495b1640 10 monclient(hunting): picked mon.noname-c con 0x7f302c0039c0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:56:21.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.570+0000 7f30495b1640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f30441cf970 0x7f30441240e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:56:21.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.570+0000 7f30495b1640 10 monclient(hunting): picked mon.noname-b con 0x7f30441cf970 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:56:21.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.570+0000 7f30495b1640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f30441cfd90 0x7f3044124620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:56:21.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.570+0000 7f30495b1640 10 monclient(hunting): picked mon.noname-a con 0x7f30441cfd90 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:56:21.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.570+0000 7f30495b1640 10 monclient(hunting): start opening mon connection 2024-09-16T03:56:21.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.570+0000 7f30495b1640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f3044135910 con 0x7f30441cfd90 2024-09-16T03:56:21.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.570+0000 7f30495b1640 10 monclient(hunting): start opening mon connection 2024-09-16T03:56:21.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.570+0000 7f30495b1640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f304409ef80 con 0x7f30441cf970 2024-09-16T03:56:21.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.570+0000 7f30495b1640 10 monclient(hunting): start opening mon connection 2024-09-16T03:56:21.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.570+0000 7f30495b1640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f3044135a90 con 0x7f302c0039c0 2024-09-16T03:56:21.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.570+0000 7f30495b1640 10 monclient(hunting): _renew_subs 2024-09-16T03:56:21.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.570+0000 7f30495b1640 10 monclient(hunting): authenticate will time out at 1828.541177s 2024-09-16T03:56:21.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.570+0000 7f3042ffd640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f302c0039c0 0x7f30441cf430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:56:21.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.570+0000 7f30437fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f30441cfd90 0x7f3044124620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:56:21.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.570+0000 7f3042ffd640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f302c0039c0 0x7f30441cf430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:40918/0 (socket says 172.21.15.42:40918) 2024-09-16T03:56:21.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.570+0000 7f3042ffd640 1 -- 172.21.15.42:0/1796368712 learned_addr learned my addr 172.21.15.42:0/1796368712 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:56:21.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.570+0000 7f3042ffd640 10 monclient(hunting): get_auth_request con 0x7f302c0039c0 auth_method 0 2024-09-16T03:56:21.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.570+0000 7f3042ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:56:21.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.570+0000 7f3042ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:56:21.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.570+0000 7f3042ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:56:21.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.570+0000 7f30437fe640 10 monclient(hunting): get_auth_request con 0x7f30441cfd90 auth_method 0 2024-09-16T03:56:21.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.570+0000 7f30437fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:56:21.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.570+0000 7f30437fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:56:21.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.570+0000 7f30437fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:56:21.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.570+0000 7f30427fc640 1 -- 172.21.15.42:0/1796368712 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f30441cf970 msgr2=0x7f30441240e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T03:56:21.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.570+0000 7f30427fc640 1 --2- 172.21.15.42:0/1796368712 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f30441cf970 0x7f30441240e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:56:21.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.570+0000 7f3042ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:56:21.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.570+0000 7f3042ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:56:21.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.570+0000 7f3042ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:56:21.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.570+0000 7f30437fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:56:21.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.570+0000 7f30437fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:56:21.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.570+0000 7f30437fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:56:21.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.570+0000 7f3042ffd640 10 monclient(hunting): handle_auth_done global_id 854333 payload 824 2024-09-16T03:56:21.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.570+0000 7f3042ffd640 1 -- 172.21.15.42:0/1796368712 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f30441cf970 msgr2=0x7f30441240e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:56:21.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.570+0000 7f3042ffd640 1 --2- 172.21.15.42:0/1796368712 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f30441cf970 0x7f30441240e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:56:21.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.570+0000 7f3042ffd640 1 -- 172.21.15.42:0/1796368712 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f30441cfd90 msgr2=0x7f3044124620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:56:21.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.570+0000 7f3042ffd640 1 --2- 172.21.15.42:0/1796368712 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f30441cfd90 0x7f3044124620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:56:21.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.570+0000 7f3042ffd640 10 monclient: _finish_hunting 0 2024-09-16T03:56:21.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.570+0000 7f3042ffd640 1 monclient: found mon.noname-c 2024-09-16T03:56:21.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.570+0000 7f3042ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:56:21.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.570+0000 7f3042ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:56:21.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.570+0000 7f3042ffd640 1 -- 172.21.15.42:0/1796368712 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3044128990 con 0x7f302c0039c0 2024-09-16T03:56:21.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.570+0000 7f30437fe640 1 --2- 172.21.15.42:0/1796368712 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f30441cfd90 0x7f3044124620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:56:21.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.570+0000 7f3042ffd640 1 --2- 172.21.15.42:0/1796368712 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f302c0039c0 0x7f30441cf430 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f303401c420 tx=0x7f302c005840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:56:21.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.570+0000 7f302b7fe640 10 client.?.objecter ms_handle_connect 0x7f302c0039c0 2024-09-16T03:56:21.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.570+0000 7f302b7fe640 10 client.?.objecter resend_mon_ops 2024-09-16T03:56:21.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.570+0000 7f302b7fe640 1 -- 172.21.15.42:0/1796368712 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30340bb070 con 0x7f302c0039c0 2024-09-16T03:56:21.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.570+0000 7f302b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:56:21.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.570+0000 7f302b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:56:21.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.570+0000 7f302b7fe640 10 monclient: dump: 2024-09-16T03:56:21.595 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:56:21.596 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:56:21.596 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:21.596 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:21.596 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:56:21.596 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:56:21.596 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:56:21.596 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:56:21.596 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:56:21.596 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:56:21.596 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.570+0000 7f302b7fe640 10 monclient: _finish_auth 0 2024-09-16T03:56:21.596 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.570+0000 7f302b7fe640 10 monclient: _check_auth_tickets 2024-09-16T03:56:21.596 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.570+0000 7f302b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:56:21.596 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.570+0000 7f302b7fe640 1 -- 172.21.15.42:0/1796368712 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3034009500 con 0x7f302c0039c0 2024-09-16T03:56:21.597 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.570+0000 7f30495b1640 5 monclient: authenticate success, global_id 854333 2024-09-16T03:56:21.597 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.570+0000 7f30495b1640 10 monclient: _renew_subs 2024-09-16T03:56:21.597 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.570+0000 7f30495b1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:56:21.597 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.570+0000 7f30495b1640 1 -- 172.21.15.42:0/1796368712 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3044127cd0 con 0x7f302c0039c0 2024-09-16T03:56:21.597 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.570+0000 7f302b7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:56:21.597 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.570+0000 7f302b7fe640 1 -- 172.21.15.42:0/1796368712 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3034005650 con 0x7f302c0039c0 2024-09-16T03:56:21.597 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.570+0000 7f302b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:56:21.597 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.570+0000 7f302b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:56:21.597 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.570+0000 7f302b7fe640 10 monclient: dump: 2024-09-16T03:56:21.597 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:56:21.597 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:56:21.597 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:21.597 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:21.597 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:56:21.598 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:56:21.598 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:56:21.598 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:56:21.598 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:56:21.598 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:56:21.598 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.570+0000 7f3040ff9640 4 set_mon_vals no callback set 2024-09-16T03:56:21.598 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.570+0000 7f30495b1640 10 client.854333.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:56:21.598 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.570+0000 7f30495b1640 10 monclient: _renew_subs 2024-09-16T03:56:21.598 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.570+0000 7f30495b1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:56:21.598 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.570+0000 7f30495b1640 1 -- 172.21.15.42:0/1796368712 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f304409d3e0 con 0x7f302c0039c0 2024-09-16T03:56:21.598 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.570+0000 7f30495b1640 1 librados: init done 2024-09-16T03:56:21.598 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.570+0000 7f30495b1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:56:21.598 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.570+0000 7f30495b1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:56:21.598 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.570+0000 7f30495b1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:56:21.598 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.570+0000 7f30495b1640 1 -- 172.21.15.42:0/1796368712 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3044135e20 con 0x7f302c0039c0 2024-09-16T03:56:21.599 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.570+0000 7f302b7fe640 1 -- 172.21.15.42:0/1796368712 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f30340be070 con 0x7f302c0039c0 2024-09-16T03:56:21.599 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.570+0000 7f302b7fe640 1 --2- 172.21.15.42:0/1796368712 >> v2:172.21.15.154:6812/254845342 conn(0x7f30300377c0 0x7f3030039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:56:21.599 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.570+0000 7f302b7fe640 1 -- 172.21.15.42:0/1796368712 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(442..442 src has 330..442) ==== 5741+0+0 (secure 0 0 0) 0x7f30340e2760 con 0x7f302c0039c0 2024-09-16T03:56:21.599 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.570+0000 7f302b7fe640 10 client.854333.objecter ms_dispatch 0x7f3044000ba0 osd_map(442..442 src has 330..442) 2024-09-16T03:56:21.599 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.570+0000 7f302b7fe640 3 client.854333.objecter handle_osd_map got epochs [442,442] > 0 2024-09-16T03:56:21.599 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.570+0000 7f302b7fe640 3 client.854333.objecter handle_osd_map decoding full epoch 442 2024-09-16T03:56:21.599 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.570+0000 7f302b7fe640 20 client.854333.objecter dump_active .. 0 homeless 2024-09-16T03:56:21.599 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.570+0000 7f30427fc640 1 --2- 172.21.15.42:0/1796368712 >> v2:172.21.15.154:6812/254845342 conn(0x7f30300377c0 0x7f3030039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:56:21.599 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.570+0000 7f30427fc640 10 monclient: get_auth_request con 0x7f30300377c0 auth_method 0 2024-09-16T03:56:21.599 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.574+0000 7f30427fc640 1 --2- 172.21.15.42:0/1796368712 >> v2:172.21.15.154:6812/254845342 conn(0x7f30300377c0 0x7f3030039c80 secure :-1 s=READY pgs=3658 cs=0 l=1 rev1=1 crypto rx=0x7f3038001060 tx=0x7f3038007900 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:56:21.599 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.574+0000 7f302b7fe640 10 client.854333.objecter ms_handle_connect 0x7f30300377c0 2024-09-16T03:56:21.600 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.578+0000 7f302b7fe640 1 -- 172.21.15.42:0/1796368712 <== mon.2 v2:172.21.15.42:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f3044128990 con 0x7f302c0039c0 2024-09-16T03:56:21.600 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.578+0000 7f302b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:56:21.600 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.578+0000 7f302b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:56:21.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.802+0000 7f30495b1640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:56:21.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.802+0000 7f30495b1640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:56:21.806 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.802+0000 7f30495b1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:56:21.806 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.802+0000 7f30495b1640 1 -- 172.21.15.42:0/1796368712 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f30441265a0 con 0x7f302c0039c0 2024-09-16T03:56:21.806 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.802+0000 7f302b7fe640 1 -- 172.21.15.42:0/1796368712 <== mon.2 v2:172.21.15.42:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7f30340160f0 con 0x7f302c0039c0 2024-09-16T03:56:21.806 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.802+0000 7f302b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:56:21.806 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.802+0000 7f302b7fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:56:21.806 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":128,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":4,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:56:21.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.802+0000 7f30495b1640 10 librados: watch_flush enter 2024-09-16T03:56:21.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.802+0000 7f30495b1640 10 librados: watch_flush exit 2024-09-16T03:56:21.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.802+0000 7f30495b1640 20 client.854333.objecter shutdown clearing up homeless session... 2024-09-16T03:56:21.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.802+0000 7f30495b1640 10 client.854333.objecter successfully canceled tick 2024-09-16T03:56:21.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.802+0000 7f30495b1640 1 -- 172.21.15.42:0/1796368712 >> v2:172.21.15.154:6812/254845342 conn(0x7f30300377c0 msgr2=0x7f3030039c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:56:21.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.802+0000 7f30495b1640 1 --2- 172.21.15.42:0/1796368712 >> v2:172.21.15.154:6812/254845342 conn(0x7f30300377c0 0x7f3030039c80 secure :-1 s=READY pgs=3658 cs=0 l=1 rev1=1 crypto rx=0x7f3038001060 tx=0x7f3038007900 comp rx=0 tx=0).stop 2024-09-16T03:56:21.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.802+0000 7f30495b1640 10 monclient: shutdown 2024-09-16T03:56:21.810 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.802+0000 7f30495b1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:56:21.810 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.802+0000 7f30495b1640 1 -- 172.21.15.42:0/1796368712 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f302c0039c0 msgr2=0x7f30441cf430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:56:21.810 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.802+0000 7f30495b1640 1 --2- 172.21.15.42:0/1796368712 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f302c0039c0 0x7f30441cf430 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f303401c420 tx=0x7f302c005840 comp rx=0 tx=0).stop 2024-09-16T03:56:21.810 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.802+0000 7f30495b1640 1 -- 172.21.15.42:0/1796368712 shutdown_connections 2024-09-16T03:56:21.810 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.802+0000 7f30495b1640 1 --2- 172.21.15.42:0/1796368712 >> v2:172.21.15.154:6812/254845342 conn(0x7f30300377c0 0x7f3030039c80 unknown :-1 s=CLOSED pgs=3658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:56:21.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.802+0000 7f30495b1640 1 --2- 172.21.15.42:0/1796368712 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f30441cfd90 0x7f3044124620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:56:21.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.802+0000 7f30495b1640 1 --2- 172.21.15.42:0/1796368712 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f30441cf970 0x7f30441240e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:56:21.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.802+0000 7f30495b1640 1 --2- 172.21.15.42:0/1796368712 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f302c0039c0 0x7f30441cf430 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:56:21.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.802+0000 7f30495b1640 1 -- 172.21.15.42:0/1796368712 >> 172.21.15.42:0/1796368712 conn(0x7f30440a2420 msgr2=0x7f30440a4dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:56:21.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.802+0000 7f30495b1640 1 -- 172.21.15.42:0/1796368712 shutdown_connections 2024-09-16T03:56:21.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.802+0000 7f30495b1640 1 -- 172.21.15.42:0/1796368712 wait complete. 2024-09-16T03:56:21.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:21.802+0000 7f30495b1640 1 librados: shutdown 2024-09-16T03:56:24.835 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:56:24.968 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.962+0000 7fc3a51cd640 10 monclient: get_monmap_and_config 2024-09-16T03:56:24.968 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.962+0000 7fc3a51cd640 10 monclient: build_initial_monmap 2024-09-16T03:56:24.968 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.962+0000 7fc3a51cd640 10 monclient: monmap: 2024-09-16T03:56:24.968 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:56:24.968 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:56:24.968 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:56:24.966385+0000 2024-09-16T03:56:24.968 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:56:24.966385+0000 2024-09-16T03:56:24.968 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:56:24.968 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:56:24.968 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:56:24.969 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:56:24.969 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:56:24.969 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:56:24.969 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:56:24.969 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:56:24.971 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.962+0000 7fc3a51cd640 1 Processor -- start 2024-09-16T03:56:24.971 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.962+0000 7fc3a51cd640 1 -- start start 2024-09-16T03:56:24.971 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.962+0000 7fc3a51cd640 10 monclient: init 2024-09-16T03:56:24.971 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.966+0000 7fc3a51cd640 10 monclient: _reopen_session rank -1 2024-09-16T03:56:24.971 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.966+0000 7fc3a51cd640 10 monclient: _add_conns ranks=[0,1,3,2,4] 2024-09-16T03:56:24.971 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.966+0000 7fc3a51cd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc3a00a39b0 addr v1:172.21.15.42:6789/0 2024-09-16T03:56:24.971 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.966+0000 7fc3a51cd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc3a00a3d90 addr v1:172.21.15.154:6789/0 2024-09-16T03:56:24.972 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.966+0000 7fc3a51cd640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fc3a013daa0 0x7fc3a013fe90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:56:24.972 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.966+0000 7fc3a51cd640 10 monclient(hunting): picked mon.noname-a con 0x7fc3a013daa0 addr v2:172.21.15.42:3300/0 2024-09-16T03:56:24.972 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.966+0000 7fc3a51cd640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc3a0134290 con 0x7fc3a00a39b0 2024-09-16T03:56:24.972 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.966+0000 7fc3a51cd640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc3a009c140 con 0x7fc3a00a3d90 2024-09-16T03:56:24.972 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.966+0000 7fc3a51cd640 10 monclient(hunting): start opening mon connection 2024-09-16T03:56:24.972 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.966+0000 7fc3a51cd640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7fc3a009c420 con 0x7fc3a013daa0 2024-09-16T03:56:24.972 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.966+0000 7fc3a51cd640 10 monclient(hunting): _renew_subs 2024-09-16T03:56:24.972 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.966+0000 7fc3a51cd640 10 monclient(hunting): authenticate will time out at 1831.936698s 2024-09-16T03:56:24.972 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.966+0000 7fc39ed9d640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fc3a00a39b0 0x7fc3a0133b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:37466/0 (socket says 172.21.15.42:37466) 2024-09-16T03:56:24.972 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.966+0000 7fc39ed9d640 1 -- 172.21.15.42:0/1631084180 learned_addr learned my addr 172.21.15.42:0/1631084180 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:56:24.972 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.966+0000 7fc39659c640 1 -- 172.21.15.42:0/1631084180 >> v1:172.21.15.154:6789/0 conn(0x7fc3a00a3d90 legacy=0x7fc3a013c380 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.154:6789/0 2024-09-16T03:56:24.972 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.966+0000 7fc39f59e640 1 --2- 172.21.15.42:0/1631084180 >> v2:172.21.15.42:3300/0 conn(0x7fc3a013daa0 0x7fc3a013fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:56:24.972 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.966+0000 7fc39f59e640 10 monclient(hunting): get_auth_request con 0x7fc3a013daa0 auth_method 0 2024-09-16T03:56:24.973 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.966+0000 7fc39f59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:56:24.973 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.966+0000 7fc39f59e640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:56:24.973 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.966+0000 7fc39f59e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:56:24.973 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.966+0000 7fc39f59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:56:24.973 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.966+0000 7fc39f59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:56:24.973 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.966+0000 7fc39f59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:56:24.973 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.966+0000 7fc39e59c640 1 -- 172.21.15.42:0/1631084180 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3675781629 0 0) 0x7fc3a0134290 con 0x7fc3a00a39b0 2024-09-16T03:56:24.973 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.966+0000 7fc39e59c640 10 monclient(hunting): _negotiate 2024-09-16T03:56:24.973 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.966+0000 7fc39e59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:56:24.973 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.966+0000 7fc39e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:56:24.973 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.966+0000 7fc39e59c640 10 monclient(hunting): my global_id is 824757 2024-09-16T03:56:24.973 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.966+0000 7fc39e59c640 1 -- 172.21.15.42:0/1631084180 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc38c003530 con 0x7fc3a00a39b0 2024-09-16T03:56:24.973 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.966+0000 7fc39f59e640 10 monclient(hunting): handle_auth_done global_id 824754 payload 274 2024-09-16T03:56:24.974 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.966+0000 7fc39f59e640 1 -- 172.21.15.42:0/1631084180 >> v1:172.21.15.154:6789/0 conn(0x7fc3a00a3d90 legacy=0x7fc3a013c380 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:56:24.974 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.966+0000 7fc39f59e640 1 -- 172.21.15.42:0/1631084180 >> v1:172.21.15.42:6789/0 conn(0x7fc3a00a39b0 legacy=0x7fc3a0133b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:56:24.974 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.966+0000 7fc39f59e640 10 monclient: _finish_hunting 0 2024-09-16T03:56:24.974 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.966+0000 7fc39f59e640 1 monclient: found mon.noname-a 2024-09-16T03:56:24.974 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.966+0000 7fc39f59e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:56:24.974 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.966+0000 7fc39f59e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:56:24.974 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.966+0000 7fc39f59e640 1 -- 172.21.15.42:0/1631084180 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3a006dd60 con 0x7fc3a013daa0 2024-09-16T03:56:24.980 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.966+0000 7fc39f59e640 1 --2- 172.21.15.42:0/1631084180 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc3a013daa0 0x7fc3a013fe90 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fc398009d90 tx=0x7fc39801c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce88cdee34efb523 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:56:24.980 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.966+0000 7fc39e59c640 1 -- 172.21.15.42:0/1631084180 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3980a66c0 con 0x7fc3a013daa0 2024-09-16T03:56:24.980 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.966+0000 7fc39e59c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:56:24.980 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.966+0000 7fc39e59c640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:56:24.980 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.966+0000 7fc39e59c640 10 monclient: dump: 2024-09-16T03:56:24.980 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:56:24.980 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:56:24.980 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:24.980 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:24.981 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:56:24.981 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:56:24.981 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:56:24.981 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:56:24.981 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:56:24.981 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:56:24.981 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.966+0000 7fc39e59c640 10 monclient: _finish_auth 0 2024-09-16T03:56:24.981 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.966+0000 7fc39e59c640 10 monclient: _check_auth_tickets 2024-09-16T03:56:24.981 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.966+0000 7fc39e59c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:56:24.981 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.966+0000 7fc3a51cd640 5 monclient: authenticate success, global_id 824754 2024-09-16T03:56:24.981 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.966+0000 7fc3a51cd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:56:24.981 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.966+0000 7fc39e59c640 1 -- 172.21.15.42:0/1631084180 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc3980a6860 con 0x7fc3a013daa0 2024-09-16T03:56:24.982 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.966+0000 7fc39e59c640 10 monclient: handle_config config(0 keys) 2024-09-16T03:56:24.982 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.966+0000 7fc3a51cd640 10 monclient: get_monmap_and_config success 2024-09-16T03:56:24.982 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.966+0000 7fc3a51cd640 4 set_mon_vals no callback set 2024-09-16T03:56:24.982 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.966+0000 7fc39e59c640 1 -- 172.21.15.42:0/1631084180 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3980a6c10 con 0x7fc3a013daa0 2024-09-16T03:56:24.982 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.966+0000 7fc39e59c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:56:24.982 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.966+0000 7fc39e59c640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:56:24.982 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.966+0000 7fc39e59c640 10 monclient: dump: 2024-09-16T03:56:24.982 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:56:24.982 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:56:24.982 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:24.982 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:24.982 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:56:24.983 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:56:24.983 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:56:24.983 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:56:24.983 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:56:24.983 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:56:24.983 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.970+0000 7fc3a51cd640 10 monclient: shutdown 2024-09-16T03:56:24.983 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.970+0000 7fc3a51cd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:56:24.983 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.970+0000 7fc3a51cd640 1 -- 172.21.15.42:0/1631084180 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc3a013daa0 msgr2=0x7fc3a013fe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:56:24.983 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.970+0000 7fc3a51cd640 1 --2- 172.21.15.42:0/1631084180 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc3a013daa0 0x7fc3a013fe90 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fc398009d90 tx=0x7fc39801c7f0 comp rx=0 tx=0).stop 2024-09-16T03:56:24.983 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.970+0000 7fc3a51cd640 1 -- 172.21.15.42:0/1631084180 shutdown_connections 2024-09-16T03:56:24.983 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.970+0000 7fc3a51cd640 1 --2- 172.21.15.42:0/1631084180 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc3a013daa0 0x7fc3a013fe90 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:56:24.983 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.970+0000 7fc3a51cd640 1 -- 172.21.15.42:0/1631084180 >> 172.21.15.42:0/1631084180 conn(0x7fc3a00a9fb0 msgr2=0x7fc3a00aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:56:24.983 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.970+0000 7fc3a51cd640 1 -- 172.21.15.42:0/1631084180 shutdown_connections 2024-09-16T03:56:24.984 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.970+0000 7fc3a51cd640 1 -- 172.21.15.42:0/1631084180 wait complete. 2024-09-16T03:56:24.984 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.970+0000 7fc3a51cd640 10 monclient: build_initial_monmap 2024-09-16T03:56:24.984 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.970+0000 7fc3a51cd640 10 monclient: monmap: 2024-09-16T03:56:24.984 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:56:24.984 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:56:24.984 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:56:24.984 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:56:24.984 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:56:24.984 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:56:24.984 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:56:24.984 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:56:24.984 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:56:24.984 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:56:24.984 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.970+0000 7fc3a51cd640 1 librados: starting msgr at 2024-09-16T03:56:24.985 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.970+0000 7fc3a51cd640 1 librados: starting objecter 2024-09-16T03:56:24.985 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.970+0000 7fc3a51cd640 1 Processor -- start 2024-09-16T03:56:24.985 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.970+0000 7fc3a51cd640 1 -- start start 2024-09-16T03:56:24.985 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.970+0000 7fc3a51cd640 1 librados: setting wanted keys 2024-09-16T03:56:24.985 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.970+0000 7fc3a51cd640 1 librados: calling monclient init 2024-09-16T03:56:24.986 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.970+0000 7fc3a51cd640 10 monclient: init 2024-09-16T03:56:24.986 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.970+0000 7fc3a51cd640 10 monclient: _reopen_session rank -1 2024-09-16T03:56:24.986 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.970+0000 7fc3a51cd640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:56:24.986 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.970+0000 7fc3a51cd640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc3a00a39b0 0x7fc3a0130560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:56:24.986 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.970+0000 7fc3a51cd640 10 monclient(hunting): picked mon.noname-c con 0x7fc3a00a39b0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:56:24.986 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.970+0000 7fc3a51cd640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc3a00a3d90 0x7fc3a0130aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:56:24.986 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.970+0000 7fc3a51cd640 10 monclient(hunting): picked mon.noname-a con 0x7fc3a00a3d90 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:56:24.986 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.970+0000 7fc3a51cd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc3a013daa0 0x7fc3a013a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:56:24.986 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.970+0000 7fc3a51cd640 10 monclient(hunting): picked mon.noname-b con 0x7fc3a013daa0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:56:24.986 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.970+0000 7fc3a51cd640 10 monclient(hunting): start opening mon connection 2024-09-16T03:56:24.986 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.970+0000 7fc3a51cd640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fc3a0142f80 con 0x7fc3a00a3d90 2024-09-16T03:56:24.987 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.970+0000 7fc3a51cd640 10 monclient(hunting): start opening mon connection 2024-09-16T03:56:24.987 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.970+0000 7fc3a51cd640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc3a0142e00 con 0x7fc3a013daa0 2024-09-16T03:56:24.987 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.970+0000 7fc3a51cd640 10 monclient(hunting): start opening mon connection 2024-09-16T03:56:24.987 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.970+0000 7fc3a51cd640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fc3a0143100 con 0x7fc3a00a39b0 2024-09-16T03:56:24.987 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.970+0000 7fc3a51cd640 10 monclient(hunting): _renew_subs 2024-09-16T03:56:24.987 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.970+0000 7fc3a51cd640 10 monclient(hunting): authenticate will time out at 1831.941926s 2024-09-16T03:56:24.987 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.970+0000 7fc39659c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc3a00a3d90 0x7fc3a0130aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:56:24.987 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.970+0000 7fc39ed9d640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc3a00a39b0 0x7fc3a0130560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:56:24.987 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.970+0000 7fc39f59e640 1 -- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc3a013daa0 msgr2=0x7fc3a013a010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T03:56:24.987 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.970+0000 7fc39659c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc3a00a3d90 0x7fc3a0130aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:53236/0 (socket says 172.21.15.42:53236) 2024-09-16T03:56:24.987 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.970+0000 7fc39ed9d640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc3a00a39b0 0x7fc3a0130560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:49984/0 (socket says 172.21.15.42:49984) 2024-09-16T03:56:24.987 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.970+0000 7fc39659c640 1 -- 172.21.15.42:0/3865617355 learned_addr learned my addr 172.21.15.42:0/3865617355 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:56:24.987 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.970+0000 7fc39f59e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc3a013daa0 0x7fc3a013a010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:56:24.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.970+0000 7fc39659c640 10 monclient(hunting): get_auth_request con 0x7fc3a00a3d90 auth_method 0 2024-09-16T03:56:24.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.970+0000 7fc39659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:56:24.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.970+0000 7fc39659c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:56:24.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.970+0000 7fc39659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:56:24.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.970+0000 7fc39ed9d640 10 monclient(hunting): get_auth_request con 0x7fc3a00a39b0 auth_method 0 2024-09-16T03:56:24.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.970+0000 7fc39ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:56:24.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.970+0000 7fc39ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:56:24.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.970+0000 7fc39ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:56:24.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.970+0000 7fc39ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:56:24.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.970+0000 7fc39ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:56:24.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.970+0000 7fc39ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:56:24.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.970+0000 7fc39659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:56:24.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.970+0000 7fc39659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:56:24.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.970+0000 7fc39659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:56:24.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.970+0000 7fc39ed9d640 10 monclient(hunting): handle_auth_done global_id 854336 payload 824 2024-09-16T03:56:24.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.970+0000 7fc39ed9d640 1 -- 172.21.15.42:0/3865617355 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc3a013daa0 msgr2=0x7fc3a013a010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:56:24.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.970+0000 7fc39ed9d640 1 --2- 172.21.15.42:0/3865617355 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc3a013daa0 0x7fc3a013a010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:56:24.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.970+0000 7fc39ed9d640 1 -- 172.21.15.42:0/3865617355 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc3a00a3d90 msgr2=0x7fc3a0130aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:56:24.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.970+0000 7fc39ed9d640 1 --2- 172.21.15.42:0/3865617355 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc3a00a3d90 0x7fc3a0130aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:56:24.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.970+0000 7fc39ed9d640 10 monclient: _finish_hunting 0 2024-09-16T03:56:24.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.970+0000 7fc39ed9d640 1 monclient: found mon.noname-c 2024-09-16T03:56:24.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.970+0000 7fc39ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:56:24.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.970+0000 7fc39ed9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:56:24.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.970+0000 7fc39ed9d640 1 -- 172.21.15.42:0/3865617355 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3a009d6f0 con 0x7fc3a00a39b0 2024-09-16T03:56:24.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.970+0000 7fc39659c640 1 --2- 172.21.15.42:0/3865617355 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc3a00a3d90 0x7fc3a0130aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:56:24.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.970+0000 7fc39ed9d640 1 --2- 172.21.15.42:0/3865617355 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc3a00a39b0 0x7fc3a0130560 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fc39000c390 tx=0x7fc390001e80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:56:24.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.970+0000 7fc397fff640 10 client.?.objecter ms_handle_connect 0x7fc3a00a39b0 2024-09-16T03:56:24.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.970+0000 7fc397fff640 10 client.?.objecter resend_mon_ops 2024-09-16T03:56:24.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.970+0000 7fc397fff640 1 -- 172.21.15.42:0/3865617355 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc390017070 con 0x7fc3a00a39b0 2024-09-16T03:56:24.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.970+0000 7fc397fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:56:24.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.970+0000 7fc397fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:56:24.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.970+0000 7fc397fff640 10 monclient: dump: 2024-09-16T03:56:24.990 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:56:24.990 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:56:24.990 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:24.990 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:24.990 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:56:24.990 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:56:24.990 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:56:24.990 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:56:24.991 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:56:24.991 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:56:24.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.970+0000 7fc397fff640 10 monclient: _finish_auth 0 2024-09-16T03:56:24.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.970+0000 7fc397fff640 10 monclient: _check_auth_tickets 2024-09-16T03:56:24.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.970+0000 7fc397fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:56:24.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.970+0000 7fc397fff640 1 -- 172.21.15.42:0/3865617355 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc390005cf0 con 0x7fc3a00a39b0 2024-09-16T03:56:24.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.970+0000 7fc3a51cd640 5 monclient: authenticate success, global_id 854336 2024-09-16T03:56:24.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.970+0000 7fc3a51cd640 10 monclient: _renew_subs 2024-09-16T03:56:24.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.970+0000 7fc3a51cd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:56:24.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.970+0000 7fc3a51cd640 1 -- 172.21.15.42:0/3865617355 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3a013d380 con 0x7fc3a00a39b0 2024-09-16T03:56:24.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.970+0000 7fc397fff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:56:24.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.970+0000 7fc397fff640 1 -- 172.21.15.42:0/3865617355 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3900072f0 con 0x7fc3a00a39b0 2024-09-16T03:56:24.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.970+0000 7fc397fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:56:24.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.970+0000 7fc397fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:56:24.992 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.970+0000 7fc397fff640 10 monclient: dump: 2024-09-16T03:56:24.992 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:56:24.992 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:56:24.992 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:24.992 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:24.992 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:56:24.992 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:56:24.992 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:56:24.992 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:56:24.992 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:56:24.992 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:56:24.992 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.970+0000 7fc39dd9b640 4 set_mon_vals no callback set 2024-09-16T03:56:24.992 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.970+0000 7fc3a51cd640 10 client.854336.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:56:24.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.970+0000 7fc3a51cd640 10 monclient: _renew_subs 2024-09-16T03:56:24.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.970+0000 7fc3a51cd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:56:24.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.970+0000 7fc3a51cd640 1 -- 172.21.15.42:0/3865617355 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3a006f680 con 0x7fc3a00a39b0 2024-09-16T03:56:24.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.970+0000 7fc3a51cd640 1 librados: init done 2024-09-16T03:56:24.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.970+0000 7fc3a51cd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:56:24.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.970+0000 7fc3a51cd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:56:24.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.970+0000 7fc3a51cd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:56:24.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.970+0000 7fc3a51cd640 1 -- 172.21.15.42:0/3865617355 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc3a0133b80 con 0x7fc3a00a39b0 2024-09-16T03:56:24.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.974+0000 7fc397fff640 1 -- 172.21.15.42:0/3865617355 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fc39001a070 con 0x7fc3a00a39b0 2024-09-16T03:56:24.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.974+0000 7fc397fff640 1 --2- 172.21.15.42:0/3865617355 >> v2:172.21.15.154:6812/254845342 conn(0x7fc3800377c0 0x7fc380039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:56:24.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.974+0000 7fc397fff640 1 -- 172.21.15.42:0/3865617355 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(442..442 src has 330..442) ==== 5741+0+0 (secure 0 0 0) 0x7fc3900481a0 con 0x7fc3a00a39b0 2024-09-16T03:56:24.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.974+0000 7fc397fff640 10 client.854336.objecter ms_dispatch 0x7fc3a0000ba0 osd_map(442..442 src has 330..442) 2024-09-16T03:56:24.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.974+0000 7fc397fff640 3 client.854336.objecter handle_osd_map got epochs [442,442] > 0 2024-09-16T03:56:24.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.974+0000 7fc397fff640 3 client.854336.objecter handle_osd_map decoding full epoch 442 2024-09-16T03:56:24.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.974+0000 7fc397fff640 20 client.854336.objecter dump_active .. 0 homeless 2024-09-16T03:56:24.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.974+0000 7fc39659c640 1 --2- 172.21.15.42:0/3865617355 >> v2:172.21.15.154:6812/254845342 conn(0x7fc3800377c0 0x7fc380039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:56:24.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.974+0000 7fc39659c640 10 monclient: get_auth_request con 0x7fc3800377c0 auth_method 0 2024-09-16T03:56:24.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.974+0000 7fc39659c640 1 --2- 172.21.15.42:0/3865617355 >> v2:172.21.15.154:6812/254845342 conn(0x7fc3800377c0 0x7fc380039c80 secure :-1 s=READY pgs=3659 cs=0 l=1 rev1=1 crypto rx=0x7fc3a013ce10 tx=0x7fc388003ba0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:56:24.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.974+0000 7fc397fff640 10 client.854336.objecter ms_handle_connect 0x7fc3800377c0 2024-09-16T03:56:24.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.978+0000 7fc397fff640 1 -- 172.21.15.42:0/3865617355 <== mon.2 v2:172.21.15.42:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7fc390019e10 con 0x7fc3a00a39b0 2024-09-16T03:56:24.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.978+0000 7fc397fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:56:24.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:24.978+0000 7fc397fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:56:25.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:25.198+0000 7fc3a51cd640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:56:25.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:25.198+0000 7fc3a51cd640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:56:25.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:25.198+0000 7fc3a51cd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:56:25.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:25.198+0000 7fc3a51cd640 1 -- 172.21.15.42:0/3865617355 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fc3a009d4d0 con 0x7fc3a00a39b0 2024-09-16T03:56:25.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:25.198+0000 7fc397fff640 1 -- 172.21.15.42:0/3865617355 <== mon.2 v2:172.21.15.42:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7fc390010b40 con 0x7fc3a00a39b0 2024-09-16T03:56:25.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:25.198+0000 7fc397fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:56:25.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:25.198+0000 7fc397fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:56:25.205 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":128,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:56:25.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:25.202+0000 7fc3a51cd640 10 librados: watch_flush enter 2024-09-16T03:56:25.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:25.202+0000 7fc3a51cd640 10 librados: watch_flush exit 2024-09-16T03:56:25.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:25.202+0000 7fc3a51cd640 20 client.854336.objecter shutdown clearing up homeless session... 2024-09-16T03:56:25.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:25.202+0000 7fc3a51cd640 10 client.854336.objecter successfully canceled tick 2024-09-16T03:56:25.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:25.202+0000 7fc3a51cd640 1 -- 172.21.15.42:0/3865617355 >> v2:172.21.15.154:6812/254845342 conn(0x7fc3800377c0 msgr2=0x7fc380039c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:56:25.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:25.202+0000 7fc3a51cd640 1 --2- 172.21.15.42:0/3865617355 >> v2:172.21.15.154:6812/254845342 conn(0x7fc3800377c0 0x7fc380039c80 secure :-1 s=READY pgs=3659 cs=0 l=1 rev1=1 crypto rx=0x7fc3a013ce10 tx=0x7fc388003ba0 comp rx=0 tx=0).stop 2024-09-16T03:56:25.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:25.202+0000 7fc3a51cd640 10 monclient: shutdown 2024-09-16T03:56:25.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:25.202+0000 7fc3a51cd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:56:25.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:25.202+0000 7fc3a51cd640 1 -- 172.21.15.42:0/3865617355 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc3a00a39b0 msgr2=0x7fc3a0130560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:56:25.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:25.202+0000 7fc3a51cd640 1 --2- 172.21.15.42:0/3865617355 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc3a00a39b0 0x7fc3a0130560 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fc39000c390 tx=0x7fc390001e80 comp rx=0 tx=0).stop 2024-09-16T03:56:25.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:25.202+0000 7fc3a51cd640 1 -- 172.21.15.42:0/3865617355 shutdown_connections 2024-09-16T03:56:25.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:25.202+0000 7fc3a51cd640 1 --2- 172.21.15.42:0/3865617355 >> v2:172.21.15.154:6812/254845342 conn(0x7fc3800377c0 0x7fc380039c80 unknown :-1 s=CLOSED pgs=3659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:56:25.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:25.202+0000 7fc3a51cd640 1 --2- 172.21.15.42:0/3865617355 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc3a013daa0 0x7fc3a013a010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:56:25.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:25.202+0000 7fc3a51cd640 1 --2- 172.21.15.42:0/3865617355 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc3a00a3d90 0x7fc3a0130aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:56:25.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:25.202+0000 7fc3a51cd640 1 --2- 172.21.15.42:0/3865617355 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc3a00a39b0 0x7fc3a0130560 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:56:25.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:25.202+0000 7fc3a51cd640 1 -- 172.21.15.42:0/3865617355 >> 172.21.15.42:0/3865617355 conn(0x7fc3a00a9fb0 msgr2=0x7fc3a0133020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:56:25.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:25.202+0000 7fc3a51cd640 1 -- 172.21.15.42:0/3865617355 shutdown_connections 2024-09-16T03:56:25.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:25.202+0000 7fc3a51cd640 1 -- 172.21.15.42:0/3865617355 wait complete. 2024-09-16T03:56:25.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:25.202+0000 7fc3a51cd640 1 librados: shutdown 2024-09-16T03:56:28.224 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:56:28.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.350+0000 7f983ddad640 10 monclient: get_monmap_and_config 2024-09-16T03:56:28.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.350+0000 7f983ddad640 10 monclient: build_initial_monmap 2024-09-16T03:56:28.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.350+0000 7f983ddad640 10 monclient: monmap: 2024-09-16T03:56:28.356 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:56:28.356 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:56:28.356 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:56:28.354298+0000 2024-09-16T03:56:28.356 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:56:28.354298+0000 2024-09-16T03:56:28.356 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:56:28.356 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:56:28.356 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:56:28.357 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:56:28.357 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:56:28.357 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:56:28.357 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:56:28.357 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:56:28.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.350+0000 7f983ddad640 1 Processor -- start 2024-09-16T03:56:28.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.350+0000 7f983ddad640 1 -- start start 2024-09-16T03:56:28.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.350+0000 7f983ddad640 10 monclient: init 2024-09-16T03:56:28.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.350+0000 7f983ddad640 10 monclient: _reopen_session rank -1 2024-09-16T03:56:28.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.350+0000 7f983ddad640 10 monclient: _add_conns ranks=[4,3,0,2,1] 2024-09-16T03:56:28.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.350+0000 7f983ddad640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f98380a1330 0x7f98380a1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:56:28.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.350+0000 7f983ddad640 10 monclient(hunting): picked mon.noname-c con 0x7f98380a1330 addr v2:172.21.15.154:3300/0 2024-09-16T03:56:28.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.350+0000 7f983ddad640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f983806dac0 0x7f98380a1c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:56:28.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.350+0000 7f983ddad640 10 monclient(hunting): picked mon.noname-a con 0x7f983806dac0 addr v2:172.21.15.42:3300/0 2024-09-16T03:56:28.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.350+0000 7f983ddad640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9838135100 addr v1:172.21.15.42:6789/0 2024-09-16T03:56:28.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.350+0000 7f983ddad640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f983809c7d0 con 0x7f9838135100 2024-09-16T03:56:28.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.350+0000 7f983ddad640 10 monclient(hunting): start opening mon connection 2024-09-16T03:56:28.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.350+0000 7f983ddad640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f983809c090 con 0x7f983806dac0 2024-09-16T03:56:28.360 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.350+0000 7f983ddad640 10 monclient(hunting): start opening mon connection 2024-09-16T03:56:28.360 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.350+0000 7f983ddad640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f983809f610 con 0x7f98380a1330 2024-09-16T03:56:28.360 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.350+0000 7f983ddad640 10 monclient(hunting): _renew_subs 2024-09-16T03:56:28.360 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.350+0000 7f983ddad640 10 monclient(hunting): authenticate will time out at 1835.323818s 2024-09-16T03:56:28.360 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.354+0000 7f98377fe640 1 -- >> v2:172.21.15.154:3300/0 conn(0x7f98380a1330 msgr2=0x7f98380a1710 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T03:56:28.360 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.354+0000 7f9837fff640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f9838135100 0x7f9838135550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:37480/0 (socket says 172.21.15.42:37480) 2024-09-16T03:56:28.360 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.354+0000 7f98377fe640 1 --2- 172.21.15.42:0/1536886567 >> v2:172.21.15.154:3300/0 conn(0x7f98380a1330 0x7f98380a1710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:56:28.360 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.354+0000 7f9837fff640 1 -- 172.21.15.42:0/1536886567 learned_addr learned my addr 172.21.15.42:0/1536886567 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:56:28.360 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.354+0000 7f9836ffd640 1 --2- 172.21.15.42:0/1536886567 >> v2:172.21.15.42:3300/0 conn(0x7f983806dac0 0x7f98380a1c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:56:28.360 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.354+0000 7f9836ffd640 10 monclient(hunting): get_auth_request con 0x7f983806dac0 auth_method 0 2024-09-16T03:56:28.361 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.354+0000 7f9836ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:56:28.361 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.354+0000 7f9836ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:56:28.361 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.354+0000 7f9836ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:56:28.361 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.354+0000 7f9836ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:56:28.361 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.354+0000 7f9836ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:56:28.361 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.354+0000 7f9836ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:56:28.361 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.354+0000 7f9836ffd640 10 monclient(hunting): handle_auth_done global_id 824763 payload 274 2024-09-16T03:56:28.361 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.354+0000 7f9836ffd640 1 -- 172.21.15.42:0/1536886567 >> v2:172.21.15.154:3300/0 conn(0x7f98380a1330 msgr2=0x7f98380a1710 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:56:28.361 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.354+0000 7f9836ffd640 1 --2- 172.21.15.42:0/1536886567 >> v2:172.21.15.154:3300/0 conn(0x7f98380a1330 0x7f98380a1710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:56:28.361 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.354+0000 7f9836ffd640 1 -- 172.21.15.42:0/1536886567 >> v1:172.21.15.42:6789/0 conn(0x7f9838135100 legacy=0x7f9838135550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:56:28.361 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.354+0000 7f9836ffd640 10 monclient: _finish_hunting 0 2024-09-16T03:56:28.361 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.354+0000 7f9836ffd640 1 monclient: found mon.noname-a 2024-09-16T03:56:28.361 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.354+0000 7f9836ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:56:28.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.354+0000 7f9836ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:56:28.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.354+0000 7f9836ffd640 1 -- 172.21.15.42:0/1536886567 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f983809f8f0 con 0x7f983806dac0 2024-09-16T03:56:28.367 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.354+0000 7f9836ffd640 1 --2- 172.21.15.42:0/1536886567 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f983806dac0 0x7f98380a1c50 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f983806add0 tx=0x7f982c01c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21e137000a43233 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:56:28.367 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.354+0000 7f98367fc640 1 -- 172.21.15.42:0/1536886567 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f982c0a7070 con 0x7f983806dac0 2024-09-16T03:56:28.367 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.354+0000 7f98367fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:56:28.367 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.354+0000 7f98367fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:56:28.367 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.354+0000 7f98367fc640 10 monclient: dump: 2024-09-16T03:56:28.367 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:56:28.367 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:56:28.367 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:28.367 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:28.368 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:56:28.368 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:56:28.368 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:56:28.368 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:56:28.368 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:56:28.368 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:56:28.368 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.354+0000 7f98367fc640 10 monclient: _finish_auth 0 2024-09-16T03:56:28.368 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.354+0000 7f98367fc640 10 monclient: _check_auth_tickets 2024-09-16T03:56:28.368 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.354+0000 7f98367fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:56:28.368 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.354+0000 7f983ddad640 5 monclient: authenticate success, global_id 824763 2024-09-16T03:56:28.368 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.354+0000 7f98367fc640 1 -- 172.21.15.42:0/1536886567 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f982c0b0070 con 0x7f983806dac0 2024-09-16T03:56:28.368 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.354+0000 7f983ddad640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:56:28.368 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.354+0000 7f98367fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:56:28.369 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.354+0000 7f983ddad640 10 monclient: get_monmap_and_config success 2024-09-16T03:56:28.369 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.354+0000 7f983ddad640 4 set_mon_vals no callback set 2024-09-16T03:56:28.369 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.354+0000 7f98367fc640 1 -- 172.21.15.42:0/1536886567 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f982c0ab890 con 0x7f983806dac0 2024-09-16T03:56:28.369 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.354+0000 7f98367fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:56:28.369 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.354+0000 7f98367fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:56:28.369 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.354+0000 7f98367fc640 10 monclient: dump: 2024-09-16T03:56:28.369 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:56:28.369 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:56:28.369 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:28.369 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:28.369 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:56:28.369 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:56:28.369 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:56:28.370 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:56:28.370 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:56:28.370 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:56:28.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.354+0000 7f983ddad640 10 monclient: shutdown 2024-09-16T03:56:28.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.354+0000 7f983ddad640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:56:28.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.354+0000 7f983ddad640 1 -- 172.21.15.42:0/1536886567 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f983806dac0 msgr2=0x7f98380a1c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:56:28.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.354+0000 7f983ddad640 1 --2- 172.21.15.42:0/1536886567 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f983806dac0 0x7f98380a1c50 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f983806add0 tx=0x7f982c01c450 comp rx=0 tx=0).stop 2024-09-16T03:56:28.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.354+0000 7f983ddad640 1 -- 172.21.15.42:0/1536886567 shutdown_connections 2024-09-16T03:56:28.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.354+0000 7f983ddad640 1 --2- 172.21.15.42:0/1536886567 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f983806dac0 0x7f98380a1c50 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:56:28.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.354+0000 7f983ddad640 1 --2- 172.21.15.42:0/1536886567 >> v2:172.21.15.154:3300/0 conn(0x7f98380a1330 0x7f98380a1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:56:28.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.354+0000 7f983ddad640 1 -- 172.21.15.42:0/1536886567 >> 172.21.15.42:0/1536886567 conn(0x7f98380a2420 msgr2=0x7f98380a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:56:28.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.354+0000 7f983ddad640 1 -- 172.21.15.42:0/1536886567 shutdown_connections 2024-09-16T03:56:28.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.354+0000 7f983ddad640 1 -- 172.21.15.42:0/1536886567 wait complete. 2024-09-16T03:56:28.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.358+0000 7f983ddad640 10 monclient: build_initial_monmap 2024-09-16T03:56:28.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.358+0000 7f983ddad640 10 monclient: monmap: 2024-09-16T03:56:28.371 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:56:28.371 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:56:28.371 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:56:28.371 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:56:28.371 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:56:28.371 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:56:28.371 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:56:28.371 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:56:28.371 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:56:28.371 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:56:28.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.358+0000 7f983ddad640 1 librados: starting msgr at 2024-09-16T03:56:28.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.358+0000 7f983ddad640 1 librados: starting objecter 2024-09-16T03:56:28.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.358+0000 7f983ddad640 1 Processor -- start 2024-09-16T03:56:28.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.358+0000 7f983ddad640 1 -- start start 2024-09-16T03:56:28.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.358+0000 7f983ddad640 1 librados: setting wanted keys 2024-09-16T03:56:28.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.358+0000 7f983ddad640 1 librados: calling monclient init 2024-09-16T03:56:28.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.358+0000 7f983ddad640 10 monclient: init 2024-09-16T03:56:28.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.358+0000 7f983ddad640 10 monclient: _reopen_session rank -1 2024-09-16T03:56:28.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.358+0000 7f983ddad640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T03:56:28.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.358+0000 7f983ddad640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f98380a1330 0x7f983813b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:56:28.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.358+0000 7f983ddad640 10 monclient(hunting): picked mon.noname-a con 0x7f98380a1330 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:56:28.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.358+0000 7f983ddad640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9838135100 0x7f9838134550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:56:28.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.358+0000 7f983ddad640 10 monclient(hunting): picked mon.noname-b con 0x7f9838135100 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:56:28.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.358+0000 7f983ddad640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9838134a90 0x7f983809cb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:56:28.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.358+0000 7f983ddad640 10 monclient(hunting): picked mon.noname-c con 0x7f9838134a90 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:56:28.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.358+0000 7f983ddad640 10 monclient(hunting): start opening mon connection 2024-09-16T03:56:28.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.358+0000 7f983ddad640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f983813fb80 con 0x7f98380a1330 2024-09-16T03:56:28.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.358+0000 7f983ddad640 10 monclient(hunting): start opening mon connection 2024-09-16T03:56:28.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.358+0000 7f983ddad640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f983809ef80 con 0x7f9838135100 2024-09-16T03:56:28.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.358+0000 7f983ddad640 10 monclient(hunting): start opening mon connection 2024-09-16T03:56:28.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.358+0000 7f983ddad640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f983813fd00 con 0x7f9838134a90 2024-09-16T03:56:28.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.358+0000 7f983ddad640 10 monclient(hunting): _renew_subs 2024-09-16T03:56:28.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.358+0000 7f983ddad640 10 monclient(hunting): authenticate will time out at 1835.328930s 2024-09-16T03:56:28.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.358+0000 7f98377fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f98380a1330 0x7f983813b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:56:28.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.358+0000 7f98377fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f98380a1330 0x7f983813b300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:53246/0 (socket says 172.21.15.42:53246) 2024-09-16T03:56:28.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.358+0000 7f98377fe640 1 -- 172.21.15.42:0/646765576 learned_addr learned my addr 172.21.15.42:0/646765576 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:56:28.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.358+0000 7f98377fe640 10 monclient(hunting): get_auth_request con 0x7f98380a1330 auth_method 0 2024-09-16T03:56:28.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.358+0000 7f98377fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:56:28.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.358+0000 7f98377fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:56:28.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.358+0000 7f98377fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:56:28.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.358+0000 7f9837fff640 1 --2- 172.21.15.42:0/646765576 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9838134a90 0x7f983809cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:56:28.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.358+0000 7f9836ffd640 1 -- 172.21.15.42:0/646765576 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9838135100 msgr2=0x7f9838134550 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T03:56:28.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.358+0000 7f9836ffd640 1 --2- 172.21.15.42:0/646765576 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9838135100 0x7f9838134550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:56:28.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.358+0000 7f9837fff640 10 monclient(hunting): get_auth_request con 0x7f9838134a90 auth_method 0 2024-09-16T03:56:28.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.358+0000 7f9837fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:56:28.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.358+0000 7f9837fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:56:28.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.358+0000 7f9837fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:56:28.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.358+0000 7f98377fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:56:28.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.358+0000 7f98377fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:56:28.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.358+0000 7f98377fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:56:28.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.358+0000 7f9837fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:56:28.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.358+0000 7f9837fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:56:28.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.358+0000 7f9837fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:56:28.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.358+0000 7f98377fe640 10 monclient(hunting): handle_auth_done global_id 824766 payload 824 2024-09-16T03:56:28.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.358+0000 7f98377fe640 1 -- 172.21.15.42:0/646765576 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9838134a90 msgr2=0x7f983809cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:56:28.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.358+0000 7f98377fe640 1 --2- 172.21.15.42:0/646765576 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9838134a90 0x7f983809cb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:56:28.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.358+0000 7f98377fe640 1 -- 172.21.15.42:0/646765576 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9838135100 msgr2=0x7f9838134550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:56:28.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.358+0000 7f98377fe640 1 --2- 172.21.15.42:0/646765576 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9838135100 0x7f9838134550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:56:28.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.358+0000 7f98377fe640 10 monclient: _finish_hunting 0 2024-09-16T03:56:28.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.358+0000 7f98377fe640 1 monclient: found mon.noname-a 2024-09-16T03:56:28.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.358+0000 7f98377fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:56:28.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.358+0000 7f98377fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:56:28.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.358+0000 7f98377fe640 1 -- 172.21.15.42:0/646765576 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98381cf590 con 0x7f98380a1330 2024-09-16T03:56:28.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.358+0000 7f98377fe640 1 --2- 172.21.15.42:0/646765576 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f98380a1330 0x7f983813b300 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f982800b530 tx=0x7f982800b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:56:28.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.358+0000 7f981ffff640 10 client.?.objecter ms_handle_connect 0x7f98380a1330 2024-09-16T03:56:28.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.358+0000 7f981ffff640 10 client.?.objecter resend_mon_ops 2024-09-16T03:56:28.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.358+0000 7f981ffff640 1 -- 172.21.15.42:0/646765576 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9828013020 con 0x7f98380a1330 2024-09-16T03:56:28.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.358+0000 7f981ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:56:28.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.358+0000 7f981ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:56:28.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.358+0000 7f981ffff640 10 monclient: dump: 2024-09-16T03:56:28.376 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:56:28.376 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:56:28.376 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:28.376 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:28.376 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:56:28.376 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:56:28.376 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:56:28.376 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:56:28.377 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:56:28.377 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:56:28.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.358+0000 7f981ffff640 10 monclient: _finish_auth 0 2024-09-16T03:56:28.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.358+0000 7f981ffff640 10 monclient: _check_auth_tickets 2024-09-16T03:56:28.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.358+0000 7f981ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:56:28.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.358+0000 7f981ffff640 1 -- 172.21.15.42:0/646765576 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9828003df0 con 0x7f98380a1330 2024-09-16T03:56:28.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.358+0000 7f983ddad640 5 monclient: authenticate success, global_id 824766 2024-09-16T03:56:28.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.358+0000 7f983ddad640 10 monclient: _renew_subs 2024-09-16T03:56:28.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.358+0000 7f983ddad640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:56:28.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.358+0000 7f983ddad640 1 -- 172.21.15.42:0/646765576 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98380a5110 con 0x7f98380a1330 2024-09-16T03:56:28.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.358+0000 7f981ffff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:56:28.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.358+0000 7f981ffff640 1 -- 172.21.15.42:0/646765576 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98280052b0 con 0x7f98380a1330 2024-09-16T03:56:28.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.358+0000 7f981ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:56:28.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.358+0000 7f981ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:56:28.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.358+0000 7f981ffff640 10 monclient: dump: 2024-09-16T03:56:28.378 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:56:28.378 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:56:28.378 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:28.378 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:28.378 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:56:28.378 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:56:28.378 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:56:28.378 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:56:28.378 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:56:28.378 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:56:28.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.358+0000 7f983ddad640 10 client.824766.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:56:28.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.358+0000 7f983ddad640 10 monclient: _renew_subs 2024-09-16T03:56:28.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.358+0000 7f983ddad640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:56:28.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.358+0000 7f983ddad640 1 -- 172.21.15.42:0/646765576 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9838071b80 con 0x7f98380a1330 2024-09-16T03:56:28.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.358+0000 7f98357fa640 4 set_mon_vals no callback set 2024-09-16T03:56:28.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.358+0000 7f983ddad640 1 librados: init done 2024-09-16T03:56:28.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.358+0000 7f981ffff640 1 -- 172.21.15.42:0/646765576 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f9828016020 con 0x7f98380a1330 2024-09-16T03:56:28.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.358+0000 7f983ddad640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:56:28.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.358+0000 7f983ddad640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:56:28.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.358+0000 7f983ddad640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:56:28.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.358+0000 7f983ddad640 1 -- 172.21.15.42:0/646765576 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9838073a30 con 0x7f98380a1330 2024-09-16T03:56:28.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.358+0000 7f981ffff640 1 --2- 172.21.15.42:0/646765576 >> v2:172.21.15.154:6812/254845342 conn(0x7f9824037820 0x7f9824039ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:56:28.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.358+0000 7f981ffff640 1 -- 172.21.15.42:0/646765576 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(442..442 src has 330..442) ==== 5741+0+0 (secure 0 0 0) 0x7f9828048aa0 con 0x7f98380a1330 2024-09-16T03:56:28.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.358+0000 7f981ffff640 10 client.824766.objecter ms_dispatch 0x7f9838000ba0 osd_map(442..442 src has 330..442) 2024-09-16T03:56:28.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.358+0000 7f981ffff640 3 client.824766.objecter handle_osd_map got epochs [442,442] > 0 2024-09-16T03:56:28.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.358+0000 7f981ffff640 3 client.824766.objecter handle_osd_map decoding full epoch 442 2024-09-16T03:56:28.380 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.358+0000 7f981ffff640 20 client.824766.objecter dump_active .. 0 homeless 2024-09-16T03:56:28.380 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.358+0000 7f9836ffd640 1 --2- 172.21.15.42:0/646765576 >> v2:172.21.15.154:6812/254845342 conn(0x7f9824037820 0x7f9824039ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:56:28.380 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.358+0000 7f9836ffd640 10 monclient: get_auth_request con 0x7f9824037820 auth_method 0 2024-09-16T03:56:28.380 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.362+0000 7f9836ffd640 1 --2- 172.21.15.42:0/646765576 >> v2:172.21.15.154:6812/254845342 conn(0x7f9824037820 0x7f9824039ce0 secure :-1 s=READY pgs=3660 cs=0 l=1 rev1=1 crypto rx=0x7f982c0af000 tx=0x7f982c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:56:28.380 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.362+0000 7f981ffff640 10 client.824766.objecter ms_handle_connect 0x7f9824037820 2024-09-16T03:56:28.380 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.362+0000 7f981ffff640 1 -- 172.21.15.42:0/646765576 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f982801d020 con 0x7f98380a1330 2024-09-16T03:56:28.380 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.362+0000 7f981ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:56:28.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.362+0000 7f981ffff640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:56:28.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.586+0000 7f983ddad640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:56:28.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.586+0000 7f983ddad640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:56:28.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.586+0000 7f983ddad640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:56:28.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.586+0000 7f983ddad640 1 -- 172.21.15.42:0/646765576 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f9838135550 con 0x7f98380a1330 2024-09-16T03:56:28.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.586+0000 7f981ffff640 1 -- 172.21.15.42:0/646765576 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1562 (secure 0 0 0) 0x7f98380a5110 con 0x7f98380a1330 2024-09-16T03:56:28.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.586+0000 7f981ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:56:28.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.586+0000 7f981ffff640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:56:28.594 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":128,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:56:28.596 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.590+0000 7f983ddad640 10 librados: watch_flush enter 2024-09-16T03:56:28.596 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.590+0000 7f983ddad640 10 librados: watch_flush exit 2024-09-16T03:56:28.596 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.590+0000 7f983ddad640 20 client.824766.objecter shutdown clearing up homeless session... 2024-09-16T03:56:28.596 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.590+0000 7f983ddad640 10 client.824766.objecter successfully canceled tick 2024-09-16T03:56:28.596 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.590+0000 7f983ddad640 1 -- 172.21.15.42:0/646765576 >> v2:172.21.15.154:6812/254845342 conn(0x7f9824037820 msgr2=0x7f9824039ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:56:28.596 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.590+0000 7f983ddad640 1 --2- 172.21.15.42:0/646765576 >> v2:172.21.15.154:6812/254845342 conn(0x7f9824037820 0x7f9824039ce0 secure :-1 s=READY pgs=3660 cs=0 l=1 rev1=1 crypto rx=0x7f982c0af000 tx=0x7f982c008040 comp rx=0 tx=0).stop 2024-09-16T03:56:28.597 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.590+0000 7f983ddad640 10 monclient: shutdown 2024-09-16T03:56:28.597 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.590+0000 7f983ddad640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:56:28.597 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.590+0000 7f983ddad640 1 -- 172.21.15.42:0/646765576 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f98380a1330 msgr2=0x7f983813b300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:56:28.597 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.590+0000 7f983ddad640 1 --2- 172.21.15.42:0/646765576 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f98380a1330 0x7f983813b300 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f982800b530 tx=0x7f982800b9f0 comp rx=0 tx=0).stop 2024-09-16T03:56:28.597 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.590+0000 7f983ddad640 1 -- 172.21.15.42:0/646765576 shutdown_connections 2024-09-16T03:56:28.597 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.590+0000 7f983ddad640 1 --2- 172.21.15.42:0/646765576 >> v2:172.21.15.154:6812/254845342 conn(0x7f9824037820 0x7f9824039ce0 unknown :-1 s=CLOSED pgs=3660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:56:28.597 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.590+0000 7f983ddad640 1 --2- 172.21.15.42:0/646765576 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9838134a90 0x7f983809cb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:56:28.597 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.590+0000 7f983ddad640 1 --2- 172.21.15.42:0/646765576 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9838135100 0x7f9838134550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:56:28.597 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.590+0000 7f983ddad640 1 --2- 172.21.15.42:0/646765576 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f98380a1330 0x7f983813b300 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:56:28.597 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.590+0000 7f983ddad640 1 -- 172.21.15.42:0/646765576 >> 172.21.15.42:0/646765576 conn(0x7f98380a2420 msgr2=0x7f983813c590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:56:28.598 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.590+0000 7f983ddad640 1 -- 172.21.15.42:0/646765576 shutdown_connections 2024-09-16T03:56:28.598 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.590+0000 7f983ddad640 1 -- 172.21.15.42:0/646765576 wait complete. 2024-09-16T03:56:28.598 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:28.590+0000 7f983ddad640 1 librados: shutdown 2024-09-16T03:56:31.612 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:56:31.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:31.742+0000 7f84bfdb8640 10 monclient: get_monmap_and_config 2024-09-16T03:56:31.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:31.742+0000 7f84bfdb8640 10 monclient: build_initial_monmap 2024-09-16T03:56:31.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:31.742+0000 7f84bfdb8640 10 monclient: monmap: 2024-09-16T03:56:31.746 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:56:31.747 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:56:31.747 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:56:31.745226+0000 2024-09-16T03:56:31.747 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:56:31.745226+0000 2024-09-16T03:56:31.747 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:56:31.747 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:56:31.747 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:56:31.747 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:56:31.747 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:56:31.747 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:56:31.747 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:56:31.747 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:56:31.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:31.742+0000 7f84bfdb8640 1 Processor -- start 2024-09-16T03:56:31.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:31.742+0000 7f84bfdb8640 1 -- start start 2024-09-16T03:56:31.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:31.742+0000 7f84bfdb8640 10 monclient: init 2024-09-16T03:56:31.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:31.742+0000 7f84bfdb8640 10 monclient: _reopen_session rank -1 2024-09-16T03:56:31.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:31.742+0000 7f84bfdb8640 10 monclient: _add_conns ranks=[0,2,1,4,3] 2024-09-16T03:56:31.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:31.742+0000 7f84bfdb8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f84b806db90 addr v1:172.21.15.42:6789/0 2024-09-16T03:56:31.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:31.742+0000 7f84bfdb8640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f84b81340c0 0x7f84b813c190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:56:31.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:31.742+0000 7f84bfdb8640 10 monclient(hunting): picked mon.noname-b con 0x7f84b81340c0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:56:31.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:31.742+0000 7f84bfdb8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f84b813c6d0 addr v1:172.21.15.154:6789/0 2024-09-16T03:56:31.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:31.742+0000 7f84bfdb8640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f84b806ece0 con 0x7f84b806db90 2024-09-16T03:56:31.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:31.742+0000 7f84bfdb8640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f84b809bf80 con 0x7f84b813c6d0 2024-09-16T03:56:31.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:31.742+0000 7f84bfdb8640 10 monclient(hunting): start opening mon connection 2024-09-16T03:56:31.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:31.742+0000 7f84bfdb8640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f84b806df70 con 0x7f84b81340c0 2024-09-16T03:56:31.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:31.742+0000 7f84bfdb8640 10 monclient(hunting): _renew_subs 2024-09-16T03:56:31.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:31.742+0000 7f84bfdb8640 10 monclient(hunting): authenticate will time out at 1838.714726s 2024-09-16T03:56:31.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:31.742+0000 7f84bd353640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f84b81340c0 0x7f84b813c190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:56:31.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:31.742+0000 7f84bdb54640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f84b806db90 0x7f84b80a37f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:37496/0 (socket says 172.21.15.42:37496) 2024-09-16T03:56:31.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:31.742+0000 7f84be355640 1 -- >> v1:172.21.15.154:6789/0 conn(0x7f84b813c6d0 legacy=0x7f84b813eac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.154:6789/0 2024-09-16T03:56:31.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:31.742+0000 7f84bdb54640 1 -- 172.21.15.42:0/1540466343 learned_addr learned my addr 172.21.15.42:0/1540466343 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:56:31.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:31.742+0000 7f84bd353640 10 monclient(hunting): get_auth_request con 0x7f84b81340c0 auth_method 0 2024-09-16T03:56:31.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:31.742+0000 7f84bd353640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:56:31.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:31.742+0000 7f84bd353640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:56:31.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:31.742+0000 7f84bd353640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:56:31.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:31.742+0000 7f84bd353640 1 -- 172.21.15.42:0/1540466343 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f84b81340c0 msgr2=0x7f84b813c190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T03:56:31.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:31.742+0000 7f84bd353640 1 -- 172.21.15.42:0/1540466343 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f84b81340c0 msgr2=0x7f84b813c190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:56:31.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:31.742+0000 7f84bd353640 1 --2- 172.21.15.42:0/1540466343 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f84b81340c0 0x7f84b813c190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:56:31.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:31.742+0000 7f84bd353640 1 --2- 172.21.15.42:0/1540466343 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f84b81340c0 0x7f84b813c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:56:31.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:31.946+0000 7f84be355640 1 -- 172.21.15.42:0/1540466343 >> v1:172.21.15.154:6789/0 conn(0x7f84b813c6d0 legacy=0x7f84b813eac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.154:6789/0 2024-09-16T03:56:31.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:31.946+0000 7f84bd353640 1 --2- 172.21.15.42:0/1540466343 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f84b81340c0 0x7f84b813c190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:56:31.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:31.946+0000 7f84bd353640 10 monclient(hunting): get_auth_request con 0x7f84b81340c0 auth_method 0 2024-09-16T03:56:31.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:31.946+0000 7f84bd353640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:56:31.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:31.946+0000 7f84bd353640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:56:31.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:31.946+0000 7f84bd353640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:56:31.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:31.946+0000 7f84bd353640 1 -- 172.21.15.42:0/1540466343 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f84b81340c0 msgr2=0x7f84b813c190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T03:56:31.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:31.946+0000 7f84bd353640 1 -- 172.21.15.42:0/1540466343 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f84b81340c0 msgr2=0x7f84b813c190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:56:31.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:31.946+0000 7f84bd353640 1 --2- 172.21.15.42:0/1540466343 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f84b81340c0 0x7f84b813c190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:56:31.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:31.946+0000 7f84bd353640 1 --2- 172.21.15.42:0/1540466343 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f84b81340c0 0x7f84b813c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T03:56:32.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:32.350+0000 7f84be355640 1 -- 172.21.15.42:0/1540466343 >> v1:172.21.15.154:6789/0 conn(0x7f84b813c6d0 legacy=0x7f84b813eac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.154:6789/0 2024-09-16T03:56:32.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:32.350+0000 7f84bd353640 1 --2- 172.21.15.42:0/1540466343 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f84b81340c0 0x7f84b813c190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:56:32.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:32.350+0000 7f84bd353640 10 monclient(hunting): get_auth_request con 0x7f84b81340c0 auth_method 0 2024-09-16T03:56:32.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:32.350+0000 7f84bd353640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:56:32.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:32.350+0000 7f84bd353640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:56:32.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:32.350+0000 7f84bd353640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:56:32.355 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:32.350+0000 7f84bd353640 1 -- 172.21.15.42:0/1540466343 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f84b81340c0 msgr2=0x7f84b813c190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T03:56:32.355 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:32.350+0000 7f84bd353640 1 -- 172.21.15.42:0/1540466343 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f84b81340c0 msgr2=0x7f84b813c190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:56:32.355 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:32.350+0000 7f84bd353640 1 --2- 172.21.15.42:0/1540466343 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f84b81340c0 0x7f84b813c190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:56:32.355 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:32.350+0000 7f84bd353640 1 --2- 172.21.15.42:0/1540466343 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f84b81340c0 0x7f84b813c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T03:56:33.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:33.154+0000 7f84be355640 1 -- 172.21.15.42:0/1540466343 >> v1:172.21.15.154:6789/0 conn(0x7f84b813c6d0 legacy=0x7f84b813eac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.154:6789/0 2024-09-16T03:56:33.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:33.154+0000 7f84bd353640 1 --2- 172.21.15.42:0/1540466343 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f84b81340c0 0x7f84b813c190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:56:33.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:33.154+0000 7f84bd353640 10 monclient(hunting): get_auth_request con 0x7f84b81340c0 auth_method 0 2024-09-16T03:56:33.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:33.154+0000 7f84bd353640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:56:33.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:33.154+0000 7f84bd353640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:56:33.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:33.154+0000 7f84bd353640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:56:33.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:33.154+0000 7f84bd353640 1 -- 172.21.15.42:0/1540466343 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f84b81340c0 msgr2=0x7f84b813c190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T03:56:33.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:33.154+0000 7f84bd353640 1 -- 172.21.15.42:0/1540466343 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f84b81340c0 msgr2=0x7f84b813c190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:56:33.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:33.154+0000 7f84bd353640 1 --2- 172.21.15.42:0/1540466343 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f84b81340c0 0x7f84b813c190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:56:33.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:33.154+0000 7f84bd353640 1 --2- 172.21.15.42:0/1540466343 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f84b81340c0 0x7f84b813c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T03:56:34.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:34.742+0000 7f84af7fe640 10 monclient(hunting): tick 2024-09-16T03:56:34.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:34.742+0000 7f84af7fe640 10 monclient(hunting): _check_auth_tickets 2024-09-16T03:56:34.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:34.742+0000 7f84af7fe640 1 monclient(hunting): continuing hunt 2024-09-16T03:56:34.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:34.742+0000 7f84af7fe640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T03:56:34.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:34.742+0000 7f84af7fe640 1 -- 172.21.15.42:0/1540466343 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f84b81340c0 msgr2=0x7f84b813c190 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:56:34.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:34.742+0000 7f84af7fe640 1 --2- 172.21.15.42:0/1540466343 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f84b81340c0 0x7f84b813c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:56:34.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:34.742+0000 7f84af7fe640 1 -- 172.21.15.42:0/1540466343 >> v1:172.21.15.154:6789/0 conn(0x7f84b813c6d0 legacy=0x7f84b813eac0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:56:34.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:34.742+0000 7f84af7fe640 1 -- 172.21.15.42:0/1540466343 >> v1:172.21.15.42:6789/0 conn(0x7f84b806db90 legacy=0x7f84b80a37f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:56:34.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:34.742+0000 7f84af7fe640 10 monclient: _add_conns ranks=[3,4] 2024-09-16T03:56:34.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:34.742+0000 7f84af7fe640 1 --2- 172.21.15.42:0/1540466343 >> v2:172.21.15.42:3300/0 conn(0x7f849c0036e0 0x7f849c005b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:56:34.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:34.742+0000 7f84af7fe640 10 monclient(hunting): picked mon.noname-a con 0x7f849c0036e0 addr v2:172.21.15.42:3300/0 2024-09-16T03:56:34.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:34.742+0000 7f84af7fe640 1 --2- 172.21.15.42:0/1540466343 >> v2:172.21.15.154:3300/0 conn(0x7f849c006250 0x7f849c008710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:56:34.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:34.742+0000 7f84af7fe640 10 monclient(hunting): picked mon.noname-c con 0x7f849c006250 addr v2:172.21.15.154:3300/0 2024-09-16T03:56:34.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:34.742+0000 7f84af7fe640 10 monclient(hunting): start opening mon connection 2024-09-16T03:56:34.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:34.742+0000 7f84af7fe640 1 -- 172.21.15.42:0/1540466343 --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f84b806df70 con 0x7f849c0036e0 2024-09-16T03:56:34.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:34.742+0000 7f84af7fe640 10 monclient(hunting): start opening mon connection 2024-09-16T03:56:34.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:34.742+0000 7f84af7fe640 1 -- 172.21.15.42:0/1540466343 --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f849c008dd0 con 0x7f849c006250 2024-09-16T03:56:34.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:34.742+0000 7f84af7fe640 10 monclient(hunting): _renew_subs 2024-09-16T03:56:34.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:34.742+0000 7f84bd353640 1 -- 172.21.15.42:0/1540466343 >> v2:172.21.15.154:3300/0 conn(0x7f849c006250 msgr2=0x7f849c008710 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T03:56:34.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:34.742+0000 7f84bdb54640 1 --2- 172.21.15.42:0/1540466343 >> v2:172.21.15.42:3300/0 conn(0x7f849c0036e0 0x7f849c005b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:56:34.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:34.742+0000 7f84bd353640 1 --2- 172.21.15.42:0/1540466343 >> v2:172.21.15.154:3300/0 conn(0x7f849c006250 0x7f849c008710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:56:34.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:34.742+0000 7f84bdb54640 10 monclient(hunting): get_auth_request con 0x7f849c0036e0 auth_method 0 2024-09-16T03:56:34.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:34.742+0000 7f84bdb54640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:56:34.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:34.742+0000 7f84bdb54640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:56:34.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:34.742+0000 7f84bdb54640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:56:34.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:34.742+0000 7f84bdb54640 1 -- 172.21.15.42:0/1540466343 >> v2:172.21.15.42:3300/0 conn(0x7f849c0036e0 msgr2=0x7f849c005b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T03:56:34.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:34.742+0000 7f84bdb54640 1 -- 172.21.15.42:0/1540466343 >> v2:172.21.15.42:3300/0 conn(0x7f849c0036e0 msgr2=0x7f849c005b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:56:34.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:34.742+0000 7f84bdb54640 1 --2- 172.21.15.42:0/1540466343 >> v2:172.21.15.42:3300/0 conn(0x7f849c0036e0 0x7f849c005b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:56:34.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:34.742+0000 7f84bdb54640 1 --2- 172.21.15.42:0/1540466343 >> v2:172.21.15.42:3300/0 conn(0x7f849c0036e0 0x7f849c005b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:56:34.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:34.946+0000 7f84bd353640 1 -- 172.21.15.42:0/1540466343 >> v2:172.21.15.154:3300/0 conn(0x7f849c006250 msgr2=0x7f849c008710 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T03:56:34.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:34.946+0000 7f84bd353640 1 --2- 172.21.15.42:0/1540466343 >> v2:172.21.15.154:3300/0 conn(0x7f849c006250 0x7f849c008710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T03:56:34.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:34.946+0000 7f84bdb54640 1 --2- 172.21.15.42:0/1540466343 >> v2:172.21.15.42:3300/0 conn(0x7f849c0036e0 0x7f849c005b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:56:34.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:34.946+0000 7f84bdb54640 10 monclient(hunting): get_auth_request con 0x7f849c0036e0 auth_method 0 2024-09-16T03:56:34.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:34.946+0000 7f84bdb54640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:56:34.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:34.946+0000 7f84bdb54640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:56:34.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:34.946+0000 7f84bdb54640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:56:34.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:34.946+0000 7f84bdb54640 1 -- 172.21.15.42:0/1540466343 >> v2:172.21.15.42:3300/0 conn(0x7f849c0036e0 msgr2=0x7f849c005b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T03:56:34.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:34.946+0000 7f84bdb54640 1 -- 172.21.15.42:0/1540466343 >> v2:172.21.15.42:3300/0 conn(0x7f849c0036e0 msgr2=0x7f849c005b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:56:34.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:34.946+0000 7f84bdb54640 1 --2- 172.21.15.42:0/1540466343 >> v2:172.21.15.42:3300/0 conn(0x7f849c0036e0 0x7f849c005b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:56:34.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:34.946+0000 7f84bdb54640 1 --2- 172.21.15.42:0/1540466343 >> v2:172.21.15.42:3300/0 conn(0x7f849c0036e0 0x7f849c005b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T03:56:35.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:35.350+0000 7f84bd353640 1 -- 172.21.15.42:0/1540466343 >> v2:172.21.15.154:3300/0 conn(0x7f849c006250 msgr2=0x7f849c008710 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T03:56:35.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:35.350+0000 7f84bdb54640 1 --2- 172.21.15.42:0/1540466343 >> v2:172.21.15.42:3300/0 conn(0x7f849c0036e0 0x7f849c005b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:56:35.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:35.350+0000 7f84bd353640 1 --2- 172.21.15.42:0/1540466343 >> v2:172.21.15.154:3300/0 conn(0x7f849c006250 0x7f849c008710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T03:56:35.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:35.350+0000 7f84bdb54640 10 monclient(hunting): get_auth_request con 0x7f849c0036e0 auth_method 0 2024-09-16T03:56:35.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:35.350+0000 7f84bdb54640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:56:35.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:35.350+0000 7f84bdb54640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:56:35.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:35.350+0000 7f84bdb54640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:56:35.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:35.350+0000 7f84bdb54640 1 -- 172.21.15.42:0/1540466343 >> v2:172.21.15.42:3300/0 conn(0x7f849c0036e0 msgr2=0x7f849c005b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T03:56:35.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:35.350+0000 7f84bdb54640 1 -- 172.21.15.42:0/1540466343 >> v2:172.21.15.42:3300/0 conn(0x7f849c0036e0 msgr2=0x7f849c005b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:56:35.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:35.350+0000 7f84bdb54640 1 --2- 172.21.15.42:0/1540466343 >> v2:172.21.15.42:3300/0 conn(0x7f849c0036e0 0x7f849c005b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:56:35.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:35.350+0000 7f84bdb54640 1 --2- 172.21.15.42:0/1540466343 >> v2:172.21.15.42:3300/0 conn(0x7f849c0036e0 0x7f849c005b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T03:56:36.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:36.154+0000 7f84bd353640 1 -- 172.21.15.42:0/1540466343 >> v2:172.21.15.154:3300/0 conn(0x7f849c006250 msgr2=0x7f849c008710 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T03:56:36.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:36.154+0000 7f84bdb54640 1 --2- 172.21.15.42:0/1540466343 >> v2:172.21.15.42:3300/0 conn(0x7f849c0036e0 0x7f849c005b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:56:36.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:36.154+0000 7f84bd353640 1 --2- 172.21.15.42:0/1540466343 >> v2:172.21.15.154:3300/0 conn(0x7f849c006250 0x7f849c008710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T03:56:36.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:36.154+0000 7f84bdb54640 10 monclient(hunting): get_auth_request con 0x7f849c0036e0 auth_method 0 2024-09-16T03:56:36.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:36.154+0000 7f84bdb54640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:56:36.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:36.154+0000 7f84bdb54640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:56:36.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:36.154+0000 7f84bdb54640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:56:36.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:36.154+0000 7f84bdb54640 1 -- 172.21.15.42:0/1540466343 >> v2:172.21.15.42:3300/0 conn(0x7f849c0036e0 msgr2=0x7f849c005b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T03:56:36.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:36.154+0000 7f84bdb54640 1 -- 172.21.15.42:0/1540466343 >> v2:172.21.15.42:3300/0 conn(0x7f849c0036e0 msgr2=0x7f849c005b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:56:36.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:36.154+0000 7f84bdb54640 1 --2- 172.21.15.42:0/1540466343 >> v2:172.21.15.42:3300/0 conn(0x7f849c0036e0 0x7f849c005b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:56:36.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:36.154+0000 7f84bdb54640 1 --2- 172.21.15.42:0/1540466343 >> v2:172.21.15.42:3300/0 conn(0x7f849c0036e0 0x7f849c005b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T03:56:37.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.746+0000 7f84af7fe640 10 monclient(hunting): tick 2024-09-16T03:56:37.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.746+0000 7f84af7fe640 10 monclient(hunting): _check_auth_tickets 2024-09-16T03:56:37.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.746+0000 7f84af7fe640 1 monclient(hunting): continuing hunt 2024-09-16T03:56:37.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.746+0000 7f84af7fe640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T03:56:37.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.746+0000 7f84af7fe640 1 -- 172.21.15.42:0/1540466343 >> v2:172.21.15.154:3300/0 conn(0x7f849c006250 msgr2=0x7f849c008710 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:56:37.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.746+0000 7f84af7fe640 1 --2- 172.21.15.42:0/1540466343 >> v2:172.21.15.154:3300/0 conn(0x7f849c006250 0x7f849c008710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:56:37.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.746+0000 7f84af7fe640 1 -- 172.21.15.42:0/1540466343 >> v2:172.21.15.42:3300/0 conn(0x7f849c0036e0 msgr2=0x7f849c005b60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:56:37.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.746+0000 7f84af7fe640 1 --2- 172.21.15.42:0/1540466343 >> v2:172.21.15.42:3300/0 conn(0x7f849c0036e0 0x7f849c005b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:56:37.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.746+0000 7f84af7fe640 10 monclient: _add_conns ranks=[1,3,0,2,4] 2024-09-16T03:56:37.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.746+0000 7f84be355640 1 -- 172.21.15.42:0/1540466343 reap_dead start 2024-09-16T03:56:37.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.746+0000 7f84af7fe640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f84b813c6d0 addr v1:172.21.15.154:6789/0 2024-09-16T03:56:37.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.746+0000 7f84af7fe640 1 --2- 172.21.15.42:0/1540466343 >> v2:172.21.15.42:3300/0 conn(0x7f849c003b50 0x7f849c006700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:56:37.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.746+0000 7f84af7fe640 10 monclient(hunting): picked mon.noname-a con 0x7f849c003b50 addr v2:172.21.15.42:3300/0 2024-09-16T03:56:37.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.746+0000 7f84af7fe640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f849c006c40 addr v1:172.21.15.42:6789/0 2024-09-16T03:56:37.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.746+0000 7f84af7fe640 1 -- 172.21.15.42:0/1540466343 --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f84b81344d0 con 0x7f849c006c40 2024-09-16T03:56:37.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.746+0000 7f84af7fe640 1 -- 172.21.15.42:0/1540466343 --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f849c0088d0 con 0x7f84b813c6d0 2024-09-16T03:56:37.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.746+0000 7f84af7fe640 10 monclient(hunting): start opening mon connection 2024-09-16T03:56:37.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.746+0000 7f84af7fe640 1 -- 172.21.15.42:0/1540466343 --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f84b806df70 con 0x7f849c003b50 2024-09-16T03:56:37.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.746+0000 7f84af7fe640 10 monclient(hunting): _renew_subs 2024-09-16T03:56:37.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.746+0000 7f84bdb54640 1 -- 172.21.15.42:0/1540466343 >> v1:172.21.15.154:6789/0 conn(0x7f84b813c6d0 legacy=0x7f849c00b270 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.154:6789/0 2024-09-16T03:56:37.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.746+0000 7f84bd353640 1 --2- 172.21.15.42:0/1540466343 >> v2:172.21.15.42:3300/0 conn(0x7f849c003b50 0x7f849c006700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:56:37.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.746+0000 7f84bd353640 10 monclient(hunting): get_auth_request con 0x7f849c003b50 auth_method 0 2024-09-16T03:56:37.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.746+0000 7f84bd353640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:56:37.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.746+0000 7f84bd353640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:56:37.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.746+0000 7f84bd353640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:56:37.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.746+0000 7f84bd353640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:56:37.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.746+0000 7f84bd353640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:56:37.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.746+0000 7f84bd353640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:56:37.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.746+0000 7f84bd353640 10 monclient(hunting): handle_auth_done global_id 824769 payload 274 2024-09-16T03:56:37.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.746+0000 7f84bd353640 1 -- 172.21.15.42:0/1540466343 >> v1:172.21.15.154:6789/0 conn(0x7f84b813c6d0 legacy=0x7f849c00b270 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:56:37.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.746+0000 7f84bd353640 1 -- 172.21.15.42:0/1540466343 >> v1:172.21.15.42:6789/0 conn(0x7f849c006c40 legacy=0x7f849c0070f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:56:37.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.746+0000 7f84bd353640 10 monclient: _finish_hunting 0 2024-09-16T03:56:37.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.746+0000 7f84bd353640 1 monclient: found mon.noname-a 2024-09-16T03:56:37.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.746+0000 7f84bd353640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:56:37.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.746+0000 7f84bd353640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:56:37.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.746+0000 7f84bd353640 1 -- 172.21.15.42:0/1540466343 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f849c0109c0 con 0x7f849c003b50 2024-09-16T03:56:37.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.746+0000 7f84bcb52640 1 -- 172.21.15.42:0/1540466343 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1525109559 0 0) 0x7f84b81344d0 con 0x7f849c006c40 2024-09-16T03:56:37.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.746+0000 7f84bcb52640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T03:56:37.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.750+0000 7f84bd353640 1 --2- 172.21.15.42:0/1540466343 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f849c003b50 0x7f849c006700 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f84a8004660 tx=0x7f84a801d230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2bdf886b916d21e0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:56:37.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.750+0000 7f84bcb52640 1 -- 172.21.15.42:0/1540466343 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84a80b3070 con 0x7f849c003b50 2024-09-16T03:56:37.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.750+0000 7f84bcb52640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:56:37.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.750+0000 7f84bcb52640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:56:37.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.750+0000 7f84bcb52640 10 monclient: dump: 2024-09-16T03:56:37.757 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:56:37.757 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:56:37.757 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:37.757 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:37.757 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:56:37.757 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:56:37.757 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:56:37.757 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:56:37.757 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:56:37.757 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:56:37.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.750+0000 7f84bcb52640 10 monclient: _finish_auth 0 2024-09-16T03:56:37.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.750+0000 7f84bcb52640 10 monclient: _check_auth_tickets 2024-09-16T03:56:37.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.750+0000 7f84bcb52640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:56:37.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.750+0000 7f84bcb52640 1 -- 172.21.15.42:0/1540466343 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f84a80ae410 con 0x7f849c003b50 2024-09-16T03:56:37.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.750+0000 7f84bcb52640 10 monclient: handle_config config(0 keys) 2024-09-16T03:56:37.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.750+0000 7f84bcb52640 1 -- 172.21.15.42:0/1540466343 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84a800f040 con 0x7f849c003b50 2024-09-16T03:56:37.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.750+0000 7f84bcb52640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:56:37.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.750+0000 7f84bcb52640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:56:37.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.750+0000 7f84bcb52640 10 monclient: dump: 2024-09-16T03:56:37.758 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:56:37.758 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:56:37.759 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:37.759 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:37.759 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:56:37.759 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:56:37.759 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:56:37.759 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:56:37.759 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:56:37.759 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:56:37.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.750+0000 7f84bfdb8640 5 monclient: authenticate success, global_id 824769 2024-09-16T03:56:37.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.750+0000 7f84bfdb8640 10 monclient: get_monmap_and_config success 2024-09-16T03:56:37.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.750+0000 7f84bfdb8640 4 set_mon_vals no callback set 2024-09-16T03:56:37.767 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.754+0000 7f84bfdb8640 10 monclient: shutdown 2024-09-16T03:56:37.767 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.754+0000 7f84bfdb8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:56:37.767 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.754+0000 7f84bfdb8640 1 -- 172.21.15.42:0/1540466343 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f849c003b50 msgr2=0x7f849c006700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:56:37.767 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.754+0000 7f84bfdb8640 1 --2- 172.21.15.42:0/1540466343 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f849c003b50 0x7f849c006700 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f84a8004660 tx=0x7f84a801d230 comp rx=0 tx=0).stop 2024-09-16T03:56:37.768 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.754+0000 7f84bfdb8640 1 -- 172.21.15.42:0/1540466343 shutdown_connections 2024-09-16T03:56:37.768 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.754+0000 7f84bfdb8640 1 --2- 172.21.15.42:0/1540466343 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f849c003b50 0x7f849c006700 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:56:37.768 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.754+0000 7f84bfdb8640 1 -- 172.21.15.42:0/1540466343 >> 172.21.15.42:0/1540466343 conn(0x7f84b80a9fb0 msgr2=0x7f84b80aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:56:37.768 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.754+0000 7f84bfdb8640 1 -- 172.21.15.42:0/1540466343 shutdown_connections 2024-09-16T03:56:37.768 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.754+0000 7f84bfdb8640 1 -- 172.21.15.42:0/1540466343 wait complete. 2024-09-16T03:56:37.768 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.754+0000 7f84bfdb8640 10 monclient: build_initial_monmap 2024-09-16T03:56:37.768 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.754+0000 7f84bfdb8640 10 monclient: monmap: 2024-09-16T03:56:37.768 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:56:37.768 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:56:37.768 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:56:37.768 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:56:37.768 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:56:37.768 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:56:37.769 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:56:37.769 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:56:37.769 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:56:37.769 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:56:37.769 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.754+0000 7f84bfdb8640 1 librados: starting msgr at 2024-09-16T03:56:37.769 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.754+0000 7f84bfdb8640 1 librados: starting objecter 2024-09-16T03:56:37.769 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.754+0000 7f84bfdb8640 1 Processor -- start 2024-09-16T03:56:37.769 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.754+0000 7f84bfdb8640 1 -- start start 2024-09-16T03:56:37.769 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.754+0000 7f84bfdb8640 1 librados: setting wanted keys 2024-09-16T03:56:37.769 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.754+0000 7f84bfdb8640 1 librados: calling monclient init 2024-09-16T03:56:37.769 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.754+0000 7f84bfdb8640 10 monclient: init 2024-09-16T03:56:37.769 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.754+0000 7f84bfdb8640 10 monclient: _reopen_session rank -1 2024-09-16T03:56:37.770 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.754+0000 7f84bfdb8640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:56:37.770 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.754+0000 7f84bfdb8640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f849c003b50 0x7f84b81ced90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:56:37.770 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.754+0000 7f84bfdb8640 10 monclient(hunting): picked mon.noname-c con 0x7f849c003b50 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:56:37.770 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.754+0000 7f84bfdb8640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f849c006c40 0x7f84b80a2940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:56:37.770 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.754+0000 7f84bfdb8640 10 monclient(hunting): picked mon.noname-a con 0x7f849c006c40 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:56:37.770 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.754+0000 7f84bfdb8640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f84b813c6d0 0x7f84b80a2e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:56:37.770 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.754+0000 7f84bfdb8640 10 monclient(hunting): picked mon.noname-b con 0x7f84b813c6d0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:56:37.770 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.754+0000 7f84bfdb8640 10 monclient(hunting): start opening mon connection 2024-09-16T03:56:37.770 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.754+0000 7f84bfdb8640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f84b81371a0 con 0x7f849c006c40 2024-09-16T03:56:37.770 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.754+0000 7f84bfdb8640 10 monclient(hunting): start opening mon connection 2024-09-16T03:56:37.770 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.754+0000 7f84bfdb8640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f84b8137020 con 0x7f84b813c6d0 2024-09-16T03:56:37.770 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.754+0000 7f84bfdb8640 10 monclient(hunting): start opening mon connection 2024-09-16T03:56:37.770 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.754+0000 7f84bfdb8640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f84b8137320 con 0x7f849c003b50 2024-09-16T03:56:37.771 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.754+0000 7f84bfdb8640 10 monclient(hunting): _renew_subs 2024-09-16T03:56:37.771 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.754+0000 7f84bfdb8640 10 monclient(hunting): authenticate will time out at 1844.727117s 2024-09-16T03:56:37.771 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.754+0000 7f84be355640 1 -- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f84b813c6d0 msgr2=0x7f84b80a2e80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T03:56:37.771 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.754+0000 7f84be355640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f84b813c6d0 0x7f84b80a2e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:56:37.771 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.754+0000 7f84bd353640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f849c006c40 0x7f84b80a2940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:56:37.771 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.754+0000 7f84bd353640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f849c006c40 0x7f84b80a2940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:44726/0 (socket says 172.21.15.42:44726) 2024-09-16T03:56:37.771 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.754+0000 7f84bd353640 1 -- 172.21.15.42:0/2456468565 learned_addr learned my addr 172.21.15.42:0/2456468565 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:56:37.771 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.754+0000 7f84bd353640 10 monclient(hunting): get_auth_request con 0x7f849c006c40 auth_method 0 2024-09-16T03:56:37.771 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.754+0000 7f84bd353640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:56:37.771 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.754+0000 7f84bd353640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:56:37.771 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.754+0000 7f84bd353640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:56:37.771 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.754+0000 7f84bdb54640 1 --2- 172.21.15.42:0/2456468565 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f849c003b50 0x7f84b81ced90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:56:37.771 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.754+0000 7f84bdb54640 10 monclient(hunting): get_auth_request con 0x7f849c003b50 auth_method 0 2024-09-16T03:56:37.771 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.754+0000 7f84bdb54640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:56:37.771 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.754+0000 7f84bdb54640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:56:37.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.754+0000 7f84bdb54640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:56:37.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.754+0000 7f84bd353640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:56:37.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.754+0000 7f84bd353640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:56:37.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.754+0000 7f84bd353640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:56:37.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.754+0000 7f84bdb54640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:56:37.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.754+0000 7f84bdb54640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:56:37.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.754+0000 7f84bdb54640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:56:37.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.754+0000 7f84bd353640 10 monclient(hunting): handle_auth_done global_id 824775 payload 824 2024-09-16T03:56:37.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.754+0000 7f84bd353640 1 -- 172.21.15.42:0/2456468565 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f849c003b50 msgr2=0x7f84b81ced90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:56:37.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.754+0000 7f84bd353640 1 --2- 172.21.15.42:0/2456468565 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f849c003b50 0x7f84b81ced90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:56:37.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.754+0000 7f84bd353640 1 -- 172.21.15.42:0/2456468565 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f84b813c6d0 msgr2=0x7f84b80a2e80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:56:37.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.754+0000 7f84bd353640 1 --2- 172.21.15.42:0/2456468565 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f84b813c6d0 0x7f84b80a2e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:56:37.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.754+0000 7f84bd353640 10 monclient: _finish_hunting 0 2024-09-16T03:56:37.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.754+0000 7f84bd353640 1 monclient: found mon.noname-a 2024-09-16T03:56:37.773 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.754+0000 7f84bd353640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:56:37.773 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.754+0000 7f84bd353640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:56:37.773 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.754+0000 7f84bd353640 1 -- 172.21.15.42:0/2456468565 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84b80a89f0 con 0x7f849c006c40 2024-09-16T03:56:37.773 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.754+0000 7f84bdb54640 1 --2- 172.21.15.42:0/2456468565 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f849c003b50 0x7f84b81ced90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:56:37.773 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.758+0000 7f84bd353640 1 --2- 172.21.15.42:0/2456468565 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f849c006c40 0x7f84b80a2940 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f84a80b1040 tx=0x7f84a8019fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:56:37.773 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.758+0000 7f84ae7fc640 10 client.?.objecter ms_handle_connect 0x7f849c006c40 2024-09-16T03:56:37.773 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.758+0000 7f84ae7fc640 10 client.?.objecter resend_mon_ops 2024-09-16T03:56:37.773 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.758+0000 7f84ae7fc640 1 -- 172.21.15.42:0/2456468565 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84a80b3040 con 0x7f849c006c40 2024-09-16T03:56:37.773 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.758+0000 7f84ae7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:56:37.773 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.758+0000 7f84ae7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:56:37.773 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.758+0000 7f84ae7fc640 10 monclient: dump: 2024-09-16T03:56:37.773 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:56:37.773 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:56:37.774 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:37.774 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:37.774 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:56:37.774 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:56:37.774 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:56:37.774 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:56:37.774 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:56:37.774 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:56:37.774 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.758+0000 7f84ae7fc640 10 monclient: _finish_auth 0 2024-09-16T03:56:37.774 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.758+0000 7f84ae7fc640 10 monclient: _check_auth_tickets 2024-09-16T03:56:37.774 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.758+0000 7f84ae7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:56:37.774 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.758+0000 7f84ae7fc640 1 -- 172.21.15.42:0/2456468565 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f84a80b7630 con 0x7f849c006c40 2024-09-16T03:56:37.774 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.758+0000 7f84bfdb8640 5 monclient: authenticate success, global_id 824775 2024-09-16T03:56:37.774 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.758+0000 7f84bfdb8640 10 monclient: _renew_subs 2024-09-16T03:56:37.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.758+0000 7f84bfdb8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:56:37.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.758+0000 7f84bfdb8640 1 -- 172.21.15.42:0/2456468565 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84b809f120 con 0x7f849c006c40 2024-09-16T03:56:37.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.758+0000 7f84ae7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:56:37.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.758+0000 7f84ae7fc640 1 -- 172.21.15.42:0/2456468565 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84a80b79e0 con 0x7f849c006c40 2024-09-16T03:56:37.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.758+0000 7f84ae7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:56:37.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.758+0000 7f84ae7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:56:37.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.758+0000 7f84ae7fc640 10 monclient: dump: 2024-09-16T03:56:37.775 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:56:37.775 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:56:37.775 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:37.775 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:37.775 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:56:37.775 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:56:37.776 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:56:37.776 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:56:37.776 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:56:37.776 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:56:37.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.758+0000 7f84af7fe640 4 set_mon_vals no callback set 2024-09-16T03:56:37.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.758+0000 7f84bfdb8640 10 client.824775.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:56:37.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.758+0000 7f84bfdb8640 10 monclient: _renew_subs 2024-09-16T03:56:37.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.758+0000 7f84bfdb8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:56:37.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.758+0000 7f84bfdb8640 1 -- 172.21.15.42:0/2456468565 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84b809fc80 con 0x7f849c006c40 2024-09-16T03:56:37.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.758+0000 7f84bfdb8640 1 librados: init done 2024-09-16T03:56:37.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.758+0000 7f84bfdb8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:56:37.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.758+0000 7f84bfdb8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:56:37.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.758+0000 7f84bfdb8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:56:37.777 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.758+0000 7f84bfdb8640 1 -- 172.21.15.42:0/2456468565 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f84b809ff40 con 0x7f849c006c40 2024-09-16T03:56:37.777 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.758+0000 7f84ae7fc640 1 -- 172.21.15.42:0/2456468565 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f84a80c0060 con 0x7f849c006c40 2024-09-16T03:56:37.777 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.758+0000 7f84ae7fc640 1 --2- 172.21.15.42:0/2456468565 >> v2:172.21.15.154:6812/254845342 conn(0x7f849c039110 0x7f849c03b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:56:37.777 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.758+0000 7f84ae7fc640 1 -- 172.21.15.42:0/2456468565 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(443..443 src has 330..443) ==== 5356+0+0 (secure 0 0 0) 0x7f84a80e3650 con 0x7f849c006c40 2024-09-16T03:56:37.777 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.758+0000 7f84ae7fc640 10 client.824775.objecter ms_dispatch 0x7f84b8000ba0 osd_map(443..443 src has 330..443) 2024-09-16T03:56:37.777 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.758+0000 7f84ae7fc640 3 client.824775.objecter handle_osd_map got epochs [443,443] > 0 2024-09-16T03:56:37.777 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.758+0000 7f84ae7fc640 3 client.824775.objecter handle_osd_map decoding full epoch 443 2024-09-16T03:56:37.777 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.758+0000 7f84ae7fc640 20 client.824775.objecter dump_active .. 0 homeless 2024-09-16T03:56:37.777 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.758+0000 7f84bdb54640 1 --2- 172.21.15.42:0/2456468565 >> v2:172.21.15.154:6812/254845342 conn(0x7f849c039110 0x7f849c03b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:56:37.777 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.758+0000 7f84bdb54640 10 monclient: get_auth_request con 0x7f849c039110 auth_method 0 2024-09-16T03:56:37.777 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.758+0000 7f84bdb54640 1 --2- 172.21.15.42:0/2456468565 >> v2:172.21.15.154:6812/254845342 conn(0x7f849c039110 0x7f849c03b5d0 secure :-1 s=READY pgs=3661 cs=0 l=1 rev1=1 crypto rx=0x7f84b4007fd0 tx=0x7f84b4009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:56:37.777 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.758+0000 7f84ae7fc640 10 client.824775.objecter ms_handle_connect 0x7f849c039110 2024-09-16T03:56:37.778 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.766+0000 7f84ae7fc640 1 -- 172.21.15.42:0/2456468565 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f84a8016f80 con 0x7f849c006c40 2024-09-16T03:56:37.778 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.766+0000 7f84ae7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:56:37.778 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.766+0000 7f84ae7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:56:37.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.994+0000 7f84bfdb8640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:56:37.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.994+0000 7f84bfdb8640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:56:37.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.994+0000 7f84bfdb8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:56:37.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.994+0000 7f84bfdb8640 1 -- 172.21.15.42:0/2456468565 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f84b806f660 con 0x7f849c006c40 2024-09-16T03:56:37.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.994+0000 7f84ae7fc640 1 -- 172.21.15.42:0/2456468565 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1555 (secure 0 0 0) 0x7f84a80b6550 con 0x7f849c006c40 2024-09-16T03:56:37.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.994+0000 7f84ae7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:56:37.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.994+0000 7f84ae7fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:56:37.999 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":132,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:56:38.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.994+0000 7f84bfdb8640 10 librados: watch_flush enter 2024-09-16T03:56:38.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.994+0000 7f84bfdb8640 10 librados: watch_flush exit 2024-09-16T03:56:38.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.994+0000 7f84bfdb8640 20 client.824775.objecter shutdown clearing up homeless session... 2024-09-16T03:56:38.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.994+0000 7f84bfdb8640 10 client.824775.objecter successfully canceled tick 2024-09-16T03:56:38.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.998+0000 7f84bfdb8640 1 -- 172.21.15.42:0/2456468565 >> v2:172.21.15.154:6812/254845342 conn(0x7f849c039110 msgr2=0x7f849c03b5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:56:38.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.998+0000 7f84bfdb8640 1 --2- 172.21.15.42:0/2456468565 >> v2:172.21.15.154:6812/254845342 conn(0x7f849c039110 0x7f849c03b5d0 secure :-1 s=READY pgs=3661 cs=0 l=1 rev1=1 crypto rx=0x7f84b4007fd0 tx=0x7f84b4009040 comp rx=0 tx=0).stop 2024-09-16T03:56:38.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.998+0000 7f84bfdb8640 10 monclient: shutdown 2024-09-16T03:56:38.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.998+0000 7f84bfdb8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:56:38.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.998+0000 7f84bfdb8640 1 -- 172.21.15.42:0/2456468565 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f849c006c40 msgr2=0x7f84b80a2940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:56:38.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.998+0000 7f84bfdb8640 1 --2- 172.21.15.42:0/2456468565 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f849c006c40 0x7f84b80a2940 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f84a80b1040 tx=0x7f84a8019fd0 comp rx=0 tx=0).stop 2024-09-16T03:56:38.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.998+0000 7f84bfdb8640 1 -- 172.21.15.42:0/2456468565 shutdown_connections 2024-09-16T03:56:38.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.998+0000 7f84bfdb8640 1 --2- 172.21.15.42:0/2456468565 >> v2:172.21.15.154:6812/254845342 conn(0x7f849c039110 0x7f849c03b5d0 unknown :-1 s=CLOSED pgs=3661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:56:38.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.998+0000 7f84bfdb8640 1 --2- 172.21.15.42:0/2456468565 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f84b813c6d0 0x7f84b80a2e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:56:38.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.998+0000 7f84bfdb8640 1 --2- 172.21.15.42:0/2456468565 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f849c006c40 0x7f84b80a2940 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:56:38.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.998+0000 7f84bfdb8640 1 --2- 172.21.15.42:0/2456468565 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f849c003b50 0x7f84b81ced90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:56:38.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.998+0000 7f84bfdb8640 1 -- 172.21.15.42:0/2456468565 >> 172.21.15.42:0/2456468565 conn(0x7f84b80a9fb0 msgr2=0x7f84b8131f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:56:38.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.998+0000 7f84bfdb8640 1 -- 172.21.15.42:0/2456468565 shutdown_connections 2024-09-16T03:56:38.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.998+0000 7f84bfdb8640 1 -- 172.21.15.42:0/2456468565 wait complete. 2024-09-16T03:56:38.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:37.998+0000 7f84bfdb8640 1 librados: shutdown 2024-09-16T03:56:38.018 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T03:56:38.018 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-16T03:56:38.019 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T03:56:38.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.138+0000 7fd00da70640 10 monclient: get_monmap_and_config 2024-09-16T03:56:38.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.138+0000 7fd00da70640 10 monclient: build_initial_monmap 2024-09-16T03:56:38.143 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.138+0000 7fd00da70640 10 monclient: monmap: 2024-09-16T03:56:38.143 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:56:38.143 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:56:38.143 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:56:38.140794+0000 2024-09-16T03:56:38.143 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:56:38.140794+0000 2024-09-16T03:56:38.143 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:56:38.143 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:56:38.143 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:56:38.143 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:56:38.143 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:56:38.143 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:56:38.143 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:56:38.143 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:56:38.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.138+0000 7fd00da70640 1 Processor -- start 2024-09-16T03:56:38.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.138+0000 7fd00da70640 1 -- start start 2024-09-16T03:56:38.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.138+0000 7fd00da70640 10 monclient: init 2024-09-16T03:56:38.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.138+0000 7fd00da70640 10 monclient: _reopen_session rank -1 2024-09-16T03:56:38.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.138+0000 7fd00da70640 10 monclient: _add_conns ranks=[0,4,1,3,2] 2024-09-16T03:56:38.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.138+0000 7fd00da70640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd0080a39b0 addr v1:172.21.15.42:6789/0 2024-09-16T03:56:38.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.138+0000 7fd00da70640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fd0080a3d90 0x7fd00813c380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:56:38.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.138+0000 7fd00da70640 10 monclient(hunting): picked mon.noname-c con 0x7fd0080a3d90 addr v2:172.21.15.154:3300/0 2024-09-16T03:56:38.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.138+0000 7fd00da70640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd00813c8c0 addr v1:172.21.15.154:6789/0 2024-09-16T03:56:38.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.138+0000 7fd00da70640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd008134290 con 0x7fd0080a39b0 2024-09-16T03:56:38.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.138+0000 7fd00da70640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd00809c140 con 0x7fd00813c8c0 2024-09-16T03:56:38.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.138+0000 7fd00da70640 10 monclient(hunting): start opening mon connection 2024-09-16T03:56:38.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.138+0000 7fd00da70640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fd00809c420 con 0x7fd0080a3d90 2024-09-16T03:56:38.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.138+0000 7fd00da70640 10 monclient(hunting): _renew_subs 2024-09-16T03:56:38.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.138+0000 7fd00da70640 10 monclient(hunting): authenticate will time out at 1845.110232s 2024-09-16T03:56:38.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.138+0000 7fd0077fe640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fd0080a39b0 0x7fd008133b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:47998/0 (socket says 172.21.15.42:47998) 2024-09-16T03:56:38.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.138+0000 7fd0077fe640 1 -- 172.21.15.42:0/2104093700 learned_addr learned my addr 172.21.15.42:0/2104093700 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:56:38.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.138+0000 7fd007fff640 1 -- 172.21.15.42:0/2104093700 >> v1:172.21.15.154:6789/0 conn(0x7fd00813c8c0 legacy=0x7fd00813ecb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.154:6789/0 2024-09-16T03:56:38.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.138+0000 7fd006ffd640 1 -- 172.21.15.42:0/2104093700 >> v2:172.21.15.154:3300/0 conn(0x7fd0080a3d90 msgr2=0x7fd00813c380 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T03:56:38.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.138+0000 7fd006ffd640 1 --2- 172.21.15.42:0/2104093700 >> v2:172.21.15.154:3300/0 conn(0x7fd0080a3d90 0x7fd00813c380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:56:38.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.138+0000 7fd0067fc640 1 -- 172.21.15.42:0/2104093700 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 136567719 0 0) 0x7fd008134290 con 0x7fd0080a39b0 2024-09-16T03:56:38.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.138+0000 7fd0067fc640 10 monclient(hunting): _negotiate 2024-09-16T03:56:38.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.138+0000 7fd0067fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:56:38.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.138+0000 7fd0067fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:56:38.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.138+0000 7fd0067fc640 10 monclient(hunting): my global_id is 824778 2024-09-16T03:56:38.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.138+0000 7fd0067fc640 1 -- 172.21.15.42:0/2104093700 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcff0003530 con 0x7fd0080a39b0 2024-09-16T03:56:38.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.138+0000 7fd0067fc640 1 -- 172.21.15.42:0/2104093700 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1547144990 0 0) 0x7fcff0003530 con 0x7fd0080a39b0 2024-09-16T03:56:38.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.138+0000 7fd0067fc640 1 -- 172.21.15.42:0/2104093700 >> v2:172.21.15.154:3300/0 conn(0x7fd0080a3d90 msgr2=0x7fd00813c380 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:56:38.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.138+0000 7fd0067fc640 1 --2- 172.21.15.42:0/2104093700 >> v2:172.21.15.154:3300/0 conn(0x7fd0080a3d90 0x7fd00813c380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:56:38.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.138+0000 7fd0067fc640 1 -- 172.21.15.42:0/2104093700 >> v1:172.21.15.154:6789/0 conn(0x7fd00813c8c0 legacy=0x7fd00813ecb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:56:38.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.138+0000 7fd0067fc640 10 monclient: _finish_hunting 0 2024-09-16T03:56:38.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.138+0000 7fd0067fc640 1 monclient: found mon.noname-a-legacy 2024-09-16T03:56:38.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.138+0000 7fd0067fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:56:38.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.138+0000 7fd0067fc640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.42:6789/0 2024-09-16T03:56:38.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.138+0000 7fd0067fc640 1 -- 172.21.15.42:0/2104093700 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd00806dd60 con 0x7fd0080a39b0 2024-09-16T03:56:38.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.138+0000 7fd0067fc640 10 monclient: _finish_auth 0 2024-09-16T03:56:38.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.138+0000 7fd0067fc640 10 monclient: _check_auth_tickets 2024-09-16T03:56:38.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.138+0000 7fd0067fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:56:38.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.138+0000 7fd0067fc640 1 -- 172.21.15.42:0/2104093700 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2177413734 0 0) 0x7fd000001a00 con 0x7fd0080a39b0 2024-09-16T03:56:38.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.138+0000 7fd0067fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:56:38.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.138+0000 7fd0067fc640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T03:56:38.157 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.138+0000 7fd0067fc640 10 monclient: dump: 2024-09-16T03:56:38.157 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:56:38.157 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:56:38.157 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:38.157 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:38.157 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:56:38.157 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:56:38.157 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:56:38.157 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:56:38.157 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:56:38.157 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:56:38.157 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.138+0000 7fd0067fc640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] but i'm connected to v1:172.21.15.42:6789/0, reconnecting 2024-09-16T03:56:38.157 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.138+0000 7fd0067fc640 10 monclient: _reopen_session rank -1 2024-09-16T03:56:38.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.138+0000 7fd0067fc640 1 -- 172.21.15.42:0/2104093700 >> v1:172.21.15.42:6789/0 conn(0x7fd0080a39b0 legacy=0x7fd008133b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:56:38.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.138+0000 7fd0067fc640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:56:38.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.138+0000 7fd0067fc640 1 --2- 172.21.15.42:0/2104093700 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fcff00039c0 0x7fcff0003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:56:38.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.138+0000 7fd0067fc640 10 monclient(hunting): picked mon.c con 0x7fcff00039c0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:56:38.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.138+0000 7fd0067fc640 1 --2- 172.21.15.42:0/2104093700 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcff0004810 0x7fcff0004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:56:38.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.138+0000 7fd0067fc640 10 monclient(hunting): picked mon.a con 0x7fcff0004810 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:56:38.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.138+0000 7fd0067fc640 1 --2- 172.21.15.42:0/2104093700 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcff00053d0 0x7fcff000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:56:38.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.138+0000 7fd0067fc640 10 monclient(hunting): picked mon.b con 0x7fcff00053d0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:56:38.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.138+0000 7fd0067fc640 10 monclient(hunting): start opening mon connection 2024-09-16T03:56:38.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.138+0000 7fd0067fc640 1 -- 172.21.15.42:0/2104093700 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fd00809c420 con 0x7fcff0004810 2024-09-16T03:56:38.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.138+0000 7fd0067fc640 10 monclient(hunting): start opening mon connection 2024-09-16T03:56:38.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.138+0000 7fd0067fc640 1 -- 172.21.15.42:0/2104093700 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fcff000c270 con 0x7fcff00053d0 2024-09-16T03:56:38.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.138+0000 7fd0067fc640 10 monclient(hunting): start opening mon connection 2024-09-16T03:56:38.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.138+0000 7fd0067fc640 1 -- 172.21.15.42:0/2104093700 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fcff000c470 con 0x7fcff00039c0 2024-09-16T03:56:38.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.138+0000 7fd0067fc640 10 monclient(hunting): _renew_subs 2024-09-16T03:56:38.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.138+0000 7fd0067fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:56:38.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.138+0000 7fd0077fe640 1 --2- 172.21.15.42:0/2104093700 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fcff00039c0 0x7fcff0003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:56:38.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.138+0000 7fd006ffd640 1 --2- 172.21.15.42:0/2104093700 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcff0004810 0x7fcff0004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:56:38.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.138+0000 7fd0077fe640 10 monclient(hunting): get_auth_request con 0x7fcff00039c0 auth_method 0 2024-09-16T03:56:38.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.138+0000 7fd0077fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:56:38.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.138+0000 7fd0077fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:56:38.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.138+0000 7fd0077fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:56:38.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.138+0000 7fd006ffd640 10 monclient(hunting): get_auth_request con 0x7fcff0004810 auth_method 0 2024-09-16T03:56:38.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.138+0000 7fd006ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:56:38.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.138+0000 7fd006ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:56:38.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.138+0000 7fd006ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:56:38.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.138+0000 7fd007fff640 1 -- 172.21.15.42:0/2104093700 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcff00053d0 msgr2=0x7fcff000b9d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T03:56:38.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.138+0000 7fd007fff640 1 --2- 172.21.15.42:0/2104093700 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcff00053d0 0x7fcff000b9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:56:38.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.138+0000 7fd006ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:56:38.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.138+0000 7fd006ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:56:38.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.138+0000 7fd006ffd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T03:56:38.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.138+0000 7fd0077fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:56:38.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.138+0000 7fd0077fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:56:38.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.138+0000 7fd0077fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T03:56:38.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.138+0000 7fd006ffd640 10 monclient(hunting): handle_auth_done global_id 824778 payload 293 2024-09-16T03:56:38.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.138+0000 7fd006ffd640 1 -- 172.21.15.42:0/2104093700 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fcff00039c0 msgr2=0x7fcff0003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:56:38.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.138+0000 7fd006ffd640 1 --2- 172.21.15.42:0/2104093700 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fcff00039c0 0x7fcff0003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:56:38.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.138+0000 7fd006ffd640 1 -- 172.21.15.42:0/2104093700 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcff00053d0 msgr2=0x7fcff000b9d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:56:38.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.138+0000 7fd006ffd640 1 --2- 172.21.15.42:0/2104093700 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcff00053d0 0x7fcff000b9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:56:38.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.138+0000 7fd006ffd640 10 monclient: _finish_hunting 0 2024-09-16T03:56:38.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.138+0000 7fd006ffd640 1 monclient: found mon.a 2024-09-16T03:56:38.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.138+0000 7fd006ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:56:38.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.138+0000 7fd006ffd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:56:38.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.138+0000 7fd006ffd640 1 -- 172.21.15.42:0/2104093700 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcff0003530 con 0x7fcff0004810 2024-09-16T03:56:38.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.138+0000 7fd007fff640 1 -- 172.21.15.42:0/2104093700 reap_dead start 2024-09-16T03:56:38.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.138+0000 7fd006ffd640 10 monclient: _finish_auth 0 2024-09-16T03:56:38.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.138+0000 7fd006ffd640 10 monclient: _check_auth_tickets 2024-09-16T03:56:38.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.138+0000 7fd006ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:56:38.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.138+0000 7fd00da70640 5 monclient: authenticate success, global_id 824778 2024-09-16T03:56:38.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.138+0000 7fd00da70640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:56:38.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.142+0000 7fd006ffd640 1 --2- 172.21.15.42:0/2104093700 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcff0004810 0x7fcff0004c90 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fcffc003940 tx=0x7fcffc01c420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa079360269ece00 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:56:38.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.142+0000 7fd0067fc640 1 -- 172.21.15.42:0/2104093700 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcffc009520 con 0x7fcff0004810 2024-09-16T03:56:38.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.142+0000 7fd0067fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:56:38.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.142+0000 7fd0067fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:56:38.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.142+0000 7fd0067fc640 10 monclient: dump: 2024-09-16T03:56:38.162 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:56:38.162 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:56:38.162 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:38.162 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:38.162 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:56:38.162 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:56:38.162 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:56:38.162 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:56:38.162 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:56:38.162 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:56:38.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.142+0000 7fd0067fc640 1 -- 172.21.15.42:0/2104093700 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcffc003bc0 con 0x7fcff0004810 2024-09-16T03:56:38.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.142+0000 7fd00da70640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:56:38.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.142+0000 7fd0067fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:56:38.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.142+0000 7fd00da70640 10 monclient: get_monmap_and_config success 2024-09-16T03:56:38.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.142+0000 7fd0067fc640 1 -- 172.21.15.42:0/2104093700 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcffc005070 con 0x7fcff0004810 2024-09-16T03:56:38.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.142+0000 7fd0067fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:56:38.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.142+0000 7fd00da70640 4 set_mon_vals no callback set 2024-09-16T03:56:38.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.142+0000 7fd0067fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:56:38.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.142+0000 7fd0067fc640 10 monclient: dump: 2024-09-16T03:56:38.164 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:56:38.164 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:56:38.164 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:38.164 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:38.165 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:56:38.165 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:56:38.165 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:56:38.165 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:56:38.165 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:56:38.165 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:56:38.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.142+0000 7fd00da70640 10 monclient: shutdown 2024-09-16T03:56:38.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.142+0000 7fd00da70640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:56:38.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.142+0000 7fd00da70640 1 -- 172.21.15.42:0/2104093700 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcff0004810 msgr2=0x7fcff0004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:56:38.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.142+0000 7fd00da70640 1 --2- 172.21.15.42:0/2104093700 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcff0004810 0x7fcff0004c90 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fcffc003940 tx=0x7fcffc01c420 comp rx=0 tx=0).stop 2024-09-16T03:56:38.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.142+0000 7fd00da70640 1 -- 172.21.15.42:0/2104093700 shutdown_connections 2024-09-16T03:56:38.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.142+0000 7fd00da70640 1 --2- 172.21.15.42:0/2104093700 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcff0004810 0x7fcff0004c90 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:56:38.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.142+0000 7fd00da70640 1 -- 172.21.15.42:0/2104093700 >> 172.21.15.42:0/2104093700 conn(0x7fd0080a9fb0 msgr2=0x7fd0080aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:56:38.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.142+0000 7fd00da70640 1 -- 172.21.15.42:0/2104093700 shutdown_connections 2024-09-16T03:56:38.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.142+0000 7fd00da70640 1 -- 172.21.15.42:0/2104093700 wait complete. 2024-09-16T03:56:38.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.142+0000 7fd00da70640 10 monclient: build_initial_monmap 2024-09-16T03:56:38.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.142+0000 7fd00da70640 10 monclient: monmap: 2024-09-16T03:56:38.166 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:56:38.166 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:56:38.166 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:56:38.166 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:56:38.166 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:56:38.166 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:56:38.166 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:56:38.166 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:56:38.166 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:56:38.167 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:56:38.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.142+0000 7fd00da70640 1 librados: starting msgr at 2024-09-16T03:56:38.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.142+0000 7fd00da70640 1 librados: starting objecter 2024-09-16T03:56:38.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.142+0000 7fd00da70640 1 Processor -- start 2024-09-16T03:56:38.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.142+0000 7fd00da70640 1 -- start start 2024-09-16T03:56:38.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.142+0000 7fd00da70640 1 librados: setting wanted keys 2024-09-16T03:56:38.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.142+0000 7fd00da70640 1 librados: calling monclient init 2024-09-16T03:56:38.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.142+0000 7fd00da70640 10 monclient: init 2024-09-16T03:56:38.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.146+0000 7fd00da70640 10 monclient: _reopen_session rank -1 2024-09-16T03:56:38.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.146+0000 7fd00da70640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T03:56:38.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.146+0000 7fd00da70640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcff0004810 0x7fd0080a0270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:56:38.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.146+0000 7fd00da70640 10 monclient(hunting): picked mon.noname-b con 0x7fcff0004810 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:56:38.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.146+0000 7fd00da70640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd0080a07b0 0x7fd0080a0c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:56:38.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.146+0000 7fd00da70640 10 monclient(hunting): picked mon.noname-c con 0x7fd0080a07b0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:56:38.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.146+0000 7fd00da70640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd0080a1150 0x7fd0081d5ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:56:38.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.146+0000 7fd00da70640 10 monclient(hunting): picked mon.noname-a con 0x7fd0080a1150 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:56:38.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.146+0000 7fd00da70640 10 monclient(hunting): start opening mon connection 2024-09-16T03:56:38.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.146+0000 7fd00da70640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fd008137ba0 con 0x7fd0080a1150 2024-09-16T03:56:38.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.146+0000 7fd00da70640 10 monclient(hunting): start opening mon connection 2024-09-16T03:56:38.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.146+0000 7fd00da70640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd008137a20 con 0x7fcff0004810 2024-09-16T03:56:38.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.146+0000 7fd00da70640 10 monclient(hunting): start opening mon connection 2024-09-16T03:56:38.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.146+0000 7fd00da70640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fd008137d20 con 0x7fd0080a07b0 2024-09-16T03:56:38.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.146+0000 7fd00da70640 10 monclient(hunting): _renew_subs 2024-09-16T03:56:38.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.146+0000 7fd00da70640 10 monclient(hunting): authenticate will time out at 1845.116342s 2024-09-16T03:56:38.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.146+0000 7fd007fff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd0080a1150 0x7fd0081d5ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:56:38.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.146+0000 7fd006ffd640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd0080a07b0 0x7fd0080a0c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:56:38.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.146+0000 7fd0077fe640 1 -- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcff0004810 msgr2=0x7fd0080a0270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T03:56:38.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.146+0000 7fd007fff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd0080a1150 0x7fd0081d5ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:44748/0 (socket says 172.21.15.42:44748) 2024-09-16T03:56:38.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.146+0000 7fd007fff640 1 -- 172.21.15.42:0/4176662321 learned_addr learned my addr 172.21.15.42:0/4176662321 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:56:38.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.146+0000 7fd007fff640 10 monclient(hunting): get_auth_request con 0x7fd0080a1150 auth_method 0 2024-09-16T03:56:38.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.146+0000 7fd007fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:56:38.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.146+0000 7fd007fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:56:38.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.146+0000 7fd007fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:56:38.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.146+0000 7fd0077fe640 1 --2- 172.21.15.42:0/4176662321 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcff0004810 0x7fd0080a0270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:56:38.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.146+0000 7fd006ffd640 10 monclient(hunting): get_auth_request con 0x7fd0080a07b0 auth_method 0 2024-09-16T03:56:38.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.146+0000 7fd006ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:56:38.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.146+0000 7fd006ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:56:38.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.146+0000 7fd006ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:56:38.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.146+0000 7fd007fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:56:38.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.146+0000 7fd007fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:56:38.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.146+0000 7fd007fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:56:38.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.146+0000 7fd006ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:56:38.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.146+0000 7fd006ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:56:38.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.146+0000 7fd006ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:56:38.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.146+0000 7fd007fff640 10 monclient(hunting): handle_auth_done global_id 824781 payload 824 2024-09-16T03:56:38.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.146+0000 7fd007fff640 1 -- 172.21.15.42:0/4176662321 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd0080a07b0 msgr2=0x7fd0080a0c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:56:38.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.146+0000 7fd007fff640 1 --2- 172.21.15.42:0/4176662321 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd0080a07b0 0x7fd0080a0c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:56:38.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.146+0000 7fd007fff640 1 -- 172.21.15.42:0/4176662321 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcff0004810 msgr2=0x7fd0080a0270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:56:38.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.146+0000 7fd007fff640 1 --2- 172.21.15.42:0/4176662321 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcff0004810 0x7fd0080a0270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:56:38.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.146+0000 7fd007fff640 10 monclient: _finish_hunting 0 2024-09-16T03:56:38.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.146+0000 7fd007fff640 1 monclient: found mon.noname-a 2024-09-16T03:56:38.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.146+0000 7fd007fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:56:38.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.146+0000 7fd007fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:56:38.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.146+0000 7fd007fff640 1 -- 172.21.15.42:0/4176662321 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0080a2150 con 0x7fd0080a1150 2024-09-16T03:56:38.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.146+0000 7fd006ffd640 1 --2- 172.21.15.42:0/4176662321 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd0080a07b0 0x7fd0080a0c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:56:38.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.146+0000 7fd007fff640 1 --2- 172.21.15.42:0/4176662321 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd0080a1150 0x7fd0081d5ac0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fd00806e7a0 tx=0x7fcff800b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:56:38.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.146+0000 7fcfeffff640 10 client.?.objecter ms_handle_connect 0x7fd0080a1150 2024-09-16T03:56:38.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.146+0000 7fcfeffff640 10 client.?.objecter resend_mon_ops 2024-09-16T03:56:38.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.146+0000 7fcfeffff640 1 -- 172.21.15.42:0/4176662321 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcff8003dd0 con 0x7fd0080a1150 2024-09-16T03:56:38.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.146+0000 7fcfeffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:56:38.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.146+0000 7fcfeffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:56:38.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.146+0000 7fcfeffff640 10 monclient: dump: 2024-09-16T03:56:38.171 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:56:38.172 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:56:38.172 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:38.172 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:38.172 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:56:38.172 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:56:38.172 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:56:38.172 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:56:38.172 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:56:38.172 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:56:38.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.146+0000 7fcfeffff640 10 monclient: _finish_auth 0 2024-09-16T03:56:38.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.146+0000 7fcfeffff640 10 monclient: _check_auth_tickets 2024-09-16T03:56:38.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.146+0000 7fcfeffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:56:38.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.146+0000 7fd00da70640 5 monclient: authenticate success, global_id 824781 2024-09-16T03:56:38.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.146+0000 7fd00da70640 10 monclient: _renew_subs 2024-09-16T03:56:38.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.146+0000 7fcfeffff640 1 -- 172.21.15.42:0/4176662321 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcff80060d0 con 0x7fd0080a1150 2024-09-16T03:56:38.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.146+0000 7fd00da70640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:56:38.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.146+0000 7fd00da70640 1 -- 172.21.15.42:0/4176662321 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0081207f0 con 0x7fd0080a1150 2024-09-16T03:56:38.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.146+0000 7fcfeffff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:56:38.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.146+0000 7fcfeffff640 1 -- 172.21.15.42:0/4176662321 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcff80064c0 con 0x7fd0080a1150 2024-09-16T03:56:38.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.146+0000 7fcfeffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:56:38.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.146+0000 7fcfeffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:56:38.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.146+0000 7fcfeffff640 10 monclient: dump: 2024-09-16T03:56:38.173 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:56:38.173 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:56:38.173 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:38.173 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:38.174 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:56:38.174 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:56:38.175 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:56:38.175 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:56:38.175 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:56:38.175 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:56:38.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.146+0000 7fd00da70640 10 client.824781.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:56:38.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.146+0000 7fd005ffb640 4 set_mon_vals no callback set 2024-09-16T03:56:38.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.146+0000 7fd00da70640 10 monclient: _renew_subs 2024-09-16T03:56:38.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.146+0000 7fd00da70640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:56:38.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.146+0000 7fd00da70640 1 -- 172.21.15.42:0/4176662321 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0080a9440 con 0x7fd0080a1150 2024-09-16T03:56:38.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.146+0000 7fd00da70640 1 librados: init done 2024-09-16T03:56:38.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.146+0000 7fd00da70640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:56:38.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.146+0000 7fd00da70640 1 --2- 172.21.15.42:0/4176662321 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd00809c8d0 0x7fd00809ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:56:38.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.146+0000 7fd00da70640 10 monclient(hunting): start opening mon connection 2024-09-16T03:56:38.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.146+0000 7fd00da70640 1 -- 172.21.15.42:0/4176662321 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fd008134230 con 0x7fd00809c8d0 2024-09-16T03:56:38.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.146+0000 7fd0077fe640 1 --2- 172.21.15.42:0/4176662321 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd00809c8d0 0x7fd00809ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:56:38.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.146+0000 7fd0077fe640 10 monclient: get_auth_request con 0x7fd00809c8d0 auth_method 0 2024-09-16T03:56:38.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.146+0000 7fd0077fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:56:38.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.146+0000 7fd0077fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:56:38.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.146+0000 7fd0077fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:56:38.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.146+0000 7fcfeffff640 1 -- 172.21.15.42:0/4176662321 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fcff8010070 con 0x7fd0080a1150 2024-09-16T03:56:38.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.146+0000 7fd0077fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:56:38.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.146+0000 7fd0077fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:56:38.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.146+0000 7fd0077fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:56:38.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.146+0000 7fcfeffff640 1 --2- 172.21.15.42:0/4176662321 >> v2:172.21.15.154:6812/254845342 conn(0x7fcff40377c0 0x7fcff4039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:56:38.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.146+0000 7fd0077fe640 10 monclient(hunting): handle_auth_done global_id 824784 payload 824 2024-09-16T03:56:38.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.146+0000 7fd0077fe640 1 -- 172.21.15.42:0/4176662321 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd0080a5750 con 0x7fd00809c8d0 2024-09-16T03:56:38.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.146+0000 7fcfeffff640 1 -- 172.21.15.42:0/4176662321 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(443..443 src has 330..443) ==== 5356+0+0 (secure 0 0 0) 0x7fcff8048370 con 0x7fd0080a1150 2024-09-16T03:56:38.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.146+0000 7fcfeffff640 10 client.824781.objecter ms_dispatch 0x7fd008000ba0 osd_map(443..443 src has 330..443) 2024-09-16T03:56:38.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.146+0000 7fcfeffff640 3 client.824781.objecter handle_osd_map got epochs [443,443] > 0 2024-09-16T03:56:38.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.146+0000 7fcfeffff640 3 client.824781.objecter handle_osd_map decoding full epoch 443 2024-09-16T03:56:38.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.146+0000 7fcfeffff640 20 client.824781.objecter dump_active .. 0 homeless 2024-09-16T03:56:38.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.146+0000 7fcfeffff640 10 client.824781.objecter ms_handle_connect 0x7fd00809c8d0 2024-09-16T03:56:38.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.146+0000 7fcfeffff640 10 client.824781.objecter resend_mon_ops 2024-09-16T03:56:38.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.146+0000 7fd0077fe640 1 --2- 172.21.15.42:0/4176662321 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd00809c8d0 0x7fd00809ccb0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fd0080a4270 tx=0x7fd000007a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:56:38.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.146+0000 7fcfeffff640 1 -- 172.21.15.42:0/4176662321 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd000017020 con 0x7fd00809c8d0 2024-09-16T03:56:38.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.146+0000 7fcfeffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:56:38.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.146+0000 7fcfeffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:56:38.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.146+0000 7fcfeffff640 10 monclient: dump: 2024-09-16T03:56:38.177 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:56:38.177 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:56:38.178 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:38.178 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:38.178 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:56:38.178 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:56:38.178 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:56:38.178 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:56:38.178 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:56:38.178 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:56:38.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.146+0000 7fd006ffd640 1 --2- 172.21.15.42:0/4176662321 >> v2:172.21.15.154:6812/254845342 conn(0x7fcff40377c0 0x7fcff4039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:56:38.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.146+0000 7fd006ffd640 10 monclient: get_auth_request con 0x7fcff40377c0 auth_method 0 2024-09-16T03:56:38.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.146+0000 7fcfeffff640 1 -- 172.21.15.42:0/4176662321 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fd00000aaf0 con 0x7fd00809c8d0 2024-09-16T03:56:38.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.146+0000 7fcfeffff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:56:38.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.146+0000 7fcfeffff640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:56:38.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.146+0000 7fcfeffff640 1 -- 172.21.15.42:0/4176662321 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd00809c8d0 msgr2=0x7fd00809ccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:56:38.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.146+0000 7fcfeffff640 1 --2- 172.21.15.42:0/4176662321 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd00809c8d0 0x7fd00809ccb0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fd0080a4270 tx=0x7fd000007a20 comp rx=0 tx=0).stop 2024-09-16T03:56:38.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.146+0000 7fcfeffff640 1 -- 172.21.15.42:0/4176662321 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd00809c8d0 msgr2=0x7fd00809ccb0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:56:38.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.146+0000 7fcfeffff640 1 --2- 172.21.15.42:0/4176662321 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd00809c8d0 0x7fd00809ccb0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:56:38.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.146+0000 7fd006ffd640 1 --2- 172.21.15.42:0/4176662321 >> v2:172.21.15.154:6812/254845342 conn(0x7fcff40377c0 0x7fcff4039c80 secure :-1 s=READY pgs=3662 cs=0 l=1 rev1=1 crypto rx=0x7fcffc0aa000 tx=0x7fcffc005a90 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:56:38.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.146+0000 7fcfeffff640 10 client.824781.objecter ms_handle_connect 0x7fcff40377c0 2024-09-16T03:56:38.181 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.154+0000 7fd00da70640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:56:38.181 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.154+0000 7fd00da70640 1 --2- 172.21.15.42:0/4176662321 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd00809d1f0 0x7fd0081da020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:56:38.181 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.154+0000 7fd00da70640 10 monclient(hunting): start opening mon connection 2024-09-16T03:56:38.181 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.154+0000 7fd00da70640 1 -- 172.21.15.42:0/4176662321 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fd008068e40 con 0x7fd00809d1f0 2024-09-16T03:56:38.181 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.154+0000 7fd0077fe640 1 --2- 172.21.15.42:0/4176662321 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd00809d1f0 0x7fd0081da020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:56:38.181 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.154+0000 7fd0077fe640 10 monclient: get_auth_request con 0x7fd00809d1f0 auth_method 0 2024-09-16T03:56:38.181 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.154+0000 7fd0077fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:56:38.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.154+0000 7fd0077fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:56:38.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.154+0000 7fd0077fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:56:38.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.154+0000 7fd0077fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:56:38.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.154+0000 7fd0077fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:56:38.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.154+0000 7fd0077fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:56:38.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.154+0000 7fd0077fe640 10 monclient(hunting): handle_auth_done global_id 824787 payload 824 2024-09-16T03:56:38.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.154+0000 7fd0077fe640 1 -- 172.21.15.42:0/4176662321 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fd0080a3700 con 0x7fd00809d1f0 2024-09-16T03:56:38.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.154+0000 7fcfeffff640 10 client.824781.objecter ms_handle_connect 0x7fd00809d1f0 2024-09-16T03:56:38.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.154+0000 7fcfeffff640 10 client.824781.objecter resend_mon_ops 2024-09-16T03:56:38.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.154+0000 7fd0077fe640 1 --2- 172.21.15.42:0/4176662321 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd00809d1f0 0x7fd0081da020 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fd0080a4270 tx=0x7fd000001e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:56:38.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.154+0000 7fcfeffff640 1 -- 172.21.15.42:0/4176662321 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd000017050 con 0x7fd00809d1f0 2024-09-16T03:56:38.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.154+0000 7fcfeffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:56:38.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.154+0000 7fcfeffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:56:38.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.154+0000 7fcfeffff640 10 monclient: dump: 2024-09-16T03:56:38.183 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:56:38.183 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:56:38.183 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:38.183 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:38.183 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:56:38.183 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:56:38.183 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:56:38.183 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:56:38.183 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:56:38.183 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:56:38.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.154+0000 7fcfeffff640 1 -- 172.21.15.42:0/4176662321 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4871 (secure 0 0 0) 0x7fd000013a20 con 0x7fd00809d1f0 2024-09-16T03:56:38.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.154+0000 7fcfeffff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:56:38.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.154+0000 7fcfeffff640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:56:38.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.154+0000 7fcfeffff640 1 -- 172.21.15.42:0/4176662321 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd00809d1f0 msgr2=0x7fd0081da020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:56:38.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.154+0000 7fcfeffff640 1 --2- 172.21.15.42:0/4176662321 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd00809d1f0 0x7fd0081da020 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fd0080a4270 tx=0x7fd000001e00 comp rx=0 tx=0).stop 2024-09-16T03:56:38.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.154+0000 7fcfeffff640 1 -- 172.21.15.42:0/4176662321 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd00809d1f0 msgr2=0x7fd0081da020 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:56:38.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.154+0000 7fcfeffff640 1 --2- 172.21.15.42:0/4176662321 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd00809d1f0 0x7fd0081da020 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:56:38.184 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:56:38.184 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:56:38.184 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:56:38.184 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "leader", 2024-09-16T03:56:38.184 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 132, 2024-09-16T03:56:38.184 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 288009, 2024-09-16T03:56:38.185 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:56:38.185 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:56:38.185 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T03:56:38.185 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:56:38.185 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 1, 2024-09-16T03:56:38.185 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:56:38.185 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:56:38.185 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:56:38.185 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:56:38.185 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:56:38.185 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:56:38.185 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:56:38.185 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:56:38.185 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:56:38.186 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:56:38.186 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:56:38.186 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:56:38.186 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:56:38.186 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:56:38.186 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:56:38.186 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:56:38.186 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:56:38.186 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:56:38.186 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:56:38.186 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:56:38.186 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:56:38.186 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:56:38.186 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:56:38.187 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:56:38.187 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:56:38.187 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:56:38.187 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:56:38.187 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:56:38.187 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:56:38.187 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:56:38.187 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:56:38.187 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:56:38.187 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:56:38.187 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:56:38.187 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:56:38.187 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:56:38.187 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:56:38.188 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:56:38.188 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:56:38.188 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:56:38.188 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:56:38.188 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:56:38.188 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:56:38.188 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:56:38.188 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:56:38.188 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:56:38.188 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:56:38.188 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:56:38.188 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:56:38.188 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:56:38.188 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:56:38.189 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:56:38.189 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:56:38.189 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:56:38.189 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:56:38.189 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:56:38.189 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:56:38.189 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:56:38.189 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:56:38.189 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:56:38.189 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:56:38.189 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:56:38.189 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:56:38.189 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:56:38.189 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:56:38.190 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:56:38.190 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:56:38.190 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:56:38.190 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:56:38.190 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:56:38.190 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:56:38.190 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:56:38.190 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:56:38.190 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:56:38.190 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:56:38.190 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:56:38.190 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:56:38.190 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:56:38.190 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:56:38.190 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:56:38.191 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:56:38.191 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:56:38.191 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:56:38.191 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:56:38.191 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:56:38.191 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:56:38.191 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:56:38.191 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:56:38.191 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:56:38.191 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:56:38.191 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:56:38.191 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:56:38.191 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:56:38.191 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:56:38.192 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:56:38.192 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:56:38.192 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:56:38.192 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:56:38.192 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:56:38.192 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:56:38.192 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:56:38.192 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:56:38.192 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:56:38.192 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:56:38.192 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:56:38.192 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:56:38.192 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:56:38.193 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:56:38.193 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:56:38.193 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:56:38.193 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:56:38.193 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:56:38.193 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:56:38.193 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:56:38.193 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:56:38.193 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:56:38.193 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:56:38.193 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:56:38.193 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:56:38.193 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:56:38.193 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:56:38.193 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:56:38.194 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:56:38.194 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:56:38.194 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:56:38.194 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:56:38.194 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:56:38.194 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:56:38.194 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:56:38.194 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:56:38.194 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:56:38.194 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:56:38.194 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:56:38.194 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:56:38.194 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:56:38.194 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:56:38.195 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:56:38.195 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:56:38.195 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:56:38.195 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:56:38.195 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:56:38.195 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T03:56:38.195 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:56:38.195 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:56:38.195 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:56:38.195 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 6 2024-09-16T03:56:38.195 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:56:38.195 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:56:38.195 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:56:38.195 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:56:38.196 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:56:38.196 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:56:38.196 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 8 2024-09-16T03:56:38.196 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:56:38.196 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:56:38.196 INFO:teuthology.orchestra.run.smithi042.stdout: "mgr": [ 2024-09-16T03:56:38.196 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:56:38.196 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:56:38.196 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:56:38.196 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:56:38.196 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:56:38.196 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:56:38.196 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:56:38.196 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:56:38.197 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:56:38.198 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.154+0000 7fd00da70640 10 librados: watch_flush enter 2024-09-16T03:56:38.199 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.154+0000 7fd00da70640 10 librados: watch_flush exit 2024-09-16T03:56:38.199 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.154+0000 7fd00da70640 20 client.824781.objecter shutdown clearing up homeless session... 2024-09-16T03:56:38.199 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.154+0000 7fd00da70640 10 client.824781.objecter successfully canceled tick 2024-09-16T03:56:38.199 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.154+0000 7fd00da70640 1 -- 172.21.15.42:0/4176662321 >> v2:172.21.15.154:6812/254845342 conn(0x7fcff40377c0 msgr2=0x7fcff4039c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:56:38.199 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.154+0000 7fd00da70640 1 --2- 172.21.15.42:0/4176662321 >> v2:172.21.15.154:6812/254845342 conn(0x7fcff40377c0 0x7fcff4039c80 secure :-1 s=READY pgs=3662 cs=0 l=1 rev1=1 crypto rx=0x7fcffc0aa000 tx=0x7fcffc005a90 comp rx=0 tx=0).stop 2024-09-16T03:56:38.199 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.154+0000 7fd00da70640 10 monclient: shutdown 2024-09-16T03:56:38.199 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.154+0000 7fd00da70640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:56:38.199 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.154+0000 7fd00da70640 1 -- 172.21.15.42:0/4176662321 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd0080a1150 msgr2=0x7fd0081d5ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:56:38.199 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.154+0000 7fd007fff640 1 -- 172.21.15.42:0/4176662321 reap_dead start 2024-09-16T03:56:38.199 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.154+0000 7fd00da70640 1 --2- 172.21.15.42:0/4176662321 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd0080a1150 0x7fd0081d5ac0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fd00806e7a0 tx=0x7fcff800b820 comp rx=0 tx=0).stop 2024-09-16T03:56:38.199 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.154+0000 7fd00da70640 1 -- 172.21.15.42:0/4176662321 shutdown_connections 2024-09-16T03:56:38.199 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.154+0000 7fd00da70640 1 --2- 172.21.15.42:0/4176662321 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd0080a1150 0x7fd0081d5ac0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:56:38.199 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.154+0000 7fd00da70640 1 -- 172.21.15.42:0/4176662321 >> 172.21.15.42:0/4176662321 conn(0x7fd0080a9fb0 msgr2=0x7fd00813bb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:56:38.200 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.154+0000 7fd00da70640 1 -- 172.21.15.42:0/4176662321 shutdown_connections 2024-09-16T03:56:38.200 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.154+0000 7fd00da70640 1 -- 172.21.15.42:0/4176662321 wait complete. 2024-09-16T03:56:38.200 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.154+0000 7fd00da70640 1 librados: shutdown 2024-09-16T03:56:38.200 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T03:56:38.200 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-16T03:56:38.200 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T03:56:38.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.314+0000 7fb6c3e81640 10 monclient: get_monmap_and_config 2024-09-16T03:56:38.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.314+0000 7fb6c3e81640 10 monclient: build_initial_monmap 2024-09-16T03:56:38.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.314+0000 7fb6c3e81640 10 monclient: monmap: 2024-09-16T03:56:38.318 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:56:38.318 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:56:38.318 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:56:38.316479+0000 2024-09-16T03:56:38.318 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:56:38.316479+0000 2024-09-16T03:56:38.318 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:56:38.318 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:56:38.318 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:56:38.318 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:56:38.318 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:56:38.318 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:56:38.318 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:56:38.319 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:56:38.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.314+0000 7fb6c3e81640 1 Processor -- start 2024-09-16T03:56:38.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.314+0000 7fb6c3e81640 1 -- start start 2024-09-16T03:56:38.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.314+0000 7fb6c3e81640 10 monclient: init 2024-09-16T03:56:38.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.314+0000 7fb6c3e81640 10 monclient: _reopen_session rank -1 2024-09-16T03:56:38.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.314+0000 7fb6c3e81640 10 monclient: _add_conns ranks=[4,1,0,3,2] 2024-09-16T03:56:38.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.314+0000 7fb6c3e81640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fb6bc09fb00 0x7fb6bc09ba50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:56:38.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.314+0000 7fb6c3e81640 10 monclient(hunting): picked mon.noname-c con 0x7fb6bc09fb00 addr v2:172.21.15.154:3300/0 2024-09-16T03:56:38.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.314+0000 7fb6c3e81640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb6bc09bf90 addr v1:172.21.15.154:6789/0 2024-09-16T03:56:38.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.314+0000 7fb6c3e81640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb6bc09c400 addr v1:172.21.15.42:6789/0 2024-09-16T03:56:38.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.314+0000 7fb6c3e81640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb6bc0a1e20 con 0x7fb6bc09c400 2024-09-16T03:56:38.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.314+0000 7fb6c3e81640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb6bc09c850 con 0x7fb6bc09bf90 2024-09-16T03:56:38.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.314+0000 7fb6c3e81640 10 monclient(hunting): start opening mon connection 2024-09-16T03:56:38.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.314+0000 7fb6c3e81640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fb6bc09e250 con 0x7fb6bc09fb00 2024-09-16T03:56:38.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.314+0000 7fb6c3e81640 10 monclient(hunting): _renew_subs 2024-09-16T03:56:38.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.314+0000 7fb6c3e81640 10 monclient(hunting): authenticate will time out at 1845.285847s 2024-09-16T03:56:38.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.314+0000 7fb6c1c1d640 1 -- >> v2:172.21.15.154:3300/0 conn(0x7fb6bc09fb00 msgr2=0x7fb6bc09ba50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T03:56:38.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.314+0000 7fb6c1c1d640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fb6bc09fb00 0x7fb6bc09ba50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:56:38.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.314+0000 7fb6c141c640 1 -- >> v1:172.21.15.154:6789/0 conn(0x7fb6bc09bf90 legacy=0x7fb6bc138110 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.154:6789/0 2024-09-16T03:56:38.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.314+0000 7fb6c241e640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fb6bc09c400 0x7fb6bc13b840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:48012/0 (socket says 172.21.15.42:48012) 2024-09-16T03:56:38.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.314+0000 7fb6c241e640 1 -- 172.21.15.42:0/2745751120 learned_addr learned my addr 172.21.15.42:0/2745751120 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:56:38.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.314+0000 7fb6c0c1b640 1 -- 172.21.15.42:0/2745751120 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2697485166 0 0) 0x7fb6bc0a1e20 con 0x7fb6bc09c400 2024-09-16T03:56:38.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.314+0000 7fb6c0c1b640 10 monclient(hunting): _negotiate 2024-09-16T03:56:38.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.314+0000 7fb6c0c1b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:56:38.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.314+0000 7fb6c0c1b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:56:38.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.314+0000 7fb6c0c1b640 10 monclient(hunting): my global_id is 824790 2024-09-16T03:56:38.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.314+0000 7fb6c0c1b640 1 -- 172.21.15.42:0/2745751120 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb6a8003530 con 0x7fb6bc09c400 2024-09-16T03:56:38.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.314+0000 7fb6c0c1b640 1 -- 172.21.15.42:0/2745751120 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3159073977 0 0) 0x7fb6a8003530 con 0x7fb6bc09c400 2024-09-16T03:56:38.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.314+0000 7fb6c0c1b640 1 -- 172.21.15.42:0/2745751120 >> v2:172.21.15.154:3300/0 conn(0x7fb6bc09fb00 msgr2=0x7fb6bc09ba50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:56:38.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.314+0000 7fb6c0c1b640 1 --2- 172.21.15.42:0/2745751120 >> v2:172.21.15.154:3300/0 conn(0x7fb6bc09fb00 0x7fb6bc09ba50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:56:38.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.314+0000 7fb6c0c1b640 1 -- 172.21.15.42:0/2745751120 >> v1:172.21.15.154:6789/0 conn(0x7fb6bc09bf90 legacy=0x7fb6bc138110 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:56:38.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.314+0000 7fb6c0c1b640 10 monclient: _finish_hunting 0 2024-09-16T03:56:38.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.314+0000 7fb6c0c1b640 1 monclient: found mon.noname-a-legacy 2024-09-16T03:56:38.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.314+0000 7fb6c0c1b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:56:38.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.314+0000 7fb6c0c1b640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.42:6789/0 2024-09-16T03:56:38.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.314+0000 7fb6c0c1b640 1 -- 172.21.15.42:0/2745751120 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6bc09ea30 con 0x7fb6bc09c400 2024-09-16T03:56:38.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.314+0000 7fb6c0c1b640 10 monclient: _finish_auth 0 2024-09-16T03:56:38.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.314+0000 7fb6c0c1b640 10 monclient: _check_auth_tickets 2024-09-16T03:56:38.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.314+0000 7fb6c0c1b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:56:38.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.314+0000 7fb6c0c1b640 1 -- 172.21.15.42:0/2745751120 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2177413734 0 0) 0x7fb6b4001680 con 0x7fb6bc09c400 2024-09-16T03:56:38.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.314+0000 7fb6c3e81640 5 monclient: authenticate success, global_id 824790 2024-09-16T03:56:38.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.314+0000 7fb6c3e81640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:56:38.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.314+0000 7fb6c0c1b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:56:38.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.314+0000 7fb6c0c1b640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T03:56:38.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.314+0000 7fb6c0c1b640 10 monclient: dump: 2024-09-16T03:56:38.336 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:56:38.336 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:56:38.336 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:38.336 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:38.336 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:56:38.336 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:56:38.336 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:56:38.336 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:56:38.336 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:56:38.336 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:56:38.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.314+0000 7fb6c0c1b640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] but i'm connected to v1:172.21.15.42:6789/0, reconnecting 2024-09-16T03:56:38.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.314+0000 7fb6c0c1b640 10 monclient: _reopen_session rank -1 2024-09-16T03:56:38.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.314+0000 7fb6c0c1b640 1 -- 172.21.15.42:0/2745751120 >> v1:172.21.15.42:6789/0 conn(0x7fb6bc09c400 legacy=0x7fb6bc13b840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:56:38.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.314+0000 7fb6c0c1b640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T03:56:38.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.314+0000 7fb6c0c1b640 1 --2- 172.21.15.42:0/2745751120 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb6a80039c0 0x7fb6a8003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:56:38.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.314+0000 7fb6c0c1b640 10 monclient(hunting): picked mon.a con 0x7fb6a80039c0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:56:38.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.314+0000 7fb6c0c1b640 1 --2- 172.21.15.42:0/2745751120 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb6a8004810 0x7fb6a8004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:56:38.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.314+0000 7fb6c0c1b640 10 monclient(hunting): picked mon.b con 0x7fb6a8004810 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:56:38.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.314+0000 7fb6c0c1b640 1 --2- 172.21.15.42:0/2745751120 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb6a80053d0 0x7fb6a800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:56:38.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.314+0000 7fb6c0c1b640 10 monclient(hunting): picked mon.c con 0x7fb6a80053d0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:56:38.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.314+0000 7fb6c0c1b640 10 monclient(hunting): start opening mon connection 2024-09-16T03:56:38.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.314+0000 7fb6c0c1b640 1 -- 172.21.15.42:0/2745751120 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fb6bc09e250 con 0x7fb6a80039c0 2024-09-16T03:56:38.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.314+0000 7fb6c0c1b640 10 monclient(hunting): start opening mon connection 2024-09-16T03:56:38.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.314+0000 7fb6c0c1b640 1 -- 172.21.15.42:0/2745751120 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fb6a800c270 con 0x7fb6a8004810 2024-09-16T03:56:38.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.314+0000 7fb6c0c1b640 10 monclient(hunting): start opening mon connection 2024-09-16T03:56:38.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.314+0000 7fb6c0c1b640 1 -- 172.21.15.42:0/2745751120 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fb6a800c470 con 0x7fb6a80053d0 2024-09-16T03:56:38.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.314+0000 7fb6c0c1b640 10 monclient(hunting): _renew_subs 2024-09-16T03:56:38.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.314+0000 7fb6c0c1b640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:56:38.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.314+0000 7fb6c3e81640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-16T03:56:38.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.314+0000 7fb6c1c1d640 1 --2- 172.21.15.42:0/2745751120 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb6a80039c0 0x7fb6a8003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:56:38.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.314+0000 7fb6c1c1d640 10 monclient(hunting): get_auth_request con 0x7fb6a80039c0 auth_method 0 2024-09-16T03:56:38.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.314+0000 7fb6c1c1d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:56:38.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.314+0000 7fb6c141c640 1 -- 172.21.15.42:0/2745751120 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb6a8004810 msgr2=0x7fb6a8004c90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T03:56:38.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.314+0000 7fb6c1c1d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:56:38.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.314+0000 7fb6c1c1d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:56:38.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.314+0000 7fb6c141c640 1 --2- 172.21.15.42:0/2745751120 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb6a8004810 0x7fb6a8004c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:56:38.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.314+0000 7fb6c241e640 1 --2- 172.21.15.42:0/2745751120 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb6a80053d0 0x7fb6a800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:56:38.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.314+0000 7fb6c241e640 10 monclient(hunting): get_auth_request con 0x7fb6a80053d0 auth_method 0 2024-09-16T03:56:38.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.314+0000 7fb6c241e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:56:38.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.314+0000 7fb6c241e640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:56:38.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.314+0000 7fb6c241e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:56:38.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.314+0000 7fb6c1c1d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:56:38.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.314+0000 7fb6c1c1d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:56:38.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.314+0000 7fb6c1c1d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T03:56:38.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.314+0000 7fb6c241e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:56:38.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.314+0000 7fb6c241e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:56:38.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.314+0000 7fb6c241e640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T03:56:38.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.314+0000 7fb6c1c1d640 10 monclient(hunting): handle_auth_done global_id 824790 payload 293 2024-09-16T03:56:38.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.314+0000 7fb6c1c1d640 1 -- 172.21.15.42:0/2745751120 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb6a80053d0 msgr2=0x7fb6a800b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:56:38.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.314+0000 7fb6c1c1d640 1 --2- 172.21.15.42:0/2745751120 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb6a80053d0 0x7fb6a800b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:56:38.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.314+0000 7fb6c1c1d640 1 -- 172.21.15.42:0/2745751120 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb6a8004810 msgr2=0x7fb6a8004c90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:56:38.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.314+0000 7fb6c1c1d640 1 --2- 172.21.15.42:0/2745751120 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb6a8004810 0x7fb6a8004c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:56:38.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.314+0000 7fb6c1c1d640 10 monclient: _finish_hunting 0 2024-09-16T03:56:38.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.314+0000 7fb6c1c1d640 1 monclient: found mon.a 2024-09-16T03:56:38.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.314+0000 7fb6c1c1d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:56:38.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.314+0000 7fb6c1c1d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:56:38.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.314+0000 7fb6c1c1d640 1 -- 172.21.15.42:0/2745751120 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6a8003530 con 0x7fb6a80039c0 2024-09-16T03:56:38.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.314+0000 7fb6c1c1d640 10 monclient: _finish_auth 0 2024-09-16T03:56:38.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.314+0000 7fb6c1c1d640 10 monclient: _check_auth_tickets 2024-09-16T03:56:38.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.314+0000 7fb6c1c1d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:56:38.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.314+0000 7fb6c241e640 1 --2- 172.21.15.42:0/2745751120 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb6a80053d0 0x7fb6a800b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:56:38.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.314+0000 7fb6c241e640 1 -- 172.21.15.42:0/2745751120 reap_dead start 2024-09-16T03:56:38.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.318+0000 7fb6c1c1d640 1 --2- 172.21.15.42:0/2745751120 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb6a80039c0 0x7fb6a8003dc0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fb6ac003940 tx=0x7fb6ac01c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b5bc4201de41315 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:56:38.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.318+0000 7fb6c0c1b640 1 -- 172.21.15.42:0/2745751120 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6ac009520 con 0x7fb6a80039c0 2024-09-16T03:56:38.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.318+0000 7fb6c0c1b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:56:38.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.318+0000 7fb6c0c1b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:56:38.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.318+0000 7fb6c0c1b640 10 monclient: dump: 2024-09-16T03:56:38.341 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:56:38.341 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:56:38.341 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:38.341 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:38.341 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:56:38.341 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:56:38.341 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:56:38.342 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:56:38.342 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:56:38.342 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:56:38.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.318+0000 7fb6c3e81640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:56:38.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.318+0000 7fb6c0c1b640 1 -- 172.21.15.42:0/2745751120 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb6ac003bc0 con 0x7fb6a80039c0 2024-09-16T03:56:38.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.318+0000 7fb6c0c1b640 10 monclient: handle_config config(0 keys) 2024-09-16T03:56:38.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.318+0000 7fb6c3e81640 10 monclient: get_monmap_and_config success 2024-09-16T03:56:38.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.318+0000 7fb6c3e81640 4 set_mon_vals no callback set 2024-09-16T03:56:38.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.318+0000 7fb6c0c1b640 1 -- 172.21.15.42:0/2745751120 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6ac005070 con 0x7fb6a80039c0 2024-09-16T03:56:38.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.318+0000 7fb6c0c1b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:56:38.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.318+0000 7fb6c0c1b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:56:38.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.318+0000 7fb6c0c1b640 10 monclient: dump: 2024-09-16T03:56:38.342 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:56:38.343 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:56:38.343 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:38.343 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:38.343 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:56:38.343 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:56:38.343 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:56:38.343 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:56:38.343 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:56:38.343 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:56:38.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.318+0000 7fb6c3e81640 10 monclient: shutdown 2024-09-16T03:56:38.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.318+0000 7fb6c3e81640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:56:38.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.318+0000 7fb6c3e81640 1 -- 172.21.15.42:0/2745751120 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb6a80039c0 msgr2=0x7fb6a8003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:56:38.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.318+0000 7fb6c3e81640 1 --2- 172.21.15.42:0/2745751120 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb6a80039c0 0x7fb6a8003dc0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fb6ac003940 tx=0x7fb6ac01c400 comp rx=0 tx=0).stop 2024-09-16T03:56:38.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.318+0000 7fb6c3e81640 1 -- 172.21.15.42:0/2745751120 shutdown_connections 2024-09-16T03:56:38.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.318+0000 7fb6c3e81640 1 --2- 172.21.15.42:0/2745751120 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb6a80039c0 0x7fb6a8003dc0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:56:38.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.318+0000 7fb6c3e81640 1 -- 172.21.15.42:0/2745751120 >> 172.21.15.42:0/2745751120 conn(0x7fb6bc0a2420 msgr2=0x7fb6bc0a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:56:38.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.318+0000 7fb6c3e81640 1 -- 172.21.15.42:0/2745751120 shutdown_connections 2024-09-16T03:56:38.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.318+0000 7fb6c3e81640 1 -- 172.21.15.42:0/2745751120 wait complete. 2024-09-16T03:56:38.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.318+0000 7fb6c3e81640 10 monclient: build_initial_monmap 2024-09-16T03:56:38.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.318+0000 7fb6c3e81640 10 monclient: monmap: 2024-09-16T03:56:38.344 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:56:38.344 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:56:38.344 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:56:38.344 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:56:38.344 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:56:38.344 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:56:38.344 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:56:38.345 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:56:38.345 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:56:38.345 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:56:38.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.318+0000 7fb6c3e81640 1 librados: starting msgr at 2024-09-16T03:56:38.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.318+0000 7fb6c3e81640 1 librados: starting objecter 2024-09-16T03:56:38.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.318+0000 7fb6c3e81640 1 Processor -- start 2024-09-16T03:56:38.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.318+0000 7fb6c3e81640 1 -- start start 2024-09-16T03:56:38.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.318+0000 7fb6c3e81640 1 librados: setting wanted keys 2024-09-16T03:56:38.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.318+0000 7fb6c3e81640 1 librados: calling monclient init 2024-09-16T03:56:38.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.318+0000 7fb6c3e81640 10 monclient: init 2024-09-16T03:56:38.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.318+0000 7fb6c3e81640 10 monclient: _reopen_session rank -1 2024-09-16T03:56:38.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.318+0000 7fb6c3e81640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T03:56:38.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.318+0000 7fb6c3e81640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb6a80039c0 0x7fb6bc1cf180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:56:38.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.318+0000 7fb6c3e81640 10 monclient(hunting): picked mon.noname-b con 0x7fb6a80039c0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:56:38.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.318+0000 7fb6c3e81640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb6bc1332d0 0x7fb6bc133730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:56:38.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.318+0000 7fb6c3e81640 10 monclient(hunting): picked mon.noname-c con 0x7fb6bc1332d0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:56:38.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.318+0000 7fb6c3e81640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb6bc11bf70 0x7fb6bc11c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:56:38.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.318+0000 7fb6c3e81640 10 monclient(hunting): picked mon.noname-a con 0x7fb6bc11bf70 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:56:38.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.318+0000 7fb6c3e81640 10 monclient(hunting): start opening mon connection 2024-09-16T03:56:38.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.318+0000 7fb6c3e81640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fb6bc0a3170 con 0x7fb6bc11bf70 2024-09-16T03:56:38.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.318+0000 7fb6c3e81640 10 monclient(hunting): start opening mon connection 2024-09-16T03:56:38.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.318+0000 7fb6c3e81640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fb6bc09d0b0 con 0x7fb6a80039c0 2024-09-16T03:56:38.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.318+0000 7fb6c3e81640 10 monclient(hunting): start opening mon connection 2024-09-16T03:56:38.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.318+0000 7fb6c3e81640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fb6bc0a32f0 con 0x7fb6bc1332d0 2024-09-16T03:56:38.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.318+0000 7fb6c3e81640 10 monclient(hunting): _renew_subs 2024-09-16T03:56:38.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.318+0000 7fb6c3e81640 10 monclient(hunting): authenticate will time out at 1845.291583s 2024-09-16T03:56:38.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.318+0000 7fb6c241e640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb6bc11bf70 0x7fb6bc11c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:56:38.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.318+0000 7fb6c141c640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb6bc1332d0 0x7fb6bc133730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:56:38.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.318+0000 7fb6c1c1d640 1 -- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb6a80039c0 msgr2=0x7fb6bc1cf180 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T03:56:38.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.318+0000 7fb6c141c640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb6bc1332d0 0x7fb6bc133730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:58046/0 (socket says 172.21.15.42:58046) 2024-09-16T03:56:38.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.318+0000 7fb6c241e640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb6bc11bf70 0x7fb6bc11c3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:44764/0 (socket says 172.21.15.42:44764) 2024-09-16T03:56:38.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.318+0000 7fb6c141c640 1 -- 172.21.15.42:0/823162261 learned_addr learned my addr 172.21.15.42:0/823162261 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:56:38.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.318+0000 7fb6c141c640 10 monclient(hunting): get_auth_request con 0x7fb6bc1332d0 auth_method 0 2024-09-16T03:56:38.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.318+0000 7fb6c141c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:56:38.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.318+0000 7fb6c141c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:56:38.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.318+0000 7fb6c141c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:56:38.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.318+0000 7fb6c241e640 10 monclient(hunting): get_auth_request con 0x7fb6bc11bf70 auth_method 0 2024-09-16T03:56:38.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.318+0000 7fb6c241e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:56:38.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.318+0000 7fb6c241e640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:56:38.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.318+0000 7fb6c241e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:56:38.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.318+0000 7fb6c1c1d640 1 --2- 172.21.15.42:0/823162261 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb6a80039c0 0x7fb6bc1cf180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:56:38.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.318+0000 7fb6c141c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:56:38.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.318+0000 7fb6c141c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:56:38.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.318+0000 7fb6c141c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:56:38.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.318+0000 7fb6c241e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:56:38.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.318+0000 7fb6c241e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:56:38.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.318+0000 7fb6c241e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:56:38.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.322+0000 7fb6c141c640 10 monclient(hunting): handle_auth_done global_id 854348 payload 824 2024-09-16T03:56:38.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.322+0000 7fb6c141c640 1 -- 172.21.15.42:0/823162261 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb6a80039c0 msgr2=0x7fb6bc1cf180 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:56:38.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.322+0000 7fb6c141c640 1 --2- 172.21.15.42:0/823162261 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb6a80039c0 0x7fb6bc1cf180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:56:38.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.322+0000 7fb6c141c640 1 -- 172.21.15.42:0/823162261 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb6bc11bf70 msgr2=0x7fb6bc11c3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:56:38.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.322+0000 7fb6c141c640 1 --2- 172.21.15.42:0/823162261 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb6bc11bf70 0x7fb6bc11c3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:56:38.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.322+0000 7fb6c141c640 10 monclient: _finish_hunting 0 2024-09-16T03:56:38.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.322+0000 7fb6c141c640 1 monclient: found mon.noname-c 2024-09-16T03:56:38.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.322+0000 7fb6c141c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:56:38.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.322+0000 7fb6c141c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:56:38.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.322+0000 7fb6c141c640 1 -- 172.21.15.42:0/823162261 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6bc11f9a0 con 0x7fb6bc1332d0 2024-09-16T03:56:38.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.322+0000 7fb6c141c640 1 --2- 172.21.15.42:0/823162261 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb6bc1332d0 0x7fb6bc133730 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fb6a400b9c0 tx=0x7fb6a400bdc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:56:38.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.322+0000 7fb6ba7fc640 10 client.?.objecter ms_handle_connect 0x7fb6bc1332d0 2024-09-16T03:56:38.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.322+0000 7fb6ba7fc640 10 client.?.objecter resend_mon_ops 2024-09-16T03:56:38.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.322+0000 7fb6ba7fc640 1 -- 172.21.15.42:0/823162261 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6a400cac0 con 0x7fb6bc1332d0 2024-09-16T03:56:38.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.322+0000 7fb6ba7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:56:38.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.322+0000 7fb6ba7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:56:38.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.322+0000 7fb6ba7fc640 10 monclient: dump: 2024-09-16T03:56:38.350 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:56:38.350 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:56:38.350 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:38.350 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:38.350 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:56:38.350 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:56:38.350 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:56:38.350 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:56:38.350 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:56:38.350 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:56:38.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.322+0000 7fb6ba7fc640 10 monclient: _finish_auth 0 2024-09-16T03:56:38.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.322+0000 7fb6ba7fc640 10 monclient: _check_auth_tickets 2024-09-16T03:56:38.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.322+0000 7fb6ba7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:56:38.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.322+0000 7fb6ba7fc640 1 -- 172.21.15.42:0/823162261 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb6a4010070 con 0x7fb6bc1332d0 2024-09-16T03:56:38.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.322+0000 7fb6c3e81640 5 monclient: authenticate success, global_id 854348 2024-09-16T03:56:38.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.322+0000 7fb6c3e81640 10 monclient: _renew_subs 2024-09-16T03:56:38.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.322+0000 7fb6c3e81640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:56:38.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.322+0000 7fb6c3e81640 1 -- 172.21.15.42:0/823162261 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6bc13f4c0 con 0x7fb6bc1332d0 2024-09-16T03:56:38.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.322+0000 7fb6ba7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:56:38.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.322+0000 7fb6ba7fc640 1 -- 172.21.15.42:0/823162261 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6a4015470 con 0x7fb6bc1332d0 2024-09-16T03:56:38.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.322+0000 7fb6ba7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:56:38.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.322+0000 7fb6ba7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:56:38.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.322+0000 7fb6c3e81640 10 client.854348.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:56:38.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.322+0000 7fb6bbfff640 4 set_mon_vals no callback set 2024-09-16T03:56:38.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.322+0000 7fb6ba7fc640 10 monclient: dump: 2024-09-16T03:56:38.351 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:56:38.352 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:56:38.352 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:38.352 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:38.352 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:56:38.352 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:56:38.352 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:56:38.352 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:56:38.352 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:56:38.352 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:56:38.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.322+0000 7fb6c3e81640 10 monclient: _renew_subs 2024-09-16T03:56:38.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.322+0000 7fb6c3e81640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:56:38.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.322+0000 7fb6c3e81640 1 -- 172.21.15.42:0/823162261 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6bc070250 con 0x7fb6bc1332d0 2024-09-16T03:56:38.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.322+0000 7fb6c3e81640 1 librados: init done 2024-09-16T03:56:38.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.322+0000 7fb6c3e81640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:56:38.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.322+0000 7fb6c3e81640 1 --2- 172.21.15.42:0/823162261 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb6bc11f3e0 0x7fb6bc02d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:56:38.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.322+0000 7fb6c3e81640 10 monclient(hunting): start opening mon connection 2024-09-16T03:56:38.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.322+0000 7fb6c3e81640 1 -- 172.21.15.42:0/823162261 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fb6bc136100 con 0x7fb6bc11f3e0 2024-09-16T03:56:38.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.322+0000 7fb6c1c1d640 1 --2- 172.21.15.42:0/823162261 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb6bc11f3e0 0x7fb6bc02d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:56:38.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.322+0000 7fb6ba7fc640 1 -- 172.21.15.42:0/823162261 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fb6a401e3f0 con 0x7fb6bc1332d0 2024-09-16T03:56:38.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.322+0000 7fb6c1c1d640 10 monclient: get_auth_request con 0x7fb6bc11f3e0 auth_method 0 2024-09-16T03:56:38.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.322+0000 7fb6c1c1d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:56:38.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.322+0000 7fb6c1c1d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:56:38.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.322+0000 7fb6c1c1d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:56:38.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.322+0000 7fb6c1c1d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:56:38.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.322+0000 7fb6c1c1d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:56:38.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.322+0000 7fb6c1c1d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:56:38.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.322+0000 7fb6ba7fc640 1 --2- 172.21.15.42:0/823162261 >> v2:172.21.15.154:6812/254845342 conn(0x7fb69c0377c0 0x7fb69c039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:56:38.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.322+0000 7fb6ba7fc640 1 -- 172.21.15.42:0/823162261 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(443..443 src has 330..443) ==== 5356+0+0 (secure 0 0 0) 0x7fb6a4048ea0 con 0x7fb6bc1332d0 2024-09-16T03:56:38.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.322+0000 7fb6ba7fc640 10 client.854348.objecter ms_dispatch 0x7fb6bc000ba0 osd_map(443..443 src has 330..443) 2024-09-16T03:56:38.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.322+0000 7fb6ba7fc640 3 client.854348.objecter handle_osd_map got epochs [443,443] > 0 2024-09-16T03:56:38.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.322+0000 7fb6ba7fc640 3 client.854348.objecter handle_osd_map decoding full epoch 443 2024-09-16T03:56:38.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.322+0000 7fb6c1c1d640 10 monclient(hunting): handle_auth_done global_id 854351 payload 824 2024-09-16T03:56:38.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.322+0000 7fb6ba7fc640 20 client.854348.objecter dump_active .. 0 homeless 2024-09-16T03:56:38.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.322+0000 7fb6c1c1d640 1 -- 172.21.15.42:0/823162261 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb6bc11fd40 con 0x7fb6bc11f3e0 2024-09-16T03:56:38.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.322+0000 7fb6ba7fc640 10 client.854348.objecter ms_handle_connect 0x7fb6bc11f3e0 2024-09-16T03:56:38.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.322+0000 7fb6ba7fc640 10 client.854348.objecter resend_mon_ops 2024-09-16T03:56:38.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.322+0000 7fb6c1c1d640 1 --2- 172.21.15.42:0/823162261 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb6bc11f3e0 0x7fb6bc02d7d0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fb6ac006fd0 tx=0x7fb6a8005840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:56:38.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.322+0000 7fb6c241e640 1 --2- 172.21.15.42:0/823162261 >> v2:172.21.15.154:6812/254845342 conn(0x7fb69c0377c0 0x7fb69c039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:56:38.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.322+0000 7fb6ba7fc640 1 -- 172.21.15.42:0/823162261 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6ac009500 con 0x7fb6bc11f3e0 2024-09-16T03:56:38.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.322+0000 7fb6ba7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:56:38.355 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.322+0000 7fb6ba7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:56:38.355 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.322+0000 7fb6ba7fc640 10 monclient: dump: 2024-09-16T03:56:38.355 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:56:38.355 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:56:38.355 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:38.355 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:38.355 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:56:38.355 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:56:38.355 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:56:38.355 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:56:38.355 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:56:38.355 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:56:38.355 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.322+0000 7fb6c241e640 10 monclient: get_auth_request con 0x7fb69c0377c0 auth_method 0 2024-09-16T03:56:38.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.322+0000 7fb6ba7fc640 1 -- 172.21.15.42:0/823162261 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fb6ac009500 con 0x7fb6bc11f3e0 2024-09-16T03:56:38.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.322+0000 7fb6ba7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:56:38.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.322+0000 7fb6ba7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:56:38.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.322+0000 7fb6ba7fc640 1 -- 172.21.15.42:0/823162261 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb6bc11f3e0 msgr2=0x7fb6bc02d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:56:38.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.322+0000 7fb6ba7fc640 1 --2- 172.21.15.42:0/823162261 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb6bc11f3e0 0x7fb6bc02d7d0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fb6ac006fd0 tx=0x7fb6a8005840 comp rx=0 tx=0).stop 2024-09-16T03:56:38.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.322+0000 7fb6ba7fc640 1 -- 172.21.15.42:0/823162261 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb6bc11f3e0 msgr2=0x7fb6bc02d7d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:56:38.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.322+0000 7fb6ba7fc640 1 --2- 172.21.15.42:0/823162261 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb6bc11f3e0 0x7fb6bc02d7d0 secure :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fb6ac006fd0 tx=0x7fb6a8005840 comp rx=0 tx=0).stop 2024-09-16T03:56:38.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.322+0000 7fb6ba7fc640 10 client.854348.objecter ms_handle_connect 0x7fb69c0377c0 2024-09-16T03:56:38.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.322+0000 7fb6c241e640 1 --2- 172.21.15.42:0/823162261 >> v2:172.21.15.154:6812/254845342 conn(0x7fb69c0377c0 0x7fb69c039c80 secure :-1 s=READY pgs=3663 cs=0 l=1 rev1=1 crypto rx=0x7fb6bc0a6910 tx=0x7fb6b4007900 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:56:38.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.330+0000 7fb6c3e81640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:56:38.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.330+0000 7fb6c3e81640 1 --2- 172.21.15.42:0/823162261 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb6bc11c930 0x7fb6bc062ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:56:38.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.330+0000 7fb6c3e81640 10 monclient(hunting): start opening mon connection 2024-09-16T03:56:38.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.330+0000 7fb6c3e81640 1 -- 172.21.15.42:0/823162261 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fb6bc13be30 con 0x7fb6bc11c930 2024-09-16T03:56:38.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.330+0000 7fb6c1c1d640 1 --2- 172.21.15.42:0/823162261 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb6bc11c930 0x7fb6bc062ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:56:38.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.330+0000 7fb6c1c1d640 10 monclient: get_auth_request con 0x7fb6bc11c930 auth_method 0 2024-09-16T03:56:38.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.330+0000 7fb6c1c1d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:56:38.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.330+0000 7fb6c1c1d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:56:38.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.330+0000 7fb6c1c1d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:56:38.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.330+0000 7fb6c1c1d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:56:38.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.330+0000 7fb6c1c1d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:56:38.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.330+0000 7fb6c1c1d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:56:38.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.330+0000 7fb6c1c1d640 10 monclient(hunting): handle_auth_done global_id 854354 payload 824 2024-09-16T03:56:38.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.330+0000 7fb6c1c1d640 1 -- 172.21.15.42:0/823162261 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fb6bc11cd40 con 0x7fb6bc11c930 2024-09-16T03:56:38.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.330+0000 7fb6ba7fc640 10 client.854348.objecter ms_handle_connect 0x7fb6bc11c930 2024-09-16T03:56:38.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.330+0000 7fb6ba7fc640 10 client.854348.objecter resend_mon_ops 2024-09-16T03:56:38.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.330+0000 7fb6c1c1d640 1 --2- 172.21.15.42:0/823162261 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb6bc11c930 0x7fb6bc062ea0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fb6ac003940 tx=0x7fb6ac0090e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:56:38.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.330+0000 7fb6ba7fc640 1 -- 172.21.15.42:0/823162261 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6ac0bd870 con 0x7fb6bc11c930 2024-09-16T03:56:38.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.330+0000 7fb6ba7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:56:38.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.330+0000 7fb6ba7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:56:38.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.330+0000 7fb6ba7fc640 10 monclient: dump: 2024-09-16T03:56:38.359 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:56:38.360 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:56:38.360 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:38.360 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:38.360 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:56:38.360 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:56:38.360 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:56:38.360 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:56:38.360 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:56:38.360 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:56:38.360 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.330+0000 7fb6ba7fc640 1 -- 172.21.15.42:0/823162261 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4535 (secure 0 0 0) 0x7fb6ac0bd870 con 0x7fb6bc11c930 2024-09-16T03:56:38.360 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.330+0000 7fb6ba7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:56:38.360 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.330+0000 7fb6ba7fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:56:38.360 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.330+0000 7fb6ba7fc640 1 -- 172.21.15.42:0/823162261 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb6bc11c930 msgr2=0x7fb6bc062ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:56:38.361 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.330+0000 7fb6ba7fc640 1 --2- 172.21.15.42:0/823162261 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb6bc11c930 0x7fb6bc062ea0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fb6ac003940 tx=0x7fb6ac0090e0 comp rx=0 tx=0).stop 2024-09-16T03:56:38.361 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.330+0000 7fb6ba7fc640 1 -- 172.21.15.42:0/823162261 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb6bc11c930 msgr2=0x7fb6bc062ea0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:56:38.361 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.330+0000 7fb6ba7fc640 1 --2- 172.21.15.42:0/823162261 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb6bc11c930 0x7fb6bc062ea0 secure :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fb6ac003940 tx=0x7fb6ac0090e0 comp rx=0 tx=0).stop 2024-09-16T03:56:38.361 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:56:38.362 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:56:38.362 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:56:38.362 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T03:56:38.362 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 132, 2024-09-16T03:56:38.362 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 131766, 2024-09-16T03:56:38.362 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:56:38.362 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:56:38.362 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T03:56:38.362 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:56:38.362 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 1, 2024-09-16T03:56:38.363 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:56:38.363 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:56:38.363 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:56:38.363 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:56:38.363 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:56:38.363 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:56:38.363 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:56:38.363 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:56:38.363 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:56:38.363 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:56:38.363 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:56:38.363 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:56:38.364 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:56:38.364 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:56:38.364 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:56:38.364 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:56:38.364 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:56:38.364 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:56:38.364 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:56:38.364 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:56:38.364 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:56:38.364 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:56:38.364 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:56:38.364 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:56:38.365 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:56:38.365 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:56:38.365 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:56:38.365 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:56:38.365 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:56:38.365 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:56:38.365 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:56:38.365 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:56:38.365 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:56:38.365 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:56:38.365 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:56:38.365 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:56:38.365 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:56:38.365 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:56:38.366 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:56:38.366 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:56:38.366 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:56:38.366 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:56:38.366 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:56:38.366 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:56:38.366 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:56:38.366 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:56:38.367 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:56:38.367 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:56:38.367 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:56:38.367 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:56:38.367 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:56:38.367 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:56:38.367 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:56:38.367 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:56:38.367 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:56:38.367 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:56:38.367 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:56:38.367 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:56:38.367 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:56:38.368 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:56:38.368 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:56:38.368 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:56:38.368 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:56:38.368 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:56:38.368 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:56:38.368 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:56:38.368 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:56:38.369 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:56:38.369 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:56:38.369 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:56:38.369 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:56:38.369 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:56:38.369 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:56:38.369 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:56:38.369 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:56:38.369 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:56:38.369 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:56:38.369 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:56:38.369 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:56:38.369 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:56:38.370 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:56:38.370 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:56:38.370 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:56:38.370 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:56:38.370 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:56:38.370 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:56:38.370 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:56:38.370 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:56:38.370 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:56:38.370 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:56:38.370 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:56:38.370 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:56:38.370 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:56:38.370 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:56:38.371 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:56:38.371 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:56:38.371 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:56:38.371 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:56:38.371 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:56:38.371 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:56:38.371 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:56:38.371 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:56:38.371 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:56:38.371 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:56:38.371 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:56:38.371 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:56:38.371 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:56:38.371 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:56:38.372 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:56:38.372 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:56:38.372 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:56:38.372 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:56:38.372 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:56:38.372 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:56:38.372 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:56:38.372 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:56:38.372 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:56:38.372 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:56:38.372 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:56:38.372 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:56:38.372 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:56:38.373 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:56:38.373 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:56:38.373 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:56:38.373 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:56:38.373 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:56:38.373 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:56:38.373 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:56:38.373 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:56:38.373 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:56:38.373 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:56:38.373 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:56:38.373 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:56:38.373 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:56:38.373 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:56:38.374 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:56:38.374 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:56:38.374 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:56:38.374 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:56:38.374 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:56:38.374 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:56:38.374 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:56:38.374 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:56:38.374 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:56:38.374 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:56:38.374 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 2 2024-09-16T03:56:38.374 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:56:38.374 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:56:38.374 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:56:38.374 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:56:38.375 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:56:38.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.330+0000 7fb6c3e81640 10 librados: watch_flush enter 2024-09-16T03:56:38.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.330+0000 7fb6c3e81640 10 librados: watch_flush exit 2024-09-16T03:56:38.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.330+0000 7fb6c3e81640 20 client.854348.objecter shutdown clearing up homeless session... 2024-09-16T03:56:38.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.330+0000 7fb6c3e81640 10 client.854348.objecter successfully canceled tick 2024-09-16T03:56:38.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.330+0000 7fb6c3e81640 1 -- 172.21.15.42:0/823162261 >> v2:172.21.15.154:6812/254845342 conn(0x7fb69c0377c0 msgr2=0x7fb69c039c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:56:38.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.330+0000 7fb6c3e81640 1 --2- 172.21.15.42:0/823162261 >> v2:172.21.15.154:6812/254845342 conn(0x7fb69c0377c0 0x7fb69c039c80 secure :-1 s=READY pgs=3663 cs=0 l=1 rev1=1 crypto rx=0x7fb6bc0a6910 tx=0x7fb6b4007900 comp rx=0 tx=0).stop 2024-09-16T03:56:38.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.330+0000 7fb6c3e81640 10 monclient: shutdown 2024-09-16T03:56:38.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.330+0000 7fb6c3e81640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:56:38.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.330+0000 7fb6c3e81640 1 -- 172.21.15.42:0/823162261 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb6bc1332d0 msgr2=0x7fb6bc133730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:56:38.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.330+0000 7fb6c3e81640 1 --2- 172.21.15.42:0/823162261 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb6bc1332d0 0x7fb6bc133730 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fb6a400b9c0 tx=0x7fb6a400bdc0 comp rx=0 tx=0).stop 2024-09-16T03:56:38.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.330+0000 7fb6c241e640 1 -- 172.21.15.42:0/823162261 reap_dead start 2024-09-16T03:56:38.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.330+0000 7fb6c241e640 1 -- 172.21.15.42:0/823162261 reap_dead start 2024-09-16T03:56:38.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.330+0000 7fb6c3e81640 1 -- 172.21.15.42:0/823162261 shutdown_connections 2024-09-16T03:56:38.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.330+0000 7fb6c3e81640 1 -- 172.21.15.42:0/823162261 >> 172.21.15.42:0/823162261 conn(0x7fb6bc0a2420 msgr2=0x7fb6bc0712a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:56:38.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.330+0000 7fb6c3e81640 1 -- 172.21.15.42:0/823162261 shutdown_connections 2024-09-16T03:56:38.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.330+0000 7fb6c3e81640 1 -- 172.21.15.42:0/823162261 wait complete. 2024-09-16T03:56:38.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:38.330+0000 7fb6c3e81640 1 librados: shutdown 2024-09-16T03:56:38.378 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T03:56:38.378 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-16T03:56:50.475 INFO:tasks.ceph.osd.2.smithi042.stderr:2024-09-16T03:56:50.470+0000 7fe6e3cac640 -1 snap_mapper.add_oid found existing snaps mapped on 103:add70e79:test-rados-api-smithi154-28216-97::foo:21, removing 2024-09-16T03:56:50.476 INFO:tasks.ceph.osd.5.smithi154.stderr:2024-09-16T03:56:50.480+0000 7f056eece640 -1 snap_mapper.add_oid found existing snaps mapped on 103:add70e79:test-rados-api-smithi154-28216-97::foo:21, removing 2024-09-16T03:56:58.379 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-16T03:56:58.379 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-09-16T03:56:58.379 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-16T03:56:58.379 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-16T03:56:58.382 INFO:tasks.ceph.mon.b:Started 2024-09-16T03:56:58.382 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T03:56:58.382 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:56:58.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.514+0000 7fb81ef5d640 10 monclient: get_monmap_and_config 2024-09-16T03:56:58.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.514+0000 7fb81ef5d640 10 monclient: build_initial_monmap 2024-09-16T03:56:58.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.514+0000 7fb81ef5d640 10 monclient: monmap: 2024-09-16T03:56:58.518 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:56:58.518 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:56:58.518 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:56:58.516809+0000 2024-09-16T03:56:58.518 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:56:58.516809+0000 2024-09-16T03:56:58.519 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:56:58.519 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:56:58.519 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:56:58.519 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:56:58.519 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:56:58.519 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:56:58.519 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:56:58.519 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:56:58.521 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.514+0000 7fb81ef5d640 1 Processor -- start 2024-09-16T03:56:58.521 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.514+0000 7fb81ef5d640 1 -- start start 2024-09-16T03:56:58.521 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.514+0000 7fb81ef5d640 10 monclient: init 2024-09-16T03:56:58.521 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.514+0000 7fb81ef5d640 10 monclient: _reopen_session rank -1 2024-09-16T03:56:58.522 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.514+0000 7fb81ef5d640 10 monclient: _add_conns ranks=[4,1,3,0,2] 2024-09-16T03:56:58.522 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.514+0000 7fb81ef5d640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fb8180a1330 0x7fb8180a1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:56:58.522 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.514+0000 7fb81ef5d640 10 monclient(hunting): picked mon.noname-c con 0x7fb8180a1330 addr v2:172.21.15.154:3300/0 2024-09-16T03:56:58.522 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.514+0000 7fb81ef5d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb81806dac0 addr v1:172.21.15.154:6789/0 2024-09-16T03:56:58.522 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.514+0000 7fb81ef5d640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb8180a1c50 0x7fb818135810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:56:58.522 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.514+0000 7fb81ef5d640 10 monclient(hunting): picked mon.noname-a con 0x7fb8180a1c50 addr v2:172.21.15.42:3300/0 2024-09-16T03:56:58.522 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.514+0000 7fb81ef5d640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb81809c090 con 0x7fb81806dac0 2024-09-16T03:56:58.522 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.514+0000 7fb81ef5d640 10 monclient(hunting): start opening mon connection 2024-09-16T03:56:58.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.514+0000 7fb81ef5d640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7fb81809f610 con 0x7fb8180a1c50 2024-09-16T03:56:58.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.514+0000 7fb81ef5d640 10 monclient(hunting): start opening mon connection 2024-09-16T03:56:58.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.514+0000 7fb81ef5d640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fb81809f8f0 con 0x7fb8180a1330 2024-09-16T03:56:58.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.514+0000 7fb81ef5d640 10 monclient(hunting): _renew_subs 2024-09-16T03:56:58.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.514+0000 7fb81ef5d640 10 monclient(hunting): authenticate will time out at 1865.486246s 2024-09-16T03:56:58.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.514+0000 7fb81d4fa640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb8180a1c50 0x7fb818135810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:56:58.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.514+0000 7fb81d4fa640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb8180a1c50 0x7fb818135810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:55760/0 (socket says 172.21.15.42:55760) 2024-09-16T03:56:58.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.514+0000 7fb81ccf9640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fb8180a1330 0x7fb8180a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:56:58.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.514+0000 7fb817fff640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7fb81806dac0 0x7fb818135100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.42:58524/0 (socket says 172.21.15.42:58524) 2024-09-16T03:56:58.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.514+0000 7fb81d4fa640 1 -- 172.21.15.42:0/4167799096 learned_addr learned my addr 172.21.15.42:0/4167799096 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:56:58.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.514+0000 7fb81d4fa640 10 monclient(hunting): get_auth_request con 0x7fb8180a1c50 auth_method 0 2024-09-16T03:56:58.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.514+0000 7fb81d4fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:56:58.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.514+0000 7fb81d4fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:56:58.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.514+0000 7fb81d4fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:56:58.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.514+0000 7fb81ccf9640 10 monclient(hunting): get_auth_request con 0x7fb8180a1330 auth_method 0 2024-09-16T03:56:58.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.514+0000 7fb81ccf9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:56:58.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.514+0000 7fb81ccf9640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:56:58.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.514+0000 7fb81ccf9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:56:58.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.514+0000 7fb81d4fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:56:58.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.514+0000 7fb81d4fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:56:58.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.514+0000 7fb81d4fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:56:58.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.514+0000 7fb81ccf9640 1 -- 172.21.15.42:0/4167799096 >> v2:172.21.15.154:3300/0 conn(0x7fb8180a1330 msgr2=0x7fb8180a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T03:56:58.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.514+0000 7fb81ccf9640 1 -- 172.21.15.42:0/4167799096 >> v2:172.21.15.154:3300/0 conn(0x7fb8180a1330 msgr2=0x7fb8180a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:56:58.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.514+0000 7fb81ccf9640 1 --2- 172.21.15.42:0/4167799096 >> v2:172.21.15.154:3300/0 conn(0x7fb8180a1330 0x7fb8180a1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:56:58.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.514+0000 7fb81ccf9640 1 --2- 172.21.15.42:0/4167799096 >> v2:172.21.15.154:3300/0 conn(0x7fb8180a1330 0x7fb8180a1710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:56:58.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.514+0000 7fb81d4fa640 10 monclient(hunting): handle_auth_done global_id 824796 payload 274 2024-09-16T03:56:58.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.514+0000 7fb81d4fa640 1 -- 172.21.15.42:0/4167799096 >> v2:172.21.15.154:3300/0 conn(0x7fb8180a1330 msgr2=0x7fb8180a1710 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:56:58.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.514+0000 7fb81d4fa640 1 --2- 172.21.15.42:0/4167799096 >> v2:172.21.15.154:3300/0 conn(0x7fb8180a1330 0x7fb8180a1710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:56:58.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.514+0000 7fb81d4fa640 1 -- 172.21.15.42:0/4167799096 >> v1:172.21.15.154:6789/0 conn(0x7fb81806dac0 legacy=0x7fb818135100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:56:58.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.514+0000 7fb81d4fa640 10 monclient: _finish_hunting 0 2024-09-16T03:56:58.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.514+0000 7fb81d4fa640 1 monclient: found mon.noname-a 2024-09-16T03:56:58.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.514+0000 7fb81d4fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:56:58.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.514+0000 7fb81d4fa640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:56:58.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.514+0000 7fb81d4fa640 1 -- 172.21.15.42:0/4167799096 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb81809fbd0 con 0x7fb8180a1c50 2024-09-16T03:56:58.532 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.518+0000 7fb81d4fa640 1 --2- 172.21.15.42:0/4167799096 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb8180a1c50 0x7fb818135810 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fb81806add0 tx=0x7fb81001c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=199cdd3c2c8e44ea server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:56:58.532 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.518+0000 7fb8177fe640 1 -- 172.21.15.42:0/4167799096 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8100a5790 con 0x7fb8180a1c50 2024-09-16T03:56:58.532 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.518+0000 7fb8177fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:56:58.532 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.518+0000 7fb8177fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:56:58.532 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.518+0000 7fb8177fe640 10 monclient: dump: 2024-09-16T03:56:58.532 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:56:58.532 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:56:58.533 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:58.533 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:58.533 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:56:58.533 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:56:58.533 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:56:58.534 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:56:58.534 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:56:58.534 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:56:58.534 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.518+0000 7fb8177fe640 10 monclient: _finish_auth 0 2024-09-16T03:56:58.534 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.518+0000 7fb8177fe640 10 monclient: _check_auth_tickets 2024-09-16T03:56:58.534 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.518+0000 7fb8177fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:56:58.534 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.518+0000 7fb8177fe640 1 -- 172.21.15.42:0/4167799096 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb8100a5930 con 0x7fb8180a1c50 2024-09-16T03:56:58.534 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.518+0000 7fb81ef5d640 5 monclient: authenticate success, global_id 824796 2024-09-16T03:56:58.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.518+0000 7fb81ef5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:56:58.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.518+0000 7fb8177fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:56:58.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.518+0000 7fb81ef5d640 10 monclient: get_monmap_and_config success 2024-09-16T03:56:58.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.518+0000 7fb81ef5d640 4 set_mon_vals no callback set 2024-09-16T03:56:58.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.518+0000 7fb8177fe640 1 -- 172.21.15.42:0/4167799096 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8100a5d00 con 0x7fb8180a1c50 2024-09-16T03:56:58.536 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.518+0000 7fb8177fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:56:58.537 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.518+0000 7fb8177fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:56:58.537 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.518+0000 7fb8177fe640 10 monclient: dump: 2024-09-16T03:56:58.537 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:56:58.537 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:56:58.537 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:58.537 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:58.537 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:56:58.537 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:56:58.539 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:56:58.539 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:56:58.539 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:56:58.539 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:56:58.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.518+0000 7fb81ef5d640 10 monclient: shutdown 2024-09-16T03:56:58.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.518+0000 7fb81ef5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:56:58.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.518+0000 7fb81ef5d640 1 -- 172.21.15.42:0/4167799096 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb8180a1c50 msgr2=0x7fb818135810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:56:58.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.518+0000 7fb81ef5d640 1 --2- 172.21.15.42:0/4167799096 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb8180a1c50 0x7fb818135810 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fb81806add0 tx=0x7fb81001c8b0 comp rx=0 tx=0).stop 2024-09-16T03:56:58.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.518+0000 7fb81ef5d640 1 -- 172.21.15.42:0/4167799096 shutdown_connections 2024-09-16T03:56:58.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.518+0000 7fb81ef5d640 1 --2- 172.21.15.42:0/4167799096 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb8180a1c50 0x7fb818135810 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:56:58.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.518+0000 7fb81ef5d640 1 --2- 172.21.15.42:0/4167799096 >> v2:172.21.15.154:3300/0 conn(0x7fb8180a1330 0x7fb8180a1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:56:58.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.518+0000 7fb81ef5d640 1 -- 172.21.15.42:0/4167799096 >> 172.21.15.42:0/4167799096 conn(0x7fb8180a2420 msgr2=0x7fb8180a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:56:58.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.518+0000 7fb81ef5d640 1 -- 172.21.15.42:0/4167799096 shutdown_connections 2024-09-16T03:56:58.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.518+0000 7fb81ef5d640 1 -- 172.21.15.42:0/4167799096 wait complete. 2024-09-16T03:56:58.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.518+0000 7fb81ef5d640 10 monclient: build_initial_monmap 2024-09-16T03:56:58.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.518+0000 7fb81ef5d640 10 monclient: monmap: 2024-09-16T03:56:58.540 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:56:58.540 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:56:58.540 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:56:58.540 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:56:58.541 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:56:58.541 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:56:58.541 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:56:58.541 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:56:58.541 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:56:58.541 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:56:58.541 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.518+0000 7fb81ef5d640 1 librados: starting msgr at 2024-09-16T03:56:58.541 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.518+0000 7fb81ef5d640 1 librados: starting objecter 2024-09-16T03:56:58.541 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.518+0000 7fb81ef5d640 1 Processor -- start 2024-09-16T03:56:58.542 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.518+0000 7fb81ef5d640 1 -- start start 2024-09-16T03:56:58.542 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.518+0000 7fb81ef5d640 1 librados: setting wanted keys 2024-09-16T03:56:58.542 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.518+0000 7fb81ef5d640 1 librados: calling monclient init 2024-09-16T03:56:58.542 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.518+0000 7fb81ef5d640 10 monclient: init 2024-09-16T03:56:58.542 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.518+0000 7fb81ef5d640 10 monclient: _reopen_session rank -1 2024-09-16T03:56:58.542 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.518+0000 7fb81ef5d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:56:58.542 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.518+0000 7fb81ef5d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb8180a1330 0x7fb8181ceef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:56:58.542 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.518+0000 7fb81ef5d640 10 monclient(hunting): picked mon.noname-b con 0x7fb8180a1330 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:56:58.542 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.518+0000 7fb81ef5d640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb8180a1c50 0x7fb8181352c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:56:58.542 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.518+0000 7fb81ef5d640 10 monclient(hunting): picked mon.noname-a con 0x7fb8180a1c50 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:56:58.542 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.518+0000 7fb81ef5d640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb818135800 0x7fb81811dce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:56:58.542 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.518+0000 7fb81ef5d640 10 monclient(hunting): picked mon.noname-c con 0x7fb818135800 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:56:58.542 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.518+0000 7fb81ef5d640 10 monclient(hunting): start opening mon connection 2024-09-16T03:56:58.543 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.518+0000 7fb81ef5d640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fb818135d50 con 0x7fb8180a1c50 2024-09-16T03:56:58.543 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.518+0000 7fb81ef5d640 10 monclient(hunting): start opening mon connection 2024-09-16T03:56:58.543 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.518+0000 7fb81ef5d640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fb81809ef80 con 0x7fb8180a1330 2024-09-16T03:56:58.543 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.518+0000 7fb81ef5d640 10 monclient(hunting): start opening mon connection 2024-09-16T03:56:58.543 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.518+0000 7fb81ef5d640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fb818135ed0 con 0x7fb818135800 2024-09-16T03:56:58.543 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.518+0000 7fb81ef5d640 10 monclient(hunting): _renew_subs 2024-09-16T03:56:58.543 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.518+0000 7fb81ef5d640 10 monclient(hunting): authenticate will time out at 1865.491449s 2024-09-16T03:56:58.543 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.518+0000 7fb817fff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb8180a1c50 0x7fb8181352c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:56:58.543 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.518+0000 7fb81d4fa640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb818135800 0x7fb81811dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:56:58.543 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.518+0000 7fb817fff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb8180a1c50 0x7fb8181352c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:55766/0 (socket says 172.21.15.42:55766) 2024-09-16T03:56:58.543 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.518+0000 7fb81d4fa640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb818135800 0x7fb81811dce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:44972/0 (socket says 172.21.15.42:44972) 2024-09-16T03:56:58.543 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.518+0000 7fb817fff640 1 -- 172.21.15.42:0/1572829143 learned_addr learned my addr 172.21.15.42:0/1572829143 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:56:58.544 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.518+0000 7fb817fff640 10 monclient(hunting): get_auth_request con 0x7fb8180a1c50 auth_method 0 2024-09-16T03:56:58.544 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.518+0000 7fb817fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:56:58.544 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.518+0000 7fb817fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:56:58.544 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.518+0000 7fb817fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:56:58.544 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.518+0000 7fb81d4fa640 10 monclient(hunting): get_auth_request con 0x7fb818135800 auth_method 0 2024-09-16T03:56:58.545 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.518+0000 7fb81d4fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:56:58.545 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.518+0000 7fb81d4fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:56:58.545 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.518+0000 7fb81d4fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:56:58.545 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.518+0000 7fb81ccf9640 1 --2- 172.21.15.42:0/1572829143 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb8180a1330 0x7fb8181ceef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:56:58.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.518+0000 7fb817fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:56:58.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.518+0000 7fb817fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:56:58.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.518+0000 7fb817fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:56:58.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.518+0000 7fb81ccf9640 10 monclient(hunting): get_auth_request con 0x7fb8180a1330 auth_method 0 2024-09-16T03:56:58.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.518+0000 7fb81ccf9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:56:58.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.518+0000 7fb81ccf9640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:56:58.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.518+0000 7fb81ccf9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:56:58.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.518+0000 7fb81d4fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:56:58.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.518+0000 7fb81d4fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:56:58.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.518+0000 7fb81d4fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:56:58.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.518+0000 7fb81ccf9640 1 -- 172.21.15.42:0/1572829143 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb8180a1330 msgr2=0x7fb8181ceef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-16T03:56:58.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.518+0000 7fb81ccf9640 1 -- 172.21.15.42:0/1572829143 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb8180a1330 msgr2=0x7fb8181ceef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T03:56:58.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.518+0000 7fb81ccf9640 1 --2- 172.21.15.42:0/1572829143 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb8180a1330 0x7fb8181ceef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:56:58.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.518+0000 7fb81ccf9640 1 --2- 172.21.15.42:0/1572829143 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb8180a1330 0x7fb8181ceef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:56:58.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.518+0000 7fb817fff640 10 monclient(hunting): handle_auth_done global_id 824799 payload 824 2024-09-16T03:56:58.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.518+0000 7fb817fff640 1 -- 172.21.15.42:0/1572829143 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb818135800 msgr2=0x7fb81811dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:56:58.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.518+0000 7fb817fff640 1 --2- 172.21.15.42:0/1572829143 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb818135800 0x7fb81811dce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:56:58.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.518+0000 7fb817fff640 1 -- 172.21.15.42:0/1572829143 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb8180a1330 msgr2=0x7fb8181ceef0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:56:58.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.518+0000 7fb817fff640 1 --2- 172.21.15.42:0/1572829143 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb8180a1330 0x7fb8181ceef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:56:58.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.518+0000 7fb817fff640 10 monclient: _finish_hunting 0 2024-09-16T03:56:58.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.518+0000 7fb817fff640 1 monclient: found mon.noname-a 2024-09-16T03:56:58.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.518+0000 7fb817fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:56:58.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.522+0000 7fb817fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:56:58.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.522+0000 7fb817fff640 1 -- 172.21.15.42:0/1572829143 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb818121990 con 0x7fb8180a1c50 2024-09-16T03:56:58.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.522+0000 7fb817fff640 1 --2- 172.21.15.42:0/1572829143 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb8180a1c50 0x7fb8181352c0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fb80800c450 tx=0x7fb80800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:56:58.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.522+0000 7fb8157fa640 10 client.?.objecter ms_handle_connect 0x7fb8180a1c50 2024-09-16T03:56:58.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.522+0000 7fb8157fa640 10 client.?.objecter resend_mon_ops 2024-09-16T03:56:58.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.522+0000 7fb8157fa640 1 -- 172.21.15.42:0/1572829143 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb808018070 con 0x7fb8180a1c50 2024-09-16T03:56:58.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.522+0000 7fb8157fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:56:58.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.522+0000 7fb8157fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:56:58.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.522+0000 7fb8157fa640 10 monclient: dump: 2024-09-16T03:56:58.548 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:56:58.548 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:56:58.549 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:58.549 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:58.549 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:56:58.549 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:56:58.549 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:56:58.549 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:56:58.549 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:56:58.549 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:56:58.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.522+0000 7fb8157fa640 10 monclient: _finish_auth 0 2024-09-16T03:56:58.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.522+0000 7fb8157fa640 10 monclient: _check_auth_tickets 2024-09-16T03:56:58.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.522+0000 7fb8157fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:56:58.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.522+0000 7fb8157fa640 1 -- 172.21.15.42:0/1572829143 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb808007480 con 0x7fb8180a1c50 2024-09-16T03:56:58.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.522+0000 7fb81ef5d640 5 monclient: authenticate success, global_id 824799 2024-09-16T03:56:58.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.522+0000 7fb81ef5d640 10 monclient: _renew_subs 2024-09-16T03:56:58.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.522+0000 7fb81ef5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:56:58.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.522+0000 7fb81ef5d640 1 -- 172.21.15.42:0/1572829143 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8181cf430 con 0x7fb8180a1c50 2024-09-16T03:56:58.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.522+0000 7fb8157fa640 10 monclient: handle_config config(0 keys) 2024-09-16T03:56:58.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.522+0000 7fb8157fa640 1 -- 172.21.15.42:0/1572829143 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8080047f0 con 0x7fb8180a1c50 2024-09-16T03:56:58.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.522+0000 7fb8157fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:56:58.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.522+0000 7fb8157fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:56:58.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.522+0000 7fb8157fa640 10 monclient: dump: 2024-09-16T03:56:58.550 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:56:58.551 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:56:58.551 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:58.551 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:56:58.551 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:56:58.551 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:56:58.551 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:56:58.551 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:56:58.551 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:56:58.551 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:56:58.551 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.522+0000 7fb81ef5d640 10 client.824799.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:56:58.552 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.522+0000 7fb81ef5d640 10 monclient: _renew_subs 2024-09-16T03:56:58.552 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.522+0000 7fb8167fc640 4 set_mon_vals no callback set 2024-09-16T03:56:58.552 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.522+0000 7fb81ef5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:56:58.552 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.522+0000 7fb81ef5d640 1 -- 172.21.15.42:0/1572829143 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb81813c8c0 con 0x7fb8180a1c50 2024-09-16T03:56:58.552 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.522+0000 7fb81ef5d640 1 librados: init done 2024-09-16T03:56:58.552 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.522+0000 7fb81ef5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:56:58.552 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.522+0000 7fb81ef5d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:56:58.552 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.522+0000 7fb81ef5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:56:58.552 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.522+0000 7fb81ef5d640 1 -- 172.21.15.42:0/1572829143 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb81809d3b0 con 0x7fb8180a1c50 2024-09-16T03:56:58.552 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.522+0000 7fb8157fa640 1 -- 172.21.15.42:0/1572829143 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fb80801b070 con 0x7fb8180a1c50 2024-09-16T03:56:58.552 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.522+0000 7fb8157fa640 1 --2- 172.21.15.42:0/1572829143 >> v2:172.21.15.154:6812/254845342 conn(0x7fb7f8037840 0x7fb7f8039d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:56:58.552 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.522+0000 7fb8157fa640 1 -- 172.21.15.42:0/1572829143 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(454..454 src has 330..454) ==== 5747+0+0 (secure 0 0 0) 0x7fb8080480e0 con 0x7fb8180a1c50 2024-09-16T03:56:58.552 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.522+0000 7fb8157fa640 10 client.824799.objecter ms_dispatch 0x7fb818000ba0 osd_map(454..454 src has 330..454) 2024-09-16T03:56:58.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.522+0000 7fb8157fa640 3 client.824799.objecter handle_osd_map got epochs [454,454] > 0 2024-09-16T03:56:58.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.522+0000 7fb8157fa640 3 client.824799.objecter handle_osd_map decoding full epoch 454 2024-09-16T03:56:58.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.522+0000 7fb8157fa640 20 client.824799.objecter dump_active .. 0 homeless 2024-09-16T03:56:58.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.522+0000 7fb81ccf9640 1 --2- 172.21.15.42:0/1572829143 >> v2:172.21.15.154:6812/254845342 conn(0x7fb7f8037840 0x7fb7f8039d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:56:58.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.522+0000 7fb81ccf9640 10 monclient: get_auth_request con 0x7fb7f8037840 auth_method 0 2024-09-16T03:56:58.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.522+0000 7fb81ccf9640 1 --2- 172.21.15.42:0/1572829143 >> v2:172.21.15.154:6812/254845342 conn(0x7fb7f8037840 0x7fb7f8039d00 secure :-1 s=READY pgs=3665 cs=0 l=1 rev1=1 crypto rx=0x7fb800006aa0 tx=0x7fb800006a30 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:56:58.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.522+0000 7fb8157fa640 10 client.824799.objecter ms_handle_connect 0x7fb7f8037840 2024-09-16T03:56:58.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.526+0000 7fb8157fa640 1 -- 172.21.15.42:0/1572829143 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7fb8080119e0 con 0x7fb8180a1c50 2024-09-16T03:56:58.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.526+0000 7fb8157fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:56:58.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.526+0000 7fb8157fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:56:58.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.750+0000 7fb81ef5d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:56:58.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.750+0000 7fb81ef5d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:56:58.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.750+0000 7fb81ef5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:56:58.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.750+0000 7fb81ef5d640 1 -- 172.21.15.42:0/1572829143 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fb8180a1710 con 0x7fb8180a1c50 2024-09-16T03:56:58.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.750+0000 7fb8157fa640 1 -- 172.21.15.42:0/1572829143 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7fb808004d20 con 0x7fb8180a1c50 2024-09-16T03:56:58.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.750+0000 7fb8157fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:56:58.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.750+0000 7fb8157fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:56:58.756 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":132,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":22,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:56:58.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.754+0000 7fb81ef5d640 10 librados: watch_flush enter 2024-09-16T03:56:58.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.754+0000 7fb81ef5d640 10 librados: watch_flush exit 2024-09-16T03:56:58.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.754+0000 7fb81ef5d640 20 client.824799.objecter shutdown clearing up homeless session... 2024-09-16T03:56:58.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.754+0000 7fb81ef5d640 10 client.824799.objecter successfully canceled tick 2024-09-16T03:56:58.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.754+0000 7fb81ef5d640 1 -- 172.21.15.42:0/1572829143 >> v2:172.21.15.154:6812/254845342 conn(0x7fb7f8037840 msgr2=0x7fb7f8039d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:56:58.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.754+0000 7fb81ef5d640 1 --2- 172.21.15.42:0/1572829143 >> v2:172.21.15.154:6812/254845342 conn(0x7fb7f8037840 0x7fb7f8039d00 secure :-1 s=READY pgs=3665 cs=0 l=1 rev1=1 crypto rx=0x7fb800006aa0 tx=0x7fb800006a30 comp rx=0 tx=0).stop 2024-09-16T03:56:58.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.754+0000 7fb81ef5d640 10 monclient: shutdown 2024-09-16T03:56:58.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.754+0000 7fb81ef5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:56:58.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.754+0000 7fb81ef5d640 1 -- 172.21.15.42:0/1572829143 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb8180a1c50 msgr2=0x7fb8181352c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:56:58.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.754+0000 7fb81ef5d640 1 --2- 172.21.15.42:0/1572829143 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb8180a1c50 0x7fb8181352c0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fb80800c450 tx=0x7fb80800c910 comp rx=0 tx=0).stop 2024-09-16T03:56:58.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.754+0000 7fb81ef5d640 1 -- 172.21.15.42:0/1572829143 shutdown_connections 2024-09-16T03:56:58.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.754+0000 7fb81ef5d640 1 --2- 172.21.15.42:0/1572829143 >> v2:172.21.15.154:6812/254845342 conn(0x7fb7f8037840 0x7fb7f8039d00 unknown :-1 s=CLOSED pgs=3665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:56:58.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.754+0000 7fb81ef5d640 1 --2- 172.21.15.42:0/1572829143 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb818135800 0x7fb81811dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:56:58.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.754+0000 7fb81ef5d640 1 --2- 172.21.15.42:0/1572829143 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb8180a1c50 0x7fb8181352c0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:56:58.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.754+0000 7fb81ef5d640 1 --2- 172.21.15.42:0/1572829143 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb8180a1330 0x7fb8181ceef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:56:58.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.754+0000 7fb81ef5d640 1 -- 172.21.15.42:0/1572829143 >> 172.21.15.42:0/1572829143 conn(0x7fb8180a2420 msgr2=0x7fb8180a46b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:56:58.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.754+0000 7fb81ef5d640 1 -- 172.21.15.42:0/1572829143 shutdown_connections 2024-09-16T03:56:58.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.754+0000 7fb81ef5d640 1 -- 172.21.15.42:0/1572829143 wait complete. 2024-09-16T03:56:58.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:56:58.754+0000 7fb81ef5d640 1 librados: shutdown 2024-09-16T03:57:01.774 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:57:01.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:01.902+0000 7fc187af1640 10 monclient: get_monmap_and_config 2024-09-16T03:57:01.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:01.902+0000 7fc187af1640 10 monclient: build_initial_monmap 2024-09-16T03:57:01.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:01.902+0000 7fc187af1640 10 monclient: monmap: 2024-09-16T03:57:01.907 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:57:01.907 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:57:01.907 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:57:01.905396+0000 2024-09-16T03:57:01.907 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:57:01.905396+0000 2024-09-16T03:57:01.907 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:57:01.907 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:57:01.907 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:57:01.907 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:57:01.907 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:57:01.907 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:57:01.908 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:57:01.908 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:57:01.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:01.902+0000 7fc187af1640 1 Processor -- start 2024-09-16T03:57:01.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:01.902+0000 7fc187af1640 1 -- start start 2024-09-16T03:57:01.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:01.902+0000 7fc187af1640 10 monclient: init 2024-09-16T03:57:01.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:01.902+0000 7fc187af1640 10 monclient: _reopen_session rank -1 2024-09-16T03:57:01.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:01.902+0000 7fc187af1640 10 monclient: _add_conns ranks=[2,4,0,3,1] 2024-09-16T03:57:01.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:01.902+0000 7fc187af1640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc1800a39b0 0x7fc180133b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:57:01.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:01.902+0000 7fc187af1640 10 monclient(hunting): picked mon.noname-b con 0x7fc1800a39b0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:57:01.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:01.902+0000 7fc187af1640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fc1800a3d90 0x7fc180137000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:57:01.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:01.902+0000 7fc187af1640 10 monclient(hunting): picked mon.noname-c con 0x7fc1800a3d90 addr v2:172.21.15.154:3300/0 2024-09-16T03:57:01.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:01.902+0000 7fc187af1640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc180137540 addr v1:172.21.15.42:6789/0 2024-09-16T03:57:01.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:01.902+0000 7fc187af1640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc18009c140 con 0x7fc180137540 2024-09-16T03:57:01.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:01.902+0000 7fc187af1640 10 monclient(hunting): start opening mon connection 2024-09-16T03:57:01.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:01.902+0000 7fc187af1640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fc18009c420 con 0x7fc1800a3d90 2024-09-16T03:57:01.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:01.902+0000 7fc187af1640 10 monclient(hunting): start opening mon connection 2024-09-16T03:57:01.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:01.902+0000 7fc187af1640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fc18006dd60 con 0x7fc1800a39b0 2024-09-16T03:57:01.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:01.902+0000 7fc187af1640 10 monclient(hunting): _renew_subs 2024-09-16T03:57:01.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:01.902+0000 7fc187af1640 10 monclient(hunting): authenticate will time out at 1868.875047s 2024-09-16T03:57:01.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:01.902+0000 7fc18588d640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc1800a39b0 0x7fc180133b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:57:01.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:01.902+0000 7fc18588d640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc1800a39b0 0x7fc180133b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:44986/0 (socket says 172.21.15.42:44986) 2024-09-16T03:57:01.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:01.902+0000 7fc18588d640 1 -- 172.21.15.42:0/1036523732 learned_addr learned my addr 172.21.15.42:0/1036523732 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:57:01.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:01.902+0000 7fc18588d640 10 monclient(hunting): get_auth_request con 0x7fc1800a39b0 auth_method 0 2024-09-16T03:57:01.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:01.902+0000 7fc18588d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:57:01.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:01.902+0000 7fc18588d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:57:01.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:01.902+0000 7fc18588d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:57:01.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:01.902+0000 7fc18588d640 1 -- 172.21.15.42:0/1036523732 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc1800a39b0 msgr2=0x7fc180133b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T03:57:01.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:01.902+0000 7fc18588d640 1 -- 172.21.15.42:0/1036523732 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc1800a39b0 msgr2=0x7fc180133b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:57:01.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:01.902+0000 7fc18588d640 1 --2- 172.21.15.42:0/1036523732 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc1800a39b0 0x7fc180133b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:57:01.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:01.902+0000 7fc18588d640 1 --2- 172.21.15.42:0/1036523732 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc1800a39b0 0x7fc180133b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:57:01.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:01.902+0000 7fc18508c640 1 --2- 172.21.15.42:0/1036523732 >> v2:172.21.15.154:3300/0 conn(0x7fc1800a3d90 0x7fc180137000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:57:01.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:01.902+0000 7fc18508c640 10 monclient(hunting): get_auth_request con 0x7fc1800a3d90 auth_method 0 2024-09-16T03:57:01.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:01.902+0000 7fc18508c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:57:01.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:01.902+0000 7fc18508c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:57:01.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:01.902+0000 7fc18508c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:57:01.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:01.902+0000 7fc18508c640 1 -- 172.21.15.42:0/1036523732 >> v2:172.21.15.154:3300/0 conn(0x7fc1800a3d90 msgr2=0x7fc180137000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T03:57:01.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:01.902+0000 7fc18508c640 1 -- 172.21.15.42:0/1036523732 >> v2:172.21.15.154:3300/0 conn(0x7fc1800a3d90 msgr2=0x7fc180137000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:57:01.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:01.902+0000 7fc18508c640 1 --2- 172.21.15.42:0/1036523732 >> v2:172.21.15.154:3300/0 conn(0x7fc1800a3d90 0x7fc180137000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:57:01.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:01.902+0000 7fc18508c640 1 --2- 172.21.15.42:0/1036523732 >> v2:172.21.15.154:3300/0 conn(0x7fc1800a3d90 0x7fc180137000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:57:02.110 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:02.106+0000 7fc18588d640 1 --2- 172.21.15.42:0/1036523732 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc1800a39b0 0x7fc180133b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:57:02.110 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:02.106+0000 7fc18588d640 10 monclient(hunting): get_auth_request con 0x7fc1800a39b0 auth_method 0 2024-09-16T03:57:02.110 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:02.106+0000 7fc18588d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:57:02.110 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:02.106+0000 7fc18588d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:57:02.110 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:02.106+0000 7fc18588d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:57:02.110 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:02.106+0000 7fc18588d640 1 -- 172.21.15.42:0/1036523732 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc1800a39b0 msgr2=0x7fc180133b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T03:57:02.111 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:02.106+0000 7fc18588d640 1 -- 172.21.15.42:0/1036523732 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc1800a39b0 msgr2=0x7fc180133b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:57:02.111 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:02.106+0000 7fc18588d640 1 --2- 172.21.15.42:0/1036523732 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc1800a39b0 0x7fc180133b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:57:02.111 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:02.106+0000 7fc18588d640 1 --2- 172.21.15.42:0/1036523732 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc1800a39b0 0x7fc180133b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T03:57:02.111 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:02.106+0000 7fc18508c640 1 --2- 172.21.15.42:0/1036523732 >> v2:172.21.15.154:3300/0 conn(0x7fc1800a3d90 0x7fc180137000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:57:02.111 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:02.106+0000 7fc18508c640 10 monclient(hunting): get_auth_request con 0x7fc1800a3d90 auth_method 0 2024-09-16T03:57:02.112 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:02.106+0000 7fc18508c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:57:02.112 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:02.106+0000 7fc18508c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:57:02.112 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:02.106+0000 7fc18508c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:57:02.112 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:02.106+0000 7fc18508c640 1 -- 172.21.15.42:0/1036523732 >> v2:172.21.15.154:3300/0 conn(0x7fc1800a3d90 msgr2=0x7fc180137000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T03:57:02.112 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:02.106+0000 7fc18508c640 1 -- 172.21.15.42:0/1036523732 >> v2:172.21.15.154:3300/0 conn(0x7fc1800a3d90 msgr2=0x7fc180137000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:57:02.112 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:02.106+0000 7fc18508c640 1 --2- 172.21.15.42:0/1036523732 >> v2:172.21.15.154:3300/0 conn(0x7fc1800a3d90 0x7fc180137000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:57:02.112 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:02.106+0000 7fc18508c640 1 --2- 172.21.15.42:0/1036523732 >> v2:172.21.15.154:3300/0 conn(0x7fc1800a3d90 0x7fc180137000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T03:57:02.514 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:02.510+0000 7fc18588d640 1 --2- 172.21.15.42:0/1036523732 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc1800a39b0 0x7fc180133b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:57:02.514 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:02.510+0000 7fc18588d640 10 monclient(hunting): get_auth_request con 0x7fc1800a39b0 auth_method 0 2024-09-16T03:57:02.514 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:02.510+0000 7fc18588d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:57:02.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:02.510+0000 7fc18588d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:57:02.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:02.510+0000 7fc18588d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:57:02.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:02.510+0000 7fc18508c640 1 --2- 172.21.15.42:0/1036523732 >> v2:172.21.15.154:3300/0 conn(0x7fc1800a3d90 0x7fc180137000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:57:02.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:02.510+0000 7fc18588d640 1 -- 172.21.15.42:0/1036523732 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc1800a39b0 msgr2=0x7fc180133b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T03:57:02.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:02.510+0000 7fc18588d640 1 -- 172.21.15.42:0/1036523732 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc1800a39b0 msgr2=0x7fc180133b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:57:02.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:02.510+0000 7fc18588d640 1 --2- 172.21.15.42:0/1036523732 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc1800a39b0 0x7fc180133b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:57:02.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:02.510+0000 7fc18508c640 10 monclient(hunting): get_auth_request con 0x7fc1800a3d90 auth_method 0 2024-09-16T03:57:02.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:02.510+0000 7fc18508c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:57:02.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:02.510+0000 7fc18508c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:57:02.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:02.510+0000 7fc18508c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:57:02.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:02.510+0000 7fc18588d640 1 --2- 172.21.15.42:0/1036523732 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc1800a39b0 0x7fc180133b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T03:57:02.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:02.510+0000 7fc18508c640 1 -- 172.21.15.42:0/1036523732 >> v2:172.21.15.154:3300/0 conn(0x7fc1800a3d90 msgr2=0x7fc180137000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T03:57:02.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:02.510+0000 7fc18508c640 1 -- 172.21.15.42:0/1036523732 >> v2:172.21.15.154:3300/0 conn(0x7fc1800a3d90 msgr2=0x7fc180137000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:57:02.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:02.510+0000 7fc18508c640 1 --2- 172.21.15.42:0/1036523732 >> v2:172.21.15.154:3300/0 conn(0x7fc1800a3d90 0x7fc180137000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:57:02.516 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:02.510+0000 7fc18508c640 1 --2- 172.21.15.42:0/1036523732 >> v2:172.21.15.154:3300/0 conn(0x7fc1800a3d90 0x7fc180137000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T03:57:03.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:03.314+0000 7fc18588d640 1 --2- 172.21.15.42:0/1036523732 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc1800a39b0 0x7fc180133b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:57:03.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:03.314+0000 7fc18588d640 10 monclient(hunting): get_auth_request con 0x7fc1800a39b0 auth_method 0 2024-09-16T03:57:03.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:03.314+0000 7fc18588d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:57:03.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:03.314+0000 7fc18588d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:57:03.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:03.314+0000 7fc18588d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:57:03.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:03.314+0000 7fc18508c640 1 --2- 172.21.15.42:0/1036523732 >> v2:172.21.15.154:3300/0 conn(0x7fc1800a3d90 0x7fc180137000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:57:03.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:03.314+0000 7fc18508c640 10 monclient(hunting): get_auth_request con 0x7fc1800a3d90 auth_method 0 2024-09-16T03:57:03.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:03.314+0000 7fc18508c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:57:03.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:03.314+0000 7fc18588d640 1 -- 172.21.15.42:0/1036523732 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc1800a39b0 msgr2=0x7fc180133b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T03:57:03.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:03.314+0000 7fc18508c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:57:03.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:03.314+0000 7fc18508c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:57:03.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:03.314+0000 7fc18588d640 1 -- 172.21.15.42:0/1036523732 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc1800a39b0 msgr2=0x7fc180133b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:57:03.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:03.314+0000 7fc18588d640 1 --2- 172.21.15.42:0/1036523732 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc1800a39b0 0x7fc180133b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:57:03.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:03.314+0000 7fc18588d640 1 --2- 172.21.15.42:0/1036523732 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc1800a39b0 0x7fc180133b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T03:57:03.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:03.314+0000 7fc18508c640 1 -- 172.21.15.42:0/1036523732 >> v2:172.21.15.154:3300/0 conn(0x7fc1800a3d90 msgr2=0x7fc180137000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T03:57:03.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:03.314+0000 7fc18508c640 1 -- 172.21.15.42:0/1036523732 >> v2:172.21.15.154:3300/0 conn(0x7fc1800a3d90 msgr2=0x7fc180137000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:57:03.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:03.314+0000 7fc18508c640 1 --2- 172.21.15.42:0/1036523732 >> v2:172.21.15.154:3300/0 conn(0x7fc1800a3d90 0x7fc180137000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:57:03.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:03.314+0000 7fc18508c640 1 --2- 172.21.15.42:0/1036523732 >> v2:172.21.15.154:3300/0 conn(0x7fc1800a3d90 0x7fc180137000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T03:57:04.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:04.902+0000 7fc1737fe640 10 monclient(hunting): tick 2024-09-16T03:57:04.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:04.902+0000 7fc1737fe640 10 monclient(hunting): _check_auth_tickets 2024-09-16T03:57:04.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:04.902+0000 7fc1737fe640 1 monclient(hunting): continuing hunt 2024-09-16T03:57:04.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:04.902+0000 7fc1737fe640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T03:57:04.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:04.902+0000 7fc1737fe640 1 -- 172.21.15.42:0/1036523732 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc1800a39b0 msgr2=0x7fc180133b80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:57:04.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:04.902+0000 7fc1737fe640 1 --2- 172.21.15.42:0/1036523732 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc1800a39b0 0x7fc180133b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:57:04.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:04.902+0000 7fc1737fe640 1 -- 172.21.15.42:0/1036523732 >> v2:172.21.15.154:3300/0 conn(0x7fc1800a3d90 msgr2=0x7fc180137000 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:57:04.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:04.902+0000 7fc1737fe640 1 --2- 172.21.15.42:0/1036523732 >> v2:172.21.15.154:3300/0 conn(0x7fc1800a3d90 0x7fc180137000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:57:04.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:04.902+0000 7fc1737fe640 1 -- 172.21.15.42:0/1036523732 >> v1:172.21.15.42:6789/0 conn(0x7fc180137540 legacy=0x7fc18013e390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:57:04.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:04.902+0000 7fc1737fe640 10 monclient: _add_conns ranks=[1,3] 2024-09-16T03:57:04.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:04.902+0000 7fc1737fe640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc1640036f0 addr v1:172.21.15.154:6789/0 2024-09-16T03:57:04.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:04.902+0000 7fc1737fe640 1 --2- 172.21.15.42:0/1036523732 >> v2:172.21.15.42:3300/0 conn(0x7fc164007400 0x7fc1640098c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:57:04.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:04.902+0000 7fc1737fe640 10 monclient(hunting): picked mon.noname-a con 0x7fc164007400 addr v2:172.21.15.42:3300/0 2024-09-16T03:57:04.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:04.902+0000 7fc1737fe640 1 -- 172.21.15.42:0/1036523732 --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc1800a5e20 con 0x7fc1640036f0 2024-09-16T03:57:04.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:04.902+0000 7fc1737fe640 10 monclient(hunting): start opening mon connection 2024-09-16T03:57:04.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:04.902+0000 7fc1737fe640 1 -- 172.21.15.42:0/1036523732 --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7fc18009c420 con 0x7fc164007400 2024-09-16T03:57:04.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:04.902+0000 7fc1737fe640 10 monclient(hunting): _renew_subs 2024-09-16T03:57:04.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:04.902+0000 7fc18508c640 1 --2- 172.21.15.42:0/1036523732 >> v2:172.21.15.42:3300/0 conn(0x7fc164007400 0x7fc1640098c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:57:04.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:04.902+0000 7fc18508c640 10 monclient(hunting): get_auth_request con 0x7fc164007400 auth_method 0 2024-09-16T03:57:04.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:04.902+0000 7fc18508c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:57:04.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:04.902+0000 7fc18508c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:57:04.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:04.902+0000 7fc18508c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:57:04.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:04.906+0000 7fc18508c640 1 -- 172.21.15.42:0/1036523732 >> v2:172.21.15.42:3300/0 conn(0x7fc164007400 msgr2=0x7fc1640098c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T03:57:04.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:04.906+0000 7fc18508c640 1 -- 172.21.15.42:0/1036523732 >> v2:172.21.15.42:3300/0 conn(0x7fc164007400 msgr2=0x7fc1640098c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:57:04.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:04.906+0000 7fc18508c640 1 --2- 172.21.15.42:0/1036523732 >> v2:172.21.15.42:3300/0 conn(0x7fc164007400 0x7fc1640098c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:57:04.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:04.906+0000 7fc18508c640 1 --2- 172.21.15.42:0/1036523732 >> v2:172.21.15.42:3300/0 conn(0x7fc164007400 0x7fc1640098c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:57:05.114 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:05.110+0000 7fc18508c640 1 --2- 172.21.15.42:0/1036523732 >> v2:172.21.15.42:3300/0 conn(0x7fc164007400 0x7fc1640098c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:57:05.114 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:05.110+0000 7fc18508c640 10 monclient(hunting): get_auth_request con 0x7fc164007400 auth_method 0 2024-09-16T03:57:05.114 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:05.110+0000 7fc18508c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:57:05.114 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:05.110+0000 7fc18508c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:57:05.114 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:05.110+0000 7fc18508c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:57:05.114 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:05.110+0000 7fc18508c640 1 -- 172.21.15.42:0/1036523732 >> v2:172.21.15.42:3300/0 conn(0x7fc164007400 msgr2=0x7fc1640098c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T03:57:05.114 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:05.110+0000 7fc18508c640 1 -- 172.21.15.42:0/1036523732 >> v2:172.21.15.42:3300/0 conn(0x7fc164007400 msgr2=0x7fc1640098c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:57:05.114 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:05.110+0000 7fc18508c640 1 --2- 172.21.15.42:0/1036523732 >> v2:172.21.15.42:3300/0 conn(0x7fc164007400 0x7fc1640098c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:57:05.114 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:05.110+0000 7fc18508c640 1 --2- 172.21.15.42:0/1036523732 >> v2:172.21.15.42:3300/0 conn(0x7fc164007400 0x7fc1640098c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T03:57:05.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:05.514+0000 7fc18508c640 1 --2- 172.21.15.42:0/1036523732 >> v2:172.21.15.42:3300/0 conn(0x7fc164007400 0x7fc1640098c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:57:05.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:05.514+0000 7fc18508c640 10 monclient(hunting): get_auth_request con 0x7fc164007400 auth_method 0 2024-09-16T03:57:05.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:05.514+0000 7fc18508c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:57:05.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:05.514+0000 7fc18508c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:57:05.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:05.514+0000 7fc18508c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:57:05.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:05.514+0000 7fc18508c640 1 -- 172.21.15.42:0/1036523732 >> v2:172.21.15.42:3300/0 conn(0x7fc164007400 msgr2=0x7fc1640098c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T03:57:05.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:05.514+0000 7fc18508c640 1 -- 172.21.15.42:0/1036523732 >> v2:172.21.15.42:3300/0 conn(0x7fc164007400 msgr2=0x7fc1640098c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:57:05.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:05.514+0000 7fc18508c640 1 --2- 172.21.15.42:0/1036523732 >> v2:172.21.15.42:3300/0 conn(0x7fc164007400 0x7fc1640098c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:57:05.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:05.514+0000 7fc18508c640 1 --2- 172.21.15.42:0/1036523732 >> v2:172.21.15.42:3300/0 conn(0x7fc164007400 0x7fc1640098c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T03:57:06.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.318+0000 7fc18508c640 1 --2- 172.21.15.42:0/1036523732 >> v2:172.21.15.42:3300/0 conn(0x7fc164007400 0x7fc1640098c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:57:06.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.318+0000 7fc18508c640 10 monclient(hunting): get_auth_request con 0x7fc164007400 auth_method 0 2024-09-16T03:57:06.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.318+0000 7fc18508c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:57:06.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.318+0000 7fc18508c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:57:06.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.318+0000 7fc18508c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:57:06.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.318+0000 7fc18508c640 1 -- 172.21.15.42:0/1036523732 >> v2:172.21.15.42:3300/0 conn(0x7fc164007400 msgr2=0x7fc1640098c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T03:57:06.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.318+0000 7fc18508c640 1 -- 172.21.15.42:0/1036523732 >> v2:172.21.15.42:3300/0 conn(0x7fc164007400 msgr2=0x7fc1640098c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:57:06.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.318+0000 7fc18508c640 1 --2- 172.21.15.42:0/1036523732 >> v2:172.21.15.42:3300/0 conn(0x7fc164007400 0x7fc1640098c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:57:06.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.318+0000 7fc18508c640 1 --2- 172.21.15.42:0/1036523732 >> v2:172.21.15.42:3300/0 conn(0x7fc164007400 0x7fc1640098c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T03:57:06.639 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.634+0000 7fc18488b640 1 -- 172.21.15.42:0/1036523732 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2462301343 0 0) 0x7fc1800a5e20 con 0x7fc1640036f0 2024-09-16T03:57:06.639 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.634+0000 7fc18488b640 10 monclient(hunting): _negotiate 2024-09-16T03:57:06.639 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.634+0000 7fc18488b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:57:06.639 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.634+0000 7fc18488b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:57:06.640 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.634+0000 7fc18488b640 10 monclient(hunting): my global_id is 884098 2024-09-16T03:57:06.640 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.634+0000 7fc18488b640 1 -- 172.21.15.42:0/1036523732 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc16c003530 con 0x7fc1640036f0 2024-09-16T03:57:06.642 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.634+0000 7fc18488b640 1 -- 172.21.15.42:0/1036523732 <== mon.1 v1:172.21.15.154:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 95075078 0 0) 0x7fc16c003530 con 0x7fc1640036f0 2024-09-16T03:57:06.642 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.634+0000 7fc18488b640 1 -- 172.21.15.42:0/1036523732 >> v2:172.21.15.42:3300/0 conn(0x7fc164007400 msgr2=0x7fc1640098c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:57:06.642 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.634+0000 7fc18488b640 1 --2- 172.21.15.42:0/1036523732 >> v2:172.21.15.42:3300/0 conn(0x7fc164007400 0x7fc1640098c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:57:06.642 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.634+0000 7fc18488b640 10 monclient: _finish_hunting 0 2024-09-16T03:57:06.643 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.634+0000 7fc18488b640 1 monclient: found mon.noname-c-legacy 2024-09-16T03:57:06.643 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.634+0000 7fc18488b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:57:06.643 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.634+0000 7fc18488b640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.154:6789/0 2024-09-16T03:57:06.643 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.634+0000 7fc18488b640 1 -- 172.21.15.42:0/1036523732 --> v1:172.21.15.154:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc16400afd0 con 0x7fc1640036f0 2024-09-16T03:57:06.643 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.634+0000 7fc18488b640 10 monclient: _finish_auth 0 2024-09-16T03:57:06.643 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.634+0000 7fc18488b640 10 monclient: _check_auth_tickets 2024-09-16T03:57:06.643 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.634+0000 7fc18488b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:57:06.643 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.634+0000 7fc18488b640 1 -- 172.21.15.42:0/1036523732 <== mon.1 v1:172.21.15.154:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2177413734 0 0) 0x7fc17400f040 con 0x7fc1640036f0 2024-09-16T03:57:06.643 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.634+0000 7fc187af1640 5 monclient: authenticate success, global_id 884098 2024-09-16T03:57:06.644 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.634+0000 7fc187af1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:57:06.644 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.634+0000 7fc18488b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:57:06.644 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.634+0000 7fc18488b640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-16T03:57:06.644 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.634+0000 7fc18488b640 10 monclient: dump: 2024-09-16T03:57:06.644 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:57:06.644 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:57:06.644 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:57:06.644 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:57:06.644 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:57:06.644 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:57:06.646 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:57:06.646 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:57:06.646 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:57:06.646 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:57:06.646 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.634+0000 7fc18488b640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] but i'm connected to v1:172.21.15.154:6789/0, reconnecting 2024-09-16T03:57:06.646 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.634+0000 7fc18488b640 10 monclient: _reopen_session rank -1 2024-09-16T03:57:06.646 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.634+0000 7fc18488b640 1 -- 172.21.15.42:0/1036523732 >> v1:172.21.15.154:6789/0 conn(0x7fc1640036f0 legacy=0x7fc164005b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:57:06.646 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.634+0000 7fc18608e640 1 -- 172.21.15.42:0/1036523732 reap_dead start 2024-09-16T03:57:06.647 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.634+0000 7fc18488b640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:57:06.647 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.634+0000 7fc18488b640 1 --2- 172.21.15.42:0/1036523732 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc16c003a10 0x7fc16c003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:57:06.647 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.634+0000 7fc18488b640 10 monclient(hunting): picked mon.c con 0x7fc16c003a10 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:57:06.647 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.634+0000 7fc18488b640 1 --2- 172.21.15.42:0/1036523732 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc16c004810 0x7fc16c004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:57:06.647 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.634+0000 7fc18488b640 10 monclient(hunting): picked mon.b con 0x7fc16c004810 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:57:06.647 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.634+0000 7fc18488b640 1 --2- 172.21.15.42:0/1036523732 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc16c0053d0 0x7fc16c00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:57:06.647 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.634+0000 7fc18488b640 10 monclient(hunting): picked mon.a con 0x7fc16c0053d0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:57:06.647 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.634+0000 7fc18488b640 10 monclient(hunting): start opening mon connection 2024-09-16T03:57:06.647 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.634+0000 7fc18488b640 1 -- 172.21.15.42:0/1036523732 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fc18009c420 con 0x7fc16c0053d0 2024-09-16T03:57:06.647 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.634+0000 7fc18488b640 10 monclient(hunting): start opening mon connection 2024-09-16T03:57:06.647 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.634+0000 7fc18488b640 1 -- 172.21.15.42:0/1036523732 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc16c00c290 con 0x7fc16c004810 2024-09-16T03:57:06.647 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.634+0000 7fc18488b640 10 monclient(hunting): start opening mon connection 2024-09-16T03:57:06.647 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.634+0000 7fc18488b640 1 -- 172.21.15.42:0/1036523732 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fc16c00c490 con 0x7fc16c003a10 2024-09-16T03:57:06.647 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.634+0000 7fc18488b640 10 monclient(hunting): _renew_subs 2024-09-16T03:57:06.648 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.634+0000 7fc18488b640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:57:06.648 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.634+0000 7fc18588d640 1 --2- 172.21.15.42:0/1036523732 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc16c003a10 0x7fc16c003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:57:06.648 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.634+0000 7fc187af1640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-16T03:57:06.648 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.634+0000 7fc18608e640 1 --2- 172.21.15.42:0/1036523732 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc16c0053d0 0x7fc16c00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:57:06.648 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.634+0000 7fc18588d640 10 monclient(hunting): get_auth_request con 0x7fc16c003a10 auth_method 0 2024-09-16T03:57:06.648 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.634+0000 7fc18588d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:57:06.648 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.634+0000 7fc18588d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:57:06.648 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.634+0000 7fc18588d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:57:06.648 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.634+0000 7fc18608e640 10 monclient(hunting): get_auth_request con 0x7fc16c0053d0 auth_method 0 2024-09-16T03:57:06.648 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.634+0000 7fc18608e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:57:06.648 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.634+0000 7fc18608e640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:57:06.648 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.634+0000 7fc18608e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:57:06.648 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.634+0000 7fc18508c640 1 --2- 172.21.15.42:0/1036523732 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc16c004810 0x7fc16c004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:57:06.648 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.634+0000 7fc18508c640 10 monclient(hunting): get_auth_request con 0x7fc16c004810 auth_method 0 2024-09-16T03:57:06.649 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.634+0000 7fc18508c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:57:06.649 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.634+0000 7fc18508c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:57:06.649 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.634+0000 7fc18508c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:57:06.649 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.634+0000 7fc18608e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:57:06.649 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.634+0000 7fc18608e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:57:06.649 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.638+0000 7fc18608e640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T03:57:06.649 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.638+0000 7fc18588d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:57:06.650 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.638+0000 7fc18588d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:57:06.650 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.638+0000 7fc18588d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T03:57:06.650 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.638+0000 7fc18508c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:57:06.650 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.638+0000 7fc18508c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:57:06.650 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.638+0000 7fc18508c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T03:57:06.650 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.638+0000 7fc18608e640 10 monclient(hunting): handle_auth_done global_id 884098 payload 293 2024-09-16T03:57:06.650 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.638+0000 7fc18608e640 1 -- 172.21.15.42:0/1036523732 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc16c003a10 msgr2=0x7fc16c003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:57:06.650 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.638+0000 7fc18608e640 1 --2- 172.21.15.42:0/1036523732 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc16c003a10 0x7fc16c003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:57:06.650 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.638+0000 7fc18608e640 1 -- 172.21.15.42:0/1036523732 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc16c004810 msgr2=0x7fc16c004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:57:06.650 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.638+0000 7fc18608e640 1 --2- 172.21.15.42:0/1036523732 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc16c004810 0x7fc16c004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:57:06.650 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.638+0000 7fc18608e640 10 monclient: _finish_hunting 0 2024-09-16T03:57:06.650 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.638+0000 7fc18608e640 1 monclient: found mon.a 2024-09-16T03:57:06.650 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.638+0000 7fc18608e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:57:06.651 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.638+0000 7fc18608e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:57:06.651 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.638+0000 7fc18608e640 1 -- 172.21.15.42:0/1036523732 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc16c003530 con 0x7fc16c0053d0 2024-09-16T03:57:06.651 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.638+0000 7fc18608e640 10 monclient: _finish_auth 0 2024-09-16T03:57:06.651 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.638+0000 7fc18608e640 10 monclient: _check_auth_tickets 2024-09-16T03:57:06.651 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.638+0000 7fc18608e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:57:06.657 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.642+0000 7fc18608e640 1 --2- 172.21.15.42:0/1036523732 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc16c0053d0 0x7fc16c00b9d0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fc180137920 tx=0x7fc17801bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=530220541529b006 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:57:06.657 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.642+0000 7fc18488b640 1 -- 172.21.15.42:0/1036523732 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1780012a0 con 0x7fc16c0053d0 2024-09-16T03:57:06.657 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.642+0000 7fc18488b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:57:06.657 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.642+0000 7fc18488b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:57:06.657 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.642+0000 7fc18488b640 10 monclient: dump: 2024-09-16T03:57:06.657 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:57:06.657 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:57:06.657 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:57:06.658 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:57:06.658 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:57:06.658 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:57:06.658 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:57:06.658 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:57:06.658 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:57:06.658 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:57:06.658 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.642+0000 7fc187af1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:57:06.658 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.642+0000 7fc18488b640 1 -- 172.21.15.42:0/1036523732 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc178004020 con 0x7fc16c0053d0 2024-09-16T03:57:06.658 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.642+0000 7fc18488b640 10 monclient: handle_config config(0 keys) 2024-09-16T03:57:06.658 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.642+0000 7fc187af1640 10 monclient: get_monmap_and_config success 2024-09-16T03:57:06.658 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.642+0000 7fc187af1640 4 set_mon_vals no callback set 2024-09-16T03:57:06.658 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.642+0000 7fc18488b640 1 -- 172.21.15.42:0/1036523732 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc178004430 con 0x7fc16c0053d0 2024-09-16T03:57:06.658 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.642+0000 7fc18488b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:57:06.659 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.642+0000 7fc18488b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:57:06.659 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.642+0000 7fc18488b640 10 monclient: dump: 2024-09-16T03:57:06.659 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:57:06.659 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:57:06.659 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:57:06.659 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:57:06.659 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:57:06.659 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:57:06.659 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:57:06.659 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:57:06.659 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:57:06.659 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:57:06.659 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.642+0000 7fc187af1640 10 monclient: shutdown 2024-09-16T03:57:06.660 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.642+0000 7fc187af1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:57:06.660 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.642+0000 7fc187af1640 1 -- 172.21.15.42:0/1036523732 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc16c0053d0 msgr2=0x7fc16c00b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:57:06.660 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.642+0000 7fc187af1640 1 --2- 172.21.15.42:0/1036523732 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc16c0053d0 0x7fc16c00b9d0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fc180137920 tx=0x7fc17801bf80 comp rx=0 tx=0).stop 2024-09-16T03:57:06.660 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.642+0000 7fc187af1640 1 -- 172.21.15.42:0/1036523732 shutdown_connections 2024-09-16T03:57:06.660 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.642+0000 7fc187af1640 1 --2- 172.21.15.42:0/1036523732 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc16c0053d0 0x7fc16c00b9d0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:57:06.660 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.642+0000 7fc187af1640 1 --2- 172.21.15.42:0/1036523732 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc16c004810 0x7fc16c004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:57:06.660 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.642+0000 7fc187af1640 1 --2- 172.21.15.42:0/1036523732 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc16c003a10 0x7fc16c003e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:57:06.660 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.642+0000 7fc187af1640 1 -- 172.21.15.42:0/1036523732 >> 172.21.15.42:0/1036523732 conn(0x7fc1800a9fb0 msgr2=0x7fc1800aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:57:06.660 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.642+0000 7fc187af1640 1 -- 172.21.15.42:0/1036523732 shutdown_connections 2024-09-16T03:57:06.660 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.642+0000 7fc187af1640 1 -- 172.21.15.42:0/1036523732 wait complete. 2024-09-16T03:57:06.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.646+0000 7fc187af1640 10 monclient: build_initial_monmap 2024-09-16T03:57:06.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.646+0000 7fc187af1640 10 monclient: monmap: 2024-09-16T03:57:06.661 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:57:06.661 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:57:06.661 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:57:06.661 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:57:06.661 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:57:06.661 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:57:06.661 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:57:06.661 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:57:06.661 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:57:06.661 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:57:06.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.646+0000 7fc187af1640 1 librados: starting msgr at 2024-09-16T03:57:06.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.646+0000 7fc187af1640 1 librados: starting objecter 2024-09-16T03:57:06.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.646+0000 7fc187af1640 1 Processor -- start 2024-09-16T03:57:06.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.646+0000 7fc187af1640 1 -- start start 2024-09-16T03:57:06.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.646+0000 7fc187af1640 1 librados: setting wanted keys 2024-09-16T03:57:06.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.646+0000 7fc187af1640 1 librados: calling monclient init 2024-09-16T03:57:06.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.646+0000 7fc187af1640 10 monclient: init 2024-09-16T03:57:06.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.646+0000 7fc187af1640 10 monclient: _reopen_session rank -1 2024-09-16T03:57:06.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.646+0000 7fc187af1640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:57:06.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.646+0000 7fc187af1640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc16c003a10 0x7fc1801cc010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:57:06.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.646+0000 7fc187af1640 10 monclient(hunting): picked mon.noname-c con 0x7fc16c003a10 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:57:06.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.646+0000 7fc187af1640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc16c004810 0x7fc1801304c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:57:06.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.646+0000 7fc187af1640 10 monclient(hunting): picked mon.noname-a con 0x7fc16c004810 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:57:06.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.646+0000 7fc187af1640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc16c0053d0 0x7fc180130a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:57:06.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.646+0000 7fc187af1640 10 monclient(hunting): picked mon.noname-b con 0x7fc16c0053d0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:57:06.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.646+0000 7fc187af1640 10 monclient(hunting): start opening mon connection 2024-09-16T03:57:06.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.646+0000 7fc187af1640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fc180137390 con 0x7fc16c004810 2024-09-16T03:57:06.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.646+0000 7fc187af1640 10 monclient(hunting): start opening mon connection 2024-09-16T03:57:06.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.646+0000 7fc187af1640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc180137210 con 0x7fc16c0053d0 2024-09-16T03:57:06.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.646+0000 7fc187af1640 10 monclient(hunting): start opening mon connection 2024-09-16T03:57:06.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.646+0000 7fc187af1640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fc1800a97c0 con 0x7fc16c003a10 2024-09-16T03:57:06.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.646+0000 7fc187af1640 10 monclient(hunting): _renew_subs 2024-09-16T03:57:06.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.646+0000 7fc187af1640 10 monclient(hunting): authenticate will time out at 1873.618113s 2024-09-16T03:57:06.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.646+0000 7fc18588d640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc16c003a10 0x7fc1801cc010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:57:06.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.646+0000 7fc18588d640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc16c003a10 0x7fc1801cc010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:35998/0 (socket says 172.21.15.42:35998) 2024-09-16T03:57:06.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.646+0000 7fc18588d640 1 -- 172.21.15.42:0/2457273605 learned_addr learned my addr 172.21.15.42:0/2457273605 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:57:06.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.646+0000 7fc18588d640 10 monclient(hunting): get_auth_request con 0x7fc16c003a10 auth_method 0 2024-09-16T03:57:06.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.646+0000 7fc18588d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:57:06.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.646+0000 7fc18588d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:57:06.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.646+0000 7fc18588d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:57:06.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.646+0000 7fc18608e640 1 --2- 172.21.15.42:0/2457273605 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc16c0053d0 0x7fc180130a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:57:06.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.646+0000 7fc18608e640 10 monclient(hunting): get_auth_request con 0x7fc16c0053d0 auth_method 0 2024-09-16T03:57:06.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.646+0000 7fc18608e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:57:06.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.646+0000 7fc18608e640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:57:06.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.646+0000 7fc18608e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:57:06.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.646+0000 7fc18508c640 1 --2- 172.21.15.42:0/2457273605 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc16c004810 0x7fc1801304c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:57:06.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.646+0000 7fc18588d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:57:06.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.646+0000 7fc18588d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:57:06.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.646+0000 7fc18588d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:57:06.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.646+0000 7fc18508c640 10 monclient(hunting): get_auth_request con 0x7fc16c004810 auth_method 0 2024-09-16T03:57:06.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.646+0000 7fc18508c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:57:06.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.646+0000 7fc18508c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:57:06.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.646+0000 7fc18508c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:57:06.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.646+0000 7fc18508c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:57:06.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.646+0000 7fc18508c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:57:06.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.646+0000 7fc18508c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:57:06.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.646+0000 7fc18608e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:57:06.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.646+0000 7fc18608e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:57:06.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.646+0000 7fc18608e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:57:06.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.646+0000 7fc18588d640 10 monclient(hunting): handle_auth_done global_id 854360 payload 824 2024-09-16T03:57:06.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.646+0000 7fc18588d640 1 -- 172.21.15.42:0/2457273605 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc16c0053d0 msgr2=0x7fc180130a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:57:06.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.646+0000 7fc18588d640 1 --2- 172.21.15.42:0/2457273605 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc16c0053d0 0x7fc180130a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:57:06.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.646+0000 7fc18588d640 1 -- 172.21.15.42:0/2457273605 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc16c004810 msgr2=0x7fc1801304c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:57:06.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.646+0000 7fc18588d640 1 --2- 172.21.15.42:0/2457273605 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc16c004810 0x7fc1801304c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:57:06.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.646+0000 7fc18588d640 10 monclient: _finish_hunting 0 2024-09-16T03:57:06.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.646+0000 7fc18588d640 1 monclient: found mon.noname-c 2024-09-16T03:57:06.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.646+0000 7fc18588d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:57:06.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.646+0000 7fc18588d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:57:06.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.646+0000 7fc18588d640 1 -- 172.21.15.42:0/2457273605 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc18006f680 con 0x7fc16c003a10 2024-09-16T03:57:06.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.646+0000 7fc18588d640 1 --2- 172.21.15.42:0/2457273605 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc16c003a10 0x7fc1801cc010 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fc174000c10 tx=0x7fc174017da0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:57:06.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.646+0000 7fc1727fc640 10 client.?.objecter ms_handle_connect 0x7fc16c003a10 2024-09-16T03:57:06.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.646+0000 7fc1727fc640 10 client.?.objecter resend_mon_ops 2024-09-16T03:57:06.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.646+0000 7fc1727fc640 1 -- 172.21.15.42:0/2457273605 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc17401d070 con 0x7fc16c003a10 2024-09-16T03:57:06.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.646+0000 7fc1727fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:57:06.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.646+0000 7fc1727fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:57:06.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.646+0000 7fc1727fc640 10 monclient: dump: 2024-09-16T03:57:06.666 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:57:06.666 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:57:06.666 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:57:06.666 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:57:06.666 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:57:06.667 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:57:06.667 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:57:06.667 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:57:06.667 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:57:06.667 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:57:06.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.646+0000 7fc1727fc640 10 monclient: _finish_auth 0 2024-09-16T03:57:06.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.646+0000 7fc1727fc640 10 monclient: _check_auth_tickets 2024-09-16T03:57:06.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.646+0000 7fc1727fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:57:06.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.646+0000 7fc1727fc640 1 -- 172.21.15.42:0/2457273605 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc174000e50 con 0x7fc16c003a10 2024-09-16T03:57:06.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.646+0000 7fc187af1640 5 monclient: authenticate success, global_id 854360 2024-09-16T03:57:06.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.646+0000 7fc187af1640 10 monclient: _renew_subs 2024-09-16T03:57:06.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.646+0000 7fc187af1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:57:06.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.646+0000 7fc187af1640 1 -- 172.21.15.42:0/2457273605 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1800a3320 con 0x7fc16c003a10 2024-09-16T03:57:06.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.646+0000 7fc1727fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:57:06.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.646+0000 7fc1727fc640 1 -- 172.21.15.42:0/2457273605 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc174014cc0 con 0x7fc16c003a10 2024-09-16T03:57:06.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.646+0000 7fc1727fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:57:06.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.646+0000 7fc1727fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:57:06.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.646+0000 7fc1727fc640 10 monclient: dump: 2024-09-16T03:57:06.668 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:57:06.668 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:57:06.668 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:57:06.668 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:57:06.668 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:57:06.668 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:57:06.668 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:57:06.668 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:57:06.668 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:57:06.668 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:57:06.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.646+0000 7fc173fff640 4 set_mon_vals no callback set 2024-09-16T03:57:06.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.646+0000 7fc187af1640 10 client.854360.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:57:06.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.646+0000 7fc187af1640 10 monclient: _renew_subs 2024-09-16T03:57:06.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.646+0000 7fc187af1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:57:06.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.646+0000 7fc187af1640 1 -- 172.21.15.42:0/2457273605 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc180120730 con 0x7fc16c003a10 2024-09-16T03:57:06.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.646+0000 7fc187af1640 1 librados: init done 2024-09-16T03:57:06.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.650+0000 7fc1727fc640 1 -- 172.21.15.42:0/2457273605 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fc174019450 con 0x7fc16c003a10 2024-09-16T03:57:06.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.650+0000 7fc187af1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:57:06.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.650+0000 7fc1727fc640 1 --2- 172.21.15.42:0/2457273605 >> v2:172.21.15.154:6812/254845342 conn(0x7fc164037fd0 0x7fc16403a490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:57:06.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.650+0000 7fc187af1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:57:06.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.650+0000 7fc187af1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:57:06.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.650+0000 7fc187af1640 1 -- 172.21.15.42:0/2457273605 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc180137950 con 0x7fc16c003a10 2024-09-16T03:57:06.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.650+0000 7fc1727fc640 1 -- 172.21.15.42:0/2457273605 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(457..457 src has 330..457) ==== 5747+0+0 (secure 0 0 0) 0x7fc17404cc70 con 0x7fc16c003a10 2024-09-16T03:57:06.670 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.650+0000 7fc1727fc640 10 client.854360.objecter ms_dispatch 0x7fc180000ba0 osd_map(457..457 src has 330..457) 2024-09-16T03:57:06.670 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.650+0000 7fc1727fc640 3 client.854360.objecter handle_osd_map got epochs [457,457] > 0 2024-09-16T03:57:06.670 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.650+0000 7fc1727fc640 3 client.854360.objecter handle_osd_map decoding full epoch 457 2024-09-16T03:57:06.670 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.650+0000 7fc1727fc640 20 client.854360.objecter dump_active .. 0 homeless 2024-09-16T03:57:06.670 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.650+0000 7fc18508c640 1 --2- 172.21.15.42:0/2457273605 >> v2:172.21.15.154:6812/254845342 conn(0x7fc164037fd0 0x7fc16403a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:57:06.670 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.650+0000 7fc18508c640 10 monclient: get_auth_request con 0x7fc164037fd0 auth_method 0 2024-09-16T03:57:06.670 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.650+0000 7fc1727fc640 10 client.854360.objecter ms_handle_connect 0x7fc164037fd0 2024-09-16T03:57:06.670 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.650+0000 7fc18508c640 1 --2- 172.21.15.42:0/2457273605 >> v2:172.21.15.154:6812/254845342 conn(0x7fc164037fd0 0x7fc16403a490 secure :-1 s=READY pgs=3666 cs=0 l=1 rev1=1 crypto rx=0x7fc1801206c0 tx=0x7fc17c00b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:57:06.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.654+0000 7fc1727fc640 1 -- 172.21.15.42:0/2457273605 <== mon.2 v2:172.21.15.42:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7fc174019740 con 0x7fc16c003a10 2024-09-16T03:57:06.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.654+0000 7fc1727fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:57:06.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.654+0000 7fc1727fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:57:06.893 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.886+0000 7fc187af1640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:57:06.893 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.886+0000 7fc187af1640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:57:06.893 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.886+0000 7fc187af1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:57:06.893 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.886+0000 7fc187af1640 1 -- 172.21.15.42:0/2457273605 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fc1800a5750 con 0x7fc16c003a10 2024-09-16T03:57:06.894 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.890+0000 7fc1727fc640 1 -- 172.21.15.42:0/2457273605 <== mon.2 v2:172.21.15.42:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7fc17402a730 con 0x7fc16c003a10 2024-09-16T03:57:06.894 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.890+0000 7fc1727fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:57:06.894 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.890+0000 7fc1727fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:57:06.894 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":140,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:57:06.897 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.890+0000 7fc187af1640 10 librados: watch_flush enter 2024-09-16T03:57:06.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.890+0000 7fc187af1640 10 librados: watch_flush exit 2024-09-16T03:57:06.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.890+0000 7fc187af1640 20 client.854360.objecter shutdown clearing up homeless session... 2024-09-16T03:57:06.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.890+0000 7fc187af1640 10 client.854360.objecter successfully canceled tick 2024-09-16T03:57:06.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.890+0000 7fc187af1640 1 -- 172.21.15.42:0/2457273605 >> v2:172.21.15.154:6812/254845342 conn(0x7fc164037fd0 msgr2=0x7fc16403a490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:57:06.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.890+0000 7fc187af1640 1 --2- 172.21.15.42:0/2457273605 >> v2:172.21.15.154:6812/254845342 conn(0x7fc164037fd0 0x7fc16403a490 secure :-1 s=READY pgs=3666 cs=0 l=1 rev1=1 crypto rx=0x7fc1801206c0 tx=0x7fc17c00b040 comp rx=0 tx=0).stop 2024-09-16T03:57:06.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.890+0000 7fc187af1640 10 monclient: shutdown 2024-09-16T03:57:06.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.890+0000 7fc187af1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:57:06.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.890+0000 7fc187af1640 1 -- 172.21.15.42:0/2457273605 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc16c003a10 msgr2=0x7fc1801cc010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:57:06.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.890+0000 7fc187af1640 1 --2- 172.21.15.42:0/2457273605 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc16c003a10 0x7fc1801cc010 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fc174000c10 tx=0x7fc174017da0 comp rx=0 tx=0).stop 2024-09-16T03:57:06.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.890+0000 7fc187af1640 1 -- 172.21.15.42:0/2457273605 shutdown_connections 2024-09-16T03:57:06.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.890+0000 7fc187af1640 1 --2- 172.21.15.42:0/2457273605 >> v2:172.21.15.154:6812/254845342 conn(0x7fc164037fd0 0x7fc16403a490 unknown :-1 s=CLOSED pgs=3666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:57:06.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.890+0000 7fc187af1640 1 --2- 172.21.15.42:0/2457273605 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc16c0053d0 0x7fc180130a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:57:06.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.890+0000 7fc187af1640 1 --2- 172.21.15.42:0/2457273605 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc16c004810 0x7fc1801304c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:57:06.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.890+0000 7fc187af1640 1 --2- 172.21.15.42:0/2457273605 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc16c003a10 0x7fc1801cc010 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:57:06.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.890+0000 7fc187af1640 1 -- 172.21.15.42:0/2457273605 >> 172.21.15.42:0/2457273605 conn(0x7fc1800a9fb0 msgr2=0x7fc1800a8490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:57:06.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.890+0000 7fc187af1640 1 -- 172.21.15.42:0/2457273605 shutdown_connections 2024-09-16T03:57:06.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.890+0000 7fc187af1640 1 -- 172.21.15.42:0/2457273605 wait complete. 2024-09-16T03:57:06.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:06.890+0000 7fc187af1640 1 librados: shutdown 2024-09-16T03:57:06.916 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T03:57:06.916 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T03:57:06.916 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T03:57:07.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.030+0000 7f52c2ecf640 10 monclient: get_monmap_and_config 2024-09-16T03:57:07.036 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.030+0000 7f52c2ecf640 10 monclient: build_initial_monmap 2024-09-16T03:57:07.036 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.030+0000 7f52c2ecf640 10 monclient: monmap: 2024-09-16T03:57:07.036 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:57:07.036 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:57:07.036 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:57:07.034510+0000 2024-09-16T03:57:07.036 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:57:07.034510+0000 2024-09-16T03:57:07.036 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:57:07.036 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:57:07.037 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:57:07.037 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:57:07.037 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:57:07.037 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:57:07.037 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:57:07.037 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:57:07.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.030+0000 7f52c2ecf640 1 Processor -- start 2024-09-16T03:57:07.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.030+0000 7f52c2ecf640 1 -- start start 2024-09-16T03:57:07.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.030+0000 7f52c2ecf640 10 monclient: init 2024-09-16T03:57:07.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.030+0000 7f52c2ecf640 10 monclient: _reopen_session rank -1 2024-09-16T03:57:07.048 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.030+0000 7f52c2ecf640 10 monclient: _add_conns ranks=[4,1,3,0,2] 2024-09-16T03:57:07.048 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.030+0000 7f52c2ecf640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f52bc0a1330 0x7f52bc09f180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:57:07.048 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.030+0000 7f52c2ecf640 10 monclient(hunting): picked mon.noname-c con 0x7f52bc0a1330 addr v2:172.21.15.154:3300/0 2024-09-16T03:57:07.048 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.030+0000 7f52c2ecf640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f52bc0a1710 addr v1:172.21.15.154:6789/0 2024-09-16T03:57:07.048 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.030+0000 7f52c2ecf640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f52bc1393a0 0x7f52bc13b790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:57:07.048 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.030+0000 7f52c2ecf640 10 monclient(hunting): picked mon.noname-a con 0x7f52bc1393a0 addr v2:172.21.15.42:3300/0 2024-09-16T03:57:07.048 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.030+0000 7f52c2ecf640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f52bc09e4c0 con 0x7f52bc0a1710 2024-09-16T03:57:07.048 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.030+0000 7f52c2ecf640 10 monclient(hunting): start opening mon connection 2024-09-16T03:57:07.048 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.030+0000 7f52c2ecf640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f52bc09edf0 con 0x7f52bc1393a0 2024-09-16T03:57:07.048 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.030+0000 7f52c2ecf640 10 monclient(hunting): start opening mon connection 2024-09-16T03:57:07.048 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.030+0000 7f52c2ecf640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f52bc09cd60 con 0x7f52bc0a1330 2024-09-16T03:57:07.048 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.030+0000 7f52c2ecf640 10 monclient(hunting): _renew_subs 2024-09-16T03:57:07.048 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.030+0000 7f52c2ecf640 10 monclient(hunting): authenticate will time out at 1874.003936s 2024-09-16T03:57:07.048 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.034+0000 7f52c146c640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f52bc1393a0 0x7f52bc13b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:57:07.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.034+0000 7f52c146c640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f52bc1393a0 0x7f52bc13b790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:58176/0 (socket says 172.21.15.42:58176) 2024-09-16T03:57:07.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.034+0000 7f52c146c640 1 -- 172.21.15.42:0/377334828 learned_addr learned my addr 172.21.15.42:0/377334828 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:57:07.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.034+0000 7f52c146c640 10 monclient(hunting): get_auth_request con 0x7f52bc1393a0 auth_method 0 2024-09-16T03:57:07.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.034+0000 7f52c146c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:57:07.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.034+0000 7f52c146c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:57:07.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.034+0000 7f52c146c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:57:07.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.034+0000 7f52c146c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:57:07.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.034+0000 7f52c146c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:57:07.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.034+0000 7f52c146c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:57:07.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.034+0000 7f52c0c6b640 1 --2- 172.21.15.42:0/377334828 >> v2:172.21.15.154:3300/0 conn(0x7f52bc0a1330 0x7f52bc09f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:57:07.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.034+0000 7f52c0c6b640 10 monclient(hunting): get_auth_request con 0x7f52bc0a1330 auth_method 0 2024-09-16T03:57:07.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.034+0000 7f52c0c6b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:57:07.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.034+0000 7f52c0c6b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:57:07.050 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.034+0000 7f52c0c6b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:57:07.050 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.034+0000 7f52c146c640 10 monclient(hunting): handle_auth_done global_id 824805 payload 274 2024-09-16T03:57:07.050 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.034+0000 7f52c146c640 1 -- 172.21.15.42:0/377334828 >> v2:172.21.15.154:3300/0 conn(0x7f52bc0a1330 msgr2=0x7f52bc09f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:57:07.050 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.034+0000 7f52c146c640 1 --2- 172.21.15.42:0/377334828 >> v2:172.21.15.154:3300/0 conn(0x7f52bc0a1330 0x7f52bc09f180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:57:07.050 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.034+0000 7f52c146c640 1 -- 172.21.15.42:0/377334828 >> v1:172.21.15.154:6789/0 conn(0x7f52bc0a1710 legacy=0x7f52bc0a1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:57:07.050 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.034+0000 7f52c146c640 10 monclient: _finish_hunting 0 2024-09-16T03:57:07.050 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.034+0000 7f52c146c640 1 monclient: found mon.noname-a 2024-09-16T03:57:07.050 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.034+0000 7f52c146c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:57:07.050 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.034+0000 7f52c146c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:57:07.050 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.034+0000 7f52c146c640 1 -- 172.21.15.42:0/377334828 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52bc09f6c0 con 0x7f52bc1393a0 2024-09-16T03:57:07.050 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.034+0000 7f52c146c640 1 --2- 172.21.15.42:0/377334828 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f52bc1393a0 0x7f52bc13b790 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f52bc06add0 tx=0x7f52b401c870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=602aaaf521d274b5 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:57:07.050 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.034+0000 7f52bb7fe640 1 -- 172.21.15.42:0/377334828 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52b40a5640 con 0x7f52bc1393a0 2024-09-16T03:57:07.050 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.034+0000 7f52bb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:57:07.050 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.034+0000 7f52bb7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:57:07.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.034+0000 7f52bb7fe640 10 monclient: dump: 2024-09-16T03:57:07.051 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:57:07.051 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:57:07.051 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:57:07.051 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:57:07.051 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:57:07.051 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:57:07.051 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:57:07.051 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:57:07.051 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:57:07.051 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:57:07.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.034+0000 7f52bb7fe640 10 monclient: _finish_auth 0 2024-09-16T03:57:07.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.034+0000 7f52bb7fe640 10 monclient: _check_auth_tickets 2024-09-16T03:57:07.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.034+0000 7f52bb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:57:07.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.034+0000 7f52c2ecf640 5 monclient: authenticate success, global_id 824805 2024-09-16T03:57:07.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.034+0000 7f52c2ecf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:57:07.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.034+0000 7f52bb7fe640 1 -- 172.21.15.42:0/377334828 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f52b40a57e0 con 0x7f52bc1393a0 2024-09-16T03:57:07.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.034+0000 7f52bb7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:57:07.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.034+0000 7f52bb7fe640 1 -- 172.21.15.42:0/377334828 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52b40a5bb0 con 0x7f52bc1393a0 2024-09-16T03:57:07.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.034+0000 7f52bb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:57:07.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.034+0000 7f52bb7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:57:07.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.034+0000 7f52bb7fe640 10 monclient: dump: 2024-09-16T03:57:07.052 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:57:07.052 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:57:07.052 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:57:07.052 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:57:07.053 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:57:07.053 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:57:07.053 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:57:07.053 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:57:07.053 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:57:07.053 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:57:07.053 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.034+0000 7f52c2ecf640 10 monclient: get_monmap_and_config success 2024-09-16T03:57:07.053 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.034+0000 7f52c2ecf640 4 set_mon_vals no callback set 2024-09-16T03:57:07.053 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.034+0000 7f52c2ecf640 10 monclient: shutdown 2024-09-16T03:57:07.053 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.034+0000 7f52c2ecf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:57:07.053 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.034+0000 7f52c2ecf640 1 -- 172.21.15.42:0/377334828 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f52bc1393a0 msgr2=0x7f52bc13b790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:57:07.053 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.034+0000 7f52c2ecf640 1 --2- 172.21.15.42:0/377334828 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f52bc1393a0 0x7f52bc13b790 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f52bc06add0 tx=0x7f52b401c870 comp rx=0 tx=0).stop 2024-09-16T03:57:07.053 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.034+0000 7f52c2ecf640 1 -- 172.21.15.42:0/377334828 shutdown_connections 2024-09-16T03:57:07.053 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.034+0000 7f52c2ecf640 1 --2- 172.21.15.42:0/377334828 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f52bc1393a0 0x7f52bc13b790 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:57:07.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.034+0000 7f52c2ecf640 1 --2- 172.21.15.42:0/377334828 >> v2:172.21.15.154:3300/0 conn(0x7f52bc0a1330 0x7f52bc09f180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:57:07.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.034+0000 7f52c2ecf640 1 -- 172.21.15.42:0/377334828 >> 172.21.15.42:0/377334828 conn(0x7f52bc0a2420 msgr2=0x7f52bc0a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:57:07.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.034+0000 7f52c2ecf640 1 -- 172.21.15.42:0/377334828 shutdown_connections 2024-09-16T03:57:07.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.034+0000 7f52c2ecf640 1 -- 172.21.15.42:0/377334828 wait complete. 2024-09-16T03:57:07.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.038+0000 7f52c2ecf640 10 monclient: build_initial_monmap 2024-09-16T03:57:07.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.038+0000 7f52c2ecf640 10 monclient: monmap: 2024-09-16T03:57:07.054 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:57:07.054 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:57:07.054 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:57:07.054 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:57:07.054 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:57:07.054 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:57:07.054 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:57:07.054 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:57:07.055 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:57:07.055 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:57:07.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.038+0000 7f52c2ecf640 1 librados: starting msgr at 2024-09-16T03:57:07.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.038+0000 7f52c2ecf640 1 librados: starting objecter 2024-09-16T03:57:07.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.038+0000 7f52c2ecf640 1 Processor -- start 2024-09-16T03:57:07.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.038+0000 7f52c2ecf640 1 -- start start 2024-09-16T03:57:07.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.038+0000 7f52c2ecf640 1 librados: setting wanted keys 2024-09-16T03:57:07.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.038+0000 7f52c2ecf640 1 librados: calling monclient init 2024-09-16T03:57:07.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.038+0000 7f52c2ecf640 10 monclient: init 2024-09-16T03:57:07.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.038+0000 7f52c2ecf640 10 monclient: _reopen_session rank -1 2024-09-16T03:57:07.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.038+0000 7f52c2ecf640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T03:57:07.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.038+0000 7f52c2ecf640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f52bc0a1330 0x7f52bc1cd3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:57:07.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.038+0000 7f52c2ecf640 10 monclient(hunting): picked mon.noname-a con 0x7f52bc0a1330 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:57:07.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.038+0000 7f52c2ecf640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f52bc0a1710 0x7f52bc0704a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:57:07.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.038+0000 7f52c2ecf640 10 monclient(hunting): picked mon.noname-b con 0x7f52bc0a1710 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:57:07.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.038+0000 7f52c2ecf640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f52bc1393a0 0x7f52bc0709e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:57:07.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.038+0000 7f52c2ecf640 10 monclient(hunting): picked mon.noname-c con 0x7f52bc1393a0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:57:07.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.038+0000 7f52c2ecf640 10 monclient(hunting): start opening mon connection 2024-09-16T03:57:07.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.038+0000 7f52c2ecf640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f52bc13c8b0 con 0x7f52bc0a1330 2024-09-16T03:57:07.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.038+0000 7f52c2ecf640 10 monclient(hunting): start opening mon connection 2024-09-16T03:57:07.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.038+0000 7f52c2ecf640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f52bc13c730 con 0x7f52bc0a1710 2024-09-16T03:57:07.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.038+0000 7f52c2ecf640 10 monclient(hunting): start opening mon connection 2024-09-16T03:57:07.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.038+0000 7f52c2ecf640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f52bc13ca30 con 0x7f52bc1393a0 2024-09-16T03:57:07.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.038+0000 7f52c2ecf640 10 monclient(hunting): _renew_subs 2024-09-16T03:57:07.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.038+0000 7f52c2ecf640 10 monclient(hunting): authenticate will time out at 1874.009660s 2024-09-16T03:57:07.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.038+0000 7f52c146c640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f52bc1393a0 0x7f52bc0709e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:57:07.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.038+0000 7f52c0c6b640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f52bc0a1330 0x7f52bc1cd3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:57:07.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.038+0000 7f52c146c640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f52bc1393a0 0x7f52bc0709e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:36008/0 (socket says 172.21.15.42:36008) 2024-09-16T03:57:07.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.038+0000 7f52c0c6b640 10 monclient(hunting): get_auth_request con 0x7f52bc0a1330 auth_method 0 2024-09-16T03:57:07.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.038+0000 7f52c146c640 1 -- 172.21.15.42:0/1664593606 learned_addr learned my addr 172.21.15.42:0/1664593606 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:57:07.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.038+0000 7f52c0c6b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:57:07.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.038+0000 7f52c0c6b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:57:07.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.038+0000 7f52c0c6b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:57:07.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.038+0000 7f52c146c640 10 monclient(hunting): get_auth_request con 0x7f52bc1393a0 auth_method 0 2024-09-16T03:57:07.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.038+0000 7f52c146c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:57:07.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.038+0000 7f52c146c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:57:07.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.038+0000 7f52c146c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:57:07.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.038+0000 7f52c0c6b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:57:07.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.038+0000 7f52c0c6b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:57:07.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.038+0000 7f52c0c6b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:57:07.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.038+0000 7f52c146c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:57:07.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.038+0000 7f52c146c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:57:07.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.038+0000 7f52c146c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:57:07.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.038+0000 7f52bbfff640 1 --2- 172.21.15.42:0/1664593606 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f52bc0a1710 0x7f52bc0704a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:57:07.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.038+0000 7f52bbfff640 10 monclient(hunting): get_auth_request con 0x7f52bc0a1710 auth_method 0 2024-09-16T03:57:07.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.038+0000 7f52bbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:57:07.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.038+0000 7f52bbfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:57:07.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.038+0000 7f52bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:57:07.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.038+0000 7f52c0c6b640 10 monclient(hunting): handle_auth_done global_id 824808 payload 824 2024-09-16T03:57:07.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.038+0000 7f52c0c6b640 1 -- 172.21.15.42:0/1664593606 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f52bc1393a0 msgr2=0x7f52bc0709e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:57:07.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.038+0000 7f52c0c6b640 1 --2- 172.21.15.42:0/1664593606 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f52bc1393a0 0x7f52bc0709e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:57:07.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.038+0000 7f52c0c6b640 1 -- 172.21.15.42:0/1664593606 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f52bc0a1710 msgr2=0x7f52bc0704a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:57:07.059 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.038+0000 7f52c0c6b640 1 --2- 172.21.15.42:0/1664593606 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f52bc0a1710 0x7f52bc0704a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:57:07.059 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.038+0000 7f52c0c6b640 10 monclient: _finish_hunting 0 2024-09-16T03:57:07.060 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.038+0000 7f52c0c6b640 1 monclient: found mon.noname-a 2024-09-16T03:57:07.060 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.038+0000 7f52c0c6b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:57:07.060 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.038+0000 7f52c0c6b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:57:07.060 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.038+0000 7f52c0c6b640 1 -- 172.21.15.42:0/1664593606 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52bc11d780 con 0x7f52bc0a1330 2024-09-16T03:57:07.060 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.038+0000 7f52c146c640 1 --2- 172.21.15.42:0/1664593606 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f52bc1393a0 0x7f52bc0709e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:57:07.060 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.038+0000 7f52bbfff640 1 --2- 172.21.15.42:0/1664593606 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f52bc0a1710 0x7f52bc0704a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T03:57:07.060 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.038+0000 7f52c0c6b640 1 --2- 172.21.15.42:0/1664593606 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f52bc0a1330 0x7f52bc1cd3c0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f52ac00d5f0 tx=0x7f52ac00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:57:07.060 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.038+0000 7f52b97fa640 10 client.?.objecter ms_handle_connect 0x7f52bc0a1330 2024-09-16T03:57:07.060 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.038+0000 7f52b97fa640 10 client.?.objecter resend_mon_ops 2024-09-16T03:57:07.060 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.038+0000 7f52b97fa640 1 -- 172.21.15.42:0/1664593606 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52ac0125f0 con 0x7f52bc0a1330 2024-09-16T03:57:07.060 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.038+0000 7f52b97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:57:07.060 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.038+0000 7f52b97fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:57:07.060 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.038+0000 7f52b97fa640 10 monclient: dump: 2024-09-16T03:57:07.060 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:57:07.061 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:57:07.061 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:57:07.061 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:57:07.061 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:57:07.061 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:57:07.061 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:57:07.061 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:57:07.061 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:57:07.061 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:57:07.061 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.038+0000 7f52b97fa640 10 monclient: _finish_auth 0 2024-09-16T03:57:07.061 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.038+0000 7f52b97fa640 10 monclient: _check_auth_tickets 2024-09-16T03:57:07.061 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.038+0000 7f52b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:57:07.061 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.038+0000 7f52b97fa640 1 -- 172.21.15.42:0/1664593606 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f52ac019070 con 0x7f52bc0a1330 2024-09-16T03:57:07.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.038+0000 7f52b97fa640 10 monclient: handle_config config(0 keys) 2024-09-16T03:57:07.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.038+0000 7f52b97fa640 1 -- 172.21.15.42:0/1664593606 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52ac0129b0 con 0x7f52bc0a1330 2024-09-16T03:57:07.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.038+0000 7f52b97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:57:07.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.038+0000 7f52b97fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:57:07.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.038+0000 7f52b97fa640 10 monclient: dump: 2024-09-16T03:57:07.062 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:57:07.062 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:57:07.062 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:57:07.062 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:57:07.062 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:57:07.062 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:57:07.062 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:57:07.062 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:57:07.062 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:57:07.062 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:57:07.063 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.038+0000 7f52baffd640 4 set_mon_vals no callback set 2024-09-16T03:57:07.063 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.038+0000 7f52c2ecf640 5 monclient: authenticate success, global_id 824808 2024-09-16T03:57:07.063 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.038+0000 7f52c2ecf640 10 monclient: _renew_subs 2024-09-16T03:57:07.063 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.038+0000 7f52c2ecf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:57:07.063 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.038+0000 7f52c2ecf640 1 -- 172.21.15.42:0/1664593606 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52bc13b340 con 0x7f52bc0a1330 2024-09-16T03:57:07.063 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.038+0000 7f52c2ecf640 10 client.824808.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:57:07.063 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.038+0000 7f52c2ecf640 10 monclient: _renew_subs 2024-09-16T03:57:07.063 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.038+0000 7f52c2ecf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:57:07.063 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.038+0000 7f52c2ecf640 1 -- 172.21.15.42:0/1664593606 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52bc072140 con 0x7f52bc0a1330 2024-09-16T03:57:07.063 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.038+0000 7f52c2ecf640 1 librados: init done 2024-09-16T03:57:07.063 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.038+0000 7f52c2ecf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:57:07.063 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.038+0000 7f52c2ecf640 1 --2- 172.21.15.42:0/1664593606 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f52bc1cd900 0x7f52bc02c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:57:07.063 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.038+0000 7f52c2ecf640 10 monclient(hunting): start opening mon connection 2024-09-16T03:57:07.063 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.038+0000 7f52c2ecf640 1 -- 172.21.15.42:0/1664593606 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f52bc09f180 con 0x7f52bc1cd900 2024-09-16T03:57:07.064 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.042+0000 7f52b97fa640 1 -- 172.21.15.42:0/1664593606 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f52ac012b50 con 0x7f52bc0a1330 2024-09-16T03:57:07.064 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.042+0000 7f52bbfff640 1 --2- 172.21.15.42:0/1664593606 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f52bc1cd900 0x7f52bc02c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:57:07.064 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.042+0000 7f52bbfff640 10 monclient: get_auth_request con 0x7f52bc1cd900 auth_method 0 2024-09-16T03:57:07.064 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.042+0000 7f52bbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:57:07.064 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.042+0000 7f52bbfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:57:07.064 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.042+0000 7f52bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:57:07.064 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.042+0000 7f52bbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:57:07.064 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.042+0000 7f52bbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:57:07.064 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.042+0000 7f52bbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:57:07.064 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.042+0000 7f52b97fa640 1 --2- 172.21.15.42:0/1664593606 >> v2:172.21.15.154:6812/254845342 conn(0x7f52a8037810 0x7f52a8039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:57:07.064 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.042+0000 7f52b97fa640 1 -- 172.21.15.42:0/1664593606 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(457..457 src has 330..457) ==== 5747+0+0 (secure 0 0 0) 0x7f52ac049310 con 0x7f52bc0a1330 2024-09-16T03:57:07.065 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.042+0000 7f52b97fa640 10 client.824808.objecter ms_dispatch 0x7f52bc000ba0 osd_map(457..457 src has 330..457) 2024-09-16T03:57:07.065 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.042+0000 7f52b97fa640 3 client.824808.objecter handle_osd_map got epochs [457,457] > 0 2024-09-16T03:57:07.065 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.042+0000 7f52b97fa640 3 client.824808.objecter handle_osd_map decoding full epoch 457 2024-09-16T03:57:07.065 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.042+0000 7f52bbfff640 10 monclient(hunting): handle_auth_done global_id 824811 payload 824 2024-09-16T03:57:07.065 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.042+0000 7f52bbfff640 1 -- 172.21.15.42:0/1664593606 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f52bc138aa0 con 0x7f52bc1cd900 2024-09-16T03:57:07.065 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.042+0000 7f52b97fa640 20 client.824808.objecter dump_active .. 0 homeless 2024-09-16T03:57:07.065 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.042+0000 7f52b97fa640 10 client.824808.objecter ms_handle_connect 0x7f52bc1cd900 2024-09-16T03:57:07.065 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.042+0000 7f52b97fa640 10 client.824808.objecter resend_mon_ops 2024-09-16T03:57:07.065 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.042+0000 7f52bbfff640 1 --2- 172.21.15.42:0/1664593606 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f52bc1cd900 0x7f52bc02c650 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f52bc071c20 tx=0x7f52b000e960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:57:07.065 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.042+0000 7f52c146c640 1 --2- 172.21.15.42:0/1664593606 >> v2:172.21.15.154:6812/254845342 conn(0x7f52a8037810 0x7f52a8039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:57:07.065 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.042+0000 7f52c146c640 10 monclient: get_auth_request con 0x7f52a8037810 auth_method 0 2024-09-16T03:57:07.065 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.042+0000 7f52b97fa640 1 -- 172.21.15.42:0/1664593606 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52b0007030 con 0x7f52bc1cd900 2024-09-16T03:57:07.065 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.042+0000 7f52b97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:57:07.065 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.042+0000 7f52b97fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:57:07.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.042+0000 7f52b97fa640 10 monclient: dump: 2024-09-16T03:57:07.066 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:57:07.066 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:57:07.066 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:57:07.066 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:57:07.066 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:57:07.066 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:57:07.066 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:57:07.066 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:57:07.066 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:57:07.066 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:57:07.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.042+0000 7f52b97fa640 1 -- 172.21.15.42:0/1664593606 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f52b0007030 con 0x7f52bc1cd900 2024-09-16T03:57:07.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.042+0000 7f52b97fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:57:07.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.042+0000 7f52b97fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:57:07.067 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.042+0000 7f52b97fa640 1 -- 172.21.15.42:0/1664593606 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f52bc1cd900 msgr2=0x7f52bc02c650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:57:07.067 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.042+0000 7f52b97fa640 1 --2- 172.21.15.42:0/1664593606 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f52bc1cd900 0x7f52bc02c650 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f52bc071c20 tx=0x7f52b000e960 comp rx=0 tx=0).stop 2024-09-16T03:57:07.067 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.042+0000 7f52b97fa640 1 -- 172.21.15.42:0/1664593606 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f52bc1cd900 msgr2=0x7f52bc02c650 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:57:07.067 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.042+0000 7f52b97fa640 1 --2- 172.21.15.42:0/1664593606 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f52bc1cd900 0x7f52bc02c650 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:57:07.067 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.042+0000 7f52b97fa640 10 client.824808.objecter ms_handle_connect 0x7f52a8037810 2024-09-16T03:57:07.067 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.042+0000 7f52c146c640 1 --2- 172.21.15.42:0/1664593606 >> v2:172.21.15.154:6812/254845342 conn(0x7f52a8037810 0x7f52a8039cd0 secure :-1 s=READY pgs=3667 cs=0 l=1 rev1=1 crypto rx=0x7f52b401a4a0 tx=0x7f52b40a7070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:57:07.069 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.046+0000 7f52c2ecf640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:57:07.069 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.046+0000 7f52c2ecf640 1 --2- 172.21.15.42:0/1664593606 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f52bc02cb90 0x7f52bc02cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:57:07.069 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.046+0000 7f52c2ecf640 10 monclient(hunting): start opening mon connection 2024-09-16T03:57:07.069 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.046+0000 7f52c2ecf640 1 -- 172.21.15.42:0/1664593606 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f52bc069040 con 0x7f52bc02cb90 2024-09-16T03:57:07.069 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.046+0000 7f52bbfff640 1 --2- 172.21.15.42:0/1664593606 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f52bc02cb90 0x7f52bc02cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:57:07.070 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.046+0000 7f52bbfff640 10 monclient: get_auth_request con 0x7f52bc02cb90 auth_method 0 2024-09-16T03:57:07.070 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.046+0000 7f52bbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:57:07.070 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.046+0000 7f52bbfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:57:07.070 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.046+0000 7f52bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:57:07.070 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.046+0000 7f52bbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:57:07.070 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.046+0000 7f52bbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:57:07.070 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.046+0000 7f52bbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:57:07.070 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.046+0000 7f52bbfff640 10 monclient(hunting): handle_auth_done global_id 824814 payload 824 2024-09-16T03:57:07.070 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.046+0000 7f52bbfff640 1 -- 172.21.15.42:0/1664593606 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f52bc134700 con 0x7f52bc02cb90 2024-09-16T03:57:07.070 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.046+0000 7f52bbfff640 1 --2- 172.21.15.42:0/1664593606 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f52bc02cb90 0x7f52bc02cfa0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f52bc071c20 tx=0x7f52b0003770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:57:07.070 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.046+0000 7f52b97fa640 10 client.824808.objecter ms_handle_connect 0x7f52bc02cb90 2024-09-16T03:57:07.070 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.046+0000 7f52b97fa640 10 client.824808.objecter resend_mon_ops 2024-09-16T03:57:07.070 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.050+0000 7f52b97fa640 1 -- 172.21.15.42:0/1664593606 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52b001eb80 con 0x7f52bc02cb90 2024-09-16T03:57:07.071 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.050+0000 7f52b97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:57:07.071 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.050+0000 7f52b97fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:57:07.071 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.050+0000 7f52b97fa640 10 monclient: dump: 2024-09-16T03:57:07.071 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:57:07.071 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:57:07.071 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:57:07.071 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:57:07.071 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:57:07.071 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:57:07.071 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:57:07.071 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:57:07.071 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:57:07.071 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:57:07.071 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.050+0000 7f52b97fa640 1 -- 172.21.15.42:0/1664593606 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4882 (secure 0 0 0) 0x7f52b001eb80 con 0x7f52bc02cb90 2024-09-16T03:57:07.071 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.050+0000 7f52b97fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:57:07.072 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.050+0000 7f52b97fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:57:07.072 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.050+0000 7f52b97fa640 1 -- 172.21.15.42:0/1664593606 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f52bc02cb90 msgr2=0x7f52bc02cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:57:07.072 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.050+0000 7f52b97fa640 1 --2- 172.21.15.42:0/1664593606 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f52bc02cb90 0x7f52bc02cfa0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f52bc071c20 tx=0x7f52b0003770 comp rx=0 tx=0).stop 2024-09-16T03:57:07.072 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.050+0000 7f52b97fa640 1 -- 172.21.15.42:0/1664593606 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f52bc02cb90 msgr2=0x7f52bc02cfa0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:57:07.072 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.050+0000 7f52b97fa640 1 --2- 172.21.15.42:0/1664593606 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f52bc02cb90 0x7f52bc02cfa0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:57:07.072 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:57:07.072 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:57:07.072 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:57:07.073 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "leader", 2024-09-16T03:57:07.073 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 140, 2024-09-16T03:57:07.073 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 316905, 2024-09-16T03:57:07.073 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:57:07.073 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:57:07.073 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T03:57:07.073 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T03:57:07.073 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:57:07.073 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 0, 2024-09-16T03:57:07.073 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:57:07.073 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:57:07.073 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:57:07.074 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:57:07.074 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:57:07.074 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:57:07.074 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:57:07.074 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:57:07.074 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:57:07.074 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:57:07.074 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:57:07.074 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:57:07.074 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:57:07.074 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:57:07.074 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:57:07.074 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:57:07.075 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:57:07.075 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:57:07.075 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:57:07.075 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:57:07.075 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:57:07.075 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:57:07.075 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:57:07.075 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:57:07.075 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:57:07.075 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:57:07.075 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:57:07.075 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:57:07.075 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:57:07.076 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:57:07.076 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:57:07.076 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:57:07.076 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:57:07.076 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:57:07.076 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:57:07.076 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:57:07.076 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:57:07.076 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:57:07.076 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:57:07.076 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:57:07.076 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:57:07.077 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:57:07.077 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:57:07.077 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:57:07.077 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:57:07.077 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:57:07.077 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:57:07.077 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:57:07.077 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:57:07.077 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:57:07.077 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:57:07.077 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:57:07.077 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:57:07.077 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:57:07.078 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:57:07.078 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:57:07.078 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:57:07.078 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:57:07.078 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:57:07.078 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:57:07.078 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:57:07.078 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:57:07.078 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:57:07.078 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:57:07.078 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:57:07.078 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:57:07.078 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:57:07.078 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:57:07.079 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:57:07.079 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:57:07.079 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:57:07.079 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:57:07.079 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:57:07.079 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:57:07.079 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:57:07.079 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:57:07.079 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:57:07.079 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:57:07.079 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:57:07.079 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:57:07.079 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:57:07.079 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:57:07.080 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:57:07.080 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:57:07.080 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:57:07.080 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:57:07.080 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:57:07.080 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:57:07.080 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:57:07.080 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:57:07.080 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:57:07.080 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:57:07.080 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:57:07.080 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:57:07.080 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:57:07.081 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:57:07.081 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:57:07.081 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:57:07.081 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:57:07.081 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:57:07.081 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:57:07.081 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:57:07.081 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:57:07.081 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:57:07.081 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:57:07.081 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:57:07.081 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:57:07.081 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:57:07.081 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:57:07.081 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:57:07.082 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:57:07.082 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:57:07.082 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:57:07.082 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:57:07.082 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:57:07.082 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:57:07.082 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:57:07.082 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:57:07.082 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:57:07.082 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:57:07.082 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:57:07.082 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:57:07.082 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:57:07.082 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:57:07.083 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:57:07.083 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:57:07.083 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:57:07.083 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:57:07.083 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:57:07.083 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:57:07.083 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:57:07.083 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:57:07.083 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:57:07.083 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:57:07.083 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:57:07.083 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:57:07.083 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:57:07.083 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:57:07.084 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:57:07.084 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:57:07.084 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:57:07.084 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T03:57:07.084 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:57:07.084 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:57:07.084 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:57:07.084 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 6 2024-09-16T03:57:07.084 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:57:07.084 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:57:07.084 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:57:07.084 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:57:07.084 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:57:07.084 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:57:07.085 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 8 2024-09-16T03:57:07.085 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:57:07.085 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:57:07.085 INFO:teuthology.orchestra.run.smithi042.stdout: "mgr": [ 2024-09-16T03:57:07.085 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:57:07.085 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:57:07.085 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:57:07.085 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:57:07.085 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:57:07.085 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:57:07.085 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:57:07.085 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:57:07.085 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:57:07.087 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.050+0000 7f52c2ecf640 10 librados: watch_flush enter 2024-09-16T03:57:07.087 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.050+0000 7f52c2ecf640 10 librados: watch_flush exit 2024-09-16T03:57:07.087 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.050+0000 7f52c2ecf640 20 client.824808.objecter shutdown clearing up homeless session... 2024-09-16T03:57:07.087 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.050+0000 7f52c2ecf640 10 client.824808.objecter successfully canceled tick 2024-09-16T03:57:07.087 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.050+0000 7f52c2ecf640 1 -- 172.21.15.42:0/1664593606 >> v2:172.21.15.154:6812/254845342 conn(0x7f52a8037810 msgr2=0x7f52a8039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:57:07.087 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.050+0000 7f52c2ecf640 1 --2- 172.21.15.42:0/1664593606 >> v2:172.21.15.154:6812/254845342 conn(0x7f52a8037810 0x7f52a8039cd0 secure :-1 s=READY pgs=3667 cs=0 l=1 rev1=1 crypto rx=0x7f52b401a4a0 tx=0x7f52b40a7070 comp rx=0 tx=0).stop 2024-09-16T03:57:07.087 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.050+0000 7f52c2ecf640 10 monclient: shutdown 2024-09-16T03:57:07.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.050+0000 7f52c2ecf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:57:07.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.050+0000 7f52c2ecf640 1 -- 172.21.15.42:0/1664593606 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f52bc0a1330 msgr2=0x7f52bc1cd3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:57:07.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.050+0000 7f52c2ecf640 1 --2- 172.21.15.42:0/1664593606 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f52bc0a1330 0x7f52bc1cd3c0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f52ac00d5f0 tx=0x7f52ac00dab0 comp rx=0 tx=0).stop 2024-09-16T03:57:07.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.050+0000 7f52c146c640 1 -- 172.21.15.42:0/1664593606 reap_dead start 2024-09-16T03:57:07.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.050+0000 7f52c146c640 1 -- 172.21.15.42:0/1664593606 reap_dead start 2024-09-16T03:57:07.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.050+0000 7f52c2ecf640 1 -- 172.21.15.42:0/1664593606 shutdown_connections 2024-09-16T03:57:07.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.050+0000 7f52c2ecf640 1 -- 172.21.15.42:0/1664593606 >> 172.21.15.42:0/1664593606 conn(0x7f52bc0a2420 msgr2=0x7f52bc13a080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:57:07.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.050+0000 7f52c2ecf640 1 -- 172.21.15.42:0/1664593606 shutdown_connections 2024-09-16T03:57:07.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.050+0000 7f52c2ecf640 1 -- 172.21.15.42:0/1664593606 wait complete. 2024-09-16T03:57:07.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.050+0000 7f52c2ecf640 1 librados: shutdown 2024-09-16T03:57:07.088 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:57:07.089 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T03:57:07.089 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T03:57:07.205 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.198+0000 7fdf85ca2640 10 monclient: get_monmap_and_config 2024-09-16T03:57:07.205 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.198+0000 7fdf85ca2640 10 monclient: build_initial_monmap 2024-09-16T03:57:07.205 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.198+0000 7fdf85ca2640 10 monclient: monmap: 2024-09-16T03:57:07.206 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:57:07.206 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:57:07.206 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:57:07.203507+0000 2024-09-16T03:57:07.206 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:57:07.203507+0000 2024-09-16T03:57:07.206 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:57:07.206 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:57:07.206 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:57:07.206 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:57:07.206 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:57:07.206 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:57:07.206 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:57:07.207 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:57:07.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.202+0000 7fdf85ca2640 1 Processor -- start 2024-09-16T03:57:07.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.202+0000 7fdf85ca2640 1 -- start start 2024-09-16T03:57:07.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.202+0000 7fdf85ca2640 10 monclient: init 2024-09-16T03:57:07.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.202+0000 7fdf85ca2640 10 monclient: _reopen_session rank -1 2024-09-16T03:57:07.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.202+0000 7fdf85ca2640 10 monclient: _add_conns ranks=[1,4,3,2,0] 2024-09-16T03:57:07.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.202+0000 7fdf85ca2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fdf8009fb00 addr v1:172.21.15.154:6789/0 2024-09-16T03:57:07.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.202+0000 7fdf85ca2640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fdf8009c160 0x7fdf80139120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:57:07.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.202+0000 7fdf85ca2640 10 monclient(hunting): picked mon.noname-c con 0x7fdf8009c160 addr v2:172.21.15.154:3300/0 2024-09-16T03:57:07.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.202+0000 7fdf85ca2640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fdf8009c5d0 0x7fdf8013b670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:57:07.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.202+0000 7fdf85ca2640 10 monclient(hunting): picked mon.noname-a con 0x7fdf8009c5d0 addr v2:172.21.15.42:3300/0 2024-09-16T03:57:07.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.202+0000 7fdf85ca2640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdf800a1e20 con 0x7fdf8009fb00 2024-09-16T03:57:07.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.202+0000 7fdf85ca2640 10 monclient(hunting): start opening mon connection 2024-09-16T03:57:07.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.202+0000 7fdf85ca2640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7fdf8009e250 con 0x7fdf8009c5d0 2024-09-16T03:57:07.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.202+0000 7fdf85ca2640 10 monclient(hunting): start opening mon connection 2024-09-16T03:57:07.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.202+0000 7fdf85ca2640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fdf8009ea30 con 0x7fdf8009c160 2024-09-16T03:57:07.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.202+0000 7fdf85ca2640 10 monclient(hunting): _renew_subs 2024-09-16T03:57:07.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.202+0000 7fdf85ca2640 10 monclient(hunting): authenticate will time out at 1874.173110s 2024-09-16T03:57:07.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.202+0000 7fdf7ffff640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fdf8009c5d0 0x7fdf8013b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:57:07.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.202+0000 7fdf7ffff640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fdf8009c5d0 0x7fdf8013b670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:58212/0 (socket says 172.21.15.42:58212) 2024-09-16T03:57:07.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.202+0000 7fdf7ffff640 1 -- 172.21.15.42:0/3198984084 learned_addr learned my addr 172.21.15.42:0/3198984084 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:57:07.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.202+0000 7fdf7ffff640 10 monclient(hunting): get_auth_request con 0x7fdf8009c5d0 auth_method 0 2024-09-16T03:57:07.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.202+0000 7fdf7ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:57:07.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.202+0000 7fdf7ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:57:07.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.202+0000 7fdf7ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:57:07.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.202+0000 7fdf7ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:57:07.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.202+0000 7fdf7ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:57:07.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.202+0000 7fdf7ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:57:07.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.202+0000 7fdf7effd640 1 --2- 172.21.15.42:0/3198984084 >> v2:172.21.15.154:3300/0 conn(0x7fdf8009c160 0x7fdf80139120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:57:07.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.202+0000 7fdf7effd640 10 monclient(hunting): get_auth_request con 0x7fdf8009c160 auth_method 0 2024-09-16T03:57:07.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.202+0000 7fdf7effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:57:07.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.202+0000 7fdf7effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:57:07.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.202+0000 7fdf7effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:57:07.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.202+0000 7fdf7ffff640 10 monclient(hunting): handle_auth_done global_id 824817 payload 274 2024-09-16T03:57:07.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.202+0000 7fdf7ffff640 1 -- 172.21.15.42:0/3198984084 >> v2:172.21.15.154:3300/0 conn(0x7fdf8009c160 msgr2=0x7fdf80139120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:57:07.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.202+0000 7fdf7ffff640 1 --2- 172.21.15.42:0/3198984084 >> v2:172.21.15.154:3300/0 conn(0x7fdf8009c160 0x7fdf80139120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:57:07.214 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.202+0000 7fdf7ffff640 1 -- 172.21.15.42:0/3198984084 >> v1:172.21.15.154:6789/0 conn(0x7fdf8009fb00 legacy=0x7fdf8009ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:57:07.214 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.202+0000 7fdf7ffff640 10 monclient: _finish_hunting 0 2024-09-16T03:57:07.214 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.202+0000 7fdf7ffff640 1 monclient: found mon.noname-a 2024-09-16T03:57:07.214 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.202+0000 7fdf7ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:57:07.214 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.202+0000 7fdf7ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:57:07.214 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.202+0000 7fdf7ffff640 1 -- 172.21.15.42:0/3198984084 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf8009fee0 con 0x7fdf8009c5d0 2024-09-16T03:57:07.214 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.202+0000 7fdf7ffff640 1 --2- 172.21.15.42:0/3198984084 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdf8009c5d0 0x7fdf8013b670 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fdf8006c650 tx=0x7fdf7801c870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d24535f2b2a017c5 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:57:07.214 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.202+0000 7fdf7e7fc640 1 -- 172.21.15.42:0/3198984084 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf780a56b0 con 0x7fdf8009c5d0 2024-09-16T03:57:07.214 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.202+0000 7fdf7e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:57:07.214 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.202+0000 7fdf7e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:57:07.214 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.202+0000 7fdf7e7fc640 10 monclient: dump: 2024-09-16T03:57:07.214 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:57:07.214 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:57:07.215 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:57:07.215 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:57:07.215 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:57:07.215 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:57:07.215 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:57:07.215 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:57:07.215 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:57:07.215 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:57:07.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.202+0000 7fdf7e7fc640 10 monclient: _finish_auth 0 2024-09-16T03:57:07.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.202+0000 7fdf7e7fc640 10 monclient: _check_auth_tickets 2024-09-16T03:57:07.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.202+0000 7fdf7e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:57:07.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.202+0000 7fdf7e7fc640 1 -- 172.21.15.42:0/3198984084 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdf780a5850 con 0x7fdf8009c5d0 2024-09-16T03:57:07.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.202+0000 7fdf85ca2640 5 monclient: authenticate success, global_id 824817 2024-09-16T03:57:07.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.202+0000 7fdf85ca2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:57:07.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.202+0000 7fdf7e7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:57:07.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.202+0000 7fdf85ca2640 10 monclient: get_monmap_and_config success 2024-09-16T03:57:07.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.202+0000 7fdf7e7fc640 1 -- 172.21.15.42:0/3198984084 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf780a5c20 con 0x7fdf8009c5d0 2024-09-16T03:57:07.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.202+0000 7fdf85ca2640 4 set_mon_vals no callback set 2024-09-16T03:57:07.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.202+0000 7fdf7e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:57:07.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.202+0000 7fdf7e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:57:07.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.202+0000 7fdf7e7fc640 10 monclient: dump: 2024-09-16T03:57:07.216 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:57:07.216 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:57:07.216 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:57:07.216 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:57:07.216 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:57:07.216 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:57:07.217 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:57:07.217 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:57:07.217 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:57:07.217 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:57:07.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.206+0000 7fdf85ca2640 10 monclient: shutdown 2024-09-16T03:57:07.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.206+0000 7fdf85ca2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:57:07.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.206+0000 7fdf85ca2640 1 -- 172.21.15.42:0/3198984084 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdf8009c5d0 msgr2=0x7fdf8013b670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:57:07.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.206+0000 7fdf85ca2640 1 --2- 172.21.15.42:0/3198984084 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdf8009c5d0 0x7fdf8013b670 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fdf8006c650 tx=0x7fdf7801c870 comp rx=0 tx=0).stop 2024-09-16T03:57:07.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.206+0000 7fdf85ca2640 1 -- 172.21.15.42:0/3198984084 shutdown_connections 2024-09-16T03:57:07.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.206+0000 7fdf85ca2640 1 --2- 172.21.15.42:0/3198984084 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdf8009c5d0 0x7fdf8013b670 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:57:07.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.206+0000 7fdf85ca2640 1 --2- 172.21.15.42:0/3198984084 >> v2:172.21.15.154:3300/0 conn(0x7fdf8009c160 0x7fdf80139120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:57:07.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.206+0000 7fdf85ca2640 1 -- 172.21.15.42:0/3198984084 >> 172.21.15.42:0/3198984084 conn(0x7fdf800a2420 msgr2=0x7fdf800a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:57:07.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.206+0000 7fdf85ca2640 1 -- 172.21.15.42:0/3198984084 shutdown_connections 2024-09-16T03:57:07.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.206+0000 7fdf85ca2640 1 -- 172.21.15.42:0/3198984084 wait complete. 2024-09-16T03:57:07.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.206+0000 7fdf85ca2640 10 monclient: build_initial_monmap 2024-09-16T03:57:07.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.206+0000 7fdf85ca2640 10 monclient: monmap: 2024-09-16T03:57:07.218 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:57:07.218 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:57:07.218 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:57:07.218 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:57:07.218 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:57:07.218 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:57:07.218 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:57:07.218 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:57:07.218 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:57:07.218 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:57:07.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.206+0000 7fdf85ca2640 1 librados: starting msgr at 2024-09-16T03:57:07.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.206+0000 7fdf85ca2640 1 librados: starting objecter 2024-09-16T03:57:07.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.206+0000 7fdf85ca2640 1 Processor -- start 2024-09-16T03:57:07.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.206+0000 7fdf85ca2640 1 -- start start 2024-09-16T03:57:07.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.206+0000 7fdf85ca2640 1 librados: setting wanted keys 2024-09-16T03:57:07.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.206+0000 7fdf85ca2640 1 librados: calling monclient init 2024-09-16T03:57:07.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.206+0000 7fdf85ca2640 10 monclient: init 2024-09-16T03:57:07.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.206+0000 7fdf85ca2640 10 monclient: _reopen_session rank -1 2024-09-16T03:57:07.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.206+0000 7fdf85ca2640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T03:57:07.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.206+0000 7fdf85ca2640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdf8009c160 0x7fdf801ceef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:57:07.227 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.206+0000 7fdf85ca2640 10 monclient(hunting): picked mon.noname-b con 0x7fdf8009c160 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:57:07.227 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.206+0000 7fdf85ca2640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fdf8009c5d0 0x7fdf801359b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:57:07.227 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.206+0000 7fdf85ca2640 10 monclient(hunting): picked mon.noname-c con 0x7fdf8009c5d0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:57:07.227 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.206+0000 7fdf85ca2640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdf8009fb00 0x7fdf80135ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:57:07.227 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.206+0000 7fdf85ca2640 10 monclient(hunting): picked mon.noname-a con 0x7fdf8009fb00 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:57:07.227 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.206+0000 7fdf85ca2640 10 monclient(hunting): start opening mon connection 2024-09-16T03:57:07.227 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.206+0000 7fdf85ca2640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fdf800a3330 con 0x7fdf8009fb00 2024-09-16T03:57:07.227 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.206+0000 7fdf85ca2640 10 monclient(hunting): start opening mon connection 2024-09-16T03:57:07.227 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.206+0000 7fdf85ca2640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fdf800a31b0 con 0x7fdf8009c160 2024-09-16T03:57:07.227 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.206+0000 7fdf85ca2640 10 monclient(hunting): start opening mon connection 2024-09-16T03:57:07.227 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.206+0000 7fdf85ca2640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fdf8009d380 con 0x7fdf8009c5d0 2024-09-16T03:57:07.227 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.206+0000 7fdf85ca2640 10 monclient(hunting): _renew_subs 2024-09-16T03:57:07.227 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.206+0000 7fdf85ca2640 10 monclient(hunting): authenticate will time out at 1874.178087s 2024-09-16T03:57:07.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.206+0000 7fdf7ffff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdf8009fb00 0x7fdf80135ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:57:07.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.206+0000 7fdf7effd640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fdf8009c5d0 0x7fdf801359b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:57:07.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.206+0000 7fdf7ffff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdf8009fb00 0x7fdf80135ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:58228/0 (socket says 172.21.15.42:58228) 2024-09-16T03:57:07.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.206+0000 7fdf7effd640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fdf8009c5d0 0x7fdf801359b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:36018/0 (socket says 172.21.15.42:36018) 2024-09-16T03:57:07.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.206+0000 7fdf7ffff640 1 -- 172.21.15.42:0/377838358 learned_addr learned my addr 172.21.15.42:0/377838358 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:57:07.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.206+0000 7fdf7ffff640 10 monclient(hunting): get_auth_request con 0x7fdf8009fb00 auth_method 0 2024-09-16T03:57:07.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.206+0000 7fdf7ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:57:07.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.206+0000 7fdf7ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:57:07.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.206+0000 7fdf7ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:57:07.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.206+0000 7fdf7effd640 10 monclient(hunting): get_auth_request con 0x7fdf8009c5d0 auth_method 0 2024-09-16T03:57:07.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.206+0000 7fdf7effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:57:07.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.206+0000 7fdf7effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:57:07.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.206+0000 7fdf7effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:57:07.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.206+0000 7fdf7ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:57:07.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.206+0000 7fdf7ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:57:07.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.206+0000 7fdf7ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:57:07.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.206+0000 7fdf7effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:57:07.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.206+0000 7fdf7effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:57:07.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.206+0000 7fdf7f7fe640 1 --2- 172.21.15.42:0/377838358 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdf8009c160 0x7fdf801ceef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:57:07.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.206+0000 7fdf7effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:57:07.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.206+0000 7fdf7f7fe640 10 monclient(hunting): get_auth_request con 0x7fdf8009c160 auth_method 0 2024-09-16T03:57:07.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.206+0000 7fdf7f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:57:07.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.206+0000 7fdf7f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:57:07.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.206+0000 7fdf7f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:57:07.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.206+0000 7fdf7ffff640 10 monclient(hunting): handle_auth_done global_id 824820 payload 824 2024-09-16T03:57:07.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.206+0000 7fdf7ffff640 1 -- 172.21.15.42:0/377838358 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fdf8009c5d0 msgr2=0x7fdf801359b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:57:07.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.206+0000 7fdf7ffff640 1 --2- 172.21.15.42:0/377838358 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fdf8009c5d0 0x7fdf801359b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:57:07.230 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.206+0000 7fdf7ffff640 1 -- 172.21.15.42:0/377838358 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdf8009c160 msgr2=0x7fdf801ceef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:57:07.230 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.206+0000 7fdf7ffff640 1 --2- 172.21.15.42:0/377838358 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdf8009c160 0x7fdf801ceef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:57:07.230 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.206+0000 7fdf7ffff640 10 monclient: _finish_hunting 0 2024-09-16T03:57:07.230 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.206+0000 7fdf7ffff640 1 monclient: found mon.noname-a 2024-09-16T03:57:07.230 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.206+0000 7fdf7ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:57:07.230 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.206+0000 7fdf7ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:57:07.230 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.206+0000 7fdf7ffff640 1 -- 172.21.15.42:0/377838358 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf80122080 con 0x7fdf8009fb00 2024-09-16T03:57:07.230 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.206+0000 7fdf7effd640 1 --2- 172.21.15.42:0/377838358 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fdf8009c5d0 0x7fdf801359b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:57:07.230 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.206+0000 7fdf7f7fe640 1 --2- 172.21.15.42:0/377838358 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdf8009c160 0x7fdf801ceef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T03:57:07.230 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.206+0000 7fdf7ffff640 1 --2- 172.21.15.42:0/377838358 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdf8009fb00 0x7fdf80135ef0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fdf7801a4a0 tx=0x7fdf78019610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:57:07.230 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.206+0000 7fdf77fff640 10 client.?.objecter ms_handle_connect 0x7fdf8009fb00 2024-09-16T03:57:07.230 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.206+0000 7fdf77fff640 10 client.?.objecter resend_mon_ops 2024-09-16T03:57:07.230 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.206+0000 7fdf77fff640 1 -- 172.21.15.42:0/377838358 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf780bc020 con 0x7fdf8009fb00 2024-09-16T03:57:07.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.206+0000 7fdf77fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:57:07.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.206+0000 7fdf77fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:57:07.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.206+0000 7fdf77fff640 10 monclient: dump: 2024-09-16T03:57:07.231 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:57:07.231 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:57:07.231 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:57:07.231 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:57:07.231 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:57:07.231 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:57:07.231 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:57:07.231 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:57:07.231 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:57:07.231 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:57:07.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.206+0000 7fdf77fff640 10 monclient: _finish_auth 0 2024-09-16T03:57:07.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.206+0000 7fdf77fff640 10 monclient: _check_auth_tickets 2024-09-16T03:57:07.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.206+0000 7fdf77fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:57:07.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.206+0000 7fdf77fff640 1 -- 172.21.15.42:0/377838358 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdf780b8690 con 0x7fdf8009fb00 2024-09-16T03:57:07.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.206+0000 7fdf85ca2640 5 monclient: authenticate success, global_id 824820 2024-09-16T03:57:07.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.206+0000 7fdf85ca2640 10 monclient: _renew_subs 2024-09-16T03:57:07.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.206+0000 7fdf85ca2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:57:07.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.206+0000 7fdf85ca2640 1 -- 172.21.15.42:0/377838358 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf800a3ee0 con 0x7fdf8009fb00 2024-09-16T03:57:07.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.206+0000 7fdf77fff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:57:07.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.206+0000 7fdf77fff640 1 -- 172.21.15.42:0/377838358 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf780b8a80 con 0x7fdf8009fb00 2024-09-16T03:57:07.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.206+0000 7fdf77fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:57:07.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.206+0000 7fdf77fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:57:07.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.206+0000 7fdf77fff640 10 monclient: dump: 2024-09-16T03:57:07.232 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:57:07.232 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:57:07.233 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:57:07.233 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:57:07.233 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:57:07.233 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:57:07.233 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:57:07.233 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:57:07.233 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:57:07.233 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:57:07.233 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.206+0000 7fdf7d7fa640 4 set_mon_vals no callback set 2024-09-16T03:57:07.233 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.206+0000 7fdf85ca2640 10 client.824820.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:57:07.233 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.206+0000 7fdf85ca2640 10 monclient: _renew_subs 2024-09-16T03:57:07.233 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.206+0000 7fdf85ca2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:57:07.233 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.206+0000 7fdf85ca2640 1 -- 172.21.15.42:0/377838358 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf80121da0 con 0x7fdf8009fb00 2024-09-16T03:57:07.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.206+0000 7fdf85ca2640 1 librados: init done 2024-09-16T03:57:07.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.206+0000 7fdf85ca2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:57:07.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.206+0000 7fdf85ca2640 1 --2- 172.21.15.42:0/377838358 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fdf80071230 0x7fdf8002c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:57:07.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.206+0000 7fdf85ca2640 10 monclient(hunting): start opening mon connection 2024-09-16T03:57:07.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.206+0000 7fdf85ca2640 1 -- 172.21.15.42:0/377838358 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fdf80073a80 con 0x7fdf80071230 2024-09-16T03:57:07.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.210+0000 7fdf7f7fe640 1 --2- 172.21.15.42:0/377838358 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fdf80071230 0x7fdf8002c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:57:07.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.210+0000 7fdf7f7fe640 10 monclient: get_auth_request con 0x7fdf80071230 auth_method 0 2024-09-16T03:57:07.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.210+0000 7fdf7f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:57:07.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.210+0000 7fdf7f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:57:07.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.210+0000 7fdf7f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:57:07.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.210+0000 7fdf77fff640 1 -- 172.21.15.42:0/377838358 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fdf780bf020 con 0x7fdf8009fb00 2024-09-16T03:57:07.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.210+0000 7fdf7f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:57:07.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.210+0000 7fdf7f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:57:07.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.210+0000 7fdf7f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:57:07.235 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.210+0000 7fdf7f7fe640 10 monclient(hunting): handle_auth_done global_id 854369 payload 824 2024-09-16T03:57:07.235 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.210+0000 7fdf77fff640 1 --2- 172.21.15.42:0/377838358 >> v2:172.21.15.154:6812/254845342 conn(0x7fdf60037810 0x7fdf60039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:57:07.235 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.210+0000 7fdf7f7fe640 1 -- 172.21.15.42:0/377838358 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fdf8009ba50 con 0x7fdf80071230 2024-09-16T03:57:07.235 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.210+0000 7fdf77fff640 1 -- 172.21.15.42:0/377838358 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(457..457 src has 330..457) ==== 5747+0+0 (secure 0 0 0) 0x7fdf780e39e0 con 0x7fdf8009fb00 2024-09-16T03:57:07.235 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.210+0000 7fdf77fff640 10 client.824820.objecter ms_dispatch 0x7fdf80000ba0 osd_map(457..457 src has 330..457) 2024-09-16T03:57:07.235 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.210+0000 7fdf77fff640 3 client.824820.objecter handle_osd_map got epochs [457,457] > 0 2024-09-16T03:57:07.235 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.210+0000 7fdf77fff640 3 client.824820.objecter handle_osd_map decoding full epoch 457 2024-09-16T03:57:07.235 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.210+0000 7fdf77fff640 20 client.824820.objecter dump_active .. 0 homeless 2024-09-16T03:57:07.235 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.210+0000 7fdf77fff640 10 client.824820.objecter ms_handle_connect 0x7fdf80071230 2024-09-16T03:57:07.235 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.210+0000 7fdf77fff640 10 client.824820.objecter resend_mon_ops 2024-09-16T03:57:07.235 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.210+0000 7fdf7f7fe640 1 --2- 172.21.15.42:0/377838358 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fdf80071230 0x7fdf8002c650 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fdf7000e500 tx=0x7fdf7000e9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:57:07.235 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.210+0000 7fdf7effd640 1 --2- 172.21.15.42:0/377838358 >> v2:172.21.15.154:6812/254845342 conn(0x7fdf60037810 0x7fdf60039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:57:07.235 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.210+0000 7fdf77fff640 1 -- 172.21.15.42:0/377838358 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf70007030 con 0x7fdf80071230 2024-09-16T03:57:07.235 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.210+0000 7fdf77fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:57:07.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.210+0000 7fdf77fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:57:07.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.210+0000 7fdf77fff640 10 monclient: dump: 2024-09-16T03:57:07.236 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:57:07.236 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:57:07.236 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:57:07.236 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:57:07.236 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:57:07.236 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:57:07.236 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:57:07.236 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:57:07.236 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:57:07.236 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:57:07.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.210+0000 7fdf7effd640 10 monclient: get_auth_request con 0x7fdf60037810 auth_method 0 2024-09-16T03:57:07.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.210+0000 7fdf77fff640 1 -- 172.21.15.42:0/377838358 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fdf70007030 con 0x7fdf80071230 2024-09-16T03:57:07.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.210+0000 7fdf77fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:57:07.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.210+0000 7fdf77fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:57:07.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.210+0000 7fdf77fff640 1 -- 172.21.15.42:0/377838358 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fdf80071230 msgr2=0x7fdf8002c650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:57:07.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.210+0000 7fdf77fff640 1 --2- 172.21.15.42:0/377838358 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fdf80071230 0x7fdf8002c650 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fdf7000e500 tx=0x7fdf7000e9c0 comp rx=0 tx=0).stop 2024-09-16T03:57:07.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.210+0000 7fdf77fff640 1 -- 172.21.15.42:0/377838358 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fdf80071230 msgr2=0x7fdf8002c650 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:57:07.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.210+0000 7fdf77fff640 1 --2- 172.21.15.42:0/377838358 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fdf80071230 0x7fdf8002c650 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:57:07.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.210+0000 7fdf7effd640 1 --2- 172.21.15.42:0/377838358 >> v2:172.21.15.154:6812/254845342 conn(0x7fdf60037810 0x7fdf60039cd0 secure :-1 s=READY pgs=3668 cs=0 l=1 rev1=1 crypto rx=0x7fdf801cffb0 tx=0x7fdf68008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:57:07.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.210+0000 7fdf77fff640 10 client.824820.objecter ms_handle_connect 0x7fdf60037810 2024-09-16T03:57:07.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.214+0000 7fdf85ca2640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:57:07.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.214+0000 7fdf85ca2640 1 --2- 172.21.15.42:0/377838358 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fdf80071610 0x7fdf8002cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:57:07.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.214+0000 7fdf85ca2640 10 monclient(hunting): start opening mon connection 2024-09-16T03:57:07.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.214+0000 7fdf85ca2640 1 -- 172.21.15.42:0/377838358 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fdf80136540 con 0x7fdf80071610 2024-09-16T03:57:07.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.214+0000 7fdf7f7fe640 1 --2- 172.21.15.42:0/377838358 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fdf80071610 0x7fdf8002cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:57:07.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.214+0000 7fdf7f7fe640 10 monclient: get_auth_request con 0x7fdf80071610 auth_method 0 2024-09-16T03:57:07.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.214+0000 7fdf7f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:57:07.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.214+0000 7fdf7f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:57:07.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.214+0000 7fdf7f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:57:07.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.214+0000 7fdf7f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:57:07.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.214+0000 7fdf7f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:57:07.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.214+0000 7fdf7f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:57:07.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.214+0000 7fdf7f7fe640 10 monclient(hunting): handle_auth_done global_id 854372 payload 824 2024-09-16T03:57:07.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.214+0000 7fdf7f7fe640 1 -- 172.21.15.42:0/377838358 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fdf80137e20 con 0x7fdf80071610 2024-09-16T03:57:07.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.214+0000 7fdf7f7fe640 1 --2- 172.21.15.42:0/377838358 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fdf80071610 0x7fdf8002cb90 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fdf70009010 tx=0x7fdf70003770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:57:07.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.214+0000 7fdf77fff640 10 client.824820.objecter ms_handle_connect 0x7fdf80071610 2024-09-16T03:57:07.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.214+0000 7fdf77fff640 10 client.824820.objecter resend_mon_ops 2024-09-16T03:57:07.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.214+0000 7fdf77fff640 1 -- 172.21.15.42:0/377838358 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf7001ed10 con 0x7fdf80071610 2024-09-16T03:57:07.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.214+0000 7fdf77fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:57:07.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.214+0000 7fdf77fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:57:07.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.214+0000 7fdf77fff640 10 monclient: dump: 2024-09-16T03:57:07.239 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:57:07.239 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:57:07.239 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:57:07.239 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:57:07.239 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:57:07.239 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:57:07.239 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:57:07.239 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:57:07.239 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:57:07.239 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:57:07.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.218+0000 7fdf77fff640 1 -- 172.21.15.42:0/377838358 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4546 (secure 0 0 0) 0x7fdf7001ed10 con 0x7fdf80071610 2024-09-16T03:57:07.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.218+0000 7fdf77fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:57:07.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.218+0000 7fdf77fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:57:07.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.218+0000 7fdf77fff640 1 -- 172.21.15.42:0/377838358 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fdf80071610 msgr2=0x7fdf8002cb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:57:07.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.218+0000 7fdf77fff640 1 --2- 172.21.15.42:0/377838358 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fdf80071610 0x7fdf8002cb90 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fdf70009010 tx=0x7fdf70003770 comp rx=0 tx=0).stop 2024-09-16T03:57:07.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.218+0000 7fdf77fff640 1 -- 172.21.15.42:0/377838358 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fdf80071610 msgr2=0x7fdf8002cb90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:57:07.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.218+0000 7fdf77fff640 1 --2- 172.21.15.42:0/377838358 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fdf80071610 0x7fdf8002cb90 secure :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fdf70009010 tx=0x7fdf70003770 comp rx=0 tx=0).stop 2024-09-16T03:57:07.240 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:57:07.240 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:57:07.241 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:57:07.241 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T03:57:07.241 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 140, 2024-09-16T03:57:07.241 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 160650, 2024-09-16T03:57:07.241 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:57:07.241 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:57:07.241 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T03:57:07.241 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T03:57:07.241 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:57:07.241 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 0, 2024-09-16T03:57:07.241 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:57:07.241 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:57:07.241 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:57:07.242 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:57:07.242 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:57:07.242 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:57:07.242 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:57:07.242 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:57:07.242 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:57:07.242 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:57:07.242 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:57:07.242 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:57:07.242 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:57:07.242 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:57:07.242 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:57:07.242 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:57:07.242 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:57:07.243 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:57:07.243 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:57:07.243 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:57:07.243 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:57:07.243 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:57:07.243 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:57:07.243 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:57:07.243 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:57:07.243 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:57:07.243 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:57:07.243 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:57:07.243 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:57:07.243 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:57:07.243 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:57:07.244 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:57:07.244 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:57:07.244 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:57:07.244 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:57:07.244 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:57:07.244 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:57:07.244 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:57:07.244 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:57:07.244 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:57:07.244 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:57:07.244 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:57:07.244 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:57:07.244 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:57:07.244 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:57:07.245 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:57:07.245 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:57:07.245 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:57:07.245 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:57:07.245 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:57:07.245 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:57:07.245 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:57:07.245 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:57:07.245 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:57:07.245 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:57:07.245 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:57:07.245 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:57:07.245 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:57:07.245 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:57:07.246 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:57:07.246 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:57:07.246 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:57:07.246 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:57:07.246 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:57:07.246 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:57:07.246 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:57:07.246 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:57:07.246 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:57:07.246 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:57:07.246 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:57:07.246 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:57:07.246 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:57:07.246 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:57:07.247 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:57:07.247 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:57:07.247 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:57:07.247 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:57:07.247 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:57:07.247 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:57:07.247 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:57:07.247 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:57:07.247 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:57:07.247 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:57:07.247 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:57:07.247 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:57:07.247 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:57:07.247 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:57:07.248 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:57:07.248 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:57:07.248 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:57:07.248 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:57:07.248 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:57:07.248 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:57:07.248 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:57:07.248 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:57:07.248 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:57:07.248 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:57:07.248 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:57:07.248 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:57:07.248 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:57:07.248 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:57:07.249 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:57:07.249 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:57:07.249 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:57:07.249 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:57:07.249 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:57:07.249 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:57:07.249 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:57:07.249 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:57:07.249 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:57:07.249 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:57:07.249 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:57:07.249 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:57:07.249 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:57:07.249 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:57:07.250 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:57:07.250 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:57:07.250 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:57:07.250 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:57:07.250 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:57:07.250 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:57:07.250 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:57:07.250 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:57:07.250 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:57:07.250 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:57:07.250 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:57:07.250 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:57:07.250 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:57:07.251 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:57:07.251 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:57:07.251 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:57:07.251 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:57:07.251 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:57:07.251 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:57:07.251 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:57:07.251 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:57:07.251 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:57:07.251 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:57:07.251 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:57:07.251 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:57:07.251 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:57:07.251 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:57:07.252 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:57:07.252 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:57:07.252 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:57:07.252 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:57:07.252 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:57:07.252 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:57:07.252 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:57:07.252 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:57:07.252 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:57:07.254 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.218+0000 7fdf85ca2640 10 librados: watch_flush enter 2024-09-16T03:57:07.254 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.218+0000 7fdf85ca2640 10 librados: watch_flush exit 2024-09-16T03:57:07.254 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.218+0000 7fdf85ca2640 20 client.824820.objecter shutdown clearing up homeless session... 2024-09-16T03:57:07.254 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.218+0000 7fdf85ca2640 10 client.824820.objecter successfully canceled tick 2024-09-16T03:57:07.254 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.218+0000 7fdf85ca2640 1 -- 172.21.15.42:0/377838358 >> v2:172.21.15.154:6812/254845342 conn(0x7fdf60037810 msgr2=0x7fdf60039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:57:07.255 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.218+0000 7fdf85ca2640 1 --2- 172.21.15.42:0/377838358 >> v2:172.21.15.154:6812/254845342 conn(0x7fdf60037810 0x7fdf60039cd0 secure :-1 s=READY pgs=3668 cs=0 l=1 rev1=1 crypto rx=0x7fdf801cffb0 tx=0x7fdf68008040 comp rx=0 tx=0).stop 2024-09-16T03:57:07.255 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.218+0000 7fdf85ca2640 10 monclient: shutdown 2024-09-16T03:57:07.255 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.218+0000 7fdf85ca2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:57:07.255 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.218+0000 7fdf85ca2640 1 -- 172.21.15.42:0/377838358 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdf8009fb00 msgr2=0x7fdf80135ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:57:07.255 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.218+0000 7fdf85ca2640 1 --2- 172.21.15.42:0/377838358 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdf8009fb00 0x7fdf80135ef0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fdf7801a4a0 tx=0x7fdf78019610 comp rx=0 tx=0).stop 2024-09-16T03:57:07.255 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.218+0000 7fdf7ffff640 1 -- 172.21.15.42:0/377838358 reap_dead start 2024-09-16T03:57:07.255 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.218+0000 7fdf85ca2640 1 -- 172.21.15.42:0/377838358 shutdown_connections 2024-09-16T03:57:07.255 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.218+0000 7fdf85ca2640 1 --2- 172.21.15.42:0/377838358 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdf8009fb00 0x7fdf80135ef0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:57:07.255 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.218+0000 7fdf85ca2640 1 -- 172.21.15.42:0/377838358 >> 172.21.15.42:0/377838358 conn(0x7fdf800a2420 msgr2=0x7fdf80138b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:57:07.255 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.218+0000 7fdf85ca2640 1 -- 172.21.15.42:0/377838358 shutdown_connections 2024-09-16T03:57:07.255 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.218+0000 7fdf85ca2640 1 -- 172.21.15.42:0/377838358 wait complete. 2024-09-16T03:57:07.255 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.218+0000 7fdf85ca2640 1 librados: shutdown 2024-09-16T03:57:07.256 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:57:07.256 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T03:57:07.256 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T03:57:07.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.370+0000 7f38d17d0640 10 monclient: get_monmap_and_config 2024-09-16T03:57:07.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.370+0000 7f38d17d0640 10 monclient: build_initial_monmap 2024-09-16T03:57:07.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.370+0000 7f38d17d0640 10 monclient: monmap: 2024-09-16T03:57:07.376 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:57:07.376 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:57:07.376 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:57:07.374799+0000 2024-09-16T03:57:07.376 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:57:07.374799+0000 2024-09-16T03:57:07.377 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:57:07.377 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:57:07.377 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:57:07.377 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:57:07.377 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:57:07.377 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:57:07.377 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:57:07.377 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:57:07.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.370+0000 7f38d17d0640 1 Processor -- start 2024-09-16T03:57:07.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.370+0000 7f38d17d0640 1 -- start start 2024-09-16T03:57:07.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.370+0000 7f38d17d0640 10 monclient: init 2024-09-16T03:57:07.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.374+0000 7f38d17d0640 10 monclient: _reopen_session rank -1 2024-09-16T03:57:07.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.374+0000 7f38d17d0640 10 monclient: _add_conns ranks=[1,0,2,3,4] 2024-09-16T03:57:07.380 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.374+0000 7f38d17d0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f38cc0a1330 addr v1:172.21.15.154:6789/0 2024-09-16T03:57:07.380 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.374+0000 7f38d17d0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f38cc06dac0 addr v1:172.21.15.42:6789/0 2024-09-16T03:57:07.380 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.374+0000 7f38d17d0640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f38cc0a1e20 0x7f38cc13da70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:57:07.380 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.374+0000 7f38d17d0640 10 monclient(hunting): picked mon.noname-b con 0x7f38cc0a1e20 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:57:07.380 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.374+0000 7f38d17d0640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f38cc09c090 con 0x7f38cc06dac0 2024-09-16T03:57:07.380 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.374+0000 7f38d17d0640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f38cc09f610 con 0x7f38cc0a1330 2024-09-16T03:57:07.380 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.374+0000 7f38d17d0640 10 monclient(hunting): start opening mon connection 2024-09-16T03:57:07.380 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.374+0000 7f38d17d0640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f38cc09c890 con 0x7f38cc0a1e20 2024-09-16T03:57:07.380 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.374+0000 7f38d17d0640 10 monclient(hunting): _renew_subs 2024-09-16T03:57:07.380 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.374+0000 7f38d17d0640 10 monclient(hunting): authenticate will time out at 1874.344351s 2024-09-16T03:57:07.380 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.374+0000 7f38cb7fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f38cc0a1e20 0x7f38cc13da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:57:07.380 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.374+0000 7f38cb7fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f38cc0a1e20 0x7f38cc13da70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:36054/0 (socket says 172.21.15.42:36054) 2024-09-16T03:57:07.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.374+0000 7f38cb7fe640 1 -- 172.21.15.42:0/3007517903 learned_addr learned my addr 172.21.15.42:0/3007517903 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:57:07.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.374+0000 7f38cb7fe640 10 monclient(hunting): get_auth_request con 0x7f38cc0a1e20 auth_method 0 2024-09-16T03:57:07.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.374+0000 7f38cb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:57:07.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.374+0000 7f38cb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:57:07.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.374+0000 7f38cb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:57:07.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.374+0000 7f38cb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:57:07.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.374+0000 7f38cb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:57:07.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.374+0000 7f38cb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:57:07.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.374+0000 7f38cb7fe640 10 monclient(hunting): handle_auth_done global_id 854375 payload 274 2024-09-16T03:57:07.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.374+0000 7f38cb7fe640 1 -- 172.21.15.42:0/3007517903 >> v1:172.21.15.154:6789/0 conn(0x7f38cc0a1330 legacy=0x7f38cc0a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:57:07.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.374+0000 7f38cb7fe640 1 -- 172.21.15.42:0/3007517903 >> v1:172.21.15.42:6789/0 conn(0x7f38cc06dac0 legacy=0x7f38cc13a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:57:07.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.374+0000 7f38cb7fe640 10 monclient: _finish_hunting 0 2024-09-16T03:57:07.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.374+0000 7f38cb7fe640 1 monclient: found mon.noname-b 2024-09-16T03:57:07.382 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.374+0000 7f38cb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:57:07.382 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.374+0000 7f38cb7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T03:57:07.382 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.374+0000 7f38cb7fe640 1 -- 172.21.15.42:0/3007517903 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38cc09f8f0 con 0x7f38cc0a1e20 2024-09-16T03:57:07.389 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.374+0000 7f38cb7fe640 1 --2- 172.21.15.42:0/3007517903 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f38cc0a1e20 0x7f38cc13da70 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f38bc0098a0 tx=0x7f38bc01c460 comp rx=0 tx=0).ready entity=mon.2 client_cookie=379698ae21aa2a43 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:57:07.389 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.374+0000 7f38c9ffb640 1 -- 172.21.15.42:0/3007517903 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38bc0ab680 con 0x7f38cc0a1e20 2024-09-16T03:57:07.389 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.374+0000 7f38c9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:57:07.389 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.374+0000 7f38c9ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:57:07.389 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.374+0000 7f38c9ffb640 10 monclient: dump: 2024-09-16T03:57:07.389 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:57:07.389 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:57:07.389 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:57:07.389 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:57:07.389 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:57:07.390 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:57:07.390 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:57:07.390 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:57:07.390 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:57:07.390 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:57:07.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.374+0000 7f38c9ffb640 10 monclient: _finish_auth 0 2024-09-16T03:57:07.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.374+0000 7f38c9ffb640 10 monclient: _check_auth_tickets 2024-09-16T03:57:07.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.374+0000 7f38c9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:57:07.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.374+0000 7f38d17d0640 5 monclient: authenticate success, global_id 854375 2024-09-16T03:57:07.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.374+0000 7f38d17d0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:57:07.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.374+0000 7f38c9ffb640 1 -- 172.21.15.42:0/3007517903 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f38bc0b0070 con 0x7f38cc0a1e20 2024-09-16T03:57:07.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.374+0000 7f38c9ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T03:57:07.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.374+0000 7f38d17d0640 10 monclient: get_monmap_and_config success 2024-09-16T03:57:07.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.374+0000 7f38d17d0640 4 set_mon_vals no callback set 2024-09-16T03:57:07.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.374+0000 7f38c9ffb640 1 -- 172.21.15.42:0/3007517903 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38bc0aba40 con 0x7f38cc0a1e20 2024-09-16T03:57:07.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.374+0000 7f38c9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:57:07.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.374+0000 7f38c9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:57:07.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.374+0000 7f38c9ffb640 10 monclient: dump: 2024-09-16T03:57:07.391 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:57:07.391 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:57:07.391 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:57:07.391 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:57:07.391 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:57:07.391 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:57:07.392 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:57:07.392 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:57:07.392 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:57:07.392 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:57:07.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.374+0000 7f38d17d0640 10 monclient: shutdown 2024-09-16T03:57:07.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.374+0000 7f38d17d0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:57:07.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.374+0000 7f38d17d0640 1 -- 172.21.15.42:0/3007517903 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f38cc0a1e20 msgr2=0x7f38cc13da70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:57:07.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.374+0000 7f38d17d0640 1 --2- 172.21.15.42:0/3007517903 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f38cc0a1e20 0x7f38cc13da70 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f38bc0098a0 tx=0x7f38bc01c460 comp rx=0 tx=0).stop 2024-09-16T03:57:07.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.374+0000 7f38d17d0640 1 -- 172.21.15.42:0/3007517903 shutdown_connections 2024-09-16T03:57:07.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.374+0000 7f38d17d0640 1 --2- 172.21.15.42:0/3007517903 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f38cc0a1e20 0x7f38cc13da70 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:57:07.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.374+0000 7f38d17d0640 1 -- 172.21.15.42:0/3007517903 >> 172.21.15.42:0/3007517903 conn(0x7f38cc0a2420 msgr2=0x7f38cc0a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:57:07.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.374+0000 7f38d17d0640 1 -- 172.21.15.42:0/3007517903 shutdown_connections 2024-09-16T03:57:07.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.374+0000 7f38d17d0640 1 -- 172.21.15.42:0/3007517903 wait complete. 2024-09-16T03:57:07.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.378+0000 7f38d17d0640 10 monclient: build_initial_monmap 2024-09-16T03:57:07.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.378+0000 7f38d17d0640 10 monclient: monmap: 2024-09-16T03:57:07.393 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:57:07.393 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:57:07.393 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:57:07.393 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:57:07.393 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:57:07.393 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:57:07.393 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:57:07.393 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:57:07.393 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:57:07.393 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:57:07.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.378+0000 7f38d17d0640 1 librados: starting msgr at 2024-09-16T03:57:07.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.378+0000 7f38d17d0640 1 librados: starting objecter 2024-09-16T03:57:07.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.378+0000 7f38d17d0640 1 Processor -- start 2024-09-16T03:57:07.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.378+0000 7f38d17d0640 1 -- start start 2024-09-16T03:57:07.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.378+0000 7f38d17d0640 1 librados: setting wanted keys 2024-09-16T03:57:07.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.378+0000 7f38d17d0640 1 librados: calling monclient init 2024-09-16T03:57:07.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.378+0000 7f38d17d0640 10 monclient: init 2024-09-16T03:57:07.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.378+0000 7f38d17d0640 10 monclient: _reopen_session rank -1 2024-09-16T03:57:07.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.378+0000 7f38d17d0640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:57:07.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.378+0000 7f38d17d0640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f38cc0a1330 0x7f38cc11bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:57:07.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.378+0000 7f38d17d0640 10 monclient(hunting): picked mon.noname-b con 0x7f38cc0a1330 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:57:07.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.378+0000 7f38d17d0640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f38cc0a1e20 0x7f38cc11c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:57:07.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.378+0000 7f38d17d0640 10 monclient(hunting): picked mon.noname-a con 0x7f38cc0a1e20 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:57:07.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.378+0000 7f38d17d0640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f38cc11c9f0 0x7f38cc134930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:57:07.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.378+0000 7f38d17d0640 10 monclient(hunting): picked mon.noname-c con 0x7f38cc11c9f0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:57:07.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.378+0000 7f38d17d0640 10 monclient(hunting): start opening mon connection 2024-09-16T03:57:07.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.378+0000 7f38d17d0640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f38cc140950 con 0x7f38cc0a1e20 2024-09-16T03:57:07.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.378+0000 7f38d17d0640 10 monclient(hunting): start opening mon connection 2024-09-16T03:57:07.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.378+0000 7f38d17d0640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f38cc1407d0 con 0x7f38cc0a1330 2024-09-16T03:57:07.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.378+0000 7f38d17d0640 10 monclient(hunting): start opening mon connection 2024-09-16T03:57:07.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.378+0000 7f38d17d0640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f38cc140ad0 con 0x7f38cc11c9f0 2024-09-16T03:57:07.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.378+0000 7f38d17d0640 10 monclient(hunting): _renew_subs 2024-09-16T03:57:07.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.378+0000 7f38d17d0640 10 monclient(hunting): authenticate will time out at 1874.349428s 2024-09-16T03:57:07.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.378+0000 7f38cb7fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f38cc11c9f0 0x7f38cc134930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:57:07.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.378+0000 7f38ca7fc640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f38cc0a1e20 0x7f38cc11c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:57:07.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.378+0000 7f38ca7fc640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f38cc0a1e20 0x7f38cc11c4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:58238/0 (socket says 172.21.15.42:58238) 2024-09-16T03:57:07.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.378+0000 7f38ca7fc640 1 -- 172.21.15.42:0/3896412451 learned_addr learned my addr 172.21.15.42:0/3896412451 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:57:07.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.378+0000 7f38ca7fc640 10 monclient(hunting): get_auth_request con 0x7f38cc0a1e20 auth_method 0 2024-09-16T03:57:07.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.378+0000 7f38ca7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:57:07.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.378+0000 7f38ca7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:57:07.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.378+0000 7f38ca7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:57:07.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.378+0000 7f38cb7fe640 10 monclient(hunting): get_auth_request con 0x7f38cc11c9f0 auth_method 0 2024-09-16T03:57:07.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.378+0000 7f38cb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:57:07.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.378+0000 7f38cb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:57:07.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.378+0000 7f38cb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:57:07.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.378+0000 7f38ca7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:57:07.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.378+0000 7f38ca7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:57:07.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.378+0000 7f38ca7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:57:07.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.378+0000 7f38cb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:57:07.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.378+0000 7f38cb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:57:07.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.378+0000 7f38cb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:57:07.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.378+0000 7f38caffd640 1 --2- 172.21.15.42:0/3896412451 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f38cc0a1330 0x7f38cc11bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:57:07.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.378+0000 7f38caffd640 10 monclient(hunting): get_auth_request con 0x7f38cc0a1330 auth_method 0 2024-09-16T03:57:07.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.378+0000 7f38caffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:57:07.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.378+0000 7f38caffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:57:07.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.378+0000 7f38caffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:57:07.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.378+0000 7f38cb7fe640 10 monclient(hunting): handle_auth_done global_id 854378 payload 824 2024-09-16T03:57:07.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.378+0000 7f38cb7fe640 1 -- 172.21.15.42:0/3896412451 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f38cc0a1330 msgr2=0x7f38cc11bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:57:07.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.378+0000 7f38cb7fe640 1 --2- 172.21.15.42:0/3896412451 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f38cc0a1330 0x7f38cc11bf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:57:07.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.378+0000 7f38cb7fe640 1 -- 172.21.15.42:0/3896412451 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f38cc0a1e20 msgr2=0x7f38cc11c4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:57:07.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.378+0000 7f38cb7fe640 1 --2- 172.21.15.42:0/3896412451 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f38cc0a1e20 0x7f38cc11c4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:57:07.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.378+0000 7f38cb7fe640 10 monclient: _finish_hunting 0 2024-09-16T03:57:07.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.378+0000 7f38cb7fe640 1 monclient: found mon.noname-c 2024-09-16T03:57:07.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.378+0000 7f38cb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:57:07.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.378+0000 7f38cb7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:57:07.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.378+0000 7f38cb7fe640 1 -- 172.21.15.42:0/3896412451 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38cc11d350 con 0x7f38cc11c9f0 2024-09-16T03:57:07.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.378+0000 7f38ca7fc640 1 --2- 172.21.15.42:0/3896412451 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f38cc0a1e20 0x7f38cc11c4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:57:07.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.378+0000 7f38cb7fe640 1 --2- 172.21.15.42:0/3896412451 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f38cc11c9f0 0x7f38cc134930 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f38bc0ac000 tx=0x7f38bc019200 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:57:07.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.378+0000 7f38c37fe640 10 client.?.objecter ms_handle_connect 0x7f38cc11c9f0 2024-09-16T03:57:07.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.378+0000 7f38c37fe640 10 client.?.objecter resend_mon_ops 2024-09-16T03:57:07.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.378+0000 7f38c37fe640 1 -- 172.21.15.42:0/3896412451 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38bc0b5810 con 0x7f38cc11c9f0 2024-09-16T03:57:07.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.378+0000 7f38c37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:57:07.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.378+0000 7f38c37fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:57:07.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.378+0000 7f38c37fe640 10 monclient: dump: 2024-09-16T03:57:07.400 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:57:07.401 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:57:07.401 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:57:07.401 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:57:07.401 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:57:07.401 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:57:07.401 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:57:07.401 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:57:07.401 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:57:07.401 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:57:07.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.378+0000 7f38c37fe640 10 monclient: _finish_auth 0 2024-09-16T03:57:07.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.378+0000 7f38c37fe640 10 monclient: _check_auth_tickets 2024-09-16T03:57:07.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.378+0000 7f38c37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:57:07.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.378+0000 7f38c37fe640 1 -- 172.21.15.42:0/3896412451 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f38bc0a7070 con 0x7f38cc11c9f0 2024-09-16T03:57:07.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.378+0000 7f38c37fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:57:07.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.378+0000 7f38c37fe640 1 -- 172.21.15.42:0/3896412451 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38bc0b5bb0 con 0x7f38cc11c9f0 2024-09-16T03:57:07.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.378+0000 7f38c37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:57:07.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.378+0000 7f38c37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:57:07.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.378+0000 7f38c37fe640 10 monclient: dump: 2024-09-16T03:57:07.402 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:57:07.402 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:57:07.402 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:57:07.402 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:57:07.402 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:57:07.402 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:57:07.402 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:57:07.403 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:57:07.403 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:57:07.403 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:57:07.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.378+0000 7f38d17d0640 5 monclient: authenticate success, global_id 854378 2024-09-16T03:57:07.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.378+0000 7f38c97fa640 4 set_mon_vals no callback set 2024-09-16T03:57:07.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.378+0000 7f38d17d0640 10 monclient: _renew_subs 2024-09-16T03:57:07.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.378+0000 7f38d17d0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:57:07.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.378+0000 7f38d17d0640 1 -- 172.21.15.42:0/3896412451 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38cc11e800 con 0x7f38cc11c9f0 2024-09-16T03:57:07.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.378+0000 7f38d17d0640 10 client.854378.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:57:07.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.378+0000 7f38d17d0640 10 monclient: _renew_subs 2024-09-16T03:57:07.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.378+0000 7f38d17d0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:57:07.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.378+0000 7f38d17d0640 1 -- 172.21.15.42:0/3896412451 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f38cc06e660 con 0x7f38cc11c9f0 2024-09-16T03:57:07.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.378+0000 7f38d17d0640 1 librados: init done 2024-09-16T03:57:07.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.378+0000 7f38d17d0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:57:07.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.378+0000 7f38c37fe640 1 -- 172.21.15.42:0/3896412451 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f38bc0bd020 con 0x7f38cc11c9f0 2024-09-16T03:57:07.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.378+0000 7f38d17d0640 1 --2- 172.21.15.42:0/3896412451 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f38cc06dac0 0x7f38cc11e240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:57:07.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.378+0000 7f38d17d0640 10 monclient(hunting): start opening mon connection 2024-09-16T03:57:07.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.378+0000 7f38d17d0640 1 -- 172.21.15.42:0/3896412451 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f38cc0a1710 con 0x7f38cc06dac0 2024-09-16T03:57:07.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.378+0000 7f38c37fe640 1 --2- 172.21.15.42:0/3896412451 >> v2:172.21.15.154:6812/254845342 conn(0x7f38ac037810 0x7f38ac039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:57:07.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.378+0000 7f38c37fe640 1 -- 172.21.15.42:0/3896412451 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(457..457 src has 330..457) ==== 5747+0+0 (secure 0 0 0) 0x7f38bc0e3940 con 0x7f38cc11c9f0 2024-09-16T03:57:07.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.378+0000 7f38c37fe640 10 client.854378.objecter ms_dispatch 0x7f38cc000ba0 osd_map(457..457 src has 330..457) 2024-09-16T03:57:07.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.378+0000 7f38c37fe640 3 client.854378.objecter handle_osd_map got epochs [457,457] > 0 2024-09-16T03:57:07.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.378+0000 7f38c37fe640 3 client.854378.objecter handle_osd_map decoding full epoch 457 2024-09-16T03:57:07.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.378+0000 7f38c37fe640 20 client.854378.objecter dump_active .. 0 homeless 2024-09-16T03:57:07.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.378+0000 7f38caffd640 1 --2- 172.21.15.42:0/3896412451 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f38cc06dac0 0x7f38cc11e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:57:07.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.378+0000 7f38caffd640 10 monclient: get_auth_request con 0x7f38cc06dac0 auth_method 0 2024-09-16T03:57:07.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.378+0000 7f38caffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:57:07.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.378+0000 7f38caffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:57:07.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.378+0000 7f38caffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:57:07.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.378+0000 7f38ca7fc640 1 --2- 172.21.15.42:0/3896412451 >> v2:172.21.15.154:6812/254845342 conn(0x7f38ac037810 0x7f38ac039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:57:07.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.378+0000 7f38ca7fc640 10 monclient: get_auth_request con 0x7f38ac037810 auth_method 0 2024-09-16T03:57:07.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.378+0000 7f38caffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:57:07.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.378+0000 7f38caffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:57:07.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.378+0000 7f38caffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:57:07.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.382+0000 7f38caffd640 10 monclient(hunting): handle_auth_done global_id 884119 payload 824 2024-09-16T03:57:07.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.382+0000 7f38caffd640 1 -- 172.21.15.42:0/3896412451 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f38cc11d0d0 con 0x7f38cc06dac0 2024-09-16T03:57:07.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.382+0000 7f38caffd640 1 -- 172.21.15.42:0/3896412451 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f38cc06dac0 msgr2=0x7f38cc11e240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-16T03:57:07.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.382+0000 7f38caffd640 1 -- 172.21.15.42:0/3896412451 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f38cc06dac0 msgr2=0x7f38cc11e240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T03:57:07.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.382+0000 7f38caffd640 1 --2- 172.21.15.42:0/3896412451 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f38cc06dac0 0x7f38cc11e240 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rev1=1 crypto rx=0x7f38c400e7c0 tx=0x7f38c400ec80 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:57:07.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.382+0000 7f38caffd640 1 --2- 172.21.15.42:0/3896412451 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f38cc06dac0 0x7f38cc11e240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:57:07.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.382+0000 7f38c37fe640 10 client.854378.objecter ms_handle_connect 0x7f38ac037810 2024-09-16T03:57:07.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.382+0000 7f38ca7fc640 1 --2- 172.21.15.42:0/3896412451 >> v2:172.21.15.154:6812/254845342 conn(0x7f38ac037810 0x7f38ac039cd0 secure :-1 s=READY pgs=3669 cs=0 l=1 rev1=1 crypto rx=0x7f38cc13c900 tx=0x7f38b4009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:57:07.590 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.586+0000 7f38caffd640 1 --2- 172.21.15.42:0/3896412451 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f38cc06dac0 0x7f38cc11e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:57:07.590 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.586+0000 7f38caffd640 10 monclient: get_auth_request con 0x7f38cc06dac0 auth_method 0 2024-09-16T03:57:07.590 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.586+0000 7f38caffd640 10 monclient: get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:57:07.590 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.586+0000 7f38caffd640 10 monclient: _init_auth method 2 2024-09-16T03:57:07.590 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.586+0000 7f38caffd640 10 monclient: _init_auth already have auth, reseting 2024-09-16T03:57:07.590 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.586+0000 7f38caffd640 10 monclient: handle_auth_reply_more payload 9 2024-09-16T03:57:07.590 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.586+0000 7f38caffd640 10 monclient: handle_auth_reply_more payload_len 9 2024-09-16T03:57:07.590 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.586+0000 7f38caffd640 10 monclient: handle_auth_reply_more responding with 132 bytes 2024-09-16T03:57:07.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.586+0000 7f38caffd640 10 monclient: handle_auth_done global_id 884119 payload 293 2024-09-16T03:57:07.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.586+0000 7f38caffd640 1 --2- 172.21.15.42:0/3896412451 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f38cc06dac0 0x7f38cc11e240 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f38c400e7c0 tx=0x7f38c4009040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:57:07.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.586+0000 7f38c37fe640 10 client.854378.objecter ms_handle_connect 0x7f38cc06dac0 2024-09-16T03:57:07.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.586+0000 7f38c37fe640 10 client.854378.objecter resend_mon_ops 2024-09-16T03:57:07.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.586+0000 7f38c37fe640 1 -- 172.21.15.42:0/3896412451 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38c400c7f0 con 0x7f38cc06dac0 2024-09-16T03:57:07.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.586+0000 7f38c37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:57:07.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.586+0000 7f38c37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:57:07.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.586+0000 7f38c37fe640 10 monclient: dump: 2024-09-16T03:57:07.592 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:57:07.592 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:57:07.592 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:57:07.592 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:57:07.592 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:57:07.592 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:57:07.592 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:57:07.592 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:57:07.592 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:57:07.592 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:57:07.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.586+0000 7f38c37fe640 1 -- 172.21.15.42:0/3896412451 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f38c400cc50 con 0x7f38cc06dac0 2024-09-16T03:57:07.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.586+0000 7f38c37fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:57:07.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.586+0000 7f38c37fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:57:07.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.586+0000 7f38c37fe640 1 -- 172.21.15.42:0/3896412451 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f38cc06dac0 msgr2=0x7f38cc11e240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:57:07.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.586+0000 7f38c37fe640 1 --2- 172.21.15.42:0/3896412451 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f38cc06dac0 0x7f38cc11e240 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f38c400e7c0 tx=0x7f38c4009040 comp rx=0 tx=0).stop 2024-09-16T03:57:07.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.586+0000 7f38c37fe640 1 -- 172.21.15.42:0/3896412451 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f38cc06dac0 msgr2=0x7f38cc11e240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:57:07.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.586+0000 7f38c37fe640 1 --2- 172.21.15.42:0/3896412451 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f38cc06dac0 0x7f38cc11e240 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:57:07.599 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.594+0000 7f38d17d0640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:57:07.600 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.594+0000 7f38d17d0640 1 --2- 172.21.15.42:0/3896412451 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f38cc02c650 0x7f38cc02ca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:57:07.600 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.594+0000 7f38d17d0640 10 monclient(hunting): start opening mon connection 2024-09-16T03:57:07.600 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.594+0000 7f38d17d0640 1 -- 172.21.15.42:0/3896412451 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f38cc138a40 con 0x7f38cc02c650 2024-09-16T03:57:07.600 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.594+0000 7f38caffd640 1 --2- 172.21.15.42:0/3896412451 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f38cc02c650 0x7f38cc02ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:57:07.600 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.594+0000 7f38caffd640 10 monclient: get_auth_request con 0x7f38cc02c650 auth_method 0 2024-09-16T03:57:07.600 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.594+0000 7f38caffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:57:07.600 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.594+0000 7f38caffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:57:07.600 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.594+0000 7f38caffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:57:07.600 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.594+0000 7f38caffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:57:07.600 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.594+0000 7f38caffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:57:07.600 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.594+0000 7f38caffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:57:07.601 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.594+0000 7f38caffd640 10 monclient(hunting): handle_auth_done global_id 884122 payload 824 2024-09-16T03:57:07.601 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.594+0000 7f38caffd640 1 -- 172.21.15.42:0/3896412451 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f38cc069040 con 0x7f38cc02c650 2024-09-16T03:57:07.601 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.594+0000 7f38c37fe640 10 client.854378.objecter ms_handle_connect 0x7f38cc02c650 2024-09-16T03:57:07.601 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.594+0000 7f38c37fe640 10 client.854378.objecter resend_mon_ops 2024-09-16T03:57:07.601 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.594+0000 7f38caffd640 1 --2- 172.21.15.42:0/3896412451 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f38cc02c650 0x7f38cc02ca60 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f38c400e7c0 tx=0x7f38c400ec80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:57:07.606 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.602+0000 7f38c37fe640 1 -- 172.21.15.42:0/3896412451 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38c4007000 con 0x7f38cc02c650 2024-09-16T03:57:07.606 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.602+0000 7f38c37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:57:07.606 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.602+0000 7f38c37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:57:07.606 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.602+0000 7f38c37fe640 10 monclient: dump: 2024-09-16T03:57:07.607 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:57:07.607 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:57:07.607 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:57:07.607 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:57:07.607 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:57:07.607 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:57:07.607 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:57:07.607 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:57:07.607 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:57:07.607 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:57:07.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.602+0000 7f38c37fe640 1 -- 172.21.15.42:0/3896412451 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4544 (secure 0 0 0) 0x7f38c4007000 con 0x7f38cc02c650 2024-09-16T03:57:07.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.602+0000 7f38c37fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:57:07.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.602+0000 7f38c37fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:57:07.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.602+0000 7f38c37fe640 1 -- 172.21.15.42:0/3896412451 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f38cc02c650 msgr2=0x7f38cc02ca60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:57:07.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.602+0000 7f38c37fe640 1 --2- 172.21.15.42:0/3896412451 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f38cc02c650 0x7f38cc02ca60 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f38c400e7c0 tx=0x7f38c400ec80 comp rx=0 tx=0).stop 2024-09-16T03:57:07.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.602+0000 7f38c37fe640 1 -- 172.21.15.42:0/3896412451 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f38cc02c650 msgr2=0x7f38cc02ca60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:57:07.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.602+0000 7f38c37fe640 1 --2- 172.21.15.42:0/3896412451 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f38cc02c650 0x7f38cc02ca60 secure :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f38c400e7c0 tx=0x7f38c400ec80 comp rx=0 tx=0).stop 2024-09-16T03:57:07.609 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:57:07.609 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:57:07.609 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:57:07.609 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T03:57:07.609 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 140, 2024-09-16T03:57:07.609 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 9139, 2024-09-16T03:57:07.609 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:57:07.609 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:57:07.609 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T03:57:07.609 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T03:57:07.609 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:57:07.610 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 1, 2024-09-16T03:57:07.610 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:57:07.610 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:57:07.610 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:57:07.610 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:57:07.610 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:57:07.610 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:57:07.610 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:57:07.610 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:57:07.610 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:57:07.610 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:57:07.610 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:57:07.610 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:57:07.610 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:57:07.611 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:57:07.611 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:57:07.611 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:57:07.611 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:57:07.611 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:57:07.611 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:57:07.611 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:57:07.611 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:57:07.611 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:57:07.611 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:57:07.611 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:57:07.611 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:57:07.611 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:57:07.612 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:57:07.612 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:57:07.612 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:57:07.612 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:57:07.612 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:57:07.612 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:57:07.612 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:57:07.612 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:57:07.612 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:57:07.612 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:57:07.612 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:57:07.612 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:57:07.612 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:57:07.613 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:57:07.613 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:57:07.613 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:57:07.613 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:57:07.613 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:57:07.613 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:57:07.613 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:57:07.613 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:57:07.613 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:57:07.613 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:57:07.613 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:57:07.613 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:57:07.613 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:57:07.614 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:57:07.614 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:57:07.614 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:57:07.614 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:57:07.614 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:57:07.614 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:57:07.614 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:57:07.614 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:57:07.614 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:57:07.614 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:57:07.614 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:57:07.614 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:57:07.614 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:57:07.615 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:57:07.615 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:57:07.615 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:57:07.615 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:57:07.615 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:57:07.615 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:57:07.615 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:57:07.615 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:57:07.615 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:57:07.615 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:57:07.615 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:57:07.615 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:57:07.615 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:57:07.615 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:57:07.616 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:57:07.616 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:57:07.616 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:57:07.616 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:57:07.616 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:57:07.616 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:57:07.616 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:57:07.616 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:57:07.616 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:57:07.616 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:57:07.616 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:57:07.616 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:57:07.616 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:57:07.617 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:57:07.617 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:57:07.617 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:57:07.617 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:57:07.617 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:57:07.617 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:57:07.617 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:57:07.617 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:57:07.617 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:57:07.617 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:57:07.617 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:57:07.617 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:57:07.617 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:57:07.618 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:57:07.618 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:57:07.618 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:57:07.618 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:57:07.618 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:57:07.618 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:57:07.618 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:57:07.618 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:57:07.618 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:57:07.618 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:57:07.618 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:57:07.618 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:57:07.618 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:57:07.618 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:57:07.619 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:57:07.619 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:57:07.619 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:57:07.619 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:57:07.619 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:57:07.619 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:57:07.619 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:57:07.619 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:57:07.619 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:57:07.619 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:57:07.619 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:57:07.619 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:57:07.619 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:57:07.620 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:57:07.620 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:57:07.620 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:57:07.620 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:57:07.620 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:57:07.620 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:57:07.620 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:57:07.620 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:57:07.620 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:57:07.620 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:57:07.620 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:57:07.620 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:57:07.620 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:57:07.621 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:57:07.621 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:57:07.621 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:57:07.621 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:57:07.621 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:57:07.621 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:57:07.622 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.602+0000 7f38d17d0640 10 librados: watch_flush enter 2024-09-16T03:57:07.622 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.602+0000 7f38d17d0640 10 librados: watch_flush exit 2024-09-16T03:57:07.622 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.602+0000 7f38d17d0640 20 client.854378.objecter shutdown clearing up homeless session... 2024-09-16T03:57:07.622 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.602+0000 7f38d17d0640 10 client.854378.objecter successfully canceled tick 2024-09-16T03:57:07.622 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.602+0000 7f38d17d0640 1 -- 172.21.15.42:0/3896412451 >> v2:172.21.15.154:6812/254845342 conn(0x7f38ac037810 msgr2=0x7f38ac039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:57:07.622 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.602+0000 7f38d17d0640 1 --2- 172.21.15.42:0/3896412451 >> v2:172.21.15.154:6812/254845342 conn(0x7f38ac037810 0x7f38ac039cd0 secure :-1 s=READY pgs=3669 cs=0 l=1 rev1=1 crypto rx=0x7f38cc13c900 tx=0x7f38b4009040 comp rx=0 tx=0).stop 2024-09-16T03:57:07.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.602+0000 7f38d17d0640 10 monclient: shutdown 2024-09-16T03:57:07.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.602+0000 7f38d17d0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:57:07.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.602+0000 7f38d17d0640 1 -- 172.21.15.42:0/3896412451 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f38cc11c9f0 msgr2=0x7f38cc134930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:57:07.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.602+0000 7f38d17d0640 1 --2- 172.21.15.42:0/3896412451 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f38cc11c9f0 0x7f38cc134930 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f38bc0ac000 tx=0x7f38bc019200 comp rx=0 tx=0).stop 2024-09-16T03:57:07.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.602+0000 7f38cb7fe640 1 -- 172.21.15.42:0/3896412451 reap_dead start 2024-09-16T03:57:07.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.602+0000 7f38d17d0640 1 -- 172.21.15.42:0/3896412451 shutdown_connections 2024-09-16T03:57:07.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.602+0000 7f38d17d0640 1 --2- 172.21.15.42:0/3896412451 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f38cc11c9f0 0x7f38cc134930 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:57:07.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.602+0000 7f38d17d0640 1 -- 172.21.15.42:0/3896412451 >> 172.21.15.42:0/3896412451 conn(0x7f38cc0a2420 msgr2=0x7f38cc0a4ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:57:07.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.602+0000 7f38d17d0640 1 -- 172.21.15.42:0/3896412451 shutdown_connections 2024-09-16T03:57:07.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.602+0000 7f38d17d0640 1 -- 172.21.15.42:0/3896412451 wait complete. 2024-09-16T03:57:07.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.602+0000 7f38d17d0640 1 librados: shutdown 2024-09-16T03:57:07.635 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:57:07.635 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T03:57:07.635 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T03:57:07.765 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.758+0000 7f64691b2640 10 monclient: get_monmap_and_config 2024-09-16T03:57:07.765 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.758+0000 7f64691b2640 10 monclient: build_initial_monmap 2024-09-16T03:57:07.765 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.758+0000 7f64691b2640 10 monclient: monmap: 2024-09-16T03:57:07.765 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:57:07.765 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:57:07.766 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:57:07.763883+0000 2024-09-16T03:57:07.766 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:57:07.763883+0000 2024-09-16T03:57:07.766 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:57:07.766 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:57:07.766 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:57:07.766 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:57:07.766 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:57:07.766 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:57:07.766 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:57:07.766 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:57:07.777 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.762+0000 7f64691b2640 1 Processor -- start 2024-09-16T03:57:07.777 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.762+0000 7f64691b2640 1 -- start start 2024-09-16T03:57:07.777 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.762+0000 7f64691b2640 10 monclient: init 2024-09-16T03:57:07.777 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.762+0000 7f64691b2640 10 monclient: _reopen_session rank -1 2024-09-16T03:57:07.777 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.762+0000 7f64691b2640 10 monclient: _add_conns ranks=[1,0,4,3,2] 2024-09-16T03:57:07.777 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.762+0000 7f64691b2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f64640a1330 addr v1:172.21.15.154:6789/0 2024-09-16T03:57:07.777 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.762+0000 7f64691b2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f646406dac0 addr v1:172.21.15.42:6789/0 2024-09-16T03:57:07.777 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.762+0000 7f64691b2640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f64640a1e20 0x7f646413da70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:57:07.777 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.762+0000 7f64691b2640 10 monclient(hunting): picked mon.noname-c con 0x7f64640a1e20 addr v2:172.21.15.154:3300/0 2024-09-16T03:57:07.777 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.762+0000 7f64691b2640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f646409c090 con 0x7f646406dac0 2024-09-16T03:57:07.778 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.762+0000 7f64691b2640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f646409f610 con 0x7f64640a1330 2024-09-16T03:57:07.778 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.762+0000 7f64691b2640 10 monclient(hunting): start opening mon connection 2024-09-16T03:57:07.778 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.762+0000 7f64691b2640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f646409f8f0 con 0x7f64640a1e20 2024-09-16T03:57:07.778 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.762+0000 7f64691b2640 10 monclient(hunting): _renew_subs 2024-09-16T03:57:07.778 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.762+0000 7f64691b2640 10 monclient(hunting): authenticate will time out at 1874.733350s 2024-09-16T03:57:07.778 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.762+0000 7f646259c640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f646406dac0 0x7f646413a340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:44288/0 (socket says 172.21.15.42:44288) 2024-09-16T03:57:07.778 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.762+0000 7f646259c640 1 -- 172.21.15.42:0/1794586780 learned_addr learned my addr 172.21.15.42:0/1794586780 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:57:07.778 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.762+0000 7f646359e640 1 --2- 172.21.15.42:0/1794586780 >> v2:172.21.15.154:3300/0 conn(0x7f64640a1e20 0x7f646413da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:57:07.778 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.762+0000 7f646359e640 10 monclient(hunting): get_auth_request con 0x7f64640a1e20 auth_method 0 2024-09-16T03:57:07.778 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.762+0000 7f646359e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:57:07.778 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.762+0000 7f646359e640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:57:07.778 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.762+0000 7f646359e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:57:07.778 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.762+0000 7f6461d9b640 1 -- 172.21.15.42:0/1794586780 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1824077833 0 0) 0x7f646409c090 con 0x7f646406dac0 2024-09-16T03:57:07.779 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.762+0000 7f6461d9b640 10 monclient(hunting): _negotiate 2024-09-16T03:57:07.779 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.762+0000 7f6461d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:57:07.779 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.762+0000 7f6461d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:57:07.779 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.762+0000 7f6461d9b640 10 monclient(hunting): my global_id is 824829 2024-09-16T03:57:07.779 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.762+0000 7f6461d9b640 1 -- 172.21.15.42:0/1794586780 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6444003530 con 0x7f646406dac0 2024-09-16T03:57:07.779 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.762+0000 7f646359e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:57:07.779 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.762+0000 7f646359e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:57:07.779 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.762+0000 7f646359e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:57:07.779 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.762+0000 7f6462d9d640 1 -- 172.21.15.42:0/1794586780 >> v1:172.21.15.154:6789/0 conn(0x7f64640a1330 legacy=0x7f64640a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T03:57:07.779 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.762+0000 7f6462d9d640 1 -- 172.21.15.42:0/1794586780 >> v1:172.21.15.154:6789/0 conn(0x7f64640a1330 legacy=0x7f64640a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:57:07.779 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.762+0000 7f6462d9d640 1 --1- 172.21.15.42:0/1794586780 >> v1:172.21.15.154:6789/0 conn(0x7f64640a1330 0x7f64640a1710 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=0).handle_connect_reply_1 read connect reply failed 2024-09-16T03:57:07.779 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.762+0000 7f6461d9b640 1 -- 172.21.15.42:0/1794586780 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2973127593 0 0) 0x7f6444003530 con 0x7f646406dac0 2024-09-16T03:57:07.779 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.762+0000 7f6461d9b640 1 -- 172.21.15.42:0/1794586780 >> v2:172.21.15.154:3300/0 conn(0x7f64640a1e20 msgr2=0x7f646413da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:57:07.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.762+0000 7f6461d9b640 1 --2- 172.21.15.42:0/1794586780 >> v2:172.21.15.154:3300/0 conn(0x7f64640a1e20 0x7f646413da70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:57:07.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.762+0000 7f6461d9b640 1 -- 172.21.15.42:0/1794586780 >> v1:172.21.15.154:6789/0 conn(0x7f64640a1330 legacy=0x7f64640a1710 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:57:07.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.762+0000 7f6461d9b640 10 monclient: _finish_hunting 0 2024-09-16T03:57:07.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.762+0000 7f6461d9b640 1 monclient: found mon.noname-a-legacy 2024-09-16T03:57:07.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.762+0000 7f6461d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:57:07.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.762+0000 7f6461d9b640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.42:6789/0 2024-09-16T03:57:07.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.762+0000 7f6461d9b640 1 -- 172.21.15.42:0/1794586780 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f646409fbd0 con 0x7f646406dac0 2024-09-16T03:57:07.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.762+0000 7f6461d9b640 10 monclient: _finish_auth 0 2024-09-16T03:57:07.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.762+0000 7f6461d9b640 10 monclient: _check_auth_tickets 2024-09-16T03:57:07.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.762+0000 7f6461d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:57:07.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.762+0000 7f6461d9b640 1 -- 172.21.15.42:0/1794586780 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2177413734 0 0) 0x7f644c001a00 con 0x7f646406dac0 2024-09-16T03:57:07.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.762+0000 7f6461d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:57:07.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.762+0000 7f6461d9b640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T03:57:07.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.762+0000 7f6461d9b640 10 monclient: dump: 2024-09-16T03:57:07.781 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:57:07.781 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:57:07.781 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:57:07.781 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:57:07.781 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:57:07.781 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:57:07.781 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:57:07.781 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:57:07.781 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:57:07.781 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:57:07.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.762+0000 7f6461d9b640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] but i'm connected to v1:172.21.15.42:6789/0, reconnecting 2024-09-16T03:57:07.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.762+0000 7f6461d9b640 10 monclient: _reopen_session rank -1 2024-09-16T03:57:07.782 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.762+0000 7f6461d9b640 1 -- 172.21.15.42:0/1794586780 >> v1:172.21.15.42:6789/0 conn(0x7f646406dac0 legacy=0x7f646413a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:57:07.782 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.762+0000 7f6461d9b640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:57:07.782 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.762+0000 7f6461d9b640 1 --2- 172.21.15.42:0/1794586780 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f64440039c0 0x7f6444003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:57:07.782 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.762+0000 7f6461d9b640 10 monclient(hunting): picked mon.c con 0x7f64440039c0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:57:07.782 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.762+0000 7f6461d9b640 1 --2- 172.21.15.42:0/1794586780 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6444004810 0x7f6444004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:57:07.782 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.762+0000 7f6461d9b640 10 monclient(hunting): picked mon.b con 0x7f6444004810 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:57:07.782 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.762+0000 7f6461d9b640 1 --2- 172.21.15.42:0/1794586780 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f64440051d0 0x7f644400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:57:07.782 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.762+0000 7f6461d9b640 10 monclient(hunting): picked mon.a con 0x7f64440051d0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:57:07.782 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.762+0000 7f6461d9b640 10 monclient(hunting): start opening mon connection 2024-09-16T03:57:07.782 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.762+0000 7f6461d9b640 1 -- 172.21.15.42:0/1794586780 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f646409f8f0 con 0x7f64440051d0 2024-09-16T03:57:07.782 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.762+0000 7f6461d9b640 10 monclient(hunting): start opening mon connection 2024-09-16T03:57:07.782 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.762+0000 7f6461d9b640 1 -- 172.21.15.42:0/1794586780 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f644400c100 con 0x7f6444004810 2024-09-16T03:57:07.782 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.762+0000 7f6461d9b640 10 monclient(hunting): start opening mon connection 2024-09-16T03:57:07.783 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.762+0000 7f6461d9b640 1 -- 172.21.15.42:0/1794586780 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f644400c300 con 0x7f64440039c0 2024-09-16T03:57:07.783 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.762+0000 7f6461d9b640 10 monclient(hunting): _renew_subs 2024-09-16T03:57:07.783 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.762+0000 7f6461d9b640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:57:07.783 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.762+0000 7f646359e640 1 --2- 172.21.15.42:0/1794586780 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f64440051d0 0x7f644400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:57:07.783 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.762+0000 7f6462d9d640 1 --2- 172.21.15.42:0/1794586780 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f64440039c0 0x7f6444003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:57:07.783 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.762+0000 7f646359e640 10 monclient(hunting): get_auth_request con 0x7f64440051d0 auth_method 0 2024-09-16T03:57:07.783 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.762+0000 7f646359e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:57:07.783 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.762+0000 7f646359e640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:57:07.783 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.762+0000 7f646359e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:57:07.783 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.762+0000 7f6462d9d640 10 monclient(hunting): get_auth_request con 0x7f64440039c0 auth_method 0 2024-09-16T03:57:07.783 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.762+0000 7f6462d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:57:07.783 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.762+0000 7f6462d9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:57:07.783 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.762+0000 7f6462d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:57:07.783 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.762+0000 7f646359e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:57:07.784 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.762+0000 7f646359e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:57:07.784 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.762+0000 7f646359e640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T03:57:07.784 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.762+0000 7f6462d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:57:07.784 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.762+0000 7f6462d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:57:07.784 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.762+0000 7f6462d9d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T03:57:07.784 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.762+0000 7f646359e640 1 -- 172.21.15.42:0/1794586780 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f64440051d0 msgr2=0x7f644400b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T03:57:07.784 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.762+0000 7f646359e640 1 -- 172.21.15.42:0/1794586780 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f64440051d0 msgr2=0x7f644400b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:57:07.784 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.762+0000 7f646359e640 1 --2- 172.21.15.42:0/1794586780 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f64440051d0 0x7f644400b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:57:07.784 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.762+0000 7f646259c640 1 --2- 172.21.15.42:0/1794586780 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6444004810 0x7f6444004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:57:07.784 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.762+0000 7f646359e640 1 --2- 172.21.15.42:0/1794586780 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f64440051d0 0x7f644400b9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:57:07.784 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.762+0000 7f646259c640 10 monclient(hunting): get_auth_request con 0x7f6444004810 auth_method 0 2024-09-16T03:57:07.784 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.762+0000 7f646259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:57:07.784 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.762+0000 7f646259c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:57:07.785 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.762+0000 7f646259c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:57:07.785 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.762+0000 7f6462d9d640 10 monclient(hunting): handle_auth_done global_id 824829 payload 293 2024-09-16T03:57:07.785 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.762+0000 7f6462d9d640 1 -- 172.21.15.42:0/1794586780 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6444004810 msgr2=0x7f6444004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:57:07.785 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.762+0000 7f6462d9d640 1 --2- 172.21.15.42:0/1794586780 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6444004810 0x7f6444004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:57:07.785 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.762+0000 7f6462d9d640 1 -- 172.21.15.42:0/1794586780 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f64440051d0 msgr2=0x7f644400b9d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:57:07.785 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.762+0000 7f6462d9d640 1 --2- 172.21.15.42:0/1794586780 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f64440051d0 0x7f644400b9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:57:07.785 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.762+0000 7f6462d9d640 10 monclient: _finish_hunting 0 2024-09-16T03:57:07.785 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.762+0000 7f6462d9d640 1 monclient: found mon.c 2024-09-16T03:57:07.785 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.762+0000 7f6462d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:57:07.785 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.762+0000 7f6462d9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:57:07.785 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.762+0000 7f646359e640 1 -- 172.21.15.42:0/1794586780 reap_dead start 2024-09-16T03:57:07.785 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.762+0000 7f6462d9d640 1 -- 172.21.15.42:0/1794586780 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6444003530 con 0x7f64440039c0 2024-09-16T03:57:07.785 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.762+0000 7f6462d9d640 10 monclient: _finish_auth 0 2024-09-16T03:57:07.785 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.762+0000 7f6462d9d640 10 monclient: _check_auth_tickets 2024-09-16T03:57:07.786 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.762+0000 7f6462d9d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:57:07.786 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.762+0000 7f64691b2640 5 monclient: authenticate success, global_id 824829 2024-09-16T03:57:07.786 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.762+0000 7f64691b2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:57:07.786 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.766+0000 7f6462d9d640 1 --2- 172.21.15.42:0/1794586780 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f64440039c0 0x7f6444003dc0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f64540036f0 tx=0x7f645401c300 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bebfb5f33d89457 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:57:07.786 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.766+0000 7f6461d9b640 1 -- 172.21.15.42:0/1794586780 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64540039e0 con 0x7f64440039c0 2024-09-16T03:57:07.786 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.766+0000 7f6461d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:57:07.786 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.766+0000 7f6461d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:57:07.786 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.766+0000 7f6461d9b640 10 monclient: dump: 2024-09-16T03:57:07.786 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:57:07.786 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:57:07.786 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:57:07.786 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:57:07.786 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:57:07.786 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:57:07.787 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:57:07.787 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:57:07.787 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:57:07.787 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:57:07.787 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.766+0000 7f6461d9b640 1 -- 172.21.15.42:0/1794586780 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6454007290 con 0x7f64440039c0 2024-09-16T03:57:07.787 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.766+0000 7f64691b2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:57:07.787 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.766+0000 7f6461d9b640 10 monclient: handle_config config(0 keys) 2024-09-16T03:57:07.787 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.766+0000 7f6461d9b640 1 -- 172.21.15.42:0/1794586780 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64540057d0 con 0x7f64440039c0 2024-09-16T03:57:07.787 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.766+0000 7f64691b2640 10 monclient: get_monmap_and_config success 2024-09-16T03:57:07.787 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.766+0000 7f64691b2640 4 set_mon_vals no callback set 2024-09-16T03:57:07.787 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.766+0000 7f6461d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:57:07.787 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.766+0000 7f6461d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:57:07.787 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.766+0000 7f6461d9b640 10 monclient: dump: 2024-09-16T03:57:07.787 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:57:07.788 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:57:07.788 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:57:07.788 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:57:07.788 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:57:07.788 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:57:07.788 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:57:07.788 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:57:07.788 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:57:07.789 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:57:07.789 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.766+0000 7f64691b2640 10 monclient: shutdown 2024-09-16T03:57:07.789 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.766+0000 7f64691b2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:57:07.789 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.766+0000 7f64691b2640 1 -- 172.21.15.42:0/1794586780 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f64440039c0 msgr2=0x7f6444003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:57:07.789 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.766+0000 7f64691b2640 1 --2- 172.21.15.42:0/1794586780 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f64440039c0 0x7f6444003dc0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f64540036f0 tx=0x7f645401c300 comp rx=0 tx=0).stop 2024-09-16T03:57:07.789 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.766+0000 7f64691b2640 1 -- 172.21.15.42:0/1794586780 shutdown_connections 2024-09-16T03:57:07.789 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.766+0000 7f64691b2640 1 --2- 172.21.15.42:0/1794586780 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f64440039c0 0x7f6444003dc0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:57:07.789 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.766+0000 7f64691b2640 1 -- 172.21.15.42:0/1794586780 >> 172.21.15.42:0/1794586780 conn(0x7f64640a2420 msgr2=0x7f64640a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:57:07.789 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.766+0000 7f64691b2640 1 -- 172.21.15.42:0/1794586780 shutdown_connections 2024-09-16T03:57:07.789 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.766+0000 7f64691b2640 1 -- 172.21.15.42:0/1794586780 wait complete. 2024-09-16T03:57:07.789 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.766+0000 7f64691b2640 10 monclient: build_initial_monmap 2024-09-16T03:57:07.789 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.766+0000 7f64691b2640 10 monclient: monmap: 2024-09-16T03:57:07.789 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:57:07.790 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:57:07.790 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:57:07.790 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:57:07.790 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:57:07.790 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:57:07.790 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:57:07.790 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:57:07.790 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:57:07.790 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:57:07.790 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.766+0000 7f64691b2640 1 librados: starting msgr at 2024-09-16T03:57:07.790 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.766+0000 7f64691b2640 1 librados: starting objecter 2024-09-16T03:57:07.790 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.766+0000 7f64691b2640 1 Processor -- start 2024-09-16T03:57:07.790 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.766+0000 7f64691b2640 1 -- start start 2024-09-16T03:57:07.791 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.766+0000 7f64691b2640 1 librados: setting wanted keys 2024-09-16T03:57:07.791 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.766+0000 7f64691b2640 1 librados: calling monclient init 2024-09-16T03:57:07.791 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.766+0000 7f64691b2640 10 monclient: init 2024-09-16T03:57:07.791 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.766+0000 7f64691b2640 10 monclient: _reopen_session rank -1 2024-09-16T03:57:07.791 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.766+0000 7f64691b2640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T03:57:07.791 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.766+0000 7f64691b2640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f64440039c0 0x7f6464121850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:57:07.791 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.766+0000 7f64691b2640 10 monclient(hunting): picked mon.noname-a con 0x7f64440039c0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:57:07.791 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.766+0000 7f64691b2640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6464121d90 0x7f64641221f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:57:07.791 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.766+0000 7f64691b2640 10 monclient(hunting): picked mon.noname-b con 0x7f6464121d90 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:57:07.791 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.766+0000 7f64691b2640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f6464122730 0x7f64641d0150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:57:07.791 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.766+0000 7f64691b2640 10 monclient(hunting): picked mon.noname-c con 0x7f6464122730 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:57:07.791 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.766+0000 7f64691b2640 10 monclient(hunting): start opening mon connection 2024-09-16T03:57:07.791 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.766+0000 7f64691b2640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f646409eef0 con 0x7f64440039c0 2024-09-16T03:57:07.791 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.766+0000 7f64691b2640 10 monclient(hunting): start opening mon connection 2024-09-16T03:57:07.792 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.766+0000 7f64691b2640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f646409ed70 con 0x7f6464121d90 2024-09-16T03:57:07.792 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.766+0000 7f64691b2640 10 monclient(hunting): start opening mon connection 2024-09-16T03:57:07.792 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.766+0000 7f64691b2640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f64640a1710 con 0x7f6464122730 2024-09-16T03:57:07.792 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.766+0000 7f64691b2640 10 monclient(hunting): _renew_subs 2024-09-16T03:57:07.792 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.766+0000 7f64691b2640 10 monclient(hunting): authenticate will time out at 1874.739390s 2024-09-16T03:57:07.792 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.766+0000 7f6462d9d640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f64440039c0 0x7f6464121850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:57:07.792 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.766+0000 7f646359e640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f6464122730 0x7f64641d0150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:57:07.792 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.766+0000 7f6462d9d640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f64440039c0 0x7f6464121850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:58256/0 (socket says 172.21.15.42:58256) 2024-09-16T03:57:07.792 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.766+0000 7f6462d9d640 1 -- 172.21.15.42:0/3811283929 learned_addr learned my addr 172.21.15.42:0/3811283929 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:57:07.792 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.766+0000 7f6462d9d640 10 monclient(hunting): get_auth_request con 0x7f64440039c0 auth_method 0 2024-09-16T03:57:07.792 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.766+0000 7f6462d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:57:07.792 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.766+0000 7f6462d9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:57:07.792 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.766+0000 7f6462d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:57:07.792 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.766+0000 7f646359e640 10 monclient(hunting): get_auth_request con 0x7f6464122730 auth_method 0 2024-09-16T03:57:07.793 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.766+0000 7f646359e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:57:07.793 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.766+0000 7f646359e640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:57:07.793 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.766+0000 7f646359e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:57:07.793 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.766+0000 7f646359e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:57:07.793 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.766+0000 7f646359e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:57:07.793 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.766+0000 7f646359e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:57:07.793 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.766+0000 7f6462d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:57:07.793 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.766+0000 7f6462d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:57:07.793 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.766+0000 7f6462d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:57:07.793 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.766+0000 7f646259c640 1 --2- 172.21.15.42:0/3811283929 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6464121d90 0x7f64641221f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:57:07.793 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.766+0000 7f646359e640 10 monclient(hunting): handle_auth_done global_id 854381 payload 824 2024-09-16T03:57:07.793 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.766+0000 7f646359e640 1 -- 172.21.15.42:0/3811283929 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6464121d90 msgr2=0x7f64641221f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:57:07.794 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.766+0000 7f646359e640 1 --2- 172.21.15.42:0/3811283929 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6464121d90 0x7f64641221f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:57:07.794 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.766+0000 7f646359e640 1 -- 172.21.15.42:0/3811283929 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f64440039c0 msgr2=0x7f6464121850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:57:07.794 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.766+0000 7f646359e640 1 --2- 172.21.15.42:0/3811283929 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f64440039c0 0x7f6464121850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:57:07.794 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.766+0000 7f646359e640 10 monclient: _finish_hunting 0 2024-09-16T03:57:07.794 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.766+0000 7f646359e640 1 monclient: found mon.noname-c 2024-09-16T03:57:07.794 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.766+0000 7f646359e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:57:07.794 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.766+0000 7f646359e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:57:07.794 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.766+0000 7f646359e640 1 -- 172.21.15.42:0/3811283929 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6464123db0 con 0x7f6464122730 2024-09-16T03:57:07.794 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.766+0000 7f646259c640 10 monclient: get_auth_request con 0x7f6464121d90 auth_method 0 2024-09-16T03:57:07.794 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.766+0000 7f646259c640 1 --2- 172.21.15.42:0/3811283929 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6464121d90 0x7f64641221f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T03:57:07.794 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.766+0000 7f6462d9d640 1 --2- 172.21.15.42:0/3811283929 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f64440039c0 0x7f6464121850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:57:07.794 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.770+0000 7f646359e640 1 --2- 172.21.15.42:0/3811283929 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f6464122730 0x7f64641d0150 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f64640a22c0 tx=0x7f645c011cf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:57:07.794 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.770+0000 7f645b7fe640 10 client.?.objecter ms_handle_connect 0x7f6464122730 2024-09-16T03:57:07.794 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.770+0000 7f645b7fe640 10 client.?.objecter resend_mon_ops 2024-09-16T03:57:07.795 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.770+0000 7f645b7fe640 1 -- 172.21.15.42:0/3811283929 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f645c019020 con 0x7f6464122730 2024-09-16T03:57:07.795 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.770+0000 7f645b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:57:07.795 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.770+0000 7f645b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:57:07.795 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.770+0000 7f645b7fe640 10 monclient: dump: 2024-09-16T03:57:07.795 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:57:07.795 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:57:07.795 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:57:07.795 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:57:07.795 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:57:07.795 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:57:07.795 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:57:07.795 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:57:07.795 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:57:07.795 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:57:07.796 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.770+0000 7f645b7fe640 10 monclient: _finish_auth 0 2024-09-16T03:57:07.796 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.770+0000 7f645b7fe640 10 monclient: _check_auth_tickets 2024-09-16T03:57:07.797 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.770+0000 7f645b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:57:07.797 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.770+0000 7f645b7fe640 1 -- 172.21.15.42:0/3811283929 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f645c004240 con 0x7f6464122730 2024-09-16T03:57:07.797 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.770+0000 7f64691b2640 5 monclient: authenticate success, global_id 854381 2024-09-16T03:57:07.797 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.770+0000 7f64691b2640 10 monclient: _renew_subs 2024-09-16T03:57:07.797 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.770+0000 7f64691b2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:57:07.797 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.770+0000 7f64691b2640 1 -- 172.21.15.42:0/3811283929 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6464125540 con 0x7f6464122730 2024-09-16T03:57:07.797 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.770+0000 7f645b7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:57:07.797 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.770+0000 7f645b7fe640 1 -- 172.21.15.42:0/3811283929 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f645c00f770 con 0x7f6464122730 2024-09-16T03:57:07.797 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.770+0000 7f645b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:57:07.797 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.770+0000 7f645b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:57:07.797 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.770+0000 7f645b7fe640 10 monclient: dump: 2024-09-16T03:57:07.797 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:57:07.797 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:57:07.798 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:57:07.798 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:57:07.798 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:57:07.798 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:57:07.798 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:57:07.798 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:57:07.798 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:57:07.798 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:57:07.798 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.770+0000 7f64691b2640 10 client.854381.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:57:07.798 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.770+0000 7f64691b2640 10 monclient: _renew_subs 2024-09-16T03:57:07.798 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.770+0000 7f64691b2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:57:07.798 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.770+0000 7f6460d99640 4 set_mon_vals no callback set 2024-09-16T03:57:07.798 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.770+0000 7f64691b2640 1 -- 172.21.15.42:0/3811283929 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f646406e380 con 0x7f6464122730 2024-09-16T03:57:07.798 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.770+0000 7f64691b2640 1 librados: init done 2024-09-16T03:57:07.799 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.770+0000 7f64691b2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:57:07.799 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.770+0000 7f64691b2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:57:07.799 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.770+0000 7f64691b2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:57:07.799 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.770+0000 7f645b7fe640 1 -- 172.21.15.42:0/3811283929 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f645c01c020 con 0x7f6464122730 2024-09-16T03:57:07.799 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.770+0000 7f64691b2640 1 -- 172.21.15.42:0/3811283929 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6464073a30 con 0x7f6464122730 2024-09-16T03:57:07.799 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.770+0000 7f645b7fe640 1 --2- 172.21.15.42:0/3811283929 >> v2:172.21.15.154:6812/254845342 conn(0x7f6448037810 0x7f6448039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:57:07.799 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.770+0000 7f645b7fe640 1 -- 172.21.15.42:0/3811283929 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(458..458 src has 330..458) ==== 5713+0+0 (secure 0 0 0) 0x7f645c048e40 con 0x7f6464122730 2024-09-16T03:57:07.799 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.770+0000 7f645b7fe640 10 client.854381.objecter ms_dispatch 0x7f6464000ba0 osd_map(458..458 src has 330..458) 2024-09-16T03:57:07.799 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.770+0000 7f645b7fe640 3 client.854381.objecter handle_osd_map got epochs [458,458] > 0 2024-09-16T03:57:07.799 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.770+0000 7f645b7fe640 3 client.854381.objecter handle_osd_map decoding full epoch 458 2024-09-16T03:57:07.799 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.770+0000 7f645b7fe640 20 client.854381.objecter dump_active .. 0 homeless 2024-09-16T03:57:07.799 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.770+0000 7f6462d9d640 1 --2- 172.21.15.42:0/3811283929 >> v2:172.21.15.154:6812/254845342 conn(0x7f6448037810 0x7f6448039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:57:07.799 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.770+0000 7f6462d9d640 10 monclient: get_auth_request con 0x7f6448037810 auth_method 0 2024-09-16T03:57:07.800 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.770+0000 7f6462d9d640 1 --2- 172.21.15.42:0/3811283929 >> v2:172.21.15.154:6812/254845342 conn(0x7f6448037810 0x7f6448039cd0 secure :-1 s=READY pgs=3670 cs=0 l=1 rev1=1 crypto rx=0x7f6454007bd0 tx=0x7f6454007040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:57:07.800 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.770+0000 7f645b7fe640 10 client.854381.objecter ms_handle_connect 0x7f6448037810 2024-09-16T03:57:07.800 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.774+0000 7f645b7fe640 1 -- 172.21.15.42:0/3811283929 <== mon.2 v2:172.21.15.42:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f645c023030 con 0x7f6464122730 2024-09-16T03:57:07.800 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.774+0000 7f645b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:57:07.800 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.774+0000 7f645b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:57:08.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.998+0000 7f64691b2640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T03:57:08.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.998+0000 7f64691b2640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T03:57:08.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.998+0000 7f64691b2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:57:08.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:07.998+0000 7f64691b2640 1 -- 172.21.15.42:0/3811283929 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f64640a1aa0 con 0x7f6464122730 2024-09-16T03:57:08.007 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.002+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.007 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.002+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=29,config=2,health=10,kv=10,logm=49} crc {auth=2171661892,config=692619927,health=3614246832,kv=1987072213,logm=1393579257}) 2024-09-16T03:57:08.007 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.002+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=27,config=2,health=10,kv=10,logm=51} crc {auth=183950580,config=692619927,health=2066015120,kv=1987072213,logm=1462649948}) 2024-09-16T03:57:08.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:08.002+0000 7f645b7fe640 1 -- 172.21.15.42:0/3811283929 <== mon.2 v2:172.21.15.42:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f645c01d3b0 con 0x7f6464122730 2024-09-16T03:57:08.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:08.002+0000 7f645b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T03:57:08.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:08.002+0000 7f645b7fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:57:08.008 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.002+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.008 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.002+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=29,config=2,health=10,kv=10,logm=49} crc {auth=2171661892,config=692619927,health=3614246832,kv=1987072213,logm=1393579257}) 2024-09-16T03:57:08.008 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.002+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=24,config=2,health=12,kv=9,logm=53} crc {auth=1726488078,config=692619927,health=744740284,kv=542242125,logm=983952493}) 2024-09-16T03:57:08.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:08.002+0000 7f64691b2640 10 librados: watch_flush enter 2024-09-16T03:57:08.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:08.002+0000 7f64691b2640 10 librados: watch_flush exit 2024-09-16T03:57:08.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:08.002+0000 7f64691b2640 20 client.854381.objecter shutdown clearing up homeless session... 2024-09-16T03:57:08.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:08.002+0000 7f64691b2640 10 client.854381.objecter successfully canceled tick 2024-09-16T03:57:08.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:08.002+0000 7f64691b2640 1 -- 172.21.15.42:0/3811283929 >> v2:172.21.15.154:6812/254845342 conn(0x7f6448037810 msgr2=0x7f6448039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:57:08.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:08.002+0000 7f64691b2640 1 --2- 172.21.15.42:0/3811283929 >> v2:172.21.15.154:6812/254845342 conn(0x7f6448037810 0x7f6448039cd0 secure :-1 s=READY pgs=3670 cs=0 l=1 rev1=1 crypto rx=0x7f6454007bd0 tx=0x7f6454007040 comp rx=0 tx=0).stop 2024-09-16T03:57:08.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:08.002+0000 7f64691b2640 10 monclient: shutdown 2024-09-16T03:57:08.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:08.002+0000 7f64691b2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:57:08.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:08.002+0000 7f64691b2640 1 -- 172.21.15.42:0/3811283929 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f6464122730 msgr2=0x7f64641d0150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:57:08.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:08.002+0000 7f64691b2640 1 --2- 172.21.15.42:0/3811283929 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f6464122730 0x7f64641d0150 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f64640a22c0 tx=0x7f645c011cf0 comp rx=0 tx=0).stop 2024-09-16T03:57:08.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:08.002+0000 7f64691b2640 1 -- 172.21.15.42:0/3811283929 shutdown_connections 2024-09-16T03:57:08.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:08.002+0000 7f64691b2640 1 --2- 172.21.15.42:0/3811283929 >> v2:172.21.15.154:6812/254845342 conn(0x7f6448037810 0x7f6448039cd0 unknown :-1 s=CLOSED pgs=3670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:57:08.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:08.002+0000 7f64691b2640 1 --2- 172.21.15.42:0/3811283929 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f6464122730 0x7f64641d0150 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:57:08.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:08.002+0000 7f64691b2640 1 --2- 172.21.15.42:0/3811283929 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6464121d90 0x7f64641221f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:57:08.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:08.002+0000 7f64691b2640 1 --2- 172.21.15.42:0/3811283929 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f64440039c0 0x7f6464121850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:57:08.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:08.006+0000 7f64691b2640 1 -- 172.21.15.42:0/3811283929 >> 172.21.15.42:0/3811283929 conn(0x7f64640a2420 msgr2=0x7f64640717d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:57:08.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:08.006+0000 7f64691b2640 1 -- 172.21.15.42:0/3811283929 shutdown_connections 2024-09-16T03:57:08.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:08.006+0000 7f64691b2640 1 -- 172.21.15.42:0/3811283929 wait complete. 2024-09-16T03:57:08.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:08.006+0000 7f64691b2640 1 librados: shutdown 2024-09-16T03:57:08.011 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.006+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.012 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.006+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=15998725}) 2024-09-16T03:57:08.012 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.006+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4212037535}) 2024-09-16T03:57:08.012 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.006+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.012 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.006+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=15998725}) 2024-09-16T03:57:08.012 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.006+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3810662833}) 2024-09-16T03:57:08.013 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.006+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.013 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.006+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2043790295}) 2024-09-16T03:57:08.013 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.006+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=454009497}) 2024-09-16T03:57:08.013 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.006+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.013 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.006+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2043790295}) 2024-09-16T03:57:08.013 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.006+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=860840732}) 2024-09-16T03:57:08.015 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.010+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.015 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.010+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3644959677}) 2024-09-16T03:57:08.015 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.010+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3373057350}) 2024-09-16T03:57:08.015 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.010+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.016 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.010+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3644959677}) 2024-09-16T03:57:08.016 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.010+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4170473936}) 2024-09-16T03:57:08.018 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.014+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.019 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.014+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3411193895}) 2024-09-16T03:57:08.019 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.014+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=79037694}) 2024-09-16T03:57:08.019 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.014+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.019 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.014+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3411193895}) 2024-09-16T03:57:08.019 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.014+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2987547008}) 2024-09-16T03:57:08.021 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.014+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.021 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.014+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1851097289}) 2024-09-16T03:57:08.021 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.014+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4144630880}) 2024-09-16T03:57:08.021 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.014+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.021 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.014+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1851097289}) 2024-09-16T03:57:08.021 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.014+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2548150572}) 2024-09-16T03:57:08.023 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.018+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.023 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.018+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2884419733}) 2024-09-16T03:57:08.023 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.018+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4074100671}) 2024-09-16T03:57:08.023 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.018+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.023 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.018+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2884419733}) 2024-09-16T03:57:08.023 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.018+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2493937008}) 2024-09-16T03:57:08.025 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.018+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.025 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.018+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4217434323}) 2024-09-16T03:57:08.025 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.018+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2161141367}) 2024-09-16T03:57:08.025 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.018+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.025 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.018+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4217434323}) 2024-09-16T03:57:08.025 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.022+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2243241461}) 2024-09-16T03:57:08.027 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.022+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.027 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.022+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3157186844}) 2024-09-16T03:57:08.027 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.022+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=563146636}) 2024-09-16T03:57:08.027 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.022+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.028 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.022+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3157186844}) 2024-09-16T03:57:08.028 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.022+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=600656868}) 2024-09-16T03:57:08.028 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T03:57:08.029 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.026+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.029 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.026+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1656941101}) 2024-09-16T03:57:08.029 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.026+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3013802695}) 2024-09-16T03:57:08.030 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.026+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.030 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.026+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1656941101}) 2024-09-16T03:57:08.030 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.026+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=776117780}) 2024-09-16T03:57:08.031 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.026+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.031 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.026+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3829991006}) 2024-09-16T03:57:08.031 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.026+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3399498089}) 2024-09-16T03:57:08.031 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.026+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.031 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.026+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3829991006}) 2024-09-16T03:57:08.031 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.026+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2917433643}) 2024-09-16T03:57:08.033 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.026+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.033 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.026+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=406177187}) 2024-09-16T03:57:08.033 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.026+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2590639158}) 2024-09-16T03:57:08.033 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.026+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.033 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.026+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=406177187}) 2024-09-16T03:57:08.033 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.026+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2380178903}) 2024-09-16T03:57:08.035 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.030+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.035 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.030+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=163572822}) 2024-09-16T03:57:08.035 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.030+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=465715446}) 2024-09-16T03:57:08.035 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.030+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.035 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.030+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=163572822}) 2024-09-16T03:57:08.035 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.030+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=317995887}) 2024-09-16T03:57:08.037 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.030+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.037 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.030+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1388758213}) 2024-09-16T03:57:08.037 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.030+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3220360003}) 2024-09-16T03:57:08.037 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.030+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.037 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.030+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1388758213}) 2024-09-16T03:57:08.038 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.030+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4031015314}) 2024-09-16T03:57:08.039 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.034+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.039 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.034+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2127606363}) 2024-09-16T03:57:08.039 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.034+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2167625052}) 2024-09-16T03:57:08.039 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.034+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.039 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.034+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2127606363}) 2024-09-16T03:57:08.040 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.034+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4078278843}) 2024-09-16T03:57:08.041 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.034+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.041 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.034+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1524800213}) 2024-09-16T03:57:08.041 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.034+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4158182921}) 2024-09-16T03:57:08.041 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.034+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.042 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.034+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1524800213}) 2024-09-16T03:57:08.042 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.034+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3187525766}) 2024-09-16T03:57:08.043 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.038+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.043 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.038+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1437640005}) 2024-09-16T03:57:08.043 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.038+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3745328985}) 2024-09-16T03:57:08.043 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.038+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.043 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.038+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1437640005}) 2024-09-16T03:57:08.044 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.038+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1973260473}) 2024-09-16T03:57:08.045 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.038+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.045 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.038+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=535837844}) 2024-09-16T03:57:08.045 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.038+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2813556731}) 2024-09-16T03:57:08.045 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.038+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.045 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.038+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=535837844}) 2024-09-16T03:57:08.045 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.038+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=588479713}) 2024-09-16T03:57:08.048 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.042+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.048 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.042+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=448682156}) 2024-09-16T03:57:08.048 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.042+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=57403105}) 2024-09-16T03:57:08.048 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.042+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.048 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.042+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=448682156}) 2024-09-16T03:57:08.048 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.042+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1067361333}) 2024-09-16T03:57:08.049 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.046+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.050 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.046+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1945441454}) 2024-09-16T03:57:08.050 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.046+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1155329120}) 2024-09-16T03:57:08.050 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.046+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.050 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.046+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1945441454}) 2024-09-16T03:57:08.050 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.046+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2804382754}) 2024-09-16T03:57:08.051 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.046+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.052 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.046+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2151540027}) 2024-09-16T03:57:08.052 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.046+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1137123284}) 2024-09-16T03:57:08.052 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.046+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.052 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.046+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2151540027}) 2024-09-16T03:57:08.052 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.046+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3112492906}) 2024-09-16T03:57:08.055 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.050+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.055 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.050+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3919339797}) 2024-09-16T03:57:08.055 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.050+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=12040854}) 2024-09-16T03:57:08.055 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.050+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.055 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.050+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3919339797}) 2024-09-16T03:57:08.055 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.050+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1423685502}) 2024-09-16T03:57:08.056 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.050+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.056 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.050+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1929147587}) 2024-09-16T03:57:08.056 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.050+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=112695514}) 2024-09-16T03:57:08.056 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.050+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.056 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.050+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1929147587}) 2024-09-16T03:57:08.056 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.050+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=134766020}) 2024-09-16T03:57:08.057 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.054+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.058 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.054+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1868050637}) 2024-09-16T03:57:08.058 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.054+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1505720270}) 2024-09-16T03:57:08.058 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.054+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.058 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.054+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1868050637}) 2024-09-16T03:57:08.058 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.054+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2461223545}) 2024-09-16T03:57:08.060 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.054+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.060 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.054+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1197652575}) 2024-09-16T03:57:08.060 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.054+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4189651486}) 2024-09-16T03:57:08.060 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.054+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.060 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.054+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1197652575}) 2024-09-16T03:57:08.060 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.054+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1633666447}) 2024-09-16T03:57:08.062 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.058+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.063 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.058+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=370203162}) 2024-09-16T03:57:08.063 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.058+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2136581868}) 2024-09-16T03:57:08.063 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.058+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.063 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.058+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=370203162}) 2024-09-16T03:57:08.063 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.058+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3312274530}) 2024-09-16T03:57:08.064 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.058+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.064 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.058+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1342266231}) 2024-09-16T03:57:08.064 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.058+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2099285556}) 2024-09-16T03:57:08.064 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.058+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.064 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.058+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1342266231}) 2024-09-16T03:57:08.064 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.058+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2940729274}) 2024-09-16T03:57:08.066 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.062+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.066 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.062+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1459767608}) 2024-09-16T03:57:08.066 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.062+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2312193337}) 2024-09-16T03:57:08.066 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.062+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.066 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.062+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1459767608}) 2024-09-16T03:57:08.066 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.062+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3690486665}) 2024-09-16T03:57:08.067 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.062+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.068 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.062+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1747394332}) 2024-09-16T03:57:08.068 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.062+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1442614098}) 2024-09-16T03:57:08.068 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.062+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.068 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.062+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1747394332}) 2024-09-16T03:57:08.068 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.062+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2957058670}) 2024-09-16T03:57:08.070 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.066+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.070 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.066+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4000920641}) 2024-09-16T03:57:08.070 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.066+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1261799278}) 2024-09-16T03:57:08.070 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.066+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.070 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.066+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4000920641}) 2024-09-16T03:57:08.070 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.066+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1824387505}) 2024-09-16T03:57:08.072 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.066+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.072 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.066+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3687143833}) 2024-09-16T03:57:08.072 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.066+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3835246043}) 2024-09-16T03:57:08.072 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.066+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.072 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.066+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3687143833}) 2024-09-16T03:57:08.072 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.066+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1725228532}) 2024-09-16T03:57:08.073 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.070+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.074 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.070+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1411688513}) 2024-09-16T03:57:08.074 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.070+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4170912395}) 2024-09-16T03:57:08.074 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.070+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.074 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.070+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1411688513}) 2024-09-16T03:57:08.074 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.070+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3134840379}) 2024-09-16T03:57:08.076 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.070+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.076 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.070+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=107563460}) 2024-09-16T03:57:08.076 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.070+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1262991712}) 2024-09-16T03:57:08.076 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.070+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.076 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.070+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=107563460}) 2024-09-16T03:57:08.076 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.070+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1697601284}) 2024-09-16T03:57:08.077 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.074+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.077 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.074+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2087304392}) 2024-09-16T03:57:08.077 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.074+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3211141737}) 2024-09-16T03:57:08.077 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.074+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.078 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.074+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2087304392}) 2024-09-16T03:57:08.078 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.074+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3055505223}) 2024-09-16T03:57:08.079 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.074+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.079 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.074+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2623908849}) 2024-09-16T03:57:08.080 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.074+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2429907482}) 2024-09-16T03:57:08.080 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.074+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.080 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.074+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2623908849}) 2024-09-16T03:57:08.080 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.074+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4275787784}) 2024-09-16T03:57:08.081 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.078+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.082 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.078+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2370379527}) 2024-09-16T03:57:08.082 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.078+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2916155975}) 2024-09-16T03:57:08.082 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.078+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.082 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.078+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2370379527}) 2024-09-16T03:57:08.082 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.078+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2705287164}) 2024-09-16T03:57:08.084 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.078+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.084 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.078+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=755737299}) 2024-09-16T03:57:08.084 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.078+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=476101989}) 2024-09-16T03:57:08.084 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.078+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.084 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.078+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=755737299}) 2024-09-16T03:57:08.084 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.078+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=485263598}) 2024-09-16T03:57:08.086 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.082+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.086 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.082+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3372588211}) 2024-09-16T03:57:08.086 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.082+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1870811628}) 2024-09-16T03:57:08.086 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.082+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.086 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.082+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3372588211}) 2024-09-16T03:57:08.086 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.082+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=598337119}) 2024-09-16T03:57:08.088 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.082+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.088 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.082+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3989872697}) 2024-09-16T03:57:08.088 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.082+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=912767861}) 2024-09-16T03:57:08.088 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.082+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.088 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.082+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3989872697}) 2024-09-16T03:57:08.089 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.082+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2891020284}) 2024-09-16T03:57:08.090 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.086+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.091 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.086+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3034044610}) 2024-09-16T03:57:08.091 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.086+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=270494705}) 2024-09-16T03:57:08.091 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.086+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.091 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.086+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3034044610}) 2024-09-16T03:57:08.091 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.086+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=29319402}) 2024-09-16T03:57:08.092 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.086+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.093 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.086+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=203920096}) 2024-09-16T03:57:08.093 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.086+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4223983786}) 2024-09-16T03:57:08.093 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.086+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.093 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.086+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=203920096}) 2024-09-16T03:57:08.093 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.086+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2339473016}) 2024-09-16T03:57:08.095 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.090+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.095 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.090+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3214537751}) 2024-09-16T03:57:08.095 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.090+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2182274898}) 2024-09-16T03:57:08.095 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.090+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.095 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.090+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3214537751}) 2024-09-16T03:57:08.095 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.090+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=82860665}) 2024-09-16T03:57:08.097 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.090+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.097 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.090+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1685541994}) 2024-09-16T03:57:08.098 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.090+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4253185047}) 2024-09-16T03:57:08.098 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.090+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.098 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.090+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1685541994}) 2024-09-16T03:57:08.098 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.090+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=479871464}) 2024-09-16T03:57:08.100 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.094+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.100 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.094+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1477572339}) 2024-09-16T03:57:08.100 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.094+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=908494829}) 2024-09-16T03:57:08.101 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.094+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.101 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.094+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1477572339}) 2024-09-16T03:57:08.101 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.094+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3138510558}) 2024-09-16T03:57:08.102 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.098+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.102 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.098+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2413282025}) 2024-09-16T03:57:08.102 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.098+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3606224873}) 2024-09-16T03:57:08.102 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.098+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.102 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.098+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2413282025}) 2024-09-16T03:57:08.102 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.098+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2514473267}) 2024-09-16T03:57:08.104 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.098+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.104 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.098+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=983499018}) 2024-09-16T03:57:08.104 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.098+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3591245145}) 2024-09-16T03:57:08.104 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.098+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.104 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.098+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=983499018}) 2024-09-16T03:57:08.104 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.098+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2005974515}) 2024-09-16T03:57:08.106 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.102+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.106 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.102+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=467529511}) 2024-09-16T03:57:08.106 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.102+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4208003966}) 2024-09-16T03:57:08.107 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.102+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.107 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.102+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=467529511}) 2024-09-16T03:57:08.107 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.102+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=902120019}) 2024-09-16T03:57:08.108 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.102+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.108 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.102+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1989614787}) 2024-09-16T03:57:08.109 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.102+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=955827998}) 2024-09-16T03:57:08.109 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.102+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.109 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.102+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1989614787}) 2024-09-16T03:57:08.109 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.102+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1161032957}) 2024-09-16T03:57:08.111 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.106+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.111 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.106+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3545112516}) 2024-09-16T03:57:08.111 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.106+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1821385386}) 2024-09-16T03:57:08.111 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.106+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.111 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.106+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3545112516}) 2024-09-16T03:57:08.111 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.106+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1924930808}) 2024-09-16T03:57:08.113 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.106+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.113 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.106+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2965103271}) 2024-09-16T03:57:08.113 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.106+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3475324489}) 2024-09-16T03:57:08.113 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.106+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.113 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.106+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2965103271}) 2024-09-16T03:57:08.113 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.106+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2604055442}) 2024-09-16T03:57:08.115 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.110+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.115 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.110+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1303576951}) 2024-09-16T03:57:08.115 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.110+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=472686696}) 2024-09-16T03:57:08.115 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.110+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.115 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.110+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1303576951}) 2024-09-16T03:57:08.115 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.110+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2163582829}) 2024-09-16T03:57:08.117 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.110+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.117 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.110+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3889943583}) 2024-09-16T03:57:08.117 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.110+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3836470143}) 2024-09-16T03:57:08.117 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.110+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.117 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.110+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3889943583}) 2024-09-16T03:57:08.117 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.110+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2831986043}) 2024-09-16T03:57:08.119 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.114+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.119 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.114+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2167755719}) 2024-09-16T03:57:08.119 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.114+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2594579506}) 2024-09-16T03:57:08.119 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.114+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.119 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.114+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2167755719}) 2024-09-16T03:57:08.119 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.114+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=907524556}) 2024-09-16T03:57:08.121 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.114+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.121 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.118+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2276744139}) 2024-09-16T03:57:08.121 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.118+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3851806374}) 2024-09-16T03:57:08.121 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.118+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.122 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.118+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2276744139}) 2024-09-16T03:57:08.122 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.118+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=614102741}) 2024-09-16T03:57:08.123 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.118+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.123 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.118+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1214978584}) 2024-09-16T03:57:08.123 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.118+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2777817501}) 2024-09-16T03:57:08.123 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.118+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.123 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.118+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1214978584}) 2024-09-16T03:57:08.124 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.118+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1046621521}) 2024-09-16T03:57:08.125 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.122+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.125 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.122+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2879930009}) 2024-09-16T03:57:08.126 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.122+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1872230219}) 2024-09-16T03:57:08.126 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.122+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.126 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.122+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2879930009}) 2024-09-16T03:57:08.126 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.122+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4219925042}) 2024-09-16T03:57:08.127 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.122+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.127 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.122+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=9735220}) 2024-09-16T03:57:08.127 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.122+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2938761531}) 2024-09-16T03:57:08.127 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.122+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.128 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.122+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=9735220}) 2024-09-16T03:57:08.128 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.122+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3172907611}) 2024-09-16T03:57:08.130 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.126+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.130 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.126+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3903601446}) 2024-09-16T03:57:08.130 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.126+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2012780106}) 2024-09-16T03:57:08.130 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.126+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.130 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.126+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3903601446}) 2024-09-16T03:57:08.130 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.126+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1451081922}) 2024-09-16T03:57:08.132 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.126+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.132 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.126+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3568417019}) 2024-09-16T03:57:08.132 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.126+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4271055758}) 2024-09-16T03:57:08.133 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.126+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.133 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.126+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3568417019}) 2024-09-16T03:57:08.133 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.126+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2861206179}) 2024-09-16T03:57:08.134 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.130+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.135 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.130+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=478527829}) 2024-09-16T03:57:08.135 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.130+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1275930638}) 2024-09-16T03:57:08.135 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.130+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.135 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.130+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=478527829}) 2024-09-16T03:57:08.135 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.130+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=830005893}) 2024-09-16T03:57:08.137 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.130+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.137 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.130+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3745568666}) 2024-09-16T03:57:08.137 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.130+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3924780274}) 2024-09-16T03:57:08.137 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.134+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.137 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.134+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3745568666}) 2024-09-16T03:57:08.138 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.134+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1140618012}) 2024-09-16T03:57:08.140 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.134+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.140 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.134+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3179230396}) 2024-09-16T03:57:08.140 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.134+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1501200072}) 2024-09-16T03:57:08.140 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.134+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.140 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.134+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3179230396}) 2024-09-16T03:57:08.140 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.134+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=575431624}) 2024-09-16T03:57:08.142 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.138+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.142 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.138+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=642149484}) 2024-09-16T03:57:08.142 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.138+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2647403291}) 2024-09-16T03:57:08.142 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.138+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.142 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.138+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=642149484}) 2024-09-16T03:57:08.142 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.138+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=76245133}) 2024-09-16T03:57:08.144 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.138+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.144 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.138+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4055134127}) 2024-09-16T03:57:08.144 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.138+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3930316406}) 2024-09-16T03:57:08.144 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.138+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.144 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.138+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4055134127}) 2024-09-16T03:57:08.144 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.138+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1414451103}) 2024-09-16T03:57:08.146 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.142+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.146 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.142+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3572540858}) 2024-09-16T03:57:08.146 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.142+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1846577351}) 2024-09-16T03:57:08.146 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.142+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.146 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.142+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3572540858}) 2024-09-16T03:57:08.146 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.142+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2754651086}) 2024-09-16T03:57:08.148 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.142+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.148 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.142+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1004851068}) 2024-09-16T03:57:08.148 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.142+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2675846012}) 2024-09-16T03:57:08.149 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.142+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.149 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.142+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1004851068}) 2024-09-16T03:57:08.149 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.142+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1900041147}) 2024-09-16T03:57:08.151 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.146+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.151 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.146+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2188345559}) 2024-09-16T03:57:08.151 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.146+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1758008747}) 2024-09-16T03:57:08.151 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.146+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.151 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.146+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2188345559}) 2024-09-16T03:57:08.151 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.146+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3900671827}) 2024-09-16T03:57:08.153 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.146+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.153 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.146+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1495139264}) 2024-09-16T03:57:08.153 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.146+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=791377549}) 2024-09-16T03:57:08.153 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.146+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.153 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.146+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1495139264}) 2024-09-16T03:57:08.153 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.146+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=267079580}) 2024-09-16T03:57:08.157 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.150+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.158 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.150+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1209509147}) 2024-09-16T03:57:08.158 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.150+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3781141073}) 2024-09-16T03:57:08.158 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.150+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.158 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.150+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1209509147}) 2024-09-16T03:57:08.158 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.150+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1564196081}) 2024-09-16T03:57:08.158 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.154+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.158 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.154+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=599537344}) 2024-09-16T03:57:08.158 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.154+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2538896867}) 2024-09-16T03:57:08.158 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.154+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.158 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.154+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=599537344}) 2024-09-16T03:57:08.158 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.154+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1748743283}) 2024-09-16T03:57:08.160 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.154+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.160 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.154+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=423362799}) 2024-09-16T03:57:08.160 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.154+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3396389543}) 2024-09-16T03:57:08.160 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.154+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.160 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.154+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=423362799}) 2024-09-16T03:57:08.160 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.154+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=328884206}) 2024-09-16T03:57:08.162 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.158+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.162 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.158+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4282184850}) 2024-09-16T03:57:08.162 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.158+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=956179994}) 2024-09-16T03:57:08.162 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.158+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.162 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.158+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4282184850}) 2024-09-16T03:57:08.162 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.158+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1504986036}) 2024-09-16T03:57:08.164 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.158+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.165 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.158+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=198263548}) 2024-09-16T03:57:08.165 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.158+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=721966310}) 2024-09-16T03:57:08.165 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.158+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.165 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.158+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=198263548}) 2024-09-16T03:57:08.165 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.158+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1204128830}) 2024-09-16T03:57:08.167 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.162+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.167 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.162+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4055858722}) 2024-09-16T03:57:08.167 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.162+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2385064215}) 2024-09-16T03:57:08.167 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.162+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.167 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.162+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4055858722}) 2024-09-16T03:57:08.168 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.162+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=459103813}) 2024-09-16T03:57:08.169 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.162+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.169 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.162+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=295843367}) 2024-09-16T03:57:08.169 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.162+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3159659125}) 2024-09-16T03:57:08.169 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.162+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.169 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.162+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=295843367}) 2024-09-16T03:57:08.169 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.162+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3689962874}) 2024-09-16T03:57:08.171 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.166+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.171 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.166+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3391869904}) 2024-09-16T03:57:08.171 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.166+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=411458165}) 2024-09-16T03:57:08.171 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.166+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.171 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.166+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3391869904}) 2024-09-16T03:57:08.171 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.166+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=768346390}) 2024-09-16T03:57:08.173 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.166+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.173 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.170+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=617423756}) 2024-09-16T03:57:08.173 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.170+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2700766766}) 2024-09-16T03:57:08.174 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.170+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.174 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.170+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=617423756}) 2024-09-16T03:57:08.174 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.170+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=93160902}) 2024-09-16T03:57:08.176 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.170+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.176 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.170+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3731521991}) 2024-09-16T03:57:08.176 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.170+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3906772909}) 2024-09-16T03:57:08.176 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.170+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.176 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.170+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3731521991}) 2024-09-16T03:57:08.176 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.170+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1851070427}) 2024-09-16T03:57:08.178 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.174+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.178 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.174+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=39522925}) 2024-09-16T03:57:08.178 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.174+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3974542461}) 2024-09-16T03:57:08.178 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.174+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.178 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.174+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=39522925}) 2024-09-16T03:57:08.178 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.174+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4097601855}) 2024-09-16T03:57:08.180 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.174+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.181 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.174+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3517380082}) 2024-09-16T03:57:08.181 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.174+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=491975219}) 2024-09-16T03:57:08.181 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.174+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.181 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.174+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3517380082}) 2024-09-16T03:57:08.181 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.174+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3926646073}) 2024-09-16T03:57:08.182 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.178+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.183 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.178+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1775289417}) 2024-09-16T03:57:08.183 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.178+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4148969006}) 2024-09-16T03:57:08.183 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.178+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.183 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.178+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1775289417}) 2024-09-16T03:57:08.183 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.178+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1684525045}) 2024-09-16T03:57:08.185 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.178+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.185 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.178+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=189294818}) 2024-09-16T03:57:08.185 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.178+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1090541776}) 2024-09-16T03:57:08.185 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.178+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.185 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.178+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=189294818}) 2024-09-16T03:57:08.185 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.178+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3043609500}) 2024-09-16T03:57:08.187 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.182+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.187 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.182+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4198323131}) 2024-09-16T03:57:08.187 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.182+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1854311266}) 2024-09-16T03:57:08.187 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.182+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.187 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.182+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4198323131}) 2024-09-16T03:57:08.188 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.182+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3473250250}) 2024-09-16T03:57:08.189 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.182+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.189 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.182+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=805786499}) 2024-09-16T03:57:08.189 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.182+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1534760433}) 2024-09-16T03:57:08.189 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.182+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.189 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.186+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=805786499}) 2024-09-16T03:57:08.189 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.186+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2842963598}) 2024-09-16T03:57:08.191 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.186+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.191 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.186+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2538958341}) 2024-09-16T03:57:08.191 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.186+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=189760841}) 2024-09-16T03:57:08.191 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.186+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.191 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.186+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2538958341}) 2024-09-16T03:57:08.191 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.186+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4054042157}) 2024-09-16T03:57:08.193 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.190+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.193 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.190+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3199048617}) 2024-09-16T03:57:08.193 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.190+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3732916041}) 2024-09-16T03:57:08.194 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.190+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.194 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.190+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3199048617}) 2024-09-16T03:57:08.194 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.190+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3253947458}) 2024-09-16T03:57:08.196 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.190+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.196 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.190+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2079444124}) 2024-09-16T03:57:08.196 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.190+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2921577738}) 2024-09-16T03:57:08.196 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.190+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.196 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.190+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2079444124}) 2024-09-16T03:57:08.196 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.190+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=202900672}) 2024-09-16T03:57:08.199 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.194+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.199 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.194+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=414812697}) 2024-09-16T03:57:08.199 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.194+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1534255565}) 2024-09-16T03:57:08.199 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.194+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.199 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.194+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=414812697}) 2024-09-16T03:57:08.199 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.194+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2046946180}) 2024-09-16T03:57:08.200 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.194+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.200 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.194+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=439366054}) 2024-09-16T03:57:08.200 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.194+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2730167817}) 2024-09-16T03:57:08.200 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.194+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.200 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.194+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=439366054}) 2024-09-16T03:57:08.200 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.194+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3042611607}) 2024-09-16T03:57:08.203 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.198+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.203 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.198+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=49,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=10,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=6} crc {logm=7929332,mdsmap=4088524775,mgr=460109776,mgr_command_descs=2107673600,mgr_metadata=1367272202,mgrstat=3656281681,mon_config_key=1822420877,monmap=3937513981,osd_metadata=951161650,osd_pg_creating=960408616,osd_snap=2628328398}) 2024-09-16T03:57:08.203 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.198+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=50,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=3} crc {logm=3863578293,mdsmap=4088524775,mgr=460109776,mgr_command_descs=2107673600,mgr_metadata=1367272202,mgrstat=3225855592,mon_config_key=85057112,monmap=3937513981,osd_metadata=4182959540,osd_pg_creating=960408616,osd_snap=379090904}) 2024-09-16T03:57:08.203 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.198+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.203 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.198+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=49,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=10,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=6} crc {logm=7929332,mdsmap=4088524775,mgr=460109776,mgr_command_descs=2107673600,mgr_metadata=1367272202,mgrstat=3656281681,mon_config_key=1822420877,monmap=3937513981,osd_metadata=951161650,osd_pg_creating=960408616,osd_snap=2628328398}) 2024-09-16T03:57:08.204 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.198+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=50,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=11,monmap=2,osd_metadata=6,osd_pg_creating=1,osd_snap=5} crc {logm=1795358033,mdsmap=4088524775,mgr=911650295,mgr_command_descs=2107673600,mgr_metadata=1367272202,mgrstat=3225855592,mon_config_key=85057112,monmap=268092033,osd_metadata=4182959540,osd_pg_creating=960408616,osd_snap=3292740240}) 2024-09-16T03:57:08.205 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.198+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.205 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.198+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=28,osdmap=72} crc {osd_snap=266092045,osdmap=2594165333}) 2024-09-16T03:57:08.205 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.198+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=29,osdmap=71} crc {osd_snap=1968460099,osdmap=857027269}) 2024-09-16T03:57:08.206 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.198+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.206 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.198+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=28,osdmap=72} crc {osd_snap=266092045,osdmap=2594165333}) 2024-09-16T03:57:08.206 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.198+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=30,osdmap=70} crc {osd_snap=195464536,osdmap=3455531215}) 2024-09-16T03:57:08.209 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.202+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.209 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.202+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1549330204}) 2024-09-16T03:57:08.210 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.206+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3237219013}) 2024-09-16T03:57:08.210 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.206+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.210 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.206+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1549330204}) 2024-09-16T03:57:08.210 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.206+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2338438930}) 2024-09-16T03:57:08.222 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.218+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.222 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.218+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=25} crc {osdmap=1555426865}) 2024-09-16T03:57:08.222 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.218+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=24} crc {osdmap=2555306344}) 2024-09-16T03:57:08.222 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.218+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:57:08.222 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.218+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=25} crc {osdmap=1555426865}) 2024-09-16T03:57:08.223 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:57:08.218+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=23} crc {osdmap=879039809}) 2024-09-16T03:57:09.029 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T03:57:09.029 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:57:09.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.158+0000 7fc206ff1640 10 monclient: get_monmap_and_config 2024-09-16T03:57:09.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.158+0000 7fc206ff1640 10 monclient: build_initial_monmap 2024-09-16T03:57:09.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.158+0000 7fc206ff1640 10 monclient: monmap: 2024-09-16T03:57:09.163 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:57:09.163 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:57:09.164 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:57:09.161727+0000 2024-09-16T03:57:09.164 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:57:09.161727+0000 2024-09-16T03:57:09.164 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:57:09.164 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:57:09.164 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:57:09.164 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:57:09.164 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:57:09.164 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:57:09.164 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:57:09.164 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:57:09.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.158+0000 7fc206ff1640 1 Processor -- start 2024-09-16T03:57:09.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.158+0000 7fc206ff1640 1 -- start start 2024-09-16T03:57:09.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.158+0000 7fc206ff1640 10 monclient: init 2024-09-16T03:57:09.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.158+0000 7fc206ff1640 10 monclient: _reopen_session rank -1 2024-09-16T03:57:09.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.158+0000 7fc206ff1640 10 monclient: _add_conns ranks=[0,4,3,2,1] 2024-09-16T03:57:09.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.158+0000 7fc206ff1640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc2000a1330 addr v1:172.21.15.42:6789/0 2024-09-16T03:57:09.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.158+0000 7fc206ff1640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fc20006dac0 0x7fc20013a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:57:09.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.158+0000 7fc206ff1640 10 monclient(hunting): picked mon.noname-c con 0x7fc20006dac0 addr v2:172.21.15.154:3300/0 2024-09-16T03:57:09.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.158+0000 7fc206ff1640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fc2000a1e20 0x7fc20013c890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:57:09.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.158+0000 7fc206ff1640 10 monclient(hunting): picked mon.noname-a con 0x7fc2000a1e20 addr v2:172.21.15.42:3300/0 2024-09-16T03:57:09.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.158+0000 7fc206ff1640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc20009c090 con 0x7fc2000a1330 2024-09-16T03:57:09.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.158+0000 7fc206ff1640 10 monclient(hunting): start opening mon connection 2024-09-16T03:57:09.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.158+0000 7fc206ff1640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7fc20009f610 con 0x7fc2000a1e20 2024-09-16T03:57:09.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.158+0000 7fc206ff1640 10 monclient(hunting): start opening mon connection 2024-09-16T03:57:09.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.158+0000 7fc206ff1640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fc20009f8f0 con 0x7fc20006dac0 2024-09-16T03:57:09.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.158+0000 7fc206ff1640 10 monclient(hunting): _renew_subs 2024-09-16T03:57:09.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.158+0000 7fc206ff1640 10 monclient(hunting): authenticate will time out at 1876.131716s 2024-09-16T03:57:09.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.158+0000 7fc204d8d640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fc2000a1330 0x7fc2000a1710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:44294/0 (socket says 172.21.15.42:44294) 2024-09-16T03:57:09.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.158+0000 7fc204d8d640 1 -- 172.21.15.42:0/2956679333 learned_addr learned my addr 172.21.15.42:0/2956679333 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:57:09.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.162+0000 7fc20558e640 1 --2- 172.21.15.42:0/2956679333 >> v2:172.21.15.42:3300/0 conn(0x7fc2000a1e20 0x7fc20013c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:57:09.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.162+0000 7fc20558e640 10 monclient(hunting): get_auth_request con 0x7fc2000a1e20 auth_method 0 2024-09-16T03:57:09.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.162+0000 7fc20558e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:57:09.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.162+0000 7fc20558e640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:57:09.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.162+0000 7fc20558e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:57:09.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.162+0000 7fc1fffff640 1 --2- 172.21.15.42:0/2956679333 >> v2:172.21.15.154:3300/0 conn(0x7fc20006dac0 0x7fc20013a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:57:09.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.162+0000 7fc20558e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:57:09.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.162+0000 7fc20558e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:57:09.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.162+0000 7fc20558e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:57:09.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.162+0000 7fc1fffff640 10 monclient(hunting): get_auth_request con 0x7fc20006dac0 auth_method 0 2024-09-16T03:57:09.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.162+0000 7fc1fffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:57:09.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.162+0000 7fc1fffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:57:09.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.162+0000 7fc1fffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:57:09.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.162+0000 7fc1ff7fe640 1 -- 172.21.15.42:0/2956679333 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 596169990 0 0) 0x7fc20009c090 con 0x7fc2000a1330 2024-09-16T03:57:09.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.162+0000 7fc1ff7fe640 10 monclient(hunting): _negotiate 2024-09-16T03:57:09.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.162+0000 7fc1ff7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:57:09.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.162+0000 7fc1ff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:57:09.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.162+0000 7fc1ff7fe640 10 monclient(hunting): my global_id is 824838 2024-09-16T03:57:09.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.162+0000 7fc1ff7fe640 1 -- 172.21.15.42:0/2956679333 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc1e8003530 con 0x7fc2000a1330 2024-09-16T03:57:09.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.162+0000 7fc1fffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:57:09.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.162+0000 7fc1fffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:57:09.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.162+0000 7fc1fffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:57:09.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.162+0000 7fc20558e640 10 monclient(hunting): handle_auth_done global_id 824835 payload 274 2024-09-16T03:57:09.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.162+0000 7fc20558e640 1 -- 172.21.15.42:0/2956679333 >> v2:172.21.15.154:3300/0 conn(0x7fc20006dac0 msgr2=0x7fc20013a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:57:09.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.162+0000 7fc20558e640 1 --2- 172.21.15.42:0/2956679333 >> v2:172.21.15.154:3300/0 conn(0x7fc20006dac0 0x7fc20013a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:57:09.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.162+0000 7fc20558e640 1 -- 172.21.15.42:0/2956679333 >> v1:172.21.15.42:6789/0 conn(0x7fc2000a1330 legacy=0x7fc2000a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:57:09.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.162+0000 7fc20558e640 10 monclient: _finish_hunting 0 2024-09-16T03:57:09.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.162+0000 7fc20558e640 1 monclient: found mon.noname-a 2024-09-16T03:57:09.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.162+0000 7fc20558e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:57:09.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.162+0000 7fc20558e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:57:09.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.162+0000 7fc20558e640 1 -- 172.21.15.42:0/2956679333 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc20009fbd0 con 0x7fc2000a1e20 2024-09-16T03:57:09.181 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.162+0000 7fc20558e640 1 --2- 172.21.15.42:0/2956679333 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc2000a1e20 0x7fc20013c890 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fc20006da10 tx=0x7fc1f801c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86d45be6a46bd9f9 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:57:09.181 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.162+0000 7fc1ff7fe640 1 -- 172.21.15.42:0/2956679333 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1f80b2070 con 0x7fc2000a1e20 2024-09-16T03:57:09.181 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.162+0000 7fc1ff7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:57:09.181 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.162+0000 7fc1ff7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:57:09.181 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.162+0000 7fc1ff7fe640 10 monclient: dump: 2024-09-16T03:57:09.181 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:57:09.181 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:57:09.181 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:57:09.181 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:57:09.181 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:57:09.181 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:57:09.182 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:57:09.182 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:57:09.182 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:57:09.182 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:57:09.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.162+0000 7fc1ff7fe640 10 monclient: _finish_auth 0 2024-09-16T03:57:09.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.162+0000 7fc1ff7fe640 10 monclient: _check_auth_tickets 2024-09-16T03:57:09.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.162+0000 7fc1ff7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:57:09.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.162+0000 7fc206ff1640 5 monclient: authenticate success, global_id 824835 2024-09-16T03:57:09.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.162+0000 7fc206ff1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:57:09.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.162+0000 7fc1ff7fe640 1 -- 172.21.15.42:0/2956679333 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc1f80a57d0 con 0x7fc2000a1e20 2024-09-16T03:57:09.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.162+0000 7fc1ff7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:57:09.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.162+0000 7fc206ff1640 10 monclient: get_monmap_and_config success 2024-09-16T03:57:09.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.162+0000 7fc206ff1640 4 set_mon_vals no callback set 2024-09-16T03:57:09.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.162+0000 7fc1ff7fe640 1 -- 172.21.15.42:0/2956679333 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1f80a5b80 con 0x7fc2000a1e20 2024-09-16T03:57:09.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.162+0000 7fc1ff7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:57:09.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.162+0000 7fc1ff7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:57:09.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.162+0000 7fc1ff7fe640 10 monclient: dump: 2024-09-16T03:57:09.183 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:57:09.183 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:57:09.183 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:57:09.183 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:57:09.183 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:57:09.183 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:57:09.183 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:57:09.183 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:57:09.183 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:57:09.184 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:57:09.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.162+0000 7fc206ff1640 10 monclient: shutdown 2024-09-16T03:57:09.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.162+0000 7fc206ff1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:57:09.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.162+0000 7fc206ff1640 1 -- 172.21.15.42:0/2956679333 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc2000a1e20 msgr2=0x7fc20013c890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:57:09.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.162+0000 7fc206ff1640 1 --2- 172.21.15.42:0/2956679333 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc2000a1e20 0x7fc20013c890 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fc20006da10 tx=0x7fc1f801c910 comp rx=0 tx=0).stop 2024-09-16T03:57:09.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.162+0000 7fc206ff1640 1 -- 172.21.15.42:0/2956679333 shutdown_connections 2024-09-16T03:57:09.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.162+0000 7fc206ff1640 1 --2- 172.21.15.42:0/2956679333 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc2000a1e20 0x7fc20013c890 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:57:09.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.162+0000 7fc206ff1640 1 --2- 172.21.15.42:0/2956679333 >> v2:172.21.15.154:3300/0 conn(0x7fc20006dac0 0x7fc20013a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:57:09.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.162+0000 7fc206ff1640 1 -- 172.21.15.42:0/2956679333 >> 172.21.15.42:0/2956679333 conn(0x7fc2000a2420 msgr2=0x7fc2000a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:57:09.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.162+0000 7fc206ff1640 1 -- 172.21.15.42:0/2956679333 shutdown_connections 2024-09-16T03:57:09.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.162+0000 7fc206ff1640 1 -- 172.21.15.42:0/2956679333 wait complete. 2024-09-16T03:57:09.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.166+0000 7fc206ff1640 10 monclient: build_initial_monmap 2024-09-16T03:57:09.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.166+0000 7fc206ff1640 10 monclient: monmap: 2024-09-16T03:57:09.185 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:57:09.185 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:57:09.185 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:57:09.185 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:57:09.185 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:57:09.185 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:57:09.185 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:57:09.185 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:57:09.185 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:57:09.186 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:57:09.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.166+0000 7fc206ff1640 1 librados: starting msgr at 2024-09-16T03:57:09.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.166+0000 7fc206ff1640 1 librados: starting objecter 2024-09-16T03:57:09.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.166+0000 7fc206ff1640 1 Processor -- start 2024-09-16T03:57:09.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.166+0000 7fc206ff1640 1 -- start start 2024-09-16T03:57:09.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.166+0000 7fc206ff1640 1 librados: setting wanted keys 2024-09-16T03:57:09.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.166+0000 7fc206ff1640 1 librados: calling monclient init 2024-09-16T03:57:09.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.166+0000 7fc206ff1640 10 monclient: init 2024-09-16T03:57:09.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.166+0000 7fc206ff1640 10 monclient: _reopen_session rank -1 2024-09-16T03:57:09.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.166+0000 7fc206ff1640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:57:09.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.166+0000 7fc206ff1640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc2000a1330 0x7fc2001ccc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:57:09.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.166+0000 7fc206ff1640 10 monclient(hunting): picked mon.noname-c con 0x7fc2000a1330 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:57:09.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.166+0000 7fc206ff1640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc2000a1e20 0x7fc2001352c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:57:09.187 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.166+0000 7fc206ff1640 10 monclient(hunting): picked mon.noname-a con 0x7fc2000a1e20 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:57:09.187 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.166+0000 7fc206ff1640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc200135800 0x7fc20011bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:57:09.187 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.166+0000 7fc206ff1640 10 monclient(hunting): picked mon.noname-b con 0x7fc200135800 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:57:09.187 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.166+0000 7fc206ff1640 10 monclient(hunting): start opening mon connection 2024-09-16T03:57:09.187 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.166+0000 7fc206ff1640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fc20013f800 con 0x7fc2000a1e20 2024-09-16T03:57:09.187 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.166+0000 7fc206ff1640 10 monclient(hunting): start opening mon connection 2024-09-16T03:57:09.187 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.166+0000 7fc206ff1640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc20009ef80 con 0x7fc200135800 2024-09-16T03:57:09.187 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.166+0000 7fc206ff1640 10 monclient(hunting): start opening mon connection 2024-09-16T03:57:09.187 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.166+0000 7fc206ff1640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fc20013f980 con 0x7fc2000a1330 2024-09-16T03:57:09.187 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.166+0000 7fc206ff1640 10 monclient(hunting): _renew_subs 2024-09-16T03:57:09.187 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.166+0000 7fc206ff1640 10 monclient(hunting): authenticate will time out at 1876.137141s 2024-09-16T03:57:09.188 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.166+0000 7fc204d8d640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc2000a1330 0x7fc2001ccc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:57:09.188 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.166+0000 7fc1fffff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc2000a1e20 0x7fc2001352c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:57:09.188 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.166+0000 7fc1fffff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc2000a1e20 0x7fc2001352c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:58282/0 (socket says 172.21.15.42:58282) 2024-09-16T03:57:09.188 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.166+0000 7fc1fffff640 1 -- 172.21.15.42:0/3184305338 learned_addr learned my addr 172.21.15.42:0/3184305338 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:57:09.188 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.166+0000 7fc1fffff640 10 monclient(hunting): get_auth_request con 0x7fc2000a1e20 auth_method 0 2024-09-16T03:57:09.188 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.166+0000 7fc1fffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:57:09.188 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.166+0000 7fc1fffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:57:09.188 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.166+0000 7fc1fffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:57:09.188 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.166+0000 7fc204d8d640 10 monclient(hunting): get_auth_request con 0x7fc2000a1330 auth_method 0 2024-09-16T03:57:09.188 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.166+0000 7fc204d8d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:57:09.189 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.166+0000 7fc204d8d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:57:09.189 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.166+0000 7fc204d8d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:57:09.189 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.166+0000 7fc1fffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:57:09.189 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.166+0000 7fc1fffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:57:09.189 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.166+0000 7fc1fffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:57:09.189 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.166+0000 7fc204d8d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:57:09.189 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.166+0000 7fc204d8d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:57:09.189 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.166+0000 7fc204d8d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:57:09.189 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.166+0000 7fc20558e640 1 --2- 172.21.15.42:0/3184305338 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc200135800 0x7fc20011bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:57:09.190 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.166+0000 7fc20558e640 10 monclient(hunting): get_auth_request con 0x7fc200135800 auth_method 0 2024-09-16T03:57:09.190 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.166+0000 7fc20558e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:57:09.190 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.166+0000 7fc20558e640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:57:09.190 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.166+0000 7fc20558e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:57:09.190 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.166+0000 7fc204d8d640 10 monclient(hunting): handle_auth_done global_id 854384 payload 824 2024-09-16T03:57:09.190 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.166+0000 7fc204d8d640 1 -- 172.21.15.42:0/3184305338 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc200135800 msgr2=0x7fc20011bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:57:09.190 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.166+0000 7fc204d8d640 1 --2- 172.21.15.42:0/3184305338 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc200135800 0x7fc20011bf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:57:09.190 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.166+0000 7fc204d8d640 1 -- 172.21.15.42:0/3184305338 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc2000a1e20 msgr2=0x7fc2001352c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:57:09.190 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.166+0000 7fc204d8d640 1 --2- 172.21.15.42:0/3184305338 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc2000a1e20 0x7fc2001352c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:57:09.190 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.166+0000 7fc204d8d640 10 monclient: _finish_hunting 0 2024-09-16T03:57:09.191 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.166+0000 7fc204d8d640 1 monclient: found mon.noname-c 2024-09-16T03:57:09.191 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.166+0000 7fc204d8d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:57:09.191 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.166+0000 7fc204d8d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:57:09.191 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.166+0000 7fc204d8d640 1 -- 172.21.15.42:0/3184305338 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc200073a30 con 0x7fc2000a1330 2024-09-16T03:57:09.191 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.166+0000 7fc1fffff640 1 --2- 172.21.15.42:0/3184305338 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc2000a1e20 0x7fc2001352c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:57:09.191 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.166+0000 7fc20558e640 1 --2- 172.21.15.42:0/3184305338 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc200135800 0x7fc20011bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T03:57:09.191 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.166+0000 7fc204d8d640 1 --2- 172.21.15.42:0/3184305338 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc2000a1330 0x7fc2001ccc10 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fc1f0007fd0 tx=0x7fc1f000c890 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:57:09.191 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.166+0000 7fc1fd7fa640 10 client.?.objecter ms_handle_connect 0x7fc2000a1330 2024-09-16T03:57:09.191 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.166+0000 7fc1fd7fa640 10 client.?.objecter resend_mon_ops 2024-09-16T03:57:09.191 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.166+0000 7fc1fd7fa640 1 -- 172.21.15.42:0/3184305338 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1f0017070 con 0x7fc2000a1330 2024-09-16T03:57:09.191 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.166+0000 7fc1fd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:57:09.191 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.166+0000 7fc1fd7fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:57:09.192 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.166+0000 7fc1fd7fa640 10 monclient: dump: 2024-09-16T03:57:09.192 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:57:09.192 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:57:09.192 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:57:09.192 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:57:09.192 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:57:09.192 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:57:09.192 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:57:09.192 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:57:09.192 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:57:09.192 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:57:09.193 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.166+0000 7fc1fd7fa640 10 monclient: _finish_auth 0 2024-09-16T03:57:09.193 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.166+0000 7fc1fd7fa640 10 monclient: _check_auth_tickets 2024-09-16T03:57:09.193 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.166+0000 7fc1fd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:57:09.193 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.166+0000 7fc1fd7fa640 1 -- 172.21.15.42:0/3184305338 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc1f0005cf0 con 0x7fc2000a1330 2024-09-16T03:57:09.193 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.166+0000 7fc1fd7fa640 10 monclient: handle_config config(0 keys) 2024-09-16T03:57:09.193 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.166+0000 7fc1fd7fa640 1 -- 172.21.15.42:0/3184305338 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1f00072f0 con 0x7fc2000a1330 2024-09-16T03:57:09.193 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.166+0000 7fc1fd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:57:09.193 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.166+0000 7fc1fd7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:57:09.193 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.166+0000 7fc1fd7fa640 10 monclient: dump: 2024-09-16T03:57:09.193 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:57:09.194 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:57:09.194 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:57:09.194 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:57:09.194 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:57:09.194 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:57:09.194 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:57:09.194 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:57:09.194 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:57:09.194 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:57:09.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.166+0000 7fc206ff1640 5 monclient: authenticate success, global_id 854384 2024-09-16T03:57:09.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.166+0000 7fc206ff1640 10 monclient: _renew_subs 2024-09-16T03:57:09.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.166+0000 7fc1feffd640 4 set_mon_vals no callback set 2024-09-16T03:57:09.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.166+0000 7fc206ff1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:57:09.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.166+0000 7fc206ff1640 1 -- 172.21.15.42:0/3184305338 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc20013c380 con 0x7fc2000a1330 2024-09-16T03:57:09.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.166+0000 7fc206ff1640 10 client.854384.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:57:09.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.166+0000 7fc206ff1640 10 monclient: _renew_subs 2024-09-16T03:57:09.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.166+0000 7fc206ff1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:57:09.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.166+0000 7fc206ff1640 1 -- 172.21.15.42:0/3184305338 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc20011fc50 con 0x7fc2000a1330 2024-09-16T03:57:09.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.166+0000 7fc206ff1640 1 librados: init done 2024-09-16T03:57:09.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.166+0000 7fc206ff1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:57:09.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.166+0000 7fc206ff1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:57:09.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.166+0000 7fc206ff1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:57:09.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.166+0000 7fc206ff1640 1 -- 172.21.15.42:0/3184305338 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc2000a1710 con 0x7fc2000a1330 2024-09-16T03:57:09.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.166+0000 7fc1fd7fa640 1 -- 172.21.15.42:0/3184305338 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fc1f001a070 con 0x7fc2000a1330 2024-09-16T03:57:09.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.166+0000 7fc1fd7fa640 1 --2- 172.21.15.42:0/3184305338 >> v2:172.21.15.154:6812/254845342 conn(0x7fc1ec037810 0x7fc1ec039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:57:09.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.166+0000 7fc1fd7fa640 1 -- 172.21.15.42:0/3184305338 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(459..459 src has 330..459) ==== 5328+0+0 (secure 0 0 0) 0x7fc1f0048110 con 0x7fc2000a1330 2024-09-16T03:57:09.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.166+0000 7fc1fd7fa640 10 client.854384.objecter ms_dispatch 0x7fc200000ba0 osd_map(459..459 src has 330..459) 2024-09-16T03:57:09.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.166+0000 7fc1fd7fa640 3 client.854384.objecter handle_osd_map got epochs [459,459] > 0 2024-09-16T03:57:09.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.166+0000 7fc1fd7fa640 3 client.854384.objecter handle_osd_map decoding full epoch 459 2024-09-16T03:57:09.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.166+0000 7fc1fd7fa640 20 client.854384.objecter dump_active .. 0 homeless 2024-09-16T03:57:09.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.166+0000 7fc1fffff640 1 --2- 172.21.15.42:0/3184305338 >> v2:172.21.15.154:6812/254845342 conn(0x7fc1ec037810 0x7fc1ec039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:57:09.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.166+0000 7fc1fffff640 10 monclient: get_auth_request con 0x7fc1ec037810 auth_method 0 2024-09-16T03:57:09.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.170+0000 7fc1fd7fa640 10 client.854384.objecter ms_handle_connect 0x7fc1ec037810 2024-09-16T03:57:09.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.170+0000 7fc1fffff640 1 --2- 172.21.15.42:0/3184305338 >> v2:172.21.15.154:6812/254845342 conn(0x7fc1ec037810 0x7fc1ec039cd0 secure :-1 s=READY pgs=3671 cs=0 l=1 rev1=1 crypto rx=0x7fc1f4011310 tx=0x7fc1f4009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:57:09.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.170+0000 7fc1fd7fa640 1 -- 172.21.15.42:0/3184305338 <== mon.2 v2:172.21.15.42:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7fc200073a30 con 0x7fc2000a1330 2024-09-16T03:57:09.197 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.174+0000 7fc1fd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:57:09.197 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.174+0000 7fc1fd7fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:57:09.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.394+0000 7fc206ff1640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:57:09.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.394+0000 7fc206ff1640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:57:09.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.394+0000 7fc206ff1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:57:09.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.394+0000 7fc206ff1640 1 -- 172.21.15.42:0/3184305338 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fc20006e090 con 0x7fc2000a1330 2024-09-16T03:57:09.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.398+0000 7fc1fd7fa640 1 -- 172.21.15.42:0/3184305338 <== mon.2 v2:172.21.15.42:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7fc1f0019e10 con 0x7fc2000a1330 2024-09-16T03:57:09.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.398+0000 7fc1fd7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:57:09.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.398+0000 7fc1fd7fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:57:09.403 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":140,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:57:09.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.398+0000 7fc206ff1640 10 librados: watch_flush enter 2024-09-16T03:57:09.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.398+0000 7fc206ff1640 10 librados: watch_flush exit 2024-09-16T03:57:09.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.398+0000 7fc206ff1640 20 client.854384.objecter shutdown clearing up homeless session... 2024-09-16T03:57:09.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.398+0000 7fc206ff1640 10 client.854384.objecter successfully canceled tick 2024-09-16T03:57:09.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.398+0000 7fc206ff1640 1 -- 172.21.15.42:0/3184305338 >> v2:172.21.15.154:6812/254845342 conn(0x7fc1ec037810 msgr2=0x7fc1ec039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:57:09.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.398+0000 7fc206ff1640 1 --2- 172.21.15.42:0/3184305338 >> v2:172.21.15.154:6812/254845342 conn(0x7fc1ec037810 0x7fc1ec039cd0 secure :-1 s=READY pgs=3671 cs=0 l=1 rev1=1 crypto rx=0x7fc1f4011310 tx=0x7fc1f4009190 comp rx=0 tx=0).stop 2024-09-16T03:57:09.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.398+0000 7fc206ff1640 10 monclient: shutdown 2024-09-16T03:57:09.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.398+0000 7fc206ff1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:57:09.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.398+0000 7fc206ff1640 1 -- 172.21.15.42:0/3184305338 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc2000a1330 msgr2=0x7fc2001ccc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:57:09.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.398+0000 7fc206ff1640 1 --2- 172.21.15.42:0/3184305338 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc2000a1330 0x7fc2001ccc10 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fc1f0007fd0 tx=0x7fc1f000c890 comp rx=0 tx=0).stop 2024-09-16T03:57:09.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.398+0000 7fc206ff1640 1 -- 172.21.15.42:0/3184305338 shutdown_connections 2024-09-16T03:57:09.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.398+0000 7fc206ff1640 1 --2- 172.21.15.42:0/3184305338 >> v2:172.21.15.154:6812/254845342 conn(0x7fc1ec037810 0x7fc1ec039cd0 unknown :-1 s=CLOSED pgs=3671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:57:09.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.398+0000 7fc206ff1640 1 --2- 172.21.15.42:0/3184305338 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc200135800 0x7fc20011bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:57:09.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.398+0000 7fc206ff1640 1 --2- 172.21.15.42:0/3184305338 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc2000a1e20 0x7fc2001352c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:57:09.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.398+0000 7fc206ff1640 1 --2- 172.21.15.42:0/3184305338 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc2000a1330 0x7fc2001ccc10 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:57:09.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.398+0000 7fc206ff1640 1 -- 172.21.15.42:0/3184305338 >> 172.21.15.42:0/3184305338 conn(0x7fc2000a2420 msgr2=0x7fc2000a4dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:57:09.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.398+0000 7fc206ff1640 1 -- 172.21.15.42:0/3184305338 shutdown_connections 2024-09-16T03:57:09.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.398+0000 7fc206ff1640 1 -- 172.21.15.42:0/3184305338 wait complete. 2024-09-16T03:57:09.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.398+0000 7fc206ff1640 1 librados: shutdown 2024-09-16T03:57:09.424 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T03:57:09.424 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T03:57:09.424 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T03:57:09.425 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T03:57:09.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.546+0000 7f628d77a640 10 monclient: get_monmap_and_config 2024-09-16T03:57:09.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.546+0000 7f628d77a640 10 monclient: build_initial_monmap 2024-09-16T03:57:09.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.546+0000 7f628d77a640 10 monclient: monmap: 2024-09-16T03:57:09.550 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:57:09.550 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:57:09.550 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:57:09.548705+0000 2024-09-16T03:57:09.550 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:57:09.548705+0000 2024-09-16T03:57:09.550 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:57:09.550 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:57:09.551 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:57:09.551 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:57:09.551 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:57:09.551 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:57:09.551 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:57:09.551 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:57:09.556 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.546+0000 7f628d77a640 1 Processor -- start 2024-09-16T03:57:09.556 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.546+0000 7f628d77a640 1 -- start start 2024-09-16T03:57:09.556 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.546+0000 7f628d77a640 10 monclient: init 2024-09-16T03:57:09.556 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.546+0000 7f628d77a640 10 monclient: _reopen_session rank -1 2024-09-16T03:57:09.556 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.546+0000 7f628d77a640 10 monclient: _add_conns ranks=[1,4,0,2,3] 2024-09-16T03:57:09.556 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.546+0000 7f628d77a640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f628809fb00 addr v1:172.21.15.154:6789/0 2024-09-16T03:57:09.556 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.546+0000 7f628d77a640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f628809c160 0x7f6288139120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:57:09.556 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.546+0000 7f628d77a640 10 monclient(hunting): picked mon.noname-c con 0x7f628809c160 addr v2:172.21.15.154:3300/0 2024-09-16T03:57:09.556 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.546+0000 7f628d77a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f628809c5d0 addr v1:172.21.15.42:6789/0 2024-09-16T03:57:09.556 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.546+0000 7f628d77a640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f62880a1e20 con 0x7f628809c5d0 2024-09-16T03:57:09.557 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.546+0000 7f628d77a640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f628809ea30 con 0x7f628809fb00 2024-09-16T03:57:09.557 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.546+0000 7f628d77a640 10 monclient(hunting): start opening mon connection 2024-09-16T03:57:09.557 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.546+0000 7f628d77a640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f628809e250 con 0x7f628809c160 2024-09-16T03:57:09.557 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.546+0000 7f628d77a640 10 monclient(hunting): _renew_subs 2024-09-16T03:57:09.557 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.546+0000 7f628d77a640 10 monclient(hunting): authenticate will time out at 1876.518304s 2024-09-16T03:57:09.557 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.546+0000 7f62877fe640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f628809c5d0 0x7f628813b670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:44306/0 (socket says 172.21.15.42:44306) 2024-09-16T03:57:09.557 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.546+0000 7f62877fe640 1 -- 172.21.15.42:0/4106708780 learned_addr learned my addr 172.21.15.42:0/4106708780 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:57:09.557 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.546+0000 7f627ffff640 1 --2- 172.21.15.42:0/4106708780 >> v2:172.21.15.154:3300/0 conn(0x7f628809c160 0x7f6288139120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:57:09.557 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.546+0000 7f627ffff640 10 monclient(hunting): get_auth_request con 0x7f628809c160 auth_method 0 2024-09-16T03:57:09.557 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.546+0000 7f627ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:57:09.557 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.546+0000 7f627ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:57:09.557 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.546+0000 7f627ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:57:09.557 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.546+0000 7f62867fc640 1 -- 172.21.15.42:0/4106708780 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1972794025 0 0) 0x7f62880a1e20 con 0x7f628809c5d0 2024-09-16T03:57:09.557 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.546+0000 7f62867fc640 10 monclient(hunting): _negotiate 2024-09-16T03:57:09.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.546+0000 7f62867fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:57:09.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.546+0000 7f62867fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:57:09.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.546+0000 7f62867fc640 10 monclient(hunting): my global_id is 824844 2024-09-16T03:57:09.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.546+0000 7f62867fc640 1 -- 172.21.15.42:0/4106708780 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6274004640 con 0x7f628809c5d0 2024-09-16T03:57:09.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.546+0000 7f627ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:57:09.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.546+0000 7f627ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:57:09.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.546+0000 7f627ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:57:09.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.546+0000 7f62867fc640 1 -- 172.21.15.42:0/4106708780 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3911948310 0 0) 0x7f6274004640 con 0x7f628809c5d0 2024-09-16T03:57:09.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.546+0000 7f62867fc640 1 -- 172.21.15.42:0/4106708780 >> v2:172.21.15.154:3300/0 conn(0x7f628809c160 msgr2=0x7f6288139120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:57:09.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.546+0000 7f62867fc640 1 --2- 172.21.15.42:0/4106708780 >> v2:172.21.15.154:3300/0 conn(0x7f628809c160 0x7f6288139120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:57:09.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.546+0000 7f62867fc640 1 -- 172.21.15.42:0/4106708780 >> v1:172.21.15.154:6789/0 conn(0x7f628809fb00 legacy=0x7f628809ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:57:09.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.546+0000 7f62867fc640 10 monclient: _finish_hunting 0 2024-09-16T03:57:09.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.546+0000 7f62867fc640 1 monclient: found mon.noname-a-legacy 2024-09-16T03:57:09.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.546+0000 7f62867fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:57:09.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.546+0000 7f62867fc640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.42:6789/0 2024-09-16T03:57:09.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.546+0000 7f62867fc640 1 -- 172.21.15.42:0/4106708780 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f628809fee0 con 0x7f628809c5d0 2024-09-16T03:57:09.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.546+0000 7f62867fc640 10 monclient: _finish_auth 0 2024-09-16T03:57:09.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.546+0000 7f62867fc640 10 monclient: _check_auth_tickets 2024-09-16T03:57:09.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.546+0000 7f62867fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:57:09.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.546+0000 7f62867fc640 1 -- 172.21.15.42:0/4106708780 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2177413734 0 0) 0x7f6280001a00 con 0x7f628809c5d0 2024-09-16T03:57:09.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.546+0000 7f62867fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:57:09.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.546+0000 7f62867fc640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T03:57:09.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.546+0000 7f62867fc640 10 monclient: dump: 2024-09-16T03:57:09.559 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:57:09.559 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:57:09.559 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:57:09.559 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:57:09.560 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:57:09.560 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:57:09.560 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:57:09.560 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:57:09.560 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:57:09.560 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:57:09.560 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.546+0000 7f62867fc640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] but i'm connected to v1:172.21.15.42:6789/0, reconnecting 2024-09-16T03:57:09.560 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.546+0000 7f62867fc640 10 monclient: _reopen_session rank -1 2024-09-16T03:57:09.560 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.546+0000 7f62867fc640 1 -- 172.21.15.42:0/4106708780 >> v1:172.21.15.42:6789/0 conn(0x7f628809c5d0 legacy=0x7f628813b670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:57:09.560 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.546+0000 7f62867fc640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T03:57:09.560 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.546+0000 7f62867fc640 1 --2- 172.21.15.42:0/4106708780 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6274004ad0 0x7f6274004f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:57:09.560 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.546+0000 7f62867fc640 10 monclient(hunting): picked mon.b con 0x7f6274004ad0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:57:09.560 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.546+0000 7f62867fc640 1 --2- 172.21.15.42:0/4106708780 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f6274005920 0x7f6274005da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:57:09.560 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.546+0000 7f62867fc640 10 monclient(hunting): picked mon.c con 0x7f6274005920 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:57:09.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.546+0000 7f62867fc640 1 --2- 172.21.15.42:0/4106708780 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6274006400 0x7f627400cae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:57:09.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.546+0000 7f62867fc640 10 monclient(hunting): picked mon.a con 0x7f6274006400 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:57:09.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.546+0000 7f62867fc640 10 monclient(hunting): start opening mon connection 2024-09-16T03:57:09.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.546+0000 7f62867fc640 1 -- 172.21.15.42:0/4106708780 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f628809e250 con 0x7f6274006400 2024-09-16T03:57:09.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.546+0000 7f62867fc640 10 monclient(hunting): start opening mon connection 2024-09-16T03:57:09.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.546+0000 7f62867fc640 1 -- 172.21.15.42:0/4106708780 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f627400d2d0 con 0x7f6274004ad0 2024-09-16T03:57:09.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.546+0000 7f62867fc640 10 monclient(hunting): start opening mon connection 2024-09-16T03:57:09.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.546+0000 7f62867fc640 1 -- 172.21.15.42:0/4106708780 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f627400d4d0 con 0x7f6274005920 2024-09-16T03:57:09.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.546+0000 7f62867fc640 10 monclient(hunting): _renew_subs 2024-09-16T03:57:09.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.546+0000 7f62867fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:57:09.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.546+0000 7f627ffff640 1 --2- 172.21.15.42:0/4106708780 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f6274005920 0x7f6274005da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:57:09.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.546+0000 7f62877fe640 1 --2- 172.21.15.42:0/4106708780 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6274006400 0x7f627400cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:57:09.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.546+0000 7f627ffff640 10 monclient(hunting): get_auth_request con 0x7f6274005920 auth_method 0 2024-09-16T03:57:09.562 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.546+0000 7f627ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:57:09.562 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.546+0000 7f627ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:57:09.562 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.546+0000 7f627ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:57:09.562 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.546+0000 7f62877fe640 10 monclient(hunting): get_auth_request con 0x7f6274006400 auth_method 0 2024-09-16T03:57:09.562 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.546+0000 7f62877fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:57:09.562 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.546+0000 7f62877fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:57:09.562 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.546+0000 7f62877fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:57:09.562 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.546+0000 7f627ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:57:09.562 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.546+0000 7f627ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:57:09.562 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.546+0000 7f627ffff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T03:57:09.562 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.546+0000 7f6286ffd640 1 --2- 172.21.15.42:0/4106708780 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6274004ad0 0x7f6274004f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:57:09.562 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.546+0000 7f62877fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:57:09.562 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.546+0000 7f62877fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:57:09.562 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.546+0000 7f62877fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T03:57:09.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.546+0000 7f6286ffd640 10 monclient(hunting): get_auth_request con 0x7f6274004ad0 auth_method 0 2024-09-16T03:57:09.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.546+0000 7f6286ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:57:09.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.546+0000 7f6286ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:57:09.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.546+0000 7f6286ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:57:09.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.546+0000 7f627ffff640 10 monclient(hunting): handle_auth_done global_id 824844 payload 293 2024-09-16T03:57:09.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.546+0000 7f627ffff640 1 -- 172.21.15.42:0/4106708780 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6274004ad0 msgr2=0x7f6274004f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:57:09.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.546+0000 7f627ffff640 1 --2- 172.21.15.42:0/4106708780 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6274004ad0 0x7f6274004f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:57:09.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.546+0000 7f627ffff640 1 -- 172.21.15.42:0/4106708780 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6274006400 msgr2=0x7f627400cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:57:09.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.546+0000 7f627ffff640 1 --2- 172.21.15.42:0/4106708780 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6274006400 0x7f627400cae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:57:09.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.546+0000 7f627ffff640 10 monclient: _finish_hunting 0 2024-09-16T03:57:09.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.546+0000 7f627ffff640 1 monclient: found mon.c 2024-09-16T03:57:09.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.546+0000 7f627ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:57:09.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.546+0000 7f627ffff640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:57:09.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.546+0000 7f627ffff640 1 -- 172.21.15.42:0/4106708780 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6274004640 con 0x7f6274005920 2024-09-16T03:57:09.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.546+0000 7f627ffff640 10 monclient: _finish_auth 0 2024-09-16T03:57:09.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.546+0000 7f627ffff640 10 monclient: _check_auth_tickets 2024-09-16T03:57:09.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.546+0000 7f627ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:57:09.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.546+0000 7f628d77a640 5 monclient: authenticate success, global_id 824844 2024-09-16T03:57:09.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.546+0000 7f628d77a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:57:09.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.546+0000 7f62877fe640 1 --2- 172.21.15.42:0/4106708780 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6274006400 0x7f627400cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:57:09.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.546+0000 7f62877fe640 1 -- 172.21.15.42:0/4106708780 reap_dead start 2024-09-16T03:57:09.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.546+0000 7f6286ffd640 1 --2- 172.21.15.42:0/4106708780 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6274004ad0 0x7f6274004f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T03:57:09.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.550+0000 7f627ffff640 1 --2- 172.21.15.42:0/4106708780 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f6274005920 0x7f6274005da0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f6270009ae0 tx=0x7f627001ca50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d1767e082ec1eda8 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:57:09.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.550+0000 7f62867fc640 1 -- 172.21.15.42:0/4106708780 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62700b2070 con 0x7f6274005920 2024-09-16T03:57:09.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.550+0000 7f62867fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:57:09.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.550+0000 7f62867fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:57:09.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.550+0000 7f62867fc640 10 monclient: dump: 2024-09-16T03:57:09.564 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:57:09.565 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:57:09.565 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:57:09.565 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:57:09.565 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:57:09.565 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:57:09.565 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:57:09.565 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:57:09.565 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:57:09.565 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:57:09.565 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.550+0000 7f628d77a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:57:09.565 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.550+0000 7f62867fc640 1 -- 172.21.15.42:0/4106708780 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f62700a55a0 con 0x7f6274005920 2024-09-16T03:57:09.565 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.550+0000 7f62867fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:57:09.565 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.550+0000 7f628d77a640 10 monclient: get_monmap_and_config success 2024-09-16T03:57:09.565 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.550+0000 7f628d77a640 4 set_mon_vals no callback set 2024-09-16T03:57:09.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.550+0000 7f62867fc640 1 -- 172.21.15.42:0/4106708780 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62700096b0 con 0x7f6274005920 2024-09-16T03:57:09.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.550+0000 7f62867fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:57:09.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.550+0000 7f62867fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:57:09.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.550+0000 7f62867fc640 10 monclient: dump: 2024-09-16T03:57:09.566 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:57:09.566 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:57:09.566 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:57:09.566 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:57:09.566 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:57:09.566 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:57:09.566 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:57:09.566 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:57:09.566 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:57:09.566 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:57:09.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.550+0000 7f628d77a640 10 monclient: shutdown 2024-09-16T03:57:09.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.550+0000 7f628d77a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:57:09.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.550+0000 7f628d77a640 1 -- 172.21.15.42:0/4106708780 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f6274005920 msgr2=0x7f6274005da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:57:09.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.550+0000 7f628d77a640 1 --2- 172.21.15.42:0/4106708780 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f6274005920 0x7f6274005da0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f6270009ae0 tx=0x7f627001ca50 comp rx=0 tx=0).stop 2024-09-16T03:57:09.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.550+0000 7f628d77a640 1 -- 172.21.15.42:0/4106708780 shutdown_connections 2024-09-16T03:57:09.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.550+0000 7f628d77a640 1 --2- 172.21.15.42:0/4106708780 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f6274005920 0x7f6274005da0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:57:09.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.550+0000 7f628d77a640 1 -- 172.21.15.42:0/4106708780 >> 172.21.15.42:0/4106708780 conn(0x7f62880a2420 msgr2=0x7f62880a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:57:09.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.550+0000 7f628d77a640 1 -- 172.21.15.42:0/4106708780 shutdown_connections 2024-09-16T03:57:09.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.550+0000 7f628d77a640 1 -- 172.21.15.42:0/4106708780 wait complete. 2024-09-16T03:57:09.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.550+0000 7f628d77a640 10 monclient: build_initial_monmap 2024-09-16T03:57:09.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.550+0000 7f628d77a640 10 monclient: monmap: 2024-09-16T03:57:09.574 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:57:09.574 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:57:09.574 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:57:09.574 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:57:09.574 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:57:09.574 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:57:09.574 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:57:09.574 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:57:09.574 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:57:09.574 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:57:09.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.550+0000 7f628d77a640 1 librados: starting msgr at 2024-09-16T03:57:09.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.550+0000 7f628d77a640 1 librados: starting objecter 2024-09-16T03:57:09.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.550+0000 7f628d77a640 1 Processor -- start 2024-09-16T03:57:09.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.550+0000 7f628d77a640 1 -- start start 2024-09-16T03:57:09.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.550+0000 7f628d77a640 1 librados: setting wanted keys 2024-09-16T03:57:09.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.550+0000 7f628d77a640 1 librados: calling monclient init 2024-09-16T03:57:09.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.550+0000 7f628d77a640 10 monclient: init 2024-09-16T03:57:09.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.554+0000 7f628d77a640 10 monclient: _reopen_session rank -1 2024-09-16T03:57:09.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.554+0000 7f628d77a640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:57:09.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.554+0000 7f628d77a640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6274005920 0x7f62881cb5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:57:09.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.554+0000 7f628d77a640 10 monclient(hunting): picked mon.noname-b con 0x7f6274005920 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:57:09.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.554+0000 7f628d77a640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f62881cbb00 0x7f6288121b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:57:09.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.554+0000 7f628d77a640 10 monclient(hunting): picked mon.noname-a con 0x7f62881cbb00 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:57:09.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.554+0000 7f628d77a640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f6288122070 0x7f62881224f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:57:09.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.554+0000 7f628d77a640 10 monclient(hunting): picked mon.noname-c con 0x7f6288122070 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:57:09.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.554+0000 7f628d77a640 10 monclient(hunting): start opening mon connection 2024-09-16T03:57:09.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.554+0000 7f628d77a640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f628809bfe0 con 0x7f62881cbb00 2024-09-16T03:57:09.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.554+0000 7f628d77a640 10 monclient(hunting): start opening mon connection 2024-09-16T03:57:09.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.554+0000 7f628d77a640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f628809be60 con 0x7f6274005920 2024-09-16T03:57:09.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.554+0000 7f628d77a640 10 monclient(hunting): start opening mon connection 2024-09-16T03:57:09.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.554+0000 7f628d77a640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f62880a3170 con 0x7f6288122070 2024-09-16T03:57:09.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.554+0000 7f628d77a640 10 monclient(hunting): _renew_subs 2024-09-16T03:57:09.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.554+0000 7f628d77a640 10 monclient(hunting): authenticate will time out at 1876.524314s 2024-09-16T03:57:09.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.554+0000 7f62877fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f6288122070 0x7f62881224f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:57:09.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.554+0000 7f62877fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f6288122070 0x7f62881224f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:36102/0 (socket says 172.21.15.42:36102) 2024-09-16T03:57:09.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.554+0000 7f62877fe640 1 -- 172.21.15.42:0/1253775204 learned_addr learned my addr 172.21.15.42:0/1253775204 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:57:09.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.554+0000 7f62877fe640 10 monclient(hunting): get_auth_request con 0x7f6288122070 auth_method 0 2024-09-16T03:57:09.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.554+0000 7f62877fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:57:09.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.554+0000 7f62877fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:57:09.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.554+0000 7f62877fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:57:09.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.554+0000 7f627ffff640 1 --2- 172.21.15.42:0/1253775204 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f62881cbb00 0x7f6288121b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:57:09.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.554+0000 7f627ffff640 10 monclient(hunting): get_auth_request con 0x7f62881cbb00 auth_method 0 2024-09-16T03:57:09.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.554+0000 7f627ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:57:09.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.554+0000 7f627ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:57:09.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.554+0000 7f627ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:57:09.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.554+0000 7f62877fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:57:09.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.554+0000 7f62877fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:57:09.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.554+0000 7f62877fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:57:09.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.554+0000 7f627ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:57:09.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.554+0000 7f627ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:57:09.578 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.554+0000 7f627ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:57:09.578 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.554+0000 7f6286ffd640 1 --2- 172.21.15.42:0/1253775204 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6274005920 0x7f62881cb5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:57:09.578 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.554+0000 7f62877fe640 10 monclient(hunting): handle_auth_done global_id 854387 payload 824 2024-09-16T03:57:09.578 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.554+0000 7f62877fe640 1 -- 172.21.15.42:0/1253775204 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6274005920 msgr2=0x7f62881cb5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:57:09.578 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.554+0000 7f62877fe640 1 --2- 172.21.15.42:0/1253775204 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6274005920 0x7f62881cb5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:57:09.578 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.554+0000 7f62877fe640 1 -- 172.21.15.42:0/1253775204 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f62881cbb00 msgr2=0x7f6288121b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:57:09.578 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.554+0000 7f62877fe640 1 --2- 172.21.15.42:0/1253775204 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f62881cbb00 0x7f6288121b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:57:09.578 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.554+0000 7f62877fe640 10 monclient: _finish_hunting 0 2024-09-16T03:57:09.578 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.554+0000 7f62877fe640 1 monclient: found mon.noname-c 2024-09-16T03:57:09.578 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.554+0000 7f62877fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:57:09.578 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.554+0000 7f62877fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:57:09.578 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.554+0000 7f62877fe640 1 -- 172.21.15.42:0/1253775204 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6288071eb0 con 0x7f6288122070 2024-09-16T03:57:09.578 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.554+0000 7f6286ffd640 10 monclient: get_auth_request con 0x7f6274005920 auth_method 0 2024-09-16T03:57:09.578 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.554+0000 7f6286ffd640 1 --2- 172.21.15.42:0/1253775204 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6274005920 0x7f62881cb5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T03:57:09.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.554+0000 7f62877fe640 1 --2- 172.21.15.42:0/1253775204 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f6288122070 0x7f62881224f0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f62880a68b0 tx=0x7f6280007430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:57:09.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.554+0000 7f627f7fe640 10 client.?.objecter ms_handle_connect 0x7f6288122070 2024-09-16T03:57:09.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.554+0000 7f627f7fe640 10 client.?.objecter resend_mon_ops 2024-09-16T03:57:09.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.554+0000 7f627f7fe640 1 -- 172.21.15.42:0/1253775204 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f628000a480 con 0x7f6288122070 2024-09-16T03:57:09.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.554+0000 7f627f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:57:09.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.554+0000 7f627f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:57:09.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.554+0000 7f627f7fe640 10 monclient: dump: 2024-09-16T03:57:09.579 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:57:09.579 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:57:09.579 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:57:09.579 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:57:09.579 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:57:09.579 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:57:09.579 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:57:09.580 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:57:09.580 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:57:09.580 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:57:09.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.554+0000 7f627f7fe640 10 monclient: _finish_auth 0 2024-09-16T03:57:09.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.554+0000 7f627f7fe640 10 monclient: _check_auth_tickets 2024-09-16T03:57:09.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.554+0000 7f627f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:57:09.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.554+0000 7f627f7fe640 1 -- 172.21.15.42:0/1253775204 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6280018070 con 0x7f6288122070 2024-09-16T03:57:09.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.554+0000 7f627f7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:57:09.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.554+0000 7f627f7fe640 1 -- 172.21.15.42:0/1253775204 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62800135f0 con 0x7f6288122070 2024-09-16T03:57:09.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.554+0000 7f627f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:57:09.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.554+0000 7f627f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:57:09.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.554+0000 7f627f7fe640 10 monclient: dump: 2024-09-16T03:57:09.580 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:57:09.581 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:57:09.581 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:57:09.581 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:57:09.581 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:57:09.581 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:57:09.581 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:57:09.581 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:57:09.581 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:57:09.581 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:57:09.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.554+0000 7f628d77a640 5 monclient: authenticate success, global_id 854387 2024-09-16T03:57:09.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.554+0000 7f628d77a640 10 monclient: _renew_subs 2024-09-16T03:57:09.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.554+0000 7f628d77a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:57:09.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.554+0000 7f62857fa640 4 set_mon_vals no callback set 2024-09-16T03:57:09.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.554+0000 7f628d77a640 1 -- 172.21.15.42:0/1253775204 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6288125500 con 0x7f6288122070 2024-09-16T03:57:09.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.554+0000 7f628d77a640 10 client.854387.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:57:09.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.554+0000 7f628d77a640 10 monclient: _renew_subs 2024-09-16T03:57:09.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.554+0000 7f628d77a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:57:09.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.554+0000 7f628d77a640 1 -- 172.21.15.42:0/1253775204 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f62881257e0 con 0x7f6288122070 2024-09-16T03:57:09.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.554+0000 7f628d77a640 1 librados: init done 2024-09-16T03:57:09.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.554+0000 7f628d77a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:57:09.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.554+0000 7f628d77a640 1 --2- 172.21.15.42:0/1253775204 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f628802d7d0 0x7f6288122a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:57:09.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.554+0000 7f628d77a640 10 monclient(hunting): start opening mon connection 2024-09-16T03:57:09.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.554+0000 7f628d77a640 1 -- 172.21.15.42:0/1253775204 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f628802dbb0 con 0x7f628802d7d0 2024-09-16T03:57:09.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.554+0000 7f6286ffd640 1 --2- 172.21.15.42:0/1253775204 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f628802d7d0 0x7f6288122a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:57:09.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.554+0000 7f6286ffd640 10 monclient: get_auth_request con 0x7f628802d7d0 auth_method 0 2024-09-16T03:57:09.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.554+0000 7f6286ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:57:09.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.554+0000 7f6286ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:57:09.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.554+0000 7f6286ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:57:09.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.554+0000 7f627f7fe640 1 -- 172.21.15.42:0/1253775204 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f6280013790 con 0x7f6288122070 2024-09-16T03:57:09.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.554+0000 7f627f7fe640 1 --2- 172.21.15.42:0/1253775204 >> v2:172.21.15.154:6812/254845342 conn(0x7f6268037810 0x7f6268039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:57:09.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.554+0000 7f627f7fe640 1 -- 172.21.15.42:0/1253775204 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(459..459 src has 330..459) ==== 5328+0+0 (secure 0 0 0) 0x7f6280049290 con 0x7f6288122070 2024-09-16T03:57:09.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.554+0000 7f627f7fe640 10 client.854387.objecter ms_dispatch 0x7f6288000ba0 osd_map(459..459 src has 330..459) 2024-09-16T03:57:09.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.554+0000 7f627f7fe640 3 client.854387.objecter handle_osd_map got epochs [459,459] > 0 2024-09-16T03:57:09.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.554+0000 7f627f7fe640 3 client.854387.objecter handle_osd_map decoding full epoch 459 2024-09-16T03:57:09.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.554+0000 7f627f7fe640 20 client.854387.objecter dump_active .. 0 homeless 2024-09-16T03:57:09.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.554+0000 7f6286ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:57:09.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.554+0000 7f6286ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:57:09.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.554+0000 7f6286ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:57:09.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.554+0000 7f6286ffd640 10 monclient(hunting): handle_auth_done global_id 824850 payload 824 2024-09-16T03:57:09.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.554+0000 7f6286ffd640 1 -- 172.21.15.42:0/1253775204 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f628806e440 con 0x7f628802d7d0 2024-09-16T03:57:09.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.554+0000 7f627ffff640 1 --2- 172.21.15.42:0/1253775204 >> v2:172.21.15.154:6812/254845342 conn(0x7f6268037810 0x7f6268039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:57:09.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.554+0000 7f627ffff640 10 monclient: get_auth_request con 0x7f6268037810 auth_method 0 2024-09-16T03:57:09.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.554+0000 7f627f7fe640 10 client.854387.objecter ms_handle_connect 0x7f628802d7d0 2024-09-16T03:57:09.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.554+0000 7f627f7fe640 10 client.854387.objecter resend_mon_ops 2024-09-16T03:57:09.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.554+0000 7f6286ffd640 1 --2- 172.21.15.42:0/1253775204 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f628802d7d0 0x7f6288122a30 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f628806a7c0 tx=0x7f627800f8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:57:09.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.554+0000 7f627f7fe640 1 -- 172.21.15.42:0/1253775204 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62780052e0 con 0x7f628802d7d0 2024-09-16T03:57:09.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.554+0000 7f627f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:57:09.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.554+0000 7f627f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:57:09.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.554+0000 7f627f7fe640 10 monclient: dump: 2024-09-16T03:57:09.584 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:57:09.584 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:57:09.584 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:57:09.584 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:57:09.584 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:57:09.585 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:57:09.585 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:57:09.585 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:57:09.585 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:57:09.585 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:57:09.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.554+0000 7f627f7fe640 1 -- 172.21.15.42:0/1253775204 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f6278018070 con 0x7f628802d7d0 2024-09-16T03:57:09.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.554+0000 7f627f7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:57:09.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.554+0000 7f627f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:57:09.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.554+0000 7f627ffff640 1 --2- 172.21.15.42:0/1253775204 >> v2:172.21.15.154:6812/254845342 conn(0x7f6268037810 0x7f6268039cd0 secure :-1 s=READY pgs=3672 cs=0 l=1 rev1=1 crypto rx=0x7f62700af000 tx=0x7f62700046d0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:57:09.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.554+0000 7f627f7fe640 1 -- 172.21.15.42:0/1253775204 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f628802d7d0 msgr2=0x7f6288122a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:57:09.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.554+0000 7f627f7fe640 1 --2- 172.21.15.42:0/1253775204 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f628802d7d0 0x7f6288122a30 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f628806a7c0 tx=0x7f627800f8e0 comp rx=0 tx=0).stop 2024-09-16T03:57:09.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.554+0000 7f627f7fe640 1 -- 172.21.15.42:0/1253775204 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f628802d7d0 msgr2=0x7f6288122a30 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:57:09.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.554+0000 7f627f7fe640 1 --2- 172.21.15.42:0/1253775204 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f628802d7d0 0x7f6288122a30 secure :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f628806a7c0 tx=0x7f627800f8e0 comp rx=0 tx=0).stop 2024-09-16T03:57:09.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.554+0000 7f627f7fe640 10 client.854387.objecter ms_handle_connect 0x7f6268037810 2024-09-16T03:57:09.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.562+0000 7f628d77a640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:57:09.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.562+0000 7f628d77a640 1 --2- 172.21.15.42:0/1253775204 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6288062ea0 0x7f62880632b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:57:09.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.562+0000 7f628d77a640 10 monclient(hunting): start opening mon connection 2024-09-16T03:57:09.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.562+0000 7f628d77a640 1 -- 172.21.15.42:0/1253775204 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f628809d380 con 0x7f6288062ea0 2024-09-16T03:57:09.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.562+0000 7f6286ffd640 1 --2- 172.21.15.42:0/1253775204 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6288062ea0 0x7f62880632b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:57:09.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.562+0000 7f6286ffd640 10 monclient: get_auth_request con 0x7f6288062ea0 auth_method 0 2024-09-16T03:57:09.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.562+0000 7f6286ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:57:09.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.562+0000 7f6286ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:57:09.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.562+0000 7f6286ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:57:09.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.562+0000 7f6286ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:57:09.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.562+0000 7f6286ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:57:09.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.562+0000 7f6286ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:57:09.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.562+0000 7f6286ffd640 10 monclient(hunting): handle_auth_done global_id 824853 payload 824 2024-09-16T03:57:09.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.562+0000 7f6286ffd640 1 -- 172.21.15.42:0/1253775204 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f628811e690 con 0x7f6288062ea0 2024-09-16T03:57:09.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.562+0000 7f6286ffd640 1 --2- 172.21.15.42:0/1253775204 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6288062ea0 0x7f62880632b0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f6278017000 tx=0x7f6278001130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:57:09.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.562+0000 7f627f7fe640 10 client.854387.objecter ms_handle_connect 0x7f6288062ea0 2024-09-16T03:57:09.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.562+0000 7f627f7fe640 10 client.854387.objecter resend_mon_ops 2024-09-16T03:57:09.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.562+0000 7f627f7fe640 1 -- 172.21.15.42:0/1253775204 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6278003c90 con 0x7f6288062ea0 2024-09-16T03:57:09.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.562+0000 7f627f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:57:09.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.562+0000 7f627f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:57:09.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.562+0000 7f627f7fe640 10 monclient: dump: 2024-09-16T03:57:09.587 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:57:09.587 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:57:09.587 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:57:09.588 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:57:09.588 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:57:09.588 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:57:09.588 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:57:09.588 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:57:09.588 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:57:09.588 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:57:09.588 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.562+0000 7f627f7fe640 1 -- 172.21.15.42:0/1253775204 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4882 (secure 0 0 0) 0x7f6278003c90 con 0x7f6288062ea0 2024-09-16T03:57:09.589 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.562+0000 7f627f7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:57:09.589 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.562+0000 7f627f7fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:57:09.589 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.562+0000 7f627f7fe640 1 -- 172.21.15.42:0/1253775204 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6288062ea0 msgr2=0x7f62880632b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:57:09.589 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.562+0000 7f627f7fe640 1 --2- 172.21.15.42:0/1253775204 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6288062ea0 0x7f62880632b0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f6278017000 tx=0x7f6278001130 comp rx=0 tx=0).stop 2024-09-16T03:57:09.589 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.562+0000 7f627f7fe640 1 -- 172.21.15.42:0/1253775204 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6288062ea0 msgr2=0x7f62880632b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:57:09.589 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.562+0000 7f627f7fe640 1 --2- 172.21.15.42:0/1253775204 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6288062ea0 0x7f62880632b0 secure :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f6278017000 tx=0x7f6278001130 comp rx=0 tx=0).stop 2024-09-16T03:57:09.589 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:57:09.589 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:57:09.589 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:57:09.590 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "leader", 2024-09-16T03:57:09.590 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 140, 2024-09-16T03:57:09.590 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 319417, 2024-09-16T03:57:09.590 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:57:09.590 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:57:09.590 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T03:57:09.590 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T03:57:09.590 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:57:09.590 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 2, 2024-09-16T03:57:09.590 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:57:09.590 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:57:09.590 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:57:09.590 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:57:09.591 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:57:09.591 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:57:09.591 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:57:09.591 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:57:09.591 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:57:09.591 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:57:09.591 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:57:09.591 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:57:09.591 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:57:09.591 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:57:09.591 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:57:09.591 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:57:09.591 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:57:09.591 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:57:09.591 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:57:09.592 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:57:09.592 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:57:09.592 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:57:09.592 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:57:09.592 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:57:09.592 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:57:09.592 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:57:09.592 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:57:09.592 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:57:09.592 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:57:09.592 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:57:09.592 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:57:09.592 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:57:09.593 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:57:09.593 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:57:09.593 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:57:09.593 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:57:09.593 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:57:09.593 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:57:09.593 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:57:09.593 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:57:09.593 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:57:09.593 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:57:09.593 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:57:09.593 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:57:09.594 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:57:09.594 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:57:09.594 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:57:09.594 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:57:09.594 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:57:09.594 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:57:09.594 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:57:09.594 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:57:09.594 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:57:09.594 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:57:09.594 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:57:09.594 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:57:09.594 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:57:09.594 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:57:09.595 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:57:09.595 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:57:09.595 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:57:09.595 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:57:09.595 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:57:09.595 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:57:09.595 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:57:09.595 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:57:09.595 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:57:09.595 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:57:09.595 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:57:09.595 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:57:09.595 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:57:09.595 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:57:09.596 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:57:09.596 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:57:09.596 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:57:09.596 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:57:09.596 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:57:09.596 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:57:09.596 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:57:09.596 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:57:09.596 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:57:09.596 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:57:09.596 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:57:09.596 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:57:09.596 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:57:09.597 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:57:09.597 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:57:09.597 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:57:09.597 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:57:09.597 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:57:09.597 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:57:09.597 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:57:09.597 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:57:09.597 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:57:09.597 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:57:09.597 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:57:09.597 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:57:09.597 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:57:09.597 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:57:09.598 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:57:09.598 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:57:09.598 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:57:09.598 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:57:09.598 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:57:09.598 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:57:09.598 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:57:09.598 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:57:09.598 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:57:09.598 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:57:09.598 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:57:09.598 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:57:09.598 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:57:09.599 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:57:09.599 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:57:09.599 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:57:09.599 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:57:09.599 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:57:09.599 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:57:09.599 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:57:09.599 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:57:09.599 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:57:09.599 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:57:09.599 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:57:09.599 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:57:09.599 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:57:09.599 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:57:09.600 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:57:09.600 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:57:09.600 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:57:09.600 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:57:09.600 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:57:09.600 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:57:09.600 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:57:09.600 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:57:09.600 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:57:09.600 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:57:09.600 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:57:09.600 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:57:09.600 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:57:09.601 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:57:09.601 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:57:09.601 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T03:57:09.601 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:57:09.601 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:57:09.601 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:57:09.601 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 6 2024-09-16T03:57:09.601 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:57:09.601 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:57:09.601 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:57:09.601 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:57:09.601 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:57:09.601 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:57:09.601 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 7 2024-09-16T03:57:09.602 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:57:09.602 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:57:09.602 INFO:teuthology.orchestra.run.smithi042.stdout: "mgr": [ 2024-09-16T03:57:09.602 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:57:09.602 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:57:09.602 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:57:09.602 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:57:09.602 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:57:09.602 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:57:09.602 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:57:09.602 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:57:09.602 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:57:09.604 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.562+0000 7f628d77a640 10 librados: watch_flush enter 2024-09-16T03:57:09.604 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.562+0000 7f628d77a640 10 librados: watch_flush exit 2024-09-16T03:57:09.604 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.562+0000 7f628d77a640 20 client.854387.objecter shutdown clearing up homeless session... 2024-09-16T03:57:09.604 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.562+0000 7f628d77a640 10 client.854387.objecter successfully canceled tick 2024-09-16T03:57:09.604 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.562+0000 7f628d77a640 1 -- 172.21.15.42:0/1253775204 >> v2:172.21.15.154:6812/254845342 conn(0x7f6268037810 msgr2=0x7f6268039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:57:09.604 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.562+0000 7f628d77a640 1 --2- 172.21.15.42:0/1253775204 >> v2:172.21.15.154:6812/254845342 conn(0x7f6268037810 0x7f6268039cd0 secure :-1 s=READY pgs=3672 cs=0 l=1 rev1=1 crypto rx=0x7f62700af000 tx=0x7f62700046d0 comp rx=0 tx=0).stop 2024-09-16T03:57:09.605 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.562+0000 7f628d77a640 10 monclient: shutdown 2024-09-16T03:57:09.605 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.562+0000 7f628d77a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:57:09.605 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.562+0000 7f628d77a640 1 -- 172.21.15.42:0/1253775204 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f6288122070 msgr2=0x7f62881224f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:57:09.605 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.562+0000 7f628d77a640 1 --2- 172.21.15.42:0/1253775204 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f6288122070 0x7f62881224f0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f62880a68b0 tx=0x7f6280007430 comp rx=0 tx=0).stop 2024-09-16T03:57:09.605 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.562+0000 7f62877fe640 1 -- 172.21.15.42:0/1253775204 reap_dead start 2024-09-16T03:57:09.605 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.562+0000 7f628d77a640 1 -- 172.21.15.42:0/1253775204 shutdown_connections 2024-09-16T03:57:09.605 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.562+0000 7f628d77a640 1 --2- 172.21.15.42:0/1253775204 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f6288122070 0x7f62881224f0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:57:09.605 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.562+0000 7f628d77a640 1 -- 172.21.15.42:0/1253775204 >> 172.21.15.42:0/1253775204 conn(0x7f62880a2420 msgr2=0x7f62880a4ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:57:09.605 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.562+0000 7f628d77a640 1 -- 172.21.15.42:0/1253775204 shutdown_connections 2024-09-16T03:57:09.605 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.562+0000 7f628d77a640 1 -- 172.21.15.42:0/1253775204 wait complete. 2024-09-16T03:57:09.605 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.562+0000 7f628d77a640 1 librados: shutdown 2024-09-16T03:57:09.606 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:57:09.606 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T03:57:09.606 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T03:57:09.724 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.718+0000 7ff4d6d53640 10 monclient: get_monmap_and_config 2024-09-16T03:57:09.725 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.718+0000 7ff4d6d53640 10 monclient: build_initial_monmap 2024-09-16T03:57:09.725 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.718+0000 7ff4d6d53640 10 monclient: monmap: 2024-09-16T03:57:09.725 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:57:09.725 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:57:09.725 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:57:09.723271+0000 2024-09-16T03:57:09.725 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:57:09.723271+0000 2024-09-16T03:57:09.725 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:57:09.726 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:57:09.726 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:57:09.726 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:57:09.726 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:57:09.726 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:57:09.726 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:57:09.726 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:57:09.728 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.718+0000 7ff4d6d53640 1 Processor -- start 2024-09-16T03:57:09.728 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.722+0000 7ff4d6d53640 1 -- start start 2024-09-16T03:57:09.729 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.722+0000 7ff4d6d53640 10 monclient: init 2024-09-16T03:57:09.729 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.722+0000 7ff4d6d53640 10 monclient: _reopen_session rank -1 2024-09-16T03:57:09.729 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.722+0000 7ff4d6d53640 10 monclient: _add_conns ranks=[4,0,2,1,3] 2024-09-16T03:57:09.729 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.722+0000 7ff4d6d53640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7ff4d00a1330 0x7ff4d009f180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:57:09.729 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.722+0000 7ff4d6d53640 10 monclient(hunting): picked mon.noname-c con 0x7ff4d00a1330 addr v2:172.21.15.154:3300/0 2024-09-16T03:57:09.729 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.722+0000 7ff4d6d53640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff4d00a1710 addr v1:172.21.15.42:6789/0 2024-09-16T03:57:09.729 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.722+0000 7ff4d6d53640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff4d01393a0 0x7ff4d013b790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:57:09.729 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.722+0000 7ff4d6d53640 10 monclient(hunting): picked mon.noname-b con 0x7ff4d01393a0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:57:09.729 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.722+0000 7ff4d6d53640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff4d009e4c0 con 0x7ff4d00a1710 2024-09-16T03:57:09.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.722+0000 7ff4d6d53640 10 monclient(hunting): start opening mon connection 2024-09-16T03:57:09.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.722+0000 7ff4d6d53640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7ff4d009edf0 con 0x7ff4d00a1330 2024-09-16T03:57:09.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.722+0000 7ff4d6d53640 10 monclient(hunting): start opening mon connection 2024-09-16T03:57:09.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.722+0000 7ff4d6d53640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7ff4d009cd60 con 0x7ff4d01393a0 2024-09-16T03:57:09.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.722+0000 7ff4d6d53640 10 monclient(hunting): _renew_subs 2024-09-16T03:57:09.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.722+0000 7ff4d6d53640 10 monclient(hunting): authenticate will time out at 1876.692689s 2024-09-16T03:57:09.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.722+0000 7ff4d52f0640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff4d01393a0 0x7ff4d013b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:57:09.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.722+0000 7ff4cffff640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7ff4d00a1710 0x7ff4d00a1b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:44318/0 (socket says 172.21.15.42:44318) 2024-09-16T03:57:09.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.722+0000 7ff4cffff640 1 -- 172.21.15.42:0/837025857 learned_addr learned my addr 172.21.15.42:0/837025857 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:57:09.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.722+0000 7ff4d52f0640 10 monclient(hunting): get_auth_request con 0x7ff4d01393a0 auth_method 0 2024-09-16T03:57:09.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.722+0000 7ff4d52f0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:57:09.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.722+0000 7ff4d52f0640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:57:09.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.722+0000 7ff4d52f0640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:57:09.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.722+0000 7ff4d52f0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:57:09.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.722+0000 7ff4d52f0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:57:09.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.722+0000 7ff4d52f0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:57:09.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.722+0000 7ff4d52f0640 10 monclient(hunting): handle_auth_done global_id 854390 payload 274 2024-09-16T03:57:09.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.722+0000 7ff4d52f0640 1 -- 172.21.15.42:0/837025857 >> v2:172.21.15.154:3300/0 conn(0x7ff4d00a1330 msgr2=0x7ff4d009f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:57:09.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.722+0000 7ff4d4aef640 1 --2- 172.21.15.42:0/837025857 >> v2:172.21.15.154:3300/0 conn(0x7ff4d00a1330 0x7ff4d009f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:57:09.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.722+0000 7ff4d52f0640 1 --2- 172.21.15.42:0/837025857 >> v2:172.21.15.154:3300/0 conn(0x7ff4d00a1330 0x7ff4d009f180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:57:09.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.722+0000 7ff4cf7fe640 1 -- 172.21.15.42:0/837025857 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 418859357 0 0) 0x7ff4d009e4c0 con 0x7ff4d00a1710 2024-09-16T03:57:09.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.722+0000 7ff4d52f0640 1 -- 172.21.15.42:0/837025857 >> v1:172.21.15.42:6789/0 conn(0x7ff4d00a1710 legacy=0x7ff4d00a1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:57:09.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.722+0000 7ff4d52f0640 10 monclient: _finish_hunting 0 2024-09-16T03:57:09.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.722+0000 7ff4d52f0640 1 monclient: found mon.noname-b 2024-09-16T03:57:09.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.722+0000 7ff4d52f0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:57:09.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.722+0000 7ff4d52f0640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T03:57:09.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.722+0000 7ff4d52f0640 1 -- 172.21.15.42:0/837025857 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4d009f6c0 con 0x7ff4d01393a0 2024-09-16T03:57:09.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.722+0000 7ff4d4aef640 10 monclient: get_auth_request con 0x7ff4d00a1330 auth_method 0 2024-09-16T03:57:09.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.722+0000 7ff4d4aef640 1 --2- 172.21.15.42:0/837025857 >> v2:172.21.15.154:3300/0 conn(0x7ff4d00a1330 0x7ff4d009f180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T03:57:09.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.722+0000 7ff4cf7fe640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T03:57:09.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.726+0000 7ff4d52f0640 1 --2- 172.21.15.42:0/837025857 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff4d01393a0 0x7ff4d013b790 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7ff4b80097c0 tx=0x7ff4b801c360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=142292b1030f5078 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:57:09.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.726+0000 7ff4cf7fe640 1 -- 172.21.15.42:0/837025857 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4b80b0070 con 0x7ff4d01393a0 2024-09-16T03:57:09.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.726+0000 7ff4cf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:57:09.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.726+0000 7ff4cf7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:57:09.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.726+0000 7ff4cf7fe640 10 monclient: dump: 2024-09-16T03:57:09.741 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:57:09.741 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:57:09.741 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:57:09.741 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:57:09.741 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:57:09.741 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:57:09.741 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:57:09.741 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:57:09.741 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:57:09.741 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:57:09.742 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.726+0000 7ff4cf7fe640 10 monclient: _finish_auth 0 2024-09-16T03:57:09.742 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.726+0000 7ff4cf7fe640 10 monclient: _check_auth_tickets 2024-09-16T03:57:09.742 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.726+0000 7ff4cf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:57:09.742 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.726+0000 7ff4d6d53640 5 monclient: authenticate success, global_id 854390 2024-09-16T03:57:09.742 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.726+0000 7ff4cf7fe640 1 -- 172.21.15.42:0/837025857 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4b80ab600 con 0x7ff4d01393a0 2024-09-16T03:57:09.742 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.726+0000 7ff4d6d53640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:57:09.742 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.726+0000 7ff4cf7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:57:09.742 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.726+0000 7ff4cf7fe640 1 -- 172.21.15.42:0/837025857 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4b80ab9d0 con 0x7ff4d01393a0 2024-09-16T03:57:09.742 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.726+0000 7ff4d6d53640 10 monclient: get_monmap_and_config success 2024-09-16T03:57:09.742 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.726+0000 7ff4d6d53640 4 set_mon_vals no callback set 2024-09-16T03:57:09.742 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.726+0000 7ff4cf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:57:09.742 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.726+0000 7ff4cf7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:57:09.742 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.726+0000 7ff4cf7fe640 10 monclient: dump: 2024-09-16T03:57:09.742 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:57:09.743 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:57:09.743 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:57:09.743 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:57:09.743 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:57:09.743 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:57:09.743 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:57:09.743 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:57:09.743 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:57:09.743 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:57:09.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.726+0000 7ff4d6d53640 10 monclient: shutdown 2024-09-16T03:57:09.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.726+0000 7ff4d6d53640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:57:09.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.726+0000 7ff4d6d53640 1 -- 172.21.15.42:0/837025857 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff4d01393a0 msgr2=0x7ff4d013b790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:57:09.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.726+0000 7ff4d6d53640 1 --2- 172.21.15.42:0/837025857 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff4d01393a0 0x7ff4d013b790 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7ff4b80097c0 tx=0x7ff4b801c360 comp rx=0 tx=0).stop 2024-09-16T03:57:09.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.726+0000 7ff4d6d53640 1 -- 172.21.15.42:0/837025857 shutdown_connections 2024-09-16T03:57:09.744 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.726+0000 7ff4d6d53640 1 --2- 172.21.15.42:0/837025857 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff4d01393a0 0x7ff4d013b790 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:57:09.744 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.726+0000 7ff4d6d53640 1 --2- 172.21.15.42:0/837025857 >> v2:172.21.15.154:3300/0 conn(0x7ff4d00a1330 0x7ff4d009f180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:57:09.744 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.726+0000 7ff4d6d53640 1 -- 172.21.15.42:0/837025857 >> 172.21.15.42:0/837025857 conn(0x7ff4d00a2420 msgr2=0x7ff4d00a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:57:09.744 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.726+0000 7ff4d6d53640 1 -- 172.21.15.42:0/837025857 shutdown_connections 2024-09-16T03:57:09.744 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.726+0000 7ff4d6d53640 1 -- 172.21.15.42:0/837025857 wait complete. 2024-09-16T03:57:09.744 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.726+0000 7ff4d6d53640 10 monclient: build_initial_monmap 2024-09-16T03:57:09.744 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.726+0000 7ff4d6d53640 10 monclient: monmap: 2024-09-16T03:57:09.744 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:57:09.744 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:57:09.744 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:57:09.744 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:57:09.744 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:57:09.744 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:57:09.744 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:57:09.745 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:57:09.745 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:57:09.745 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:57:09.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.726+0000 7ff4d6d53640 1 librados: starting msgr at 2024-09-16T03:57:09.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.726+0000 7ff4d6d53640 1 librados: starting objecter 2024-09-16T03:57:09.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.726+0000 7ff4d6d53640 1 Processor -- start 2024-09-16T03:57:09.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.726+0000 7ff4d6d53640 1 -- start start 2024-09-16T03:57:09.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.726+0000 7ff4d6d53640 1 librados: setting wanted keys 2024-09-16T03:57:09.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.726+0000 7ff4d6d53640 1 librados: calling monclient init 2024-09-16T03:57:09.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.726+0000 7ff4d6d53640 10 monclient: init 2024-09-16T03:57:09.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.726+0000 7ff4d6d53640 10 monclient: _reopen_session rank -1 2024-09-16T03:57:09.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.726+0000 7ff4d6d53640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:57:09.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.726+0000 7ff4d6d53640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff4d00a1330 0x7ff4d0134030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:57:09.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.726+0000 7ff4d6d53640 10 monclient(hunting): picked mon.noname-a con 0x7ff4d00a1330 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:57:09.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.726+0000 7ff4d6d53640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff4d00a1710 0x7ff4d0134570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:57:09.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.726+0000 7ff4d6d53640 10 monclient(hunting): picked mon.noname-c con 0x7ff4d00a1710 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:57:09.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.726+0000 7ff4d6d53640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff4d01393a0 0x7ff4d011dc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:57:09.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.726+0000 7ff4d6d53640 10 monclient(hunting): picked mon.noname-b con 0x7ff4d01393a0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:57:09.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.726+0000 7ff4d6d53640 10 monclient(hunting): start opening mon connection 2024-09-16T03:57:09.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.726+0000 7ff4d6d53640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7ff4d013e880 con 0x7ff4d00a1330 2024-09-16T03:57:09.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.726+0000 7ff4d6d53640 10 monclient(hunting): start opening mon connection 2024-09-16T03:57:09.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.726+0000 7ff4d6d53640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff4d013e700 con 0x7ff4d01393a0 2024-09-16T03:57:09.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.726+0000 7ff4d6d53640 10 monclient(hunting): start opening mon connection 2024-09-16T03:57:09.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.726+0000 7ff4d6d53640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7ff4d013ea00 con 0x7ff4d00a1710 2024-09-16T03:57:09.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.726+0000 7ff4d6d53640 10 monclient(hunting): _renew_subs 2024-09-16T03:57:09.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.726+0000 7ff4d6d53640 10 monclient(hunting): authenticate will time out at 1876.699161s 2024-09-16T03:57:09.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.726+0000 7ff4d4aef640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff4d00a1330 0x7ff4d0134030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:57:09.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.726+0000 7ff4cffff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff4d00a1710 0x7ff4d0134570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:57:09.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.726+0000 7ff4d4aef640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff4d00a1330 0x7ff4d0134030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:58312/0 (socket says 172.21.15.42:58312) 2024-09-16T03:57:09.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.726+0000 7ff4d4aef640 1 -- 172.21.15.42:0/1831477496 learned_addr learned my addr 172.21.15.42:0/1831477496 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:57:09.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.726+0000 7ff4d4aef640 10 monclient(hunting): get_auth_request con 0x7ff4d00a1330 auth_method 0 2024-09-16T03:57:09.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.726+0000 7ff4d4aef640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:57:09.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.726+0000 7ff4d4aef640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:57:09.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.726+0000 7ff4d4aef640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:57:09.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.726+0000 7ff4cffff640 10 monclient(hunting): get_auth_request con 0x7ff4d00a1710 auth_method 0 2024-09-16T03:57:09.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.726+0000 7ff4cffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:57:09.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.726+0000 7ff4cffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:57:09.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.726+0000 7ff4cffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:57:09.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.726+0000 7ff4d4aef640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:57:09.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.726+0000 7ff4d4aef640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:57:09.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.726+0000 7ff4d4aef640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:57:09.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.726+0000 7ff4cffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:57:09.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.726+0000 7ff4cffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:57:09.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.726+0000 7ff4cffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:57:09.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.726+0000 7ff4d52f0640 1 --2- 172.21.15.42:0/1831477496 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff4d01393a0 0x7ff4d011dc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:57:09.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.726+0000 7ff4d52f0640 10 monclient(hunting): get_auth_request con 0x7ff4d01393a0 auth_method 0 2024-09-16T03:57:09.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.726+0000 7ff4d52f0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:57:09.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.726+0000 7ff4d52f0640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:57:09.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.726+0000 7ff4d52f0640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:57:09.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.726+0000 7ff4d4aef640 10 monclient(hunting): handle_auth_done global_id 824859 payload 824 2024-09-16T03:57:09.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.726+0000 7ff4d4aef640 1 -- 172.21.15.42:0/1831477496 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff4d00a1710 msgr2=0x7ff4d0134570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:57:09.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.726+0000 7ff4d4aef640 1 --2- 172.21.15.42:0/1831477496 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff4d00a1710 0x7ff4d0134570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:57:09.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.726+0000 7ff4d4aef640 1 -- 172.21.15.42:0/1831477496 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff4d01393a0 msgr2=0x7ff4d011dc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:57:09.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.726+0000 7ff4d4aef640 1 --2- 172.21.15.42:0/1831477496 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff4d01393a0 0x7ff4d011dc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:57:09.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.726+0000 7ff4d4aef640 10 monclient: _finish_hunting 0 2024-09-16T03:57:09.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.726+0000 7ff4d4aef640 1 monclient: found mon.noname-a 2024-09-16T03:57:09.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.726+0000 7ff4d4aef640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:57:09.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.726+0000 7ff4d4aef640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:57:09.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.726+0000 7ff4d4aef640 1 -- 172.21.15.42:0/1831477496 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4d0121860 con 0x7ff4d00a1330 2024-09-16T03:57:09.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.726+0000 7ff4cffff640 1 --2- 172.21.15.42:0/1831477496 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff4d00a1710 0x7ff4d0134570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:57:09.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.726+0000 7ff4d4aef640 1 --2- 172.21.15.42:0/1831477496 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff4d00a1330 0x7ff4d0134030 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7ff4c800c7c0 tx=0x7ff4c800cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:57:09.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.726+0000 7ff4cd7fa640 10 client.?.objecter ms_handle_connect 0x7ff4d00a1330 2024-09-16T03:57:09.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.726+0000 7ff4cd7fa640 10 client.?.objecter resend_mon_ops 2024-09-16T03:57:09.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.730+0000 7ff4cd7fa640 1 -- 172.21.15.42:0/1831477496 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4c8007a30 con 0x7ff4d00a1330 2024-09-16T03:57:09.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.730+0000 7ff4cd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:57:09.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.730+0000 7ff4cd7fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:57:09.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.730+0000 7ff4cd7fa640 10 monclient: dump: 2024-09-16T03:57:09.750 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:57:09.750 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:57:09.750 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:57:09.750 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:57:09.750 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:57:09.751 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:57:09.751 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:57:09.751 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:57:09.751 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:57:09.751 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:57:09.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.730+0000 7ff4cd7fa640 10 monclient: _finish_auth 0 2024-09-16T03:57:09.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.730+0000 7ff4cd7fa640 10 monclient: _check_auth_tickets 2024-09-16T03:57:09.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.730+0000 7ff4cd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:57:09.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.730+0000 7ff4cd7fa640 1 -- 172.21.15.42:0/1831477496 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4c8007bd0 con 0x7ff4d00a1330 2024-09-16T03:57:09.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.730+0000 7ff4cd7fa640 10 monclient: handle_config config(0 keys) 2024-09-16T03:57:09.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.730+0000 7ff4cd7fa640 1 -- 172.21.15.42:0/1831477496 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4c80113f0 con 0x7ff4d00a1330 2024-09-16T03:57:09.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.730+0000 7ff4cd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:57:09.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.730+0000 7ff4cd7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:57:09.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.730+0000 7ff4cd7fa640 10 monclient: dump: 2024-09-16T03:57:09.752 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:57:09.752 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:57:09.752 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:57:09.752 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:57:09.752 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:57:09.752 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:57:09.752 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:57:09.752 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:57:09.753 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:57:09.753 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:57:09.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.730+0000 7ff4d6d53640 5 monclient: authenticate success, global_id 824859 2024-09-16T03:57:09.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.730+0000 7ff4d6d53640 10 monclient: _renew_subs 2024-09-16T03:57:09.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.730+0000 7ff4d6d53640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:57:09.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.730+0000 7ff4ceffd640 4 set_mon_vals no callback set 2024-09-16T03:57:09.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.730+0000 7ff4d6d53640 1 -- 172.21.15.42:0/1831477496 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4d0073a30 con 0x7ff4d00a1330 2024-09-16T03:57:09.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.730+0000 7ff4d6d53640 10 client.824859.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:57:09.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.730+0000 7ff4d6d53640 10 monclient: _renew_subs 2024-09-16T03:57:09.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.730+0000 7ff4d6d53640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:57:09.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.730+0000 7ff4d6d53640 1 -- 172.21.15.42:0/1831477496 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4d00a0c90 con 0x7ff4d00a1330 2024-09-16T03:57:09.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.730+0000 7ff4d6d53640 1 librados: init done 2024-09-16T03:57:09.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.730+0000 7ff4d6d53640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:57:09.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.730+0000 7ff4d6d53640 1 --2- 172.21.15.42:0/1831477496 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff4d011e190 0x7ff4d011e570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:57:09.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.730+0000 7ff4d6d53640 10 monclient(hunting): start opening mon connection 2024-09-16T03:57:09.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.730+0000 7ff4d6d53640 1 -- 172.21.15.42:0/1831477496 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7ff4d009f180 con 0x7ff4d011e190 2024-09-16T03:57:09.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.730+0000 7ff4cffff640 1 --2- 172.21.15.42:0/1831477496 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff4d011e190 0x7ff4d011e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:57:09.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.730+0000 7ff4cffff640 10 monclient: get_auth_request con 0x7ff4d011e190 auth_method 0 2024-09-16T03:57:09.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.730+0000 7ff4cffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:57:09.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.730+0000 7ff4cffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:57:09.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.730+0000 7ff4cffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:57:09.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.730+0000 7ff4cffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:57:09.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.730+0000 7ff4cffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:57:09.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.730+0000 7ff4cffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:57:09.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.730+0000 7ff4cd7fa640 1 -- 172.21.15.42:0/1831477496 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7ff4c8011590 con 0x7ff4d00a1330 2024-09-16T03:57:09.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.730+0000 7ff4cffff640 10 monclient(hunting): handle_auth_done global_id 854396 payload 824 2024-09-16T03:57:09.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.730+0000 7ff4cffff640 1 -- 172.21.15.42:0/1831477496 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff4d01215e0 con 0x7ff4d011e190 2024-09-16T03:57:09.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.730+0000 7ff4cd7fa640 1 --2- 172.21.15.42:0/1831477496 >> v2:172.21.15.154:6812/254845342 conn(0x7ff4b4037810 0x7ff4b4039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:57:09.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.730+0000 7ff4cd7fa640 1 -- 172.21.15.42:0/1831477496 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(460..460 src has 330..460) ==== 5701+0+0 (secure 0 0 0) 0x7ff4c8049100 con 0x7ff4d00a1330 2024-09-16T03:57:09.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.730+0000 7ff4cd7fa640 10 client.824859.objecter ms_dispatch 0x7ff4d0000ba0 osd_map(460..460 src has 330..460) 2024-09-16T03:57:09.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.730+0000 7ff4cffff640 1 --2- 172.21.15.42:0/1831477496 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff4d011e190 0x7ff4d011e570 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7ff4d011f450 tx=0x7ff4c000d830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:57:09.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.730+0000 7ff4cd7fa640 3 client.824859.objecter handle_osd_map got epochs [460,460] > 0 2024-09-16T03:57:09.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.730+0000 7ff4cd7fa640 3 client.824859.objecter handle_osd_map decoding full epoch 460 2024-09-16T03:57:09.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.730+0000 7ff4cd7fa640 20 client.824859.objecter dump_active .. 0 homeless 2024-09-16T03:57:09.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.730+0000 7ff4cd7fa640 10 client.824859.objecter ms_handle_connect 0x7ff4d011e190 2024-09-16T03:57:09.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.730+0000 7ff4cd7fa640 10 client.824859.objecter resend_mon_ops 2024-09-16T03:57:09.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.730+0000 7ff4cd7fa640 1 -- 172.21.15.42:0/1831477496 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4c0019070 con 0x7ff4d011e190 2024-09-16T03:57:09.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.730+0000 7ff4cd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:57:09.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.730+0000 7ff4cd7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:57:09.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.730+0000 7ff4cd7fa640 10 monclient: dump: 2024-09-16T03:57:09.756 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:57:09.756 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:57:09.756 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:57:09.756 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:57:09.756 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:57:09.757 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:57:09.757 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:57:09.757 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:57:09.757 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:57:09.757 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:57:09.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.730+0000 7ff4d52f0640 1 --2- 172.21.15.42:0/1831477496 >> v2:172.21.15.154:6812/254845342 conn(0x7ff4b4037810 0x7ff4b4039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:57:09.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.730+0000 7ff4cd7fa640 1 -- 172.21.15.42:0/1831477496 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7ff4c0007320 con 0x7ff4d011e190 2024-09-16T03:57:09.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.730+0000 7ff4d52f0640 10 monclient: get_auth_request con 0x7ff4b4037810 auth_method 0 2024-09-16T03:57:09.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.730+0000 7ff4cd7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:57:09.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.730+0000 7ff4cd7fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:57:09.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.730+0000 7ff4cd7fa640 1 -- 172.21.15.42:0/1831477496 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff4d011e190 msgr2=0x7ff4d011e570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:57:09.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.730+0000 7ff4cd7fa640 1 --2- 172.21.15.42:0/1831477496 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff4d011e190 0x7ff4d011e570 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7ff4d011f450 tx=0x7ff4c000d830 comp rx=0 tx=0).stop 2024-09-16T03:57:09.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.730+0000 7ff4cd7fa640 1 -- 172.21.15.42:0/1831477496 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff4d011e190 msgr2=0x7ff4d011e570 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:57:09.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.730+0000 7ff4cd7fa640 1 --2- 172.21.15.42:0/1831477496 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff4d011e190 0x7ff4d011e570 secure :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7ff4d011f450 tx=0x7ff4c000d830 comp rx=0 tx=0).stop 2024-09-16T03:57:09.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.730+0000 7ff4d52f0640 1 --2- 172.21.15.42:0/1831477496 >> v2:172.21.15.154:6812/254845342 conn(0x7ff4b4037810 0x7ff4b4039cd0 secure :-1 s=READY pgs=3673 cs=0 l=1 rev1=1 crypto rx=0x7ff4b80ad000 tx=0x7ff4b80a6000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:57:09.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.730+0000 7ff4cd7fa640 10 client.824859.objecter ms_handle_connect 0x7ff4b4037810 2024-09-16T03:57:09.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.738+0000 7ff4d6d53640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:57:09.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.738+0000 7ff4d6d53640 1 --2- 172.21.15.42:0/1831477496 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff4d011eab0 0x7ff4d0137390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:57:09.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.738+0000 7ff4d6d53640 10 monclient(hunting): start opening mon connection 2024-09-16T03:57:09.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.738+0000 7ff4d6d53640 1 -- 172.21.15.42:0/1831477496 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7ff4d011eec0 con 0x7ff4d011eab0 2024-09-16T03:57:09.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.738+0000 7ff4cffff640 1 --2- 172.21.15.42:0/1831477496 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff4d011eab0 0x7ff4d0137390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:57:09.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.738+0000 7ff4cffff640 10 monclient: get_auth_request con 0x7ff4d011eab0 auth_method 0 2024-09-16T03:57:09.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.738+0000 7ff4cffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:57:09.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.738+0000 7ff4cffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:57:09.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.738+0000 7ff4cffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:57:09.761 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.738+0000 7ff4cffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:57:09.761 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.738+0000 7ff4cffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:57:09.761 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.738+0000 7ff4cffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:57:09.761 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.738+0000 7ff4cffff640 10 monclient(hunting): handle_auth_done global_id 854399 payload 824 2024-09-16T03:57:09.761 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.738+0000 7ff4cffff640 1 -- 172.21.15.42:0/1831477496 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ff4d01203b0 con 0x7ff4d011eab0 2024-09-16T03:57:09.761 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.738+0000 7ff4cffff640 1 --2- 172.21.15.42:0/1831477496 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff4d011eab0 0x7ff4d0137390 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7ff4c0013000 tx=0x7ff4c0005b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:57:09.761 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.738+0000 7ff4cd7fa640 10 client.824859.objecter ms_handle_connect 0x7ff4d011eab0 2024-09-16T03:57:09.761 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.738+0000 7ff4cd7fa640 10 client.824859.objecter resend_mon_ops 2024-09-16T03:57:09.761 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.738+0000 7ff4cd7fa640 1 -- 172.21.15.42:0/1831477496 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4c0019050 con 0x7ff4d011eab0 2024-09-16T03:57:09.761 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.738+0000 7ff4cd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:57:09.761 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.738+0000 7ff4cd7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:57:09.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.738+0000 7ff4cd7fa640 10 monclient: dump: 2024-09-16T03:57:09.762 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:57:09.762 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:57:09.762 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:57:09.762 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:57:09.762 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:57:09.762 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:57:09.762 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:57:09.762 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:57:09.762 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:57:09.762 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:57:09.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.738+0000 7ff4cd7fa640 1 -- 172.21.15.42:0/1831477496 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4546 (secure 0 0 0) 0x7ff4c001a040 con 0x7ff4d011eab0 2024-09-16T03:57:09.763 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.738+0000 7ff4cd7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:57:09.763 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.738+0000 7ff4cd7fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:57:09.763 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.738+0000 7ff4cd7fa640 1 -- 172.21.15.42:0/1831477496 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff4d011eab0 msgr2=0x7ff4d0137390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:57:09.763 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.738+0000 7ff4cd7fa640 1 --2- 172.21.15.42:0/1831477496 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff4d011eab0 0x7ff4d0137390 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7ff4c0013000 tx=0x7ff4c0005b80 comp rx=0 tx=0).stop 2024-09-16T03:57:09.763 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.738+0000 7ff4cd7fa640 1 -- 172.21.15.42:0/1831477496 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff4d011eab0 msgr2=0x7ff4d0137390 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:57:09.763 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.738+0000 7ff4cd7fa640 1 --2- 172.21.15.42:0/1831477496 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff4d011eab0 0x7ff4d0137390 secure :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7ff4c0013000 tx=0x7ff4c0005b80 comp rx=0 tx=0).stop 2024-09-16T03:57:09.763 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:57:09.764 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:57:09.764 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:57:09.764 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T03:57:09.764 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 140, 2024-09-16T03:57:09.764 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 163174, 2024-09-16T03:57:09.764 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:57:09.764 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:57:09.764 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T03:57:09.765 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T03:57:09.765 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:57:09.765 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 3, 2024-09-16T03:57:09.765 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:57:09.765 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:57:09.765 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:57:09.765 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:57:09.765 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:57:09.765 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:57:09.765 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:57:09.766 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:57:09.766 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:57:09.766 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:57:09.766 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:57:09.766 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:57:09.766 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:57:09.766 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:57:09.766 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:57:09.766 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:57:09.766 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:57:09.766 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:57:09.766 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:57:09.766 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:57:09.767 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:57:09.767 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:57:09.767 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:57:09.767 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:57:09.767 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:57:09.767 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:57:09.767 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:57:09.767 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:57:09.767 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:57:09.767 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:57:09.767 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:57:09.767 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:57:09.768 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:57:09.768 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:57:09.768 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:57:09.768 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:57:09.768 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:57:09.768 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:57:09.768 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:57:09.768 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:57:09.768 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:57:09.768 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:57:09.768 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:57:09.768 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:57:09.768 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:57:09.769 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:57:09.769 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:57:09.769 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:57:09.769 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:57:09.769 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:57:09.769 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:57:09.769 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:57:09.769 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:57:09.769 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:57:09.769 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:57:09.769 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:57:09.769 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:57:09.769 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:57:09.770 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:57:09.770 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:57:09.770 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:57:09.770 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:57:09.770 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:57:09.770 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:57:09.770 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:57:09.770 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:57:09.770 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:57:09.770 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:57:09.770 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:57:09.770 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:57:09.771 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:57:09.771 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:57:09.771 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:57:09.771 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:57:09.771 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:57:09.771 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:57:09.771 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:57:09.771 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:57:09.771 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:57:09.771 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:57:09.771 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:57:09.771 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:57:09.772 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:57:09.772 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:57:09.772 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:57:09.772 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:57:09.772 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:57:09.772 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:57:09.772 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:57:09.772 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:57:09.772 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:57:09.772 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:57:09.772 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:57:09.772 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:57:09.772 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:57:09.772 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:57:09.773 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:57:09.773 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:57:09.773 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:57:09.773 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:57:09.773 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:57:09.773 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:57:09.773 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:57:09.773 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:57:09.773 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:57:09.773 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:57:09.773 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:57:09.773 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:57:09.773 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:57:09.774 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:57:09.774 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:57:09.774 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:57:09.774 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:57:09.774 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:57:09.774 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:57:09.774 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:57:09.774 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:57:09.774 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:57:09.774 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:57:09.774 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:57:09.774 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:57:09.774 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:57:09.774 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:57:09.775 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:57:09.775 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:57:09.775 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:57:09.775 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:57:09.775 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:57:09.775 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:57:09.775 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:57:09.775 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:57:09.775 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:57:09.775 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:57:09.775 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:57:09.775 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:57:09.776 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:57:09.776 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:57:09.776 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:57:09.776 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:57:09.776 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:57:09.776 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:57:09.776 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:57:09.777 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:57:09.777 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:57:09.777 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:57:09.777 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:57:09.777 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:57:09.777 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:57:09.777 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:57:09.777 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:57:09.777 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:57:09.779 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.738+0000 7ff4d6d53640 10 librados: watch_flush enter 2024-09-16T03:57:09.779 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.738+0000 7ff4d6d53640 10 librados: watch_flush exit 2024-09-16T03:57:09.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.738+0000 7ff4d6d53640 20 client.824859.objecter shutdown clearing up homeless session... 2024-09-16T03:57:09.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.738+0000 7ff4d6d53640 10 client.824859.objecter successfully canceled tick 2024-09-16T03:57:09.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.738+0000 7ff4d6d53640 1 -- 172.21.15.42:0/1831477496 >> v2:172.21.15.154:6812/254845342 conn(0x7ff4b4037810 msgr2=0x7ff4b4039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:57:09.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.738+0000 7ff4d6d53640 1 --2- 172.21.15.42:0/1831477496 >> v2:172.21.15.154:6812/254845342 conn(0x7ff4b4037810 0x7ff4b4039cd0 secure :-1 s=READY pgs=3673 cs=0 l=1 rev1=1 crypto rx=0x7ff4b80ad000 tx=0x7ff4b80a6000 comp rx=0 tx=0).stop 2024-09-16T03:57:09.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.738+0000 7ff4d6d53640 10 monclient: shutdown 2024-09-16T03:57:09.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.738+0000 7ff4d6d53640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:57:09.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.738+0000 7ff4d6d53640 1 -- 172.21.15.42:0/1831477496 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff4d00a1330 msgr2=0x7ff4d0134030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:57:09.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.738+0000 7ff4d6d53640 1 --2- 172.21.15.42:0/1831477496 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff4d00a1330 0x7ff4d0134030 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7ff4c800c7c0 tx=0x7ff4c800cc80 comp rx=0 tx=0).stop 2024-09-16T03:57:09.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.738+0000 7ff4d52f0640 1 -- 172.21.15.42:0/1831477496 reap_dead start 2024-09-16T03:57:09.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.738+0000 7ff4d52f0640 1 -- 172.21.15.42:0/1831477496 reap_dead start 2024-09-16T03:57:09.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.738+0000 7ff4d6d53640 1 -- 172.21.15.42:0/1831477496 shutdown_connections 2024-09-16T03:57:09.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.738+0000 7ff4d6d53640 1 -- 172.21.15.42:0/1831477496 >> 172.21.15.42:0/1831477496 conn(0x7ff4d00a2420 msgr2=0x7ff4d00a46b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:57:09.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.738+0000 7ff4d6d53640 1 -- 172.21.15.42:0/1831477496 shutdown_connections 2024-09-16T03:57:09.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.738+0000 7ff4d6d53640 1 -- 172.21.15.42:0/1831477496 wait complete. 2024-09-16T03:57:09.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.738+0000 7ff4d6d53640 1 librados: shutdown 2024-09-16T03:57:09.781 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:57:09.781 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T03:57:09.781 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T03:57:09.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.898+0000 7f94a6509640 10 monclient: get_monmap_and_config 2024-09-16T03:57:09.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.898+0000 7f94a6509640 10 monclient: build_initial_monmap 2024-09-16T03:57:09.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.898+0000 7f94a6509640 10 monclient: monmap: 2024-09-16T03:57:09.902 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:57:09.902 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:57:09.902 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:57:09.900324+0000 2024-09-16T03:57:09.902 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:57:09.900324+0000 2024-09-16T03:57:09.902 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:57:09.902 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:57:09.902 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:57:09.902 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:57:09.902 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:57:09.903 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:57:09.903 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:57:09.903 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:57:09.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.898+0000 7f94a6509640 1 Processor -- start 2024-09-16T03:57:09.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.898+0000 7f94a6509640 1 -- start start 2024-09-16T03:57:09.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.898+0000 7f94a6509640 10 monclient: init 2024-09-16T03:57:09.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.898+0000 7f94a6509640 10 monclient: _reopen_session rank -1 2024-09-16T03:57:09.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.898+0000 7f94a6509640 10 monclient: _add_conns ranks=[0,1,2,4,3] 2024-09-16T03:57:09.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.898+0000 7f94a6509640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f94a00a39b0 addr v1:172.21.15.42:6789/0 2024-09-16T03:57:09.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.898+0000 7f94a6509640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f94a00a3d90 addr v1:172.21.15.154:6789/0 2024-09-16T03:57:09.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.898+0000 7f94a6509640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f94a013daa0 0x7f94a013fe90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:57:09.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.898+0000 7f94a6509640 10 monclient(hunting): picked mon.noname-b con 0x7f94a013daa0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:57:09.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.898+0000 7f94a6509640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f94a0134290 con 0x7f94a00a39b0 2024-09-16T03:57:09.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.898+0000 7f94a6509640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f94a009c140 con 0x7f94a00a3d90 2024-09-16T03:57:09.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.898+0000 7f94a6509640 10 monclient(hunting): start opening mon connection 2024-09-16T03:57:09.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.898+0000 7f94a6509640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f94a009c420 con 0x7f94a013daa0 2024-09-16T03:57:09.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.898+0000 7f94a6509640 10 monclient(hunting): _renew_subs 2024-09-16T03:57:09.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.898+0000 7f94a6509640 10 monclient(hunting): authenticate will time out at 1876.869739s 2024-09-16T03:57:09.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.898+0000 7f94a4aa6640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f94a013daa0 0x7f94a013fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:57:09.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.898+0000 7f949ffff640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f94a00a39b0 0x7f94a0133b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:44322/0 (socket says 172.21.15.42:44322) 2024-09-16T03:57:09.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.898+0000 7f949ffff640 1 -- 172.21.15.42:0/462244597 learned_addr learned my addr 172.21.15.42:0/462244597 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:57:09.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.898+0000 7f94a4aa6640 10 monclient(hunting): get_auth_request con 0x7f94a013daa0 auth_method 0 2024-09-16T03:57:09.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.898+0000 7f94a4aa6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:57:09.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.898+0000 7f94a4aa6640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:57:09.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.898+0000 7f94a4aa6640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:57:09.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.898+0000 7f94a4aa6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:57:09.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.898+0000 7f94a4aa6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:57:09.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.898+0000 7f94a4aa6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:57:09.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.898+0000 7f949effd640 1 -- 172.21.15.42:0/462244597 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1362646679 0 0) 0x7f94a0134290 con 0x7f94a00a39b0 2024-09-16T03:57:09.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.898+0000 7f949effd640 10 monclient(hunting): _negotiate 2024-09-16T03:57:09.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.898+0000 7f949effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:57:09.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.898+0000 7f949effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:57:09.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.898+0000 7f949effd640 10 monclient(hunting): my global_id is 824862 2024-09-16T03:57:09.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.898+0000 7f949effd640 1 -- 172.21.15.42:0/462244597 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9488003530 con 0x7f94a00a39b0 2024-09-16T03:57:09.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.898+0000 7f94a4aa6640 10 monclient(hunting): handle_auth_done global_id 854402 payload 274 2024-09-16T03:57:09.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.898+0000 7f94a4aa6640 1 -- 172.21.15.42:0/462244597 >> v1:172.21.15.154:6789/0 conn(0x7f94a00a3d90 legacy=0x7f94a013c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:57:09.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.898+0000 7f94a4aa6640 1 -- 172.21.15.42:0/462244597 >> v1:172.21.15.42:6789/0 conn(0x7f94a00a39b0 legacy=0x7f94a0133b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:57:09.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.898+0000 7f94a4aa6640 10 monclient: _finish_hunting 0 2024-09-16T03:57:09.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.898+0000 7f94a4aa6640 1 monclient: found mon.noname-b 2024-09-16T03:57:09.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.898+0000 7f94a4aa6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:57:09.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.898+0000 7f94a4aa6640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T03:57:09.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.898+0000 7f94a4aa6640 1 -- 172.21.15.42:0/462244597 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94a006dd60 con 0x7f94a013daa0 2024-09-16T03:57:09.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.902+0000 7f94a4aa6640 1 --2- 172.21.15.42:0/462244597 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f94a013daa0 0x7f94a013fe90 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f9498009dc0 tx=0x7f949801c850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d4edf908d33d0281 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:57:09.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.902+0000 7f949effd640 1 -- 172.21.15.42:0/462244597 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94980b2070 con 0x7f94a013daa0 2024-09-16T03:57:09.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.902+0000 7f949effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:57:09.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.902+0000 7f949effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:57:09.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.902+0000 7f949effd640 10 monclient: dump: 2024-09-16T03:57:09.912 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:57:09.912 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:57:09.912 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:57:09.912 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:57:09.912 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:57:09.912 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:57:09.912 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:57:09.912 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:57:09.912 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:57:09.912 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:57:09.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.902+0000 7f949effd640 10 monclient: _finish_auth 0 2024-09-16T03:57:09.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.902+0000 7f949effd640 10 monclient: _check_auth_tickets 2024-09-16T03:57:09.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.902+0000 7f949effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:57:09.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.902+0000 7f949effd640 1 -- 172.21.15.42:0/462244597 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f94980a5710 con 0x7f94a013daa0 2024-09-16T03:57:09.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.902+0000 7f949effd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:57:09.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.902+0000 7f949effd640 1 -- 172.21.15.42:0/462244597 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94980a5ae0 con 0x7f94a013daa0 2024-09-16T03:57:09.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.902+0000 7f94a6509640 5 monclient: authenticate success, global_id 854402 2024-09-16T03:57:09.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.902+0000 7f94a6509640 10 monclient: get_monmap_and_config success 2024-09-16T03:57:09.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.902+0000 7f94a6509640 4 set_mon_vals no callback set 2024-09-16T03:57:09.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.902+0000 7f949effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:57:09.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.902+0000 7f949effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:57:09.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.902+0000 7f949effd640 10 monclient: dump: 2024-09-16T03:57:09.913 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:57:09.914 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:57:09.914 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:57:09.914 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:57:09.914 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:57:09.914 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:57:09.914 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:57:09.914 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:57:09.914 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:57:09.914 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:57:09.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.902+0000 7f94a6509640 10 monclient: shutdown 2024-09-16T03:57:09.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.902+0000 7f94a6509640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:57:09.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.902+0000 7f94a6509640 1 -- 172.21.15.42:0/462244597 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f94a013daa0 msgr2=0x7f94a013fe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:57:09.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.902+0000 7f94a6509640 1 --2- 172.21.15.42:0/462244597 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f94a013daa0 0x7f94a013fe90 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f9498009dc0 tx=0x7f949801c850 comp rx=0 tx=0).stop 2024-09-16T03:57:09.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.902+0000 7f94a6509640 1 -- 172.21.15.42:0/462244597 shutdown_connections 2024-09-16T03:57:09.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.902+0000 7f94a6509640 1 --2- 172.21.15.42:0/462244597 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f94a013daa0 0x7f94a013fe90 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:57:09.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.902+0000 7f94a6509640 1 -- 172.21.15.42:0/462244597 >> 172.21.15.42:0/462244597 conn(0x7f94a00a9fb0 msgr2=0x7f94a00aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:57:09.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.902+0000 7f94a6509640 1 -- 172.21.15.42:0/462244597 shutdown_connections 2024-09-16T03:57:09.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.902+0000 7f94a6509640 1 -- 172.21.15.42:0/462244597 wait complete. 2024-09-16T03:57:09.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.902+0000 7f94a6509640 10 monclient: build_initial_monmap 2024-09-16T03:57:09.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.902+0000 7f94a6509640 10 monclient: monmap: 2024-09-16T03:57:09.915 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:57:09.915 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:57:09.915 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:57:09.915 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:57:09.915 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:57:09.915 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:57:09.915 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:57:09.915 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:57:09.915 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:57:09.916 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:57:09.916 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.902+0000 7f94a6509640 1 librados: starting msgr at 2024-09-16T03:57:09.916 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.902+0000 7f94a6509640 1 librados: starting objecter 2024-09-16T03:57:09.916 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.902+0000 7f94a6509640 1 Processor -- start 2024-09-16T03:57:09.916 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.902+0000 7f94a6509640 1 -- start start 2024-09-16T03:57:09.916 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.902+0000 7f94a6509640 1 librados: setting wanted keys 2024-09-16T03:57:09.916 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.902+0000 7f94a6509640 1 librados: calling monclient init 2024-09-16T03:57:09.916 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.902+0000 7f94a6509640 10 monclient: init 2024-09-16T03:57:09.929 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.902+0000 7f94a6509640 10 monclient: _reopen_session rank -1 2024-09-16T03:57:09.929 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.902+0000 7f94a6509640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:57:09.929 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.902+0000 7f94a6509640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f94a00a39b0 0x7f94a0130be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:57:09.929 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.902+0000 7f94a6509640 10 monclient(hunting): picked mon.noname-c con 0x7f94a00a39b0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:57:09.929 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.902+0000 7f94a6509640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f94a00a3d90 0x7f94a0139fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:57:09.930 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.902+0000 7f94a6509640 10 monclient(hunting): picked mon.noname-b con 0x7f94a00a3d90 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:57:09.930 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.902+0000 7f94a6509640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f94a013daa0 0x7f94a013a510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:57:09.930 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.902+0000 7f94a6509640 10 monclient(hunting): picked mon.noname-a con 0x7f94a013daa0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:57:09.930 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.902+0000 7f94a6509640 10 monclient(hunting): start opening mon connection 2024-09-16T03:57:09.930 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.902+0000 7f94a6509640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f94a0142f80 con 0x7f94a013daa0 2024-09-16T03:57:09.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.902+0000 7f94a6509640 10 monclient(hunting): start opening mon connection 2024-09-16T03:57:09.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.902+0000 7f94a6509640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f94a0142e00 con 0x7f94a00a3d90 2024-09-16T03:57:09.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.902+0000 7f94a6509640 10 monclient(hunting): start opening mon connection 2024-09-16T03:57:09.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.902+0000 7f94a6509640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f94a0143100 con 0x7f94a00a39b0 2024-09-16T03:57:09.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.902+0000 7f94a6509640 10 monclient(hunting): _renew_subs 2024-09-16T03:57:09.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.902+0000 7f94a6509640 10 monclient(hunting): authenticate will time out at 1876.875287s 2024-09-16T03:57:09.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.902+0000 7f949ffff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f94a00a39b0 0x7f94a0130be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:57:09.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.902+0000 7f949ffff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f94a00a39b0 0x7f94a0130be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:36162/0 (socket says 172.21.15.42:36162) 2024-09-16T03:57:09.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.902+0000 7f94a4aa6640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f94a013daa0 0x7f94a013a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:57:09.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.902+0000 7f949ffff640 1 -- 172.21.15.42:0/3369392345 learned_addr learned my addr 172.21.15.42:0/3369392345 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:57:09.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.902+0000 7f949ffff640 10 monclient(hunting): get_auth_request con 0x7f94a00a39b0 auth_method 0 2024-09-16T03:57:09.933 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.902+0000 7f949ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:57:09.933 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.902+0000 7f949ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:57:09.933 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.902+0000 7f949ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:57:09.933 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.902+0000 7f94a4aa6640 10 monclient(hunting): get_auth_request con 0x7f94a013daa0 auth_method 0 2024-09-16T03:57:09.933 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.902+0000 7f94a4aa6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:57:09.934 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.902+0000 7f94a4aa6640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:57:09.934 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.902+0000 7f94a4aa6640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:57:09.934 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.902+0000 7f949ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:57:09.934 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.902+0000 7f949ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:57:09.934 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.902+0000 7f949ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:57:09.934 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.902+0000 7f94a4aa6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:57:09.934 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.902+0000 7f94a4aa6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:57:09.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.902+0000 7f94a4aa6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:57:09.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.902+0000 7f949f7fe640 1 --2- 172.21.15.42:0/3369392345 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f94a00a3d90 0x7f94a0139fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:57:09.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.902+0000 7f949f7fe640 10 monclient(hunting): get_auth_request con 0x7f94a00a3d90 auth_method 0 2024-09-16T03:57:09.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.902+0000 7f949f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:57:09.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.902+0000 7f949f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:57:09.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.902+0000 7f949f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:57:09.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.902+0000 7f94a4aa6640 10 monclient(hunting): handle_auth_done global_id 824865 payload 824 2024-09-16T03:57:09.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.902+0000 7f94a4aa6640 1 -- 172.21.15.42:0/3369392345 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f94a00a39b0 msgr2=0x7f94a0130be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:57:09.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.902+0000 7f94a4aa6640 1 --2- 172.21.15.42:0/3369392345 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f94a00a39b0 0x7f94a0130be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:57:09.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.902+0000 7f94a4aa6640 1 -- 172.21.15.42:0/3369392345 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f94a00a3d90 msgr2=0x7f94a0139fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:57:09.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.902+0000 7f94a4aa6640 1 --2- 172.21.15.42:0/3369392345 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f94a00a3d90 0x7f94a0139fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:57:09.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.902+0000 7f94a4aa6640 10 monclient: _finish_hunting 0 2024-09-16T03:57:09.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.902+0000 7f94a4aa6640 1 monclient: found mon.noname-a 2024-09-16T03:57:09.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.902+0000 7f94a4aa6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:57:09.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.902+0000 7f94a4aa6640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:57:09.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.902+0000 7f94a4aa6640 1 -- 172.21.15.42:0/3369392345 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94a00a95a0 con 0x7f94a013daa0 2024-09-16T03:57:09.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.902+0000 7f949ffff640 1 --2- 172.21.15.42:0/3369392345 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f94a00a39b0 0x7f94a0130be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:57:09.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.906+0000 7f94a4aa6640 1 --2- 172.21.15.42:0/3369392345 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f94a013daa0 0x7f94a013a510 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f94980ac000 tx=0x7f94980195f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:57:09.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.906+0000 7f949cff9640 10 client.?.objecter ms_handle_connect 0x7f94a013daa0 2024-09-16T03:57:09.938 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.906+0000 7f949cff9640 10 client.?.objecter resend_mon_ops 2024-09-16T03:57:09.938 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.906+0000 7f949cff9640 1 -- 172.21.15.42:0/3369392345 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94980b2050 con 0x7f94a013daa0 2024-09-16T03:57:09.938 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.906+0000 7f949cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:57:09.938 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.906+0000 7f949cff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:57:09.938 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.906+0000 7f949cff9640 10 monclient: dump: 2024-09-16T03:57:09.938 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:57:09.938 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:57:09.938 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:57:09.938 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:57:09.938 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:57:09.938 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:57:09.938 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:57:09.938 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:57:09.939 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:57:09.939 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:57:09.939 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.906+0000 7f949cff9640 10 monclient: _finish_auth 0 2024-09-16T03:57:09.939 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.906+0000 7f949cff9640 10 monclient: _check_auth_tickets 2024-09-16T03:57:09.939 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.906+0000 7f949cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:57:09.939 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.906+0000 7f949cff9640 1 -- 172.21.15.42:0/3369392345 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f94980be070 con 0x7f94a013daa0 2024-09-16T03:57:09.939 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.906+0000 7f94a6509640 5 monclient: authenticate success, global_id 824865 2024-09-16T03:57:09.939 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.906+0000 7f94a6509640 10 monclient: _renew_subs 2024-09-16T03:57:09.939 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.906+0000 7f94a6509640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:57:09.939 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.906+0000 7f94a6509640 1 -- 172.21.15.42:0/3369392345 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94a013d340 con 0x7f94a013daa0 2024-09-16T03:57:09.939 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.906+0000 7f949cff9640 10 monclient: handle_config config(0 keys) 2024-09-16T03:57:09.939 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.906+0000 7f949cff9640 1 -- 172.21.15.42:0/3369392345 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9498009450 con 0x7f94a013daa0 2024-09-16T03:57:09.939 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.906+0000 7f949cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:57:09.939 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.906+0000 7f949cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:57:09.940 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.906+0000 7f949cff9640 10 monclient: dump: 2024-09-16T03:57:09.940 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:57:09.940 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:57:09.940 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:57:09.940 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:57:09.940 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:57:09.940 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:57:09.940 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:57:09.940 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:57:09.940 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:57:09.941 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:57:09.941 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.906+0000 7f949dffb640 4 set_mon_vals no callback set 2024-09-16T03:57:09.941 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.906+0000 7f94a6509640 10 client.824865.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:57:09.941 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.906+0000 7f94a6509640 10 monclient: _renew_subs 2024-09-16T03:57:09.941 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.906+0000 7f94a6509640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:57:09.941 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.906+0000 7f94a6509640 1 -- 172.21.15.42:0/3369392345 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94a006f680 con 0x7f94a013daa0 2024-09-16T03:57:09.941 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.906+0000 7f94a6509640 1 librados: init done 2024-09-16T03:57:09.941 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.906+0000 7f94a6509640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:57:09.941 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.906+0000 7f949cff9640 1 -- 172.21.15.42:0/3369392345 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f94980adc40 con 0x7f94a013daa0 2024-09-16T03:57:09.941 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.906+0000 7f94a6509640 1 --2- 172.21.15.42:0/3369392345 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f94a013b210 0x7f94a002c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:57:09.941 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.906+0000 7f94a6509640 10 monclient(hunting): start opening mon connection 2024-09-16T03:57:09.941 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.906+0000 7f94a6509640 1 -- 172.21.15.42:0/3369392345 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f94a0133b80 con 0x7f94a013b210 2024-09-16T03:57:09.941 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.906+0000 7f949cff9640 1 --2- 172.21.15.42:0/3369392345 >> v2:172.21.15.154:6812/254845342 conn(0x7f948c037810 0x7f948c039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:57:09.941 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.906+0000 7f949cff9640 1 -- 172.21.15.42:0/3369392345 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(460..460 src has 330..460) ==== 5701+0+0 (secure 0 0 0) 0x7f94980e40c0 con 0x7f94a013daa0 2024-09-16T03:57:09.942 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.906+0000 7f949cff9640 10 client.824865.objecter ms_dispatch 0x7f94a0000ba0 osd_map(460..460 src has 330..460) 2024-09-16T03:57:09.942 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.906+0000 7f949cff9640 3 client.824865.objecter handle_osd_map got epochs [460,460] > 0 2024-09-16T03:57:09.942 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.906+0000 7f949cff9640 3 client.824865.objecter handle_osd_map decoding full epoch 460 2024-09-16T03:57:09.942 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.906+0000 7f949cff9640 20 client.824865.objecter dump_active .. 0 homeless 2024-09-16T03:57:09.942 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.906+0000 7f949ffff640 1 --2- 172.21.15.42:0/3369392345 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f94a013b210 0x7f94a002c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:57:09.942 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.906+0000 7f949ffff640 10 monclient: get_auth_request con 0x7f94a013b210 auth_method 0 2024-09-16T03:57:09.942 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.906+0000 7f949ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:57:09.942 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.906+0000 7f949ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:57:09.942 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.906+0000 7f949ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:57:09.943 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.906+0000 7f949f7fe640 1 --2- 172.21.15.42:0/3369392345 >> v2:172.21.15.154:6812/254845342 conn(0x7f948c037810 0x7f948c039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:57:09.943 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.906+0000 7f949ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:57:09.943 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.906+0000 7f949ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:57:09.943 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.906+0000 7f949ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:57:09.943 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.906+0000 7f949f7fe640 10 monclient: get_auth_request con 0x7f948c037810 auth_method 0 2024-09-16T03:57:09.943 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.906+0000 7f949ffff640 10 monclient(hunting): handle_auth_done global_id 884146 payload 824 2024-09-16T03:57:09.943 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.906+0000 7f949ffff640 1 -- 172.21.15.42:0/3369392345 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f94a00a5750 con 0x7f94a013b210 2024-09-16T03:57:09.943 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.906+0000 7f949cff9640 10 client.824865.objecter ms_handle_connect 0x7f94a013b210 2024-09-16T03:57:09.943 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.906+0000 7f949cff9640 10 client.824865.objecter resend_mon_ops 2024-09-16T03:57:09.943 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.906+0000 7f949ffff640 1 --2- 172.21.15.42:0/3369392345 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f94a013b210 0x7f94a002c650 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f9490000f70 tx=0x7f949000ff70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:57:09.943 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.906+0000 7f949cff9640 10 client.824865.objecter ms_handle_connect 0x7f948c037810 2024-09-16T03:57:09.943 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.906+0000 7f949f7fe640 1 --2- 172.21.15.42:0/3369392345 >> v2:172.21.15.154:6812/254845342 conn(0x7f948c037810 0x7f948c039cd0 secure :-1 s=READY pgs=3674 cs=0 l=1 rev1=1 crypto rx=0x7f94a0131610 tx=0x7f9494009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:57:09.943 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.906+0000 7f949cff9640 1 -- 172.21.15.42:0/3369392345 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f949000db10 con 0x7f94a013b210 2024-09-16T03:57:09.943 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.906+0000 7f949cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:57:09.944 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.906+0000 7f949cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:57:09.944 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.906+0000 7f949cff9640 10 monclient: dump: 2024-09-16T03:57:09.944 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:57:09.944 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:57:09.944 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:57:09.944 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:57:09.944 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:57:09.944 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:57:09.944 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:57:09.944 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:57:09.944 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:57:09.944 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:57:09.944 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.906+0000 7f949cff9640 1 -- 172.21.15.42:0/3369392345 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f9490018070 con 0x7f94a013b210 2024-09-16T03:57:09.944 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.906+0000 7f949cff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:57:09.945 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.906+0000 7f949cff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:57:09.945 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.906+0000 7f949cff9640 1 -- 172.21.15.42:0/3369392345 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f94a013b210 msgr2=0x7f94a002c650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:57:09.945 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.906+0000 7f949cff9640 1 --2- 172.21.15.42:0/3369392345 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f94a013b210 0x7f94a002c650 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f9490000f70 tx=0x7f949000ff70 comp rx=0 tx=0).stop 2024-09-16T03:57:09.945 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.906+0000 7f949cff9640 1 -- 172.21.15.42:0/3369392345 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f94a013b210 msgr2=0x7f94a002c650 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:57:09.945 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.906+0000 7f949cff9640 1 --2- 172.21.15.42:0/3369392345 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f94a013b210 0x7f94a002c650 secure :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f9490000f70 tx=0x7f949000ff70 comp rx=0 tx=0).stop 2024-09-16T03:57:09.945 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.914+0000 7f94a6509640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:57:09.945 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.914+0000 7f94a6509640 1 --2- 172.21.15.42:0/3369392345 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f94a002cb90 0x7f94a002cf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:57:09.945 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.914+0000 7f94a6509640 10 monclient(hunting): start opening mon connection 2024-09-16T03:57:09.945 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.914+0000 7f94a6509640 1 -- 172.21.15.42:0/3369392345 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f94a01362c0 con 0x7f94a002cb90 2024-09-16T03:57:09.945 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.914+0000 7f949ffff640 1 --2- 172.21.15.42:0/3369392345 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f94a002cb90 0x7f94a002cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:57:09.945 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.914+0000 7f949ffff640 10 monclient: get_auth_request con 0x7f94a002cb90 auth_method 0 2024-09-16T03:57:09.945 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.914+0000 7f949ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:57:09.946 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.914+0000 7f949ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:57:09.946 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.914+0000 7f949ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:57:09.946 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.914+0000 7f949ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:57:09.946 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.914+0000 7f949ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:57:09.946 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.914+0000 7f949ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:57:09.946 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.914+0000 7f949ffff640 10 monclient(hunting): handle_auth_done global_id 884149 payload 824 2024-09-16T03:57:09.946 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.914+0000 7f949ffff640 1 -- 172.21.15.42:0/3369392345 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f94a013c550 con 0x7f94a002cb90 2024-09-16T03:57:09.946 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.914+0000 7f949ffff640 1 --2- 172.21.15.42:0/3369392345 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f94a002cb90 0x7f94a002cf70 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f9490017000 tx=0x7f9490001160 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:57:09.946 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.914+0000 7f949cff9640 10 client.824865.objecter ms_handle_connect 0x7f94a002cb90 2024-09-16T03:57:09.946 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.914+0000 7f949cff9640 10 client.824865.objecter resend_mon_ops 2024-09-16T03:57:09.946 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.914+0000 7f949cff9640 1 -- 172.21.15.42:0/3369392345 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94900073a0 con 0x7f94a002cb90 2024-09-16T03:57:09.946 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.914+0000 7f949cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:57:09.946 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.914+0000 7f949cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:57:09.946 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.914+0000 7f949cff9640 10 monclient: dump: 2024-09-16T03:57:09.947 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:57:09.947 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:57:09.947 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:57:09.947 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:57:09.947 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:57:09.947 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:57:09.947 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:57:09.947 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:57:09.947 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:57:09.947 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:57:09.947 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.918+0000 7f949cff9640 1 -- 172.21.15.42:0/3369392345 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4545 (secure 0 0 0) 0x7f94900073a0 con 0x7f94a002cb90 2024-09-16T03:57:09.947 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.918+0000 7f949cff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:57:09.947 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.918+0000 7f949cff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:57:09.947 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.918+0000 7f949cff9640 1 -- 172.21.15.42:0/3369392345 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f94a002cb90 msgr2=0x7f94a002cf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:57:09.948 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.918+0000 7f949cff9640 1 --2- 172.21.15.42:0/3369392345 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f94a002cb90 0x7f94a002cf70 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f9490017000 tx=0x7f9490001160 comp rx=0 tx=0).stop 2024-09-16T03:57:09.948 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.918+0000 7f949cff9640 1 -- 172.21.15.42:0/3369392345 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f94a002cb90 msgr2=0x7f94a002cf70 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:57:09.948 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.918+0000 7f949cff9640 1 --2- 172.21.15.42:0/3369392345 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f94a002cb90 0x7f94a002cf70 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:57:09.948 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:57:09.948 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:57:09.948 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:57:09.948 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T03:57:09.948 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 140, 2024-09-16T03:57:09.948 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 11455, 2024-09-16T03:57:09.949 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:57:09.949 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:57:09.949 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T03:57:09.949 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T03:57:09.949 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:57:09.949 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 3, 2024-09-16T03:57:09.949 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:57:09.949 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:57:09.949 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:57:09.949 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:57:09.949 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:57:09.949 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:57:09.949 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:57:09.949 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:57:09.950 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:57:09.950 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:57:09.950 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:57:09.950 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:57:09.950 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:57:09.950 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:57:09.950 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:57:09.950 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:57:09.950 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:57:09.950 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:57:09.950 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:57:09.950 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:57:09.950 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:57:09.950 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:57:09.951 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:57:09.951 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:57:09.951 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:57:09.951 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:57:09.951 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:57:09.951 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:57:09.951 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:57:09.951 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:57:09.951 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:57:09.951 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:57:09.951 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:57:09.951 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:57:09.951 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:57:09.951 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:57:09.952 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:57:09.952 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:57:09.952 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:57:09.952 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:57:09.952 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:57:09.952 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:57:09.952 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:57:09.952 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:57:09.952 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:57:09.952 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:57:09.953 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:57:09.953 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:57:09.953 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:57:09.953 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:57:09.953 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:57:09.953 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:57:09.953 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:57:09.953 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:57:09.953 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:57:09.953 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:57:09.953 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:57:09.953 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:57:09.953 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:57:09.954 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:57:09.954 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:57:09.954 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:57:09.954 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:57:09.954 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:57:09.954 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:57:09.954 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:57:09.954 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:57:09.954 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:57:09.954 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:57:09.954 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:57:09.954 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:57:09.954 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:57:09.955 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:57:09.955 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:57:09.955 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:57:09.955 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:57:09.955 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:57:09.955 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:57:09.955 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:57:09.955 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:57:09.955 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:57:09.955 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:57:09.955 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:57:09.955 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:57:09.955 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:57:09.955 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:57:09.956 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:57:09.956 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:57:09.956 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:57:09.956 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:57:09.956 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:57:09.956 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:57:09.956 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:57:09.956 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:57:09.956 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:57:09.956 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:57:09.956 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:57:09.956 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:57:09.956 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:57:09.956 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:57:09.957 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:57:09.957 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:57:09.957 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:57:09.957 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:57:09.957 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:57:09.957 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:57:09.957 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:57:09.957 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:57:09.957 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:57:09.957 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:57:09.957 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:57:09.957 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:57:09.957 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:57:09.957 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:57:09.958 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:57:09.958 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:57:09.958 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:57:09.958 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:57:09.958 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:57:09.958 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:57:09.958 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:57:09.958 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:57:09.958 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:57:09.958 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:57:09.958 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:57:09.958 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:57:09.958 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:57:09.958 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:57:09.958 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:57:09.959 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:57:09.959 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:57:09.959 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:57:09.959 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:57:09.959 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:57:09.959 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:57:09.959 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:57:09.959 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:57:09.959 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:57:09.959 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:57:09.959 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:57:09.959 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:57:09.959 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:57:09.959 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:57:09.960 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:57:09.960 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:57:09.960 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:57:09.960 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:57:09.960 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:57:09.960 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:57:09.960 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:57:09.960 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:57:09.962 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.918+0000 7f94a6509640 10 librados: watch_flush enter 2024-09-16T03:57:09.962 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.918+0000 7f94a6509640 10 librados: watch_flush exit 2024-09-16T03:57:09.962 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.918+0000 7f94a6509640 20 client.824865.objecter shutdown clearing up homeless session... 2024-09-16T03:57:09.962 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.918+0000 7f94a6509640 10 client.824865.objecter successfully canceled tick 2024-09-16T03:57:09.962 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.918+0000 7f94a6509640 1 -- 172.21.15.42:0/3369392345 >> v2:172.21.15.154:6812/254845342 conn(0x7f948c037810 msgr2=0x7f948c039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:57:09.962 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.918+0000 7f94a6509640 1 --2- 172.21.15.42:0/3369392345 >> v2:172.21.15.154:6812/254845342 conn(0x7f948c037810 0x7f948c039cd0 secure :-1 s=READY pgs=3674 cs=0 l=1 rev1=1 crypto rx=0x7f94a0131610 tx=0x7f9494009040 comp rx=0 tx=0).stop 2024-09-16T03:57:09.962 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.918+0000 7f94a6509640 10 monclient: shutdown 2024-09-16T03:57:09.962 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.918+0000 7f94a6509640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:57:09.962 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.918+0000 7f94a6509640 1 -- 172.21.15.42:0/3369392345 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f94a013daa0 msgr2=0x7f94a013a510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:57:09.962 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.918+0000 7f94a6509640 1 --2- 172.21.15.42:0/3369392345 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f94a013daa0 0x7f94a013a510 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f94980ac000 tx=0x7f94980195f0 comp rx=0 tx=0).stop 2024-09-16T03:57:09.962 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.918+0000 7f94a4aa6640 1 -- 172.21.15.42:0/3369392345 reap_dead start 2024-09-16T03:57:09.962 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.918+0000 7f94a6509640 1 -- 172.21.15.42:0/3369392345 shutdown_connections 2024-09-16T03:57:09.962 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.918+0000 7f94a6509640 1 --2- 172.21.15.42:0/3369392345 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f94a013daa0 0x7f94a013a510 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:57:09.963 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.918+0000 7f94a6509640 1 -- 172.21.15.42:0/3369392345 >> 172.21.15.42:0/3369392345 conn(0x7f94a00a9fb0 msgr2=0x7f94a00a8e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:57:09.963 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.918+0000 7f94a6509640 1 -- 172.21.15.42:0/3369392345 shutdown_connections 2024-09-16T03:57:09.963 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.918+0000 7f94a6509640 1 -- 172.21.15.42:0/3369392345 wait complete. 2024-09-16T03:57:09.963 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:09.918+0000 7f94a6509640 1 librados: shutdown 2024-09-16T03:57:09.963 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:57:09.963 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-09-16T03:57:09.963 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['c'] 2024-09-16T03:57:09.963 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-09-16T03:57:09.964 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-16T03:57:09.964 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-16T03:57:09.964 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T03:57:09.986 INFO:tasks.ceph.mon.b:Stopped 2024-09-16T03:57:09.986 INFO:tasks.mon_thrash:Sending STOP to mon c 2024-09-16T03:57:09.986 INFO:tasks.ceph.mon.c:Sent signal 19 2024-09-16T03:57:09.986 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-16T03:57:24.987 INFO:tasks.mon_thrash:Sending CONT to mon c 2024-09-16T03:57:24.988 INFO:tasks.ceph.mon.c:Sent signal 18 2024-09-16T03:57:24.988 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T03:57:24.989 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:57:25.119 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:25.114+0000 7f307ae3c640 10 monclient: get_monmap_and_config 2024-09-16T03:57:25.119 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:25.114+0000 7f307ae3c640 10 monclient: build_initial_monmap 2024-09-16T03:57:25.119 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:25.114+0000 7f307ae3c640 10 monclient: monmap: 2024-09-16T03:57:25.119 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:57:25.119 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:57:25.120 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:57:25.117929+0000 2024-09-16T03:57:25.120 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:57:25.117929+0000 2024-09-16T03:57:25.120 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:57:25.120 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:57:25.120 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:57:25.120 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:57:25.120 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:57:25.120 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:57:25.120 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:57:25.120 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:57:25.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:25.114+0000 7f307ae3c640 1 Processor -- start 2024-09-16T03:57:25.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:25.114+0000 7f307ae3c640 1 -- start start 2024-09-16T03:57:25.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:25.114+0000 7f307ae3c640 10 monclient: init 2024-09-16T03:57:25.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:25.114+0000 7f307ae3c640 10 monclient: _reopen_session rank -1 2024-09-16T03:57:25.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:25.114+0000 7f307ae3c640 10 monclient: _add_conns ranks=[1,0,4,3,2] 2024-09-16T03:57:25.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:25.114+0000 7f307ae3c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f30740a1330 addr v1:172.21.15.154:6789/0 2024-09-16T03:57:25.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:25.114+0000 7f307ae3c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f307406dac0 addr v1:172.21.15.42:6789/0 2024-09-16T03:57:25.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:25.114+0000 7f307ae3c640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f30740a1e20 0x7f307413da70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:57:25.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:25.114+0000 7f307ae3c640 10 monclient(hunting): picked mon.noname-c con 0x7f30740a1e20 addr v2:172.21.15.154:3300/0 2024-09-16T03:57:25.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:25.114+0000 7f307ae3c640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f307409c090 con 0x7f307406dac0 2024-09-16T03:57:25.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:25.114+0000 7f307ae3c640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f307409f610 con 0x7f30740a1330 2024-09-16T03:57:25.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:25.114+0000 7f307ae3c640 10 monclient(hunting): start opening mon connection 2024-09-16T03:57:25.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:25.114+0000 7f307ae3c640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f307409f8f0 con 0x7f30740a1e20 2024-09-16T03:57:25.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:25.114+0000 7f307ae3c640 10 monclient(hunting): _renew_subs 2024-09-16T03:57:25.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:25.114+0000 7f307ae3c640 10 monclient(hunting): authenticate will time out at 1892.087622s 2024-09-16T03:57:25.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:25.114+0000 7f3078bd8640 1 -- >> v1:172.21.15.154:6789/0 conn(0x7f30740a1330 legacy=0x7f30740a1710 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.154:6789/0 2024-09-16T03:57:25.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:25.114+0000 7f3073fff640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f307406dac0 0x7f307413a340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:33062/0 (socket says 172.21.15.42:33062) 2024-09-16T03:57:25.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:25.114+0000 7f3073fff640 1 -- 172.21.15.42:0/3149630460 learned_addr learned my addr 172.21.15.42:0/3149630460 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:57:25.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:25.114+0000 7f30793d9640 1 -- 172.21.15.42:0/3149630460 >> v2:172.21.15.154:3300/0 conn(0x7f30740a1e20 msgr2=0x7f307413da70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T03:57:25.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:25.114+0000 7f30793d9640 1 --2- 172.21.15.42:0/3149630460 >> v2:172.21.15.154:3300/0 conn(0x7f30740a1e20 0x7f307413da70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:57:25.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:25.318+0000 7f30793d9640 1 -- 172.21.15.42:0/3149630460 >> v2:172.21.15.154:3300/0 conn(0x7f30740a1e20 msgr2=0x7f307413da70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T03:57:25.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:25.318+0000 7f3078bd8640 1 -- 172.21.15.42:0/3149630460 >> v1:172.21.15.154:6789/0 conn(0x7f30740a1330 legacy=0x7f30740a1710 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.154:6789/0 2024-09-16T03:57:25.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:25.318+0000 7f30793d9640 1 --2- 172.21.15.42:0/3149630460 >> v2:172.21.15.154:3300/0 conn(0x7f30740a1e20 0x7f307413da70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T03:57:25.725 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:25.722+0000 7f30793d9640 1 -- 172.21.15.42:0/3149630460 >> v2:172.21.15.154:3300/0 conn(0x7f30740a1e20 msgr2=0x7f307413da70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T03:57:25.726 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:25.722+0000 7f3078bd8640 1 -- 172.21.15.42:0/3149630460 >> v1:172.21.15.154:6789/0 conn(0x7f30740a1330 legacy=0x7f30740a1710 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.154:6789/0 2024-09-16T03:57:25.726 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:25.722+0000 7f30793d9640 1 --2- 172.21.15.42:0/3149630460 >> v2:172.21.15.154:3300/0 conn(0x7f30740a1e20 0x7f307413da70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T03:57:26.530 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:26.526+0000 7f3078bd8640 1 -- 172.21.15.42:0/3149630460 >> v1:172.21.15.154:6789/0 conn(0x7f30740a1330 legacy=0x7f30740a1710 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.154:6789/0 2024-09-16T03:57:26.530 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:26.526+0000 7f30793d9640 1 -- 172.21.15.42:0/3149630460 >> v2:172.21.15.154:3300/0 conn(0x7f30740a1e20 msgr2=0x7f307413da70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T03:57:26.530 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:26.526+0000 7f30793d9640 1 --2- 172.21.15.42:0/3149630460 >> v2:172.21.15.154:3300/0 conn(0x7f30740a1e20 0x7f307413da70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T03:57:28.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:28.118+0000 7f30727fc640 10 monclient(hunting): tick 2024-09-16T03:57:28.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:28.118+0000 7f30727fc640 10 monclient(hunting): _check_auth_tickets 2024-09-16T03:57:28.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:28.118+0000 7f30727fc640 1 monclient(hunting): continuing hunt 2024-09-16T03:57:28.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:28.118+0000 7f30727fc640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T03:57:28.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:28.118+0000 7f30727fc640 1 -- 172.21.15.42:0/3149630460 >> v2:172.21.15.154:3300/0 conn(0x7f30740a1e20 msgr2=0x7f307413da70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:57:28.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:28.118+0000 7f30727fc640 1 --2- 172.21.15.42:0/3149630460 >> v2:172.21.15.154:3300/0 conn(0x7f30740a1e20 0x7f307413da70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:57:28.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:28.118+0000 7f30727fc640 1 -- 172.21.15.42:0/3149630460 >> v1:172.21.15.154:6789/0 conn(0x7f30740a1330 legacy=0x7f30740a1710 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:57:28.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:28.118+0000 7f30727fc640 1 -- 172.21.15.42:0/3149630460 >> v1:172.21.15.42:6789/0 conn(0x7f307406dac0 legacy=0x7f307413a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:57:28.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:28.118+0000 7f30727fc640 10 monclient: _add_conns ranks=[3,2] 2024-09-16T03:57:28.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:28.118+0000 7f30727fc640 1 --2- 172.21.15.42:0/3149630460 >> v2:172.21.15.42:3300/0 conn(0x7f30600036d0 0x7f3060005b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:57:28.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:28.118+0000 7f30727fc640 10 monclient(hunting): picked mon.noname-a con 0x7f30600036d0 addr v2:172.21.15.42:3300/0 2024-09-16T03:57:28.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:28.118+0000 7f30727fc640 1 --2- 172.21.15.42:0/3149630460 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3060006140 0x7f3060008600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:57:28.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:28.118+0000 7f30727fc640 10 monclient(hunting): picked mon.noname-b con 0x7f3060006140 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:57:28.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:28.118+0000 7f30727fc640 10 monclient(hunting): start opening mon connection 2024-09-16T03:57:28.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:28.118+0000 7f30727fc640 1 -- 172.21.15.42:0/3149630460 --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f307409f8f0 con 0x7f30600036d0 2024-09-16T03:57:28.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:28.118+0000 7f30727fc640 10 monclient(hunting): start opening mon connection 2024-09-16T03:57:28.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:28.118+0000 7f30727fc640 1 -- 172.21.15.42:0/3149630460 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f3060008d30 con 0x7f3060006140 2024-09-16T03:57:28.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:28.118+0000 7f30727fc640 10 monclient(hunting): _renew_subs 2024-09-16T03:57:28.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:28.118+0000 7f3078bd8640 1 --2- 172.21.15.42:0/3149630460 >> v2:172.21.15.42:3300/0 conn(0x7f30600036d0 0x7f3060005b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:57:28.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:28.118+0000 7f3073fff640 1 --2- 172.21.15.42:0/3149630460 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3060006140 0x7f3060008600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:57:28.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:28.118+0000 7f3078bd8640 10 monclient(hunting): get_auth_request con 0x7f30600036d0 auth_method 0 2024-09-16T03:57:28.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:28.118+0000 7f3078bd8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:57:28.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:28.118+0000 7f3078bd8640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:57:28.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:28.118+0000 7f3078bd8640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:57:28.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:28.118+0000 7f3073fff640 10 monclient(hunting): get_auth_request con 0x7f3060006140 auth_method 0 2024-09-16T03:57:28.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:28.118+0000 7f3073fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:57:28.125 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:28.118+0000 7f3073fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:57:28.125 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:28.118+0000 7f3073fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:57:28.125 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:28.118+0000 7f3078bd8640 1 -- 172.21.15.42:0/3149630460 >> v2:172.21.15.42:3300/0 conn(0x7f30600036d0 msgr2=0x7f3060005b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T03:57:28.125 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:28.118+0000 7f3078bd8640 1 -- 172.21.15.42:0/3149630460 >> v2:172.21.15.42:3300/0 conn(0x7f30600036d0 msgr2=0x7f3060005b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:57:28.125 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:28.118+0000 7f3078bd8640 1 --2- 172.21.15.42:0/3149630460 >> v2:172.21.15.42:3300/0 conn(0x7f30600036d0 0x7f3060005b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:57:28.125 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:28.118+0000 7f3073fff640 1 -- 172.21.15.42:0/3149630460 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3060006140 msgr2=0x7f3060008600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T03:57:28.125 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:28.118+0000 7f3073fff640 1 -- 172.21.15.42:0/3149630460 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3060006140 msgr2=0x7f3060008600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:57:28.125 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:28.118+0000 7f3078bd8640 1 --2- 172.21.15.42:0/3149630460 >> v2:172.21.15.42:3300/0 conn(0x7f30600036d0 0x7f3060005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:57:28.125 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:28.118+0000 7f3073fff640 1 --2- 172.21.15.42:0/3149630460 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3060006140 0x7f3060008600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:57:28.125 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:28.118+0000 7f3073fff640 1 --2- 172.21.15.42:0/3149630460 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3060006140 0x7f3060008600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:57:28.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:28.322+0000 7f3073fff640 1 --2- 172.21.15.42:0/3149630460 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3060006140 0x7f3060008600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:57:28.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:28.322+0000 7f3073fff640 10 monclient(hunting): get_auth_request con 0x7f3060006140 auth_method 0 2024-09-16T03:57:28.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:28.322+0000 7f3073fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:57:28.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:28.322+0000 7f3073fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:57:28.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:28.322+0000 7f3073fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:57:28.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:28.322+0000 7f3078bd8640 1 --2- 172.21.15.42:0/3149630460 >> v2:172.21.15.42:3300/0 conn(0x7f30600036d0 0x7f3060005b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:57:28.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:28.322+0000 7f3078bd8640 10 monclient(hunting): get_auth_request con 0x7f30600036d0 auth_method 0 2024-09-16T03:57:28.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:28.322+0000 7f3078bd8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:57:28.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:28.322+0000 7f3078bd8640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:57:28.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:28.322+0000 7f3078bd8640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:57:28.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:28.322+0000 7f3073fff640 1 -- 172.21.15.42:0/3149630460 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3060006140 msgr2=0x7f3060008600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T03:57:28.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:28.322+0000 7f3073fff640 1 -- 172.21.15.42:0/3149630460 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3060006140 msgr2=0x7f3060008600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:57:28.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:28.322+0000 7f3073fff640 1 --2- 172.21.15.42:0/3149630460 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3060006140 0x7f3060008600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:57:28.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:28.322+0000 7f3073fff640 1 --2- 172.21.15.42:0/3149630460 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3060006140 0x7f3060008600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T03:57:28.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:28.322+0000 7f3078bd8640 1 -- 172.21.15.42:0/3149630460 >> v2:172.21.15.42:3300/0 conn(0x7f30600036d0 msgr2=0x7f3060005b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T03:57:28.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:28.322+0000 7f3078bd8640 1 -- 172.21.15.42:0/3149630460 >> v2:172.21.15.42:3300/0 conn(0x7f30600036d0 msgr2=0x7f3060005b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:57:28.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:28.322+0000 7f3078bd8640 1 --2- 172.21.15.42:0/3149630460 >> v2:172.21.15.42:3300/0 conn(0x7f30600036d0 0x7f3060005b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:57:28.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:28.322+0000 7f3078bd8640 1 --2- 172.21.15.42:0/3149630460 >> v2:172.21.15.42:3300/0 conn(0x7f30600036d0 0x7f3060005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T03:57:28.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:28.726+0000 7f3078bd8640 1 --2- 172.21.15.42:0/3149630460 >> v2:172.21.15.42:3300/0 conn(0x7f30600036d0 0x7f3060005b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:57:28.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:28.726+0000 7f3073fff640 1 --2- 172.21.15.42:0/3149630460 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3060006140 0x7f3060008600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:57:28.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:28.726+0000 7f3078bd8640 10 monclient(hunting): get_auth_request con 0x7f30600036d0 auth_method 0 2024-09-16T03:57:28.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:28.726+0000 7f3078bd8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:57:28.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:28.726+0000 7f3078bd8640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:57:28.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:28.726+0000 7f3078bd8640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:57:28.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:28.726+0000 7f3073fff640 10 monclient(hunting): get_auth_request con 0x7f3060006140 auth_method 0 2024-09-16T03:57:28.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:28.726+0000 7f3073fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:57:28.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:28.726+0000 7f3073fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:57:28.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:28.726+0000 7f3073fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:57:28.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:28.726+0000 7f3078bd8640 1 -- 172.21.15.42:0/3149630460 >> v2:172.21.15.42:3300/0 conn(0x7f30600036d0 msgr2=0x7f3060005b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T03:57:28.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:28.726+0000 7f3078bd8640 1 -- 172.21.15.42:0/3149630460 >> v2:172.21.15.42:3300/0 conn(0x7f30600036d0 msgr2=0x7f3060005b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:57:28.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:28.726+0000 7f3078bd8640 1 --2- 172.21.15.42:0/3149630460 >> v2:172.21.15.42:3300/0 conn(0x7f30600036d0 0x7f3060005b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:57:28.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:28.726+0000 7f3073fff640 1 -- 172.21.15.42:0/3149630460 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3060006140 msgr2=0x7f3060008600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T03:57:28.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:28.726+0000 7f3073fff640 1 -- 172.21.15.42:0/3149630460 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3060006140 msgr2=0x7f3060008600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:57:28.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:28.726+0000 7f3073fff640 1 --2- 172.21.15.42:0/3149630460 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3060006140 0x7f3060008600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:57:28.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:28.726+0000 7f3078bd8640 1 --2- 172.21.15.42:0/3149630460 >> v2:172.21.15.42:3300/0 conn(0x7f30600036d0 0x7f3060005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T03:57:28.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:28.726+0000 7f3073fff640 1 --2- 172.21.15.42:0/3149630460 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3060006140 0x7f3060008600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T03:57:29.534 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:29.530+0000 7f3073fff640 1 --2- 172.21.15.42:0/3149630460 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3060006140 0x7f3060008600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:57:29.534 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:29.530+0000 7f3078bd8640 1 --2- 172.21.15.42:0/3149630460 >> v2:172.21.15.42:3300/0 conn(0x7f30600036d0 0x7f3060005b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:57:29.534 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:29.530+0000 7f3073fff640 10 monclient(hunting): get_auth_request con 0x7f3060006140 auth_method 0 2024-09-16T03:57:29.534 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:29.530+0000 7f3073fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:57:29.534 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:29.530+0000 7f3073fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:57:29.534 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:29.530+0000 7f3073fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:57:29.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:29.530+0000 7f3078bd8640 10 monclient(hunting): get_auth_request con 0x7f30600036d0 auth_method 0 2024-09-16T03:57:29.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:29.530+0000 7f3078bd8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:57:29.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:29.530+0000 7f3078bd8640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:57:29.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:29.530+0000 7f3078bd8640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:57:29.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:29.530+0000 7f3073fff640 1 -- 172.21.15.42:0/3149630460 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3060006140 msgr2=0x7f3060008600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T03:57:29.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:29.530+0000 7f3073fff640 1 -- 172.21.15.42:0/3149630460 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3060006140 msgr2=0x7f3060008600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:57:29.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:29.530+0000 7f3078bd8640 1 -- 172.21.15.42:0/3149630460 >> v2:172.21.15.42:3300/0 conn(0x7f30600036d0 msgr2=0x7f3060005b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T03:57:29.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:29.530+0000 7f3073fff640 1 --2- 172.21.15.42:0/3149630460 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3060006140 0x7f3060008600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:57:29.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:29.530+0000 7f3078bd8640 1 -- 172.21.15.42:0/3149630460 >> v2:172.21.15.42:3300/0 conn(0x7f30600036d0 msgr2=0x7f3060005b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:57:29.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:29.530+0000 7f3078bd8640 1 --2- 172.21.15.42:0/3149630460 >> v2:172.21.15.42:3300/0 conn(0x7f30600036d0 0x7f3060005b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:57:29.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:29.530+0000 7f3078bd8640 1 --2- 172.21.15.42:0/3149630460 >> v2:172.21.15.42:3300/0 conn(0x7f30600036d0 0x7f3060005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T03:57:29.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:29.530+0000 7f3073fff640 1 --2- 172.21.15.42:0/3149630460 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3060006140 0x7f3060008600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T03:57:31.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.118+0000 7f30727fc640 10 monclient(hunting): tick 2024-09-16T03:57:31.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.118+0000 7f30727fc640 10 monclient(hunting): _check_auth_tickets 2024-09-16T03:57:31.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.118+0000 7f30727fc640 1 monclient(hunting): continuing hunt 2024-09-16T03:57:31.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.118+0000 7f30727fc640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T03:57:31.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.118+0000 7f30727fc640 1 -- 172.21.15.42:0/3149630460 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3060006140 msgr2=0x7f3060008600 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:57:31.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.118+0000 7f30727fc640 1 --2- 172.21.15.42:0/3149630460 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3060006140 0x7f3060008600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:57:31.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.118+0000 7f30727fc640 1 -- 172.21.15.42:0/3149630460 >> v2:172.21.15.42:3300/0 conn(0x7f30600036d0 msgr2=0x7f3060005b50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:57:31.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.118+0000 7f30727fc640 1 --2- 172.21.15.42:0/3149630460 >> v2:172.21.15.42:3300/0 conn(0x7f30600036d0 0x7f3060005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:57:31.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.118+0000 7f30793d9640 1 -- 172.21.15.42:0/3149630460 reap_dead start 2024-09-16T03:57:31.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.118+0000 7f30727fc640 10 monclient: _add_conns ranks=[3,4,0,1,2] 2024-09-16T03:57:31.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.118+0000 7f30727fc640 1 --2- 172.21.15.42:0/3149630460 >> v2:172.21.15.42:3300/0 conn(0x7f3060003b40 0x7f30600065f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:57:31.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.118+0000 7f30727fc640 10 monclient(hunting): picked mon.noname-a con 0x7f3060003b40 addr v2:172.21.15.42:3300/0 2024-09-16T03:57:31.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.118+0000 7f30727fc640 1 --2- 172.21.15.42:0/3149630460 >> v2:172.21.15.154:3300/0 conn(0x7f3060006b30 0x7f3060006f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:57:31.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.118+0000 7f30727fc640 10 monclient(hunting): picked mon.noname-c con 0x7f3060006b30 addr v2:172.21.15.154:3300/0 2024-09-16T03:57:31.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.118+0000 7f30727fc640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f30600074a0 addr v1:172.21.15.42:6789/0 2024-09-16T03:57:31.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.118+0000 7f30727fc640 1 -- 172.21.15.42:0/3149630460 --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f307409fbd0 con 0x7f30600074a0 2024-09-16T03:57:31.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.118+0000 7f30727fc640 10 monclient(hunting): start opening mon connection 2024-09-16T03:57:31.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.118+0000 7f30727fc640 1 -- 172.21.15.42:0/3149630460 --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f307409f8f0 con 0x7f3060003b40 2024-09-16T03:57:31.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.118+0000 7f30727fc640 10 monclient(hunting): start opening mon connection 2024-09-16T03:57:31.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.118+0000 7f30727fc640 1 -- 172.21.15.42:0/3149630460 --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f3060008d30 con 0x7f3060006b30 2024-09-16T03:57:31.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.118+0000 7f30727fc640 10 monclient(hunting): _renew_subs 2024-09-16T03:57:31.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.118+0000 7f3073fff640 1 -- 172.21.15.42:0/3149630460 >> v2:172.21.15.154:3300/0 conn(0x7f3060006b30 msgr2=0x7f3060006f30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T03:57:31.125 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.118+0000 7f3073fff640 1 --2- 172.21.15.42:0/3149630460 >> v2:172.21.15.154:3300/0 conn(0x7f3060006b30 0x7f3060006f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:57:31.125 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.118+0000 7f3078bd8640 1 --2- 172.21.15.42:0/3149630460 >> v2:172.21.15.42:3300/0 conn(0x7f3060003b40 0x7f30600065f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:57:31.125 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.118+0000 7f3078bd8640 10 monclient(hunting): get_auth_request con 0x7f3060003b40 auth_method 0 2024-09-16T03:57:31.125 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.118+0000 7f3078bd8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:57:31.126 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.118+0000 7f3078bd8640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:57:31.126 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.118+0000 7f3078bd8640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:57:31.126 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.118+0000 7f3078bd8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:57:31.126 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.118+0000 7f3078bd8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:57:31.126 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.118+0000 7f3078bd8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:57:31.126 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.118+0000 7f3078bd8640 10 monclient(hunting): handle_auth_done global_id 824868 payload 274 2024-09-16T03:57:31.126 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.118+0000 7f3078bd8640 1 -- 172.21.15.42:0/3149630460 >> v2:172.21.15.154:3300/0 conn(0x7f3060006b30 msgr2=0x7f3060006f30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:57:31.126 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.118+0000 7f3078bd8640 1 --2- 172.21.15.42:0/3149630460 >> v2:172.21.15.154:3300/0 conn(0x7f3060006b30 0x7f3060006f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:57:31.126 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.118+0000 7f3078bd8640 1 -- 172.21.15.42:0/3149630460 >> v1:172.21.15.42:6789/0 conn(0x7f30600074a0 legacy=0x7f3060007950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:57:31.126 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.118+0000 7f3078bd8640 10 monclient: _finish_hunting 0 2024-09-16T03:57:31.126 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.118+0000 7f3078bd8640 1 monclient: found mon.noname-a 2024-09-16T03:57:31.126 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.118+0000 7f3078bd8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:57:31.126 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.118+0000 7f3078bd8640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:57:31.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.118+0000 7f3078bd8640 1 -- 172.21.15.42:0/3149630460 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f307409f610 con 0x7f3060003b40 2024-09-16T03:57:31.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.118+0000 7f30737fe640 1 -- 172.21.15.42:0/3149630460 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3875766241 0 0) 0x7f307409fbd0 con 0x7f30600074a0 2024-09-16T03:57:31.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.118+0000 7f30737fe640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T03:57:31.128 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.122+0000 7f3078bd8640 1 --2- 172.21.15.42:0/3149630460 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3060003b40 0x7f30600065f0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f3060008c00 tx=0x7f306401d4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88ea4c5e78d0c6cd server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:57:31.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.122+0000 7f30737fe640 1 -- 172.21.15.42:0/3149630460 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30640ad460 con 0x7f3060003b40 2024-09-16T03:57:31.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.122+0000 7f30737fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:57:31.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.122+0000 7f30737fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:57:31.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.122+0000 7f30737fe640 10 monclient: dump: 2024-09-16T03:57:31.129 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:57:31.129 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:57:31.129 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:57:31.129 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:57:31.129 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:57:31.129 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:57:31.130 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:57:31.130 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:57:31.130 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:57:31.130 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:57:31.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.122+0000 7f30737fe640 10 monclient: _finish_auth 0 2024-09-16T03:57:31.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.122+0000 7f30737fe640 10 monclient: _check_auth_tickets 2024-09-16T03:57:31.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.122+0000 7f30737fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:57:31.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.122+0000 7f30737fe640 1 -- 172.21.15.42:0/3149630460 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f30640b2070 con 0x7f3060003b40 2024-09-16T03:57:31.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.122+0000 7f307ae3c640 5 monclient: authenticate success, global_id 824868 2024-09-16T03:57:31.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.122+0000 7f307ae3c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:57:31.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.122+0000 7f30737fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:57:31.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.122+0000 7f307ae3c640 10 monclient: get_monmap_and_config success 2024-09-16T03:57:31.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.122+0000 7f307ae3c640 4 set_mon_vals no callback set 2024-09-16T03:57:31.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.122+0000 7f30737fe640 1 -- 172.21.15.42:0/3149630460 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30640ad820 con 0x7f3060003b40 2024-09-16T03:57:31.131 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.122+0000 7f30737fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:57:31.131 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.122+0000 7f30737fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:57:31.131 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.122+0000 7f30737fe640 10 monclient: dump: 2024-09-16T03:57:31.131 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:57:31.131 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:57:31.131 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:57:31.131 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:57:31.131 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:57:31.131 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:57:31.132 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:57:31.132 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:57:31.132 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:57:31.132 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:57:31.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.126+0000 7f307ae3c640 10 monclient: shutdown 2024-09-16T03:57:31.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.126+0000 7f307ae3c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:57:31.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.126+0000 7f307ae3c640 1 -- 172.21.15.42:0/3149630460 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3060003b40 msgr2=0x7f30600065f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:57:31.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.126+0000 7f307ae3c640 1 --2- 172.21.15.42:0/3149630460 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3060003b40 0x7f30600065f0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f3060008c00 tx=0x7f306401d4b0 comp rx=0 tx=0).stop 2024-09-16T03:57:31.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.126+0000 7f307ae3c640 1 -- 172.21.15.42:0/3149630460 shutdown_connections 2024-09-16T03:57:31.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.126+0000 7f307ae3c640 1 --2- 172.21.15.42:0/3149630460 >> v2:172.21.15.154:3300/0 conn(0x7f3060006b30 0x7f3060006f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:57:31.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.126+0000 7f307ae3c640 1 --2- 172.21.15.42:0/3149630460 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3060003b40 0x7f30600065f0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:57:31.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.126+0000 7f307ae3c640 1 -- 172.21.15.42:0/3149630460 >> 172.21.15.42:0/3149630460 conn(0x7f30740a2420 msgr2=0x7f30740a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:57:31.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.126+0000 7f307ae3c640 1 -- 172.21.15.42:0/3149630460 shutdown_connections 2024-09-16T03:57:31.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.126+0000 7f307ae3c640 1 -- 172.21.15.42:0/3149630460 wait complete. 2024-09-16T03:57:31.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.126+0000 7f307ae3c640 10 monclient: build_initial_monmap 2024-09-16T03:57:31.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.126+0000 7f307ae3c640 10 monclient: monmap: 2024-09-16T03:57:31.139 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:57:31.139 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:57:31.139 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:57:31.139 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:57:31.139 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:57:31.139 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:57:31.139 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:57:31.139 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:57:31.139 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:57:31.140 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:57:31.140 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.126+0000 7f307ae3c640 1 librados: starting msgr at 2024-09-16T03:57:31.140 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.126+0000 7f307ae3c640 1 librados: starting objecter 2024-09-16T03:57:31.140 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.126+0000 7f307ae3c640 1 Processor -- start 2024-09-16T03:57:31.140 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.126+0000 7f307ae3c640 1 -- start start 2024-09-16T03:57:31.140 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.126+0000 7f307ae3c640 1 librados: setting wanted keys 2024-09-16T03:57:31.140 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.126+0000 7f307ae3c640 1 librados: calling monclient init 2024-09-16T03:57:31.140 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.126+0000 7f307ae3c640 10 monclient: init 2024-09-16T03:57:31.140 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.126+0000 7f307ae3c640 10 monclient: _reopen_session rank -1 2024-09-16T03:57:31.140 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.126+0000 7f307ae3c640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:57:31.140 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.126+0000 7f307ae3c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3060003b40 0x7f30741cc3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:57:31.140 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.126+0000 7f307ae3c640 10 monclient(hunting): picked mon.noname-b con 0x7f3060003b40 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:57:31.140 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.126+0000 7f307ae3c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3060006b30 0x7f30741cc8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:57:31.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.126+0000 7f307ae3c640 10 monclient(hunting): picked mon.noname-a con 0x7f3060006b30 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:57:31.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.126+0000 7f307ae3c640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f30600074a0 0x7f3074123c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:57:31.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.126+0000 7f307ae3c640 10 monclient(hunting): picked mon.noname-c con 0x7f30600074a0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:57:31.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.126+0000 7f307ae3c640 10 monclient(hunting): start opening mon connection 2024-09-16T03:57:31.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.126+0000 7f307ae3c640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f307409eef0 con 0x7f3060006b30 2024-09-16T03:57:31.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.126+0000 7f307ae3c640 10 monclient(hunting): start opening mon connection 2024-09-16T03:57:31.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.126+0000 7f307ae3c640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f307409ed70 con 0x7f3060003b40 2024-09-16T03:57:31.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.126+0000 7f307ae3c640 10 monclient(hunting): start opening mon connection 2024-09-16T03:57:31.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.126+0000 7f307ae3c640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f30740a1710 con 0x7f30600074a0 2024-09-16T03:57:31.145 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.126+0000 7f307ae3c640 10 monclient(hunting): _renew_subs 2024-09-16T03:57:31.145 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.126+0000 7f307ae3c640 10 monclient(hunting): authenticate will time out at 1898.099819s 2024-09-16T03:57:31.145 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.126+0000 7f3078bd8640 1 -- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3060003b40 msgr2=0x7f30741cc3a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T03:57:31.145 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.126+0000 7f3078bd8640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3060003b40 0x7f30741cc3a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:57:31.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.126+0000 7f3073fff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3060006b30 0x7f30741cc8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:57:31.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.126+0000 7f30793d9640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f30600074a0 0x7f3074123c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:57:31.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.130+0000 7f3073fff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3060006b30 0x7f30741cc8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:36372/0 (socket says 172.21.15.42:36372) 2024-09-16T03:57:31.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.130+0000 7f3073fff640 1 -- 172.21.15.42:0/674415286 learned_addr learned my addr 172.21.15.42:0/674415286 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:57:31.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.130+0000 7f3073fff640 10 monclient(hunting): get_auth_request con 0x7f3060006b30 auth_method 0 2024-09-16T03:57:31.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.130+0000 7f3073fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:57:31.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.130+0000 7f3073fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:57:31.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.130+0000 7f3073fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:57:31.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.130+0000 7f30793d9640 10 monclient(hunting): get_auth_request con 0x7f30600074a0 auth_method 0 2024-09-16T03:57:31.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.130+0000 7f30793d9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:57:31.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.130+0000 7f30793d9640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:57:31.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.130+0000 7f30793d9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:57:31.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.130+0000 7f3073fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:57:31.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.130+0000 7f3073fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:57:31.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.130+0000 7f3073fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:57:31.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.130+0000 7f30793d9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:57:31.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.130+0000 7f30793d9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:57:31.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.130+0000 7f30793d9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:57:31.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.130+0000 7f3073fff640 10 monclient(hunting): handle_auth_done global_id 824874 payload 824 2024-09-16T03:57:31.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.130+0000 7f3073fff640 1 -- 172.21.15.42:0/674415286 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f30600074a0 msgr2=0x7f3074123c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:57:31.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.130+0000 7f3073fff640 1 --2- 172.21.15.42:0/674415286 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f30600074a0 0x7f3074123c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:57:31.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.130+0000 7f3073fff640 1 -- 172.21.15.42:0/674415286 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3060003b40 msgr2=0x7f30741cc3a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:57:31.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.130+0000 7f3073fff640 1 --2- 172.21.15.42:0/674415286 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3060003b40 0x7f30741cc3a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:57:31.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.130+0000 7f3073fff640 10 monclient: _finish_hunting 0 2024-09-16T03:57:31.148 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.130+0000 7f3073fff640 1 monclient: found mon.noname-a 2024-09-16T03:57:31.148 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.130+0000 7f3073fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:57:31.148 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.130+0000 7f3073fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:57:31.148 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.130+0000 7f3073fff640 1 -- 172.21.15.42:0/674415286 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30741278f0 con 0x7f3060006b30 2024-09-16T03:57:31.148 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.130+0000 7f30793d9640 1 --2- 172.21.15.42:0/674415286 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f30600074a0 0x7f3074123c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:57:31.148 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.130+0000 7f3073fff640 1 --2- 172.21.15.42:0/674415286 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3060006b30 0x7f30741cc8e0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f30680076c0 tx=0x7f3068001640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:57:31.148 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.130+0000 7f30717fa640 10 client.?.objecter ms_handle_connect 0x7f3060006b30 2024-09-16T03:57:31.148 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.130+0000 7f30717fa640 10 client.?.objecter resend_mon_ops 2024-09-16T03:57:31.148 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.130+0000 7f30717fa640 1 -- 172.21.15.42:0/674415286 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3068004a00 con 0x7f3060006b30 2024-09-16T03:57:31.148 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.130+0000 7f30717fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:57:31.148 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.130+0000 7f30717fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:57:31.148 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.130+0000 7f30717fa640 10 monclient: dump: 2024-09-16T03:57:31.149 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:57:31.149 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:57:31.149 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:57:31.149 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:57:31.149 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:57:31.149 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:57:31.149 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:57:31.149 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:57:31.149 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:57:31.149 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:57:31.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.130+0000 7f30717fa640 10 monclient: _finish_auth 0 2024-09-16T03:57:31.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.130+0000 7f30717fa640 10 monclient: _check_auth_tickets 2024-09-16T03:57:31.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.130+0000 7f30717fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:57:31.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.130+0000 7f307ae3c640 5 monclient: authenticate success, global_id 824874 2024-09-16T03:57:31.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.130+0000 7f307ae3c640 10 monclient: _renew_subs 2024-09-16T03:57:31.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.130+0000 7f307ae3c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:57:31.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.130+0000 7f307ae3c640 1 -- 172.21.15.42:0/674415286 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3074127bd0 con 0x7f3060006b30 2024-09-16T03:57:31.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.130+0000 7f30717fa640 1 -- 172.21.15.42:0/674415286 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3068004ba0 con 0x7f3060006b30 2024-09-16T03:57:31.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.130+0000 7f30717fa640 10 monclient: handle_config config(0 keys) 2024-09-16T03:57:31.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.130+0000 7f30717fa640 1 -- 172.21.15.42:0/674415286 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3068005050 con 0x7f3060006b30 2024-09-16T03:57:31.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.130+0000 7f30717fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:57:31.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.130+0000 7f30717fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:57:31.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.130+0000 7f30717fa640 10 monclient: dump: 2024-09-16T03:57:31.150 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:57:31.150 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:57:31.150 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:57:31.150 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:57:31.151 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:57:31.151 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:57:31.151 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:57:31.151 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:57:31.151 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:57:31.151 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:57:31.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.130+0000 7f30727fc640 4 set_mon_vals no callback set 2024-09-16T03:57:31.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.130+0000 7f307ae3c640 10 client.824874.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:57:31.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.130+0000 7f307ae3c640 10 monclient: _renew_subs 2024-09-16T03:57:31.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.130+0000 7f307ae3c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:57:31.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.130+0000 7f307ae3c640 1 -- 172.21.15.42:0/674415286 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3074127610 con 0x7f3060006b30 2024-09-16T03:57:31.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.130+0000 7f307ae3c640 1 librados: init done 2024-09-16T03:57:31.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.130+0000 7f30717fa640 1 -- 172.21.15.42:0/674415286 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f30680051f0 con 0x7f3060006b30 2024-09-16T03:57:31.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.130+0000 7f307ae3c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:57:31.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.130+0000 7f307ae3c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:57:31.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.130+0000 7f307ae3c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:57:31.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.130+0000 7f307ae3c640 1 -- 172.21.15.42:0/674415286 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f30740a1aa0 con 0x7f3060006b30 2024-09-16T03:57:31.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.130+0000 7f30717fa640 1 --2- 172.21.15.42:0/674415286 >> v2:172.21.15.154:6812/254845342 conn(0x7f306003c320 0x7f306003e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:57:31.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.130+0000 7f30717fa640 1 -- 172.21.15.42:0/674415286 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(461..461 src has 330..461) ==== 5714+0+0 (secure 0 0 0) 0x7f30680498e0 con 0x7f3060006b30 2024-09-16T03:57:31.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.130+0000 7f30717fa640 10 client.824874.objecter ms_dispatch 0x7f3074000ba0 osd_map(461..461 src has 330..461) 2024-09-16T03:57:31.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.130+0000 7f30717fa640 3 client.824874.objecter handle_osd_map got epochs [461,461] > 0 2024-09-16T03:57:31.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.130+0000 7f30717fa640 3 client.824874.objecter handle_osd_map decoding full epoch 461 2024-09-16T03:57:31.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.130+0000 7f30717fa640 20 client.824874.objecter dump_active .. 0 homeless 2024-09-16T03:57:31.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.130+0000 7f3078bd8640 1 --2- 172.21.15.42:0/674415286 >> v2:172.21.15.154:6812/254845342 conn(0x7f306003c320 0x7f306003e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:57:31.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.130+0000 7f3078bd8640 10 monclient: get_auth_request con 0x7f306003c320 auth_method 0 2024-09-16T03:57:31.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.134+0000 7f30717fa640 10 client.824874.objecter ms_handle_connect 0x7f306003c320 2024-09-16T03:57:31.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.134+0000 7f3078bd8640 1 --2- 172.21.15.42:0/674415286 >> v2:172.21.15.154:6812/254845342 conn(0x7f306003c320 0x7f306003e7e0 secure :-1 s=READY pgs=3681 cs=0 l=1 rev1=1 crypto rx=0x7f30640b1000 tx=0x7f3064005040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:57:31.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.134+0000 7f30717fa640 1 -- 172.21.15.42:0/674415286 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f30680163c0 con 0x7f3060006b30 2024-09-16T03:57:31.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.134+0000 7f30717fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:57:31.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.134+0000 7f30717fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:57:31.368 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.362+0000 7f307ae3c640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:57:31.368 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.362+0000 7f307ae3c640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:57:31.369 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.362+0000 7f307ae3c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:57:31.369 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.362+0000 7f307ae3c640 1 -- 172.21.15.42:0/674415286 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f30741261c0 con 0x7f3060006b30 2024-09-16T03:57:31.369 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.362+0000 7f30717fa640 1 -- 172.21.15.42:0/674415286 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1555 (secure 0 0 0) 0x7f3068017e20 con 0x7f3060006b30 2024-09-16T03:57:31.369 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.366+0000 7f30717fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:57:31.369 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.366+0000 7f30717fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:57:31.370 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":142,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:57:31.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.366+0000 7f307ae3c640 10 librados: watch_flush enter 2024-09-16T03:57:31.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.366+0000 7f307ae3c640 10 librados: watch_flush exit 2024-09-16T03:57:31.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.366+0000 7f307ae3c640 20 client.824874.objecter shutdown clearing up homeless session... 2024-09-16T03:57:31.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.366+0000 7f307ae3c640 10 client.824874.objecter successfully canceled tick 2024-09-16T03:57:31.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.366+0000 7f307ae3c640 1 -- 172.21.15.42:0/674415286 >> v2:172.21.15.154:6812/254845342 conn(0x7f306003c320 msgr2=0x7f306003e7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:57:31.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.366+0000 7f307ae3c640 1 --2- 172.21.15.42:0/674415286 >> v2:172.21.15.154:6812/254845342 conn(0x7f306003c320 0x7f306003e7e0 secure :-1 s=READY pgs=3681 cs=0 l=1 rev1=1 crypto rx=0x7f30640b1000 tx=0x7f3064005040 comp rx=0 tx=0).stop 2024-09-16T03:57:31.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.366+0000 7f307ae3c640 10 monclient: shutdown 2024-09-16T03:57:31.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.366+0000 7f307ae3c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:57:31.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.366+0000 7f307ae3c640 1 -- 172.21.15.42:0/674415286 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3060006b30 msgr2=0x7f30741cc8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:57:31.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.366+0000 7f307ae3c640 1 --2- 172.21.15.42:0/674415286 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3060006b30 0x7f30741cc8e0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f30680076c0 tx=0x7f3068001640 comp rx=0 tx=0).stop 2024-09-16T03:57:31.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.366+0000 7f307ae3c640 1 -- 172.21.15.42:0/674415286 shutdown_connections 2024-09-16T03:57:31.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.366+0000 7f307ae3c640 1 --2- 172.21.15.42:0/674415286 >> v2:172.21.15.154:6812/254845342 conn(0x7f306003c320 0x7f306003e7e0 unknown :-1 s=CLOSED pgs=3681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:57:31.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.366+0000 7f307ae3c640 1 --2- 172.21.15.42:0/674415286 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f30600074a0 0x7f3074123c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:57:31.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.366+0000 7f307ae3c640 1 --2- 172.21.15.42:0/674415286 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3060006b30 0x7f30741cc8e0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:57:31.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.366+0000 7f307ae3c640 1 --2- 172.21.15.42:0/674415286 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3060003b40 0x7f30741cc3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:57:31.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.366+0000 7f307ae3c640 1 -- 172.21.15.42:0/674415286 >> 172.21.15.42:0/674415286 conn(0x7f30740a2420 msgr2=0x7f30740a45b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:57:31.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.366+0000 7f307ae3c640 1 -- 172.21.15.42:0/674415286 shutdown_connections 2024-09-16T03:57:31.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.366+0000 7f307ae3c640 1 -- 172.21.15.42:0/674415286 wait complete. 2024-09-16T03:57:31.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.366+0000 7f307ae3c640 1 librados: shutdown 2024-09-16T03:57:31.389 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T03:57:31.389 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-16T03:57:31.389 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T03:57:31.514 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.510+0000 7fa59f23e640 10 monclient: get_monmap_and_config 2024-09-16T03:57:31.514 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.510+0000 7fa59f23e640 10 monclient: build_initial_monmap 2024-09-16T03:57:31.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.510+0000 7fa59f23e640 10 monclient: monmap: 2024-09-16T03:57:31.515 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:57:31.515 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:57:31.515 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:57:31.513269+0000 2024-09-16T03:57:31.515 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:57:31.513269+0000 2024-09-16T03:57:31.515 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:57:31.515 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:57:31.515 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:57:31.515 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:57:31.515 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:57:31.515 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:57:31.515 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:57:31.515 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:57:31.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.510+0000 7fa59f23e640 1 Processor -- start 2024-09-16T03:57:31.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.510+0000 7fa59f23e640 1 -- start start 2024-09-16T03:57:31.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.510+0000 7fa59f23e640 10 monclient: init 2024-09-16T03:57:31.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.510+0000 7fa59f23e640 10 monclient: _reopen_session rank -1 2024-09-16T03:57:31.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.510+0000 7fa59f23e640 10 monclient: _add_conns ranks=[3,2,1,4,0] 2024-09-16T03:57:31.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.510+0000 7fa59f23e640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fa5980a1330 0x7fa5980a1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:57:31.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.510+0000 7fa59f23e640 10 monclient(hunting): picked mon.noname-a con 0x7fa5980a1330 addr v2:172.21.15.42:3300/0 2024-09-16T03:57:31.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.510+0000 7fa59f23e640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa59806dac0 0x7fa5980a1c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:57:31.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.510+0000 7fa59f23e640 10 monclient(hunting): picked mon.noname-b con 0x7fa59806dac0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:57:31.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.510+0000 7fa59f23e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa598135100 addr v1:172.21.15.154:6789/0 2024-09-16T03:57:31.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.510+0000 7fa59f23e640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa59809c7d0 con 0x7fa598135100 2024-09-16T03:57:31.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.510+0000 7fa59f23e640 10 monclient(hunting): start opening mon connection 2024-09-16T03:57:31.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.510+0000 7fa59f23e640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7fa59809c090 con 0x7fa5980a1330 2024-09-16T03:57:31.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.510+0000 7fa59f23e640 10 monclient(hunting): start opening mon connection 2024-09-16T03:57:31.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.510+0000 7fa59f23e640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fa59809f610 con 0x7fa59806dac0 2024-09-16T03:57:31.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.510+0000 7fa59f23e640 10 monclient(hunting): _renew_subs 2024-09-16T03:57:31.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.510+0000 7fa59f23e640 10 monclient(hunting): authenticate will time out at 1898.482679s 2024-09-16T03:57:31.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.510+0000 7fa59cfda640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fa5980a1330 0x7fa5980a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:57:31.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.510+0000 7fa597fff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa59806dac0 0x7fa5980a1c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:57:31.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.510+0000 7fa59d7db640 1 -- >> v1:172.21.15.154:6789/0 conn(0x7fa598135100 legacy=0x7fa598135550 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.154:6789/0 2024-09-16T03:57:31.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.510+0000 7fa59cfda640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fa5980a1330 0x7fa5980a1710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:36388/0 (socket says 172.21.15.42:36388) 2024-09-16T03:57:31.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.510+0000 7fa597fff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa59806dac0 0x7fa5980a1c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:41816/0 (socket says 172.21.15.42:41816) 2024-09-16T03:57:31.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.510+0000 7fa59cfda640 1 -- 172.21.15.42:0/307417610 learned_addr learned my addr 172.21.15.42:0/307417610 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:57:31.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.510+0000 7fa59cfda640 10 monclient(hunting): get_auth_request con 0x7fa5980a1330 auth_method 0 2024-09-16T03:57:31.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.510+0000 7fa59cfda640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:57:31.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.510+0000 7fa59cfda640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:57:31.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.510+0000 7fa59cfda640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:57:31.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.510+0000 7fa597fff640 10 monclient(hunting): get_auth_request con 0x7fa59806dac0 auth_method 0 2024-09-16T03:57:31.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.510+0000 7fa597fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:57:31.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.510+0000 7fa597fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:57:31.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.510+0000 7fa597fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:57:31.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.510+0000 7fa59cfda640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:57:31.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.510+0000 7fa59cfda640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:57:31.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.510+0000 7fa59cfda640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:57:31.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.510+0000 7fa597fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:57:31.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.510+0000 7fa597fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:57:31.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.510+0000 7fa597fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:57:31.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.510+0000 7fa59cfda640 10 monclient(hunting): handle_auth_done global_id 824877 payload 274 2024-09-16T03:57:31.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.510+0000 7fa59cfda640 1 -- 172.21.15.42:0/307417610 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa59806dac0 msgr2=0x7fa5980a1c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:57:31.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.510+0000 7fa59cfda640 1 --2- 172.21.15.42:0/307417610 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa59806dac0 0x7fa5980a1c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:57:31.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.510+0000 7fa59cfda640 1 -- 172.21.15.42:0/307417610 >> v1:172.21.15.154:6789/0 conn(0x7fa598135100 legacy=0x7fa598135550 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:57:31.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.510+0000 7fa59cfda640 10 monclient: _finish_hunting 0 2024-09-16T03:57:31.521 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.510+0000 7fa59cfda640 1 monclient: found mon.noname-a 2024-09-16T03:57:31.521 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.510+0000 7fa59cfda640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:57:31.521 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.510+0000 7fa59cfda640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:57:31.521 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.510+0000 7fa59cfda640 1 -- 172.21.15.42:0/307417610 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa59809f8f0 con 0x7fa5980a1330 2024-09-16T03:57:31.521 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.510+0000 7fa597fff640 1 --2- 172.21.15.42:0/307417610 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa59806dac0 0x7fa5980a1c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:57:31.528 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.514+0000 7fa59cfda640 1 --2- 172.21.15.42:0/307417610 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa5980a1330 0x7fa5980a1710 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fa588009870 tx=0x7fa58801c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac48986ff3001e49 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:57:31.528 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.514+0000 7fa5977fe640 1 -- 172.21.15.42:0/307417610 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5880b1070 con 0x7fa5980a1330 2024-09-16T03:57:31.528 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.514+0000 7fa5977fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:57:31.528 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.514+0000 7fa5977fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:57:31.528 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.514+0000 7fa5977fe640 10 monclient: dump: 2024-09-16T03:57:31.528 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:57:31.528 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:57:31.528 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:57:31.528 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:57:31.528 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:57:31.528 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:57:31.528 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:57:31.529 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:57:31.529 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:57:31.529 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:57:31.529 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.514+0000 7fa5977fe640 10 monclient: _finish_auth 0 2024-09-16T03:57:31.529 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.514+0000 7fa5977fe640 10 monclient: _check_auth_tickets 2024-09-16T03:57:31.529 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.514+0000 7fa5977fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:57:31.529 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.514+0000 7fa5977fe640 1 -- 172.21.15.42:0/307417610 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa5880ac630 con 0x7fa5980a1330 2024-09-16T03:57:31.529 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.514+0000 7fa5977fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:57:31.529 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.514+0000 7fa5977fe640 1 -- 172.21.15.42:0/307417610 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5880a8050 con 0x7fa5980a1330 2024-09-16T03:57:31.529 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.514+0000 7fa5977fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:57:31.529 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.514+0000 7fa5977fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:57:31.529 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.514+0000 7fa5977fe640 10 monclient: dump: 2024-09-16T03:57:31.529 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:57:31.529 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:57:31.530 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:57:31.530 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:57:31.530 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:57:31.530 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:57:31.530 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:57:31.530 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:57:31.530 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:57:31.530 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:57:31.530 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.514+0000 7fa59f23e640 5 monclient: authenticate success, global_id 824877 2024-09-16T03:57:31.530 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.514+0000 7fa59f23e640 10 monclient: get_monmap_and_config success 2024-09-16T03:57:31.530 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.514+0000 7fa59f23e640 4 set_mon_vals no callback set 2024-09-16T03:57:31.530 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.514+0000 7fa59f23e640 10 monclient: shutdown 2024-09-16T03:57:31.530 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.514+0000 7fa59f23e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:57:31.530 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.514+0000 7fa59f23e640 1 -- 172.21.15.42:0/307417610 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa5980a1330 msgr2=0x7fa5980a1710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:57:31.531 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.514+0000 7fa59f23e640 1 --2- 172.21.15.42:0/307417610 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa5980a1330 0x7fa5980a1710 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fa588009870 tx=0x7fa58801c500 comp rx=0 tx=0).stop 2024-09-16T03:57:31.531 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.514+0000 7fa59f23e640 1 -- 172.21.15.42:0/307417610 shutdown_connections 2024-09-16T03:57:31.531 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.514+0000 7fa59f23e640 1 --2- 172.21.15.42:0/307417610 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa59806dac0 0x7fa5980a1c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:57:31.531 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.514+0000 7fa59f23e640 1 --2- 172.21.15.42:0/307417610 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa5980a1330 0x7fa5980a1710 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:57:31.531 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.514+0000 7fa59f23e640 1 -- 172.21.15.42:0/307417610 >> 172.21.15.42:0/307417610 conn(0x7fa5980a2420 msgr2=0x7fa5980a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:57:31.531 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.514+0000 7fa59f23e640 1 -- 172.21.15.42:0/307417610 shutdown_connections 2024-09-16T03:57:31.531 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.514+0000 7fa59f23e640 1 -- 172.21.15.42:0/307417610 wait complete. 2024-09-16T03:57:31.531 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.514+0000 7fa59f23e640 10 monclient: build_initial_monmap 2024-09-16T03:57:31.531 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.514+0000 7fa59f23e640 10 monclient: monmap: 2024-09-16T03:57:31.531 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:57:31.531 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:57:31.531 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:57:31.531 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:57:31.531 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:57:31.532 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:57:31.532 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:57:31.532 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:57:31.532 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:57:31.532 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:57:31.532 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.514+0000 7fa59f23e640 1 librados: starting msgr at 2024-09-16T03:57:31.532 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.514+0000 7fa59f23e640 1 librados: starting objecter 2024-09-16T03:57:31.532 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.514+0000 7fa59f23e640 1 Processor -- start 2024-09-16T03:57:31.532 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.514+0000 7fa59f23e640 1 -- start start 2024-09-16T03:57:31.532 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.514+0000 7fa59f23e640 1 librados: setting wanted keys 2024-09-16T03:57:31.532 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.514+0000 7fa59f23e640 1 librados: calling monclient init 2024-09-16T03:57:31.532 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.514+0000 7fa59f23e640 10 monclient: init 2024-09-16T03:57:31.532 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.518+0000 7fa59f23e640 10 monclient: _reopen_session rank -1 2024-09-16T03:57:31.532 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.518+0000 7fa59f23e640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T03:57:31.533 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.518+0000 7fa59f23e640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa5980a1330 0x7fa598134120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:57:31.533 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.518+0000 7fa59f23e640 10 monclient(hunting): picked mon.noname-a con 0x7fa5980a1330 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:57:31.533 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.518+0000 7fa59f23e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa598135100 0x7fa598134660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:57:31.533 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.518+0000 7fa59f23e640 10 monclient(hunting): picked mon.noname-b con 0x7fa598135100 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:57:31.533 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.518+0000 7fa59f23e640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa59811bf70 0x7fa59811c400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:57:31.533 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.518+0000 7fa59f23e640 10 monclient(hunting): picked mon.noname-c con 0x7fa59811bf70 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:57:31.533 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.518+0000 7fa59f23e640 10 monclient(hunting): start opening mon connection 2024-09-16T03:57:31.533 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.518+0000 7fa59f23e640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fa59813fd00 con 0x7fa5980a1330 2024-09-16T03:57:31.533 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.518+0000 7fa59f23e640 10 monclient(hunting): start opening mon connection 2024-09-16T03:57:31.533 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.518+0000 7fa59f23e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa59813fb80 con 0x7fa598135100 2024-09-16T03:57:31.533 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.518+0000 7fa59f23e640 10 monclient(hunting): start opening mon connection 2024-09-16T03:57:31.533 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.518+0000 7fa59f23e640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fa59813fe80 con 0x7fa59811bf70 2024-09-16T03:57:31.533 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.518+0000 7fa59f23e640 10 monclient(hunting): _renew_subs 2024-09-16T03:57:31.533 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.518+0000 7fa59f23e640 10 monclient(hunting): authenticate will time out at 1898.488484s 2024-09-16T03:57:31.533 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.518+0000 7fa59cfda640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa5980a1330 0x7fa598134120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:57:31.534 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.518+0000 7fa59cfda640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa5980a1330 0x7fa598134120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:36390/0 (socket says 172.21.15.42:36390) 2024-09-16T03:57:31.534 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.518+0000 7fa59cfda640 1 -- 172.21.15.42:0/3166358253 learned_addr learned my addr 172.21.15.42:0/3166358253 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:57:31.534 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.518+0000 7fa59cfda640 10 monclient(hunting): get_auth_request con 0x7fa5980a1330 auth_method 0 2024-09-16T03:57:31.534 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.518+0000 7fa59cfda640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:57:31.534 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.518+0000 7fa59cfda640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:57:31.534 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.518+0000 7fa59cfda640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:57:31.534 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.518+0000 7fa59d7db640 1 --2- 172.21.15.42:0/3166358253 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa59811bf70 0x7fa59811c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:57:31.534 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.518+0000 7fa59d7db640 10 monclient(hunting): get_auth_request con 0x7fa59811bf70 auth_method 0 2024-09-16T03:57:31.534 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.518+0000 7fa59d7db640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:57:31.534 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.518+0000 7fa59d7db640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:57:31.534 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.518+0000 7fa59d7db640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:57:31.534 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.518+0000 7fa59cfda640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:57:31.534 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.518+0000 7fa59cfda640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:57:31.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.518+0000 7fa59cfda640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:57:31.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.518+0000 7fa597fff640 1 -- 172.21.15.42:0/3166358253 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa598135100 msgr2=0x7fa598134660 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T03:57:31.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.518+0000 7fa597fff640 1 --2- 172.21.15.42:0/3166358253 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa598135100 0x7fa598134660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:57:31.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.518+0000 7fa59d7db640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:57:31.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.518+0000 7fa59d7db640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:57:31.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.518+0000 7fa59d7db640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:57:31.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.518+0000 7fa59cfda640 10 monclient(hunting): handle_auth_done global_id 824880 payload 824 2024-09-16T03:57:31.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.518+0000 7fa59cfda640 1 -- 172.21.15.42:0/3166358253 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa59811bf70 msgr2=0x7fa59811c400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:57:31.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.518+0000 7fa59cfda640 1 --2- 172.21.15.42:0/3166358253 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa59811bf70 0x7fa59811c400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:57:31.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.518+0000 7fa59cfda640 1 -- 172.21.15.42:0/3166358253 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa598135100 msgr2=0x7fa598134660 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:57:31.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.518+0000 7fa59cfda640 1 --2- 172.21.15.42:0/3166358253 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa598135100 0x7fa598134660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:57:31.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.518+0000 7fa59cfda640 10 monclient: _finish_hunting 0 2024-09-16T03:57:31.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.518+0000 7fa59cfda640 1 monclient: found mon.noname-a 2024-09-16T03:57:31.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.518+0000 7fa59cfda640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:57:31.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.518+0000 7fa59cfda640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:57:31.536 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.518+0000 7fa59cfda640 1 -- 172.21.15.42:0/3166358253 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5980a5210 con 0x7fa5980a1330 2024-09-16T03:57:31.536 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.518+0000 7fa59cfda640 1 --2- 172.21.15.42:0/3166358253 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa5980a1330 0x7fa598134120 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fa5880af040 tx=0x7fa5980a22d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:57:31.536 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.518+0000 7fa5957fa640 10 client.?.objecter ms_handle_connect 0x7fa5980a1330 2024-09-16T03:57:31.536 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.518+0000 7fa5957fa640 10 client.?.objecter resend_mon_ops 2024-09-16T03:57:31.536 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.518+0000 7fa5957fa640 1 -- 172.21.15.42:0/3166358253 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5880bc070 con 0x7fa5980a1330 2024-09-16T03:57:31.536 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.518+0000 7fa5957fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:57:31.536 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.518+0000 7fa5957fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:57:31.536 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.518+0000 7fa5957fa640 10 monclient: dump: 2024-09-16T03:57:31.536 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:57:31.536 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:57:31.536 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:57:31.536 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:57:31.537 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:57:31.537 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:57:31.537 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:57:31.537 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:57:31.537 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:57:31.537 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:57:31.537 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.518+0000 7fa5957fa640 10 monclient: _finish_auth 0 2024-09-16T03:57:31.537 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.518+0000 7fa5957fa640 10 monclient: _check_auth_tickets 2024-09-16T03:57:31.537 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.518+0000 7fa5957fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:57:31.537 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.518+0000 7fa59f23e640 5 monclient: authenticate success, global_id 824880 2024-09-16T03:57:31.537 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.518+0000 7fa5957fa640 1 -- 172.21.15.42:0/3166358253 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa5880ab5e0 con 0x7fa5980a1330 2024-09-16T03:57:31.537 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.518+0000 7fa5957fa640 10 monclient: handle_config config(0 keys) 2024-09-16T03:57:31.537 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.518+0000 7fa59f23e640 10 monclient: _renew_subs 2024-09-16T03:57:31.537 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.518+0000 7fa5957fa640 1 -- 172.21.15.42:0/3166358253 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5880b1070 con 0x7fa5980a1330 2024-09-16T03:57:31.537 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.518+0000 7fa59f23e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:57:31.538 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.518+0000 7fa59f23e640 1 -- 172.21.15.42:0/3166358253 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa598071b40 con 0x7fa5980a1330 2024-09-16T03:57:31.538 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.518+0000 7fa5957fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:57:31.538 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.518+0000 7fa5957fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:57:31.538 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.518+0000 7fa596ffd640 4 set_mon_vals no callback set 2024-09-16T03:57:31.538 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.518+0000 7fa5957fa640 10 monclient: dump: 2024-09-16T03:57:31.538 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:57:31.538 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:57:31.538 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:57:31.538 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:57:31.538 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:57:31.538 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:57:31.538 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:57:31.538 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:57:31.538 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:57:31.539 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:57:31.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.518+0000 7fa59f23e640 10 client.824880.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:57:31.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.518+0000 7fa59f23e640 10 monclient: _renew_subs 2024-09-16T03:57:31.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.518+0000 7fa59f23e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:57:31.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.518+0000 7fa59f23e640 1 -- 172.21.15.42:0/3166358253 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa59811e8c0 con 0x7fa5980a1330 2024-09-16T03:57:31.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.518+0000 7fa59f23e640 1 librados: init done 2024-09-16T03:57:31.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.518+0000 7fa59f23e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:57:31.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.518+0000 7fa59f23e640 1 --2- 172.21.15.42:0/3166358253 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa59806dac0 0x7fa59811c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:57:31.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.518+0000 7fa59f23e640 10 monclient(hunting): start opening mon connection 2024-09-16T03:57:31.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.518+0000 7fa59f23e640 1 -- 172.21.15.42:0/3166358253 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fa59811d520 con 0x7fa59806dac0 2024-09-16T03:57:31.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.518+0000 7fa597fff640 1 --2- 172.21.15.42:0/3166358253 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa59806dac0 0x7fa59811c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:57:31.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.518+0000 7fa5957fa640 1 -- 172.21.15.42:0/3166358253 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fa5880bf070 con 0x7fa5980a1330 2024-09-16T03:57:31.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.518+0000 7fa597fff640 10 monclient: get_auth_request con 0x7fa59806dac0 auth_method 0 2024-09-16T03:57:31.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.518+0000 7fa597fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:57:31.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.518+0000 7fa597fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:57:31.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.518+0000 7fa597fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:57:31.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.518+0000 7fa597fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:57:31.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.518+0000 7fa597fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:57:31.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.518+0000 7fa597fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:57:31.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.518+0000 7fa597fff640 10 monclient(hunting): handle_auth_done global_id 824883 payload 824 2024-09-16T03:57:31.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.518+0000 7fa5957fa640 1 --2- 172.21.15.42:0/3166358253 >> v2:172.21.15.154:6812/254845342 conn(0x7fa5840377c0 0x7fa584039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:57:31.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.518+0000 7fa597fff640 1 -- 172.21.15.42:0/3166358253 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa598135550 con 0x7fa59806dac0 2024-09-16T03:57:31.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.518+0000 7fa5957fa640 1 -- 172.21.15.42:0/3166358253 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(461..461 src has 330..461) ==== 5714+0+0 (secure 0 0 0) 0x7fa5880e2880 con 0x7fa5980a1330 2024-09-16T03:57:31.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.518+0000 7fa5957fa640 10 client.824880.objecter ms_dispatch 0x7fa598000ba0 osd_map(461..461 src has 330..461) 2024-09-16T03:57:31.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.518+0000 7fa5957fa640 3 client.824880.objecter handle_osd_map got epochs [461,461] > 0 2024-09-16T03:57:31.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.518+0000 7fa5957fa640 3 client.824880.objecter handle_osd_map decoding full epoch 461 2024-09-16T03:57:31.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.518+0000 7fa5957fa640 20 client.824880.objecter dump_active .. 0 homeless 2024-09-16T03:57:31.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.518+0000 7fa597fff640 1 --2- 172.21.15.42:0/3166358253 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa59806dac0 0x7fa59811c940 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fa59813bd50 tx=0x7fa58c00ec00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:57:31.541 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.518+0000 7fa5957fa640 10 client.824880.objecter ms_handle_connect 0x7fa59806dac0 2024-09-16T03:57:31.541 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.518+0000 7fa5957fa640 10 client.824880.objecter resend_mon_ops 2024-09-16T03:57:31.541 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.518+0000 7fa5957fa640 1 -- 172.21.15.42:0/3166358253 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa58c00ee50 con 0x7fa59806dac0 2024-09-16T03:57:31.541 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.518+0000 7fa59d7db640 1 --2- 172.21.15.42:0/3166358253 >> v2:172.21.15.154:6812/254845342 conn(0x7fa5840377c0 0x7fa584039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:57:31.541 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.518+0000 7fa5957fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:57:31.541 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.518+0000 7fa5957fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:57:31.541 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.518+0000 7fa5957fa640 10 monclient: dump: 2024-09-16T03:57:31.541 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:57:31.541 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:57:31.541 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:57:31.541 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:57:31.541 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:57:31.541 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:57:31.541 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:57:31.542 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:57:31.542 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:57:31.542 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:57:31.542 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.518+0000 7fa59d7db640 10 monclient: get_auth_request con 0x7fa5840377c0 auth_method 0 2024-09-16T03:57:31.542 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.518+0000 7fa5957fa640 1 -- 172.21.15.42:0/3166358253 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fa58c00ee50 con 0x7fa59806dac0 2024-09-16T03:57:31.542 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.518+0000 7fa5957fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:57:31.542 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.518+0000 7fa5957fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:57:31.542 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.518+0000 7fa5957fa640 1 -- 172.21.15.42:0/3166358253 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa59806dac0 msgr2=0x7fa59811c940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:57:31.542 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.518+0000 7fa5957fa640 1 --2- 172.21.15.42:0/3166358253 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa59806dac0 0x7fa59811c940 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fa59813bd50 tx=0x7fa58c00ec00 comp rx=0 tx=0).stop 2024-09-16T03:57:31.542 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.518+0000 7fa5957fa640 1 -- 172.21.15.42:0/3166358253 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa59806dac0 msgr2=0x7fa59811c940 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:57:31.542 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.518+0000 7fa5957fa640 1 --2- 172.21.15.42:0/3166358253 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa59806dac0 0x7fa59811c940 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:57:31.542 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.518+0000 7fa59d7db640 1 --2- 172.21.15.42:0/3166358253 >> v2:172.21.15.154:6812/254845342 conn(0x7fa5840377c0 0x7fa584039c80 secure :-1 s=READY pgs=3682 cs=0 l=1 rev1=1 crypto rx=0x7fa590003bd0 tx=0x7fa59000d040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:57:31.542 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.518+0000 7fa5957fa640 10 client.824880.objecter ms_handle_connect 0x7fa5840377c0 2024-09-16T03:57:31.544 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.526+0000 7fa59f23e640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:57:31.544 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.526+0000 7fa59f23e640 1 --2- 172.21.15.42:0/3166358253 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa59811eb00 0x7fa598137330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:57:31.544 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.526+0000 7fa59f23e640 10 monclient(hunting): start opening mon connection 2024-09-16T03:57:31.544 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.526+0000 7fa59f23e640 1 -- 172.21.15.42:0/3166358253 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fa59811ef10 con 0x7fa59811eb00 2024-09-16T03:57:31.544 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.526+0000 7fa597fff640 1 --2- 172.21.15.42:0/3166358253 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa59811eb00 0x7fa598137330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:57:31.544 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.526+0000 7fa597fff640 10 monclient: get_auth_request con 0x7fa59811eb00 auth_method 0 2024-09-16T03:57:31.544 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.526+0000 7fa597fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:57:31.544 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.526+0000 7fa597fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:57:31.544 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.526+0000 7fa597fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:57:31.545 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.526+0000 7fa597fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:57:31.545 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.526+0000 7fa597fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:57:31.545 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.526+0000 7fa597fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:57:31.545 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.526+0000 7fa597fff640 10 monclient(hunting): handle_auth_done global_id 824886 payload 824 2024-09-16T03:57:31.545 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.526+0000 7fa597fff640 1 -- 172.21.15.42:0/3166358253 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fa598134ba0 con 0x7fa59811eb00 2024-09-16T03:57:31.545 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.526+0000 7fa5957fa640 10 client.824880.objecter ms_handle_connect 0x7fa59811eb00 2024-09-16T03:57:31.545 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.526+0000 7fa5957fa640 10 client.824880.objecter resend_mon_ops 2024-09-16T03:57:31.545 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.526+0000 7fa597fff640 1 --2- 172.21.15.42:0/3166358253 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa59811eb00 0x7fa598137330 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fa59813bd50 tx=0x7fa58c004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:57:31.545 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.526+0000 7fa5957fa640 1 -- 172.21.15.42:0/3166358253 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa58c0059a0 con 0x7fa59811eb00 2024-09-16T03:57:31.545 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.526+0000 7fa5957fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:57:31.545 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.526+0000 7fa5957fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:57:31.545 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.526+0000 7fa5957fa640 10 monclient: dump: 2024-09-16T03:57:31.545 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:57:31.545 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:57:31.546 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:57:31.546 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:57:31.546 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:57:31.546 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:57:31.546 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:57:31.546 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:57:31.546 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:57:31.546 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:57:31.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.526+0000 7fa5957fa640 1 -- 172.21.15.42:0/3166358253 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4871 (secure 0 0 0) 0x7fa58c0059a0 con 0x7fa59811eb00 2024-09-16T03:57:31.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.526+0000 7fa5957fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:57:31.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.526+0000 7fa5957fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:57:31.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.526+0000 7fa5957fa640 1 -- 172.21.15.42:0/3166358253 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa59811eb00 msgr2=0x7fa598137330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:57:31.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.526+0000 7fa5957fa640 1 --2- 172.21.15.42:0/3166358253 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa59811eb00 0x7fa598137330 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fa59813bd50 tx=0x7fa58c004740 comp rx=0 tx=0).stop 2024-09-16T03:57:31.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.526+0000 7fa5957fa640 1 -- 172.21.15.42:0/3166358253 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa59811eb00 msgr2=0x7fa598137330 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:57:31.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.526+0000 7fa5957fa640 1 --2- 172.21.15.42:0/3166358253 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa59811eb00 0x7fa598137330 secure :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fa59813bd50 tx=0x7fa58c004740 comp rx=0 tx=0).stop 2024-09-16T03:57:31.547 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:57:31.547 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:57:31.547 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:57:31.547 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "leader", 2024-09-16T03:57:31.547 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 142, 2024-09-16T03:57:31.547 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 341380, 2024-09-16T03:57:31.547 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:57:31.547 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:57:31.547 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T03:57:31.547 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:57:31.548 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 1, 2024-09-16T03:57:31.548 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:57:31.548 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:57:31.548 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:57:31.548 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:57:31.548 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:57:31.548 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:57:31.548 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:57:31.548 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:57:31.548 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:57:31.548 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:57:31.548 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:57:31.548 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:57:31.549 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:57:31.549 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:57:31.549 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:57:31.549 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:57:31.549 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:57:31.549 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:57:31.549 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:57:31.549 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:57:31.549 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:57:31.549 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:57:31.549 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:57:31.549 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:57:31.549 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:57:31.549 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:57:31.549 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:57:31.550 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:57:31.550 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:57:31.550 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:57:31.550 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:57:31.550 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:57:31.550 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:57:31.550 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:57:31.550 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:57:31.550 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:57:31.550 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:57:31.550 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:57:31.550 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:57:31.550 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:57:31.550 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:57:31.551 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:57:31.551 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:57:31.551 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:57:31.551 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:57:31.551 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:57:31.551 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:57:31.551 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:57:31.551 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:57:31.551 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:57:31.551 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:57:31.551 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:57:31.551 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:57:31.551 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:57:31.551 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:57:31.552 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:57:31.552 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:57:31.552 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:57:31.552 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:57:31.552 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:57:31.552 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:57:31.552 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:57:31.552 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:57:31.552 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:57:31.552 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:57:31.552 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:57:31.552 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:57:31.552 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:57:31.553 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:57:31.553 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:57:31.553 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:57:31.553 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:57:31.553 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:57:31.553 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:57:31.553 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:57:31.553 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:57:31.553 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:57:31.553 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:57:31.556 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:57:31.556 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:57:31.556 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:57:31.557 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:57:31.557 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:57:31.557 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:57:31.557 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:57:31.557 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:57:31.557 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:57:31.557 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:57:31.557 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:57:31.557 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:57:31.557 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:57:31.557 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:57:31.557 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:57:31.557 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:57:31.557 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:57:31.558 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:57:31.558 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:57:31.558 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:57:31.558 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:57:31.558 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:57:31.558 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:57:31.558 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:57:31.558 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:57:31.558 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:57:31.558 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:57:31.558 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:57:31.558 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:57:31.558 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:57:31.558 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:57:31.559 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:57:31.559 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:57:31.559 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:57:31.559 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:57:31.559 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:57:31.559 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:57:31.559 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:57:31.559 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:57:31.559 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:57:31.559 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:57:31.559 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:57:31.559 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:57:31.559 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:57:31.559 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:57:31.559 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:57:31.560 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:57:31.560 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:57:31.560 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:57:31.560 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:57:31.560 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:57:31.560 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:57:31.560 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:57:31.560 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:57:31.560 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:57:31.560 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:57:31.560 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:57:31.560 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:57:31.560 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:57:31.560 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:57:31.561 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:57:31.561 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:57:31.561 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:57:31.561 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T03:57:31.561 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:57:31.561 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:57:31.561 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:57:31.561 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 6 2024-09-16T03:57:31.561 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:57:31.561 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:57:31.561 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:57:31.561 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:57:31.561 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:57:31.561 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:57:31.562 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 8 2024-09-16T03:57:31.562 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:57:31.562 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:57:31.562 INFO:teuthology.orchestra.run.smithi042.stdout: "mgr": [ 2024-09-16T03:57:31.562 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:57:31.562 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:57:31.562 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:57:31.562 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:57:31.562 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:57:31.562 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:57:31.563 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:57:31.563 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:57:31.563 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:57:31.565 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.526+0000 7fa59f23e640 10 librados: watch_flush enter 2024-09-16T03:57:31.565 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.526+0000 7fa59f23e640 10 librados: watch_flush exit 2024-09-16T03:57:31.565 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.526+0000 7fa59f23e640 20 client.824880.objecter shutdown clearing up homeless session... 2024-09-16T03:57:31.565 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.526+0000 7fa59f23e640 10 client.824880.objecter successfully canceled tick 2024-09-16T03:57:31.565 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.526+0000 7fa59f23e640 1 -- 172.21.15.42:0/3166358253 >> v2:172.21.15.154:6812/254845342 conn(0x7fa5840377c0 msgr2=0x7fa584039c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:57:31.565 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.526+0000 7fa59f23e640 1 --2- 172.21.15.42:0/3166358253 >> v2:172.21.15.154:6812/254845342 conn(0x7fa5840377c0 0x7fa584039c80 secure :-1 s=READY pgs=3682 cs=0 l=1 rev1=1 crypto rx=0x7fa590003bd0 tx=0x7fa59000d040 comp rx=0 tx=0).stop 2024-09-16T03:57:31.565 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.526+0000 7fa59f23e640 10 monclient: shutdown 2024-09-16T03:57:31.565 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.526+0000 7fa59f23e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:57:31.565 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.526+0000 7fa59f23e640 1 -- 172.21.15.42:0/3166358253 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa5980a1330 msgr2=0x7fa598134120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:57:31.565 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.526+0000 7fa59f23e640 1 --2- 172.21.15.42:0/3166358253 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa5980a1330 0x7fa598134120 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fa5880af040 tx=0x7fa5980a22d0 comp rx=0 tx=0).stop 2024-09-16T03:57:31.565 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.526+0000 7fa59d7db640 1 -- 172.21.15.42:0/3166358253 reap_dead start 2024-09-16T03:57:31.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.526+0000 7fa59d7db640 1 -- 172.21.15.42:0/3166358253 reap_dead start 2024-09-16T03:57:31.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.526+0000 7fa59f23e640 1 -- 172.21.15.42:0/3166358253 shutdown_connections 2024-09-16T03:57:31.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.526+0000 7fa59f23e640 1 -- 172.21.15.42:0/3166358253 >> 172.21.15.42:0/3166358253 conn(0x7fa5980a2420 msgr2=0x7fa5980a3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:57:31.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.526+0000 7fa59f23e640 1 -- 172.21.15.42:0/3166358253 shutdown_connections 2024-09-16T03:57:31.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.526+0000 7fa59f23e640 1 -- 172.21.15.42:0/3166358253 wait complete. 2024-09-16T03:57:31.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.526+0000 7fa59f23e640 1 librados: shutdown 2024-09-16T03:57:31.566 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T03:57:31.566 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-16T03:57:31.566 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T03:57:31.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.678+0000 7fc881357640 10 monclient: get_monmap_and_config 2024-09-16T03:57:31.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.678+0000 7fc881357640 10 monclient: build_initial_monmap 2024-09-16T03:57:31.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.678+0000 7fc881357640 10 monclient: monmap: 2024-09-16T03:57:31.683 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:57:31.683 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:57:31.683 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:57:31.681282+0000 2024-09-16T03:57:31.683 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:57:31.681282+0000 2024-09-16T03:57:31.683 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:57:31.683 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:57:31.683 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:57:31.683 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:57:31.683 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:57:31.683 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:57:31.683 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:57:31.684 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:57:31.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.678+0000 7fc881357640 1 Processor -- start 2024-09-16T03:57:31.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.678+0000 7fc881357640 1 -- start start 2024-09-16T03:57:31.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.678+0000 7fc881357640 10 monclient: init 2024-09-16T03:57:31.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.678+0000 7fc881357640 10 monclient: _reopen_session rank -1 2024-09-16T03:57:31.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.678+0000 7fc881357640 10 monclient: _add_conns ranks=[0,4,1,3,2] 2024-09-16T03:57:31.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.678+0000 7fc881357640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc87c0a1330 addr v1:172.21.15.42:6789/0 2024-09-16T03:57:31.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.678+0000 7fc881357640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fc87c06dac0 0x7fc87c13a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:57:31.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.678+0000 7fc881357640 10 monclient(hunting): picked mon.noname-c con 0x7fc87c06dac0 addr v2:172.21.15.154:3300/0 2024-09-16T03:57:31.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.678+0000 7fc881357640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc87c0a1e20 addr v1:172.21.15.154:6789/0 2024-09-16T03:57:31.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.678+0000 7fc881357640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc87c09c090 con 0x7fc87c0a1330 2024-09-16T03:57:31.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.678+0000 7fc881357640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc87c09f610 con 0x7fc87c0a1e20 2024-09-16T03:57:31.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.678+0000 7fc881357640 10 monclient(hunting): start opening mon connection 2024-09-16T03:57:31.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.678+0000 7fc881357640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fc87c09f8f0 con 0x7fc87c06dac0 2024-09-16T03:57:31.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.678+0000 7fc881357640 10 monclient(hunting): _renew_subs 2024-09-16T03:57:31.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.678+0000 7fc881357640 10 monclient(hunting): authenticate will time out at 1898.650595s 2024-09-16T03:57:31.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.678+0000 7fc87affd640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fc87c0a1330 0x7fc87c0a1710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:33076/0 (socket says 172.21.15.42:33076) 2024-09-16T03:57:31.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.678+0000 7fc87affd640 1 -- 172.21.15.42:0/2308657989 learned_addr learned my addr 172.21.15.42:0/2308657989 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:57:31.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.678+0000 7fc87b7fe640 1 -- 172.21.15.42:0/2308657989 >> v1:172.21.15.154:6789/0 conn(0x7fc87c0a1e20 legacy=0x7fc87c13c890 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.154:6789/0 2024-09-16T03:57:31.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.678+0000 7fc87a7fc640 1 -- 172.21.15.42:0/2308657989 >> v2:172.21.15.154:3300/0 conn(0x7fc87c06dac0 msgr2=0x7fc87c13a340 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T03:57:31.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.678+0000 7fc87a7fc640 1 --2- 172.21.15.42:0/2308657989 >> v2:172.21.15.154:3300/0 conn(0x7fc87c06dac0 0x7fc87c13a340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:57:31.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.678+0000 7fc879ffb640 1 -- 172.21.15.42:0/2308657989 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2635002098 0 0) 0x7fc87c09c090 con 0x7fc87c0a1330 2024-09-16T03:57:31.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.678+0000 7fc879ffb640 10 monclient(hunting): _negotiate 2024-09-16T03:57:31.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.678+0000 7fc879ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:57:31.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.678+0000 7fc879ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:57:31.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.678+0000 7fc879ffb640 10 monclient(hunting): my global_id is 824889 2024-09-16T03:57:31.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.678+0000 7fc879ffb640 1 -- 172.21.15.42:0/2308657989 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc868003530 con 0x7fc87c0a1330 2024-09-16T03:57:31.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.678+0000 7fc879ffb640 1 -- 172.21.15.42:0/2308657989 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2454549685 0 0) 0x7fc868003530 con 0x7fc87c0a1330 2024-09-16T03:57:31.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.678+0000 7fc879ffb640 1 -- 172.21.15.42:0/2308657989 >> v2:172.21.15.154:3300/0 conn(0x7fc87c06dac0 msgr2=0x7fc87c13a340 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:57:31.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.678+0000 7fc879ffb640 1 --2- 172.21.15.42:0/2308657989 >> v2:172.21.15.154:3300/0 conn(0x7fc87c06dac0 0x7fc87c13a340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:57:31.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.678+0000 7fc879ffb640 1 -- 172.21.15.42:0/2308657989 >> v1:172.21.15.154:6789/0 conn(0x7fc87c0a1e20 legacy=0x7fc87c13c890 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:57:31.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.678+0000 7fc879ffb640 10 monclient: _finish_hunting 0 2024-09-16T03:57:31.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.678+0000 7fc879ffb640 1 monclient: found mon.noname-a-legacy 2024-09-16T03:57:31.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.678+0000 7fc879ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:57:31.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.678+0000 7fc879ffb640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.42:6789/0 2024-09-16T03:57:31.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.678+0000 7fc879ffb640 1 -- 172.21.15.42:0/2308657989 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc87c09fbd0 con 0x7fc87c0a1330 2024-09-16T03:57:31.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.678+0000 7fc879ffb640 10 monclient: _finish_auth 0 2024-09-16T03:57:31.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.678+0000 7fc879ffb640 10 monclient: _check_auth_tickets 2024-09-16T03:57:31.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.678+0000 7fc879ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:57:31.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.678+0000 7fc879ffb640 1 -- 172.21.15.42:0/2308657989 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2177413734 0 0) 0x7fc86c001a00 con 0x7fc87c0a1330 2024-09-16T03:57:31.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.678+0000 7fc879ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:57:31.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.678+0000 7fc879ffb640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T03:57:31.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.678+0000 7fc879ffb640 10 monclient: dump: 2024-09-16T03:57:31.692 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:57:31.692 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:57:31.692 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:57:31.692 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:57:31.692 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:57:31.692 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:57:31.692 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:57:31.692 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:57:31.692 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:57:31.693 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:57:31.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.678+0000 7fc879ffb640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] but i'm connected to v1:172.21.15.42:6789/0, reconnecting 2024-09-16T03:57:31.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.678+0000 7fc879ffb640 10 monclient: _reopen_session rank -1 2024-09-16T03:57:31.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.678+0000 7fc879ffb640 1 -- 172.21.15.42:0/2308657989 >> v1:172.21.15.42:6789/0 conn(0x7fc87c0a1330 legacy=0x7fc87c0a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:57:31.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.678+0000 7fc879ffb640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:57:31.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.678+0000 7fc879ffb640 1 --2- 172.21.15.42:0/2308657989 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc8680039c0 0x7fc868003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:57:31.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.678+0000 7fc879ffb640 10 monclient(hunting): picked mon.b con 0x7fc8680039c0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:57:31.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.678+0000 7fc879ffb640 1 --2- 172.21.15.42:0/2308657989 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc868004810 0x7fc868004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:57:31.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.678+0000 7fc879ffb640 10 monclient(hunting): picked mon.a con 0x7fc868004810 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:57:31.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.678+0000 7fc879ffb640 1 --2- 172.21.15.42:0/2308657989 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc8680052f0 0x7fc86800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:57:31.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.678+0000 7fc879ffb640 10 monclient(hunting): picked mon.c con 0x7fc8680052f0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:57:31.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.678+0000 7fc879ffb640 10 monclient(hunting): start opening mon connection 2024-09-16T03:57:31.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.678+0000 7fc879ffb640 1 -- 172.21.15.42:0/2308657989 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fc87c09f8f0 con 0x7fc868004810 2024-09-16T03:57:31.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.678+0000 7fc879ffb640 10 monclient(hunting): start opening mon connection 2024-09-16T03:57:31.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.678+0000 7fc879ffb640 1 -- 172.21.15.42:0/2308657989 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc86800c240 con 0x7fc8680039c0 2024-09-16T03:57:31.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.678+0000 7fc879ffb640 10 monclient(hunting): start opening mon connection 2024-09-16T03:57:31.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.678+0000 7fc879ffb640 1 -- 172.21.15.42:0/2308657989 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fc86800c440 con 0x7fc8680052f0 2024-09-16T03:57:31.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.678+0000 7fc879ffb640 10 monclient(hunting): _renew_subs 2024-09-16T03:57:31.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.678+0000 7fc879ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:57:31.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.678+0000 7fc87a7fc640 1 --2- 172.21.15.42:0/2308657989 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc868004810 0x7fc868004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:57:31.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.678+0000 7fc87affd640 1 -- 172.21.15.42:0/2308657989 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc8680039c0 msgr2=0x7fc868003dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T03:57:31.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.678+0000 7fc87b7fe640 1 --2- 172.21.15.42:0/2308657989 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc8680052f0 0x7fc86800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:57:31.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.678+0000 7fc87affd640 1 --2- 172.21.15.42:0/2308657989 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc8680039c0 0x7fc868003dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:57:31.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.678+0000 7fc87b7fe640 10 monclient(hunting): get_auth_request con 0x7fc8680052f0 auth_method 0 2024-09-16T03:57:31.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.678+0000 7fc87b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:57:31.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.678+0000 7fc87b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:57:31.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.678+0000 7fc87b7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:57:31.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.678+0000 7fc87a7fc640 10 monclient(hunting): get_auth_request con 0x7fc868004810 auth_method 0 2024-09-16T03:57:31.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.678+0000 7fc87a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:57:31.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.678+0000 7fc87a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:57:31.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.678+0000 7fc87a7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:57:31.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.678+0000 7fc87b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:57:31.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.678+0000 7fc87b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:57:31.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.678+0000 7fc87b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T03:57:31.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.678+0000 7fc87a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:57:31.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.678+0000 7fc87a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:57:31.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.678+0000 7fc87a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T03:57:31.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.678+0000 7fc87b7fe640 10 monclient(hunting): handle_auth_done global_id 824889 payload 293 2024-09-16T03:57:31.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.678+0000 7fc87b7fe640 1 -- 172.21.15.42:0/2308657989 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc8680039c0 msgr2=0x7fc868003dc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:57:31.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.678+0000 7fc87b7fe640 1 --2- 172.21.15.42:0/2308657989 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc8680039c0 0x7fc868003dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:57:31.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.678+0000 7fc87b7fe640 1 -- 172.21.15.42:0/2308657989 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc868004810 msgr2=0x7fc868004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:57:31.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.678+0000 7fc87b7fe640 1 --2- 172.21.15.42:0/2308657989 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc868004810 0x7fc868004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:57:31.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.678+0000 7fc87b7fe640 10 monclient: _finish_hunting 0 2024-09-16T03:57:31.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.678+0000 7fc87b7fe640 1 monclient: found mon.c 2024-09-16T03:57:31.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.678+0000 7fc87b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:57:31.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.678+0000 7fc87b7fe640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:57:31.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.678+0000 7fc87b7fe640 1 -- 172.21.15.42:0/2308657989 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc868003530 con 0x7fc8680052f0 2024-09-16T03:57:31.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.678+0000 7fc87b7fe640 10 monclient: _finish_auth 0 2024-09-16T03:57:31.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.678+0000 7fc87b7fe640 10 monclient: _check_auth_tickets 2024-09-16T03:57:31.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.678+0000 7fc87b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:57:31.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.678+0000 7fc881357640 5 monclient: authenticate success, global_id 824889 2024-09-16T03:57:31.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.678+0000 7fc881357640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:57:31.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.682+0000 7fc87b7fe640 1 -- 172.21.15.42:0/2308657989 reap_dead start 2024-09-16T03:57:31.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.682+0000 7fc87b7fe640 1 --2- 172.21.15.42:0/2308657989 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc8680052f0 0x7fc86800b9d0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fc874006fd0 tx=0x7fc87401c3e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2a69eb0bf5e5c446 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:57:31.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.682+0000 7fc879ffb640 1 -- 172.21.15.42:0/2308657989 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc874009520 con 0x7fc8680052f0 2024-09-16T03:57:31.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.682+0000 7fc879ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:57:31.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.682+0000 7fc879ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:57:31.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.682+0000 7fc879ffb640 10 monclient: dump: 2024-09-16T03:57:31.697 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:57:31.697 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:57:31.697 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:57:31.697 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:57:31.698 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:57:31.698 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:57:31.698 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:57:31.698 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:57:31.698 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:57:31.698 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:57:31.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.682+0000 7fc881357640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:57:31.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.682+0000 7fc879ffb640 1 -- 172.21.15.42:0/2308657989 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc874003bc0 con 0x7fc8680052f0 2024-09-16T03:57:31.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.682+0000 7fc879ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T03:57:31.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.682+0000 7fc881357640 10 monclient: get_monmap_and_config success 2024-09-16T03:57:31.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.682+0000 7fc881357640 4 set_mon_vals no callback set 2024-09-16T03:57:31.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.682+0000 7fc879ffb640 1 -- 172.21.15.42:0/2308657989 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc874005070 con 0x7fc8680052f0 2024-09-16T03:57:31.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.682+0000 7fc879ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:57:31.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.682+0000 7fc879ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:57:31.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.682+0000 7fc879ffb640 10 monclient: dump: 2024-09-16T03:57:31.699 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:57:31.699 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:57:31.699 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:57:31.699 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:57:31.699 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:57:31.699 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:57:31.699 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:57:31.699 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:57:31.699 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:57:31.699 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:57:31.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.682+0000 7fc881357640 10 monclient: shutdown 2024-09-16T03:57:31.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.682+0000 7fc881357640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:57:31.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.682+0000 7fc881357640 1 -- 172.21.15.42:0/2308657989 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc8680052f0 msgr2=0x7fc86800b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:57:31.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.682+0000 7fc881357640 1 --2- 172.21.15.42:0/2308657989 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc8680052f0 0x7fc86800b9d0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fc874006fd0 tx=0x7fc87401c3e0 comp rx=0 tx=0).stop 2024-09-16T03:57:31.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.682+0000 7fc881357640 1 -- 172.21.15.42:0/2308657989 shutdown_connections 2024-09-16T03:57:31.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.682+0000 7fc881357640 1 --2- 172.21.15.42:0/2308657989 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc8680052f0 0x7fc86800b9d0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:57:31.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.682+0000 7fc881357640 1 -- 172.21.15.42:0/2308657989 >> 172.21.15.42:0/2308657989 conn(0x7fc87c0a2420 msgr2=0x7fc87c0a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:57:31.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.682+0000 7fc881357640 1 -- 172.21.15.42:0/2308657989 shutdown_connections 2024-09-16T03:57:31.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.682+0000 7fc881357640 1 -- 172.21.15.42:0/2308657989 wait complete. 2024-09-16T03:57:31.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.682+0000 7fc881357640 10 monclient: build_initial_monmap 2024-09-16T03:57:31.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.682+0000 7fc881357640 10 monclient: monmap: 2024-09-16T03:57:31.701 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:57:31.701 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:57:31.701 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:57:31.701 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:57:31.701 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:57:31.701 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:57:31.701 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:57:31.701 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:57:31.701 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:57:31.701 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:57:31.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.682+0000 7fc881357640 1 librados: starting msgr at 2024-09-16T03:57:31.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.682+0000 7fc881357640 1 librados: starting objecter 2024-09-16T03:57:31.702 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.682+0000 7fc881357640 1 Processor -- start 2024-09-16T03:57:31.702 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.682+0000 7fc881357640 1 -- start start 2024-09-16T03:57:31.702 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.682+0000 7fc881357640 1 librados: setting wanted keys 2024-09-16T03:57:31.702 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.682+0000 7fc881357640 1 librados: calling monclient init 2024-09-16T03:57:31.702 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.682+0000 7fc881357640 10 monclient: init 2024-09-16T03:57:31.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.686+0000 7fc881357640 10 monclient: _reopen_session rank -1 2024-09-16T03:57:31.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.686+0000 7fc881357640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:57:31.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.686+0000 7fc881357640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc8680052f0 0x7fc87c1ccaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:57:31.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.686+0000 7fc881357640 10 monclient(hunting): picked mon.noname-c con 0x7fc8680052f0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:57:31.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.686+0000 7fc881357640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc87c13c6c0 0x7fc87c13cb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:57:31.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.686+0000 7fc881357640 10 monclient(hunting): picked mon.noname-a con 0x7fc87c13c6c0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:57:31.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.686+0000 7fc881357640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc87c137200 0x7fc87c1376b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:57:31.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.686+0000 7fc881357640 10 monclient(hunting): picked mon.noname-b con 0x7fc87c137200 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:57:31.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.686+0000 7fc881357640 10 monclient(hunting): start opening mon connection 2024-09-16T03:57:31.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.686+0000 7fc881357640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fc87c0a1aa0 con 0x7fc87c13c6c0 2024-09-16T03:57:31.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.686+0000 7fc881357640 10 monclient(hunting): start opening mon connection 2024-09-16T03:57:31.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.686+0000 7fc881357640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc87c0a1920 con 0x7fc87c137200 2024-09-16T03:57:31.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.686+0000 7fc881357640 10 monclient(hunting): start opening mon connection 2024-09-16T03:57:31.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.686+0000 7fc881357640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fc87c0a1c20 con 0x7fc8680052f0 2024-09-16T03:57:31.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.686+0000 7fc881357640 10 monclient(hunting): _renew_subs 2024-09-16T03:57:31.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.686+0000 7fc881357640 10 monclient(hunting): authenticate will time out at 1898.656371s 2024-09-16T03:57:31.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.686+0000 7fc87affd640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc8680052f0 0x7fc87c1ccaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:57:31.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.686+0000 7fc87a7fc640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc87c13c6c0 0x7fc87c13cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:57:31.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.686+0000 7fc87affd640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc8680052f0 0x7fc87c1ccaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:41828/0 (socket says 172.21.15.42:41828) 2024-09-16T03:57:31.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.686+0000 7fc87affd640 1 -- 172.21.15.42:0/177920199 learned_addr learned my addr 172.21.15.42:0/177920199 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:57:31.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.686+0000 7fc87affd640 10 monclient(hunting): get_auth_request con 0x7fc8680052f0 auth_method 0 2024-09-16T03:57:31.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.686+0000 7fc87affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:57:31.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.686+0000 7fc87affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:57:31.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.686+0000 7fc87affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:57:31.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.686+0000 7fc87a7fc640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc87c13c6c0 0x7fc87c13cb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:36412/0 (socket says 172.21.15.42:36412) 2024-09-16T03:57:31.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.686+0000 7fc87a7fc640 10 monclient(hunting): get_auth_request con 0x7fc87c13c6c0 auth_method 0 2024-09-16T03:57:31.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.686+0000 7fc87a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:57:31.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.686+0000 7fc87a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:57:31.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.686+0000 7fc87a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:57:31.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.686+0000 7fc87b7fe640 1 -- 172.21.15.42:0/177920199 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc87c137200 msgr2=0x7fc87c1376b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T03:57:31.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.686+0000 7fc87b7fe640 1 --2- 172.21.15.42:0/177920199 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc87c137200 0x7fc87c1376b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:57:31.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.686+0000 7fc87affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:57:31.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.686+0000 7fc87affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:57:31.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.686+0000 7fc87affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:57:31.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.686+0000 7fc87a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:57:31.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.686+0000 7fc87a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:57:31.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.686+0000 7fc87a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:57:31.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.686+0000 7fc87affd640 10 monclient(hunting): handle_auth_done global_id 854417 payload 824 2024-09-16T03:57:31.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.686+0000 7fc87affd640 1 -- 172.21.15.42:0/177920199 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc87c137200 msgr2=0x7fc87c1376b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:57:31.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.686+0000 7fc87affd640 1 --2- 172.21.15.42:0/177920199 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc87c137200 0x7fc87c1376b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:57:31.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.686+0000 7fc87affd640 1 -- 172.21.15.42:0/177920199 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc87c13c6c0 msgr2=0x7fc87c13cb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:57:31.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.686+0000 7fc87affd640 1 --2- 172.21.15.42:0/177920199 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc87c13c6c0 0x7fc87c13cb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:57:31.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.686+0000 7fc87affd640 10 monclient: _finish_hunting 0 2024-09-16T03:57:31.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.686+0000 7fc87affd640 1 monclient: found mon.noname-c 2024-09-16T03:57:31.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.686+0000 7fc87affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:57:31.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.686+0000 7fc87affd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:57:31.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.686+0000 7fc87affd640 1 -- 172.21.15.42:0/177920199 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc87c13abd0 con 0x7fc8680052f0 2024-09-16T03:57:31.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.686+0000 7fc87a7fc640 1 --2- 172.21.15.42:0/177920199 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc87c13c6c0 0x7fc87c13cb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:57:31.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.686+0000 7fc87affd640 1 --2- 172.21.15.42:0/177920199 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc8680052f0 0x7fc87c1ccaa0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fc86c00cbe0 tx=0x7fc86c00a430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:57:31.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.686+0000 7fc8737fe640 10 client.?.objecter ms_handle_connect 0x7fc8680052f0 2024-09-16T03:57:31.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.686+0000 7fc8737fe640 10 client.?.objecter resend_mon_ops 2024-09-16T03:57:31.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.686+0000 7fc8737fe640 1 -- 172.21.15.42:0/177920199 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc86c004d70 con 0x7fc8680052f0 2024-09-16T03:57:31.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.686+0000 7fc8737fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:57:31.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.686+0000 7fc8737fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:57:31.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.686+0000 7fc8737fe640 10 monclient: dump: 2024-09-16T03:57:31.712 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:57:31.712 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:57:31.712 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:57:31.712 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:57:31.712 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:57:31.712 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:57:31.712 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:57:31.712 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:57:31.712 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:57:31.712 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:57:31.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.686+0000 7fc8737fe640 10 monclient: _finish_auth 0 2024-09-16T03:57:31.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.686+0000 7fc8737fe640 10 monclient: _check_auth_tickets 2024-09-16T03:57:31.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.686+0000 7fc8737fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:57:31.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.686+0000 7fc8737fe640 1 -- 172.21.15.42:0/177920199 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc86c00ab40 con 0x7fc8680052f0 2024-09-16T03:57:31.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.686+0000 7fc881357640 5 monclient: authenticate success, global_id 854417 2024-09-16T03:57:31.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.686+0000 7fc881357640 10 monclient: _renew_subs 2024-09-16T03:57:31.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.686+0000 7fc881357640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:57:31.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.686+0000 7fc881357640 1 -- 172.21.15.42:0/177920199 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc87c13d540 con 0x7fc8680052f0 2024-09-16T03:57:31.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.686+0000 7fc8737fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:57:31.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.686+0000 7fc8737fe640 1 -- 172.21.15.42:0/177920199 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc86c0070d0 con 0x7fc8680052f0 2024-09-16T03:57:31.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.686+0000 7fc8737fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:57:31.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.686+0000 7fc8737fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:57:31.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.686+0000 7fc8737fe640 10 monclient: dump: 2024-09-16T03:57:31.713 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:57:31.714 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:57:31.714 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:57:31.714 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:57:31.714 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:57:31.714 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:57:31.714 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:57:31.714 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:57:31.714 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:57:31.714 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:57:31.714 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.686+0000 7fc8797fa640 4 set_mon_vals no callback set 2024-09-16T03:57:31.714 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.686+0000 7fc881357640 10 client.854417.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:57:31.714 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.686+0000 7fc881357640 10 monclient: _renew_subs 2024-09-16T03:57:31.714 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.686+0000 7fc881357640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:57:31.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.686+0000 7fc881357640 1 -- 172.21.15.42:0/177920199 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc87c073a30 con 0x7fc8680052f0 2024-09-16T03:57:31.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.686+0000 7fc881357640 1 librados: init done 2024-09-16T03:57:31.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.686+0000 7fc881357640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:57:31.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.686+0000 7fc881357640 1 --2- 172.21.15.42:0/177920199 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc87c02d7d0 0x7fc87c137bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:57:31.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.686+0000 7fc881357640 10 monclient(hunting): start opening mon connection 2024-09-16T03:57:31.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.686+0000 7fc881357640 1 -- 172.21.15.42:0/177920199 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fc87c13d080 con 0x7fc87c02d7d0 2024-09-16T03:57:31.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.686+0000 7fc87a7fc640 1 --2- 172.21.15.42:0/177920199 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc87c02d7d0 0x7fc87c137bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:57:31.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.686+0000 7fc8737fe640 1 -- 172.21.15.42:0/177920199 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fc86c011070 con 0x7fc8680052f0 2024-09-16T03:57:31.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.686+0000 7fc87a7fc640 10 monclient: get_auth_request con 0x7fc87c02d7d0 auth_method 0 2024-09-16T03:57:31.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.686+0000 7fc87a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:57:31.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.686+0000 7fc87a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:57:31.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.686+0000 7fc87a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:57:31.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.686+0000 7fc87a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:57:31.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.686+0000 7fc87a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:57:31.716 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.686+0000 7fc87a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:57:31.716 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.686+0000 7fc8737fe640 1 --2- 172.21.15.42:0/177920199 >> v2:172.21.15.154:6812/254845342 conn(0x7fc85c0377f0 0x7fc85c039cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:57:31.716 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.686+0000 7fc8737fe640 1 -- 172.21.15.42:0/177920199 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(461..461 src has 330..461) ==== 5714+0+0 (secure 0 0 0) 0x7fc86c049010 con 0x7fc8680052f0 2024-09-16T03:57:31.716 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.686+0000 7fc8737fe640 10 client.854417.objecter ms_dispatch 0x7fc87c000ba0 osd_map(461..461 src has 330..461) 2024-09-16T03:57:31.716 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.686+0000 7fc8737fe640 3 client.854417.objecter handle_osd_map got epochs [461,461] > 0 2024-09-16T03:57:31.716 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.686+0000 7fc8737fe640 3 client.854417.objecter handle_osd_map decoding full epoch 461 2024-09-16T03:57:31.716 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.686+0000 7fc87a7fc640 10 monclient(hunting): handle_auth_done global_id 854420 payload 824 2024-09-16T03:57:31.716 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.686+0000 7fc8737fe640 20 client.854417.objecter dump_active .. 0 homeless 2024-09-16T03:57:31.716 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.686+0000 7fc87a7fc640 1 -- 172.21.15.42:0/177920199 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc87c02dbb0 con 0x7fc87c02d7d0 2024-09-16T03:57:31.716 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.686+0000 7fc8737fe640 10 client.854417.objecter ms_handle_connect 0x7fc87c02d7d0 2024-09-16T03:57:31.716 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.686+0000 7fc8737fe640 10 client.854417.objecter resend_mon_ops 2024-09-16T03:57:31.716 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.686+0000 7fc87a7fc640 1 --2- 172.21.15.42:0/177920199 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc87c02d7d0 0x7fc87c137bf0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fc864003890 tx=0x7fc864009dd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:57:31.716 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.686+0000 7fc87b7fe640 1 --2- 172.21.15.42:0/177920199 >> v2:172.21.15.154:6812/254845342 conn(0x7fc85c0377f0 0x7fc85c039cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:57:31.717 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.686+0000 7fc8737fe640 1 -- 172.21.15.42:0/177920199 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc864011a90 con 0x7fc87c02d7d0 2024-09-16T03:57:31.717 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.686+0000 7fc8737fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:57:31.717 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.686+0000 7fc8737fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:57:31.717 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.686+0000 7fc8737fe640 10 monclient: dump: 2024-09-16T03:57:31.717 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:57:31.717 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:57:31.717 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:57:31.717 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:57:31.717 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:57:31.717 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:57:31.717 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:57:31.717 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:57:31.717 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:57:31.717 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:57:31.718 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.686+0000 7fc87b7fe640 10 monclient: get_auth_request con 0x7fc85c0377f0 auth_method 0 2024-09-16T03:57:31.718 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.686+0000 7fc8737fe640 1 -- 172.21.15.42:0/177920199 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fc864011a90 con 0x7fc87c02d7d0 2024-09-16T03:57:31.718 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.686+0000 7fc8737fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:57:31.718 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.686+0000 7fc8737fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:57:31.718 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.686+0000 7fc8737fe640 1 -- 172.21.15.42:0/177920199 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc87c02d7d0 msgr2=0x7fc87c137bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:57:31.718 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.686+0000 7fc8737fe640 1 --2- 172.21.15.42:0/177920199 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc87c02d7d0 0x7fc87c137bf0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fc864003890 tx=0x7fc864009dd0 comp rx=0 tx=0).stop 2024-09-16T03:57:31.718 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.686+0000 7fc8737fe640 1 -- 172.21.15.42:0/177920199 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc87c02d7d0 msgr2=0x7fc87c137bf0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:57:31.718 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.686+0000 7fc8737fe640 1 --2- 172.21.15.42:0/177920199 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc87c02d7d0 0x7fc87c137bf0 secure :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fc864003890 tx=0x7fc864009dd0 comp rx=0 tx=0).stop 2024-09-16T03:57:31.718 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.686+0000 7fc8737fe640 10 client.854417.objecter ms_handle_connect 0x7fc85c0377f0 2024-09-16T03:57:31.718 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.686+0000 7fc87b7fe640 1 --2- 172.21.15.42:0/177920199 >> v2:172.21.15.154:6812/254845342 conn(0x7fc85c0377f0 0x7fc85c039cb0 secure :-1 s=READY pgs=3683 cs=0 l=1 rev1=1 crypto rx=0x7fc86800c610 tx=0x7fc874008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:57:31.718 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.694+0000 7fc881357640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:57:31.718 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.694+0000 7fc881357640 1 --2- 172.21.15.42:0/177920199 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc87c062ea0 0x7fc87c0632b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:57:31.719 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.694+0000 7fc881357640 10 monclient(hunting): start opening mon connection 2024-09-16T03:57:31.719 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.694+0000 7fc881357640 1 -- 172.21.15.42:0/177920199 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fc87c1329e0 con 0x7fc87c062ea0 2024-09-16T03:57:31.719 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.694+0000 7fc87a7fc640 1 --2- 172.21.15.42:0/177920199 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc87c062ea0 0x7fc87c0632b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:57:31.719 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.694+0000 7fc87a7fc640 10 monclient: get_auth_request con 0x7fc87c062ea0 auth_method 0 2024-09-16T03:57:31.719 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.694+0000 7fc87a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:57:31.719 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.694+0000 7fc87a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:57:31.719 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.694+0000 7fc87a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:57:31.719 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.694+0000 7fc87a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:57:31.719 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.694+0000 7fc87a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:57:31.719 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.694+0000 7fc87a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:57:31.719 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.694+0000 7fc87a7fc640 10 monclient(hunting): handle_auth_done global_id 854423 payload 824 2024-09-16T03:57:31.719 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.694+0000 7fc87a7fc640 1 -- 172.21.15.42:0/177920199 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fc87c139780 con 0x7fc87c062ea0 2024-09-16T03:57:31.719 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.694+0000 7fc8737fe640 10 client.854417.objecter ms_handle_connect 0x7fc87c062ea0 2024-09-16T03:57:31.720 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.694+0000 7fc87a7fc640 1 --2- 172.21.15.42:0/177920199 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc87c062ea0 0x7fc87c0632b0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fc86401b000 tx=0x7fc864005950 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:57:31.720 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.694+0000 7fc8737fe640 10 client.854417.objecter resend_mon_ops 2024-09-16T03:57:31.720 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.694+0000 7fc8737fe640 1 -- 172.21.15.42:0/177920199 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc864014440 con 0x7fc87c062ea0 2024-09-16T03:57:31.720 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.694+0000 7fc8737fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:57:31.720 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.694+0000 7fc8737fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:57:31.720 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.694+0000 7fc8737fe640 10 monclient: dump: 2024-09-16T03:57:31.720 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:57:31.720 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:57:31.720 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:57:31.720 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:57:31.720 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:57:31.720 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:57:31.720 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:57:31.721 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:57:31.721 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:57:31.721 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:57:31.721 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.694+0000 7fc8737fe640 1 -- 172.21.15.42:0/177920199 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4535 (secure 0 0 0) 0x7fc864014440 con 0x7fc87c062ea0 2024-09-16T03:57:31.721 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.694+0000 7fc8737fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:57:31.721 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.694+0000 7fc8737fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:57:31.721 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.694+0000 7fc8737fe640 1 -- 172.21.15.42:0/177920199 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc87c062ea0 msgr2=0x7fc87c0632b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:57:31.721 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.694+0000 7fc8737fe640 1 --2- 172.21.15.42:0/177920199 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc87c062ea0 0x7fc87c0632b0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fc86401b000 tx=0x7fc864005950 comp rx=0 tx=0).stop 2024-09-16T03:57:31.721 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.694+0000 7fc8737fe640 1 -- 172.21.15.42:0/177920199 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc87c062ea0 msgr2=0x7fc87c0632b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:57:31.721 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.694+0000 7fc8737fe640 1 --2- 172.21.15.42:0/177920199 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc87c062ea0 0x7fc87c0632b0 secure :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fc86401b000 tx=0x7fc864005950 comp rx=0 tx=0).stop 2024-09-16T03:57:31.722 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:57:31.722 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:57:31.722 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:57:31.722 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T03:57:31.722 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 142, 2024-09-16T03:57:31.722 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 185130, 2024-09-16T03:57:31.722 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:57:31.722 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:57:31.722 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T03:57:31.722 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:57:31.722 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 1, 2024-09-16T03:57:31.722 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:57:31.723 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:57:31.723 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:57:31.723 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:57:31.723 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:57:31.723 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:57:31.723 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:57:31.723 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:57:31.723 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:57:31.723 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:57:31.723 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:57:31.723 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:57:31.723 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:57:31.723 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:57:31.723 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:57:31.724 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:57:31.724 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:57:31.724 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:57:31.724 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:57:31.724 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:57:31.724 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:57:31.724 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:57:31.724 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:57:31.724 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:57:31.724 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:57:31.724 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:57:31.724 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:57:31.724 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:57:31.725 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:57:31.725 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:57:31.725 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:57:31.725 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:57:31.725 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:57:31.725 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:57:31.725 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:57:31.725 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:57:31.725 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:57:31.725 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:57:31.725 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:57:31.725 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:57:31.725 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:57:31.725 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:57:31.726 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:57:31.726 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:57:31.726 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:57:31.726 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:57:31.726 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:57:31.726 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:57:31.726 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:57:31.726 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:57:31.726 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:57:31.726 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:57:31.726 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:57:31.726 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:57:31.726 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:57:31.726 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:57:31.727 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:57:31.727 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:57:31.727 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:57:31.727 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:57:31.727 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:57:31.727 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:57:31.727 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:57:31.727 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:57:31.727 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:57:31.727 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:57:31.728 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:57:31.728 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:57:31.728 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:57:31.728 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:57:31.728 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:57:31.728 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:57:31.728 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:57:31.728 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:57:31.728 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:57:31.728 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:57:31.728 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:57:31.728 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:57:31.728 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:57:31.729 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:57:31.729 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:57:31.729 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:57:31.729 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:57:31.729 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:57:31.729 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:57:31.729 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:57:31.729 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:57:31.729 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:57:31.729 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:57:31.729 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:57:31.729 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:57:31.729 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:57:31.730 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:57:31.730 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:57:31.730 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:57:31.730 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:57:31.730 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:57:31.730 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:57:31.730 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:57:31.730 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:57:31.730 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:57:31.730 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:57:31.730 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:57:31.730 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:57:31.730 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:57:31.730 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:57:31.731 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:57:31.731 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:57:31.731 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:57:31.731 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:57:31.731 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:57:31.731 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:57:31.731 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:57:31.731 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:57:31.731 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:57:31.731 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:57:31.731 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:57:31.731 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:57:31.731 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:57:31.732 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:57:31.732 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:57:31.732 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:57:31.732 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:57:31.732 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:57:31.732 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:57:31.732 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:57:31.732 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:57:31.732 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:57:31.732 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:57:31.732 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:57:31.732 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:57:31.732 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:57:31.732 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:57:31.733 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:57:31.733 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:57:31.733 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:57:31.733 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:57:31.733 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:57:31.733 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:57:31.733 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:57:31.733 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:57:31.733 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:57:31.733 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:57:31.733 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:57:31.733 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:57:31.733 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 2 2024-09-16T03:57:31.734 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:57:31.734 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:57:31.734 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:57:31.734 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:57:31.734 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:57:31.736 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.694+0000 7fc881357640 10 librados: watch_flush enter 2024-09-16T03:57:31.736 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.694+0000 7fc881357640 10 librados: watch_flush exit 2024-09-16T03:57:31.736 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.694+0000 7fc881357640 20 client.854417.objecter shutdown clearing up homeless session... 2024-09-16T03:57:31.736 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.694+0000 7fc881357640 10 client.854417.objecter successfully canceled tick 2024-09-16T03:57:31.736 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.694+0000 7fc881357640 1 -- 172.21.15.42:0/177920199 >> v2:172.21.15.154:6812/254845342 conn(0x7fc85c0377f0 msgr2=0x7fc85c039cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:57:31.736 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.694+0000 7fc881357640 1 --2- 172.21.15.42:0/177920199 >> v2:172.21.15.154:6812/254845342 conn(0x7fc85c0377f0 0x7fc85c039cb0 secure :-1 s=READY pgs=3683 cs=0 l=1 rev1=1 crypto rx=0x7fc86800c610 tx=0x7fc874008040 comp rx=0 tx=0).stop 2024-09-16T03:57:31.736 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.694+0000 7fc881357640 10 monclient: shutdown 2024-09-16T03:57:31.736 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.694+0000 7fc881357640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:57:31.736 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.694+0000 7fc881357640 1 -- 172.21.15.42:0/177920199 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc8680052f0 msgr2=0x7fc87c1ccaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:57:31.736 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.694+0000 7fc881357640 1 --2- 172.21.15.42:0/177920199 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc8680052f0 0x7fc87c1ccaa0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fc86c00cbe0 tx=0x7fc86c00a430 comp rx=0 tx=0).stop 2024-09-16T03:57:31.736 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.694+0000 7fc87b7fe640 1 -- 172.21.15.42:0/177920199 reap_dead start 2024-09-16T03:57:31.736 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.694+0000 7fc87b7fe640 1 -- 172.21.15.42:0/177920199 reap_dead start 2024-09-16T03:57:31.737 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.694+0000 7fc881357640 1 -- 172.21.15.42:0/177920199 shutdown_connections 2024-09-16T03:57:31.737 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.694+0000 7fc881357640 1 -- 172.21.15.42:0/177920199 >> 172.21.15.42:0/177920199 conn(0x7fc87c0a2420 msgr2=0x7fc87c0a3f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:57:31.737 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.694+0000 7fc881357640 1 -- 172.21.15.42:0/177920199 shutdown_connections 2024-09-16T03:57:31.737 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.694+0000 7fc881357640 1 -- 172.21.15.42:0/177920199 wait complete. 2024-09-16T03:57:31.737 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:57:31.694+0000 7fc881357640 1 librados: shutdown 2024-09-16T03:57:31.737 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T03:57:31.737 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-16T03:57:51.738 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-16T03:57:51.738 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-09-16T03:57:51.738 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-16T03:57:51.738 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-16T03:57:51.741 INFO:tasks.ceph.mon.b:Started 2024-09-16T03:57:51.741 INFO:tasks.mon_thrash:Sending STOP to mon c 2024-09-16T03:57:51.742 INFO:tasks.ceph.mon.c:Sent signal 19 2024-09-16T03:57:51.742 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-16T03:58:06.742 INFO:tasks.mon_thrash:Sending CONT to mon c 2024-09-16T03:58:06.743 INFO:tasks.ceph.mon.c:Sent signal 18 2024-09-16T03:58:06.743 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T03:58:06.743 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:58:06.874 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:06.869+0000 7f47edf65640 10 monclient: get_monmap_and_config 2024-09-16T03:58:06.875 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:06.869+0000 7f47edf65640 10 monclient: build_initial_monmap 2024-09-16T03:58:06.875 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:06.869+0000 7f47edf65640 10 monclient: monmap: 2024-09-16T03:58:06.875 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:58:06.875 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:58:06.875 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:58:06.873210+0000 2024-09-16T03:58:06.875 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:58:06.873210+0000 2024-09-16T03:58:06.875 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:58:06.875 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:06.875 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:58:06.875 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:58:06.875 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:58:06.876 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:58:06.876 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:58:06.876 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:06.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:06.869+0000 7f47edf65640 1 Processor -- start 2024-09-16T03:58:06.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:06.869+0000 7f47edf65640 1 -- start start 2024-09-16T03:58:06.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:06.869+0000 7f47edf65640 10 monclient: init 2024-09-16T03:58:06.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:06.869+0000 7f47edf65640 10 monclient: _reopen_session rank -1 2024-09-16T03:58:06.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:06.869+0000 7f47edf65640 10 monclient: _add_conns ranks=[2,0,3,4,1] 2024-09-16T03:58:06.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:06.869+0000 7f47edf65640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f47e80a1330 0x7f47e80a1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:06.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:06.869+0000 7f47edf65640 10 monclient(hunting): picked mon.noname-b con 0x7f47e80a1330 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:58:06.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:06.869+0000 7f47edf65640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f47e809c920 addr v1:172.21.15.42:6789/0 2024-09-16T03:58:06.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:06.869+0000 7f47edf65640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f47e8072d40 0x7f47e80a1c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:06.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:06.869+0000 7f47edf65640 10 monclient(hunting): picked mon.noname-a con 0x7f47e8072d40 addr v2:172.21.15.42:3300/0 2024-09-16T03:58:06.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:06.869+0000 7f47edf65640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f47e8072350 con 0x7f47e809c920 2024-09-16T03:58:06.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:06.869+0000 7f47edf65640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:06.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:06.869+0000 7f47edf65640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f47e8072b50 con 0x7f47e8072d40 2024-09-16T03:58:06.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:06.869+0000 7f47edf65640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:06.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:06.869+0000 7f47edf65640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f47e80a2190 con 0x7f47e80a1330 2024-09-16T03:58:06.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:06.869+0000 7f47edf65640 10 monclient(hunting): _renew_subs 2024-09-16T03:58:06.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:06.869+0000 7f47edf65640 10 monclient(hunting): authenticate will time out at 1933.842629s 2024-09-16T03:58:06.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:06.869+0000 7f47e77fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f47e80a1330 0x7f47e80a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:06.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:06.869+0000 7f47e7fff640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f47e8072d40 0x7f47e80a1c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:06.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:06.869+0000 7f47e6ffd640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f47e809c920 0x7f47e8137120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:46158/0 (socket says 172.21.15.42:46158) 2024-09-16T03:58:06.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:06.869+0000 7f47e7fff640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f47e8072d40 0x7f47e80a1c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:35904/0 (socket says 172.21.15.42:35904) 2024-09-16T03:58:06.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:06.869+0000 7f47e77fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f47e80a1330 0x7f47e80a1710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:48962/0 (socket says 172.21.15.42:48962) 2024-09-16T03:58:06.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:06.869+0000 7f47e6ffd640 1 -- 172.21.15.42:0/110423500 learned_addr learned my addr 172.21.15.42:0/110423500 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:58:06.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:06.869+0000 7f47e7fff640 10 monclient(hunting): get_auth_request con 0x7f47e8072d40 auth_method 0 2024-09-16T03:58:06.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:06.869+0000 7f47e7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:06.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:06.869+0000 7f47e7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:06.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:06.869+0000 7f47e7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:58:06.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:06.869+0000 7f47e77fe640 10 monclient(hunting): get_auth_request con 0x7f47e80a1330 auth_method 0 2024-09-16T03:58:06.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:06.869+0000 7f47e77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:06.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:06.869+0000 7f47e77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:06.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:06.869+0000 7f47e77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:58:06.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:06.869+0000 7f47e77fe640 1 -- 172.21.15.42:0/110423500 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f47e80a1330 msgr2=0x7f47e80a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T03:58:06.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:06.869+0000 7f47e7fff640 1 -- 172.21.15.42:0/110423500 >> v2:172.21.15.42:3300/0 conn(0x7f47e8072d40 msgr2=0x7f47e80a1c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T03:58:06.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:06.869+0000 7f47e77fe640 1 -- 172.21.15.42:0/110423500 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f47e80a1330 msgr2=0x7f47e80a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:58:06.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:06.869+0000 7f47e7fff640 1 -- 172.21.15.42:0/110423500 >> v2:172.21.15.42:3300/0 conn(0x7f47e8072d40 msgr2=0x7f47e80a1c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:58:06.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:06.869+0000 7f47e77fe640 1 --2- 172.21.15.42:0/110423500 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f47e80a1330 0x7f47e80a1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:58:06.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:06.869+0000 7f47e7fff640 1 --2- 172.21.15.42:0/110423500 >> v2:172.21.15.42:3300/0 conn(0x7f47e8072d40 0x7f47e80a1c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:58:06.883 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:06.869+0000 7f47e77fe640 1 --2- 172.21.15.42:0/110423500 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f47e80a1330 0x7f47e80a1710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:58:06.883 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:06.869+0000 7f47e7fff640 1 --2- 172.21.15.42:0/110423500 >> v2:172.21.15.42:3300/0 conn(0x7f47e8072d40 0x7f47e80a1c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:58:07.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:07.073+0000 7f47e77fe640 1 --2- 172.21.15.42:0/110423500 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f47e80a1330 0x7f47e80a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:07.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:07.073+0000 7f47e7fff640 1 --2- 172.21.15.42:0/110423500 >> v2:172.21.15.42:3300/0 conn(0x7f47e8072d40 0x7f47e80a1c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:07.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:07.073+0000 7f47e77fe640 10 monclient(hunting): get_auth_request con 0x7f47e80a1330 auth_method 0 2024-09-16T03:58:07.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:07.073+0000 7f47e77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:07.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:07.073+0000 7f47e77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:07.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:07.073+0000 7f47e77fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:58:07.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:07.073+0000 7f47e7fff640 10 monclient(hunting): get_auth_request con 0x7f47e8072d40 auth_method 0 2024-09-16T03:58:07.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:07.073+0000 7f47e7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:07.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:07.073+0000 7f47e7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:07.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:07.073+0000 7f47e7fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:58:07.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:07.073+0000 7f47e7fff640 1 -- 172.21.15.42:0/110423500 >> v2:172.21.15.42:3300/0 conn(0x7f47e8072d40 msgr2=0x7f47e80a1c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T03:58:07.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:07.073+0000 7f47e7fff640 1 -- 172.21.15.42:0/110423500 >> v2:172.21.15.42:3300/0 conn(0x7f47e8072d40 msgr2=0x7f47e80a1c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:58:07.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:07.073+0000 7f47e7fff640 1 --2- 172.21.15.42:0/110423500 >> v2:172.21.15.42:3300/0 conn(0x7f47e8072d40 0x7f47e80a1c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:58:07.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:07.073+0000 7f47e77fe640 1 -- 172.21.15.42:0/110423500 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f47e80a1330 msgr2=0x7f47e80a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T03:58:07.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:07.073+0000 7f47e77fe640 1 -- 172.21.15.42:0/110423500 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f47e80a1330 msgr2=0x7f47e80a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:58:07.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:07.073+0000 7f47e77fe640 1 --2- 172.21.15.42:0/110423500 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f47e80a1330 0x7f47e80a1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:58:07.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:07.073+0000 7f47e7fff640 1 --2- 172.21.15.42:0/110423500 >> v2:172.21.15.42:3300/0 conn(0x7f47e8072d40 0x7f47e80a1c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T03:58:07.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:07.073+0000 7f47e77fe640 1 --2- 172.21.15.42:0/110423500 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f47e80a1330 0x7f47e80a1710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T03:58:07.483 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:07.477+0000 7f47e77fe640 1 --2- 172.21.15.42:0/110423500 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f47e80a1330 0x7f47e80a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:07.483 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:07.477+0000 7f47e7fff640 1 --2- 172.21.15.42:0/110423500 >> v2:172.21.15.42:3300/0 conn(0x7f47e8072d40 0x7f47e80a1c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:07.483 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:07.477+0000 7f47e77fe640 10 monclient(hunting): get_auth_request con 0x7f47e80a1330 auth_method 0 2024-09-16T03:58:07.483 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:07.477+0000 7f47e77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:07.484 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:07.477+0000 7f47e77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:07.484 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:07.477+0000 7f47e77fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:58:07.484 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:07.477+0000 7f47e7fff640 10 monclient(hunting): get_auth_request con 0x7f47e8072d40 auth_method 0 2024-09-16T03:58:07.484 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:07.477+0000 7f47e7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:07.484 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:07.477+0000 7f47e7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:07.485 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:07.477+0000 7f47e7fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:58:07.485 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:07.477+0000 7f47e77fe640 1 -- 172.21.15.42:0/110423500 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f47e80a1330 msgr2=0x7f47e80a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T03:58:07.485 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:07.477+0000 7f47e77fe640 1 -- 172.21.15.42:0/110423500 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f47e80a1330 msgr2=0x7f47e80a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:58:07.485 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:07.477+0000 7f47e77fe640 1 --2- 172.21.15.42:0/110423500 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f47e80a1330 0x7f47e80a1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:58:07.485 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:07.477+0000 7f47e77fe640 1 --2- 172.21.15.42:0/110423500 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f47e80a1330 0x7f47e80a1710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T03:58:07.485 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:07.477+0000 7f47e7fff640 1 -- 172.21.15.42:0/110423500 >> v2:172.21.15.42:3300/0 conn(0x7f47e8072d40 msgr2=0x7f47e80a1c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T03:58:07.486 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:07.477+0000 7f47e7fff640 1 -- 172.21.15.42:0/110423500 >> v2:172.21.15.42:3300/0 conn(0x7f47e8072d40 msgr2=0x7f47e80a1c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:58:07.486 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:07.477+0000 7f47e7fff640 1 --2- 172.21.15.42:0/110423500 >> v2:172.21.15.42:3300/0 conn(0x7f47e8072d40 0x7f47e80a1c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:58:07.486 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:07.477+0000 7f47e7fff640 1 --2- 172.21.15.42:0/110423500 >> v2:172.21.15.42:3300/0 conn(0x7f47e8072d40 0x7f47e80a1c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T03:58:08.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:08.281+0000 7f47e7fff640 1 --2- 172.21.15.42:0/110423500 >> v2:172.21.15.42:3300/0 conn(0x7f47e8072d40 0x7f47e80a1c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:08.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:08.281+0000 7f47e7fff640 10 monclient(hunting): get_auth_request con 0x7f47e8072d40 auth_method 0 2024-09-16T03:58:08.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:08.281+0000 7f47e7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:08.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:08.281+0000 7f47e7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:08.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:08.281+0000 7f47e7fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:58:08.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:08.281+0000 7f47e77fe640 1 --2- 172.21.15.42:0/110423500 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f47e80a1330 0x7f47e80a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:08.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:08.281+0000 7f47e77fe640 10 monclient(hunting): get_auth_request con 0x7f47e80a1330 auth_method 0 2024-09-16T03:58:08.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:08.281+0000 7f47e77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:08.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:08.281+0000 7f47e77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:08.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:08.281+0000 7f47e77fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:58:08.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:08.281+0000 7f47e7fff640 1 -- 172.21.15.42:0/110423500 >> v2:172.21.15.42:3300/0 conn(0x7f47e8072d40 msgr2=0x7f47e80a1c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T03:58:08.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:08.281+0000 7f47e7fff640 1 -- 172.21.15.42:0/110423500 >> v2:172.21.15.42:3300/0 conn(0x7f47e8072d40 msgr2=0x7f47e80a1c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:58:08.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:08.281+0000 7f47e7fff640 1 --2- 172.21.15.42:0/110423500 >> v2:172.21.15.42:3300/0 conn(0x7f47e8072d40 0x7f47e80a1c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:58:08.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:08.281+0000 7f47e7fff640 1 --2- 172.21.15.42:0/110423500 >> v2:172.21.15.42:3300/0 conn(0x7f47e8072d40 0x7f47e80a1c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T03:58:08.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:08.281+0000 7f47e77fe640 1 -- 172.21.15.42:0/110423500 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f47e80a1330 msgr2=0x7f47e80a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T03:58:08.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:08.281+0000 7f47e77fe640 1 -- 172.21.15.42:0/110423500 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f47e80a1330 msgr2=0x7f47e80a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:58:08.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:08.281+0000 7f47e77fe640 1 --2- 172.21.15.42:0/110423500 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f47e80a1330 0x7f47e80a1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:58:08.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:08.281+0000 7f47e77fe640 1 --2- 172.21.15.42:0/110423500 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f47e80a1330 0x7f47e80a1710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T03:58:09.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:09.869+0000 7f47e57fa640 10 monclient(hunting): tick 2024-09-16T03:58:09.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:09.869+0000 7f47e57fa640 10 monclient(hunting): _check_auth_tickets 2024-09-16T03:58:09.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:09.869+0000 7f47e57fa640 1 monclient(hunting): continuing hunt 2024-09-16T03:58:09.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:09.869+0000 7f47e57fa640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T03:58:09.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:09.869+0000 7f47e57fa640 1 -- 172.21.15.42:0/110423500 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f47e80a1330 msgr2=0x7f47e80a1710 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:58:09.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:09.869+0000 7f47e57fa640 1 --2- 172.21.15.42:0/110423500 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f47e80a1330 0x7f47e80a1710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:09.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:09.869+0000 7f47e57fa640 1 -- 172.21.15.42:0/110423500 >> v2:172.21.15.42:3300/0 conn(0x7f47e8072d40 msgr2=0x7f47e80a1c50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:58:09.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:09.869+0000 7f47e57fa640 1 --2- 172.21.15.42:0/110423500 >> v2:172.21.15.42:3300/0 conn(0x7f47e8072d40 0x7f47e80a1c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:09.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:09.869+0000 7f47e57fa640 1 -- 172.21.15.42:0/110423500 >> v1:172.21.15.42:6789/0 conn(0x7f47e809c920 legacy=0x7f47e8137120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:09.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:09.869+0000 7f47e57fa640 10 monclient: _add_conns ranks=[1,4] 2024-09-16T03:58:09.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:09.869+0000 7f47e57fa640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f47d40036b0 addr v1:172.21.15.154:6789/0 2024-09-16T03:58:09.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:09.869+0000 7f47e57fa640 1 --2- 172.21.15.42:0/110423500 >> v2:172.21.15.154:3300/0 conn(0x7f47d4007400 0x7f47d40098c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:09.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:09.869+0000 7f47e57fa640 10 monclient(hunting): picked mon.noname-c con 0x7f47d4007400 addr v2:172.21.15.154:3300/0 2024-09-16T03:58:09.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:09.869+0000 7f47e57fa640 1 -- 172.21.15.42:0/110423500 --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f47e809bc60 con 0x7f47d40036b0 2024-09-16T03:58:09.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:09.869+0000 7f47e57fa640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:09.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:09.869+0000 7f47e57fa640 1 -- 172.21.15.42:0/110423500 --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f47e8072b50 con 0x7f47d4007400 2024-09-16T03:58:09.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:09.869+0000 7f47e57fa640 10 monclient(hunting): _renew_subs 2024-09-16T03:58:09.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:09.869+0000 7f47e6ffd640 1 --2- 172.21.15.42:0/110423500 >> v2:172.21.15.154:3300/0 conn(0x7f47d4007400 0x7f47d40098c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:09.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:09.869+0000 7f47e6ffd640 10 monclient(hunting): get_auth_request con 0x7f47d4007400 auth_method 0 2024-09-16T03:58:09.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:09.869+0000 7f47e6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:09.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:09.869+0000 7f47e6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:09.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:09.869+0000 7f47e6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:58:09.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:09.869+0000 7f47e6ffd640 1 -- 172.21.15.42:0/110423500 >> v2:172.21.15.154:3300/0 conn(0x7f47d4007400 msgr2=0x7f47d40098c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T03:58:09.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:09.869+0000 7f47e6ffd640 1 -- 172.21.15.42:0/110423500 >> v2:172.21.15.154:3300/0 conn(0x7f47d4007400 msgr2=0x7f47d40098c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:58:09.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:09.869+0000 7f47e6ffd640 1 --2- 172.21.15.42:0/110423500 >> v2:172.21.15.154:3300/0 conn(0x7f47d4007400 0x7f47d40098c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:58:09.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:09.873+0000 7f47e6ffd640 1 --2- 172.21.15.42:0/110423500 >> v2:172.21.15.154:3300/0 conn(0x7f47d4007400 0x7f47d40098c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:58:10.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:10.077+0000 7f47e6ffd640 1 --2- 172.21.15.42:0/110423500 >> v2:172.21.15.154:3300/0 conn(0x7f47d4007400 0x7f47d40098c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:10.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:10.077+0000 7f47e6ffd640 10 monclient(hunting): get_auth_request con 0x7f47d4007400 auth_method 0 2024-09-16T03:58:10.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:10.077+0000 7f47e6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:10.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:10.077+0000 7f47e6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:10.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:10.077+0000 7f47e6ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:58:10.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:10.077+0000 7f47e6ffd640 1 -- 172.21.15.42:0/110423500 >> v2:172.21.15.154:3300/0 conn(0x7f47d4007400 msgr2=0x7f47d40098c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T03:58:10.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:10.077+0000 7f47e6ffd640 1 -- 172.21.15.42:0/110423500 >> v2:172.21.15.154:3300/0 conn(0x7f47d4007400 msgr2=0x7f47d40098c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:58:10.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:10.077+0000 7f47e6ffd640 1 --2- 172.21.15.42:0/110423500 >> v2:172.21.15.154:3300/0 conn(0x7f47d4007400 0x7f47d40098c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:58:10.083 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:10.077+0000 7f47e6ffd640 1 --2- 172.21.15.42:0/110423500 >> v2:172.21.15.154:3300/0 conn(0x7f47d4007400 0x7f47d40098c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T03:58:10.486 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:10.481+0000 7f47e6ffd640 1 --2- 172.21.15.42:0/110423500 >> v2:172.21.15.154:3300/0 conn(0x7f47d4007400 0x7f47d40098c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:10.486 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:10.481+0000 7f47e6ffd640 10 monclient(hunting): get_auth_request con 0x7f47d4007400 auth_method 0 2024-09-16T03:58:10.486 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:10.481+0000 7f47e6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:10.486 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:10.481+0000 7f47e6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:10.486 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:10.481+0000 7f47e6ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:58:10.486 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:10.481+0000 7f47e6ffd640 1 -- 172.21.15.42:0/110423500 >> v2:172.21.15.154:3300/0 conn(0x7f47d4007400 msgr2=0x7f47d40098c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T03:58:10.486 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:10.481+0000 7f47e6ffd640 1 -- 172.21.15.42:0/110423500 >> v2:172.21.15.154:3300/0 conn(0x7f47d4007400 msgr2=0x7f47d40098c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:58:10.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:10.481+0000 7f47e6ffd640 1 --2- 172.21.15.42:0/110423500 >> v2:172.21.15.154:3300/0 conn(0x7f47d4007400 0x7f47d40098c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:58:10.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:10.481+0000 7f47e6ffd640 1 --2- 172.21.15.42:0/110423500 >> v2:172.21.15.154:3300/0 conn(0x7f47d4007400 0x7f47d40098c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T03:58:11.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.285+0000 7f47e6ffd640 1 --2- 172.21.15.42:0/110423500 >> v2:172.21.15.154:3300/0 conn(0x7f47d4007400 0x7f47d40098c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:11.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.285+0000 7f47e6ffd640 10 monclient(hunting): get_auth_request con 0x7f47d4007400 auth_method 0 2024-09-16T03:58:11.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.285+0000 7f47e6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:11.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.285+0000 7f47e6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:11.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.285+0000 7f47e6ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:58:11.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.285+0000 7f47e6ffd640 1 -- 172.21.15.42:0/110423500 >> v2:172.21.15.154:3300/0 conn(0x7f47d4007400 msgr2=0x7f47d40098c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T03:58:11.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.285+0000 7f47e6ffd640 1 -- 172.21.15.42:0/110423500 >> v2:172.21.15.154:3300/0 conn(0x7f47d4007400 msgr2=0x7f47d40098c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:58:11.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.285+0000 7f47e6ffd640 1 --2- 172.21.15.42:0/110423500 >> v2:172.21.15.154:3300/0 conn(0x7f47d4007400 0x7f47d40098c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:58:11.291 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.285+0000 7f47e6ffd640 1 --2- 172.21.15.42:0/110423500 >> v2:172.21.15.154:3300/0 conn(0x7f47d4007400 0x7f47d40098c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T03:58:11.590 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.585+0000 7f47e67fc640 1 -- 172.21.15.42:0/110423500 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3746007030 0 0) 0x7f47e809bc60 con 0x7f47d40036b0 2024-09-16T03:58:11.590 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.585+0000 7f47e67fc640 10 monclient(hunting): _negotiate 2024-09-16T03:58:11.590 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.585+0000 7f47e67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:11.590 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.585+0000 7f47e67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:58:11.590 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.585+0000 7f47e67fc640 10 monclient(hunting): my global_id is 904099 2024-09-16T03:58:11.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.585+0000 7f47e67fc640 1 -- 172.21.15.42:0/110423500 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f47d0003530 con 0x7f47d40036b0 2024-09-16T03:58:11.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.585+0000 7f47e67fc640 1 -- 172.21.15.42:0/110423500 <== mon.1 v1:172.21.15.154:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2803181587 0 0) 0x7f47d0003530 con 0x7f47d40036b0 2024-09-16T03:58:11.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.585+0000 7f47e67fc640 1 -- 172.21.15.42:0/110423500 >> v2:172.21.15.154:3300/0 conn(0x7f47d4007400 msgr2=0x7f47d40098c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:58:11.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.585+0000 7f47e67fc640 1 --2- 172.21.15.42:0/110423500 >> v2:172.21.15.154:3300/0 conn(0x7f47d4007400 0x7f47d40098c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:11.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.585+0000 7f47e67fc640 10 monclient: _finish_hunting 0 2024-09-16T03:58:11.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.585+0000 7f47e67fc640 1 monclient: found mon.noname-c-legacy 2024-09-16T03:58:11.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.585+0000 7f47e67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:58:11.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.585+0000 7f47e67fc640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.154:6789/0 2024-09-16T03:58:11.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.585+0000 7f47e67fc640 1 -- 172.21.15.42:0/110423500 --> v1:172.21.15.154:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47d400afd0 con 0x7f47d40036b0 2024-09-16T03:58:11.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.585+0000 7f47e67fc640 10 monclient: _finish_auth 0 2024-09-16T03:58:11.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.585+0000 7f47e67fc640 10 monclient: _check_auth_tickets 2024-09-16T03:58:11.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.585+0000 7f47e67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:58:11.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.585+0000 7f47e67fc640 1 -- 172.21.15.42:0/110423500 <== mon.1 v1:172.21.15.154:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2177413734 0 0) 0x7f47d800f040 con 0x7f47d40036b0 2024-09-16T03:58:11.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.585+0000 7f47e67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:58:11.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.585+0000 7f47e67fc640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-16T03:58:11.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.585+0000 7f47e67fc640 10 monclient: dump: 2024-09-16T03:58:11.595 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:58:11.595 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:58:11.595 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:11.595 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:11.595 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:58:11.595 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:11.595 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:58:11.596 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:58:11.596 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:58:11.596 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:11.596 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.585+0000 7f47e67fc640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] but i'm connected to v1:172.21.15.154:6789/0, reconnecting 2024-09-16T03:58:11.596 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.585+0000 7f47e67fc640 10 monclient: _reopen_session rank -1 2024-09-16T03:58:11.596 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.585+0000 7f47e67fc640 1 -- 172.21.15.42:0/110423500 >> v1:172.21.15.154:6789/0 conn(0x7f47d40036b0 legacy=0x7f47d4005b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:11.596 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.585+0000 7f47e7fff640 1 -- 172.21.15.42:0/110423500 reap_dead start 2024-09-16T03:58:11.596 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.585+0000 7f47e67fc640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:58:11.596 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.585+0000 7f47e67fc640 1 --2- 172.21.15.42:0/110423500 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f47d0003a10 0x7f47d0003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:11.596 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.585+0000 7f47e67fc640 10 monclient(hunting): picked mon.c con 0x7f47d0003a10 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:58:11.596 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.585+0000 7f47e67fc640 1 --2- 172.21.15.42:0/110423500 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f47d0004810 0x7f47d0004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:11.596 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.585+0000 7f47e67fc640 10 monclient(hunting): picked mon.b con 0x7f47d0004810 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:58:11.596 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.585+0000 7f47e67fc640 1 --2- 172.21.15.42:0/110423500 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f47d00053d0 0x7f47d000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:11.597 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.585+0000 7f47e67fc640 10 monclient(hunting): picked mon.a con 0x7f47d00053d0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:58:11.597 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.585+0000 7f47e67fc640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:11.597 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.585+0000 7f47e67fc640 1 -- 172.21.15.42:0/110423500 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f47e8072b50 con 0x7f47d00053d0 2024-09-16T03:58:11.597 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.585+0000 7f47e67fc640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:11.597 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.585+0000 7f47e67fc640 1 -- 172.21.15.42:0/110423500 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f47d000c290 con 0x7f47d0004810 2024-09-16T03:58:11.597 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.585+0000 7f47e67fc640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:11.597 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.585+0000 7f47e67fc640 1 -- 172.21.15.42:0/110423500 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f47d000c490 con 0x7f47d0003a10 2024-09-16T03:58:11.597 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.585+0000 7f47e67fc640 10 monclient(hunting): _renew_subs 2024-09-16T03:58:11.597 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.585+0000 7f47e67fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:58:11.597 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.585+0000 7f47e77fe640 1 --2- 172.21.15.42:0/110423500 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f47d0003a10 0x7f47d0003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:11.597 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.585+0000 7f47e77fe640 10 monclient(hunting): get_auth_request con 0x7f47d0003a10 auth_method 0 2024-09-16T03:58:11.597 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.585+0000 7f47e77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:11.597 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.585+0000 7f47e77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:11.597 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.585+0000 7f47e77fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:58:11.598 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.585+0000 7f47e7fff640 1 --2- 172.21.15.42:0/110423500 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f47d00053d0 0x7f47d000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:11.598 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.585+0000 7f47e6ffd640 1 --2- 172.21.15.42:0/110423500 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f47d0004810 0x7f47d0004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:11.598 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.585+0000 7f47e6ffd640 10 monclient(hunting): get_auth_request con 0x7f47d0004810 auth_method 0 2024-09-16T03:58:11.598 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.585+0000 7f47e6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:11.598 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.585+0000 7f47e6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:11.598 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.585+0000 7f47e6ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:58:11.598 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.585+0000 7f47e7fff640 10 monclient(hunting): get_auth_request con 0x7f47d00053d0 auth_method 0 2024-09-16T03:58:11.598 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.585+0000 7f47e7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:11.598 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.585+0000 7f47e7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:11.598 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.585+0000 7f47e7fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:58:11.598 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.585+0000 7f47e77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:58:11.598 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.585+0000 7f47e77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:58:11.598 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.585+0000 7f47e77fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T03:58:11.598 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.585+0000 7f47e6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:58:11.599 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.585+0000 7f47e6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:58:11.599 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.585+0000 7f47e6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T03:58:11.599 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.585+0000 7f47e7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:58:11.599 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.585+0000 7f47e7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:58:11.599 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.585+0000 7f47e7fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T03:58:11.599 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.585+0000 7f47e77fe640 10 monclient(hunting): handle_auth_done global_id 904099 payload 293 2024-09-16T03:58:11.599 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.585+0000 7f47e77fe640 1 -- 172.21.15.42:0/110423500 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f47d0004810 msgr2=0x7f47d0004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:58:11.599 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.585+0000 7f47e77fe640 1 --2- 172.21.15.42:0/110423500 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f47d0004810 0x7f47d0004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:11.599 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.585+0000 7f47e77fe640 1 -- 172.21.15.42:0/110423500 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f47d00053d0 msgr2=0x7f47d000b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:58:11.599 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.585+0000 7f47e77fe640 1 --2- 172.21.15.42:0/110423500 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f47d00053d0 0x7f47d000b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:11.599 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.585+0000 7f47e77fe640 10 monclient: _finish_hunting 0 2024-09-16T03:58:11.599 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.585+0000 7f47e77fe640 1 monclient: found mon.c 2024-09-16T03:58:11.599 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.585+0000 7f47e77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:58:11.600 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.585+0000 7f47e77fe640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:58:11.600 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.585+0000 7f47e77fe640 1 -- 172.21.15.42:0/110423500 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47d0003530 con 0x7f47d0003a10 2024-09-16T03:58:11.600 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.585+0000 7f47e77fe640 10 monclient: _finish_auth 0 2024-09-16T03:58:11.600 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.585+0000 7f47e77fe640 10 monclient: _check_auth_tickets 2024-09-16T03:58:11.600 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.585+0000 7f47e77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:58:11.600 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.585+0000 7f47edf65640 5 monclient: authenticate success, global_id 904099 2024-09-16T03:58:11.600 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.585+0000 7f47edf65640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:58:11.605 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.589+0000 7f47e77fe640 1 --2- 172.21.15.42:0/110423500 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f47d0003a10 0x7f47d0003e10 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f47d800eff0 tx=0x7f47d801ffa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=314005e87f886486 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:58:11.605 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.589+0000 7f47e67fc640 1 -- 172.21.15.42:0/110423500 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47d80af070 con 0x7f47d0003a10 2024-09-16T03:58:11.605 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.589+0000 7f47e67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:58:11.605 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.589+0000 7f47e67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:58:11.605 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.589+0000 7f47e67fc640 10 monclient: dump: 2024-09-16T03:58:11.605 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:58:11.606 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:58:11.606 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:11.606 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:11.606 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:58:11.606 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:11.606 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:58:11.606 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:58:11.606 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:58:11.606 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:11.606 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.589+0000 7f47edf65640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:58:11.606 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.589+0000 7f47e67fc640 1 -- 172.21.15.42:0/110423500 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f47d80a9b70 con 0x7f47d0003a10 2024-09-16T03:58:11.606 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.589+0000 7f47e67fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:58:11.606 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.589+0000 7f47edf65640 10 monclient: get_monmap_and_config success 2024-09-16T03:58:11.606 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.589+0000 7f47edf65640 4 set_mon_vals no callback set 2024-09-16T03:58:11.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.589+0000 7f47e67fc640 1 -- 172.21.15.42:0/110423500 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47d800ec40 con 0x7f47d0003a10 2024-09-16T03:58:11.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.589+0000 7f47e67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:58:11.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.589+0000 7f47e67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:58:11.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.589+0000 7f47e67fc640 10 monclient: dump: 2024-09-16T03:58:11.607 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:58:11.607 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:58:11.607 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:11.607 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:11.607 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:58:11.607 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:11.607 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:58:11.607 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:58:11.607 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:58:11.607 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:11.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.589+0000 7f47edf65640 10 monclient: shutdown 2024-09-16T03:58:11.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.589+0000 7f47edf65640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:58:11.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.589+0000 7f47edf65640 1 -- 172.21.15.42:0/110423500 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f47d0003a10 msgr2=0x7f47d0003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:11.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.589+0000 7f47edf65640 1 --2- 172.21.15.42:0/110423500 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f47d0003a10 0x7f47d0003e10 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f47d800eff0 tx=0x7f47d801ffa0 comp rx=0 tx=0).stop 2024-09-16T03:58:11.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.589+0000 7f47edf65640 1 -- 172.21.15.42:0/110423500 shutdown_connections 2024-09-16T03:58:11.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.589+0000 7f47edf65640 1 --2- 172.21.15.42:0/110423500 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f47d00053d0 0x7f47d000b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:11.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.589+0000 7f47edf65640 1 --2- 172.21.15.42:0/110423500 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f47d0004810 0x7f47d0004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:11.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.589+0000 7f47edf65640 1 --2- 172.21.15.42:0/110423500 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f47d0003a10 0x7f47d0003e10 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:11.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.589+0000 7f47edf65640 1 -- 172.21.15.42:0/110423500 >> 172.21.15.42:0/110423500 conn(0x7f47e80a2420 msgr2=0x7f47e80a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:58:11.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.593+0000 7f47edf65640 1 -- 172.21.15.42:0/110423500 shutdown_connections 2024-09-16T03:58:11.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.593+0000 7f47edf65640 1 -- 172.21.15.42:0/110423500 wait complete. 2024-09-16T03:58:11.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.593+0000 7f47edf65640 10 monclient: build_initial_monmap 2024-09-16T03:58:11.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.593+0000 7f47edf65640 10 monclient: monmap: 2024-09-16T03:58:11.608 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:58:11.609 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:58:11.609 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:58:11.609 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:58:11.609 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:58:11.609 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:11.609 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:58:11.609 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:58:11.609 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:58:11.609 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:11.609 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.593+0000 7f47edf65640 1 librados: starting msgr at 2024-09-16T03:58:11.609 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.593+0000 7f47edf65640 1 librados: starting objecter 2024-09-16T03:58:11.609 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.593+0000 7f47edf65640 1 Processor -- start 2024-09-16T03:58:11.609 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.593+0000 7f47edf65640 1 -- start start 2024-09-16T03:58:11.609 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.593+0000 7f47edf65640 1 librados: setting wanted keys 2024-09-16T03:58:11.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.593+0000 7f47edf65640 1 librados: calling monclient init 2024-09-16T03:58:11.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.593+0000 7f47edf65640 10 monclient: init 2024-09-16T03:58:11.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.593+0000 7f47edf65640 10 monclient: _reopen_session rank -1 2024-09-16T03:58:11.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.593+0000 7f47edf65640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T03:58:11.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.593+0000 7f47edf65640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f47d0003a10 0x7f47e8136460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:11.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.593+0000 7f47edf65640 10 monclient(hunting): picked mon.noname-a con 0x7f47d0003a10 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:58:11.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.593+0000 7f47edf65640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f47d0004810 0x7f47e81369a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:11.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.593+0000 7f47edf65640 10 monclient(hunting): picked mon.noname-b con 0x7f47d0004810 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:58:11.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.593+0000 7f47edf65640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f47d00053d0 0x7f47e811ff10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:11.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.593+0000 7f47edf65640 10 monclient(hunting): picked mon.noname-c con 0x7f47d00053d0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:58:11.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.593+0000 7f47edf65640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:11.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.593+0000 7f47edf65640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f47e80a1710 con 0x7f47d0003a10 2024-09-16T03:58:11.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.593+0000 7f47edf65640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:11.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.593+0000 7f47edf65640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f47e809bfa0 con 0x7f47d0004810 2024-09-16T03:58:11.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.593+0000 7f47edf65640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:11.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.593+0000 7f47edf65640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f47e80a1890 con 0x7f47d00053d0 2024-09-16T03:58:11.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.593+0000 7f47edf65640 10 monclient(hunting): _renew_subs 2024-09-16T03:58:11.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.593+0000 7f47edf65640 10 monclient(hunting): authenticate will time out at 1938.566593s 2024-09-16T03:58:11.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.593+0000 7f47e77fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f47d0003a10 0x7f47e8136460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:11.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.593+0000 7f47e77fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f47d0003a10 0x7f47e8136460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:35962/0 (socket says 172.21.15.42:35962) 2024-09-16T03:58:11.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.593+0000 7f47e77fe640 1 -- 172.21.15.42:0/1111057885 learned_addr learned my addr 172.21.15.42:0/1111057885 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:58:11.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.593+0000 7f47e77fe640 10 monclient(hunting): get_auth_request con 0x7f47d0003a10 auth_method 0 2024-09-16T03:58:11.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.593+0000 7f47e77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:11.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.593+0000 7f47e77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:11.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.593+0000 7f47e77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:58:11.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.593+0000 7f47e7fff640 1 --2- 172.21.15.42:0/1111057885 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f47d00053d0 0x7f47e811ff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:11.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.593+0000 7f47e77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:58:11.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.593+0000 7f47e77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:58:11.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.593+0000 7f47e77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:58:11.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.593+0000 7f47e7fff640 10 monclient(hunting): get_auth_request con 0x7f47d00053d0 auth_method 0 2024-09-16T03:58:11.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.593+0000 7f47e7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:11.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.593+0000 7f47e7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:11.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.593+0000 7f47e7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:58:11.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.593+0000 7f47e6ffd640 1 --2- 172.21.15.42:0/1111057885 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f47d0004810 0x7f47e81369a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:11.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.593+0000 7f47e6ffd640 10 monclient(hunting): get_auth_request con 0x7f47d0004810 auth_method 0 2024-09-16T03:58:11.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.593+0000 7f47e6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:11.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.593+0000 7f47e6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:11.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.593+0000 7f47e6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:58:11.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.593+0000 7f47e7fff640 1 -- 172.21.15.42:0/1111057885 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f47d00053d0 msgr2=0x7f47e811ff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-16T03:58:11.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.593+0000 7f47e7fff640 1 -- 172.21.15.42:0/1111057885 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f47d00053d0 msgr2=0x7f47e811ff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T03:58:11.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.593+0000 7f47e7fff640 1 --2- 172.21.15.42:0/1111057885 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f47d00053d0 0x7f47e811ff10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:58:11.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.593+0000 7f47e7fff640 1 --2- 172.21.15.42:0/1111057885 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f47d00053d0 0x7f47e811ff10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:58:11.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.593+0000 7f47e77fe640 10 monclient(hunting): handle_auth_done global_id 824907 payload 824 2024-09-16T03:58:11.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.593+0000 7f47e77fe640 1 -- 172.21.15.42:0/1111057885 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f47d00053d0 msgr2=0x7f47e811ff10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:58:11.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.593+0000 7f47e77fe640 1 --2- 172.21.15.42:0/1111057885 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f47d00053d0 0x7f47e811ff10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:11.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.593+0000 7f47e77fe640 1 -- 172.21.15.42:0/1111057885 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f47d0004810 msgr2=0x7f47e81369a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:11.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.593+0000 7f47e77fe640 1 --2- 172.21.15.42:0/1111057885 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f47d0004810 0x7f47e81369a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:11.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.593+0000 7f47e77fe640 10 monclient: _finish_hunting 0 2024-09-16T03:58:11.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.593+0000 7f47e77fe640 1 monclient: found mon.noname-a 2024-09-16T03:58:11.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.593+0000 7f47e77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:58:11.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.593+0000 7f47e77fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:58:11.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.593+0000 7f47e77fe640 1 -- 172.21.15.42:0/1111057885 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47e8121b30 con 0x7f47d0003a10 2024-09-16T03:58:11.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.593+0000 7f47e6ffd640 1 --2- 172.21.15.42:0/1111057885 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f47d0004810 0x7f47e81369a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T03:58:11.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.593+0000 7f47e77fe640 1 --2- 172.21.15.42:0/1111057885 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f47d0003a10 0x7f47e8136460 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f47d800eff0 tx=0x7f47d801cda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:58:11.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.593+0000 7f47cffff640 10 client.?.objecter ms_handle_connect 0x7f47d0003a10 2024-09-16T03:58:11.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.593+0000 7f47cffff640 10 client.?.objecter resend_mon_ops 2024-09-16T03:58:11.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.593+0000 7f47cffff640 1 -- 172.21.15.42:0/1111057885 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47d80c2070 con 0x7f47d0003a10 2024-09-16T03:58:11.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.593+0000 7f47cffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:58:11.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.593+0000 7f47cffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:58:11.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.593+0000 7f47cffff640 10 monclient: dump: 2024-09-16T03:58:11.614 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:58:11.614 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:58:11.614 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:11.614 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:11.614 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:58:11.614 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:11.614 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:58:11.614 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:58:11.615 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:58:11.615 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:11.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.593+0000 7f47cffff640 10 monclient: _finish_auth 0 2024-09-16T03:58:11.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.593+0000 7f47cffff640 10 monclient: _check_auth_tickets 2024-09-16T03:58:11.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.593+0000 7f47cffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:58:11.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.593+0000 7f47cffff640 1 -- 172.21.15.42:0/1111057885 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f47d80bd430 con 0x7f47d0003a10 2024-09-16T03:58:11.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.593+0000 7f47cffff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:58:11.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.593+0000 7f47cffff640 1 -- 172.21.15.42:0/1111057885 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47d80af040 con 0x7f47d0003a10 2024-09-16T03:58:11.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.593+0000 7f47edf65640 5 monclient: authenticate success, global_id 824907 2024-09-16T03:58:11.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.593+0000 7f47edf65640 10 monclient: _renew_subs 2024-09-16T03:58:11.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.593+0000 7f47edf65640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:58:11.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.593+0000 7f47edf65640 1 -- 172.21.15.42:0/1111057885 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47e8122690 con 0x7f47d0003a10 2024-09-16T03:58:11.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.593+0000 7f47cffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:58:11.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.593+0000 7f47e5ffb640 4 set_mon_vals no callback set 2024-09-16T03:58:11.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.593+0000 7f47cffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:58:11.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.593+0000 7f47cffff640 10 monclient: dump: 2024-09-16T03:58:11.616 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:58:11.616 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:58:11.616 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:11.616 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:11.616 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:58:11.616 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:11.616 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:58:11.616 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:58:11.617 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:58:11.617 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:11.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.593+0000 7f47edf65640 10 client.824907.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:58:11.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.593+0000 7f47edf65640 10 monclient: _renew_subs 2024-09-16T03:58:11.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.593+0000 7f47edf65640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:58:11.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.593+0000 7f47edf65640 1 -- 172.21.15.42:0/1111057885 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47e8073a30 con 0x7f47d0003a10 2024-09-16T03:58:11.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.593+0000 7f47edf65640 1 librados: init done 2024-09-16T03:58:11.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.597+0000 7f47edf65640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:58:11.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.597+0000 7f47edf65640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:58:11.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.597+0000 7f47edf65640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:58:11.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.597+0000 7f47edf65640 1 -- 172.21.15.42:0/1111057885 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f47e80a1c60 con 0x7f47d0003a10 2024-09-16T03:58:11.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.597+0000 7f47cffff640 1 -- 172.21.15.42:0/1111057885 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f47d80c5070 con 0x7f47d0003a10 2024-09-16T03:58:11.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.597+0000 7f47cffff640 1 --2- 172.21.15.42:0/1111057885 >> v2:172.21.15.154:6812/254845342 conn(0x7f47d4037f90 0x7f47d403a450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:11.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.597+0000 7f47cffff640 1 -- 172.21.15.42:0/1111057885 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(482..482 src has 366..482) ==== 5795+0+0 (secure 0 0 0) 0x7f47d80e7700 con 0x7f47d0003a10 2024-09-16T03:58:11.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.597+0000 7f47cffff640 10 client.824907.objecter ms_dispatch 0x7f47e8000ba0 osd_map(482..482 src has 366..482) 2024-09-16T03:58:11.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.597+0000 7f47cffff640 3 client.824907.objecter handle_osd_map got epochs [482,482] > 0 2024-09-16T03:58:11.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.597+0000 7f47cffff640 3 client.824907.objecter handle_osd_map decoding full epoch 482 2024-09-16T03:58:11.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.597+0000 7f47cffff640 20 client.824907.objecter dump_active .. 0 homeless 2024-09-16T03:58:11.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.597+0000 7f47e6ffd640 1 --2- 172.21.15.42:0/1111057885 >> v2:172.21.15.154:6812/254845342 conn(0x7f47d4037f90 0x7f47d403a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:11.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.597+0000 7f47e6ffd640 10 monclient: get_auth_request con 0x7f47d4037f90 auth_method 0 2024-09-16T03:58:11.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.597+0000 7f47cffff640 10 client.824907.objecter ms_handle_connect 0x7f47d4037f90 2024-09-16T03:58:11.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.597+0000 7f47e6ffd640 1 --2- 172.21.15.42:0/1111057885 >> v2:172.21.15.154:6812/254845342 conn(0x7f47d4037f90 0x7f47d403a450 secure :-1 s=READY pgs=3688 cs=0 l=1 rev1=1 crypto rx=0x7f47e8121960 tx=0x7f47dc009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:58:11.619 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.601+0000 7f47cffff640 1 -- 172.21.15.42:0/1111057885 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f47d80cc070 con 0x7f47d0003a10 2024-09-16T03:58:11.619 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.601+0000 7f47cffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:58:11.619 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.601+0000 7f47cffff640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:58:11.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.833+0000 7f47edf65640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:58:11.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.833+0000 7f47edf65640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:58:11.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.833+0000 7f47edf65640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:58:11.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.833+0000 7f47edf65640 1 -- 172.21.15.42:0/1111057885 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f47e80a3c00 con 0x7f47d0003a10 2024-09-16T03:58:11.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.861+0000 7f47cffff640 1 -- 172.21.15.42:0/1111057885 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7f47d8019d10 con 0x7f47d0003a10 2024-09-16T03:58:11.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.861+0000 7f47cffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:58:11.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.861+0000 7f47cffff640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:58:11.867 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":148,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:58:11.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.861+0000 7f47edf65640 10 librados: watch_flush enter 2024-09-16T03:58:11.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.865+0000 7f47edf65640 10 librados: watch_flush exit 2024-09-16T03:58:11.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.865+0000 7f47edf65640 20 client.824907.objecter shutdown clearing up homeless session... 2024-09-16T03:58:11.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.865+0000 7f47edf65640 10 client.824907.objecter successfully canceled tick 2024-09-16T03:58:11.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.865+0000 7f47edf65640 1 -- 172.21.15.42:0/1111057885 >> v2:172.21.15.154:6812/254845342 conn(0x7f47d4037f90 msgr2=0x7f47d403a450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:11.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.865+0000 7f47edf65640 1 --2- 172.21.15.42:0/1111057885 >> v2:172.21.15.154:6812/254845342 conn(0x7f47d4037f90 0x7f47d403a450 secure :-1 s=READY pgs=3688 cs=0 l=1 rev1=1 crypto rx=0x7f47e8121960 tx=0x7f47dc009040 comp rx=0 tx=0).stop 2024-09-16T03:58:11.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.865+0000 7f47edf65640 10 monclient: shutdown 2024-09-16T03:58:11.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.865+0000 7f47edf65640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:58:11.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.865+0000 7f47edf65640 1 -- 172.21.15.42:0/1111057885 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f47d0003a10 msgr2=0x7f47e8136460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:11.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.865+0000 7f47edf65640 1 --2- 172.21.15.42:0/1111057885 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f47d0003a10 0x7f47e8136460 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f47d800eff0 tx=0x7f47d801cda0 comp rx=0 tx=0).stop 2024-09-16T03:58:11.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.865+0000 7f47edf65640 1 -- 172.21.15.42:0/1111057885 shutdown_connections 2024-09-16T03:58:11.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.865+0000 7f47edf65640 1 --2- 172.21.15.42:0/1111057885 >> v2:172.21.15.154:6812/254845342 conn(0x7f47d4037f90 0x7f47d403a450 unknown :-1 s=CLOSED pgs=3688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:11.871 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.865+0000 7f47edf65640 1 --2- 172.21.15.42:0/1111057885 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f47d00053d0 0x7f47e811ff10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:11.871 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.865+0000 7f47edf65640 1 --2- 172.21.15.42:0/1111057885 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f47d0004810 0x7f47e81369a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:11.871 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.865+0000 7f47edf65640 1 --2- 172.21.15.42:0/1111057885 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f47d0003a10 0x7f47e8136460 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:11.871 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.865+0000 7f47edf65640 1 -- 172.21.15.42:0/1111057885 >> 172.21.15.42:0/1111057885 conn(0x7f47e80a2420 msgr2=0x7f47e81375e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:58:11.871 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.865+0000 7f47edf65640 1 -- 172.21.15.42:0/1111057885 shutdown_connections 2024-09-16T03:58:11.871 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.865+0000 7f47edf65640 1 -- 172.21.15.42:0/1111057885 wait complete. 2024-09-16T03:58:11.871 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:11.865+0000 7f47edf65640 1 librados: shutdown 2024-09-16T03:58:11.888 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T03:58:11.888 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T03:58:11.889 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T03:58:12.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.009+0000 7f1072c9e640 10 monclient: get_monmap_and_config 2024-09-16T03:58:12.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.009+0000 7f1072c9e640 10 monclient: build_initial_monmap 2024-09-16T03:58:12.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.009+0000 7f1072c9e640 10 monclient: monmap: 2024-09-16T03:58:12.014 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:58:12.014 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:58:12.014 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:58:12.012800+0000 2024-09-16T03:58:12.015 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:58:12.012800+0000 2024-09-16T03:58:12.015 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:58:12.015 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:12.015 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:58:12.015 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:58:12.015 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:58:12.015 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:58:12.015 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:58:12.015 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:12.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.009+0000 7f1072c9e640 1 Processor -- start 2024-09-16T03:58:12.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.009+0000 7f1072c9e640 1 -- start start 2024-09-16T03:58:12.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.009+0000 7f1072c9e640 10 monclient: init 2024-09-16T03:58:12.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.009+0000 7f1072c9e640 10 monclient: _reopen_session rank -1 2024-09-16T03:58:12.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.009+0000 7f1072c9e640 10 monclient: _add_conns ranks=[0,1,4,3,2] 2024-09-16T03:58:12.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.009+0000 7f1072c9e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f106c0a1330 addr v1:172.21.15.42:6789/0 2024-09-16T03:58:12.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.009+0000 7f1072c9e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f106c0a1710 addr v1:172.21.15.154:6789/0 2024-09-16T03:58:12.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.009+0000 7f1072c9e640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f106c13a3b0 0x7f106c13c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:12.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.009+0000 7f1072c9e640 10 monclient(hunting): picked mon.noname-c con 0x7f106c13a3b0 addr v2:172.21.15.154:3300/0 2024-09-16T03:58:12.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.009+0000 7f1072c9e640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f106c09e4c0 con 0x7f106c0a1330 2024-09-16T03:58:12.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.009+0000 7f1072c9e640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f106c0a3170 con 0x7f106c0a1710 2024-09-16T03:58:12.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.009+0000 7f1072c9e640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:12.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.009+0000 7f1072c9e640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f106c09edf0 con 0x7f106c13a3b0 2024-09-16T03:58:12.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.009+0000 7f1072c9e640 10 monclient(hunting): _renew_subs 2024-09-16T03:58:12.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.009+0000 7f1072c9e640 10 monclient(hunting): authenticate will time out at 1938.982203s 2024-09-16T03:58:12.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.009+0000 7f1070a3a640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f106c0a1330 0x7f106c09f180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:46164/0 (socket says 172.21.15.42:46164) 2024-09-16T03:58:12.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.009+0000 7f1070a3a640 1 -- 172.21.15.42:0/2841943599 learned_addr learned my addr 172.21.15.42:0/2841943599 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:58:12.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.009+0000 7f107123b640 1 --2- 172.21.15.42:0/2841943599 >> v2:172.21.15.154:3300/0 conn(0x7f106c13a3b0 0x7f106c13c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:12.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.009+0000 7f107123b640 10 monclient(hunting): get_auth_request con 0x7f106c13a3b0 auth_method 0 2024-09-16T03:58:12.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.009+0000 7f107123b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:12.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.009+0000 7f107123b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:12.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.009+0000 7f107123b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:58:12.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.009+0000 7f106b7fe640 1 -- 172.21.15.42:0/2841943599 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2200796857 0 0) 0x7f106c09e4c0 con 0x7f106c0a1330 2024-09-16T03:58:12.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.009+0000 7f106b7fe640 10 monclient(hunting): _negotiate 2024-09-16T03:58:12.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.009+0000 7f106b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:12.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.009+0000 7f106b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:58:12.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.009+0000 7f106b7fe640 10 monclient(hunting): my global_id is 824910 2024-09-16T03:58:12.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.009+0000 7f106b7fe640 1 -- 172.21.15.42:0/2841943599 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1058003530 con 0x7f106c0a1330 2024-09-16T03:58:12.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.009+0000 7f107123b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:58:12.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.009+0000 7f107123b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:58:12.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.009+0000 7f107123b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:58:12.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.009+0000 7f106b7fe640 1 -- 172.21.15.42:0/2841943599 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1423291640 0 0) 0x7f1058003530 con 0x7f106c0a1330 2024-09-16T03:58:12.029 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.009+0000 7f106b7fe640 1 -- 172.21.15.42:0/2841943599 >> v2:172.21.15.154:3300/0 conn(0x7f106c13a3b0 msgr2=0x7f106c13c7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:58:12.029 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.009+0000 7f106b7fe640 1 --2- 172.21.15.42:0/2841943599 >> v2:172.21.15.154:3300/0 conn(0x7f106c13a3b0 0x7f106c13c7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:12.029 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.009+0000 7f106b7fe640 1 -- 172.21.15.42:0/2841943599 >> v1:172.21.15.154:6789/0 conn(0x7f106c0a1710 legacy=0x7f106c0a1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:12.029 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.009+0000 7f106b7fe640 10 monclient: _finish_hunting 0 2024-09-16T03:58:12.029 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.009+0000 7f106b7fe640 1 monclient: found mon.noname-a-legacy 2024-09-16T03:58:12.029 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.009+0000 7f106b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:58:12.029 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.009+0000 7f106b7fe640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.42:6789/0 2024-09-16T03:58:12.029 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.009+0000 7f106b7fe640 1 -- 172.21.15.42:0/2841943599 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f106c09d030 con 0x7f106c0a1330 2024-09-16T03:58:12.029 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.009+0000 7f106b7fe640 10 monclient: _finish_auth 0 2024-09-16T03:58:12.029 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.009+0000 7f106b7fe640 10 monclient: _check_auth_tickets 2024-09-16T03:58:12.029 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.009+0000 7f106b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:58:12.029 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.009+0000 7f106b7fe640 1 -- 172.21.15.42:0/2841943599 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2177413734 0 0) 0x7f105c001a00 con 0x7f106c0a1330 2024-09-16T03:58:12.029 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.009+0000 7f1072c9e640 5 monclient: authenticate success, global_id 824910 2024-09-16T03:58:12.029 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.009+0000 7f1072c9e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:58:12.030 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.009+0000 7f106b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:58:12.030 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.009+0000 7f106b7fe640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T03:58:12.030 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.009+0000 7f106b7fe640 10 monclient: dump: 2024-09-16T03:58:12.030 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:58:12.030 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:58:12.030 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:12.030 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:12.030 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:58:12.030 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:12.030 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:58:12.030 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:58:12.030 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:58:12.030 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:12.030 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.009+0000 7f106b7fe640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] but i'm connected to v1:172.21.15.42:6789/0, reconnecting 2024-09-16T03:58:12.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.009+0000 7f106b7fe640 10 monclient: _reopen_session rank -1 2024-09-16T03:58:12.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.009+0000 7f106b7fe640 1 -- 172.21.15.42:0/2841943599 >> v1:172.21.15.42:6789/0 conn(0x7f106c0a1330 legacy=0x7f106c09f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:12.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.009+0000 7f106b7fe640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:58:12.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.009+0000 7f106b7fe640 1 --2- 172.21.15.42:0/2841943599 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f10580039c0 0x7f1058003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:12.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.009+0000 7f106b7fe640 10 monclient(hunting): picked mon.c con 0x7f10580039c0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:58:12.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.009+0000 7f106b7fe640 1 --2- 172.21.15.42:0/2841943599 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1058004810 0x7f1058004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:12.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.009+0000 7f106b7fe640 10 monclient(hunting): picked mon.b con 0x7f1058004810 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:58:12.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.009+0000 7f106b7fe640 1 --2- 172.21.15.42:0/2841943599 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f10580052f0 0x7f105800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:12.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.009+0000 7f106b7fe640 10 monclient(hunting): picked mon.a con 0x7f10580052f0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:58:12.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.009+0000 7f106b7fe640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:12.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.009+0000 7f106b7fe640 1 -- 172.21.15.42:0/2841943599 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f106c09edf0 con 0x7f10580052f0 2024-09-16T03:58:12.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.009+0000 7f106b7fe640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:12.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.009+0000 7f106b7fe640 1 -- 172.21.15.42:0/2841943599 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f105800c1c0 con 0x7f1058004810 2024-09-16T03:58:12.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.009+0000 7f106b7fe640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:12.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.009+0000 7f106b7fe640 1 -- 172.21.15.42:0/2841943599 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f105800c3c0 con 0x7f10580039c0 2024-09-16T03:58:12.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.009+0000 7f106b7fe640 10 monclient(hunting): _renew_subs 2024-09-16T03:58:12.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.009+0000 7f106b7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:58:12.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.009+0000 7f1072c9e640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-16T03:58:12.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.009+0000 7f1070a3a640 1 --2- 172.21.15.42:0/2841943599 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f10580039c0 0x7f1058003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:12.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.009+0000 7f1070a3a640 10 monclient(hunting): get_auth_request con 0x7f10580039c0 auth_method 0 2024-09-16T03:58:12.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.009+0000 7f1070a3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:12.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.009+0000 7f1070a3a640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:12.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.009+0000 7f1070a3a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:58:12.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.009+0000 7f107123b640 1 --2- 172.21.15.42:0/2841943599 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f10580052f0 0x7f105800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:12.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.009+0000 7f107123b640 10 monclient(hunting): get_auth_request con 0x7f10580052f0 auth_method 0 2024-09-16T03:58:12.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.009+0000 7f107123b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:12.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.009+0000 7f107123b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:12.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.009+0000 7f107123b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:58:12.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.009+0000 7f1070a3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:58:12.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.009+0000 7f1070a3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:58:12.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.009+0000 7f1070a3a640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T03:58:12.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.009+0000 7f107123b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:58:12.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.009+0000 7f107123b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:58:12.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.009+0000 7f107123b640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T03:58:12.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.009+0000 7f106bfff640 1 --2- 172.21.15.42:0/2841943599 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1058004810 0x7f1058004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:12.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.009+0000 7f106bfff640 10 monclient(hunting): get_auth_request con 0x7f1058004810 auth_method 0 2024-09-16T03:58:12.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.009+0000 7f106bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:12.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.009+0000 7f106bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:12.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.009+0000 7f106bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:58:12.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.009+0000 7f1070a3a640 10 monclient(hunting): handle_auth_done global_id 824910 payload 293 2024-09-16T03:58:12.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.009+0000 7f1070a3a640 1 -- 172.21.15.42:0/2841943599 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1058004810 msgr2=0x7f1058004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:58:12.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.009+0000 7f1070a3a640 1 --2- 172.21.15.42:0/2841943599 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1058004810 0x7f1058004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:12.034 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.009+0000 7f1070a3a640 1 -- 172.21.15.42:0/2841943599 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f10580052f0 msgr2=0x7f105800b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:58:12.034 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.009+0000 7f1070a3a640 1 --2- 172.21.15.42:0/2841943599 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f10580052f0 0x7f105800b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:12.034 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.009+0000 7f1070a3a640 10 monclient: _finish_hunting 0 2024-09-16T03:58:12.034 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.009+0000 7f1070a3a640 1 monclient: found mon.c 2024-09-16T03:58:12.034 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.009+0000 7f1070a3a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:58:12.034 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.009+0000 7f1070a3a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:58:12.034 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.009+0000 7f1070a3a640 1 -- 172.21.15.42:0/2841943599 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1058003530 con 0x7f10580039c0 2024-09-16T03:58:12.034 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.009+0000 7f1070a3a640 10 monclient: _finish_auth 0 2024-09-16T03:58:12.034 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.009+0000 7f1070a3a640 10 monclient: _check_auth_tickets 2024-09-16T03:58:12.034 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.009+0000 7f1070a3a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:58:12.034 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.009+0000 7f107123b640 1 --2- 172.21.15.42:0/2841943599 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f10580052f0 0x7f105800b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:58:12.034 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.009+0000 7f107123b640 1 -- 172.21.15.42:0/2841943599 reap_dead start 2024-09-16T03:58:12.034 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.013+0000 7f1070a3a640 1 --2- 172.21.15.42:0/2841943599 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f10580039c0 0x7f1058003de0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f105c002240 tx=0x7f105c01d2c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1c7f1eed4992d809 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:58:12.034 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.013+0000 7f106b7fe640 1 -- 172.21.15.42:0/2841943599 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f105c00a4a0 con 0x7f10580039c0 2024-09-16T03:58:12.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.013+0000 7f106b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:58:12.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.013+0000 7f106b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:58:12.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.013+0000 7f106b7fe640 10 monclient: dump: 2024-09-16T03:58:12.035 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:58:12.035 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:58:12.035 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:12.035 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:12.035 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:58:12.035 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:12.035 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:58:12.035 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:58:12.035 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:58:12.035 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:12.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.013+0000 7f106b7fe640 1 -- 172.21.15.42:0/2841943599 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f105c0ad650 con 0x7f10580039c0 2024-09-16T03:58:12.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.013+0000 7f1072c9e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:58:12.036 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.013+0000 7f106b7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:58:12.036 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.013+0000 7f1072c9e640 10 monclient: get_monmap_and_config success 2024-09-16T03:58:12.036 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.013+0000 7f106b7fe640 1 -- 172.21.15.42:0/2841943599 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f105c0ada80 con 0x7f10580039c0 2024-09-16T03:58:12.036 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.013+0000 7f1072c9e640 4 set_mon_vals no callback set 2024-09-16T03:58:12.036 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.013+0000 7f106b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:58:12.036 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.013+0000 7f106b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:58:12.036 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.013+0000 7f106b7fe640 10 monclient: dump: 2024-09-16T03:58:12.036 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:58:12.036 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:58:12.036 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:12.036 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:12.036 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:58:12.036 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:12.036 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:58:12.037 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:58:12.037 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:58:12.037 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:12.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.013+0000 7f1072c9e640 10 monclient: shutdown 2024-09-16T03:58:12.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.013+0000 7f1072c9e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:58:12.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.013+0000 7f1072c9e640 1 -- 172.21.15.42:0/2841943599 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f10580039c0 msgr2=0x7f1058003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:12.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.013+0000 7f1072c9e640 1 --2- 172.21.15.42:0/2841943599 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f10580039c0 0x7f1058003de0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f105c002240 tx=0x7f105c01d2c0 comp rx=0 tx=0).stop 2024-09-16T03:58:12.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.013+0000 7f1072c9e640 1 -- 172.21.15.42:0/2841943599 shutdown_connections 2024-09-16T03:58:12.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.013+0000 7f1072c9e640 1 --2- 172.21.15.42:0/2841943599 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f10580039c0 0x7f1058003de0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:12.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.013+0000 7f1072c9e640 1 -- 172.21.15.42:0/2841943599 >> 172.21.15.42:0/2841943599 conn(0x7f106c0a2420 msgr2=0x7f106c0a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:58:12.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.013+0000 7f1072c9e640 1 -- 172.21.15.42:0/2841943599 shutdown_connections 2024-09-16T03:58:12.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.013+0000 7f1072c9e640 1 -- 172.21.15.42:0/2841943599 wait complete. 2024-09-16T03:58:12.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.013+0000 7f1072c9e640 10 monclient: build_initial_monmap 2024-09-16T03:58:12.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.013+0000 7f1072c9e640 10 monclient: monmap: 2024-09-16T03:58:12.038 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:58:12.038 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:58:12.038 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:58:12.038 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:58:12.038 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:58:12.038 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:12.038 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:58:12.038 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:58:12.038 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:58:12.038 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:12.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.013+0000 7f1072c9e640 1 librados: starting msgr at 2024-09-16T03:58:12.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.013+0000 7f1072c9e640 1 librados: starting objecter 2024-09-16T03:58:12.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.013+0000 7f1072c9e640 1 Processor -- start 2024-09-16T03:58:12.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.013+0000 7f1072c9e640 1 -- start start 2024-09-16T03:58:12.039 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.013+0000 7f1072c9e640 1 librados: setting wanted keys 2024-09-16T03:58:12.039 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.013+0000 7f1072c9e640 1 librados: calling monclient init 2024-09-16T03:58:12.039 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.013+0000 7f1072c9e640 10 monclient: init 2024-09-16T03:58:12.039 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.017+0000 7f1072c9e640 10 monclient: _reopen_session rank -1 2024-09-16T03:58:12.039 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.017+0000 7f1072c9e640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:58:12.039 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.017+0000 7f1072c9e640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f10580039c0 0x7f106c1cf800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:12.039 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.017+0000 7f1072c9e640 10 monclient(hunting): picked mon.noname-a con 0x7f10580039c0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:58:12.039 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.017+0000 7f1072c9e640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f106c1cfd40 0x7f106c120920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:12.039 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.017+0000 7f1072c9e640 10 monclient(hunting): picked mon.noname-c con 0x7f106c1cfd40 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:58:12.039 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.017+0000 7f1072c9e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f106c1d0180 0x7f106c120e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:12.039 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.017+0000 7f1072c9e640 10 monclient(hunting): picked mon.noname-b con 0x7f106c1d0180 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:58:12.039 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.017+0000 7f1072c9e640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:12.039 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.017+0000 7f1072c9e640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f106c134480 con 0x7f10580039c0 2024-09-16T03:58:12.039 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.017+0000 7f1072c9e640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:12.040 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.017+0000 7f1072c9e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f106c134300 con 0x7f106c1d0180 2024-09-16T03:58:12.040 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.017+0000 7f1072c9e640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:12.040 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.017+0000 7f1072c9e640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f106c134600 con 0x7f106c1cfd40 2024-09-16T03:58:12.040 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.017+0000 7f1072c9e640 10 monclient(hunting): _renew_subs 2024-09-16T03:58:12.040 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.017+0000 7f1072c9e640 10 monclient(hunting): authenticate will time out at 1938.988340s 2024-09-16T03:58:12.040 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.017+0000 7f1070a3a640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f10580039c0 0x7f106c1cf800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:12.040 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.017+0000 7f1070a3a640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f10580039c0 0x7f106c1cf800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:35982/0 (socket says 172.21.15.42:35982) 2024-09-16T03:58:12.040 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.017+0000 7f106bfff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f106c1cfd40 0x7f106c120920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:12.040 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.017+0000 7f1070a3a640 1 -- 172.21.15.42:0/534455802 learned_addr learned my addr 172.21.15.42:0/534455802 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:58:12.040 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.017+0000 7f1070a3a640 10 monclient(hunting): get_auth_request con 0x7f10580039c0 auth_method 0 2024-09-16T03:58:12.040 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.017+0000 7f1070a3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:12.040 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.017+0000 7f1070a3a640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:12.040 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.017+0000 7f1070a3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:58:12.040 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.017+0000 7f106bfff640 10 monclient(hunting): get_auth_request con 0x7f106c1cfd40 auth_method 0 2024-09-16T03:58:12.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.017+0000 7f106bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:12.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.017+0000 7f106bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:12.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.017+0000 7f106bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:58:12.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.017+0000 7f1070a3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:58:12.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.017+0000 7f1070a3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:58:12.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.017+0000 7f1070a3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:58:12.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.017+0000 7f106bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:58:12.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.017+0000 7f106bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:58:12.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.017+0000 7f106bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:58:12.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.017+0000 7f107123b640 1 --2- 172.21.15.42:0/534455802 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f106c1d0180 0x7f106c120e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:12.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.017+0000 7f107123b640 10 monclient(hunting): get_auth_request con 0x7f106c1d0180 auth_method 0 2024-09-16T03:58:12.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.017+0000 7f107123b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:12.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.017+0000 7f107123b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:12.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.017+0000 7f107123b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:58:12.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.017+0000 7f106bfff640 10 monclient(hunting): handle_auth_done global_id 854435 payload 824 2024-09-16T03:58:12.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.017+0000 7f106bfff640 1 -- 172.21.15.42:0/534455802 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f106c1d0180 msgr2=0x7f106c120e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:12.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.017+0000 7f106bfff640 1 --2- 172.21.15.42:0/534455802 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f106c1d0180 0x7f106c120e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:12.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.017+0000 7f106bfff640 1 -- 172.21.15.42:0/534455802 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f10580039c0 msgr2=0x7f106c1cf800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:12.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.017+0000 7f106bfff640 1 --2- 172.21.15.42:0/534455802 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f10580039c0 0x7f106c1cf800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:12.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.017+0000 7f106bfff640 10 monclient: _finish_hunting 0 2024-09-16T03:58:12.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.017+0000 7f106bfff640 1 monclient: found mon.noname-c 2024-09-16T03:58:12.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.017+0000 7f106bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:58:12.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.017+0000 7f106bfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:58:12.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.017+0000 7f106bfff640 1 -- 172.21.15.42:0/534455802 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f106c1245d0 con 0x7f106c1cfd40 2024-09-16T03:58:12.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.017+0000 7f1070a3a640 1 --2- 172.21.15.42:0/534455802 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f10580039c0 0x7f106c1cf800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:58:12.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.017+0000 7f107123b640 1 --2- 172.21.15.42:0/534455802 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f106c1d0180 0x7f106c120e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T03:58:12.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.017+0000 7f106bfff640 1 --2- 172.21.15.42:0/534455802 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f106c1cfd40 0x7f106c120920 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f105400e440 tx=0x7f105400e900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:58:12.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.017+0000 7f10697fa640 10 client.?.objecter ms_handle_connect 0x7f106c1cfd40 2024-09-16T03:58:12.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.017+0000 7f10697fa640 10 client.?.objecter resend_mon_ops 2024-09-16T03:58:12.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.017+0000 7f10697fa640 1 -- 172.21.15.42:0/534455802 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10540052e0 con 0x7f106c1cfd40 2024-09-16T03:58:12.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.017+0000 7f10697fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:58:12.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.017+0000 7f10697fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:58:12.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.017+0000 7f10697fa640 10 monclient: dump: 2024-09-16T03:58:12.043 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:58:12.043 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:58:12.043 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:12.043 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:12.043 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:58:12.043 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:12.043 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:58:12.043 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:58:12.043 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:58:12.043 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:12.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.017+0000 7f10697fa640 10 monclient: _finish_auth 0 2024-09-16T03:58:12.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.017+0000 7f10697fa640 10 monclient: _check_auth_tickets 2024-09-16T03:58:12.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.017+0000 7f10697fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:58:12.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.017+0000 7f10697fa640 1 -- 172.21.15.42:0/534455802 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1054007020 con 0x7f106c1cfd40 2024-09-16T03:58:12.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.017+0000 7f1072c9e640 5 monclient: authenticate success, global_id 854435 2024-09-16T03:58:12.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.017+0000 7f1072c9e640 10 monclient: _renew_subs 2024-09-16T03:58:12.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.017+0000 7f1072c9e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:58:12.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.017+0000 7f1072c9e640 1 -- 172.21.15.42:0/534455802 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f106c122a20 con 0x7f106c1cfd40 2024-09-16T03:58:12.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.017+0000 7f10697fa640 10 monclient: handle_config config(0 keys) 2024-09-16T03:58:12.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.017+0000 7f10697fa640 1 -- 172.21.15.42:0/534455802 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1054005480 con 0x7f106c1cfd40 2024-09-16T03:58:12.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.017+0000 7f10697fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:58:12.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.017+0000 7f10697fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:58:12.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.017+0000 7f106affd640 4 set_mon_vals no callback set 2024-09-16T03:58:12.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.017+0000 7f10697fa640 10 monclient: dump: 2024-09-16T03:58:12.045 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:58:12.045 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:58:12.045 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:12.045 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:12.045 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:58:12.045 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:12.045 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:58:12.045 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:58:12.045 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:58:12.045 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:12.045 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.017+0000 7f1072c9e640 10 client.854435.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:58:12.045 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.017+0000 7f1072c9e640 10 monclient: _renew_subs 2024-09-16T03:58:12.045 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.017+0000 7f1072c9e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:58:12.045 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.017+0000 7f1072c9e640 1 -- 172.21.15.42:0/534455802 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f106c0a4830 con 0x7f106c1cfd40 2024-09-16T03:58:12.045 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.017+0000 7f1072c9e640 1 librados: init done 2024-09-16T03:58:12.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.017+0000 7f1072c9e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:58:12.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.017+0000 7f1072c9e640 1 --2- 172.21.15.42:0/534455802 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f106c1213a0 0x7f106c121780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:12.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.017+0000 7f1072c9e640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:12.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.017+0000 7f1072c9e640 1 -- 172.21.15.42:0/534455802 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f106c133170 con 0x7f106c1213a0 2024-09-16T03:58:12.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.017+0000 7f1070a3a640 1 --2- 172.21.15.42:0/534455802 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f106c1213a0 0x7f106c121780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:12.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.017+0000 7f1070a3a640 10 monclient: get_auth_request con 0x7f106c1213a0 auth_method 0 2024-09-16T03:58:12.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.017+0000 7f1070a3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:12.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.017+0000 7f1070a3a640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:12.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.017+0000 7f1070a3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:58:12.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.017+0000 7f1070a3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:58:12.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.017+0000 7f1070a3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:58:12.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.017+0000 7f10697fa640 1 -- 172.21.15.42:0/534455802 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f105401a020 con 0x7f106c1cfd40 2024-09-16T03:58:12.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.017+0000 7f1070a3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:58:12.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.017+0000 7f1070a3a640 10 monclient(hunting): handle_auth_done global_id 824916 payload 824 2024-09-16T03:58:12.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.017+0000 7f1070a3a640 1 -- 172.21.15.42:0/534455802 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f106c0a4d60 con 0x7f106c1213a0 2024-09-16T03:58:12.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.017+0000 7f10697fa640 1 --2- 172.21.15.42:0/534455802 >> v2:172.21.15.154:6812/254845342 conn(0x7f104c037810 0x7f104c039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:12.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.017+0000 7f10697fa640 1 -- 172.21.15.42:0/534455802 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(482..482 src has 366..482) ==== 5795+0+0 (secure 0 0 0) 0x7f1054048520 con 0x7f106c1cfd40 2024-09-16T03:58:12.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.017+0000 7f10697fa640 10 client.854435.objecter ms_dispatch 0x7f106c000ba0 osd_map(482..482 src has 366..482) 2024-09-16T03:58:12.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.017+0000 7f10697fa640 3 client.854435.objecter handle_osd_map got epochs [482,482] > 0 2024-09-16T03:58:12.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.017+0000 7f10697fa640 3 client.854435.objecter handle_osd_map decoding full epoch 482 2024-09-16T03:58:12.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.017+0000 7f1070a3a640 1 --2- 172.21.15.42:0/534455802 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f106c1213a0 0x7f106c121780 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f105c002240 tx=0x7f1058005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:58:12.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.017+0000 7f10697fa640 20 client.854435.objecter dump_active .. 0 homeless 2024-09-16T03:58:12.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.017+0000 7f10697fa640 10 client.854435.objecter ms_handle_connect 0x7f106c1213a0 2024-09-16T03:58:12.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.017+0000 7f10697fa640 10 client.854435.objecter resend_mon_ops 2024-09-16T03:58:12.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.017+0000 7f10697fa640 1 -- 172.21.15.42:0/534455802 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f105c0be070 con 0x7f106c1213a0 2024-09-16T03:58:12.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.017+0000 7f10697fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:58:12.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.017+0000 7f10697fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:58:12.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.017+0000 7f10697fa640 10 monclient: dump: 2024-09-16T03:58:12.048 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:58:12.048 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:58:12.048 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:12.048 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:12.048 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:58:12.048 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:12.048 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:58:12.048 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:58:12.048 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:58:12.048 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:12.048 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.017+0000 7f107123b640 1 --2- 172.21.15.42:0/534455802 >> v2:172.21.15.154:6812/254845342 conn(0x7f104c037810 0x7f104c039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:12.048 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.017+0000 7f107123b640 10 monclient: get_auth_request con 0x7f104c037810 auth_method 0 2024-09-16T03:58:12.048 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.017+0000 7f10697fa640 1 -- 172.21.15.42:0/534455802 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f105c0b3040 con 0x7f106c1213a0 2024-09-16T03:58:12.048 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.017+0000 7f10697fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:58:12.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.017+0000 7f10697fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:58:12.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.017+0000 7f10697fa640 1 -- 172.21.15.42:0/534455802 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f106c1213a0 msgr2=0x7f106c121780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:12.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.017+0000 7f10697fa640 1 --2- 172.21.15.42:0/534455802 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f106c1213a0 0x7f106c121780 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f105c002240 tx=0x7f1058005840 comp rx=0 tx=0).stop 2024-09-16T03:58:12.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.017+0000 7f10697fa640 1 -- 172.21.15.42:0/534455802 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f106c1213a0 msgr2=0x7f106c121780 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:58:12.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.017+0000 7f10697fa640 1 --2- 172.21.15.42:0/534455802 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f106c1213a0 0x7f106c121780 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:12.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.017+0000 7f10697fa640 10 client.854435.objecter ms_handle_connect 0x7f104c037810 2024-09-16T03:58:12.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.017+0000 7f107123b640 1 --2- 172.21.15.42:0/534455802 >> v2:172.21.15.154:6812/254845342 conn(0x7f104c037810 0x7f104c039cd0 secure :-1 s=READY pgs=3689 cs=0 l=1 rev1=1 crypto rx=0x7f106c0a1af0 tx=0x7f1064008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:58:12.050 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.025+0000 7f1072c9e640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:58:12.050 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.025+0000 7f1072c9e640 1 --2- 172.21.15.42:0/534455802 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f106c06e910 0x7f106c06ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:12.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.025+0000 7f1072c9e640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:12.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.025+0000 7f1072c9e640 1 -- 172.21.15.42:0/534455802 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f106c134990 con 0x7f106c06e910 2024-09-16T03:58:12.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.025+0000 7f1070a3a640 1 --2- 172.21.15.42:0/534455802 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f106c06e910 0x7f106c06ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:12.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.025+0000 7f1070a3a640 10 monclient: get_auth_request con 0x7f106c06e910 auth_method 0 2024-09-16T03:58:12.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.025+0000 7f1070a3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:12.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.025+0000 7f1070a3a640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:12.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.025+0000 7f1070a3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:58:12.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.025+0000 7f1070a3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:58:12.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.025+0000 7f1070a3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:58:12.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.025+0000 7f1070a3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:58:12.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.025+0000 7f1070a3a640 10 monclient(hunting): handle_auth_done global_id 824919 payload 824 2024-09-16T03:58:12.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.025+0000 7f1070a3a640 1 -- 172.21.15.42:0/534455802 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f106c121cc0 con 0x7f106c06e910 2024-09-16T03:58:12.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.025+0000 7f10697fa640 10 client.854435.objecter ms_handle_connect 0x7f106c06e910 2024-09-16T03:58:12.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.025+0000 7f10697fa640 10 client.854435.objecter resend_mon_ops 2024-09-16T03:58:12.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.025+0000 7f1070a3a640 1 --2- 172.21.15.42:0/534455802 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f106c06e910 0x7f106c06ed20 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f106c074530 tx=0x7f105c007000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:58:12.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.025+0000 7f10697fa640 1 -- 172.21.15.42:0/534455802 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f105c001a00 con 0x7f106c06e910 2024-09-16T03:58:12.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.025+0000 7f10697fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:58:12.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.025+0000 7f10697fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:58:12.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.025+0000 7f10697fa640 10 monclient: dump: 2024-09-16T03:58:12.052 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:58:12.052 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:58:12.052 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:12.052 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:12.052 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:58:12.052 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:12.052 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:58:12.052 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:58:12.052 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:58:12.053 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:12.053 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.025+0000 7f10697fa640 1 -- 172.21.15.42:0/534455802 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4882 (secure 0 0 0) 0x7f105c0c50d0 con 0x7f106c06e910 2024-09-16T03:58:12.053 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.025+0000 7f10697fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:58:12.053 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.025+0000 7f10697fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:58:12.053 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.025+0000 7f10697fa640 1 -- 172.21.15.42:0/534455802 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f106c06e910 msgr2=0x7f106c06ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:12.053 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.025+0000 7f10697fa640 1 --2- 172.21.15.42:0/534455802 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f106c06e910 0x7f106c06ed20 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f106c074530 tx=0x7f105c007000 comp rx=0 tx=0).stop 2024-09-16T03:58:12.053 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.025+0000 7f10697fa640 1 -- 172.21.15.42:0/534455802 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f106c06e910 msgr2=0x7f106c06ed20 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:58:12.053 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.025+0000 7f10697fa640 1 --2- 172.21.15.42:0/534455802 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f106c06e910 0x7f106c06ed20 secure :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f106c074530 tx=0x7f105c007000 comp rx=0 tx=0).stop 2024-09-16T03:58:12.053 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:58:12.053 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:58:12.054 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:58:12.054 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "leader", 2024-09-16T03:58:12.054 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 148, 2024-09-16T03:58:12.054 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 381880, 2024-09-16T03:58:12.054 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:58:12.054 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:58:12.054 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T03:58:12.054 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T03:58:12.054 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:58:12.054 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 0, 2024-09-16T03:58:12.054 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:58:12.054 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:58:12.054 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:58:12.054 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:58:12.055 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:58:12.055 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:58:12.055 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:58:12.055 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:58:12.055 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:58:12.055 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:58:12.055 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:58:12.055 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:58:12.055 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:58:12.055 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:58:12.055 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:58:12.055 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:58:12.055 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:58:12.055 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:58:12.055 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:58:12.056 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:58:12.056 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:58:12.056 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:58:12.056 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:58:12.056 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:58:12.056 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:58:12.056 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:58:12.056 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:58:12.056 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:58:12.056 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:58:12.056 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:58:12.056 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:58:12.056 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:58:12.056 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:58:12.057 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:58:12.057 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:58:12.057 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:58:12.057 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:58:12.057 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:58:12.057 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:58:12.057 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:58:12.057 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:58:12.057 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:58:12.057 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:58:12.057 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:58:12.057 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:58:12.057 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:58:12.057 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:58:12.058 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:58:12.058 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:58:12.058 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:58:12.058 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:58:12.058 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:58:12.058 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:58:12.058 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:58:12.058 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:58:12.058 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:58:12.058 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:58:12.058 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:58:12.058 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:58:12.058 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:58:12.058 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:58:12.059 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:58:12.059 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:58:12.059 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:58:12.059 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:58:12.059 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:58:12.059 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:58:12.059 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:58:12.059 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:58:12.059 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:58:12.059 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:58:12.059 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:58:12.059 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:58:12.059 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:58:12.059 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:58:12.059 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:58:12.060 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:58:12.060 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:58:12.060 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:58:12.060 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:58:12.060 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:58:12.060 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:58:12.060 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:58:12.060 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:58:12.060 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:58:12.060 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:58:12.060 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:58:12.060 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:58:12.060 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:58:12.060 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:58:12.061 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:58:12.061 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:58:12.061 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:58:12.061 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:58:12.061 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:58:12.061 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:58:12.061 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:58:12.061 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:58:12.061 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:58:12.061 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:58:12.061 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:58:12.061 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:58:12.061 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:58:12.061 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:58:12.062 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:58:12.062 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:58:12.062 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:58:12.062 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:58:12.062 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:58:12.062 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:58:12.062 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:58:12.062 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:58:12.062 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:58:12.062 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:58:12.062 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:58:12.062 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:58:12.062 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:58:12.062 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:58:12.062 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:58:12.063 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:58:12.063 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:58:12.063 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:58:12.063 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:58:12.063 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:58:12.063 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:58:12.063 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:58:12.063 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:58:12.063 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:58:12.063 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:58:12.063 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:58:12.063 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:58:12.063 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:58:12.063 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:58:12.063 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:58:12.064 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:58:12.064 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:58:12.064 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:58:12.064 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:58:12.064 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:58:12.064 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:58:12.064 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:58:12.064 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T03:58:12.064 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:58:12.064 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:58:12.064 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:58:12.064 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 5 2024-09-16T03:58:12.064 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:58:12.065 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:58:12.065 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:58:12.065 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:58:12.065 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:58:12.065 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:58:12.065 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 7 2024-09-16T03:58:12.065 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:58:12.065 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:58:12.065 INFO:teuthology.orchestra.run.smithi042.stdout: "mgr": [ 2024-09-16T03:58:12.065 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:58:12.065 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:58:12.065 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:58:12.065 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:58:12.065 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:58:12.066 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:58:12.066 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:58:12.066 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:58:12.066 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:58:12.067 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.025+0000 7f1072c9e640 10 librados: watch_flush enter 2024-09-16T03:58:12.068 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.025+0000 7f1072c9e640 10 librados: watch_flush exit 2024-09-16T03:58:12.068 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.025+0000 7f1072c9e640 20 client.854435.objecter shutdown clearing up homeless session... 2024-09-16T03:58:12.068 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.025+0000 7f1072c9e640 10 client.854435.objecter successfully canceled tick 2024-09-16T03:58:12.068 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.025+0000 7f1072c9e640 1 -- 172.21.15.42:0/534455802 >> v2:172.21.15.154:6812/254845342 conn(0x7f104c037810 msgr2=0x7f104c039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:12.068 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.025+0000 7f1072c9e640 1 --2- 172.21.15.42:0/534455802 >> v2:172.21.15.154:6812/254845342 conn(0x7f104c037810 0x7f104c039cd0 secure :-1 s=READY pgs=3689 cs=0 l=1 rev1=1 crypto rx=0x7f106c0a1af0 tx=0x7f1064008040 comp rx=0 tx=0).stop 2024-09-16T03:58:12.068 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.025+0000 7f1072c9e640 10 monclient: shutdown 2024-09-16T03:58:12.068 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.025+0000 7f1072c9e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:58:12.068 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.025+0000 7f1072c9e640 1 -- 172.21.15.42:0/534455802 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f106c1cfd40 msgr2=0x7f106c120920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:12.068 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.025+0000 7f1072c9e640 1 --2- 172.21.15.42:0/534455802 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f106c1cfd40 0x7f106c120920 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f105400e440 tx=0x7f105400e900 comp rx=0 tx=0).stop 2024-09-16T03:58:12.068 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.025+0000 7f107123b640 1 -- 172.21.15.42:0/534455802 reap_dead start 2024-09-16T03:58:12.068 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.025+0000 7f107123b640 1 -- 172.21.15.42:0/534455802 reap_dead start 2024-09-16T03:58:12.068 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.025+0000 7f1072c9e640 1 -- 172.21.15.42:0/534455802 shutdown_connections 2024-09-16T03:58:12.068 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.025+0000 7f1072c9e640 1 -- 172.21.15.42:0/534455802 >> 172.21.15.42:0/534455802 conn(0x7f106c0a2420 msgr2=0x7f106c1399a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:58:12.069 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.025+0000 7f1072c9e640 1 -- 172.21.15.42:0/534455802 shutdown_connections 2024-09-16T03:58:12.069 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.025+0000 7f1072c9e640 1 -- 172.21.15.42:0/534455802 wait complete. 2024-09-16T03:58:12.069 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.025+0000 7f1072c9e640 1 librados: shutdown 2024-09-16T03:58:12.069 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:58:12.069 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T03:58:12.069 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T03:58:12.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.181+0000 7fbc4bdc4640 10 monclient: get_monmap_and_config 2024-09-16T03:58:12.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.181+0000 7fbc4bdc4640 10 monclient: build_initial_monmap 2024-09-16T03:58:12.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.181+0000 7fbc4bdc4640 10 monclient: monmap: 2024-09-16T03:58:12.186 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:58:12.186 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:58:12.186 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:58:12.184676+0000 2024-09-16T03:58:12.186 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:58:12.184676+0000 2024-09-16T03:58:12.186 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:58:12.186 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:12.187 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:58:12.187 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:58:12.187 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:58:12.187 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:58:12.187 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:58:12.187 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:12.189 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.181+0000 7fbc4bdc4640 1 Processor -- start 2024-09-16T03:58:12.189 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.181+0000 7fbc4bdc4640 1 -- start start 2024-09-16T03:58:12.190 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.181+0000 7fbc4bdc4640 10 monclient: init 2024-09-16T03:58:12.190 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.181+0000 7fbc4bdc4640 10 monclient: _reopen_session rank -1 2024-09-16T03:58:12.190 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.181+0000 7fbc4bdc4640 10 monclient: _add_conns ranks=[4,0,2,3,1] 2024-09-16T03:58:12.190 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.181+0000 7fbc4bdc4640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fbc440a1330 0x7fbc440a1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:12.190 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.181+0000 7fbc4bdc4640 10 monclient(hunting): picked mon.noname-c con 0x7fbc440a1330 addr v2:172.21.15.154:3300/0 2024-09-16T03:58:12.190 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.181+0000 7fbc4bdc4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbc4406dac0 addr v1:172.21.15.42:6789/0 2024-09-16T03:58:12.190 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.181+0000 7fbc4bdc4640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbc440a1c50 0x7fbc44135810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:12.190 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.181+0000 7fbc4bdc4640 10 monclient(hunting): picked mon.noname-b con 0x7fbc440a1c50 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:58:12.190 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.181+0000 7fbc4bdc4640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbc4409c090 con 0x7fbc4406dac0 2024-09-16T03:58:12.190 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.181+0000 7fbc4bdc4640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:12.190 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.181+0000 7fbc4bdc4640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fbc4409f610 con 0x7fbc440a1330 2024-09-16T03:58:12.190 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.181+0000 7fbc4bdc4640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:12.190 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.181+0000 7fbc4bdc4640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fbc4409f8f0 con 0x7fbc440a1c50 2024-09-16T03:58:12.191 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.181+0000 7fbc4bdc4640 10 monclient(hunting): _renew_subs 2024-09-16T03:58:12.191 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.181+0000 7fbc4bdc4640 10 monclient(hunting): authenticate will time out at 1939.154221s 2024-09-16T03:58:12.191 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.181+0000 7fbc4a361640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbc440a1c50 0x7fbc44135810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:12.191 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.181+0000 7fbc4a361640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbc440a1c50 0x7fbc44135810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:49018/0 (socket says 172.21.15.42:49018) 2024-09-16T03:58:12.191 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.181+0000 7fbc4a361640 1 -- 172.21.15.42:0/1777103197 learned_addr learned my addr 172.21.15.42:0/1777103197 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:58:12.191 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.181+0000 7fbc4a361640 10 monclient(hunting): get_auth_request con 0x7fbc440a1c50 auth_method 0 2024-09-16T03:58:12.191 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.181+0000 7fbc4a361640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:12.191 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.181+0000 7fbc4a361640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:12.191 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.181+0000 7fbc4a361640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:58:12.191 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.181+0000 7fbc4a361640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:58:12.191 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.181+0000 7fbc4a361640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:58:12.191 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.181+0000 7fbc4a361640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:58:12.191 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.181+0000 7fbc49b60640 1 --2- 172.21.15.42:0/1777103197 >> v2:172.21.15.154:3300/0 conn(0x7fbc440a1330 0x7fbc440a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:12.191 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.181+0000 7fbc49b60640 10 monclient(hunting): get_auth_request con 0x7fbc440a1330 auth_method 0 2024-09-16T03:58:12.192 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.181+0000 7fbc49b60640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:12.192 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.181+0000 7fbc49b60640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:12.192 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.181+0000 7fbc49b60640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:58:12.192 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.181+0000 7fbc4a361640 10 monclient(hunting): handle_auth_done global_id 854438 payload 274 2024-09-16T03:58:12.192 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.181+0000 7fbc4a361640 1 -- 172.21.15.42:0/1777103197 >> v2:172.21.15.154:3300/0 conn(0x7fbc440a1330 msgr2=0x7fbc440a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:58:12.192 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.181+0000 7fbc4a361640 1 --2- 172.21.15.42:0/1777103197 >> v2:172.21.15.154:3300/0 conn(0x7fbc440a1330 0x7fbc440a1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:12.192 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.181+0000 7fbc4a361640 1 -- 172.21.15.42:0/1777103197 >> v1:172.21.15.42:6789/0 conn(0x7fbc4406dac0 legacy=0x7fbc44135100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:12.192 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.181+0000 7fbc4a361640 10 monclient: _finish_hunting 0 2024-09-16T03:58:12.192 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.181+0000 7fbc4a361640 1 monclient: found mon.noname-b 2024-09-16T03:58:12.192 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.181+0000 7fbc4a361640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:58:12.192 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.181+0000 7fbc4a361640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T03:58:12.192 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.181+0000 7fbc4a361640 1 -- 172.21.15.42:0/1777103197 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc4409fbd0 con 0x7fbc440a1c50 2024-09-16T03:58:12.192 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.181+0000 7fbc48b5e640 1 -- 172.21.15.42:0/1777103197 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2052991705 0 0) 0x7fbc4409c090 con 0x7fbc4406dac0 2024-09-16T03:58:12.192 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.181+0000 7fbc48b5e640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T03:58:12.193 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.181+0000 7fbc49b60640 1 --2- 172.21.15.42:0/1777103197 >> v2:172.21.15.154:3300/0 conn(0x7fbc440a1330 0x7fbc440a1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T03:58:12.199 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.185+0000 7fbc4a361640 1 --2- 172.21.15.42:0/1777103197 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbc440a1c50 0x7fbc44135810 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fbc34009870 tx=0x7fbc3401c4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=af9f7541dd1b6c6a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:58:12.199 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.185+0000 7fbc48b5e640 1 -- 172.21.15.42:0/1777103197 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc340b1070 con 0x7fbc440a1c50 2024-09-16T03:58:12.199 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.185+0000 7fbc48b5e640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:58:12.199 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.185+0000 7fbc48b5e640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:58:12.199 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.185+0000 7fbc48b5e640 10 monclient: dump: 2024-09-16T03:58:12.199 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:58:12.199 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:58:12.199 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:12.199 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:12.199 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:58:12.200 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:12.200 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:58:12.200 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:58:12.200 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:58:12.200 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:12.200 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.185+0000 7fbc48b5e640 10 monclient: _finish_auth 0 2024-09-16T03:58:12.200 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.185+0000 7fbc48b5e640 10 monclient: _check_auth_tickets 2024-09-16T03:58:12.200 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.185+0000 7fbc48b5e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:58:12.200 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.185+0000 7fbc48b5e640 1 -- 172.21.15.42:0/1777103197 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc340ac620 con 0x7fbc440a1c50 2024-09-16T03:58:12.200 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.185+0000 7fbc48b5e640 10 monclient: handle_config config(0 keys) 2024-09-16T03:58:12.200 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.185+0000 7fbc48b5e640 1 -- 172.21.15.42:0/1777103197 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc340ac9f0 con 0x7fbc440a1c50 2024-09-16T03:58:12.200 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.185+0000 7fbc48b5e640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:58:12.200 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.185+0000 7fbc48b5e640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:58:12.201 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.185+0000 7fbc48b5e640 10 monclient: dump: 2024-09-16T03:58:12.201 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:58:12.201 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:58:12.201 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:12.201 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:12.201 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:58:12.201 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:12.201 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:58:12.201 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:58:12.201 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:58:12.201 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:12.201 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.185+0000 7fbc4bdc4640 5 monclient: authenticate success, global_id 854438 2024-09-16T03:58:12.201 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.185+0000 7fbc4bdc4640 10 monclient: get_monmap_and_config success 2024-09-16T03:58:12.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.185+0000 7fbc4bdc4640 4 set_mon_vals no callback set 2024-09-16T03:58:12.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.185+0000 7fbc4bdc4640 10 monclient: shutdown 2024-09-16T03:58:12.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.185+0000 7fbc4bdc4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:58:12.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.185+0000 7fbc4bdc4640 1 -- 172.21.15.42:0/1777103197 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbc440a1c50 msgr2=0x7fbc44135810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:12.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.185+0000 7fbc4bdc4640 1 --2- 172.21.15.42:0/1777103197 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbc440a1c50 0x7fbc44135810 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fbc34009870 tx=0x7fbc3401c4b0 comp rx=0 tx=0).stop 2024-09-16T03:58:12.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.185+0000 7fbc4bdc4640 1 -- 172.21.15.42:0/1777103197 shutdown_connections 2024-09-16T03:58:12.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.185+0000 7fbc4bdc4640 1 --2- 172.21.15.42:0/1777103197 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbc440a1c50 0x7fbc44135810 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:12.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.185+0000 7fbc4bdc4640 1 --2- 172.21.15.42:0/1777103197 >> v2:172.21.15.154:3300/0 conn(0x7fbc440a1330 0x7fbc440a1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:12.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.185+0000 7fbc4bdc4640 1 -- 172.21.15.42:0/1777103197 >> 172.21.15.42:0/1777103197 conn(0x7fbc440a2420 msgr2=0x7fbc440a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:58:12.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.185+0000 7fbc4bdc4640 1 -- 172.21.15.42:0/1777103197 shutdown_connections 2024-09-16T03:58:12.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.185+0000 7fbc4bdc4640 1 -- 172.21.15.42:0/1777103197 wait complete. 2024-09-16T03:58:12.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.189+0000 7fbc4bdc4640 10 monclient: build_initial_monmap 2024-09-16T03:58:12.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.189+0000 7fbc4bdc4640 10 monclient: monmap: 2024-09-16T03:58:12.202 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:58:12.203 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:58:12.203 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:58:12.203 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:58:12.203 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:58:12.203 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:12.203 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:58:12.203 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:58:12.203 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:58:12.203 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:12.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.189+0000 7fbc4bdc4640 1 librados: starting msgr at 2024-09-16T03:58:12.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.189+0000 7fbc4bdc4640 1 librados: starting objecter 2024-09-16T03:58:12.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.189+0000 7fbc4bdc4640 1 Processor -- start 2024-09-16T03:58:12.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.189+0000 7fbc4bdc4640 1 -- start start 2024-09-16T03:58:12.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.189+0000 7fbc4bdc4640 1 librados: setting wanted keys 2024-09-16T03:58:12.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.189+0000 7fbc4bdc4640 1 librados: calling monclient init 2024-09-16T03:58:12.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.189+0000 7fbc4bdc4640 10 monclient: init 2024-09-16T03:58:12.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.189+0000 7fbc4bdc4640 10 monclient: _reopen_session rank -1 2024-09-16T03:58:12.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.189+0000 7fbc4bdc4640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:58:12.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.189+0000 7fbc4bdc4640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbc440a1330 0x7fbc4411bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:12.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.189+0000 7fbc4bdc4640 10 monclient(hunting): picked mon.noname-a con 0x7fbc440a1330 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:58:12.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.189+0000 7fbc4bdc4640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbc440a1c50 0x7fbc4411c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:12.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.189+0000 7fbc4bdc4640 10 monclient(hunting): picked mon.noname-c con 0x7fbc440a1c50 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:58:12.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.189+0000 7fbc4bdc4640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbc4411c9f0 0x7fbc44134580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:12.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.189+0000 7fbc4bdc4640 10 monclient(hunting): picked mon.noname-b con 0x7fbc4411c9f0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:58:12.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.189+0000 7fbc4bdc4640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:12.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.189+0000 7fbc4bdc4640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fbc44135d50 con 0x7fbc440a1330 2024-09-16T03:58:12.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.189+0000 7fbc4bdc4640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:12.205 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.189+0000 7fbc4bdc4640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fbc4409ef80 con 0x7fbc4411c9f0 2024-09-16T03:58:12.205 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.189+0000 7fbc4bdc4640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:12.205 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.189+0000 7fbc4bdc4640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fbc44135ed0 con 0x7fbc440a1c50 2024-09-16T03:58:12.205 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.189+0000 7fbc4bdc4640 10 monclient(hunting): _renew_subs 2024-09-16T03:58:12.205 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.189+0000 7fbc4bdc4640 10 monclient(hunting): authenticate will time out at 1939.161454s 2024-09-16T03:58:12.205 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.189+0000 7fbc49b60640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbc440a1330 0x7fbc4411bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:12.205 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.189+0000 7fbc4935f640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbc440a1c50 0x7fbc4411c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:12.205 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.189+0000 7fbc49b60640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbc440a1330 0x7fbc4411bf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:36012/0 (socket says 172.21.15.42:36012) 2024-09-16T03:58:12.205 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.189+0000 7fbc49b60640 1 -- 172.21.15.42:0/449632986 learned_addr learned my addr 172.21.15.42:0/449632986 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:58:12.205 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.189+0000 7fbc49b60640 10 monclient(hunting): get_auth_request con 0x7fbc440a1330 auth_method 0 2024-09-16T03:58:12.205 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.189+0000 7fbc49b60640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:12.205 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.189+0000 7fbc49b60640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:12.205 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.189+0000 7fbc49b60640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:58:12.205 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.189+0000 7fbc4935f640 10 monclient(hunting): get_auth_request con 0x7fbc440a1c50 auth_method 0 2024-09-16T03:58:12.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.189+0000 7fbc4935f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:12.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.189+0000 7fbc4935f640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:12.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.189+0000 7fbc4935f640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:58:12.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.189+0000 7fbc49b60640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:58:12.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.189+0000 7fbc49b60640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:58:12.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.189+0000 7fbc49b60640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:58:12.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.189+0000 7fbc4935f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:58:12.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.189+0000 7fbc4935f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:58:12.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.189+0000 7fbc4935f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:58:12.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.189+0000 7fbc49b60640 10 monclient(hunting): handle_auth_done global_id 824925 payload 824 2024-09-16T03:58:12.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.189+0000 7fbc4a361640 1 --2- 172.21.15.42:0/449632986 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbc4411c9f0 0x7fbc44134580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:12.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.189+0000 7fbc49b60640 1 -- 172.21.15.42:0/449632986 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbc440a1c50 msgr2=0x7fbc4411c4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:12.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.189+0000 7fbc49b60640 1 --2- 172.21.15.42:0/449632986 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbc440a1c50 0x7fbc4411c4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:12.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.189+0000 7fbc49b60640 1 -- 172.21.15.42:0/449632986 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbc4411c9f0 msgr2=0x7fbc44134580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:12.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.189+0000 7fbc49b60640 1 --2- 172.21.15.42:0/449632986 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbc4411c9f0 0x7fbc44134580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:12.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.189+0000 7fbc49b60640 10 monclient: _finish_hunting 0 2024-09-16T03:58:12.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.189+0000 7fbc49b60640 1 monclient: found mon.noname-a 2024-09-16T03:58:12.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.189+0000 7fbc49b60640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:58:12.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.189+0000 7fbc49b60640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:58:12.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.189+0000 7fbc49b60640 1 -- 172.21.15.42:0/449632986 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc4411f610 con 0x7fbc440a1330 2024-09-16T03:58:12.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.189+0000 7fbc4a361640 10 monclient: get_auth_request con 0x7fbc4411c9f0 auth_method 0 2024-09-16T03:58:12.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.189+0000 7fbc4a361640 1 --2- 172.21.15.42:0/449632986 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbc4411c9f0 0x7fbc44134580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T03:58:12.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.189+0000 7fbc49b60640 1 --2- 172.21.15.42:0/449632986 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbc440a1330 0x7fbc4411bf70 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fbc3800e810 tx=0x7fbc3800ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:58:12.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.189+0000 7fbc427fc640 10 client.?.objecter ms_handle_connect 0x7fbc440a1330 2024-09-16T03:58:12.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.189+0000 7fbc427fc640 10 client.?.objecter resend_mon_ops 2024-09-16T03:58:12.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.189+0000 7fbc427fc640 1 -- 172.21.15.42:0/449632986 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc38017070 con 0x7fbc440a1330 2024-09-16T03:58:12.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.189+0000 7fbc427fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:58:12.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.189+0000 7fbc427fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:58:12.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.189+0000 7fbc427fc640 10 monclient: dump: 2024-09-16T03:58:12.208 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:58:12.208 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:58:12.208 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:12.208 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:12.208 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:58:12.208 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:12.208 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:58:12.208 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:58:12.208 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:58:12.208 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:12.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.189+0000 7fbc427fc640 10 monclient: _finish_auth 0 2024-09-16T03:58:12.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.189+0000 7fbc427fc640 10 monclient: _check_auth_tickets 2024-09-16T03:58:12.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.189+0000 7fbc427fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:58:12.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.189+0000 7fbc4bdc4640 5 monclient: authenticate success, global_id 824925 2024-09-16T03:58:12.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.189+0000 7fbc4bdc4640 10 monclient: _renew_subs 2024-09-16T03:58:12.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.189+0000 7fbc427fc640 1 -- 172.21.15.42:0/449632986 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc3800c8d0 con 0x7fbc440a1330 2024-09-16T03:58:12.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.189+0000 7fbc4bdc4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:58:12.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.189+0000 7fbc4bdc4640 1 -- 172.21.15.42:0/449632986 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc4406e410 con 0x7fbc440a1330 2024-09-16T03:58:12.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.189+0000 7fbc427fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:58:12.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.189+0000 7fbc427fc640 1 -- 172.21.15.42:0/449632986 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc3800f040 con 0x7fbc440a1330 2024-09-16T03:58:12.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.189+0000 7fbc427fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:58:12.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.189+0000 7fbc427fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:58:12.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.189+0000 7fbc427fc640 10 monclient: dump: 2024-09-16T03:58:12.209 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:58:12.209 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:58:12.210 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:12.210 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:12.210 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:58:12.210 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:12.210 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:58:12.210 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:58:12.210 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:58:12.210 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:12.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.189+0000 7fbc43fff640 4 set_mon_vals no callback set 2024-09-16T03:58:12.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.189+0000 7fbc4bdc4640 10 client.824925.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:58:12.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.189+0000 7fbc4bdc4640 10 monclient: _renew_subs 2024-09-16T03:58:12.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.189+0000 7fbc4bdc4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:58:12.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.189+0000 7fbc4bdc4640 1 -- 172.21.15.42:0/449632986 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc4411f050 con 0x7fbc440a1330 2024-09-16T03:58:12.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.189+0000 7fbc4bdc4640 1 librados: init done 2024-09-16T03:58:12.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.189+0000 7fbc4bdc4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:58:12.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.189+0000 7fbc4bdc4640 1 --2- 172.21.15.42:0/449632986 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbc4406dac0 0x7fbc4402c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:12.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.189+0000 7fbc4bdc4640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:12.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.189+0000 7fbc4bdc4640 1 -- 172.21.15.42:0/449632986 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fbc4411e0a0 con 0x7fbc4406dac0 2024-09-16T03:58:12.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.189+0000 7fbc4935f640 1 --2- 172.21.15.42:0/449632986 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbc4406dac0 0x7fbc4402c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:12.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.189+0000 7fbc4935f640 10 monclient: get_auth_request con 0x7fbc4406dac0 auth_method 0 2024-09-16T03:58:12.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.189+0000 7fbc4935f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:12.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.189+0000 7fbc4935f640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:12.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.189+0000 7fbc4935f640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:58:12.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.189+0000 7fbc427fc640 1 -- 172.21.15.42:0/449632986 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fbc3801a070 con 0x7fbc440a1330 2024-09-16T03:58:12.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.189+0000 7fbc4935f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:58:12.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.189+0000 7fbc4935f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:58:12.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.189+0000 7fbc4935f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:58:12.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.189+0000 7fbc4935f640 10 monclient(hunting): handle_auth_done global_id 854444 payload 824 2024-09-16T03:58:12.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.189+0000 7fbc4935f640 1 -- 172.21.15.42:0/449632986 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbc440a1710 con 0x7fbc4406dac0 2024-09-16T03:58:12.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.189+0000 7fbc427fc640 1 --2- 172.21.15.42:0/449632986 >> v2:172.21.15.154:6812/254845342 conn(0x7fbc30037810 0x7fbc30039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:12.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.189+0000 7fbc427fc640 1 -- 172.21.15.42:0/449632986 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(482..482 src has 366..482) ==== 5795+0+0 (secure 0 0 0) 0x7fbc38048bb0 con 0x7fbc440a1330 2024-09-16T03:58:12.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.189+0000 7fbc427fc640 10 client.824925.objecter ms_dispatch 0x7fbc44000ba0 osd_map(482..482 src has 366..482) 2024-09-16T03:58:12.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.189+0000 7fbc427fc640 3 client.824925.objecter handle_osd_map got epochs [482,482] > 0 2024-09-16T03:58:12.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.189+0000 7fbc427fc640 3 client.824925.objecter handle_osd_map decoding full epoch 482 2024-09-16T03:58:12.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.189+0000 7fbc4935f640 1 --2- 172.21.15.42:0/449632986 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbc4406dac0 0x7fbc4402c650 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fbc3c0012f0 tx=0x7fbc3c00ff70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:58:12.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.189+0000 7fbc427fc640 20 client.824925.objecter dump_active .. 0 homeless 2024-09-16T03:58:12.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.189+0000 7fbc427fc640 10 client.824925.objecter ms_handle_connect 0x7fbc4406dac0 2024-09-16T03:58:12.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.189+0000 7fbc427fc640 10 client.824925.objecter resend_mon_ops 2024-09-16T03:58:12.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.193+0000 7fbc427fc640 1 -- 172.21.15.42:0/449632986 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc3c003b90 con 0x7fbc4406dac0 2024-09-16T03:58:12.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.193+0000 7fbc427fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:58:12.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.193+0000 7fbc427fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:58:12.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.193+0000 7fbc427fc640 10 monclient: dump: 2024-09-16T03:58:12.213 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:58:12.213 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:58:12.213 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:12.213 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:12.213 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:58:12.213 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:12.213 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:58:12.213 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:58:12.213 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:58:12.213 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:12.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.193+0000 7fbc4a361640 1 --2- 172.21.15.42:0/449632986 >> v2:172.21.15.154:6812/254845342 conn(0x7fbc30037810 0x7fbc30039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:12.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.193+0000 7fbc4a361640 10 monclient: get_auth_request con 0x7fbc30037810 auth_method 0 2024-09-16T03:58:12.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.193+0000 7fbc427fc640 1 -- 172.21.15.42:0/449632986 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fbc3c003b90 con 0x7fbc4406dac0 2024-09-16T03:58:12.214 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.193+0000 7fbc427fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:58:12.214 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.193+0000 7fbc427fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:58:12.214 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.193+0000 7fbc427fc640 1 -- 172.21.15.42:0/449632986 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbc4406dac0 msgr2=0x7fbc4402c650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:12.214 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.193+0000 7fbc427fc640 1 --2- 172.21.15.42:0/449632986 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbc4406dac0 0x7fbc4402c650 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fbc3c0012f0 tx=0x7fbc3c00ff70 comp rx=0 tx=0).stop 2024-09-16T03:58:12.214 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.193+0000 7fbc427fc640 1 -- 172.21.15.42:0/449632986 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbc4406dac0 msgr2=0x7fbc4402c650 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:58:12.214 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.193+0000 7fbc427fc640 1 --2- 172.21.15.42:0/449632986 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbc4406dac0 0x7fbc4402c650 secure :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fbc3c0012f0 tx=0x7fbc3c00ff70 comp rx=0 tx=0).stop 2024-09-16T03:58:12.214 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.193+0000 7fbc4a361640 1 --2- 172.21.15.42:0/449632986 >> v2:172.21.15.154:6812/254845342 conn(0x7fbc30037810 0x7fbc30039cd0 secure :-1 s=READY pgs=3690 cs=0 l=1 rev1=1 crypto rx=0x7fbc340ae000 tx=0x7fbc340a7000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:58:12.214 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.193+0000 7fbc427fc640 10 client.824925.objecter ms_handle_connect 0x7fbc30037810 2024-09-16T03:58:12.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.197+0000 7fbc4bdc4640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:58:12.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.197+0000 7fbc4bdc4640 1 --2- 172.21.15.42:0/449632986 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbc4402cb90 0x7fbc4402cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:12.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.197+0000 7fbc4bdc4640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:12.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.197+0000 7fbc4bdc4640 1 -- 172.21.15.42:0/449632986 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fbc44139a40 con 0x7fbc4402cb90 2024-09-16T03:58:12.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.197+0000 7fbc4935f640 1 --2- 172.21.15.42:0/449632986 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbc4402cb90 0x7fbc4402cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:12.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.197+0000 7fbc4935f640 10 monclient: get_auth_request con 0x7fbc4402cb90 auth_method 0 2024-09-16T03:58:12.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.197+0000 7fbc4935f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:12.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.197+0000 7fbc4935f640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:12.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.197+0000 7fbc4935f640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:58:12.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.197+0000 7fbc4935f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:58:12.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.197+0000 7fbc4935f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:58:12.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.197+0000 7fbc4935f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:58:12.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.197+0000 7fbc4935f640 10 monclient(hunting): handle_auth_done global_id 854447 payload 824 2024-09-16T03:58:12.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.197+0000 7fbc4935f640 1 -- 172.21.15.42:0/449632986 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fbc44069040 con 0x7fbc4402cb90 2024-09-16T03:58:12.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.197+0000 7fbc427fc640 10 client.824925.objecter ms_handle_connect 0x7fbc4402cb90 2024-09-16T03:58:12.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.197+0000 7fbc427fc640 10 client.824925.objecter resend_mon_ops 2024-09-16T03:58:12.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.197+0000 7fbc4935f640 1 --2- 172.21.15.42:0/449632986 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbc4402cb90 0x7fbc4402cfa0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fbc3c0012f0 tx=0x7fbc3c001260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:58:12.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.197+0000 7fbc427fc640 1 -- 172.21.15.42:0/449632986 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc3c003e10 con 0x7fbc4402cb90 2024-09-16T03:58:12.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.197+0000 7fbc427fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:58:12.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.197+0000 7fbc427fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:58:12.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.197+0000 7fbc427fc640 10 monclient: dump: 2024-09-16T03:58:12.218 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:58:12.218 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:58:12.218 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:12.218 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:12.218 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:58:12.218 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:12.218 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:58:12.218 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:58:12.218 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:58:12.218 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:12.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.197+0000 7fbc427fc640 1 -- 172.21.15.42:0/449632986 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4713 (secure 0 0 0) 0x7fbc3c003e10 con 0x7fbc4402cb90 2024-09-16T03:58:12.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.197+0000 7fbc427fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:58:12.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.197+0000 7fbc427fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:58:12.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.197+0000 7fbc427fc640 1 -- 172.21.15.42:0/449632986 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbc4402cb90 msgr2=0x7fbc4402cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:12.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.197+0000 7fbc427fc640 1 --2- 172.21.15.42:0/449632986 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbc4402cb90 0x7fbc4402cfa0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fbc3c0012f0 tx=0x7fbc3c001260 comp rx=0 tx=0).stop 2024-09-16T03:58:12.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.197+0000 7fbc427fc640 1 -- 172.21.15.42:0/449632986 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbc4402cb90 msgr2=0x7fbc4402cfa0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:58:12.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.197+0000 7fbc427fc640 1 --2- 172.21.15.42:0/449632986 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbc4402cb90 0x7fbc4402cfa0 secure :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fbc3c0012f0 tx=0x7fbc3c001260 comp rx=0 tx=0).stop 2024-09-16T03:58:12.219 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:58:12.219 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:58:12.219 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:58:12.219 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T03:58:12.220 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 148, 2024-09-16T03:58:12.220 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 225633, 2024-09-16T03:58:12.220 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:58:12.220 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:58:12.220 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T03:58:12.220 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T03:58:12.220 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:58:12.220 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 0, 2024-09-16T03:58:12.220 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:58:12.220 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:58:12.220 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:58:12.220 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:58:12.221 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:58:12.221 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:58:12.221 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:58:12.221 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:58:12.221 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:58:12.221 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:58:12.221 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:58:12.221 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:58:12.221 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:58:12.221 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:58:12.221 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:58:12.222 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:58:12.222 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:58:12.222 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:58:12.222 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:58:12.222 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:58:12.222 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:58:12.222 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:58:12.222 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:58:12.222 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:58:12.222 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:58:12.222 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:58:12.222 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:58:12.222 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:58:12.223 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:58:12.223 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:58:12.223 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:58:12.223 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:58:12.223 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:58:12.223 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:58:12.223 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:58:12.223 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:58:12.223 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:58:12.223 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:58:12.223 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:58:12.223 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:58:12.223 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:58:12.223 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:58:12.224 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:58:12.224 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:58:12.224 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:58:12.224 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:58:12.224 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:58:12.224 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:58:12.224 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:58:12.224 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:58:12.224 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:58:12.224 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:58:12.224 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:58:12.224 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:58:12.224 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:58:12.225 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:58:12.225 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:58:12.225 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:58:12.225 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:58:12.225 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:58:12.225 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:58:12.225 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:58:12.225 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:58:12.225 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:58:12.225 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:58:12.225 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:58:12.225 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:58:12.225 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:58:12.226 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:58:12.226 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:58:12.226 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:58:12.226 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:58:12.226 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:58:12.226 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:58:12.226 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:58:12.226 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:58:12.226 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:58:12.226 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:58:12.226 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:58:12.226 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:58:12.226 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:58:12.226 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:58:12.227 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:58:12.227 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:58:12.227 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:58:12.227 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:58:12.227 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:58:12.227 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:58:12.227 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:58:12.227 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:58:12.227 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:58:12.227 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:58:12.227 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:58:12.227 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:58:12.227 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:58:12.227 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:58:12.228 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:58:12.228 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:58:12.228 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:58:12.228 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:58:12.228 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:58:12.228 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:58:12.228 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:58:12.228 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:58:12.228 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:58:12.228 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:58:12.230 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:58:12.230 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:58:12.230 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:58:12.230 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:58:12.230 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:58:12.230 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:58:12.231 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:58:12.231 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:58:12.231 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:58:12.231 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:58:12.231 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:58:12.231 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:58:12.231 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:58:12.231 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:58:12.231 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:58:12.231 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:58:12.231 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:58:12.231 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:58:12.231 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:58:12.231 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:58:12.232 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:58:12.232 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:58:12.232 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:58:12.232 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:58:12.232 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:58:12.232 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:58:12.232 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:58:12.232 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:58:12.232 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:58:12.232 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:58:12.232 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:58:12.232 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:58:12.232 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:58:12.233 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:58:12.233 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:58:12.233 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T03:58:12.233 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:58:12.233 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:58:12.233 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:58:12.233 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:58:12.233 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:58:12.233 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:58:12.233 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:58:12.236 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:58:12.236 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:58:12.236 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:58:12.236 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 2 2024-09-16T03:58:12.236 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:58:12.236 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:58:12.236 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:58:12.236 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:58:12.236 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:58:12.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.197+0000 7fbc4bdc4640 10 librados: watch_flush enter 2024-09-16T03:58:12.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.197+0000 7fbc4bdc4640 10 librados: watch_flush exit 2024-09-16T03:58:12.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.197+0000 7fbc4bdc4640 20 client.824925.objecter shutdown clearing up homeless session... 2024-09-16T03:58:12.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.197+0000 7fbc4bdc4640 10 client.824925.objecter successfully canceled tick 2024-09-16T03:58:12.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.197+0000 7fbc4bdc4640 1 -- 172.21.15.42:0/449632986 >> v2:172.21.15.154:6812/254845342 conn(0x7fbc30037810 msgr2=0x7fbc30039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:12.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.197+0000 7fbc4bdc4640 1 --2- 172.21.15.42:0/449632986 >> v2:172.21.15.154:6812/254845342 conn(0x7fbc30037810 0x7fbc30039cd0 secure :-1 s=READY pgs=3690 cs=0 l=1 rev1=1 crypto rx=0x7fbc340ae000 tx=0x7fbc340a7000 comp rx=0 tx=0).stop 2024-09-16T03:58:12.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.197+0000 7fbc4bdc4640 10 monclient: shutdown 2024-09-16T03:58:12.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.197+0000 7fbc4bdc4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:58:12.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.197+0000 7fbc4bdc4640 1 -- 172.21.15.42:0/449632986 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbc440a1330 msgr2=0x7fbc4411bf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:12.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.197+0000 7fbc4bdc4640 1 --2- 172.21.15.42:0/449632986 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbc440a1330 0x7fbc4411bf70 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fbc3800e810 tx=0x7fbc3800ecd0 comp rx=0 tx=0).stop 2024-09-16T03:58:12.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.197+0000 7fbc4a361640 1 -- 172.21.15.42:0/449632986 reap_dead start 2024-09-16T03:58:12.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.197+0000 7fbc4a361640 1 -- 172.21.15.42:0/449632986 reap_dead start 2024-09-16T03:58:12.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.197+0000 7fbc4bdc4640 1 -- 172.21.15.42:0/449632986 shutdown_connections 2024-09-16T03:58:12.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.197+0000 7fbc4bdc4640 1 -- 172.21.15.42:0/449632986 >> 172.21.15.42:0/449632986 conn(0x7fbc440a2420 msgr2=0x7fbc4406f8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:58:12.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.201+0000 7fbc4bdc4640 1 -- 172.21.15.42:0/449632986 shutdown_connections 2024-09-16T03:58:12.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.201+0000 7fbc4bdc4640 1 -- 172.21.15.42:0/449632986 wait complete. 2024-09-16T03:58:12.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.201+0000 7fbc4bdc4640 1 librados: shutdown 2024-09-16T03:58:12.239 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:58:12.240 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T03:58:12.240 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T03:58:12.366 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.361+0000 7ff49295f640 10 monclient: get_monmap_and_config 2024-09-16T03:58:12.366 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.361+0000 7ff49295f640 10 monclient: build_initial_monmap 2024-09-16T03:58:12.366 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.361+0000 7ff49295f640 10 monclient: monmap: 2024-09-16T03:58:12.366 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:58:12.366 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:58:12.366 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:58:12.364543+0000 2024-09-16T03:58:12.366 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:58:12.364543+0000 2024-09-16T03:58:12.366 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:58:12.366 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:12.366 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:58:12.366 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:58:12.367 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:58:12.367 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:58:12.367 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:58:12.367 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:12.369 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.361+0000 7ff49295f640 1 Processor -- start 2024-09-16T03:58:12.369 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.361+0000 7ff49295f640 1 -- start start 2024-09-16T03:58:12.369 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.361+0000 7ff49295f640 10 monclient: init 2024-09-16T03:58:12.369 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.361+0000 7ff49295f640 10 monclient: _reopen_session rank -1 2024-09-16T03:58:12.369 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.361+0000 7ff49295f640 10 monclient: _add_conns ranks=[1,2,4,3,0] 2024-09-16T03:58:12.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.361+0000 7ff49295f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff48c0a1330 addr v1:172.21.15.154:6789/0 2024-09-16T03:58:12.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.361+0000 7ff49295f640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff48c0a1710 0x7ff48c0a1b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:12.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.361+0000 7ff49295f640 10 monclient(hunting): picked mon.noname-b con 0x7ff48c0a1710 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:58:12.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.361+0000 7ff49295f640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7ff48c1393a0 0x7ff48c13b790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:12.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.361+0000 7ff49295f640 10 monclient(hunting): picked mon.noname-c con 0x7ff48c1393a0 addr v2:172.21.15.154:3300/0 2024-09-16T03:58:12.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.361+0000 7ff49295f640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff48c09e4c0 con 0x7ff48c0a1330 2024-09-16T03:58:12.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.361+0000 7ff49295f640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:12.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.361+0000 7ff49295f640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7ff48c09edf0 con 0x7ff48c1393a0 2024-09-16T03:58:12.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.361+0000 7ff49295f640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:12.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.361+0000 7ff49295f640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7ff48c09cd60 con 0x7ff48c0a1710 2024-09-16T03:58:12.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.361+0000 7ff49295f640 10 monclient(hunting): _renew_subs 2024-09-16T03:58:12.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.361+0000 7ff49295f640 10 monclient(hunting): authenticate will time out at 1939.334014s 2024-09-16T03:58:12.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.361+0000 7ff48b7fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff48c0a1710 0x7ff48c0a1b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:12.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.361+0000 7ff48b7fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff48c0a1710 0x7ff48c0a1b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:49048/0 (socket says 172.21.15.42:49048) 2024-09-16T03:58:12.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.361+0000 7ff48b7fe640 1 -- 172.21.15.42:0/2274589244 learned_addr learned my addr 172.21.15.42:0/2274589244 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:58:12.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.361+0000 7ff48b7fe640 10 monclient(hunting): get_auth_request con 0x7ff48c0a1710 auth_method 0 2024-09-16T03:58:12.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.361+0000 7ff48b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:12.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.361+0000 7ff48b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:12.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.361+0000 7ff48b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:58:12.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.361+0000 7ff48b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:58:12.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.361+0000 7ff48b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:58:12.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.361+0000 7ff48b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:58:12.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.361+0000 7ff490efc640 1 --2- 172.21.15.42:0/2274589244 >> v2:172.21.15.154:3300/0 conn(0x7ff48c1393a0 0x7ff48c13b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:12.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.361+0000 7ff490efc640 10 monclient(hunting): get_auth_request con 0x7ff48c1393a0 auth_method 0 2024-09-16T03:58:12.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.361+0000 7ff490efc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:12.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.361+0000 7ff490efc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:12.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.361+0000 7ff490efc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:58:12.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.361+0000 7ff48b7fe640 10 monclient(hunting): handle_auth_done global_id 854450 payload 274 2024-09-16T03:58:12.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.361+0000 7ff48b7fe640 1 -- 172.21.15.42:0/2274589244 >> v2:172.21.15.154:3300/0 conn(0x7ff48c1393a0 msgr2=0x7ff48c13b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:58:12.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.361+0000 7ff48b7fe640 1 --2- 172.21.15.42:0/2274589244 >> v2:172.21.15.154:3300/0 conn(0x7ff48c1393a0 0x7ff48c13b790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:12.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.361+0000 7ff48b7fe640 1 -- 172.21.15.42:0/2274589244 >> v1:172.21.15.154:6789/0 conn(0x7ff48c0a1330 legacy=0x7ff48c09f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:58:12.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.361+0000 7ff48b7fe640 10 monclient: _finish_hunting 0 2024-09-16T03:58:12.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.361+0000 7ff48b7fe640 1 monclient: found mon.noname-b 2024-09-16T03:58:12.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.361+0000 7ff48b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:58:12.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.361+0000 7ff48b7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T03:58:12.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.361+0000 7ff48b7fe640 1 -- 172.21.15.42:0/2274589244 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff48c0a2050 con 0x7ff48c0a1710 2024-09-16T03:58:12.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.361+0000 7ff490efc640 1 --2- 172.21.15.42:0/2274589244 >> v2:172.21.15.154:3300/0 conn(0x7ff48c1393a0 0x7ff48c13b790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T03:58:12.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.365+0000 7ff48b7fe640 1 --2- 172.21.15.42:0/2274589244 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff48c0a1710 0x7ff48c0a1b10 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7ff47400bb10 tx=0x7ff47401eaf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bae44c90d92f21cf server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:58:12.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.365+0000 7ff48affd640 1 -- 172.21.15.42:0/2274589244 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4740b4070 con 0x7ff48c0a1710 2024-09-16T03:58:12.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.365+0000 7ff48affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:58:12.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.365+0000 7ff48affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:58:12.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.365+0000 7ff48affd640 10 monclient: dump: 2024-09-16T03:58:12.378 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:58:12.378 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:58:12.379 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:12.379 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:12.379 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:58:12.379 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:12.379 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:58:12.379 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:58:12.379 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:58:12.379 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:12.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.365+0000 7ff48affd640 10 monclient: _finish_auth 0 2024-09-16T03:58:12.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.365+0000 7ff48affd640 10 monclient: _check_auth_tickets 2024-09-16T03:58:12.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.365+0000 7ff48affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:58:12.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.365+0000 7ff48affd640 1 -- 172.21.15.42:0/2274589244 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4740a7ad0 con 0x7ff48c0a1710 2024-09-16T03:58:12.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.365+0000 7ff49295f640 5 monclient: authenticate success, global_id 854450 2024-09-16T03:58:12.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.365+0000 7ff49295f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:58:12.380 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.365+0000 7ff48affd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:58:12.380 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.365+0000 7ff49295f640 10 monclient: get_monmap_and_config success 2024-09-16T03:58:12.380 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.365+0000 7ff48affd640 1 -- 172.21.15.42:0/2274589244 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4740af440 con 0x7ff48c0a1710 2024-09-16T03:58:12.380 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.365+0000 7ff48affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:58:12.380 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.365+0000 7ff49295f640 4 set_mon_vals no callback set 2024-09-16T03:58:12.380 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.365+0000 7ff48affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:58:12.380 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.365+0000 7ff48affd640 10 monclient: dump: 2024-09-16T03:58:12.380 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:58:12.380 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:58:12.380 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:12.380 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:12.380 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:58:12.380 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:12.381 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:58:12.381 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:58:12.381 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:58:12.381 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:12.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.365+0000 7ff49295f640 10 monclient: shutdown 2024-09-16T03:58:12.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.365+0000 7ff49295f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:58:12.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.365+0000 7ff49295f640 1 -- 172.21.15.42:0/2274589244 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff48c0a1710 msgr2=0x7ff48c0a1b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:12.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.365+0000 7ff49295f640 1 --2- 172.21.15.42:0/2274589244 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff48c0a1710 0x7ff48c0a1b10 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7ff47400bb10 tx=0x7ff47401eaf0 comp rx=0 tx=0).stop 2024-09-16T03:58:12.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.365+0000 7ff49295f640 1 -- 172.21.15.42:0/2274589244 shutdown_connections 2024-09-16T03:58:12.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.365+0000 7ff49295f640 1 --2- 172.21.15.42:0/2274589244 >> v2:172.21.15.154:3300/0 conn(0x7ff48c1393a0 0x7ff48c13b790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:12.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.365+0000 7ff49295f640 1 --2- 172.21.15.42:0/2274589244 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff48c0a1710 0x7ff48c0a1b10 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:12.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.365+0000 7ff49295f640 1 -- 172.21.15.42:0/2274589244 >> 172.21.15.42:0/2274589244 conn(0x7ff48c0a2420 msgr2=0x7ff48c0a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:58:12.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.365+0000 7ff49295f640 1 -- 172.21.15.42:0/2274589244 shutdown_connections 2024-09-16T03:58:12.382 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.365+0000 7ff49295f640 1 -- 172.21.15.42:0/2274589244 wait complete. 2024-09-16T03:58:12.382 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.365+0000 7ff49295f640 10 monclient: build_initial_monmap 2024-09-16T03:58:12.382 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.365+0000 7ff49295f640 10 monclient: monmap: 2024-09-16T03:58:12.382 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:58:12.382 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:58:12.382 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:58:12.382 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:58:12.382 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:58:12.382 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:12.382 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:58:12.383 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:58:12.383 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:58:12.383 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:12.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.365+0000 7ff49295f640 1 librados: starting msgr at 2024-09-16T03:58:12.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.365+0000 7ff49295f640 1 librados: starting objecter 2024-09-16T03:58:12.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.365+0000 7ff49295f640 1 Processor -- start 2024-09-16T03:58:12.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.365+0000 7ff49295f640 1 -- start start 2024-09-16T03:58:12.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.365+0000 7ff49295f640 1 librados: setting wanted keys 2024-09-16T03:58:12.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.365+0000 7ff49295f640 1 librados: calling monclient init 2024-09-16T03:58:12.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.365+0000 7ff49295f640 10 monclient: init 2024-09-16T03:58:12.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.365+0000 7ff49295f640 10 monclient: _reopen_session rank -1 2024-09-16T03:58:12.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.365+0000 7ff49295f640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T03:58:12.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.365+0000 7ff49295f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff48c0a1330 0x7ff48c1cae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:12.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.365+0000 7ff49295f640 10 monclient(hunting): picked mon.noname-b con 0x7ff48c0a1330 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:58:12.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.365+0000 7ff49295f640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff48c0a1710 0x7ff48c1313d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:12.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.365+0000 7ff49295f640 10 monclient(hunting): picked mon.noname-c con 0x7ff48c0a1710 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:58:12.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.365+0000 7ff49295f640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff48c1393a0 0x7ff48c06e950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:12.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.365+0000 7ff49295f640 10 monclient(hunting): picked mon.noname-a con 0x7ff48c1393a0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:58:12.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.365+0000 7ff49295f640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:12.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.365+0000 7ff49295f640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7ff48c09c5f0 con 0x7ff48c1393a0 2024-09-16T03:58:12.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.365+0000 7ff49295f640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:12.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.365+0000 7ff49295f640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff48c09c470 con 0x7ff48c0a1330 2024-09-16T03:58:12.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.365+0000 7ff49295f640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:12.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.365+0000 7ff49295f640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7ff48c09c770 con 0x7ff48c0a1710 2024-09-16T03:58:12.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.365+0000 7ff49295f640 10 monclient(hunting): _renew_subs 2024-09-16T03:58:12.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.365+0000 7ff49295f640 10 monclient(hunting): authenticate will time out at 1939.339395s 2024-09-16T03:58:12.385 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.365+0000 7ff490efc640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff48c1393a0 0x7ff48c06e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:12.385 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.365+0000 7ff48b7fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff48c0a1710 0x7ff48c1313d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:12.385 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.365+0000 7ff490efc640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff48c1393a0 0x7ff48c06e950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:36026/0 (socket says 172.21.15.42:36026) 2024-09-16T03:58:12.385 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.365+0000 7ff490efc640 1 -- 172.21.15.42:0/38491130 learned_addr learned my addr 172.21.15.42:0/38491130 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:58:12.385 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.365+0000 7ff490efc640 10 monclient(hunting): get_auth_request con 0x7ff48c1393a0 auth_method 0 2024-09-16T03:58:12.385 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.365+0000 7ff490efc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:12.385 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.365+0000 7ff490efc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:12.385 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.365+0000 7ff490efc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:58:12.385 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.365+0000 7ff48b7fe640 10 monclient(hunting): get_auth_request con 0x7ff48c0a1710 auth_method 0 2024-09-16T03:58:12.385 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.365+0000 7ff48b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:12.385 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.365+0000 7ff48b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:12.385 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.365+0000 7ff48b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:58:12.385 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.365+0000 7ff48bfff640 1 --2- 172.21.15.42:0/38491130 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff48c0a1330 0x7ff48c1cae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:12.385 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.365+0000 7ff490efc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:58:12.386 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.365+0000 7ff490efc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:58:12.386 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.365+0000 7ff490efc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:58:12.386 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.365+0000 7ff48b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:58:12.386 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.365+0000 7ff48b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:58:12.386 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.365+0000 7ff48b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:58:12.386 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.365+0000 7ff48bfff640 10 monclient(hunting): get_auth_request con 0x7ff48c0a1330 auth_method 0 2024-09-16T03:58:12.386 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.365+0000 7ff48bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:12.386 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.365+0000 7ff48bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:12.386 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.365+0000 7ff48bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:58:12.386 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.365+0000 7ff490efc640 10 monclient(hunting): handle_auth_done global_id 824928 payload 824 2024-09-16T03:58:12.386 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.365+0000 7ff490efc640 1 -- 172.21.15.42:0/38491130 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff48c0a1710 msgr2=0x7ff48c1313d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:12.386 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.365+0000 7ff490efc640 1 --2- 172.21.15.42:0/38491130 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff48c0a1710 0x7ff48c1313d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:12.386 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.365+0000 7ff490efc640 1 -- 172.21.15.42:0/38491130 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff48c0a1330 msgr2=0x7ff48c1cae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:12.386 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.365+0000 7ff490efc640 1 --2- 172.21.15.42:0/38491130 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff48c0a1330 0x7ff48c1cae60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:12.387 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.365+0000 7ff490efc640 10 monclient: _finish_hunting 0 2024-09-16T03:58:12.387 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.365+0000 7ff490efc640 1 monclient: found mon.noname-a 2024-09-16T03:58:12.387 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.365+0000 7ff490efc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:58:12.387 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.365+0000 7ff490efc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:58:12.387 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.365+0000 7ff490efc640 1 -- 172.21.15.42:0/38491130 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff48c1cbce0 con 0x7ff48c1393a0 2024-09-16T03:58:12.387 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.369+0000 7ff48bfff640 1 --2- 172.21.15.42:0/38491130 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff48c0a1330 0x7ff48c1cae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T03:58:12.387 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.369+0000 7ff48b7fe640 1 --2- 172.21.15.42:0/38491130 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff48c0a1710 0x7ff48c1313d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:58:12.387 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.369+0000 7ff488ff9640 10 client.?.objecter ms_handle_connect 0x7ff48c1393a0 2024-09-16T03:58:12.387 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.369+0000 7ff488ff9640 10 client.?.objecter resend_mon_ops 2024-09-16T03:58:12.387 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.369+0000 7ff490efc640 1 --2- 172.21.15.42:0/38491130 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff48c1393a0 0x7ff48c06e950 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7ff47c00e5e0 tx=0x7ff47c00eaa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:58:12.387 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.369+0000 7ff488ff9640 1 -- 172.21.15.42:0/38491130 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff47c017070 con 0x7ff48c1393a0 2024-09-16T03:58:12.387 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.369+0000 7ff488ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:58:12.387 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.369+0000 7ff488ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:58:12.388 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.369+0000 7ff488ff9640 10 monclient: dump: 2024-09-16T03:58:12.388 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:58:12.388 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:58:12.388 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:12.388 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:12.388 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:58:12.388 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:12.388 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:58:12.388 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:58:12.388 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:58:12.388 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:12.388 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.369+0000 7ff488ff9640 10 monclient: _finish_auth 0 2024-09-16T03:58:12.388 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.369+0000 7ff488ff9640 10 monclient: _check_auth_tickets 2024-09-16T03:58:12.388 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.369+0000 7ff488ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:58:12.389 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.369+0000 7ff488ff9640 1 -- 172.21.15.42:0/38491130 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff47c013610 con 0x7ff48c1393a0 2024-09-16T03:58:12.389 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.369+0000 7ff488ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T03:58:12.389 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.369+0000 7ff488ff9640 1 -- 172.21.15.42:0/38491130 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff47c00f040 con 0x7ff48c1393a0 2024-09-16T03:58:12.389 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.369+0000 7ff49295f640 5 monclient: authenticate success, global_id 824928 2024-09-16T03:58:12.389 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.369+0000 7ff49295f640 10 monclient: _renew_subs 2024-09-16T03:58:12.389 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.369+0000 7ff49295f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:58:12.389 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.369+0000 7ff49295f640 1 -- 172.21.15.42:0/38491130 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff48c073a30 con 0x7ff48c1393a0 2024-09-16T03:58:12.389 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.369+0000 7ff488ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:58:12.389 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.369+0000 7ff488ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:58:12.389 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.369+0000 7ff489ffb640 4 set_mon_vals no callback set 2024-09-16T03:58:12.389 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.369+0000 7ff488ff9640 10 monclient: dump: 2024-09-16T03:58:12.389 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:58:12.389 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:58:12.389 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:12.390 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:12.390 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:58:12.390 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:12.390 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:58:12.390 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:58:12.390 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:58:12.390 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:12.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.369+0000 7ff49295f640 10 client.824928.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:58:12.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.369+0000 7ff49295f640 10 monclient: _renew_subs 2024-09-16T03:58:12.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.369+0000 7ff49295f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:58:12.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.369+0000 7ff49295f640 1 -- 172.21.15.42:0/38491130 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff48c139a10 con 0x7ff48c1393a0 2024-09-16T03:58:12.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.369+0000 7ff49295f640 1 librados: init done 2024-09-16T03:58:12.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.369+0000 7ff49295f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:58:12.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.369+0000 7ff49295f640 1 --2- 172.21.15.42:0/38491130 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff48c131910 0x7ff48c0724e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:12.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.369+0000 7ff49295f640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:12.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.369+0000 7ff49295f640 1 -- 172.21.15.42:0/38491130 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff48c0a1b10 con 0x7ff48c131910 2024-09-16T03:58:12.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.369+0000 7ff488ff9640 1 -- 172.21.15.42:0/38491130 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7ff47c01a070 con 0x7ff48c1393a0 2024-09-16T03:58:12.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.369+0000 7ff488ff9640 1 --2- 172.21.15.42:0/38491130 >> v2:172.21.15.154:6812/254845342 conn(0x7ff46c037810 0x7ff46c039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:12.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.369+0000 7ff488ff9640 1 -- 172.21.15.42:0/38491130 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(482..482 src has 366..482) ==== 5795+0+0 (secure 0 0 0) 0x7ff47c048b50 con 0x7ff48c1393a0 2024-09-16T03:58:12.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.369+0000 7ff488ff9640 10 client.824928.objecter ms_dispatch 0x7ff48c000ba0 osd_map(482..482 src has 366..482) 2024-09-16T03:58:12.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.369+0000 7ff488ff9640 3 client.824928.objecter handle_osd_map got epochs [482,482] > 0 2024-09-16T03:58:12.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.369+0000 7ff488ff9640 3 client.824928.objecter handle_osd_map decoding full epoch 482 2024-09-16T03:58:12.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.369+0000 7ff48bfff640 1 --2- 172.21.15.42:0/38491130 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff48c131910 0x7ff48c0724e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:12.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.369+0000 7ff48bfff640 10 monclient: get_auth_request con 0x7ff48c131910 auth_method 0 2024-09-16T03:58:12.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.369+0000 7ff48bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:12.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.369+0000 7ff48bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:12.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.369+0000 7ff48bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:58:12.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.369+0000 7ff488ff9640 20 client.824928.objecter dump_active .. 0 homeless 2024-09-16T03:58:12.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.369+0000 7ff48bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:58:12.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.369+0000 7ff48bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:58:12.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.369+0000 7ff48bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:58:12.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.369+0000 7ff48b7fe640 1 --2- 172.21.15.42:0/38491130 >> v2:172.21.15.154:6812/254845342 conn(0x7ff46c037810 0x7ff46c039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:12.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.369+0000 7ff48b7fe640 10 monclient: get_auth_request con 0x7ff46c037810 auth_method 0 2024-09-16T03:58:12.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.369+0000 7ff48bfff640 10 monclient(hunting): handle_auth_done global_id 904120 payload 824 2024-09-16T03:58:12.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.369+0000 7ff48bfff640 1 -- 172.21.15.42:0/38491130 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff48c13ba90 con 0x7ff48c131910 2024-09-16T03:58:12.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.369+0000 7ff488ff9640 10 client.824928.objecter ms_handle_connect 0x7ff48c131910 2024-09-16T03:58:12.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.369+0000 7ff488ff9640 10 client.824928.objecter resend_mon_ops 2024-09-16T03:58:12.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.369+0000 7ff48bfff640 1 --2- 172.21.15.42:0/38491130 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff48c131910 0x7ff48c0724e0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7ff48400e7c0 tx=0x7ff48400ec80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:58:12.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.369+0000 7ff488ff9640 1 -- 172.21.15.42:0/38491130 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff484003b90 con 0x7ff48c131910 2024-09-16T03:58:12.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.369+0000 7ff488ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:58:12.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.369+0000 7ff488ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:58:12.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.369+0000 7ff488ff9640 10 monclient: dump: 2024-09-16T03:58:12.393 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:58:12.393 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:58:12.393 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:12.393 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:12.393 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:58:12.393 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:12.393 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:58:12.393 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:58:12.393 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:58:12.393 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:12.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.369+0000 7ff48b7fe640 1 --2- 172.21.15.42:0/38491130 >> v2:172.21.15.154:6812/254845342 conn(0x7ff46c037810 0x7ff46c039cd0 secure :-1 s=READY pgs=3691 cs=0 l=1 rev1=1 crypto rx=0x7ff4740b1000 tx=0x7ff4740a9070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:58:12.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.369+0000 7ff488ff9640 10 client.824928.objecter ms_handle_connect 0x7ff46c037810 2024-09-16T03:58:12.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.369+0000 7ff488ff9640 1 -- 172.21.15.42:0/38491130 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7ff484003b90 con 0x7ff48c131910 2024-09-16T03:58:12.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.369+0000 7ff488ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:58:12.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.369+0000 7ff488ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:58:12.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.369+0000 7ff488ff9640 1 -- 172.21.15.42:0/38491130 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff48c131910 msgr2=0x7ff48c0724e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:12.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.369+0000 7ff488ff9640 1 --2- 172.21.15.42:0/38491130 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff48c131910 0x7ff48c0724e0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7ff48400e7c0 tx=0x7ff48400ec80 comp rx=0 tx=0).stop 2024-09-16T03:58:12.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.373+0000 7ff488ff9640 1 -- 172.21.15.42:0/38491130 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff48c131910 msgr2=0x7ff48c0724e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:58:12.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.373+0000 7ff488ff9640 1 --2- 172.21.15.42:0/38491130 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff48c131910 0x7ff48c0724e0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:12.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.377+0000 7ff49295f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:58:12.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.377+0000 7ff49295f640 1 --2- 172.21.15.42:0/38491130 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff48c02c650 0x7ff48c02ca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:12.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.377+0000 7ff49295f640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:12.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.377+0000 7ff49295f640 1 -- 172.21.15.42:0/38491130 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff48c136a90 con 0x7ff48c02c650 2024-09-16T03:58:12.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.377+0000 7ff48bfff640 1 --2- 172.21.15.42:0/38491130 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff48c02c650 0x7ff48c02ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:12.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.377+0000 7ff48bfff640 10 monclient: get_auth_request con 0x7ff48c02c650 auth_method 0 2024-09-16T03:58:12.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.377+0000 7ff48bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:12.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.377+0000 7ff48bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:12.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.377+0000 7ff48bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:58:12.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.377+0000 7ff48bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:58:12.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.377+0000 7ff48bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:58:12.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.377+0000 7ff48bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:58:12.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.377+0000 7ff48bfff640 10 monclient(hunting): handle_auth_done global_id 904123 payload 824 2024-09-16T03:58:12.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.377+0000 7ff48bfff640 1 -- 172.21.15.42:0/38491130 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ff48c069040 con 0x7ff48c02c650 2024-09-16T03:58:12.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.377+0000 7ff48bfff640 1 --2- 172.21.15.42:0/38491130 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff48c02c650 0x7ff48c02ca30 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7ff48400e7c0 tx=0x7ff484001260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:58:12.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.377+0000 7ff488ff9640 10 client.824928.objecter ms_handle_connect 0x7ff48c02c650 2024-09-16T03:58:12.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.377+0000 7ff488ff9640 10 client.824928.objecter resend_mon_ops 2024-09-16T03:58:12.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.377+0000 7ff488ff9640 1 -- 172.21.15.42:0/38491130 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff48401cc50 con 0x7ff48c02c650 2024-09-16T03:58:12.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.377+0000 7ff488ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:58:12.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.377+0000 7ff488ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:58:12.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.377+0000 7ff488ff9640 10 monclient: dump: 2024-09-16T03:58:12.397 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:58:12.397 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:58:12.397 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:12.397 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:12.397 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:58:12.397 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:12.397 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:58:12.398 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:58:12.398 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:58:12.398 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:12.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.381+0000 7ff488ff9640 1 -- 172.21.15.42:0/38491130 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4545 (secure 0 0 0) 0x7ff4840050d0 con 0x7ff48c02c650 2024-09-16T03:58:12.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.381+0000 7ff488ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:58:12.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.381+0000 7ff488ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:58:12.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.381+0000 7ff488ff9640 1 -- 172.21.15.42:0/38491130 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff48c02c650 msgr2=0x7ff48c02ca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:12.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.381+0000 7ff488ff9640 1 --2- 172.21.15.42:0/38491130 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff48c02c650 0x7ff48c02ca30 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7ff48400e7c0 tx=0x7ff484001260 comp rx=0 tx=0).stop 2024-09-16T03:58:12.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.381+0000 7ff488ff9640 1 -- 172.21.15.42:0/38491130 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff48c02c650 msgr2=0x7ff48c02ca30 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:58:12.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.381+0000 7ff488ff9640 1 --2- 172.21.15.42:0/38491130 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff48c02c650 0x7ff48c02ca30 secure :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7ff48400e7c0 tx=0x7ff484001260 comp rx=0 tx=0).stop 2024-09-16T03:58:12.398 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:58:12.399 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:58:12.399 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:58:12.399 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T03:58:12.399 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 148, 2024-09-16T03:58:12.399 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 20563, 2024-09-16T03:58:12.399 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:58:12.399 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:58:12.399 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T03:58:12.399 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T03:58:12.399 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:58:12.399 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 0, 2024-09-16T03:58:12.399 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:58:12.399 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:58:12.400 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:58:12.400 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:58:12.400 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:58:12.400 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:58:12.400 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:58:12.400 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:58:12.400 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:58:12.400 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:58:12.400 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:58:12.400 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:58:12.400 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:58:12.400 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:58:12.400 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:58:12.401 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:58:12.401 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:58:12.401 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:58:12.401 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:58:12.401 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:58:12.401 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:58:12.401 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:58:12.401 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:58:12.401 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:58:12.401 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:58:12.401 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:58:12.401 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:58:12.401 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:58:12.401 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:58:12.402 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:58:12.402 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:58:12.402 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:58:12.402 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:58:12.402 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:58:12.402 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:58:12.402 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:58:12.402 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:58:12.402 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:58:12.402 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:58:12.402 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:58:12.402 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:58:12.402 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:58:12.403 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:58:12.403 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:58:12.403 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:58:12.403 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:58:12.403 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:58:12.403 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:58:12.403 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:58:12.403 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:58:12.403 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:58:12.403 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:58:12.403 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:58:12.403 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:58:12.403 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:58:12.403 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:58:12.404 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:58:12.404 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:58:12.404 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:58:12.404 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:58:12.404 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:58:12.404 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:58:12.404 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:58:12.404 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:58:12.404 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:58:12.404 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:58:12.404 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:58:12.404 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:58:12.404 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:58:12.404 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:58:12.405 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:58:12.405 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:58:12.405 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:58:12.405 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:58:12.405 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:58:12.405 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:58:12.405 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:58:12.405 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:58:12.405 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:58:12.405 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:58:12.405 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:58:12.405 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:58:12.405 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:58:12.405 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:58:12.406 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:58:12.406 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:58:12.406 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:58:12.406 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:58:12.406 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:58:12.406 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:58:12.406 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:58:12.406 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:58:12.406 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:58:12.406 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:58:12.406 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:58:12.406 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:58:12.406 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:58:12.406 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:58:12.407 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:58:12.407 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:58:12.407 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:58:12.407 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:58:12.407 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:58:12.407 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:58:12.407 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:58:12.407 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:58:12.407 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:58:12.407 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:58:12.407 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:58:12.407 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:58:12.407 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:58:12.407 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:58:12.408 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:58:12.408 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:58:12.408 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:58:12.408 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:58:12.408 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:58:12.408 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:58:12.408 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:58:12.408 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:58:12.408 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:58:12.408 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:58:12.408 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:58:12.408 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:58:12.408 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:58:12.408 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:58:12.409 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:58:12.409 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:58:12.409 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:58:12.409 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:58:12.409 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:58:12.409 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:58:12.409 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:58:12.409 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:58:12.409 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:58:12.409 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:58:12.409 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:58:12.409 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:58:12.409 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:58:12.409 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:58:12.410 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:58:12.410 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:58:12.410 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:58:12.410 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:58:12.410 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:58:12.410 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:58:12.410 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:58:12.410 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:58:12.410 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:58:12.410 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:58:12.410 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:58:12.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.381+0000 7ff49295f640 10 librados: watch_flush enter 2024-09-16T03:58:12.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.381+0000 7ff49295f640 10 librados: watch_flush exit 2024-09-16T03:58:12.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.381+0000 7ff49295f640 20 client.824928.objecter shutdown clearing up homeless session... 2024-09-16T03:58:12.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.381+0000 7ff49295f640 10 client.824928.objecter successfully canceled tick 2024-09-16T03:58:12.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.381+0000 7ff49295f640 1 -- 172.21.15.42:0/38491130 >> v2:172.21.15.154:6812/254845342 conn(0x7ff46c037810 msgr2=0x7ff46c039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:12.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.381+0000 7ff49295f640 1 --2- 172.21.15.42:0/38491130 >> v2:172.21.15.154:6812/254845342 conn(0x7ff46c037810 0x7ff46c039cd0 secure :-1 s=READY pgs=3691 cs=0 l=1 rev1=1 crypto rx=0x7ff4740b1000 tx=0x7ff4740a9070 comp rx=0 tx=0).stop 2024-09-16T03:58:12.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.381+0000 7ff49295f640 10 monclient: shutdown 2024-09-16T03:58:12.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.381+0000 7ff49295f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:58:12.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.381+0000 7ff49295f640 1 -- 172.21.15.42:0/38491130 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff48c1393a0 msgr2=0x7ff48c06e950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:12.413 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.381+0000 7ff49295f640 1 --2- 172.21.15.42:0/38491130 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff48c1393a0 0x7ff48c06e950 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7ff47c00e5e0 tx=0x7ff47c00eaa0 comp rx=0 tx=0).stop 2024-09-16T03:58:12.413 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.381+0000 7ff490efc640 1 -- 172.21.15.42:0/38491130 reap_dead start 2024-09-16T03:58:12.413 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.381+0000 7ff490efc640 1 -- 172.21.15.42:0/38491130 reap_dead start 2024-09-16T03:58:12.413 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.381+0000 7ff49295f640 1 -- 172.21.15.42:0/38491130 shutdown_connections 2024-09-16T03:58:12.413 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.381+0000 7ff49295f640 1 -- 172.21.15.42:0/38491130 >> 172.21.15.42:0/38491130 conn(0x7ff48c0a2420 msgr2=0x7ff48c1342f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:58:12.413 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.381+0000 7ff49295f640 1 -- 172.21.15.42:0/38491130 shutdown_connections 2024-09-16T03:58:12.413 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.381+0000 7ff49295f640 1 -- 172.21.15.42:0/38491130 wait complete. 2024-09-16T03:58:12.413 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.381+0000 7ff49295f640 1 librados: shutdown 2024-09-16T03:58:12.413 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:58:12.413 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T03:58:12.414 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T03:58:12.532 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.525+0000 7fbfb6b6d640 10 monclient: get_monmap_and_config 2024-09-16T03:58:12.533 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.525+0000 7fbfb6b6d640 10 monclient: build_initial_monmap 2024-09-16T03:58:12.533 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.525+0000 7fbfb6b6d640 10 monclient: monmap: 2024-09-16T03:58:12.533 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:58:12.533 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:58:12.533 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:58:12.531408+0000 2024-09-16T03:58:12.533 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:58:12.531408+0000 2024-09-16T03:58:12.533 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:58:12.533 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:12.533 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:58:12.533 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:58:12.534 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:58:12.534 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:58:12.534 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:58:12.534 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:12.536 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.529+0000 7fbfb6b6d640 1 Processor -- start 2024-09-16T03:58:12.536 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.529+0000 7fbfb6b6d640 1 -- start start 2024-09-16T03:58:12.536 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.529+0000 7fbfb6b6d640 10 monclient: init 2024-09-16T03:58:12.536 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.529+0000 7fbfb6b6d640 10 monclient: _reopen_session rank -1 2024-09-16T03:58:12.536 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.529+0000 7fbfb6b6d640 10 monclient: _add_conns ranks=[3,2,1,0,4] 2024-09-16T03:58:12.536 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.529+0000 7fbfb6b6d640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fbfb00a1330 0x7fbfb00a1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:12.537 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.529+0000 7fbfb6b6d640 10 monclient(hunting): picked mon.noname-a con 0x7fbfb00a1330 addr v2:172.21.15.42:3300/0 2024-09-16T03:58:12.537 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.529+0000 7fbfb6b6d640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbfb006dac0 0x7fbfb00a1c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:12.537 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.529+0000 7fbfb6b6d640 10 monclient(hunting): picked mon.noname-b con 0x7fbfb006dac0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:58:12.537 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.529+0000 7fbfb6b6d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbfb0135100 addr v1:172.21.15.154:6789/0 2024-09-16T03:58:12.537 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.529+0000 7fbfb6b6d640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbfb009c7d0 con 0x7fbfb0135100 2024-09-16T03:58:12.537 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.529+0000 7fbfb6b6d640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:12.537 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.529+0000 7fbfb6b6d640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7fbfb009c090 con 0x7fbfb00a1330 2024-09-16T03:58:12.537 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.529+0000 7fbfb6b6d640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:12.537 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.529+0000 7fbfb6b6d640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fbfb009f610 con 0x7fbfb006dac0 2024-09-16T03:58:12.537 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.529+0000 7fbfb6b6d640 10 monclient(hunting): _renew_subs 2024-09-16T03:58:12.537 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.529+0000 7fbfb6b6d640 10 monclient(hunting): authenticate will time out at 1939.500891s 2024-09-16T03:58:12.537 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.529+0000 7fbfb4909640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fbfb00a1330 0x7fbfb00a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:12.537 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.529+0000 7fbfaffff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbfb006dac0 0x7fbfb00a1c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:12.538 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.529+0000 7fbfaffff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbfb006dac0 0x7fbfb00a1c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:49074/0 (socket says 172.21.15.42:49074) 2024-09-16T03:58:12.538 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.529+0000 7fbfb4909640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fbfb00a1330 0x7fbfb00a1710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:36036/0 (socket says 172.21.15.42:36036) 2024-09-16T03:58:12.538 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.529+0000 7fbfaffff640 1 -- 172.21.15.42:0/2377352081 learned_addr learned my addr 172.21.15.42:0/2377352081 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:58:12.538 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.529+0000 7fbfaffff640 10 monclient(hunting): get_auth_request con 0x7fbfb006dac0 auth_method 0 2024-09-16T03:58:12.538 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.529+0000 7fbfaffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:12.538 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.529+0000 7fbfaffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:12.538 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.529+0000 7fbfaffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:58:12.538 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.529+0000 7fbfb4909640 10 monclient(hunting): get_auth_request con 0x7fbfb00a1330 auth_method 0 2024-09-16T03:58:12.538 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.529+0000 7fbfb4909640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:12.538 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.529+0000 7fbfb4909640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:12.538 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.529+0000 7fbfb4909640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:58:12.538 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.529+0000 7fbfaffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:58:12.538 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.529+0000 7fbfaffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:58:12.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.529+0000 7fbfaffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:58:12.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.529+0000 7fbfb4909640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:58:12.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.529+0000 7fbfb4909640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:58:12.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.529+0000 7fbfb4909640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:58:12.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.529+0000 7fbfaffff640 10 monclient(hunting): handle_auth_done global_id 854456 payload 274 2024-09-16T03:58:12.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.529+0000 7fbfaffff640 1 -- 172.21.15.42:0/2377352081 >> v2:172.21.15.42:3300/0 conn(0x7fbfb00a1330 msgr2=0x7fbfb00a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:58:12.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.529+0000 7fbfaffff640 1 --2- 172.21.15.42:0/2377352081 >> v2:172.21.15.42:3300/0 conn(0x7fbfb00a1330 0x7fbfb00a1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:12.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.529+0000 7fbfaffff640 1 -- 172.21.15.42:0/2377352081 >> v1:172.21.15.154:6789/0 conn(0x7fbfb0135100 legacy=0x7fbfb0135550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:12.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.529+0000 7fbfaffff640 10 monclient: _finish_hunting 0 2024-09-16T03:58:12.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.529+0000 7fbfaffff640 1 monclient: found mon.noname-b 2024-09-16T03:58:12.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.529+0000 7fbfaffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:58:12.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.529+0000 7fbfaffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T03:58:12.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.529+0000 7fbfaffff640 1 -- 172.21.15.42:0/2377352081 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfb009f8f0 con 0x7fbfb006dac0 2024-09-16T03:58:12.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.529+0000 7fbfb4909640 1 --2- 172.21.15.42:0/2377352081 >> v2:172.21.15.42:3300/0 conn(0x7fbfb00a1330 0x7fbfb00a1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:58:12.545 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.533+0000 7fbfaffff640 1 --2- 172.21.15.42:0/2377352081 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbfb006dac0 0x7fbfb00a1c50 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fbfa4009870 tx=0x7fbfa401c540 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f59c6f6b0c3fc400 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:58:12.545 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.533+0000 7fbfaf7fe640 1 -- 172.21.15.42:0/2377352081 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfa40ab650 con 0x7fbfb006dac0 2024-09-16T03:58:12.545 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.533+0000 7fbfaf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:58:12.545 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.533+0000 7fbfaf7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:58:12.545 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.533+0000 7fbfaf7fe640 10 monclient: dump: 2024-09-16T03:58:12.545 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:58:12.545 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:58:12.545 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:12.545 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:12.545 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:58:12.545 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:12.546 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:58:12.546 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:58:12.546 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:58:12.546 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:12.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.533+0000 7fbfaf7fe640 10 monclient: _finish_auth 0 2024-09-16T03:58:12.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.533+0000 7fbfaf7fe640 10 monclient: _check_auth_tickets 2024-09-16T03:58:12.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.533+0000 7fbfaf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:58:12.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.533+0000 7fbfb6b6d640 5 monclient: authenticate success, global_id 854456 2024-09-16T03:58:12.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.533+0000 7fbfb6b6d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:58:12.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.533+0000 7fbfaf7fe640 1 -- 172.21.15.42:0/2377352081 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbfa40b0070 con 0x7fbfb006dac0 2024-09-16T03:58:12.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.533+0000 7fbfaf7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:58:12.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.533+0000 7fbfb6b6d640 10 monclient: get_monmap_and_config success 2024-09-16T03:58:12.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.533+0000 7fbfb6b6d640 4 set_mon_vals no callback set 2024-09-16T03:58:12.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.533+0000 7fbfaf7fe640 1 -- 172.21.15.42:0/2377352081 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfa40aba10 con 0x7fbfb006dac0 2024-09-16T03:58:12.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.533+0000 7fbfaf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:58:12.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.533+0000 7fbfaf7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:58:12.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.533+0000 7fbfaf7fe640 10 monclient: dump: 2024-09-16T03:58:12.547 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:58:12.547 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:58:12.547 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:12.547 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:12.547 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:58:12.547 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:12.547 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:58:12.547 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:58:12.547 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:58:12.547 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:12.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.533+0000 7fbfb6b6d640 10 monclient: shutdown 2024-09-16T03:58:12.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.533+0000 7fbfb6b6d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:58:12.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.533+0000 7fbfb6b6d640 1 -- 172.21.15.42:0/2377352081 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbfb006dac0 msgr2=0x7fbfb00a1c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:12.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.533+0000 7fbfb6b6d640 1 --2- 172.21.15.42:0/2377352081 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbfb006dac0 0x7fbfb00a1c50 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fbfa4009870 tx=0x7fbfa401c540 comp rx=0 tx=0).stop 2024-09-16T03:58:12.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.533+0000 7fbfb6b6d640 1 -- 172.21.15.42:0/2377352081 shutdown_connections 2024-09-16T03:58:12.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.533+0000 7fbfb6b6d640 1 --2- 172.21.15.42:0/2377352081 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbfb006dac0 0x7fbfb00a1c50 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:12.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.533+0000 7fbfb6b6d640 1 --2- 172.21.15.42:0/2377352081 >> v2:172.21.15.42:3300/0 conn(0x7fbfb00a1330 0x7fbfb00a1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:12.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.533+0000 7fbfb6b6d640 1 -- 172.21.15.42:0/2377352081 >> 172.21.15.42:0/2377352081 conn(0x7fbfb00a2420 msgr2=0x7fbfb00a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:58:12.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.533+0000 7fbfb6b6d640 1 -- 172.21.15.42:0/2377352081 shutdown_connections 2024-09-16T03:58:12.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.533+0000 7fbfb6b6d640 1 -- 172.21.15.42:0/2377352081 wait complete. 2024-09-16T03:58:12.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.533+0000 7fbfb6b6d640 10 monclient: build_initial_monmap 2024-09-16T03:58:12.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.533+0000 7fbfb6b6d640 10 monclient: monmap: 2024-09-16T03:58:12.548 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:58:12.549 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:58:12.549 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:58:12.549 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:58:12.549 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:58:12.549 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:12.549 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:58:12.549 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:58:12.549 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:58:12.549 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:12.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.533+0000 7fbfb6b6d640 1 librados: starting msgr at 2024-09-16T03:58:12.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.533+0000 7fbfb6b6d640 1 librados: starting objecter 2024-09-16T03:58:12.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.533+0000 7fbfb6b6d640 1 Processor -- start 2024-09-16T03:58:12.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.533+0000 7fbfb6b6d640 1 -- start start 2024-09-16T03:58:12.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.533+0000 7fbfb6b6d640 1 librados: setting wanted keys 2024-09-16T03:58:12.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.533+0000 7fbfb6b6d640 1 librados: calling monclient init 2024-09-16T03:58:12.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.533+0000 7fbfb6b6d640 10 monclient: init 2024-09-16T03:58:12.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.533+0000 7fbfb6b6d640 10 monclient: _reopen_session rank -1 2024-09-16T03:58:12.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.533+0000 7fbfb6b6d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:58:12.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.533+0000 7fbfb6b6d640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbfb00a1330 0x7fbfb013bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:12.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.533+0000 7fbfb6b6d640 10 monclient(hunting): picked mon.noname-a con 0x7fbfb00a1330 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:58:12.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.533+0000 7fbfb6b6d640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbfb0135100 0x7fbfb01343a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:12.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.533+0000 7fbfb6b6d640 10 monclient(hunting): picked mon.noname-c con 0x7fbfb0135100 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:58:12.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.533+0000 7fbfb6b6d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbfb01348e0 0x7fbfb01cf0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:12.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.533+0000 7fbfb6b6d640 10 monclient(hunting): picked mon.noname-b con 0x7fbfb01348e0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:58:12.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.533+0000 7fbfb6b6d640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:12.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.533+0000 7fbfb6b6d640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fbfb013fd00 con 0x7fbfb00a1330 2024-09-16T03:58:12.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.533+0000 7fbfb6b6d640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:12.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.533+0000 7fbfb6b6d640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fbfb013fb80 con 0x7fbfb01348e0 2024-09-16T03:58:12.551 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.533+0000 7fbfb6b6d640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:12.551 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.533+0000 7fbfb6b6d640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fbfb013fe80 con 0x7fbfb0135100 2024-09-16T03:58:12.551 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.533+0000 7fbfb6b6d640 10 monclient(hunting): _renew_subs 2024-09-16T03:58:12.551 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.533+0000 7fbfb6b6d640 10 monclient(hunting): authenticate will time out at 1939.507201s 2024-09-16T03:58:12.551 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.533+0000 7fbfb4909640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbfb00a1330 0x7fbfb013bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:12.551 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.533+0000 7fbfb4909640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbfb00a1330 0x7fbfb013bfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:36040/0 (socket says 172.21.15.42:36040) 2024-09-16T03:58:12.551 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.533+0000 7fbfb4909640 1 -- 172.21.15.42:0/1720043992 learned_addr learned my addr 172.21.15.42:0/1720043992 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:58:12.551 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.533+0000 7fbfb4909640 10 monclient(hunting): get_auth_request con 0x7fbfb00a1330 auth_method 0 2024-09-16T03:58:12.551 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.533+0000 7fbfb4909640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:12.551 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.533+0000 7fbfb4909640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:12.551 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.533+0000 7fbfb4909640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:58:12.552 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.533+0000 7fbfaffff640 1 --2- 172.21.15.42:0/1720043992 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbfb0135100 0x7fbfb01343a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:12.552 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.533+0000 7fbfaffff640 10 monclient(hunting): get_auth_request con 0x7fbfb0135100 auth_method 0 2024-09-16T03:58:12.552 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.533+0000 7fbfaffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:12.552 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.533+0000 7fbfaffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:12.552 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.533+0000 7fbfaffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:58:12.552 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.533+0000 7fbfb4909640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:58:12.552 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.533+0000 7fbfb4909640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:58:12.552 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.533+0000 7fbfb4909640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:58:12.552 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.533+0000 7fbfaffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:58:12.552 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.533+0000 7fbfaffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:58:12.552 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.533+0000 7fbfaffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:58:12.552 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.533+0000 7fbfb510a640 1 --2- 172.21.15.42:0/1720043992 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbfb01348e0 0x7fbfb01cf0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:12.552 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.533+0000 7fbfb510a640 10 monclient(hunting): get_auth_request con 0x7fbfb01348e0 auth_method 0 2024-09-16T03:58:12.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.533+0000 7fbfb510a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:12.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.533+0000 7fbfb510a640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:12.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.533+0000 7fbfb510a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:58:12.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.533+0000 7fbfb4909640 10 monclient(hunting): handle_auth_done global_id 824934 payload 824 2024-09-16T03:58:12.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.533+0000 7fbfb4909640 1 -- 172.21.15.42:0/1720043992 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbfb0135100 msgr2=0x7fbfb01343a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:12.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.533+0000 7fbfb4909640 1 --2- 172.21.15.42:0/1720043992 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbfb0135100 0x7fbfb01343a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:12.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.533+0000 7fbfb4909640 1 -- 172.21.15.42:0/1720043992 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbfb01348e0 msgr2=0x7fbfb01cf0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:12.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.533+0000 7fbfb4909640 1 --2- 172.21.15.42:0/1720043992 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbfb01348e0 0x7fbfb01cf0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:12.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.533+0000 7fbfb4909640 10 monclient: _finish_hunting 0 2024-09-16T03:58:12.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.533+0000 7fbfb4909640 1 monclient: found mon.noname-a 2024-09-16T03:58:12.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.533+0000 7fbfb4909640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:58:12.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.533+0000 7fbfb4909640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:58:12.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.533+0000 7fbfb4909640 1 -- 172.21.15.42:0/1720043992 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfb0120d50 con 0x7fbfb00a1330 2024-09-16T03:58:12.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.533+0000 7fbfaffff640 1 --2- 172.21.15.42:0/1720043992 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbfb0135100 0x7fbfb01343a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:58:12.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.533+0000 7fbfb4909640 1 --2- 172.21.15.42:0/1720043992 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbfb00a1330 0x7fbfb013bfb0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fbfa000ebe0 tx=0x7fbfa000c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:58:12.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.533+0000 7fbfad7fa640 10 client.?.objecter ms_handle_connect 0x7fbfb00a1330 2024-09-16T03:58:12.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.537+0000 7fbfad7fa640 10 client.?.objecter resend_mon_ops 2024-09-16T03:58:12.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.537+0000 7fbfad7fa640 1 -- 172.21.15.42:0/1720043992 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfa0017070 con 0x7fbfb00a1330 2024-09-16T03:58:12.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.537+0000 7fbfad7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:58:12.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.537+0000 7fbfad7fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:58:12.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.537+0000 7fbfad7fa640 10 monclient: dump: 2024-09-16T03:58:12.554 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:58:12.554 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:58:12.554 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:12.554 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:12.554 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:58:12.554 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:12.555 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:58:12.555 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:58:12.555 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:58:12.555 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:12.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.537+0000 7fbfad7fa640 10 monclient: _finish_auth 0 2024-09-16T03:58:12.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.537+0000 7fbfad7fa640 10 monclient: _check_auth_tickets 2024-09-16T03:58:12.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.537+0000 7fbfad7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:58:12.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.537+0000 7fbfad7fa640 1 -- 172.21.15.42:0/1720043992 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbfa000cc30 con 0x7fbfb00a1330 2024-09-16T03:58:12.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.537+0000 7fbfb6b6d640 5 monclient: authenticate success, global_id 824934 2024-09-16T03:58:12.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.537+0000 7fbfb6b6d640 10 monclient: _renew_subs 2024-09-16T03:58:12.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.537+0000 7fbfb6b6d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:58:12.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.537+0000 7fbfb6b6d640 1 -- 172.21.15.42:0/1720043992 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbfb0071a70 con 0x7fbfb00a1330 2024-09-16T03:58:12.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.537+0000 7fbfad7fa640 10 monclient: handle_config config(0 keys) 2024-09-16T03:58:12.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.537+0000 7fbfad7fa640 1 -- 172.21.15.42:0/1720043992 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfa0013470 con 0x7fbfb00a1330 2024-09-16T03:58:12.556 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.537+0000 7fbfad7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:58:12.556 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.537+0000 7fbfad7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:58:12.556 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.537+0000 7fbfad7fa640 10 monclient: dump: 2024-09-16T03:58:12.556 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:58:12.556 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:58:12.556 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:12.556 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:12.556 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:58:12.556 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:12.556 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:58:12.556 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:58:12.556 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:58:12.556 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:12.556 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.537+0000 7fbfb6b6d640 10 client.824934.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:58:12.557 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.537+0000 7fbfb6b6d640 10 monclient: _renew_subs 2024-09-16T03:58:12.557 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.537+0000 7fbfb6b6d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:58:12.557 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.537+0000 7fbfb6b6d640 1 -- 172.21.15.42:0/1720043992 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbfb0120a70 con 0x7fbfb00a1330 2024-09-16T03:58:12.557 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.537+0000 7fbfaeffd640 4 set_mon_vals no callback set 2024-09-16T03:58:12.557 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.537+0000 7fbfb6b6d640 1 librados: init done 2024-09-16T03:58:12.557 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.537+0000 7fbfb6b6d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:58:12.557 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.537+0000 7fbfb6b6d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:58:12.557 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.537+0000 7fbfb6b6d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:58:12.557 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.537+0000 7fbfb6b6d640 1 -- 172.21.15.42:0/1720043992 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbfb0135510 con 0x7fbfb00a1330 2024-09-16T03:58:12.557 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.537+0000 7fbfad7fa640 1 -- 172.21.15.42:0/1720043992 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fbfa001a070 con 0x7fbfb00a1330 2024-09-16T03:58:12.557 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.537+0000 7fbfad7fa640 1 --2- 172.21.15.42:0/1720043992 >> v2:172.21.15.154:6812/254845342 conn(0x7fbf9c037810 0x7fbf9c039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:12.557 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.537+0000 7fbfad7fa640 1 -- 172.21.15.42:0/1720043992 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(482..482 src has 366..482) ==== 5795+0+0 (secure 0 0 0) 0x7fbfa0048f90 con 0x7fbfb00a1330 2024-09-16T03:58:12.557 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.537+0000 7fbfad7fa640 10 client.824934.objecter ms_dispatch 0x7fbfb0000ba0 osd_map(482..482 src has 366..482) 2024-09-16T03:58:12.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.537+0000 7fbfad7fa640 3 client.824934.objecter handle_osd_map got epochs [482,482] > 0 2024-09-16T03:58:12.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.537+0000 7fbfad7fa640 3 client.824934.objecter handle_osd_map decoding full epoch 482 2024-09-16T03:58:12.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.537+0000 7fbfad7fa640 20 client.824934.objecter dump_active .. 0 homeless 2024-09-16T03:58:12.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.537+0000 7fbfaffff640 1 --2- 172.21.15.42:0/1720043992 >> v2:172.21.15.154:6812/254845342 conn(0x7fbf9c037810 0x7fbf9c039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:12.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.537+0000 7fbfaffff640 10 monclient: get_auth_request con 0x7fbf9c037810 auth_method 0 2024-09-16T03:58:12.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.537+0000 7fbfad7fa640 10 client.824934.objecter ms_handle_connect 0x7fbf9c037810 2024-09-16T03:58:12.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.537+0000 7fbfaffff640 1 --2- 172.21.15.42:0/1720043992 >> v2:172.21.15.154:6812/254845342 conn(0x7fbf9c037810 0x7fbf9c039cd0 secure :-1 s=READY pgs=3692 cs=0 l=1 rev1=1 crypto rx=0x7fbfa40af000 tx=0x7fbfa4004740 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:58:12.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.541+0000 7fbfad7fa640 1 -- 172.21.15.42:0/1720043992 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7fbfa0011350 con 0x7fbfb00a1330 2024-09-16T03:58:12.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.541+0000 7fbfad7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:58:12.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.541+0000 7fbfad7fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:58:12.786 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.781+0000 7fbfb6b6d640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T03:58:12.786 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.781+0000 7fbfb6b6d640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T03:58:12.786 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.781+0000 7fbfb6b6d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:58:12.786 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.781+0000 7fbfb6b6d640 1 -- 172.21.15.42:0/1720043992 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7fbfb0069040 con 0x7fbfb00a1330 2024-09-16T03:58:12.788 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.781+0000 7fbfad7fa640 1 -- 172.21.15.42:0/1720043992 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7fbfa0013610 con 0x7fbfb00a1330 2024-09-16T03:58:12.788 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.781+0000 7fbfad7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T03:58:12.788 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.781+0000 7fbfad7fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:58:12.789 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.781+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.789 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.781+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=31,config=2,health=11,kv=9,logm=47} crc {auth=4158835543,config=692619927,health=540224681,kv=3915008329,logm=2327135771}) 2024-09-16T03:58:12.789 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.781+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=31,config=1,health=12,kv=10,logm=46} crc {auth=888610198,config=1228700967,health=1487215615,kv=3446188498,logm=1150612604}) 2024-09-16T03:58:12.789 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.781+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.789 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.781+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=31,config=2,health=11,kv=9,logm=47} crc {auth=4158835543,config=692619927,health=540224681,kv=3915008329,logm=2327135771}) 2024-09-16T03:58:12.789 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.781+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=28,config=2,health=12,kv=9,logm=49} crc {auth=3662100421,config=692619927,health=1487215615,kv=3276479304,logm=2779284600}) 2024-09-16T03:58:12.792 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.785+0000 7fbfb6b6d640 10 librados: watch_flush enter 2024-09-16T03:58:12.792 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.785+0000 7fbfb6b6d640 10 librados: watch_flush exit 2024-09-16T03:58:12.792 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.785+0000 7fbfb6b6d640 20 client.824934.objecter shutdown clearing up homeless session... 2024-09-16T03:58:12.792 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.785+0000 7fbfb6b6d640 10 client.824934.objecter successfully canceled tick 2024-09-16T03:58:12.792 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.785+0000 7fbfb6b6d640 1 -- 172.21.15.42:0/1720043992 >> v2:172.21.15.154:6812/254845342 conn(0x7fbf9c037810 msgr2=0x7fbf9c039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:12.792 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.785+0000 7fbfb6b6d640 1 --2- 172.21.15.42:0/1720043992 >> v2:172.21.15.154:6812/254845342 conn(0x7fbf9c037810 0x7fbf9c039cd0 secure :-1 s=READY pgs=3692 cs=0 l=1 rev1=1 crypto rx=0x7fbfa40af000 tx=0x7fbfa4004740 comp rx=0 tx=0).stop 2024-09-16T03:58:12.792 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.785+0000 7fbfb6b6d640 10 monclient: shutdown 2024-09-16T03:58:12.792 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.785+0000 7fbfb6b6d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:58:12.792 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.785+0000 7fbfb6b6d640 1 -- 172.21.15.42:0/1720043992 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbfb00a1330 msgr2=0x7fbfb013bfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:12.792 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.785+0000 7fbfb6b6d640 1 --2- 172.21.15.42:0/1720043992 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbfb00a1330 0x7fbfb013bfb0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fbfa000ebe0 tx=0x7fbfa000c520 comp rx=0 tx=0).stop 2024-09-16T03:58:12.793 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.785+0000 7fbfb6b6d640 1 -- 172.21.15.42:0/1720043992 shutdown_connections 2024-09-16T03:58:12.793 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.785+0000 7fbfb6b6d640 1 --2- 172.21.15.42:0/1720043992 >> v2:172.21.15.154:6812/254845342 conn(0x7fbf9c037810 0x7fbf9c039cd0 unknown :-1 s=CLOSED pgs=3692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:12.793 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.785+0000 7fbfb6b6d640 1 --2- 172.21.15.42:0/1720043992 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbfb01348e0 0x7fbfb01cf0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:12.793 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.785+0000 7fbfb6b6d640 1 --2- 172.21.15.42:0/1720043992 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbfb0135100 0x7fbfb01343a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:12.793 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.785+0000 7fbfb6b6d640 1 --2- 172.21.15.42:0/1720043992 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbfb00a1330 0x7fbfb013bfb0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:12.793 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.785+0000 7fbfb6b6d640 1 -- 172.21.15.42:0/1720043992 >> 172.21.15.42:0/1720043992 conn(0x7fbfb00a2420 msgr2=0x7fbfb00a3720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:58:12.793 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.785+0000 7fbfb6b6d640 1 -- 172.21.15.42:0/1720043992 shutdown_connections 2024-09-16T03:58:12.793 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.785+0000 7fbfb6b6d640 1 -- 172.21.15.42:0/1720043992 wait complete. 2024-09-16T03:58:12.793 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:12.785+0000 7fbfb6b6d640 1 librados: shutdown 2024-09-16T03:58:12.794 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.785+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.794 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.785+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=283238157}) 2024-09-16T03:58:12.794 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.785+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2611159300}) 2024-09-16T03:58:12.794 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.785+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.794 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.785+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=283238157}) 2024-09-16T03:58:12.794 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.785+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3153464089}) 2024-09-16T03:58:12.795 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.789+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.795 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.789+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4127343403}) 2024-09-16T03:58:12.795 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.789+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2758853954}) 2024-09-16T03:58:12.795 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.789+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.795 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.789+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4127343403}) 2024-09-16T03:58:12.796 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.789+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=306483334}) 2024-09-16T03:58:12.799 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.793+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.799 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.793+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2720986498}) 2024-09-16T03:58:12.799 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.793+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3283228766}) 2024-09-16T03:58:12.799 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.793+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.799 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.793+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2720986498}) 2024-09-16T03:58:12.799 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.793+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2558124030}) 2024-09-16T03:58:12.802 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.797+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.802 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.797+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3225272148}) 2024-09-16T03:58:12.802 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.797+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3461671302}) 2024-09-16T03:58:12.802 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.797+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.802 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.797+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3225272148}) 2024-09-16T03:58:12.802 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.797+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1750325939}) 2024-09-16T03:58:12.804 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.797+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.804 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.797+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1203305025}) 2024-09-16T03:58:12.805 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.797+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2069521682}) 2024-09-16T03:58:12.805 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.797+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.805 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.797+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1203305025}) 2024-09-16T03:58:12.805 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.797+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=256216690}) 2024-09-16T03:58:12.806 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.801+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.806 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.801+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1803386779}) 2024-09-16T03:58:12.807 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.801+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2381412835}) 2024-09-16T03:58:12.807 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.801+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.807 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.801+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1803386779}) 2024-09-16T03:58:12.807 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.801+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3299086833}) 2024-09-16T03:58:12.809 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.801+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.809 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.801+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2651091816}) 2024-09-16T03:58:12.809 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.801+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=108146216}) 2024-09-16T03:58:12.809 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.801+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.809 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.801+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2651091816}) 2024-09-16T03:58:12.809 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.801+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3535996251}) 2024-09-16T03:58:12.811 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.805+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.811 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.805+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2159164695}) 2024-09-16T03:58:12.811 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.805+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3937267264}) 2024-09-16T03:58:12.811 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.805+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.811 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.805+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2159164695}) 2024-09-16T03:58:12.811 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.805+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3385731816}) 2024-09-16T03:58:12.813 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T03:58:12.813 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.805+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.813 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.805+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=81250338}) 2024-09-16T03:58:12.813 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.805+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3616056818}) 2024-09-16T03:58:12.813 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.805+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.813 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.805+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=81250338}) 2024-09-16T03:58:12.814 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.805+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2455186855}) 2024-09-16T03:58:12.816 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.809+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.816 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.809+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3092231176}) 2024-09-16T03:58:12.816 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.809+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1012239160}) 2024-09-16T03:58:12.816 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.809+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.816 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.809+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3092231176}) 2024-09-16T03:58:12.816 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.809+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2332362189}) 2024-09-16T03:58:12.817 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.809+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.817 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.809+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=14005917}) 2024-09-16T03:58:12.817 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.809+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=17862715}) 2024-09-16T03:58:12.817 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.809+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.817 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.809+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=14005917}) 2024-09-16T03:58:12.817 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.809+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3156621770}) 2024-09-16T03:58:12.819 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.813+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.819 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.813+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=89903808}) 2024-09-16T03:58:12.819 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.813+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3020187544}) 2024-09-16T03:58:12.819 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.813+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.819 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.813+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=89903808}) 2024-09-16T03:58:12.819 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.813+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1907003179}) 2024-09-16T03:58:12.821 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.813+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.821 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.813+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1236349673}) 2024-09-16T03:58:12.821 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.813+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3319356571}) 2024-09-16T03:58:12.821 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.813+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.821 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.813+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1236349673}) 2024-09-16T03:58:12.821 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.813+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4061611153}) 2024-09-16T03:58:12.823 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.817+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.823 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.817+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4256644703}) 2024-09-16T03:58:12.823 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.817+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1092920651}) 2024-09-16T03:58:12.823 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.817+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.823 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.817+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4256644703}) 2024-09-16T03:58:12.823 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.817+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=660947061}) 2024-09-16T03:58:12.825 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.817+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.825 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.817+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2469902484}) 2024-09-16T03:58:12.825 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.817+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1957361638}) 2024-09-16T03:58:12.825 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.817+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.825 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.817+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2469902484}) 2024-09-16T03:58:12.825 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.817+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3226734806}) 2024-09-16T03:58:12.826 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.821+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.827 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.821+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=766506080}) 2024-09-16T03:58:12.827 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.821+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2791275675}) 2024-09-16T03:58:12.827 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.821+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.827 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.821+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=766506080}) 2024-09-16T03:58:12.827 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.821+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2532573870}) 2024-09-16T03:58:12.828 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.821+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.828 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.821+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1155478721}) 2024-09-16T03:58:12.829 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.821+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2007709923}) 2024-09-16T03:58:12.829 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.821+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.829 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.821+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1155478721}) 2024-09-16T03:58:12.829 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.821+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1846590465}) 2024-09-16T03:58:12.831 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.825+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.831 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.825+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2019152580}) 2024-09-16T03:58:12.831 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.825+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=789662434}) 2024-09-16T03:58:12.831 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.825+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.831 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.825+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2019152580}) 2024-09-16T03:58:12.831 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.825+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1719207105}) 2024-09-16T03:58:12.832 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.825+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.833 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.825+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=443549061}) 2024-09-16T03:58:12.833 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.825+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1015742542}) 2024-09-16T03:58:12.833 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.825+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.833 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.825+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=443549061}) 2024-09-16T03:58:12.833 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.825+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=402433513}) 2024-09-16T03:58:12.835 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.829+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.835 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.829+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=188440303}) 2024-09-16T03:58:12.835 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.829+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4015356345}) 2024-09-16T03:58:12.835 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.829+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.835 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.829+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=188440303}) 2024-09-16T03:58:12.835 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.829+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=447055193}) 2024-09-16T03:58:12.837 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.829+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.837 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.829+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3224366667}) 2024-09-16T03:58:12.837 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.829+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2557592564}) 2024-09-16T03:58:12.837 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.829+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.837 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.829+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3224366667}) 2024-09-16T03:58:12.837 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.829+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2456801681}) 2024-09-16T03:58:12.839 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.833+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.839 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.833+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1414939815}) 2024-09-16T03:58:12.839 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.833+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2374339765}) 2024-09-16T03:58:12.839 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.833+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.839 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.833+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1414939815}) 2024-09-16T03:58:12.839 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.833+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3650832923}) 2024-09-16T03:58:12.841 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.833+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.841 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.833+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=458153076}) 2024-09-16T03:58:12.842 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.833+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=536160491}) 2024-09-16T03:58:12.842 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.833+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.842 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.833+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=458153076}) 2024-09-16T03:58:12.842 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.833+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3267542892}) 2024-09-16T03:58:12.843 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.837+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.843 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.837+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2591673152}) 2024-09-16T03:58:12.843 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.837+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1824773667}) 2024-09-16T03:58:12.844 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.837+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.844 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.837+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2591673152}) 2024-09-16T03:58:12.844 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.837+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2238381013}) 2024-09-16T03:58:12.845 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.837+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.845 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.837+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3652521331}) 2024-09-16T03:58:12.845 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.837+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=321690064}) 2024-09-16T03:58:12.845 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.837+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.845 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.837+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3652521331}) 2024-09-16T03:58:12.845 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.837+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=470612603}) 2024-09-16T03:58:12.847 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.841+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.847 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.841+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1810917409}) 2024-09-16T03:58:12.847 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.841+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1775160423}) 2024-09-16T03:58:12.847 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.841+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.847 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.841+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1810917409}) 2024-09-16T03:58:12.848 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.841+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3510004202}) 2024-09-16T03:58:12.849 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.841+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.849 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.841+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1042435009}) 2024-09-16T03:58:12.850 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.845+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3034270423}) 2024-09-16T03:58:12.850 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.845+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.850 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.845+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1042435009}) 2024-09-16T03:58:12.850 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.845+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3140713182}) 2024-09-16T03:58:12.851 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.845+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.852 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.845+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1751702424}) 2024-09-16T03:58:12.852 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.845+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4130106312}) 2024-09-16T03:58:12.852 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.845+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.852 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.845+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1751702424}) 2024-09-16T03:58:12.852 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.845+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=37219469}) 2024-09-16T03:58:12.854 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.849+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.854 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.849+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2443883525}) 2024-09-16T03:58:12.854 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.849+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1874256793}) 2024-09-16T03:58:12.854 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.849+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.854 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.849+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2443883525}) 2024-09-16T03:58:12.854 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.849+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1986397960}) 2024-09-16T03:58:12.855 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.849+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.856 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.849+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3484539520}) 2024-09-16T03:58:12.856 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.849+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3194612581}) 2024-09-16T03:58:12.856 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.849+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.856 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.849+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3484539520}) 2024-09-16T03:58:12.856 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.849+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=32340584}) 2024-09-16T03:58:12.857 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.853+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.857 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.853+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3495843545}) 2024-09-16T03:58:12.858 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.853+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=322630611}) 2024-09-16T03:58:12.858 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.853+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.858 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.853+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3495843545}) 2024-09-16T03:58:12.858 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.853+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4161665663}) 2024-09-16T03:58:12.859 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.853+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.859 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.853+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=594213281}) 2024-09-16T03:58:12.860 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.853+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2817619151}) 2024-09-16T03:58:12.860 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.853+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.860 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.853+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=594213281}) 2024-09-16T03:58:12.860 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.853+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1157998122}) 2024-09-16T03:58:12.862 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.857+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.862 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.857+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=138111544}) 2024-09-16T03:58:12.862 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.857+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3678554608}) 2024-09-16T03:58:12.862 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.857+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.862 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.857+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=138111544}) 2024-09-16T03:58:12.862 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.857+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2384721141}) 2024-09-16T03:58:12.863 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.857+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.863 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.857+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=612986601}) 2024-09-16T03:58:12.864 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.857+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1321839029}) 2024-09-16T03:58:12.864 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.857+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.864 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.857+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=612986601}) 2024-09-16T03:58:12.864 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.857+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=629108899}) 2024-09-16T03:58:12.865 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.861+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.865 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.861+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1207077311}) 2024-09-16T03:58:12.866 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.861+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=767737029}) 2024-09-16T03:58:12.866 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.861+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.866 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.861+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1207077311}) 2024-09-16T03:58:12.866 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.861+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2693079624}) 2024-09-16T03:58:12.871 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.865+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.871 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.865+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3148132414}) 2024-09-16T03:58:12.872 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.865+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=606751814}) 2024-09-16T03:58:12.872 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.865+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.872 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.865+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3148132414}) 2024-09-16T03:58:12.872 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.865+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=352747864}) 2024-09-16T03:58:12.873 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.869+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.874 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.869+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1765407313}) 2024-09-16T03:58:12.874 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.869+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3374159479}) 2024-09-16T03:58:12.874 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.869+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.874 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.869+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1765407313}) 2024-09-16T03:58:12.874 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.869+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1003114724}) 2024-09-16T03:58:12.876 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.869+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.876 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.869+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=30866253}) 2024-09-16T03:58:12.876 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.869+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=790316244}) 2024-09-16T03:58:12.876 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.869+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.876 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.869+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=30866253}) 2024-09-16T03:58:12.876 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.869+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3217629758}) 2024-09-16T03:58:12.878 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.873+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.878 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.873+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2432986184}) 2024-09-16T03:58:12.878 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.873+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=130842017}) 2024-09-16T03:58:12.878 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.873+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.878 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.873+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2432986184}) 2024-09-16T03:58:12.878 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.873+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=699519773}) 2024-09-16T03:58:12.880 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.873+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.880 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.873+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2090538793}) 2024-09-16T03:58:12.880 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.873+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2448736262}) 2024-09-16T03:58:12.880 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.873+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.880 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.873+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2090538793}) 2024-09-16T03:58:12.881 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.873+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=939984236}) 2024-09-16T03:58:12.883 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.877+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.883 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.877+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2626206262}) 2024-09-16T03:58:12.883 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.877+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3081029809}) 2024-09-16T03:58:12.883 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.877+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.883 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.877+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2626206262}) 2024-09-16T03:58:12.883 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.877+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=107078734}) 2024-09-16T03:58:12.885 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.877+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.885 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.877+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2415443676}) 2024-09-16T03:58:12.885 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.877+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1090141063}) 2024-09-16T03:58:12.885 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.877+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.885 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.881+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2415443676}) 2024-09-16T03:58:12.886 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.881+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1739200812}) 2024-09-16T03:58:12.887 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.881+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.888 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.881+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=316018417}) 2024-09-16T03:58:12.888 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.881+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=930658916}) 2024-09-16T03:58:12.888 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.881+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.888 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.881+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=316018417}) 2024-09-16T03:58:12.888 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.881+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4185205562}) 2024-09-16T03:58:12.890 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.885+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.890 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.885+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3586999880}) 2024-09-16T03:58:12.890 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.885+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=636524034}) 2024-09-16T03:58:12.890 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.885+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.890 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.885+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3586999880}) 2024-09-16T03:58:12.890 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.885+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1339523103}) 2024-09-16T03:58:12.892 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.885+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.892 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.885+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3196215447}) 2024-09-16T03:58:12.892 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.885+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=778121476}) 2024-09-16T03:58:12.892 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.885+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.892 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.885+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3196215447}) 2024-09-16T03:58:12.892 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.885+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3265151665}) 2024-09-16T03:58:12.894 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.889+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.894 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.889+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=544824031}) 2024-09-16T03:58:12.894 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.889+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=137784798}) 2024-09-16T03:58:12.894 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.889+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.895 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.889+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=544824031}) 2024-09-16T03:58:12.895 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.889+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=118680776}) 2024-09-16T03:58:12.896 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.889+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.896 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.889+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3497771771}) 2024-09-16T03:58:12.897 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.889+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1274275701}) 2024-09-16T03:58:12.897 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.889+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.897 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.889+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3497771771}) 2024-09-16T03:58:12.897 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.889+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1905035096}) 2024-09-16T03:58:12.898 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.893+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.899 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.893+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4111428164}) 2024-09-16T03:58:12.899 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.893+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2001671484}) 2024-09-16T03:58:12.899 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.893+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.899 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.893+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4111428164}) 2024-09-16T03:58:12.899 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.893+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=310485006}) 2024-09-16T03:58:12.901 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.893+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.901 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.893+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3663988333}) 2024-09-16T03:58:12.901 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.893+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3205259021}) 2024-09-16T03:58:12.901 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.893+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.901 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.893+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3663988333}) 2024-09-16T03:58:12.901 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.893+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4089542710}) 2024-09-16T03:58:12.903 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.897+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.903 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.897+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=378527294}) 2024-09-16T03:58:12.903 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.897+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3466719817}) 2024-09-16T03:58:12.903 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.897+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.903 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.897+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=378527294}) 2024-09-16T03:58:12.903 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.897+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2603230309}) 2024-09-16T03:58:12.904 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.897+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.905 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.897+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3016531539}) 2024-09-16T03:58:12.905 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.897+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1366717789}) 2024-09-16T03:58:12.905 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.897+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.905 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.897+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3016531539}) 2024-09-16T03:58:12.905 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.897+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3236004191}) 2024-09-16T03:58:12.907 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.901+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.907 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.901+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3701132011}) 2024-09-16T03:58:12.907 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.901+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3180097241}) 2024-09-16T03:58:12.907 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.901+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.907 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.901+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3701132011}) 2024-09-16T03:58:12.907 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.901+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=391412073}) 2024-09-16T03:58:12.908 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.901+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.909 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.901+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3373483291}) 2024-09-16T03:58:12.909 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.901+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1421612472}) 2024-09-16T03:58:12.909 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.901+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.909 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.901+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3373483291}) 2024-09-16T03:58:12.909 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.901+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1064447744}) 2024-09-16T03:58:12.910 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.905+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.911 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.905+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1068219097}) 2024-09-16T03:58:12.911 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.905+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2034062317}) 2024-09-16T03:58:12.911 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.905+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.911 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.905+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1068219097}) 2024-09-16T03:58:12.911 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.905+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2871661177}) 2024-09-16T03:58:12.913 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.905+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.913 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.905+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=654699433}) 2024-09-16T03:58:12.913 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.905+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2005551967}) 2024-09-16T03:58:12.913 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.905+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.913 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.905+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=654699433}) 2024-09-16T03:58:12.913 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.905+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2611528482}) 2024-09-16T03:58:12.915 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.909+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.915 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.909+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3189821054}) 2024-09-16T03:58:12.915 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.909+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2901122808}) 2024-09-16T03:58:12.915 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.909+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.915 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.909+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3189821054}) 2024-09-16T03:58:12.915 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.909+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2644739593}) 2024-09-16T03:58:12.917 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.909+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.917 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.909+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1639822067}) 2024-09-16T03:58:12.917 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.909+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=114409019}) 2024-09-16T03:58:12.917 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.909+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.917 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.909+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1639822067}) 2024-09-16T03:58:12.917 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.909+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2353533105}) 2024-09-16T03:58:12.918 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.913+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.918 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.913+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2492093937}) 2024-09-16T03:58:12.919 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.913+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2208730032}) 2024-09-16T03:58:12.919 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.913+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.919 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.913+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2492093937}) 2024-09-16T03:58:12.919 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.913+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3933322481}) 2024-09-16T03:58:12.920 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.913+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.921 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.913+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2006094019}) 2024-09-16T03:58:12.921 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.913+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2803007923}) 2024-09-16T03:58:12.921 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.913+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.921 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.913+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2006094019}) 2024-09-16T03:58:12.921 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.913+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3102189672}) 2024-09-16T03:58:12.922 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.917+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.922 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.917+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3014622200}) 2024-09-16T03:58:12.922 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.917+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1516158972}) 2024-09-16T03:58:12.922 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.917+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.922 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.917+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3014622200}) 2024-09-16T03:58:12.922 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.917+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1349219360}) 2024-09-16T03:58:12.924 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.917+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.924 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.917+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2727906226}) 2024-09-16T03:58:12.924 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.917+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2540457014}) 2024-09-16T03:58:12.924 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.917+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.924 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.917+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2727906226}) 2024-09-16T03:58:12.924 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.917+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2407252479}) 2024-09-16T03:58:12.926 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.921+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.926 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.921+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1432286904}) 2024-09-16T03:58:12.926 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.921+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=226169306}) 2024-09-16T03:58:12.926 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.921+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.926 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.921+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1432286904}) 2024-09-16T03:58:12.926 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.921+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3990553779}) 2024-09-16T03:58:12.928 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.921+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.928 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.921+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2455096509}) 2024-09-16T03:58:12.928 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.921+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=514148715}) 2024-09-16T03:58:12.928 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.921+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.928 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.921+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2455096509}) 2024-09-16T03:58:12.928 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.921+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2220368747}) 2024-09-16T03:58:12.930 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.925+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.930 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.925+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1507080511}) 2024-09-16T03:58:12.930 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.925+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3153798003}) 2024-09-16T03:58:12.930 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.925+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.930 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.925+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1507080511}) 2024-09-16T03:58:12.930 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.925+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1756356425}) 2024-09-16T03:58:12.932 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.925+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.932 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.925+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3033651060}) 2024-09-16T03:58:12.932 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.925+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2582418034}) 2024-09-16T03:58:12.932 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.925+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.932 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.925+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3033651060}) 2024-09-16T03:58:12.932 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.925+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2683136708}) 2024-09-16T03:58:12.934 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.929+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.934 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.929+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3923323815}) 2024-09-16T03:58:12.934 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.929+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=592575793}) 2024-09-16T03:58:12.934 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.929+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.934 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.929+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3923323815}) 2024-09-16T03:58:12.934 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.929+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=504237124}) 2024-09-16T03:58:12.936 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.929+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.936 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.929+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=589059507}) 2024-09-16T03:58:12.936 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.929+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2833299365}) 2024-09-16T03:58:12.936 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.929+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.936 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.929+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=589059507}) 2024-09-16T03:58:12.936 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.929+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2322693564}) 2024-09-16T03:58:12.937 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.933+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.938 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.933+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2848119793}) 2024-09-16T03:58:12.938 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.933+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3508549311}) 2024-09-16T03:58:12.938 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.933+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.938 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.933+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2848119793}) 2024-09-16T03:58:12.938 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.933+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2923951046}) 2024-09-16T03:58:12.939 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.933+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.939 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.933+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2206504258}) 2024-09-16T03:58:12.939 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.933+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2064914901}) 2024-09-16T03:58:12.939 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.933+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.940 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.933+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2206504258}) 2024-09-16T03:58:12.940 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.933+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=756405693}) 2024-09-16T03:58:12.941 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.933+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.941 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.933+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=331972671}) 2024-09-16T03:58:12.941 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.937+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=716010334}) 2024-09-16T03:58:12.942 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.937+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.942 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.937+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=331972671}) 2024-09-16T03:58:12.942 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.937+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=873226492}) 2024-09-16T03:58:12.944 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.937+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.944 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.937+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1983605344}) 2024-09-16T03:58:12.944 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.937+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3883646875}) 2024-09-16T03:58:12.944 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.937+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.944 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.937+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1983605344}) 2024-09-16T03:58:12.944 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.937+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=491306017}) 2024-09-16T03:58:12.946 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.941+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.946 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.941+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=913859212}) 2024-09-16T03:58:12.946 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.941+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1623818393}) 2024-09-16T03:58:12.946 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.941+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.946 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.941+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=913859212}) 2024-09-16T03:58:12.946 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.941+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=309637461}) 2024-09-16T03:58:12.948 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.941+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.948 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.941+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1279525902}) 2024-09-16T03:58:12.948 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.941+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3733174327}) 2024-09-16T03:58:12.948 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.941+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.948 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.941+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1279525902}) 2024-09-16T03:58:12.948 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.941+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=891129106}) 2024-09-16T03:58:12.950 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.945+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.950 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.945+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1563542808}) 2024-09-16T03:58:12.950 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.945+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=844574388}) 2024-09-16T03:58:12.950 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.945+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.950 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.945+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1563542808}) 2024-09-16T03:58:12.951 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.945+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3865775481}) 2024-09-16T03:58:12.952 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.945+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.952 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.945+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2042600064}) 2024-09-16T03:58:12.952 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.945+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1262821010}) 2024-09-16T03:58:12.952 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.945+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.952 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.945+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2042600064}) 2024-09-16T03:58:12.953 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.945+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=589333004}) 2024-09-16T03:58:12.955 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.949+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.955 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.949+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3032102509}) 2024-09-16T03:58:12.955 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.949+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3497832413}) 2024-09-16T03:58:12.955 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.949+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.955 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.949+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3032102509}) 2024-09-16T03:58:12.955 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.949+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3672992019}) 2024-09-16T03:58:12.957 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.949+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.957 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.949+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3307310067}) 2024-09-16T03:58:12.957 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.949+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1567288755}) 2024-09-16T03:58:12.957 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.949+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.957 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.949+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3307310067}) 2024-09-16T03:58:12.957 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.949+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3062933094}) 2024-09-16T03:58:12.959 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.953+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.959 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.953+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2973136982}) 2024-09-16T03:58:12.959 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.953+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2691303643}) 2024-09-16T03:58:12.959 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.953+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.959 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.953+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2973136982}) 2024-09-16T03:58:12.959 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.953+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1296823571}) 2024-09-16T03:58:12.961 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.957+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.961 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.957+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4163970040}) 2024-09-16T03:58:12.962 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.957+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2329383107}) 2024-09-16T03:58:12.962 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.957+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.962 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.957+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4163970040}) 2024-09-16T03:58:12.962 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.957+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1668484726}) 2024-09-16T03:58:12.964 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.957+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.964 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.957+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=123125799}) 2024-09-16T03:58:12.964 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.957+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3418314108}) 2024-09-16T03:58:12.964 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.957+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.964 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.957+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=123125799}) 2024-09-16T03:58:12.964 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.957+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3575722990}) 2024-09-16T03:58:12.966 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.961+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.966 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.961+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3473849222}) 2024-09-16T03:58:12.966 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.961+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3240794111}) 2024-09-16T03:58:12.966 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.961+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.967 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.961+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3473849222}) 2024-09-16T03:58:12.967 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.961+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4172879130}) 2024-09-16T03:58:12.968 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.961+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.968 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.961+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1232948785}) 2024-09-16T03:58:12.968 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.961+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=792205642}) 2024-09-16T03:58:12.968 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.961+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.969 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.961+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1232948785}) 2024-09-16T03:58:12.969 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.961+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=822277940}) 2024-09-16T03:58:12.970 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.965+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.970 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.965+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3918850871}) 2024-09-16T03:58:12.970 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.965+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3036362879}) 2024-09-16T03:58:12.971 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.965+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.971 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.965+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3918850871}) 2024-09-16T03:58:12.971 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.965+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=178656698}) 2024-09-16T03:58:12.972 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.965+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.973 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.965+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=17553509}) 2024-09-16T03:58:12.973 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.965+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1479186839}) 2024-09-16T03:58:12.973 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.965+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.973 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.965+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=17553509}) 2024-09-16T03:58:12.973 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.965+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=569221618}) 2024-09-16T03:58:12.975 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.969+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.975 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.969+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3317337180}) 2024-09-16T03:58:12.975 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.969+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=732015685}) 2024-09-16T03:58:12.975 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.969+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.975 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.969+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3317337180}) 2024-09-16T03:58:12.975 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.969+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3779631958}) 2024-09-16T03:58:12.977 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.969+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.977 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.969+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1269825582}) 2024-09-16T03:58:12.977 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.969+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2609329573}) 2024-09-16T03:58:12.977 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.969+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.977 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.969+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1269825582}) 2024-09-16T03:58:12.977 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.969+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1698556470}) 2024-09-16T03:58:12.979 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.973+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.979 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.973+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1216581676}) 2024-09-16T03:58:12.979 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.973+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2336316352}) 2024-09-16T03:58:12.979 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.973+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.979 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.973+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1216581676}) 2024-09-16T03:58:12.979 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.973+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1952334943}) 2024-09-16T03:58:12.981 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.973+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.981 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.973+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2571641669}) 2024-09-16T03:58:12.981 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.973+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2383637480}) 2024-09-16T03:58:12.981 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.973+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.981 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.973+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2571641669}) 2024-09-16T03:58:12.982 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.977+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1205773742}) 2024-09-16T03:58:12.984 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.977+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.984 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.977+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1542156643}) 2024-09-16T03:58:12.984 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.977+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2565215858}) 2024-09-16T03:58:12.984 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.977+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.984 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.977+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1542156643}) 2024-09-16T03:58:12.984 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.977+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3679278901}) 2024-09-16T03:58:12.986 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.981+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.987 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.981+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1926911959}) 2024-09-16T03:58:12.987 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.981+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=805137324}) 2024-09-16T03:58:12.987 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.981+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.987 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.981+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1926911959}) 2024-09-16T03:58:12.987 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.981+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1907233745}) 2024-09-16T03:58:12.989 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.981+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.989 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.981+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3673768603}) 2024-09-16T03:58:12.989 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.981+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=471786822}) 2024-09-16T03:58:12.989 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.981+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.989 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.981+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3673768603}) 2024-09-16T03:58:12.989 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.981+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3683876502}) 2024-09-16T03:58:12.991 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.985+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.991 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.985+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2348551967}) 2024-09-16T03:58:12.991 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.985+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2645620746}) 2024-09-16T03:58:12.991 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.985+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.991 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.985+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2348551967}) 2024-09-16T03:58:12.991 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.985+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3168463450}) 2024-09-16T03:58:12.993 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.985+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.993 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.985+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1115592584}) 2024-09-16T03:58:12.993 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.985+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1526822595}) 2024-09-16T03:58:12.993 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.985+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.993 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.985+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1115592584}) 2024-09-16T03:58:12.993 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.985+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=33061970}) 2024-09-16T03:58:12.995 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.989+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.996 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.989+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1981896931}) 2024-09-16T03:58:12.996 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.989+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1553162656}) 2024-09-16T03:58:12.996 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.989+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.996 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.989+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1981896931}) 2024-09-16T03:58:12.996 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.989+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1618404641}) 2024-09-16T03:58:12.998 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.993+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.998 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.993+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3338565452}) 2024-09-16T03:58:12.998 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.993+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1192717762}) 2024-09-16T03:58:12.998 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.993+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:12.998 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.993+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3338565452}) 2024-09-16T03:58:12.998 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.993+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=512997233}) 2024-09-16T03:58:13.001 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.993+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:13.001 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.993+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=11,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=15,mon_config_key=10,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=34,osdmap=8} crc {logm=2441205416,mdsmap=4088524775,mgr=390803354,mgr_command_descs=2107673600,mgr_metadata=1367272202,mgrstat=294462938,mon_config_key=3969788172,monmap=3937513981,osd_metadata=4182959540,osd_pg_creating=3128641836,osd_snap=4248563283,osdmap=4070841208}) 2024-09-16T03:58:13.001 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.993+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=11,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=15,mon_config_key=10,monmap=2,osd_metadata=6,osd_pg_creating=1,osd_snap=35,osdmap=7} crc {logm=2441205416,mdsmap=4088524775,mgr=460109776,mgr_command_descs=2107673600,mgr_metadata=1367272202,mgrstat=294462938,mon_config_key=1599078736,monmap=268092033,osd_metadata=4182959540,osd_pg_creating=3128641836,osd_snap=1977785798,osdmap=36603834}) 2024-09-16T03:58:13.001 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.993+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:13.001 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.993+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=11,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=15,mon_config_key=10,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=34,osdmap=8} crc {logm=2441205416,mdsmap=4088524775,mgr=390803354,mgr_command_descs=2107673600,mgr_metadata=1367272202,mgrstat=294462938,mon_config_key=3969788172,monmap=3937513981,osd_metadata=4182959540,osd_pg_creating=3128641836,osd_snap=4248563283,osdmap=4070841208}) 2024-09-16T03:58:13.001 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.993+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=11,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=14,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=35,osdmap=6} crc {logm=2441205416,mdsmap=4088524775,mgr=3558091671,mgr_command_descs=2107673600,mgr_metadata=1367272202,mgrstat=4129583780,mon_config_key=85057112,monmap=3937513981,osd_metadata=4182959540,osd_pg_creating=3128641836,osd_snap=2875164478,osdmap=35335747}) 2024-09-16T03:58:13.003 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.997+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:13.003 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.997+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1000113238}) 2024-09-16T03:58:13.003 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.997+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3144069520}) 2024-09-16T03:58:13.003 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.997+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:13.003 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.997+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1000113238}) 2024-09-16T03:58:13.004 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:12.997+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1507978100}) 2024-09-16T03:58:13.018 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:13.013+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:13.018 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:13.013+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=78} crc {osdmap=72434029}) 2024-09-16T03:58:13.019 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:13.013+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=81} crc {osdmap=493559605}) 2024-09-16T03:58:13.019 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:13.013+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:13.019 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:13.013+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=78} crc {osdmap=72434029}) 2024-09-16T03:58:13.019 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:13.013+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=78} crc {osdmap=204703552}) 2024-09-16T03:58:13.813 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T03:58:13.813 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:58:13.943 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.937+0000 7f4aa5fe0640 10 monclient: get_monmap_and_config 2024-09-16T03:58:13.943 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.937+0000 7f4aa5fe0640 10 monclient: build_initial_monmap 2024-09-16T03:58:13.943 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.937+0000 7f4aa5fe0640 10 monclient: monmap: 2024-09-16T03:58:13.943 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:58:13.944 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:58:13.944 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:58:13.941884+0000 2024-09-16T03:58:13.944 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:58:13.941884+0000 2024-09-16T03:58:13.944 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:58:13.944 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:13.944 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:58:13.944 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:58:13.944 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:58:13.944 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:58:13.944 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:58:13.944 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:13.947 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.937+0000 7f4aa5fe0640 1 Processor -- start 2024-09-16T03:58:13.947 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.937+0000 7f4aa5fe0640 1 -- start start 2024-09-16T03:58:13.947 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.937+0000 7f4aa5fe0640 10 monclient: init 2024-09-16T03:58:13.947 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.937+0000 7f4aa5fe0640 10 monclient: _reopen_session rank -1 2024-09-16T03:58:13.947 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.937+0000 7f4aa5fe0640 10 monclient: _add_conns ranks=[4,1,2,0,3] 2024-09-16T03:58:13.947 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.937+0000 7f4aa5fe0640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f4aa00a1330 0x7f4aa009f180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:13.947 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.937+0000 7f4aa5fe0640 10 monclient(hunting): picked mon.noname-c con 0x7f4aa00a1330 addr v2:172.21.15.154:3300/0 2024-09-16T03:58:13.947 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.937+0000 7f4aa5fe0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4aa00a1710 addr v1:172.21.15.154:6789/0 2024-09-16T03:58:13.948 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.937+0000 7f4aa5fe0640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4aa01393a0 0x7f4aa013b790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:13.948 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.937+0000 7f4aa5fe0640 10 monclient(hunting): picked mon.noname-b con 0x7f4aa01393a0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:58:13.948 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.937+0000 7f4aa5fe0640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4aa009e4c0 con 0x7f4aa00a1710 2024-09-16T03:58:13.948 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.937+0000 7f4aa5fe0640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:13.948 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.937+0000 7f4aa5fe0640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f4aa009edf0 con 0x7f4aa00a1330 2024-09-16T03:58:13.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.937+0000 7f4aa5fe0640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:13.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.937+0000 7f4aa5fe0640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f4aa009cd60 con 0x7f4aa01393a0 2024-09-16T03:58:13.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.937+0000 7f4aa5fe0640 10 monclient(hunting): _renew_subs 2024-09-16T03:58:13.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.937+0000 7f4aa5fe0640 10 monclient(hunting): authenticate will time out at 1940.911323s 2024-09-16T03:58:13.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.937+0000 7f4a9ffff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4aa01393a0 0x7f4aa013b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:13.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.937+0000 7f4a9ffff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4aa01393a0 0x7f4aa013b790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:49092/0 (socket says 172.21.15.42:49092) 2024-09-16T03:58:13.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.937+0000 7f4a9ffff640 1 -- 172.21.15.42:0/1991567096 learned_addr learned my addr 172.21.15.42:0/1991567096 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:58:13.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.937+0000 7f4a9ffff640 10 monclient(hunting): get_auth_request con 0x7f4aa01393a0 auth_method 0 2024-09-16T03:58:13.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.937+0000 7f4a9ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:13.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.937+0000 7f4a9ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:13.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.937+0000 7f4a9ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:58:13.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.937+0000 7f4a9ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:58:13.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.937+0000 7f4a9ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:58:13.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.937+0000 7f4a9ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:58:13.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.937+0000 7f4a9f7fe640 1 --2- 172.21.15.42:0/1991567096 >> v2:172.21.15.154:3300/0 conn(0x7f4aa00a1330 0x7f4aa009f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:13.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.937+0000 7f4a9f7fe640 10 monclient(hunting): get_auth_request con 0x7f4aa00a1330 auth_method 0 2024-09-16T03:58:13.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.937+0000 7f4a9f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:13.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.937+0000 7f4a9f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:13.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.937+0000 7f4a9f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:58:13.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.937+0000 7f4a9ffff640 10 monclient(hunting): handle_auth_done global_id 854462 payload 274 2024-09-16T03:58:13.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.937+0000 7f4a9ffff640 1 -- 172.21.15.42:0/1991567096 >> v2:172.21.15.154:3300/0 conn(0x7f4aa00a1330 msgr2=0x7f4aa009f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:58:13.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.937+0000 7f4a9ffff640 1 --2- 172.21.15.42:0/1991567096 >> v2:172.21.15.154:3300/0 conn(0x7f4aa00a1330 0x7f4aa009f180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:13.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.937+0000 7f4a9ffff640 1 -- 172.21.15.42:0/1991567096 >> v1:172.21.15.154:6789/0 conn(0x7f4aa00a1710 legacy=0x7f4aa00a1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:13.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.937+0000 7f4a9ffff640 10 monclient: _finish_hunting 0 2024-09-16T03:58:13.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.937+0000 7f4a9ffff640 1 monclient: found mon.noname-b 2024-09-16T03:58:13.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.937+0000 7f4a9ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:58:13.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.937+0000 7f4a9ffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T03:58:13.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.937+0000 7f4a9ffff640 1 -- 172.21.15.42:0/1991567096 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4aa009f6c0 con 0x7f4aa01393a0 2024-09-16T03:58:13.956 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.941+0000 7f4a9ffff640 1 --2- 172.21.15.42:0/1991567096 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4aa01393a0 0x7f4aa013b790 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f4a98009e50 tx=0x7f4a9801c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6d7a9614603d97f8 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:58:13.956 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.941+0000 7f4a9e7fc640 1 -- 172.21.15.42:0/1991567096 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a980a56c0 con 0x7f4aa01393a0 2024-09-16T03:58:13.956 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.941+0000 7f4a9e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:58:13.957 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.941+0000 7f4a9e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:58:13.957 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.941+0000 7f4a9e7fc640 10 monclient: dump: 2024-09-16T03:58:13.957 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:58:13.957 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:58:13.957 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:13.957 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:13.957 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:58:13.957 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:13.957 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:58:13.957 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:58:13.957 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:58:13.957 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:13.958 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.941+0000 7f4a9e7fc640 10 monclient: _finish_auth 0 2024-09-16T03:58:13.958 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.941+0000 7f4a9e7fc640 10 monclient: _check_auth_tickets 2024-09-16T03:58:13.958 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.941+0000 7f4a9e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:58:13.958 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.941+0000 7f4a9e7fc640 1 -- 172.21.15.42:0/1991567096 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a980a5860 con 0x7f4aa01393a0 2024-09-16T03:58:13.958 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.941+0000 7f4aa5fe0640 5 monclient: authenticate success, global_id 854462 2024-09-16T03:58:13.958 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.941+0000 7f4aa5fe0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:58:13.958 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.941+0000 7f4a9e7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:58:13.958 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.941+0000 7f4aa5fe0640 10 monclient: get_monmap_and_config success 2024-09-16T03:58:13.958 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.941+0000 7f4a9e7fc640 1 -- 172.21.15.42:0/1991567096 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a980a5c30 con 0x7f4aa01393a0 2024-09-16T03:58:13.958 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.941+0000 7f4aa5fe0640 4 set_mon_vals no callback set 2024-09-16T03:58:13.958 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.941+0000 7f4a9e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:58:13.958 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.941+0000 7f4a9e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:58:13.959 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.941+0000 7f4a9e7fc640 10 monclient: dump: 2024-09-16T03:58:13.959 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:58:13.959 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:58:13.959 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:13.959 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:13.959 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:58:13.959 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:13.959 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:58:13.959 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:58:13.959 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:58:13.959 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:13.959 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.941+0000 7f4aa5fe0640 10 monclient: shutdown 2024-09-16T03:58:13.959 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.941+0000 7f4aa5fe0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:58:13.960 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.941+0000 7f4aa5fe0640 1 -- 172.21.15.42:0/1991567096 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4aa01393a0 msgr2=0x7f4aa013b790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:13.960 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.941+0000 7f4aa5fe0640 1 --2- 172.21.15.42:0/1991567096 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4aa01393a0 0x7f4aa013b790 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f4a98009e50 tx=0x7f4a9801c8f0 comp rx=0 tx=0).stop 2024-09-16T03:58:13.960 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.941+0000 7f4aa5fe0640 1 -- 172.21.15.42:0/1991567096 shutdown_connections 2024-09-16T03:58:13.960 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.941+0000 7f4aa5fe0640 1 --2- 172.21.15.42:0/1991567096 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4aa01393a0 0x7f4aa013b790 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:13.960 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.941+0000 7f4aa5fe0640 1 --2- 172.21.15.42:0/1991567096 >> v2:172.21.15.154:3300/0 conn(0x7f4aa00a1330 0x7f4aa009f180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:13.960 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.941+0000 7f4aa5fe0640 1 -- 172.21.15.42:0/1991567096 >> 172.21.15.42:0/1991567096 conn(0x7f4aa00a2420 msgr2=0x7f4aa00a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:58:13.960 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.941+0000 7f4aa5fe0640 1 -- 172.21.15.42:0/1991567096 shutdown_connections 2024-09-16T03:58:13.960 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.941+0000 7f4aa5fe0640 1 -- 172.21.15.42:0/1991567096 wait complete. 2024-09-16T03:58:13.960 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.945+0000 7f4aa5fe0640 10 monclient: build_initial_monmap 2024-09-16T03:58:13.960 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.945+0000 7f4aa5fe0640 10 monclient: monmap: 2024-09-16T03:58:13.960 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:58:13.960 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:58:13.960 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:58:13.961 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:58:13.961 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:58:13.961 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:13.961 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:58:13.961 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:58:13.961 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:58:13.961 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:13.961 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.945+0000 7f4aa5fe0640 1 librados: starting msgr at 2024-09-16T03:58:13.961 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.945+0000 7f4aa5fe0640 1 librados: starting objecter 2024-09-16T03:58:13.961 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.945+0000 7f4aa5fe0640 1 Processor -- start 2024-09-16T03:58:13.961 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.945+0000 7f4aa5fe0640 1 -- start start 2024-09-16T03:58:13.961 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.945+0000 7f4aa5fe0640 1 librados: setting wanted keys 2024-09-16T03:58:13.961 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.945+0000 7f4aa5fe0640 1 librados: calling monclient init 2024-09-16T03:58:13.962 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.945+0000 7f4aa5fe0640 10 monclient: init 2024-09-16T03:58:13.962 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.945+0000 7f4aa5fe0640 10 monclient: _reopen_session rank -1 2024-09-16T03:58:13.962 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.945+0000 7f4aa5fe0640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:58:13.962 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.945+0000 7f4aa5fe0640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4aa00a1330 0x7f4aa013cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:13.962 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.945+0000 7f4aa5fe0640 10 monclient(hunting): picked mon.noname-a con 0x7f4aa00a1330 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:58:13.962 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.945+0000 7f4aa5fe0640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4aa00a1710 0x7f4aa013d220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:13.962 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.945+0000 7f4aa5fe0640 10 monclient(hunting): picked mon.noname-c con 0x7f4aa00a1710 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:58:13.962 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.945+0000 7f4aa5fe0640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4aa01393a0 0x7f4aa013d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:13.962 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.945+0000 7f4aa5fe0640 10 monclient(hunting): picked mon.noname-b con 0x7f4aa01393a0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:58:13.962 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.945+0000 7f4aa5fe0640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:13.962 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.945+0000 7f4aa5fe0640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f4aa013c8b0 con 0x7f4aa00a1330 2024-09-16T03:58:13.962 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.945+0000 7f4aa5fe0640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:13.963 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.945+0000 7f4aa5fe0640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f4aa013c730 con 0x7f4aa01393a0 2024-09-16T03:58:13.963 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.945+0000 7f4aa5fe0640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:13.963 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.945+0000 7f4aa5fe0640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f4aa013ca30 con 0x7f4aa00a1710 2024-09-16T03:58:13.963 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.945+0000 7f4aa5fe0640 10 monclient(hunting): _renew_subs 2024-09-16T03:58:13.963 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.945+0000 7f4aa5fe0640 10 monclient(hunting): authenticate will time out at 1940.917890s 2024-09-16T03:58:13.963 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.945+0000 7f4a9f7fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4aa00a1330 0x7f4aa013cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:13.963 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.945+0000 7f4a9effd640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4aa00a1710 0x7f4aa013d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:13.963 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.945+0000 7f4a9f7fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4aa00a1330 0x7f4aa013cce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:36044/0 (socket says 172.21.15.42:36044) 2024-09-16T03:58:13.963 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.945+0000 7f4a9f7fe640 1 -- 172.21.15.42:0/1958562724 learned_addr learned my addr 172.21.15.42:0/1958562724 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:58:13.963 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.945+0000 7f4a9f7fe640 10 monclient(hunting): get_auth_request con 0x7f4aa00a1330 auth_method 0 2024-09-16T03:58:13.963 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.945+0000 7f4a9f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:13.964 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.945+0000 7f4a9f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:13.964 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.945+0000 7f4a9f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:58:13.964 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.945+0000 7f4a9effd640 10 monclient(hunting): get_auth_request con 0x7f4aa00a1710 auth_method 0 2024-09-16T03:58:13.964 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.945+0000 7f4a9effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:13.964 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.945+0000 7f4a9effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:13.964 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.945+0000 7f4a9effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:58:13.964 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.945+0000 7f4a9effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:58:13.964 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.945+0000 7f4a9effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:58:13.964 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.945+0000 7f4a9effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:58:13.964 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.945+0000 7f4a9f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:58:13.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.945+0000 7f4a9f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:58:13.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.945+0000 7f4a9ffff640 1 --2- 172.21.15.42:0/1958562724 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4aa01393a0 0x7f4aa013d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:13.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.945+0000 7f4a9f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:58:13.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.945+0000 7f4a9ffff640 10 monclient(hunting): get_auth_request con 0x7f4aa01393a0 auth_method 0 2024-09-16T03:58:13.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.945+0000 7f4a9ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:13.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.945+0000 7f4a9ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:13.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.945+0000 7f4a9ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:58:13.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.945+0000 7f4a9effd640 10 monclient(hunting): handle_auth_done global_id 854465 payload 824 2024-09-16T03:58:13.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.945+0000 7f4a9effd640 1 -- 172.21.15.42:0/1958562724 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4aa01393a0 msgr2=0x7f4aa013d760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:13.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.945+0000 7f4a9effd640 1 --2- 172.21.15.42:0/1958562724 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4aa01393a0 0x7f4aa013d760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:13.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.945+0000 7f4a9effd640 1 -- 172.21.15.42:0/1958562724 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4aa00a1330 msgr2=0x7f4aa013cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:13.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.945+0000 7f4a9effd640 1 --2- 172.21.15.42:0/1958562724 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4aa00a1330 0x7f4aa013cce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:13.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.945+0000 7f4a9effd640 10 monclient: _finish_hunting 0 2024-09-16T03:58:13.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.945+0000 7f4a9effd640 1 monclient: found mon.noname-c 2024-09-16T03:58:13.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.945+0000 7f4a9effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:58:13.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.945+0000 7f4a9effd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:58:13.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.945+0000 7f4a9effd640 1 -- 172.21.15.42:0/1958562724 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4aa00721d0 con 0x7f4aa00a1710 2024-09-16T03:58:13.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.945+0000 7f4a9effd640 1 --2- 172.21.15.42:0/1958562724 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4aa00a1710 0x7f4aa013d220 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f4a90000f70 tx=0x7f4a9000c820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:58:13.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.945+0000 7f4a97fff640 10 client.?.objecter ms_handle_connect 0x7f4aa00a1710 2024-09-16T03:58:13.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.945+0000 7f4a97fff640 10 client.?.objecter resend_mon_ops 2024-09-16T03:58:13.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.945+0000 7f4a97fff640 1 -- 172.21.15.42:0/1958562724 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a90017070 con 0x7f4aa00a1710 2024-09-16T03:58:13.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.945+0000 7f4a97fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:58:13.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.945+0000 7f4a97fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:58:13.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.945+0000 7f4a97fff640 10 monclient: dump: 2024-09-16T03:58:13.966 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:58:13.966 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:58:13.967 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:13.967 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:13.967 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:58:13.967 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:13.967 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:58:13.967 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:58:13.967 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:58:13.967 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:13.967 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.945+0000 7f4a97fff640 10 monclient: _finish_auth 0 2024-09-16T03:58:13.967 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.945+0000 7f4a97fff640 10 monclient: _check_auth_tickets 2024-09-16T03:58:13.967 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.945+0000 7f4a97fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:58:13.967 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.945+0000 7f4a97fff640 1 -- 172.21.15.42:0/1958562724 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a900052d0 con 0x7f4aa00a1710 2024-09-16T03:58:13.967 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.945+0000 7f4aa5fe0640 5 monclient: authenticate success, global_id 854465 2024-09-16T03:58:13.967 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.945+0000 7f4aa5fe0640 10 monclient: _renew_subs 2024-09-16T03:58:13.968 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.945+0000 7f4aa5fe0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:58:13.968 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.945+0000 7f4aa5fe0640 1 -- 172.21.15.42:0/1958562724 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4aa013b320 con 0x7f4aa00a1710 2024-09-16T03:58:13.968 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.945+0000 7f4a97fff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:58:13.968 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.945+0000 7f4a97fff640 1 -- 172.21.15.42:0/1958562724 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a900054c0 con 0x7f4aa00a1710 2024-09-16T03:58:13.968 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.945+0000 7f4a97fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:58:13.968 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.945+0000 7f4a97fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:58:13.968 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.945+0000 7f4a9dffb640 4 set_mon_vals no callback set 2024-09-16T03:58:13.968 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.945+0000 7f4a97fff640 10 monclient: dump: 2024-09-16T03:58:13.968 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:58:13.968 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:58:13.968 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:13.968 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:13.968 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:58:13.969 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:13.969 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:58:13.969 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:58:13.969 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:58:13.969 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:13.969 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.945+0000 7f4aa5fe0640 10 client.854465.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:58:13.969 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.945+0000 7f4aa5fe0640 10 monclient: _renew_subs 2024-09-16T03:58:13.969 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.945+0000 7f4aa5fe0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:58:13.969 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.945+0000 7f4aa5fe0640 1 -- 172.21.15.42:0/1958562724 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f4aa013e910 con 0x7f4aa00a1710 2024-09-16T03:58:13.969 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.945+0000 7f4aa5fe0640 1 librados: init done 2024-09-16T03:58:13.969 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.945+0000 7f4aa5fe0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:58:13.969 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.945+0000 7f4a97fff640 1 -- 172.21.15.42:0/1958562724 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f4a9001a070 con 0x7f4aa00a1710 2024-09-16T03:58:13.969 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.945+0000 7f4aa5fe0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:58:13.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.945+0000 7f4aa5fe0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:58:13.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.945+0000 7f4aa5fe0640 1 -- 172.21.15.42:0/1958562724 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4aa009f180 con 0x7f4aa00a1710 2024-09-16T03:58:13.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.949+0000 7f4a97fff640 1 --2- 172.21.15.42:0/1958562724 >> v2:172.21.15.154:6812/254845342 conn(0x7f4a80037860 0x7f4a80039d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:13.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.949+0000 7f4a97fff640 1 -- 172.21.15.42:0/1958562724 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(484..484 src has 366..484) ==== 6181+0+0 (secure 0 0 0) 0x7f4a900484c0 con 0x7f4aa00a1710 2024-09-16T03:58:13.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.949+0000 7f4a97fff640 10 client.854465.objecter ms_dispatch 0x7f4aa0000ba0 osd_map(484..484 src has 366..484) 2024-09-16T03:58:13.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.949+0000 7f4a97fff640 3 client.854465.objecter handle_osd_map got epochs [484,484] > 0 2024-09-16T03:58:13.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.949+0000 7f4a97fff640 3 client.854465.objecter handle_osd_map decoding full epoch 484 2024-09-16T03:58:13.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.949+0000 7f4a97fff640 20 client.854465.objecter dump_active .. 0 homeless 2024-09-16T03:58:13.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.949+0000 7f4a9f7fe640 1 --2- 172.21.15.42:0/1958562724 >> v2:172.21.15.154:6812/254845342 conn(0x7f4a80037860 0x7f4a80039d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:13.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.949+0000 7f4a9f7fe640 10 monclient: get_auth_request con 0x7f4a80037860 auth_method 0 2024-09-16T03:58:13.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.949+0000 7f4a97fff640 10 client.854465.objecter ms_handle_connect 0x7f4a80037860 2024-09-16T03:58:13.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.949+0000 7f4a9f7fe640 1 --2- 172.21.15.42:0/1958562724 >> v2:172.21.15.154:6812/254845342 conn(0x7f4a80037860 0x7f4a80039d20 secure :-1 s=READY pgs=3693 cs=0 l=1 rev1=1 crypto rx=0x7f4a88007b10 tx=0x7f4a88008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:58:13.971 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.953+0000 7f4a97fff640 1 -- 172.21.15.42:0/1958562724 <== mon.2 v2:172.21.15.42:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f4a90011590 con 0x7f4aa00a1710 2024-09-16T03:58:13.971 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.953+0000 7f4a97fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:58:13.971 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:13.953+0000 7f4a97fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:58:14.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.181+0000 7f4aa5fe0640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:58:14.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.181+0000 7f4aa5fe0640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:58:14.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.181+0000 7f4aa5fe0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:58:14.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.181+0000 7f4aa5fe0640 1 -- 172.21.15.42:0/1958562724 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f4aa0069040 con 0x7f4aa00a1710 2024-09-16T03:58:14.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.181+0000 7f4a97fff640 1 -- 172.21.15.42:0/1958562724 <== mon.2 v2:172.21.15.42:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7f4a90003bd0 con 0x7f4aa00a1710 2024-09-16T03:58:14.187 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.181+0000 7f4a97fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:58:14.187 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.181+0000 7f4a97fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:58:14.187 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":148,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:58:14.190 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.181+0000 7f4aa5fe0640 10 librados: watch_flush enter 2024-09-16T03:58:14.190 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.185+0000 7f4aa5fe0640 10 librados: watch_flush exit 2024-09-16T03:58:14.190 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.185+0000 7f4aa5fe0640 20 client.854465.objecter shutdown clearing up homeless session... 2024-09-16T03:58:14.190 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.185+0000 7f4aa5fe0640 10 client.854465.objecter successfully canceled tick 2024-09-16T03:58:14.190 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.185+0000 7f4aa5fe0640 1 -- 172.21.15.42:0/1958562724 >> v2:172.21.15.154:6812/254845342 conn(0x7f4a80037860 msgr2=0x7f4a80039d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:14.190 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.185+0000 7f4aa5fe0640 1 --2- 172.21.15.42:0/1958562724 >> v2:172.21.15.154:6812/254845342 conn(0x7f4a80037860 0x7f4a80039d20 secure :-1 s=READY pgs=3693 cs=0 l=1 rev1=1 crypto rx=0x7f4a88007b10 tx=0x7f4a88008040 comp rx=0 tx=0).stop 2024-09-16T03:58:14.190 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.185+0000 7f4aa5fe0640 10 monclient: shutdown 2024-09-16T03:58:14.190 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.185+0000 7f4aa5fe0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:58:14.190 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.185+0000 7f4aa5fe0640 1 -- 172.21.15.42:0/1958562724 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4aa00a1710 msgr2=0x7f4aa013d220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:14.191 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.185+0000 7f4aa5fe0640 1 --2- 172.21.15.42:0/1958562724 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4aa00a1710 0x7f4aa013d220 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f4a90000f70 tx=0x7f4a9000c820 comp rx=0 tx=0).stop 2024-09-16T03:58:14.191 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.185+0000 7f4aa5fe0640 1 -- 172.21.15.42:0/1958562724 shutdown_connections 2024-09-16T03:58:14.191 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.185+0000 7f4aa5fe0640 1 --2- 172.21.15.42:0/1958562724 >> v2:172.21.15.154:6812/254845342 conn(0x7f4a80037860 0x7f4a80039d20 unknown :-1 s=CLOSED pgs=3693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:14.191 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.185+0000 7f4aa5fe0640 1 --2- 172.21.15.42:0/1958562724 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4aa01393a0 0x7f4aa013d760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:14.191 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.185+0000 7f4aa5fe0640 1 --2- 172.21.15.42:0/1958562724 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4aa00a1710 0x7f4aa013d220 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:14.191 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.185+0000 7f4aa5fe0640 1 --2- 172.21.15.42:0/1958562724 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4aa00a1330 0x7f4aa013cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:14.191 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.185+0000 7f4aa5fe0640 1 -- 172.21.15.42:0/1958562724 >> 172.21.15.42:0/1958562724 conn(0x7f4aa00a2420 msgr2=0x7f4aa00a4cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:58:14.191 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.185+0000 7f4aa5fe0640 1 -- 172.21.15.42:0/1958562724 shutdown_connections 2024-09-16T03:58:14.191 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.185+0000 7f4aa5fe0640 1 -- 172.21.15.42:0/1958562724 wait complete. 2024-09-16T03:58:14.191 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.185+0000 7f4aa5fe0640 1 librados: shutdown 2024-09-16T03:58:14.207 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T03:58:14.207 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T03:58:14.207 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T03:58:14.207 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T03:58:14.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.329+0000 7f3afcb4f640 10 monclient: get_monmap_and_config 2024-09-16T03:58:14.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.329+0000 7f3afcb4f640 10 monclient: build_initial_monmap 2024-09-16T03:58:14.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.329+0000 7f3afcb4f640 10 monclient: monmap: 2024-09-16T03:58:14.334 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:58:14.334 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:58:14.334 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:58:14.332635+0000 2024-09-16T03:58:14.334 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:58:14.332635+0000 2024-09-16T03:58:14.334 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:58:14.335 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:14.335 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:58:14.335 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:58:14.335 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:58:14.335 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:58:14.335 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:58:14.335 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:14.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.329+0000 7f3afcb4f640 1 Processor -- start 2024-09-16T03:58:14.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.329+0000 7f3afcb4f640 1 -- start start 2024-09-16T03:58:14.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.329+0000 7f3afcb4f640 10 monclient: init 2024-09-16T03:58:14.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.329+0000 7f3afcb4f640 10 monclient: _reopen_session rank -1 2024-09-16T03:58:14.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.329+0000 7f3afcb4f640 10 monclient: _add_conns ranks=[0,3,4,2,1] 2024-09-16T03:58:14.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.329+0000 7f3afcb4f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3af80a1330 addr v1:172.21.15.42:6789/0 2024-09-16T03:58:14.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.329+0000 7f3afcb4f640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f3af80a1710 0x7f3af80a1b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:14.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.329+0000 7f3afcb4f640 10 monclient(hunting): picked mon.noname-a con 0x7f3af80a1710 addr v2:172.21.15.42:3300/0 2024-09-16T03:58:14.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.329+0000 7f3afcb4f640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f3af81393a0 0x7f3af813b790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:14.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.329+0000 7f3afcb4f640 10 monclient(hunting): picked mon.noname-c con 0x7f3af81393a0 addr v2:172.21.15.154:3300/0 2024-09-16T03:58:14.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.329+0000 7f3afcb4f640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3af809e4c0 con 0x7f3af80a1330 2024-09-16T03:58:14.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.329+0000 7f3afcb4f640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:14.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.329+0000 7f3afcb4f640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f3af809edf0 con 0x7f3af80a1710 2024-09-16T03:58:14.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.329+0000 7f3afcb4f640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:14.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.329+0000 7f3afcb4f640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f3af809cd60 con 0x7f3af81393a0 2024-09-16T03:58:14.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.329+0000 7f3afcb4f640 10 monclient(hunting): _renew_subs 2024-09-16T03:58:14.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.329+0000 7f3afcb4f640 10 monclient(hunting): authenticate will time out at 1941.302033s 2024-09-16T03:58:14.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.329+0000 7f3af659c640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f3af80a1330 0x7f3af809f180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:56636/0 (socket says 172.21.15.42:56636) 2024-09-16T03:58:14.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.329+0000 7f3af659c640 1 -- 172.21.15.42:0/2239831440 learned_addr learned my addr 172.21.15.42:0/2239831440 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:58:14.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.329+0000 7f3af5d9b640 1 --2- 172.21.15.42:0/2239831440 >> v2:172.21.15.42:3300/0 conn(0x7f3af80a1710 0x7f3af80a1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:14.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.329+0000 7f3af5d9b640 10 monclient(hunting): get_auth_request con 0x7f3af80a1710 auth_method 0 2024-09-16T03:58:14.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.329+0000 7f3af5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:14.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.329+0000 7f3af5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:14.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.329+0000 7f3af5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:58:14.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.329+0000 7f3af5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:58:14.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.329+0000 7f3af5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:58:14.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.329+0000 7f3af5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:58:14.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.329+0000 7f3af6d9d640 1 --2- 172.21.15.42:0/2239831440 >> v2:172.21.15.154:3300/0 conn(0x7f3af81393a0 0x7f3af813b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:14.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.329+0000 7f3af6d9d640 10 monclient(hunting): get_auth_request con 0x7f3af81393a0 auth_method 0 2024-09-16T03:58:14.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.329+0000 7f3af6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:14.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.329+0000 7f3af6d9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:14.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.329+0000 7f3af6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:58:14.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.329+0000 7f3af5d9b640 10 monclient(hunting): handle_auth_done global_id 824940 payload 274 2024-09-16T03:58:14.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.329+0000 7f3af5d9b640 1 -- 172.21.15.42:0/2239831440 >> v2:172.21.15.154:3300/0 conn(0x7f3af81393a0 msgr2=0x7f3af813b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:58:14.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.329+0000 7f3af5d9b640 1 --2- 172.21.15.42:0/2239831440 >> v2:172.21.15.154:3300/0 conn(0x7f3af81393a0 0x7f3af813b790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:14.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.329+0000 7f3af5d9b640 1 -- 172.21.15.42:0/2239831440 >> v1:172.21.15.42:6789/0 conn(0x7f3af80a1330 legacy=0x7f3af809f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:14.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.329+0000 7f3af5d9b640 10 monclient: _finish_hunting 0 2024-09-16T03:58:14.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.329+0000 7f3af5d9b640 1 monclient: found mon.noname-a 2024-09-16T03:58:14.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.329+0000 7f3af5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:58:14.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.329+0000 7f3af559a640 1 -- 172.21.15.42:0/2239831440 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2262060453 0 0) 0x7f3af809e4c0 con 0x7f3af80a1330 2024-09-16T03:58:14.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.329+0000 7f3af5d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:58:14.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.329+0000 7f3af5d9b640 1 -- 172.21.15.42:0/2239831440 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3af80a2080 con 0x7f3af80a1710 2024-09-16T03:58:14.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.329+0000 7f3af559a640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T03:58:14.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.333+0000 7f3af5d9b640 1 --2- 172.21.15.42:0/2239831440 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3af80a1710 0x7f3af80a1b40 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f3af809df50 tx=0x7f3ae001c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f79bc0b546a66516 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:58:14.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.333+0000 7f3af559a640 1 -- 172.21.15.42:0/2239831440 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ae00a66c0 con 0x7f3af80a1710 2024-09-16T03:58:14.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.333+0000 7f3af559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:58:14.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.333+0000 7f3af559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:58:14.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.333+0000 7f3af559a640 10 monclient: dump: 2024-09-16T03:58:14.348 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:58:14.348 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:58:14.348 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:14.348 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:14.348 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:58:14.348 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:14.348 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:58:14.348 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:58:14.348 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:58:14.348 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:14.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.333+0000 7f3af559a640 10 monclient: _finish_auth 0 2024-09-16T03:58:14.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.333+0000 7f3af559a640 10 monclient: _check_auth_tickets 2024-09-16T03:58:14.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.333+0000 7f3af559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:58:14.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.333+0000 7f3afcb4f640 5 monclient: authenticate success, global_id 824940 2024-09-16T03:58:14.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.333+0000 7f3afcb4f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:58:14.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.333+0000 7f3af559a640 1 -- 172.21.15.42:0/2239831440 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3ae00a6860 con 0x7f3af80a1710 2024-09-16T03:58:14.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.333+0000 7f3af559a640 10 monclient: handle_config config(0 keys) 2024-09-16T03:58:14.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.333+0000 7f3af559a640 1 -- 172.21.15.42:0/2239831440 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ae00a6c30 con 0x7f3af80a1710 2024-09-16T03:58:14.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.333+0000 7f3af559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:58:14.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.333+0000 7f3af559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:58:14.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.333+0000 7f3af559a640 10 monclient: dump: 2024-09-16T03:58:14.349 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:58:14.349 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:58:14.349 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:14.349 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:14.349 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:58:14.349 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:14.350 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:58:14.350 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:58:14.350 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:58:14.350 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:14.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.333+0000 7f3afcb4f640 10 monclient: get_monmap_and_config success 2024-09-16T03:58:14.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.333+0000 7f3afcb4f640 4 set_mon_vals no callback set 2024-09-16T03:58:14.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.333+0000 7f3afcb4f640 10 monclient: shutdown 2024-09-16T03:58:14.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.333+0000 7f3afcb4f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:58:14.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.333+0000 7f3afcb4f640 1 -- 172.21.15.42:0/2239831440 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3af80a1710 msgr2=0x7f3af80a1b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:14.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.333+0000 7f3afcb4f640 1 --2- 172.21.15.42:0/2239831440 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3af80a1710 0x7f3af80a1b40 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f3af809df50 tx=0x7f3ae001c7f0 comp rx=0 tx=0).stop 2024-09-16T03:58:14.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.333+0000 7f3afcb4f640 1 -- 172.21.15.42:0/2239831440 shutdown_connections 2024-09-16T03:58:14.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.333+0000 7f3afcb4f640 1 --2- 172.21.15.42:0/2239831440 >> v2:172.21.15.154:3300/0 conn(0x7f3af81393a0 0x7f3af813b790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:14.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.333+0000 7f3afcb4f640 1 --2- 172.21.15.42:0/2239831440 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3af80a1710 0x7f3af80a1b40 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:14.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.333+0000 7f3afcb4f640 1 -- 172.21.15.42:0/2239831440 >> 172.21.15.42:0/2239831440 conn(0x7f3af80a2420 msgr2=0x7f3af80a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:58:14.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.333+0000 7f3afcb4f640 1 -- 172.21.15.42:0/2239831440 shutdown_connections 2024-09-16T03:58:14.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.333+0000 7f3afcb4f640 1 -- 172.21.15.42:0/2239831440 wait complete. 2024-09-16T03:58:14.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.333+0000 7f3afcb4f640 10 monclient: build_initial_monmap 2024-09-16T03:58:14.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.333+0000 7f3afcb4f640 10 monclient: monmap: 2024-09-16T03:58:14.351 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:58:14.351 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:58:14.351 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:58:14.351 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:58:14.351 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:58:14.351 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:14.351 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:58:14.351 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:58:14.351 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:58:14.351 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:14.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.333+0000 7f3afcb4f640 1 librados: starting msgr at 2024-09-16T03:58:14.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.333+0000 7f3afcb4f640 1 librados: starting objecter 2024-09-16T03:58:14.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.333+0000 7f3afcb4f640 1 Processor -- start 2024-09-16T03:58:14.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.333+0000 7f3afcb4f640 1 -- start start 2024-09-16T03:58:14.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.333+0000 7f3afcb4f640 1 librados: setting wanted keys 2024-09-16T03:58:14.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.333+0000 7f3afcb4f640 1 librados: calling monclient init 2024-09-16T03:58:14.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.333+0000 7f3afcb4f640 10 monclient: init 2024-09-16T03:58:14.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.333+0000 7f3afcb4f640 10 monclient: _reopen_session rank -1 2024-09-16T03:58:14.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.333+0000 7f3afcb4f640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:58:14.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.333+0000 7f3afcb4f640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3af80a1330 0x7f3af81cb5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:14.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.333+0000 7f3afcb4f640 10 monclient(hunting): picked mon.noname-c con 0x7f3af80a1330 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:58:14.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.333+0000 7f3afcb4f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3af80a1710 0x7f3af8134890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:14.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.333+0000 7f3afcb4f640 10 monclient(hunting): picked mon.noname-b con 0x7f3af80a1710 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:58:14.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.333+0000 7f3afcb4f640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3af81393a0 0x7f3af81ceae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:14.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.333+0000 7f3afcb4f640 10 monclient(hunting): picked mon.noname-a con 0x7f3af81393a0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:58:14.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.333+0000 7f3afcb4f640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:14.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.333+0000 7f3afcb4f640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f3af809bde0 con 0x7f3af81393a0 2024-09-16T03:58:14.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.333+0000 7f3afcb4f640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:14.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.333+0000 7f3afcb4f640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f3af809bc60 con 0x7f3af80a1710 2024-09-16T03:58:14.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.333+0000 7f3afcb4f640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:14.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.333+0000 7f3afcb4f640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f3af809bf60 con 0x7f3af80a1330 2024-09-16T03:58:14.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.333+0000 7f3afcb4f640 10 monclient(hunting): _renew_subs 2024-09-16T03:58:14.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.333+0000 7f3afcb4f640 10 monclient(hunting): authenticate will time out at 1941.307527s 2024-09-16T03:58:14.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.333+0000 7f3af6d9d640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3af81393a0 0x7f3af81ceae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:14.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.333+0000 7f3af6d9d640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3af81393a0 0x7f3af81ceae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:50262/0 (socket says 172.21.15.42:50262) 2024-09-16T03:58:14.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.333+0000 7f3af659c640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3af80a1330 0x7f3af81cb5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:14.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.333+0000 7f3af6d9d640 1 -- 172.21.15.42:0/3472372112 learned_addr learned my addr 172.21.15.42:0/3472372112 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:58:14.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.333+0000 7f3af6d9d640 10 monclient(hunting): get_auth_request con 0x7f3af81393a0 auth_method 0 2024-09-16T03:58:14.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.333+0000 7f3af6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:14.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.333+0000 7f3af6d9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:14.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.333+0000 7f3af6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:58:14.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.333+0000 7f3af659c640 10 monclient(hunting): get_auth_request con 0x7f3af80a1330 auth_method 0 2024-09-16T03:58:14.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.333+0000 7f3af659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:14.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.333+0000 7f3af659c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:14.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.333+0000 7f3af659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:58:14.355 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.333+0000 7f3af6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:58:14.355 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.333+0000 7f3af6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:58:14.355 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.333+0000 7f3af6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:58:14.355 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.333+0000 7f3af5d9b640 1 --2- 172.21.15.42:0/3472372112 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3af80a1710 0x7f3af8134890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:14.355 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.333+0000 7f3af659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:58:14.355 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.333+0000 7f3af659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:58:14.355 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.333+0000 7f3af659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:58:14.355 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.333+0000 7f3af5d9b640 10 monclient(hunting): get_auth_request con 0x7f3af80a1710 auth_method 0 2024-09-16T03:58:14.355 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.333+0000 7f3af5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:14.355 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.333+0000 7f3af5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:14.355 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.333+0000 7f3af5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:58:14.355 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.337+0000 7f3af5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:58:14.355 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.337+0000 7f3af5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:58:14.355 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.337+0000 7f3af5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:58:14.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.337+0000 7f3af659c640 10 monclient(hunting): handle_auth_done global_id 854468 payload 824 2024-09-16T03:58:14.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.337+0000 7f3af659c640 1 -- 172.21.15.42:0/3472372112 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3af80a1710 msgr2=0x7f3af8134890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:14.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.337+0000 7f3af659c640 1 --2- 172.21.15.42:0/3472372112 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3af80a1710 0x7f3af8134890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:14.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.337+0000 7f3af659c640 1 -- 172.21.15.42:0/3472372112 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3af81393a0 msgr2=0x7f3af81ceae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:14.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.337+0000 7f3af659c640 1 --2- 172.21.15.42:0/3472372112 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3af81393a0 0x7f3af81ceae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:14.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.337+0000 7f3af659c640 10 monclient: _finish_hunting 0 2024-09-16T03:58:14.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.337+0000 7f3af659c640 1 monclient: found mon.noname-c 2024-09-16T03:58:14.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.337+0000 7f3af659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:58:14.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.337+0000 7f3af659c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:58:14.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.337+0000 7f3af659c640 1 -- 172.21.15.42:0/3472372112 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3af8071e10 con 0x7f3af80a1330 2024-09-16T03:58:14.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.337+0000 7f3af6d9d640 1 --2- 172.21.15.42:0/3472372112 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3af81393a0 0x7f3af81ceae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:58:14.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.337+0000 7f3af659c640 1 --2- 172.21.15.42:0/3472372112 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3af80a1330 0x7f3af81cb5e0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f3ae800c460 tx=0x7f3ae800c920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:58:14.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.337+0000 7f3aeeffd640 10 client.?.objecter ms_handle_connect 0x7f3af80a1330 2024-09-16T03:58:14.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.337+0000 7f3aeeffd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:58:14.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.337+0000 7f3aeeffd640 1 -- 172.21.15.42:0/3472372112 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ae8018070 con 0x7f3af80a1330 2024-09-16T03:58:14.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.337+0000 7f3aeeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:58:14.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.337+0000 7f3aeeffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:58:14.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.337+0000 7f3aeeffd640 10 monclient: dump: 2024-09-16T03:58:14.357 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:58:14.357 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:58:14.357 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:14.357 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:14.357 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:58:14.357 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:14.357 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:58:14.357 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:58:14.357 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:58:14.358 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:14.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.337+0000 7f3aeeffd640 10 monclient: _finish_auth 0 2024-09-16T03:58:14.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.337+0000 7f3aeeffd640 10 monclient: _check_auth_tickets 2024-09-16T03:58:14.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.337+0000 7f3aeeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:58:14.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.337+0000 7f3afcb4f640 5 monclient: authenticate success, global_id 854468 2024-09-16T03:58:14.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.337+0000 7f3aeeffd640 1 -- 172.21.15.42:0/3472372112 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3ae8007500 con 0x7f3af80a1330 2024-09-16T03:58:14.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.337+0000 7f3aeeffd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:58:14.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.337+0000 7f3aeeffd640 1 -- 172.21.15.42:0/3472372112 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ae8004a30 con 0x7f3af80a1330 2024-09-16T03:58:14.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.337+0000 7f3aeeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:58:14.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.337+0000 7f3aeeffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:58:14.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.337+0000 7f3aeeffd640 10 monclient: dump: 2024-09-16T03:58:14.358 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:58:14.359 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:58:14.359 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:14.359 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:14.359 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:58:14.359 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:14.359 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:58:14.359 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:58:14.359 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:58:14.359 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:14.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.337+0000 7f3aeffff640 4 set_mon_vals no callback set 2024-09-16T03:58:14.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.337+0000 7f3afcb4f640 10 monclient: _renew_subs 2024-09-16T03:58:14.360 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.337+0000 7f3afcb4f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:58:14.360 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.337+0000 7f3afcb4f640 1 -- 172.21.15.42:0/3472372112 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3af8073a30 con 0x7f3af80a1330 2024-09-16T03:58:14.360 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.337+0000 7f3afcb4f640 10 client.854468.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:58:14.360 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.337+0000 7f3afcb4f640 10 monclient: _renew_subs 2024-09-16T03:58:14.360 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.337+0000 7f3afcb4f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:58:14.360 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.337+0000 7f3afcb4f640 1 -- 172.21.15.42:0/3472372112 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f3af811ff50 con 0x7f3af80a1330 2024-09-16T03:58:14.360 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.337+0000 7f3afcb4f640 1 librados: init done 2024-09-16T03:58:14.360 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.337+0000 7f3afcb4f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:58:14.360 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.337+0000 7f3afcb4f640 1 --2- 172.21.15.42:0/3472372112 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3af81cf020 0x7f3af81cf400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:14.360 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.337+0000 7f3afcb4f640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:14.360 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.337+0000 7f3afcb4f640 1 -- 172.21.15.42:0/3472372112 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f3af8072360 con 0x7f3af81cf020 2024-09-16T03:58:14.361 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.337+0000 7f3aeeffd640 1 -- 172.21.15.42:0/3472372112 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f3ae8004bd0 con 0x7f3af80a1330 2024-09-16T03:58:14.361 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.337+0000 7f3af5d9b640 1 --2- 172.21.15.42:0/3472372112 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3af81cf020 0x7f3af81cf400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:14.361 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.337+0000 7f3af5d9b640 10 monclient: get_auth_request con 0x7f3af81cf020 auth_method 0 2024-09-16T03:58:14.361 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.337+0000 7f3af5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:14.361 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.337+0000 7f3af5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:14.361 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.337+0000 7f3af5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:58:14.361 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.337+0000 7f3af5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:58:14.361 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.337+0000 7f3af5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:58:14.361 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.337+0000 7f3af5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:58:14.361 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.337+0000 7f3aeeffd640 1 --2- 172.21.15.42:0/3472372112 >> v2:172.21.15.154:6812/254845342 conn(0x7f3adc037840 0x7f3adc039d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:14.361 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.337+0000 7f3aeeffd640 1 -- 172.21.15.42:0/3472372112 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(484..484 src has 366..484) ==== 6181+0+0 (secure 0 0 0) 0x7f3ae80484f0 con 0x7f3af80a1330 2024-09-16T03:58:14.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.337+0000 7f3aeeffd640 10 client.854468.objecter ms_dispatch 0x7f3af8000ba0 osd_map(484..484 src has 366..484) 2024-09-16T03:58:14.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.337+0000 7f3aeeffd640 3 client.854468.objecter handle_osd_map got epochs [484,484] > 0 2024-09-16T03:58:14.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.337+0000 7f3aeeffd640 3 client.854468.objecter handle_osd_map decoding full epoch 484 2024-09-16T03:58:14.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.337+0000 7f3af5d9b640 10 monclient(hunting): handle_auth_done global_id 824949 payload 824 2024-09-16T03:58:14.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.337+0000 7f3af5d9b640 1 -- 172.21.15.42:0/3472372112 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3af81cf940 con 0x7f3af81cf020 2024-09-16T03:58:14.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.337+0000 7f3aeeffd640 20 client.854468.objecter dump_active .. 0 homeless 2024-09-16T03:58:14.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.337+0000 7f3af6d9d640 1 --2- 172.21.15.42:0/3472372112 >> v2:172.21.15.154:6812/254845342 conn(0x7f3adc037840 0x7f3adc039d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:14.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.337+0000 7f3aeeffd640 10 client.854468.objecter ms_handle_connect 0x7f3af81cf020 2024-09-16T03:58:14.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.337+0000 7f3aeeffd640 10 client.854468.objecter resend_mon_ops 2024-09-16T03:58:14.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.337+0000 7f3af5d9b640 1 --2- 172.21.15.42:0/3472372112 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3af81cf020 0x7f3af81cf400 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f3af806add0 tx=0x7f3ae0019590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:58:14.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.337+0000 7f3af6d9d640 10 monclient: get_auth_request con 0x7f3adc037840 auth_method 0 2024-09-16T03:58:14.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.337+0000 7f3aeeffd640 1 -- 172.21.15.42:0/3472372112 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ae0009450 con 0x7f3af81cf020 2024-09-16T03:58:14.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.337+0000 7f3aeeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:58:14.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.337+0000 7f3aeeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:58:14.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.337+0000 7f3aeeffd640 10 monclient: dump: 2024-09-16T03:58:14.363 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:58:14.363 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:58:14.363 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:14.363 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:14.363 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:58:14.363 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:14.363 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:58:14.363 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:58:14.363 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:58:14.364 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:14.364 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.337+0000 7f3aeeffd640 1 -- 172.21.15.42:0/3472372112 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f3ae0003ba0 con 0x7f3af81cf020 2024-09-16T03:58:14.364 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.337+0000 7f3aeeffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:58:14.364 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.337+0000 7f3aeeffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:58:14.364 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.337+0000 7f3aeeffd640 1 -- 172.21.15.42:0/3472372112 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3af81cf020 msgr2=0x7f3af81cf400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:14.364 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.337+0000 7f3aeeffd640 1 --2- 172.21.15.42:0/3472372112 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3af81cf020 0x7f3af81cf400 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f3af806add0 tx=0x7f3ae0019590 comp rx=0 tx=0).stop 2024-09-16T03:58:14.364 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.337+0000 7f3aeeffd640 1 -- 172.21.15.42:0/3472372112 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3af81cf020 msgr2=0x7f3af81cf400 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:58:14.364 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.337+0000 7f3aeeffd640 1 --2- 172.21.15.42:0/3472372112 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3af81cf020 0x7f3af81cf400 secure :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f3af806add0 tx=0x7f3ae0019590 comp rx=0 tx=0).stop 2024-09-16T03:58:14.364 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.337+0000 7f3aeeffd640 10 client.854468.objecter ms_handle_connect 0x7f3adc037840 2024-09-16T03:58:14.364 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.337+0000 7f3af6d9d640 1 --2- 172.21.15.42:0/3472372112 >> v2:172.21.15.154:6812/254845342 conn(0x7f3adc037840 0x7f3adc039d00 secure :-1 s=READY pgs=3694 cs=0 l=1 rev1=1 crypto rx=0x7f3af811fba0 tx=0x7f3af0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:58:14.366 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.345+0000 7f3afcb4f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:58:14.366 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.345+0000 7f3afcb4f640 1 --2- 172.21.15.42:0/3472372112 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3af8135380 0x7f3af8135790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:14.366 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.345+0000 7f3afcb4f640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:14.366 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.345+0000 7f3afcb4f640 1 -- 172.21.15.42:0/3472372112 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f3af80a1b40 con 0x7f3af8135380 2024-09-16T03:58:14.366 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.345+0000 7f3af5d9b640 1 --2- 172.21.15.42:0/3472372112 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3af8135380 0x7f3af8135790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:14.366 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.345+0000 7f3af5d9b640 10 monclient: get_auth_request con 0x7f3af8135380 auth_method 0 2024-09-16T03:58:14.366 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.345+0000 7f3af5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:14.366 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.345+0000 7f3af5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:14.367 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.345+0000 7f3af5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:58:14.367 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.345+0000 7f3af5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:58:14.367 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.345+0000 7f3af5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:58:14.367 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.345+0000 7f3af5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:58:14.367 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.345+0000 7f3af5d9b640 10 monclient(hunting): handle_auth_done global_id 824952 payload 824 2024-09-16T03:58:14.367 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.345+0000 7f3af5d9b640 1 -- 172.21.15.42:0/3472372112 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f3af811e5f0 con 0x7f3af8135380 2024-09-16T03:58:14.367 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.345+0000 7f3af5d9b640 1 --2- 172.21.15.42:0/3472372112 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3af8135380 0x7f3af8135790 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f3ae001c830 tx=0x7f3ae00a9000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:58:14.367 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.345+0000 7f3aeeffd640 10 client.854468.objecter ms_handle_connect 0x7f3af8135380 2024-09-16T03:58:14.367 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.345+0000 7f3aeeffd640 10 client.854468.objecter resend_mon_ops 2024-09-16T03:58:14.367 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.345+0000 7f3aeeffd640 1 -- 172.21.15.42:0/3472372112 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ae00b9970 con 0x7f3af8135380 2024-09-16T03:58:14.367 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.345+0000 7f3aeeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:58:14.367 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.345+0000 7f3aeeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:58:14.367 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.345+0000 7f3aeeffd640 10 monclient: dump: 2024-09-16T03:58:14.368 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:58:14.368 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:58:14.368 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:14.368 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:14.368 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:58:14.368 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:14.368 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:58:14.368 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:58:14.368 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:58:14.368 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:14.368 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.345+0000 7f3aeeffd640 1 -- 172.21.15.42:0/3472372112 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4882 (secure 0 0 0) 0x7f3ae00cd450 con 0x7f3af8135380 2024-09-16T03:58:14.368 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.345+0000 7f3aeeffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:58:14.368 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.345+0000 7f3aeeffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:58:14.369 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.345+0000 7f3aeeffd640 1 -- 172.21.15.42:0/3472372112 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3af8135380 msgr2=0x7f3af8135790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:14.369 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.345+0000 7f3aeeffd640 1 --2- 172.21.15.42:0/3472372112 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3af8135380 0x7f3af8135790 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f3ae001c830 tx=0x7f3ae00a9000 comp rx=0 tx=0).stop 2024-09-16T03:58:14.369 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.345+0000 7f3aeeffd640 1 -- 172.21.15.42:0/3472372112 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3af8135380 msgr2=0x7f3af8135790 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:58:14.369 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.345+0000 7f3aeeffd640 1 --2- 172.21.15.42:0/3472372112 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3af8135380 0x7f3af8135790 secure :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f3ae001c830 tx=0x7f3ae00a9000 comp rx=0 tx=0).stop 2024-09-16T03:58:14.369 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:58:14.369 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:58:14.369 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:58:14.369 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "leader", 2024-09-16T03:58:14.369 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 148, 2024-09-16T03:58:14.369 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 384200, 2024-09-16T03:58:14.370 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:58:14.370 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:58:14.370 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T03:58:14.370 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T03:58:14.370 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:58:14.370 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 2, 2024-09-16T03:58:14.370 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:58:14.370 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:58:14.370 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:58:14.370 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:58:14.370 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:58:14.370 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:58:14.370 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:58:14.371 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:58:14.371 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:58:14.371 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:58:14.371 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:58:14.371 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:58:14.371 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:58:14.371 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:58:14.371 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:58:14.371 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:58:14.371 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:58:14.371 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:58:14.371 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:58:14.371 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:58:14.372 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:58:14.372 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:58:14.372 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:58:14.372 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:58:14.372 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:58:14.372 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:58:14.372 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:58:14.372 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:58:14.372 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:58:14.372 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:58:14.372 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:58:14.372 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:58:14.373 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:58:14.373 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:58:14.373 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:58:14.373 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:58:14.373 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:58:14.373 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:58:14.373 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:58:14.373 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:58:14.373 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:58:14.373 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:58:14.373 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:58:14.373 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:58:14.373 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:58:14.374 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:58:14.374 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:58:14.374 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:58:14.374 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:58:14.374 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:58:14.374 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:58:14.374 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:58:14.374 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:58:14.374 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:58:14.374 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:58:14.374 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:58:14.374 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:58:14.374 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:58:14.374 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:58:14.375 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:58:14.375 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:58:14.375 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:58:14.375 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:58:14.375 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:58:14.375 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:58:14.375 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:58:14.375 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:58:14.375 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:58:14.375 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:58:14.375 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:58:14.375 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:58:14.375 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:58:14.375 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:58:14.376 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:58:14.376 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:58:14.376 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:58:14.376 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:58:14.376 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:58:14.376 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:58:14.376 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:58:14.376 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:58:14.376 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:58:14.376 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:58:14.376 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:58:14.376 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:58:14.376 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:58:14.376 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:58:14.377 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:58:14.377 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:58:14.377 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:58:14.377 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:58:14.377 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:58:14.377 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:58:14.377 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:58:14.377 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:58:14.377 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:58:14.377 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:58:14.377 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:58:14.377 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:58:14.377 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:58:14.378 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:58:14.378 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:58:14.378 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:58:14.378 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:58:14.378 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:58:14.378 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:58:14.378 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:58:14.378 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:58:14.378 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:58:14.378 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:58:14.378 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:58:14.378 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:58:14.378 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:58:14.378 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:58:14.379 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:58:14.379 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:58:14.379 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:58:14.379 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:58:14.379 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:58:14.379 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:58:14.379 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:58:14.379 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:58:14.379 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:58:14.379 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:58:14.379 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:58:14.379 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:58:14.379 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:58:14.379 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:58:14.380 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:58:14.380 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:58:14.380 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:58:14.380 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:58:14.380 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:58:14.380 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:58:14.380 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:58:14.380 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:58:14.380 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:58:14.380 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:58:14.380 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:58:14.380 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:58:14.380 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:58:14.380 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T03:58:14.381 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:58:14.381 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:58:14.381 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:58:14.381 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 5 2024-09-16T03:58:14.381 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:58:14.381 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:58:14.381 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:58:14.381 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:58:14.381 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:58:14.381 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:58:14.381 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 7 2024-09-16T03:58:14.381 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:58:14.381 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:58:14.381 INFO:teuthology.orchestra.run.smithi042.stdout: "mgr": [ 2024-09-16T03:58:14.382 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:58:14.382 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:58:14.382 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:58:14.382 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:58:14.382 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:58:14.382 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:58:14.382 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:58:14.382 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:58:14.382 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:58:14.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.345+0000 7f3afcb4f640 10 librados: watch_flush enter 2024-09-16T03:58:14.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.345+0000 7f3afcb4f640 10 librados: watch_flush exit 2024-09-16T03:58:14.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.345+0000 7f3afcb4f640 20 client.854468.objecter shutdown clearing up homeless session... 2024-09-16T03:58:14.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.345+0000 7f3afcb4f640 10 client.854468.objecter successfully canceled tick 2024-09-16T03:58:14.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.345+0000 7f3afcb4f640 1 -- 172.21.15.42:0/3472372112 >> v2:172.21.15.154:6812/254845342 conn(0x7f3adc037840 msgr2=0x7f3adc039d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:14.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.345+0000 7f3afcb4f640 1 --2- 172.21.15.42:0/3472372112 >> v2:172.21.15.154:6812/254845342 conn(0x7f3adc037840 0x7f3adc039d00 secure :-1 s=READY pgs=3694 cs=0 l=1 rev1=1 crypto rx=0x7f3af811fba0 tx=0x7f3af0008040 comp rx=0 tx=0).stop 2024-09-16T03:58:14.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.345+0000 7f3afcb4f640 10 monclient: shutdown 2024-09-16T03:58:14.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.345+0000 7f3afcb4f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:58:14.385 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.345+0000 7f3afcb4f640 1 -- 172.21.15.42:0/3472372112 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3af80a1330 msgr2=0x7f3af81cb5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:14.385 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.345+0000 7f3afcb4f640 1 --2- 172.21.15.42:0/3472372112 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3af80a1330 0x7f3af81cb5e0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f3ae800c460 tx=0x7f3ae800c920 comp rx=0 tx=0).stop 2024-09-16T03:58:14.385 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.345+0000 7f3af6d9d640 1 -- 172.21.15.42:0/3472372112 reap_dead start 2024-09-16T03:58:14.385 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.345+0000 7f3af6d9d640 1 -- 172.21.15.42:0/3472372112 reap_dead start 2024-09-16T03:58:14.385 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.345+0000 7f3afcb4f640 1 -- 172.21.15.42:0/3472372112 shutdown_connections 2024-09-16T03:58:14.385 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.345+0000 7f3afcb4f640 1 -- 172.21.15.42:0/3472372112 >> 172.21.15.42:0/3472372112 conn(0x7f3af80a2420 msgr2=0x7f3af81388c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:58:14.385 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.345+0000 7f3afcb4f640 1 -- 172.21.15.42:0/3472372112 shutdown_connections 2024-09-16T03:58:14.385 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.345+0000 7f3afcb4f640 1 -- 172.21.15.42:0/3472372112 wait complete. 2024-09-16T03:58:14.385 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.345+0000 7f3afcb4f640 1 librados: shutdown 2024-09-16T03:58:14.385 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:58:14.386 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T03:58:14.386 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T03:58:14.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.497+0000 7fcb5f6d6640 10 monclient: get_monmap_and_config 2024-09-16T03:58:14.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.497+0000 7fcb5f6d6640 10 monclient: build_initial_monmap 2024-09-16T03:58:14.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.497+0000 7fcb5f6d6640 10 monclient: monmap: 2024-09-16T03:58:14.505 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:58:14.505 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:58:14.505 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:58:14.502992+0000 2024-09-16T03:58:14.505 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:58:14.502992+0000 2024-09-16T03:58:14.505 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:58:14.505 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:14.505 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:58:14.505 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:58:14.505 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:58:14.505 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:58:14.505 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:58:14.505 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:14.508 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.497+0000 7fcb5f6d6640 1 Processor -- start 2024-09-16T03:58:14.508 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.497+0000 7fcb5f6d6640 1 -- start start 2024-09-16T03:58:14.508 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.497+0000 7fcb5f6d6640 10 monclient: init 2024-09-16T03:58:14.508 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.501+0000 7fcb5f6d6640 10 monclient: _reopen_session rank -1 2024-09-16T03:58:14.508 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.501+0000 7fcb5f6d6640 10 monclient: _add_conns ranks=[3,4,0,1,2] 2024-09-16T03:58:14.508 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.501+0000 7fcb5f6d6640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fcb580a1330 0x7fcb5809f180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:14.508 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.501+0000 7fcb5f6d6640 10 monclient(hunting): picked mon.noname-a con 0x7fcb580a1330 addr v2:172.21.15.42:3300/0 2024-09-16T03:58:14.508 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.501+0000 7fcb5f6d6640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fcb580a1710 0x7fcb580a1b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:14.509 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.501+0000 7fcb5f6d6640 10 monclient(hunting): picked mon.noname-c con 0x7fcb580a1710 addr v2:172.21.15.154:3300/0 2024-09-16T03:58:14.509 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.501+0000 7fcb5f6d6640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fcb581340f0 addr v1:172.21.15.42:6789/0 2024-09-16T03:58:14.509 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.501+0000 7fcb5f6d6640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcb5809e4c0 con 0x7fcb581340f0 2024-09-16T03:58:14.509 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.501+0000 7fcb5f6d6640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:14.509 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.501+0000 7fcb5f6d6640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7fcb5809edf0 con 0x7fcb580a1330 2024-09-16T03:58:14.509 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.501+0000 7fcb5f6d6640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:14.509 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.501+0000 7fcb5f6d6640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fcb5809cd60 con 0x7fcb580a1710 2024-09-16T03:58:14.509 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.501+0000 7fcb5f6d6640 10 monclient(hunting): _renew_subs 2024-09-16T03:58:14.509 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.501+0000 7fcb5f6d6640 10 monclient(hunting): authenticate will time out at 1941.472606s 2024-09-16T03:58:14.509 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.501+0000 7fcb5d472640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fcb580a1330 0x7fcb5809f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:14.509 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.501+0000 7fcb5d472640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fcb580a1330 0x7fcb5809f180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:50292/0 (socket says 172.21.15.42:50292) 2024-09-16T03:58:14.509 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.501+0000 7fcb5d472640 1 -- 172.21.15.42:0/682539965 learned_addr learned my addr 172.21.15.42:0/682539965 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:58:14.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.501+0000 7fcb5dc73640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fcb581340f0 0x7fcb58134540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:56638/0 (socket says 172.21.15.42:56638) 2024-09-16T03:58:14.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.501+0000 7fcb5d472640 10 monclient(hunting): get_auth_request con 0x7fcb580a1330 auth_method 0 2024-09-16T03:58:14.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.501+0000 7fcb5d472640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:14.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.501+0000 7fcb5d472640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:14.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.501+0000 7fcb5d472640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:58:14.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.501+0000 7fcb5d472640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:58:14.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.501+0000 7fcb5d472640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:58:14.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.501+0000 7fcb5d472640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:58:14.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.501+0000 7fcb5cc71640 1 --2- 172.21.15.42:0/682539965 >> v2:172.21.15.154:3300/0 conn(0x7fcb580a1710 0x7fcb580a1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:14.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.501+0000 7fcb5cc71640 10 monclient(hunting): get_auth_request con 0x7fcb580a1710 auth_method 0 2024-09-16T03:58:14.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.501+0000 7fcb5cc71640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:14.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.501+0000 7fcb5cc71640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:14.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.501+0000 7fcb5cc71640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:58:14.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.501+0000 7fcb5d472640 10 monclient(hunting): handle_auth_done global_id 824955 payload 274 2024-09-16T03:58:14.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.501+0000 7fcb5d472640 1 -- 172.21.15.42:0/682539965 >> v2:172.21.15.154:3300/0 conn(0x7fcb580a1710 msgr2=0x7fcb580a1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:58:14.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.501+0000 7fcb5d472640 1 --2- 172.21.15.42:0/682539965 >> v2:172.21.15.154:3300/0 conn(0x7fcb580a1710 0x7fcb580a1b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:14.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.501+0000 7fcb5d472640 1 -- 172.21.15.42:0/682539965 >> v1:172.21.15.42:6789/0 conn(0x7fcb581340f0 legacy=0x7fcb58134540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:14.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.501+0000 7fcb5d472640 10 monclient: _finish_hunting 0 2024-09-16T03:58:14.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.501+0000 7fcb5d472640 1 monclient: found mon.noname-a 2024-09-16T03:58:14.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.501+0000 7fcb5d472640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:58:14.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.501+0000 7fcb4ffff640 1 -- 172.21.15.42:0/682539965 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3570403325 0 0) 0x7fcb5809e4c0 con 0x7fcb581340f0 2024-09-16T03:58:14.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.501+0000 7fcb5d472640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:58:14.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.501+0000 7fcb5d472640 1 -- 172.21.15.42:0/682539965 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb580a2080 con 0x7fcb580a1330 2024-09-16T03:58:14.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.501+0000 7fcb4ffff640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T03:58:14.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.501+0000 7fcb5d472640 1 --2- 172.21.15.42:0/682539965 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcb580a1330 0x7fcb5809f180 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fcb58073490 tx=0x7fcb5401eb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbec3bccb4e704ce server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:58:14.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.501+0000 7fcb4ffff640 1 -- 172.21.15.42:0/682539965 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb540b4070 con 0x7fcb580a1330 2024-09-16T03:58:14.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.501+0000 7fcb4ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:58:14.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.501+0000 7fcb4ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:58:14.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.501+0000 7fcb4ffff640 10 monclient: dump: 2024-09-16T03:58:14.519 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:58:14.519 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:58:14.519 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:14.519 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:14.519 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:58:14.519 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:14.519 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:58:14.519 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:58:14.519 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:58:14.520 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:14.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.501+0000 7fcb4ffff640 10 monclient: _finish_auth 0 2024-09-16T03:58:14.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.501+0000 7fcb4ffff640 10 monclient: _check_auth_tickets 2024-09-16T03:58:14.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.501+0000 7fcb4ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:58:14.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.501+0000 7fcb5f6d6640 5 monclient: authenticate success, global_id 824955 2024-09-16T03:58:14.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.501+0000 7fcb4ffff640 1 -- 172.21.15.42:0/682539965 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcb540a7a20 con 0x7fcb580a1330 2024-09-16T03:58:14.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.501+0000 7fcb5f6d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:58:14.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.501+0000 7fcb4ffff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:58:14.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.501+0000 7fcb4ffff640 1 -- 172.21.15.42:0/682539965 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb540a7df0 con 0x7fcb580a1330 2024-09-16T03:58:14.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.501+0000 7fcb4ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:58:14.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.501+0000 7fcb4ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:58:14.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.501+0000 7fcb4ffff640 10 monclient: dump: 2024-09-16T03:58:14.521 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:58:14.521 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:58:14.521 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:14.521 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:14.521 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:58:14.521 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:14.521 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:58:14.521 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:58:14.521 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:58:14.521 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:14.521 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.501+0000 7fcb5f6d6640 10 monclient: get_monmap_and_config success 2024-09-16T03:58:14.522 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.501+0000 7fcb5f6d6640 4 set_mon_vals no callback set 2024-09-16T03:58:14.522 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.505+0000 7fcb5f6d6640 10 monclient: shutdown 2024-09-16T03:58:14.522 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.505+0000 7fcb5f6d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:58:14.522 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.505+0000 7fcb5f6d6640 1 -- 172.21.15.42:0/682539965 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcb580a1330 msgr2=0x7fcb5809f180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:14.522 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.505+0000 7fcb5f6d6640 1 --2- 172.21.15.42:0/682539965 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcb580a1330 0x7fcb5809f180 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fcb58073490 tx=0x7fcb5401eb40 comp rx=0 tx=0).stop 2024-09-16T03:58:14.522 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.505+0000 7fcb5f6d6640 1 -- 172.21.15.42:0/682539965 shutdown_connections 2024-09-16T03:58:14.522 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.505+0000 7fcb5f6d6640 1 --2- 172.21.15.42:0/682539965 >> v2:172.21.15.154:3300/0 conn(0x7fcb580a1710 0x7fcb580a1b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:14.522 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.505+0000 7fcb5f6d6640 1 --2- 172.21.15.42:0/682539965 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcb580a1330 0x7fcb5809f180 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:14.522 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.505+0000 7fcb5f6d6640 1 -- 172.21.15.42:0/682539965 >> 172.21.15.42:0/682539965 conn(0x7fcb580a2420 msgr2=0x7fcb580a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:58:14.522 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.505+0000 7fcb5f6d6640 1 -- 172.21.15.42:0/682539965 shutdown_connections 2024-09-16T03:58:14.522 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.505+0000 7fcb5f6d6640 1 -- 172.21.15.42:0/682539965 wait complete. 2024-09-16T03:58:14.522 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.505+0000 7fcb5f6d6640 10 monclient: build_initial_monmap 2024-09-16T03:58:14.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.505+0000 7fcb5f6d6640 10 monclient: monmap: 2024-09-16T03:58:14.523 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:58:14.523 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:58:14.523 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:58:14.523 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:58:14.523 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:58:14.523 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:14.523 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:58:14.523 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:58:14.523 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:58:14.523 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:14.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.505+0000 7fcb5f6d6640 1 librados: starting msgr at 2024-09-16T03:58:14.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.505+0000 7fcb5f6d6640 1 librados: starting objecter 2024-09-16T03:58:14.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.505+0000 7fcb5f6d6640 1 Processor -- start 2024-09-16T03:58:14.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.505+0000 7fcb5f6d6640 1 -- start start 2024-09-16T03:58:14.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.505+0000 7fcb5f6d6640 1 librados: setting wanted keys 2024-09-16T03:58:14.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.505+0000 7fcb5f6d6640 1 librados: calling monclient init 2024-09-16T03:58:14.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.505+0000 7fcb5f6d6640 10 monclient: init 2024-09-16T03:58:14.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.505+0000 7fcb5f6d6640 10 monclient: _reopen_session rank -1 2024-09-16T03:58:14.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.505+0000 7fcb5f6d6640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T03:58:14.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.505+0000 7fcb5f6d6640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcb580a1330 0x7fcb581cb3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:14.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.505+0000 7fcb5f6d6640 10 monclient(hunting): picked mon.noname-a con 0x7fcb580a1330 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:58:14.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.505+0000 7fcb5f6d6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcb580a1710 0x7fcb581cb900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:14.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.505+0000 7fcb5f6d6640 10 monclient(hunting): picked mon.noname-b con 0x7fcb580a1710 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:58:14.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.505+0000 7fcb5f6d6640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fcb581340f0 0x7fcb58139bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:14.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.505+0000 7fcb5f6d6640 10 monclient(hunting): picked mon.noname-c con 0x7fcb581340f0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:58:14.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.505+0000 7fcb5f6d6640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:14.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.505+0000 7fcb5f6d6640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fcb5809c3e0 con 0x7fcb580a1330 2024-09-16T03:58:14.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.505+0000 7fcb5f6d6640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:14.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.505+0000 7fcb5f6d6640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fcb5809c260 con 0x7fcb580a1710 2024-09-16T03:58:14.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.505+0000 7fcb5f6d6640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:14.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.505+0000 7fcb5f6d6640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fcb5809c560 con 0x7fcb581340f0 2024-09-16T03:58:14.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.505+0000 7fcb5f6d6640 10 monclient(hunting): _renew_subs 2024-09-16T03:58:14.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.505+0000 7fcb5f6d6640 10 monclient(hunting): authenticate will time out at 1941.477858s 2024-09-16T03:58:14.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.505+0000 7fcb5d472640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcb580a1330 0x7fcb581cb3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:14.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.505+0000 7fcb5d472640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcb580a1330 0x7fcb581cb3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:50294/0 (socket says 172.21.15.42:50294) 2024-09-16T03:58:14.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.505+0000 7fcb5d472640 1 -- 172.21.15.42:0/2929948969 learned_addr learned my addr 172.21.15.42:0/2929948969 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:58:14.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.505+0000 7fcb5d472640 10 monclient(hunting): get_auth_request con 0x7fcb580a1330 auth_method 0 2024-09-16T03:58:14.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.505+0000 7fcb5d472640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:14.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.505+0000 7fcb5d472640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:14.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.505+0000 7fcb5d472640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:58:14.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.505+0000 7fcb5dc73640 1 --2- 172.21.15.42:0/2929948969 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fcb581340f0 0x7fcb58139bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:14.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.505+0000 7fcb5d472640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:58:14.527 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.505+0000 7fcb5d472640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:58:14.527 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.505+0000 7fcb5d472640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:58:14.527 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.505+0000 7fcb5dc73640 10 monclient(hunting): get_auth_request con 0x7fcb581340f0 auth_method 0 2024-09-16T03:58:14.527 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.505+0000 7fcb5dc73640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:14.527 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.505+0000 7fcb5dc73640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:14.527 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.505+0000 7fcb5dc73640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:58:14.527 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.505+0000 7fcb5dc73640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:58:14.527 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.505+0000 7fcb5dc73640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:58:14.527 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.505+0000 7fcb5dc73640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:58:14.527 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.505+0000 7fcb5d472640 10 monclient(hunting): handle_auth_done global_id 824961 payload 824 2024-09-16T03:58:14.527 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.505+0000 7fcb5cc71640 1 --2- 172.21.15.42:0/2929948969 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcb580a1710 0x7fcb581cb900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:14.528 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.505+0000 7fcb5d472640 1 -- 172.21.15.42:0/2929948969 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fcb581340f0 msgr2=0x7fcb58139bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:14.528 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.505+0000 7fcb5d472640 1 --2- 172.21.15.42:0/2929948969 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fcb581340f0 0x7fcb58139bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:14.528 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.505+0000 7fcb5d472640 1 -- 172.21.15.42:0/2929948969 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcb580a1710 msgr2=0x7fcb581cb900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:14.528 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.505+0000 7fcb5d472640 1 --2- 172.21.15.42:0/2929948969 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcb580a1710 0x7fcb581cb900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:14.528 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.505+0000 7fcb5d472640 10 monclient: _finish_hunting 0 2024-09-16T03:58:14.528 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.505+0000 7fcb5d472640 1 monclient: found mon.noname-a 2024-09-16T03:58:14.528 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.505+0000 7fcb5d472640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:58:14.528 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.505+0000 7fcb5d472640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:58:14.528 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.505+0000 7fcb5d472640 1 -- 172.21.15.42:0/2929948969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb5811d180 con 0x7fcb580a1330 2024-09-16T03:58:14.529 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.505+0000 7fcb5d472640 1 --2- 172.21.15.42:0/2929948969 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcb580a1330 0x7fcb581cb3c0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fcb5401b750 tx=0x7fcb5401b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:58:14.529 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.505+0000 7fcb4dffb640 10 client.?.objecter ms_handle_connect 0x7fcb580a1330 2024-09-16T03:58:14.532 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.505+0000 7fcb4dffb640 10 client.?.objecter resend_mon_ops 2024-09-16T03:58:14.532 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.505+0000 7fcb4dffb640 1 -- 172.21.15.42:0/2929948969 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb540be070 con 0x7fcb580a1330 2024-09-16T03:58:14.532 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.505+0000 7fcb4dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:58:14.532 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.505+0000 7fcb4dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:58:14.532 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.505+0000 7fcb4dffb640 10 monclient: dump: 2024-09-16T03:58:14.533 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:58:14.533 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:58:14.533 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:14.533 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:14.533 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:58:14.533 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:14.533 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:58:14.533 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:58:14.533 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:58:14.534 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:14.534 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.505+0000 7fcb4dffb640 10 monclient: _finish_auth 0 2024-09-16T03:58:14.534 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.505+0000 7fcb4dffb640 10 monclient: _check_auth_tickets 2024-09-16T03:58:14.534 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.505+0000 7fcb4dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:58:14.534 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.505+0000 7fcb5f6d6640 5 monclient: authenticate success, global_id 824961 2024-09-16T03:58:14.534 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.505+0000 7fcb5f6d6640 10 monclient: _renew_subs 2024-09-16T03:58:14.534 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.505+0000 7fcb4dffb640 1 -- 172.21.15.42:0/2929948969 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcb54004220 con 0x7fcb580a1330 2024-09-16T03:58:14.534 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.505+0000 7fcb5f6d6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:58:14.534 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.505+0000 7fcb5f6d6640 1 -- 172.21.15.42:0/2929948969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb58071ad0 con 0x7fcb580a1330 2024-09-16T03:58:14.534 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.505+0000 7fcb4dffb640 10 monclient: handle_config config(0 keys) 2024-09-16T03:58:14.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.505+0000 7fcb4dffb640 1 -- 172.21.15.42:0/2929948969 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb540b4040 con 0x7fcb580a1330 2024-09-16T03:58:14.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.505+0000 7fcb4dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:58:14.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.505+0000 7fcb4dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:58:14.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.505+0000 7fcb4dffb640 10 monclient: dump: 2024-09-16T03:58:14.535 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:58:14.535 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:58:14.535 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:14.535 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:14.535 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:58:14.535 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:14.535 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:58:14.535 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:58:14.536 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:58:14.536 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:14.536 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.505+0000 7fcb4f7fe640 4 set_mon_vals no callback set 2024-09-16T03:58:14.536 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.505+0000 7fcb5f6d6640 10 client.824961.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:58:14.536 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.505+0000 7fcb5f6d6640 10 monclient: _renew_subs 2024-09-16T03:58:14.536 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.505+0000 7fcb5f6d6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:58:14.536 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.505+0000 7fcb5f6d6640 1 -- 172.21.15.42:0/2929948969 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb5811cea0 con 0x7fcb580a1330 2024-09-16T03:58:14.536 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.505+0000 7fcb5f6d6640 1 librados: init done 2024-09-16T03:58:14.536 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.505+0000 7fcb5f6d6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:58:14.536 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.505+0000 7fcb5f6d6640 1 --2- 172.21.15.42:0/2929948969 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fcb5813a0f0 0x7fcb5813a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:14.536 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.505+0000 7fcb5f6d6640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:14.536 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.505+0000 7fcb5f6d6640 1 -- 172.21.15.42:0/2929948969 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fcb58073a30 con 0x7fcb5813a0f0 2024-09-16T03:58:14.537 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.505+0000 7fcb5cc71640 1 --2- 172.21.15.42:0/2929948969 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fcb5813a0f0 0x7fcb5813a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:14.537 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.505+0000 7fcb5cc71640 10 monclient: get_auth_request con 0x7fcb5813a0f0 auth_method 0 2024-09-16T03:58:14.537 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.505+0000 7fcb5cc71640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:14.537 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.505+0000 7fcb5cc71640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:14.537 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.505+0000 7fcb5cc71640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:58:14.537 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.505+0000 7fcb5cc71640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:58:14.537 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.505+0000 7fcb5cc71640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:58:14.537 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.505+0000 7fcb5cc71640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:58:14.537 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.509+0000 7fcb4dffb640 1 -- 172.21.15.42:0/2929948969 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fcb54003620 con 0x7fcb580a1330 2024-09-16T03:58:14.537 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.509+0000 7fcb5cc71640 10 monclient(hunting): handle_auth_done global_id 854474 payload 824 2024-09-16T03:58:14.537 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.509+0000 7fcb5cc71640 1 -- 172.21.15.42:0/2929948969 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fcb5809f180 con 0x7fcb5813a0f0 2024-09-16T03:58:14.538 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.509+0000 7fcb4dffb640 1 --2- 172.21.15.42:0/2929948969 >> v2:172.21.15.154:6812/254845342 conn(0x7fcb3c037810 0x7fcb3c039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:14.538 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.509+0000 7fcb4dffb640 1 -- 172.21.15.42:0/2929948969 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(484..484 src has 366..484) ==== 6181+0+0 (secure 0 0 0) 0x7fcb540e5da0 con 0x7fcb580a1330 2024-09-16T03:58:14.538 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.509+0000 7fcb4dffb640 10 client.824961.objecter ms_dispatch 0x7fcb58000ba0 osd_map(484..484 src has 366..484) 2024-09-16T03:58:14.538 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.509+0000 7fcb4dffb640 3 client.824961.objecter handle_osd_map got epochs [484,484] > 0 2024-09-16T03:58:14.538 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.509+0000 7fcb5cc71640 1 --2- 172.21.15.42:0/2929948969 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fcb5813a0f0 0x7fcb5813a4d0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fcb5813b330 tx=0x7fcb4800e990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:58:14.538 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.509+0000 7fcb4dffb640 3 client.824961.objecter handle_osd_map decoding full epoch 484 2024-09-16T03:58:14.538 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.509+0000 7fcb4dffb640 20 client.824961.objecter dump_active .. 0 homeless 2024-09-16T03:58:14.538 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.509+0000 7fcb4dffb640 10 client.824961.objecter ms_handle_connect 0x7fcb5813a0f0 2024-09-16T03:58:14.538 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.509+0000 7fcb4dffb640 10 client.824961.objecter resend_mon_ops 2024-09-16T03:58:14.538 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.509+0000 7fcb4dffb640 1 -- 172.21.15.42:0/2929948969 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb48017070 con 0x7fcb5813a0f0 2024-09-16T03:58:14.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.509+0000 7fcb4dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:58:14.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.509+0000 7fcb4dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:58:14.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.509+0000 7fcb4dffb640 10 monclient: dump: 2024-09-16T03:58:14.539 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:58:14.539 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:58:14.539 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:14.539 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:14.539 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:58:14.539 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:14.540 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:58:14.540 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:58:14.540 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:58:14.540 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:14.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.509+0000 7fcb5dc73640 1 --2- 172.21.15.42:0/2929948969 >> v2:172.21.15.154:6812/254845342 conn(0x7fcb3c037810 0x7fcb3c039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:14.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.509+0000 7fcb5dc73640 10 monclient: get_auth_request con 0x7fcb3c037810 auth_method 0 2024-09-16T03:58:14.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.509+0000 7fcb4dffb640 1 -- 172.21.15.42:0/2929948969 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fcb48013610 con 0x7fcb5813a0f0 2024-09-16T03:58:14.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.509+0000 7fcb4dffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:58:14.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.509+0000 7fcb4dffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:58:14.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.509+0000 7fcb4dffb640 1 -- 172.21.15.42:0/2929948969 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fcb5813a0f0 msgr2=0x7fcb5813a4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:14.541 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.509+0000 7fcb4dffb640 1 --2- 172.21.15.42:0/2929948969 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fcb5813a0f0 0x7fcb5813a4d0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fcb5813b330 tx=0x7fcb4800e990 comp rx=0 tx=0).stop 2024-09-16T03:58:14.541 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.509+0000 7fcb4dffb640 1 -- 172.21.15.42:0/2929948969 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fcb5813a0f0 msgr2=0x7fcb5813a4d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:58:14.541 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.509+0000 7fcb4dffb640 1 --2- 172.21.15.42:0/2929948969 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fcb5813a0f0 0x7fcb5813a4d0 secure :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fcb5813b330 tx=0x7fcb4800e990 comp rx=0 tx=0).stop 2024-09-16T03:58:14.541 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.509+0000 7fcb4dffb640 10 client.824961.objecter ms_handle_connect 0x7fcb3c037810 2024-09-16T03:58:14.541 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.509+0000 7fcb5dc73640 1 --2- 172.21.15.42:0/2929948969 >> v2:172.21.15.154:6812/254845342 conn(0x7fcb3c037810 0x7fcb3c039cd0 secure :-1 s=READY pgs=3695 cs=0 l=1 rev1=1 crypto rx=0x7fcb500018c0 tx=0x7fcb50001850 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:58:14.543 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.513+0000 7fcb5f6d6640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:58:14.544 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.513+0000 7fcb5f6d6640 1 --2- 172.21.15.42:0/2929948969 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fcb5813aa10 0x7fcb58136390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:14.544 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.513+0000 7fcb5f6d6640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:14.544 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.513+0000 7fcb5f6d6640 1 -- 172.21.15.42:0/2929948969 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fcb5811c2f0 con 0x7fcb5813aa10 2024-09-16T03:58:14.544 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.513+0000 7fcb5cc71640 1 --2- 172.21.15.42:0/2929948969 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fcb5813aa10 0x7fcb58136390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:14.544 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.513+0000 7fcb5cc71640 10 monclient: get_auth_request con 0x7fcb5813aa10 auth_method 0 2024-09-16T03:58:14.544 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.513+0000 7fcb5cc71640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:14.544 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.513+0000 7fcb5cc71640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:14.544 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.513+0000 7fcb5cc71640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:58:14.545 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.513+0000 7fcb5cc71640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:58:14.545 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.513+0000 7fcb5cc71640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:58:14.545 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.513+0000 7fcb5cc71640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:58:14.545 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.513+0000 7fcb5cc71640 10 monclient(hunting): handle_auth_done global_id 854477 payload 824 2024-09-16T03:58:14.545 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.513+0000 7fcb5cc71640 1 -- 172.21.15.42:0/2929948969 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fcb5813c130 con 0x7fcb5813aa10 2024-09-16T03:58:14.545 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.513+0000 7fcb5cc71640 1 --2- 172.21.15.42:0/2929948969 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fcb5813aa10 0x7fcb58136390 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fcb5813b330 tx=0x7fcb48004740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:58:14.545 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.513+0000 7fcb4dffb640 10 client.824961.objecter ms_handle_connect 0x7fcb5813aa10 2024-09-16T03:58:14.545 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.513+0000 7fcb4dffb640 10 client.824961.objecter resend_mon_ops 2024-09-16T03:58:14.545 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.513+0000 7fcb4dffb640 1 -- 172.21.15.42:0/2929948969 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb4800f040 con 0x7fcb5813aa10 2024-09-16T03:58:14.545 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.513+0000 7fcb4dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:58:14.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.513+0000 7fcb4dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:58:14.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.513+0000 7fcb4dffb640 10 monclient: dump: 2024-09-16T03:58:14.546 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:58:14.546 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:58:14.546 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:14.546 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:14.546 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:58:14.546 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:14.546 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:58:14.547 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:58:14.547 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:58:14.547 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:14.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.517+0000 7fcb4dffb640 1 -- 172.21.15.42:0/2929948969 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4713 (secure 0 0 0) 0x7fcb48012370 con 0x7fcb5813aa10 2024-09-16T03:58:14.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.517+0000 7fcb4dffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:58:14.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.517+0000 7fcb4dffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:58:14.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.517+0000 7fcb4dffb640 1 -- 172.21.15.42:0/2929948969 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fcb5813aa10 msgr2=0x7fcb58136390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:14.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.517+0000 7fcb4dffb640 1 --2- 172.21.15.42:0/2929948969 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fcb5813aa10 0x7fcb58136390 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fcb5813b330 tx=0x7fcb48004740 comp rx=0 tx=0).stop 2024-09-16T03:58:14.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.517+0000 7fcb4dffb640 1 -- 172.21.15.42:0/2929948969 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fcb5813aa10 msgr2=0x7fcb58136390 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:58:14.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.517+0000 7fcb4dffb640 1 --2- 172.21.15.42:0/2929948969 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fcb5813aa10 0x7fcb58136390 secure :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fcb5813b330 tx=0x7fcb48004740 comp rx=0 tx=0).stop 2024-09-16T03:58:14.548 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:58:14.548 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:58:14.548 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:58:14.548 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T03:58:14.548 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 148, 2024-09-16T03:58:14.548 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 227949, 2024-09-16T03:58:14.548 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:58:14.548 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:58:14.548 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T03:58:14.548 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T03:58:14.549 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:58:14.549 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 2, 2024-09-16T03:58:14.549 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:58:14.549 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:58:14.549 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:58:14.549 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:58:14.549 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:58:14.549 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:58:14.549 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:58:14.549 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:58:14.549 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:58:14.549 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:58:14.550 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:58:14.550 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:58:14.550 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:58:14.550 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:58:14.550 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:58:14.550 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:58:14.550 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:58:14.550 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:58:14.550 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:58:14.550 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:58:14.550 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:58:14.550 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:58:14.551 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:58:14.551 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:58:14.551 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:58:14.551 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:58:14.551 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:58:14.551 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:58:14.551 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:58:14.551 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:58:14.551 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:58:14.551 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:58:14.551 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:58:14.552 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:58:14.552 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:58:14.552 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:58:14.552 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:58:14.552 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:58:14.552 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:58:14.552 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:58:14.552 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:58:14.552 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:58:14.552 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:58:14.552 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:58:14.552 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:58:14.553 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:58:14.553 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:58:14.553 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:58:14.553 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:58:14.553 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:58:14.553 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:58:14.553 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:58:14.553 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:58:14.553 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:58:14.553 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:58:14.553 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:58:14.554 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:58:14.554 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:58:14.554 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:58:14.554 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:58:14.554 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:58:14.554 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:58:14.554 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:58:14.554 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:58:14.554 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:58:14.554 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:58:14.554 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:58:14.555 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:58:14.555 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:58:14.555 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:58:14.555 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:58:14.555 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:58:14.555 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:58:14.555 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:58:14.555 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:58:14.555 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:58:14.555 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:58:14.555 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:58:14.555 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:58:14.556 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:58:14.556 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:58:14.556 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:58:14.556 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:58:14.556 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:58:14.556 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:58:14.556 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:58:14.556 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:58:14.556 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:58:14.556 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:58:14.556 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:58:14.556 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:58:14.557 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:58:14.557 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:58:14.557 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:58:14.557 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:58:14.557 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:58:14.557 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:58:14.557 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:58:14.557 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:58:14.557 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:58:14.557 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:58:14.557 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:58:14.558 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:58:14.558 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:58:14.558 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:58:14.558 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:58:14.558 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:58:14.558 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:58:14.558 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:58:14.558 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:58:14.558 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:58:14.558 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:58:14.558 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:58:14.558 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:58:14.558 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:58:14.559 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:58:14.559 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:58:14.559 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:58:14.559 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:58:14.559 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:58:14.559 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:58:14.559 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:58:14.559 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:58:14.559 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:58:14.559 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:58:14.559 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:58:14.559 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:58:14.560 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:58:14.560 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:58:14.560 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:58:14.560 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:58:14.560 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:58:14.560 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:58:14.560 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:58:14.560 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:58:14.560 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:58:14.560 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:58:14.560 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:58:14.560 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:58:14.561 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:58:14.561 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:58:14.561 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T03:58:14.561 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:58:14.561 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:58:14.561 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:58:14.561 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:58:14.561 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:58:14.561 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:58:14.561 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:58:14.561 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:58:14.561 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:58:14.562 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:58:14.562 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 2 2024-09-16T03:58:14.562 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:58:14.562 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:58:14.562 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:58:14.562 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:58:14.562 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:58:14.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.517+0000 7fcb5f6d6640 10 librados: watch_flush enter 2024-09-16T03:58:14.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.517+0000 7fcb5f6d6640 10 librados: watch_flush exit 2024-09-16T03:58:14.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.517+0000 7fcb5f6d6640 20 client.824961.objecter shutdown clearing up homeless session... 2024-09-16T03:58:14.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.517+0000 7fcb5f6d6640 10 client.824961.objecter successfully canceled tick 2024-09-16T03:58:14.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.517+0000 7fcb5f6d6640 1 -- 172.21.15.42:0/2929948969 >> v2:172.21.15.154:6812/254845342 conn(0x7fcb3c037810 msgr2=0x7fcb3c039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:14.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.517+0000 7fcb5f6d6640 1 --2- 172.21.15.42:0/2929948969 >> v2:172.21.15.154:6812/254845342 conn(0x7fcb3c037810 0x7fcb3c039cd0 secure :-1 s=READY pgs=3695 cs=0 l=1 rev1=1 crypto rx=0x7fcb500018c0 tx=0x7fcb50001850 comp rx=0 tx=0).stop 2024-09-16T03:58:14.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.517+0000 7fcb5f6d6640 10 monclient: shutdown 2024-09-16T03:58:14.565 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.517+0000 7fcb5f6d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:58:14.565 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.517+0000 7fcb5f6d6640 1 -- 172.21.15.42:0/2929948969 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcb580a1330 msgr2=0x7fcb581cb3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:14.565 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.517+0000 7fcb5f6d6640 1 --2- 172.21.15.42:0/2929948969 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcb580a1330 0x7fcb581cb3c0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fcb5401b750 tx=0x7fcb5401b8e0 comp rx=0 tx=0).stop 2024-09-16T03:58:14.565 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.517+0000 7fcb5dc73640 1 -- 172.21.15.42:0/2929948969 reap_dead start 2024-09-16T03:58:14.565 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.517+0000 7fcb5dc73640 1 -- 172.21.15.42:0/2929948969 reap_dead start 2024-09-16T03:58:14.565 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.517+0000 7fcb5f6d6640 1 -- 172.21.15.42:0/2929948969 shutdown_connections 2024-09-16T03:58:14.565 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.517+0000 7fcb5f6d6640 1 -- 172.21.15.42:0/2929948969 >> 172.21.15.42:0/2929948969 conn(0x7fcb580a2420 msgr2=0x7fcb580701d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:58:14.565 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.517+0000 7fcb5f6d6640 1 -- 172.21.15.42:0/2929948969 shutdown_connections 2024-09-16T03:58:14.565 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.517+0000 7fcb5f6d6640 1 -- 172.21.15.42:0/2929948969 wait complete. 2024-09-16T03:58:14.565 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.517+0000 7fcb5f6d6640 1 librados: shutdown 2024-09-16T03:58:14.566 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:58:14.566 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T03:58:14.566 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T03:58:14.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.681+0000 7f0d35581640 10 monclient: get_monmap_and_config 2024-09-16T03:58:14.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.681+0000 7f0d35581640 10 monclient: build_initial_monmap 2024-09-16T03:58:14.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.681+0000 7f0d35581640 10 monclient: monmap: 2024-09-16T03:58:14.689 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:58:14.689 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:58:14.690 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:58:14.687860+0000 2024-09-16T03:58:14.690 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:58:14.687860+0000 2024-09-16T03:58:14.690 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:58:14.690 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:14.690 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:58:14.690 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:58:14.690 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:58:14.690 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:58:14.690 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:58:14.690 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:14.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.685+0000 7f0d35581640 1 Processor -- start 2024-09-16T03:58:14.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.685+0000 7f0d35581640 1 -- start start 2024-09-16T03:58:14.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.685+0000 7f0d35581640 10 monclient: init 2024-09-16T03:58:14.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.685+0000 7f0d35581640 10 monclient: _reopen_session rank -1 2024-09-16T03:58:14.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.685+0000 7f0d35581640 10 monclient: _add_conns ranks=[4,2,3,0,1] 2024-09-16T03:58:14.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.685+0000 7f0d35581640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f0d300a1330 0x7f0d300a1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:14.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.685+0000 7f0d35581640 10 monclient(hunting): picked mon.noname-c con 0x7f0d300a1330 addr v2:172.21.15.154:3300/0 2024-09-16T03:58:14.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.685+0000 7f0d35581640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0d3006dac0 0x7f0d300a1c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:14.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.685+0000 7f0d35581640 10 monclient(hunting): picked mon.noname-b con 0x7f0d3006dac0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:58:14.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.685+0000 7f0d35581640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f0d30135100 0x7f0d30135550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:14.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.685+0000 7f0d35581640 10 monclient(hunting): picked mon.noname-a con 0x7f0d30135100 addr v2:172.21.15.42:3300/0 2024-09-16T03:58:14.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.685+0000 7f0d35581640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:14.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.685+0000 7f0d35581640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f0d3009c7d0 con 0x7f0d30135100 2024-09-16T03:58:14.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.685+0000 7f0d35581640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:14.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.685+0000 7f0d35581640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f0d3009c090 con 0x7f0d300a1330 2024-09-16T03:58:14.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.685+0000 7f0d35581640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:14.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.685+0000 7f0d35581640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f0d3009f610 con 0x7f0d3006dac0 2024-09-16T03:58:14.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.685+0000 7f0d35581640 10 monclient(hunting): _renew_subs 2024-09-16T03:58:14.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.685+0000 7f0d35581640 10 monclient(hunting): authenticate will time out at 1941.657264s 2024-09-16T03:58:14.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.685+0000 7f0d2f7fe640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f0d30135100 0x7f0d30135550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:14.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.685+0000 7f0d2f7fe640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f0d30135100 0x7f0d30135550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:50308/0 (socket says 172.21.15.42:50308) 2024-09-16T03:58:14.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.685+0000 7f0d2f7fe640 1 -- 172.21.15.42:0/1154051864 learned_addr learned my addr 172.21.15.42:0/1154051864 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:58:14.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.685+0000 7f0d2e7fc640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0d3006dac0 0x7f0d300a1c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:14.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.685+0000 7f0d2f7fe640 10 monclient(hunting): get_auth_request con 0x7f0d30135100 auth_method 0 2024-09-16T03:58:14.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.685+0000 7f0d2f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:14.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.685+0000 7f0d2f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:14.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.685+0000 7f0d2f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:58:14.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.685+0000 7f0d2e7fc640 10 monclient(hunting): get_auth_request con 0x7f0d3006dac0 auth_method 0 2024-09-16T03:58:14.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.685+0000 7f0d2e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:14.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.685+0000 7f0d2e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:14.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.685+0000 7f0d2e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:58:14.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.685+0000 7f0d2f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:58:14.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.685+0000 7f0d2f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:58:14.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.685+0000 7f0d2f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:58:14.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.685+0000 7f0d2e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:58:14.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.685+0000 7f0d2e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:58:14.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.685+0000 7f0d2e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:58:14.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.685+0000 7f0d2effd640 1 --2- 172.21.15.42:0/1154051864 >> v2:172.21.15.154:3300/0 conn(0x7f0d300a1330 0x7f0d300a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:14.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.685+0000 7f0d2effd640 10 monclient(hunting): get_auth_request con 0x7f0d300a1330 auth_method 0 2024-09-16T03:58:14.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.685+0000 7f0d2effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:14.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.685+0000 7f0d2effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:14.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.685+0000 7f0d2effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:58:14.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.685+0000 7f0d2e7fc640 10 monclient(hunting): handle_auth_done global_id 854480 payload 274 2024-09-16T03:58:14.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.685+0000 7f0d2e7fc640 1 -- 172.21.15.42:0/1154051864 >> v2:172.21.15.154:3300/0 conn(0x7f0d300a1330 msgr2=0x7f0d300a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:58:14.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.685+0000 7f0d2e7fc640 1 --2- 172.21.15.42:0/1154051864 >> v2:172.21.15.154:3300/0 conn(0x7f0d300a1330 0x7f0d300a1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:14.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.685+0000 7f0d2e7fc640 1 -- 172.21.15.42:0/1154051864 >> v2:172.21.15.42:3300/0 conn(0x7f0d30135100 msgr2=0x7f0d30135550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:58:14.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.685+0000 7f0d2e7fc640 1 --2- 172.21.15.42:0/1154051864 >> v2:172.21.15.42:3300/0 conn(0x7f0d30135100 0x7f0d30135550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:14.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.685+0000 7f0d2e7fc640 10 monclient: _finish_hunting 0 2024-09-16T03:58:14.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.685+0000 7f0d2e7fc640 1 monclient: found mon.noname-b 2024-09-16T03:58:14.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.685+0000 7f0d2e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:58:14.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.685+0000 7f0d2e7fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T03:58:14.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.685+0000 7f0d2e7fc640 1 -- 172.21.15.42:0/1154051864 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d3009f8f0 con 0x7f0d3006dac0 2024-09-16T03:58:14.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.685+0000 7f0d2f7fe640 1 --2- 172.21.15.42:0/1154051864 >> v2:172.21.15.42:3300/0 conn(0x7f0d30135100 0x7f0d30135550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:58:14.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.689+0000 7f0d2e7fc640 1 --2- 172.21.15.42:0/1154051864 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0d3006dac0 0x7f0d300a1c50 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f0d3006add0 tx=0x7f0d2001c460 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5c33593aed82613a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:58:14.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.689+0000 7f0d2dffb640 1 -- 172.21.15.42:0/1154051864 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d200b0070 con 0x7f0d3006dac0 2024-09-16T03:58:14.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.689+0000 7f0d2dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:58:14.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.689+0000 7f0d2dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:58:14.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.689+0000 7f0d2dffb640 10 monclient: dump: 2024-09-16T03:58:14.698 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:58:14.698 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:58:14.698 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:14.698 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:14.698 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:58:14.698 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:14.698 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:58:14.698 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:58:14.698 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:58:14.698 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:14.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.689+0000 7f0d2dffb640 10 monclient: _finish_auth 0 2024-09-16T03:58:14.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.689+0000 7f0d2dffb640 10 monclient: _check_auth_tickets 2024-09-16T03:58:14.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.689+0000 7f0d2dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:58:14.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.689+0000 7f0d2dffb640 1 -- 172.21.15.42:0/1154051864 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0d200ab600 con 0x7f0d3006dac0 2024-09-16T03:58:14.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.689+0000 7f0d35581640 5 monclient: authenticate success, global_id 854480 2024-09-16T03:58:14.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.689+0000 7f0d35581640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:58:14.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.689+0000 7f0d2dffb640 10 monclient: handle_config config(0 keys) 2024-09-16T03:58:14.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.689+0000 7f0d35581640 10 monclient: get_monmap_and_config success 2024-09-16T03:58:14.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.689+0000 7f0d35581640 4 set_mon_vals no callback set 2024-09-16T03:58:14.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.689+0000 7f0d2dffb640 1 -- 172.21.15.42:0/1154051864 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d200ab9d0 con 0x7f0d3006dac0 2024-09-16T03:58:14.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.689+0000 7f0d2dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:58:14.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.689+0000 7f0d2dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:58:14.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.689+0000 7f0d2dffb640 10 monclient: dump: 2024-09-16T03:58:14.699 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:58:14.699 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:58:14.699 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:14.699 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:14.699 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:58:14.700 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:14.700 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:58:14.700 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:58:14.700 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:58:14.700 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:14.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.689+0000 7f0d35581640 10 monclient: shutdown 2024-09-16T03:58:14.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.689+0000 7f0d35581640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:58:14.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.689+0000 7f0d35581640 1 -- 172.21.15.42:0/1154051864 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0d3006dac0 msgr2=0x7f0d300a1c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:14.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.689+0000 7f0d35581640 1 --2- 172.21.15.42:0/1154051864 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0d3006dac0 0x7f0d300a1c50 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f0d3006add0 tx=0x7f0d2001c460 comp rx=0 tx=0).stop 2024-09-16T03:58:14.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.689+0000 7f0d35581640 1 -- 172.21.15.42:0/1154051864 shutdown_connections 2024-09-16T03:58:14.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.689+0000 7f0d35581640 1 --2- 172.21.15.42:0/1154051864 >> v2:172.21.15.42:3300/0 conn(0x7f0d30135100 0x7f0d30135550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:14.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.689+0000 7f0d35581640 1 --2- 172.21.15.42:0/1154051864 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0d3006dac0 0x7f0d300a1c50 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:14.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.689+0000 7f0d35581640 1 --2- 172.21.15.42:0/1154051864 >> v2:172.21.15.154:3300/0 conn(0x7f0d300a1330 0x7f0d300a1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:14.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.689+0000 7f0d35581640 1 -- 172.21.15.42:0/1154051864 >> 172.21.15.42:0/1154051864 conn(0x7f0d300a2420 msgr2=0x7f0d300a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:58:14.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.689+0000 7f0d35581640 1 -- 172.21.15.42:0/1154051864 shutdown_connections 2024-09-16T03:58:14.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.689+0000 7f0d35581640 1 -- 172.21.15.42:0/1154051864 wait complete. 2024-09-16T03:58:14.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.689+0000 7f0d35581640 10 monclient: build_initial_monmap 2024-09-16T03:58:14.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.689+0000 7f0d35581640 10 monclient: monmap: 2024-09-16T03:58:14.706 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:58:14.706 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:58:14.706 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:58:14.706 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:58:14.706 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:58:14.706 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:14.706 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:58:14.707 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:58:14.707 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:58:14.707 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:14.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.689+0000 7f0d35581640 1 librados: starting msgr at 2024-09-16T03:58:14.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.689+0000 7f0d35581640 1 librados: starting objecter 2024-09-16T03:58:14.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.689+0000 7f0d35581640 1 Processor -- start 2024-09-16T03:58:14.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.689+0000 7f0d35581640 1 -- start start 2024-09-16T03:58:14.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.689+0000 7f0d35581640 1 librados: setting wanted keys 2024-09-16T03:58:14.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.689+0000 7f0d35581640 1 librados: calling monclient init 2024-09-16T03:58:14.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.689+0000 7f0d35581640 10 monclient: init 2024-09-16T03:58:14.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.689+0000 7f0d35581640 10 monclient: _reopen_session rank -1 2024-09-16T03:58:14.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.689+0000 7f0d35581640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T03:58:14.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.689+0000 7f0d35581640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0d300a1330 0x7f0d30133650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:14.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.689+0000 7f0d35581640 10 monclient(hunting): picked mon.noname-a con 0x7f0d300a1330 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:58:14.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.689+0000 7f0d35581640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0d30135100 0x7f0d30133b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:14.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.689+0000 7f0d35581640 10 monclient(hunting): picked mon.noname-b con 0x7f0d30135100 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:58:14.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.689+0000 7f0d35581640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0d301cf3c0 0x7f0d3011d100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:14.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.689+0000 7f0d35581640 10 monclient(hunting): picked mon.noname-c con 0x7f0d301cf3c0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:58:14.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.689+0000 7f0d35581640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:14.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.689+0000 7f0d35581640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f0d3009eef0 con 0x7f0d300a1330 2024-09-16T03:58:14.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.689+0000 7f0d35581640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:14.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.689+0000 7f0d35581640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f0d3009ed70 con 0x7f0d30135100 2024-09-16T03:58:14.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.689+0000 7f0d35581640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:14.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.689+0000 7f0d35581640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f0d3013eb70 con 0x7f0d301cf3c0 2024-09-16T03:58:14.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.689+0000 7f0d35581640 10 monclient(hunting): _renew_subs 2024-09-16T03:58:14.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.689+0000 7f0d35581640 10 monclient(hunting): authenticate will time out at 1941.662604s 2024-09-16T03:58:14.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.689+0000 7f0d2effd640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0d300a1330 0x7f0d30133650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:14.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.689+0000 7f0d2f7fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0d301cf3c0 0x7f0d3011d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:14.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.689+0000 7f0d2effd640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0d300a1330 0x7f0d30133650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:50310/0 (socket says 172.21.15.42:50310) 2024-09-16T03:58:14.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.689+0000 7f0d2effd640 1 -- 172.21.15.42:0/4185497516 learned_addr learned my addr 172.21.15.42:0/4185497516 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:58:14.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.689+0000 7f0d2effd640 10 monclient(hunting): get_auth_request con 0x7f0d300a1330 auth_method 0 2024-09-16T03:58:14.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.689+0000 7f0d2effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:14.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.689+0000 7f0d2effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:14.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.689+0000 7f0d2effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:58:14.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.689+0000 7f0d2f7fe640 10 monclient(hunting): get_auth_request con 0x7f0d301cf3c0 auth_method 0 2024-09-16T03:58:14.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.689+0000 7f0d2f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:14.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.689+0000 7f0d2f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:14.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.689+0000 7f0d2f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:58:14.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.689+0000 7f0d2f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:58:14.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.689+0000 7f0d2f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:58:14.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.689+0000 7f0d2f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:58:14.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.689+0000 7f0d2effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:58:14.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.689+0000 7f0d2effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:58:14.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.689+0000 7f0d2effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:58:14.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.689+0000 7f0d2e7fc640 1 --2- 172.21.15.42:0/4185497516 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0d30135100 0x7f0d30133b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:14.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.689+0000 7f0d2f7fe640 10 monclient(hunting): handle_auth_done global_id 854483 payload 824 2024-09-16T03:58:14.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.689+0000 7f0d2f7fe640 1 -- 172.21.15.42:0/4185497516 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0d30135100 msgr2=0x7f0d30133b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:14.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.689+0000 7f0d2f7fe640 1 --2- 172.21.15.42:0/4185497516 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0d30135100 0x7f0d30133b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:14.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.689+0000 7f0d2f7fe640 1 -- 172.21.15.42:0/4185497516 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0d300a1330 msgr2=0x7f0d30133650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:14.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.689+0000 7f0d2f7fe640 1 --2- 172.21.15.42:0/4185497516 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0d300a1330 0x7f0d30133650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:14.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.689+0000 7f0d2f7fe640 10 monclient: _finish_hunting 0 2024-09-16T03:58:14.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.689+0000 7f0d2f7fe640 1 monclient: found mon.noname-c 2024-09-16T03:58:14.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.689+0000 7f0d2f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:58:14.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.689+0000 7f0d2f7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:58:14.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.689+0000 7f0d2f7fe640 1 -- 172.21.15.42:0/4185497516 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d30120d30 con 0x7f0d301cf3c0 2024-09-16T03:58:14.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.689+0000 7f0d2e7fc640 10 monclient: get_auth_request con 0x7f0d30135100 auth_method 0 2024-09-16T03:58:14.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.689+0000 7f0d2e7fc640 1 --2- 172.21.15.42:0/4185497516 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0d30135100 0x7f0d30133b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T03:58:14.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.689+0000 7f0d2effd640 1 --2- 172.21.15.42:0/4185497516 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0d300a1330 0x7f0d30133650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:58:14.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.689+0000 7f0d2f7fe640 1 --2- 172.21.15.42:0/4185497516 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0d301cf3c0 0x7f0d3011d100 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f0d2800efc0 tx=0x7f0d2800c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:58:14.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.689+0000 7f0d177fe640 10 client.?.objecter ms_handle_connect 0x7f0d301cf3c0 2024-09-16T03:58:14.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.689+0000 7f0d177fe640 10 client.?.objecter resend_mon_ops 2024-09-16T03:58:14.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.689+0000 7f0d177fe640 1 -- 172.21.15.42:0/4185497516 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d2800cb20 con 0x7f0d301cf3c0 2024-09-16T03:58:14.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.689+0000 7f0d177fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:58:14.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.689+0000 7f0d177fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:58:14.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.689+0000 7f0d177fe640 10 monclient: dump: 2024-09-16T03:58:14.711 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:58:14.711 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:58:14.711 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:14.711 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:14.711 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:58:14.711 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:14.712 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:58:14.712 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:58:14.712 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:58:14.712 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:14.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.689+0000 7f0d177fe640 10 monclient: _finish_auth 0 2024-09-16T03:58:14.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.689+0000 7f0d177fe640 10 monclient: _check_auth_tickets 2024-09-16T03:58:14.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.689+0000 7f0d177fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:58:14.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.689+0000 7f0d177fe640 1 -- 172.21.15.42:0/4185497516 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0d2800cce0 con 0x7f0d301cf3c0 2024-09-16T03:58:14.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.689+0000 7f0d177fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:58:14.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.689+0000 7f0d177fe640 1 -- 172.21.15.42:0/4185497516 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d28003be0 con 0x7f0d301cf3c0 2024-09-16T03:58:14.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.689+0000 7f0d177fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:58:14.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.689+0000 7f0d177fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:58:14.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.689+0000 7f0d177fe640 10 monclient: dump: 2024-09-16T03:58:14.712 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:58:14.712 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:58:14.713 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:14.713 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:14.713 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:58:14.713 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:14.713 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:58:14.713 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:58:14.713 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:58:14.713 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:14.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.689+0000 7f0d35581640 5 monclient: authenticate success, global_id 854483 2024-09-16T03:58:14.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.689+0000 7f0d35581640 10 monclient: _renew_subs 2024-09-16T03:58:14.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.689+0000 7f0d35581640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:58:14.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.689+0000 7f0d35581640 1 -- 172.21.15.42:0/4185497516 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d30121010 con 0x7f0d301cf3c0 2024-09-16T03:58:14.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.689+0000 7f0d2d7fa640 4 set_mon_vals no callback set 2024-09-16T03:58:14.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.689+0000 7f0d35581640 10 client.854483.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:58:14.714 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.689+0000 7f0d35581640 10 monclient: _renew_subs 2024-09-16T03:58:14.714 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.689+0000 7f0d35581640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:58:14.714 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.689+0000 7f0d35581640 1 -- 172.21.15.42:0/4185497516 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d300a3c50 con 0x7f0d301cf3c0 2024-09-16T03:58:14.714 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.689+0000 7f0d35581640 1 librados: init done 2024-09-16T03:58:14.714 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.693+0000 7f0d35581640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:58:14.714 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.693+0000 7f0d35581640 1 --2- 172.21.15.42:0/4185497516 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0d3002c650 0x7f0d3002ca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:14.714 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.693+0000 7f0d35581640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:14.714 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.693+0000 7f0d35581640 1 -- 172.21.15.42:0/4185497516 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f0d301340d0 con 0x7f0d3002c650 2024-09-16T03:58:14.714 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.693+0000 7f0d177fe640 1 -- 172.21.15.42:0/4185497516 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f0d2801b030 con 0x7f0d301cf3c0 2024-09-16T03:58:14.714 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.693+0000 7f0d177fe640 1 --2- 172.21.15.42:0/4185497516 >> v2:172.21.15.154:6812/254845342 conn(0x7f0d1c037810 0x7f0d1c039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:14.714 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.693+0000 7f0d2effd640 1 --2- 172.21.15.42:0/4185497516 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0d3002c650 0x7f0d3002ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:14.714 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.693+0000 7f0d177fe640 1 -- 172.21.15.42:0/4185497516 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(485..485 src has 366..485) ==== 6560+0+0 (secure 0 0 0) 0x7f0d280494f0 con 0x7f0d301cf3c0 2024-09-16T03:58:14.714 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.693+0000 7f0d2effd640 10 monclient: get_auth_request con 0x7f0d3002c650 auth_method 0 2024-09-16T03:58:14.714 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.693+0000 7f0d177fe640 10 client.854483.objecter ms_dispatch 0x7f0d30000ba0 osd_map(485..485 src has 366..485) 2024-09-16T03:58:14.714 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.693+0000 7f0d2effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:14.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.693+0000 7f0d2effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:14.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.693+0000 7f0d177fe640 3 client.854483.objecter handle_osd_map got epochs [485,485] > 0 2024-09-16T03:58:14.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.693+0000 7f0d2effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:58:14.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.693+0000 7f0d177fe640 3 client.854483.objecter handle_osd_map decoding full epoch 485 2024-09-16T03:58:14.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.693+0000 7f0d177fe640 20 client.854483.objecter dump_active .. 0 homeless 2024-09-16T03:58:14.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.693+0000 7f0d2effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:58:14.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.693+0000 7f0d2effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:58:14.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.693+0000 7f0d2effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:58:14.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.693+0000 7f0d2e7fc640 1 --2- 172.21.15.42:0/4185497516 >> v2:172.21.15.154:6812/254845342 conn(0x7f0d1c037810 0x7f0d1c039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:14.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.693+0000 7f0d2e7fc640 10 monclient: get_auth_request con 0x7f0d1c037810 auth_method 0 2024-09-16T03:58:14.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.693+0000 7f0d2effd640 10 monclient(hunting): handle_auth_done global_id 904150 payload 824 2024-09-16T03:58:14.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.693+0000 7f0d2effd640 1 -- 172.21.15.42:0/4185497516 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0d30135550 con 0x7f0d3002c650 2024-09-16T03:58:14.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.693+0000 7f0d2effd640 1 --2- 172.21.15.42:0/4185497516 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0d3002c650 0x7f0d3002ca30 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f0d24010cb0 tx=0x7f0d2400e5a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:58:14.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.693+0000 7f0d177fe640 10 client.854483.objecter ms_handle_connect 0x7f0d3002c650 2024-09-16T03:58:14.716 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.693+0000 7f0d177fe640 10 client.854483.objecter resend_mon_ops 2024-09-16T03:58:14.716 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.693+0000 7f0d177fe640 10 client.854483.objecter ms_handle_connect 0x7f0d1c037810 2024-09-16T03:58:14.716 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.693+0000 7f0d2e7fc640 1 --2- 172.21.15.42:0/4185497516 >> v2:172.21.15.154:6812/254845342 conn(0x7f0d1c037810 0x7f0d1c039cd0 secure :-1 s=READY pgs=3696 cs=0 l=1 rev1=1 crypto rx=0x7f0d200ad000 tx=0x7f0d20008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:58:14.716 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.693+0000 7f0d177fe640 1 -- 172.21.15.42:0/4185497516 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d24018070 con 0x7f0d3002c650 2024-09-16T03:58:14.716 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.693+0000 7f0d177fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:58:14.716 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.693+0000 7f0d177fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:58:14.716 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.693+0000 7f0d177fe640 10 monclient: dump: 2024-09-16T03:58:14.716 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:58:14.716 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:58:14.716 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:14.716 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:14.716 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:58:14.716 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:14.716 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:58:14.716 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:58:14.717 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:58:14.717 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:14.717 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.693+0000 7f0d177fe640 1 -- 172.21.15.42:0/4185497516 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f0d2401d450 con 0x7f0d3002c650 2024-09-16T03:58:14.717 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.693+0000 7f0d177fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:58:14.717 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.693+0000 7f0d177fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:58:14.717 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.693+0000 7f0d177fe640 1 -- 172.21.15.42:0/4185497516 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0d3002c650 msgr2=0x7f0d3002ca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:14.717 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.693+0000 7f0d177fe640 1 --2- 172.21.15.42:0/4185497516 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0d3002c650 0x7f0d3002ca30 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f0d24010cb0 tx=0x7f0d2400e5a0 comp rx=0 tx=0).stop 2024-09-16T03:58:14.717 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.693+0000 7f0d177fe640 1 -- 172.21.15.42:0/4185497516 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0d3002c650 msgr2=0x7f0d3002ca30 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:58:14.717 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.693+0000 7f0d177fe640 1 --2- 172.21.15.42:0/4185497516 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0d3002c650 0x7f0d3002ca30 secure :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f0d24010cb0 tx=0x7f0d2400e5a0 comp rx=0 tx=0).stop 2024-09-16T03:58:14.718 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.701+0000 7f0d35581640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:58:14.719 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.701+0000 7f0d35581640 1 --2- 172.21.15.42:0/4185497516 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0d3002cf70 0x7f0d3002d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:14.719 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.701+0000 7f0d35581640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:14.719 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.701+0000 7f0d35581640 1 -- 172.21.15.42:0/4185497516 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f0d300741a0 con 0x7f0d3002cf70 2024-09-16T03:58:14.719 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.701+0000 7f0d2effd640 1 --2- 172.21.15.42:0/4185497516 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0d3002cf70 0x7f0d3002d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:14.719 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.701+0000 7f0d2effd640 10 monclient: get_auth_request con 0x7f0d3002cf70 auth_method 0 2024-09-16T03:58:14.719 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.701+0000 7f0d2effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:14.719 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.701+0000 7f0d2effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:14.719 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.701+0000 7f0d2effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:58:14.719 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.701+0000 7f0d2effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:58:14.719 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.701+0000 7f0d2effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:58:14.719 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.701+0000 7f0d2effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:58:14.719 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.701+0000 7f0d2effd640 10 monclient(hunting): handle_auth_done global_id 904153 payload 824 2024-09-16T03:58:14.719 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.701+0000 7f0d2effd640 1 -- 172.21.15.42:0/4185497516 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f0d3011f600 con 0x7f0d3002cf70 2024-09-16T03:58:14.720 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.701+0000 7f0d2effd640 1 --2- 172.21.15.42:0/4185497516 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0d3002cf70 0x7f0d3002d350 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f0d24010cb0 tx=0x7f0d240046d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:58:14.720 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.701+0000 7f0d177fe640 10 client.854483.objecter ms_handle_connect 0x7f0d3002cf70 2024-09-16T03:58:14.720 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.701+0000 7f0d177fe640 10 client.854483.objecter resend_mon_ops 2024-09-16T03:58:14.720 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.701+0000 7f0d177fe640 1 -- 172.21.15.42:0/4185497516 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d24014990 con 0x7f0d3002cf70 2024-09-16T03:58:14.720 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.701+0000 7f0d177fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:58:14.720 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.701+0000 7f0d177fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:58:14.720 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.701+0000 7f0d177fe640 10 monclient: dump: 2024-09-16T03:58:14.720 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:58:14.720 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:58:14.720 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:14.720 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:14.720 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:58:14.720 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:14.720 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:58:14.721 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:58:14.721 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:58:14.721 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:14.721 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.701+0000 7f0d177fe640 1 -- 172.21.15.42:0/4185497516 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4545 (secure 0 0 0) 0x7f0d24014990 con 0x7f0d3002cf70 2024-09-16T03:58:14.721 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.701+0000 7f0d177fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:58:14.721 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.701+0000 7f0d177fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:58:14.721 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.701+0000 7f0d177fe640 1 -- 172.21.15.42:0/4185497516 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0d3002cf70 msgr2=0x7f0d3002d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:14.721 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.701+0000 7f0d177fe640 1 --2- 172.21.15.42:0/4185497516 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0d3002cf70 0x7f0d3002d350 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f0d24010cb0 tx=0x7f0d240046d0 comp rx=0 tx=0).stop 2024-09-16T03:58:14.721 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.701+0000 7f0d177fe640 1 -- 172.21.15.42:0/4185497516 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0d3002cf70 msgr2=0x7f0d3002d350 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:58:14.721 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.701+0000 7f0d177fe640 1 --2- 172.21.15.42:0/4185497516 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0d3002cf70 0x7f0d3002d350 secure :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f0d24010cb0 tx=0x7f0d240046d0 comp rx=0 tx=0).stop 2024-09-16T03:58:14.721 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:58:14.722 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:58:14.722 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:58:14.722 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T03:58:14.722 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 148, 2024-09-16T03:58:14.722 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 22883, 2024-09-16T03:58:14.722 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:58:14.722 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:58:14.722 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T03:58:14.722 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T03:58:14.722 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:58:14.722 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 2, 2024-09-16T03:58:14.722 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:58:14.722 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:58:14.723 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:58:14.723 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:58:14.723 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:58:14.723 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:58:14.723 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:58:14.723 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:58:14.723 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:58:14.723 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:58:14.723 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:58:14.723 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:58:14.723 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:58:14.723 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:58:14.723 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:58:14.723 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:58:14.723 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:58:14.724 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:58:14.724 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:58:14.724 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:58:14.724 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:58:14.724 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:58:14.724 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:58:14.724 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:58:14.724 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:58:14.724 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:58:14.724 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:58:14.724 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:58:14.724 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:58:14.724 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:58:14.725 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:58:14.725 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:58:14.725 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:58:14.725 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:58:14.725 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:58:14.725 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:58:14.725 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:58:14.725 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:58:14.725 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:58:14.725 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:58:14.725 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:58:14.725 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:58:14.725 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:58:14.725 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:58:14.726 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:58:14.726 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:58:14.726 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:58:14.726 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:58:14.726 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:58:14.726 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:58:14.726 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:58:14.726 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:58:14.726 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:58:14.726 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:58:14.726 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:58:14.726 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:58:14.726 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:58:14.726 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:58:14.726 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:58:14.727 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:58:14.727 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:58:14.727 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:58:14.727 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:58:14.727 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:58:14.727 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:58:14.727 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:58:14.727 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:58:14.727 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:58:14.727 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:58:14.727 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:58:14.727 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:58:14.727 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:58:14.727 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:58:14.727 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:58:14.728 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:58:14.728 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:58:14.728 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:58:14.728 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:58:14.728 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:58:14.728 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:58:14.728 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:58:14.728 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:58:14.728 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:58:14.728 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:58:14.728 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:58:14.728 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:58:14.728 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:58:14.729 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:58:14.729 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:58:14.729 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:58:14.729 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:58:14.729 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:58:14.729 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:58:14.729 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:58:14.729 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:58:14.729 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:58:14.729 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:58:14.729 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:58:14.729 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:58:14.729 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:58:14.730 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:58:14.730 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:58:14.730 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:58:14.730 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:58:14.730 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:58:14.730 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:58:14.730 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:58:14.730 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:58:14.730 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:58:14.730 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:58:14.730 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:58:14.730 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:58:14.730 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:58:14.730 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:58:14.731 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:58:14.731 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:58:14.731 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:58:14.731 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:58:14.731 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:58:14.731 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:58:14.731 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:58:14.731 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:58:14.731 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:58:14.731 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:58:14.731 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:58:14.731 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:58:14.731 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:58:14.731 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:58:14.732 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:58:14.732 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:58:14.732 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:58:14.732 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:58:14.732 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:58:14.732 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:58:14.732 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:58:14.732 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:58:14.732 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:58:14.732 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:58:14.732 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:58:14.732 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:58:14.732 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:58:14.732 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:58:14.733 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:58:14.733 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:58:14.733 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:58:14.733 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:58:14.736 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:58:14.736 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:58:14.737 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:58:14.737 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:58:14.737 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:58:14.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.701+0000 7f0d35581640 10 librados: watch_flush enter 2024-09-16T03:58:14.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.701+0000 7f0d35581640 10 librados: watch_flush exit 2024-09-16T03:58:14.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.701+0000 7f0d35581640 20 client.854483.objecter shutdown clearing up homeless session... 2024-09-16T03:58:14.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.701+0000 7f0d35581640 10 client.854483.objecter successfully canceled tick 2024-09-16T03:58:14.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.701+0000 7f0d35581640 1 -- 172.21.15.42:0/4185497516 >> v2:172.21.15.154:6812/254845342 conn(0x7f0d1c037810 msgr2=0x7f0d1c039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:14.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.701+0000 7f0d35581640 1 --2- 172.21.15.42:0/4185497516 >> v2:172.21.15.154:6812/254845342 conn(0x7f0d1c037810 0x7f0d1c039cd0 secure :-1 s=READY pgs=3696 cs=0 l=1 rev1=1 crypto rx=0x7f0d200ad000 tx=0x7f0d20008040 comp rx=0 tx=0).stop 2024-09-16T03:58:14.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.701+0000 7f0d35581640 10 monclient: shutdown 2024-09-16T03:58:14.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.701+0000 7f0d35581640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:58:14.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.701+0000 7f0d35581640 1 -- 172.21.15.42:0/4185497516 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0d301cf3c0 msgr2=0x7f0d3011d100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:14.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.701+0000 7f0d35581640 1 --2- 172.21.15.42:0/4185497516 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0d301cf3c0 0x7f0d3011d100 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f0d2800efc0 tx=0x7f0d2800c490 comp rx=0 tx=0).stop 2024-09-16T03:58:14.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.701+0000 7f0d2f7fe640 1 -- 172.21.15.42:0/4185497516 reap_dead start 2024-09-16T03:58:14.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.701+0000 7f0d2f7fe640 1 -- 172.21.15.42:0/4185497516 reap_dead start 2024-09-16T03:58:14.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.701+0000 7f0d35581640 1 -- 172.21.15.42:0/4185497516 shutdown_connections 2024-09-16T03:58:14.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.701+0000 7f0d35581640 1 -- 172.21.15.42:0/4185497516 >> 172.21.15.42:0/4185497516 conn(0x7f0d300a2420 msgr2=0x7f0d30134590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:58:14.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.701+0000 7f0d35581640 1 -- 172.21.15.42:0/4185497516 shutdown_connections 2024-09-16T03:58:14.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.701+0000 7f0d35581640 1 -- 172.21.15.42:0/4185497516 wait complete. 2024-09-16T03:58:14.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.701+0000 7f0d35581640 1 librados: shutdown 2024-09-16T03:58:14.740 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:58:14.740 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-09-16T03:58:14.741 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-16T03:58:14.741 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-09-16T03:58:14.741 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-16T03:58:14.741 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-16T03:58:14.741 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T03:58:14.756 INFO:tasks.ceph.mon.c:Stopped 2024-09-16T03:58:14.756 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T03:58:14.756 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:58:14.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.913+0000 7f0c61d2f640 10 monclient: get_monmap_and_config 2024-09-16T03:58:14.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.913+0000 7f0c61d2f640 10 monclient: build_initial_monmap 2024-09-16T03:58:14.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.913+0000 7f0c61d2f640 10 monclient: monmap: 2024-09-16T03:58:14.919 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:58:14.919 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:58:14.919 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:58:14.917619+0000 2024-09-16T03:58:14.919 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:58:14.917619+0000 2024-09-16T03:58:14.919 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:58:14.919 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:14.920 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:58:14.920 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:58:14.920 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:58:14.920 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:58:14.920 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:58:14.920 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:14.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.913+0000 7f0c61d2f640 1 Processor -- start 2024-09-16T03:58:14.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.913+0000 7f0c61d2f640 1 -- start start 2024-09-16T03:58:14.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.913+0000 7f0c61d2f640 10 monclient: init 2024-09-16T03:58:14.923 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.913+0000 7f0c61d2f640 10 monclient: _reopen_session rank -1 2024-09-16T03:58:14.923 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.913+0000 7f0c61d2f640 10 monclient: _add_conns ranks=[3,4,0,1,2] 2024-09-16T03:58:14.923 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.913+0000 7f0c61d2f640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f0c5c0a1330 0x7f0c5c0a1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:14.923 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.913+0000 7f0c61d2f640 10 monclient(hunting): picked mon.noname-a con 0x7f0c5c0a1330 addr v2:172.21.15.42:3300/0 2024-09-16T03:58:14.923 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.913+0000 7f0c61d2f640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f0c5c06dac0 0x7f0c5c0a1c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:14.923 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.913+0000 7f0c61d2f640 10 monclient(hunting): picked mon.noname-c con 0x7f0c5c06dac0 addr v2:172.21.15.154:3300/0 2024-09-16T03:58:14.923 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.913+0000 7f0c61d2f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0c5c135100 addr v1:172.21.15.42:6789/0 2024-09-16T03:58:14.923 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.913+0000 7f0c61d2f640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c5c09c7d0 con 0x7f0c5c135100 2024-09-16T03:58:14.923 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.913+0000 7f0c61d2f640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:14.923 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.913+0000 7f0c61d2f640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f0c5c09c090 con 0x7f0c5c0a1330 2024-09-16T03:58:14.923 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.913+0000 7f0c61d2f640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:14.923 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.913+0000 7f0c61d2f640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f0c5c09f610 con 0x7f0c5c06dac0 2024-09-16T03:58:14.923 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.913+0000 7f0c61d2f640 10 monclient(hunting): _renew_subs 2024-09-16T03:58:14.924 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.913+0000 7f0c61d2f640 10 monclient(hunting): authenticate will time out at 1941.887128s 2024-09-16T03:58:14.924 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.913+0000 7f0c5bfff640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f0c5c135100 0x7f0c5c135550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:56648/0 (socket says 172.21.15.42:56648) 2024-09-16T03:58:14.924 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.913+0000 7f0c5bfff640 1 -- 172.21.15.42:0/34388239 learned_addr learned my addr 172.21.15.42:0/34388239 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:58:14.924 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.913+0000 7f0c5b7fe640 1 --2- 172.21.15.42:0/34388239 >> v2:172.21.15.42:3300/0 conn(0x7f0c5c0a1330 0x7f0c5c0a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:14.924 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.913+0000 7f0c5b7fe640 10 monclient(hunting): get_auth_request con 0x7f0c5c0a1330 auth_method 0 2024-09-16T03:58:14.924 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.913+0000 7f0c5b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:14.924 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.913+0000 7f0c5b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:14.924 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.913+0000 7f0c5b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:58:14.924 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.913+0000 7f0c5affd640 1 --2- 172.21.15.42:0/34388239 >> v2:172.21.15.154:3300/0 conn(0x7f0c5c06dac0 0x7f0c5c0a1c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:14.924 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.913+0000 7f0c5affd640 10 monclient(hunting): get_auth_request con 0x7f0c5c06dac0 auth_method 0 2024-09-16T03:58:14.924 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.913+0000 7f0c5affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:14.924 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.913+0000 7f0c5affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:14.924 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.913+0000 7f0c5affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:58:14.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.913+0000 7f0c5b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:58:14.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.913+0000 7f0c5b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:58:14.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.913+0000 7f0c5b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:58:14.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.913+0000 7f0c5a7fc640 1 -- 172.21.15.42:0/34388239 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2683473196 0 0) 0x7f0c5c09c7d0 con 0x7f0c5c135100 2024-09-16T03:58:14.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.913+0000 7f0c5a7fc640 10 monclient(hunting): _negotiate 2024-09-16T03:58:14.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.913+0000 7f0c5a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:14.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.913+0000 7f0c5a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:58:14.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.913+0000 7f0c5a7fc640 10 monclient(hunting): my global_id is 824970 2024-09-16T03:58:14.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.913+0000 7f0c5a7fc640 1 -- 172.21.15.42:0/34388239 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0c44003530 con 0x7f0c5c135100 2024-09-16T03:58:14.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.913+0000 7f0c5affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:58:14.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.913+0000 7f0c5affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:58:14.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.913+0000 7f0c5affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:58:14.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.913+0000 7f0c5b7fe640 10 monclient(hunting): handle_auth_done global_id 824973 payload 274 2024-09-16T03:58:14.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.913+0000 7f0c5b7fe640 1 -- 172.21.15.42:0/34388239 >> v2:172.21.15.154:3300/0 conn(0x7f0c5c06dac0 msgr2=0x7f0c5c0a1c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:58:14.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.913+0000 7f0c5b7fe640 1 --2- 172.21.15.42:0/34388239 >> v2:172.21.15.154:3300/0 conn(0x7f0c5c06dac0 0x7f0c5c0a1c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:14.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.913+0000 7f0c5b7fe640 1 -- 172.21.15.42:0/34388239 >> v1:172.21.15.42:6789/0 conn(0x7f0c5c135100 legacy=0x7f0c5c135550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:14.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.913+0000 7f0c5b7fe640 10 monclient: _finish_hunting 0 2024-09-16T03:58:14.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.913+0000 7f0c5b7fe640 1 monclient: found mon.noname-a 2024-09-16T03:58:14.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.913+0000 7f0c5b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:58:14.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.913+0000 7f0c5b7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:58:14.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.913+0000 7f0c5b7fe640 1 -- 172.21.15.42:0/34388239 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c5c09f8f0 con 0x7f0c5c0a1330 2024-09-16T03:58:14.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.917+0000 7f0c5b7fe640 1 --2- 172.21.15.42:0/34388239 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0c5c0a1330 0x7f0c5c0a1710 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f0c5c06da10 tx=0x7f0c4c01c430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7f4dad0b8c3a16c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:58:14.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.917+0000 7f0c5a7fc640 1 -- 172.21.15.42:0/34388239 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c4c0ab650 con 0x7f0c5c0a1330 2024-09-16T03:58:14.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.917+0000 7f0c5a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:58:14.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.917+0000 7f0c5a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:58:14.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.917+0000 7f0c5a7fc640 10 monclient: dump: 2024-09-16T03:58:14.932 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:58:14.932 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:58:14.932 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:14.932 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:14.932 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:58:14.932 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:14.932 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:58:14.932 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:58:14.932 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:58:14.932 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:14.933 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.917+0000 7f0c5a7fc640 10 monclient: _finish_auth 0 2024-09-16T03:58:14.933 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.917+0000 7f0c5a7fc640 10 monclient: _check_auth_tickets 2024-09-16T03:58:14.933 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.917+0000 7f0c5a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:58:14.933 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.917+0000 7f0c5a7fc640 1 -- 172.21.15.42:0/34388239 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c4c0b0070 con 0x7f0c5c0a1330 2024-09-16T03:58:14.933 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.917+0000 7f0c61d2f640 5 monclient: authenticate success, global_id 824973 2024-09-16T03:58:14.933 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.917+0000 7f0c5a7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:58:14.933 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.917+0000 7f0c5a7fc640 1 -- 172.21.15.42:0/34388239 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c4c0aba10 con 0x7f0c5c0a1330 2024-09-16T03:58:14.933 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.917+0000 7f0c61d2f640 10 monclient: get_monmap_and_config success 2024-09-16T03:58:14.933 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.917+0000 7f0c61d2f640 4 set_mon_vals no callback set 2024-09-16T03:58:14.933 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.917+0000 7f0c5a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:58:14.933 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.917+0000 7f0c5a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:58:14.933 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.917+0000 7f0c5a7fc640 10 monclient: dump: 2024-09-16T03:58:14.933 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:58:14.933 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:58:14.934 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:14.934 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:14.934 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:58:14.934 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:14.934 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:58:14.934 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:58:14.934 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:58:14.934 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:14.934 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.917+0000 7f0c61d2f640 10 monclient: shutdown 2024-09-16T03:58:14.934 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.917+0000 7f0c61d2f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:58:14.934 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.917+0000 7f0c61d2f640 1 -- 172.21.15.42:0/34388239 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0c5c0a1330 msgr2=0x7f0c5c0a1710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:14.934 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.917+0000 7f0c61d2f640 1 --2- 172.21.15.42:0/34388239 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0c5c0a1330 0x7f0c5c0a1710 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f0c5c06da10 tx=0x7f0c4c01c430 comp rx=0 tx=0).stop 2024-09-16T03:58:14.934 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.917+0000 7f0c61d2f640 1 -- 172.21.15.42:0/34388239 shutdown_connections 2024-09-16T03:58:14.934 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.917+0000 7f0c61d2f640 1 --2- 172.21.15.42:0/34388239 >> v2:172.21.15.154:3300/0 conn(0x7f0c5c06dac0 0x7f0c5c0a1c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:14.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.917+0000 7f0c61d2f640 1 --2- 172.21.15.42:0/34388239 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0c5c0a1330 0x7f0c5c0a1710 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:14.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.917+0000 7f0c61d2f640 1 -- 172.21.15.42:0/34388239 >> 172.21.15.42:0/34388239 conn(0x7f0c5c0a2420 msgr2=0x7f0c5c0a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:58:14.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.917+0000 7f0c61d2f640 1 -- 172.21.15.42:0/34388239 shutdown_connections 2024-09-16T03:58:14.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.917+0000 7f0c61d2f640 1 -- 172.21.15.42:0/34388239 wait complete. 2024-09-16T03:58:14.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.917+0000 7f0c61d2f640 10 monclient: build_initial_monmap 2024-09-16T03:58:14.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.917+0000 7f0c61d2f640 10 monclient: monmap: 2024-09-16T03:58:14.935 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:58:14.935 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:58:14.935 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:58:14.935 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:58:14.935 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:58:14.935 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:14.935 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:58:14.935 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:58:14.935 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:58:14.936 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:14.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.917+0000 7f0c61d2f640 1 librados: starting msgr at 2024-09-16T03:58:14.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.917+0000 7f0c61d2f640 1 librados: starting objecter 2024-09-16T03:58:14.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.917+0000 7f0c61d2f640 1 Processor -- start 2024-09-16T03:58:14.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.917+0000 7f0c61d2f640 1 -- start start 2024-09-16T03:58:14.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.917+0000 7f0c61d2f640 1 librados: setting wanted keys 2024-09-16T03:58:14.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.917+0000 7f0c61d2f640 1 librados: calling monclient init 2024-09-16T03:58:14.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.917+0000 7f0c61d2f640 10 monclient: init 2024-09-16T03:58:14.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.921+0000 7f0c61d2f640 10 monclient: _reopen_session rank -1 2024-09-16T03:58:14.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.921+0000 7f0c61d2f640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:58:14.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.921+0000 7f0c61d2f640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0c5c0a1330 0x7f0c5c13c060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:14.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.921+0000 7f0c61d2f640 10 monclient(hunting): picked mon.noname-a con 0x7f0c5c0a1330 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:58:14.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.921+0000 7f0c61d2f640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0c5c135100 0x7f0c5c134450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:14.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.921+0000 7f0c61d2f640 10 monclient(hunting): picked mon.noname-c con 0x7f0c5c135100 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:58:14.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.921+0000 7f0c61d2f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0c5c134990 0x7f0c5c1cf0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:14.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.921+0000 7f0c61d2f640 10 monclient(hunting): picked mon.noname-b con 0x7f0c5c134990 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:58:14.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.921+0000 7f0c61d2f640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:14.941 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.921+0000 7f0c61d2f640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f0c5c13fb80 con 0x7f0c5c0a1330 2024-09-16T03:58:14.941 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.921+0000 7f0c61d2f640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:14.941 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.921+0000 7f0c61d2f640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f0c5c09ef80 con 0x7f0c5c134990 2024-09-16T03:58:14.941 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.921+0000 7f0c61d2f640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:14.941 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.921+0000 7f0c61d2f640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f0c5c13fd00 con 0x7f0c5c135100 2024-09-16T03:58:14.941 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.921+0000 7f0c61d2f640 10 monclient(hunting): _renew_subs 2024-09-16T03:58:14.941 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.921+0000 7f0c61d2f640 10 monclient(hunting): authenticate will time out at 1941.892512s 2024-09-16T03:58:14.941 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.921+0000 7f0c5affd640 1 -- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0c5c135100 msgr2=0x7f0c5c134450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T03:58:14.941 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.921+0000 7f0c5affd640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0c5c135100 0x7f0c5c134450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:58:14.941 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.921+0000 7f0c5b7fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0c5c0a1330 0x7f0c5c13c060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:14.941 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.921+0000 7f0c5b7fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0c5c0a1330 0x7f0c5c13c060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:50330/0 (socket says 172.21.15.42:50330) 2024-09-16T03:58:14.941 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.921+0000 7f0c5b7fe640 1 -- 172.21.15.42:0/480562459 learned_addr learned my addr 172.21.15.42:0/480562459 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:58:14.942 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.921+0000 7f0c5b7fe640 10 monclient(hunting): get_auth_request con 0x7f0c5c0a1330 auth_method 0 2024-09-16T03:58:14.942 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.921+0000 7f0c5b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:14.942 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.921+0000 7f0c5b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:14.942 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.921+0000 7f0c5b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:58:14.942 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.921+0000 7f0c5b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:58:14.942 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.921+0000 7f0c5b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:58:14.942 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.921+0000 7f0c5b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:58:14.942 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.921+0000 7f0c5b7fe640 10 monclient(hunting): handle_auth_done global_id 824976 payload 824 2024-09-16T03:58:14.942 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.921+0000 7f0c5bfff640 1 --2- 172.21.15.42:0/480562459 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0c5c134990 0x7f0c5c1cf0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:14.942 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.921+0000 7f0c5b7fe640 1 -- 172.21.15.42:0/480562459 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0c5c135100 msgr2=0x7f0c5c134450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:58:14.942 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.921+0000 7f0c5b7fe640 1 --2- 172.21.15.42:0/480562459 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0c5c135100 0x7f0c5c134450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:14.942 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.921+0000 7f0c5b7fe640 1 -- 172.21.15.42:0/480562459 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0c5c134990 msgr2=0x7f0c5c1cf0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:14.942 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.921+0000 7f0c5b7fe640 1 --2- 172.21.15.42:0/480562459 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0c5c134990 0x7f0c5c1cf0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:14.943 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.921+0000 7f0c5b7fe640 10 monclient: _finish_hunting 0 2024-09-16T03:58:14.943 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.921+0000 7f0c5b7fe640 1 monclient: found mon.noname-a 2024-09-16T03:58:14.943 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.921+0000 7f0c5b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:58:14.943 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.921+0000 7f0c5b7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:58:14.943 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.921+0000 7f0c5b7fe640 1 -- 172.21.15.42:0/480562459 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c5c13ccf0 con 0x7f0c5c0a1330 2024-09-16T03:58:14.943 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.921+0000 7f0c5bfff640 10 monclient: get_auth_request con 0x7f0c5c134990 auth_method 0 2024-09-16T03:58:14.943 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.921+0000 7f0c5bfff640 1 --2- 172.21.15.42:0/480562459 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0c5c134990 0x7f0c5c1cf0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T03:58:14.943 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.921+0000 7f0c5b7fe640 1 --2- 172.21.15.42:0/480562459 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0c5c0a1330 0x7f0c5c13c060 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f0c4c0ad040 tx=0x7f0c4c01ac70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:58:14.943 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.921+0000 7f0c43fff640 10 client.?.objecter ms_handle_connect 0x7f0c5c0a1330 2024-09-16T03:58:14.943 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.921+0000 7f0c43fff640 10 client.?.objecter resend_mon_ops 2024-09-16T03:58:14.943 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.921+0000 7f0c43fff640 1 -- 172.21.15.42:0/480562459 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c4c0bf410 con 0x7f0c5c0a1330 2024-09-16T03:58:14.943 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.921+0000 7f0c43fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:58:14.943 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.921+0000 7f0c43fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:58:14.943 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.921+0000 7f0c43fff640 10 monclient: dump: 2024-09-16T03:58:14.943 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:58:14.944 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:58:14.944 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:14.944 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:14.944 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:58:14.944 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:14.944 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:58:14.944 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:58:14.944 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:58:14.944 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:14.944 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.921+0000 7f0c43fff640 10 monclient: _finish_auth 0 2024-09-16T03:58:14.944 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.921+0000 7f0c43fff640 10 monclient: _check_auth_tickets 2024-09-16T03:58:14.944 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.921+0000 7f0c43fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:58:14.944 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.921+0000 7f0c43fff640 1 -- 172.21.15.42:0/480562459 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c4c0bf5d0 con 0x7f0c5c0a1330 2024-09-16T03:58:14.945 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.921+0000 7f0c61d2f640 5 monclient: authenticate success, global_id 824976 2024-09-16T03:58:14.945 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.921+0000 7f0c61d2f640 10 monclient: _renew_subs 2024-09-16T03:58:14.945 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.921+0000 7f0c61d2f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:58:14.945 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.921+0000 7f0c61d2f640 1 -- 172.21.15.42:0/480562459 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c5c0a5230 con 0x7f0c5c0a1330 2024-09-16T03:58:14.945 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.921+0000 7f0c43fff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:58:14.945 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.921+0000 7f0c43fff640 1 -- 172.21.15.42:0/480562459 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c4c0bf9c0 con 0x7f0c5c0a1330 2024-09-16T03:58:14.945 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.921+0000 7f0c43fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:58:14.945 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.921+0000 7f0c43fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:58:14.945 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.921+0000 7f0c43fff640 10 monclient: dump: 2024-09-16T03:58:14.945 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:58:14.946 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:58:14.946 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:14.946 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:14.946 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:58:14.946 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:14.946 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:58:14.946 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:58:14.946 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:58:14.946 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:14.946 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.921+0000 7f0c61d2f640 10 client.824976.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:58:14.946 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.921+0000 7f0c61d2f640 10 monclient: _renew_subs 2024-09-16T03:58:14.946 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.921+0000 7f0c61d2f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:58:14.946 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.921+0000 7f0c61d2f640 1 -- 172.21.15.42:0/480562459 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c5c120560 con 0x7f0c5c0a1330 2024-09-16T03:58:14.946 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.921+0000 7f0c597fa640 4 set_mon_vals no callback set 2024-09-16T03:58:14.947 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.921+0000 7f0c61d2f640 1 librados: init done 2024-09-16T03:58:14.947 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.921+0000 7f0c61d2f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:58:14.947 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.921+0000 7f0c61d2f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:58:14.947 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.921+0000 7f0c61d2f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:58:14.947 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.921+0000 7f0c61d2f640 1 -- 172.21.15.42:0/480562459 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0c5c135550 con 0x7f0c5c0a1330 2024-09-16T03:58:14.947 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.921+0000 7f0c43fff640 1 -- 172.21.15.42:0/480562459 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f0c4c0bd020 con 0x7f0c5c0a1330 2024-09-16T03:58:14.947 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.921+0000 7f0c43fff640 1 --2- 172.21.15.42:0/480562459 >> v2:172.21.15.154:6812/254845342 conn(0x7f0c480377f0 0x7f0c48039cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:14.947 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.921+0000 7f0c43fff640 1 -- 172.21.15.42:0/480562459 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(485..485 src has 366..485) ==== 6560+0+0 (secure 0 0 0) 0x7f0c4c0e3b00 con 0x7f0c5c0a1330 2024-09-16T03:58:14.947 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.921+0000 7f0c43fff640 10 client.824976.objecter ms_dispatch 0x7f0c5c000ba0 osd_map(485..485 src has 366..485) 2024-09-16T03:58:14.947 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.921+0000 7f0c43fff640 3 client.824976.objecter handle_osd_map got epochs [485,485] > 0 2024-09-16T03:58:14.947 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.921+0000 7f0c43fff640 3 client.824976.objecter handle_osd_map decoding full epoch 485 2024-09-16T03:58:14.947 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.921+0000 7f0c43fff640 20 client.824976.objecter dump_active .. 0 homeless 2024-09-16T03:58:14.947 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.921+0000 7f0c5affd640 1 --2- 172.21.15.42:0/480562459 >> v2:172.21.15.154:6812/254845342 conn(0x7f0c480377f0 0x7f0c48039cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:14.948 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.921+0000 7f0c5affd640 10 monclient: get_auth_request con 0x7f0c480377f0 auth_method 0 2024-09-16T03:58:14.948 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.925+0000 7f0c5affd640 1 --2- 172.21.15.42:0/480562459 >> v2:172.21.15.154:6812/254845342 conn(0x7f0c480377f0 0x7f0c48039cb0 secure :-1 s=READY pgs=3697 cs=0 l=1 rev1=1 crypto rx=0x7f0c5c09e2e0 tx=0x7f0c50009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:58:14.948 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.925+0000 7f0c43fff640 10 client.824976.objecter ms_handle_connect 0x7f0c480377f0 2024-09-16T03:58:14.948 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.925+0000 7f0c43fff640 1 -- 172.21.15.42:0/480562459 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f0c4c016050 con 0x7f0c5c0a1330 2024-09-16T03:58:14.948 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.925+0000 7f0c43fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:58:14.948 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:14.925+0000 7f0c43fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:58:15.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:15.145+0000 7f0c61d2f640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:58:15.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:15.145+0000 7f0c61d2f640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:58:15.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:15.145+0000 7f0c61d2f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:58:15.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:15.145+0000 7f0c61d2f640 1 -- 172.21.15.42:0/480562459 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f0c5c0a47a0 con 0x7f0c5c0a1330 2024-09-16T03:58:15.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:15.145+0000 7f0c43fff640 1 -- 172.21.15.42:0/480562459 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7f0c4c0b5d00 con 0x7f0c5c0a1330 2024-09-16T03:58:15.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:15.145+0000 7f0c43fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:58:15.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:15.145+0000 7f0c43fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:58:15.153 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":148,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:58:15.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:15.149+0000 7f0c61d2f640 10 librados: watch_flush enter 2024-09-16T03:58:15.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:15.149+0000 7f0c61d2f640 10 librados: watch_flush exit 2024-09-16T03:58:15.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:15.149+0000 7f0c61d2f640 20 client.824976.objecter shutdown clearing up homeless session... 2024-09-16T03:58:15.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:15.149+0000 7f0c61d2f640 10 client.824976.objecter successfully canceled tick 2024-09-16T03:58:15.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:15.149+0000 7f0c61d2f640 1 -- 172.21.15.42:0/480562459 >> v2:172.21.15.154:6812/254845342 conn(0x7f0c480377f0 msgr2=0x7f0c48039cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:15.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:15.149+0000 7f0c61d2f640 1 --2- 172.21.15.42:0/480562459 >> v2:172.21.15.154:6812/254845342 conn(0x7f0c480377f0 0x7f0c48039cb0 secure :-1 s=READY pgs=3697 cs=0 l=1 rev1=1 crypto rx=0x7f0c5c09e2e0 tx=0x7f0c50009190 comp rx=0 tx=0).stop 2024-09-16T03:58:15.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:15.149+0000 7f0c61d2f640 10 monclient: shutdown 2024-09-16T03:58:15.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:15.149+0000 7f0c61d2f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:58:15.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:15.149+0000 7f0c61d2f640 1 -- 172.21.15.42:0/480562459 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0c5c0a1330 msgr2=0x7f0c5c13c060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:15.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:15.149+0000 7f0c61d2f640 1 --2- 172.21.15.42:0/480562459 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0c5c0a1330 0x7f0c5c13c060 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f0c4c0ad040 tx=0x7f0c4c01ac70 comp rx=0 tx=0).stop 2024-09-16T03:58:15.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:15.149+0000 7f0c61d2f640 1 -- 172.21.15.42:0/480562459 shutdown_connections 2024-09-16T03:58:15.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:15.149+0000 7f0c61d2f640 1 --2- 172.21.15.42:0/480562459 >> v2:172.21.15.154:6812/254845342 conn(0x7f0c480377f0 0x7f0c48039cb0 unknown :-1 s=CLOSED pgs=3697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:15.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:15.149+0000 7f0c61d2f640 1 --2- 172.21.15.42:0/480562459 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0c5c134990 0x7f0c5c1cf0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:15.157 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:15.149+0000 7f0c61d2f640 1 --2- 172.21.15.42:0/480562459 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0c5c135100 0x7f0c5c134450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:15.157 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:15.149+0000 7f0c61d2f640 1 --2- 172.21.15.42:0/480562459 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0c5c0a1330 0x7f0c5c13c060 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:15.157 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:15.149+0000 7f0c61d2f640 1 -- 172.21.15.42:0/480562459 >> 172.21.15.42:0/480562459 conn(0x7f0c5c0a2420 msgr2=0x7f0c5c0a4040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:58:15.157 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:15.149+0000 7f0c61d2f640 1 -- 172.21.15.42:0/480562459 shutdown_connections 2024-09-16T03:58:15.157 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:15.149+0000 7f0c61d2f640 1 -- 172.21.15.42:0/480562459 wait complete. 2024-09-16T03:58:15.157 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:15.149+0000 7f0c61d2f640 1 librados: shutdown 2024-09-16T03:58:18.172 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:58:18.307 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.301+0000 7f6587302640 10 monclient: get_monmap_and_config 2024-09-16T03:58:18.307 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.301+0000 7f6587302640 10 monclient: build_initial_monmap 2024-09-16T03:58:18.307 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.301+0000 7f6587302640 10 monclient: monmap: 2024-09-16T03:58:18.307 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:58:18.307 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:58:18.307 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:58:18.305641+0000 2024-09-16T03:58:18.307 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:58:18.305641+0000 2024-09-16T03:58:18.307 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:58:18.308 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:18.308 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:58:18.308 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:58:18.308 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:58:18.308 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:58:18.308 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:58:18.308 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:18.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.301+0000 7f6587302640 1 Processor -- start 2024-09-16T03:58:18.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.301+0000 7f6587302640 1 -- start start 2024-09-16T03:58:18.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.301+0000 7f6587302640 10 monclient: init 2024-09-16T03:58:18.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.301+0000 7f6587302640 10 monclient: _reopen_session rank -1 2024-09-16T03:58:18.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.301+0000 7f6587302640 10 monclient: _add_conns ranks=[4,1,0,3,2] 2024-09-16T03:58:18.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.301+0000 7f6587302640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f65800a39b0 0x7f6580133b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:18.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.301+0000 7f6587302640 10 monclient(hunting): picked mon.noname-c con 0x7f65800a39b0 addr v2:172.21.15.154:3300/0 2024-09-16T03:58:18.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.301+0000 7f6587302640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f65800a3d90 addr v1:172.21.15.154:6789/0 2024-09-16T03:58:18.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.301+0000 7f6587302640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6580137710 addr v1:172.21.15.42:6789/0 2024-09-16T03:58:18.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.301+0000 7f6587302640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f658009c140 con 0x7f6580137710 2024-09-16T03:58:18.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.301+0000 7f6587302640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f658009c420 con 0x7f65800a3d90 2024-09-16T03:58:18.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.301+0000 7f6587302640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:18.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.301+0000 7f6587302640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f658006dd60 con 0x7f65800a39b0 2024-09-16T03:58:18.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.301+0000 7f6587302640 10 monclient(hunting): _renew_subs 2024-09-16T03:58:18.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.301+0000 7f6587302640 10 monclient(hunting): authenticate will time out at 1945.275126s 2024-09-16T03:58:18.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.301+0000 7f658589f640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f6580137710 0x7f658013f3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:56652/0 (socket says 172.21.15.42:56652) 2024-09-16T03:58:18.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.301+0000 7f658589f640 1 -- 172.21.15.42:0/2934865559 learned_addr learned my addr 172.21.15.42:0/2934865559 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:58:18.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.301+0000 7f658509e640 1 --2- 172.21.15.42:0/2934865559 >> v2:172.21.15.154:3300/0 conn(0x7f65800a39b0 0x7f6580133b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:18.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.301+0000 7f658509e640 10 monclient(hunting): get_auth_request con 0x7f65800a39b0 auth_method 0 2024-09-16T03:58:18.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.301+0000 7f658509e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:18.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.301+0000 7f658509e640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:18.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.301+0000 7f658509e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:58:18.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.301+0000 7f6573fff640 1 -- 172.21.15.42:0/2934865559 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1137274715 0 0) 0x7f658009c140 con 0x7f6580137710 2024-09-16T03:58:18.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.301+0000 7f6573fff640 10 monclient(hunting): _negotiate 2024-09-16T03:58:18.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.301+0000 7f6573fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:18.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.301+0000 7f6573fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:58:18.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.301+0000 7f6573fff640 10 monclient(hunting): my global_id is 824979 2024-09-16T03:58:18.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.301+0000 7f6573fff640 1 -- 172.21.15.42:0/2934865559 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f656c003530 con 0x7f6580137710 2024-09-16T03:58:18.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.301+0000 7f658509e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:58:18.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.301+0000 7f658509e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:58:18.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.301+0000 7f658509e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:58:18.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.301+0000 7f6573fff640 1 -- 172.21.15.42:0/2934865559 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 638038003 0 0) 0x7f656c003530 con 0x7f6580137710 2024-09-16T03:58:18.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.301+0000 7f6573fff640 1 -- 172.21.15.42:0/2934865559 >> v2:172.21.15.154:3300/0 conn(0x7f65800a39b0 msgr2=0x7f6580133b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:58:18.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.301+0000 7f6573fff640 1 --2- 172.21.15.42:0/2934865559 >> v2:172.21.15.154:3300/0 conn(0x7f65800a39b0 0x7f6580133b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:18.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.301+0000 7f6573fff640 1 -- 172.21.15.42:0/2934865559 >> v1:172.21.15.154:6789/0 conn(0x7f65800a3d90 legacy=0x7f6580137000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:18.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.301+0000 7f6573fff640 10 monclient: _finish_hunting 0 2024-09-16T03:58:18.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.301+0000 7f6573fff640 1 monclient: found mon.noname-a-legacy 2024-09-16T03:58:18.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.301+0000 7f6573fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:58:18.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.301+0000 7f6573fff640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.42:6789/0 2024-09-16T03:58:18.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.301+0000 7f6573fff640 1 -- 172.21.15.42:0/2934865559 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65800a5e20 con 0x7f6580137710 2024-09-16T03:58:18.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.301+0000 7f6573fff640 10 monclient: _finish_auth 0 2024-09-16T03:58:18.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.301+0000 7f6573fff640 10 monclient: _check_auth_tickets 2024-09-16T03:58:18.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.301+0000 7f6573fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:58:18.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.301+0000 7f6573fff640 1 -- 172.21.15.42:0/2934865559 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2177413734 0 0) 0x7f6578001a00 con 0x7f6580137710 2024-09-16T03:58:18.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.301+0000 7f6573fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:58:18.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.301+0000 7f6573fff640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T03:58:18.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.301+0000 7f6573fff640 10 monclient: dump: 2024-09-16T03:58:18.323 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:58:18.323 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:58:18.323 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:18.323 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:18.323 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:58:18.324 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:18.324 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:58:18.324 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:58:18.324 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:58:18.324 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:18.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.301+0000 7f6573fff640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] but i'm connected to v1:172.21.15.42:6789/0, reconnecting 2024-09-16T03:58:18.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.301+0000 7f6573fff640 10 monclient: _reopen_session rank -1 2024-09-16T03:58:18.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.301+0000 7f6573fff640 1 -- 172.21.15.42:0/2934865559 >> v1:172.21.15.42:6789/0 conn(0x7f6580137710 legacy=0x7f658013f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:18.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.301+0000 7f6573fff640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:58:18.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.301+0000 7f6573fff640 1 --2- 172.21.15.42:0/2934865559 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f656c0039c0 0x7f656c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:18.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.305+0000 7f6573fff640 10 monclient(hunting): picked mon.a con 0x7f656c0039c0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:58:18.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.305+0000 7f6573fff640 1 --2- 172.21.15.42:0/2934865559 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f656c004810 0x7f656c004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:18.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.305+0000 7f6573fff640 10 monclient(hunting): picked mon.c con 0x7f656c004810 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:58:18.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.305+0000 7f6573fff640 1 --2- 172.21.15.42:0/2934865559 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f656c0052f0 0x7f656c00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:18.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.305+0000 7f6573fff640 10 monclient(hunting): picked mon.b con 0x7f656c0052f0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:58:18.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.305+0000 7f6573fff640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:18.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.305+0000 7f6573fff640 1 -- 172.21.15.42:0/2934865559 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f658006dd60 con 0x7f656c0039c0 2024-09-16T03:58:18.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.305+0000 7f6573fff640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:18.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.305+0000 7f6573fff640 1 -- 172.21.15.42:0/2934865559 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f656c00c1c0 con 0x7f656c0052f0 2024-09-16T03:58:18.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.305+0000 7f6573fff640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:18.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.305+0000 7f6573fff640 1 -- 172.21.15.42:0/2934865559 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f656c00c3c0 con 0x7f656c004810 2024-09-16T03:58:18.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.305+0000 7f6573fff640 10 monclient(hunting): _renew_subs 2024-09-16T03:58:18.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.305+0000 7f6573fff640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:58:18.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.305+0000 7f658489d640 1 -- 172.21.15.42:0/2934865559 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f656c004810 msgr2=0x7f656c004c90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T03:58:18.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.305+0000 7f658489d640 1 --2- 172.21.15.42:0/2934865559 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f656c004810 0x7f656c004c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:58:18.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.305+0000 7f658509e640 1 --2- 172.21.15.42:0/2934865559 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f656c0039c0 0x7f656c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:18.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.305+0000 7f658509e640 10 monclient(hunting): get_auth_request con 0x7f656c0039c0 auth_method 0 2024-09-16T03:58:18.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.305+0000 7f658509e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:18.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.305+0000 7f658509e640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:18.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.305+0000 7f658509e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:58:18.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.305+0000 7f658509e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:58:18.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.305+0000 7f658509e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:58:18.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.305+0000 7f658509e640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T03:58:18.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.305+0000 7f658589f640 1 --2- 172.21.15.42:0/2934865559 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f656c0052f0 0x7f656c00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:18.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.305+0000 7f658589f640 10 monclient(hunting): get_auth_request con 0x7f656c0052f0 auth_method 0 2024-09-16T03:58:18.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.305+0000 7f658589f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:18.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.305+0000 7f658589f640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:18.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.305+0000 7f658589f640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:58:18.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.305+0000 7f658589f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:58:18.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.305+0000 7f658589f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:58:18.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.305+0000 7f658589f640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T03:58:18.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.305+0000 7f658509e640 10 monclient(hunting): handle_auth_done global_id 824979 payload 293 2024-09-16T03:58:18.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.305+0000 7f658509e640 1 -- 172.21.15.42:0/2934865559 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f656c004810 msgr2=0x7f656c004c90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:58:18.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.305+0000 7f658509e640 1 --2- 172.21.15.42:0/2934865559 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f656c004810 0x7f656c004c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:18.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.305+0000 7f658509e640 1 -- 172.21.15.42:0/2934865559 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f656c0052f0 msgr2=0x7f656c00b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:58:18.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.305+0000 7f658509e640 1 --2- 172.21.15.42:0/2934865559 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f656c0052f0 0x7f656c00b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:18.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.305+0000 7f658509e640 10 monclient: _finish_hunting 0 2024-09-16T03:58:18.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.305+0000 7f658509e640 1 monclient: found mon.a 2024-09-16T03:58:18.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.305+0000 7f658509e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:58:18.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.305+0000 7f658589f640 1 -- 172.21.15.42:0/2934865559 reap_dead start 2024-09-16T03:58:18.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.305+0000 7f658509e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:58:18.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.305+0000 7f658509e640 1 -- 172.21.15.42:0/2934865559 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f656c003530 con 0x7f656c0039c0 2024-09-16T03:58:18.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.305+0000 7f658509e640 10 monclient: _finish_auth 0 2024-09-16T03:58:18.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.305+0000 7f658509e640 10 monclient: _check_auth_tickets 2024-09-16T03:58:18.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.305+0000 7f658509e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:58:18.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.305+0000 7f6587302640 5 monclient: authenticate success, global_id 824979 2024-09-16T03:58:18.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.305+0000 7f6587302640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:58:18.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.305+0000 7f658509e640 1 --2- 172.21.15.42:0/2934865559 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f656c0039c0 0x7f656c003de0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f657c009ab0 tx=0x7f657c01c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d8087fa0cbe46e4 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:58:18.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.305+0000 7f6573fff640 1 -- 172.21.15.42:0/2934865559 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f657c0096b0 con 0x7f656c0039c0 2024-09-16T03:58:18.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.305+0000 7f6573fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:58:18.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.305+0000 7f6573fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:58:18.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.305+0000 7f6573fff640 10 monclient: dump: 2024-09-16T03:58:18.328 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:58:18.328 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:58:18.328 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:18.328 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:18.328 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:58:18.329 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:18.329 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:58:18.329 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:58:18.329 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:58:18.329 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:18.329 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.305+0000 7f6587302640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:58:18.329 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.305+0000 7f6573fff640 1 -- 172.21.15.42:0/2934865559 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f657c0a9450 con 0x7f656c0039c0 2024-09-16T03:58:18.329 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.305+0000 7f6573fff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:58:18.329 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.305+0000 7f6573fff640 1 -- 172.21.15.42:0/2934865559 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f657c0a9840 con 0x7f656c0039c0 2024-09-16T03:58:18.329 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.305+0000 7f6587302640 10 monclient: get_monmap_and_config success 2024-09-16T03:58:18.329 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.305+0000 7f6573fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:58:18.329 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.305+0000 7f6587302640 4 set_mon_vals no callback set 2024-09-16T03:58:18.329 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.305+0000 7f6573fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:58:18.329 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.305+0000 7f6573fff640 10 monclient: dump: 2024-09-16T03:58:18.330 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:58:18.330 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:58:18.330 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:18.330 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:18.330 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:58:18.330 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:18.330 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:58:18.330 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:58:18.330 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:58:18.330 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:18.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.305+0000 7f6587302640 10 monclient: shutdown 2024-09-16T03:58:18.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.305+0000 7f6587302640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:58:18.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.305+0000 7f6587302640 1 -- 172.21.15.42:0/2934865559 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f656c0039c0 msgr2=0x7f656c003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:18.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.305+0000 7f6587302640 1 --2- 172.21.15.42:0/2934865559 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f656c0039c0 0x7f656c003de0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f657c009ab0 tx=0x7f657c01c970 comp rx=0 tx=0).stop 2024-09-16T03:58:18.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.305+0000 7f6587302640 1 -- 172.21.15.42:0/2934865559 shutdown_connections 2024-09-16T03:58:18.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.305+0000 7f6587302640 1 --2- 172.21.15.42:0/2934865559 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f656c0039c0 0x7f656c003de0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:18.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.305+0000 7f6587302640 1 -- 172.21.15.42:0/2934865559 >> 172.21.15.42:0/2934865559 conn(0x7f65800a9fb0 msgr2=0x7f65800aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:58:18.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.305+0000 7f6587302640 1 -- 172.21.15.42:0/2934865559 shutdown_connections 2024-09-16T03:58:18.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.305+0000 7f6587302640 1 -- 172.21.15.42:0/2934865559 wait complete. 2024-09-16T03:58:18.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.309+0000 7f6587302640 10 monclient: build_initial_monmap 2024-09-16T03:58:18.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.309+0000 7f6587302640 10 monclient: monmap: 2024-09-16T03:58:18.331 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:58:18.331 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:58:18.331 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:58:18.331 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:58:18.331 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:58:18.332 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:18.332 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:58:18.332 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:58:18.332 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:58:18.332 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:18.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.309+0000 7f6587302640 1 librados: starting msgr at 2024-09-16T03:58:18.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.309+0000 7f6587302640 1 librados: starting objecter 2024-09-16T03:58:18.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.309+0000 7f6587302640 1 Processor -- start 2024-09-16T03:58:18.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.309+0000 7f6587302640 1 -- start start 2024-09-16T03:58:18.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.309+0000 7f6587302640 1 librados: setting wanted keys 2024-09-16T03:58:18.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.309+0000 7f6587302640 1 librados: calling monclient init 2024-09-16T03:58:18.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.309+0000 7f6587302640 10 monclient: init 2024-09-16T03:58:18.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.309+0000 7f6587302640 10 monclient: _reopen_session rank -1 2024-09-16T03:58:18.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.309+0000 7f6587302640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:58:18.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.309+0000 7f6587302640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f656c0039c0 0x7f65801cf9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:18.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.309+0000 7f6587302640 10 monclient(hunting): picked mon.noname-c con 0x7f656c0039c0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:58:18.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.309+0000 7f6587302640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f65801cff00 0x7f65801d0320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:18.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.309+0000 7f6587302640 10 monclient(hunting): picked mon.noname-a con 0x7f65801cff00 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:58:18.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.309+0000 7f6587302640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6580120200 0x7f6580120680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:18.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.309+0000 7f6587302640 10 monclient(hunting): picked mon.noname-b con 0x7f6580120200 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:58:18.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.309+0000 7f6587302640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:18.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.309+0000 7f6587302640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f658013a580 con 0x7f65801cff00 2024-09-16T03:58:18.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.309+0000 7f6587302640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:18.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.309+0000 7f6587302640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f658006dee0 con 0x7f6580120200 2024-09-16T03:58:18.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.309+0000 7f6587302640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:18.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.309+0000 7f6587302640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f658013a700 con 0x7f656c0039c0 2024-09-16T03:58:18.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.309+0000 7f6587302640 10 monclient(hunting): _renew_subs 2024-09-16T03:58:18.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.309+0000 7f6587302640 10 monclient(hunting): authenticate will time out at 1945.281142s 2024-09-16T03:58:18.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.309+0000 7f658509e640 1 -- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f656c0039c0 msgr2=0x7f65801cf9c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T03:58:18.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.309+0000 7f658509e640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f656c0039c0 0x7f65801cf9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:58:18.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.309+0000 7f658489d640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f65801cff00 0x7f65801d0320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:18.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.309+0000 7f658489d640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f65801cff00 0x7f65801d0320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:50344/0 (socket says 172.21.15.42:50344) 2024-09-16T03:58:18.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.309+0000 7f658489d640 1 -- 172.21.15.42:0/4285610185 learned_addr learned my addr 172.21.15.42:0/4285610185 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:58:18.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.309+0000 7f658489d640 10 monclient(hunting): get_auth_request con 0x7f65801cff00 auth_method 0 2024-09-16T03:58:18.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.309+0000 7f658489d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:18.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.309+0000 7f658489d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:18.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.309+0000 7f658489d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:58:18.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.309+0000 7f658489d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:58:18.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.309+0000 7f658489d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:58:18.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.309+0000 7f658489d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:58:18.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.309+0000 7f658589f640 1 --2- 172.21.15.42:0/4285610185 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6580120200 0x7f6580120680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:18.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.309+0000 7f658589f640 10 monclient(hunting): get_auth_request con 0x7f6580120200 auth_method 0 2024-09-16T03:58:18.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.309+0000 7f658589f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:18.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.309+0000 7f658589f640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:18.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.309+0000 7f658589f640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:58:18.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.309+0000 7f658589f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:58:18.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.309+0000 7f658589f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:58:18.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.309+0000 7f658589f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:58:18.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.309+0000 7f658489d640 10 monclient(hunting): handle_auth_done global_id 824982 payload 824 2024-09-16T03:58:18.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.309+0000 7f658489d640 1 -- 172.21.15.42:0/4285610185 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f656c0039c0 msgr2=0x7f65801cf9c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:58:18.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.309+0000 7f658489d640 1 --2- 172.21.15.42:0/4285610185 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f656c0039c0 0x7f65801cf9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:18.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.309+0000 7f658489d640 1 -- 172.21.15.42:0/4285610185 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6580120200 msgr2=0x7f6580120680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:18.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.309+0000 7f658489d640 1 --2- 172.21.15.42:0/4285610185 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6580120200 0x7f6580120680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:18.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.309+0000 7f658489d640 10 monclient: _finish_hunting 0 2024-09-16T03:58:18.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.309+0000 7f658489d640 1 monclient: found mon.noname-a 2024-09-16T03:58:18.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.309+0000 7f658489d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:58:18.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.309+0000 7f658489d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:58:18.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.309+0000 7f658489d640 1 -- 172.21.15.42:0/4285610185 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6580122660 con 0x7f65801cff00 2024-09-16T03:58:18.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.309+0000 7f658489d640 1 --2- 172.21.15.42:0/4285610185 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f65801cff00 0x7f65801d0320 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f657400c500 tx=0x7f657400c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:58:18.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.309+0000 7f6571ffb640 10 client.?.objecter ms_handle_connect 0x7f65801cff00 2024-09-16T03:58:18.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.309+0000 7f6571ffb640 10 client.?.objecter resend_mon_ops 2024-09-16T03:58:18.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.309+0000 7f6571ffb640 1 -- 172.21.15.42:0/4285610185 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6574017070 con 0x7f65801cff00 2024-09-16T03:58:18.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.309+0000 7f6571ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:58:18.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.309+0000 7f6571ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:58:18.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.309+0000 7f6571ffb640 10 monclient: dump: 2024-09-16T03:58:18.338 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:58:18.338 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:58:18.339 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:18.339 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:18.339 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:58:18.339 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:18.339 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:58:18.339 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:58:18.339 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:58:18.339 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:18.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.309+0000 7f6571ffb640 10 monclient: _finish_auth 0 2024-09-16T03:58:18.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.309+0000 7f6571ffb640 10 monclient: _check_auth_tickets 2024-09-16T03:58:18.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.309+0000 7f6571ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:58:18.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.309+0000 7f6571ffb640 1 -- 172.21.15.42:0/4285610185 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6574007480 con 0x7f65801cff00 2024-09-16T03:58:18.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.309+0000 7f6571ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T03:58:18.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.309+0000 7f6571ffb640 1 -- 172.21.15.42:0/4285610185 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6574005a10 con 0x7f65801cff00 2024-09-16T03:58:18.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.309+0000 7f6571ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:58:18.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.309+0000 7f6571ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:58:18.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.309+0000 7f6571ffb640 10 monclient: dump: 2024-09-16T03:58:18.340 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:58:18.340 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:58:18.340 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:18.340 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:18.340 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:58:18.340 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:18.340 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:58:18.340 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:58:18.340 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:58:18.341 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:18.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.309+0000 7f65737fe640 4 set_mon_vals no callback set 2024-09-16T03:58:18.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.309+0000 7f6587302640 5 monclient: authenticate success, global_id 824982 2024-09-16T03:58:18.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.309+0000 7f6587302640 10 monclient: _renew_subs 2024-09-16T03:58:18.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.309+0000 7f6587302640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:58:18.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.309+0000 7f6587302640 1 -- 172.21.15.42:0/4285610185 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6580122940 con 0x7f65801cff00 2024-09-16T03:58:18.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.309+0000 7f6587302640 10 client.824982.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:58:18.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.309+0000 7f6587302640 10 monclient: _renew_subs 2024-09-16T03:58:18.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.309+0000 7f6587302640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:58:18.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.309+0000 7f6587302640 1 -- 172.21.15.42:0/4285610185 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65801333a0 con 0x7f65801cff00 2024-09-16T03:58:18.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.309+0000 7f6587302640 1 librados: init done 2024-09-16T03:58:18.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.309+0000 7f6587302640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:58:18.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.309+0000 7f6571ffb640 1 -- 172.21.15.42:0/4285610185 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f6574005bb0 con 0x7f65801cff00 2024-09-16T03:58:18.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.309+0000 7f6587302640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:58:18.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.309+0000 7f6587302640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:58:18.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.309+0000 7f6587302640 1 -- 172.21.15.42:0/4285610185 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f658013a9a0 con 0x7f65801cff00 2024-09-16T03:58:18.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.309+0000 7f6571ffb640 1 --2- 172.21.15.42:0/4285610185 >> v2:172.21.15.154:6812/254845342 conn(0x7f65640377c0 0x7f6564039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:18.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.309+0000 7f6571ffb640 1 -- 172.21.15.42:0/4285610185 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(485..485 src has 366..485) ==== 6560+0+0 (secure 0 0 0) 0x7f6574048340 con 0x7f65801cff00 2024-09-16T03:58:18.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.309+0000 7f6571ffb640 10 client.824982.objecter ms_dispatch 0x7f6580000ba0 osd_map(485..485 src has 366..485) 2024-09-16T03:58:18.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.309+0000 7f6571ffb640 3 client.824982.objecter handle_osd_map got epochs [485,485] > 0 2024-09-16T03:58:18.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.309+0000 7f6571ffb640 3 client.824982.objecter handle_osd_map decoding full epoch 485 2024-09-16T03:58:18.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.313+0000 7f6571ffb640 20 client.824982.objecter dump_active .. 0 homeless 2024-09-16T03:58:18.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.313+0000 7f658509e640 1 --2- 172.21.15.42:0/4285610185 >> v2:172.21.15.154:6812/254845342 conn(0x7f65640377c0 0x7f6564039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:18.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.313+0000 7f658509e640 10 monclient: get_auth_request con 0x7f65640377c0 auth_method 0 2024-09-16T03:58:18.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.313+0000 7f658509e640 1 --2- 172.21.15.42:0/4285610185 >> v2:172.21.15.154:6812/254845342 conn(0x7f65640377c0 0x7f6564039c80 secure :-1 s=READY pgs=3698 cs=0 l=1 rev1=1 crypto rx=0x7f657c0ae000 tx=0x7f657c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:58:18.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.313+0000 7f6571ffb640 10 client.824982.objecter ms_handle_connect 0x7f65640377c0 2024-09-16T03:58:18.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.317+0000 7f6571ffb640 1 -- 172.21.15.42:0/4285610185 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f657401ad20 con 0x7f65801cff00 2024-09-16T03:58:18.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.317+0000 7f6571ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:58:18.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.317+0000 7f6571ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:58:18.551 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.545+0000 7f6587302640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:58:18.551 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.545+0000 7f6587302640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:58:18.551 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.545+0000 7f6587302640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:58:18.551 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.545+0000 7f6587302640 1 -- 172.21.15.42:0/4285610185 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f65800a5750 con 0x7f65801cff00 2024-09-16T03:58:18.551 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.545+0000 7f6571ffb640 1 -- 172.21.15.42:0/4285610185 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7f6574019dc0 con 0x7f65801cff00 2024-09-16T03:58:18.551 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.545+0000 7f6571ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:58:18.552 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.545+0000 7f6571ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:58:18.552 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":148,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:58:18.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.549+0000 7f6587302640 10 librados: watch_flush enter 2024-09-16T03:58:18.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.549+0000 7f6587302640 10 librados: watch_flush exit 2024-09-16T03:58:18.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.549+0000 7f6587302640 20 client.824982.objecter shutdown clearing up homeless session... 2024-09-16T03:58:18.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.549+0000 7f6587302640 10 client.824982.objecter successfully canceled tick 2024-09-16T03:58:18.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.549+0000 7f6587302640 1 -- 172.21.15.42:0/4285610185 >> v2:172.21.15.154:6812/254845342 conn(0x7f65640377c0 msgr2=0x7f6564039c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:18.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.549+0000 7f6587302640 1 --2- 172.21.15.42:0/4285610185 >> v2:172.21.15.154:6812/254845342 conn(0x7f65640377c0 0x7f6564039c80 secure :-1 s=READY pgs=3698 cs=0 l=1 rev1=1 crypto rx=0x7f657c0ae000 tx=0x7f657c008040 comp rx=0 tx=0).stop 2024-09-16T03:58:18.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.549+0000 7f6587302640 10 monclient: shutdown 2024-09-16T03:58:18.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.549+0000 7f6587302640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:58:18.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.549+0000 7f6587302640 1 -- 172.21.15.42:0/4285610185 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f65801cff00 msgr2=0x7f65801d0320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:18.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.549+0000 7f6587302640 1 --2- 172.21.15.42:0/4285610185 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f65801cff00 0x7f65801d0320 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f657400c500 tx=0x7f657400c900 comp rx=0 tx=0).stop 2024-09-16T03:58:18.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.549+0000 7f6587302640 1 -- 172.21.15.42:0/4285610185 shutdown_connections 2024-09-16T03:58:18.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.549+0000 7f6587302640 1 --2- 172.21.15.42:0/4285610185 >> v2:172.21.15.154:6812/254845342 conn(0x7f65640377c0 0x7f6564039c80 unknown :-1 s=CLOSED pgs=3698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:18.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.549+0000 7f6587302640 1 --2- 172.21.15.42:0/4285610185 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6580120200 0x7f6580120680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:18.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.549+0000 7f6587302640 1 --2- 172.21.15.42:0/4285610185 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f65801cff00 0x7f65801d0320 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:18.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.549+0000 7f6587302640 1 --2- 172.21.15.42:0/4285610185 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f656c0039c0 0x7f65801cf9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:18.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.549+0000 7f6587302640 1 -- 172.21.15.42:0/4285610185 >> 172.21.15.42:0/4285610185 conn(0x7f65800a9fb0 msgr2=0x7f65800a7a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:58:18.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.549+0000 7f6587302640 1 -- 172.21.15.42:0/4285610185 shutdown_connections 2024-09-16T03:58:18.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.549+0000 7f6587302640 1 -- 172.21.15.42:0/4285610185 wait complete. 2024-09-16T03:58:18.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:18.549+0000 7f6587302640 1 librados: shutdown 2024-09-16T03:58:21.572 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:58:21.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.701+0000 7fe2f30e2640 10 monclient: get_monmap_and_config 2024-09-16T03:58:21.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.701+0000 7fe2f30e2640 10 monclient: build_initial_monmap 2024-09-16T03:58:21.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.701+0000 7fe2f30e2640 10 monclient: monmap: 2024-09-16T03:58:21.707 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:58:21.707 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:58:21.707 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:58:21.705888+0000 2024-09-16T03:58:21.708 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:58:21.705888+0000 2024-09-16T03:58:21.708 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:58:21.708 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:21.708 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:58:21.708 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:58:21.708 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:58:21.708 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:58:21.708 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:58:21.708 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:21.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.701+0000 7fe2f30e2640 1 Processor -- start 2024-09-16T03:58:21.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.701+0000 7fe2f30e2640 1 -- start start 2024-09-16T03:58:21.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.701+0000 7fe2f30e2640 10 monclient: init 2024-09-16T03:58:21.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.701+0000 7fe2f30e2640 10 monclient: _reopen_session rank -1 2024-09-16T03:58:21.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.701+0000 7fe2f30e2640 10 monclient: _add_conns ranks=[3,1,0,2,4] 2024-09-16T03:58:21.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.701+0000 7fe2f30e2640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe2ec0a1330 0x7fe2ec0a1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:21.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.701+0000 7fe2f30e2640 10 monclient(hunting): picked mon.noname-a con 0x7fe2ec0a1330 addr v2:172.21.15.42:3300/0 2024-09-16T03:58:21.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.701+0000 7fe2f30e2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe2ec06dac0 addr v1:172.21.15.154:6789/0 2024-09-16T03:58:21.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.701+0000 7fe2f30e2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe2ec0a1c50 addr v1:172.21.15.42:6789/0 2024-09-16T03:58:21.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.701+0000 7fe2f30e2640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe2ec135f20 con 0x7fe2ec0a1c50 2024-09-16T03:58:21.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.701+0000 7fe2f30e2640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe2ec09c090 con 0x7fe2ec06dac0 2024-09-16T03:58:21.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.701+0000 7fe2f30e2640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:21.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.701+0000 7fe2f30e2640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7fe2ec09f610 con 0x7fe2ec0a1330 2024-09-16T03:58:21.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.701+0000 7fe2f30e2640 10 monclient(hunting): _renew_subs 2024-09-16T03:58:21.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.701+0000 7fe2f30e2640 10 monclient(hunting): authenticate will time out at 1948.675387s 2024-09-16T03:58:21.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.701+0000 7fe2f167f640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fe2ec0a1c50 0x7fe2ec135810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:56658/0 (socket says 172.21.15.42:56658) 2024-09-16T03:58:21.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.701+0000 7fe2f167f640 1 -- 172.21.15.42:0/1156013195 learned_addr learned my addr 172.21.15.42:0/1156013195 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:58:21.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.701+0000 7fe2f0e7e640 1 --2- 172.21.15.42:0/1156013195 >> v2:172.21.15.42:3300/0 conn(0x7fe2ec0a1330 0x7fe2ec0a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:21.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.701+0000 7fe2f0e7e640 10 monclient(hunting): get_auth_request con 0x7fe2ec0a1330 auth_method 0 2024-09-16T03:58:21.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.701+0000 7fe2f0e7e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:21.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.701+0000 7fe2f0e7e640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:21.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.701+0000 7fe2f0e7e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:58:21.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.701+0000 7fe2f0e7e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:58:21.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.701+0000 7fe2f0e7e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:58:21.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.701+0000 7fe2f0e7e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:58:21.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.705+0000 7fe2f0e7e640 10 monclient(hunting): handle_auth_done global_id 824985 payload 274 2024-09-16T03:58:21.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.705+0000 7fe2f0e7e640 1 -- 172.21.15.42:0/1156013195 >> v1:172.21.15.154:6789/0 conn(0x7fe2ec06dac0 legacy=0x7fe2ec135100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:21.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.705+0000 7fe2f0e7e640 1 -- 172.21.15.42:0/1156013195 >> v1:172.21.15.42:6789/0 conn(0x7fe2ec0a1c50 legacy=0x7fe2ec135810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:21.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.705+0000 7fe2f0e7e640 10 monclient: _finish_hunting 0 2024-09-16T03:58:21.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.705+0000 7fe2f0e7e640 1 monclient: found mon.noname-a 2024-09-16T03:58:21.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.705+0000 7fe2f0e7e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:58:21.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.705+0000 7fe2f0e7e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:58:21.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.705+0000 7fe2f0e7e640 1 -- 172.21.15.42:0/1156013195 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2ec09f8f0 con 0x7fe2ec0a1330 2024-09-16T03:58:21.718 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.705+0000 7fe2f0e7e640 1 --2- 172.21.15.42:0/1156013195 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe2ec0a1330 0x7fe2ec0a1710 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fe2dc0097c0 tx=0x7fe2dc01c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a2189c597343cd2 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:58:21.719 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.705+0000 7fe2eb7fe640 1 -- 172.21.15.42:0/1156013195 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2dc0b1070 con 0x7fe2ec0a1330 2024-09-16T03:58:21.719 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.705+0000 7fe2eb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:58:21.719 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.705+0000 7fe2eb7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:58:21.719 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.705+0000 7fe2eb7fe640 10 monclient: dump: 2024-09-16T03:58:21.719 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:58:21.719 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:58:21.719 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:21.719 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:21.719 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:58:21.720 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:21.720 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:58:21.720 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:58:21.720 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:58:21.720 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:21.720 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.705+0000 7fe2eb7fe640 10 monclient: _finish_auth 0 2024-09-16T03:58:21.720 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.705+0000 7fe2eb7fe640 10 monclient: _check_auth_tickets 2024-09-16T03:58:21.720 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.705+0000 7fe2eb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:58:21.720 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.705+0000 7fe2eb7fe640 1 -- 172.21.15.42:0/1156013195 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe2dc0a8050 con 0x7fe2ec0a1330 2024-09-16T03:58:21.720 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.705+0000 7fe2f30e2640 5 monclient: authenticate success, global_id 824985 2024-09-16T03:58:21.720 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.705+0000 7fe2f30e2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:58:21.720 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.705+0000 7fe2eb7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:58:21.720 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.705+0000 7fe2f30e2640 10 monclient: get_monmap_and_config success 2024-09-16T03:58:21.720 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.705+0000 7fe2eb7fe640 1 -- 172.21.15.42:0/1156013195 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2dc0ac860 con 0x7fe2ec0a1330 2024-09-16T03:58:21.721 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.705+0000 7fe2eb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:58:21.721 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.705+0000 7fe2f30e2640 4 set_mon_vals no callback set 2024-09-16T03:58:21.721 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.705+0000 7fe2eb7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:58:21.721 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.705+0000 7fe2eb7fe640 10 monclient: dump: 2024-09-16T03:58:21.721 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:58:21.721 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:58:21.721 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:21.721 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:21.721 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:58:21.722 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:21.722 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:58:21.722 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:58:21.722 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:58:21.722 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:21.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.705+0000 7fe2f30e2640 10 monclient: shutdown 2024-09-16T03:58:21.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.705+0000 7fe2f30e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:58:21.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.705+0000 7fe2f30e2640 1 -- 172.21.15.42:0/1156013195 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe2ec0a1330 msgr2=0x7fe2ec0a1710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:21.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.705+0000 7fe2f30e2640 1 --2- 172.21.15.42:0/1156013195 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe2ec0a1330 0x7fe2ec0a1710 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fe2dc0097c0 tx=0x7fe2dc01c3c0 comp rx=0 tx=0).stop 2024-09-16T03:58:21.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.705+0000 7fe2f30e2640 1 -- 172.21.15.42:0/1156013195 shutdown_connections 2024-09-16T03:58:21.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.705+0000 7fe2f30e2640 1 --2- 172.21.15.42:0/1156013195 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe2ec0a1330 0x7fe2ec0a1710 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:21.724 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.705+0000 7fe2f30e2640 1 -- 172.21.15.42:0/1156013195 >> 172.21.15.42:0/1156013195 conn(0x7fe2ec0a2420 msgr2=0x7fe2ec0a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:58:21.724 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.709+0000 7fe2f30e2640 1 -- 172.21.15.42:0/1156013195 shutdown_connections 2024-09-16T03:58:21.724 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.709+0000 7fe2f30e2640 1 -- 172.21.15.42:0/1156013195 wait complete. 2024-09-16T03:58:21.724 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.709+0000 7fe2f30e2640 10 monclient: build_initial_monmap 2024-09-16T03:58:21.724 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.709+0000 7fe2f30e2640 10 monclient: monmap: 2024-09-16T03:58:21.724 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:58:21.725 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:58:21.725 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:58:21.725 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:58:21.725 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:58:21.725 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:21.725 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:58:21.725 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:58:21.726 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:58:21.726 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:21.726 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.709+0000 7fe2f30e2640 1 librados: starting msgr at 2024-09-16T03:58:21.726 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.709+0000 7fe2f30e2640 1 librados: starting objecter 2024-09-16T03:58:21.726 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.709+0000 7fe2f30e2640 1 Processor -- start 2024-09-16T03:58:21.726 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.709+0000 7fe2f30e2640 1 -- start start 2024-09-16T03:58:21.726 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.709+0000 7fe2f30e2640 1 librados: setting wanted keys 2024-09-16T03:58:21.727 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.709+0000 7fe2f30e2640 1 librados: calling monclient init 2024-09-16T03:58:21.727 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.709+0000 7fe2f30e2640 10 monclient: init 2024-09-16T03:58:21.727 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.709+0000 7fe2f30e2640 10 monclient: _reopen_session rank -1 2024-09-16T03:58:21.727 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.709+0000 7fe2f30e2640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T03:58:21.727 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.709+0000 7fe2f30e2640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe2ec0a1330 0x7fe2ec1ccd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:21.727 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.709+0000 7fe2f30e2640 10 monclient(hunting): picked mon.noname-a con 0x7fe2ec0a1330 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:58:21.727 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.709+0000 7fe2f30e2640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe2ec0a1c50 0x7fe2ec13cf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:21.728 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.709+0000 7fe2f30e2640 10 monclient(hunting): picked mon.noname-b con 0x7fe2ec0a1c50 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:58:21.728 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.709+0000 7fe2f30e2640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe2ec1350d0 0x7fe2ec11bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:21.728 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.709+0000 7fe2f30e2640 10 monclient(hunting): picked mon.noname-c con 0x7fe2ec1350d0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:58:21.728 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.709+0000 7fe2f30e2640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:21.728 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.709+0000 7fe2f30e2640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fe2ec09eef0 con 0x7fe2ec0a1330 2024-09-16T03:58:21.728 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.709+0000 7fe2f30e2640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:21.729 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.709+0000 7fe2f30e2640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe2ec09ed70 con 0x7fe2ec0a1c50 2024-09-16T03:58:21.729 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.709+0000 7fe2f30e2640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:21.729 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.709+0000 7fe2f30e2640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fe2ec140b90 con 0x7fe2ec1350d0 2024-09-16T03:58:21.729 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.709+0000 7fe2f30e2640 10 monclient(hunting): _renew_subs 2024-09-16T03:58:21.729 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.709+0000 7fe2f30e2640 10 monclient(hunting): authenticate will time out at 1948.681695s 2024-09-16T03:58:21.729 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.709+0000 7fe2f167f640 1 -- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe2ec1350d0 msgr2=0x7fe2ec11bf70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T03:58:21.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.709+0000 7fe2f167f640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe2ec1350d0 0x7fe2ec11bf70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:58:21.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.709+0000 7fe2f0e7e640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe2ec0a1330 0x7fe2ec1ccd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:21.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.709+0000 7fe2f0e7e640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe2ec0a1330 0x7fe2ec1ccd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:50348/0 (socket says 172.21.15.42:50348) 2024-09-16T03:58:21.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.709+0000 7fe2f0e7e640 1 -- 172.21.15.42:0/2343356202 learned_addr learned my addr 172.21.15.42:0/2343356202 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:58:21.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.709+0000 7fe2f0e7e640 10 monclient(hunting): get_auth_request con 0x7fe2ec0a1330 auth_method 0 2024-09-16T03:58:21.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.709+0000 7fe2f0e7e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:21.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.709+0000 7fe2f0e7e640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:21.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.709+0000 7fe2f0e7e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:58:21.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.709+0000 7fe2f0e7e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:58:21.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.709+0000 7fe2f0e7e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:58:21.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.709+0000 7fe2f0e7e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:58:21.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.709+0000 7fe2f0e7e640 10 monclient(hunting): handle_auth_done global_id 824988 payload 824 2024-09-16T03:58:21.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.709+0000 7fe2ebfff640 1 --2- 172.21.15.42:0/2343356202 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe2ec0a1c50 0x7fe2ec13cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:21.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.709+0000 7fe2f0e7e640 1 -- 172.21.15.42:0/2343356202 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe2ec1350d0 msgr2=0x7fe2ec11bf70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:58:21.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.709+0000 7fe2f0e7e640 1 --2- 172.21.15.42:0/2343356202 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe2ec1350d0 0x7fe2ec11bf70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:21.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.709+0000 7fe2f0e7e640 1 -- 172.21.15.42:0/2343356202 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe2ec0a1c50 msgr2=0x7fe2ec13cf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:21.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.709+0000 7fe2f0e7e640 1 --2- 172.21.15.42:0/2343356202 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe2ec0a1c50 0x7fe2ec13cf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:21.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.709+0000 7fe2f0e7e640 10 monclient: _finish_hunting 0 2024-09-16T03:58:21.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.709+0000 7fe2f0e7e640 1 monclient: found mon.noname-a 2024-09-16T03:58:21.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.709+0000 7fe2f0e7e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:58:21.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.709+0000 7fe2f0e7e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:58:21.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.709+0000 7fe2f0e7e640 1 -- 172.21.15.42:0/2343356202 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2ec13db50 con 0x7fe2ec0a1330 2024-09-16T03:58:21.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.709+0000 7fe2f0e7e640 1 --2- 172.21.15.42:0/2343356202 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe2ec0a1330 0x7fe2ec1ccd20 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fe2dc0af040 tx=0x7fe2dc019160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:58:21.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.709+0000 7fe2e97fa640 10 client.?.objecter ms_handle_connect 0x7fe2ec0a1330 2024-09-16T03:58:21.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.709+0000 7fe2e97fa640 10 client.?.objecter resend_mon_ops 2024-09-16T03:58:21.733 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.709+0000 7fe2e97fa640 1 -- 172.21.15.42:0/2343356202 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2dc0b9020 con 0x7fe2ec0a1330 2024-09-16T03:58:21.733 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.709+0000 7fe2e97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:58:21.733 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.709+0000 7fe2e97fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:58:21.733 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.709+0000 7fe2e97fa640 10 monclient: dump: 2024-09-16T03:58:21.733 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:58:21.733 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:58:21.733 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:21.733 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:21.733 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:58:21.733 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:21.733 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:58:21.733 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:58:21.734 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:58:21.734 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:21.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.709+0000 7fe2e97fa640 10 monclient: _finish_auth 0 2024-09-16T03:58:21.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.709+0000 7fe2e97fa640 10 monclient: _check_auth_tickets 2024-09-16T03:58:21.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.709+0000 7fe2e97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:58:21.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.709+0000 7fe2f30e2640 5 monclient: authenticate success, global_id 824988 2024-09-16T03:58:21.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.709+0000 7fe2e97fa640 1 -- 172.21.15.42:0/2343356202 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe2dc0b5410 con 0x7fe2ec0a1330 2024-09-16T03:58:21.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.709+0000 7fe2e97fa640 10 monclient: handle_config config(0 keys) 2024-09-16T03:58:21.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.709+0000 7fe2f30e2640 10 monclient: _renew_subs 2024-09-16T03:58:21.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.709+0000 7fe2e97fa640 1 -- 172.21.15.42:0/2343356202 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2dc0b1040 con 0x7fe2ec0a1330 2024-09-16T03:58:21.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.709+0000 7fe2f30e2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:58:21.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.709+0000 7fe2f30e2640 1 -- 172.21.15.42:0/2343356202 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2ec06e710 con 0x7fe2ec0a1330 2024-09-16T03:58:21.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.709+0000 7fe2e97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:58:21.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.709+0000 7fe2e97fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:58:21.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.709+0000 7fe2e97fa640 10 monclient: dump: 2024-09-16T03:58:21.735 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:58:21.735 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:58:21.735 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:21.735 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:21.735 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:58:21.736 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:21.736 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:58:21.736 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:58:21.736 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:58:21.736 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:21.736 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.709+0000 7fe2eaffd640 4 set_mon_vals no callback set 2024-09-16T03:58:21.736 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.709+0000 7fe2f30e2640 10 client.824988.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:58:21.736 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.709+0000 7fe2f30e2640 10 monclient: _renew_subs 2024-09-16T03:58:21.736 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.709+0000 7fe2f30e2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:58:21.736 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.709+0000 7fe2f30e2640 1 -- 172.21.15.42:0/2343356202 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2ec073a30 con 0x7fe2ec0a1330 2024-09-16T03:58:21.736 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.709+0000 7fe2f30e2640 1 librados: init done 2024-09-16T03:58:21.736 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.709+0000 7fe2f30e2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:58:21.737 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.709+0000 7fe2f30e2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:58:21.737 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.709+0000 7fe2f30e2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:58:21.737 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.709+0000 7fe2f30e2640 1 -- 172.21.15.42:0/2343356202 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe2ec0a1710 con 0x7fe2ec0a1330 2024-09-16T03:58:21.737 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.709+0000 7fe2e97fa640 1 -- 172.21.15.42:0/2343356202 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fe2dc0bc020 con 0x7fe2ec0a1330 2024-09-16T03:58:21.737 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.709+0000 7fe2e97fa640 1 --2- 172.21.15.42:0/2343356202 >> v2:172.21.15.154:6812/254845342 conn(0x7fe2d80377c0 0x7fe2d8039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:21.737 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.709+0000 7fe2e97fa640 1 -- 172.21.15.42:0/2343356202 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(485..485 src has 366..485) ==== 6560+0+0 (secure 0 0 0) 0x7fe2dc0e2f50 con 0x7fe2ec0a1330 2024-09-16T03:58:21.737 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.709+0000 7fe2e97fa640 10 client.824988.objecter ms_dispatch 0x7fe2ec000ba0 osd_map(485..485 src has 366..485) 2024-09-16T03:58:21.737 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.709+0000 7fe2e97fa640 3 client.824988.objecter handle_osd_map got epochs [485,485] > 0 2024-09-16T03:58:21.738 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.709+0000 7fe2e97fa640 3 client.824988.objecter handle_osd_map decoding full epoch 485 2024-09-16T03:58:21.738 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.709+0000 7fe2e97fa640 20 client.824988.objecter dump_active .. 0 homeless 2024-09-16T03:58:21.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.713+0000 7fe2ebfff640 1 --2- 172.21.15.42:0/2343356202 >> v2:172.21.15.154:6812/254845342 conn(0x7fe2d80377c0 0x7fe2d8039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:21.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.713+0000 7fe2ebfff640 10 monclient: get_auth_request con 0x7fe2d80377c0 auth_method 0 2024-09-16T03:58:21.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.713+0000 7fe2ebfff640 1 --2- 172.21.15.42:0/2343356202 >> v2:172.21.15.154:6812/254845342 conn(0x7fe2d80377c0 0x7fe2d8039c80 secure :-1 s=READY pgs=3699 cs=0 l=1 rev1=1 crypto rx=0x7fe2ec13e340 tx=0x7fe2e0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:58:21.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.713+0000 7fe2e97fa640 10 client.824988.objecter ms_handle_connect 0x7fe2d80377c0 2024-09-16T03:58:21.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.717+0000 7fe2e97fa640 1 -- 172.21.15.42:0/2343356202 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7fe2dc0c3070 con 0x7fe2ec0a1330 2024-09-16T03:58:21.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.717+0000 7fe2e97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:58:21.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.717+0000 7fe2e97fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:58:21.945 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.941+0000 7fe2f30e2640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:58:21.946 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.941+0000 7fe2f30e2640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:58:21.946 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.941+0000 7fe2f30e2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:58:21.946 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.941+0000 7fe2f30e2640 1 -- 172.21.15.42:0/2343356202 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fe2ec069040 con 0x7fe2ec0a1330 2024-09-16T03:58:21.946 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.941+0000 7fe2e97fa640 1 -- 172.21.15.42:0/2343356202 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1562 (secure 0 0 0) 0x7fe2ec06e710 con 0x7fe2ec0a1330 2024-09-16T03:58:21.946 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.941+0000 7fe2e97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:58:21.946 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.941+0000 7fe2e97fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:58:21.947 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":148,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:58:21.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.941+0000 7fe2f30e2640 10 librados: watch_flush enter 2024-09-16T03:58:21.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.941+0000 7fe2f30e2640 10 librados: watch_flush exit 2024-09-16T03:58:21.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.941+0000 7fe2f30e2640 20 client.824988.objecter shutdown clearing up homeless session... 2024-09-16T03:58:21.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.941+0000 7fe2f30e2640 10 client.824988.objecter successfully canceled tick 2024-09-16T03:58:21.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.941+0000 7fe2f30e2640 1 -- 172.21.15.42:0/2343356202 >> v2:172.21.15.154:6812/254845342 conn(0x7fe2d80377c0 msgr2=0x7fe2d8039c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:21.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.941+0000 7fe2f30e2640 1 --2- 172.21.15.42:0/2343356202 >> v2:172.21.15.154:6812/254845342 conn(0x7fe2d80377c0 0x7fe2d8039c80 secure :-1 s=READY pgs=3699 cs=0 l=1 rev1=1 crypto rx=0x7fe2ec13e340 tx=0x7fe2e0009040 comp rx=0 tx=0).stop 2024-09-16T03:58:21.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.941+0000 7fe2f30e2640 10 monclient: shutdown 2024-09-16T03:58:21.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.941+0000 7fe2f30e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:58:21.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.941+0000 7fe2f30e2640 1 -- 172.21.15.42:0/2343356202 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe2ec0a1330 msgr2=0x7fe2ec1ccd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:21.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.941+0000 7fe2f30e2640 1 --2- 172.21.15.42:0/2343356202 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe2ec0a1330 0x7fe2ec1ccd20 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fe2dc0af040 tx=0x7fe2dc019160 comp rx=0 tx=0).stop 2024-09-16T03:58:21.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.941+0000 7fe2f30e2640 1 -- 172.21.15.42:0/2343356202 shutdown_connections 2024-09-16T03:58:21.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.941+0000 7fe2f30e2640 1 --2- 172.21.15.42:0/2343356202 >> v2:172.21.15.154:6812/254845342 conn(0x7fe2d80377c0 0x7fe2d8039c80 unknown :-1 s=CLOSED pgs=3699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:21.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.941+0000 7fe2f30e2640 1 --2- 172.21.15.42:0/2343356202 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe2ec1350d0 0x7fe2ec11bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:21.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.941+0000 7fe2f30e2640 1 --2- 172.21.15.42:0/2343356202 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe2ec0a1c50 0x7fe2ec13cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:21.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.941+0000 7fe2f30e2640 1 --2- 172.21.15.42:0/2343356202 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe2ec0a1330 0x7fe2ec1ccd20 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:21.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.941+0000 7fe2f30e2640 1 -- 172.21.15.42:0/2343356202 >> 172.21.15.42:0/2343356202 conn(0x7fe2ec0a2420 msgr2=0x7fe2ec0a4ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:58:21.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.941+0000 7fe2f30e2640 1 -- 172.21.15.42:0/2343356202 shutdown_connections 2024-09-16T03:58:21.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.941+0000 7fe2f30e2640 1 -- 172.21.15.42:0/2343356202 wait complete. 2024-09-16T03:58:21.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:21.941+0000 7fe2f30e2640 1 librados: shutdown 2024-09-16T03:58:24.966 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:58:25.100 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:25.093+0000 7f60f6c11640 10 monclient: get_monmap_and_config 2024-09-16T03:58:25.100 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:25.093+0000 7f60f6c11640 10 monclient: build_initial_monmap 2024-09-16T03:58:25.100 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:25.093+0000 7f60f6c11640 10 monclient: monmap: 2024-09-16T03:58:25.100 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:58:25.100 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:58:25.100 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:58:25.098508+0000 2024-09-16T03:58:25.100 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:58:25.098508+0000 2024-09-16T03:58:25.100 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:58:25.100 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:25.100 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:58:25.101 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:58:25.101 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:58:25.101 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:58:25.101 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:58:25.101 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:25.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:25.093+0000 7f60f6c11640 1 Processor -- start 2024-09-16T03:58:25.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:25.093+0000 7f60f6c11640 1 -- start start 2024-09-16T03:58:25.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:25.093+0000 7f60f6c11640 10 monclient: init 2024-09-16T03:58:25.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:25.097+0000 7f60f6c11640 10 monclient: _reopen_session rank -1 2024-09-16T03:58:25.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:25.097+0000 7f60f6c11640 10 monclient: _add_conns ranks=[4,2,0,1,3] 2024-09-16T03:58:25.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:25.097+0000 7f60f6c11640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f60f00a39b0 0x7f60f0133b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:25.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:25.097+0000 7f60f6c11640 10 monclient(hunting): picked mon.noname-c con 0x7f60f00a39b0 addr v2:172.21.15.154:3300/0 2024-09-16T03:58:25.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:25.097+0000 7f60f6c11640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f60f00a3d90 0x7f60f0137000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:25.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:25.097+0000 7f60f6c11640 10 monclient(hunting): picked mon.noname-b con 0x7f60f00a3d90 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:58:25.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:25.097+0000 7f60f6c11640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f60f0137540 addr v1:172.21.15.42:6789/0 2024-09-16T03:58:25.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:25.097+0000 7f60f6c11640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f60f009c140 con 0x7f60f0137540 2024-09-16T03:58:25.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:25.097+0000 7f60f6c11640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:25.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:25.097+0000 7f60f6c11640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f60f009c420 con 0x7f60f00a39b0 2024-09-16T03:58:25.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:25.097+0000 7f60f6c11640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:25.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:25.097+0000 7f60f6c11640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f60f006dd60 con 0x7f60f00a3d90 2024-09-16T03:58:25.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:25.097+0000 7f60f6c11640 10 monclient(hunting): _renew_subs 2024-09-16T03:58:25.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:25.097+0000 7f60f6c11640 10 monclient(hunting): authenticate will time out at 1952.068695s 2024-09-16T03:58:25.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:25.097+0000 7f60e7fff640 1 -- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f60f00a3d90 msgr2=0x7f60f0137000 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T03:58:25.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:25.097+0000 7f60e7fff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f60f00a3d90 0x7f60f0137000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:58:25.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:25.097+0000 7f60f51ae640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f60f0137540 0x7f60f013e390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:50086/0 (socket says 172.21.15.42:50086) 2024-09-16T03:58:25.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:25.097+0000 7f60f51ae640 1 -- 172.21.15.42:0/1160615497 learned_addr learned my addr 172.21.15.42:0/1160615497 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:58:25.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:25.097+0000 7f60f49ad640 1 --2- 172.21.15.42:0/1160615497 >> v2:172.21.15.154:3300/0 conn(0x7f60f00a39b0 0x7f60f0133b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:25.105 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:25.097+0000 7f60f49ad640 10 monclient(hunting): get_auth_request con 0x7f60f00a39b0 auth_method 0 2024-09-16T03:58:25.105 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:25.097+0000 7f60f49ad640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:25.105 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:25.097+0000 7f60f49ad640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:25.105 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:25.097+0000 7f60f49ad640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:58:25.105 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:25.097+0000 7f60f49ad640 1 -- 172.21.15.42:0/1160615497 >> v2:172.21.15.154:3300/0 conn(0x7f60f00a39b0 msgr2=0x7f60f0133b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T03:58:25.105 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:25.097+0000 7f60f49ad640 1 -- 172.21.15.42:0/1160615497 >> v2:172.21.15.154:3300/0 conn(0x7f60f00a39b0 msgr2=0x7f60f0133b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:58:25.105 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:25.097+0000 7f60f49ad640 1 --2- 172.21.15.42:0/1160615497 >> v2:172.21.15.154:3300/0 conn(0x7f60f00a39b0 0x7f60f0133b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:58:25.105 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:25.097+0000 7f60f49ad640 1 --2- 172.21.15.42:0/1160615497 >> v2:172.21.15.154:3300/0 conn(0x7f60f00a39b0 0x7f60f0133b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:58:25.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:25.301+0000 7f60e7fff640 1 -- 172.21.15.42:0/1160615497 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f60f00a3d90 msgr2=0x7f60f0137000 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T03:58:25.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:25.301+0000 7f60e7fff640 1 --2- 172.21.15.42:0/1160615497 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f60f00a3d90 0x7f60f0137000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T03:58:25.306 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:25.301+0000 7f60f49ad640 1 --2- 172.21.15.42:0/1160615497 >> v2:172.21.15.154:3300/0 conn(0x7f60f00a39b0 0x7f60f0133b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:25.306 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:25.301+0000 7f60f49ad640 10 monclient(hunting): get_auth_request con 0x7f60f00a39b0 auth_method 0 2024-09-16T03:58:25.306 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:25.301+0000 7f60f49ad640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:25.306 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:25.301+0000 7f60f49ad640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:25.307 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:25.301+0000 7f60f49ad640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:58:25.307 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:25.301+0000 7f60f49ad640 1 -- 172.21.15.42:0/1160615497 >> v2:172.21.15.154:3300/0 conn(0x7f60f00a39b0 msgr2=0x7f60f0133b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T03:58:25.307 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:25.301+0000 7f60f49ad640 1 -- 172.21.15.42:0/1160615497 >> v2:172.21.15.154:3300/0 conn(0x7f60f00a39b0 msgr2=0x7f60f0133b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:58:25.307 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:25.301+0000 7f60f49ad640 1 --2- 172.21.15.42:0/1160615497 >> v2:172.21.15.154:3300/0 conn(0x7f60f00a39b0 0x7f60f0133b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:58:25.307 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:25.301+0000 7f60f49ad640 1 --2- 172.21.15.42:0/1160615497 >> v2:172.21.15.154:3300/0 conn(0x7f60f00a39b0 0x7f60f0133b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T03:58:25.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:25.705+0000 7f60e7fff640 1 -- 172.21.15.42:0/1160615497 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f60f00a3d90 msgr2=0x7f60f0137000 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T03:58:25.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:25.705+0000 7f60e7fff640 1 --2- 172.21.15.42:0/1160615497 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f60f00a3d90 0x7f60f0137000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T03:58:25.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:25.705+0000 7f60f49ad640 1 --2- 172.21.15.42:0/1160615497 >> v2:172.21.15.154:3300/0 conn(0x7f60f00a39b0 0x7f60f0133b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:25.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:25.705+0000 7f60f49ad640 10 monclient(hunting): get_auth_request con 0x7f60f00a39b0 auth_method 0 2024-09-16T03:58:25.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:25.705+0000 7f60f49ad640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:25.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:25.705+0000 7f60f49ad640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:25.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:25.705+0000 7f60f49ad640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:58:25.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:25.705+0000 7f60f49ad640 1 -- 172.21.15.42:0/1160615497 >> v2:172.21.15.154:3300/0 conn(0x7f60f00a39b0 msgr2=0x7f60f0133b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T03:58:25.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:25.705+0000 7f60f49ad640 1 -- 172.21.15.42:0/1160615497 >> v2:172.21.15.154:3300/0 conn(0x7f60f00a39b0 msgr2=0x7f60f0133b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:58:25.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:25.705+0000 7f60f49ad640 1 --2- 172.21.15.42:0/1160615497 >> v2:172.21.15.154:3300/0 conn(0x7f60f00a39b0 0x7f60f0133b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:58:25.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:25.705+0000 7f60f49ad640 1 --2- 172.21.15.42:0/1160615497 >> v2:172.21.15.154:3300/0 conn(0x7f60f00a39b0 0x7f60f0133b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T03:58:26.514 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:26.509+0000 7f60e7fff640 1 -- 172.21.15.42:0/1160615497 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f60f00a3d90 msgr2=0x7f60f0137000 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T03:58:26.514 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:26.509+0000 7f60e7fff640 1 --2- 172.21.15.42:0/1160615497 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f60f00a3d90 0x7f60f0137000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T03:58:26.514 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:26.509+0000 7f60f49ad640 1 --2- 172.21.15.42:0/1160615497 >> v2:172.21.15.154:3300/0 conn(0x7f60f00a39b0 0x7f60f0133b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:26.514 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:26.509+0000 7f60f49ad640 10 monclient(hunting): get_auth_request con 0x7f60f00a39b0 auth_method 0 2024-09-16T03:58:26.514 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:26.509+0000 7f60f49ad640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:26.514 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:26.509+0000 7f60f49ad640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:26.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:26.509+0000 7f60f49ad640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:58:26.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:26.509+0000 7f60f49ad640 1 -- 172.21.15.42:0/1160615497 >> v2:172.21.15.154:3300/0 conn(0x7f60f00a39b0 msgr2=0x7f60f0133b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T03:58:26.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:26.509+0000 7f60f49ad640 1 -- 172.21.15.42:0/1160615497 >> v2:172.21.15.154:3300/0 conn(0x7f60f00a39b0 msgr2=0x7f60f0133b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:58:26.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:26.509+0000 7f60f49ad640 1 --2- 172.21.15.42:0/1160615497 >> v2:172.21.15.154:3300/0 conn(0x7f60f00a39b0 0x7f60f0133b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:58:26.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:26.509+0000 7f60f49ad640 1 --2- 172.21.15.42:0/1160615497 >> v2:172.21.15.154:3300/0 conn(0x7f60f00a39b0 0x7f60f0133b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T03:58:28.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:28.097+0000 7f60e67fc640 10 monclient(hunting): tick 2024-09-16T03:58:28.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:28.097+0000 7f60e67fc640 10 monclient(hunting): _check_auth_tickets 2024-09-16T03:58:28.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:28.097+0000 7f60e67fc640 1 monclient(hunting): continuing hunt 2024-09-16T03:58:28.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:28.097+0000 7f60e67fc640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T03:58:28.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:28.097+0000 7f60e67fc640 1 -- 172.21.15.42:0/1160615497 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f60f00a3d90 msgr2=0x7f60f0137000 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:58:28.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:28.097+0000 7f60e67fc640 1 --2- 172.21.15.42:0/1160615497 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f60f00a3d90 0x7f60f0137000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:28.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:28.097+0000 7f60e67fc640 1 -- 172.21.15.42:0/1160615497 >> v2:172.21.15.154:3300/0 conn(0x7f60f00a39b0 msgr2=0x7f60f0133b80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:58:28.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:28.097+0000 7f60e67fc640 1 --2- 172.21.15.42:0/1160615497 >> v2:172.21.15.154:3300/0 conn(0x7f60f00a39b0 0x7f60f0133b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:28.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:28.097+0000 7f60e67fc640 1 -- 172.21.15.42:0/1160615497 >> v1:172.21.15.42:6789/0 conn(0x7f60f0137540 legacy=0x7f60f013e390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:28.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:28.097+0000 7f60e67fc640 10 monclient: _add_conns ranks=[1,3] 2024-09-16T03:58:28.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:28.097+0000 7f60e67fc640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f60dc0036f0 addr v1:172.21.15.154:6789/0 2024-09-16T03:58:28.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:28.097+0000 7f60e67fc640 1 --2- 172.21.15.42:0/1160615497 >> v2:172.21.15.42:3300/0 conn(0x7f60dc007420 0x7f60dc0098c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:28.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:28.097+0000 7f60e67fc640 10 monclient(hunting): picked mon.noname-a con 0x7f60dc007420 addr v2:172.21.15.42:3300/0 2024-09-16T03:58:28.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:28.097+0000 7f60e67fc640 1 -- 172.21.15.42:0/1160615497 --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f60f00a5e20 con 0x7f60dc0036f0 2024-09-16T03:58:28.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:28.097+0000 7f60e67fc640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:28.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:28.097+0000 7f60e67fc640 1 -- 172.21.15.42:0/1160615497 --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f60f009c420 con 0x7f60dc007420 2024-09-16T03:58:28.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:28.097+0000 7f60e67fc640 10 monclient(hunting): _renew_subs 2024-09-16T03:58:28.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:28.097+0000 7f60e7fff640 1 --2- 172.21.15.42:0/1160615497 >> v2:172.21.15.42:3300/0 conn(0x7f60dc007420 0x7f60dc0098c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:28.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:28.097+0000 7f60e7fff640 10 monclient(hunting): get_auth_request con 0x7f60dc007420 auth_method 0 2024-09-16T03:58:28.105 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:28.097+0000 7f60e7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:28.105 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:28.097+0000 7f60e7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:28.105 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:28.097+0000 7f60e7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:58:28.105 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:28.097+0000 7f60e7fff640 1 -- 172.21.15.42:0/1160615497 >> v2:172.21.15.42:3300/0 conn(0x7f60dc007420 msgr2=0x7f60dc0098c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T03:58:28.105 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:28.097+0000 7f60e7fff640 1 -- 172.21.15.42:0/1160615497 >> v2:172.21.15.42:3300/0 conn(0x7f60dc007420 msgr2=0x7f60dc0098c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:58:28.105 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:28.097+0000 7f60e7fff640 1 --2- 172.21.15.42:0/1160615497 >> v2:172.21.15.42:3300/0 conn(0x7f60dc007420 0x7f60dc0098c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:58:28.105 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:28.097+0000 7f60e7fff640 1 --2- 172.21.15.42:0/1160615497 >> v2:172.21.15.42:3300/0 conn(0x7f60dc007420 0x7f60dc0098c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:58:28.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:28.301+0000 7f60e7fff640 1 --2- 172.21.15.42:0/1160615497 >> v2:172.21.15.42:3300/0 conn(0x7f60dc007420 0x7f60dc0098c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:28.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:28.301+0000 7f60e7fff640 10 monclient(hunting): get_auth_request con 0x7f60dc007420 auth_method 0 2024-09-16T03:58:28.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:28.301+0000 7f60e7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:28.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:28.301+0000 7f60e7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:28.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:28.301+0000 7f60e7fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:58:28.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:28.301+0000 7f60e7fff640 1 -- 172.21.15.42:0/1160615497 >> v2:172.21.15.42:3300/0 conn(0x7f60dc007420 msgr2=0x7f60dc0098c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T03:58:28.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:28.301+0000 7f60e7fff640 1 -- 172.21.15.42:0/1160615497 >> v2:172.21.15.42:3300/0 conn(0x7f60dc007420 msgr2=0x7f60dc0098c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:58:28.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:28.301+0000 7f60e7fff640 1 --2- 172.21.15.42:0/1160615497 >> v2:172.21.15.42:3300/0 conn(0x7f60dc007420 0x7f60dc0098c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:58:28.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:28.301+0000 7f60e7fff640 1 --2- 172.21.15.42:0/1160615497 >> v2:172.21.15.42:3300/0 conn(0x7f60dc007420 0x7f60dc0098c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T03:58:28.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:28.705+0000 7f60e7fff640 1 --2- 172.21.15.42:0/1160615497 >> v2:172.21.15.42:3300/0 conn(0x7f60dc007420 0x7f60dc0098c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:28.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:28.705+0000 7f60e7fff640 10 monclient(hunting): get_auth_request con 0x7f60dc007420 auth_method 0 2024-09-16T03:58:28.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:28.705+0000 7f60e7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:28.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:28.705+0000 7f60e7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:28.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:28.705+0000 7f60e7fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:58:28.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:28.705+0000 7f60e7fff640 1 -- 172.21.15.42:0/1160615497 >> v2:172.21.15.42:3300/0 conn(0x7f60dc007420 msgr2=0x7f60dc0098c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T03:58:28.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:28.705+0000 7f60e7fff640 1 -- 172.21.15.42:0/1160615497 >> v2:172.21.15.42:3300/0 conn(0x7f60dc007420 msgr2=0x7f60dc0098c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:58:28.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:28.705+0000 7f60e7fff640 1 --2- 172.21.15.42:0/1160615497 >> v2:172.21.15.42:3300/0 conn(0x7f60dc007420 0x7f60dc0098c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:58:28.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:28.705+0000 7f60e7fff640 1 --2- 172.21.15.42:0/1160615497 >> v2:172.21.15.42:3300/0 conn(0x7f60dc007420 0x7f60dc0098c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T03:58:29.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.509+0000 7f60e7fff640 1 --2- 172.21.15.42:0/1160615497 >> v2:172.21.15.42:3300/0 conn(0x7f60dc007420 0x7f60dc0098c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:29.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.509+0000 7f60e7fff640 10 monclient(hunting): get_auth_request con 0x7f60dc007420 auth_method 0 2024-09-16T03:58:29.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.509+0000 7f60e7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:29.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.509+0000 7f60e7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:29.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.509+0000 7f60e7fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:58:29.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.509+0000 7f60e7fff640 1 -- 172.21.15.42:0/1160615497 >> v2:172.21.15.42:3300/0 conn(0x7f60dc007420 msgr2=0x7f60dc0098c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T03:58:29.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.509+0000 7f60e7fff640 1 -- 172.21.15.42:0/1160615497 >> v2:172.21.15.42:3300/0 conn(0x7f60dc007420 msgr2=0x7f60dc0098c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:58:29.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.509+0000 7f60e7fff640 1 --2- 172.21.15.42:0/1160615497 >> v2:172.21.15.42:3300/0 conn(0x7f60dc007420 0x7f60dc0098c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:58:29.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.509+0000 7f60e7fff640 1 --2- 172.21.15.42:0/1160615497 >> v2:172.21.15.42:3300/0 conn(0x7f60dc007420 0x7f60dc0098c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T03:58:29.714 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.709+0000 7f60e77fe640 1 -- 172.21.15.42:0/1160615497 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3062012517 0 0) 0x7f60f00a5e20 con 0x7f60dc0036f0 2024-09-16T03:58:29.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.709+0000 7f60e77fe640 10 monclient(hunting): _negotiate 2024-09-16T03:58:29.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.709+0000 7f60e77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:29.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.709+0000 7f60e77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:58:29.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.709+0000 7f60e77fe640 10 monclient(hunting): my global_id is 904165 2024-09-16T03:58:29.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.709+0000 7f60e77fe640 1 -- 172.21.15.42:0/1160615497 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f60d8003530 con 0x7f60dc0036f0 2024-09-16T03:58:29.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.737+0000 7f60e77fe640 1 -- 172.21.15.42:0/1160615497 <== mon.1 v1:172.21.15.154:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3288219723 0 0) 0x7f60d8003530 con 0x7f60dc0036f0 2024-09-16T03:58:29.744 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.737+0000 7f60e77fe640 1 -- 172.21.15.42:0/1160615497 >> v2:172.21.15.42:3300/0 conn(0x7f60dc007420 msgr2=0x7f60dc0098c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:58:29.744 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.737+0000 7f60e77fe640 1 --2- 172.21.15.42:0/1160615497 >> v2:172.21.15.42:3300/0 conn(0x7f60dc007420 0x7f60dc0098c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:29.744 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.737+0000 7f60e77fe640 10 monclient: _finish_hunting 0 2024-09-16T03:58:29.744 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.737+0000 7f60e77fe640 1 monclient: found mon.noname-c-legacy 2024-09-16T03:58:29.744 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.737+0000 7f60e77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:58:29.744 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.737+0000 7f60e77fe640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.154:6789/0 2024-09-16T03:58:29.744 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.737+0000 7f60e77fe640 1 -- 172.21.15.42:0/1160615497 --> v1:172.21.15.154:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60dc00afd0 con 0x7f60dc0036f0 2024-09-16T03:58:29.744 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.737+0000 7f60e77fe640 10 monclient: _finish_auth 0 2024-09-16T03:58:29.744 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.737+0000 7f60e77fe640 10 monclient: _check_auth_tickets 2024-09-16T03:58:29.744 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.737+0000 7f60e77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:58:29.744 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.737+0000 7f60e77fe640 1 -- 172.21.15.42:0/1160615497 <== mon.1 v1:172.21.15.154:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2177413734 0 0) 0x7f60ec00f040 con 0x7f60dc0036f0 2024-09-16T03:58:29.744 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.737+0000 7f60f6c11640 5 monclient: authenticate success, global_id 904165 2024-09-16T03:58:29.744 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.737+0000 7f60e77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:58:29.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.737+0000 7f60e77fe640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-16T03:58:29.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.737+0000 7f60e77fe640 10 monclient: dump: 2024-09-16T03:58:29.745 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:58:29.745 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:58:29.745 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:29.745 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:29.745 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:58:29.745 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:29.745 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:58:29.745 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:58:29.745 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:58:29.745 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:29.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.737+0000 7f60e77fe640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] but i'm connected to v1:172.21.15.154:6789/0, reconnecting 2024-09-16T03:58:29.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.737+0000 7f60e77fe640 10 monclient: _reopen_session rank -1 2024-09-16T03:58:29.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.737+0000 7f60e77fe640 1 -- 172.21.15.42:0/1160615497 >> v1:172.21.15.154:6789/0 conn(0x7f60dc0036f0 legacy=0x7f60dc005b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:29.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.737+0000 7f60e77fe640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:58:29.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.737+0000 7f60e77fe640 1 --2- 172.21.15.42:0/1160615497 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f60f00a3d90 0x7f60d80039c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:29.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.737+0000 7f60f49ad640 1 -- 172.21.15.42:0/1160615497 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f60f00a3d90 msgr2=0x7f60d80039c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T03:58:29.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.737+0000 7f60f49ad640 1 --2- 172.21.15.42:0/1160615497 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f60f00a3d90 0x7f60d80039c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:58:29.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.737+0000 7f60e77fe640 10 monclient(hunting): picked mon.c con 0x7f60f00a3d90 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:58:29.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.737+0000 7f60e77fe640 1 --2- 172.21.15.42:0/1160615497 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f60d8003ff0 0x7f60d8004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:29.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.737+0000 7f60e77fe640 10 monclient(hunting): picked mon.a con 0x7f60d8003ff0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:58:29.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.737+0000 7f60e77fe640 1 --2- 172.21.15.42:0/1160615497 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f60d8004d50 0x7f60d80051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:29.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.737+0000 7f60e77fe640 10 monclient(hunting): picked mon.b con 0x7f60d8004d50 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:58:29.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.737+0000 7f60e77fe640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:29.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.737+0000 7f60e77fe640 1 -- 172.21.15.42:0/1160615497 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f60f009c420 con 0x7f60d8003ff0 2024-09-16T03:58:29.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.737+0000 7f60e77fe640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:29.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.737+0000 7f60e77fe640 1 -- 172.21.15.42:0/1160615497 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f60d800bbe0 con 0x7f60d8004d50 2024-09-16T03:58:29.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.737+0000 7f60e77fe640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:29.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.737+0000 7f60e77fe640 1 -- 172.21.15.42:0/1160615497 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f60d800bde0 con 0x7f60f00a3d90 2024-09-16T03:58:29.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.737+0000 7f60e77fe640 10 monclient(hunting): _renew_subs 2024-09-16T03:58:29.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.737+0000 7f60e77fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:58:29.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.737+0000 7f60f51ae640 1 -- 172.21.15.42:0/1160615497 reap_dead start 2024-09-16T03:58:29.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.737+0000 7f60f6c11640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-16T03:58:29.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.741+0000 7f60e7fff640 1 --2- 172.21.15.42:0/1160615497 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f60d8003ff0 0x7f60d8004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:29.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.741+0000 7f60f51ae640 1 --2- 172.21.15.42:0/1160615497 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f60d8004d50 0x7f60d80051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:29.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.741+0000 7f60f51ae640 10 monclient(hunting): get_auth_request con 0x7f60d8004d50 auth_method 0 2024-09-16T03:58:29.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.741+0000 7f60f51ae640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:29.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.741+0000 7f60f51ae640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:29.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.741+0000 7f60f51ae640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:58:29.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.741+0000 7f60e7fff640 10 monclient(hunting): get_auth_request con 0x7f60d8003ff0 auth_method 0 2024-09-16T03:58:29.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.741+0000 7f60e7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:29.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.741+0000 7f60e7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:29.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.741+0000 7f60e7fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:58:29.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.741+0000 7f60e7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:58:29.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.741+0000 7f60e7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:58:29.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.741+0000 7f60e7fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T03:58:29.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.741+0000 7f60f51ae640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:58:29.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.741+0000 7f60f51ae640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:58:29.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.741+0000 7f60f51ae640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T03:58:29.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.741+0000 7f60e7fff640 10 monclient(hunting): handle_auth_done global_id 904165 payload 293 2024-09-16T03:58:29.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.741+0000 7f60e7fff640 1 -- 172.21.15.42:0/1160615497 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f60f00a3d90 msgr2=0x7f60d80039c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:58:29.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.741+0000 7f60e7fff640 1 --2- 172.21.15.42:0/1160615497 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f60f00a3d90 0x7f60d80039c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:29.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.741+0000 7f60e7fff640 1 -- 172.21.15.42:0/1160615497 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f60d8004d50 msgr2=0x7f60d80051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:58:29.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.741+0000 7f60e7fff640 1 --2- 172.21.15.42:0/1160615497 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f60d8004d50 0x7f60d80051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:29.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.741+0000 7f60e7fff640 10 monclient: _finish_hunting 0 2024-09-16T03:58:29.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.741+0000 7f60e7fff640 1 monclient: found mon.a 2024-09-16T03:58:29.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.741+0000 7f60e7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:58:29.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.741+0000 7f60e7fff640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:58:29.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.741+0000 7f60e7fff640 1 -- 172.21.15.42:0/1160615497 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60d8003530 con 0x7f60d8003ff0 2024-09-16T03:58:29.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.741+0000 7f60e7fff640 10 monclient: _finish_auth 0 2024-09-16T03:58:29.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.741+0000 7f60e7fff640 10 monclient: _check_auth_tickets 2024-09-16T03:58:29.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.741+0000 7f60e7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:58:29.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.741+0000 7f60f51ae640 1 --2- 172.21.15.42:0/1160615497 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f60d8004d50 0x7f60d80051d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:58:29.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.749+0000 7f60e7fff640 1 --2- 172.21.15.42:0/1160615497 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f60d8003ff0 0x7f60d8004810 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f60e0009ed0 tx=0x7f60e001d4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f7049760c0d41a8 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:58:29.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.749+0000 7f60e77fe640 1 -- 172.21.15.42:0/1160615497 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60e00b3070 con 0x7f60d8003ff0 2024-09-16T03:58:29.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.749+0000 7f60e77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:58:29.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.749+0000 7f60e77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:58:29.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.749+0000 7f60e77fe640 10 monclient: dump: 2024-09-16T03:58:29.756 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:58:29.756 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:58:29.756 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:29.756 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:29.756 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:58:29.756 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:29.756 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:58:29.756 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:58:29.756 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:58:29.756 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:29.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.749+0000 7f60e77fe640 1 -- 172.21.15.42:0/1160615497 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f60e00a5e50 con 0x7f60d8003ff0 2024-09-16T03:58:29.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.749+0000 7f60f6c11640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:58:29.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.749+0000 7f60e77fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:58:29.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.749+0000 7f60f6c11640 10 monclient: get_monmap_and_config success 2024-09-16T03:58:29.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.749+0000 7f60f6c11640 4 set_mon_vals no callback set 2024-09-16T03:58:29.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.749+0000 7f60e77fe640 1 -- 172.21.15.42:0/1160615497 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60e0009ad0 con 0x7f60d8003ff0 2024-09-16T03:58:29.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.749+0000 7f60e77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:58:29.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.749+0000 7f60e77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:58:29.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.749+0000 7f60e77fe640 10 monclient: dump: 2024-09-16T03:58:29.757 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:58:29.757 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:58:29.757 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:29.757 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:29.757 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:58:29.757 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:29.757 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:58:29.758 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:58:29.758 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:58:29.758 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:29.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.749+0000 7f60f6c11640 10 monclient: shutdown 2024-09-16T03:58:29.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.749+0000 7f60f6c11640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:58:29.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.749+0000 7f60f6c11640 1 -- 172.21.15.42:0/1160615497 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f60d8003ff0 msgr2=0x7f60d8004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:29.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.749+0000 7f60f6c11640 1 --2- 172.21.15.42:0/1160615497 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f60d8003ff0 0x7f60d8004810 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f60e0009ed0 tx=0x7f60e001d4f0 comp rx=0 tx=0).stop 2024-09-16T03:58:29.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.749+0000 7f60f6c11640 1 -- 172.21.15.42:0/1160615497 shutdown_connections 2024-09-16T03:58:29.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.749+0000 7f60f6c11640 1 --2- 172.21.15.42:0/1160615497 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f60d8004d50 0x7f60d80051d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:29.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.749+0000 7f60f6c11640 1 --2- 172.21.15.42:0/1160615497 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f60f00a3d90 0x7f60d80039c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:29.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.749+0000 7f60f6c11640 1 --2- 172.21.15.42:0/1160615497 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f60d8003ff0 0x7f60d8004810 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:29.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.749+0000 7f60f6c11640 1 -- 172.21.15.42:0/1160615497 >> 172.21.15.42:0/1160615497 conn(0x7f60f00a9fb0 msgr2=0x7f60f00aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:58:29.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.749+0000 7f60f6c11640 1 -- 172.21.15.42:0/1160615497 shutdown_connections 2024-09-16T03:58:29.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.749+0000 7f60f6c11640 1 -- 172.21.15.42:0/1160615497 wait complete. 2024-09-16T03:58:29.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.753+0000 7f60f6c11640 10 monclient: build_initial_monmap 2024-09-16T03:58:29.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.753+0000 7f60f6c11640 10 monclient: monmap: 2024-09-16T03:58:29.760 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:58:29.760 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:58:29.760 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:58:29.760 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:58:29.760 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:58:29.760 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:29.760 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:58:29.760 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:58:29.760 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:58:29.760 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:29.761 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.753+0000 7f60f6c11640 1 librados: starting msgr at 2024-09-16T03:58:29.761 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.753+0000 7f60f6c11640 1 librados: starting objecter 2024-09-16T03:58:29.761 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.753+0000 7f60f6c11640 1 Processor -- start 2024-09-16T03:58:29.761 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.753+0000 7f60f6c11640 1 -- start start 2024-09-16T03:58:29.761 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.753+0000 7f60f6c11640 1 librados: setting wanted keys 2024-09-16T03:58:29.761 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.753+0000 7f60f6c11640 1 librados: calling monclient init 2024-09-16T03:58:29.761 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.753+0000 7f60f6c11640 10 monclient: init 2024-09-16T03:58:29.765 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.753+0000 7f60f6c11640 10 monclient: _reopen_session rank -1 2024-09-16T03:58:29.765 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.753+0000 7f60f6c11640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:58:29.766 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.753+0000 7f60f6c11640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f60d8004d50 0x7f60f01cfa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:29.766 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.753+0000 7f60f6c11640 10 monclient(hunting): picked mon.noname-b con 0x7f60d8004d50 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:58:29.766 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.753+0000 7f60f6c11640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f60f00a3d90 0x7f60f01cff60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:29.766 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.753+0000 7f60f6c11640 10 monclient(hunting): picked mon.noname-a con 0x7f60f00a3d90 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:58:29.766 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.753+0000 7f60f6c11640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f60f01d04a0 0x7f60f0120200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:29.766 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.753+0000 7f60f6c11640 10 monclient(hunting): picked mon.noname-c con 0x7f60f01d04a0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:58:29.766 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.753+0000 7f60f6c11640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:29.766 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.753+0000 7f60f6c11640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f60f013a580 con 0x7f60f00a3d90 2024-09-16T03:58:29.766 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.753+0000 7f60f6c11640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:29.766 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.753+0000 7f60f6c11640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f60f013a700 con 0x7f60d8004d50 2024-09-16T03:58:29.766 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.753+0000 7f60f6c11640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:29.766 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.753+0000 7f60f6c11640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f60f013a880 con 0x7f60f01d04a0 2024-09-16T03:58:29.766 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.753+0000 7f60f6c11640 10 monclient(hunting): _renew_subs 2024-09-16T03:58:29.766 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.753+0000 7f60f6c11640 10 monclient(hunting): authenticate will time out at 1956.726613s 2024-09-16T03:58:29.767 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.753+0000 7f60f51ae640 1 -- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f60f01d04a0 msgr2=0x7f60f0120200 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T03:58:29.767 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.753+0000 7f60f51ae640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f60f01d04a0 0x7f60f0120200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:58:29.767 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.753+0000 7f60e7fff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f60f00a3d90 0x7f60f01cff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:29.767 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.753+0000 7f60e7fff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f60f00a3d90 0x7f60f01cff60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:40130/0 (socket says 172.21.15.42:40130) 2024-09-16T03:58:29.767 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.753+0000 7f60e7fff640 1 -- 172.21.15.42:0/4293207215 learned_addr learned my addr 172.21.15.42:0/4293207215 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:58:29.767 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.753+0000 7f60e7fff640 10 monclient(hunting): get_auth_request con 0x7f60f00a3d90 auth_method 0 2024-09-16T03:58:29.767 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.753+0000 7f60e7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:29.767 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.753+0000 7f60e7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:29.767 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.753+0000 7f60e7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:58:29.767 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.753+0000 7f60e7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:58:29.767 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.753+0000 7f60e7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:58:29.767 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.753+0000 7f60e7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:58:29.767 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.753+0000 7f60f49ad640 1 --2- 172.21.15.42:0/4293207215 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f60d8004d50 0x7f60f01cfa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:29.768 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.753+0000 7f60f49ad640 10 monclient(hunting): get_auth_request con 0x7f60d8004d50 auth_method 0 2024-09-16T03:58:29.768 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.753+0000 7f60f49ad640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:29.768 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.753+0000 7f60f49ad640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:29.768 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.753+0000 7f60f49ad640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:58:29.768 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.753+0000 7f60e7fff640 10 monclient(hunting): handle_auth_done global_id 824991 payload 824 2024-09-16T03:58:29.768 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.753+0000 7f60e7fff640 1 -- 172.21.15.42:0/4293207215 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f60f01d04a0 msgr2=0x7f60f0120200 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:58:29.768 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.753+0000 7f60e7fff640 1 --2- 172.21.15.42:0/4293207215 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f60f01d04a0 0x7f60f0120200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:29.768 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.753+0000 7f60e7fff640 1 -- 172.21.15.42:0/4293207215 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f60d8004d50 msgr2=0x7f60f01cfa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:29.768 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.753+0000 7f60e7fff640 1 --2- 172.21.15.42:0/4293207215 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f60d8004d50 0x7f60f01cfa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:29.768 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.753+0000 7f60e7fff640 10 monclient: _finish_hunting 0 2024-09-16T03:58:29.768 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.753+0000 7f60e7fff640 1 monclient: found mon.noname-a 2024-09-16T03:58:29.768 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.753+0000 7f60e7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:58:29.768 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.753+0000 7f60e7fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:58:29.768 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.753+0000 7f60e7fff640 1 -- 172.21.15.42:0/4293207215 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60f01228c0 con 0x7f60f00a3d90 2024-09-16T03:58:29.769 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.753+0000 7f60f49ad640 1 --2- 172.21.15.42:0/4293207215 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f60d8004d50 0x7f60f01cfa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T03:58:29.769 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.753+0000 7f60e7fff640 1 --2- 172.21.15.42:0/4293207215 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f60f00a3d90 0x7f60f01cff60 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f60e001d320 tx=0x7f60e0005040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:58:29.769 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.753+0000 7f60e57fa640 10 client.?.objecter ms_handle_connect 0x7f60f00a3d90 2024-09-16T03:58:29.769 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.753+0000 7f60e57fa640 10 client.?.objecter resend_mon_ops 2024-09-16T03:58:29.769 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.753+0000 7f60e57fa640 1 -- 172.21.15.42:0/4293207215 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60e00b3040 con 0x7f60f00a3d90 2024-09-16T03:58:29.769 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.753+0000 7f60e57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:58:29.769 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.753+0000 7f60e57fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:58:29.769 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.753+0000 7f60e57fa640 10 monclient: dump: 2024-09-16T03:58:29.769 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:58:29.769 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:58:29.769 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:29.769 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:29.769 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:58:29.769 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:29.770 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:58:29.770 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:58:29.770 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:58:29.770 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:29.770 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.753+0000 7f60e57fa640 10 monclient: _finish_auth 0 2024-09-16T03:58:29.770 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.753+0000 7f60e57fa640 10 monclient: _check_auth_tickets 2024-09-16T03:58:29.770 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.753+0000 7f60e57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:58:29.770 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.753+0000 7f60e57fa640 1 -- 172.21.15.42:0/4293207215 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f60e00c0070 con 0x7f60f00a3d90 2024-09-16T03:58:29.770 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.753+0000 7f60e57fa640 10 monclient: handle_config config(0 keys) 2024-09-16T03:58:29.770 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.753+0000 7f60e57fa640 1 -- 172.21.15.42:0/4293207215 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60e0009ad0 con 0x7f60f00a3d90 2024-09-16T03:58:29.770 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.753+0000 7f60e57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:58:29.770 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.753+0000 7f60e57fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:58:29.770 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.753+0000 7f60e57fa640 10 monclient: dump: 2024-09-16T03:58:29.770 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:58:29.771 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:58:29.771 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:29.771 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:29.771 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:58:29.771 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:29.771 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:58:29.771 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:58:29.771 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:58:29.771 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:29.771 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.753+0000 7f60f6c11640 5 monclient: authenticate success, global_id 824991 2024-09-16T03:58:29.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.753+0000 7f60f6c11640 10 monclient: _renew_subs 2024-09-16T03:58:29.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.757+0000 7f60f6c11640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:58:29.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.757+0000 7f60f6c11640 1 -- 172.21.15.42:0/4293207215 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60f00a8fa0 con 0x7f60f00a3d90 2024-09-16T03:58:29.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.753+0000 7f60e6ffd640 4 set_mon_vals no callback set 2024-09-16T03:58:29.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.757+0000 7f60f6c11640 10 client.824991.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:58:29.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.757+0000 7f60f6c11640 10 monclient: _renew_subs 2024-09-16T03:58:29.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.757+0000 7f60f6c11640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:58:29.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.757+0000 7f60f6c11640 1 -- 172.21.15.42:0/4293207215 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60f0121410 con 0x7f60f00a3d90 2024-09-16T03:58:29.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.757+0000 7f60f6c11640 1 librados: init done 2024-09-16T03:58:29.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.757+0000 7f60e57fa640 1 -- 172.21.15.42:0/4293207215 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f60e00b6070 con 0x7f60f00a3d90 2024-09-16T03:58:29.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.757+0000 7f60f6c11640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:58:29.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.757+0000 7f60f6c11640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:58:29.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.757+0000 7f60f6c11640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:58:29.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.757+0000 7f60f6c11640 1 -- 172.21.15.42:0/4293207215 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f60f013aa00 con 0x7f60f00a3d90 2024-09-16T03:58:29.773 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.757+0000 7f60e57fa640 1 --2- 172.21.15.42:0/4293207215 >> v2:172.21.15.154:6812/254845342 conn(0x7f60dc0384d0 0x7f60dc03a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:29.773 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.757+0000 7f60e57fa640 1 -- 172.21.15.42:0/4293207215 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(486..486 src has 366..486) ==== 6573+0+0 (secure 0 0 0) 0x7f60e00e4210 con 0x7f60f00a3d90 2024-09-16T03:58:29.773 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.757+0000 7f60e57fa640 10 client.824991.objecter ms_dispatch 0x7f60f0000ba0 osd_map(486..486 src has 366..486) 2024-09-16T03:58:29.773 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.757+0000 7f60e57fa640 3 client.824991.objecter handle_osd_map got epochs [486,486] > 0 2024-09-16T03:58:29.773 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.757+0000 7f60e57fa640 3 client.824991.objecter handle_osd_map decoding full epoch 486 2024-09-16T03:58:29.773 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.757+0000 7f60e57fa640 20 client.824991.objecter dump_active .. 0 homeless 2024-09-16T03:58:29.774 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.757+0000 7f60f49ad640 1 --2- 172.21.15.42:0/4293207215 >> v2:172.21.15.154:6812/254845342 conn(0x7f60dc0384d0 0x7f60dc03a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:29.774 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.757+0000 7f60f49ad640 10 monclient: get_auth_request con 0x7f60dc0384d0 auth_method 0 2024-09-16T03:58:29.774 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.757+0000 7f60e57fa640 10 client.824991.objecter ms_handle_connect 0x7f60dc0384d0 2024-09-16T03:58:29.774 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.757+0000 7f60f49ad640 1 --2- 172.21.15.42:0/4293207215 >> v2:172.21.15.154:6812/254845342 conn(0x7f60dc0384d0 0x7f60dc03a990 secure :-1 s=READY pgs=3701 cs=0 l=1 rev1=1 crypto rx=0x7f60ec000eb0 tx=0x7f60ec00f740 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:58:29.774 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.765+0000 7f60e57fa640 1 -- 172.21.15.42:0/4293207215 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f60e0017500 con 0x7f60f00a3d90 2024-09-16T03:58:29.774 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.765+0000 7f60e57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:58:29.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.765+0000 7f60e57fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:58:29.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.993+0000 7f60f6c11640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:58:29.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.993+0000 7f60f6c11640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:58:29.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.993+0000 7f60f6c11640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:58:29.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.993+0000 7f60f6c11640 1 -- 172.21.15.42:0/4293207215 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f60f00a5750 con 0x7f60f00a3d90 2024-09-16T03:58:29.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.993+0000 7f60e57fa640 1 -- 172.21.15.42:0/4293207215 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1555 (secure 0 0 0) 0x7f60e00039c0 con 0x7f60f00a3d90 2024-09-16T03:58:29.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.993+0000 7f60e57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:58:29.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.993+0000 7f60e57fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:58:30.000 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":152,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:58:30.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.993+0000 7f60f6c11640 10 librados: watch_flush enter 2024-09-16T03:58:30.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.997+0000 7f60f6c11640 10 librados: watch_flush exit 2024-09-16T03:58:30.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.997+0000 7f60f6c11640 20 client.824991.objecter shutdown clearing up homeless session... 2024-09-16T03:58:30.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.997+0000 7f60f6c11640 10 client.824991.objecter successfully canceled tick 2024-09-16T03:58:30.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.997+0000 7f60f6c11640 1 -- 172.21.15.42:0/4293207215 >> v2:172.21.15.154:6812/254845342 conn(0x7f60dc0384d0 msgr2=0x7f60dc03a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:30.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.997+0000 7f60f6c11640 1 --2- 172.21.15.42:0/4293207215 >> v2:172.21.15.154:6812/254845342 conn(0x7f60dc0384d0 0x7f60dc03a990 secure :-1 s=READY pgs=3701 cs=0 l=1 rev1=1 crypto rx=0x7f60ec000eb0 tx=0x7f60ec00f740 comp rx=0 tx=0).stop 2024-09-16T03:58:30.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.997+0000 7f60f6c11640 10 monclient: shutdown 2024-09-16T03:58:30.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.997+0000 7f60f6c11640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:58:30.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.997+0000 7f60f6c11640 1 -- 172.21.15.42:0/4293207215 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f60f00a3d90 msgr2=0x7f60f01cff60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:30.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.997+0000 7f60f6c11640 1 --2- 172.21.15.42:0/4293207215 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f60f00a3d90 0x7f60f01cff60 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f60e001d320 tx=0x7f60e0005040 comp rx=0 tx=0).stop 2024-09-16T03:58:30.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.997+0000 7f60f6c11640 1 -- 172.21.15.42:0/4293207215 shutdown_connections 2024-09-16T03:58:30.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.997+0000 7f60f6c11640 1 --2- 172.21.15.42:0/4293207215 >> v2:172.21.15.154:6812/254845342 conn(0x7f60dc0384d0 0x7f60dc03a990 unknown :-1 s=CLOSED pgs=3701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:30.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.997+0000 7f60f6c11640 1 --2- 172.21.15.42:0/4293207215 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f60f01d04a0 0x7f60f0120200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:30.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.997+0000 7f60f6c11640 1 --2- 172.21.15.42:0/4293207215 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f60f00a3d90 0x7f60f01cff60 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:30.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.997+0000 7f60f6c11640 1 --2- 172.21.15.42:0/4293207215 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f60d8004d50 0x7f60f01cfa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:30.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.997+0000 7f60f6c11640 1 -- 172.21.15.42:0/4293207215 >> 172.21.15.42:0/4293207215 conn(0x7f60f00a9fb0 msgr2=0x7f60f01368a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:58:30.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.997+0000 7f60f6c11640 1 -- 172.21.15.42:0/4293207215 shutdown_connections 2024-09-16T03:58:30.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.997+0000 7f60f6c11640 1 -- 172.21.15.42:0/4293207215 wait complete. 2024-09-16T03:58:30.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:29.997+0000 7f60f6c11640 1 librados: shutdown 2024-09-16T03:58:30.021 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T03:58:30.021 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-16T03:58:30.021 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T03:58:30.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.133+0000 7f7f9ee8c640 10 monclient: get_monmap_and_config 2024-09-16T03:58:30.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.133+0000 7f7f9ee8c640 10 monclient: build_initial_monmap 2024-09-16T03:58:30.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.133+0000 7f7f9ee8c640 10 monclient: monmap: 2024-09-16T03:58:30.138 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:58:30.138 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:58:30.138 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:58:30.136826+0000 2024-09-16T03:58:30.138 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:58:30.136826+0000 2024-09-16T03:58:30.138 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:58:30.139 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:30.139 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:58:30.139 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:58:30.139 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:58:30.139 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:58:30.139 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:58:30.139 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:30.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.133+0000 7f7f9ee8c640 1 Processor -- start 2024-09-16T03:58:30.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.133+0000 7f7f9ee8c640 1 -- start start 2024-09-16T03:58:30.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.133+0000 7f7f9ee8c640 10 monclient: init 2024-09-16T03:58:30.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.133+0000 7f7f9ee8c640 10 monclient: _reopen_session rank -1 2024-09-16T03:58:30.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.133+0000 7f7f9ee8c640 10 monclient: _add_conns ranks=[1,4,2,0,3] 2024-09-16T03:58:30.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.133+0000 7f7f9ee8c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7f980a1330 addr v1:172.21.15.154:6789/0 2024-09-16T03:58:30.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.133+0000 7f7f9ee8c640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f7f980a1710 0x7f7f980a1b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:30.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.133+0000 7f7f9ee8c640 10 monclient(hunting): picked mon.noname-c con 0x7f7f980a1710 addr v2:172.21.15.154:3300/0 2024-09-16T03:58:30.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.133+0000 7f7f9ee8c640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7f981393a0 0x7f7f9813b790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:30.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.133+0000 7f7f9ee8c640 10 monclient(hunting): picked mon.noname-b con 0x7f7f981393a0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:58:30.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.133+0000 7f7f9ee8c640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7f9809e4c0 con 0x7f7f980a1330 2024-09-16T03:58:30.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.133+0000 7f7f9ee8c640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:30.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.133+0000 7f7f9ee8c640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f7f9809edf0 con 0x7f7f980a1710 2024-09-16T03:58:30.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.133+0000 7f7f9ee8c640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:30.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.133+0000 7f7f9ee8c640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f7f9809cd60 con 0x7f7f981393a0 2024-09-16T03:58:30.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.133+0000 7f7f9ee8c640 10 monclient(hunting): _renew_subs 2024-09-16T03:58:30.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.133+0000 7f7f9ee8c640 10 monclient(hunting): authenticate will time out at 1957.106289s 2024-09-16T03:58:30.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.133+0000 7f7f9d429640 1 -- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7f981393a0 msgr2=0x7f7f9813b790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T03:58:30.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.133+0000 7f7f9d429640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7f981393a0 0x7f7f9813b790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:58:30.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.133+0000 7f7f97fff640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f7f980a1710 0x7f7f980a1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:30.143 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.133+0000 7f7f97fff640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f7f980a1710 0x7f7f980a1b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.42:48056/0 (socket says 172.21.15.42:48056) 2024-09-16T03:58:30.143 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.133+0000 7f7f97fff640 1 -- 172.21.15.42:0/922844797 learned_addr learned my addr 172.21.15.42:0/922844797 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:58:30.143 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.133+0000 7f7f97fff640 10 monclient(hunting): get_auth_request con 0x7f7f980a1710 auth_method 0 2024-09-16T03:58:30.143 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.133+0000 7f7f97fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:30.143 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.133+0000 7f7f97fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:30.143 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.133+0000 7f7f97fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:58:30.143 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.133+0000 7f7f97fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:58:30.143 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.133+0000 7f7f97fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:58:30.143 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.133+0000 7f7f97fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:58:30.143 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.133+0000 7f7f97fff640 10 monclient(hunting): handle_auth_done global_id 904171 payload 274 2024-09-16T03:58:30.143 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.133+0000 7f7f97fff640 1 -- 172.21.15.42:0/922844797 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7f981393a0 msgr2=0x7f7f9813b790 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:58:30.143 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.133+0000 7f7f97fff640 1 --2- 172.21.15.42:0/922844797 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7f981393a0 0x7f7f9813b790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:30.143 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.133+0000 7f7f97fff640 1 -- 172.21.15.42:0/922844797 >> v1:172.21.15.154:6789/0 conn(0x7f7f980a1330 legacy=0x7f7f9809f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:30.143 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.133+0000 7f7f97fff640 10 monclient: _finish_hunting 0 2024-09-16T03:58:30.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.133+0000 7f7f97fff640 1 monclient: found mon.noname-c 2024-09-16T03:58:30.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.133+0000 7f7f97fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:58:30.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.133+0000 7f7f97fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.154:3300/0 2024-09-16T03:58:30.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.133+0000 7f7f97fff640 1 -- 172.21.15.42:0/922844797 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f980a2080 con 0x7f7f980a1710 2024-09-16T03:58:30.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.137+0000 7f7f97fff640 1 --2- 172.21.15.42:0/922844797 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7f980a1710 0x7f7f980a1b40 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f7f8800bb10 tx=0x7f7f8801eb10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=48eb55d700f5268b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:58:30.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.137+0000 7f7f977fe640 1 -- 172.21.15.42:0/922844797 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f880b4070 con 0x7f7f980a1710 2024-09-16T03:58:30.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.137+0000 7f7f977fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:58:30.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.137+0000 7f7f977fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:58:30.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.137+0000 7f7f977fe640 10 monclient: dump: 2024-09-16T03:58:30.151 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:58:30.151 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:58:30.151 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:30.151 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:30.151 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:58:30.151 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:30.151 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:58:30.152 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:58:30.152 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:58:30.152 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:30.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.137+0000 7f7f977fe640 10 monclient: _finish_auth 0 2024-09-16T03:58:30.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.137+0000 7f7f977fe640 10 monclient: _check_auth_tickets 2024-09-16T03:58:30.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.137+0000 7f7f977fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:58:30.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.137+0000 7f7f977fe640 1 -- 172.21.15.42:0/922844797 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f880a79f0 con 0x7f7f980a1710 2024-09-16T03:58:30.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.137+0000 7f7f977fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:58:30.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.137+0000 7f7f977fe640 1 -- 172.21.15.42:0/922844797 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f880a7dc0 con 0x7f7f980a1710 2024-09-16T03:58:30.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.137+0000 7f7f9ee8c640 5 monclient: authenticate success, global_id 904171 2024-09-16T03:58:30.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.137+0000 7f7f9ee8c640 10 monclient: get_monmap_and_config success 2024-09-16T03:58:30.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.137+0000 7f7f9ee8c640 4 set_mon_vals no callback set 2024-09-16T03:58:30.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.137+0000 7f7f977fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:58:30.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.137+0000 7f7f977fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:58:30.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.137+0000 7f7f977fe640 10 monclient: dump: 2024-09-16T03:58:30.153 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:58:30.153 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:58:30.153 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:30.153 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:30.153 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:58:30.153 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:30.153 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:58:30.153 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:58:30.153 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:58:30.153 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:30.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.137+0000 7f7f9ee8c640 10 monclient: shutdown 2024-09-16T03:58:30.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.137+0000 7f7f9ee8c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:58:30.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.137+0000 7f7f9ee8c640 1 -- 172.21.15.42:0/922844797 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7f980a1710 msgr2=0x7f7f980a1b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:30.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.137+0000 7f7f9ee8c640 1 --2- 172.21.15.42:0/922844797 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7f980a1710 0x7f7f980a1b40 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f7f8800bb10 tx=0x7f7f8801eb10 comp rx=0 tx=0).stop 2024-09-16T03:58:30.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.137+0000 7f7f9ee8c640 1 -- 172.21.15.42:0/922844797 shutdown_connections 2024-09-16T03:58:30.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.137+0000 7f7f9ee8c640 1 --2- 172.21.15.42:0/922844797 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7f981393a0 0x7f7f9813b790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:30.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.137+0000 7f7f9ee8c640 1 --2- 172.21.15.42:0/922844797 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7f980a1710 0x7f7f980a1b40 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:30.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.137+0000 7f7f9ee8c640 1 -- 172.21.15.42:0/922844797 >> 172.21.15.42:0/922844797 conn(0x7f7f980a2420 msgr2=0x7f7f980a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:58:30.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.137+0000 7f7f9ee8c640 1 -- 172.21.15.42:0/922844797 shutdown_connections 2024-09-16T03:58:30.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.137+0000 7f7f9ee8c640 1 -- 172.21.15.42:0/922844797 wait complete. 2024-09-16T03:58:30.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.137+0000 7f7f9ee8c640 10 monclient: build_initial_monmap 2024-09-16T03:58:30.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.137+0000 7f7f9ee8c640 10 monclient: monmap: 2024-09-16T03:58:30.154 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:58:30.154 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:58:30.154 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:58:30.154 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:58:30.155 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:58:30.155 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:30.155 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:58:30.155 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:58:30.155 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:58:30.155 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:30.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.137+0000 7f7f9ee8c640 1 librados: starting msgr at 2024-09-16T03:58:30.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.137+0000 7f7f9ee8c640 1 librados: starting objecter 2024-09-16T03:58:30.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.137+0000 7f7f9ee8c640 1 Processor -- start 2024-09-16T03:58:30.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.137+0000 7f7f9ee8c640 1 -- start start 2024-09-16T03:58:30.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.137+0000 7f7f9ee8c640 1 librados: setting wanted keys 2024-09-16T03:58:30.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.137+0000 7f7f9ee8c640 1 librados: calling monclient init 2024-09-16T03:58:30.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.137+0000 7f7f9ee8c640 10 monclient: init 2024-09-16T03:58:30.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.141+0000 7f7f9ee8c640 10 monclient: _reopen_session rank -1 2024-09-16T03:58:30.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.141+0000 7f7f9ee8c640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:58:30.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.141+0000 7f7f9ee8c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7f980a1330 0x7f7f9811bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:30.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.141+0000 7f7f9ee8c640 10 monclient(hunting): picked mon.noname-a con 0x7f7f980a1330 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:58:30.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.141+0000 7f7f9ee8c640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7f980a1710 0x7f7f9811c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:30.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.141+0000 7f7f9ee8c640 10 monclient(hunting): picked mon.noname-c con 0x7f7f980a1710 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:58:30.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.141+0000 7f7f9ee8c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7f981393a0 0x7f7f9811c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:30.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.141+0000 7f7f9ee8c640 10 monclient(hunting): picked mon.noname-b con 0x7f7f981393a0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:58:30.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.141+0000 7f7f9ee8c640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:30.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.141+0000 7f7f9ee8c640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f7f9809c670 con 0x7f7f980a1330 2024-09-16T03:58:30.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.141+0000 7f7f9ee8c640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:30.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.141+0000 7f7f9ee8c640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f7f9809c4f0 con 0x7f7f981393a0 2024-09-16T03:58:30.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.141+0000 7f7f9ee8c640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:30.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.141+0000 7f7f9ee8c640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f7f9809c7f0 con 0x7f7f980a1710 2024-09-16T03:58:30.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.141+0000 7f7f9ee8c640 10 monclient(hunting): _renew_subs 2024-09-16T03:58:30.157 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.141+0000 7f7f9ee8c640 10 monclient(hunting): authenticate will time out at 1957.112311s 2024-09-16T03:58:30.157 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.141+0000 7f7f97fff640 1 -- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7f980a1710 msgr2=0x7f7f9811c4b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T03:58:30.157 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.141+0000 7f7f97fff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7f980a1710 0x7f7f9811c4b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:58:30.157 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.141+0000 7f7f9cc28640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7f980a1330 0x7f7f9811bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:30.157 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.141+0000 7f7f9cc28640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7f980a1330 0x7f7f9811bf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:40134/0 (socket says 172.21.15.42:40134) 2024-09-16T03:58:30.157 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.141+0000 7f7f9cc28640 1 -- 172.21.15.42:0/4093917873 learned_addr learned my addr 172.21.15.42:0/4093917873 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:58:30.157 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.141+0000 7f7f9cc28640 10 monclient(hunting): get_auth_request con 0x7f7f980a1330 auth_method 0 2024-09-16T03:58:30.157 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.141+0000 7f7f9cc28640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:30.157 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.141+0000 7f7f9cc28640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:30.157 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.141+0000 7f7f9cc28640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:58:30.157 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.141+0000 7f7f9cc28640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:58:30.157 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.141+0000 7f7f9cc28640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:58:30.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.141+0000 7f7f9cc28640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:58:30.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.141+0000 7f7f9d429640 1 --2- 172.21.15.42:0/4093917873 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7f981393a0 0x7f7f9811c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:30.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.141+0000 7f7f9d429640 10 monclient(hunting): get_auth_request con 0x7f7f981393a0 auth_method 0 2024-09-16T03:58:30.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.141+0000 7f7f9d429640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:30.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.141+0000 7f7f9d429640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:30.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.141+0000 7f7f9d429640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:58:30.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.141+0000 7f7f9cc28640 10 monclient(hunting): handle_auth_done global_id 824994 payload 824 2024-09-16T03:58:30.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.141+0000 7f7f9cc28640 1 -- 172.21.15.42:0/4093917873 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7f980a1710 msgr2=0x7f7f9811c4b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:58:30.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.141+0000 7f7f9cc28640 1 --2- 172.21.15.42:0/4093917873 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7f980a1710 0x7f7f9811c4b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:30.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.141+0000 7f7f9cc28640 1 -- 172.21.15.42:0/4093917873 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7f981393a0 msgr2=0x7f7f9811c9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:30.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.141+0000 7f7f9cc28640 1 --2- 172.21.15.42:0/4093917873 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7f981393a0 0x7f7f9811c9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:30.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.141+0000 7f7f9cc28640 10 monclient: _finish_hunting 0 2024-09-16T03:58:30.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.141+0000 7f7f9cc28640 1 monclient: found mon.noname-a 2024-09-16T03:58:30.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.141+0000 7f7f9cc28640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:58:30.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.141+0000 7f7f9cc28640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:58:30.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.141+0000 7f7f9cc28640 1 -- 172.21.15.42:0/4093917873 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f9811ee80 con 0x7f7f980a1330 2024-09-16T03:58:30.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.141+0000 7f7f9d429640 1 --2- 172.21.15.42:0/4093917873 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7f981393a0 0x7f7f9811c9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T03:58:30.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.141+0000 7f7f9cc28640 1 --2- 172.21.15.42:0/4093917873 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7f980a1330 0x7f7f9811bf70 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f7f8c00c500 tx=0x7f7f8c00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:58:30.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.141+0000 7f7f957fa640 10 client.?.objecter ms_handle_connect 0x7f7f980a1330 2024-09-16T03:58:30.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.141+0000 7f7f957fa640 10 client.?.objecter resend_mon_ops 2024-09-16T03:58:30.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.141+0000 7f7f957fa640 1 -- 172.21.15.42:0/4093917873 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f8c016020 con 0x7f7f980a1330 2024-09-16T03:58:30.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.141+0000 7f7f957fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:58:30.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.141+0000 7f7f957fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:58:30.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.141+0000 7f7f957fa640 10 monclient: dump: 2024-09-16T03:58:30.159 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:58:30.159 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:58:30.159 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:30.160 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:30.160 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:58:30.160 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:30.160 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:58:30.160 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:58:30.160 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:58:30.160 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:30.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.141+0000 7f7f957fa640 10 monclient: _finish_auth 0 2024-09-16T03:58:30.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.141+0000 7f7f957fa640 10 monclient: _check_auth_tickets 2024-09-16T03:58:30.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.141+0000 7f7f957fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:58:30.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.141+0000 7f7f957fa640 1 -- 172.21.15.42:0/4093917873 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f8c005220 con 0x7f7f980a1330 2024-09-16T03:58:30.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.141+0000 7f7f9ee8c640 5 monclient: authenticate success, global_id 824994 2024-09-16T03:58:30.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.141+0000 7f7f9ee8c640 10 monclient: _renew_subs 2024-09-16T03:58:30.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.141+0000 7f7f9ee8c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:58:30.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.141+0000 7f7f9ee8c640 1 -- 172.21.15.42:0/4093917873 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f98139980 con 0x7f7f980a1330 2024-09-16T03:58:30.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.141+0000 7f7f957fa640 10 monclient: handle_config config(0 keys) 2024-09-16T03:58:30.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.141+0000 7f7f957fa640 1 -- 172.21.15.42:0/4093917873 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f8c007030 con 0x7f7f980a1330 2024-09-16T03:58:30.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.141+0000 7f7f957fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:58:30.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.141+0000 7f7f957fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:58:30.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.141+0000 7f7f957fa640 10 monclient: dump: 2024-09-16T03:58:30.161 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:58:30.161 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:58:30.161 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:30.161 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:30.161 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:58:30.161 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:30.161 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:58:30.162 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:58:30.162 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:58:30.162 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:30.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.141+0000 7f7f967fc640 4 set_mon_vals no callback set 2024-09-16T03:58:30.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.141+0000 7f7f9ee8c640 10 client.824994.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:58:30.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.141+0000 7f7f9ee8c640 10 monclient: _renew_subs 2024-09-16T03:58:30.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.141+0000 7f7f9ee8c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:58:30.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.141+0000 7f7f9ee8c640 1 -- 172.21.15.42:0/4093917873 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f9811eba0 con 0x7f7f980a1330 2024-09-16T03:58:30.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.141+0000 7f7f9ee8c640 1 librados: init done 2024-09-16T03:58:30.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.141+0000 7f7f9ee8c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:58:30.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.141+0000 7f7f9ee8c640 1 --2- 172.21.15.42:0/4093917873 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7f98071ba0 0x7f7f98133df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:30.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.141+0000 7f7f9ee8c640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:30.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.141+0000 7f7f9ee8c640 1 -- 172.21.15.42:0/4093917873 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f7f9811e7b0 con 0x7f7f98071ba0 2024-09-16T03:58:30.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.141+0000 7f7f97fff640 1 --2- 172.21.15.42:0/4093917873 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7f98071ba0 0x7f7f98133df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:30.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.141+0000 7f7f97fff640 10 monclient: get_auth_request con 0x7f7f98071ba0 auth_method 0 2024-09-16T03:58:30.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.141+0000 7f7f97fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:30.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.141+0000 7f7f97fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:30.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.141+0000 7f7f97fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:58:30.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.141+0000 7f7f97fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:58:30.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.141+0000 7f7f97fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:58:30.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.141+0000 7f7f97fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:58:30.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.141+0000 7f7f957fa640 1 -- 172.21.15.42:0/4093917873 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f7f8c0071f0 con 0x7f7f980a1330 2024-09-16T03:58:30.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.141+0000 7f7f97fff640 10 monclient(hunting): handle_auth_done global_id 824997 payload 824 2024-09-16T03:58:30.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.141+0000 7f7f97fff640 1 -- 172.21.15.42:0/4093917873 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7f98072810 con 0x7f7f98071ba0 2024-09-16T03:58:30.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.141+0000 7f7f957fa640 1 --2- 172.21.15.42:0/4093917873 >> v2:172.21.15.154:6812/254845342 conn(0x7f7f840377c0 0x7f7f84039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:30.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.141+0000 7f7f957fa640 1 -- 172.21.15.42:0/4093917873 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(486..486 src has 366..486) ==== 6573+0+0 (secure 0 0 0) 0x7f7f8c0486a0 con 0x7f7f980a1330 2024-09-16T03:58:30.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.141+0000 7f7f957fa640 10 client.824994.objecter ms_dispatch 0x7f7f98000ba0 osd_map(486..486 src has 366..486) 2024-09-16T03:58:30.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.141+0000 7f7f957fa640 3 client.824994.objecter handle_osd_map got epochs [486,486] > 0 2024-09-16T03:58:30.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.141+0000 7f7f957fa640 3 client.824994.objecter handle_osd_map decoding full epoch 486 2024-09-16T03:58:30.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.141+0000 7f7f97fff640 1 --2- 172.21.15.42:0/4093917873 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7f98071ba0 0x7f7f98133df0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f7f9806add0 tx=0x7f7f8801b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:58:30.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.141+0000 7f7f957fa640 20 client.824994.objecter dump_active .. 0 homeless 2024-09-16T03:58:30.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.141+0000 7f7f957fa640 10 client.824994.objecter ms_handle_connect 0x7f7f98071ba0 2024-09-16T03:58:30.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.141+0000 7f7f957fa640 10 client.824994.objecter resend_mon_ops 2024-09-16T03:58:30.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.141+0000 7f7f957fa640 1 -- 172.21.15.42:0/4093917873 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f880be070 con 0x7f7f98071ba0 2024-09-16T03:58:30.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.141+0000 7f7f957fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:58:30.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.141+0000 7f7f957fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:58:30.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.141+0000 7f7f957fa640 10 monclient: dump: 2024-09-16T03:58:30.164 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:58:30.165 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:58:30.165 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:30.165 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:30.165 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:58:30.165 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:30.165 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:58:30.165 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:58:30.165 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:58:30.165 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:30.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.141+0000 7f7f9d429640 1 --2- 172.21.15.42:0/4093917873 >> v2:172.21.15.154:6812/254845342 conn(0x7f7f840377c0 0x7f7f84039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:30.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.141+0000 7f7f9d429640 10 monclient: get_auth_request con 0x7f7f840377c0 auth_method 0 2024-09-16T03:58:30.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.141+0000 7f7f957fa640 1 -- 172.21.15.42:0/4093917873 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f7f880b9660 con 0x7f7f98071ba0 2024-09-16T03:58:30.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.141+0000 7f7f957fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:58:30.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.141+0000 7f7f957fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:58:30.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.141+0000 7f7f957fa640 1 -- 172.21.15.42:0/4093917873 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7f98071ba0 msgr2=0x7f7f98133df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:30.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.141+0000 7f7f957fa640 1 --2- 172.21.15.42:0/4093917873 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7f98071ba0 0x7f7f98133df0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f7f9806add0 tx=0x7f7f8801b8b0 comp rx=0 tx=0).stop 2024-09-16T03:58:30.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.141+0000 7f7f957fa640 1 -- 172.21.15.42:0/4093917873 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7f98071ba0 msgr2=0x7f7f98133df0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:58:30.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.141+0000 7f7f957fa640 1 --2- 172.21.15.42:0/4093917873 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7f98071ba0 0x7f7f98133df0 secure :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f7f9806add0 tx=0x7f7f8801b8b0 comp rx=0 tx=0).stop 2024-09-16T03:58:30.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.141+0000 7f7f957fa640 10 client.824994.objecter ms_handle_connect 0x7f7f840377c0 2024-09-16T03:58:30.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.141+0000 7f7f9d429640 1 --2- 172.21.15.42:0/4093917873 >> v2:172.21.15.154:6812/254845342 conn(0x7f7f840377c0 0x7f7f84039c80 secure :-1 s=READY pgs=3702 cs=0 l=1 rev1=1 crypto rx=0x7f7f9809df70 tx=0x7f7f900094b0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:58:30.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.149+0000 7f7f9ee8c640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:58:30.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.149+0000 7f7f9ee8c640 1 --2- 172.21.15.42:0/4093917873 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7f98134330 0x7f7f98134710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:30.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.149+0000 7f7f9ee8c640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:30.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.149+0000 7f7f9ee8c640 1 -- 172.21.15.42:0/4093917873 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f7f980a1b40 con 0x7f7f98134330 2024-09-16T03:58:30.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.149+0000 7f7f97fff640 1 --2- 172.21.15.42:0/4093917873 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7f98134330 0x7f7f98134710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:30.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.149+0000 7f7f97fff640 10 monclient: get_auth_request con 0x7f7f98134330 auth_method 0 2024-09-16T03:58:30.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.149+0000 7f7f97fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:30.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.149+0000 7f7f97fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:30.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.149+0000 7f7f97fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:58:30.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.149+0000 7f7f97fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:58:30.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.149+0000 7f7f97fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:58:30.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.149+0000 7f7f97fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:58:30.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.149+0000 7f7f97fff640 10 monclient(hunting): handle_auth_done global_id 825000 payload 824 2024-09-16T03:58:30.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.149+0000 7f7f97fff640 1 -- 172.21.15.42:0/4093917873 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f7f98134c50 con 0x7f7f98134330 2024-09-16T03:58:30.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.149+0000 7f7f957fa640 10 client.824994.objecter ms_handle_connect 0x7f7f98134330 2024-09-16T03:58:30.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.149+0000 7f7f97fff640 1 --2- 172.21.15.42:0/4093917873 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7f98134330 0x7f7f98134710 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f7f8801d1c0 tx=0x7f7f8800b450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:58:30.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.149+0000 7f7f957fa640 10 client.824994.objecter resend_mon_ops 2024-09-16T03:58:30.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.149+0000 7f7f957fa640 1 -- 172.21.15.42:0/4093917873 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f880b4040 con 0x7f7f98134330 2024-09-16T03:58:30.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.149+0000 7f7f957fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:58:30.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.149+0000 7f7f957fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:58:30.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.149+0000 7f7f957fa640 10 monclient: dump: 2024-09-16T03:58:30.169 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:58:30.169 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:58:30.169 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:30.169 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:30.169 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:58:30.170 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:30.170 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:58:30.170 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:58:30.170 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:58:30.170 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:30.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.149+0000 7f7f957fa640 1 -- 172.21.15.42:0/4093917873 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4871 (secure 0 0 0) 0x7f7f880b9e70 con 0x7f7f98134330 2024-09-16T03:58:30.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.149+0000 7f7f957fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:58:30.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.149+0000 7f7f957fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:58:30.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.149+0000 7f7f957fa640 1 -- 172.21.15.42:0/4093917873 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7f98134330 msgr2=0x7f7f98134710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:30.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.149+0000 7f7f957fa640 1 --2- 172.21.15.42:0/4093917873 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7f98134330 0x7f7f98134710 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f7f8801d1c0 tx=0x7f7f8800b450 comp rx=0 tx=0).stop 2024-09-16T03:58:30.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.149+0000 7f7f957fa640 1 -- 172.21.15.42:0/4093917873 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7f98134330 msgr2=0x7f7f98134710 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:58:30.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.149+0000 7f7f957fa640 1 --2- 172.21.15.42:0/4093917873 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7f98134330 0x7f7f98134710 secure :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f7f8801d1c0 tx=0x7f7f8800b450 comp rx=0 tx=0).stop 2024-09-16T03:58:30.171 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:58:30.171 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:58:30.171 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:58:30.171 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "leader", 2024-09-16T03:58:30.171 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 152, 2024-09-16T03:58:30.171 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 400004, 2024-09-16T03:58:30.171 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:58:30.171 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:58:30.171 INFO:teuthology.orchestra.run.smithi042.stdout: 1 2024-09-16T03:58:30.171 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:58:30.171 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 0, 2024-09-16T03:58:30.171 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:58:30.172 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:58:30.172 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:58:30.172 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:58:30.172 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:58:30.172 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:58:30.172 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:58:30.172 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:58:30.172 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:58:30.172 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:58:30.172 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:58:30.172 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:58:30.172 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:58:30.172 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:58:30.172 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:58:30.173 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:58:30.173 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:58:30.173 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:58:30.173 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:58:30.173 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:58:30.173 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:58:30.173 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:58:30.173 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:58:30.173 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:58:30.173 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:58:30.173 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:58:30.173 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:58:30.173 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:58:30.174 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:58:30.174 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:58:30.174 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:58:30.174 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:58:30.174 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:58:30.174 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:58:30.174 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:58:30.174 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:58:30.174 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:58:30.174 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:58:30.174 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:58:30.174 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:58:30.174 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:58:30.174 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:58:30.175 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:58:30.175 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:58:30.175 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:58:30.175 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:58:30.175 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:58:30.175 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:58:30.175 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:58:30.175 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:58:30.175 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:58:30.175 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:58:30.175 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:58:30.175 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:58:30.175 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:58:30.175 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:58:30.176 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:58:30.176 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:58:30.176 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:58:30.176 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:58:30.176 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:58:30.176 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:58:30.176 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:58:30.176 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:58:30.176 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:58:30.176 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:58:30.176 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:58:30.176 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:58:30.176 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:58:30.176 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:58:30.177 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:58:30.177 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:58:30.177 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:58:30.177 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:58:30.177 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:58:30.177 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:58:30.177 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:58:30.177 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:58:30.177 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:58:30.177 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:58:30.177 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:58:30.177 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:58:30.177 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:58:30.178 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:58:30.178 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:58:30.178 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:58:30.178 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:58:30.178 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:58:30.178 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:58:30.178 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:58:30.178 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:58:30.178 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:58:30.178 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:58:30.178 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:58:30.178 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:58:30.178 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:58:30.178 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:58:30.178 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:58:30.179 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:58:30.179 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:58:30.179 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:58:30.179 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:58:30.179 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:58:30.179 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:58:30.179 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:58:30.179 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:58:30.179 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:58:30.179 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:58:30.179 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:58:30.179 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:58:30.179 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:58:30.179 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:58:30.180 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:58:30.180 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:58:30.180 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:58:30.180 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:58:30.180 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:58:30.180 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:58:30.180 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:58:30.180 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:58:30.180 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:58:30.180 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:58:30.180 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:58:30.180 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:58:30.180 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:58:30.180 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:58:30.180 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:58:30.181 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:58:30.181 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:58:30.181 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:58:30.181 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:58:30.181 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:58:30.181 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:58:30.181 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:58:30.181 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:58:30.181 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:58:30.181 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:58:30.181 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:58:30.181 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:58:30.181 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:58:30.181 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:58:30.182 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T03:58:30.182 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:58:30.182 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:58:30.182 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:58:30.182 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 6 2024-09-16T03:58:30.182 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:58:30.182 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:58:30.182 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:58:30.182 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:58:30.182 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:58:30.182 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:58:30.182 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 8 2024-09-16T03:58:30.182 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:58:30.182 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:58:30.182 INFO:teuthology.orchestra.run.smithi042.stdout: "mgr": [ 2024-09-16T03:58:30.183 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:58:30.183 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:58:30.183 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:58:30.183 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:58:30.183 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:58:30.183 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:58:30.183 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:58:30.183 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:58:30.183 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:58:30.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.149+0000 7f7f9ee8c640 10 librados: watch_flush enter 2024-09-16T03:58:30.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.149+0000 7f7f9ee8c640 10 librados: watch_flush exit 2024-09-16T03:58:30.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.149+0000 7f7f9ee8c640 20 client.824994.objecter shutdown clearing up homeless session... 2024-09-16T03:58:30.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.149+0000 7f7f9ee8c640 10 client.824994.objecter successfully canceled tick 2024-09-16T03:58:30.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.149+0000 7f7f9ee8c640 1 -- 172.21.15.42:0/4093917873 >> v2:172.21.15.154:6812/254845342 conn(0x7f7f840377c0 msgr2=0x7f7f84039c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:30.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.149+0000 7f7f9ee8c640 1 --2- 172.21.15.42:0/4093917873 >> v2:172.21.15.154:6812/254845342 conn(0x7f7f840377c0 0x7f7f84039c80 secure :-1 s=READY pgs=3702 cs=0 l=1 rev1=1 crypto rx=0x7f7f9809df70 tx=0x7f7f900094b0 comp rx=0 tx=0).stop 2024-09-16T03:58:30.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.149+0000 7f7f9ee8c640 10 monclient: shutdown 2024-09-16T03:58:30.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.149+0000 7f7f9ee8c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:58:30.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.149+0000 7f7f9ee8c640 1 -- 172.21.15.42:0/4093917873 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7f980a1330 msgr2=0x7f7f9811bf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:30.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.149+0000 7f7f9ee8c640 1 --2- 172.21.15.42:0/4093917873 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7f980a1330 0x7f7f9811bf70 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f7f8c00c500 tx=0x7f7f8c00c9c0 comp rx=0 tx=0).stop 2024-09-16T03:58:30.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.149+0000 7f7f9d429640 1 -- 172.21.15.42:0/4093917873 reap_dead start 2024-09-16T03:58:30.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.149+0000 7f7f9ee8c640 1 -- 172.21.15.42:0/4093917873 shutdown_connections 2024-09-16T03:58:30.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.149+0000 7f7f9ee8c640 1 --2- 172.21.15.42:0/4093917873 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7f980a1330 0x7f7f9811bf70 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:30.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.149+0000 7f7f9ee8c640 1 -- 172.21.15.42:0/4093917873 >> 172.21.15.42:0/4093917873 conn(0x7f7f980a2420 msgr2=0x7f7f980a3720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:58:30.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.149+0000 7f7f9ee8c640 1 -- 172.21.15.42:0/4093917873 shutdown_connections 2024-09-16T03:58:30.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.149+0000 7f7f9ee8c640 1 -- 172.21.15.42:0/4093917873 wait complete. 2024-09-16T03:58:30.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.149+0000 7f7f9ee8c640 1 librados: shutdown 2024-09-16T03:58:30.186 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T03:58:30.187 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-16T03:58:30.187 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T03:58:30.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.301+0000 7ff8d79c5640 10 monclient: get_monmap_and_config 2024-09-16T03:58:30.306 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.301+0000 7ff8d79c5640 10 monclient: build_initial_monmap 2024-09-16T03:58:30.306 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.301+0000 7ff8d79c5640 10 monclient: monmap: 2024-09-16T03:58:30.306 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:58:30.306 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:58:30.306 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:58:30.304280+0000 2024-09-16T03:58:30.306 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:58:30.304280+0000 2024-09-16T03:58:30.306 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:58:30.306 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:30.306 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:58:30.306 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:58:30.306 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:58:30.306 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:58:30.306 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:58:30.307 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:30.308 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.301+0000 7ff8d79c5640 1 Processor -- start 2024-09-16T03:58:30.308 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.301+0000 7ff8d79c5640 1 -- start start 2024-09-16T03:58:30.308 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.301+0000 7ff8d79c5640 10 monclient: init 2024-09-16T03:58:30.309 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.301+0000 7ff8d79c5640 10 monclient: _reopen_session rank -1 2024-09-16T03:58:30.309 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.301+0000 7ff8d79c5640 10 monclient: _add_conns ranks=[0,3,1,2,4] 2024-09-16T03:58:30.309 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.301+0000 7ff8d79c5640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff8d00a39b0 addr v1:172.21.15.42:6789/0 2024-09-16T03:58:30.309 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.301+0000 7ff8d79c5640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff8d00a3d90 0x7ff8d013c380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:30.309 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.301+0000 7ff8d79c5640 10 monclient(hunting): picked mon.noname-a con 0x7ff8d00a3d90 addr v2:172.21.15.42:3300/0 2024-09-16T03:58:30.309 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.301+0000 7ff8d79c5640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff8d013c8c0 addr v1:172.21.15.154:6789/0 2024-09-16T03:58:30.309 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.301+0000 7ff8d79c5640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff8d0134290 con 0x7ff8d00a39b0 2024-09-16T03:58:30.309 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.301+0000 7ff8d79c5640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff8d009c140 con 0x7ff8d013c8c0 2024-09-16T03:58:30.309 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.301+0000 7ff8d79c5640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:30.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.301+0000 7ff8d79c5640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7ff8d009c420 con 0x7ff8d00a3d90 2024-09-16T03:58:30.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.301+0000 7ff8d79c5640 10 monclient(hunting): _renew_subs 2024-09-16T03:58:30.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.301+0000 7ff8d79c5640 10 monclient(hunting): authenticate will time out at 1957.273805s 2024-09-16T03:58:30.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.301+0000 7ff8d5761640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7ff8d00a39b0 0x7ff8d0133b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:50102/0 (socket says 172.21.15.42:50102) 2024-09-16T03:58:30.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.301+0000 7ff8d5761640 1 -- 172.21.15.42:0/1448898991 learned_addr learned my addr 172.21.15.42:0/1448898991 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:58:30.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.301+0000 7ff8d4f60640 1 --2- 172.21.15.42:0/1448898991 >> v2:172.21.15.42:3300/0 conn(0x7ff8d00a3d90 0x7ff8d013c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:30.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.301+0000 7ff8d4f60640 10 monclient(hunting): get_auth_request con 0x7ff8d00a3d90 auth_method 0 2024-09-16T03:58:30.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.301+0000 7ff8d4f60640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:30.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.301+0000 7ff8d4f60640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:30.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.301+0000 7ff8d4f60640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:58:30.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.301+0000 7ff8d4f60640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:58:30.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.301+0000 7ff8d4f60640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:58:30.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.301+0000 7ff8d4f60640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:58:30.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.301+0000 7ff8cffff640 1 -- 172.21.15.42:0/1448898991 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1849819986 0 0) 0x7ff8d0134290 con 0x7ff8d00a39b0 2024-09-16T03:58:30.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.301+0000 7ff8cffff640 10 monclient(hunting): _negotiate 2024-09-16T03:58:30.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.301+0000 7ff8cffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:30.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.301+0000 7ff8cffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:58:30.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.301+0000 7ff8cffff640 10 monclient(hunting): my global_id is 825003 2024-09-16T03:58:30.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.301+0000 7ff8cffff640 1 -- 172.21.15.42:0/1448898991 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff8b8003530 con 0x7ff8d00a39b0 2024-09-16T03:58:30.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.301+0000 7ff8d4f60640 10 monclient(hunting): handle_auth_done global_id 825006 payload 274 2024-09-16T03:58:30.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.301+0000 7ff8d4f60640 1 -- 172.21.15.42:0/1448898991 >> v1:172.21.15.154:6789/0 conn(0x7ff8d013c8c0 legacy=0x7ff8d013ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:30.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.301+0000 7ff8d4f60640 1 -- 172.21.15.42:0/1448898991 >> v1:172.21.15.42:6789/0 conn(0x7ff8d00a39b0 legacy=0x7ff8d0133b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:30.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.301+0000 7ff8cffff640 1 -- 172.21.15.42:0/1448898991 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2506243419 0 0) 0x7ff8b8003530 con 0x7ff8d00a39b0 2024-09-16T03:58:30.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.301+0000 7ff8d4f60640 10 monclient: _finish_hunting 0 2024-09-16T03:58:30.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.301+0000 7ff8d4f60640 1 monclient: found mon.noname-a 2024-09-16T03:58:30.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.301+0000 7ff8d4f60640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:58:30.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.301+0000 7ff8d4f60640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:58:30.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.301+0000 7ff8d4f60640 1 -- 172.21.15.42:0/1448898991 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8d006dd60 con 0x7ff8d00a3d90 2024-09-16T03:58:30.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.301+0000 7ff8cffff640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T03:58:30.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.305+0000 7ff8d4f60640 1 --2- 172.21.15.42:0/1448898991 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff8d00a3d90 0x7ff8d013c380 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7ff8c40097c0 tx=0x7ff8c401c280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80837fa270551a31 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:58:30.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.305+0000 7ff8cffff640 1 -- 172.21.15.42:0/1448898991 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8c40b0070 con 0x7ff8d00a3d90 2024-09-16T03:58:30.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.305+0000 7ff8cffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:58:30.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.305+0000 7ff8cffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:58:30.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.305+0000 7ff8cffff640 10 monclient: dump: 2024-09-16T03:58:30.319 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:58:30.319 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:58:30.320 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:30.320 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:30.320 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:58:30.320 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:30.320 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:58:30.320 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:58:30.320 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:58:30.320 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:30.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.305+0000 7ff8cffff640 10 monclient: _finish_auth 0 2024-09-16T03:58:30.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.305+0000 7ff8cffff640 10 monclient: _check_auth_tickets 2024-09-16T03:58:30.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.305+0000 7ff8cffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:58:30.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.305+0000 7ff8cffff640 1 -- 172.21.15.42:0/1448898991 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff8c40ab400 con 0x7ff8d00a3d90 2024-09-16T03:58:30.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.305+0000 7ff8cffff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:58:30.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.305+0000 7ff8cffff640 1 -- 172.21.15.42:0/1448898991 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8c8001a00 con 0x7ff8d00a3d90 2024-09-16T03:58:30.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.305+0000 7ff8d79c5640 5 monclient: authenticate success, global_id 825006 2024-09-16T03:58:30.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.305+0000 7ff8d79c5640 10 monclient: get_monmap_and_config success 2024-09-16T03:58:30.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.305+0000 7ff8d79c5640 4 set_mon_vals no callback set 2024-09-16T03:58:30.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.305+0000 7ff8cffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:58:30.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.305+0000 7ff8cffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:58:30.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.305+0000 7ff8cffff640 10 monclient: dump: 2024-09-16T03:58:30.321 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:58:30.321 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:58:30.321 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:30.321 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:30.321 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:58:30.321 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:30.321 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:58:30.322 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:58:30.322 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:58:30.322 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:30.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.305+0000 7ff8d79c5640 10 monclient: shutdown 2024-09-16T03:58:30.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.305+0000 7ff8d79c5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:58:30.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.305+0000 7ff8d79c5640 1 -- 172.21.15.42:0/1448898991 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff8d00a3d90 msgr2=0x7ff8d013c380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:30.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.305+0000 7ff8d79c5640 1 --2- 172.21.15.42:0/1448898991 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff8d00a3d90 0x7ff8d013c380 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7ff8c40097c0 tx=0x7ff8c401c280 comp rx=0 tx=0).stop 2024-09-16T03:58:30.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.305+0000 7ff8d79c5640 1 -- 172.21.15.42:0/1448898991 shutdown_connections 2024-09-16T03:58:30.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.305+0000 7ff8d79c5640 1 --2- 172.21.15.42:0/1448898991 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff8d00a3d90 0x7ff8d013c380 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:30.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.305+0000 7ff8d79c5640 1 -- 172.21.15.42:0/1448898991 >> 172.21.15.42:0/1448898991 conn(0x7ff8d00a9fb0 msgr2=0x7ff8d00aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:58:30.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.305+0000 7ff8d79c5640 1 -- 172.21.15.42:0/1448898991 shutdown_connections 2024-09-16T03:58:30.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.305+0000 7ff8d79c5640 1 -- 172.21.15.42:0/1448898991 wait complete. 2024-09-16T03:58:30.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.305+0000 7ff8d79c5640 10 monclient: build_initial_monmap 2024-09-16T03:58:30.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.305+0000 7ff8d79c5640 10 monclient: monmap: 2024-09-16T03:58:30.323 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:58:30.323 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:58:30.323 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:58:30.323 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:58:30.323 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:58:30.323 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:30.323 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:58:30.323 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:58:30.323 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:58:30.323 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:30.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.305+0000 7ff8d79c5640 1 librados: starting msgr at 2024-09-16T03:58:30.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.305+0000 7ff8d79c5640 1 librados: starting objecter 2024-09-16T03:58:30.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.305+0000 7ff8d79c5640 1 Processor -- start 2024-09-16T03:58:30.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.305+0000 7ff8d79c5640 1 -- start start 2024-09-16T03:58:30.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.305+0000 7ff8d79c5640 1 librados: setting wanted keys 2024-09-16T03:58:30.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.305+0000 7ff8d79c5640 1 librados: calling monclient init 2024-09-16T03:58:30.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.305+0000 7ff8d79c5640 10 monclient: init 2024-09-16T03:58:30.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.305+0000 7ff8d79c5640 10 monclient: _reopen_session rank -1 2024-09-16T03:58:30.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.305+0000 7ff8d79c5640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:58:30.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.305+0000 7ff8d79c5640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff8d00a39b0 0x7ff8d012c520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:30.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.305+0000 7ff8d79c5640 10 monclient(hunting): picked mon.noname-c con 0x7ff8d00a39b0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:58:30.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.305+0000 7ff8d79c5640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff8d00a3d90 0x7ff8d012ca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:30.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.305+0000 7ff8d79c5640 10 monclient(hunting): picked mon.noname-b con 0x7ff8d00a3d90 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:58:30.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.305+0000 7ff8d79c5640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff8d013c8c0 0x7ff8d0139580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:30.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.305+0000 7ff8d79c5640 10 monclient(hunting): picked mon.noname-a con 0x7ff8d013c8c0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:58:30.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.305+0000 7ff8d79c5640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:30.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.305+0000 7ff8d79c5640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7ff8d0142f80 con 0x7ff8d013c8c0 2024-09-16T03:58:30.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.305+0000 7ff8d79c5640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:30.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.305+0000 7ff8d79c5640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff8d0142e00 con 0x7ff8d00a3d90 2024-09-16T03:58:30.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.305+0000 7ff8d79c5640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:30.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.305+0000 7ff8d79c5640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7ff8d0143100 con 0x7ff8d00a39b0 2024-09-16T03:58:30.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.305+0000 7ff8d79c5640 10 monclient(hunting): _renew_subs 2024-09-16T03:58:30.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.305+0000 7ff8d79c5640 10 monclient(hunting): authenticate will time out at 1957.279155s 2024-09-16T03:58:30.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.305+0000 7ff8d5761640 1 -- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff8d00a39b0 msgr2=0x7ff8d012c520 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T03:58:30.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.305+0000 7ff8d5761640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff8d00a39b0 0x7ff8d012c520 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:58:30.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.305+0000 7ff8d5f62640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff8d013c8c0 0x7ff8d0139580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:30.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.305+0000 7ff8d5f62640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff8d013c8c0 0x7ff8d0139580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:40188/0 (socket says 172.21.15.42:40188) 2024-09-16T03:58:30.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.305+0000 7ff8d5f62640 1 -- 172.21.15.42:0/2999341162 learned_addr learned my addr 172.21.15.42:0/2999341162 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:58:30.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.305+0000 7ff8d5f62640 10 monclient(hunting): get_auth_request con 0x7ff8d013c8c0 auth_method 0 2024-09-16T03:58:30.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.305+0000 7ff8d5f62640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:30.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.305+0000 7ff8d5f62640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:30.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.305+0000 7ff8d5f62640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:58:30.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.305+0000 7ff8d5f62640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:58:30.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.305+0000 7ff8d5f62640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:58:30.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.305+0000 7ff8d5f62640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:58:30.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.305+0000 7ff8d4f60640 1 --2- 172.21.15.42:0/2999341162 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff8d00a3d90 0x7ff8d012ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:30.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.305+0000 7ff8d4f60640 10 monclient(hunting): get_auth_request con 0x7ff8d00a3d90 auth_method 0 2024-09-16T03:58:30.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.305+0000 7ff8d4f60640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:30.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.305+0000 7ff8d4f60640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:30.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.305+0000 7ff8d4f60640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:58:30.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.305+0000 7ff8d5f62640 10 monclient(hunting): handle_auth_done global_id 825009 payload 824 2024-09-16T03:58:30.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.305+0000 7ff8d5f62640 1 -- 172.21.15.42:0/2999341162 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff8d00a39b0 msgr2=0x7ff8d012c520 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:58:30.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.305+0000 7ff8d5f62640 1 --2- 172.21.15.42:0/2999341162 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff8d00a39b0 0x7ff8d012c520 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:30.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.305+0000 7ff8d5f62640 1 -- 172.21.15.42:0/2999341162 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff8d00a3d90 msgr2=0x7ff8d012ca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:30.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.305+0000 7ff8d5f62640 1 --2- 172.21.15.42:0/2999341162 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff8d00a3d90 0x7ff8d012ca60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:30.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.305+0000 7ff8d5f62640 10 monclient: _finish_hunting 0 2024-09-16T03:58:30.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.305+0000 7ff8d5f62640 1 monclient: found mon.noname-a 2024-09-16T03:58:30.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.305+0000 7ff8d5f62640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:58:30.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.305+0000 7ff8d5f62640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:58:30.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.305+0000 7ff8d5f62640 1 -- 172.21.15.42:0/2999341162 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8d0134bd0 con 0x7ff8d013c8c0 2024-09-16T03:58:30.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.305+0000 7ff8d4f60640 1 --2- 172.21.15.42:0/2999341162 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff8d00a3d90 0x7ff8d012ca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T03:58:30.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.305+0000 7ff8d5f62640 1 --2- 172.21.15.42:0/2999341162 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff8d013c8c0 0x7ff8d0139580 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7ff8c000c430 tx=0x7ff8c000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:58:30.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.305+0000 7ff8cdffb640 10 client.?.objecter ms_handle_connect 0x7ff8d013c8c0 2024-09-16T03:58:30.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.305+0000 7ff8cdffb640 10 client.?.objecter resend_mon_ops 2024-09-16T03:58:30.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.309+0000 7ff8cdffb640 1 -- 172.21.15.42:0/2999341162 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8c0005220 con 0x7ff8d013c8c0 2024-09-16T03:58:30.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.309+0000 7ff8cdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:58:30.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.309+0000 7ff8cdffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:58:30.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.309+0000 7ff8cdffb640 10 monclient: dump: 2024-09-16T03:58:30.328 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:58:30.328 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:58:30.328 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:30.328 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:30.328 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:58:30.328 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:30.328 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:58:30.328 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:58:30.328 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:58:30.328 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:30.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.309+0000 7ff8cdffb640 10 monclient: _finish_auth 0 2024-09-16T03:58:30.329 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.309+0000 7ff8cdffb640 10 monclient: _check_auth_tickets 2024-09-16T03:58:30.329 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.309+0000 7ff8cdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:58:30.329 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.309+0000 7ff8d79c5640 5 monclient: authenticate success, global_id 825009 2024-09-16T03:58:30.329 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.309+0000 7ff8cdffb640 1 -- 172.21.15.42:0/2999341162 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff8c00053c0 con 0x7ff8d013c8c0 2024-09-16T03:58:30.329 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.309+0000 7ff8cdffb640 10 monclient: handle_config config(0 keys) 2024-09-16T03:58:30.329 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.309+0000 7ff8d79c5640 10 monclient: _renew_subs 2024-09-16T03:58:30.329 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.309+0000 7ff8cdffb640 1 -- 172.21.15.42:0/2999341162 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8c0007270 con 0x7ff8d013c8c0 2024-09-16T03:58:30.329 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.309+0000 7ff8d79c5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:58:30.329 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.309+0000 7ff8d79c5640 1 -- 172.21.15.42:0/2999341162 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8d012db20 con 0x7ff8d013c8c0 2024-09-16T03:58:30.329 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.309+0000 7ff8cdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:58:30.329 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.309+0000 7ff8cdffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:58:30.329 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.309+0000 7ff8cf7fe640 4 set_mon_vals no callback set 2024-09-16T03:58:30.329 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.309+0000 7ff8cdffb640 10 monclient: dump: 2024-09-16T03:58:30.329 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:58:30.330 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:58:30.330 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:30.330 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:30.330 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:58:30.330 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:30.330 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:58:30.330 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:58:30.330 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:58:30.330 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:30.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.309+0000 7ff8d79c5640 10 client.825009.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:58:30.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.309+0000 7ff8d79c5640 10 monclient: _renew_subs 2024-09-16T03:58:30.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.309+0000 7ff8d79c5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:58:30.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.309+0000 7ff8d79c5640 1 -- 172.21.15.42:0/2999341162 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8d013afe0 con 0x7ff8d013c8c0 2024-09-16T03:58:30.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.309+0000 7ff8d79c5640 1 librados: init done 2024-09-16T03:58:30.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.309+0000 7ff8d79c5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:58:30.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.309+0000 7ff8d79c5640 1 --2- 172.21.15.42:0/2999341162 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff8d0139ac0 0x7ff8d0139ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:30.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.309+0000 7ff8d79c5640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:30.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.309+0000 7ff8d79c5640 1 -- 172.21.15.42:0/2999341162 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff8d006f680 con 0x7ff8d0139ac0 2024-09-16T03:58:30.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.309+0000 7ff8cdffb640 1 -- 172.21.15.42:0/2999341162 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7ff8c0019020 con 0x7ff8d013c8c0 2024-09-16T03:58:30.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.309+0000 7ff8cdffb640 1 --2- 172.21.15.42:0/2999341162 >> v2:172.21.15.154:6812/254845342 conn(0x7ff8bc0377c0 0x7ff8bc039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:30.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.309+0000 7ff8d5761640 1 --2- 172.21.15.42:0/2999341162 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff8d0139ac0 0x7ff8d0139ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:30.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.309+0000 7ff8cdffb640 1 -- 172.21.15.42:0/2999341162 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(486..486 src has 366..486) ==== 6573+0+0 (secure 0 0 0) 0x7ff8c00486b0 con 0x7ff8d013c8c0 2024-09-16T03:58:30.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.309+0000 7ff8cdffb640 10 client.825009.objecter ms_dispatch 0x7ff8d0000ba0 osd_map(486..486 src has 366..486) 2024-09-16T03:58:30.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.309+0000 7ff8cdffb640 3 client.825009.objecter handle_osd_map got epochs [486,486] > 0 2024-09-16T03:58:30.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.309+0000 7ff8cdffb640 3 client.825009.objecter handle_osd_map decoding full epoch 486 2024-09-16T03:58:30.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.309+0000 7ff8d5761640 10 monclient: get_auth_request con 0x7ff8d0139ac0 auth_method 0 2024-09-16T03:58:30.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.309+0000 7ff8d5761640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:30.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.309+0000 7ff8d5761640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:30.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.309+0000 7ff8d5761640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:58:30.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.309+0000 7ff8cdffb640 20 client.825009.objecter dump_active .. 0 homeless 2024-09-16T03:58:30.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.309+0000 7ff8d5761640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:58:30.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.309+0000 7ff8d5761640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:58:30.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.309+0000 7ff8d5761640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:58:30.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.309+0000 7ff8d4f60640 1 --2- 172.21.15.42:0/2999341162 >> v2:172.21.15.154:6812/254845342 conn(0x7ff8bc0377c0 0x7ff8bc039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:30.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.309+0000 7ff8d4f60640 10 monclient: get_auth_request con 0x7ff8bc0377c0 auth_method 0 2024-09-16T03:58:30.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.309+0000 7ff8d5761640 10 monclient(hunting): handle_auth_done global_id 904186 payload 824 2024-09-16T03:58:30.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.309+0000 7ff8d5761640 1 -- 172.21.15.42:0/2999341162 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff8d0133b80 con 0x7ff8d0139ac0 2024-09-16T03:58:30.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.309+0000 7ff8cdffb640 10 client.825009.objecter ms_handle_connect 0x7ff8d0139ac0 2024-09-16T03:58:30.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.309+0000 7ff8cdffb640 10 client.825009.objecter resend_mon_ops 2024-09-16T03:58:30.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.309+0000 7ff8d5761640 1 --2- 172.21.15.42:0/2999341162 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff8d0139ac0 0x7ff8d0139ea0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ff8c8004c00 tx=0x7ff8c800a430 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:58:30.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.309+0000 7ff8cdffb640 10 client.825009.objecter ms_handle_connect 0x7ff8bc0377c0 2024-09-16T03:58:30.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.309+0000 7ff8d4f60640 1 --2- 172.21.15.42:0/2999341162 >> v2:172.21.15.154:6812/254845342 conn(0x7ff8bc0377c0 0x7ff8bc039c80 secure :-1 s=READY pgs=3703 cs=0 l=1 rev1=1 crypto rx=0x7ff8c40ad000 tx=0x7ff8c40a6000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:58:30.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.309+0000 7ff8cdffb640 1 -- 172.21.15.42:0/2999341162 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8c8012400 con 0x7ff8d0139ac0 2024-09-16T03:58:30.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.309+0000 7ff8cdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:58:30.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.309+0000 7ff8cdffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:58:30.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.309+0000 7ff8cdffb640 10 monclient: dump: 2024-09-16T03:58:30.333 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:58:30.333 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:58:30.333 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:30.333 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:30.333 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:58:30.333 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:30.333 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:58:30.333 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:58:30.334 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:58:30.334 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:30.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.309+0000 7ff8cdffb640 1 -- 172.21.15.42:0/2999341162 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7ff8c8012400 con 0x7ff8d0139ac0 2024-09-16T03:58:30.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.309+0000 7ff8cdffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:58:30.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.309+0000 7ff8cdffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:58:30.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.309+0000 7ff8cdffb640 1 -- 172.21.15.42:0/2999341162 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff8d0139ac0 msgr2=0x7ff8d0139ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:30.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.309+0000 7ff8cdffb640 1 --2- 172.21.15.42:0/2999341162 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff8d0139ac0 0x7ff8d0139ea0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ff8c8004c00 tx=0x7ff8c800a430 comp rx=0 tx=0).stop 2024-09-16T03:58:30.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.309+0000 7ff8cdffb640 1 -- 172.21.15.42:0/2999341162 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff8d0139ac0 msgr2=0x7ff8d0139ea0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:58:30.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.309+0000 7ff8cdffb640 1 --2- 172.21.15.42:0/2999341162 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff8d0139ac0 0x7ff8d0139ea0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:30.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.317+0000 7ff8d79c5640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:58:30.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.317+0000 7ff8d79c5640 1 --2- 172.21.15.42:0/2999341162 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff8d013a3e0 0x7ff8d013cca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:30.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.317+0000 7ff8d79c5640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:30.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.317+0000 7ff8d79c5640 1 -- 172.21.15.42:0/2999341162 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff8d013c6a0 con 0x7ff8d013a3e0 2024-09-16T03:58:30.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.317+0000 7ff8d5761640 1 --2- 172.21.15.42:0/2999341162 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff8d013a3e0 0x7ff8d013cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:30.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.317+0000 7ff8d5761640 10 monclient: get_auth_request con 0x7ff8d013a3e0 auth_method 0 2024-09-16T03:58:30.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.317+0000 7ff8d5761640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:30.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.317+0000 7ff8d5761640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:30.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.317+0000 7ff8d5761640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:58:30.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.317+0000 7ff8d5761640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:58:30.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.317+0000 7ff8d5761640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:58:30.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.317+0000 7ff8d5761640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:58:30.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.317+0000 7ff8d5761640 10 monclient(hunting): handle_auth_done global_id 904189 payload 824 2024-09-16T03:58:30.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.317+0000 7ff8d5761640 1 -- 172.21.15.42:0/2999341162 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ff8d00a5750 con 0x7ff8d013a3e0 2024-09-16T03:58:30.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.317+0000 7ff8d5761640 1 --2- 172.21.15.42:0/2999341162 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff8d013a3e0 0x7ff8d013cca0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7ff8c8002240 tx=0x7ff8c800f000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:58:30.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.317+0000 7ff8cdffb640 10 client.825009.objecter ms_handle_connect 0x7ff8d013a3e0 2024-09-16T03:58:30.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.317+0000 7ff8cdffb640 10 client.825009.objecter resend_mon_ops 2024-09-16T03:58:30.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.317+0000 7ff8cdffb640 1 -- 172.21.15.42:0/2999341162 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8c8012d90 con 0x7ff8d013a3e0 2024-09-16T03:58:30.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.317+0000 7ff8cdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:58:30.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.317+0000 7ff8cdffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:58:30.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.317+0000 7ff8cdffb640 10 monclient: dump: 2024-09-16T03:58:30.337 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:58:30.337 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:58:30.337 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:30.337 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:30.337 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:58:30.337 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:30.337 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:58:30.337 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:58:30.337 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:58:30.338 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:30.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.317+0000 7ff8cdffb640 1 -- 172.21.15.42:0/2999341162 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4534 (secure 0 0 0) 0x7ff8c8012d90 con 0x7ff8d013a3e0 2024-09-16T03:58:30.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.317+0000 7ff8cdffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:58:30.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.317+0000 7ff8cdffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:58:30.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.317+0000 7ff8cdffb640 1 -- 172.21.15.42:0/2999341162 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff8d013a3e0 msgr2=0x7ff8d013cca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:30.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.317+0000 7ff8cdffb640 1 --2- 172.21.15.42:0/2999341162 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff8d013a3e0 0x7ff8d013cca0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7ff8c8002240 tx=0x7ff8c800f000 comp rx=0 tx=0).stop 2024-09-16T03:58:30.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.317+0000 7ff8cdffb640 1 -- 172.21.15.42:0/2999341162 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff8d013a3e0 msgr2=0x7ff8d013cca0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:58:30.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.317+0000 7ff8cdffb640 1 --2- 172.21.15.42:0/2999341162 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff8d013a3e0 0x7ff8d013cca0 secure :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7ff8c8002240 tx=0x7ff8c800f000 comp rx=0 tx=0).stop 2024-09-16T03:58:30.338 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:58:30.338 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:58:30.338 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:58:30.339 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T03:58:30.339 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 152, 2024-09-16T03:58:30.339 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 38503, 2024-09-16T03:58:30.339 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:58:30.339 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:58:30.339 INFO:teuthology.orchestra.run.smithi042.stdout: 1 2024-09-16T03:58:30.339 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:58:30.339 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 0, 2024-09-16T03:58:30.339 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:58:30.339 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:58:30.339 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:58:30.339 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:58:30.339 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:58:30.339 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:58:30.340 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:58:30.340 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:58:30.340 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:58:30.340 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:58:30.340 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:58:30.340 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:58:30.340 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:58:30.340 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:58:30.340 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:58:30.340 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:58:30.340 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:58:30.340 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:58:30.340 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:58:30.340 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:58:30.341 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:58:30.341 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:58:30.341 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:58:30.341 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:58:30.341 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:58:30.341 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:58:30.341 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:58:30.341 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:58:30.341 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:58:30.341 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:58:30.341 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:58:30.341 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:58:30.341 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:58:30.341 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:58:30.342 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:58:30.342 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:58:30.342 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:58:30.342 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:58:30.342 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:58:30.342 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:58:30.342 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:58:30.342 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:58:30.342 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:58:30.342 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:58:30.342 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:58:30.342 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:58:30.342 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:58:30.342 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:58:30.343 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:58:30.343 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:58:30.343 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:58:30.343 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:58:30.343 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:58:30.343 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:58:30.343 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:58:30.343 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:58:30.343 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:58:30.343 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:58:30.343 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:58:30.343 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:58:30.343 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:58:30.343 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:58:30.344 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:58:30.344 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:58:30.344 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:58:30.344 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:58:30.344 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:58:30.344 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:58:30.344 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:58:30.344 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:58:30.344 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:58:30.344 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:58:30.344 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:58:30.344 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:58:30.344 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:58:30.345 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:58:30.345 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:58:30.345 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:58:30.345 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:58:30.345 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:58:30.345 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:58:30.345 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:58:30.345 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:58:30.345 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:58:30.345 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:58:30.345 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:58:30.345 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:58:30.345 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:58:30.346 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:58:30.346 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:58:30.346 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:58:30.346 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:58:30.346 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:58:30.346 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:58:30.346 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:58:30.346 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:58:30.346 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:58:30.346 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:58:30.346 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:58:30.346 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:58:30.346 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:58:30.346 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:58:30.346 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:58:30.347 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:58:30.347 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:58:30.347 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:58:30.347 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:58:30.347 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:58:30.347 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:58:30.347 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:58:30.347 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:58:30.347 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:58:30.347 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:58:30.347 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:58:30.347 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:58:30.347 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:58:30.347 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:58:30.347 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:58:30.348 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:58:30.348 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:58:30.348 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:58:30.348 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:58:30.348 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:58:30.348 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:58:30.348 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:58:30.348 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:58:30.348 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:58:30.348 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:58:30.348 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:58:30.348 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:58:30.348 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:58:30.348 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:58:30.349 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:58:30.349 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:58:30.349 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:58:30.349 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:58:30.349 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:58:30.349 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:58:30.349 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:58:30.349 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:58:30.349 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:58:30.349 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:58:30.349 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:58:30.349 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:58:30.349 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:58:30.349 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 2 2024-09-16T03:58:30.350 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:58:30.350 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:58:30.350 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:58:30.350 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:58:30.350 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:58:30.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.317+0000 7ff8d79c5640 10 librados: watch_flush enter 2024-09-16T03:58:30.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.317+0000 7ff8d79c5640 10 librados: watch_flush exit 2024-09-16T03:58:30.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.317+0000 7ff8d79c5640 20 client.825009.objecter shutdown clearing up homeless session... 2024-09-16T03:58:30.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.317+0000 7ff8d79c5640 10 client.825009.objecter successfully canceled tick 2024-09-16T03:58:30.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.321+0000 7ff8d79c5640 1 -- 172.21.15.42:0/2999341162 >> v2:172.21.15.154:6812/254845342 conn(0x7ff8bc0377c0 msgr2=0x7ff8bc039c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:30.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.321+0000 7ff8d79c5640 1 --2- 172.21.15.42:0/2999341162 >> v2:172.21.15.154:6812/254845342 conn(0x7ff8bc0377c0 0x7ff8bc039c80 secure :-1 s=READY pgs=3703 cs=0 l=1 rev1=1 crypto rx=0x7ff8c40ad000 tx=0x7ff8c40a6000 comp rx=0 tx=0).stop 2024-09-16T03:58:30.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.321+0000 7ff8d79c5640 10 monclient: shutdown 2024-09-16T03:58:30.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.321+0000 7ff8d79c5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:58:30.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.321+0000 7ff8d79c5640 1 -- 172.21.15.42:0/2999341162 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff8d013c8c0 msgr2=0x7ff8d0139580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:30.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.321+0000 7ff8d79c5640 1 --2- 172.21.15.42:0/2999341162 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff8d013c8c0 0x7ff8d0139580 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7ff8c000c430 tx=0x7ff8c000c8f0 comp rx=0 tx=0).stop 2024-09-16T03:58:30.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.321+0000 7ff8d5f62640 1 -- 172.21.15.42:0/2999341162 reap_dead start 2024-09-16T03:58:30.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.321+0000 7ff8d79c5640 1 -- 172.21.15.42:0/2999341162 shutdown_connections 2024-09-16T03:58:30.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.321+0000 7ff8d79c5640 1 --2- 172.21.15.42:0/2999341162 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff8d013c8c0 0x7ff8d0139580 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:30.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.321+0000 7ff8d79c5640 1 -- 172.21.15.42:0/2999341162 >> 172.21.15.42:0/2999341162 conn(0x7ff8d00a9fb0 msgr2=0x7ff8d00a8870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:58:30.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.321+0000 7ff8d79c5640 1 -- 172.21.15.42:0/2999341162 shutdown_connections 2024-09-16T03:58:30.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.321+0000 7ff8d79c5640 1 -- 172.21.15.42:0/2999341162 wait complete. 2024-09-16T03:58:30.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:30.321+0000 7ff8d79c5640 1 librados: shutdown 2024-09-16T03:58:30.353 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T03:58:30.353 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-16T03:58:50.354 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-16T03:58:50.354 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-09-16T03:58:50.354 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-16T03:58:50.355 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-16T03:58:50.361 INFO:tasks.ceph.mon.c:Started 2024-09-16T03:58:50.361 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T03:58:50.361 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:58:50.485 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.477+0000 7f97b9f1d640 10 monclient: get_monmap_and_config 2024-09-16T03:58:50.486 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.477+0000 7f97b9f1d640 10 monclient: build_initial_monmap 2024-09-16T03:58:50.486 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.477+0000 7f97b9f1d640 10 monclient: monmap: 2024-09-16T03:58:50.486 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:58:50.486 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:58:50.486 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:58:50.482275+0000 2024-09-16T03:58:50.486 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:58:50.482275+0000 2024-09-16T03:58:50.486 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:58:50.487 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:50.487 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:58:50.487 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:58:50.487 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:58:50.487 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:58:50.487 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:58:50.487 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:50.488 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.477+0000 7f97b9f1d640 1 Processor -- start 2024-09-16T03:58:50.489 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.477+0000 7f97b9f1d640 1 -- start start 2024-09-16T03:58:50.489 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.477+0000 7f97b9f1d640 10 monclient: init 2024-09-16T03:58:50.489 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.477+0000 7f97b9f1d640 10 monclient: _reopen_session rank -1 2024-09-16T03:58:50.489 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.477+0000 7f97b9f1d640 10 monclient: _add_conns ranks=[0,3,2,1,4] 2024-09-16T03:58:50.489 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.477+0000 7f97b9f1d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f97b40a39b0 addr v1:172.21.15.42:6789/0 2024-09-16T03:58:50.489 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.477+0000 7f97b9f1d640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f97b40a3d90 0x7f97b413c380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:50.489 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.477+0000 7f97b9f1d640 10 monclient(hunting): picked mon.noname-a con 0x7f97b40a3d90 addr v2:172.21.15.42:3300/0 2024-09-16T03:58:50.490 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.477+0000 7f97b9f1d640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f97b413c8c0 0x7f97b413ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:50.490 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.477+0000 7f97b9f1d640 10 monclient(hunting): picked mon.noname-b con 0x7f97b413c8c0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:58:50.490 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.477+0000 7f97b9f1d640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f97b4134290 con 0x7f97b40a39b0 2024-09-16T03:58:50.490 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.477+0000 7f97b9f1d640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:50.490 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.477+0000 7f97b9f1d640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f97b409c140 con 0x7f97b40a3d90 2024-09-16T03:58:50.491 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.477+0000 7f97b9f1d640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:50.491 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.477+0000 7f97b9f1d640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f97b409c420 con 0x7f97b413c8c0 2024-09-16T03:58:50.491 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.477+0000 7f97b9f1d640 10 monclient(hunting): _renew_subs 2024-09-16T03:58:50.491 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.477+0000 7f97b9f1d640 10 monclient(hunting): authenticate will time out at 1977.451298s 2024-09-16T03:58:50.491 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.477+0000 7f97b2ffd640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f97b40a3d90 0x7f97b413c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:50.491 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.477+0000 7f97b37fe640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f97b40a39b0 0x7f97b4133b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:48020/0 (socket says 172.21.15.42:48020) 2024-09-16T03:58:50.500 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.477+0000 7f97b37fe640 1 -- 172.21.15.42:0/4156801354 learned_addr learned my addr 172.21.15.42:0/4156801354 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:58:50.501 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.477+0000 7f97b2ffd640 10 monclient(hunting): get_auth_request con 0x7f97b40a3d90 auth_method 0 2024-09-16T03:58:50.501 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.477+0000 7f97b3fff640 1 --2- 172.21.15.42:0/4156801354 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f97b413c8c0 0x7f97b413ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:50.501 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.477+0000 7f97b2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:50.501 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.477+0000 7f97b2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:50.501 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.477+0000 7f97b2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:58:50.501 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.477+0000 7f97b3fff640 10 monclient(hunting): get_auth_request con 0x7f97b413c8c0 auth_method 0 2024-09-16T03:58:50.501 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.477+0000 7f97b3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:50.501 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.477+0000 7f97b3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:50.501 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.477+0000 7f97b3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:58:50.501 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.477+0000 7f97b2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:58:50.502 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.477+0000 7f97b2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:58:50.502 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.477+0000 7f97b2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:58:50.502 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.477+0000 7f97b3fff640 1 -- 172.21.15.42:0/4156801354 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f97b413c8c0 msgr2=0x7f97b413ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T03:58:50.502 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.477+0000 7f97b3fff640 1 -- 172.21.15.42:0/4156801354 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f97b413c8c0 msgr2=0x7f97b413ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:58:50.502 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.477+0000 7f97b3fff640 1 --2- 172.21.15.42:0/4156801354 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f97b413c8c0 0x7f97b413ecb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:58:50.502 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.477+0000 7f97b3fff640 1 --2- 172.21.15.42:0/4156801354 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f97b413c8c0 0x7f97b413ecb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:58:50.502 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.477+0000 7f97b2ffd640 10 monclient(hunting): handle_auth_done global_id 825018 payload 274 2024-09-16T03:58:50.502 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.477+0000 7f97b2ffd640 1 -- 172.21.15.42:0/4156801354 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f97b413c8c0 msgr2=0x7f97b413ecb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:58:50.502 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.477+0000 7f97b2ffd640 1 --2- 172.21.15.42:0/4156801354 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f97b413c8c0 0x7f97b413ecb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:50.502 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.477+0000 7f97b2ffd640 1 -- 172.21.15.42:0/4156801354 >> v1:172.21.15.42:6789/0 conn(0x7f97b40a39b0 legacy=0x7f97b4133b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:50.502 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.477+0000 7f97b27fc640 1 -- 172.21.15.42:0/4156801354 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1181821097 0 0) 0x7f97b4134290 con 0x7f97b40a39b0 2024-09-16T03:58:50.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.477+0000 7f97b2ffd640 10 monclient: _finish_hunting 0 2024-09-16T03:58:50.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.477+0000 7f97b2ffd640 1 monclient: found mon.noname-a 2024-09-16T03:58:50.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.477+0000 7f97b2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:58:50.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.477+0000 7f97b2ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:58:50.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.477+0000 7f97b2ffd640 1 -- 172.21.15.42:0/4156801354 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97b406dd60 con 0x7f97b40a3d90 2024-09-16T03:58:50.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.477+0000 7f97b27fc640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T03:58:50.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.481+0000 7f97b2ffd640 1 --2- 172.21.15.42:0/4156801354 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f97b40a3d90 0x7f97b413c380 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f97a80097c0 tx=0x7f97a801c360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c03a2eff57054f3 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:58:50.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.481+0000 7f97b27fc640 1 -- 172.21.15.42:0/4156801354 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97a80b0070 con 0x7f97b40a3d90 2024-09-16T03:58:50.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.481+0000 7f97b27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:58:50.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.481+0000 7f97b27fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:58:50.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.481+0000 7f97b27fc640 10 monclient: dump: 2024-09-16T03:58:50.504 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:58:50.504 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:58:50.504 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:50.504 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:50.504 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:58:50.504 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:50.504 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:58:50.504 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:58:50.504 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:58:50.504 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:50.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.481+0000 7f97b27fc640 10 monclient: _finish_auth 0 2024-09-16T03:58:50.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.481+0000 7f97b27fc640 10 monclient: _check_auth_tickets 2024-09-16T03:58:50.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.481+0000 7f97b27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:58:50.505 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.481+0000 7f97b27fc640 1 -- 172.21.15.42:0/4156801354 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f97a80ab600 con 0x7f97b40a3d90 2024-09-16T03:58:50.505 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.481+0000 7f97b9f1d640 5 monclient: authenticate success, global_id 825018 2024-09-16T03:58:50.505 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.481+0000 7f97b9f1d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:58:50.505 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.481+0000 7f97b27fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:58:50.505 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.481+0000 7f97b27fc640 1 -- 172.21.15.42:0/4156801354 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97a80ab9d0 con 0x7f97b40a3d90 2024-09-16T03:58:50.505 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.481+0000 7f97b9f1d640 10 monclient: get_monmap_and_config success 2024-09-16T03:58:50.505 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.481+0000 7f97b9f1d640 4 set_mon_vals no callback set 2024-09-16T03:58:50.505 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.481+0000 7f97b27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:58:50.505 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.481+0000 7f97b27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:58:50.505 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.481+0000 7f97b27fc640 10 monclient: dump: 2024-09-16T03:58:50.505 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:58:50.505 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:58:50.505 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:50.505 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:50.506 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:58:50.506 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:50.506 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:58:50.506 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:58:50.506 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:58:50.506 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:50.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.481+0000 7f97b9f1d640 10 monclient: shutdown 2024-09-16T03:58:50.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.481+0000 7f97b9f1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:58:50.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.481+0000 7f97b9f1d640 1 -- 172.21.15.42:0/4156801354 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f97b40a3d90 msgr2=0x7f97b413c380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:50.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.481+0000 7f97b9f1d640 1 --2- 172.21.15.42:0/4156801354 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f97b40a3d90 0x7f97b413c380 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f97a80097c0 tx=0x7f97a801c360 comp rx=0 tx=0).stop 2024-09-16T03:58:50.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.481+0000 7f97b9f1d640 1 -- 172.21.15.42:0/4156801354 shutdown_connections 2024-09-16T03:58:50.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.481+0000 7f97b9f1d640 1 --2- 172.21.15.42:0/4156801354 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f97b413c8c0 0x7f97b413ecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:50.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.481+0000 7f97b9f1d640 1 --2- 172.21.15.42:0/4156801354 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f97b40a3d90 0x7f97b413c380 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:50.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.481+0000 7f97b9f1d640 1 -- 172.21.15.42:0/4156801354 >> 172.21.15.42:0/4156801354 conn(0x7f97b40a9fb0 msgr2=0x7f97b40aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:58:50.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.481+0000 7f97b9f1d640 1 -- 172.21.15.42:0/4156801354 shutdown_connections 2024-09-16T03:58:50.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.481+0000 7f97b9f1d640 1 -- 172.21.15.42:0/4156801354 wait complete. 2024-09-16T03:58:50.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.481+0000 7f97b9f1d640 10 monclient: build_initial_monmap 2024-09-16T03:58:50.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.481+0000 7f97b9f1d640 10 monclient: monmap: 2024-09-16T03:58:50.508 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:58:50.508 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:58:50.508 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:58:50.508 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:58:50.508 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:58:50.508 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:50.508 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:58:50.508 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:58:50.509 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:58:50.509 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:50.509 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.481+0000 7f97b9f1d640 1 librados: starting msgr at 2024-09-16T03:58:50.509 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.481+0000 7f97b9f1d640 1 librados: starting objecter 2024-09-16T03:58:50.509 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.481+0000 7f97b9f1d640 1 Processor -- start 2024-09-16T03:58:50.509 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.481+0000 7f97b9f1d640 1 -- start start 2024-09-16T03:58:50.509 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.481+0000 7f97b9f1d640 1 librados: setting wanted keys 2024-09-16T03:58:50.509 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.481+0000 7f97b9f1d640 1 librados: calling monclient init 2024-09-16T03:58:50.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.481+0000 7f97b9f1d640 10 monclient: init 2024-09-16T03:58:50.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.485+0000 7f97b9f1d640 10 monclient: _reopen_session rank -1 2024-09-16T03:58:50.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.485+0000 7f97b9f1d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:58:50.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.485+0000 7f97b9f1d640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f97b40a39b0 0x7f97b412d620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:50.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.485+0000 7f97b9f1d640 10 monclient(hunting): picked mon.noname-a con 0x7f97b40a39b0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:58:50.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.485+0000 7f97b9f1d640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f97b40a3d90 0x7f97b412db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:50.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.485+0000 7f97b9f1d640 10 monclient(hunting): picked mon.noname-c con 0x7f97b40a3d90 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:58:50.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.485+0000 7f97b9f1d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f97b413c8c0 0x7f97b41cf0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:50.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.485+0000 7f97b9f1d640 10 monclient(hunting): picked mon.noname-b con 0x7f97b413c8c0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:58:50.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.485+0000 7f97b9f1d640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:50.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.485+0000 7f97b9f1d640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f97b4141da0 con 0x7f97b40a39b0 2024-09-16T03:58:50.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.485+0000 7f97b9f1d640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:50.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.485+0000 7f97b9f1d640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f97b4141c20 con 0x7f97b413c8c0 2024-09-16T03:58:50.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.485+0000 7f97b9f1d640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:50.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.485+0000 7f97b9f1d640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f97b4141f20 con 0x7f97b40a3d90 2024-09-16T03:58:50.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.485+0000 7f97b9f1d640 10 monclient(hunting): _renew_subs 2024-09-16T03:58:50.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.485+0000 7f97b9f1d640 10 monclient(hunting): authenticate will time out at 1977.456229s 2024-09-16T03:58:50.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.485+0000 7f97b37fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f97b40a39b0 0x7f97b412d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:50.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.485+0000 7f97b2ffd640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f97b40a3d90 0x7f97b412db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:50.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.485+0000 7f97b37fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f97b40a39b0 0x7f97b412d620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:42028/0 (socket says 172.21.15.42:42028) 2024-09-16T03:58:50.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.485+0000 7f97b2ffd640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f97b40a3d90 0x7f97b412db60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:49768/0 (socket says 172.21.15.42:49768) 2024-09-16T03:58:50.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.485+0000 7f97b37fe640 1 -- 172.21.15.42:0/119916643 learned_addr learned my addr 172.21.15.42:0/119916643 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:58:50.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.485+0000 7f97b37fe640 10 monclient(hunting): get_auth_request con 0x7f97b40a39b0 auth_method 0 2024-09-16T03:58:50.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.485+0000 7f97b37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:50.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.485+0000 7f97b37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:50.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.485+0000 7f97b37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:58:50.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.485+0000 7f97b2ffd640 10 monclient(hunting): get_auth_request con 0x7f97b40a3d90 auth_method 0 2024-09-16T03:58:50.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.485+0000 7f97b2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:50.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.485+0000 7f97b2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:50.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.485+0000 7f97b2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:58:50.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.485+0000 7f97b37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:58:50.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.485+0000 7f97b37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:58:50.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.485+0000 7f97b37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:58:50.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.485+0000 7f97b2ffd640 1 -- 172.21.15.42:0/119916643 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f97b40a3d90 msgr2=0x7f97b412db60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-16T03:58:50.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.485+0000 7f97b2ffd640 1 -- 172.21.15.42:0/119916643 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f97b40a3d90 msgr2=0x7f97b412db60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T03:58:50.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.485+0000 7f97b2ffd640 1 --2- 172.21.15.42:0/119916643 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f97b40a3d90 0x7f97b412db60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:58:50.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.485+0000 7f97b3fff640 1 --2- 172.21.15.42:0/119916643 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f97b413c8c0 0x7f97b41cf0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:50.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.485+0000 7f97b2ffd640 1 --2- 172.21.15.42:0/119916643 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f97b40a3d90 0x7f97b412db60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:58:50.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.485+0000 7f97b37fe640 10 monclient(hunting): handle_auth_done global_id 825024 payload 824 2024-09-16T03:58:50.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.485+0000 7f97b37fe640 1 -- 172.21.15.42:0/119916643 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f97b40a3d90 msgr2=0x7f97b412db60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:58:50.513 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.485+0000 7f97b37fe640 1 --2- 172.21.15.42:0/119916643 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f97b40a3d90 0x7f97b412db60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:50.513 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.485+0000 7f97b37fe640 1 -- 172.21.15.42:0/119916643 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f97b413c8c0 msgr2=0x7f97b41cf0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:50.513 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.485+0000 7f97b37fe640 1 --2- 172.21.15.42:0/119916643 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f97b413c8c0 0x7f97b41cf0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:50.513 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.485+0000 7f97b37fe640 10 monclient: _finish_hunting 0 2024-09-16T03:58:50.513 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.485+0000 7f97b37fe640 1 monclient: found mon.noname-a 2024-09-16T03:58:50.513 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.485+0000 7f97b37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:58:50.513 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.485+0000 7f97b37fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:58:50.513 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.485+0000 7f97b37fe640 1 -- 172.21.15.42:0/119916643 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97b4132b90 con 0x7f97b40a39b0 2024-09-16T03:58:50.513 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.485+0000 7f97b3fff640 10 monclient: get_auth_request con 0x7f97b413c8c0 auth_method 0 2024-09-16T03:58:50.513 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.485+0000 7f97b3fff640 1 --2- 172.21.15.42:0/119916643 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f97b413c8c0 0x7f97b41cf0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T03:58:50.513 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.485+0000 7f97b37fe640 1 --2- 172.21.15.42:0/119916643 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f97b40a39b0 0x7f97b412d620 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f97a400c520 tx=0x7f97a400c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:58:50.513 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.485+0000 7f979bfff640 10 client.?.objecter ms_handle_connect 0x7f97b40a39b0 2024-09-16T03:58:50.513 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.485+0000 7f979bfff640 10 client.?.objecter resend_mon_ops 2024-09-16T03:58:50.514 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.485+0000 7f979bfff640 1 -- 172.21.15.42:0/119916643 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97a4018070 con 0x7f97b40a39b0 2024-09-16T03:58:50.514 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.485+0000 7f979bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:58:50.514 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.485+0000 7f979bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:58:50.514 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.485+0000 7f979bfff640 10 monclient: dump: 2024-09-16T03:58:50.514 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:58:50.514 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:58:50.514 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:50.514 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:50.514 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:58:50.514 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:50.514 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:58:50.514 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:58:50.514 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:58:50.515 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:50.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.485+0000 7f979bfff640 10 monclient: _finish_auth 0 2024-09-16T03:58:50.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.485+0000 7f979bfff640 10 monclient: _check_auth_tickets 2024-09-16T03:58:50.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.485+0000 7f979bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:58:50.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.485+0000 7f979bfff640 1 -- 172.21.15.42:0/119916643 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f97a40074e0 con 0x7f97b40a39b0 2024-09-16T03:58:50.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.485+0000 7f97b9f1d640 5 monclient: authenticate success, global_id 825024 2024-09-16T03:58:50.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.485+0000 7f97b9f1d640 10 monclient: _renew_subs 2024-09-16T03:58:50.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.485+0000 7f97b9f1d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:58:50.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.485+0000 7f97b9f1d640 1 -- 172.21.15.42:0/119916643 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97b413c350 con 0x7f97b40a39b0 2024-09-16T03:58:50.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.485+0000 7f979bfff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:58:50.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.485+0000 7f979bfff640 1 -- 172.21.15.42:0/119916643 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97a40049f0 con 0x7f97b40a39b0 2024-09-16T03:58:50.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.485+0000 7f979bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:58:50.516 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.485+0000 7f979bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:58:50.516 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.485+0000 7f979bfff640 10 monclient: dump: 2024-09-16T03:58:50.516 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:58:50.516 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:58:50.516 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:50.516 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:50.516 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:58:50.516 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:50.516 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:58:50.516 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:58:50.516 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:58:50.516 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:50.516 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.485+0000 7f97b9f1d640 10 client.825024.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:58:50.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.485+0000 7f97b9f1d640 10 monclient: _renew_subs 2024-09-16T03:58:50.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.485+0000 7f97b9f1d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:58:50.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.485+0000 7f97b9f1d640 1 -- 172.21.15.42:0/119916643 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97b413a920 con 0x7f97b40a39b0 2024-09-16T03:58:50.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.485+0000 7f97b9f1d640 1 librados: init done 2024-09-16T03:58:50.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.485+0000 7f97b1ffb640 4 set_mon_vals no callback set 2024-09-16T03:58:50.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.485+0000 7f979bfff640 1 -- 172.21.15.42:0/119916643 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f97a401b070 con 0x7f97b40a39b0 2024-09-16T03:58:50.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.485+0000 7f97b9f1d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:58:50.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.485+0000 7f97b9f1d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:58:50.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.485+0000 7f97b9f1d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:58:50.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.485+0000 7f97b9f1d640 1 -- 172.21.15.42:0/119916643 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97b4133b80 con 0x7f97b40a39b0 2024-09-16T03:58:50.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.485+0000 7f979bfff640 1 --2- 172.21.15.42:0/119916643 >> v2:172.21.15.154:6812/254845342 conn(0x7f97a0037840 0x7f97a0039d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:50.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.485+0000 7f979bfff640 1 -- 172.21.15.42:0/119916643 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(507..507 src has 366..507) ==== 7041+0+0 (secure 0 0 0) 0x7f97a40488a0 con 0x7f97b40a39b0 2024-09-16T03:58:50.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.485+0000 7f979bfff640 10 client.825024.objecter ms_dispatch 0x7f97b4000ba0 osd_map(507..507 src has 366..507) 2024-09-16T03:58:50.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.485+0000 7f979bfff640 3 client.825024.objecter handle_osd_map got epochs [507,507] > 0 2024-09-16T03:58:50.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.485+0000 7f979bfff640 3 client.825024.objecter handle_osd_map decoding full epoch 507 2024-09-16T03:58:50.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.485+0000 7f979bfff640 20 client.825024.objecter dump_active .. 0 homeless 2024-09-16T03:58:50.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.485+0000 7f97b2ffd640 1 --2- 172.21.15.42:0/119916643 >> v2:172.21.15.154:6812/254845342 conn(0x7f97a0037840 0x7f97a0039d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:50.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.485+0000 7f97b2ffd640 10 monclient: get_auth_request con 0x7f97a0037840 auth_method 0 2024-09-16T03:58:50.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.485+0000 7f97b2ffd640 1 --2- 172.21.15.42:0/119916643 >> v2:172.21.15.154:6812/254845342 conn(0x7f97a0037840 0x7f97a0039d00 secure :-1 s=READY pgs=3708 cs=0 l=1 rev1=1 crypto rx=0x7f97a80ad000 tx=0x7f97a80a6000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:58:50.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.485+0000 7f979bfff640 10 client.825024.objecter ms_handle_connect 0x7f97a0037840 2024-09-16T03:58:50.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.489+0000 7f979bfff640 1 -- 172.21.15.42:0/119916643 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f97a401ae10 con 0x7f97b40a39b0 2024-09-16T03:58:50.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.489+0000 7f979bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:58:50.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.489+0000 7f979bfff640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:58:50.724 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.717+0000 7f97b9f1d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:58:50.724 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.717+0000 7f97b9f1d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:58:50.724 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.717+0000 7f97b9f1d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:58:50.724 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.717+0000 7f97b9f1d640 1 -- 172.21.15.42:0/119916643 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f97b40a5750 con 0x7f97b40a39b0 2024-09-16T03:58:50.725 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.717+0000 7f979bfff640 1 -- 172.21.15.42:0/119916643 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7f97a4010e00 con 0x7f97b40a39b0 2024-09-16T03:58:50.725 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.717+0000 7f979bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:58:50.725 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.717+0000 7f979bfff640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:58:50.725 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":152,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":21,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:58:50.727 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.721+0000 7f97b9f1d640 10 librados: watch_flush enter 2024-09-16T03:58:50.727 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.721+0000 7f97b9f1d640 10 librados: watch_flush exit 2024-09-16T03:58:50.727 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.721+0000 7f97b9f1d640 20 client.825024.objecter shutdown clearing up homeless session... 2024-09-16T03:58:50.727 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.721+0000 7f97b9f1d640 10 client.825024.objecter successfully canceled tick 2024-09-16T03:58:50.727 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.721+0000 7f97b9f1d640 1 -- 172.21.15.42:0/119916643 >> v2:172.21.15.154:6812/254845342 conn(0x7f97a0037840 msgr2=0x7f97a0039d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:50.728 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.721+0000 7f97b9f1d640 1 --2- 172.21.15.42:0/119916643 >> v2:172.21.15.154:6812/254845342 conn(0x7f97a0037840 0x7f97a0039d00 secure :-1 s=READY pgs=3708 cs=0 l=1 rev1=1 crypto rx=0x7f97a80ad000 tx=0x7f97a80a6000 comp rx=0 tx=0).stop 2024-09-16T03:58:50.728 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.721+0000 7f97b9f1d640 10 monclient: shutdown 2024-09-16T03:58:50.728 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.721+0000 7f97b9f1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:58:50.728 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.721+0000 7f97b9f1d640 1 -- 172.21.15.42:0/119916643 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f97b40a39b0 msgr2=0x7f97b412d620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:50.728 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.721+0000 7f97b9f1d640 1 --2- 172.21.15.42:0/119916643 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f97b40a39b0 0x7f97b412d620 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f97a400c520 tx=0x7f97a400c9e0 comp rx=0 tx=0).stop 2024-09-16T03:58:50.728 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.721+0000 7f97b9f1d640 1 -- 172.21.15.42:0/119916643 shutdown_connections 2024-09-16T03:58:50.728 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.721+0000 7f97b9f1d640 1 --2- 172.21.15.42:0/119916643 >> v2:172.21.15.154:6812/254845342 conn(0x7f97a0037840 0x7f97a0039d00 unknown :-1 s=CLOSED pgs=3708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:50.728 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.721+0000 7f97b9f1d640 1 --2- 172.21.15.42:0/119916643 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f97b413c8c0 0x7f97b41cf0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:50.728 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.721+0000 7f97b9f1d640 1 --2- 172.21.15.42:0/119916643 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f97b40a3d90 0x7f97b412db60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:50.728 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.721+0000 7f97b9f1d640 1 --2- 172.21.15.42:0/119916643 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f97b40a39b0 0x7f97b412d620 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:50.728 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.721+0000 7f97b9f1d640 1 -- 172.21.15.42:0/119916643 >> 172.21.15.42:0/119916643 conn(0x7f97b40a9fb0 msgr2=0x7f97b4131980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:58:50.728 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.721+0000 7f97b9f1d640 1 -- 172.21.15.42:0/119916643 shutdown_connections 2024-09-16T03:58:50.728 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.721+0000 7f97b9f1d640 1 -- 172.21.15.42:0/119916643 wait complete. 2024-09-16T03:58:50.728 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:50.721+0000 7f97b9f1d640 1 librados: shutdown 2024-09-16T03:58:53.746 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:58:53.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.873+0000 7fef9b134640 10 monclient: get_monmap_and_config 2024-09-16T03:58:53.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.873+0000 7fef9b134640 10 monclient: build_initial_monmap 2024-09-16T03:58:53.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.873+0000 7fef9b134640 10 monclient: monmap: 2024-09-16T03:58:53.880 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:58:53.881 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:58:53.881 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:58:53.878720+0000 2024-09-16T03:58:53.881 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:58:53.878720+0000 2024-09-16T03:58:53.881 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:58:53.881 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:53.881 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:58:53.881 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:58:53.881 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:58:53.881 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:58:53.881 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:58:53.881 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:53.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.873+0000 7fef9b134640 1 Processor -- start 2024-09-16T03:58:53.888 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.873+0000 7fef9b134640 1 -- start start 2024-09-16T03:58:53.888 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.873+0000 7fef9b134640 10 monclient: init 2024-09-16T03:58:53.888 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.877+0000 7fef9b134640 10 monclient: _reopen_session rank -1 2024-09-16T03:58:53.888 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.877+0000 7fef9b134640 10 monclient: _add_conns ranks=[0,4,1,2,3] 2024-09-16T03:58:53.888 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.877+0000 7fef9b134640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fef940a1330 addr v1:172.21.15.42:6789/0 2024-09-16T03:58:53.888 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.877+0000 7fef9b134640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fef9406dac0 0x7fef9413a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:53.888 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.877+0000 7fef9b134640 10 monclient(hunting): picked mon.noname-c con 0x7fef9406dac0 addr v2:172.21.15.154:3300/0 2024-09-16T03:58:53.888 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.877+0000 7fef9b134640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fef940a1e20 addr v1:172.21.15.154:6789/0 2024-09-16T03:58:53.888 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.877+0000 7fef9b134640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fef9409c090 con 0x7fef940a1330 2024-09-16T03:58:53.889 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.877+0000 7fef9b134640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fef9409f610 con 0x7fef940a1e20 2024-09-16T03:58:53.889 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.877+0000 7fef9b134640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:53.889 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.877+0000 7fef9b134640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fef9409f8f0 con 0x7fef9406dac0 2024-09-16T03:58:53.889 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.877+0000 7fef9b134640 10 monclient(hunting): _renew_subs 2024-09-16T03:58:53.889 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.877+0000 7fef9b134640 10 monclient(hunting): authenticate will time out at 1980.848216s 2024-09-16T03:58:53.889 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.877+0000 7fef98ed0640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fef940a1330 0x7fef940a1710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:48026/0 (socket says 172.21.15.42:48026) 2024-09-16T03:58:53.889 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.877+0000 7fef98ed0640 1 -- 172.21.15.42:0/1891670468 learned_addr learned my addr 172.21.15.42:0/1891670468 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:58:53.889 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.877+0000 7fef937fe640 1 -- 172.21.15.42:0/1891670468 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2611879132 0 0) 0x7fef9409c090 con 0x7fef940a1330 2024-09-16T03:58:53.889 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.877+0000 7fef937fe640 10 monclient(hunting): _negotiate 2024-09-16T03:58:53.889 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.877+0000 7fef937fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:53.889 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.877+0000 7fef937fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:58:53.889 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.877+0000 7fef937fe640 10 monclient(hunting): my global_id is 825027 2024-09-16T03:58:53.890 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.877+0000 7fef93fff640 1 --2- 172.21.15.42:0/1891670468 >> v2:172.21.15.154:3300/0 conn(0x7fef9406dac0 0x7fef9413a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:53.890 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.877+0000 7fef937fe640 1 -- 172.21.15.42:0/1891670468 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fef80003530 con 0x7fef940a1330 2024-09-16T03:58:53.890 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.877+0000 7fef93fff640 10 monclient(hunting): get_auth_request con 0x7fef9406dac0 auth_method 0 2024-09-16T03:58:53.890 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.877+0000 7fef93fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:53.890 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.877+0000 7fef93fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:53.890 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.877+0000 7fef93fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:58:53.890 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.877+0000 7fef93fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:58:53.890 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.877+0000 7fef93fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:58:53.890 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.877+0000 7fef93fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:58:53.890 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.877+0000 7fef937fe640 1 -- 172.21.15.42:0/1891670468 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3625186757 0 0) 0x7fef80003530 con 0x7fef940a1330 2024-09-16T03:58:53.890 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.877+0000 7fef937fe640 1 -- 172.21.15.42:0/1891670468 >> v2:172.21.15.154:3300/0 conn(0x7fef9406dac0 msgr2=0x7fef9413a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:58:53.890 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.877+0000 7fef937fe640 1 --2- 172.21.15.42:0/1891670468 >> v2:172.21.15.154:3300/0 conn(0x7fef9406dac0 0x7fef9413a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:53.891 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.877+0000 7fef937fe640 1 -- 172.21.15.42:0/1891670468 >> v1:172.21.15.154:6789/0 conn(0x7fef940a1e20 legacy=0x7fef9413c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:53.891 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.877+0000 7fef937fe640 10 monclient: _finish_hunting 0 2024-09-16T03:58:53.891 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.877+0000 7fef937fe640 1 monclient: found mon.noname-a-legacy 2024-09-16T03:58:53.891 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.877+0000 7fef937fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:58:53.891 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.877+0000 7fef937fe640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.42:6789/0 2024-09-16T03:58:53.891 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.877+0000 7fef937fe640 1 -- 172.21.15.42:0/1891670468 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef9409fbd0 con 0x7fef940a1330 2024-09-16T03:58:53.891 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.877+0000 7fef937fe640 10 monclient: _finish_auth 0 2024-09-16T03:58:53.891 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.877+0000 7fef937fe640 10 monclient: _check_auth_tickets 2024-09-16T03:58:53.891 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.877+0000 7fef937fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:58:53.891 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.877+0000 7fef9b134640 5 monclient: authenticate success, global_id 825027 2024-09-16T03:58:53.891 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.877+0000 7fef9b134640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:58:53.891 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.877+0000 7fef937fe640 1 -- 172.21.15.42:0/1891670468 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2177413734 0 0) 0x7fef84001680 con 0x7fef940a1330 2024-09-16T03:58:53.892 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.877+0000 7fef937fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:58:53.892 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.877+0000 7fef937fe640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T03:58:53.892 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.877+0000 7fef937fe640 10 monclient: dump: 2024-09-16T03:58:53.892 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:58:53.892 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:58:53.892 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:53.892 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:53.892 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:58:53.892 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:53.892 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:58:53.892 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:58:53.892 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:58:53.892 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:53.892 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.877+0000 7fef937fe640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] but i'm connected to v1:172.21.15.42:6789/0, reconnecting 2024-09-16T03:58:53.893 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.877+0000 7fef937fe640 10 monclient: _reopen_session rank -1 2024-09-16T03:58:53.893 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.877+0000 7fef937fe640 1 -- 172.21.15.42:0/1891670468 >> v1:172.21.15.42:6789/0 conn(0x7fef940a1330 legacy=0x7fef940a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:53.893 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.877+0000 7fef937fe640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:58:53.893 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.877+0000 7fef937fe640 1 --2- 172.21.15.42:0/1891670468 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fef800039c0 0x7fef80003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:53.893 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.877+0000 7fef937fe640 10 monclient(hunting): picked mon.c con 0x7fef800039c0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:58:53.893 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.877+0000 7fef937fe640 1 --2- 172.21.15.42:0/1891670468 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fef80004810 0x7fef80004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:53.893 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.877+0000 7fef937fe640 10 monclient(hunting): picked mon.a con 0x7fef80004810 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:58:53.893 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.877+0000 7fef937fe640 1 --2- 172.21.15.42:0/1891670468 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fef800051d0 0x7fef8000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:53.893 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.877+0000 7fef937fe640 10 monclient(hunting): picked mon.b con 0x7fef800051d0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:58:53.893 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.877+0000 7fef937fe640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:53.893 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.877+0000 7fef937fe640 1 -- 172.21.15.42:0/1891670468 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fef9409f8f0 con 0x7fef80004810 2024-09-16T03:58:53.893 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.877+0000 7fef937fe640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:53.893 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.877+0000 7fef937fe640 1 -- 172.21.15.42:0/1891670468 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fef8000c170 con 0x7fef800051d0 2024-09-16T03:58:53.894 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.877+0000 7fef937fe640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:53.894 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.877+0000 7fef937fe640 1 -- 172.21.15.42:0/1891670468 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fef8000c370 con 0x7fef800039c0 2024-09-16T03:58:53.894 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.877+0000 7fef937fe640 10 monclient(hunting): _renew_subs 2024-09-16T03:58:53.894 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.877+0000 7fef937fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:58:53.894 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.877+0000 7fef9b134640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-16T03:58:53.894 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.877+0000 7fef93fff640 1 --2- 172.21.15.42:0/1891670468 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fef80004810 0x7fef80004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:53.894 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.877+0000 7fef93fff640 10 monclient(hunting): get_auth_request con 0x7fef80004810 auth_method 0 2024-09-16T03:58:53.894 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.877+0000 7fef93fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:53.894 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.877+0000 7fef93fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:53.894 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.877+0000 7fef93fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:58:53.894 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.877+0000 7fef98ed0640 1 --2- 172.21.15.42:0/1891670468 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fef800039c0 0x7fef80003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:53.894 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.877+0000 7fef93fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:58:53.894 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.877+0000 7fef93fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:58:53.894 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.877+0000 7fef93fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T03:58:53.895 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.877+0000 7fef98ed0640 10 monclient(hunting): get_auth_request con 0x7fef800039c0 auth_method 0 2024-09-16T03:58:53.895 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.877+0000 7fef98ed0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:53.895 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.877+0000 7fef98ed0640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:53.895 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.877+0000 7fef98ed0640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:58:53.895 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.877+0000 7fef996d1640 1 --2- 172.21.15.42:0/1891670468 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fef800051d0 0x7fef8000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:53.895 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.877+0000 7fef98ed0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:58:53.895 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.877+0000 7fef98ed0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:58:53.895 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.877+0000 7fef98ed0640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T03:58:53.895 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.877+0000 7fef996d1640 10 monclient(hunting): get_auth_request con 0x7fef800051d0 auth_method 0 2024-09-16T03:58:53.895 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.877+0000 7fef996d1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:53.895 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.877+0000 7fef996d1640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:53.895 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.877+0000 7fef996d1640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:58:53.895 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.877+0000 7fef93fff640 10 monclient(hunting): handle_auth_done global_id 825027 payload 293 2024-09-16T03:58:53.896 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.877+0000 7fef93fff640 1 -- 172.21.15.42:0/1891670468 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fef800039c0 msgr2=0x7fef80003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:58:53.896 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.877+0000 7fef93fff640 1 --2- 172.21.15.42:0/1891670468 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fef800039c0 0x7fef80003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:53.896 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.877+0000 7fef93fff640 1 -- 172.21.15.42:0/1891670468 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fef800051d0 msgr2=0x7fef8000b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:58:53.896 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.877+0000 7fef93fff640 1 --2- 172.21.15.42:0/1891670468 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fef800051d0 0x7fef8000b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:53.896 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.877+0000 7fef93fff640 10 monclient: _finish_hunting 0 2024-09-16T03:58:53.896 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.877+0000 7fef93fff640 1 monclient: found mon.a 2024-09-16T03:58:53.896 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.877+0000 7fef93fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:58:53.896 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.877+0000 7fef996d1640 1 -- 172.21.15.42:0/1891670468 reap_dead start 2024-09-16T03:58:53.896 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.877+0000 7fef93fff640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:58:53.896 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.877+0000 7fef93fff640 1 -- 172.21.15.42:0/1891670468 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef80003530 con 0x7fef80004810 2024-09-16T03:58:53.896 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.877+0000 7fef93fff640 10 monclient: _finish_auth 0 2024-09-16T03:58:53.896 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.877+0000 7fef93fff640 10 monclient: _check_auth_tickets 2024-09-16T03:58:53.896 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.877+0000 7fef93fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:58:53.896 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.881+0000 7fef93fff640 1 --2- 172.21.15.42:0/1891670468 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fef80004810 0x7fef80004c90 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fef7c009ae0 tx=0x7fef7c01ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41e2f7813043d878 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:58:53.897 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.881+0000 7fef937fe640 1 -- 172.21.15.42:0/1891670468 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef7c0096b0 con 0x7fef80004810 2024-09-16T03:58:53.897 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.881+0000 7fef937fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:58:53.897 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.881+0000 7fef937fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:58:53.897 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.881+0000 7fef937fe640 10 monclient: dump: 2024-09-16T03:58:53.897 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:58:53.897 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:58:53.897 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:53.897 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:53.897 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:58:53.897 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:53.897 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:58:53.898 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:58:53.898 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:58:53.898 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:53.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.881+0000 7fef937fe640 1 -- 172.21.15.42:0/1891670468 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fef7c0a6430 con 0x7fef80004810 2024-09-16T03:58:53.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.881+0000 7fef9b134640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:58:53.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.881+0000 7fef937fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:58:53.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.881+0000 7fef9b134640 10 monclient: get_monmap_and_config success 2024-09-16T03:58:53.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.881+0000 7fef937fe640 1 -- 172.21.15.42:0/1891670468 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef7c0a6820 con 0x7fef80004810 2024-09-16T03:58:53.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.881+0000 7fef9b134640 4 set_mon_vals no callback set 2024-09-16T03:58:53.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.881+0000 7fef937fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:58:53.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.881+0000 7fef937fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:58:53.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.881+0000 7fef937fe640 10 monclient: dump: 2024-09-16T03:58:53.899 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:58:53.899 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:58:53.900 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:53.900 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:53.900 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:58:53.900 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:53.900 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:58:53.900 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:58:53.901 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:58:53.901 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:53.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.881+0000 7fef9b134640 10 monclient: shutdown 2024-09-16T03:58:53.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.881+0000 7fef9b134640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:58:53.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.881+0000 7fef9b134640 1 -- 172.21.15.42:0/1891670468 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fef80004810 msgr2=0x7fef80004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:53.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.881+0000 7fef9b134640 1 --2- 172.21.15.42:0/1891670468 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fef80004810 0x7fef80004c90 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fef7c009ae0 tx=0x7fef7c01ca50 comp rx=0 tx=0).stop 2024-09-16T03:58:53.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.881+0000 7fef9b134640 1 -- 172.21.15.42:0/1891670468 shutdown_connections 2024-09-16T03:58:53.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.881+0000 7fef9b134640 1 --2- 172.21.15.42:0/1891670468 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fef80004810 0x7fef80004c90 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:53.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.881+0000 7fef9b134640 1 -- 172.21.15.42:0/1891670468 >> 172.21.15.42:0/1891670468 conn(0x7fef940a2420 msgr2=0x7fef940a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:58:53.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.881+0000 7fef9b134640 1 -- 172.21.15.42:0/1891670468 shutdown_connections 2024-09-16T03:58:53.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.881+0000 7fef9b134640 1 -- 172.21.15.42:0/1891670468 wait complete. 2024-09-16T03:58:53.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.881+0000 7fef9b134640 10 monclient: build_initial_monmap 2024-09-16T03:58:53.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.881+0000 7fef9b134640 10 monclient: monmap: 2024-09-16T03:58:53.903 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:58:53.903 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:58:53.903 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:58:53.903 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:58:53.903 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:58:53.903 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:53.904 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:58:53.904 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:58:53.904 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:58:53.904 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:53.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.881+0000 7fef9b134640 1 librados: starting msgr at 2024-09-16T03:58:53.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.881+0000 7fef9b134640 1 librados: starting objecter 2024-09-16T03:58:53.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.881+0000 7fef9b134640 1 Processor -- start 2024-09-16T03:58:53.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.881+0000 7fef9b134640 1 -- start start 2024-09-16T03:58:53.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.881+0000 7fef9b134640 1 librados: setting wanted keys 2024-09-16T03:58:53.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.881+0000 7fef9b134640 1 librados: calling monclient init 2024-09-16T03:58:53.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.881+0000 7fef9b134640 10 monclient: init 2024-09-16T03:58:53.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.881+0000 7fef9b134640 10 monclient: _reopen_session rank -1 2024-09-16T03:58:53.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.881+0000 7fef9b134640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:58:53.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.881+0000 7fef9b134640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fef80004810 0x7fef941cd6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:53.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.881+0000 7fef9b134640 10 monclient(hunting): picked mon.noname-b con 0x7fef80004810 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:58:53.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.881+0000 7fef9b134640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fef941cdc30 0x7fef94122bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:53.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.881+0000 7fef9b134640 10 monclient(hunting): picked mon.noname-a con 0x7fef941cdc30 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:58:53.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.881+0000 7fef9b134640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fef94123110 0x7fef94123590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:53.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.881+0000 7fef9b134640 10 monclient(hunting): picked mon.noname-c con 0x7fef94123110 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:58:53.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.881+0000 7fef9b134640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:53.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.881+0000 7fef9b134640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fef94135910 con 0x7fef941cdc30 2024-09-16T03:58:53.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.881+0000 7fef9b134640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:53.916 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.881+0000 7fef9b134640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fef9409ef80 con 0x7fef80004810 2024-09-16T03:58:53.916 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.881+0000 7fef9b134640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:53.916 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.881+0000 7fef9b134640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fef94135a90 con 0x7fef94123110 2024-09-16T03:58:53.916 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.881+0000 7fef9b134640 10 monclient(hunting): _renew_subs 2024-09-16T03:58:53.916 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.881+0000 7fef9b134640 10 monclient(hunting): authenticate will time out at 1980.855079s 2024-09-16T03:58:53.917 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.881+0000 7fef996d1640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fef94123110 0x7fef94123590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:53.917 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.881+0000 7fef93fff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fef941cdc30 0x7fef94122bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:53.917 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.881+0000 7fef93fff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fef941cdc30 0x7fef94122bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:42076/0 (socket says 172.21.15.42:42076) 2024-09-16T03:58:53.917 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.881+0000 7fef996d1640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fef94123110 0x7fef94123590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:49786/0 (socket says 172.21.15.42:49786) 2024-09-16T03:58:53.917 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.881+0000 7fef93fff640 1 -- 172.21.15.42:0/2417829217 learned_addr learned my addr 172.21.15.42:0/2417829217 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:58:53.917 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.881+0000 7fef93fff640 10 monclient(hunting): get_auth_request con 0x7fef941cdc30 auth_method 0 2024-09-16T03:58:53.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.881+0000 7fef93fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:53.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.881+0000 7fef93fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:53.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.881+0000 7fef93fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:58:53.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.881+0000 7fef996d1640 10 monclient(hunting): get_auth_request con 0x7fef94123110 auth_method 0 2024-09-16T03:58:53.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.881+0000 7fef996d1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:53.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.881+0000 7fef996d1640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:53.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.881+0000 7fef996d1640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:58:53.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.881+0000 7fef93fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:58:53.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.881+0000 7fef93fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:58:53.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.881+0000 7fef93fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:58:53.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.881+0000 7fef996d1640 1 -- 172.21.15.42:0/2417829217 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fef94123110 msgr2=0x7fef94123590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-16T03:58:53.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.881+0000 7fef996d1640 1 -- 172.21.15.42:0/2417829217 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fef94123110 msgr2=0x7fef94123590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T03:58:53.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.881+0000 7fef996d1640 1 --2- 172.21.15.42:0/2417829217 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fef94123110 0x7fef94123590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:58:53.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.881+0000 7fef98ed0640 1 --2- 172.21.15.42:0/2417829217 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fef80004810 0x7fef941cd6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:53.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.881+0000 7fef996d1640 1 --2- 172.21.15.42:0/2417829217 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fef94123110 0x7fef94123590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:58:53.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.881+0000 7fef98ed0640 10 monclient(hunting): get_auth_request con 0x7fef80004810 auth_method 0 2024-09-16T03:58:53.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.881+0000 7fef98ed0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:53.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.881+0000 7fef98ed0640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:53.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.881+0000 7fef98ed0640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:58:53.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.881+0000 7fef93fff640 10 monclient(hunting): handle_auth_done global_id 825030 payload 824 2024-09-16T03:58:53.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.881+0000 7fef93fff640 1 -- 172.21.15.42:0/2417829217 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fef94123110 msgr2=0x7fef94123590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:58:53.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.881+0000 7fef93fff640 1 --2- 172.21.15.42:0/2417829217 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fef94123110 0x7fef94123590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:53.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.881+0000 7fef93fff640 1 -- 172.21.15.42:0/2417829217 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fef80004810 msgr2=0x7fef941cd6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:53.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.881+0000 7fef93fff640 1 --2- 172.21.15.42:0/2417829217 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fef80004810 0x7fef941cd6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:53.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.881+0000 7fef93fff640 10 monclient: _finish_hunting 0 2024-09-16T03:58:53.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.881+0000 7fef93fff640 1 monclient: found mon.noname-a 2024-09-16T03:58:53.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.881+0000 7fef93fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:58:53.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.881+0000 7fef93fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:58:53.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.881+0000 7fef93fff640 1 -- 172.21.15.42:0/2417829217 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef940712f0 con 0x7fef941cdc30 2024-09-16T03:58:53.923 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.881+0000 7fef93fff640 1 --2- 172.21.15.42:0/2417829217 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fef941cdc30 0x7fef94122bd0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fef7c009ae0 tx=0x7fef7c0197f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:58:53.923 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.881+0000 7fef917fa640 10 client.?.objecter ms_handle_connect 0x7fef941cdc30 2024-09-16T03:58:53.923 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.881+0000 7fef917fa640 10 client.?.objecter resend_mon_ops 2024-09-16T03:58:53.923 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.885+0000 7fef917fa640 1 -- 172.21.15.42:0/2417829217 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef7c0be070 con 0x7fef941cdc30 2024-09-16T03:58:53.923 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.885+0000 7fef917fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:58:53.923 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.885+0000 7fef917fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:58:53.923 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.885+0000 7fef917fa640 10 monclient: dump: 2024-09-16T03:58:53.924 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:58:53.924 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:58:53.924 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:53.924 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:53.924 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:58:53.924 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:53.925 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:58:53.925 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:58:53.925 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:58:53.925 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:53.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.885+0000 7fef917fa640 10 monclient: _finish_auth 0 2024-09-16T03:58:53.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.885+0000 7fef917fa640 10 monclient: _check_auth_tickets 2024-09-16T03:58:53.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.885+0000 7fef917fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:58:53.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.885+0000 7fef917fa640 1 -- 172.21.15.42:0/2417829217 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fef7c00f000 con 0x7fef941cdc30 2024-09-16T03:58:53.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.885+0000 7fef917fa640 10 monclient: handle_config config(0 keys) 2024-09-16T03:58:53.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.885+0000 7fef917fa640 1 -- 172.21.15.42:0/2417829217 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef7c0b9410 con 0x7fef941cdc30 2024-09-16T03:58:53.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.885+0000 7fef9b134640 5 monclient: authenticate success, global_id 825030 2024-09-16T03:58:53.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.885+0000 7fef9b134640 10 monclient: _renew_subs 2024-09-16T03:58:53.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.885+0000 7fef9b134640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:58:53.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.885+0000 7fef9b134640 1 -- 172.21.15.42:0/2417829217 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef94124c70 con 0x7fef941cdc30 2024-09-16T03:58:53.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.885+0000 7fef917fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:58:53.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.885+0000 7fef917fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:58:53.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.885+0000 7fef92ffd640 4 set_mon_vals no callback set 2024-09-16T03:58:53.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.885+0000 7fef917fa640 10 monclient: dump: 2024-09-16T03:58:53.926 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:58:53.927 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:58:53.927 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:53.927 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:53.927 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:58:53.927 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:53.927 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:58:53.927 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:58:53.927 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:58:53.927 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:53.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.885+0000 7fef9b134640 10 client.825030.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:58:53.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.885+0000 7fef9b134640 10 monclient: _renew_subs 2024-09-16T03:58:53.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.885+0000 7fef9b134640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:58:53.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.885+0000 7fef9b134640 1 -- 172.21.15.42:0/2417829217 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef94126500 con 0x7fef941cdc30 2024-09-16T03:58:53.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.885+0000 7fef9b134640 1 librados: init done 2024-09-16T03:58:53.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.885+0000 7fef917fa640 1 -- 172.21.15.42:0/2417829217 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fef7c0a7410 con 0x7fef941cdc30 2024-09-16T03:58:53.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.885+0000 7fef9b134640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:58:53.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.885+0000 7fef9b134640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:58:53.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.885+0000 7fef9b134640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:58:53.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.885+0000 7fef9b134640 1 -- 172.21.15.42:0/2417829217 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fef94135e20 con 0x7fef941cdc30 2024-09-16T03:58:53.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.885+0000 7fef917fa640 1 --2- 172.21.15.42:0/2417829217 >> v2:172.21.15.154:6812/254845342 conn(0x7fef74037810 0x7fef74039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:53.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.885+0000 7fef917fa640 1 -- 172.21.15.42:0/2417829217 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(510..510 src has 366..510) ==== 6235+0+0 (secure 0 0 0) 0x7fef7c0e2a90 con 0x7fef941cdc30 2024-09-16T03:58:53.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.885+0000 7fef917fa640 10 client.825030.objecter ms_dispatch 0x7fef94000ba0 osd_map(510..510 src has 366..510) 2024-09-16T03:58:53.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.885+0000 7fef917fa640 3 client.825030.objecter handle_osd_map got epochs [510,510] > 0 2024-09-16T03:58:53.929 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.885+0000 7fef917fa640 3 client.825030.objecter handle_osd_map decoding full epoch 510 2024-09-16T03:58:53.929 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.885+0000 7fef917fa640 20 client.825030.objecter dump_active .. 0 homeless 2024-09-16T03:58:53.929 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.885+0000 7fef98ed0640 1 --2- 172.21.15.42:0/2417829217 >> v2:172.21.15.154:6812/254845342 conn(0x7fef74037810 0x7fef74039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:53.929 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.885+0000 7fef98ed0640 10 monclient: get_auth_request con 0x7fef74037810 auth_method 0 2024-09-16T03:58:53.929 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.885+0000 7fef917fa640 10 client.825030.objecter ms_handle_connect 0x7fef74037810 2024-09-16T03:58:53.929 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.885+0000 7fef98ed0640 1 --2- 172.21.15.42:0/2417829217 >> v2:172.21.15.154:6812/254845342 conn(0x7fef74037810 0x7fef74039cd0 secure :-1 s=READY pgs=3709 cs=0 l=1 rev1=1 crypto rx=0x7fef84000c10 tx=0x7fef84009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:58:53.929 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.889+0000 7fef917fa640 1 -- 172.21.15.42:0/2417829217 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7fef940712f0 con 0x7fef941cdc30 2024-09-16T03:58:53.929 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.889+0000 7fef917fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:58:53.929 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:53.889+0000 7fef917fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:58:54.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:54.117+0000 7fef9b134640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:58:54.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:54.117+0000 7fef9b134640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:58:54.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:54.117+0000 7fef9b134640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:58:54.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:54.117+0000 7fef9b134640 1 -- 172.21.15.42:0/2417829217 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fef9411f690 con 0x7fef941cdc30 2024-09-16T03:58:54.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:54.117+0000 7fef917fa640 1 -- 172.21.15.42:0/2417829217 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7fef7c0ac020 con 0x7fef941cdc30 2024-09-16T03:58:54.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:54.117+0000 7fef917fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:58:54.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:54.117+0000 7fef917fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:58:54.124 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":152,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":24,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:58:54.126 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:54.121+0000 7fef9b134640 10 librados: watch_flush enter 2024-09-16T03:58:54.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:54.121+0000 7fef9b134640 10 librados: watch_flush exit 2024-09-16T03:58:54.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:54.121+0000 7fef9b134640 20 client.825030.objecter shutdown clearing up homeless session... 2024-09-16T03:58:54.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:54.121+0000 7fef9b134640 10 client.825030.objecter successfully canceled tick 2024-09-16T03:58:54.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:54.121+0000 7fef9b134640 1 -- 172.21.15.42:0/2417829217 >> v2:172.21.15.154:6812/254845342 conn(0x7fef74037810 msgr2=0x7fef74039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:54.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:54.121+0000 7fef9b134640 1 --2- 172.21.15.42:0/2417829217 >> v2:172.21.15.154:6812/254845342 conn(0x7fef74037810 0x7fef74039cd0 secure :-1 s=READY pgs=3709 cs=0 l=1 rev1=1 crypto rx=0x7fef84000c10 tx=0x7fef84009040 comp rx=0 tx=0).stop 2024-09-16T03:58:54.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:54.121+0000 7fef9b134640 10 monclient: shutdown 2024-09-16T03:58:54.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:54.121+0000 7fef9b134640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:58:54.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:54.121+0000 7fef9b134640 1 -- 172.21.15.42:0/2417829217 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fef941cdc30 msgr2=0x7fef94122bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:54.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:54.121+0000 7fef9b134640 1 --2- 172.21.15.42:0/2417829217 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fef941cdc30 0x7fef94122bd0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fef7c009ae0 tx=0x7fef7c0197f0 comp rx=0 tx=0).stop 2024-09-16T03:58:54.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:54.121+0000 7fef9b134640 1 -- 172.21.15.42:0/2417829217 shutdown_connections 2024-09-16T03:58:54.128 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:54.121+0000 7fef9b134640 1 --2- 172.21.15.42:0/2417829217 >> v2:172.21.15.154:6812/254845342 conn(0x7fef74037810 0x7fef74039cd0 unknown :-1 s=CLOSED pgs=3709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:54.128 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:54.121+0000 7fef9b134640 1 --2- 172.21.15.42:0/2417829217 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fef94123110 0x7fef94123590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:54.128 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:54.121+0000 7fef9b134640 1 --2- 172.21.15.42:0/2417829217 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fef941cdc30 0x7fef94122bd0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:54.128 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:54.121+0000 7fef9b134640 1 --2- 172.21.15.42:0/2417829217 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fef80004810 0x7fef941cd6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:54.128 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:54.121+0000 7fef9b134640 1 -- 172.21.15.42:0/2417829217 >> 172.21.15.42:0/2417829217 conn(0x7fef940a2420 msgr2=0x7fef9406eca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:58:54.128 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:54.121+0000 7fef9b134640 1 -- 172.21.15.42:0/2417829217 shutdown_connections 2024-09-16T03:58:54.128 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:54.121+0000 7fef9b134640 1 -- 172.21.15.42:0/2417829217 wait complete. 2024-09-16T03:58:54.128 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:54.121+0000 7fef9b134640 1 librados: shutdown 2024-09-16T03:58:57.145 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:58:57.279 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.273+0000 7f9760a24640 10 monclient: get_monmap_and_config 2024-09-16T03:58:57.279 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.273+0000 7f9760a24640 10 monclient: build_initial_monmap 2024-09-16T03:58:57.279 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.273+0000 7f9760a24640 10 monclient: monmap: 2024-09-16T03:58:57.279 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:58:57.279 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:58:57.279 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:58:57.277632+0000 2024-09-16T03:58:57.279 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:58:57.277632+0000 2024-09-16T03:58:57.279 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:58:57.279 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:57.279 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:58:57.279 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:58:57.280 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:58:57.280 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:58:57.280 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:58:57.280 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:57.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.273+0000 7f9760a24640 1 Processor -- start 2024-09-16T03:58:57.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.273+0000 7f9760a24640 1 -- start start 2024-09-16T03:58:57.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.273+0000 7f9760a24640 10 monclient: init 2024-09-16T03:58:57.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.273+0000 7f9760a24640 10 monclient: _reopen_session rank -1 2024-09-16T03:58:57.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.273+0000 7f9760a24640 10 monclient: _add_conns ranks=[3,2,4,0,1] 2024-09-16T03:58:57.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.273+0000 7f9760a24640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f975c06db90 0x7f975c1340c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:57.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.273+0000 7f9760a24640 10 monclient(hunting): picked mon.noname-a con 0x7f975c06db90 addr v2:172.21.15.42:3300/0 2024-09-16T03:58:57.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.273+0000 7f9760a24640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f975c0a37f0 0x7f975c0a3c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:57.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.273+0000 7f9760a24640 10 monclient(hunting): picked mon.noname-b con 0x7f975c0a37f0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:58:57.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.273+0000 7f9760a24640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f975c136e10 0x7f975c137240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:57.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.273+0000 7f9760a24640 10 monclient(hunting): picked mon.noname-c con 0x7f975c136e10 addr v2:172.21.15.154:3300/0 2024-09-16T03:58:57.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.273+0000 7f9760a24640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:57.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.273+0000 7f9760a24640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f975c06df70 con 0x7f975c06db90 2024-09-16T03:58:57.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.273+0000 7f9760a24640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:57.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.273+0000 7f9760a24640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f975c06ece0 con 0x7f975c136e10 2024-09-16T03:58:57.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.273+0000 7f9760a24640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:57.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.273+0000 7f9760a24640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f975c09bf80 con 0x7f975c0a37f0 2024-09-16T03:58:57.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.273+0000 7f9760a24640 10 monclient(hunting): _renew_subs 2024-09-16T03:58:57.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.273+0000 7f9760a24640 10 monclient(hunting): authenticate will time out at 1984.247895s 2024-09-16T03:58:57.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.277+0000 7f9751d9b640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f975c0a37f0 0x7f975c0a3c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:57.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.277+0000 7f975a59c640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f975c06db90 0x7f975c1340c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:57.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.277+0000 7f975a59c640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f975c06db90 0x7f975c1340c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:33438/0 (socket says 172.21.15.42:33438) 2024-09-16T03:58:57.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.277+0000 7f975a59c640 1 -- 172.21.15.42:0/2805027643 learned_addr learned my addr 172.21.15.42:0/2805027643 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:58:57.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.277+0000 7f9751d9b640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f975c0a37f0 0x7f975c0a3c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:57348/0 (socket says 172.21.15.42:57348) 2024-09-16T03:58:57.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.277+0000 7f975a59c640 10 monclient(hunting): get_auth_request con 0x7f975c06db90 auth_method 0 2024-09-16T03:58:57.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.277+0000 7f975a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:57.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.277+0000 7f975a59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:57.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.277+0000 7f975a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:58:57.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.277+0000 7f9751d9b640 10 monclient(hunting): get_auth_request con 0x7f975c0a37f0 auth_method 0 2024-09-16T03:58:57.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.277+0000 7f9751d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:57.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.277+0000 7f9751d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:57.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.277+0000 7f9751d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:58:57.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.277+0000 7f975a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:58:57.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.277+0000 7f975a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:58:57.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.277+0000 7f975a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:58:57.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.277+0000 7f9751d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:58:57.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.277+0000 7f9751d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:58:57.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.277+0000 7f975ad9d640 1 --2- 172.21.15.42:0/2805027643 >> v2:172.21.15.154:3300/0 conn(0x7f975c136e10 0x7f975c137240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:57.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.277+0000 7f9751d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:58:57.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.277+0000 7f975ad9d640 10 monclient(hunting): get_auth_request con 0x7f975c136e10 auth_method 0 2024-09-16T03:58:57.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.277+0000 7f975ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:57.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.277+0000 7f975ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:57.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.277+0000 7f975ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:58:57.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.277+0000 7f9751d9b640 10 monclient(hunting): handle_auth_done global_id 924101 payload 274 2024-09-16T03:58:57.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.277+0000 7f9751d9b640 1 -- 172.21.15.42:0/2805027643 >> v2:172.21.15.154:3300/0 conn(0x7f975c136e10 msgr2=0x7f975c137240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:58:57.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.277+0000 7f9751d9b640 1 --2- 172.21.15.42:0/2805027643 >> v2:172.21.15.154:3300/0 conn(0x7f975c136e10 0x7f975c137240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:57.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.277+0000 7f9751d9b640 1 -- 172.21.15.42:0/2805027643 >> v2:172.21.15.42:3300/0 conn(0x7f975c06db90 msgr2=0x7f975c1340c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:58:57.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.277+0000 7f9751d9b640 1 --2- 172.21.15.42:0/2805027643 >> v2:172.21.15.42:3300/0 conn(0x7f975c06db90 0x7f975c1340c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:57.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.277+0000 7f9751d9b640 10 monclient: _finish_hunting 0 2024-09-16T03:58:57.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.277+0000 7f9751d9b640 1 monclient: found mon.noname-b 2024-09-16T03:58:57.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.277+0000 7f9751d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:58:57.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.277+0000 7f9751d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T03:58:57.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.277+0000 7f9751d9b640 1 -- 172.21.15.42:0/2805027643 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f975c134780 con 0x7f975c0a37f0 2024-09-16T03:58:57.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.277+0000 7f975a59c640 1 --2- 172.21.15.42:0/2805027643 >> v2:172.21.15.42:3300/0 conn(0x7f975c06db90 0x7f975c1340c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:58:57.293 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.277+0000 7f9751d9b640 1 --2- 172.21.15.42:0/2805027643 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f975c0a37f0 0x7f975c0a3c00 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f975c131410 tx=0x7f974401c340 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c24b5435b5e10fc0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:58:57.293 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.277+0000 7f9759d9b640 1 -- 172.21.15.42:0/2805027643 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97440b0070 con 0x7f975c0a37f0 2024-09-16T03:58:57.293 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.277+0000 7f9759d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:58:57.293 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.277+0000 7f9759d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:58:57.293 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.277+0000 7f9759d9b640 10 monclient: dump: 2024-09-16T03:58:57.293 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:58:57.294 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:58:57.294 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:57.294 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:57.294 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:58:57.294 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:57.294 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:58:57.294 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:58:57.294 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:58:57.294 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:57.295 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.277+0000 7f9759d9b640 10 monclient: _finish_auth 0 2024-09-16T03:58:57.295 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.277+0000 7f9759d9b640 10 monclient: _check_auth_tickets 2024-09-16T03:58:57.295 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.277+0000 7f9759d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:58:57.295 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.277+0000 7f9760a24640 5 monclient: authenticate success, global_id 924101 2024-09-16T03:58:57.295 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.277+0000 7f9760a24640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:58:57.295 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.277+0000 7f9759d9b640 1 -- 172.21.15.42:0/2805027643 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f97440ab600 con 0x7f975c0a37f0 2024-09-16T03:58:57.295 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.277+0000 7f9759d9b640 10 monclient: handle_config config(0 keys) 2024-09-16T03:58:57.295 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.277+0000 7f9760a24640 10 monclient: get_monmap_and_config success 2024-09-16T03:58:57.295 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.277+0000 7f9760a24640 4 set_mon_vals no callback set 2024-09-16T03:58:57.295 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.277+0000 7f9759d9b640 1 -- 172.21.15.42:0/2805027643 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97440ab9d0 con 0x7f975c0a37f0 2024-09-16T03:58:57.295 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.277+0000 7f9759d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:58:57.295 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.277+0000 7f9759d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:58:57.295 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.277+0000 7f9759d9b640 10 monclient: dump: 2024-09-16T03:58:57.295 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:58:57.295 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:58:57.296 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:57.296 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:57.296 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:58:57.296 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:57.296 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:58:57.296 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:58:57.296 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:58:57.296 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:57.296 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.277+0000 7f9760a24640 10 monclient: shutdown 2024-09-16T03:58:57.296 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.277+0000 7f9760a24640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:58:57.296 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.277+0000 7f9760a24640 1 -- 172.21.15.42:0/2805027643 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f975c0a37f0 msgr2=0x7f975c0a3c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:57.296 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.277+0000 7f9760a24640 1 --2- 172.21.15.42:0/2805027643 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f975c0a37f0 0x7f975c0a3c00 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f975c131410 tx=0x7f974401c340 comp rx=0 tx=0).stop 2024-09-16T03:58:57.296 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.277+0000 7f9760a24640 1 -- 172.21.15.42:0/2805027643 shutdown_connections 2024-09-16T03:58:57.296 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.277+0000 7f9760a24640 1 --2- 172.21.15.42:0/2805027643 >> v2:172.21.15.154:3300/0 conn(0x7f975c136e10 0x7f975c137240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:57.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.277+0000 7f9760a24640 1 --2- 172.21.15.42:0/2805027643 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f975c0a37f0 0x7f975c0a3c00 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:57.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.277+0000 7f9760a24640 1 --2- 172.21.15.42:0/2805027643 >> v2:172.21.15.42:3300/0 conn(0x7f975c06db90 0x7f975c1340c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:57.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.277+0000 7f9760a24640 1 -- 172.21.15.42:0/2805027643 >> 172.21.15.42:0/2805027643 conn(0x7f975c0a9fb0 msgr2=0x7f975c0aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:58:57.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.277+0000 7f9760a24640 1 -- 172.21.15.42:0/2805027643 shutdown_connections 2024-09-16T03:58:57.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.277+0000 7f9760a24640 1 -- 172.21.15.42:0/2805027643 wait complete. 2024-09-16T03:58:57.300 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.281+0000 7f9760a24640 10 monclient: build_initial_monmap 2024-09-16T03:58:57.301 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.281+0000 7f9760a24640 10 monclient: monmap: 2024-09-16T03:58:57.301 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:58:57.301 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:58:57.301 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:58:57.301 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:58:57.301 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:58:57.301 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:57.301 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:58:57.301 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:58:57.302 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:58:57.302 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:57.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.281+0000 7f9760a24640 1 librados: starting msgr at 2024-09-16T03:58:57.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.281+0000 7f9760a24640 1 librados: starting objecter 2024-09-16T03:58:57.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.281+0000 7f9760a24640 1 Processor -- start 2024-09-16T03:58:57.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.281+0000 7f9760a24640 1 -- start start 2024-09-16T03:58:57.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.281+0000 7f9760a24640 1 librados: setting wanted keys 2024-09-16T03:58:57.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.281+0000 7f9760a24640 1 librados: calling monclient init 2024-09-16T03:58:57.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.281+0000 7f9760a24640 10 monclient: init 2024-09-16T03:58:57.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.281+0000 7f9760a24640 10 monclient: _reopen_session rank -1 2024-09-16T03:58:57.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.281+0000 7f9760a24640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:58:57.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.281+0000 7f9760a24640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f975c06db90 0x7f975c0a3210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:57.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.281+0000 7f9760a24640 10 monclient(hunting): picked mon.noname-a con 0x7f975c06db90 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:58:57.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.281+0000 7f9760a24640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f975c0a37f0 0x7f975c09c710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:57.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.281+0000 7f9760a24640 10 monclient(hunting): picked mon.noname-c con 0x7f975c0a37f0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:58:57.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.281+0000 7f9760a24640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f975c136e10 0x7f975c09cc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:57.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.281+0000 7f9760a24640 10 monclient(hunting): picked mon.noname-b con 0x7f975c136e10 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:58:57.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.281+0000 7f9760a24640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:57.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.281+0000 7f9760a24640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f975c140bd0 con 0x7f975c06db90 2024-09-16T03:58:57.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.281+0000 7f9760a24640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:57.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.281+0000 7f9760a24640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f975c137c80 con 0x7f975c136e10 2024-09-16T03:58:57.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.281+0000 7f9760a24640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:57.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.281+0000 7f9760a24640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f975c140d50 con 0x7f975c0a37f0 2024-09-16T03:58:57.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.281+0000 7f9760a24640 10 monclient(hunting): _renew_subs 2024-09-16T03:58:57.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.281+0000 7f9760a24640 10 monclient(hunting): authenticate will time out at 1984.253425s 2024-09-16T03:58:57.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.281+0000 7f975a59c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f975c06db90 0x7f975c0a3210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:57.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.281+0000 7f975a59c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f975c06db90 0x7f975c0a3210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:33440/0 (socket says 172.21.15.42:33440) 2024-09-16T03:58:57.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.281+0000 7f9751d9b640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f975c0a37f0 0x7f975c09c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:57.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.281+0000 7f975a59c640 1 -- 172.21.15.42:0/3625325372 learned_addr learned my addr 172.21.15.42:0/3625325372 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:58:57.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.281+0000 7f975a59c640 10 monclient(hunting): get_auth_request con 0x7f975c06db90 auth_method 0 2024-09-16T03:58:57.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.281+0000 7f975a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:57.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.281+0000 7f975a59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:57.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.281+0000 7f975a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:58:57.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.281+0000 7f9751d9b640 10 monclient(hunting): get_auth_request con 0x7f975c0a37f0 auth_method 0 2024-09-16T03:58:57.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.281+0000 7f9751d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:57.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.281+0000 7f9751d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:57.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.281+0000 7f9751d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:58:57.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.281+0000 7f9751d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:58:57.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.281+0000 7f9751d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:58:57.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.281+0000 7f9751d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:58:57.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.281+0000 7f975a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:58:57.306 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.281+0000 7f975a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:58:57.306 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.281+0000 7f975a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:58:57.306 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.281+0000 7f975ad9d640 1 --2- 172.21.15.42:0/3625325372 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f975c136e10 0x7f975c09cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:57.306 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.281+0000 7f975ad9d640 10 monclient(hunting): get_auth_request con 0x7f975c136e10 auth_method 0 2024-09-16T03:58:57.306 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.281+0000 7f975ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:57.306 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.281+0000 7f975ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:57.306 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.281+0000 7f975ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:58:57.306 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.281+0000 7f9751d9b640 10 monclient(hunting): handle_auth_done global_id 924104 payload 824 2024-09-16T03:58:57.306 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.281+0000 7f9751d9b640 1 -- 172.21.15.42:0/3625325372 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f975c136e10 msgr2=0x7f975c09cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:57.306 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.281+0000 7f9751d9b640 1 --2- 172.21.15.42:0/3625325372 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f975c136e10 0x7f975c09cc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:57.306 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.281+0000 7f9751d9b640 1 -- 172.21.15.42:0/3625325372 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f975c06db90 msgr2=0x7f975c0a3210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:57.306 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.281+0000 7f9751d9b640 1 --2- 172.21.15.42:0/3625325372 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f975c06db90 0x7f975c0a3210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:57.307 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.281+0000 7f9751d9b640 10 monclient: _finish_hunting 0 2024-09-16T03:58:57.307 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.281+0000 7f9751d9b640 1 monclient: found mon.noname-c 2024-09-16T03:58:57.307 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.281+0000 7f9751d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:58:57.307 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.281+0000 7f9751d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:58:57.307 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.281+0000 7f9751d9b640 1 -- 172.21.15.42:0/3625325372 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f975c09fa70 con 0x7f975c0a37f0 2024-09-16T03:58:57.307 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.281+0000 7f975a59c640 1 --2- 172.21.15.42:0/3625325372 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f975c06db90 0x7f975c0a3210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:58:57.307 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.281+0000 7f9751d9b640 1 --2- 172.21.15.42:0/3625325372 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f975c0a37f0 0x7f975c09c710 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f97440ab000 tx=0x7f97440190e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:58:57.307 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.281+0000 7f97537fe640 10 client.?.objecter ms_handle_connect 0x7f975c0a37f0 2024-09-16T03:58:57.307 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.281+0000 7f97537fe640 10 client.?.objecter resend_mon_ops 2024-09-16T03:58:57.307 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.281+0000 7f97537fe640 1 -- 172.21.15.42:0/3625325372 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97440b8020 con 0x7f975c0a37f0 2024-09-16T03:58:57.307 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.281+0000 7f97537fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:58:57.307 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.281+0000 7f97537fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:58:57.308 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.281+0000 7f97537fe640 10 monclient: dump: 2024-09-16T03:58:57.308 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:58:57.308 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:58:57.308 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:57.308 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:57.308 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:58:57.308 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:57.308 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:58:57.308 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:58:57.308 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:58:57.308 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:57.309 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.281+0000 7f97537fe640 10 monclient: _finish_auth 0 2024-09-16T03:58:57.309 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.281+0000 7f97537fe640 10 monclient: _check_auth_tickets 2024-09-16T03:58:57.309 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.281+0000 7f97537fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:58:57.309 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.281+0000 7f97537fe640 1 -- 172.21.15.42:0/3625325372 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f97440b4410 con 0x7f975c0a37f0 2024-09-16T03:58:57.309 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.281+0000 7f97537fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:58:57.309 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.281+0000 7f97537fe640 1 -- 172.21.15.42:0/3625325372 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97440b0040 con 0x7f975c0a37f0 2024-09-16T03:58:57.309 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.281+0000 7f97537fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:58:57.309 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.281+0000 7f97537fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:58:57.309 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.281+0000 7f97537fe640 10 monclient: dump: 2024-09-16T03:58:57.309 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:58:57.309 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:58:57.309 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:57.309 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:57.310 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:58:57.310 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:57.310 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:58:57.310 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:58:57.310 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:58:57.310 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:57.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.281+0000 7f975959a640 4 set_mon_vals no callback set 2024-09-16T03:58:57.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.281+0000 7f9760a24640 5 monclient: authenticate success, global_id 924104 2024-09-16T03:58:57.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.281+0000 7f9760a24640 10 monclient: _renew_subs 2024-09-16T03:58:57.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.281+0000 7f9760a24640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:58:57.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.281+0000 7f9760a24640 1 -- 172.21.15.42:0/3625325372 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f975c0a8b90 con 0x7f975c0a37f0 2024-09-16T03:58:57.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.281+0000 7f9760a24640 10 client.924104.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:58:57.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.281+0000 7f9760a24640 10 monclient: _renew_subs 2024-09-16T03:58:57.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.281+0000 7f9760a24640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:58:57.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.281+0000 7f9760a24640 1 -- 172.21.15.42:0/3625325372 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f975c09f790 con 0x7f975c0a37f0 2024-09-16T03:58:57.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.281+0000 7f9760a24640 1 librados: init done 2024-09-16T03:58:57.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.281+0000 7f97537fe640 1 -- 172.21.15.42:0/3625325372 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f97440bb020 con 0x7f975c0a37f0 2024-09-16T03:58:57.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.281+0000 7f9760a24640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:58:57.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.281+0000 7f9760a24640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:58:57.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.281+0000 7f9760a24640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:58:57.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.281+0000 7f9760a24640 1 -- 172.21.15.42:0/3625325372 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f975c0a3c00 con 0x7f975c0a37f0 2024-09-16T03:58:57.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.285+0000 7f97537fe640 1 --2- 172.21.15.42:0/3625325372 >> v2:172.21.15.154:6812/254845342 conn(0x7f973c037810 0x7f973c039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:57.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.285+0000 7f97537fe640 1 -- 172.21.15.42:0/3625325372 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(513..513 src has 366..513) ==== 5372+0+0 (secure 0 0 0) 0x7f97440e29a0 con 0x7f975c0a37f0 2024-09-16T03:58:57.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.285+0000 7f97537fe640 10 client.924104.objecter ms_dispatch 0x7f975c000ba0 osd_map(513..513 src has 366..513) 2024-09-16T03:58:57.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.285+0000 7f97537fe640 3 client.924104.objecter handle_osd_map got epochs [513,513] > 0 2024-09-16T03:58:57.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.285+0000 7f97537fe640 3 client.924104.objecter handle_osd_map decoding full epoch 513 2024-09-16T03:58:57.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.285+0000 7f97537fe640 20 client.924104.objecter dump_active .. 0 homeless 2024-09-16T03:58:57.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.285+0000 7f975a59c640 1 --2- 172.21.15.42:0/3625325372 >> v2:172.21.15.154:6812/254845342 conn(0x7f973c037810 0x7f973c039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:57.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.285+0000 7f975a59c640 10 monclient: get_auth_request con 0x7f973c037810 auth_method 0 2024-09-16T03:58:57.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.285+0000 7f97537fe640 10 client.924104.objecter ms_handle_connect 0x7f973c037810 2024-09-16T03:58:57.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.285+0000 7f975a59c640 1 --2- 172.21.15.42:0/3625325372 >> v2:172.21.15.154:6812/254845342 conn(0x7f973c037810 0x7f973c039cd0 secure :-1 s=READY pgs=3711 cs=0 l=1 rev1=1 crypto rx=0x7f974c009630 tx=0x7f974c009240 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:58:57.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.289+0000 7f97537fe640 1 -- 172.21.15.42:0/3625325372 <== mon.2 v2:172.21.15.42:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f97440c2030 con 0x7f975c0a37f0 2024-09-16T03:58:57.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.289+0000 7f97537fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:58:57.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.289+0000 7f97537fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:58:57.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.517+0000 7f9760a24640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:58:57.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.517+0000 7f9760a24640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:58:57.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.517+0000 7f9760a24640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:58:57.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.517+0000 7f9760a24640 1 -- 172.21.15.42:0/3625325372 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f975c06f680 con 0x7f975c0a37f0 2024-09-16T03:58:57.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.517+0000 7f97537fe640 1 -- 172.21.15.42:0/3625325372 <== mon.2 v2:172.21.15.42:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7f9744016050 con 0x7f975c0a37f0 2024-09-16T03:58:57.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.517+0000 7f97537fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:58:57.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.517+0000 7f97537fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:58:57.525 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":154,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:58:57.527 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.521+0000 7f9760a24640 10 librados: watch_flush enter 2024-09-16T03:58:57.527 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.521+0000 7f9760a24640 10 librados: watch_flush exit 2024-09-16T03:58:57.527 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.521+0000 7f9760a24640 20 client.924104.objecter shutdown clearing up homeless session... 2024-09-16T03:58:57.527 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.521+0000 7f9760a24640 10 client.924104.objecter successfully canceled tick 2024-09-16T03:58:57.527 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.521+0000 7f9760a24640 1 -- 172.21.15.42:0/3625325372 >> v2:172.21.15.154:6812/254845342 conn(0x7f973c037810 msgr2=0x7f973c039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:57.527 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.521+0000 7f9760a24640 1 --2- 172.21.15.42:0/3625325372 >> v2:172.21.15.154:6812/254845342 conn(0x7f973c037810 0x7f973c039cd0 secure :-1 s=READY pgs=3711 cs=0 l=1 rev1=1 crypto rx=0x7f974c009630 tx=0x7f974c009240 comp rx=0 tx=0).stop 2024-09-16T03:58:57.527 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.521+0000 7f9760a24640 10 monclient: shutdown 2024-09-16T03:58:57.527 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.521+0000 7f9760a24640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:58:57.527 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.521+0000 7f9760a24640 1 -- 172.21.15.42:0/3625325372 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f975c0a37f0 msgr2=0x7f975c09c710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:57.527 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.521+0000 7f9760a24640 1 --2- 172.21.15.42:0/3625325372 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f975c0a37f0 0x7f975c09c710 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f97440ab000 tx=0x7f97440190e0 comp rx=0 tx=0).stop 2024-09-16T03:58:57.527 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.521+0000 7f9760a24640 1 -- 172.21.15.42:0/3625325372 shutdown_connections 2024-09-16T03:58:57.528 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.521+0000 7f9760a24640 1 --2- 172.21.15.42:0/3625325372 >> v2:172.21.15.154:6812/254845342 conn(0x7f973c037810 0x7f973c039cd0 unknown :-1 s=CLOSED pgs=3711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:57.528 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.521+0000 7f9760a24640 1 --2- 172.21.15.42:0/3625325372 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f975c136e10 0x7f975c09cc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:57.528 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.521+0000 7f9760a24640 1 --2- 172.21.15.42:0/3625325372 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f975c0a37f0 0x7f975c09c710 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:57.528 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.521+0000 7f9760a24640 1 --2- 172.21.15.42:0/3625325372 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f975c06db90 0x7f975c0a3210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:57.528 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.521+0000 7f9760a24640 1 -- 172.21.15.42:0/3625325372 >> 172.21.15.42:0/3625325372 conn(0x7f975c0a9fb0 msgr2=0x7f975c0a8210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:58:57.528 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.521+0000 7f9760a24640 1 -- 172.21.15.42:0/3625325372 shutdown_connections 2024-09-16T03:58:57.528 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.521+0000 7f9760a24640 1 -- 172.21.15.42:0/3625325372 wait complete. 2024-09-16T03:58:57.528 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.521+0000 7f9760a24640 1 librados: shutdown 2024-09-16T03:58:57.543 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T03:58:57.543 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T03:58:57.544 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T03:58:57.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.657+0000 7efdb9dc5640 10 monclient: get_monmap_and_config 2024-09-16T03:58:57.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.657+0000 7efdb9dc5640 10 monclient: build_initial_monmap 2024-09-16T03:58:57.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.657+0000 7efdb9dc5640 10 monclient: monmap: 2024-09-16T03:58:57.664 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:58:57.665 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:58:57.665 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:58:57.662915+0000 2024-09-16T03:58:57.665 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:58:57.662915+0000 2024-09-16T03:58:57.665 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:58:57.665 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:57.665 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:58:57.665 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:58:57.665 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:58:57.665 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:58:57.665 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:58:57.665 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:57.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.657+0000 7efdb9dc5640 1 Processor -- start 2024-09-16T03:58:57.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.657+0000 7efdb9dc5640 1 -- start start 2024-09-16T03:58:57.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.657+0000 7efdb9dc5640 10 monclient: init 2024-09-16T03:58:57.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.661+0000 7efdb9dc5640 10 monclient: _reopen_session rank -1 2024-09-16T03:58:57.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.661+0000 7efdb9dc5640 10 monclient: _add_conns ranks=[3,1,2,0,4] 2024-09-16T03:58:57.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.661+0000 7efdb9dc5640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7efdb406db90 0x7efdb41340c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:57.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.661+0000 7efdb9dc5640 10 monclient(hunting): picked mon.noname-a con 0x7efdb406db90 addr v2:172.21.15.42:3300/0 2024-09-16T03:58:57.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.661+0000 7efdb9dc5640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7efdb40a37f0 addr v1:172.21.15.154:6789/0 2024-09-16T03:58:57.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.661+0000 7efdb9dc5640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7efdb40a3c00 0x7efdb4137520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:57.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.661+0000 7efdb9dc5640 10 monclient(hunting): picked mon.noname-b con 0x7efdb40a3c00 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:58:57.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.661+0000 7efdb9dc5640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efdb406ece0 con 0x7efdb40a37f0 2024-09-16T03:58:57.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.661+0000 7efdb9dc5640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:57.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.661+0000 7efdb9dc5640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7efdb406df70 con 0x7efdb406db90 2024-09-16T03:58:57.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.661+0000 7efdb9dc5640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:57.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.661+0000 7efdb9dc5640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7efdb409bf80 con 0x7efdb40a3c00 2024-09-16T03:58:57.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.661+0000 7efdb9dc5640 10 monclient(hunting): _renew_subs 2024-09-16T03:58:57.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.661+0000 7efdb9dc5640 10 monclient(hunting): authenticate will time out at 1984.632382s 2024-09-16T03:58:57.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.661+0000 7efdb3fff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7efdb40a3c00 0x7efdb4137520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:57.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.661+0000 7efdb3fff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7efdb40a3c00 0x7efdb4137520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:57370/0 (socket says 172.21.15.42:57370) 2024-09-16T03:58:57.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.661+0000 7efdb37fe640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7efdb406db90 0x7efdb41340c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:57.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.661+0000 7efdb3fff640 1 -- 172.21.15.42:0/1768607027 learned_addr learned my addr 172.21.15.42:0/1768607027 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:58:57.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.661+0000 7efdb3fff640 10 monclient(hunting): get_auth_request con 0x7efdb40a3c00 auth_method 0 2024-09-16T03:58:57.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.661+0000 7efdb3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:57.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.661+0000 7efdb3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:57.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.661+0000 7efdb3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:58:57.670 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.661+0000 7efdb37fe640 10 monclient(hunting): get_auth_request con 0x7efdb406db90 auth_method 0 2024-09-16T03:58:57.670 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.661+0000 7efdb37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:57.670 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.661+0000 7efdb37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:57.670 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.661+0000 7efdb37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:58:57.670 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.661+0000 7efdb3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:58:57.670 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.661+0000 7efdb3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:58:57.670 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.661+0000 7efdb3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:58:57.670 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.661+0000 7efdb37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:58:57.670 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.661+0000 7efdb37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:58:57.670 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.661+0000 7efdb37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:58:57.670 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.661+0000 7efdb3fff640 10 monclient(hunting): handle_auth_done global_id 924107 payload 274 2024-09-16T03:58:57.670 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.661+0000 7efdb3fff640 1 -- 172.21.15.42:0/1768607027 >> v2:172.21.15.42:3300/0 conn(0x7efdb406db90 msgr2=0x7efdb41340c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:58:57.670 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.661+0000 7efdb3fff640 1 --2- 172.21.15.42:0/1768607027 >> v2:172.21.15.42:3300/0 conn(0x7efdb406db90 0x7efdb41340c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:57.670 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.661+0000 7efdb3fff640 1 -- 172.21.15.42:0/1768607027 >> v1:172.21.15.154:6789/0 conn(0x7efdb40a37f0 legacy=0x7efdb4136e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:58:57.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.661+0000 7efdb3fff640 10 monclient: _finish_hunting 0 2024-09-16T03:58:57.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.661+0000 7efdb3fff640 1 monclient: found mon.noname-b 2024-09-16T03:58:57.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.661+0000 7efdb3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:58:57.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.661+0000 7efdb3fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T03:58:57.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.661+0000 7efdb3fff640 1 -- 172.21.15.42:0/1768607027 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efdb4134780 con 0x7efdb40a3c00 2024-09-16T03:58:57.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.661+0000 7efdb37fe640 1 --2- 172.21.15.42:0/1768607027 >> v2:172.21.15.42:3300/0 conn(0x7efdb406db90 0x7efdb41340c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:58:57.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.661+0000 7efdb3fff640 1 --2- 172.21.15.42:0/1768607027 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7efdb40a3c00 0x7efdb4137520 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7efdac009e50 tx=0x7efdac01d310 comp rx=0 tx=0).ready entity=mon.2 client_cookie=994313c6a084f547 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:58:57.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.661+0000 7efdb27fc640 1 -- 172.21.15.42:0/1768607027 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdac0a5640 con 0x7efdb40a3c00 2024-09-16T03:58:57.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.661+0000 7efdb27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:58:57.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.661+0000 7efdb27fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:58:57.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.661+0000 7efdb27fc640 10 monclient: dump: 2024-09-16T03:58:57.678 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:58:57.678 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:58:57.678 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:57.678 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:57.678 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:58:57.678 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:57.678 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:58:57.678 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:58:57.678 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:58:57.678 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:57.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.661+0000 7efdb27fc640 10 monclient: _finish_auth 0 2024-09-16T03:58:57.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.661+0000 7efdb27fc640 10 monclient: _check_auth_tickets 2024-09-16T03:58:57.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.661+0000 7efdb27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:58:57.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.661+0000 7efdb9dc5640 5 monclient: authenticate success, global_id 924107 2024-09-16T03:58:57.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.661+0000 7efdb27fc640 1 -- 172.21.15.42:0/1768607027 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efdac0a57e0 con 0x7efdb40a3c00 2024-09-16T03:58:57.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.661+0000 7efdb9dc5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:58:57.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.661+0000 7efdb27fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:58:57.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.661+0000 7efdb9dc5640 10 monclient: get_monmap_and_config success 2024-09-16T03:58:57.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.661+0000 7efdb9dc5640 4 set_mon_vals no callback set 2024-09-16T03:58:57.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.661+0000 7efdb27fc640 1 -- 172.21.15.42:0/1768607027 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdac0a5bb0 con 0x7efdb40a3c00 2024-09-16T03:58:57.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.661+0000 7efdb27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:58:57.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.661+0000 7efdb27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:58:57.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.661+0000 7efdb27fc640 10 monclient: dump: 2024-09-16T03:58:57.679 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:58:57.680 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:58:57.680 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:57.680 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:57.680 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:58:57.680 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:57.680 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:58:57.680 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:58:57.680 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:58:57.680 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:57.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.661+0000 7efdb9dc5640 10 monclient: shutdown 2024-09-16T03:58:57.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.661+0000 7efdb9dc5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:58:57.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.661+0000 7efdb9dc5640 1 -- 172.21.15.42:0/1768607027 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7efdb40a3c00 msgr2=0x7efdb4137520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:57.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.661+0000 7efdb9dc5640 1 --2- 172.21.15.42:0/1768607027 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7efdb40a3c00 0x7efdb4137520 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7efdac009e50 tx=0x7efdac01d310 comp rx=0 tx=0).stop 2024-09-16T03:58:57.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.661+0000 7efdb9dc5640 1 -- 172.21.15.42:0/1768607027 shutdown_connections 2024-09-16T03:58:57.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.661+0000 7efdb9dc5640 1 --2- 172.21.15.42:0/1768607027 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7efdb40a3c00 0x7efdb4137520 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:57.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.661+0000 7efdb9dc5640 1 --2- 172.21.15.42:0/1768607027 >> v2:172.21.15.42:3300/0 conn(0x7efdb406db90 0x7efdb41340c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:57.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.661+0000 7efdb9dc5640 1 -- 172.21.15.42:0/1768607027 >> 172.21.15.42:0/1768607027 conn(0x7efdb40a9fb0 msgr2=0x7efdb40aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:58:57.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.661+0000 7efdb9dc5640 1 -- 172.21.15.42:0/1768607027 shutdown_connections 2024-09-16T03:58:57.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.661+0000 7efdb9dc5640 1 -- 172.21.15.42:0/1768607027 wait complete. 2024-09-16T03:58:57.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.665+0000 7efdb9dc5640 10 monclient: build_initial_monmap 2024-09-16T03:58:57.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.665+0000 7efdb9dc5640 10 monclient: monmap: 2024-09-16T03:58:57.681 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:58:57.681 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:58:57.681 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:58:57.681 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:58:57.681 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:58:57.681 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:57.682 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:58:57.682 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:58:57.682 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:58:57.682 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:57.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.665+0000 7efdb9dc5640 1 librados: starting msgr at 2024-09-16T03:58:57.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.665+0000 7efdb9dc5640 1 librados: starting objecter 2024-09-16T03:58:57.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.665+0000 7efdb9dc5640 1 Processor -- start 2024-09-16T03:58:57.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.665+0000 7efdb9dc5640 1 -- start start 2024-09-16T03:58:57.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.665+0000 7efdb9dc5640 1 librados: setting wanted keys 2024-09-16T03:58:57.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.665+0000 7efdb9dc5640 1 librados: calling monclient init 2024-09-16T03:58:57.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.665+0000 7efdb9dc5640 10 monclient: init 2024-09-16T03:58:57.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.665+0000 7efdb9dc5640 10 monclient: _reopen_session rank -1 2024-09-16T03:58:57.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.665+0000 7efdb9dc5640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T03:58:57.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.665+0000 7efdb9dc5640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efdb406db90 0x7efdb41cf0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:57.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.665+0000 7efdb9dc5640 10 monclient(hunting): picked mon.noname-a con 0x7efdb406db90 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:58:57.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.665+0000 7efdb9dc5640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efdb40a37f0 0x7efdb413e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:57.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.665+0000 7efdb9dc5640 10 monclient(hunting): picked mon.noname-b con 0x7efdb40a37f0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:58:57.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.665+0000 7efdb9dc5640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7efdb40a3c00 0x7efdb40a1960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:57.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.665+0000 7efdb9dc5640 10 monclient(hunting): picked mon.noname-c con 0x7efdb40a3c00 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:58:57.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.665+0000 7efdb9dc5640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:57.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.665+0000 7efdb9dc5640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7efdb41419d0 con 0x7efdb406db90 2024-09-16T03:58:57.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.665+0000 7efdb9dc5640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:57.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.665+0000 7efdb9dc5640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7efdb4137c70 con 0x7efdb40a37f0 2024-09-16T03:58:57.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.665+0000 7efdb9dc5640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:57.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.665+0000 7efdb9dc5640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7efdb4141b50 con 0x7efdb40a3c00 2024-09-16T03:58:57.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.665+0000 7efdb9dc5640 10 monclient(hunting): _renew_subs 2024-09-16T03:58:57.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.665+0000 7efdb9dc5640 10 monclient(hunting): authenticate will time out at 1984.637303s 2024-09-16T03:58:57.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.665+0000 7efdb37fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efdb406db90 0x7efdb41cf0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:57.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.665+0000 7efdb3fff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7efdb40a3c00 0x7efdb40a1960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:57.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.665+0000 7efdb37fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efdb406db90 0x7efdb41cf0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:33448/0 (socket says 172.21.15.42:33448) 2024-09-16T03:58:57.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.665+0000 7efdb37fe640 1 -- 172.21.15.42:0/2445242997 learned_addr learned my addr 172.21.15.42:0/2445242997 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:58:57.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.665+0000 7efdb37fe640 10 monclient(hunting): get_auth_request con 0x7efdb406db90 auth_method 0 2024-09-16T03:58:57.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.665+0000 7efdb37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:57.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.665+0000 7efdb37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:57.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.665+0000 7efdb37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:58:57.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.665+0000 7efdb3fff640 10 monclient(hunting): get_auth_request con 0x7efdb40a3c00 auth_method 0 2024-09-16T03:58:57.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.665+0000 7efdb3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:57.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.665+0000 7efdb3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:57.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.665+0000 7efdb3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:58:57.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.665+0000 7efdb37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:58:57.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.665+0000 7efdb37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:58:57.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.665+0000 7efdb37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:58:57.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.665+0000 7efdb3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:58:57.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.665+0000 7efdb3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:58:57.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.665+0000 7efdb3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:58:57.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.665+0000 7efdb3fff640 10 monclient(hunting): handle_auth_done global_id 924110 payload 824 2024-09-16T03:58:57.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.665+0000 7efdb3fff640 1 -- 172.21.15.42:0/2445242997 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efdb40a37f0 msgr2=0x7efdb413e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:57.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.665+0000 7efdb2ffd640 1 --2- 172.21.15.42:0/2445242997 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efdb40a37f0 0x7efdb413e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:57.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.665+0000 7efdb3fff640 1 --2- 172.21.15.42:0/2445242997 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efdb40a37f0 0x7efdb413e7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:57.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.665+0000 7efdb3fff640 1 -- 172.21.15.42:0/2445242997 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efdb406db90 msgr2=0x7efdb41cf0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:57.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.665+0000 7efdb3fff640 1 --2- 172.21.15.42:0/2445242997 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efdb406db90 0x7efdb41cf0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:57.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.665+0000 7efdb3fff640 10 monclient: _finish_hunting 0 2024-09-16T03:58:57.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.665+0000 7efdb3fff640 1 monclient: found mon.noname-c 2024-09-16T03:58:57.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.665+0000 7efdb3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:58:57.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.665+0000 7efdb3fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:58:57.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.665+0000 7efdb3fff640 1 -- 172.21.15.42:0/2445242997 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efdb406f6a0 con 0x7efdb40a3c00 2024-09-16T03:58:57.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.665+0000 7efdb2ffd640 10 monclient: get_auth_request con 0x7efdb40a37f0 auth_method 0 2024-09-16T03:58:57.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.665+0000 7efdb37fe640 1 --2- 172.21.15.42:0/2445242997 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efdb406db90 0x7efdb41cf0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:58:57.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.665+0000 7efdb2ffd640 1 --2- 172.21.15.42:0/2445242997 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efdb40a37f0 0x7efdb413e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T03:58:57.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.665+0000 7efdb3fff640 1 --2- 172.21.15.42:0/2445242997 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7efdb40a3c00 0x7efdb40a1960 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7efdac0ab000 tx=0x7efdac019650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:58:57.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.665+0000 7efdabfff640 10 client.?.objecter ms_handle_connect 0x7efdb40a3c00 2024-09-16T03:58:57.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.665+0000 7efdabfff640 10 client.?.objecter resend_mon_ops 2024-09-16T03:58:57.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.665+0000 7efdabfff640 1 -- 172.21.15.42:0/2445242997 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdac0bc030 con 0x7efdb40a3c00 2024-09-16T03:58:57.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.665+0000 7efdabfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:58:57.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.665+0000 7efdabfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:58:57.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.665+0000 7efdabfff640 10 monclient: dump: 2024-09-16T03:58:57.687 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:58:57.687 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:58:57.687 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:57.687 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:57.687 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:58:57.687 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:57.687 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:58:57.687 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:58:57.687 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:58:57.687 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:57.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.665+0000 7efdabfff640 10 monclient: _finish_auth 0 2024-09-16T03:58:57.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.665+0000 7efdabfff640 10 monclient: _check_auth_tickets 2024-09-16T03:58:57.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.665+0000 7efdabfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:58:57.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.665+0000 7efdb9dc5640 5 monclient: authenticate success, global_id 924110 2024-09-16T03:58:57.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.665+0000 7efdabfff640 1 -- 172.21.15.42:0/2445242997 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efdac0b8470 con 0x7efdb40a3c00 2024-09-16T03:58:57.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.665+0000 7efdb9dc5640 10 monclient: _renew_subs 2024-09-16T03:58:57.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.665+0000 7efdb9dc5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:58:57.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.665+0000 7efdb9dc5640 1 -- 172.21.15.42:0/2445242997 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efdb4133680 con 0x7efdb40a3c00 2024-09-16T03:58:57.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.665+0000 7efdabfff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:58:57.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.665+0000 7efdabfff640 1 -- 172.21.15.42:0/2445242997 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdac009510 con 0x7efdb40a3c00 2024-09-16T03:58:57.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.665+0000 7efdabfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:58:57.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.665+0000 7efdabfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:58:57.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.665+0000 7efdabfff640 10 monclient: dump: 2024-09-16T03:58:57.688 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:58:57.688 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:58:57.688 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:57.688 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:57.689 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:58:57.689 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:57.689 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:58:57.689 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:58:57.689 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:58:57.689 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:57.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.665+0000 7efdb1ffb640 4 set_mon_vals no callback set 2024-09-16T03:58:57.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.665+0000 7efdb9dc5640 10 client.924110.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:58:57.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.665+0000 7efdb9dc5640 10 monclient: _renew_subs 2024-09-16T03:58:57.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.665+0000 7efdb9dc5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:58:57.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.665+0000 7efdb9dc5640 1 -- 172.21.15.42:0/2445242997 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7efdb4138d70 con 0x7efdb40a3c00 2024-09-16T03:58:57.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.665+0000 7efdb9dc5640 1 librados: init done 2024-09-16T03:58:57.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.665+0000 7efdb9dc5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:58:57.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.665+0000 7efdb9dc5640 1 --2- 172.21.15.42:0/2445242997 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efdb413ed10 0x7efdb4139be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:57.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.665+0000 7efdb9dc5640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:57.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.665+0000 7efdb9dc5640 1 -- 172.21.15.42:0/2445242997 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7efdb4139800 con 0x7efdb413ed10 2024-09-16T03:58:57.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.665+0000 7efdabfff640 1 -- 172.21.15.42:0/2445242997 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7efdac0bf070 con 0x7efdb40a3c00 2024-09-16T03:58:57.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.665+0000 7efdb37fe640 1 --2- 172.21.15.42:0/2445242997 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efdb413ed10 0x7efdb4139be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:57.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.665+0000 7efdb37fe640 10 monclient: get_auth_request con 0x7efdb413ed10 auth_method 0 2024-09-16T03:58:57.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.665+0000 7efdb37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:57.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.665+0000 7efdb37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:57.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.665+0000 7efdb37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:58:57.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.665+0000 7efdb37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:58:57.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.665+0000 7efdb37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:58:57.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.665+0000 7efdb37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:58:57.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.665+0000 7efdabfff640 1 --2- 172.21.15.42:0/2445242997 >> v2:172.21.15.154:6812/254845342 conn(0x7efd94037810 0x7efd94039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:57.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.665+0000 7efdabfff640 1 -- 172.21.15.42:0/2445242997 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(514..514 src has 366..514) ==== 5040+0+0 (secure 0 0 0) 0x7efdac0e35e0 con 0x7efdb40a3c00 2024-09-16T03:58:57.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.665+0000 7efdabfff640 10 client.924110.objecter ms_dispatch 0x7efdb4000ba0 osd_map(514..514 src has 366..514) 2024-09-16T03:58:57.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.665+0000 7efdabfff640 3 client.924110.objecter handle_osd_map got epochs [514,514] > 0 2024-09-16T03:58:57.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.665+0000 7efdabfff640 3 client.924110.objecter handle_osd_map decoding full epoch 514 2024-09-16T03:58:57.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.665+0000 7efdb37fe640 10 monclient(hunting): handle_auth_done global_id 825045 payload 824 2024-09-16T03:58:57.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.665+0000 7efdb37fe640 1 -- 172.21.15.42:0/2445242997 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7efdb413a120 con 0x7efdb413ed10 2024-09-16T03:58:57.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.665+0000 7efdabfff640 20 client.924110.objecter dump_active .. 0 homeless 2024-09-16T03:58:57.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.665+0000 7efdb37fe640 1 --2- 172.21.15.42:0/2445242997 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efdb413ed10 0x7efdb4139be0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7efda4011db0 tx=0x7efda400f610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:58:57.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.665+0000 7efdabfff640 10 client.924110.objecter ms_handle_connect 0x7efdb413ed10 2024-09-16T03:58:57.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.665+0000 7efdabfff640 10 client.924110.objecter resend_mon_ops 2024-09-16T03:58:57.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.665+0000 7efdb2ffd640 1 --2- 172.21.15.42:0/2445242997 >> v2:172.21.15.154:6812/254845342 conn(0x7efd94037810 0x7efd94039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:57.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.665+0000 7efdabfff640 1 -- 172.21.15.42:0/2445242997 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efda4014440 con 0x7efdb413ed10 2024-09-16T03:58:57.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.665+0000 7efdabfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:58:57.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.665+0000 7efdabfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:58:57.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.665+0000 7efdabfff640 10 monclient: dump: 2024-09-16T03:58:57.692 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:58:57.692 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:58:57.692 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:57.692 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:57.692 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:58:57.692 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:57.692 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:58:57.692 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:58:57.692 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:58:57.692 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:57.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.665+0000 7efdb2ffd640 10 monclient: get_auth_request con 0x7efd94037810 auth_method 0 2024-09-16T03:58:57.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.669+0000 7efdabfff640 1 -- 172.21.15.42:0/2445242997 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7efda4014440 con 0x7efdb413ed10 2024-09-16T03:58:57.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.669+0000 7efdabfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:58:57.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.669+0000 7efdabfff640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:58:57.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.669+0000 7efdabfff640 1 -- 172.21.15.42:0/2445242997 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efdb413ed10 msgr2=0x7efdb4139be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:57.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.669+0000 7efdabfff640 1 --2- 172.21.15.42:0/2445242997 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efdb413ed10 0x7efdb4139be0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7efda4011db0 tx=0x7efda400f610 comp rx=0 tx=0).stop 2024-09-16T03:58:57.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.669+0000 7efdabfff640 1 -- 172.21.15.42:0/2445242997 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efdb413ed10 msgr2=0x7efdb4139be0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:58:57.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.669+0000 7efdabfff640 1 --2- 172.21.15.42:0/2445242997 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efdb413ed10 0x7efdb4139be0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:57.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.669+0000 7efdb2ffd640 1 --2- 172.21.15.42:0/2445242997 >> v2:172.21.15.154:6812/254845342 conn(0x7efd94037810 0x7efd94039cd0 secure :-1 s=READY pgs=3712 cs=0 l=1 rev1=1 crypto rx=0x7efdb40a3200 tx=0x7efd9c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:58:57.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.669+0000 7efdabfff640 10 client.924110.objecter ms_handle_connect 0x7efd94037810 2024-09-16T03:58:57.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.673+0000 7efdb9dc5640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:58:57.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.673+0000 7efdb9dc5640 1 --2- 172.21.15.42:0/2445242997 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efdb402c650 0x7efdb402ca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:57.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.673+0000 7efdb9dc5640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:57.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.673+0000 7efdb9dc5640 1 -- 172.21.15.42:0/2445242997 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7efdb41cf610 con 0x7efdb402c650 2024-09-16T03:58:57.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.673+0000 7efdb37fe640 1 --2- 172.21.15.42:0/2445242997 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efdb402c650 0x7efdb402ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:57.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.673+0000 7efdb37fe640 10 monclient: get_auth_request con 0x7efdb402c650 auth_method 0 2024-09-16T03:58:57.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.673+0000 7efdb37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:57.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.673+0000 7efdb37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:57.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.673+0000 7efdb37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:58:57.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.673+0000 7efdb37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:58:57.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.673+0000 7efdb37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:58:57.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.673+0000 7efdb37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:58:57.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.673+0000 7efdb37fe640 10 monclient(hunting): handle_auth_done global_id 825048 payload 824 2024-09-16T03:58:57.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.673+0000 7efdb37fe640 1 -- 172.21.15.42:0/2445242997 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7efdb409de20 con 0x7efdb402c650 2024-09-16T03:58:57.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.673+0000 7efdb37fe640 1 --2- 172.21.15.42:0/2445242997 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efdb402c650 0x7efdb402ca60 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7efda401c000 tx=0x7efda40046d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:58:57.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.673+0000 7efdabfff640 10 client.924110.objecter ms_handle_connect 0x7efdb402c650 2024-09-16T03:58:57.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.673+0000 7efdabfff640 10 client.924110.objecter resend_mon_ops 2024-09-16T03:58:57.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.673+0000 7efdabfff640 1 -- 172.21.15.42:0/2445242997 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efda4022cb0 con 0x7efdb402c650 2024-09-16T03:58:57.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.673+0000 7efdabfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:58:57.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.673+0000 7efdabfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:58:57.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.673+0000 7efdabfff640 10 monclient: dump: 2024-09-16T03:58:57.696 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:58:57.696 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:58:57.696 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:57.696 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:57.696 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:58:57.696 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:57.696 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:58:57.696 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:58:57.696 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:58:57.697 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:57.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.673+0000 7efdabfff640 1 -- 172.21.15.42:0/2445242997 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4882 (secure 0 0 0) 0x7efda4022cb0 con 0x7efdb402c650 2024-09-16T03:58:57.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.673+0000 7efdabfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:58:57.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.673+0000 7efdabfff640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:58:57.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.673+0000 7efdabfff640 1 -- 172.21.15.42:0/2445242997 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efdb402c650 msgr2=0x7efdb402ca60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:57.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.673+0000 7efdabfff640 1 --2- 172.21.15.42:0/2445242997 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efdb402c650 0x7efdb402ca60 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7efda401c000 tx=0x7efda40046d0 comp rx=0 tx=0).stop 2024-09-16T03:58:57.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.673+0000 7efdabfff640 1 -- 172.21.15.42:0/2445242997 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efdb402c650 msgr2=0x7efdb402ca60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:58:57.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.673+0000 7efdabfff640 1 --2- 172.21.15.42:0/2445242997 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efdb402c650 0x7efdb402ca60 secure :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7efda401c000 tx=0x7efda40046d0 comp rx=0 tx=0).stop 2024-09-16T03:58:57.697 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:58:57.698 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:58:57.698 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:58:57.698 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "leader", 2024-09-16T03:58:57.698 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 154, 2024-09-16T03:58:57.698 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 427528, 2024-09-16T03:58:57.698 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:58:57.698 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:58:57.698 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T03:58:57.698 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T03:58:57.698 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:58:57.698 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 2, 2024-09-16T03:58:57.698 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:58:57.698 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:58:57.698 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:58:57.699 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:58:57.699 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:58:57.699 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:58:57.699 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:58:57.699 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:58:57.699 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:58:57.699 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:58:57.699 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:58:57.699 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:58:57.699 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:58:57.699 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:58:57.699 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:58:57.699 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:58:57.699 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:58:57.700 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:58:57.700 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:58:57.700 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:58:57.700 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:58:57.700 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:58:57.700 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:58:57.700 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:58:57.700 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:58:57.700 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:58:57.700 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:58:57.700 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:58:57.700 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:58:57.700 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:58:57.700 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:58:57.701 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:58:57.701 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:58:57.701 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:58:57.701 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:58:57.701 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:58:57.701 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:58:57.701 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:58:57.701 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:58:57.701 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:58:57.701 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:58:57.701 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:58:57.701 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:58:57.701 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:58:57.702 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:58:57.702 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:58:57.702 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:58:57.702 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:58:57.702 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:58:57.702 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:58:57.702 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:58:57.702 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:58:57.702 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:58:57.702 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:58:57.702 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:58:57.702 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:58:57.702 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:58:57.702 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:58:57.702 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:58:57.703 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:58:57.703 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:58:57.703 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:58:57.703 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:58:57.703 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:58:57.703 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:58:57.703 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:58:57.703 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:58:57.703 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:58:57.703 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:58:57.703 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:58:57.703 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:58:57.703 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:58:57.703 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:58:57.704 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:58:57.704 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:58:57.704 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:58:57.704 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:58:57.704 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:58:57.704 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:58:57.704 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:58:57.704 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:58:57.704 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:58:57.704 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:58:57.704 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:58:57.704 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:58:57.704 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:58:57.704 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:58:57.705 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:58:57.705 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:58:57.705 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:58:57.705 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:58:57.705 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:58:57.705 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:58:57.705 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:58:57.705 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:58:57.705 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:58:57.705 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:58:57.705 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:58:57.705 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:58:57.705 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:58:57.706 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:58:57.706 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:58:57.706 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:58:57.706 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:58:57.706 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:58:57.706 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:58:57.706 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:58:57.706 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:58:57.706 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:58:57.706 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:58:57.706 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:58:57.706 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:58:57.708 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:58:57.708 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:58:57.708 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:58:57.708 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:58:57.708 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:58:57.708 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:58:57.709 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:58:57.709 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:58:57.709 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:58:57.709 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:58:57.709 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:58:57.709 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:58:57.709 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:58:57.709 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:58:57.709 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:58:57.709 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:58:57.709 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:58:57.709 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:58:57.710 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:58:57.710 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:58:57.710 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:58:57.710 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:58:57.710 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:58:57.710 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:58:57.710 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:58:57.710 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:58:57.710 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:58:57.710 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:58:57.710 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:58:57.710 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T03:58:57.711 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:58:57.711 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:58:57.711 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:58:57.711 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 5 2024-09-16T03:58:57.711 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:58:57.711 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:58:57.711 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:58:57.711 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:58:57.711 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:58:57.711 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:58:57.711 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 6 2024-09-16T03:58:57.711 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:58:57.711 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:58:57.711 INFO:teuthology.orchestra.run.smithi042.stdout: "mgr": [ 2024-09-16T03:58:57.712 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:58:57.712 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:58:57.712 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:58:57.712 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:58:57.712 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:58:57.712 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:58:57.712 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:58:57.712 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:58:57.712 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:58:57.714 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.673+0000 7efdb9dc5640 10 librados: watch_flush enter 2024-09-16T03:58:57.714 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.673+0000 7efdb9dc5640 10 librados: watch_flush exit 2024-09-16T03:58:57.714 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.673+0000 7efdb9dc5640 20 client.924110.objecter shutdown clearing up homeless session... 2024-09-16T03:58:57.714 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.673+0000 7efdb9dc5640 10 client.924110.objecter successfully canceled tick 2024-09-16T03:58:57.714 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.673+0000 7efdb9dc5640 1 -- 172.21.15.42:0/2445242997 >> v2:172.21.15.154:6812/254845342 conn(0x7efd94037810 msgr2=0x7efd94039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:57.714 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.673+0000 7efdb9dc5640 1 --2- 172.21.15.42:0/2445242997 >> v2:172.21.15.154:6812/254845342 conn(0x7efd94037810 0x7efd94039cd0 secure :-1 s=READY pgs=3712 cs=0 l=1 rev1=1 crypto rx=0x7efdb40a3200 tx=0x7efd9c009040 comp rx=0 tx=0).stop 2024-09-16T03:58:57.714 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.673+0000 7efdb9dc5640 10 monclient: shutdown 2024-09-16T03:58:57.714 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.673+0000 7efdb9dc5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:58:57.714 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.673+0000 7efdb9dc5640 1 -- 172.21.15.42:0/2445242997 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7efdb40a3c00 msgr2=0x7efdb40a1960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:57.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.673+0000 7efdb9dc5640 1 --2- 172.21.15.42:0/2445242997 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7efdb40a3c00 0x7efdb40a1960 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7efdac0ab000 tx=0x7efdac019650 comp rx=0 tx=0).stop 2024-09-16T03:58:57.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.673+0000 7efdb3fff640 1 -- 172.21.15.42:0/2445242997 reap_dead start 2024-09-16T03:58:57.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.673+0000 7efdb3fff640 1 -- 172.21.15.42:0/2445242997 reap_dead start 2024-09-16T03:58:57.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.673+0000 7efdb9dc5640 1 -- 172.21.15.42:0/2445242997 shutdown_connections 2024-09-16T03:58:57.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.673+0000 7efdb9dc5640 1 -- 172.21.15.42:0/2445242997 >> 172.21.15.42:0/2445242997 conn(0x7efdb40a9fb0 msgr2=0x7efdb4136ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:58:57.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.673+0000 7efdb9dc5640 1 -- 172.21.15.42:0/2445242997 shutdown_connections 2024-09-16T03:58:57.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.673+0000 7efdb9dc5640 1 -- 172.21.15.42:0/2445242997 wait complete. 2024-09-16T03:58:57.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.673+0000 7efdb9dc5640 1 librados: shutdown 2024-09-16T03:58:57.715 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:58:57.715 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T03:58:57.716 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T03:58:57.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.821+0000 7f821a6ec640 10 monclient: get_monmap_and_config 2024-09-16T03:58:57.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.821+0000 7f821a6ec640 10 monclient: build_initial_monmap 2024-09-16T03:58:57.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.821+0000 7f821a6ec640 10 monclient: monmap: 2024-09-16T03:58:57.830 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:58:57.830 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:58:57.830 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:58:57.828035+0000 2024-09-16T03:58:57.830 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:58:57.828035+0000 2024-09-16T03:58:57.830 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:58:57.830 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:57.830 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:58:57.830 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:58:57.830 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:58:57.830 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:58:57.830 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:58:57.830 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:57.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.825+0000 7f821a6ec640 1 Processor -- start 2024-09-16T03:58:57.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.825+0000 7f821a6ec640 1 -- start start 2024-09-16T03:58:57.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.825+0000 7f821a6ec640 10 monclient: init 2024-09-16T03:58:57.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.825+0000 7f821a6ec640 10 monclient: _reopen_session rank -1 2024-09-16T03:58:57.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.825+0000 7f821a6ec640 10 monclient: _add_conns ranks=[0,1,4,3,2] 2024-09-16T03:58:57.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.825+0000 7f821a6ec640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f821406dc10 addr v1:172.21.15.42:6789/0 2024-09-16T03:58:57.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.825+0000 7f821a6ec640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f82140a38e0 addr v1:172.21.15.154:6789/0 2024-09-16T03:58:57.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.825+0000 7f821a6ec640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f82140a3d30 0x7f821413f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:57.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.825+0000 7f821a6ec640 10 monclient(hunting): picked mon.noname-c con 0x7f82140a3d30 addr v2:172.21.15.154:3300/0 2024-09-16T03:58:57.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.825+0000 7f821a6ec640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f821409be00 con 0x7f821406dc10 2024-09-16T03:58:57.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.825+0000 7f821a6ec640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8214134080 con 0x7f82140a38e0 2024-09-16T03:58:57.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.825+0000 7f821a6ec640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:57.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.825+0000 7f821a6ec640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f821406dff0 con 0x7f82140a3d30 2024-09-16T03:58:57.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.825+0000 7f821a6ec640 10 monclient(hunting): _renew_subs 2024-09-16T03:58:57.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.825+0000 7f821a6ec640 10 monclient(hunting): authenticate will time out at 1984.797970s 2024-09-16T03:58:57.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.825+0000 7f8213fff640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f821406dc10 0x7f8214133970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:37236/0 (socket says 172.21.15.42:37236) 2024-09-16T03:58:57.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.825+0000 7f8213fff640 1 -- 172.21.15.42:0/2883710731 learned_addr learned my addr 172.21.15.42:0/2883710731 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:58:57.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.825+0000 7f8218c89640 1 --2- 172.21.15.42:0/2883710731 >> v2:172.21.15.154:3300/0 conn(0x7f82140a3d30 0x7f821413f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:57.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.825+0000 7f8218c89640 10 monclient(hunting): get_auth_request con 0x7f82140a3d30 auth_method 0 2024-09-16T03:58:57.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.825+0000 7f8218c89640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:57.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.825+0000 7f8218c89640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:57.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.825+0000 7f8218c89640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:58:57.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.825+0000 7f8212ffd640 1 -- 172.21.15.42:0/2883710731 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 488297237 0 0) 0x7f821409be00 con 0x7f821406dc10 2024-09-16T03:58:57.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.825+0000 7f8212ffd640 10 monclient(hunting): _negotiate 2024-09-16T03:58:57.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.825+0000 7f8212ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:57.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.825+0000 7f8212ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:58:57.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.825+0000 7f8212ffd640 10 monclient(hunting): my global_id is 825051 2024-09-16T03:58:57.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.825+0000 7f8212ffd640 1 -- 172.21.15.42:0/2883710731 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f81fc003530 con 0x7f821406dc10 2024-09-16T03:58:57.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.825+0000 7f8218c89640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:58:57.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.825+0000 7f8218c89640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:58:57.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.825+0000 7f8218c89640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:58:57.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.825+0000 7f8212ffd640 1 -- 172.21.15.42:0/2883710731 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1756066718 0 0) 0x7f81fc003530 con 0x7f821406dc10 2024-09-16T03:58:57.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.825+0000 7f8212ffd640 1 -- 172.21.15.42:0/2883710731 >> v2:172.21.15.154:3300/0 conn(0x7f82140a3d30 msgr2=0x7f821413f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:58:57.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.825+0000 7f8212ffd640 1 --2- 172.21.15.42:0/2883710731 >> v2:172.21.15.154:3300/0 conn(0x7f82140a3d30 0x7f821413f920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:57.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.825+0000 7f8212ffd640 1 -- 172.21.15.42:0/2883710731 >> v1:172.21.15.154:6789/0 conn(0x7f82140a38e0 legacy=0x7f821413c1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:57.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.825+0000 7f8212ffd640 10 monclient: _finish_hunting 0 2024-09-16T03:58:57.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.825+0000 7f8212ffd640 1 monclient: found mon.noname-a-legacy 2024-09-16T03:58:57.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.825+0000 7f8212ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:58:57.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.825+0000 7f8212ffd640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.42:6789/0 2024-09-16T03:58:57.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.825+0000 7f8212ffd640 1 -- 172.21.15.42:0/2883710731 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f821409c350 con 0x7f821406dc10 2024-09-16T03:58:57.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.825+0000 7f8212ffd640 10 monclient: _finish_auth 0 2024-09-16T03:58:57.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.825+0000 7f8212ffd640 10 monclient: _check_auth_tickets 2024-09-16T03:58:57.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.825+0000 7f8212ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:58:57.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.825+0000 7f8212ffd640 1 -- 172.21.15.42:0/2883710731 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2177413734 0 0) 0x7f8204001680 con 0x7f821406dc10 2024-09-16T03:58:57.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.825+0000 7f8212ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:58:57.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.825+0000 7f8212ffd640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T03:58:57.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.825+0000 7f8212ffd640 10 monclient: dump: 2024-09-16T03:58:57.838 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:58:57.838 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:58:57.838 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:57.838 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:57.838 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:58:57.838 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:57.839 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:58:57.839 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:58:57.839 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:58:57.839 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:57.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.825+0000 7f8212ffd640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] but i'm connected to v1:172.21.15.42:6789/0, reconnecting 2024-09-16T03:58:57.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.825+0000 7f8212ffd640 10 monclient: _reopen_session rank -1 2024-09-16T03:58:57.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.825+0000 7f8212ffd640 1 -- 172.21.15.42:0/2883710731 >> v1:172.21.15.42:6789/0 conn(0x7f821406dc10 legacy=0x7f8214133970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:57.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.825+0000 7f8212ffd640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:58:57.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.825+0000 7f8212ffd640 1 --2- 172.21.15.42:0/2883710731 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f81fc0039c0 0x7f81fc003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:57.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.825+0000 7f8212ffd640 10 monclient(hunting): picked mon.b con 0x7f81fc0039c0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:58:57.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.825+0000 7f8212ffd640 1 --2- 172.21.15.42:0/2883710731 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f81fc004810 0x7f81fc004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:57.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.825+0000 7f8212ffd640 10 monclient(hunting): picked mon.a con 0x7f81fc004810 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:58:57.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.825+0000 7f8212ffd640 1 --2- 172.21.15.42:0/2883710731 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f81fc0052f0 0x7f81fc00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:57.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.825+0000 7f8212ffd640 10 monclient(hunting): picked mon.c con 0x7f81fc0052f0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:58:57.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.825+0000 7f8212ffd640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:57.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.825+0000 7f8212ffd640 1 -- 172.21.15.42:0/2883710731 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f821406dff0 con 0x7f81fc004810 2024-09-16T03:58:57.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.825+0000 7f8212ffd640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:57.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.825+0000 7f8212ffd640 1 -- 172.21.15.42:0/2883710731 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f81fc00c1c0 con 0x7f81fc0039c0 2024-09-16T03:58:57.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.825+0000 7f8212ffd640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:57.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.825+0000 7f8212ffd640 1 -- 172.21.15.42:0/2883710731 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f81fc00c3c0 con 0x7f81fc0052f0 2024-09-16T03:58:57.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.825+0000 7f8212ffd640 10 monclient(hunting): _renew_subs 2024-09-16T03:58:57.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.825+0000 7f8212ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:58:57.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.825+0000 7f82137fe640 1 --2- 172.21.15.42:0/2883710731 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f81fc004810 0x7f81fc004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:57.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.825+0000 7f8218c89640 1 --2- 172.21.15.42:0/2883710731 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f81fc0052f0 0x7f81fc00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:57.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.825+0000 7f82137fe640 10 monclient(hunting): get_auth_request con 0x7f81fc004810 auth_method 0 2024-09-16T03:58:57.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.825+0000 7f82137fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:57.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.825+0000 7f82137fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:57.841 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.825+0000 7f82137fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:58:57.841 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.825+0000 7f8218c89640 10 monclient(hunting): get_auth_request con 0x7f81fc0052f0 auth_method 0 2024-09-16T03:58:57.841 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.825+0000 7f8218c89640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:57.841 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.825+0000 7f8218c89640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:57.841 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.825+0000 7f8218c89640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:58:57.841 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.825+0000 7f82137fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:58:57.841 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.825+0000 7f82137fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:58:57.841 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.825+0000 7f82137fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T03:58:57.841 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.825+0000 7f8218c89640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:58:57.841 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.825+0000 7f8218c89640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:58:57.841 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.825+0000 7f8218c89640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T03:58:57.841 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.825+0000 7f82137fe640 10 monclient(hunting): handle_auth_done global_id 825051 payload 293 2024-09-16T03:58:57.841 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.825+0000 7f82137fe640 1 -- 172.21.15.42:0/2883710731 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f81fc0052f0 msgr2=0x7f81fc00b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:58:57.842 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.825+0000 7f8213fff640 1 --2- 172.21.15.42:0/2883710731 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f81fc0039c0 0x7f81fc003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:57.842 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.825+0000 7f82137fe640 1 --2- 172.21.15.42:0/2883710731 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f81fc0052f0 0x7f81fc00b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:57.842 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.825+0000 7f82137fe640 1 -- 172.21.15.42:0/2883710731 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f81fc0039c0 msgr2=0x7f81fc003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:58:57.842 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.825+0000 7f82137fe640 1 --2- 172.21.15.42:0/2883710731 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f81fc0039c0 0x7f81fc003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:57.842 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.825+0000 7f82137fe640 10 monclient: _finish_hunting 0 2024-09-16T03:58:57.842 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.825+0000 7f82137fe640 1 monclient: found mon.a 2024-09-16T03:58:57.842 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.825+0000 7f82137fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:58:57.842 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.825+0000 7f82137fe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:58:57.842 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.825+0000 7f82137fe640 1 -- 172.21.15.42:0/2883710731 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81fc003530 con 0x7f81fc004810 2024-09-16T03:58:57.842 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.825+0000 7f82137fe640 10 monclient: _finish_auth 0 2024-09-16T03:58:57.842 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.825+0000 7f82137fe640 10 monclient: _check_auth_tickets 2024-09-16T03:58:57.842 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.825+0000 7f82137fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:58:57.842 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.825+0000 7f8213fff640 10 monclient: get_auth_request con 0x7f81fc0039c0 auth_method 0 2024-09-16T03:58:57.843 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.825+0000 7f821a6ec640 5 monclient: authenticate success, global_id 825051 2024-09-16T03:58:57.843 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.825+0000 7f8213fff640 1 --2- 172.21.15.42:0/2883710731 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f81fc0039c0 0x7f81fc003de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T03:58:57.843 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.825+0000 7f821a6ec640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:58:57.843 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.825+0000 7f8218c89640 1 -- 172.21.15.42:0/2883710731 reap_dead start 2024-09-16T03:58:57.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.829+0000 7f82137fe640 1 --2- 172.21.15.42:0/2883710731 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f81fc004810 0x7f81fc004c90 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f8208001750 tx=0x7f820801c380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e942787a7107385 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:58:57.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.829+0000 7f8212ffd640 1 -- 172.21.15.42:0/2883710731 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82080013a0 con 0x7f81fc004810 2024-09-16T03:58:57.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.829+0000 7f8212ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:58:57.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.829+0000 7f8212ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:58:57.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.829+0000 7f8212ffd640 10 monclient: dump: 2024-09-16T03:58:57.851 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:58:57.851 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:58:57.851 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:57.852 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:57.852 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:58:57.852 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:57.852 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:58:57.852 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:58:57.852 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:58:57.852 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:57.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.829+0000 7f8212ffd640 1 -- 172.21.15.42:0/2883710731 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f82080052f0 con 0x7f81fc004810 2024-09-16T03:58:57.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.829+0000 7f821a6ec640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:58:57.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.829+0000 7f8212ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:58:57.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.829+0000 7f821a6ec640 10 monclient: get_monmap_and_config success 2024-09-16T03:58:57.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.829+0000 7f821a6ec640 4 set_mon_vals no callback set 2024-09-16T03:58:57.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.829+0000 7f8212ffd640 1 -- 172.21.15.42:0/2883710731 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8208003840 con 0x7f81fc004810 2024-09-16T03:58:57.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.829+0000 7f8212ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:58:57.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.829+0000 7f8212ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:58:57.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.829+0000 7f8212ffd640 10 monclient: dump: 2024-09-16T03:58:57.853 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:58:57.853 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:58:57.853 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:57.853 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:57.853 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:58:57.853 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:57.853 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:58:57.853 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:58:57.853 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:58:57.853 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:57.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.829+0000 7f821a6ec640 10 monclient: shutdown 2024-09-16T03:58:57.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.829+0000 7f821a6ec640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:58:57.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.829+0000 7f821a6ec640 1 -- 172.21.15.42:0/2883710731 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f81fc004810 msgr2=0x7f81fc004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:57.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.829+0000 7f821a6ec640 1 --2- 172.21.15.42:0/2883710731 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f81fc004810 0x7f81fc004c90 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f8208001750 tx=0x7f820801c380 comp rx=0 tx=0).stop 2024-09-16T03:58:57.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.829+0000 7f821a6ec640 1 -- 172.21.15.42:0/2883710731 shutdown_connections 2024-09-16T03:58:57.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.829+0000 7f821a6ec640 1 --2- 172.21.15.42:0/2883710731 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f81fc004810 0x7f81fc004c90 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:57.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.829+0000 7f821a6ec640 1 -- 172.21.15.42:0/2883710731 >> 172.21.15.42:0/2883710731 conn(0x7f82140a9fb0 msgr2=0x7f82140aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:58:57.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.829+0000 7f821a6ec640 1 -- 172.21.15.42:0/2883710731 shutdown_connections 2024-09-16T03:58:57.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.829+0000 7f821a6ec640 1 -- 172.21.15.42:0/2883710731 wait complete. 2024-09-16T03:58:57.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.829+0000 7f821a6ec640 10 monclient: build_initial_monmap 2024-09-16T03:58:57.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.829+0000 7f821a6ec640 10 monclient: monmap: 2024-09-16T03:58:57.854 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:58:57.854 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:58:57.854 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:58:57.854 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:58:57.855 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:58:57.855 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:57.855 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:58:57.855 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:58:57.855 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:58:57.855 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:57.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.829+0000 7f821a6ec640 1 librados: starting msgr at 2024-09-16T03:58:57.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.829+0000 7f821a6ec640 1 librados: starting objecter 2024-09-16T03:58:57.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.829+0000 7f821a6ec640 1 Processor -- start 2024-09-16T03:58:57.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.829+0000 7f821a6ec640 1 -- start start 2024-09-16T03:58:57.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.829+0000 7f821a6ec640 1 librados: setting wanted keys 2024-09-16T03:58:57.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.829+0000 7f821a6ec640 1 librados: calling monclient init 2024-09-16T03:58:57.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.829+0000 7f821a6ec640 10 monclient: init 2024-09-16T03:58:57.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.829+0000 7f821a6ec640 10 monclient: _reopen_session rank -1 2024-09-16T03:58:57.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.829+0000 7f821a6ec640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T03:58:57.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.829+0000 7f821a6ec640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f81fc004810 0x7f82141cf370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:57.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.829+0000 7f821a6ec640 10 monclient(hunting): picked mon.noname-b con 0x7f81fc004810 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:58:57.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.829+0000 7f821a6ec640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f821412d5b0 0x7f821412da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:57.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.829+0000 7f821a6ec640 10 monclient(hunting): picked mon.noname-c con 0x7f821412d5b0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:58:57.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.829+0000 7f821a6ec640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f821409f780 0x7f821409fc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:57.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.829+0000 7f821a6ec640 10 monclient(hunting): picked mon.noname-a con 0x7f821409f780 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:58:57.857 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.829+0000 7f821a6ec640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:57.857 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.829+0000 7f821a6ec640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f82140a4f40 con 0x7f821409f780 2024-09-16T03:58:57.858 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.829+0000 7f821a6ec640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:57.858 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.829+0000 7f821a6ec640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f82140a4dc0 con 0x7f81fc004810 2024-09-16T03:58:57.858 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.829+0000 7f821a6ec640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:57.858 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.829+0000 7f821a6ec640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f82140a50c0 con 0x7f821412d5b0 2024-09-16T03:58:57.858 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.829+0000 7f821a6ec640 10 monclient(hunting): _renew_subs 2024-09-16T03:58:57.858 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.829+0000 7f821a6ec640 10 monclient(hunting): authenticate will time out at 1984.803864s 2024-09-16T03:58:57.858 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.829+0000 7f8218c89640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f821409f780 0x7f821409fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:57.858 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.833+0000 7f82137fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f821412d5b0 0x7f821412da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:57.858 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.833+0000 7f8218c89640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f821409f780 0x7f821409fc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:33466/0 (socket says 172.21.15.42:33466) 2024-09-16T03:58:57.858 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.833+0000 7f8218c89640 1 -- 172.21.15.42:0/1115892858 learned_addr learned my addr 172.21.15.42:0/1115892858 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:58:57.859 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.833+0000 7f8218c89640 10 monclient(hunting): get_auth_request con 0x7f821409f780 auth_method 0 2024-09-16T03:58:57.859 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.833+0000 7f8218c89640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:57.859 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.833+0000 7f8218c89640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:57.859 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.833+0000 7f8218c89640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:58:57.859 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.833+0000 7f82137fe640 10 monclient(hunting): get_auth_request con 0x7f821412d5b0 auth_method 0 2024-09-16T03:58:57.859 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.833+0000 7f82137fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:57.859 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.833+0000 7f82137fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:57.859 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.833+0000 7f82137fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:58:57.859 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.833+0000 7f82137fe640 1 -- 172.21.15.42:0/1115892858 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f821412d5b0 msgr2=0x7f821412da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-16T03:58:57.859 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.833+0000 7f82137fe640 1 -- 172.21.15.42:0/1115892858 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f821412d5b0 msgr2=0x7f821412da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T03:58:57.859 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.833+0000 7f82137fe640 1 --2- 172.21.15.42:0/1115892858 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f821412d5b0 0x7f821412da10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:58:57.859 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.833+0000 7f8218c89640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:58:57.860 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.833+0000 7f8218c89640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:58:57.860 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.833+0000 7f8218c89640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:58:57.860 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.833+0000 7f82137fe640 1 --2- 172.21.15.42:0/1115892858 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f821412d5b0 0x7f821412da10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:58:57.860 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.833+0000 7f8213fff640 1 --2- 172.21.15.42:0/1115892858 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f81fc004810 0x7f82141cf370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:57.860 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.833+0000 7f8213fff640 10 monclient(hunting): get_auth_request con 0x7f81fc004810 auth_method 0 2024-09-16T03:58:57.860 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.833+0000 7f8213fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:57.860 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.833+0000 7f8213fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:57.860 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.833+0000 7f8213fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:58:57.860 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.833+0000 7f8218c89640 10 monclient(hunting): handle_auth_done global_id 825054 payload 824 2024-09-16T03:58:57.860 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.833+0000 7f8218c89640 1 -- 172.21.15.42:0/1115892858 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f821412d5b0 msgr2=0x7f821412da10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:58:57.860 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.833+0000 7f8218c89640 1 --2- 172.21.15.42:0/1115892858 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f821412d5b0 0x7f821412da10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:57.860 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.833+0000 7f8218c89640 1 -- 172.21.15.42:0/1115892858 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f81fc004810 msgr2=0x7f82141cf370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:57.861 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.833+0000 7f8218c89640 1 --2- 172.21.15.42:0/1115892858 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f81fc004810 0x7f82141cf370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:57.861 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.833+0000 7f8218c89640 10 monclient: _finish_hunting 0 2024-09-16T03:58:57.861 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.833+0000 7f8218c89640 1 monclient: found mon.noname-a 2024-09-16T03:58:57.861 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.833+0000 7f8218c89640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:58:57.861 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.833+0000 7f8218c89640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:58:57.861 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.833+0000 7f8218c89640 1 -- 172.21.15.42:0/1115892858 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f821412df50 con 0x7f821409f780 2024-09-16T03:58:57.861 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.833+0000 7f8218c89640 1 --2- 172.21.15.42:0/1115892858 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f821409f780 0x7f821409fc10 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f821406e960 tx=0x7f820c00f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:58:57.861 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.833+0000 7f8210ff9640 10 client.?.objecter ms_handle_connect 0x7f821409f780 2024-09-16T03:58:57.861 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.833+0000 7f8210ff9640 10 client.?.objecter resend_mon_ops 2024-09-16T03:58:57.861 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.833+0000 7f8210ff9640 1 -- 172.21.15.42:0/1115892858 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f820c00fde0 con 0x7f821409f780 2024-09-16T03:58:57.861 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.833+0000 7f8210ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:58:57.861 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.833+0000 7f8210ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:58:57.862 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.833+0000 7f8210ff9640 10 monclient: dump: 2024-09-16T03:58:57.862 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:58:57.862 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:58:57.862 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:57.862 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:57.862 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:58:57.862 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:57.862 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:58:57.863 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:58:57.863 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:58:57.863 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:57.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.833+0000 7f8210ff9640 10 monclient: _finish_auth 0 2024-09-16T03:58:57.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.833+0000 7f8210ff9640 10 monclient: _check_auth_tickets 2024-09-16T03:58:57.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.833+0000 7f8210ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:58:57.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.833+0000 7f8210ff9640 1 -- 172.21.15.42:0/1115892858 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f820c015430 con 0x7f821409f780 2024-09-16T03:58:57.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.833+0000 7f821a6ec640 5 monclient: authenticate success, global_id 825054 2024-09-16T03:58:57.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.833+0000 7f821a6ec640 10 monclient: _renew_subs 2024-09-16T03:58:57.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.833+0000 7f821a6ec640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:58:57.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.833+0000 7f821a6ec640 1 -- 172.21.15.42:0/1115892858 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82141cf8b0 con 0x7f821409f780 2024-09-16T03:58:57.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.833+0000 7f8210ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T03:58:57.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.833+0000 7f8210ff9640 1 -- 172.21.15.42:0/1115892858 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f820c015820 con 0x7f821409f780 2024-09-16T03:58:57.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.833+0000 7f8210ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:58:57.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.833+0000 7f8210ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:58:57.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.833+0000 7f8210ff9640 10 monclient: dump: 2024-09-16T03:58:57.864 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:58:57.864 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:58:57.864 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:57.864 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:57.864 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:58:57.864 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:57.864 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:58:57.864 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:58:57.864 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:58:57.864 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:57.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.833+0000 7f82127fc640 4 set_mon_vals no callback set 2024-09-16T03:58:57.865 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.833+0000 7f821a6ec640 10 client.825054.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:58:57.865 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.833+0000 7f821a6ec640 10 monclient: _renew_subs 2024-09-16T03:58:57.865 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.833+0000 7f821a6ec640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:58:57.865 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.833+0000 7f821a6ec640 1 -- 172.21.15.42:0/1115892858 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82140a2df0 con 0x7f821409f780 2024-09-16T03:58:57.865 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.833+0000 7f821a6ec640 1 librados: init done 2024-09-16T03:58:57.865 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.833+0000 7f821a6ec640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:58:57.865 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.833+0000 7f821a6ec640 1 --2- 172.21.15.42:0/1115892858 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f82140a0150 0x7f821409c870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:57.865 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.833+0000 7f821a6ec640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:57.865 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.833+0000 7f821a6ec640 1 -- 172.21.15.42:0/1115892858 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f82140a85f0 con 0x7f82140a0150 2024-09-16T03:58:57.865 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.833+0000 7f8210ff9640 1 -- 172.21.15.42:0/1115892858 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f820c0159c0 con 0x7f821409f780 2024-09-16T03:58:57.865 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.833+0000 7f8213fff640 1 --2- 172.21.15.42:0/1115892858 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f82140a0150 0x7f821409c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:57.865 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.833+0000 7f8213fff640 10 monclient: get_auth_request con 0x7f82140a0150 auth_method 0 2024-09-16T03:58:57.865 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.833+0000 7f8213fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:57.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.833+0000 7f8213fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:57.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.833+0000 7f8213fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:58:57.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.833+0000 7f8213fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:58:57.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.833+0000 7f8213fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:58:57.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.833+0000 7f8213fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:58:57.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.833+0000 7f8210ff9640 1 --2- 172.21.15.42:0/1115892858 >> v2:172.21.15.154:6812/254845342 conn(0x7f8200037840 0x7f8200039d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:57.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.833+0000 7f8210ff9640 1 -- 172.21.15.42:0/1115892858 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(514..514 src has 366..514) ==== 5040+0+0 (secure 0 0 0) 0x7f820c049ae0 con 0x7f821409f780 2024-09-16T03:58:57.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.833+0000 7f8210ff9640 10 client.825054.objecter ms_dispatch 0x7f8214000ba0 osd_map(514..514 src has 366..514) 2024-09-16T03:58:57.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.833+0000 7f8210ff9640 3 client.825054.objecter handle_osd_map got epochs [514,514] > 0 2024-09-16T03:58:57.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.833+0000 7f8213fff640 10 monclient(hunting): handle_auth_done global_id 924116 payload 824 2024-09-16T03:58:57.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.833+0000 7f8210ff9640 3 client.825054.objecter handle_osd_map decoding full epoch 514 2024-09-16T03:58:57.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.833+0000 7f8213fff640 1 -- 172.21.15.42:0/1115892858 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f821413a1e0 con 0x7f82140a0150 2024-09-16T03:58:57.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.833+0000 7f8210ff9640 20 client.825054.objecter dump_active .. 0 homeless 2024-09-16T03:58:57.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.833+0000 7f82137fe640 1 --2- 172.21.15.42:0/1115892858 >> v2:172.21.15.154:6812/254845342 conn(0x7f8200037840 0x7f8200039d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:57.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.833+0000 7f8210ff9640 10 client.825054.objecter ms_handle_connect 0x7f82140a0150 2024-09-16T03:58:57.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.833+0000 7f8210ff9640 10 client.825054.objecter resend_mon_ops 2024-09-16T03:58:57.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.833+0000 7f8213fff640 1 --2- 172.21.15.42:0/1115892858 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f82140a0150 0x7f821409c870 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f8204000fc0 tx=0x7f820400fc70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:58:57.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.833+0000 7f82137fe640 10 monclient: get_auth_request con 0x7f8200037840 auth_method 0 2024-09-16T03:58:57.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.833+0000 7f8210ff9640 1 -- 172.21.15.42:0/1115892858 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8204017020 con 0x7f82140a0150 2024-09-16T03:58:57.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.833+0000 7f8210ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:58:57.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.833+0000 7f8210ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:58:57.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.833+0000 7f8210ff9640 10 monclient: dump: 2024-09-16T03:58:57.867 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:58:57.867 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:58:57.867 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:57.867 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:57.868 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:58:57.868 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:57.868 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:58:57.868 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:58:57.868 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:58:57.868 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:57.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.833+0000 7f8210ff9640 1 -- 172.21.15.42:0/1115892858 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f820400d910 con 0x7f82140a0150 2024-09-16T03:58:57.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.833+0000 7f8210ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:58:57.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.833+0000 7f8210ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:58:57.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.833+0000 7f8210ff9640 1 -- 172.21.15.42:0/1115892858 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f82140a0150 msgr2=0x7f821409c870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:57.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.833+0000 7f8210ff9640 1 --2- 172.21.15.42:0/1115892858 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f82140a0150 0x7f821409c870 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f8204000fc0 tx=0x7f820400fc70 comp rx=0 tx=0).stop 2024-09-16T03:58:57.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.833+0000 7f8210ff9640 1 -- 172.21.15.42:0/1115892858 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f82140a0150 msgr2=0x7f821409c870 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:58:57.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.833+0000 7f8210ff9640 1 --2- 172.21.15.42:0/1115892858 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f82140a0150 0x7f821409c870 secure :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f8204000fc0 tx=0x7f820400fc70 comp rx=0 tx=0).stop 2024-09-16T03:58:57.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.833+0000 7f8210ff9640 10 client.825054.objecter ms_handle_connect 0x7f8200037840 2024-09-16T03:58:57.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.833+0000 7f82137fe640 1 --2- 172.21.15.42:0/1115892858 >> v2:172.21.15.154:6812/254845342 conn(0x7f8200037840 0x7f8200039d00 secure :-1 s=READY pgs=3713 cs=0 l=1 rev1=1 crypto rx=0x7f81fc0057f0 tx=0x7f82080011a0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:58:57.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.841+0000 7f821a6ec640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:58:57.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.841+0000 7f821a6ec640 1 --2- 172.21.15.42:0/1115892858 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f821409cdb0 0x7f821409d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:57.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.841+0000 7f821a6ec640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:57.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.841+0000 7f821a6ec640 1 -- 172.21.15.42:0/1115892858 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f82140a1a80 con 0x7f821409cdb0 2024-09-16T03:58:57.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.841+0000 7f8213fff640 1 --2- 172.21.15.42:0/1115892858 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f821409cdb0 0x7f821409d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:57.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.841+0000 7f8213fff640 10 monclient: get_auth_request con 0x7f821409cdb0 auth_method 0 2024-09-16T03:58:57.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.841+0000 7f8213fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:57.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.841+0000 7f8213fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:57.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.841+0000 7f8213fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:58:57.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.841+0000 7f8213fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:58:57.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.841+0000 7f8213fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:58:57.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.841+0000 7f8213fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:58:57.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.841+0000 7f8213fff640 10 monclient(hunting): handle_auth_done global_id 924119 payload 824 2024-09-16T03:58:57.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.841+0000 7f8213fff640 1 -- 172.21.15.42:0/1115892858 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f82140a0530 con 0x7f821409cdb0 2024-09-16T03:58:57.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.841+0000 7f8210ff9640 10 client.825054.objecter ms_handle_connect 0x7f821409cdb0 2024-09-16T03:58:57.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.841+0000 7f8210ff9640 10 client.825054.objecter resend_mon_ops 2024-09-16T03:58:57.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.841+0000 7f8213fff640 1 --2- 172.21.15.42:0/1115892858 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f821409cdb0 0x7f821409d1c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f8204001980 tx=0x7f82040018a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:58:57.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.841+0000 7f8210ff9640 1 -- 172.21.15.42:0/1115892858 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8204017030 con 0x7f821409cdb0 2024-09-16T03:58:57.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.841+0000 7f8210ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:58:57.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.841+0000 7f8210ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:58:57.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.841+0000 7f8210ff9640 10 monclient: dump: 2024-09-16T03:58:57.870 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:58:57.870 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:58:57.871 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:57.871 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:57.871 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:58:57.871 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:57.871 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:58:57.871 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:58:57.871 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:58:57.871 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:57.871 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.841+0000 7f8210ff9640 1 -- 172.21.15.42:0/1115892858 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4544 (secure 0 0 0) 0x7f8204013680 con 0x7f821409cdb0 2024-09-16T03:58:57.871 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.841+0000 7f8210ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:58:57.871 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.841+0000 7f8210ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:58:57.871 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.841+0000 7f8210ff9640 1 -- 172.21.15.42:0/1115892858 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f821409cdb0 msgr2=0x7f821409d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:57.871 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.841+0000 7f8210ff9640 1 --2- 172.21.15.42:0/1115892858 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f821409cdb0 0x7f821409d1c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f8204001980 tx=0x7f82040018a0 comp rx=0 tx=0).stop 2024-09-16T03:58:57.871 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.841+0000 7f8210ff9640 1 -- 172.21.15.42:0/1115892858 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f821409cdb0 msgr2=0x7f821409d1c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:58:57.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.841+0000 7f8210ff9640 1 --2- 172.21.15.42:0/1115892858 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f821409cdb0 0x7f821409d1c0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:57.872 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:58:57.872 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:58:57.872 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:58:57.872 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T03:58:57.872 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 154, 2024-09-16T03:58:57.873 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 7395, 2024-09-16T03:58:57.873 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:58:57.873 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:58:57.873 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T03:58:57.873 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T03:58:57.873 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:58:57.873 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 2, 2024-09-16T03:58:57.873 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:58:57.873 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:58:57.873 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:58:57.873 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:58:57.873 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:58:57.873 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:58:57.874 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:58:57.874 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:58:57.874 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:58:57.874 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:58:57.874 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:58:57.874 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:58:57.874 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:58:57.874 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:58:57.874 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:58:57.874 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:58:57.874 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:58:57.874 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:58:57.874 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:58:57.875 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:58:57.875 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:58:57.875 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:58:57.875 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:58:57.875 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:58:57.875 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:58:57.875 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:58:57.875 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:58:57.875 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:58:57.875 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:58:57.875 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:58:57.875 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:58:57.875 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:58:57.876 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:58:57.876 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:58:57.876 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:58:57.876 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:58:57.876 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:58:57.876 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:58:57.876 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:58:57.876 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:58:57.876 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:58:57.876 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:58:57.876 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:58:57.876 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:58:57.876 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:58:57.877 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:58:57.877 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:58:57.877 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:58:57.877 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:58:57.877 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:58:57.877 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:58:57.877 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:58:57.877 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:58:57.877 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:58:57.877 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:58:57.877 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:58:57.877 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:58:57.877 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:58:57.878 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:58:57.878 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:58:57.878 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:58:57.878 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:58:57.878 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:58:57.878 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:58:57.878 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:58:57.878 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:58:57.878 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:58:57.878 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:58:57.878 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:58:57.878 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:58:57.878 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:58:57.879 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:58:57.879 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:58:57.879 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:58:57.879 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:58:57.879 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:58:57.879 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:58:57.879 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:58:57.879 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:58:57.879 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:58:57.879 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:58:57.879 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:58:57.879 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:58:57.879 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:58:57.880 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:58:57.880 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:58:57.880 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:58:57.880 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:58:57.880 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:58:57.880 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:58:57.880 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:58:57.880 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:58:57.880 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:58:57.880 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:58:57.880 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:58:57.880 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:58:57.880 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:58:57.881 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:58:57.881 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:58:57.881 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:58:57.881 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:58:57.881 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:58:57.881 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:58:57.881 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:58:57.881 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:58:57.881 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:58:57.881 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:58:57.881 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:58:57.881 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:58:57.881 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:58:57.882 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:58:57.882 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:58:57.882 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:58:57.882 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:58:57.882 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:58:57.882 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:58:57.882 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:58:57.882 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:58:57.882 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:58:57.882 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:58:57.882 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:58:57.882 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:58:57.882 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:58:57.883 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:58:57.883 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:58:57.883 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:58:57.883 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:58:57.883 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:58:57.883 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:58:57.883 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:58:57.883 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:58:57.883 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:58:57.883 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:58:57.883 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:58:57.883 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:58:57.883 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:58:57.883 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:58:57.884 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:58:57.884 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:58:57.884 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:58:57.884 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:58:57.884 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:58:57.884 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:58:57.884 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:58:57.884 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:58:57.884 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:58:57.884 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:58:57.884 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:58:57.884 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:58:57.884 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:58:57.885 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:58:57.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.841+0000 7f821a6ec640 10 librados: watch_flush enter 2024-09-16T03:58:57.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.841+0000 7f821a6ec640 10 librados: watch_flush exit 2024-09-16T03:58:57.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.841+0000 7f821a6ec640 20 client.825054.objecter shutdown clearing up homeless session... 2024-09-16T03:58:57.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.841+0000 7f821a6ec640 10 client.825054.objecter successfully canceled tick 2024-09-16T03:58:57.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.841+0000 7f821a6ec640 1 -- 172.21.15.42:0/1115892858 >> v2:172.21.15.154:6812/254845342 conn(0x7f8200037840 msgr2=0x7f8200039d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:57.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.841+0000 7f821a6ec640 1 --2- 172.21.15.42:0/1115892858 >> v2:172.21.15.154:6812/254845342 conn(0x7f8200037840 0x7f8200039d00 secure :-1 s=READY pgs=3713 cs=0 l=1 rev1=1 crypto rx=0x7f81fc0057f0 tx=0x7f82080011a0 comp rx=0 tx=0).stop 2024-09-16T03:58:57.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.841+0000 7f821a6ec640 10 monclient: shutdown 2024-09-16T03:58:57.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.841+0000 7f821a6ec640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:58:57.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.841+0000 7f821a6ec640 1 -- 172.21.15.42:0/1115892858 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f821409f780 msgr2=0x7f821409fc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:57.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.841+0000 7f821a6ec640 1 --2- 172.21.15.42:0/1115892858 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f821409f780 0x7f821409fc10 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f821406e960 tx=0x7f820c00f580 comp rx=0 tx=0).stop 2024-09-16T03:58:57.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.841+0000 7f8218c89640 1 -- 172.21.15.42:0/1115892858 reap_dead start 2024-09-16T03:58:57.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.841+0000 7f8218c89640 1 -- 172.21.15.42:0/1115892858 reap_dead start 2024-09-16T03:58:57.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.841+0000 7f821a6ec640 1 -- 172.21.15.42:0/1115892858 shutdown_connections 2024-09-16T03:58:57.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.841+0000 7f821a6ec640 1 -- 172.21.15.42:0/1115892858 >> 172.21.15.42:0/1115892858 conn(0x7f82140a9fb0 msgr2=0x7f82140a8b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:58:57.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.841+0000 7f821a6ec640 1 -- 172.21.15.42:0/1115892858 shutdown_connections 2024-09-16T03:58:57.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.841+0000 7f821a6ec640 1 -- 172.21.15.42:0/1115892858 wait complete. 2024-09-16T03:58:57.888 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.841+0000 7f821a6ec640 1 librados: shutdown 2024-09-16T03:58:57.888 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:58:57.888 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T03:58:57.888 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T03:58:58.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.993+0000 7fc977617640 10 monclient: get_monmap_and_config 2024-09-16T03:58:58.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.993+0000 7fc977617640 10 monclient: build_initial_monmap 2024-09-16T03:58:58.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.993+0000 7fc977617640 10 monclient: monmap: 2024-09-16T03:58:58.001 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:58:58.001 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:58:58.001 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:58:57.999819+0000 2024-09-16T03:58:58.002 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:58:57.999819+0000 2024-09-16T03:58:58.002 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:58:58.002 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:58.002 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:58:58.002 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:58:58.002 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:58:58.002 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:58:58.002 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:58:58.002 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:58.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.997+0000 7fc977617640 1 Processor -- start 2024-09-16T03:58:58.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.997+0000 7fc977617640 1 -- start start 2024-09-16T03:58:58.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.997+0000 7fc977617640 10 monclient: init 2024-09-16T03:58:58.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.997+0000 7fc977617640 10 monclient: _reopen_session rank -1 2024-09-16T03:58:58.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.997+0000 7fc977617640 10 monclient: _add_conns ranks=[1,3,2,0,4] 2024-09-16T03:58:58.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.997+0000 7fc977617640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc9700a39b0 addr v1:172.21.15.154:6789/0 2024-09-16T03:58:58.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.997+0000 7fc977617640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fc9700a3d90 0x7fc97013c380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:58.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.997+0000 7fc977617640 10 monclient(hunting): picked mon.noname-a con 0x7fc9700a3d90 addr v2:172.21.15.42:3300/0 2024-09-16T03:58:58.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.997+0000 7fc977617640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc97013c8c0 0x7fc97013ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:58.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.997+0000 7fc977617640 10 monclient(hunting): picked mon.noname-b con 0x7fc97013c8c0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:58:58.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.997+0000 7fc977617640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc970134290 con 0x7fc9700a39b0 2024-09-16T03:58:58.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.997+0000 7fc977617640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:58.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.997+0000 7fc977617640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7fc97009c140 con 0x7fc9700a3d90 2024-09-16T03:58:58.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.997+0000 7fc977617640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:58.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.997+0000 7fc977617640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fc97009c420 con 0x7fc97013c8c0 2024-09-16T03:58:58.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.997+0000 7fc977617640 10 monclient(hunting): _renew_subs 2024-09-16T03:58:58.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.997+0000 7fc977617640 10 monclient(hunting): authenticate will time out at 1984.969169s 2024-09-16T03:58:58.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.997+0000 7fc974bb2640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fc9700a3d90 0x7fc97013c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:58.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.997+0000 7fc975bb4640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc97013c8c0 0x7fc97013ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:58.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.997+0000 7fc974bb2640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fc9700a3d90 0x7fc97013c380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:33474/0 (socket says 172.21.15.42:33474) 2024-09-16T03:58:58.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.997+0000 7fc975bb4640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc97013c8c0 0x7fc97013ecb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:57428/0 (socket says 172.21.15.42:57428) 2024-09-16T03:58:58.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.997+0000 7fc974bb2640 1 -- 172.21.15.42:0/2202197158 learned_addr learned my addr 172.21.15.42:0/2202197158 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:58:58.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.997+0000 7fc974bb2640 10 monclient(hunting): get_auth_request con 0x7fc9700a3d90 auth_method 0 2024-09-16T03:58:58.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.997+0000 7fc974bb2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:58.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.997+0000 7fc974bb2640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:58.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.997+0000 7fc974bb2640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:58:58.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.997+0000 7fc975bb4640 10 monclient(hunting): get_auth_request con 0x7fc97013c8c0 auth_method 0 2024-09-16T03:58:58.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.997+0000 7fc975bb4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:58.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.997+0000 7fc975bb4640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:58.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.997+0000 7fc975bb4640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:58:58.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.997+0000 7fc975bb4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:58:58.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.997+0000 7fc975bb4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:58:58.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.997+0000 7fc975bb4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:58:58.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.997+0000 7fc974bb2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:58:58.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.997+0000 7fc974bb2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:58:58.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.997+0000 7fc974bb2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:58:58.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.997+0000 7fc975bb4640 10 monclient(hunting): handle_auth_done global_id 924122 payload 274 2024-09-16T03:58:58.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.997+0000 7fc975bb4640 1 -- 172.21.15.42:0/2202197158 >> v2:172.21.15.42:3300/0 conn(0x7fc9700a3d90 msgr2=0x7fc97013c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:58:58.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.997+0000 7fc975bb4640 1 --2- 172.21.15.42:0/2202197158 >> v2:172.21.15.42:3300/0 conn(0x7fc9700a3d90 0x7fc97013c380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:58.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.997+0000 7fc975bb4640 1 -- 172.21.15.42:0/2202197158 >> v1:172.21.15.154:6789/0 conn(0x7fc9700a39b0 legacy=0x7fc970133b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:58.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.997+0000 7fc975bb4640 10 monclient: _finish_hunting 0 2024-09-16T03:58:58.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.997+0000 7fc975bb4640 1 monclient: found mon.noname-b 2024-09-16T03:58:58.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.997+0000 7fc975bb4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:58:58.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.997+0000 7fc975bb4640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T03:58:58.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:57.997+0000 7fc975bb4640 1 -- 172.21.15.42:0/2202197158 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc97006dd60 con 0x7fc97013c8c0 2024-09-16T03:58:58.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.001+0000 7fc975bb4640 1 --2- 172.21.15.42:0/2202197158 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc97013c8c0 0x7fc97013ecb0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fc96c0097c0 tx=0x7fc96c01c360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cf4cd54fbd60bc9b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:58:58.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.001+0000 7fc967fff640 1 -- 172.21.15.42:0/2202197158 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc96c0b0070 con 0x7fc97013c8c0 2024-09-16T03:58:58.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.001+0000 7fc967fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:58:58.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.001+0000 7fc967fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:58:58.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.001+0000 7fc967fff640 10 monclient: dump: 2024-09-16T03:58:58.014 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:58:58.014 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:58:58.014 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:58.014 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:58.014 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:58:58.014 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:58.015 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:58:58.015 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:58:58.015 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:58:58.015 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:58.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.001+0000 7fc967fff640 10 monclient: _finish_auth 0 2024-09-16T03:58:58.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.001+0000 7fc967fff640 10 monclient: _check_auth_tickets 2024-09-16T03:58:58.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.001+0000 7fc967fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:58:58.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.001+0000 7fc977617640 5 monclient: authenticate success, global_id 924122 2024-09-16T03:58:58.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.001+0000 7fc967fff640 1 -- 172.21.15.42:0/2202197158 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc96c0ab600 con 0x7fc97013c8c0 2024-09-16T03:58:58.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.001+0000 7fc977617640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:58:58.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.001+0000 7fc967fff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:58:58.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.001+0000 7fc977617640 10 monclient: get_monmap_and_config success 2024-09-16T03:58:58.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.001+0000 7fc977617640 4 set_mon_vals no callback set 2024-09-16T03:58:58.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.001+0000 7fc967fff640 1 -- 172.21.15.42:0/2202197158 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc96c0ab9d0 con 0x7fc97013c8c0 2024-09-16T03:58:58.016 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.001+0000 7fc967fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:58:58.016 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.001+0000 7fc967fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:58:58.016 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.001+0000 7fc967fff640 10 monclient: dump: 2024-09-16T03:58:58.016 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:58:58.016 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:58:58.016 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:58.016 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:58.016 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:58:58.016 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:58.016 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:58:58.016 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:58:58.016 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:58:58.016 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:58.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.001+0000 7fc977617640 10 monclient: shutdown 2024-09-16T03:58:58.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.001+0000 7fc977617640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:58:58.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.001+0000 7fc977617640 1 -- 172.21.15.42:0/2202197158 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc97013c8c0 msgr2=0x7fc97013ecb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:58.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.001+0000 7fc977617640 1 --2- 172.21.15.42:0/2202197158 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc97013c8c0 0x7fc97013ecb0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fc96c0097c0 tx=0x7fc96c01c360 comp rx=0 tx=0).stop 2024-09-16T03:58:58.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.001+0000 7fc977617640 1 -- 172.21.15.42:0/2202197158 shutdown_connections 2024-09-16T03:58:58.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.001+0000 7fc977617640 1 --2- 172.21.15.42:0/2202197158 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc97013c8c0 0x7fc97013ecb0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:58.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.001+0000 7fc977617640 1 --2- 172.21.15.42:0/2202197158 >> v2:172.21.15.42:3300/0 conn(0x7fc9700a3d90 0x7fc97013c380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:58.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.001+0000 7fc977617640 1 -- 172.21.15.42:0/2202197158 >> 172.21.15.42:0/2202197158 conn(0x7fc9700a9fb0 msgr2=0x7fc9700aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:58:58.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.001+0000 7fc977617640 1 -- 172.21.15.42:0/2202197158 shutdown_connections 2024-09-16T03:58:58.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.001+0000 7fc977617640 1 -- 172.21.15.42:0/2202197158 wait complete. 2024-09-16T03:58:58.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.001+0000 7fc977617640 10 monclient: build_initial_monmap 2024-09-16T03:58:58.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.001+0000 7fc977617640 10 monclient: monmap: 2024-09-16T03:58:58.017 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:58:58.018 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:58:58.018 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:58:58.018 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:58:58.018 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:58:58.018 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:58.018 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:58:58.018 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:58:58.018 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:58:58.018 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:58.018 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.001+0000 7fc977617640 1 librados: starting msgr at 2024-09-16T03:58:58.018 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.001+0000 7fc977617640 1 librados: starting objecter 2024-09-16T03:58:58.018 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.001+0000 7fc977617640 1 Processor -- start 2024-09-16T03:58:58.018 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.001+0000 7fc977617640 1 -- start start 2024-09-16T03:58:58.019 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.001+0000 7fc977617640 1 librados: setting wanted keys 2024-09-16T03:58:58.019 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.001+0000 7fc977617640 1 librados: calling monclient init 2024-09-16T03:58:58.019 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.001+0000 7fc977617640 10 monclient: init 2024-09-16T03:58:58.019 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.001+0000 7fc977617640 10 monclient: _reopen_session rank -1 2024-09-16T03:58:58.019 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.001+0000 7fc977617640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:58:58.019 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.001+0000 7fc977617640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc9700a39b0 0x7fc97013b280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:58.019 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.001+0000 7fc977617640 10 monclient(hunting): picked mon.noname-c con 0x7fc9700a39b0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:58:58.019 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.001+0000 7fc977617640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc9700a3d90 0x7fc97012d620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:58.019 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.001+0000 7fc977617640 10 monclient(hunting): picked mon.noname-b con 0x7fc9700a3d90 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:58:58.019 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.001+0000 7fc977617640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc97013c8c0 0x7fc97012db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:58.019 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.001+0000 7fc977617640 10 monclient(hunting): picked mon.noname-a con 0x7fc97013c8c0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:58:58.019 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.001+0000 7fc977617640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:58.019 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.001+0000 7fc977617640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fc970141da0 con 0x7fc97013c8c0 2024-09-16T03:58:58.020 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.001+0000 7fc977617640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:58.020 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.001+0000 7fc977617640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc970141c20 con 0x7fc9700a3d90 2024-09-16T03:58:58.020 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.001+0000 7fc977617640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:58.020 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.001+0000 7fc977617640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fc970141f20 con 0x7fc9700a39b0 2024-09-16T03:58:58.020 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.001+0000 7fc977617640 10 monclient(hunting): _renew_subs 2024-09-16T03:58:58.020 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.001+0000 7fc977617640 10 monclient(hunting): authenticate will time out at 1984.974407s 2024-09-16T03:58:58.020 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.001+0000 7fc9753b3640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc9700a39b0 0x7fc97013b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:58.020 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.001+0000 7fc975bb4640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc97013c8c0 0x7fc97012db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:58.020 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.001+0000 7fc9753b3640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc9700a39b0 0x7fc97013b280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:57436/0 (socket says 172.21.15.42:57436) 2024-09-16T03:58:58.020 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.001+0000 7fc9753b3640 1 -- 172.21.15.42:0/3808555087 learned_addr learned my addr 172.21.15.42:0/3808555087 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:58:58.020 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.001+0000 7fc9753b3640 10 monclient(hunting): get_auth_request con 0x7fc9700a39b0 auth_method 0 2024-09-16T03:58:58.020 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.001+0000 7fc9753b3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:58.020 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.001+0000 7fc9753b3640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:58.020 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.001+0000 7fc9753b3640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:58:58.021 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.001+0000 7fc975bb4640 10 monclient(hunting): get_auth_request con 0x7fc97013c8c0 auth_method 0 2024-09-16T03:58:58.021 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.001+0000 7fc975bb4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:58.021 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.001+0000 7fc975bb4640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:58.021 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.001+0000 7fc975bb4640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:58:58.021 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.001+0000 7fc975bb4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:58:58.021 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.001+0000 7fc975bb4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:58:58.021 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.001+0000 7fc975bb4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:58:58.021 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.001+0000 7fc9753b3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:58:58.021 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.001+0000 7fc9753b3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:58:58.021 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.001+0000 7fc9753b3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:58:58.021 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.001+0000 7fc974bb2640 1 --2- 172.21.15.42:0/3808555087 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc9700a3d90 0x7fc97012d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:58.021 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.001+0000 7fc974bb2640 10 monclient(hunting): get_auth_request con 0x7fc9700a3d90 auth_method 0 2024-09-16T03:58:58.021 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.001+0000 7fc974bb2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:58.022 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.001+0000 7fc974bb2640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:58.022 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.001+0000 7fc974bb2640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:58:58.022 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.001+0000 7fc975bb4640 10 monclient(hunting): handle_auth_done global_id 825060 payload 824 2024-09-16T03:58:58.022 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.001+0000 7fc975bb4640 1 -- 172.21.15.42:0/3808555087 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc9700a39b0 msgr2=0x7fc97013b280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:58.022 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.001+0000 7fc975bb4640 1 --2- 172.21.15.42:0/3808555087 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc9700a39b0 0x7fc97013b280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:58.022 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.001+0000 7fc975bb4640 1 -- 172.21.15.42:0/3808555087 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc9700a3d90 msgr2=0x7fc97012d620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:58.022 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.001+0000 7fc975bb4640 1 --2- 172.21.15.42:0/3808555087 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc9700a3d90 0x7fc97012d620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:58.022 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.001+0000 7fc975bb4640 10 monclient: _finish_hunting 0 2024-09-16T03:58:58.022 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.001+0000 7fc975bb4640 1 monclient: found mon.noname-a 2024-09-16T03:58:58.022 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.001+0000 7fc975bb4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:58:58.022 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.001+0000 7fc975bb4640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:58:58.022 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.001+0000 7fc975bb4640 1 -- 172.21.15.42:0/3808555087 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc97006f680 con 0x7fc97013c8c0 2024-09-16T03:58:58.022 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.001+0000 7fc975bb4640 1 --2- 172.21.15.42:0/3808555087 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc97013c8c0 0x7fc97012db60 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fc97009c5a0 tx=0x7fc96c019100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:58:58.022 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.001+0000 7fc965ffb640 10 client.?.objecter ms_handle_connect 0x7fc97013c8c0 2024-09-16T03:58:58.023 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.001+0000 7fc965ffb640 10 client.?.objecter resend_mon_ops 2024-09-16T03:58:58.023 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.001+0000 7fc965ffb640 1 -- 172.21.15.42:0/3808555087 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc96c0b0040 con 0x7fc97013c8c0 2024-09-16T03:58:58.023 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.001+0000 7fc965ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:58:58.023 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.001+0000 7fc965ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:58:58.023 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.001+0000 7fc965ffb640 10 monclient: dump: 2024-09-16T03:58:58.023 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:58:58.023 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:58:58.023 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:58.023 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:58.023 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:58:58.023 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:58.023 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:58:58.023 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:58:58.024 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:58:58.024 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:58.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.001+0000 7fc965ffb640 10 monclient: _finish_auth 0 2024-09-16T03:58:58.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.001+0000 7fc965ffb640 10 monclient: _check_auth_tickets 2024-09-16T03:58:58.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.001+0000 7fc965ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:58:58.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.001+0000 7fc965ffb640 1 -- 172.21.15.42:0/3808555087 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc96c0b9070 con 0x7fc97013c8c0 2024-09-16T03:58:58.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.001+0000 7fc977617640 5 monclient: authenticate success, global_id 825060 2024-09-16T03:58:58.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.001+0000 7fc977617640 10 monclient: _renew_subs 2024-09-16T03:58:58.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.001+0000 7fc977617640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:58:58.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.001+0000 7fc977617640 1 -- 172.21.15.42:0/3808555087 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc97013c300 con 0x7fc97013c8c0 2024-09-16T03:58:58.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.001+0000 7fc965ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T03:58:58.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.001+0000 7fc965ffb640 1 -- 172.21.15.42:0/3808555087 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc96c0b4690 con 0x7fc97013c8c0 2024-09-16T03:58:58.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.001+0000 7fc965ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:58:58.025 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.001+0000 7fc965ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:58:58.025 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.001+0000 7fc965ffb640 10 monclient: dump: 2024-09-16T03:58:58.025 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:58:58.025 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:58:58.025 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:58.025 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:58.025 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:58:58.025 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:58.025 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:58:58.025 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:58:58.025 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:58:58.025 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:58.025 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.001+0000 7fc977617640 10 client.825060.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:58:58.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.001+0000 7fc977617640 10 monclient: _renew_subs 2024-09-16T03:58:58.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.001+0000 7fc977617640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:58:58.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.001+0000 7fc977617640 1 -- 172.21.15.42:0/3808555087 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc97013a5c0 con 0x7fc97013c8c0 2024-09-16T03:58:58.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.001+0000 7fc977617640 1 librados: init done 2024-09-16T03:58:58.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.001+0000 7fc9677fe640 4 set_mon_vals no callback set 2024-09-16T03:58:58.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.001+0000 7fc977617640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:58:58.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.001+0000 7fc977617640 1 --2- 172.21.15.42:0/3808555087 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc9701cf0c0 0x7fc9701cf4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:58.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.001+0000 7fc977617640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:58.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.001+0000 7fc977617640 1 -- 172.21.15.42:0/3808555087 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc97013a920 con 0x7fc9701cf0c0 2024-09-16T03:58:58.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.005+0000 7fc965ffb640 1 -- 172.21.15.42:0/3808555087 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fc96c0b4830 con 0x7fc97013c8c0 2024-09-16T03:58:58.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.005+0000 7fc965ffb640 1 --2- 172.21.15.42:0/3808555087 >> v2:172.21.15.154:6812/254845342 conn(0x7fc954037810 0x7fc954039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:58.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.005+0000 7fc965ffb640 1 -- 172.21.15.42:0/3808555087 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(514..514 src has 366..514) ==== 5040+0+0 (secure 0 0 0) 0x7fc96c0e28d0 con 0x7fc97013c8c0 2024-09-16T03:58:58.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.005+0000 7fc965ffb640 10 client.825060.objecter ms_dispatch 0x7fc970000ba0 osd_map(514..514 src has 366..514) 2024-09-16T03:58:58.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.005+0000 7fc965ffb640 3 client.825060.objecter handle_osd_map got epochs [514,514] > 0 2024-09-16T03:58:58.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.005+0000 7fc965ffb640 3 client.825060.objecter handle_osd_map decoding full epoch 514 2024-09-16T03:58:58.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.005+0000 7fc965ffb640 20 client.825060.objecter dump_active .. 0 homeless 2024-09-16T03:58:58.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.005+0000 7fc974bb2640 1 --2- 172.21.15.42:0/3808555087 >> v2:172.21.15.154:6812/254845342 conn(0x7fc954037810 0x7fc954039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:58.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.005+0000 7fc974bb2640 10 monclient: get_auth_request con 0x7fc954037810 auth_method 0 2024-09-16T03:58:58.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.005+0000 7fc9753b3640 1 --2- 172.21.15.42:0/3808555087 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc9701cf0c0 0x7fc9701cf4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:58.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.005+0000 7fc9753b3640 10 monclient: get_auth_request con 0x7fc9701cf0c0 auth_method 0 2024-09-16T03:58:58.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.005+0000 7fc9753b3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:58.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.005+0000 7fc9753b3640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:58.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.005+0000 7fc9753b3640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:58:58.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.005+0000 7fc9753b3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:58:58.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.005+0000 7fc9753b3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:58:58.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.005+0000 7fc9753b3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:58:58.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.005+0000 7fc965ffb640 10 client.825060.objecter ms_handle_connect 0x7fc954037810 2024-09-16T03:58:58.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.005+0000 7fc974bb2640 1 --2- 172.21.15.42:0/3808555087 >> v2:172.21.15.154:6812/254845342 conn(0x7fc954037810 0x7fc954039cd0 secure :-1 s=READY pgs=3714 cs=0 l=1 rev1=1 crypto rx=0x7fc970131300 tx=0x7fc960008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:58:58.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.005+0000 7fc9753b3640 10 monclient(hunting): handle_auth_done global_id 904228 payload 824 2024-09-16T03:58:58.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.005+0000 7fc9753b3640 1 -- 172.21.15.42:0/3808555087 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc970139790 con 0x7fc9701cf0c0 2024-09-16T03:58:58.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.005+0000 7fc965ffb640 10 client.825060.objecter ms_handle_connect 0x7fc9701cf0c0 2024-09-16T03:58:58.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.005+0000 7fc965ffb640 10 client.825060.objecter resend_mon_ops 2024-09-16T03:58:58.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.005+0000 7fc9753b3640 1 --2- 172.21.15.42:0/3808555087 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc9701cf0c0 0x7fc9701cf4a0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fc96800fb10 tx=0x7fc9680020e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:58:58.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.005+0000 7fc965ffb640 1 -- 172.21.15.42:0/3808555087 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc968019070 con 0x7fc9701cf0c0 2024-09-16T03:58:58.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.005+0000 7fc965ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:58:58.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.005+0000 7fc965ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:58:58.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.005+0000 7fc965ffb640 10 monclient: dump: 2024-09-16T03:58:58.028 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:58:58.028 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:58:58.028 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:58.029 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:58.029 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:58:58.029 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:58.029 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:58:58.029 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:58:58.029 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:58:58.029 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:58.029 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.005+0000 7fc965ffb640 1 -- 172.21.15.42:0/3808555087 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fc968006e30 con 0x7fc9701cf0c0 2024-09-16T03:58:58.029 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.005+0000 7fc965ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:58:58.029 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.005+0000 7fc965ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:58:58.029 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.005+0000 7fc965ffb640 1 -- 172.21.15.42:0/3808555087 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc9701cf0c0 msgr2=0x7fc9701cf4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:58.029 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.005+0000 7fc965ffb640 1 --2- 172.21.15.42:0/3808555087 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc9701cf0c0 0x7fc9701cf4a0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fc96800fb10 tx=0x7fc9680020e0 comp rx=0 tx=0).stop 2024-09-16T03:58:58.029 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.005+0000 7fc965ffb640 1 -- 172.21.15.42:0/3808555087 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc9701cf0c0 msgr2=0x7fc9701cf4a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:58:58.030 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.005+0000 7fc965ffb640 1 --2- 172.21.15.42:0/3808555087 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc9701cf0c0 0x7fc9701cf4a0 secure :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fc96800fb10 tx=0x7fc9680020e0 comp rx=0 tx=0).stop 2024-09-16T03:58:58.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.013+0000 7fc977617640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:58:58.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.013+0000 7fc977617640 1 --2- 172.21.15.42:0/3808555087 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc9701cf9e0 0x7fc970135bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:58.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.013+0000 7fc977617640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:58.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.013+0000 7fc977617640 1 -- 172.21.15.42:0/3808555087 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc9700a5750 con 0x7fc9701cf9e0 2024-09-16T03:58:58.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.013+0000 7fc9753b3640 1 --2- 172.21.15.42:0/3808555087 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc9701cf9e0 0x7fc970135bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:58.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.013+0000 7fc9753b3640 10 monclient: get_auth_request con 0x7fc9701cf9e0 auth_method 0 2024-09-16T03:58:58.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.013+0000 7fc9753b3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:58.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.013+0000 7fc9753b3640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:58.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.013+0000 7fc9753b3640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:58:58.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.013+0000 7fc9753b3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:58:58.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.013+0000 7fc9753b3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:58:58.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.013+0000 7fc9753b3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:58:58.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.013+0000 7fc9753b3640 10 monclient(hunting): handle_auth_done global_id 904231 payload 824 2024-09-16T03:58:58.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.013+0000 7fc9753b3640 1 -- 172.21.15.42:0/3808555087 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fc970068e40 con 0x7fc9701cf9e0 2024-09-16T03:58:58.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.013+0000 7fc9753b3640 1 --2- 172.21.15.42:0/3808555087 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc9701cf9e0 0x7fc970135bc0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fc96800fb10 tx=0x7fc968001260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:58:58.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.013+0000 7fc965ffb640 10 client.825060.objecter ms_handle_connect 0x7fc9701cf9e0 2024-09-16T03:58:58.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.013+0000 7fc965ffb640 10 client.825060.objecter resend_mon_ops 2024-09-16T03:58:58.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.013+0000 7fc965ffb640 1 -- 172.21.15.42:0/3808555087 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc968019070 con 0x7fc9701cf9e0 2024-09-16T03:58:58.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.013+0000 7fc965ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:58:58.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.013+0000 7fc965ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:58:58.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.013+0000 7fc965ffb640 10 monclient: dump: 2024-09-16T03:58:58.033 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:58:58.033 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:58:58.033 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:58.033 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:58.033 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:58:58.033 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:58.033 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:58:58.033 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:58:58.033 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:58:58.033 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:58.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.013+0000 7fc965ffb640 1 -- 172.21.15.42:0/3808555087 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4712 (secure 0 0 0) 0x7fc96801a040 con 0x7fc9701cf9e0 2024-09-16T03:58:58.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.013+0000 7fc965ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:58:58.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.013+0000 7fc965ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:58:58.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.013+0000 7fc965ffb640 1 -- 172.21.15.42:0/3808555087 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc9701cf9e0 msgr2=0x7fc970135bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:58.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.013+0000 7fc965ffb640 1 --2- 172.21.15.42:0/3808555087 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc9701cf9e0 0x7fc970135bc0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fc96800fb10 tx=0x7fc968001260 comp rx=0 tx=0).stop 2024-09-16T03:58:58.034 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.013+0000 7fc965ffb640 1 -- 172.21.15.42:0/3808555087 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc9701cf9e0 msgr2=0x7fc970135bc0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:58:58.034 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.013+0000 7fc965ffb640 1 --2- 172.21.15.42:0/3808555087 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc9701cf9e0 0x7fc970135bc0 secure :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fc96800fb10 tx=0x7fc968001260 comp rx=0 tx=0).stop 2024-09-16T03:58:58.034 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:58:58.034 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:58:58.034 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:58:58.034 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T03:58:58.034 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 154, 2024-09-16T03:58:58.034 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 66198, 2024-09-16T03:58:58.034 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:58:58.034 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:58:58.035 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T03:58:58.035 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T03:58:58.035 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:58:58.035 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 2, 2024-09-16T03:58:58.035 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:58:58.035 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:58:58.035 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:58:58.035 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:58:58.035 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:58:58.035 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:58:58.035 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:58:58.035 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:58:58.035 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:58:58.035 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:58:58.035 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:58:58.036 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:58:58.036 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:58:58.036 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:58:58.036 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:58:58.036 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:58:58.036 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:58:58.036 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:58:58.036 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:58:58.036 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:58:58.036 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:58:58.036 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:58:58.036 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:58:58.036 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:58:58.037 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:58:58.037 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:58:58.037 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:58:58.037 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:58:58.037 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:58:58.037 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:58:58.037 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:58:58.037 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:58:58.037 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:58:58.037 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:58:58.037 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:58:58.037 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:58:58.037 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:58:58.037 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:58:58.038 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:58:58.038 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:58:58.038 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:58:58.038 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:58:58.038 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:58:58.038 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:58:58.038 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:58:58.038 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:58:58.038 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:58:58.038 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:58:58.038 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:58:58.038 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:58:58.038 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:58:58.038 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:58:58.039 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:58:58.039 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:58:58.039 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:58:58.039 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:58:58.039 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:58:58.039 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:58:58.039 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:58:58.039 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:58:58.039 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:58:58.039 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:58:58.039 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:58:58.039 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:58:58.039 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:58:58.040 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:58:58.040 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:58:58.040 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:58:58.040 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:58:58.040 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:58:58.040 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:58:58.040 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:58:58.040 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:58:58.040 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:58:58.040 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:58:58.040 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:58:58.040 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:58:58.040 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:58:58.041 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:58:58.041 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:58:58.041 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:58:58.041 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:58:58.041 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:58:58.041 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:58:58.041 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:58:58.041 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:58:58.041 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:58:58.041 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:58:58.041 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:58:58.041 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:58:58.041 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:58:58.041 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:58:58.042 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:58:58.042 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:58:58.042 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:58:58.042 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:58:58.042 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:58:58.042 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:58:58.042 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:58:58.042 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:58:58.042 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:58:58.042 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:58:58.042 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:58:58.042 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:58:58.042 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:58:58.042 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:58:58.043 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:58:58.043 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:58:58.043 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:58:58.043 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:58:58.043 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:58:58.043 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:58:58.043 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:58:58.043 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:58:58.043 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:58:58.043 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:58:58.043 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:58:58.043 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:58:58.043 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:58:58.044 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:58:58.044 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:58:58.044 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:58:58.044 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:58:58.044 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:58:58.044 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:58:58.044 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:58:58.044 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:58:58.044 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:58:58.044 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:58:58.044 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:58:58.044 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:58:58.044 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:58:58.044 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:58:58.045 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:58:58.045 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:58:58.045 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:58:58.045 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:58:58.045 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:58:58.045 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:58:58.045 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:58:58.045 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:58:58.045 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T03:58:58.045 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:58:58.045 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:58:58.045 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:58:58.045 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:58:58.046 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:58:58.046 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:58:58.046 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:58:58.046 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:58:58.046 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:58:58.046 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:58:58.046 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 2 2024-09-16T03:58:58.046 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:58:58.046 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:58:58.046 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:58:58.046 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:58:58.046 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:58:58.048 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.013+0000 7fc977617640 10 librados: watch_flush enter 2024-09-16T03:58:58.048 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.013+0000 7fc977617640 10 librados: watch_flush exit 2024-09-16T03:58:58.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.013+0000 7fc977617640 20 client.825060.objecter shutdown clearing up homeless session... 2024-09-16T03:58:58.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.013+0000 7fc977617640 10 client.825060.objecter successfully canceled tick 2024-09-16T03:58:58.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.013+0000 7fc977617640 1 -- 172.21.15.42:0/3808555087 >> v2:172.21.15.154:6812/254845342 conn(0x7fc954037810 msgr2=0x7fc954039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:58.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.013+0000 7fc977617640 1 --2- 172.21.15.42:0/3808555087 >> v2:172.21.15.154:6812/254845342 conn(0x7fc954037810 0x7fc954039cd0 secure :-1 s=READY pgs=3714 cs=0 l=1 rev1=1 crypto rx=0x7fc970131300 tx=0x7fc960008040 comp rx=0 tx=0).stop 2024-09-16T03:58:58.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.013+0000 7fc977617640 10 monclient: shutdown 2024-09-16T03:58:58.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.013+0000 7fc977617640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:58:58.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.013+0000 7fc977617640 1 -- 172.21.15.42:0/3808555087 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc97013c8c0 msgr2=0x7fc97012db60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:58.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.013+0000 7fc977617640 1 --2- 172.21.15.42:0/3808555087 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc97013c8c0 0x7fc97012db60 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fc97009c5a0 tx=0x7fc96c019100 comp rx=0 tx=0).stop 2024-09-16T03:58:58.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.013+0000 7fc975bb4640 1 -- 172.21.15.42:0/3808555087 reap_dead start 2024-09-16T03:58:58.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.013+0000 7fc977617640 1 -- 172.21.15.42:0/3808555087 shutdown_connections 2024-09-16T03:58:58.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.013+0000 7fc977617640 1 --2- 172.21.15.42:0/3808555087 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc97013c8c0 0x7fc97012db60 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:58.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.013+0000 7fc977617640 1 -- 172.21.15.42:0/3808555087 >> 172.21.15.42:0/3808555087 conn(0x7fc9700a9fb0 msgr2=0x7fc970131980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:58:58.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.017+0000 7fc977617640 1 -- 172.21.15.42:0/3808555087 shutdown_connections 2024-09-16T03:58:58.050 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.017+0000 7fc977617640 1 -- 172.21.15.42:0/3808555087 wait complete. 2024-09-16T03:58:58.050 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.017+0000 7fc977617640 1 librados: shutdown 2024-09-16T03:58:58.050 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:58:58.050 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T03:58:58.050 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T03:58:58.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.157+0000 7fdb37f17640 10 monclient: get_monmap_and_config 2024-09-16T03:58:58.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.157+0000 7fdb37f17640 10 monclient: build_initial_monmap 2024-09-16T03:58:58.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.161+0000 7fdb37f17640 10 monclient: monmap: 2024-09-16T03:58:58.165 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:58:58.166 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:58:58.166 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:58:58.164110+0000 2024-09-16T03:58:58.166 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:58:58.164110+0000 2024-09-16T03:58:58.166 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:58:58.166 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:58.166 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:58:58.166 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:58:58.166 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:58:58.166 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:58:58.166 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:58:58.166 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:58.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.161+0000 7fdb37f17640 1 Processor -- start 2024-09-16T03:58:58.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.161+0000 7fdb37f17640 1 -- start start 2024-09-16T03:58:58.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.161+0000 7fdb37f17640 10 monclient: init 2024-09-16T03:58:58.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.161+0000 7fdb37f17640 10 monclient: _reopen_session rank -1 2024-09-16T03:58:58.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.161+0000 7fdb37f17640 10 monclient: _add_conns ranks=[4,1,3,0,2] 2024-09-16T03:58:58.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.161+0000 7fdb37f17640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fdb300a39b0 0x7fdb30133b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:58.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.161+0000 7fdb37f17640 10 monclient(hunting): picked mon.noname-c con 0x7fdb300a39b0 addr v2:172.21.15.154:3300/0 2024-09-16T03:58:58.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.161+0000 7fdb37f17640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fdb300a3d90 addr v1:172.21.15.154:6789/0 2024-09-16T03:58:58.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.161+0000 7fdb37f17640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fdb30137710 0x7fdb3013f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:58.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.161+0000 7fdb37f17640 10 monclient(hunting): picked mon.noname-a con 0x7fdb30137710 addr v2:172.21.15.42:3300/0 2024-09-16T03:58:58.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.161+0000 7fdb37f17640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdb3009c140 con 0x7fdb300a3d90 2024-09-16T03:58:58.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.161+0000 7fdb37f17640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:58.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.161+0000 7fdb37f17640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7fdb3009c420 con 0x7fdb30137710 2024-09-16T03:58:58.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.161+0000 7fdb37f17640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:58.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.161+0000 7fdb37f17640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fdb3006dd60 con 0x7fdb300a39b0 2024-09-16T03:58:58.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.161+0000 7fdb37f17640 10 monclient(hunting): _renew_subs 2024-09-16T03:58:58.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.161+0000 7fdb37f17640 10 monclient(hunting): authenticate will time out at 1985.133533s 2024-09-16T03:58:58.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.161+0000 7fdb364b4640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fdb30137710 0x7fdb3013f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:58.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.161+0000 7fdb364b4640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fdb30137710 0x7fdb3013f3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:33484/0 (socket says 172.21.15.42:33484) 2024-09-16T03:58:58.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.161+0000 7fdb364b4640 1 -- 172.21.15.42:0/3528419562 learned_addr learned my addr 172.21.15.42:0/3528419562 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:58:58.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.161+0000 7fdb364b4640 10 monclient(hunting): get_auth_request con 0x7fdb30137710 auth_method 0 2024-09-16T03:58:58.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.161+0000 7fdb364b4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:58.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.161+0000 7fdb364b4640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:58.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.161+0000 7fdb364b4640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:58:58.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.161+0000 7fdb364b4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:58:58.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.161+0000 7fdb364b4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:58:58.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.161+0000 7fdb364b4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:58:58.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.161+0000 7fdb35cb3640 1 --2- 172.21.15.42:0/3528419562 >> v2:172.21.15.154:3300/0 conn(0x7fdb300a39b0 0x7fdb30133b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:58.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.161+0000 7fdb35cb3640 10 monclient(hunting): get_auth_request con 0x7fdb300a39b0 auth_method 0 2024-09-16T03:58:58.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.161+0000 7fdb35cb3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:58.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.161+0000 7fdb35cb3640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:58.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.161+0000 7fdb35cb3640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:58:58.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.161+0000 7fdb364b4640 10 monclient(hunting): handle_auth_done global_id 825063 payload 274 2024-09-16T03:58:58.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.161+0000 7fdb364b4640 1 -- 172.21.15.42:0/3528419562 >> v2:172.21.15.154:3300/0 conn(0x7fdb300a39b0 msgr2=0x7fdb30133b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:58:58.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.161+0000 7fdb364b4640 1 --2- 172.21.15.42:0/3528419562 >> v2:172.21.15.154:3300/0 conn(0x7fdb300a39b0 0x7fdb30133b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:58.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.161+0000 7fdb364b4640 1 -- 172.21.15.42:0/3528419562 >> v1:172.21.15.154:6789/0 conn(0x7fdb300a3d90 legacy=0x7fdb30137000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:58:58.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.161+0000 7fdb364b4640 10 monclient: _finish_hunting 0 2024-09-16T03:58:58.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.161+0000 7fdb364b4640 1 monclient: found mon.noname-a 2024-09-16T03:58:58.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.161+0000 7fdb364b4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:58:58.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.161+0000 7fdb364b4640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:58:58.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.161+0000 7fdb364b4640 1 -- 172.21.15.42:0/3528419562 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb300a5e20 con 0x7fdb30137710 2024-09-16T03:58:58.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.165+0000 7fdb364b4640 1 --2- 172.21.15.42:0/3528419562 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdb30137710 0x7fdb3013f3a0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fdb300a7750 tx=0x7fdb2801c870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8f68db9e1ce94ce server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:58:58.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.165+0000 7fdb34cb1640 1 -- 172.21.15.42:0/3528419562 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb280a5530 con 0x7fdb30137710 2024-09-16T03:58:58.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.165+0000 7fdb34cb1640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:58:58.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.165+0000 7fdb34cb1640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:58:58.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.165+0000 7fdb34cb1640 10 monclient: dump: 2024-09-16T03:58:58.177 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:58:58.177 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:58:58.177 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:58.177 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:58.177 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:58:58.177 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:58.178 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:58:58.178 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:58:58.178 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:58:58.178 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:58.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.165+0000 7fdb34cb1640 10 monclient: _finish_auth 0 2024-09-16T03:58:58.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.165+0000 7fdb34cb1640 10 monclient: _check_auth_tickets 2024-09-16T03:58:58.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.165+0000 7fdb34cb1640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:58:58.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.165+0000 7fdb34cb1640 1 -- 172.21.15.42:0/3528419562 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb280a56d0 con 0x7fdb30137710 2024-09-16T03:58:58.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.165+0000 7fdb34cb1640 10 monclient: handle_config config(0 keys) 2024-09-16T03:58:58.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.165+0000 7fdb34cb1640 1 -- 172.21.15.42:0/3528419562 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb280a5aa0 con 0x7fdb30137710 2024-09-16T03:58:58.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.165+0000 7fdb37f17640 5 monclient: authenticate success, global_id 825063 2024-09-16T03:58:58.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.165+0000 7fdb37f17640 10 monclient: get_monmap_and_config success 2024-09-16T03:58:58.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.165+0000 7fdb37f17640 4 set_mon_vals no callback set 2024-09-16T03:58:58.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.165+0000 7fdb34cb1640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:58:58.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.165+0000 7fdb34cb1640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:58:58.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.165+0000 7fdb34cb1640 10 monclient: dump: 2024-09-16T03:58:58.179 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:58:58.179 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:58:58.179 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:58.179 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:58.179 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:58:58.179 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:58.179 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:58:58.179 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:58:58.179 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:58:58.179 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:58.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.165+0000 7fdb37f17640 10 monclient: shutdown 2024-09-16T03:58:58.180 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.165+0000 7fdb37f17640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:58:58.180 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.165+0000 7fdb37f17640 1 -- 172.21.15.42:0/3528419562 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdb30137710 msgr2=0x7fdb3013f3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:58.180 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.165+0000 7fdb37f17640 1 --2- 172.21.15.42:0/3528419562 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdb30137710 0x7fdb3013f3a0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fdb300a7750 tx=0x7fdb2801c870 comp rx=0 tx=0).stop 2024-09-16T03:58:58.180 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.165+0000 7fdb37f17640 1 -- 172.21.15.42:0/3528419562 shutdown_connections 2024-09-16T03:58:58.180 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.165+0000 7fdb37f17640 1 --2- 172.21.15.42:0/3528419562 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdb30137710 0x7fdb3013f3a0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:58.180 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.165+0000 7fdb37f17640 1 --2- 172.21.15.42:0/3528419562 >> v2:172.21.15.154:3300/0 conn(0x7fdb300a39b0 0x7fdb30133b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:58.180 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.165+0000 7fdb37f17640 1 -- 172.21.15.42:0/3528419562 >> 172.21.15.42:0/3528419562 conn(0x7fdb300a9fb0 msgr2=0x7fdb300aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:58:58.180 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.165+0000 7fdb37f17640 1 -- 172.21.15.42:0/3528419562 shutdown_connections 2024-09-16T03:58:58.180 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.165+0000 7fdb37f17640 1 -- 172.21.15.42:0/3528419562 wait complete. 2024-09-16T03:58:58.180 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.165+0000 7fdb37f17640 10 monclient: build_initial_monmap 2024-09-16T03:58:58.180 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.165+0000 7fdb37f17640 10 monclient: monmap: 2024-09-16T03:58:58.180 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:58:58.180 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:58:58.180 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:58:58.181 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:58:58.181 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:58:58.181 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:58.181 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:58:58.181 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:58:58.181 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:58:58.181 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:58.181 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.165+0000 7fdb37f17640 1 librados: starting msgr at 2024-09-16T03:58:58.181 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.165+0000 7fdb37f17640 1 librados: starting objecter 2024-09-16T03:58:58.181 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.165+0000 7fdb37f17640 1 Processor -- start 2024-09-16T03:58:58.181 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.165+0000 7fdb37f17640 1 -- start start 2024-09-16T03:58:58.181 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.165+0000 7fdb37f17640 1 librados: setting wanted keys 2024-09-16T03:58:58.181 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.165+0000 7fdb37f17640 1 librados: calling monclient init 2024-09-16T03:58:58.181 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.165+0000 7fdb37f17640 10 monclient: init 2024-09-16T03:58:58.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.165+0000 7fdb37f17640 10 monclient: _reopen_session rank -1 2024-09-16T03:58:58.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.165+0000 7fdb37f17640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:58:58.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.165+0000 7fdb37f17640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdb300a39b0 0x7fdb301cf590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:58.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.165+0000 7fdb37f17640 10 monclient(hunting): picked mon.noname-a con 0x7fdb300a39b0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:58:58.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.165+0000 7fdb37f17640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fdb300a3d90 0x7fdb301cfad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:58.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.165+0000 7fdb37f17640 10 monclient(hunting): picked mon.noname-c con 0x7fdb300a3d90 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:58:58.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.165+0000 7fdb37f17640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdb30137710 0x7fdb301d0010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:58.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.165+0000 7fdb37f17640 10 monclient(hunting): picked mon.noname-b con 0x7fdb30137710 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:58:58.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.165+0000 7fdb37f17640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:58.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.165+0000 7fdb37f17640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fdb30142310 con 0x7fdb300a39b0 2024-09-16T03:58:58.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.165+0000 7fdb37f17640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:58.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.165+0000 7fdb37f17640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fdb30142490 con 0x7fdb30137710 2024-09-16T03:58:58.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.165+0000 7fdb37f17640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:58.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.165+0000 7fdb37f17640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fdb30142610 con 0x7fdb300a3d90 2024-09-16T03:58:58.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.165+0000 7fdb37f17640 10 monclient(hunting): _renew_subs 2024-09-16T03:58:58.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.165+0000 7fdb37f17640 10 monclient(hunting): authenticate will time out at 1985.138812s 2024-09-16T03:58:58.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.165+0000 7fdb35cb3640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdb300a39b0 0x7fdb301cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:58.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.165+0000 7fdb354b2640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fdb300a3d90 0x7fdb301cfad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:58.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.165+0000 7fdb35cb3640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdb300a39b0 0x7fdb301cf590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:33498/0 (socket says 172.21.15.42:33498) 2024-09-16T03:58:58.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.165+0000 7fdb354b2640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fdb300a3d90 0x7fdb301cfad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:57438/0 (socket says 172.21.15.42:57438) 2024-09-16T03:58:58.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.165+0000 7fdb35cb3640 1 -- 172.21.15.42:0/1540716551 learned_addr learned my addr 172.21.15.42:0/1540716551 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:58:58.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.165+0000 7fdb35cb3640 10 monclient(hunting): get_auth_request con 0x7fdb300a39b0 auth_method 0 2024-09-16T03:58:58.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.165+0000 7fdb35cb3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:58.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.165+0000 7fdb35cb3640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:58.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.165+0000 7fdb35cb3640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:58:58.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.165+0000 7fdb354b2640 10 monclient(hunting): get_auth_request con 0x7fdb300a3d90 auth_method 0 2024-09-16T03:58:58.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.165+0000 7fdb354b2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:58.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.165+0000 7fdb354b2640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:58.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.165+0000 7fdb354b2640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:58:58.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.165+0000 7fdb354b2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:58:58.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.165+0000 7fdb354b2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:58:58.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.165+0000 7fdb354b2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:58:58.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.165+0000 7fdb35cb3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:58:58.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.165+0000 7fdb35cb3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:58:58.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.165+0000 7fdb35cb3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:58:58.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.165+0000 7fdb364b4640 1 --2- 172.21.15.42:0/1540716551 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdb30137710 0x7fdb301d0010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:58.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.165+0000 7fdb364b4640 10 monclient(hunting): get_auth_request con 0x7fdb30137710 auth_method 0 2024-09-16T03:58:58.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.165+0000 7fdb364b4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:58.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.165+0000 7fdb364b4640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:58.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.165+0000 7fdb364b4640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:58:58.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.165+0000 7fdb354b2640 10 monclient(hunting): handle_auth_done global_id 924128 payload 824 2024-09-16T03:58:58.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.165+0000 7fdb354b2640 1 -- 172.21.15.42:0/1540716551 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdb30137710 msgr2=0x7fdb301d0010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:58.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.165+0000 7fdb354b2640 1 --2- 172.21.15.42:0/1540716551 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdb30137710 0x7fdb301d0010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:58.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.165+0000 7fdb354b2640 1 -- 172.21.15.42:0/1540716551 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdb300a39b0 msgr2=0x7fdb301cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:58.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.165+0000 7fdb354b2640 1 --2- 172.21.15.42:0/1540716551 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdb300a39b0 0x7fdb301cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:58.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.165+0000 7fdb354b2640 10 monclient: _finish_hunting 0 2024-09-16T03:58:58.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.165+0000 7fdb354b2640 1 monclient: found mon.noname-c 2024-09-16T03:58:58.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.165+0000 7fdb354b2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:58:58.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.165+0000 7fdb354b2640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:58:58.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.165+0000 7fdb354b2640 1 -- 172.21.15.42:0/1540716551 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb3006f680 con 0x7fdb300a3d90 2024-09-16T03:58:58.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.165+0000 7fdb35cb3640 1 --2- 172.21.15.42:0/1540716551 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdb300a39b0 0x7fdb301cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:58:58.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.165+0000 7fdb354b2640 1 --2- 172.21.15.42:0/1540716551 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fdb300a3d90 0x7fdb301cfad0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fdb2000c410 tx=0x7fdb2000c8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:58:58.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.165+0000 7fdb267fc640 10 client.?.objecter ms_handle_connect 0x7fdb300a3d90 2024-09-16T03:58:58.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.165+0000 7fdb267fc640 10 client.?.objecter resend_mon_ops 2024-09-16T03:58:58.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.165+0000 7fdb267fc640 1 -- 172.21.15.42:0/1540716551 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb20017070 con 0x7fdb300a3d90 2024-09-16T03:58:58.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.165+0000 7fdb267fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:58:58.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.165+0000 7fdb267fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:58:58.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.165+0000 7fdb267fc640 10 monclient: dump: 2024-09-16T03:58:58.186 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:58:58.186 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:58:58.186 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:58.186 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:58.186 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:58:58.186 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:58.186 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:58:58.186 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:58:58.186 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:58:58.186 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:58.187 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.165+0000 7fdb267fc640 10 monclient: _finish_auth 0 2024-09-16T03:58:58.187 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.165+0000 7fdb267fc640 10 monclient: _check_auth_tickets 2024-09-16T03:58:58.187 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.165+0000 7fdb267fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:58:58.187 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.165+0000 7fdb267fc640 1 -- 172.21.15.42:0/1540716551 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb200069e0 con 0x7fdb300a3d90 2024-09-16T03:58:58.187 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.165+0000 7fdb37f17640 5 monclient: authenticate success, global_id 924128 2024-09-16T03:58:58.187 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.165+0000 7fdb37f17640 10 monclient: _renew_subs 2024-09-16T03:58:58.187 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.165+0000 7fdb37f17640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:58:58.187 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.165+0000 7fdb37f17640 1 -- 172.21.15.42:0/1540716551 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb30133340 con 0x7fdb300a3d90 2024-09-16T03:58:58.187 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.165+0000 7fdb267fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:58:58.187 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.165+0000 7fdb267fc640 1 -- 172.21.15.42:0/1540716551 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb20006e10 con 0x7fdb300a3d90 2024-09-16T03:58:58.187 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.165+0000 7fdb267fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:58:58.187 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.165+0000 7fdb267fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:58:58.187 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.165+0000 7fdb267fc640 10 monclient: dump: 2024-09-16T03:58:58.187 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:58:58.188 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:58:58.188 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:58.188 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:58.188 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:58:58.188 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:58.188 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:58:58.188 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:58:58.188 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:58:58.188 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:58.188 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.165+0000 7fdb37f17640 10 client.924128.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:58:58.188 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.165+0000 7fdb27fff640 4 set_mon_vals no callback set 2024-09-16T03:58:58.188 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.165+0000 7fdb37f17640 10 monclient: _renew_subs 2024-09-16T03:58:58.188 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.165+0000 7fdb37f17640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:58:58.189 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.165+0000 7fdb37f17640 1 -- 172.21.15.42:0/1540716551 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb3013f0e0 con 0x7fdb300a3d90 2024-09-16T03:58:58.189 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.165+0000 7fdb37f17640 1 librados: init done 2024-09-16T03:58:58.189 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.169+0000 7fdb267fc640 1 -- 172.21.15.42:0/1540716551 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fdb2001a070 con 0x7fdb300a3d90 2024-09-16T03:58:58.189 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.169+0000 7fdb37f17640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:58:58.189 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.169+0000 7fdb37f17640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:58:58.189 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.169+0000 7fdb37f17640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:58:58.189 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.169+0000 7fdb37f17640 1 -- 172.21.15.42:0/1540716551 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdb30133b80 con 0x7fdb300a3d90 2024-09-16T03:58:58.189 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.169+0000 7fdb267fc640 1 --2- 172.21.15.42:0/1540716551 >> v2:172.21.15.154:6812/254845342 conn(0x7fdb14037810 0x7fdb14039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:58.189 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.169+0000 7fdb267fc640 1 -- 172.21.15.42:0/1540716551 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(514..514 src has 366..514) ==== 5040+0+0 (secure 0 0 0) 0x7fdb20047fe0 con 0x7fdb300a3d90 2024-09-16T03:58:58.189 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.169+0000 7fdb267fc640 10 client.924128.objecter ms_dispatch 0x7fdb30000ba0 osd_map(514..514 src has 366..514) 2024-09-16T03:58:58.189 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.169+0000 7fdb267fc640 3 client.924128.objecter handle_osd_map got epochs [514,514] > 0 2024-09-16T03:58:58.189 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.169+0000 7fdb267fc640 3 client.924128.objecter handle_osd_map decoding full epoch 514 2024-09-16T03:58:58.189 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.169+0000 7fdb267fc640 20 client.924128.objecter dump_active .. 0 homeless 2024-09-16T03:58:58.190 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.169+0000 7fdb35cb3640 1 --2- 172.21.15.42:0/1540716551 >> v2:172.21.15.154:6812/254845342 conn(0x7fdb14037810 0x7fdb14039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:58.190 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.169+0000 7fdb35cb3640 10 monclient: get_auth_request con 0x7fdb14037810 auth_method 0 2024-09-16T03:58:58.190 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.169+0000 7fdb267fc640 10 client.924128.objecter ms_handle_connect 0x7fdb14037810 2024-09-16T03:58:58.190 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.169+0000 7fdb35cb3640 1 --2- 172.21.15.42:0/1540716551 >> v2:172.21.15.154:6812/254845342 conn(0x7fdb14037810 0x7fdb14039cd0 secure :-1 s=READY pgs=3715 cs=0 l=1 rev1=1 crypto rx=0x7fdb2c007f00 tx=0x7fdb2c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:58:58.190 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.173+0000 7fdb267fc640 1 -- 172.21.15.42:0/1540716551 <== mon.2 v2:172.21.15.42:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7fdb20007210 con 0x7fdb300a3d90 2024-09-16T03:58:58.190 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.173+0000 7fdb267fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:58:58.190 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.173+0000 7fdb267fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:58:58.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.389+0000 7fdb37f17640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T03:58:58.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.389+0000 7fdb37f17640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T03:58:58.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.389+0000 7fdb37f17640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:58:58.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.389+0000 7fdb37f17640 1 -- 172.21.15.42:0/1540716551 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7fdb300a5750 con 0x7fdb300a3d90 2024-09-16T03:58:58.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.393+0000 7fdb267fc640 1 -- 172.21.15.42:0/1540716551 <== mon.2 v2:172.21.15.42:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7fdb2001a520 con 0x7fdb300a3d90 2024-09-16T03:58:58.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.393+0000 7fdb267fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T03:58:58.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.393+0000 7fdb267fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:58:58.400 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.393+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.400 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.393+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=26,config=2,health=11,kv=10,logm=51} crc {auth=4134856540,config=692619927,health=2740141040,kv=439326260,logm=1241494152}) 2024-09-16T03:58:58.400 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.393+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=26,config=2,health=11,kv=10,logm=51} crc {auth=2348597881,config=692619927,health=4220590635,kv=3446188498,logm=3627758088}) 2024-09-16T03:58:58.400 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.393+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.400 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.393+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=26,config=2,health=11,kv=10,logm=51} crc {auth=4134856540,config=692619927,health=2740141040,kv=439326260,logm=1241494152}) 2024-09-16T03:58:58.401 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.393+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=24,config=2,health=12,kv=9,logm=53} crc {auth=3244750034,config=692619927,health=2386882462,kv=950665868,logm=412243907}) 2024-09-16T03:58:58.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.393+0000 7fdb37f17640 10 librados: watch_flush enter 2024-09-16T03:58:58.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.393+0000 7fdb37f17640 10 librados: watch_flush exit 2024-09-16T03:58:58.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.393+0000 7fdb37f17640 20 client.924128.objecter shutdown clearing up homeless session... 2024-09-16T03:58:58.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.393+0000 7fdb37f17640 10 client.924128.objecter successfully canceled tick 2024-09-16T03:58:58.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.397+0000 7fdb37f17640 1 -- 172.21.15.42:0/1540716551 >> v2:172.21.15.154:6812/254845342 conn(0x7fdb14037810 msgr2=0x7fdb14039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:58.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.397+0000 7fdb37f17640 1 --2- 172.21.15.42:0/1540716551 >> v2:172.21.15.154:6812/254845342 conn(0x7fdb14037810 0x7fdb14039cd0 secure :-1 s=READY pgs=3715 cs=0 l=1 rev1=1 crypto rx=0x7fdb2c007f00 tx=0x7fdb2c008040 comp rx=0 tx=0).stop 2024-09-16T03:58:58.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.397+0000 7fdb37f17640 10 monclient: shutdown 2024-09-16T03:58:58.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.397+0000 7fdb37f17640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:58:58.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.397+0000 7fdb37f17640 1 -- 172.21.15.42:0/1540716551 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fdb300a3d90 msgr2=0x7fdb301cfad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:58.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.397+0000 7fdb37f17640 1 --2- 172.21.15.42:0/1540716551 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fdb300a3d90 0x7fdb301cfad0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fdb2000c410 tx=0x7fdb2000c8d0 comp rx=0 tx=0).stop 2024-09-16T03:58:58.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.397+0000 7fdb37f17640 1 -- 172.21.15.42:0/1540716551 shutdown_connections 2024-09-16T03:58:58.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.397+0000 7fdb37f17640 1 --2- 172.21.15.42:0/1540716551 >> v2:172.21.15.154:6812/254845342 conn(0x7fdb14037810 0x7fdb14039cd0 unknown :-1 s=CLOSED pgs=3715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:58.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.397+0000 7fdb37f17640 1 --2- 172.21.15.42:0/1540716551 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdb30137710 0x7fdb301d0010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:58.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.397+0000 7fdb37f17640 1 --2- 172.21.15.42:0/1540716551 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fdb300a3d90 0x7fdb301cfad0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:58.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.397+0000 7fdb37f17640 1 --2- 172.21.15.42:0/1540716551 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdb300a39b0 0x7fdb301cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:58.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.397+0000 7fdb37f17640 1 -- 172.21.15.42:0/1540716551 >> 172.21.15.42:0/1540716551 conn(0x7fdb300a9fb0 msgr2=0x7fdb30131980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:58:58.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.397+0000 7fdb37f17640 1 -- 172.21.15.42:0/1540716551 shutdown_connections 2024-09-16T03:58:58.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.397+0000 7fdb37f17640 1 -- 172.21.15.42:0/1540716551 wait complete. 2024-09-16T03:58:58.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:58.397+0000 7fdb37f17640 1 librados: shutdown 2024-09-16T03:58:58.404 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.397+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.404 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.397+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=594940942}) 2024-09-16T03:58:58.404 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.397+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1999382724}) 2024-09-16T03:58:58.404 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.397+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.404 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.397+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=594940942}) 2024-09-16T03:58:58.404 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.397+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1332911618}) 2024-09-16T03:58:58.406 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.401+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.406 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.401+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3338139980}) 2024-09-16T03:58:58.406 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.401+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3451560844}) 2024-09-16T03:58:58.407 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.401+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.407 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.401+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3338139980}) 2024-09-16T03:58:58.407 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.401+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2078315073}) 2024-09-16T03:58:58.410 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.405+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.410 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.405+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1785947559}) 2024-09-16T03:58:58.410 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.405+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=84861587}) 2024-09-16T03:58:58.410 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.405+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.410 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.405+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1785947559}) 2024-09-16T03:58:58.410 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.405+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1618045691}) 2024-09-16T03:58:58.413 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.405+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.413 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.405+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3780975097}) 2024-09-16T03:58:58.413 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.405+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=575292934}) 2024-09-16T03:58:58.413 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.405+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.413 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.405+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3780975097}) 2024-09-16T03:58:58.413 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.405+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2734150329}) 2024-09-16T03:58:58.415 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.409+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.415 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.409+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3671905487}) 2024-09-16T03:58:58.416 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.409+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1428512134}) 2024-09-16T03:58:58.416 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.409+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.416 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.409+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3671905487}) 2024-09-16T03:58:58.416 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.409+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1150081885}) 2024-09-16T03:58:58.417 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.413+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.417 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.413+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4134042065}) 2024-09-16T03:58:58.418 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.413+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=983610537}) 2024-09-16T03:58:58.418 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.413+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.418 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.413+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4134042065}) 2024-09-16T03:58:58.418 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.413+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3813392389}) 2024-09-16T03:58:58.419 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T03:58:58.419 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.413+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.420 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.413+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3305989068}) 2024-09-16T03:58:58.420 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.413+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=424847551}) 2024-09-16T03:58:58.420 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.413+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.420 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.413+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3305989068}) 2024-09-16T03:58:58.420 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.413+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3493654287}) 2024-09-16T03:58:58.421 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.413+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.421 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.413+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1536619658}) 2024-09-16T03:58:58.421 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.413+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=865461066}) 2024-09-16T03:58:58.421 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.413+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.421 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.413+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1536619658}) 2024-09-16T03:58:58.421 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.413+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=966116019}) 2024-09-16T03:58:58.423 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.417+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.423 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.417+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=561636410}) 2024-09-16T03:58:58.423 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.417+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2472813413}) 2024-09-16T03:58:58.423 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.417+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.423 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.417+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=561636410}) 2024-09-16T03:58:58.423 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.417+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1715096358}) 2024-09-16T03:58:58.425 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.417+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.425 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.417+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=297301400}) 2024-09-16T03:58:58.425 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.417+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1628338013}) 2024-09-16T03:58:58.425 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.417+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.425 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.417+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=297301400}) 2024-09-16T03:58:58.425 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.417+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1613685693}) 2024-09-16T03:58:58.427 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.421+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.427 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.421+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=103849986}) 2024-09-16T03:58:58.427 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.421+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2940808340}) 2024-09-16T03:58:58.427 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.421+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.427 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.421+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=103849986}) 2024-09-16T03:58:58.427 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.421+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4268620010}) 2024-09-16T03:58:58.429 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.421+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.429 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.421+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1019109657}) 2024-09-16T03:58:58.429 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.421+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3500483510}) 2024-09-16T03:58:58.429 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.421+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.429 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.421+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1019109657}) 2024-09-16T03:58:58.429 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.421+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3773232317}) 2024-09-16T03:58:58.431 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.425+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.431 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.425+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3152208901}) 2024-09-16T03:58:58.431 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.425+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2240229389}) 2024-09-16T03:58:58.431 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.425+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.431 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.425+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3152208901}) 2024-09-16T03:58:58.431 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.425+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=753527091}) 2024-09-16T03:58:58.432 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.425+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.433 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.425+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1005250831}) 2024-09-16T03:58:58.433 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.425+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1364982586}) 2024-09-16T03:58:58.433 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.425+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.433 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.425+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1005250831}) 2024-09-16T03:58:58.433 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.425+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1895022075}) 2024-09-16T03:58:58.434 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.429+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.434 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.429+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3448573329}) 2024-09-16T03:58:58.435 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.429+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=895216356}) 2024-09-16T03:58:58.435 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.429+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.435 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.429+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3448573329}) 2024-09-16T03:58:58.435 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.429+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2043535890}) 2024-09-16T03:58:58.437 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.429+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.437 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.429+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1228457371}) 2024-09-16T03:58:58.437 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.429+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1460204329}) 2024-09-16T03:58:58.437 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.429+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.437 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.429+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1228457371}) 2024-09-16T03:58:58.437 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.429+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1250076771}) 2024-09-16T03:58:58.439 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.433+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.439 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.433+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2397361726}) 2024-09-16T03:58:58.439 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.433+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1324456135}) 2024-09-16T03:58:58.439 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.433+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.439 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.433+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2397361726}) 2024-09-16T03:58:58.439 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.433+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=762163903}) 2024-09-16T03:58:58.441 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.433+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.441 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.433+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2241551658}) 2024-09-16T03:58:58.441 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.433+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2165673022}) 2024-09-16T03:58:58.441 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.433+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.441 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.433+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2241551658}) 2024-09-16T03:58:58.441 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.433+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2441563506}) 2024-09-16T03:58:58.443 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.437+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.443 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.437+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=208366307}) 2024-09-16T03:58:58.443 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.437+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=140329285}) 2024-09-16T03:58:58.443 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.437+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.443 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.437+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=208366307}) 2024-09-16T03:58:58.443 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.437+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=856160235}) 2024-09-16T03:58:58.445 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.437+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.445 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.437+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2029488552}) 2024-09-16T03:58:58.445 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.437+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1442085957}) 2024-09-16T03:58:58.445 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.437+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.445 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.437+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2029488552}) 2024-09-16T03:58:58.445 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.437+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2903799222}) 2024-09-16T03:58:58.447 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.441+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.447 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.441+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2874166941}) 2024-09-16T03:58:58.447 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.441+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3874002133}) 2024-09-16T03:58:58.447 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.441+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.447 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.441+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2874166941}) 2024-09-16T03:58:58.447 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.441+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1725161101}) 2024-09-16T03:58:58.449 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.441+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.449 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.441+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1160335732}) 2024-09-16T03:58:58.449 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.441+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3424812417}) 2024-09-16T03:58:58.449 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.441+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.449 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.441+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1160335732}) 2024-09-16T03:58:58.449 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.441+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2233030834}) 2024-09-16T03:58:58.451 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.445+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.451 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.445+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1403949108}) 2024-09-16T03:58:58.452 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.445+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2517464941}) 2024-09-16T03:58:58.452 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.445+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.452 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.445+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1403949108}) 2024-09-16T03:58:58.452 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.445+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1173445660}) 2024-09-16T03:58:58.454 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.449+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.454 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.449+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2643499401}) 2024-09-16T03:58:58.454 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.449+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3407528996}) 2024-09-16T03:58:58.454 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.449+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.454 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.449+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2643499401}) 2024-09-16T03:58:58.454 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.449+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2533939993}) 2024-09-16T03:58:58.456 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.449+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.456 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.449+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3495828165}) 2024-09-16T03:58:58.456 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.449+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3872767304}) 2024-09-16T03:58:58.456 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.449+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.456 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.449+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3495828165}) 2024-09-16T03:58:58.456 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.449+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=330018408}) 2024-09-16T03:58:58.458 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.453+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.458 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.453+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2424297504}) 2024-09-16T03:58:58.458 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.453+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3553628697}) 2024-09-16T03:58:58.458 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.453+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.458 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.453+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2424297504}) 2024-09-16T03:58:58.458 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.453+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1447059757}) 2024-09-16T03:58:58.460 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.453+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.460 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.453+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3268460965}) 2024-09-16T03:58:58.460 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.453+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2858878227}) 2024-09-16T03:58:58.460 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.453+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.461 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.453+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3268460965}) 2024-09-16T03:58:58.461 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.453+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2692869820}) 2024-09-16T03:58:58.463 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.457+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.463 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.457+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1154796084}) 2024-09-16T03:58:58.463 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.457+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3652000870}) 2024-09-16T03:58:58.463 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.457+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.463 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.457+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1154796084}) 2024-09-16T03:58:58.463 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.457+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2768832447}) 2024-09-16T03:58:58.465 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.457+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.465 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.457+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2728280365}) 2024-09-16T03:58:58.465 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.457+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=901673294}) 2024-09-16T03:58:58.465 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.457+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.465 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.457+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2728280365}) 2024-09-16T03:58:58.465 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.457+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=479416232}) 2024-09-16T03:58:58.467 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.461+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.468 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.461+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3902705985}) 2024-09-16T03:58:58.468 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.461+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2768939083}) 2024-09-16T03:58:58.468 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.461+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.468 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.461+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3902705985}) 2024-09-16T03:58:58.468 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.461+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2923170243}) 2024-09-16T03:58:58.471 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.465+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.471 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.465+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3333899830}) 2024-09-16T03:58:58.472 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.465+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=692683970}) 2024-09-16T03:58:58.472 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.465+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.472 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.465+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3333899830}) 2024-09-16T03:58:58.472 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.465+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=27262047}) 2024-09-16T03:58:58.475 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.469+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.475 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.469+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2182185206}) 2024-09-16T03:58:58.475 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.469+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=977976415}) 2024-09-16T03:58:58.476 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.469+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.476 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.469+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2182185206}) 2024-09-16T03:58:58.476 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.469+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1298064947}) 2024-09-16T03:58:58.485 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.477+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.485 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.477+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3582256060}) 2024-09-16T03:58:58.485 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.477+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2156950073}) 2024-09-16T03:58:58.485 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.477+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.486 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.477+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3582256060}) 2024-09-16T03:58:58.486 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.477+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3526896430}) 2024-09-16T03:58:58.494 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.489+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.494 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.489+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2088102965}) 2024-09-16T03:58:58.494 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.489+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3114236579}) 2024-09-16T03:58:58.495 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.489+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.495 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.489+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2088102965}) 2024-09-16T03:58:58.495 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.489+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=897681497}) 2024-09-16T03:58:58.496 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.489+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.497 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.489+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1760811563}) 2024-09-16T03:58:58.497 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.489+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3387872859}) 2024-09-16T03:58:58.497 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.489+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.497 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.489+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1760811563}) 2024-09-16T03:58:58.497 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.489+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3623113029}) 2024-09-16T03:58:58.498 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.493+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.498 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.493+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=777021424}) 2024-09-16T03:58:58.498 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.493+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2391321057}) 2024-09-16T03:58:58.498 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.493+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.498 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.493+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=777021424}) 2024-09-16T03:58:58.498 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.493+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1087193299}) 2024-09-16T03:58:58.500 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.493+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.500 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.493+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2521593170}) 2024-09-16T03:58:58.500 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.493+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=457340661}) 2024-09-16T03:58:58.500 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.493+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.500 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.493+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2521593170}) 2024-09-16T03:58:58.500 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.493+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3943376527}) 2024-09-16T03:58:58.502 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.497+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.502 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.497+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3586728491}) 2024-09-16T03:58:58.502 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.497+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=377310315}) 2024-09-16T03:58:58.502 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.497+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.502 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.497+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3586728491}) 2024-09-16T03:58:58.502 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.497+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=757425357}) 2024-09-16T03:58:58.503 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.497+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.503 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.497+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3824904449}) 2024-09-16T03:58:58.504 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.497+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2450654370}) 2024-09-16T03:58:58.504 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.497+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.504 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.497+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3824904449}) 2024-09-16T03:58:58.504 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.497+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3675983193}) 2024-09-16T03:58:58.505 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.501+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.505 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.501+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=751455605}) 2024-09-16T03:58:58.506 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.501+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3670340132}) 2024-09-16T03:58:58.506 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.501+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.506 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.501+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=751455605}) 2024-09-16T03:58:58.506 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.501+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2512050612}) 2024-09-16T03:58:58.507 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.501+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.507 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.501+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=206593340}) 2024-09-16T03:58:58.507 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.501+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3686584367}) 2024-09-16T03:58:58.508 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.501+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.508 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.501+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=206593340}) 2024-09-16T03:58:58.508 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.501+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=305560789}) 2024-09-16T03:58:58.509 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.505+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.509 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.505+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1207675260}) 2024-09-16T03:58:58.509 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.505+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=153793085}) 2024-09-16T03:58:58.510 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.505+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.510 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.505+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1207675260}) 2024-09-16T03:58:58.510 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.505+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=875683055}) 2024-09-16T03:58:58.511 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.505+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.511 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.505+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4199405859}) 2024-09-16T03:58:58.511 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.505+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2860712072}) 2024-09-16T03:58:58.511 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.505+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.512 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.505+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4199405859}) 2024-09-16T03:58:58.512 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.505+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2444179461}) 2024-09-16T03:58:58.513 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.505+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.513 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.505+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1985755937}) 2024-09-16T03:58:58.513 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.509+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=881540546}) 2024-09-16T03:58:58.513 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.509+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.513 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.509+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1985755937}) 2024-09-16T03:58:58.513 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.509+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3241091007}) 2024-09-16T03:58:58.516 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.509+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.517 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.509+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2151430445}) 2024-09-16T03:58:58.517 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.509+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1398209000}) 2024-09-16T03:58:58.517 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.509+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.517 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.509+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2151430445}) 2024-09-16T03:58:58.517 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.509+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2874528735}) 2024-09-16T03:58:58.518 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.513+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.518 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.513+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=38955150}) 2024-09-16T03:58:58.519 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.513+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=163992539}) 2024-09-16T03:58:58.519 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.513+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.519 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.513+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=38955150}) 2024-09-16T03:58:58.519 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.513+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1338190031}) 2024-09-16T03:58:58.520 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.513+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.520 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.513+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1182330148}) 2024-09-16T03:58:58.521 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.513+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3172398007}) 2024-09-16T03:58:58.521 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.513+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.521 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.513+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1182330148}) 2024-09-16T03:58:58.521 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.513+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=538068147}) 2024-09-16T03:58:58.522 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.517+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.523 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.517+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=486163062}) 2024-09-16T03:58:58.523 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.517+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3779617069}) 2024-09-16T03:58:58.523 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.517+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.523 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.517+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=486163062}) 2024-09-16T03:58:58.523 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.517+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2318916476}) 2024-09-16T03:58:58.525 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.517+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.525 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.517+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=914809455}) 2024-09-16T03:58:58.525 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.517+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4127833089}) 2024-09-16T03:58:58.525 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.517+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.525 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.517+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=914809455}) 2024-09-16T03:58:58.525 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.517+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1433974536}) 2024-09-16T03:58:58.527 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.521+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.528 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.521+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1357338180}) 2024-09-16T03:58:58.528 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.521+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2205699575}) 2024-09-16T03:58:58.528 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.521+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.528 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.521+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1357338180}) 2024-09-16T03:58:58.528 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.521+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4195111487}) 2024-09-16T03:58:58.530 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.525+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.530 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.525+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3949936533}) 2024-09-16T03:58:58.530 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.525+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2149787435}) 2024-09-16T03:58:58.530 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.525+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.530 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.525+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3949936533}) 2024-09-16T03:58:58.530 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.525+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3675871844}) 2024-09-16T03:58:58.532 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.525+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.532 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.525+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2095412964}) 2024-09-16T03:58:58.532 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.525+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2225633529}) 2024-09-16T03:58:58.532 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.525+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.532 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.525+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2095412964}) 2024-09-16T03:58:58.532 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.525+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1303000389}) 2024-09-16T03:58:58.535 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.529+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.535 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.529+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3023755601}) 2024-09-16T03:58:58.535 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.529+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3465921029}) 2024-09-16T03:58:58.535 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.529+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.535 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.529+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3023755601}) 2024-09-16T03:58:58.535 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.529+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1904042719}) 2024-09-16T03:58:58.537 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.529+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.537 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.529+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2774454858}) 2024-09-16T03:58:58.537 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.529+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=160954811}) 2024-09-16T03:58:58.537 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.529+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.538 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.529+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2774454858}) 2024-09-16T03:58:58.538 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.529+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3373189776}) 2024-09-16T03:58:58.539 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.533+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.539 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.533+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3194527583}) 2024-09-16T03:58:58.539 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.533+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2505866762}) 2024-09-16T03:58:58.539 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.533+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.539 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.533+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3194527583}) 2024-09-16T03:58:58.539 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.533+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2359784581}) 2024-09-16T03:58:58.541 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.533+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.541 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.533+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3789770189}) 2024-09-16T03:58:58.541 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.533+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1872470715}) 2024-09-16T03:58:58.541 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.533+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.541 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.533+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3789770189}) 2024-09-16T03:58:58.541 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.533+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2469671066}) 2024-09-16T03:58:58.543 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.537+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.543 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.537+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4002094723}) 2024-09-16T03:58:58.543 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.537+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1508251356}) 2024-09-16T03:58:58.543 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.537+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.543 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.537+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4002094723}) 2024-09-16T03:58:58.543 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.537+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2074247306}) 2024-09-16T03:58:58.545 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.537+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.545 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.537+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1543755753}) 2024-09-16T03:58:58.545 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.537+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1476319335}) 2024-09-16T03:58:58.545 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.537+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.545 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.537+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1543755753}) 2024-09-16T03:58:58.545 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.537+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4125403862}) 2024-09-16T03:58:58.547 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.541+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.547 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.541+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2445379002}) 2024-09-16T03:58:58.547 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.541+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=909106771}) 2024-09-16T03:58:58.547 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.541+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.547 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.541+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2445379002}) 2024-09-16T03:58:58.547 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.541+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2014781804}) 2024-09-16T03:58:58.549 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.541+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.549 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.541+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2158175345}) 2024-09-16T03:58:58.549 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.541+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4011754145}) 2024-09-16T03:58:58.549 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.541+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.549 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.541+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2158175345}) 2024-09-16T03:58:58.549 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.541+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3029762255}) 2024-09-16T03:58:58.551 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.545+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.551 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.545+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=810846559}) 2024-09-16T03:58:58.551 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.545+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=470309801}) 2024-09-16T03:58:58.551 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.545+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.551 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.545+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=810846559}) 2024-09-16T03:58:58.551 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.545+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=506004591}) 2024-09-16T03:58:58.553 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.545+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.553 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.545+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1759117474}) 2024-09-16T03:58:58.553 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.545+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3643336988}) 2024-09-16T03:58:58.553 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.545+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.553 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.545+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1759117474}) 2024-09-16T03:58:58.553 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.545+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=739842313}) 2024-09-16T03:58:58.555 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.549+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.555 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.549+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3233790265}) 2024-09-16T03:58:58.555 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.549+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2725996371}) 2024-09-16T03:58:58.555 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.549+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.555 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.549+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3233790265}) 2024-09-16T03:58:58.555 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.549+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1783258620}) 2024-09-16T03:58:58.557 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.549+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.557 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.549+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3781037682}) 2024-09-16T03:58:58.557 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.549+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2695795894}) 2024-09-16T03:58:58.557 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.549+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.557 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.549+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3781037682}) 2024-09-16T03:58:58.557 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.549+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2242980667}) 2024-09-16T03:58:58.559 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.553+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.559 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.553+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1961004931}) 2024-09-16T03:58:58.559 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.553+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4154161674}) 2024-09-16T03:58:58.559 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.553+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.559 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.553+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1961004931}) 2024-09-16T03:58:58.559 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.553+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=769567725}) 2024-09-16T03:58:58.561 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.553+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.561 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.553+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3487061863}) 2024-09-16T03:58:58.561 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.553+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2335995832}) 2024-09-16T03:58:58.561 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.553+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.561 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.553+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3487061863}) 2024-09-16T03:58:58.561 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.553+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1939189987}) 2024-09-16T03:58:58.563 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.557+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.563 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.557+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1846710030}) 2024-09-16T03:58:58.563 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.557+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3372211986}) 2024-09-16T03:58:58.563 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.557+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.563 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.557+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1846710030}) 2024-09-16T03:58:58.563 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.557+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2648907602}) 2024-09-16T03:58:58.565 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.557+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.565 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.557+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2279507180}) 2024-09-16T03:58:58.565 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.557+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2713244763}) 2024-09-16T03:58:58.565 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.557+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.565 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.557+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2279507180}) 2024-09-16T03:58:58.565 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.557+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4256575143}) 2024-09-16T03:58:58.567 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.561+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.567 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.561+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1242395864}) 2024-09-16T03:58:58.567 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.561+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1406704341}) 2024-09-16T03:58:58.567 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.561+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.567 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.561+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1242395864}) 2024-09-16T03:58:58.567 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.561+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1653540480}) 2024-09-16T03:58:58.569 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.561+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.569 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.561+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1307900716}) 2024-09-16T03:58:58.569 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.561+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2222585151}) 2024-09-16T03:58:58.569 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.561+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.569 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.561+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1307900716}) 2024-09-16T03:58:58.569 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.561+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=627752110}) 2024-09-16T03:58:58.571 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.565+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.571 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.565+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1866639714}) 2024-09-16T03:58:58.571 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.565+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3341378795}) 2024-09-16T03:58:58.571 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.565+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.571 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.565+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1866639714}) 2024-09-16T03:58:58.571 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.565+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=391535165}) 2024-09-16T03:58:58.573 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.565+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.573 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.565+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=695078485}) 2024-09-16T03:58:58.573 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.565+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=112145330}) 2024-09-16T03:58:58.573 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.565+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.573 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.565+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=695078485}) 2024-09-16T03:58:58.573 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.565+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1363407129}) 2024-09-16T03:58:58.575 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.569+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.575 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.569+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1832335774}) 2024-09-16T03:58:58.575 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.569+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=466656505}) 2024-09-16T03:58:58.575 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.569+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.575 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.569+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1832335774}) 2024-09-16T03:58:58.575 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.569+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3228090366}) 2024-09-16T03:58:58.577 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.569+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.577 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.569+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3701640179}) 2024-09-16T03:58:58.577 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.569+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2024769140}) 2024-09-16T03:58:58.577 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.569+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.577 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.569+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3701640179}) 2024-09-16T03:58:58.577 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.569+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1817886450}) 2024-09-16T03:58:58.579 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.573+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.579 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.573+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2606143910}) 2024-09-16T03:58:58.579 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.573+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3330538213}) 2024-09-16T03:58:58.579 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.573+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.579 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.573+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2606143910}) 2024-09-16T03:58:58.579 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.573+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4215245872}) 2024-09-16T03:58:58.581 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.573+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.581 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.573+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=940077540}) 2024-09-16T03:58:58.581 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.573+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=826042430}) 2024-09-16T03:58:58.581 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.573+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.581 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.573+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=940077540}) 2024-09-16T03:58:58.582 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.577+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1977941423}) 2024-09-16T03:58:58.583 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.577+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.583 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.577+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1544410461}) 2024-09-16T03:58:58.583 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.577+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=47948032}) 2024-09-16T03:58:58.583 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.577+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.583 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.577+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1544410461}) 2024-09-16T03:58:58.584 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.577+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2474699788}) 2024-09-16T03:58:58.585 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.581+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.585 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.581+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4283883593}) 2024-09-16T03:58:58.585 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.581+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2330085820}) 2024-09-16T03:58:58.586 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.581+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.586 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.581+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4283883593}) 2024-09-16T03:58:58.586 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.581+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3637000680}) 2024-09-16T03:58:58.587 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.581+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.587 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.581+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3715495431}) 2024-09-16T03:58:58.587 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.581+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2697118244}) 2024-09-16T03:58:58.588 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.581+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.588 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.581+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3715495431}) 2024-09-16T03:58:58.588 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.581+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3152438319}) 2024-09-16T03:58:58.590 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.585+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.590 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.585+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3126667829}) 2024-09-16T03:58:58.590 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.585+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3920586640}) 2024-09-16T03:58:58.590 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.585+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.590 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.585+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3126667829}) 2024-09-16T03:58:58.590 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.585+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=41258357}) 2024-09-16T03:58:58.592 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.585+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.592 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.585+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1992320799}) 2024-09-16T03:58:58.592 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.585+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2757584889}) 2024-09-16T03:58:58.592 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.585+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.592 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.585+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1992320799}) 2024-09-16T03:58:58.592 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.585+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3166944232}) 2024-09-16T03:58:58.594 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.589+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.594 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.589+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=144362141}) 2024-09-16T03:58:58.594 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.589+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3989051148}) 2024-09-16T03:58:58.594 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.589+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.594 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.589+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=144362141}) 2024-09-16T03:58:58.594 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.589+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3051035137}) 2024-09-16T03:58:58.596 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.589+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.596 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.589+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=731402096}) 2024-09-16T03:58:58.596 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.589+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3776495659}) 2024-09-16T03:58:58.596 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.589+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.596 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.589+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=731402096}) 2024-09-16T03:58:58.596 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.589+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1572636057}) 2024-09-16T03:58:58.598 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.593+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.598 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.593+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=891368414}) 2024-09-16T03:58:58.598 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.593+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2281134489}) 2024-09-16T03:58:58.598 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.593+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.598 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.593+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=891368414}) 2024-09-16T03:58:58.599 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.593+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1831143981}) 2024-09-16T03:58:58.600 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.593+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.600 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.593+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=624052039}) 2024-09-16T03:58:58.600 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.593+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2554692055}) 2024-09-16T03:58:58.601 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.593+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.601 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.593+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=624052039}) 2024-09-16T03:58:58.601 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.593+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=756429621}) 2024-09-16T03:58:58.602 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.597+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.602 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.597+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1791398542}) 2024-09-16T03:58:58.603 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.597+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3849068763}) 2024-09-16T03:58:58.603 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.597+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.603 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.597+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1791398542}) 2024-09-16T03:58:58.603 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.597+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1872260560}) 2024-09-16T03:58:58.605 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.597+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.605 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.597+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1649635263}) 2024-09-16T03:58:58.605 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.597+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2744868058}) 2024-09-16T03:58:58.605 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.597+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.605 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.597+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1649635263}) 2024-09-16T03:58:58.605 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.597+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1494329820}) 2024-09-16T03:58:58.607 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.601+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.607 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.601+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=179679452}) 2024-09-16T03:58:58.607 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.601+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=109096785}) 2024-09-16T03:58:58.607 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.601+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.607 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.601+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=179679452}) 2024-09-16T03:58:58.607 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.601+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4156540679}) 2024-09-16T03:58:58.609 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.601+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.609 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.601+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4198052218}) 2024-09-16T03:58:58.609 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.601+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1229703520}) 2024-09-16T03:58:58.609 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.601+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.609 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.601+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4198052218}) 2024-09-16T03:58:58.609 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.605+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4099005204}) 2024-09-16T03:58:58.611 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.605+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.611 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.605+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3175244394}) 2024-09-16T03:58:58.611 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.605+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3488587733}) 2024-09-16T03:58:58.611 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.605+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.612 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.605+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3175244394}) 2024-09-16T03:58:58.612 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.605+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=433525312}) 2024-09-16T03:58:58.614 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.609+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.614 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.609+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1877627489}) 2024-09-16T03:58:58.614 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.609+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2002512928}) 2024-09-16T03:58:58.614 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.609+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.614 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.609+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1877627489}) 2024-09-16T03:58:58.614 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.609+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1577501229}) 2024-09-16T03:58:58.615 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.609+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.615 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.609+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3551358601}) 2024-09-16T03:58:58.616 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.609+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4010014777}) 2024-09-16T03:58:58.616 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.609+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.616 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.609+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3551358601}) 2024-09-16T03:58:58.616 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.609+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3903632960}) 2024-09-16T03:58:58.617 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.613+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.617 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.613+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2631735273}) 2024-09-16T03:58:58.617 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.613+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4120151612}) 2024-09-16T03:58:58.618 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.613+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.618 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.613+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2631735273}) 2024-09-16T03:58:58.618 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.613+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=821417731}) 2024-09-16T03:58:58.619 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.613+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.619 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.613+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3121192149}) 2024-09-16T03:58:58.619 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.613+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1657712623}) 2024-09-16T03:58:58.619 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.613+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.619 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.613+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3121192149}) 2024-09-16T03:58:58.619 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.613+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=898952398}) 2024-09-16T03:58:58.621 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.613+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.621 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.613+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4199072541}) 2024-09-16T03:58:58.621 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.613+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=972169696}) 2024-09-16T03:58:58.621 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.613+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.621 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.613+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4199072541}) 2024-09-16T03:58:58.621 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.613+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=699127381}) 2024-09-16T03:58:58.623 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.617+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.623 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.617+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=435530757}) 2024-09-16T03:58:58.623 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.617+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2089290668}) 2024-09-16T03:58:58.623 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.617+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.623 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.617+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=435530757}) 2024-09-16T03:58:58.623 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.617+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=729815363}) 2024-09-16T03:58:58.624 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.617+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.625 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.617+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=738765193}) 2024-09-16T03:58:58.625 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.617+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3005801854}) 2024-09-16T03:58:58.625 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.617+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.625 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.617+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=738765193}) 2024-09-16T03:58:58.625 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.617+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1401632800}) 2024-09-16T03:58:58.626 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.621+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.626 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.621+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2790979947}) 2024-09-16T03:58:58.627 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.621+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=660923019}) 2024-09-16T03:58:58.627 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.621+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.627 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.621+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2790979947}) 2024-09-16T03:58:58.627 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.621+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2994584931}) 2024-09-16T03:58:58.628 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.621+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.628 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.621+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=46015191}) 2024-09-16T03:58:58.629 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.621+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3964642311}) 2024-09-16T03:58:58.629 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.621+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.629 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.621+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=46015191}) 2024-09-16T03:58:58.629 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.621+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2538112058}) 2024-09-16T03:58:58.630 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.625+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.630 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.625+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2836876553}) 2024-09-16T03:58:58.630 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.625+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=978419209}) 2024-09-16T03:58:58.631 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.625+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.631 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.625+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2836876553}) 2024-09-16T03:58:58.631 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.625+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3524197198}) 2024-09-16T03:58:58.632 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.625+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.632 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.625+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4247950915}) 2024-09-16T03:58:58.632 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.625+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2217486002}) 2024-09-16T03:58:58.633 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.625+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.633 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.625+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4247950915}) 2024-09-16T03:58:58.633 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.625+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2947233158}) 2024-09-16T03:58:58.634 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.629+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.634 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.629+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=263315293}) 2024-09-16T03:58:58.634 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.629+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3830030396}) 2024-09-16T03:58:58.634 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.629+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.634 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.629+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=263315293}) 2024-09-16T03:58:58.634 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.629+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3427636242}) 2024-09-16T03:58:58.636 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.629+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.636 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.629+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1964793897}) 2024-09-16T03:58:58.636 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.629+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=102054879}) 2024-09-16T03:58:58.636 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.629+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.636 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.629+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1964793897}) 2024-09-16T03:58:58.636 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.629+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2832035641}) 2024-09-16T03:58:58.637 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.629+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.637 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.629+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1208844401}) 2024-09-16T03:58:58.637 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.629+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3371220572}) 2024-09-16T03:58:58.637 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.629+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.637 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.629+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1208844401}) 2024-09-16T03:58:58.637 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.629+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1391633058}) 2024-09-16T03:58:58.640 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.633+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.640 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.633+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=7,mdsmap=2,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=11,monmap=2,osd_metadata=6,osd_pg_creating=1,osd_snap=36,osdmap=13} crc {logm=1824989944,mdsmap=210808725,mgr=1406439472,mgr_command_descs=2107673600,mgr_metadata=1367272202,mgrstat=260816498,mon_config_key=85057112,monmap=3140514009,osd_metadata=4182959540,osd_pg_creating=1662927693,osd_snap=2225968841,osdmap=1530663110}) 2024-09-16T03:58:58.640 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.633+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=7,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=35,osdmap=12} crc {logm=3996759702,mdsmap=4088524775,mgr=460109776,mgr_command_descs=2107673600,mgr_metadata=1367272202,mgrstat=906102532,mon_config_key=85057112,monmap=3937513981,osd_metadata=4182959540,osd_pg_creating=1662927693,osd_snap=1526327510,osdmap=2732019648}) 2024-09-16T03:58:58.640 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.633+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.640 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.633+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=7,mdsmap=2,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=11,monmap=2,osd_metadata=6,osd_pg_creating=1,osd_snap=36,osdmap=13} crc {logm=1824989944,mdsmap=210808725,mgr=1406439472,mgr_command_descs=2107673600,mgr_metadata=1367272202,mgrstat=260816498,mon_config_key=85057112,monmap=3140514009,osd_metadata=4182959540,osd_pg_creating=1662927693,osd_snap=2225968841,osdmap=1530663110}) 2024-09-16T03:58:58.641 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.633+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=6,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=11,monmap=2,osd_metadata=6,osd_pg_creating=1,osd_snap=34,osdmap=14} crc {logm=109461130,mdsmap=4088524775,mgr=3159792332,mgr_command_descs=2107673600,mgr_metadata=1367272202,mgrstat=763562916,mon_config_key=85057112,monmap=3140514009,osd_metadata=4182959540,osd_pg_creating=1662927694,osd_snap=2995134633,osdmap=3849997485}) 2024-09-16T03:58:58.642 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.637+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.642 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.637+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2280952486}) 2024-09-16T03:58:58.642 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.637+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3984993085}) 2024-09-16T03:58:58.642 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.637+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.642 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.637+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2280952486}) 2024-09-16T03:58:58.643 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.637+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1297594039}) 2024-09-16T03:58:58.646 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.641+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.646 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.641+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=406737274}) 2024-09-16T03:58:58.646 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.641+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=4260502239}) 2024-09-16T03:58:58.646 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.641+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.646 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.641+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=406737274}) 2024-09-16T03:58:58.646 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.641+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=192873054}) 2024-09-16T03:58:58.660 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.653+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:58:58.660 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.653+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=22} crc {osdmap=1489258246}) 2024-09-16T03:58:58.660 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:58:58.653+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=21} crc {osdmap=1129257237}) 2024-09-16T03:58:59.420 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T03:58:59.420 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:58:59.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.553+0000 7fbd6b798640 10 monclient: get_monmap_and_config 2024-09-16T03:58:59.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.553+0000 7fbd6b798640 10 monclient: build_initial_monmap 2024-09-16T03:58:59.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.553+0000 7fbd6b798640 10 monclient: monmap: 2024-09-16T03:58:59.559 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:58:59.559 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:58:59.559 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:58:59.557289+0000 2024-09-16T03:58:59.559 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:58:59.557289+0000 2024-09-16T03:58:59.559 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:58:59.559 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:59.559 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:58:59.559 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:58:59.560 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:58:59.560 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:58:59.560 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:58:59.560 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:59.570 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.553+0000 7fbd6b798640 1 Processor -- start 2024-09-16T03:58:59.570 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.553+0000 7fbd6b798640 1 -- start start 2024-09-16T03:58:59.570 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.553+0000 7fbd6b798640 10 monclient: init 2024-09-16T03:58:59.570 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.553+0000 7fbd6b798640 10 monclient: _reopen_session rank -1 2024-09-16T03:58:59.570 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.553+0000 7fbd6b798640 10 monclient: _add_conns ranks=[1,0,4,2,3] 2024-09-16T03:58:59.570 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.553+0000 7fbd6b798640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbd640a67e0 addr v1:172.21.15.154:6789/0 2024-09-16T03:58:59.570 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.553+0000 7fbd6b798640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbd64072560 addr v1:172.21.15.42:6789/0 2024-09-16T03:58:59.570 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.553+0000 7fbd6b798640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fbd640a72d0 0x7fbd6413da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:59.570 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.553+0000 7fbd6b798640 10 monclient(hunting): picked mon.noname-c con 0x7fbd640a72d0 addr v2:172.21.15.154:3300/0 2024-09-16T03:58:59.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.553+0000 7fbd6b798640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbd64071b80 con 0x7fbd64072560 2024-09-16T03:58:59.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.553+0000 7fbd6b798640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbd64072ed0 con 0x7fbd640a67e0 2024-09-16T03:58:59.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.553+0000 7fbd6b798640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:59.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.553+0000 7fbd6b798640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fbd64070030 con 0x7fbd640a72d0 2024-09-16T03:58:59.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.553+0000 7fbd6b798640 10 monclient(hunting): _renew_subs 2024-09-16T03:58:59.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.553+0000 7fbd6b798640 10 monclient(hunting): authenticate will time out at 1986.526863s 2024-09-16T03:58:59.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.553+0000 7fbd68d33640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fbd64072560 0x7fbd6413a2d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:37248/0 (socket says 172.21.15.42:37248) 2024-09-16T03:58:59.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.553+0000 7fbd68d33640 1 -- 172.21.15.42:0/488013748 learned_addr learned my addr 172.21.15.42:0/488013748 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:58:59.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.553+0000 7fbd5bfff640 1 -- 172.21.15.42:0/488013748 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3914675259 0 0) 0x7fbd64071b80 con 0x7fbd64072560 2024-09-16T03:58:59.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.553+0000 7fbd5bfff640 10 monclient(hunting): _negotiate 2024-09-16T03:58:59.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.553+0000 7fbd5bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:59.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.553+0000 7fbd5bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:58:59.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.553+0000 7fbd5bfff640 10 monclient(hunting): my global_id is 825075 2024-09-16T03:58:59.572 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.553+0000 7fbd5bfff640 1 -- 172.21.15.42:0/488013748 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbd50003530 con 0x7fbd64072560 2024-09-16T03:58:59.572 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.553+0000 7fbd69d35640 1 --2- 172.21.15.42:0/488013748 >> v2:172.21.15.154:3300/0 conn(0x7fbd640a72d0 0x7fbd6413da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:59.572 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.553+0000 7fbd69d35640 10 monclient(hunting): get_auth_request con 0x7fbd640a72d0 auth_method 0 2024-09-16T03:58:59.572 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.553+0000 7fbd69d35640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:59.572 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.553+0000 7fbd69d35640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:59.572 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.553+0000 7fbd69d35640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:58:59.572 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.553+0000 7fbd69d35640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:58:59.572 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.553+0000 7fbd69d35640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:58:59.572 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.553+0000 7fbd69d35640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:58:59.572 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.553+0000 7fbd5bfff640 1 -- 172.21.15.42:0/488013748 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2541799502 0 0) 0x7fbd50003530 con 0x7fbd64072560 2024-09-16T03:58:59.572 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.553+0000 7fbd5bfff640 1 -- 172.21.15.42:0/488013748 >> v2:172.21.15.154:3300/0 conn(0x7fbd640a72d0 msgr2=0x7fbd6413da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:58:59.572 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.553+0000 7fbd5bfff640 1 --2- 172.21.15.42:0/488013748 >> v2:172.21.15.154:3300/0 conn(0x7fbd640a72d0 0x7fbd6413da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:59.573 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.553+0000 7fbd5bfff640 1 -- 172.21.15.42:0/488013748 >> v1:172.21.15.154:6789/0 conn(0x7fbd640a67e0 legacy=0x7fbd640a6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:59.573 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.553+0000 7fbd5bfff640 10 monclient: _finish_hunting 0 2024-09-16T03:58:59.573 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.553+0000 7fbd5bfff640 1 monclient: found mon.noname-a-legacy 2024-09-16T03:58:59.573 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.553+0000 7fbd5bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:58:59.573 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.553+0000 7fbd5bfff640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.42:6789/0 2024-09-16T03:58:59.573 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.553+0000 7fbd5bfff640 1 -- 172.21.15.42:0/488013748 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd640712e0 con 0x7fbd64072560 2024-09-16T03:58:59.573 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.553+0000 7fbd5bfff640 10 monclient: _finish_auth 0 2024-09-16T03:58:59.573 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.553+0000 7fbd5bfff640 10 monclient: _check_auth_tickets 2024-09-16T03:58:59.573 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.553+0000 7fbd5bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:58:59.573 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.553+0000 7fbd5bfff640 1 -- 172.21.15.42:0/488013748 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2177413734 0 0) 0x7fbd54001680 con 0x7fbd64072560 2024-09-16T03:58:59.573 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.553+0000 7fbd5bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:58:59.573 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.553+0000 7fbd5bfff640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T03:58:59.573 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.553+0000 7fbd5bfff640 10 monclient: dump: 2024-09-16T03:58:59.573 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:58:59.574 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:58:59.574 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:59.574 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:59.574 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:58:59.574 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:59.574 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:58:59.574 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:58:59.574 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:58:59.574 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:59.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.553+0000 7fbd5bfff640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] but i'm connected to v1:172.21.15.42:6789/0, reconnecting 2024-09-16T03:58:59.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.553+0000 7fbd5bfff640 10 monclient: _reopen_session rank -1 2024-09-16T03:58:59.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.553+0000 7fbd5bfff640 1 -- 172.21.15.42:0/488013748 >> v1:172.21.15.42:6789/0 conn(0x7fbd64072560 legacy=0x7fbd6413a2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:59.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.553+0000 7fbd5bfff640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:58:59.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.553+0000 7fbd5bfff640 1 --2- 172.21.15.42:0/488013748 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbd500039c0 0x7fbd50003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:59.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.553+0000 7fbd5bfff640 10 monclient(hunting): picked mon.c con 0x7fbd500039c0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:58:59.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.553+0000 7fbd5bfff640 1 --2- 172.21.15.42:0/488013748 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbd50004810 0x7fbd50004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:59.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.553+0000 7fbd5bfff640 10 monclient(hunting): picked mon.b con 0x7fbd50004810 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:58:59.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.553+0000 7fbd5bfff640 1 --2- 172.21.15.42:0/488013748 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbd500051d0 0x7fbd5000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:59.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.553+0000 7fbd5bfff640 10 monclient(hunting): picked mon.a con 0x7fbd500051d0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:58:59.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.553+0000 7fbd5bfff640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:59.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.553+0000 7fbd5bfff640 1 -- 172.21.15.42:0/488013748 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fbd64070030 con 0x7fbd500051d0 2024-09-16T03:58:59.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.553+0000 7fbd5bfff640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:59.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.553+0000 7fbd5bfff640 1 -- 172.21.15.42:0/488013748 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fbd5000c120 con 0x7fbd50004810 2024-09-16T03:58:59.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.553+0000 7fbd5bfff640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:59.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.553+0000 7fbd5bfff640 1 -- 172.21.15.42:0/488013748 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fbd5000c320 con 0x7fbd500039c0 2024-09-16T03:58:59.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.553+0000 7fbd5bfff640 10 monclient(hunting): _renew_subs 2024-09-16T03:58:59.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.553+0000 7fbd5bfff640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:58:59.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.553+0000 7fbd69d35640 1 --2- 172.21.15.42:0/488013748 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbd500051d0 0x7fbd5000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:59.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.553+0000 7fbd69534640 1 --2- 172.21.15.42:0/488013748 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbd500039c0 0x7fbd50003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:59.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.553+0000 7fbd69d35640 10 monclient(hunting): get_auth_request con 0x7fbd500051d0 auth_method 0 2024-09-16T03:58:59.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.553+0000 7fbd69d35640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:59.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.553+0000 7fbd69d35640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:59.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.553+0000 7fbd69d35640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:58:59.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.557+0000 7fbd69534640 10 monclient(hunting): get_auth_request con 0x7fbd500039c0 auth_method 0 2024-09-16T03:58:59.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.557+0000 7fbd69534640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:59.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.557+0000 7fbd69534640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:59.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.557+0000 7fbd69534640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:58:59.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.557+0000 7fbd69d35640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:58:59.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.557+0000 7fbd69d35640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:58:59.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.557+0000 7fbd69d35640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T03:58:59.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.557+0000 7fbd69534640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:58:59.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.557+0000 7fbd69534640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:58:59.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.557+0000 7fbd69534640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T03:58:59.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.557+0000 7fbd68d33640 1 --2- 172.21.15.42:0/488013748 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbd50004810 0x7fbd50004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:59.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.557+0000 7fbd68d33640 10 monclient(hunting): get_auth_request con 0x7fbd50004810 auth_method 0 2024-09-16T03:58:59.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.557+0000 7fbd68d33640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:59.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.557+0000 7fbd68d33640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:59.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.557+0000 7fbd68d33640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:58:59.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.557+0000 7fbd69d35640 10 monclient(hunting): handle_auth_done global_id 825075 payload 293 2024-09-16T03:58:59.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.557+0000 7fbd69d35640 1 -- 172.21.15.42:0/488013748 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbd500039c0 msgr2=0x7fbd50003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:58:59.578 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.557+0000 7fbd69d35640 1 --2- 172.21.15.42:0/488013748 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbd500039c0 0x7fbd50003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:59.578 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.557+0000 7fbd69d35640 1 -- 172.21.15.42:0/488013748 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbd50004810 msgr2=0x7fbd50004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:58:59.578 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.557+0000 7fbd69d35640 1 --2- 172.21.15.42:0/488013748 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbd50004810 0x7fbd50004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:59.578 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.557+0000 7fbd69d35640 10 monclient: _finish_hunting 0 2024-09-16T03:58:59.578 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.557+0000 7fbd69d35640 1 monclient: found mon.a 2024-09-16T03:58:59.578 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.557+0000 7fbd69d35640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:58:59.578 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.557+0000 7fbd69d35640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:58:59.578 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.557+0000 7fbd69d35640 1 -- 172.21.15.42:0/488013748 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd50003530 con 0x7fbd500051d0 2024-09-16T03:58:59.578 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.557+0000 7fbd69d35640 10 monclient: _finish_auth 0 2024-09-16T03:58:59.578 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.557+0000 7fbd69d35640 10 monclient: _check_auth_tickets 2024-09-16T03:58:59.578 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.557+0000 7fbd69d35640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:58:59.578 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.557+0000 7fbd6b798640 5 monclient: authenticate success, global_id 825075 2024-09-16T03:58:59.578 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.557+0000 7fbd6b798640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:58:59.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.557+0000 7fbd69534640 1 --2- 172.21.15.42:0/488013748 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbd500039c0 0x7fbd50003da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:58:59.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.557+0000 7fbd69d35640 1 -- 172.21.15.42:0/488013748 reap_dead start 2024-09-16T03:58:59.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.557+0000 7fbd69d35640 1 --2- 172.21.15.42:0/488013748 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbd500051d0 0x7fbd5000b9d0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fbd5c009fa0 tx=0x7fbd5c01cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=268f4ae678c086b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:58:59.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.557+0000 7fbd5bfff640 1 -- 172.21.15.42:0/488013748 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd5c0b2070 con 0x7fbd500051d0 2024-09-16T03:58:59.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.557+0000 7fbd5bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:58:59.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.557+0000 7fbd5bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:58:59.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.557+0000 7fbd5bfff640 10 monclient: dump: 2024-09-16T03:58:59.579 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:58:59.579 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:58:59.579 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:59.579 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:59.579 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:58:59.579 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:59.580 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:58:59.580 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:58:59.580 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:58:59.580 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:59.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.557+0000 7fbd5bfff640 1 -- 172.21.15.42:0/488013748 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbd5c0a5d30 con 0x7fbd500051d0 2024-09-16T03:58:59.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.557+0000 7fbd6b798640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:58:59.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.557+0000 7fbd5bfff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:58:59.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.557+0000 7fbd6b798640 10 monclient: get_monmap_and_config success 2024-09-16T03:58:59.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.557+0000 7fbd5bfff640 1 -- 172.21.15.42:0/488013748 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd60001460 con 0x7fbd500051d0 2024-09-16T03:58:59.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.557+0000 7fbd5bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:58:59.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.557+0000 7fbd6b798640 4 set_mon_vals no callback set 2024-09-16T03:58:59.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.557+0000 7fbd5bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:58:59.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.557+0000 7fbd5bfff640 10 monclient: dump: 2024-09-16T03:58:59.580 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:58:59.581 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:58:59.581 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:59.581 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:59.581 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:58:59.581 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:59.581 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:58:59.581 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:58:59.581 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:58:59.581 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:59.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.557+0000 7fbd6b798640 10 monclient: shutdown 2024-09-16T03:58:59.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.557+0000 7fbd6b798640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:58:59.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.557+0000 7fbd6b798640 1 -- 172.21.15.42:0/488013748 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbd500051d0 msgr2=0x7fbd5000b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:59.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.557+0000 7fbd6b798640 1 --2- 172.21.15.42:0/488013748 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbd500051d0 0x7fbd5000b9d0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fbd5c009fa0 tx=0x7fbd5c01cc60 comp rx=0 tx=0).stop 2024-09-16T03:58:59.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.557+0000 7fbd6b798640 1 -- 172.21.15.42:0/488013748 shutdown_connections 2024-09-16T03:58:59.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.557+0000 7fbd6b798640 1 --2- 172.21.15.42:0/488013748 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbd500051d0 0x7fbd5000b9d0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:59.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.557+0000 7fbd6b798640 1 -- 172.21.15.42:0/488013748 >> 172.21.15.42:0/488013748 conn(0x7fbd640a7ed0 msgr2=0x7fbd640a43d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:58:59.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.557+0000 7fbd6b798640 1 -- 172.21.15.42:0/488013748 shutdown_connections 2024-09-16T03:58:59.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.557+0000 7fbd6b798640 1 -- 172.21.15.42:0/488013748 wait complete. 2024-09-16T03:58:59.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.557+0000 7fbd6b798640 10 monclient: build_initial_monmap 2024-09-16T03:58:59.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.557+0000 7fbd6b798640 10 monclient: monmap: 2024-09-16T03:58:59.582 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:58:59.582 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:58:59.582 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:58:59.582 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:58:59.582 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:58:59.582 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:59.582 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:58:59.583 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:58:59.583 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:58:59.583 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:59.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.561+0000 7fbd6b798640 1 librados: starting msgr at 2024-09-16T03:58:59.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.561+0000 7fbd6b798640 1 librados: starting objecter 2024-09-16T03:58:59.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.561+0000 7fbd6b798640 1 Processor -- start 2024-09-16T03:58:59.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.561+0000 7fbd6b798640 1 -- start start 2024-09-16T03:58:59.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.561+0000 7fbd6b798640 1 librados: setting wanted keys 2024-09-16T03:58:59.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.561+0000 7fbd6b798640 1 librados: calling monclient init 2024-09-16T03:58:59.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.561+0000 7fbd6b798640 10 monclient: init 2024-09-16T03:58:59.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.561+0000 7fbd6b798640 10 monclient: _reopen_session rank -1 2024-09-16T03:58:59.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.561+0000 7fbd6b798640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:58:59.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.561+0000 7fbd6b798640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbd500051d0 0x7fbd641cba60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:59.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.561+0000 7fbd6b798640 10 monclient(hunting): picked mon.noname-b con 0x7fbd500051d0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:58:59.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.561+0000 7fbd6b798640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbd64134640 0x7fbd6406d920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:59.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.561+0000 7fbd6b798640 10 monclient(hunting): picked mon.noname-a con 0x7fbd64134640 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:58:59.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.561+0000 7fbd6b798640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbd64134a40 0x7fbd6406de60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:59.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.561+0000 7fbd6b798640 10 monclient(hunting): picked mon.noname-c con 0x7fbd64134a40 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:58:59.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.561+0000 7fbd6b798640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:59.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.561+0000 7fbd6b798640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fbd640a4d40 con 0x7fbd64134640 2024-09-16T03:58:59.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.561+0000 7fbd6b798640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:59.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.561+0000 7fbd6b798640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fbd640702b0 con 0x7fbd500051d0 2024-09-16T03:58:59.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.561+0000 7fbd6b798640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:59.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.561+0000 7fbd6b798640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fbd640a4ec0 con 0x7fbd64134a40 2024-09-16T03:58:59.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.561+0000 7fbd6b798640 10 monclient(hunting): _renew_subs 2024-09-16T03:58:59.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.561+0000 7fbd6b798640 10 monclient(hunting): authenticate will time out at 1986.532867s 2024-09-16T03:58:59.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.561+0000 7fbd68d33640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbd64134640 0x7fbd6406d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:59.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.561+0000 7fbd69d35640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbd64134a40 0x7fbd6406de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:59.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.561+0000 7fbd68d33640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbd64134640 0x7fbd6406d920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:33514/0 (socket says 172.21.15.42:33514) 2024-09-16T03:58:59.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.561+0000 7fbd68d33640 1 -- 172.21.15.42:0/1679192868 learned_addr learned my addr 172.21.15.42:0/1679192868 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:58:59.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.561+0000 7fbd69d35640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbd64134a40 0x7fbd6406de60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:57462/0 (socket says 172.21.15.42:57462) 2024-09-16T03:58:59.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.561+0000 7fbd68d33640 10 monclient(hunting): get_auth_request con 0x7fbd64134640 auth_method 0 2024-09-16T03:58:59.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.561+0000 7fbd68d33640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:59.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.561+0000 7fbd68d33640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:59.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.561+0000 7fbd68d33640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:58:59.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.561+0000 7fbd69d35640 10 monclient(hunting): get_auth_request con 0x7fbd64134a40 auth_method 0 2024-09-16T03:58:59.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.561+0000 7fbd69d35640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:59.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.561+0000 7fbd69d35640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:59.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.561+0000 7fbd69d35640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:58:59.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.561+0000 7fbd69d35640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:58:59.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.561+0000 7fbd69d35640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:58:59.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.561+0000 7fbd69d35640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:58:59.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.561+0000 7fbd69534640 1 --2- 172.21.15.42:0/1679192868 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbd500051d0 0x7fbd641cba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:59.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.561+0000 7fbd68d33640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:58:59.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.561+0000 7fbd68d33640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:58:59.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.561+0000 7fbd68d33640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:58:59.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.561+0000 7fbd69534640 10 monclient(hunting): get_auth_request con 0x7fbd500051d0 auth_method 0 2024-09-16T03:58:59.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.561+0000 7fbd69534640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:59.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.561+0000 7fbd69534640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:59.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.561+0000 7fbd69534640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:58:59.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.561+0000 7fbd69534640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:58:59.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.561+0000 7fbd69534640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:58:59.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.561+0000 7fbd69534640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:58:59.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.561+0000 7fbd69d35640 10 monclient(hunting): handle_auth_done global_id 924134 payload 824 2024-09-16T03:58:59.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.561+0000 7fbd69d35640 1 -- 172.21.15.42:0/1679192868 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbd500051d0 msgr2=0x7fbd641cba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:59.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.561+0000 7fbd69d35640 1 --2- 172.21.15.42:0/1679192868 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbd500051d0 0x7fbd641cba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:59.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.561+0000 7fbd69d35640 1 -- 172.21.15.42:0/1679192868 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbd64134640 msgr2=0x7fbd6406d920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:59.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.561+0000 7fbd69d35640 1 --2- 172.21.15.42:0/1679192868 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbd64134640 0x7fbd6406d920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:59.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.561+0000 7fbd69d35640 10 monclient: _finish_hunting 0 2024-09-16T03:58:59.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.561+0000 7fbd69d35640 1 monclient: found mon.noname-c 2024-09-16T03:58:59.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.561+0000 7fbd69d35640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:58:59.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.561+0000 7fbd69d35640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:58:59.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.561+0000 7fbd69d35640 1 -- 172.21.15.42:0/1679192868 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd64123a80 con 0x7fbd64134a40 2024-09-16T03:58:59.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.561+0000 7fbd68d33640 1 --2- 172.21.15.42:0/1679192868 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbd64134640 0x7fbd6406d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:58:59.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.561+0000 7fbd69d35640 1 --2- 172.21.15.42:0/1679192868 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbd64134a40 0x7fbd6406de60 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fbd640a4360 tx=0x7fbd5c009750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:58:59.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.561+0000 7fbd59ffb640 10 client.?.objecter ms_handle_connect 0x7fbd64134a40 2024-09-16T03:58:59.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.561+0000 7fbd59ffb640 10 client.?.objecter resend_mon_ops 2024-09-16T03:58:59.588 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.561+0000 7fbd59ffb640 1 -- 172.21.15.42:0/1679192868 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd5c0b2040 con 0x7fbd64134a40 2024-09-16T03:58:59.588 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.561+0000 7fbd59ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:58:59.588 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.561+0000 7fbd59ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:58:59.588 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.561+0000 7fbd59ffb640 10 monclient: dump: 2024-09-16T03:58:59.588 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:58:59.588 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:58:59.588 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:59.588 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:59.588 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:58:59.588 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:59.588 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:58:59.588 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:58:59.588 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:58:59.589 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:59.589 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.561+0000 7fbd59ffb640 10 monclient: _finish_auth 0 2024-09-16T03:58:59.589 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.561+0000 7fbd59ffb640 10 monclient: _check_auth_tickets 2024-09-16T03:58:59.589 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.561+0000 7fbd59ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:58:59.589 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.561+0000 7fbd59ffb640 1 -- 172.21.15.42:0/1679192868 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbd5c0bb070 con 0x7fbd64134a40 2024-09-16T03:58:59.589 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.561+0000 7fbd6b798640 5 monclient: authenticate success, global_id 924134 2024-09-16T03:58:59.589 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.561+0000 7fbd59ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T03:58:59.589 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.561+0000 7fbd59ffb640 1 -- 172.21.15.42:0/1679192868 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd5c009bf0 con 0x7fbd64134a40 2024-09-16T03:58:59.589 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.561+0000 7fbd59ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:58:59.589 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.561+0000 7fbd59ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:58:59.589 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.561+0000 7fbd5b7fe640 4 set_mon_vals no callback set 2024-09-16T03:58:59.589 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.561+0000 7fbd59ffb640 10 monclient: dump: 2024-09-16T03:58:59.589 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:58:59.590 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:58:59.590 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:59.590 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:59.590 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:58:59.590 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:59.590 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:58:59.590 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:58:59.590 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:58:59.590 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:59.590 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.561+0000 7fbd6b798640 10 monclient: _renew_subs 2024-09-16T03:58:59.590 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.561+0000 7fbd6b798640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:58:59.590 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.561+0000 7fbd6b798640 1 -- 172.21.15.42:0/1679192868 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd64123d60 con 0x7fbd64134a40 2024-09-16T03:58:59.590 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.561+0000 7fbd6b798640 10 client.924134.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:58:59.590 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.561+0000 7fbd6b798640 10 monclient: _renew_subs 2024-09-16T03:58:59.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.561+0000 7fbd6b798640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:58:59.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.561+0000 7fbd6b798640 1 -- 172.21.15.42:0/1679192868 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd64073970 con 0x7fbd64134a40 2024-09-16T03:58:59.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.561+0000 7fbd6b798640 1 librados: init done 2024-09-16T03:58:59.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.561+0000 7fbd59ffb640 1 -- 172.21.15.42:0/1679192868 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fbd5c0c0460 con 0x7fbd64134a40 2024-09-16T03:58:59.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.561+0000 7fbd6b798640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:58:59.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.561+0000 7fbd6b798640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:58:59.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.561+0000 7fbd6b798640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:58:59.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.561+0000 7fbd6b798640 1 -- 172.21.15.42:0/1679192868 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbd640a6bc0 con 0x7fbd64134a40 2024-09-16T03:58:59.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.561+0000 7fbd59ffb640 1 --2- 172.21.15.42:0/1679192868 >> v2:172.21.15.154:6812/254845342 conn(0x7fbd48037810 0x7fbd48039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:59.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.561+0000 7fbd59ffb640 1 -- 172.21.15.42:0/1679192868 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(516..516 src has 366..516) ==== 5258+0+0 (secure 0 0 0) 0x7fbd5c0e3630 con 0x7fbd64134a40 2024-09-16T03:58:59.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.561+0000 7fbd59ffb640 10 client.924134.objecter ms_dispatch 0x7fbd64000ba0 osd_map(516..516 src has 366..516) 2024-09-16T03:58:59.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.561+0000 7fbd59ffb640 3 client.924134.objecter handle_osd_map got epochs [516,516] > 0 2024-09-16T03:58:59.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.561+0000 7fbd59ffb640 3 client.924134.objecter handle_osd_map decoding full epoch 516 2024-09-16T03:58:59.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.561+0000 7fbd59ffb640 20 client.924134.objecter dump_active .. 0 homeless 2024-09-16T03:58:59.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.561+0000 7fbd69534640 1 --2- 172.21.15.42:0/1679192868 >> v2:172.21.15.154:6812/254845342 conn(0x7fbd48037810 0x7fbd48039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:59.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.561+0000 7fbd69534640 10 monclient: get_auth_request con 0x7fbd48037810 auth_method 0 2024-09-16T03:58:59.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.565+0000 7fbd59ffb640 10 client.924134.objecter ms_handle_connect 0x7fbd48037810 2024-09-16T03:58:59.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.565+0000 7fbd69534640 1 --2- 172.21.15.42:0/1679192868 >> v2:172.21.15.154:6812/254845342 conn(0x7fbd48037810 0x7fbd48039cd0 secure :-1 s=READY pgs=3717 cs=0 l=1 rev1=1 crypto rx=0x7fbd600014a0 tx=0x7fbd60001430 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:58:59.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.569+0000 7fbd59ffb640 1 -- 172.21.15.42:0/1679192868 <== mon.2 v2:172.21.15.42:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7fbd5c016970 con 0x7fbd64134a40 2024-09-16T03:58:59.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.569+0000 7fbd59ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:58:59.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.569+0000 7fbd59ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:58:59.801 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.793+0000 7fbd6b798640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:58:59.801 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.793+0000 7fbd6b798640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:58:59.801 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.793+0000 7fbd6b798640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:58:59.801 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.793+0000 7fbd6b798640 1 -- 172.21.15.42:0/1679192868 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fbd64134f60 con 0x7fbd64134a40 2024-09-16T03:58:59.802 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.797+0000 7fbd59ffb640 1 -- 172.21.15.42:0/1679192868 <== mon.2 v2:172.21.15.42:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7fbd5c0e56e0 con 0x7fbd64134a40 2024-09-16T03:58:59.802 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.797+0000 7fbd59ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:58:59.802 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.797+0000 7fbd59ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:58:59.802 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":154,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":4,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:58:59.804 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.797+0000 7fbd6b798640 10 librados: watch_flush enter 2024-09-16T03:58:59.804 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.797+0000 7fbd6b798640 10 librados: watch_flush exit 2024-09-16T03:58:59.804 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.797+0000 7fbd6b798640 20 client.924134.objecter shutdown clearing up homeless session... 2024-09-16T03:58:59.804 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.797+0000 7fbd6b798640 10 client.924134.objecter successfully canceled tick 2024-09-16T03:58:59.804 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.797+0000 7fbd6b798640 1 -- 172.21.15.42:0/1679192868 >> v2:172.21.15.154:6812/254845342 conn(0x7fbd48037810 msgr2=0x7fbd48039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:59.804 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.797+0000 7fbd6b798640 1 --2- 172.21.15.42:0/1679192868 >> v2:172.21.15.154:6812/254845342 conn(0x7fbd48037810 0x7fbd48039cd0 secure :-1 s=READY pgs=3717 cs=0 l=1 rev1=1 crypto rx=0x7fbd600014a0 tx=0x7fbd60001430 comp rx=0 tx=0).stop 2024-09-16T03:58:59.804 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.797+0000 7fbd6b798640 10 monclient: shutdown 2024-09-16T03:58:59.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.797+0000 7fbd6b798640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:58:59.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.797+0000 7fbd6b798640 1 -- 172.21.15.42:0/1679192868 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbd64134a40 msgr2=0x7fbd6406de60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:59.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.797+0000 7fbd6b798640 1 --2- 172.21.15.42:0/1679192868 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbd64134a40 0x7fbd6406de60 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fbd640a4360 tx=0x7fbd5c009750 comp rx=0 tx=0).stop 2024-09-16T03:58:59.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.797+0000 7fbd6b798640 1 -- 172.21.15.42:0/1679192868 shutdown_connections 2024-09-16T03:58:59.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.797+0000 7fbd6b798640 1 --2- 172.21.15.42:0/1679192868 >> v2:172.21.15.154:6812/254845342 conn(0x7fbd48037810 0x7fbd48039cd0 unknown :-1 s=CLOSED pgs=3717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:59.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.797+0000 7fbd6b798640 1 --2- 172.21.15.42:0/1679192868 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbd64134a40 0x7fbd6406de60 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:59.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.797+0000 7fbd6b798640 1 --2- 172.21.15.42:0/1679192868 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbd64134640 0x7fbd6406d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:59.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.797+0000 7fbd6b798640 1 --2- 172.21.15.42:0/1679192868 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbd500051d0 0x7fbd641cba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:59.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.797+0000 7fbd6b798640 1 -- 172.21.15.42:0/1679192868 >> 172.21.15.42:0/1679192868 conn(0x7fbd640a7ed0 msgr2=0x7fbd640a2340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:58:59.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.797+0000 7fbd6b798640 1 -- 172.21.15.42:0/1679192868 shutdown_connections 2024-09-16T03:58:59.806 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.797+0000 7fbd6b798640 1 -- 172.21.15.42:0/1679192868 wait complete. 2024-09-16T03:58:59.806 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.797+0000 7fbd6b798640 1 librados: shutdown 2024-09-16T03:58:59.821 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T03:58:59.821 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T03:58:59.821 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T03:58:59.822 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T03:58:59.941 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.937+0000 7f9f1a21c640 10 monclient: get_monmap_and_config 2024-09-16T03:58:59.941 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.937+0000 7f9f1a21c640 10 monclient: build_initial_monmap 2024-09-16T03:58:59.942 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.937+0000 7f9f1a21c640 10 monclient: monmap: 2024-09-16T03:58:59.942 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:58:59.942 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:58:59.942 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:58:59.940293+0000 2024-09-16T03:58:59.942 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:58:59.940293+0000 2024-09-16T03:58:59.942 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:58:59.942 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:59.942 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:58:59.942 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:58:59.942 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:58:59.942 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:58:59.942 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:58:59.942 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:59.945 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.937+0000 7f9f1a21c640 1 Processor -- start 2024-09-16T03:58:59.945 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.937+0000 7f9f1a21c640 1 -- start start 2024-09-16T03:58:59.945 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.937+0000 7f9f1a21c640 10 monclient: init 2024-09-16T03:58:59.945 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.937+0000 7f9f1a21c640 10 monclient: _reopen_session rank -1 2024-09-16T03:58:59.945 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.937+0000 7f9f1a21c640 10 monclient: _add_conns ranks=[2,0,4,3,1] 2024-09-16T03:58:59.945 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.937+0000 7f9f1a21c640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9f14130a40 0x7f9f140a2250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:59.945 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.937+0000 7f9f1a21c640 10 monclient(hunting): picked mon.noname-b con 0x7f9f14130a40 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:58:59.945 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.937+0000 7f9f1a21c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9f14130e20 addr v1:172.21.15.42:6789/0 2024-09-16T03:58:59.946 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.937+0000 7f9f1a21c640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f9f14131260 0x7f9f14072b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:59.946 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.937+0000 7f9f1a21c640 10 monclient(hunting): picked mon.noname-c con 0x7f9f14131260 addr v2:172.21.15.154:3300/0 2024-09-16T03:58:59.946 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.937+0000 7f9f1a21c640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9f140a0fd0 con 0x7f9f14130e20 2024-09-16T03:58:59.946 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.937+0000 7f9f1a21c640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:59.946 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.937+0000 7f9f1a21c640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f9f140a12b0 con 0x7f9f14131260 2024-09-16T03:58:59.946 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.937+0000 7f9f1a21c640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:59.946 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.937+0000 7f9f1a21c640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f9f140a1590 con 0x7f9f14130a40 2024-09-16T03:58:59.946 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.937+0000 7f9f1a21c640 10 monclient(hunting): _renew_subs 2024-09-16T03:58:59.946 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.937+0000 7f9f1a21c640 10 monclient(hunting): authenticate will time out at 1986.909913s 2024-09-16T03:58:59.946 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.937+0000 7f9f12ffd640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f9f14130e20 0x7f9f14072440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:37260/0 (socket says 172.21.15.42:37260) 2024-09-16T03:58:59.946 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.937+0000 7f9f137fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9f14130a40 0x7f9f140a2250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:59.946 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.937+0000 7f9f12ffd640 1 -- 172.21.15.42:0/1318803270 learned_addr learned my addr 172.21.15.42:0/1318803270 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:58:59.946 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.937+0000 7f9f137fe640 10 monclient(hunting): get_auth_request con 0x7f9f14130a40 auth_method 0 2024-09-16T03:58:59.946 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.937+0000 7f9f137fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:59.947 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.937+0000 7f9f137fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:59.947 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.937+0000 7f9f137fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:58:59.947 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.937+0000 7f9f137fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:58:59.947 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.937+0000 7f9f137fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:58:59.947 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.937+0000 7f9f137fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:58:59.947 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.937+0000 7f9f13fff640 1 --2- 172.21.15.42:0/1318803270 >> v2:172.21.15.154:3300/0 conn(0x7f9f14131260 0x7f9f14072b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:59.947 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.937+0000 7f9f13fff640 10 monclient(hunting): get_auth_request con 0x7f9f14131260 auth_method 0 2024-09-16T03:58:59.947 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.937+0000 7f9f13fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:59.947 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.937+0000 7f9f13fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:59.947 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.937+0000 7f9f13fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:58:59.947 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.937+0000 7f9f127fc640 1 -- 172.21.15.42:0/1318803270 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2860462531 0 0) 0x7f9f140a0fd0 con 0x7f9f14130e20 2024-09-16T03:58:59.947 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.937+0000 7f9f127fc640 10 monclient(hunting): _negotiate 2024-09-16T03:58:59.947 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.937+0000 7f9f127fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:59.948 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.937+0000 7f9f127fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:58:59.948 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.937+0000 7f9f127fc640 10 monclient(hunting): my global_id is 825081 2024-09-16T03:58:59.948 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.937+0000 7f9f127fc640 1 -- 172.21.15.42:0/1318803270 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9efc003530 con 0x7f9f14130e20 2024-09-16T03:58:59.948 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.937+0000 7f9f137fe640 10 monclient(hunting): handle_auth_done global_id 924137 payload 274 2024-09-16T03:58:59.948 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.937+0000 7f9f137fe640 1 -- 172.21.15.42:0/1318803270 >> v2:172.21.15.154:3300/0 conn(0x7f9f14131260 msgr2=0x7f9f14072b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:58:59.948 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.937+0000 7f9f137fe640 1 --2- 172.21.15.42:0/1318803270 >> v2:172.21.15.154:3300/0 conn(0x7f9f14131260 0x7f9f14072b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:59.948 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.937+0000 7f9f137fe640 1 -- 172.21.15.42:0/1318803270 >> v1:172.21.15.42:6789/0 conn(0x7f9f14130e20 legacy=0x7f9f14072440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:59.948 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.937+0000 7f9f137fe640 10 monclient: _finish_hunting 0 2024-09-16T03:58:59.948 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.937+0000 7f9f137fe640 1 monclient: found mon.noname-b 2024-09-16T03:58:59.948 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.937+0000 7f9f137fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:58:59.948 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.937+0000 7f9f137fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T03:58:59.948 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.937+0000 7f9f137fe640 1 -- 172.21.15.42:0/1318803270 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f1409ccc0 con 0x7f9f14130a40 2024-09-16T03:58:59.948 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.937+0000 7f9f13fff640 1 --2- 172.21.15.42:0/1318803270 >> v2:172.21.15.154:3300/0 conn(0x7f9f14131260 0x7f9f14072b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T03:58:59.960 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.941+0000 7f9f137fe640 1 --2- 172.21.15.42:0/1318803270 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9f14130a40 0x7f9f140a2250 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f9f040097c0 tx=0x7f9f0401c360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e24aba25a6deb998 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:58:59.960 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.941+0000 7f9f127fc640 1 -- 172.21.15.42:0/1318803270 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f040b2070 con 0x7f9f14130a40 2024-09-16T03:58:59.960 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.941+0000 7f9f127fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:58:59.960 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.941+0000 7f9f127fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:58:59.960 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.941+0000 7f9f127fc640 10 monclient: dump: 2024-09-16T03:58:59.961 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:58:59.961 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:58:59.961 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:59.961 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:59.961 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:58:59.961 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:59.961 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:58:59.961 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:58:59.961 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:58:59.961 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:59.961 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.941+0000 7f9f127fc640 10 monclient: _finish_auth 0 2024-09-16T03:58:59.961 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.941+0000 7f9f127fc640 10 monclient: _check_auth_tickets 2024-09-16T03:58:59.961 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.941+0000 7f9f127fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:58:59.961 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.941+0000 7f9f1a21c640 5 monclient: authenticate success, global_id 924137 2024-09-16T03:58:59.962 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.941+0000 7f9f1a21c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:58:59.962 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.941+0000 7f9f127fc640 1 -- 172.21.15.42:0/1318803270 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f140732a0 con 0x7f9f14130a40 2024-09-16T03:58:59.962 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.941+0000 7f9f127fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:58:59.962 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.941+0000 7f9f1a21c640 10 monclient: get_monmap_and_config success 2024-09-16T03:58:59.962 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.941+0000 7f9f127fc640 1 -- 172.21.15.42:0/1318803270 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f040ad880 con 0x7f9f14130a40 2024-09-16T03:58:59.962 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.941+0000 7f9f1a21c640 4 set_mon_vals no callback set 2024-09-16T03:58:59.962 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.941+0000 7f9f127fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:58:59.962 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.941+0000 7f9f127fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:58:59.962 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.941+0000 7f9f127fc640 10 monclient: dump: 2024-09-16T03:58:59.962 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:58:59.962 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:58:59.962 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:59.962 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:59.963 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:58:59.963 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:59.963 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:58:59.963 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:58:59.963 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:58:59.963 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:59.963 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.941+0000 7f9f1a21c640 10 monclient: shutdown 2024-09-16T03:58:59.963 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.941+0000 7f9f1a21c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:58:59.963 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.941+0000 7f9f1a21c640 1 -- 172.21.15.42:0/1318803270 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9f14130a40 msgr2=0x7f9f140a2250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:59.963 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.941+0000 7f9f1a21c640 1 --2- 172.21.15.42:0/1318803270 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9f14130a40 0x7f9f140a2250 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f9f040097c0 tx=0x7f9f0401c360 comp rx=0 tx=0).stop 2024-09-16T03:58:59.963 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.941+0000 7f9f1a21c640 1 -- 172.21.15.42:0/1318803270 shutdown_connections 2024-09-16T03:58:59.963 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.941+0000 7f9f1a21c640 1 --2- 172.21.15.42:0/1318803270 >> v2:172.21.15.154:3300/0 conn(0x7f9f14131260 0x7f9f14072b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:59.963 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.941+0000 7f9f1a21c640 1 --2- 172.21.15.42:0/1318803270 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9f14130a40 0x7f9f140a2250 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:59.963 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.941+0000 7f9f1a21c640 1 -- 172.21.15.42:0/1318803270 >> 172.21.15.42:0/1318803270 conn(0x7f9f1406bc40 msgr2=0x7f9f1406c050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:58:59.964 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.941+0000 7f9f1a21c640 1 -- 172.21.15.42:0/1318803270 shutdown_connections 2024-09-16T03:58:59.964 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.941+0000 7f9f1a21c640 1 -- 172.21.15.42:0/1318803270 wait complete. 2024-09-16T03:58:59.964 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.941+0000 7f9f1a21c640 10 monclient: build_initial_monmap 2024-09-16T03:58:59.964 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.941+0000 7f9f1a21c640 10 monclient: monmap: 2024-09-16T03:58:59.964 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:58:59.964 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:58:59.964 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:58:59.964 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:58:59.964 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:58:59.964 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:59.964 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:58:59.964 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:58:59.964 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:58:59.964 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:59.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.941+0000 7f9f1a21c640 1 librados: starting msgr at 2024-09-16T03:58:59.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.941+0000 7f9f1a21c640 1 librados: starting objecter 2024-09-16T03:58:59.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.941+0000 7f9f1a21c640 1 Processor -- start 2024-09-16T03:58:59.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.941+0000 7f9f1a21c640 1 -- start start 2024-09-16T03:58:59.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.941+0000 7f9f1a21c640 1 librados: setting wanted keys 2024-09-16T03:58:59.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.941+0000 7f9f1a21c640 1 librados: calling monclient init 2024-09-16T03:58:59.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.941+0000 7f9f1a21c640 10 monclient: init 2024-09-16T03:58:59.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.941+0000 7f9f1a21c640 10 monclient: _reopen_session rank -1 2024-09-16T03:58:59.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.941+0000 7f9f1a21c640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:58:59.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.941+0000 7f9f1a21c640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9f14130a40 0x7f9f141cf010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:59.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.941+0000 7f9f1a21c640 10 monclient(hunting): picked mon.noname-c con 0x7f9f14130a40 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:58:59.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.941+0000 7f9f1a21c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9f14130e20 0x7f9f1413dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:59.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.941+0000 7f9f1a21c640 10 monclient(hunting): picked mon.noname-b con 0x7f9f14130e20 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:58:59.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.941+0000 7f9f1a21c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9f14131260 0x7f9f1413e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:59.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.941+0000 7f9f1a21c640 10 monclient(hunting): picked mon.noname-a con 0x7f9f14131260 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:58:59.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.941+0000 7f9f1a21c640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:59.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.941+0000 7f9f1a21c640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f9f14141e10 con 0x7f9f14131260 2024-09-16T03:58:59.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.941+0000 7f9f1a21c640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:59.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.941+0000 7f9f1a21c640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f9f14141c90 con 0x7f9f14130e20 2024-09-16T03:58:59.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.941+0000 7f9f1a21c640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:59.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.941+0000 7f9f1a21c640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f9f14141f90 con 0x7f9f14130a40 2024-09-16T03:58:59.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.941+0000 7f9f1a21c640 10 monclient(hunting): _renew_subs 2024-09-16T03:58:59.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.941+0000 7f9f1a21c640 10 monclient(hunting): authenticate will time out at 1986.915106s 2024-09-16T03:58:59.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.941+0000 7f9f13fff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9f14131260 0x7f9f1413e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:59.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.941+0000 7f9f13fff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9f14131260 0x7f9f1413e2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:33524/0 (socket says 172.21.15.42:33524) 2024-09-16T03:58:59.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.941+0000 7f9f13fff640 1 -- 172.21.15.42:0/443630319 learned_addr learned my addr 172.21.15.42:0/443630319 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:58:59.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.941+0000 7f9f13fff640 10 monclient(hunting): get_auth_request con 0x7f9f14131260 auth_method 0 2024-09-16T03:58:59.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.941+0000 7f9f13fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:59.967 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.941+0000 7f9f13fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:59.967 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.941+0000 7f9f13fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:58:59.967 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.941+0000 7f9f137fe640 1 --2- 172.21.15.42:0/443630319 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9f14130a40 0x7f9f141cf010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:59.967 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.941+0000 7f9f137fe640 10 monclient(hunting): get_auth_request con 0x7f9f14130a40 auth_method 0 2024-09-16T03:58:59.967 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.941+0000 7f9f137fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:59.967 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.941+0000 7f9f137fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:59.967 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.941+0000 7f9f137fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:58:59.967 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.941+0000 7f9f13fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:58:59.967 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.941+0000 7f9f13fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:58:59.967 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.941+0000 7f9f13fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:58:59.967 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.941+0000 7f9f137fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:58:59.967 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.941+0000 7f9f137fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:58:59.967 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.941+0000 7f9f12ffd640 1 --2- 172.21.15.42:0/443630319 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9f14130e20 0x7f9f1413dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:59.967 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.941+0000 7f9f137fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:58:59.968 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.941+0000 7f9f12ffd640 10 monclient(hunting): get_auth_request con 0x7f9f14130e20 auth_method 0 2024-09-16T03:58:59.968 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.941+0000 7f9f12ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:59.968 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.941+0000 7f9f12ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:59.968 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.941+0000 7f9f12ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:58:59.968 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.941+0000 7f9f13fff640 10 monclient(hunting): handle_auth_done global_id 825084 payload 824 2024-09-16T03:58:59.968 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.941+0000 7f9f13fff640 1 -- 172.21.15.42:0/443630319 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9f14130a40 msgr2=0x7f9f141cf010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:59.968 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.941+0000 7f9f13fff640 1 --2- 172.21.15.42:0/443630319 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9f14130a40 0x7f9f141cf010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:59.968 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.941+0000 7f9f13fff640 1 -- 172.21.15.42:0/443630319 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9f14130e20 msgr2=0x7f9f1413dd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:59.968 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.941+0000 7f9f13fff640 1 --2- 172.21.15.42:0/443630319 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9f14130e20 0x7f9f1413dd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:58:59.968 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.941+0000 7f9f13fff640 10 monclient: _finish_hunting 0 2024-09-16T03:58:59.968 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.941+0000 7f9f13fff640 1 monclient: found mon.noname-a 2024-09-16T03:58:59.968 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.941+0000 7f9f13fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:58:59.968 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.941+0000 7f9f13fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:58:59.968 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.941+0000 7f9f13fff640 1 -- 172.21.15.42:0/443630319 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f1413c580 con 0x7f9f14131260 2024-09-16T03:58:59.969 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.941+0000 7f9f13fff640 1 --2- 172.21.15.42:0/443630319 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9f14131260 0x7f9f1413e2d0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f9f0c009f70 tx=0x7f9f0c00d560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:58:59.969 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.941+0000 7f9efbfff640 10 client.?.objecter ms_handle_connect 0x7f9f14131260 2024-09-16T03:58:59.969 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.941+0000 7f9efbfff640 10 client.?.objecter resend_mon_ops 2024-09-16T03:58:59.969 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.945+0000 7f9efbfff640 1 -- 172.21.15.42:0/443630319 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f0c01a070 con 0x7f9f14131260 2024-09-16T03:58:59.969 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.945+0000 7f9efbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:58:59.969 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.945+0000 7f9efbfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:58:59.969 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.945+0000 7f9efbfff640 10 monclient: dump: 2024-09-16T03:58:59.969 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:58:59.969 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:58:59.969 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:59.969 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:59.969 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:58:59.969 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:59.969 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:58:59.970 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:58:59.970 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:58:59.970 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:59.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.945+0000 7f9efbfff640 10 monclient: _finish_auth 0 2024-09-16T03:58:59.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.945+0000 7f9efbfff640 10 monclient: _check_auth_tickets 2024-09-16T03:58:59.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.945+0000 7f9efbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:58:59.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.945+0000 7f9f1a21c640 5 monclient: authenticate success, global_id 825084 2024-09-16T03:58:59.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.945+0000 7f9f1a21c640 10 monclient: _renew_subs 2024-09-16T03:58:59.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.945+0000 7f9efbfff640 1 -- 172.21.15.42:0/443630319 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f0c003bc0 con 0x7f9f14131260 2024-09-16T03:58:59.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.945+0000 7f9f1a21c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:58:59.971 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.945+0000 7f9f1a21c640 1 -- 172.21.15.42:0/443630319 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f141cf550 con 0x7f9f14131260 2024-09-16T03:58:59.971 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.945+0000 7f9efbfff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:58:59.971 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.945+0000 7f9efbfff640 1 -- 172.21.15.42:0/443630319 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f0c004030 con 0x7f9f14131260 2024-09-16T03:58:59.971 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.945+0000 7f9efbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:58:59.971 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.945+0000 7f9efbfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:58:59.971 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.945+0000 7f9efbfff640 10 monclient: dump: 2024-09-16T03:58:59.971 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:58:59.971 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:58:59.971 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:59.971 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:59.971 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:58:59.971 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:59.971 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:58:59.971 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:58:59.972 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:58:59.972 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:59.972 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.945+0000 7f9f11ffb640 4 set_mon_vals no callback set 2024-09-16T03:58:59.972 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.945+0000 7f9f1a21c640 10 client.825084.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:58:59.972 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.945+0000 7f9f1a21c640 10 monclient: _renew_subs 2024-09-16T03:58:59.972 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.945+0000 7f9f1a21c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:58:59.972 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.945+0000 7f9f1a21c640 1 -- 172.21.15.42:0/443630319 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f1409d150 con 0x7f9f14131260 2024-09-16T03:58:59.972 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.945+0000 7f9f1a21c640 1 librados: init done 2024-09-16T03:58:59.972 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.945+0000 7f9f1a21c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:58:59.972 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.945+0000 7f9efbfff640 1 -- 172.21.15.42:0/443630319 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f9f0c01d070 con 0x7f9f14131260 2024-09-16T03:58:59.972 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.945+0000 7f9f1a21c640 1 --2- 172.21.15.42:0/443630319 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9f141388d0 0x7f9f14138cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:59.972 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.945+0000 7f9f1a21c640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:59.972 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.945+0000 7f9f1a21c640 1 -- 172.21.15.42:0/443630319 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f9f140732a0 con 0x7f9f141388d0 2024-09-16T03:58:59.973 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.945+0000 7f9f137fe640 1 --2- 172.21.15.42:0/443630319 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9f141388d0 0x7f9f14138cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:59.973 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.945+0000 7f9f137fe640 10 monclient: get_auth_request con 0x7f9f141388d0 auth_method 0 2024-09-16T03:58:59.973 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.945+0000 7f9f137fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:59.973 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.945+0000 7f9f137fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:59.973 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.945+0000 7f9f137fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:58:59.973 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.945+0000 7f9efbfff640 1 --2- 172.21.15.42:0/443630319 >> v2:172.21.15.154:6812/254845342 conn(0x7f9f00037810 0x7f9f00039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:59.973 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.945+0000 7f9f137fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:58:59.973 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.945+0000 7f9f137fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:58:59.973 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.945+0000 7f9f137fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:58:59.973 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.945+0000 7f9efbfff640 1 -- 172.21.15.42:0/443630319 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(516..516 src has 366..516) ==== 5258+0+0 (secure 0 0 0) 0x7f9f0c048fe0 con 0x7f9f14131260 2024-09-16T03:58:59.973 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.945+0000 7f9efbfff640 10 client.825084.objecter ms_dispatch 0x7f9f14000ba0 osd_map(516..516 src has 366..516) 2024-09-16T03:58:59.973 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.945+0000 7f9efbfff640 3 client.825084.objecter handle_osd_map got epochs [516,516] > 0 2024-09-16T03:58:59.973 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.945+0000 7f9efbfff640 3 client.825084.objecter handle_osd_map decoding full epoch 516 2024-09-16T03:58:59.973 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.945+0000 7f9f137fe640 10 monclient(hunting): handle_auth_done global_id 825087 payload 824 2024-09-16T03:58:59.974 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.945+0000 7f9efbfff640 20 client.825084.objecter dump_active .. 0 homeless 2024-09-16T03:58:59.974 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.945+0000 7f9f137fe640 1 -- 172.21.15.42:0/443630319 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9f140a2250 con 0x7f9f141388d0 2024-09-16T03:58:59.974 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.945+0000 7f9f12ffd640 1 --2- 172.21.15.42:0/443630319 >> v2:172.21.15.154:6812/254845342 conn(0x7f9f00037810 0x7f9f00039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:59.974 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.945+0000 7f9f12ffd640 10 monclient: get_auth_request con 0x7f9f00037810 auth_method 0 2024-09-16T03:58:59.974 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.945+0000 7f9efbfff640 10 client.825084.objecter ms_handle_connect 0x7f9f141388d0 2024-09-16T03:58:59.974 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.945+0000 7f9efbfff640 10 client.825084.objecter resend_mon_ops 2024-09-16T03:58:59.974 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.945+0000 7f9f137fe640 1 --2- 172.21.15.42:0/443630319 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9f141388d0 0x7f9f14138cb0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f9f1406ca10 tx=0x7f9f04019100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:58:59.974 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.945+0000 7f9efbfff640 1 -- 172.21.15.42:0/443630319 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f040bc030 con 0x7f9f141388d0 2024-09-16T03:58:59.974 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.945+0000 7f9efbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:58:59.974 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.945+0000 7f9efbfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:58:59.974 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.945+0000 7f9efbfff640 10 monclient: dump: 2024-09-16T03:58:59.974 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:58:59.975 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:58:59.975 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:59.975 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:59.975 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:58:59.975 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:59.975 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:58:59.975 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:58:59.975 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:58:59.975 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:59.975 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.945+0000 7f9efbfff640 1 -- 172.21.15.42:0/443630319 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f9f040b3040 con 0x7f9f141388d0 2024-09-16T03:58:59.975 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.945+0000 7f9efbfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:58:59.975 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.945+0000 7f9efbfff640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:58:59.975 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.945+0000 7f9efbfff640 1 -- 172.21.15.42:0/443630319 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9f141388d0 msgr2=0x7f9f14138cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:59.975 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.945+0000 7f9efbfff640 1 --2- 172.21.15.42:0/443630319 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9f141388d0 0x7f9f14138cb0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f9f1406ca10 tx=0x7f9f04019100 comp rx=0 tx=0).stop 2024-09-16T03:58:59.976 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.945+0000 7f9efbfff640 1 -- 172.21.15.42:0/443630319 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9f141388d0 msgr2=0x7f9f14138cb0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:58:59.976 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.945+0000 7f9efbfff640 1 --2- 172.21.15.42:0/443630319 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9f141388d0 0x7f9f14138cb0 secure :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f9f1406ca10 tx=0x7f9f04019100 comp rx=0 tx=0).stop 2024-09-16T03:58:59.976 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.945+0000 7f9f12ffd640 1 --2- 172.21.15.42:0/443630319 >> v2:172.21.15.154:6812/254845342 conn(0x7f9f00037810 0x7f9f00039cd0 secure :-1 s=READY pgs=3718 cs=0 l=1 rev1=1 crypto rx=0x7f9f14073d80 tx=0x7f9f08009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:58:59.976 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.945+0000 7f9efbfff640 10 client.825084.objecter ms_handle_connect 0x7f9f00037810 2024-09-16T03:58:59.976 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.953+0000 7f9f1a21c640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:58:59.976 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.953+0000 7f9f1a21c640 1 --2- 172.21.15.42:0/443630319 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9f141391f0 0x7f9f14134a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:58:59.976 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.953+0000 7f9f1a21c640 10 monclient(hunting): start opening mon connection 2024-09-16T03:58:59.976 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.953+0000 7f9f1a21c640 1 -- 172.21.15.42:0/443630319 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f9f1413ae50 con 0x7f9f141391f0 2024-09-16T03:58:59.976 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.953+0000 7f9f137fe640 1 --2- 172.21.15.42:0/443630319 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9f141391f0 0x7f9f14134a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:58:59.976 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.953+0000 7f9f137fe640 10 monclient: get_auth_request con 0x7f9f141391f0 auth_method 0 2024-09-16T03:58:59.976 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.953+0000 7f9f137fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:58:59.976 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.953+0000 7f9f137fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:58:59.976 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.953+0000 7f9f137fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:58:59.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.953+0000 7f9f137fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:58:59.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.953+0000 7f9f137fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:58:59.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.953+0000 7f9f137fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:58:59.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.953+0000 7f9f137fe640 10 monclient(hunting): handle_auth_done global_id 825090 payload 824 2024-09-16T03:58:59.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.953+0000 7f9f137fe640 1 -- 172.21.15.42:0/443630319 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f9f1406b3e0 con 0x7f9f141391f0 2024-09-16T03:58:59.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.953+0000 7f9f137fe640 1 --2- 172.21.15.42:0/443630319 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9f141391f0 0x7f9f14134a60 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f9f14073420 tx=0x7f9f04009190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:58:59.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.953+0000 7f9efbfff640 10 client.825084.objecter ms_handle_connect 0x7f9f141391f0 2024-09-16T03:58:59.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.953+0000 7f9efbfff640 10 client.825084.objecter resend_mon_ops 2024-09-16T03:58:59.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.953+0000 7f9efbfff640 1 -- 172.21.15.42:0/443630319 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f040bc030 con 0x7f9f141391f0 2024-09-16T03:58:59.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.953+0000 7f9efbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:58:59.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.953+0000 7f9efbfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:58:59.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.953+0000 7f9efbfff640 10 monclient: dump: 2024-09-16T03:58:59.977 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:58:59.978 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:58:59.978 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:59.978 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:58:59.978 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:58:59.978 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:58:59.978 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:58:59.978 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:58:59.978 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:58:59.978 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:58:59.978 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.953+0000 7f9efbfff640 1 -- 172.21.15.42:0/443630319 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4882 (secure 0 0 0) 0x7f9f040ad5c0 con 0x7f9f141391f0 2024-09-16T03:58:59.978 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.953+0000 7f9efbfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:58:59.978 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.953+0000 7f9efbfff640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:58:59.978 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.953+0000 7f9efbfff640 1 -- 172.21.15.42:0/443630319 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9f141391f0 msgr2=0x7f9f14134a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:59.978 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.953+0000 7f9efbfff640 1 --2- 172.21.15.42:0/443630319 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9f141391f0 0x7f9f14134a60 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f9f14073420 tx=0x7f9f04009190 comp rx=0 tx=0).stop 2024-09-16T03:58:59.979 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.953+0000 7f9efbfff640 1 -- 172.21.15.42:0/443630319 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9f141391f0 msgr2=0x7f9f14134a60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:58:59.979 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.953+0000 7f9efbfff640 1 --2- 172.21.15.42:0/443630319 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9f141391f0 0x7f9f14134a60 secure :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f9f14073420 tx=0x7f9f04009190 comp rx=0 tx=0).stop 2024-09-16T03:58:59.979 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:58:59.979 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:58:59.979 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:58:59.979 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "leader", 2024-09-16T03:58:59.979 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 154, 2024-09-16T03:58:59.979 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 429808, 2024-09-16T03:58:59.979 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:58:59.980 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:58:59.980 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T03:58:59.980 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T03:58:59.980 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:58:59.980 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 4, 2024-09-16T03:58:59.980 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:58:59.980 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:58:59.980 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:58:59.980 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:58:59.980 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:58:59.980 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:58:59.980 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:58:59.980 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:58:59.981 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:58:59.981 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:58:59.981 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:58:59.981 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:58:59.981 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:58:59.981 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:58:59.981 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:58:59.981 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:58:59.981 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:58:59.981 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:58:59.981 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:58:59.981 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:58:59.981 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:58:59.981 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:58:59.982 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:58:59.982 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:58:59.982 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:58:59.982 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:58:59.982 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:58:59.982 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:58:59.982 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:58:59.982 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:58:59.982 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:58:59.982 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:58:59.982 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:58:59.982 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:58:59.982 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:58:59.982 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:58:59.983 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:58:59.983 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:58:59.983 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:58:59.983 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:58:59.983 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:58:59.983 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:58:59.983 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:58:59.983 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:58:59.983 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:58:59.983 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:58:59.983 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:58:59.983 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:58:59.983 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:58:59.983 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:58:59.984 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:58:59.984 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:58:59.984 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:58:59.984 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:58:59.984 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:58:59.984 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:58:59.984 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:58:59.984 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:58:59.984 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:58:59.984 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:58:59.984 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:58:59.984 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:58:59.984 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:58:59.984 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:58:59.985 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:58:59.985 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:58:59.985 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:58:59.985 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:58:59.985 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:58:59.985 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:58:59.985 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:58:59.985 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:58:59.985 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:58:59.985 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:58:59.985 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:58:59.985 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:58:59.985 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:58:59.985 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:58:59.986 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:58:59.986 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:58:59.986 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:58:59.986 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:58:59.986 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:58:59.986 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:58:59.986 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:58:59.986 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:58:59.986 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:58:59.986 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:58:59.986 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:58:59.986 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:58:59.986 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:58:59.986 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:58:59.986 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:58:59.987 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:58:59.987 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:58:59.987 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:58:59.987 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:58:59.987 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:58:59.987 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:58:59.987 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:58:59.987 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:58:59.987 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:58:59.987 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:58:59.987 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:58:59.987 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:58:59.987 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:58:59.987 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:58:59.988 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:58:59.988 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:58:59.988 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:58:59.988 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:58:59.988 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:58:59.988 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:58:59.988 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:58:59.988 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:58:59.988 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:58:59.988 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:58:59.988 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:58:59.988 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:58:59.988 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:58:59.989 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:58:59.989 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:58:59.989 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:58:59.989 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:58:59.989 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:58:59.989 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:58:59.989 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:58:59.989 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:58:59.989 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:58:59.989 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:58:59.989 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:58:59.989 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:58:59.989 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:58:59.990 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:58:59.990 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:58:59.990 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:58:59.990 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:58:59.990 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:58:59.990 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:58:59.990 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:58:59.990 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:58:59.990 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T03:58:59.990 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:58:59.990 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:58:59.990 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:58:59.990 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 5 2024-09-16T03:58:59.990 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:58:59.991 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:58:59.991 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:58:59.991 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:58:59.991 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:58:59.991 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:58:59.991 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 7 2024-09-16T03:58:59.991 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:58:59.991 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:58:59.991 INFO:teuthology.orchestra.run.smithi042.stdout: "mgr": [ 2024-09-16T03:58:59.991 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:58:59.991 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:58:59.991 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:58:59.991 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:58:59.991 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:58:59.991 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:58:59.992 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:58:59.992 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:58:59.992 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:58:59.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.953+0000 7f9f1a21c640 10 librados: watch_flush enter 2024-09-16T03:58:59.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.953+0000 7f9f1a21c640 10 librados: watch_flush exit 2024-09-16T03:58:59.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.953+0000 7f9f1a21c640 20 client.825084.objecter shutdown clearing up homeless session... 2024-09-16T03:58:59.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.953+0000 7f9f1a21c640 10 client.825084.objecter successfully canceled tick 2024-09-16T03:58:59.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.953+0000 7f9f1a21c640 1 -- 172.21.15.42:0/443630319 >> v2:172.21.15.154:6812/254845342 conn(0x7f9f00037810 msgr2=0x7f9f00039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:59.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.953+0000 7f9f1a21c640 1 --2- 172.21.15.42:0/443630319 >> v2:172.21.15.154:6812/254845342 conn(0x7f9f00037810 0x7f9f00039cd0 secure :-1 s=READY pgs=3718 cs=0 l=1 rev1=1 crypto rx=0x7f9f14073d80 tx=0x7f9f08009040 comp rx=0 tx=0).stop 2024-09-16T03:58:59.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.953+0000 7f9f1a21c640 10 monclient: shutdown 2024-09-16T03:58:59.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.953+0000 7f9f1a21c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:58:59.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.953+0000 7f9f1a21c640 1 -- 172.21.15.42:0/443630319 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9f14131260 msgr2=0x7f9f1413e2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:58:59.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.953+0000 7f9f1a21c640 1 --2- 172.21.15.42:0/443630319 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9f14131260 0x7f9f1413e2d0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f9f0c009f70 tx=0x7f9f0c00d560 comp rx=0 tx=0).stop 2024-09-16T03:58:59.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.953+0000 7f9f13fff640 1 -- 172.21.15.42:0/443630319 reap_dead start 2024-09-16T03:58:59.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.953+0000 7f9f13fff640 1 -- 172.21.15.42:0/443630319 reap_dead start 2024-09-16T03:58:59.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.953+0000 7f9f1a21c640 1 -- 172.21.15.42:0/443630319 shutdown_connections 2024-09-16T03:58:59.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.953+0000 7f9f1a21c640 1 -- 172.21.15.42:0/443630319 >> 172.21.15.42:0/443630319 conn(0x7f9f1406bc40 msgr2=0x7f9f14132f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:58:59.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.953+0000 7f9f1a21c640 1 -- 172.21.15.42:0/443630319 shutdown_connections 2024-09-16T03:58:59.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.953+0000 7f9f1a21c640 1 -- 172.21.15.42:0/443630319 wait complete. 2024-09-16T03:58:59.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:58:59.953+0000 7f9f1a21c640 1 librados: shutdown 2024-09-16T03:58:59.995 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:58:59.995 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T03:58:59.995 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T03:59:00.118 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.113+0000 7fa1f7e2b640 10 monclient: get_monmap_and_config 2024-09-16T03:59:00.118 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.113+0000 7fa1f7e2b640 10 monclient: build_initial_monmap 2024-09-16T03:59:00.118 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.113+0000 7fa1f7e2b640 10 monclient: monmap: 2024-09-16T03:59:00.118 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:59:00.118 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:59:00.118 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:59:00.116877+0000 2024-09-16T03:59:00.118 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:59:00.116877+0000 2024-09-16T03:59:00.119 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:59:00.119 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:59:00.119 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:59:00.119 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:59:00.119 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:59:00.119 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:59:00.119 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:59:00.119 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:59:00.125 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.113+0000 7fa1f7e2b640 1 Processor -- start 2024-09-16T03:59:00.125 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.113+0000 7fa1f7e2b640 1 -- start start 2024-09-16T03:59:00.125 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.113+0000 7fa1f7e2b640 10 monclient: init 2024-09-16T03:59:00.125 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.113+0000 7fa1f7e2b640 10 monclient: _reopen_session rank -1 2024-09-16T03:59:00.125 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.113+0000 7fa1f7e2b640 10 monclient: _add_conns ranks=[0,1,4,2,3] 2024-09-16T03:59:00.125 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.113+0000 7fa1f7e2b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa1f006db90 addr v1:172.21.15.42:6789/0 2024-09-16T03:59:00.125 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.113+0000 7fa1f7e2b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa1f01340c0 addr v1:172.21.15.154:6789/0 2024-09-16T03:59:00.125 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.113+0000 7fa1f7e2b640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fa1f013d8b0 0x7fa1f013fca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:59:00.125 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.113+0000 7fa1f7e2b640 10 monclient(hunting): picked mon.noname-c con 0x7fa1f013d8b0 addr v2:172.21.15.154:3300/0 2024-09-16T03:59:00.125 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.113+0000 7fa1f7e2b640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa1f006ece0 con 0x7fa1f006db90 2024-09-16T03:59:00.125 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.113+0000 7fa1f7e2b640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa1f009bf80 con 0x7fa1f01340c0 2024-09-16T03:59:00.126 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.113+0000 7fa1f7e2b640 10 monclient(hunting): start opening mon connection 2024-09-16T03:59:00.126 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.113+0000 7fa1f7e2b640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fa1f006df70 con 0x7fa1f013d8b0 2024-09-16T03:59:00.126 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.113+0000 7fa1f7e2b640 10 monclient(hunting): _renew_subs 2024-09-16T03:59:00.126 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.113+0000 7fa1f7e2b640 10 monclient(hunting): authenticate will time out at 1987.086335s 2024-09-16T03:59:00.126 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.113+0000 7fa1f5bc7640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fa1f006db90 0x7fa1f00a37f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:37266/0 (socket says 172.21.15.42:37266) 2024-09-16T03:59:00.126 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.113+0000 7fa1f5bc7640 1 -- 172.21.15.42:0/35603066 learned_addr learned my addr 172.21.15.42:0/35603066 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:59:00.126 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.113+0000 7fa1f63c8640 1 --2- 172.21.15.42:0/35603066 >> v2:172.21.15.154:3300/0 conn(0x7fa1f013d8b0 0x7fa1f013fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:59:00.126 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.113+0000 7fa1f4bc5640 1 -- 172.21.15.42:0/35603066 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 842522738 0 0) 0x7fa1f006ece0 con 0x7fa1f006db90 2024-09-16T03:59:00.126 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.113+0000 7fa1f4bc5640 10 monclient(hunting): _negotiate 2024-09-16T03:59:00.126 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.113+0000 7fa1f4bc5640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:59:00.126 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.113+0000 7fa1f4bc5640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:59:00.126 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.113+0000 7fa1f4bc5640 10 monclient(hunting): my global_id is 825093 2024-09-16T03:59:00.126 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.113+0000 7fa1f4bc5640 1 -- 172.21.15.42:0/35603066 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa1dc003530 con 0x7fa1f006db90 2024-09-16T03:59:00.126 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.113+0000 7fa1f63c8640 10 monclient(hunting): get_auth_request con 0x7fa1f013d8b0 auth_method 0 2024-09-16T03:59:00.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.113+0000 7fa1f63c8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:59:00.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.113+0000 7fa1f63c8640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:59:00.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.113+0000 7fa1f63c8640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:59:00.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.113+0000 7fa1f63c8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:59:00.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.113+0000 7fa1f63c8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:59:00.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.113+0000 7fa1f63c8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:59:00.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.113+0000 7fa1f4bc5640 1 -- 172.21.15.42:0/35603066 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 444902603 0 0) 0x7fa1dc003530 con 0x7fa1f006db90 2024-09-16T03:59:00.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.113+0000 7fa1f4bc5640 1 -- 172.21.15.42:0/35603066 >> v2:172.21.15.154:3300/0 conn(0x7fa1f013d8b0 msgr2=0x7fa1f013fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:59:00.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.113+0000 7fa1f4bc5640 1 --2- 172.21.15.42:0/35603066 >> v2:172.21.15.154:3300/0 conn(0x7fa1f013d8b0 0x7fa1f013fca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:59:00.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.113+0000 7fa1f4bc5640 1 -- 172.21.15.42:0/35603066 >> v1:172.21.15.154:6789/0 conn(0x7fa1f01340c0 legacy=0x7fa1f013c190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:59:00.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.113+0000 7fa1f4bc5640 10 monclient: _finish_hunting 0 2024-09-16T03:59:00.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.113+0000 7fa1f4bc5640 1 monclient: found mon.noname-a-legacy 2024-09-16T03:59:00.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.113+0000 7fa1f4bc5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:59:00.128 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.113+0000 7fa1f4bc5640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.42:6789/0 2024-09-16T03:59:00.128 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.113+0000 7fa1f4bc5640 1 -- 172.21.15.42:0/35603066 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1f01344d0 con 0x7fa1f006db90 2024-09-16T03:59:00.128 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.113+0000 7fa1f4bc5640 10 monclient: _finish_auth 0 2024-09-16T03:59:00.128 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.113+0000 7fa1f4bc5640 10 monclient: _check_auth_tickets 2024-09-16T03:59:00.128 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.113+0000 7fa1f4bc5640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:59:00.128 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.113+0000 7fa1f7e2b640 5 monclient: authenticate success, global_id 825093 2024-09-16T03:59:00.128 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.113+0000 7fa1f4bc5640 1 -- 172.21.15.42:0/35603066 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2177413734 0 0) 0x7fa1ec001680 con 0x7fa1f006db90 2024-09-16T03:59:00.128 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.113+0000 7fa1f4bc5640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:59:00.128 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.113+0000 7fa1f4bc5640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T03:59:00.128 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.113+0000 7fa1f4bc5640 10 monclient: dump: 2024-09-16T03:59:00.128 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:59:00.128 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:59:00.128 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:00.128 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:00.129 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:59:00.129 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:59:00.129 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:59:00.129 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:59:00.129 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:59:00.129 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:59:00.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.113+0000 7fa1f4bc5640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] but i'm connected to v1:172.21.15.42:6789/0, reconnecting 2024-09-16T03:59:00.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.113+0000 7fa1f4bc5640 10 monclient: _reopen_session rank -1 2024-09-16T03:59:00.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.113+0000 7fa1f4bc5640 1 -- 172.21.15.42:0/35603066 >> v1:172.21.15.42:6789/0 conn(0x7fa1f006db90 legacy=0x7fa1f00a37f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:59:00.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.113+0000 7fa1f4bc5640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:59:00.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.113+0000 7fa1f4bc5640 1 --2- 172.21.15.42:0/35603066 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa1dc0039c0 0x7fa1dc003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:59:00.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.113+0000 7fa1f4bc5640 10 monclient(hunting): picked mon.a con 0x7fa1dc0039c0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:59:00.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.113+0000 7fa1f4bc5640 1 --2- 172.21.15.42:0/35603066 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa1dc004810 0x7fa1dc004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:59:00.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.113+0000 7fa1f4bc5640 10 monclient(hunting): picked mon.c con 0x7fa1dc004810 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:59:00.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.113+0000 7fa1f4bc5640 1 --2- 172.21.15.42:0/35603066 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa1dc0052f0 0x7fa1dc00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:59:00.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.113+0000 7fa1f4bc5640 10 monclient(hunting): picked mon.b con 0x7fa1dc0052f0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:59:00.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.113+0000 7fa1f4bc5640 10 monclient(hunting): start opening mon connection 2024-09-16T03:59:00.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.113+0000 7fa1f4bc5640 1 -- 172.21.15.42:0/35603066 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fa1f006df70 con 0x7fa1dc0039c0 2024-09-16T03:59:00.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.113+0000 7fa1f4bc5640 10 monclient(hunting): start opening mon connection 2024-09-16T03:59:00.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.113+0000 7fa1f4bc5640 1 -- 172.21.15.42:0/35603066 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa1dc00c1c0 con 0x7fa1dc0052f0 2024-09-16T03:59:00.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.113+0000 7fa1f4bc5640 10 monclient(hunting): start opening mon connection 2024-09-16T03:59:00.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.113+0000 7fa1f4bc5640 1 -- 172.21.15.42:0/35603066 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fa1dc00c3c0 con 0x7fa1dc004810 2024-09-16T03:59:00.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.113+0000 7fa1f4bc5640 10 monclient(hunting): _renew_subs 2024-09-16T03:59:00.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.113+0000 7fa1f4bc5640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:59:00.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.113+0000 7fa1f7e2b640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-16T03:59:00.131 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.113+0000 7fa1f5bc7640 1 --2- 172.21.15.42:0/35603066 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa1dc0039c0 0x7fa1dc003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:59:00.131 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.113+0000 7fa1f5bc7640 10 monclient(hunting): get_auth_request con 0x7fa1dc0039c0 auth_method 0 2024-09-16T03:59:00.131 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.113+0000 7fa1f5bc7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:59:00.131 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.113+0000 7fa1f5bc7640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:59:00.131 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.113+0000 7fa1f5bc7640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:59:00.131 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.113+0000 7fa1f53c6640 1 --2- 172.21.15.42:0/35603066 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa1dc004810 0x7fa1dc004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:59:00.131 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.113+0000 7fa1f53c6640 10 monclient(hunting): get_auth_request con 0x7fa1dc004810 auth_method 0 2024-09-16T03:59:00.131 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.113+0000 7fa1f53c6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:59:00.131 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.113+0000 7fa1f53c6640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:59:00.131 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.113+0000 7fa1f53c6640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:59:00.131 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.113+0000 7fa1f5bc7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:59:00.131 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.113+0000 7fa1f5bc7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:59:00.131 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.113+0000 7fa1f63c8640 1 --2- 172.21.15.42:0/35603066 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa1dc0052f0 0x7fa1dc00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:59:00.132 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.113+0000 7fa1f5bc7640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T03:59:00.132 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.113+0000 7fa1f63c8640 10 monclient(hunting): get_auth_request con 0x7fa1dc0052f0 auth_method 0 2024-09-16T03:59:00.132 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.113+0000 7fa1f63c8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:59:00.132 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.113+0000 7fa1f63c8640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:59:00.132 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.113+0000 7fa1f63c8640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:59:00.132 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.113+0000 7fa1f53c6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:59:00.132 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.113+0000 7fa1f53c6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:59:00.132 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.113+0000 7fa1f53c6640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T03:59:00.132 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.113+0000 7fa1f5bc7640 10 monclient(hunting): handle_auth_done global_id 825093 payload 293 2024-09-16T03:59:00.132 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.113+0000 7fa1f5bc7640 1 -- 172.21.15.42:0/35603066 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa1dc004810 msgr2=0x7fa1dc004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:59:00.132 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.113+0000 7fa1f5bc7640 1 --2- 172.21.15.42:0/35603066 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa1dc004810 0x7fa1dc004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:59:00.132 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.113+0000 7fa1f5bc7640 1 -- 172.21.15.42:0/35603066 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa1dc0052f0 msgr2=0x7fa1dc00b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:59:00.132 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.113+0000 7fa1f5bc7640 1 --2- 172.21.15.42:0/35603066 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa1dc0052f0 0x7fa1dc00b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:59:00.133 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.113+0000 7fa1f5bc7640 10 monclient: _finish_hunting 0 2024-09-16T03:59:00.133 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.113+0000 7fa1f5bc7640 1 monclient: found mon.a 2024-09-16T03:59:00.133 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.113+0000 7fa1f5bc7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:59:00.133 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.113+0000 7fa1f5bc7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:59:00.133 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.113+0000 7fa1f5bc7640 1 -- 172.21.15.42:0/35603066 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1dc003530 con 0x7fa1dc0039c0 2024-09-16T03:59:00.133 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.113+0000 7fa1f5bc7640 10 monclient: _finish_auth 0 2024-09-16T03:59:00.133 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.113+0000 7fa1f5bc7640 10 monclient: _check_auth_tickets 2024-09-16T03:59:00.133 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.113+0000 7fa1f5bc7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:59:00.133 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.113+0000 7fa1f63c8640 1 --2- 172.21.15.42:0/35603066 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa1dc0052f0 0x7fa1dc00b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T03:59:00.133 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.113+0000 7fa1f53c6640 1 --2- 172.21.15.42:0/35603066 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa1dc004810 0x7fa1dc004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:59:00.133 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.113+0000 7fa1f63c8640 1 -- 172.21.15.42:0/35603066 reap_dead start 2024-09-16T03:59:00.133 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.117+0000 7fa1f5bc7640 1 --2- 172.21.15.42:0/35603066 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa1dc0039c0 0x7fa1dc003de0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fa1ec001ec0 tx=0x7fa1ec01d660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb6d2a23aa21785 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:59:00.133 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.117+0000 7fa1f4bc5640 1 -- 172.21.15.42:0/35603066 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1ec007d40 con 0x7fa1dc0039c0 2024-09-16T03:59:00.133 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.117+0000 7fa1f4bc5640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:59:00.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.117+0000 7fa1f4bc5640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:59:00.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.117+0000 7fa1f4bc5640 10 monclient: dump: 2024-09-16T03:59:00.134 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:59:00.134 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:59:00.134 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:00.134 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:00.134 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:59:00.134 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:59:00.134 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:59:00.134 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:59:00.134 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:59:00.134 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:59:00.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.117+0000 7fa1f7e2b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:59:00.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.117+0000 7fa1f4bc5640 1 -- 172.21.15.42:0/35603066 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa1ec0a9020 con 0x7fa1dc0039c0 2024-09-16T03:59:00.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.117+0000 7fa1f4bc5640 10 monclient: handle_config config(0 keys) 2024-09-16T03:59:00.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.117+0000 7fa1f7e2b640 10 monclient: get_monmap_and_config success 2024-09-16T03:59:00.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.117+0000 7fa1f4bc5640 1 -- 172.21.15.42:0/35603066 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1ec005ae0 con 0x7fa1dc0039c0 2024-09-16T03:59:00.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.117+0000 7fa1f7e2b640 4 set_mon_vals no callback set 2024-09-16T03:59:00.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.117+0000 7fa1f4bc5640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:59:00.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.117+0000 7fa1f4bc5640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:59:00.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.117+0000 7fa1f4bc5640 10 monclient: dump: 2024-09-16T03:59:00.135 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:59:00.135 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:59:00.135 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:00.135 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:00.135 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:59:00.135 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:59:00.136 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:59:00.136 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:59:00.136 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:59:00.136 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:59:00.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.117+0000 7fa1f7e2b640 10 monclient: shutdown 2024-09-16T03:59:00.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.117+0000 7fa1f7e2b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:59:00.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.117+0000 7fa1f7e2b640 1 -- 172.21.15.42:0/35603066 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa1dc0039c0 msgr2=0x7fa1dc003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:59:00.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.117+0000 7fa1f7e2b640 1 --2- 172.21.15.42:0/35603066 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa1dc0039c0 0x7fa1dc003de0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fa1ec001ec0 tx=0x7fa1ec01d660 comp rx=0 tx=0).stop 2024-09-16T03:59:00.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.117+0000 7fa1f7e2b640 1 -- 172.21.15.42:0/35603066 shutdown_connections 2024-09-16T03:59:00.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.117+0000 7fa1f7e2b640 1 --2- 172.21.15.42:0/35603066 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa1dc0039c0 0x7fa1dc003de0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:59:00.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.117+0000 7fa1f7e2b640 1 -- 172.21.15.42:0/35603066 >> 172.21.15.42:0/35603066 conn(0x7fa1f00a9fb0 msgr2=0x7fa1f00aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:59:00.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.117+0000 7fa1f7e2b640 1 -- 172.21.15.42:0/35603066 shutdown_connections 2024-09-16T03:59:00.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.117+0000 7fa1f7e2b640 1 -- 172.21.15.42:0/35603066 wait complete. 2024-09-16T03:59:00.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.117+0000 7fa1f7e2b640 10 monclient: build_initial_monmap 2024-09-16T03:59:00.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.117+0000 7fa1f7e2b640 10 monclient: monmap: 2024-09-16T03:59:00.137 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:59:00.137 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:59:00.137 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:59:00.137 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:59:00.137 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:59:00.137 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:59:00.137 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:59:00.137 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:59:00.137 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:59:00.137 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:59:00.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.117+0000 7fa1f7e2b640 1 librados: starting msgr at 2024-09-16T03:59:00.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.117+0000 7fa1f7e2b640 1 librados: starting objecter 2024-09-16T03:59:00.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.117+0000 7fa1f7e2b640 1 Processor -- start 2024-09-16T03:59:00.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.117+0000 7fa1f7e2b640 1 -- start start 2024-09-16T03:59:00.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.117+0000 7fa1f7e2b640 1 librados: setting wanted keys 2024-09-16T03:59:00.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.117+0000 7fa1f7e2b640 1 librados: calling monclient init 2024-09-16T03:59:00.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.117+0000 7fa1f7e2b640 10 monclient: init 2024-09-16T03:59:00.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.121+0000 7fa1f7e2b640 10 monclient: _reopen_session rank -1 2024-09-16T03:59:00.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.121+0000 7fa1f7e2b640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:59:00.145 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.121+0000 7fa1f7e2b640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa1dc0039c0 0x7fa1f009d580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:59:00.145 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.121+0000 7fa1f7e2b640 10 monclient(hunting): picked mon.noname-c con 0x7fa1dc0039c0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:59:00.145 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.121+0000 7fa1f7e2b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa1f009dac0 0x7fa1f009df20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:59:00.145 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.121+0000 7fa1f7e2b640 10 monclient(hunting): picked mon.noname-b con 0x7fa1f009dac0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:59:00.145 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.121+0000 7fa1f7e2b640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa1f01cede0 0x7fa1f01cf290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:59:00.145 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.121+0000 7fa1f7e2b640 10 monclient(hunting): picked mon.noname-a con 0x7fa1f01cede0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:59:00.145 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.121+0000 7fa1f7e2b640 10 monclient(hunting): start opening mon connection 2024-09-16T03:59:00.145 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.121+0000 7fa1f7e2b640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fa1f0137200 con 0x7fa1f01cede0 2024-09-16T03:59:00.145 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.121+0000 7fa1f7e2b640 10 monclient(hunting): start opening mon connection 2024-09-16T03:59:00.145 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.121+0000 7fa1f7e2b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa1f0137080 con 0x7fa1f009dac0 2024-09-16T03:59:00.145 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.121+0000 7fa1f7e2b640 10 monclient(hunting): start opening mon connection 2024-09-16T03:59:00.145 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.121+0000 7fa1f7e2b640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fa1f0137380 con 0x7fa1dc0039c0 2024-09-16T03:59:00.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.121+0000 7fa1f7e2b640 10 monclient(hunting): _renew_subs 2024-09-16T03:59:00.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.121+0000 7fa1f7e2b640 10 monclient(hunting): authenticate will time out at 1987.092347s 2024-09-16T03:59:00.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.121+0000 7fa1f5bc7640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa1dc0039c0 0x7fa1f009d580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:59:00.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.121+0000 7fa1f63c8640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa1f01cede0 0x7fa1f01cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:59:00.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.121+0000 7fa1f5bc7640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa1dc0039c0 0x7fa1f009d580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:57494/0 (socket says 172.21.15.42:57494) 2024-09-16T03:59:00.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.121+0000 7fa1f5bc7640 1 -- 172.21.15.42:0/2911157046 learned_addr learned my addr 172.21.15.42:0/2911157046 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:59:00.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.121+0000 7fa1f63c8640 10 monclient(hunting): get_auth_request con 0x7fa1f01cede0 auth_method 0 2024-09-16T03:59:00.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.121+0000 7fa1f63c8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:59:00.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.121+0000 7fa1f63c8640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:59:00.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.121+0000 7fa1f63c8640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:59:00.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.121+0000 7fa1f5bc7640 10 monclient(hunting): get_auth_request con 0x7fa1dc0039c0 auth_method 0 2024-09-16T03:59:00.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.121+0000 7fa1f5bc7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:59:00.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.121+0000 7fa1f5bc7640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:59:00.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.121+0000 7fa1f5bc7640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:59:00.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.121+0000 7fa1f63c8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:59:00.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.121+0000 7fa1f63c8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:59:00.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.121+0000 7fa1f63c8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:59:00.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.121+0000 7fa1f5bc7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:59:00.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.121+0000 7fa1f5bc7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:59:00.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.121+0000 7fa1f5bc7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:59:00.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.121+0000 7fa1f53c6640 1 --2- 172.21.15.42:0/2911157046 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa1f009dac0 0x7fa1f009df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:59:00.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.121+0000 7fa1f53c6640 10 monclient(hunting): get_auth_request con 0x7fa1f009dac0 auth_method 0 2024-09-16T03:59:00.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.121+0000 7fa1f53c6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:59:00.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.121+0000 7fa1f53c6640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:59:00.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.121+0000 7fa1f53c6640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:59:00.148 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.121+0000 7fa1f63c8640 10 monclient(hunting): handle_auth_done global_id 825096 payload 824 2024-09-16T03:59:00.148 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.121+0000 7fa1f63c8640 1 -- 172.21.15.42:0/2911157046 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa1dc0039c0 msgr2=0x7fa1f009d580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:59:00.148 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.121+0000 7fa1f63c8640 1 --2- 172.21.15.42:0/2911157046 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa1dc0039c0 0x7fa1f009d580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:59:00.148 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.121+0000 7fa1f63c8640 1 -- 172.21.15.42:0/2911157046 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa1f009dac0 msgr2=0x7fa1f009df20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:59:00.148 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.121+0000 7fa1f63c8640 1 --2- 172.21.15.42:0/2911157046 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa1f009dac0 0x7fa1f009df20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:59:00.148 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.121+0000 7fa1f63c8640 10 monclient: _finish_hunting 0 2024-09-16T03:59:00.148 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.121+0000 7fa1f63c8640 1 monclient: found mon.noname-a 2024-09-16T03:59:00.148 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.121+0000 7fa1f63c8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:59:00.148 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.121+0000 7fa1f63c8640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:59:00.148 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.121+0000 7fa1f63c8640 1 -- 172.21.15.42:0/2911157046 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1f00a0ed0 con 0x7fa1f01cede0 2024-09-16T03:59:00.148 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.121+0000 7fa1f5bc7640 1 --2- 172.21.15.42:0/2911157046 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa1dc0039c0 0x7fa1f009d580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:59:00.148 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.121+0000 7fa1f53c6640 1 --2- 172.21.15.42:0/2911157046 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa1f009dac0 0x7fa1f009df20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T03:59:00.148 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.121+0000 7fa1f63c8640 1 --2- 172.21.15.42:0/2911157046 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa1f01cede0 0x7fa1f01cf290 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fa1f00a5ec0 tx=0x7fa1e8010590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:59:00.148 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.121+0000 7fa1e67fc640 10 client.?.objecter ms_handle_connect 0x7fa1f01cede0 2024-09-16T03:59:00.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.121+0000 7fa1e67fc640 10 client.?.objecter resend_mon_ops 2024-09-16T03:59:00.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.121+0000 7fa1e67fc640 1 -- 172.21.15.42:0/2911157046 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1e801a020 con 0x7fa1f01cede0 2024-09-16T03:59:00.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.121+0000 7fa1e67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:59:00.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.121+0000 7fa1e67fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:59:00.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.121+0000 7fa1e67fc640 10 monclient: dump: 2024-09-16T03:59:00.149 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:59:00.149 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:59:00.149 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:00.149 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:00.149 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:59:00.149 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:59:00.149 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:59:00.149 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:59:00.150 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:59:00.150 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:59:00.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.121+0000 7fa1e67fc640 10 monclient: _finish_auth 0 2024-09-16T03:59:00.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.121+0000 7fa1e67fc640 10 monclient: _check_auth_tickets 2024-09-16T03:59:00.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.121+0000 7fa1e67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:59:00.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.121+0000 7fa1e67fc640 1 -- 172.21.15.42:0/2911157046 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa1e8015440 con 0x7fa1f01cede0 2024-09-16T03:59:00.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.121+0000 7fa1f7e2b640 5 monclient: authenticate success, global_id 825096 2024-09-16T03:59:00.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.121+0000 7fa1f7e2b640 10 monclient: _renew_subs 2024-09-16T03:59:00.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.121+0000 7fa1f7e2b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:59:00.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.121+0000 7fa1f7e2b640 1 -- 172.21.15.42:0/2911157046 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1f00a11b0 con 0x7fa1f01cede0 2024-09-16T03:59:00.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.121+0000 7fa1e67fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:59:00.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.121+0000 7fa1e67fc640 1 -- 172.21.15.42:0/2911157046 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1e80157f0 con 0x7fa1f01cede0 2024-09-16T03:59:00.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.121+0000 7fa1e67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:59:00.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.121+0000 7fa1e67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:59:00.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.121+0000 7fa1e67fc640 10 monclient: dump: 2024-09-16T03:59:00.151 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:59:00.151 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:59:00.151 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:00.151 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:00.151 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:59:00.151 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:59:00.151 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:59:00.151 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:59:00.151 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:59:00.151 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:59:00.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.121+0000 7fa1e77fe640 4 set_mon_vals no callback set 2024-09-16T03:59:00.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.121+0000 7fa1f7e2b640 10 client.825096.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:59:00.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.121+0000 7fa1f7e2b640 10 monclient: _renew_subs 2024-09-16T03:59:00.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.121+0000 7fa1f7e2b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:59:00.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.121+0000 7fa1f7e2b640 1 -- 172.21.15.42:0/2911157046 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1f006f6a0 con 0x7fa1f01cede0 2024-09-16T03:59:00.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.121+0000 7fa1f7e2b640 1 librados: init done 2024-09-16T03:59:00.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.121+0000 7fa1f7e2b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:59:00.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.121+0000 7fa1f7e2b640 1 --2- 172.21.15.42:0/2911157046 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa1f01cf7d0 0x7fa1f013a180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:59:00.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.121+0000 7fa1f7e2b640 10 monclient(hunting): start opening mon connection 2024-09-16T03:59:00.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.121+0000 7fa1f7e2b640 1 -- 172.21.15.42:0/2911157046 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fa1f0132cc0 con 0x7fa1f01cf7d0 2024-09-16T03:59:00.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.121+0000 7fa1f5bc7640 1 --2- 172.21.15.42:0/2911157046 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa1f01cf7d0 0x7fa1f013a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:59:00.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.121+0000 7fa1f5bc7640 10 monclient: get_auth_request con 0x7fa1f01cf7d0 auth_method 0 2024-09-16T03:59:00.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.121+0000 7fa1f5bc7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:59:00.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.121+0000 7fa1f5bc7640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:59:00.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.121+0000 7fa1f5bc7640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:59:00.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.121+0000 7fa1e67fc640 1 -- 172.21.15.42:0/2911157046 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fa1e801d020 con 0x7fa1f01cede0 2024-09-16T03:59:00.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.121+0000 7fa1f5bc7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:59:00.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.121+0000 7fa1f5bc7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:59:00.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.121+0000 7fa1f5bc7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:59:00.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.121+0000 7fa1f5bc7640 10 monclient(hunting): handle_auth_done global_id 924146 payload 824 2024-09-16T03:59:00.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.121+0000 7fa1f5bc7640 1 -- 172.21.15.42:0/2911157046 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa1f0137710 con 0x7fa1f01cf7d0 2024-09-16T03:59:00.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.121+0000 7fa1e67fc640 1 --2- 172.21.15.42:0/2911157046 >> v2:172.21.15.154:6812/254845342 conn(0x7fa1d4037810 0x7fa1d4039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:59:00.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.121+0000 7fa1e67fc640 1 -- 172.21.15.42:0/2911157046 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(516..516 src has 366..516) ==== 5258+0+0 (secure 0 0 0) 0x7fa1e80496f0 con 0x7fa1f01cede0 2024-09-16T03:59:00.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.121+0000 7fa1e67fc640 10 client.825096.objecter ms_dispatch 0x7fa1f0000ba0 osd_map(516..516 src has 366..516) 2024-09-16T03:59:00.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.121+0000 7fa1e67fc640 3 client.825096.objecter handle_osd_map got epochs [516,516] > 0 2024-09-16T03:59:00.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.121+0000 7fa1e67fc640 3 client.825096.objecter handle_osd_map decoding full epoch 516 2024-09-16T03:59:00.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.121+0000 7fa1f5bc7640 1 --2- 172.21.15.42:0/2911157046 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa1f01cf7d0 0x7fa1f013a180 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fa1ec0ab040 tx=0x7fa1ec0199a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:59:00.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.121+0000 7fa1e67fc640 20 client.825096.objecter dump_active .. 0 homeless 2024-09-16T03:59:00.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.121+0000 7fa1e67fc640 10 client.825096.objecter ms_handle_connect 0x7fa1f01cf7d0 2024-09-16T03:59:00.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.121+0000 7fa1e67fc640 10 client.825096.objecter resend_mon_ops 2024-09-16T03:59:00.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.121+0000 7fa1e67fc640 1 -- 172.21.15.42:0/2911157046 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1ec007d40 con 0x7fa1f01cf7d0 2024-09-16T03:59:00.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.121+0000 7fa1e67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:59:00.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.121+0000 7fa1e67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:59:00.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.121+0000 7fa1e67fc640 10 monclient: dump: 2024-09-16T03:59:00.154 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:59:00.154 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:59:00.154 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:00.154 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:00.154 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:59:00.155 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:59:00.155 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:59:00.155 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:59:00.155 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:59:00.155 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:59:00.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.121+0000 7fa1f53c6640 1 --2- 172.21.15.42:0/2911157046 >> v2:172.21.15.154:6812/254845342 conn(0x7fa1d4037810 0x7fa1d4039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:59:00.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.121+0000 7fa1f53c6640 10 monclient: get_auth_request con 0x7fa1d4037810 auth_method 0 2024-09-16T03:59:00.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.121+0000 7fa1e67fc640 1 -- 172.21.15.42:0/2911157046 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fa1ec0a9040 con 0x7fa1f01cf7d0 2024-09-16T03:59:00.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.121+0000 7fa1e67fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:59:00.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.121+0000 7fa1e67fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:59:00.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.121+0000 7fa1e67fc640 1 -- 172.21.15.42:0/2911157046 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa1f01cf7d0 msgr2=0x7fa1f013a180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:59:00.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.121+0000 7fa1e67fc640 1 --2- 172.21.15.42:0/2911157046 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa1f01cf7d0 0x7fa1f013a180 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fa1ec0ab040 tx=0x7fa1ec0199a0 comp rx=0 tx=0).stop 2024-09-16T03:59:00.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.121+0000 7fa1e67fc640 1 -- 172.21.15.42:0/2911157046 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa1f01cf7d0 msgr2=0x7fa1f013a180 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:59:00.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.121+0000 7fa1e67fc640 1 --2- 172.21.15.42:0/2911157046 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa1f01cf7d0 0x7fa1f013a180 secure :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fa1ec0ab040 tx=0x7fa1ec0199a0 comp rx=0 tx=0).stop 2024-09-16T03:59:00.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.121+0000 7fa1f53c6640 1 --2- 172.21.15.42:0/2911157046 >> v2:172.21.15.154:6812/254845342 conn(0x7fa1d4037810 0x7fa1d4039cd0 secure :-1 s=READY pgs=3719 cs=0 l=1 rev1=1 crypto rx=0x7fa1e0000c10 tx=0x7fa1e0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:59:00.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.121+0000 7fa1e67fc640 10 client.825096.objecter ms_handle_connect 0x7fa1d4037810 2024-09-16T03:59:00.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.129+0000 7fa1f7e2b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:59:00.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.129+0000 7fa1f7e2b640 1 --2- 172.21.15.42:0/2911157046 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa1f013a6c0 0x7fa1f013aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:59:00.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.129+0000 7fa1f7e2b640 10 monclient(hunting): start opening mon connection 2024-09-16T03:59:00.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.129+0000 7fa1f7e2b640 1 -- 172.21.15.42:0/2911157046 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fa1f009fa60 con 0x7fa1f013a6c0 2024-09-16T03:59:00.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.129+0000 7fa1f5bc7640 1 --2- 172.21.15.42:0/2911157046 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa1f013a6c0 0x7fa1f013aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:59:00.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.129+0000 7fa1f5bc7640 10 monclient: get_auth_request con 0x7fa1f013a6c0 auth_method 0 2024-09-16T03:59:00.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.129+0000 7fa1f5bc7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:59:00.157 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.129+0000 7fa1f5bc7640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:59:00.157 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.129+0000 7fa1f5bc7640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:59:00.157 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.129+0000 7fa1f5bc7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:59:00.157 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.129+0000 7fa1f5bc7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:59:00.157 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.129+0000 7fa1f5bc7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:59:00.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.129+0000 7fa1f5bc7640 10 monclient(hunting): handle_auth_done global_id 924149 payload 824 2024-09-16T03:59:00.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.129+0000 7fa1f5bc7640 1 -- 172.21.15.42:0/2911157046 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fa1f0068e40 con 0x7fa1f013a6c0 2024-09-16T03:59:00.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.129+0000 7fa1f5bc7640 1 --2- 172.21.15.42:0/2911157046 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa1f013a6c0 0x7fa1f013aad0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fa1f009def0 tx=0x7fa1dc005840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:59:00.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.129+0000 7fa1e67fc640 10 client.825096.objecter ms_handle_connect 0x7fa1f013a6c0 2024-09-16T03:59:00.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.129+0000 7fa1e67fc640 10 client.825096.objecter resend_mon_ops 2024-09-16T03:59:00.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.129+0000 7fa1e67fc640 1 -- 172.21.15.42:0/2911157046 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1ec0074b0 con 0x7fa1f013a6c0 2024-09-16T03:59:00.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.129+0000 7fa1e67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:59:00.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.129+0000 7fa1e67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:59:00.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.129+0000 7fa1e67fc640 10 monclient: dump: 2024-09-16T03:59:00.161 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:59:00.161 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:59:00.161 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:00.161 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:00.161 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:59:00.161 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:59:00.161 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:59:00.161 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:59:00.161 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:59:00.161 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:59:00.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.129+0000 7fa1e67fc640 1 -- 172.21.15.42:0/2911157046 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4544 (secure 0 0 0) 0x7fa1ec0074b0 con 0x7fa1f013a6c0 2024-09-16T03:59:00.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.129+0000 7fa1e67fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:59:00.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.129+0000 7fa1e67fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:59:00.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.129+0000 7fa1e67fc640 1 -- 172.21.15.42:0/2911157046 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa1f013a6c0 msgr2=0x7fa1f013aad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:59:00.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.129+0000 7fa1e67fc640 1 --2- 172.21.15.42:0/2911157046 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa1f013a6c0 0x7fa1f013aad0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fa1f009def0 tx=0x7fa1dc005840 comp rx=0 tx=0).stop 2024-09-16T03:59:00.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.129+0000 7fa1e67fc640 1 -- 172.21.15.42:0/2911157046 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa1f013a6c0 msgr2=0x7fa1f013aad0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:59:00.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.129+0000 7fa1e67fc640 1 --2- 172.21.15.42:0/2911157046 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa1f013a6c0 0x7fa1f013aad0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:59:00.162 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:59:00.162 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:59:00.163 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:59:00.163 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T03:59:00.163 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 154, 2024-09-16T03:59:00.163 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 9683, 2024-09-16T03:59:00.163 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:59:00.163 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:59:00.163 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T03:59:00.163 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T03:59:00.163 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:59:00.163 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 4, 2024-09-16T03:59:00.163 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:59:00.163 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:59:00.163 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:59:00.164 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:59:00.164 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:59:00.164 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:59:00.164 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:59:00.164 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:59:00.164 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:59:00.164 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:59:00.164 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:59:00.164 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:59:00.164 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:59:00.164 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:59:00.164 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:59:00.164 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:59:00.165 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:59:00.165 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:59:00.165 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:59:00.165 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:59:00.165 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:59:00.165 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:59:00.165 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:59:00.165 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:59:00.165 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:59:00.165 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:59:00.165 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:59:00.165 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:59:00.166 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:59:00.166 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:59:00.166 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:59:00.166 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:59:00.166 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:59:00.166 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:59:00.166 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:59:00.166 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:59:00.166 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:59:00.166 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:59:00.166 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:59:00.166 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:59:00.167 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:59:00.167 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:59:00.167 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:59:00.167 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:59:00.167 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:59:00.167 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:59:00.167 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:59:00.167 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:59:00.167 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:59:00.167 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:59:00.167 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:59:00.167 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:59:00.167 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:59:00.168 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:59:00.168 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:59:00.168 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:59:00.168 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:59:00.168 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:59:00.168 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:59:00.168 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:59:00.168 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:59:00.168 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:59:00.168 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:59:00.168 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:59:00.168 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:59:00.168 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:59:00.168 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:59:00.169 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:59:00.169 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:59:00.169 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:59:00.169 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:59:00.169 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:59:00.169 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:59:00.169 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:59:00.169 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:59:00.169 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:59:00.169 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:59:00.169 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:59:00.169 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:59:00.169 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:59:00.169 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:59:00.170 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:59:00.170 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:59:00.170 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:59:00.170 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:59:00.170 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:59:00.170 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:59:00.170 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:59:00.170 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:59:00.170 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:59:00.170 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:59:00.170 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:59:00.170 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:59:00.170 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:59:00.171 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:59:00.171 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:59:00.171 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:59:00.171 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:59:00.171 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:59:00.171 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:59:00.171 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:59:00.171 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:59:00.171 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:59:00.171 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:59:00.171 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:59:00.171 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:59:00.171 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:59:00.171 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:59:00.172 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:59:00.172 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:59:00.172 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:59:00.172 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:59:00.172 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:59:00.172 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:59:00.172 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:59:00.172 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:59:00.172 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:59:00.172 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:59:00.172 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:59:00.172 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:59:00.172 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:59:00.172 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:59:00.172 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:59:00.173 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:59:00.173 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:59:00.173 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:59:00.173 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:59:00.173 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:59:00.173 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:59:00.173 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:59:00.173 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:59:00.173 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:59:00.173 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:59:00.173 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:59:00.173 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:59:00.173 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:59:00.174 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:59:00.174 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:59:00.174 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:59:00.174 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:59:00.174 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:59:00.174 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:59:00.174 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:59:00.174 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:59:00.174 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:59:00.174 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:59:00.174 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:59:00.174 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:59:00.174 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:59:00.174 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:59:00.175 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:59:00.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.129+0000 7fa1f7e2b640 10 librados: watch_flush enter 2024-09-16T03:59:00.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.129+0000 7fa1f7e2b640 10 librados: watch_flush exit 2024-09-16T03:59:00.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.129+0000 7fa1f7e2b640 20 client.825096.objecter shutdown clearing up homeless session... 2024-09-16T03:59:00.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.129+0000 7fa1f7e2b640 10 client.825096.objecter successfully canceled tick 2024-09-16T03:59:00.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.129+0000 7fa1f7e2b640 1 -- 172.21.15.42:0/2911157046 >> v2:172.21.15.154:6812/254845342 conn(0x7fa1d4037810 msgr2=0x7fa1d4039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:59:00.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.129+0000 7fa1f7e2b640 1 --2- 172.21.15.42:0/2911157046 >> v2:172.21.15.154:6812/254845342 conn(0x7fa1d4037810 0x7fa1d4039cd0 secure :-1 s=READY pgs=3719 cs=0 l=1 rev1=1 crypto rx=0x7fa1e0000c10 tx=0x7fa1e0009040 comp rx=0 tx=0).stop 2024-09-16T03:59:00.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.129+0000 7fa1f7e2b640 10 monclient: shutdown 2024-09-16T03:59:00.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.129+0000 7fa1f7e2b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:59:00.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.129+0000 7fa1f7e2b640 1 -- 172.21.15.42:0/2911157046 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa1f01cede0 msgr2=0x7fa1f01cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:59:00.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.129+0000 7fa1f7e2b640 1 --2- 172.21.15.42:0/2911157046 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa1f01cede0 0x7fa1f01cf290 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fa1f00a5ec0 tx=0x7fa1e8010590 comp rx=0 tx=0).stop 2024-09-16T03:59:00.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.129+0000 7fa1f63c8640 1 -- 172.21.15.42:0/2911157046 reap_dead start 2024-09-16T03:59:00.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.129+0000 7fa1f63c8640 1 -- 172.21.15.42:0/2911157046 reap_dead start 2024-09-16T03:59:00.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.129+0000 7fa1f7e2b640 1 -- 172.21.15.42:0/2911157046 shutdown_connections 2024-09-16T03:59:00.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.129+0000 7fa1f7e2b640 1 -- 172.21.15.42:0/2911157046 >> 172.21.15.42:0/2911157046 conn(0x7fa1f00a9fb0 msgr2=0x7fa1f00a9160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:59:00.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.129+0000 7fa1f7e2b640 1 -- 172.21.15.42:0/2911157046 shutdown_connections 2024-09-16T03:59:00.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.129+0000 7fa1f7e2b640 1 -- 172.21.15.42:0/2911157046 wait complete. 2024-09-16T03:59:00.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.129+0000 7fa1f7e2b640 1 librados: shutdown 2024-09-16T03:59:00.178 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:59:00.178 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T03:59:00.178 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T03:59:00.298 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.293+0000 7f70d3059640 10 monclient: get_monmap_and_config 2024-09-16T03:59:00.298 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.293+0000 7f70d3059640 10 monclient: build_initial_monmap 2024-09-16T03:59:00.298 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.293+0000 7f70d3059640 10 monclient: monmap: 2024-09-16T03:59:00.299 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:59:00.299 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:59:00.299 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:59:00.296883+0000 2024-09-16T03:59:00.299 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:59:00.296883+0000 2024-09-16T03:59:00.299 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:59:00.299 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:59:00.299 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:59:00.299 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:59:00.299 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:59:00.299 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:59:00.299 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:59:00.299 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:59:00.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.293+0000 7f70d3059640 1 Processor -- start 2024-09-16T03:59:00.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.293+0000 7f70d3059640 1 -- start start 2024-09-16T03:59:00.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.293+0000 7f70d3059640 10 monclient: init 2024-09-16T03:59:00.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.293+0000 7f70d3059640 10 monclient: _reopen_session rank -1 2024-09-16T03:59:00.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.293+0000 7f70d3059640 10 monclient: _add_conns ranks=[1,2,3,0,4] 2024-09-16T03:59:00.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.293+0000 7f70d3059640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f70cc0a1330 addr v1:172.21.15.154:6789/0 2024-09-16T03:59:00.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.293+0000 7f70d3059640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f70cc06dac0 0x7f70cc13a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:59:00.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.293+0000 7f70d3059640 10 monclient(hunting): picked mon.noname-b con 0x7f70cc06dac0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:59:00.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.293+0000 7f70d3059640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f70cc0a1e20 0x7f70cc13c890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:59:00.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.293+0000 7f70d3059640 10 monclient(hunting): picked mon.noname-a con 0x7f70cc0a1e20 addr v2:172.21.15.42:3300/0 2024-09-16T03:59:00.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.293+0000 7f70d3059640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f70cc09c7d0 con 0x7f70cc0a1330 2024-09-16T03:59:00.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.293+0000 7f70d3059640 10 monclient(hunting): start opening mon connection 2024-09-16T03:59:00.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.293+0000 7f70d3059640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f70cc09c090 con 0x7f70cc0a1e20 2024-09-16T03:59:00.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.293+0000 7f70d3059640 10 monclient(hunting): start opening mon connection 2024-09-16T03:59:00.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.293+0000 7f70d3059640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f70cc09f610 con 0x7f70cc06dac0 2024-09-16T03:59:00.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.293+0000 7f70d3059640 10 monclient(hunting): _renew_subs 2024-09-16T03:59:00.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.293+0000 7f70d3059640 10 monclient(hunting): authenticate will time out at 1987.266592s 2024-09-16T03:59:00.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.293+0000 7f70cbfff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f70cc06dac0 0x7f70cc13a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:59:00.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.293+0000 7f70cbfff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f70cc06dac0 0x7f70cc13a340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:57524/0 (socket says 172.21.15.42:57524) 2024-09-16T03:59:00.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.293+0000 7f70d15f6640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f70cc0a1e20 0x7f70cc13c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:59:00.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.293+0000 7f70cbfff640 1 -- 172.21.15.42:0/3837874711 learned_addr learned my addr 172.21.15.42:0/3837874711 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:59:00.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.293+0000 7f70cbfff640 10 monclient(hunting): get_auth_request con 0x7f70cc06dac0 auth_method 0 2024-09-16T03:59:00.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.293+0000 7f70cbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:59:00.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.293+0000 7f70cbfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:59:00.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.293+0000 7f70cbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:59:00.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.293+0000 7f70d15f6640 10 monclient(hunting): get_auth_request con 0x7f70cc0a1e20 auth_method 0 2024-09-16T03:59:00.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.293+0000 7f70d15f6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:59:00.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.293+0000 7f70d15f6640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:59:00.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.293+0000 7f70d15f6640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:59:00.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.293+0000 7f70cbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:59:00.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.293+0000 7f70cbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:59:00.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.293+0000 7f70cbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:59:00.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.293+0000 7f70d15f6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:59:00.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.293+0000 7f70d15f6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:59:00.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.293+0000 7f70d15f6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:59:00.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.293+0000 7f70cbfff640 10 monclient(hunting): handle_auth_done global_id 924152 payload 274 2024-09-16T03:59:00.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.293+0000 7f70cbfff640 1 -- 172.21.15.42:0/3837874711 >> v2:172.21.15.42:3300/0 conn(0x7f70cc0a1e20 msgr2=0x7f70cc13c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:59:00.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.293+0000 7f70cbfff640 1 --2- 172.21.15.42:0/3837874711 >> v2:172.21.15.42:3300/0 conn(0x7f70cc0a1e20 0x7f70cc13c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:59:00.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.293+0000 7f70cbfff640 1 -- 172.21.15.42:0/3837874711 >> v1:172.21.15.154:6789/0 conn(0x7f70cc0a1330 legacy=0x7f70cc0a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:59:00.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.293+0000 7f70cbfff640 10 monclient: _finish_hunting 0 2024-09-16T03:59:00.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.293+0000 7f70cbfff640 1 monclient: found mon.noname-b 2024-09-16T03:59:00.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.293+0000 7f70cbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:59:00.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.293+0000 7f70cbfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T03:59:00.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.293+0000 7f70cbfff640 1 -- 172.21.15.42:0/3837874711 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70cc09f8f0 con 0x7f70cc06dac0 2024-09-16T03:59:00.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.297+0000 7f70cbfff640 1 --2- 172.21.15.42:0/3837874711 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f70cc06dac0 0x7f70cc13a340 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f70b4009e50 tx=0x7f70b401c9e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7b2734842936c25f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:59:00.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.297+0000 7f70cb7fe640 1 -- 172.21.15.42:0/3837874711 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70b40a58d0 con 0x7f70cc06dac0 2024-09-16T03:59:00.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.297+0000 7f70cb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:59:00.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.297+0000 7f70cb7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:59:00.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.297+0000 7f70cb7fe640 10 monclient: dump: 2024-09-16T03:59:00.312 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:59:00.312 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:59:00.312 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:00.313 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:00.313 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:59:00.313 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:59:00.313 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:59:00.313 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:59:00.313 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:59:00.313 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:59:00.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.297+0000 7f70cb7fe640 10 monclient: _finish_auth 0 2024-09-16T03:59:00.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.297+0000 7f70cb7fe640 10 monclient: _check_auth_tickets 2024-09-16T03:59:00.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.297+0000 7f70cb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:59:00.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.297+0000 7f70cb7fe640 1 -- 172.21.15.42:0/3837874711 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f70b40a5a70 con 0x7f70cc06dac0 2024-09-16T03:59:00.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.297+0000 7f70cb7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:59:00.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.297+0000 7f70cb7fe640 1 -- 172.21.15.42:0/3837874711 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70b40a5e40 con 0x7f70cc06dac0 2024-09-16T03:59:00.314 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.297+0000 7f70cb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:59:00.314 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.297+0000 7f70cb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:59:00.314 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.297+0000 7f70cb7fe640 10 monclient: dump: 2024-09-16T03:59:00.314 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:59:00.314 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:59:00.314 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:00.314 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:00.314 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:59:00.314 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:59:00.314 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:59:00.314 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:59:00.314 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:59:00.314 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:59:00.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.297+0000 7f70d3059640 5 monclient: authenticate success, global_id 924152 2024-09-16T03:59:00.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.297+0000 7f70d3059640 10 monclient: get_monmap_and_config success 2024-09-16T03:59:00.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.297+0000 7f70d3059640 4 set_mon_vals no callback set 2024-09-16T03:59:00.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.297+0000 7f70d3059640 10 monclient: shutdown 2024-09-16T03:59:00.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.297+0000 7f70d3059640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:59:00.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.297+0000 7f70d3059640 1 -- 172.21.15.42:0/3837874711 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f70cc06dac0 msgr2=0x7f70cc13a340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:59:00.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.297+0000 7f70d3059640 1 --2- 172.21.15.42:0/3837874711 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f70cc06dac0 0x7f70cc13a340 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f70b4009e50 tx=0x7f70b401c9e0 comp rx=0 tx=0).stop 2024-09-16T03:59:00.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.297+0000 7f70d3059640 1 -- 172.21.15.42:0/3837874711 shutdown_connections 2024-09-16T03:59:00.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.297+0000 7f70d3059640 1 --2- 172.21.15.42:0/3837874711 >> v2:172.21.15.42:3300/0 conn(0x7f70cc0a1e20 0x7f70cc13c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:59:00.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.297+0000 7f70d3059640 1 --2- 172.21.15.42:0/3837874711 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f70cc06dac0 0x7f70cc13a340 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:59:00.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.297+0000 7f70d3059640 1 -- 172.21.15.42:0/3837874711 >> 172.21.15.42:0/3837874711 conn(0x7f70cc0a2420 msgr2=0x7f70cc0a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:59:00.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.297+0000 7f70d3059640 1 -- 172.21.15.42:0/3837874711 shutdown_connections 2024-09-16T03:59:00.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.297+0000 7f70d3059640 1 -- 172.21.15.42:0/3837874711 wait complete. 2024-09-16T03:59:00.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.297+0000 7f70d3059640 10 monclient: build_initial_monmap 2024-09-16T03:59:00.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.297+0000 7f70d3059640 10 monclient: monmap: 2024-09-16T03:59:00.316 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:59:00.316 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:59:00.316 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:59:00.316 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:59:00.316 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:59:00.316 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:59:00.316 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:59:00.316 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:59:00.316 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:59:00.317 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:59:00.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.297+0000 7f70d3059640 1 librados: starting msgr at 2024-09-16T03:59:00.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.297+0000 7f70d3059640 1 librados: starting objecter 2024-09-16T03:59:00.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.297+0000 7f70d3059640 1 Processor -- start 2024-09-16T03:59:00.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.297+0000 7f70d3059640 1 -- start start 2024-09-16T03:59:00.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.297+0000 7f70d3059640 1 librados: setting wanted keys 2024-09-16T03:59:00.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.297+0000 7f70d3059640 1 librados: calling monclient init 2024-09-16T03:59:00.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.297+0000 7f70d3059640 10 monclient: init 2024-09-16T03:59:00.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.301+0000 7f70d3059640 10 monclient: _reopen_session rank -1 2024-09-16T03:59:00.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.301+0000 7f70d3059640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:59:00.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.301+0000 7f70d3059640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f70cc0a1330 0x7f70cc1ced80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:59:00.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.301+0000 7f70d3059640 10 monclient(hunting): picked mon.noname-c con 0x7f70cc0a1330 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:59:00.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.301+0000 7f70d3059640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f70cc0a1e20 0x7f70cc1355a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:59:00.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.301+0000 7f70d3059640 10 monclient(hunting): picked mon.noname-a con 0x7f70cc0a1e20 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:59:00.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.301+0000 7f70d3059640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f70cc135ae0 0x7f70cc09cb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:59:00.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.301+0000 7f70d3059640 10 monclient(hunting): picked mon.noname-b con 0x7f70cc135ae0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:59:00.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.301+0000 7f70d3059640 10 monclient(hunting): start opening mon connection 2024-09-16T03:59:00.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.301+0000 7f70d3059640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f70cc13f770 con 0x7f70cc0a1e20 2024-09-16T03:59:00.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.301+0000 7f70d3059640 10 monclient(hunting): start opening mon connection 2024-09-16T03:59:00.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.301+0000 7f70d3059640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f70cc13f5f0 con 0x7f70cc135ae0 2024-09-16T03:59:00.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.301+0000 7f70d3059640 10 monclient(hunting): start opening mon connection 2024-09-16T03:59:00.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.301+0000 7f70d3059640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f70cc13f8f0 con 0x7f70cc0a1330 2024-09-16T03:59:00.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.301+0000 7f70d3059640 10 monclient(hunting): _renew_subs 2024-09-16T03:59:00.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.301+0000 7f70d3059640 10 monclient(hunting): authenticate will time out at 1987.272334s 2024-09-16T03:59:00.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.301+0000 7f70cbfff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f70cc0a1e20 0x7f70cc1355a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:59:00.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.301+0000 7f70d0df5640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f70cc0a1330 0x7f70cc1ced80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:59:00.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.301+0000 7f70cbfff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f70cc0a1e20 0x7f70cc1355a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:33566/0 (socket says 172.21.15.42:33566) 2024-09-16T03:59:00.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.301+0000 7f70cbfff640 1 -- 172.21.15.42:0/2754167261 learned_addr learned my addr 172.21.15.42:0/2754167261 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:59:00.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.301+0000 7f70cbfff640 10 monclient(hunting): get_auth_request con 0x7f70cc0a1e20 auth_method 0 2024-09-16T03:59:00.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.301+0000 7f70cbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:59:00.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.301+0000 7f70cbfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:59:00.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.301+0000 7f70cbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:59:00.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.301+0000 7f70d0df5640 10 monclient(hunting): get_auth_request con 0x7f70cc0a1330 auth_method 0 2024-09-16T03:59:00.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.301+0000 7f70d0df5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:59:00.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.301+0000 7f70d0df5640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:59:00.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.301+0000 7f70d0df5640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:59:00.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.301+0000 7f70d0df5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:59:00.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.301+0000 7f70d0df5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:59:00.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.301+0000 7f70d15f6640 1 --2- 172.21.15.42:0/2754167261 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f70cc135ae0 0x7f70cc09cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:59:00.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.301+0000 7f70d0df5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:59:00.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.301+0000 7f70cbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:59:00.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.301+0000 7f70cbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:59:00.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.301+0000 7f70cbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:59:00.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.301+0000 7f70d15f6640 10 monclient(hunting): get_auth_request con 0x7f70cc135ae0 auth_method 0 2024-09-16T03:59:00.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.301+0000 7f70d15f6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:59:00.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.301+0000 7f70d15f6640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:59:00.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.301+0000 7f70d15f6640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:59:00.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.301+0000 7f70d0df5640 10 monclient(hunting): handle_auth_done global_id 924155 payload 824 2024-09-16T03:59:00.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.301+0000 7f70d0df5640 1 -- 172.21.15.42:0/2754167261 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f70cc135ae0 msgr2=0x7f70cc09cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:59:00.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.301+0000 7f70d0df5640 1 --2- 172.21.15.42:0/2754167261 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f70cc135ae0 0x7f70cc09cb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:59:00.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.301+0000 7f70d0df5640 1 -- 172.21.15.42:0/2754167261 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f70cc0a1e20 msgr2=0x7f70cc1355a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:59:00.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.301+0000 7f70d0df5640 1 --2- 172.21.15.42:0/2754167261 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f70cc0a1e20 0x7f70cc1355a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:59:00.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.301+0000 7f70d0df5640 10 monclient: _finish_hunting 0 2024-09-16T03:59:00.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.301+0000 7f70d0df5640 1 monclient: found mon.noname-c 2024-09-16T03:59:00.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.301+0000 7f70d0df5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:59:00.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.301+0000 7f70d0df5640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:59:00.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.301+0000 7f70d0df5640 1 -- 172.21.15.42:0/2754167261 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70cc121f50 con 0x7f70cc0a1330 2024-09-16T03:59:00.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.301+0000 7f70cbfff640 1 --2- 172.21.15.42:0/2754167261 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f70cc0a1e20 0x7f70cc1355a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:59:00.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.301+0000 7f70d0df5640 1 --2- 172.21.15.42:0/2754167261 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f70cc0a1330 0x7f70cc1ced80 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f70c4001ec0 tx=0x7f70c400c890 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:59:00.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.301+0000 7f70c97fa640 10 client.?.objecter ms_handle_connect 0x7f70cc0a1330 2024-09-16T03:59:00.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.301+0000 7f70c97fa640 10 client.?.objecter resend_mon_ops 2024-09-16T03:59:00.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.301+0000 7f70c97fa640 1 -- 172.21.15.42:0/2754167261 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70c4017070 con 0x7f70cc0a1330 2024-09-16T03:59:00.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.301+0000 7f70c97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:59:00.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.301+0000 7f70c97fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:59:00.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.301+0000 7f70c97fa640 10 monclient: dump: 2024-09-16T03:59:00.323 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:59:00.323 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:59:00.323 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:00.323 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:00.323 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:59:00.323 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:59:00.323 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:59:00.324 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:59:00.324 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:59:00.324 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:59:00.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.301+0000 7f70c97fa640 10 monclient: _finish_auth 0 2024-09-16T03:59:00.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.301+0000 7f70c97fa640 10 monclient: _check_auth_tickets 2024-09-16T03:59:00.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.301+0000 7f70c97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:59:00.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.301+0000 7f70c97fa640 1 -- 172.21.15.42:0/2754167261 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f70c40052d0 con 0x7f70cc0a1330 2024-09-16T03:59:00.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.301+0000 7f70d3059640 5 monclient: authenticate success, global_id 924155 2024-09-16T03:59:00.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.301+0000 7f70d3059640 10 monclient: _renew_subs 2024-09-16T03:59:00.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.301+0000 7f70d3059640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:59:00.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.301+0000 7f70d3059640 1 -- 172.21.15.42:0/2754167261 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70cc073a30 con 0x7f70cc0a1330 2024-09-16T03:59:00.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.301+0000 7f70c97fa640 10 monclient: handle_config config(0 keys) 2024-09-16T03:59:00.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.301+0000 7f70c97fa640 1 -- 172.21.15.42:0/2754167261 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70c4003840 con 0x7f70cc0a1330 2024-09-16T03:59:00.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.301+0000 7f70c97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:59:00.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.301+0000 7f70d3059640 10 client.924155.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:59:00.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.301+0000 7f70c97fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:59:00.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.301+0000 7f70c97fa640 10 monclient: dump: 2024-09-16T03:59:00.325 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:59:00.325 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:59:00.325 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:00.325 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:00.325 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:59:00.325 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:59:00.325 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:59:00.325 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:59:00.325 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:59:00.325 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:59:00.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.301+0000 7f70caffd640 4 set_mon_vals no callback set 2024-09-16T03:59:00.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.301+0000 7f70d3059640 10 monclient: _renew_subs 2024-09-16T03:59:00.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.301+0000 7f70d3059640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:59:00.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.301+0000 7f70d3059640 1 -- 172.21.15.42:0/2754167261 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f70cc13c8e0 con 0x7f70cc0a1330 2024-09-16T03:59:00.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.301+0000 7f70d3059640 1 librados: init done 2024-09-16T03:59:00.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.301+0000 7f70d3059640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:59:00.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.301+0000 7f70d3059640 1 --2- 172.21.15.42:0/2754167261 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f70cc06dac0 0x7f70cc09d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:59:00.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.301+0000 7f70d3059640 10 monclient(hunting): start opening mon connection 2024-09-16T03:59:00.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.301+0000 7f70d3059640 1 -- 172.21.15.42:0/2754167261 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f70cc135ef0 con 0x7f70cc06dac0 2024-09-16T03:59:00.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.301+0000 7f70c97fa640 1 -- 172.21.15.42:0/2754167261 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f70c401a070 con 0x7f70cc0a1330 2024-09-16T03:59:00.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.301+0000 7f70c97fa640 1 --2- 172.21.15.42:0/2754167261 >> v2:172.21.15.154:6812/254845342 conn(0x7f70ac037810 0x7f70ac039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:59:00.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.301+0000 7f70cbfff640 1 --2- 172.21.15.42:0/2754167261 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f70cc06dac0 0x7f70cc09d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:59:00.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.301+0000 7f70cbfff640 10 monclient: get_auth_request con 0x7f70cc06dac0 auth_method 0 2024-09-16T03:59:00.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.301+0000 7f70cbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:59:00.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.301+0000 7f70cbfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:59:00.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.301+0000 7f70cbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:59:00.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.301+0000 7f70c97fa640 1 -- 172.21.15.42:0/2754167261 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(516..516 src has 366..516) ==== 5258+0+0 (secure 0 0 0) 0x7f70c40480c0 con 0x7f70cc0a1330 2024-09-16T03:59:00.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.301+0000 7f70c97fa640 10 client.924155.objecter ms_dispatch 0x7f70cc000ba0 osd_map(516..516 src has 366..516) 2024-09-16T03:59:00.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.301+0000 7f70c97fa640 3 client.924155.objecter handle_osd_map got epochs [516,516] > 0 2024-09-16T03:59:00.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.301+0000 7f70c97fa640 3 client.924155.objecter handle_osd_map decoding full epoch 516 2024-09-16T03:59:00.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.301+0000 7f70c97fa640 20 client.924155.objecter dump_active .. 0 homeless 2024-09-16T03:59:00.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.301+0000 7f70cbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:59:00.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.301+0000 7f70cbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:59:00.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.301+0000 7f70cbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:59:00.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.301+0000 7f70d15f6640 1 --2- 172.21.15.42:0/2754167261 >> v2:172.21.15.154:6812/254845342 conn(0x7f70ac037810 0x7f70ac039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:59:00.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.301+0000 7f70d15f6640 10 monclient: get_auth_request con 0x7f70ac037810 auth_method 0 2024-09-16T03:59:00.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.301+0000 7f70cbfff640 10 monclient(hunting): handle_auth_done global_id 904276 payload 824 2024-09-16T03:59:00.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.301+0000 7f70cbfff640 1 -- 172.21.15.42:0/2754167261 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f70cc0a1710 con 0x7f70cc06dac0 2024-09-16T03:59:00.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.301+0000 7f70c97fa640 10 client.924155.objecter ms_handle_connect 0x7f70cc06dac0 2024-09-16T03:59:00.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.301+0000 7f70cbfff640 1 --2- 172.21.15.42:0/2754167261 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f70cc06dac0 0x7f70cc09d0a0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f70b401a610 tx=0x7f70b4019780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:59:00.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.301+0000 7f70c97fa640 10 client.924155.objecter resend_mon_ops 2024-09-16T03:59:00.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.301+0000 7f70c97fa640 10 client.924155.objecter ms_handle_connect 0x7f70ac037810 2024-09-16T03:59:00.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.301+0000 7f70d15f6640 1 --2- 172.21.15.42:0/2754167261 >> v2:172.21.15.154:6812/254845342 conn(0x7f70ac037810 0x7f70ac039cd0 secure :-1 s=READY pgs=3720 cs=0 l=1 rev1=1 crypto rx=0x7f70bc009af0 tx=0x7f70bc008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:59:00.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.301+0000 7f70c97fa640 1 -- 172.21.15.42:0/2754167261 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70b4009510 con 0x7f70cc06dac0 2024-09-16T03:59:00.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.301+0000 7f70c97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:59:00.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.301+0000 7f70c97fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:59:00.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.301+0000 7f70c97fa640 10 monclient: dump: 2024-09-16T03:59:00.328 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:59:00.329 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:59:00.329 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:00.329 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:00.329 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:59:00.329 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:59:00.329 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:59:00.329 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:59:00.329 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:59:00.329 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:59:00.329 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.305+0000 7f70c97fa640 1 -- 172.21.15.42:0/2754167261 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f70b40c0440 con 0x7f70cc06dac0 2024-09-16T03:59:00.329 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.305+0000 7f70c97fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:59:00.329 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.305+0000 7f70c97fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:59:00.329 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.305+0000 7f70c97fa640 1 -- 172.21.15.42:0/2754167261 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f70cc06dac0 msgr2=0x7f70cc09d0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:59:00.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.305+0000 7f70c97fa640 1 --2- 172.21.15.42:0/2754167261 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f70cc06dac0 0x7f70cc09d0a0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f70b401a610 tx=0x7f70b4019780 comp rx=0 tx=0).stop 2024-09-16T03:59:00.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.305+0000 7f70c97fa640 1 -- 172.21.15.42:0/2754167261 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f70cc06dac0 msgr2=0x7f70cc09d0a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:59:00.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.305+0000 7f70c97fa640 1 --2- 172.21.15.42:0/2754167261 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f70cc06dac0 0x7f70cc09d0a0 secure :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f70b401a610 tx=0x7f70b4019780 comp rx=0 tx=0).stop 2024-09-16T03:59:00.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.309+0000 7f70d3059640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:59:00.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.309+0000 7f70d3059640 1 --2- 172.21.15.42:0/2754167261 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f70cc09d5e0 0x7f70cc138330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:59:00.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.309+0000 7f70d3059640 10 monclient(hunting): start opening mon connection 2024-09-16T03:59:00.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.309+0000 7f70d3059640 1 -- 172.21.15.42:0/2754167261 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f70cc09d9f0 con 0x7f70cc09d5e0 2024-09-16T03:59:00.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.309+0000 7f70cbfff640 1 --2- 172.21.15.42:0/2754167261 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f70cc09d5e0 0x7f70cc138330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:59:00.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.309+0000 7f70cbfff640 10 monclient: get_auth_request con 0x7f70cc09d5e0 auth_method 0 2024-09-16T03:59:00.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.309+0000 7f70cbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:59:00.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.309+0000 7f70cbfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:59:00.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.309+0000 7f70cbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:59:00.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.309+0000 7f70cbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:59:00.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.309+0000 7f70cbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:59:00.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.309+0000 7f70cbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:59:00.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.309+0000 7f70cbfff640 10 monclient(hunting): handle_auth_done global_id 904279 payload 824 2024-09-16T03:59:00.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.309+0000 7f70cbfff640 1 -- 172.21.15.42:0/2754167261 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f70cc1207c0 con 0x7f70cc09d5e0 2024-09-16T03:59:00.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.313+0000 7f70cbfff640 1 --2- 172.21.15.42:0/2754167261 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f70cc09d5e0 0x7f70cc138330 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f70b401ca20 tx=0x7f70b40a7000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:59:00.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.313+0000 7f70c97fa640 10 client.924155.objecter ms_handle_connect 0x7f70cc09d5e0 2024-09-16T03:59:00.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.313+0000 7f70c97fa640 10 client.924155.objecter resend_mon_ops 2024-09-16T03:59:00.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.313+0000 7f70c97fa640 1 -- 172.21.15.42:0/2754167261 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70b4004260 con 0x7f70cc09d5e0 2024-09-16T03:59:00.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.313+0000 7f70c97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:59:00.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.313+0000 7f70c97fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:59:00.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.313+0000 7f70c97fa640 10 monclient: dump: 2024-09-16T03:59:00.333 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:59:00.333 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:59:00.333 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:00.333 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:00.333 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:59:00.333 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:59:00.333 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:59:00.333 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:59:00.333 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:59:00.333 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:59:00.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.313+0000 7f70c97fa640 1 -- 172.21.15.42:0/2754167261 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4712 (secure 0 0 0) 0x7f70b4009af0 con 0x7f70cc09d5e0 2024-09-16T03:59:00.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.313+0000 7f70c97fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:59:00.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.313+0000 7f70c97fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:59:00.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.313+0000 7f70c97fa640 1 -- 172.21.15.42:0/2754167261 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f70cc09d5e0 msgr2=0x7f70cc138330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:59:00.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.313+0000 7f70c97fa640 1 --2- 172.21.15.42:0/2754167261 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f70cc09d5e0 0x7f70cc138330 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f70b401ca20 tx=0x7f70b40a7000 comp rx=0 tx=0).stop 2024-09-16T03:59:00.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.313+0000 7f70c97fa640 1 -- 172.21.15.42:0/2754167261 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f70cc09d5e0 msgr2=0x7f70cc138330 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:59:00.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.313+0000 7f70c97fa640 1 --2- 172.21.15.42:0/2754167261 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f70cc09d5e0 0x7f70cc138330 secure :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f70b401ca20 tx=0x7f70b40a7000 comp rx=0 tx=0).stop 2024-09-16T03:59:00.334 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:59:00.334 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:59:00.334 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:59:00.334 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T03:59:00.335 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 154, 2024-09-16T03:59:00.335 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 68494, 2024-09-16T03:59:00.335 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:59:00.335 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:59:00.335 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T03:59:00.335 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T03:59:00.335 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:59:00.335 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 5, 2024-09-16T03:59:00.335 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:59:00.335 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:59:00.335 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:59:00.335 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:59:00.335 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:59:00.335 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:59:00.336 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:59:00.336 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:59:00.336 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:59:00.336 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:59:00.336 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:59:00.336 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:59:00.336 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:59:00.336 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:59:00.336 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:59:00.336 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:59:00.336 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:59:00.336 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:59:00.336 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:59:00.336 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:59:00.337 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:59:00.337 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:59:00.337 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:59:00.337 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:59:00.337 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:59:00.337 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:59:00.337 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:59:00.337 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:59:00.337 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:59:00.337 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:59:00.337 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:59:00.337 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:59:00.337 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:59:00.337 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:59:00.338 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:59:00.338 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:59:00.338 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:59:00.338 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:59:00.338 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:59:00.338 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:59:00.338 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:59:00.338 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:59:00.338 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:59:00.338 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:59:00.339 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:59:00.339 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:59:00.339 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:59:00.339 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:59:00.339 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:59:00.339 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:59:00.339 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:59:00.339 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:59:00.339 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:59:00.339 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:59:00.339 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:59:00.339 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:59:00.339 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:59:00.339 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:59:00.339 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:59:00.340 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:59:00.340 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:59:00.340 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:59:00.340 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:59:00.340 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:59:00.340 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:59:00.340 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:59:00.340 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:59:00.340 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:59:00.340 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:59:00.340 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:59:00.340 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:59:00.340 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:59:00.341 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:59:00.341 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:59:00.341 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:59:00.341 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:59:00.341 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:59:00.341 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:59:00.341 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:59:00.342 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:59:00.342 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:59:00.342 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:59:00.342 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:59:00.342 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:59:00.342 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:59:00.342 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:59:00.342 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:59:00.342 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:59:00.342 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:59:00.342 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:59:00.342 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:59:00.342 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:59:00.342 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:59:00.343 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:59:00.343 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:59:00.343 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:59:00.343 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:59:00.343 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:59:00.343 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:59:00.343 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:59:00.343 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:59:00.343 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:59:00.343 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:59:00.343 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:59:00.343 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:59:00.343 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:59:00.343 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:59:00.344 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:59:00.344 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:59:00.344 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:59:00.344 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:59:00.344 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:59:00.344 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:59:00.344 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:59:00.344 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:59:00.344 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:59:00.344 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:59:00.344 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:59:00.344 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:59:00.344 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:59:00.344 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:59:00.345 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:59:00.345 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:59:00.345 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:59:00.345 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:59:00.345 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:59:00.345 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:59:00.345 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:59:00.345 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:59:00.345 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:59:00.345 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:59:00.345 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:59:00.345 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:59:00.345 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:59:00.345 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:59:00.346 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:59:00.346 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:59:00.346 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:59:00.346 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:59:00.346 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:59:00.346 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:59:00.346 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T03:59:00.346 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:59:00.346 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:59:00.346 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:59:00.346 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:59:00.346 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:59:00.346 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:59:00.346 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:59:00.347 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:59:00.347 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:59:00.347 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:59:00.347 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 2 2024-09-16T03:59:00.347 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:59:00.347 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:59:00.347 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:59:00.347 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:59:00.347 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:59:00.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.313+0000 7f70d3059640 10 librados: watch_flush enter 2024-09-16T03:59:00.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.313+0000 7f70d3059640 10 librados: watch_flush exit 2024-09-16T03:59:00.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.313+0000 7f70d3059640 20 client.924155.objecter shutdown clearing up homeless session... 2024-09-16T03:59:00.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.313+0000 7f70d3059640 10 client.924155.objecter successfully canceled tick 2024-09-16T03:59:00.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.313+0000 7f70d3059640 1 -- 172.21.15.42:0/2754167261 >> v2:172.21.15.154:6812/254845342 conn(0x7f70ac037810 msgr2=0x7f70ac039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:59:00.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.313+0000 7f70d3059640 1 --2- 172.21.15.42:0/2754167261 >> v2:172.21.15.154:6812/254845342 conn(0x7f70ac037810 0x7f70ac039cd0 secure :-1 s=READY pgs=3720 cs=0 l=1 rev1=1 crypto rx=0x7f70bc009af0 tx=0x7f70bc008040 comp rx=0 tx=0).stop 2024-09-16T03:59:00.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.313+0000 7f70d3059640 10 monclient: shutdown 2024-09-16T03:59:00.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.313+0000 7f70d3059640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:59:00.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.313+0000 7f70d3059640 1 -- 172.21.15.42:0/2754167261 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f70cc0a1330 msgr2=0x7f70cc1ced80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:59:00.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.313+0000 7f70d3059640 1 --2- 172.21.15.42:0/2754167261 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f70cc0a1330 0x7f70cc1ced80 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f70c4001ec0 tx=0x7f70c400c890 comp rx=0 tx=0).stop 2024-09-16T03:59:00.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.313+0000 7f70d15f6640 1 -- 172.21.15.42:0/2754167261 reap_dead start 2024-09-16T03:59:00.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.313+0000 7f70d15f6640 1 -- 172.21.15.42:0/2754167261 reap_dead start 2024-09-16T03:59:00.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.313+0000 7f70d3059640 1 -- 172.21.15.42:0/2754167261 shutdown_connections 2024-09-16T03:59:00.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.313+0000 7f70d3059640 1 -- 172.21.15.42:0/2754167261 >> 172.21.15.42:0/2754167261 conn(0x7f70cc0a2420 msgr2=0x7f70cc0a45b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:59:00.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.313+0000 7f70d3059640 1 -- 172.21.15.42:0/2754167261 shutdown_connections 2024-09-16T03:59:00.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.313+0000 7f70d3059640 1 -- 172.21.15.42:0/2754167261 wait complete. 2024-09-16T03:59:00.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.313+0000 7f70d3059640 1 librados: shutdown 2024-09-16T03:59:00.350 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:59:00.351 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-09-16T03:59:00.351 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-16T03:59:00.351 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-09-16T03:59:00.351 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-16T03:59:00.351 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-16T03:59:00.351 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T03:59:00.362 INFO:tasks.ceph.mon.c:Stopped 2024-09-16T03:59:00.362 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T03:59:00.363 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:59:00.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.517+0000 7f236d8a2640 10 monclient: get_monmap_and_config 2024-09-16T03:59:00.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.517+0000 7f236d8a2640 10 monclient: build_initial_monmap 2024-09-16T03:59:00.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.517+0000 7f236d8a2640 10 monclient: monmap: 2024-09-16T03:59:00.523 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:59:00.523 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:59:00.523 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:59:00.521450+0000 2024-09-16T03:59:00.523 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:59:00.521450+0000 2024-09-16T03:59:00.523 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:59:00.523 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:59:00.524 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:59:00.524 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:59:00.524 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:59:00.524 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:59:00.524 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:59:00.524 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:59:00.529 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.517+0000 7f236d8a2640 1 Processor -- start 2024-09-16T03:59:00.529 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.517+0000 7f236d8a2640 1 -- start start 2024-09-16T03:59:00.529 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.517+0000 7f236d8a2640 10 monclient: init 2024-09-16T03:59:00.530 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.517+0000 7f236d8a2640 10 monclient: _reopen_session rank -1 2024-09-16T03:59:00.530 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.517+0000 7f236d8a2640 10 monclient: _add_conns ranks=[0,2,4,3,1] 2024-09-16T03:59:00.530 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.517+0000 7f236d8a2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f23680a1330 addr v1:172.21.15.42:6789/0 2024-09-16T03:59:00.530 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.517+0000 7f236d8a2640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f23680a1710 0x7f23680a1b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:59:00.530 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.517+0000 7f236d8a2640 10 monclient(hunting): picked mon.noname-b con 0x7f23680a1710 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:59:00.530 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.517+0000 7f236d8a2640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f23681393a0 0x7f236813b790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:59:00.530 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.517+0000 7f236d8a2640 10 monclient(hunting): picked mon.noname-c con 0x7f23681393a0 addr v2:172.21.15.154:3300/0 2024-09-16T03:59:00.530 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.517+0000 7f236d8a2640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f236809e4c0 con 0x7f23680a1330 2024-09-16T03:59:00.530 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.517+0000 7f236d8a2640 10 monclient(hunting): start opening mon connection 2024-09-16T03:59:00.530 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.517+0000 7f236d8a2640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f236809edf0 con 0x7f23681393a0 2024-09-16T03:59:00.530 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.517+0000 7f236d8a2640 10 monclient(hunting): start opening mon connection 2024-09-16T03:59:00.530 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.517+0000 7f236d8a2640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f236809cd60 con 0x7f23680a1710 2024-09-16T03:59:00.530 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.517+0000 7f236d8a2640 10 monclient(hunting): _renew_subs 2024-09-16T03:59:00.530 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.517+0000 7f236d8a2640 10 monclient(hunting): authenticate will time out at 1987.490981s 2024-09-16T03:59:00.531 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.517+0000 7f235ffff640 1 -- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f23680a1710 msgr2=0x7f23680a1b10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T03:59:00.531 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.517+0000 7f235ffff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f23680a1710 0x7f23680a1b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:59:00.531 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.517+0000 7f2366ffd640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f23680a1330 0x7f236809f180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:37276/0 (socket says 172.21.15.42:37276) 2024-09-16T03:59:00.531 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.517+0000 7f2366ffd640 1 -- 172.21.15.42:0/3678947539 learned_addr learned my addr 172.21.15.42:0/3678947539 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:59:00.531 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.517+0000 7f23677fe640 1 --2- 172.21.15.42:0/3678947539 >> v2:172.21.15.154:3300/0 conn(0x7f23681393a0 0x7f236813b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:59:00.531 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.517+0000 7f23667fc640 1 -- 172.21.15.42:0/3678947539 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1357493310 0 0) 0x7f236809e4c0 con 0x7f23680a1330 2024-09-16T03:59:00.531 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.517+0000 7f23667fc640 10 monclient(hunting): _negotiate 2024-09-16T03:59:00.531 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.517+0000 7f23667fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:59:00.531 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.517+0000 7f23667fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:59:00.531 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.517+0000 7f23667fc640 10 monclient(hunting): my global_id is 825105 2024-09-16T03:59:00.531 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.517+0000 7f23667fc640 1 -- 172.21.15.42:0/3678947539 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2354003530 con 0x7f23680a1330 2024-09-16T03:59:00.531 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.517+0000 7f23677fe640 10 monclient(hunting): get_auth_request con 0x7f23681393a0 auth_method 0 2024-09-16T03:59:00.531 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.517+0000 7f23677fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:59:00.532 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.517+0000 7f23677fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:59:00.532 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.517+0000 7f23677fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:59:00.532 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.517+0000 7f23677fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:59:00.532 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.517+0000 7f23677fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:59:00.532 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.517+0000 7f23677fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:59:00.532 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.517+0000 7f23667fc640 1 -- 172.21.15.42:0/3678947539 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1230375357 0 0) 0x7f2354003530 con 0x7f23680a1330 2024-09-16T03:59:00.532 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.517+0000 7f23667fc640 1 -- 172.21.15.42:0/3678947539 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f23680a1710 msgr2=0x7f23680a1b10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:59:00.532 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.517+0000 7f23667fc640 1 --2- 172.21.15.42:0/3678947539 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f23680a1710 0x7f23680a1b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:59:00.532 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.517+0000 7f23667fc640 1 -- 172.21.15.42:0/3678947539 >> v2:172.21.15.154:3300/0 conn(0x7f23681393a0 msgr2=0x7f236813b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:59:00.532 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.517+0000 7f23667fc640 1 --2- 172.21.15.42:0/3678947539 >> v2:172.21.15.154:3300/0 conn(0x7f23681393a0 0x7f236813b790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:59:00.533 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.517+0000 7f23667fc640 10 monclient: _finish_hunting 0 2024-09-16T03:59:00.533 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.517+0000 7f23667fc640 1 monclient: found mon.noname-a-legacy 2024-09-16T03:59:00.533 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.517+0000 7f23667fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:59:00.533 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.517+0000 7f23667fc640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.42:6789/0 2024-09-16T03:59:00.533 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.517+0000 7f23667fc640 1 -- 172.21.15.42:0/3678947539 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23680a2050 con 0x7f23680a1330 2024-09-16T03:59:00.533 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.517+0000 7f23667fc640 10 monclient: _finish_auth 0 2024-09-16T03:59:00.533 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.517+0000 7f23667fc640 10 monclient: _check_auth_tickets 2024-09-16T03:59:00.533 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.517+0000 7f23667fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:59:00.533 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.517+0000 7f23667fc640 1 -- 172.21.15.42:0/3678947539 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2177413734 0 0) 0x7f2358001680 con 0x7f23680a1330 2024-09-16T03:59:00.533 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.517+0000 7f236d8a2640 5 monclient: authenticate success, global_id 825105 2024-09-16T03:59:00.533 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.517+0000 7f236d8a2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:59:00.533 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.517+0000 7f23667fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:59:00.533 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.517+0000 7f23667fc640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T03:59:00.534 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.517+0000 7f23667fc640 10 monclient: dump: 2024-09-16T03:59:00.534 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:59:00.534 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:59:00.534 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:00.534 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:00.534 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:59:00.534 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:59:00.534 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:59:00.534 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:59:00.534 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:59:00.534 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:59:00.534 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.517+0000 7f23667fc640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] but i'm connected to v1:172.21.15.42:6789/0, reconnecting 2024-09-16T03:59:00.534 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.517+0000 7f23667fc640 10 monclient: _reopen_session rank -1 2024-09-16T03:59:00.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.517+0000 7f23667fc640 1 -- 172.21.15.42:0/3678947539 >> v1:172.21.15.42:6789/0 conn(0x7f23680a1330 legacy=0x7f236809f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:59:00.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.517+0000 7f23667fc640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:59:00.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.517+0000 7f23667fc640 1 --2- 172.21.15.42:0/3678947539 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f23680a1710 0x7f23540039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:59:00.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.517+0000 7f23667fc640 10 monclient(hunting): picked mon.c con 0x7f23680a1710 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:59:00.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.517+0000 7f23667fc640 1 --2- 172.21.15.42:0/3678947539 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2354003f60 0x7f2354004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:59:00.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.517+0000 7f23667fc640 10 monclient(hunting): picked mon.a con 0x7f2354003f60 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:59:00.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.517+0000 7f23667fc640 1 --2- 172.21.15.42:0/3678947539 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2354004d50 0x7f23540051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:59:00.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.517+0000 7f23667fc640 10 monclient(hunting): picked mon.b con 0x7f2354004d50 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:59:00.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.517+0000 7f23667fc640 10 monclient(hunting): start opening mon connection 2024-09-16T03:59:00.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.517+0000 7f23667fc640 1 -- 172.21.15.42:0/3678947539 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f236809edf0 con 0x7f2354003f60 2024-09-16T03:59:00.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.517+0000 7f23667fc640 10 monclient(hunting): start opening mon connection 2024-09-16T03:59:00.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.517+0000 7f23667fc640 1 -- 172.21.15.42:0/3678947539 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f236809cd60 con 0x7f2354004d50 2024-09-16T03:59:00.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.517+0000 7f23667fc640 10 monclient(hunting): start opening mon connection 2024-09-16T03:59:00.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.517+0000 7f23667fc640 1 -- 172.21.15.42:0/3678947539 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f235400bc40 con 0x7f23680a1710 2024-09-16T03:59:00.536 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.517+0000 7f23667fc640 10 monclient(hunting): _renew_subs 2024-09-16T03:59:00.536 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.517+0000 7f23667fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:59:00.536 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.517+0000 7f236d8a2640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-16T03:59:00.536 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.517+0000 7f2366ffd640 1 -- 172.21.15.42:0/3678947539 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f23680a1710 msgr2=0x7f23540039e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T03:59:00.536 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.517+0000 7f2366ffd640 1 --2- 172.21.15.42:0/3678947539 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f23680a1710 0x7f23540039e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:59:00.536 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.521+0000 7f235ffff640 1 --2- 172.21.15.42:0/3678947539 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2354003f60 0x7f2354004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:59:00.536 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.521+0000 7f235ffff640 10 monclient(hunting): get_auth_request con 0x7f2354003f60 auth_method 0 2024-09-16T03:59:00.536 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.521+0000 7f235ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:59:00.536 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.521+0000 7f235ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:59:00.536 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.521+0000 7f235ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:59:00.536 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.521+0000 7f235ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:59:00.536 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.521+0000 7f235ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:59:00.536 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.521+0000 7f23677fe640 1 --2- 172.21.15.42:0/3678947539 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2354004d50 0x7f23540051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:59:00.536 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.521+0000 7f235ffff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T03:59:00.537 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.521+0000 7f23677fe640 10 monclient(hunting): get_auth_request con 0x7f2354004d50 auth_method 0 2024-09-16T03:59:00.537 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.521+0000 7f23677fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:59:00.537 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.521+0000 7f23677fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:59:00.537 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.521+0000 7f23677fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:59:00.537 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.521+0000 7f23677fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:59:00.537 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.521+0000 7f23677fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:59:00.537 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.521+0000 7f23677fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T03:59:00.537 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.521+0000 7f235ffff640 10 monclient(hunting): handle_auth_done global_id 825105 payload 293 2024-09-16T03:59:00.537 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.521+0000 7f235ffff640 1 -- 172.21.15.42:0/3678947539 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f23680a1710 msgr2=0x7f23540039e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:59:00.537 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.521+0000 7f235ffff640 1 --2- 172.21.15.42:0/3678947539 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f23680a1710 0x7f23540039e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:59:00.537 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.521+0000 7f235ffff640 1 -- 172.21.15.42:0/3678947539 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2354004d50 msgr2=0x7f23540051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:59:00.537 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.521+0000 7f235ffff640 1 --2- 172.21.15.42:0/3678947539 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2354004d50 0x7f23540051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:59:00.537 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.521+0000 7f235ffff640 10 monclient: _finish_hunting 0 2024-09-16T03:59:00.537 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.521+0000 7f235ffff640 1 monclient: found mon.a 2024-09-16T03:59:00.538 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.521+0000 7f235ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:59:00.538 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.521+0000 7f235ffff640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:59:00.538 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.521+0000 7f235ffff640 1 -- 172.21.15.42:0/3678947539 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2354003530 con 0x7f2354003f60 2024-09-16T03:59:00.538 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.521+0000 7f235ffff640 10 monclient: _finish_auth 0 2024-09-16T03:59:00.538 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.521+0000 7f235ffff640 10 monclient: _check_auth_tickets 2024-09-16T03:59:00.538 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.521+0000 7f235ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:59:00.538 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.521+0000 7f235ffff640 1 --2- 172.21.15.42:0/3678947539 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2354003f60 0x7f2354004810 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f2350009900 tx=0x7f235001c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c509797d4deb6fbb server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:59:00.538 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.521+0000 7f23667fc640 1 -- 172.21.15.42:0/3678947539 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2350009500 con 0x7f2354003f60 2024-09-16T03:59:00.538 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.521+0000 7f23667fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:59:00.538 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.521+0000 7f23667fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:59:00.538 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.521+0000 7f23667fc640 10 monclient: dump: 2024-09-16T03:59:00.538 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:59:00.538 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:59:00.538 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:00.538 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:00.539 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:59:00.539 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:59:00.539 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:59:00.539 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:59:00.539 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:59:00.539 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:59:00.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.521+0000 7f23667fc640 1 -- 172.21.15.42:0/3678947539 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2350003bb0 con 0x7f2354003f60 2024-09-16T03:59:00.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.521+0000 7f236d8a2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:59:00.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.521+0000 7f23667fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:59:00.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.521+0000 7f236d8a2640 10 monclient: get_monmap_and_config success 2024-09-16T03:59:00.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.521+0000 7f236d8a2640 4 set_mon_vals no callback set 2024-09-16T03:59:00.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.521+0000 7f23667fc640 1 -- 172.21.15.42:0/3678947539 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2350005070 con 0x7f2354003f60 2024-09-16T03:59:00.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.521+0000 7f23667fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:59:00.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.521+0000 7f23667fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:59:00.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.521+0000 7f23667fc640 10 monclient: dump: 2024-09-16T03:59:00.540 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:59:00.540 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:59:00.540 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:00.540 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:00.540 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:59:00.540 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:59:00.540 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:59:00.540 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:59:00.540 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:59:00.540 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:59:00.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.521+0000 7f236d8a2640 10 monclient: shutdown 2024-09-16T03:59:00.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.521+0000 7f236d8a2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:59:00.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.521+0000 7f236d8a2640 1 -- 172.21.15.42:0/3678947539 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2354003f60 msgr2=0x7f2354004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:59:00.541 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.521+0000 7f236d8a2640 1 --2- 172.21.15.42:0/3678947539 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2354003f60 0x7f2354004810 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f2350009900 tx=0x7f235001c400 comp rx=0 tx=0).stop 2024-09-16T03:59:00.541 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.521+0000 7f23677fe640 1 -- 172.21.15.42:0/3678947539 reap_dead start 2024-09-16T03:59:00.541 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.521+0000 7f236d8a2640 1 -- 172.21.15.42:0/3678947539 shutdown_connections 2024-09-16T03:59:00.541 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.521+0000 7f236d8a2640 1 -- 172.21.15.42:0/3678947539 >> 172.21.15.42:0/3678947539 conn(0x7f23680a2420 msgr2=0x7f23680a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:59:00.541 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.521+0000 7f236d8a2640 1 -- 172.21.15.42:0/3678947539 shutdown_connections 2024-09-16T03:59:00.541 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.521+0000 7f236d8a2640 1 -- 172.21.15.42:0/3678947539 wait complete. 2024-09-16T03:59:00.541 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.525+0000 7f236d8a2640 10 monclient: build_initial_monmap 2024-09-16T03:59:00.541 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.525+0000 7f236d8a2640 10 monclient: monmap: 2024-09-16T03:59:00.541 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:59:00.541 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:59:00.541 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:59:00.541 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:59:00.541 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:59:00.541 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:59:00.542 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:59:00.542 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:59:00.542 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:59:00.542 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:59:00.542 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.525+0000 7f236d8a2640 1 librados: starting msgr at 2024-09-16T03:59:00.542 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.525+0000 7f236d8a2640 1 librados: starting objecter 2024-09-16T03:59:00.542 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.525+0000 7f236d8a2640 1 Processor -- start 2024-09-16T03:59:00.542 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.525+0000 7f236d8a2640 1 -- start start 2024-09-16T03:59:00.542 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.525+0000 7f236d8a2640 1 librados: setting wanted keys 2024-09-16T03:59:00.542 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.525+0000 7f236d8a2640 1 librados: calling monclient init 2024-09-16T03:59:00.542 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.525+0000 7f236d8a2640 10 monclient: init 2024-09-16T03:59:00.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.525+0000 7f236d8a2640 10 monclient: _reopen_session rank -1 2024-09-16T03:59:00.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.525+0000 7f236d8a2640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:59:00.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.525+0000 7f236d8a2640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2354003f60 0x7f23681cf960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:59:00.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.525+0000 7f236d8a2640 10 monclient(hunting): picked mon.noname-b con 0x7f2354003f60 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:59:00.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.525+0000 7f236d8a2640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f23681cfea0 0x7f23681d02e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:59:00.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.525+0000 7f236d8a2640 10 monclient(hunting): picked mon.noname-a con 0x7f23681cfea0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:59:00.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.525+0000 7f236d8a2640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f236811f620 0x7f236811faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:59:00.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.525+0000 7f236d8a2640 10 monclient(hunting): picked mon.noname-c con 0x7f236811f620 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:59:00.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.525+0000 7f236d8a2640 10 monclient(hunting): start opening mon connection 2024-09-16T03:59:00.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.525+0000 7f236d8a2640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f236813c870 con 0x7f23681cfea0 2024-09-16T03:59:00.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.525+0000 7f236d8a2640 10 monclient(hunting): start opening mon connection 2024-09-16T03:59:00.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.525+0000 7f236d8a2640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f236813c6f0 con 0x7f2354003f60 2024-09-16T03:59:00.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.525+0000 7f236d8a2640 10 monclient(hunting): start opening mon connection 2024-09-16T03:59:00.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.525+0000 7f236d8a2640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f236813c9f0 con 0x7f236811f620 2024-09-16T03:59:00.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.525+0000 7f236d8a2640 10 monclient(hunting): _renew_subs 2024-09-16T03:59:00.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.525+0000 7f236d8a2640 10 monclient(hunting): authenticate will time out at 1987.497100s 2024-09-16T03:59:00.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.525+0000 7f23677fe640 1 -- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f236811f620 msgr2=0x7f236811faa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T03:59:00.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.525+0000 7f23677fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f236811f620 0x7f236811faa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:59:00.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.525+0000 7f235ffff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f23681cfea0 0x7f23681d02e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:59:00.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.525+0000 7f235ffff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f23681cfea0 0x7f23681d02e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:33592/0 (socket says 172.21.15.42:33592) 2024-09-16T03:59:00.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.525+0000 7f235ffff640 1 -- 172.21.15.42:0/2744277092 learned_addr learned my addr 172.21.15.42:0/2744277092 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:59:00.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.525+0000 7f235ffff640 10 monclient(hunting): get_auth_request con 0x7f23681cfea0 auth_method 0 2024-09-16T03:59:00.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.525+0000 7f235ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:59:00.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.525+0000 7f235ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:59:00.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.525+0000 7f235ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:59:00.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.525+0000 7f235ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:59:00.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.525+0000 7f235ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:59:00.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.525+0000 7f235ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:59:00.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.525+0000 7f2366ffd640 1 --2- 172.21.15.42:0/2744277092 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2354003f60 0x7f23681cf960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:59:00.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.525+0000 7f2366ffd640 10 monclient(hunting): get_auth_request con 0x7f2354003f60 auth_method 0 2024-09-16T03:59:00.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.525+0000 7f2366ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:59:00.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.525+0000 7f2366ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:59:00.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.525+0000 7f2366ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:59:00.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.525+0000 7f235ffff640 10 monclient(hunting): handle_auth_done global_id 825108 payload 824 2024-09-16T03:59:00.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.525+0000 7f235ffff640 1 -- 172.21.15.42:0/2744277092 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f236811f620 msgr2=0x7f236811faa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:59:00.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.525+0000 7f235ffff640 1 --2- 172.21.15.42:0/2744277092 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f236811f620 0x7f236811faa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:59:00.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.525+0000 7f235ffff640 1 -- 172.21.15.42:0/2744277092 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2354003f60 msgr2=0x7f23681cf960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:59:00.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.525+0000 7f235ffff640 1 --2- 172.21.15.42:0/2744277092 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2354003f60 0x7f23681cf960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:59:00.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.525+0000 7f235ffff640 10 monclient: _finish_hunting 0 2024-09-16T03:59:00.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.525+0000 7f235ffff640 1 monclient: found mon.noname-a 2024-09-16T03:59:00.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.525+0000 7f235ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:59:00.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.525+0000 7f235ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:59:00.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.525+0000 7f235ffff640 1 -- 172.21.15.42:0/2744277092 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2368123230 con 0x7f23681cfea0 2024-09-16T03:59:00.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.525+0000 7f2366ffd640 1 --2- 172.21.15.42:0/2744277092 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2354003f60 0x7f23681cf960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T03:59:00.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.525+0000 7f235ffff640 1 --2- 172.21.15.42:0/2744277092 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f23681cfea0 0x7f23681d02e0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f235001c230 tx=0x7f2354005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:59:00.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.525+0000 7f2364ff9640 10 client.?.objecter ms_handle_connect 0x7f23681cfea0 2024-09-16T03:59:00.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.525+0000 7f2364ff9640 10 client.?.objecter resend_mon_ops 2024-09-16T03:59:00.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.525+0000 7f2364ff9640 1 -- 172.21.15.42:0/2744277092 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23500bc070 con 0x7f23681cfea0 2024-09-16T03:59:00.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.525+0000 7f2364ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:59:00.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.525+0000 7f2364ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:59:00.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.525+0000 7f2364ff9640 10 monclient: dump: 2024-09-16T03:59:00.550 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:59:00.550 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:59:00.550 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:00.550 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:00.550 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:59:00.550 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:59:00.550 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:59:00.550 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:59:00.550 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:59:00.550 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:59:00.551 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.525+0000 7f2364ff9640 10 monclient: _finish_auth 0 2024-09-16T03:59:00.551 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.525+0000 7f2364ff9640 10 monclient: _check_auth_tickets 2024-09-16T03:59:00.551 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.525+0000 7f2364ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:59:00.551 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.525+0000 7f2364ff9640 1 -- 172.21.15.42:0/2744277092 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2350003970 con 0x7f23681cfea0 2024-09-16T03:59:00.551 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.525+0000 7f2364ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T03:59:00.551 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.525+0000 7f2364ff9640 1 -- 172.21.15.42:0/2744277092 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2350006dd0 con 0x7f23681cfea0 2024-09-16T03:59:00.551 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.525+0000 7f236d8a2640 5 monclient: authenticate success, global_id 825108 2024-09-16T03:59:00.551 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.525+0000 7f236d8a2640 10 monclient: _renew_subs 2024-09-16T03:59:00.551 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.525+0000 7f236d8a2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:59:00.551 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.525+0000 7f236d8a2640 1 -- 172.21.15.42:0/2744277092 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2368121680 con 0x7f23681cfea0 2024-09-16T03:59:00.551 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.525+0000 7f2365ffb640 4 set_mon_vals no callback set 2024-09-16T03:59:00.551 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.525+0000 7f2364ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:59:00.551 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.525+0000 7f2364ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:59:00.551 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.525+0000 7f2364ff9640 10 monclient: dump: 2024-09-16T03:59:00.552 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:59:00.552 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:59:00.552 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:00.552 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:00.552 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:59:00.552 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:59:00.552 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:59:00.552 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:59:00.552 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:59:00.552 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:59:00.552 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.525+0000 7f236d8a2640 10 client.825108.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:59:00.552 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.525+0000 7f236d8a2640 10 monclient: _renew_subs 2024-09-16T03:59:00.552 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.525+0000 7f236d8a2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:59:00.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.525+0000 7f236d8a2640 1 -- 172.21.15.42:0/2744277092 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2368073a30 con 0x7f23681cfea0 2024-09-16T03:59:00.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.525+0000 7f236d8a2640 1 librados: init done 2024-09-16T03:59:00.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.525+0000 7f236d8a2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:59:00.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.525+0000 7f236d8a2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:59:00.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.525+0000 7f236d8a2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:59:00.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.525+0000 7f2364ff9640 1 -- 172.21.15.42:0/2744277092 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f23500bf070 con 0x7f23681cfea0 2024-09-16T03:59:00.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.525+0000 7f236d8a2640 1 -- 172.21.15.42:0/2744277092 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f236809f140 con 0x7f23681cfea0 2024-09-16T03:59:00.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.525+0000 7f2364ff9640 1 --2- 172.21.15.42:0/2744277092 >> v2:172.21.15.154:6812/254845342 conn(0x7f23480377f0 0x7f2348039cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:59:00.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.525+0000 7f2364ff9640 1 -- 172.21.15.42:0/2744277092 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(516..516 src has 451..516) ==== 5258+0+0 (secure 0 0 0) 0x7f23500e1fa0 con 0x7f23681cfea0 2024-09-16T03:59:00.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.525+0000 7f2364ff9640 10 client.825108.objecter ms_dispatch 0x7f2368000ba0 osd_map(516..516 src has 451..516) 2024-09-16T03:59:00.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.525+0000 7f2364ff9640 3 client.825108.objecter handle_osd_map got epochs [516,516] > 0 2024-09-16T03:59:00.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.525+0000 7f2364ff9640 3 client.825108.objecter handle_osd_map decoding full epoch 516 2024-09-16T03:59:00.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.525+0000 7f2364ff9640 20 client.825108.objecter dump_active .. 0 homeless 2024-09-16T03:59:00.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.525+0000 7f2366ffd640 1 --2- 172.21.15.42:0/2744277092 >> v2:172.21.15.154:6812/254845342 conn(0x7f23480377f0 0x7f2348039cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:59:00.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.525+0000 7f2366ffd640 10 monclient: get_auth_request con 0x7f23480377f0 auth_method 0 2024-09-16T03:59:00.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.529+0000 7f2366ffd640 1 --2- 172.21.15.42:0/2744277092 >> v2:172.21.15.154:6812/254845342 conn(0x7f23480377f0 0x7f2348039cb0 secure :-1 s=READY pgs=3721 cs=0 l=1 rev1=1 crypto rx=0x7f2358001a50 tx=0x7f2358001d40 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:59:00.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.529+0000 7f2364ff9640 10 client.825108.objecter ms_handle_connect 0x7f23480377f0 2024-09-16T03:59:00.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.533+0000 7f2364ff9640 1 -- 172.21.15.42:0/2744277092 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f2350016110 con 0x7f23681cfea0 2024-09-16T03:59:00.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.533+0000 7f2364ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:59:00.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.533+0000 7f2364ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:59:00.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.753+0000 7f236d8a2640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:59:00.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.753+0000 7f236d8a2640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:59:00.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.753+0000 7f236d8a2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:59:00.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.753+0000 7f236d8a2640 1 -- 172.21.15.42:0/2744277092 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f2368073f80 con 0x7f23681cfea0 2024-09-16T03:59:00.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.753+0000 7f2364ff9640 1 -- 172.21.15.42:0/2744277092 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7f23500aba20 con 0x7f23681cfea0 2024-09-16T03:59:00.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.753+0000 7f2364ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:59:00.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.753+0000 7f2364ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:59:00.760 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":154,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:59:00.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.753+0000 7f236d8a2640 10 librados: watch_flush enter 2024-09-16T03:59:00.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.753+0000 7f236d8a2640 10 librados: watch_flush exit 2024-09-16T03:59:00.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.753+0000 7f236d8a2640 20 client.825108.objecter shutdown clearing up homeless session... 2024-09-16T03:59:00.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.753+0000 7f236d8a2640 10 client.825108.objecter successfully canceled tick 2024-09-16T03:59:00.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.753+0000 7f236d8a2640 1 -- 172.21.15.42:0/2744277092 >> v2:172.21.15.154:6812/254845342 conn(0x7f23480377f0 msgr2=0x7f2348039cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:59:00.763 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.753+0000 7f236d8a2640 1 --2- 172.21.15.42:0/2744277092 >> v2:172.21.15.154:6812/254845342 conn(0x7f23480377f0 0x7f2348039cb0 secure :-1 s=READY pgs=3721 cs=0 l=1 rev1=1 crypto rx=0x7f2358001a50 tx=0x7f2358001d40 comp rx=0 tx=0).stop 2024-09-16T03:59:00.763 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.753+0000 7f236d8a2640 10 monclient: shutdown 2024-09-16T03:59:00.763 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.753+0000 7f236d8a2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:59:00.763 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.753+0000 7f236d8a2640 1 -- 172.21.15.42:0/2744277092 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f23681cfea0 msgr2=0x7f23681d02e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:59:00.763 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.753+0000 7f236d8a2640 1 --2- 172.21.15.42:0/2744277092 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f23681cfea0 0x7f23681d02e0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f235001c230 tx=0x7f2354005840 comp rx=0 tx=0).stop 2024-09-16T03:59:00.763 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.753+0000 7f236d8a2640 1 -- 172.21.15.42:0/2744277092 shutdown_connections 2024-09-16T03:59:00.763 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.753+0000 7f236d8a2640 1 --2- 172.21.15.42:0/2744277092 >> v2:172.21.15.154:6812/254845342 conn(0x7f23480377f0 0x7f2348039cb0 unknown :-1 s=CLOSED pgs=3721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:59:00.763 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.753+0000 7f236d8a2640 1 --2- 172.21.15.42:0/2744277092 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f236811f620 0x7f236811faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:59:00.763 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.753+0000 7f236d8a2640 1 --2- 172.21.15.42:0/2744277092 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f23681cfea0 0x7f23681d02e0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:59:00.763 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.753+0000 7f236d8a2640 1 --2- 172.21.15.42:0/2744277092 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2354003f60 0x7f23681cf960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:59:00.763 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.753+0000 7f236d8a2640 1 -- 172.21.15.42:0/2744277092 >> 172.21.15.42:0/2744277092 conn(0x7f23680a2420 msgr2=0x7f2368134850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:59:00.763 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.753+0000 7f236d8a2640 1 -- 172.21.15.42:0/2744277092 shutdown_connections 2024-09-16T03:59:00.763 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.753+0000 7f236d8a2640 1 -- 172.21.15.42:0/2744277092 wait complete. 2024-09-16T03:59:00.763 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:00.753+0000 7f236d8a2640 1 librados: shutdown 2024-09-16T03:59:03.777 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:59:03.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.901+0000 7f8b685c0640 10 monclient: get_monmap_and_config 2024-09-16T03:59:03.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.901+0000 7f8b685c0640 10 monclient: build_initial_monmap 2024-09-16T03:59:03.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.901+0000 7f8b685c0640 10 monclient: monmap: 2024-09-16T03:59:03.908 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:59:03.908 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:59:03.908 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:59:03.906115+0000 2024-09-16T03:59:03.908 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:59:03.906115+0000 2024-09-16T03:59:03.908 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:59:03.908 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:59:03.908 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:59:03.908 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:59:03.908 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:59:03.908 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:59:03.908 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:59:03.908 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:59:03.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.901+0000 7f8b685c0640 1 Processor -- start 2024-09-16T03:59:03.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.901+0000 7f8b685c0640 1 -- start start 2024-09-16T03:59:03.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.901+0000 7f8b685c0640 10 monclient: init 2024-09-16T03:59:03.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.901+0000 7f8b685c0640 10 monclient: _reopen_session rank -1 2024-09-16T03:59:03.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.901+0000 7f8b685c0640 10 monclient: _add_conns ranks=[4,0,1,2,3] 2024-09-16T03:59:03.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.901+0000 7f8b685c0640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f8b600a1330 0x7f8b600a1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:59:03.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.901+0000 7f8b685c0640 10 monclient(hunting): picked mon.noname-c con 0x7f8b600a1330 addr v2:172.21.15.154:3300/0 2024-09-16T03:59:03.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.901+0000 7f8b685c0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8b6006dac0 addr v1:172.21.15.42:6789/0 2024-09-16T03:59:03.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.901+0000 7f8b685c0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8b600a1c50 addr v1:172.21.15.154:6789/0 2024-09-16T03:59:03.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.901+0000 7f8b685c0640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8b60135f20 con 0x7f8b6006dac0 2024-09-16T03:59:03.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.901+0000 7f8b685c0640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8b6009c090 con 0x7f8b600a1c50 2024-09-16T03:59:03.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.901+0000 7f8b685c0640 10 monclient(hunting): start opening mon connection 2024-09-16T03:59:03.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.901+0000 7f8b685c0640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f8b6009f610 con 0x7f8b600a1330 2024-09-16T03:59:03.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.901+0000 7f8b685c0640 10 monclient(hunting): _renew_subs 2024-09-16T03:59:03.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.901+0000 7f8b685c0640 10 monclient(hunting): authenticate will time out at 1990.875500s 2024-09-16T03:59:03.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.901+0000 7f8b65b5b640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f8b6006dac0 0x7f8b60135100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:37284/0 (socket says 172.21.15.42:37284) 2024-09-16T03:59:03.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.901+0000 7f8b65b5b640 1 -- 172.21.15.42:0/1486010635 learned_addr learned my addr 172.21.15.42:0/1486010635 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:59:03.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.905+0000 7f8b6635c640 1 --2- 172.21.15.42:0/1486010635 >> v2:172.21.15.154:3300/0 conn(0x7f8b600a1330 0x7f8b600a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:59:03.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.905+0000 7f8b6535a640 1 -- 172.21.15.42:0/1486010635 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3284290924 0 0) 0x7f8b60135f20 con 0x7f8b6006dac0 2024-09-16T03:59:03.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.905+0000 7f8b6535a640 10 monclient(hunting): _negotiate 2024-09-16T03:59:03.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.905+0000 7f8b6535a640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:59:03.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.905+0000 7f8b6535a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:59:03.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.905+0000 7f8b6535a640 10 monclient(hunting): my global_id is 825111 2024-09-16T03:59:03.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.905+0000 7f8b6535a640 1 -- 172.21.15.42:0/1486010635 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8b4c003530 con 0x7f8b6006dac0 2024-09-16T03:59:03.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.905+0000 7f8b6635c640 10 monclient(hunting): get_auth_request con 0x7f8b600a1330 auth_method 0 2024-09-16T03:59:03.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.905+0000 7f8b6635c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:59:03.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.905+0000 7f8b6635c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:59:03.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.905+0000 7f8b6635c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:59:03.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.905+0000 7f8b6635c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:59:03.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.905+0000 7f8b6635c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:59:03.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.905+0000 7f8b6635c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:59:03.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.905+0000 7f8b6535a640 1 -- 172.21.15.42:0/1486010635 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1762046299 0 0) 0x7f8b4c003530 con 0x7f8b6006dac0 2024-09-16T03:59:03.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.905+0000 7f8b6535a640 1 -- 172.21.15.42:0/1486010635 >> v2:172.21.15.154:3300/0 conn(0x7f8b600a1330 msgr2=0x7f8b600a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:59:03.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.905+0000 7f8b6535a640 1 --2- 172.21.15.42:0/1486010635 >> v2:172.21.15.154:3300/0 conn(0x7f8b600a1330 0x7f8b600a1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:59:03.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.905+0000 7f8b6535a640 1 -- 172.21.15.42:0/1486010635 >> v1:172.21.15.154:6789/0 conn(0x7f8b600a1c50 legacy=0x7f8b60135810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:59:03.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.905+0000 7f8b6535a640 10 monclient: _finish_hunting 0 2024-09-16T03:59:03.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.905+0000 7f8b6535a640 1 monclient: found mon.noname-a-legacy 2024-09-16T03:59:03.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.905+0000 7f8b6535a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:59:03.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.905+0000 7f8b6535a640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.42:6789/0 2024-09-16T03:59:03.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.905+0000 7f8b6535a640 1 -- 172.21.15.42:0/1486010635 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b6009f8f0 con 0x7f8b6006dac0 2024-09-16T03:59:03.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.905+0000 7f8b6535a640 10 monclient: _finish_auth 0 2024-09-16T03:59:03.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.905+0000 7f8b6535a640 10 monclient: _check_auth_tickets 2024-09-16T03:59:03.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.905+0000 7f8b6535a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:59:03.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.905+0000 7f8b6535a640 1 -- 172.21.15.42:0/1486010635 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2177413734 0 0) 0x7f8b5c001680 con 0x7f8b6006dac0 2024-09-16T03:59:03.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.905+0000 7f8b6535a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:59:03.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.905+0000 7f8b6535a640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T03:59:03.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.905+0000 7f8b6535a640 10 monclient: dump: 2024-09-16T03:59:03.915 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:59:03.915 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:59:03.915 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:03.915 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:03.916 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:59:03.916 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:59:03.916 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:59:03.916 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:59:03.916 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:59:03.916 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:59:03.916 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.905+0000 7f8b6535a640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] but i'm connected to v1:172.21.15.42:6789/0, reconnecting 2024-09-16T03:59:03.916 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.905+0000 7f8b6535a640 10 monclient: _reopen_session rank -1 2024-09-16T03:59:03.916 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.905+0000 7f8b6535a640 1 -- 172.21.15.42:0/1486010635 >> v1:172.21.15.42:6789/0 conn(0x7f8b6006dac0 legacy=0x7f8b60135100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:59:03.916 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.905+0000 7f8b6535a640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:59:03.916 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.905+0000 7f8b6535a640 1 --2- 172.21.15.42:0/1486010635 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8b4c0039c0 0x7f8b4c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:59:03.916 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.905+0000 7f8b6535a640 10 monclient(hunting): picked mon.b con 0x7f8b4c0039c0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:59:03.916 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.905+0000 7f8b6535a640 1 --2- 172.21.15.42:0/1486010635 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8b4c004810 0x7f8b4c004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:59:03.917 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.905+0000 7f8b6535a640 10 monclient(hunting): picked mon.a con 0x7f8b4c004810 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:59:03.917 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.905+0000 7f8b6535a640 1 --2- 172.21.15.42:0/1486010635 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f8b4c0051d0 0x7f8b4c00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:59:03.917 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.905+0000 7f8b6535a640 10 monclient(hunting): picked mon.c con 0x7f8b4c0051d0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:59:03.917 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.905+0000 7f8b6535a640 10 monclient(hunting): start opening mon connection 2024-09-16T03:59:03.917 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.905+0000 7f8b6535a640 1 -- 172.21.15.42:0/1486010635 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f8b6009f610 con 0x7f8b4c004810 2024-09-16T03:59:03.917 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.905+0000 7f8b6535a640 10 monclient(hunting): start opening mon connection 2024-09-16T03:59:03.917 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.905+0000 7f8b6535a640 1 -- 172.21.15.42:0/1486010635 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f8b4c00c170 con 0x7f8b4c0039c0 2024-09-16T03:59:03.917 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.905+0000 7f8b6535a640 10 monclient(hunting): start opening mon connection 2024-09-16T03:59:03.917 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.905+0000 7f8b6535a640 1 -- 172.21.15.42:0/1486010635 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f8b4c00c370 con 0x7f8b4c0051d0 2024-09-16T03:59:03.917 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.905+0000 7f8b6535a640 10 monclient(hunting): _renew_subs 2024-09-16T03:59:03.917 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.905+0000 7f8b6535a640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:59:03.917 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.905+0000 7f8b66b5d640 1 -- 172.21.15.42:0/1486010635 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f8b4c0051d0 msgr2=0x7f8b4c00b9d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T03:59:03.917 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.905+0000 7f8b65b5b640 1 --2- 172.21.15.42:0/1486010635 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8b4c004810 0x7f8b4c004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:59:03.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.905+0000 7f8b66b5d640 1 --2- 172.21.15.42:0/1486010635 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f8b4c0051d0 0x7f8b4c00b9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:59:03.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.905+0000 7f8b65b5b640 10 monclient(hunting): get_auth_request con 0x7f8b4c004810 auth_method 0 2024-09-16T03:59:03.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.905+0000 7f8b65b5b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:59:03.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.905+0000 7f8b65b5b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:59:03.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.905+0000 7f8b65b5b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:59:03.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.905+0000 7f8b65b5b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:59:03.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.905+0000 7f8b65b5b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:59:03.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.905+0000 7f8b65b5b640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T03:59:03.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.905+0000 7f8b65b5b640 10 monclient(hunting): handle_auth_done global_id 825111 payload 293 2024-09-16T03:59:03.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.905+0000 7f8b65b5b640 1 -- 172.21.15.42:0/1486010635 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f8b4c0051d0 msgr2=0x7f8b4c00b9d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:59:03.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.905+0000 7f8b65b5b640 1 --2- 172.21.15.42:0/1486010635 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f8b4c0051d0 0x7f8b4c00b9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:59:03.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.905+0000 7f8b6635c640 1 --2- 172.21.15.42:0/1486010635 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8b4c0039c0 0x7f8b4c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:59:03.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.905+0000 7f8b65b5b640 1 -- 172.21.15.42:0/1486010635 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8b4c0039c0 msgr2=0x7f8b4c003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:59:03.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.905+0000 7f8b65b5b640 1 --2- 172.21.15.42:0/1486010635 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8b4c0039c0 0x7f8b4c003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:59:03.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.905+0000 7f8b65b5b640 10 monclient: _finish_hunting 0 2024-09-16T03:59:03.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.905+0000 7f8b65b5b640 1 monclient: found mon.a 2024-09-16T03:59:03.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.905+0000 7f8b66b5d640 1 -- 172.21.15.42:0/1486010635 reap_dead start 2024-09-16T03:59:03.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.905+0000 7f8b65b5b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:59:03.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.905+0000 7f8b65b5b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:59:03.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.905+0000 7f8b65b5b640 1 -- 172.21.15.42:0/1486010635 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b4c003530 con 0x7f8b4c004810 2024-09-16T03:59:03.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.905+0000 7f8b65b5b640 10 monclient: _finish_auth 0 2024-09-16T03:59:03.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.905+0000 7f8b65b5b640 10 monclient: _check_auth_tickets 2024-09-16T03:59:03.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.905+0000 7f8b65b5b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:59:03.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.905+0000 7f8b685c0640 5 monclient: authenticate success, global_id 825111 2024-09-16T03:59:03.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.905+0000 7f8b685c0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:59:03.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.905+0000 7f8b6635c640 10 monclient: get_auth_request con 0x7f8b4c0039c0 auth_method 0 2024-09-16T03:59:03.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.905+0000 7f8b6635c640 1 --2- 172.21.15.42:0/1486010635 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8b4c0039c0 0x7f8b4c003de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T03:59:03.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.905+0000 7f8b65b5b640 1 --2- 172.21.15.42:0/1486010635 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8b4c004810 0x7f8b4c004c90 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f8b5c00c240 tx=0x7f8b5c01cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4dc0a67c0f1b54e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:59:03.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.905+0000 7f8b6535a640 1 -- 172.21.15.42:0/1486010635 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b5c007d20 con 0x7f8b4c004810 2024-09-16T03:59:03.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.905+0000 7f8b6535a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:59:03.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.905+0000 7f8b6535a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:59:03.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.905+0000 7f8b6535a640 10 monclient: dump: 2024-09-16T03:59:03.927 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:59:03.927 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:59:03.928 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:03.928 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:03.928 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:59:03.928 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:59:03.928 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:59:03.928 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:59:03.928 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:59:03.928 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:59:03.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.905+0000 7f8b6535a640 1 -- 172.21.15.42:0/1486010635 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8b5c0a9070 con 0x7f8b4c004810 2024-09-16T03:59:03.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.905+0000 7f8b6535a640 10 monclient: handle_config config(0 keys) 2024-09-16T03:59:03.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.905+0000 7f8b6535a640 1 -- 172.21.15.42:0/1486010635 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b5c005d40 con 0x7f8b4c004810 2024-09-16T03:59:03.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.905+0000 7f8b6535a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:59:03.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.905+0000 7f8b6535a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:59:03.929 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.905+0000 7f8b6535a640 10 monclient: dump: 2024-09-16T03:59:03.929 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:59:03.929 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:59:03.929 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:03.929 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:03.929 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:59:03.929 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:59:03.929 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:59:03.929 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:59:03.929 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:59:03.929 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:59:03.929 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.905+0000 7f8b685c0640 10 monclient: get_monmap_and_config success 2024-09-16T03:59:03.929 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.905+0000 7f8b685c0640 4 set_mon_vals no callback set 2024-09-16T03:59:03.929 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.905+0000 7f8b685c0640 10 monclient: shutdown 2024-09-16T03:59:03.930 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.905+0000 7f8b685c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:59:03.930 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.905+0000 7f8b685c0640 1 -- 172.21.15.42:0/1486010635 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8b4c004810 msgr2=0x7f8b4c004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:59:03.930 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.905+0000 7f8b685c0640 1 --2- 172.21.15.42:0/1486010635 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8b4c004810 0x7f8b4c004c90 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f8b5c00c240 tx=0x7f8b5c01cc30 comp rx=0 tx=0).stop 2024-09-16T03:59:03.930 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.905+0000 7f8b685c0640 1 -- 172.21.15.42:0/1486010635 shutdown_connections 2024-09-16T03:59:03.930 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.905+0000 7f8b685c0640 1 --2- 172.21.15.42:0/1486010635 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8b4c004810 0x7f8b4c004c90 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:59:03.930 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.905+0000 7f8b685c0640 1 -- 172.21.15.42:0/1486010635 >> 172.21.15.42:0/1486010635 conn(0x7f8b600a2420 msgr2=0x7f8b600a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:59:03.930 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.909+0000 7f8b685c0640 1 -- 172.21.15.42:0/1486010635 shutdown_connections 2024-09-16T03:59:03.930 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.909+0000 7f8b685c0640 1 -- 172.21.15.42:0/1486010635 wait complete. 2024-09-16T03:59:03.930 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.909+0000 7f8b685c0640 10 monclient: build_initial_monmap 2024-09-16T03:59:03.930 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.909+0000 7f8b685c0640 10 monclient: monmap: 2024-09-16T03:59:03.930 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:59:03.930 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:59:03.930 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:59:03.931 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:59:03.931 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:59:03.931 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:59:03.931 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:59:03.931 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:59:03.931 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:59:03.931 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:59:03.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.909+0000 7f8b685c0640 1 librados: starting msgr at 2024-09-16T03:59:03.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.909+0000 7f8b685c0640 1 librados: starting objecter 2024-09-16T03:59:03.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.909+0000 7f8b685c0640 1 Processor -- start 2024-09-16T03:59:03.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.909+0000 7f8b685c0640 1 -- start start 2024-09-16T03:59:03.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.909+0000 7f8b685c0640 1 librados: setting wanted keys 2024-09-16T03:59:03.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.909+0000 7f8b685c0640 1 librados: calling monclient init 2024-09-16T03:59:03.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.909+0000 7f8b685c0640 10 monclient: init 2024-09-16T03:59:03.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.909+0000 7f8b685c0640 10 monclient: _reopen_session rank -1 2024-09-16T03:59:03.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.909+0000 7f8b685c0640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T03:59:03.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.909+0000 7f8b685c0640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8b4c004810 0x7f8b601cd760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:59:03.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.909+0000 7f8b685c0640 10 monclient(hunting): picked mon.noname-b con 0x7f8b4c004810 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:59:03.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.909+0000 7f8b685c0640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f8b601cdca0 0x7f8b60122a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:59:03.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.909+0000 7f8b685c0640 10 monclient(hunting): picked mon.noname-c con 0x7f8b601cdca0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:59:03.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.909+0000 7f8b685c0640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8b60122fc0 0x7f8b60123440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:59:03.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.909+0000 7f8b685c0640 10 monclient(hunting): picked mon.noname-a con 0x7f8b60122fc0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:59:03.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.909+0000 7f8b685c0640 10 monclient(hunting): start opening mon connection 2024-09-16T03:59:03.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.909+0000 7f8b685c0640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f8b6009eef0 con 0x7f8b60122fc0 2024-09-16T03:59:03.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.909+0000 7f8b685c0640 10 monclient(hunting): start opening mon connection 2024-09-16T03:59:03.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.909+0000 7f8b685c0640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f8b6009ed70 con 0x7f8b4c004810 2024-09-16T03:59:03.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.909+0000 7f8b685c0640 10 monclient(hunting): start opening mon connection 2024-09-16T03:59:03.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.909+0000 7f8b685c0640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f8b60135910 con 0x7f8b601cdca0 2024-09-16T03:59:03.933 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.909+0000 7f8b685c0640 10 monclient(hunting): _renew_subs 2024-09-16T03:59:03.933 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.909+0000 7f8b685c0640 10 monclient(hunting): authenticate will time out at 1990.882019s 2024-09-16T03:59:03.933 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.909+0000 7f8b65b5b640 1 -- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f8b601cdca0 msgr2=0x7f8b60122a80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T03:59:03.933 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.909+0000 7f8b65b5b640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f8b601cdca0 0x7f8b60122a80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:59:03.934 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.909+0000 7f8b66b5d640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8b60122fc0 0x7f8b60123440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:59:03.934 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.909+0000 7f8b66b5d640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8b60122fc0 0x7f8b60123440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:33604/0 (socket says 172.21.15.42:33604) 2024-09-16T03:59:03.934 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.909+0000 7f8b66b5d640 1 -- 172.21.15.42:0/666452411 learned_addr learned my addr 172.21.15.42:0/666452411 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:59:03.934 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.909+0000 7f8b66b5d640 10 monclient(hunting): get_auth_request con 0x7f8b60122fc0 auth_method 0 2024-09-16T03:59:03.934 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.909+0000 7f8b66b5d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:59:03.934 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.909+0000 7f8b66b5d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:59:03.934 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.909+0000 7f8b66b5d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:59:03.934 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.909+0000 7f8b66b5d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:59:03.934 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.909+0000 7f8b66b5d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:59:03.934 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.909+0000 7f8b66b5d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:59:03.934 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.909+0000 7f8b6635c640 1 --2- 172.21.15.42:0/666452411 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8b4c004810 0x7f8b601cd760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:59:03.934 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.909+0000 7f8b6635c640 10 monclient(hunting): get_auth_request con 0x7f8b4c004810 auth_method 0 2024-09-16T03:59:03.934 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.909+0000 7f8b6635c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:59:03.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.909+0000 7f8b6635c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:59:03.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.909+0000 7f8b6635c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:59:03.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.909+0000 7f8b66b5d640 10 monclient(hunting): handle_auth_done global_id 825114 payload 824 2024-09-16T03:59:03.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.909+0000 7f8b66b5d640 1 -- 172.21.15.42:0/666452411 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f8b601cdca0 msgr2=0x7f8b60122a80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:59:03.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.909+0000 7f8b66b5d640 1 --2- 172.21.15.42:0/666452411 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f8b601cdca0 0x7f8b60122a80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:59:03.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.909+0000 7f8b66b5d640 1 -- 172.21.15.42:0/666452411 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8b4c004810 msgr2=0x7f8b601cd760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:59:03.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.909+0000 7f8b66b5d640 1 --2- 172.21.15.42:0/666452411 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8b4c004810 0x7f8b601cd760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:59:03.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.909+0000 7f8b66b5d640 10 monclient: _finish_hunting 0 2024-09-16T03:59:03.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.909+0000 7f8b66b5d640 1 monclient: found mon.noname-a 2024-09-16T03:59:03.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.909+0000 7f8b66b5d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:59:03.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.909+0000 7f8b66b5d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:59:03.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.909+0000 7f8b66b5d640 1 -- 172.21.15.42:0/666452411 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b60126a10 con 0x7f8b60122fc0 2024-09-16T03:59:03.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.909+0000 7f8b66b5d640 1 --2- 172.21.15.42:0/666452411 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8b60122fc0 0x7f8b60123440 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f8b60068f60 tx=0x7f8b5800cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:59:03.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.909+0000 7f8b56ffd640 10 client.?.objecter ms_handle_connect 0x7f8b60122fc0 2024-09-16T03:59:03.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.909+0000 7f8b56ffd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:59:03.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.909+0000 7f8b56ffd640 1 -- 172.21.15.42:0/666452411 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b58017070 con 0x7f8b60122fc0 2024-09-16T03:59:03.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.909+0000 7f8b56ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:59:03.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.909+0000 7f8b56ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:59:03.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.909+0000 7f8b56ffd640 10 monclient: dump: 2024-09-16T03:59:03.936 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:59:03.936 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:59:03.936 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:03.936 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:03.936 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:59:03.936 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:59:03.936 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:59:03.937 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:59:03.937 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:59:03.937 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:59:03.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.909+0000 7f8b56ffd640 10 monclient: _finish_auth 0 2024-09-16T03:59:03.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.909+0000 7f8b56ffd640 10 monclient: _check_auth_tickets 2024-09-16T03:59:03.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.909+0000 7f8b56ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:59:03.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.909+0000 7f8b56ffd640 1 -- 172.21.15.42:0/666452411 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8b58005d50 con 0x7f8b60122fc0 2024-09-16T03:59:03.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.909+0000 7f8b685c0640 5 monclient: authenticate success, global_id 825114 2024-09-16T03:59:03.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.909+0000 7f8b685c0640 10 monclient: _renew_subs 2024-09-16T03:59:03.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.909+0000 7f8b685c0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:59:03.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.909+0000 7f8b685c0640 1 -- 172.21.15.42:0/666452411 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b6006e620 con 0x7f8b60122fc0 2024-09-16T03:59:03.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.909+0000 7f8b56ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:59:03.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.909+0000 7f8b56ffd640 1 -- 172.21.15.42:0/666452411 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b5800a880 con 0x7f8b60122fc0 2024-09-16T03:59:03.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.909+0000 7f8b56ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:59:03.938 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.909+0000 7f8b56ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:59:03.938 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.909+0000 7f8b56ffd640 10 monclient: dump: 2024-09-16T03:59:03.938 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:59:03.938 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:59:03.938 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:03.938 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:03.938 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:59:03.938 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:59:03.938 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:59:03.938 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:59:03.938 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:59:03.938 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:59:03.938 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.909+0000 7f8b57fff640 4 set_mon_vals no callback set 2024-09-16T03:59:03.939 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.909+0000 7f8b685c0640 10 client.825114.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:59:03.939 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.909+0000 7f8b685c0640 10 monclient: _renew_subs 2024-09-16T03:59:03.939 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.909+0000 7f8b685c0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:59:03.939 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.909+0000 7f8b685c0640 1 -- 172.21.15.42:0/666452411 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b60126450 con 0x7f8b60122fc0 2024-09-16T03:59:03.939 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.909+0000 7f8b685c0640 1 librados: init done 2024-09-16T03:59:03.939 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.909+0000 7f8b685c0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:59:03.939 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.909+0000 7f8b685c0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:59:03.939 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.909+0000 7f8b685c0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:59:03.939 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.909+0000 7f8b685c0640 1 -- 172.21.15.42:0/666452411 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b60135ca0 con 0x7f8b60122fc0 2024-09-16T03:59:03.939 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.909+0000 7f8b56ffd640 1 -- 172.21.15.42:0/666452411 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f8b5801a070 con 0x7f8b60122fc0 2024-09-16T03:59:03.940 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.913+0000 7f8b56ffd640 1 --2- 172.21.15.42:0/666452411 >> v2:172.21.15.154:6812/254845342 conn(0x7f8b440377c0 0x7f8b44039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:59:03.940 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.913+0000 7f8b56ffd640 1 -- 172.21.15.42:0/666452411 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(516..516 src has 451..516) ==== 5258+0+0 (secure 0 0 0) 0x7f8b58047f30 con 0x7f8b60122fc0 2024-09-16T03:59:03.940 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.913+0000 7f8b56ffd640 10 client.825114.objecter ms_dispatch 0x7f8b60000ba0 osd_map(516..516 src has 451..516) 2024-09-16T03:59:03.940 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.913+0000 7f8b56ffd640 3 client.825114.objecter handle_osd_map got epochs [516,516] > 0 2024-09-16T03:59:03.940 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.913+0000 7f8b56ffd640 3 client.825114.objecter handle_osd_map decoding full epoch 516 2024-09-16T03:59:03.940 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.913+0000 7f8b56ffd640 20 client.825114.objecter dump_active .. 0 homeless 2024-09-16T03:59:03.940 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.913+0000 7f8b6635c640 1 --2- 172.21.15.42:0/666452411 >> v2:172.21.15.154:6812/254845342 conn(0x7f8b440377c0 0x7f8b44039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:59:03.940 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.913+0000 7f8b6635c640 10 monclient: get_auth_request con 0x7f8b440377c0 auth_method 0 2024-09-16T03:59:03.940 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.913+0000 7f8b56ffd640 10 client.825114.objecter ms_handle_connect 0x7f8b440377c0 2024-09-16T03:59:03.940 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.913+0000 7f8b6635c640 1 --2- 172.21.15.42:0/666452411 >> v2:172.21.15.154:6812/254845342 conn(0x7f8b440377c0 0x7f8b44039c80 secure :-1 s=READY pgs=3722 cs=0 l=1 rev1=1 crypto rx=0x7f8b60051960 tx=0x7f8b50008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:59:03.940 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.917+0000 7f8b56ffd640 1 -- 172.21.15.42:0/666452411 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f8b5800adf0 con 0x7f8b60122fc0 2024-09-16T03:59:03.940 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.917+0000 7f8b56ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:59:03.940 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:03.917+0000 7f8b56ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:59:04.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:04.141+0000 7f8b685c0640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:59:04.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:04.141+0000 7f8b685c0640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:59:04.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:04.141+0000 7f8b685c0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:59:04.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:04.141+0000 7f8b685c0640 1 -- 172.21.15.42:0/666452411 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f8b6011f690 con 0x7f8b60122fc0 2024-09-16T03:59:04.148 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:04.141+0000 7f8b56ffd640 1 -- 172.21.15.42:0/666452411 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7f8b580073f0 con 0x7f8b60122fc0 2024-09-16T03:59:04.148 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:04.141+0000 7f8b56ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:59:04.148 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:04.141+0000 7f8b56ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:59:04.148 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":154,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:59:04.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:04.145+0000 7f8b685c0640 10 librados: watch_flush enter 2024-09-16T03:59:04.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:04.145+0000 7f8b685c0640 10 librados: watch_flush exit 2024-09-16T03:59:04.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:04.145+0000 7f8b685c0640 20 client.825114.objecter shutdown clearing up homeless session... 2024-09-16T03:59:04.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:04.145+0000 7f8b685c0640 10 client.825114.objecter successfully canceled tick 2024-09-16T03:59:04.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:04.145+0000 7f8b685c0640 1 -- 172.21.15.42:0/666452411 >> v2:172.21.15.154:6812/254845342 conn(0x7f8b440377c0 msgr2=0x7f8b44039c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:59:04.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:04.145+0000 7f8b685c0640 1 --2- 172.21.15.42:0/666452411 >> v2:172.21.15.154:6812/254845342 conn(0x7f8b440377c0 0x7f8b44039c80 secure :-1 s=READY pgs=3722 cs=0 l=1 rev1=1 crypto rx=0x7f8b60051960 tx=0x7f8b50008040 comp rx=0 tx=0).stop 2024-09-16T03:59:04.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:04.145+0000 7f8b685c0640 10 monclient: shutdown 2024-09-16T03:59:04.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:04.145+0000 7f8b685c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:59:04.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:04.145+0000 7f8b685c0640 1 -- 172.21.15.42:0/666452411 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8b60122fc0 msgr2=0x7f8b60123440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:59:04.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:04.145+0000 7f8b685c0640 1 --2- 172.21.15.42:0/666452411 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8b60122fc0 0x7f8b60123440 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f8b60068f60 tx=0x7f8b5800cb90 comp rx=0 tx=0).stop 2024-09-16T03:59:04.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:04.145+0000 7f8b685c0640 1 -- 172.21.15.42:0/666452411 shutdown_connections 2024-09-16T03:59:04.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:04.145+0000 7f8b685c0640 1 --2- 172.21.15.42:0/666452411 >> v2:172.21.15.154:6812/254845342 conn(0x7f8b440377c0 0x7f8b44039c80 unknown :-1 s=CLOSED pgs=3722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:59:04.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:04.145+0000 7f8b685c0640 1 --2- 172.21.15.42:0/666452411 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8b60122fc0 0x7f8b60123440 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:59:04.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:04.145+0000 7f8b685c0640 1 --2- 172.21.15.42:0/666452411 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f8b601cdca0 0x7f8b60122a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:59:04.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:04.145+0000 7f8b685c0640 1 --2- 172.21.15.42:0/666452411 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8b4c004810 0x7f8b601cd760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:59:04.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:04.145+0000 7f8b685c0640 1 -- 172.21.15.42:0/666452411 >> 172.21.15.42:0/666452411 conn(0x7f8b600a2420 msgr2=0x7f8b60071690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:59:04.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:04.145+0000 7f8b685c0640 1 -- 172.21.15.42:0/666452411 shutdown_connections 2024-09-16T03:59:04.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:04.145+0000 7f8b685c0640 1 -- 172.21.15.42:0/666452411 wait complete. 2024-09-16T03:59:04.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:04.145+0000 7f8b685c0640 1 librados: shutdown 2024-09-16T03:59:07.168 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:59:07.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.293+0000 7fc3188bf640 10 monclient: get_monmap_and_config 2024-09-16T03:59:07.300 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.293+0000 7fc3188bf640 10 monclient: build_initial_monmap 2024-09-16T03:59:07.300 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.293+0000 7fc3188bf640 10 monclient: monmap: 2024-09-16T03:59:07.300 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:59:07.300 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:59:07.300 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:59:07.298255+0000 2024-09-16T03:59:07.300 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:59:07.298255+0000 2024-09-16T03:59:07.300 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:59:07.300 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:59:07.300 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:59:07.300 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:59:07.300 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:59:07.301 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:59:07.301 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:59:07.301 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:59:07.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.293+0000 7fc3188bf640 1 Processor -- start 2024-09-16T03:59:07.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.293+0000 7fc3188bf640 1 -- start start 2024-09-16T03:59:07.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.293+0000 7fc3188bf640 10 monclient: init 2024-09-16T03:59:07.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.297+0000 7fc3188bf640 10 monclient: _reopen_session rank -1 2024-09-16T03:59:07.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.297+0000 7fc3188bf640 10 monclient: _add_conns ranks=[3,4,2,1,0] 2024-09-16T03:59:07.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.297+0000 7fc3188bf640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fc3140a1330 0x7fc31409f180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:59:07.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.297+0000 7fc3188bf640 10 monclient(hunting): picked mon.noname-a con 0x7fc3140a1330 addr v2:172.21.15.42:3300/0 2024-09-16T03:59:07.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.297+0000 7fc3188bf640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fc3140a1710 0x7fc3140a1b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:59:07.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.297+0000 7fc3188bf640 10 monclient(hunting): picked mon.noname-c con 0x7fc3140a1710 addr v2:172.21.15.154:3300/0 2024-09-16T03:59:07.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.297+0000 7fc3188bf640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc3141340f0 0x7fc314134580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:59:07.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.297+0000 7fc3188bf640 10 monclient(hunting): picked mon.noname-b con 0x7fc3141340f0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:59:07.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.297+0000 7fc3188bf640 10 monclient(hunting): start opening mon connection 2024-09-16T03:59:07.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.297+0000 7fc3188bf640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7fc31409edf0 con 0x7fc3140a1330 2024-09-16T03:59:07.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.297+0000 7fc3188bf640 10 monclient(hunting): start opening mon connection 2024-09-16T03:59:07.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.297+0000 7fc3188bf640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fc31409cd60 con 0x7fc3140a1710 2024-09-16T03:59:07.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.297+0000 7fc3188bf640 10 monclient(hunting): start opening mon connection 2024-09-16T03:59:07.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.297+0000 7fc3188bf640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fc31409e4c0 con 0x7fc3141340f0 2024-09-16T03:59:07.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.297+0000 7fc3188bf640 10 monclient(hunting): _renew_subs 2024-09-16T03:59:07.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.297+0000 7fc3188bf640 10 monclient(hunting): authenticate will time out at 1994.268427s 2024-09-16T03:59:07.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.297+0000 7fc312d9d640 1 -- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc3141340f0 msgr2=0x7fc314134580 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T03:59:07.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.297+0000 7fc312d9d640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc3141340f0 0x7fc314134580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:59:07.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.297+0000 7fc31259c640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fc3140a1330 0x7fc31409f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:59:07.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.297+0000 7fc31259c640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fc3140a1330 0x7fc31409f180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:44208/0 (socket says 172.21.15.42:44208) 2024-09-16T03:59:07.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.297+0000 7fc31259c640 1 -- 172.21.15.42:0/1034080974 learned_addr learned my addr 172.21.15.42:0/1034080974 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:59:07.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.297+0000 7fc31259c640 10 monclient(hunting): get_auth_request con 0x7fc3140a1330 auth_method 0 2024-09-16T03:59:07.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.297+0000 7fc31259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:59:07.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.297+0000 7fc31259c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:59:07.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.297+0000 7fc31259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:59:07.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.297+0000 7fc31259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:59:07.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.297+0000 7fc31259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:59:07.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.297+0000 7fc31259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:59:07.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.297+0000 7fc30bfff640 1 --2- 172.21.15.42:0/1034080974 >> v2:172.21.15.154:3300/0 conn(0x7fc3140a1710 0x7fc3140a1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:59:07.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.297+0000 7fc30bfff640 10 monclient(hunting): get_auth_request con 0x7fc3140a1710 auth_method 0 2024-09-16T03:59:07.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.297+0000 7fc30bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:59:07.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.297+0000 7fc30bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:59:07.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.297+0000 7fc30bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:59:07.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.297+0000 7fc31259c640 10 monclient(hunting): handle_auth_done global_id 825117 payload 274 2024-09-16T03:59:07.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.297+0000 7fc31259c640 1 -- 172.21.15.42:0/1034080974 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc3141340f0 msgr2=0x7fc314134580 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:59:07.306 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.297+0000 7fc31259c640 1 --2- 172.21.15.42:0/1034080974 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc3141340f0 0x7fc314134580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:59:07.306 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.297+0000 7fc31259c640 1 -- 172.21.15.42:0/1034080974 >> v2:172.21.15.154:3300/0 conn(0x7fc3140a1710 msgr2=0x7fc3140a1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:59:07.306 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.297+0000 7fc31259c640 1 --2- 172.21.15.42:0/1034080974 >> v2:172.21.15.154:3300/0 conn(0x7fc3140a1710 0x7fc3140a1b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:59:07.306 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.297+0000 7fc31259c640 10 monclient: _finish_hunting 0 2024-09-16T03:59:07.306 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.297+0000 7fc31259c640 1 monclient: found mon.noname-a 2024-09-16T03:59:07.306 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.297+0000 7fc31259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:59:07.306 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.297+0000 7fc31259c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:59:07.306 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.297+0000 7fc31259c640 1 -- 172.21.15.42:0/1034080974 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3140a2080 con 0x7fc3140a1330 2024-09-16T03:59:07.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.297+0000 7fc31259c640 1 --2- 172.21.15.42:0/1034080974 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc3140a1330 0x7fc31409f180 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fc314073490 tx=0x7fc2fc01c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbcd874515a4ca7c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:59:07.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.297+0000 7fc311d9b640 1 -- 172.21.15.42:0/1034080974 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2fc0b0070 con 0x7fc3140a1330 2024-09-16T03:59:07.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.297+0000 7fc311d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:59:07.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.297+0000 7fc311d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:59:07.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.297+0000 7fc311d9b640 10 monclient: dump: 2024-09-16T03:59:07.311 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:59:07.311 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:59:07.311 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:07.312 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:07.312 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:59:07.312 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:59:07.312 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:59:07.312 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:59:07.312 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:59:07.312 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:59:07.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.297+0000 7fc311d9b640 10 monclient: _finish_auth 0 2024-09-16T03:59:07.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.297+0000 7fc311d9b640 10 monclient: _check_auth_tickets 2024-09-16T03:59:07.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.297+0000 7fc311d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:59:07.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.297+0000 7fc311d9b640 1 -- 172.21.15.42:0/1034080974 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2fc0ab600 con 0x7fc3140a1330 2024-09-16T03:59:07.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.297+0000 7fc311d9b640 10 monclient: handle_config config(0 keys) 2024-09-16T03:59:07.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.297+0000 7fc311d9b640 1 -- 172.21.15.42:0/1034080974 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2fc0ab9d0 con 0x7fc3140a1330 2024-09-16T03:59:07.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.297+0000 7fc3188bf640 5 monclient: authenticate success, global_id 825117 2024-09-16T03:59:07.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.297+0000 7fc3188bf640 10 monclient: get_monmap_and_config success 2024-09-16T03:59:07.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.297+0000 7fc3188bf640 4 set_mon_vals no callback set 2024-09-16T03:59:07.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.297+0000 7fc311d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:59:07.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.297+0000 7fc311d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:59:07.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.297+0000 7fc311d9b640 10 monclient: dump: 2024-09-16T03:59:07.313 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:59:07.313 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:59:07.313 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:07.313 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:07.313 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:59:07.313 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:59:07.313 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:59:07.314 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:59:07.314 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:59:07.314 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:59:07.314 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.301+0000 7fc3188bf640 10 monclient: shutdown 2024-09-16T03:59:07.314 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.301+0000 7fc3188bf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:59:07.314 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.301+0000 7fc3188bf640 1 -- 172.21.15.42:0/1034080974 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc3140a1330 msgr2=0x7fc31409f180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:59:07.314 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.301+0000 7fc3188bf640 1 --2- 172.21.15.42:0/1034080974 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc3140a1330 0x7fc31409f180 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fc314073490 tx=0x7fc2fc01c400 comp rx=0 tx=0).stop 2024-09-16T03:59:07.314 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.301+0000 7fc3188bf640 1 -- 172.21.15.42:0/1034080974 shutdown_connections 2024-09-16T03:59:07.314 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.301+0000 7fc3188bf640 1 --2- 172.21.15.42:0/1034080974 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc3141340f0 0x7fc314134580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:59:07.314 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.301+0000 7fc3188bf640 1 --2- 172.21.15.42:0/1034080974 >> v2:172.21.15.154:3300/0 conn(0x7fc3140a1710 0x7fc3140a1b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:59:07.314 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.301+0000 7fc3188bf640 1 --2- 172.21.15.42:0/1034080974 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc3140a1330 0x7fc31409f180 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:59:07.314 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.301+0000 7fc3188bf640 1 -- 172.21.15.42:0/1034080974 >> 172.21.15.42:0/1034080974 conn(0x7fc3140a2420 msgr2=0x7fc3140a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:59:07.314 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.301+0000 7fc3188bf640 1 -- 172.21.15.42:0/1034080974 shutdown_connections 2024-09-16T03:59:07.314 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.301+0000 7fc3188bf640 1 -- 172.21.15.42:0/1034080974 wait complete. 2024-09-16T03:59:07.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.301+0000 7fc3188bf640 10 monclient: build_initial_monmap 2024-09-16T03:59:07.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.301+0000 7fc3188bf640 10 monclient: monmap: 2024-09-16T03:59:07.315 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:59:07.315 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:59:07.315 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:59:07.315 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:59:07.315 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:59:07.315 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:59:07.315 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:59:07.315 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:59:07.315 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:59:07.316 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:59:07.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.301+0000 7fc3188bf640 1 librados: starting msgr at 2024-09-16T03:59:07.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.301+0000 7fc3188bf640 1 librados: starting objecter 2024-09-16T03:59:07.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.301+0000 7fc3188bf640 1 Processor -- start 2024-09-16T03:59:07.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.301+0000 7fc3188bf640 1 -- start start 2024-09-16T03:59:07.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.301+0000 7fc3188bf640 1 librados: setting wanted keys 2024-09-16T03:59:07.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.301+0000 7fc3188bf640 1 librados: calling monclient init 2024-09-16T03:59:07.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.301+0000 7fc3188bf640 10 monclient: init 2024-09-16T03:59:07.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.301+0000 7fc3188bf640 10 monclient: _reopen_session rank -1 2024-09-16T03:59:07.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.301+0000 7fc3188bf640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T03:59:07.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.301+0000 7fc3188bf640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc3140a1330 0x7fc3141cd040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:59:07.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.301+0000 7fc3188bf640 10 monclient(hunting): picked mon.noname-a con 0x7fc3140a1330 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:59:07.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.301+0000 7fc3188bf640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc3140a1710 0x7fc314133440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:59:07.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.301+0000 7fc3188bf640 10 monclient(hunting): picked mon.noname-b con 0x7fc3140a1710 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:59:07.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.301+0000 7fc3188bf640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc3141340f0 0x7fc314133980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:59:07.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.301+0000 7fc3188bf640 10 monclient(hunting): picked mon.noname-c con 0x7fc3141340f0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:59:07.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.301+0000 7fc3188bf640 10 monclient(hunting): start opening mon connection 2024-09-16T03:59:07.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.301+0000 7fc3188bf640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fc31409d1b0 con 0x7fc3140a1330 2024-09-16T03:59:07.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.301+0000 7fc3188bf640 10 monclient(hunting): start opening mon connection 2024-09-16T03:59:07.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.301+0000 7fc3188bf640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc31409d030 con 0x7fc3140a1710 2024-09-16T03:59:07.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.301+0000 7fc3188bf640 10 monclient(hunting): start opening mon connection 2024-09-16T03:59:07.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.301+0000 7fc3188bf640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fc31413dbd0 con 0x7fc3141340f0 2024-09-16T03:59:07.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.301+0000 7fc3188bf640 10 monclient(hunting): _renew_subs 2024-09-16T03:59:07.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.301+0000 7fc3188bf640 10 monclient(hunting): authenticate will time out at 1994.273844s 2024-09-16T03:59:07.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.301+0000 7fc312d9d640 1 -- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc3141340f0 msgr2=0x7fc314133980 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T03:59:07.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.301+0000 7fc312d9d640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc3141340f0 0x7fc314133980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:59:07.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.301+0000 7fc31259c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc3140a1330 0x7fc3141cd040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:59:07.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.301+0000 7fc31259c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc3140a1330 0x7fc3141cd040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:44220/0 (socket says 172.21.15.42:44220) 2024-09-16T03:59:07.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.301+0000 7fc31259c640 1 -- 172.21.15.42:0/2102323273 learned_addr learned my addr 172.21.15.42:0/2102323273 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:59:07.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.301+0000 7fc31259c640 10 monclient(hunting): get_auth_request con 0x7fc3140a1330 auth_method 0 2024-09-16T03:59:07.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.301+0000 7fc31259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:59:07.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.301+0000 7fc31259c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:59:07.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.301+0000 7fc31259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:59:07.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.301+0000 7fc31259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:59:07.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.301+0000 7fc31259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:59:07.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.301+0000 7fc31259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:59:07.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.301+0000 7fc30bfff640 1 --2- 172.21.15.42:0/2102323273 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc3140a1710 0x7fc314133440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:59:07.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.301+0000 7fc30bfff640 10 monclient(hunting): get_auth_request con 0x7fc3140a1710 auth_method 0 2024-09-16T03:59:07.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.301+0000 7fc30bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:59:07.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.301+0000 7fc30bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:59:07.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.301+0000 7fc30bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:59:07.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.301+0000 7fc31259c640 10 monclient(hunting): handle_auth_done global_id 825120 payload 824 2024-09-16T03:59:07.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.301+0000 7fc31259c640 1 -- 172.21.15.42:0/2102323273 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc3141340f0 msgr2=0x7fc314133980 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:59:07.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.301+0000 7fc31259c640 1 --2- 172.21.15.42:0/2102323273 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc3141340f0 0x7fc314133980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:59:07.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.301+0000 7fc31259c640 1 -- 172.21.15.42:0/2102323273 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc3140a1710 msgr2=0x7fc314133440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:59:07.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.301+0000 7fc31259c640 1 --2- 172.21.15.42:0/2102323273 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc3140a1710 0x7fc314133440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:59:07.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.301+0000 7fc31259c640 10 monclient: _finish_hunting 0 2024-09-16T03:59:07.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.301+0000 7fc31259c640 1 monclient: found mon.noname-a 2024-09-16T03:59:07.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.301+0000 7fc31259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:59:07.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.301+0000 7fc31259c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:59:07.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.301+0000 7fc31259c640 1 -- 172.21.15.42:0/2102323273 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc31411f830 con 0x7fc3140a1330 2024-09-16T03:59:07.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.301+0000 7fc31259c640 1 --2- 172.21.15.42:0/2102323273 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc3140a1330 0x7fc3141cd040 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fc2fc0ae040 tx=0x7fc2fc0191a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:59:07.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.301+0000 7fc30affd640 10 client.?.objecter ms_handle_connect 0x7fc3140a1330 2024-09-16T03:59:07.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.301+0000 7fc30affd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:59:07.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.301+0000 7fc30affd640 1 -- 172.21.15.42:0/2102323273 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2fc0b0040 con 0x7fc3140a1330 2024-09-16T03:59:07.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.301+0000 7fc30affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:59:07.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.301+0000 7fc30affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:59:07.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.301+0000 7fc30affd640 10 monclient: dump: 2024-09-16T03:59:07.320 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:59:07.320 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:59:07.320 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:07.320 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:07.320 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:59:07.320 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:59:07.320 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:59:07.320 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:59:07.321 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:59:07.321 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:59:07.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.301+0000 7fc30affd640 10 monclient: _finish_auth 0 2024-09-16T03:59:07.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.301+0000 7fc30affd640 10 monclient: _check_auth_tickets 2024-09-16T03:59:07.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.301+0000 7fc30affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:59:07.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.301+0000 7fc30affd640 1 -- 172.21.15.42:0/2102323273 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2fc0b9070 con 0x7fc3140a1330 2024-09-16T03:59:07.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.301+0000 7fc30affd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:59:07.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.301+0000 7fc30affd640 1 -- 172.21.15.42:0/2102323273 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2fc0b4640 con 0x7fc3140a1330 2024-09-16T03:59:07.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.301+0000 7fc30affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:59:07.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.301+0000 7fc30affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:59:07.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.301+0000 7fc30affd640 10 monclient: dump: 2024-09-16T03:59:07.321 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:59:07.321 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:59:07.321 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:07.322 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:07.322 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:59:07.322 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:59:07.322 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:59:07.322 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:59:07.322 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:59:07.322 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:59:07.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.301+0000 7fc3188bf640 5 monclient: authenticate success, global_id 825120 2024-09-16T03:59:07.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.301+0000 7fc3188bf640 10 monclient: _renew_subs 2024-09-16T03:59:07.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.301+0000 7fc31159a640 4 set_mon_vals no callback set 2024-09-16T03:59:07.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.301+0000 7fc3188bf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:59:07.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.301+0000 7fc3188bf640 1 -- 172.21.15.42:0/2102323273 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc31411fdf0 con 0x7fc3140a1330 2024-09-16T03:59:07.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.301+0000 7fc3188bf640 10 client.825120.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:59:07.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.301+0000 7fc3188bf640 10 monclient: _renew_subs 2024-09-16T03:59:07.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.301+0000 7fc3188bf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:59:07.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.301+0000 7fc3188bf640 1 -- 172.21.15.42:0/2102323273 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc31411f550 con 0x7fc3140a1330 2024-09-16T03:59:07.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.301+0000 7fc3188bf640 1 librados: init done 2024-09-16T03:59:07.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.301+0000 7fc3188bf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:59:07.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.301+0000 7fc30affd640 1 -- 172.21.15.42:0/2102323273 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fc2fc0b47e0 con 0x7fc3140a1330 2024-09-16T03:59:07.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.301+0000 7fc3188bf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:59:07.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.301+0000 7fc3188bf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:59:07.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.301+0000 7fc3188bf640 1 -- 172.21.15.42:0/2102323273 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc3140a1b40 con 0x7fc3140a1330 2024-09-16T03:59:07.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.301+0000 7fc30affd640 1 --2- 172.21.15.42:0/2102323273 >> v2:172.21.15.154:6812/254845342 conn(0x7fc2f80377c0 0x7fc2f8039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:59:07.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.301+0000 7fc30affd640 1 -- 172.21.15.42:0/2102323273 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(516..516 src has 451..516) ==== 5258+0+0 (secure 0 0 0) 0x7fc2fc0e2b00 con 0x7fc3140a1330 2024-09-16T03:59:07.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.301+0000 7fc30affd640 10 client.825120.objecter ms_dispatch 0x7fc314000ba0 osd_map(516..516 src has 451..516) 2024-09-16T03:59:07.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.301+0000 7fc30affd640 3 client.825120.objecter handle_osd_map got epochs [516,516] > 0 2024-09-16T03:59:07.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.301+0000 7fc30affd640 3 client.825120.objecter handle_osd_map decoding full epoch 516 2024-09-16T03:59:07.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.305+0000 7fc30affd640 20 client.825120.objecter dump_active .. 0 homeless 2024-09-16T03:59:07.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.305+0000 7fc30bfff640 1 --2- 172.21.15.42:0/2102323273 >> v2:172.21.15.154:6812/254845342 conn(0x7fc2f80377c0 0x7fc2f8039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:59:07.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.305+0000 7fc30bfff640 10 monclient: get_auth_request con 0x7fc2f80377c0 auth_method 0 2024-09-16T03:59:07.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.305+0000 7fc30affd640 10 client.825120.objecter ms_handle_connect 0x7fc2f80377c0 2024-09-16T03:59:07.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.305+0000 7fc30bfff640 1 --2- 172.21.15.42:0/2102323273 >> v2:172.21.15.154:6812/254845342 conn(0x7fc2f80377c0 0x7fc2f8039c80 secure :-1 s=READY pgs=3723 cs=0 l=1 rev1=1 crypto rx=0x7fc31411d6e0 tx=0x7fc304008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:59:07.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.309+0000 7fc30affd640 1 -- 172.21.15.42:0/2102323273 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7fc2fc016110 con 0x7fc3140a1330 2024-09-16T03:59:07.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.309+0000 7fc30affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:59:07.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.309+0000 7fc30affd640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:59:07.537 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.529+0000 7fc3188bf640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:59:07.537 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.529+0000 7fc3188bf640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:59:07.537 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.529+0000 7fc3188bf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:59:07.537 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.529+0000 7fc3188bf640 1 -- 172.21.15.42:0/2102323273 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fc31411e3e0 con 0x7fc3140a1330 2024-09-16T03:59:07.538 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.533+0000 7fc30affd640 1 -- 172.21.15.42:0/2102323273 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1562 (secure 0 0 0) 0x7fc2fc0b3560 con 0x7fc3140a1330 2024-09-16T03:59:07.538 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.533+0000 7fc30affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:59:07.538 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.533+0000 7fc30affd640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:59:07.538 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":154,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:59:07.541 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.533+0000 7fc3188bf640 10 librados: watch_flush enter 2024-09-16T03:59:07.541 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.533+0000 7fc3188bf640 10 librados: watch_flush exit 2024-09-16T03:59:07.541 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.533+0000 7fc3188bf640 20 client.825120.objecter shutdown clearing up homeless session... 2024-09-16T03:59:07.541 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.533+0000 7fc3188bf640 10 client.825120.objecter successfully canceled tick 2024-09-16T03:59:07.541 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.533+0000 7fc3188bf640 1 -- 172.21.15.42:0/2102323273 >> v2:172.21.15.154:6812/254845342 conn(0x7fc2f80377c0 msgr2=0x7fc2f8039c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:59:07.541 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.533+0000 7fc3188bf640 1 --2- 172.21.15.42:0/2102323273 >> v2:172.21.15.154:6812/254845342 conn(0x7fc2f80377c0 0x7fc2f8039c80 secure :-1 s=READY pgs=3723 cs=0 l=1 rev1=1 crypto rx=0x7fc31411d6e0 tx=0x7fc304008040 comp rx=0 tx=0).stop 2024-09-16T03:59:07.541 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.533+0000 7fc3188bf640 10 monclient: shutdown 2024-09-16T03:59:07.541 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.533+0000 7fc3188bf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:59:07.541 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.533+0000 7fc3188bf640 1 -- 172.21.15.42:0/2102323273 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc3140a1330 msgr2=0x7fc3141cd040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:59:07.541 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.533+0000 7fc3188bf640 1 --2- 172.21.15.42:0/2102323273 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc3140a1330 0x7fc3141cd040 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fc2fc0ae040 tx=0x7fc2fc0191a0 comp rx=0 tx=0).stop 2024-09-16T03:59:07.541 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.533+0000 7fc3188bf640 1 -- 172.21.15.42:0/2102323273 shutdown_connections 2024-09-16T03:59:07.544 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.533+0000 7fc3188bf640 1 --2- 172.21.15.42:0/2102323273 >> v2:172.21.15.154:6812/254845342 conn(0x7fc2f80377c0 0x7fc2f8039c80 unknown :-1 s=CLOSED pgs=3723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:59:07.544 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.533+0000 7fc3188bf640 1 --2- 172.21.15.42:0/2102323273 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc3141340f0 0x7fc314133980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:59:07.545 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.533+0000 7fc3188bf640 1 --2- 172.21.15.42:0/2102323273 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc3140a1710 0x7fc314133440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:59:07.545 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.533+0000 7fc3188bf640 1 --2- 172.21.15.42:0/2102323273 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc3140a1330 0x7fc3141cd040 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:59:07.545 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.533+0000 7fc3188bf640 1 -- 172.21.15.42:0/2102323273 >> 172.21.15.42:0/2102323273 conn(0x7fc3140a2420 msgr2=0x7fc31413ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:59:07.545 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.533+0000 7fc3188bf640 1 -- 172.21.15.42:0/2102323273 shutdown_connections 2024-09-16T03:59:07.545 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.533+0000 7fc3188bf640 1 -- 172.21.15.42:0/2102323273 wait complete. 2024-09-16T03:59:07.545 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:07.533+0000 7fc3188bf640 1 librados: shutdown 2024-09-16T03:59:10.556 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:59:10.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:10.681+0000 7f87eabbb640 10 monclient: get_monmap_and_config 2024-09-16T03:59:10.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:10.681+0000 7f87eabbb640 10 monclient: build_initial_monmap 2024-09-16T03:59:10.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:10.681+0000 7f87eabbb640 10 monclient: monmap: 2024-09-16T03:59:10.688 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:59:10.688 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:59:10.688 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:59:10.686145+0000 2024-09-16T03:59:10.688 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:59:10.686145+0000 2024-09-16T03:59:10.688 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:59:10.688 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:59:10.688 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:59:10.688 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:59:10.688 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:59:10.688 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:59:10.688 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:59:10.689 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:59:10.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:10.681+0000 7f87eabbb640 1 Processor -- start 2024-09-16T03:59:10.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:10.681+0000 7f87eabbb640 1 -- start start 2024-09-16T03:59:10.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:10.681+0000 7f87eabbb640 10 monclient: init 2024-09-16T03:59:10.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:10.681+0000 7f87eabbb640 10 monclient: _reopen_session rank -1 2024-09-16T03:59:10.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:10.681+0000 7f87eabbb640 10 monclient: _add_conns ranks=[3,0,4,1,2] 2024-09-16T03:59:10.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:10.681+0000 7f87eabbb640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f87e40a1330 0x7f87e40a1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:59:10.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:10.681+0000 7f87eabbb640 10 monclient(hunting): picked mon.noname-a con 0x7f87e40a1330 addr v2:172.21.15.42:3300/0 2024-09-16T03:59:10.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:10.681+0000 7f87eabbb640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f87e406dac0 addr v1:172.21.15.42:6789/0 2024-09-16T03:59:10.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:10.681+0000 7f87eabbb640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f87e40a1c50 0x7f87e4135810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:59:10.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:10.681+0000 7f87eabbb640 10 monclient(hunting): picked mon.noname-c con 0x7f87e40a1c50 addr v2:172.21.15.154:3300/0 2024-09-16T03:59:10.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:10.681+0000 7f87eabbb640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f87e409c090 con 0x7f87e406dac0 2024-09-16T03:59:10.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:10.681+0000 7f87eabbb640 10 monclient(hunting): start opening mon connection 2024-09-16T03:59:10.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:10.681+0000 7f87eabbb640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f87e409f610 con 0x7f87e40a1330 2024-09-16T03:59:10.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:10.681+0000 7f87eabbb640 10 monclient(hunting): start opening mon connection 2024-09-16T03:59:10.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:10.681+0000 7f87eabbb640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f87e409f8f0 con 0x7f87e40a1c50 2024-09-16T03:59:10.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:10.681+0000 7f87eabbb640 10 monclient(hunting): _renew_subs 2024-09-16T03:59:10.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:10.681+0000 7f87eabbb640 10 monclient(hunting): authenticate will time out at 1997.655848s 2024-09-16T03:59:10.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:10.685+0000 7f87e8957640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f87e40a1330 0x7f87e40a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:59:10.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:10.685+0000 7f87e3fff640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f87e406dac0 0x7f87e4135100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:48804/0 (socket says 172.21.15.42:48804) 2024-09-16T03:59:10.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:10.685+0000 7f87e3fff640 1 -- 172.21.15.42:0/3475885585 learned_addr learned my addr 172.21.15.42:0/3475885585 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:59:10.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:10.685+0000 7f87e8957640 10 monclient(hunting): get_auth_request con 0x7f87e40a1330 auth_method 0 2024-09-16T03:59:10.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:10.685+0000 7f87e8957640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:59:10.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:10.685+0000 7f87e8957640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:59:10.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:10.685+0000 7f87e8957640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:59:10.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:10.685+0000 7f87e8957640 1 -- 172.21.15.42:0/3475885585 >> v2:172.21.15.42:3300/0 conn(0x7f87e40a1330 msgr2=0x7f87e40a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T03:59:10.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:10.685+0000 7f87e8957640 1 -- 172.21.15.42:0/3475885585 >> v2:172.21.15.42:3300/0 conn(0x7f87e40a1330 msgr2=0x7f87e40a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:59:10.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:10.685+0000 7f87e8957640 1 --2- 172.21.15.42:0/3475885585 >> v2:172.21.15.42:3300/0 conn(0x7f87e40a1330 0x7f87e40a1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:59:10.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:10.685+0000 7f87e8957640 1 --2- 172.21.15.42:0/3475885585 >> v2:172.21.15.42:3300/0 conn(0x7f87e40a1330 0x7f87e40a1710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:59:10.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:10.685+0000 7f87e9158640 1 --2- 172.21.15.42:0/3475885585 >> v2:172.21.15.154:3300/0 conn(0x7f87e40a1c50 0x7f87e4135810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:59:10.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:10.685+0000 7f87e9158640 10 monclient(hunting): get_auth_request con 0x7f87e40a1c50 auth_method 0 2024-09-16T03:59:10.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:10.685+0000 7f87e9158640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:59:10.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:10.685+0000 7f87e9158640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:59:10.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:10.685+0000 7f87e9158640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:59:10.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:10.685+0000 7f87e9158640 1 -- 172.21.15.42:0/3475885585 >> v2:172.21.15.154:3300/0 conn(0x7f87e40a1c50 msgr2=0x7f87e4135810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T03:59:10.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:10.685+0000 7f87e9158640 1 -- 172.21.15.42:0/3475885585 >> v2:172.21.15.154:3300/0 conn(0x7f87e40a1c50 msgr2=0x7f87e4135810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:59:10.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:10.685+0000 7f87e9158640 1 --2- 172.21.15.42:0/3475885585 >> v2:172.21.15.154:3300/0 conn(0x7f87e40a1c50 0x7f87e4135810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:59:10.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:10.685+0000 7f87e9158640 1 --2- 172.21.15.42:0/3475885585 >> v2:172.21.15.154:3300/0 conn(0x7f87e40a1c50 0x7f87e4135810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:59:10.894 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:10.889+0000 7f87e8957640 1 --2- 172.21.15.42:0/3475885585 >> v2:172.21.15.42:3300/0 conn(0x7f87e40a1330 0x7f87e40a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:59:10.895 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:10.889+0000 7f87e8957640 10 monclient(hunting): get_auth_request con 0x7f87e40a1330 auth_method 0 2024-09-16T03:59:10.895 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:10.889+0000 7f87e8957640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:59:10.895 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:10.889+0000 7f87e8957640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:59:10.895 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:10.889+0000 7f87e8957640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:59:10.895 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:10.889+0000 7f87e9158640 1 --2- 172.21.15.42:0/3475885585 >> v2:172.21.15.154:3300/0 conn(0x7f87e40a1c50 0x7f87e4135810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:59:10.895 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:10.889+0000 7f87e9158640 10 monclient(hunting): get_auth_request con 0x7f87e40a1c50 auth_method 0 2024-09-16T03:59:10.895 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:10.889+0000 7f87e9158640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:59:10.895 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:10.889+0000 7f87e9158640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:59:10.895 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:10.889+0000 7f87e9158640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:59:10.895 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:10.889+0000 7f87e8957640 1 -- 172.21.15.42:0/3475885585 >> v2:172.21.15.42:3300/0 conn(0x7f87e40a1330 msgr2=0x7f87e40a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T03:59:10.895 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:10.889+0000 7f87e8957640 1 -- 172.21.15.42:0/3475885585 >> v2:172.21.15.42:3300/0 conn(0x7f87e40a1330 msgr2=0x7f87e40a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:59:10.895 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:10.889+0000 7f87e8957640 1 --2- 172.21.15.42:0/3475885585 >> v2:172.21.15.42:3300/0 conn(0x7f87e40a1330 0x7f87e40a1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:59:10.895 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:10.889+0000 7f87e8957640 1 --2- 172.21.15.42:0/3475885585 >> v2:172.21.15.42:3300/0 conn(0x7f87e40a1330 0x7f87e40a1710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T03:59:10.896 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:10.889+0000 7f87e9158640 1 -- 172.21.15.42:0/3475885585 >> v2:172.21.15.154:3300/0 conn(0x7f87e40a1c50 msgr2=0x7f87e4135810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T03:59:10.896 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:10.889+0000 7f87e9158640 1 -- 172.21.15.42:0/3475885585 >> v2:172.21.15.154:3300/0 conn(0x7f87e40a1c50 msgr2=0x7f87e4135810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:59:10.896 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:10.889+0000 7f87e9158640 1 --2- 172.21.15.42:0/3475885585 >> v2:172.21.15.154:3300/0 conn(0x7f87e40a1c50 0x7f87e4135810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:59:10.896 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:10.889+0000 7f87e9158640 1 --2- 172.21.15.42:0/3475885585 >> v2:172.21.15.154:3300/0 conn(0x7f87e40a1c50 0x7f87e4135810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T03:59:11.298 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:11.293+0000 7f87e8957640 1 --2- 172.21.15.42:0/3475885585 >> v2:172.21.15.42:3300/0 conn(0x7f87e40a1330 0x7f87e40a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:59:11.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:11.293+0000 7f87e8957640 10 monclient(hunting): get_auth_request con 0x7f87e40a1330 auth_method 0 2024-09-16T03:59:11.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:11.293+0000 7f87e8957640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:59:11.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:11.293+0000 7f87e8957640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:59:11.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:11.293+0000 7f87e8957640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:59:11.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:11.293+0000 7f87e9158640 1 --2- 172.21.15.42:0/3475885585 >> v2:172.21.15.154:3300/0 conn(0x7f87e40a1c50 0x7f87e4135810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:59:11.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:11.293+0000 7f87e9158640 10 monclient(hunting): get_auth_request con 0x7f87e40a1c50 auth_method 0 2024-09-16T03:59:11.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:11.293+0000 7f87e9158640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:59:11.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:11.293+0000 7f87e9158640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:59:11.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:11.293+0000 7f87e9158640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:59:11.300 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:11.293+0000 7f87e8957640 1 -- 172.21.15.42:0/3475885585 >> v2:172.21.15.42:3300/0 conn(0x7f87e40a1330 msgr2=0x7f87e40a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T03:59:11.300 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:11.293+0000 7f87e8957640 1 -- 172.21.15.42:0/3475885585 >> v2:172.21.15.42:3300/0 conn(0x7f87e40a1330 msgr2=0x7f87e40a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:59:11.300 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:11.293+0000 7f87e8957640 1 --2- 172.21.15.42:0/3475885585 >> v2:172.21.15.42:3300/0 conn(0x7f87e40a1330 0x7f87e40a1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:59:11.300 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:11.293+0000 7f87e9158640 1 -- 172.21.15.42:0/3475885585 >> v2:172.21.15.154:3300/0 conn(0x7f87e40a1c50 msgr2=0x7f87e4135810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T03:59:11.300 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:11.293+0000 7f87e9158640 1 -- 172.21.15.42:0/3475885585 >> v2:172.21.15.154:3300/0 conn(0x7f87e40a1c50 msgr2=0x7f87e4135810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:59:11.300 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:11.293+0000 7f87e8957640 1 --2- 172.21.15.42:0/3475885585 >> v2:172.21.15.42:3300/0 conn(0x7f87e40a1330 0x7f87e40a1710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T03:59:11.301 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:11.293+0000 7f87e9158640 1 --2- 172.21.15.42:0/3475885585 >> v2:172.21.15.154:3300/0 conn(0x7f87e40a1c50 0x7f87e4135810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:59:11.301 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:11.293+0000 7f87e9158640 1 --2- 172.21.15.42:0/3475885585 >> v2:172.21.15.154:3300/0 conn(0x7f87e40a1c50 0x7f87e4135810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T03:59:12.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:12.097+0000 7f87e8957640 1 --2- 172.21.15.42:0/3475885585 >> v2:172.21.15.42:3300/0 conn(0x7f87e40a1330 0x7f87e40a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:59:12.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:12.097+0000 7f87e8957640 10 monclient(hunting): get_auth_request con 0x7f87e40a1330 auth_method 0 2024-09-16T03:59:12.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:12.097+0000 7f87e8957640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:59:12.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:12.097+0000 7f87e8957640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:59:12.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:12.097+0000 7f87e8957640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:59:12.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:12.097+0000 7f87e9158640 1 --2- 172.21.15.42:0/3475885585 >> v2:172.21.15.154:3300/0 conn(0x7f87e40a1c50 0x7f87e4135810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:59:12.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:12.097+0000 7f87e9158640 10 monclient(hunting): get_auth_request con 0x7f87e40a1c50 auth_method 0 2024-09-16T03:59:12.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:12.097+0000 7f87e9158640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:59:12.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:12.097+0000 7f87e9158640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:59:12.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:12.097+0000 7f87e9158640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:59:12.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:12.097+0000 7f87e8957640 1 -- 172.21.15.42:0/3475885585 >> v2:172.21.15.42:3300/0 conn(0x7f87e40a1330 msgr2=0x7f87e40a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T03:59:12.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:12.097+0000 7f87e8957640 1 -- 172.21.15.42:0/3475885585 >> v2:172.21.15.42:3300/0 conn(0x7f87e40a1330 msgr2=0x7f87e40a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:59:12.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:12.097+0000 7f87e8957640 1 --2- 172.21.15.42:0/3475885585 >> v2:172.21.15.42:3300/0 conn(0x7f87e40a1330 0x7f87e40a1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:59:12.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:12.097+0000 7f87e8957640 1 --2- 172.21.15.42:0/3475885585 >> v2:172.21.15.42:3300/0 conn(0x7f87e40a1330 0x7f87e40a1710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T03:59:12.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:12.097+0000 7f87e9158640 1 -- 172.21.15.42:0/3475885585 >> v2:172.21.15.154:3300/0 conn(0x7f87e40a1c50 msgr2=0x7f87e4135810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T03:59:12.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:12.097+0000 7f87e9158640 1 -- 172.21.15.42:0/3475885585 >> v2:172.21.15.154:3300/0 conn(0x7f87e40a1c50 msgr2=0x7f87e4135810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:59:12.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:12.097+0000 7f87e9158640 1 --2- 172.21.15.42:0/3475885585 >> v2:172.21.15.154:3300/0 conn(0x7f87e40a1c50 0x7f87e4135810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:59:12.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:12.097+0000 7f87e9158640 1 --2- 172.21.15.42:0/3475885585 >> v2:172.21.15.154:3300/0 conn(0x7f87e40a1c50 0x7f87e4135810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T03:59:13.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:13.685+0000 7f87e27fc640 10 monclient(hunting): tick 2024-09-16T03:59:13.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:13.685+0000 7f87e27fc640 10 monclient(hunting): _check_auth_tickets 2024-09-16T03:59:13.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:13.685+0000 7f87e27fc640 1 monclient(hunting): continuing hunt 2024-09-16T03:59:13.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:13.685+0000 7f87e27fc640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T03:59:13.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:13.685+0000 7f87e27fc640 1 -- 172.21.15.42:0/3475885585 >> v2:172.21.15.154:3300/0 conn(0x7f87e40a1c50 msgr2=0x7f87e4135810 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:59:13.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:13.685+0000 7f87e27fc640 1 --2- 172.21.15.42:0/3475885585 >> v2:172.21.15.154:3300/0 conn(0x7f87e40a1c50 0x7f87e4135810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:59:13.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:13.685+0000 7f87e27fc640 1 -- 172.21.15.42:0/3475885585 >> v2:172.21.15.42:3300/0 conn(0x7f87e40a1330 msgr2=0x7f87e40a1710 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:59:13.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:13.685+0000 7f87e27fc640 1 --2- 172.21.15.42:0/3475885585 >> v2:172.21.15.42:3300/0 conn(0x7f87e40a1330 0x7f87e40a1710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:59:13.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:13.685+0000 7f87e27fc640 1 -- 172.21.15.42:0/3475885585 >> v1:172.21.15.42:6789/0 conn(0x7f87e406dac0 legacy=0x7f87e4135100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:59:13.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:13.685+0000 7f87e27fc640 10 monclient: _add_conns ranks=[2,1] 2024-09-16T03:59:13.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:13.685+0000 7f87e27fc640 1 --2- 172.21.15.42:0/3475885585 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f87d00036a0 0x7f87d0005ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:59:13.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:13.685+0000 7f87e27fc640 10 monclient(hunting): picked mon.noname-b con 0x7f87d00036a0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:59:13.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:13.685+0000 7f87e27fc640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f87d0006160 addr v1:172.21.15.154:6789/0 2024-09-16T03:59:13.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:13.685+0000 7f87e27fc640 1 -- 172.21.15.42:0/3475885585 --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f87e409fbd0 con 0x7f87d0006160 2024-09-16T03:59:13.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:13.685+0000 7f87e27fc640 10 monclient(hunting): start opening mon connection 2024-09-16T03:59:13.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:13.685+0000 7f87e27fc640 1 -- 172.21.15.42:0/3475885585 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f87e409f610 con 0x7f87d00036a0 2024-09-16T03:59:13.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:13.685+0000 7f87e27fc640 10 monclient(hunting): _renew_subs 2024-09-16T03:59:13.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:13.685+0000 7f87e8957640 1 -- 172.21.15.42:0/3475885585 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f87d00036a0 msgr2=0x7f87d0005ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T03:59:13.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:13.685+0000 7f87e8957640 1 --2- 172.21.15.42:0/3475885585 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f87d00036a0 0x7f87d0005ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:59:13.893 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:13.889+0000 7f87e8957640 1 -- 172.21.15.42:0/3475885585 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f87d00036a0 msgr2=0x7f87d0005ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T03:59:13.893 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:13.889+0000 7f87e8957640 1 --2- 172.21.15.42:0/3475885585 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f87d00036a0 0x7f87d0005ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T03:59:14.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.293+0000 7f87e8957640 1 -- 172.21.15.42:0/3475885585 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f87d00036a0 msgr2=0x7f87d0005ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T03:59:14.298 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.293+0000 7f87e8957640 1 --2- 172.21.15.42:0/3475885585 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f87d00036a0 0x7f87d0005ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T03:59:14.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.581+0000 7f87e37fe640 1 -- 172.21.15.42:0/3475885585 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 865716245 0 0) 0x7f87e409fbd0 con 0x7f87d0006160 2024-09-16T03:59:14.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.581+0000 7f87e37fe640 10 monclient(hunting): _negotiate 2024-09-16T03:59:14.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.581+0000 7f87e37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:59:14.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.581+0000 7f87e37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:59:14.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.581+0000 7f87e37fe640 10 monclient(hunting): my global_id is 904300 2024-09-16T03:59:14.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.581+0000 7f87e37fe640 1 -- 172.21.15.42:0/3475885585 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f87cc003530 con 0x7f87d0006160 2024-09-16T03:59:14.589 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.581+0000 7f87e37fe640 1 -- 172.21.15.42:0/3475885585 <== mon.1 v1:172.21.15.154:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4083352278 0 0) 0x7f87cc003530 con 0x7f87d0006160 2024-09-16T03:59:14.589 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.581+0000 7f87e37fe640 1 -- 172.21.15.42:0/3475885585 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f87d00036a0 msgr2=0x7f87d0005ad0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:59:14.589 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.581+0000 7f87e37fe640 1 --2- 172.21.15.42:0/3475885585 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f87d00036a0 0x7f87d0005ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:59:14.589 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.581+0000 7f87e37fe640 10 monclient: _finish_hunting 0 2024-09-16T03:59:14.589 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.581+0000 7f87e37fe640 1 monclient: found mon.noname-c-legacy 2024-09-16T03:59:14.589 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.581+0000 7f87e37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:59:14.589 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.581+0000 7f87e37fe640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.154:6789/0 2024-09-16T03:59:14.589 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.581+0000 7f87e37fe640 1 -- 172.21.15.42:0/3475885585 --> v1:172.21.15.154:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87d000afc0 con 0x7f87d0006160 2024-09-16T03:59:14.589 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.581+0000 7f87e37fe640 10 monclient: _finish_auth 0 2024-09-16T03:59:14.590 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.581+0000 7f87e37fe640 10 monclient: _check_auth_tickets 2024-09-16T03:59:14.590 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.581+0000 7f87e37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:59:14.590 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.581+0000 7f87e37fe640 1 -- 172.21.15.42:0/3475885585 <== mon.1 v1:172.21.15.154:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2177413734 0 0) 0x7f87d8001a40 con 0x7f87d0006160 2024-09-16T03:59:14.590 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.581+0000 7f87eabbb640 5 monclient: authenticate success, global_id 904300 2024-09-16T03:59:14.590 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.581+0000 7f87eabbb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:59:14.590 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.581+0000 7f87e37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:59:14.590 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.581+0000 7f87e37fe640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-16T03:59:14.590 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.581+0000 7f87e37fe640 10 monclient: dump: 2024-09-16T03:59:14.590 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:59:14.590 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:59:14.590 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:14.590 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:14.591 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:59:14.591 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:59:14.591 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:59:14.591 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:59:14.591 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:59:14.591 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:59:14.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.581+0000 7f87e37fe640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] but i'm connected to v1:172.21.15.154:6789/0, reconnecting 2024-09-16T03:59:14.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.581+0000 7f87e37fe640 10 monclient: _reopen_session rank -1 2024-09-16T03:59:14.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.581+0000 7f87e37fe640 1 -- 172.21.15.42:0/3475885585 >> v1:172.21.15.154:6789/0 conn(0x7f87d0006160 legacy=0x7f87d0008620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:59:14.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.581+0000 7f87e9158640 1 -- 172.21.15.42:0/3475885585 reap_dead start 2024-09-16T03:59:14.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.581+0000 7f87e37fe640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T03:59:14.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.581+0000 7f87e37fe640 1 --2- 172.21.15.42:0/3475885585 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f87cc0039c0 0x7f87cc003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:59:14.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.581+0000 7f87e37fe640 10 monclient(hunting): picked mon.b con 0x7f87cc0039c0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:59:14.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.581+0000 7f87e37fe640 1 --2- 172.21.15.42:0/3475885585 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f87cc004810 0x7f87cc004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:59:14.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.581+0000 7f87e37fe640 10 monclient(hunting): picked mon.c con 0x7f87cc004810 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:59:14.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.581+0000 7f87e37fe640 1 --2- 172.21.15.42:0/3475885585 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f87cc0052f0 0x7f87cc00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:59:14.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.581+0000 7f87e37fe640 10 monclient(hunting): picked mon.a con 0x7f87cc0052f0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:59:14.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.581+0000 7f87e37fe640 10 monclient(hunting): start opening mon connection 2024-09-16T03:59:14.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.581+0000 7f87e37fe640 1 -- 172.21.15.42:0/3475885585 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f87e409f610 con 0x7f87cc0052f0 2024-09-16T03:59:14.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.581+0000 7f87e37fe640 10 monclient(hunting): start opening mon connection 2024-09-16T03:59:14.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.581+0000 7f87e37fe640 1 -- 172.21.15.42:0/3475885585 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f87cc00c240 con 0x7f87cc0039c0 2024-09-16T03:59:14.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.581+0000 7f87e37fe640 10 monclient(hunting): start opening mon connection 2024-09-16T03:59:14.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.581+0000 7f87e37fe640 1 -- 172.21.15.42:0/3475885585 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f87cc00c440 con 0x7f87cc004810 2024-09-16T03:59:14.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.581+0000 7f87e37fe640 10 monclient(hunting): _renew_subs 2024-09-16T03:59:14.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.581+0000 7f87e37fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:59:14.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.581+0000 7f87eabbb640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-16T03:59:14.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.581+0000 7f87e3fff640 1 -- 172.21.15.42:0/3475885585 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f87cc004810 msgr2=0x7f87cc004c90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T03:59:14.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.581+0000 7f87e3fff640 1 --2- 172.21.15.42:0/3475885585 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f87cc004810 0x7f87cc004c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:59:14.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.581+0000 7f87e8957640 1 --2- 172.21.15.42:0/3475885585 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f87cc0039c0 0x7f87cc003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:59:14.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.581+0000 7f87e9158640 1 --2- 172.21.15.42:0/3475885585 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f87cc0052f0 0x7f87cc00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:59:14.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.581+0000 7f87e8957640 10 monclient(hunting): get_auth_request con 0x7f87cc0039c0 auth_method 0 2024-09-16T03:59:14.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.581+0000 7f87e8957640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:59:14.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.581+0000 7f87e8957640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:59:14.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.581+0000 7f87e8957640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:59:14.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.581+0000 7f87e9158640 10 monclient(hunting): get_auth_request con 0x7f87cc0052f0 auth_method 0 2024-09-16T03:59:14.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.581+0000 7f87e9158640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:59:14.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.581+0000 7f87e9158640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:59:14.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.581+0000 7f87e9158640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:59:14.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.581+0000 7f87e8957640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:59:14.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.581+0000 7f87e8957640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:59:14.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.581+0000 7f87e8957640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T03:59:14.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.581+0000 7f87e9158640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:59:14.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.581+0000 7f87e9158640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:59:14.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.581+0000 7f87e9158640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T03:59:14.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.581+0000 7f87e9158640 10 monclient(hunting): handle_auth_done global_id 904300 payload 293 2024-09-16T03:59:14.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.581+0000 7f87e9158640 1 -- 172.21.15.42:0/3475885585 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f87cc004810 msgr2=0x7f87cc004c90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:59:14.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.581+0000 7f87e9158640 1 --2- 172.21.15.42:0/3475885585 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f87cc004810 0x7f87cc004c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:59:14.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.581+0000 7f87e9158640 1 -- 172.21.15.42:0/3475885585 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f87cc0039c0 msgr2=0x7f87cc003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:59:14.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.581+0000 7f87e9158640 1 --2- 172.21.15.42:0/3475885585 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f87cc0039c0 0x7f87cc003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:59:14.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.581+0000 7f87e9158640 10 monclient: _finish_hunting 0 2024-09-16T03:59:14.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.581+0000 7f87e9158640 1 monclient: found mon.a 2024-09-16T03:59:14.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.581+0000 7f87e9158640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:59:14.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.581+0000 7f87e9158640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:59:14.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.581+0000 7f87e9158640 1 -- 172.21.15.42:0/3475885585 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87cc003530 con 0x7f87cc0052f0 2024-09-16T03:59:14.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.581+0000 7f87e9158640 10 monclient: _finish_auth 0 2024-09-16T03:59:14.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.581+0000 7f87e9158640 10 monclient: _check_auth_tickets 2024-09-16T03:59:14.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.581+0000 7f87e9158640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:59:14.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.581+0000 7f87e8957640 1 --2- 172.21.15.42:0/3475885585 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f87cc0039c0 0x7f87cc003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:59:14.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.589+0000 7f87e9158640 1 --2- 172.21.15.42:0/3475885585 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f87cc0052f0 0x7f87cc00b9d0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f87e40a2030 tx=0x7f87dc01cea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73fbc0f25a839cd0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:59:14.599 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.593+0000 7f87e37fe640 1 -- 172.21.15.42:0/3475885585 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87dc0ab070 con 0x7f87cc0052f0 2024-09-16T03:59:14.599 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.593+0000 7f87e37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:59:14.599 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.593+0000 7f87e37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:59:14.599 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.593+0000 7f87e37fe640 10 monclient: dump: 2024-09-16T03:59:14.599 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:59:14.599 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:59:14.599 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:14.599 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:14.599 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:59:14.599 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:59:14.599 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:59:14.599 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:59:14.600 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:59:14.600 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:59:14.600 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.593+0000 7f87eabbb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:59:14.600 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.593+0000 7f87e37fe640 1 -- 172.21.15.42:0/3475885585 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f87dc0a5c90 con 0x7f87cc0052f0 2024-09-16T03:59:14.600 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.593+0000 7f87e37fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:59:14.600 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.593+0000 7f87eabbb640 10 monclient: get_monmap_and_config success 2024-09-16T03:59:14.600 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.593+0000 7f87eabbb640 4 set_mon_vals no callback set 2024-09-16T03:59:14.600 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.593+0000 7f87e37fe640 1 -- 172.21.15.42:0/3475885585 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87dc00f040 con 0x7f87cc0052f0 2024-09-16T03:59:14.600 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.593+0000 7f87e37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:59:14.600 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.593+0000 7f87e37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:59:14.600 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.593+0000 7f87e37fe640 10 monclient: dump: 2024-09-16T03:59:14.601 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:59:14.601 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:59:14.601 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:14.601 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:14.601 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:59:14.601 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:59:14.601 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:59:14.601 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:59:14.601 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:59:14.601 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:59:14.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.593+0000 7f87eabbb640 10 monclient: shutdown 2024-09-16T03:59:14.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.593+0000 7f87eabbb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:59:14.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.593+0000 7f87eabbb640 1 -- 172.21.15.42:0/3475885585 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f87cc0052f0 msgr2=0x7f87cc00b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:59:14.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.593+0000 7f87eabbb640 1 --2- 172.21.15.42:0/3475885585 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f87cc0052f0 0x7f87cc00b9d0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f87e40a2030 tx=0x7f87dc01cea0 comp rx=0 tx=0).stop 2024-09-16T03:59:14.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.593+0000 7f87eabbb640 1 -- 172.21.15.42:0/3475885585 shutdown_connections 2024-09-16T03:59:14.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.593+0000 7f87eabbb640 1 --2- 172.21.15.42:0/3475885585 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f87cc0052f0 0x7f87cc00b9d0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:59:14.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.593+0000 7f87eabbb640 1 --2- 172.21.15.42:0/3475885585 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f87cc004810 0x7f87cc004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:59:14.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.593+0000 7f87eabbb640 1 --2- 172.21.15.42:0/3475885585 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f87cc0039c0 0x7f87cc003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:59:14.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.593+0000 7f87eabbb640 1 -- 172.21.15.42:0/3475885585 >> 172.21.15.42:0/3475885585 conn(0x7f87e40a2420 msgr2=0x7f87e40a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:59:14.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.593+0000 7f87eabbb640 1 -- 172.21.15.42:0/3475885585 shutdown_connections 2024-09-16T03:59:14.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.593+0000 7f87eabbb640 1 -- 172.21.15.42:0/3475885585 wait complete. 2024-09-16T03:59:14.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.597+0000 7f87eabbb640 10 monclient: build_initial_monmap 2024-09-16T03:59:14.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.597+0000 7f87eabbb640 10 monclient: monmap: 2024-09-16T03:59:14.608 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:59:14.608 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:59:14.608 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:59:14.608 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:59:14.608 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:59:14.608 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:59:14.608 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:59:14.608 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:59:14.608 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:59:14.608 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:59:14.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.597+0000 7f87eabbb640 1 librados: starting msgr at 2024-09-16T03:59:14.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.597+0000 7f87eabbb640 1 librados: starting objecter 2024-09-16T03:59:14.609 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.597+0000 7f87eabbb640 1 Processor -- start 2024-09-16T03:59:14.609 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.597+0000 7f87eabbb640 1 -- start start 2024-09-16T03:59:14.609 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.597+0000 7f87eabbb640 1 librados: setting wanted keys 2024-09-16T03:59:14.609 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.597+0000 7f87eabbb640 1 librados: calling monclient init 2024-09-16T03:59:14.609 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.597+0000 7f87eabbb640 10 monclient: init 2024-09-16T03:59:14.609 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.597+0000 7f87eabbb640 10 monclient: _reopen_session rank -1 2024-09-16T03:59:14.609 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.597+0000 7f87eabbb640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:59:14.609 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.597+0000 7f87eabbb640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f87cc0039c0 0x7f87e41cb880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:59:14.609 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.597+0000 7f87eabbb640 10 monclient(hunting): picked mon.noname-c con 0x7f87cc0039c0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:59:14.609 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.597+0000 7f87eabbb640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f87cc004810 0x7f87e4135170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:59:14.609 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.597+0000 7f87eabbb640 10 monclient(hunting): picked mon.noname-b con 0x7f87cc004810 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:59:14.609 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.597+0000 7f87eabbb640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f87cc0052f0 0x7f87e41356b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:59:14.609 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.597+0000 7f87eabbb640 10 monclient(hunting): picked mon.noname-a con 0x7f87cc0052f0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:59:14.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.597+0000 7f87eabbb640 10 monclient(hunting): start opening mon connection 2024-09-16T03:59:14.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.597+0000 7f87eabbb640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f87e40a32f0 con 0x7f87cc0052f0 2024-09-16T03:59:14.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.597+0000 7f87eabbb640 10 monclient(hunting): start opening mon connection 2024-09-16T03:59:14.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.597+0000 7f87eabbb640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f87e40a3170 con 0x7f87cc004810 2024-09-16T03:59:14.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.597+0000 7f87eabbb640 10 monclient(hunting): start opening mon connection 2024-09-16T03:59:14.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.597+0000 7f87eabbb640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f87e409ed70 con 0x7f87cc0039c0 2024-09-16T03:59:14.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.597+0000 7f87eabbb640 10 monclient(hunting): _renew_subs 2024-09-16T03:59:14.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.597+0000 7f87eabbb640 10 monclient(hunting): authenticate will time out at 2001.569725s 2024-09-16T03:59:14.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.597+0000 7f87e8957640 1 -- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f87cc0039c0 msgr2=0x7f87e41cb880 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T03:59:14.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.597+0000 7f87e8957640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f87cc0039c0 0x7f87e41cb880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:59:14.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.597+0000 7f87e9158640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f87cc0052f0 0x7f87e41356b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:59:14.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.597+0000 7f87e9158640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f87cc0052f0 0x7f87e41356b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:39610/0 (socket says 172.21.15.42:39610) 2024-09-16T03:59:14.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.597+0000 7f87e9158640 1 -- 172.21.15.42:0/215136185 learned_addr learned my addr 172.21.15.42:0/215136185 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:59:14.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.597+0000 7f87e9158640 10 monclient(hunting): get_auth_request con 0x7f87cc0052f0 auth_method 0 2024-09-16T03:59:14.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.597+0000 7f87e9158640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:59:14.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.597+0000 7f87e9158640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:59:14.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.597+0000 7f87e9158640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:59:14.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.597+0000 7f87e9158640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:59:14.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.597+0000 7f87e9158640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:59:14.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.597+0000 7f87e9158640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:59:14.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.597+0000 7f87e3fff640 1 --2- 172.21.15.42:0/215136185 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f87cc004810 0x7f87e4135170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:59:14.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.597+0000 7f87e3fff640 10 monclient(hunting): get_auth_request con 0x7f87cc004810 auth_method 0 2024-09-16T03:59:14.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.597+0000 7f87e3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:59:14.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.597+0000 7f87e3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:59:14.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.597+0000 7f87e3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:59:14.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.597+0000 7f87e9158640 10 monclient(hunting): handle_auth_done global_id 825123 payload 824 2024-09-16T03:59:14.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.597+0000 7f87e9158640 1 -- 172.21.15.42:0/215136185 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f87cc0039c0 msgr2=0x7f87e41cb880 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:59:14.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.597+0000 7f87e9158640 1 --2- 172.21.15.42:0/215136185 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f87cc0039c0 0x7f87e41cb880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:59:14.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.597+0000 7f87e9158640 1 -- 172.21.15.42:0/215136185 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f87cc004810 msgr2=0x7f87e4135170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:59:14.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.597+0000 7f87e9158640 1 --2- 172.21.15.42:0/215136185 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f87cc004810 0x7f87e4135170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:59:14.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.597+0000 7f87e9158640 10 monclient: _finish_hunting 0 2024-09-16T03:59:14.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.597+0000 7f87e9158640 1 monclient: found mon.noname-a 2024-09-16T03:59:14.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.597+0000 7f87e9158640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:59:14.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.597+0000 7f87e9158640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:59:14.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.597+0000 7f87e9158640 1 -- 172.21.15.42:0/215136185 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87e4122850 con 0x7f87cc0052f0 2024-09-16T03:59:14.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.597+0000 7f87e9158640 1 --2- 172.21.15.42:0/215136185 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f87cc0052f0 0x7f87e41356b0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f87dc000eb0 tx=0x7f87dc019c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:59:14.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.597+0000 7f87e17fa640 10 client.?.objecter ms_handle_connect 0x7f87cc0052f0 2024-09-16T03:59:14.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.597+0000 7f87e17fa640 10 client.?.objecter resend_mon_ops 2024-09-16T03:59:14.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.597+0000 7f87e17fa640 1 -- 172.21.15.42:0/215136185 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87dc0ab040 con 0x7f87cc0052f0 2024-09-16T03:59:14.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.597+0000 7f87e17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:59:14.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.597+0000 7f87e17fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:59:14.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.597+0000 7f87e17fa640 10 monclient: dump: 2024-09-16T03:59:14.613 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:59:14.613 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:59:14.613 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:14.613 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:14.613 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:59:14.613 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:59:14.614 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:59:14.614 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:59:14.614 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:59:14.614 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:59:14.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.597+0000 7f87e17fa640 10 monclient: _finish_auth 0 2024-09-16T03:59:14.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.597+0000 7f87e17fa640 10 monclient: _check_auth_tickets 2024-09-16T03:59:14.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.597+0000 7f87e17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:59:14.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.597+0000 7f87eabbb640 5 monclient: authenticate success, global_id 825123 2024-09-16T03:59:14.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.597+0000 7f87e17fa640 1 -- 172.21.15.42:0/215136185 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f87dc0bf070 con 0x7f87cc0052f0 2024-09-16T03:59:14.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.597+0000 7f87e17fa640 10 monclient: handle_config config(0 keys) 2024-09-16T03:59:14.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.597+0000 7f87eabbb640 10 monclient: _renew_subs 2024-09-16T03:59:14.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.597+0000 7f87e17fa640 1 -- 172.21.15.42:0/215136185 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87dc0b9400 con 0x7f87cc0052f0 2024-09-16T03:59:14.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.597+0000 7f87eabbb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:59:14.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.597+0000 7f87eabbb640 1 -- 172.21.15.42:0/215136185 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87e41cbdc0 con 0x7f87cc0052f0 2024-09-16T03:59:14.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.597+0000 7f87e17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:59:14.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.597+0000 7f87e17fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:59:14.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.597+0000 7f87e2ffd640 4 set_mon_vals no callback set 2024-09-16T03:59:14.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.597+0000 7f87e17fa640 10 monclient: dump: 2024-09-16T03:59:14.615 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:59:14.615 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:59:14.615 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:14.615 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:14.615 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:59:14.615 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:59:14.615 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:59:14.615 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:59:14.615 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:59:14.615 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:59:14.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.597+0000 7f87eabbb640 10 client.825123.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:59:14.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.597+0000 7f87eabbb640 10 monclient: _renew_subs 2024-09-16T03:59:14.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.597+0000 7f87eabbb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:59:14.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.597+0000 7f87eabbb640 1 -- 172.21.15.42:0/215136185 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87e40a0e40 con 0x7f87cc0052f0 2024-09-16T03:59:14.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.597+0000 7f87eabbb640 1 librados: init done 2024-09-16T03:59:14.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.601+0000 7f87eabbb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:59:14.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.601+0000 7f87eabbb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:59:14.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.601+0000 7f87eabbb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:59:14.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.601+0000 7f87eabbb640 1 -- 172.21.15.42:0/215136185 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f87e409f8f0 con 0x7f87cc0052f0 2024-09-16T03:59:14.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.601+0000 7f87e17fa640 1 -- 172.21.15.42:0/215136185 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f87dc0b95a0 con 0x7f87cc0052f0 2024-09-16T03:59:14.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.601+0000 7f87e17fa640 1 --2- 172.21.15.42:0/215136185 >> v2:172.21.15.154:6812/254845342 conn(0x7f87d0038040 0x7f87d003a500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:59:14.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.601+0000 7f87e17fa640 1 -- 172.21.15.42:0/215136185 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(516..516 src has 451..516) ==== 5258+0+0 (secure 0 0 0) 0x7f87dc0e3d00 con 0x7f87cc0052f0 2024-09-16T03:59:14.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.601+0000 7f87e17fa640 10 client.825123.objecter ms_dispatch 0x7f87e4000ba0 osd_map(516..516 src has 451..516) 2024-09-16T03:59:14.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.601+0000 7f87e17fa640 3 client.825123.objecter handle_osd_map got epochs [516,516] > 0 2024-09-16T03:59:14.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.601+0000 7f87e17fa640 3 client.825123.objecter handle_osd_map decoding full epoch 516 2024-09-16T03:59:14.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.601+0000 7f87e17fa640 20 client.825123.objecter dump_active .. 0 homeless 2024-09-16T03:59:14.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.601+0000 7f87e8957640 1 --2- 172.21.15.42:0/215136185 >> v2:172.21.15.154:6812/254845342 conn(0x7f87d0038040 0x7f87d003a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:59:14.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.601+0000 7f87e8957640 10 monclient: get_auth_request con 0x7f87d0038040 auth_method 0 2024-09-16T03:59:14.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.601+0000 7f87e8957640 1 --2- 172.21.15.42:0/215136185 >> v2:172.21.15.154:6812/254845342 conn(0x7f87d0038040 0x7f87d003a500 secure :-1 s=READY pgs=3724 cs=0 l=1 rev1=1 crypto rx=0x7f87d4000ee0 tx=0x7f87d400e440 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:59:14.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.601+0000 7f87e17fa640 10 client.825123.objecter ms_handle_connect 0x7f87d0038040 2024-09-16T03:59:14.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.613+0000 7f87e17fa640 1 -- 172.21.15.42:0/215136185 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f87dc016c40 con 0x7f87cc0052f0 2024-09-16T03:59:14.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.613+0000 7f87e17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:59:14.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.613+0000 7f87e17fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:59:14.843 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.837+0000 7f87eabbb640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:59:14.844 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.837+0000 7f87eabbb640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:59:14.844 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.837+0000 7f87eabbb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:59:14.844 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.837+0000 7f87eabbb640 1 -- 172.21.15.42:0/215136185 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f87e4121120 con 0x7f87cc0052f0 2024-09-16T03:59:14.844 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.837+0000 7f87e17fa640 1 -- 172.21.15.42:0/215136185 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1555 (secure 0 0 0) 0x7f87dc0e2460 con 0x7f87cc0052f0 2024-09-16T03:59:14.844 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.837+0000 7f87e17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:59:14.844 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.837+0000 7f87e17fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:59:14.845 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":158,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:59:14.847 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.841+0000 7f87eabbb640 10 librados: watch_flush enter 2024-09-16T03:59:14.847 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.841+0000 7f87eabbb640 10 librados: watch_flush exit 2024-09-16T03:59:14.847 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.841+0000 7f87eabbb640 20 client.825123.objecter shutdown clearing up homeless session... 2024-09-16T03:59:14.847 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.841+0000 7f87eabbb640 10 client.825123.objecter successfully canceled tick 2024-09-16T03:59:14.847 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.841+0000 7f87eabbb640 1 -- 172.21.15.42:0/215136185 >> v2:172.21.15.154:6812/254845342 conn(0x7f87d0038040 msgr2=0x7f87d003a500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:59:14.847 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.841+0000 7f87eabbb640 1 --2- 172.21.15.42:0/215136185 >> v2:172.21.15.154:6812/254845342 conn(0x7f87d0038040 0x7f87d003a500 secure :-1 s=READY pgs=3724 cs=0 l=1 rev1=1 crypto rx=0x7f87d4000ee0 tx=0x7f87d400e440 comp rx=0 tx=0).stop 2024-09-16T03:59:14.847 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.841+0000 7f87eabbb640 10 monclient: shutdown 2024-09-16T03:59:14.847 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.841+0000 7f87eabbb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:59:14.847 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.841+0000 7f87eabbb640 1 -- 172.21.15.42:0/215136185 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f87cc0052f0 msgr2=0x7f87e41356b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:59:14.847 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.841+0000 7f87eabbb640 1 --2- 172.21.15.42:0/215136185 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f87cc0052f0 0x7f87e41356b0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f87dc000eb0 tx=0x7f87dc019c90 comp rx=0 tx=0).stop 2024-09-16T03:59:14.847 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.841+0000 7f87eabbb640 1 -- 172.21.15.42:0/215136185 shutdown_connections 2024-09-16T03:59:14.847 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.841+0000 7f87eabbb640 1 --2- 172.21.15.42:0/215136185 >> v2:172.21.15.154:6812/254845342 conn(0x7f87d0038040 0x7f87d003a500 unknown :-1 s=CLOSED pgs=3724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:59:14.847 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.841+0000 7f87eabbb640 1 --2- 172.21.15.42:0/215136185 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f87cc0052f0 0x7f87e41356b0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:59:14.848 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.841+0000 7f87eabbb640 1 --2- 172.21.15.42:0/215136185 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f87cc004810 0x7f87e4135170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:59:14.848 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.841+0000 7f87eabbb640 1 --2- 172.21.15.42:0/215136185 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f87cc0039c0 0x7f87e41cb880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:59:14.848 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.841+0000 7f87eabbb640 1 -- 172.21.15.42:0/215136185 >> 172.21.15.42:0/215136185 conn(0x7f87e40a2420 msgr2=0x7f87e40a45b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:59:14.848 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.841+0000 7f87eabbb640 1 -- 172.21.15.42:0/215136185 shutdown_connections 2024-09-16T03:59:14.848 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.841+0000 7f87eabbb640 1 -- 172.21.15.42:0/215136185 wait complete. 2024-09-16T03:59:14.848 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.841+0000 7f87eabbb640 1 librados: shutdown 2024-09-16T03:59:14.864 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T03:59:14.864 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-16T03:59:14.864 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T03:59:14.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.985+0000 7fac51ba1640 10 monclient: get_monmap_and_config 2024-09-16T03:59:14.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.985+0000 7fac51ba1640 10 monclient: build_initial_monmap 2024-09-16T03:59:14.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.985+0000 7fac51ba1640 10 monclient: monmap: 2024-09-16T03:59:14.990 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:59:14.990 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:59:14.990 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:59:14.988393+0000 2024-09-16T03:59:14.990 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:59:14.988393+0000 2024-09-16T03:59:14.990 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:59:14.990 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:59:14.990 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:59:14.990 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:59:14.990 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:59:14.991 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:59:14.991 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:59:14.991 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:59:14.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.985+0000 7fac51ba1640 1 Processor -- start 2024-09-16T03:59:14.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.985+0000 7fac51ba1640 1 -- start start 2024-09-16T03:59:14.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.985+0000 7fac51ba1640 10 monclient: init 2024-09-16T03:59:14.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.985+0000 7fac51ba1640 10 monclient: _reopen_session rank -1 2024-09-16T03:59:14.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.985+0000 7fac51ba1640 10 monclient: _add_conns ranks=[2,1,4,3,0] 2024-09-16T03:59:14.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.985+0000 7fac51ba1640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fac4c0a3450 0x7fac4c0a3830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:59:14.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.985+0000 7fac51ba1640 10 monclient(hunting): picked mon.noname-b con 0x7fac4c0a3450 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:59:14.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.985+0000 7fac51ba1640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fac4c0a0ea0 addr v1:172.21.15.154:6789/0 2024-09-16T03:59:14.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.985+0000 7fac51ba1640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fac4c0a3d70 0x7fac4c137930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:59:14.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.985+0000 7fac51ba1640 10 monclient(hunting): picked mon.noname-c con 0x7fac4c0a3d70 addr v2:172.21.15.154:3300/0 2024-09-16T03:59:14.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.985+0000 7fac51ba1640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fac4c0a00e0 con 0x7fac4c0a0ea0 2024-09-16T03:59:14.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.985+0000 7fac51ba1640 10 monclient(hunting): start opening mon connection 2024-09-16T03:59:14.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.985+0000 7fac51ba1640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fac4c0a1b40 con 0x7fac4c0a3d70 2024-09-16T03:59:14.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.985+0000 7fac51ba1640 10 monclient(hunting): start opening mon connection 2024-09-16T03:59:14.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.985+0000 7fac51ba1640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fac4c0a1d90 con 0x7fac4c0a3450 2024-09-16T03:59:14.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.985+0000 7fac51ba1640 10 monclient(hunting): _renew_subs 2024-09-16T03:59:14.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.985+0000 7fac51ba1640 10 monclient(hunting): authenticate will time out at 2001.957800s 2024-09-16T03:59:14.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.985+0000 7fac4b7fe640 1 -- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fac4c0a3450 msgr2=0x7fac4c0a3830 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T03:59:14.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.985+0000 7fac4b7fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fac4c0a3450 0x7fac4c0a3830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:59:14.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.985+0000 7fac4bfff640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fac4c0a3d70 0x7fac4c137930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:59:14.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.985+0000 7fac4affd640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7fac4c0a0ea0 0x7fac4c137220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.42:47654/0 (socket says 172.21.15.42:47654) 2024-09-16T03:59:14.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.985+0000 7fac4affd640 1 -- 172.21.15.42:0/2453287221 learned_addr learned my addr 172.21.15.42:0/2453287221 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:59:14.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.985+0000 7fac4bfff640 10 monclient(hunting): get_auth_request con 0x7fac4c0a3d70 auth_method 0 2024-09-16T03:59:14.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.985+0000 7fac4bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:59:14.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.985+0000 7fac4bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:59:14.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.985+0000 7fac4bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:59:14.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.985+0000 7fac4bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:59:14.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.985+0000 7fac4bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:59:14.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.985+0000 7fac4bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:59:14.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.985+0000 7fac4a7fc640 1 -- 172.21.15.42:0/2453287221 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1141006257 0 0) 0x7fac4c0a00e0 con 0x7fac4c0a0ea0 2024-09-16T03:59:14.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.985+0000 7fac4a7fc640 10 monclient(hunting): _negotiate 2024-09-16T03:59:14.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.985+0000 7fac4a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:59:14.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.985+0000 7fac4a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:59:14.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.985+0000 7fac4a7fc640 10 monclient(hunting): my global_id is 904309 2024-09-16T03:59:14.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.985+0000 7fac4a7fc640 1 -- 172.21.15.42:0/2453287221 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fac34003530 con 0x7fac4c0a0ea0 2024-09-16T03:59:14.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.985+0000 7fac4bfff640 10 monclient(hunting): handle_auth_done global_id 904306 payload 274 2024-09-16T03:59:14.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.985+0000 7fac4bfff640 1 -- 172.21.15.42:0/2453287221 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fac4c0a3450 msgr2=0x7fac4c0a3830 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:59:14.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.985+0000 7fac4bfff640 1 --2- 172.21.15.42:0/2453287221 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fac4c0a3450 0x7fac4c0a3830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:59:14.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.985+0000 7fac4bfff640 1 -- 172.21.15.42:0/2453287221 >> v1:172.21.15.154:6789/0 conn(0x7fac4c0a0ea0 legacy=0x7fac4c137220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:59:14.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.985+0000 7fac4bfff640 10 monclient: _finish_hunting 0 2024-09-16T03:59:14.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.985+0000 7fac4bfff640 1 monclient: found mon.noname-c 2024-09-16T03:59:14.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.985+0000 7fac4bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:59:14.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.985+0000 7fac4bfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.154:3300/0 2024-09-16T03:59:14.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.985+0000 7fac4bfff640 1 -- 172.21.15.42:0/2453287221 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac4c09fea0 con 0x7fac4c0a3d70 2024-09-16T03:59:15.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.989+0000 7fac4bfff640 1 --2- 172.21.15.42:0/2453287221 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fac4c0a3d70 0x7fac4c137930 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fac44009d90 tx=0x7fac4401c850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7fd63b97946dcd50 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:59:15.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.989+0000 7fac4a7fc640 1 -- 172.21.15.42:0/2453287221 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac440b2070 con 0x7fac4c0a3d70 2024-09-16T03:59:15.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.989+0000 7fac4a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:59:15.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.989+0000 7fac4a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:59:15.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.989+0000 7fac4a7fc640 10 monclient: dump: 2024-09-16T03:59:15.003 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:59:15.003 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:59:15.003 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:15.003 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:15.003 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:59:15.003 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:59:15.003 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:59:15.003 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:59:15.003 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:59:15.003 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:59:15.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.989+0000 7fac4a7fc640 10 monclient: _finish_auth 0 2024-09-16T03:59:15.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.989+0000 7fac4a7fc640 10 monclient: _check_auth_tickets 2024-09-16T03:59:15.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.989+0000 7fac4a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:59:15.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.989+0000 7fac4a7fc640 1 -- 172.21.15.42:0/2453287221 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac440a5820 con 0x7fac4c0a3d70 2024-09-16T03:59:15.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.989+0000 7fac4a7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:59:15.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.989+0000 7fac4a7fc640 1 -- 172.21.15.42:0/2453287221 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac440a5bf0 con 0x7fac4c0a3d70 2024-09-16T03:59:15.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.989+0000 7fac4a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:59:15.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.989+0000 7fac4a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:59:15.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.989+0000 7fac4a7fc640 10 monclient: dump: 2024-09-16T03:59:15.004 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:59:15.004 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:59:15.004 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:15.004 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:15.004 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:59:15.004 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:59:15.004 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:59:15.004 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:59:15.004 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:59:15.005 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:59:15.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.989+0000 7fac51ba1640 5 monclient: authenticate success, global_id 904306 2024-09-16T03:59:15.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.989+0000 7fac51ba1640 10 monclient: get_monmap_and_config success 2024-09-16T03:59:15.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.989+0000 7fac51ba1640 4 set_mon_vals no callback set 2024-09-16T03:59:15.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.989+0000 7fac51ba1640 10 monclient: shutdown 2024-09-16T03:59:15.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.989+0000 7fac51ba1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:59:15.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.989+0000 7fac51ba1640 1 -- 172.21.15.42:0/2453287221 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fac4c0a3d70 msgr2=0x7fac4c137930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:59:15.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.989+0000 7fac51ba1640 1 --2- 172.21.15.42:0/2453287221 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fac4c0a3d70 0x7fac4c137930 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fac44009d90 tx=0x7fac4401c850 comp rx=0 tx=0).stop 2024-09-16T03:59:15.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.989+0000 7fac51ba1640 1 -- 172.21.15.42:0/2453287221 shutdown_connections 2024-09-16T03:59:15.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.989+0000 7fac51ba1640 1 --2- 172.21.15.42:0/2453287221 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fac4c0a3d70 0x7fac4c137930 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:59:15.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.989+0000 7fac51ba1640 1 --2- 172.21.15.42:0/2453287221 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fac4c0a3450 0x7fac4c0a3830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:59:15.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.989+0000 7fac51ba1640 1 -- 172.21.15.42:0/2453287221 >> 172.21.15.42:0/2453287221 conn(0x7fac4c0756e0 msgr2=0x7fac4c075af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:59:15.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.989+0000 7fac51ba1640 1 -- 172.21.15.42:0/2453287221 shutdown_connections 2024-09-16T03:59:15.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.989+0000 7fac51ba1640 1 -- 172.21.15.42:0/2453287221 wait complete. 2024-09-16T03:59:15.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.989+0000 7fac51ba1640 10 monclient: build_initial_monmap 2024-09-16T03:59:15.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.989+0000 7fac51ba1640 10 monclient: monmap: 2024-09-16T03:59:15.006 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:59:15.006 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:59:15.006 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:59:15.006 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:59:15.006 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:59:15.006 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:59:15.006 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:59:15.006 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:59:15.006 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:59:15.006 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:59:15.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.989+0000 7fac51ba1640 1 librados: starting msgr at 2024-09-16T03:59:15.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.989+0000 7fac51ba1640 1 librados: starting objecter 2024-09-16T03:59:15.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.989+0000 7fac51ba1640 1 Processor -- start 2024-09-16T03:59:15.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.989+0000 7fac51ba1640 1 -- start start 2024-09-16T03:59:15.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.989+0000 7fac51ba1640 1 librados: setting wanted keys 2024-09-16T03:59:15.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.989+0000 7fac51ba1640 1 librados: calling monclient init 2024-09-16T03:59:15.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.989+0000 7fac51ba1640 10 monclient: init 2024-09-16T03:59:15.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.989+0000 7fac51ba1640 10 monclient: _reopen_session rank -1 2024-09-16T03:59:15.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.989+0000 7fac51ba1640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:59:15.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.989+0000 7fac51ba1640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fac4c0a0ea0 0x7fac4c11e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:59:15.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.989+0000 7fac51ba1640 10 monclient(hunting): picked mon.noname-c con 0x7fac4c0a0ea0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:59:15.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.989+0000 7fac51ba1640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fac4c0a3450 0x7fac4c11e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:59:15.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.989+0000 7fac51ba1640 10 monclient(hunting): picked mon.noname-b con 0x7fac4c0a3450 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:59:15.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.989+0000 7fac51ba1640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fac4c0a3d70 0x7fac4c11eb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:59:15.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.989+0000 7fac51ba1640 10 monclient(hunting): picked mon.noname-a con 0x7fac4c0a3d70 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:59:15.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.989+0000 7fac51ba1640 10 monclient(hunting): start opening mon connection 2024-09-16T03:59:15.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.989+0000 7fac51ba1640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fac4c137ff0 con 0x7fac4c0a3d70 2024-09-16T03:59:15.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.989+0000 7fac51ba1640 10 monclient(hunting): start opening mon connection 2024-09-16T03:59:15.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.989+0000 7fac51ba1640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fac4c137e70 con 0x7fac4c0a3450 2024-09-16T03:59:15.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.989+0000 7fac51ba1640 10 monclient(hunting): start opening mon connection 2024-09-16T03:59:15.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.989+0000 7fac51ba1640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fac4c141ca0 con 0x7fac4c0a0ea0 2024-09-16T03:59:15.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.989+0000 7fac4b7fe640 1 -- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fac4c0a0ea0 msgr2=0x7fac4c11e090 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T03:59:15.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.989+0000 7fac51ba1640 10 monclient(hunting): _renew_subs 2024-09-16T03:59:15.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.989+0000 7fac51ba1640 10 monclient(hunting): authenticate will time out at 2001.963910s 2024-09-16T03:59:15.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.989+0000 7fac4b7fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fac4c0a0ea0 0x7fac4c11e090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:59:15.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.993+0000 7fac4bfff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fac4c0a3d70 0x7fac4c11eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:59:15.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.993+0000 7fac4bfff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fac4c0a3d70 0x7fac4c11eb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:39614/0 (socket says 172.21.15.42:39614) 2024-09-16T03:59:15.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.993+0000 7fac4bfff640 1 -- 172.21.15.42:0/4280791378 learned_addr learned my addr 172.21.15.42:0/4280791378 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:59:15.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.993+0000 7fac4bfff640 10 monclient(hunting): get_auth_request con 0x7fac4c0a3d70 auth_method 0 2024-09-16T03:59:15.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.993+0000 7fac4bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:59:15.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.993+0000 7fac4bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:59:15.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.993+0000 7fac4bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:59:15.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.993+0000 7fac4affd640 1 --2- 172.21.15.42:0/4280791378 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fac4c0a3450 0x7fac4c11e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:59:15.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.993+0000 7fac4bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:59:15.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.993+0000 7fac4bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:59:15.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.993+0000 7fac4bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:59:15.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.993+0000 7fac4affd640 10 monclient(hunting): get_auth_request con 0x7fac4c0a3450 auth_method 0 2024-09-16T03:59:15.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.993+0000 7fac4affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:59:15.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.993+0000 7fac4affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:59:15.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.993+0000 7fac4affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:59:15.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.993+0000 7fac4affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:59:15.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.993+0000 7fac4affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:59:15.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.993+0000 7fac4affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:59:15.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.993+0000 7fac4bfff640 10 monclient(hunting): handle_auth_done global_id 825126 payload 824 2024-09-16T03:59:15.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.993+0000 7fac4bfff640 1 -- 172.21.15.42:0/4280791378 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fac4c0a0ea0 msgr2=0x7fac4c11e090 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:59:15.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.993+0000 7fac4bfff640 1 --2- 172.21.15.42:0/4280791378 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fac4c0a0ea0 0x7fac4c11e090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:59:15.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.993+0000 7fac4bfff640 1 -- 172.21.15.42:0/4280791378 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fac4c0a3450 msgr2=0x7fac4c11e5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:59:15.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.993+0000 7fac4bfff640 1 --2- 172.21.15.42:0/4280791378 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fac4c0a3450 0x7fac4c11e5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:59:15.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.993+0000 7fac4bfff640 10 monclient: _finish_hunting 0 2024-09-16T03:59:15.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.993+0000 7fac4bfff640 1 monclient: found mon.noname-a 2024-09-16T03:59:15.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.993+0000 7fac4bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:59:15.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.993+0000 7fac4bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:59:15.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.993+0000 7fac4bfff640 1 -- 172.21.15.42:0/4280791378 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac4c0710d0 con 0x7fac4c0a3d70 2024-09-16T03:59:15.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.993+0000 7fac4bfff640 1 --2- 172.21.15.42:0/4280791378 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fac4c0a3d70 0x7fac4c11eb10 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fac4401af00 tx=0x7fac440195f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:59:15.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.993+0000 7fac33fff640 10 client.?.objecter ms_handle_connect 0x7fac4c0a3d70 2024-09-16T03:59:15.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.993+0000 7fac33fff640 10 client.?.objecter resend_mon_ops 2024-09-16T03:59:15.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.993+0000 7fac33fff640 1 -- 172.21.15.42:0/4280791378 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac440b2050 con 0x7fac4c0a3d70 2024-09-16T03:59:15.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.993+0000 7fac33fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:59:15.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.993+0000 7fac33fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:59:15.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.993+0000 7fac33fff640 10 monclient: dump: 2024-09-16T03:59:15.011 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:59:15.011 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:59:15.011 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:15.011 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:15.011 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:59:15.011 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:59:15.011 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:59:15.011 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:59:15.011 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:59:15.011 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:59:15.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.993+0000 7fac33fff640 10 monclient: _finish_auth 0 2024-09-16T03:59:15.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.993+0000 7fac33fff640 10 monclient: _check_auth_tickets 2024-09-16T03:59:15.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.993+0000 7fac33fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:59:15.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.993+0000 7fac51ba1640 5 monclient: authenticate success, global_id 825126 2024-09-16T03:59:15.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.993+0000 7fac51ba1640 10 monclient: _renew_subs 2024-09-16T03:59:15.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.993+0000 7fac33fff640 1 -- 172.21.15.42:0/4280791378 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac440be070 con 0x7fac4c0a3d70 2024-09-16T03:59:15.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.993+0000 7fac51ba1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:59:15.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.993+0000 7fac51ba1640 1 -- 172.21.15.42:0/4280791378 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac4c120020 con 0x7fac4c0a3d70 2024-09-16T03:59:15.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.993+0000 7fac33fff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:59:15.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.993+0000 7fac33fff640 1 -- 172.21.15.42:0/4280791378 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac440ad850 con 0x7fac4c0a3d70 2024-09-16T03:59:15.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.993+0000 7fac33fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:59:15.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.993+0000 7fac33fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:59:15.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.993+0000 7fac33fff640 10 monclient: dump: 2024-09-16T03:59:15.012 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:59:15.013 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:59:15.013 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:15.013 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:15.013 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:59:15.013 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:59:15.013 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:59:15.013 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:59:15.013 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:59:15.013 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:59:15.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.993+0000 7fac51ba1640 10 client.825126.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:59:15.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.993+0000 7fac51ba1640 10 monclient: _renew_subs 2024-09-16T03:59:15.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.993+0000 7fac497fa640 4 set_mon_vals no callback set 2024-09-16T03:59:15.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.993+0000 7fac51ba1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:59:15.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.993+0000 7fac51ba1640 1 -- 172.21.15.42:0/4280791378 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac4c06e440 con 0x7fac4c0a3d70 2024-09-16T03:59:15.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.993+0000 7fac51ba1640 1 librados: init done 2024-09-16T03:59:15.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.993+0000 7fac51ba1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:59:15.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.993+0000 7fac51ba1640 1 --2- 172.21.15.42:0/4280791378 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fac4c136770 0x7fac4c136b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:59:15.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.993+0000 7fac51ba1640 10 monclient(hunting): start opening mon connection 2024-09-16T03:59:15.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.993+0000 7fac51ba1640 1 -- 172.21.15.42:0/4280791378 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fac4c121380 con 0x7fac4c136770 2024-09-16T03:59:15.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.993+0000 7fac4b7fe640 1 --2- 172.21.15.42:0/4280791378 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fac4c136770 0x7fac4c136b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:59:15.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.993+0000 7fac4b7fe640 10 monclient: get_auth_request con 0x7fac4c136770 auth_method 0 2024-09-16T03:59:15.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.993+0000 7fac4b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:59:15.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.993+0000 7fac4b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:59:15.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.993+0000 7fac4b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:59:15.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.993+0000 7fac4b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:59:15.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.993+0000 7fac4b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:59:15.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.993+0000 7fac4b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:59:15.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.993+0000 7fac33fff640 1 -- 172.21.15.42:0/4280791378 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fac440ad9f0 con 0x7fac4c0a3d70 2024-09-16T03:59:15.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.993+0000 7fac4b7fe640 10 monclient(hunting): handle_auth_done global_id 825129 payload 824 2024-09-16T03:59:15.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.993+0000 7fac4b7fe640 1 -- 172.21.15.42:0/4280791378 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fac4c0a3830 con 0x7fac4c136770 2024-09-16T03:59:15.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.993+0000 7fac33fff640 1 --2- 172.21.15.42:0/4280791378 >> v2:172.21.15.154:6812/254845342 conn(0x7fac380377c0 0x7fac38039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:59:15.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.993+0000 7fac33fff640 1 -- 172.21.15.42:0/4280791378 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(516..516 src has 451..516) ==== 5258+0+0 (secure 0 0 0) 0x7fac440e3a60 con 0x7fac4c0a3d70 2024-09-16T03:59:15.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.993+0000 7fac33fff640 10 client.825126.objecter ms_dispatch 0x7fac4c000ba0 osd_map(516..516 src has 451..516) 2024-09-16T03:59:15.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.993+0000 7fac33fff640 3 client.825126.objecter handle_osd_map got epochs [516,516] > 0 2024-09-16T03:59:15.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.993+0000 7fac33fff640 3 client.825126.objecter handle_osd_map decoding full epoch 516 2024-09-16T03:59:15.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.993+0000 7fac4b7fe640 1 --2- 172.21.15.42:0/4280791378 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fac4c136770 0x7fac4c136b50 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fac4000b450 tx=0x7fac4000b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:59:15.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.993+0000 7fac33fff640 20 client.825126.objecter dump_active .. 0 homeless 2024-09-16T03:59:15.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.993+0000 7fac33fff640 10 client.825126.objecter ms_handle_connect 0x7fac4c136770 2024-09-16T03:59:15.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.993+0000 7fac33fff640 10 client.825126.objecter resend_mon_ops 2024-09-16T03:59:15.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.993+0000 7fac33fff640 1 -- 172.21.15.42:0/4280791378 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac40003e20 con 0x7fac4c136770 2024-09-16T03:59:15.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.993+0000 7fac33fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:59:15.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.993+0000 7fac33fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:59:15.016 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.993+0000 7fac33fff640 10 monclient: dump: 2024-09-16T03:59:15.016 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:59:15.016 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:59:15.016 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:15.016 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:15.016 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:59:15.016 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:59:15.016 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:59:15.016 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:59:15.016 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:59:15.016 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:59:15.016 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.993+0000 7fac4affd640 1 --2- 172.21.15.42:0/4280791378 >> v2:172.21.15.154:6812/254845342 conn(0x7fac380377c0 0x7fac38039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:59:15.016 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.993+0000 7fac4affd640 10 monclient: get_auth_request con 0x7fac380377c0 auth_method 0 2024-09-16T03:59:15.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.993+0000 7fac33fff640 1 -- 172.21.15.42:0/4280791378 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fac40015070 con 0x7fac4c136770 2024-09-16T03:59:15.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.993+0000 7fac33fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:59:15.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.993+0000 7fac33fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:59:15.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.993+0000 7fac33fff640 1 -- 172.21.15.42:0/4280791378 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fac4c136770 msgr2=0x7fac4c136b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:59:15.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.993+0000 7fac33fff640 1 --2- 172.21.15.42:0/4280791378 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fac4c136770 0x7fac4c136b50 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fac4000b450 tx=0x7fac4000b910 comp rx=0 tx=0).stop 2024-09-16T03:59:15.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.993+0000 7fac33fff640 1 -- 172.21.15.42:0/4280791378 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fac4c136770 msgr2=0x7fac4c136b50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:59:15.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.993+0000 7fac33fff640 1 --2- 172.21.15.42:0/4280791378 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fac4c136770 0x7fac4c136b50 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:59:15.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.993+0000 7fac33fff640 10 client.825126.objecter ms_handle_connect 0x7fac380377c0 2024-09-16T03:59:15.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:14.993+0000 7fac4affd640 1 --2- 172.21.15.42:0/4280791378 >> v2:172.21.15.154:6812/254845342 conn(0x7fac380377c0 0x7fac38039c80 secure :-1 s=READY pgs=3725 cs=0 l=1 rev1=1 crypto rx=0x7fac3c007fd0 tx=0x7fac3c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:59:15.019 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.001+0000 7fac51ba1640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:59:15.019 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.001+0000 7fac51ba1640 1 --2- 172.21.15.42:0/4280791378 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fac4c13edd0 0x7fac4c137090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:59:15.019 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.001+0000 7fac51ba1640 10 monclient(hunting): start opening mon connection 2024-09-16T03:59:15.019 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.001+0000 7fac51ba1640 1 -- 172.21.15.42:0/4280791378 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fac4c1375d0 con 0x7fac4c13edd0 2024-09-16T03:59:15.019 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.001+0000 7fac4b7fe640 1 --2- 172.21.15.42:0/4280791378 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fac4c13edd0 0x7fac4c137090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:59:15.019 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.001+0000 7fac4b7fe640 10 monclient: get_auth_request con 0x7fac4c13edd0 auth_method 0 2024-09-16T03:59:15.019 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.001+0000 7fac4b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:59:15.019 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.001+0000 7fac4b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:59:15.020 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.001+0000 7fac4b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:59:15.020 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.001+0000 7fac4b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:59:15.020 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.001+0000 7fac4b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:59:15.020 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.001+0000 7fac4b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:59:15.020 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.001+0000 7fac4b7fe640 10 monclient(hunting): handle_auth_done global_id 825132 payload 824 2024-09-16T03:59:15.020 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.001+0000 7fac4b7fe640 1 -- 172.21.15.42:0/4280791378 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fac4c068f60 con 0x7fac4c13edd0 2024-09-16T03:59:15.020 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.001+0000 7fac4b7fe640 1 --2- 172.21.15.42:0/4280791378 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fac4c13edd0 0x7fac4c137090 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fac4000b450 tx=0x7fac40005a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:59:15.020 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.001+0000 7fac33fff640 10 client.825126.objecter ms_handle_connect 0x7fac4c13edd0 2024-09-16T03:59:15.020 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.001+0000 7fac33fff640 10 client.825126.objecter resend_mon_ops 2024-09-16T03:59:15.020 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.001+0000 7fac33fff640 1 -- 172.21.15.42:0/4280791378 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac40010480 con 0x7fac4c13edd0 2024-09-16T03:59:15.020 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.001+0000 7fac33fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:59:15.020 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.001+0000 7fac33fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:59:15.020 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.001+0000 7fac33fff640 10 monclient: dump: 2024-09-16T03:59:15.021 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:59:15.021 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:59:15.021 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:15.021 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:15.021 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:59:15.021 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:59:15.021 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:59:15.021 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:59:15.021 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:59:15.021 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:59:15.021 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.001+0000 7fac33fff640 1 -- 172.21.15.42:0/4280791378 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4871 (secure 0 0 0) 0x7fac40010480 con 0x7fac4c13edd0 2024-09-16T03:59:15.021 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.001+0000 7fac33fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:59:15.021 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.001+0000 7fac33fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:59:15.021 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.001+0000 7fac33fff640 1 -- 172.21.15.42:0/4280791378 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fac4c13edd0 msgr2=0x7fac4c137090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:59:15.022 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.001+0000 7fac33fff640 1 --2- 172.21.15.42:0/4280791378 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fac4c13edd0 0x7fac4c137090 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fac4000b450 tx=0x7fac40005a30 comp rx=0 tx=0).stop 2024-09-16T03:59:15.022 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.001+0000 7fac33fff640 1 -- 172.21.15.42:0/4280791378 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fac4c13edd0 msgr2=0x7fac4c137090 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:59:15.022 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.001+0000 7fac33fff640 1 --2- 172.21.15.42:0/4280791378 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fac4c13edd0 0x7fac4c137090 secure :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fac4000b450 tx=0x7fac40005a30 comp rx=0 tx=0).stop 2024-09-16T03:59:15.022 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:59:15.022 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:59:15.022 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:59:15.022 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "leader", 2024-09-16T03:59:15.022 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 158, 2024-09-16T03:59:15.022 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 444855, 2024-09-16T03:59:15.022 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:59:15.022 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:59:15.023 INFO:teuthology.orchestra.run.smithi042.stdout: 1 2024-09-16T03:59:15.023 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:59:15.023 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 0, 2024-09-16T03:59:15.023 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:59:15.023 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:59:15.023 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:59:15.023 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:59:15.023 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:59:15.023 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:59:15.023 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:59:15.023 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:59:15.023 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:59:15.023 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:59:15.024 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:59:15.024 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:59:15.024 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:59:15.024 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:59:15.024 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:59:15.024 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:59:15.024 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:59:15.024 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:59:15.024 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:59:15.024 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:59:15.024 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:59:15.024 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:59:15.024 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:59:15.025 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:59:15.025 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:59:15.025 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:59:15.025 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:59:15.025 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:59:15.025 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:59:15.025 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:59:15.025 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:59:15.025 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:59:15.025 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:59:15.025 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:59:15.025 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:59:15.025 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:59:15.026 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:59:15.026 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:59:15.026 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:59:15.026 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:59:15.026 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:59:15.026 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:59:15.026 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:59:15.026 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:59:15.026 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:59:15.026 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:59:15.026 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:59:15.026 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:59:15.026 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:59:15.026 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:59:15.027 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:59:15.027 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:59:15.027 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:59:15.027 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:59:15.027 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:59:15.027 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:59:15.027 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:59:15.027 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:59:15.027 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:59:15.027 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:59:15.027 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:59:15.027 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:59:15.027 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:59:15.027 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:59:15.028 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:59:15.028 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:59:15.028 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:59:15.028 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:59:15.028 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:59:15.028 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:59:15.028 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:59:15.028 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:59:15.028 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:59:15.028 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:59:15.028 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:59:15.028 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:59:15.028 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:59:15.028 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:59:15.029 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:59:15.029 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:59:15.029 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:59:15.029 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:59:15.029 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:59:15.029 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:59:15.029 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:59:15.029 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:59:15.029 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:59:15.029 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:59:15.029 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:59:15.029 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:59:15.029 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:59:15.029 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:59:15.030 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:59:15.030 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:59:15.030 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:59:15.030 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:59:15.030 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:59:15.030 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:59:15.030 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:59:15.030 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:59:15.030 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:59:15.030 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:59:15.030 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:59:15.030 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:59:15.030 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:59:15.030 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:59:15.030 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:59:15.031 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:59:15.031 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:59:15.031 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:59:15.031 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:59:15.031 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:59:15.031 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:59:15.031 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:59:15.031 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:59:15.031 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:59:15.031 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:59:15.031 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:59:15.031 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:59:15.031 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:59:15.031 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:59:15.032 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:59:15.032 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:59:15.032 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:59:15.032 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:59:15.032 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:59:15.032 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:59:15.032 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:59:15.032 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:59:15.032 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:59:15.032 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:59:15.032 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:59:15.032 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:59:15.032 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:59:15.033 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:59:15.033 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:59:15.033 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:59:15.033 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:59:15.033 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:59:15.033 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:59:15.033 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:59:15.033 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T03:59:15.033 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:59:15.033 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:59:15.033 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:59:15.033 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 5 2024-09-16T03:59:15.033 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:59:15.033 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:59:15.033 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:59:15.034 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:59:15.034 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:59:15.034 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:59:15.034 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 7 2024-09-16T03:59:15.034 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:59:15.034 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:59:15.034 INFO:teuthology.orchestra.run.smithi042.stdout: "mgr": [ 2024-09-16T03:59:15.034 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:59:15.034 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:59:15.034 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:59:15.034 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:59:15.034 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:59:15.034 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:59:15.034 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:59:15.034 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:59:15.035 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:59:15.036 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.001+0000 7fac51ba1640 10 librados: watch_flush enter 2024-09-16T03:59:15.036 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.001+0000 7fac51ba1640 10 librados: watch_flush exit 2024-09-16T03:59:15.036 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.001+0000 7fac51ba1640 20 client.825126.objecter shutdown clearing up homeless session... 2024-09-16T03:59:15.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.001+0000 7fac51ba1640 10 client.825126.objecter successfully canceled tick 2024-09-16T03:59:15.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.001+0000 7fac51ba1640 1 -- 172.21.15.42:0/4280791378 >> v2:172.21.15.154:6812/254845342 conn(0x7fac380377c0 msgr2=0x7fac38039c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:59:15.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.001+0000 7fac51ba1640 1 --2- 172.21.15.42:0/4280791378 >> v2:172.21.15.154:6812/254845342 conn(0x7fac380377c0 0x7fac38039c80 secure :-1 s=READY pgs=3725 cs=0 l=1 rev1=1 crypto rx=0x7fac3c007fd0 tx=0x7fac3c009040 comp rx=0 tx=0).stop 2024-09-16T03:59:15.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.001+0000 7fac51ba1640 10 monclient: shutdown 2024-09-16T03:59:15.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.001+0000 7fac51ba1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:59:15.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.001+0000 7fac51ba1640 1 -- 172.21.15.42:0/4280791378 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fac4c0a3d70 msgr2=0x7fac4c11eb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:59:15.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.001+0000 7fac51ba1640 1 --2- 172.21.15.42:0/4280791378 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fac4c0a3d70 0x7fac4c11eb10 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fac4401af00 tx=0x7fac440195f0 comp rx=0 tx=0).stop 2024-09-16T03:59:15.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.001+0000 7fac4bfff640 1 -- 172.21.15.42:0/4280791378 reap_dead start 2024-09-16T03:59:15.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.001+0000 7fac4bfff640 1 -- 172.21.15.42:0/4280791378 reap_dead start 2024-09-16T03:59:15.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.001+0000 7fac51ba1640 1 -- 172.21.15.42:0/4280791378 shutdown_connections 2024-09-16T03:59:15.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.001+0000 7fac51ba1640 1 -- 172.21.15.42:0/4280791378 >> 172.21.15.42:0/4280791378 conn(0x7fac4c0756e0 msgr2=0x7fac4c06f7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:59:15.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.001+0000 7fac51ba1640 1 -- 172.21.15.42:0/4280791378 shutdown_connections 2024-09-16T03:59:15.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.001+0000 7fac51ba1640 1 -- 172.21.15.42:0/4280791378 wait complete. 2024-09-16T03:59:15.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.001+0000 7fac51ba1640 1 librados: shutdown 2024-09-16T03:59:15.038 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T03:59:15.038 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-16T03:59:15.038 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T03:59:15.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.145+0000 7fe75737f640 10 monclient: get_monmap_and_config 2024-09-16T03:59:15.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.145+0000 7fe75737f640 10 monclient: build_initial_monmap 2024-09-16T03:59:15.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.145+0000 7fe75737f640 10 monclient: monmap: 2024-09-16T03:59:15.150 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:59:15.150 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:59:15.150 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:59:15.148505+0000 2024-09-16T03:59:15.150 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:59:15.148505+0000 2024-09-16T03:59:15.150 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:59:15.150 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:59:15.150 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:59:15.151 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:59:15.151 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:59:15.151 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:59:15.151 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:59:15.151 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:59:15.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.145+0000 7fe75737f640 1 Processor -- start 2024-09-16T03:59:15.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.145+0000 7fe75737f640 1 -- start start 2024-09-16T03:59:15.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.145+0000 7fe75737f640 10 monclient: init 2024-09-16T03:59:15.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.145+0000 7fe75737f640 10 monclient: _reopen_session rank -1 2024-09-16T03:59:15.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.145+0000 7fe75737f640 10 monclient: _add_conns ranks=[1,3,2,0,4] 2024-09-16T03:59:15.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.145+0000 7fe75737f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe7500a1330 addr v1:172.21.15.154:6789/0 2024-09-16T03:59:15.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.145+0000 7fe75737f640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe75006dac0 0x7fe75013a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:59:15.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.145+0000 7fe75737f640 10 monclient(hunting): picked mon.noname-a con 0x7fe75006dac0 addr v2:172.21.15.42:3300/0 2024-09-16T03:59:15.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.145+0000 7fe75737f640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe7500a1e20 0x7fe75013c890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:59:15.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.145+0000 7fe75737f640 10 monclient(hunting): picked mon.noname-b con 0x7fe7500a1e20 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:59:15.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.145+0000 7fe75737f640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe75009c090 con 0x7fe7500a1330 2024-09-16T03:59:15.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.145+0000 7fe75737f640 10 monclient(hunting): start opening mon connection 2024-09-16T03:59:15.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.145+0000 7fe75737f640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7fe75009c890 con 0x7fe75006dac0 2024-09-16T03:59:15.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.145+0000 7fe75737f640 10 monclient(hunting): start opening mon connection 2024-09-16T03:59:15.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.145+0000 7fe75737f640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fe75009f610 con 0x7fe7500a1e20 2024-09-16T03:59:15.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.145+0000 7fe75737f640 10 monclient(hunting): _renew_subs 2024-09-16T03:59:15.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.145+0000 7fe75737f640 10 monclient(hunting): authenticate will time out at 2002.117694s 2024-09-16T03:59:15.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.145+0000 7fe75591c640 1 -- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe7500a1e20 msgr2=0x7fe75013c890 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T03:59:15.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.145+0000 7fe75591c640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe7500a1e20 0x7fe75013c890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:59:15.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.145+0000 7fe75491a640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe75006dac0 0x7fe75013a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:59:15.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.145+0000 7fe75491a640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe75006dac0 0x7fe75013a340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:39638/0 (socket says 172.21.15.42:39638) 2024-09-16T03:59:15.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.145+0000 7fe75491a640 1 -- 172.21.15.42:0/3703171826 learned_addr learned my addr 172.21.15.42:0/3703171826 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:59:15.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.145+0000 7fe75491a640 10 monclient(hunting): get_auth_request con 0x7fe75006dac0 auth_method 0 2024-09-16T03:59:15.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.145+0000 7fe75491a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:59:15.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.145+0000 7fe75491a640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:59:15.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.145+0000 7fe75491a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:59:15.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.145+0000 7fe75491a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:59:15.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.145+0000 7fe75491a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:59:15.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.145+0000 7fe75491a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:59:15.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.145+0000 7fe75491a640 10 monclient(hunting): handle_auth_done global_id 825135 payload 274 2024-09-16T03:59:15.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.145+0000 7fe75491a640 1 -- 172.21.15.42:0/3703171826 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe7500a1e20 msgr2=0x7fe75013c890 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:59:15.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.145+0000 7fe75491a640 1 --2- 172.21.15.42:0/3703171826 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe7500a1e20 0x7fe75013c890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:59:15.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.145+0000 7fe75491a640 1 -- 172.21.15.42:0/3703171826 >> v1:172.21.15.154:6789/0 conn(0x7fe7500a1330 legacy=0x7fe7500a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:59:15.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.145+0000 7fe75491a640 10 monclient: _finish_hunting 0 2024-09-16T03:59:15.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.145+0000 7fe75491a640 1 monclient: found mon.noname-a 2024-09-16T03:59:15.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.145+0000 7fe75491a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:59:15.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.145+0000 7fe75491a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:59:15.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.145+0000 7fe75491a640 1 -- 172.21.15.42:0/3703171826 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe75009f8f0 con 0x7fe75006dac0 2024-09-16T03:59:15.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.149+0000 7fe75491a640 1 --2- 172.21.15.42:0/3703171826 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe75006dac0 0x7fe75013a340 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fe740009870 tx=0x7fe74001c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f50697ed33c3623 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:59:15.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.149+0000 7fe747fff640 1 -- 172.21.15.42:0/3703171826 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7400ab640 con 0x7fe75006dac0 2024-09-16T03:59:15.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.149+0000 7fe747fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:59:15.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.149+0000 7fe747fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:59:15.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.149+0000 7fe747fff640 10 monclient: dump: 2024-09-16T03:59:15.162 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:59:15.162 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:59:15.162 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:15.162 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:15.162 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:59:15.163 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:59:15.163 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:59:15.163 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:59:15.163 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:59:15.163 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:59:15.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.149+0000 7fe747fff640 10 monclient: _finish_auth 0 2024-09-16T03:59:15.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.149+0000 7fe747fff640 10 monclient: _check_auth_tickets 2024-09-16T03:59:15.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.149+0000 7fe747fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:59:15.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.149+0000 7fe747fff640 1 -- 172.21.15.42:0/3703171826 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe7400b0070 con 0x7fe75006dac0 2024-09-16T03:59:15.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.149+0000 7fe75737f640 5 monclient: authenticate success, global_id 825135 2024-09-16T03:59:15.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.149+0000 7fe75737f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:59:15.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.149+0000 7fe747fff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:59:15.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.149+0000 7fe75737f640 10 monclient: get_monmap_and_config success 2024-09-16T03:59:15.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.149+0000 7fe747fff640 1 -- 172.21.15.42:0/3703171826 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7400aba00 con 0x7fe75006dac0 2024-09-16T03:59:15.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.149+0000 7fe747fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:59:15.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.149+0000 7fe75737f640 4 set_mon_vals no callback set 2024-09-16T03:59:15.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.149+0000 7fe747fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:59:15.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.149+0000 7fe747fff640 10 monclient: dump: 2024-09-16T03:59:15.164 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:59:15.164 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:59:15.164 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:15.164 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:15.164 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:59:15.164 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:59:15.164 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:59:15.164 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:59:15.164 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:59:15.164 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:59:15.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.149+0000 7fe75737f640 10 monclient: shutdown 2024-09-16T03:59:15.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.149+0000 7fe75737f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:59:15.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.149+0000 7fe75737f640 1 -- 172.21.15.42:0/3703171826 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe75006dac0 msgr2=0x7fe75013a340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:59:15.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.149+0000 7fe75737f640 1 --2- 172.21.15.42:0/3703171826 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe75006dac0 0x7fe75013a340 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fe740009870 tx=0x7fe74001c460 comp rx=0 tx=0).stop 2024-09-16T03:59:15.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.149+0000 7fe75737f640 1 -- 172.21.15.42:0/3703171826 shutdown_connections 2024-09-16T03:59:15.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.149+0000 7fe75737f640 1 --2- 172.21.15.42:0/3703171826 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe7500a1e20 0x7fe75013c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:59:15.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.149+0000 7fe75737f640 1 --2- 172.21.15.42:0/3703171826 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe75006dac0 0x7fe75013a340 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:59:15.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.149+0000 7fe75737f640 1 -- 172.21.15.42:0/3703171826 >> 172.21.15.42:0/3703171826 conn(0x7fe7500a2420 msgr2=0x7fe7500a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:59:15.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.149+0000 7fe75737f640 1 -- 172.21.15.42:0/3703171826 shutdown_connections 2024-09-16T03:59:15.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.149+0000 7fe75737f640 1 -- 172.21.15.42:0/3703171826 wait complete. 2024-09-16T03:59:15.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.149+0000 7fe75737f640 10 monclient: build_initial_monmap 2024-09-16T03:59:15.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.149+0000 7fe75737f640 10 monclient: monmap: 2024-09-16T03:59:15.165 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:59:15.165 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:59:15.165 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:59:15.165 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:59:15.166 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:59:15.166 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:59:15.166 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:59:15.166 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:59:15.166 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:59:15.166 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:59:15.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.149+0000 7fe75737f640 1 librados: starting msgr at 2024-09-16T03:59:15.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.149+0000 7fe75737f640 1 librados: starting objecter 2024-09-16T03:59:15.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.149+0000 7fe75737f640 1 Processor -- start 2024-09-16T03:59:15.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.149+0000 7fe75737f640 1 -- start start 2024-09-16T03:59:15.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.149+0000 7fe75737f640 1 librados: setting wanted keys 2024-09-16T03:59:15.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.149+0000 7fe75737f640 1 librados: calling monclient init 2024-09-16T03:59:15.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.149+0000 7fe75737f640 10 monclient: init 2024-09-16T03:59:15.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.149+0000 7fe75737f640 10 monclient: _reopen_session rank -1 2024-09-16T03:59:15.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.149+0000 7fe75737f640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T03:59:15.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.149+0000 7fe75737f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe7500a1330 0x7fe750132940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:59:15.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.149+0000 7fe75737f640 10 monclient(hunting): picked mon.noname-b con 0x7fe7500a1330 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:59:15.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.149+0000 7fe75737f640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe7500a1e20 0x7fe75009cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:59:15.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.149+0000 7fe75737f640 10 monclient(hunting): picked mon.noname-c con 0x7fe7500a1e20 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:59:15.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.149+0000 7fe75737f640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe75009d0d0 0x7fe75009d560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:59:15.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.149+0000 7fe75737f640 10 monclient(hunting): picked mon.noname-a con 0x7fe75009d0d0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:59:15.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.149+0000 7fe75737f640 10 monclient(hunting): start opening mon connection 2024-09-16T03:59:15.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.149+0000 7fe75737f640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fe75013f770 con 0x7fe75009d0d0 2024-09-16T03:59:15.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.149+0000 7fe75737f640 10 monclient(hunting): start opening mon connection 2024-09-16T03:59:15.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.149+0000 7fe75737f640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe75013f5f0 con 0x7fe7500a1330 2024-09-16T03:59:15.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.149+0000 7fe75737f640 10 monclient(hunting): start opening mon connection 2024-09-16T03:59:15.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.149+0000 7fe75737f640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fe75013f8f0 con 0x7fe7500a1e20 2024-09-16T03:59:15.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.149+0000 7fe75737f640 10 monclient(hunting): _renew_subs 2024-09-16T03:59:15.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.149+0000 7fe75737f640 10 monclient(hunting): authenticate will time out at 2002.122807s 2024-09-16T03:59:15.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.149+0000 7fe75491a640 1 -- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe7500a1e20 msgr2=0x7fe75009cb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T03:59:15.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.149+0000 7fe75491a640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe7500a1e20 0x7fe75009cb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:59:15.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.149+0000 7fe75591c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe75009d0d0 0x7fe75009d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:59:15.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.149+0000 7fe75591c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe75009d0d0 0x7fe75009d560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:39640/0 (socket says 172.21.15.42:39640) 2024-09-16T03:59:15.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.149+0000 7fe75591c640 1 -- 172.21.15.42:0/3518897564 learned_addr learned my addr 172.21.15.42:0/3518897564 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:59:15.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.149+0000 7fe75591c640 10 monclient(hunting): get_auth_request con 0x7fe75009d0d0 auth_method 0 2024-09-16T03:59:15.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.149+0000 7fe75591c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:59:15.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.149+0000 7fe75591c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:59:15.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.149+0000 7fe75591c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:59:15.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.149+0000 7fe75591c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:59:15.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.149+0000 7fe75591c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:59:15.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.149+0000 7fe75591c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:59:15.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.149+0000 7fe75511b640 1 --2- 172.21.15.42:0/3518897564 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe7500a1330 0x7fe750132940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:59:15.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.149+0000 7fe75511b640 10 monclient(hunting): get_auth_request con 0x7fe7500a1330 auth_method 0 2024-09-16T03:59:15.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.149+0000 7fe75511b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:59:15.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.149+0000 7fe75511b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:59:15.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.149+0000 7fe75511b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:59:15.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.149+0000 7fe75591c640 10 monclient(hunting): handle_auth_done global_id 825138 payload 824 2024-09-16T03:59:15.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.149+0000 7fe75591c640 1 -- 172.21.15.42:0/3518897564 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe7500a1e20 msgr2=0x7fe75009cb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:59:15.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.149+0000 7fe75591c640 1 --2- 172.21.15.42:0/3518897564 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe7500a1e20 0x7fe75009cb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:59:15.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.149+0000 7fe75591c640 1 -- 172.21.15.42:0/3518897564 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe7500a1330 msgr2=0x7fe750132940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:59:15.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.149+0000 7fe75591c640 1 --2- 172.21.15.42:0/3518897564 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe7500a1330 0x7fe750132940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:59:15.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.149+0000 7fe75591c640 10 monclient: _finish_hunting 0 2024-09-16T03:59:15.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.149+0000 7fe75591c640 1 monclient: found mon.noname-a 2024-09-16T03:59:15.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.149+0000 7fe75591c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:59:15.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.149+0000 7fe75591c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:59:15.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.149+0000 7fe75591c640 1 -- 172.21.15.42:0/3518897564 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe75013ac50 con 0x7fe75009d0d0 2024-09-16T03:59:15.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.149+0000 7fe75591c640 1 --2- 172.21.15.42:0/3518897564 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe75009d0d0 0x7fe75009d560 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fe74800be60 tx=0x7fe74800bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:59:15.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.149+0000 7fe745ffb640 10 client.?.objecter ms_handle_connect 0x7fe75009d0d0 2024-09-16T03:59:15.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.149+0000 7fe745ffb640 10 client.?.objecter resend_mon_ops 2024-09-16T03:59:15.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.149+0000 7fe745ffb640 1 -- 172.21.15.42:0/3518897564 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe748016430 con 0x7fe75009d0d0 2024-09-16T03:59:15.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.149+0000 7fe745ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:59:15.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.149+0000 7fe745ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:59:15.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.149+0000 7fe745ffb640 10 monclient: dump: 2024-09-16T03:59:15.170 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:59:15.170 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:59:15.170 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:15.170 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:15.171 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:59:15.171 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:59:15.171 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:59:15.171 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:59:15.171 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:59:15.171 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:59:15.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.149+0000 7fe745ffb640 10 monclient: _finish_auth 0 2024-09-16T03:59:15.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.149+0000 7fe745ffb640 10 monclient: _check_auth_tickets 2024-09-16T03:59:15.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.149+0000 7fe745ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:59:15.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.149+0000 7fe745ffb640 1 -- 172.21.15.42:0/3518897564 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe748010030 con 0x7fe75009d0d0 2024-09-16T03:59:15.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.149+0000 7fe745ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T03:59:15.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.149+0000 7fe745ffb640 1 -- 172.21.15.42:0/3518897564 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe748016810 con 0x7fe75009d0d0 2024-09-16T03:59:15.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.149+0000 7fe75737f640 5 monclient: authenticate success, global_id 825138 2024-09-16T03:59:15.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.149+0000 7fe75737f640 10 monclient: _renew_subs 2024-09-16T03:59:15.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.149+0000 7fe75737f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:59:15.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.149+0000 7fe75737f640 1 -- 172.21.15.42:0/3518897564 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7501393c0 con 0x7fe75009d0d0 2024-09-16T03:59:15.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.149+0000 7fe7477fe640 4 set_mon_vals no callback set 2024-09-16T03:59:15.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.149+0000 7fe745ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:59:15.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.149+0000 7fe745ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:59:15.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.149+0000 7fe745ffb640 10 monclient: dump: 2024-09-16T03:59:15.172 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:59:15.172 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:59:15.172 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:15.172 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:15.172 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:59:15.172 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:59:15.172 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:59:15.172 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:59:15.173 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:59:15.173 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:59:15.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.149+0000 7fe75737f640 10 client.825138.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:59:15.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.149+0000 7fe75737f640 10 monclient: _renew_subs 2024-09-16T03:59:15.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.149+0000 7fe75737f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:59:15.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.149+0000 7fe75737f640 1 -- 172.21.15.42:0/3518897564 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe75006e2d0 con 0x7fe75009d0d0 2024-09-16T03:59:15.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.149+0000 7fe75737f640 1 librados: init done 2024-09-16T03:59:15.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.153+0000 7fe75737f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:59:15.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.153+0000 7fe75737f640 1 --2- 172.21.15.42:0/3518897564 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe75006dac0 0x7fe750136320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:59:15.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.153+0000 7fe75737f640 10 monclient(hunting): start opening mon connection 2024-09-16T03:59:15.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.153+0000 7fe75737f640 1 -- 172.21.15.42:0/3518897564 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe750139a80 con 0x7fe75006dac0 2024-09-16T03:59:15.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.153+0000 7fe745ffb640 1 -- 172.21.15.42:0/3518897564 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fe7480153e0 con 0x7fe75009d0d0 2024-09-16T03:59:15.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.153+0000 7fe745ffb640 1 --2- 172.21.15.42:0/3518897564 >> v2:172.21.15.154:6812/254845342 conn(0x7fe7340377c0 0x7fe734039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:59:15.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.153+0000 7fe75511b640 1 --2- 172.21.15.42:0/3518897564 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe75006dac0 0x7fe750136320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:59:15.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.153+0000 7fe745ffb640 1 -- 172.21.15.42:0/3518897564 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(516..516 src has 451..516) ==== 5258+0+0 (secure 0 0 0) 0x7fe748049280 con 0x7fe75009d0d0 2024-09-16T03:59:15.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.153+0000 7fe75511b640 10 monclient: get_auth_request con 0x7fe75006dac0 auth_method 0 2024-09-16T03:59:15.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.153+0000 7fe745ffb640 10 client.825138.objecter ms_dispatch 0x7fe750000ba0 osd_map(516..516 src has 451..516) 2024-09-16T03:59:15.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.153+0000 7fe75511b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:59:15.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.153+0000 7fe745ffb640 3 client.825138.objecter handle_osd_map got epochs [516,516] > 0 2024-09-16T03:59:15.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.153+0000 7fe75511b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:59:15.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.153+0000 7fe75511b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:59:15.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.153+0000 7fe745ffb640 3 client.825138.objecter handle_osd_map decoding full epoch 516 2024-09-16T03:59:15.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.153+0000 7fe745ffb640 20 client.825138.objecter dump_active .. 0 homeless 2024-09-16T03:59:15.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.153+0000 7fe75511b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:59:15.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.153+0000 7fe75511b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:59:15.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.153+0000 7fe75511b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:59:15.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.153+0000 7fe75491a640 1 --2- 172.21.15.42:0/3518897564 >> v2:172.21.15.154:6812/254845342 conn(0x7fe7340377c0 0x7fe734039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:59:15.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.153+0000 7fe75491a640 10 monclient: get_auth_request con 0x7fe7340377c0 auth_method 0 2024-09-16T03:59:15.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.153+0000 7fe75511b640 10 monclient(hunting): handle_auth_done global_id 904318 payload 824 2024-09-16T03:59:15.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.153+0000 7fe75511b640 1 -- 172.21.15.42:0/3518897564 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe7500a1710 con 0x7fe75006dac0 2024-09-16T03:59:15.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.153+0000 7fe745ffb640 10 client.825138.objecter ms_handle_connect 0x7fe75006dac0 2024-09-16T03:59:15.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.153+0000 7fe745ffb640 10 client.825138.objecter resend_mon_ops 2024-09-16T03:59:15.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.153+0000 7fe75511b640 1 --2- 172.21.15.42:0/3518897564 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe75006dac0 0x7fe750136320 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fe74c00e450 tx=0x7fe74c00e910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:59:15.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.153+0000 7fe745ffb640 10 client.825138.objecter ms_handle_connect 0x7fe7340377c0 2024-09-16T03:59:15.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.153+0000 7fe75491a640 1 --2- 172.21.15.42:0/3518897564 >> v2:172.21.15.154:6812/254845342 conn(0x7fe7340377c0 0x7fe734039c80 secure :-1 s=READY pgs=3726 cs=0 l=1 rev1=1 crypto rx=0x7fe7400af000 tx=0x7fe740008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:59:15.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.153+0000 7fe745ffb640 1 -- 172.21.15.42:0/3518897564 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe74c007240 con 0x7fe75006dac0 2024-09-16T03:59:15.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.153+0000 7fe745ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:59:15.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.153+0000 7fe745ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:59:15.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.153+0000 7fe745ffb640 10 monclient: dump: 2024-09-16T03:59:15.175 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:59:15.175 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:59:15.176 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:15.176 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:15.176 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:59:15.176 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:59:15.176 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:59:15.176 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:59:15.176 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:59:15.176 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:59:15.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.153+0000 7fe745ffb640 1 -- 172.21.15.42:0/3518897564 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fe74c007240 con 0x7fe75006dac0 2024-09-16T03:59:15.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.153+0000 7fe745ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:59:15.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.153+0000 7fe745ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:59:15.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.153+0000 7fe745ffb640 1 -- 172.21.15.42:0/3518897564 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe75006dac0 msgr2=0x7fe750136320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:59:15.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.153+0000 7fe745ffb640 1 --2- 172.21.15.42:0/3518897564 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe75006dac0 0x7fe750136320 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fe74c00e450 tx=0x7fe74c00e910 comp rx=0 tx=0).stop 2024-09-16T03:59:15.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.153+0000 7fe745ffb640 1 -- 172.21.15.42:0/3518897564 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe75006dac0 msgr2=0x7fe750136320 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:59:15.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.153+0000 7fe745ffb640 1 --2- 172.21.15.42:0/3518897564 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe75006dac0 0x7fe750136320 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:59:15.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.161+0000 7fe75737f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:59:15.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.161+0000 7fe75737f640 1 --2- 172.21.15.42:0/3518897564 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe750136860 0x7fe750136c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:59:15.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.161+0000 7fe75737f640 10 monclient(hunting): start opening mon connection 2024-09-16T03:59:15.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.161+0000 7fe75737f640 1 -- 172.21.15.42:0/3518897564 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe750132e80 con 0x7fe750136860 2024-09-16T03:59:15.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.161+0000 7fe75511b640 1 --2- 172.21.15.42:0/3518897564 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe750136860 0x7fe750136c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:59:15.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.161+0000 7fe75511b640 10 monclient: get_auth_request con 0x7fe750136860 auth_method 0 2024-09-16T03:59:15.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.161+0000 7fe75511b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:59:15.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.161+0000 7fe75511b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:59:15.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.161+0000 7fe75511b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:59:15.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.161+0000 7fe75511b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:59:15.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.161+0000 7fe75511b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:59:15.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.161+0000 7fe75511b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:59:15.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.161+0000 7fe75511b640 10 monclient(hunting): handle_auth_done global_id 904321 payload 824 2024-09-16T03:59:15.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.161+0000 7fe75511b640 1 -- 172.21.15.42:0/3518897564 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fe750069040 con 0x7fe750136860 2024-09-16T03:59:15.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.161+0000 7fe75511b640 1 --2- 172.21.15.42:0/3518897564 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe750136860 0x7fe750136c40 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fe74c00efd0 tx=0x7fe74c003770 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:59:15.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.161+0000 7fe745ffb640 10 client.825138.objecter ms_handle_connect 0x7fe750136860 2024-09-16T03:59:15.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.161+0000 7fe745ffb640 10 client.825138.objecter resend_mon_ops 2024-09-16T03:59:15.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.161+0000 7fe745ffb640 1 -- 172.21.15.42:0/3518897564 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe74c01ea40 con 0x7fe750136860 2024-09-16T03:59:15.180 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.161+0000 7fe745ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:59:15.180 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.161+0000 7fe745ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:59:15.180 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.161+0000 7fe745ffb640 10 monclient: dump: 2024-09-16T03:59:15.180 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:59:15.180 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:59:15.180 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:15.180 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:15.180 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:59:15.180 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:59:15.180 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:59:15.180 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:59:15.180 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:59:15.181 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:59:15.181 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.161+0000 7fe745ffb640 1 -- 172.21.15.42:0/3518897564 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4701 (secure 0 0 0) 0x7fe74c029460 con 0x7fe750136860 2024-09-16T03:59:15.181 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.161+0000 7fe745ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:59:15.181 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.161+0000 7fe745ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:59:15.181 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.161+0000 7fe745ffb640 1 -- 172.21.15.42:0/3518897564 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe750136860 msgr2=0x7fe750136c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:59:15.181 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.161+0000 7fe745ffb640 1 --2- 172.21.15.42:0/3518897564 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe750136860 0x7fe750136c40 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fe74c00efd0 tx=0x7fe74c003770 comp rx=0 tx=0).stop 2024-09-16T03:59:15.181 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.161+0000 7fe745ffb640 1 -- 172.21.15.42:0/3518897564 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe750136860 msgr2=0x7fe750136c40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:59:15.181 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.161+0000 7fe745ffb640 1 --2- 172.21.15.42:0/3518897564 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe750136860 0x7fe750136c40 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:59:15.182 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:59:15.182 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:59:15.182 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:59:15.182 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T03:59:15.182 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 158, 2024-09-16T03:59:15.182 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 83342, 2024-09-16T03:59:15.182 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:59:15.182 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:59:15.182 INFO:teuthology.orchestra.run.smithi042.stdout: 1 2024-09-16T03:59:15.182 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:59:15.182 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 0, 2024-09-16T03:59:15.183 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:59:15.183 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:59:15.183 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:59:15.183 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:59:15.183 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:59:15.183 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:59:15.183 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:59:15.183 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:59:15.183 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:59:15.183 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:59:15.183 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:59:15.183 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:59:15.183 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:59:15.183 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:59:15.184 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:59:15.184 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:59:15.184 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:59:15.184 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:59:15.184 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:59:15.184 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:59:15.184 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:59:15.184 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:59:15.184 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:59:15.184 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:59:15.184 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:59:15.184 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:59:15.184 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:59:15.185 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:59:15.185 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:59:15.185 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:59:15.185 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:59:15.185 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:59:15.185 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:59:15.185 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:59:15.185 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:59:15.185 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:59:15.185 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:59:15.185 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:59:15.185 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:59:15.185 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:59:15.186 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:59:15.186 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:59:15.186 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:59:15.186 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:59:15.186 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:59:15.186 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:59:15.186 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:59:15.186 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:59:15.186 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:59:15.186 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:59:15.186 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:59:15.186 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:59:15.186 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:59:15.186 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:59:15.187 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:59:15.187 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:59:15.187 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:59:15.187 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:59:15.187 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:59:15.187 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:59:15.187 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:59:15.187 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:59:15.187 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:59:15.187 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:59:15.187 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:59:15.187 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:59:15.187 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:59:15.187 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:59:15.188 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:59:15.188 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:59:15.188 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:59:15.188 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:59:15.188 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:59:15.188 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:59:15.188 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:59:15.188 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:59:15.188 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:59:15.188 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:59:15.188 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:59:15.188 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:59:15.188 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:59:15.188 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:59:15.189 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:59:15.189 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:59:15.189 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:59:15.189 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:59:15.189 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:59:15.189 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:59:15.189 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:59:15.189 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:59:15.189 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:59:15.189 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:59:15.189 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:59:15.189 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:59:15.189 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:59:15.189 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:59:15.190 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:59:15.190 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:59:15.190 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:59:15.190 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:59:15.190 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:59:15.190 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:59:15.190 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:59:15.190 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:59:15.190 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:59:15.190 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:59:15.190 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:59:15.190 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:59:15.190 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:59:15.190 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:59:15.191 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:59:15.191 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:59:15.191 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:59:15.191 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:59:15.191 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:59:15.191 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:59:15.191 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:59:15.191 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:59:15.191 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:59:15.191 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:59:15.191 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:59:15.191 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:59:15.191 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:59:15.191 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:59:15.192 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:59:15.192 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:59:15.192 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:59:15.192 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:59:15.192 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:59:15.192 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:59:15.192 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:59:15.192 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:59:15.192 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:59:15.192 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:59:15.192 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:59:15.192 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:59:15.192 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:59:15.192 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:59:15.193 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:59:15.193 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:59:15.193 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:59:15.193 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T03:59:15.193 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:59:15.193 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:59:15.193 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:59:15.193 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:59:15.193 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:59:15.193 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:59:15.193 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:59:15.193 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:59:15.193 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:59:15.193 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:59:15.194 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 2 2024-09-16T03:59:15.194 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:59:15.194 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:59:15.194 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:59:15.194 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:59:15.194 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:59:15.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.161+0000 7fe75737f640 10 librados: watch_flush enter 2024-09-16T03:59:15.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.161+0000 7fe75737f640 10 librados: watch_flush exit 2024-09-16T03:59:15.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.161+0000 7fe75737f640 20 client.825138.objecter shutdown clearing up homeless session... 2024-09-16T03:59:15.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.161+0000 7fe75737f640 10 client.825138.objecter successfully canceled tick 2024-09-16T03:59:15.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.161+0000 7fe75737f640 1 -- 172.21.15.42:0/3518897564 >> v2:172.21.15.154:6812/254845342 conn(0x7fe7340377c0 msgr2=0x7fe734039c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:59:15.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.161+0000 7fe75737f640 1 --2- 172.21.15.42:0/3518897564 >> v2:172.21.15.154:6812/254845342 conn(0x7fe7340377c0 0x7fe734039c80 secure :-1 s=READY pgs=3726 cs=0 l=1 rev1=1 crypto rx=0x7fe7400af000 tx=0x7fe740008040 comp rx=0 tx=0).stop 2024-09-16T03:59:15.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.161+0000 7fe75737f640 10 monclient: shutdown 2024-09-16T03:59:15.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.161+0000 7fe75737f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:59:15.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.161+0000 7fe75737f640 1 -- 172.21.15.42:0/3518897564 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe75009d0d0 msgr2=0x7fe75009d560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:59:15.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.161+0000 7fe75737f640 1 --2- 172.21.15.42:0/3518897564 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe75009d0d0 0x7fe75009d560 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fe74800be60 tx=0x7fe74800bf50 comp rx=0 tx=0).stop 2024-09-16T03:59:15.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.161+0000 7fe75591c640 1 -- 172.21.15.42:0/3518897564 reap_dead start 2024-09-16T03:59:15.197 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.161+0000 7fe75737f640 1 -- 172.21.15.42:0/3518897564 shutdown_connections 2024-09-16T03:59:15.197 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.161+0000 7fe75737f640 1 --2- 172.21.15.42:0/3518897564 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe75009d0d0 0x7fe75009d560 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:59:15.197 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.161+0000 7fe75737f640 1 -- 172.21.15.42:0/3518897564 >> 172.21.15.42:0/3518897564 conn(0x7fe7500a2420 msgr2=0x7fe750135300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:59:15.197 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.161+0000 7fe75737f640 1 -- 172.21.15.42:0/3518897564 shutdown_connections 2024-09-16T03:59:15.197 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.161+0000 7fe75737f640 1 -- 172.21.15.42:0/3518897564 wait complete. 2024-09-16T03:59:15.197 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:15.161+0000 7fe75737f640 1 librados: shutdown 2024-09-16T03:59:15.197 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T03:59:15.197 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-16T03:59:35.198 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-16T03:59:35.198 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-09-16T03:59:35.198 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-16T03:59:35.199 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-16T03:59:35.201 INFO:tasks.ceph.mon.c:Started 2024-09-16T03:59:35.202 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T03:59:35.202 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:59:35.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.329+0000 7fd38ade1640 10 monclient: get_monmap_and_config 2024-09-16T03:59:35.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.329+0000 7fd38ade1640 10 monclient: build_initial_monmap 2024-09-16T03:59:35.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.329+0000 7fd38ade1640 10 monclient: monmap: 2024-09-16T03:59:35.335 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:59:35.335 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:59:35.335 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:59:35.334034+0000 2024-09-16T03:59:35.336 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:59:35.334034+0000 2024-09-16T03:59:35.336 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:59:35.336 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:59:35.336 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:59:35.336 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:59:35.336 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:59:35.336 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:59:35.336 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:59:35.336 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:59:35.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.329+0000 7fd38ade1640 1 Processor -- start 2024-09-16T03:59:35.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.329+0000 7fd38ade1640 1 -- start start 2024-09-16T03:59:35.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.329+0000 7fd38ade1640 10 monclient: init 2024-09-16T03:59:35.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.329+0000 7fd38ade1640 10 monclient: _reopen_session rank -1 2024-09-16T03:59:35.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.329+0000 7fd38ade1640 10 monclient: _add_conns ranks=[4,1,2,3,0] 2024-09-16T03:59:35.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.329+0000 7fd38ade1640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fd3840a39b0 0x7fd384133b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:59:35.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.329+0000 7fd38ade1640 10 monclient(hunting): picked mon.noname-c con 0x7fd3840a39b0 addr v2:172.21.15.154:3300/0 2024-09-16T03:59:35.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.329+0000 7fd38ade1640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd3840a3d90 addr v1:172.21.15.154:6789/0 2024-09-16T03:59:35.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.329+0000 7fd38ade1640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd384137710 0x7fd38413f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:59:35.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.329+0000 7fd38ade1640 10 monclient(hunting): picked mon.noname-b con 0x7fd384137710 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:59:35.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.329+0000 7fd38ade1640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd38409c140 con 0x7fd3840a3d90 2024-09-16T03:59:35.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.329+0000 7fd38ade1640 10 monclient(hunting): start opening mon connection 2024-09-16T03:59:35.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.329+0000 7fd38ade1640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fd38409c420 con 0x7fd3840a39b0 2024-09-16T03:59:35.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.329+0000 7fd38ade1640 10 monclient(hunting): start opening mon connection 2024-09-16T03:59:35.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.329+0000 7fd38ade1640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fd38406dd60 con 0x7fd384137710 2024-09-16T03:59:35.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.329+0000 7fd38ade1640 10 monclient(hunting): _renew_subs 2024-09-16T03:59:35.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.329+0000 7fd38ade1640 10 monclient(hunting): authenticate will time out at 2022.303193s 2024-09-16T03:59:35.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.329+0000 7fd38937e640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd384137710 0x7fd38413f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:59:35.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.329+0000 7fd38937e640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd384137710 0x7fd38413f3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:40472/0 (socket says 172.21.15.42:40472) 2024-09-16T03:59:35.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.329+0000 7fd38937e640 1 -- 172.21.15.42:0/1341247531 learned_addr learned my addr 172.21.15.42:0/1341247531 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:59:35.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.329+0000 7fd38937e640 10 monclient(hunting): get_auth_request con 0x7fd384137710 auth_method 0 2024-09-16T03:59:35.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.329+0000 7fd38937e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:59:35.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.329+0000 7fd38937e640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:59:35.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.329+0000 7fd38937e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:59:35.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.329+0000 7fd38937e640 1 -- 172.21.15.42:0/1341247531 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd384137710 msgr2=0x7fd38413f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T03:59:35.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.329+0000 7fd38937e640 1 -- 172.21.15.42:0/1341247531 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd384137710 msgr2=0x7fd38413f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:59:35.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.329+0000 7fd38937e640 1 --2- 172.21.15.42:0/1341247531 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd384137710 0x7fd38413f3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:59:35.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.329+0000 7fd388b7d640 1 --2- 172.21.15.42:0/1341247531 >> v2:172.21.15.154:3300/0 conn(0x7fd3840a39b0 0x7fd384133b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:59:35.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.329+0000 7fd38937e640 1 --2- 172.21.15.42:0/1341247531 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd384137710 0x7fd38413f3a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:59:35.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.329+0000 7fd388b7d640 10 monclient(hunting): get_auth_request con 0x7fd3840a39b0 auth_method 0 2024-09-16T03:59:35.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.329+0000 7fd388b7d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:59:35.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.329+0000 7fd388b7d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:59:35.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.329+0000 7fd388b7d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:59:35.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.329+0000 7fd388b7d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:59:35.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.329+0000 7fd388b7d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:59:35.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.329+0000 7fd388b7d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:59:35.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.333+0000 7fd388b7d640 10 monclient(hunting): handle_auth_done global_id 904324 payload 274 2024-09-16T03:59:35.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.333+0000 7fd388b7d640 1 -- 172.21.15.42:0/1341247531 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd384137710 msgr2=0x7fd38413f3a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:59:35.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.333+0000 7fd388b7d640 1 --2- 172.21.15.42:0/1341247531 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd384137710 0x7fd38413f3a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:59:35.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.333+0000 7fd388b7d640 1 -- 172.21.15.42:0/1341247531 >> v1:172.21.15.154:6789/0 conn(0x7fd3840a3d90 legacy=0x7fd384137000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:59:35.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.333+0000 7fd388b7d640 10 monclient: _finish_hunting 0 2024-09-16T03:59:35.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.333+0000 7fd388b7d640 1 monclient: found mon.noname-c 2024-09-16T03:59:35.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.333+0000 7fd388b7d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:59:35.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.333+0000 7fd388b7d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.154:3300/0 2024-09-16T03:59:35.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.333+0000 7fd388b7d640 1 -- 172.21.15.42:0/1341247531 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3840a5e20 con 0x7fd3840a39b0 2024-09-16T03:59:35.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.333+0000 7fd388b7d640 1 --2- 172.21.15.42:0/1341247531 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd3840a39b0 0x7fd384133b80 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fd3740097c0 tx=0x7fd37401c360 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7c6deb7fb74a5ae1 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:59:35.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.333+0000 7fd3837fe640 1 -- 172.21.15.42:0/1341247531 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3740b0070 con 0x7fd3840a39b0 2024-09-16T03:59:35.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.333+0000 7fd3837fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:59:35.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.333+0000 7fd3837fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:59:35.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.333+0000 7fd3837fe640 10 monclient: dump: 2024-09-16T03:59:35.347 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:59:35.347 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:59:35.347 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:35.348 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:35.348 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:59:35.348 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:59:35.348 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:59:35.348 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:59:35.348 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:59:35.348 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:59:35.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.333+0000 7fd3837fe640 10 monclient: _finish_auth 0 2024-09-16T03:59:35.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.333+0000 7fd3837fe640 10 monclient: _check_auth_tickets 2024-09-16T03:59:35.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.333+0000 7fd3837fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:59:35.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.333+0000 7fd3837fe640 1 -- 172.21.15.42:0/1341247531 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd3740ab600 con 0x7fd3840a39b0 2024-09-16T03:59:35.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.333+0000 7fd3837fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:59:35.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.333+0000 7fd3837fe640 1 -- 172.21.15.42:0/1341247531 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3740ab9d0 con 0x7fd3840a39b0 2024-09-16T03:59:35.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.333+0000 7fd38ade1640 5 monclient: authenticate success, global_id 904324 2024-09-16T03:59:35.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.333+0000 7fd38ade1640 10 monclient: get_monmap_and_config success 2024-09-16T03:59:35.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.333+0000 7fd38ade1640 4 set_mon_vals no callback set 2024-09-16T03:59:35.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.333+0000 7fd3837fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:59:35.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.333+0000 7fd3837fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:59:35.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.333+0000 7fd3837fe640 10 monclient: dump: 2024-09-16T03:59:35.349 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:59:35.349 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:59:35.349 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:35.349 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:35.349 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:59:35.349 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:59:35.349 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:59:35.349 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:59:35.349 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:59:35.350 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:59:35.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.333+0000 7fd38ade1640 10 monclient: shutdown 2024-09-16T03:59:35.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.333+0000 7fd38ade1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:59:35.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.333+0000 7fd38ade1640 1 -- 172.21.15.42:0/1341247531 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd3840a39b0 msgr2=0x7fd384133b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:59:35.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.333+0000 7fd38ade1640 1 --2- 172.21.15.42:0/1341247531 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd3840a39b0 0x7fd384133b80 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fd3740097c0 tx=0x7fd37401c360 comp rx=0 tx=0).stop 2024-09-16T03:59:35.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.333+0000 7fd38ade1640 1 -- 172.21.15.42:0/1341247531 shutdown_connections 2024-09-16T03:59:35.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.333+0000 7fd38ade1640 1 --2- 172.21.15.42:0/1341247531 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd384137710 0x7fd38413f3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:59:35.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.333+0000 7fd38ade1640 1 --2- 172.21.15.42:0/1341247531 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd3840a39b0 0x7fd384133b80 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:59:35.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.333+0000 7fd38ade1640 1 -- 172.21.15.42:0/1341247531 >> 172.21.15.42:0/1341247531 conn(0x7fd3840a9fb0 msgr2=0x7fd3840aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:59:35.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.333+0000 7fd38ade1640 1 -- 172.21.15.42:0/1341247531 shutdown_connections 2024-09-16T03:59:35.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.333+0000 7fd38ade1640 1 -- 172.21.15.42:0/1341247531 wait complete. 2024-09-16T03:59:35.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.333+0000 7fd38ade1640 10 monclient: build_initial_monmap 2024-09-16T03:59:35.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.333+0000 7fd38ade1640 10 monclient: monmap: 2024-09-16T03:59:35.351 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:59:35.351 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:59:35.351 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:59:35.351 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:59:35.351 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:59:35.351 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:59:35.351 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:59:35.351 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:59:35.351 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:59:35.351 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:59:35.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.337+0000 7fd38ade1640 1 librados: starting msgr at 2024-09-16T03:59:35.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.337+0000 7fd38ade1640 1 librados: starting objecter 2024-09-16T03:59:35.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.337+0000 7fd38ade1640 1 Processor -- start 2024-09-16T03:59:35.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.337+0000 7fd38ade1640 1 -- start start 2024-09-16T03:59:35.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.337+0000 7fd38ade1640 1 librados: setting wanted keys 2024-09-16T03:59:35.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.337+0000 7fd38ade1640 1 librados: calling monclient init 2024-09-16T03:59:35.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.337+0000 7fd38ade1640 10 monclient: init 2024-09-16T03:59:35.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.337+0000 7fd38ade1640 10 monclient: _reopen_session rank -1 2024-09-16T03:59:35.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.337+0000 7fd38ade1640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T03:59:35.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.337+0000 7fd38ade1640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd3840a39b0 0x7fd38412d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:59:35.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.337+0000 7fd38ade1640 10 monclient(hunting): picked mon.noname-c con 0x7fd3840a39b0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:59:35.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.337+0000 7fd38ade1640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd3840a3d90 0x7fd38412dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:59:35.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.337+0000 7fd38ade1640 10 monclient(hunting): picked mon.noname-a con 0x7fd3840a3d90 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:59:35.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.337+0000 7fd38ade1640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd384137710 0x7fd3841cf620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:59:35.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.337+0000 7fd38ade1640 10 monclient(hunting): picked mon.noname-b con 0x7fd384137710 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:59:35.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.337+0000 7fd38ade1640 10 monclient(hunting): start opening mon connection 2024-09-16T03:59:35.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.337+0000 7fd38ade1640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fd384142310 con 0x7fd3840a3d90 2024-09-16T03:59:35.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.337+0000 7fd38ade1640 10 monclient(hunting): start opening mon connection 2024-09-16T03:59:35.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.337+0000 7fd38ade1640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd384142490 con 0x7fd384137710 2024-09-16T03:59:35.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.337+0000 7fd38ade1640 10 monclient(hunting): start opening mon connection 2024-09-16T03:59:35.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.337+0000 7fd38ade1640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fd384142610 con 0x7fd3840a39b0 2024-09-16T03:59:35.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.337+0000 7fd38ade1640 10 monclient(hunting): _renew_subs 2024-09-16T03:59:35.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.337+0000 7fd38ade1640 10 monclient(hunting): authenticate will time out at 2022.308841s 2024-09-16T03:59:35.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.337+0000 7fd388b7d640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd3840a39b0 0x7fd38412d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:59:35.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.337+0000 7fd383fff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd3840a3d90 0x7fd38412dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:59:35.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.337+0000 7fd388b7d640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd3840a39b0 0x7fd38412d6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:40486/0 (socket says 172.21.15.42:40486) 2024-09-16T03:59:35.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.337+0000 7fd388b7d640 1 -- 172.21.15.42:0/3157805954 learned_addr learned my addr 172.21.15.42:0/3157805954 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:59:35.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.337+0000 7fd388b7d640 10 monclient(hunting): get_auth_request con 0x7fd3840a39b0 auth_method 0 2024-09-16T03:59:35.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.337+0000 7fd388b7d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:59:35.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.337+0000 7fd388b7d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:59:35.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.337+0000 7fd388b7d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:59:35.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.337+0000 7fd383fff640 10 monclient(hunting): get_auth_request con 0x7fd3840a3d90 auth_method 0 2024-09-16T03:59:35.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.337+0000 7fd383fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:59:35.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.337+0000 7fd383fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:59:35.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.337+0000 7fd383fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:59:35.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.337+0000 7fd388b7d640 1 -- 172.21.15.42:0/3157805954 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd3840a39b0 msgr2=0x7fd38412d6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-16T03:59:35.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.337+0000 7fd388b7d640 1 -- 172.21.15.42:0/3157805954 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd3840a39b0 msgr2=0x7fd38412d6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T03:59:35.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.337+0000 7fd383fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:59:35.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.337+0000 7fd388b7d640 1 --2- 172.21.15.42:0/3157805954 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd3840a39b0 0x7fd38412d6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:59:35.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.337+0000 7fd383fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:59:35.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.337+0000 7fd383fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:59:35.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.337+0000 7fd388b7d640 1 --2- 172.21.15.42:0/3157805954 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd3840a39b0 0x7fd38412d6b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:59:35.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.337+0000 7fd38937e640 1 --2- 172.21.15.42:0/3157805954 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd384137710 0x7fd3841cf620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:59:35.355 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.337+0000 7fd38937e640 10 monclient(hunting): get_auth_request con 0x7fd384137710 auth_method 0 2024-09-16T03:59:35.355 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.337+0000 7fd38937e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:59:35.355 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.337+0000 7fd38937e640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:59:35.355 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.337+0000 7fd38937e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:59:35.355 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.337+0000 7fd383fff640 10 monclient(hunting): handle_auth_done global_id 825141 payload 824 2024-09-16T03:59:35.355 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.337+0000 7fd383fff640 1 -- 172.21.15.42:0/3157805954 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd3840a39b0 msgr2=0x7fd38412d6b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:59:35.355 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.337+0000 7fd383fff640 1 --2- 172.21.15.42:0/3157805954 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd3840a39b0 0x7fd38412d6b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:59:35.355 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.337+0000 7fd383fff640 1 -- 172.21.15.42:0/3157805954 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd384137710 msgr2=0x7fd3841cf620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:59:35.355 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.337+0000 7fd383fff640 1 --2- 172.21.15.42:0/3157805954 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd384137710 0x7fd3841cf620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:59:35.355 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.337+0000 7fd383fff640 10 monclient: _finish_hunting 0 2024-09-16T03:59:35.355 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.337+0000 7fd383fff640 1 monclient: found mon.noname-a 2024-09-16T03:59:35.355 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.337+0000 7fd383fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:59:35.355 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.337+0000 7fd383fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:59:35.355 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.337+0000 7fd383fff640 1 -- 172.21.15.42:0/3157805954 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3840a09e0 con 0x7fd3840a3d90 2024-09-16T03:59:35.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.337+0000 7fd383fff640 1 --2- 172.21.15.42:0/3157805954 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd3840a3d90 0x7fd38412dbf0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fd37800c3c0 tx=0x7fd37800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:59:35.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.337+0000 7fd3817fa640 10 client.?.objecter ms_handle_connect 0x7fd3840a3d90 2024-09-16T03:59:35.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.337+0000 7fd3817fa640 10 client.?.objecter resend_mon_ops 2024-09-16T03:59:35.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.337+0000 7fd3817fa640 1 -- 172.21.15.42:0/3157805954 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3780052d0 con 0x7fd3840a3d90 2024-09-16T03:59:35.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.337+0000 7fd3817fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:59:35.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.337+0000 7fd3817fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:59:35.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.337+0000 7fd3817fa640 10 monclient: dump: 2024-09-16T03:59:35.356 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:59:35.356 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:59:35.356 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:35.356 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:35.356 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:59:35.356 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:59:35.356 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:59:35.357 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:59:35.357 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:59:35.357 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:59:35.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.337+0000 7fd3817fa640 10 monclient: _finish_auth 0 2024-09-16T03:59:35.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.337+0000 7fd3817fa640 10 monclient: _check_auth_tickets 2024-09-16T03:59:35.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.337+0000 7fd3817fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:59:35.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.337+0000 7fd38ade1640 5 monclient: authenticate success, global_id 825141 2024-09-16T03:59:35.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.337+0000 7fd38ade1640 10 monclient: _renew_subs 2024-09-16T03:59:35.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.337+0000 7fd3817fa640 1 -- 172.21.15.42:0/3157805954 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd378005470 con 0x7fd3840a3d90 2024-09-16T03:59:35.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.337+0000 7fd38ade1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:59:35.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.337+0000 7fd38ade1640 1 -- 172.21.15.42:0/3157805954 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3840a22a0 con 0x7fd3840a3d90 2024-09-16T03:59:35.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.337+0000 7fd3817fa640 10 monclient: handle_config config(0 keys) 2024-09-16T03:59:35.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.337+0000 7fd3817fa640 1 -- 172.21.15.42:0/3157805954 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd378007260 con 0x7fd3840a3d90 2024-09-16T03:59:35.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.337+0000 7fd3817fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:59:35.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.337+0000 7fd3817fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:59:35.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.337+0000 7fd3817fa640 10 monclient: dump: 2024-09-16T03:59:35.358 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:59:35.358 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:59:35.358 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:35.358 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:35.358 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:59:35.358 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:59:35.358 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:59:35.358 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:59:35.358 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:59:35.358 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:59:35.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.337+0000 7fd382ffd640 4 set_mon_vals no callback set 2024-09-16T03:59:35.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.337+0000 7fd38ade1640 10 client.825141.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:59:35.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.337+0000 7fd38ade1640 10 monclient: _renew_subs 2024-09-16T03:59:35.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.337+0000 7fd38ade1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:59:35.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.337+0000 7fd38ade1640 1 -- 172.21.15.42:0/3157805954 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3840a2580 con 0x7fd3840a3d90 2024-09-16T03:59:35.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.337+0000 7fd38ade1640 1 librados: init done 2024-09-16T03:59:35.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.337+0000 7fd3817fa640 1 -- 172.21.15.42:0/3157805954 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fd378007400 con 0x7fd3840a3d90 2024-09-16T03:59:35.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.337+0000 7fd38ade1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:59:35.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.337+0000 7fd38ade1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:59:35.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.337+0000 7fd38ade1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:59:35.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.337+0000 7fd38ade1640 1 -- 172.21.15.42:0/3157805954 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd38413f2c0 con 0x7fd3840a3d90 2024-09-16T03:59:35.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.337+0000 7fd3817fa640 1 --2- 172.21.15.42:0/3157805954 >> v2:172.21.15.154:6812/254845342 conn(0x7fd370037810 0x7fd370039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:59:35.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.337+0000 7fd3817fa640 1 -- 172.21.15.42:0/3157805954 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(536..536 src has 451..536) ==== 6173+0+0 (secure 0 0 0) 0x7fd378048590 con 0x7fd3840a3d90 2024-09-16T03:59:35.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.337+0000 7fd3817fa640 10 client.825141.objecter ms_dispatch 0x7fd384000ba0 osd_map(536..536 src has 451..536) 2024-09-16T03:59:35.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.337+0000 7fd3817fa640 3 client.825141.objecter handle_osd_map got epochs [536,536] > 0 2024-09-16T03:59:35.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.337+0000 7fd3817fa640 3 client.825141.objecter handle_osd_map decoding full epoch 536 2024-09-16T03:59:35.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.337+0000 7fd3817fa640 20 client.825141.objecter dump_active .. 0 homeless 2024-09-16T03:59:35.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.337+0000 7fd388b7d640 1 --2- 172.21.15.42:0/3157805954 >> v2:172.21.15.154:6812/254845342 conn(0x7fd370037810 0x7fd370039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:59:35.360 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.337+0000 7fd388b7d640 10 monclient: get_auth_request con 0x7fd370037810 auth_method 0 2024-09-16T03:59:35.360 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.341+0000 7fd388b7d640 1 --2- 172.21.15.42:0/3157805954 >> v2:172.21.15.154:6812/254845342 conn(0x7fd370037810 0x7fd370039cd0 secure :-1 s=READY pgs=3729 cs=0 l=1 rev1=1 crypto rx=0x7fd37401aa10 tx=0x7fd3740a6000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:59:35.360 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.341+0000 7fd3817fa640 10 client.825141.objecter ms_handle_connect 0x7fd370037810 2024-09-16T03:59:35.360 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.345+0000 7fd3817fa640 1 -- 172.21.15.42:0/3157805954 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7fd378006890 con 0x7fd3840a3d90 2024-09-16T03:59:35.360 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.345+0000 7fd3817fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:59:35.360 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.345+0000 7fd3817fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:59:35.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.573+0000 7fd38ade1640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:59:35.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.573+0000 7fd38ade1640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:59:35.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.573+0000 7fd38ade1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:59:35.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.573+0000 7fd38ade1640 1 -- 172.21.15.42:0/3157805954 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fd38406f680 con 0x7fd3840a3d90 2024-09-16T03:59:35.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.573+0000 7fd3817fa640 1 -- 172.21.15.42:0/3157805954 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7fd378003600 con 0x7fd3840a3d90 2024-09-16T03:59:35.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.573+0000 7fd3817fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:59:35.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.573+0000 7fd3817fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:59:35.580 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":158,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":21,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:59:35.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.577+0000 7fd38ade1640 10 librados: watch_flush enter 2024-09-16T03:59:35.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.577+0000 7fd38ade1640 10 librados: watch_flush exit 2024-09-16T03:59:35.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.577+0000 7fd38ade1640 20 client.825141.objecter shutdown clearing up homeless session... 2024-09-16T03:59:35.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.577+0000 7fd38ade1640 10 client.825141.objecter successfully canceled tick 2024-09-16T03:59:35.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.577+0000 7fd38ade1640 1 -- 172.21.15.42:0/3157805954 >> v2:172.21.15.154:6812/254845342 conn(0x7fd370037810 msgr2=0x7fd370039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:59:35.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.577+0000 7fd38ade1640 1 --2- 172.21.15.42:0/3157805954 >> v2:172.21.15.154:6812/254845342 conn(0x7fd370037810 0x7fd370039cd0 secure :-1 s=READY pgs=3729 cs=0 l=1 rev1=1 crypto rx=0x7fd37401aa10 tx=0x7fd3740a6000 comp rx=0 tx=0).stop 2024-09-16T03:59:35.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.577+0000 7fd38ade1640 10 monclient: shutdown 2024-09-16T03:59:35.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.577+0000 7fd38ade1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:59:35.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.577+0000 7fd38ade1640 1 -- 172.21.15.42:0/3157805954 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd3840a3d90 msgr2=0x7fd38412dbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:59:35.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.577+0000 7fd38ade1640 1 --2- 172.21.15.42:0/3157805954 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd3840a3d90 0x7fd38412dbf0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fd37800c3c0 tx=0x7fd37800c880 comp rx=0 tx=0).stop 2024-09-16T03:59:35.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.577+0000 7fd38ade1640 1 -- 172.21.15.42:0/3157805954 shutdown_connections 2024-09-16T03:59:35.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.577+0000 7fd38ade1640 1 --2- 172.21.15.42:0/3157805954 >> v2:172.21.15.154:6812/254845342 conn(0x7fd370037810 0x7fd370039cd0 unknown :-1 s=CLOSED pgs=3729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:59:35.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.577+0000 7fd38ade1640 1 --2- 172.21.15.42:0/3157805954 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd384137710 0x7fd3841cf620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:59:35.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.577+0000 7fd38ade1640 1 --2- 172.21.15.42:0/3157805954 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd3840a3d90 0x7fd38412dbf0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:59:35.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.577+0000 7fd38ade1640 1 --2- 172.21.15.42:0/3157805954 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd3840a39b0 0x7fd38412d6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:59:35.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.577+0000 7fd38ade1640 1 -- 172.21.15.42:0/3157805954 >> 172.21.15.42:0/3157805954 conn(0x7fd3840a9fb0 msgr2=0x7fd384131980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:59:35.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.577+0000 7fd38ade1640 1 -- 172.21.15.42:0/3157805954 shutdown_connections 2024-09-16T03:59:35.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.577+0000 7fd38ade1640 1 -- 172.21.15.42:0/3157805954 wait complete. 2024-09-16T03:59:35.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:35.577+0000 7fd38ade1640 1 librados: shutdown 2024-09-16T03:59:38.600 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:59:38.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.729+0000 7f02e2d93640 10 monclient: get_monmap_and_config 2024-09-16T03:59:38.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.729+0000 7f02e2d93640 10 monclient: build_initial_monmap 2024-09-16T03:59:38.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.729+0000 7f02e2d93640 10 monclient: monmap: 2024-09-16T03:59:38.735 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:59:38.736 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:59:38.736 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:59:38.733342+0000 2024-09-16T03:59:38.736 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:59:38.733342+0000 2024-09-16T03:59:38.736 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:59:38.736 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:59:38.736 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:59:38.737 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:59:38.737 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:59:38.737 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:59:38.737 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:59:38.737 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:59:38.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.729+0000 7f02e2d93640 1 Processor -- start 2024-09-16T03:59:38.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.729+0000 7f02e2d93640 1 -- start start 2024-09-16T03:59:38.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.729+0000 7f02e2d93640 10 monclient: init 2024-09-16T03:59:38.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.729+0000 7f02e2d93640 10 monclient: _reopen_session rank -1 2024-09-16T03:59:38.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.729+0000 7f02e2d93640 10 monclient: _add_conns ranks=[2,4,0,3,1] 2024-09-16T03:59:38.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.729+0000 7f02e2d93640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f02dc0a1330 0x7f02dc0a1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:59:38.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.729+0000 7f02e2d93640 10 monclient(hunting): picked mon.noname-b con 0x7f02dc0a1330 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:59:38.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.729+0000 7f02e2d93640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f02dc06dac0 0x7f02dc0a1c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:59:38.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.729+0000 7f02e2d93640 10 monclient(hunting): picked mon.noname-c con 0x7f02dc06dac0 addr v2:172.21.15.154:3300/0 2024-09-16T03:59:38.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.729+0000 7f02e2d93640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f02dc135100 addr v1:172.21.15.42:6789/0 2024-09-16T03:59:38.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.729+0000 7f02e2d93640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f02dc09c7d0 con 0x7f02dc135100 2024-09-16T03:59:38.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.729+0000 7f02e2d93640 10 monclient(hunting): start opening mon connection 2024-09-16T03:59:38.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.729+0000 7f02e2d93640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f02dc09c090 con 0x7f02dc06dac0 2024-09-16T03:59:38.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.729+0000 7f02e2d93640 10 monclient(hunting): start opening mon connection 2024-09-16T03:59:38.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.729+0000 7f02e2d93640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f02dc09f610 con 0x7f02dc0a1330 2024-09-16T03:59:38.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.729+0000 7f02e2d93640 10 monclient(hunting): _renew_subs 2024-09-16T03:59:38.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.729+0000 7f02e2d93640 10 monclient(hunting): authenticate will time out at 2025.703284s 2024-09-16T03:59:38.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.729+0000 7f02e0b2f640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f02dc0a1330 0x7f02dc0a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:59:38.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.729+0000 7f02e1330640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f02dc135100 0x7f02dc135550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:52078/0 (socket says 172.21.15.42:52078) 2024-09-16T03:59:38.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.729+0000 7f02e0b2f640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f02dc0a1330 0x7f02dc0a1710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:40494/0 (socket says 172.21.15.42:40494) 2024-09-16T03:59:38.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.729+0000 7f02e1330640 1 -- 172.21.15.42:0/1455638975 learned_addr learned my addr 172.21.15.42:0/1455638975 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:59:38.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.729+0000 7f02e0b2f640 10 monclient(hunting): get_auth_request con 0x7f02dc0a1330 auth_method 0 2024-09-16T03:59:38.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.729+0000 7f02e0b2f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:59:38.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.729+0000 7f02e0b2f640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:59:38.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.729+0000 7f02e0b2f640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:59:38.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.729+0000 7f02e0b2f640 1 -- 172.21.15.42:0/1455638975 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f02dc0a1330 msgr2=0x7f02dc0a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T03:59:38.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.729+0000 7f02e0b2f640 1 -- 172.21.15.42:0/1455638975 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f02dc0a1330 msgr2=0x7f02dc0a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:59:38.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.729+0000 7f02e0b2f640 1 --2- 172.21.15.42:0/1455638975 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f02dc0a1330 0x7f02dc0a1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:59:38.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.729+0000 7f02e0b2f640 1 --2- 172.21.15.42:0/1455638975 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f02dc0a1330 0x7f02dc0a1710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:59:38.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.729+0000 7f02dbfff640 1 --2- 172.21.15.42:0/1455638975 >> v2:172.21.15.154:3300/0 conn(0x7f02dc06dac0 0x7f02dc0a1c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:59:38.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.729+0000 7f02dbfff640 10 monclient(hunting): get_auth_request con 0x7f02dc06dac0 auth_method 0 2024-09-16T03:59:38.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.729+0000 7f02dbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:59:38.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.729+0000 7f02dbfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:59:38.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.729+0000 7f02dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:59:38.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.729+0000 7f02db7fe640 1 -- 172.21.15.42:0/1455638975 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 315451511 0 0) 0x7f02dc09c7d0 con 0x7f02dc135100 2024-09-16T03:59:38.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.729+0000 7f02db7fe640 10 monclient(hunting): _negotiate 2024-09-16T03:59:38.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.729+0000 7f02db7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:59:38.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.729+0000 7f02db7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:59:38.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.729+0000 7f02db7fe640 10 monclient(hunting): my global_id is 825144 2024-09-16T03:59:38.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.729+0000 7f02db7fe640 1 -- 172.21.15.42:0/1455638975 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f02c4003530 con 0x7f02dc135100 2024-09-16T03:59:38.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.729+0000 7f02dbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:59:38.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.729+0000 7f02dbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:59:38.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.729+0000 7f02dbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:59:38.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.733+0000 7f02db7fe640 1 -- 172.21.15.42:0/1455638975 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3704129580 0 0) 0x7f02c4003530 con 0x7f02dc135100 2024-09-16T03:59:38.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.733+0000 7f02db7fe640 1 -- 172.21.15.42:0/1455638975 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f02dc0a1330 msgr2=0x7f02dc0a1710 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:59:38.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.733+0000 7f02db7fe640 1 --2- 172.21.15.42:0/1455638975 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f02dc0a1330 0x7f02dc0a1710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:59:38.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.733+0000 7f02db7fe640 1 -- 172.21.15.42:0/1455638975 >> v2:172.21.15.154:3300/0 conn(0x7f02dc06dac0 msgr2=0x7f02dc0a1c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:59:38.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.733+0000 7f02db7fe640 1 --2- 172.21.15.42:0/1455638975 >> v2:172.21.15.154:3300/0 conn(0x7f02dc06dac0 0x7f02dc0a1c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:59:38.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.733+0000 7f02db7fe640 10 monclient: _finish_hunting 0 2024-09-16T03:59:38.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.733+0000 7f02db7fe640 1 monclient: found mon.noname-a-legacy 2024-09-16T03:59:38.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.733+0000 7f02db7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:59:38.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.733+0000 7f02db7fe640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.42:6789/0 2024-09-16T03:59:38.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.733+0000 7f02db7fe640 1 -- 172.21.15.42:0/1455638975 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02dc09f8f0 con 0x7f02dc135100 2024-09-16T03:59:38.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.733+0000 7f02db7fe640 10 monclient: _finish_auth 0 2024-09-16T03:59:38.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.733+0000 7f02db7fe640 10 monclient: _check_auth_tickets 2024-09-16T03:59:38.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.733+0000 7f02db7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:59:38.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.733+0000 7f02db7fe640 1 -- 172.21.15.42:0/1455638975 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2177413734 0 0) 0x7f02d4001680 con 0x7f02dc135100 2024-09-16T03:59:38.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.733+0000 7f02e2d93640 5 monclient: authenticate success, global_id 825144 2024-09-16T03:59:38.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.733+0000 7f02e2d93640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:59:38.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.733+0000 7f02db7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:59:38.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.733+0000 7f02db7fe640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T03:59:38.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.733+0000 7f02db7fe640 10 monclient: dump: 2024-09-16T03:59:38.760 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:59:38.760 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:59:38.760 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:38.760 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:38.760 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:59:38.760 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:59:38.760 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:59:38.760 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:59:38.760 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:59:38.761 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:59:38.761 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.733+0000 7f02db7fe640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] but i'm connected to v1:172.21.15.42:6789/0, reconnecting 2024-09-16T03:59:38.761 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.733+0000 7f02db7fe640 10 monclient: _reopen_session rank -1 2024-09-16T03:59:38.761 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.733+0000 7f02db7fe640 1 -- 172.21.15.42:0/1455638975 >> v1:172.21.15.42:6789/0 conn(0x7f02dc135100 legacy=0x7f02dc135550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:59:38.761 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.733+0000 7f02db7fe640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T03:59:38.761 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.733+0000 7f02db7fe640 1 --2- 172.21.15.42:0/1455638975 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f02c40039c0 0x7f02c4003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:59:38.761 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.733+0000 7f02db7fe640 10 monclient(hunting): picked mon.b con 0x7f02c40039c0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:59:38.761 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.733+0000 7f02db7fe640 1 --2- 172.21.15.42:0/1455638975 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f02dc0a1330 0x7f02c4004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:59:38.761 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.733+0000 7f02db7fe640 10 monclient(hunting): picked mon.c con 0x7f02dc0a1330 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:59:38.761 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.733+0000 7f02db7fe640 1 --2- 172.21.15.42:0/1455638975 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f02c4004d50 0x7f02c40051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:59:38.761 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.733+0000 7f02db7fe640 10 monclient(hunting): picked mon.a con 0x7f02c4004d50 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:59:38.761 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.733+0000 7f02db7fe640 10 monclient(hunting): start opening mon connection 2024-09-16T03:59:38.761 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.733+0000 7f02db7fe640 1 -- 172.21.15.42:0/1455638975 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f02dc09c090 con 0x7f02c4004d50 2024-09-16T03:59:38.761 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.733+0000 7f02db7fe640 10 monclient(hunting): start opening mon connection 2024-09-16T03:59:38.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.733+0000 7f02db7fe640 1 -- 172.21.15.42:0/1455638975 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f02dc09f610 con 0x7f02c40039c0 2024-09-16T03:59:38.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.733+0000 7f02db7fe640 10 monclient(hunting): start opening mon connection 2024-09-16T03:59:38.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.733+0000 7f02db7fe640 1 -- 172.21.15.42:0/1455638975 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f02c400ba80 con 0x7f02dc0a1330 2024-09-16T03:59:38.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.733+0000 7f02db7fe640 10 monclient(hunting): _renew_subs 2024-09-16T03:59:38.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.733+0000 7f02db7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T03:59:38.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.733+0000 7f02dbfff640 1 --2- 172.21.15.42:0/1455638975 >> v2:172.21.15.154:3300/0 conn(0x7f02dc06dac0 0x7f02dc0a1c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:59:38.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.733+0000 7f02e2d93640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-16T03:59:38.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.733+0000 7f02e1330640 1 --2- 172.21.15.42:0/1455638975 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f02c4004d50 0x7f02c40051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:59:38.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.733+0000 7f02e1330640 10 monclient(hunting): get_auth_request con 0x7f02c4004d50 auth_method 0 2024-09-16T03:59:38.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.733+0000 7f02e1330640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:59:38.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.733+0000 7f02e1330640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:59:38.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.733+0000 7f02e1330640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:59:38.763 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.733+0000 7f02dbfff640 1 --2- 172.21.15.42:0/1455638975 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f02dc0a1330 0x7f02c4004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:59:38.763 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.733+0000 7f02dbfff640 10 monclient(hunting): get_auth_request con 0x7f02dc0a1330 auth_method 0 2024-09-16T03:59:38.763 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.733+0000 7f02dbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:59:38.763 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.733+0000 7f02dbfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:59:38.763 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.733+0000 7f02dbfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:59:38.763 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.733+0000 7f02dbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:59:38.763 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.733+0000 7f02dbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:59:38.764 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.733+0000 7f02e0b2f640 1 --2- 172.21.15.42:0/1455638975 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f02c40039c0 0x7f02c4003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:59:38.764 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.733+0000 7f02dbfff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T03:59:38.764 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.733+0000 7f02e1330640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:59:38.764 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.733+0000 7f02e1330640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:59:38.764 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.733+0000 7f02e1330640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T03:59:38.764 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.733+0000 7f02e0b2f640 10 monclient(hunting): get_auth_request con 0x7f02c40039c0 auth_method 0 2024-09-16T03:59:38.764 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.733+0000 7f02e0b2f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:59:38.764 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.733+0000 7f02e0b2f640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:59:38.764 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.733+0000 7f02e0b2f640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:59:38.764 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.733+0000 7f02dbfff640 10 monclient(hunting): handle_auth_done global_id 825144 payload 293 2024-09-16T03:59:38.764 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.733+0000 7f02dbfff640 1 -- 172.21.15.42:0/1455638975 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f02c40039c0 msgr2=0x7f02c4003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:59:38.764 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.733+0000 7f02dbfff640 1 --2- 172.21.15.42:0/1455638975 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f02c40039c0 0x7f02c4003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:59:38.764 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.733+0000 7f02dbfff640 1 -- 172.21.15.42:0/1455638975 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f02c4004d50 msgr2=0x7f02c40051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:59:38.765 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.733+0000 7f02dbfff640 1 --2- 172.21.15.42:0/1455638975 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f02c4004d50 0x7f02c40051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:59:38.765 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.733+0000 7f02dbfff640 10 monclient: _finish_hunting 0 2024-09-16T03:59:38.765 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.733+0000 7f02dbfff640 1 monclient: found mon.c 2024-09-16T03:59:38.765 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.733+0000 7f02dbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:59:38.765 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.733+0000 7f02dbfff640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:59:38.765 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.733+0000 7f02dbfff640 1 -- 172.21.15.42:0/1455638975 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02c4003530 con 0x7f02dc0a1330 2024-09-16T03:59:38.765 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.733+0000 7f02dbfff640 10 monclient: _finish_auth 0 2024-09-16T03:59:38.765 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.733+0000 7f02dbfff640 10 monclient: _check_auth_tickets 2024-09-16T03:59:38.765 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.733+0000 7f02dbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:59:38.765 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.733+0000 7f02dbfff640 1 --2- 172.21.15.42:0/1455638975 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f02dc0a1330 0x7f02c4004810 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f02d0009c90 tx=0x7f02d001c900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f598e09b39244d82 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:59:38.765 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.733+0000 7f02db7fe640 1 -- 172.21.15.42:0/1455638975 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02d0009860 con 0x7f02dc0a1330 2024-09-16T03:59:38.766 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.733+0000 7f02db7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:59:38.766 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.733+0000 7f02db7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:59:38.766 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.733+0000 7f02db7fe640 10 monclient: dump: 2024-09-16T03:59:38.766 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:59:38.766 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:59:38.766 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:38.766 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:38.766 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:59:38.767 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:59:38.767 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:59:38.767 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:59:38.767 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:59:38.767 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:59:38.767 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.733+0000 7f02e2d93640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:59:38.767 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.733+0000 7f02db7fe640 1 -- 172.21.15.42:0/1455638975 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f02d0004020 con 0x7f02dc0a1330 2024-09-16T03:59:38.767 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.733+0000 7f02db7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:59:38.767 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.733+0000 7f02e2d93640 10 monclient: get_monmap_and_config success 2024-09-16T03:59:38.767 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.733+0000 7f02e2d93640 4 set_mon_vals no callback set 2024-09-16T03:59:38.767 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.733+0000 7f02db7fe640 1 -- 172.21.15.42:0/1455638975 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02d0004410 con 0x7f02dc0a1330 2024-09-16T03:59:38.767 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.733+0000 7f02db7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:59:38.768 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.733+0000 7f02db7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:59:38.768 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.733+0000 7f02db7fe640 10 monclient: dump: 2024-09-16T03:59:38.768 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:59:38.768 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:59:38.768 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:38.769 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:38.769 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:59:38.769 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:59:38.769 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:59:38.769 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:59:38.769 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:59:38.769 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:59:38.769 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.733+0000 7f02e2d93640 10 monclient: shutdown 2024-09-16T03:59:38.769 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.733+0000 7f02e2d93640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:59:38.769 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.733+0000 7f02e2d93640 1 -- 172.21.15.42:0/1455638975 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f02dc0a1330 msgr2=0x7f02c4004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:59:38.770 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.733+0000 7f02e2d93640 1 --2- 172.21.15.42:0/1455638975 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f02dc0a1330 0x7f02c4004810 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f02d0009c90 tx=0x7f02d001c900 comp rx=0 tx=0).stop 2024-09-16T03:59:38.770 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.733+0000 7f02e1330640 1 -- 172.21.15.42:0/1455638975 reap_dead start 2024-09-16T03:59:38.770 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.733+0000 7f02e2d93640 1 -- 172.21.15.42:0/1455638975 shutdown_connections 2024-09-16T03:59:38.770 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.733+0000 7f02e2d93640 1 -- 172.21.15.42:0/1455638975 >> 172.21.15.42:0/1455638975 conn(0x7f02dc0a2420 msgr2=0x7f02dc0a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:59:38.770 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.733+0000 7f02e2d93640 1 -- 172.21.15.42:0/1455638975 shutdown_connections 2024-09-16T03:59:38.770 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.737+0000 7f02e2d93640 1 -- 172.21.15.42:0/1455638975 wait complete. 2024-09-16T03:59:38.770 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.737+0000 7f02e2d93640 10 monclient: build_initial_monmap 2024-09-16T03:59:38.770 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.737+0000 7f02e2d93640 10 monclient: monmap: 2024-09-16T03:59:38.770 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:59:38.770 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:59:38.770 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:59:38.770 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:59:38.770 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:59:38.771 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:59:38.771 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:59:38.771 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:59:38.771 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:59:38.771 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:59:38.771 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.737+0000 7f02e2d93640 1 librados: starting msgr at 2024-09-16T03:59:38.771 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.737+0000 7f02e2d93640 1 librados: starting objecter 2024-09-16T03:59:38.771 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.737+0000 7f02e2d93640 1 Processor -- start 2024-09-16T03:59:38.771 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.737+0000 7f02e2d93640 1 -- start start 2024-09-16T03:59:38.771 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.737+0000 7f02e2d93640 1 librados: setting wanted keys 2024-09-16T03:59:38.771 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.737+0000 7f02e2d93640 1 librados: calling monclient init 2024-09-16T03:59:38.771 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.737+0000 7f02e2d93640 10 monclient: init 2024-09-16T03:59:38.771 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.737+0000 7f02e2d93640 10 monclient: _reopen_session rank -1 2024-09-16T03:59:38.771 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.737+0000 7f02e2d93640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:59:38.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.737+0000 7f02e2d93640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f02dc0a1330 0x7f02dc1ccc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:59:38.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.737+0000 7f02e2d93640 10 monclient(hunting): picked mon.noname-b con 0x7f02dc0a1330 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:59:38.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.737+0000 7f02e2d93640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f02dc133af0 0x7f02dc133f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:59:38.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.737+0000 7f02e2d93640 10 monclient(hunting): picked mon.noname-a con 0x7f02dc133af0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:59:38.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.737+0000 7f02e2d93640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f02dc1c9700 0x7f02dc1c9b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:59:38.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.737+0000 7f02e2d93640 10 monclient(hunting): picked mon.noname-c con 0x7f02dc1c9700 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:59:38.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.737+0000 7f02e2d93640 10 monclient(hunting): start opening mon connection 2024-09-16T03:59:38.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.737+0000 7f02e2d93640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f02dc136320 con 0x7f02dc133af0 2024-09-16T03:59:38.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.737+0000 7f02e2d93640 10 monclient(hunting): start opening mon connection 2024-09-16T03:59:38.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.737+0000 7f02e2d93640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f02dc09ef80 con 0x7f02dc0a1330 2024-09-16T03:59:38.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.737+0000 7f02e2d93640 10 monclient(hunting): start opening mon connection 2024-09-16T03:59:38.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.737+0000 7f02e2d93640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f02dc1364a0 con 0x7f02dc1c9700 2024-09-16T03:59:38.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.737+0000 7f02e2d93640 10 monclient(hunting): _renew_subs 2024-09-16T03:59:38.773 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.737+0000 7f02e2d93640 10 monclient(hunting): authenticate will time out at 2025.709468s 2024-09-16T03:59:38.773 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.737+0000 7f02dbfff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f02dc133af0 0x7f02dc133f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:59:38.773 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.737+0000 7f02e1330640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f02dc1c9700 0x7f02dc1c9b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:59:38.773 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.737+0000 7f02dbfff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f02dc133af0 0x7f02dc133f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:58788/0 (socket says 172.21.15.42:58788) 2024-09-16T03:59:38.773 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.737+0000 7f02e1330640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f02dc1c9700 0x7f02dc1c9b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:40516/0 (socket says 172.21.15.42:40516) 2024-09-16T03:59:38.773 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.737+0000 7f02dbfff640 1 -- 172.21.15.42:0/318836247 learned_addr learned my addr 172.21.15.42:0/318836247 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:59:38.773 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.737+0000 7f02dbfff640 10 monclient(hunting): get_auth_request con 0x7f02dc133af0 auth_method 0 2024-09-16T03:59:38.773 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.737+0000 7f02dbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:59:38.773 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.737+0000 7f02dbfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:59:38.773 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.737+0000 7f02dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:59:38.773 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.737+0000 7f02e1330640 10 monclient(hunting): get_auth_request con 0x7f02dc1c9700 auth_method 0 2024-09-16T03:59:38.773 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.737+0000 7f02e1330640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:59:38.774 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.737+0000 7f02e1330640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:59:38.774 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.737+0000 7f02e1330640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:59:38.774 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.737+0000 7f02dbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:59:38.774 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.737+0000 7f02dbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:59:38.774 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.737+0000 7f02e1330640 1 -- 172.21.15.42:0/318836247 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f02dc1c9700 msgr2=0x7f02dc1c9b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-16T03:59:38.774 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.737+0000 7f02e1330640 1 -- 172.21.15.42:0/318836247 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f02dc1c9700 msgr2=0x7f02dc1c9b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T03:59:38.774 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.737+0000 7f02e1330640 1 --2- 172.21.15.42:0/318836247 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f02dc1c9700 0x7f02dc1c9b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:59:38.774 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.737+0000 7f02dbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:59:38.774 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.737+0000 7f02e1330640 1 --2- 172.21.15.42:0/318836247 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f02dc1c9700 0x7f02dc1c9b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:59:38.774 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.737+0000 7f02e0b2f640 1 --2- 172.21.15.42:0/318836247 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f02dc0a1330 0x7f02dc1ccc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:59:38.774 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.737+0000 7f02e0b2f640 10 monclient(hunting): get_auth_request con 0x7f02dc0a1330 auth_method 0 2024-09-16T03:59:38.774 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.737+0000 7f02e0b2f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:59:38.774 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.737+0000 7f02e0b2f640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:59:38.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.737+0000 7f02e0b2f640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:59:38.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.737+0000 7f02dbfff640 10 monclient(hunting): handle_auth_done global_id 825147 payload 824 2024-09-16T03:59:38.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.737+0000 7f02dbfff640 1 -- 172.21.15.42:0/318836247 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f02dc1c9700 msgr2=0x7f02dc1c9b60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:59:38.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.737+0000 7f02dbfff640 1 --2- 172.21.15.42:0/318836247 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f02dc1c9700 0x7f02dc1c9b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:59:38.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.737+0000 7f02dbfff640 1 -- 172.21.15.42:0/318836247 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f02dc0a1330 msgr2=0x7f02dc1ccc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:59:38.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.737+0000 7f02dbfff640 1 --2- 172.21.15.42:0/318836247 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f02dc0a1330 0x7f02dc1ccc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:59:38.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.737+0000 7f02dbfff640 10 monclient: _finish_hunting 0 2024-09-16T03:59:38.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.737+0000 7f02dbfff640 1 monclient: found mon.noname-a 2024-09-16T03:59:38.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.737+0000 7f02dbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:59:38.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.737+0000 7f02dbfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:59:38.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.737+0000 7f02dbfff640 1 -- 172.21.15.42:0/318836247 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02dc127230 con 0x7f02dc133af0 2024-09-16T03:59:38.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.737+0000 7f02dbfff640 1 --2- 172.21.15.42:0/318836247 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f02dc133af0 0x7f02dc133f70 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f02d00ad040 tx=0x7f02c4005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:59:38.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.737+0000 7f02d97fa640 10 client.?.objecter ms_handle_connect 0x7f02dc133af0 2024-09-16T03:59:38.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.737+0000 7f02d97fa640 10 client.?.objecter resend_mon_ops 2024-09-16T03:59:38.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.737+0000 7f02d97fa640 1 -- 172.21.15.42:0/318836247 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02d00ba070 con 0x7f02dc133af0 2024-09-16T03:59:38.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.737+0000 7f02d97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:59:38.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.737+0000 7f02d97fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:59:38.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.737+0000 7f02d97fa640 10 monclient: dump: 2024-09-16T03:59:38.776 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:59:38.776 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:59:38.776 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:38.776 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:38.776 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:59:38.776 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:59:38.776 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:59:38.776 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:59:38.776 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:59:38.776 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:59:38.777 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.737+0000 7f02d97fa640 10 monclient: _finish_auth 0 2024-09-16T03:59:38.777 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.737+0000 7f02d97fa640 10 monclient: _check_auth_tickets 2024-09-16T03:59:38.777 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.737+0000 7f02d97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:59:38.777 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.737+0000 7f02d97fa640 1 -- 172.21.15.42:0/318836247 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f02d000f080 con 0x7f02dc133af0 2024-09-16T03:59:38.777 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.737+0000 7f02e2d93640 5 monclient: authenticate success, global_id 825147 2024-09-16T03:59:38.777 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.737+0000 7f02d97fa640 10 monclient: handle_config config(0 keys) 2024-09-16T03:59:38.777 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.737+0000 7f02d97fa640 1 -- 172.21.15.42:0/318836247 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02d0005ba0 con 0x7f02dc133af0 2024-09-16T03:59:38.777 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.737+0000 7f02e2d93640 10 monclient: _renew_subs 2024-09-16T03:59:38.777 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.737+0000 7f02e2d93640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:59:38.777 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.737+0000 7f02e2d93640 1 -- 172.21.15.42:0/318836247 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02dc1256c0 con 0x7f02dc133af0 2024-09-16T03:59:38.777 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.737+0000 7f02d97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:59:38.777 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.737+0000 7f02d97fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:59:38.777 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.737+0000 7f02d97fa640 10 monclient: dump: 2024-09-16T03:59:38.777 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:59:38.778 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:59:38.778 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:38.778 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:38.778 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:59:38.778 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:59:38.778 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:59:38.778 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:59:38.778 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:59:38.779 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:59:38.779 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.737+0000 7f02daffd640 4 set_mon_vals no callback set 2024-09-16T03:59:38.779 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.737+0000 7f02e2d93640 10 client.825147.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:59:38.779 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.737+0000 7f02e2d93640 10 monclient: _renew_subs 2024-09-16T03:59:38.779 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.737+0000 7f02e2d93640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:59:38.779 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.737+0000 7f02e2d93640 1 -- 172.21.15.42:0/318836247 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02dc09c9d0 con 0x7f02dc133af0 2024-09-16T03:59:38.779 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.737+0000 7f02e2d93640 1 librados: init done 2024-09-16T03:59:38.779 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.737+0000 7f02e2d93640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:59:38.779 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.737+0000 7f02e2d93640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:59:38.779 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.737+0000 7f02e2d93640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:59:38.779 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.737+0000 7f02e2d93640 1 -- 172.21.15.42:0/318836247 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f02dc0a1b20 con 0x7f02dc133af0 2024-09-16T03:59:38.779 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.737+0000 7f02d97fa640 1 -- 172.21.15.42:0/318836247 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f02d00bd070 con 0x7f02dc133af0 2024-09-16T03:59:38.779 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.737+0000 7f02d97fa640 1 --2- 172.21.15.42:0/318836247 >> v2:172.21.15.154:6812/254845342 conn(0x7f02c8037810 0x7f02c8039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:59:38.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.737+0000 7f02d97fa640 1 -- 172.21.15.42:0/318836247 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(539..539 src has 451..539) ==== 6168+0+0 (secure 0 0 0) 0x7f02d00e2ff0 con 0x7f02dc133af0 2024-09-16T03:59:38.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.737+0000 7f02d97fa640 10 client.825147.objecter ms_dispatch 0x7f02dc000ba0 osd_map(539..539 src has 451..539) 2024-09-16T03:59:38.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.737+0000 7f02d97fa640 3 client.825147.objecter handle_osd_map got epochs [539,539] > 0 2024-09-16T03:59:38.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.737+0000 7f02d97fa640 3 client.825147.objecter handle_osd_map decoding full epoch 539 2024-09-16T03:59:38.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.737+0000 7f02d97fa640 20 client.825147.objecter dump_active .. 0 homeless 2024-09-16T03:59:38.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.737+0000 7f02e0b2f640 1 --2- 172.21.15.42:0/318836247 >> v2:172.21.15.154:6812/254845342 conn(0x7f02c8037810 0x7f02c8039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:59:38.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.741+0000 7f02e0b2f640 10 monclient: get_auth_request con 0x7f02c8037810 auth_method 0 2024-09-16T03:59:38.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.741+0000 7f02d97fa640 10 client.825147.objecter ms_handle_connect 0x7f02c8037810 2024-09-16T03:59:38.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.741+0000 7f02e0b2f640 1 --2- 172.21.15.42:0/318836247 >> v2:172.21.15.154:6812/254845342 conn(0x7f02c8037810 0x7f02c8039cd0 secure :-1 s=READY pgs=3730 cs=0 l=1 rev1=1 crypto rx=0x7f02cc003fd0 tx=0x7f02cc0039a0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:59:38.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.745+0000 7f02d97fa640 1 -- 172.21.15.42:0/318836247 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f02d0016960 con 0x7f02dc133af0 2024-09-16T03:59:38.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.745+0000 7f02d97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:59:38.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.745+0000 7f02d97fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:59:38.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.969+0000 7f02e2d93640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:59:38.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.969+0000 7f02e2d93640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:59:38.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.969+0000 7f02e2d93640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:59:38.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.969+0000 7f02e2d93640 1 -- 172.21.15.42:0/318836247 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f02dc124aa0 con 0x7f02dc133af0 2024-09-16T03:59:38.978 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.973+0000 7f02d97fa640 1 -- 172.21.15.42:0/318836247 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7f02d0005df0 con 0x7f02dc133af0 2024-09-16T03:59:38.978 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.973+0000 7f02d97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:59:38.978 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.973+0000 7f02d97fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:59:38.978 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":158,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":24,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:59:38.980 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.973+0000 7f02e2d93640 10 librados: watch_flush enter 2024-09-16T03:59:38.980 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.973+0000 7f02e2d93640 10 librados: watch_flush exit 2024-09-16T03:59:38.980 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.973+0000 7f02e2d93640 20 client.825147.objecter shutdown clearing up homeless session... 2024-09-16T03:59:38.980 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.973+0000 7f02e2d93640 10 client.825147.objecter successfully canceled tick 2024-09-16T03:59:38.980 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.973+0000 7f02e2d93640 1 -- 172.21.15.42:0/318836247 >> v2:172.21.15.154:6812/254845342 conn(0x7f02c8037810 msgr2=0x7f02c8039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:59:38.980 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.973+0000 7f02e2d93640 1 --2- 172.21.15.42:0/318836247 >> v2:172.21.15.154:6812/254845342 conn(0x7f02c8037810 0x7f02c8039cd0 secure :-1 s=READY pgs=3730 cs=0 l=1 rev1=1 crypto rx=0x7f02cc003fd0 tx=0x7f02cc0039a0 comp rx=0 tx=0).stop 2024-09-16T03:59:38.980 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.973+0000 7f02e2d93640 10 monclient: shutdown 2024-09-16T03:59:38.980 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.973+0000 7f02e2d93640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:59:38.981 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.973+0000 7f02e2d93640 1 -- 172.21.15.42:0/318836247 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f02dc133af0 msgr2=0x7f02dc133f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:59:38.981 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.973+0000 7f02e2d93640 1 --2- 172.21.15.42:0/318836247 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f02dc133af0 0x7f02dc133f70 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f02d00ad040 tx=0x7f02c4005840 comp rx=0 tx=0).stop 2024-09-16T03:59:38.981 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.973+0000 7f02e2d93640 1 -- 172.21.15.42:0/318836247 shutdown_connections 2024-09-16T03:59:38.981 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.973+0000 7f02e2d93640 1 --2- 172.21.15.42:0/318836247 >> v2:172.21.15.154:6812/254845342 conn(0x7f02c8037810 0x7f02c8039cd0 unknown :-1 s=CLOSED pgs=3730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:59:38.981 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.973+0000 7f02e2d93640 1 --2- 172.21.15.42:0/318836247 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f02dc1c9700 0x7f02dc1c9b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:59:38.981 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.973+0000 7f02e2d93640 1 --2- 172.21.15.42:0/318836247 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f02dc133af0 0x7f02dc133f70 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:59:38.981 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.973+0000 7f02e2d93640 1 --2- 172.21.15.42:0/318836247 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f02dc0a1330 0x7f02dc1ccc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:59:38.981 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.973+0000 7f02e2d93640 1 -- 172.21.15.42:0/318836247 >> 172.21.15.42:0/318836247 conn(0x7f02dc0a2420 msgr2=0x7f02dc1346d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:59:38.981 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.973+0000 7f02e2d93640 1 -- 172.21.15.42:0/318836247 shutdown_connections 2024-09-16T03:59:38.981 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.973+0000 7f02e2d93640 1 -- 172.21.15.42:0/318836247 wait complete. 2024-09-16T03:59:38.981 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:38.973+0000 7f02e2d93640 1 librados: shutdown 2024-09-16T03:59:41.997 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:59:42.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.125+0000 7f54a0e75640 10 monclient: get_monmap_and_config 2024-09-16T03:59:42.131 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.125+0000 7f54a0e75640 10 monclient: build_initial_monmap 2024-09-16T03:59:42.131 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.125+0000 7f54a0e75640 10 monclient: monmap: 2024-09-16T03:59:42.131 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:59:42.131 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:59:42.131 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:59:42.129183+0000 2024-09-16T03:59:42.131 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:59:42.129183+0000 2024-09-16T03:59:42.131 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:59:42.131 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:59:42.131 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:59:42.131 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:59:42.131 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:59:42.131 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:59:42.131 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:59:42.132 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:59:42.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.125+0000 7f54a0e75640 1 Processor -- start 2024-09-16T03:59:42.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.125+0000 7f54a0e75640 1 -- start start 2024-09-16T03:59:42.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.125+0000 7f54a0e75640 10 monclient: init 2024-09-16T03:59:42.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.125+0000 7f54a0e75640 10 monclient: _reopen_session rank -1 2024-09-16T03:59:42.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.125+0000 7f54a0e75640 10 monclient: _add_conns ranks=[1,2,3,4,0] 2024-09-16T03:59:42.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.125+0000 7f54a0e75640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f549c06db90 addr v1:172.21.15.154:6789/0 2024-09-16T03:59:42.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.125+0000 7f54a0e75640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f549c1340c0 0x7f549c13c190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:59:42.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.125+0000 7f54a0e75640 10 monclient(hunting): picked mon.noname-b con 0x7f549c1340c0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:59:42.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.125+0000 7f54a0e75640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f549c13c6d0 0x7f549c13eac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:59:42.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.125+0000 7f54a0e75640 10 monclient(hunting): picked mon.noname-a con 0x7f549c13c6d0 addr v2:172.21.15.42:3300/0 2024-09-16T03:59:42.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.125+0000 7f54a0e75640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f549c06ece0 con 0x7f549c06db90 2024-09-16T03:59:42.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.125+0000 7f54a0e75640 10 monclient(hunting): start opening mon connection 2024-09-16T03:59:42.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.125+0000 7f54a0e75640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f549c06df70 con 0x7f549c13c6d0 2024-09-16T03:59:42.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.125+0000 7f54a0e75640 10 monclient(hunting): start opening mon connection 2024-09-16T03:59:42.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.125+0000 7f54a0e75640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f549c09bf80 con 0x7f549c1340c0 2024-09-16T03:59:42.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.125+0000 7f54a0e75640 10 monclient(hunting): _renew_subs 2024-09-16T03:59:42.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.125+0000 7f54a0e75640 10 monclient(hunting): authenticate will time out at 2029.099493s 2024-09-16T03:59:42.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.125+0000 7f5499d9b640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f549c1340c0 0x7f549c13c190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:59:42.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.125+0000 7f549ad9d640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f549c13c6d0 0x7f549c13eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:59:42.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.125+0000 7f5499d9b640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f549c1340c0 0x7f549c13c190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:40546/0 (socket says 172.21.15.42:40546) 2024-09-16T03:59:42.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.125+0000 7f5499d9b640 1 -- 172.21.15.42:0/798296951 learned_addr learned my addr 172.21.15.42:0/798296951 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:59:42.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.125+0000 7f549ad9d640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f549c13c6d0 0x7f549c13eac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:58790/0 (socket says 172.21.15.42:58790) 2024-09-16T03:59:42.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.125+0000 7f5499d9b640 10 monclient(hunting): get_auth_request con 0x7f549c1340c0 auth_method 0 2024-09-16T03:59:42.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.125+0000 7f5499d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:59:42.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.125+0000 7f5499d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:59:42.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.125+0000 7f5499d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:59:42.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.125+0000 7f549ad9d640 10 monclient(hunting): get_auth_request con 0x7f549c13c6d0 auth_method 0 2024-09-16T03:59:42.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.125+0000 7f549ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:59:42.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.125+0000 7f549ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:59:42.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.125+0000 7f549ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:59:42.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.125+0000 7f549ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:59:42.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.125+0000 7f549ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:59:42.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.125+0000 7f549ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:59:42.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.125+0000 7f5499d9b640 1 -- 172.21.15.42:0/798296951 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f549c1340c0 msgr2=0x7f549c13c190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T03:59:42.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.125+0000 7f5499d9b640 1 -- 172.21.15.42:0/798296951 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f549c1340c0 msgr2=0x7f549c13c190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:59:42.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.125+0000 7f5499d9b640 1 --2- 172.21.15.42:0/798296951 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f549c1340c0 0x7f549c13c190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:59:42.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.125+0000 7f5499d9b640 1 --2- 172.21.15.42:0/798296951 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f549c1340c0 0x7f549c13c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:59:42.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.129+0000 7f549ad9d640 10 monclient(hunting): handle_auth_done global_id 825150 payload 274 2024-09-16T03:59:42.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.129+0000 7f549ad9d640 1 -- 172.21.15.42:0/798296951 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f549c1340c0 msgr2=0x7f549c13c190 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:59:42.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.129+0000 7f549ad9d640 1 --2- 172.21.15.42:0/798296951 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f549c1340c0 0x7f549c13c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:59:42.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.129+0000 7f549ad9d640 1 -- 172.21.15.42:0/798296951 >> v1:172.21.15.154:6789/0 conn(0x7f549c06db90 legacy=0x7f549c0a37f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:59:42.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.129+0000 7f549ad9d640 10 monclient: _finish_hunting 0 2024-09-16T03:59:42.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.129+0000 7f549ad9d640 1 monclient: found mon.noname-a 2024-09-16T03:59:42.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.129+0000 7f549ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:59:42.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.129+0000 7f549ad9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:59:42.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.129+0000 7f549ad9d640 1 -- 172.21.15.42:0/798296951 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f549c1344d0 con 0x7f549c13c6d0 2024-09-16T03:59:42.143 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.129+0000 7f549ad9d640 1 --2- 172.21.15.42:0/798296951 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f549c13c6d0 0x7f549c13eac0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f5494009d90 tx=0x7f549401c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce2c118b25ba9726 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:59:42.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.129+0000 7f549959a640 1 -- 172.21.15.42:0/798296951 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54940b2070 con 0x7f549c13c6d0 2024-09-16T03:59:42.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.129+0000 7f549959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:59:42.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.129+0000 7f549959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:59:42.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.129+0000 7f549959a640 10 monclient: dump: 2024-09-16T03:59:42.144 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:59:42.144 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:59:42.144 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:42.144 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:42.144 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:59:42.144 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:59:42.144 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:59:42.145 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:59:42.145 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:59:42.145 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:59:42.145 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.129+0000 7f549959a640 10 monclient: _finish_auth 0 2024-09-16T03:59:42.145 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.129+0000 7f549959a640 10 monclient: _check_auth_tickets 2024-09-16T03:59:42.145 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.129+0000 7f549959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:59:42.145 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.129+0000 7f54a0e75640 5 monclient: authenticate success, global_id 825150 2024-09-16T03:59:42.145 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.129+0000 7f549959a640 1 -- 172.21.15.42:0/798296951 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f54940a58f0 con 0x7f549c13c6d0 2024-09-16T03:59:42.145 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.129+0000 7f549959a640 10 monclient: handle_config config(0 keys) 2024-09-16T03:59:42.145 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.129+0000 7f54a0e75640 10 monclient: get_monmap_and_config success 2024-09-16T03:59:42.145 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.129+0000 7f549959a640 1 -- 172.21.15.42:0/798296951 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54940a5cc0 con 0x7f549c13c6d0 2024-09-16T03:59:42.145 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.129+0000 7f54a0e75640 4 set_mon_vals no callback set 2024-09-16T03:59:42.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.129+0000 7f549959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:59:42.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.129+0000 7f549959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:59:42.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.129+0000 7f549959a640 10 monclient: dump: 2024-09-16T03:59:42.146 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:59:42.146 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:59:42.146 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:42.146 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:42.146 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:59:42.146 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:59:42.146 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:59:42.146 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:59:42.146 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:59:42.147 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:59:42.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.129+0000 7f54a0e75640 10 monclient: shutdown 2024-09-16T03:59:42.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.129+0000 7f54a0e75640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:59:42.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.129+0000 7f54a0e75640 1 -- 172.21.15.42:0/798296951 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f549c13c6d0 msgr2=0x7f549c13eac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:59:42.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.129+0000 7f54a0e75640 1 --2- 172.21.15.42:0/798296951 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f549c13c6d0 0x7f549c13eac0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f5494009d90 tx=0x7f549401c830 comp rx=0 tx=0).stop 2024-09-16T03:59:42.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.129+0000 7f54a0e75640 1 -- 172.21.15.42:0/798296951 shutdown_connections 2024-09-16T03:59:42.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.129+0000 7f54a0e75640 1 --2- 172.21.15.42:0/798296951 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f549c13c6d0 0x7f549c13eac0 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:59:42.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.129+0000 7f54a0e75640 1 --2- 172.21.15.42:0/798296951 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f549c1340c0 0x7f549c13c190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:59:42.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.129+0000 7f54a0e75640 1 -- 172.21.15.42:0/798296951 >> 172.21.15.42:0/798296951 conn(0x7f549c0a9fb0 msgr2=0x7f549c0aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:59:42.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.129+0000 7f54a0e75640 1 -- 172.21.15.42:0/798296951 shutdown_connections 2024-09-16T03:59:42.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.129+0000 7f54a0e75640 1 -- 172.21.15.42:0/798296951 wait complete. 2024-09-16T03:59:42.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.129+0000 7f54a0e75640 10 monclient: build_initial_monmap 2024-09-16T03:59:42.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.129+0000 7f54a0e75640 10 monclient: monmap: 2024-09-16T03:59:42.148 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:59:42.148 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:59:42.148 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:59:42.148 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:59:42.148 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:59:42.148 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:59:42.148 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:59:42.148 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:59:42.148 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:59:42.148 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:59:42.148 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.129+0000 7f54a0e75640 1 librados: starting msgr at 2024-09-16T03:59:42.148 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.129+0000 7f54a0e75640 1 librados: starting objecter 2024-09-16T03:59:42.148 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.133+0000 7f54a0e75640 1 Processor -- start 2024-09-16T03:59:42.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.133+0000 7f54a0e75640 1 -- start start 2024-09-16T03:59:42.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.133+0000 7f54a0e75640 1 librados: setting wanted keys 2024-09-16T03:59:42.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.133+0000 7f54a0e75640 1 librados: calling monclient init 2024-09-16T03:59:42.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.133+0000 7f54a0e75640 10 monclient: init 2024-09-16T03:59:42.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.133+0000 7f54a0e75640 10 monclient: _reopen_session rank -1 2024-09-16T03:59:42.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.133+0000 7f54a0e75640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T03:59:42.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.133+0000 7f54a0e75640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f549c06db90 0x7f549c0a1810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:59:42.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.133+0000 7f54a0e75640 10 monclient(hunting): picked mon.noname-b con 0x7f549c06db90 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:59:42.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.133+0000 7f54a0e75640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f549c1340c0 0x7f549c0a1d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:59:42.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.133+0000 7f54a0e75640 10 monclient(hunting): picked mon.noname-c con 0x7f549c1340c0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:59:42.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.133+0000 7f54a0e75640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f549c13c6d0 0x7f549c0a2290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:59:42.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.133+0000 7f54a0e75640 10 monclient(hunting): picked mon.noname-a con 0x7f549c13c6d0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:59:42.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.133+0000 7f54a0e75640 10 monclient(hunting): start opening mon connection 2024-09-16T03:59:42.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.133+0000 7f54a0e75640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f549c141820 con 0x7f549c13c6d0 2024-09-16T03:59:42.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.133+0000 7f54a0e75640 10 monclient(hunting): start opening mon connection 2024-09-16T03:59:42.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.133+0000 7f54a0e75640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f549c1419a0 con 0x7f549c06db90 2024-09-16T03:59:42.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.133+0000 7f54a0e75640 10 monclient(hunting): start opening mon connection 2024-09-16T03:59:42.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.133+0000 7f54a0e75640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f549c141b20 con 0x7f549c1340c0 2024-09-16T03:59:42.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.133+0000 7f54a0e75640 10 monclient(hunting): _renew_subs 2024-09-16T03:59:42.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.133+0000 7f54a0e75640 10 monclient(hunting): authenticate will time out at 2029.104622s 2024-09-16T03:59:42.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.133+0000 7f5499d9b640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f549c1340c0 0x7f549c0a1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:59:42.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.133+0000 7f549ad9d640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f549c13c6d0 0x7f549c0a2290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:59:42.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.133+0000 7f549ad9d640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f549c13c6d0 0x7f549c0a2290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:58806/0 (socket says 172.21.15.42:58806) 2024-09-16T03:59:42.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.133+0000 7f549ad9d640 1 -- 172.21.15.42:0/2339986332 learned_addr learned my addr 172.21.15.42:0/2339986332 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:59:42.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.133+0000 7f549ad9d640 10 monclient(hunting): get_auth_request con 0x7f549c13c6d0 auth_method 0 2024-09-16T03:59:42.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.133+0000 7f549ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:59:42.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.133+0000 7f549ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:59:42.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.133+0000 7f549ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:59:42.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.133+0000 7f5499d9b640 10 monclient(hunting): get_auth_request con 0x7f549c1340c0 auth_method 0 2024-09-16T03:59:42.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.133+0000 7f5499d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:59:42.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.133+0000 7f5499d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:59:42.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.133+0000 7f5499d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:59:42.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.133+0000 7f549ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:59:42.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.133+0000 7f549ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:59:42.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.133+0000 7f549ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:59:42.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.133+0000 7f5499d9b640 1 -- 172.21.15.42:0/2339986332 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f549c1340c0 msgr2=0x7f549c0a1d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-16T03:59:42.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.133+0000 7f5499d9b640 1 -- 172.21.15.42:0/2339986332 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f549c1340c0 msgr2=0x7f549c0a1d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T03:59:42.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.133+0000 7f5499d9b640 1 --2- 172.21.15.42:0/2339986332 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f549c1340c0 0x7f549c0a1d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:59:42.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.133+0000 7f5499d9b640 1 --2- 172.21.15.42:0/2339986332 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f549c1340c0 0x7f549c0a1d50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:59:42.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.133+0000 7f549a59c640 1 --2- 172.21.15.42:0/2339986332 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f549c06db90 0x7f549c0a1810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:59:42.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.133+0000 7f549a59c640 10 monclient(hunting): get_auth_request con 0x7f549c06db90 auth_method 0 2024-09-16T03:59:42.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.133+0000 7f549a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:59:42.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.133+0000 7f549a59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:59:42.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.133+0000 7f549a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:59:42.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.133+0000 7f549ad9d640 10 monclient(hunting): handle_auth_done global_id 825153 payload 824 2024-09-16T03:59:42.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.133+0000 7f549ad9d640 1 -- 172.21.15.42:0/2339986332 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f549c1340c0 msgr2=0x7f549c0a1d50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:59:42.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.133+0000 7f549ad9d640 1 --2- 172.21.15.42:0/2339986332 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f549c1340c0 0x7f549c0a1d50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:59:42.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.133+0000 7f549ad9d640 1 -- 172.21.15.42:0/2339986332 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f549c06db90 msgr2=0x7f549c0a1810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:59:42.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.133+0000 7f549ad9d640 1 --2- 172.21.15.42:0/2339986332 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f549c06db90 0x7f549c0a1810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:59:42.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.133+0000 7f549ad9d640 10 monclient: _finish_hunting 0 2024-09-16T03:59:42.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.133+0000 7f549ad9d640 1 monclient: found mon.noname-a 2024-09-16T03:59:42.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.133+0000 7f549ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:59:42.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.133+0000 7f549ad9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:59:42.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.133+0000 7f549ad9d640 1 -- 172.21.15.42:0/2339986332 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f549c133390 con 0x7f549c13c6d0 2024-09-16T03:59:42.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.133+0000 7f549ad9d640 1 --2- 172.21.15.42:0/2339986332 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f549c13c6d0 0x7f549c0a2290 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f54940b0040 tx=0x7f54940195d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:59:42.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.133+0000 7f548affd640 10 client.?.objecter ms_handle_connect 0x7f549c13c6d0 2024-09-16T03:59:42.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.133+0000 7f548affd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:59:42.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.133+0000 7f548affd640 1 -- 172.21.15.42:0/2339986332 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54940b2050 con 0x7f549c13c6d0 2024-09-16T03:59:42.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.133+0000 7f548affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:59:42.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.133+0000 7f548affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:59:42.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.133+0000 7f548affd640 10 monclient: dump: 2024-09-16T03:59:42.153 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:59:42.153 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:59:42.153 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:42.153 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:42.153 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:59:42.153 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:59:42.153 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:59:42.154 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:59:42.154 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:59:42.154 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:59:42.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.133+0000 7f548affd640 10 monclient: _finish_auth 0 2024-09-16T03:59:42.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.133+0000 7f548affd640 10 monclient: _check_auth_tickets 2024-09-16T03:59:42.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.133+0000 7f548affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:59:42.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.133+0000 7f54a0e75640 5 monclient: authenticate success, global_id 825153 2024-09-16T03:59:42.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.133+0000 7f54a0e75640 10 monclient: _renew_subs 2024-09-16T03:59:42.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.133+0000 7f548affd640 1 -- 172.21.15.42:0/2339986332 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f54940be070 con 0x7f549c13c6d0 2024-09-16T03:59:42.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.133+0000 7f54a0e75640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:59:42.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.133+0000 7f54a0e75640 1 -- 172.21.15.42:0/2339986332 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f549c132ed0 con 0x7f549c13c6d0 2024-09-16T03:59:42.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.133+0000 7f548affd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:59:42.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.133+0000 7f548affd640 1 -- 172.21.15.42:0/2339986332 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54940adc40 con 0x7f549c13c6d0 2024-09-16T03:59:42.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.133+0000 7f548affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:59:42.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.133+0000 7f548affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:59:42.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.133+0000 7f548affd640 10 monclient: dump: 2024-09-16T03:59:42.155 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:59:42.155 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:59:42.155 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:42.155 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:42.155 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:59:42.155 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:59:42.155 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:59:42.155 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:59:42.155 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:59:42.155 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:59:42.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.133+0000 7f5498d99640 4 set_mon_vals no callback set 2024-09-16T03:59:42.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.133+0000 7f54a0e75640 10 client.825153.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:59:42.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.133+0000 7f54a0e75640 10 monclient: _renew_subs 2024-09-16T03:59:42.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.133+0000 7f54a0e75640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:59:42.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.133+0000 7f54a0e75640 1 -- 172.21.15.42:0/2339986332 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f549c136390 con 0x7f549c13c6d0 2024-09-16T03:59:42.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.133+0000 7f54a0e75640 1 librados: init done 2024-09-16T03:59:42.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.133+0000 7f54a0e75640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:59:42.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.133+0000 7f54a0e75640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:59:42.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.133+0000 7f54a0e75640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:59:42.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.133+0000 7f54a0e75640 1 -- 172.21.15.42:0/2339986332 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f549c0a37f0 con 0x7f549c13c6d0 2024-09-16T03:59:42.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.133+0000 7f548affd640 1 -- 172.21.15.42:0/2339986332 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f54940c3460 con 0x7f549c13c6d0 2024-09-16T03:59:42.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.133+0000 7f548affd640 1 --2- 172.21.15.42:0/2339986332 >> v2:172.21.15.154:6812/254845342 conn(0x7f547c037810 0x7f547c039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:59:42.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.133+0000 7f548affd640 1 -- 172.21.15.42:0/2339986332 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(540..540 src has 451..540) ==== 6168+0+0 (secure 0 0 0) 0x7f54940e46c0 con 0x7f549c13c6d0 2024-09-16T03:59:42.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.133+0000 7f548affd640 10 client.825153.objecter ms_dispatch 0x7f549c000ba0 osd_map(540..540 src has 451..540) 2024-09-16T03:59:42.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.133+0000 7f548affd640 3 client.825153.objecter handle_osd_map got epochs [540,540] > 0 2024-09-16T03:59:42.157 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.133+0000 7f548affd640 3 client.825153.objecter handle_osd_map decoding full epoch 540 2024-09-16T03:59:42.157 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.133+0000 7f548affd640 20 client.825153.objecter dump_active .. 0 homeless 2024-09-16T03:59:42.157 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.133+0000 7f549a59c640 1 --2- 172.21.15.42:0/2339986332 >> v2:172.21.15.154:6812/254845342 conn(0x7f547c037810 0x7f547c039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:59:42.157 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.133+0000 7f549a59c640 10 monclient: get_auth_request con 0x7f547c037810 auth_method 0 2024-09-16T03:59:42.157 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.137+0000 7f549a59c640 1 --2- 172.21.15.42:0/2339986332 >> v2:172.21.15.154:6812/254845342 conn(0x7f547c037810 0x7f547c039cd0 secure :-1 s=READY pgs=3731 cs=0 l=1 rev1=1 crypto rx=0x7f5490000f70 tx=0x7f5490009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:59:42.157 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.137+0000 7f548affd640 10 client.825153.objecter ms_handle_connect 0x7f547c037810 2024-09-16T03:59:42.157 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.137+0000 7f548affd640 1 -- 172.21.15.42:0/2339986332 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f5494016540 con 0x7f549c13c6d0 2024-09-16T03:59:42.157 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.137+0000 7f548affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:59:42.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.137+0000 7f548affd640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:59:42.369 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.365+0000 7f54a0e75640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:59:42.369 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.365+0000 7f54a0e75640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:59:42.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.365+0000 7f54a0e75640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:59:42.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.365+0000 7f54a0e75640 1 -- 172.21.15.42:0/2339986332 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f549c134f40 con 0x7f549c13c6d0 2024-09-16T03:59:42.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.365+0000 7f548affd640 1 -- 172.21.15.42:0/2339986332 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7f54940b6e20 con 0x7f549c13c6d0 2024-09-16T03:59:42.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.365+0000 7f548affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:59:42.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.365+0000 7f548affd640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:59:42.371 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":160,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:59:42.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.365+0000 7f54a0e75640 10 librados: watch_flush enter 2024-09-16T03:59:42.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.365+0000 7f54a0e75640 10 librados: watch_flush exit 2024-09-16T03:59:42.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.365+0000 7f54a0e75640 20 client.825153.objecter shutdown clearing up homeless session... 2024-09-16T03:59:42.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.365+0000 7f54a0e75640 10 client.825153.objecter successfully canceled tick 2024-09-16T03:59:42.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.365+0000 7f54a0e75640 1 -- 172.21.15.42:0/2339986332 >> v2:172.21.15.154:6812/254845342 conn(0x7f547c037810 msgr2=0x7f547c039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:59:42.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.365+0000 7f54a0e75640 1 --2- 172.21.15.42:0/2339986332 >> v2:172.21.15.154:6812/254845342 conn(0x7f547c037810 0x7f547c039cd0 secure :-1 s=READY pgs=3731 cs=0 l=1 rev1=1 crypto rx=0x7f5490000f70 tx=0x7f5490009040 comp rx=0 tx=0).stop 2024-09-16T03:59:42.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.365+0000 7f54a0e75640 10 monclient: shutdown 2024-09-16T03:59:42.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.365+0000 7f54a0e75640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:59:42.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.365+0000 7f54a0e75640 1 -- 172.21.15.42:0/2339986332 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f549c13c6d0 msgr2=0x7f549c0a2290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:59:42.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.365+0000 7f54a0e75640 1 --2- 172.21.15.42:0/2339986332 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f549c13c6d0 0x7f549c0a2290 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f54940b0040 tx=0x7f54940195d0 comp rx=0 tx=0).stop 2024-09-16T03:59:42.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.365+0000 7f54a0e75640 1 -- 172.21.15.42:0/2339986332 shutdown_connections 2024-09-16T03:59:42.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.365+0000 7f54a0e75640 1 --2- 172.21.15.42:0/2339986332 >> v2:172.21.15.154:6812/254845342 conn(0x7f547c037810 0x7f547c039cd0 unknown :-1 s=CLOSED pgs=3731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:59:42.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.365+0000 7f54a0e75640 1 --2- 172.21.15.42:0/2339986332 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f549c13c6d0 0x7f549c0a2290 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:59:42.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.365+0000 7f54a0e75640 1 --2- 172.21.15.42:0/2339986332 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f549c1340c0 0x7f549c0a1d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:59:42.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.365+0000 7f54a0e75640 1 --2- 172.21.15.42:0/2339986332 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f549c06db90 0x7f549c0a1810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:59:42.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.365+0000 7f54a0e75640 1 -- 172.21.15.42:0/2339986332 >> 172.21.15.42:0/2339986332 conn(0x7f549c0a9fb0 msgr2=0x7f549c0a7a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:59:42.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.365+0000 7f54a0e75640 1 -- 172.21.15.42:0/2339986332 shutdown_connections 2024-09-16T03:59:42.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.365+0000 7f54a0e75640 1 -- 172.21.15.42:0/2339986332 wait complete. 2024-09-16T03:59:42.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.365+0000 7f54a0e75640 1 librados: shutdown 2024-09-16T03:59:42.388 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T03:59:42.388 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T03:59:42.389 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T03:59:42.514 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.509+0000 7f39e6c5f640 10 monclient: get_monmap_and_config 2024-09-16T03:59:42.514 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.509+0000 7f39e6c5f640 10 monclient: build_initial_monmap 2024-09-16T03:59:42.514 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.509+0000 7f39e6c5f640 10 monclient: monmap: 2024-09-16T03:59:42.514 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:59:42.514 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:59:42.514 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:59:42.512464+0000 2024-09-16T03:59:42.514 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:59:42.512464+0000 2024-09-16T03:59:42.514 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:59:42.514 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:59:42.514 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:59:42.514 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:59:42.515 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:59:42.515 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:59:42.515 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:59:42.515 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:59:42.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.509+0000 7f39e6c5f640 1 Processor -- start 2024-09-16T03:59:42.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.509+0000 7f39e6c5f640 1 -- start start 2024-09-16T03:59:42.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.509+0000 7f39e6c5f640 10 monclient: init 2024-09-16T03:59:42.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.509+0000 7f39e6c5f640 10 monclient: _reopen_session rank -1 2024-09-16T03:59:42.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.509+0000 7f39e6c5f640 10 monclient: _add_conns ranks=[2,3,4,0,1] 2024-09-16T03:59:42.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.509+0000 7f39e6c5f640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f39e00a3450 0x7f39e00a3830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:59:42.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.509+0000 7f39e6c5f640 10 monclient(hunting): picked mon.noname-b con 0x7f39e00a3450 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:59:42.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.509+0000 7f39e6c5f640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f39e00a0280 0x7f39e00a3d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:59:42.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.509+0000 7f39e6c5f640 10 monclient(hunting): picked mon.noname-a con 0x7f39e00a0280 addr v2:172.21.15.42:3300/0 2024-09-16T03:59:42.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.509+0000 7f39e6c5f640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f39e00a0e50 0x7f39e0137220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:59:42.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.509+0000 7f39e6c5f640 10 monclient(hunting): picked mon.noname-c con 0x7f39e00a0e50 addr v2:172.21.15.154:3300/0 2024-09-16T03:59:42.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.509+0000 7f39e6c5f640 10 monclient(hunting): start opening mon connection 2024-09-16T03:59:42.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.509+0000 7f39e6c5f640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f39e0072490 con 0x7f39e00a0280 2024-09-16T03:59:42.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.509+0000 7f39e6c5f640 10 monclient(hunting): start opening mon connection 2024-09-16T03:59:42.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.509+0000 7f39e6c5f640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f39e00a0700 con 0x7f39e00a0e50 2024-09-16T03:59:42.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.509+0000 7f39e6c5f640 10 monclient(hunting): start opening mon connection 2024-09-16T03:59:42.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.509+0000 7f39e6c5f640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f39e00a42b0 con 0x7f39e00a3450 2024-09-16T03:59:42.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.509+0000 7f39e6c5f640 10 monclient(hunting): _renew_subs 2024-09-16T03:59:42.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.509+0000 7f39e6c5f640 10 monclient(hunting): authenticate will time out at 2029.481824s 2024-09-16T03:59:42.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.509+0000 7f39e49fb640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f39e00a3450 0x7f39e00a3830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:59:42.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.509+0000 7f39e49fb640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f39e00a3450 0x7f39e00a3830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:40568/0 (socket says 172.21.15.42:40568) 2024-09-16T03:59:42.521 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.509+0000 7f39e49fb640 1 -- 172.21.15.42:0/2318479593 learned_addr learned my addr 172.21.15.42:0/2318479593 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:59:42.521 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.509+0000 7f39e49fb640 10 monclient(hunting): get_auth_request con 0x7f39e00a3450 auth_method 0 2024-09-16T03:59:42.521 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.509+0000 7f39e49fb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:59:42.521 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.509+0000 7f39e49fb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:59:42.521 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.509+0000 7f39e49fb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:59:42.521 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.509+0000 7f39dffff640 1 --2- 172.21.15.42:0/2318479593 >> v2:172.21.15.42:3300/0 conn(0x7f39e00a0280 0x7f39e00a3d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:59:42.521 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.509+0000 7f39dffff640 10 monclient(hunting): get_auth_request con 0x7f39e00a0280 auth_method 0 2024-09-16T03:59:42.521 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.509+0000 7f39dffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:59:42.521 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.509+0000 7f39dffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:59:42.521 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.509+0000 7f39dffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:59:42.521 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.509+0000 7f39e49fb640 1 -- 172.21.15.42:0/2318479593 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f39e00a3450 msgr2=0x7f39e00a3830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T03:59:42.521 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.509+0000 7f39e49fb640 1 -- 172.21.15.42:0/2318479593 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f39e00a3450 msgr2=0x7f39e00a3830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T03:59:42.522 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.509+0000 7f39e49fb640 1 --2- 172.21.15.42:0/2318479593 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f39e00a3450 0x7f39e00a3830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:59:42.522 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.509+0000 7f39e49fb640 1 --2- 172.21.15.42:0/2318479593 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f39e00a3450 0x7f39e00a3830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:59:42.522 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.509+0000 7f39dffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:59:42.522 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.509+0000 7f39dffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:59:42.522 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.509+0000 7f39dffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:59:42.522 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.509+0000 7f39e51fc640 1 --2- 172.21.15.42:0/2318479593 >> v2:172.21.15.154:3300/0 conn(0x7f39e00a0e50 0x7f39e0137220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:59:42.522 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.509+0000 7f39e51fc640 10 monclient(hunting): get_auth_request con 0x7f39e00a0e50 auth_method 0 2024-09-16T03:59:42.522 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.509+0000 7f39e51fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:59:42.522 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.509+0000 7f39e51fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:59:42.522 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.509+0000 7f39e51fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:59:42.522 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.509+0000 7f39dffff640 10 monclient(hunting): handle_auth_done global_id 825156 payload 274 2024-09-16T03:59:42.522 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.509+0000 7f39dffff640 1 -- 172.21.15.42:0/2318479593 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f39e00a3450 msgr2=0x7f39e00a3830 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T03:59:42.522 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.509+0000 7f39dffff640 1 --2- 172.21.15.42:0/2318479593 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f39e00a3450 0x7f39e00a3830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:59:42.522 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.509+0000 7f39dffff640 1 -- 172.21.15.42:0/2318479593 >> v2:172.21.15.154:3300/0 conn(0x7f39e00a0e50 msgr2=0x7f39e0137220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:59:42.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.509+0000 7f39dffff640 1 --2- 172.21.15.42:0/2318479593 >> v2:172.21.15.154:3300/0 conn(0x7f39e00a0e50 0x7f39e0137220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:59:42.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.509+0000 7f39dffff640 10 monclient: _finish_hunting 0 2024-09-16T03:59:42.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.509+0000 7f39dffff640 1 monclient: found mon.noname-a 2024-09-16T03:59:42.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.509+0000 7f39dffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:59:42.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.509+0000 7f39dffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:59:42.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.509+0000 7f39dffff640 1 -- 172.21.15.42:0/2318479593 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39e00a1a80 con 0x7f39e00a0280 2024-09-16T03:59:42.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.513+0000 7f39dffff640 1 --2- 172.21.15.42:0/2318479593 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f39e00a0280 0x7f39e00a3d70 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f39e00a44f0 tx=0x7f39cc01c080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29a8f60080ba0bad server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:59:42.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.513+0000 7f39df7fe640 1 -- 172.21.15.42:0/2318479593 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39cc0b1070 con 0x7f39e00a0280 2024-09-16T03:59:42.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.513+0000 7f39df7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:59:42.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.513+0000 7f39df7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:59:42.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.513+0000 7f39df7fe640 10 monclient: dump: 2024-09-16T03:59:42.523 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:59:42.523 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:59:42.524 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:42.524 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:42.524 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:59:42.524 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:59:42.524 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:59:42.524 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:59:42.524 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:59:42.524 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:59:42.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.513+0000 7f39df7fe640 10 monclient: _finish_auth 0 2024-09-16T03:59:42.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.513+0000 7f39df7fe640 10 monclient: _check_auth_tickets 2024-09-16T03:59:42.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.513+0000 7f39df7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:59:42.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.513+0000 7f39df7fe640 1 -- 172.21.15.42:0/2318479593 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f39cc0a4df0 con 0x7f39e00a0280 2024-09-16T03:59:42.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.513+0000 7f39e6c5f640 5 monclient: authenticate success, global_id 825156 2024-09-16T03:59:42.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.513+0000 7f39e6c5f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:59:42.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.513+0000 7f39df7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:59:42.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.513+0000 7f39e6c5f640 10 monclient: get_monmap_and_config success 2024-09-16T03:59:42.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.513+0000 7f39df7fe640 1 -- 172.21.15.42:0/2318479593 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39cc0ac660 con 0x7f39e00a0280 2024-09-16T03:59:42.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.513+0000 7f39df7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:59:42.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.513+0000 7f39e6c5f640 4 set_mon_vals no callback set 2024-09-16T03:59:42.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.513+0000 7f39df7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:59:42.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.513+0000 7f39df7fe640 10 monclient: dump: 2024-09-16T03:59:42.525 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:59:42.525 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:59:42.525 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:42.525 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:42.526 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:59:42.526 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:59:42.526 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:59:42.526 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:59:42.526 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:59:42.526 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:59:42.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.513+0000 7f39e6c5f640 10 monclient: shutdown 2024-09-16T03:59:42.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.513+0000 7f39e6c5f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:59:42.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.513+0000 7f39e6c5f640 1 -- 172.21.15.42:0/2318479593 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f39e00a0280 msgr2=0x7f39e00a3d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:59:42.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.513+0000 7f39e6c5f640 1 --2- 172.21.15.42:0/2318479593 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f39e00a0280 0x7f39e00a3d70 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f39e00a44f0 tx=0x7f39cc01c080 comp rx=0 tx=0).stop 2024-09-16T03:59:42.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.513+0000 7f39e6c5f640 1 -- 172.21.15.42:0/2318479593 shutdown_connections 2024-09-16T03:59:42.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.513+0000 7f39e6c5f640 1 --2- 172.21.15.42:0/2318479593 >> v2:172.21.15.154:3300/0 conn(0x7f39e00a0e50 0x7f39e0137220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:59:42.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.513+0000 7f39e6c5f640 1 --2- 172.21.15.42:0/2318479593 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f39e00a0280 0x7f39e00a3d70 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:59:42.527 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.513+0000 7f39e6c5f640 1 --2- 172.21.15.42:0/2318479593 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f39e00a3450 0x7f39e00a3830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:59:42.527 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.513+0000 7f39e6c5f640 1 -- 172.21.15.42:0/2318479593 >> 172.21.15.42:0/2318479593 conn(0x7f39e00a4540 msgr2=0x7f39e00a4950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:59:42.527 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.513+0000 7f39e6c5f640 1 -- 172.21.15.42:0/2318479593 shutdown_connections 2024-09-16T03:59:42.527 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.513+0000 7f39e6c5f640 1 -- 172.21.15.42:0/2318479593 wait complete. 2024-09-16T03:59:42.533 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.513+0000 7f39e6c5f640 10 monclient: build_initial_monmap 2024-09-16T03:59:42.533 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.513+0000 7f39e6c5f640 10 monclient: monmap: 2024-09-16T03:59:42.533 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:59:42.533 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:59:42.534 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:59:42.534 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:59:42.534 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:59:42.534 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:59:42.534 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:59:42.534 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:59:42.534 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:59:42.534 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:59:42.534 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.513+0000 7f39e6c5f640 1 librados: starting msgr at 2024-09-16T03:59:42.534 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.513+0000 7f39e6c5f640 1 librados: starting objecter 2024-09-16T03:59:42.534 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.513+0000 7f39e6c5f640 1 Processor -- start 2024-09-16T03:59:42.534 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.513+0000 7f39e6c5f640 1 -- start start 2024-09-16T03:59:42.534 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.513+0000 7f39e6c5f640 1 librados: setting wanted keys 2024-09-16T03:59:42.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.513+0000 7f39e6c5f640 1 librados: calling monclient init 2024-09-16T03:59:42.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.513+0000 7f39e6c5f640 10 monclient: init 2024-09-16T03:59:42.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.513+0000 7f39e6c5f640 10 monclient: _reopen_session rank -1 2024-09-16T03:59:42.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.513+0000 7f39e6c5f640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:59:42.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.513+0000 7f39e6c5f640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f39e00a0280 0x7f39e01cf830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:59:42.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.513+0000 7f39e6c5f640 10 monclient(hunting): picked mon.noname-c con 0x7f39e00a0280 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:59:42.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.513+0000 7f39e6c5f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f39e00a0e50 0x7f39e0136760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:59:42.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.513+0000 7f39e6c5f640 10 monclient(hunting): picked mon.noname-b con 0x7f39e00a0e50 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:59:42.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.513+0000 7f39e6c5f640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f39e00a3450 0x7f39e011e310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:59:42.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.513+0000 7f39e6c5f640 10 monclient(hunting): picked mon.noname-a con 0x7f39e00a3450 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:59:42.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.513+0000 7f39e6c5f640 10 monclient(hunting): start opening mon connection 2024-09-16T03:59:42.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.513+0000 7f39e6c5f640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f39e01410b0 con 0x7f39e00a3450 2024-09-16T03:59:42.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.513+0000 7f39e6c5f640 10 monclient(hunting): start opening mon connection 2024-09-16T03:59:42.536 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.513+0000 7f39e6c5f640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f39e0072720 con 0x7f39e00a0e50 2024-09-16T03:59:42.536 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.513+0000 7f39e6c5f640 10 monclient(hunting): start opening mon connection 2024-09-16T03:59:42.536 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.513+0000 7f39e6c5f640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f39e0141230 con 0x7f39e00a0280 2024-09-16T03:59:42.536 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.513+0000 7f39e6c5f640 10 monclient(hunting): _renew_subs 2024-09-16T03:59:42.536 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.513+0000 7f39e6c5f640 10 monclient(hunting): authenticate will time out at 2029.486925s 2024-09-16T03:59:42.536 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.513+0000 7f39e49fb640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f39e00a0280 0x7f39e01cf830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:59:42.536 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.513+0000 7f39e51fc640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f39e00a3450 0x7f39e011e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:59:42.536 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.513+0000 7f39e49fb640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f39e00a0280 0x7f39e01cf830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:40580/0 (socket says 172.21.15.42:40580) 2024-09-16T03:59:42.536 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.513+0000 7f39e49fb640 1 -- 172.21.15.42:0/3349464524 learned_addr learned my addr 172.21.15.42:0/3349464524 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:59:42.536 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.513+0000 7f39e49fb640 10 monclient(hunting): get_auth_request con 0x7f39e00a0280 auth_method 0 2024-09-16T03:59:42.536 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.513+0000 7f39e49fb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:59:42.536 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.513+0000 7f39e49fb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:59:42.537 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.513+0000 7f39e49fb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:59:42.537 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.513+0000 7f39e51fc640 10 monclient(hunting): get_auth_request con 0x7f39e00a3450 auth_method 0 2024-09-16T03:59:42.537 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.513+0000 7f39e51fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:59:42.537 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.513+0000 7f39e51fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:59:42.537 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.513+0000 7f39e51fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:59:42.537 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.513+0000 7f39e51fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:59:42.537 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.513+0000 7f39e51fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:59:42.537 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.513+0000 7f39e49fb640 1 -- 172.21.15.42:0/3349464524 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f39e00a0280 msgr2=0x7f39e01cf830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-16T03:59:42.537 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.513+0000 7f39e51fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:59:42.537 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.513+0000 7f39e49fb640 1 -- 172.21.15.42:0/3349464524 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f39e00a0280 msgr2=0x7f39e01cf830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T03:59:42.537 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.513+0000 7f39e49fb640 1 --2- 172.21.15.42:0/3349464524 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f39e00a0280 0x7f39e01cf830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:59:42.537 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.513+0000 7f39e49fb640 1 --2- 172.21.15.42:0/3349464524 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f39e00a0280 0x7f39e01cf830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:59:42.537 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.513+0000 7f39dffff640 1 --2- 172.21.15.42:0/3349464524 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f39e00a0e50 0x7f39e0136760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:59:42.538 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.513+0000 7f39dffff640 10 monclient(hunting): get_auth_request con 0x7f39e00a0e50 auth_method 0 2024-09-16T03:59:42.538 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.513+0000 7f39dffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:59:42.538 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.513+0000 7f39dffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:59:42.538 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.513+0000 7f39dffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:59:42.538 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.513+0000 7f39e51fc640 10 monclient(hunting): handle_auth_done global_id 825159 payload 824 2024-09-16T03:59:42.538 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.513+0000 7f39e51fc640 1 -- 172.21.15.42:0/3349464524 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f39e00a0280 msgr2=0x7f39e01cf830 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:59:42.538 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.513+0000 7f39e51fc640 1 --2- 172.21.15.42:0/3349464524 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f39e00a0280 0x7f39e01cf830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:59:42.538 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.513+0000 7f39e51fc640 1 -- 172.21.15.42:0/3349464524 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f39e00a0e50 msgr2=0x7f39e0136760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:59:42.538 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.513+0000 7f39e51fc640 1 --2- 172.21.15.42:0/3349464524 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f39e00a0e50 0x7f39e0136760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:59:42.538 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.513+0000 7f39e51fc640 10 monclient: _finish_hunting 0 2024-09-16T03:59:42.538 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.513+0000 7f39e51fc640 1 monclient: found mon.noname-a 2024-09-16T03:59:42.538 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.513+0000 7f39e51fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:59:42.538 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.513+0000 7f39e51fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:59:42.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.513+0000 7f39e51fc640 1 -- 172.21.15.42:0/3349464524 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39e0121e20 con 0x7f39e00a3450 2024-09-16T03:59:42.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.513+0000 7f39e51fc640 1 --2- 172.21.15.42:0/3349464524 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f39e00a3450 0x7f39e011e310 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f39d8007d90 tx=0x7f39d800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:59:42.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.513+0000 7f39dd7fa640 10 client.?.objecter ms_handle_connect 0x7f39e00a3450 2024-09-16T03:59:42.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.513+0000 7f39dd7fa640 10 client.?.objecter resend_mon_ops 2024-09-16T03:59:42.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.513+0000 7f39dd7fa640 1 -- 172.21.15.42:0/3349464524 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39d800ed70 con 0x7f39e00a3450 2024-09-16T03:59:42.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.513+0000 7f39dd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:59:42.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.513+0000 7f39dd7fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:59:42.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.513+0000 7f39dd7fa640 10 monclient: dump: 2024-09-16T03:59:42.539 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:59:42.539 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:59:42.539 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:42.539 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:42.540 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:59:42.540 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:59:42.540 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:59:42.540 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:59:42.540 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:59:42.540 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:59:42.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.513+0000 7f39dd7fa640 10 monclient: _finish_auth 0 2024-09-16T03:59:42.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.513+0000 7f39dd7fa640 10 monclient: _check_auth_tickets 2024-09-16T03:59:42.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.513+0000 7f39dd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:59:42.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.513+0000 7f39dd7fa640 1 -- 172.21.15.42:0/3349464524 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f39d80039a0 con 0x7f39e00a3450 2024-09-16T03:59:42.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.513+0000 7f39e6c5f640 5 monclient: authenticate success, global_id 825159 2024-09-16T03:59:42.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.513+0000 7f39e6c5f640 10 monclient: _renew_subs 2024-09-16T03:59:42.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.513+0000 7f39e6c5f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:59:42.541 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.513+0000 7f39e6c5f640 1 -- 172.21.15.42:0/3349464524 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39e01cfd70 con 0x7f39e00a3450 2024-09-16T03:59:42.541 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.513+0000 7f39dd7fa640 10 monclient: handle_config config(0 keys) 2024-09-16T03:59:42.541 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.513+0000 7f39dd7fa640 1 -- 172.21.15.42:0/3349464524 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39d8003d90 con 0x7f39e00a3450 2024-09-16T03:59:42.541 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.513+0000 7f39dd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:59:42.541 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.513+0000 7f39dd7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:59:42.541 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.513+0000 7f39dd7fa640 10 monclient: dump: 2024-09-16T03:59:42.541 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:59:42.541 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:59:42.541 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:42.541 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:42.541 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:59:42.541 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:59:42.542 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:59:42.542 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:59:42.542 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:59:42.542 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:59:42.542 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.513+0000 7f39e6c5f640 10 client.825159.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:59:42.542 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.513+0000 7f39e6c5f640 10 monclient: _renew_subs 2024-09-16T03:59:42.542 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.513+0000 7f39e6c5f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:59:42.542 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.513+0000 7f39e6c5f640 1 -- 172.21.15.42:0/3349464524 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39e00a71b0 con 0x7f39e00a3450 2024-09-16T03:59:42.542 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.513+0000 7f39e6c5f640 1 librados: init done 2024-09-16T03:59:42.542 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.513+0000 7f39de7fc640 4 set_mon_vals no callback set 2024-09-16T03:59:42.542 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.517+0000 7f39e6c5f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:59:42.542 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.517+0000 7f39e6c5f640 1 --2- 172.21.15.42:0/3349464524 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f39e0136ca0 0x7f39e011e850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:59:42.542 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.517+0000 7f39e6c5f640 10 monclient(hunting): start opening mon connection 2024-09-16T03:59:42.543 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.517+0000 7f39e6c5f640 1 -- 172.21.15.42:0/3349464524 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f39e0121860 con 0x7f39e0136ca0 2024-09-16T03:59:42.543 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.517+0000 7f39e49fb640 1 --2- 172.21.15.42:0/3349464524 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f39e0136ca0 0x7f39e011e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:59:42.543 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.517+0000 7f39e49fb640 10 monclient: get_auth_request con 0x7f39e0136ca0 auth_method 0 2024-09-16T03:59:42.543 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.517+0000 7f39e49fb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:59:42.543 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.517+0000 7f39e49fb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:59:42.543 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.517+0000 7f39e49fb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:59:42.543 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.517+0000 7f39dd7fa640 1 -- 172.21.15.42:0/3349464524 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f39d801b020 con 0x7f39e00a3450 2024-09-16T03:59:42.543 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.517+0000 7f39e49fb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:59:42.543 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.517+0000 7f39e49fb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:59:42.543 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.517+0000 7f39e49fb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:59:42.543 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.517+0000 7f39dd7fa640 1 --2- 172.21.15.42:0/3349464524 >> v2:172.21.15.154:6812/254845342 conn(0x7f39c4037810 0x7f39c4039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:59:42.544 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.517+0000 7f39dd7fa640 1 -- 172.21.15.42:0/3349464524 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(540..540 src has 451..540) ==== 6168+0+0 (secure 0 0 0) 0x7f39d8049260 con 0x7f39e00a3450 2024-09-16T03:59:42.544 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.517+0000 7f39dd7fa640 10 client.825159.objecter ms_dispatch 0x7f39e0000ba0 osd_map(540..540 src has 451..540) 2024-09-16T03:59:42.544 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.517+0000 7f39dd7fa640 3 client.825159.objecter handle_osd_map got epochs [540,540] > 0 2024-09-16T03:59:42.544 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.517+0000 7f39dd7fa640 3 client.825159.objecter handle_osd_map decoding full epoch 540 2024-09-16T03:59:42.544 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.517+0000 7f39e49fb640 10 monclient(hunting): handle_auth_done global_id 825162 payload 824 2024-09-16T03:59:42.544 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.517+0000 7f39e49fb640 1 -- 172.21.15.42:0/3349464524 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f39e0137220 con 0x7f39e0136ca0 2024-09-16T03:59:42.544 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.517+0000 7f39dd7fa640 20 client.825159.objecter dump_active .. 0 homeless 2024-09-16T03:59:42.544 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.517+0000 7f39dd7fa640 10 client.825159.objecter ms_handle_connect 0x7f39e0136ca0 2024-09-16T03:59:42.544 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.517+0000 7f39dd7fa640 10 client.825159.objecter resend_mon_ops 2024-09-16T03:59:42.544 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.517+0000 7f39e49fb640 1 --2- 172.21.15.42:0/3349464524 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f39e0136ca0 0x7f39e011e850 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f39d40091d0 tx=0x7f39d4010c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:59:42.544 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.517+0000 7f39dffff640 1 --2- 172.21.15.42:0/3349464524 >> v2:172.21.15.154:6812/254845342 conn(0x7f39c4037810 0x7f39c4039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:59:42.544 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.517+0000 7f39dffff640 10 monclient: get_auth_request con 0x7f39c4037810 auth_method 0 2024-09-16T03:59:42.544 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.517+0000 7f39dd7fa640 1 -- 172.21.15.42:0/3349464524 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39d401a020 con 0x7f39e0136ca0 2024-09-16T03:59:42.545 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.517+0000 7f39dd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:59:42.545 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.517+0000 7f39dd7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:59:42.545 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.517+0000 7f39dd7fa640 10 monclient: dump: 2024-09-16T03:59:42.545 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:59:42.545 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:59:42.545 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:42.545 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:42.545 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:59:42.545 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:59:42.545 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:59:42.545 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:59:42.545 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:59:42.546 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:59:42.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.517+0000 7f39dd7fa640 1 -- 172.21.15.42:0/3349464524 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f39d4010e70 con 0x7f39e0136ca0 2024-09-16T03:59:42.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.517+0000 7f39dd7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:59:42.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.517+0000 7f39dd7fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:59:42.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.517+0000 7f39dd7fa640 1 -- 172.21.15.42:0/3349464524 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f39e0136ca0 msgr2=0x7f39e011e850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:59:42.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.517+0000 7f39dd7fa640 1 --2- 172.21.15.42:0/3349464524 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f39e0136ca0 0x7f39e011e850 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f39d40091d0 tx=0x7f39d4010c20 comp rx=0 tx=0).stop 2024-09-16T03:59:42.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.517+0000 7f39dd7fa640 1 -- 172.21.15.42:0/3349464524 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f39e0136ca0 msgr2=0x7f39e011e850 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:59:42.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.517+0000 7f39dd7fa640 1 --2- 172.21.15.42:0/3349464524 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f39e0136ca0 0x7f39e011e850 secure :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f39d40091d0 tx=0x7f39d4010c20 comp rx=0 tx=0).stop 2024-09-16T03:59:42.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.517+0000 7f39dd7fa640 10 client.825159.objecter ms_handle_connect 0x7f39c4037810 2024-09-16T03:59:42.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.517+0000 7f39dffff640 1 --2- 172.21.15.42:0/3349464524 >> v2:172.21.15.154:6812/254845342 conn(0x7f39c4037810 0x7f39c4039cd0 secure :-1 s=READY pgs=3732 cs=0 l=1 rev1=1 crypto rx=0x7f39cc0ae000 tx=0x7f39cc008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:59:42.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.521+0000 7f39e6c5f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:59:42.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.521+0000 7f39e6c5f640 1 --2- 172.21.15.42:0/3349464524 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f39e011ed90 0x7f39e006f610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:59:42.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.521+0000 7f39e6c5f640 10 monclient(hunting): start opening mon connection 2024-09-16T03:59:42.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.521+0000 7f39e6c5f640 1 -- 172.21.15.42:0/3349464524 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f39e011f170 con 0x7f39e011ed90 2024-09-16T03:59:42.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.525+0000 7f39e49fb640 1 --2- 172.21.15.42:0/3349464524 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f39e011ed90 0x7f39e006f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:59:42.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.525+0000 7f39e49fb640 10 monclient: get_auth_request con 0x7f39e011ed90 auth_method 0 2024-09-16T03:59:42.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.525+0000 7f39e49fb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:59:42.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.525+0000 7f39e49fb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:59:42.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.525+0000 7f39e49fb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:59:42.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.525+0000 7f39e49fb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:59:42.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.525+0000 7f39e49fb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:59:42.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.525+0000 7f39e49fb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:59:42.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.525+0000 7f39e49fb640 10 monclient(hunting): handle_auth_done global_id 825165 payload 824 2024-09-16T03:59:42.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.525+0000 7f39e49fb640 1 -- 172.21.15.42:0/3349464524 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f39e0121c00 con 0x7f39e011ed90 2024-09-16T03:59:42.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.525+0000 7f39dd7fa640 10 client.825159.objecter ms_handle_connect 0x7f39e011ed90 2024-09-16T03:59:42.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.525+0000 7f39dd7fa640 10 client.825159.objecter resend_mon_ops 2024-09-16T03:59:42.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.525+0000 7f39e49fb640 1 --2- 172.21.15.42:0/3349464524 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f39e011ed90 0x7f39e006f610 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f39d40091d0 tx=0x7f39d40046d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:59:42.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.525+0000 7f39dd7fa640 1 -- 172.21.15.42:0/3349464524 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39d401a040 con 0x7f39e011ed90 2024-09-16T03:59:42.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.525+0000 7f39dd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:59:42.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.525+0000 7f39dd7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:59:42.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.525+0000 7f39dd7fa640 10 monclient: dump: 2024-09-16T03:59:42.548 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:59:42.548 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:59:42.548 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:42.548 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:42.548 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:59:42.548 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:59:42.548 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:59:42.549 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:59:42.549 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:59:42.549 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:59:42.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.525+0000 7f39dd7fa640 1 -- 172.21.15.42:0/3349464524 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4882 (secure 0 0 0) 0x7f39d402b430 con 0x7f39e011ed90 2024-09-16T03:59:42.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.525+0000 7f39dd7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:59:42.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.525+0000 7f39dd7fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:59:42.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.525+0000 7f39dd7fa640 1 -- 172.21.15.42:0/3349464524 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f39e011ed90 msgr2=0x7f39e006f610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:59:42.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.525+0000 7f39dd7fa640 1 --2- 172.21.15.42:0/3349464524 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f39e011ed90 0x7f39e006f610 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f39d40091d0 tx=0x7f39d40046d0 comp rx=0 tx=0).stop 2024-09-16T03:59:42.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.525+0000 7f39dd7fa640 1 -- 172.21.15.42:0/3349464524 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f39e011ed90 msgr2=0x7f39e006f610 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:59:42.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.525+0000 7f39dd7fa640 1 --2- 172.21.15.42:0/3349464524 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f39e011ed90 0x7f39e006f610 secure :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f39d40091d0 tx=0x7f39d40046d0 comp rx=0 tx=0).stop 2024-09-16T03:59:42.550 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:59:42.550 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:59:42.550 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:59:42.550 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "leader", 2024-09-16T03:59:42.550 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 160, 2024-09-16T03:59:42.550 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 472379, 2024-09-16T03:59:42.550 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:59:42.550 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:59:42.550 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T03:59:42.550 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T03:59:42.550 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:59:42.551 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 1, 2024-09-16T03:59:42.551 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:59:42.551 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:59:42.551 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:59:42.551 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:59:42.551 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:59:42.551 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:59:42.551 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:59:42.551 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:59:42.551 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:59:42.551 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:59:42.551 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:59:42.552 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:59:42.552 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:59:42.552 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:59:42.552 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:59:42.552 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:59:42.552 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:59:42.552 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:59:42.552 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:59:42.552 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:59:42.552 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:59:42.552 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:59:42.552 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:59:42.553 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:59:42.553 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:59:42.553 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:59:42.553 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:59:42.553 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:59:42.553 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:59:42.553 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:59:42.553 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:59:42.553 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:59:42.553 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:59:42.553 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:59:42.553 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:59:42.553 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:59:42.554 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:59:42.554 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:59:42.554 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:59:42.554 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:59:42.554 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:59:42.554 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:59:42.554 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:59:42.554 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:59:42.554 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:59:42.554 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:59:42.554 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:59:42.554 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:59:42.555 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:59:42.555 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:59:42.555 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:59:42.555 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:59:42.555 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:59:42.555 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:59:42.555 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:59:42.555 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:59:42.555 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:59:42.555 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:59:42.555 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:59:42.555 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:59:42.555 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:59:42.555 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:59:42.556 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:59:42.556 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:59:42.556 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:59:42.556 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:59:42.556 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:59:42.556 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:59:42.556 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:59:42.556 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:59:42.556 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:59:42.556 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:59:42.556 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:59:42.556 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:59:42.557 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:59:42.557 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:59:42.557 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:59:42.557 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:59:42.557 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:59:42.557 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:59:42.557 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:59:42.557 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:59:42.557 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:59:42.557 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:59:42.557 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:59:42.557 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:59:42.557 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:59:42.557 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:59:42.558 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:59:42.558 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:59:42.558 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:59:42.558 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:59:42.558 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:59:42.558 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:59:42.558 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:59:42.558 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:59:42.558 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:59:42.558 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:59:42.558 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:59:42.558 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:59:42.558 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:59:42.559 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:59:42.559 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:59:42.559 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:59:42.559 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:59:42.559 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:59:42.559 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:59:42.559 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:59:42.559 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:59:42.559 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:59:42.559 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:59:42.559 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:59:42.559 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:59:42.559 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:59:42.560 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:59:42.560 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:59:42.560 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:59:42.560 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:59:42.560 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:59:42.560 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:59:42.560 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:59:42.560 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:59:42.560 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:59:42.560 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:59:42.560 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:59:42.560 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:59:42.561 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:59:42.561 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:59:42.561 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:59:42.561 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:59:42.561 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:59:42.561 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:59:42.561 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:59:42.561 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:59:42.561 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:59:42.561 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:59:42.561 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:59:42.561 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:59:42.561 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:59:42.561 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:59:42.562 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:59:42.562 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T03:59:42.562 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:59:42.562 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:59:42.562 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:59:42.562 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 5 2024-09-16T03:59:42.562 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:59:42.562 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:59:42.562 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:59:42.562 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:59:42.562 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:59:42.562 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:59:42.563 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 7 2024-09-16T03:59:42.563 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:59:42.563 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:59:42.563 INFO:teuthology.orchestra.run.smithi042.stdout: "mgr": [ 2024-09-16T03:59:42.563 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:59:42.563 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:59:42.563 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:59:42.563 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:59:42.563 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:59:42.563 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:59:42.563 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:59:42.563 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:59:42.563 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:59:42.565 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.525+0000 7f39e6c5f640 10 librados: watch_flush enter 2024-09-16T03:59:42.565 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.525+0000 7f39e6c5f640 10 librados: watch_flush exit 2024-09-16T03:59:42.565 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.525+0000 7f39e6c5f640 20 client.825159.objecter shutdown clearing up homeless session... 2024-09-16T03:59:42.565 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.525+0000 7f39e6c5f640 10 client.825159.objecter successfully canceled tick 2024-09-16T03:59:42.565 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.525+0000 7f39e6c5f640 1 -- 172.21.15.42:0/3349464524 >> v2:172.21.15.154:6812/254845342 conn(0x7f39c4037810 msgr2=0x7f39c4039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:59:42.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.525+0000 7f39e6c5f640 1 --2- 172.21.15.42:0/3349464524 >> v2:172.21.15.154:6812/254845342 conn(0x7f39c4037810 0x7f39c4039cd0 secure :-1 s=READY pgs=3732 cs=0 l=1 rev1=1 crypto rx=0x7f39cc0ae000 tx=0x7f39cc008040 comp rx=0 tx=0).stop 2024-09-16T03:59:42.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.525+0000 7f39e6c5f640 10 monclient: shutdown 2024-09-16T03:59:42.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.525+0000 7f39e6c5f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:59:42.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.525+0000 7f39e6c5f640 1 -- 172.21.15.42:0/3349464524 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f39e00a3450 msgr2=0x7f39e011e310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:59:42.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.525+0000 7f39e51fc640 1 -- 172.21.15.42:0/3349464524 reap_dead start 2024-09-16T03:59:42.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.525+0000 7f39e6c5f640 1 --2- 172.21.15.42:0/3349464524 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f39e00a3450 0x7f39e011e310 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f39d8007d90 tx=0x7f39d800e4b0 comp rx=0 tx=0).stop 2024-09-16T03:59:42.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.525+0000 7f39e6c5f640 1 -- 172.21.15.42:0/3349464524 shutdown_connections 2024-09-16T03:59:42.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.525+0000 7f39e6c5f640 1 --2- 172.21.15.42:0/3349464524 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f39e00a3450 0x7f39e011e310 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:59:42.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.525+0000 7f39e6c5f640 1 -- 172.21.15.42:0/3349464524 >> 172.21.15.42:0/3349464524 conn(0x7f39e00a4540 msgr2=0x7f39e006ec80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:59:42.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.525+0000 7f39e6c5f640 1 -- 172.21.15.42:0/3349464524 shutdown_connections 2024-09-16T03:59:42.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.525+0000 7f39e6c5f640 1 -- 172.21.15.42:0/3349464524 wait complete. 2024-09-16T03:59:42.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.525+0000 7f39e6c5f640 1 librados: shutdown 2024-09-16T03:59:42.567 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:59:42.567 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T03:59:42.567 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T03:59:42.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.681+0000 7fd5ee54c640 10 monclient: get_monmap_and_config 2024-09-16T03:59:42.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.681+0000 7fd5ee54c640 10 monclient: build_initial_monmap 2024-09-16T03:59:42.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.681+0000 7fd5ee54c640 10 monclient: monmap: 2024-09-16T03:59:42.686 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:59:42.686 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:59:42.686 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:59:42.684375+0000 2024-09-16T03:59:42.686 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:59:42.684375+0000 2024-09-16T03:59:42.686 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:59:42.686 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:59:42.686 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:59:42.687 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:59:42.687 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:59:42.687 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:59:42.687 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:59:42.687 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:59:42.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.681+0000 7fd5ee54c640 1 Processor -- start 2024-09-16T03:59:42.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.681+0000 7fd5ee54c640 1 -- start start 2024-09-16T03:59:42.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.681+0000 7fd5ee54c640 10 monclient: init 2024-09-16T03:59:42.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.681+0000 7fd5ee54c640 10 monclient: _reopen_session rank -1 2024-09-16T03:59:42.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.681+0000 7fd5ee54c640 10 monclient: _add_conns ranks=[3,0,1,2,4] 2024-09-16T03:59:42.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.681+0000 7fd5ee54c640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd5e80a1330 0x7fd5e809f180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:59:42.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.681+0000 7fd5ee54c640 10 monclient(hunting): picked mon.noname-a con 0x7fd5e80a1330 addr v2:172.21.15.42:3300/0 2024-09-16T03:59:42.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.681+0000 7fd5ee54c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd5e80a1710 addr v1:172.21.15.42:6789/0 2024-09-16T03:59:42.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.681+0000 7fd5ee54c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd5e81393a0 addr v1:172.21.15.154:6789/0 2024-09-16T03:59:42.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.681+0000 7fd5ee54c640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd5e809e4c0 con 0x7fd5e80a1710 2024-09-16T03:59:42.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.681+0000 7fd5ee54c640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd5e809f6c0 con 0x7fd5e81393a0 2024-09-16T03:59:42.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.681+0000 7fd5ee54c640 10 monclient(hunting): start opening mon connection 2024-09-16T03:59:42.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.681+0000 7fd5ee54c640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7fd5e809edf0 con 0x7fd5e80a1330 2024-09-16T03:59:42.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.681+0000 7fd5ee54c640 10 monclient(hunting): _renew_subs 2024-09-16T03:59:42.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.681+0000 7fd5ee54c640 10 monclient(hunting): authenticate will time out at 2029.653788s 2024-09-16T03:59:42.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.681+0000 7fd5e77fe640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fd5e80a1710 0x7fd5e80a1b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:52094/0 (socket says 172.21.15.42:52094) 2024-09-16T03:59:42.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.681+0000 7fd5e77fe640 1 -- 172.21.15.42:0/1031906919 learned_addr learned my addr 172.21.15.42:0/1031906919 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:59:42.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.681+0000 7fd5e7fff640 1 --2- 172.21.15.42:0/1031906919 >> v2:172.21.15.42:3300/0 conn(0x7fd5e80a1330 0x7fd5e809f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:59:42.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.681+0000 7fd5e7fff640 10 monclient(hunting): get_auth_request con 0x7fd5e80a1330 auth_method 0 2024-09-16T03:59:42.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.681+0000 7fd5e7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:59:42.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.681+0000 7fd5e7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:59:42.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.681+0000 7fd5e7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:59:42.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.681+0000 7fd5e7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:59:42.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.681+0000 7fd5e7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:59:42.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.681+0000 7fd5e7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:59:42.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.681+0000 7fd5e6ffd640 1 -- 172.21.15.42:0/1031906919 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 820696634 0 0) 0x7fd5e809e4c0 con 0x7fd5e80a1710 2024-09-16T03:59:42.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.681+0000 7fd5e6ffd640 10 monclient(hunting): _negotiate 2024-09-16T03:59:42.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.681+0000 7fd5e6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:59:42.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.681+0000 7fd5e6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:59:42.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.681+0000 7fd5e6ffd640 10 monclient(hunting): my global_id is 825171 2024-09-16T03:59:42.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.681+0000 7fd5e6ffd640 1 -- 172.21.15.42:0/1031906919 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd5d4003530 con 0x7fd5e80a1710 2024-09-16T03:59:42.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.681+0000 7fd5e7fff640 10 monclient(hunting): handle_auth_done global_id 825168 payload 274 2024-09-16T03:59:42.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.681+0000 7fd5e7fff640 1 -- 172.21.15.42:0/1031906919 >> v1:172.21.15.154:6789/0 conn(0x7fd5e81393a0 legacy=0x7fd5e813b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:59:42.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.681+0000 7fd5e7fff640 1 -- 172.21.15.42:0/1031906919 >> v1:172.21.15.42:6789/0 conn(0x7fd5e80a1710 legacy=0x7fd5e80a1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:59:42.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.681+0000 7fd5e7fff640 10 monclient: _finish_hunting 0 2024-09-16T03:59:42.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.681+0000 7fd5e7fff640 1 monclient: found mon.noname-a 2024-09-16T03:59:42.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.681+0000 7fd5e7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:59:42.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.681+0000 7fd5e7fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:59:42.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.681+0000 7fd5e7fff640 1 -- 172.21.15.42:0/1031906919 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5e80a3170 con 0x7fd5e80a1330 2024-09-16T03:59:42.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.685+0000 7fd5e7fff640 1 --2- 172.21.15.42:0/1031906919 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd5e80a1330 0x7fd5e809f180 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fd5d8009d90 tx=0x7fd5d801c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=580f7d2ce18996ef server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:59:42.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.685+0000 7fd5e6ffd640 1 -- 172.21.15.42:0/1031906919 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5d80b2070 con 0x7fd5e80a1330 2024-09-16T03:59:42.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.685+0000 7fd5e6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:59:42.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.685+0000 7fd5e6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:59:42.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.685+0000 7fd5e6ffd640 10 monclient: dump: 2024-09-16T03:59:42.698 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:59:42.699 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:59:42.699 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:42.699 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:42.699 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:59:42.699 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:59:42.699 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:59:42.699 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:59:42.699 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:59:42.699 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:59:42.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.685+0000 7fd5e6ffd640 10 monclient: _finish_auth 0 2024-09-16T03:59:42.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.685+0000 7fd5e6ffd640 10 monclient: _check_auth_tickets 2024-09-16T03:59:42.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.685+0000 7fd5e6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:59:42.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.685+0000 7fd5e6ffd640 1 -- 172.21.15.42:0/1031906919 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd5d80a5800 con 0x7fd5e80a1330 2024-09-16T03:59:42.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.685+0000 7fd5ee54c640 5 monclient: authenticate success, global_id 825168 2024-09-16T03:59:42.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.685+0000 7fd5ee54c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:59:42.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.685+0000 7fd5e6ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:59:42.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.685+0000 7fd5ee54c640 10 monclient: get_monmap_and_config success 2024-09-16T03:59:42.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.685+0000 7fd5ee54c640 4 set_mon_vals no callback set 2024-09-16T03:59:42.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.685+0000 7fd5e6ffd640 1 -- 172.21.15.42:0/1031906919 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5d80a5bd0 con 0x7fd5e80a1330 2024-09-16T03:59:42.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.685+0000 7fd5e6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:59:42.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.685+0000 7fd5e6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:59:42.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.685+0000 7fd5e6ffd640 10 monclient: dump: 2024-09-16T03:59:42.700 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:59:42.700 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:59:42.700 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:42.700 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:42.700 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:59:42.701 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:59:42.701 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:59:42.701 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:59:42.701 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:59:42.701 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:59:42.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.685+0000 7fd5ee54c640 10 monclient: shutdown 2024-09-16T03:59:42.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.685+0000 7fd5ee54c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:59:42.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.685+0000 7fd5ee54c640 1 -- 172.21.15.42:0/1031906919 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd5e80a1330 msgr2=0x7fd5e809f180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:59:42.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.685+0000 7fd5ee54c640 1 --2- 172.21.15.42:0/1031906919 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd5e80a1330 0x7fd5e809f180 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fd5d8009d90 tx=0x7fd5d801c830 comp rx=0 tx=0).stop 2024-09-16T03:59:42.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.685+0000 7fd5ee54c640 1 -- 172.21.15.42:0/1031906919 shutdown_connections 2024-09-16T03:59:42.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.685+0000 7fd5ee54c640 1 --2- 172.21.15.42:0/1031906919 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd5e80a1330 0x7fd5e809f180 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:59:42.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.685+0000 7fd5ee54c640 1 -- 172.21.15.42:0/1031906919 >> 172.21.15.42:0/1031906919 conn(0x7fd5e80a2420 msgr2=0x7fd5e80a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:59:42.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.685+0000 7fd5ee54c640 1 -- 172.21.15.42:0/1031906919 shutdown_connections 2024-09-16T03:59:42.702 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.685+0000 7fd5ee54c640 1 -- 172.21.15.42:0/1031906919 wait complete. 2024-09-16T03:59:42.702 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.685+0000 7fd5ee54c640 10 monclient: build_initial_monmap 2024-09-16T03:59:42.702 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.685+0000 7fd5ee54c640 10 monclient: monmap: 2024-09-16T03:59:42.702 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:59:42.702 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:59:42.702 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:59:42.702 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:59:42.702 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:59:42.702 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:59:42.702 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:59:42.702 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:59:42.702 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:59:42.702 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:59:42.702 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.685+0000 7fd5ee54c640 1 librados: starting msgr at 2024-09-16T03:59:42.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.685+0000 7fd5ee54c640 1 librados: starting objecter 2024-09-16T03:59:42.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.685+0000 7fd5ee54c640 1 Processor -- start 2024-09-16T03:59:42.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.685+0000 7fd5ee54c640 1 -- start start 2024-09-16T03:59:42.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.685+0000 7fd5ee54c640 1 librados: setting wanted keys 2024-09-16T03:59:42.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.685+0000 7fd5ee54c640 1 librados: calling monclient init 2024-09-16T03:59:42.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.685+0000 7fd5ee54c640 10 monclient: init 2024-09-16T03:59:42.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.689+0000 7fd5ee54c640 10 monclient: _reopen_session rank -1 2024-09-16T03:59:42.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.689+0000 7fd5ee54c640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T03:59:42.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.689+0000 7fd5ee54c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd5e80a1330 0x7fd5e81ccf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:59:42.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.689+0000 7fd5ee54c640 10 monclient(hunting): picked mon.noname-b con 0x7fd5e80a1330 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:59:42.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.689+0000 7fd5ee54c640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd5e80a1710 0x7fd5e81cd460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:59:42.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.689+0000 7fd5ee54c640 10 monclient(hunting): picked mon.noname-c con 0x7fd5e80a1710 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:59:42.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.689+0000 7fd5ee54c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd5e81393a0 0x7fd5e806f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:59:42.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.689+0000 7fd5ee54c640 10 monclient(hunting): picked mon.noname-a con 0x7fd5e81393a0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:59:42.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.689+0000 7fd5ee54c640 10 monclient(hunting): start opening mon connection 2024-09-16T03:59:42.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.689+0000 7fd5ee54c640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fd5e813f8e0 con 0x7fd5e81393a0 2024-09-16T03:59:42.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.689+0000 7fd5ee54c640 10 monclient(hunting): start opening mon connection 2024-09-16T03:59:42.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.689+0000 7fd5ee54c640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd5e809cd60 con 0x7fd5e80a1330 2024-09-16T03:59:42.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.689+0000 7fd5ee54c640 10 monclient(hunting): start opening mon connection 2024-09-16T03:59:42.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.689+0000 7fd5ee54c640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fd5e813fa60 con 0x7fd5e80a1710 2024-09-16T03:59:42.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.689+0000 7fd5ee54c640 10 monclient(hunting): _renew_subs 2024-09-16T03:59:42.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.689+0000 7fd5ee54c640 10 monclient(hunting): authenticate will time out at 2029.660167s 2024-09-16T03:59:42.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.689+0000 7fd5e77fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd5e80a1710 0x7fd5e81cd460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:59:42.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.689+0000 7fd5e77fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd5e80a1710 0x7fd5e81cd460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:40586/0 (socket says 172.21.15.42:40586) 2024-09-16T03:59:42.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.689+0000 7fd5e77fe640 1 -- 172.21.15.42:0/1015001970 learned_addr learned my addr 172.21.15.42:0/1015001970 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:59:42.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.689+0000 7fd5e77fe640 10 monclient(hunting): get_auth_request con 0x7fd5e80a1710 auth_method 0 2024-09-16T03:59:42.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.689+0000 7fd5e77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:59:42.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.689+0000 7fd5e77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:59:42.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.689+0000 7fd5e77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:59:42.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.689+0000 7fd5ecae9640 1 --2- 172.21.15.42:0/1015001970 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd5e81393a0 0x7fd5e806f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:59:42.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.689+0000 7fd5ecae9640 10 monclient(hunting): get_auth_request con 0x7fd5e81393a0 auth_method 0 2024-09-16T03:59:42.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.689+0000 7fd5ecae9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:59:42.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.689+0000 7fd5ecae9640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:59:42.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.689+0000 7fd5ecae9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:59:42.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.689+0000 7fd5e77fe640 1 -- 172.21.15.42:0/1015001970 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd5e80a1710 msgr2=0x7fd5e81cd460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-16T03:59:42.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.689+0000 7fd5e77fe640 1 -- 172.21.15.42:0/1015001970 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd5e80a1710 msgr2=0x7fd5e81cd460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T03:59:42.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.689+0000 7fd5e77fe640 1 --2- 172.21.15.42:0/1015001970 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd5e80a1710 0x7fd5e81cd460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:59:42.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.689+0000 7fd5e7fff640 1 --2- 172.21.15.42:0/1015001970 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd5e80a1330 0x7fd5e81ccf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:59:42.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.689+0000 7fd5e77fe640 1 --2- 172.21.15.42:0/1015001970 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd5e80a1710 0x7fd5e81cd460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:59:42.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.689+0000 7fd5ecae9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:59:42.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.689+0000 7fd5ecae9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:59:42.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.689+0000 7fd5ecae9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:59:42.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.689+0000 7fd5e7fff640 10 monclient(hunting): get_auth_request con 0x7fd5e80a1330 auth_method 0 2024-09-16T03:59:42.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.689+0000 7fd5e7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:59:42.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.689+0000 7fd5e7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:59:42.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.689+0000 7fd5e7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:59:42.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.689+0000 7fd5e7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:59:42.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.689+0000 7fd5e7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:59:42.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.689+0000 7fd5e7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:59:42.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.689+0000 7fd5ecae9640 10 monclient(hunting): handle_auth_done global_id 825174 payload 824 2024-09-16T03:59:42.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.689+0000 7fd5ecae9640 1 -- 172.21.15.42:0/1015001970 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd5e80a1710 msgr2=0x7fd5e81cd460 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:59:42.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.689+0000 7fd5ecae9640 1 --2- 172.21.15.42:0/1015001970 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd5e80a1710 0x7fd5e81cd460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:59:42.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.689+0000 7fd5ecae9640 1 -- 172.21.15.42:0/1015001970 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd5e80a1330 msgr2=0x7fd5e81ccf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:59:42.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.689+0000 7fd5ecae9640 1 --2- 172.21.15.42:0/1015001970 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd5e80a1330 0x7fd5e81ccf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:59:42.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.689+0000 7fd5ecae9640 10 monclient: _finish_hunting 0 2024-09-16T03:59:42.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.689+0000 7fd5ecae9640 1 monclient: found mon.noname-a 2024-09-16T03:59:42.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.689+0000 7fd5ecae9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:59:42.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.689+0000 7fd5ecae9640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:59:42.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.689+0000 7fd5ecae9640 1 -- 172.21.15.42:0/1015001970 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5e811c780 con 0x7fd5e81393a0 2024-09-16T03:59:42.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.689+0000 7fd5ecae9640 1 --2- 172.21.15.42:0/1015001970 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd5e81393a0 0x7fd5e806f1e0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fd5e000c3f0 tx=0x7fd5e000c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:59:42.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.689+0000 7fd5e4ff9640 10 client.?.objecter ms_handle_connect 0x7fd5e81393a0 2024-09-16T03:59:42.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.689+0000 7fd5e4ff9640 10 client.?.objecter resend_mon_ops 2024-09-16T03:59:42.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.689+0000 7fd5e4ff9640 1 -- 172.21.15.42:0/1015001970 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5e00052d0 con 0x7fd5e81393a0 2024-09-16T03:59:42.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.689+0000 7fd5e4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:59:42.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.689+0000 7fd5e4ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:59:42.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.689+0000 7fd5e4ff9640 10 monclient: dump: 2024-09-16T03:59:42.708 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:59:42.708 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:59:42.708 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:42.708 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:42.708 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:59:42.708 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:59:42.708 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:59:42.708 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:59:42.708 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:59:42.708 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:59:42.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.689+0000 7fd5e4ff9640 10 monclient: _finish_auth 0 2024-09-16T03:59:42.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.689+0000 7fd5e4ff9640 10 monclient: _check_auth_tickets 2024-09-16T03:59:42.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.689+0000 7fd5e4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:59:42.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.689+0000 7fd5e4ff9640 1 -- 172.21.15.42:0/1015001970 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd5e0005470 con 0x7fd5e81393a0 2024-09-16T03:59:42.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.689+0000 7fd5e4ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T03:59:42.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.689+0000 7fd5e4ff9640 1 -- 172.21.15.42:0/1015001970 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5e0007260 con 0x7fd5e81393a0 2024-09-16T03:59:42.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.689+0000 7fd5ee54c640 5 monclient: authenticate success, global_id 825174 2024-09-16T03:59:42.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.689+0000 7fd5ee54c640 10 monclient: _renew_subs 2024-09-16T03:59:42.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.689+0000 7fd5ee54c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:59:42.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.689+0000 7fd5ee54c640 1 -- 172.21.15.42:0/1015001970 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5e813c180 con 0x7fd5e81393a0 2024-09-16T03:59:42.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.689+0000 7fd5e67fc640 4 set_mon_vals no callback set 2024-09-16T03:59:42.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.689+0000 7fd5e4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:59:42.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.689+0000 7fd5e4ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:59:42.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.689+0000 7fd5e4ff9640 10 monclient: dump: 2024-09-16T03:59:42.709 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:59:42.710 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:59:42.710 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:42.710 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:42.710 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:59:42.710 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:59:42.710 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:59:42.710 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:59:42.710 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:59:42.710 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:59:42.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.689+0000 7fd5ee54c640 10 client.825174.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:59:42.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.689+0000 7fd5ee54c640 10 monclient: _renew_subs 2024-09-16T03:59:42.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.689+0000 7fd5ee54c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:59:42.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.689+0000 7fd5ee54c640 1 -- 172.21.15.42:0/1015001970 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5e811c1c0 con 0x7fd5e81393a0 2024-09-16T03:59:42.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.689+0000 7fd5ee54c640 1 librados: init done 2024-09-16T03:59:42.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.689+0000 7fd5ee54c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:59:42.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.689+0000 7fd5ee54c640 1 --2- 172.21.15.42:0/1015001970 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd5e806fee0 0x7fd5e802c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:59:42.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.689+0000 7fd5ee54c640 10 monclient(hunting): start opening mon connection 2024-09-16T03:59:42.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.689+0000 7fd5ee54c640 1 -- 172.21.15.42:0/1015001970 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fd5e81381b0 con 0x7fd5e806fee0 2024-09-16T03:59:42.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.689+0000 7fd5e7fff640 1 --2- 172.21.15.42:0/1015001970 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd5e806fee0 0x7fd5e802c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:59:42.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.689+0000 7fd5e7fff640 10 monclient: get_auth_request con 0x7fd5e806fee0 auth_method 0 2024-09-16T03:59:42.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.689+0000 7fd5e7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:59:42.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.689+0000 7fd5e7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:59:42.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.689+0000 7fd5e7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:59:42.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.689+0000 7fd5e4ff9640 1 -- 172.21.15.42:0/1015001970 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fd5e0007400 con 0x7fd5e81393a0 2024-09-16T03:59:42.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.689+0000 7fd5e7fff640 1 -- 172.21.15.42:0/1015001970 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd5e806fee0 msgr2=0x7fd5e802c650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-16T03:59:42.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.689+0000 7fd5e7fff640 1 -- 172.21.15.42:0/1015001970 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd5e806fee0 msgr2=0x7fd5e802c650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T03:59:42.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.689+0000 7fd5e7fff640 1 --2- 172.21.15.42:0/1015001970 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd5e806fee0 0x7fd5e802c650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:59:42.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.689+0000 7fd5e7fff640 1 --2- 172.21.15.42:0/1015001970 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd5e806fee0 0x7fd5e802c650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T03:59:42.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.689+0000 7fd5e4ff9640 1 --2- 172.21.15.42:0/1015001970 >> v2:172.21.15.154:6812/254845342 conn(0x7fd5c8037810 0x7fd5c8039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:59:42.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.689+0000 7fd5e4ff9640 1 -- 172.21.15.42:0/1015001970 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(540..540 src has 451..540) ==== 6168+0+0 (secure 0 0 0) 0x7fd5e00484e0 con 0x7fd5e81393a0 2024-09-16T03:59:42.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.689+0000 7fd5e4ff9640 10 client.825174.objecter ms_dispatch 0x7fd5e8000ba0 osd_map(540..540 src has 451..540) 2024-09-16T03:59:42.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.689+0000 7fd5e4ff9640 3 client.825174.objecter handle_osd_map got epochs [540,540] > 0 2024-09-16T03:59:42.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.689+0000 7fd5e4ff9640 3 client.825174.objecter handle_osd_map decoding full epoch 540 2024-09-16T03:59:42.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.689+0000 7fd5e4ff9640 20 client.825174.objecter dump_active .. 0 homeless 2024-09-16T03:59:42.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.689+0000 7fd5e77fe640 1 --2- 172.21.15.42:0/1015001970 >> v2:172.21.15.154:6812/254845342 conn(0x7fd5c8037810 0x7fd5c8039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:59:42.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.689+0000 7fd5e77fe640 10 monclient: get_auth_request con 0x7fd5c8037810 auth_method 0 2024-09-16T03:59:42.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.689+0000 7fd5e4ff9640 10 client.825174.objecter ms_handle_connect 0x7fd5c8037810 2024-09-16T03:59:42.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.689+0000 7fd5e77fe640 1 --2- 172.21.15.42:0/1015001970 >> v2:172.21.15.154:6812/254845342 conn(0x7fd5c8037810 0x7fd5c8039cd0 secure :-1 s=READY pgs=3733 cs=0 l=1 rev1=1 crypto rx=0x7fd5e8071bb0 tx=0x7fd5d0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:59:42.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.893+0000 7fd5e7fff640 1 --2- 172.21.15.42:0/1015001970 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd5e806fee0 0x7fd5e802c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:59:42.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.893+0000 7fd5e7fff640 10 monclient: get_auth_request con 0x7fd5e806fee0 auth_method 0 2024-09-16T03:59:42.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.893+0000 7fd5e7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:59:42.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.893+0000 7fd5e7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:59:42.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.893+0000 7fd5e7fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:59:42.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.893+0000 7fd5e7fff640 1 -- 172.21.15.42:0/1015001970 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd5e806fee0 msgr2=0x7fd5e802c650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-16T03:59:42.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.893+0000 7fd5e7fff640 1 -- 172.21.15.42:0/1015001970 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd5e806fee0 msgr2=0x7fd5e802c650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T03:59:42.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.893+0000 7fd5e7fff640 1 --2- 172.21.15.42:0/1015001970 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd5e806fee0 0x7fd5e802c650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:59:42.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:42.893+0000 7fd5e7fff640 1 --2- 172.21.15.42:0/1015001970 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd5e806fee0 0x7fd5e802c650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T03:59:43.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:43.297+0000 7fd5e7fff640 1 --2- 172.21.15.42:0/1015001970 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd5e806fee0 0x7fd5e802c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:59:43.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:43.297+0000 7fd5e7fff640 10 monclient: get_auth_request con 0x7fd5e806fee0 auth_method 0 2024-09-16T03:59:43.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:43.297+0000 7fd5e7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:59:43.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:43.297+0000 7fd5e7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:59:43.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:43.297+0000 7fd5e7fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:59:43.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:43.297+0000 7fd5e7fff640 1 -- 172.21.15.42:0/1015001970 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd5e806fee0 msgr2=0x7fd5e802c650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-16T03:59:43.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:43.297+0000 7fd5e7fff640 1 -- 172.21.15.42:0/1015001970 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd5e806fee0 msgr2=0x7fd5e802c650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T03:59:43.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:43.297+0000 7fd5e7fff640 1 --2- 172.21.15.42:0/1015001970 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd5e806fee0 0x7fd5e802c650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:59:43.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:43.297+0000 7fd5e7fff640 1 --2- 172.21.15.42:0/1015001970 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd5e806fee0 0x7fd5e802c650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T03:59:44.106 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:44.101+0000 7fd5e7fff640 1 --2- 172.21.15.42:0/1015001970 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd5e806fee0 0x7fd5e802c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:59:44.106 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:44.101+0000 7fd5e7fff640 10 monclient: get_auth_request con 0x7fd5e806fee0 auth_method 0 2024-09-16T03:59:44.106 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:44.101+0000 7fd5e7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:59:44.106 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:44.101+0000 7fd5e7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:59:44.106 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:44.101+0000 7fd5e7fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:59:44.106 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:44.101+0000 7fd5e7fff640 1 -- 172.21.15.42:0/1015001970 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd5e806fee0 msgr2=0x7fd5e802c650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-16T03:59:44.106 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:44.101+0000 7fd5e7fff640 1 -- 172.21.15.42:0/1015001970 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd5e806fee0 msgr2=0x7fd5e802c650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T03:59:44.107 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:44.101+0000 7fd5e7fff640 1 --2- 172.21.15.42:0/1015001970 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd5e806fee0 0x7fd5e802c650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:59:44.107 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:44.101+0000 7fd5e7fff640 1 --2- 172.21.15.42:0/1015001970 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd5e806fee0 0x7fd5e802c650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T03:59:45.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:45.689+0000 7fd5df7fe640 10 monclient: tick 2024-09-16T03:59:45.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:45.689+0000 7fd5df7fe640 10 monclient: _check_auth_tickets 2024-09-16T03:59:45.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:45.689+0000 7fd5df7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:59:45.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:45.689+0000 7fd5df7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:59:45.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:45.705+0000 7fd5e7fff640 1 --2- 172.21.15.42:0/1015001970 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd5e806fee0 0x7fd5e802c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:59:45.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:45.705+0000 7fd5e7fff640 10 monclient: get_auth_request con 0x7fd5e806fee0 auth_method 0 2024-09-16T03:59:45.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:45.705+0000 7fd5e7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:59:45.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:45.705+0000 7fd5e7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:59:45.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:45.705+0000 7fd5e7fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T03:59:45.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:45.705+0000 7fd5e7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:59:45.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:45.705+0000 7fd5e7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:59:45.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:45.705+0000 7fd5e7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:59:45.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:45.705+0000 7fd5e7fff640 1 -- 172.21.15.42:0/1015001970 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd5e806fee0 msgr2=0x7fd5e802c650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-16T03:59:45.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:45.705+0000 7fd5e7fff640 1 -- 172.21.15.42:0/1015001970 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd5e806fee0 msgr2=0x7fd5e802c650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T03:59:45.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:45.705+0000 7fd5e7fff640 1 --2- 172.21.15.42:0/1015001970 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd5e806fee0 0x7fd5e802c650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T03:59:45.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:45.705+0000 7fd5e7fff640 1 --2- 172.21.15.42:0/1015001970 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd5e806fee0 0x7fd5e802c650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-09-16T03:59:46.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.689+0000 7fd5df7fe640 10 monclient: tick 2024-09-16T03:59:46.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.689+0000 7fd5df7fe640 10 monclient: _check_auth_tickets 2024-09-16T03:59:46.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.689+0000 7fd5df7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:59:46.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.689+0000 7fd5df7fe640 5 monclient: _check_tell_commands timeout tell command 1 2024-09-16T03:59:46.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.689+0000 7fd5df7fe640 1 -- 172.21.15.42:0/1015001970 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd5e806fee0 msgr2=0x7fd5e802c650 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T03:59:46.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.689+0000 7fd5df7fe640 1 --2- 172.21.15.42:0/1015001970 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd5e806fee0 0x7fd5e802c650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:59:46.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.689+0000 7fd5df7fe640 1 --2- 172.21.15.42:0/1015001970 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd5c0003550 0x7fd5c00059e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:59:46.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.689+0000 7fd5df7fe640 1 -- 172.21.15.42:0/1015001970 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd5e806fee0 msgr2=0x7fd5e802c650 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:59:46.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.689+0000 7fd5df7fe640 1 --2- 172.21.15.42:0/1015001970 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd5e806fee0 0x7fd5e802c650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:59:46.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.689+0000 7fd5df7fe640 10 monclient(hunting): start opening mon connection 2024-09-16T03:59:46.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.689+0000 7fd5df7fe640 1 -- 172.21.15.42:0/1015001970 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fd5e81381b0 con 0x7fd5c0003550 2024-09-16T03:59:46.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.689+0000 7fd5e7fff640 1 --2- 172.21.15.42:0/1015001970 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd5c0003550 0x7fd5c00059e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:59:46.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.689+0000 7fd5e7fff640 10 monclient: get_auth_request con 0x7fd5c0003550 auth_method 0 2024-09-16T03:59:46.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.689+0000 7fd5e7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:59:46.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.689+0000 7fd5e7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:59:46.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.689+0000 7fd5e7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:59:46.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.689+0000 7fd5e7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:59:46.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.689+0000 7fd5e7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:59:46.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.689+0000 7fd5e7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:59:46.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.689+0000 7fd5e7fff640 10 monclient(hunting): handle_auth_done global_id 934103 payload 824 2024-09-16T03:59:46.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.689+0000 7fd5e7fff640 1 -- 172.21.15.42:0/1015001970 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd5c00060c0 con 0x7fd5c0003550 2024-09-16T03:59:46.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.689+0000 7fd5e7fff640 1 --2- 172.21.15.42:0/1015001970 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd5c0003550 0x7fd5c00059e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fd5e8073490 tx=0x7fd5d80195d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:59:46.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.689+0000 7fd5e4ff9640 10 client.825174.objecter ms_handle_connect 0x7fd5c0003550 2024-09-16T03:59:46.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.689+0000 7fd5e4ff9640 10 client.825174.objecter resend_mon_ops 2024-09-16T03:59:46.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.689+0000 7fd5e4ff9640 1 -- 172.21.15.42:0/1015001970 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5d80c7070 con 0x7fd5c0003550 2024-09-16T03:59:46.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.689+0000 7fd5e4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:59:46.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.689+0000 7fd5e4ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:59:46.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.689+0000 7fd5e4ff9640 10 monclient: dump: 2024-09-16T03:59:46.701 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:59:46.701 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:59:46.701 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:46.702 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:46.702 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:59:46.702 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:59:46.702 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:59:46.702 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:59:46.702 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:59:46.702 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:59:46.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.689+0000 7fd5e4ff9640 1 -- 172.21.15.42:0/1015001970 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fd5d80b3040 con 0x7fd5c0003550 2024-09-16T03:59:46.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.689+0000 7fd5e4ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:59:46.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.689+0000 7fd5e4ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:59:46.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.689+0000 7fd5e4ff9640 1 -- 172.21.15.42:0/1015001970 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd5c0003550 msgr2=0x7fd5c00059e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:59:46.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.689+0000 7fd5e4ff9640 1 --2- 172.21.15.42:0/1015001970 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd5c0003550 0x7fd5c00059e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fd5e8073490 tx=0x7fd5d80195d0 comp rx=0 tx=0).stop 2024-09-16T03:59:46.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.689+0000 7fd5e4ff9640 1 -- 172.21.15.42:0/1015001970 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd5c0003550 msgr2=0x7fd5c00059e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:59:46.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.689+0000 7fd5e4ff9640 1 --2- 172.21.15.42:0/1015001970 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd5c0003550 0x7fd5c00059e0 secure :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fd5e8073490 tx=0x7fd5d80195d0 comp rx=0 tx=0).stop 2024-09-16T03:59:46.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.697+0000 7fd5ee54c640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:59:46.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.697+0000 7fd5ee54c640 1 --2- 172.21.15.42:0/1015001970 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd5e80702c0 0x7fd5e802cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:59:46.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.697+0000 7fd5ee54c640 10 monclient(hunting): start opening mon connection 2024-09-16T03:59:46.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.697+0000 7fd5ee54c640 1 -- 172.21.15.42:0/1015001970 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fd5e8069040 con 0x7fd5e80702c0 2024-09-16T03:59:46.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.697+0000 7fd5e7fff640 1 --2- 172.21.15.42:0/1015001970 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd5e80702c0 0x7fd5e802cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:59:46.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.697+0000 7fd5e7fff640 10 monclient: get_auth_request con 0x7fd5e80702c0 auth_method 0 2024-09-16T03:59:46.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.697+0000 7fd5e7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:59:46.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.697+0000 7fd5e7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:59:46.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.697+0000 7fd5e7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:59:46.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.697+0000 7fd5e7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:59:46.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.697+0000 7fd5e7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:59:46.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.697+0000 7fd5e7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:59:46.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.697+0000 7fd5e7fff640 10 monclient(hunting): handle_auth_done global_id 934106 payload 824 2024-09-16T03:59:46.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.697+0000 7fd5e7fff640 1 -- 172.21.15.42:0/1015001970 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fd5e81cd9a0 con 0x7fd5e80702c0 2024-09-16T03:59:46.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.697+0000 7fd5e7fff640 1 --2- 172.21.15.42:0/1015001970 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd5e80702c0 0x7fd5e802cb90 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fd5d801a460 tx=0x7fd5d80ba040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:59:46.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.697+0000 7fd5e4ff9640 10 client.825174.objecter ms_handle_connect 0x7fd5e80702c0 2024-09-16T03:59:46.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.697+0000 7fd5e4ff9640 10 client.825174.objecter resend_mon_ops 2024-09-16T03:59:46.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.701+0000 7fd5e4ff9640 1 -- 172.21.15.42:0/1015001970 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5d80b2040 con 0x7fd5e80702c0 2024-09-16T03:59:46.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.701+0000 7fd5e4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:59:46.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.701+0000 7fd5e4ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:59:46.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.701+0000 7fd5e4ff9640 10 monclient: dump: 2024-09-16T03:59:46.708 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:59:46.708 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:59:46.708 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:46.708 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:46.708 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:59:46.709 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:59:46.709 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:59:46.709 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:59:46.709 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:59:46.709 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:59:46.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.701+0000 7fd5e4ff9640 1 -- 172.21.15.42:0/1015001970 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4545 (secure 0 0 0) 0x7fd5d80ad9d0 con 0x7fd5e80702c0 2024-09-16T03:59:46.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.701+0000 7fd5e4ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:59:46.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.701+0000 7fd5e4ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:59:46.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.701+0000 7fd5e4ff9640 1 -- 172.21.15.42:0/1015001970 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd5e80702c0 msgr2=0x7fd5e802cb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:59:46.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.701+0000 7fd5e4ff9640 1 --2- 172.21.15.42:0/1015001970 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd5e80702c0 0x7fd5e802cb90 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fd5d801a460 tx=0x7fd5d80ba040 comp rx=0 tx=0).stop 2024-09-16T03:59:46.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.701+0000 7fd5e4ff9640 1 -- 172.21.15.42:0/1015001970 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd5e80702c0 msgr2=0x7fd5e802cb90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:59:46.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.701+0000 7fd5e4ff9640 1 --2- 172.21.15.42:0/1015001970 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd5e80702c0 0x7fd5e802cb90 secure :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fd5d801a460 tx=0x7fd5d80ba040 comp rx=0 tx=0).stop 2024-09-16T03:59:46.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.701+0000 7fd5ecae9640 1 -- 172.21.15.42:0/1015001970 reap_dead start 2024-09-16T03:59:46.710 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:59:46.710 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:59:46.711 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:59:46.711 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T03:59:46.711 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 160, 2024-09-16T03:59:46.711 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 11427, 2024-09-16T03:59:46.711 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:59:46.711 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:59:46.711 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T03:59:46.711 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T03:59:46.711 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:59:46.712 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 5, 2024-09-16T03:59:46.712 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:59:46.712 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:59:46.712 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:59:46.712 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:59:46.712 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:59:46.712 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:59:46.712 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:59:46.712 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:59:46.712 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:59:46.712 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:59:46.712 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:59:46.712 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:59:46.713 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:59:46.713 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:59:46.713 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:59:46.713 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:59:46.713 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:59:46.713 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:59:46.713 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:59:46.713 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:59:46.713 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:59:46.713 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:59:46.713 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:59:46.713 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:59:46.713 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:59:46.713 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:59:46.714 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:59:46.714 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:59:46.714 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:59:46.714 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:59:46.714 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:59:46.714 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:59:46.714 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:59:46.714 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:59:46.714 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:59:46.714 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:59:46.714 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:59:46.715 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:59:46.715 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:59:46.715 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:59:46.715 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:59:46.715 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:59:46.715 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:59:46.715 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:59:46.715 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:59:46.715 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:59:46.715 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:59:46.715 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:59:46.715 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:59:46.715 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:59:46.716 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:59:46.716 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:59:46.716 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:59:46.716 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:59:46.716 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:59:46.716 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:59:46.716 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:59:46.716 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:59:46.716 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:59:46.716 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:59:46.716 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:59:46.717 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:59:46.717 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:59:46.717 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:59:46.717 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:59:46.717 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:59:46.717 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:59:46.717 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:59:46.717 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:59:46.717 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:59:46.717 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:59:46.717 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:59:46.717 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:59:46.718 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:59:46.718 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:59:46.718 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:59:46.718 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:59:46.718 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:59:46.718 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:59:46.718 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:59:46.718 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:59:46.718 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:59:46.718 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:59:46.719 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:59:46.719 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:59:46.719 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:59:46.719 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:59:46.719 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:59:46.719 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:59:46.719 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:59:46.719 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:59:46.719 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:59:46.719 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:59:46.719 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:59:46.719 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:59:46.719 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:59:46.719 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:59:46.720 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:59:46.720 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:59:46.720 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:59:46.720 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:59:46.720 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:59:46.720 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:59:46.720 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:59:46.720 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:59:46.720 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:59:46.720 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:59:46.720 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:59:46.720 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:59:46.721 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:59:46.721 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:59:46.721 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:59:46.721 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:59:46.721 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:59:46.721 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:59:46.721 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:59:46.721 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:59:46.722 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:59:46.722 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:59:46.722 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:59:46.722 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:59:46.722 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:59:46.722 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:59:46.722 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:59:46.722 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:59:46.722 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:59:46.722 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:59:46.722 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:59:46.722 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:59:46.722 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:59:46.723 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:59:46.723 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:59:46.723 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:59:46.723 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:59:46.723 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:59:46.723 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:59:46.723 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:59:46.723 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:59:46.723 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:59:46.723 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:59:46.723 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:59:46.723 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:59:46.723 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:59:46.723 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:59:46.724 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:59:46.724 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:59:46.724 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:59:46.724 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:59:46.724 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:59:46.724 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:59:46.724 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:59:46.725 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.701+0000 7fd5ee54c640 10 librados: watch_flush enter 2024-09-16T03:59:46.725 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.701+0000 7fd5ee54c640 10 librados: watch_flush exit 2024-09-16T03:59:46.725 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.701+0000 7fd5ee54c640 20 client.825174.objecter shutdown clearing up homeless session... 2024-09-16T03:59:46.726 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.701+0000 7fd5ee54c640 10 client.825174.objecter successfully canceled tick 2024-09-16T03:59:46.726 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.701+0000 7fd5ee54c640 1 -- 172.21.15.42:0/1015001970 >> v2:172.21.15.154:6812/254845342 conn(0x7fd5c8037810 msgr2=0x7fd5c8039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:59:46.726 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.701+0000 7fd5ee54c640 1 --2- 172.21.15.42:0/1015001970 >> v2:172.21.15.154:6812/254845342 conn(0x7fd5c8037810 0x7fd5c8039cd0 secure :-1 s=READY pgs=3733 cs=0 l=1 rev1=1 crypto rx=0x7fd5e8071bb0 tx=0x7fd5d0009040 comp rx=0 tx=0).stop 2024-09-16T03:59:46.726 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.701+0000 7fd5ee54c640 10 monclient: shutdown 2024-09-16T03:59:46.726 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.701+0000 7fd5ee54c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:59:46.726 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.701+0000 7fd5ee54c640 1 -- 172.21.15.42:0/1015001970 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd5e81393a0 msgr2=0x7fd5e806f1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:59:46.726 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.701+0000 7fd5ee54c640 1 --2- 172.21.15.42:0/1015001970 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd5e81393a0 0x7fd5e806f1e0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fd5e000c3f0 tx=0x7fd5e000c8b0 comp rx=0 tx=0).stop 2024-09-16T03:59:46.726 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.701+0000 7fd5ee54c640 1 -- 172.21.15.42:0/1015001970 shutdown_connections 2024-09-16T03:59:46.726 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.701+0000 7fd5ee54c640 1 --2- 172.21.15.42:0/1015001970 >> v2:172.21.15.154:6812/254845342 conn(0x7fd5c8037810 0x7fd5c8039cd0 unknown :-1 s=CLOSED pgs=3733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:59:46.726 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.701+0000 7fd5ee54c640 1 --2- 172.21.15.42:0/1015001970 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd5e81393a0 0x7fd5e806f1e0 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:59:46.726 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.701+0000 7fd5ee54c640 1 -- 172.21.15.42:0/1015001970 >> 172.21.15.42:0/1015001970 conn(0x7fd5e80a2420 msgr2=0x7fd5e8135380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:59:46.726 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.701+0000 7fd5ee54c640 1 -- 172.21.15.42:0/1015001970 shutdown_connections 2024-09-16T03:59:46.726 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.701+0000 7fd5ee54c640 1 -- 172.21.15.42:0/1015001970 wait complete. 2024-09-16T03:59:46.727 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.701+0000 7fd5ee54c640 1 librados: shutdown 2024-09-16T03:59:46.728 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:59:46.729 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T03:59:46.729 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T03:59:46.861 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.857+0000 7f15a1d9b640 10 monclient: get_monmap_and_config 2024-09-16T03:59:46.861 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.857+0000 7f15a1d9b640 10 monclient: build_initial_monmap 2024-09-16T03:59:46.862 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.857+0000 7f15a1d9b640 10 monclient: monmap: 2024-09-16T03:59:46.862 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:59:46.862 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:59:46.862 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:59:46.860186+0000 2024-09-16T03:59:46.862 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:59:46.860186+0000 2024-09-16T03:59:46.862 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:59:46.862 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:59:46.862 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:59:46.862 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:59:46.862 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:59:46.862 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:59:46.862 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:59:46.862 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:59:46.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.857+0000 7f15a1d9b640 1 Processor -- start 2024-09-16T03:59:46.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.857+0000 7f15a1d9b640 1 -- start start 2024-09-16T03:59:46.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.857+0000 7f15a1d9b640 10 monclient: init 2024-09-16T03:59:46.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.857+0000 7f15a1d9b640 10 monclient: _reopen_session rank -1 2024-09-16T03:59:46.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.857+0000 7f15a1d9b640 10 monclient: _add_conns ranks=[4,2,0,1,3] 2024-09-16T03:59:46.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.857+0000 7f15a1d9b640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f159c0a1330 0x7f159c09f180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:59:46.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.857+0000 7f15a1d9b640 10 monclient(hunting): picked mon.noname-c con 0x7f159c0a1330 addr v2:172.21.15.154:3300/0 2024-09-16T03:59:46.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.857+0000 7f15a1d9b640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f159c0a1710 0x7f159c0a1b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:59:46.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.857+0000 7f15a1d9b640 10 monclient(hunting): picked mon.noname-b con 0x7f159c0a1710 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:59:46.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.857+0000 7f15a1d9b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f159c1340f0 addr v1:172.21.15.42:6789/0 2024-09-16T03:59:46.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.857+0000 7f15a1d9b640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f159c09e4c0 con 0x7f159c1340f0 2024-09-16T03:59:46.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.857+0000 7f15a1d9b640 10 monclient(hunting): start opening mon connection 2024-09-16T03:59:46.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.857+0000 7f15a1d9b640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f159c09edf0 con 0x7f159c0a1330 2024-09-16T03:59:46.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.857+0000 7f15a1d9b640 10 monclient(hunting): start opening mon connection 2024-09-16T03:59:46.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.857+0000 7f15a1d9b640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f159c09cd60 con 0x7f159c0a1710 2024-09-16T03:59:46.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.857+0000 7f15a1d9b640 10 monclient(hunting): _renew_subs 2024-09-16T03:59:46.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.857+0000 7f15a1d9b640 10 monclient(hunting): authenticate will time out at 2033.829572s 2024-09-16T03:59:46.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.857+0000 7f159affd640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f159c0a1710 0x7f159c0a1b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:59:46.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.857+0000 7f159bfff640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f159c1340f0 0x7f159c1345a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:34804/0 (socket says 172.21.15.42:34804) 2024-09-16T03:59:46.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.857+0000 7f159bfff640 1 -- 172.21.15.42:0/147897527 learned_addr learned my addr 172.21.15.42:0/147897527 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:59:46.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.857+0000 7f159affd640 10 monclient(hunting): get_auth_request con 0x7f159c0a1710 auth_method 0 2024-09-16T03:59:46.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.857+0000 7f159affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:59:46.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.857+0000 7f159affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:59:46.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.857+0000 7f159affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:59:46.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.857+0000 7f159affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:59:46.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.857+0000 7f159affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:59:46.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.857+0000 7f159affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:59:46.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.857+0000 7f159b7fe640 1 --2- 172.21.15.42:0/147897527 >> v2:172.21.15.154:3300/0 conn(0x7f159c0a1330 0x7f159c09f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:59:46.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.857+0000 7f159b7fe640 10 monclient(hunting): get_auth_request con 0x7f159c0a1330 auth_method 0 2024-09-16T03:59:46.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.857+0000 7f159b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:59:46.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.857+0000 7f159b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:59:46.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.857+0000 7f159b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:59:46.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.857+0000 7f159a7fc640 1 -- 172.21.15.42:0/147897527 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2221927725 0 0) 0x7f159c09e4c0 con 0x7f159c1340f0 2024-09-16T03:59:46.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.857+0000 7f159a7fc640 10 monclient(hunting): _negotiate 2024-09-16T03:59:46.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.857+0000 7f159a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:59:46.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.857+0000 7f159a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:59:46.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.857+0000 7f159a7fc640 10 monclient(hunting): my global_id is 825177 2024-09-16T03:59:46.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.857+0000 7f159a7fc640 1 -- 172.21.15.42:0/147897527 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1584003530 con 0x7f159c1340f0 2024-09-16T03:59:46.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.857+0000 7f159affd640 10 monclient(hunting): handle_auth_done global_id 934109 payload 274 2024-09-16T03:59:46.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.857+0000 7f159affd640 1 -- 172.21.15.42:0/147897527 >> v2:172.21.15.154:3300/0 conn(0x7f159c0a1330 msgr2=0x7f159c09f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:59:46.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.857+0000 7f159affd640 1 --2- 172.21.15.42:0/147897527 >> v2:172.21.15.154:3300/0 conn(0x7f159c0a1330 0x7f159c09f180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:59:46.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.857+0000 7f159affd640 1 -- 172.21.15.42:0/147897527 >> v1:172.21.15.42:6789/0 conn(0x7f159c1340f0 legacy=0x7f159c1345a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:59:46.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.857+0000 7f159affd640 10 monclient: _finish_hunting 0 2024-09-16T03:59:46.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.857+0000 7f159affd640 1 monclient: found mon.noname-b 2024-09-16T03:59:46.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.857+0000 7f159affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:59:46.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.857+0000 7f159affd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T03:59:46.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.857+0000 7f159affd640 1 -- 172.21.15.42:0/147897527 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f159c0a2050 con 0x7f159c0a1710 2024-09-16T03:59:46.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.861+0000 7f159affd640 1 --2- 172.21.15.42:0/147897527 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f159c0a1710 0x7f159c0a1b10 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f15900097f0 tx=0x7f159001c360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c288a0442583f47 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:59:46.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.861+0000 7f159a7fc640 1 -- 172.21.15.42:0/147897527 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15900b0070 con 0x7f159c0a1710 2024-09-16T03:59:46.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.861+0000 7f159a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:59:46.871 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.861+0000 7f159a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:59:46.871 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.861+0000 7f159a7fc640 10 monclient: dump: 2024-09-16T03:59:46.871 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:59:46.871 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:59:46.871 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:46.871 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:46.871 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:59:46.871 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:59:46.871 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:59:46.871 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:59:46.871 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:59:46.871 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:59:46.871 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.861+0000 7f159a7fc640 10 monclient: _finish_auth 0 2024-09-16T03:59:46.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.861+0000 7f159a7fc640 10 monclient: _check_auth_tickets 2024-09-16T03:59:46.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.861+0000 7f159a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:59:46.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.861+0000 7f15a1d9b640 5 monclient: authenticate success, global_id 934109 2024-09-16T03:59:46.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.861+0000 7f15a1d9b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:59:46.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.861+0000 7f159a7fc640 1 -- 172.21.15.42:0/147897527 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f15900ab600 con 0x7f159c0a1710 2024-09-16T03:59:46.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.861+0000 7f159a7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T03:59:46.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.861+0000 7f159a7fc640 1 -- 172.21.15.42:0/147897527 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15900ab9b0 con 0x7f159c0a1710 2024-09-16T03:59:46.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.861+0000 7f159a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:59:46.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.861+0000 7f159a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:59:46.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.861+0000 7f159a7fc640 10 monclient: dump: 2024-09-16T03:59:46.872 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:59:46.872 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:59:46.872 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:46.873 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:46.873 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:59:46.873 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:59:46.873 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:59:46.873 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:59:46.873 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:59:46.873 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:59:46.873 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.861+0000 7f15a1d9b640 10 monclient: get_monmap_and_config success 2024-09-16T03:59:46.873 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.861+0000 7f15a1d9b640 4 set_mon_vals no callback set 2024-09-16T03:59:46.873 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.861+0000 7f15a1d9b640 10 monclient: shutdown 2024-09-16T03:59:46.873 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.861+0000 7f15a1d9b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:59:46.874 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.861+0000 7f15a1d9b640 1 -- 172.21.15.42:0/147897527 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f159c0a1710 msgr2=0x7f159c0a1b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:59:46.874 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.861+0000 7f15a1d9b640 1 --2- 172.21.15.42:0/147897527 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f159c0a1710 0x7f159c0a1b10 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f15900097f0 tx=0x7f159001c360 comp rx=0 tx=0).stop 2024-09-16T03:59:46.874 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.861+0000 7f15a1d9b640 1 -- 172.21.15.42:0/147897527 shutdown_connections 2024-09-16T03:59:46.874 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.861+0000 7f15a1d9b640 1 --2- 172.21.15.42:0/147897527 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f159c0a1710 0x7f159c0a1b10 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:59:46.874 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.861+0000 7f15a1d9b640 1 --2- 172.21.15.42:0/147897527 >> v2:172.21.15.154:3300/0 conn(0x7f159c0a1330 0x7f159c09f180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:59:46.874 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.861+0000 7f15a1d9b640 1 -- 172.21.15.42:0/147897527 >> 172.21.15.42:0/147897527 conn(0x7f159c0a2420 msgr2=0x7f159c0a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:59:46.874 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.861+0000 7f15a1d9b640 1 -- 172.21.15.42:0/147897527 shutdown_connections 2024-09-16T03:59:46.874 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.861+0000 7f15a1d9b640 1 -- 172.21.15.42:0/147897527 wait complete. 2024-09-16T03:59:46.883 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.861+0000 7f15a1d9b640 10 monclient: build_initial_monmap 2024-09-16T03:59:46.883 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.861+0000 7f15a1d9b640 10 monclient: monmap: 2024-09-16T03:59:46.883 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:59:46.883 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:59:46.883 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:59:46.883 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:59:46.883 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:59:46.883 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:59:46.884 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:59:46.884 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:59:46.884 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:59:46.884 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:59:46.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.861+0000 7f15a1d9b640 1 librados: starting msgr at 2024-09-16T03:59:46.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.861+0000 7f15a1d9b640 1 librados: starting objecter 2024-09-16T03:59:46.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.861+0000 7f15a1d9b640 1 Processor -- start 2024-09-16T03:59:46.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.861+0000 7f15a1d9b640 1 -- start start 2024-09-16T03:59:46.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.861+0000 7f15a1d9b640 1 librados: setting wanted keys 2024-09-16T03:59:46.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.861+0000 7f15a1d9b640 1 librados: calling monclient init 2024-09-16T03:59:46.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.861+0000 7f15a1d9b640 10 monclient: init 2024-09-16T03:59:46.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.861+0000 7f15a1d9b640 10 monclient: _reopen_session rank -1 2024-09-16T03:59:46.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.861+0000 7f15a1d9b640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T03:59:46.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.861+0000 7f15a1d9b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f159c0a1330 0x7f159c1cf020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:59:46.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.861+0000 7f15a1d9b640 10 monclient(hunting): picked mon.noname-b con 0x7f159c0a1330 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:59:46.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.861+0000 7f15a1d9b640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f159c0a1710 0x7f159c133250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:59:46.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.861+0000 7f15a1d9b640 10 monclient(hunting): picked mon.noname-c con 0x7f159c0a1710 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:59:46.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.861+0000 7f15a1d9b640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f159c1340f0 0x7f159c133790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:59:46.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.861+0000 7f15a1d9b640 10 monclient(hunting): picked mon.noname-a con 0x7f159c1340f0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:59:46.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.861+0000 7f15a1d9b640 10 monclient(hunting): start opening mon connection 2024-09-16T03:59:46.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.861+0000 7f15a1d9b640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f159c13ed60 con 0x7f159c1340f0 2024-09-16T03:59:46.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.861+0000 7f15a1d9b640 10 monclient(hunting): start opening mon connection 2024-09-16T03:59:46.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.861+0000 7f15a1d9b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f159c13ebe0 con 0x7f159c0a1330 2024-09-16T03:59:46.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.861+0000 7f15a1d9b640 10 monclient(hunting): start opening mon connection 2024-09-16T03:59:46.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.861+0000 7f15a1d9b640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f159c13eee0 con 0x7f159c0a1710 2024-09-16T03:59:46.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.861+0000 7f15a1d9b640 10 monclient(hunting): _renew_subs 2024-09-16T03:59:46.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.861+0000 7f15a1d9b640 10 monclient(hunting): authenticate will time out at 2033.834861s 2024-09-16T03:59:46.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.861+0000 7f159affd640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f159c0a1710 0x7f159c133250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:59:46.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.861+0000 7f159affd640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f159c0a1710 0x7f159c133250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:44200/0 (socket says 172.21.15.42:44200) 2024-09-16T03:59:46.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.861+0000 7f159affd640 1 -- 172.21.15.42:0/491827627 learned_addr learned my addr 172.21.15.42:0/491827627 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:59:46.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.861+0000 7f159affd640 10 monclient(hunting): get_auth_request con 0x7f159c0a1710 auth_method 0 2024-09-16T03:59:46.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.861+0000 7f159affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:59:46.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.861+0000 7f159affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:59:46.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.861+0000 7f159affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:59:46.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.861+0000 7f159bfff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f159c1340f0 0x7f159c133790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:59:46.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.861+0000 7f159bfff640 10 monclient(hunting): get_auth_request con 0x7f159c1340f0 auth_method 0 2024-09-16T03:59:46.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.861+0000 7f159bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:59:46.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.861+0000 7f159bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:59:46.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.861+0000 7f159bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:59:46.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.861+0000 7f159affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:59:46.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.861+0000 7f159affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:59:46.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.861+0000 7f159affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:59:46.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.861+0000 7f159bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:59:46.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.861+0000 7f159bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:59:46.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.861+0000 7f159bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:59:46.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.861+0000 7f159affd640 10 monclient(hunting): handle_auth_done global_id 934112 payload 824 2024-09-16T03:59:46.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.861+0000 7f159affd640 1 -- 172.21.15.42:0/491827627 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f159c0a1330 msgr2=0x7f159c1cf020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:59:46.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.861+0000 7f159b7fe640 1 --2- 172.21.15.42:0/491827627 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f159c0a1330 0x7f159c1cf020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:59:46.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.861+0000 7f159affd640 1 --2- 172.21.15.42:0/491827627 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f159c0a1330 0x7f159c1cf020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:59:46.888 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.861+0000 7f159affd640 1 -- 172.21.15.42:0/491827627 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f159c1340f0 msgr2=0x7f159c133790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:59:46.888 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.861+0000 7f159affd640 1 --2- 172.21.15.42:0/491827627 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f159c1340f0 0x7f159c133790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:59:46.888 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.861+0000 7f159affd640 10 monclient: _finish_hunting 0 2024-09-16T03:59:46.888 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.861+0000 7f159affd640 1 monclient: found mon.noname-c 2024-09-16T03:59:46.888 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.861+0000 7f159affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:59:46.888 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.861+0000 7f159affd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:59:46.888 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.861+0000 7f159affd640 1 -- 172.21.15.42:0/491827627 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f159c120c10 con 0x7f159c0a1710 2024-09-16T03:59:46.888 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.861+0000 7f159affd640 1 --2- 172.21.15.42:0/491827627 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f159c0a1710 0x7f159c133250 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f1590019f90 tx=0x7f1590019100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:59:46.888 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.861+0000 7f1583fff640 10 client.?.objecter ms_handle_connect 0x7f159c0a1710 2024-09-16T03:59:46.888 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.861+0000 7f1583fff640 10 client.?.objecter resend_mon_ops 2024-09-16T03:59:46.888 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.861+0000 7f1583fff640 1 -- 172.21.15.42:0/491827627 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15900b0040 con 0x7f159c0a1710 2024-09-16T03:59:46.888 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.861+0000 7f1583fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:59:46.888 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.861+0000 7f1583fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:59:46.889 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.861+0000 7f1583fff640 10 monclient: dump: 2024-09-16T03:59:46.889 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:59:46.889 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:59:46.889 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:46.889 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:46.889 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:59:46.889 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:59:46.889 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:59:46.889 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:59:46.889 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:59:46.889 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:59:46.889 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.861+0000 7f1583fff640 10 monclient: _finish_auth 0 2024-09-16T03:59:46.889 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.861+0000 7f1583fff640 10 monclient: _check_auth_tickets 2024-09-16T03:59:46.889 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.861+0000 7f1583fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:59:46.890 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.861+0000 7f1583fff640 1 -- 172.21.15.42:0/491827627 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f15900ab9b0 con 0x7f159c0a1710 2024-09-16T03:59:46.890 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.861+0000 7f1583fff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:59:46.890 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.861+0000 7f1583fff640 1 -- 172.21.15.42:0/491827627 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15900abd60 con 0x7f159c0a1710 2024-09-16T03:59:46.890 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.861+0000 7f1583fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:59:46.890 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.861+0000 7f1583fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:59:46.890 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.861+0000 7f1583fff640 10 monclient: dump: 2024-09-16T03:59:46.890 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:59:46.890 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:59:46.890 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:46.890 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:46.890 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:59:46.890 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:59:46.891 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:59:46.891 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:59:46.891 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:59:46.891 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:59:46.891 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.861+0000 7f15a1d9b640 5 monclient: authenticate success, global_id 934112 2024-09-16T03:59:46.891 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.861+0000 7f15a1d9b640 10 monclient: _renew_subs 2024-09-16T03:59:46.891 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.861+0000 7f15a1d9b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:59:46.891 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.861+0000 7f15a1d9b640 1 -- 172.21.15.42:0/491827627 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f159c1cf560 con 0x7f159c0a1710 2024-09-16T03:59:46.891 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.861+0000 7f1599ffb640 4 set_mon_vals no callback set 2024-09-16T03:59:46.891 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.861+0000 7f15a1d9b640 10 client.934112.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:59:46.891 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.861+0000 7f15a1d9b640 10 monclient: _renew_subs 2024-09-16T03:59:46.891 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.861+0000 7f15a1d9b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:59:46.891 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.861+0000 7f15a1d9b640 1 -- 172.21.15.42:0/491827627 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f159c073a30 con 0x7f159c0a1710 2024-09-16T03:59:46.891 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.861+0000 7f15a1d9b640 1 librados: init done 2024-09-16T03:59:46.892 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.865+0000 7f15a1d9b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:59:46.892 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.865+0000 7f15a1d9b640 1 --2- 172.21.15.42:0/491827627 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f159c11cc80 0x7f159c11d060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:59:46.892 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.865+0000 7f15a1d9b640 10 monclient(hunting): start opening mon connection 2024-09-16T03:59:46.892 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.865+0000 7f15a1d9b640 1 -- 172.21.15.42:0/491827627 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f159c09f180 con 0x7f159c11cc80 2024-09-16T03:59:46.892 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.865+0000 7f1583fff640 1 -- 172.21.15.42:0/491827627 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f1590003620 con 0x7f159c0a1710 2024-09-16T03:59:46.892 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.865+0000 7f1583fff640 1 --2- 172.21.15.42:0/491827627 >> v2:172.21.15.154:6812/254845342 conn(0x7f1588037810 0x7f1588039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:59:46.892 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.865+0000 7f159b7fe640 1 --2- 172.21.15.42:0/491827627 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f159c11cc80 0x7f159c11d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:59:46.892 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.865+0000 7f1583fff640 1 -- 172.21.15.42:0/491827627 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(540..540 src has 451..540) ==== 6168+0+0 (secure 0 0 0) 0x7f15900e2b10 con 0x7f159c0a1710 2024-09-16T03:59:46.892 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.865+0000 7f1583fff640 10 client.934112.objecter ms_dispatch 0x7f159c000ba0 osd_map(540..540 src has 451..540) 2024-09-16T03:59:46.892 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.865+0000 7f1583fff640 3 client.934112.objecter handle_osd_map got epochs [540,540] > 0 2024-09-16T03:59:46.892 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.865+0000 7f1583fff640 3 client.934112.objecter handle_osd_map decoding full epoch 540 2024-09-16T03:59:46.892 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.865+0000 7f159b7fe640 10 monclient: get_auth_request con 0x7f159c11cc80 auth_method 0 2024-09-16T03:59:46.892 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.865+0000 7f159b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:59:46.892 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.865+0000 7f159b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:59:46.893 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.865+0000 7f159b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:59:46.893 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.865+0000 7f1583fff640 20 client.934112.objecter dump_active .. 0 homeless 2024-09-16T03:59:46.893 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.865+0000 7f159b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:59:46.893 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.865+0000 7f159b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:59:46.893 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.865+0000 7f159b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:59:46.893 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.865+0000 7f159bfff640 1 --2- 172.21.15.42:0/491827627 >> v2:172.21.15.154:6812/254845342 conn(0x7f1588037810 0x7f1588039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:59:46.893 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.865+0000 7f159bfff640 10 monclient: get_auth_request con 0x7f1588037810 auth_method 0 2024-09-16T03:59:46.893 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.865+0000 7f159b7fe640 10 monclient(hunting): handle_auth_done global_id 904357 payload 824 2024-09-16T03:59:46.893 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.865+0000 7f159b7fe640 1 -- 172.21.15.42:0/491827627 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f159c11e920 con 0x7f159c11cc80 2024-09-16T03:59:46.893 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.865+0000 7f159b7fe640 1 --2- 172.21.15.42:0/491827627 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f159c11cc80 0x7f159c11d060 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f158c00e530 tx=0x7f158c00e9f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:59:46.893 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.865+0000 7f1583fff640 10 client.934112.objecter ms_handle_connect 0x7f159c11cc80 2024-09-16T03:59:46.893 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.865+0000 7f1583fff640 10 client.934112.objecter resend_mon_ops 2024-09-16T03:59:46.893 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.865+0000 7f1583fff640 10 client.934112.objecter ms_handle_connect 0x7f1588037810 2024-09-16T03:59:46.894 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.865+0000 7f159bfff640 1 --2- 172.21.15.42:0/491827627 >> v2:172.21.15.154:6812/254845342 conn(0x7f1588037810 0x7f1588039cd0 secure :-1 s=READY pgs=3734 cs=0 l=1 rev1=1 crypto rx=0x7f1594005c10 tx=0x7f1594009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:59:46.894 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.865+0000 7f1583fff640 1 -- 172.21.15.42:0/491827627 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f158c017070 con 0x7f159c11cc80 2024-09-16T03:59:46.894 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.865+0000 7f1583fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:59:46.894 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.865+0000 7f1583fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:59:46.894 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.865+0000 7f1583fff640 10 monclient: dump: 2024-09-16T03:59:46.894 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:59:46.894 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:59:46.894 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:46.894 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:46.894 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:59:46.894 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:59:46.895 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:59:46.895 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:59:46.895 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:59:46.895 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:59:46.895 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.865+0000 7f1583fff640 1 -- 172.21.15.42:0/491827627 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f158c013610 con 0x7f159c11cc80 2024-09-16T03:59:46.895 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.865+0000 7f1583fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:59:46.895 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.865+0000 7f1583fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:59:46.895 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.865+0000 7f1583fff640 1 -- 172.21.15.42:0/491827627 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f159c11cc80 msgr2=0x7f159c11d060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:59:46.895 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.865+0000 7f1583fff640 1 --2- 172.21.15.42:0/491827627 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f159c11cc80 0x7f159c11d060 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f158c00e530 tx=0x7f158c00e9f0 comp rx=0 tx=0).stop 2024-09-16T03:59:46.895 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.865+0000 7f1583fff640 1 -- 172.21.15.42:0/491827627 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f159c11cc80 msgr2=0x7f159c11d060 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:59:46.895 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.865+0000 7f1583fff640 1 --2- 172.21.15.42:0/491827627 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f159c11cc80 0x7f159c11d060 secure :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f158c00e530 tx=0x7f158c00e9f0 comp rx=0 tx=0).stop 2024-09-16T03:59:46.895 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.873+0000 7f15a1d9b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:59:46.895 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.873+0000 7f15a1d9b640 1 --2- 172.21.15.42:0/491827627 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f159c11d5a0 0x7f159c06f910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:59:46.896 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.873+0000 7f15a1d9b640 10 monclient(hunting): start opening mon connection 2024-09-16T03:59:46.896 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.873+0000 7f15a1d9b640 1 -- 172.21.15.42:0/491827627 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f159c120450 con 0x7f159c11d5a0 2024-09-16T03:59:46.896 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.873+0000 7f159b7fe640 1 --2- 172.21.15.42:0/491827627 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f159c11d5a0 0x7f159c06f910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:59:46.896 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.873+0000 7f159b7fe640 10 monclient: get_auth_request con 0x7f159c11d5a0 auth_method 0 2024-09-16T03:59:46.896 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.873+0000 7f159b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:59:46.896 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.873+0000 7f159b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:59:46.896 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.873+0000 7f159b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:59:46.896 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.873+0000 7f159b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:59:46.896 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.873+0000 7f159b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:59:46.896 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.873+0000 7f159b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:59:46.896 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.873+0000 7f159b7fe640 10 monclient(hunting): handle_auth_done global_id 904360 payload 824 2024-09-16T03:59:46.896 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.873+0000 7f159b7fe640 1 -- 172.21.15.42:0/491827627 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f159c069040 con 0x7f159c11d5a0 2024-09-16T03:59:46.896 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.873+0000 7f159b7fe640 1 --2- 172.21.15.42:0/491827627 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f159c11d5a0 0x7f159c06f910 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f158c00e530 tx=0x7f158c004740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:59:46.897 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.873+0000 7f1583fff640 10 client.934112.objecter ms_handle_connect 0x7f159c11d5a0 2024-09-16T03:59:46.897 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.873+0000 7f1583fff640 10 client.934112.objecter resend_mon_ops 2024-09-16T03:59:46.897 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.873+0000 7f1583fff640 1 -- 172.21.15.42:0/491827627 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f158c00f040 con 0x7f159c11d5a0 2024-09-16T03:59:46.897 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.873+0000 7f1583fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:59:46.897 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.873+0000 7f1583fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:59:46.897 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.873+0000 7f1583fff640 10 monclient: dump: 2024-09-16T03:59:46.897 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:59:46.897 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:59:46.897 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:46.897 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:46.897 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:59:46.898 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:59:46.898 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:59:46.898 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:59:46.898 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:59:46.898 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:59:46.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.873+0000 7f1583fff640 1 -- 172.21.15.42:0/491827627 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4713 (secure 0 0 0) 0x7f158c00c3c0 con 0x7f159c11d5a0 2024-09-16T03:59:46.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.873+0000 7f1583fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:59:46.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.873+0000 7f1583fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:59:46.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.873+0000 7f1583fff640 1 -- 172.21.15.42:0/491827627 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f159c11d5a0 msgr2=0x7f159c06f910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:59:46.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.877+0000 7f1583fff640 1 --2- 172.21.15.42:0/491827627 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f159c11d5a0 0x7f159c06f910 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f158c00e530 tx=0x7f158c004740 comp rx=0 tx=0).stop 2024-09-16T03:59:46.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.877+0000 7f1583fff640 1 -- 172.21.15.42:0/491827627 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f159c11d5a0 msgr2=0x7f159c06f910 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:59:46.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.877+0000 7f1583fff640 1 --2- 172.21.15.42:0/491827627 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f159c11d5a0 0x7f159c06f910 secure :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f158c00e530 tx=0x7f158c004740 comp rx=0 tx=0).stop 2024-09-16T03:59:46.900 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:59:46.900 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:59:46.900 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:59:46.900 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T03:59:46.900 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 160, 2024-09-16T03:59:46.900 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 115061, 2024-09-16T03:59:46.900 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:59:46.900 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:59:46.900 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T03:59:46.900 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T03:59:46.900 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:59:46.900 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 5, 2024-09-16T03:59:46.901 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:59:46.901 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:59:46.901 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:59:46.901 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:59:46.901 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:59:46.901 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:59:46.901 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:59:46.901 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:59:46.902 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:59:46.902 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:59:46.902 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:59:46.902 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:59:46.902 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:59:46.902 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:59:46.902 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:59:46.902 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:59:46.902 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:59:46.902 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:59:46.902 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:59:46.902 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:59:46.902 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:59:46.902 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:59:46.902 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:59:46.903 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:59:46.903 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:59:46.903 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:59:46.903 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:59:46.903 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:59:46.903 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:59:46.903 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:59:46.903 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:59:46.903 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:59:46.903 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:59:46.903 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:59:46.904 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:59:46.904 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:59:46.904 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:59:46.904 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:59:46.904 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:59:46.904 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:59:46.904 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:59:46.904 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:59:46.904 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:59:46.904 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:59:46.905 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:59:46.905 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:59:46.905 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:59:46.905 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:59:46.905 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:59:46.905 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:59:46.905 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:59:46.905 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:59:46.905 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:59:46.905 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:59:46.905 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:59:46.905 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:59:46.905 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:59:46.905 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:59:46.906 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:59:46.906 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:59:46.906 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:59:46.906 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:59:46.906 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:59:46.906 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:59:46.906 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:59:46.906 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:59:46.906 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:59:46.906 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:59:46.906 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:59:46.906 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:59:46.906 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:59:46.906 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:59:46.906 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:59:46.907 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:59:46.907 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:59:46.907 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:59:46.907 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:59:46.907 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:59:46.907 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:59:46.907 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:59:46.907 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:59:46.907 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:59:46.907 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:59:46.907 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:59:46.907 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:59:46.907 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:59:46.907 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:59:46.908 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:59:46.908 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:59:46.908 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:59:46.908 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:59:46.908 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:59:46.908 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:59:46.908 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:59:46.908 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:59:46.908 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:59:46.908 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:59:46.908 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:59:46.908 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:59:46.908 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:59:46.908 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:59:46.909 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:59:46.909 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:59:46.909 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:59:46.909 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:59:46.909 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:59:46.909 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:59:46.909 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:59:46.909 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:59:46.909 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:59:46.909 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:59:46.909 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:59:46.909 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:59:46.909 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:59:46.909 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:59:46.910 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:59:46.910 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:59:46.910 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:59:46.910 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:59:46.910 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:59:46.910 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:59:46.910 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:59:46.910 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:59:46.910 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:59:46.910 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:59:46.910 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:59:46.910 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:59:46.910 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:59:46.910 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:59:46.911 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:59:46.911 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:59:46.911 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:59:46.911 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:59:46.911 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:59:46.911 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:59:46.911 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:59:46.911 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:59:46.911 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:59:46.911 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:59:46.911 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:59:46.912 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:59:46.912 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T03:59:46.912 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:59:46.912 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:59:46.912 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:59:46.912 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:59:46.912 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:59:46.912 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:59:46.912 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:59:46.912 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:59:46.912 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:59:46.913 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:59:46.913 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 2 2024-09-16T03:59:46.913 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:59:46.913 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:59:46.913 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:59:46.913 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:59:46.913 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:59:46.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.877+0000 7f15a1d9b640 10 librados: watch_flush enter 2024-09-16T03:59:46.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.877+0000 7f15a1d9b640 10 librados: watch_flush exit 2024-09-16T03:59:46.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.877+0000 7f15a1d9b640 20 client.934112.objecter shutdown clearing up homeless session... 2024-09-16T03:59:46.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.877+0000 7f15a1d9b640 10 client.934112.objecter successfully canceled tick 2024-09-16T03:59:46.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.877+0000 7f15a1d9b640 1 -- 172.21.15.42:0/491827627 >> v2:172.21.15.154:6812/254845342 conn(0x7f1588037810 msgr2=0x7f1588039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:59:46.916 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.877+0000 7f15a1d9b640 1 --2- 172.21.15.42:0/491827627 >> v2:172.21.15.154:6812/254845342 conn(0x7f1588037810 0x7f1588039cd0 secure :-1 s=READY pgs=3734 cs=0 l=1 rev1=1 crypto rx=0x7f1594005c10 tx=0x7f1594009040 comp rx=0 tx=0).stop 2024-09-16T03:59:46.916 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.877+0000 7f15a1d9b640 10 monclient: shutdown 2024-09-16T03:59:46.916 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.877+0000 7f15a1d9b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:59:46.916 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.877+0000 7f15a1d9b640 1 -- 172.21.15.42:0/491827627 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f159c0a1710 msgr2=0x7f159c133250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:59:46.916 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.877+0000 7f159bfff640 1 -- 172.21.15.42:0/491827627 reap_dead start 2024-09-16T03:59:46.916 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.877+0000 7f15a1d9b640 1 --2- 172.21.15.42:0/491827627 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f159c0a1710 0x7f159c133250 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f1590019f90 tx=0x7f1590019100 comp rx=0 tx=0).stop 2024-09-16T03:59:46.916 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.877+0000 7f15a1d9b640 1 -- 172.21.15.42:0/491827627 shutdown_connections 2024-09-16T03:59:46.916 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.877+0000 7f15a1d9b640 1 --2- 172.21.15.42:0/491827627 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f159c0a1710 0x7f159c133250 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:59:46.916 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.877+0000 7f15a1d9b640 1 -- 172.21.15.42:0/491827627 >> 172.21.15.42:0/491827627 conn(0x7f159c0a2420 msgr2=0x7f159c09ba50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:59:46.916 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.877+0000 7f15a1d9b640 1 -- 172.21.15.42:0/491827627 shutdown_connections 2024-09-16T03:59:46.917 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.877+0000 7f15a1d9b640 1 -- 172.21.15.42:0/491827627 wait complete. 2024-09-16T03:59:46.917 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:46.877+0000 7f15a1d9b640 1 librados: shutdown 2024-09-16T03:59:46.917 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:59:46.917 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T03:59:46.917 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T03:59:47.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.025+0000 7f3db2a4b640 10 monclient: get_monmap_and_config 2024-09-16T03:59:47.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.025+0000 7f3db2a4b640 10 monclient: build_initial_monmap 2024-09-16T03:59:47.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.025+0000 7f3db2a4b640 10 monclient: monmap: 2024-09-16T03:59:47.031 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:59:47.031 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:59:47.031 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:59:47.029578+0000 2024-09-16T03:59:47.031 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:59:47.029578+0000 2024-09-16T03:59:47.032 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:59:47.032 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:59:47.032 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:59:47.032 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:59:47.032 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:59:47.033 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:59:47.033 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:59:47.033 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:59:47.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.025+0000 7f3db2a4b640 1 Processor -- start 2024-09-16T03:59:47.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.025+0000 7f3db2a4b640 1 -- start start 2024-09-16T03:59:47.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.025+0000 7f3db2a4b640 10 monclient: init 2024-09-16T03:59:47.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.025+0000 7f3db2a4b640 10 monclient: _reopen_session rank -1 2024-09-16T03:59:47.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.025+0000 7f3db2a4b640 10 monclient: _add_conns ranks=[4,2,1,3,0] 2024-09-16T03:59:47.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.025+0000 7f3db2a4b640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f3dac0a1330 0x7f3dac09f180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:59:47.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.025+0000 7f3db2a4b640 10 monclient(hunting): picked mon.noname-c con 0x7f3dac0a1330 addr v2:172.21.15.154:3300/0 2024-09-16T03:59:47.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.025+0000 7f3db2a4b640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3dac0a1710 0x7f3dac0a1b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:59:47.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.025+0000 7f3db2a4b640 10 monclient(hunting): picked mon.noname-b con 0x7f3dac0a1710 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:59:47.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.025+0000 7f3db2a4b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3dac1340f0 addr v1:172.21.15.154:6789/0 2024-09-16T03:59:47.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.025+0000 7f3db2a4b640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3dac09e4c0 con 0x7f3dac1340f0 2024-09-16T03:59:47.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.025+0000 7f3db2a4b640 10 monclient(hunting): start opening mon connection 2024-09-16T03:59:47.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.025+0000 7f3db2a4b640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f3dac09edf0 con 0x7f3dac0a1330 2024-09-16T03:59:47.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.025+0000 7f3db2a4b640 10 monclient(hunting): start opening mon connection 2024-09-16T03:59:47.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.025+0000 7f3db2a4b640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f3dac09cd60 con 0x7f3dac0a1710 2024-09-16T03:59:47.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.025+0000 7f3db2a4b640 10 monclient(hunting): _renew_subs 2024-09-16T03:59:47.039 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.025+0000 7f3db2a4b640 10 monclient(hunting): authenticate will time out at 2033.999660s 2024-09-16T03:59:47.039 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.025+0000 7f3da3fff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3dac0a1710 0x7f3dac0a1b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:59:47.039 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.025+0000 7f3da3fff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3dac0a1710 0x7f3dac0a1b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:44206/0 (socket says 172.21.15.42:44206) 2024-09-16T03:59:47.039 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.025+0000 7f3da3fff640 1 -- 172.21.15.42:0/3811806818 learned_addr learned my addr 172.21.15.42:0/3811806818 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:59:47.039 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.025+0000 7f3da3fff640 10 monclient(hunting): get_auth_request con 0x7f3dac0a1710 auth_method 0 2024-09-16T03:59:47.039 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.025+0000 7f3da3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:59:47.039 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.025+0000 7f3da3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:59:47.039 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.025+0000 7f3da3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:59:47.039 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.029+0000 7f3da3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:59:47.039 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.029+0000 7f3da3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:59:47.039 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.029+0000 7f3da3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:59:47.039 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.029+0000 7f3dabfff640 1 --2- 172.21.15.42:0/3811806818 >> v2:172.21.15.154:3300/0 conn(0x7f3dac0a1330 0x7f3dac09f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:59:47.039 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.029+0000 7f3dabfff640 10 monclient(hunting): get_auth_request con 0x7f3dac0a1330 auth_method 0 2024-09-16T03:59:47.039 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.029+0000 7f3dabfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:59:47.040 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.029+0000 7f3dabfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:59:47.040 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.029+0000 7f3dabfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:59:47.040 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.029+0000 7f3da3fff640 10 monclient(hunting): handle_auth_done global_id 934115 payload 274 2024-09-16T03:59:47.040 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.029+0000 7f3da3fff640 1 -- 172.21.15.42:0/3811806818 >> v2:172.21.15.154:3300/0 conn(0x7f3dac0a1330 msgr2=0x7f3dac09f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:59:47.040 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.029+0000 7f3da3fff640 1 --2- 172.21.15.42:0/3811806818 >> v2:172.21.15.154:3300/0 conn(0x7f3dac0a1330 0x7f3dac09f180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:59:47.040 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.029+0000 7f3da3fff640 1 -- 172.21.15.42:0/3811806818 >> v1:172.21.15.154:6789/0 conn(0x7f3dac1340f0 legacy=0x7f3dac1345a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:59:47.040 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.029+0000 7f3da3fff640 10 monclient: _finish_hunting 0 2024-09-16T03:59:47.040 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.029+0000 7f3da3fff640 1 monclient: found mon.noname-b 2024-09-16T03:59:47.040 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.029+0000 7f3da3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:59:47.040 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.029+0000 7f3da3fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T03:59:47.040 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.029+0000 7f3da3fff640 1 -- 172.21.15.42:0/3811806818 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3dac0a2050 con 0x7f3dac0a1710 2024-09-16T03:59:47.040 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.029+0000 7f3da3fff640 1 --2- 172.21.15.42:0/3811806818 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3dac0a1710 0x7f3dac0a1b10 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f3d940098a0 tx=0x7f3d9401c3c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dbdb11d3a776154a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:59:47.040 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.029+0000 7f3dab7fe640 1 -- 172.21.15.42:0/3811806818 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d940b0070 con 0x7f3dac0a1710 2024-09-16T03:59:47.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.029+0000 7f3dab7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:59:47.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.029+0000 7f3dab7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:59:47.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.029+0000 7f3dab7fe640 10 monclient: dump: 2024-09-16T03:59:47.041 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:59:47.041 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:59:47.042 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:47.042 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:47.042 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:59:47.042 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:59:47.042 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:59:47.042 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:59:47.042 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:59:47.042 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:59:47.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.029+0000 7f3dab7fe640 10 monclient: _finish_auth 0 2024-09-16T03:59:47.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.029+0000 7f3dab7fe640 10 monclient: _check_auth_tickets 2024-09-16T03:59:47.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.029+0000 7f3dab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:59:47.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.029+0000 7f3dab7fe640 1 -- 172.21.15.42:0/3811806818 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d940ab600 con 0x7f3dac0a1710 2024-09-16T03:59:47.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.029+0000 7f3dab7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:59:47.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.029+0000 7f3dab7fe640 1 -- 172.21.15.42:0/3811806818 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d940ab9d0 con 0x7f3dac0a1710 2024-09-16T03:59:47.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.029+0000 7f3dab7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:59:47.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.029+0000 7f3dab7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:59:47.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.029+0000 7f3dab7fe640 10 monclient: dump: 2024-09-16T03:59:47.044 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:59:47.044 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:59:47.044 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:47.044 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:47.044 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:59:47.044 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:59:47.044 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:59:47.044 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:59:47.044 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:59:47.045 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:59:47.045 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.029+0000 7f3db2a4b640 5 monclient: authenticate success, global_id 934115 2024-09-16T03:59:47.045 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.029+0000 7f3db2a4b640 10 monclient: get_monmap_and_config success 2024-09-16T03:59:47.045 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.029+0000 7f3db2a4b640 4 set_mon_vals no callback set 2024-09-16T03:59:47.045 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.033+0000 7f3db2a4b640 10 monclient: shutdown 2024-09-16T03:59:47.045 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.033+0000 7f3db2a4b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:59:47.045 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.033+0000 7f3db2a4b640 1 -- 172.21.15.42:0/3811806818 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3dac0a1710 msgr2=0x7f3dac0a1b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:59:47.045 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.033+0000 7f3db2a4b640 1 --2- 172.21.15.42:0/3811806818 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3dac0a1710 0x7f3dac0a1b10 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f3d940098a0 tx=0x7f3d9401c3c0 comp rx=0 tx=0).stop 2024-09-16T03:59:47.045 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.033+0000 7f3db2a4b640 1 -- 172.21.15.42:0/3811806818 shutdown_connections 2024-09-16T03:59:47.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.033+0000 7f3db2a4b640 1 --2- 172.21.15.42:0/3811806818 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3dac0a1710 0x7f3dac0a1b10 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:59:47.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.033+0000 7f3db2a4b640 1 --2- 172.21.15.42:0/3811806818 >> v2:172.21.15.154:3300/0 conn(0x7f3dac0a1330 0x7f3dac09f180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:59:47.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.033+0000 7f3db2a4b640 1 -- 172.21.15.42:0/3811806818 >> 172.21.15.42:0/3811806818 conn(0x7f3dac0a2420 msgr2=0x7f3dac0a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:59:47.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.033+0000 7f3db2a4b640 1 -- 172.21.15.42:0/3811806818 shutdown_connections 2024-09-16T03:59:47.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.033+0000 7f3db2a4b640 1 -- 172.21.15.42:0/3811806818 wait complete. 2024-09-16T03:59:47.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.033+0000 7f3db2a4b640 10 monclient: build_initial_monmap 2024-09-16T03:59:47.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.033+0000 7f3db2a4b640 10 monclient: monmap: 2024-09-16T03:59:47.054 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:59:47.055 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:59:47.055 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:59:47.055 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:59:47.055 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:59:47.055 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:59:47.055 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:59:47.056 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:59:47.056 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:59:47.056 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:59:47.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.033+0000 7f3db2a4b640 1 librados: starting msgr at 2024-09-16T03:59:47.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.033+0000 7f3db2a4b640 1 librados: starting objecter 2024-09-16T03:59:47.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.033+0000 7f3db2a4b640 1 Processor -- start 2024-09-16T03:59:47.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.033+0000 7f3db2a4b640 1 -- start start 2024-09-16T03:59:47.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.033+0000 7f3db2a4b640 1 librados: setting wanted keys 2024-09-16T03:59:47.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.033+0000 7f3db2a4b640 1 librados: calling monclient init 2024-09-16T03:59:47.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.033+0000 7f3db2a4b640 10 monclient: init 2024-09-16T03:59:47.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.033+0000 7f3db2a4b640 10 monclient: _reopen_session rank -1 2024-09-16T03:59:47.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.033+0000 7f3db2a4b640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T03:59:47.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.033+0000 7f3db2a4b640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3dac0a1330 0x7f3dac09ba50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:59:47.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.033+0000 7f3db2a4b640 10 monclient(hunting): picked mon.noname-c con 0x7f3dac0a1330 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:59:47.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.033+0000 7f3db2a4b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3dac0a1710 0x7f3dac133440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:59:47.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.033+0000 7f3db2a4b640 10 monclient(hunting): picked mon.noname-b con 0x7f3dac0a1710 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:59:47.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.033+0000 7f3db2a4b640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3dac1340f0 0x7f3dac133980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:59:47.059 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.033+0000 7f3db2a4b640 10 monclient(hunting): picked mon.noname-a con 0x7f3dac1340f0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:59:47.059 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.033+0000 7f3db2a4b640 10 monclient(hunting): start opening mon connection 2024-09-16T03:59:47.059 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.033+0000 7f3db2a4b640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f3dac13ed60 con 0x7f3dac1340f0 2024-09-16T03:59:47.059 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.033+0000 7f3db2a4b640 10 monclient(hunting): start opening mon connection 2024-09-16T03:59:47.059 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.033+0000 7f3db2a4b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f3dac13ebe0 con 0x7f3dac0a1710 2024-09-16T03:59:47.060 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.033+0000 7f3db2a4b640 10 monclient(hunting): start opening mon connection 2024-09-16T03:59:47.060 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.033+0000 7f3db2a4b640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f3dac13eee0 con 0x7f3dac0a1330 2024-09-16T03:59:47.060 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.033+0000 7f3db2a4b640 10 monclient(hunting): _renew_subs 2024-09-16T03:59:47.060 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.033+0000 7f3db2a4b640 10 monclient(hunting): authenticate will time out at 2034.005694s 2024-09-16T03:59:47.060 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.033+0000 7f3dabfff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3dac0a1330 0x7f3dac09ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:59:47.061 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.033+0000 7f3db0fe8640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3dac1340f0 0x7f3dac133980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:59:47.061 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.033+0000 7f3dabfff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3dac0a1330 0x7f3dac09ba50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:44220/0 (socket says 172.21.15.42:44220) 2024-09-16T03:59:47.061 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.033+0000 7f3dabfff640 1 -- 172.21.15.42:0/1177799955 learned_addr learned my addr 172.21.15.42:0/1177799955 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:59:47.061 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.033+0000 7f3dabfff640 10 monclient(hunting): get_auth_request con 0x7f3dac0a1330 auth_method 0 2024-09-16T03:59:47.061 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.033+0000 7f3dabfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:59:47.061 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.033+0000 7f3dabfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:59:47.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.033+0000 7f3dabfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:59:47.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.033+0000 7f3db0fe8640 10 monclient(hunting): get_auth_request con 0x7f3dac1340f0 auth_method 0 2024-09-16T03:59:47.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.033+0000 7f3db0fe8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:59:47.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.033+0000 7f3db0fe8640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:59:47.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.033+0000 7f3db0fe8640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:59:47.063 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.033+0000 7f3dabfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:59:47.063 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.033+0000 7f3dabfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:59:47.063 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.033+0000 7f3dabfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:59:47.063 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.033+0000 7f3db0fe8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:59:47.063 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.033+0000 7f3db0fe8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:59:47.063 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.033+0000 7f3db0fe8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:59:47.064 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.033+0000 7f3da3fff640 1 --2- 172.21.15.42:0/1177799955 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3dac0a1710 0x7f3dac133440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:59:47.064 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.033+0000 7f3da3fff640 10 monclient(hunting): get_auth_request con 0x7f3dac0a1710 auth_method 0 2024-09-16T03:59:47.064 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.033+0000 7f3da3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:59:47.064 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.033+0000 7f3da3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:59:47.064 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.033+0000 7f3da3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:59:47.064 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.033+0000 7f3dabfff640 10 monclient(hunting): handle_auth_done global_id 934118 payload 824 2024-09-16T03:59:47.065 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.033+0000 7f3dabfff640 1 -- 172.21.15.42:0/1177799955 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3dac0a1710 msgr2=0x7f3dac133440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:59:47.065 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.033+0000 7f3dabfff640 1 --2- 172.21.15.42:0/1177799955 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3dac0a1710 0x7f3dac133440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:59:47.065 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.033+0000 7f3dabfff640 1 -- 172.21.15.42:0/1177799955 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3dac1340f0 msgr2=0x7f3dac133980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:59:47.065 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.033+0000 7f3dabfff640 1 --2- 172.21.15.42:0/1177799955 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3dac1340f0 0x7f3dac133980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:59:47.065 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.033+0000 7f3dabfff640 10 monclient: _finish_hunting 0 2024-09-16T03:59:47.065 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.033+0000 7f3dabfff640 1 monclient: found mon.noname-c 2024-09-16T03:59:47.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.033+0000 7f3dabfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:59:47.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.033+0000 7f3dabfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:59:47.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.033+0000 7f3dabfff640 1 -- 172.21.15.42:0/1177799955 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3dac072560 con 0x7f3dac0a1330 2024-09-16T03:59:47.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.033+0000 7f3db0fe8640 1 --2- 172.21.15.42:0/1177799955 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3dac1340f0 0x7f3dac133980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:59:47.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.033+0000 7f3da3fff640 1 --2- 172.21.15.42:0/1177799955 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3dac0a1710 0x7f3dac133440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T03:59:47.067 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.033+0000 7f3dabfff640 1 --2- 172.21.15.42:0/1177799955 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3dac0a1330 0x7f3dac09ba50 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f3d9c00d830 tx=0x7f3d9c00dcf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:59:47.067 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.033+0000 7f3da97fa640 10 client.?.objecter ms_handle_connect 0x7f3dac0a1330 2024-09-16T03:59:47.067 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.033+0000 7f3da97fa640 10 client.?.objecter resend_mon_ops 2024-09-16T03:59:47.067 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.033+0000 7f3da97fa640 1 -- 172.21.15.42:0/1177799955 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d9c00b6c0 con 0x7f3dac0a1330 2024-09-16T03:59:47.067 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.033+0000 7f3da97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:59:47.067 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.033+0000 7f3da97fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:59:47.067 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.033+0000 7f3da97fa640 10 monclient: dump: 2024-09-16T03:59:47.067 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:59:47.068 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:59:47.068 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:47.068 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:47.068 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:59:47.068 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:59:47.068 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:59:47.068 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:59:47.069 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:59:47.069 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:59:47.069 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.033+0000 7f3da97fa640 10 monclient: _finish_auth 0 2024-09-16T03:59:47.069 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.033+0000 7f3da97fa640 10 monclient: _check_auth_tickets 2024-09-16T03:59:47.069 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.033+0000 7f3da97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:59:47.069 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.033+0000 7f3da97fa640 1 -- 172.21.15.42:0/1177799955 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d9c019070 con 0x7f3dac0a1330 2024-09-16T03:59:47.069 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.033+0000 7f3db2a4b640 5 monclient: authenticate success, global_id 934118 2024-09-16T03:59:47.069 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.033+0000 7f3db2a4b640 10 monclient: _renew_subs 2024-09-16T03:59:47.070 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.033+0000 7f3db2a4b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:59:47.070 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.033+0000 7f3db2a4b640 1 -- 172.21.15.42:0/1177799955 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3dac073a30 con 0x7f3dac0a1330 2024-09-16T03:59:47.070 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.033+0000 7f3da97fa640 10 monclient: handle_config config(0 keys) 2024-09-16T03:59:47.070 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.033+0000 7f3da97fa640 1 -- 172.21.15.42:0/1177799955 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d9c00bae0 con 0x7f3dac0a1330 2024-09-16T03:59:47.070 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.033+0000 7f3da97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:59:47.070 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.033+0000 7f3da97fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:59:47.070 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.033+0000 7f3da97fa640 10 monclient: dump: 2024-09-16T03:59:47.070 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:59:47.071 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:59:47.071 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:47.071 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:47.071 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:59:47.071 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:59:47.071 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:59:47.071 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:59:47.072 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:59:47.072 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:59:47.072 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.033+0000 7f3db2a4b640 10 client.934118.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:59:47.072 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.033+0000 7f3db2a4b640 10 monclient: _renew_subs 2024-09-16T03:59:47.072 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.033+0000 7f3daaffd640 4 set_mon_vals no callback set 2024-09-16T03:59:47.072 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.033+0000 7f3db2a4b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:59:47.072 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.033+0000 7f3db2a4b640 1 -- 172.21.15.42:0/1177799955 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f3dac120b20 con 0x7f3dac0a1330 2024-09-16T03:59:47.072 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.033+0000 7f3db2a4b640 1 librados: init done 2024-09-16T03:59:47.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.033+0000 7f3db2a4b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:59:47.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.033+0000 7f3db2a4b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:59:47.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.033+0000 7f3db2a4b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:59:47.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.033+0000 7f3da97fa640 1 -- 172.21.15.42:0/1177799955 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f3d9c00bc80 con 0x7f3dac0a1330 2024-09-16T03:59:47.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.033+0000 7f3db2a4b640 1 -- 172.21.15.42:0/1177799955 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3dac09f180 con 0x7f3dac0a1330 2024-09-16T03:59:47.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.033+0000 7f3da97fa640 1 --2- 172.21.15.42:0/1177799955 >> v2:172.21.15.154:6812/254845342 conn(0x7f3d8c037810 0x7f3d8c039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:59:47.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.033+0000 7f3da97fa640 1 -- 172.21.15.42:0/1177799955 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(540..540 src has 451..540) ==== 6168+0+0 (secure 0 0 0) 0x7f3d9c0495a0 con 0x7f3dac0a1330 2024-09-16T03:59:47.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.033+0000 7f3da97fa640 10 client.934118.objecter ms_dispatch 0x7f3dac000ba0 osd_map(540..540 src has 451..540) 2024-09-16T03:59:47.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.033+0000 7f3da97fa640 3 client.934118.objecter handle_osd_map got epochs [540,540] > 0 2024-09-16T03:59:47.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.033+0000 7f3da97fa640 3 client.934118.objecter handle_osd_map decoding full epoch 540 2024-09-16T03:59:47.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.037+0000 7f3da97fa640 20 client.934118.objecter dump_active .. 0 homeless 2024-09-16T03:59:47.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.037+0000 7f3da3fff640 1 --2- 172.21.15.42:0/1177799955 >> v2:172.21.15.154:6812/254845342 conn(0x7f3d8c037810 0x7f3d8c039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:59:47.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.037+0000 7f3da3fff640 10 monclient: get_auth_request con 0x7f3d8c037810 auth_method 0 2024-09-16T03:59:47.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.037+0000 7f3da97fa640 10 client.934118.objecter ms_handle_connect 0x7f3d8c037810 2024-09-16T03:59:47.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.037+0000 7f3da3fff640 1 --2- 172.21.15.42:0/1177799955 >> v2:172.21.15.154:6812/254845342 conn(0x7f3d8c037810 0x7f3d8c039cd0 secure :-1 s=READY pgs=3735 cs=0 l=1 rev1=1 crypto rx=0x7f3d940ad000 tx=0x7f3d94008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:59:47.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.041+0000 7f3da97fa640 1 -- 172.21.15.42:0/1177799955 <== mon.2 v2:172.21.15.42:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f3d9c04bd20 con 0x7f3dac0a1330 2024-09-16T03:59:47.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.041+0000 7f3da97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:59:47.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.041+0000 7f3da97fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:59:47.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.265+0000 7f3db2a4b640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T03:59:47.271 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.265+0000 7f3db2a4b640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T03:59:47.271 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.265+0000 7f3db2a4b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:59:47.271 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.265+0000 7f3db2a4b640 1 -- 172.21.15.42:0/1177799955 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f3dac09c5f0 con 0x7f3dac0a1330 2024-09-16T03:59:47.272 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.265+0000 7f3da97fa640 1 -- 172.21.15.42:0/1177799955 <== mon.2 v2:172.21.15.42:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f3d9c01e890 con 0x7f3dac0a1330 2024-09-16T03:59:47.272 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.265+0000 7f3da97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T03:59:47.273 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.265+0000 7f3da97fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:59:47.274 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.269+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.274 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.269+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=25,config=2,health=11,kv=7,logm=55} crc {auth=2324573606,config=692619927,health=2651125269,kv=3538517832,logm=397844274}) 2024-09-16T03:59:47.274 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.269+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=27,config=2,health=11,kv=8,logm=52} crc {auth=3794679188,config=692619927,health=951188301,kv=2024801497,logm=249912215}) 2024-09-16T03:59:47.274 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.269+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.274 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.269+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=25,config=2,health=11,kv=7,logm=55} crc {auth=2324573606,config=692619927,health=2651125269,kv=3538517832,logm=397844274}) 2024-09-16T03:59:47.274 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.269+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=28,config=2,health=11,kv=10,logm=49} crc {auth=1669457763,config=692619927,health=2510701432,kv=3446188498,logm=1185498153}) 2024-09-16T03:59:47.275 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.269+0000 7f3db2a4b640 10 librados: watch_flush enter 2024-09-16T03:59:47.275 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.269+0000 7f3db2a4b640 10 librados: watch_flush exit 2024-09-16T03:59:47.275 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.269+0000 7f3db2a4b640 20 client.934118.objecter shutdown clearing up homeless session... 2024-09-16T03:59:47.275 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.269+0000 7f3db2a4b640 10 client.934118.objecter successfully canceled tick 2024-09-16T03:59:47.275 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.269+0000 7f3db2a4b640 1 -- 172.21.15.42:0/1177799955 >> v2:172.21.15.154:6812/254845342 conn(0x7f3d8c037810 msgr2=0x7f3d8c039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:59:47.275 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.269+0000 7f3db2a4b640 1 --2- 172.21.15.42:0/1177799955 >> v2:172.21.15.154:6812/254845342 conn(0x7f3d8c037810 0x7f3d8c039cd0 secure :-1 s=READY pgs=3735 cs=0 l=1 rev1=1 crypto rx=0x7f3d940ad000 tx=0x7f3d94008040 comp rx=0 tx=0).stop 2024-09-16T03:59:47.275 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.269+0000 7f3db2a4b640 10 monclient: shutdown 2024-09-16T03:59:47.276 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.269+0000 7f3db2a4b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:59:47.276 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.269+0000 7f3db2a4b640 1 -- 172.21.15.42:0/1177799955 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3dac0a1330 msgr2=0x7f3dac09ba50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:59:47.276 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.269+0000 7f3db2a4b640 1 --2- 172.21.15.42:0/1177799955 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3dac0a1330 0x7f3dac09ba50 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f3d9c00d830 tx=0x7f3d9c00dcf0 comp rx=0 tx=0).stop 2024-09-16T03:59:47.276 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.269+0000 7f3db2a4b640 1 -- 172.21.15.42:0/1177799955 shutdown_connections 2024-09-16T03:59:47.276 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.269+0000 7f3db2a4b640 1 --2- 172.21.15.42:0/1177799955 >> v2:172.21.15.154:6812/254845342 conn(0x7f3d8c037810 0x7f3d8c039cd0 unknown :-1 s=CLOSED pgs=3735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:59:47.276 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.269+0000 7f3db2a4b640 1 --2- 172.21.15.42:0/1177799955 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3dac1340f0 0x7f3dac133980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:59:47.276 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.269+0000 7f3db2a4b640 1 --2- 172.21.15.42:0/1177799955 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3dac0a1710 0x7f3dac133440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:59:47.276 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.269+0000 7f3db2a4b640 1 --2- 172.21.15.42:0/1177799955 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3dac0a1330 0x7f3dac09ba50 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:59:47.276 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.269+0000 7f3db2a4b640 1 -- 172.21.15.42:0/1177799955 >> 172.21.15.42:0/1177799955 conn(0x7f3dac0a2420 msgr2=0x7f3dac06d900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:59:47.276 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.269+0000 7f3db2a4b640 1 -- 172.21.15.42:0/1177799955 shutdown_connections 2024-09-16T03:59:47.276 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.269+0000 7f3db2a4b640 1 -- 172.21.15.42:0/1177799955 wait complete. 2024-09-16T03:59:47.276 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:47.269+0000 7f3db2a4b640 1 librados: shutdown 2024-09-16T03:59:47.277 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.269+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.277 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.269+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=400630121}) 2024-09-16T03:59:47.277 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.269+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3170013489}) 2024-09-16T03:59:47.277 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.269+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.277 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.269+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=400630121}) 2024-09-16T03:59:47.277 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.269+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4178680454}) 2024-09-16T03:59:47.280 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.273+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.280 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.273+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3062887525}) 2024-09-16T03:59:47.280 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.273+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1751489602}) 2024-09-16T03:59:47.280 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.273+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.280 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.273+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3062887525}) 2024-09-16T03:59:47.280 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.273+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3323923947}) 2024-09-16T03:59:47.283 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.277+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.283 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.277+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3960637490}) 2024-09-16T03:59:47.283 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.277+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3259399287}) 2024-09-16T03:59:47.283 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.277+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.283 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.277+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3960637490}) 2024-09-16T03:59:47.283 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.277+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2574373197}) 2024-09-16T03:59:47.286 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.281+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.286 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.281+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1610414243}) 2024-09-16T03:59:47.286 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.281+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3608499168}) 2024-09-16T03:59:47.286 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.281+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.286 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.281+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1610414243}) 2024-09-16T03:59:47.286 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.281+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=815259654}) 2024-09-16T03:59:47.288 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.281+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.288 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.281+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2314369840}) 2024-09-16T03:59:47.288 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.281+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3488724173}) 2024-09-16T03:59:47.288 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.281+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.288 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.281+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2314369840}) 2024-09-16T03:59:47.288 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.281+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3538104960}) 2024-09-16T03:59:47.290 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.285+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.290 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.285+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1160800226}) 2024-09-16T03:59:47.290 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.285+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=823239549}) 2024-09-16T03:59:47.290 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.285+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.290 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.285+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1160800226}) 2024-09-16T03:59:47.290 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.285+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2645983946}) 2024-09-16T03:59:47.291 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T03:59:47.292 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.285+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.292 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.285+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2529724554}) 2024-09-16T03:59:47.292 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.285+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=878681968}) 2024-09-16T03:59:47.292 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.285+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.292 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.285+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2529724554}) 2024-09-16T03:59:47.292 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.285+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3435900142}) 2024-09-16T03:59:47.293 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.289+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.293 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.289+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2747907746}) 2024-09-16T03:59:47.293 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.289+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4077976298}) 2024-09-16T03:59:47.293 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.289+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.294 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.289+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2747907746}) 2024-09-16T03:59:47.294 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.289+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3319625204}) 2024-09-16T03:59:47.297 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.289+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.297 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.289+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=338661697}) 2024-09-16T03:59:47.297 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.289+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1902676328}) 2024-09-16T03:59:47.297 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.289+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.297 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.289+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=338661697}) 2024-09-16T03:59:47.298 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.289+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=860648371}) 2024-09-16T03:59:47.298 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.289+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.298 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.289+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2428025349}) 2024-09-16T03:59:47.298 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.289+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=489037911}) 2024-09-16T03:59:47.298 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.289+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.298 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.289+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2428025349}) 2024-09-16T03:59:47.298 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.289+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1351459562}) 2024-09-16T03:59:47.299 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.293+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.299 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.293+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=965519100}) 2024-09-16T03:59:47.299 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.293+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2569748843}) 2024-09-16T03:59:47.299 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.293+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.299 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.293+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=965519100}) 2024-09-16T03:59:47.299 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.293+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2696093100}) 2024-09-16T03:59:47.301 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.293+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.301 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.293+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3788541670}) 2024-09-16T03:59:47.301 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.293+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1750583793}) 2024-09-16T03:59:47.301 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.293+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.301 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.293+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3788541670}) 2024-09-16T03:59:47.301 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.293+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=910291840}) 2024-09-16T03:59:47.304 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.297+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.304 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.297+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3985540500}) 2024-09-16T03:59:47.304 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.297+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3140324458}) 2024-09-16T03:59:47.304 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.297+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.304 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.297+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3985540500}) 2024-09-16T03:59:47.304 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.297+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=439848065}) 2024-09-16T03:59:47.307 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.301+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.307 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.301+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2733924625}) 2024-09-16T03:59:47.307 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.301+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2918595089}) 2024-09-16T03:59:47.307 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.301+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.307 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.301+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2733924625}) 2024-09-16T03:59:47.308 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.301+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2304993089}) 2024-09-16T03:59:47.309 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.305+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.309 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.305+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2409739196}) 2024-09-16T03:59:47.310 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.305+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4114670196}) 2024-09-16T03:59:47.310 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.305+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.310 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.305+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2409739196}) 2024-09-16T03:59:47.310 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.305+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4169922564}) 2024-09-16T03:59:47.312 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.305+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.312 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.305+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4064460362}) 2024-09-16T03:59:47.312 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.305+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1603393557}) 2024-09-16T03:59:47.312 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.305+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.312 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.305+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4064460362}) 2024-09-16T03:59:47.312 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.305+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=308133345}) 2024-09-16T03:59:47.314 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.309+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.314 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.309+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1150899495}) 2024-09-16T03:59:47.314 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.309+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=10821679}) 2024-09-16T03:59:47.314 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.309+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.314 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.309+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1150899495}) 2024-09-16T03:59:47.314 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.309+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3616480681}) 2024-09-16T03:59:47.316 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.309+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.316 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.309+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1888271917}) 2024-09-16T03:59:47.316 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.309+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3934057600}) 2024-09-16T03:59:47.316 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.309+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.317 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.309+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1888271917}) 2024-09-16T03:59:47.317 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.309+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3831393294}) 2024-09-16T03:59:47.318 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.313+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.319 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.313+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1925316261}) 2024-09-16T03:59:47.319 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.313+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4015075533}) 2024-09-16T03:59:47.319 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.313+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.319 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.313+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1925316261}) 2024-09-16T03:59:47.319 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.313+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=932208826}) 2024-09-16T03:59:47.321 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.313+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.321 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.313+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=725862056}) 2024-09-16T03:59:47.321 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.313+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1319789534}) 2024-09-16T03:59:47.321 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.313+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.321 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.313+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=725862056}) 2024-09-16T03:59:47.321 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.313+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3991369283}) 2024-09-16T03:59:47.323 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.317+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.323 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.317+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=838134264}) 2024-09-16T03:59:47.323 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.317+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3722416712}) 2024-09-16T03:59:47.323 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.317+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.324 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.317+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=838134264}) 2024-09-16T03:59:47.324 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.317+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3731430169}) 2024-09-16T03:59:47.325 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.317+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.325 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.317+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1340611149}) 2024-09-16T03:59:47.325 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.317+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2781085317}) 2024-09-16T03:59:47.325 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.317+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.326 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.317+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1340611149}) 2024-09-16T03:59:47.326 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.317+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2740186791}) 2024-09-16T03:59:47.327 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.321+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.327 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.321+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3534081875}) 2024-09-16T03:59:47.327 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.321+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4109047106}) 2024-09-16T03:59:47.328 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.321+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.328 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.321+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3534081875}) 2024-09-16T03:59:47.328 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.321+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2606653331}) 2024-09-16T03:59:47.329 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.321+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.330 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.321+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=688158836}) 2024-09-16T03:59:47.330 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.321+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=280587490}) 2024-09-16T03:59:47.330 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.321+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.330 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.321+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=688158836}) 2024-09-16T03:59:47.330 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.321+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2461671098}) 2024-09-16T03:59:47.332 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.325+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.332 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.325+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3504218217}) 2024-09-16T03:59:47.332 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.325+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2290587369}) 2024-09-16T03:59:47.332 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.325+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.332 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.325+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3504218217}) 2024-09-16T03:59:47.333 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.325+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3057586955}) 2024-09-16T03:59:47.335 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.325+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.335 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.325+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1734641018}) 2024-09-16T03:59:47.335 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.325+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1548146949}) 2024-09-16T03:59:47.336 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.325+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.336 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.325+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1734641018}) 2024-09-16T03:59:47.336 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.325+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3666778372}) 2024-09-16T03:59:47.336 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.329+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.336 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.329+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=915396323}) 2024-09-16T03:59:47.336 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.329+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2633276610}) 2024-09-16T03:59:47.336 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.329+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.337 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.329+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=915396323}) 2024-09-16T03:59:47.337 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.329+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1071411175}) 2024-09-16T03:59:47.338 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.329+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.338 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.329+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3547489250}) 2024-09-16T03:59:47.338 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.329+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1583222806}) 2024-09-16T03:59:47.338 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.329+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.339 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.329+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3547489250}) 2024-09-16T03:59:47.339 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.329+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=568703573}) 2024-09-16T03:59:47.341 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.333+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.341 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.333+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=364058437}) 2024-09-16T03:59:47.342 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.333+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1188559484}) 2024-09-16T03:59:47.342 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.333+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.342 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.333+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=364058437}) 2024-09-16T03:59:47.342 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.333+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2855332338}) 2024-09-16T03:59:47.342 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.333+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.342 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.333+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3277834560}) 2024-09-16T03:59:47.342 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.333+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=425417252}) 2024-09-16T03:59:47.343 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.333+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.343 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.333+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3277834560}) 2024-09-16T03:59:47.343 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.333+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=949241047}) 2024-09-16T03:59:47.345 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.337+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.345 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.337+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4113221149}) 2024-09-16T03:59:47.346 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.337+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2652040850}) 2024-09-16T03:59:47.346 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.337+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.346 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.337+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4113221149}) 2024-09-16T03:59:47.346 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.337+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=800125970}) 2024-09-16T03:59:47.346 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.337+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.346 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.337+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=579580712}) 2024-09-16T03:59:47.346 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.337+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=691621336}) 2024-09-16T03:59:47.346 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.337+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.346 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.337+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=579580712}) 2024-09-16T03:59:47.346 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.337+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=109150525}) 2024-09-16T03:59:47.347 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.341+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.347 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.341+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4010117513}) 2024-09-16T03:59:47.348 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.341+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1492418354}) 2024-09-16T03:59:47.348 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.341+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.348 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.341+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4010117513}) 2024-09-16T03:59:47.348 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.341+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2226988344}) 2024-09-16T03:59:47.349 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.341+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.349 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.341+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1295784695}) 2024-09-16T03:59:47.349 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.341+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2455351221}) 2024-09-16T03:59:47.350 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.341+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.350 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.341+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1295784695}) 2024-09-16T03:59:47.350 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.341+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3474739298}) 2024-09-16T03:59:47.351 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.345+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.351 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.345+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3799783744}) 2024-09-16T03:59:47.351 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.345+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1273560732}) 2024-09-16T03:59:47.351 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.345+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.351 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.345+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3799783744}) 2024-09-16T03:59:47.351 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.345+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2189124679}) 2024-09-16T03:59:47.353 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.345+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.353 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.345+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=356774739}) 2024-09-16T03:59:47.353 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.345+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2109441703}) 2024-09-16T03:59:47.353 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.345+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.353 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.345+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=356774739}) 2024-09-16T03:59:47.354 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.345+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4006106225}) 2024-09-16T03:59:47.355 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.349+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.355 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.349+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=75821975}) 2024-09-16T03:59:47.355 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.349+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2054940679}) 2024-09-16T03:59:47.355 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.349+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.355 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.349+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=75821975}) 2024-09-16T03:59:47.355 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.349+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2146125132}) 2024-09-16T03:59:47.358 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.349+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.358 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.349+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4059584328}) 2024-09-16T03:59:47.358 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.349+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=894479938}) 2024-09-16T03:59:47.358 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.349+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.358 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.349+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4059584328}) 2024-09-16T03:59:47.358 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.349+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1400077742}) 2024-09-16T03:59:47.359 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.353+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.359 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.353+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1634959877}) 2024-09-16T03:59:47.360 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.353+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1342522433}) 2024-09-16T03:59:47.360 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.353+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.360 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.353+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1634959877}) 2024-09-16T03:59:47.360 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.353+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4170059782}) 2024-09-16T03:59:47.361 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.353+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.361 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.353+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3440144923}) 2024-09-16T03:59:47.362 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.353+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=720688082}) 2024-09-16T03:59:47.362 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.353+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.362 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.353+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3440144923}) 2024-09-16T03:59:47.362 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.357+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1718949543}) 2024-09-16T03:59:47.364 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.357+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.364 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.357+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3047822042}) 2024-09-16T03:59:47.364 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.357+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2508244212}) 2024-09-16T03:59:47.364 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.357+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.364 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.357+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3047822042}) 2024-09-16T03:59:47.364 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.357+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1548198917}) 2024-09-16T03:59:47.365 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.361+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.366 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.361+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3455689556}) 2024-09-16T03:59:47.366 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.361+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2965254725}) 2024-09-16T03:59:47.366 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.361+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.366 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.361+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3455689556}) 2024-09-16T03:59:47.366 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.361+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2987946192}) 2024-09-16T03:59:47.368 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.361+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.368 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.361+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=841896579}) 2024-09-16T03:59:47.368 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.361+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2703677316}) 2024-09-16T03:59:47.368 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.361+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.368 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.361+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=841896579}) 2024-09-16T03:59:47.368 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.361+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4200006538}) 2024-09-16T03:59:47.370 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.361+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.370 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.361+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3912076872}) 2024-09-16T03:59:47.370 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.365+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2638392287}) 2024-09-16T03:59:47.370 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.365+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.370 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.365+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3912076872}) 2024-09-16T03:59:47.370 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.365+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=272616160}) 2024-09-16T03:59:47.371 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.365+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.371 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.365+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3746696893}) 2024-09-16T03:59:47.371 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.365+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3461270046}) 2024-09-16T03:59:47.372 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.365+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.372 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.365+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3746696893}) 2024-09-16T03:59:47.372 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.365+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=863437511}) 2024-09-16T03:59:47.373 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.365+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.373 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.365+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1085208048}) 2024-09-16T03:59:47.374 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.365+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=217677901}) 2024-09-16T03:59:47.374 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.369+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.374 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.369+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1085208048}) 2024-09-16T03:59:47.374 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.369+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2018624849}) 2024-09-16T03:59:47.376 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.369+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.376 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.369+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=373880686}) 2024-09-16T03:59:47.376 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.369+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1295741933}) 2024-09-16T03:59:47.376 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.369+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.376 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.369+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=373880686}) 2024-09-16T03:59:47.376 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.369+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3864253775}) 2024-09-16T03:59:47.378 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.373+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.378 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.373+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1101096551}) 2024-09-16T03:59:47.378 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.373+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2491440391}) 2024-09-16T03:59:47.378 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.373+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.378 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.373+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1101096551}) 2024-09-16T03:59:47.378 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.373+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=894338569}) 2024-09-16T03:59:47.380 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.373+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.380 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.373+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=56568524}) 2024-09-16T03:59:47.380 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.373+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=834304700}) 2024-09-16T03:59:47.380 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.373+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.380 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.373+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=56568524}) 2024-09-16T03:59:47.381 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.373+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1347434256}) 2024-09-16T03:59:47.382 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.377+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.383 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.377+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3265949158}) 2024-09-16T03:59:47.383 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.377+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2469370850}) 2024-09-16T03:59:47.383 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.377+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.383 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.377+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3265949158}) 2024-09-16T03:59:47.383 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.377+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2704948561}) 2024-09-16T03:59:47.384 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.377+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.384 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.377+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1273244494}) 2024-09-16T03:59:47.384 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.377+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1831084011}) 2024-09-16T03:59:47.385 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.377+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.385 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.377+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1273244494}) 2024-09-16T03:59:47.385 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.377+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1412446046}) 2024-09-16T03:59:47.386 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.381+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.386 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.381+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1729800179}) 2024-09-16T03:59:47.386 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.381+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1470644713}) 2024-09-16T03:59:47.386 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.381+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.386 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.381+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1729800179}) 2024-09-16T03:59:47.386 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.381+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=968864390}) 2024-09-16T03:59:47.388 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.381+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.388 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.381+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2215419187}) 2024-09-16T03:59:47.389 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.381+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3479700771}) 2024-09-16T03:59:47.389 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.381+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.389 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.381+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2215419187}) 2024-09-16T03:59:47.389 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.381+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3157021817}) 2024-09-16T03:59:47.390 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.385+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.390 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.385+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1685699344}) 2024-09-16T03:59:47.390 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.385+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3642000741}) 2024-09-16T03:59:47.390 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.385+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.391 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.385+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1685699344}) 2024-09-16T03:59:47.391 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.385+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=836528742}) 2024-09-16T03:59:47.392 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.385+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.392 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.385+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2894704897}) 2024-09-16T03:59:47.392 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.385+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4149948052}) 2024-09-16T03:59:47.393 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.385+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.393 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.385+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2894704897}) 2024-09-16T03:59:47.393 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.385+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=304568632}) 2024-09-16T03:59:47.394 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.389+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.394 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.389+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4160331779}) 2024-09-16T03:59:47.394 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.389+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1225521888}) 2024-09-16T03:59:47.394 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.389+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.395 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.389+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4160331779}) 2024-09-16T03:59:47.395 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.389+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1753558343}) 2024-09-16T03:59:47.396 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.389+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.396 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.389+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=942783846}) 2024-09-16T03:59:47.396 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.389+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3150700762}) 2024-09-16T03:59:47.397 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.389+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.397 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.389+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=942783846}) 2024-09-16T03:59:47.397 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.389+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2671707548}) 2024-09-16T03:59:47.399 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.393+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.399 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.393+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3618395235}) 2024-09-16T03:59:47.399 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.393+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=399023780}) 2024-09-16T03:59:47.399 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.393+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.399 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.393+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3618395235}) 2024-09-16T03:59:47.400 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.393+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2903918143}) 2024-09-16T03:59:47.400 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.393+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.400 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.393+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4275284435}) 2024-09-16T03:59:47.400 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.393+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3296614712}) 2024-09-16T03:59:47.401 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.393+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.401 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.393+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4275284435}) 2024-09-16T03:59:47.401 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.393+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1163864130}) 2024-09-16T03:59:47.402 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.397+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.402 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.397+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=871120562}) 2024-09-16T03:59:47.403 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.397+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1738625176}) 2024-09-16T03:59:47.403 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.397+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.403 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.397+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=871120562}) 2024-09-16T03:59:47.403 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.397+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1933486515}) 2024-09-16T03:59:47.404 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.397+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.405 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.397+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3725960661}) 2024-09-16T03:59:47.405 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.397+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3345579446}) 2024-09-16T03:59:47.405 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.397+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.405 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.397+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3725960661}) 2024-09-16T03:59:47.405 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.397+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1993503954}) 2024-09-16T03:59:47.407 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.401+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.407 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.401+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1967972725}) 2024-09-16T03:59:47.407 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.401+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=24539165}) 2024-09-16T03:59:47.407 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.401+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.407 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.401+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1967972725}) 2024-09-16T03:59:47.407 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.401+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3928315911}) 2024-09-16T03:59:47.409 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.401+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.409 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.401+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=95688828}) 2024-09-16T03:59:47.409 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.401+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3477946156}) 2024-09-16T03:59:47.409 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.401+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.409 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.401+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=95688828}) 2024-09-16T03:59:47.409 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.401+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2405822324}) 2024-09-16T03:59:47.411 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.405+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.411 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.405+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1486116992}) 2024-09-16T03:59:47.411 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.405+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1120136255}) 2024-09-16T03:59:47.411 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.405+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.411 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.405+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1486116992}) 2024-09-16T03:59:47.411 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.405+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2243571460}) 2024-09-16T03:59:47.412 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.405+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.413 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.405+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=23139803}) 2024-09-16T03:59:47.413 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.405+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=76347081}) 2024-09-16T03:59:47.413 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.405+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.413 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.405+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=23139803}) 2024-09-16T03:59:47.413 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.405+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=801002429}) 2024-09-16T03:59:47.414 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.409+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.415 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.409+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1497530191}) 2024-09-16T03:59:47.415 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.409+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1680552286}) 2024-09-16T03:59:47.415 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.409+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.415 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.409+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1497530191}) 2024-09-16T03:59:47.415 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.409+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4124183017}) 2024-09-16T03:59:47.416 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.409+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.417 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.409+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3444666225}) 2024-09-16T03:59:47.417 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.409+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4190959643}) 2024-09-16T03:59:47.417 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.409+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.417 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.409+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3444666225}) 2024-09-16T03:59:47.417 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.409+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1391914086}) 2024-09-16T03:59:47.419 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.413+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.419 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.413+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2584502610}) 2024-09-16T03:59:47.419 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.413+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1727853862}) 2024-09-16T03:59:47.419 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.413+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.419 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.413+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2584502610}) 2024-09-16T03:59:47.419 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.413+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3109794948}) 2024-09-16T03:59:47.421 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.413+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.421 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.413+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1358438016}) 2024-09-16T03:59:47.421 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.413+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1133480193}) 2024-09-16T03:59:47.421 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.413+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.422 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.413+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1358438016}) 2024-09-16T03:59:47.422 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.413+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4097633623}) 2024-09-16T03:59:47.424 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.417+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.424 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.417+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2379443515}) 2024-09-16T03:59:47.424 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.417+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=756727083}) 2024-09-16T03:59:47.424 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.417+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.424 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.417+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2379443515}) 2024-09-16T03:59:47.425 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.417+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1752354915}) 2024-09-16T03:59:47.426 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.421+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.426 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.421+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=745790414}) 2024-09-16T03:59:47.426 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.421+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1616021308}) 2024-09-16T03:59:47.426 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.421+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.427 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.421+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=745790414}) 2024-09-16T03:59:47.427 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.421+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3357234828}) 2024-09-16T03:59:47.428 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.421+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.428 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.421+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=683389575}) 2024-09-16T03:59:47.429 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.421+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2249125965}) 2024-09-16T03:59:47.429 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.421+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.429 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.421+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=683389575}) 2024-09-16T03:59:47.429 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.421+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2332614837}) 2024-09-16T03:59:47.430 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.425+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.430 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.425+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=775863026}) 2024-09-16T03:59:47.430 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.425+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=967926410}) 2024-09-16T03:59:47.431 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.425+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.431 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.425+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=775863026}) 2024-09-16T03:59:47.431 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.425+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1209674634}) 2024-09-16T03:59:47.432 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.425+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.432 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.425+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1639929277}) 2024-09-16T03:59:47.432 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.425+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=697154276}) 2024-09-16T03:59:47.433 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.425+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.433 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.425+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1639929277}) 2024-09-16T03:59:47.433 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.425+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2806689511}) 2024-09-16T03:59:47.434 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.429+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.435 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.429+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3914002805}) 2024-09-16T03:59:47.435 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.429+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1004335360}) 2024-09-16T03:59:47.435 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.429+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.435 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.429+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3914002805}) 2024-09-16T03:59:47.435 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.429+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1763348978}) 2024-09-16T03:59:47.437 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.429+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.437 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.429+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=687141419}) 2024-09-16T03:59:47.437 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.429+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2234975941}) 2024-09-16T03:59:47.437 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.429+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.437 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.429+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=687141419}) 2024-09-16T03:59:47.437 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.429+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1597374277}) 2024-09-16T03:59:47.439 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.433+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.439 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.433+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1189832181}) 2024-09-16T03:59:47.439 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.433+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=684022492}) 2024-09-16T03:59:47.439 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.433+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.439 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.433+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1189832181}) 2024-09-16T03:59:47.439 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.433+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=539533155}) 2024-09-16T03:59:47.441 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.433+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.441 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.433+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4115635585}) 2024-09-16T03:59:47.441 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.433+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=556155932}) 2024-09-16T03:59:47.441 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.433+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.441 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.433+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4115635585}) 2024-09-16T03:59:47.441 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.433+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2799373207}) 2024-09-16T03:59:47.442 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.437+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.442 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.437+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1279078238}) 2024-09-16T03:59:47.442 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.437+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1513498349}) 2024-09-16T03:59:47.442 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.437+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.443 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.437+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1279078238}) 2024-09-16T03:59:47.443 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.437+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1894028619}) 2024-09-16T03:59:47.445 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.437+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.445 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.437+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3705886727}) 2024-09-16T03:59:47.445 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.437+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2996200376}) 2024-09-16T03:59:47.445 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.437+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.445 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.437+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3705886727}) 2024-09-16T03:59:47.445 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.437+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1845498550}) 2024-09-16T03:59:47.447 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.441+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.447 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.441+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3891719447}) 2024-09-16T03:59:47.447 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.441+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=519583068}) 2024-09-16T03:59:47.447 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.441+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.447 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.441+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3891719447}) 2024-09-16T03:59:47.448 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.441+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2861938280}) 2024-09-16T03:59:47.449 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.441+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.449 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.441+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3660004079}) 2024-09-16T03:59:47.450 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.441+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3552238866}) 2024-09-16T03:59:47.450 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.441+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.450 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.441+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3660004079}) 2024-09-16T03:59:47.450 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.441+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2712499001}) 2024-09-16T03:59:47.451 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.445+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.451 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.445+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=625037282}) 2024-09-16T03:59:47.452 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.445+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=122909859}) 2024-09-16T03:59:47.452 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.445+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.452 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.445+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=625037282}) 2024-09-16T03:59:47.452 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.445+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=511432664}) 2024-09-16T03:59:47.453 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.445+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.454 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.445+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2826611511}) 2024-09-16T03:59:47.454 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.445+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4009474143}) 2024-09-16T03:59:47.454 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.445+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.454 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.445+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2826611511}) 2024-09-16T03:59:47.454 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.445+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2508276578}) 2024-09-16T03:59:47.456 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.449+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.456 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.449+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3016351098}) 2024-09-16T03:59:47.456 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.449+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=306176805}) 2024-09-16T03:59:47.456 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.449+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.456 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.449+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3016351098}) 2024-09-16T03:59:47.457 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.449+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3262245238}) 2024-09-16T03:59:47.458 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.449+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.458 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.449+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2093208423}) 2024-09-16T03:59:47.458 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.449+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4016720450}) 2024-09-16T03:59:47.458 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.449+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.458 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.453+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2093208423}) 2024-09-16T03:59:47.458 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.453+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=452377700}) 2024-09-16T03:59:47.459 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.453+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.459 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.453+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3185147578}) 2024-09-16T03:59:47.460 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.453+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1220684627}) 2024-09-16T03:59:47.460 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.453+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.460 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.453+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3185147578}) 2024-09-16T03:59:47.460 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.453+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3005752409}) 2024-09-16T03:59:47.462 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.457+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.462 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.457+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3203480577}) 2024-09-16T03:59:47.462 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.457+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1457873408}) 2024-09-16T03:59:47.462 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.457+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.462 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.457+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3203480577}) 2024-09-16T03:59:47.462 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.457+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=421856774}) 2024-09-16T03:59:47.464 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.457+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.464 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.457+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4178821890}) 2024-09-16T03:59:47.464 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.457+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=135395524}) 2024-09-16T03:59:47.464 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.457+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.464 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.457+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4178821890}) 2024-09-16T03:59:47.464 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.457+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2534558827}) 2024-09-16T03:59:47.466 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.461+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.466 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.461+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2707634194}) 2024-09-16T03:59:47.466 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.461+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3584759375}) 2024-09-16T03:59:47.466 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.461+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.467 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.461+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2707634194}) 2024-09-16T03:59:47.467 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.461+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=167858816}) 2024-09-16T03:59:47.468 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.461+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.468 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.461+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=550675927}) 2024-09-16T03:59:47.469 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.461+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3779569202}) 2024-09-16T03:59:47.469 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.461+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.469 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.461+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=550675927}) 2024-09-16T03:59:47.469 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.461+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=34625198}) 2024-09-16T03:59:47.476 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.469+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.477 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.469+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=791359034}) 2024-09-16T03:59:47.477 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.469+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3604994726}) 2024-09-16T03:59:47.477 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.469+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.477 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.469+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=791359034}) 2024-09-16T03:59:47.477 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.469+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=100431365}) 2024-09-16T03:59:47.481 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.473+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.481 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.473+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=118879442}) 2024-09-16T03:59:47.481 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.473+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1703366358}) 2024-09-16T03:59:47.481 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.473+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.481 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.473+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=118879442}) 2024-09-16T03:59:47.481 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.473+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=174191859}) 2024-09-16T03:59:47.513 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.505+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.513 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.505+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=670127578}) 2024-09-16T03:59:47.513 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.505+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3024316298}) 2024-09-16T03:59:47.513 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.505+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.513 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.505+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=670127578}) 2024-09-16T03:59:47.513 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.505+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3931594797}) 2024-09-16T03:59:47.514 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.509+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.515 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.509+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1200492087}) 2024-09-16T03:59:47.515 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.509+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1302959916}) 2024-09-16T03:59:47.515 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.509+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.515 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.509+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1200492087}) 2024-09-16T03:59:47.515 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.509+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=299652171}) 2024-09-16T03:59:47.516 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.509+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.517 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.509+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=429786380}) 2024-09-16T03:59:47.517 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.509+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3993764095}) 2024-09-16T03:59:47.517 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.509+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.517 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.509+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=429786380}) 2024-09-16T03:59:47.517 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.509+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2772592799}) 2024-09-16T03:59:47.518 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.513+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.519 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.513+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3157588731}) 2024-09-16T03:59:47.519 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.513+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1560106020}) 2024-09-16T03:59:47.519 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.513+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.519 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.513+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3157588731}) 2024-09-16T03:59:47.519 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.513+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3157769462}) 2024-09-16T03:59:47.520 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.513+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.520 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.513+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=287076567}) 2024-09-16T03:59:47.521 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.513+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1150443611}) 2024-09-16T03:59:47.521 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.513+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.521 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.513+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=287076567}) 2024-09-16T03:59:47.521 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.513+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=449521108}) 2024-09-16T03:59:47.524 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.517+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.524 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.517+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=611923833}) 2024-09-16T03:59:47.524 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.517+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=917102433}) 2024-09-16T03:59:47.524 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.517+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.525 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.517+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=611923833}) 2024-09-16T03:59:47.525 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.517+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3498079169}) 2024-09-16T03:59:47.525 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.517+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.525 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.517+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=904340521}) 2024-09-16T03:59:47.525 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.517+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2912753261}) 2024-09-16T03:59:47.525 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.517+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.526 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.517+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=904340521}) 2024-09-16T03:59:47.526 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.517+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3531684858}) 2024-09-16T03:59:47.527 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.521+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.527 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.521+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2828331148}) 2024-09-16T03:59:47.527 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.521+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=648856866}) 2024-09-16T03:59:47.527 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.521+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.527 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.521+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2828331148}) 2024-09-16T03:59:47.528 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.521+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2975309641}) 2024-09-16T03:59:47.529 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.521+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.529 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.521+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2494236803}) 2024-09-16T03:59:47.529 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.521+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2093574309}) 2024-09-16T03:59:47.529 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.521+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.529 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.521+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2494236803}) 2024-09-16T03:59:47.530 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.521+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3226132411}) 2024-09-16T03:59:47.530 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.525+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.531 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.525+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1173635116}) 2024-09-16T03:59:47.531 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.525+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1584096235}) 2024-09-16T03:59:47.531 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.525+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.531 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.525+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1173635116}) 2024-09-16T03:59:47.531 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.525+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=884375959}) 2024-09-16T03:59:47.532 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.525+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.533 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.525+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4177879764}) 2024-09-16T03:59:47.533 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.525+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=663259156}) 2024-09-16T03:59:47.533 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.525+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.533 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.525+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4177879764}) 2024-09-16T03:59:47.533 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.525+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1271453885}) 2024-09-16T03:59:47.534 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.525+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.534 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.525+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3476065788}) 2024-09-16T03:59:47.534 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.525+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2162803165}) 2024-09-16T03:59:47.534 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.525+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.534 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.525+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3476065788}) 2024-09-16T03:59:47.534 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.525+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3163251064}) 2024-09-16T03:59:47.535 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.529+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.535 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.529+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3401198375}) 2024-09-16T03:59:47.535 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.529+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=989674445}) 2024-09-16T03:59:47.535 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.529+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.535 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.529+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3401198375}) 2024-09-16T03:59:47.535 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.529+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1315726596}) 2024-09-16T03:59:47.537 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.529+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.537 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.529+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2687334039}) 2024-09-16T03:59:47.537 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.529+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=139728721}) 2024-09-16T03:59:47.537 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.529+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.537 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.529+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2687334039}) 2024-09-16T03:59:47.537 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.529+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4168490794}) 2024-09-16T03:59:47.538 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.533+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.538 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.533+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2637250480}) 2024-09-16T03:59:47.538 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.533+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2239453328}) 2024-09-16T03:59:47.539 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.533+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.539 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.533+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2637250480}) 2024-09-16T03:59:47.539 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.533+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2092558526}) 2024-09-16T03:59:47.540 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.533+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.540 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.533+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=120741837}) 2024-09-16T03:59:47.540 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.533+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3480041722}) 2024-09-16T03:59:47.541 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.533+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.541 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.533+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=120741837}) 2024-09-16T03:59:47.541 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.533+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1654944895}) 2024-09-16T03:59:47.542 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.537+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.542 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.537+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3102420252}) 2024-09-16T03:59:47.542 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.537+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1966969834}) 2024-09-16T03:59:47.542 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.537+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.542 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.537+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3102420252}) 2024-09-16T03:59:47.542 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.537+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3951415789}) 2024-09-16T03:59:47.543 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.537+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.543 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.537+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=654226368}) 2024-09-16T03:59:47.543 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.537+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=422877175}) 2024-09-16T03:59:47.543 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.537+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.543 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.537+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=654226368}) 2024-09-16T03:59:47.543 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.537+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2675414185}) 2024-09-16T03:59:47.544 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.537+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.544 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.537+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=332349704}) 2024-09-16T03:59:47.544 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.537+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1546684515}) 2024-09-16T03:59:47.545 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.537+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.545 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.537+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=332349704}) 2024-09-16T03:59:47.545 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.537+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3230473122}) 2024-09-16T03:59:47.547 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.541+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.547 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.541+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=84795506}) 2024-09-16T03:59:47.547 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.541+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=848016030}) 2024-09-16T03:59:47.547 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.541+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.547 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.541+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=84795506}) 2024-09-16T03:59:47.547 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.541+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3522207556}) 2024-09-16T03:59:47.547 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.541+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.547 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.541+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2082746622}) 2024-09-16T03:59:47.547 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.541+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3066910661}) 2024-09-16T03:59:47.548 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.541+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.548 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.541+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2082746622}) 2024-09-16T03:59:47.548 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.541+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3545082291}) 2024-09-16T03:59:47.549 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.541+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.549 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.541+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4127659289}) 2024-09-16T03:59:47.549 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.541+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2929012396}) 2024-09-16T03:59:47.549 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.541+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.549 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.541+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4127659289}) 2024-09-16T03:59:47.549 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.541+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2364341701}) 2024-09-16T03:59:47.551 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.545+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.551 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.545+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=10,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=10,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=34,osdmap=15} crc {logm=4011232822,mdsmap=4088524775,mgr=2815798751,mgr_command_descs=2107673600,mgr_metadata=1367272202,mgrstat=2383015758,mon_config_key=3448559827,monmap=3937513981,osd_metadata=4182959540,osd_pg_creating=2472660455,osd_snap=315335804,osdmap=3446153427}) 2024-09-16T03:59:47.551 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.545+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=10,mdsmap=3,mgr=7,mgr_command_descs=1,mgrstat=9,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=35,osdmap=14} crc {logm=4011232822,mdsmap=4088524775,mgr=1406439472,mgr_command_descs=2107673600,mgrstat=2383015758,mon_config_key=85057112,monmap=3937513981,osd_metadata=4182959540,osd_pg_creating=2472660455,osd_snap=3014786947,osdmap=3588516724}) 2024-09-16T03:59:47.551 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.545+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.551 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.545+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=10,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=10,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=34,osdmap=15} crc {logm=4011232822,mdsmap=4088524775,mgr=2815798751,mgr_command_descs=2107673600,mgr_metadata=1367272202,mgrstat=2383015758,mon_config_key=3448559827,monmap=3937513981,osd_metadata=4182959540,osd_pg_creating=2472660455,osd_snap=315335804,osdmap=3446153427}) 2024-09-16T03:59:47.551 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.545+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=10,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=10,monmap=3,osd_metadata=4,osd_pg_creating=1,osd_snap=33,osdmap=19} crc {logm=4011232822,mdsmap=210808725,mgr=460109776,mgr_command_descs=2107673600,mgr_metadata=1367272202,mgrstat=1414238737,mon_config_key=1530417952,monmap=3937513981,osd_metadata=3875794834,osd_pg_creating=2472660455,osd_snap=1149087657,osdmap=3552217664}) 2024-09-16T03:59:47.553 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.549+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.554 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.549+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=946627687}) 2024-09-16T03:59:47.554 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.549+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=275805382}) 2024-09-16T03:59:47.554 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.549+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.554 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.549+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=946627687}) 2024-09-16T03:59:47.554 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.549+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1396341038}) 2024-09-16T03:59:47.566 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.557+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.566 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.557+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=34} crc {osdmap=4212005326}) 2024-09-16T03:59:47.566 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.557+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=33} crc {osdmap=406865686}) 2024-09-16T03:59:47.566 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.557+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T03:59:47.566 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.557+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=34} crc {osdmap=4212005326}) 2024-09-16T03:59:47.566 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T03:59:47.557+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=34} crc {osdmap=4267392281}) 2024-09-16T03:59:48.292 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T03:59:48.293 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T03:59:48.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.417+0000 7f654192b640 10 monclient: get_monmap_and_config 2024-09-16T03:59:48.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.417+0000 7f654192b640 10 monclient: build_initial_monmap 2024-09-16T03:59:48.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.417+0000 7f654192b640 10 monclient: monmap: 2024-09-16T03:59:48.423 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:59:48.423 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:59:48.423 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:59:48.421929+0000 2024-09-16T03:59:48.424 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:59:48.421929+0000 2024-09-16T03:59:48.424 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:59:48.424 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:59:48.424 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:59:48.424 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:59:48.424 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:59:48.424 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:59:48.424 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:59:48.424 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:59:48.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.417+0000 7f654192b640 1 Processor -- start 2024-09-16T03:59:48.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.417+0000 7f654192b640 1 -- start start 2024-09-16T03:59:48.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.417+0000 7f654192b640 10 monclient: init 2024-09-16T03:59:48.437 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.417+0000 7f654192b640 10 monclient: _reopen_session rank -1 2024-09-16T03:59:48.437 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.417+0000 7f654192b640 10 monclient: _add_conns ranks=[0,3,1,2,4] 2024-09-16T03:59:48.437 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.417+0000 7f654192b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f653c0a1330 addr v1:172.21.15.42:6789/0 2024-09-16T03:59:48.437 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.417+0000 7f654192b640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f653c06dac0 0x7f653c13a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:59:48.437 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.417+0000 7f654192b640 10 monclient(hunting): picked mon.noname-a con 0x7f653c06dac0 addr v2:172.21.15.42:3300/0 2024-09-16T03:59:48.437 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.417+0000 7f654192b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f653c0a1e20 addr v1:172.21.15.154:6789/0 2024-09-16T03:59:48.437 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.417+0000 7f654192b640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f653c09c090 con 0x7f653c0a1330 2024-09-16T03:59:48.437 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.417+0000 7f654192b640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f653c09f610 con 0x7f653c0a1e20 2024-09-16T03:59:48.437 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.417+0000 7f654192b640 10 monclient(hunting): start opening mon connection 2024-09-16T03:59:48.437 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.417+0000 7f654192b640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f653c09f8f0 con 0x7f653c06dac0 2024-09-16T03:59:48.437 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.417+0000 7f654192b640 10 monclient(hunting): _renew_subs 2024-09-16T03:59:48.438 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.417+0000 7f654192b640 10 monclient(hunting): authenticate will time out at 2035.391316s 2024-09-16T03:59:48.438 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.417+0000 7f653affd640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f653c0a1330 0x7f653c0a1710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:34808/0 (socket says 172.21.15.42:34808) 2024-09-16T03:59:48.438 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.417+0000 7f653affd640 1 -- 172.21.15.42:0/2841203167 learned_addr learned my addr 172.21.15.42:0/2841203167 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:59:48.438 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.417+0000 7f653a7fc640 1 --2- 172.21.15.42:0/2841203167 >> v2:172.21.15.42:3300/0 conn(0x7f653c06dac0 0x7f653c13a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:59:48.438 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.417+0000 7f653a7fc640 10 monclient(hunting): get_auth_request con 0x7f653c06dac0 auth_method 0 2024-09-16T03:59:48.438 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.417+0000 7f653a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:59:48.438 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.417+0000 7f653a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:59:48.438 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.417+0000 7f653a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:59:48.439 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.417+0000 7f653a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:59:48.439 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.417+0000 7f653a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:59:48.439 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.417+0000 7f653a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:59:48.439 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.417+0000 7f6539ffb640 1 -- 172.21.15.42:0/2841203167 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2347210027 0 0) 0x7f653c09c090 con 0x7f653c0a1330 2024-09-16T03:59:48.439 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.417+0000 7f6539ffb640 10 monclient(hunting): _negotiate 2024-09-16T03:59:48.439 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.417+0000 7f6539ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:59:48.439 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.417+0000 7f6539ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:59:48.439 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.417+0000 7f6539ffb640 10 monclient(hunting): my global_id is 825189 2024-09-16T03:59:48.439 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.417+0000 7f6539ffb640 1 -- 172.21.15.42:0/2841203167 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6524003530 con 0x7f653c0a1330 2024-09-16T03:59:48.439 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.417+0000 7f653a7fc640 10 monclient(hunting): handle_auth_done global_id 825186 payload 274 2024-09-16T03:59:48.439 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.417+0000 7f653a7fc640 1 -- 172.21.15.42:0/2841203167 >> v1:172.21.15.154:6789/0 conn(0x7f653c0a1e20 legacy=0x7f653c13c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:59:48.439 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.417+0000 7f653a7fc640 1 -- 172.21.15.42:0/2841203167 >> v1:172.21.15.42:6789/0 conn(0x7f653c0a1330 legacy=0x7f653c0a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:59:48.439 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.417+0000 7f653a7fc640 10 monclient: _finish_hunting 0 2024-09-16T03:59:48.440 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.417+0000 7f653a7fc640 1 monclient: found mon.noname-a 2024-09-16T03:59:48.440 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.417+0000 7f653a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:59:48.440 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.417+0000 7f653a7fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:59:48.440 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.417+0000 7f653a7fc640 1 -- 172.21.15.42:0/2841203167 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f653c09fbd0 con 0x7f653c06dac0 2024-09-16T03:59:48.440 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.421+0000 7f653a7fc640 1 --2- 172.21.15.42:0/2841203167 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f653c06dac0 0x7f653c13a340 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f652c0097c0 tx=0x7f652c01c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27bf40b75b6705f7 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:59:48.440 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.421+0000 7f6539ffb640 1 -- 172.21.15.42:0/2841203167 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f652c0b0070 con 0x7f653c06dac0 2024-09-16T03:59:48.440 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.421+0000 7f6539ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:59:48.440 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.421+0000 7f6539ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:59:48.440 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.421+0000 7f6539ffb640 10 monclient: dump: 2024-09-16T03:59:48.440 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:59:48.440 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:59:48.440 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:48.441 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:48.441 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:59:48.441 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:59:48.441 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:59:48.441 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:59:48.441 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:59:48.441 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:59:48.441 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.421+0000 7f6539ffb640 10 monclient: _finish_auth 0 2024-09-16T03:59:48.441 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.421+0000 7f6539ffb640 10 monclient: _check_auth_tickets 2024-09-16T03:59:48.441 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.421+0000 7f6539ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:59:48.441 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.421+0000 7f6539ffb640 1 -- 172.21.15.42:0/2841203167 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f652c0ab600 con 0x7f653c06dac0 2024-09-16T03:59:48.441 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.421+0000 7f654192b640 5 monclient: authenticate success, global_id 825186 2024-09-16T03:59:48.441 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.421+0000 7f654192b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:59:48.441 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.421+0000 7f6539ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T03:59:48.442 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.421+0000 7f654192b640 10 monclient: get_monmap_and_config success 2024-09-16T03:59:48.442 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.421+0000 7f654192b640 4 set_mon_vals no callback set 2024-09-16T03:59:48.442 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.421+0000 7f6539ffb640 1 -- 172.21.15.42:0/2841203167 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f652c0ab9d0 con 0x7f653c06dac0 2024-09-16T03:59:48.442 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.421+0000 7f6539ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:59:48.442 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.421+0000 7f6539ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:59:48.442 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.421+0000 7f6539ffb640 10 monclient: dump: 2024-09-16T03:59:48.442 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:59:48.442 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:59:48.442 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:48.442 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:48.442 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:59:48.442 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:59:48.442 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:59:48.443 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:59:48.443 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:59:48.443 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:59:48.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.421+0000 7f654192b640 10 monclient: shutdown 2024-09-16T03:59:48.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.421+0000 7f654192b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:59:48.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.421+0000 7f654192b640 1 -- 172.21.15.42:0/2841203167 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f653c06dac0 msgr2=0x7f653c13a340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:59:48.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.421+0000 7f654192b640 1 --2- 172.21.15.42:0/2841203167 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f653c06dac0 0x7f653c13a340 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f652c0097c0 tx=0x7f652c01c480 comp rx=0 tx=0).stop 2024-09-16T03:59:48.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.421+0000 7f654192b640 1 -- 172.21.15.42:0/2841203167 shutdown_connections 2024-09-16T03:59:48.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.421+0000 7f654192b640 1 --2- 172.21.15.42:0/2841203167 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f653c06dac0 0x7f653c13a340 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:59:48.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.421+0000 7f654192b640 1 -- 172.21.15.42:0/2841203167 >> 172.21.15.42:0/2841203167 conn(0x7f653c0a2420 msgr2=0x7f653c0a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:59:48.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.421+0000 7f654192b640 1 -- 172.21.15.42:0/2841203167 shutdown_connections 2024-09-16T03:59:48.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.421+0000 7f654192b640 1 -- 172.21.15.42:0/2841203167 wait complete. 2024-09-16T03:59:48.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.421+0000 7f654192b640 10 monclient: build_initial_monmap 2024-09-16T03:59:48.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.421+0000 7f654192b640 10 monclient: monmap: 2024-09-16T03:59:48.444 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:59:48.444 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:59:48.444 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:59:48.445 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:59:48.445 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:59:48.445 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:59:48.445 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:59:48.445 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:59:48.445 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:59:48.445 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:59:48.445 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.421+0000 7f654192b640 1 librados: starting msgr at 2024-09-16T03:59:48.445 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.421+0000 7f654192b640 1 librados: starting objecter 2024-09-16T03:59:48.446 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.421+0000 7f654192b640 1 Processor -- start 2024-09-16T03:59:48.446 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.421+0000 7f654192b640 1 -- start start 2024-09-16T03:59:48.446 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.421+0000 7f654192b640 1 librados: setting wanted keys 2024-09-16T03:59:48.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.421+0000 7f654192b640 1 librados: calling monclient init 2024-09-16T03:59:48.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.421+0000 7f654192b640 10 monclient: init 2024-09-16T03:59:48.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.425+0000 7f654192b640 10 monclient: _reopen_session rank -1 2024-09-16T03:59:48.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.425+0000 7f654192b640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:59:48.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.425+0000 7f654192b640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f653c0a1330 0x7f653c1cedd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:59:48.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.425+0000 7f654192b640 10 monclient(hunting): picked mon.noname-a con 0x7f653c0a1330 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:59:48.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.425+0000 7f654192b640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f653c0a1e20 0x7f653c135240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:59:48.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.425+0000 7f654192b640 10 monclient(hunting): picked mon.noname-c con 0x7f653c0a1e20 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:59:48.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.425+0000 7f654192b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f653c135780 0x7f653c11cca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:59:48.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.425+0000 7f654192b640 10 monclient(hunting): picked mon.noname-b con 0x7f653c135780 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:59:48.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.425+0000 7f654192b640 10 monclient(hunting): start opening mon connection 2024-09-16T03:59:48.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.425+0000 7f654192b640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f653c1409e0 con 0x7f653c0a1330 2024-09-16T03:59:48.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.425+0000 7f654192b640 10 monclient(hunting): start opening mon connection 2024-09-16T03:59:48.448 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.425+0000 7f654192b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f653c09efb0 con 0x7f653c135780 2024-09-16T03:59:48.448 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.425+0000 7f654192b640 10 monclient(hunting): start opening mon connection 2024-09-16T03:59:48.448 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.425+0000 7f654192b640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f653c140b60 con 0x7f653c0a1e20 2024-09-16T03:59:48.448 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.425+0000 7f654192b640 10 monclient(hunting): _renew_subs 2024-09-16T03:59:48.448 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.425+0000 7f654192b640 10 monclient(hunting): authenticate will time out at 2035.396370s 2024-09-16T03:59:48.448 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.425+0000 7f653affd640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f653c0a1330 0x7f653c1cedd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:59:48.448 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.425+0000 7f653affd640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f653c0a1330 0x7f653c1cedd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:36814/0 (socket says 172.21.15.42:36814) 2024-09-16T03:59:48.448 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.425+0000 7f653affd640 1 -- 172.21.15.42:0/190060821 learned_addr learned my addr 172.21.15.42:0/190060821 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:59:48.448 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.425+0000 7f653affd640 10 monclient(hunting): get_auth_request con 0x7f653c0a1330 auth_method 0 2024-09-16T03:59:48.449 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.425+0000 7f653affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:59:48.449 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.425+0000 7f653affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:59:48.449 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.425+0000 7f653affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:59:48.449 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.425+0000 7f653a7fc640 1 --2- 172.21.15.42:0/190060821 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f653c0a1e20 0x7f653c135240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:59:48.449 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.425+0000 7f653a7fc640 10 monclient(hunting): get_auth_request con 0x7f653c0a1e20 auth_method 0 2024-09-16T03:59:48.449 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.425+0000 7f653a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:59:48.449 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.425+0000 7f653a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:59:48.449 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.425+0000 7f653a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:59:48.449 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.425+0000 7f653a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:59:48.449 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.425+0000 7f653a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:59:48.450 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.425+0000 7f653a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:59:48.450 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.425+0000 7f653b7fe640 1 --2- 172.21.15.42:0/190060821 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f653c135780 0x7f653c11cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:59:48.450 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.425+0000 7f653affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:59:48.450 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.425+0000 7f653affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:59:48.450 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.425+0000 7f653affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:59:48.450 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.425+0000 7f653b7fe640 10 monclient(hunting): get_auth_request con 0x7f653c135780 auth_method 0 2024-09-16T03:59:48.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.425+0000 7f653b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:59:48.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.425+0000 7f653b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:59:48.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.425+0000 7f653b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:59:48.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.425+0000 7f653b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:59:48.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.425+0000 7f653b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:59:48.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.425+0000 7f653b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:59:48.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.425+0000 7f653a7fc640 10 monclient(hunting): handle_auth_done global_id 934121 payload 824 2024-09-16T03:59:48.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.425+0000 7f653a7fc640 1 -- 172.21.15.42:0/190060821 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f653c135780 msgr2=0x7f653c11cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:59:48.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.425+0000 7f653a7fc640 1 --2- 172.21.15.42:0/190060821 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f653c135780 0x7f653c11cca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:59:48.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.425+0000 7f653a7fc640 1 -- 172.21.15.42:0/190060821 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f653c0a1330 msgr2=0x7f653c1cedd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:59:48.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.425+0000 7f653a7fc640 1 --2- 172.21.15.42:0/190060821 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f653c0a1330 0x7f653c1cedd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:59:48.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.425+0000 7f653a7fc640 10 monclient: _finish_hunting 0 2024-09-16T03:59:48.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.425+0000 7f653a7fc640 1 monclient: found mon.noname-c 2024-09-16T03:59:48.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.425+0000 7f653a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:59:48.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.425+0000 7f653a7fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:59:48.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.425+0000 7f653a7fc640 1 -- 172.21.15.42:0/190060821 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f653c120900 con 0x7f653c0a1e20 2024-09-16T03:59:48.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.425+0000 7f653affd640 1 --2- 172.21.15.42:0/190060821 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f653c0a1330 0x7f653c1cedd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:59:48.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.425+0000 7f653a7fc640 1 --2- 172.21.15.42:0/190060821 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f653c0a1e20 0x7f653c135240 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f653c0a6900 tx=0x7f652c019220 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:59:48.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.425+0000 7f6523fff640 10 client.?.objecter ms_handle_connect 0x7f653c0a1e20 2024-09-16T03:59:48.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.425+0000 7f6523fff640 10 client.?.objecter resend_mon_ops 2024-09-16T03:59:48.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.425+0000 7f6523fff640 1 -- 172.21.15.42:0/190060821 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f652c0b8020 con 0x7f653c0a1e20 2024-09-16T03:59:48.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.425+0000 7f6523fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:59:48.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.425+0000 7f6523fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:59:48.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.425+0000 7f6523fff640 10 monclient: dump: 2024-09-16T03:59:48.453 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:59:48.453 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:59:48.453 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:48.454 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:48.454 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:59:48.454 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:59:48.454 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:59:48.454 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:59:48.454 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:59:48.454 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:59:48.455 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.425+0000 7f6523fff640 10 monclient: _finish_auth 0 2024-09-16T03:59:48.455 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.425+0000 7f6523fff640 10 monclient: _check_auth_tickets 2024-09-16T03:59:48.455 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.425+0000 7f6523fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:59:48.455 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.425+0000 7f654192b640 5 monclient: authenticate success, global_id 934121 2024-09-16T03:59:48.455 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.425+0000 7f654192b640 10 monclient: _renew_subs 2024-09-16T03:59:48.455 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.425+0000 7f6523fff640 1 -- 172.21.15.42:0/190060821 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f652c0b4410 con 0x7f653c0a1e20 2024-09-16T03:59:48.455 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.425+0000 7f654192b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:59:48.456 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.425+0000 7f654192b640 1 -- 172.21.15.42:0/190060821 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f653c06e240 con 0x7f653c0a1e20 2024-09-16T03:59:48.456 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.425+0000 7f6523fff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:59:48.456 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.425+0000 7f6523fff640 1 -- 172.21.15.42:0/190060821 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f652c0b0040 con 0x7f653c0a1e20 2024-09-16T03:59:48.456 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.425+0000 7f6523fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:59:48.456 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.425+0000 7f6523fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:59:48.456 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.425+0000 7f6523fff640 10 monclient: dump: 2024-09-16T03:59:48.456 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:59:48.456 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:59:48.456 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:48.457 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:48.457 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:59:48.457 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:59:48.457 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:59:48.457 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:59:48.457 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:59:48.457 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:59:48.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.425+0000 7f654192b640 10 client.934121.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:59:48.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.425+0000 7f654192b640 10 monclient: _renew_subs 2024-09-16T03:59:48.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.425+0000 7f654192b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:59:48.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.425+0000 7f654192b640 1 -- 172.21.15.42:0/190060821 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f653c120620 con 0x7f653c0a1e20 2024-09-16T03:59:48.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.425+0000 7f65397fa640 4 set_mon_vals no callback set 2024-09-16T03:59:48.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.425+0000 7f654192b640 1 librados: init done 2024-09-16T03:59:48.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.425+0000 7f6523fff640 1 -- 172.21.15.42:0/190060821 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f652c0bb020 con 0x7f653c0a1e20 2024-09-16T03:59:48.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.425+0000 7f654192b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:59:48.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.425+0000 7f654192b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:59:48.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.425+0000 7f654192b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:59:48.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.425+0000 7f6523fff640 1 --2- 172.21.15.42:0/190060821 >> v2:172.21.15.154:6812/254845342 conn(0x7f6528037810 0x7f6528039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:59:48.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.425+0000 7f654192b640 1 -- 172.21.15.42:0/190060821 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f653c0a1710 con 0x7f653c0a1e20 2024-09-16T03:59:48.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.425+0000 7f6523fff640 1 -- 172.21.15.42:0/190060821 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(540..540 src has 451..540) ==== 6168+0+0 (secure 0 0 0) 0x7f652c0e2be0 con 0x7f653c0a1e20 2024-09-16T03:59:48.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.425+0000 7f6523fff640 10 client.934121.objecter ms_dispatch 0x7f653c000ba0 osd_map(540..540 src has 451..540) 2024-09-16T03:59:48.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.425+0000 7f6523fff640 3 client.934121.objecter handle_osd_map got epochs [540,540] > 0 2024-09-16T03:59:48.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.425+0000 7f6523fff640 3 client.934121.objecter handle_osd_map decoding full epoch 540 2024-09-16T03:59:48.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.425+0000 7f6523fff640 20 client.934121.objecter dump_active .. 0 homeless 2024-09-16T03:59:48.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.425+0000 7f653affd640 1 --2- 172.21.15.42:0/190060821 >> v2:172.21.15.154:6812/254845342 conn(0x7f6528037810 0x7f6528039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:59:48.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.425+0000 7f653affd640 10 monclient: get_auth_request con 0x7f6528037810 auth_method 0 2024-09-16T03:59:48.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.425+0000 7f653affd640 1 --2- 172.21.15.42:0/190060821 >> v2:172.21.15.154:6812/254845342 conn(0x7f6528037810 0x7f6528039cd0 secure :-1 s=READY pgs=3736 cs=0 l=1 rev1=1 crypto rx=0x7f65300011f0 tx=0x7f6530009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:59:48.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.425+0000 7f6523fff640 10 client.934121.objecter ms_handle_connect 0x7f6528037810 2024-09-16T03:59:48.460 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.429+0000 7f6523fff640 1 -- 172.21.15.42:0/190060821 <== mon.2 v2:172.21.15.42:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f652c0c2030 con 0x7f653c0a1e20 2024-09-16T03:59:48.460 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.429+0000 7f6523fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:59:48.460 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.429+0000 7f6523fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:59:48.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.657+0000 7f654192b640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T03:59:48.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.657+0000 7f654192b640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T03:59:48.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.657+0000 7f654192b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:59:48.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.657+0000 7f654192b640 1 -- 172.21.15.42:0/190060821 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f653c11f1d0 con 0x7f653c0a1e20 2024-09-16T03:59:48.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.657+0000 7f6523fff640 1 -- 172.21.15.42:0/190060821 <== mon.2 v2:172.21.15.42:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7f652c016070 con 0x7f653c0a1e20 2024-09-16T03:59:48.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.657+0000 7f6523fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T03:59:48.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.657+0000 7f6523fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:59:48.665 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":160,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T03:59:48.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.661+0000 7f654192b640 10 librados: watch_flush enter 2024-09-16T03:59:48.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.661+0000 7f654192b640 10 librados: watch_flush exit 2024-09-16T03:59:48.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.661+0000 7f654192b640 20 client.934121.objecter shutdown clearing up homeless session... 2024-09-16T03:59:48.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.661+0000 7f654192b640 10 client.934121.objecter successfully canceled tick 2024-09-16T03:59:48.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.661+0000 7f654192b640 1 -- 172.21.15.42:0/190060821 >> v2:172.21.15.154:6812/254845342 conn(0x7f6528037810 msgr2=0x7f6528039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:59:48.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.661+0000 7f654192b640 1 --2- 172.21.15.42:0/190060821 >> v2:172.21.15.154:6812/254845342 conn(0x7f6528037810 0x7f6528039cd0 secure :-1 s=READY pgs=3736 cs=0 l=1 rev1=1 crypto rx=0x7f65300011f0 tx=0x7f6530009040 comp rx=0 tx=0).stop 2024-09-16T03:59:48.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.661+0000 7f654192b640 10 monclient: shutdown 2024-09-16T03:59:48.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.661+0000 7f654192b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:59:48.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.661+0000 7f654192b640 1 -- 172.21.15.42:0/190060821 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f653c0a1e20 msgr2=0x7f653c135240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:59:48.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.661+0000 7f654192b640 1 --2- 172.21.15.42:0/190060821 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f653c0a1e20 0x7f653c135240 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f653c0a6900 tx=0x7f652c019220 comp rx=0 tx=0).stop 2024-09-16T03:59:48.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.661+0000 7f654192b640 1 -- 172.21.15.42:0/190060821 shutdown_connections 2024-09-16T03:59:48.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.661+0000 7f654192b640 1 --2- 172.21.15.42:0/190060821 >> v2:172.21.15.154:6812/254845342 conn(0x7f6528037810 0x7f6528039cd0 unknown :-1 s=CLOSED pgs=3736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:59:48.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.661+0000 7f654192b640 1 --2- 172.21.15.42:0/190060821 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f653c135780 0x7f653c11cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:59:48.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.661+0000 7f654192b640 1 --2- 172.21.15.42:0/190060821 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f653c0a1e20 0x7f653c135240 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:59:48.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.661+0000 7f654192b640 1 --2- 172.21.15.42:0/190060821 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f653c0a1330 0x7f653c1cedd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:59:48.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.661+0000 7f654192b640 1 -- 172.21.15.42:0/190060821 >> 172.21.15.42:0/190060821 conn(0x7f653c0a2420 msgr2=0x7f653c0a45b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:59:48.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.661+0000 7f654192b640 1 -- 172.21.15.42:0/190060821 shutdown_connections 2024-09-16T03:59:48.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.661+0000 7f654192b640 1 -- 172.21.15.42:0/190060821 wait complete. 2024-09-16T03:59:48.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.661+0000 7f654192b640 1 librados: shutdown 2024-09-16T03:59:48.685 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T03:59:48.685 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T03:59:48.685 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T03:59:48.686 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T03:59:48.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.809+0000 7fe58b387640 10 monclient: get_monmap_and_config 2024-09-16T03:59:48.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.809+0000 7fe58b387640 10 monclient: build_initial_monmap 2024-09-16T03:59:48.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.809+0000 7fe58b387640 10 monclient: monmap: 2024-09-16T03:59:48.814 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:59:48.814 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:59:48.814 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:59:48.812393+0000 2024-09-16T03:59:48.814 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:59:48.812393+0000 2024-09-16T03:59:48.815 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:59:48.815 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:59:48.815 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:59:48.815 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:59:48.815 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:59:48.815 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:59:48.815 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:59:48.815 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:59:48.826 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.809+0000 7fe58b387640 1 Processor -- start 2024-09-16T03:59:48.826 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.809+0000 7fe58b387640 1 -- start start 2024-09-16T03:59:48.826 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.809+0000 7fe58b387640 10 monclient: init 2024-09-16T03:59:48.826 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.809+0000 7fe58b387640 10 monclient: _reopen_session rank -1 2024-09-16T03:59:48.826 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.809+0000 7fe58b387640 10 monclient: _add_conns ranks=[0,3,1,4,2] 2024-09-16T03:59:48.826 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.809+0000 7fe58b387640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe5840a1330 addr v1:172.21.15.42:6789/0 2024-09-16T03:59:48.826 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.809+0000 7fe58b387640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe58406dac0 0x7fe58413a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:59:48.826 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.809+0000 7fe58b387640 10 monclient(hunting): picked mon.noname-a con 0x7fe58406dac0 addr v2:172.21.15.42:3300/0 2024-09-16T03:59:48.826 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.809+0000 7fe58b387640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe5840a1e20 addr v1:172.21.15.154:6789/0 2024-09-16T03:59:48.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.809+0000 7fe58b387640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe58409c090 con 0x7fe5840a1330 2024-09-16T03:59:48.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.809+0000 7fe58b387640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe58409f610 con 0x7fe5840a1e20 2024-09-16T03:59:48.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.809+0000 7fe58b387640 10 monclient(hunting): start opening mon connection 2024-09-16T03:59:48.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.809+0000 7fe58b387640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7fe58409f8f0 con 0x7fe58406dac0 2024-09-16T03:59:48.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.809+0000 7fe58b387640 10 monclient(hunting): _renew_subs 2024-09-16T03:59:48.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.809+0000 7fe58b387640 10 monclient(hunting): authenticate will time out at 2035.781634s 2024-09-16T03:59:48.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.809+0000 7fe588922640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe58406dac0 0x7fe58413a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:59:48.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.809+0000 7fe588922640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe58406dac0 0x7fe58413a340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:36826/0 (socket says 172.21.15.42:36826) 2024-09-16T03:59:48.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.809+0000 7fe589123640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fe5840a1330 0x7fe5840a1710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:34824/0 (socket says 172.21.15.42:34824) 2024-09-16T03:59:48.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.809+0000 7fe588922640 1 -- 172.21.15.42:0/1689926545 learned_addr learned my addr 172.21.15.42:0/1689926545 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:59:48.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.809+0000 7fe588922640 10 monclient(hunting): get_auth_request con 0x7fe58406dac0 auth_method 0 2024-09-16T03:59:48.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.809+0000 7fe588922640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:59:48.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.809+0000 7fe588922640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:59:48.828 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.809+0000 7fe588922640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:59:48.828 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.809+0000 7fe588922640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:59:48.828 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.809+0000 7fe588922640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:59:48.828 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.809+0000 7fe588922640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:59:48.828 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.809+0000 7fe588922640 10 monclient(hunting): handle_auth_done global_id 825195 payload 274 2024-09-16T03:59:48.828 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.809+0000 7fe588922640 1 -- 172.21.15.42:0/1689926545 >> v1:172.21.15.154:6789/0 conn(0x7fe5840a1e20 legacy=0x7fe58413c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T03:59:48.828 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.809+0000 7fe588922640 1 -- 172.21.15.42:0/1689926545 >> v1:172.21.15.42:6789/0 conn(0x7fe5840a1330 legacy=0x7fe5840a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:59:48.828 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.809+0000 7fe588922640 10 monclient: _finish_hunting 0 2024-09-16T03:59:48.828 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.809+0000 7fe588922640 1 monclient: found mon.noname-a 2024-09-16T03:59:48.828 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.809+0000 7fe588922640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:59:48.828 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.809+0000 7fe588922640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:59:48.828 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.809+0000 7fe588922640 1 -- 172.21.15.42:0/1689926545 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe58409fbd0 con 0x7fe58406dac0 2024-09-16T03:59:48.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.809+0000 7fe57bfff640 1 -- 172.21.15.42:0/1689926545 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1292807822 0 0) 0x7fe58409c090 con 0x7fe5840a1330 2024-09-16T03:59:48.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.809+0000 7fe57bfff640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T03:59:48.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.813+0000 7fe588922640 1 --2- 172.21.15.42:0/1689926545 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe58406dac0 0x7fe58413a340 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fe574009870 tx=0x7fe57401c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe903831e275abb5 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:59:48.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.813+0000 7fe57bfff640 1 -- 172.21.15.42:0/1689926545 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5740b0070 con 0x7fe58406dac0 2024-09-16T03:59:48.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.813+0000 7fe57bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:59:48.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.813+0000 7fe57bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:59:48.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.813+0000 7fe57bfff640 10 monclient: dump: 2024-09-16T03:59:48.830 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:59:48.830 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:59:48.830 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:48.830 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:48.831 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:59:48.831 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:59:48.831 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:59:48.831 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:59:48.831 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:59:48.831 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:59:48.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.813+0000 7fe57bfff640 10 monclient: _finish_auth 0 2024-09-16T03:59:48.832 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.813+0000 7fe57bfff640 10 monclient: _check_auth_tickets 2024-09-16T03:59:48.832 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.813+0000 7fe57bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:59:48.832 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.813+0000 7fe58b387640 5 monclient: authenticate success, global_id 825195 2024-09-16T03:59:48.832 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.813+0000 7fe58b387640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:59:48.832 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.813+0000 7fe57bfff640 1 -- 172.21.15.42:0/1689926545 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe5740ab600 con 0x7fe58406dac0 2024-09-16T03:59:48.832 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.813+0000 7fe57bfff640 10 monclient: handle_config config(0 keys) 2024-09-16T03:59:48.832 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.813+0000 7fe58b387640 10 monclient: get_monmap_and_config success 2024-09-16T03:59:48.832 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.813+0000 7fe58b387640 4 set_mon_vals no callback set 2024-09-16T03:59:48.832 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.813+0000 7fe57bfff640 1 -- 172.21.15.42:0/1689926545 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5740ab9d0 con 0x7fe58406dac0 2024-09-16T03:59:48.832 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.813+0000 7fe57bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:59:48.832 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.813+0000 7fe57bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:59:48.832 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.813+0000 7fe57bfff640 10 monclient: dump: 2024-09-16T03:59:48.833 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:59:48.833 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:59:48.833 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:48.833 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:48.833 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:59:48.833 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:59:48.833 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:59:48.833 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:59:48.833 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:59:48.833 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:59:48.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.813+0000 7fe58b387640 10 monclient: shutdown 2024-09-16T03:59:48.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.813+0000 7fe58b387640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:59:48.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.813+0000 7fe58b387640 1 -- 172.21.15.42:0/1689926545 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe58406dac0 msgr2=0x7fe58413a340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:59:48.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.813+0000 7fe58b387640 1 --2- 172.21.15.42:0/1689926545 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe58406dac0 0x7fe58413a340 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fe574009870 tx=0x7fe57401c500 comp rx=0 tx=0).stop 2024-09-16T03:59:48.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.813+0000 7fe58b387640 1 -- 172.21.15.42:0/1689926545 shutdown_connections 2024-09-16T03:59:48.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.813+0000 7fe58b387640 1 --2- 172.21.15.42:0/1689926545 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe58406dac0 0x7fe58413a340 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:59:48.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.813+0000 7fe58b387640 1 -- 172.21.15.42:0/1689926545 >> 172.21.15.42:0/1689926545 conn(0x7fe5840a2420 msgr2=0x7fe5840a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:59:48.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.813+0000 7fe58b387640 1 -- 172.21.15.42:0/1689926545 shutdown_connections 2024-09-16T03:59:48.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.813+0000 7fe58b387640 1 -- 172.21.15.42:0/1689926545 wait complete. 2024-09-16T03:59:48.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.813+0000 7fe58b387640 10 monclient: build_initial_monmap 2024-09-16T03:59:48.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.813+0000 7fe58b387640 10 monclient: monmap: 2024-09-16T03:59:48.834 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:59:48.834 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:59:48.834 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:59:48.834 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:59:48.834 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:59:48.835 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:59:48.835 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:59:48.835 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:59:48.835 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:59:48.835 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:59:48.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.813+0000 7fe58b387640 1 librados: starting msgr at 2024-09-16T03:59:48.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.813+0000 7fe58b387640 1 librados: starting objecter 2024-09-16T03:59:48.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.813+0000 7fe58b387640 1 Processor -- start 2024-09-16T03:59:48.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.813+0000 7fe58b387640 1 -- start start 2024-09-16T03:59:48.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.813+0000 7fe58b387640 1 librados: setting wanted keys 2024-09-16T03:59:48.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.813+0000 7fe58b387640 1 librados: calling monclient init 2024-09-16T03:59:48.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.813+0000 7fe58b387640 10 monclient: init 2024-09-16T03:59:48.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.813+0000 7fe58b387640 10 monclient: _reopen_session rank -1 2024-09-16T03:59:48.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.813+0000 7fe58b387640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T03:59:48.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.813+0000 7fe58b387640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe5840a1330 0x7fe58411bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:59:48.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.813+0000 7fe58b387640 10 monclient(hunting): picked mon.noname-b con 0x7fe5840a1330 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:59:48.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.813+0000 7fe58b387640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe5840a1e20 0x7fe58411c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:59:48.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.813+0000 7fe58b387640 10 monclient(hunting): picked mon.noname-c con 0x7fe5840a1e20 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:59:48.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.813+0000 7fe58b387640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe58411c9f0 0x7fe5841348d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:59:48.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.813+0000 7fe58b387640 10 monclient(hunting): picked mon.noname-a con 0x7fe58411c9f0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:59:48.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.813+0000 7fe58b387640 10 monclient(hunting): start opening mon connection 2024-09-16T03:59:48.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.813+0000 7fe58b387640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fe584140b60 con 0x7fe58411c9f0 2024-09-16T03:59:48.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.813+0000 7fe58b387640 10 monclient(hunting): start opening mon connection 2024-09-16T03:59:48.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.813+0000 7fe58b387640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe5841409e0 con 0x7fe5840a1330 2024-09-16T03:59:48.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.813+0000 7fe58b387640 10 monclient(hunting): start opening mon connection 2024-09-16T03:59:48.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.813+0000 7fe58b387640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fe584140ce0 con 0x7fe5840a1e20 2024-09-16T03:59:48.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.813+0000 7fe58b387640 10 monclient(hunting): _renew_subs 2024-09-16T03:59:48.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.813+0000 7fe58b387640 10 monclient(hunting): authenticate will time out at 2035.786687s 2024-09-16T03:59:48.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.813+0000 7fe589924640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe58411c9f0 0x7fe5841348d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:59:48.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.813+0000 7fe588922640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe5840a1e20 0x7fe58411c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:59:48.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.813+0000 7fe589924640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe58411c9f0 0x7fe5841348d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:36842/0 (socket says 172.21.15.42:36842) 2024-09-16T03:59:48.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.813+0000 7fe589924640 1 -- 172.21.15.42:0/400687140 learned_addr learned my addr 172.21.15.42:0/400687140 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:59:48.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.813+0000 7fe589924640 10 monclient(hunting): get_auth_request con 0x7fe58411c9f0 auth_method 0 2024-09-16T03:59:48.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.813+0000 7fe588922640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe5840a1e20 0x7fe58411c4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:44230/0 (socket says 172.21.15.42:44230) 2024-09-16T03:59:48.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.813+0000 7fe589924640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:59:48.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.813+0000 7fe589924640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:59:48.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.813+0000 7fe589924640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:59:48.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.813+0000 7fe588922640 10 monclient(hunting): get_auth_request con 0x7fe5840a1e20 auth_method 0 2024-09-16T03:59:48.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.813+0000 7fe588922640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:59:48.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.813+0000 7fe588922640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:59:48.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.813+0000 7fe588922640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:59:48.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.813+0000 7fe589924640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:59:48.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.813+0000 7fe589924640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:59:48.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.813+0000 7fe589924640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:59:48.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.813+0000 7fe588922640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:59:48.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.813+0000 7fe588922640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:59:48.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.813+0000 7fe588922640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:59:48.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.813+0000 7fe589924640 10 monclient(hunting): handle_auth_done global_id 825201 payload 824 2024-09-16T03:59:48.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.813+0000 7fe589123640 1 --2- 172.21.15.42:0/400687140 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe5840a1330 0x7fe58411bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:59:48.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.813+0000 7fe589924640 1 -- 172.21.15.42:0/400687140 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe5840a1e20 msgr2=0x7fe58411c4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:59:48.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.813+0000 7fe589924640 1 --2- 172.21.15.42:0/400687140 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe5840a1e20 0x7fe58411c4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:59:48.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.813+0000 7fe589924640 1 -- 172.21.15.42:0/400687140 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe5840a1330 msgr2=0x7fe58411bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:59:48.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.813+0000 7fe589924640 1 --2- 172.21.15.42:0/400687140 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe5840a1330 0x7fe58411bf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:59:48.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.813+0000 7fe589924640 10 monclient: _finish_hunting 0 2024-09-16T03:59:48.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.813+0000 7fe589924640 1 monclient: found mon.noname-a 2024-09-16T03:59:48.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.813+0000 7fe589924640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:59:48.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.813+0000 7fe589924640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:59:48.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.813+0000 7fe589924640 1 -- 172.21.15.42:0/400687140 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe58411e800 con 0x7fe58411c9f0 2024-09-16T03:59:48.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.813+0000 7fe589123640 10 monclient: get_auth_request con 0x7fe5840a1330 auth_method 0 2024-09-16T03:59:48.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.813+0000 7fe589123640 1 --2- 172.21.15.42:0/400687140 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe5840a1330 0x7fe58411bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T03:59:48.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.813+0000 7fe589924640 1 --2- 172.21.15.42:0/400687140 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe58411c9f0 0x7fe5841348d0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fe58000c3c0 tx=0x7fe58000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:59:48.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.813+0000 7fe579ffb640 10 client.?.objecter ms_handle_connect 0x7fe58411c9f0 2024-09-16T03:59:48.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.813+0000 7fe579ffb640 10 client.?.objecter resend_mon_ops 2024-09-16T03:59:48.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.813+0000 7fe579ffb640 1 -- 172.21.15.42:0/400687140 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe580011790 con 0x7fe58411c9f0 2024-09-16T03:59:48.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.813+0000 7fe579ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:59:48.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.813+0000 7fe579ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:59:48.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.813+0000 7fe579ffb640 10 monclient: dump: 2024-09-16T03:59:48.840 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:59:48.841 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:59:48.841 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:48.841 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:48.841 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:59:48.841 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:59:48.841 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:59:48.841 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:59:48.841 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:59:48.841 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:59:48.841 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.813+0000 7fe579ffb640 10 monclient: _finish_auth 0 2024-09-16T03:59:48.841 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.813+0000 7fe579ffb640 10 monclient: _check_auth_tickets 2024-09-16T03:59:48.841 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.813+0000 7fe579ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:59:48.841 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.813+0000 7fe579ffb640 1 -- 172.21.15.42:0/400687140 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe580017070 con 0x7fe58411c9f0 2024-09-16T03:59:48.842 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.813+0000 7fe579ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T03:59:48.842 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.813+0000 7fe579ffb640 1 -- 172.21.15.42:0/400687140 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe580011b70 con 0x7fe58411c9f0 2024-09-16T03:59:48.842 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.813+0000 7fe579ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:59:48.842 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.813+0000 7fe579ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:59:48.842 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.813+0000 7fe579ffb640 10 monclient: dump: 2024-09-16T03:59:48.842 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:59:48.842 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:59:48.842 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:48.842 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:48.842 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:59:48.842 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:59:48.842 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:59:48.842 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:59:48.842 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:59:48.843 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:59:48.843 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.813+0000 7fe58b387640 5 monclient: authenticate success, global_id 825201 2024-09-16T03:59:48.843 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.813+0000 7fe57b7fe640 4 set_mon_vals no callback set 2024-09-16T03:59:48.843 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.813+0000 7fe58b387640 10 monclient: _renew_subs 2024-09-16T03:59:48.843 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.813+0000 7fe58b387640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:59:48.843 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.813+0000 7fe58b387640 1 -- 172.21.15.42:0/400687140 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe58413d540 con 0x7fe58411c9f0 2024-09-16T03:59:48.843 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.813+0000 7fe58b387640 10 client.825201.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:59:48.843 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.813+0000 7fe58b387640 10 monclient: _renew_subs 2024-09-16T03:59:48.843 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.813+0000 7fe58b387640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T03:59:48.843 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.813+0000 7fe58b387640 1 -- 172.21.15.42:0/400687140 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe58411e520 con 0x7fe58411c9f0 2024-09-16T03:59:48.843 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.813+0000 7fe58b387640 1 librados: init done 2024-09-16T03:59:48.843 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.817+0000 7fe58b387640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:59:48.844 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.817+0000 7fe58b387640 1 --2- 172.21.15.42:0/400687140 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe58406dac0 0x7fe584134e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:59:48.844 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.817+0000 7fe58b387640 10 monclient(hunting): start opening mon connection 2024-09-16T03:59:48.844 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.817+0000 7fe58b387640 1 -- 172.21.15.42:0/400687140 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fe58411d3c0 con 0x7fe58406dac0 2024-09-16T03:59:48.844 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.817+0000 7fe589123640 1 --2- 172.21.15.42:0/400687140 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe58406dac0 0x7fe584134e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:59:48.844 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.817+0000 7fe589123640 10 monclient: get_auth_request con 0x7fe58406dac0 auth_method 0 2024-09-16T03:59:48.844 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.817+0000 7fe589123640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:59:48.844 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.817+0000 7fe589123640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:59:48.844 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.817+0000 7fe589123640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:59:48.844 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.817+0000 7fe589123640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:59:48.844 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.817+0000 7fe589123640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:59:48.844 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.817+0000 7fe589123640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:59:48.845 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.817+0000 7fe579ffb640 1 -- 172.21.15.42:0/400687140 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fe580011d10 con 0x7fe58411c9f0 2024-09-16T03:59:48.845 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.817+0000 7fe589123640 10 monclient(hunting): handle_auth_done global_id 825204 payload 824 2024-09-16T03:59:48.845 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.817+0000 7fe589123640 1 -- 172.21.15.42:0/400687140 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe5840a1710 con 0x7fe58406dac0 2024-09-16T03:59:48.845 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.817+0000 7fe579ffb640 1 --2- 172.21.15.42:0/400687140 >> v2:172.21.15.154:6812/254845342 conn(0x7fe568037840 0x7fe568039d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:59:48.845 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.817+0000 7fe579ffb640 1 -- 172.21.15.42:0/400687140 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(540..540 src has 451..540) ==== 6168+0+0 (secure 0 0 0) 0x7fe580048db0 con 0x7fe58411c9f0 2024-09-16T03:59:48.845 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.817+0000 7fe579ffb640 10 client.825201.objecter ms_dispatch 0x7fe584000ba0 osd_map(540..540 src has 451..540) 2024-09-16T03:59:48.845 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.817+0000 7fe579ffb640 3 client.825201.objecter handle_osd_map got epochs [540,540] > 0 2024-09-16T03:59:48.845 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.817+0000 7fe579ffb640 3 client.825201.objecter handle_osd_map decoding full epoch 540 2024-09-16T03:59:48.845 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.817+0000 7fe589123640 1 --2- 172.21.15.42:0/400687140 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe58406dac0 0x7fe584134e10 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fe57c00da60 tx=0x7fe57c00df20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:59:48.846 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.817+0000 7fe579ffb640 20 client.825201.objecter dump_active .. 0 homeless 2024-09-16T03:59:48.846 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.817+0000 7fe579ffb640 10 client.825201.objecter ms_handle_connect 0x7fe58406dac0 2024-09-16T03:59:48.846 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.817+0000 7fe579ffb640 10 client.825201.objecter resend_mon_ops 2024-09-16T03:59:48.846 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.817+0000 7fe579ffb640 1 -- 172.21.15.42:0/400687140 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe57c018070 con 0x7fe58406dac0 2024-09-16T03:59:48.846 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.817+0000 7fe579ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:59:48.846 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.817+0000 7fe579ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:59:48.846 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.817+0000 7fe579ffb640 10 monclient: dump: 2024-09-16T03:59:48.846 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:59:48.846 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:59:48.846 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:48.846 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:48.846 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:59:48.847 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:59:48.847 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:59:48.847 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:59:48.847 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:59:48.847 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:59:48.847 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.817+0000 7fe588922640 1 --2- 172.21.15.42:0/400687140 >> v2:172.21.15.154:6812/254845342 conn(0x7fe568037840 0x7fe568039d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:59:48.847 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.817+0000 7fe588922640 10 monclient: get_auth_request con 0x7fe568037840 auth_method 0 2024-09-16T03:59:48.847 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.817+0000 7fe579ffb640 1 -- 172.21.15.42:0/400687140 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fe57c00b720 con 0x7fe58406dac0 2024-09-16T03:59:48.847 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.817+0000 7fe579ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:59:48.847 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.817+0000 7fe579ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:59:48.847 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.817+0000 7fe579ffb640 1 -- 172.21.15.42:0/400687140 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe58406dac0 msgr2=0x7fe584134e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:59:48.847 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.817+0000 7fe579ffb640 1 --2- 172.21.15.42:0/400687140 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe58406dac0 0x7fe584134e10 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fe57c00da60 tx=0x7fe57c00df20 comp rx=0 tx=0).stop 2024-09-16T03:59:48.847 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.817+0000 7fe579ffb640 1 -- 172.21.15.42:0/400687140 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe58406dac0 msgr2=0x7fe584134e10 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:59:48.847 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.817+0000 7fe579ffb640 1 --2- 172.21.15.42:0/400687140 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe58406dac0 0x7fe584134e10 secure :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fe57c00da60 tx=0x7fe57c00df20 comp rx=0 tx=0).stop 2024-09-16T03:59:48.848 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.817+0000 7fe588922640 1 --2- 172.21.15.42:0/400687140 >> v2:172.21.15.154:6812/254845342 conn(0x7fe568037840 0x7fe568039d00 secure :-1 s=READY pgs=3737 cs=0 l=1 rev1=1 crypto rx=0x7fe5740ad000 tx=0x7fe5740a4f90 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:59:48.848 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.817+0000 7fe579ffb640 10 client.825201.objecter ms_handle_connect 0x7fe568037840 2024-09-16T03:59:48.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.821+0000 7fe58b387640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:59:48.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.821+0000 7fe58b387640 1 --2- 172.21.15.42:0/400687140 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe584135350 0x7fe584138330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:59:48.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.821+0000 7fe58b387640 10 monclient(hunting): start opening mon connection 2024-09-16T03:59:48.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.821+0000 7fe58b387640 1 -- 172.21.15.42:0/400687140 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fe584135730 con 0x7fe584135350 2024-09-16T03:59:48.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.825+0000 7fe589123640 1 --2- 172.21.15.42:0/400687140 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe584135350 0x7fe584138330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:59:48.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.825+0000 7fe589123640 10 monclient: get_auth_request con 0x7fe584135350 auth_method 0 2024-09-16T03:59:48.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.825+0000 7fe589123640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:59:48.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.825+0000 7fe589123640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:59:48.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.825+0000 7fe589123640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:59:48.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.825+0000 7fe589123640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:59:48.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.825+0000 7fe589123640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:59:48.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.825+0000 7fe589123640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:59:48.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.825+0000 7fe589123640 10 monclient(hunting): handle_auth_done global_id 825207 payload 824 2024-09-16T03:59:48.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.825+0000 7fe589123640 1 -- 172.21.15.42:0/400687140 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fe58411d0d0 con 0x7fe584135350 2024-09-16T03:59:48.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.825+0000 7fe589123640 1 --2- 172.21.15.42:0/400687140 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe584135350 0x7fe584138330 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fe57c00da60 tx=0x7fe57c001410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:59:48.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.825+0000 7fe579ffb640 10 client.825201.objecter ms_handle_connect 0x7fe584135350 2024-09-16T03:59:48.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.825+0000 7fe579ffb640 10 client.825201.objecter resend_mon_ops 2024-09-16T03:59:48.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.825+0000 7fe579ffb640 1 -- 172.21.15.42:0/400687140 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe57c018050 con 0x7fe584135350 2024-09-16T03:59:48.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.825+0000 7fe579ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:59:48.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.825+0000 7fe579ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:59:48.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.825+0000 7fe579ffb640 10 monclient: dump: 2024-09-16T03:59:48.852 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:59:48.852 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:59:48.852 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:48.852 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:48.852 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:59:48.853 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:59:48.853 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:59:48.853 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:59:48.853 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:59:48.853 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:59:48.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.825+0000 7fe579ffb640 1 -- 172.21.15.42:0/400687140 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4715 (secure 0 0 0) 0x7fe57c019040 con 0x7fe584135350 2024-09-16T03:59:48.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.825+0000 7fe579ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:59:48.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.825+0000 7fe579ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:59:48.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.825+0000 7fe579ffb640 1 -- 172.21.15.42:0/400687140 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe584135350 msgr2=0x7fe584138330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:59:48.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.825+0000 7fe579ffb640 1 --2- 172.21.15.42:0/400687140 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe584135350 0x7fe584138330 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fe57c00da60 tx=0x7fe57c001410 comp rx=0 tx=0).stop 2024-09-16T03:59:48.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.825+0000 7fe579ffb640 1 -- 172.21.15.42:0/400687140 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe584135350 msgr2=0x7fe584138330 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:59:48.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.825+0000 7fe579ffb640 1 --2- 172.21.15.42:0/400687140 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe584135350 0x7fe584138330 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:59:48.854 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:59:48.854 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:59:48.854 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:59:48.855 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "leader", 2024-09-16T03:59:48.855 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 160, 2024-09-16T03:59:48.855 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 478679, 2024-09-16T03:59:48.855 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:59:48.855 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:59:48.855 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T03:59:48.855 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T03:59:48.855 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:59:48.855 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 7, 2024-09-16T03:59:48.855 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:59:48.855 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:59:48.855 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:59:48.855 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:59:48.856 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:59:48.856 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:59:48.856 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:59:48.856 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:59:48.856 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:59:48.856 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:59:48.856 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:59:48.856 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:59:48.856 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:59:48.856 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:59:48.856 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:59:48.856 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:59:48.856 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:59:48.857 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:59:48.857 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:59:48.857 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:59:48.857 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:59:48.857 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:59:48.857 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:59:48.857 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:59:48.857 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:59:48.858 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:59:48.858 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:59:48.858 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:59:48.858 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:59:48.858 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:59:48.858 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:59:48.858 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:59:48.858 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:59:48.859 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:59:48.859 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:59:48.859 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:59:48.859 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:59:48.859 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:59:48.859 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:59:48.859 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:59:48.859 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:59:48.860 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:59:48.860 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:59:48.860 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:59:48.860 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:59:48.860 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:59:48.860 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:59:48.860 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:59:48.860 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:59:48.861 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:59:48.861 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:59:48.861 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:59:48.861 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:59:48.861 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:59:48.861 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:59:48.861 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:59:48.861 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:59:48.861 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:59:48.862 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:59:48.862 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:59:48.862 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:59:48.862 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:59:48.862 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:59:48.862 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:59:48.862 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:59:48.862 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:59:48.862 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:59:48.862 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:59:48.862 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:59:48.862 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:59:48.862 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:59:48.863 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:59:48.863 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:59:48.863 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:59:48.863 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:59:48.863 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:59:48.863 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:59:48.863 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:59:48.863 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:59:48.863 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:59:48.863 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:59:48.863 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:59:48.863 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:59:48.864 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:59:48.864 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:59:48.864 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:59:48.864 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:59:48.864 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:59:48.864 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:59:48.864 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:59:48.864 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:59:48.864 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:59:48.864 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:59:48.865 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:59:48.865 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:59:48.865 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:59:48.865 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:59:48.865 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:59:48.865 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:59:48.865 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:59:48.865 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:59:48.865 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:59:48.866 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:59:48.866 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:59:48.866 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:59:48.866 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:59:48.866 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:59:48.866 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:59:48.866 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:59:48.866 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:59:48.866 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:59:48.867 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:59:48.867 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:59:48.867 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:59:48.867 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:59:48.867 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:59:48.867 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:59:48.867 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:59:48.867 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:59:48.867 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:59:48.867 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:59:48.868 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:59:48.868 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:59:48.868 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:59:48.868 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:59:48.868 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:59:48.868 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:59:48.868 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:59:48.868 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:59:48.868 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:59:48.868 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:59:48.868 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:59:48.868 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:59:48.868 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:59:48.869 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:59:48.869 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:59:48.869 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:59:48.869 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:59:48.869 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:59:48.869 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:59:48.869 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:59:48.869 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T03:59:48.869 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:59:48.869 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:59:48.869 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:59:48.869 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 5 2024-09-16T03:59:48.869 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:59:48.869 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:59:48.870 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:59:48.870 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:59:48.870 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:59:48.870 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:59:48.870 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 7 2024-09-16T03:59:48.870 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:59:48.870 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:59:48.870 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:59:48.870 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:59:48.870 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:59:48.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.825+0000 7fe58b387640 10 librados: watch_flush enter 2024-09-16T03:59:48.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.825+0000 7fe58b387640 10 librados: watch_flush exit 2024-09-16T03:59:48.873 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.825+0000 7fe58b387640 20 client.825201.objecter shutdown clearing up homeless session... 2024-09-16T03:59:48.873 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.825+0000 7fe58b387640 10 client.825201.objecter successfully canceled tick 2024-09-16T03:59:48.873 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.825+0000 7fe58b387640 1 -- 172.21.15.42:0/400687140 >> v2:172.21.15.154:6812/254845342 conn(0x7fe568037840 msgr2=0x7fe568039d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:59:48.873 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.825+0000 7fe58b387640 1 --2- 172.21.15.42:0/400687140 >> v2:172.21.15.154:6812/254845342 conn(0x7fe568037840 0x7fe568039d00 secure :-1 s=READY pgs=3737 cs=0 l=1 rev1=1 crypto rx=0x7fe5740ad000 tx=0x7fe5740a4f90 comp rx=0 tx=0).stop 2024-09-16T03:59:48.873 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.825+0000 7fe58b387640 10 monclient: shutdown 2024-09-16T03:59:48.873 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.825+0000 7fe58b387640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:59:48.873 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.825+0000 7fe58b387640 1 -- 172.21.15.42:0/400687140 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe58411c9f0 msgr2=0x7fe5841348d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:59:48.873 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.825+0000 7fe58b387640 1 --2- 172.21.15.42:0/400687140 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe58411c9f0 0x7fe5841348d0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fe58000c3c0 tx=0x7fe58000c880 comp rx=0 tx=0).stop 2024-09-16T03:59:48.873 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.825+0000 7fe589924640 1 -- 172.21.15.42:0/400687140 reap_dead start 2024-09-16T03:59:48.873 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.825+0000 7fe589924640 1 -- 172.21.15.42:0/400687140 reap_dead start 2024-09-16T03:59:48.873 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.825+0000 7fe58b387640 1 -- 172.21.15.42:0/400687140 shutdown_connections 2024-09-16T03:59:48.873 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.825+0000 7fe58b387640 1 -- 172.21.15.42:0/400687140 >> 172.21.15.42:0/400687140 conn(0x7fe5840a2420 msgr2=0x7fe5840a45b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:59:48.873 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.825+0000 7fe58b387640 1 -- 172.21.15.42:0/400687140 shutdown_connections 2024-09-16T03:59:48.874 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.825+0000 7fe58b387640 1 -- 172.21.15.42:0/400687140 wait complete. 2024-09-16T03:59:48.874 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.825+0000 7fe58b387640 1 librados: shutdown 2024-09-16T03:59:48.874 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:59:48.874 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T03:59:48.874 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T03:59:48.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.993+0000 7f9838db4640 10 monclient: get_monmap_and_config 2024-09-16T03:59:48.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.993+0000 7f9838db4640 10 monclient: build_initial_monmap 2024-09-16T03:59:48.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.993+0000 7f9838db4640 10 monclient: monmap: 2024-09-16T03:59:48.999 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:59:48.999 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:59:48.999 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:59:48.996718+0000 2024-09-16T03:59:48.999 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:59:48.996718+0000 2024-09-16T03:59:49.000 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:59:49.000 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:59:49.000 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:59:49.000 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:59:49.000 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:59:49.000 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:59:49.001 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:59:49.001 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:59:49.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.993+0000 7f9838db4640 1 Processor -- start 2024-09-16T03:59:49.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.993+0000 7f9838db4640 1 -- start start 2024-09-16T03:59:49.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.993+0000 7f9838db4640 10 monclient: init 2024-09-16T03:59:49.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.993+0000 7f9838db4640 10 monclient: _reopen_session rank -1 2024-09-16T03:59:49.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.993+0000 7f9838db4640 10 monclient: _add_conns ranks=[3,1,0,2,4] 2024-09-16T03:59:49.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.993+0000 7f9838db4640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f98340a39b0 0x7f9834133b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:59:49.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.993+0000 7f9838db4640 10 monclient(hunting): picked mon.noname-a con 0x7f98340a39b0 addr v2:172.21.15.42:3300/0 2024-09-16T03:59:49.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.993+0000 7f9838db4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f98340a3d90 addr v1:172.21.15.154:6789/0 2024-09-16T03:59:49.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.993+0000 7f9838db4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9834137710 addr v1:172.21.15.42:6789/0 2024-09-16T03:59:49.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.993+0000 7f9838db4640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f983409c140 con 0x7f9834137710 2024-09-16T03:59:49.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.993+0000 7f9838db4640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f983409c420 con 0x7f98340a3d90 2024-09-16T03:59:49.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.993+0000 7f9838db4640 10 monclient(hunting): start opening mon connection 2024-09-16T03:59:49.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.993+0000 7f9838db4640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f983406dd60 con 0x7f98340a39b0 2024-09-16T03:59:49.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.993+0000 7f9838db4640 10 monclient(hunting): _renew_subs 2024-09-16T03:59:49.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.993+0000 7f9838db4640 10 monclient(hunting): authenticate will time out at 2035.966192s 2024-09-16T03:59:49.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.993+0000 7f9832d9d640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f9834137710 0x7f983413f3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:34826/0 (socket says 172.21.15.42:34826) 2024-09-16T03:59:49.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.993+0000 7f9832d9d640 1 -- 172.21.15.42:0/2895770176 learned_addr learned my addr 172.21.15.42:0/2895770176 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:59:49.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.993+0000 7f983259c640 1 --2- 172.21.15.42:0/2895770176 >> v2:172.21.15.42:3300/0 conn(0x7f98340a39b0 0x7f9834133b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:59:49.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.993+0000 7f983259c640 10 monclient(hunting): get_auth_request con 0x7f98340a39b0 auth_method 0 2024-09-16T03:59:49.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.993+0000 7f983259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:59:49.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.993+0000 7f983259c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:59:49.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.993+0000 7f983259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:59:49.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.993+0000 7f983259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:59:49.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.993+0000 7f983259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:59:49.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.993+0000 7f983259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:59:49.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.993+0000 7f983159a640 1 -- 172.21.15.42:0/2895770176 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 113170724 0 0) 0x7f983409c140 con 0x7f9834137710 2024-09-16T03:59:49.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.993+0000 7f983159a640 10 monclient(hunting): _negotiate 2024-09-16T03:59:49.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.993+0000 7f983159a640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:59:49.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.993+0000 7f983159a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:59:49.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.993+0000 7f983159a640 10 monclient(hunting): my global_id is 825213 2024-09-16T03:59:49.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.993+0000 7f983159a640 1 -- 172.21.15.42:0/2895770176 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f981c003530 con 0x7f9834137710 2024-09-16T03:59:49.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.993+0000 7f983259c640 10 monclient(hunting): handle_auth_done global_id 825210 payload 274 2024-09-16T03:59:49.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.993+0000 7f983259c640 1 -- 172.21.15.42:0/2895770176 >> v1:172.21.15.154:6789/0 conn(0x7f98340a3d90 legacy=0x7f9834137000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:59:49.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.993+0000 7f983259c640 1 -- 172.21.15.42:0/2895770176 >> v1:172.21.15.42:6789/0 conn(0x7f9834137710 legacy=0x7f983413f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:59:49.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.993+0000 7f983259c640 10 monclient: _finish_hunting 0 2024-09-16T03:59:49.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.993+0000 7f983259c640 1 monclient: found mon.noname-a 2024-09-16T03:59:49.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.993+0000 7f983259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:59:49.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.993+0000 7f983259c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T03:59:49.016 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.993+0000 7f983259c640 1 -- 172.21.15.42:0/2895770176 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98340a5e20 con 0x7f98340a39b0 2024-09-16T03:59:49.016 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.997+0000 7f983259c640 1 --2- 172.21.15.42:0/2895770176 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f98340a39b0 0x7f9834133b80 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f98240097c0 tx=0x7f982401c360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eddf9607ac6afacd server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:59:49.016 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.997+0000 7f983159a640 1 -- 172.21.15.42:0/2895770176 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98240b0070 con 0x7f98340a39b0 2024-09-16T03:59:49.016 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.997+0000 7f983159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:59:49.016 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.997+0000 7f983159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T03:59:49.016 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.997+0000 7f983159a640 10 monclient: dump: 2024-09-16T03:59:49.016 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:59:49.016 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:59:49.016 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:49.016 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:49.017 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:59:49.017 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:59:49.017 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:59:49.017 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:59:49.017 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:59:49.017 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:59:49.018 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.997+0000 7f983159a640 10 monclient: _finish_auth 0 2024-09-16T03:59:49.018 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.997+0000 7f983159a640 10 monclient: _check_auth_tickets 2024-09-16T03:59:49.018 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.997+0000 7f983159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:59:49.018 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.997+0000 7f9838db4640 5 monclient: authenticate success, global_id 825210 2024-09-16T03:59:49.018 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.997+0000 7f9838db4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T03:59:49.018 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.997+0000 7f983159a640 1 -- 172.21.15.42:0/2895770176 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f98240ab600 con 0x7f98340a39b0 2024-09-16T03:59:49.019 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.997+0000 7f983159a640 10 monclient: handle_config config(0 keys) 2024-09-16T03:59:49.019 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.997+0000 7f983159a640 1 -- 172.21.15.42:0/2895770176 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98240ab9b0 con 0x7f98340a39b0 2024-09-16T03:59:49.019 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.997+0000 7f983159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:59:49.019 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.997+0000 7f983159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T03:59:49.019 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.997+0000 7f983159a640 10 monclient: dump: 2024-09-16T03:59:49.019 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:59:49.020 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:59:49.020 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:49.020 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:49.020 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:59:49.020 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:59:49.020 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:59:49.021 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:59:49.021 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:59:49.021 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:59:49.021 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.997+0000 7f9838db4640 10 monclient: get_monmap_and_config success 2024-09-16T03:59:49.021 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.997+0000 7f9838db4640 4 set_mon_vals no callback set 2024-09-16T03:59:49.021 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.997+0000 7f9838db4640 10 monclient: shutdown 2024-09-16T03:59:49.021 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.997+0000 7f9838db4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:59:49.022 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.997+0000 7f9838db4640 1 -- 172.21.15.42:0/2895770176 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f98340a39b0 msgr2=0x7f9834133b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:59:49.022 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.997+0000 7f9838db4640 1 --2- 172.21.15.42:0/2895770176 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f98340a39b0 0x7f9834133b80 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f98240097c0 tx=0x7f982401c360 comp rx=0 tx=0).stop 2024-09-16T03:59:49.022 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.997+0000 7f9838db4640 1 -- 172.21.15.42:0/2895770176 shutdown_connections 2024-09-16T03:59:49.023 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.997+0000 7f9838db4640 1 --2- 172.21.15.42:0/2895770176 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f98340a39b0 0x7f9834133b80 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:59:49.023 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.997+0000 7f9838db4640 1 -- 172.21.15.42:0/2895770176 >> 172.21.15.42:0/2895770176 conn(0x7f98340a9fb0 msgr2=0x7f98340aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:59:49.023 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.997+0000 7f9838db4640 1 -- 172.21.15.42:0/2895770176 shutdown_connections 2024-09-16T03:59:49.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.997+0000 7f9838db4640 1 -- 172.21.15.42:0/2895770176 wait complete. 2024-09-16T03:59:49.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.997+0000 7f9838db4640 10 monclient: build_initial_monmap 2024-09-16T03:59:49.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.997+0000 7f9838db4640 10 monclient: monmap: 2024-09-16T03:59:49.024 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:59:49.024 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:59:49.024 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:59:49.024 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:59:49.024 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:59:49.024 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:59:49.024 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:59:49.024 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:59:49.024 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:59:49.024 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:59:49.025 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.997+0000 7f9838db4640 1 librados: starting msgr at 2024-09-16T03:59:49.025 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.997+0000 7f9838db4640 1 librados: starting objecter 2024-09-16T03:59:49.025 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.997+0000 7f9838db4640 1 Processor -- start 2024-09-16T03:59:49.025 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.997+0000 7f9838db4640 1 -- start start 2024-09-16T03:59:49.025 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.997+0000 7f9838db4640 1 librados: setting wanted keys 2024-09-16T03:59:49.025 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.997+0000 7f9838db4640 1 librados: calling monclient init 2024-09-16T03:59:49.025 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:48.997+0000 7f9838db4640 10 monclient: init 2024-09-16T03:59:49.025 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.001+0000 7f9838db4640 10 monclient: _reopen_session rank -1 2024-09-16T03:59:49.025 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.001+0000 7f9838db4640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T03:59:49.025 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.001+0000 7f9838db4640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f98340a39b0 0x7f98341cfd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:59:49.025 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.001+0000 7f9838db4640 10 monclient(hunting): picked mon.noname-a con 0x7f98340a39b0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:59:49.025 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.001+0000 7f9838db4640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f98340a3d90 0x7f98341d02b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:59:49.025 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.001+0000 7f9838db4640 10 monclient(hunting): picked mon.noname-c con 0x7f98340a3d90 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:59:49.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.001+0000 7f9838db4640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9834137710 0x7f98340a1640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:59:49.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.001+0000 7f9838db4640 10 monclient(hunting): picked mon.noname-b con 0x7f9834137710 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:59:49.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.001+0000 7f9838db4640 10 monclient(hunting): start opening mon connection 2024-09-16T03:59:49.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.001+0000 7f9838db4640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f9834143900 con 0x7f98340a39b0 2024-09-16T03:59:49.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.001+0000 7f9838db4640 10 monclient(hunting): start opening mon connection 2024-09-16T03:59:49.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.001+0000 7f9838db4640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f983406dee0 con 0x7f9834137710 2024-09-16T03:59:49.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.001+0000 7f9838db4640 10 monclient(hunting): start opening mon connection 2024-09-16T03:59:49.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.001+0000 7f9838db4640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f9834143a80 con 0x7f98340a3d90 2024-09-16T03:59:49.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.001+0000 7f9838db4640 10 monclient(hunting): _renew_subs 2024-09-16T03:59:49.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.001+0000 7f9838db4640 10 monclient(hunting): authenticate will time out at 2035.972370s 2024-09-16T03:59:49.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.001+0000 7f9831d9b640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f98340a3d90 0x7f98341d02b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:59:49.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.001+0000 7f983259c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f98340a39b0 0x7f98341cfd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:59:49.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.001+0000 7f9831d9b640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f98340a3d90 0x7f98341d02b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:44232/0 (socket says 172.21.15.42:44232) 2024-09-16T03:59:49.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.001+0000 7f983259c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f98340a39b0 0x7f98341cfd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:36888/0 (socket says 172.21.15.42:36888) 2024-09-16T03:59:49.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.001+0000 7f9831d9b640 1 -- 172.21.15.42:0/3805307117 learned_addr learned my addr 172.21.15.42:0/3805307117 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:59:49.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.001+0000 7f9831d9b640 10 monclient(hunting): get_auth_request con 0x7f98340a3d90 auth_method 0 2024-09-16T03:59:49.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.001+0000 7f9831d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:59:49.029 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.001+0000 7f9831d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:59:49.029 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.001+0000 7f9831d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:59:49.029 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.001+0000 7f983259c640 10 monclient(hunting): get_auth_request con 0x7f98340a39b0 auth_method 0 2024-09-16T03:59:49.029 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.001+0000 7f983259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:59:49.029 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.001+0000 7f983259c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:59:49.029 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.001+0000 7f983259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:59:49.029 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.001+0000 7f9831d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:59:49.029 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.001+0000 7f9831d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:59:49.029 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.001+0000 7f9831d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:59:49.029 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.001+0000 7f983259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:59:49.030 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.001+0000 7f983259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:59:49.030 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.001+0000 7f983259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:59:49.030 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.001+0000 7f9831d9b640 10 monclient(hunting): handle_auth_done global_id 934127 payload 824 2024-09-16T03:59:49.030 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.001+0000 7f9832d9d640 1 --2- 172.21.15.42:0/3805307117 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9834137710 0x7f98340a1640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:59:49.030 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.001+0000 7f9831d9b640 1 -- 172.21.15.42:0/3805307117 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9834137710 msgr2=0x7f98340a1640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:59:49.030 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.001+0000 7f9831d9b640 1 --2- 172.21.15.42:0/3805307117 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9834137710 0x7f98340a1640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:59:49.030 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.001+0000 7f9831d9b640 1 -- 172.21.15.42:0/3805307117 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f98340a39b0 msgr2=0x7f98341cfd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:59:49.030 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.001+0000 7f9831d9b640 1 --2- 172.21.15.42:0/3805307117 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f98340a39b0 0x7f98341cfd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:59:49.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.001+0000 7f9831d9b640 10 monclient: _finish_hunting 0 2024-09-16T03:59:49.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.001+0000 7f9831d9b640 1 monclient: found mon.noname-c 2024-09-16T03:59:49.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.001+0000 7f9831d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:59:49.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.001+0000 7f9831d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:59:49.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.001+0000 7f9831d9b640 1 -- 172.21.15.42:0/3805307117 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9834139cc0 con 0x7f98340a3d90 2024-09-16T03:59:49.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.001+0000 7f983259c640 1 --2- 172.21.15.42:0/3805307117 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f98340a39b0 0x7f98341cfd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T03:59:49.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.001+0000 7f9831d9b640 1 --2- 172.21.15.42:0/3805307117 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f98340a3d90 0x7f98341d02b0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f982800c420 tx=0x7f982800c8e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:59:49.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.001+0000 7f9822ffd640 10 client.?.objecter ms_handle_connect 0x7f98340a3d90 2024-09-16T03:59:49.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.001+0000 7f9822ffd640 10 client.?.objecter resend_mon_ops 2024-09-16T03:59:49.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.001+0000 7f9822ffd640 1 -- 172.21.15.42:0/3805307117 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98280052d0 con 0x7f98340a3d90 2024-09-16T03:59:49.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.001+0000 7f9822ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:59:49.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.001+0000 7f9822ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:59:49.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.001+0000 7f9822ffd640 10 monclient: dump: 2024-09-16T03:59:49.032 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:59:49.032 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:59:49.032 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:49.033 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:49.033 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:59:49.033 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:59:49.033 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:59:49.033 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:59:49.033 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:59:49.033 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:59:49.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.001+0000 7f9822ffd640 10 monclient: _finish_auth 0 2024-09-16T03:59:49.034 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.001+0000 7f9822ffd640 10 monclient: _check_auth_tickets 2024-09-16T03:59:49.034 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.001+0000 7f9822ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:59:49.034 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.001+0000 7f9822ffd640 1 -- 172.21.15.42:0/3805307117 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9828005470 con 0x7f98340a3d90 2024-09-16T03:59:49.034 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.001+0000 7f9822ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T03:59:49.034 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.001+0000 7f9822ffd640 1 -- 172.21.15.42:0/3805307117 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9828007260 con 0x7f98340a3d90 2024-09-16T03:59:49.034 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.001+0000 7f9838db4640 5 monclient: authenticate success, global_id 934127 2024-09-16T03:59:49.034 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.001+0000 7f9838db4640 10 monclient: _renew_subs 2024-09-16T03:59:49.034 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.001+0000 7f9838db4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:59:49.034 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.001+0000 7f9838db4640 1 -- 172.21.15.42:0/3805307117 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9834136e90 con 0x7f98340a3d90 2024-09-16T03:59:49.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.001+0000 7f9830d99640 4 set_mon_vals no callback set 2024-09-16T03:59:49.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.001+0000 7f9822ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:59:49.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.001+0000 7f9822ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:59:49.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.001+0000 7f9822ffd640 10 monclient: dump: 2024-09-16T03:59:49.035 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:59:49.035 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:59:49.035 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:49.036 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:49.036 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:59:49.036 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:59:49.036 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:59:49.036 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:59:49.036 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:59:49.037 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:59:49.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.001+0000 7f9838db4640 10 client.934127.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:59:49.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.001+0000 7f9838db4640 10 monclient: _renew_subs 2024-09-16T03:59:49.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.001+0000 7f9838db4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:59:49.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.001+0000 7f9838db4640 1 -- 172.21.15.42:0/3805307117 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f983406f680 con 0x7f98340a3d90 2024-09-16T03:59:49.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.001+0000 7f9838db4640 1 librados: init done 2024-09-16T03:59:49.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.001+0000 7f9838db4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:59:49.039 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.001+0000 7f9838db4640 1 --2- 172.21.15.42:0/3805307117 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9834139610 0x7f98340a1b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:59:49.039 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.001+0000 7f9838db4640 10 monclient(hunting): start opening mon connection 2024-09-16T03:59:49.039 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.001+0000 7f9838db4640 1 -- 172.21.15.42:0/3805307117 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f9834133b80 con 0x7f9834139610 2024-09-16T03:59:49.039 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.001+0000 7f983259c640 1 --2- 172.21.15.42:0/3805307117 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9834139610 0x7f98340a1b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:59:49.039 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.001+0000 7f983259c640 10 monclient: get_auth_request con 0x7f9834139610 auth_method 0 2024-09-16T03:59:49.039 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.001+0000 7f983259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:59:49.040 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.001+0000 7f983259c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:59:49.040 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.001+0000 7f983259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:59:49.040 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.001+0000 7f9822ffd640 1 -- 172.21.15.42:0/3805307117 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f9828007400 con 0x7f98340a3d90 2024-09-16T03:59:49.040 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.001+0000 7f983259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:59:49.040 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.001+0000 7f983259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:59:49.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.001+0000 7f983259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:59:49.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.001+0000 7f9822ffd640 1 --2- 172.21.15.42:0/3805307117 >> v2:172.21.15.154:6812/254845342 conn(0x7f9814037810 0x7f9814039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:59:49.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.001+0000 7f983259c640 10 monclient(hunting): handle_auth_done global_id 934130 payload 824 2024-09-16T03:59:49.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.001+0000 7f983259c640 1 -- 172.21.15.42:0/3805307117 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f98340a5750 con 0x7f9834139610 2024-09-16T03:59:49.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.001+0000 7f9822ffd640 1 -- 172.21.15.42:0/3805307117 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(540..540 src has 451..540) ==== 6168+0+0 (secure 0 0 0) 0x7f9828048630 con 0x7f98340a3d90 2024-09-16T03:59:49.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.001+0000 7f9822ffd640 10 client.934127.objecter ms_dispatch 0x7f9834000ba0 osd_map(540..540 src has 451..540) 2024-09-16T03:59:49.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.001+0000 7f9822ffd640 3 client.934127.objecter handle_osd_map got epochs [540,540] > 0 2024-09-16T03:59:49.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.001+0000 7f9822ffd640 3 client.934127.objecter handle_osd_map decoding full epoch 540 2024-09-16T03:59:49.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.001+0000 7f9822ffd640 20 client.934127.objecter dump_active .. 0 homeless 2024-09-16T03:59:49.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.001+0000 7f9822ffd640 10 client.934127.objecter ms_handle_connect 0x7f9834139610 2024-09-16T03:59:49.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.001+0000 7f9822ffd640 10 client.934127.objecter resend_mon_ops 2024-09-16T03:59:49.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.001+0000 7f983259c640 1 --2- 172.21.15.42:0/3805307117 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9834139610 0x7f98340a1b80 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f983406b540 tx=0x7f9824019100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:59:49.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.001+0000 7f9822ffd640 1 -- 172.21.15.42:0/3805307117 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98240ba070 con 0x7f9834139610 2024-09-16T03:59:49.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.001+0000 7f9822ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:59:49.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.001+0000 7f9822ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:59:49.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.001+0000 7f9822ffd640 10 monclient: dump: 2024-09-16T03:59:49.042 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:59:49.042 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:59:49.042 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:49.043 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:49.043 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:59:49.043 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:59:49.043 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:59:49.043 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:59:49.043 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:59:49.043 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:59:49.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.001+0000 7f9832d9d640 1 --2- 172.21.15.42:0/3805307117 >> v2:172.21.15.154:6812/254845342 conn(0x7f9814037810 0x7f9814039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:59:49.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.001+0000 7f9832d9d640 10 monclient: get_auth_request con 0x7f9814037810 auth_method 0 2024-09-16T03:59:49.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.001+0000 7f9822ffd640 1 -- 172.21.15.42:0/3805307117 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f98240b1040 con 0x7f9834139610 2024-09-16T03:59:49.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.001+0000 7f9822ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:59:49.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.001+0000 7f9822ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:59:49.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.001+0000 7f9822ffd640 1 -- 172.21.15.42:0/3805307117 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9834139610 msgr2=0x7f98340a1b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:59:49.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.001+0000 7f9822ffd640 1 --2- 172.21.15.42:0/3805307117 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9834139610 0x7f98340a1b80 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f983406b540 tx=0x7f9824019100 comp rx=0 tx=0).stop 2024-09-16T03:59:49.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.001+0000 7f9822ffd640 1 -- 172.21.15.42:0/3805307117 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9834139610 msgr2=0x7f98340a1b80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:59:49.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.001+0000 7f9822ffd640 1 --2- 172.21.15.42:0/3805307117 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9834139610 0x7f98340a1b80 secure :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f983406b540 tx=0x7f9824019100 comp rx=0 tx=0).stop 2024-09-16T03:59:49.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.001+0000 7f9822ffd640 10 client.934127.objecter ms_handle_connect 0x7f9814037810 2024-09-16T03:59:49.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.001+0000 7f9832d9d640 1 --2- 172.21.15.42:0/3805307117 >> v2:172.21.15.154:6812/254845342 conn(0x7f9814037810 0x7f9814039cd0 secure :-1 s=READY pgs=3738 cs=0 l=1 rev1=1 crypto rx=0x7f982c001570 tx=0x7f982c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:59:49.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.009+0000 7f9838db4640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:59:49.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.009+0000 7f9838db4640 1 --2- 172.21.15.42:0/3805307117 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f98340a20c0 0x7f98340a24a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:59:49.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.009+0000 7f9838db4640 10 monclient(hunting): start opening mon connection 2024-09-16T03:59:49.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.009+0000 7f9838db4640 1 -- 172.21.15.42:0/3805307117 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f98340a29e0 con 0x7f98340a20c0 2024-09-16T03:59:49.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.009+0000 7f983259c640 1 --2- 172.21.15.42:0/3805307117 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f98340a20c0 0x7f98340a24a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:59:49.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.009+0000 7f983259c640 10 monclient: get_auth_request con 0x7f98340a20c0 auth_method 0 2024-09-16T03:59:49.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.009+0000 7f983259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:59:49.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.009+0000 7f983259c640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:59:49.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.009+0000 7f983259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:59:49.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.009+0000 7f983259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:59:49.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.009+0000 7f983259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:59:49.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.009+0000 7f983259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:59:49.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.009+0000 7f983259c640 10 monclient(hunting): handle_auth_done global_id 934133 payload 824 2024-09-16T03:59:49.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.009+0000 7f983259c640 1 -- 172.21.15.42:0/3805307117 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f9834068e40 con 0x7f98340a20c0 2024-09-16T03:59:49.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.009+0000 7f983259c640 1 --2- 172.21.15.42:0/3805307117 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f98340a20c0 0x7f98340a24a0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f9824019f90 tx=0x7f9824009190 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:59:49.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.009+0000 7f9822ffd640 10 client.934127.objecter ms_handle_connect 0x7f98340a20c0 2024-09-16T03:59:49.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.009+0000 7f9822ffd640 10 client.934127.objecter resend_mon_ops 2024-09-16T03:59:49.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.009+0000 7f9822ffd640 1 -- 172.21.15.42:0/3805307117 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98240b0070 con 0x7f98340a20c0 2024-09-16T03:59:49.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.009+0000 7f9822ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:59:49.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.009+0000 7f9822ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:59:49.048 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.009+0000 7f9822ffd640 10 monclient: dump: 2024-09-16T03:59:49.048 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:59:49.048 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:59:49.048 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:49.048 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:49.048 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:59:49.048 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:59:49.048 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:59:49.048 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:59:49.048 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:59:49.048 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:59:49.048 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.009+0000 7f9822ffd640 1 -- 172.21.15.42:0/3805307117 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4545 (secure 0 0 0) 0x7f98240aba00 con 0x7f98340a20c0 2024-09-16T03:59:49.048 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.009+0000 7f9822ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:59:49.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.009+0000 7f9822ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:59:49.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.009+0000 7f9822ffd640 1 -- 172.21.15.42:0/3805307117 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f98340a20c0 msgr2=0x7f98340a24a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:59:49.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.009+0000 7f9822ffd640 1 --2- 172.21.15.42:0/3805307117 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f98340a20c0 0x7f98340a24a0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f9824019f90 tx=0x7f9824009190 comp rx=0 tx=0).stop 2024-09-16T03:59:49.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.009+0000 7f9822ffd640 1 -- 172.21.15.42:0/3805307117 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f98340a20c0 msgr2=0x7f98340a24a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:59:49.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.009+0000 7f9822ffd640 1 --2- 172.21.15.42:0/3805307117 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f98340a20c0 0x7f98340a24a0 secure :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f9824019f90 tx=0x7f9824009190 comp rx=0 tx=0).stop 2024-09-16T03:59:49.049 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:59:49.049 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:59:49.049 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:59:49.049 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T03:59:49.049 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 160, 2024-09-16T03:59:49.050 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 13735, 2024-09-16T03:59:49.050 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:59:49.050 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:59:49.050 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T03:59:49.050 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T03:59:49.050 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:59:49.050 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 8, 2024-09-16T03:59:49.050 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:59:49.050 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:59:49.050 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:59:49.050 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:59:49.050 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:59:49.050 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:59:49.051 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:59:49.051 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:59:49.051 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:59:49.051 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:59:49.051 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:59:49.051 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:59:49.051 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:59:49.051 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:59:49.051 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:59:49.051 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:59:49.051 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:59:49.051 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:59:49.051 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:59:49.052 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:59:49.052 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:59:49.052 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:59:49.052 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:59:49.052 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:59:49.052 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:59:49.052 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:59:49.052 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:59:49.052 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:59:49.052 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:59:49.052 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:59:49.052 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:59:49.052 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:59:49.053 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:59:49.053 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:59:49.053 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:59:49.053 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:59:49.053 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:59:49.053 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:59:49.053 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:59:49.053 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:59:49.053 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:59:49.054 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:59:49.054 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:59:49.055 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:59:49.055 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:59:49.055 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:59:49.055 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:59:49.055 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:59:49.055 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:59:49.055 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:59:49.055 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:59:49.055 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:59:49.055 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:59:49.055 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:59:49.055 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:59:49.055 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:59:49.056 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:59:49.056 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:59:49.056 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:59:49.056 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:59:49.056 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:59:49.056 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:59:49.056 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:59:49.056 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:59:49.056 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:59:49.056 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:59:49.056 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:59:49.057 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:59:49.057 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:59:49.057 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:59:49.057 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:59:49.057 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:59:49.057 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:59:49.057 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:59:49.057 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:59:49.057 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:59:49.057 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:59:49.057 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:59:49.057 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:59:49.058 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:59:49.058 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:59:49.058 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:59:49.058 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:59:49.058 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:59:49.058 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:59:49.058 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:59:49.058 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:59:49.058 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:59:49.058 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:59:49.058 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:59:49.058 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:59:49.059 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:59:49.059 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:59:49.059 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:59:49.059 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:59:49.059 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:59:49.059 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:59:49.059 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:59:49.059 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:59:49.059 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:59:49.059 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:59:49.059 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:59:49.060 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:59:49.060 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:59:49.060 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:59:49.060 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:59:49.060 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:59:49.060 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:59:49.060 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:59:49.060 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:59:49.060 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:59:49.060 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:59:49.060 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:59:49.060 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:59:49.060 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:59:49.061 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:59:49.061 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:59:49.061 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:59:49.061 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:59:49.061 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:59:49.061 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:59:49.061 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:59:49.061 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:59:49.061 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:59:49.062 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:59:49.062 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:59:49.062 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:59:49.062 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:59:49.062 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:59:49.062 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:59:49.062 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:59:49.062 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:59:49.062 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:59:49.062 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:59:49.062 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:59:49.062 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:59:49.062 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:59:49.062 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:59:49.063 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:59:49.063 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:59:49.063 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:59:49.063 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:59:49.063 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:59:49.063 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:59:49.063 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:59:49.063 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 2 2024-09-16T03:59:49.063 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:59:49.063 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:59:49.063 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:59:49.063 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:59:49.064 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:59:49.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.009+0000 7f9838db4640 10 librados: watch_flush enter 2024-09-16T03:59:49.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.009+0000 7f9838db4640 10 librados: watch_flush exit 2024-09-16T03:59:49.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.009+0000 7f9838db4640 20 client.934127.objecter shutdown clearing up homeless session... 2024-09-16T03:59:49.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.009+0000 7f9838db4640 10 client.934127.objecter successfully canceled tick 2024-09-16T03:59:49.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.009+0000 7f9838db4640 1 -- 172.21.15.42:0/3805307117 >> v2:172.21.15.154:6812/254845342 conn(0x7f9814037810 msgr2=0x7f9814039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:59:49.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.009+0000 7f9838db4640 1 --2- 172.21.15.42:0/3805307117 >> v2:172.21.15.154:6812/254845342 conn(0x7f9814037810 0x7f9814039cd0 secure :-1 s=READY pgs=3738 cs=0 l=1 rev1=1 crypto rx=0x7f982c001570 tx=0x7f982c009040 comp rx=0 tx=0).stop 2024-09-16T03:59:49.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.009+0000 7f9838db4640 10 monclient: shutdown 2024-09-16T03:59:49.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.009+0000 7f9838db4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:59:49.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.009+0000 7f9838db4640 1 -- 172.21.15.42:0/3805307117 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f98340a3d90 msgr2=0x7f98341d02b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:59:49.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.009+0000 7f9838db4640 1 --2- 172.21.15.42:0/3805307117 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f98340a3d90 0x7f98341d02b0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f982800c420 tx=0x7f982800c8e0 comp rx=0 tx=0).stop 2024-09-16T03:59:49.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.009+0000 7f9832d9d640 1 -- 172.21.15.42:0/3805307117 reap_dead start 2024-09-16T03:59:49.067 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.009+0000 7f9832d9d640 1 -- 172.21.15.42:0/3805307117 reap_dead start 2024-09-16T03:59:49.067 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.009+0000 7f9838db4640 1 -- 172.21.15.42:0/3805307117 shutdown_connections 2024-09-16T03:59:49.067 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.009+0000 7f9838db4640 1 -- 172.21.15.42:0/3805307117 >> 172.21.15.42:0/3805307117 conn(0x7f98340a9fb0 msgr2=0x7f98341332e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:59:49.067 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.009+0000 7f9838db4640 1 -- 172.21.15.42:0/3805307117 shutdown_connections 2024-09-16T03:59:49.067 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.009+0000 7f9838db4640 1 -- 172.21.15.42:0/3805307117 wait complete. 2024-09-16T03:59:49.067 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.009+0000 7f9838db4640 1 librados: shutdown 2024-09-16T03:59:49.067 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:59:49.067 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T03:59:49.068 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T03:59:49.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.189+0000 7f8446ec1640 10 monclient: get_monmap_and_config 2024-09-16T03:59:49.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.189+0000 7f8446ec1640 10 monclient: build_initial_monmap 2024-09-16T03:59:49.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.189+0000 7f8446ec1640 10 monclient: monmap: 2024-09-16T03:59:49.194 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:59:49.195 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:59:49.195 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:59:49.192488+0000 2024-09-16T03:59:49.195 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:59:49.192488+0000 2024-09-16T03:59:49.195 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:59:49.195 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:59:49.195 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T03:59:49.196 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T03:59:49.196 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T03:59:49.196 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T03:59:49.196 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T03:59:49.196 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:59:49.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.189+0000 7f8446ec1640 1 Processor -- start 2024-09-16T03:59:49.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.189+0000 7f8446ec1640 1 -- start start 2024-09-16T03:59:49.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.189+0000 7f8446ec1640 10 monclient: init 2024-09-16T03:59:49.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.189+0000 7f8446ec1640 10 monclient: _reopen_session rank -1 2024-09-16T03:59:49.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.189+0000 7f8446ec1640 10 monclient: _add_conns ranks=[1,2,0,4,3] 2024-09-16T03:59:49.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.189+0000 7f8446ec1640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f84400a1330 addr v1:172.21.15.154:6789/0 2024-09-16T03:59:49.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.189+0000 7f8446ec1640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f844006dac0 0x7f844013a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:59:49.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.189+0000 7f8446ec1640 10 monclient(hunting): picked mon.noname-b con 0x7f844006dac0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:59:49.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.189+0000 7f8446ec1640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f84400a1e20 addr v1:172.21.15.42:6789/0 2024-09-16T03:59:49.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.189+0000 7f8446ec1640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f844009c7d0 con 0x7f84400a1e20 2024-09-16T03:59:49.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.189+0000 7f8446ec1640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f844009c090 con 0x7f84400a1330 2024-09-16T03:59:49.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.189+0000 7f8446ec1640 10 monclient(hunting): start opening mon connection 2024-09-16T03:59:49.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.189+0000 7f8446ec1640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f844009f610 con 0x7f844006dac0 2024-09-16T03:59:49.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.189+0000 7f8446ec1640 10 monclient(hunting): _renew_subs 2024-09-16T03:59:49.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.189+0000 7f8446ec1640 10 monclient(hunting): authenticate will time out at 2036.162027s 2024-09-16T03:59:49.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.189+0000 7f843ffff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f844006dac0 0x7f844013a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:59:49.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.189+0000 7f844545e640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f84400a1e20 0x7f844013c890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:34832/0 (socket says 172.21.15.42:34832) 2024-09-16T03:59:49.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.189+0000 7f844545e640 1 -- 172.21.15.42:0/2255309321 learned_addr learned my addr 172.21.15.42:0/2255309321 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T03:59:49.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.189+0000 7f843ffff640 10 monclient(hunting): get_auth_request con 0x7f844006dac0 auth_method 0 2024-09-16T03:59:49.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.189+0000 7f843ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:59:49.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.189+0000 7f843ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:59:49.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.189+0000 7f843ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:59:49.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.189+0000 7f843ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:59:49.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.189+0000 7f843ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:59:49.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.189+0000 7f844545e640 1 -- 172.21.15.42:0/2255309321 >> v1:172.21.15.42:6789/0 conn(0x7f84400a1e20 legacy=0x7f844013c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 15 2024-09-16T03:59:49.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.189+0000 7f844545e640 1 -- 172.21.15.42:0/2255309321 >> v1:172.21.15.42:6789/0 conn(0x7f84400a1e20 legacy=0x7f844013c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T03:59:49.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.189+0000 7f844545e640 1 --1- 172.21.15.42:0/2255309321 >> v1:172.21.15.42:6789/0 conn(0x7f84400a1e20 0x7f844013c890 :-1 s=OPENED pgs=485 cs=1 l=1).handle_message read tag failed 2024-09-16T03:59:49.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.189+0000 7f843ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:59:49.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.189+0000 7f844545e640 1 --1- 172.21.15.42:0/2255309321 >> v1:172.21.15.42:6789/0 conn(0x7f84400a1e20 0x7f844013c890 :-1 s=OPENED pgs=485 cs=1 l=1).fault on lossy channel, failing 2024-09-16T03:59:49.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.189+0000 7f843f7fe640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.42:6789/0 2024-09-16T03:59:49.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.189+0000 7f843ffff640 10 monclient(hunting): handle_auth_done global_id 934136 payload 274 2024-09-16T03:59:49.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.189+0000 7f843ffff640 1 -- 172.21.15.42:0/2255309321 >> v1:172.21.15.154:6789/0 conn(0x7f84400a1330 legacy=0x7f84400a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:59:49.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.189+0000 7f843ffff640 1 -- 172.21.15.42:0/2255309321 >> v1:172.21.15.42:6789/0 conn(0x7f84400a1e20 legacy=0x7f844013c890 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:59:49.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.189+0000 7f843ffff640 10 monclient: _finish_hunting 0 2024-09-16T03:59:49.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.189+0000 7f843ffff640 1 monclient: found mon.noname-b 2024-09-16T03:59:49.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.189+0000 7f843ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:59:49.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.189+0000 7f843ffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T03:59:49.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.189+0000 7f843ffff640 1 -- 172.21.15.42:0/2255309321 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f844009f8f0 con 0x7f844006dac0 2024-09-16T03:59:49.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.193+0000 7f843ffff640 1 --2- 172.21.15.42:0/2255309321 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f844006dac0 0x7f844013a340 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f844006add0 tx=0x7f843001c920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6dedc86cf44d6f83 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:59:49.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.193+0000 7f843f7fe640 1 -- 172.21.15.42:0/2255309321 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84300b2070 con 0x7f844006dac0 2024-09-16T03:59:49.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.193+0000 7f843f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:59:49.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.193+0000 7f843f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T03:59:49.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.193+0000 7f843f7fe640 10 monclient: dump: 2024-09-16T03:59:49.219 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:59:49.219 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:59:49.219 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:49.219 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:49.219 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:59:49.220 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:59:49.220 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:59:49.220 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:59:49.220 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:59:49.220 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:59:49.221 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.193+0000 7f843f7fe640 10 monclient: _finish_auth 0 2024-09-16T03:59:49.221 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.193+0000 7f843f7fe640 10 monclient: _check_auth_tickets 2024-09-16T03:59:49.221 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.193+0000 7f843f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:59:49.222 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.193+0000 7f843f7fe640 1 -- 172.21.15.42:0/2255309321 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f84300a5980 con 0x7f844006dac0 2024-09-16T03:59:49.222 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.193+0000 7f843f7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T03:59:49.222 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.193+0000 7f843f7fe640 1 -- 172.21.15.42:0/2255309321 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84300a5d50 con 0x7f844006dac0 2024-09-16T03:59:49.222 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.193+0000 7f8446ec1640 5 monclient: authenticate success, global_id 934136 2024-09-16T03:59:49.222 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.193+0000 7f8446ec1640 10 monclient: get_monmap_and_config success 2024-09-16T03:59:49.222 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.193+0000 7f8446ec1640 4 set_mon_vals no callback set 2024-09-16T03:59:49.222 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.193+0000 7f843f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:59:49.222 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.193+0000 7f843f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:59:49.222 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.193+0000 7f843f7fe640 10 monclient: dump: 2024-09-16T03:59:49.223 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:59:49.223 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:59:49.223 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:49.223 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:49.223 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:59:49.223 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:59:49.223 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:59:49.223 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:59:49.223 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:59:49.223 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:59:49.223 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.193+0000 7f8446ec1640 10 monclient: shutdown 2024-09-16T03:59:49.224 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.193+0000 7f8446ec1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:59:49.224 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.193+0000 7f8446ec1640 1 -- 172.21.15.42:0/2255309321 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f844006dac0 msgr2=0x7f844013a340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:59:49.224 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.193+0000 7f8446ec1640 1 --2- 172.21.15.42:0/2255309321 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f844006dac0 0x7f844013a340 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f844006add0 tx=0x7f843001c920 comp rx=0 tx=0).stop 2024-09-16T03:59:49.224 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.193+0000 7f8446ec1640 1 -- 172.21.15.42:0/2255309321 shutdown_connections 2024-09-16T03:59:49.224 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.193+0000 7f8446ec1640 1 --2- 172.21.15.42:0/2255309321 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f844006dac0 0x7f844013a340 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:59:49.224 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.193+0000 7f8446ec1640 1 -- 172.21.15.42:0/2255309321 >> 172.21.15.42:0/2255309321 conn(0x7f84400a2420 msgr2=0x7f84400a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:59:49.224 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.193+0000 7f8446ec1640 1 -- 172.21.15.42:0/2255309321 shutdown_connections 2024-09-16T03:59:49.224 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.193+0000 7f8446ec1640 1 -- 172.21.15.42:0/2255309321 wait complete. 2024-09-16T03:59:49.224 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.193+0000 7f8446ec1640 10 monclient: build_initial_monmap 2024-09-16T03:59:49.224 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.193+0000 7f8446ec1640 10 monclient: monmap: 2024-09-16T03:59:49.225 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T03:59:49.225 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T03:59:49.225 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T03:59:49.225 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T03:59:49.225 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T03:59:49.225 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:59:49.225 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T03:59:49.225 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T03:59:49.225 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T03:59:49.225 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:59:49.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.193+0000 7f8446ec1640 1 librados: starting msgr at 2024-09-16T03:59:49.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.193+0000 7f8446ec1640 1 librados: starting objecter 2024-09-16T03:59:49.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.193+0000 7f8446ec1640 1 Processor -- start 2024-09-16T03:59:49.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.193+0000 7f8446ec1640 1 -- start start 2024-09-16T03:59:49.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.193+0000 7f8446ec1640 1 librados: setting wanted keys 2024-09-16T03:59:49.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.193+0000 7f8446ec1640 1 librados: calling monclient init 2024-09-16T03:59:49.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.193+0000 7f8446ec1640 10 monclient: init 2024-09-16T03:59:49.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.193+0000 7f8446ec1640 10 monclient: _reopen_session rank -1 2024-09-16T03:59:49.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.193+0000 7f8446ec1640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T03:59:49.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.193+0000 7f8446ec1640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f84400a1330 0x7f84401cf110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:59:49.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.193+0000 7f8446ec1640 10 monclient(hunting): picked mon.noname-b con 0x7f84400a1330 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T03:59:49.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.193+0000 7f8446ec1640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f84400a1e20 0x7f8440135470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:59:49.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.193+0000 7f8446ec1640 10 monclient(hunting): picked mon.noname-a con 0x7f84400a1e20 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T03:59:49.227 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.193+0000 7f8446ec1640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f84401359b0 0x7f844011cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:59:49.227 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.193+0000 7f8446ec1640 10 monclient(hunting): picked mon.noname-c con 0x7f84401359b0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T03:59:49.227 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.193+0000 7f8446ec1640 10 monclient(hunting): start opening mon connection 2024-09-16T03:59:49.227 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.193+0000 7f8446ec1640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f8440140950 con 0x7f84400a1e20 2024-09-16T03:59:49.227 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.193+0000 7f8446ec1640 10 monclient(hunting): start opening mon connection 2024-09-16T03:59:49.227 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.193+0000 7f8446ec1640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f84401407d0 con 0x7f84400a1330 2024-09-16T03:59:49.227 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.193+0000 7f8446ec1640 10 monclient(hunting): start opening mon connection 2024-09-16T03:59:49.227 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.193+0000 7f8446ec1640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f8440140ad0 con 0x7f84401359b0 2024-09-16T03:59:49.227 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.193+0000 7f8446ec1640 10 monclient(hunting): _renew_subs 2024-09-16T03:59:49.227 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.193+0000 7f8446ec1640 10 monclient(hunting): authenticate will time out at 2036.166997s 2024-09-16T03:59:49.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.193+0000 7f844545e640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f84401359b0 0x7f844011cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:59:49.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.193+0000 7f844545e640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f84401359b0 0x7f844011cfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:44248/0 (socket says 172.21.15.42:44248) 2024-09-16T03:59:49.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.193+0000 7f844545e640 1 -- 172.21.15.42:0/3505672517 learned_addr learned my addr 172.21.15.42:0/3505672517 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T03:59:49.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.193+0000 7f844545e640 10 monclient(hunting): get_auth_request con 0x7f84401359b0 auth_method 0 2024-09-16T03:59:49.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.193+0000 7f844545e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:59:49.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.193+0000 7f844545e640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:59:49.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.193+0000 7f844545e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:59:49.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.193+0000 7f843ffff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f84400a1e20 0x7f8440135470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:59:49.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.193+0000 7f843ffff640 10 monclient(hunting): get_auth_request con 0x7f84400a1e20 auth_method 0 2024-09-16T03:59:49.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.193+0000 7f843ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:59:49.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.193+0000 7f843ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:59:49.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.193+0000 7f843ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:59:49.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.193+0000 7f844545e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:59:49.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.193+0000 7f844545e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:59:49.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.193+0000 7f8444c5d640 1 --2- 172.21.15.42:0/3505672517 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f84400a1330 0x7f84401cf110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:59:49.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.193+0000 7f844545e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:59:49.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.193+0000 7f8444c5d640 10 monclient(hunting): get_auth_request con 0x7f84400a1330 auth_method 0 2024-09-16T03:59:49.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.193+0000 7f8444c5d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:59:49.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.193+0000 7f8444c5d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:59:49.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.193+0000 7f8444c5d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:59:49.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.193+0000 7f843ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:59:49.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.193+0000 7f843ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:59:49.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.193+0000 7f843ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:59:49.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.193+0000 7f8444c5d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:59:49.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.193+0000 7f8444c5d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:59:49.230 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.193+0000 7f8444c5d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:59:49.230 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.193+0000 7f844545e640 10 monclient(hunting): handle_auth_done global_id 934139 payload 824 2024-09-16T03:59:49.230 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.193+0000 7f844545e640 1 -- 172.21.15.42:0/3505672517 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f84400a1330 msgr2=0x7f84401cf110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:59:49.230 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.193+0000 7f844545e640 1 --2- 172.21.15.42:0/3505672517 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f84400a1330 0x7f84401cf110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:59:49.230 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.193+0000 7f844545e640 1 -- 172.21.15.42:0/3505672517 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f84400a1e20 msgr2=0x7f8440135470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:59:49.230 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.193+0000 7f844545e640 1 --2- 172.21.15.42:0/3505672517 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f84400a1e20 0x7f8440135470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:59:49.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.193+0000 7f844545e640 10 monclient: _finish_hunting 0 2024-09-16T03:59:49.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.193+0000 7f844545e640 1 monclient: found mon.noname-c 2024-09-16T03:59:49.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.193+0000 7f844545e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T03:59:49.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.193+0000 7f844545e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T03:59:49.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.193+0000 7f844545e640 1 -- 172.21.15.42:0/3505672517 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8440120e30 con 0x7f84401359b0 2024-09-16T03:59:49.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.193+0000 7f844545e640 1 --2- 172.21.15.42:0/3505672517 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f84401359b0 0x7f844011cfc0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f843800c4d0 tx=0x7f843800c990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:59:49.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.193+0000 7f843d7fa640 10 client.?.objecter ms_handle_connect 0x7f84401359b0 2024-09-16T03:59:49.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.193+0000 7f843d7fa640 10 client.?.objecter resend_mon_ops 2024-09-16T03:59:49.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.197+0000 7f843d7fa640 1 -- 172.21.15.42:0/3505672517 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8438017070 con 0x7f84401359b0 2024-09-16T03:59:49.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.197+0000 7f843d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:59:49.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.197+0000 7f843d7fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T03:59:49.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.197+0000 7f843d7fa640 10 monclient: dump: 2024-09-16T03:59:49.233 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:59:49.233 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:59:49.233 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:49.233 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:49.233 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:59:49.234 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:59:49.234 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:59:49.234 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:59:49.234 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:59:49.234 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:59:49.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.197+0000 7f843d7fa640 10 monclient: _finish_auth 0 2024-09-16T03:59:49.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.197+0000 7f843d7fa640 10 monclient: _check_auth_tickets 2024-09-16T03:59:49.235 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.197+0000 7f843d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T03:59:49.235 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.197+0000 7f843d7fa640 1 -- 172.21.15.42:0/3505672517 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8438005d10 con 0x7f84401359b0 2024-09-16T03:59:49.235 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.197+0000 7f8446ec1640 5 monclient: authenticate success, global_id 934139 2024-09-16T03:59:49.235 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.197+0000 7f8446ec1640 10 monclient: _renew_subs 2024-09-16T03:59:49.235 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.197+0000 7f8446ec1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:59:49.235 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.197+0000 7f8446ec1640 1 -- 172.21.15.42:0/3505672517 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f844009cea0 con 0x7f84401359b0 2024-09-16T03:59:49.235 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.197+0000 7f843d7fa640 10 monclient: handle_config config(0 keys) 2024-09-16T03:59:49.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.197+0000 7f843d7fa640 1 -- 172.21.15.42:0/3505672517 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84380072b0 con 0x7f84401359b0 2024-09-16T03:59:49.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.197+0000 7f843d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:59:49.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.197+0000 7f843d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T03:59:49.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.197+0000 7f843d7fa640 10 monclient: dump: 2024-09-16T03:59:49.236 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:59:49.236 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:59:49.236 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:49.236 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:49.237 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:59:49.237 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:59:49.237 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:59:49.237 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:59:49.237 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:59:49.237 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:59:49.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.197+0000 7f843effd640 4 set_mon_vals no callback set 2024-09-16T03:59:49.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.197+0000 7f8446ec1640 10 client.934139.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T03:59:49.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.197+0000 7f8446ec1640 10 monclient: _renew_subs 2024-09-16T03:59:49.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.197+0000 7f8446ec1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T03:59:49.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.197+0000 7f8446ec1640 1 -- 172.21.15.42:0/3505672517 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f8440073a30 con 0x7f84401359b0 2024-09-16T03:59:49.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.197+0000 7f8446ec1640 1 librados: init done 2024-09-16T03:59:49.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.197+0000 7f8446ec1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T03:59:49.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.197+0000 7f8446ec1640 1 --2- 172.21.15.42:0/3505672517 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f84401208d0 0x7f844011d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:59:49.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.197+0000 7f8446ec1640 10 monclient(hunting): start opening mon connection 2024-09-16T03:59:49.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.197+0000 7f8446ec1640 1 -- 172.21.15.42:0/3505672517 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f8440120cb0 con 0x7f84401208d0 2024-09-16T03:59:49.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.197+0000 7f843d7fa640 1 -- 172.21.15.42:0/3505672517 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f843801a070 con 0x7f84401359b0 2024-09-16T03:59:49.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.197+0000 7f843d7fa640 1 --2- 172.21.15.42:0/3505672517 >> v2:172.21.15.154:6812/254845342 conn(0x7f8424037840 0x7f8424039d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:59:49.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.197+0000 7f843d7fa640 1 -- 172.21.15.42:0/3505672517 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(540..540 src has 451..540) ==== 6168+0+0 (secure 0 0 0) 0x7f8438048420 con 0x7f84401359b0 2024-09-16T03:59:49.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.197+0000 7f843d7fa640 10 client.934139.objecter ms_dispatch 0x7f8440000ba0 osd_map(540..540 src has 451..540) 2024-09-16T03:59:49.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.197+0000 7f843d7fa640 3 client.934139.objecter handle_osd_map got epochs [540,540] > 0 2024-09-16T03:59:49.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.197+0000 7f843d7fa640 3 client.934139.objecter handle_osd_map decoding full epoch 540 2024-09-16T03:59:49.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.197+0000 7f843d7fa640 20 client.934139.objecter dump_active .. 0 homeless 2024-09-16T03:59:49.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.197+0000 7f8444c5d640 1 --2- 172.21.15.42:0/3505672517 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f84401208d0 0x7f844011d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:59:49.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.197+0000 7f8444c5d640 10 monclient: get_auth_request con 0x7f84401208d0 auth_method 0 2024-09-16T03:59:49.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.197+0000 7f8444c5d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:59:49.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.197+0000 7f8444c5d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:59:49.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.197+0000 7f8444c5d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:59:49.242 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.197+0000 7f843ffff640 1 --2- 172.21.15.42:0/3505672517 >> v2:172.21.15.154:6812/254845342 conn(0x7f8424037840 0x7f8424039d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:59:49.242 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.197+0000 7f843ffff640 10 monclient: get_auth_request con 0x7f8424037840 auth_method 0 2024-09-16T03:59:49.242 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.197+0000 7f8444c5d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:59:49.242 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.197+0000 7f8444c5d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:59:49.242 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.197+0000 7f8444c5d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:59:49.242 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.197+0000 7f8444c5d640 10 monclient(hunting): handle_auth_done global_id 904378 payload 824 2024-09-16T03:59:49.242 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.197+0000 7f8444c5d640 1 -- 172.21.15.42:0/3505672517 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f84400a1710 con 0x7f84401208d0 2024-09-16T03:59:49.242 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.197+0000 7f843d7fa640 10 client.934139.objecter ms_handle_connect 0x7f8424037840 2024-09-16T03:59:49.242 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.197+0000 7f843ffff640 1 --2- 172.21.15.42:0/3505672517 >> v2:172.21.15.154:6812/254845342 conn(0x7f8424037840 0x7f8424039d00 secure :-1 s=READY pgs=3739 cs=0 l=1 rev1=1 crypto rx=0x7f84300af000 tx=0x7f84300a7070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:59:49.243 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.197+0000 7f843d7fa640 10 client.934139.objecter ms_handle_connect 0x7f84401208d0 2024-09-16T03:59:49.243 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.197+0000 7f843d7fa640 10 client.934139.objecter resend_mon_ops 2024-09-16T03:59:49.243 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.197+0000 7f8444c5d640 1 --2- 172.21.15.42:0/3505672517 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f84401208d0 0x7f844011d500 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f842800f540 tx=0x7f842800fa00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:59:49.243 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.197+0000 7f843d7fa640 1 -- 172.21.15.42:0/3505672517 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8428019070 con 0x7f84401208d0 2024-09-16T03:59:49.243 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.197+0000 7f843d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:59:49.243 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.197+0000 7f843d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:59:49.243 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.197+0000 7f843d7fa640 10 monclient: dump: 2024-09-16T03:59:49.244 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:59:49.244 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:59:49.244 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:49.244 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:49.244 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:59:49.244 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:59:49.244 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:59:49.244 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:59:49.244 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:59:49.244 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:59:49.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.197+0000 7f843d7fa640 1 -- 172.21.15.42:0/3505672517 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f8428005320 con 0x7f84401208d0 2024-09-16T03:59:49.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.197+0000 7f843d7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T03:59:49.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.197+0000 7f843d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T03:59:49.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.197+0000 7f843d7fa640 1 -- 172.21.15.42:0/3505672517 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f84401208d0 msgr2=0x7f844011d500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:59:49.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.197+0000 7f843d7fa640 1 --2- 172.21.15.42:0/3505672517 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f84401208d0 0x7f844011d500 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f842800f540 tx=0x7f842800fa00 comp rx=0 tx=0).stop 2024-09-16T03:59:49.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.197+0000 7f843d7fa640 1 -- 172.21.15.42:0/3505672517 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f84401208d0 msgr2=0x7f844011d500 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:59:49.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.197+0000 7f843d7fa640 1 --2- 172.21.15.42:0/3505672517 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f84401208d0 0x7f844011d500 secure :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f842800f540 tx=0x7f842800fa00 comp rx=0 tx=0).stop 2024-09-16T03:59:49.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.205+0000 7f8446ec1640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T03:59:49.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.205+0000 7f8446ec1640 1 --2- 172.21.15.42:0/3505672517 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f844011da40 0x7f844006dac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T03:59:49.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.205+0000 7f8446ec1640 10 monclient(hunting): start opening mon connection 2024-09-16T03:59:49.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.205+0000 7f8446ec1640 1 -- 172.21.15.42:0/3505672517 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f844011de50 con 0x7f844011da40 2024-09-16T03:59:49.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.205+0000 7f8444c5d640 1 --2- 172.21.15.42:0/3505672517 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f844011da40 0x7f844006dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T03:59:49.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.205+0000 7f8444c5d640 10 monclient: get_auth_request con 0x7f844011da40 auth_method 0 2024-09-16T03:59:49.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.205+0000 7f8444c5d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T03:59:49.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.205+0000 7f8444c5d640 10 monclient(hunting): _init_auth method 2 2024-09-16T03:59:49.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.205+0000 7f8444c5d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T03:59:49.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.205+0000 7f8444c5d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T03:59:49.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.205+0000 7f8444c5d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T03:59:49.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.205+0000 7f8444c5d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T03:59:49.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.205+0000 7f8444c5d640 10 monclient(hunting): handle_auth_done global_id 904381 payload 824 2024-09-16T03:59:49.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.205+0000 7f8444c5d640 1 -- 172.21.15.42:0/3505672517 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f8440120670 con 0x7f844011da40 2024-09-16T03:59:49.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.205+0000 7f843d7fa640 10 client.934139.objecter ms_handle_connect 0x7f844011da40 2024-09-16T03:59:49.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.205+0000 7f843d7fa640 10 client.934139.objecter resend_mon_ops 2024-09-16T03:59:49.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.205+0000 7f8444c5d640 1 --2- 172.21.15.42:0/3505672517 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f844011da40 0x7f844006dac0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f842800f540 tx=0x7f8428007f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T03:59:49.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.205+0000 7f843d7fa640 1 -- 172.21.15.42:0/3505672517 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8428019070 con 0x7f844011da40 2024-09-16T03:59:49.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.205+0000 7f843d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T03:59:49.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.205+0000 7f843d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T03:59:49.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.205+0000 7f843d7fa640 10 monclient: dump: 2024-09-16T03:59:49.247 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T03:59:49.247 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T03:59:49.247 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:49.247 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T03:59:49.247 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T03:59:49.247 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T03:59:49.247 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T03:59:49.247 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T03:59:49.248 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T03:59:49.248 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T03:59:49.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.205+0000 7f843d7fa640 1 -- 172.21.15.42:0/3505672517 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4880 (secure 0 0 0) 0x7f8428007510 con 0x7f844011da40 2024-09-16T03:59:49.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.205+0000 7f843d7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T03:59:49.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.205+0000 7f843d7fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T03:59:49.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.205+0000 7f843d7fa640 1 -- 172.21.15.42:0/3505672517 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f844011da40 msgr2=0x7f844006dac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:59:49.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.205+0000 7f843d7fa640 1 --2- 172.21.15.42:0/3505672517 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f844011da40 0x7f844006dac0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f842800f540 tx=0x7f8428007f90 comp rx=0 tx=0).stop 2024-09-16T03:59:49.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.205+0000 7f843d7fa640 1 -- 172.21.15.42:0/3505672517 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f844011da40 msgr2=0x7f844006dac0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T03:59:49.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.205+0000 7f843d7fa640 1 --2- 172.21.15.42:0/3505672517 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f844011da40 0x7f844006dac0 secure :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f842800f540 tx=0x7f8428007f90 comp rx=0 tx=0).stop 2024-09-16T03:59:49.249 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T03:59:49.249 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:59:49.249 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:59:49.249 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T03:59:49.249 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 160, 2024-09-16T03:59:49.249 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 117393, 2024-09-16T03:59:49.249 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T03:59:49.249 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T03:59:49.249 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T03:59:49.249 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T03:59:49.249 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:59:49.249 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 8, 2024-09-16T03:59:49.250 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:59:49.250 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T03:59:49.250 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T03:59:49.250 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:59:49.250 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:59:49.250 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:59:49.250 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:59:49.250 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:59:49.250 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:59:49.250 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:59:49.250 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:59:49.250 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:59:49.250 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:59:49.250 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:59:49.251 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:59:49.251 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T03:59:49.251 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T03:59:49.251 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:59:49.251 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:59:49.251 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:59:49.251 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:59:49.251 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:59:49.251 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:59:49.251 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:59:49.252 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:59:49.252 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:59:49.252 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:59:49.252 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:59:49.252 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:59:49.252 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:59:49.252 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T03:59:49.252 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T03:59:49.252 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T03:59:49.252 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T03:59:49.253 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T03:59:49.253 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T03:59:49.253 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:59:49.253 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T03:59:49.253 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T03:59:49.253 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T03:59:49.253 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T03:59:49.254 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T03:59:49.254 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T03:59:49.254 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T03:59:49.254 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T03:59:49.254 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T03:59:49.254 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T03:59:49.255 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T03:59:49.255 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T03:59:49.255 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T03:59:49.255 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T03:59:49.255 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T03:59:49.255 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T03:59:49.255 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T03:59:49.255 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T03:59:49.255 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T03:59:49.255 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T03:59:49.256 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T03:59:49.256 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:59:49.256 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T03:59:49.256 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:59:49.256 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T03:59:49.256 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:59:49.256 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T03:59:49.256 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T03:59:49.257 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:59:49.257 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:59:49.257 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:59:49.257 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:59:49.257 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T03:59:49.257 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:59:49.257 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:59:49.257 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:59:49.258 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:59:49.258 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T03:59:49.258 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:59:49.258 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:59:49.258 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:59:49.258 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:59:49.258 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T03:59:49.258 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T03:59:49.258 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:59:49.258 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:59:49.258 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:59:49.259 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:59:49.259 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:59:49.259 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T03:59:49.259 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T03:59:49.259 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:59:49.259 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:59:49.259 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:59:49.259 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:59:49.259 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T03:59:49.260 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:59:49.260 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:59:49.260 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:59:49.260 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:59:49.260 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T03:59:49.260 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:59:49.260 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:59:49.260 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:59:49.261 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:59:49.261 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T03:59:49.261 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T03:59:49.261 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:59:49.261 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:59:49.261 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:59:49.261 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:59:49.261 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:59:49.261 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T03:59:49.262 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T03:59:49.262 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T03:59:49.262 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T03:59:49.263 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:59:49.263 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T03:59:49.263 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T03:59:49.263 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:59:49.263 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:59:49.263 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:59:49.263 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T03:59:49.264 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T03:59:49.264 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T03:59:49.264 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:59:49.264 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:59:49.264 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:59:49.264 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T03:59:49.264 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T03:59:49.265 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T03:59:49.265 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T03:59:49.265 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T03:59:49.265 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:59:49.265 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:59:49.265 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:59:49.265 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T03:59:49.265 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T03:59:49.266 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:59:49.266 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:59:49.266 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:59:49.266 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:59:49.266 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:59:49.266 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:59:49.266 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T03:59:49.266 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:59:49.266 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:59:49.267 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:59:49.267 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:59:49.267 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:59:49.267 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:59:49.267 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T03:59:49.267 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:59:49.267 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:59:49.267 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:59:49.268 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 2 2024-09-16T03:59:49.268 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:59:49.268 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T03:59:49.268 INFO:teuthology.orchestra.run.smithi042.stdout: "mgr": [ 2024-09-16T03:59:49.268 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T03:59:49.268 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T03:59:49.268 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T03:59:49.268 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T03:59:49.268 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T03:59:49.268 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T03:59:49.269 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T03:59:49.269 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T03:59:49.269 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T03:59:49.271 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.209+0000 7f8446ec1640 10 librados: watch_flush enter 2024-09-16T03:59:49.271 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.209+0000 7f8446ec1640 10 librados: watch_flush exit 2024-09-16T03:59:49.271 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.209+0000 7f8446ec1640 20 client.934139.objecter shutdown clearing up homeless session... 2024-09-16T03:59:49.271 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.209+0000 7f8446ec1640 10 client.934139.objecter successfully canceled tick 2024-09-16T03:59:49.271 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.209+0000 7f8446ec1640 1 -- 172.21.15.42:0/3505672517 >> v2:172.21.15.154:6812/254845342 conn(0x7f8424037840 msgr2=0x7f8424039d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:59:49.271 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.209+0000 7f8446ec1640 1 --2- 172.21.15.42:0/3505672517 >> v2:172.21.15.154:6812/254845342 conn(0x7f8424037840 0x7f8424039d00 secure :-1 s=READY pgs=3739 cs=0 l=1 rev1=1 crypto rx=0x7f84300af000 tx=0x7f84300a7070 comp rx=0 tx=0).stop 2024-09-16T03:59:49.271 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.209+0000 7f8446ec1640 10 monclient: shutdown 2024-09-16T03:59:49.271 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.209+0000 7f8446ec1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T03:59:49.272 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.209+0000 7f8446ec1640 1 -- 172.21.15.42:0/3505672517 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f84401359b0 msgr2=0x7f844011cfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T03:59:49.272 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.209+0000 7f8446ec1640 1 --2- 172.21.15.42:0/3505672517 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f84401359b0 0x7f844011cfc0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f843800c4d0 tx=0x7f843800c990 comp rx=0 tx=0).stop 2024-09-16T03:59:49.272 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.209+0000 7f844545e640 1 -- 172.21.15.42:0/3505672517 reap_dead start 2024-09-16T03:59:49.272 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.209+0000 7f8446ec1640 1 -- 172.21.15.42:0/3505672517 shutdown_connections 2024-09-16T03:59:49.272 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.209+0000 7f8446ec1640 1 --2- 172.21.15.42:0/3505672517 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f84401359b0 0x7f844011cfc0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T03:59:49.272 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.209+0000 7f8446ec1640 1 -- 172.21.15.42:0/3505672517 >> 172.21.15.42:0/3505672517 conn(0x7f84400a2420 msgr2=0x7f844009e430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T03:59:49.272 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.209+0000 7f8446ec1640 1 -- 172.21.15.42:0/3505672517 shutdown_connections 2024-09-16T03:59:49.272 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.209+0000 7f8446ec1640 1 -- 172.21.15.42:0/3505672517 wait complete. 2024-09-16T03:59:49.272 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T03:59:49.209+0000 7f8446ec1640 1 librados: shutdown 2024-09-16T03:59:49.273 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T03:59:49.273 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-09-16T03:59:49.273 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['b'] 2024-09-16T03:59:49.273 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-09-16T03:59:49.273 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-16T03:59:49.274 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-16T03:59:49.274 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T03:59:49.285 INFO:tasks.ceph.mon.c:Stopped 2024-09-16T03:59:49.285 INFO:tasks.mon_thrash:Sending STOP to mon b 2024-09-16T03:59:49.286 INFO:tasks.ceph.mon.b:Sent signal 19 2024-09-16T03:59:49.286 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-16T04:00:04.289 INFO:tasks.mon_thrash:Sending CONT to mon b 2024-09-16T04:00:04.290 INFO:tasks.ceph.mon.b:Sent signal 18 2024-09-16T04:00:04.290 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T04:00:04.290 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:00:04.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:04.416+0000 7f0dfa6dc640 10 monclient: get_monmap_and_config 2024-09-16T04:00:04.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:04.416+0000 7f0dfa6dc640 10 monclient: build_initial_monmap 2024-09-16T04:00:04.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:04.416+0000 7f0dfa6dc640 10 monclient: monmap: 2024-09-16T04:00:04.423 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:00:04.423 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:00:04.424 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:00:04.421785+0000 2024-09-16T04:00:04.424 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:00:04.421785+0000 2024-09-16T04:00:04.424 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:00:04.424 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:00:04.424 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:00:04.424 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:00:04.424 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:00:04.424 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:00:04.424 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:00:04.424 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:00:04.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:04.416+0000 7f0dfa6dc640 1 Processor -- start 2024-09-16T04:00:04.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:04.416+0000 7f0dfa6dc640 1 -- start start 2024-09-16T04:00:04.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:04.416+0000 7f0dfa6dc640 10 monclient: init 2024-09-16T04:00:04.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:04.416+0000 7f0dfa6dc640 10 monclient: _reopen_session rank -1 2024-09-16T04:00:04.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:04.416+0000 7f0dfa6dc640 10 monclient: _add_conns ranks=[2,0,1,4,3] 2024-09-16T04:00:04.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:04.416+0000 7f0dfa6dc640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0df40a39b0 0x7f0df4133b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:00:04.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:04.416+0000 7f0dfa6dc640 10 monclient(hunting): picked mon.noname-b con 0x7f0df40a39b0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:00:04.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:04.416+0000 7f0dfa6dc640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0df40a3d90 addr v1:172.21.15.42:6789/0 2024-09-16T04:00:04.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:04.416+0000 7f0dfa6dc640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0df4137710 addr v1:172.21.15.154:6789/0 2024-09-16T04:00:04.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:04.416+0000 7f0dfa6dc640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0df409c140 con 0x7f0df40a3d90 2024-09-16T04:00:04.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:04.416+0000 7f0dfa6dc640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0df409c420 con 0x7f0df4137710 2024-09-16T04:00:04.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:04.416+0000 7f0dfa6dc640 10 monclient(hunting): start opening mon connection 2024-09-16T04:00:04.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:04.416+0000 7f0dfa6dc640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f0df406dd60 con 0x7f0df40a39b0 2024-09-16T04:00:04.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:04.416+0000 7f0dfa6dc640 10 monclient(hunting): _renew_subs 2024-09-16T04:00:04.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:04.416+0000 7f0dfa6dc640 10 monclient(hunting): authenticate will time out at 2051.391361s 2024-09-16T04:00:04.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:04.416+0000 7f0df3fff640 1 -- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0df40a39b0 msgr2=0x7f0df4133b80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T04:00:04.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:04.416+0000 7f0df3fff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0df40a39b0 0x7f0df4133b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:00:04.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:04.416+0000 7f0df37fe640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f0df40a3d90 0x7f0df4137000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:33498/0 (socket says 172.21.15.42:33498) 2024-09-16T04:00:04.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:04.416+0000 7f0df37fe640 1 -- 172.21.15.42:0/3566365887 learned_addr learned my addr 172.21.15.42:0/3566365887 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T04:00:04.625 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:04.620+0000 7f0df3fff640 1 -- 172.21.15.42:0/3566365887 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0df40a39b0 msgr2=0x7f0df4133b80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T04:00:04.625 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:04.620+0000 7f0df3fff640 1 --2- 172.21.15.42:0/3566365887 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0df40a39b0 0x7f0df4133b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T04:00:05.029 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:05.024+0000 7f0df3fff640 1 -- 172.21.15.42:0/3566365887 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0df40a39b0 msgr2=0x7f0df4133b80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T04:00:05.030 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:05.024+0000 7f0df3fff640 1 --2- 172.21.15.42:0/3566365887 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0df40a39b0 0x7f0df4133b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T04:00:05.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:05.828+0000 7f0df3fff640 1 -- 172.21.15.42:0/3566365887 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0df40a39b0 msgr2=0x7f0df4133b80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T04:00:05.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:05.828+0000 7f0df3fff640 1 --2- 172.21.15.42:0/3566365887 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0df40a39b0 0x7f0df4133b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T04:00:07.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:07.416+0000 7f0df1ffb640 10 monclient(hunting): tick 2024-09-16T04:00:07.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:07.416+0000 7f0df1ffb640 10 monclient(hunting): _check_auth_tickets 2024-09-16T04:00:07.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:07.416+0000 7f0df1ffb640 1 monclient(hunting): continuing hunt 2024-09-16T04:00:07.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:07.416+0000 7f0df1ffb640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T04:00:07.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:07.416+0000 7f0df1ffb640 1 -- 172.21.15.42:0/3566365887 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0df40a39b0 msgr2=0x7f0df4133b80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:00:07.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:07.416+0000 7f0df1ffb640 1 --2- 172.21.15.42:0/3566365887 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0df40a39b0 0x7f0df4133b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:00:07.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:07.416+0000 7f0df1ffb640 1 -- 172.21.15.42:0/3566365887 >> v1:172.21.15.154:6789/0 conn(0x7f0df4137710 legacy=0x7f0df413f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:00:07.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:07.416+0000 7f0df1ffb640 1 -- 172.21.15.42:0/3566365887 >> v1:172.21.15.42:6789/0 conn(0x7f0df40a3d90 legacy=0x7f0df4137000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:00:07.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:07.416+0000 7f0df1ffb640 10 monclient: _add_conns ranks=[3,4] 2024-09-16T04:00:07.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:07.416+0000 7f0df1ffb640 1 --2- 172.21.15.42:0/3566365887 >> v2:172.21.15.42:3300/0 conn(0x7f0de0003720 0x7f0de0005ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:00:07.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:07.416+0000 7f0df1ffb640 10 monclient(hunting): picked mon.noname-a con 0x7f0de0003720 addr v2:172.21.15.42:3300/0 2024-09-16T04:00:07.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:07.416+0000 7f0df1ffb640 1 --2- 172.21.15.42:0/3566365887 >> v2:172.21.15.154:3300/0 conn(0x7f0de0006290 0x7f0de0008750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:00:07.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:07.416+0000 7f0df1ffb640 10 monclient(hunting): picked mon.noname-c con 0x7f0de0006290 addr v2:172.21.15.154:3300/0 2024-09-16T04:00:07.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:07.416+0000 7f0df1ffb640 10 monclient(hunting): start opening mon connection 2024-09-16T04:00:07.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:07.416+0000 7f0df1ffb640 1 -- 172.21.15.42:0/3566365887 --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f0df406dd60 con 0x7f0de0003720 2024-09-16T04:00:07.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:07.416+0000 7f0df1ffb640 10 monclient(hunting): start opening mon connection 2024-09-16T04:00:07.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:07.416+0000 7f0df1ffb640 1 -- 172.21.15.42:0/3566365887 --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f0de0008e10 con 0x7f0de0006290 2024-09-16T04:00:07.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:07.416+0000 7f0df1ffb640 10 monclient(hunting): _renew_subs 2024-09-16T04:00:07.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:07.416+0000 7f0df3fff640 1 --2- 172.21.15.42:0/3566365887 >> v2:172.21.15.42:3300/0 conn(0x7f0de0003720 0x7f0de0005ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:00:07.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:07.420+0000 7f0df3fff640 10 monclient(hunting): get_auth_request con 0x7f0de0003720 auth_method 0 2024-09-16T04:00:07.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:07.420+0000 7f0df3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:00:07.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:07.420+0000 7f0df3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:00:07.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:07.420+0000 7f0df3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:00:07.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:07.420+0000 7f0df37fe640 1 --2- 172.21.15.42:0/3566365887 >> v2:172.21.15.154:3300/0 conn(0x7f0de0006290 0x7f0de0008750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:00:07.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:07.420+0000 7f0df37fe640 10 monclient(hunting): get_auth_request con 0x7f0de0006290 auth_method 0 2024-09-16T04:00:07.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:07.420+0000 7f0df37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:00:07.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:07.420+0000 7f0df37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:00:07.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:07.420+0000 7f0df37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:00:07.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:07.420+0000 7f0df3fff640 1 -- 172.21.15.42:0/3566365887 >> v2:172.21.15.42:3300/0 conn(0x7f0de0003720 msgr2=0x7f0de0005ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:00:07.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:07.420+0000 7f0df3fff640 1 -- 172.21.15.42:0/3566365887 >> v2:172.21.15.42:3300/0 conn(0x7f0de0003720 msgr2=0x7f0de0005ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:00:07.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:07.420+0000 7f0df3fff640 1 --2- 172.21.15.42:0/3566365887 >> v2:172.21.15.42:3300/0 conn(0x7f0de0003720 0x7f0de0005ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:00:07.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:07.420+0000 7f0df3fff640 1 --2- 172.21.15.42:0/3566365887 >> v2:172.21.15.42:3300/0 conn(0x7f0de0003720 0x7f0de0005ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:00:07.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:07.420+0000 7f0df37fe640 1 -- 172.21.15.42:0/3566365887 >> v2:172.21.15.154:3300/0 conn(0x7f0de0006290 msgr2=0x7f0de0008750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:00:07.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:07.420+0000 7f0df37fe640 1 -- 172.21.15.42:0/3566365887 >> v2:172.21.15.154:3300/0 conn(0x7f0de0006290 msgr2=0x7f0de0008750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:00:07.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:07.420+0000 7f0df37fe640 1 --2- 172.21.15.42:0/3566365887 >> v2:172.21.15.154:3300/0 conn(0x7f0de0006290 0x7f0de0008750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:00:07.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:07.420+0000 7f0df37fe640 1 --2- 172.21.15.42:0/3566365887 >> v2:172.21.15.154:3300/0 conn(0x7f0de0006290 0x7f0de0008750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:00:07.630 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:07.624+0000 7f0df3fff640 1 --2- 172.21.15.42:0/3566365887 >> v2:172.21.15.42:3300/0 conn(0x7f0de0003720 0x7f0de0005ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:00:07.630 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:07.624+0000 7f0df3fff640 10 monclient(hunting): get_auth_request con 0x7f0de0003720 auth_method 0 2024-09-16T04:00:07.630 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:07.624+0000 7f0df3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:00:07.630 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:07.624+0000 7f0df3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:00:07.631 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:07.624+0000 7f0df3fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:00:07.631 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:07.624+0000 7f0df3fff640 1 -- 172.21.15.42:0/3566365887 >> v2:172.21.15.42:3300/0 conn(0x7f0de0003720 msgr2=0x7f0de0005ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:00:07.631 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:07.624+0000 7f0df3fff640 1 -- 172.21.15.42:0/3566365887 >> v2:172.21.15.42:3300/0 conn(0x7f0de0003720 msgr2=0x7f0de0005ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:00:07.631 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:07.624+0000 7f0df3fff640 1 --2- 172.21.15.42:0/3566365887 >> v2:172.21.15.42:3300/0 conn(0x7f0de0003720 0x7f0de0005ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:00:07.631 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:07.624+0000 7f0df37fe640 1 --2- 172.21.15.42:0/3566365887 >> v2:172.21.15.154:3300/0 conn(0x7f0de0006290 0x7f0de0008750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:00:07.631 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:07.624+0000 7f0df3fff640 1 --2- 172.21.15.42:0/3566365887 >> v2:172.21.15.42:3300/0 conn(0x7f0de0003720 0x7f0de0005ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T04:00:07.631 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:07.624+0000 7f0df37fe640 10 monclient(hunting): get_auth_request con 0x7f0de0006290 auth_method 0 2024-09-16T04:00:07.631 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:07.624+0000 7f0df37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:00:07.631 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:07.624+0000 7f0df37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:00:07.631 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:07.624+0000 7f0df37fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:00:07.631 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:07.624+0000 7f0df37fe640 1 -- 172.21.15.42:0/3566365887 >> v2:172.21.15.154:3300/0 conn(0x7f0de0006290 msgr2=0x7f0de0008750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:00:07.631 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:07.624+0000 7f0df37fe640 1 -- 172.21.15.42:0/3566365887 >> v2:172.21.15.154:3300/0 conn(0x7f0de0006290 msgr2=0x7f0de0008750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:00:07.631 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:07.624+0000 7f0df37fe640 1 --2- 172.21.15.42:0/3566365887 >> v2:172.21.15.154:3300/0 conn(0x7f0de0006290 0x7f0de0008750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:00:07.632 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:07.624+0000 7f0df37fe640 1 --2- 172.21.15.42:0/3566365887 >> v2:172.21.15.154:3300/0 conn(0x7f0de0006290 0x7f0de0008750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T04:00:08.034 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:08.028+0000 7f0df3fff640 1 --2- 172.21.15.42:0/3566365887 >> v2:172.21.15.42:3300/0 conn(0x7f0de0003720 0x7f0de0005ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:00:08.034 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:08.028+0000 7f0df3fff640 10 monclient(hunting): get_auth_request con 0x7f0de0003720 auth_method 0 2024-09-16T04:00:08.034 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:08.028+0000 7f0df3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:00:08.034 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:08.028+0000 7f0df3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:00:08.034 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:08.028+0000 7f0df3fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:00:08.034 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:08.028+0000 7f0df37fe640 1 --2- 172.21.15.42:0/3566365887 >> v2:172.21.15.154:3300/0 conn(0x7f0de0006290 0x7f0de0008750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:00:08.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:08.028+0000 7f0df3fff640 1 -- 172.21.15.42:0/3566365887 >> v2:172.21.15.42:3300/0 conn(0x7f0de0003720 msgr2=0x7f0de0005ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:00:08.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:08.028+0000 7f0df3fff640 1 -- 172.21.15.42:0/3566365887 >> v2:172.21.15.42:3300/0 conn(0x7f0de0003720 msgr2=0x7f0de0005ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:00:08.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:08.028+0000 7f0df3fff640 1 --2- 172.21.15.42:0/3566365887 >> v2:172.21.15.42:3300/0 conn(0x7f0de0003720 0x7f0de0005ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:00:08.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:08.028+0000 7f0df37fe640 10 monclient(hunting): get_auth_request con 0x7f0de0006290 auth_method 0 2024-09-16T04:00:08.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:08.028+0000 7f0df37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:00:08.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:08.028+0000 7f0df37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:00:08.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:08.028+0000 7f0df37fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:00:08.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:08.028+0000 7f0df3fff640 1 --2- 172.21.15.42:0/3566365887 >> v2:172.21.15.42:3300/0 conn(0x7f0de0003720 0x7f0de0005ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T04:00:08.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:08.028+0000 7f0df37fe640 1 -- 172.21.15.42:0/3566365887 >> v2:172.21.15.154:3300/0 conn(0x7f0de0006290 msgr2=0x7f0de0008750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:00:08.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:08.028+0000 7f0df37fe640 1 -- 172.21.15.42:0/3566365887 >> v2:172.21.15.154:3300/0 conn(0x7f0de0006290 msgr2=0x7f0de0008750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:00:08.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:08.028+0000 7f0df37fe640 1 --2- 172.21.15.42:0/3566365887 >> v2:172.21.15.154:3300/0 conn(0x7f0de0006290 0x7f0de0008750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:00:08.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:08.028+0000 7f0df37fe640 1 --2- 172.21.15.42:0/3566365887 >> v2:172.21.15.154:3300/0 conn(0x7f0de0006290 0x7f0de0008750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T04:00:08.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:08.832+0000 7f0df3fff640 1 --2- 172.21.15.42:0/3566365887 >> v2:172.21.15.42:3300/0 conn(0x7f0de0003720 0x7f0de0005ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:00:08.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:08.832+0000 7f0df3fff640 10 monclient(hunting): get_auth_request con 0x7f0de0003720 auth_method 0 2024-09-16T04:00:08.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:08.832+0000 7f0df3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:00:08.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:08.832+0000 7f0df3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:00:08.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:08.832+0000 7f0df3fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:00:08.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:08.832+0000 7f0df37fe640 1 --2- 172.21.15.42:0/3566365887 >> v2:172.21.15.154:3300/0 conn(0x7f0de0006290 0x7f0de0008750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:00:08.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:08.832+0000 7f0df3fff640 1 -- 172.21.15.42:0/3566365887 >> v2:172.21.15.42:3300/0 conn(0x7f0de0003720 msgr2=0x7f0de0005ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:00:08.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:08.832+0000 7f0df3fff640 1 -- 172.21.15.42:0/3566365887 >> v2:172.21.15.42:3300/0 conn(0x7f0de0003720 msgr2=0x7f0de0005ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:00:08.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:08.832+0000 7f0df3fff640 1 --2- 172.21.15.42:0/3566365887 >> v2:172.21.15.42:3300/0 conn(0x7f0de0003720 0x7f0de0005ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:00:08.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:08.832+0000 7f0df37fe640 10 monclient(hunting): get_auth_request con 0x7f0de0006290 auth_method 0 2024-09-16T04:00:08.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:08.832+0000 7f0df37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:00:08.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:08.832+0000 7f0df37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:00:08.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:08.832+0000 7f0df37fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:00:08.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:08.832+0000 7f0df3fff640 1 --2- 172.21.15.42:0/3566365887 >> v2:172.21.15.42:3300/0 conn(0x7f0de0003720 0x7f0de0005ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T04:00:08.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:08.832+0000 7f0df37fe640 1 -- 172.21.15.42:0/3566365887 >> v2:172.21.15.154:3300/0 conn(0x7f0de0006290 msgr2=0x7f0de0008750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:00:08.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:08.832+0000 7f0df37fe640 1 -- 172.21.15.42:0/3566365887 >> v2:172.21.15.154:3300/0 conn(0x7f0de0006290 msgr2=0x7f0de0008750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:00:08.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:08.832+0000 7f0df37fe640 1 --2- 172.21.15.42:0/3566365887 >> v2:172.21.15.154:3300/0 conn(0x7f0de0006290 0x7f0de0008750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:00:08.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:08.832+0000 7f0df37fe640 1 --2- 172.21.15.42:0/3566365887 >> v2:172.21.15.154:3300/0 conn(0x7f0de0006290 0x7f0de0008750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T04:00:10.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.416+0000 7f0df1ffb640 10 monclient(hunting): tick 2024-09-16T04:00:10.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.416+0000 7f0df1ffb640 10 monclient(hunting): _check_auth_tickets 2024-09-16T04:00:10.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.416+0000 7f0df1ffb640 1 monclient(hunting): continuing hunt 2024-09-16T04:00:10.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.416+0000 7f0df1ffb640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T04:00:10.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.416+0000 7f0df1ffb640 1 -- 172.21.15.42:0/3566365887 >> v2:172.21.15.154:3300/0 conn(0x7f0de0006290 msgr2=0x7f0de0008750 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:00:10.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.416+0000 7f0df1ffb640 1 --2- 172.21.15.42:0/3566365887 >> v2:172.21.15.154:3300/0 conn(0x7f0de0006290 0x7f0de0008750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:00:10.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.416+0000 7f0df1ffb640 1 -- 172.21.15.42:0/3566365887 >> v2:172.21.15.42:3300/0 conn(0x7f0de0003720 msgr2=0x7f0de0005ba0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:00:10.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.416+0000 7f0df1ffb640 1 --2- 172.21.15.42:0/3566365887 >> v2:172.21.15.42:3300/0 conn(0x7f0de0003720 0x7f0de0005ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:00:10.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.416+0000 7f0df1ffb640 10 monclient: _add_conns ranks=[0,1,4,3,2] 2024-09-16T04:00:10.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.416+0000 7f0df8c79640 1 -- 172.21.15.42:0/3566365887 reap_dead start 2024-09-16T04:00:10.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.420+0000 7f0df1ffb640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0df40a3d90 addr v1:172.21.15.42:6789/0 2024-09-16T04:00:10.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.420+0000 7f0df1ffb640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0de0003b30 addr v1:172.21.15.154:6789/0 2024-09-16T04:00:10.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.420+0000 7f0df1ffb640 1 --2- 172.21.15.42:0/3566365887 >> v2:172.21.15.154:3300/0 conn(0x7f0de0007de0 0x7f0de0008260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:00:10.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.420+0000 7f0df1ffb640 10 monclient(hunting): picked mon.noname-c con 0x7f0de0007de0 addr v2:172.21.15.154:3300/0 2024-09-16T04:00:10.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.420+0000 7f0df1ffb640 1 -- 172.21.15.42:0/3566365887 --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0df40a5e20 con 0x7f0df40a3d90 2024-09-16T04:00:10.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.420+0000 7f0df1ffb640 1 -- 172.21.15.42:0/3566365887 --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0de00088f0 con 0x7f0de0003b30 2024-09-16T04:00:10.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.420+0000 7f0df1ffb640 10 monclient(hunting): start opening mon connection 2024-09-16T04:00:10.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.420+0000 7f0df1ffb640 1 -- 172.21.15.42:0/3566365887 --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f0df406dd60 con 0x7f0de0007de0 2024-09-16T04:00:10.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.420+0000 7f0df1ffb640 10 monclient(hunting): _renew_subs 2024-09-16T04:00:10.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.420+0000 7f0df8c79640 1 --2- 172.21.15.42:0/3566365887 >> v2:172.21.15.154:3300/0 conn(0x7f0de0007de0 0x7f0de0008260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:00:10.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.420+0000 7f0df8c79640 10 monclient(hunting): get_auth_request con 0x7f0de0007de0 auth_method 0 2024-09-16T04:00:10.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.420+0000 7f0df8c79640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:00:10.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.420+0000 7f0df8c79640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:00:10.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.420+0000 7f0df8c79640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:00:10.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.420+0000 7f0df2ffd640 1 -- 172.21.15.42:0/3566365887 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4139213425 0 0) 0x7f0df40a5e20 con 0x7f0df40a3d90 2024-09-16T04:00:10.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.420+0000 7f0df2ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:00:10.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.420+0000 7f0df2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:00:10.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.420+0000 7f0df2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:00:10.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.420+0000 7f0df2ffd640 10 monclient(hunting): my global_id is 825222 2024-09-16T04:00:10.432 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.420+0000 7f0df2ffd640 1 -- 172.21.15.42:0/3566365887 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0ddc003530 con 0x7f0df40a3d90 2024-09-16T04:00:10.432 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.420+0000 7f0df8c79640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:00:10.432 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.420+0000 7f0df8c79640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:00:10.432 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.420+0000 7f0df8c79640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:00:10.432 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.420+0000 7f0df2ffd640 1 -- 172.21.15.42:0/3566365887 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1650332200 0 0) 0x7f0ddc003530 con 0x7f0df40a3d90 2024-09-16T04:00:10.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.420+0000 7f0df2ffd640 1 -- 172.21.15.42:0/3566365887 >> v2:172.21.15.154:3300/0 conn(0x7f0de0007de0 msgr2=0x7f0de0008260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:00:10.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.420+0000 7f0df2ffd640 1 --2- 172.21.15.42:0/3566365887 >> v2:172.21.15.154:3300/0 conn(0x7f0de0007de0 0x7f0de0008260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:00:10.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.420+0000 7f0df2ffd640 1 -- 172.21.15.42:0/3566365887 >> v1:172.21.15.154:6789/0 conn(0x7f0de0003b30 legacy=0x7f0de00066c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:00:10.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.420+0000 7f0df2ffd640 10 monclient: _finish_hunting 0 2024-09-16T04:00:10.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.420+0000 7f0df2ffd640 1 monclient: found mon.noname-a-legacy 2024-09-16T04:00:10.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.420+0000 7f0df2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:00:10.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.420+0000 7f0df2ffd640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.42:6789/0 2024-09-16T04:00:10.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.420+0000 7f0df2ffd640 1 -- 172.21.15.42:0/3566365887 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0de0010960 con 0x7f0df40a3d90 2024-09-16T04:00:10.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.420+0000 7f0df2ffd640 10 monclient: _finish_auth 0 2024-09-16T04:00:10.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.420+0000 7f0df2ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:00:10.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.420+0000 7f0df2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:00:10.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.420+0000 7f0df2ffd640 1 -- 172.21.15.42:0/3566365887 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2177413734 0 0) 0x7f0de80098b0 con 0x7f0df40a3d90 2024-09-16T04:00:10.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.420+0000 7f0dfa6dc640 5 monclient: authenticate success, global_id 825222 2024-09-16T04:00:10.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.420+0000 7f0df8c79640 1 --2- 172.21.15.42:0/3566365887 >> v2:172.21.15.154:3300/0 conn(0x7f0de0007de0 0x7f0de0008260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:00:10.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.420+0000 7f0df2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:00:10.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.420+0000 7f0df2ffd640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T04:00:10.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.420+0000 7f0df2ffd640 10 monclient: dump: 2024-09-16T04:00:10.435 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:00:10.435 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:00:10.435 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:00:10.435 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:00:10.435 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:00:10.435 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:00:10.435 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:00:10.435 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:00:10.435 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:00:10.435 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:00:10.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.420+0000 7f0df2ffd640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] but i'm connected to v1:172.21.15.42:6789/0, reconnecting 2024-09-16T04:00:10.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.420+0000 7f0df2ffd640 10 monclient: _reopen_session rank -1 2024-09-16T04:00:10.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.420+0000 7f0df2ffd640 1 -- 172.21.15.42:0/3566365887 >> v1:172.21.15.42:6789/0 conn(0x7f0df40a3d90 legacy=0x7f0de000b210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:00:10.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.420+0000 7f0df2ffd640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:00:10.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.420+0000 7f0df2ffd640 1 --2- 172.21.15.42:0/3566365887 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0ddc0039c0 0x7f0ddc003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:00:10.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.420+0000 7f0df2ffd640 10 monclient(hunting): picked mon.b con 0x7f0ddc0039c0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:00:10.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.420+0000 7f0df2ffd640 1 --2- 172.21.15.42:0/3566365887 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0ddc004810 0x7f0ddc004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:00:10.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.420+0000 7f0df2ffd640 10 monclient(hunting): picked mon.c con 0x7f0ddc004810 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:00:10.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.420+0000 7f0df2ffd640 1 --2- 172.21.15.42:0/3566365887 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0ddc0051d0 0x7f0ddc00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:00:10.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.420+0000 7f0df2ffd640 10 monclient(hunting): picked mon.a con 0x7f0ddc0051d0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:00:10.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.420+0000 7f0df2ffd640 10 monclient(hunting): start opening mon connection 2024-09-16T04:00:10.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.420+0000 7f0df37fe640 1 -- 172.21.15.42:0/3566365887 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0ddc004810 msgr2=0x7f0ddc004c90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T04:00:10.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.420+0000 7f0df2ffd640 1 -- 172.21.15.42:0/3566365887 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f0df406dd60 con 0x7f0ddc0051d0 2024-09-16T04:00:10.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.420+0000 7f0df2ffd640 10 monclient(hunting): start opening mon connection 2024-09-16T04:00:10.437 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.420+0000 7f0df37fe640 1 --2- 172.21.15.42:0/3566365887 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0ddc004810 0x7f0ddc004c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:00:10.437 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.420+0000 7f0df2ffd640 1 -- 172.21.15.42:0/3566365887 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f0ddc00c0d0 con 0x7f0ddc0039c0 2024-09-16T04:00:10.437 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.420+0000 7f0df2ffd640 10 monclient(hunting): start opening mon connection 2024-09-16T04:00:10.437 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.420+0000 7f0df2ffd640 1 -- 172.21.15.42:0/3566365887 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f0ddc00c2d0 con 0x7f0ddc004810 2024-09-16T04:00:10.437 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.420+0000 7f0df2ffd640 10 monclient(hunting): _renew_subs 2024-09-16T04:00:10.437 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.420+0000 7f0df2ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:00:10.437 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.420+0000 7f0dfa6dc640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-16T04:00:10.437 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.420+0000 7f0df8c79640 1 --2- 172.21.15.42:0/3566365887 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0ddc0051d0 0x7f0ddc00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:00:10.437 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.420+0000 7f0df8c79640 10 monclient(hunting): get_auth_request con 0x7f0ddc0051d0 auth_method 0 2024-09-16T04:00:10.437 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.420+0000 7f0df8c79640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:00:10.437 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.420+0000 7f0df3fff640 1 --2- 172.21.15.42:0/3566365887 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0ddc0039c0 0x7f0ddc003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:00:10.437 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.420+0000 7f0df8c79640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:00:10.438 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.420+0000 7f0df8c79640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:00:10.438 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.420+0000 7f0df3fff640 10 monclient(hunting): get_auth_request con 0x7f0ddc0039c0 auth_method 0 2024-09-16T04:00:10.438 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.420+0000 7f0df3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:00:10.438 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.420+0000 7f0df3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:00:10.438 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.420+0000 7f0df3fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:00:10.440 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.420+0000 7f0df8c79640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:00:10.440 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.420+0000 7f0df8c79640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:00:10.440 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.420+0000 7f0df8c79640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:00:10.440 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.420+0000 7f0df3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:00:10.440 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.420+0000 7f0df3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:00:10.440 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.420+0000 7f0df3fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:00:10.440 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.420+0000 7f0df8c79640 10 monclient(hunting): handle_auth_done global_id 825222 payload 293 2024-09-16T04:00:10.440 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.420+0000 7f0df8c79640 1 -- 172.21.15.42:0/3566365887 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0ddc004810 msgr2=0x7f0ddc004c90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:00:10.440 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.420+0000 7f0df8c79640 1 --2- 172.21.15.42:0/3566365887 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0ddc004810 0x7f0ddc004c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:00:10.441 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.420+0000 7f0df8c79640 1 -- 172.21.15.42:0/3566365887 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0ddc0039c0 msgr2=0x7f0ddc003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:00:10.441 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.420+0000 7f0df8c79640 1 --2- 172.21.15.42:0/3566365887 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0ddc0039c0 0x7f0ddc003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:00:10.441 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.420+0000 7f0df8c79640 10 monclient: _finish_hunting 0 2024-09-16T04:00:10.441 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.420+0000 7f0df8c79640 1 monclient: found mon.a 2024-09-16T04:00:10.441 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.420+0000 7f0df8c79640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:00:10.441 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.420+0000 7f0df8c79640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:00:10.441 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.420+0000 7f0df8c79640 1 -- 172.21.15.42:0/3566365887 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ddc003530 con 0x7f0ddc0051d0 2024-09-16T04:00:10.442 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.420+0000 7f0df8c79640 10 monclient: _finish_auth 0 2024-09-16T04:00:10.442 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.420+0000 7f0df8c79640 10 monclient: _check_auth_tickets 2024-09-16T04:00:10.442 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.420+0000 7f0df8c79640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:00:10.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.428+0000 7f0df8c79640 1 -- 172.21.15.42:0/3566365887 reap_dead start 2024-09-16T04:00:10.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.428+0000 7f0df8c79640 1 --2- 172.21.15.42:0/3566365887 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0ddc0051d0 0x7f0ddc00b9d0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f0df409bfd0 tx=0x7f0dec01cbd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba0237e2e7401c9b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:00:10.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.428+0000 7f0df2ffd640 1 -- 172.21.15.42:0/3566365887 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dec001d30 con 0x7f0ddc0051d0 2024-09-16T04:00:10.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.428+0000 7f0df2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:00:10.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.428+0000 7f0df2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:00:10.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.428+0000 7f0df2ffd640 10 monclient: dump: 2024-09-16T04:00:10.460 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:00:10.460 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:00:10.460 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:00:10.460 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:00:10.461 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:00:10.461 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:00:10.461 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:00:10.461 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:00:10.461 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:00:10.462 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:00:10.462 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.428+0000 7f0dfa6dc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:00:10.462 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.428+0000 7f0df2ffd640 1 -- 172.21.15.42:0/3566365887 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0dec0a56b0 con 0x7f0ddc0051d0 2024-09-16T04:00:10.462 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.428+0000 7f0df2ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:00:10.462 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.428+0000 7f0dfa6dc640 10 monclient: get_monmap_and_config success 2024-09-16T04:00:10.462 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.428+0000 7f0dfa6dc640 4 set_mon_vals no callback set 2024-09-16T04:00:10.463 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.428+0000 7f0df2ffd640 1 -- 172.21.15.42:0/3566365887 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dec0a5a80 con 0x7f0ddc0051d0 2024-09-16T04:00:10.463 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.428+0000 7f0df2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:00:10.463 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.428+0000 7f0df2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:00:10.463 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.428+0000 7f0df2ffd640 10 monclient: dump: 2024-09-16T04:00:10.463 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:00:10.463 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:00:10.463 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:00:10.463 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:00:10.463 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:00:10.463 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:00:10.463 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:00:10.463 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:00:10.464 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:00:10.464 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:00:10.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.428+0000 7f0dfa6dc640 10 monclient: shutdown 2024-09-16T04:00:10.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.428+0000 7f0dfa6dc640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:00:10.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.428+0000 7f0dfa6dc640 1 -- 172.21.15.42:0/3566365887 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0ddc0051d0 msgr2=0x7f0ddc00b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:00:10.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.428+0000 7f0dfa6dc640 1 --2- 172.21.15.42:0/3566365887 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0ddc0051d0 0x7f0ddc00b9d0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f0df409bfd0 tx=0x7f0dec01cbd0 comp rx=0 tx=0).stop 2024-09-16T04:00:10.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.428+0000 7f0dfa6dc640 1 -- 172.21.15.42:0/3566365887 shutdown_connections 2024-09-16T04:00:10.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.428+0000 7f0dfa6dc640 1 --2- 172.21.15.42:0/3566365887 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0ddc0051d0 0x7f0ddc00b9d0 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:00:10.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.428+0000 7f0dfa6dc640 1 -- 172.21.15.42:0/3566365887 >> 172.21.15.42:0/3566365887 conn(0x7f0df40a9fb0 msgr2=0x7f0df40aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:00:10.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.428+0000 7f0dfa6dc640 1 -- 172.21.15.42:0/3566365887 shutdown_connections 2024-09-16T04:00:10.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.428+0000 7f0dfa6dc640 1 -- 172.21.15.42:0/3566365887 wait complete. 2024-09-16T04:00:10.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.432+0000 7f0dfa6dc640 10 monclient: build_initial_monmap 2024-09-16T04:00:10.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.432+0000 7f0dfa6dc640 10 monclient: monmap: 2024-09-16T04:00:10.465 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:00:10.465 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:00:10.465 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:00:10.465 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:00:10.465 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:00:10.465 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:00:10.465 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:00:10.465 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:00:10.465 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:00:10.465 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:00:10.466 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.432+0000 7f0dfa6dc640 1 librados: starting msgr at 2024-09-16T04:00:10.466 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.432+0000 7f0dfa6dc640 1 librados: starting objecter 2024-09-16T04:00:10.466 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.432+0000 7f0dfa6dc640 1 Processor -- start 2024-09-16T04:00:10.466 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.432+0000 7f0dfa6dc640 1 -- start start 2024-09-16T04:00:10.466 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.432+0000 7f0dfa6dc640 1 librados: setting wanted keys 2024-09-16T04:00:10.466 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.432+0000 7f0dfa6dc640 1 librados: calling monclient init 2024-09-16T04:00:10.466 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.432+0000 7f0dfa6dc640 10 monclient: init 2024-09-16T04:00:10.466 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.432+0000 7f0dfa6dc640 10 monclient: _reopen_session rank -1 2024-09-16T04:00:10.466 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.432+0000 7f0dfa6dc640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:00:10.466 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.432+0000 7f0dfa6dc640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0ddc0051d0 0x7f0df4120200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:00:10.467 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.432+0000 7f0dfa6dc640 10 monclient(hunting): picked mon.noname-a con 0x7f0ddc0051d0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:00:10.467 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.432+0000 7f0dfa6dc640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0df4120740 0x7f0df4120b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:00:10.467 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.432+0000 7f0dfa6dc640 10 monclient(hunting): picked mon.noname-b con 0x7f0df4120740 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:00:10.467 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.432+0000 7f0dfa6dc640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0df41ce9b0 0x7f0df41cee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:00:10.467 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.432+0000 7f0dfa6dc640 10 monclient(hunting): picked mon.noname-c con 0x7f0df41ce9b0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:00:10.467 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.432+0000 7f0dfa6dc640 10 monclient(hunting): start opening mon connection 2024-09-16T04:00:10.467 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.432+0000 7f0dfa6dc640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f0df413a580 con 0x7f0ddc0051d0 2024-09-16T04:00:10.467 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.432+0000 7f0dfa6dc640 10 monclient(hunting): start opening mon connection 2024-09-16T04:00:10.467 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.432+0000 7f0dfa6dc640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f0df406dee0 con 0x7f0df4120740 2024-09-16T04:00:10.467 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.432+0000 7f0dfa6dc640 10 monclient(hunting): start opening mon connection 2024-09-16T04:00:10.467 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.432+0000 7f0dfa6dc640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f0df413a700 con 0x7f0df41ce9b0 2024-09-16T04:00:10.467 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.432+0000 7f0dfa6dc640 10 monclient(hunting): _renew_subs 2024-09-16T04:00:10.467 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.432+0000 7f0dfa6dc640 10 monclient(hunting): authenticate will time out at 2057.406597s 2024-09-16T04:00:10.468 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.432+0000 7f0df8c79640 1 -- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0df41ce9b0 msgr2=0x7f0df41cee10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T04:00:10.468 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.432+0000 7f0df8c79640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0df41ce9b0 0x7f0df41cee10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:00:10.468 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.432+0000 7f0df3fff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0ddc0051d0 0x7f0df4120200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:00:10.468 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.432+0000 7f0df3fff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0ddc0051d0 0x7f0df4120200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:59692/0 (socket says 172.21.15.42:59692) 2024-09-16T04:00:10.468 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.432+0000 7f0df3fff640 1 -- 172.21.15.42:0/4200161412 learned_addr learned my addr 172.21.15.42:0/4200161412 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:00:10.468 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.432+0000 7f0df3fff640 10 monclient(hunting): get_auth_request con 0x7f0ddc0051d0 auth_method 0 2024-09-16T04:00:10.468 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.432+0000 7f0df3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:00:10.468 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.432+0000 7f0df3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:00:10.468 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.432+0000 7f0df3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:00:10.468 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.432+0000 7f0df3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:00:10.468 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.432+0000 7f0df3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:00:10.469 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.432+0000 7f0df3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:00:10.469 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.432+0000 7f0df37fe640 1 --2- 172.21.15.42:0/4200161412 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0df4120740 0x7f0df4120b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:00:10.469 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.432+0000 7f0df37fe640 10 monclient(hunting): get_auth_request con 0x7f0df4120740 auth_method 0 2024-09-16T04:00:10.469 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.432+0000 7f0df37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:00:10.469 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.432+0000 7f0df37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:00:10.469 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.432+0000 7f0df37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:00:10.469 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.432+0000 7f0df3fff640 10 monclient(hunting): handle_auth_done global_id 825225 payload 824 2024-09-16T04:00:10.469 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.432+0000 7f0df3fff640 1 -- 172.21.15.42:0/4200161412 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0df41ce9b0 msgr2=0x7f0df41cee10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:00:10.469 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.432+0000 7f0df3fff640 1 --2- 172.21.15.42:0/4200161412 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0df41ce9b0 0x7f0df41cee10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:00:10.469 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.432+0000 7f0df3fff640 1 -- 172.21.15.42:0/4200161412 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0df4120740 msgr2=0x7f0df4120b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:00:10.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.432+0000 7f0df3fff640 1 --2- 172.21.15.42:0/4200161412 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0df4120740 0x7f0df4120b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:00:10.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.432+0000 7f0df3fff640 10 monclient: _finish_hunting 0 2024-09-16T04:00:10.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.432+0000 7f0df3fff640 1 monclient: found mon.noname-a 2024-09-16T04:00:10.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.432+0000 7f0df3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:00:10.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.432+0000 7f0df3fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:00:10.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.432+0000 7f0df3fff640 1 -- 172.21.15.42:0/4200161412 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0df4122b80 con 0x7f0ddc0051d0 2024-09-16T04:00:10.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.432+0000 7f0df3fff640 1 --2- 172.21.15.42:0/4200161412 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0ddc0051d0 0x7f0df4120200 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f0de800bb70 tx=0x7f0de8005480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:00:10.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.432+0000 7f0df0ff9640 10 client.?.objecter ms_handle_connect 0x7f0ddc0051d0 2024-09-16T04:00:10.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.432+0000 7f0df0ff9640 10 client.?.objecter resend_mon_ops 2024-09-16T04:00:10.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.432+0000 7f0df0ff9640 1 -- 172.21.15.42:0/4200161412 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0de801d070 con 0x7f0ddc0051d0 2024-09-16T04:00:10.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.432+0000 7f0df0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:00:10.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.432+0000 7f0df0ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:00:10.471 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.432+0000 7f0df0ff9640 10 monclient: dump: 2024-09-16T04:00:10.471 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:00:10.471 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:00:10.471 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:00:10.471 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:00:10.471 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:00:10.471 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:00:10.471 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:00:10.471 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:00:10.471 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:00:10.471 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:00:10.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.432+0000 7f0df0ff9640 10 monclient: _finish_auth 0 2024-09-16T04:00:10.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.432+0000 7f0df0ff9640 10 monclient: _check_auth_tickets 2024-09-16T04:00:10.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.432+0000 7f0df0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:00:10.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.432+0000 7f0df0ff9640 1 -- 172.21.15.42:0/4200161412 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0de8019440 con 0x7f0ddc0051d0 2024-09-16T04:00:10.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.432+0000 7f0df0ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T04:00:10.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.432+0000 7f0df0ff9640 1 -- 172.21.15.42:0/4200161412 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0de8019830 con 0x7f0ddc0051d0 2024-09-16T04:00:10.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.432+0000 7f0dfa6dc640 5 monclient: authenticate success, global_id 825225 2024-09-16T04:00:10.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.432+0000 7f0dfa6dc640 10 monclient: _renew_subs 2024-09-16T04:00:10.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.432+0000 7f0df27fc640 4 set_mon_vals no callback set 2024-09-16T04:00:10.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.432+0000 7f0dfa6dc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:00:10.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.432+0000 7f0dfa6dc640 1 -- 172.21.15.42:0/4200161412 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0df4133250 con 0x7f0ddc0051d0 2024-09-16T04:00:10.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.432+0000 7f0df0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:00:10.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.432+0000 7f0df0ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:00:10.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.436+0000 7f0df0ff9640 10 monclient: dump: 2024-09-16T04:00:10.473 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:00:10.473 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:00:10.473 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:00:10.473 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:00:10.473 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:00:10.473 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:00:10.473 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:00:10.474 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:00:10.474 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:00:10.474 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:00:10.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.436+0000 7f0dfa6dc640 10 client.825225.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:00:10.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.436+0000 7f0dfa6dc640 10 monclient: _renew_subs 2024-09-16T04:00:10.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.436+0000 7f0dfa6dc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:00:10.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.436+0000 7f0dfa6dc640 1 -- 172.21.15.42:0/4200161412 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0df4130dc0 con 0x7f0ddc0051d0 2024-09-16T04:00:10.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.436+0000 7f0dfa6dc640 1 librados: init done 2024-09-16T04:00:10.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.436+0000 7f0dfa6dc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:00:10.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.436+0000 7f0dfa6dc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:00:10.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.436+0000 7f0dfa6dc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:00:10.475 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.436+0000 7f0dfa6dc640 1 -- 172.21.15.42:0/4200161412 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0df413a9a0 con 0x7f0ddc0051d0 2024-09-16T04:00:10.475 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.436+0000 7f0df0ff9640 1 -- 172.21.15.42:0/4200161412 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f0de8020070 con 0x7f0ddc0051d0 2024-09-16T04:00:10.475 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.436+0000 7f0df0ff9640 1 --2- 172.21.15.42:0/4200161412 >> v2:172.21.15.154:6812/254845342 conn(0x7f0de0038ed0 0x7f0de003b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:00:10.475 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.436+0000 7f0df0ff9640 1 -- 172.21.15.42:0/4200161412 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(541..541 src has 451..541) ==== 6280+0+0 (secure 0 0 0) 0x7f0de804a4a0 con 0x7f0ddc0051d0 2024-09-16T04:00:10.475 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.436+0000 7f0df0ff9640 10 client.825225.objecter ms_dispatch 0x7f0df4000ba0 osd_map(541..541 src has 451..541) 2024-09-16T04:00:10.475 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.436+0000 7f0df0ff9640 3 client.825225.objecter handle_osd_map got epochs [541,541] > 0 2024-09-16T04:00:10.475 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.436+0000 7f0df0ff9640 3 client.825225.objecter handle_osd_map decoding full epoch 541 2024-09-16T04:00:10.475 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.436+0000 7f0df0ff9640 20 client.825225.objecter dump_active .. 0 homeless 2024-09-16T04:00:10.476 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.436+0000 7f0df37fe640 1 --2- 172.21.15.42:0/4200161412 >> v2:172.21.15.154:6812/254845342 conn(0x7f0de0038ed0 0x7f0de003b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:00:10.476 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.436+0000 7f0df37fe640 10 monclient: get_auth_request con 0x7f0de0038ed0 auth_method 0 2024-09-16T04:00:10.476 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.436+0000 7f0df37fe640 1 --2- 172.21.15.42:0/4200161412 >> v2:172.21.15.154:6812/254845342 conn(0x7f0de0038ed0 0x7f0de003b390 secure :-1 s=READY pgs=3740 cs=0 l=1 rev1=1 crypto rx=0x7f0df4121660 tx=0x7f0de4009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:00:10.476 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.436+0000 7f0df0ff9640 10 client.825225.objecter ms_handle_connect 0x7f0de0038ed0 2024-09-16T04:00:10.476 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.440+0000 7f0df0ff9640 1 -- 172.21.15.42:0/4200161412 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f0de8020520 con 0x7f0ddc0051d0 2024-09-16T04:00:10.476 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.440+0000 7f0df0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:00:10.476 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.440+0000 7f0df0ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:00:10.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.676+0000 7f0dfa6dc640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:00:10.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.676+0000 7f0dfa6dc640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:00:10.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.676+0000 7f0dfa6dc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:00:10.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.676+0000 7f0dfa6dc640 1 -- 172.21.15.42:0/4200161412 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f0df40a5750 con 0x7f0ddc0051d0 2024-09-16T04:00:10.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.676+0000 7f0df0ff9640 1 -- 172.21.15.42:0/4200161412 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1555 (secure 0 0 0) 0x7f0de8020780 con 0x7f0ddc0051d0 2024-09-16T04:00:10.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.676+0000 7f0df0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:00:10.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.676+0000 7f0df0ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:00:10.685 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":162,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:00:10.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.676+0000 7f0dfa6dc640 10 librados: watch_flush enter 2024-09-16T04:00:10.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.680+0000 7f0dfa6dc640 10 librados: watch_flush exit 2024-09-16T04:00:10.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.680+0000 7f0dfa6dc640 20 client.825225.objecter shutdown clearing up homeless session... 2024-09-16T04:00:10.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.680+0000 7f0dfa6dc640 10 client.825225.objecter successfully canceled tick 2024-09-16T04:00:10.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.680+0000 7f0dfa6dc640 1 -- 172.21.15.42:0/4200161412 >> v2:172.21.15.154:6812/254845342 conn(0x7f0de0038ed0 msgr2=0x7f0de003b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:00:10.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.680+0000 7f0dfa6dc640 1 --2- 172.21.15.42:0/4200161412 >> v2:172.21.15.154:6812/254845342 conn(0x7f0de0038ed0 0x7f0de003b390 secure :-1 s=READY pgs=3740 cs=0 l=1 rev1=1 crypto rx=0x7f0df4121660 tx=0x7f0de4009040 comp rx=0 tx=0).stop 2024-09-16T04:00:10.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.680+0000 7f0dfa6dc640 10 monclient: shutdown 2024-09-16T04:00:10.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.680+0000 7f0dfa6dc640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:00:10.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.680+0000 7f0dfa6dc640 1 -- 172.21.15.42:0/4200161412 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0ddc0051d0 msgr2=0x7f0df4120200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:00:10.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.680+0000 7f0dfa6dc640 1 --2- 172.21.15.42:0/4200161412 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0ddc0051d0 0x7f0df4120200 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f0de800bb70 tx=0x7f0de8005480 comp rx=0 tx=0).stop 2024-09-16T04:00:10.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.680+0000 7f0dfa6dc640 1 -- 172.21.15.42:0/4200161412 shutdown_connections 2024-09-16T04:00:10.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.680+0000 7f0dfa6dc640 1 --2- 172.21.15.42:0/4200161412 >> v2:172.21.15.154:6812/254845342 conn(0x7f0de0038ed0 0x7f0de003b390 unknown :-1 s=CLOSED pgs=3740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:00:10.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.680+0000 7f0dfa6dc640 1 --2- 172.21.15.42:0/4200161412 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0df41ce9b0 0x7f0df41cee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:00:10.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.680+0000 7f0dfa6dc640 1 --2- 172.21.15.42:0/4200161412 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0df4120740 0x7f0df4120b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:00:10.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.680+0000 7f0dfa6dc640 1 --2- 172.21.15.42:0/4200161412 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0ddc0051d0 0x7f0df4120200 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:00:10.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.680+0000 7f0dfa6dc640 1 -- 172.21.15.42:0/4200161412 >> 172.21.15.42:0/4200161412 conn(0x7f0df40a9fb0 msgr2=0x7f0df4131980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:00:10.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.680+0000 7f0dfa6dc640 1 -- 172.21.15.42:0/4200161412 shutdown_connections 2024-09-16T04:00:10.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.680+0000 7f0dfa6dc640 1 -- 172.21.15.42:0/4200161412 wait complete. 2024-09-16T04:00:10.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.680+0000 7f0dfa6dc640 1 librados: shutdown 2024-09-16T04:00:10.703 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T04:00:10.703 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-16T04:00:10.704 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T04:00:10.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.832+0000 7f097a4db640 10 monclient: get_monmap_and_config 2024-09-16T04:00:10.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.832+0000 7f097a4db640 10 monclient: build_initial_monmap 2024-09-16T04:00:10.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.832+0000 7f097a4db640 10 monclient: monmap: 2024-09-16T04:00:10.841 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:00:10.841 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:00:10.841 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:00:10.838935+0000 2024-09-16T04:00:10.841 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:00:10.838935+0000 2024-09-16T04:00:10.841 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:00:10.841 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:00:10.842 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:00:10.842 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:00:10.842 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:00:10.842 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:00:10.842 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:00:10.842 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:00:10.845 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.836+0000 7f097a4db640 1 Processor -- start 2024-09-16T04:00:10.845 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.836+0000 7f097a4db640 1 -- start start 2024-09-16T04:00:10.845 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.836+0000 7f097a4db640 10 monclient: init 2024-09-16T04:00:10.845 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.836+0000 7f097a4db640 10 monclient: _reopen_session rank -1 2024-09-16T04:00:10.846 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.836+0000 7f097a4db640 10 monclient: _add_conns ranks=[2,4,1,0,3] 2024-09-16T04:00:10.846 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.836+0000 7f097a4db640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0974130b20 0x7f0974130f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:00:10.846 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.836+0000 7f097a4db640 10 monclient(hunting): picked mon.noname-b con 0x7f0974130b20 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:00:10.846 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.836+0000 7f097a4db640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f09740a2250 0x7f0974072d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:00:10.846 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.836+0000 7f097a4db640 10 monclient(hunting): picked mon.noname-c con 0x7f09740a2250 addr v2:172.21.15.154:3300/0 2024-09-16T04:00:10.846 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.836+0000 7f097a4db640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0974073250 addr v1:172.21.15.154:6789/0 2024-09-16T04:00:10.846 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.836+0000 7f097a4db640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f09740a1040 con 0x7f0974073250 2024-09-16T04:00:10.846 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.836+0000 7f097a4db640 10 monclient(hunting): start opening mon connection 2024-09-16T04:00:10.846 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.836+0000 7f097a4db640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f09740a12b0 con 0x7f09740a2250 2024-09-16T04:00:10.846 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.836+0000 7f097a4db640 10 monclient(hunting): start opening mon connection 2024-09-16T04:00:10.846 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.836+0000 7f097a4db640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f09740a1590 con 0x7f0974130b20 2024-09-16T04:00:10.846 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.836+0000 7f097a4db640 10 monclient(hunting): _renew_subs 2024-09-16T04:00:10.847 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.836+0000 7f097a4db640 10 monclient(hunting): authenticate will time out at 2057.808887s 2024-09-16T04:00:10.847 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.836+0000 7f0973fff640 1 -- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0974130b20 msgr2=0x7f0974130f00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T04:00:10.847 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.836+0000 7f0973fff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0974130b20 0x7f0974130f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:00:10.847 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.836+0000 7f096b7fe640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f09740a2250 0x7f0974072d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:00:10.847 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.836+0000 7f096b7fe640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f09740a2250 0x7f0974072d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.42:42416/0 (socket says 172.21.15.42:42416) 2024-09-16T04:00:10.847 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.836+0000 7f096b7fe640 1 -- 172.21.15.42:0/1250393151 learned_addr learned my addr 172.21.15.42:0/1250393151 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:00:10.847 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.836+0000 7f096b7fe640 10 monclient(hunting): get_auth_request con 0x7f09740a2250 auth_method 0 2024-09-16T04:00:10.847 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.836+0000 7f096b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:00:10.847 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.836+0000 7f096b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:00:10.847 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.836+0000 7f096b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:00:10.847 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.836+0000 7f096b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:00:10.847 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.836+0000 7f096b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:00:10.847 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.836+0000 7f096b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:00:10.847 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.836+0000 7f096b7fe640 10 monclient(hunting): handle_auth_done global_id 904393 payload 274 2024-09-16T04:00:10.848 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.836+0000 7f096b7fe640 1 -- 172.21.15.42:0/1250393151 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0974130b20 msgr2=0x7f0974130f00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:00:10.848 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.836+0000 7f096b7fe640 1 --2- 172.21.15.42:0/1250393151 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0974130b20 0x7f0974130f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:00:10.848 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.836+0000 7f096b7fe640 1 -- 172.21.15.42:0/1250393151 >> v1:172.21.15.154:6789/0 conn(0x7f0974073250 legacy=0x7f097413e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:00:10.848 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.836+0000 7f096b7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:00:10.848 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.836+0000 7f096b7fe640 1 monclient: found mon.noname-c 2024-09-16T04:00:10.848 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.836+0000 7f096b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:00:10.848 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.836+0000 7f096b7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.154:3300/0 2024-09-16T04:00:10.848 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.836+0000 7f096b7fe640 1 -- 172.21.15.42:0/1250393151 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09740a1870 con 0x7f09740a2250 2024-09-16T04:00:10.848 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.840+0000 7f096b7fe640 1 --2- 172.21.15.42:0/1250393151 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f09740a2250 0x7f0974072d10 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f095c009870 tx=0x7f095c01c440 comp rx=0 tx=0).ready entity=mon.1 client_cookie=585d7261dc8c9791 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:00:10.860 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.840+0000 7f09737fe640 1 -- 172.21.15.42:0/1250393151 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f095c0ab660 con 0x7f09740a2250 2024-09-16T04:00:10.860 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.840+0000 7f09737fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:00:10.860 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.840+0000 7f09737fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:00:10.861 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.840+0000 7f09737fe640 10 monclient: dump: 2024-09-16T04:00:10.861 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:00:10.861 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:00:10.861 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:00:10.861 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:00:10.861 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:00:10.861 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:00:10.861 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:00:10.861 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:00:10.861 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:00:10.862 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:00:10.862 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.840+0000 7f09737fe640 10 monclient: _finish_auth 0 2024-09-16T04:00:10.862 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.840+0000 7f09737fe640 10 monclient: _check_auth_tickets 2024-09-16T04:00:10.862 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.840+0000 7f09737fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:00:10.862 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.840+0000 7f09737fe640 1 -- 172.21.15.42:0/1250393151 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f095c0b0070 con 0x7f09740a2250 2024-09-16T04:00:10.862 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.840+0000 7f09737fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:00:10.862 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.840+0000 7f09737fe640 1 -- 172.21.15.42:0/1250393151 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f095c0aba20 con 0x7f09740a2250 2024-09-16T04:00:10.862 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.840+0000 7f09737fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:00:10.862 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.840+0000 7f09737fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:00:10.862 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.840+0000 7f09737fe640 10 monclient: dump: 2024-09-16T04:00:10.863 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:00:10.863 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:00:10.863 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:00:10.863 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:00:10.863 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:00:10.863 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:00:10.863 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:00:10.863 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:00:10.863 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:00:10.863 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:00:10.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.840+0000 7f097a4db640 5 monclient: authenticate success, global_id 904393 2024-09-16T04:00:10.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.840+0000 7f097a4db640 10 monclient: get_monmap_and_config success 2024-09-16T04:00:10.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.840+0000 7f097a4db640 4 set_mon_vals no callback set 2024-09-16T04:00:10.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.840+0000 7f097a4db640 10 monclient: shutdown 2024-09-16T04:00:10.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.840+0000 7f097a4db640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:00:10.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.840+0000 7f097a4db640 1 -- 172.21.15.42:0/1250393151 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f09740a2250 msgr2=0x7f0974072d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:00:10.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.840+0000 7f097a4db640 1 --2- 172.21.15.42:0/1250393151 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f09740a2250 0x7f0974072d10 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f095c009870 tx=0x7f095c01c440 comp rx=0 tx=0).stop 2024-09-16T04:00:10.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.840+0000 7f097a4db640 1 -- 172.21.15.42:0/1250393151 shutdown_connections 2024-09-16T04:00:10.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.840+0000 7f097a4db640 1 --2- 172.21.15.42:0/1250393151 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f09740a2250 0x7f0974072d10 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:00:10.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.840+0000 7f097a4db640 1 --2- 172.21.15.42:0/1250393151 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0974130b20 0x7f0974130f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:00:10.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.840+0000 7f097a4db640 1 -- 172.21.15.42:0/1250393151 >> 172.21.15.42:0/1250393151 conn(0x7f09740a6210 msgr2=0x7f09740a6620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:00:10.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.840+0000 7f097a4db640 1 -- 172.21.15.42:0/1250393151 shutdown_connections 2024-09-16T04:00:10.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.840+0000 7f097a4db640 1 -- 172.21.15.42:0/1250393151 wait complete. 2024-09-16T04:00:10.865 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.840+0000 7f097a4db640 10 monclient: build_initial_monmap 2024-09-16T04:00:10.865 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.840+0000 7f097a4db640 10 monclient: monmap: 2024-09-16T04:00:10.865 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:00:10.865 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:00:10.865 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:00:10.865 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:00:10.865 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:00:10.865 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:00:10.865 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:00:10.866 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:00:10.866 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:00:10.866 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:00:10.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.840+0000 7f097a4db640 1 librados: starting msgr at 2024-09-16T04:00:10.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.840+0000 7f097a4db640 1 librados: starting objecter 2024-09-16T04:00:10.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.840+0000 7f097a4db640 1 Processor -- start 2024-09-16T04:00:10.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.840+0000 7f097a4db640 1 -- start start 2024-09-16T04:00:10.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.840+0000 7f097a4db640 1 librados: setting wanted keys 2024-09-16T04:00:10.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.840+0000 7f097a4db640 1 librados: calling monclient init 2024-09-16T04:00:10.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.840+0000 7f097a4db640 10 monclient: init 2024-09-16T04:00:10.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.840+0000 7f097a4db640 10 monclient: _reopen_session rank -1 2024-09-16T04:00:10.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.840+0000 7f097a4db640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:00:10.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.840+0000 7f097a4db640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0974073250 0x7f097413e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:00:10.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.840+0000 7f097a4db640 10 monclient(hunting): picked mon.noname-a con 0x7f0974073250 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:00:10.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.840+0000 7f097a4db640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f09740a2250 0x7f097413e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:00:10.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.840+0000 7f097a4db640 10 monclient(hunting): picked mon.noname-c con 0x7f09740a2250 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:00:10.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.840+0000 7f097a4db640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0974130b20 0x7f09741cf210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:00:10.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.840+0000 7f097a4db640 10 monclient(hunting): picked mon.noname-b con 0x7f0974130b20 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:00:10.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.840+0000 7f097a4db640 10 monclient(hunting): start opening mon connection 2024-09-16T04:00:10.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.840+0000 7f097a4db640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f0974075ac0 con 0x7f0974073250 2024-09-16T04:00:10.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.840+0000 7f097a4db640 10 monclient(hunting): start opening mon connection 2024-09-16T04:00:10.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.840+0000 7f097a4db640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f0974075940 con 0x7f0974130b20 2024-09-16T04:00:10.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.840+0000 7f097a4db640 10 monclient(hunting): start opening mon connection 2024-09-16T04:00:10.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.840+0000 7f097a4db640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f0974131440 con 0x7f09740a2250 2024-09-16T04:00:10.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.840+0000 7f097a4db640 10 monclient(hunting): _renew_subs 2024-09-16T04:00:10.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.840+0000 7f097a4db640 10 monclient(hunting): authenticate will time out at 2057.814660s 2024-09-16T04:00:10.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.840+0000 7f096b7fe640 1 -- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f09740a2250 msgr2=0x7f097413e6e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T04:00:10.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.840+0000 7f096b7fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f09740a2250 0x7f097413e6e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:00:10.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.840+0000 7f0973fff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0974073250 0x7f097413e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:00:10.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.840+0000 7f0973fff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0974073250 0x7f097413e1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:59698/0 (socket says 172.21.15.42:59698) 2024-09-16T04:00:10.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.840+0000 7f0973fff640 1 -- 172.21.15.42:0/377451630 learned_addr learned my addr 172.21.15.42:0/377451630 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:00:10.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.840+0000 7f0973fff640 10 monclient(hunting): get_auth_request con 0x7f0974073250 auth_method 0 2024-09-16T04:00:10.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.840+0000 7f0973fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:00:10.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.840+0000 7f0973fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:00:10.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.840+0000 7f0973fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:00:10.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.840+0000 7f0973fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:00:10.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.840+0000 7f0973fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:00:10.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.840+0000 7f0973fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:00:10.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.840+0000 7f0978a78640 1 --2- 172.21.15.42:0/377451630 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0974130b20 0x7f09741cf210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:00:10.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.840+0000 7f0978a78640 10 monclient(hunting): get_auth_request con 0x7f0974130b20 auth_method 0 2024-09-16T04:00:10.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.840+0000 7f0978a78640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:00:10.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.840+0000 7f0978a78640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:00:10.871 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.840+0000 7f0978a78640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:00:10.871 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.840+0000 7f0973fff640 10 monclient(hunting): handle_auth_done global_id 825228 payload 824 2024-09-16T04:00:10.871 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.840+0000 7f0973fff640 1 -- 172.21.15.42:0/377451630 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f09740a2250 msgr2=0x7f097413e6e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:00:10.871 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.840+0000 7f0973fff640 1 --2- 172.21.15.42:0/377451630 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f09740a2250 0x7f097413e6e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:00:10.871 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.840+0000 7f0973fff640 1 -- 172.21.15.42:0/377451630 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0974130b20 msgr2=0x7f09741cf210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:00:10.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.840+0000 7f0973fff640 1 --2- 172.21.15.42:0/377451630 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0974130b20 0x7f09741cf210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:00:10.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.840+0000 7f0973fff640 10 monclient: _finish_hunting 0 2024-09-16T04:00:10.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.840+0000 7f0973fff640 1 monclient: found mon.noname-a 2024-09-16T04:00:10.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.840+0000 7f0973fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:00:10.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.840+0000 7f0973fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:00:10.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.840+0000 7f0973fff640 1 -- 172.21.15.42:0/377451630 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09740a01b0 con 0x7f0974073250 2024-09-16T04:00:10.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.840+0000 7f0978a78640 1 --2- 172.21.15.42:0/377451630 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0974130b20 0x7f09741cf210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T04:00:10.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.840+0000 7f0973fff640 1 --2- 172.21.15.42:0/377451630 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0974073250 0x7f097413e1a0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f096400b530 tx=0x7f096400b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:00:10.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.840+0000 7f09717fa640 10 client.?.objecter ms_handle_connect 0x7f0974073250 2024-09-16T04:00:10.873 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.840+0000 7f09717fa640 10 client.?.objecter resend_mon_ops 2024-09-16T04:00:10.873 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.840+0000 7f09717fa640 1 -- 172.21.15.42:0/377451630 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0964013020 con 0x7f0974073250 2024-09-16T04:00:10.873 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.840+0000 7f09717fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:00:10.873 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.840+0000 7f09717fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:00:10.873 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.840+0000 7f09717fa640 10 monclient: dump: 2024-09-16T04:00:10.873 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:00:10.873 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:00:10.873 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:00:10.873 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:00:10.873 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:00:10.873 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:00:10.873 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:00:10.873 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:00:10.874 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:00:10.874 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:00:10.874 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.840+0000 7f09717fa640 10 monclient: _finish_auth 0 2024-09-16T04:00:10.874 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.840+0000 7f09717fa640 10 monclient: _check_auth_tickets 2024-09-16T04:00:10.874 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.840+0000 7f09717fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:00:10.874 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.840+0000 7f097a4db640 5 monclient: authenticate success, global_id 825228 2024-09-16T04:00:10.874 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.840+0000 7f09717fa640 1 -- 172.21.15.42:0/377451630 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0964003df0 con 0x7f0974073250 2024-09-16T04:00:10.874 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.840+0000 7f097a4db640 10 monclient: _renew_subs 2024-09-16T04:00:10.874 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.840+0000 7f097a4db640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:00:10.874 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.840+0000 7f097a4db640 1 -- 172.21.15.42:0/377451630 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0974139c40 con 0x7f0974073250 2024-09-16T04:00:10.874 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.840+0000 7f09717fa640 10 monclient: handle_config config(0 keys) 2024-09-16T04:00:10.874 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.840+0000 7f09717fa640 1 -- 172.21.15.42:0/377451630 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09640052b0 con 0x7f0974073250 2024-09-16T04:00:10.874 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.840+0000 7f09717fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:00:10.875 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.840+0000 7f09717fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:00:10.875 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.840+0000 7f09717fa640 10 monclient: dump: 2024-09-16T04:00:10.875 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:00:10.875 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:00:10.875 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:00:10.875 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:00:10.875 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:00:10.875 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:00:10.875 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:00:10.875 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:00:10.875 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:00:10.875 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:00:10.876 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.840+0000 7f0972ffd640 4 set_mon_vals no callback set 2024-09-16T04:00:10.876 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.840+0000 7f097a4db640 10 client.825228.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:00:10.876 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.840+0000 7f097a4db640 10 monclient: _renew_subs 2024-09-16T04:00:10.876 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.840+0000 7f097a4db640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:00:10.876 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.840+0000 7f097a4db640 1 -- 172.21.15.42:0/377451630 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09740a43f0 con 0x7f0974073250 2024-09-16T04:00:10.876 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.840+0000 7f097a4db640 1 librados: init done 2024-09-16T04:00:10.876 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.844+0000 7f097a4db640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:00:10.876 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.844+0000 7f097a4db640 1 --2- 172.21.15.42:0/377451630 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f09741372c0 0x7f09741cf750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:00:10.876 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.844+0000 7f097a4db640 10 monclient(hunting): start opening mon connection 2024-09-16T04:00:10.876 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.844+0000 7f097a4db640 1 -- 172.21.15.42:0/377451630 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f0974072d10 con 0x7f09741372c0 2024-09-16T04:00:10.876 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.844+0000 7f096b7fe640 1 --2- 172.21.15.42:0/377451630 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f09741372c0 0x7f09741cf750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:00:10.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.844+0000 7f096b7fe640 10 monclient: get_auth_request con 0x7f09741372c0 auth_method 0 2024-09-16T04:00:10.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.844+0000 7f096b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:00:10.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.844+0000 7f096b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:00:10.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.844+0000 7f096b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:00:10.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.844+0000 7f09717fa640 1 -- 172.21.15.42:0/377451630 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f0964016020 con 0x7f0974073250 2024-09-16T04:00:10.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.844+0000 7f096b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:00:10.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.844+0000 7f096b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:00:10.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.844+0000 7f096b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:00:10.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.844+0000 7f096b7fe640 10 monclient(hunting): handle_auth_done global_id 825231 payload 824 2024-09-16T04:00:10.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.844+0000 7f09717fa640 1 --2- 172.21.15.42:0/377451630 >> v2:172.21.15.154:6812/254845342 conn(0x7f09540377c0 0x7f0954039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:00:10.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.844+0000 7f096b7fe640 1 -- 172.21.15.42:0/377451630 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f09740008e0 con 0x7f09741372c0 2024-09-16T04:00:10.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.844+0000 7f09717fa640 1 -- 172.21.15.42:0/377451630 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(542..542 src has 451..542) ==== 6224+0+0 (secure 0 0 0) 0x7f0964048d10 con 0x7f0974073250 2024-09-16T04:00:10.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.844+0000 7f09717fa640 10 client.825228.objecter ms_dispatch 0x7f0974000ba0 osd_map(542..542 src has 451..542) 2024-09-16T04:00:10.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.844+0000 7f09717fa640 3 client.825228.objecter handle_osd_map got epochs [542,542] > 0 2024-09-16T04:00:10.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.844+0000 7f09717fa640 3 client.825228.objecter handle_osd_map decoding full epoch 542 2024-09-16T04:00:10.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.844+0000 7f09717fa640 20 client.825228.objecter dump_active .. 0 homeless 2024-09-16T04:00:10.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.844+0000 7f09717fa640 10 client.825228.objecter ms_handle_connect 0x7f09741372c0 2024-09-16T04:00:10.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.844+0000 7f09717fa640 10 client.825228.objecter resend_mon_ops 2024-09-16T04:00:10.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.844+0000 7f096b7fe640 1 --2- 172.21.15.42:0/377451630 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f09741372c0 0x7f09741cf750 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f097409cdd0 tx=0x7f095c01ac80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:00:10.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.844+0000 7f0978a78640 1 --2- 172.21.15.42:0/377451630 >> v2:172.21.15.154:6812/254845342 conn(0x7f09540377c0 0x7f0954039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:00:10.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.844+0000 7f0978a78640 10 monclient: get_auth_request con 0x7f09540377c0 auth_method 0 2024-09-16T04:00:10.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.844+0000 7f09717fa640 1 -- 172.21.15.42:0/377451630 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f095c004220 con 0x7f09741372c0 2024-09-16T04:00:10.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.844+0000 7f09717fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:00:10.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.844+0000 7f09717fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:00:10.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.844+0000 7f09717fa640 10 monclient: dump: 2024-09-16T04:00:10.879 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:00:10.879 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:00:10.879 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:00:10.879 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:00:10.879 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:00:10.879 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:00:10.879 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:00:10.879 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:00:10.879 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:00:10.880 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:00:10.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.844+0000 7f09717fa640 1 -- 172.21.15.42:0/377451630 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f095c0b0040 con 0x7f09741372c0 2024-09-16T04:00:10.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.844+0000 7f09717fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:00:10.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.844+0000 7f09717fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:00:10.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.844+0000 7f09717fa640 1 -- 172.21.15.42:0/377451630 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f09741372c0 msgr2=0x7f09741cf750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:00:10.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.844+0000 7f09717fa640 1 --2- 172.21.15.42:0/377451630 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f09741372c0 0x7f09741cf750 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f097409cdd0 tx=0x7f095c01ac80 comp rx=0 tx=0).stop 2024-09-16T04:00:10.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.844+0000 7f09717fa640 1 -- 172.21.15.42:0/377451630 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f09741372c0 msgr2=0x7f09741cf750 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:00:10.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.844+0000 7f09717fa640 1 --2- 172.21.15.42:0/377451630 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f09741372c0 0x7f09741cf750 secure :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f097409cdd0 tx=0x7f095c01ac80 comp rx=0 tx=0).stop 2024-09-16T04:00:10.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.844+0000 7f09717fa640 10 client.825228.objecter ms_handle_connect 0x7f09540377c0 2024-09-16T04:00:10.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.844+0000 7f0978a78640 1 --2- 172.21.15.42:0/377451630 >> v2:172.21.15.154:6812/254845342 conn(0x7f09540377c0 0x7f0954039c80 secure :-1 s=READY pgs=3741 cs=0 l=1 rev1=1 crypto rx=0x7f096c0012f0 tx=0x7f096c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:00:10.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.848+0000 7f097a4db640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:00:10.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.848+0000 7f097a4db640 1 --2- 172.21.15.42:0/377451630 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f09741cfc90 0x7f097412e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:00:10.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.848+0000 7f097a4db640 10 monclient(hunting): start opening mon connection 2024-09-16T04:00:10.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.848+0000 7f097a4db640 1 -- 172.21.15.42:0/377451630 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f09741d0070 con 0x7f09741cfc90 2024-09-16T04:00:10.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.848+0000 7f096b7fe640 1 --2- 172.21.15.42:0/377451630 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f09741cfc90 0x7f097412e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:00:10.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.848+0000 7f096b7fe640 10 monclient: get_auth_request con 0x7f09741cfc90 auth_method 0 2024-09-16T04:00:10.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.848+0000 7f096b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:00:10.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.848+0000 7f096b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:00:10.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.848+0000 7f096b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:00:10.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.852+0000 7f096b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:00:10.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.852+0000 7f096b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:00:10.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.852+0000 7f096b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:00:10.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.852+0000 7f096b7fe640 10 monclient(hunting): handle_auth_done global_id 825234 payload 824 2024-09-16T04:00:10.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.852+0000 7f096b7fe640 1 -- 172.21.15.42:0/377451630 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f0974139480 con 0x7f09741cfc90 2024-09-16T04:00:10.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.852+0000 7f096b7fe640 1 --2- 172.21.15.42:0/377451630 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f09741cfc90 0x7f097412e9c0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f095c01aaf0 tx=0x7f095c0191e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:00:10.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.852+0000 7f09717fa640 10 client.825228.objecter ms_handle_connect 0x7f09741cfc90 2024-09-16T04:00:10.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.852+0000 7f09717fa640 10 client.825228.objecter resend_mon_ops 2024-09-16T04:00:10.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.852+0000 7f09717fa640 1 -- 172.21.15.42:0/377451630 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f095c003c40 con 0x7f09741cfc90 2024-09-16T04:00:10.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.852+0000 7f09717fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:00:10.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.852+0000 7f09717fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:00:10.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.852+0000 7f09717fa640 10 monclient: dump: 2024-09-16T04:00:10.882 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:00:10.882 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:00:10.882 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:00:10.883 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:00:10.883 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:00:10.883 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:00:10.883 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:00:10.883 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:00:10.883 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:00:10.883 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:00:10.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.852+0000 7f09717fa640 1 -- 172.21.15.42:0/377451630 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4704 (secure 0 0 0) 0x7f095c003c40 con 0x7f09741cfc90 2024-09-16T04:00:10.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.852+0000 7f09717fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:00:10.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.852+0000 7f09717fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:00:10.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.852+0000 7f09717fa640 1 -- 172.21.15.42:0/377451630 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f09741cfc90 msgr2=0x7f097412e9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:00:10.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.852+0000 7f09717fa640 1 --2- 172.21.15.42:0/377451630 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f09741cfc90 0x7f097412e9c0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f095c01aaf0 tx=0x7f095c0191e0 comp rx=0 tx=0).stop 2024-09-16T04:00:10.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.852+0000 7f09717fa640 1 -- 172.21.15.42:0/377451630 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f09741cfc90 msgr2=0x7f097412e9c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:00:10.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.852+0000 7f09717fa640 1 --2- 172.21.15.42:0/377451630 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f09741cfc90 0x7f097412e9c0 secure :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f095c01aaf0 tx=0x7f095c0191e0 comp rx=0 tx=0).stop 2024-09-16T04:00:10.885 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T04:00:10.886 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:00:10.886 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:00:10.886 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "leader", 2024-09-16T04:00:10.886 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 162, 2024-09-16T04:00:10.886 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 500707, 2024-09-16T04:00:10.886 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T04:00:10.886 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T04:00:10.886 INFO:teuthology.orchestra.run.smithi042.stdout: 1 2024-09-16T04:00:10.886 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:00:10.887 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 1, 2024-09-16T04:00:10.887 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:00:10.887 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T04:00:10.887 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T04:00:10.887 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:00:10.887 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:00:10.887 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:00:10.887 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:00:10.887 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:00:10.887 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:00:10.887 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:00:10.887 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:00:10.887 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:00:10.887 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:00:10.888 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:00:10.888 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:00:10.888 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:00:10.888 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T04:00:10.888 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:00:10.888 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:00:10.888 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:00:10.888 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:00:10.888 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:00:10.888 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:00:10.888 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:00:10.889 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:00:10.889 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:00:10.889 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:00:10.889 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:00:10.889 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:00:10.889 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:00:10.889 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T04:00:10.889 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T04:00:10.889 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T04:00:10.889 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T04:00:10.889 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T04:00:10.890 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T04:00:10.890 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:00:10.890 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:00:10.890 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T04:00:10.890 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T04:00:10.890 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T04:00:10.891 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T04:00:10.891 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T04:00:10.891 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T04:00:10.891 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T04:00:10.891 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:00:10.891 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T04:00:10.891 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:00:10.891 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:00:10.891 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:00:10.891 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:00:10.892 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:00:10.892 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:00:10.892 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:00:10.892 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:00:10.892 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:00:10.892 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:00:10.892 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:00:10.892 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:00:10.892 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T04:00:10.892 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:00:10.893 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T04:00:10.893 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:00:10.893 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:00:10.893 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:00:10.893 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:00:10.893 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:00:10.893 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:00:10.893 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:00:10.893 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T04:00:10.894 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:00:10.894 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:00:10.894 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:00:10.894 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:00:10.894 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T04:00:10.894 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:00:10.894 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:00:10.894 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:00:10.894 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:00:10.894 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T04:00:10.894 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T04:00:10.894 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:00:10.895 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:00:10.895 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:00:10.895 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:00:10.895 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:00:10.895 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:00:10.895 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:00:10.895 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:00:10.896 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:00:10.896 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:00:10.896 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:00:10.896 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T04:00:10.896 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:00:10.896 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:00:10.896 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:00:10.897 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:00:10.897 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T04:00:10.897 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:00:10.897 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:00:10.897 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:00:10.897 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:00:10.898 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T04:00:10.898 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T04:00:10.898 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:00:10.898 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:00:10.898 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:00:10.898 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:00:10.898 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:00:10.898 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:00:10.898 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:00:10.898 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:00:10.898 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:00:10.899 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:00:10.899 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:00:10.899 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T04:00:10.899 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:00:10.899 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:00:10.899 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:00:10.899 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:00:10.899 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T04:00:10.899 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:00:10.900 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:00:10.900 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:00:10.900 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:00:10.900 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T04:00:10.900 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T04:00:10.900 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:00:10.901 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:00:10.901 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:00:10.901 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:00:10.901 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:00:10.901 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:00:10.901 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T04:00:10.901 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T04:00:10.901 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:00:10.901 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:00:10.901 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:00:10.901 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:00:10.901 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:00:10.901 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:00:10.902 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T04:00:10.902 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:00:10.902 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:00:10.902 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:00:10.902 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 5 2024-09-16T04:00:10.902 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:00:10.902 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:00:10.902 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T04:00:10.903 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:00:10.903 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:00:10.903 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:00:10.903 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 7 2024-09-16T04:00:10.903 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:00:10.903 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:00:10.903 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:00:10.903 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T04:00:10.903 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T04:00:10.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.852+0000 7f097a4db640 10 librados: watch_flush enter 2024-09-16T04:00:10.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.852+0000 7f097a4db640 10 librados: watch_flush exit 2024-09-16T04:00:10.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.852+0000 7f097a4db640 20 client.825228.objecter shutdown clearing up homeless session... 2024-09-16T04:00:10.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.852+0000 7f097a4db640 10 client.825228.objecter successfully canceled tick 2024-09-16T04:00:10.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.852+0000 7f097a4db640 1 -- 172.21.15.42:0/377451630 >> v2:172.21.15.154:6812/254845342 conn(0x7f09540377c0 msgr2=0x7f0954039c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:00:10.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.852+0000 7f097a4db640 1 --2- 172.21.15.42:0/377451630 >> v2:172.21.15.154:6812/254845342 conn(0x7f09540377c0 0x7f0954039c80 secure :-1 s=READY pgs=3741 cs=0 l=1 rev1=1 crypto rx=0x7f096c0012f0 tx=0x7f096c009040 comp rx=0 tx=0).stop 2024-09-16T04:00:10.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.852+0000 7f097a4db640 10 monclient: shutdown 2024-09-16T04:00:10.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.852+0000 7f097a4db640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:00:10.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.852+0000 7f097a4db640 1 -- 172.21.15.42:0/377451630 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0974073250 msgr2=0x7f097413e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:00:10.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.852+0000 7f097a4db640 1 --2- 172.21.15.42:0/377451630 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0974073250 0x7f097413e1a0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f096400b530 tx=0x7f096400b9f0 comp rx=0 tx=0).stop 2024-09-16T04:00:10.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.852+0000 7f0978a78640 1 -- 172.21.15.42:0/377451630 reap_dead start 2024-09-16T04:00:10.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.852+0000 7f097a4db640 1 -- 172.21.15.42:0/377451630 shutdown_connections 2024-09-16T04:00:10.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.852+0000 7f097a4db640 1 --2- 172.21.15.42:0/377451630 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0974073250 0x7f097413e1a0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:00:10.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.852+0000 7f097a4db640 1 -- 172.21.15.42:0/377451630 >> 172.21.15.42:0/377451630 conn(0x7f09740a6210 msgr2=0x7f0974135b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:00:10.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.852+0000 7f097a4db640 1 -- 172.21.15.42:0/377451630 shutdown_connections 2024-09-16T04:00:10.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.852+0000 7f097a4db640 1 -- 172.21.15.42:0/377451630 wait complete. 2024-09-16T04:00:10.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:10.852+0000 7f097a4db640 1 librados: shutdown 2024-09-16T04:00:10.907 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T04:00:10.908 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-16T04:00:10.908 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T04:00:11.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.020+0000 7f922e5d7640 10 monclient: get_monmap_and_config 2024-09-16T04:00:11.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.020+0000 7f922e5d7640 10 monclient: build_initial_monmap 2024-09-16T04:00:11.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.020+0000 7f922e5d7640 10 monclient: monmap: 2024-09-16T04:00:11.026 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:00:11.026 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:00:11.027 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:00:11.024917+0000 2024-09-16T04:00:11.027 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:00:11.024917+0000 2024-09-16T04:00:11.027 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:00:11.027 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:00:11.027 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:00:11.027 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:00:11.027 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:00:11.027 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:00:11.027 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:00:11.027 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:00:11.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.020+0000 7f922e5d7640 1 Processor -- start 2024-09-16T04:00:11.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.020+0000 7f922e5d7640 1 -- start start 2024-09-16T04:00:11.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.020+0000 7f922e5d7640 10 monclient: init 2024-09-16T04:00:11.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.020+0000 7f922e5d7640 10 monclient: _reopen_session rank -1 2024-09-16T04:00:11.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.020+0000 7f922e5d7640 10 monclient: _add_conns ranks=[4,0,2,3,1] 2024-09-16T04:00:11.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.020+0000 7f922e5d7640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f922806db90 0x7f92281340c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:00:11.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.020+0000 7f922e5d7640 10 monclient(hunting): picked mon.noname-c con 0x7f922806db90 addr v2:172.21.15.154:3300/0 2024-09-16T04:00:11.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.020+0000 7f922e5d7640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f92280a37f0 addr v1:172.21.15.42:6789/0 2024-09-16T04:00:11.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.020+0000 7f922e5d7640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f92280a3c00 0x7f9228137520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:00:11.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.020+0000 7f922e5d7640 10 monclient(hunting): picked mon.noname-b con 0x7f92280a3c00 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:00:11.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.020+0000 7f922e5d7640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f922806ece0 con 0x7f92280a37f0 2024-09-16T04:00:11.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.020+0000 7f922e5d7640 10 monclient(hunting): start opening mon connection 2024-09-16T04:00:11.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.020+0000 7f922e5d7640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f922806df70 con 0x7f922806db90 2024-09-16T04:00:11.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.020+0000 7f922e5d7640 10 monclient(hunting): start opening mon connection 2024-09-16T04:00:11.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.020+0000 7f922e5d7640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f922809bf80 con 0x7f92280a3c00 2024-09-16T04:00:11.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.020+0000 7f922e5d7640 10 monclient(hunting): _renew_subs 2024-09-16T04:00:11.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.020+0000 7f922e5d7640 10 monclient(hunting): authenticate will time out at 2057.994600s 2024-09-16T04:00:11.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.020+0000 7f922cb74640 1 -- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f92280a3c00 msgr2=0x7f9228137520 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T04:00:11.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.020+0000 7f922cb74640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f92280a3c00 0x7f9228137520 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:00:11.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.020+0000 7f92277fe640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f92280a37f0 0x7f9228136e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:33518/0 (socket says 172.21.15.42:33518) 2024-09-16T04:00:11.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.020+0000 7f92277fe640 1 -- 172.21.15.42:0/2801319709 learned_addr learned my addr 172.21.15.42:0/2801319709 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T04:00:11.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.020+0000 7f9227fff640 1 --2- 172.21.15.42:0/2801319709 >> v2:172.21.15.154:3300/0 conn(0x7f922806db90 0x7f92281340c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:00:11.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.020+0000 7f9227fff640 10 monclient(hunting): get_auth_request con 0x7f922806db90 auth_method 0 2024-09-16T04:00:11.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.020+0000 7f9227fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:00:11.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.020+0000 7f9227fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:00:11.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.020+0000 7f9227fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:00:11.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.020+0000 7f9226ffd640 1 -- 172.21.15.42:0/2801319709 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 108808612 0 0) 0x7f922806ece0 con 0x7f92280a37f0 2024-09-16T04:00:11.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.020+0000 7f9226ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:00:11.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.020+0000 7f9226ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:00:11.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.020+0000 7f9226ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:00:11.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.020+0000 7f9226ffd640 10 monclient(hunting): my global_id is 825237 2024-09-16T04:00:11.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.020+0000 7f9226ffd640 1 -- 172.21.15.42:0/2801319709 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9214003530 con 0x7f92280a37f0 2024-09-16T04:00:11.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.020+0000 7f9227fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:00:11.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.020+0000 7f9227fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:00:11.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.020+0000 7f9227fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:00:11.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.020+0000 7f9226ffd640 1 -- 172.21.15.42:0/2801319709 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2882734011 0 0) 0x7f9214003530 con 0x7f92280a37f0 2024-09-16T04:00:11.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.020+0000 7f9226ffd640 1 -- 172.21.15.42:0/2801319709 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f92280a3c00 msgr2=0x7f9228137520 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:00:11.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.020+0000 7f9226ffd640 1 --2- 172.21.15.42:0/2801319709 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f92280a3c00 0x7f9228137520 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:00:11.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.020+0000 7f9226ffd640 1 -- 172.21.15.42:0/2801319709 >> v2:172.21.15.154:3300/0 conn(0x7f922806db90 msgr2=0x7f92281340c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:00:11.034 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.020+0000 7f9226ffd640 1 --2- 172.21.15.42:0/2801319709 >> v2:172.21.15.154:3300/0 conn(0x7f922806db90 0x7f92281340c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:00:11.034 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.020+0000 7f9226ffd640 10 monclient: _finish_hunting 0 2024-09-16T04:00:11.034 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.020+0000 7f9226ffd640 1 monclient: found mon.noname-a-legacy 2024-09-16T04:00:11.034 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.020+0000 7f9226ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:00:11.034 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.020+0000 7f9226ffd640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.42:6789/0 2024-09-16T04:00:11.034 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.020+0000 7f9226ffd640 1 -- 172.21.15.42:0/2801319709 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9228134780 con 0x7f92280a37f0 2024-09-16T04:00:11.034 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.020+0000 7f9226ffd640 10 monclient: _finish_auth 0 2024-09-16T04:00:11.034 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.020+0000 7f9226ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:00:11.034 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.020+0000 7f9226ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:00:11.034 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.020+0000 7f9226ffd640 1 -- 172.21.15.42:0/2801319709 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2177413734 0 0) 0x7f9218001a00 con 0x7f92280a37f0 2024-09-16T04:00:11.034 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.020+0000 7f922e5d7640 5 monclient: authenticate success, global_id 825237 2024-09-16T04:00:11.034 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.020+0000 7f922e5d7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:00:11.034 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.020+0000 7f9226ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:00:11.034 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.020+0000 7f9226ffd640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T04:00:11.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.020+0000 7f9226ffd640 10 monclient: dump: 2024-09-16T04:00:11.035 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:00:11.035 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:00:11.035 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:00:11.035 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:00:11.035 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:00:11.035 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:00:11.035 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:00:11.035 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:00:11.035 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:00:11.035 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:00:11.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.020+0000 7f9226ffd640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] but i'm connected to v1:172.21.15.42:6789/0, reconnecting 2024-09-16T04:00:11.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.020+0000 7f9226ffd640 10 monclient: _reopen_session rank -1 2024-09-16T04:00:11.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.020+0000 7f9226ffd640 1 -- 172.21.15.42:0/2801319709 >> v1:172.21.15.42:6789/0 conn(0x7f92280a37f0 legacy=0x7f9228136e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:00:11.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.020+0000 7f9226ffd640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:00:11.036 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.020+0000 7f9226ffd640 1 --2- 172.21.15.42:0/2801319709 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f92140039c0 0x7f9214003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:00:11.036 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.020+0000 7f9226ffd640 10 monclient(hunting): picked mon.b con 0x7f92140039c0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:00:11.036 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.020+0000 7f9226ffd640 1 --2- 172.21.15.42:0/2801319709 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f92280a3c00 0x7f9214004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:00:11.036 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.020+0000 7f9226ffd640 10 monclient(hunting): picked mon.c con 0x7f92280a3c00 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:00:11.036 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.020+0000 7f9226ffd640 1 --2- 172.21.15.42:0/2801319709 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9214004d50 0x7f92140051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:00:11.036 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.020+0000 7f9226ffd640 10 monclient(hunting): picked mon.a con 0x7f9214004d50 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:00:11.036 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.020+0000 7f9226ffd640 10 monclient(hunting): start opening mon connection 2024-09-16T04:00:11.036 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.020+0000 7f9226ffd640 1 -- 172.21.15.42:0/2801319709 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f922806df70 con 0x7f9214004d50 2024-09-16T04:00:11.036 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.020+0000 7f9226ffd640 10 monclient(hunting): start opening mon connection 2024-09-16T04:00:11.036 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.020+0000 7f9226ffd640 1 -- 172.21.15.42:0/2801319709 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f922809bf80 con 0x7f92140039c0 2024-09-16T04:00:11.036 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.020+0000 7f9226ffd640 10 monclient(hunting): start opening mon connection 2024-09-16T04:00:11.036 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.020+0000 7f9226ffd640 1 -- 172.21.15.42:0/2801319709 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f921400bbe0 con 0x7f92280a3c00 2024-09-16T04:00:11.036 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.020+0000 7f9226ffd640 10 monclient(hunting): _renew_subs 2024-09-16T04:00:11.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.020+0000 7f9226ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:00:11.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.020+0000 7f92277fe640 1 -- 172.21.15.42:0/2801319709 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f92280a3c00 msgr2=0x7f9214004810 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T04:00:11.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.020+0000 7f92277fe640 1 --2- 172.21.15.42:0/2801319709 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f92280a3c00 0x7f9214004810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:00:11.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.020+0000 7f922e5d7640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-16T04:00:11.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.020+0000 7f922cb74640 1 --2- 172.21.15.42:0/2801319709 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9214004d50 0x7f92140051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:00:11.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.020+0000 7f922cb74640 10 monclient(hunting): get_auth_request con 0x7f9214004d50 auth_method 0 2024-09-16T04:00:11.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.020+0000 7f922cb74640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:00:11.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.020+0000 7f922cb74640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:00:11.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.020+0000 7f922cb74640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:00:11.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.020+0000 7f922cb74640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:00:11.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.020+0000 7f922cb74640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:00:11.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.020+0000 7f922cb74640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:00:11.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.024+0000 7f922cb74640 10 monclient(hunting): handle_auth_done global_id 825237 payload 293 2024-09-16T04:00:11.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.024+0000 7f922cb74640 1 -- 172.21.15.42:0/2801319709 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f92280a3c00 msgr2=0x7f9214004810 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:00:11.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.024+0000 7f922cb74640 1 --2- 172.21.15.42:0/2801319709 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f92280a3c00 0x7f9214004810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:00:11.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.024+0000 7f922cb74640 1 -- 172.21.15.42:0/2801319709 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f92140039c0 msgr2=0x7f9214003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:00:11.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.024+0000 7f922cb74640 1 --2- 172.21.15.42:0/2801319709 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f92140039c0 0x7f9214003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:00:11.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.024+0000 7f922cb74640 10 monclient: _finish_hunting 0 2024-09-16T04:00:11.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.024+0000 7f922cb74640 1 monclient: found mon.a 2024-09-16T04:00:11.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.024+0000 7f922cb74640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:00:11.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.024+0000 7f922cb74640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:00:11.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.024+0000 7f922cb74640 1 -- 172.21.15.42:0/2801319709 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9214003530 con 0x7f9214004d50 2024-09-16T04:00:11.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.024+0000 7f922cb74640 10 monclient: _finish_auth 0 2024-09-16T04:00:11.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.024+0000 7f922cb74640 10 monclient: _check_auth_tickets 2024-09-16T04:00:11.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.024+0000 7f922cb74640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:00:11.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.024+0000 7f922cb74640 1 --2- 172.21.15.42:0/2801319709 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9214004d50 0x7f92140051d0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f9210003990 tx=0x7f921001ce80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6f25cfba7a741d1 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:00:11.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.024+0000 7f9226ffd640 1 -- 172.21.15.42:0/2801319709 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9210009520 con 0x7f9214004d50 2024-09-16T04:00:11.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.024+0000 7f9226ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:00:11.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.024+0000 7f9226ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:00:11.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.024+0000 7f9226ffd640 10 monclient: dump: 2024-09-16T04:00:11.044 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:00:11.044 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:00:11.044 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:00:11.044 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:00:11.045 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:00:11.045 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:00:11.045 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:00:11.045 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:00:11.045 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:00:11.045 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:00:11.045 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.024+0000 7f9226ffd640 1 -- 172.21.15.42:0/2801319709 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f92100039c0 con 0x7f9214004d50 2024-09-16T04:00:11.045 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.024+0000 7f922e5d7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:00:11.045 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.024+0000 7f9226ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:00:11.045 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.024+0000 7f922e5d7640 10 monclient: get_monmap_and_config success 2024-09-16T04:00:11.045 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.024+0000 7f922e5d7640 4 set_mon_vals no callback set 2024-09-16T04:00:11.045 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.024+0000 7f9226ffd640 1 -- 172.21.15.42:0/2801319709 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9210003db0 con 0x7f9214004d50 2024-09-16T04:00:11.045 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.024+0000 7f9226ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:00:11.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.024+0000 7f9226ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:00:11.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.024+0000 7f9226ffd640 10 monclient: dump: 2024-09-16T04:00:11.046 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:00:11.046 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:00:11.046 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:00:11.046 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:00:11.046 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:00:11.046 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:00:11.046 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:00:11.046 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:00:11.046 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:00:11.046 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:00:11.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.024+0000 7f922e5d7640 10 monclient: shutdown 2024-09-16T04:00:11.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.024+0000 7f922e5d7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:00:11.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.024+0000 7f922e5d7640 1 -- 172.21.15.42:0/2801319709 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9214004d50 msgr2=0x7f92140051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:00:11.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.024+0000 7f922e5d7640 1 --2- 172.21.15.42:0/2801319709 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9214004d50 0x7f92140051d0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f9210003990 tx=0x7f921001ce80 comp rx=0 tx=0).stop 2024-09-16T04:00:11.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.024+0000 7f922cb74640 1 -- 172.21.15.42:0/2801319709 reap_dead start 2024-09-16T04:00:11.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.024+0000 7f922e5d7640 1 -- 172.21.15.42:0/2801319709 shutdown_connections 2024-09-16T04:00:11.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.024+0000 7f922e5d7640 1 -- 172.21.15.42:0/2801319709 >> 172.21.15.42:0/2801319709 conn(0x7f92280a9fb0 msgr2=0x7f92280aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:00:11.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.024+0000 7f922e5d7640 1 -- 172.21.15.42:0/2801319709 shutdown_connections 2024-09-16T04:00:11.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.024+0000 7f922e5d7640 1 -- 172.21.15.42:0/2801319709 wait complete. 2024-09-16T04:00:11.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.024+0000 7f922e5d7640 10 monclient: build_initial_monmap 2024-09-16T04:00:11.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.024+0000 7f922e5d7640 10 monclient: monmap: 2024-09-16T04:00:11.047 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:00:11.047 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:00:11.047 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:00:11.047 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:00:11.047 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:00:11.048 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:00:11.048 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:00:11.048 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:00:11.048 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:00:11.048 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:00:11.048 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.024+0000 7f922e5d7640 1 librados: starting msgr at 2024-09-16T04:00:11.048 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.024+0000 7f922e5d7640 1 librados: starting objecter 2024-09-16T04:00:11.048 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.028+0000 7f922e5d7640 1 Processor -- start 2024-09-16T04:00:11.048 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.028+0000 7f922e5d7640 1 -- start start 2024-09-16T04:00:11.048 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.028+0000 7f922e5d7640 1 librados: setting wanted keys 2024-09-16T04:00:11.048 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.028+0000 7f922e5d7640 1 librados: calling monclient init 2024-09-16T04:00:11.048 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.028+0000 7f922e5d7640 10 monclient: init 2024-09-16T04:00:11.048 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.028+0000 7f922e5d7640 10 monclient: _reopen_session rank -1 2024-09-16T04:00:11.048 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.028+0000 7f922e5d7640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:00:11.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.028+0000 7f922e5d7640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f92281cf690 0x7f92281cfa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:00:11.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.028+0000 7f922e5d7640 10 monclient(hunting): picked mon.noname-c con 0x7f92281cf690 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:00:11.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.028+0000 7f922e5d7640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f92281cffb0 0x7f9228139830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:00:11.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.028+0000 7f922e5d7640 10 monclient(hunting): picked mon.noname-b con 0x7f92281cffb0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:00:11.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.028+0000 7f922e5d7640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9228139d70 0x7f922813a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:00:11.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.028+0000 7f922e5d7640 10 monclient(hunting): picked mon.noname-a con 0x7f9228139d70 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:00:11.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.028+0000 7f922e5d7640 10 monclient(hunting): start opening mon connection 2024-09-16T04:00:11.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.028+0000 7f922e5d7640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f922813f3c0 con 0x7f9228139d70 2024-09-16T04:00:11.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.028+0000 7f922e5d7640 10 monclient(hunting): start opening mon connection 2024-09-16T04:00:11.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.028+0000 7f922e5d7640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f92280a4100 con 0x7f92281cffb0 2024-09-16T04:00:11.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.028+0000 7f922e5d7640 10 monclient(hunting): start opening mon connection 2024-09-16T04:00:11.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.028+0000 7f922e5d7640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f922813f540 con 0x7f92281cf690 2024-09-16T04:00:11.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.028+0000 7f922e5d7640 10 monclient(hunting): _renew_subs 2024-09-16T04:00:11.050 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.028+0000 7f922e5d7640 10 monclient(hunting): authenticate will time out at 2058.000802s 2024-09-16T04:00:11.050 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.028+0000 7f9227fff640 1 -- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f92281cf690 msgr2=0x7f92281cfa70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T04:00:11.050 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.028+0000 7f9227fff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f92281cf690 0x7f92281cfa70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:00:11.050 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.028+0000 7f922cb74640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9228139d70 0x7f922813a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:00:11.050 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.028+0000 7f922cb74640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9228139d70 0x7f922813a1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:59724/0 (socket says 172.21.15.42:59724) 2024-09-16T04:00:11.050 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.028+0000 7f922cb74640 1 -- 172.21.15.42:0/2457836754 learned_addr learned my addr 172.21.15.42:0/2457836754 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:00:11.050 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.028+0000 7f922cb74640 10 monclient(hunting): get_auth_request con 0x7f9228139d70 auth_method 0 2024-09-16T04:00:11.050 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.028+0000 7f922cb74640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:00:11.050 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.028+0000 7f922cb74640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:00:11.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.028+0000 7f922cb74640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:00:11.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.028+0000 7f922cb74640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:00:11.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.028+0000 7f922cb74640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:00:11.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.028+0000 7f922cb74640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:00:11.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.028+0000 7f922cb74640 10 monclient(hunting): handle_auth_done global_id 825240 payload 824 2024-09-16T04:00:11.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.028+0000 7f92277fe640 1 --2- 172.21.15.42:0/2457836754 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f92281cffb0 0x7f9228139830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:00:11.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.028+0000 7f922cb74640 1 -- 172.21.15.42:0/2457836754 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f92281cf690 msgr2=0x7f92281cfa70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:00:11.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.028+0000 7f922cb74640 1 --2- 172.21.15.42:0/2457836754 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f92281cf690 0x7f92281cfa70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:00:11.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.028+0000 7f922cb74640 1 -- 172.21.15.42:0/2457836754 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f92281cffb0 msgr2=0x7f9228139830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:00:11.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.028+0000 7f922cb74640 1 --2- 172.21.15.42:0/2457836754 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f92281cffb0 0x7f9228139830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:00:11.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.028+0000 7f922cb74640 10 monclient: _finish_hunting 0 2024-09-16T04:00:11.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.028+0000 7f922cb74640 1 monclient: found mon.noname-a 2024-09-16T04:00:11.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.028+0000 7f922cb74640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:00:11.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.028+0000 7f922cb74640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:00:11.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.028+0000 7f922cb74640 1 -- 172.21.15.42:0/2457836754 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9228136ef0 con 0x7f9228139d70 2024-09-16T04:00:11.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.028+0000 7f92277fe640 10 monclient: get_auth_request con 0x7f92281cffb0 auth_method 0 2024-09-16T04:00:11.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.028+0000 7f92277fe640 1 --2- 172.21.15.42:0/2457836754 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f92281cffb0 0x7f9228139830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T04:00:11.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.028+0000 7f922cb74640 1 --2- 172.21.15.42:0/2457836754 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9228139d70 0x7f922813a1f0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f9210000eb0 tx=0x7f9214005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:00:11.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.028+0000 7f9224ff9640 10 client.?.objecter ms_handle_connect 0x7f9228139d70 2024-09-16T04:00:11.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.028+0000 7f9224ff9640 10 client.?.objecter resend_mon_ops 2024-09-16T04:00:11.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.028+0000 7f9224ff9640 1 -- 172.21.15.42:0/2457836754 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9210009500 con 0x7f9228139d70 2024-09-16T04:00:11.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.028+0000 7f9224ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:00:11.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.028+0000 7f9224ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:00:11.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.028+0000 7f9224ff9640 10 monclient: dump: 2024-09-16T04:00:11.052 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:00:11.052 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:00:11.052 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:00:11.053 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:00:11.053 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:00:11.053 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:00:11.053 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:00:11.053 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:00:11.053 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:00:11.053 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:00:11.053 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.028+0000 7f9224ff9640 10 monclient: _finish_auth 0 2024-09-16T04:00:11.053 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.028+0000 7f9224ff9640 10 monclient: _check_auth_tickets 2024-09-16T04:00:11.053 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.028+0000 7f9224ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:00:11.053 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.028+0000 7f9224ff9640 1 -- 172.21.15.42:0/2457836754 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f92100a4e00 con 0x7f9228139d70 2024-09-16T04:00:11.053 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.028+0000 7f9224ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T04:00:11.053 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.028+0000 7f9224ff9640 1 -- 172.21.15.42:0/2457836754 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9210003d70 con 0x7f9228139d70 2024-09-16T04:00:11.053 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.028+0000 7f9224ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:00:11.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.028+0000 7f9224ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:00:11.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.028+0000 7f9224ff9640 10 monclient: dump: 2024-09-16T04:00:11.054 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:00:11.054 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:00:11.054 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:00:11.054 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:00:11.054 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:00:11.054 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:00:11.054 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:00:11.054 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:00:11.054 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:00:11.054 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:00:11.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.028+0000 7f922e5d7640 5 monclient: authenticate success, global_id 825240 2024-09-16T04:00:11.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.028+0000 7f922e5d7640 10 monclient: _renew_subs 2024-09-16T04:00:11.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.028+0000 7f922e5d7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:00:11.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.028+0000 7f92267fc640 4 set_mon_vals no callback set 2024-09-16T04:00:11.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.028+0000 7f922e5d7640 1 -- 172.21.15.42:0/2457836754 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f922813bfc0 con 0x7f9228139d70 2024-09-16T04:00:11.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.028+0000 7f922e5d7640 10 client.825240.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:00:11.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.028+0000 7f922e5d7640 10 monclient: _renew_subs 2024-09-16T04:00:11.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.028+0000 7f922e5d7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:00:11.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.028+0000 7f922e5d7640 1 -- 172.21.15.42:0/2457836754 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f922809ccd0 con 0x7f9228139d70 2024-09-16T04:00:11.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.028+0000 7f922e5d7640 1 librados: init done 2024-09-16T04:00:11.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.028+0000 7f922e5d7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:00:11.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.028+0000 7f922e5d7640 1 --2- 172.21.15.42:0/2457836754 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f922813a730 0x7f92281349e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:00:11.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.028+0000 7f9224ff9640 1 -- 172.21.15.42:0/2457836754 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f92100050d0 con 0x7f9228139d70 2024-09-16T04:00:11.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.028+0000 7f922e5d7640 10 monclient(hunting): start opening mon connection 2024-09-16T04:00:11.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.028+0000 7f922e5d7640 1 -- 172.21.15.42:0/2457836754 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f92281340c0 con 0x7f922813a730 2024-09-16T04:00:11.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.028+0000 7f9224ff9640 1 --2- 172.21.15.42:0/2457836754 >> v2:172.21.15.154:6812/254845342 conn(0x7f92080377b0 0x7f9208039c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:00:11.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.028+0000 7f9224ff9640 1 -- 172.21.15.42:0/2457836754 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(542..542 src has 451..542) ==== 6224+0+0 (secure 0 0 0) 0x7f92100e1670 con 0x7f9228139d70 2024-09-16T04:00:11.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.028+0000 7f9224ff9640 10 client.825240.objecter ms_dispatch 0x7f9228000ba0 osd_map(542..542 src has 451..542) 2024-09-16T04:00:11.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.028+0000 7f9224ff9640 3 client.825240.objecter handle_osd_map got epochs [542,542] > 0 2024-09-16T04:00:11.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.028+0000 7f9224ff9640 3 client.825240.objecter handle_osd_map decoding full epoch 542 2024-09-16T04:00:11.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.028+0000 7f9224ff9640 20 client.825240.objecter dump_active .. 0 homeless 2024-09-16T04:00:11.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.028+0000 7f9227fff640 1 --2- 172.21.15.42:0/2457836754 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f922813a730 0x7f92281349e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:00:11.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.028+0000 7f9227fff640 10 monclient: get_auth_request con 0x7f922813a730 auth_method 0 2024-09-16T04:00:11.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.028+0000 7f9227fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:00:11.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.028+0000 7f9227fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:00:11.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.028+0000 7f9227fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:00:11.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.028+0000 7f92277fe640 1 --2- 172.21.15.42:0/2457836754 >> v2:172.21.15.154:6812/254845342 conn(0x7f92080377b0 0x7f9208039c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:00:11.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.028+0000 7f92277fe640 10 monclient: get_auth_request con 0x7f92080377b0 auth_method 0 2024-09-16T04:00:11.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.028+0000 7f9227fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:00:11.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.028+0000 7f9227fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:00:11.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.028+0000 7f9227fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:00:11.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.028+0000 7f9227fff640 10 monclient(hunting): handle_auth_done global_id 904402 payload 824 2024-09-16T04:00:11.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.028+0000 7f9227fff640 1 -- 172.21.15.42:0/2457836754 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f922813c1f0 con 0x7f922813a730 2024-09-16T04:00:11.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.032+0000 7f9224ff9640 10 client.825240.objecter ms_handle_connect 0x7f922813a730 2024-09-16T04:00:11.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.032+0000 7f9224ff9640 10 client.825240.objecter resend_mon_ops 2024-09-16T04:00:11.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.032+0000 7f9227fff640 1 --2- 172.21.15.42:0/2457836754 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f922813a730 0x7f92281349e0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f922000e810 tx=0x7f922000ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:00:11.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.032+0000 7f9224ff9640 10 client.825240.objecter ms_handle_connect 0x7f92080377b0 2024-09-16T04:00:11.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.032+0000 7f92277fe640 1 --2- 172.21.15.42:0/2457836754 >> v2:172.21.15.154:6812/254845342 conn(0x7f92080377b0 0x7f9208039c70 secure :-1 s=READY pgs=3742 cs=0 l=1 rev1=1 crypto rx=0x7f9218004bd0 tx=0x7f9218009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:00:11.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.032+0000 7f9224ff9640 1 -- 172.21.15.42:0/2457836754 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9220017070 con 0x7f922813a730 2024-09-16T04:00:11.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.032+0000 7f9224ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:00:11.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.032+0000 7f9224ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:00:11.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.032+0000 7f9224ff9640 10 monclient: dump: 2024-09-16T04:00:11.058 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:00:11.058 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:00:11.058 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:00:11.058 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:00:11.058 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:00:11.058 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:00:11.058 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:00:11.058 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:00:11.058 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:00:11.058 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:00:11.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.032+0000 7f9224ff9640 1 -- 172.21.15.42:0/2457836754 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f922000c850 con 0x7f922813a730 2024-09-16T04:00:11.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.032+0000 7f9224ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:00:11.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.032+0000 7f9224ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:00:11.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.032+0000 7f9224ff9640 1 -- 172.21.15.42:0/2457836754 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f922813a730 msgr2=0x7f92281349e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:00:11.059 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.032+0000 7f9224ff9640 1 --2- 172.21.15.42:0/2457836754 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f922813a730 0x7f92281349e0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f922000e810 tx=0x7f922000ecd0 comp rx=0 tx=0).stop 2024-09-16T04:00:11.059 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.032+0000 7f9224ff9640 1 -- 172.21.15.42:0/2457836754 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f922813a730 msgr2=0x7f92281349e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:00:11.059 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.032+0000 7f9224ff9640 1 --2- 172.21.15.42:0/2457836754 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f922813a730 0x7f92281349e0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:00:11.061 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.036+0000 7f922e5d7640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:00:11.061 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.036+0000 7f922e5d7640 1 --2- 172.21.15.42:0/2457836754 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9228134f20 0x7f9228135330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:00:11.061 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.036+0000 7f922e5d7640 10 monclient(hunting): start opening mon connection 2024-09-16T04:00:11.061 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.036+0000 7f922e5d7640 1 -- 172.21.15.42:0/2457836754 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f9228068e40 con 0x7f9228134f20 2024-09-16T04:00:11.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.040+0000 7f9227fff640 1 --2- 172.21.15.42:0/2457836754 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9228134f20 0x7f9228135330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:00:11.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.040+0000 7f9227fff640 10 monclient: get_auth_request con 0x7f9228134f20 auth_method 0 2024-09-16T04:00:11.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.040+0000 7f9227fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:00:11.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.040+0000 7f9227fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:00:11.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.040+0000 7f9227fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:00:11.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.040+0000 7f9227fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:00:11.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.040+0000 7f9227fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:00:11.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.040+0000 7f9227fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:00:11.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.040+0000 7f9227fff640 10 monclient(hunting): handle_auth_done global_id 904405 payload 824 2024-09-16T04:00:11.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.040+0000 7f9227fff640 1 -- 172.21.15.42:0/2457836754 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f92280a3530 con 0x7f9228134f20 2024-09-16T04:00:11.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.040+0000 7f9227fff640 1 --2- 172.21.15.42:0/2457836754 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9228134f20 0x7f9228135330 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f9220019000 tx=0x7f92200046d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:00:11.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.040+0000 7f9224ff9640 10 client.825240.objecter ms_handle_connect 0x7f9228134f20 2024-09-16T04:00:11.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.040+0000 7f9224ff9640 10 client.825240.objecter resend_mon_ops 2024-09-16T04:00:11.063 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.040+0000 7f9224ff9640 1 -- 172.21.15.42:0/2457836754 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9220017070 con 0x7f9228134f20 2024-09-16T04:00:11.063 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.040+0000 7f9224ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:00:11.063 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.040+0000 7f9224ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:00:11.063 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.040+0000 7f9224ff9640 10 monclient: dump: 2024-09-16T04:00:11.063 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:00:11.063 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:00:11.063 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:00:11.063 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:00:11.063 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:00:11.063 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:00:11.063 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:00:11.063 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:00:11.063 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:00:11.064 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:00:11.064 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.040+0000 7f9224ff9640 1 -- 172.21.15.42:0/2457836754 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4869 (secure 0 0 0) 0x7f9220006650 con 0x7f9228134f20 2024-09-16T04:00:11.064 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.040+0000 7f9224ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:00:11.064 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.040+0000 7f9224ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:00:11.064 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.040+0000 7f9224ff9640 1 -- 172.21.15.42:0/2457836754 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9228134f20 msgr2=0x7f9228135330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:00:11.064 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.040+0000 7f9224ff9640 1 --2- 172.21.15.42:0/2457836754 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9228134f20 0x7f9228135330 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f9220019000 tx=0x7f92200046d0 comp rx=0 tx=0).stop 2024-09-16T04:00:11.064 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.040+0000 7f9224ff9640 1 -- 172.21.15.42:0/2457836754 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9228134f20 msgr2=0x7f9228135330 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:00:11.064 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.040+0000 7f9224ff9640 1 --2- 172.21.15.42:0/2457836754 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9228134f20 0x7f9228135330 secure :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f9220019000 tx=0x7f92200046d0 comp rx=0 tx=0).stop 2024-09-16T04:00:11.064 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T04:00:11.064 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:00:11.065 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:00:11.065 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T04:00:11.065 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 162, 2024-09-16T04:00:11.065 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 139224, 2024-09-16T04:00:11.065 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T04:00:11.065 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T04:00:11.065 INFO:teuthology.orchestra.run.smithi042.stdout: 1 2024-09-16T04:00:11.065 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:00:11.065 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 1, 2024-09-16T04:00:11.065 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:00:11.065 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T04:00:11.065 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T04:00:11.065 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:00:11.065 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:00:11.066 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:00:11.066 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:00:11.066 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:00:11.066 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:00:11.066 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:00:11.066 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:00:11.066 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:00:11.066 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:00:11.066 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:00:11.066 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:00:11.066 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:00:11.066 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T04:00:11.066 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:00:11.066 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:00:11.067 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:00:11.067 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:00:11.067 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:00:11.067 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:00:11.067 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:00:11.067 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:00:11.067 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:00:11.067 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:00:11.067 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:00:11.067 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:00:11.067 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:00:11.067 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T04:00:11.067 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T04:00:11.067 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T04:00:11.068 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T04:00:11.068 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T04:00:11.068 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T04:00:11.068 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:00:11.068 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:00:11.068 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T04:00:11.068 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T04:00:11.068 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T04:00:11.068 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T04:00:11.068 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T04:00:11.068 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T04:00:11.068 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T04:00:11.068 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:00:11.069 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T04:00:11.069 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:00:11.069 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:00:11.069 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:00:11.069 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:00:11.069 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:00:11.069 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:00:11.069 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:00:11.069 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:00:11.069 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:00:11.069 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:00:11.069 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:00:11.069 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:00:11.069 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T04:00:11.070 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:00:11.070 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T04:00:11.070 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:00:11.070 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:00:11.070 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:00:11.070 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:00:11.070 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:00:11.070 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:00:11.070 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:00:11.070 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T04:00:11.070 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:00:11.070 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:00:11.070 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:00:11.070 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:00:11.071 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T04:00:11.071 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:00:11.071 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:00:11.071 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:00:11.071 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:00:11.071 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T04:00:11.071 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T04:00:11.071 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:00:11.071 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:00:11.071 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:00:11.071 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:00:11.071 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:00:11.071 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:00:11.071 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:00:11.072 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:00:11.072 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:00:11.072 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:00:11.072 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:00:11.072 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T04:00:11.072 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:00:11.072 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:00:11.072 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:00:11.072 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:00:11.072 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T04:00:11.072 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:00:11.072 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:00:11.072 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:00:11.072 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:00:11.073 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T04:00:11.073 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T04:00:11.073 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:00:11.073 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:00:11.073 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:00:11.073 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:00:11.073 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:00:11.073 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:00:11.073 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:00:11.073 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:00:11.073 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:00:11.073 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:00:11.073 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:00:11.073 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T04:00:11.074 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:00:11.074 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:00:11.074 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:00:11.074 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:00:11.074 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T04:00:11.074 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:00:11.074 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:00:11.074 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:00:11.074 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:00:11.074 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T04:00:11.074 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T04:00:11.074 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:00:11.074 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:00:11.074 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:00:11.075 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:00:11.075 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:00:11.075 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:00:11.075 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T04:00:11.075 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T04:00:11.075 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:00:11.075 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:00:11.075 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:00:11.075 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:00:11.075 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:00:11.075 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:00:11.075 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T04:00:11.075 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:00:11.075 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:00:11.076 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:00:11.076 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:00:11.076 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:00:11.076 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:00:11.076 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T04:00:11.076 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:00:11.076 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:00:11.076 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:00:11.076 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 2 2024-09-16T04:00:11.076 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:00:11.076 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:00:11.076 INFO:teuthology.orchestra.run.smithi042.stdout: "mgr": [ 2024-09-16T04:00:11.076 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:00:11.076 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:00:11.077 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:00:11.077 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:00:11.077 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:00:11.077 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:00:11.077 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:00:11.077 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T04:00:11.077 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T04:00:11.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.040+0000 7f922e5d7640 10 librados: watch_flush enter 2024-09-16T04:00:11.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.040+0000 7f922e5d7640 10 librados: watch_flush exit 2024-09-16T04:00:11.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.040+0000 7f922e5d7640 20 client.825240.objecter shutdown clearing up homeless session... 2024-09-16T04:00:11.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.040+0000 7f922e5d7640 10 client.825240.objecter successfully canceled tick 2024-09-16T04:00:11.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.040+0000 7f922e5d7640 1 -- 172.21.15.42:0/2457836754 >> v2:172.21.15.154:6812/254845342 conn(0x7f92080377b0 msgr2=0x7f9208039c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:00:11.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.040+0000 7f922e5d7640 1 --2- 172.21.15.42:0/2457836754 >> v2:172.21.15.154:6812/254845342 conn(0x7f92080377b0 0x7f9208039c70 secure :-1 s=READY pgs=3742 cs=0 l=1 rev1=1 crypto rx=0x7f9218004bd0 tx=0x7f9218009040 comp rx=0 tx=0).stop 2024-09-16T04:00:11.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.040+0000 7f922e5d7640 10 monclient: shutdown 2024-09-16T04:00:11.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.040+0000 7f922e5d7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:00:11.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.040+0000 7f922e5d7640 1 -- 172.21.15.42:0/2457836754 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9228139d70 msgr2=0x7f922813a1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:00:11.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.040+0000 7f922e5d7640 1 --2- 172.21.15.42:0/2457836754 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9228139d70 0x7f922813a1f0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f9210000eb0 tx=0x7f9214005840 comp rx=0 tx=0).stop 2024-09-16T04:00:11.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.040+0000 7f922cb74640 1 -- 172.21.15.42:0/2457836754 reap_dead start 2024-09-16T04:00:11.080 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.040+0000 7f922cb74640 1 -- 172.21.15.42:0/2457836754 reap_dead start 2024-09-16T04:00:11.080 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.040+0000 7f922e5d7640 1 -- 172.21.15.42:0/2457836754 shutdown_connections 2024-09-16T04:00:11.080 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.040+0000 7f922e5d7640 1 -- 172.21.15.42:0/2457836754 >> 172.21.15.42:0/2457836754 conn(0x7f92280a9fb0 msgr2=0x7f92281307c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:00:11.080 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.040+0000 7f922e5d7640 1 -- 172.21.15.42:0/2457836754 shutdown_connections 2024-09-16T04:00:11.080 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.040+0000 7f922e5d7640 1 -- 172.21.15.42:0/2457836754 wait complete. 2024-09-16T04:00:11.080 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:11.040+0000 7f922e5d7640 1 librados: shutdown 2024-09-16T04:00:11.080 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T04:00:11.080 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-16T04:00:31.081 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-16T04:00:31.082 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-09-16T04:00:31.082 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-16T04:00:31.082 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-16T04:00:31.085 INFO:tasks.ceph.mon.c:Started 2024-09-16T04:00:31.085 INFO:tasks.mon_thrash:Sending STOP to mon b 2024-09-16T04:00:31.086 INFO:tasks.ceph.mon.b:Sent signal 19 2024-09-16T04:00:31.086 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-16T04:00:46.089 INFO:tasks.mon_thrash:Sending CONT to mon b 2024-09-16T04:00:46.090 INFO:tasks.ceph.mon.b:Sent signal 18 2024-09-16T04:00:46.090 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T04:00:46.090 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:00:46.223 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:46.216+0000 7f3c59b13640 10 monclient: get_monmap_and_config 2024-09-16T04:00:46.223 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:46.216+0000 7f3c59b13640 10 monclient: build_initial_monmap 2024-09-16T04:00:46.223 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:46.216+0000 7f3c59b13640 10 monclient: monmap: 2024-09-16T04:00:46.223 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:00:46.223 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:00:46.223 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:00:46.221609+0000 2024-09-16T04:00:46.223 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:00:46.221609+0000 2024-09-16T04:00:46.224 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:00:46.224 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:00:46.224 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:00:46.224 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:00:46.224 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:00:46.224 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:00:46.224 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:00:46.224 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:00:46.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:46.216+0000 7f3c59b13640 1 Processor -- start 2024-09-16T04:00:46.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:46.216+0000 7f3c59b13640 1 -- start start 2024-09-16T04:00:46.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:46.216+0000 7f3c59b13640 10 monclient: init 2024-09-16T04:00:46.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:46.216+0000 7f3c59b13640 10 monclient: _reopen_session rank -1 2024-09-16T04:00:46.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:46.216+0000 7f3c59b13640 10 monclient: _add_conns ranks=[3,2,4,1,0] 2024-09-16T04:00:46.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:46.216+0000 7f3c59b13640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f3c540a2f10 0x7f3c540a1330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:00:46.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:46.216+0000 7f3c59b13640 10 monclient(hunting): picked mon.noname-a con 0x7f3c540a2f10 addr v2:172.21.15.42:3300/0 2024-09-16T04:00:46.227 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:46.216+0000 7f3c59b13640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3c5409e070 0x7f3c540a1870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:00:46.227 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:46.216+0000 7f3c59b13640 10 monclient(hunting): picked mon.noname-b con 0x7f3c5409e070 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:00:46.227 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:46.216+0000 7f3c59b13640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f3c540a1db0 0x7f3c5413b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:00:46.227 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:46.216+0000 7f3c59b13640 10 monclient(hunting): picked mon.noname-c con 0x7f3c540a1db0 addr v2:172.21.15.154:3300/0 2024-09-16T04:00:46.227 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:46.216+0000 7f3c59b13640 10 monclient(hunting): start opening mon connection 2024-09-16T04:00:46.227 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:46.216+0000 7f3c59b13640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f3c540a32f0 con 0x7f3c540a2f10 2024-09-16T04:00:46.227 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:46.216+0000 7f3c59b13640 10 monclient(hunting): start opening mon connection 2024-09-16T04:00:46.227 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:46.216+0000 7f3c59b13640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f3c5409dc50 con 0x7f3c540a1db0 2024-09-16T04:00:46.227 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:46.216+0000 7f3c59b13640 10 monclient(hunting): start opening mon connection 2024-09-16T04:00:46.227 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:46.216+0000 7f3c59b13640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f3c54073290 con 0x7f3c5409e070 2024-09-16T04:00:46.227 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:46.216+0000 7f3c59b13640 10 monclient(hunting): _renew_subs 2024-09-16T04:00:46.227 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:46.216+0000 7f3c59b13640 10 monclient(hunting): authenticate will time out at 2093.191090s 2024-09-16T04:00:46.227 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:46.216+0000 7f3c52ffd640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3c5409e070 0x7f3c540a1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:00:46.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:46.216+0000 7f3c52ffd640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3c5409e070 0x7f3c540a1870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:53202/0 (socket says 172.21.15.42:53202) 2024-09-16T04:00:46.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:46.216+0000 7f3c52ffd640 1 -- 172.21.15.42:0/1068448438 learned_addr learned my addr 172.21.15.42:0/1068448438 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:00:46.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:46.216+0000 7f3c52ffd640 10 monclient(hunting): get_auth_request con 0x7f3c5409e070 auth_method 0 2024-09-16T04:00:46.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:46.216+0000 7f3c52ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:00:46.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:46.216+0000 7f3c52ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:00:46.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:46.216+0000 7f3c52ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:00:46.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:46.216+0000 7f3c537fe640 1 --2- 172.21.15.42:0/1068448438 >> v2:172.21.15.42:3300/0 conn(0x7f3c540a2f10 0x7f3c540a1330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:00:46.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:46.216+0000 7f3c537fe640 10 monclient(hunting): get_auth_request con 0x7f3c540a2f10 auth_method 0 2024-09-16T04:00:46.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:46.216+0000 7f3c537fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:00:46.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:46.216+0000 7f3c537fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:00:46.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:46.216+0000 7f3c537fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:00:46.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:46.216+0000 7f3c52ffd640 1 -- 172.21.15.42:0/1068448438 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3c5409e070 msgr2=0x7f3c540a1870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:00:46.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:46.216+0000 7f3c52ffd640 1 -- 172.21.15.42:0/1068448438 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3c5409e070 msgr2=0x7f3c540a1870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:00:46.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:46.216+0000 7f3c52ffd640 1 --2- 172.21.15.42:0/1068448438 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3c5409e070 0x7f3c540a1870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:00:46.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:46.216+0000 7f3c52ffd640 1 --2- 172.21.15.42:0/1068448438 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3c5409e070 0x7f3c540a1870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:00:46.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:46.216+0000 7f3c537fe640 1 -- 172.21.15.42:0/1068448438 >> v2:172.21.15.42:3300/0 conn(0x7f3c540a2f10 msgr2=0x7f3c540a1330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:00:46.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:46.216+0000 7f3c537fe640 1 -- 172.21.15.42:0/1068448438 >> v2:172.21.15.42:3300/0 conn(0x7f3c540a2f10 msgr2=0x7f3c540a1330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:00:46.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:46.216+0000 7f3c537fe640 1 --2- 172.21.15.42:0/1068448438 >> v2:172.21.15.42:3300/0 conn(0x7f3c540a2f10 0x7f3c540a1330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:00:46.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:46.216+0000 7f3c537fe640 1 --2- 172.21.15.42:0/1068448438 >> v2:172.21.15.42:3300/0 conn(0x7f3c540a2f10 0x7f3c540a1330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:00:46.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:46.216+0000 7f3c53fff640 1 --2- 172.21.15.42:0/1068448438 >> v2:172.21.15.154:3300/0 conn(0x7f3c540a1db0 0x7f3c5413b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:00:46.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:46.216+0000 7f3c53fff640 10 monclient(hunting): get_auth_request con 0x7f3c540a1db0 auth_method 0 2024-09-16T04:00:46.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:46.216+0000 7f3c53fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:00:46.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:46.216+0000 7f3c53fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:00:46.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:46.216+0000 7f3c53fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:00:46.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:46.216+0000 7f3c53fff640 1 -- 172.21.15.42:0/1068448438 >> v2:172.21.15.154:3300/0 conn(0x7f3c540a1db0 msgr2=0x7f3c5413b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T04:00:46.230 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:46.216+0000 7f3c53fff640 1 -- 172.21.15.42:0/1068448438 >> v2:172.21.15.154:3300/0 conn(0x7f3c540a1db0 msgr2=0x7f3c5413b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:00:46.230 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:46.216+0000 7f3c53fff640 1 --2- 172.21.15.42:0/1068448438 >> v2:172.21.15.154:3300/0 conn(0x7f3c540a1db0 0x7f3c5413b8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:00:46.230 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:46.216+0000 7f3c53fff640 1 --2- 172.21.15.42:0/1068448438 >> v2:172.21.15.154:3300/0 conn(0x7f3c540a1db0 0x7f3c5413b8f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:00:46.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:46.420+0000 7f3c52ffd640 1 --2- 172.21.15.42:0/1068448438 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3c5409e070 0x7f3c540a1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:00:46.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:46.420+0000 7f3c537fe640 1 --2- 172.21.15.42:0/1068448438 >> v2:172.21.15.42:3300/0 conn(0x7f3c540a2f10 0x7f3c540a1330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:00:46.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:46.420+0000 7f3c537fe640 10 monclient(hunting): get_auth_request con 0x7f3c540a2f10 auth_method 0 2024-09-16T04:00:46.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:46.420+0000 7f3c537fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:00:46.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:46.420+0000 7f3c537fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:00:46.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:46.420+0000 7f3c537fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:00:46.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:46.420+0000 7f3c52ffd640 10 monclient(hunting): get_auth_request con 0x7f3c5409e070 auth_method 0 2024-09-16T04:00:46.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:46.420+0000 7f3c52ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:00:46.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:46.420+0000 7f3c52ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:00:46.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:46.420+0000 7f3c52ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:00:46.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:46.420+0000 7f3c53fff640 1 --2- 172.21.15.42:0/1068448438 >> v2:172.21.15.154:3300/0 conn(0x7f3c540a1db0 0x7f3c5413b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:00:46.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:46.420+0000 7f3c53fff640 10 monclient(hunting): get_auth_request con 0x7f3c540a1db0 auth_method 0 2024-09-16T04:00:46.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:46.420+0000 7f3c53fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:00:46.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:46.420+0000 7f3c53fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:00:46.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:46.420+0000 7f3c53fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:00:46.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:46.420+0000 7f3c537fe640 1 -- 172.21.15.42:0/1068448438 >> v2:172.21.15.42:3300/0 conn(0x7f3c540a2f10 msgr2=0x7f3c540a1330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:00:46.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:46.420+0000 7f3c52ffd640 1 -- 172.21.15.42:0/1068448438 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3c5409e070 msgr2=0x7f3c540a1870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:00:46.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:46.420+0000 7f3c537fe640 1 -- 172.21.15.42:0/1068448438 >> v2:172.21.15.42:3300/0 conn(0x7f3c540a2f10 msgr2=0x7f3c540a1330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:00:46.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:46.420+0000 7f3c52ffd640 1 -- 172.21.15.42:0/1068448438 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3c5409e070 msgr2=0x7f3c540a1870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:00:46.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:46.420+0000 7f3c52ffd640 1 --2- 172.21.15.42:0/1068448438 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3c5409e070 0x7f3c540a1870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:00:46.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:46.420+0000 7f3c537fe640 1 --2- 172.21.15.42:0/1068448438 >> v2:172.21.15.42:3300/0 conn(0x7f3c540a2f10 0x7f3c540a1330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:00:46.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:46.420+0000 7f3c52ffd640 1 --2- 172.21.15.42:0/1068448438 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3c5409e070 0x7f3c540a1870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T04:00:46.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:46.420+0000 7f3c537fe640 1 --2- 172.21.15.42:0/1068448438 >> v2:172.21.15.42:3300/0 conn(0x7f3c540a2f10 0x7f3c540a1330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T04:00:46.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:46.420+0000 7f3c53fff640 1 -- 172.21.15.42:0/1068448438 >> v2:172.21.15.154:3300/0 conn(0x7f3c540a1db0 msgr2=0x7f3c5413b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T04:00:46.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:46.420+0000 7f3c53fff640 1 -- 172.21.15.42:0/1068448438 >> v2:172.21.15.154:3300/0 conn(0x7f3c540a1db0 msgr2=0x7f3c5413b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:00:46.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:46.420+0000 7f3c53fff640 1 --2- 172.21.15.42:0/1068448438 >> v2:172.21.15.154:3300/0 conn(0x7f3c540a1db0 0x7f3c5413b8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:00:46.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:46.420+0000 7f3c53fff640 1 --2- 172.21.15.42:0/1068448438 >> v2:172.21.15.154:3300/0 conn(0x7f3c540a1db0 0x7f3c5413b8f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T04:00:46.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:46.824+0000 7f3c52ffd640 1 --2- 172.21.15.42:0/1068448438 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3c5409e070 0x7f3c540a1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:00:46.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:46.824+0000 7f3c537fe640 1 --2- 172.21.15.42:0/1068448438 >> v2:172.21.15.42:3300/0 conn(0x7f3c540a2f10 0x7f3c540a1330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:00:46.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:46.824+0000 7f3c537fe640 10 monclient(hunting): get_auth_request con 0x7f3c540a2f10 auth_method 0 2024-09-16T04:00:46.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:46.824+0000 7f3c537fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:00:46.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:46.824+0000 7f3c537fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:00:46.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:46.824+0000 7f3c537fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:00:46.832 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:46.824+0000 7f3c52ffd640 10 monclient(hunting): get_auth_request con 0x7f3c5409e070 auth_method 0 2024-09-16T04:00:46.832 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:46.824+0000 7f3c52ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:00:46.832 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:46.824+0000 7f3c52ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:00:46.832 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:46.824+0000 7f3c52ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:00:46.832 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:46.824+0000 7f3c53fff640 1 --2- 172.21.15.42:0/1068448438 >> v2:172.21.15.154:3300/0 conn(0x7f3c540a1db0 0x7f3c5413b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:00:46.832 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:46.824+0000 7f3c53fff640 10 monclient(hunting): get_auth_request con 0x7f3c540a1db0 auth_method 0 2024-09-16T04:00:46.832 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:46.824+0000 7f3c53fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:00:46.832 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:46.824+0000 7f3c53fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:00:46.832 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:46.824+0000 7f3c53fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:00:46.832 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:46.824+0000 7f3c537fe640 1 -- 172.21.15.42:0/1068448438 >> v2:172.21.15.42:3300/0 conn(0x7f3c540a2f10 msgr2=0x7f3c540a1330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T04:00:46.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:46.824+0000 7f3c537fe640 1 -- 172.21.15.42:0/1068448438 >> v2:172.21.15.42:3300/0 conn(0x7f3c540a2f10 msgr2=0x7f3c540a1330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:00:46.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:46.824+0000 7f3c537fe640 1 --2- 172.21.15.42:0/1068448438 >> v2:172.21.15.42:3300/0 conn(0x7f3c540a2f10 0x7f3c540a1330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:00:46.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:46.824+0000 7f3c52ffd640 1 -- 172.21.15.42:0/1068448438 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3c5409e070 msgr2=0x7f3c540a1870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:00:46.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:46.824+0000 7f3c52ffd640 1 -- 172.21.15.42:0/1068448438 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3c5409e070 msgr2=0x7f3c540a1870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:00:46.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:46.824+0000 7f3c52ffd640 1 --2- 172.21.15.42:0/1068448438 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3c5409e070 0x7f3c540a1870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:00:46.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:46.824+0000 7f3c537fe640 1 --2- 172.21.15.42:0/1068448438 >> v2:172.21.15.42:3300/0 conn(0x7f3c540a2f10 0x7f3c540a1330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T04:00:46.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:46.824+0000 7f3c53fff640 1 -- 172.21.15.42:0/1068448438 >> v2:172.21.15.154:3300/0 conn(0x7f3c540a1db0 msgr2=0x7f3c5413b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:00:46.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:46.824+0000 7f3c53fff640 1 -- 172.21.15.42:0/1068448438 >> v2:172.21.15.154:3300/0 conn(0x7f3c540a1db0 msgr2=0x7f3c5413b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:00:46.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:46.824+0000 7f3c53fff640 1 --2- 172.21.15.42:0/1068448438 >> v2:172.21.15.154:3300/0 conn(0x7f3c540a1db0 0x7f3c5413b8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:00:46.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:46.824+0000 7f3c52ffd640 1 --2- 172.21.15.42:0/1068448438 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3c5409e070 0x7f3c540a1870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T04:00:46.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:46.824+0000 7f3c53fff640 1 --2- 172.21.15.42:0/1068448438 >> v2:172.21.15.154:3300/0 conn(0x7f3c540a1db0 0x7f3c5413b8f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T04:00:47.632 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:47.628+0000 7f3c537fe640 1 --2- 172.21.15.42:0/1068448438 >> v2:172.21.15.42:3300/0 conn(0x7f3c540a2f10 0x7f3c540a1330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:00:47.632 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:47.628+0000 7f3c537fe640 10 monclient(hunting): get_auth_request con 0x7f3c540a2f10 auth_method 0 2024-09-16T04:00:47.632 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:47.628+0000 7f3c537fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:00:47.633 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:47.628+0000 7f3c537fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:00:47.633 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:47.628+0000 7f3c537fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:00:47.633 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:47.628+0000 7f3c52ffd640 1 --2- 172.21.15.42:0/1068448438 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3c5409e070 0x7f3c540a1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:00:47.633 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:47.628+0000 7f3c52ffd640 10 monclient(hunting): get_auth_request con 0x7f3c5409e070 auth_method 0 2024-09-16T04:00:47.633 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:47.628+0000 7f3c52ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:00:47.633 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:47.628+0000 7f3c52ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:00:47.633 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:47.628+0000 7f3c52ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:00:47.633 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:47.628+0000 7f3c537fe640 1 -- 172.21.15.42:0/1068448438 >> v2:172.21.15.42:3300/0 conn(0x7f3c540a2f10 msgr2=0x7f3c540a1330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:00:47.633 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:47.628+0000 7f3c537fe640 1 -- 172.21.15.42:0/1068448438 >> v2:172.21.15.42:3300/0 conn(0x7f3c540a2f10 msgr2=0x7f3c540a1330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:00:47.633 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:47.628+0000 7f3c537fe640 1 --2- 172.21.15.42:0/1068448438 >> v2:172.21.15.42:3300/0 conn(0x7f3c540a2f10 0x7f3c540a1330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:00:47.633 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:47.628+0000 7f3c53fff640 1 --2- 172.21.15.42:0/1068448438 >> v2:172.21.15.154:3300/0 conn(0x7f3c540a1db0 0x7f3c5413b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:00:47.633 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:47.628+0000 7f3c52ffd640 1 -- 172.21.15.42:0/1068448438 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3c5409e070 msgr2=0x7f3c540a1870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:00:47.634 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:47.628+0000 7f3c52ffd640 1 -- 172.21.15.42:0/1068448438 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3c5409e070 msgr2=0x7f3c540a1870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:00:47.634 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:47.628+0000 7f3c52ffd640 1 --2- 172.21.15.42:0/1068448438 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3c5409e070 0x7f3c540a1870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:00:47.634 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:47.628+0000 7f3c53fff640 10 monclient(hunting): get_auth_request con 0x7f3c540a1db0 auth_method 0 2024-09-16T04:00:47.634 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:47.628+0000 7f3c53fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:00:47.634 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:47.628+0000 7f3c53fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:00:47.634 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:47.628+0000 7f3c53fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:00:47.634 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:47.628+0000 7f3c537fe640 1 --2- 172.21.15.42:0/1068448438 >> v2:172.21.15.42:3300/0 conn(0x7f3c540a2f10 0x7f3c540a1330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T04:00:47.634 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:47.628+0000 7f3c52ffd640 1 --2- 172.21.15.42:0/1068448438 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3c5409e070 0x7f3c540a1870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T04:00:47.634 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:47.628+0000 7f3c53fff640 1 -- 172.21.15.42:0/1068448438 >> v2:172.21.15.154:3300/0 conn(0x7f3c540a1db0 msgr2=0x7f3c5413b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T04:00:47.634 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:47.628+0000 7f3c53fff640 1 -- 172.21.15.42:0/1068448438 >> v2:172.21.15.154:3300/0 conn(0x7f3c540a1db0 msgr2=0x7f3c5413b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:00:47.634 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:47.628+0000 7f3c53fff640 1 --2- 172.21.15.42:0/1068448438 >> v2:172.21.15.154:3300/0 conn(0x7f3c540a1db0 0x7f3c5413b8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:00:47.634 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:47.628+0000 7f3c53fff640 1 --2- 172.21.15.42:0/1068448438 >> v2:172.21.15.154:3300/0 conn(0x7f3c540a1db0 0x7f3c5413b8f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T04:00:49.225 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:49.220+0000 7f3c517fa640 10 monclient(hunting): tick 2024-09-16T04:00:49.225 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:49.220+0000 7f3c517fa640 10 monclient(hunting): _check_auth_tickets 2024-09-16T04:00:49.225 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:49.220+0000 7f3c517fa640 1 monclient(hunting): continuing hunt 2024-09-16T04:00:49.225 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:49.220+0000 7f3c517fa640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T04:00:49.225 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:49.220+0000 7f3c517fa640 1 -- 172.21.15.42:0/1068448438 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3c5409e070 msgr2=0x7f3c540a1870 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:00:49.225 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:49.220+0000 7f3c517fa640 1 --2- 172.21.15.42:0/1068448438 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3c5409e070 0x7f3c540a1870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:00:49.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:49.220+0000 7f3c517fa640 1 -- 172.21.15.42:0/1068448438 >> v2:172.21.15.154:3300/0 conn(0x7f3c540a1db0 msgr2=0x7f3c5413b8f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:00:49.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:49.220+0000 7f3c517fa640 1 --2- 172.21.15.42:0/1068448438 >> v2:172.21.15.154:3300/0 conn(0x7f3c540a1db0 0x7f3c5413b8f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:00:49.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:49.220+0000 7f3c517fa640 1 -- 172.21.15.42:0/1068448438 >> v2:172.21.15.42:3300/0 conn(0x7f3c540a2f10 msgr2=0x7f3c540a1330 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:00:49.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:49.220+0000 7f3c517fa640 1 --2- 172.21.15.42:0/1068448438 >> v2:172.21.15.42:3300/0 conn(0x7f3c540a2f10 0x7f3c540a1330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:00:49.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:49.220+0000 7f3c517fa640 10 monclient: _add_conns ranks=[0,1] 2024-09-16T04:00:49.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:49.220+0000 7f3c517fa640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3c400036a0 addr v1:172.21.15.42:6789/0 2024-09-16T04:00:49.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:49.220+0000 7f3c517fa640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3c400073b0 addr v1:172.21.15.154:6789/0 2024-09-16T04:00:49.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:49.220+0000 7f3c517fa640 1 -- 172.21.15.42:0/1068448438 --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3c5409e600 con 0x7f3c400036a0 2024-09-16T04:00:49.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:49.220+0000 7f3c517fa640 1 -- 172.21.15.42:0/1068448438 --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3c4000c130 con 0x7f3c400073b0 2024-09-16T04:00:49.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:49.220+0000 7f3c517fa640 10 monclient(hunting): _renew_subs 2024-09-16T04:00:50.537 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.528+0000 7f3c527fc640 1 -- 172.21.15.42:0/1068448438 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3181143378 0 0) 0x7f3c5409e600 con 0x7f3c400036a0 2024-09-16T04:00:50.537 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.528+0000 7f3c527fc640 10 monclient(hunting): _negotiate 2024-09-16T04:00:50.537 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.528+0000 7f3c527fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:00:50.537 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.528+0000 7f3c527fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:00:50.537 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.528+0000 7f3c527fc640 10 monclient(hunting): my global_id is 825243 2024-09-16T04:00:50.538 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.528+0000 7f3c527fc640 1 -- 172.21.15.42:0/1068448438 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3c34003530 con 0x7f3c400036a0 2024-09-16T04:00:50.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.532+0000 7f3c527fc640 1 -- 172.21.15.42:0/1068448438 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1103244522 0 0) 0x7f3c34003530 con 0x7f3c400036a0 2024-09-16T04:00:50.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.532+0000 7f3c527fc640 1 -- 172.21.15.42:0/1068448438 >> v1:172.21.15.154:6789/0 conn(0x7f3c400073b0 legacy=0x7f3c40009870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:00:50.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.532+0000 7f3c527fc640 10 monclient: _finish_hunting 0 2024-09-16T04:00:50.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.532+0000 7f3c527fc640 1 monclient: found mon.noname-a-legacy 2024-09-16T04:00:50.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.532+0000 7f3c527fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:00:50.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.532+0000 7f3c527fc640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.42:6789/0 2024-09-16T04:00:50.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.532+0000 7f3c527fc640 1 -- 172.21.15.42:0/1068448438 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c4000d3d0 con 0x7f3c400036a0 2024-09-16T04:00:50.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.532+0000 7f3c527fc640 10 monclient: _finish_auth 0 2024-09-16T04:00:50.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.532+0000 7f3c527fc640 10 monclient: _check_auth_tickets 2024-09-16T04:00:50.541 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.532+0000 7f3c527fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:00:50.541 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.532+0000 7f3c527fc640 1 -- 172.21.15.42:0/1068448438 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2177413734 0 0) 0x7f3c4400f040 con 0x7f3c400036a0 2024-09-16T04:00:50.541 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.532+0000 7f3c527fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:00:50.541 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.532+0000 7f3c527fc640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T04:00:50.541 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.532+0000 7f3c527fc640 10 monclient: dump: 2024-09-16T04:00:50.541 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:00:50.541 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:00:50.541 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:00:50.541 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:00:50.541 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:00:50.541 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:00:50.541 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:00:50.541 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:00:50.542 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:00:50.542 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:00:50.542 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.532+0000 7f3c527fc640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] but i'm connected to v1:172.21.15.42:6789/0, reconnecting 2024-09-16T04:00:50.542 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.532+0000 7f3c527fc640 10 monclient: _reopen_session rank -1 2024-09-16T04:00:50.542 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.532+0000 7f3c527fc640 1 -- 172.21.15.42:0/1068448438 >> v1:172.21.15.42:6789/0 conn(0x7f3c400036a0 legacy=0x7f3c40005ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:00:50.542 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.532+0000 7f3c527fc640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:00:50.542 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.532+0000 7f3c53fff640 1 -- 172.21.15.42:0/1068448438 reap_dead start 2024-09-16T04:00:50.542 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.532+0000 7f3c527fc640 1 --2- 172.21.15.42:0/1068448438 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3c34003a90 0x7f3c34003e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:00:50.542 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.532+0000 7f3c527fc640 10 monclient(hunting): picked mon.a con 0x7f3c34003a90 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:00:50.542 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.532+0000 7f3c527fc640 1 --2- 172.21.15.42:0/1068448438 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3c34004810 0x7f3c34004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:00:50.542 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.532+0000 7f3c527fc640 10 monclient(hunting): picked mon.c con 0x7f3c34004810 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:00:50.542 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.532+0000 7f3c537fe640 1 --2- 172.21.15.42:0/1068448438 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3c34003a90 0x7f3c34003e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:00:50.542 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.532+0000 7f3c52ffd640 1 --2- 172.21.15.42:0/1068448438 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3c34004810 0x7f3c34004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:00:50.543 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.532+0000 7f3c527fc640 1 --2- 172.21.15.42:0/1068448438 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3c340099c0 0x7f3c3400bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:00:50.543 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.532+0000 7f3c527fc640 10 monclient(hunting): picked mon.b con 0x7f3c340099c0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:00:50.544 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.532+0000 7f3c527fc640 10 monclient(hunting): start opening mon connection 2024-09-16T04:00:50.544 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.532+0000 7f3c527fc640 1 -- 172.21.15.42:0/1068448438 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f3c3400c3a0 con 0x7f3c34003a90 2024-09-16T04:00:50.544 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.532+0000 7f3c527fc640 10 monclient(hunting): start opening mon connection 2024-09-16T04:00:50.544 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.536+0000 7f3c527fc640 1 -- 172.21.15.42:0/1068448438 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f3c3400c550 con 0x7f3c340099c0 2024-09-16T04:00:50.544 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.536+0000 7f3c53fff640 1 --2- 172.21.15.42:0/1068448438 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3c340099c0 0x7f3c3400bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:00:50.544 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.536+0000 7f3c527fc640 10 monclient(hunting): start opening mon connection 2024-09-16T04:00:50.544 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.536+0000 7f3c527fc640 1 -- 172.21.15.42:0/1068448438 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f3c3400c750 con 0x7f3c34004810 2024-09-16T04:00:50.544 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.536+0000 7f3c527fc640 10 monclient(hunting): _renew_subs 2024-09-16T04:00:50.545 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.536+0000 7f3c527fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:00:50.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.536+0000 7f3c537fe640 10 monclient(hunting): get_auth_request con 0x7f3c34003a90 auth_method 0 2024-09-16T04:00:50.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.536+0000 7f3c537fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:00:50.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.536+0000 7f3c537fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:00:50.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.536+0000 7f3c537fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:00:50.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.536+0000 7f3c537fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:00:50.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.536+0000 7f3c537fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:00:50.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.536+0000 7f3c537fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:00:50.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.536+0000 7f3c537fe640 10 monclient(hunting): handle_auth_done global_id 825243 payload 293 2024-09-16T04:00:50.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.536+0000 7f3c537fe640 1 -- 172.21.15.42:0/1068448438 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3c34004810 msgr2=0x7f3c34004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:00:50.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.536+0000 7f3c537fe640 1 --2- 172.21.15.42:0/1068448438 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3c34004810 0x7f3c34004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:00:50.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.536+0000 7f3c537fe640 1 -- 172.21.15.42:0/1068448438 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3c340099c0 msgr2=0x7f3c3400bdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:00:50.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.536+0000 7f3c537fe640 1 --2- 172.21.15.42:0/1068448438 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3c340099c0 0x7f3c3400bdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:00:50.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.536+0000 7f3c537fe640 10 monclient: _finish_hunting 0 2024-09-16T04:00:50.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.536+0000 7f3c537fe640 1 monclient: found mon.a 2024-09-16T04:00:50.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.536+0000 7f3c537fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:00:50.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.536+0000 7f3c537fe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:00:50.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.536+0000 7f3c537fe640 1 -- 172.21.15.42:0/1068448438 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c34003530 con 0x7f3c34003a90 2024-09-16T04:00:50.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.536+0000 7f3c537fe640 10 monclient: _finish_auth 0 2024-09-16T04:00:50.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.536+0000 7f3c537fe640 10 monclient: _check_auth_tickets 2024-09-16T04:00:50.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.536+0000 7f3c537fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:00:50.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.540+0000 7f3c52ffd640 10 monclient: get_auth_request con 0x7f3c34004810 auth_method 0 2024-09-16T04:00:50.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.540+0000 7f3c52ffd640 1 --2- 172.21.15.42:0/1068448438 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3c34004810 0x7f3c34004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T04:00:50.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.540+0000 7f3c53fff640 10 monclient: get_auth_request con 0x7f3c340099c0 auth_method 0 2024-09-16T04:00:50.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.540+0000 7f3c53fff640 1 --2- 172.21.15.42:0/1068448438 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3c340099c0 0x7f3c3400bdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T04:00:50.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.540+0000 7f3c59b13640 5 monclient: authenticate success, global_id 825243 2024-09-16T04:00:50.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.540+0000 7f3c59b13640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:00:50.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.540+0000 7f3c537fe640 1 --2- 172.21.15.42:0/1068448438 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3c34003a90 0x7f3c34003e90 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f3c4400efd0 tx=0x7f3c4401d890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=460d07c2eabe55d2 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:00:50.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.540+0000 7f3c527fc640 1 -- 172.21.15.42:0/1068448438 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c4400ec20 con 0x7f3c34003a90 2024-09-16T04:00:50.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.540+0000 7f3c527fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:00:50.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.540+0000 7f3c527fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:00:50.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.540+0000 7f3c527fc640 10 monclient: dump: 2024-09-16T04:00:50.549 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:00:50.549 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:00:50.549 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:00:50.549 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:00:50.549 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:00:50.549 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:00:50.549 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:00:50.549 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:00:50.549 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:00:50.549 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:00:50.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.540+0000 7f3c527fc640 1 -- 172.21.15.42:0/1068448438 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3c440a7710 con 0x7f3c34003a90 2024-09-16T04:00:50.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.540+0000 7f3c527fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:00:50.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.540+0000 7f3c527fc640 1 -- 172.21.15.42:0/1068448438 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c440a7b00 con 0x7f3c34003a90 2024-09-16T04:00:50.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.540+0000 7f3c527fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:00:50.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.540+0000 7f3c527fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:00:50.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.540+0000 7f3c527fc640 10 monclient: dump: 2024-09-16T04:00:50.550 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:00:50.550 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:00:50.550 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:00:50.550 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:00:50.550 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:00:50.550 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:00:50.550 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:00:50.550 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:00:50.550 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:00:50.550 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:00:50.551 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.540+0000 7f3c59b13640 10 monclient: get_monmap_and_config success 2024-09-16T04:00:50.551 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.540+0000 7f3c59b13640 4 set_mon_vals no callback set 2024-09-16T04:00:50.551 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.544+0000 7f3c59b13640 10 monclient: shutdown 2024-09-16T04:00:50.551 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.544+0000 7f3c59b13640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:00:50.551 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.544+0000 7f3c59b13640 1 -- 172.21.15.42:0/1068448438 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3c34003a90 msgr2=0x7f3c34003e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:00:50.552 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.544+0000 7f3c59b13640 1 --2- 172.21.15.42:0/1068448438 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3c34003a90 0x7f3c34003e90 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f3c4400efd0 tx=0x7f3c4401d890 comp rx=0 tx=0).stop 2024-09-16T04:00:50.552 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.544+0000 7f3c59b13640 1 -- 172.21.15.42:0/1068448438 shutdown_connections 2024-09-16T04:00:50.552 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.544+0000 7f3c59b13640 1 --2- 172.21.15.42:0/1068448438 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3c340099c0 0x7f3c3400bdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:00:50.552 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.544+0000 7f3c59b13640 1 --2- 172.21.15.42:0/1068448438 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3c34004810 0x7f3c34004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:00:50.552 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.544+0000 7f3c59b13640 1 --2- 172.21.15.42:0/1068448438 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3c34003a90 0x7f3c34003e90 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:00:50.552 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.544+0000 7f3c59b13640 1 -- 172.21.15.42:0/1068448438 >> 172.21.15.42:0/1068448438 conn(0x7f3c540a2420 msgr2=0x7f3c540a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:00:50.552 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.544+0000 7f3c59b13640 1 -- 172.21.15.42:0/1068448438 shutdown_connections 2024-09-16T04:00:50.552 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.544+0000 7f3c59b13640 1 -- 172.21.15.42:0/1068448438 wait complete. 2024-09-16T04:00:50.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.544+0000 7f3c59b13640 10 monclient: build_initial_monmap 2024-09-16T04:00:50.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.544+0000 7f3c59b13640 10 monclient: monmap: 2024-09-16T04:00:50.553 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:00:50.553 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:00:50.553 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:00:50.553 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:00:50.553 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:00:50.553 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:00:50.553 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:00:50.553 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:00:50.554 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:00:50.554 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:00:50.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.544+0000 7f3c59b13640 1 librados: starting msgr at 2024-09-16T04:00:50.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.544+0000 7f3c59b13640 1 librados: starting objecter 2024-09-16T04:00:50.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.544+0000 7f3c59b13640 1 Processor -- start 2024-09-16T04:00:50.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.544+0000 7f3c59b13640 1 -- start start 2024-09-16T04:00:50.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.544+0000 7f3c59b13640 1 librados: setting wanted keys 2024-09-16T04:00:50.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.544+0000 7f3c59b13640 1 librados: calling monclient init 2024-09-16T04:00:50.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.544+0000 7f3c59b13640 10 monclient: init 2024-09-16T04:00:50.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.548+0000 7f3c59b13640 10 monclient: _reopen_session rank -1 2024-09-16T04:00:50.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.548+0000 7f3c59b13640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:00:50.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.548+0000 7f3c59b13640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3c34003a90 0x7f3c541d2b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:00:50.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.548+0000 7f3c59b13640 10 monclient(hunting): picked mon.noname-c con 0x7f3c34003a90 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:00:50.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.548+0000 7f3c59b13640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3c34004810 0x7f3c541d30c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:00:50.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.548+0000 7f3c59b13640 10 monclient(hunting): picked mon.noname-b con 0x7f3c34004810 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:00:50.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.548+0000 7f3c59b13640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3c340099c0 0x7f3c541281d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:00:50.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.548+0000 7f3c59b13640 10 monclient(hunting): picked mon.noname-a con 0x7f3c340099c0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:00:50.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.548+0000 7f3c59b13640 10 monclient(hunting): start opening mon connection 2024-09-16T04:00:50.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.548+0000 7f3c59b13640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f3c5413ba70 con 0x7f3c340099c0 2024-09-16T04:00:50.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.548+0000 7f3c59b13640 10 monclient(hunting): start opening mon connection 2024-09-16T04:00:50.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.548+0000 7f3c59b13640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f3c5413b8f0 con 0x7f3c34004810 2024-09-16T04:00:50.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.548+0000 7f3c59b13640 10 monclient(hunting): start opening mon connection 2024-09-16T04:00:50.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.548+0000 7f3c59b13640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f3c5413bbf0 con 0x7f3c34003a90 2024-09-16T04:00:50.560 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.548+0000 7f3c59b13640 10 monclient(hunting): _renew_subs 2024-09-16T04:00:50.560 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.548+0000 7f3c59b13640 10 monclient(hunting): authenticate will time out at 2097.520640s 2024-09-16T04:00:50.560 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.548+0000 7f3c537fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3c34003a90 0x7f3c541d2b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:00:50.560 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.548+0000 7f3c537fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3c34003a90 0x7f3c541d2b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:53246/0 (socket says 172.21.15.42:53246) 2024-09-16T04:00:50.560 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.548+0000 7f3c53fff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3c340099c0 0x7f3c541281d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:00:50.560 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.548+0000 7f3c537fe640 1 -- 172.21.15.42:0/3051361223 learned_addr learned my addr 172.21.15.42:0/3051361223 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:00:50.560 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.548+0000 7f3c537fe640 10 monclient(hunting): get_auth_request con 0x7f3c34003a90 auth_method 0 2024-09-16T04:00:50.560 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.548+0000 7f3c537fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:00:50.560 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.548+0000 7f3c537fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:00:50.560 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.548+0000 7f3c537fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:00:50.560 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.548+0000 7f3c53fff640 10 monclient(hunting): get_auth_request con 0x7f3c340099c0 auth_method 0 2024-09-16T04:00:50.560 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.548+0000 7f3c53fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:00:50.560 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.548+0000 7f3c53fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:00:50.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.548+0000 7f3c53fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:00:50.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.548+0000 7f3c52ffd640 1 --2- 172.21.15.42:0/3051361223 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3c34004810 0x7f3c541d30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:00:50.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.548+0000 7f3c537fe640 1 -- 172.21.15.42:0/3051361223 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3c34003a90 msgr2=0x7f3c541d2b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-16T04:00:50.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.548+0000 7f3c537fe640 1 -- 172.21.15.42:0/3051361223 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3c34003a90 msgr2=0x7f3c541d2b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:00:50.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.548+0000 7f3c537fe640 1 --2- 172.21.15.42:0/3051361223 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3c34003a90 0x7f3c541d2b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:00:50.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.548+0000 7f3c52ffd640 10 monclient(hunting): get_auth_request con 0x7f3c34004810 auth_method 0 2024-09-16T04:00:50.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.548+0000 7f3c52ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:00:50.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.548+0000 7f3c52ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:00:50.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.548+0000 7f3c52ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:00:50.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.548+0000 7f3c537fe640 1 --2- 172.21.15.42:0/3051361223 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3c34003a90 0x7f3c541d2b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:00:50.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.548+0000 7f3c53fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:00:50.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.548+0000 7f3c53fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:00:50.562 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.548+0000 7f3c53fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:00:50.562 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.548+0000 7f3c52ffd640 1 -- 172.21.15.42:0/3051361223 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3c34004810 msgr2=0x7f3c541d30c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-16T04:00:50.562 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.548+0000 7f3c52ffd640 1 -- 172.21.15.42:0/3051361223 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3c34004810 msgr2=0x7f3c541d30c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:00:50.562 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.548+0000 7f3c52ffd640 1 --2- 172.21.15.42:0/3051361223 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3c34004810 0x7f3c541d30c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:00:50.562 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.548+0000 7f3c52ffd640 1 --2- 172.21.15.42:0/3051361223 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3c34004810 0x7f3c541d30c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:00:50.562 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.548+0000 7f3c53fff640 10 monclient(hunting): handle_auth_done global_id 825246 payload 824 2024-09-16T04:00:50.562 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.548+0000 7f3c53fff640 1 -- 172.21.15.42:0/3051361223 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3c34003a90 msgr2=0x7f3c541d2b80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:00:50.562 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.548+0000 7f3c53fff640 1 --2- 172.21.15.42:0/3051361223 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3c34003a90 0x7f3c541d2b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:00:50.562 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.548+0000 7f3c53fff640 1 -- 172.21.15.42:0/3051361223 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3c34004810 msgr2=0x7f3c541d30c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:00:50.562 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.548+0000 7f3c53fff640 1 --2- 172.21.15.42:0/3051361223 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3c34004810 0x7f3c541d30c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:00:50.562 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.548+0000 7f3c53fff640 10 monclient: _finish_hunting 0 2024-09-16T04:00:50.562 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.548+0000 7f3c53fff640 1 monclient: found mon.noname-a 2024-09-16T04:00:50.562 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.548+0000 7f3c53fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:00:50.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.548+0000 7f3c53fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:00:50.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.548+0000 7f3c53fff640 1 -- 172.21.15.42:0/3051361223 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c5412bba0 con 0x7f3c340099c0 2024-09-16T04:00:50.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.548+0000 7f3c53fff640 1 --2- 172.21.15.42:0/3051361223 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3c340099c0 0x7f3c541281d0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f3c540a37d0 tx=0x7f3c4c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:00:50.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.548+0000 7f3c4bfff640 10 client.?.objecter ms_handle_connect 0x7f3c340099c0 2024-09-16T04:00:50.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.548+0000 7f3c4bfff640 10 client.?.objecter resend_mon_ops 2024-09-16T04:00:50.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.548+0000 7f3c4bfff640 1 -- 172.21.15.42:0/3051361223 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c4c01b070 con 0x7f3c340099c0 2024-09-16T04:00:50.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.548+0000 7f3c4bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:00:50.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.548+0000 7f3c4bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:00:50.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.548+0000 7f3c4bfff640 10 monclient: dump: 2024-09-16T04:00:50.563 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:00:50.563 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:00:50.563 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:00:50.563 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:00:50.563 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:00:50.564 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:00:50.564 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:00:50.564 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:00:50.564 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:00:50.564 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:00:50.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.548+0000 7f3c4bfff640 10 monclient: _finish_auth 0 2024-09-16T04:00:50.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.548+0000 7f3c4bfff640 10 monclient: _check_auth_tickets 2024-09-16T04:00:50.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.548+0000 7f3c4bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:00:50.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.548+0000 7f3c4bfff640 1 -- 172.21.15.42:0/3051361223 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3c4c013a10 con 0x7f3c340099c0 2024-09-16T04:00:50.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.548+0000 7f3c4bfff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:00:50.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.548+0000 7f3c4bfff640 1 -- 172.21.15.42:0/3051361223 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c4c013de0 con 0x7f3c340099c0 2024-09-16T04:00:50.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.548+0000 7f3c59b13640 5 monclient: authenticate success, global_id 825246 2024-09-16T04:00:50.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.548+0000 7f3c59b13640 10 monclient: _renew_subs 2024-09-16T04:00:50.565 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.548+0000 7f3c59b13640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:00:50.565 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.548+0000 7f3c59b13640 1 -- 172.21.15.42:0/3051361223 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c5412a310 con 0x7f3c340099c0 2024-09-16T04:00:50.565 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.548+0000 7f3c51ffb640 4 set_mon_vals no callback set 2024-09-16T04:00:50.565 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.548+0000 7f3c4bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:00:50.565 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.548+0000 7f3c4bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:00:50.565 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.548+0000 7f3c4bfff640 10 monclient: dump: 2024-09-16T04:00:50.565 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:00:50.565 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:00:50.565 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:00:50.565 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:00:50.565 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:00:50.565 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:00:50.565 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:00:50.565 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:00:50.566 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:00:50.566 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:00:50.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.548+0000 7f3c59b13640 10 client.825246.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:00:50.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.548+0000 7f3c59b13640 10 monclient: _renew_subs 2024-09-16T04:00:50.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.548+0000 7f3c59b13640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:00:50.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.548+0000 7f3c59b13640 1 -- 172.21.15.42:0/3051361223 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c5412be80 con 0x7f3c340099c0 2024-09-16T04:00:50.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.548+0000 7f3c59b13640 1 librados: init done 2024-09-16T04:00:50.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.548+0000 7f3c59b13640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:00:50.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.548+0000 7f3c59b13640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:00:50.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.548+0000 7f3c59b13640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:00:50.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.548+0000 7f3c4bfff640 1 -- 172.21.15.42:0/3051361223 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f3c4c01e070 con 0x7f3c340099c0 2024-09-16T04:00:50.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.548+0000 7f3c59b13640 1 -- 172.21.15.42:0/3051361223 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3c540a4480 con 0x7f3c340099c0 2024-09-16T04:00:50.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.548+0000 7f3c4bfff640 1 --2- 172.21.15.42:0/3051361223 >> v2:172.21.15.154:6812/254845342 conn(0x7f3c40038360 0x7f3c4003a820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:00:50.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.548+0000 7f3c4bfff640 1 -- 172.21.15.42:0/3051361223 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(563..563 src has 451..563) ==== 6216+0+0 (secure 0 0 0) 0x7f3c4c049d20 con 0x7f3c340099c0 2024-09-16T04:00:50.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.548+0000 7f3c4bfff640 10 client.825246.objecter ms_dispatch 0x7f3c54000ba0 osd_map(563..563 src has 451..563) 2024-09-16T04:00:50.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.548+0000 7f3c4bfff640 3 client.825246.objecter handle_osd_map got epochs [563,563] > 0 2024-09-16T04:00:50.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.548+0000 7f3c4bfff640 3 client.825246.objecter handle_osd_map decoding full epoch 563 2024-09-16T04:00:50.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.548+0000 7f3c4bfff640 20 client.825246.objecter dump_active .. 0 homeless 2024-09-16T04:00:50.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.548+0000 7f3c537fe640 1 --2- 172.21.15.42:0/3051361223 >> v2:172.21.15.154:6812/254845342 conn(0x7f3c40038360 0x7f3c4003a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:00:50.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.548+0000 7f3c537fe640 10 monclient: get_auth_request con 0x7f3c40038360 auth_method 0 2024-09-16T04:00:50.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.552+0000 7f3c4bfff640 10 client.825246.objecter ms_handle_connect 0x7f3c40038360 2024-09-16T04:00:50.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.552+0000 7f3c537fe640 1 --2- 172.21.15.42:0/3051361223 >> v2:172.21.15.154:6812/254845342 conn(0x7f3c40038360 0x7f3c4003a820 secure :-1 s=READY pgs=3744 cs=0 l=1 rev1=1 crypto rx=0x7f3c440101c0 tx=0x7f3c44004740 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:00:50.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.556+0000 7f3c4bfff640 1 -- 172.21.15.42:0/3051361223 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f3c4c025030 con 0x7f3c340099c0 2024-09-16T04:00:50.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.556+0000 7f3c4bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:00:50.568 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.556+0000 7f3c4bfff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:00:50.786 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.780+0000 7f3c59b13640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:00:50.786 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.780+0000 7f3c59b13640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:00:50.786 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.780+0000 7f3c59b13640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:00:50.787 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.780+0000 7f3c59b13640 1 -- 172.21.15.42:0/3051361223 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f3c54069040 con 0x7f3c340099c0 2024-09-16T04:00:50.787 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.780+0000 7f3c4bfff640 1 -- 172.21.15.42:0/3051361223 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1555 (secure 0 0 0) 0x7f3c4c010030 con 0x7f3c340099c0 2024-09-16T04:00:50.787 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.780+0000 7f3c4bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:00:50.787 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.780+0000 7f3c4bfff640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:00:50.787 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":164,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:00:50.789 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.780+0000 7f3c59b13640 10 librados: watch_flush enter 2024-09-16T04:00:50.789 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.780+0000 7f3c59b13640 10 librados: watch_flush exit 2024-09-16T04:00:50.789 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.780+0000 7f3c59b13640 20 client.825246.objecter shutdown clearing up homeless session... 2024-09-16T04:00:50.789 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.780+0000 7f3c59b13640 10 client.825246.objecter successfully canceled tick 2024-09-16T04:00:50.789 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.780+0000 7f3c59b13640 1 -- 172.21.15.42:0/3051361223 >> v2:172.21.15.154:6812/254845342 conn(0x7f3c40038360 msgr2=0x7f3c4003a820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:00:50.790 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.780+0000 7f3c59b13640 1 --2- 172.21.15.42:0/3051361223 >> v2:172.21.15.154:6812/254845342 conn(0x7f3c40038360 0x7f3c4003a820 secure :-1 s=READY pgs=3744 cs=0 l=1 rev1=1 crypto rx=0x7f3c440101c0 tx=0x7f3c44004740 comp rx=0 tx=0).stop 2024-09-16T04:00:50.790 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.780+0000 7f3c59b13640 10 monclient: shutdown 2024-09-16T04:00:50.790 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.780+0000 7f3c59b13640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:00:50.790 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.780+0000 7f3c59b13640 1 -- 172.21.15.42:0/3051361223 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3c340099c0 msgr2=0x7f3c541281d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:00:50.790 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.780+0000 7f3c59b13640 1 --2- 172.21.15.42:0/3051361223 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3c340099c0 0x7f3c541281d0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f3c540a37d0 tx=0x7f3c4c004750 comp rx=0 tx=0).stop 2024-09-16T04:00:50.790 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.780+0000 7f3c59b13640 1 -- 172.21.15.42:0/3051361223 shutdown_connections 2024-09-16T04:00:50.790 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.780+0000 7f3c59b13640 1 --2- 172.21.15.42:0/3051361223 >> v2:172.21.15.154:6812/254845342 conn(0x7f3c40038360 0x7f3c4003a820 unknown :-1 s=CLOSED pgs=3744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:00:50.790 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.780+0000 7f3c59b13640 1 --2- 172.21.15.42:0/3051361223 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3c340099c0 0x7f3c541281d0 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:00:50.790 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.780+0000 7f3c59b13640 1 --2- 172.21.15.42:0/3051361223 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3c34004810 0x7f3c541d30c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:00:50.790 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.780+0000 7f3c59b13640 1 --2- 172.21.15.42:0/3051361223 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3c34003a90 0x7f3c541d2b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:00:50.790 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.780+0000 7f3c59b13640 1 -- 172.21.15.42:0/3051361223 >> 172.21.15.42:0/3051361223 conn(0x7f3c540a2420 msgr2=0x7f3c5406f910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:00:50.790 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.784+0000 7f3c59b13640 1 -- 172.21.15.42:0/3051361223 shutdown_connections 2024-09-16T04:00:50.791 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.784+0000 7f3c59b13640 1 -- 172.21.15.42:0/3051361223 wait complete. 2024-09-16T04:00:50.791 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:50.784+0000 7f3c59b13640 1 librados: shutdown 2024-09-16T04:00:53.806 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:00:53.939 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.932+0000 7f2a25774640 10 monclient: get_monmap_and_config 2024-09-16T04:00:53.940 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.932+0000 7f2a25774640 10 monclient: build_initial_monmap 2024-09-16T04:00:53.940 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.932+0000 7f2a25774640 10 monclient: monmap: 2024-09-16T04:00:53.940 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:00:53.940 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:00:53.940 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:00:53.937996+0000 2024-09-16T04:00:53.940 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:00:53.937996+0000 2024-09-16T04:00:53.940 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:00:53.940 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:00:53.940 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:00:53.940 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:00:53.941 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:00:53.941 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:00:53.941 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:00:53.941 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:00:53.943 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.932+0000 7f2a25774640 1 Processor -- start 2024-09-16T04:00:53.943 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.932+0000 7f2a25774640 1 -- start start 2024-09-16T04:00:53.943 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.932+0000 7f2a25774640 10 monclient: init 2024-09-16T04:00:53.943 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.932+0000 7f2a25774640 10 monclient: _reopen_session rank -1 2024-09-16T04:00:53.943 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.932+0000 7f2a25774640 10 monclient: _add_conns ranks=[2,3,4,0,1] 2024-09-16T04:00:53.943 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.932+0000 7f2a25774640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2a200a39b0 0x7f2a20133b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:00:53.943 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.932+0000 7f2a25774640 10 monclient(hunting): picked mon.noname-b con 0x7f2a200a39b0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:00:53.943 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.932+0000 7f2a25774640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f2a200a3d90 0x7f2a20137000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:00:53.943 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.932+0000 7f2a25774640 10 monclient(hunting): picked mon.noname-a con 0x7f2a200a3d90 addr v2:172.21.15.42:3300/0 2024-09-16T04:00:53.943 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.932+0000 7f2a25774640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f2a20137540 0x7f2a20137950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:00:53.943 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.932+0000 7f2a25774640 10 monclient(hunting): picked mon.noname-c con 0x7f2a20137540 addr v2:172.21.15.154:3300/0 2024-09-16T04:00:53.943 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.932+0000 7f2a25774640 10 monclient(hunting): start opening mon connection 2024-09-16T04:00:53.944 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.932+0000 7f2a25774640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f2a20137e90 con 0x7f2a200a3d90 2024-09-16T04:00:53.944 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.932+0000 7f2a25774640 10 monclient(hunting): start opening mon connection 2024-09-16T04:00:53.944 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.932+0000 7f2a25774640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f2a2009c140 con 0x7f2a20137540 2024-09-16T04:00:53.944 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.932+0000 7f2a25774640 10 monclient(hunting): start opening mon connection 2024-09-16T04:00:53.944 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.932+0000 7f2a25774640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f2a2009c420 con 0x7f2a200a39b0 2024-09-16T04:00:53.944 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.932+0000 7f2a25774640 10 monclient(hunting): _renew_subs 2024-09-16T04:00:53.944 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.932+0000 7f2a25774640 10 monclient(hunting): authenticate will time out at 2100.907850s 2024-09-16T04:00:53.944 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.932+0000 7f2a1e7fc640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f2a200a3d90 0x7f2a20137000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:00:53.944 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.932+0000 7f2a1effd640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2a200a39b0 0x7f2a20133b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:00:53.944 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.936+0000 7f2a1e7fc640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f2a200a3d90 0x7f2a20137000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:56410/0 (socket says 172.21.15.42:56410) 2024-09-16T04:00:53.944 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.936+0000 7f2a1effd640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2a200a39b0 0x7f2a20133b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:53268/0 (socket says 172.21.15.42:53268) 2024-09-16T04:00:53.945 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.936+0000 7f2a1e7fc640 1 -- 172.21.15.42:0/1483206687 learned_addr learned my addr 172.21.15.42:0/1483206687 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:00:53.945 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.936+0000 7f2a1e7fc640 10 monclient(hunting): get_auth_request con 0x7f2a200a3d90 auth_method 0 2024-09-16T04:00:53.945 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.936+0000 7f2a1e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:00:53.945 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.936+0000 7f2a1e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:00:53.945 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.936+0000 7f2a1e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:00:53.945 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.936+0000 7f2a1effd640 10 monclient(hunting): get_auth_request con 0x7f2a200a39b0 auth_method 0 2024-09-16T04:00:53.945 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.936+0000 7f2a1effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:00:53.945 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.936+0000 7f2a1effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:00:53.946 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.936+0000 7f2a1effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:00:53.946 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.936+0000 7f2a1e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:00:53.946 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.936+0000 7f2a1e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:00:53.946 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.936+0000 7f2a1e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:00:53.946 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.936+0000 7f2a1effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:00:53.946 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.936+0000 7f2a1effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:00:53.946 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.936+0000 7f2a1effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:00:53.946 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.936+0000 7f2a1e7fc640 10 monclient(hunting): handle_auth_done global_id 825249 payload 274 2024-09-16T04:00:53.946 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.936+0000 7f2a1e7fc640 1 -- 172.21.15.42:0/1483206687 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2a200a39b0 msgr2=0x7f2a20133b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:00:53.946 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.936+0000 7f2a1e7fc640 1 --2- 172.21.15.42:0/1483206687 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2a200a39b0 0x7f2a20133b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:00:53.946 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.936+0000 7f2a1f7fe640 1 --2- 172.21.15.42:0/1483206687 >> v2:172.21.15.154:3300/0 conn(0x7f2a20137540 0x7f2a20137950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:00:53.946 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.936+0000 7f2a1e7fc640 1 -- 172.21.15.42:0/1483206687 >> v2:172.21.15.154:3300/0 conn(0x7f2a20137540 msgr2=0x7f2a20137950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:00:53.946 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.936+0000 7f2a1e7fc640 1 --2- 172.21.15.42:0/1483206687 >> v2:172.21.15.154:3300/0 conn(0x7f2a20137540 0x7f2a20137950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:00:53.947 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.936+0000 7f2a1e7fc640 10 monclient: _finish_hunting 0 2024-09-16T04:00:53.947 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.936+0000 7f2a1e7fc640 1 monclient: found mon.noname-a 2024-09-16T04:00:53.947 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.936+0000 7f2a1e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:00:53.947 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.936+0000 7f2a1e7fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:00:53.947 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.936+0000 7f2a1e7fc640 1 -- 172.21.15.42:0/1483206687 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a2006dd60 con 0x7f2a200a3d90 2024-09-16T04:00:53.947 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.936+0000 7f2a1f7fe640 10 monclient: get_auth_request con 0x7f2a20137540 auth_method 0 2024-09-16T04:00:53.947 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.936+0000 7f2a1effd640 1 --2- 172.21.15.42:0/1483206687 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2a200a39b0 0x7f2a20133b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:00:53.948 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.936+0000 7f2a1f7fe640 1 --2- 172.21.15.42:0/1483206687 >> v2:172.21.15.154:3300/0 conn(0x7f2a20137540 0x7f2a20137950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T04:00:53.954 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.936+0000 7f2a1e7fc640 1 --2- 172.21.15.42:0/1483206687 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2a200a3d90 0x7f2a20137000 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f2a200a56c0 tx=0x7f2a1001c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68262d72244c2c07 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:00:53.954 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.936+0000 7f2a1dffb640 1 -- 172.21.15.42:0/1483206687 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a100a5530 con 0x7f2a200a3d90 2024-09-16T04:00:53.954 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.936+0000 7f2a1dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:00:53.954 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.936+0000 7f2a1dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:00:53.954 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.936+0000 7f2a1dffb640 10 monclient: dump: 2024-09-16T04:00:53.955 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:00:53.955 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:00:53.955 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:00:53.955 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:00:53.955 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:00:53.955 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:00:53.955 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:00:53.955 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:00:53.955 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:00:53.955 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:00:53.955 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.936+0000 7f2a1dffb640 10 monclient: _finish_auth 0 2024-09-16T04:00:53.955 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.936+0000 7f2a1dffb640 10 monclient: _check_auth_tickets 2024-09-16T04:00:53.955 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.936+0000 7f2a1dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:00:53.955 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.936+0000 7f2a25774640 5 monclient: authenticate success, global_id 825249 2024-09-16T04:00:53.956 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.936+0000 7f2a25774640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:00:53.956 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.936+0000 7f2a1dffb640 1 -- 172.21.15.42:0/1483206687 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a100a56d0 con 0x7f2a200a3d90 2024-09-16T04:00:53.956 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.936+0000 7f2a1dffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:00:53.956 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.936+0000 7f2a25774640 10 monclient: get_monmap_and_config success 2024-09-16T04:00:53.956 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.936+0000 7f2a1dffb640 1 -- 172.21.15.42:0/1483206687 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a100a5aa0 con 0x7f2a200a3d90 2024-09-16T04:00:53.956 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.936+0000 7f2a1dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:00:53.956 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.936+0000 7f2a25774640 4 set_mon_vals no callback set 2024-09-16T04:00:53.956 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.936+0000 7f2a1dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:00:53.956 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.936+0000 7f2a1dffb640 10 monclient: dump: 2024-09-16T04:00:53.956 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:00:53.956 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:00:53.956 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:00:53.957 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:00:53.957 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:00:53.957 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:00:53.957 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:00:53.957 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:00:53.957 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:00:53.957 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:00:53.957 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.936+0000 7f2a25774640 10 monclient: shutdown 2024-09-16T04:00:53.957 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.936+0000 7f2a25774640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:00:53.957 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.936+0000 7f2a25774640 1 -- 172.21.15.42:0/1483206687 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2a200a3d90 msgr2=0x7f2a20137000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:00:53.957 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.936+0000 7f2a25774640 1 --2- 172.21.15.42:0/1483206687 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2a200a3d90 0x7f2a20137000 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f2a200a56c0 tx=0x7f2a1001c8b0 comp rx=0 tx=0).stop 2024-09-16T04:00:53.957 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.936+0000 7f2a25774640 1 -- 172.21.15.42:0/1483206687 shutdown_connections 2024-09-16T04:00:53.957 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.936+0000 7f2a25774640 1 --2- 172.21.15.42:0/1483206687 >> v2:172.21.15.154:3300/0 conn(0x7f2a20137540 0x7f2a20137950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:00:53.957 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.936+0000 7f2a25774640 1 --2- 172.21.15.42:0/1483206687 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2a200a3d90 0x7f2a20137000 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:00:53.958 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.936+0000 7f2a25774640 1 --2- 172.21.15.42:0/1483206687 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2a200a39b0 0x7f2a20133b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:00:53.958 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.936+0000 7f2a25774640 1 -- 172.21.15.42:0/1483206687 >> 172.21.15.42:0/1483206687 conn(0x7f2a200a9fb0 msgr2=0x7f2a200aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:00:53.958 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.936+0000 7f2a25774640 1 -- 172.21.15.42:0/1483206687 shutdown_connections 2024-09-16T04:00:53.958 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.936+0000 7f2a25774640 1 -- 172.21.15.42:0/1483206687 wait complete. 2024-09-16T04:00:53.958 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.936+0000 7f2a25774640 10 monclient: build_initial_monmap 2024-09-16T04:00:53.958 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.936+0000 7f2a25774640 10 monclient: monmap: 2024-09-16T04:00:53.959 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:00:53.959 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:00:53.959 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:00:53.959 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:00:53.959 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:00:53.959 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:00:53.959 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:00:53.959 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:00:53.959 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:00:53.959 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:00:53.959 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.940+0000 7f2a25774640 1 librados: starting msgr at 2024-09-16T04:00:53.960 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.940+0000 7f2a25774640 1 librados: starting objecter 2024-09-16T04:00:53.960 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.940+0000 7f2a25774640 1 Processor -- start 2024-09-16T04:00:53.960 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.940+0000 7f2a25774640 1 -- start start 2024-09-16T04:00:53.960 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.940+0000 7f2a25774640 1 librados: setting wanted keys 2024-09-16T04:00:53.960 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.940+0000 7f2a25774640 1 librados: calling monclient init 2024-09-16T04:00:53.960 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.940+0000 7f2a25774640 10 monclient: init 2024-09-16T04:00:53.960 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.940+0000 7f2a25774640 10 monclient: _reopen_session rank -1 2024-09-16T04:00:53.960 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.940+0000 7f2a25774640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:00:53.960 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.940+0000 7f2a25774640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2a200a39b0 0x7f2a200a2d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:00:53.960 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.940+0000 7f2a25774640 10 monclient(hunting): picked mon.noname-a con 0x7f2a200a39b0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:00:53.960 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.940+0000 7f2a25774640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2a200a3d90 0x7f2a200a32d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:00:53.960 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.940+0000 7f2a25774640 10 monclient(hunting): picked mon.noname-c con 0x7f2a200a3d90 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:00:53.960 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.940+0000 7f2a25774640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2a20137540 0x7f2a2013b280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:00:53.960 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.940+0000 7f2a25774640 10 monclient(hunting): picked mon.noname-b con 0x7f2a20137540 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:00:53.962 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.940+0000 7f2a25774640 10 monclient(hunting): start opening mon connection 2024-09-16T04:00:53.962 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.940+0000 7f2a25774640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f2a20140dc0 con 0x7f2a200a39b0 2024-09-16T04:00:53.962 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.940+0000 7f2a25774640 10 monclient(hunting): start opening mon connection 2024-09-16T04:00:53.962 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.940+0000 7f2a25774640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f2a20140f40 con 0x7f2a20137540 2024-09-16T04:00:53.962 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.940+0000 7f2a25774640 10 monclient(hunting): start opening mon connection 2024-09-16T04:00:53.962 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.940+0000 7f2a25774640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f2a201410c0 con 0x7f2a200a3d90 2024-09-16T04:00:53.962 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.940+0000 7f2a25774640 10 monclient(hunting): _renew_subs 2024-09-16T04:00:53.963 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.940+0000 7f2a25774640 10 monclient(hunting): authenticate will time out at 2100.912896s 2024-09-16T04:00:53.964 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.940+0000 7f2a1effd640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2a200a39b0 0x7f2a200a2d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:00:53.964 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.940+0000 7f2a1e7fc640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2a200a3d90 0x7f2a200a32d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:00:53.964 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.940+0000 7f2a1effd640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2a200a39b0 0x7f2a200a2d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:56426/0 (socket says 172.21.15.42:56426) 2024-09-16T04:00:53.964 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.940+0000 7f2a1effd640 1 -- 172.21.15.42:0/2745306532 learned_addr learned my addr 172.21.15.42:0/2745306532 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:00:53.964 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.940+0000 7f2a1effd640 10 monclient(hunting): get_auth_request con 0x7f2a200a39b0 auth_method 0 2024-09-16T04:00:53.964 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.940+0000 7f2a1effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:00:53.964 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.940+0000 7f2a1effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:00:53.964 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.940+0000 7f2a1effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:00:53.964 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.940+0000 7f2a1e7fc640 10 monclient(hunting): get_auth_request con 0x7f2a200a3d90 auth_method 0 2024-09-16T04:00:53.964 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.940+0000 7f2a1e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:00:53.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.940+0000 7f2a1e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:00:53.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.940+0000 7f2a1e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:00:53.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.940+0000 7f2a1effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:00:53.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.940+0000 7f2a1effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:00:53.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.940+0000 7f2a1effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:00:53.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.940+0000 7f2a1e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:00:53.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.940+0000 7f2a1e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:00:53.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.940+0000 7f2a1e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:00:53.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.940+0000 7f2a1f7fe640 1 --2- 172.21.15.42:0/2745306532 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2a20137540 0x7f2a2013b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:00:53.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.940+0000 7f2a1f7fe640 10 monclient(hunting): get_auth_request con 0x7f2a20137540 auth_method 0 2024-09-16T04:00:53.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.940+0000 7f2a1f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:00:53.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.940+0000 7f2a1f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:00:53.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.940+0000 7f2a1f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:00:53.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.940+0000 7f2a1effd640 10 monclient(hunting): handle_auth_done global_id 825252 payload 824 2024-09-16T04:00:53.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.940+0000 7f2a1effd640 1 -- 172.21.15.42:0/2745306532 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2a200a3d90 msgr2=0x7f2a200a32d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:00:53.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.940+0000 7f2a1effd640 1 --2- 172.21.15.42:0/2745306532 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2a200a3d90 0x7f2a200a32d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:00:53.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.940+0000 7f2a1effd640 1 -- 172.21.15.42:0/2745306532 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2a20137540 msgr2=0x7f2a2013b280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:00:53.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.940+0000 7f2a1effd640 1 --2- 172.21.15.42:0/2745306532 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2a20137540 0x7f2a2013b280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:00:53.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.940+0000 7f2a1effd640 10 monclient: _finish_hunting 0 2024-09-16T04:00:53.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.940+0000 7f2a1effd640 1 monclient: found mon.noname-a 2024-09-16T04:00:53.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.940+0000 7f2a1effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:00:53.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.940+0000 7f2a1effd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:00:53.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.940+0000 7f2a1effd640 1 -- 172.21.15.42:0/2745306532 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a2013d3c0 con 0x7f2a200a39b0 2024-09-16T04:00:53.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.940+0000 7f2a1e7fc640 1 --2- 172.21.15.42:0/2745306532 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2a200a3d90 0x7f2a200a32d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:00:53.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.940+0000 7f2a1effd640 1 --2- 172.21.15.42:0/2745306532 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2a200a39b0 0x7f2a200a2d90 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f2a0800e980 tx=0x7f2a0800ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:00:53.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.940+0000 7f2a177fe640 10 client.?.objecter ms_handle_connect 0x7f2a200a39b0 2024-09-16T04:00:53.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.940+0000 7f2a177fe640 10 client.?.objecter resend_mon_ops 2024-09-16T04:00:53.967 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.940+0000 7f2a177fe640 1 -- 172.21.15.42:0/2745306532 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a08017070 con 0x7f2a200a39b0 2024-09-16T04:00:53.967 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.940+0000 7f2a177fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:00:53.967 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.940+0000 7f2a177fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:00:53.967 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.940+0000 7f2a177fe640 10 monclient: dump: 2024-09-16T04:00:53.967 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:00:53.967 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:00:53.967 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:00:53.967 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:00:53.967 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:00:53.967 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:00:53.967 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:00:53.967 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:00:53.967 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:00:53.967 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:00:53.968 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.940+0000 7f2a177fe640 10 monclient: _finish_auth 0 2024-09-16T04:00:53.968 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.940+0000 7f2a177fe640 10 monclient: _check_auth_tickets 2024-09-16T04:00:53.968 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.940+0000 7f2a177fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:00:53.968 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.940+0000 7f2a177fe640 1 -- 172.21.15.42:0/2745306532 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a0800ca00 con 0x7f2a200a39b0 2024-09-16T04:00:53.968 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.940+0000 7f2a25774640 5 monclient: authenticate success, global_id 825252 2024-09-16T04:00:53.968 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.940+0000 7f2a25774640 10 monclient: _renew_subs 2024-09-16T04:00:53.968 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.940+0000 7f2a25774640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:00:53.968 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.940+0000 7f2a25774640 1 -- 172.21.15.42:0/2745306532 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a2009d820 con 0x7f2a200a39b0 2024-09-16T04:00:53.968 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.940+0000 7f2a177fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:00:53.968 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.940+0000 7f2a177fe640 1 -- 172.21.15.42:0/2745306532 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a0800cdd0 con 0x7f2a200a39b0 2024-09-16T04:00:53.968 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.940+0000 7f2a177fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:00:53.968 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.940+0000 7f2a177fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:00:53.968 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.940+0000 7f2a177fe640 10 monclient: dump: 2024-09-16T04:00:53.968 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:00:53.969 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:00:53.969 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:00:53.969 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:00:53.969 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:00:53.969 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:00:53.969 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:00:53.969 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:00:53.969 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:00:53.969 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:00:53.969 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.940+0000 7f2a25774640 10 client.825252.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:00:53.969 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.940+0000 7f2a25774640 10 monclient: _renew_subs 2024-09-16T04:00:53.969 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.940+0000 7f2a25774640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:00:53.969 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.940+0000 7f2a1cff9640 4 set_mon_vals no callback set 2024-09-16T04:00:53.969 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.940+0000 7f2a25774640 1 -- 172.21.15.42:0/2745306532 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a200a4e00 con 0x7f2a200a39b0 2024-09-16T04:00:53.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.940+0000 7f2a25774640 1 librados: init done 2024-09-16T04:00:53.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.940+0000 7f2a177fe640 1 -- 172.21.15.42:0/2745306532 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f2a0801a070 con 0x7f2a200a39b0 2024-09-16T04:00:53.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.940+0000 7f2a25774640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:00:53.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.940+0000 7f2a25774640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:00:53.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.940+0000 7f2a25774640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:00:53.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.940+0000 7f2a25774640 1 -- 172.21.15.42:0/2745306532 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2a2006f680 con 0x7f2a200a39b0 2024-09-16T04:00:53.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.940+0000 7f2a177fe640 1 --2- 172.21.15.42:0/2745306532 >> v2:172.21.15.154:6812/254845342 conn(0x7f2a00037810 0x7f2a00039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:00:53.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.940+0000 7f2a177fe640 1 -- 172.21.15.42:0/2745306532 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(566..566 src has 451..566) ==== 6224+0+0 (secure 0 0 0) 0x7f2a080490c0 con 0x7f2a200a39b0 2024-09-16T04:00:53.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.940+0000 7f2a177fe640 10 client.825252.objecter ms_dispatch 0x7f2a20000ba0 osd_map(566..566 src has 451..566) 2024-09-16T04:00:53.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.940+0000 7f2a177fe640 3 client.825252.objecter handle_osd_map got epochs [566,566] > 0 2024-09-16T04:00:53.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.940+0000 7f2a177fe640 3 client.825252.objecter handle_osd_map decoding full epoch 566 2024-09-16T04:00:53.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.940+0000 7f2a177fe640 20 client.825252.objecter dump_active .. 0 homeless 2024-09-16T04:00:53.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.940+0000 7f2a1e7fc640 1 --2- 172.21.15.42:0/2745306532 >> v2:172.21.15.154:6812/254845342 conn(0x7f2a00037810 0x7f2a00039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:00:53.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.940+0000 7f2a1e7fc640 10 monclient: get_auth_request con 0x7f2a00037810 auth_method 0 2024-09-16T04:00:53.971 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.944+0000 7f2a1e7fc640 1 --2- 172.21.15.42:0/2745306532 >> v2:172.21.15.154:6812/254845342 conn(0x7f2a00037810 0x7f2a00039cd0 secure :-1 s=READY pgs=3745 cs=0 l=1 rev1=1 crypto rx=0x7f2a100ae000 tx=0x7f2a100a7070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:00:53.971 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.944+0000 7f2a177fe640 10 client.825252.objecter ms_handle_connect 0x7f2a00037810 2024-09-16T04:00:53.971 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.944+0000 7f2a177fe640 1 -- 172.21.15.42:0/2745306532 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f2a08012df0 con 0x7f2a200a39b0 2024-09-16T04:00:53.971 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.944+0000 7f2a177fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:00:53.971 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:53.944+0000 7f2a177fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:00:54.181 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.172+0000 7f2a25774640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:00:54.181 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.172+0000 7f2a25774640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:00:54.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.172+0000 7f2a25774640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:00:54.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.172+0000 7f2a25774640 1 -- 172.21.15.42:0/2745306532 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f2a20137000 con 0x7f2a200a39b0 2024-09-16T04:00:54.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.176+0000 7f2a177fe640 1 -- 172.21.15.42:0/2745306532 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7f2a08013990 con 0x7f2a200a39b0 2024-09-16T04:00:54.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.176+0000 7f2a177fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:00:54.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.176+0000 7f2a177fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:00:54.182 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":168,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:00:54.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.176+0000 7f2a25774640 10 librados: watch_flush enter 2024-09-16T04:00:54.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.176+0000 7f2a25774640 10 librados: watch_flush exit 2024-09-16T04:00:54.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.176+0000 7f2a25774640 20 client.825252.objecter shutdown clearing up homeless session... 2024-09-16T04:00:54.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.176+0000 7f2a25774640 10 client.825252.objecter successfully canceled tick 2024-09-16T04:00:54.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.176+0000 7f2a25774640 1 -- 172.21.15.42:0/2745306532 >> v2:172.21.15.154:6812/254845342 conn(0x7f2a00037810 msgr2=0x7f2a00039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:00:54.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.176+0000 7f2a25774640 1 --2- 172.21.15.42:0/2745306532 >> v2:172.21.15.154:6812/254845342 conn(0x7f2a00037810 0x7f2a00039cd0 secure :-1 s=READY pgs=3745 cs=0 l=1 rev1=1 crypto rx=0x7f2a100ae000 tx=0x7f2a100a7070 comp rx=0 tx=0).stop 2024-09-16T04:00:54.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.176+0000 7f2a25774640 10 monclient: shutdown 2024-09-16T04:00:54.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.176+0000 7f2a25774640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:00:54.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.176+0000 7f2a25774640 1 -- 172.21.15.42:0/2745306532 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2a200a39b0 msgr2=0x7f2a200a2d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:00:54.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.176+0000 7f2a25774640 1 --2- 172.21.15.42:0/2745306532 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2a200a39b0 0x7f2a200a2d90 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f2a0800e980 tx=0x7f2a0800ee40 comp rx=0 tx=0).stop 2024-09-16T04:00:54.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.176+0000 7f2a25774640 1 -- 172.21.15.42:0/2745306532 shutdown_connections 2024-09-16T04:00:54.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.176+0000 7f2a25774640 1 --2- 172.21.15.42:0/2745306532 >> v2:172.21.15.154:6812/254845342 conn(0x7f2a00037810 0x7f2a00039cd0 unknown :-1 s=CLOSED pgs=3745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:00:54.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.176+0000 7f2a25774640 1 --2- 172.21.15.42:0/2745306532 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2a20137540 0x7f2a2013b280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:00:54.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.176+0000 7f2a25774640 1 --2- 172.21.15.42:0/2745306532 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2a200a3d90 0x7f2a200a32d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:00:54.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.176+0000 7f2a25774640 1 --2- 172.21.15.42:0/2745306532 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2a200a39b0 0x7f2a200a2d90 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:00:54.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.176+0000 7f2a25774640 1 -- 172.21.15.42:0/2745306532 >> 172.21.15.42:0/2745306532 conn(0x7f2a200a9fb0 msgr2=0x7f2a200a8000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:00:54.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.176+0000 7f2a25774640 1 -- 172.21.15.42:0/2745306532 shutdown_connections 2024-09-16T04:00:54.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.176+0000 7f2a25774640 1 -- 172.21.15.42:0/2745306532 wait complete. 2024-09-16T04:00:54.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.176+0000 7f2a25774640 1 librados: shutdown 2024-09-16T04:00:54.200 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T04:00:54.200 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T04:00:54.200 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T04:00:54.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.320+0000 7fe0af2be640 10 monclient: get_monmap_and_config 2024-09-16T04:00:54.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.320+0000 7fe0af2be640 10 monclient: build_initial_monmap 2024-09-16T04:00:54.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.320+0000 7fe0af2be640 10 monclient: monmap: 2024-09-16T04:00:54.326 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:00:54.326 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:00:54.326 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:00:54.324493+0000 2024-09-16T04:00:54.326 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:00:54.324493+0000 2024-09-16T04:00:54.326 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:00:54.326 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:00:54.326 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:00:54.327 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:00:54.327 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:00:54.327 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:00:54.327 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:00:54.327 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:00:54.329 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.320+0000 7fe0af2be640 1 Processor -- start 2024-09-16T04:00:54.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.320+0000 7fe0af2be640 1 -- start start 2024-09-16T04:00:54.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.320+0000 7fe0af2be640 10 monclient: init 2024-09-16T04:00:54.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.320+0000 7fe0af2be640 10 monclient: _reopen_session rank -1 2024-09-16T04:00:54.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.320+0000 7fe0af2be640 10 monclient: _add_conns ranks=[4,3,1,2,0] 2024-09-16T04:00:54.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.320+0000 7fe0af2be640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fe0a80a1330 0x7fe0a80a1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:00:54.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.320+0000 7fe0af2be640 10 monclient(hunting): picked mon.noname-c con 0x7fe0a80a1330 addr v2:172.21.15.154:3300/0 2024-09-16T04:00:54.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.320+0000 7fe0af2be640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe0a809c920 0x7fe0a80a1c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:00:54.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.320+0000 7fe0af2be640 10 monclient(hunting): picked mon.noname-a con 0x7fe0a809c920 addr v2:172.21.15.42:3300/0 2024-09-16T04:00:54.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.320+0000 7fe0af2be640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe0a8072d40 addr v1:172.21.15.154:6789/0 2024-09-16T04:00:54.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.320+0000 7fe0af2be640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe0a8072350 con 0x7fe0a8072d40 2024-09-16T04:00:54.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.320+0000 7fe0af2be640 10 monclient(hunting): start opening mon connection 2024-09-16T04:00:54.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.320+0000 7fe0af2be640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7fe0a8072b50 con 0x7fe0a809c920 2024-09-16T04:00:54.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.320+0000 7fe0af2be640 10 monclient(hunting): start opening mon connection 2024-09-16T04:00:54.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.320+0000 7fe0af2be640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fe0a80a2190 con 0x7fe0a80a1330 2024-09-16T04:00:54.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.320+0000 7fe0af2be640 10 monclient(hunting): _renew_subs 2024-09-16T04:00:54.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.320+0000 7fe0af2be640 10 monclient(hunting): authenticate will time out at 2101.293917s 2024-09-16T04:00:54.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.320+0000 7fe0ac859640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe0a809c920 0x7fe0a80a1c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:00:54.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.320+0000 7fe0ac859640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe0a809c920 0x7fe0a80a1c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:44048/0 (socket says 172.21.15.42:44048) 2024-09-16T04:00:54.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.320+0000 7fe0ac859640 1 -- 172.21.15.42:0/197907406 learned_addr learned my addr 172.21.15.42:0/197907406 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:00:54.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.320+0000 7fe0ac859640 10 monclient(hunting): get_auth_request con 0x7fe0a809c920 auth_method 0 2024-09-16T04:00:54.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.320+0000 7fe0ac859640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:00:54.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.320+0000 7fe0ac859640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:00:54.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.320+0000 7fe0ac859640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:00:54.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.320+0000 7fe0ac859640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:00:54.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.320+0000 7fe0ac859640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:00:54.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.320+0000 7fe0ac859640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:00:54.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.320+0000 7fe0ad05a640 1 --2- 172.21.15.42:0/197907406 >> v2:172.21.15.154:3300/0 conn(0x7fe0a80a1330 0x7fe0a80a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:00:54.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.320+0000 7fe0ad05a640 10 monclient(hunting): get_auth_request con 0x7fe0a80a1330 auth_method 0 2024-09-16T04:00:54.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.320+0000 7fe0ad05a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:00:54.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.320+0000 7fe0ad05a640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:00:54.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.320+0000 7fe0ad05a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:00:54.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.320+0000 7fe0ac859640 10 monclient(hunting): handle_auth_done global_id 825255 payload 274 2024-09-16T04:00:54.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.320+0000 7fe0ac859640 1 -- 172.21.15.42:0/197907406 >> v2:172.21.15.154:3300/0 conn(0x7fe0a80a1330 msgr2=0x7fe0a80a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:00:54.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.320+0000 7fe0ac859640 1 --2- 172.21.15.42:0/197907406 >> v2:172.21.15.154:3300/0 conn(0x7fe0a80a1330 0x7fe0a80a1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:00:54.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.320+0000 7fe0ac859640 1 -- 172.21.15.42:0/197907406 >> v1:172.21.15.154:6789/0 conn(0x7fe0a8072d40 legacy=0x7fe0a8137120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:00:54.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.320+0000 7fe0ac859640 10 monclient: _finish_hunting 0 2024-09-16T04:00:54.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.320+0000 7fe0ac859640 1 monclient: found mon.noname-a 2024-09-16T04:00:54.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.320+0000 7fe0ac859640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:00:54.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.320+0000 7fe0ac859640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:00:54.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.320+0000 7fe0ac859640 1 -- 172.21.15.42:0/197907406 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0a809bc60 con 0x7fe0a809c920 2024-09-16T04:00:54.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.324+0000 7fe0ac859640 1 --2- 172.21.15.42:0/197907406 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe0a809c920 0x7fe0a80a1c50 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fe0a80a2340 tx=0x7fe09801c430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8d24a2468bcd669 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:00:54.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.324+0000 7fe09ffff640 1 -- 172.21.15.42:0/197907406 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0980b0070 con 0x7fe0a809c920 2024-09-16T04:00:54.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.324+0000 7fe09ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:00:54.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.324+0000 7fe09ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:00:54.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.324+0000 7fe09ffff640 10 monclient: dump: 2024-09-16T04:00:54.340 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:00:54.340 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:00:54.340 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:00:54.340 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:00:54.340 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:00:54.340 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:00:54.340 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:00:54.340 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:00:54.340 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:00:54.340 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:00:54.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.324+0000 7fe09ffff640 10 monclient: _finish_auth 0 2024-09-16T04:00:54.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.324+0000 7fe09ffff640 10 monclient: _check_auth_tickets 2024-09-16T04:00:54.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.324+0000 7fe09ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:00:54.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.324+0000 7fe09ffff640 1 -- 172.21.15.42:0/197907406 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe0980ab600 con 0x7fe0a809c920 2024-09-16T04:00:54.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.324+0000 7fe0af2be640 5 monclient: authenticate success, global_id 825255 2024-09-16T04:00:54.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.324+0000 7fe0af2be640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:00:54.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.324+0000 7fe09ffff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:00:54.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.324+0000 7fe0af2be640 10 monclient: get_monmap_and_config success 2024-09-16T04:00:54.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.324+0000 7fe0af2be640 4 set_mon_vals no callback set 2024-09-16T04:00:54.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.324+0000 7fe09ffff640 1 -- 172.21.15.42:0/197907406 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0980ab9d0 con 0x7fe0a809c920 2024-09-16T04:00:54.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.324+0000 7fe09ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:00:54.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.324+0000 7fe09ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:00:54.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.324+0000 7fe09ffff640 10 monclient: dump: 2024-09-16T04:00:54.341 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:00:54.341 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:00:54.342 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:00:54.342 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:00:54.342 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:00:54.342 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:00:54.342 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:00:54.342 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:00:54.342 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:00:54.342 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:00:54.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.324+0000 7fe0af2be640 10 monclient: shutdown 2024-09-16T04:00:54.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.324+0000 7fe0af2be640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:00:54.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.324+0000 7fe0af2be640 1 -- 172.21.15.42:0/197907406 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe0a809c920 msgr2=0x7fe0a80a1c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:00:54.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.324+0000 7fe0af2be640 1 --2- 172.21.15.42:0/197907406 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe0a809c920 0x7fe0a80a1c50 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fe0a80a2340 tx=0x7fe09801c430 comp rx=0 tx=0).stop 2024-09-16T04:00:54.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.324+0000 7fe0af2be640 1 -- 172.21.15.42:0/197907406 shutdown_connections 2024-09-16T04:00:54.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.324+0000 7fe0af2be640 1 --2- 172.21.15.42:0/197907406 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe0a809c920 0x7fe0a80a1c50 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:00:54.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.324+0000 7fe0af2be640 1 --2- 172.21.15.42:0/197907406 >> v2:172.21.15.154:3300/0 conn(0x7fe0a80a1330 0x7fe0a80a1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:00:54.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.324+0000 7fe0af2be640 1 -- 172.21.15.42:0/197907406 >> 172.21.15.42:0/197907406 conn(0x7fe0a80a2420 msgr2=0x7fe0a80a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:00:54.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.324+0000 7fe0af2be640 1 -- 172.21.15.42:0/197907406 shutdown_connections 2024-09-16T04:00:54.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.324+0000 7fe0af2be640 1 -- 172.21.15.42:0/197907406 wait complete. 2024-09-16T04:00:54.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.324+0000 7fe0af2be640 10 monclient: build_initial_monmap 2024-09-16T04:00:54.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.324+0000 7fe0af2be640 10 monclient: monmap: 2024-09-16T04:00:54.343 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:00:54.343 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:00:54.343 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:00:54.343 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:00:54.343 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:00:54.343 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:00:54.343 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:00:54.344 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:00:54.344 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:00:54.344 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:00:54.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.324+0000 7fe0af2be640 1 librados: starting msgr at 2024-09-16T04:00:54.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.324+0000 7fe0af2be640 1 librados: starting objecter 2024-09-16T04:00:54.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.324+0000 7fe0af2be640 1 Processor -- start 2024-09-16T04:00:54.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.324+0000 7fe0af2be640 1 -- start start 2024-09-16T04:00:54.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.324+0000 7fe0af2be640 1 librados: setting wanted keys 2024-09-16T04:00:54.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.324+0000 7fe0af2be640 1 librados: calling monclient init 2024-09-16T04:00:54.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.324+0000 7fe0af2be640 10 monclient: init 2024-09-16T04:00:54.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.324+0000 7fe0af2be640 10 monclient: _reopen_session rank -1 2024-09-16T04:00:54.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.328+0000 7fe0af2be640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:00:54.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.328+0000 7fe0af2be640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe0a8072d40 0x7fe0a81cf0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:00:54.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.328+0000 7fe0af2be640 10 monclient(hunting): picked mon.noname-a con 0x7fe0a8072d40 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:00:54.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.328+0000 7fe0af2be640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe0a809c920 0x7fe0a8133550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:00:54.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.328+0000 7fe0af2be640 10 monclient(hunting): picked mon.noname-c con 0x7fe0a809c920 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:00:54.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.328+0000 7fe0af2be640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe0a80a1330 0x7fe0a813d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:00:54.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.328+0000 7fe0af2be640 10 monclient(hunting): picked mon.noname-b con 0x7fe0a80a1330 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:00:54.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.328+0000 7fe0af2be640 10 monclient(hunting): start opening mon connection 2024-09-16T04:00:54.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.328+0000 7fe0af2be640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fe0a8141a60 con 0x7fe0a8072d40 2024-09-16T04:00:54.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.328+0000 7fe0af2be640 10 monclient(hunting): start opening mon connection 2024-09-16T04:00:54.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.328+0000 7fe0af2be640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe0a81418e0 con 0x7fe0a80a1330 2024-09-16T04:00:54.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.328+0000 7fe0af2be640 10 monclient(hunting): start opening mon connection 2024-09-16T04:00:54.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.328+0000 7fe0af2be640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fe0a8141be0 con 0x7fe0a809c920 2024-09-16T04:00:54.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.328+0000 7fe0af2be640 10 monclient(hunting): _renew_subs 2024-09-16T04:00:54.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.328+0000 7fe0af2be640 10 monclient(hunting): authenticate will time out at 2101.300098s 2024-09-16T04:00:54.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.328+0000 7fe0ad05a640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe0a8072d40 0x7fe0a81cf0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:00:54.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.328+0000 7fe0ac859640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe0a809c920 0x7fe0a8133550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:00:54.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.328+0000 7fe0ad05a640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe0a8072d40 0x7fe0a81cf0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:44054/0 (socket says 172.21.15.42:44054) 2024-09-16T04:00:54.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.328+0000 7fe0ad05a640 1 -- 172.21.15.42:0/1641112605 learned_addr learned my addr 172.21.15.42:0/1641112605 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:00:54.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.328+0000 7fe0ad05a640 10 monclient(hunting): get_auth_request con 0x7fe0a8072d40 auth_method 0 2024-09-16T04:00:54.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.328+0000 7fe0ad05a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:00:54.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.328+0000 7fe0ad05a640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:00:54.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.328+0000 7fe0ad05a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:00:54.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.328+0000 7fe0ac859640 10 monclient(hunting): get_auth_request con 0x7fe0a809c920 auth_method 0 2024-09-16T04:00:54.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.328+0000 7fe0ac859640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:00:54.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.328+0000 7fe0ac859640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:00:54.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.328+0000 7fe0ac859640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:00:54.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.328+0000 7fe0ad05a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:00:54.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.328+0000 7fe0ad05a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:00:54.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.328+0000 7fe0ad05a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:00:54.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.328+0000 7fe0ac859640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:00:54.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.328+0000 7fe0ac859640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:00:54.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.328+0000 7fe0ac859640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:00:54.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.328+0000 7fe0ad85b640 1 --2- 172.21.15.42:0/1641112605 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe0a80a1330 0x7fe0a813d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:00:54.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.328+0000 7fe0ad05a640 10 monclient(hunting): handle_auth_done global_id 825258 payload 824 2024-09-16T04:00:54.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.328+0000 7fe0ad05a640 1 -- 172.21.15.42:0/1641112605 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe0a809c920 msgr2=0x7fe0a8133550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:00:54.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.328+0000 7fe0ad05a640 1 --2- 172.21.15.42:0/1641112605 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe0a809c920 0x7fe0a8133550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:00:54.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.328+0000 7fe0ad05a640 1 -- 172.21.15.42:0/1641112605 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe0a80a1330 msgr2=0x7fe0a813d1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:00:54.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.328+0000 7fe0ad05a640 1 --2- 172.21.15.42:0/1641112605 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe0a80a1330 0x7fe0a813d1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:00:54.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.328+0000 7fe0ad05a640 10 monclient: _finish_hunting 0 2024-09-16T04:00:54.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.328+0000 7fe0ad05a640 1 monclient: found mon.noname-a 2024-09-16T04:00:54.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.328+0000 7fe0ad05a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:00:54.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.328+0000 7fe0ad05a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:00:54.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.328+0000 7fe0ad05a640 1 -- 172.21.15.42:0/1641112605 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0a81cffc0 con 0x7fe0a8072d40 2024-09-16T04:00:54.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.328+0000 7fe0ad85b640 10 monclient: get_auth_request con 0x7fe0a80a1330 auth_method 0 2024-09-16T04:00:54.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.328+0000 7fe0ad85b640 1 --2- 172.21.15.42:0/1641112605 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe0a80a1330 0x7fe0a813d1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T04:00:54.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.328+0000 7fe0ac859640 1 --2- 172.21.15.42:0/1641112605 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe0a809c920 0x7fe0a8133550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:00:54.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.328+0000 7fe0ad05a640 1 --2- 172.21.15.42:0/1641112605 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe0a8072d40 0x7fe0a81cf0b0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fe0a400d800 tx=0x7fe0a400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:00:54.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.328+0000 7fe09dffb640 10 client.?.objecter ms_handle_connect 0x7fe0a8072d40 2024-09-16T04:00:54.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.328+0000 7fe09dffb640 10 client.?.objecter resend_mon_ops 2024-09-16T04:00:54.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.328+0000 7fe09dffb640 1 -- 172.21.15.42:0/1641112605 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0a400b6c0 con 0x7fe0a8072d40 2024-09-16T04:00:54.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.328+0000 7fe09dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:00:54.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.328+0000 7fe09dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:00:54.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.328+0000 7fe09dffb640 10 monclient: dump: 2024-09-16T04:00:54.348 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:00:54.348 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:00:54.349 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:00:54.349 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:00:54.349 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:00:54.349 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:00:54.349 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:00:54.349 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:00:54.349 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:00:54.349 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:00:54.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.328+0000 7fe09dffb640 10 monclient: _finish_auth 0 2024-09-16T04:00:54.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.328+0000 7fe09dffb640 10 monclient: _check_auth_tickets 2024-09-16T04:00:54.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.328+0000 7fe09dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:00:54.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.328+0000 7fe09dffb640 1 -- 172.21.15.42:0/1641112605 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe0a4019070 con 0x7fe0a8072d40 2024-09-16T04:00:54.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.328+0000 7fe0af2be640 5 monclient: authenticate success, global_id 825258 2024-09-16T04:00:54.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.328+0000 7fe0af2be640 10 monclient: _renew_subs 2024-09-16T04:00:54.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.328+0000 7fe0af2be640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:00:54.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.328+0000 7fe0af2be640 1 -- 172.21.15.42:0/1641112605 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0a811dc00 con 0x7fe0a8072d40 2024-09-16T04:00:54.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.328+0000 7fe09dffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:00:54.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.328+0000 7fe09dffb640 1 -- 172.21.15.42:0/1641112605 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0a400ba60 con 0x7fe0a8072d40 2024-09-16T04:00:54.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.328+0000 7fe09dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:00:54.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.328+0000 7fe09dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:00:54.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.328+0000 7fe09dffb640 10 monclient: dump: 2024-09-16T04:00:54.350 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:00:54.350 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:00:54.350 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:00:54.350 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:00:54.351 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:00:54.351 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:00:54.351 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:00:54.351 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:00:54.351 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:00:54.351 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:00:54.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.328+0000 7fe0af2be640 10 client.825258.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:00:54.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.328+0000 7fe0af2be640 10 monclient: _renew_subs 2024-09-16T04:00:54.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.328+0000 7fe09f7fe640 4 set_mon_vals no callback set 2024-09-16T04:00:54.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.328+0000 7fe0af2be640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:00:54.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.328+0000 7fe0af2be640 1 -- 172.21.15.42:0/1641112605 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0a80a4bf0 con 0x7fe0a8072d40 2024-09-16T04:00:54.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.328+0000 7fe0af2be640 1 librados: init done 2024-09-16T04:00:54.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.328+0000 7fe0af2be640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:00:54.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.328+0000 7fe0af2be640 1 --2- 172.21.15.42:0/1641112605 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe0a8133a90 0x7fe0a813d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:00:54.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.328+0000 7fe0af2be640 10 monclient(hunting): start opening mon connection 2024-09-16T04:00:54.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.328+0000 7fe0af2be640 1 -- 172.21.15.42:0/1641112605 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fe0a811c970 con 0x7fe0a8133a90 2024-09-16T04:00:54.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.328+0000 7fe0ac859640 1 --2- 172.21.15.42:0/1641112605 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe0a8133a90 0x7fe0a813d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:00:54.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.328+0000 7fe0ac859640 10 monclient: get_auth_request con 0x7fe0a8133a90 auth_method 0 2024-09-16T04:00:54.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.328+0000 7fe0ac859640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:00:54.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.328+0000 7fe0ac859640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:00:54.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.328+0000 7fe0ac859640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:00:54.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.328+0000 7fe09dffb640 1 -- 172.21.15.42:0/1641112605 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fe0a400bc00 con 0x7fe0a8072d40 2024-09-16T04:00:54.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.328+0000 7fe0ac859640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:00:54.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.328+0000 7fe0ac859640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:00:54.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.328+0000 7fe0ac859640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:00:54.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.328+0000 7fe0ac859640 10 monclient(hunting): handle_auth_done global_id 825261 payload 824 2024-09-16T04:00:54.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.328+0000 7fe0ac859640 1 -- 172.21.15.42:0/1641112605 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe0a8073a30 con 0x7fe0a8133a90 2024-09-16T04:00:54.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.328+0000 7fe09dffb640 1 --2- 172.21.15.42:0/1641112605 >> v2:172.21.15.154:6812/254845342 conn(0x7fe08c037810 0x7fe08c039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:00:54.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.328+0000 7fe09dffb640 1 -- 172.21.15.42:0/1641112605 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(566..566 src has 451..566) ==== 6224+0+0 (secure 0 0 0) 0x7fe0a40494f0 con 0x7fe0a8072d40 2024-09-16T04:00:54.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.328+0000 7fe09dffb640 10 client.825258.objecter ms_dispatch 0x7fe0a8000ba0 osd_map(566..566 src has 451..566) 2024-09-16T04:00:54.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.328+0000 7fe09dffb640 3 client.825258.objecter handle_osd_map got epochs [566,566] > 0 2024-09-16T04:00:54.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.328+0000 7fe09dffb640 3 client.825258.objecter handle_osd_map decoding full epoch 566 2024-09-16T04:00:54.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.328+0000 7fe09dffb640 20 client.825258.objecter dump_active .. 0 homeless 2024-09-16T04:00:54.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.328+0000 7fe09dffb640 10 client.825258.objecter ms_handle_connect 0x7fe0a8133a90 2024-09-16T04:00:54.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.328+0000 7fe09dffb640 10 client.825258.objecter resend_mon_ops 2024-09-16T04:00:54.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.328+0000 7fe0ac859640 1 --2- 172.21.15.42:0/1641112605 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe0a8133a90 0x7fe0a813d6f0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fe0a8068190 tx=0x7fe09801ac70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:00:54.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.328+0000 7fe09dffb640 1 -- 172.21.15.42:0/1641112605 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0980b0040 con 0x7fe0a8133a90 2024-09-16T04:00:54.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.328+0000 7fe09dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:00:54.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.328+0000 7fe09dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:00:54.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.328+0000 7fe09dffb640 10 monclient: dump: 2024-09-16T04:00:54.354 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:00:54.354 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:00:54.354 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:00:54.354 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:00:54.354 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:00:54.354 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:00:54.354 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:00:54.354 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:00:54.354 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:00:54.354 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:00:54.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.328+0000 7fe0ad85b640 1 --2- 172.21.15.42:0/1641112605 >> v2:172.21.15.154:6812/254845342 conn(0x7fe08c037810 0x7fe08c039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:00:54.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.328+0000 7fe0ad85b640 10 monclient: get_auth_request con 0x7fe08c037810 auth_method 0 2024-09-16T04:00:54.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.328+0000 7fe09dffb640 1 -- 172.21.15.42:0/1641112605 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fe0980b6670 con 0x7fe0a8133a90 2024-09-16T04:00:54.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.328+0000 7fe09dffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:00:54.355 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.328+0000 7fe09dffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:00:54.355 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.328+0000 7fe09dffb640 1 -- 172.21.15.42:0/1641112605 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe0a8133a90 msgr2=0x7fe0a813d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:00:54.355 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.328+0000 7fe09dffb640 1 --2- 172.21.15.42:0/1641112605 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe0a8133a90 0x7fe0a813d6f0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fe0a8068190 tx=0x7fe09801ac70 comp rx=0 tx=0).stop 2024-09-16T04:00:54.355 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.328+0000 7fe09dffb640 1 -- 172.21.15.42:0/1641112605 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe0a8133a90 msgr2=0x7fe0a813d6f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:00:54.355 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.328+0000 7fe09dffb640 1 --2- 172.21.15.42:0/1641112605 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe0a8133a90 0x7fe0a813d6f0 secure :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fe0a8068190 tx=0x7fe09801ac70 comp rx=0 tx=0).stop 2024-09-16T04:00:54.355 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.328+0000 7fe09dffb640 10 client.825258.objecter ms_handle_connect 0x7fe08c037810 2024-09-16T04:00:54.355 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.328+0000 7fe0ad85b640 1 --2- 172.21.15.42:0/1641112605 >> v2:172.21.15.154:6812/254845342 conn(0x7fe08c037810 0x7fe08c039cd0 secure :-1 s=READY pgs=3746 cs=0 l=1 rev1=1 crypto rx=0x7fe0a0007fd0 tx=0x7fe0a0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:00:54.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.336+0000 7fe0af2be640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:00:54.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.336+0000 7fe0af2be640 1 --2- 172.21.15.42:0/1641112605 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe0a813dc30 0x7fe0a8138650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:00:54.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.336+0000 7fe0af2be640 10 monclient(hunting): start opening mon connection 2024-09-16T04:00:54.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.336+0000 7fe0af2be640 1 -- 172.21.15.42:0/1641112605 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fe0a813e040 con 0x7fe0a813dc30 2024-09-16T04:00:54.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.336+0000 7fe0ac859640 1 --2- 172.21.15.42:0/1641112605 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe0a813dc30 0x7fe0a8138650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:00:54.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.336+0000 7fe0ac859640 10 monclient: get_auth_request con 0x7fe0a813dc30 auth_method 0 2024-09-16T04:00:54.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.336+0000 7fe0ac859640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:00:54.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.336+0000 7fe0ac859640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:00:54.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.336+0000 7fe0ac859640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:00:54.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.336+0000 7fe0ac859640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:00:54.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.336+0000 7fe0ac859640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:00:54.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.336+0000 7fe0ac859640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:00:54.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.336+0000 7fe0ac859640 10 monclient(hunting): handle_auth_done global_id 825264 payload 824 2024-09-16T04:00:54.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.336+0000 7fe0ac859640 1 -- 172.21.15.42:0/1641112605 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fe0a8137120 con 0x7fe0a813dc30 2024-09-16T04:00:54.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.336+0000 7fe09dffb640 10 client.825258.objecter ms_handle_connect 0x7fe0a813dc30 2024-09-16T04:00:54.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.336+0000 7fe09dffb640 10 client.825258.objecter resend_mon_ops 2024-09-16T04:00:54.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.336+0000 7fe0ac859640 1 --2- 172.21.15.42:0/1641112605 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe0a813dc30 0x7fe0a8138650 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fe09801c470 tx=0x7fe0980191d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:00:54.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.336+0000 7fe09dffb640 1 -- 172.21.15.42:0/1641112605 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0980b0050 con 0x7fe0a813dc30 2024-09-16T04:00:54.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.336+0000 7fe09dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:00:54.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.336+0000 7fe09dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:00:54.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.336+0000 7fe09dffb640 10 monclient: dump: 2024-09-16T04:00:54.358 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:00:54.358 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:00:54.358 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:00:54.358 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:00:54.359 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:00:54.359 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:00:54.359 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:00:54.359 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:00:54.359 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:00:54.359 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:00:54.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.336+0000 7fe09dffb640 1 -- 172.21.15.42:0/1641112605 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4715 (secure 0 0 0) 0x7fe0980bf750 con 0x7fe0a813dc30 2024-09-16T04:00:54.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.336+0000 7fe09dffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:00:54.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.336+0000 7fe09dffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:00:54.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.336+0000 7fe09dffb640 1 -- 172.21.15.42:0/1641112605 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe0a813dc30 msgr2=0x7fe0a8138650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:00:54.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.336+0000 7fe09dffb640 1 --2- 172.21.15.42:0/1641112605 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe0a813dc30 0x7fe0a8138650 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fe09801c470 tx=0x7fe0980191d0 comp rx=0 tx=0).stop 2024-09-16T04:00:54.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.336+0000 7fe09dffb640 1 -- 172.21.15.42:0/1641112605 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe0a813dc30 msgr2=0x7fe0a8138650 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:00:54.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.336+0000 7fe09dffb640 1 --2- 172.21.15.42:0/1641112605 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe0a813dc30 0x7fe0a8138650 secure :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fe09801c470 tx=0x7fe0980191d0 comp rx=0 tx=0).stop 2024-09-16T04:00:54.360 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T04:00:54.360 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:00:54.360 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:00:54.360 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "leader", 2024-09-16T04:00:54.360 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 168, 2024-09-16T04:00:54.360 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 544191, 2024-09-16T04:00:54.360 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T04:00:54.360 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T04:00:54.360 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T04:00:54.360 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T04:00:54.360 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:00:54.361 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 3, 2024-09-16T04:00:54.361 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:00:54.361 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T04:00:54.361 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T04:00:54.361 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:00:54.361 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:00:54.361 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:00:54.361 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:00:54.361 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:00:54.361 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:00:54.361 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:00:54.361 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:00:54.361 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:00:54.362 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:00:54.362 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:00:54.362 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:00:54.362 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:00:54.362 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T04:00:54.362 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:00:54.362 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:00:54.362 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:00:54.362 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:00:54.362 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:00:54.362 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:00:54.362 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:00:54.362 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:00:54.362 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:00:54.363 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:00:54.363 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:00:54.363 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:00:54.363 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:00:54.363 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T04:00:54.363 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T04:00:54.363 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T04:00:54.363 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T04:00:54.363 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T04:00:54.363 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T04:00:54.363 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:00:54.363 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:00:54.363 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T04:00:54.363 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T04:00:54.364 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T04:00:54.364 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T04:00:54.364 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T04:00:54.364 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T04:00:54.364 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T04:00:54.364 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:00:54.364 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T04:00:54.364 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:00:54.364 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:00:54.364 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:00:54.364 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:00:54.364 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:00:54.364 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:00:54.364 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:00:54.365 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:00:54.365 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:00:54.365 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:00:54.365 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:00:54.365 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:00:54.365 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T04:00:54.365 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:00:54.365 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T04:00:54.365 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:00:54.365 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:00:54.365 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:00:54.365 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:00:54.365 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:00:54.366 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:00:54.366 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:00:54.366 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T04:00:54.366 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:00:54.366 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:00:54.366 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:00:54.366 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:00:54.366 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T04:00:54.366 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:00:54.366 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:00:54.366 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:00:54.366 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:00:54.366 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T04:00:54.366 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T04:00:54.367 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:00:54.367 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:00:54.367 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:00:54.367 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:00:54.367 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:00:54.367 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:00:54.367 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:00:54.367 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:00:54.367 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:00:54.367 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:00:54.367 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:00:54.367 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T04:00:54.367 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:00:54.368 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:00:54.368 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:00:54.368 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:00:54.368 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T04:00:54.368 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:00:54.368 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:00:54.368 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:00:54.368 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:00:54.368 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T04:00:54.368 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T04:00:54.368 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:00:54.368 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:00:54.368 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:00:54.368 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:00:54.369 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:00:54.369 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:00:54.369 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:00:54.369 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:00:54.369 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:00:54.369 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:00:54.369 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:00:54.369 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T04:00:54.369 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:00:54.369 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:00:54.369 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:00:54.369 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:00:54.369 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T04:00:54.369 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:00:54.370 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:00:54.370 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:00:54.370 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:00:54.370 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T04:00:54.370 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T04:00:54.370 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:00:54.370 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:00:54.370 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:00:54.370 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:00:54.370 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:00:54.370 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:00:54.370 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T04:00:54.370 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T04:00:54.370 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:00:54.371 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:00:54.371 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:00:54.371 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:00:54.371 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:00:54.371 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:00:54.371 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T04:00:54.371 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:00:54.371 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:00:54.371 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:00:54.371 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 5 2024-09-16T04:00:54.371 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:00:54.371 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:00:54.371 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T04:00:54.371 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:00:54.372 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:00:54.372 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:00:54.372 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 7 2024-09-16T04:00:54.372 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:00:54.372 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:00:54.372 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:00:54.372 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T04:00:54.372 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T04:00:54.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.336+0000 7fe0af2be640 10 librados: watch_flush enter 2024-09-16T04:00:54.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.336+0000 7fe0af2be640 10 librados: watch_flush exit 2024-09-16T04:00:54.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.336+0000 7fe0af2be640 20 client.825258.objecter shutdown clearing up homeless session... 2024-09-16T04:00:54.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.336+0000 7fe0af2be640 10 client.825258.objecter successfully canceled tick 2024-09-16T04:00:54.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.336+0000 7fe0af2be640 1 -- 172.21.15.42:0/1641112605 >> v2:172.21.15.154:6812/254845342 conn(0x7fe08c037810 msgr2=0x7fe08c039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:00:54.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.336+0000 7fe0af2be640 1 --2- 172.21.15.42:0/1641112605 >> v2:172.21.15.154:6812/254845342 conn(0x7fe08c037810 0x7fe08c039cd0 secure :-1 s=READY pgs=3746 cs=0 l=1 rev1=1 crypto rx=0x7fe0a0007fd0 tx=0x7fe0a0009040 comp rx=0 tx=0).stop 2024-09-16T04:00:54.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.336+0000 7fe0af2be640 10 monclient: shutdown 2024-09-16T04:00:54.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.336+0000 7fe0af2be640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:00:54.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.336+0000 7fe0af2be640 1 -- 172.21.15.42:0/1641112605 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe0a8072d40 msgr2=0x7fe0a81cf0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:00:54.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.336+0000 7fe0af2be640 1 --2- 172.21.15.42:0/1641112605 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe0a8072d40 0x7fe0a81cf0b0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fe0a400d800 tx=0x7fe0a400dcc0 comp rx=0 tx=0).stop 2024-09-16T04:00:54.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.336+0000 7fe0ad85b640 1 -- 172.21.15.42:0/1641112605 reap_dead start 2024-09-16T04:00:54.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.336+0000 7fe0af2be640 1 -- 172.21.15.42:0/1641112605 shutdown_connections 2024-09-16T04:00:54.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.336+0000 7fe0af2be640 1 --2- 172.21.15.42:0/1641112605 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe0a8072d40 0x7fe0a81cf0b0 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:00:54.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.336+0000 7fe0af2be640 1 -- 172.21.15.42:0/1641112605 >> 172.21.15.42:0/1641112605 conn(0x7fe0a80a2420 msgr2=0x7fe0a809cd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:00:54.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.336+0000 7fe0af2be640 1 -- 172.21.15.42:0/1641112605 shutdown_connections 2024-09-16T04:00:54.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.336+0000 7fe0af2be640 1 -- 172.21.15.42:0/1641112605 wait complete. 2024-09-16T04:00:54.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.336+0000 7fe0af2be640 1 librados: shutdown 2024-09-16T04:00:54.375 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:00:54.375 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T04:00:54.376 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T04:00:54.498 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.492+0000 7f56c99a2640 10 monclient: get_monmap_and_config 2024-09-16T04:00:54.498 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.492+0000 7f56c99a2640 10 monclient: build_initial_monmap 2024-09-16T04:00:54.498 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.492+0000 7f56c99a2640 10 monclient: monmap: 2024-09-16T04:00:54.499 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:00:54.499 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:00:54.499 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:00:54.497270+0000 2024-09-16T04:00:54.499 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:00:54.497270+0000 2024-09-16T04:00:54.499 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:00:54.499 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:00:54.499 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:00:54.499 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:00:54.499 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:00:54.499 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:00:54.499 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:00:54.499 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:00:54.502 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.492+0000 7f56c99a2640 1 Processor -- start 2024-09-16T04:00:54.502 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.492+0000 7f56c99a2640 1 -- start start 2024-09-16T04:00:54.502 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.492+0000 7f56c99a2640 10 monclient: init 2024-09-16T04:00:54.502 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.492+0000 7f56c99a2640 10 monclient: _reopen_session rank -1 2024-09-16T04:00:54.502 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.492+0000 7f56c99a2640 10 monclient: _add_conns ranks=[4,3,0,1,2] 2024-09-16T04:00:54.502 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.492+0000 7f56c99a2640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f56c40a1330 0x7f56c40a1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:00:54.502 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.492+0000 7f56c99a2640 10 monclient(hunting): picked mon.noname-c con 0x7f56c40a1330 addr v2:172.21.15.154:3300/0 2024-09-16T04:00:54.502 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.492+0000 7f56c99a2640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f56c406dac0 0x7f56c40a1c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:00:54.502 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.492+0000 7f56c99a2640 10 monclient(hunting): picked mon.noname-a con 0x7f56c406dac0 addr v2:172.21.15.42:3300/0 2024-09-16T04:00:54.502 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.492+0000 7f56c99a2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f56c4135100 addr v1:172.21.15.42:6789/0 2024-09-16T04:00:54.502 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.492+0000 7f56c99a2640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f56c409c7d0 con 0x7f56c4135100 2024-09-16T04:00:54.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.492+0000 7f56c99a2640 10 monclient(hunting): start opening mon connection 2024-09-16T04:00:54.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.492+0000 7f56c99a2640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f56c409c090 con 0x7f56c406dac0 2024-09-16T04:00:54.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.492+0000 7f56c99a2640 10 monclient(hunting): start opening mon connection 2024-09-16T04:00:54.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.492+0000 7f56c99a2640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f56c409f610 con 0x7f56c40a1330 2024-09-16T04:00:54.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.492+0000 7f56c99a2640 10 monclient(hunting): _renew_subs 2024-09-16T04:00:54.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.492+0000 7f56c99a2640 10 monclient(hunting): authenticate will time out at 2101.466726s 2024-09-16T04:00:54.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.492+0000 7f56c27fc640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f56c406dac0 0x7f56c40a1c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:00:54.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.492+0000 7f56c37fe640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f56c4135100 0x7f56c4135550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:56220/0 (socket says 172.21.15.42:56220) 2024-09-16T04:00:54.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.492+0000 7f56c37fe640 1 -- 172.21.15.42:0/2259039897 learned_addr learned my addr 172.21.15.42:0/2259039897 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T04:00:54.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.492+0000 7f56c27fc640 10 monclient(hunting): get_auth_request con 0x7f56c406dac0 auth_method 0 2024-09-16T04:00:54.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.492+0000 7f56c27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:00:54.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.492+0000 7f56c27fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:00:54.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.492+0000 7f56c27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:00:54.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.492+0000 7f56c27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:00:54.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.492+0000 7f56c27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:00:54.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.492+0000 7f56c27fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:00:54.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.492+0000 7f56c2ffd640 1 --2- 172.21.15.42:0/2259039897 >> v2:172.21.15.154:3300/0 conn(0x7f56c40a1330 0x7f56c40a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:00:54.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.492+0000 7f56c2ffd640 10 monclient(hunting): get_auth_request con 0x7f56c40a1330 auth_method 0 2024-09-16T04:00:54.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.492+0000 7f56c2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:00:54.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.492+0000 7f56c2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:00:54.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.492+0000 7f56c2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:00:54.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.492+0000 7f56c1ffb640 1 -- 172.21.15.42:0/2259039897 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4211982578 0 0) 0x7f56c409c7d0 con 0x7f56c4135100 2024-09-16T04:00:54.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.492+0000 7f56c1ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:00:54.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.492+0000 7f56c1ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:00:54.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.492+0000 7f56c1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:00:54.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.492+0000 7f56c1ffb640 10 monclient(hunting): my global_id is 825270 2024-09-16T04:00:54.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.492+0000 7f56c1ffb640 1 -- 172.21.15.42:0/2259039897 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f56a4003530 con 0x7f56c4135100 2024-09-16T04:00:54.505 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.492+0000 7f56c2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:00:54.505 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.492+0000 7f56c2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:00:54.505 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.492+0000 7f56c2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:00:54.505 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.492+0000 7f56c27fc640 10 monclient(hunting): handle_auth_done global_id 825267 payload 274 2024-09-16T04:00:54.505 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.492+0000 7f56c27fc640 1 -- 172.21.15.42:0/2259039897 >> v2:172.21.15.154:3300/0 conn(0x7f56c40a1330 msgr2=0x7f56c40a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:00:54.505 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.492+0000 7f56c27fc640 1 --2- 172.21.15.42:0/2259039897 >> v2:172.21.15.154:3300/0 conn(0x7f56c40a1330 0x7f56c40a1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:00:54.505 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.492+0000 7f56c27fc640 1 -- 172.21.15.42:0/2259039897 >> v1:172.21.15.42:6789/0 conn(0x7f56c4135100 legacy=0x7f56c4135550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:00:54.505 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.492+0000 7f56c27fc640 10 monclient: _finish_hunting 0 2024-09-16T04:00:54.505 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.492+0000 7f56c27fc640 1 monclient: found mon.noname-a 2024-09-16T04:00:54.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.492+0000 7f56c27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:00:54.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.492+0000 7f56c27fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:00:54.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.492+0000 7f56c27fc640 1 -- 172.21.15.42:0/2259039897 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56c409f8f0 con 0x7f56c406dac0 2024-09-16T04:00:54.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.496+0000 7f56c27fc640 1 --2- 172.21.15.42:0/2259039897 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f56c406dac0 0x7f56c40a1c50 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f56c406add0 tx=0x7f56ac01c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2b9203264ffea9d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:00:54.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.496+0000 7f56c1ffb640 1 -- 172.21.15.42:0/2259039897 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56ac0b2070 con 0x7f56c406dac0 2024-09-16T04:00:54.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.496+0000 7f56c1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:00:54.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.496+0000 7f56c1ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:00:54.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.496+0000 7f56c1ffb640 10 monclient: dump: 2024-09-16T04:00:54.512 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:00:54.512 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:00:54.512 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:00:54.512 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:00:54.512 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:00:54.512 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:00:54.512 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:00:54.512 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:00:54.512 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:00:54.513 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:00:54.513 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.496+0000 7f56c1ffb640 10 monclient: _finish_auth 0 2024-09-16T04:00:54.513 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.496+0000 7f56c1ffb640 10 monclient: _check_auth_tickets 2024-09-16T04:00:54.513 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.496+0000 7f56c1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:00:54.513 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.496+0000 7f56c1ffb640 1 -- 172.21.15.42:0/2259039897 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f56ac0a59c0 con 0x7f56c406dac0 2024-09-16T04:00:54.513 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.496+0000 7f56c1ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:00:54.513 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.496+0000 7f56c1ffb640 1 -- 172.21.15.42:0/2259039897 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56ac0a5d90 con 0x7f56c406dac0 2024-09-16T04:00:54.513 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.496+0000 7f56c1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:00:54.513 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.496+0000 7f56c1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:00:54.513 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.496+0000 7f56c1ffb640 10 monclient: dump: 2024-09-16T04:00:54.513 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:00:54.513 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:00:54.513 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:00:54.513 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:00:54.514 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:00:54.514 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:00:54.514 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:00:54.514 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:00:54.514 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:00:54.514 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:00:54.514 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.496+0000 7f56c99a2640 5 monclient: authenticate success, global_id 825267 2024-09-16T04:00:54.514 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.496+0000 7f56c99a2640 10 monclient: get_monmap_and_config success 2024-09-16T04:00:54.514 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.496+0000 7f56c99a2640 4 set_mon_vals no callback set 2024-09-16T04:00:54.514 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.496+0000 7f56c99a2640 10 monclient: shutdown 2024-09-16T04:00:54.514 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.496+0000 7f56c99a2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:00:54.514 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.496+0000 7f56c99a2640 1 -- 172.21.15.42:0/2259039897 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f56c406dac0 msgr2=0x7f56c40a1c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:00:54.514 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.496+0000 7f56c99a2640 1 --2- 172.21.15.42:0/2259039897 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f56c406dac0 0x7f56c40a1c50 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f56c406add0 tx=0x7f56ac01c8e0 comp rx=0 tx=0).stop 2024-09-16T04:00:54.514 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.496+0000 7f56c99a2640 1 -- 172.21.15.42:0/2259039897 shutdown_connections 2024-09-16T04:00:54.514 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.496+0000 7f56c99a2640 1 --2- 172.21.15.42:0/2259039897 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f56c406dac0 0x7f56c40a1c50 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:00:54.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.496+0000 7f56c99a2640 1 --2- 172.21.15.42:0/2259039897 >> v2:172.21.15.154:3300/0 conn(0x7f56c40a1330 0x7f56c40a1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:00:54.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.496+0000 7f56c99a2640 1 -- 172.21.15.42:0/2259039897 >> 172.21.15.42:0/2259039897 conn(0x7f56c40a2420 msgr2=0x7f56c40a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:00:54.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.496+0000 7f56c99a2640 1 -- 172.21.15.42:0/2259039897 shutdown_connections 2024-09-16T04:00:54.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.496+0000 7f56c99a2640 1 -- 172.21.15.42:0/2259039897 wait complete. 2024-09-16T04:00:54.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.496+0000 7f56c99a2640 10 monclient: build_initial_monmap 2024-09-16T04:00:54.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.496+0000 7f56c99a2640 10 monclient: monmap: 2024-09-16T04:00:54.515 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:00:54.515 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:00:54.515 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:00:54.515 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:00:54.516 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:00:54.516 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:00:54.516 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:00:54.516 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:00:54.516 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:00:54.516 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:00:54.516 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.496+0000 7f56c99a2640 1 librados: starting msgr at 2024-09-16T04:00:54.516 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.496+0000 7f56c99a2640 1 librados: starting objecter 2024-09-16T04:00:54.516 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.496+0000 7f56c99a2640 1 Processor -- start 2024-09-16T04:00:54.516 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.496+0000 7f56c99a2640 1 -- start start 2024-09-16T04:00:54.516 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.496+0000 7f56c99a2640 1 librados: setting wanted keys 2024-09-16T04:00:54.516 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.496+0000 7f56c99a2640 1 librados: calling monclient init 2024-09-16T04:00:54.516 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.496+0000 7f56c99a2640 10 monclient: init 2024-09-16T04:00:54.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.496+0000 7f56c99a2640 10 monclient: _reopen_session rank -1 2024-09-16T04:00:54.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.496+0000 7f56c99a2640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:00:54.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.496+0000 7f56c99a2640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f56c40a1330 0x7f56c411bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:00:54.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.496+0000 7f56c99a2640 10 monclient(hunting): picked mon.noname-b con 0x7f56c40a1330 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:00:54.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.496+0000 7f56c99a2640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f56c4135100 0x7f56c411c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:00:54.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.496+0000 7f56c99a2640 10 monclient(hunting): picked mon.noname-a con 0x7f56c4135100 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:00:54.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.496+0000 7f56c99a2640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f56c411c9f0 0x7f56c4133800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:00:54.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.496+0000 7f56c99a2640 10 monclient(hunting): picked mon.noname-c con 0x7f56c411c9f0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:00:54.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.496+0000 7f56c99a2640 10 monclient(hunting): start opening mon connection 2024-09-16T04:00:54.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.496+0000 7f56c99a2640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f56c413fb80 con 0x7f56c4135100 2024-09-16T04:00:54.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.496+0000 7f56c99a2640 10 monclient(hunting): start opening mon connection 2024-09-16T04:00:54.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.496+0000 7f56c99a2640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f56c409ef80 con 0x7f56c40a1330 2024-09-16T04:00:54.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.496+0000 7f56c99a2640 10 monclient(hunting): start opening mon connection 2024-09-16T04:00:54.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.496+0000 7f56c99a2640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f56c413fd00 con 0x7f56c411c9f0 2024-09-16T04:00:54.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.496+0000 7f56c99a2640 10 monclient(hunting): _renew_subs 2024-09-16T04:00:54.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.496+0000 7f56c99a2640 10 monclient(hunting): authenticate will time out at 2101.471863s 2024-09-16T04:00:54.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.496+0000 7f56c37fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f56c411c9f0 0x7f56c4133800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:00:54.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.496+0000 7f56c27fc640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f56c4135100 0x7f56c411c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:00:54.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.500+0000 7f56c37fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f56c411c9f0 0x7f56c4133800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:40668/0 (socket says 172.21.15.42:40668) 2024-09-16T04:00:54.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.500+0000 7f56c37fe640 1 -- 172.21.15.42:0/2723270600 learned_addr learned my addr 172.21.15.42:0/2723270600 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:00:54.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.500+0000 7f56c37fe640 10 monclient(hunting): get_auth_request con 0x7f56c411c9f0 auth_method 0 2024-09-16T04:00:54.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.500+0000 7f56c37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:00:54.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.500+0000 7f56c37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:00:54.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.500+0000 7f56c37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:00:54.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.500+0000 7f56c27fc640 10 monclient(hunting): get_auth_request con 0x7f56c4135100 auth_method 0 2024-09-16T04:00:54.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.500+0000 7f56c27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:00:54.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.500+0000 7f56c27fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:00:54.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.500+0000 7f56c27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:00:54.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.500+0000 7f56c37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:00:54.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.500+0000 7f56c37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:00:54.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.500+0000 7f56c37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:00:54.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.500+0000 7f56c27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:00:54.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.500+0000 7f56c27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:00:54.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.500+0000 7f56c2ffd640 1 --2- 172.21.15.42:0/2723270600 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f56c40a1330 0x7f56c411bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:00:54.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.500+0000 7f56c27fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:00:54.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.500+0000 7f56c2ffd640 10 monclient(hunting): get_auth_request con 0x7f56c40a1330 auth_method 0 2024-09-16T04:00:54.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.500+0000 7f56c2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:00:54.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.500+0000 7f56c2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:00:54.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.500+0000 7f56c2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:00:54.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.500+0000 7f56c37fe640 10 monclient(hunting): handle_auth_done global_id 944108 payload 824 2024-09-16T04:00:54.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.500+0000 7f56c37fe640 1 -- 172.21.15.42:0/2723270600 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f56c40a1330 msgr2=0x7f56c411bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:00:54.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.500+0000 7f56c37fe640 1 --2- 172.21.15.42:0/2723270600 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f56c40a1330 0x7f56c411bf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:00:54.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.500+0000 7f56c37fe640 1 -- 172.21.15.42:0/2723270600 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f56c4135100 msgr2=0x7f56c411c4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:00:54.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.500+0000 7f56c37fe640 1 --2- 172.21.15.42:0/2723270600 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f56c4135100 0x7f56c411c4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:00:54.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.500+0000 7f56c37fe640 10 monclient: _finish_hunting 0 2024-09-16T04:00:54.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.500+0000 7f56c37fe640 1 monclient: found mon.noname-c 2024-09-16T04:00:54.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.500+0000 7f56c37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:00:54.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.500+0000 7f56c37fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T04:00:54.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.500+0000 7f56c37fe640 1 -- 172.21.15.42:0/2723270600 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56c411e720 con 0x7f56c411c9f0 2024-09-16T04:00:54.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.500+0000 7f56c2ffd640 1 --2- 172.21.15.42:0/2723270600 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f56c40a1330 0x7f56c411bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T04:00:54.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.500+0000 7f56c27fc640 1 --2- 172.21.15.42:0/2723270600 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f56c4135100 0x7f56c411c4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:00:54.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.500+0000 7f56c37fe640 1 --2- 172.21.15.42:0/2723270600 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f56c411c9f0 0x7f56c4133800 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f56b400c480 tx=0x7f56b400c940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:00:54.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.500+0000 7f56bbfff640 10 client.?.objecter ms_handle_connect 0x7f56c411c9f0 2024-09-16T04:00:54.521 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.500+0000 7f56bbfff640 10 client.?.objecter resend_mon_ops 2024-09-16T04:00:54.521 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.500+0000 7f56bbfff640 1 -- 172.21.15.42:0/2723270600 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56b4017070 con 0x7f56c411c9f0 2024-09-16T04:00:54.521 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.500+0000 7f56bbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:00:54.521 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.500+0000 7f56bbfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:00:54.521 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.500+0000 7f56bbfff640 10 monclient: dump: 2024-09-16T04:00:54.521 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:00:54.521 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:00:54.521 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:00:54.521 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:00:54.521 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:00:54.521 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:00:54.521 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:00:54.521 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:00:54.521 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:00:54.522 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:00:54.522 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.500+0000 7f56bbfff640 10 monclient: _finish_auth 0 2024-09-16T04:00:54.522 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.500+0000 7f56bbfff640 10 monclient: _check_auth_tickets 2024-09-16T04:00:54.522 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.500+0000 7f56bbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:00:54.522 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.500+0000 7f56bbfff640 1 -- 172.21.15.42:0/2723270600 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f56b4005cf0 con 0x7f56c411c9f0 2024-09-16T04:00:54.522 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.500+0000 7f56bbfff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:00:54.522 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.500+0000 7f56bbfff640 1 -- 172.21.15.42:0/2723270600 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56b40072b0 con 0x7f56c411c9f0 2024-09-16T04:00:54.522 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.500+0000 7f56bbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:00:54.522 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.500+0000 7f56bbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:00:54.522 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.500+0000 7f56bbfff640 10 monclient: dump: 2024-09-16T04:00:54.522 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:00:54.522 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:00:54.522 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:00:54.522 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:00:54.523 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:00:54.523 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:00:54.523 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:00:54.523 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:00:54.523 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:00:54.523 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:00:54.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.500+0000 7f56c17fa640 4 set_mon_vals no callback set 2024-09-16T04:00:54.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.500+0000 7f56c99a2640 5 monclient: authenticate success, global_id 944108 2024-09-16T04:00:54.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.500+0000 7f56c99a2640 10 monclient: _renew_subs 2024-09-16T04:00:54.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.500+0000 7f56c99a2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:00:54.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.500+0000 7f56c99a2640 1 -- 172.21.15.42:0/2723270600 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56c411ea00 con 0x7f56c411c9f0 2024-09-16T04:00:54.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.500+0000 7f56c99a2640 10 client.944108.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:00:54.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.500+0000 7f56c99a2640 10 monclient: _renew_subs 2024-09-16T04:00:54.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.500+0000 7f56c99a2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:00:54.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.500+0000 7f56c99a2640 1 -- 172.21.15.42:0/2723270600 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f56c40a5330 con 0x7f56c411c9f0 2024-09-16T04:00:54.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.500+0000 7f56c99a2640 1 librados: init done 2024-09-16T04:00:54.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.500+0000 7f56c99a2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:00:54.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.500+0000 7f56c99a2640 1 --2- 172.21.15.42:0/2723270600 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f56c406dac0 0x7f56c402c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:00:54.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.500+0000 7f56c99a2640 10 monclient(hunting): start opening mon connection 2024-09-16T04:00:54.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.500+0000 7f56c99a2640 1 -- 172.21.15.42:0/2723270600 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f56c411d0a0 con 0x7f56c406dac0 2024-09-16T04:00:54.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.500+0000 7f56c2ffd640 1 --2- 172.21.15.42:0/2723270600 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f56c406dac0 0x7f56c402c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:00:54.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.500+0000 7f56bbfff640 1 -- 172.21.15.42:0/2723270600 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f56b401a070 con 0x7f56c411c9f0 2024-09-16T04:00:54.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.500+0000 7f56c2ffd640 10 monclient: get_auth_request con 0x7f56c406dac0 auth_method 0 2024-09-16T04:00:54.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.500+0000 7f56c2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:00:54.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.500+0000 7f56c2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:00:54.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.500+0000 7f56c2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:00:54.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.500+0000 7f56c2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:00:54.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.500+0000 7f56c2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:00:54.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.500+0000 7f56c2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:00:54.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.500+0000 7f56bbfff640 1 --2- 172.21.15.42:0/2723270600 >> v2:172.21.15.154:6812/254845342 conn(0x7f56a8037840 0x7f56a8039d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:00:54.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.500+0000 7f56c2ffd640 10 monclient(hunting): handle_auth_done global_id 944111 payload 824 2024-09-16T04:00:54.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.500+0000 7f56c2ffd640 1 -- 172.21.15.42:0/2723270600 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f56c4073a30 con 0x7f56c406dac0 2024-09-16T04:00:54.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.500+0000 7f56bbfff640 1 -- 172.21.15.42:0/2723270600 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(566..566 src has 451..566) ==== 6224+0+0 (secure 0 0 0) 0x7f56b4048540 con 0x7f56c411c9f0 2024-09-16T04:00:54.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.500+0000 7f56bbfff640 10 client.944108.objecter ms_dispatch 0x7f56c4000ba0 osd_map(566..566 src has 451..566) 2024-09-16T04:00:54.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.500+0000 7f56bbfff640 3 client.944108.objecter handle_osd_map got epochs [566,566] > 0 2024-09-16T04:00:54.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.500+0000 7f56bbfff640 3 client.944108.objecter handle_osd_map decoding full epoch 566 2024-09-16T04:00:54.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.500+0000 7f56bbfff640 20 client.944108.objecter dump_active .. 0 homeless 2024-09-16T04:00:54.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.500+0000 7f56c2ffd640 1 --2- 172.21.15.42:0/2723270600 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f56c406dac0 0x7f56c402c650 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f56bc011830 tx=0x7f56bc011cf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:00:54.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.500+0000 7f56bbfff640 10 client.944108.objecter ms_handle_connect 0x7f56c406dac0 2024-09-16T04:00:54.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.500+0000 7f56bbfff640 10 client.944108.objecter resend_mon_ops 2024-09-16T04:00:54.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.500+0000 7f56c27fc640 1 --2- 172.21.15.42:0/2723270600 >> v2:172.21.15.154:6812/254845342 conn(0x7f56a8037840 0x7f56a8039d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:00:54.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.500+0000 7f56bbfff640 1 -- 172.21.15.42:0/2723270600 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56bc00fa30 con 0x7f56c406dac0 2024-09-16T04:00:54.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.500+0000 7f56bbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:00:54.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.500+0000 7f56bbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:00:54.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.500+0000 7f56bbfff640 10 monclient: dump: 2024-09-16T04:00:54.526 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:00:54.526 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:00:54.527 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:00:54.527 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:00:54.527 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:00:54.527 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:00:54.527 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:00:54.527 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:00:54.527 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:00:54.527 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:00:54.527 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.500+0000 7f56c27fc640 10 monclient: get_auth_request con 0x7f56a8037840 auth_method 0 2024-09-16T04:00:54.527 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.500+0000 7f56bbfff640 1 -- 172.21.15.42:0/2723270600 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f56bc00fa30 con 0x7f56c406dac0 2024-09-16T04:00:54.527 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.500+0000 7f56bbfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:00:54.527 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.500+0000 7f56bbfff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:00:54.527 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.500+0000 7f56bbfff640 1 -- 172.21.15.42:0/2723270600 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f56c406dac0 msgr2=0x7f56c402c650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:00:54.527 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.500+0000 7f56bbfff640 1 --2- 172.21.15.42:0/2723270600 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f56c406dac0 0x7f56c402c650 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f56bc011830 tx=0x7f56bc011cf0 comp rx=0 tx=0).stop 2024-09-16T04:00:54.528 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.500+0000 7f56bbfff640 1 -- 172.21.15.42:0/2723270600 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f56c406dac0 msgr2=0x7f56c402c650 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:00:54.528 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.500+0000 7f56bbfff640 1 --2- 172.21.15.42:0/2723270600 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f56c406dac0 0x7f56c402c650 secure :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f56bc011830 tx=0x7f56bc011cf0 comp rx=0 tx=0).stop 2024-09-16T04:00:54.528 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.500+0000 7f56bbfff640 10 client.944108.objecter ms_handle_connect 0x7f56a8037840 2024-09-16T04:00:54.528 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.500+0000 7f56c27fc640 1 --2- 172.21.15.42:0/2723270600 >> v2:172.21.15.154:6812/254845342 conn(0x7f56a8037840 0x7f56a8039d00 secure :-1 s=READY pgs=3747 cs=0 l=1 rev1=1 crypto rx=0x7f56ac0af000 tx=0x7f56ac0a7070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:00:54.529 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.508+0000 7f56c99a2640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:00:54.530 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.508+0000 7f56c99a2640 1 --2- 172.21.15.42:0/2723270600 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f56c402cb90 0x7f56c402cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:00:54.530 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.508+0000 7f56c99a2640 10 monclient(hunting): start opening mon connection 2024-09-16T04:00:54.530 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.508+0000 7f56c99a2640 1 -- 172.21.15.42:0/2723270600 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f56c4135550 con 0x7f56c402cb90 2024-09-16T04:00:54.530 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.508+0000 7f56c2ffd640 1 --2- 172.21.15.42:0/2723270600 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f56c402cb90 0x7f56c402cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:00:54.530 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.508+0000 7f56c2ffd640 10 monclient: get_auth_request con 0x7f56c402cb90 auth_method 0 2024-09-16T04:00:54.530 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.508+0000 7f56c2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:00:54.530 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.508+0000 7f56c2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:00:54.530 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.508+0000 7f56c2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:00:54.530 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.508+0000 7f56c2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:00:54.530 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.508+0000 7f56c2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:00:54.530 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.508+0000 7f56c2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:00:54.530 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.508+0000 7f56c2ffd640 10 monclient(hunting): handle_auth_done global_id 944114 payload 824 2024-09-16T04:00:54.530 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.508+0000 7f56c2ffd640 1 -- 172.21.15.42:0/2723270600 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f56c40708c0 con 0x7f56c402cb90 2024-09-16T04:00:54.530 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.508+0000 7f56c2ffd640 1 --2- 172.21.15.42:0/2723270600 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f56c402cb90 0x7f56c402cfa0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f56bc011830 tx=0x7f56bc0046d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:00:54.531 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.508+0000 7f56bbfff640 10 client.944108.objecter ms_handle_connect 0x7f56c402cb90 2024-09-16T04:00:54.531 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.508+0000 7f56bbfff640 10 client.944108.objecter resend_mon_ops 2024-09-16T04:00:54.531 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.508+0000 7f56bbfff640 1 -- 172.21.15.42:0/2723270600 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56bc015670 con 0x7f56c402cb90 2024-09-16T04:00:54.531 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.508+0000 7f56bbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:00:54.531 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.508+0000 7f56bbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:00:54.531 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.508+0000 7f56bbfff640 10 monclient: dump: 2024-09-16T04:00:54.531 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:00:54.531 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:00:54.531 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:00:54.531 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:00:54.531 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:00:54.531 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:00:54.531 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:00:54.531 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:00:54.532 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:00:54.532 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:00:54.532 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.508+0000 7f56bbfff640 1 -- 172.21.15.42:0/2723270600 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4545 (secure 0 0 0) 0x7f56bc015670 con 0x7f56c402cb90 2024-09-16T04:00:54.532 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.508+0000 7f56bbfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:00:54.532 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.508+0000 7f56bbfff640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:00:54.532 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.508+0000 7f56bbfff640 1 -- 172.21.15.42:0/2723270600 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f56c402cb90 msgr2=0x7f56c402cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:00:54.532 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.508+0000 7f56bbfff640 1 --2- 172.21.15.42:0/2723270600 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f56c402cb90 0x7f56c402cfa0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f56bc011830 tx=0x7f56bc0046d0 comp rx=0 tx=0).stop 2024-09-16T04:00:54.532 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.508+0000 7f56bbfff640 1 -- 172.21.15.42:0/2723270600 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f56c402cb90 msgr2=0x7f56c402cfa0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:00:54.532 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.508+0000 7f56bbfff640 1 --2- 172.21.15.42:0/2723270600 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f56c402cb90 0x7f56c402cfa0 secure :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f56bc011830 tx=0x7f56bc0046d0 comp rx=0 tx=0).stop 2024-09-16T04:00:54.532 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T04:00:54.533 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:00:54.533 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:00:54.533 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T04:00:54.533 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 168, 2024-09-16T04:00:54.533 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 23339, 2024-09-16T04:00:54.533 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T04:00:54.533 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T04:00:54.533 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T04:00:54.533 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T04:00:54.533 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:00:54.533 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 3, 2024-09-16T04:00:54.533 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:00:54.533 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T04:00:54.534 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T04:00:54.534 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:00:54.534 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:00:54.534 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:00:54.534 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:00:54.534 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:00:54.534 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:00:54.534 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:00:54.534 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:00:54.534 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:00:54.534 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:00:54.534 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:00:54.534 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:00:54.534 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:00:54.535 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T04:00:54.535 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:00:54.535 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:00:54.535 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:00:54.535 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:00:54.535 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:00:54.535 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:00:54.535 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:00:54.535 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:00:54.535 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:00:54.535 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:00:54.535 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:00:54.535 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:00:54.535 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:00:54.536 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T04:00:54.536 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T04:00:54.536 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T04:00:54.536 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T04:00:54.536 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T04:00:54.536 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T04:00:54.536 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:00:54.536 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:00:54.536 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T04:00:54.536 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T04:00:54.536 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T04:00:54.536 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T04:00:54.536 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T04:00:54.537 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T04:00:54.537 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T04:00:54.537 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:00:54.537 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T04:00:54.537 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:00:54.537 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:00:54.537 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:00:54.537 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:00:54.537 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:00:54.537 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:00:54.537 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:00:54.537 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:00:54.537 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:00:54.538 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:00:54.538 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:00:54.538 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:00:54.538 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T04:00:54.538 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:00:54.538 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T04:00:54.538 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:00:54.538 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:00:54.538 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:00:54.538 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:00:54.538 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:00:54.538 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:00:54.538 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:00:54.538 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T04:00:54.539 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:00:54.539 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:00:54.539 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:00:54.539 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:00:54.539 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T04:00:54.539 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:00:54.539 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:00:54.539 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:00:54.539 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:00:54.539 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T04:00:54.539 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T04:00:54.539 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:00:54.539 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:00:54.540 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:00:54.540 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:00:54.540 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:00:54.540 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:00:54.540 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:00:54.540 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:00:54.540 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:00:54.540 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:00:54.540 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:00:54.540 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T04:00:54.540 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:00:54.540 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:00:54.540 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:00:54.540 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:00:54.541 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T04:00:54.541 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:00:54.541 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:00:54.541 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:00:54.541 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:00:54.541 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T04:00:54.541 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T04:00:54.541 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:00:54.541 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:00:54.541 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:00:54.541 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:00:54.541 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:00:54.541 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:00:54.542 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:00:54.542 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:00:54.542 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:00:54.542 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:00:54.542 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:00:54.542 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T04:00:54.542 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:00:54.542 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:00:54.542 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:00:54.542 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:00:54.542 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T04:00:54.542 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:00:54.542 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:00:54.542 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:00:54.543 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:00:54.543 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T04:00:54.543 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T04:00:54.543 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:00:54.543 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:00:54.543 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:00:54.543 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:00:54.543 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:00:54.543 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:00:54.543 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T04:00:54.543 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T04:00:54.543 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:00:54.543 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:00:54.543 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:00:54.544 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:00:54.544 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:00:54.544 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:00:54.544 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T04:00:54.544 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:00:54.544 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:00:54.544 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:00:54.544 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 2 2024-09-16T04:00:54.544 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:00:54.544 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:00:54.544 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:00:54.544 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T04:00:54.544 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T04:00:54.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.508+0000 7f56c99a2640 10 librados: watch_flush enter 2024-09-16T04:00:54.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.508+0000 7f56c99a2640 10 librados: watch_flush exit 2024-09-16T04:00:54.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.508+0000 7f56c99a2640 20 client.944108.objecter shutdown clearing up homeless session... 2024-09-16T04:00:54.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.508+0000 7f56c99a2640 10 client.944108.objecter successfully canceled tick 2024-09-16T04:00:54.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.508+0000 7f56c99a2640 1 -- 172.21.15.42:0/2723270600 >> v2:172.21.15.154:6812/254845342 conn(0x7f56a8037840 msgr2=0x7f56a8039d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:00:54.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.508+0000 7f56c99a2640 1 --2- 172.21.15.42:0/2723270600 >> v2:172.21.15.154:6812/254845342 conn(0x7f56a8037840 0x7f56a8039d00 secure :-1 s=READY pgs=3747 cs=0 l=1 rev1=1 crypto rx=0x7f56ac0af000 tx=0x7f56ac0a7070 comp rx=0 tx=0).stop 2024-09-16T04:00:54.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.508+0000 7f56c99a2640 10 monclient: shutdown 2024-09-16T04:00:54.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.508+0000 7f56c99a2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:00:54.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.508+0000 7f56c99a2640 1 -- 172.21.15.42:0/2723270600 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f56c411c9f0 msgr2=0x7f56c4133800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:00:54.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.508+0000 7f56c99a2640 1 --2- 172.21.15.42:0/2723270600 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f56c411c9f0 0x7f56c4133800 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f56b400c480 tx=0x7f56b400c940 comp rx=0 tx=0).stop 2024-09-16T04:00:54.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.508+0000 7f56c37fe640 1 -- 172.21.15.42:0/2723270600 reap_dead start 2024-09-16T04:00:54.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.508+0000 7f56c37fe640 1 -- 172.21.15.42:0/2723270600 reap_dead start 2024-09-16T04:00:54.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.508+0000 7f56c99a2640 1 -- 172.21.15.42:0/2723270600 shutdown_connections 2024-09-16T04:00:54.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.508+0000 7f56c99a2640 1 -- 172.21.15.42:0/2723270600 >> 172.21.15.42:0/2723270600 conn(0x7f56c40a2420 msgr2=0x7f56c40719f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:00:54.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.508+0000 7f56c99a2640 1 -- 172.21.15.42:0/2723270600 shutdown_connections 2024-09-16T04:00:54.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.508+0000 7f56c99a2640 1 -- 172.21.15.42:0/2723270600 wait complete. 2024-09-16T04:00:54.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.508+0000 7f56c99a2640 1 librados: shutdown 2024-09-16T04:00:54.547 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:00:54.547 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T04:00:54.548 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T04:00:54.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.668+0000 7f5f0ad40640 10 monclient: get_monmap_and_config 2024-09-16T04:00:54.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.668+0000 7f5f0ad40640 10 monclient: build_initial_monmap 2024-09-16T04:00:54.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.668+0000 7f5f0ad40640 10 monclient: monmap: 2024-09-16T04:00:54.675 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:00:54.675 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:00:54.675 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:00:54.673130+0000 2024-09-16T04:00:54.675 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:00:54.673130+0000 2024-09-16T04:00:54.675 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:00:54.675 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:00:54.675 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:00:54.675 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:00:54.675 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:00:54.675 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:00:54.675 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:00:54.675 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:00:54.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.668+0000 7f5f0ad40640 1 Processor -- start 2024-09-16T04:00:54.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.668+0000 7f5f0ad40640 1 -- start start 2024-09-16T04:00:54.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.668+0000 7f5f0ad40640 10 monclient: init 2024-09-16T04:00:54.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.668+0000 7f5f0ad40640 10 monclient: _reopen_session rank -1 2024-09-16T04:00:54.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.668+0000 7f5f0ad40640 10 monclient: _add_conns ranks=[2,3,0,1,4] 2024-09-16T04:00:54.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.668+0000 7f5f0ad40640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5f04137500 0x7f5f041378e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:00:54.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.668+0000 7f5f0ad40640 10 monclient(hunting): picked mon.noname-b con 0x7f5f04137500 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:00:54.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.668+0000 7f5f0ad40640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f5f0406e9f0 0x7f5f040a2340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:00:54.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.668+0000 7f5f0ad40640 10 monclient(hunting): picked mon.noname-a con 0x7f5f0406e9f0 addr v2:172.21.15.42:3300/0 2024-09-16T04:00:54.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.668+0000 7f5f0ad40640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5f0409fe60 addr v1:172.21.15.42:6789/0 2024-09-16T04:00:54.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.668+0000 7f5f0ad40640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5f0406c1a0 con 0x7f5f0409fe60 2024-09-16T04:00:54.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.668+0000 7f5f0ad40640 10 monclient(hunting): start opening mon connection 2024-09-16T04:00:54.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.668+0000 7f5f0ad40640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f5f040a0b20 con 0x7f5f0406e9f0 2024-09-16T04:00:54.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.668+0000 7f5f0ad40640 10 monclient(hunting): start opening mon connection 2024-09-16T04:00:54.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.668+0000 7f5f0ad40640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f5f040a0840 con 0x7f5f04137500 2024-09-16T04:00:54.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.668+0000 7f5f0ad40640 10 monclient(hunting): _renew_subs 2024-09-16T04:00:54.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.668+0000 7f5f0ad40640 10 monclient(hunting): authenticate will time out at 2101.643457s 2024-09-16T04:00:54.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.668+0000 7f5f08adc640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5f04137500 0x7f5f041378e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:00:54.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.668+0000 7f5f08adc640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5f04137500 0x7f5f041378e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:40690/0 (socket says 172.21.15.42:40690) 2024-09-16T04:00:54.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.668+0000 7f5f08adc640 1 -- 172.21.15.42:0/1299248151 learned_addr learned my addr 172.21.15.42:0/1299248151 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:00:54.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.668+0000 7f5f08adc640 10 monclient(hunting): get_auth_request con 0x7f5f04137500 auth_method 0 2024-09-16T04:00:54.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.668+0000 7f5f08adc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:00:54.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.668+0000 7f5f08adc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:00:54.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.668+0000 7f5f08adc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:00:54.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.668+0000 7f5f03fff640 1 --2- 172.21.15.42:0/1299248151 >> v2:172.21.15.42:3300/0 conn(0x7f5f0406e9f0 0x7f5f040a2340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:00:54.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.668+0000 7f5f03fff640 10 monclient(hunting): get_auth_request con 0x7f5f0406e9f0 auth_method 0 2024-09-16T04:00:54.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.668+0000 7f5f03fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:00:54.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.668+0000 7f5f03fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:00:54.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.668+0000 7f5f03fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:00:54.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.668+0000 7f5f03fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:00:54.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.668+0000 7f5f03fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:00:54.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.668+0000 7f5f03fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:00:54.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.668+0000 7f5f08adc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:00:54.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.668+0000 7f5f08adc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:00:54.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.668+0000 7f5f08adc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:00:54.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.672+0000 7f5f03fff640 10 monclient(hunting): handle_auth_done global_id 825276 payload 274 2024-09-16T04:00:54.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.672+0000 7f5f03fff640 1 -- 172.21.15.42:0/1299248151 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5f04137500 msgr2=0x7f5f041378e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:00:54.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.672+0000 7f5f037fe640 1 -- 172.21.15.42:0/1299248151 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1319649143 0 0) 0x7f5f0406c1a0 con 0x7f5f0409fe60 2024-09-16T04:00:54.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.672+0000 7f5f03fff640 1 --2- 172.21.15.42:0/1299248151 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5f04137500 0x7f5f041378e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:00:54.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.672+0000 7f5f03fff640 1 -- 172.21.15.42:0/1299248151 >> v1:172.21.15.42:6789/0 conn(0x7f5f0409fe60 legacy=0x7f5f040a2880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:00:54.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.672+0000 7f5f03fff640 10 monclient: _finish_hunting 0 2024-09-16T04:00:54.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.672+0000 7f5f03fff640 1 monclient: found mon.noname-a 2024-09-16T04:00:54.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.672+0000 7f5f03fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:00:54.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.672+0000 7f5f03fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:00:54.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.672+0000 7f5f03fff640 1 -- 172.21.15.42:0/1299248151 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f040a0560 con 0x7f5f0406e9f0 2024-09-16T04:00:54.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.672+0000 7f5f037fe640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T04:00:54.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.672+0000 7f5f08adc640 1 --2- 172.21.15.42:0/1299248151 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5f04137500 0x7f5f041378e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:00:54.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.672+0000 7f5f03fff640 1 --2- 172.21.15.42:0/1299248151 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5f0406e9f0 0x7f5f040a2340 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f5ef00097c0 tx=0x7f5ef001c360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50c56d015fdaa062 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:00:54.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.672+0000 7f5f037fe640 1 -- 172.21.15.42:0/1299248151 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ef00b2070 con 0x7f5f0406e9f0 2024-09-16T04:00:54.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.672+0000 7f5f037fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:00:54.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.672+0000 7f5f037fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:00:54.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.672+0000 7f5f037fe640 10 monclient: dump: 2024-09-16T04:00:54.696 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:00:54.696 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:00:54.696 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:00:54.696 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:00:54.696 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:00:54.696 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:00:54.696 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:00:54.696 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:00:54.696 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:00:54.696 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:00:54.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.672+0000 7f5f037fe640 10 monclient: _finish_auth 0 2024-09-16T04:00:54.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.672+0000 7f5f037fe640 10 monclient: _check_auth_tickets 2024-09-16T04:00:54.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.672+0000 7f5f037fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:00:54.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.672+0000 7f5f037fe640 1 -- 172.21.15.42:0/1299248151 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ef00ad680 con 0x7f5f0406e9f0 2024-09-16T04:00:54.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.672+0000 7f5f0ad40640 5 monclient: authenticate success, global_id 825276 2024-09-16T04:00:54.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.672+0000 7f5f037fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:00:54.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.672+0000 7f5f0ad40640 10 monclient: get_monmap_and_config success 2024-09-16T04:00:54.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.672+0000 7f5f037fe640 1 -- 172.21.15.42:0/1299248151 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ef00ada50 con 0x7f5f0406e9f0 2024-09-16T04:00:54.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.672+0000 7f5f0ad40640 4 set_mon_vals no callback set 2024-09-16T04:00:54.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.672+0000 7f5f037fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:00:54.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.672+0000 7f5f037fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:00:54.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.672+0000 7f5f037fe640 10 monclient: dump: 2024-09-16T04:00:54.698 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:00:54.698 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:00:54.698 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:00:54.698 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:00:54.698 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:00:54.698 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:00:54.698 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:00:54.698 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:00:54.699 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:00:54.699 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:00:54.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.672+0000 7f5f0ad40640 10 monclient: shutdown 2024-09-16T04:00:54.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.672+0000 7f5f0ad40640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:00:54.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.672+0000 7f5f0ad40640 1 -- 172.21.15.42:0/1299248151 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5f0406e9f0 msgr2=0x7f5f040a2340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:00:54.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.672+0000 7f5f0ad40640 1 --2- 172.21.15.42:0/1299248151 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5f0406e9f0 0x7f5f040a2340 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f5ef00097c0 tx=0x7f5ef001c360 comp rx=0 tx=0).stop 2024-09-16T04:00:54.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.672+0000 7f5f0ad40640 1 -- 172.21.15.42:0/1299248151 shutdown_connections 2024-09-16T04:00:54.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.672+0000 7f5f0ad40640 1 --2- 172.21.15.42:0/1299248151 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5f0406e9f0 0x7f5f040a2340 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:00:54.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.672+0000 7f5f0ad40640 1 --2- 172.21.15.42:0/1299248151 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5f04137500 0x7f5f041378e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:00:54.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.672+0000 7f5f0ad40640 1 -- 172.21.15.42:0/1299248151 >> 172.21.15.42:0/1299248151 conn(0x7f5f040a3430 msgr2=0x7f5f040a3810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:00:54.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.672+0000 7f5f0ad40640 1 -- 172.21.15.42:0/1299248151 shutdown_connections 2024-09-16T04:00:54.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.672+0000 7f5f0ad40640 1 -- 172.21.15.42:0/1299248151 wait complete. 2024-09-16T04:00:54.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.672+0000 7f5f0ad40640 10 monclient: build_initial_monmap 2024-09-16T04:00:54.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.672+0000 7f5f0ad40640 10 monclient: monmap: 2024-09-16T04:00:54.700 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:00:54.700 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:00:54.700 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:00:54.700 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:00:54.700 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:00:54.700 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:00:54.700 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:00:54.701 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:00:54.701 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:00:54.701 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:00:54.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.676+0000 7f5f0ad40640 1 librados: starting msgr at 2024-09-16T04:00:54.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.676+0000 7f5f0ad40640 1 librados: starting objecter 2024-09-16T04:00:54.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.676+0000 7f5f0ad40640 1 Processor -- start 2024-09-16T04:00:54.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.676+0000 7f5f0ad40640 1 -- start start 2024-09-16T04:00:54.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.676+0000 7f5f0ad40640 1 librados: setting wanted keys 2024-09-16T04:00:54.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.676+0000 7f5f0ad40640 1 librados: calling monclient init 2024-09-16T04:00:54.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.676+0000 7f5f0ad40640 10 monclient: init 2024-09-16T04:00:54.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.676+0000 7f5f0ad40640 10 monclient: _reopen_session rank -1 2024-09-16T04:00:54.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.676+0000 7f5f0ad40640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:00:54.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.676+0000 7f5f0ad40640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5f0409fe60 0x7f5f04135ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:00:54.702 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.676+0000 7f5f0ad40640 10 monclient(hunting): picked mon.noname-b con 0x7f5f0409fe60 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:00:54.702 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.676+0000 7f5f0ad40640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5f04137500 0x7f5f04136410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:00:54.702 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.676+0000 7f5f0ad40640 10 monclient(hunting): picked mon.noname-a con 0x7f5f04137500 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:00:54.702 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.676+0000 7f5f0ad40640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5f0411cf80 0x7f5f0411d400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:00:54.702 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.676+0000 7f5f0ad40640 10 monclient(hunting): picked mon.noname-c con 0x7f5f0411cf80 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:00:54.702 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.676+0000 7f5f0ad40640 10 monclient(hunting): start opening mon connection 2024-09-16T04:00:54.702 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.676+0000 7f5f0ad40640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f5f04141d40 con 0x7f5f04137500 2024-09-16T04:00:54.702 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.676+0000 7f5f0ad40640 10 monclient(hunting): start opening mon connection 2024-09-16T04:00:54.702 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.676+0000 7f5f0ad40640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f5f04141bc0 con 0x7f5f0409fe60 2024-09-16T04:00:54.702 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.676+0000 7f5f0ad40640 10 monclient(hunting): start opening mon connection 2024-09-16T04:00:54.702 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.676+0000 7f5f0ad40640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f5f04141ec0 con 0x7f5f0411cf80 2024-09-16T04:00:54.702 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.676+0000 7f5f0ad40640 10 monclient(hunting): _renew_subs 2024-09-16T04:00:54.702 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.676+0000 7f5f0ad40640 10 monclient(hunting): authenticate will time out at 2101.648931s 2024-09-16T04:00:54.702 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.676+0000 7f5f03fff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5f04137500 0x7f5f04136410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:00:54.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.676+0000 7f5f092dd640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5f0411cf80 0x7f5f0411d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:00:54.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.676+0000 7f5f03fff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5f04137500 0x7f5f04136410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:44092/0 (socket says 172.21.15.42:44092) 2024-09-16T04:00:54.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.676+0000 7f5f03fff640 1 -- 172.21.15.42:0/4181597950 learned_addr learned my addr 172.21.15.42:0/4181597950 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:00:54.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.676+0000 7f5f03fff640 10 monclient(hunting): get_auth_request con 0x7f5f04137500 auth_method 0 2024-09-16T04:00:54.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.676+0000 7f5f03fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:00:54.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.676+0000 7f5f03fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:00:54.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.676+0000 7f5f03fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:00:54.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.676+0000 7f5f092dd640 10 monclient(hunting): get_auth_request con 0x7f5f0411cf80 auth_method 0 2024-09-16T04:00:54.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.676+0000 7f5f092dd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:00:54.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.676+0000 7f5f092dd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:00:54.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.676+0000 7f5f092dd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:00:54.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.676+0000 7f5f03fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:00:54.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.676+0000 7f5f03fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:00:54.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.676+0000 7f5f03fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:00:54.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.676+0000 7f5f092dd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:00:54.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.676+0000 7f5f092dd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:00:54.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.676+0000 7f5f092dd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:00:54.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.676+0000 7f5f08adc640 1 --2- 172.21.15.42:0/4181597950 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5f0409fe60 0x7f5f04135ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:00:54.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.676+0000 7f5f08adc640 10 monclient(hunting): get_auth_request con 0x7f5f0409fe60 auth_method 0 2024-09-16T04:00:54.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.676+0000 7f5f08adc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:00:54.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.676+0000 7f5f08adc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:00:54.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.676+0000 7f5f08adc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:00:54.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.676+0000 7f5f03fff640 10 monclient(hunting): handle_auth_done global_id 825282 payload 824 2024-09-16T04:00:54.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.676+0000 7f5f03fff640 1 -- 172.21.15.42:0/4181597950 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5f0411cf80 msgr2=0x7f5f0411d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:00:54.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.676+0000 7f5f03fff640 1 --2- 172.21.15.42:0/4181597950 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5f0411cf80 0x7f5f0411d400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:00:54.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.676+0000 7f5f03fff640 1 -- 172.21.15.42:0/4181597950 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5f0409fe60 msgr2=0x7f5f04135ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:00:54.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.676+0000 7f5f03fff640 1 --2- 172.21.15.42:0/4181597950 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5f0409fe60 0x7f5f04135ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:00:54.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.676+0000 7f5f03fff640 10 monclient: _finish_hunting 0 2024-09-16T04:00:54.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.676+0000 7f5f03fff640 1 monclient: found mon.noname-a 2024-09-16T04:00:54.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.676+0000 7f5f03fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:00:54.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.676+0000 7f5f03fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:00:54.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.676+0000 7f5f03fff640 1 -- 172.21.15.42:0/4181597950 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f0411efc0 con 0x7f5f04137500 2024-09-16T04:00:54.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.676+0000 7f5f092dd640 1 --2- 172.21.15.42:0/4181597950 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5f0411cf80 0x7f5f0411d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:00:54.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.676+0000 7f5f03fff640 1 --2- 172.21.15.42:0/4181597950 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5f04137500 0x7f5f04136410 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f5ef0019f90 tx=0x7f5ef001aba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:00:54.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.676+0000 7f5f017fa640 10 client.?.objecter ms_handle_connect 0x7f5f04137500 2024-09-16T04:00:54.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.676+0000 7f5f017fa640 10 client.?.objecter resend_mon_ops 2024-09-16T04:00:54.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.676+0000 7f5f017fa640 1 -- 172.21.15.42:0/4181597950 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ef00b2040 con 0x7f5f04137500 2024-09-16T04:00:54.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.676+0000 7f5f017fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:00:54.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.676+0000 7f5f017fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:00:54.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.676+0000 7f5f017fa640 10 monclient: dump: 2024-09-16T04:00:54.706 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:00:54.706 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:00:54.706 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:00:54.706 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:00:54.706 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:00:54.706 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:00:54.706 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:00:54.706 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:00:54.706 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:00:54.706 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:00:54.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.676+0000 7f5f017fa640 10 monclient: _finish_auth 0 2024-09-16T04:00:54.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.676+0000 7f5f017fa640 10 monclient: _check_auth_tickets 2024-09-16T04:00:54.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.676+0000 7f5f017fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:00:54.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.676+0000 7f5f017fa640 1 -- 172.21.15.42:0/4181597950 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ef00bd070 con 0x7f5f04137500 2024-09-16T04:00:54.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.676+0000 7f5f0ad40640 5 monclient: authenticate success, global_id 825282 2024-09-16T04:00:54.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.676+0000 7f5f0ad40640 10 monclient: _renew_subs 2024-09-16T04:00:54.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.676+0000 7f5f0ad40640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:00:54.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.676+0000 7f5f0ad40640 1 -- 172.21.15.42:0/4181597950 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f041209d0 con 0x7f5f04137500 2024-09-16T04:00:54.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.676+0000 7f5f017fa640 10 monclient: handle_config config(0 keys) 2024-09-16T04:00:54.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.676+0000 7f5f017fa640 1 -- 172.21.15.42:0/4181597950 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ef00add50 con 0x7f5f04137500 2024-09-16T04:00:54.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.676+0000 7f5f017fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:00:54.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.676+0000 7f5f017fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:00:54.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.676+0000 7f5f017fa640 10 monclient: dump: 2024-09-16T04:00:54.707 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:00:54.707 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:00:54.707 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:00:54.708 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:00:54.708 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:00:54.708 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:00:54.708 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:00:54.708 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:00:54.708 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:00:54.708 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:00:54.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.676+0000 7f5f02ffd640 4 set_mon_vals no callback set 2024-09-16T04:00:54.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.676+0000 7f5f0ad40640 10 client.825282.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:00:54.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.676+0000 7f5f0ad40640 10 monclient: _renew_subs 2024-09-16T04:00:54.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.676+0000 7f5f0ad40640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:00:54.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.676+0000 7f5f0ad40640 1 -- 172.21.15.42:0/4181597950 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f04071910 con 0x7f5f04137500 2024-09-16T04:00:54.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.676+0000 7f5f0ad40640 1 librados: init done 2024-09-16T04:00:54.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.676+0000 7f5f0ad40640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:00:54.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.676+0000 7f5f0ad40640 1 --2- 172.21.15.42:0/4181597950 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5f0406e9f0 0x7f5f04120240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:00:54.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.676+0000 7f5f0ad40640 10 monclient(hunting): start opening mon connection 2024-09-16T04:00:54.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.676+0000 7f5f0ad40640 1 -- 172.21.15.42:0/4181597950 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f5f04073a10 con 0x7f5f0406e9f0 2024-09-16T04:00:54.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.676+0000 7f5f017fa640 1 -- 172.21.15.42:0/4181597950 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f5ef00c2460 con 0x7f5f04137500 2024-09-16T04:00:54.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.676+0000 7f5f08adc640 1 --2- 172.21.15.42:0/4181597950 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5f0406e9f0 0x7f5f04120240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:00:54.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.676+0000 7f5f08adc640 10 monclient: get_auth_request con 0x7f5f0406e9f0 auth_method 0 2024-09-16T04:00:54.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.676+0000 7f5f08adc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:00:54.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.676+0000 7f5f08adc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:00:54.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.676+0000 7f5f08adc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:00:54.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.676+0000 7f5f017fa640 1 --2- 172.21.15.42:0/4181597950 >> v2:172.21.15.154:6812/254845342 conn(0x7f5ee8037810 0x7f5ee8039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:00:54.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.676+0000 7f5f017fa640 1 -- 172.21.15.42:0/4181597950 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(567..567 src has 451..567) ==== 6224+0+0 (secure 0 0 0) 0x7f5ef00e3130 con 0x7f5f04137500 2024-09-16T04:00:54.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.676+0000 7f5f017fa640 10 client.825282.objecter ms_dispatch 0x7f5f04000ba0 osd_map(567..567 src has 451..567) 2024-09-16T04:00:54.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.676+0000 7f5f08adc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:00:54.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.676+0000 7f5f017fa640 3 client.825282.objecter handle_osd_map got epochs [567,567] > 0 2024-09-16T04:00:54.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.676+0000 7f5f017fa640 3 client.825282.objecter handle_osd_map decoding full epoch 567 2024-09-16T04:00:54.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.676+0000 7f5f08adc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:00:54.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.676+0000 7f5f08adc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:00:54.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.676+0000 7f5f017fa640 20 client.825282.objecter dump_active .. 0 homeless 2024-09-16T04:00:54.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.676+0000 7f5f092dd640 1 --2- 172.21.15.42:0/4181597950 >> v2:172.21.15.154:6812/254845342 conn(0x7f5ee8037810 0x7f5ee8039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:00:54.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.676+0000 7f5f092dd640 10 monclient: get_auth_request con 0x7f5ee8037810 auth_method 0 2024-09-16T04:00:54.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.676+0000 7f5f08adc640 10 monclient(hunting): handle_auth_done global_id 904423 payload 824 2024-09-16T04:00:54.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.676+0000 7f5f08adc640 1 -- 172.21.15.42:0/4181597950 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5f041378e0 con 0x7f5f0406e9f0 2024-09-16T04:00:54.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.676+0000 7f5f017fa640 10 client.825282.objecter ms_handle_connect 0x7f5f0406e9f0 2024-09-16T04:00:54.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.676+0000 7f5f017fa640 10 client.825282.objecter resend_mon_ops 2024-09-16T04:00:54.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.676+0000 7f5f08adc640 1 --2- 172.21.15.42:0/4181597950 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5f0406e9f0 0x7f5f04120240 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f5ef8011600 tx=0x7f5ef8011ac0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:00:54.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.680+0000 7f5f017fa640 1 -- 172.21.15.42:0/4181597950 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ef800f930 con 0x7f5f0406e9f0 2024-09-16T04:00:54.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.680+0000 7f5f017fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:00:54.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.680+0000 7f5f017fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:00:54.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.680+0000 7f5f092dd640 1 --2- 172.21.15.42:0/4181597950 >> v2:172.21.15.154:6812/254845342 conn(0x7f5ee8037810 0x7f5ee8039cd0 secure :-1 s=READY pgs=3748 cs=0 l=1 rev1=1 crypto rx=0x7f5efc0012f0 tx=0x7f5efc009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:00:54.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.680+0000 7f5f017fa640 10 monclient: dump: 2024-09-16T04:00:54.711 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:00:54.711 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:00:54.711 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:00:54.712 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:00:54.712 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:00:54.712 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:00:54.712 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:00:54.712 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:00:54.712 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:00:54.712 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:00:54.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.680+0000 7f5f017fa640 10 client.825282.objecter ms_handle_connect 0x7f5ee8037810 2024-09-16T04:00:54.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.680+0000 7f5f017fa640 1 -- 172.21.15.42:0/4181597950 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f5ef800f930 con 0x7f5f0406e9f0 2024-09-16T04:00:54.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.680+0000 7f5f017fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:00:54.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.680+0000 7f5f017fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:00:54.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.680+0000 7f5f017fa640 1 -- 172.21.15.42:0/4181597950 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5f0406e9f0 msgr2=0x7f5f04120240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:00:54.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.680+0000 7f5f017fa640 1 --2- 172.21.15.42:0/4181597950 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5f0406e9f0 0x7f5f04120240 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f5ef8011600 tx=0x7f5ef8011ac0 comp rx=0 tx=0).stop 2024-09-16T04:00:54.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.680+0000 7f5f017fa640 1 -- 172.21.15.42:0/4181597950 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5f0406e9f0 msgr2=0x7f5f04120240 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:00:54.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.680+0000 7f5f017fa640 1 --2- 172.21.15.42:0/4181597950 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5f0406e9f0 0x7f5f04120240 secure :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f5ef8011600 tx=0x7f5ef8011ac0 comp rx=0 tx=0).stop 2024-09-16T04:00:54.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.684+0000 7f5f0ad40640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:00:54.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.684+0000 7f5f0ad40640 1 --2- 172.21.15.42:0/4181597950 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5f0411e100 0x7f5f0402c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:00:54.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.684+0000 7f5f0ad40640 10 monclient(hunting): start opening mon connection 2024-09-16T04:00:54.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.684+0000 7f5f0ad40640 1 -- 172.21.15.42:0/4181597950 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f5f0413b870 con 0x7f5f0411e100 2024-09-16T04:00:54.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.684+0000 7f5f08adc640 1 --2- 172.21.15.42:0/4181597950 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5f0411e100 0x7f5f0402c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:00:54.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.684+0000 7f5f08adc640 10 monclient: get_auth_request con 0x7f5f0411e100 auth_method 0 2024-09-16T04:00:54.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.684+0000 7f5f08adc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:00:54.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.684+0000 7f5f08adc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:00:54.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.684+0000 7f5f08adc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:00:54.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.684+0000 7f5f08adc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:00:54.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.684+0000 7f5f08adc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:00:54.714 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.684+0000 7f5f08adc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:00:54.714 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.684+0000 7f5f08adc640 10 monclient(hunting): handle_auth_done global_id 904426 payload 824 2024-09-16T04:00:54.714 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.684+0000 7f5f08adc640 1 -- 172.21.15.42:0/4181597950 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f5f04120780 con 0x7f5f0411e100 2024-09-16T04:00:54.714 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.688+0000 7f5f08adc640 1 --2- 172.21.15.42:0/4181597950 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5f0411e100 0x7f5f0402c650 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f5ef8000c10 tx=0x7f5ef80046d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:00:54.714 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.688+0000 7f5f017fa640 10 client.825282.objecter ms_handle_connect 0x7f5f0411e100 2024-09-16T04:00:54.714 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.688+0000 7f5f017fa640 10 client.825282.objecter resend_mon_ops 2024-09-16T04:00:54.714 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.688+0000 7f5f017fa640 1 -- 172.21.15.42:0/4181597950 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ef80064d0 con 0x7f5f0411e100 2024-09-16T04:00:54.714 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.688+0000 7f5f017fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:00:54.714 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.688+0000 7f5f017fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:00:54.714 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.688+0000 7f5f017fa640 10 monclient: dump: 2024-09-16T04:00:54.714 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:00:54.714 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:00:54.714 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:00:54.714 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:00:54.715 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:00:54.715 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:00:54.715 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:00:54.715 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:00:54.715 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:00:54.715 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:00:54.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.688+0000 7f5f017fa640 1 -- 172.21.15.42:0/4181597950 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4880 (secure 0 0 0) 0x7f5ef8026070 con 0x7f5f0411e100 2024-09-16T04:00:54.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.688+0000 7f5f017fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:00:54.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.688+0000 7f5f017fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:00:54.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.688+0000 7f5f017fa640 1 -- 172.21.15.42:0/4181597950 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5f0411e100 msgr2=0x7f5f0402c650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:00:54.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.688+0000 7f5f017fa640 1 --2- 172.21.15.42:0/4181597950 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5f0411e100 0x7f5f0402c650 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f5ef8000c10 tx=0x7f5ef80046d0 comp rx=0 tx=0).stop 2024-09-16T04:00:54.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.688+0000 7f5f017fa640 1 -- 172.21.15.42:0/4181597950 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5f0411e100 msgr2=0x7f5f0402c650 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:00:54.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.688+0000 7f5f017fa640 1 --2- 172.21.15.42:0/4181597950 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5f0411e100 0x7f5f0402c650 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:00:54.716 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T04:00:54.716 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:00:54.716 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:00:54.716 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T04:00:54.716 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 168, 2024-09-16T04:00:54.716 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 182870, 2024-09-16T04:00:54.716 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T04:00:54.716 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T04:00:54.716 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T04:00:54.716 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T04:00:54.717 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:00:54.717 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 3, 2024-09-16T04:00:54.717 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:00:54.717 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T04:00:54.717 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T04:00:54.717 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:00:54.717 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:00:54.717 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:00:54.717 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:00:54.717 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:00:54.717 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:00:54.717 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:00:54.717 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:00:54.717 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:00:54.718 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:00:54.718 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:00:54.718 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:00:54.718 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:00:54.718 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T04:00:54.718 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:00:54.718 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:00:54.718 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:00:54.718 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:00:54.718 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:00:54.718 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:00:54.718 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:00:54.718 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:00:54.718 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:00:54.719 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:00:54.719 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:00:54.719 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:00:54.719 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:00:54.719 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T04:00:54.719 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T04:00:54.719 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T04:00:54.719 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T04:00:54.719 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T04:00:54.719 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T04:00:54.719 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:00:54.719 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:00:54.719 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T04:00:54.719 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T04:00:54.720 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T04:00:54.720 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T04:00:54.720 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T04:00:54.720 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T04:00:54.720 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T04:00:54.720 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:00:54.720 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T04:00:54.720 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:00:54.720 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:00:54.720 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:00:54.720 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:00:54.720 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:00:54.720 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:00:54.721 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:00:54.721 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:00:54.721 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:00:54.721 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:00:54.721 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:00:54.721 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:00:54.721 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T04:00:54.721 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:00:54.721 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T04:00:54.721 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:00:54.721 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:00:54.721 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:00:54.721 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:00:54.721 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:00:54.722 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:00:54.722 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:00:54.722 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T04:00:54.722 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:00:54.722 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:00:54.722 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:00:54.722 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:00:54.722 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T04:00:54.722 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:00:54.723 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:00:54.723 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:00:54.723 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:00:54.723 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T04:00:54.723 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T04:00:54.723 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:00:54.723 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:00:54.723 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:00:54.723 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:00:54.723 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:00:54.723 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:00:54.723 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:00:54.723 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:00:54.723 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:00:54.724 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:00:54.724 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:00:54.724 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T04:00:54.724 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:00:54.724 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:00:54.724 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:00:54.724 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:00:54.724 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T04:00:54.724 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:00:54.724 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:00:54.724 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:00:54.724 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:00:54.724 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T04:00:54.725 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T04:00:54.725 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:00:54.725 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:00:54.725 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:00:54.725 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:00:54.725 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:00:54.725 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:00:54.725 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:00:54.725 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:00:54.725 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:00:54.725 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:00:54.725 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:00:54.725 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T04:00:54.725 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:00:54.725 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:00:54.726 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:00:54.726 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:00:54.726 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T04:00:54.726 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:00:54.726 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:00:54.726 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:00:54.726 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:00:54.726 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T04:00:54.726 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T04:00:54.726 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:00:54.726 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:00:54.726 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:00:54.726 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:00:54.726 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:00:54.727 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:00:54.727 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T04:00:54.727 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T04:00:54.727 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:00:54.727 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:00:54.727 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:00:54.727 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:00:54.727 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:00:54.727 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:00:54.727 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T04:00:54.727 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:00:54.727 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:00:54.727 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:00:54.727 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:00:54.728 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:00:54.728 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:00:54.728 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T04:00:54.728 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:00:54.728 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:00:54.728 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:00:54.728 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 2 2024-09-16T04:00:54.728 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:00:54.728 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:00:54.728 INFO:teuthology.orchestra.run.smithi042.stdout: "mgr": [ 2024-09-16T04:00:54.728 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:00:54.728 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:00:54.728 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:00:54.729 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:00:54.729 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:00:54.729 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:00:54.729 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:00:54.729 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T04:00:54.729 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T04:00:54.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.688+0000 7f5f0ad40640 10 librados: watch_flush enter 2024-09-16T04:00:54.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.688+0000 7f5f0ad40640 10 librados: watch_flush exit 2024-09-16T04:00:54.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.688+0000 7f5f0ad40640 20 client.825282.objecter shutdown clearing up homeless session... 2024-09-16T04:00:54.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.688+0000 7f5f0ad40640 10 client.825282.objecter successfully canceled tick 2024-09-16T04:00:54.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.688+0000 7f5f0ad40640 1 -- 172.21.15.42:0/4181597950 >> v2:172.21.15.154:6812/254845342 conn(0x7f5ee8037810 msgr2=0x7f5ee8039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:00:54.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.688+0000 7f5f0ad40640 1 --2- 172.21.15.42:0/4181597950 >> v2:172.21.15.154:6812/254845342 conn(0x7f5ee8037810 0x7f5ee8039cd0 secure :-1 s=READY pgs=3748 cs=0 l=1 rev1=1 crypto rx=0x7f5efc0012f0 tx=0x7f5efc009040 comp rx=0 tx=0).stop 2024-09-16T04:00:54.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.688+0000 7f5f0ad40640 10 monclient: shutdown 2024-09-16T04:00:54.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.688+0000 7f5f0ad40640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:00:54.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.688+0000 7f5f0ad40640 1 -- 172.21.15.42:0/4181597950 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5f04137500 msgr2=0x7f5f04136410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:00:54.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.688+0000 7f5f0ad40640 1 --2- 172.21.15.42:0/4181597950 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5f04137500 0x7f5f04136410 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f5ef0019f90 tx=0x7f5ef001aba0 comp rx=0 tx=0).stop 2024-09-16T04:00:54.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.688+0000 7f5f092dd640 1 -- 172.21.15.42:0/4181597950 reap_dead start 2024-09-16T04:00:54.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.688+0000 7f5f092dd640 1 -- 172.21.15.42:0/4181597950 reap_dead start 2024-09-16T04:00:54.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.688+0000 7f5f0ad40640 1 -- 172.21.15.42:0/4181597950 shutdown_connections 2024-09-16T04:00:54.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.688+0000 7f5f0ad40640 1 -- 172.21.15.42:0/4181597950 >> 172.21.15.42:0/4181597950 conn(0x7f5f040a3430 msgr2=0x7f5f0406ee50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:00:54.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.688+0000 7f5f0ad40640 1 -- 172.21.15.42:0/4181597950 shutdown_connections 2024-09-16T04:00:54.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.688+0000 7f5f0ad40640 1 -- 172.21.15.42:0/4181597950 wait complete. 2024-09-16T04:00:54.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.688+0000 7f5f0ad40640 1 librados: shutdown 2024-09-16T04:00:54.732 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:00:54.732 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T04:00:54.733 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T04:00:54.844 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.836+0000 7f0097c8b640 10 monclient: get_monmap_and_config 2024-09-16T04:00:54.844 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.836+0000 7f0097c8b640 10 monclient: build_initial_monmap 2024-09-16T04:00:54.844 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.836+0000 7f0097c8b640 10 monclient: monmap: 2024-09-16T04:00:54.844 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:00:54.844 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:00:54.844 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:00:54.842775+0000 2024-09-16T04:00:54.844 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:00:54.842775+0000 2024-09-16T04:00:54.845 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:00:54.845 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:00:54.845 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:00:54.845 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:00:54.845 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:00:54.845 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:00:54.845 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:00:54.845 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:00:54.847 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.836+0000 7f0097c8b640 1 Processor -- start 2024-09-16T04:00:54.847 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.836+0000 7f0097c8b640 1 -- start start 2024-09-16T04:00:54.847 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.836+0000 7f0097c8b640 10 monclient: init 2024-09-16T04:00:54.847 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.836+0000 7f0097c8b640 10 monclient: _reopen_session rank -1 2024-09-16T04:00:54.847 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.836+0000 7f0097c8b640 10 monclient: _add_conns ranks=[1,4,2,3,0] 2024-09-16T04:00:54.847 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.836+0000 7f0097c8b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f00900a39b0 addr v1:172.21.15.154:6789/0 2024-09-16T04:00:54.847 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.836+0000 7f0097c8b640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f00900a3d90 0x7f009013c380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:00:54.848 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.836+0000 7f0097c8b640 10 monclient(hunting): picked mon.noname-c con 0x7f00900a3d90 addr v2:172.21.15.154:3300/0 2024-09-16T04:00:54.848 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.840+0000 7f0097c8b640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f009013c8c0 0x7f009013ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:00:54.848 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.840+0000 7f0097c8b640 10 monclient(hunting): picked mon.noname-b con 0x7f009013c8c0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:00:54.848 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.840+0000 7f0097c8b640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0090134290 con 0x7f00900a39b0 2024-09-16T04:00:54.848 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.840+0000 7f0097c8b640 10 monclient(hunting): start opening mon connection 2024-09-16T04:00:54.848 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.840+0000 7f0097c8b640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f009009c140 con 0x7f00900a3d90 2024-09-16T04:00:54.848 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.840+0000 7f0097c8b640 10 monclient(hunting): start opening mon connection 2024-09-16T04:00:54.848 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.840+0000 7f0097c8b640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f009009c420 con 0x7f009013c8c0 2024-09-16T04:00:54.848 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.840+0000 7f0097c8b640 10 monclient(hunting): _renew_subs 2024-09-16T04:00:54.848 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.840+0000 7f0097c8b640 10 monclient(hunting): authenticate will time out at 2101.812047s 2024-09-16T04:00:54.848 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.840+0000 7f0096228640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f009013c8c0 0x7f009013ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:00:54.848 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.840+0000 7f0096228640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f009013c8c0 0x7f009013ecb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:40720/0 (socket says 172.21.15.42:40720) 2024-09-16T04:00:54.849 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.840+0000 7f0096228640 1 -- 172.21.15.42:0/4282770146 learned_addr learned my addr 172.21.15.42:0/4282770146 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:00:54.849 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.840+0000 7f0096228640 10 monclient(hunting): get_auth_request con 0x7f009013c8c0 auth_method 0 2024-09-16T04:00:54.849 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.840+0000 7f0096228640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:00:54.849 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.840+0000 7f0096228640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:00:54.849 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.840+0000 7f0096228640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:00:54.849 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.840+0000 7f0096228640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:00:54.849 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.840+0000 7f0096228640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:00:54.849 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.840+0000 7f0096228640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:00:54.849 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.840+0000 7f0095226640 1 --2- 172.21.15.42:0/4282770146 >> v2:172.21.15.154:3300/0 conn(0x7f00900a3d90 0x7f009013c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:00:54.849 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.840+0000 7f0095226640 10 monclient(hunting): get_auth_request con 0x7f00900a3d90 auth_method 0 2024-09-16T04:00:54.849 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.840+0000 7f0095226640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:00:54.849 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.840+0000 7f0095226640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:00:54.849 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.840+0000 7f0095226640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:00:54.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.840+0000 7f0096228640 10 monclient(hunting): handle_auth_done global_id 944123 payload 274 2024-09-16T04:00:54.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.840+0000 7f0096228640 1 -- 172.21.15.42:0/4282770146 >> v2:172.21.15.154:3300/0 conn(0x7f00900a3d90 msgr2=0x7f009013c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:00:54.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.840+0000 7f0096228640 1 --2- 172.21.15.42:0/4282770146 >> v2:172.21.15.154:3300/0 conn(0x7f00900a3d90 0x7f009013c380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:00:54.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.840+0000 7f0096228640 1 -- 172.21.15.42:0/4282770146 >> v1:172.21.15.154:6789/0 conn(0x7f00900a39b0 legacy=0x7f0090133b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:00:54.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.840+0000 7f0096228640 10 monclient: _finish_hunting 0 2024-09-16T04:00:54.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.840+0000 7f0096228640 1 monclient: found mon.noname-b 2024-09-16T04:00:54.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.840+0000 7f0096228640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:00:54.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.840+0000 7f0096228640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T04:00:54.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.840+0000 7f0096228640 1 -- 172.21.15.42:0/4282770146 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f009006dd60 con 0x7f009013c8c0 2024-09-16T04:00:54.859 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.840+0000 7f0096228640 1 --2- 172.21.15.42:0/4282770146 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f009013c8c0 0x7f009013ecb0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f0088009e50 tx=0x7f008801c8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3ee6d1074bb0067d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:00:54.859 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.840+0000 7f0094a25640 1 -- 172.21.15.42:0/4282770146 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00880a5530 con 0x7f009013c8c0 2024-09-16T04:00:54.859 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.840+0000 7f0094a25640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:00:54.859 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.840+0000 7f0094a25640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:00:54.859 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.840+0000 7f0094a25640 10 monclient: dump: 2024-09-16T04:00:54.859 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:00:54.859 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:00:54.859 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:00:54.859 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:00:54.859 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:00:54.859 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:00:54.859 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:00:54.860 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:00:54.860 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:00:54.860 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:00:54.860 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.840+0000 7f0094a25640 10 monclient: _finish_auth 0 2024-09-16T04:00:54.860 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.840+0000 7f0094a25640 10 monclient: _check_auth_tickets 2024-09-16T04:00:54.860 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.840+0000 7f0094a25640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:00:54.860 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.840+0000 7f0097c8b640 5 monclient: authenticate success, global_id 944123 2024-09-16T04:00:54.860 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.840+0000 7f0097c8b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:00:54.860 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.840+0000 7f0094a25640 1 -- 172.21.15.42:0/4282770146 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f00880a56d0 con 0x7f009013c8c0 2024-09-16T04:00:54.860 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.840+0000 7f0094a25640 10 monclient: handle_config config(0 keys) 2024-09-16T04:00:54.860 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.840+0000 7f0097c8b640 10 monclient: get_monmap_and_config success 2024-09-16T04:00:54.860 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.840+0000 7f0097c8b640 4 set_mon_vals no callback set 2024-09-16T04:00:54.861 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.840+0000 7f0094a25640 1 -- 172.21.15.42:0/4282770146 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00880a5aa0 con 0x7f009013c8c0 2024-09-16T04:00:54.861 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.840+0000 7f0094a25640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:00:54.861 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.840+0000 7f0094a25640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:00:54.861 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.840+0000 7f0094a25640 10 monclient: dump: 2024-09-16T04:00:54.861 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:00:54.861 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:00:54.861 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:00:54.862 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:00:54.862 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:00:54.862 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:00:54.862 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:00:54.862 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:00:54.862 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:00:54.862 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:00:54.862 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.840+0000 7f0097c8b640 10 monclient: shutdown 2024-09-16T04:00:54.862 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.840+0000 7f0097c8b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:00:54.862 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.840+0000 7f0097c8b640 1 -- 172.21.15.42:0/4282770146 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f009013c8c0 msgr2=0x7f009013ecb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:00:54.862 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.840+0000 7f0097c8b640 1 --2- 172.21.15.42:0/4282770146 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f009013c8c0 0x7f009013ecb0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f0088009e50 tx=0x7f008801c8b0 comp rx=0 tx=0).stop 2024-09-16T04:00:54.862 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.840+0000 7f0097c8b640 1 -- 172.21.15.42:0/4282770146 shutdown_connections 2024-09-16T04:00:54.862 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.840+0000 7f0097c8b640 1 --2- 172.21.15.42:0/4282770146 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f009013c8c0 0x7f009013ecb0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:00:54.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.840+0000 7f0097c8b640 1 --2- 172.21.15.42:0/4282770146 >> v2:172.21.15.154:3300/0 conn(0x7f00900a3d90 0x7f009013c380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:00:54.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.840+0000 7f0097c8b640 1 -- 172.21.15.42:0/4282770146 >> 172.21.15.42:0/4282770146 conn(0x7f00900a9fb0 msgr2=0x7f00900aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:00:54.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.840+0000 7f0097c8b640 1 -- 172.21.15.42:0/4282770146 shutdown_connections 2024-09-16T04:00:54.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.840+0000 7f0097c8b640 1 -- 172.21.15.42:0/4282770146 wait complete. 2024-09-16T04:00:54.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.844+0000 7f0097c8b640 10 monclient: build_initial_monmap 2024-09-16T04:00:54.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.844+0000 7f0097c8b640 10 monclient: monmap: 2024-09-16T04:00:54.863 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:00:54.863 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:00:54.863 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:00:54.863 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:00:54.863 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:00:54.863 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:00:54.863 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:00:54.863 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:00:54.864 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:00:54.864 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:00:54.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.844+0000 7f0097c8b640 1 librados: starting msgr at 2024-09-16T04:00:54.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.844+0000 7f0097c8b640 1 librados: starting objecter 2024-09-16T04:00:54.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.844+0000 7f0097c8b640 1 Processor -- start 2024-09-16T04:00:54.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.844+0000 7f0097c8b640 1 -- start start 2024-09-16T04:00:54.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.844+0000 7f0097c8b640 1 librados: setting wanted keys 2024-09-16T04:00:54.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.844+0000 7f0097c8b640 1 librados: calling monclient init 2024-09-16T04:00:54.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.844+0000 7f0097c8b640 10 monclient: init 2024-09-16T04:00:54.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.844+0000 7f0097c8b640 10 monclient: _reopen_session rank -1 2024-09-16T04:00:54.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.844+0000 7f0097c8b640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:00:54.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.844+0000 7f0097c8b640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f00900a39b0 0x7f009012fc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:00:54.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.844+0000 7f0097c8b640 10 monclient(hunting): picked mon.noname-c con 0x7f00900a39b0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:00:54.865 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.844+0000 7f0097c8b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f00900a3d90 0x7f00901301a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:00:54.865 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.844+0000 7f0097c8b640 10 monclient(hunting): picked mon.noname-b con 0x7f00900a3d90 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:00:54.865 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.844+0000 7f0097c8b640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f009013c8c0 0x7f00901cf1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:00:54.865 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.844+0000 7f0097c8b640 10 monclient(hunting): picked mon.noname-a con 0x7f009013c8c0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:00:54.865 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.844+0000 7f0097c8b640 10 monclient(hunting): start opening mon connection 2024-09-16T04:00:54.865 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.844+0000 7f0097c8b640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f0090141da0 con 0x7f009013c8c0 2024-09-16T04:00:54.865 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.844+0000 7f0097c8b640 10 monclient(hunting): start opening mon connection 2024-09-16T04:00:54.865 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.844+0000 7f0097c8b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f0090141c20 con 0x7f00900a3d90 2024-09-16T04:00:54.865 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.844+0000 7f0097c8b640 10 monclient(hunting): start opening mon connection 2024-09-16T04:00:54.865 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.844+0000 7f0097c8b640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f0090141f20 con 0x7f00900a39b0 2024-09-16T04:00:54.865 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.844+0000 7f0097c8b640 10 monclient(hunting): _renew_subs 2024-09-16T04:00:54.865 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.844+0000 7f0097c8b640 10 monclient(hunting): authenticate will time out at 2101.817297s 2024-09-16T04:00:54.865 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.844+0000 7f0096228640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f009013c8c0 0x7f00901cf1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:00:54.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.844+0000 7f0096228640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f009013c8c0 0x7f00901cf1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:44094/0 (socket says 172.21.15.42:44094) 2024-09-16T04:00:54.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.844+0000 7f0096228640 1 -- 172.21.15.42:0/1693899729 learned_addr learned my addr 172.21.15.42:0/1693899729 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:00:54.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.844+0000 7f0096228640 10 monclient(hunting): get_auth_request con 0x7f009013c8c0 auth_method 0 2024-09-16T04:00:54.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.844+0000 7f0096228640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:00:54.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.844+0000 7f0096228640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:00:54.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.844+0000 7f0096228640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:00:54.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.844+0000 7f0095a27640 1 --2- 172.21.15.42:0/1693899729 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f00900a39b0 0x7f009012fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:00:54.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.844+0000 7f0096228640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:00:54.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.844+0000 7f0096228640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:00:54.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.844+0000 7f0096228640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:00:54.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.844+0000 7f0095a27640 10 monclient(hunting): get_auth_request con 0x7f00900a39b0 auth_method 0 2024-09-16T04:00:54.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.844+0000 7f0095a27640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:00:54.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.844+0000 7f0095a27640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:00:54.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.844+0000 7f0095a27640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:00:54.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.844+0000 7f0095a27640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:00:54.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.844+0000 7f0095a27640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:00:54.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.844+0000 7f0095226640 1 --2- 172.21.15.42:0/1693899729 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f00900a3d90 0x7f00901301a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:00:54.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.844+0000 7f0095a27640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:00:54.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.844+0000 7f0095226640 10 monclient(hunting): get_auth_request con 0x7f00900a3d90 auth_method 0 2024-09-16T04:00:54.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.844+0000 7f0095226640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:00:54.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.844+0000 7f0095226640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:00:54.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.844+0000 7f0095226640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:00:54.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.844+0000 7f0096228640 10 monclient(hunting): handle_auth_done global_id 825285 payload 824 2024-09-16T04:00:54.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.844+0000 7f0096228640 1 -- 172.21.15.42:0/1693899729 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f00900a39b0 msgr2=0x7f009012fc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:00:54.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.844+0000 7f0096228640 1 --2- 172.21.15.42:0/1693899729 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f00900a39b0 0x7f009012fc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:00:54.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.844+0000 7f0096228640 1 -- 172.21.15.42:0/1693899729 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f00900a3d90 msgr2=0x7f00901301a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:00:54.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.844+0000 7f0096228640 1 --2- 172.21.15.42:0/1693899729 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f00900a3d90 0x7f00901301a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:00:54.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.844+0000 7f0096228640 10 monclient: _finish_hunting 0 2024-09-16T04:00:54.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.844+0000 7f0096228640 1 monclient: found mon.noname-a 2024-09-16T04:00:54.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.844+0000 7f0096228640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:00:54.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.844+0000 7f0096228640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:00:54.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.844+0000 7f0096228640 1 -- 172.21.15.42:0/1693899729 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f009013ede0 con 0x7f009013c8c0 2024-09-16T04:00:54.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.844+0000 7f0096228640 1 --2- 172.21.15.42:0/1693899729 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f009013c8c0 0x7f00901cf1d0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f00880af040 tx=0x7f0088019650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:00:54.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.844+0000 7f00867fc640 10 client.?.objecter ms_handle_connect 0x7f009013c8c0 2024-09-16T04:00:54.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.844+0000 7f00867fc640 10 client.?.objecter resend_mon_ops 2024-09-16T04:00:54.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.844+0000 7f00867fc640 1 -- 172.21.15.42:0/1693899729 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00880bc020 con 0x7f009013c8c0 2024-09-16T04:00:54.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.844+0000 7f00867fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:00:54.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.844+0000 7f00867fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:00:54.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.844+0000 7f00867fc640 10 monclient: dump: 2024-09-16T04:00:54.869 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:00:54.869 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:00:54.869 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:00:54.869 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:00:54.869 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:00:54.869 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:00:54.869 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:00:54.869 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:00:54.869 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:00:54.869 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:00:54.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.844+0000 7f00867fc640 10 monclient: _finish_auth 0 2024-09-16T04:00:54.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.844+0000 7f00867fc640 10 monclient: _check_auth_tickets 2024-09-16T04:00:54.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.844+0000 7f00867fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:00:54.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.844+0000 7f0097c8b640 5 monclient: authenticate success, global_id 825285 2024-09-16T04:00:54.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.844+0000 7f00867fc640 1 -- 172.21.15.42:0/1693899729 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f00880b88c0 con 0x7f009013c8c0 2024-09-16T04:00:54.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.844+0000 7f0097c8b640 10 monclient: _renew_subs 2024-09-16T04:00:54.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.844+0000 7f0097c8b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:00:54.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.844+0000 7f0097c8b640 1 -- 172.21.15.42:0/1693899729 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f009006f680 con 0x7f009013c8c0 2024-09-16T04:00:54.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.844+0000 7f00867fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:00:54.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.844+0000 7f00867fc640 1 -- 172.21.15.42:0/1693899729 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00880a5530 con 0x7f009013c8c0 2024-09-16T04:00:54.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.844+0000 7f00867fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:00:54.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.844+0000 7f00867fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:00:54.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.844+0000 7f00867fc640 10 monclient: dump: 2024-09-16T04:00:54.870 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:00:54.870 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:00:54.871 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:00:54.871 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:00:54.871 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:00:54.871 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:00:54.871 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:00:54.871 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:00:54.871 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:00:54.871 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:00:54.871 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.844+0000 7f0087fff640 4 set_mon_vals no callback set 2024-09-16T04:00:54.871 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.844+0000 7f0097c8b640 10 client.825285.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:00:54.871 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.844+0000 7f0097c8b640 10 monclient: _renew_subs 2024-09-16T04:00:54.871 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.844+0000 7f0097c8b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:00:54.871 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.844+0000 7f0097c8b640 1 -- 172.21.15.42:0/1693899729 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0090131550 con 0x7f009013c8c0 2024-09-16T04:00:54.871 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.844+0000 7f0097c8b640 1 librados: init done 2024-09-16T04:00:54.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.844+0000 7f00867fc640 1 -- 172.21.15.42:0/1693899729 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f00880bf020 con 0x7f009013c8c0 2024-09-16T04:00:54.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.844+0000 7f0097c8b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:00:54.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.844+0000 7f0097c8b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:00:54.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.844+0000 7f0097c8b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:00:54.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.844+0000 7f0097c8b640 1 -- 172.21.15.42:0/1693899729 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0090133b80 con 0x7f009013c8c0 2024-09-16T04:00:54.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.844+0000 7f00867fc640 1 --2- 172.21.15.42:0/1693899729 >> v2:172.21.15.154:6812/254845342 conn(0x7f0074037810 0x7f0074039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:00:54.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.844+0000 7f00867fc640 1 -- 172.21.15.42:0/1693899729 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(567..567 src has 451..567) ==== 6224+0+0 (secure 0 0 0) 0x7f00880e4430 con 0x7f009013c8c0 2024-09-16T04:00:54.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.844+0000 7f00867fc640 10 client.825285.objecter ms_dispatch 0x7f0090000ba0 osd_map(567..567 src has 451..567) 2024-09-16T04:00:54.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.844+0000 7f00867fc640 3 client.825285.objecter handle_osd_map got epochs [567,567] > 0 2024-09-16T04:00:54.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.844+0000 7f00867fc640 3 client.825285.objecter handle_osd_map decoding full epoch 567 2024-09-16T04:00:54.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.844+0000 7f00867fc640 20 client.825285.objecter dump_active .. 0 homeless 2024-09-16T04:00:54.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.844+0000 7f0095a27640 1 --2- 172.21.15.42:0/1693899729 >> v2:172.21.15.154:6812/254845342 conn(0x7f0074037810 0x7f0074039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:00:54.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.844+0000 7f0095a27640 10 monclient: get_auth_request con 0x7f0074037810 auth_method 0 2024-09-16T04:00:54.873 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.848+0000 7f0095a27640 1 --2- 172.21.15.42:0/1693899729 >> v2:172.21.15.154:6812/254845342 conn(0x7f0074037810 0x7f0074039cd0 secure :-1 s=READY pgs=3749 cs=0 l=1 rev1=1 crypto rx=0x7f008c007fd0 tx=0x7f008c007e40 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:00:54.873 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.848+0000 7f00867fc640 10 client.825285.objecter ms_handle_connect 0x7f0074037810 2024-09-16T04:00:54.873 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.852+0000 7f00867fc640 1 -- 172.21.15.42:0/1693899729 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f009006f680 con 0x7f009013c8c0 2024-09-16T04:00:54.873 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.852+0000 7f00867fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:00:54.873 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:54.852+0000 7f00867fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:00:55.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:55.080+0000 7f0097c8b640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T04:00:55.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:55.080+0000 7f0097c8b640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T04:00:55.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:55.080+0000 7f0097c8b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:00:55.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:55.080+0000 7f0097c8b640 1 -- 172.21.15.42:0/1693899729 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f00900a5750 con 0x7f009013c8c0 2024-09-16T04:00:55.093 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.088+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.093 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:55.088+0000 7f00867fc640 1 -- 172.21.15.42:0/1693899729 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f009013ede0 con 0x7f009013c8c0 2024-09-16T04:00:55.093 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:55.088+0000 7f00867fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T04:00:55.093 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:55.088+0000 7f00867fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:00:55.094 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.088+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=29,config=2,health=11,kv=9,logm=49} crc {auth=580307019,config=692619927,health=2970712953,kv=542242125,logm=3781379668}) 2024-09-16T04:00:55.094 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.088+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=24,config=2,health=11,kv=10,logm=53} crc {auth=2528366790,config=692619927,health=4196884964,kv=3446188498,logm=466285251}) 2024-09-16T04:00:55.094 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.088+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.094 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.088+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=29,config=2,health=11,kv=9,logm=49} crc {auth=580307019,config=692619927,health=2970712953,kv=542242125,logm=3781379668}) 2024-09-16T04:00:55.094 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.088+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=29,config=2,health=12,kv=10,logm=47} crc {auth=1073672464,config=692619927,health=4042312017,kv=3446188498,logm=3488172329}) 2024-09-16T04:00:55.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:55.088+0000 7f0097c8b640 10 librados: watch_flush enter 2024-09-16T04:00:55.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:55.088+0000 7f0097c8b640 10 librados: watch_flush exit 2024-09-16T04:00:55.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:55.088+0000 7f0097c8b640 20 client.825285.objecter shutdown clearing up homeless session... 2024-09-16T04:00:55.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:55.088+0000 7f0097c8b640 10 client.825285.objecter successfully canceled tick 2024-09-16T04:00:55.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:55.088+0000 7f0097c8b640 1 -- 172.21.15.42:0/1693899729 >> v2:172.21.15.154:6812/254845342 conn(0x7f0074037810 msgr2=0x7f0074039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:00:55.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:55.088+0000 7f0097c8b640 1 --2- 172.21.15.42:0/1693899729 >> v2:172.21.15.154:6812/254845342 conn(0x7f0074037810 0x7f0074039cd0 secure :-1 s=READY pgs=3749 cs=0 l=1 rev1=1 crypto rx=0x7f008c007fd0 tx=0x7f008c007e40 comp rx=0 tx=0).stop 2024-09-16T04:00:55.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:55.088+0000 7f0097c8b640 10 monclient: shutdown 2024-09-16T04:00:55.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:55.088+0000 7f0097c8b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:00:55.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:55.088+0000 7f0097c8b640 1 -- 172.21.15.42:0/1693899729 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f009013c8c0 msgr2=0x7f00901cf1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:00:55.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:55.088+0000 7f0097c8b640 1 --2- 172.21.15.42:0/1693899729 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f009013c8c0 0x7f00901cf1d0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f00880af040 tx=0x7f0088019650 comp rx=0 tx=0).stop 2024-09-16T04:00:55.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:55.088+0000 7f0097c8b640 1 -- 172.21.15.42:0/1693899729 shutdown_connections 2024-09-16T04:00:55.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:55.088+0000 7f0097c8b640 1 --2- 172.21.15.42:0/1693899729 >> v2:172.21.15.154:6812/254845342 conn(0x7f0074037810 0x7f0074039cd0 unknown :-1 s=CLOSED pgs=3749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:00:55.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:55.088+0000 7f0097c8b640 1 --2- 172.21.15.42:0/1693899729 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f009013c8c0 0x7f00901cf1d0 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:00:55.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:55.088+0000 7f0097c8b640 1 --2- 172.21.15.42:0/1693899729 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f00900a3d90 0x7f00901301a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:00:55.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:55.088+0000 7f0097c8b640 1 --2- 172.21.15.42:0/1693899729 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f00900a39b0 0x7f009012fc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:00:55.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:55.088+0000 7f0097c8b640 1 -- 172.21.15.42:0/1693899729 >> 172.21.15.42:0/1693899729 conn(0x7f00900a9fb0 msgr2=0x7f009012d340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:00:55.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:55.088+0000 7f0097c8b640 1 -- 172.21.15.42:0/1693899729 shutdown_connections 2024-09-16T04:00:55.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:55.088+0000 7f0097c8b640 1 -- 172.21.15.42:0/1693899729 wait complete. 2024-09-16T04:00:55.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:55.088+0000 7f0097c8b640 1 librados: shutdown 2024-09-16T04:00:55.098 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.092+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.098 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.092+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1582204885}) 2024-09-16T04:00:55.099 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.092+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1016580532}) 2024-09-16T04:00:55.099 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.092+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.099 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.092+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1582204885}) 2024-09-16T04:00:55.099 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.092+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1878123533}) 2024-09-16T04:00:55.105 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.096+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.106 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.096+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=719171536}) 2024-09-16T04:00:55.106 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.096+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3946283865}) 2024-09-16T04:00:55.106 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.096+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.106 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.096+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=719171536}) 2024-09-16T04:00:55.106 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.096+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2760226313}) 2024-09-16T04:00:55.109 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.100+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.109 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.100+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1792984692}) 2024-09-16T04:00:55.109 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.100+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2647573395}) 2024-09-16T04:00:55.109 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.100+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.109 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.100+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1792984692}) 2024-09-16T04:00:55.109 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.100+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2593674550}) 2024-09-16T04:00:55.113 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T04:00:55.114 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.108+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.114 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.108+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1123534419}) 2024-09-16T04:00:55.114 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.108+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=272602038}) 2024-09-16T04:00:55.114 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.108+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.114 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.108+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1123534419}) 2024-09-16T04:00:55.114 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.108+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1644433450}) 2024-09-16T04:00:55.117 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.108+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.117 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.108+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=134956937}) 2024-09-16T04:00:55.118 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.108+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=264533131}) 2024-09-16T04:00:55.118 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.108+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.118 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.112+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=134956937}) 2024-09-16T04:00:55.118 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.112+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=581423380}) 2024-09-16T04:00:55.119 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.112+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.119 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.112+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3465068857}) 2024-09-16T04:00:55.119 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.112+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4019242066}) 2024-09-16T04:00:55.120 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.112+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.120 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.112+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3465068857}) 2024-09-16T04:00:55.120 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.112+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3773957641}) 2024-09-16T04:00:55.122 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.116+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.122 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.116+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=258817909}) 2024-09-16T04:00:55.122 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.116+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3960022943}) 2024-09-16T04:00:55.122 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.116+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.122 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.116+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=258817909}) 2024-09-16T04:00:55.122 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.116+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=293278373}) 2024-09-16T04:00:55.124 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.116+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.124 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.116+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2429065704}) 2024-09-16T04:00:55.124 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.116+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2514890103}) 2024-09-16T04:00:55.124 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.116+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.124 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.116+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2429065704}) 2024-09-16T04:00:55.124 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.116+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1893383536}) 2024-09-16T04:00:55.126 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.120+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.126 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.120+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3858424468}) 2024-09-16T04:00:55.126 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.120+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3269542563}) 2024-09-16T04:00:55.127 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.120+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.127 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.120+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3858424468}) 2024-09-16T04:00:55.127 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.120+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1103978599}) 2024-09-16T04:00:55.128 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.120+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.129 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.120+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=570597953}) 2024-09-16T04:00:55.129 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.120+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1628711987}) 2024-09-16T04:00:55.129 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.120+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.129 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.120+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=570597953}) 2024-09-16T04:00:55.129 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.120+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3590996935}) 2024-09-16T04:00:55.131 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.124+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.131 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.124+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3713355077}) 2024-09-16T04:00:55.131 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.124+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3092993299}) 2024-09-16T04:00:55.131 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.124+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.131 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.124+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3713355077}) 2024-09-16T04:00:55.131 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.124+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2897814863}) 2024-09-16T04:00:55.133 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.124+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.133 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.124+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3787066662}) 2024-09-16T04:00:55.133 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.124+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3806127165}) 2024-09-16T04:00:55.133 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.124+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.133 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.124+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3787066662}) 2024-09-16T04:00:55.133 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.124+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2767343981}) 2024-09-16T04:00:55.135 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.128+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.136 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.128+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=39633553}) 2024-09-16T04:00:55.136 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.128+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2457665275}) 2024-09-16T04:00:55.136 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.128+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.136 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.128+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=39633553}) 2024-09-16T04:00:55.136 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.128+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1302910667}) 2024-09-16T04:00:55.138 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.132+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.138 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.132+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2749289643}) 2024-09-16T04:00:55.138 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.132+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2279542758}) 2024-09-16T04:00:55.138 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.132+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.138 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.132+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2749289643}) 2024-09-16T04:00:55.138 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.132+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1611359350}) 2024-09-16T04:00:55.140 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.132+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.140 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.132+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1388504250}) 2024-09-16T04:00:55.140 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.132+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1071564264}) 2024-09-16T04:00:55.140 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.132+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.140 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.132+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1388504250}) 2024-09-16T04:00:55.141 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.132+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1085212666}) 2024-09-16T04:00:55.142 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.136+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.142 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.136+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1194768215}) 2024-09-16T04:00:55.142 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.136+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=553252998}) 2024-09-16T04:00:55.142 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.136+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.143 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.136+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1194768215}) 2024-09-16T04:00:55.143 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.136+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=921893318}) 2024-09-16T04:00:55.145 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.136+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.145 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.136+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3197953711}) 2024-09-16T04:00:55.145 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.136+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2392087475}) 2024-09-16T04:00:55.145 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.136+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.145 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.136+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3197953711}) 2024-09-16T04:00:55.145 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.136+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2597708351}) 2024-09-16T04:00:55.147 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.140+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.147 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.140+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2237247935}) 2024-09-16T04:00:55.147 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.140+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1318876500}) 2024-09-16T04:00:55.147 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.140+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.147 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.140+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2237247935}) 2024-09-16T04:00:55.147 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.140+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2054767259}) 2024-09-16T04:00:55.149 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.140+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.149 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.140+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3196058396}) 2024-09-16T04:00:55.149 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.140+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1972916713}) 2024-09-16T04:00:55.149 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.140+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.149 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.140+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3196058396}) 2024-09-16T04:00:55.149 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.140+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2180192246}) 2024-09-16T04:00:55.152 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.144+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.152 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.144+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2002536919}) 2024-09-16T04:00:55.152 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.144+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3491250348}) 2024-09-16T04:00:55.152 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.144+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.152 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.144+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2002536919}) 2024-09-16T04:00:55.152 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.144+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3527015858}) 2024-09-16T04:00:55.154 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.148+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.154 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.148+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3856286538}) 2024-09-16T04:00:55.154 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.148+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1545761605}) 2024-09-16T04:00:55.154 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.148+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.154 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.148+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3856286538}) 2024-09-16T04:00:55.154 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.148+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=372770312}) 2024-09-16T04:00:55.156 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.148+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.156 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.148+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1120570894}) 2024-09-16T04:00:55.156 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.148+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1885159394}) 2024-09-16T04:00:55.156 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.148+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.156 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.148+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1120570894}) 2024-09-16T04:00:55.156 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.148+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2765505144}) 2024-09-16T04:00:55.158 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.152+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.158 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.152+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=61349006}) 2024-09-16T04:00:55.158 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.152+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3349646312}) 2024-09-16T04:00:55.158 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.152+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.158 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.152+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=61349006}) 2024-09-16T04:00:55.159 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.152+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2138491972}) 2024-09-16T04:00:55.161 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.152+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.161 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.152+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=11652135}) 2024-09-16T04:00:55.161 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.152+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1576091034}) 2024-09-16T04:00:55.161 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.152+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.161 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.152+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=11652135}) 2024-09-16T04:00:55.161 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.152+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=241454340}) 2024-09-16T04:00:55.163 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.156+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.163 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.156+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4013776908}) 2024-09-16T04:00:55.163 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.156+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2326164663}) 2024-09-16T04:00:55.163 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.156+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.163 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.156+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4013776908}) 2024-09-16T04:00:55.163 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.156+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3114789489}) 2024-09-16T04:00:55.165 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.156+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.165 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.156+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=741110336}) 2024-09-16T04:00:55.165 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.156+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=917356019}) 2024-09-16T04:00:55.165 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.156+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.165 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.156+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=741110336}) 2024-09-16T04:00:55.166 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.156+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=926464057}) 2024-09-16T04:00:55.167 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.160+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.167 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.160+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=250153182}) 2024-09-16T04:00:55.168 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.160+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2510631189}) 2024-09-16T04:00:55.168 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.160+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.168 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.160+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=250153182}) 2024-09-16T04:00:55.168 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.160+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2776306432}) 2024-09-16T04:00:55.170 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.164+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.170 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.164+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3454425101}) 2024-09-16T04:00:55.170 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.164+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3960575660}) 2024-09-16T04:00:55.170 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.164+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.170 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.164+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3454425101}) 2024-09-16T04:00:55.170 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.164+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3966364145}) 2024-09-16T04:00:55.172 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.164+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.172 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.164+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3844993746}) 2024-09-16T04:00:55.172 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.164+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3495070453}) 2024-09-16T04:00:55.172 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.164+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.172 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.164+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3844993746}) 2024-09-16T04:00:55.172 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.164+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1208073324}) 2024-09-16T04:00:55.174 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.168+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.174 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.168+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1536105709}) 2024-09-16T04:00:55.174 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.168+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1853912378}) 2024-09-16T04:00:55.174 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.168+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.174 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.168+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1536105709}) 2024-09-16T04:00:55.175 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.168+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3258291167}) 2024-09-16T04:00:55.176 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.168+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.176 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.168+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=152254773}) 2024-09-16T04:00:55.176 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.168+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1374018558}) 2024-09-16T04:00:55.176 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.168+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.177 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.168+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=152254773}) 2024-09-16T04:00:55.177 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.168+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1445781352}) 2024-09-16T04:00:55.178 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.172+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.178 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.172+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=168600437}) 2024-09-16T04:00:55.178 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.172+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3498369353}) 2024-09-16T04:00:55.179 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.172+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.179 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.172+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=168600437}) 2024-09-16T04:00:55.179 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.172+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1605826073}) 2024-09-16T04:00:55.180 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.172+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.181 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.172+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=499258966}) 2024-09-16T04:00:55.181 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.172+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=878842193}) 2024-09-16T04:00:55.181 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.172+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.181 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.172+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=499258966}) 2024-09-16T04:00:55.181 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.172+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2387470167}) 2024-09-16T04:00:55.183 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.176+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.183 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.176+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1287187329}) 2024-09-16T04:00:55.183 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.176+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2436885074}) 2024-09-16T04:00:55.183 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.176+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.183 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.176+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1287187329}) 2024-09-16T04:00:55.183 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.176+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1149561177}) 2024-09-16T04:00:55.185 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.176+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.185 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.176+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3742774769}) 2024-09-16T04:00:55.185 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.176+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1232822198}) 2024-09-16T04:00:55.185 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.176+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.185 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.176+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3742774769}) 2024-09-16T04:00:55.185 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.176+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=375111407}) 2024-09-16T04:00:55.187 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.180+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.187 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.180+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=67746608}) 2024-09-16T04:00:55.187 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.180+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=704773238}) 2024-09-16T04:00:55.187 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.180+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.188 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.180+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=67746608}) 2024-09-16T04:00:55.188 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.180+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1180788066}) 2024-09-16T04:00:55.189 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.184+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.189 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.184+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=942391467}) 2024-09-16T04:00:55.190 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.184+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=73618413}) 2024-09-16T04:00:55.190 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.184+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.190 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.184+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=942391467}) 2024-09-16T04:00:55.190 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.184+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2717216517}) 2024-09-16T04:00:55.192 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.184+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.192 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.184+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4228487247}) 2024-09-16T04:00:55.192 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.184+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2821423156}) 2024-09-16T04:00:55.192 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.184+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.192 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.184+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4228487247}) 2024-09-16T04:00:55.192 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.184+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=793005403}) 2024-09-16T04:00:55.194 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.188+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.194 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.188+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2103217148}) 2024-09-16T04:00:55.194 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.188+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=399355979}) 2024-09-16T04:00:55.194 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.188+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.194 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.188+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2103217148}) 2024-09-16T04:00:55.194 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.188+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3320950855}) 2024-09-16T04:00:55.196 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.188+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.196 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.188+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3191131589}) 2024-09-16T04:00:55.196 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.188+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=851540581}) 2024-09-16T04:00:55.196 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.188+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.196 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.188+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3191131589}) 2024-09-16T04:00:55.197 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.188+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1492297133}) 2024-09-16T04:00:55.198 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.192+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.198 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.192+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1509294177}) 2024-09-16T04:00:55.198 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.192+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2917398693}) 2024-09-16T04:00:55.198 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.192+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.198 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.192+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1509294177}) 2024-09-16T04:00:55.199 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.192+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2515318057}) 2024-09-16T04:00:55.200 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.192+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.200 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.192+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1398371203}) 2024-09-16T04:00:55.200 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.192+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4036236945}) 2024-09-16T04:00:55.201 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.192+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.201 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.192+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1398371203}) 2024-09-16T04:00:55.201 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.192+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=682648210}) 2024-09-16T04:00:55.202 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.196+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.203 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.196+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2474133469}) 2024-09-16T04:00:55.203 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.196+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1718132131}) 2024-09-16T04:00:55.203 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.196+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.203 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.196+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2474133469}) 2024-09-16T04:00:55.203 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.196+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=426929092}) 2024-09-16T04:00:55.205 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.196+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.205 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.196+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3176075072}) 2024-09-16T04:00:55.205 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.196+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2551332128}) 2024-09-16T04:00:55.205 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.196+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.205 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.196+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3176075072}) 2024-09-16T04:00:55.205 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.196+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=166788472}) 2024-09-16T04:00:55.207 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.200+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.207 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.200+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1985741847}) 2024-09-16T04:00:55.207 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.200+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2898447739}) 2024-09-16T04:00:55.207 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.200+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.207 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.200+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1985741847}) 2024-09-16T04:00:55.207 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.200+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=364011231}) 2024-09-16T04:00:55.209 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.200+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.209 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.200+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1679030848}) 2024-09-16T04:00:55.209 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.200+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1786135273}) 2024-09-16T04:00:55.209 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.200+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.209 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.200+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1679030848}) 2024-09-16T04:00:55.209 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.200+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2004781434}) 2024-09-16T04:00:55.211 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.204+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.211 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.204+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1822523119}) 2024-09-16T04:00:55.211 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.204+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2599638539}) 2024-09-16T04:00:55.212 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.204+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.212 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.204+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1822523119}) 2024-09-16T04:00:55.212 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.204+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2766856749}) 2024-09-16T04:00:55.213 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.204+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.213 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.204+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3555470216}) 2024-09-16T04:00:55.213 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.204+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4163591189}) 2024-09-16T04:00:55.213 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.204+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.213 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.204+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3555470216}) 2024-09-16T04:00:55.213 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.204+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1649413413}) 2024-09-16T04:00:55.215 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.208+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.215 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.208+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3478306164}) 2024-09-16T04:00:55.215 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.208+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2673758651}) 2024-09-16T04:00:55.215 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.208+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.215 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.208+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3478306164}) 2024-09-16T04:00:55.215 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.208+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3729689587}) 2024-09-16T04:00:55.217 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.208+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.217 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.208+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1927988565}) 2024-09-16T04:00:55.217 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.208+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=583361732}) 2024-09-16T04:00:55.217 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.208+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.217 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.208+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1927988565}) 2024-09-16T04:00:55.217 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.208+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3210292631}) 2024-09-16T04:00:55.219 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.212+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.219 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.212+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=577602848}) 2024-09-16T04:00:55.219 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.212+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3787445125}) 2024-09-16T04:00:55.219 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.212+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.219 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.212+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=577602848}) 2024-09-16T04:00:55.219 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.212+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1847594344}) 2024-09-16T04:00:55.221 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.212+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.221 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.212+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1238315652}) 2024-09-16T04:00:55.221 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.212+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2661926300}) 2024-09-16T04:00:55.221 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.212+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.221 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.212+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1238315652}) 2024-09-16T04:00:55.221 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.212+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=738116933}) 2024-09-16T04:00:55.223 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.216+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.223 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.216+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1400043479}) 2024-09-16T04:00:55.223 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.216+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=677465356}) 2024-09-16T04:00:55.223 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.216+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.223 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.216+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1400043479}) 2024-09-16T04:00:55.223 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.216+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4173888827}) 2024-09-16T04:00:55.225 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.216+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.225 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.216+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3312052610}) 2024-09-16T04:00:55.225 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.216+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1764373977}) 2024-09-16T04:00:55.225 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.216+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.225 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.216+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3312052610}) 2024-09-16T04:00:55.225 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.216+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=789353322}) 2024-09-16T04:00:55.227 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.220+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.227 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.220+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3819727195}) 2024-09-16T04:00:55.227 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.220+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=449340343}) 2024-09-16T04:00:55.227 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.220+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.227 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.220+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3819727195}) 2024-09-16T04:00:55.227 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.220+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1700886764}) 2024-09-16T04:00:55.229 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.220+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.229 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.220+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1276601502}) 2024-09-16T04:00:55.229 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.220+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2518227764}) 2024-09-16T04:00:55.229 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.220+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.229 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.220+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1276601502}) 2024-09-16T04:00:55.229 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.220+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3879165025}) 2024-09-16T04:00:55.231 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.224+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.231 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.224+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2199943020}) 2024-09-16T04:00:55.231 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.224+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1454835114}) 2024-09-16T04:00:55.231 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.224+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.231 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.224+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2199943020}) 2024-09-16T04:00:55.231 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.224+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2547772469}) 2024-09-16T04:00:55.233 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.224+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.236 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.224+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4061627978}) 2024-09-16T04:00:55.236 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.224+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1793359874}) 2024-09-16T04:00:55.237 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.224+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.237 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.224+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4061627978}) 2024-09-16T04:00:55.237 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.224+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1492649635}) 2024-09-16T04:00:55.239 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.228+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.239 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.228+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3681942775}) 2024-09-16T04:00:55.239 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.228+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1359036880}) 2024-09-16T04:00:55.239 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.228+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.239 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.228+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3681942775}) 2024-09-16T04:00:55.239 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.228+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2583045146}) 2024-09-16T04:00:55.239 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.228+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.239 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.228+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=10618721}) 2024-09-16T04:00:55.240 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.228+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1778487118}) 2024-09-16T04:00:55.240 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.228+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.240 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.228+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=10618721}) 2024-09-16T04:00:55.240 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.228+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3385664462}) 2024-09-16T04:00:55.240 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.232+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.240 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.232+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3770313320}) 2024-09-16T04:00:55.240 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.232+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2442471679}) 2024-09-16T04:00:55.240 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.232+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.240 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.232+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3770313320}) 2024-09-16T04:00:55.240 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.232+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1461883880}) 2024-09-16T04:00:55.241 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.232+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.242 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.232+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3895797231}) 2024-09-16T04:00:55.242 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.232+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3589383326}) 2024-09-16T04:00:55.242 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.232+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.242 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.232+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3895797231}) 2024-09-16T04:00:55.242 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.232+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3885024928}) 2024-09-16T04:00:55.243 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.236+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.243 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.236+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1575848398}) 2024-09-16T04:00:55.243 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.236+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=460433804}) 2024-09-16T04:00:55.243 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.236+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.243 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.236+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1575848398}) 2024-09-16T04:00:55.243 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.236+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3692620154}) 2024-09-16T04:00:55.245 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.236+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.245 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.236+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2022034743}) 2024-09-16T04:00:55.245 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.236+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3909617133}) 2024-09-16T04:00:55.245 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.236+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.245 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.236+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2022034743}) 2024-09-16T04:00:55.245 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.236+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4292277606}) 2024-09-16T04:00:55.247 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.240+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.247 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.240+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2825535407}) 2024-09-16T04:00:55.247 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.240+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1415366838}) 2024-09-16T04:00:55.247 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.240+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.247 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.240+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2825535407}) 2024-09-16T04:00:55.247 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.240+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1476630133}) 2024-09-16T04:00:55.249 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.240+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.249 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.240+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1998442139}) 2024-09-16T04:00:55.249 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.240+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1628677180}) 2024-09-16T04:00:55.249 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.240+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.249 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.240+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1998442139}) 2024-09-16T04:00:55.249 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.240+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3064687758}) 2024-09-16T04:00:55.250 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.244+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.251 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.244+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1461819601}) 2024-09-16T04:00:55.251 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.244+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=353656466}) 2024-09-16T04:00:55.251 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.244+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.251 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.244+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1461819601}) 2024-09-16T04:00:55.251 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.244+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3889767699}) 2024-09-16T04:00:55.252 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.244+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.253 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.244+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3419218626}) 2024-09-16T04:00:55.253 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.244+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3238642541}) 2024-09-16T04:00:55.253 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.244+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.253 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.244+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3419218626}) 2024-09-16T04:00:55.253 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.244+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2571635287}) 2024-09-16T04:00:55.254 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.248+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.255 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.248+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1356755390}) 2024-09-16T04:00:55.255 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.248+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1224480513}) 2024-09-16T04:00:55.255 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.248+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.255 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.248+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1356755390}) 2024-09-16T04:00:55.255 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.248+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1740884857}) 2024-09-16T04:00:55.257 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.248+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.257 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.248+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=552394594}) 2024-09-16T04:00:55.257 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.248+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2621425306}) 2024-09-16T04:00:55.257 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.248+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.257 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.248+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=552394594}) 2024-09-16T04:00:55.257 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.248+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3791525124}) 2024-09-16T04:00:55.259 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.252+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.259 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.252+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2118182375}) 2024-09-16T04:00:55.259 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.252+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3254712990}) 2024-09-16T04:00:55.259 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.252+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.259 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.252+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2118182375}) 2024-09-16T04:00:55.259 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.252+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1009248415}) 2024-09-16T04:00:55.261 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.252+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.261 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.252+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3349928865}) 2024-09-16T04:00:55.261 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.252+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4272131467}) 2024-09-16T04:00:55.261 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.252+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.261 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.252+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3349928865}) 2024-09-16T04:00:55.261 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.252+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1114681946}) 2024-09-16T04:00:55.263 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.256+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.263 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.256+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=133608009}) 2024-09-16T04:00:55.263 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.256+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=21208479}) 2024-09-16T04:00:55.263 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.256+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.263 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.256+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=133608009}) 2024-09-16T04:00:55.263 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.256+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1584566475}) 2024-09-16T04:00:55.265 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.256+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.265 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.256+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3792544212}) 2024-09-16T04:00:55.265 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.256+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3775763763}) 2024-09-16T04:00:55.265 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.256+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.265 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.256+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3792544212}) 2024-09-16T04:00:55.265 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.256+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=832058757}) 2024-09-16T04:00:55.267 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.260+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.267 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.260+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=44143808}) 2024-09-16T04:00:55.267 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.260+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2816657113}) 2024-09-16T04:00:55.267 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.260+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.267 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.260+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=44143808}) 2024-09-16T04:00:55.267 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.260+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1630670868}) 2024-09-16T04:00:55.269 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.260+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.269 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.260+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2260211725}) 2024-09-16T04:00:55.269 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.260+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1406503176}) 2024-09-16T04:00:55.269 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.260+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.269 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.260+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2260211725}) 2024-09-16T04:00:55.269 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.260+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2967951277}) 2024-09-16T04:00:55.271 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.264+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.271 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.264+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1882636524}) 2024-09-16T04:00:55.271 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.264+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=929029563}) 2024-09-16T04:00:55.271 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.264+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.271 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.264+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1882636524}) 2024-09-16T04:00:55.271 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.264+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=921074839}) 2024-09-16T04:00:55.273 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.264+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.273 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.264+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1474853572}) 2024-09-16T04:00:55.273 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.264+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=854463317}) 2024-09-16T04:00:55.273 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.264+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.273 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.264+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1474853572}) 2024-09-16T04:00:55.273 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.264+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1772726461}) 2024-09-16T04:00:55.275 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.268+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.275 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.268+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3243142135}) 2024-09-16T04:00:55.275 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.268+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2319715096}) 2024-09-16T04:00:55.275 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.268+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.275 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.268+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3243142135}) 2024-09-16T04:00:55.275 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.268+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2353479251}) 2024-09-16T04:00:55.277 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.268+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.277 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.268+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3972159868}) 2024-09-16T04:00:55.277 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.268+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4117785151}) 2024-09-16T04:00:55.277 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.268+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.277 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.268+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3972159868}) 2024-09-16T04:00:55.277 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.268+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3659317057}) 2024-09-16T04:00:55.279 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.272+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.279 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.272+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=951448554}) 2024-09-16T04:00:55.279 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.272+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2208455265}) 2024-09-16T04:00:55.279 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.272+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.279 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.272+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=951448554}) 2024-09-16T04:00:55.279 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.272+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1371044176}) 2024-09-16T04:00:55.281 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.272+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.281 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.272+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2935126239}) 2024-09-16T04:00:55.281 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.272+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3716674269}) 2024-09-16T04:00:55.281 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.272+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.281 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.272+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2935126239}) 2024-09-16T04:00:55.281 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.272+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1708823023}) 2024-09-16T04:00:55.283 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.276+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.283 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.276+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=577719704}) 2024-09-16T04:00:55.283 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.276+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4060467008}) 2024-09-16T04:00:55.283 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.276+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.283 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.276+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=577719704}) 2024-09-16T04:00:55.283 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.276+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2613714276}) 2024-09-16T04:00:55.285 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.276+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.285 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.276+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3651852097}) 2024-09-16T04:00:55.285 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.276+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3997682740}) 2024-09-16T04:00:55.285 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.276+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.286 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.276+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3651852097}) 2024-09-16T04:00:55.286 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.276+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2228269445}) 2024-09-16T04:00:55.287 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.280+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.287 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.280+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2860663538}) 2024-09-16T04:00:55.287 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.280+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=976952637}) 2024-09-16T04:00:55.287 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.280+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.287 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.280+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2860663538}) 2024-09-16T04:00:55.287 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.280+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3943322326}) 2024-09-16T04:00:55.289 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.280+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.289 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.280+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=849371866}) 2024-09-16T04:00:55.289 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.280+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1440482229}) 2024-09-16T04:00:55.289 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.280+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.289 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.280+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=849371866}) 2024-09-16T04:00:55.290 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.280+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1618638545}) 2024-09-16T04:00:55.291 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.284+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.291 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.284+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2347166495}) 2024-09-16T04:00:55.291 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.284+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3784433187}) 2024-09-16T04:00:55.291 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.284+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.291 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.284+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2347166495}) 2024-09-16T04:00:55.291 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.284+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4015817631}) 2024-09-16T04:00:55.293 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.284+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.293 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.284+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3032462526}) 2024-09-16T04:00:55.293 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.284+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3195229405}) 2024-09-16T04:00:55.293 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.284+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.293 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.284+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3032462526}) 2024-09-16T04:00:55.293 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.284+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3742617827}) 2024-09-16T04:00:55.295 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.288+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.295 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.288+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2990979326}) 2024-09-16T04:00:55.295 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.288+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2090494001}) 2024-09-16T04:00:55.295 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.288+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.295 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.288+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2990979326}) 2024-09-16T04:00:55.295 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.288+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1515305092}) 2024-09-16T04:00:55.297 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.288+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.297 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.288+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=896443576}) 2024-09-16T04:00:55.297 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.288+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3914729448}) 2024-09-16T04:00:55.297 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.288+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.297 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.288+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=896443576}) 2024-09-16T04:00:55.297 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.288+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3140141974}) 2024-09-16T04:00:55.299 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.292+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.299 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.292+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2191359329}) 2024-09-16T04:00:55.299 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.292+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3545722813}) 2024-09-16T04:00:55.299 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.292+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.299 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.292+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2191359329}) 2024-09-16T04:00:55.299 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.292+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=587467151}) 2024-09-16T04:00:55.301 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.292+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.301 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.292+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3637694821}) 2024-09-16T04:00:55.301 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.292+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1604828765}) 2024-09-16T04:00:55.301 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.292+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.301 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.292+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3637694821}) 2024-09-16T04:00:55.301 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.292+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3657533023}) 2024-09-16T04:00:55.303 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.296+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.303 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.296+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2632335412}) 2024-09-16T04:00:55.303 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.296+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2164395791}) 2024-09-16T04:00:55.303 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.296+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.303 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.296+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2632335412}) 2024-09-16T04:00:55.303 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.296+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3648618146}) 2024-09-16T04:00:55.305 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.296+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.305 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.296+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=293168415}) 2024-09-16T04:00:55.305 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.296+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=699276167}) 2024-09-16T04:00:55.305 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.296+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.305 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.296+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=293168415}) 2024-09-16T04:00:55.305 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.296+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1172627243}) 2024-09-16T04:00:55.306 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.300+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.306 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.300+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=799445245}) 2024-09-16T04:00:55.307 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.300+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=97882037}) 2024-09-16T04:00:55.307 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.300+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.307 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.300+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=799445245}) 2024-09-16T04:00:55.307 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.300+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1121112921}) 2024-09-16T04:00:55.308 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.300+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.308 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.300+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=476274785}) 2024-09-16T04:00:55.308 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.300+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2260235225}) 2024-09-16T04:00:55.309 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.300+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.309 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.300+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=476274785}) 2024-09-16T04:00:55.309 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.300+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=348914997}) 2024-09-16T04:00:55.310 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.304+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.310 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.304+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1734600730}) 2024-09-16T04:00:55.311 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.304+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=920729118}) 2024-09-16T04:00:55.311 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.304+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.311 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.304+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1734600730}) 2024-09-16T04:00:55.311 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.304+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=129110186}) 2024-09-16T04:00:55.312 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.304+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.313 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.304+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2787416674}) 2024-09-16T04:00:55.313 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.304+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2945801156}) 2024-09-16T04:00:55.313 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.304+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.313 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.304+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2787416674}) 2024-09-16T04:00:55.313 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.304+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4231042931}) 2024-09-16T04:00:55.314 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.308+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.314 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.308+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=93394290}) 2024-09-16T04:00:55.314 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.308+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2336024229}) 2024-09-16T04:00:55.315 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.308+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.315 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.308+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=93394290}) 2024-09-16T04:00:55.315 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.308+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4258586902}) 2024-09-16T04:00:55.316 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.308+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.317 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.308+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3255144658}) 2024-09-16T04:00:55.317 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.308+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4173404802}) 2024-09-16T04:00:55.317 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.308+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.317 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.308+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3255144658}) 2024-09-16T04:00:55.317 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.308+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=89310433}) 2024-09-16T04:00:55.318 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.312+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.318 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.312+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2698939976}) 2024-09-16T04:00:55.319 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.312+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3891266134}) 2024-09-16T04:00:55.319 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.312+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.319 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.312+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2698939976}) 2024-09-16T04:00:55.319 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.312+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1662764851}) 2024-09-16T04:00:55.320 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.312+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.320 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.312+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1689421037}) 2024-09-16T04:00:55.320 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.312+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=489969495}) 2024-09-16T04:00:55.321 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.312+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.321 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.312+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1689421037}) 2024-09-16T04:00:55.321 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.312+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=767276881}) 2024-09-16T04:00:55.322 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.316+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.322 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.316+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1326948141}) 2024-09-16T04:00:55.322 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.316+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4031762991}) 2024-09-16T04:00:55.322 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.316+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.323 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.316+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1326948141}) 2024-09-16T04:00:55.323 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.316+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1202075137}) 2024-09-16T04:00:55.324 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.316+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.325 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.316+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=397992620}) 2024-09-16T04:00:55.325 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.316+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2081174337}) 2024-09-16T04:00:55.325 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.316+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.325 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.316+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=397992620}) 2024-09-16T04:00:55.325 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.316+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2008353180}) 2024-09-16T04:00:55.326 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.320+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.326 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.320+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=566685167}) 2024-09-16T04:00:55.327 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.320+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2695862308}) 2024-09-16T04:00:55.327 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.320+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.327 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.320+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=566685167}) 2024-09-16T04:00:55.327 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.320+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1107844642}) 2024-09-16T04:00:55.329 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.320+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.329 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.320+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3076155889}) 2024-09-16T04:00:55.329 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.320+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2348696530}) 2024-09-16T04:00:55.329 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.320+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.329 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.320+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3076155889}) 2024-09-16T04:00:55.330 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.320+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3246522302}) 2024-09-16T04:00:55.330 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.324+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.330 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.324+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3120399497}) 2024-09-16T04:00:55.331 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.324+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3497534354}) 2024-09-16T04:00:55.331 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.324+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.331 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.324+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3120399497}) 2024-09-16T04:00:55.331 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.324+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3293856089}) 2024-09-16T04:00:55.333 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.324+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.333 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.324+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1288581526}) 2024-09-16T04:00:55.333 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.324+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3288760729}) 2024-09-16T04:00:55.333 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.324+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.333 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.324+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1288581526}) 2024-09-16T04:00:55.333 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.324+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=853850629}) 2024-09-16T04:00:55.334 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.328+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.335 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.328+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3527724221}) 2024-09-16T04:00:55.335 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.328+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4235733638}) 2024-09-16T04:00:55.335 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.328+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.335 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.328+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3527724221}) 2024-09-16T04:00:55.335 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.328+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2786210409}) 2024-09-16T04:00:55.336 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.328+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.337 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.328+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4053145185}) 2024-09-16T04:00:55.337 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.328+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4140599424}) 2024-09-16T04:00:55.337 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.328+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.337 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.328+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4053145185}) 2024-09-16T04:00:55.337 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.328+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3771612265}) 2024-09-16T04:00:55.338 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.332+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.339 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.332+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1137565027}) 2024-09-16T04:00:55.339 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.332+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4124615769}) 2024-09-16T04:00:55.339 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.332+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.339 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.332+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1137565027}) 2024-09-16T04:00:55.339 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.332+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1488005599}) 2024-09-16T04:00:55.340 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.332+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.340 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.332+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1361666582}) 2024-09-16T04:00:55.340 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.332+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2071705815}) 2024-09-16T04:00:55.341 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.332+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.341 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.332+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1361666582}) 2024-09-16T04:00:55.341 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.332+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=693631214}) 2024-09-16T04:00:55.342 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.336+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.342 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.336+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1008182656}) 2024-09-16T04:00:55.342 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.336+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1149167620}) 2024-09-16T04:00:55.343 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.336+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.343 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.336+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1008182656}) 2024-09-16T04:00:55.343 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.336+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=939469072}) 2024-09-16T04:00:55.344 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.336+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.344 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.336+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=993888358}) 2024-09-16T04:00:55.344 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.336+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2164584820}) 2024-09-16T04:00:55.344 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.336+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.344 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.336+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=993888358}) 2024-09-16T04:00:55.344 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.336+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1841929563}) 2024-09-16T04:00:55.346 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.340+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.346 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.340+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3925980544}) 2024-09-16T04:00:55.346 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.340+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3547200142}) 2024-09-16T04:00:55.346 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.340+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.346 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.340+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3925980544}) 2024-09-16T04:00:55.347 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.340+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3844854704}) 2024-09-16T04:00:55.348 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.340+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.348 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.340+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2940037138}) 2024-09-16T04:00:55.348 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.340+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3279320556}) 2024-09-16T04:00:55.348 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.340+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.348 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.340+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2940037138}) 2024-09-16T04:00:55.349 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.340+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3957785708}) 2024-09-16T04:00:55.350 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.344+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.350 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.344+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2503692912}) 2024-09-16T04:00:55.350 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.344+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3024874650}) 2024-09-16T04:00:55.350 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.344+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.350 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.344+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2503692912}) 2024-09-16T04:00:55.351 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.344+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2705631178}) 2024-09-16T04:00:55.352 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.344+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.352 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.344+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2222793267}) 2024-09-16T04:00:55.352 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.344+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2816954866}) 2024-09-16T04:00:55.352 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.344+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.352 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.344+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2222793267}) 2024-09-16T04:00:55.353 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.344+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3875457148}) 2024-09-16T04:00:55.355 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.348+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.355 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.348+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=80,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=7} crc {logm=2502864233,mdsmap=4088524775,mgr=460109776,mgr_command_descs=2107673600,mgr_metadata=1367272202,mgrstat=2079678049}) 2024-09-16T04:00:55.355 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.348+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=77,mdsmap=3,mgr=7,mgr_metadata=1,mgrstat=12} crc {logm=4114599120,mdsmap=4088524775,mgr=390803354,mgr_metadata=1367272202,mgrstat=1977987548}) 2024-09-16T04:00:55.355 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.348+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.355 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.348+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=80,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=7} crc {logm=2502864233,mdsmap=4088524775,mgr=460109776,mgr_command_descs=2107673600,mgr_metadata=1367272202,mgrstat=2079678049}) 2024-09-16T04:00:55.355 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.348+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=78,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9} crc {logm=3473865935,mdsmap=4088524775,mgr=460109776,mgr_command_descs=2107673600,mgr_metadata=1367272202,mgrstat=2926215328}) 2024-09-16T04:00:55.357 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.348+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.357 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.348+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgrstat=7,mon_config_key=10,monmap=2,osd_metadata=6,osd_snap=39,osdmap=36} crc {mgrstat=4026055356,mon_config_key=1530417952,monmap=268092033,osd_metadata=4182959540,osd_snap=657291659,osdmap=3950454859}) 2024-09-16T04:00:55.357 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.348+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mgrstat=5,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=40,osdmap=34} crc {mgrstat=3123154048,mon_config_key=85057112,monmap=3937513981,osd_metadata=1273868022,osd_pg_creating=3163125691,osd_snap=3980371759,osdmap=13612412}) 2024-09-16T04:00:55.357 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.348+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.357 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.348+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgrstat=7,mon_config_key=10,monmap=2,osd_metadata=6,osd_snap=39,osdmap=36} crc {mgrstat=4026055356,mon_config_key=1530417952,monmap=268092033,osd_metadata=4182959540,osd_snap=657291659,osdmap=3950454859}) 2024-09-16T04:00:55.357 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.348+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mgrstat=7,mon_config_key=10,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=37,osdmap=36} crc {mgrstat=4026055356,mon_config_key=1481991774,monmap=3937513981,osd_metadata=4182959540,osd_pg_creating=3163125691,osd_snap=1897778942,osdmap=3639528075}) 2024-09-16T04:00:55.360 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.352+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.360 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.352+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1986906692}) 2024-09-16T04:00:55.360 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.352+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=890035438}) 2024-09-16T04:00:55.360 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.352+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.360 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.352+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1986906692}) 2024-09-16T04:00:55.360 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.352+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2462867812}) 2024-09-16T04:00:55.373 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.364+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.374 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.368+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=44} crc {osdmap=18279332}) 2024-09-16T04:00:55.374 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.368+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=44} crc {osdmap=4015723471}) 2024-09-16T04:00:55.374 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.368+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:00:55.374 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.368+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=44} crc {osdmap=18279332}) 2024-09-16T04:00:55.374 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:00:55.368+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=46} crc {osdmap=2737035194}) 2024-09-16T04:00:56.114 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T04:00:56.115 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:00:56.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.240+0000 7f1027f4f640 10 monclient: get_monmap_and_config 2024-09-16T04:00:56.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.240+0000 7f1027f4f640 10 monclient: build_initial_monmap 2024-09-16T04:00:56.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.240+0000 7f1027f4f640 10 monclient: monmap: 2024-09-16T04:00:56.248 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:00:56.248 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:00:56.248 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:00:56.246511+0000 2024-09-16T04:00:56.248 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:00:56.246511+0000 2024-09-16T04:00:56.248 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:00:56.248 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:00:56.248 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:00:56.249 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:00:56.249 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:00:56.249 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:00:56.249 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:00:56.249 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:00:56.251 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.240+0000 7f1027f4f640 1 Processor -- start 2024-09-16T04:00:56.251 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.240+0000 7f1027f4f640 1 -- start start 2024-09-16T04:00:56.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.240+0000 7f1027f4f640 10 monclient: init 2024-09-16T04:00:56.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.240+0000 7f1027f4f640 10 monclient: _reopen_session rank -1 2024-09-16T04:00:56.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.240+0000 7f1027f4f640 10 monclient: _add_conns ranks=[1,3,4,0,2] 2024-09-16T04:00:56.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.240+0000 7f1027f4f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f10200a1330 addr v1:172.21.15.154:6789/0 2024-09-16T04:00:56.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.240+0000 7f1027f4f640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f102006dac0 0x7f102013a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:00:56.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.240+0000 7f1027f4f640 10 monclient(hunting): picked mon.noname-a con 0x7f102006dac0 addr v2:172.21.15.42:3300/0 2024-09-16T04:00:56.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.240+0000 7f1027f4f640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f10200a1e20 0x7f102013c890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:00:56.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.240+0000 7f1027f4f640 10 monclient(hunting): picked mon.noname-c con 0x7f10200a1e20 addr v2:172.21.15.154:3300/0 2024-09-16T04:00:56.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.240+0000 7f1027f4f640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f102009c090 con 0x7f10200a1330 2024-09-16T04:00:56.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.240+0000 7f1027f4f640 10 monclient(hunting): start opening mon connection 2024-09-16T04:00:56.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.240+0000 7f1027f4f640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f102009f610 con 0x7f102006dac0 2024-09-16T04:00:56.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.240+0000 7f1027f4f640 10 monclient(hunting): start opening mon connection 2024-09-16T04:00:56.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.240+0000 7f1027f4f640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f102009f8f0 con 0x7f10200a1e20 2024-09-16T04:00:56.253 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.240+0000 7f1027f4f640 10 monclient(hunting): _renew_subs 2024-09-16T04:00:56.253 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.240+0000 7f1027f4f640 10 monclient(hunting): authenticate will time out at 2103.215946s 2024-09-16T04:00:56.253 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.244+0000 7f10254ea640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f102006dac0 0x7f102013a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:00:56.253 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.244+0000 7f10254ea640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f102006dac0 0x7f102013a340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:44100/0 (socket says 172.21.15.42:44100) 2024-09-16T04:00:56.253 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.244+0000 7f10254ea640 1 -- 172.21.15.42:0/1522340659 learned_addr learned my addr 172.21.15.42:0/1522340659 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:00:56.253 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.244+0000 7f10254ea640 10 monclient(hunting): get_auth_request con 0x7f102006dac0 auth_method 0 2024-09-16T04:00:56.253 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.244+0000 7f10254ea640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:00:56.253 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.244+0000 7f10254ea640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:00:56.253 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.244+0000 7f10254ea640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:00:56.253 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.244+0000 7f10254ea640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:00:56.253 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.244+0000 7f10254ea640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:00:56.253 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.244+0000 7f10254ea640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:00:56.253 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.244+0000 7f10264ec640 1 --2- 172.21.15.42:0/1522340659 >> v2:172.21.15.154:3300/0 conn(0x7f10200a1e20 0x7f102013c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:00:56.254 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.244+0000 7f10264ec640 10 monclient(hunting): get_auth_request con 0x7f10200a1e20 auth_method 0 2024-09-16T04:00:56.254 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.244+0000 7f10264ec640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:00:56.254 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.244+0000 7f10264ec640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:00:56.254 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.244+0000 7f10264ec640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:00:56.254 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.244+0000 7f10264ec640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:00:56.254 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.244+0000 7f10264ec640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:00:56.254 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.244+0000 7f10264ec640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:00:56.254 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.244+0000 7f10254ea640 10 monclient(hunting): handle_auth_done global_id 825288 payload 274 2024-09-16T04:00:56.254 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.244+0000 7f10254ea640 1 -- 172.21.15.42:0/1522340659 >> v2:172.21.15.154:3300/0 conn(0x7f10200a1e20 msgr2=0x7f102013c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:00:56.254 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.244+0000 7f10254ea640 1 --2- 172.21.15.42:0/1522340659 >> v2:172.21.15.154:3300/0 conn(0x7f10200a1e20 0x7f102013c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:00:56.254 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.244+0000 7f10254ea640 1 -- 172.21.15.42:0/1522340659 >> v1:172.21.15.154:6789/0 conn(0x7f10200a1330 legacy=0x7f10200a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:00:56.254 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.244+0000 7f10254ea640 10 monclient: _finish_hunting 0 2024-09-16T04:00:56.254 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.244+0000 7f10254ea640 1 monclient: found mon.noname-a 2024-09-16T04:00:56.254 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.244+0000 7f10254ea640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:00:56.255 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.244+0000 7f10254ea640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:00:56.255 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.244+0000 7f10254ea640 1 -- 172.21.15.42:0/1522340659 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f102009fbd0 con 0x7f102006dac0 2024-09-16T04:00:56.260 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.244+0000 7f10254ea640 1 --2- 172.21.15.42:0/1522340659 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f102006dac0 0x7f102013a340 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f102009c800 tx=0x7f101401c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98d7497658d54813 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:00:56.260 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.244+0000 7f1024ce9640 1 -- 172.21.15.42:0/1522340659 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10140a5600 con 0x7f102006dac0 2024-09-16T04:00:56.260 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.244+0000 7f1024ce9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:00:56.261 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.244+0000 7f1024ce9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:00:56.261 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.244+0000 7f1024ce9640 10 monclient: dump: 2024-09-16T04:00:56.261 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:00:56.261 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:00:56.261 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:00:56.261 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:00:56.261 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:00:56.261 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:00:56.261 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:00:56.261 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:00:56.261 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:00:56.261 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:00:56.261 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.244+0000 7f1024ce9640 10 monclient: _finish_auth 0 2024-09-16T04:00:56.262 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.244+0000 7f1024ce9640 10 monclient: _check_auth_tickets 2024-09-16T04:00:56.262 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.244+0000 7f1024ce9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:00:56.262 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.244+0000 7f1024ce9640 1 -- 172.21.15.42:0/1522340659 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f10140a57a0 con 0x7f102006dac0 2024-09-16T04:00:56.262 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.244+0000 7f1027f4f640 5 monclient: authenticate success, global_id 825288 2024-09-16T04:00:56.262 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.244+0000 7f1027f4f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:00:56.262 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.244+0000 7f1024ce9640 10 monclient: handle_config config(0 keys) 2024-09-16T04:00:56.262 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.244+0000 7f1027f4f640 10 monclient: get_monmap_and_config success 2024-09-16T04:00:56.262 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.244+0000 7f1024ce9640 1 -- 172.21.15.42:0/1522340659 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10140a5b70 con 0x7f102006dac0 2024-09-16T04:00:56.262 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.244+0000 7f1027f4f640 4 set_mon_vals no callback set 2024-09-16T04:00:56.262 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.244+0000 7f1024ce9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:00:56.262 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.244+0000 7f1024ce9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:00:56.262 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.244+0000 7f1024ce9640 10 monclient: dump: 2024-09-16T04:00:56.262 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:00:56.263 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:00:56.263 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:00:56.263 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:00:56.263 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:00:56.263 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:00:56.263 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:00:56.263 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:00:56.263 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:00:56.263 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:00:56.263 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.248+0000 7f1027f4f640 10 monclient: shutdown 2024-09-16T04:00:56.263 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.248+0000 7f1027f4f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:00:56.263 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.248+0000 7f1027f4f640 1 -- 172.21.15.42:0/1522340659 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f102006dac0 msgr2=0x7f102013a340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:00:56.263 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.248+0000 7f1027f4f640 1 --2- 172.21.15.42:0/1522340659 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f102006dac0 0x7f102013a340 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f102009c800 tx=0x7f101401c940 comp rx=0 tx=0).stop 2024-09-16T04:00:56.263 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.248+0000 7f1027f4f640 1 -- 172.21.15.42:0/1522340659 shutdown_connections 2024-09-16T04:00:56.264 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.248+0000 7f1027f4f640 1 --2- 172.21.15.42:0/1522340659 >> v2:172.21.15.154:3300/0 conn(0x7f10200a1e20 0x7f102013c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:00:56.264 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.248+0000 7f1027f4f640 1 --2- 172.21.15.42:0/1522340659 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f102006dac0 0x7f102013a340 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:00:56.264 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.248+0000 7f1027f4f640 1 -- 172.21.15.42:0/1522340659 >> 172.21.15.42:0/1522340659 conn(0x7f10200a2420 msgr2=0x7f10200a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:00:56.264 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.248+0000 7f1027f4f640 1 -- 172.21.15.42:0/1522340659 shutdown_connections 2024-09-16T04:00:56.264 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.248+0000 7f1027f4f640 1 -- 172.21.15.42:0/1522340659 wait complete. 2024-09-16T04:00:56.264 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.248+0000 7f1027f4f640 10 monclient: build_initial_monmap 2024-09-16T04:00:56.264 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.248+0000 7f1027f4f640 10 monclient: monmap: 2024-09-16T04:00:56.264 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:00:56.264 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:00:56.264 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:00:56.264 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:00:56.264 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:00:56.265 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:00:56.265 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:00:56.265 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:00:56.265 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:00:56.265 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:00:56.265 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.248+0000 7f1027f4f640 1 librados: starting msgr at 2024-09-16T04:00:56.265 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.248+0000 7f1027f4f640 1 librados: starting objecter 2024-09-16T04:00:56.265 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.248+0000 7f1027f4f640 1 Processor -- start 2024-09-16T04:00:56.265 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.248+0000 7f1027f4f640 1 -- start start 2024-09-16T04:00:56.265 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.248+0000 7f1027f4f640 1 librados: setting wanted keys 2024-09-16T04:00:56.265 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.248+0000 7f1027f4f640 1 librados: calling monclient init 2024-09-16T04:00:56.265 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.248+0000 7f1027f4f640 10 monclient: init 2024-09-16T04:00:56.265 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.248+0000 7f1027f4f640 10 monclient: _reopen_session rank -1 2024-09-16T04:00:56.266 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.248+0000 7f1027f4f640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:00:56.266 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.248+0000 7f1027f4f640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f10200a1330 0x7f10201cef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:00:56.266 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.248+0000 7f1027f4f640 10 monclient(hunting): picked mon.noname-a con 0x7f10200a1330 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:00:56.266 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.248+0000 7f1027f4f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f10200a1e20 0x7f1020135360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:00:56.266 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.248+0000 7f1027f4f640 10 monclient(hunting): picked mon.noname-b con 0x7f10200a1e20 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:00:56.266 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.248+0000 7f1027f4f640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f10201358a0 0x7f102009cb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:00:56.266 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.248+0000 7f1027f4f640 10 monclient(hunting): picked mon.noname-c con 0x7f10201358a0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:00:56.266 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.248+0000 7f1027f4f640 10 monclient(hunting): start opening mon connection 2024-09-16T04:00:56.266 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.248+0000 7f1027f4f640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f102013f800 con 0x7f10200a1330 2024-09-16T04:00:56.266 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.248+0000 7f1027f4f640 10 monclient(hunting): start opening mon connection 2024-09-16T04:00:56.266 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.248+0000 7f1027f4f640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f102009ef80 con 0x7f10200a1e20 2024-09-16T04:00:56.266 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.248+0000 7f1027f4f640 10 monclient(hunting): start opening mon connection 2024-09-16T04:00:56.266 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.248+0000 7f1027f4f640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f102013f980 con 0x7f10201358a0 2024-09-16T04:00:56.266 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.248+0000 7f1027f4f640 10 monclient(hunting): _renew_subs 2024-09-16T04:00:56.267 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.248+0000 7f1027f4f640 10 monclient(hunting): authenticate will time out at 2103.221852s 2024-09-16T04:00:56.267 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.248+0000 7f1025ceb640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f10200a1330 0x7f10201cef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:00:56.267 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.248+0000 7f1025ceb640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f10200a1330 0x7f10201cef00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:44110/0 (socket says 172.21.15.42:44110) 2024-09-16T04:00:56.267 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.248+0000 7f1025ceb640 1 -- 172.21.15.42:0/1862895352 learned_addr learned my addr 172.21.15.42:0/1862895352 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:00:56.267 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.248+0000 7f1025ceb640 10 monclient(hunting): get_auth_request con 0x7f10200a1330 auth_method 0 2024-09-16T04:00:56.267 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.248+0000 7f1025ceb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:00:56.267 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.248+0000 7f1025ceb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:00:56.267 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.248+0000 7f1025ceb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:00:56.267 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.248+0000 7f10264ec640 1 --2- 172.21.15.42:0/1862895352 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f10201358a0 0x7f102009cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:00:56.267 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.248+0000 7f1025ceb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:00:56.267 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.248+0000 7f1025ceb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:00:56.267 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.248+0000 7f1025ceb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:00:56.267 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.248+0000 7f10264ec640 10 monclient(hunting): get_auth_request con 0x7f10201358a0 auth_method 0 2024-09-16T04:00:56.267 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.248+0000 7f10264ec640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:00:56.268 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.248+0000 7f10264ec640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:00:56.268 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.248+0000 7f10264ec640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:00:56.268 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.248+0000 7f10264ec640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:00:56.268 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.248+0000 7f10264ec640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:00:56.268 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.248+0000 7f10264ec640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:00:56.268 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.248+0000 7f10254ea640 1 --2- 172.21.15.42:0/1862895352 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f10200a1e20 0x7f1020135360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:00:56.268 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.248+0000 7f1025ceb640 10 monclient(hunting): handle_auth_done global_id 825291 payload 824 2024-09-16T04:00:56.268 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.248+0000 7f1025ceb640 1 -- 172.21.15.42:0/1862895352 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f10201358a0 msgr2=0x7f102009cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:00:56.268 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.248+0000 7f1025ceb640 1 --2- 172.21.15.42:0/1862895352 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f10201358a0 0x7f102009cb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:00:56.268 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.248+0000 7f1025ceb640 1 -- 172.21.15.42:0/1862895352 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f10200a1e20 msgr2=0x7f1020135360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:00:56.268 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.248+0000 7f1025ceb640 1 --2- 172.21.15.42:0/1862895352 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f10200a1e20 0x7f1020135360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:00:56.268 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.248+0000 7f1025ceb640 10 monclient: _finish_hunting 0 2024-09-16T04:00:56.269 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.248+0000 7f1025ceb640 1 monclient: found mon.noname-a 2024-09-16T04:00:56.269 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.248+0000 7f1025ceb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:00:56.269 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.248+0000 7f1025ceb640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:00:56.269 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.248+0000 7f1025ceb640 1 -- 172.21.15.42:0/1862895352 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1020073a30 con 0x7f10200a1330 2024-09-16T04:00:56.269 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.248+0000 7f10254ea640 10 monclient: get_auth_request con 0x7f10200a1e20 auth_method 0 2024-09-16T04:00:56.269 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.248+0000 7f10254ea640 1 --2- 172.21.15.42:0/1862895352 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f10200a1e20 0x7f1020135360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T04:00:56.269 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.248+0000 7f1025ceb640 1 --2- 172.21.15.42:0/1862895352 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f10200a1330 0x7f10201cef00 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f101000c450 tx=0x7f101000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:00:56.269 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.248+0000 7f101e7fc640 10 client.?.objecter ms_handle_connect 0x7f10200a1330 2024-09-16T04:00:56.269 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.248+0000 7f101e7fc640 10 client.?.objecter resend_mon_ops 2024-09-16T04:00:56.269 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.248+0000 7f101e7fc640 1 -- 172.21.15.42:0/1862895352 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1010016020 con 0x7f10200a1330 2024-09-16T04:00:56.269 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.248+0000 7f101e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:00:56.269 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.248+0000 7f101e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:00:56.269 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.248+0000 7f101e7fc640 10 monclient: dump: 2024-09-16T04:00:56.270 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:00:56.270 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:00:56.270 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:00:56.270 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:00:56.270 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:00:56.270 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:00:56.270 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:00:56.270 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:00:56.270 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:00:56.270 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:00:56.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.248+0000 7f101e7fc640 10 monclient: _finish_auth 0 2024-09-16T04:00:56.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.248+0000 7f101e7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:00:56.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.248+0000 7f101e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:00:56.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.248+0000 7f101e7fc640 1 -- 172.21.15.42:0/1862895352 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f10100052d0 con 0x7f10200a1330 2024-09-16T04:00:56.271 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.248+0000 7f101e7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:00:56.271 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.248+0000 7f101e7fc640 1 -- 172.21.15.42:0/1862895352 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1010007260 con 0x7f10200a1330 2024-09-16T04:00:56.271 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.248+0000 7f1027f4f640 5 monclient: authenticate success, global_id 825291 2024-09-16T04:00:56.271 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.248+0000 7f1027f4f640 10 monclient: _renew_subs 2024-09-16T04:00:56.271 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.248+0000 7f1027f4f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:00:56.271 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.248+0000 7f1027f4f640 1 -- 172.21.15.42:0/1862895352 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f102009e410 con 0x7f10200a1330 2024-09-16T04:00:56.271 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.248+0000 7f101ffff640 4 set_mon_vals no callback set 2024-09-16T04:00:56.271 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.248+0000 7f101e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:00:56.271 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.248+0000 7f101e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:00:56.271 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.248+0000 7f101e7fc640 10 monclient: dump: 2024-09-16T04:00:56.271 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:00:56.271 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:00:56.271 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:00:56.271 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:00:56.272 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:00:56.272 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:00:56.272 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:00:56.272 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:00:56.272 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:00:56.272 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:00:56.272 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.248+0000 7f1027f4f640 10 client.825291.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:00:56.272 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.248+0000 7f1027f4f640 10 monclient: _renew_subs 2024-09-16T04:00:56.272 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.248+0000 7f1027f4f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:00:56.272 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.248+0000 7f1027f4f640 1 -- 172.21.15.42:0/1862895352 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1020120580 con 0x7f10200a1330 2024-09-16T04:00:56.272 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.248+0000 7f1027f4f640 1 librados: init done 2024-09-16T04:00:56.272 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.248+0000 7f1027f4f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:00:56.272 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.248+0000 7f1027f4f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:00:56.272 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.248+0000 7f1027f4f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:00:56.273 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.248+0000 7f1027f4f640 1 -- 172.21.15.42:0/1862895352 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f10200a1710 con 0x7f10200a1330 2024-09-16T04:00:56.273 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.248+0000 7f101e7fc640 1 -- 172.21.15.42:0/1862895352 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f1010007400 con 0x7f10200a1330 2024-09-16T04:00:56.273 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.252+0000 7f101e7fc640 1 --2- 172.21.15.42:0/1862895352 >> v2:172.21.15.154:6812/254845342 conn(0x7f100c037810 0x7f100c039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:00:56.273 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.252+0000 7f101e7fc640 1 -- 172.21.15.42:0/1862895352 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(568..568 src has 451..568) ==== 6224+0+0 (secure 0 0 0) 0x7f1010048b20 con 0x7f10200a1330 2024-09-16T04:00:56.273 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.252+0000 7f101e7fc640 10 client.825291.objecter ms_dispatch 0x7f1020000ba0 osd_map(568..568 src has 451..568) 2024-09-16T04:00:56.273 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.252+0000 7f101e7fc640 3 client.825291.objecter handle_osd_map got epochs [568,568] > 0 2024-09-16T04:00:56.273 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.252+0000 7f101e7fc640 3 client.825291.objecter handle_osd_map decoding full epoch 568 2024-09-16T04:00:56.273 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.252+0000 7f101e7fc640 20 client.825291.objecter dump_active .. 0 homeless 2024-09-16T04:00:56.273 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.252+0000 7f10254ea640 1 --2- 172.21.15.42:0/1862895352 >> v2:172.21.15.154:6812/254845342 conn(0x7f100c037810 0x7f100c039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:00:56.273 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.252+0000 7f10254ea640 10 monclient: get_auth_request con 0x7f100c037810 auth_method 0 2024-09-16T04:00:56.273 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.252+0000 7f10254ea640 1 --2- 172.21.15.42:0/1862895352 >> v2:172.21.15.154:6812/254845342 conn(0x7f100c037810 0x7f100c039cd0 secure :-1 s=READY pgs=3750 cs=0 l=1 rev1=1 crypto rx=0x7f10140ae000 tx=0x7f10140a7070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:00:56.273 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.252+0000 7f101e7fc640 10 client.825291.objecter ms_handle_connect 0x7f100c037810 2024-09-16T04:00:56.274 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.256+0000 7f101e7fc640 1 -- 172.21.15.42:0/1862895352 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f1010020020 con 0x7f10200a1330 2024-09-16T04:00:56.274 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.256+0000 7f101e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:00:56.274 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.256+0000 7f101e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:00:56.489 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.480+0000 7f1027f4f640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:00:56.489 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.480+0000 7f1027f4f640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:00:56.489 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.480+0000 7f1027f4f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:00:56.489 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.480+0000 7f1027f4f640 1 -- 172.21.15.42:0/1862895352 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f10201210e0 con 0x7f10200a1330 2024-09-16T04:00:56.490 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.484+0000 7f101e7fc640 1 -- 172.21.15.42:0/1862895352 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7f1020073a30 con 0x7f10200a1330 2024-09-16T04:00:56.490 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.484+0000 7f101e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:00:56.490 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.484+0000 7f101e7fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:00:56.490 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":168,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:00:56.492 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.484+0000 7f1027f4f640 10 librados: watch_flush enter 2024-09-16T04:00:56.493 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.484+0000 7f1027f4f640 10 librados: watch_flush exit 2024-09-16T04:00:56.493 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.484+0000 7f1027f4f640 20 client.825291.objecter shutdown clearing up homeless session... 2024-09-16T04:00:56.493 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.484+0000 7f1027f4f640 10 client.825291.objecter successfully canceled tick 2024-09-16T04:00:56.493 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.484+0000 7f1027f4f640 1 -- 172.21.15.42:0/1862895352 >> v2:172.21.15.154:6812/254845342 conn(0x7f100c037810 msgr2=0x7f100c039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:00:56.493 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.484+0000 7f1027f4f640 1 --2- 172.21.15.42:0/1862895352 >> v2:172.21.15.154:6812/254845342 conn(0x7f100c037810 0x7f100c039cd0 secure :-1 s=READY pgs=3750 cs=0 l=1 rev1=1 crypto rx=0x7f10140ae000 tx=0x7f10140a7070 comp rx=0 tx=0).stop 2024-09-16T04:00:56.493 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.484+0000 7f1027f4f640 10 monclient: shutdown 2024-09-16T04:00:56.493 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.484+0000 7f1027f4f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:00:56.493 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.484+0000 7f1027f4f640 1 -- 172.21.15.42:0/1862895352 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f10200a1330 msgr2=0x7f10201cef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:00:56.493 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.484+0000 7f1027f4f640 1 --2- 172.21.15.42:0/1862895352 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f10200a1330 0x7f10201cef00 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f101000c450 tx=0x7f101000c910 comp rx=0 tx=0).stop 2024-09-16T04:00:56.493 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.484+0000 7f1027f4f640 1 -- 172.21.15.42:0/1862895352 shutdown_connections 2024-09-16T04:00:56.494 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.484+0000 7f1027f4f640 1 --2- 172.21.15.42:0/1862895352 >> v2:172.21.15.154:6812/254845342 conn(0x7f100c037810 0x7f100c039cd0 unknown :-1 s=CLOSED pgs=3750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:00:56.494 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.484+0000 7f1027f4f640 1 --2- 172.21.15.42:0/1862895352 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f10201358a0 0x7f102009cb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:00:56.494 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.484+0000 7f1027f4f640 1 --2- 172.21.15.42:0/1862895352 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f10200a1e20 0x7f1020135360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:00:56.494 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.484+0000 7f1027f4f640 1 --2- 172.21.15.42:0/1862895352 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f10200a1330 0x7f10201cef00 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:00:56.494 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.484+0000 7f1027f4f640 1 -- 172.21.15.42:0/1862895352 >> 172.21.15.42:0/1862895352 conn(0x7f10200a2420 msgr2=0x7f10200a45b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:00:56.494 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.484+0000 7f1027f4f640 1 -- 172.21.15.42:0/1862895352 shutdown_connections 2024-09-16T04:00:56.494 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.484+0000 7f1027f4f640 1 -- 172.21.15.42:0/1862895352 wait complete. 2024-09-16T04:00:56.494 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.484+0000 7f1027f4f640 1 librados: shutdown 2024-09-16T04:00:56.510 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T04:00:56.510 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T04:00:56.510 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T04:00:56.510 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T04:00:56.639 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.632+0000 7f271b1e2640 10 monclient: get_monmap_and_config 2024-09-16T04:00:56.639 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.632+0000 7f271b1e2640 10 monclient: build_initial_monmap 2024-09-16T04:00:56.639 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.632+0000 7f271b1e2640 10 monclient: monmap: 2024-09-16T04:00:56.639 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:00:56.639 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:00:56.639 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:00:56.637508+0000 2024-09-16T04:00:56.639 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:00:56.637508+0000 2024-09-16T04:00:56.639 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:00:56.639 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:00:56.639 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:00:56.639 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:00:56.640 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:00:56.640 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:00:56.640 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:00:56.640 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:00:56.640 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.632+0000 7f271b1e2640 1 Processor -- start 2024-09-16T04:00:56.640 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.632+0000 7f271b1e2640 1 -- start start 2024-09-16T04:00:56.640 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.632+0000 7f271b1e2640 10 monclient: init 2024-09-16T04:00:56.642 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.632+0000 7f271b1e2640 10 monclient: _reopen_session rank -1 2024-09-16T04:00:56.642 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.632+0000 7f271b1e2640 10 monclient: _add_conns ranks=[1,0,2,3,4] 2024-09-16T04:00:56.642 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.632+0000 7f271b1e2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f271406db90 addr v1:172.21.15.154:6789/0 2024-09-16T04:00:56.642 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.632+0000 7f271b1e2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f27141340c0 addr v1:172.21.15.42:6789/0 2024-09-16T04:00:56.642 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.632+0000 7f271b1e2640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f271413d8b0 0x7f271413fca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:00:56.643 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.632+0000 7f271b1e2640 10 monclient(hunting): picked mon.noname-b con 0x7f271413d8b0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:00:56.643 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.632+0000 7f271b1e2640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f271406ece0 con 0x7f27141340c0 2024-09-16T04:00:56.643 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.632+0000 7f271b1e2640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f271409bf80 con 0x7f271406db90 2024-09-16T04:00:56.643 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.632+0000 7f271b1e2640 10 monclient(hunting): start opening mon connection 2024-09-16T04:00:56.643 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.632+0000 7f271b1e2640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f271406df70 con 0x7f271413d8b0 2024-09-16T04:00:56.643 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.632+0000 7f271b1e2640 10 monclient(hunting): _renew_subs 2024-09-16T04:00:56.643 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.632+0000 7f271b1e2640 10 monclient(hunting): authenticate will time out at 2103.607742s 2024-09-16T04:00:56.643 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.632+0000 7f271977f640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f271413d8b0 0x7f271413fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:00:56.643 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.632+0000 7f2713fff640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f27141340c0 0x7f271413c190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:56228/0 (socket says 172.21.15.42:56228) 2024-09-16T04:00:56.643 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.632+0000 7f2713fff640 1 -- 172.21.15.42:0/4272069731 learned_addr learned my addr 172.21.15.42:0/4272069731 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T04:00:56.645 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.632+0000 7f271977f640 10 monclient(hunting): get_auth_request con 0x7f271413d8b0 auth_method 0 2024-09-16T04:00:56.645 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.632+0000 7f271977f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:00:56.645 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.632+0000 7f271977f640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:00:56.645 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.632+0000 7f271977f640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:00:56.645 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.636+0000 7f271977f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:00:56.645 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.636+0000 7f271977f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:00:56.645 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.636+0000 7f271977f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:00:56.645 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.636+0000 7f271977f640 10 monclient(hunting): handle_auth_done global_id 944132 payload 274 2024-09-16T04:00:56.645 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.636+0000 7f271977f640 1 -- 172.21.15.42:0/4272069731 >> v1:172.21.15.154:6789/0 conn(0x7f271406db90 legacy=0x7f27140a37f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:00:56.646 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.636+0000 7f271977f640 1 -- 172.21.15.42:0/4272069731 >> v1:172.21.15.42:6789/0 conn(0x7f27141340c0 legacy=0x7f271413c190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:00:56.646 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.636+0000 7f271977f640 10 monclient: _finish_hunting 0 2024-09-16T04:00:56.646 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.636+0000 7f271977f640 1 monclient: found mon.noname-b 2024-09-16T04:00:56.646 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.636+0000 7f271977f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:00:56.646 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.636+0000 7f271977f640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T04:00:56.646 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.636+0000 7f271977f640 1 -- 172.21.15.42:0/4272069731 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27141344d0 con 0x7f271413d8b0 2024-09-16T04:00:56.646 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.636+0000 7f27137fe640 1 -- 172.21.15.42:0/4272069731 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2280533540 0 0) 0x7f271406ece0 con 0x7f27141340c0 2024-09-16T04:00:56.647 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.636+0000 7f27137fe640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T04:00:56.655 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.636+0000 7f271977f640 1 --2- 172.21.15.42:0/4272069731 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f271413d8b0 0x7f271413fca0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f270c0097f0 tx=0x7f270c01c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dbf7655dae65d94b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:00:56.655 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.636+0000 7f27137fe640 1 -- 172.21.15.42:0/4272069731 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f270c0b1070 con 0x7f271413d8b0 2024-09-16T04:00:56.655 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.636+0000 7f27137fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:00:56.656 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.636+0000 7f27137fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:00:56.656 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.636+0000 7f27137fe640 10 monclient: dump: 2024-09-16T04:00:56.656 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:00:56.656 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:00:56.656 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:00:56.656 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:00:56.656 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:00:56.656 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:00:56.656 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:00:56.656 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:00:56.656 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:00:56.656 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:00:56.657 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.636+0000 7f27137fe640 10 monclient: _finish_auth 0 2024-09-16T04:00:56.657 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.636+0000 7f27137fe640 10 monclient: _check_auth_tickets 2024-09-16T04:00:56.657 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.636+0000 7f27137fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:00:56.657 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.636+0000 7f27137fe640 1 -- 172.21.15.42:0/4272069731 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f270c0ac600 con 0x7f271413d8b0 2024-09-16T04:00:56.657 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.636+0000 7f27137fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:00:56.657 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.636+0000 7f27137fe640 1 -- 172.21.15.42:0/4272069731 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f270c0ac9d0 con 0x7f271413d8b0 2024-09-16T04:00:56.657 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.636+0000 7f271b1e2640 5 monclient: authenticate success, global_id 944132 2024-09-16T04:00:56.657 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.636+0000 7f271b1e2640 10 monclient: get_monmap_and_config success 2024-09-16T04:00:56.657 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.636+0000 7f27137fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:00:56.657 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.636+0000 7f271b1e2640 4 set_mon_vals no callback set 2024-09-16T04:00:56.658 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.636+0000 7f27137fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:00:56.658 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.636+0000 7f27137fe640 10 monclient: dump: 2024-09-16T04:00:56.658 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:00:56.658 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:00:56.658 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:00:56.658 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:00:56.658 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:00:56.658 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:00:56.658 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:00:56.658 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:00:56.658 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:00:56.658 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:00:56.658 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.636+0000 7f271b1e2640 10 monclient: shutdown 2024-09-16T04:00:56.658 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.636+0000 7f271b1e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:00:56.659 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.636+0000 7f271b1e2640 1 -- 172.21.15.42:0/4272069731 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f271413d8b0 msgr2=0x7f271413fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:00:56.659 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.636+0000 7f271b1e2640 1 --2- 172.21.15.42:0/4272069731 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f271413d8b0 0x7f271413fca0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f270c0097f0 tx=0x7f270c01c3a0 comp rx=0 tx=0).stop 2024-09-16T04:00:56.659 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.636+0000 7f271b1e2640 1 -- 172.21.15.42:0/4272069731 shutdown_connections 2024-09-16T04:00:56.659 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.636+0000 7f271b1e2640 1 --2- 172.21.15.42:0/4272069731 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f271413d8b0 0x7f271413fca0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:00:56.659 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.636+0000 7f271b1e2640 1 -- 172.21.15.42:0/4272069731 >> 172.21.15.42:0/4272069731 conn(0x7f27140a9fb0 msgr2=0x7f27140aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:00:56.659 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.636+0000 7f271b1e2640 1 -- 172.21.15.42:0/4272069731 shutdown_connections 2024-09-16T04:00:56.659 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.636+0000 7f271b1e2640 1 -- 172.21.15.42:0/4272069731 wait complete. 2024-09-16T04:00:56.659 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.636+0000 7f271b1e2640 10 monclient: build_initial_monmap 2024-09-16T04:00:56.659 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.636+0000 7f271b1e2640 10 monclient: monmap: 2024-09-16T04:00:56.659 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:00:56.659 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:00:56.659 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:00:56.659 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:00:56.660 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:00:56.660 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:00:56.660 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:00:56.660 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:00:56.660 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:00:56.660 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:00:56.660 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.640+0000 7f271b1e2640 1 librados: starting msgr at 2024-09-16T04:00:56.660 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.640+0000 7f271b1e2640 1 librados: starting objecter 2024-09-16T04:00:56.660 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.640+0000 7f271b1e2640 1 Processor -- start 2024-09-16T04:00:56.660 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.640+0000 7f271b1e2640 1 -- start start 2024-09-16T04:00:56.660 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.640+0000 7f271b1e2640 1 librados: setting wanted keys 2024-09-16T04:00:56.660 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.640+0000 7f271b1e2640 1 librados: calling monclient init 2024-09-16T04:00:56.660 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.640+0000 7f271b1e2640 10 monclient: init 2024-09-16T04:00:56.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.640+0000 7f271b1e2640 10 monclient: _reopen_session rank -1 2024-09-16T04:00:56.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.640+0000 7f271b1e2640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:00:56.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.640+0000 7f271b1e2640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f271406db90 0x7f27141ceff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:00:56.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.640+0000 7f271b1e2640 10 monclient(hunting): picked mon.noname-a con 0x7f271406db90 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:00:56.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.640+0000 7f271b1e2640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f27141340c0 0x7f27141cf530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:00:56.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.640+0000 7f271b1e2640 10 monclient(hunting): picked mon.noname-c con 0x7f27141340c0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:00:56.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.640+0000 7f271b1e2640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f271413d8b0 0x7f2714139930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:00:56.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.640+0000 7f271b1e2640 10 monclient(hunting): picked mon.noname-b con 0x7f271413d8b0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:00:56.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.640+0000 7f271b1e2640 10 monclient(hunting): start opening mon connection 2024-09-16T04:00:56.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.640+0000 7f271b1e2640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f2714142b80 con 0x7f271406db90 2024-09-16T04:00:56.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.640+0000 7f271b1e2640 10 monclient(hunting): start opening mon connection 2024-09-16T04:00:56.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.640+0000 7f271b1e2640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f2714142a00 con 0x7f271413d8b0 2024-09-16T04:00:56.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.640+0000 7f271b1e2640 10 monclient(hunting): start opening mon connection 2024-09-16T04:00:56.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.640+0000 7f271b1e2640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f2714142d00 con 0x7f27141340c0 2024-09-16T04:00:56.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.640+0000 7f271b1e2640 10 monclient(hunting): _renew_subs 2024-09-16T04:00:56.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.640+0000 7f271b1e2640 10 monclient(hunting): authenticate will time out at 2103.612945s 2024-09-16T04:00:56.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.640+0000 7f2718f7e640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f271406db90 0x7f27141ceff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:00:56.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.640+0000 7f2713fff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f27141340c0 0x7f27141cf530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:00:56.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.640+0000 7f2718f7e640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f271406db90 0x7f27141ceff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:44122/0 (socket says 172.21.15.42:44122) 2024-09-16T04:00:56.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.640+0000 7f2718f7e640 1 -- 172.21.15.42:0/199164292 learned_addr learned my addr 172.21.15.42:0/199164292 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:00:56.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.640+0000 7f2718f7e640 10 monclient(hunting): get_auth_request con 0x7f271406db90 auth_method 0 2024-09-16T04:00:56.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.640+0000 7f2718f7e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:00:56.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.640+0000 7f2718f7e640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:00:56.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.640+0000 7f2718f7e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:00:56.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.640+0000 7f2713fff640 10 monclient(hunting): get_auth_request con 0x7f27141340c0 auth_method 0 2024-09-16T04:00:56.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.640+0000 7f2713fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:00:56.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.640+0000 7f2713fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:00:56.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.640+0000 7f2713fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:00:56.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.640+0000 7f2718f7e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:00:56.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.640+0000 7f2718f7e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:00:56.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.640+0000 7f2718f7e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:00:56.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.640+0000 7f2713fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:00:56.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.640+0000 7f2713fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:00:56.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.640+0000 7f2713fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:00:56.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.640+0000 7f2718f7e640 10 monclient(hunting): handle_auth_done global_id 825297 payload 824 2024-09-16T04:00:56.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.640+0000 7f271977f640 1 --2- 172.21.15.42:0/199164292 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f271413d8b0 0x7f2714139930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:00:56.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.640+0000 7f2718f7e640 1 -- 172.21.15.42:0/199164292 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f27141340c0 msgr2=0x7f27141cf530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:00:56.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.640+0000 7f2718f7e640 1 --2- 172.21.15.42:0/199164292 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f27141340c0 0x7f27141cf530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:00:56.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.640+0000 7f2718f7e640 1 -- 172.21.15.42:0/199164292 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f271413d8b0 msgr2=0x7f2714139930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:00:56.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.640+0000 7f2718f7e640 1 --2- 172.21.15.42:0/199164292 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f271413d8b0 0x7f2714139930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:00:56.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.640+0000 7f2718f7e640 10 monclient: _finish_hunting 0 2024-09-16T04:00:56.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.640+0000 7f2718f7e640 1 monclient: found mon.noname-a 2024-09-16T04:00:56.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.640+0000 7f2718f7e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:00:56.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.640+0000 7f2718f7e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:00:56.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.640+0000 7f2718f7e640 1 -- 172.21.15.42:0/199164292 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f271409da30 con 0x7f271406db90 2024-09-16T04:00:56.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.640+0000 7f2718f7e640 1 --2- 172.21.15.42:0/199164292 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f271406db90 0x7f27141ceff0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f270400c7a0 tx=0x7f270400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:00:56.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.640+0000 7f27117fa640 10 client.?.objecter ms_handle_connect 0x7f271406db90 2024-09-16T04:00:56.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.640+0000 7f27117fa640 10 client.?.objecter resend_mon_ops 2024-09-16T04:00:56.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.640+0000 7f2718f7e640 1 -- 172.21.15.42:0/199164292 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f271406db90 msgr2=0x7f27141ceff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-16T04:00:56.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.640+0000 7f2718f7e640 1 -- 172.21.15.42:0/199164292 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f271406db90 msgr2=0x7f27141ceff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:00:56.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.640+0000 7f2718f7e640 1 --2- 172.21.15.42:0/199164292 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f271406db90 0x7f27141ceff0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f270400c7a0 tx=0x7f270400cc60 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:00:56.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.640+0000 7f2718f7e640 1 --2- 172.21.15.42:0/199164292 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f271406db90 0x7f27141ceff0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f270400c7a0 tx=0x7f270400cc60 comp rx=0 tx=0).stop 2024-09-16T04:00:56.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.640+0000 7f27117fa640 10 monclient: ms_handle_reset current mon [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:00:56.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.640+0000 7f27117fa640 10 monclient: _reopen_session rank -1 2024-09-16T04:00:56.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.640+0000 7f27117fa640 1 -- 172.21.15.42:0/199164292 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f271406db90 msgr2=0x7f27141ceff0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:00:56.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.640+0000 7f27117fa640 1 --2- 172.21.15.42:0/199164292 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f271406db90 0x7f27141ceff0 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:00:56.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.640+0000 7f27117fa640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:00:56.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.640+0000 7f27117fa640 1 --2- 172.21.15.42:0/199164292 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f27141340c0 0x7f2700006e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:00:56.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.640+0000 7f27117fa640 10 monclient(hunting): picked mon.noname-c con 0x7f27141340c0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:00:56.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.640+0000 7f27117fa640 1 --2- 172.21.15.42:0/199164292 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f271413d8b0 0x7f27000098f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:00:56.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.640+0000 7f27117fa640 10 monclient(hunting): picked mon.noname-b con 0x7f271413d8b0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:00:56.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.640+0000 7f27117fa640 1 --2- 172.21.15.42:0/199164292 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f270000a350 0x7f270000c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:00:56.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.640+0000 7f27117fa640 10 monclient(hunting): picked mon.noname-a con 0x7f270000a350 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:00:56.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.640+0000 7f27117fa640 10 monclient(hunting): start opening mon connection 2024-09-16T04:00:56.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.640+0000 7f27117fa640 1 -- 172.21.15.42:0/199164292 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f270000d2e0 con 0x7f270000a350 2024-09-16T04:00:56.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.640+0000 7f27117fa640 10 monclient(hunting): start opening mon connection 2024-09-16T04:00:56.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.640+0000 7f27117fa640 1 -- 172.21.15.42:0/199164292 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f270000d490 con 0x7f271413d8b0 2024-09-16T04:00:56.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.640+0000 7f27117fa640 10 monclient(hunting): start opening mon connection 2024-09-16T04:00:56.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.640+0000 7f27117fa640 1 -- 172.21.15.42:0/199164292 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f270000d690 con 0x7f27141340c0 2024-09-16T04:00:56.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.640+0000 7f27117fa640 10 monclient(hunting): _renew_subs 2024-09-16T04:00:56.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.640+0000 7f2718f7e640 1 --2- 172.21.15.42:0/199164292 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f27141340c0 0x7f2700006e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:00:56.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.640+0000 7f271977f640 1 --2- 172.21.15.42:0/199164292 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f270000a350 0x7f270000c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:00:56.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.640+0000 7f271977f640 10 monclient(hunting): get_auth_request con 0x7f270000a350 auth_method 0 2024-09-16T04:00:56.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.640+0000 7f271977f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:00:56.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.640+0000 7f271977f640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:00:56.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.640+0000 7f271977f640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:00:56.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.640+0000 7f2718f7e640 10 monclient(hunting): get_auth_request con 0x7f27141340c0 auth_method 0 2024-09-16T04:00:56.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.640+0000 7f2718f7e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:00:56.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.640+0000 7f2718f7e640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:00:56.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.640+0000 7f2718f7e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:00:56.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.640+0000 7f2718f7e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:00:56.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.640+0000 7f2718f7e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:00:56.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.640+0000 7f2718f7e640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:00:56.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.640+0000 7f271977f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:00:56.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.640+0000 7f271977f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:00:56.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.640+0000 7f271977f640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:00:56.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.640+0000 7f2718f7e640 10 monclient(hunting): handle_auth_done global_id 825297 payload 293 2024-09-16T04:00:56.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.640+0000 7f2713fff640 1 --2- 172.21.15.42:0/199164292 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f271413d8b0 0x7f27000098f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:00:56.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.640+0000 7f2718f7e640 1 -- 172.21.15.42:0/199164292 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f271413d8b0 msgr2=0x7f27000098f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:00:56.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.640+0000 7f2718f7e640 1 --2- 172.21.15.42:0/199164292 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f271413d8b0 0x7f27000098f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:00:56.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.640+0000 7f2718f7e640 1 -- 172.21.15.42:0/199164292 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f270000a350 msgr2=0x7f270000c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:00:56.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.640+0000 7f2718f7e640 1 --2- 172.21.15.42:0/199164292 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f270000a350 0x7f270000c7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:00:56.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.640+0000 7f2718f7e640 10 monclient: _finish_hunting 0 2024-09-16T04:00:56.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.640+0000 7f2718f7e640 1 monclient: found mon.noname-c 2024-09-16T04:00:56.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.640+0000 7f2718f7e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:00:56.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.640+0000 7f2718f7e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T04:00:56.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.640+0000 7f2718f7e640 1 -- 172.21.15.42:0/199164292 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f270000d950 con 0x7f27141340c0 2024-09-16T04:00:56.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.640+0000 7f271977f640 1 --2- 172.21.15.42:0/199164292 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f270000a350 0x7f270000c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:00:56.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.640+0000 7f2718f7e640 1 --2- 172.21.15.42:0/199164292 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f27141340c0 0x7f2700006e70 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f2704015000 tx=0x7f2704001710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:00:56.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.640+0000 7f27117fa640 10 client.?.objecter ms_handle_connect 0x7f27141340c0 2024-09-16T04:00:56.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.640+0000 7f27117fa640 10 client.?.objecter resend_mon_ops 2024-09-16T04:00:56.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.640+0000 7f27117fa640 1 -- 172.21.15.42:0/199164292 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2704023020 con 0x7f27141340c0 2024-09-16T04:00:56.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.640+0000 7f27117fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:00:56.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.640+0000 7f27117fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:00:56.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.640+0000 7f27117fa640 10 monclient: dump: 2024-09-16T04:00:56.669 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:00:56.669 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:00:56.669 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:00:56.669 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:00:56.669 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:00:56.669 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:00:56.669 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:00:56.669 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:00:56.669 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:00:56.669 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:00:56.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.640+0000 7f27117fa640 10 monclient: _finish_auth 0 2024-09-16T04:00:56.670 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.640+0000 7f27117fa640 10 monclient: _check_auth_tickets 2024-09-16T04:00:56.670 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.640+0000 7f27117fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:00:56.670 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.640+0000 7f271b1e2640 5 monclient: authenticate success, global_id 825297 2024-09-16T04:00:56.670 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.640+0000 7f27117fa640 1 -- 172.21.15.42:0/199164292 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2704003de0 con 0x7f27141340c0 2024-09-16T04:00:56.670 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.640+0000 7f27117fa640 10 monclient: handle_config config(0 keys) 2024-09-16T04:00:56.670 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.640+0000 7f271b1e2640 10 monclient: _renew_subs 2024-09-16T04:00:56.670 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.640+0000 7f27117fa640 1 -- 172.21.15.42:0/199164292 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27040039c0 con 0x7f27141340c0 2024-09-16T04:00:56.670 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.640+0000 7f271b1e2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:00:56.670 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.640+0000 7f271b1e2640 1 -- 172.21.15.42:0/199164292 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f271409cd70 con 0x7f27141340c0 2024-09-16T04:00:56.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.640+0000 7f27117fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:00:56.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.640+0000 7f27117fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:00:56.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.640+0000 7f2712ffd640 4 set_mon_vals no callback set 2024-09-16T04:00:56.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.640+0000 7f27117fa640 10 monclient: dump: 2024-09-16T04:00:56.671 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:00:56.671 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:00:56.671 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:00:56.671 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:00:56.671 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:00:56.671 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:00:56.671 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:00:56.671 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:00:56.671 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:00:56.672 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:00:56.672 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.640+0000 7f271b1e2640 10 client.825297.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:00:56.672 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.640+0000 7f271b1e2640 10 monclient: _renew_subs 2024-09-16T04:00:56.672 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.640+0000 7f271b1e2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:00:56.672 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.640+0000 7f271b1e2640 1 -- 172.21.15.42:0/199164292 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f271413ff40 con 0x7f27141340c0 2024-09-16T04:00:56.672 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.640+0000 7f271b1e2640 1 librados: init done 2024-09-16T04:00:56.672 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.640+0000 7f271b1e2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:00:56.672 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.640+0000 7f271b1e2640 1 --2- 172.21.15.42:0/199164292 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f27141cf7b0 0x7f2714139930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:00:56.672 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.640+0000 7f271b1e2640 10 monclient(hunting): start opening mon connection 2024-09-16T04:00:56.672 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.640+0000 7f271b1e2640 1 -- 172.21.15.42:0/199164292 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f27140a37f0 con 0x7f27141cf7b0 2024-09-16T04:00:56.672 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.640+0000 7f2713fff640 1 --2- 172.21.15.42:0/199164292 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f27141cf7b0 0x7f2714139930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:00:56.672 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.640+0000 7f2713fff640 10 monclient: get_auth_request con 0x7f27141cf7b0 auth_method 0 2024-09-16T04:00:56.672 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.640+0000 7f2713fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:00:56.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.640+0000 7f2713fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:00:56.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.640+0000 7f2713fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:00:56.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.640+0000 7f2713fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:00:56.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.640+0000 7f2713fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:00:56.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.640+0000 7f2713fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:00:56.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.640+0000 7f2713fff640 10 monclient(hunting): handle_auth_done global_id 825300 payload 824 2024-09-16T04:00:56.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.640+0000 7f2713fff640 1 -- 172.21.15.42:0/199164292 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f27141cfb90 con 0x7f27141cf7b0 2024-09-16T04:00:56.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.644+0000 7f27117fa640 10 client.825297.objecter ms_handle_connect 0x7f27141cf7b0 2024-09-16T04:00:56.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.644+0000 7f27117fa640 10 client.825297.objecter resend_mon_ops 2024-09-16T04:00:56.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.644+0000 7f2713fff640 1 --2- 172.21.15.42:0/199164292 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f27141cf7b0 0x7f2714139930 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f271413cf60 tx=0x7f2708001750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:00:56.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.644+0000 7f27117fa640 1 -- 172.21.15.42:0/199164292 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f2704026070 con 0x7f27141340c0 2024-09-16T04:00:56.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.644+0000 7f27117fa640 1 --2- 172.21.15.42:0/199164292 >> v2:172.21.15.154:6812/254845342 conn(0x7f271406db90 0x7f2700041e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:00:56.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.644+0000 7f27117fa640 1 -- 172.21.15.42:0/199164292 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(569..569 src has 451..569) ==== 6216+0+0 (secure 0 0 0) 0x7f270404a2d0 con 0x7f27141340c0 2024-09-16T04:00:56.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.644+0000 7f27117fa640 10 client.825297.objecter ms_dispatch 0x7f2714000ba0 osd_map(569..569 src has 451..569) 2024-09-16T04:00:56.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.644+0000 7f27117fa640 3 client.825297.objecter handle_osd_map got epochs [569,569] > 0 2024-09-16T04:00:56.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.644+0000 7f27117fa640 3 client.825297.objecter handle_osd_map decoding full epoch 569 2024-09-16T04:00:56.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.644+0000 7f27117fa640 20 client.825297.objecter dump_active .. 0 homeless 2024-09-16T04:00:56.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.644+0000 7f27117fa640 1 -- 172.21.15.42:0/199164292 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2708019070 con 0x7f27141cf7b0 2024-09-16T04:00:56.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.644+0000 7f27117fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:00:56.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.644+0000 7f27117fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:00:56.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.644+0000 7f27117fa640 10 monclient: dump: 2024-09-16T04:00:56.674 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:00:56.674 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:00:56.674 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:00:56.674 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:00:56.674 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:00:56.674 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:00:56.674 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:00:56.675 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:00:56.675 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:00:56.675 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:00:56.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.644+0000 7f27117fa640 1 -- 172.21.15.42:0/199164292 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f2708013630 con 0x7f27141cf7b0 2024-09-16T04:00:56.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.644+0000 7f27117fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:00:56.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.644+0000 7f27117fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:00:56.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.644+0000 7f27117fa640 1 -- 172.21.15.42:0/199164292 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f27141cf7b0 msgr2=0x7f2714139930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:00:56.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.644+0000 7f27117fa640 1 --2- 172.21.15.42:0/199164292 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f27141cf7b0 0x7f2714139930 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f271413cf60 tx=0x7f2708001750 comp rx=0 tx=0).stop 2024-09-16T04:00:56.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.644+0000 7f27117fa640 1 -- 172.21.15.42:0/199164292 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f27141cf7b0 msgr2=0x7f2714139930 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:00:56.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.644+0000 7f27117fa640 1 --2- 172.21.15.42:0/199164292 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f27141cf7b0 0x7f2714139930 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:00:56.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.644+0000 7f271977f640 1 --2- 172.21.15.42:0/199164292 >> v2:172.21.15.154:6812/254845342 conn(0x7f271406db90 0x7f2700041e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:00:56.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.644+0000 7f271977f640 10 monclient: get_auth_request con 0x7f271406db90 auth_method 0 2024-09-16T04:00:56.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.644+0000 7f271977f640 1 --2- 172.21.15.42:0/199164292 >> v2:172.21.15.154:6812/254845342 conn(0x7f271406db90 0x7f2700041e50 secure :-1 s=READY pgs=3751 cs=0 l=1 rev1=1 crypto rx=0x7f270c0ae000 tx=0x7f270c0a7000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:00:56.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.644+0000 7f27117fa640 10 client.825297.objecter ms_handle_connect 0x7f271406db90 2024-09-16T04:00:56.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.648+0000 7f271b1e2640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:00:56.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.648+0000 7f271b1e2640 1 --2- 172.21.15.42:0/199164292 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2714139e70 0x7f271413a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:00:56.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.648+0000 7f271b1e2640 10 monclient(hunting): start opening mon connection 2024-09-16T04:00:56.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.648+0000 7f271b1e2640 1 -- 172.21.15.42:0/199164292 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f271413be80 con 0x7f2714139e70 2024-09-16T04:00:56.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.648+0000 7f2713fff640 1 --2- 172.21.15.42:0/199164292 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2714139e70 0x7f271413a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:00:56.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.648+0000 7f2713fff640 10 monclient: get_auth_request con 0x7f2714139e70 auth_method 0 2024-09-16T04:00:56.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.648+0000 7f2713fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:00:56.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.648+0000 7f2713fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:00:56.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.648+0000 7f2713fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:00:56.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.648+0000 7f2713fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:00:56.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.648+0000 7f2713fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:00:56.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.648+0000 7f2713fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:00:56.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.648+0000 7f2713fff640 10 monclient(hunting): handle_auth_done global_id 825303 payload 824 2024-09-16T04:00:56.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.648+0000 7f2713fff640 1 -- 172.21.15.42:0/199164292 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f2714068e40 con 0x7f2714139e70 2024-09-16T04:00:56.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.648+0000 7f2713fff640 1 --2- 172.21.15.42:0/199164292 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2714139e70 0x7f271413a280 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f271409cf80 tx=0x7f2708007d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:00:56.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.648+0000 7f27117fa640 10 client.825297.objecter ms_handle_connect 0x7f2714139e70 2024-09-16T04:00:56.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.648+0000 7f27117fa640 10 client.825297.objecter resend_mon_ops 2024-09-16T04:00:56.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.648+0000 7f27117fa640 1 -- 172.21.15.42:0/199164292 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2708019050 con 0x7f2714139e70 2024-09-16T04:00:56.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.648+0000 7f27117fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:00:56.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.648+0000 7f27117fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:00:56.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.648+0000 7f27117fa640 10 monclient: dump: 2024-09-16T04:00:56.679 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:00:56.679 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:00:56.679 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:00:56.679 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:00:56.679 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:00:56.679 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:00:56.679 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:00:56.679 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:00:56.679 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:00:56.680 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:00:56.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.648+0000 7f27117fa640 1 -- 172.21.15.42:0/199164292 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4715 (secure 0 0 0) 0x7f270801a040 con 0x7f2714139e70 2024-09-16T04:00:56.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.648+0000 7f27117fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:00:56.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.648+0000 7f27117fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:00:56.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.648+0000 7f27117fa640 1 -- 172.21.15.42:0/199164292 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2714139e70 msgr2=0x7f271413a280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:00:56.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.648+0000 7f27117fa640 1 --2- 172.21.15.42:0/199164292 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2714139e70 0x7f271413a280 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f271409cf80 tx=0x7f2708007d20 comp rx=0 tx=0).stop 2024-09-16T04:00:56.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.648+0000 7f27117fa640 1 -- 172.21.15.42:0/199164292 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2714139e70 msgr2=0x7f271413a280 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:00:56.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.648+0000 7f27117fa640 1 --2- 172.21.15.42:0/199164292 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2714139e70 0x7f271413a280 secure :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f271409cf80 tx=0x7f2708007d20 comp rx=0 tx=0).stop 2024-09-16T04:00:56.680 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T04:00:56.681 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:00:56.681 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:00:56.681 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "leader", 2024-09-16T04:00:56.681 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 168, 2024-09-16T04:00:56.681 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 546503, 2024-09-16T04:00:56.681 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T04:00:56.681 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T04:00:56.681 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T04:00:56.681 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T04:00:56.681 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:00:56.681 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 5, 2024-09-16T04:00:56.681 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:00:56.681 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T04:00:56.681 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T04:00:56.682 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:00:56.682 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:00:56.682 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:00:56.682 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:00:56.682 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:00:56.682 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:00:56.682 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:00:56.682 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:00:56.682 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:00:56.682 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:00:56.682 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:00:56.682 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:00:56.682 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:00:56.682 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T04:00:56.683 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:00:56.683 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:00:56.683 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:00:56.683 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:00:56.683 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:00:56.683 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:00:56.683 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:00:56.683 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:00:56.683 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:00:56.683 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:00:56.683 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:00:56.683 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:00:56.683 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:00:56.683 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T04:00:56.684 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T04:00:56.684 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T04:00:56.684 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T04:00:56.684 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T04:00:56.684 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T04:00:56.684 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:00:56.684 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:00:56.684 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T04:00:56.684 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T04:00:56.684 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T04:00:56.685 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T04:00:56.685 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T04:00:56.685 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T04:00:56.685 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T04:00:56.685 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:00:56.685 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T04:00:56.685 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:00:56.685 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:00:56.685 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:00:56.685 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:00:56.685 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:00:56.685 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:00:56.685 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:00:56.685 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:00:56.686 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:00:56.686 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:00:56.686 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:00:56.686 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:00:56.686 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T04:00:56.686 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:00:56.686 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T04:00:56.686 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:00:56.686 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:00:56.686 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:00:56.686 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:00:56.686 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:00:56.686 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:00:56.687 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:00:56.687 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T04:00:56.687 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:00:56.687 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:00:56.687 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:00:56.687 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:00:56.687 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T04:00:56.687 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:00:56.687 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:00:56.687 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:00:56.687 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:00:56.687 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T04:00:56.687 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T04:00:56.688 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:00:56.688 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:00:56.688 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:00:56.688 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:00:56.688 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:00:56.688 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:00:56.688 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:00:56.688 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:00:56.688 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:00:56.688 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:00:56.688 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:00:56.688 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T04:00:56.688 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:00:56.689 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:00:56.689 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:00:56.689 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:00:56.689 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T04:00:56.689 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:00:56.689 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:00:56.689 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:00:56.689 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:00:56.689 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T04:00:56.689 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T04:00:56.689 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:00:56.689 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:00:56.689 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:00:56.690 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:00:56.690 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:00:56.690 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:00:56.690 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:00:56.690 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:00:56.690 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:00:56.690 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:00:56.690 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:00:56.690 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T04:00:56.690 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:00:56.690 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:00:56.690 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:00:56.690 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:00:56.691 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T04:00:56.691 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:00:56.691 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:00:56.691 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:00:56.691 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:00:56.691 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T04:00:56.691 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T04:00:56.691 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:00:56.691 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:00:56.691 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:00:56.691 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:00:56.691 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:00:56.691 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:00:56.691 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T04:00:56.692 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T04:00:56.692 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:00:56.692 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:00:56.692 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:00:56.692 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:00:56.692 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:00:56.692 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:00:56.692 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T04:00:56.692 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:00:56.692 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:00:56.692 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:00:56.692 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 5 2024-09-16T04:00:56.692 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:00:56.693 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:00:56.693 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T04:00:56.693 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:00:56.693 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:00:56.693 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:00:56.693 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 6 2024-09-16T04:00:56.693 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:00:56.693 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:00:56.693 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:00:56.693 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T04:00:56.693 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T04:00:56.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.652+0000 7f271b1e2640 10 librados: watch_flush enter 2024-09-16T04:00:56.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.652+0000 7f271b1e2640 10 librados: watch_flush exit 2024-09-16T04:00:56.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.652+0000 7f271b1e2640 20 client.825297.objecter shutdown clearing up homeless session... 2024-09-16T04:00:56.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.652+0000 7f271b1e2640 10 client.825297.objecter successfully canceled tick 2024-09-16T04:00:56.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.652+0000 7f271b1e2640 1 -- 172.21.15.42:0/199164292 >> v2:172.21.15.154:6812/254845342 conn(0x7f271406db90 msgr2=0x7f2700041e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:00:56.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.652+0000 7f271b1e2640 1 --2- 172.21.15.42:0/199164292 >> v2:172.21.15.154:6812/254845342 conn(0x7f271406db90 0x7f2700041e50 secure :-1 s=READY pgs=3751 cs=0 l=1 rev1=1 crypto rx=0x7f270c0ae000 tx=0x7f270c0a7000 comp rx=0 tx=0).stop 2024-09-16T04:00:56.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.652+0000 7f271b1e2640 10 monclient: shutdown 2024-09-16T04:00:56.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.652+0000 7f271b1e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:00:56.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.652+0000 7f271b1e2640 1 -- 172.21.15.42:0/199164292 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f27141340c0 msgr2=0x7f2700006e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:00:56.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.652+0000 7f271b1e2640 1 --2- 172.21.15.42:0/199164292 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f27141340c0 0x7f2700006e70 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f2704015000 tx=0x7f2704001710 comp rx=0 tx=0).stop 2024-09-16T04:00:56.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.652+0000 7f271977f640 1 -- 172.21.15.42:0/199164292 reap_dead start 2024-09-16T04:00:56.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.652+0000 7f271977f640 1 -- 172.21.15.42:0/199164292 reap_dead start 2024-09-16T04:00:56.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.652+0000 7f271b1e2640 1 -- 172.21.15.42:0/199164292 shutdown_connections 2024-09-16T04:00:56.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.652+0000 7f271b1e2640 1 -- 172.21.15.42:0/199164292 >> 172.21.15.42:0/199164292 conn(0x7f27140a9fb0 msgr2=0x7f2714131790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:00:56.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.652+0000 7f271b1e2640 1 -- 172.21.15.42:0/199164292 shutdown_connections 2024-09-16T04:00:56.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.652+0000 7f271b1e2640 1 -- 172.21.15.42:0/199164292 wait complete. 2024-09-16T04:00:56.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.652+0000 7f271b1e2640 1 librados: shutdown 2024-09-16T04:00:56.696 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:00:56.696 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T04:00:56.697 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T04:00:56.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.808+0000 7fb0be1e9640 10 monclient: get_monmap_and_config 2024-09-16T04:00:56.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.808+0000 7fb0be1e9640 10 monclient: build_initial_monmap 2024-09-16T04:00:56.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.808+0000 7fb0be1e9640 10 monclient: monmap: 2024-09-16T04:00:56.814 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:00:56.814 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:00:56.814 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:00:56.812521+0000 2024-09-16T04:00:56.814 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:00:56.812521+0000 2024-09-16T04:00:56.814 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:00:56.814 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:00:56.814 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:00:56.814 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:00:56.814 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:00:56.815 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:00:56.815 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:00:56.815 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:00:56.818 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.808+0000 7fb0be1e9640 1 Processor -- start 2024-09-16T04:00:56.819 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.808+0000 7fb0be1e9640 1 -- start start 2024-09-16T04:00:56.819 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.808+0000 7fb0be1e9640 10 monclient: init 2024-09-16T04:00:56.819 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.808+0000 7fb0be1e9640 10 monclient: _reopen_session rank -1 2024-09-16T04:00:56.819 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.808+0000 7fb0be1e9640 10 monclient: _add_conns ranks=[2,3,4,1,0] 2024-09-16T04:00:56.819 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.808+0000 7fb0be1e9640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb0b80a1330 0x7fb0b80a1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:00:56.819 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.808+0000 7fb0be1e9640 10 monclient(hunting): picked mon.noname-b con 0x7fb0b80a1330 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:00:56.819 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.808+0000 7fb0be1e9640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb0b806dac0 0x7fb0b80a1c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:00:56.819 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.808+0000 7fb0be1e9640 10 monclient(hunting): picked mon.noname-a con 0x7fb0b806dac0 addr v2:172.21.15.42:3300/0 2024-09-16T04:00:56.819 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.808+0000 7fb0be1e9640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fb0b8135100 0x7fb0b8135550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:00:56.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.808+0000 7fb0be1e9640 10 monclient(hunting): picked mon.noname-c con 0x7fb0b8135100 addr v2:172.21.15.154:3300/0 2024-09-16T04:00:56.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.808+0000 7fb0be1e9640 10 monclient(hunting): start opening mon connection 2024-09-16T04:00:56.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.808+0000 7fb0be1e9640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7fb0b809c7d0 con 0x7fb0b806dac0 2024-09-16T04:00:56.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.808+0000 7fb0be1e9640 10 monclient(hunting): start opening mon connection 2024-09-16T04:00:56.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.808+0000 7fb0be1e9640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fb0b809c090 con 0x7fb0b8135100 2024-09-16T04:00:56.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.808+0000 7fb0be1e9640 10 monclient(hunting): start opening mon connection 2024-09-16T04:00:56.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.808+0000 7fb0be1e9640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fb0b809f610 con 0x7fb0b80a1330 2024-09-16T04:00:56.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.808+0000 7fb0be1e9640 10 monclient(hunting): _renew_subs 2024-09-16T04:00:56.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.808+0000 7fb0be1e9640 10 monclient(hunting): authenticate will time out at 2103.781910s 2024-09-16T04:00:56.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.808+0000 7fb0b77fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb0b80a1330 0x7fb0b80a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:00:56.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.808+0000 7fb0b6ffd640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb0b806dac0 0x7fb0b80a1c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:00:56.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.808+0000 7fb0b77fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb0b80a1330 0x7fb0b80a1710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:40780/0 (socket says 172.21.15.42:40780) 2024-09-16T04:00:56.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.808+0000 7fb0b77fe640 1 -- 172.21.15.42:0/418126294 learned_addr learned my addr 172.21.15.42:0/418126294 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:00:56.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.808+0000 7fb0b77fe640 10 monclient(hunting): get_auth_request con 0x7fb0b80a1330 auth_method 0 2024-09-16T04:00:56.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.808+0000 7fb0b77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:00:56.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.808+0000 7fb0b77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:00:56.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.808+0000 7fb0b77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:00:56.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.808+0000 7fb0b6ffd640 10 monclient(hunting): get_auth_request con 0x7fb0b806dac0 auth_method 0 2024-09-16T04:00:56.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.808+0000 7fb0b6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:00:56.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.808+0000 7fb0b6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:00:56.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.808+0000 7fb0b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:00:56.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.808+0000 7fb0b77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:00:56.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.808+0000 7fb0b77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:00:56.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.808+0000 7fb0b77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:00:56.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.808+0000 7fb0b6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:00:56.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.808+0000 7fb0b6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:00:56.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.808+0000 7fb0b6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:00:56.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.808+0000 7fb0b7fff640 1 --2- 172.21.15.42:0/418126294 >> v2:172.21.15.154:3300/0 conn(0x7fb0b8135100 0x7fb0b8135550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:00:56.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.808+0000 7fb0b7fff640 10 monclient(hunting): get_auth_request con 0x7fb0b8135100 auth_method 0 2024-09-16T04:00:56.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.808+0000 7fb0b7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:00:56.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.808+0000 7fb0b7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:00:56.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.808+0000 7fb0b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:00:56.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.808+0000 7fb0b77fe640 10 monclient(hunting): handle_auth_done global_id 944138 payload 274 2024-09-16T04:00:56.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.808+0000 7fb0b77fe640 1 -- 172.21.15.42:0/418126294 >> v2:172.21.15.154:3300/0 conn(0x7fb0b8135100 msgr2=0x7fb0b8135550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:00:56.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.808+0000 7fb0b77fe640 1 --2- 172.21.15.42:0/418126294 >> v2:172.21.15.154:3300/0 conn(0x7fb0b8135100 0x7fb0b8135550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:00:56.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.808+0000 7fb0b77fe640 1 -- 172.21.15.42:0/418126294 >> v2:172.21.15.42:3300/0 conn(0x7fb0b806dac0 msgr2=0x7fb0b80a1c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:00:56.823 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.808+0000 7fb0b77fe640 1 --2- 172.21.15.42:0/418126294 >> v2:172.21.15.42:3300/0 conn(0x7fb0b806dac0 0x7fb0b80a1c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:00:56.823 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.808+0000 7fb0b77fe640 10 monclient: _finish_hunting 0 2024-09-16T04:00:56.823 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.808+0000 7fb0b77fe640 1 monclient: found mon.noname-b 2024-09-16T04:00:56.823 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.808+0000 7fb0b77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:00:56.823 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.808+0000 7fb0b77fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T04:00:56.823 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.808+0000 7fb0b77fe640 1 -- 172.21.15.42:0/418126294 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0b809f8f0 con 0x7fb0b80a1330 2024-09-16T04:00:56.823 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.808+0000 7fb0b6ffd640 1 --2- 172.21.15.42:0/418126294 >> v2:172.21.15.42:3300/0 conn(0x7fb0b806dac0 0x7fb0b80a1c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:00:56.823 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.812+0000 7fb0b77fe640 1 --2- 172.21.15.42:0/418126294 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb0b80a1330 0x7fb0b80a1710 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fb0b809bb00 tx=0x7fb0a801c440 comp rx=0 tx=0).ready entity=mon.2 client_cookie=66a95a21d614e3bb server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:00:56.823 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.812+0000 7fb0b67fc640 1 -- 172.21.15.42:0/418126294 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0a80b1070 con 0x7fb0b80a1330 2024-09-16T04:00:56.823 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.812+0000 7fb0b67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:00:56.823 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.812+0000 7fb0b67fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:00:56.823 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.812+0000 7fb0b67fc640 10 monclient: dump: 2024-09-16T04:00:56.823 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:00:56.823 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:00:56.824 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:00:56.824 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:00:56.824 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:00:56.824 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:00:56.824 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:00:56.824 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:00:56.824 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:00:56.824 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:00:56.824 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.812+0000 7fb0b67fc640 10 monclient: _finish_auth 0 2024-09-16T04:00:56.824 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.812+0000 7fb0b67fc640 10 monclient: _check_auth_tickets 2024-09-16T04:00:56.824 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.812+0000 7fb0b67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:00:56.824 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.812+0000 7fb0b67fc640 1 -- 172.21.15.42:0/418126294 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb0a80ac620 con 0x7fb0b80a1330 2024-09-16T04:00:56.825 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.812+0000 7fb0be1e9640 5 monclient: authenticate success, global_id 944138 2024-09-16T04:00:56.825 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.812+0000 7fb0be1e9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:00:56.825 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.812+0000 7fb0b67fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:00:56.825 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.812+0000 7fb0be1e9640 10 monclient: get_monmap_and_config success 2024-09-16T04:00:56.825 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.812+0000 7fb0b67fc640 1 -- 172.21.15.42:0/418126294 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0a80ac9f0 con 0x7fb0b80a1330 2024-09-16T04:00:56.825 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.812+0000 7fb0b67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:00:56.825 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.812+0000 7fb0be1e9640 4 set_mon_vals no callback set 2024-09-16T04:00:56.825 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.812+0000 7fb0b67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:00:56.825 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.812+0000 7fb0b67fc640 10 monclient: dump: 2024-09-16T04:00:56.825 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:00:56.825 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:00:56.825 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:00:56.825 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:00:56.826 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:00:56.826 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:00:56.826 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:00:56.826 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:00:56.826 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:00:56.826 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:00:56.826 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.812+0000 7fb0be1e9640 10 monclient: shutdown 2024-09-16T04:00:56.826 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.812+0000 7fb0be1e9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:00:56.826 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.812+0000 7fb0be1e9640 1 -- 172.21.15.42:0/418126294 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb0b80a1330 msgr2=0x7fb0b80a1710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:00:56.826 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.812+0000 7fb0be1e9640 1 --2- 172.21.15.42:0/418126294 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb0b80a1330 0x7fb0b80a1710 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fb0b809bb00 tx=0x7fb0a801c440 comp rx=0 tx=0).stop 2024-09-16T04:00:56.826 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.812+0000 7fb0be1e9640 1 -- 172.21.15.42:0/418126294 shutdown_connections 2024-09-16T04:00:56.826 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.812+0000 7fb0be1e9640 1 --2- 172.21.15.42:0/418126294 >> v2:172.21.15.154:3300/0 conn(0x7fb0b8135100 0x7fb0b8135550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:00:56.826 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.812+0000 7fb0be1e9640 1 --2- 172.21.15.42:0/418126294 >> v2:172.21.15.42:3300/0 conn(0x7fb0b806dac0 0x7fb0b80a1c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:00:56.826 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.812+0000 7fb0be1e9640 1 --2- 172.21.15.42:0/418126294 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb0b80a1330 0x7fb0b80a1710 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:00:56.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.812+0000 7fb0be1e9640 1 -- 172.21.15.42:0/418126294 >> 172.21.15.42:0/418126294 conn(0x7fb0b80a2420 msgr2=0x7fb0b80a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:00:56.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.812+0000 7fb0be1e9640 1 -- 172.21.15.42:0/418126294 shutdown_connections 2024-09-16T04:00:56.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.812+0000 7fb0be1e9640 1 -- 172.21.15.42:0/418126294 wait complete. 2024-09-16T04:00:56.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.812+0000 7fb0be1e9640 10 monclient: build_initial_monmap 2024-09-16T04:00:56.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.812+0000 7fb0be1e9640 10 monclient: monmap: 2024-09-16T04:00:56.834 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:00:56.834 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:00:56.834 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:00:56.834 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:00:56.834 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:00:56.834 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:00:56.834 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:00:56.834 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:00:56.834 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:00:56.835 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:00:56.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.812+0000 7fb0be1e9640 1 librados: starting msgr at 2024-09-16T04:00:56.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.812+0000 7fb0be1e9640 1 librados: starting objecter 2024-09-16T04:00:56.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.812+0000 7fb0be1e9640 1 Processor -- start 2024-09-16T04:00:56.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.812+0000 7fb0be1e9640 1 -- start start 2024-09-16T04:00:56.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.812+0000 7fb0be1e9640 1 librados: setting wanted keys 2024-09-16T04:00:56.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.812+0000 7fb0be1e9640 1 librados: calling monclient init 2024-09-16T04:00:56.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.812+0000 7fb0be1e9640 10 monclient: init 2024-09-16T04:00:56.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.812+0000 7fb0be1e9640 10 monclient: _reopen_session rank -1 2024-09-16T04:00:56.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.812+0000 7fb0be1e9640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:00:56.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.812+0000 7fb0be1e9640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb0b80a1330 0x7fb0b81cef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:00:56.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.812+0000 7fb0be1e9640 10 monclient(hunting): picked mon.noname-c con 0x7fb0b80a1330 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:00:56.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.812+0000 7fb0be1e9640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb0b8135100 0x7fb0b8134590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:00:56.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.812+0000 7fb0be1e9640 10 monclient(hunting): picked mon.noname-b con 0x7fb0b8135100 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:00:56.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.812+0000 7fb0be1e9640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb0b8134ad0 0x7fb0b809cb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:00:56.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.812+0000 7fb0be1e9640 10 monclient(hunting): picked mon.noname-a con 0x7fb0b8134ad0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:00:56.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.812+0000 7fb0be1e9640 10 monclient(hunting): start opening mon connection 2024-09-16T04:00:56.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.812+0000 7fb0be1e9640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fb0b809eef0 con 0x7fb0b8134ad0 2024-09-16T04:00:56.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.812+0000 7fb0be1e9640 10 monclient(hunting): start opening mon connection 2024-09-16T04:00:56.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.812+0000 7fb0be1e9640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fb0b809ed70 con 0x7fb0b8135100 2024-09-16T04:00:56.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.812+0000 7fb0be1e9640 10 monclient(hunting): start opening mon connection 2024-09-16T04:00:56.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.812+0000 7fb0be1e9640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fb0b813eb70 con 0x7fb0b80a1330 2024-09-16T04:00:56.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.812+0000 7fb0be1e9640 10 monclient(hunting): _renew_subs 2024-09-16T04:00:56.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.812+0000 7fb0be1e9640 10 monclient(hunting): authenticate will time out at 2103.786903s 2024-09-16T04:00:56.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.812+0000 7fb0b77fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb0b80a1330 0x7fb0b81cef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:00:56.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.812+0000 7fb0b7fff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb0b8134ad0 0x7fb0b809cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:00:56.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.812+0000 7fb0b77fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb0b80a1330 0x7fb0b81cef30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:40790/0 (socket says 172.21.15.42:40790) 2024-09-16T04:00:56.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.812+0000 7fb0b77fe640 1 -- 172.21.15.42:0/3880176404 learned_addr learned my addr 172.21.15.42:0/3880176404 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:00:56.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.812+0000 7fb0b7fff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb0b8134ad0 0x7fb0b809cb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:44158/0 (socket says 172.21.15.42:44158) 2024-09-16T04:00:56.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.812+0000 7fb0b77fe640 10 monclient(hunting): get_auth_request con 0x7fb0b80a1330 auth_method 0 2024-09-16T04:00:56.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.812+0000 7fb0b77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:00:56.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.812+0000 7fb0b77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:00:56.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.812+0000 7fb0b77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:00:56.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.812+0000 7fb0b7fff640 10 monclient(hunting): get_auth_request con 0x7fb0b8134ad0 auth_method 0 2024-09-16T04:00:56.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.812+0000 7fb0b7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:00:56.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.812+0000 7fb0b7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:00:56.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.812+0000 7fb0b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:00:56.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.812+0000 7fb0b77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:00:56.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.812+0000 7fb0b77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:00:56.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.812+0000 7fb0b77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:00:56.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.812+0000 7fb0b7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:00:56.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.812+0000 7fb0b7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:00:56.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.812+0000 7fb0b7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:00:56.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.812+0000 7fb0b6ffd640 1 --2- 172.21.15.42:0/3880176404 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb0b8135100 0x7fb0b8134590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:00:56.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.812+0000 7fb0b77fe640 10 monclient(hunting): handle_auth_done global_id 944141 payload 824 2024-09-16T04:00:56.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.812+0000 7fb0b77fe640 1 -- 172.21.15.42:0/3880176404 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb0b8135100 msgr2=0x7fb0b8134590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:00:56.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.812+0000 7fb0b77fe640 1 --2- 172.21.15.42:0/3880176404 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb0b8135100 0x7fb0b8134590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:00:56.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.812+0000 7fb0b77fe640 1 -- 172.21.15.42:0/3880176404 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb0b8134ad0 msgr2=0x7fb0b809cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:00:56.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.812+0000 7fb0b77fe640 1 --2- 172.21.15.42:0/3880176404 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb0b8134ad0 0x7fb0b809cb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:00:56.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.812+0000 7fb0b77fe640 10 monclient: _finish_hunting 0 2024-09-16T04:00:56.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.812+0000 7fb0b77fe640 1 monclient: found mon.noname-c 2024-09-16T04:00:56.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.812+0000 7fb0b77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:00:56.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.812+0000 7fb0b77fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T04:00:56.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.812+0000 7fb0b77fe640 1 -- 172.21.15.42:0/3880176404 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0b8121c70 con 0x7fb0b80a1330 2024-09-16T04:00:56.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.812+0000 7fb0b6ffd640 10 monclient: get_auth_request con 0x7fb0b8135100 auth_method 0 2024-09-16T04:00:56.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.812+0000 7fb0b7fff640 1 --2- 172.21.15.42:0/3880176404 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb0b8134ad0 0x7fb0b809cb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:00:56.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.812+0000 7fb0b6ffd640 1 --2- 172.21.15.42:0/3880176404 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb0b8135100 0x7fb0b8134590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T04:00:56.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.812+0000 7fb0b77fe640 1 --2- 172.21.15.42:0/3880176404 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb0b80a1330 0x7fb0b81cef30 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fb0a80af040 tx=0x7fb0a80191e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:00:56.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.812+0000 7fb09ffff640 10 client.?.objecter ms_handle_connect 0x7fb0b80a1330 2024-09-16T04:00:56.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.812+0000 7fb09ffff640 10 client.?.objecter resend_mon_ops 2024-09-16T04:00:56.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.812+0000 7fb09ffff640 1 -- 172.21.15.42:0/3880176404 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0a80b1040 con 0x7fb0b80a1330 2024-09-16T04:00:56.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.812+0000 7fb09ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:00:56.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.812+0000 7fb09ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:00:56.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.812+0000 7fb09ffff640 10 monclient: dump: 2024-09-16T04:00:56.840 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:00:56.840 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:00:56.840 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:00:56.840 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:00:56.841 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:00:56.841 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:00:56.841 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:00:56.841 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:00:56.841 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:00:56.841 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:00:56.841 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.812+0000 7fb09ffff640 10 monclient: _finish_auth 0 2024-09-16T04:00:56.841 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.812+0000 7fb09ffff640 10 monclient: _check_auth_tickets 2024-09-16T04:00:56.841 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.812+0000 7fb09ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:00:56.841 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.812+0000 7fb09ffff640 1 -- 172.21.15.42:0/3880176404 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb0a80b5420 con 0x7fb0b80a1330 2024-09-16T04:00:56.841 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.812+0000 7fb0be1e9640 5 monclient: authenticate success, global_id 944141 2024-09-16T04:00:56.841 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.812+0000 7fb0be1e9640 10 monclient: _renew_subs 2024-09-16T04:00:56.841 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.812+0000 7fb0be1e9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:00:56.841 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.812+0000 7fb0be1e9640 1 -- 172.21.15.42:0/3880176404 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0b81cf470 con 0x7fb0b80a1330 2024-09-16T04:00:56.842 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.812+0000 7fb09ffff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:00:56.842 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.812+0000 7fb09ffff640 1 -- 172.21.15.42:0/3880176404 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0a80b57f0 con 0x7fb0b80a1330 2024-09-16T04:00:56.842 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.812+0000 7fb09ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:00:56.842 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.812+0000 7fb09ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:00:56.842 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.812+0000 7fb09ffff640 10 monclient: dump: 2024-09-16T04:00:56.842 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:00:56.842 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:00:56.842 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:00:56.842 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:00:56.842 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:00:56.842 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:00:56.842 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:00:56.842 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:00:56.843 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:00:56.843 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:00:56.843 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.812+0000 7fb0be1e9640 10 client.944141.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:00:56.843 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.812+0000 7fb0be1e9640 10 monclient: _renew_subs 2024-09-16T04:00:56.843 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.812+0000 7fb0be1e9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:00:56.843 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.812+0000 7fb0be1e9640 1 -- 172.21.15.42:0/3880176404 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0b80a4d00 con 0x7fb0b80a1330 2024-09-16T04:00:56.843 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.812+0000 7fb0be1e9640 1 librados: init done 2024-09-16T04:00:56.843 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.812+0000 7fb0b57fa640 4 set_mon_vals no callback set 2024-09-16T04:00:56.843 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.816+0000 7fb0be1e9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:00:56.843 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.816+0000 7fb0be1e9640 1 --2- 172.21.15.42:0/3880176404 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb0b806dac0 0x7fb0b809d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:00:56.843 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.816+0000 7fb0be1e9640 10 monclient(hunting): start opening mon connection 2024-09-16T04:00:56.843 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.816+0000 7fb0be1e9640 1 -- 172.21.15.42:0/3880176404 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fb0b8073a30 con 0x7fb0b806dac0 2024-09-16T04:00:56.843 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.816+0000 7fb0b6ffd640 1 --2- 172.21.15.42:0/3880176404 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb0b806dac0 0x7fb0b809d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:00:56.843 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.816+0000 7fb0b6ffd640 10 monclient: get_auth_request con 0x7fb0b806dac0 auth_method 0 2024-09-16T04:00:56.844 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.816+0000 7fb0b6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:00:56.844 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.816+0000 7fb0b6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:00:56.844 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.816+0000 7fb0b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:00:56.844 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.816+0000 7fb09ffff640 1 -- 172.21.15.42:0/3880176404 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fb0a80b5990 con 0x7fb0b80a1330 2024-09-16T04:00:56.844 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.816+0000 7fb0b6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:00:56.844 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.816+0000 7fb0b6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:00:56.844 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.816+0000 7fb0b6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:00:56.844 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.816+0000 7fb09ffff640 1 --2- 172.21.15.42:0/3880176404 >> v2:172.21.15.154:6812/254845342 conn(0x7fb0a4037840 0x7fb0a4039d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:00:56.844 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.816+0000 7fb0b6ffd640 10 monclient(hunting): handle_auth_done global_id 944144 payload 824 2024-09-16T04:00:56.844 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.816+0000 7fb0b6ffd640 1 -- 172.21.15.42:0/3880176404 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb0b8135550 con 0x7fb0b806dac0 2024-09-16T04:00:56.844 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.816+0000 7fb09ffff640 1 -- 172.21.15.42:0/3880176404 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(569..569 src has 451..569) ==== 6216+0+0 (secure 0 0 0) 0x7fb0a80e2ea0 con 0x7fb0b80a1330 2024-09-16T04:00:56.844 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.816+0000 7fb09ffff640 10 client.944141.objecter ms_dispatch 0x7fb0b8000ba0 osd_map(569..569 src has 451..569) 2024-09-16T04:00:56.844 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.816+0000 7fb09ffff640 3 client.944141.objecter handle_osd_map got epochs [569,569] > 0 2024-09-16T04:00:56.845 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.816+0000 7fb09ffff640 3 client.944141.objecter handle_osd_map decoding full epoch 569 2024-09-16T04:00:56.845 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.816+0000 7fb09ffff640 20 client.944141.objecter dump_active .. 0 homeless 2024-09-16T04:00:56.845 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.816+0000 7fb0b6ffd640 1 --2- 172.21.15.42:0/3880176404 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb0b806dac0 0x7fb0b809d0a0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fb0b809e130 tx=0x7fb0ac010c20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:00:56.845 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.816+0000 7fb09ffff640 10 client.944141.objecter ms_handle_connect 0x7fb0b806dac0 2024-09-16T04:00:56.845 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.816+0000 7fb09ffff640 10 client.944141.objecter resend_mon_ops 2024-09-16T04:00:56.845 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.816+0000 7fb09ffff640 1 -- 172.21.15.42:0/3880176404 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0ac00f040 con 0x7fb0b806dac0 2024-09-16T04:00:56.845 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.816+0000 7fb09ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:00:56.845 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.816+0000 7fb0b7fff640 1 --2- 172.21.15.42:0/3880176404 >> v2:172.21.15.154:6812/254845342 conn(0x7fb0a4037840 0x7fb0a4039d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:00:56.845 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.816+0000 7fb09ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:00:56.845 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.816+0000 7fb09ffff640 10 monclient: dump: 2024-09-16T04:00:56.845 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:00:56.845 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:00:56.845 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:00:56.846 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:00:56.846 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:00:56.846 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:00:56.846 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:00:56.846 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:00:56.846 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:00:56.846 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:00:56.846 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.816+0000 7fb0b7fff640 10 monclient: get_auth_request con 0x7fb0a4037840 auth_method 0 2024-09-16T04:00:56.846 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.816+0000 7fb09ffff640 1 -- 172.21.15.42:0/3880176404 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fb0ac010e70 con 0x7fb0b806dac0 2024-09-16T04:00:56.846 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.816+0000 7fb09ffff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:00:56.846 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.816+0000 7fb09ffff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:00:56.846 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.816+0000 7fb09ffff640 1 -- 172.21.15.42:0/3880176404 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb0b806dac0 msgr2=0x7fb0b809d0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:00:56.846 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.816+0000 7fb09ffff640 1 --2- 172.21.15.42:0/3880176404 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb0b806dac0 0x7fb0b809d0a0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fb0b809e130 tx=0x7fb0ac010c20 comp rx=0 tx=0).stop 2024-09-16T04:00:56.846 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.816+0000 7fb09ffff640 1 -- 172.21.15.42:0/3880176404 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb0b806dac0 msgr2=0x7fb0b809d0a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:00:56.847 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.816+0000 7fb09ffff640 1 --2- 172.21.15.42:0/3880176404 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb0b806dac0 0x7fb0b809d0a0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:00:56.847 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.816+0000 7fb09ffff640 10 client.944141.objecter ms_handle_connect 0x7fb0a4037840 2024-09-16T04:00:56.847 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.816+0000 7fb0b7fff640 1 --2- 172.21.15.42:0/3880176404 >> v2:172.21.15.154:6812/254845342 conn(0x7fb0a4037840 0x7fb0a4039d00 secure :-1 s=READY pgs=3752 cs=0 l=1 rev1=1 crypto rx=0x7fb0b0007740 tx=0x7fb0b0009820 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:00:56.847 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.820+0000 7fb0be1e9640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:00:56.847 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.820+0000 7fb0be1e9640 1 --2- 172.21.15.42:0/3880176404 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb0b809d5e0 0x7fb0b8137330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:00:56.847 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.820+0000 7fb0be1e9640 10 monclient(hunting): start opening mon connection 2024-09-16T04:00:56.847 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.820+0000 7fb0be1e9640 1 -- 172.21.15.42:0/3880176404 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fb0b809d9f0 con 0x7fb0b809d5e0 2024-09-16T04:00:56.847 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.824+0000 7fb0b6ffd640 1 --2- 172.21.15.42:0/3880176404 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb0b809d5e0 0x7fb0b8137330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:00:56.847 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.824+0000 7fb0b6ffd640 10 monclient: get_auth_request con 0x7fb0b809d5e0 auth_method 0 2024-09-16T04:00:56.847 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.824+0000 7fb0b6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:00:56.847 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.824+0000 7fb0b6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:00:56.847 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.824+0000 7fb0b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:00:56.847 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.824+0000 7fb0b6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:00:56.847 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.824+0000 7fb0b6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:00:56.848 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.824+0000 7fb0b6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:00:56.848 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.824+0000 7fb0b6ffd640 10 monclient(hunting): handle_auth_done global_id 944147 payload 824 2024-09-16T04:00:56.848 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.824+0000 7fb0b6ffd640 1 -- 172.21.15.42:0/3880176404 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fb0b8134eb0 con 0x7fb0b809d5e0 2024-09-16T04:00:56.848 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.824+0000 7fb0b6ffd640 1 --2- 172.21.15.42:0/3880176404 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb0b809d5e0 0x7fb0b8137330 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fb0b809e130 tx=0x7fb0ac0046d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:00:56.848 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.824+0000 7fb09ffff640 10 client.944141.objecter ms_handle_connect 0x7fb0b809d5e0 2024-09-16T04:00:56.848 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.824+0000 7fb09ffff640 10 client.944141.objecter resend_mon_ops 2024-09-16T04:00:56.848 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.824+0000 7fb09ffff640 1 -- 172.21.15.42:0/3880176404 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0ac0059a0 con 0x7fb0b809d5e0 2024-09-16T04:00:56.848 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.824+0000 7fb09ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:00:56.848 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.824+0000 7fb09ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:00:56.848 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.824+0000 7fb09ffff640 10 monclient: dump: 2024-09-16T04:00:56.848 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:00:56.848 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:00:56.848 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:00:56.849 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:00:56.849 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:00:56.849 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:00:56.849 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:00:56.849 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:00:56.849 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:00:56.849 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:00:56.849 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.824+0000 7fb09ffff640 1 -- 172.21.15.42:0/3880176404 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4545 (secure 0 0 0) 0x7fb0ac0059a0 con 0x7fb0b809d5e0 2024-09-16T04:00:56.849 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.824+0000 7fb09ffff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:00:56.849 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.824+0000 7fb09ffff640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:00:56.849 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.824+0000 7fb09ffff640 1 -- 172.21.15.42:0/3880176404 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb0b809d5e0 msgr2=0x7fb0b8137330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:00:56.849 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.824+0000 7fb09ffff640 1 --2- 172.21.15.42:0/3880176404 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb0b809d5e0 0x7fb0b8137330 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fb0b809e130 tx=0x7fb0ac0046d0 comp rx=0 tx=0).stop 2024-09-16T04:00:56.849 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.824+0000 7fb09ffff640 1 -- 172.21.15.42:0/3880176404 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb0b809d5e0 msgr2=0x7fb0b8137330 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:00:56.849 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.824+0000 7fb09ffff640 1 --2- 172.21.15.42:0/3880176404 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb0b809d5e0 0x7fb0b8137330 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:00:56.850 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T04:00:56.850 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:00:56.850 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:00:56.850 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T04:00:56.850 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 168, 2024-09-16T04:00:56.850 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 25655, 2024-09-16T04:00:56.850 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T04:00:56.850 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T04:00:56.850 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T04:00:56.850 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T04:00:56.850 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:00:56.851 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 5, 2024-09-16T04:00:56.851 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:00:56.851 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T04:00:56.851 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T04:00:56.851 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:00:56.851 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:00:56.851 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:00:56.851 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:00:56.851 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:00:56.851 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:00:56.851 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:00:56.851 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:00:56.851 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:00:56.851 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:00:56.852 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:00:56.852 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:00:56.852 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:00:56.852 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T04:00:56.852 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:00:56.852 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:00:56.852 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:00:56.852 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:00:56.852 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:00:56.852 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:00:56.852 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:00:56.852 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:00:56.852 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:00:56.852 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:00:56.853 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:00:56.853 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:00:56.853 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:00:56.853 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T04:00:56.853 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T04:00:56.853 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T04:00:56.853 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T04:00:56.853 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T04:00:56.853 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T04:00:56.853 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:00:56.853 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:00:56.853 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T04:00:56.853 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T04:00:56.853 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T04:00:56.854 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T04:00:56.854 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T04:00:56.854 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T04:00:56.854 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T04:00:56.854 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:00:56.854 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T04:00:56.854 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:00:56.854 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:00:56.854 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:00:56.854 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:00:56.854 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:00:56.854 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:00:56.854 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:00:56.854 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:00:56.855 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:00:56.855 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:00:56.855 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:00:56.855 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:00:56.855 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T04:00:56.855 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:00:56.855 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T04:00:56.855 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:00:56.855 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:00:56.855 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:00:56.855 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:00:56.855 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:00:56.855 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:00:56.855 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:00:56.856 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T04:00:56.856 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:00:56.856 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:00:56.856 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:00:56.856 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:00:56.856 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T04:00:56.856 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:00:56.856 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:00:56.856 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:00:56.856 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:00:56.856 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T04:00:56.856 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T04:00:56.856 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:00:56.856 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:00:56.857 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:00:56.857 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:00:56.857 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:00:56.857 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:00:56.857 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:00:56.857 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:00:56.857 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:00:56.857 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:00:56.857 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:00:56.857 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T04:00:56.857 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:00:56.857 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:00:56.857 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:00:56.857 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:00:56.858 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T04:00:56.858 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:00:56.858 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:00:56.858 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:00:56.858 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:00:56.858 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T04:00:56.858 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T04:00:56.858 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:00:56.858 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:00:56.858 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:00:56.858 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:00:56.858 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:00:56.858 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:00:56.858 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:00:56.859 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:00:56.859 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:00:56.859 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:00:56.859 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:00:56.859 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T04:00:56.859 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:00:56.859 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:00:56.859 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:00:56.859 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:00:56.859 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T04:00:56.859 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:00:56.859 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:00:56.859 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:00:56.859 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:00:56.859 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T04:00:56.860 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T04:00:56.860 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:00:56.860 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:00:56.860 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:00:56.860 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:00:56.860 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:00:56.860 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:00:56.860 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T04:00:56.860 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T04:00:56.860 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:00:56.860 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:00:56.860 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:00:56.860 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:00:56.860 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:00:56.861 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:00:56.861 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T04:00:56.861 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:00:56.861 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:00:56.861 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:00:56.861 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 2 2024-09-16T04:00:56.861 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:00:56.861 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:00:56.861 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:00:56.861 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T04:00:56.861 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T04:00:56.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.824+0000 7fb0be1e9640 10 librados: watch_flush enter 2024-09-16T04:00:56.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.824+0000 7fb0be1e9640 10 librados: watch_flush exit 2024-09-16T04:00:56.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.824+0000 7fb0be1e9640 20 client.944141.objecter shutdown clearing up homeless session... 2024-09-16T04:00:56.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.824+0000 7fb0be1e9640 10 client.944141.objecter successfully canceled tick 2024-09-16T04:00:56.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.824+0000 7fb0be1e9640 1 -- 172.21.15.42:0/3880176404 >> v2:172.21.15.154:6812/254845342 conn(0x7fb0a4037840 msgr2=0x7fb0a4039d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:00:56.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.824+0000 7fb0be1e9640 1 --2- 172.21.15.42:0/3880176404 >> v2:172.21.15.154:6812/254845342 conn(0x7fb0a4037840 0x7fb0a4039d00 secure :-1 s=READY pgs=3752 cs=0 l=1 rev1=1 crypto rx=0x7fb0b0007740 tx=0x7fb0b0009820 comp rx=0 tx=0).stop 2024-09-16T04:00:56.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.824+0000 7fb0be1e9640 10 monclient: shutdown 2024-09-16T04:00:56.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.824+0000 7fb0be1e9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:00:56.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.824+0000 7fb0be1e9640 1 -- 172.21.15.42:0/3880176404 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb0b80a1330 msgr2=0x7fb0b81cef30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:00:56.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.824+0000 7fb0b7fff640 1 -- 172.21.15.42:0/3880176404 reap_dead start 2024-09-16T04:00:56.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.824+0000 7fb0be1e9640 1 --2- 172.21.15.42:0/3880176404 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb0b80a1330 0x7fb0b81cef30 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fb0a80af040 tx=0x7fb0a80191e0 comp rx=0 tx=0).stop 2024-09-16T04:00:56.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.824+0000 7fb0be1e9640 1 -- 172.21.15.42:0/3880176404 shutdown_connections 2024-09-16T04:00:56.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.824+0000 7fb0be1e9640 1 --2- 172.21.15.42:0/3880176404 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb0b80a1330 0x7fb0b81cef30 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:00:56.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.824+0000 7fb0be1e9640 1 -- 172.21.15.42:0/3880176404 >> 172.21.15.42:0/3880176404 conn(0x7fb0b80a2420 msgr2=0x7fb0b80a3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:00:56.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.824+0000 7fb0be1e9640 1 -- 172.21.15.42:0/3880176404 shutdown_connections 2024-09-16T04:00:56.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.824+0000 7fb0be1e9640 1 -- 172.21.15.42:0/3880176404 wait complete. 2024-09-16T04:00:56.865 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.824+0000 7fb0be1e9640 1 librados: shutdown 2024-09-16T04:00:56.865 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:00:56.865 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T04:00:56.865 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T04:00:56.984 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.976+0000 7f09e8b0f640 10 monclient: get_monmap_and_config 2024-09-16T04:00:56.984 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.976+0000 7f09e8b0f640 10 monclient: build_initial_monmap 2024-09-16T04:00:56.984 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.976+0000 7f09e8b0f640 10 monclient: monmap: 2024-09-16T04:00:56.984 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:00:56.984 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:00:56.984 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:00:56.982558+0000 2024-09-16T04:00:56.984 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:00:56.982558+0000 2024-09-16T04:00:56.985 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:00:56.985 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:00:56.985 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:00:56.985 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:00:56.985 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:00:56.985 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:00:56.985 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:00:56.985 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:00:56.987 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.976+0000 7f09e8b0f640 1 Processor -- start 2024-09-16T04:00:56.987 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.976+0000 7f09e8b0f640 1 -- start start 2024-09-16T04:00:56.987 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.976+0000 7f09e8b0f640 10 monclient: init 2024-09-16T04:00:56.987 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.976+0000 7f09e8b0f640 10 monclient: _reopen_session rank -1 2024-09-16T04:00:56.987 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.976+0000 7f09e8b0f640 10 monclient: _add_conns ranks=[1,4,2,0,3] 2024-09-16T04:00:56.987 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.976+0000 7f09e8b0f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f09e40a67e0 addr v1:172.21.15.154:6789/0 2024-09-16T04:00:56.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.976+0000 7f09e8b0f640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f09e4072560 0x7f09e413a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:00:56.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.976+0000 7f09e8b0f640 10 monclient(hunting): picked mon.noname-c con 0x7f09e4072560 addr v2:172.21.15.154:3300/0 2024-09-16T04:00:56.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.976+0000 7f09e8b0f640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f09e40a72d0 0x7f09e413c820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:00:56.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.976+0000 7f09e8b0f640 10 monclient(hunting): picked mon.noname-b con 0x7f09e40a72d0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:00:56.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.976+0000 7f09e8b0f640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f09e4071b80 con 0x7f09e40a67e0 2024-09-16T04:00:56.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.976+0000 7f09e8b0f640 10 monclient(hunting): start opening mon connection 2024-09-16T04:00:56.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.976+0000 7f09e8b0f640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f09e4072ed0 con 0x7f09e4072560 2024-09-16T04:00:56.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.976+0000 7f09e8b0f640 10 monclient(hunting): start opening mon connection 2024-09-16T04:00:56.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.976+0000 7f09e8b0f640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f09e4070030 con 0x7f09e40a72d0 2024-09-16T04:00:56.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.976+0000 7f09e8b0f640 10 monclient(hunting): _renew_subs 2024-09-16T04:00:56.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.976+0000 7f09e8b0f640 10 monclient(hunting): authenticate will time out at 2103.951976s 2024-09-16T04:00:56.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.980+0000 7f09e2d9d640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f09e40a72d0 0x7f09e413c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:00:56.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.980+0000 7f09e2d9d640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f09e40a72d0 0x7f09e413c820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:40812/0 (socket says 172.21.15.42:40812) 2024-09-16T04:00:56.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.980+0000 7f09e2d9d640 1 -- 172.21.15.42:0/525916491 learned_addr learned my addr 172.21.15.42:0/525916491 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:00:56.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.980+0000 7f09e2d9d640 10 monclient(hunting): get_auth_request con 0x7f09e40a72d0 auth_method 0 2024-09-16T04:00:56.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.980+0000 7f09e2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:00:56.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.980+0000 7f09e2d9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:00:56.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.980+0000 7f09e2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:00:56.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.980+0000 7f09e2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:00:56.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.980+0000 7f09e2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:00:56.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.980+0000 7f09e2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:00:56.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.980+0000 7f09e1d9b640 1 --2- 172.21.15.42:0/525916491 >> v2:172.21.15.154:3300/0 conn(0x7f09e4072560 0x7f09e413a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:00:56.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.980+0000 7f09e1d9b640 10 monclient(hunting): get_auth_request con 0x7f09e4072560 auth_method 0 2024-09-16T04:00:56.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.980+0000 7f09e1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:00:56.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.980+0000 7f09e1d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:00:56.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.980+0000 7f09e1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:00:56.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.980+0000 7f09e2d9d640 10 monclient(hunting): handle_auth_done global_id 944150 payload 274 2024-09-16T04:00:56.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.980+0000 7f09e2d9d640 1 -- 172.21.15.42:0/525916491 >> v2:172.21.15.154:3300/0 conn(0x7f09e4072560 msgr2=0x7f09e413a2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:00:56.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.980+0000 7f09e2d9d640 1 --2- 172.21.15.42:0/525916491 >> v2:172.21.15.154:3300/0 conn(0x7f09e4072560 0x7f09e413a2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:00:56.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.980+0000 7f09e2d9d640 1 -- 172.21.15.42:0/525916491 >> v1:172.21.15.154:6789/0 conn(0x7f09e40a67e0 legacy=0x7f09e40a6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:00:56.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.980+0000 7f09e2d9d640 10 monclient: _finish_hunting 0 2024-09-16T04:00:56.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.980+0000 7f09e2d9d640 1 monclient: found mon.noname-b 2024-09-16T04:00:56.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.980+0000 7f09e2d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:00:56.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.980+0000 7f09e2d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T04:00:56.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.980+0000 7f09e2d9d640 1 -- 172.21.15.42:0/525916491 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09e40712e0 con 0x7f09e40a72d0 2024-09-16T04:00:56.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.980+0000 7f09e1d9b640 1 --2- 172.21.15.42:0/525916491 >> v2:172.21.15.154:3300/0 conn(0x7f09e4072560 0x7f09e413a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T04:00:56.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.980+0000 7f09e2d9d640 1 --2- 172.21.15.42:0/525916491 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f09e40a72d0 0x7f09e413c820 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f09d4009870 tx=0x7f09d401c3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2d704a7f98957943 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:00:56.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.980+0000 7f09e159a640 1 -- 172.21.15.42:0/525916491 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09d40b2070 con 0x7f09e40a72d0 2024-09-16T04:00:56.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.980+0000 7f09e159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:00:56.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.980+0000 7f09e159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:00:56.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.980+0000 7f09e159a640 10 monclient: dump: 2024-09-16T04:00:56.997 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:00:56.997 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:00:56.998 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:00:56.998 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:00:56.998 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:00:56.998 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:00:56.998 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:00:56.998 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:00:56.998 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:00:56.998 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:00:56.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.980+0000 7f09e159a640 10 monclient: _finish_auth 0 2024-09-16T04:00:56.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.980+0000 7f09e159a640 10 monclient: _check_auth_tickets 2024-09-16T04:00:56.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.980+0000 7f09e159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:00:56.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.980+0000 7f09e159a640 1 -- 172.21.15.42:0/525916491 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f09d40ad640 con 0x7f09e40a72d0 2024-09-16T04:00:56.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.980+0000 7f09e159a640 10 monclient: handle_config config(0 keys) 2024-09-16T04:00:56.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.980+0000 7f09e159a640 1 -- 172.21.15.42:0/525916491 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09d40ada10 con 0x7f09e40a72d0 2024-09-16T04:00:56.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.980+0000 7f09e8b0f640 5 monclient: authenticate success, global_id 944150 2024-09-16T04:00:56.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.980+0000 7f09e8b0f640 10 monclient: get_monmap_and_config success 2024-09-16T04:00:56.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.980+0000 7f09e8b0f640 4 set_mon_vals no callback set 2024-09-16T04:00:56.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.980+0000 7f09e159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:00:56.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.980+0000 7f09e159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:00:56.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.980+0000 7f09e159a640 10 monclient: dump: 2024-09-16T04:00:56.999 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:00:56.999 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:00:56.999 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:00:56.999 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:00:56.999 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:00:56.999 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:00:56.999 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:00:56.999 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:00:56.999 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:00:56.999 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:00:57.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.980+0000 7f09e8b0f640 10 monclient: shutdown 2024-09-16T04:00:57.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.980+0000 7f09e8b0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:00:57.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.980+0000 7f09e8b0f640 1 -- 172.21.15.42:0/525916491 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f09e40a72d0 msgr2=0x7f09e413c820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:00:57.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.980+0000 7f09e8b0f640 1 --2- 172.21.15.42:0/525916491 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f09e40a72d0 0x7f09e413c820 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f09d4009870 tx=0x7f09d401c3b0 comp rx=0 tx=0).stop 2024-09-16T04:00:57.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.980+0000 7f09e8b0f640 1 -- 172.21.15.42:0/525916491 shutdown_connections 2024-09-16T04:00:57.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.980+0000 7f09e8b0f640 1 --2- 172.21.15.42:0/525916491 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f09e40a72d0 0x7f09e413c820 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:00:57.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.980+0000 7f09e8b0f640 1 --2- 172.21.15.42:0/525916491 >> v2:172.21.15.154:3300/0 conn(0x7f09e4072560 0x7f09e413a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:00:57.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.980+0000 7f09e8b0f640 1 -- 172.21.15.42:0/525916491 >> 172.21.15.42:0/525916491 conn(0x7f09e40a7ed0 msgr2=0x7f09e40a43d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:00:57.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.980+0000 7f09e8b0f640 1 -- 172.21.15.42:0/525916491 shutdown_connections 2024-09-16T04:00:57.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.980+0000 7f09e8b0f640 1 -- 172.21.15.42:0/525916491 wait complete. 2024-09-16T04:00:57.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.984+0000 7f09e8b0f640 10 monclient: build_initial_monmap 2024-09-16T04:00:57.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.984+0000 7f09e8b0f640 10 monclient: monmap: 2024-09-16T04:00:57.000 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:00:57.001 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:00:57.001 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:00:57.001 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:00:57.001 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:00:57.001 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:00:57.001 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:00:57.001 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:00:57.001 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:00:57.001 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:00:57.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.984+0000 7f09e8b0f640 1 librados: starting msgr at 2024-09-16T04:00:57.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.984+0000 7f09e8b0f640 1 librados: starting objecter 2024-09-16T04:00:57.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.984+0000 7f09e8b0f640 1 Processor -- start 2024-09-16T04:00:57.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.984+0000 7f09e8b0f640 1 -- start start 2024-09-16T04:00:57.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.984+0000 7f09e8b0f640 1 librados: setting wanted keys 2024-09-16T04:00:57.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.984+0000 7f09e8b0f640 1 librados: calling monclient init 2024-09-16T04:00:57.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.984+0000 7f09e8b0f640 10 monclient: init 2024-09-16T04:00:57.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.984+0000 7f09e8b0f640 10 monclient: _reopen_session rank -1 2024-09-16T04:00:57.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.984+0000 7f09e8b0f640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:00:57.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.984+0000 7f09e8b0f640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f09e4072560 0x7f09e413b590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:00:57.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.984+0000 7f09e8b0f640 10 monclient(hunting): picked mon.noname-c con 0x7f09e4072560 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:00:57.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.984+0000 7f09e8b0f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f09e40a67e0 0x7f09e41cf730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:00:57.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.984+0000 7f09e8b0f640 10 monclient(hunting): picked mon.noname-b con 0x7f09e40a67e0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:00:57.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.984+0000 7f09e8b0f640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f09e40a72d0 0x7f09e41cba10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:00:57.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.984+0000 7f09e8b0f640 10 monclient(hunting): picked mon.noname-a con 0x7f09e40a72d0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:00:57.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.984+0000 7f09e8b0f640 10 monclient(hunting): start opening mon connection 2024-09-16T04:00:57.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.984+0000 7f09e8b0f640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f09e413f700 con 0x7f09e40a72d0 2024-09-16T04:00:57.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.984+0000 7f09e8b0f640 10 monclient(hunting): start opening mon connection 2024-09-16T04:00:57.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.984+0000 7f09e8b0f640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f09e413f580 con 0x7f09e40a67e0 2024-09-16T04:00:57.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.984+0000 7f09e8b0f640 10 monclient(hunting): start opening mon connection 2024-09-16T04:00:57.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.984+0000 7f09e8b0f640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f09e413f880 con 0x7f09e4072560 2024-09-16T04:00:57.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.984+0000 7f09e8b0f640 10 monclient(hunting): _renew_subs 2024-09-16T04:00:57.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.984+0000 7f09e8b0f640 10 monclient(hunting): authenticate will time out at 2103.957207s 2024-09-16T04:00:57.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.984+0000 7f09e259c640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f09e4072560 0x7f09e413b590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:00:57.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.984+0000 7f09e2d9d640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f09e40a72d0 0x7f09e41cba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:00:57.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.984+0000 7f09e2d9d640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f09e40a72d0 0x7f09e41cba10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:44160/0 (socket says 172.21.15.42:44160) 2024-09-16T04:00:57.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.984+0000 7f09e259c640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f09e4072560 0x7f09e413b590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:40826/0 (socket says 172.21.15.42:40826) 2024-09-16T04:00:57.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.984+0000 7f09e2d9d640 1 -- 172.21.15.42:0/1877181973 learned_addr learned my addr 172.21.15.42:0/1877181973 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:00:57.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.984+0000 7f09e2d9d640 10 monclient(hunting): get_auth_request con 0x7f09e40a72d0 auth_method 0 2024-09-16T04:00:57.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.984+0000 7f09e2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:00:57.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.984+0000 7f09e2d9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:00:57.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.984+0000 7f09e2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:00:57.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.984+0000 7f09e259c640 10 monclient(hunting): get_auth_request con 0x7f09e4072560 auth_method 0 2024-09-16T04:00:57.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.984+0000 7f09e259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:00:57.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.984+0000 7f09e259c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:00:57.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.984+0000 7f09e259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:00:57.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.984+0000 7f09e2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:00:57.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.984+0000 7f09e2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:00:57.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.984+0000 7f09e2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:00:57.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.984+0000 7f09e259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:00:57.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.984+0000 7f09e259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:00:57.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.984+0000 7f09e259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:00:57.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.984+0000 7f09e1d9b640 1 --2- 172.21.15.42:0/1877181973 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f09e40a67e0 0x7f09e41cf730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:00:57.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.984+0000 7f09e1d9b640 10 monclient(hunting): get_auth_request con 0x7f09e40a67e0 auth_method 0 2024-09-16T04:00:57.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.984+0000 7f09e1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:00:57.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.984+0000 7f09e1d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:00:57.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.984+0000 7f09e1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:00:57.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.984+0000 7f09e259c640 10 monclient(hunting): handle_auth_done global_id 944153 payload 824 2024-09-16T04:00:57.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.984+0000 7f09e259c640 1 -- 172.21.15.42:0/1877181973 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f09e40a67e0 msgr2=0x7f09e41cf730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:00:57.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.984+0000 7f09e259c640 1 --2- 172.21.15.42:0/1877181973 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f09e40a67e0 0x7f09e41cf730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:00:57.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.984+0000 7f09e259c640 1 -- 172.21.15.42:0/1877181973 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f09e40a72d0 msgr2=0x7f09e41cba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:00:57.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.984+0000 7f09e259c640 1 --2- 172.21.15.42:0/1877181973 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f09e40a72d0 0x7f09e41cba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:00:57.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.984+0000 7f09e259c640 10 monclient: _finish_hunting 0 2024-09-16T04:00:57.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.984+0000 7f09e259c640 1 monclient: found mon.noname-c 2024-09-16T04:00:57.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.984+0000 7f09e259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:00:57.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.984+0000 7f09e259c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T04:00:57.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.984+0000 7f09e259c640 1 -- 172.21.15.42:0/1877181973 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09e4121130 con 0x7f09e4072560 2024-09-16T04:00:57.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.984+0000 7f09e2d9d640 1 --2- 172.21.15.42:0/1877181973 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f09e40a72d0 0x7f09e41cba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:00:57.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.984+0000 7f09e259c640 1 --2- 172.21.15.42:0/1877181973 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f09e4072560 0x7f09e413b590 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f09dc00c700 tx=0x7f09dc00cbc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:00:57.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.984+0000 7f09daffd640 10 client.?.objecter ms_handle_connect 0x7f09e4072560 2024-09-16T04:00:57.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.984+0000 7f09daffd640 10 client.?.objecter resend_mon_ops 2024-09-16T04:00:57.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.984+0000 7f09daffd640 1 -- 172.21.15.42:0/1877181973 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09dc016020 con 0x7f09e4072560 2024-09-16T04:00:57.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.984+0000 7f09daffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:00:57.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.984+0000 7f09daffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:00:57.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.984+0000 7f09daffd640 10 monclient: dump: 2024-09-16T04:00:57.006 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:00:57.006 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:00:57.006 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:00:57.006 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:00:57.006 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:00:57.006 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:00:57.006 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:00:57.007 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:00:57.007 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:00:57.007 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:00:57.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.984+0000 7f09daffd640 10 monclient: _finish_auth 0 2024-09-16T04:00:57.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.984+0000 7f09daffd640 10 monclient: _check_auth_tickets 2024-09-16T04:00:57.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.984+0000 7f09daffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:00:57.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.984+0000 7f09daffd640 1 -- 172.21.15.42:0/1877181973 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f09dc00aa50 con 0x7f09e4072560 2024-09-16T04:00:57.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.984+0000 7f09daffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:00:57.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.984+0000 7f09daffd640 1 -- 172.21.15.42:0/1877181973 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09dc00ae40 con 0x7f09e4072560 2024-09-16T04:00:57.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.984+0000 7f09e8b0f640 5 monclient: authenticate success, global_id 944153 2024-09-16T04:00:57.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.984+0000 7f09e8b0f640 10 monclient: _renew_subs 2024-09-16T04:00:57.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.984+0000 7f09e8b0f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:00:57.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.984+0000 7f09e8b0f640 1 -- 172.21.15.42:0/1877181973 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09e409bd50 con 0x7f09e4072560 2024-09-16T04:00:57.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.984+0000 7f09daffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:00:57.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.984+0000 7f09daffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:00:57.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.984+0000 7f09e0d99640 4 set_mon_vals no callback set 2024-09-16T04:00:57.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.984+0000 7f09daffd640 10 monclient: dump: 2024-09-16T04:00:57.008 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:00:57.008 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:00:57.008 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:00:57.008 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:00:57.008 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:00:57.008 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:00:57.008 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:00:57.008 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:00:57.008 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:00:57.008 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:00:57.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.984+0000 7f09e8b0f640 10 client.944153.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:00:57.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.984+0000 7f09e8b0f640 10 monclient: _renew_subs 2024-09-16T04:00:57.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.984+0000 7f09e8b0f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:00:57.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.984+0000 7f09e8b0f640 1 -- 172.21.15.42:0/1877181973 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f09e4120e50 con 0x7f09e4072560 2024-09-16T04:00:57.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.984+0000 7f09e8b0f640 1 librados: init done 2024-09-16T04:00:57.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.984+0000 7f09e8b0f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:00:57.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.984+0000 7f09e8b0f640 1 --2- 172.21.15.42:0/1877181973 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f09e402c650 0x7f09e402ca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:00:57.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.984+0000 7f09e8b0f640 10 monclient(hunting): start opening mon connection 2024-09-16T04:00:57.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.984+0000 7f09e8b0f640 1 -- 172.21.15.42:0/1877181973 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f09e411f780 con 0x7f09e402c650 2024-09-16T04:00:57.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.984+0000 7f09daffd640 1 -- 172.21.15.42:0/1877181973 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f09dc019020 con 0x7f09e4072560 2024-09-16T04:00:57.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.984+0000 7f09daffd640 1 --2- 172.21.15.42:0/1877181973 >> v2:172.21.15.154:6812/254845342 conn(0x7f09c4037810 0x7f09c4039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:00:57.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.984+0000 7f09daffd640 1 -- 172.21.15.42:0/1877181973 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(569..569 src has 451..569) ==== 6216+0+0 (secure 0 0 0) 0x7f09dc049340 con 0x7f09e4072560 2024-09-16T04:00:57.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.984+0000 7f09daffd640 10 client.944153.objecter ms_dispatch 0x7f09e4000ba0 osd_map(569..569 src has 451..569) 2024-09-16T04:00:57.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.984+0000 7f09daffd640 3 client.944153.objecter handle_osd_map got epochs [569,569] > 0 2024-09-16T04:00:57.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.984+0000 7f09daffd640 3 client.944153.objecter handle_osd_map decoding full epoch 569 2024-09-16T04:00:57.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.984+0000 7f09e1d9b640 1 --2- 172.21.15.42:0/1877181973 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f09e402c650 0x7f09e402ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:00:57.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.984+0000 7f09e1d9b640 10 monclient: get_auth_request con 0x7f09e402c650 auth_method 0 2024-09-16T04:00:57.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.984+0000 7f09daffd640 20 client.944153.objecter dump_active .. 0 homeless 2024-09-16T04:00:57.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.984+0000 7f09e1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:00:57.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.984+0000 7f09e1d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:00:57.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.984+0000 7f09e1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:00:57.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.984+0000 7f09e1d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:00:57.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.984+0000 7f09e1d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:00:57.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.984+0000 7f09e1d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:00:57.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.984+0000 7f09e2d9d640 1 --2- 172.21.15.42:0/1877181973 >> v2:172.21.15.154:6812/254845342 conn(0x7f09c4037810 0x7f09c4039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:00:57.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.984+0000 7f09e2d9d640 10 monclient: get_auth_request con 0x7f09c4037810 auth_method 0 2024-09-16T04:00:57.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.984+0000 7f09e1d9b640 10 monclient(hunting): handle_auth_done global_id 904447 payload 824 2024-09-16T04:00:57.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.984+0000 7f09e1d9b640 1 -- 172.21.15.42:0/1877181973 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f09e406f500 con 0x7f09e402c650 2024-09-16T04:00:57.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.988+0000 7f09daffd640 10 client.944153.objecter ms_handle_connect 0x7f09e402c650 2024-09-16T04:00:57.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.988+0000 7f09daffd640 10 client.944153.objecter resend_mon_ops 2024-09-16T04:00:57.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.988+0000 7f09e1d9b640 1 --2- 172.21.15.42:0/1877181973 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f09e402c650 0x7f09e402ca30 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f09e413be20 tx=0x7f09cc010c20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:00:57.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.988+0000 7f09daffd640 10 client.944153.objecter ms_handle_connect 0x7f09c4037810 2024-09-16T04:00:57.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.988+0000 7f09e2d9d640 1 --2- 172.21.15.42:0/1877181973 >> v2:172.21.15.154:6812/254845342 conn(0x7f09c4037810 0x7f09c4039cd0 secure :-1 s=READY pgs=3753 cs=0 l=1 rev1=1 crypto rx=0x7f09d4009870 tx=0x7f09d4009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:00:57.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.988+0000 7f09daffd640 1 -- 172.21.15.42:0/1877181973 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09cc01a030 con 0x7f09e402c650 2024-09-16T04:00:57.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.988+0000 7f09daffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:00:57.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.988+0000 7f09daffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:00:57.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.988+0000 7f09daffd640 10 monclient: dump: 2024-09-16T04:00:57.012 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:00:57.012 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:00:57.012 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:00:57.012 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:00:57.012 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:00:57.012 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:00:57.012 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:00:57.012 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:00:57.012 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:00:57.012 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:00:57.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.988+0000 7f09daffd640 1 -- 172.21.15.42:0/1877181973 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f09cc004020 con 0x7f09e402c650 2024-09-16T04:00:57.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.988+0000 7f09daffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:00:57.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.988+0000 7f09daffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:00:57.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.988+0000 7f09daffd640 1 -- 172.21.15.42:0/1877181973 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f09e402c650 msgr2=0x7f09e402ca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:00:57.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.988+0000 7f09daffd640 1 --2- 172.21.15.42:0/1877181973 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f09e402c650 0x7f09e402ca30 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f09e413be20 tx=0x7f09cc010c20 comp rx=0 tx=0).stop 2024-09-16T04:00:57.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.988+0000 7f09daffd640 1 -- 172.21.15.42:0/1877181973 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f09e402c650 msgr2=0x7f09e402ca30 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:00:57.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.988+0000 7f09daffd640 1 --2- 172.21.15.42:0/1877181973 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f09e402c650 0x7f09e402ca30 secure :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f09e413be20 tx=0x7f09cc010c20 comp rx=0 tx=0).stop 2024-09-16T04:00:57.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.992+0000 7f09e8b0f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:00:57.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.992+0000 7f09e8b0f640 1 --2- 172.21.15.42:0/1877181973 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f09e402cf70 0x7f09e402d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:00:57.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.992+0000 7f09e8b0f640 10 monclient(hunting): start opening mon connection 2024-09-16T04:00:57.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.992+0000 7f09e8b0f640 1 -- 172.21.15.42:0/1877181973 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f09e4068e40 con 0x7f09e402cf70 2024-09-16T04:00:57.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.992+0000 7f09e1d9b640 1 --2- 172.21.15.42:0/1877181973 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f09e402cf70 0x7f09e402d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:00:57.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.992+0000 7f09e1d9b640 10 monclient: get_auth_request con 0x7f09e402cf70 auth_method 0 2024-09-16T04:00:57.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.992+0000 7f09e1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:00:57.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.992+0000 7f09e1d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:00:57.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.992+0000 7f09e1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:00:57.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.996+0000 7f09e1d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:00:57.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.996+0000 7f09e1d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:00:57.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.996+0000 7f09e1d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:00:57.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.996+0000 7f09e1d9b640 10 monclient(hunting): handle_auth_done global_id 904450 payload 824 2024-09-16T04:00:57.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.996+0000 7f09e1d9b640 1 -- 172.21.15.42:0/1877181973 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f09e40a6bc0 con 0x7f09e402cf70 2024-09-16T04:00:57.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.996+0000 7f09e1d9b640 1 --2- 172.21.15.42:0/1877181973 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f09e402cf70 0x7f09e402d380 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f09cc01a000 tx=0x7f09cc0046d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:00:57.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.996+0000 7f09daffd640 10 client.944153.objecter ms_handle_connect 0x7f09e402cf70 2024-09-16T04:00:57.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.996+0000 7f09daffd640 10 client.944153.objecter resend_mon_ops 2024-09-16T04:00:57.016 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.996+0000 7f09daffd640 1 -- 172.21.15.42:0/1877181973 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09cc020e60 con 0x7f09e402cf70 2024-09-16T04:00:57.016 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.996+0000 7f09daffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:00:57.016 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.996+0000 7f09daffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:00:57.016 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.996+0000 7f09daffd640 10 monclient: dump: 2024-09-16T04:00:57.016 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:00:57.016 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:00:57.016 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:00:57.016 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:00:57.016 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:00:57.016 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:00:57.016 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:00:57.016 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:00:57.016 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:00:57.016 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:00:57.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.996+0000 7f09daffd640 1 -- 172.21.15.42:0/1877181973 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4880 (secure 0 0 0) 0x7f09cc00e7c0 con 0x7f09e402cf70 2024-09-16T04:00:57.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.996+0000 7f09daffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:00:57.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.996+0000 7f09daffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:00:57.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.996+0000 7f09daffd640 1 -- 172.21.15.42:0/1877181973 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f09e402cf70 msgr2=0x7f09e402d380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:00:57.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.996+0000 7f09daffd640 1 --2- 172.21.15.42:0/1877181973 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f09e402cf70 0x7f09e402d380 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f09cc01a000 tx=0x7f09cc0046d0 comp rx=0 tx=0).stop 2024-09-16T04:00:57.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.996+0000 7f09daffd640 1 -- 172.21.15.42:0/1877181973 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f09e402cf70 msgr2=0x7f09e402d380 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:00:57.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.996+0000 7f09daffd640 1 --2- 172.21.15.42:0/1877181973 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f09e402cf70 0x7f09e402d380 secure :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f09cc01a000 tx=0x7f09cc0046d0 comp rx=0 tx=0).stop 2024-09-16T04:00:57.017 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T04:00:57.017 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:00:57.017 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:00:57.018 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T04:00:57.018 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 168, 2024-09-16T04:00:57.018 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 185178, 2024-09-16T04:00:57.018 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T04:00:57.018 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T04:00:57.018 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T04:00:57.018 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T04:00:57.018 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:00:57.018 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 5, 2024-09-16T04:00:57.018 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:00:57.019 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T04:00:57.019 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T04:00:57.019 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:00:57.019 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:00:57.019 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:00:57.019 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:00:57.019 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:00:57.019 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:00:57.019 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:00:57.019 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:00:57.019 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:00:57.019 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:00:57.019 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:00:57.020 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:00:57.020 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:00:57.020 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T04:00:57.020 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:00:57.020 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:00:57.020 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:00:57.020 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:00:57.020 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:00:57.020 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:00:57.020 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:00:57.020 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:00:57.020 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:00:57.020 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:00:57.020 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:00:57.021 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:00:57.021 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:00:57.021 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T04:00:57.021 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T04:00:57.021 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T04:00:57.021 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T04:00:57.021 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T04:00:57.021 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T04:00:57.021 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:00:57.021 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:00:57.021 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T04:00:57.021 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T04:00:57.021 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T04:00:57.022 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T04:00:57.022 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T04:00:57.022 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T04:00:57.022 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T04:00:57.022 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:00:57.022 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T04:00:57.022 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:00:57.022 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:00:57.022 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:00:57.022 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:00:57.022 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:00:57.022 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:00:57.022 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:00:57.023 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:00:57.023 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:00:57.023 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:00:57.023 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:00:57.023 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:00:57.023 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T04:00:57.023 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:00:57.023 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T04:00:57.023 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:00:57.023 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:00:57.023 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:00:57.023 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:00:57.023 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:00:57.023 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:00:57.023 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:00:57.024 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T04:00:57.024 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:00:57.024 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:00:57.024 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:00:57.024 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:00:57.024 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T04:00:57.024 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:00:57.024 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:00:57.024 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:00:57.024 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:00:57.024 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T04:00:57.024 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T04:00:57.024 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:00:57.025 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:00:57.025 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:00:57.025 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:00:57.025 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:00:57.025 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:00:57.025 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:00:57.025 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:00:57.025 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:00:57.025 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:00:57.025 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:00:57.025 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T04:00:57.025 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:00:57.025 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:00:57.025 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:00:57.026 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:00:57.026 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T04:00:57.026 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:00:57.026 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:00:57.026 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:00:57.026 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:00:57.026 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T04:00:57.026 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T04:00:57.026 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:00:57.026 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:00:57.026 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:00:57.026 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:00:57.026 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:00:57.026 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:00:57.026 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:00:57.027 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:00:57.027 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:00:57.027 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:00:57.027 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:00:57.027 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T04:00:57.027 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:00:57.027 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:00:57.027 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:00:57.027 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:00:57.027 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T04:00:57.027 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:00:57.027 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:00:57.027 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:00:57.027 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:00:57.028 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T04:00:57.028 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T04:00:57.028 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:00:57.028 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:00:57.028 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:00:57.028 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:00:57.028 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:00:57.028 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:00:57.028 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T04:00:57.028 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T04:00:57.028 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:00:57.028 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:00:57.028 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:00:57.028 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:00:57.029 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:00:57.029 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:00:57.029 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T04:00:57.029 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:00:57.029 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:00:57.029 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:00:57.029 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:00:57.029 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:00:57.029 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:00:57.029 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T04:00:57.029 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:00:57.029 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:00:57.029 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:00:57.029 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 2 2024-09-16T04:00:57.030 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:00:57.030 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:00:57.030 INFO:teuthology.orchestra.run.smithi042.stdout: "mgr": [ 2024-09-16T04:00:57.030 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:00:57.030 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:00:57.030 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:00:57.030 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:00:57.030 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:00:57.030 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:00:57.030 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:00:57.030 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T04:00:57.030 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T04:00:57.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.996+0000 7f09e8b0f640 10 librados: watch_flush enter 2024-09-16T04:00:57.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.996+0000 7f09e8b0f640 10 librados: watch_flush exit 2024-09-16T04:00:57.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.996+0000 7f09e8b0f640 20 client.944153.objecter shutdown clearing up homeless session... 2024-09-16T04:00:57.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.996+0000 7f09e8b0f640 10 client.944153.objecter successfully canceled tick 2024-09-16T04:00:57.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.996+0000 7f09e8b0f640 1 -- 172.21.15.42:0/1877181973 >> v2:172.21.15.154:6812/254845342 conn(0x7f09c4037810 msgr2=0x7f09c4039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:00:57.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.996+0000 7f09e8b0f640 1 --2- 172.21.15.42:0/1877181973 >> v2:172.21.15.154:6812/254845342 conn(0x7f09c4037810 0x7f09c4039cd0 secure :-1 s=READY pgs=3753 cs=0 l=1 rev1=1 crypto rx=0x7f09d4009870 tx=0x7f09d4009040 comp rx=0 tx=0).stop 2024-09-16T04:00:57.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.996+0000 7f09e8b0f640 10 monclient: shutdown 2024-09-16T04:00:57.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.996+0000 7f09e8b0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:00:57.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.996+0000 7f09e8b0f640 1 -- 172.21.15.42:0/1877181973 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f09e4072560 msgr2=0x7f09e413b590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:00:57.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.996+0000 7f09e8b0f640 1 --2- 172.21.15.42:0/1877181973 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f09e4072560 0x7f09e413b590 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f09dc00c700 tx=0x7f09dc00cbc0 comp rx=0 tx=0).stop 2024-09-16T04:00:57.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.996+0000 7f09e2d9d640 1 -- 172.21.15.42:0/1877181973 reap_dead start 2024-09-16T04:00:57.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.996+0000 7f09e2d9d640 1 -- 172.21.15.42:0/1877181973 reap_dead start 2024-09-16T04:00:57.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.996+0000 7f09e8b0f640 1 -- 172.21.15.42:0/1877181973 shutdown_connections 2024-09-16T04:00:57.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.996+0000 7f09e8b0f640 1 -- 172.21.15.42:0/1877181973 >> 172.21.15.42:0/1877181973 conn(0x7f09e40a7ed0 msgr2=0x7f09e41352a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:00:57.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.996+0000 7f09e8b0f640 1 -- 172.21.15.42:0/1877181973 shutdown_connections 2024-09-16T04:00:57.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.996+0000 7f09e8b0f640 1 -- 172.21.15.42:0/1877181973 wait complete. 2024-09-16T04:00:57.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:56.996+0000 7f09e8b0f640 1 librados: shutdown 2024-09-16T04:00:57.034 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:00:57.034 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-09-16T04:00:57.034 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-16T04:00:57.034 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-09-16T04:00:57.034 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-16T04:00:57.034 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-16T04:00:57.034 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T04:00:57.045 INFO:tasks.ceph.mon.c:Stopped 2024-09-16T04:00:57.045 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T04:00:57.045 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:00:57.205 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.196+0000 7fcaf0d67640 10 monclient: get_monmap_and_config 2024-09-16T04:00:57.205 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.196+0000 7fcaf0d67640 10 monclient: build_initial_monmap 2024-09-16T04:00:57.205 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.196+0000 7fcaf0d67640 10 monclient: monmap: 2024-09-16T04:00:57.205 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:00:57.205 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:00:57.205 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:00:57.203560+0000 2024-09-16T04:00:57.205 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:00:57.203560+0000 2024-09-16T04:00:57.205 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:00:57.205 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:00:57.205 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:00:57.205 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:00:57.206 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:00:57.206 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:00:57.206 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:00:57.206 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:00:57.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.200+0000 7fcaf0d67640 1 Processor -- start 2024-09-16T04:00:57.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.200+0000 7fcaf0d67640 1 -- start start 2024-09-16T04:00:57.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.200+0000 7fcaf0d67640 10 monclient: init 2024-09-16T04:00:57.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.200+0000 7fcaf0d67640 10 monclient: _reopen_session rank -1 2024-09-16T04:00:57.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.200+0000 7fcaf0d67640 10 monclient: _add_conns ranks=[4,3,2,1,0] 2024-09-16T04:00:57.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.200+0000 7fcaf0d67640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fcaec0a39b0 0x7fcaec133b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:00:57.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.200+0000 7fcaf0d67640 10 monclient(hunting): picked mon.noname-c con 0x7fcaec0a39b0 addr v2:172.21.15.154:3300/0 2024-09-16T04:00:57.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.200+0000 7fcaf0d67640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fcaec0a3d90 0x7fcaec137000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:00:57.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.200+0000 7fcaf0d67640 10 monclient(hunting): picked mon.noname-a con 0x7fcaec0a3d90 addr v2:172.21.15.42:3300/0 2024-09-16T04:00:57.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.200+0000 7fcaf0d67640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fcaec137540 0x7fcaec137990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:00:57.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.200+0000 7fcaf0d67640 10 monclient(hunting): picked mon.noname-b con 0x7fcaec137540 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:00:57.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.200+0000 7fcaf0d67640 10 monclient(hunting): start opening mon connection 2024-09-16T04:00:57.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.200+0000 7fcaf0d67640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7fcaec09c140 con 0x7fcaec0a3d90 2024-09-16T04:00:57.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.200+0000 7fcaf0d67640 10 monclient(hunting): start opening mon connection 2024-09-16T04:00:57.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.200+0000 7fcaf0d67640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fcaec09c420 con 0x7fcaec0a39b0 2024-09-16T04:00:57.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.200+0000 7fcaf0d67640 10 monclient(hunting): start opening mon connection 2024-09-16T04:00:57.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.200+0000 7fcaf0d67640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fcaec06dd60 con 0x7fcaec137540 2024-09-16T04:00:57.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.200+0000 7fcaf0d67640 10 monclient(hunting): _renew_subs 2024-09-16T04:00:57.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.200+0000 7fcaf0d67640 10 monclient(hunting): authenticate will time out at 2104.173342s 2024-09-16T04:00:57.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.200+0000 7fcaead9d640 1 -- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fcaec137540 msgr2=0x7fcaec137990 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T04:00:57.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.200+0000 7fcaead9d640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fcaec137540 0x7fcaec137990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:00:57.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.200+0000 7fcae9d9b640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fcaec0a3d90 0x7fcaec137000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:00:57.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.200+0000 7fcae9d9b640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fcaec0a3d90 0x7fcaec137000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:44162/0 (socket says 172.21.15.42:44162) 2024-09-16T04:00:57.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.200+0000 7fcae9d9b640 1 -- 172.21.15.42:0/1303753199 learned_addr learned my addr 172.21.15.42:0/1303753199 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:00:57.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.200+0000 7fcae9d9b640 10 monclient(hunting): get_auth_request con 0x7fcaec0a3d90 auth_method 0 2024-09-16T04:00:57.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.200+0000 7fcae9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:00:57.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.200+0000 7fcae9d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:00:57.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.200+0000 7fcae9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:00:57.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.200+0000 7fcae9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:00:57.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.200+0000 7fcae9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:00:57.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.200+0000 7fcae9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:00:57.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.200+0000 7fcaea59c640 1 --2- 172.21.15.42:0/1303753199 >> v2:172.21.15.154:3300/0 conn(0x7fcaec0a39b0 0x7fcaec133b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:00:57.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.200+0000 7fcaea59c640 10 monclient(hunting): get_auth_request con 0x7fcaec0a39b0 auth_method 0 2024-09-16T04:00:57.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.200+0000 7fcaea59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:00:57.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.200+0000 7fcaea59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:00:57.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.200+0000 7fcaea59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:00:57.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.200+0000 7fcae9d9b640 10 monclient(hunting): handle_auth_done global_id 825315 payload 274 2024-09-16T04:00:57.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.200+0000 7fcae9d9b640 1 -- 172.21.15.42:0/1303753199 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fcaec137540 msgr2=0x7fcaec137990 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:00:57.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.200+0000 7fcae9d9b640 1 --2- 172.21.15.42:0/1303753199 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fcaec137540 0x7fcaec137990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:00:57.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.200+0000 7fcae9d9b640 1 -- 172.21.15.42:0/1303753199 >> v2:172.21.15.154:3300/0 conn(0x7fcaec0a39b0 msgr2=0x7fcaec133b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:00:57.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.200+0000 7fcae9d9b640 1 --2- 172.21.15.42:0/1303753199 >> v2:172.21.15.154:3300/0 conn(0x7fcaec0a39b0 0x7fcaec133b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:00:57.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.200+0000 7fcae9d9b640 10 monclient: _finish_hunting 0 2024-09-16T04:00:57.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.200+0000 7fcae9d9b640 1 monclient: found mon.noname-a 2024-09-16T04:00:57.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.200+0000 7fcae9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:00:57.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.200+0000 7fcae9d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:00:57.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.200+0000 7fcae9d9b640 1 -- 172.21.15.42:0/1303753199 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcaec0a5e20 con 0x7fcaec0a3d90 2024-09-16T04:00:57.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.204+0000 7fcae9d9b640 1 --2- 172.21.15.42:0/1303753199 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcaec0a3d90 0x7fcaec137000 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fcaec0a7750 tx=0x7fcad401c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f7a7bfb46dbf854 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:00:57.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.204+0000 7fcae959a640 1 -- 172.21.15.42:0/1303753199 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcad40b0070 con 0x7fcaec0a3d90 2024-09-16T04:00:57.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.204+0000 7fcae959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:00:57.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.204+0000 7fcae959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:00:57.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.204+0000 7fcae959a640 10 monclient: dump: 2024-09-16T04:00:57.218 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:00:57.218 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:00:57.218 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:00:57.218 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:00:57.218 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:00:57.218 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:00:57.218 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:00:57.218 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:00:57.218 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:00:57.218 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:00:57.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.204+0000 7fcae959a640 10 monclient: _finish_auth 0 2024-09-16T04:00:57.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.204+0000 7fcae959a640 10 monclient: _check_auth_tickets 2024-09-16T04:00:57.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.204+0000 7fcae959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:00:57.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.204+0000 7fcaf0d67640 5 monclient: authenticate success, global_id 825315 2024-09-16T04:00:57.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.204+0000 7fcaf0d67640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:00:57.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.204+0000 7fcae959a640 1 -- 172.21.15.42:0/1303753199 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcad40a4e50 con 0x7fcaec0a3d90 2024-09-16T04:00:57.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.204+0000 7fcae959a640 10 monclient: handle_config config(0 keys) 2024-09-16T04:00:57.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.204+0000 7fcae959a640 1 -- 172.21.15.42:0/1303753199 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcad40ab820 con 0x7fcaec0a3d90 2024-09-16T04:00:57.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.204+0000 7fcae959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:00:57.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.204+0000 7fcae959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:00:57.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.204+0000 7fcae959a640 10 monclient: dump: 2024-09-16T04:00:57.219 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:00:57.219 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:00:57.220 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:00:57.220 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:00:57.220 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:00:57.220 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:00:57.220 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:00:57.220 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:00:57.220 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:00:57.220 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:00:57.220 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.204+0000 7fcaf0d67640 10 monclient: get_monmap_and_config success 2024-09-16T04:00:57.220 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.204+0000 7fcaf0d67640 4 set_mon_vals no callback set 2024-09-16T04:00:57.220 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.204+0000 7fcaf0d67640 10 monclient: shutdown 2024-09-16T04:00:57.220 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.204+0000 7fcaf0d67640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:00:57.220 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.204+0000 7fcaf0d67640 1 -- 172.21.15.42:0/1303753199 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcaec0a3d90 msgr2=0x7fcaec137000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:00:57.220 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.204+0000 7fcaf0d67640 1 --2- 172.21.15.42:0/1303753199 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcaec0a3d90 0x7fcaec137000 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fcaec0a7750 tx=0x7fcad401c400 comp rx=0 tx=0).stop 2024-09-16T04:00:57.221 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.204+0000 7fcaf0d67640 1 -- 172.21.15.42:0/1303753199 shutdown_connections 2024-09-16T04:00:57.221 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.204+0000 7fcaf0d67640 1 --2- 172.21.15.42:0/1303753199 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fcaec137540 0x7fcaec137990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:00:57.221 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.204+0000 7fcaf0d67640 1 --2- 172.21.15.42:0/1303753199 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcaec0a3d90 0x7fcaec137000 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:00:57.221 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.204+0000 7fcaf0d67640 1 --2- 172.21.15.42:0/1303753199 >> v2:172.21.15.154:3300/0 conn(0x7fcaec0a39b0 0x7fcaec133b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:00:57.221 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.204+0000 7fcaf0d67640 1 -- 172.21.15.42:0/1303753199 >> 172.21.15.42:0/1303753199 conn(0x7fcaec0a9fb0 msgr2=0x7fcaec0aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:00:57.221 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.204+0000 7fcaf0d67640 1 -- 172.21.15.42:0/1303753199 shutdown_connections 2024-09-16T04:00:57.221 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.204+0000 7fcaf0d67640 1 -- 172.21.15.42:0/1303753199 wait complete. 2024-09-16T04:00:57.221 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.204+0000 7fcaf0d67640 10 monclient: build_initial_monmap 2024-09-16T04:00:57.221 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.204+0000 7fcaf0d67640 10 monclient: monmap: 2024-09-16T04:00:57.221 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:00:57.221 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:00:57.221 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:00:57.221 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:00:57.221 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:00:57.222 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:00:57.222 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:00:57.222 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:00:57.222 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:00:57.222 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:00:57.222 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.204+0000 7fcaf0d67640 1 librados: starting msgr at 2024-09-16T04:00:57.222 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.204+0000 7fcaf0d67640 1 librados: starting objecter 2024-09-16T04:00:57.223 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.204+0000 7fcaf0d67640 1 Processor -- start 2024-09-16T04:00:57.223 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.204+0000 7fcaf0d67640 1 -- start start 2024-09-16T04:00:57.223 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.204+0000 7fcaf0d67640 1 librados: setting wanted keys 2024-09-16T04:00:57.223 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.204+0000 7fcaf0d67640 1 librados: calling monclient init 2024-09-16T04:00:57.223 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.204+0000 7fcaf0d67640 10 monclient: init 2024-09-16T04:00:57.223 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.204+0000 7fcaf0d67640 10 monclient: _reopen_session rank -1 2024-09-16T04:00:57.223 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.204+0000 7fcaf0d67640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:00:57.223 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.204+0000 7fcaf0d67640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcaec0a39b0 0x7fcaec0a2d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:00:57.223 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.204+0000 7fcaf0d67640 10 monclient(hunting): picked mon.noname-a con 0x7fcaec0a39b0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:00:57.223 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.204+0000 7fcaf0d67640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcaec0a3d90 0x7fcaec0a3270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:00:57.223 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.204+0000 7fcaf0d67640 10 monclient(hunting): picked mon.noname-b con 0x7fcaec0a3d90 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:00:57.224 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.204+0000 7fcaf0d67640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fcaec137540 0x7fcaec09c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:00:57.224 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.204+0000 7fcaf0d67640 10 monclient(hunting): picked mon.noname-c con 0x7fcaec137540 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:00:57.224 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.204+0000 7fcaf0d67640 10 monclient(hunting): start opening mon connection 2024-09-16T04:00:57.224 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.204+0000 7fcaf0d67640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fcaec140dc0 con 0x7fcaec0a39b0 2024-09-16T04:00:57.224 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.204+0000 7fcaf0d67640 10 monclient(hunting): start opening mon connection 2024-09-16T04:00:57.224 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.204+0000 7fcaf0d67640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fcaec140f40 con 0x7fcaec0a3d90 2024-09-16T04:00:57.224 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.204+0000 7fcaf0d67640 10 monclient(hunting): start opening mon connection 2024-09-16T04:00:57.224 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.204+0000 7fcaf0d67640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fcaec06f6f0 con 0x7fcaec137540 2024-09-16T04:00:57.224 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.204+0000 7fcaf0d67640 10 monclient(hunting): _renew_subs 2024-09-16T04:00:57.224 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.204+0000 7fcaf0d67640 10 monclient(hunting): authenticate will time out at 2104.178918s 2024-09-16T04:00:57.224 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.204+0000 7fcaead9d640 1 -- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fcaec137540 msgr2=0x7fcaec09c8d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T04:00:57.224 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.204+0000 7fcaead9d640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fcaec137540 0x7fcaec09c8d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:00:57.225 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.204+0000 7fcaea59c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcaec0a39b0 0x7fcaec0a2d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:00:57.225 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.204+0000 7fcaea59c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcaec0a39b0 0x7fcaec0a2d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:44170/0 (socket says 172.21.15.42:44170) 2024-09-16T04:00:57.225 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.204+0000 7fcaea59c640 1 -- 172.21.15.42:0/3134399147 learned_addr learned my addr 172.21.15.42:0/3134399147 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:00:57.225 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.204+0000 7fcaea59c640 10 monclient(hunting): get_auth_request con 0x7fcaec0a39b0 auth_method 0 2024-09-16T04:00:57.225 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.204+0000 7fcaea59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:00:57.225 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.204+0000 7fcaea59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:00:57.225 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.204+0000 7fcaea59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:00:57.225 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.204+0000 7fcaea59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:00:57.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.204+0000 7fcaea59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:00:57.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.204+0000 7fcae9d9b640 1 --2- 172.21.15.42:0/3134399147 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcaec0a3d90 0x7fcaec0a3270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:00:57.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.204+0000 7fcaea59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:00:57.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.204+0000 7fcae9d9b640 10 monclient(hunting): get_auth_request con 0x7fcaec0a3d90 auth_method 0 2024-09-16T04:00:57.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.204+0000 7fcae9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:00:57.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.204+0000 7fcae9d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:00:57.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.204+0000 7fcae9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:00:57.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.204+0000 7fcaea59c640 10 monclient(hunting): handle_auth_done global_id 825318 payload 824 2024-09-16T04:00:57.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.204+0000 7fcaea59c640 1 -- 172.21.15.42:0/3134399147 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fcaec137540 msgr2=0x7fcaec09c8d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:00:57.227 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.204+0000 7fcaea59c640 1 --2- 172.21.15.42:0/3134399147 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fcaec137540 0x7fcaec09c8d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:00:57.227 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.204+0000 7fcaea59c640 1 -- 172.21.15.42:0/3134399147 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcaec0a3d90 msgr2=0x7fcaec0a3270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:00:57.227 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.204+0000 7fcaea59c640 1 --2- 172.21.15.42:0/3134399147 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcaec0a3d90 0x7fcaec0a3270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:00:57.227 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.204+0000 7fcaea59c640 10 monclient: _finish_hunting 0 2024-09-16T04:00:57.227 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.204+0000 7fcaea59c640 1 monclient: found mon.noname-a 2024-09-16T04:00:57.227 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.204+0000 7fcaea59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:00:57.227 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.204+0000 7fcaea59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:00:57.227 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.204+0000 7fcaea59c640 1 -- 172.21.15.42:0/3134399147 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcaec09ef40 con 0x7fcaec0a39b0 2024-09-16T04:00:57.227 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.204+0000 7fcae9d9b640 1 --2- 172.21.15.42:0/3134399147 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcaec0a3d90 0x7fcaec0a3270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T04:00:57.227 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.204+0000 7fcaea59c640 1 --2- 172.21.15.42:0/3134399147 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcaec0a39b0 0x7fcaec0a2d30 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fcadc00d800 tx=0x7fcadc00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:00:57.227 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.204+0000 7fcae2ffd640 10 client.?.objecter ms_handle_connect 0x7fcaec0a39b0 2024-09-16T04:00:57.227 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.204+0000 7fcae2ffd640 10 client.?.objecter resend_mon_ops 2024-09-16T04:00:57.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.204+0000 7fcae2ffd640 1 -- 172.21.15.42:0/3134399147 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcadc00b6c0 con 0x7fcaec0a39b0 2024-09-16T04:00:57.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.204+0000 7fcae2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:00:57.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.204+0000 7fcae2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:00:57.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.204+0000 7fcae2ffd640 10 monclient: dump: 2024-09-16T04:00:57.228 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:00:57.228 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:00:57.228 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:00:57.228 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:00:57.228 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:00:57.228 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:00:57.228 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:00:57.228 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:00:57.228 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:00:57.228 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:00:57.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.204+0000 7fcae2ffd640 10 monclient: _finish_auth 0 2024-09-16T04:00:57.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.204+0000 7fcae2ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:00:57.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.204+0000 7fcae2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:00:57.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.204+0000 7fcae2ffd640 1 -- 172.21.15.42:0/3134399147 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcadc019070 con 0x7fcaec0a39b0 2024-09-16T04:00:57.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.204+0000 7fcaf0d67640 5 monclient: authenticate success, global_id 825318 2024-09-16T04:00:57.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.204+0000 7fcaf0d67640 10 monclient: _renew_subs 2024-09-16T04:00:57.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.204+0000 7fcaf0d67640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:00:57.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.204+0000 7fcaf0d67640 1 -- 172.21.15.42:0/3134399147 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcaec09f220 con 0x7fcaec0a39b0 2024-09-16T04:00:57.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.204+0000 7fcae2ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:00:57.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.204+0000 7fcae2ffd640 1 -- 172.21.15.42:0/3134399147 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcadc00ba60 con 0x7fcaec0a39b0 2024-09-16T04:00:57.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.204+0000 7fcae2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:00:57.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.208+0000 7fcae2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:00:57.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.208+0000 7fcae2ffd640 10 monclient: dump: 2024-09-16T04:00:57.229 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:00:57.229 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:00:57.230 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:00:57.230 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:00:57.230 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:00:57.230 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:00:57.230 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:00:57.230 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:00:57.230 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:00:57.230 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:00:57.230 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.204+0000 7fcae3fff640 4 set_mon_vals no callback set 2024-09-16T04:00:57.230 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.208+0000 7fcaf0d67640 10 client.825318.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:00:57.230 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.208+0000 7fcaf0d67640 10 monclient: _renew_subs 2024-09-16T04:00:57.230 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.208+0000 7fcaf0d67640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:00:57.230 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.208+0000 7fcaf0d67640 1 -- 172.21.15.42:0/3134399147 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcaec1cf640 con 0x7fcaec0a39b0 2024-09-16T04:00:57.230 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.208+0000 7fcaf0d67640 1 librados: init done 2024-09-16T04:00:57.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.208+0000 7fcaf0d67640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:00:57.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.208+0000 7fcaf0d67640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:00:57.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.208+0000 7fcaf0d67640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:00:57.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.208+0000 7fcaf0d67640 1 -- 172.21.15.42:0/3134399147 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcaec137000 con 0x7fcaec0a39b0 2024-09-16T04:00:57.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.208+0000 7fcae2ffd640 1 -- 172.21.15.42:0/3134399147 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fcadc00bc00 con 0x7fcaec0a39b0 2024-09-16T04:00:57.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.208+0000 7fcae2ffd640 1 --2- 172.21.15.42:0/3134399147 >> v2:172.21.15.154:6812/254845342 conn(0x7fcacc0377c0 0x7fcacc039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:00:57.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.208+0000 7fcae2ffd640 1 -- 172.21.15.42:0/3134399147 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(569..569 src has 451..569) ==== 6216+0+0 (secure 0 0 0) 0x7fcadc049590 con 0x7fcaec0a39b0 2024-09-16T04:00:57.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.208+0000 7fcae2ffd640 10 client.825318.objecter ms_dispatch 0x7fcaec000ba0 osd_map(569..569 src has 451..569) 2024-09-16T04:00:57.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.208+0000 7fcae2ffd640 3 client.825318.objecter handle_osd_map got epochs [569,569] > 0 2024-09-16T04:00:57.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.208+0000 7fcae2ffd640 3 client.825318.objecter handle_osd_map decoding full epoch 569 2024-09-16T04:00:57.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.208+0000 7fcae2ffd640 20 client.825318.objecter dump_active .. 0 homeless 2024-09-16T04:00:57.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.208+0000 7fcae9d9b640 1 --2- 172.21.15.42:0/3134399147 >> v2:172.21.15.154:6812/254845342 conn(0x7fcacc0377c0 0x7fcacc039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:00:57.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.208+0000 7fcae9d9b640 10 monclient: get_auth_request con 0x7fcacc0377c0 auth_method 0 2024-09-16T04:00:57.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.208+0000 7fcae9d9b640 1 --2- 172.21.15.42:0/3134399147 >> v2:172.21.15.154:6812/254845342 conn(0x7fcacc0377c0 0x7fcacc039c80 secure :-1 s=READY pgs=3754 cs=0 l=1 rev1=1 crypto rx=0x7fcad40ad000 tx=0x7fcad4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:00:57.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.208+0000 7fcae2ffd640 10 client.825318.objecter ms_handle_connect 0x7fcacc0377c0 2024-09-16T04:00:57.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.212+0000 7fcae2ffd640 1 -- 172.21.15.42:0/3134399147 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7fcadc01d580 con 0x7fcaec0a39b0 2024-09-16T04:00:57.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.212+0000 7fcae2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:00:57.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.212+0000 7fcae2ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:00:57.442 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.436+0000 7fcaf0d67640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:00:57.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.436+0000 7fcaf0d67640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:00:57.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.436+0000 7fcaf0d67640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:00:57.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.436+0000 7fcaf0d67640 1 -- 172.21.15.42:0/3134399147 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fcaec09f740 con 0x7fcaec0a39b0 2024-09-16T04:00:57.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.436+0000 7fcae2ffd640 1 -- 172.21.15.42:0/3134399147 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7fcadc04b6e0 con 0x7fcaec0a39b0 2024-09-16T04:00:57.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.436+0000 7fcae2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:00:57.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.436+0000 7fcae2ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:00:57.444 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":168,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:00:57.446 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.436+0000 7fcaf0d67640 10 librados: watch_flush enter 2024-09-16T04:00:57.446 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.440+0000 7fcaf0d67640 10 librados: watch_flush exit 2024-09-16T04:00:57.446 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.440+0000 7fcaf0d67640 20 client.825318.objecter shutdown clearing up homeless session... 2024-09-16T04:00:57.446 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.440+0000 7fcaf0d67640 10 client.825318.objecter successfully canceled tick 2024-09-16T04:00:57.446 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.440+0000 7fcaf0d67640 1 -- 172.21.15.42:0/3134399147 >> v2:172.21.15.154:6812/254845342 conn(0x7fcacc0377c0 msgr2=0x7fcacc039c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:00:57.446 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.440+0000 7fcaf0d67640 1 --2- 172.21.15.42:0/3134399147 >> v2:172.21.15.154:6812/254845342 conn(0x7fcacc0377c0 0x7fcacc039c80 secure :-1 s=READY pgs=3754 cs=0 l=1 rev1=1 crypto rx=0x7fcad40ad000 tx=0x7fcad4008040 comp rx=0 tx=0).stop 2024-09-16T04:00:57.446 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.440+0000 7fcaf0d67640 10 monclient: shutdown 2024-09-16T04:00:57.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.440+0000 7fcaf0d67640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:00:57.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.440+0000 7fcaf0d67640 1 -- 172.21.15.42:0/3134399147 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcaec0a39b0 msgr2=0x7fcaec0a2d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:00:57.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.440+0000 7fcaf0d67640 1 --2- 172.21.15.42:0/3134399147 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcaec0a39b0 0x7fcaec0a2d30 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fcadc00d800 tx=0x7fcadc00dcc0 comp rx=0 tx=0).stop 2024-09-16T04:00:57.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.440+0000 7fcaf0d67640 1 -- 172.21.15.42:0/3134399147 shutdown_connections 2024-09-16T04:00:57.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.440+0000 7fcaf0d67640 1 --2- 172.21.15.42:0/3134399147 >> v2:172.21.15.154:6812/254845342 conn(0x7fcacc0377c0 0x7fcacc039c80 unknown :-1 s=CLOSED pgs=3754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:00:57.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.440+0000 7fcaf0d67640 1 --2- 172.21.15.42:0/3134399147 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fcaec137540 0x7fcaec09c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:00:57.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.440+0000 7fcaf0d67640 1 --2- 172.21.15.42:0/3134399147 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fcaec0a3d90 0x7fcaec0a3270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:00:57.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.440+0000 7fcaf0d67640 1 --2- 172.21.15.42:0/3134399147 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fcaec0a39b0 0x7fcaec0a2d30 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:00:57.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.440+0000 7fcaf0d67640 1 -- 172.21.15.42:0/3134399147 >> 172.21.15.42:0/3134399147 conn(0x7fcaec0a9fb0 msgr2=0x7fcaec0a8730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:00:57.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.440+0000 7fcaf0d67640 1 -- 172.21.15.42:0/3134399147 shutdown_connections 2024-09-16T04:00:57.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.440+0000 7fcaf0d67640 1 -- 172.21.15.42:0/3134399147 wait complete. 2024-09-16T04:00:57.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:00:57.440+0000 7fcaf0d67640 1 librados: shutdown 2024-09-16T04:01:00.462 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:01:00.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.588+0000 7f4b8871e640 10 monclient: get_monmap_and_config 2024-09-16T04:01:00.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.588+0000 7f4b8871e640 10 monclient: build_initial_monmap 2024-09-16T04:01:00.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.588+0000 7f4b8871e640 10 monclient: monmap: 2024-09-16T04:01:00.595 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:01:00.595 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:01:00.595 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:01:00.593820+0000 2024-09-16T04:01:00.595 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:01:00.593820+0000 2024-09-16T04:01:00.596 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:01:00.596 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:01:00.596 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:01:00.596 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:01:00.596 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:01:00.596 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:01:00.596 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:01:00.596 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:01:00.598 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.588+0000 7f4b8871e640 1 Processor -- start 2024-09-16T04:01:00.598 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.588+0000 7f4b8871e640 1 -- start start 2024-09-16T04:01:00.598 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.588+0000 7f4b8871e640 10 monclient: init 2024-09-16T04:01:00.599 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.588+0000 7f4b8871e640 10 monclient: _reopen_session rank -1 2024-09-16T04:01:00.599 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.588+0000 7f4b8871e640 10 monclient: _add_conns ranks=[1,3,0,2,4] 2024-09-16T04:01:00.599 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.588+0000 7f4b8871e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4b800a1330 addr v1:172.21.15.154:6789/0 2024-09-16T04:01:00.599 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.588+0000 7f4b8871e640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f4b8006dac0 0x7f4b8013a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:01:00.599 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.588+0000 7f4b8871e640 10 monclient(hunting): picked mon.noname-a con 0x7f4b8006dac0 addr v2:172.21.15.42:3300/0 2024-09-16T04:01:00.599 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.588+0000 7f4b8871e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4b800a1e20 addr v1:172.21.15.42:6789/0 2024-09-16T04:01:00.599 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.588+0000 7f4b8871e640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4b8009c090 con 0x7f4b800a1e20 2024-09-16T04:01:00.599 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.588+0000 7f4b8871e640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4b8009f610 con 0x7f4b800a1330 2024-09-16T04:01:00.599 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.588+0000 7f4b8871e640 10 monclient(hunting): start opening mon connection 2024-09-16T04:01:00.599 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.588+0000 7f4b8871e640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f4b8009f8f0 con 0x7f4b8006dac0 2024-09-16T04:01:00.599 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.588+0000 7f4b8871e640 10 monclient(hunting): _renew_subs 2024-09-16T04:01:00.599 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.588+0000 7f4b8871e640 10 monclient(hunting): authenticate will time out at 2107.563243s 2024-09-16T04:01:00.599 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.588+0000 7f4b86cbb640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f4b800a1e20 0x7f4b8013c890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:56236/0 (socket says 172.21.15.42:56236) 2024-09-16T04:01:00.600 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.588+0000 7f4b86cbb640 1 -- 172.21.15.42:0/3000577658 learned_addr learned my addr 172.21.15.42:0/3000577658 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T04:01:00.600 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.588+0000 7f4b85cb9640 1 --2- 172.21.15.42:0/3000577658 >> v2:172.21.15.42:3300/0 conn(0x7f4b8006dac0 0x7f4b8013a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:01:00.600 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.588+0000 7f4b85cb9640 10 monclient(hunting): get_auth_request con 0x7f4b8006dac0 auth_method 0 2024-09-16T04:01:00.600 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.588+0000 7f4b85cb9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:01:00.600 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.588+0000 7f4b85cb9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:01:00.600 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.588+0000 7f4b85cb9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:01:00.600 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.588+0000 7f4b85cb9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:01:00.600 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.588+0000 7f4b85cb9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:01:00.600 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.588+0000 7f4b85cb9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:01:00.600 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.588+0000 7f4b854b8640 1 -- 172.21.15.42:0/3000577658 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 801294633 0 0) 0x7f4b8009c090 con 0x7f4b800a1e20 2024-09-16T04:01:00.601 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.588+0000 7f4b854b8640 10 monclient(hunting): _negotiate 2024-09-16T04:01:00.601 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.588+0000 7f4b854b8640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:01:00.601 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.588+0000 7f4b854b8640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:01:00.601 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.588+0000 7f4b854b8640 10 monclient(hunting): my global_id is 825324 2024-09-16T04:01:00.601 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.588+0000 7f4b854b8640 1 -- 172.21.15.42:0/3000577658 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4b6c003530 con 0x7f4b800a1e20 2024-09-16T04:01:00.601 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.588+0000 7f4b85cb9640 10 monclient(hunting): handle_auth_done global_id 825321 payload 274 2024-09-16T04:01:00.601 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.588+0000 7f4b85cb9640 1 -- 172.21.15.42:0/3000577658 >> v1:172.21.15.154:6789/0 conn(0x7f4b800a1330 legacy=0x7f4b800a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:01:00.601 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.588+0000 7f4b85cb9640 1 -- 172.21.15.42:0/3000577658 >> v1:172.21.15.42:6789/0 conn(0x7f4b800a1e20 legacy=0x7f4b8013c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:01:00.601 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.588+0000 7f4b85cb9640 10 monclient: _finish_hunting 0 2024-09-16T04:01:00.601 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.588+0000 7f4b85cb9640 1 monclient: found mon.noname-a 2024-09-16T04:01:00.601 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.588+0000 7f4b85cb9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:01:00.601 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.588+0000 7f4b85cb9640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:01:00.601 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.588+0000 7f4b85cb9640 1 -- 172.21.15.42:0/3000577658 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b8009fbd0 con 0x7f4b8006dac0 2024-09-16T04:01:00.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.592+0000 7f4b85cb9640 1 --2- 172.21.15.42:0/3000577658 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4b8006dac0 0x7f4b8013a340 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f4b7c0097c0 tx=0x7f4b7c01c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9e1ccc92713cf24 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:01:00.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.592+0000 7f4b854b8640 1 -- 172.21.15.42:0/3000577658 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b7c0b0070 con 0x7f4b8006dac0 2024-09-16T04:01:00.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.592+0000 7f4b854b8640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:01:00.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.592+0000 7f4b854b8640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:01:00.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.592+0000 7f4b854b8640 10 monclient: dump: 2024-09-16T04:01:00.607 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:01:00.607 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:01:00.607 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:00.607 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:00.607 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:01:00.608 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:01:00.608 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:01:00.608 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:01:00.608 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:01:00.608 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:01:00.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.592+0000 7f4b854b8640 10 monclient: _finish_auth 0 2024-09-16T04:01:00.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.592+0000 7f4b854b8640 10 monclient: _check_auth_tickets 2024-09-16T04:01:00.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.592+0000 7f4b854b8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:01:00.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.592+0000 7f4b854b8640 1 -- 172.21.15.42:0/3000577658 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4b7c0ab600 con 0x7f4b8006dac0 2024-09-16T04:01:00.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.592+0000 7f4b854b8640 10 monclient: handle_config config(0 keys) 2024-09-16T04:01:00.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.592+0000 7f4b854b8640 1 -- 172.21.15.42:0/3000577658 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b7c0ab9d0 con 0x7f4b8006dac0 2024-09-16T04:01:00.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.592+0000 7f4b8871e640 5 monclient: authenticate success, global_id 825321 2024-09-16T04:01:00.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.592+0000 7f4b8871e640 10 monclient: get_monmap_and_config success 2024-09-16T04:01:00.609 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.592+0000 7f4b8871e640 4 set_mon_vals no callback set 2024-09-16T04:01:00.609 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.592+0000 7f4b854b8640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:01:00.609 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.592+0000 7f4b854b8640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:01:00.609 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.592+0000 7f4b854b8640 10 monclient: dump: 2024-09-16T04:01:00.609 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:01:00.609 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:01:00.609 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:00.609 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:00.609 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:01:00.609 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:01:00.609 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:01:00.609 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:01:00.609 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:01:00.610 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:01:00.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.592+0000 7f4b8871e640 10 monclient: shutdown 2024-09-16T04:01:00.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.592+0000 7f4b8871e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:01:00.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.592+0000 7f4b8871e640 1 -- 172.21.15.42:0/3000577658 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4b8006dac0 msgr2=0x7f4b8013a340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:01:00.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.592+0000 7f4b8871e640 1 --2- 172.21.15.42:0/3000577658 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4b8006dac0 0x7f4b8013a340 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f4b7c0097c0 tx=0x7f4b7c01c480 comp rx=0 tx=0).stop 2024-09-16T04:01:00.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.592+0000 7f4b8871e640 1 -- 172.21.15.42:0/3000577658 shutdown_connections 2024-09-16T04:01:00.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.592+0000 7f4b8871e640 1 --2- 172.21.15.42:0/3000577658 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4b8006dac0 0x7f4b8013a340 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:00.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.592+0000 7f4b8871e640 1 -- 172.21.15.42:0/3000577658 >> 172.21.15.42:0/3000577658 conn(0x7f4b800a2420 msgr2=0x7f4b800a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:01:00.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.592+0000 7f4b8871e640 1 -- 172.21.15.42:0/3000577658 shutdown_connections 2024-09-16T04:01:00.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.592+0000 7f4b8871e640 1 -- 172.21.15.42:0/3000577658 wait complete. 2024-09-16T04:01:00.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.592+0000 7f4b8871e640 10 monclient: build_initial_monmap 2024-09-16T04:01:00.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.592+0000 7f4b8871e640 10 monclient: monmap: 2024-09-16T04:01:00.610 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:01:00.610 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:01:00.611 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:01:00.611 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:01:00.611 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:01:00.611 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:01:00.611 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:01:00.611 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:01:00.611 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:01:00.611 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:01:00.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.596+0000 7f4b8871e640 1 librados: starting msgr at 2024-09-16T04:01:00.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.596+0000 7f4b8871e640 1 librados: starting objecter 2024-09-16T04:01:00.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.596+0000 7f4b8871e640 1 Processor -- start 2024-09-16T04:01:00.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.596+0000 7f4b8871e640 1 -- start start 2024-09-16T04:01:00.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.596+0000 7f4b8871e640 1 librados: setting wanted keys 2024-09-16T04:01:00.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.596+0000 7f4b8871e640 1 librados: calling monclient init 2024-09-16T04:01:00.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.596+0000 7f4b8871e640 10 monclient: init 2024-09-16T04:01:00.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.596+0000 7f4b8871e640 10 monclient: _reopen_session rank -1 2024-09-16T04:01:00.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.596+0000 7f4b8871e640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:01:00.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.596+0000 7f4b8871e640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4b800a1330 0x7f4b8013c040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:01:00.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.596+0000 7f4b8871e640 10 monclient(hunting): picked mon.noname-c con 0x7f4b800a1330 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:01:00.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.596+0000 7f4b8871e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4b800a1e20 0x7f4b801351f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:01:00.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.596+0000 7f4b8871e640 10 monclient(hunting): picked mon.noname-b con 0x7f4b800a1e20 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:01:00.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.596+0000 7f4b8871e640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4b8011cf40 0x7f4b801ccdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:01:00.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.596+0000 7f4b8871e640 10 monclient(hunting): picked mon.noname-a con 0x7f4b8011cf40 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:01:00.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.596+0000 7f4b8871e640 10 monclient(hunting): start opening mon connection 2024-09-16T04:01:00.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.596+0000 7f4b8871e640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f4b801409e0 con 0x7f4b8011cf40 2024-09-16T04:01:00.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.596+0000 7f4b8871e640 10 monclient(hunting): start opening mon connection 2024-09-16T04:01:00.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.596+0000 7f4b8871e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f4b8009efb0 con 0x7f4b800a1e20 2024-09-16T04:01:00.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.596+0000 7f4b8871e640 10 monclient(hunting): start opening mon connection 2024-09-16T04:01:00.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.596+0000 7f4b8871e640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f4b80140b60 con 0x7f4b800a1330 2024-09-16T04:01:00.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.596+0000 7f4b8871e640 10 monclient(hunting): _renew_subs 2024-09-16T04:01:00.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.596+0000 7f4b8871e640 10 monclient(hunting): authenticate will time out at 2107.568686s 2024-09-16T04:01:00.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.596+0000 7f4b864ba640 1 -- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4b800a1330 msgr2=0x7f4b8013c040 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T04:01:00.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.596+0000 7f4b864ba640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4b800a1330 0x7f4b8013c040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:01:00.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.596+0000 7f4b86cbb640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4b8011cf40 0x7f4b801ccdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:01:00.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.596+0000 7f4b86cbb640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4b8011cf40 0x7f4b801ccdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:44194/0 (socket says 172.21.15.42:44194) 2024-09-16T04:01:00.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.596+0000 7f4b86cbb640 1 -- 172.21.15.42:0/1359136162 learned_addr learned my addr 172.21.15.42:0/1359136162 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:01:00.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.596+0000 7f4b86cbb640 10 monclient(hunting): get_auth_request con 0x7f4b8011cf40 auth_method 0 2024-09-16T04:01:00.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.596+0000 7f4b86cbb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:01:00.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.596+0000 7f4b86cbb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:01:00.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.596+0000 7f4b86cbb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:01:00.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.596+0000 7f4b86cbb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:01:00.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.596+0000 7f4b86cbb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:01:00.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.596+0000 7f4b86cbb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:01:00.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.596+0000 7f4b85cb9640 1 --2- 172.21.15.42:0/1359136162 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4b800a1e20 0x7f4b801351f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:01:00.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.596+0000 7f4b85cb9640 10 monclient(hunting): get_auth_request con 0x7f4b800a1e20 auth_method 0 2024-09-16T04:01:00.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.596+0000 7f4b85cb9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:01:00.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.596+0000 7f4b85cb9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:01:00.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.596+0000 7f4b85cb9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:01:00.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.596+0000 7f4b86cbb640 10 monclient(hunting): handle_auth_done global_id 825327 payload 824 2024-09-16T04:01:00.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.596+0000 7f4b86cbb640 1 -- 172.21.15.42:0/1359136162 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4b800a1330 msgr2=0x7f4b8013c040 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:01:00.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.596+0000 7f4b86cbb640 1 --2- 172.21.15.42:0/1359136162 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4b800a1330 0x7f4b8013c040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:00.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.596+0000 7f4b86cbb640 1 -- 172.21.15.42:0/1359136162 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4b800a1e20 msgr2=0x7f4b801351f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:01:00.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.596+0000 7f4b86cbb640 1 --2- 172.21.15.42:0/1359136162 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4b800a1e20 0x7f4b801351f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:00.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.596+0000 7f4b86cbb640 10 monclient: _finish_hunting 0 2024-09-16T04:01:00.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.596+0000 7f4b86cbb640 1 monclient: found mon.noname-a 2024-09-16T04:01:00.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.596+0000 7f4b86cbb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:01:00.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.596+0000 7f4b86cbb640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:01:00.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.596+0000 7f4b86cbb640 1 -- 172.21.15.42:0/1359136162 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b8011e890 con 0x7f4b8011cf40 2024-09-16T04:01:00.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.596+0000 7f4b86cbb640 1 --2- 172.21.15.42:0/1359136162 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4b8011cf40 0x7f4b801ccdb0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f4b7800cb40 tx=0x7f4b7800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:01:00.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.596+0000 7f4b76ffd640 10 client.?.objecter ms_handle_connect 0x7f4b8011cf40 2024-09-16T04:01:00.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.596+0000 7f4b76ffd640 10 client.?.objecter resend_mon_ops 2024-09-16T04:01:00.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.596+0000 7f4b76ffd640 1 -- 172.21.15.42:0/1359136162 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b78017070 con 0x7f4b8011cf40 2024-09-16T04:01:00.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.596+0000 7f4b76ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:01:00.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.596+0000 7f4b76ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:01:00.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.596+0000 7f4b76ffd640 10 monclient: dump: 2024-09-16T04:01:00.616 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:01:00.616 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:01:00.616 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:00.616 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:00.616 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:01:00.616 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:01:00.616 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:01:00.616 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:01:00.616 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:01:00.617 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:01:00.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.596+0000 7f4b76ffd640 10 monclient: _finish_auth 0 2024-09-16T04:01:00.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.596+0000 7f4b76ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:01:00.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.596+0000 7f4b76ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:01:00.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.596+0000 7f4b8871e640 5 monclient: authenticate success, global_id 825327 2024-09-16T04:01:00.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.596+0000 7f4b76ffd640 1 -- 172.21.15.42:0/1359136162 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4b7800ac40 con 0x7f4b8011cf40 2024-09-16T04:01:00.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.596+0000 7f4b76ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:01:00.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.596+0000 7f4b8871e640 10 monclient: _renew_subs 2024-09-16T04:01:00.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.596+0000 7f4b8871e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:01:00.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.596+0000 7f4b76ffd640 1 -- 172.21.15.42:0/1359136162 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b78012620 con 0x7f4b8011cf40 2024-09-16T04:01:00.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.596+0000 7f4b8871e640 1 -- 172.21.15.42:0/1359136162 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b8011eb70 con 0x7f4b8011cf40 2024-09-16T04:01:00.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.596+0000 7f4b77fff640 4 set_mon_vals no callback set 2024-09-16T04:01:00.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.596+0000 7f4b76ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:01:00.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.596+0000 7f4b76ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:01:00.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.596+0000 7f4b76ffd640 10 monclient: dump: 2024-09-16T04:01:00.618 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:01:00.618 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:01:00.618 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:00.618 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:00.618 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:01:00.618 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:01:00.618 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:01:00.618 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:01:00.618 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:01:00.618 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:01:00.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.596+0000 7f4b8871e640 10 client.825327.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:01:00.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.596+0000 7f4b8871e640 10 monclient: _renew_subs 2024-09-16T04:01:00.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.596+0000 7f4b8871e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:01:00.619 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.596+0000 7f4b8871e640 1 -- 172.21.15.42:0/1359136162 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b8011d720 con 0x7f4b8011cf40 2024-09-16T04:01:00.619 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.596+0000 7f4b8871e640 1 librados: init done 2024-09-16T04:01:00.619 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.596+0000 7f4b76ffd640 1 -- 172.21.15.42:0/1359136162 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f4b7801a070 con 0x7f4b8011cf40 2024-09-16T04:01:00.619 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.596+0000 7f4b8871e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:01:00.619 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.596+0000 7f4b8871e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:01:00.619 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.596+0000 7f4b8871e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:01:00.619 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.596+0000 7f4b8871e640 1 -- 172.21.15.42:0/1359136162 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4b800a1710 con 0x7f4b8011cf40 2024-09-16T04:01:00.619 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.596+0000 7f4b76ffd640 1 --2- 172.21.15.42:0/1359136162 >> v2:172.21.15.154:6812/254845342 conn(0x7f4b640377c0 0x7f4b64039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:01:00.619 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.596+0000 7f4b76ffd640 1 -- 172.21.15.42:0/1359136162 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(569..569 src has 451..569) ==== 6216+0+0 (secure 0 0 0) 0x7f4b78048eb0 con 0x7f4b8011cf40 2024-09-16T04:01:00.619 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.596+0000 7f4b76ffd640 10 client.825327.objecter ms_dispatch 0x7f4b80000ba0 osd_map(569..569 src has 451..569) 2024-09-16T04:01:00.619 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.596+0000 7f4b76ffd640 3 client.825327.objecter handle_osd_map got epochs [569,569] > 0 2024-09-16T04:01:00.619 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.596+0000 7f4b76ffd640 3 client.825327.objecter handle_osd_map decoding full epoch 569 2024-09-16T04:01:00.619 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.596+0000 7f4b76ffd640 20 client.825327.objecter dump_active .. 0 homeless 2024-09-16T04:01:00.619 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.596+0000 7f4b864ba640 1 --2- 172.21.15.42:0/1359136162 >> v2:172.21.15.154:6812/254845342 conn(0x7f4b640377c0 0x7f4b64039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:01:00.620 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.596+0000 7f4b864ba640 10 monclient: get_auth_request con 0x7f4b640377c0 auth_method 0 2024-09-16T04:01:00.620 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.596+0000 7f4b76ffd640 10 client.825327.objecter ms_handle_connect 0x7f4b640377c0 2024-09-16T04:01:00.620 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.596+0000 7f4b864ba640 1 --2- 172.21.15.42:0/1359136162 >> v2:172.21.15.154:6812/254845342 conn(0x7f4b640377c0 0x7f4b64039c80 secure :-1 s=READY pgs=3755 cs=0 l=1 rev1=1 crypto rx=0x7f4b8011ed80 tx=0x7f4b70007a00 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:01:00.620 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.600+0000 7f4b76ffd640 1 -- 172.21.15.42:0/1359136162 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f4b78007120 con 0x7f4b8011cf40 2024-09-16T04:01:00.620 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.600+0000 7f4b76ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:01:00.620 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.600+0000 7f4b76ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:01:00.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.828+0000 7f4b8871e640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:01:00.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.828+0000 7f4b8871e640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:01:00.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.828+0000 7f4b8871e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:01:00.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.828+0000 7f4b8871e640 1 -- 172.21.15.42:0/1359136162 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f4b80069040 con 0x7f4b8011cf40 2024-09-16T04:01:00.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.828+0000 7f4b76ffd640 1 -- 172.21.15.42:0/1359136162 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7f4b78011df0 con 0x7f4b8011cf40 2024-09-16T04:01:00.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.828+0000 7f4b76ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:01:00.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.828+0000 7f4b76ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:01:00.837 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":168,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":9,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:01:00.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.832+0000 7f4b8871e640 10 librados: watch_flush enter 2024-09-16T04:01:00.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.832+0000 7f4b8871e640 10 librados: watch_flush exit 2024-09-16T04:01:00.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.832+0000 7f4b8871e640 20 client.825327.objecter shutdown clearing up homeless session... 2024-09-16T04:01:00.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.832+0000 7f4b8871e640 10 client.825327.objecter successfully canceled tick 2024-09-16T04:01:00.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.832+0000 7f4b8871e640 1 -- 172.21.15.42:0/1359136162 >> v2:172.21.15.154:6812/254845342 conn(0x7f4b640377c0 msgr2=0x7f4b64039c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:01:00.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.832+0000 7f4b8871e640 1 --2- 172.21.15.42:0/1359136162 >> v2:172.21.15.154:6812/254845342 conn(0x7f4b640377c0 0x7f4b64039c80 secure :-1 s=READY pgs=3755 cs=0 l=1 rev1=1 crypto rx=0x7f4b8011ed80 tx=0x7f4b70007a00 comp rx=0 tx=0).stop 2024-09-16T04:01:00.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.832+0000 7f4b8871e640 10 monclient: shutdown 2024-09-16T04:01:00.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.832+0000 7f4b8871e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:01:00.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.832+0000 7f4b8871e640 1 -- 172.21.15.42:0/1359136162 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4b8011cf40 msgr2=0x7f4b801ccdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:01:00.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.832+0000 7f4b8871e640 1 --2- 172.21.15.42:0/1359136162 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4b8011cf40 0x7f4b801ccdb0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f4b7800cb40 tx=0x7f4b7800a060 comp rx=0 tx=0).stop 2024-09-16T04:01:00.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.832+0000 7f4b8871e640 1 -- 172.21.15.42:0/1359136162 shutdown_connections 2024-09-16T04:01:00.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.832+0000 7f4b8871e640 1 --2- 172.21.15.42:0/1359136162 >> v2:172.21.15.154:6812/254845342 conn(0x7f4b640377c0 0x7f4b64039c80 unknown :-1 s=CLOSED pgs=3755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:00.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.832+0000 7f4b8871e640 1 --2- 172.21.15.42:0/1359136162 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4b8011cf40 0x7f4b801ccdb0 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:00.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.832+0000 7f4b8871e640 1 --2- 172.21.15.42:0/1359136162 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4b800a1e20 0x7f4b801351f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:00.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.832+0000 7f4b8871e640 1 --2- 172.21.15.42:0/1359136162 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4b800a1330 0x7f4b8013c040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:00.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.832+0000 7f4b8871e640 1 -- 172.21.15.42:0/1359136162 >> 172.21.15.42:0/1359136162 conn(0x7f4b800a2420 msgr2=0x7f4b800a45b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:01:00.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.832+0000 7f4b8871e640 1 -- 172.21.15.42:0/1359136162 shutdown_connections 2024-09-16T04:01:00.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.832+0000 7f4b8871e640 1 -- 172.21.15.42:0/1359136162 wait complete. 2024-09-16T04:01:00.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:00.832+0000 7f4b8871e640 1 librados: shutdown 2024-09-16T04:01:03.857 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:01:03.987 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.980+0000 7fe4df9d2640 10 monclient: get_monmap_and_config 2024-09-16T04:01:03.987 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.980+0000 7fe4df9d2640 10 monclient: build_initial_monmap 2024-09-16T04:01:03.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.980+0000 7fe4df9d2640 10 monclient: monmap: 2024-09-16T04:01:03.988 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:01:03.988 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:01:03.988 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:01:03.986182+0000 2024-09-16T04:01:03.988 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:01:03.986182+0000 2024-09-16T04:01:03.988 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:01:03.988 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:01:03.988 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:01:03.988 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:01:03.988 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:01:03.988 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:01:03.988 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:01:03.989 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:01:03.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.980+0000 7fe4df9d2640 1 Processor -- start 2024-09-16T04:01:03.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.980+0000 7fe4df9d2640 1 -- start start 2024-09-16T04:01:03.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.980+0000 7fe4df9d2640 10 monclient: init 2024-09-16T04:01:03.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.980+0000 7fe4df9d2640 10 monclient: _reopen_session rank -1 2024-09-16T04:01:03.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.980+0000 7fe4df9d2640 10 monclient: _add_conns ranks=[1,2,3,4,0] 2024-09-16T04:01:03.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.980+0000 7fe4df9d2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe4d80a39b0 addr v1:172.21.15.154:6789/0 2024-09-16T04:01:03.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.980+0000 7fe4df9d2640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe4d80a3d90 0x7fe4d813c380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:01:03.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.980+0000 7fe4df9d2640 10 monclient(hunting): picked mon.noname-b con 0x7fe4d80a3d90 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:01:03.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.980+0000 7fe4df9d2640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe4d813c8c0 0x7fe4d813ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:01:03.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.980+0000 7fe4df9d2640 10 monclient(hunting): picked mon.noname-a con 0x7fe4d813c8c0 addr v2:172.21.15.42:3300/0 2024-09-16T04:01:03.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.980+0000 7fe4df9d2640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe4d8134290 con 0x7fe4d80a39b0 2024-09-16T04:01:03.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.980+0000 7fe4df9d2640 10 monclient(hunting): start opening mon connection 2024-09-16T04:01:03.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.980+0000 7fe4df9d2640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7fe4d809c140 con 0x7fe4d813c8c0 2024-09-16T04:01:03.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.980+0000 7fe4df9d2640 10 monclient(hunting): start opening mon connection 2024-09-16T04:01:03.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.980+0000 7fe4df9d2640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fe4d809c420 con 0x7fe4d80a3d90 2024-09-16T04:01:03.992 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.980+0000 7fe4df9d2640 10 monclient(hunting): _renew_subs 2024-09-16T04:01:03.992 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.980+0000 7fe4df9d2640 10 monclient(hunting): authenticate will time out at 2110.955732s 2024-09-16T04:01:03.992 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.980+0000 7fe4dcf6d640 1 -- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe4d80a3d90 msgr2=0x7fe4d813c380 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T04:01:03.992 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.980+0000 7fe4dcf6d640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe4d80a3d90 0x7fe4d813c380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:01:03.992 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.984+0000 7fe4ddf6f640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe4d813c8c0 0x7fe4d813ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:01:03.992 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.984+0000 7fe4ddf6f640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe4d813c8c0 0x7fe4d813ecb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:44206/0 (socket says 172.21.15.42:44206) 2024-09-16T04:01:03.992 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.984+0000 7fe4ddf6f640 1 -- 172.21.15.42:0/569309846 learned_addr learned my addr 172.21.15.42:0/569309846 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:01:03.992 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.984+0000 7fe4ddf6f640 10 monclient(hunting): get_auth_request con 0x7fe4d813c8c0 auth_method 0 2024-09-16T04:01:03.992 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.984+0000 7fe4ddf6f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:01:03.992 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.984+0000 7fe4ddf6f640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:01:03.992 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.984+0000 7fe4ddf6f640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:01:03.992 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.984+0000 7fe4ddf6f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:01:03.992 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.984+0000 7fe4ddf6f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:01:03.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.984+0000 7fe4ddf6f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:01:03.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.984+0000 7fe4ddf6f640 10 monclient(hunting): handle_auth_done global_id 825330 payload 274 2024-09-16T04:01:03.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.984+0000 7fe4ddf6f640 1 -- 172.21.15.42:0/569309846 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe4d80a3d90 msgr2=0x7fe4d813c380 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:01:03.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.984+0000 7fe4ddf6f640 1 --2- 172.21.15.42:0/569309846 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe4d80a3d90 0x7fe4d813c380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:03.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.984+0000 7fe4ddf6f640 1 -- 172.21.15.42:0/569309846 >> v1:172.21.15.154:6789/0 conn(0x7fe4d80a39b0 legacy=0x7fe4d8133b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:01:03.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.984+0000 7fe4ddf6f640 10 monclient: _finish_hunting 0 2024-09-16T04:01:03.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.984+0000 7fe4ddf6f640 1 monclient: found mon.noname-a 2024-09-16T04:01:03.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.984+0000 7fe4ddf6f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:01:03.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.984+0000 7fe4ddf6f640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:01:03.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.984+0000 7fe4ddf6f640 1 -- 172.21.15.42:0/569309846 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4d806dd60 con 0x7fe4d813c8c0 2024-09-16T04:01:04.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.984+0000 7fe4ddf6f640 1 --2- 172.21.15.42:0/569309846 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe4d813c8c0 0x7fe4d813ecb0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fe4d0009e50 tx=0x7fe4d001c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36cf4e9b6d2706f8 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:01:04.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.984+0000 7fe4cffff640 1 -- 172.21.15.42:0/569309846 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4d00a55d0 con 0x7fe4d813c8c0 2024-09-16T04:01:04.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.984+0000 7fe4cffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:01:04.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.984+0000 7fe4cffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:01:04.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.984+0000 7fe4cffff640 10 monclient: dump: 2024-09-16T04:01:04.001 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:01:04.001 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:01:04.001 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:04.002 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:04.002 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:01:04.002 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:01:04.002 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:01:04.002 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:01:04.002 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:01:04.002 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:01:04.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.984+0000 7fe4cffff640 10 monclient: _finish_auth 0 2024-09-16T04:01:04.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.984+0000 7fe4cffff640 10 monclient: _check_auth_tickets 2024-09-16T04:01:04.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.984+0000 7fe4cffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:01:04.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.984+0000 7fe4df9d2640 5 monclient: authenticate success, global_id 825330 2024-09-16T04:01:04.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.984+0000 7fe4df9d2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:01:04.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.984+0000 7fe4cffff640 1 -- 172.21.15.42:0/569309846 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe4d00a5770 con 0x7fe4d813c8c0 2024-09-16T04:01:04.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.984+0000 7fe4cffff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:01:04.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.984+0000 7fe4df9d2640 10 monclient: get_monmap_and_config success 2024-09-16T04:01:04.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.984+0000 7fe4cffff640 1 -- 172.21.15.42:0/569309846 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4d00a5b40 con 0x7fe4d813c8c0 2024-09-16T04:01:04.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.984+0000 7fe4cffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:01:04.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.984+0000 7fe4df9d2640 4 set_mon_vals no callback set 2024-09-16T04:01:04.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.984+0000 7fe4cffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:01:04.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.984+0000 7fe4cffff640 10 monclient: dump: 2024-09-16T04:01:04.003 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:01:04.003 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:01:04.003 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:04.003 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:04.003 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:01:04.003 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:01:04.003 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:01:04.003 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:01:04.004 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:01:04.004 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:01:04.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.988+0000 7fe4df9d2640 10 monclient: shutdown 2024-09-16T04:01:04.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.988+0000 7fe4df9d2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:01:04.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.988+0000 7fe4df9d2640 1 -- 172.21.15.42:0/569309846 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe4d813c8c0 msgr2=0x7fe4d813ecb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:01:04.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.988+0000 7fe4df9d2640 1 --2- 172.21.15.42:0/569309846 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe4d813c8c0 0x7fe4d813ecb0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fe4d0009e50 tx=0x7fe4d001c910 comp rx=0 tx=0).stop 2024-09-16T04:01:04.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.988+0000 7fe4df9d2640 1 -- 172.21.15.42:0/569309846 shutdown_connections 2024-09-16T04:01:04.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.988+0000 7fe4df9d2640 1 --2- 172.21.15.42:0/569309846 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe4d813c8c0 0x7fe4d813ecb0 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:04.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.988+0000 7fe4df9d2640 1 --2- 172.21.15.42:0/569309846 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe4d80a3d90 0x7fe4d813c380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:04.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.988+0000 7fe4df9d2640 1 -- 172.21.15.42:0/569309846 >> 172.21.15.42:0/569309846 conn(0x7fe4d80a9fb0 msgr2=0x7fe4d80aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:01:04.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.988+0000 7fe4df9d2640 1 -- 172.21.15.42:0/569309846 shutdown_connections 2024-09-16T04:01:04.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.988+0000 7fe4df9d2640 1 -- 172.21.15.42:0/569309846 wait complete. 2024-09-16T04:01:04.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.988+0000 7fe4df9d2640 10 monclient: build_initial_monmap 2024-09-16T04:01:04.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.988+0000 7fe4df9d2640 10 monclient: monmap: 2024-09-16T04:01:04.005 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:01:04.005 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:01:04.005 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:01:04.005 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:01:04.005 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:01:04.005 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:01:04.005 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:01:04.005 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:01:04.005 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:01:04.005 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:01:04.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.988+0000 7fe4df9d2640 1 librados: starting msgr at 2024-09-16T04:01:04.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.988+0000 7fe4df9d2640 1 librados: starting objecter 2024-09-16T04:01:04.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.988+0000 7fe4df9d2640 1 Processor -- start 2024-09-16T04:01:04.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.988+0000 7fe4df9d2640 1 -- start start 2024-09-16T04:01:04.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.988+0000 7fe4df9d2640 1 librados: setting wanted keys 2024-09-16T04:01:04.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.988+0000 7fe4df9d2640 1 librados: calling monclient init 2024-09-16T04:01:04.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.988+0000 7fe4df9d2640 10 monclient: init 2024-09-16T04:01:04.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.988+0000 7fe4df9d2640 10 monclient: _reopen_session rank -1 2024-09-16T04:01:04.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.988+0000 7fe4df9d2640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:01:04.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.988+0000 7fe4df9d2640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe4d80a39b0 0x7fe4d8138130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:01:04.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.988+0000 7fe4df9d2640 10 monclient(hunting): picked mon.noname-c con 0x7fe4d80a39b0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:01:04.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.988+0000 7fe4df9d2640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe4d80a3d90 0x7fe4d8138670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:01:04.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.988+0000 7fe4df9d2640 10 monclient(hunting): picked mon.noname-b con 0x7fe4d80a3d90 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:01:04.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.988+0000 7fe4df9d2640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe4d813c8c0 0x7fe4d8138bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:01:04.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.988+0000 7fe4df9d2640 10 monclient(hunting): picked mon.noname-a con 0x7fe4d813c8c0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:01:04.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.988+0000 7fe4df9d2640 10 monclient(hunting): start opening mon connection 2024-09-16T04:01:04.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.988+0000 7fe4df9d2640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fe4d8141da0 con 0x7fe4d813c8c0 2024-09-16T04:01:04.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.988+0000 7fe4df9d2640 10 monclient(hunting): start opening mon connection 2024-09-16T04:01:04.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.988+0000 7fe4df9d2640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe4d8141c20 con 0x7fe4d80a3d90 2024-09-16T04:01:04.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.988+0000 7fe4df9d2640 10 monclient(hunting): start opening mon connection 2024-09-16T04:01:04.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.988+0000 7fe4df9d2640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fe4d8141f20 con 0x7fe4d80a39b0 2024-09-16T04:01:04.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.988+0000 7fe4df9d2640 10 monclient(hunting): _renew_subs 2024-09-16T04:01:04.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.988+0000 7fe4df9d2640 10 monclient(hunting): authenticate will time out at 2110.962243s 2024-09-16T04:01:04.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.988+0000 7fe4dd76e640 1 -- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe4d80a39b0 msgr2=0x7fe4d8138130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T04:01:04.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.988+0000 7fe4dd76e640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe4d80a39b0 0x7fe4d8138130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:01:04.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.988+0000 7fe4ddf6f640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe4d813c8c0 0x7fe4d8138bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:01:04.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.988+0000 7fe4ddf6f640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe4d813c8c0 0x7fe4d8138bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:44212/0 (socket says 172.21.15.42:44212) 2024-09-16T04:01:04.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.988+0000 7fe4ddf6f640 1 -- 172.21.15.42:0/224249069 learned_addr learned my addr 172.21.15.42:0/224249069 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:01:04.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.988+0000 7fe4ddf6f640 10 monclient(hunting): get_auth_request con 0x7fe4d813c8c0 auth_method 0 2024-09-16T04:01:04.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.988+0000 7fe4ddf6f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:01:04.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.988+0000 7fe4ddf6f640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:01:04.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.988+0000 7fe4ddf6f640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:01:04.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.988+0000 7fe4ddf6f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:01:04.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.988+0000 7fe4ddf6f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:01:04.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.988+0000 7fe4ddf6f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:01:04.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.988+0000 7fe4dcf6d640 1 --2- 172.21.15.42:0/224249069 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe4d80a3d90 0x7fe4d8138670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:01:04.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.988+0000 7fe4dcf6d640 10 monclient(hunting): get_auth_request con 0x7fe4d80a3d90 auth_method 0 2024-09-16T04:01:04.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.988+0000 7fe4dcf6d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:01:04.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.988+0000 7fe4dcf6d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:01:04.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.988+0000 7fe4dcf6d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:01:04.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.988+0000 7fe4ddf6f640 10 monclient(hunting): handle_auth_done global_id 825333 payload 824 2024-09-16T04:01:04.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.988+0000 7fe4ddf6f640 1 -- 172.21.15.42:0/224249069 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe4d80a39b0 msgr2=0x7fe4d8138130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:01:04.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.988+0000 7fe4ddf6f640 1 --2- 172.21.15.42:0/224249069 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe4d80a39b0 0x7fe4d8138130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:04.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.988+0000 7fe4ddf6f640 1 -- 172.21.15.42:0/224249069 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe4d80a3d90 msgr2=0x7fe4d8138670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:01:04.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.988+0000 7fe4ddf6f640 1 --2- 172.21.15.42:0/224249069 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe4d80a3d90 0x7fe4d8138670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:04.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.988+0000 7fe4ddf6f640 10 monclient: _finish_hunting 0 2024-09-16T04:01:04.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.988+0000 7fe4ddf6f640 1 monclient: found mon.noname-a 2024-09-16T04:01:04.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.988+0000 7fe4ddf6f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:01:04.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.988+0000 7fe4ddf6f640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:01:04.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.988+0000 7fe4ddf6f640 1 -- 172.21.15.42:0/224249069 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4d81392d0 con 0x7fe4d813c8c0 2024-09-16T04:01:04.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.988+0000 7fe4ddf6f640 1 --2- 172.21.15.42:0/224249069 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe4d813c8c0 0x7fe4d8138bb0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fe4d0009e50 tx=0x7fe4d00196b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:01:04.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.988+0000 7fe4cdffb640 10 client.?.objecter ms_handle_connect 0x7fe4d813c8c0 2024-09-16T04:01:04.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.988+0000 7fe4cdffb640 10 client.?.objecter resend_mon_ops 2024-09-16T04:01:04.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.988+0000 7fe4cdffb640 1 -- 172.21.15.42:0/224249069 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4d00bc020 con 0x7fe4d813c8c0 2024-09-16T04:01:04.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.988+0000 7fe4cdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:01:04.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.988+0000 7fe4cdffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:01:04.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.988+0000 7fe4cdffb640 10 monclient: dump: 2024-09-16T04:01:04.010 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:01:04.010 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:01:04.010 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:04.010 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:04.010 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:01:04.010 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:01:04.010 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:01:04.010 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:01:04.010 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:01:04.010 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:01:04.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.988+0000 7fe4cdffb640 10 monclient: _finish_auth 0 2024-09-16T04:01:04.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.988+0000 7fe4cdffb640 10 monclient: _check_auth_tickets 2024-09-16T04:01:04.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.988+0000 7fe4cdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:01:04.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.988+0000 7fe4cdffb640 1 -- 172.21.15.42:0/224249069 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe4d00b8880 con 0x7fe4d813c8c0 2024-09-16T04:01:04.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.988+0000 7fe4cdffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:01:04.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.988+0000 7fe4cdffb640 1 -- 172.21.15.42:0/224249069 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4d0009510 con 0x7fe4d813c8c0 2024-09-16T04:01:04.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.988+0000 7fe4df9d2640 5 monclient: authenticate success, global_id 825333 2024-09-16T04:01:04.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.988+0000 7fe4df9d2640 10 monclient: _renew_subs 2024-09-16T04:01:04.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.988+0000 7fe4df9d2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:01:04.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.988+0000 7fe4df9d2640 1 -- 172.21.15.42:0/224249069 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4d8132a70 con 0x7fe4d813c8c0 2024-09-16T04:01:04.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.988+0000 7fe4cf7fe640 4 set_mon_vals no callback set 2024-09-16T04:01:04.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.988+0000 7fe4cdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:01:04.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.988+0000 7fe4cdffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:01:04.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.988+0000 7fe4cdffb640 10 monclient: dump: 2024-09-16T04:01:04.011 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:01:04.011 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:01:04.012 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:04.012 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:04.012 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:01:04.012 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:01:04.012 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:01:04.012 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:01:04.012 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:01:04.012 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:01:04.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.988+0000 7fe4df9d2640 10 client.825333.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:01:04.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.988+0000 7fe4df9d2640 10 monclient: _renew_subs 2024-09-16T04:01:04.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.988+0000 7fe4df9d2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:01:04.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.988+0000 7fe4df9d2640 1 -- 172.21.15.42:0/224249069 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4d813c420 con 0x7fe4d813c8c0 2024-09-16T04:01:04.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.988+0000 7fe4df9d2640 1 librados: init done 2024-09-16T04:01:04.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.992+0000 7fe4cdffb640 1 -- 172.21.15.42:0/224249069 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fe4d00bf020 con 0x7fe4d813c8c0 2024-09-16T04:01:04.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.992+0000 7fe4df9d2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:01:04.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.992+0000 7fe4df9d2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:01:04.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.992+0000 7fe4df9d2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:01:04.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.992+0000 7fe4df9d2640 1 -- 172.21.15.42:0/224249069 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe4d8133b80 con 0x7fe4d813c8c0 2024-09-16T04:01:04.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.992+0000 7fe4cdffb640 1 --2- 172.21.15.42:0/224249069 >> v2:172.21.15.154:6812/254845342 conn(0x7fe4bc0377c0 0x7fe4bc039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:01:04.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.992+0000 7fe4cdffb640 1 -- 172.21.15.42:0/224249069 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(569..569 src has 451..569) ==== 6216+0+0 (secure 0 0 0) 0x7fe4d00e41f0 con 0x7fe4d813c8c0 2024-09-16T04:01:04.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.992+0000 7fe4cdffb640 10 client.825333.objecter ms_dispatch 0x7fe4d8000ba0 osd_map(569..569 src has 451..569) 2024-09-16T04:01:04.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.992+0000 7fe4cdffb640 3 client.825333.objecter handle_osd_map got epochs [569,569] > 0 2024-09-16T04:01:04.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.992+0000 7fe4cdffb640 3 client.825333.objecter handle_osd_map decoding full epoch 569 2024-09-16T04:01:04.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.992+0000 7fe4cdffb640 20 client.825333.objecter dump_active .. 0 homeless 2024-09-16T04:01:04.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.992+0000 7fe4dd76e640 1 --2- 172.21.15.42:0/224249069 >> v2:172.21.15.154:6812/254845342 conn(0x7fe4bc0377c0 0x7fe4bc039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:01:04.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.992+0000 7fe4dd76e640 10 monclient: get_auth_request con 0x7fe4bc0377c0 auth_method 0 2024-09-16T04:01:04.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.992+0000 7fe4cdffb640 10 client.825333.objecter ms_handle_connect 0x7fe4bc0377c0 2024-09-16T04:01:04.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.992+0000 7fe4dd76e640 1 --2- 172.21.15.42:0/224249069 >> v2:172.21.15.154:6812/254845342 conn(0x7fe4bc0377c0 0x7fe4bc039c80 secure :-1 s=READY pgs=3756 cs=0 l=1 rev1=1 crypto rx=0x7fe4c8000c10 tx=0x7fe4c8007a20 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:01:04.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.996+0000 7fe4cdffb640 1 -- 172.21.15.42:0/224249069 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7fe4d00c6070 con 0x7fe4d813c8c0 2024-09-16T04:01:04.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.996+0000 7fe4cdffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:01:04.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:03.996+0000 7fe4cdffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:01:04.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:04.232+0000 7fe4df9d2640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:01:04.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:04.232+0000 7fe4df9d2640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:01:04.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:04.232+0000 7fe4df9d2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:01:04.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:04.232+0000 7fe4df9d2640 1 -- 172.21.15.42:0/224249069 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fe4d80a5750 con 0x7fe4d813c8c0 2024-09-16T04:01:04.242 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:04.232+0000 7fe4cdffb640 1 -- 172.21.15.42:0/224249069 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1562 (secure 0 0 0) 0x7fe4d8132a70 con 0x7fe4d813c8c0 2024-09-16T04:01:04.242 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:04.236+0000 7fe4cdffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:01:04.242 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:04.236+0000 7fe4cdffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:01:04.242 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":168,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":13,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:01:04.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:04.236+0000 7fe4df9d2640 10 librados: watch_flush enter 2024-09-16T04:01:04.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:04.236+0000 7fe4df9d2640 10 librados: watch_flush exit 2024-09-16T04:01:04.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:04.236+0000 7fe4df9d2640 20 client.825333.objecter shutdown clearing up homeless session... 2024-09-16T04:01:04.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:04.236+0000 7fe4df9d2640 10 client.825333.objecter successfully canceled tick 2024-09-16T04:01:04.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:04.236+0000 7fe4df9d2640 1 -- 172.21.15.42:0/224249069 >> v2:172.21.15.154:6812/254845342 conn(0x7fe4bc0377c0 msgr2=0x7fe4bc039c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:01:04.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:04.236+0000 7fe4df9d2640 1 --2- 172.21.15.42:0/224249069 >> v2:172.21.15.154:6812/254845342 conn(0x7fe4bc0377c0 0x7fe4bc039c80 secure :-1 s=READY pgs=3756 cs=0 l=1 rev1=1 crypto rx=0x7fe4c8000c10 tx=0x7fe4c8007a20 comp rx=0 tx=0).stop 2024-09-16T04:01:04.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:04.236+0000 7fe4df9d2640 10 monclient: shutdown 2024-09-16T04:01:04.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:04.236+0000 7fe4df9d2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:01:04.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:04.236+0000 7fe4df9d2640 1 -- 172.21.15.42:0/224249069 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe4d813c8c0 msgr2=0x7fe4d8138bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:01:04.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:04.236+0000 7fe4df9d2640 1 --2- 172.21.15.42:0/224249069 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe4d813c8c0 0x7fe4d8138bb0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fe4d0009e50 tx=0x7fe4d00196b0 comp rx=0 tx=0).stop 2024-09-16T04:01:04.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:04.236+0000 7fe4df9d2640 1 -- 172.21.15.42:0/224249069 shutdown_connections 2024-09-16T04:01:04.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:04.236+0000 7fe4df9d2640 1 --2- 172.21.15.42:0/224249069 >> v2:172.21.15.154:6812/254845342 conn(0x7fe4bc0377c0 0x7fe4bc039c80 unknown :-1 s=CLOSED pgs=3756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:04.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:04.236+0000 7fe4df9d2640 1 --2- 172.21.15.42:0/224249069 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe4d813c8c0 0x7fe4d8138bb0 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:04.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:04.236+0000 7fe4df9d2640 1 --2- 172.21.15.42:0/224249069 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe4d80a3d90 0x7fe4d8138670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:04.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:04.236+0000 7fe4df9d2640 1 --2- 172.21.15.42:0/224249069 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe4d80a39b0 0x7fe4d8138130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:04.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:04.236+0000 7fe4df9d2640 1 -- 172.21.15.42:0/224249069 >> 172.21.15.42:0/224249069 conn(0x7fe4d80a9fb0 msgr2=0x7fe4d8131980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:01:04.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:04.236+0000 7fe4df9d2640 1 -- 172.21.15.42:0/224249069 shutdown_connections 2024-09-16T04:01:04.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:04.236+0000 7fe4df9d2640 1 -- 172.21.15.42:0/224249069 wait complete. 2024-09-16T04:01:04.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:04.236+0000 7fe4df9d2640 1 librados: shutdown 2024-09-16T04:01:07.262 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:01:07.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:07.388+0000 7fac51492640 10 monclient: get_monmap_and_config 2024-09-16T04:01:07.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:07.388+0000 7fac51492640 10 monclient: build_initial_monmap 2024-09-16T04:01:07.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:07.388+0000 7fac51492640 10 monclient: monmap: 2024-09-16T04:01:07.395 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:01:07.395 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:01:07.396 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:01:07.393876+0000 2024-09-16T04:01:07.396 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:01:07.393876+0000 2024-09-16T04:01:07.396 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:01:07.396 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:01:07.396 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:01:07.396 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:01:07.396 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:01:07.396 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:01:07.396 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:01:07.396 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:01:07.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:07.388+0000 7fac51492640 1 Processor -- start 2024-09-16T04:01:07.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:07.388+0000 7fac51492640 1 -- start start 2024-09-16T04:01:07.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:07.388+0000 7fac51492640 10 monclient: init 2024-09-16T04:01:07.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:07.388+0000 7fac51492640 10 monclient: _reopen_session rank -1 2024-09-16T04:01:07.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:07.388+0000 7fac51492640 10 monclient: _add_conns ranks=[3,2,4,1,0] 2024-09-16T04:01:07.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:07.388+0000 7fac51492640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fac4c0a39b0 0x7fac4c133b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:01:07.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:07.388+0000 7fac51492640 10 monclient(hunting): picked mon.noname-a con 0x7fac4c0a39b0 addr v2:172.21.15.42:3300/0 2024-09-16T04:01:07.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:07.388+0000 7fac51492640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fac4c0a3d90 0x7fac4c137000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:01:07.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:07.388+0000 7fac51492640 10 monclient(hunting): picked mon.noname-b con 0x7fac4c0a3d90 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:01:07.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:07.388+0000 7fac51492640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fac4c137540 0x7fac4c137990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:01:07.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:07.388+0000 7fac51492640 10 monclient(hunting): picked mon.noname-c con 0x7fac4c137540 addr v2:172.21.15.154:3300/0 2024-09-16T04:01:07.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:07.388+0000 7fac51492640 10 monclient(hunting): start opening mon connection 2024-09-16T04:01:07.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:07.388+0000 7fac51492640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7fac4c09c140 con 0x7fac4c0a39b0 2024-09-16T04:01:07.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:07.388+0000 7fac51492640 10 monclient(hunting): start opening mon connection 2024-09-16T04:01:07.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:07.388+0000 7fac51492640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fac4c09c420 con 0x7fac4c137540 2024-09-16T04:01:07.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:07.388+0000 7fac51492640 10 monclient(hunting): start opening mon connection 2024-09-16T04:01:07.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:07.388+0000 7fac51492640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fac4c06dd60 con 0x7fac4c0a3d90 2024-09-16T04:01:07.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:07.392+0000 7fac51492640 10 monclient(hunting): _renew_subs 2024-09-16T04:01:07.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:07.392+0000 7fac51492640 10 monclient(hunting): authenticate will time out at 2114.364019s 2024-09-16T04:01:07.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:07.392+0000 7fac4a7fc640 1 -- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fac4c0a3d90 msgr2=0x7fac4c137000 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T04:01:07.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:07.392+0000 7fac4a7fc640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fac4c0a3d90 0x7fac4c137000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:01:07.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:07.392+0000 7fac4affd640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fac4c0a39b0 0x7fac4c133b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:01:07.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:07.392+0000 7fac4affd640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fac4c0a39b0 0x7fac4c133b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:40750/0 (socket says 172.21.15.42:40750) 2024-09-16T04:01:07.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:07.392+0000 7fac4affd640 1 -- 172.21.15.42:0/343489618 learned_addr learned my addr 172.21.15.42:0/343489618 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:01:07.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:07.392+0000 7fac4affd640 10 monclient(hunting): get_auth_request con 0x7fac4c0a39b0 auth_method 0 2024-09-16T04:01:07.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:07.392+0000 7fac4affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:01:07.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:07.392+0000 7fac4affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:01:07.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:07.392+0000 7fac4affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:01:07.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:07.392+0000 7fac4affd640 1 -- 172.21.15.42:0/343489618 >> v2:172.21.15.42:3300/0 conn(0x7fac4c0a39b0 msgr2=0x7fac4c133b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:01:07.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:07.392+0000 7fac4affd640 1 -- 172.21.15.42:0/343489618 >> v2:172.21.15.42:3300/0 conn(0x7fac4c0a39b0 msgr2=0x7fac4c133b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:01:07.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:07.392+0000 7fac4affd640 1 --2- 172.21.15.42:0/343489618 >> v2:172.21.15.42:3300/0 conn(0x7fac4c0a39b0 0x7fac4c133b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:01:07.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:07.392+0000 7fac4b7fe640 1 --2- 172.21.15.42:0/343489618 >> v2:172.21.15.154:3300/0 conn(0x7fac4c137540 0x7fac4c137990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:01:07.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:07.392+0000 7fac4affd640 1 --2- 172.21.15.42:0/343489618 >> v2:172.21.15.42:3300/0 conn(0x7fac4c0a39b0 0x7fac4c133b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:01:07.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:07.392+0000 7fac4b7fe640 10 monclient(hunting): get_auth_request con 0x7fac4c137540 auth_method 0 2024-09-16T04:01:07.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:07.392+0000 7fac4b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:01:07.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:07.392+0000 7fac4b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:01:07.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:07.392+0000 7fac4b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:01:07.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:07.392+0000 7fac4b7fe640 1 -- 172.21.15.42:0/343489618 >> v2:172.21.15.154:3300/0 conn(0x7fac4c137540 msgr2=0x7fac4c137990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T04:01:07.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:07.392+0000 7fac4b7fe640 1 -- 172.21.15.42:0/343489618 >> v2:172.21.15.154:3300/0 conn(0x7fac4c137540 msgr2=0x7fac4c137990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:01:07.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:07.392+0000 7fac4b7fe640 1 --2- 172.21.15.42:0/343489618 >> v2:172.21.15.154:3300/0 conn(0x7fac4c137540 0x7fac4c137990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:01:07.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:07.392+0000 7fac4b7fe640 1 --2- 172.21.15.42:0/343489618 >> v2:172.21.15.154:3300/0 conn(0x7fac4c137540 0x7fac4c137990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:01:07.602 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:07.596+0000 7fac4a7fc640 1 -- 172.21.15.42:0/343489618 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fac4c0a3d90 msgr2=0x7fac4c137000 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T04:01:07.602 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:07.596+0000 7fac4a7fc640 1 --2- 172.21.15.42:0/343489618 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fac4c0a3d90 0x7fac4c137000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T04:01:07.602 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:07.596+0000 7fac4affd640 1 --2- 172.21.15.42:0/343489618 >> v2:172.21.15.42:3300/0 conn(0x7fac4c0a39b0 0x7fac4c133b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:01:07.603 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:07.596+0000 7fac4affd640 10 monclient(hunting): get_auth_request con 0x7fac4c0a39b0 auth_method 0 2024-09-16T04:01:07.603 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:07.596+0000 7fac4affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:01:07.603 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:07.596+0000 7fac4affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:01:07.603 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:07.596+0000 7fac4affd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:01:07.603 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:07.596+0000 7fac4b7fe640 1 --2- 172.21.15.42:0/343489618 >> v2:172.21.15.154:3300/0 conn(0x7fac4c137540 0x7fac4c137990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:01:07.603 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:07.596+0000 7fac4b7fe640 10 monclient(hunting): get_auth_request con 0x7fac4c137540 auth_method 0 2024-09-16T04:01:07.603 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:07.596+0000 7fac4b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:01:07.603 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:07.596+0000 7fac4b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:01:07.603 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:07.596+0000 7fac4b7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:01:07.603 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:07.596+0000 7fac4affd640 1 -- 172.21.15.42:0/343489618 >> v2:172.21.15.42:3300/0 conn(0x7fac4c0a39b0 msgr2=0x7fac4c133b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T04:01:07.603 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:07.596+0000 7fac4affd640 1 -- 172.21.15.42:0/343489618 >> v2:172.21.15.42:3300/0 conn(0x7fac4c0a39b0 msgr2=0x7fac4c133b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:01:07.603 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:07.596+0000 7fac4affd640 1 --2- 172.21.15.42:0/343489618 >> v2:172.21.15.42:3300/0 conn(0x7fac4c0a39b0 0x7fac4c133b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:01:07.603 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:07.596+0000 7fac4affd640 1 --2- 172.21.15.42:0/343489618 >> v2:172.21.15.42:3300/0 conn(0x7fac4c0a39b0 0x7fac4c133b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T04:01:07.604 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:07.596+0000 7fac4b7fe640 1 -- 172.21.15.42:0/343489618 >> v2:172.21.15.154:3300/0 conn(0x7fac4c137540 msgr2=0x7fac4c137990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:01:07.604 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:07.596+0000 7fac4b7fe640 1 -- 172.21.15.42:0/343489618 >> v2:172.21.15.154:3300/0 conn(0x7fac4c137540 msgr2=0x7fac4c137990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:01:07.604 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:07.596+0000 7fac4b7fe640 1 --2- 172.21.15.42:0/343489618 >> v2:172.21.15.154:3300/0 conn(0x7fac4c137540 0x7fac4c137990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:01:07.604 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:07.596+0000 7fac4b7fe640 1 --2- 172.21.15.42:0/343489618 >> v2:172.21.15.154:3300/0 conn(0x7fac4c137540 0x7fac4c137990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T04:01:08.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:08.000+0000 7fac4a7fc640 1 -- 172.21.15.42:0/343489618 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fac4c0a3d90 msgr2=0x7fac4c137000 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T04:01:08.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:08.000+0000 7fac4a7fc640 1 --2- 172.21.15.42:0/343489618 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fac4c0a3d90 0x7fac4c137000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T04:01:08.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:08.000+0000 7fac4affd640 1 --2- 172.21.15.42:0/343489618 >> v2:172.21.15.42:3300/0 conn(0x7fac4c0a39b0 0x7fac4c133b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:01:08.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:08.000+0000 7fac4affd640 10 monclient(hunting): get_auth_request con 0x7fac4c0a39b0 auth_method 0 2024-09-16T04:01:08.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:08.000+0000 7fac4affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:01:08.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:08.000+0000 7fac4affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:01:08.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:08.000+0000 7fac4affd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:01:08.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:08.000+0000 7fac4b7fe640 1 --2- 172.21.15.42:0/343489618 >> v2:172.21.15.154:3300/0 conn(0x7fac4c137540 0x7fac4c137990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:01:08.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:08.000+0000 7fac4b7fe640 10 monclient(hunting): get_auth_request con 0x7fac4c137540 auth_method 0 2024-09-16T04:01:08.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:08.000+0000 7fac4affd640 1 -- 172.21.15.42:0/343489618 >> v2:172.21.15.42:3300/0 conn(0x7fac4c0a39b0 msgr2=0x7fac4c133b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T04:01:08.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:08.000+0000 7fac4b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:01:08.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:08.000+0000 7fac4affd640 1 -- 172.21.15.42:0/343489618 >> v2:172.21.15.42:3300/0 conn(0x7fac4c0a39b0 msgr2=0x7fac4c133b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:01:08.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:08.000+0000 7fac4b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:01:08.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:08.000+0000 7fac4b7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:01:08.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:08.000+0000 7fac4affd640 1 --2- 172.21.15.42:0/343489618 >> v2:172.21.15.42:3300/0 conn(0x7fac4c0a39b0 0x7fac4c133b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:01:08.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:08.000+0000 7fac4affd640 1 --2- 172.21.15.42:0/343489618 >> v2:172.21.15.42:3300/0 conn(0x7fac4c0a39b0 0x7fac4c133b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T04:01:08.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:08.000+0000 7fac4b7fe640 1 -- 172.21.15.42:0/343489618 >> v2:172.21.15.154:3300/0 conn(0x7fac4c137540 msgr2=0x7fac4c137990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:01:08.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:08.000+0000 7fac4b7fe640 1 -- 172.21.15.42:0/343489618 >> v2:172.21.15.154:3300/0 conn(0x7fac4c137540 msgr2=0x7fac4c137990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:01:08.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:08.000+0000 7fac4b7fe640 1 --2- 172.21.15.42:0/343489618 >> v2:172.21.15.154:3300/0 conn(0x7fac4c137540 0x7fac4c137990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:01:08.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:08.000+0000 7fac4b7fe640 1 --2- 172.21.15.42:0/343489618 >> v2:172.21.15.154:3300/0 conn(0x7fac4c137540 0x7fac4c137990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T04:01:08.808 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:08.804+0000 7fac4a7fc640 1 -- 172.21.15.42:0/343489618 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fac4c0a3d90 msgr2=0x7fac4c137000 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T04:01:08.808 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:08.804+0000 7fac4a7fc640 1 --2- 172.21.15.42:0/343489618 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fac4c0a3d90 0x7fac4c137000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T04:01:08.808 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:08.804+0000 7fac4affd640 1 --2- 172.21.15.42:0/343489618 >> v2:172.21.15.42:3300/0 conn(0x7fac4c0a39b0 0x7fac4c133b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:01:08.808 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:08.804+0000 7fac4affd640 10 monclient(hunting): get_auth_request con 0x7fac4c0a39b0 auth_method 0 2024-09-16T04:01:08.808 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:08.804+0000 7fac4affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:01:08.808 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:08.804+0000 7fac4affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:01:08.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:08.804+0000 7fac4affd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:01:08.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:08.804+0000 7fac4affd640 1 -- 172.21.15.42:0/343489618 >> v2:172.21.15.42:3300/0 conn(0x7fac4c0a39b0 msgr2=0x7fac4c133b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:01:08.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:08.804+0000 7fac4affd640 1 -- 172.21.15.42:0/343489618 >> v2:172.21.15.42:3300/0 conn(0x7fac4c0a39b0 msgr2=0x7fac4c133b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:01:08.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:08.804+0000 7fac4affd640 1 --2- 172.21.15.42:0/343489618 >> v2:172.21.15.42:3300/0 conn(0x7fac4c0a39b0 0x7fac4c133b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:01:08.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:08.804+0000 7fac4affd640 1 --2- 172.21.15.42:0/343489618 >> v2:172.21.15.42:3300/0 conn(0x7fac4c0a39b0 0x7fac4c133b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T04:01:08.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:08.804+0000 7fac4b7fe640 1 --2- 172.21.15.42:0/343489618 >> v2:172.21.15.154:3300/0 conn(0x7fac4c137540 0x7fac4c137990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:01:08.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:08.804+0000 7fac4b7fe640 10 monclient(hunting): get_auth_request con 0x7fac4c137540 auth_method 0 2024-09-16T04:01:08.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:08.804+0000 7fac4b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:01:08.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:08.804+0000 7fac4b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:01:08.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:08.804+0000 7fac4b7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:01:08.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:08.804+0000 7fac4b7fe640 1 -- 172.21.15.42:0/343489618 >> v2:172.21.15.154:3300/0 conn(0x7fac4c137540 msgr2=0x7fac4c137990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:01:08.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:08.804+0000 7fac4b7fe640 1 -- 172.21.15.42:0/343489618 >> v2:172.21.15.154:3300/0 conn(0x7fac4c137540 msgr2=0x7fac4c137990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:01:08.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:08.804+0000 7fac4b7fe640 1 --2- 172.21.15.42:0/343489618 >> v2:172.21.15.154:3300/0 conn(0x7fac4c137540 0x7fac4c137990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:01:08.810 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:08.804+0000 7fac4b7fe640 1 --2- 172.21.15.42:0/343489618 >> v2:172.21.15.154:3300/0 conn(0x7fac4c137540 0x7fac4c137990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T04:01:10.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:10.392+0000 7fac48ff9640 10 monclient(hunting): tick 2024-09-16T04:01:10.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:10.392+0000 7fac48ff9640 10 monclient(hunting): _check_auth_tickets 2024-09-16T04:01:10.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:10.392+0000 7fac48ff9640 1 monclient(hunting): continuing hunt 2024-09-16T04:01:10.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:10.392+0000 7fac48ff9640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T04:01:10.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:10.392+0000 7fac48ff9640 1 -- 172.21.15.42:0/343489618 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fac4c0a3d90 msgr2=0x7fac4c137000 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:01:10.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:10.392+0000 7fac48ff9640 1 --2- 172.21.15.42:0/343489618 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fac4c0a3d90 0x7fac4c137000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:10.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:10.392+0000 7fac48ff9640 1 -- 172.21.15.42:0/343489618 >> v2:172.21.15.154:3300/0 conn(0x7fac4c137540 msgr2=0x7fac4c137990 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:01:10.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:10.392+0000 7fac48ff9640 1 --2- 172.21.15.42:0/343489618 >> v2:172.21.15.154:3300/0 conn(0x7fac4c137540 0x7fac4c137990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:10.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:10.392+0000 7fac48ff9640 1 -- 172.21.15.42:0/343489618 >> v2:172.21.15.42:3300/0 conn(0x7fac4c0a39b0 msgr2=0x7fac4c133b80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:01:10.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:10.392+0000 7fac48ff9640 1 --2- 172.21.15.42:0/343489618 >> v2:172.21.15.42:3300/0 conn(0x7fac4c0a39b0 0x7fac4c133b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:10.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:10.392+0000 7fac48ff9640 10 monclient: _add_conns ranks=[0,1] 2024-09-16T04:01:10.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:10.392+0000 7fac48ff9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fac380036c0 addr v1:172.21.15.42:6789/0 2024-09-16T04:01:10.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:10.392+0000 7fac48ff9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fac380073d0 addr v1:172.21.15.154:6789/0 2024-09-16T04:01:10.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:10.392+0000 7fac48ff9640 1 -- 172.21.15.42:0/343489618 --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fac4c0a5e20 con 0x7fac380036c0 2024-09-16T04:01:10.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:10.392+0000 7fac48ff9640 1 -- 172.21.15.42:0/343489618 --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fac3800c150 con 0x7fac380073d0 2024-09-16T04:01:10.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:10.392+0000 7fac48ff9640 10 monclient(hunting): _renew_subs 2024-09-16T04:01:11.660 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.652+0000 7fac49ffb640 1 -- 172.21.15.42:0/343489618 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 167221618 0 0) 0x7fac4c0a5e20 con 0x7fac380036c0 2024-09-16T04:01:11.660 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.652+0000 7fac49ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:01:11.660 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.652+0000 7fac49ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:01:11.660 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.652+0000 7fac49ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:01:11.660 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.652+0000 7fac49ffb640 10 monclient(hunting): my global_id is 825336 2024-09-16T04:01:11.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.652+0000 7fac49ffb640 1 -- 172.21.15.42:0/343489618 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fac34003530 con 0x7fac380036c0 2024-09-16T04:01:11.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.652+0000 7fac49ffb640 1 -- 172.21.15.42:0/343489618 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 796554865 0 0) 0x7fac34003530 con 0x7fac380036c0 2024-09-16T04:01:11.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.652+0000 7fac49ffb640 1 -- 172.21.15.42:0/343489618 >> v1:172.21.15.154:6789/0 conn(0x7fac380073d0 legacy=0x7fac38009890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:01:11.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.652+0000 7fac49ffb640 10 monclient: _finish_hunting 0 2024-09-16T04:01:11.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.652+0000 7fac49ffb640 1 monclient: found mon.noname-a-legacy 2024-09-16T04:01:11.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.652+0000 7fac49ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:01:11.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.652+0000 7fac49ffb640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.42:6789/0 2024-09-16T04:01:11.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.652+0000 7fac49ffb640 1 -- 172.21.15.42:0/343489618 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac3800d390 con 0x7fac380036c0 2024-09-16T04:01:11.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.656+0000 7fac49ffb640 10 monclient: _finish_auth 0 2024-09-16T04:01:11.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.656+0000 7fac49ffb640 10 monclient: _check_auth_tickets 2024-09-16T04:01:11.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.656+0000 7fac49ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:01:11.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.656+0000 7fac49ffb640 1 -- 172.21.15.42:0/343489618 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2177413734 0 0) 0x7fac3c00f040 con 0x7fac380036c0 2024-09-16T04:01:11.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.656+0000 7fac49ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:01:11.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.656+0000 7fac49ffb640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T04:01:11.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.656+0000 7fac49ffb640 10 monclient: dump: 2024-09-16T04:01:11.664 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:01:11.664 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:01:11.664 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:11.664 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:11.664 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:01:11.664 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:01:11.664 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:01:11.665 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:01:11.665 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:01:11.665 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:01:11.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.656+0000 7fac49ffb640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] but i'm connected to v1:172.21.15.42:6789/0, reconnecting 2024-09-16T04:01:11.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.656+0000 7fac49ffb640 10 monclient: _reopen_session rank -1 2024-09-16T04:01:11.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.656+0000 7fac49ffb640 1 -- 172.21.15.42:0/343489618 >> v1:172.21.15.42:6789/0 conn(0x7fac380036c0 legacy=0x7fac38005b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:01:11.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.656+0000 7fac4b7fe640 1 -- 172.21.15.42:0/343489618 reap_dead start 2024-09-16T04:01:11.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.656+0000 7fac49ffb640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:01:11.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.656+0000 7fac49ffb640 1 --2- 172.21.15.42:0/343489618 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fac340039f0 0x7fac34003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:01:11.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.656+0000 7fac49ffb640 10 monclient(hunting): picked mon.c con 0x7fac340039f0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:01:11.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.656+0000 7fac49ffb640 1 --2- 172.21.15.42:0/343489618 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fac34004810 0x7fac34004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:01:11.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.656+0000 7fac49ffb640 10 monclient(hunting): picked mon.b con 0x7fac34004810 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:01:11.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.656+0000 7fac49ffb640 1 --2- 172.21.15.42:0/343489618 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fac340053d0 0x7fac3400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:01:11.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.656+0000 7fac49ffb640 10 monclient(hunting): picked mon.a con 0x7fac340053d0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:01:11.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.656+0000 7fac49ffb640 10 monclient(hunting): start opening mon connection 2024-09-16T04:01:11.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.656+0000 7fac49ffb640 1 -- 172.21.15.42:0/343489618 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fac3400c210 con 0x7fac340053d0 2024-09-16T04:01:11.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.656+0000 7fac49ffb640 10 monclient(hunting): start opening mon connection 2024-09-16T04:01:11.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.656+0000 7fac49ffb640 1 -- 172.21.15.42:0/343489618 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fac3400c3c0 con 0x7fac34004810 2024-09-16T04:01:11.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.656+0000 7fac49ffb640 10 monclient(hunting): start opening mon connection 2024-09-16T04:01:11.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.656+0000 7fac49ffb640 1 -- 172.21.15.42:0/343489618 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fac3400c5c0 con 0x7fac340039f0 2024-09-16T04:01:11.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.656+0000 7fac49ffb640 10 monclient(hunting): _renew_subs 2024-09-16T04:01:11.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.656+0000 7fac49ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:01:11.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.656+0000 7fac4affd640 1 -- 172.21.15.42:0/343489618 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fac340039f0 msgr2=0x7fac34003df0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T04:01:11.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.656+0000 7fac4affd640 1 --2- 172.21.15.42:0/343489618 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fac340039f0 0x7fac34003df0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:01:11.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.656+0000 7fac4b7fe640 1 --2- 172.21.15.42:0/343489618 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fac340053d0 0x7fac3400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:01:11.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.656+0000 7fac4b7fe640 10 monclient(hunting): get_auth_request con 0x7fac340053d0 auth_method 0 2024-09-16T04:01:11.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.656+0000 7fac4b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:01:11.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.656+0000 7fac4b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:01:11.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.656+0000 7fac4b7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:01:11.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.656+0000 7fac4a7fc640 1 --2- 172.21.15.42:0/343489618 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fac34004810 0x7fac34004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:01:11.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.656+0000 7fac4b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:01:11.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.656+0000 7fac4b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:01:11.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.656+0000 7fac4b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:01:11.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.656+0000 7fac4a7fc640 10 monclient(hunting): get_auth_request con 0x7fac34004810 auth_method 0 2024-09-16T04:01:11.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.656+0000 7fac4a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:01:11.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.656+0000 7fac4a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:01:11.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.656+0000 7fac4a7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:01:11.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.656+0000 7fac4a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:01:11.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.656+0000 7fac4a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:01:11.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.656+0000 7fac4a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:01:11.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.656+0000 7fac4b7fe640 10 monclient(hunting): handle_auth_done global_id 825336 payload 293 2024-09-16T04:01:11.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.656+0000 7fac4b7fe640 1 -- 172.21.15.42:0/343489618 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fac340039f0 msgr2=0x7fac34003df0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:01:11.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.656+0000 7fac4b7fe640 1 --2- 172.21.15.42:0/343489618 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fac340039f0 0x7fac34003df0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:11.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.656+0000 7fac4b7fe640 1 -- 172.21.15.42:0/343489618 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fac34004810 msgr2=0x7fac34004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:01:11.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.656+0000 7fac4b7fe640 1 --2- 172.21.15.42:0/343489618 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fac34004810 0x7fac34004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:11.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.656+0000 7fac4b7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:01:11.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.656+0000 7fac4b7fe640 1 monclient: found mon.a 2024-09-16T04:01:11.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.656+0000 7fac4b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:01:11.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.656+0000 7fac4b7fe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:01:11.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.656+0000 7fac4b7fe640 1 -- 172.21.15.42:0/343489618 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac34003530 con 0x7fac340053d0 2024-09-16T04:01:11.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.656+0000 7fac4b7fe640 10 monclient: _finish_auth 0 2024-09-16T04:01:11.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.656+0000 7fac4b7fe640 10 monclient: _check_auth_tickets 2024-09-16T04:01:11.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.656+0000 7fac4b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:01:11.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.656+0000 7fac51492640 5 monclient: authenticate success, global_id 825336 2024-09-16T04:01:11.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.656+0000 7fac51492640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:01:11.670 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.664+0000 7fac4b7fe640 1 --2- 172.21.15.42:0/343489618 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fac340053d0 0x7fac3400b9d0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fac4c0a6460 tx=0x7fac4401ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70908025fb5a38b1 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:01:11.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.668+0000 7fac49ffb640 1 -- 172.21.15.42:0/343489618 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac440ab070 con 0x7fac340053d0 2024-09-16T04:01:11.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.668+0000 7fac49ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:01:11.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.668+0000 7fac49ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:01:11.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.668+0000 7fac49ffb640 10 monclient: dump: 2024-09-16T04:01:11.676 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:01:11.676 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:01:11.676 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:11.676 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:11.676 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:01:11.676 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:01:11.677 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:01:11.677 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:01:11.677 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:01:11.677 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:01:11.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.668+0000 7fac51492640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:01:11.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.668+0000 7fac49ffb640 1 -- 172.21.15.42:0/343489618 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac440a5b90 con 0x7fac340053d0 2024-09-16T04:01:11.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.668+0000 7fac49ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:01:11.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.668+0000 7fac49ffb640 1 -- 172.21.15.42:0/343489618 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac4400f040 con 0x7fac340053d0 2024-09-16T04:01:11.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.668+0000 7fac51492640 10 monclient: get_monmap_and_config success 2024-09-16T04:01:11.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.668+0000 7fac51492640 4 set_mon_vals no callback set 2024-09-16T04:01:11.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.668+0000 7fac49ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:01:11.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.668+0000 7fac49ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:01:11.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.668+0000 7fac49ffb640 10 monclient: dump: 2024-09-16T04:01:11.679 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:01:11.679 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:01:11.679 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:11.679 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:11.679 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:01:11.679 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:01:11.679 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:01:11.680 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:01:11.680 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:01:11.680 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:01:11.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.672+0000 7fac51492640 10 monclient: shutdown 2024-09-16T04:01:11.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.672+0000 7fac51492640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:01:11.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.672+0000 7fac51492640 1 -- 172.21.15.42:0/343489618 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fac340053d0 msgr2=0x7fac3400b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:01:11.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.672+0000 7fac51492640 1 --2- 172.21.15.42:0/343489618 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fac340053d0 0x7fac3400b9d0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fac4c0a6460 tx=0x7fac4401ccc0 comp rx=0 tx=0).stop 2024-09-16T04:01:11.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.672+0000 7fac51492640 1 -- 172.21.15.42:0/343489618 shutdown_connections 2024-09-16T04:01:11.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.672+0000 7fac51492640 1 --2- 172.21.15.42:0/343489618 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fac340053d0 0x7fac3400b9d0 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:11.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.672+0000 7fac51492640 1 --2- 172.21.15.42:0/343489618 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fac34004810 0x7fac34004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:11.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.672+0000 7fac51492640 1 --2- 172.21.15.42:0/343489618 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fac340039f0 0x7fac34003df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:11.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.672+0000 7fac51492640 1 -- 172.21.15.42:0/343489618 >> 172.21.15.42:0/343489618 conn(0x7fac4c0a9fb0 msgr2=0x7fac4c0aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:01:11.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.672+0000 7fac51492640 1 -- 172.21.15.42:0/343489618 shutdown_connections 2024-09-16T04:01:11.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.672+0000 7fac51492640 1 -- 172.21.15.42:0/343489618 wait complete. 2024-09-16T04:01:11.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.672+0000 7fac51492640 10 monclient: build_initial_monmap 2024-09-16T04:01:11.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.672+0000 7fac51492640 10 monclient: monmap: 2024-09-16T04:01:11.681 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:01:11.681 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:01:11.681 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:01:11.682 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:01:11.682 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:01:11.682 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:01:11.682 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:01:11.682 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:01:11.682 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:01:11.682 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:01:11.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.676+0000 7fac51492640 1 librados: starting msgr at 2024-09-16T04:01:11.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.676+0000 7fac51492640 1 librados: starting objecter 2024-09-16T04:01:11.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.676+0000 7fac51492640 1 Processor -- start 2024-09-16T04:01:11.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.676+0000 7fac51492640 1 -- start start 2024-09-16T04:01:11.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.676+0000 7fac51492640 1 librados: setting wanted keys 2024-09-16T04:01:11.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.676+0000 7fac51492640 1 librados: calling monclient init 2024-09-16T04:01:11.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.676+0000 7fac51492640 10 monclient: init 2024-09-16T04:01:11.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.676+0000 7fac51492640 10 monclient: _reopen_session rank -1 2024-09-16T04:01:11.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.676+0000 7fac51492640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:01:11.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.676+0000 7fac51492640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fac340039f0 0x7fac4c1cbf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:01:11.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.676+0000 7fac51492640 10 monclient(hunting): picked mon.noname-c con 0x7fac340039f0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:01:11.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.676+0000 7fac51492640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fac34004810 0x7fac4c1cc470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:01:11.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.676+0000 7fac51492640 10 monclient(hunting): picked mon.noname-b con 0x7fac34004810 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:01:11.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.676+0000 7fac51492640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fac340053d0 0x7fac4c1cc9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:01:11.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.676+0000 7fac51492640 10 monclient(hunting): picked mon.noname-a con 0x7fac340053d0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:01:11.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.676+0000 7fac51492640 10 monclient(hunting): start opening mon connection 2024-09-16T04:01:11.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.676+0000 7fac51492640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fac4c137390 con 0x7fac340053d0 2024-09-16T04:01:11.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.676+0000 7fac51492640 10 monclient(hunting): start opening mon connection 2024-09-16T04:01:11.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.676+0000 7fac51492640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fac4c137210 con 0x7fac34004810 2024-09-16T04:01:11.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.676+0000 7fac51492640 10 monclient(hunting): start opening mon connection 2024-09-16T04:01:11.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.676+0000 7fac51492640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fac4c120200 con 0x7fac340039f0 2024-09-16T04:01:11.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.676+0000 7fac4affd640 1 -- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fac340039f0 msgr2=0x7fac4c1cbf30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T04:01:11.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.676+0000 7fac51492640 10 monclient(hunting): _renew_subs 2024-09-16T04:01:11.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.676+0000 7fac51492640 10 monclient(hunting): authenticate will time out at 2118.651878s 2024-09-16T04:01:11.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.676+0000 7fac4affd640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fac340039f0 0x7fac4c1cbf30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:01:11.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.680+0000 7fac4b7fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fac340053d0 0x7fac4c1cc9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:01:11.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.680+0000 7fac4a7fc640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fac34004810 0x7fac4c1cc470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:01:11.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.680+0000 7fac4b7fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fac340053d0 0x7fac4c1cc9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:40818/0 (socket says 172.21.15.42:40818) 2024-09-16T04:01:11.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.680+0000 7fac4b7fe640 1 -- 172.21.15.42:0/2079469076 learned_addr learned my addr 172.21.15.42:0/2079469076 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:01:11.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.680+0000 7fac4b7fe640 10 monclient(hunting): get_auth_request con 0x7fac340053d0 auth_method 0 2024-09-16T04:01:11.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.680+0000 7fac4b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:01:11.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.680+0000 7fac4b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:01:11.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.680+0000 7fac4b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:01:11.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.680+0000 7fac4a7fc640 10 monclient(hunting): get_auth_request con 0x7fac34004810 auth_method 0 2024-09-16T04:01:11.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.680+0000 7fac4a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:01:11.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.680+0000 7fac4a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:01:11.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.680+0000 7fac4a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:01:11.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.680+0000 7fac4b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:01:11.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.680+0000 7fac4b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:01:11.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.680+0000 7fac4b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:01:11.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.680+0000 7fac4a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:01:11.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.680+0000 7fac4a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:01:11.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.680+0000 7fac4a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:01:11.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.680+0000 7fac4b7fe640 10 monclient(hunting): handle_auth_done global_id 825339 payload 824 2024-09-16T04:01:11.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.680+0000 7fac4b7fe640 1 -- 172.21.15.42:0/2079469076 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fac340039f0 msgr2=0x7fac4c1cbf30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:01:11.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.680+0000 7fac4b7fe640 1 --2- 172.21.15.42:0/2079469076 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fac340039f0 0x7fac4c1cbf30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:11.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.680+0000 7fac4b7fe640 1 -- 172.21.15.42:0/2079469076 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fac34004810 msgr2=0x7fac4c1cc470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:01:11.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.680+0000 7fac4b7fe640 1 --2- 172.21.15.42:0/2079469076 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fac34004810 0x7fac4c1cc470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:11.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.680+0000 7fac4b7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:01:11.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.680+0000 7fac4b7fe640 1 monclient: found mon.noname-a 2024-09-16T04:01:11.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.680+0000 7fac4b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:01:11.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.680+0000 7fac4b7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:01:11.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.680+0000 7fac4b7fe640 1 -- 172.21.15.42:0/2079469076 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac4c121390 con 0x7fac340053d0 2024-09-16T04:01:11.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.684+0000 7fac4b7fe640 1 --2- 172.21.15.42:0/2079469076 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fac340053d0 0x7fac4c1cc9b0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fac44000eb0 tx=0x7fac44019ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:01:11.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.684+0000 7fac337fe640 10 client.?.objecter ms_handle_connect 0x7fac340053d0 2024-09-16T04:01:11.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.684+0000 7fac337fe640 10 client.?.objecter resend_mon_ops 2024-09-16T04:01:11.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.728+0000 7fac337fe640 1 -- 172.21.15.42:0/2079469076 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac440ab040 con 0x7fac340053d0 2024-09-16T04:01:11.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.728+0000 7fac337fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:01:11.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.728+0000 7fac337fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:01:11.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.728+0000 7fac337fe640 10 monclient: dump: 2024-09-16T04:01:11.735 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:01:11.735 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:01:11.735 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:11.735 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:11.735 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:01:11.735 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:01:11.735 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:01:11.735 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:01:11.735 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:01:11.735 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:01:11.736 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.728+0000 7fac337fe640 10 monclient: _finish_auth 0 2024-09-16T04:01:11.736 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.728+0000 7fac337fe640 10 monclient: _check_auth_tickets 2024-09-16T04:01:11.736 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.728+0000 7fac337fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:01:11.736 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.728+0000 7fac337fe640 1 -- 172.21.15.42:0/2079469076 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac440bf070 con 0x7fac340053d0 2024-09-16T04:01:11.736 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.728+0000 7fac51492640 5 monclient: authenticate success, global_id 825339 2024-09-16T04:01:11.736 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.728+0000 7fac337fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:01:11.736 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.728+0000 7fac337fe640 1 -- 172.21.15.42:0/2079469076 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac440b9640 con 0x7fac340053d0 2024-09-16T04:01:11.736 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.728+0000 7fac51492640 10 monclient: _renew_subs 2024-09-16T04:01:11.736 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.728+0000 7fac51492640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:01:11.736 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.728+0000 7fac51492640 1 -- 172.21.15.42:0/2079469076 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac4c1322e0 con 0x7fac340053d0 2024-09-16T04:01:11.736 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.728+0000 7fac497fa640 4 set_mon_vals no callback set 2024-09-16T04:01:11.737 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.728+0000 7fac337fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:01:11.737 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.728+0000 7fac337fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:01:11.737 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.728+0000 7fac337fe640 10 monclient: dump: 2024-09-16T04:01:11.737 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:01:11.737 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:01:11.737 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:11.737 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:11.737 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:01:11.737 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:01:11.737 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:01:11.737 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:01:11.737 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:01:11.737 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:01:11.737 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.728+0000 7fac51492640 10 client.825339.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:01:11.738 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.728+0000 7fac51492640 10 monclient: _renew_subs 2024-09-16T04:01:11.738 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.728+0000 7fac51492640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:01:11.738 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.728+0000 7fac51492640 1 -- 172.21.15.42:0/2079469076 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac4c1210b0 con 0x7fac340053d0 2024-09-16T04:01:11.738 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.728+0000 7fac51492640 1 librados: init done 2024-09-16T04:01:11.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.728+0000 7fac51492640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:01:11.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.728+0000 7fac51492640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:01:11.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.728+0000 7fac51492640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:01:11.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.728+0000 7fac51492640 1 -- 172.21.15.42:0/2079469076 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fac4c137990 con 0x7fac340053d0 2024-09-16T04:01:11.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.728+0000 7fac337fe640 1 -- 172.21.15.42:0/2079469076 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fac440b97e0 con 0x7fac340053d0 2024-09-16T04:01:11.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.728+0000 7fac337fe640 1 --2- 172.21.15.42:0/2079469076 >> v2:172.21.15.154:6812/254845342 conn(0x7fac380382a0 0x7fac3803a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:01:11.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.732+0000 7fac337fe640 1 -- 172.21.15.42:0/2079469076 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(570..570 src has 451..570) ==== 5843+0+0 (secure 0 0 0) 0x7fac440e4d30 con 0x7fac340053d0 2024-09-16T04:01:11.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.732+0000 7fac337fe640 10 client.825339.objecter ms_dispatch 0x7fac4c000ba0 osd_map(570..570 src has 451..570) 2024-09-16T04:01:11.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.732+0000 7fac337fe640 3 client.825339.objecter handle_osd_map got epochs [570,570] > 0 2024-09-16T04:01:11.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.732+0000 7fac337fe640 3 client.825339.objecter handle_osd_map decoding full epoch 570 2024-09-16T04:01:11.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.732+0000 7fac337fe640 20 client.825339.objecter dump_active .. 0 homeless 2024-09-16T04:01:11.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.732+0000 7fac4affd640 1 --2- 172.21.15.42:0/2079469076 >> v2:172.21.15.154:6812/254845342 conn(0x7fac380382a0 0x7fac3803a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:01:11.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.732+0000 7fac4affd640 10 monclient: get_auth_request con 0x7fac380382a0 auth_method 0 2024-09-16T04:01:11.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.732+0000 7fac337fe640 10 client.825339.objecter ms_handle_connect 0x7fac380382a0 2024-09-16T04:01:11.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.732+0000 7fac4affd640 1 --2- 172.21.15.42:0/2079469076 >> v2:172.21.15.154:6812/254845342 conn(0x7fac380382a0 0x7fac3803a760 secure :-1 s=READY pgs=3757 cs=0 l=1 rev1=1 crypto rx=0x7fac3c00c540 tx=0x7fac3c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:01:11.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.736+0000 7fac337fe640 1 -- 172.21.15.42:0/2079469076 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7fac3c00ce20 con 0x7fac340053d0 2024-09-16T04:01:11.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.736+0000 7fac337fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:01:11.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.736+0000 7fac337fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:01:11.985 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.976+0000 7fac51492640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:01:11.985 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.976+0000 7fac51492640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:01:11.985 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.976+0000 7fac51492640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:01:11.986 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.976+0000 7fac51492640 1 -- 172.21.15.42:0/2079469076 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fac4c0a5750 con 0x7fac340053d0 2024-09-16T04:01:11.986 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.980+0000 7fac337fe640 1 -- 172.21.15.42:0/2079469076 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1555 (secure 0 0 0) 0x7fac440b9ad0 con 0x7fac340053d0 2024-09-16T04:01:11.986 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.980+0000 7fac337fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:01:11.986 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.980+0000 7fac337fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:01:11.986 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":172,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:01:11.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.980+0000 7fac51492640 10 librados: watch_flush enter 2024-09-16T04:01:11.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.980+0000 7fac51492640 10 librados: watch_flush exit 2024-09-16T04:01:11.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.980+0000 7fac51492640 20 client.825339.objecter shutdown clearing up homeless session... 2024-09-16T04:01:11.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.980+0000 7fac51492640 10 client.825339.objecter successfully canceled tick 2024-09-16T04:01:11.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.980+0000 7fac51492640 1 -- 172.21.15.42:0/2079469076 >> v2:172.21.15.154:6812/254845342 conn(0x7fac380382a0 msgr2=0x7fac3803a760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:01:11.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.980+0000 7fac51492640 1 --2- 172.21.15.42:0/2079469076 >> v2:172.21.15.154:6812/254845342 conn(0x7fac380382a0 0x7fac3803a760 secure :-1 s=READY pgs=3757 cs=0 l=1 rev1=1 crypto rx=0x7fac3c00c540 tx=0x7fac3c008040 comp rx=0 tx=0).stop 2024-09-16T04:01:11.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.980+0000 7fac51492640 10 monclient: shutdown 2024-09-16T04:01:11.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.980+0000 7fac51492640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:01:11.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.980+0000 7fac51492640 1 -- 172.21.15.42:0/2079469076 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fac340053d0 msgr2=0x7fac4c1cc9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:01:11.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.980+0000 7fac51492640 1 --2- 172.21.15.42:0/2079469076 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fac340053d0 0x7fac4c1cc9b0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fac44000eb0 tx=0x7fac44019ab0 comp rx=0 tx=0).stop 2024-09-16T04:01:11.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.980+0000 7fac51492640 1 -- 172.21.15.42:0/2079469076 shutdown_connections 2024-09-16T04:01:11.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.980+0000 7fac51492640 1 --2- 172.21.15.42:0/2079469076 >> v2:172.21.15.154:6812/254845342 conn(0x7fac380382a0 0x7fac3803a760 unknown :-1 s=CLOSED pgs=3757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:11.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.980+0000 7fac51492640 1 --2- 172.21.15.42:0/2079469076 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fac340053d0 0x7fac4c1cc9b0 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:11.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.980+0000 7fac51492640 1 --2- 172.21.15.42:0/2079469076 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fac34004810 0x7fac4c1cc470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:11.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.980+0000 7fac51492640 1 --2- 172.21.15.42:0/2079469076 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fac340039f0 0x7fac4c1cbf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:11.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.980+0000 7fac51492640 1 -- 172.21.15.42:0/2079469076 >> 172.21.15.42:0/2079469076 conn(0x7fac4c0a9fb0 msgr2=0x7fac4c132830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:01:11.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.980+0000 7fac51492640 1 -- 172.21.15.42:0/2079469076 shutdown_connections 2024-09-16T04:01:11.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.980+0000 7fac51492640 1 -- 172.21.15.42:0/2079469076 wait complete. 2024-09-16T04:01:11.992 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:11.980+0000 7fac51492640 1 librados: shutdown 2024-09-16T04:01:12.006 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T04:01:12.006 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-16T04:01:12.006 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T04:01:12.126 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.120+0000 7f37ee10d640 10 monclient: get_monmap_and_config 2024-09-16T04:01:12.126 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.120+0000 7f37ee10d640 10 monclient: build_initial_monmap 2024-09-16T04:01:12.126 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.120+0000 7f37ee10d640 10 monclient: monmap: 2024-09-16T04:01:12.126 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:01:12.126 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:01:12.126 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:01:12.124723+0000 2024-09-16T04:01:12.126 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:01:12.124723+0000 2024-09-16T04:01:12.127 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:01:12.127 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:01:12.127 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:01:12.127 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:01:12.127 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:01:12.127 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:01:12.127 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:01:12.127 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:01:12.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.120+0000 7f37ee10d640 1 Processor -- start 2024-09-16T04:01:12.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.120+0000 7f37ee10d640 1 -- start start 2024-09-16T04:01:12.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.120+0000 7f37ee10d640 10 monclient: init 2024-09-16T04:01:12.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.120+0000 7f37ee10d640 10 monclient: _reopen_session rank -1 2024-09-16T04:01:12.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.120+0000 7f37ee10d640 10 monclient: _add_conns ranks=[4,0,3,1,2] 2024-09-16T04:01:12.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.120+0000 7f37ee10d640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f37e80a1330 0x7f37e809f180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:01:12.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.120+0000 7f37ee10d640 10 monclient(hunting): picked mon.noname-c con 0x7f37e80a1330 addr v2:172.21.15.154:3300/0 2024-09-16T04:01:12.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.120+0000 7f37ee10d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f37e80a1710 addr v1:172.21.15.42:6789/0 2024-09-16T04:01:12.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.120+0000 7f37ee10d640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f37e81393a0 0x7f37e813b790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:01:12.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.120+0000 7f37ee10d640 10 monclient(hunting): picked mon.noname-a con 0x7f37e81393a0 addr v2:172.21.15.42:3300/0 2024-09-16T04:01:12.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.120+0000 7f37ee10d640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f37e809e4c0 con 0x7f37e80a1710 2024-09-16T04:01:12.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.120+0000 7f37ee10d640 10 monclient(hunting): start opening mon connection 2024-09-16T04:01:12.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.120+0000 7f37ee10d640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f37e809edf0 con 0x7f37e81393a0 2024-09-16T04:01:12.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.120+0000 7f37ee10d640 10 monclient(hunting): start opening mon connection 2024-09-16T04:01:12.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.120+0000 7f37ee10d640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f37e809cd60 con 0x7f37e80a1330 2024-09-16T04:01:12.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.120+0000 7f37ee10d640 10 monclient(hunting): _renew_subs 2024-09-16T04:01:12.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.120+0000 7f37ee10d640 10 monclient(hunting): authenticate will time out at 2119.094119s 2024-09-16T04:01:12.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.120+0000 7f37e7fff640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f37e81393a0 0x7f37e813b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:01:12.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.120+0000 7f37e6ffd640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f37e80a1710 0x7f37e80a1b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:45682/0 (socket says 172.21.15.42:45682) 2024-09-16T04:01:12.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.120+0000 7f37e7fff640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f37e81393a0 0x7f37e813b790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:40832/0 (socket says 172.21.15.42:40832) 2024-09-16T04:01:12.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.120+0000 7f37e6ffd640 1 -- 172.21.15.42:0/367083717 learned_addr learned my addr 172.21.15.42:0/367083717 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T04:01:12.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.120+0000 7f37e7fff640 10 monclient(hunting): get_auth_request con 0x7f37e81393a0 auth_method 0 2024-09-16T04:01:12.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.120+0000 7f37e7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:01:12.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.120+0000 7f37e7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:01:12.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.120+0000 7f37e7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:01:12.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.120+0000 7f37e77fe640 1 --2- 172.21.15.42:0/367083717 >> v2:172.21.15.154:3300/0 conn(0x7f37e80a1330 0x7f37e809f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:01:12.131 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.120+0000 7f37e7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:01:12.131 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.120+0000 7f37e7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:01:12.131 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.120+0000 7f37e7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:01:12.131 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.120+0000 7f37e77fe640 10 monclient(hunting): get_auth_request con 0x7f37e80a1330 auth_method 0 2024-09-16T04:01:12.131 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.120+0000 7f37e77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:01:12.131 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.120+0000 7f37e77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:01:12.131 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.120+0000 7f37e77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:01:12.131 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.120+0000 7f37e7fff640 10 monclient(hunting): handle_auth_done global_id 825342 payload 274 2024-09-16T04:01:12.131 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.120+0000 7f37e7fff640 1 -- 172.21.15.42:0/367083717 >> v2:172.21.15.154:3300/0 conn(0x7f37e80a1330 msgr2=0x7f37e809f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:01:12.131 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.120+0000 7f37e7fff640 1 --2- 172.21.15.42:0/367083717 >> v2:172.21.15.154:3300/0 conn(0x7f37e80a1330 0x7f37e809f180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:12.131 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.120+0000 7f37e7fff640 1 -- 172.21.15.42:0/367083717 >> v1:172.21.15.42:6789/0 conn(0x7f37e80a1710 legacy=0x7f37e80a1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:01:12.131 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.120+0000 7f37e7fff640 10 monclient: _finish_hunting 0 2024-09-16T04:01:12.131 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.120+0000 7f37e7fff640 1 monclient: found mon.noname-a 2024-09-16T04:01:12.131 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.120+0000 7f37e7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:01:12.132 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.120+0000 7f37e7fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:01:12.132 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.120+0000 7f37e7fff640 1 -- 172.21.15.42:0/367083717 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37e809f6c0 con 0x7f37e81393a0 2024-09-16T04:01:12.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.124+0000 7f37e7fff640 1 --2- 172.21.15.42:0/367083717 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f37e81393a0 0x7f37e813b790 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f37e806add0 tx=0x7f37e001c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e16b224cfb890a9e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:01:12.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.124+0000 7f37e67fc640 1 -- 172.21.15.42:0/367083717 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37e00b2070 con 0x7f37e81393a0 2024-09-16T04:01:12.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.124+0000 7f37e67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:01:12.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.124+0000 7f37e67fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:01:12.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.124+0000 7f37e67fc640 10 monclient: dump: 2024-09-16T04:01:12.138 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:01:12.138 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:01:12.138 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:12.138 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:12.138 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:01:12.139 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:01:12.139 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:01:12.139 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:01:12.139 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:01:12.139 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:01:12.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.124+0000 7f37e67fc640 10 monclient: _finish_auth 0 2024-09-16T04:01:12.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.124+0000 7f37e67fc640 10 monclient: _check_auth_tickets 2024-09-16T04:01:12.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.124+0000 7f37e67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:01:12.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.124+0000 7f37ee10d640 5 monclient: authenticate success, global_id 825342 2024-09-16T04:01:12.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.124+0000 7f37e67fc640 1 -- 172.21.15.42:0/367083717 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f37e00a57c0 con 0x7f37e81393a0 2024-09-16T04:01:12.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.124+0000 7f37e67fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:01:12.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.124+0000 7f37ee10d640 10 monclient: get_monmap_and_config success 2024-09-16T04:01:12.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.124+0000 7f37ee10d640 4 set_mon_vals no callback set 2024-09-16T04:01:12.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.124+0000 7f37e67fc640 1 -- 172.21.15.42:0/367083717 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37e00a5b90 con 0x7f37e81393a0 2024-09-16T04:01:12.140 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.124+0000 7f37e67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:01:12.140 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.124+0000 7f37e67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:01:12.140 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.124+0000 7f37e67fc640 10 monclient: dump: 2024-09-16T04:01:12.140 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:01:12.140 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:01:12.140 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:12.140 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:12.140 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:01:12.140 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:01:12.140 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:01:12.140 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:01:12.140 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:01:12.140 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:01:12.140 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.124+0000 7f37ee10d640 10 monclient: shutdown 2024-09-16T04:01:12.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.124+0000 7f37ee10d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:01:12.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.124+0000 7f37ee10d640 1 -- 172.21.15.42:0/367083717 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f37e81393a0 msgr2=0x7f37e813b790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:01:12.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.124+0000 7f37ee10d640 1 --2- 172.21.15.42:0/367083717 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f37e81393a0 0x7f37e813b790 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f37e806add0 tx=0x7f37e001c7f0 comp rx=0 tx=0).stop 2024-09-16T04:01:12.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.124+0000 7f37ee10d640 1 -- 172.21.15.42:0/367083717 shutdown_connections 2024-09-16T04:01:12.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.124+0000 7f37ee10d640 1 --2- 172.21.15.42:0/367083717 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f37e81393a0 0x7f37e813b790 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:12.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.124+0000 7f37ee10d640 1 --2- 172.21.15.42:0/367083717 >> v2:172.21.15.154:3300/0 conn(0x7f37e80a1330 0x7f37e809f180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:12.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.124+0000 7f37ee10d640 1 -- 172.21.15.42:0/367083717 >> 172.21.15.42:0/367083717 conn(0x7f37e80a2420 msgr2=0x7f37e80a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:01:12.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.124+0000 7f37ee10d640 1 -- 172.21.15.42:0/367083717 shutdown_connections 2024-09-16T04:01:12.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.124+0000 7f37ee10d640 1 -- 172.21.15.42:0/367083717 wait complete. 2024-09-16T04:01:12.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.124+0000 7f37ee10d640 10 monclient: build_initial_monmap 2024-09-16T04:01:12.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.124+0000 7f37ee10d640 10 monclient: monmap: 2024-09-16T04:01:12.141 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:01:12.141 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:01:12.141 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:01:12.142 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:01:12.142 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:01:12.142 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:01:12.142 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:01:12.142 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:01:12.142 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:01:12.142 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:01:12.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.124+0000 7f37ee10d640 1 librados: starting msgr at 2024-09-16T04:01:12.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.124+0000 7f37ee10d640 1 librados: starting objecter 2024-09-16T04:01:12.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.124+0000 7f37ee10d640 1 Processor -- start 2024-09-16T04:01:12.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.124+0000 7f37ee10d640 1 -- start start 2024-09-16T04:01:12.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.124+0000 7f37ee10d640 1 librados: setting wanted keys 2024-09-16T04:01:12.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.124+0000 7f37ee10d640 1 librados: calling monclient init 2024-09-16T04:01:12.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.124+0000 7f37ee10d640 10 monclient: init 2024-09-16T04:01:12.143 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.124+0000 7f37ee10d640 10 monclient: _reopen_session rank -1 2024-09-16T04:01:12.143 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.124+0000 7f37ee10d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:01:12.143 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.124+0000 7f37ee10d640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f37e80a1330 0x7f37e81ccf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:01:12.143 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.124+0000 7f37ee10d640 10 monclient(hunting): picked mon.noname-c con 0x7f37e80a1330 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:01:12.143 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.124+0000 7f37ee10d640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f37e80a1710 0x7f37e8134480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:01:12.143 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.124+0000 7f37ee10d640 10 monclient(hunting): picked mon.noname-a con 0x7f37e80a1710 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:01:12.143 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.124+0000 7f37ee10d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f37e81393a0 0x7f37e81349c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:01:12.143 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.124+0000 7f37ee10d640 10 monclient(hunting): picked mon.noname-b con 0x7f37e81393a0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:01:12.143 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.124+0000 7f37ee10d640 10 monclient(hunting): start opening mon connection 2024-09-16T04:01:12.143 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.124+0000 7f37ee10d640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f37e813e880 con 0x7f37e80a1710 2024-09-16T04:01:12.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.124+0000 7f37ee10d640 10 monclient(hunting): start opening mon connection 2024-09-16T04:01:12.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.124+0000 7f37ee10d640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f37e813e700 con 0x7f37e81393a0 2024-09-16T04:01:12.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.124+0000 7f37ee10d640 10 monclient(hunting): start opening mon connection 2024-09-16T04:01:12.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.124+0000 7f37ee10d640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f37e813ea00 con 0x7f37e80a1330 2024-09-16T04:01:12.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.124+0000 7f37ee10d640 10 monclient(hunting): _renew_subs 2024-09-16T04:01:12.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.124+0000 7f37ee10d640 10 monclient(hunting): authenticate will time out at 2119.099463s 2024-09-16T04:01:12.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.124+0000 7f37e77fe640 1 -- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f37e80a1330 msgr2=0x7f37e81ccf40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T04:01:12.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.124+0000 7f37e77fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f37e80a1330 0x7f37e81ccf40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:01:12.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.124+0000 7f37e6ffd640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f37e80a1710 0x7f37e8134480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:01:12.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.124+0000 7f37e6ffd640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f37e80a1710 0x7f37e8134480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:40840/0 (socket says 172.21.15.42:40840) 2024-09-16T04:01:12.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.124+0000 7f37e6ffd640 1 -- 172.21.15.42:0/1075309647 learned_addr learned my addr 172.21.15.42:0/1075309647 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:01:12.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.124+0000 7f37e6ffd640 10 monclient(hunting): get_auth_request con 0x7f37e80a1710 auth_method 0 2024-09-16T04:01:12.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.124+0000 7f37e6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:01:12.145 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.124+0000 7f37e6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:01:12.145 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.124+0000 7f37e6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:01:12.145 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.124+0000 7f37e6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:01:12.145 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.124+0000 7f37e6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:01:12.145 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.124+0000 7f37e6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:01:12.145 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.124+0000 7f37e7fff640 1 --2- 172.21.15.42:0/1075309647 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f37e81393a0 0x7f37e81349c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:01:12.145 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.124+0000 7f37e7fff640 10 monclient(hunting): get_auth_request con 0x7f37e81393a0 auth_method 0 2024-09-16T04:01:12.145 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.124+0000 7f37e7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:01:12.145 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.124+0000 7f37e7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:01:12.145 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.124+0000 7f37e7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:01:12.145 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.124+0000 7f37e6ffd640 10 monclient(hunting): handle_auth_done global_id 825348 payload 824 2024-09-16T04:01:12.145 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.124+0000 7f37e6ffd640 1 -- 172.21.15.42:0/1075309647 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f37e80a1330 msgr2=0x7f37e81ccf40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:01:12.145 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.124+0000 7f37e6ffd640 1 --2- 172.21.15.42:0/1075309647 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f37e80a1330 0x7f37e81ccf40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:12.145 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.124+0000 7f37e6ffd640 1 -- 172.21.15.42:0/1075309647 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f37e81393a0 msgr2=0x7f37e81349c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:01:12.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.124+0000 7f37e6ffd640 1 --2- 172.21.15.42:0/1075309647 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f37e81393a0 0x7f37e81349c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:12.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.124+0000 7f37e6ffd640 10 monclient: _finish_hunting 0 2024-09-16T04:01:12.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.124+0000 7f37e6ffd640 1 monclient: found mon.noname-a 2024-09-16T04:01:12.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.124+0000 7f37e6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:01:12.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.124+0000 7f37e6ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:01:12.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.124+0000 7f37e6ffd640 1 -- 172.21.15.42:0/1075309647 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37e811de00 con 0x7f37e80a1710 2024-09-16T04:01:12.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.128+0000 7f37e6ffd640 1 --2- 172.21.15.42:0/1075309647 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f37e80a1710 0x7f37e8134480 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f37dc00c420 tx=0x7f37dc00c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:01:12.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.128+0000 7f37cffff640 10 client.?.objecter ms_handle_connect 0x7f37e80a1710 2024-09-16T04:01:12.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.128+0000 7f37cffff640 10 client.?.objecter resend_mon_ops 2024-09-16T04:01:12.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.128+0000 7f37cffff640 1 -- 172.21.15.42:0/1075309647 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37dc017070 con 0x7f37e80a1710 2024-09-16T04:01:12.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.128+0000 7f37cffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:01:12.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.128+0000 7f37cffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:01:12.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.128+0000 7f37cffff640 10 monclient: dump: 2024-09-16T04:01:12.146 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:01:12.147 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:01:12.147 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:12.147 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:12.147 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:01:12.147 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:01:12.147 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:01:12.147 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:01:12.147 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:01:12.147 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:01:12.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.128+0000 7f37cffff640 10 monclient: _finish_auth 0 2024-09-16T04:01:12.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.128+0000 7f37cffff640 10 monclient: _check_auth_tickets 2024-09-16T04:01:12.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.128+0000 7f37cffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:01:12.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.128+0000 7f37cffff640 1 -- 172.21.15.42:0/1075309647 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f37dc005cf0 con 0x7f37e80a1710 2024-09-16T04:01:12.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.128+0000 7f37ee10d640 5 monclient: authenticate success, global_id 825348 2024-09-16T04:01:12.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.128+0000 7f37ee10d640 10 monclient: _renew_subs 2024-09-16T04:01:12.148 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.128+0000 7f37ee10d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:01:12.148 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.128+0000 7f37ee10d640 1 -- 172.21.15.42:0/1075309647 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37e813b3b0 con 0x7f37e80a1710 2024-09-16T04:01:12.148 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.128+0000 7f37cffff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:01:12.148 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.128+0000 7f37cffff640 1 -- 172.21.15.42:0/1075309647 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37dc0072f0 con 0x7f37e80a1710 2024-09-16T04:01:12.148 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.128+0000 7f37cffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:01:12.148 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.128+0000 7f37cffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:01:12.148 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.128+0000 7f37cffff640 10 monclient: dump: 2024-09-16T04:01:12.148 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:01:12.148 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:01:12.148 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:12.148 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:12.148 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:01:12.148 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:01:12.148 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:01:12.149 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:01:12.149 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:01:12.149 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:01:12.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.128+0000 7f37ee10d640 10 client.825348.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:01:12.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.128+0000 7f37ee10d640 10 monclient: _renew_subs 2024-09-16T04:01:12.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.128+0000 7f37ee10d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:01:12.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.128+0000 7f37ee10d640 1 -- 172.21.15.42:0/1075309647 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37e811db20 con 0x7f37e80a1710 2024-09-16T04:01:12.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.128+0000 7f37e5ffb640 4 set_mon_vals no callback set 2024-09-16T04:01:12.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.128+0000 7f37ee10d640 1 librados: init done 2024-09-16T04:01:12.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.128+0000 7f37ee10d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:01:12.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.128+0000 7f37ee10d640 1 --2- 172.21.15.42:0/1075309647 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f37e8071010 0x7f37e802c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:01:12.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.128+0000 7f37ee10d640 10 monclient(hunting): start opening mon connection 2024-09-16T04:01:12.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.128+0000 7f37ee10d640 1 -- 172.21.15.42:0/1075309647 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f37e8073a30 con 0x7f37e8071010 2024-09-16T04:01:12.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.128+0000 7f37e77fe640 1 --2- 172.21.15.42:0/1075309647 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f37e8071010 0x7f37e802c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:01:12.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.128+0000 7f37e77fe640 10 monclient: get_auth_request con 0x7f37e8071010 auth_method 0 2024-09-16T04:01:12.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.128+0000 7f37e77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:01:12.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.128+0000 7f37e77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:01:12.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.128+0000 7f37e77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:01:12.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.128+0000 7f37cffff640 1 -- 172.21.15.42:0/1075309647 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f37dc01a070 con 0x7f37e80a1710 2024-09-16T04:01:12.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.128+0000 7f37e77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:01:12.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.128+0000 7f37e77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:01:12.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.128+0000 7f37e77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:01:12.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.128+0000 7f37e77fe640 10 monclient(hunting): handle_auth_done global_id 825351 payload 824 2024-09-16T04:01:12.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.128+0000 7f37cffff640 1 --2- 172.21.15.42:0/1075309647 >> v2:172.21.15.154:6812/254845342 conn(0x7f37d40377f0 0x7f37d4039cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:01:12.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.128+0000 7f37e77fe640 1 -- 172.21.15.42:0/1075309647 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f37e8138aa0 con 0x7f37e8071010 2024-09-16T04:01:12.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.128+0000 7f37cffff640 1 -- 172.21.15.42:0/1075309647 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(570..570 src has 451..570) ==== 5843+0+0 (secure 0 0 0) 0x7f37dc0483e0 con 0x7f37e80a1710 2024-09-16T04:01:12.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.128+0000 7f37cffff640 10 client.825348.objecter ms_dispatch 0x7f37e8000ba0 osd_map(570..570 src has 451..570) 2024-09-16T04:01:12.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.128+0000 7f37cffff640 3 client.825348.objecter handle_osd_map got epochs [570,570] > 0 2024-09-16T04:01:12.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.128+0000 7f37cffff640 3 client.825348.objecter handle_osd_map decoding full epoch 570 2024-09-16T04:01:12.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.128+0000 7f37cffff640 20 client.825348.objecter dump_active .. 0 homeless 2024-09-16T04:01:12.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.128+0000 7f37cffff640 10 client.825348.objecter ms_handle_connect 0x7f37e8071010 2024-09-16T04:01:12.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.128+0000 7f37cffff640 10 client.825348.objecter resend_mon_ops 2024-09-16T04:01:12.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.128+0000 7f37e77fe640 1 --2- 172.21.15.42:0/1075309647 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f37e8071010 0x7f37e802c650 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f37d8000f30 tx=0x7f37d800eba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:01:12.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.128+0000 7f37cffff640 1 -- 172.21.15.42:0/1075309647 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37d8017020 con 0x7f37e8071010 2024-09-16T04:01:12.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.128+0000 7f37cffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:01:12.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.128+0000 7f37cffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:01:12.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.128+0000 7f37cffff640 10 monclient: dump: 2024-09-16T04:01:12.151 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:01:12.151 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:01:12.151 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:12.151 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:12.151 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:01:12.152 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:01:12.152 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:01:12.152 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:01:12.152 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:01:12.152 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:01:12.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.128+0000 7f37e7fff640 1 --2- 172.21.15.42:0/1075309647 >> v2:172.21.15.154:6812/254845342 conn(0x7f37d40377f0 0x7f37d4039cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:01:12.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.128+0000 7f37e7fff640 10 monclient: get_auth_request con 0x7f37d40377f0 auth_method 0 2024-09-16T04:01:12.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.128+0000 7f37cffff640 1 -- 172.21.15.42:0/1075309647 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f37d800c740 con 0x7f37e8071010 2024-09-16T04:01:12.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.128+0000 7f37cffff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:01:12.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.128+0000 7f37cffff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:01:12.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.128+0000 7f37cffff640 1 -- 172.21.15.42:0/1075309647 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f37e8071010 msgr2=0x7f37e802c650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:01:12.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.128+0000 7f37cffff640 1 --2- 172.21.15.42:0/1075309647 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f37e8071010 0x7f37e802c650 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f37d8000f30 tx=0x7f37d800eba0 comp rx=0 tx=0).stop 2024-09-16T04:01:12.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.128+0000 7f37cffff640 1 -- 172.21.15.42:0/1075309647 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f37e8071010 msgr2=0x7f37e802c650 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:01:12.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.128+0000 7f37cffff640 1 --2- 172.21.15.42:0/1075309647 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f37e8071010 0x7f37e802c650 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:12.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.128+0000 7f37e7fff640 1 --2- 172.21.15.42:0/1075309647 >> v2:172.21.15.154:6812/254845342 conn(0x7f37d40377f0 0x7f37d4039cb0 secure :-1 s=READY pgs=3758 cs=0 l=1 rev1=1 crypto rx=0x7f37e00af000 tx=0x7f37e00a7070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:01:12.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.128+0000 7f37cffff640 10 client.825348.objecter ms_handle_connect 0x7f37d40377f0 2024-09-16T04:01:12.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.136+0000 7f37ee10d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:01:12.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.136+0000 7f37ee10d640 1 --2- 172.21.15.42:0/1075309647 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f37e80713f0 0x7f37e802cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:01:12.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.136+0000 7f37ee10d640 10 monclient(hunting): start opening mon connection 2024-09-16T04:01:12.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.136+0000 7f37ee10d640 1 -- 172.21.15.42:0/1075309647 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f37e80a55e0 con 0x7f37e80713f0 2024-09-16T04:01:12.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.136+0000 7f37e77fe640 1 --2- 172.21.15.42:0/1075309647 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f37e80713f0 0x7f37e802cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:01:12.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.136+0000 7f37e77fe640 10 monclient: get_auth_request con 0x7f37e80713f0 auth_method 0 2024-09-16T04:01:12.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.136+0000 7f37e77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:01:12.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.136+0000 7f37e77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:01:12.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.136+0000 7f37e77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:01:12.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.136+0000 7f37e77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:01:12.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.136+0000 7f37e77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:01:12.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.136+0000 7f37e77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:01:12.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.136+0000 7f37e77fe640 10 monclient(hunting): handle_auth_done global_id 825354 payload 824 2024-09-16T04:01:12.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.136+0000 7f37e77fe640 1 -- 172.21.15.42:0/1075309647 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f37e811d660 con 0x7f37e80713f0 2024-09-16T04:01:12.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.136+0000 7f37cffff640 10 client.825348.objecter ms_handle_connect 0x7f37e80713f0 2024-09-16T04:01:12.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.136+0000 7f37e77fe640 1 --2- 172.21.15.42:0/1075309647 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f37e80713f0 0x7f37e802cb90 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f37d8000f30 tx=0x7f37d8004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:01:12.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.136+0000 7f37cffff640 10 client.825348.objecter resend_mon_ops 2024-09-16T04:01:12.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.136+0000 7f37cffff640 1 -- 172.21.15.42:0/1075309647 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37d8017040 con 0x7f37e80713f0 2024-09-16T04:01:12.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.136+0000 7f37cffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:01:12.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.136+0000 7f37cffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:01:12.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.136+0000 7f37cffff640 10 monclient: dump: 2024-09-16T04:01:12.156 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:01:12.156 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:01:12.156 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:12.156 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:12.156 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:01:12.156 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:01:12.156 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:01:12.156 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:01:12.156 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:01:12.156 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:01:12.157 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.136+0000 7f37cffff640 1 -- 172.21.15.42:0/1075309647 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4704 (secure 0 0 0) 0x7f37d800ce90 con 0x7f37e80713f0 2024-09-16T04:01:12.157 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.136+0000 7f37cffff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:01:12.157 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.136+0000 7f37cffff640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:01:12.157 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.136+0000 7f37cffff640 1 -- 172.21.15.42:0/1075309647 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f37e80713f0 msgr2=0x7f37e802cb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:01:12.157 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.136+0000 7f37cffff640 1 --2- 172.21.15.42:0/1075309647 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f37e80713f0 0x7f37e802cb90 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f37d8000f30 tx=0x7f37d8004740 comp rx=0 tx=0).stop 2024-09-16T04:01:12.157 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.136+0000 7f37cffff640 1 -- 172.21.15.42:0/1075309647 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f37e80713f0 msgr2=0x7f37e802cb90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:01:12.157 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.136+0000 7f37cffff640 1 --2- 172.21.15.42:0/1075309647 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f37e80713f0 0x7f37e802cb90 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:12.157 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T04:01:12.157 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:01:12.157 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:01:12.157 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "leader", 2024-09-16T04:01:12.158 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 172, 2024-09-16T04:01:12.158 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 561990, 2024-09-16T04:01:12.158 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T04:01:12.158 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T04:01:12.158 INFO:teuthology.orchestra.run.smithi042.stdout: 1 2024-09-16T04:01:12.158 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:01:12.158 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 0, 2024-09-16T04:01:12.158 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:01:12.158 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T04:01:12.158 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T04:01:12.158 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:01:12.158 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:01:12.158 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:01:12.158 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:01:12.159 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:01:12.159 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:01:12.159 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:01:12.159 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:01:12.159 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:01:12.159 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:01:12.159 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:01:12.159 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:01:12.159 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:01:12.159 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T04:01:12.159 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:01:12.159 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:01:12.159 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:01:12.159 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:01:12.160 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:01:12.160 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:01:12.160 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:01:12.160 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:01:12.160 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:01:12.160 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:01:12.160 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:01:12.160 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:01:12.160 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:01:12.160 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T04:01:12.160 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T04:01:12.160 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T04:01:12.160 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T04:01:12.160 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T04:01:12.161 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T04:01:12.161 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:01:12.161 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:01:12.161 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T04:01:12.161 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T04:01:12.161 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T04:01:12.161 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T04:01:12.161 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T04:01:12.161 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T04:01:12.161 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T04:01:12.161 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:01:12.161 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T04:01:12.161 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:01:12.161 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:01:12.162 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:01:12.162 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:01:12.162 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:01:12.162 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:01:12.162 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:01:12.162 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:01:12.162 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:01:12.162 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:01:12.162 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:01:12.162 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:01:12.162 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T04:01:12.162 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:01:12.162 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T04:01:12.162 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:01:12.163 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:01:12.163 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:01:12.163 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:01:12.163 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:01:12.163 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:01:12.163 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:01:12.163 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T04:01:12.163 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:01:12.163 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:01:12.163 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:01:12.163 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:01:12.163 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T04:01:12.163 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:01:12.163 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:01:12.163 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:01:12.164 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:01:12.164 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T04:01:12.164 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T04:01:12.164 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:01:12.164 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:01:12.164 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:01:12.164 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:01:12.164 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:01:12.164 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:01:12.164 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:01:12.164 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:01:12.164 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:01:12.164 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:01:12.164 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:01:12.165 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T04:01:12.165 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:01:12.165 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:01:12.165 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:01:12.165 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:01:12.165 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T04:01:12.165 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:01:12.165 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:01:12.165 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:01:12.165 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:01:12.165 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T04:01:12.165 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T04:01:12.165 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:01:12.165 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:01:12.166 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:01:12.166 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:01:12.166 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:01:12.166 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:01:12.166 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:01:12.166 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:01:12.166 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:01:12.166 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:01:12.166 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:01:12.166 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T04:01:12.166 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:01:12.166 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:01:12.166 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:01:12.166 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:01:12.166 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T04:01:12.167 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:01:12.167 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:01:12.167 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:01:12.167 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:01:12.167 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T04:01:12.167 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T04:01:12.167 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:01:12.167 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:01:12.167 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:01:12.167 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:01:12.167 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:01:12.167 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:01:12.167 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T04:01:12.167 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T04:01:12.168 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:01:12.168 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:01:12.168 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:01:12.168 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:01:12.168 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:01:12.168 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:01:12.168 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T04:01:12.168 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:01:12.168 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:01:12.168 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:01:12.168 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 5 2024-09-16T04:01:12.168 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:01:12.168 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:01:12.168 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T04:01:12.169 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:01:12.169 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:01:12.169 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:01:12.169 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 7 2024-09-16T04:01:12.169 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:01:12.169 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:01:12.169 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:01:12.169 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T04:01:12.169 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T04:01:12.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.136+0000 7f37ee10d640 10 librados: watch_flush enter 2024-09-16T04:01:12.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.136+0000 7f37ee10d640 10 librados: watch_flush exit 2024-09-16T04:01:12.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.136+0000 7f37ee10d640 20 client.825348.objecter shutdown clearing up homeless session... 2024-09-16T04:01:12.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.136+0000 7f37ee10d640 10 client.825348.objecter successfully canceled tick 2024-09-16T04:01:12.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.136+0000 7f37ee10d640 1 -- 172.21.15.42:0/1075309647 >> v2:172.21.15.154:6812/254845342 conn(0x7f37d40377f0 msgr2=0x7f37d4039cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:01:12.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.136+0000 7f37ee10d640 1 --2- 172.21.15.42:0/1075309647 >> v2:172.21.15.154:6812/254845342 conn(0x7f37d40377f0 0x7f37d4039cb0 secure :-1 s=READY pgs=3758 cs=0 l=1 rev1=1 crypto rx=0x7f37e00af000 tx=0x7f37e00a7070 comp rx=0 tx=0).stop 2024-09-16T04:01:12.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.136+0000 7f37ee10d640 10 monclient: shutdown 2024-09-16T04:01:12.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.136+0000 7f37ee10d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:01:12.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.136+0000 7f37ee10d640 1 -- 172.21.15.42:0/1075309647 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f37e80a1710 msgr2=0x7f37e8134480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:01:12.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.136+0000 7f37ee10d640 1 --2- 172.21.15.42:0/1075309647 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f37e80a1710 0x7f37e8134480 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f37dc00c420 tx=0x7f37dc00c8e0 comp rx=0 tx=0).stop 2024-09-16T04:01:12.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.136+0000 7f37e7fff640 1 -- 172.21.15.42:0/1075309647 reap_dead start 2024-09-16T04:01:12.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.136+0000 7f37e7fff640 1 -- 172.21.15.42:0/1075309647 reap_dead start 2024-09-16T04:01:12.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.136+0000 7f37ee10d640 1 -- 172.21.15.42:0/1075309647 shutdown_connections 2024-09-16T04:01:12.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.136+0000 7f37ee10d640 1 -- 172.21.15.42:0/1075309647 >> 172.21.15.42:0/1075309647 conn(0x7f37e80a2420 msgr2=0x7f37e80a4040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:01:12.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.136+0000 7f37ee10d640 1 -- 172.21.15.42:0/1075309647 shutdown_connections 2024-09-16T04:01:12.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.136+0000 7f37ee10d640 1 -- 172.21.15.42:0/1075309647 wait complete. 2024-09-16T04:01:12.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.136+0000 7f37ee10d640 1 librados: shutdown 2024-09-16T04:01:12.172 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T04:01:12.172 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-16T04:01:12.172 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T04:01:12.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.288+0000 7f4e97fb3640 10 monclient: get_monmap_and_config 2024-09-16T04:01:12.298 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.292+0000 7f4e97fb3640 10 monclient: build_initial_monmap 2024-09-16T04:01:12.298 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.292+0000 7f4e97fb3640 10 monclient: monmap: 2024-09-16T04:01:12.298 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:01:12.298 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:01:12.298 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:01:12.296164+0000 2024-09-16T04:01:12.298 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:01:12.296164+0000 2024-09-16T04:01:12.298 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:01:12.298 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:01:12.298 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:01:12.298 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:01:12.299 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:01:12.299 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:01:12.299 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:01:12.299 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:01:12.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.292+0000 7f4e97fb3640 1 Processor -- start 2024-09-16T04:01:12.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.292+0000 7f4e97fb3640 1 -- start start 2024-09-16T04:01:12.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.292+0000 7f4e97fb3640 10 monclient: init 2024-09-16T04:01:12.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.292+0000 7f4e97fb3640 10 monclient: _reopen_session rank -1 2024-09-16T04:01:12.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.292+0000 7f4e97fb3640 10 monclient: _add_conns ranks=[2,4,3,0,1] 2024-09-16T04:01:12.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.292+0000 7f4e97fb3640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4e900a1330 0x7f4e900a1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:01:12.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.292+0000 7f4e97fb3640 10 monclient(hunting): picked mon.noname-b con 0x7f4e900a1330 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:01:12.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.292+0000 7f4e97fb3640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f4e9006dac0 0x7f4e900a1c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:01:12.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.292+0000 7f4e97fb3640 10 monclient(hunting): picked mon.noname-c con 0x7f4e9006dac0 addr v2:172.21.15.154:3300/0 2024-09-16T04:01:12.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.292+0000 7f4e97fb3640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f4e90135100 0x7f4e90135550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:01:12.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.292+0000 7f4e97fb3640 10 monclient(hunting): picked mon.noname-a con 0x7f4e90135100 addr v2:172.21.15.42:3300/0 2024-09-16T04:01:12.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.292+0000 7f4e97fb3640 10 monclient(hunting): start opening mon connection 2024-09-16T04:01:12.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.292+0000 7f4e97fb3640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f4e9009c7d0 con 0x7f4e90135100 2024-09-16T04:01:12.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.292+0000 7f4e97fb3640 10 monclient(hunting): start opening mon connection 2024-09-16T04:01:12.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.292+0000 7f4e97fb3640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f4e9009c090 con 0x7f4e9006dac0 2024-09-16T04:01:12.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.292+0000 7f4e97fb3640 10 monclient(hunting): start opening mon connection 2024-09-16T04:01:12.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.292+0000 7f4e97fb3640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f4e9009f610 con 0x7f4e900a1330 2024-09-16T04:01:12.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.292+0000 7f4e97fb3640 10 monclient(hunting): _renew_subs 2024-09-16T04:01:12.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.292+0000 7f4e97fb3640 10 monclient(hunting): authenticate will time out at 2119.265588s 2024-09-16T04:01:12.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.292+0000 7f4e95d4f640 1 -- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4e900a1330 msgr2=0x7f4e900a1710 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T04:01:12.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.292+0000 7f4e95d4f640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4e900a1330 0x7f4e900a1710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:01:12.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.292+0000 7f4e96550640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f4e90135100 0x7f4e90135550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:01:12.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.292+0000 7f4e96550640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f4e90135100 0x7f4e90135550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:40870/0 (socket says 172.21.15.42:40870) 2024-09-16T04:01:12.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.292+0000 7f4e96550640 1 -- 172.21.15.42:0/239647835 learned_addr learned my addr 172.21.15.42:0/239647835 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:01:12.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.292+0000 7f4e96550640 10 monclient(hunting): get_auth_request con 0x7f4e90135100 auth_method 0 2024-09-16T04:01:12.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.292+0000 7f4e96550640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:01:12.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.292+0000 7f4e96550640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:01:12.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.292+0000 7f4e96550640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:01:12.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.292+0000 7f4e96550640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:01:12.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.292+0000 7f4e96550640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:01:12.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.292+0000 7f4e9554e640 1 --2- 172.21.15.42:0/239647835 >> v2:172.21.15.154:3300/0 conn(0x7f4e9006dac0 0x7f4e900a1c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:01:12.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.292+0000 7f4e96550640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:01:12.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.292+0000 7f4e9554e640 10 monclient(hunting): get_auth_request con 0x7f4e9006dac0 auth_method 0 2024-09-16T04:01:12.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.292+0000 7f4e9554e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:01:12.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.292+0000 7f4e9554e640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:01:12.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.292+0000 7f4e9554e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:01:12.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.292+0000 7f4e96550640 10 monclient(hunting): handle_auth_done global_id 825357 payload 274 2024-09-16T04:01:12.306 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.292+0000 7f4e96550640 1 -- 172.21.15.42:0/239647835 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4e900a1330 msgr2=0x7f4e900a1710 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:01:12.306 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.292+0000 7f4e9554e640 1 -- 172.21.15.42:0/239647835 >> v2:172.21.15.154:3300/0 conn(0x7f4e9006dac0 msgr2=0x7f4e900a1c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:01:12.306 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.292+0000 7f4e96550640 1 --2- 172.21.15.42:0/239647835 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4e900a1330 0x7f4e900a1710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:12.306 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.292+0000 7f4e9554e640 1 -- 172.21.15.42:0/239647835 >> v2:172.21.15.154:3300/0 conn(0x7f4e9006dac0 msgr2=0x7f4e900a1c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:01:12.306 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.292+0000 7f4e9554e640 1 --2- 172.21.15.42:0/239647835 >> v2:172.21.15.154:3300/0 conn(0x7f4e9006dac0 0x7f4e900a1c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:01:12.306 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.292+0000 7f4e96550640 1 -- 172.21.15.42:0/239647835 >> v2:172.21.15.154:3300/0 conn(0x7f4e9006dac0 msgr2=0x7f4e900a1c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:01:12.306 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.292+0000 7f4e9554e640 1 --2- 172.21.15.42:0/239647835 >> v2:172.21.15.154:3300/0 conn(0x7f4e9006dac0 0x7f4e900a1c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:01:12.306 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.292+0000 7f4e96550640 1 --2- 172.21.15.42:0/239647835 >> v2:172.21.15.154:3300/0 conn(0x7f4e9006dac0 0x7f4e900a1c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:12.306 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.292+0000 7f4e96550640 10 monclient: _finish_hunting 0 2024-09-16T04:01:12.306 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.292+0000 7f4e96550640 1 monclient: found mon.noname-a 2024-09-16T04:01:12.307 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.292+0000 7f4e96550640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:01:12.307 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.292+0000 7f4e96550640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:01:12.307 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.292+0000 7f4e96550640 1 -- 172.21.15.42:0/239647835 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e9009f8f0 con 0x7f4e90135100 2024-09-16T04:01:12.307 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.296+0000 7f4e96550640 1 --2- 172.21.15.42:0/239647835 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4e90135100 0x7f4e90135550 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f4e9006c630 tx=0x7f4e8801c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12667b2e42a8bb78 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:01:12.307 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.296+0000 7f4e94d4d640 1 -- 172.21.15.42:0/239647835 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e880a5720 con 0x7f4e90135100 2024-09-16T04:01:12.307 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.296+0000 7f4e94d4d640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:01:12.307 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.296+0000 7f4e94d4d640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:01:12.307 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.296+0000 7f4e94d4d640 10 monclient: dump: 2024-09-16T04:01:12.307 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:01:12.307 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:01:12.307 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:12.307 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:12.308 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:01:12.308 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:01:12.308 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:01:12.308 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:01:12.308 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:01:12.308 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:01:12.308 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.296+0000 7f4e94d4d640 10 monclient: _finish_auth 0 2024-09-16T04:01:12.308 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.296+0000 7f4e94d4d640 10 monclient: _check_auth_tickets 2024-09-16T04:01:12.308 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.296+0000 7f4e94d4d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:01:12.308 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.296+0000 7f4e94d4d640 1 -- 172.21.15.42:0/239647835 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e880a58c0 con 0x7f4e90135100 2024-09-16T04:01:12.308 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.296+0000 7f4e94d4d640 10 monclient: handle_config config(0 keys) 2024-09-16T04:01:12.308 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.296+0000 7f4e94d4d640 1 -- 172.21.15.42:0/239647835 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e880a5c90 con 0x7f4e90135100 2024-09-16T04:01:12.308 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.296+0000 7f4e94d4d640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:01:12.309 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.296+0000 7f4e94d4d640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:01:12.309 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.296+0000 7f4e94d4d640 10 monclient: dump: 2024-09-16T04:01:12.309 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:01:12.309 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:01:12.309 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:12.309 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:12.309 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:01:12.309 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:01:12.309 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:01:12.309 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:01:12.309 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:01:12.309 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:01:12.309 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.296+0000 7f4e97fb3640 5 monclient: authenticate success, global_id 825357 2024-09-16T04:01:12.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.296+0000 7f4e97fb3640 10 monclient: get_monmap_and_config success 2024-09-16T04:01:12.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.296+0000 7f4e97fb3640 4 set_mon_vals no callback set 2024-09-16T04:01:12.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.296+0000 7f4e97fb3640 10 monclient: shutdown 2024-09-16T04:01:12.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.296+0000 7f4e97fb3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:01:12.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.296+0000 7f4e97fb3640 1 -- 172.21.15.42:0/239647835 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4e90135100 msgr2=0x7f4e90135550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:01:12.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.296+0000 7f4e97fb3640 1 --2- 172.21.15.42:0/239647835 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4e90135100 0x7f4e90135550 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f4e9006c630 tx=0x7f4e8801c9f0 comp rx=0 tx=0).stop 2024-09-16T04:01:12.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.296+0000 7f4e97fb3640 1 -- 172.21.15.42:0/239647835 shutdown_connections 2024-09-16T04:01:12.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.296+0000 7f4e97fb3640 1 --2- 172.21.15.42:0/239647835 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4e90135100 0x7f4e90135550 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:12.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.296+0000 7f4e97fb3640 1 --2- 172.21.15.42:0/239647835 >> v2:172.21.15.154:3300/0 conn(0x7f4e9006dac0 0x7f4e900a1c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:12.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.296+0000 7f4e97fb3640 1 --2- 172.21.15.42:0/239647835 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4e900a1330 0x7f4e900a1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:12.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.296+0000 7f4e97fb3640 1 -- 172.21.15.42:0/239647835 >> 172.21.15.42:0/239647835 conn(0x7f4e900a2420 msgr2=0x7f4e900a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:01:12.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.296+0000 7f4e97fb3640 1 -- 172.21.15.42:0/239647835 shutdown_connections 2024-09-16T04:01:12.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.296+0000 7f4e97fb3640 1 -- 172.21.15.42:0/239647835 wait complete. 2024-09-16T04:01:12.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.296+0000 7f4e97fb3640 10 monclient: build_initial_monmap 2024-09-16T04:01:12.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.296+0000 7f4e97fb3640 10 monclient: monmap: 2024-09-16T04:01:12.321 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:01:12.321 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:01:12.321 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:01:12.321 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:01:12.321 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:01:12.321 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:01:12.322 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:01:12.322 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:01:12.322 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:01:12.323 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:01:12.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.296+0000 7f4e97fb3640 1 librados: starting msgr at 2024-09-16T04:01:12.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.296+0000 7f4e97fb3640 1 librados: starting objecter 2024-09-16T04:01:12.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.296+0000 7f4e97fb3640 1 Processor -- start 2024-09-16T04:01:12.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.296+0000 7f4e97fb3640 1 -- start start 2024-09-16T04:01:12.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.296+0000 7f4e97fb3640 1 librados: setting wanted keys 2024-09-16T04:01:12.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.296+0000 7f4e97fb3640 1 librados: calling monclient init 2024-09-16T04:01:12.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.296+0000 7f4e97fb3640 10 monclient: init 2024-09-16T04:01:12.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.296+0000 7f4e97fb3640 10 monclient: _reopen_session rank -1 2024-09-16T04:01:12.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.296+0000 7f4e97fb3640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:01:12.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.296+0000 7f4e97fb3640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4e900a1330 0x7f4e901cf110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:01:12.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.296+0000 7f4e97fb3640 10 monclient(hunting): picked mon.noname-a con 0x7f4e900a1330 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:01:12.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.296+0000 7f4e97fb3640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4e90135100 0x7f4e90130640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:01:12.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.296+0000 7f4e97fb3640 10 monclient(hunting): picked mon.noname-c con 0x7f4e90135100 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:01:12.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.296+0000 7f4e97fb3640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4e90130b80 0x7f4e901cbc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:01:12.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.296+0000 7f4e97fb3640 10 monclient(hunting): picked mon.noname-b con 0x7f4e90130b80 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:01:12.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.296+0000 7f4e97fb3640 10 monclient(hunting): start opening mon connection 2024-09-16T04:01:12.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.296+0000 7f4e97fb3640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f4e9009eef0 con 0x7f4e900a1330 2024-09-16T04:01:12.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.296+0000 7f4e97fb3640 10 monclient(hunting): start opening mon connection 2024-09-16T04:01:12.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.296+0000 7f4e97fb3640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f4e9009ed70 con 0x7f4e90130b80 2024-09-16T04:01:12.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.296+0000 7f4e97fb3640 10 monclient(hunting): start opening mon connection 2024-09-16T04:01:12.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.296+0000 7f4e97fb3640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f4e9013eb70 con 0x7f4e90135100 2024-09-16T04:01:12.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.296+0000 7f4e97fb3640 10 monclient(hunting): _renew_subs 2024-09-16T04:01:12.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.296+0000 7f4e97fb3640 10 monclient(hunting): authenticate will time out at 2119.271192s 2024-09-16T04:01:12.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.296+0000 7f4e9554e640 1 -- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4e90135100 msgr2=0x7f4e90130640 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T04:01:12.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.296+0000 7f4e9554e640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4e90135100 0x7f4e90130640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:01:12.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.296+0000 7f4e95d4f640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4e900a1330 0x7f4e901cf110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:01:12.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.296+0000 7f4e95d4f640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4e900a1330 0x7f4e901cf110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:40886/0 (socket says 172.21.15.42:40886) 2024-09-16T04:01:12.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.296+0000 7f4e95d4f640 1 -- 172.21.15.42:0/2243873954 learned_addr learned my addr 172.21.15.42:0/2243873954 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:01:12.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.296+0000 7f4e95d4f640 10 monclient(hunting): get_auth_request con 0x7f4e900a1330 auth_method 0 2024-09-16T04:01:12.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.296+0000 7f4e95d4f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:01:12.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.296+0000 7f4e95d4f640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:01:12.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.296+0000 7f4e95d4f640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:01:12.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.296+0000 7f4e95d4f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:01:12.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.296+0000 7f4e95d4f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:01:12.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.296+0000 7f4e95d4f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:01:12.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.296+0000 7f4e96550640 1 --2- 172.21.15.42:0/2243873954 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4e90130b80 0x7f4e901cbc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:01:12.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.296+0000 7f4e96550640 10 monclient(hunting): get_auth_request con 0x7f4e90130b80 auth_method 0 2024-09-16T04:01:12.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.296+0000 7f4e96550640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:01:12.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.296+0000 7f4e96550640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:01:12.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.296+0000 7f4e96550640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:01:12.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.296+0000 7f4e95d4f640 10 monclient(hunting): handle_auth_done global_id 825360 payload 824 2024-09-16T04:01:12.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.296+0000 7f4e95d4f640 1 -- 172.21.15.42:0/2243873954 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4e90135100 msgr2=0x7f4e90130640 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:01:12.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.296+0000 7f4e95d4f640 1 --2- 172.21.15.42:0/2243873954 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4e90135100 0x7f4e90130640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:12.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.296+0000 7f4e95d4f640 1 -- 172.21.15.42:0/2243873954 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4e90130b80 msgr2=0x7f4e901cbc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:01:12.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.296+0000 7f4e95d4f640 1 --2- 172.21.15.42:0/2243873954 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4e90130b80 0x7f4e901cbc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:12.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.296+0000 7f4e95d4f640 10 monclient: _finish_hunting 0 2024-09-16T04:01:12.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.296+0000 7f4e95d4f640 1 monclient: found mon.noname-a 2024-09-16T04:01:12.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.296+0000 7f4e95d4f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:01:12.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.296+0000 7f4e95d4f640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:01:12.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.296+0000 7f4e95d4f640 1 -- 172.21.15.42:0/2243873954 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e9013b960 con 0x7f4e900a1330 2024-09-16T04:01:12.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.300+0000 7f4e95d4f640 1 --2- 172.21.15.42:0/2243873954 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4e900a1330 0x7f4e901cf110 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f4e8c00b550 tx=0x7f4e8c00ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:01:12.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.300+0000 7f4e867fc640 10 client.?.objecter ms_handle_connect 0x7f4e900a1330 2024-09-16T04:01:12.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.300+0000 7f4e867fc640 10 client.?.objecter resend_mon_ops 2024-09-16T04:01:12.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.300+0000 7f4e867fc640 1 -- 172.21.15.42:0/2243873954 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e8c003dd0 con 0x7f4e900a1330 2024-09-16T04:01:12.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.300+0000 7f4e867fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:01:12.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.300+0000 7f4e867fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:01:12.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.300+0000 7f4e867fc640 10 monclient: dump: 2024-09-16T04:01:12.328 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:01:12.329 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:01:12.329 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:12.329 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:12.329 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:01:12.329 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:01:12.329 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:01:12.329 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:01:12.329 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:01:12.329 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:01:12.329 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.300+0000 7f4e867fc640 10 monclient: _finish_auth 0 2024-09-16T04:01:12.329 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.300+0000 7f4e867fc640 10 monclient: _check_auth_tickets 2024-09-16T04:01:12.329 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.300+0000 7f4e867fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:01:12.329 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.300+0000 7f4e97fb3640 5 monclient: authenticate success, global_id 825360 2024-09-16T04:01:12.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.300+0000 7f4e867fc640 1 -- 172.21.15.42:0/2243873954 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e8c015070 con 0x7f4e900a1330 2024-09-16T04:01:12.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.300+0000 7f4e97fb3640 10 monclient: _renew_subs 2024-09-16T04:01:12.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.300+0000 7f4e97fb3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:01:12.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.300+0000 7f4e97fb3640 1 -- 172.21.15.42:0/2243873954 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e90073a30 con 0x7f4e900a1330 2024-09-16T04:01:12.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.300+0000 7f4e867fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:01:12.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.300+0000 7f4e867fc640 1 -- 172.21.15.42:0/2243873954 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e8c005330 con 0x7f4e900a1330 2024-09-16T04:01:12.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.300+0000 7f4e867fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:01:12.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.300+0000 7f4e867fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:01:12.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.300+0000 7f4e867fc640 10 monclient: dump: 2024-09-16T04:01:12.330 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:01:12.330 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:01:12.330 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:12.330 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:12.330 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:01:12.331 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:01:12.331 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:01:12.331 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:01:12.331 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:01:12.331 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:01:12.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.300+0000 7f4e97fb3640 10 client.825360.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:01:12.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.300+0000 7f4e877fe640 4 set_mon_vals no callback set 2024-09-16T04:01:12.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.300+0000 7f4e97fb3640 10 monclient: _renew_subs 2024-09-16T04:01:12.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.300+0000 7f4e97fb3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:01:12.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.300+0000 7f4e97fb3640 1 -- 172.21.15.42:0/2243873954 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e9011e000 con 0x7f4e900a1330 2024-09-16T04:01:12.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.300+0000 7f4e97fb3640 1 librados: init done 2024-09-16T04:01:12.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.300+0000 7f4e97fb3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:01:12.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.300+0000 7f4e97fb3640 1 --2- 172.21.15.42:0/2243873954 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4e9006dac0 0x7f4e901cc150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:01:12.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.300+0000 7f4e97fb3640 10 monclient(hunting): start opening mon connection 2024-09-16T04:01:12.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.300+0000 7f4e97fb3640 1 -- 172.21.15.42:0/2243873954 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f4e90135550 con 0x7f4e9006dac0 2024-09-16T04:01:12.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.300+0000 7f4e867fc640 1 -- 172.21.15.42:0/2243873954 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f4e8c0054d0 con 0x7f4e900a1330 2024-09-16T04:01:12.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.300+0000 7f4e9554e640 1 --2- 172.21.15.42:0/2243873954 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4e9006dac0 0x7f4e901cc150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:01:12.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.300+0000 7f4e867fc640 1 --2- 172.21.15.42:0/2243873954 >> v2:172.21.15.154:6812/254845342 conn(0x7f4e740377c0 0x7f4e74039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:01:12.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.300+0000 7f4e9554e640 10 monclient: get_auth_request con 0x7f4e9006dac0 auth_method 0 2024-09-16T04:01:12.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.300+0000 7f4e9554e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:01:12.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.300+0000 7f4e9554e640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:01:12.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.300+0000 7f4e9554e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:01:12.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.300+0000 7f4e867fc640 1 -- 172.21.15.42:0/2243873954 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(570..570 src has 451..570) ==== 5843+0+0 (secure 0 0 0) 0x7f4e8c048dd0 con 0x7f4e900a1330 2024-09-16T04:01:12.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.300+0000 7f4e867fc640 10 client.825360.objecter ms_dispatch 0x7f4e90000ba0 osd_map(570..570 src has 451..570) 2024-09-16T04:01:12.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.300+0000 7f4e867fc640 3 client.825360.objecter handle_osd_map got epochs [570,570] > 0 2024-09-16T04:01:12.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.300+0000 7f4e867fc640 3 client.825360.objecter handle_osd_map decoding full epoch 570 2024-09-16T04:01:12.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.300+0000 7f4e867fc640 20 client.825360.objecter dump_active .. 0 homeless 2024-09-16T04:01:12.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.300+0000 7f4e9554e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:01:12.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.300+0000 7f4e9554e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:01:12.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.300+0000 7f4e9554e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:01:12.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.300+0000 7f4e96550640 1 --2- 172.21.15.42:0/2243873954 >> v2:172.21.15.154:6812/254845342 conn(0x7f4e740377c0 0x7f4e74039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:01:12.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.300+0000 7f4e96550640 10 monclient: get_auth_request con 0x7f4e740377c0 auth_method 0 2024-09-16T04:01:12.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.300+0000 7f4e9554e640 10 monclient(hunting): handle_auth_done global_id 904480 payload 824 2024-09-16T04:01:12.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.300+0000 7f4e9554e640 1 -- 172.21.15.42:0/2243873954 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4e90071e70 con 0x7f4e9006dac0 2024-09-16T04:01:12.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.300+0000 7f4e867fc640 10 client.825360.objecter ms_handle_connect 0x7f4e9006dac0 2024-09-16T04:01:12.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.300+0000 7f4e867fc640 10 client.825360.objecter resend_mon_ops 2024-09-16T04:01:12.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.300+0000 7f4e9554e640 1 --2- 172.21.15.42:0/2243873954 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4e9006dac0 0x7f4e901cc150 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f4e80000ee0 tx=0x7f4e8000eba0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:01:12.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.300+0000 7f4e867fc640 10 client.825360.objecter ms_handle_connect 0x7f4e740377c0 2024-09-16T04:01:12.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.300+0000 7f4e96550640 1 --2- 172.21.15.42:0/2243873954 >> v2:172.21.15.154:6812/254845342 conn(0x7f4e740377c0 0x7f4e74039c80 secure :-1 s=READY pgs=3759 cs=0 l=1 rev1=1 crypto rx=0x7f4e880ae000 tx=0x7f4e88004ac0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:01:12.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.300+0000 7f4e867fc640 1 -- 172.21.15.42:0/2243873954 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e80017030 con 0x7f4e9006dac0 2024-09-16T04:01:12.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.300+0000 7f4e867fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:01:12.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.300+0000 7f4e867fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:01:12.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.300+0000 7f4e867fc640 10 monclient: dump: 2024-09-16T04:01:12.334 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:01:12.334 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:01:12.334 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:12.334 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:12.334 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:01:12.334 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:01:12.334 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:01:12.334 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:01:12.334 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:01:12.334 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:01:12.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.300+0000 7f4e867fc640 1 -- 172.21.15.42:0/2243873954 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f4e8000ecd0 con 0x7f4e9006dac0 2024-09-16T04:01:12.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.300+0000 7f4e867fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:01:12.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.300+0000 7f4e867fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:01:12.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.300+0000 7f4e867fc640 1 -- 172.21.15.42:0/2243873954 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4e9006dac0 msgr2=0x7f4e901cc150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:01:12.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.300+0000 7f4e867fc640 1 --2- 172.21.15.42:0/2243873954 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4e9006dac0 0x7f4e901cc150 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f4e80000ee0 tx=0x7f4e8000eba0 comp rx=0 tx=0).stop 2024-09-16T04:01:12.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.300+0000 7f4e867fc640 1 -- 172.21.15.42:0/2243873954 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4e9006dac0 msgr2=0x7f4e901cc150 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:01:12.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.300+0000 7f4e867fc640 1 --2- 172.21.15.42:0/2243873954 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4e9006dac0 0x7f4e901cc150 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:12.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.308+0000 7f4e97fb3640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:01:12.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.308+0000 7f4e97fb3640 1 --2- 172.21.15.42:0/2243873954 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4e901cc690 0x7f4e9009cb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:01:12.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.308+0000 7f4e97fb3640 10 monclient(hunting): start opening mon connection 2024-09-16T04:01:12.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.308+0000 7f4e97fb3640 1 -- 172.21.15.42:0/2243873954 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f4e901ccaa0 con 0x7f4e901cc690 2024-09-16T04:01:12.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.308+0000 7f4e9554e640 1 --2- 172.21.15.42:0/2243873954 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4e901cc690 0x7f4e9009cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:01:12.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.308+0000 7f4e9554e640 10 monclient: get_auth_request con 0x7f4e901cc690 auth_method 0 2024-09-16T04:01:12.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.308+0000 7f4e9554e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:01:12.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.308+0000 7f4e9554e640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:01:12.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.308+0000 7f4e9554e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:01:12.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.308+0000 7f4e9554e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:01:12.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.308+0000 7f4e9554e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:01:12.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.308+0000 7f4e9554e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:01:12.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.308+0000 7f4e9554e640 10 monclient(hunting): handle_auth_done global_id 904483 payload 824 2024-09-16T04:01:12.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.308+0000 7f4e9554e640 1 -- 172.21.15.42:0/2243873954 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f4e90069040 con 0x7f4e901cc690 2024-09-16T04:01:12.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.308+0000 7f4e9554e640 1 --2- 172.21.15.42:0/2243873954 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4e901cc690 0x7f4e9009cb60 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f4e80000ee0 tx=0x7f4e80004740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:01:12.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.308+0000 7f4e867fc640 10 client.825360.objecter ms_handle_connect 0x7f4e901cc690 2024-09-16T04:01:12.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.308+0000 7f4e867fc640 10 client.825360.objecter resend_mon_ops 2024-09-16T04:01:12.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.308+0000 7f4e867fc640 1 -- 172.21.15.42:0/2243873954 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e8000f040 con 0x7f4e901cc690 2024-09-16T04:01:12.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.308+0000 7f4e867fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:01:12.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.308+0000 7f4e867fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:01:12.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.308+0000 7f4e867fc640 10 monclient: dump: 2024-09-16T04:01:12.337 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:01:12.337 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:01:12.337 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:12.337 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:12.337 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:01:12.337 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:01:12.337 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:01:12.337 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:01:12.337 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:01:12.337 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:01:12.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.308+0000 7f4e867fc640 1 -- 172.21.15.42:0/2243873954 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4869 (secure 0 0 0) 0x7f4e80013430 con 0x7f4e901cc690 2024-09-16T04:01:12.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.308+0000 7f4e867fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:01:12.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.308+0000 7f4e867fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:01:12.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.308+0000 7f4e867fc640 1 -- 172.21.15.42:0/2243873954 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4e901cc690 msgr2=0x7f4e9009cb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:01:12.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.308+0000 7f4e867fc640 1 --2- 172.21.15.42:0/2243873954 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4e901cc690 0x7f4e9009cb60 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f4e80000ee0 tx=0x7f4e80004740 comp rx=0 tx=0).stop 2024-09-16T04:01:12.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.308+0000 7f4e867fc640 1 -- 172.21.15.42:0/2243873954 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4e901cc690 msgr2=0x7f4e9009cb60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:01:12.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.308+0000 7f4e867fc640 1 --2- 172.21.15.42:0/2243873954 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4e901cc690 0x7f4e9009cb60 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:12.338 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T04:01:12.338 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:01:12.338 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:01:12.338 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T04:01:12.339 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 172, 2024-09-16T04:01:12.339 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 200490, 2024-09-16T04:01:12.339 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T04:01:12.339 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T04:01:12.339 INFO:teuthology.orchestra.run.smithi042.stdout: 1 2024-09-16T04:01:12.339 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:01:12.339 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 0, 2024-09-16T04:01:12.339 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:01:12.339 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T04:01:12.339 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T04:01:12.339 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:01:12.339 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:01:12.339 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:01:12.340 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:01:12.340 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:01:12.340 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:01:12.340 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:01:12.340 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:01:12.340 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:01:12.340 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:01:12.340 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:01:12.340 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:01:12.340 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:01:12.340 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T04:01:12.340 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:01:12.340 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:01:12.340 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:01:12.341 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:01:12.341 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:01:12.341 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:01:12.341 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:01:12.341 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:01:12.341 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:01:12.341 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:01:12.342 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:01:12.342 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:01:12.342 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:01:12.342 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T04:01:12.342 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T04:01:12.342 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T04:01:12.342 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T04:01:12.342 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T04:01:12.342 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T04:01:12.342 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:01:12.343 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:01:12.343 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T04:01:12.343 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T04:01:12.343 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T04:01:12.343 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T04:01:12.343 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T04:01:12.343 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T04:01:12.343 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T04:01:12.343 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:01:12.343 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T04:01:12.344 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:01:12.344 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:01:12.344 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:01:12.344 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:01:12.344 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:01:12.344 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:01:12.344 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:01:12.344 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:01:12.344 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:01:12.344 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:01:12.344 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:01:12.344 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:01:12.344 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T04:01:12.345 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:01:12.345 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T04:01:12.345 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:01:12.345 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:01:12.345 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:01:12.345 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:01:12.345 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:01:12.345 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:01:12.345 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:01:12.346 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T04:01:12.346 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:01:12.346 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:01:12.346 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:01:12.346 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:01:12.346 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T04:01:12.346 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:01:12.346 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:01:12.346 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:01:12.346 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:01:12.346 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T04:01:12.346 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T04:01:12.346 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:01:12.347 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:01:12.347 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:01:12.347 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:01:12.347 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:01:12.347 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:01:12.347 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:01:12.347 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:01:12.347 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:01:12.347 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:01:12.347 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:01:12.347 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T04:01:12.347 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:01:12.348 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:01:12.348 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:01:12.348 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:01:12.348 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T04:01:12.348 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:01:12.348 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:01:12.348 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:01:12.348 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:01:12.348 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T04:01:12.348 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T04:01:12.348 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:01:12.348 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:01:12.348 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:01:12.349 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:01:12.349 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:01:12.349 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:01:12.349 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:01:12.349 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:01:12.349 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:01:12.349 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:01:12.349 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:01:12.349 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T04:01:12.349 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:01:12.349 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:01:12.349 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:01:12.350 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:01:12.350 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T04:01:12.350 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:01:12.350 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:01:12.350 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:01:12.350 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:01:12.350 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T04:01:12.350 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T04:01:12.350 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:01:12.350 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:01:12.350 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:01:12.350 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:01:12.351 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:01:12.351 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:01:12.351 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T04:01:12.351 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T04:01:12.351 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:01:12.351 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:01:12.351 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:01:12.351 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:01:12.351 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:01:12.351 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:01:12.351 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T04:01:12.351 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:01:12.352 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:01:12.352 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:01:12.352 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:01:12.352 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:01:12.352 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:01:12.352 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T04:01:12.352 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:01:12.352 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:01:12.352 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:01:12.352 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 2 2024-09-16T04:01:12.352 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:01:12.352 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:01:12.353 INFO:teuthology.orchestra.run.smithi042.stdout: "mgr": [ 2024-09-16T04:01:12.353 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:01:12.353 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:01:12.353 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:01:12.353 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:01:12.353 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:01:12.353 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:01:12.353 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:01:12.353 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T04:01:12.354 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T04:01:12.355 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.312+0000 7f4e97fb3640 10 librados: watch_flush enter 2024-09-16T04:01:12.355 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.312+0000 7f4e97fb3640 10 librados: watch_flush exit 2024-09-16T04:01:12.355 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.312+0000 7f4e97fb3640 20 client.825360.objecter shutdown clearing up homeless session... 2024-09-16T04:01:12.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.312+0000 7f4e97fb3640 10 client.825360.objecter successfully canceled tick 2024-09-16T04:01:12.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.312+0000 7f4e97fb3640 1 -- 172.21.15.42:0/2243873954 >> v2:172.21.15.154:6812/254845342 conn(0x7f4e740377c0 msgr2=0x7f4e74039c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:01:12.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.312+0000 7f4e97fb3640 1 --2- 172.21.15.42:0/2243873954 >> v2:172.21.15.154:6812/254845342 conn(0x7f4e740377c0 0x7f4e74039c80 secure :-1 s=READY pgs=3759 cs=0 l=1 rev1=1 crypto rx=0x7f4e880ae000 tx=0x7f4e88004ac0 comp rx=0 tx=0).stop 2024-09-16T04:01:12.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.312+0000 7f4e97fb3640 10 monclient: shutdown 2024-09-16T04:01:12.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.312+0000 7f4e97fb3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:01:12.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.312+0000 7f4e97fb3640 1 -- 172.21.15.42:0/2243873954 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4e900a1330 msgr2=0x7f4e901cf110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:01:12.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.312+0000 7f4e97fb3640 1 --2- 172.21.15.42:0/2243873954 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4e900a1330 0x7f4e901cf110 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f4e8c00b550 tx=0x7f4e8c00ba10 comp rx=0 tx=0).stop 2024-09-16T04:01:12.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.312+0000 7f4e96550640 1 -- 172.21.15.42:0/2243873954 reap_dead start 2024-09-16T04:01:12.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.312+0000 7f4e96550640 1 -- 172.21.15.42:0/2243873954 reap_dead start 2024-09-16T04:01:12.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.312+0000 7f4e97fb3640 1 -- 172.21.15.42:0/2243873954 shutdown_connections 2024-09-16T04:01:12.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.312+0000 7f4e97fb3640 1 -- 172.21.15.42:0/2243873954 >> 172.21.15.42:0/2243873954 conn(0x7f4e900a2420 msgr2=0x7f4e901345d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:01:12.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.312+0000 7f4e97fb3640 1 -- 172.21.15.42:0/2243873954 shutdown_connections 2024-09-16T04:01:12.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.312+0000 7f4e97fb3640 1 -- 172.21.15.42:0/2243873954 wait complete. 2024-09-16T04:01:12.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:12.312+0000 7f4e97fb3640 1 librados: shutdown 2024-09-16T04:01:12.357 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T04:01:12.357 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-16T04:01:29.592 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: require_osd_release = squid 2024-09-16T04:01:29.593 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TierFlushDuringFlush (24137 ms) 2024-09-16T04:01:29.593 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapHasChunk 2024-09-16T04:01:29.593 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: require_osd_release = squid 2024-09-16T04:01:29.593 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapHasChunk (5875 ms) 2024-09-16T04:01:29.593 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRollback 2024-09-16T04:01:29.593 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: require_osd_release = squid 2024-09-16T04:01:29.593 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRollback (5038 ms) 2024-09-16T04:01:29.593 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRollbackRefcount 2024-09-16T04:01:29.593 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: require_osd_release = squid 2024-09-16T04:01:29.593 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRollbackRefcount (34019 ms) 2024-09-16T04:01:29.593 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvictRollback 2024-09-16T04:01:29.594 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: require_osd_release = squid 2024-09-16T04:01:29.594 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvictRollback (13942 ms) 2024-09-16T04:01:29.594 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PropagateBaseTierError 2024-09-16T04:01:29.594 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PropagateBaseTierError (17249 ms) 2024-09-16T04:01:29.594 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HelloWriteReturn 2024-09-16T04:01:29.594 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: 00000000 79 6f 75 20 6d 69 67 68 74 20 73 65 65 20 74 68 |you might see th| 2024-09-16T04:01:29.594 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: 00000010 69 73 |is| 2024-09-16T04:01:29.594 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: 00000012 2024-09-16T04:01:29.594 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HelloWriteReturn (31807 ms) 2024-09-16T04:01:29.594 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TierFlushDuringUnsetDedupTier 2024-09-16T04:01:29.594 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: require_osd_release = squid 2024-09-16T04:01:29.594 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TierFlushDuringUnsetDedupTier (5960 ms) 2024-09-16T04:01:29.595 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [----------] 48 tests from LibRadosTwoPoolsPP (818227 ms total) 2024-09-16T04:01:29.595 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: 2024-09-16T04:01:29.595 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [----------] 4 tests from LibRadosTierECPP 2024-09-16T04:01:29.595 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.Dirty 2024-09-16T04:01:29.595 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.Dirty (1163 ms) 2024-09-16T04:01:29.595 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.FlushWriteRaces 2024-09-16T04:01:29.595 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.FlushWriteRaces (45826 ms) 2024-09-16T04:01:29.595 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.CallForcesPromote 2024-09-16T04:01:29.595 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.CallForcesPromote (17995 ms) 2024-09-16T04:01:29.595 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.HitSetNone 2024-09-16T04:01:29.596 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.HitSetNone (6 ms) 2024-09-16T04:01:29.596 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [----------] 4 tests from LibRadosTierECPP (64990 ms total) 2024-09-16T04:01:29.596 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: 2024-09-16T04:01:29.596 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [----------] 22 tests from LibRadosTwoPoolsECPP 2024-09-16T04:01:29.596 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Overlay 2024-09-16T04:01:29.596 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Overlay (6894 ms) 2024-09-16T04:01:29.596 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Promote 2024-09-16T04:01:29.596 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Promote (7993 ms) 2024-09-16T04:01:29.596 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteSnap 2024-09-16T04:01:29.596 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: waiting for scrub... 2024-09-16T04:01:29.597 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: done waiting 2024-09-16T04:01:29.597 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteSnap (41014 ms) 2024-09-16T04:01:29.597 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteSnapTrimRace 2024-09-16T04:01:29.597 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteSnapTrimRace (10038 ms) 2024-09-16T04:01:29.597 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Whiteout 2024-09-16T04:01:29.597 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Whiteout (7953 ms) 2024-09-16T04:01:29.597 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Evict 2024-09-16T04:01:29.597 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Evict (41186 ms) 2024-09-16T04:01:29.597 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.EvictSnap 2024-09-16T04:01:29.597 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.EvictSnap (9851 ms) 2024-09-16T04:01:29.597 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TryFlush 2024-09-16T04:01:32.358 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-16T04:01:32.358 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-09-16T04:01:32.359 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-16T04:01:32.359 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-16T04:01:32.362 INFO:tasks.ceph.mon.c:Started 2024-09-16T04:01:32.362 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T04:01:32.362 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:01:32.501 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.492+0000 7f87d8c7c640 10 monclient: get_monmap_and_config 2024-09-16T04:01:32.501 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.492+0000 7f87d8c7c640 10 monclient: build_initial_monmap 2024-09-16T04:01:32.501 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.492+0000 7f87d8c7c640 10 monclient: monmap: 2024-09-16T04:01:32.501 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:01:32.501 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:01:32.501 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:01:32.499575+0000 2024-09-16T04:01:32.501 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:01:32.499575+0000 2024-09-16T04:01:32.501 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:01:32.501 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:01:32.502 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:01:32.502 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:01:32.502 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:01:32.502 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:01:32.502 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:01:32.502 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:01:32.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.496+0000 7f87d8c7c640 1 Processor -- start 2024-09-16T04:01:32.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.496+0000 7f87d8c7c640 1 -- start start 2024-09-16T04:01:32.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.496+0000 7f87d8c7c640 10 monclient: init 2024-09-16T04:01:32.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.496+0000 7f87d8c7c640 10 monclient: _reopen_session rank -1 2024-09-16T04:01:32.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.496+0000 7f87d8c7c640 10 monclient: _add_conns ranks=[0,1,3,4,2] 2024-09-16T04:01:32.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.496+0000 7f87d8c7c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f87d40a28a0 addr v1:172.21.15.42:6789/0 2024-09-16T04:01:32.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.496+0000 7f87d8c7c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f87d406dc20 addr v1:172.21.15.154:6789/0 2024-09-16T04:01:32.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.496+0000 7f87d8c7c640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f87d40a3390 0x7f87d413fce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:01:32.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.496+0000 7f87d8c7c640 10 monclient(hunting): picked mon.noname-a con 0x7f87d40a3390 addr v2:172.21.15.42:3300/0 2024-09-16T04:01:32.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.496+0000 7f87d8c7c640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f87d40726b0 con 0x7f87d40a28a0 2024-09-16T04:01:32.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.496+0000 7f87d8c7c640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f87d40a04e0 con 0x7f87d406dc20 2024-09-16T04:01:32.505 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.496+0000 7f87d8c7c640 10 monclient(hunting): start opening mon connection 2024-09-16T04:01:32.505 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.496+0000 7f87d8c7c640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f87d40a1790 con 0x7f87d40a3390 2024-09-16T04:01:32.505 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.496+0000 7f87d8c7c640 10 monclient(hunting): _renew_subs 2024-09-16T04:01:32.505 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.496+0000 7f87d8c7c640 10 monclient(hunting): authenticate will time out at 2139.468978s 2024-09-16T04:01:32.505 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.496+0000 7f87d2d9d640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f87d40a3390 0x7f87d413fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:01:32.505 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.496+0000 7f87d259c640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f87d40a28a0 0x7f87d40a2c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:46806/0 (socket says 172.21.15.42:46806) 2024-09-16T04:01:32.505 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.496+0000 7f87d259c640 1 -- 172.21.15.42:0/2200368979 learned_addr learned my addr 172.21.15.42:0/2200368979 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T04:01:32.505 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.496+0000 7f87d2d9d640 10 monclient(hunting): get_auth_request con 0x7f87d40a3390 auth_method 0 2024-09-16T04:01:32.505 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.496+0000 7f87d2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:01:32.505 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.496+0000 7f87d2d9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:01:32.505 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.496+0000 7f87d2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:01:32.505 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.496+0000 7f87d2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:01:32.505 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.496+0000 7f87d2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:01:32.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.496+0000 7f87d2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:01:32.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.496+0000 7f87d2d9d640 10 monclient(hunting): handle_auth_done global_id 825363 payload 274 2024-09-16T04:01:32.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.496+0000 7f87d2d9d640 1 -- 172.21.15.42:0/2200368979 >> v1:172.21.15.154:6789/0 conn(0x7f87d406dc20 legacy=0x7f87d413c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:01:32.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.496+0000 7f87d2d9d640 1 -- 172.21.15.42:0/2200368979 >> v1:172.21.15.42:6789/0 conn(0x7f87d40a28a0 legacy=0x7f87d40a2c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:01:32.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.496+0000 7f87d2d9d640 10 monclient: _finish_hunting 0 2024-09-16T04:01:32.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.496+0000 7f87d2d9d640 1 monclient: found mon.noname-a 2024-09-16T04:01:32.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.496+0000 7f87d2d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:01:32.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.496+0000 7f87d2d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:01:32.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.496+0000 7f87d2d9d640 1 -- 172.21.15.42:0/2200368979 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87d40a09c0 con 0x7f87d40a3390 2024-09-16T04:01:32.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.496+0000 7f87d2d9d640 1 --2- 172.21.15.42:0/2200368979 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f87d40a3390 0x7f87d413fce0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f87cc0097c0 tx=0x7f87cc01c320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd146d849bf68d2e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:01:32.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.496+0000 7f87d159a640 1 -- 172.21.15.42:0/2200368979 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87cc0b0070 con 0x7f87d40a3390 2024-09-16T04:01:32.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.496+0000 7f87d159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:01:32.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.496+0000 7f87d159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:01:32.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.496+0000 7f87d159a640 10 monclient: dump: 2024-09-16T04:01:32.513 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:01:32.513 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:01:32.513 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:32.513 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:32.513 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:01:32.513 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:01:32.513 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:01:32.513 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:01:32.513 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:01:32.513 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:01:32.513 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.496+0000 7f87d159a640 10 monclient: _finish_auth 0 2024-09-16T04:01:32.513 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.496+0000 7f87d159a640 10 monclient: _check_auth_tickets 2024-09-16T04:01:32.513 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.496+0000 7f87d159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:01:32.514 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.496+0000 7f87d8c7c640 5 monclient: authenticate success, global_id 825363 2024-09-16T04:01:32.514 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.496+0000 7f87d159a640 1 -- 172.21.15.42:0/2200368979 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f87cc0ab600 con 0x7f87d40a3390 2024-09-16T04:01:32.514 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.496+0000 7f87d8c7c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:01:32.514 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.496+0000 7f87d159a640 10 monclient: handle_config config(0 keys) 2024-09-16T04:01:32.514 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.496+0000 7f87d8c7c640 10 monclient: get_monmap_and_config success 2024-09-16T04:01:32.514 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.496+0000 7f87d8c7c640 4 set_mon_vals no callback set 2024-09-16T04:01:32.514 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.496+0000 7f87d159a640 1 -- 172.21.15.42:0/2200368979 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87cc0ab9d0 con 0x7f87d40a3390 2024-09-16T04:01:32.514 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.496+0000 7f87d159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:01:32.514 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.496+0000 7f87d159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:01:32.514 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.496+0000 7f87d159a640 10 monclient: dump: 2024-09-16T04:01:32.514 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:01:32.514 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:01:32.514 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:32.515 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:32.515 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:01:32.515 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:01:32.515 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:01:32.515 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:01:32.515 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:01:32.515 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:01:32.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.500+0000 7f87d8c7c640 10 monclient: shutdown 2024-09-16T04:01:32.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.500+0000 7f87d8c7c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:01:32.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.500+0000 7f87d8c7c640 1 -- 172.21.15.42:0/2200368979 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f87d40a3390 msgr2=0x7f87d413fce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:01:32.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.500+0000 7f87d8c7c640 1 --2- 172.21.15.42:0/2200368979 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f87d40a3390 0x7f87d413fce0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f87cc0097c0 tx=0x7f87cc01c320 comp rx=0 tx=0).stop 2024-09-16T04:01:32.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.500+0000 7f87d8c7c640 1 -- 172.21.15.42:0/2200368979 shutdown_connections 2024-09-16T04:01:32.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.500+0000 7f87d8c7c640 1 --2- 172.21.15.42:0/2200368979 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f87d40a3390 0x7f87d413fce0 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:32.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.500+0000 7f87d8c7c640 1 -- 172.21.15.42:0/2200368979 >> 172.21.15.42:0/2200368979 conn(0x7f87d40a7e40 msgr2=0x7f87d40a8250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:01:32.516 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.500+0000 7f87d8c7c640 1 -- 172.21.15.42:0/2200368979 shutdown_connections 2024-09-16T04:01:32.516 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.500+0000 7f87d8c7c640 1 -- 172.21.15.42:0/2200368979 wait complete. 2024-09-16T04:01:32.516 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.500+0000 7f87d8c7c640 10 monclient: build_initial_monmap 2024-09-16T04:01:32.516 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.500+0000 7f87d8c7c640 10 monclient: monmap: 2024-09-16T04:01:32.516 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:01:32.516 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:01:32.516 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:01:32.516 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:01:32.516 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:01:32.516 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:01:32.516 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:01:32.516 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:01:32.516 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:01:32.517 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:01:32.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.500+0000 7f87d8c7c640 1 librados: starting msgr at 2024-09-16T04:01:32.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.500+0000 7f87d8c7c640 1 librados: starting objecter 2024-09-16T04:01:32.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.500+0000 7f87d8c7c640 1 Processor -- start 2024-09-16T04:01:32.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.500+0000 7f87d8c7c640 1 -- start start 2024-09-16T04:01:32.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.500+0000 7f87d8c7c640 1 librados: setting wanted keys 2024-09-16T04:01:32.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.500+0000 7f87d8c7c640 1 librados: calling monclient init 2024-09-16T04:01:32.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.500+0000 7f87d8c7c640 10 monclient: init 2024-09-16T04:01:32.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.500+0000 7f87d8c7c640 10 monclient: _reopen_session rank -1 2024-09-16T04:01:32.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.500+0000 7f87d8c7c640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:01:32.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.500+0000 7f87d8c7c640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f87d406dc20 0x7f87d41cf0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:01:32.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.500+0000 7f87d8c7c640 10 monclient(hunting): picked mon.noname-c con 0x7f87d406dc20 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:01:32.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.500+0000 7f87d8c7c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f87d40a28a0 0x7f87d41cf8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:01:32.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.500+0000 7f87d8c7c640 10 monclient(hunting): picked mon.noname-b con 0x7f87d40a28a0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:01:32.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.500+0000 7f87d8c7c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f87d40a3390 0x7f87d413d310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:01:32.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.500+0000 7f87d8c7c640 10 monclient(hunting): picked mon.noname-a con 0x7f87d40a3390 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:01:32.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.500+0000 7f87d8c7c640 10 monclient(hunting): start opening mon connection 2024-09-16T04:01:32.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.500+0000 7f87d8c7c640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f87d4142dd0 con 0x7f87d40a3390 2024-09-16T04:01:32.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.500+0000 7f87d8c7c640 10 monclient(hunting): start opening mon connection 2024-09-16T04:01:32.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.500+0000 7f87d8c7c640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f87d4142c50 con 0x7f87d40a28a0 2024-09-16T04:01:32.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.500+0000 7f87d8c7c640 10 monclient(hunting): start opening mon connection 2024-09-16T04:01:32.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.500+0000 7f87d8c7c640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f87d4142f50 con 0x7f87d406dc20 2024-09-16T04:01:32.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.500+0000 7f87d8c7c640 10 monclient(hunting): _renew_subs 2024-09-16T04:01:32.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.500+0000 7f87d8c7c640 10 monclient(hunting): authenticate will time out at 2139.473738s 2024-09-16T04:01:32.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.500+0000 7f87d2d9d640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f87d40a3390 0x7f87d413d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:01:32.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.500+0000 7f87d259c640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f87d406dc20 0x7f87d41cf0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:01:32.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.500+0000 7f87d2d9d640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f87d40a3390 0x7f87d413d310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:53656/0 (socket says 172.21.15.42:53656) 2024-09-16T04:01:32.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.500+0000 7f87d2d9d640 1 -- 172.21.15.42:0/3895622661 learned_addr learned my addr 172.21.15.42:0/3895622661 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:01:32.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.500+0000 7f87d2d9d640 10 monclient(hunting): get_auth_request con 0x7f87d40a3390 auth_method 0 2024-09-16T04:01:32.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.500+0000 7f87d2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:01:32.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.500+0000 7f87d2d9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:01:32.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.500+0000 7f87d2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:01:32.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.500+0000 7f87d259c640 10 monclient(hunting): get_auth_request con 0x7f87d406dc20 auth_method 0 2024-09-16T04:01:32.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.500+0000 7f87d259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:01:32.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.500+0000 7f87d259c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:01:32.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.500+0000 7f87d259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:01:32.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.500+0000 7f87d2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:01:32.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.500+0000 7f87d2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:01:32.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.500+0000 7f87d2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:01:32.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.500+0000 7f87d259c640 1 -- 172.21.15.42:0/3895622661 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f87d406dc20 msgr2=0x7f87d41cf0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-16T04:01:32.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.500+0000 7f87d259c640 1 -- 172.21.15.42:0/3895622661 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f87d406dc20 msgr2=0x7f87d41cf0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:01:32.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.500+0000 7f87d259c640 1 --2- 172.21.15.42:0/3895622661 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f87d406dc20 0x7f87d41cf0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:01:32.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.500+0000 7f87d259c640 1 --2- 172.21.15.42:0/3895622661 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f87d406dc20 0x7f87d41cf0e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:01:32.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.500+0000 7f87d1d9b640 1 --2- 172.21.15.42:0/3895622661 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f87d40a28a0 0x7f87d41cf8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:01:32.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.500+0000 7f87d2d9d640 10 monclient(hunting): handle_auth_done global_id 825369 payload 824 2024-09-16T04:01:32.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.500+0000 7f87d2d9d640 1 -- 172.21.15.42:0/3895622661 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f87d406dc20 msgr2=0x7f87d41cf0e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:01:32.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.500+0000 7f87d2d9d640 1 --2- 172.21.15.42:0/3895622661 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f87d406dc20 0x7f87d41cf0e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:32.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.500+0000 7f87d2d9d640 1 -- 172.21.15.42:0/3895622661 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f87d40a28a0 msgr2=0x7f87d41cf8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:01:32.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.500+0000 7f87d2d9d640 1 --2- 172.21.15.42:0/3895622661 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f87d40a28a0 0x7f87d41cf8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:32.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.500+0000 7f87d2d9d640 10 monclient: _finish_hunting 0 2024-09-16T04:01:32.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.500+0000 7f87d2d9d640 1 monclient: found mon.noname-a 2024-09-16T04:01:32.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.500+0000 7f87d2d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:01:32.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.500+0000 7f87d2d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:01:32.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.500+0000 7f87d2d9d640 1 -- 172.21.15.42:0/3895622661 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87d411fa50 con 0x7f87d40a3390 2024-09-16T04:01:32.521 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.500+0000 7f87d1d9b640 10 monclient: get_auth_request con 0x7f87d40a28a0 auth_method 0 2024-09-16T04:01:32.521 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.500+0000 7f87d1d9b640 1 --2- 172.21.15.42:0/3895622661 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f87d40a28a0 0x7f87d41cf8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T04:01:32.521 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.500+0000 7f87d2d9d640 1 --2- 172.21.15.42:0/3895622661 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f87d40a3390 0x7f87d413d310 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f87cc0ae040 tx=0x7f87cc0190c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:01:32.521 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.500+0000 7f87caffd640 10 client.?.objecter ms_handle_connect 0x7f87d40a3390 2024-09-16T04:01:32.521 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.500+0000 7f87caffd640 10 client.?.objecter resend_mon_ops 2024-09-16T04:01:32.521 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.500+0000 7f87caffd640 1 -- 172.21.15.42:0/3895622661 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87cc0b8020 con 0x7f87d40a3390 2024-09-16T04:01:32.521 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.500+0000 7f87caffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:01:32.521 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.500+0000 7f87caffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:01:32.521 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.500+0000 7f87caffd640 10 monclient: dump: 2024-09-16T04:01:32.521 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:01:32.521 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:01:32.521 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:32.521 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:32.521 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:01:32.522 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:01:32.522 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:01:32.522 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:01:32.522 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:01:32.522 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:01:32.522 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.500+0000 7f87caffd640 10 monclient: _finish_auth 0 2024-09-16T04:01:32.522 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.500+0000 7f87caffd640 10 monclient: _check_auth_tickets 2024-09-16T04:01:32.522 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.500+0000 7f87caffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:01:32.522 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.500+0000 7f87d8c7c640 5 monclient: authenticate success, global_id 825369 2024-09-16T04:01:32.522 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.500+0000 7f87caffd640 1 -- 172.21.15.42:0/3895622661 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f87cc0aa750 con 0x7f87d40a3390 2024-09-16T04:01:32.522 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.500+0000 7f87caffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:01:32.522 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.500+0000 7f87caffd640 1 -- 172.21.15.42:0/3895622661 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87cc0b0040 con 0x7f87d40a3390 2024-09-16T04:01:32.522 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.500+0000 7f87d8c7c640 10 monclient: _renew_subs 2024-09-16T04:01:32.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.500+0000 7f87d8c7c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:01:32.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.500+0000 7f87d8c7c640 1 -- 172.21.15.42:0/3895622661 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87d411fd30 con 0x7f87d40a3390 2024-09-16T04:01:32.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.500+0000 7f87caffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:01:32.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.500+0000 7f87caffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:01:32.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.500+0000 7f87caffd640 10 monclient: dump: 2024-09-16T04:01:32.523 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:01:32.523 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:01:32.523 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:32.523 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:32.523 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:01:32.523 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:01:32.523 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:01:32.523 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:01:32.523 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:01:32.524 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:01:32.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.500+0000 7f87cbfff640 4 set_mon_vals no callback set 2024-09-16T04:01:32.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.500+0000 7f87d8c7c640 10 client.825369.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:01:32.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.500+0000 7f87d8c7c640 10 monclient: _renew_subs 2024-09-16T04:01:32.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.500+0000 7f87d8c7c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:01:32.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.500+0000 7f87d8c7c640 1 -- 172.21.15.42:0/3895622661 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87d4073af0 con 0x7f87d40a3390 2024-09-16T04:01:32.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.500+0000 7f87d8c7c640 1 librados: init done 2024-09-16T04:01:32.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.500+0000 7f87d8c7c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:01:32.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.500+0000 7f87d8c7c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:01:32.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.500+0000 7f87d8c7c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:01:32.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.500+0000 7f87d8c7c640 1 -- 172.21.15.42:0/3895622661 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f87d40a2c80 con 0x7f87d40a3390 2024-09-16T04:01:32.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.500+0000 7f87caffd640 1 -- 172.21.15.42:0/3895622661 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f87cc0bb020 con 0x7f87d40a3390 2024-09-16T04:01:32.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.500+0000 7f87caffd640 1 --2- 172.21.15.42:0/3895622661 >> v2:172.21.15.154:6812/254845342 conn(0x7f87b4037810 0x7f87b4039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:01:32.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.500+0000 7f87caffd640 1 -- 172.21.15.42:0/3895622661 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(590..590 src has 451..590) ==== 6257+0+0 (secure 0 0 0) 0x7f87cc0e31f0 con 0x7f87d40a3390 2024-09-16T04:01:32.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.500+0000 7f87caffd640 10 client.825369.objecter ms_dispatch 0x7f87d4000ba0 osd_map(590..590 src has 451..590) 2024-09-16T04:01:32.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.500+0000 7f87caffd640 3 client.825369.objecter handle_osd_map got epochs [590,590] > 0 2024-09-16T04:01:32.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.500+0000 7f87caffd640 3 client.825369.objecter handle_osd_map decoding full epoch 590 2024-09-16T04:01:32.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.504+0000 7f87caffd640 20 client.825369.objecter dump_active .. 0 homeless 2024-09-16T04:01:32.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.504+0000 7f87d259c640 1 --2- 172.21.15.42:0/3895622661 >> v2:172.21.15.154:6812/254845342 conn(0x7f87b4037810 0x7f87b4039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:01:32.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.504+0000 7f87d259c640 10 monclient: get_auth_request con 0x7f87b4037810 auth_method 0 2024-09-16T04:01:32.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.504+0000 7f87caffd640 10 client.825369.objecter ms_handle_connect 0x7f87b4037810 2024-09-16T04:01:32.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.504+0000 7f87d259c640 1 --2- 172.21.15.42:0/3895622661 >> v2:172.21.15.154:6812/254845342 conn(0x7f87b4037810 0x7f87b4039cd0 secure :-1 s=READY pgs=3761 cs=0 l=1 rev1=1 crypto rx=0x7f87bc0012f0 tx=0x7f87bc009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:01:32.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.508+0000 7f87caffd640 1 -- 172.21.15.42:0/3895622661 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f87cc0c2030 con 0x7f87d40a3390 2024-09-16T04:01:32.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.508+0000 7f87caffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:01:32.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.508+0000 7f87caffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:01:32.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.740+0000 7f87d8c7c640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:01:32.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.740+0000 7f87d8c7c640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:01:32.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.740+0000 7f87d8c7c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:01:32.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.740+0000 7f87d8c7c640 1 -- 172.21.15.42:0/3895622661 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f87d40a72d0 con 0x7f87d40a3390 2024-09-16T04:01:32.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.740+0000 7f87caffd640 1 -- 172.21.15.42:0/3895622661 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7f87cc016030 con 0x7f87d40a3390 2024-09-16T04:01:32.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.740+0000 7f87caffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:01:32.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.740+0000 7f87caffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:01:32.748 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":172,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":21,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:01:32.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.744+0000 7f87d8c7c640 10 librados: watch_flush enter 2024-09-16T04:01:32.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.744+0000 7f87d8c7c640 10 librados: watch_flush exit 2024-09-16T04:01:32.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.744+0000 7f87d8c7c640 20 client.825369.objecter shutdown clearing up homeless session... 2024-09-16T04:01:32.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.744+0000 7f87d8c7c640 10 client.825369.objecter successfully canceled tick 2024-09-16T04:01:32.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.744+0000 7f87d8c7c640 1 -- 172.21.15.42:0/3895622661 >> v2:172.21.15.154:6812/254845342 conn(0x7f87b4037810 msgr2=0x7f87b4039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:01:32.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.744+0000 7f87d8c7c640 1 --2- 172.21.15.42:0/3895622661 >> v2:172.21.15.154:6812/254845342 conn(0x7f87b4037810 0x7f87b4039cd0 secure :-1 s=READY pgs=3761 cs=0 l=1 rev1=1 crypto rx=0x7f87bc0012f0 tx=0x7f87bc009040 comp rx=0 tx=0).stop 2024-09-16T04:01:32.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.744+0000 7f87d8c7c640 10 monclient: shutdown 2024-09-16T04:01:32.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.744+0000 7f87d8c7c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:01:32.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.744+0000 7f87d8c7c640 1 -- 172.21.15.42:0/3895622661 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f87d40a3390 msgr2=0x7f87d413d310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:01:32.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.744+0000 7f87d8c7c640 1 --2- 172.21.15.42:0/3895622661 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f87d40a3390 0x7f87d413d310 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f87cc0ae040 tx=0x7f87cc0190c0 comp rx=0 tx=0).stop 2024-09-16T04:01:32.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.744+0000 7f87d8c7c640 1 -- 172.21.15.42:0/3895622661 shutdown_connections 2024-09-16T04:01:32.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.744+0000 7f87d8c7c640 1 --2- 172.21.15.42:0/3895622661 >> v2:172.21.15.154:6812/254845342 conn(0x7f87b4037810 0x7f87b4039cd0 unknown :-1 s=CLOSED pgs=3761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:32.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.744+0000 7f87d8c7c640 1 --2- 172.21.15.42:0/3895622661 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f87d40a3390 0x7f87d413d310 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:32.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.744+0000 7f87d8c7c640 1 --2- 172.21.15.42:0/3895622661 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f87d40a28a0 0x7f87d41cf8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:32.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.744+0000 7f87d8c7c640 1 --2- 172.21.15.42:0/3895622661 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f87d406dc20 0x7f87d41cf0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:32.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.744+0000 7f87d8c7c640 1 -- 172.21.15.42:0/3895622661 >> 172.21.15.42:0/3895622661 conn(0x7f87d40a7e40 msgr2=0x7f87d4070f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:01:32.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.744+0000 7f87d8c7c640 1 -- 172.21.15.42:0/3895622661 shutdown_connections 2024-09-16T04:01:32.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.744+0000 7f87d8c7c640 1 -- 172.21.15.42:0/3895622661 wait complete. 2024-09-16T04:01:32.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:32.744+0000 7f87d8c7c640 1 librados: shutdown 2024-09-16T04:01:35.768 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:01:35.900 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.892+0000 7fefff305640 10 monclient: get_monmap_and_config 2024-09-16T04:01:35.900 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.892+0000 7fefff305640 10 monclient: build_initial_monmap 2024-09-16T04:01:35.900 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.892+0000 7fefff305640 10 monclient: monmap: 2024-09-16T04:01:35.900 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:01:35.900 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:01:35.900 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:01:35.898512+0000 2024-09-16T04:01:35.900 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:01:35.898512+0000 2024-09-16T04:01:35.900 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:01:35.901 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:01:35.901 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:01:35.901 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:01:35.901 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:01:35.901 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:01:35.901 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:01:35.901 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:01:35.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.892+0000 7fefff305640 1 Processor -- start 2024-09-16T04:01:35.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.892+0000 7fefff305640 1 -- start start 2024-09-16T04:01:35.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.892+0000 7fefff305640 10 monclient: init 2024-09-16T04:01:35.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.896+0000 7fefff305640 10 monclient: _reopen_session rank -1 2024-09-16T04:01:35.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.896+0000 7fefff305640 10 monclient: _add_conns ranks=[3,1,2,4,0] 2024-09-16T04:01:35.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.896+0000 7fefff305640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7feff806db90 0x7feff81340c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:01:35.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.896+0000 7fefff305640 10 monclient(hunting): picked mon.noname-a con 0x7feff806db90 addr v2:172.21.15.42:3300/0 2024-09-16T04:01:35.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.896+0000 7fefff305640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7feff80a37f0 addr v1:172.21.15.154:6789/0 2024-09-16T04:01:35.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.896+0000 7fefff305640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7feff80a3c00 0x7feff8137520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:01:35.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.896+0000 7fefff305640 10 monclient(hunting): picked mon.noname-b con 0x7feff80a3c00 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:01:35.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.896+0000 7fefff305640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feff806ece0 con 0x7feff80a37f0 2024-09-16T04:01:35.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.896+0000 7fefff305640 10 monclient(hunting): start opening mon connection 2024-09-16T04:01:35.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.896+0000 7fefff305640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7feff806df70 con 0x7feff806db90 2024-09-16T04:01:35.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.896+0000 7fefff305640 10 monclient(hunting): start opening mon connection 2024-09-16T04:01:35.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.896+0000 7fefff305640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7feff809bf80 con 0x7feff80a3c00 2024-09-16T04:01:35.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.896+0000 7fefff305640 10 monclient(hunting): _renew_subs 2024-09-16T04:01:35.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.896+0000 7fefff305640 10 monclient(hunting): authenticate will time out at 2142.868295s 2024-09-16T04:01:35.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.896+0000 7feffd8a2640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7feff80a3c00 0x7feff8137520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:01:35.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.896+0000 7feffd0a1640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7feff806db90 0x7feff81340c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:01:35.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.896+0000 7feffd8a2640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7feff80a3c00 0x7feff8137520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:44762/0 (socket says 172.21.15.42:44762) 2024-09-16T04:01:35.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.896+0000 7feffd8a2640 1 -- 172.21.15.42:0/2931335228 learned_addr learned my addr 172.21.15.42:0/2931335228 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:01:35.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.896+0000 7feffd8a2640 10 monclient(hunting): get_auth_request con 0x7feff80a3c00 auth_method 0 2024-09-16T04:01:35.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.896+0000 7feffd8a2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:01:35.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.896+0000 7feffd8a2640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:01:35.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.896+0000 7feffd8a2640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:01:35.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.896+0000 7feffd0a1640 10 monclient(hunting): get_auth_request con 0x7feff806db90 auth_method 0 2024-09-16T04:01:35.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.896+0000 7feffd0a1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:01:35.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.896+0000 7feffd0a1640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:01:35.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.896+0000 7feffd0a1640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:01:35.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.896+0000 7feffd8a2640 1 -- 172.21.15.42:0/2931335228 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7feff80a3c00 msgr2=0x7feff8137520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T04:01:35.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.896+0000 7feffd8a2640 1 -- 172.21.15.42:0/2931335228 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7feff80a3c00 msgr2=0x7feff8137520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:01:35.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.896+0000 7feffd8a2640 1 --2- 172.21.15.42:0/2931335228 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7feff80a3c00 0x7feff8137520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:01:35.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.896+0000 7feffd0a1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:01:35.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.896+0000 7feffd0a1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:01:35.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.896+0000 7feffd8a2640 1 --2- 172.21.15.42:0/2931335228 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7feff80a3c00 0x7feff8137520 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:01:35.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.896+0000 7feffd0a1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:01:35.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.896+0000 7feffd0a1640 10 monclient(hunting): handle_auth_done global_id 825372 payload 274 2024-09-16T04:01:35.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.896+0000 7feffd0a1640 1 -- 172.21.15.42:0/2931335228 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7feff80a3c00 msgr2=0x7feff8137520 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:01:35.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.896+0000 7feffd0a1640 1 --2- 172.21.15.42:0/2931335228 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7feff80a3c00 0x7feff8137520 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:35.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.896+0000 7feffd0a1640 1 -- 172.21.15.42:0/2931335228 >> v1:172.21.15.154:6789/0 conn(0x7feff80a37f0 legacy=0x7feff8136e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:01:35.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.896+0000 7feffd0a1640 10 monclient: _finish_hunting 0 2024-09-16T04:01:35.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.896+0000 7feffd0a1640 1 monclient: found mon.noname-a 2024-09-16T04:01:35.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.896+0000 7feffd0a1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:01:35.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.896+0000 7feffd0a1640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:01:35.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.896+0000 7feffd0a1640 1 -- 172.21.15.42:0/2931335228 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feff8134780 con 0x7feff806db90 2024-09-16T04:01:35.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.900+0000 7feffd0a1640 1 --2- 172.21.15.42:0/2931335228 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feff806db90 0x7feff81340c0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fefe80097c0 tx=0x7fefe801c360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8729eece87550961 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:01:35.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.900+0000 7feff7fff640 1 -- 172.21.15.42:0/2931335228 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefe80b0070 con 0x7feff806db90 2024-09-16T04:01:35.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.900+0000 7feff7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:01:35.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.900+0000 7feff7fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:01:35.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.900+0000 7feff7fff640 10 monclient: dump: 2024-09-16T04:01:35.909 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:01:35.909 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:01:35.909 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:35.909 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:35.909 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:01:35.909 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:01:35.909 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:01:35.910 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:01:35.910 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:01:35.910 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:01:35.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.900+0000 7feff7fff640 10 monclient: _finish_auth 0 2024-09-16T04:01:35.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.900+0000 7feff7fff640 10 monclient: _check_auth_tickets 2024-09-16T04:01:35.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.900+0000 7feff7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:01:35.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.900+0000 7feff7fff640 1 -- 172.21.15.42:0/2931335228 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fefe80ab600 con 0x7feff806db90 2024-09-16T04:01:35.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.900+0000 7feff7fff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:01:35.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.900+0000 7feff7fff640 1 -- 172.21.15.42:0/2931335228 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefe80ab9d0 con 0x7feff806db90 2024-09-16T04:01:35.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.900+0000 7feff7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:01:35.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.900+0000 7feff7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:01:35.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.900+0000 7feff7fff640 10 monclient: dump: 2024-09-16T04:01:35.910 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:01:35.911 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:01:35.911 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:35.911 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:35.911 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:01:35.911 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:01:35.911 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:01:35.911 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:01:35.911 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:01:35.911 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:01:35.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.900+0000 7fefff305640 5 monclient: authenticate success, global_id 825372 2024-09-16T04:01:35.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.900+0000 7fefff305640 10 monclient: get_monmap_and_config success 2024-09-16T04:01:35.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.900+0000 7fefff305640 4 set_mon_vals no callback set 2024-09-16T04:01:35.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.900+0000 7fefff305640 10 monclient: shutdown 2024-09-16T04:01:35.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.900+0000 7fefff305640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:01:35.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.900+0000 7fefff305640 1 -- 172.21.15.42:0/2931335228 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feff806db90 msgr2=0x7feff81340c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:01:35.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.900+0000 7fefff305640 1 --2- 172.21.15.42:0/2931335228 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feff806db90 0x7feff81340c0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fefe80097c0 tx=0x7fefe801c360 comp rx=0 tx=0).stop 2024-09-16T04:01:35.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.900+0000 7fefff305640 1 -- 172.21.15.42:0/2931335228 shutdown_connections 2024-09-16T04:01:35.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.900+0000 7fefff305640 1 --2- 172.21.15.42:0/2931335228 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7feff80a3c00 0x7feff8137520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:35.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.900+0000 7fefff305640 1 --2- 172.21.15.42:0/2931335228 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feff806db90 0x7feff81340c0 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:35.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.900+0000 7fefff305640 1 -- 172.21.15.42:0/2931335228 >> 172.21.15.42:0/2931335228 conn(0x7feff80a9fb0 msgr2=0x7feff80aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:01:35.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.900+0000 7fefff305640 1 -- 172.21.15.42:0/2931335228 shutdown_connections 2024-09-16T04:01:35.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.900+0000 7fefff305640 1 -- 172.21.15.42:0/2931335228 wait complete. 2024-09-16T04:01:35.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.900+0000 7fefff305640 10 monclient: build_initial_monmap 2024-09-16T04:01:35.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.900+0000 7fefff305640 10 monclient: monmap: 2024-09-16T04:01:35.912 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:01:35.913 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:01:35.913 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:01:35.913 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:01:35.913 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:01:35.913 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:01:35.913 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:01:35.913 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:01:35.913 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:01:35.913 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:01:35.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.904+0000 7fefff305640 1 librados: starting msgr at 2024-09-16T04:01:35.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.904+0000 7fefff305640 1 librados: starting objecter 2024-09-16T04:01:35.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.904+0000 7fefff305640 1 Processor -- start 2024-09-16T04:01:35.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.904+0000 7fefff305640 1 -- start start 2024-09-16T04:01:35.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.904+0000 7fefff305640 1 librados: setting wanted keys 2024-09-16T04:01:35.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.904+0000 7fefff305640 1 librados: calling monclient init 2024-09-16T04:01:35.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.904+0000 7fefff305640 10 monclient: init 2024-09-16T04:01:35.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.904+0000 7fefff305640 10 monclient: _reopen_session rank -1 2024-09-16T04:01:35.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.904+0000 7fefff305640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:01:35.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.904+0000 7fefff305640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feff806db90 0x7feff813e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:01:35.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.904+0000 7fefff305640 10 monclient(hunting): picked mon.noname-b con 0x7feff806db90 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:01:35.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.904+0000 7fefff305640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feff80a37f0 0x7feff81cf050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:01:35.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.904+0000 7fefff305640 10 monclient(hunting): picked mon.noname-a con 0x7feff80a37f0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:01:35.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.904+0000 7fefff305640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7feff80a3c00 0x7feff81cf590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:01:35.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.904+0000 7fefff305640 10 monclient(hunting): picked mon.noname-c con 0x7feff80a3c00 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:01:35.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.904+0000 7fefff305640 10 monclient(hunting): start opening mon connection 2024-09-16T04:01:35.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.904+0000 7fefff305640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7feff81419d0 con 0x7feff80a37f0 2024-09-16T04:01:35.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.904+0000 7fefff305640 10 monclient(hunting): start opening mon connection 2024-09-16T04:01:35.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.904+0000 7fefff305640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7feff8137c70 con 0x7feff806db90 2024-09-16T04:01:35.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.904+0000 7fefff305640 10 monclient(hunting): start opening mon connection 2024-09-16T04:01:35.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.904+0000 7fefff305640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7feff8141b50 con 0x7feff80a3c00 2024-09-16T04:01:35.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.904+0000 7fefff305640 10 monclient(hunting): _renew_subs 2024-09-16T04:01:35.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.904+0000 7fefff305640 10 monclient(hunting): authenticate will time out at 2142.877090s 2024-09-16T04:01:35.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.904+0000 7feffd8a2640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7feff80a3c00 0x7feff81cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:01:35.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.904+0000 7feffc8a0640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feff80a37f0 0x7feff81cf050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:01:35.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.904+0000 7feffd8a2640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7feff80a3c00 0x7feff81cf590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:44774/0 (socket says 172.21.15.42:44774) 2024-09-16T04:01:35.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.904+0000 7feffd8a2640 1 -- 172.21.15.42:0/2337306805 learned_addr learned my addr 172.21.15.42:0/2337306805 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:01:35.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.904+0000 7feffc8a0640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feff80a37f0 0x7feff81cf050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:40938/0 (socket says 172.21.15.42:40938) 2024-09-16T04:01:35.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.904+0000 7feffd8a2640 10 monclient(hunting): get_auth_request con 0x7feff80a3c00 auth_method 0 2024-09-16T04:01:35.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.904+0000 7feffd8a2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:01:35.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.904+0000 7feffd8a2640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:01:35.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.904+0000 7feffd8a2640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:01:35.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.904+0000 7feffc8a0640 10 monclient(hunting): get_auth_request con 0x7feff80a37f0 auth_method 0 2024-09-16T04:01:35.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.904+0000 7feffc8a0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:01:35.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.904+0000 7feffc8a0640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:01:35.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.904+0000 7feffc8a0640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:01:35.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.904+0000 7feffd8a2640 1 -- 172.21.15.42:0/2337306805 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7feff80a3c00 msgr2=0x7feff81cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-16T04:01:35.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.904+0000 7feffd8a2640 1 -- 172.21.15.42:0/2337306805 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7feff80a3c00 msgr2=0x7feff81cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:01:35.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.904+0000 7feffd8a2640 1 --2- 172.21.15.42:0/2337306805 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7feff80a3c00 0x7feff81cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:01:35.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.904+0000 7feffc8a0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:01:35.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.904+0000 7feffc8a0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:01:35.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.904+0000 7feffd8a2640 1 --2- 172.21.15.42:0/2337306805 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7feff80a3c00 0x7feff81cf590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:01:35.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.904+0000 7feffc8a0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:01:35.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.904+0000 7feffd0a1640 1 --2- 172.21.15.42:0/2337306805 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feff806db90 0x7feff813e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:01:35.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.904+0000 7feffd0a1640 10 monclient(hunting): get_auth_request con 0x7feff806db90 auth_method 0 2024-09-16T04:01:35.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.904+0000 7feffd0a1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:01:35.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.904+0000 7feffd0a1640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:01:35.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.904+0000 7feffd0a1640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:01:35.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.904+0000 7feffc8a0640 10 monclient(hunting): handle_auth_done global_id 825375 payload 824 2024-09-16T04:01:35.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.904+0000 7feffc8a0640 1 -- 172.21.15.42:0/2337306805 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7feff80a3c00 msgr2=0x7feff81cf590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:01:35.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.904+0000 7feffc8a0640 1 --2- 172.21.15.42:0/2337306805 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7feff80a3c00 0x7feff81cf590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:35.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.904+0000 7feffc8a0640 1 -- 172.21.15.42:0/2337306805 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feff806db90 msgr2=0x7feff813e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:01:35.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.904+0000 7feffc8a0640 1 --2- 172.21.15.42:0/2337306805 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feff806db90 0x7feff813e650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:35.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.904+0000 7feffc8a0640 10 monclient: _finish_hunting 0 2024-09-16T04:01:35.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.904+0000 7feffc8a0640 1 monclient: found mon.noname-a 2024-09-16T04:01:35.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.904+0000 7feffc8a0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:01:35.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.904+0000 7feffc8a0640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:01:35.923 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.904+0000 7feffc8a0640 1 -- 172.21.15.42:0/2337306805 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feff813bbe0 con 0x7feff80a37f0 2024-09-16T04:01:35.923 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.904+0000 7feffc8a0640 1 --2- 172.21.15.42:0/2337306805 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feff80a37f0 0x7feff81cf050 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fefec00c3c0 tx=0x7fefec00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:01:35.923 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.904+0000 7feff5ffb640 10 client.?.objecter ms_handle_connect 0x7feff80a37f0 2024-09-16T04:01:35.923 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.904+0000 7feff5ffb640 10 client.?.objecter resend_mon_ops 2024-09-16T04:01:35.923 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.904+0000 7feff5ffb640 1 -- 172.21.15.42:0/2337306805 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefec0052d0 con 0x7feff80a37f0 2024-09-16T04:01:35.923 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.904+0000 7feff5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:01:35.923 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.904+0000 7feff5ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:01:35.923 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.904+0000 7feff5ffb640 10 monclient: dump: 2024-09-16T04:01:35.923 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:01:35.923 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:01:35.923 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:35.923 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:35.923 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:01:35.924 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:01:35.924 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:01:35.924 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:01:35.924 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:01:35.924 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:01:35.924 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.904+0000 7feff5ffb640 10 monclient: _finish_auth 0 2024-09-16T04:01:35.924 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.904+0000 7feff5ffb640 10 monclient: _check_auth_tickets 2024-09-16T04:01:35.924 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.904+0000 7feff5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:01:35.924 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.904+0000 7feff5ffb640 1 -- 172.21.15.42:0/2337306805 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fefec005470 con 0x7feff80a37f0 2024-09-16T04:01:35.924 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.904+0000 7fefff305640 5 monclient: authenticate success, global_id 825375 2024-09-16T04:01:35.924 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.904+0000 7fefff305640 10 monclient: _renew_subs 2024-09-16T04:01:35.924 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.904+0000 7fefff305640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:01:35.924 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.904+0000 7fefff305640 1 -- 172.21.15.42:0/2337306805 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feff8133200 con 0x7feff80a37f0 2024-09-16T04:01:35.924 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.904+0000 7feff5ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:01:35.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.904+0000 7feff5ffb640 1 -- 172.21.15.42:0/2337306805 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefec007260 con 0x7feff80a37f0 2024-09-16T04:01:35.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.904+0000 7feff5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:01:35.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.904+0000 7feff5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:01:35.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.904+0000 7feff5ffb640 10 monclient: dump: 2024-09-16T04:01:35.925 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:01:35.925 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:01:35.925 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:35.925 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:35.925 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:01:35.925 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:01:35.925 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:01:35.925 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:01:35.925 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:01:35.925 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:01:35.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.904+0000 7feff77fe640 4 set_mon_vals no callback set 2024-09-16T04:01:35.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.904+0000 7fefff305640 10 client.825375.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:01:35.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.904+0000 7fefff305640 10 monclient: _renew_subs 2024-09-16T04:01:35.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.904+0000 7fefff305640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:01:35.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.904+0000 7fefff305640 1 -- 172.21.15.42:0/2337306805 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feff813d630 con 0x7feff80a37f0 2024-09-16T04:01:35.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.904+0000 7fefff305640 1 librados: init done 2024-09-16T04:01:35.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.904+0000 7fefff305640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:01:35.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.904+0000 7fefff305640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:01:35.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.904+0000 7fefff305640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:01:35.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.904+0000 7fefff305640 1 -- 172.21.15.42:0/2337306805 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feff81340c0 con 0x7feff80a37f0 2024-09-16T04:01:35.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.904+0000 7feff5ffb640 1 -- 172.21.15.42:0/2337306805 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fefec007400 con 0x7feff80a37f0 2024-09-16T04:01:35.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.904+0000 7feff5ffb640 1 --2- 172.21.15.42:0/2337306805 >> v2:172.21.15.154:6812/254845342 conn(0x7fefe4037810 0x7fefe4039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:01:35.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.904+0000 7feff5ffb640 1 -- 172.21.15.42:0/2337306805 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(594..594 src has 451..594) ==== 6252+0+0 (secure 0 0 0) 0x7fefec048770 con 0x7feff80a37f0 2024-09-16T04:01:35.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.904+0000 7feff5ffb640 10 client.825375.objecter ms_dispatch 0x7feff8000ba0 osd_map(594..594 src has 451..594) 2024-09-16T04:01:35.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.904+0000 7feff5ffb640 3 client.825375.objecter handle_osd_map got epochs [594,594] > 0 2024-09-16T04:01:35.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.904+0000 7feff5ffb640 3 client.825375.objecter handle_osd_map decoding full epoch 594 2024-09-16T04:01:35.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.904+0000 7feff5ffb640 20 client.825375.objecter dump_active .. 0 homeless 2024-09-16T04:01:35.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.904+0000 7feffd0a1640 1 --2- 172.21.15.42:0/2337306805 >> v2:172.21.15.154:6812/254845342 conn(0x7fefe4037810 0x7fefe4039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:01:35.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.904+0000 7feffd0a1640 10 monclient: get_auth_request con 0x7fefe4037810 auth_method 0 2024-09-16T04:01:35.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.908+0000 7feff5ffb640 10 client.825375.objecter ms_handle_connect 0x7fefe4037810 2024-09-16T04:01:35.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.908+0000 7feffd0a1640 1 --2- 172.21.15.42:0/2337306805 >> v2:172.21.15.154:6812/254845342 conn(0x7fefe4037810 0x7fefe4039cd0 secure :-1 s=READY pgs=3762 cs=0 l=1 rev1=1 crypto rx=0x7fefe80ad000 tx=0x7fefe8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:01:35.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.912+0000 7feff5ffb640 1 -- 172.21.15.42:0/2337306805 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7fefec006890 con 0x7feff80a37f0 2024-09-16T04:01:35.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.912+0000 7feff5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:01:35.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:35.912+0000 7feff5ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:01:36.148 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:36.140+0000 7fefff305640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:01:36.148 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:36.140+0000 7fefff305640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:01:36.148 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:36.140+0000 7fefff305640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:01:36.148 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:36.140+0000 7fefff305640 1 -- 172.21.15.42:0/2337306805 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7feff8068e40 con 0x7feff80a37f0 2024-09-16T04:01:36.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:36.140+0000 7feff5ffb640 1 -- 172.21.15.42:0/2337306805 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7fefec003600 con 0x7feff80a37f0 2024-09-16T04:01:36.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:36.140+0000 7feff5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:01:36.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:36.140+0000 7feff5ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:01:36.149 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":172,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":24,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:01:36.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:36.144+0000 7fefff305640 10 librados: watch_flush enter 2024-09-16T04:01:36.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:36.144+0000 7fefff305640 10 librados: watch_flush exit 2024-09-16T04:01:36.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:36.144+0000 7fefff305640 20 client.825375.objecter shutdown clearing up homeless session... 2024-09-16T04:01:36.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:36.144+0000 7fefff305640 10 client.825375.objecter successfully canceled tick 2024-09-16T04:01:36.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:36.144+0000 7fefff305640 1 -- 172.21.15.42:0/2337306805 >> v2:172.21.15.154:6812/254845342 conn(0x7fefe4037810 msgr2=0x7fefe4039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:01:36.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:36.144+0000 7fefff305640 1 --2- 172.21.15.42:0/2337306805 >> v2:172.21.15.154:6812/254845342 conn(0x7fefe4037810 0x7fefe4039cd0 secure :-1 s=READY pgs=3762 cs=0 l=1 rev1=1 crypto rx=0x7fefe80ad000 tx=0x7fefe8008040 comp rx=0 tx=0).stop 2024-09-16T04:01:36.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:36.144+0000 7fefff305640 10 monclient: shutdown 2024-09-16T04:01:36.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:36.144+0000 7fefff305640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:01:36.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:36.144+0000 7fefff305640 1 -- 172.21.15.42:0/2337306805 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feff80a37f0 msgr2=0x7feff81cf050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:01:36.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:36.144+0000 7fefff305640 1 --2- 172.21.15.42:0/2337306805 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feff80a37f0 0x7feff81cf050 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fefec00c3c0 tx=0x7fefec00c880 comp rx=0 tx=0).stop 2024-09-16T04:01:36.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:36.144+0000 7fefff305640 1 -- 172.21.15.42:0/2337306805 shutdown_connections 2024-09-16T04:01:36.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:36.144+0000 7fefff305640 1 --2- 172.21.15.42:0/2337306805 >> v2:172.21.15.154:6812/254845342 conn(0x7fefe4037810 0x7fefe4039cd0 unknown :-1 s=CLOSED pgs=3762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:36.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:36.144+0000 7fefff305640 1 --2- 172.21.15.42:0/2337306805 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7feff80a3c00 0x7feff81cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:36.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:36.144+0000 7fefff305640 1 --2- 172.21.15.42:0/2337306805 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feff80a37f0 0x7feff81cf050 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:36.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:36.144+0000 7fefff305640 1 --2- 172.21.15.42:0/2337306805 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feff806db90 0x7feff813e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:36.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:36.144+0000 7fefff305640 1 -- 172.21.15.42:0/2337306805 >> 172.21.15.42:0/2337306805 conn(0x7feff80a9fb0 msgr2=0x7feff80a7ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:01:36.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:36.144+0000 7fefff305640 1 -- 172.21.15.42:0/2337306805 shutdown_connections 2024-09-16T04:01:36.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:36.144+0000 7fefff305640 1 -- 172.21.15.42:0/2337306805 wait complete. 2024-09-16T04:01:36.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:36.144+0000 7fefff305640 1 librados: shutdown 2024-09-16T04:01:39.168 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:01:39.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:39.292+0000 7f2d2fa88640 10 monclient: get_monmap_and_config 2024-09-16T04:01:39.300 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:39.292+0000 7f2d2fa88640 10 monclient: build_initial_monmap 2024-09-16T04:01:39.300 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:39.292+0000 7f2d2fa88640 10 monclient: monmap: 2024-09-16T04:01:39.300 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:01:39.300 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:01:39.300 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:01:39.298131+0000 2024-09-16T04:01:39.300 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:01:39.298131+0000 2024-09-16T04:01:39.300 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:01:39.300 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:01:39.300 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:01:39.300 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:01:39.300 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:01:39.300 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:01:39.301 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:01:39.301 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:01:39.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:39.292+0000 7f2d2fa88640 1 Processor -- start 2024-09-16T04:01:39.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:39.292+0000 7f2d2fa88640 1 -- start start 2024-09-16T04:01:39.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:39.292+0000 7f2d2fa88640 10 monclient: init 2024-09-16T04:01:39.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:39.292+0000 7f2d2fa88640 10 monclient: _reopen_session rank -1 2024-09-16T04:01:39.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:39.292+0000 7f2d2fa88640 10 monclient: _add_conns ranks=[1,2,3,0,4] 2024-09-16T04:01:39.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:39.292+0000 7f2d2fa88640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2d2806db90 addr v1:172.21.15.154:6789/0 2024-09-16T04:01:39.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:39.292+0000 7f2d2fa88640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2d281340c0 0x7f2d2813c190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:01:39.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:39.292+0000 7f2d2fa88640 10 monclient(hunting): picked mon.noname-b con 0x7f2d281340c0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:01:39.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:39.292+0000 7f2d2fa88640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f2d2813c6d0 0x7f2d2813eac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:01:39.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:39.292+0000 7f2d2fa88640 10 monclient(hunting): picked mon.noname-a con 0x7f2d2813c6d0 addr v2:172.21.15.42:3300/0 2024-09-16T04:01:39.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:39.292+0000 7f2d2fa88640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2d2806ece0 con 0x7f2d2806db90 2024-09-16T04:01:39.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:39.292+0000 7f2d2fa88640 10 monclient(hunting): start opening mon connection 2024-09-16T04:01:39.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:39.292+0000 7f2d2fa88640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f2d2806df70 con 0x7f2d2813c6d0 2024-09-16T04:01:39.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:39.292+0000 7f2d2fa88640 10 monclient(hunting): start opening mon connection 2024-09-16T04:01:39.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:39.292+0000 7f2d2fa88640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f2d2809bf80 con 0x7f2d281340c0 2024-09-16T04:01:39.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:39.292+0000 7f2d2fa88640 10 monclient(hunting): _renew_subs 2024-09-16T04:01:39.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:39.292+0000 7f2d2fa88640 10 monclient(hunting): authenticate will time out at 2146.267621s 2024-09-16T04:01:39.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:39.292+0000 7f2d2d023640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2d281340c0 0x7f2d2813c190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:01:39.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:39.292+0000 7f2d2e025640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f2d2813c6d0 0x7f2d2813eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:01:39.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:39.292+0000 7f2d2d023640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2d281340c0 0x7f2d2813c190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:44786/0 (socket says 172.21.15.42:44786) 2024-09-16T04:01:39.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:39.292+0000 7f2d2e025640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f2d2813c6d0 0x7f2d2813eac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:40940/0 (socket says 172.21.15.42:40940) 2024-09-16T04:01:39.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:39.292+0000 7f2d2d023640 1 -- 172.21.15.42:0/3225714512 learned_addr learned my addr 172.21.15.42:0/3225714512 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:01:39.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:39.292+0000 7f2d2d023640 10 monclient(hunting): get_auth_request con 0x7f2d281340c0 auth_method 0 2024-09-16T04:01:39.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:39.292+0000 7f2d2d023640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:01:39.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:39.292+0000 7f2d2d023640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:01:39.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:39.292+0000 7f2d2d023640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:01:39.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:39.292+0000 7f2d2e025640 10 monclient(hunting): get_auth_request con 0x7f2d2813c6d0 auth_method 0 2024-09-16T04:01:39.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:39.292+0000 7f2d2e025640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:01:39.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:39.292+0000 7f2d2e025640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:01:39.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:39.292+0000 7f2d2e025640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:01:39.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:39.296+0000 7f2d2d023640 1 -- 172.21.15.42:0/3225714512 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2d281340c0 msgr2=0x7f2d2813c190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:01:39.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:39.296+0000 7f2d2d023640 1 -- 172.21.15.42:0/3225714512 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2d281340c0 msgr2=0x7f2d2813c190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:01:39.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:39.296+0000 7f2d2d023640 1 --2- 172.21.15.42:0/3225714512 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2d281340c0 0x7f2d2813c190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:01:39.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:39.296+0000 7f2d2e025640 1 -- 172.21.15.42:0/3225714512 >> v2:172.21.15.42:3300/0 conn(0x7f2d2813c6d0 msgr2=0x7f2d2813eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T04:01:39.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:39.296+0000 7f2d2e025640 1 -- 172.21.15.42:0/3225714512 >> v2:172.21.15.42:3300/0 conn(0x7f2d2813c6d0 msgr2=0x7f2d2813eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:01:39.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:39.296+0000 7f2d2e025640 1 --2- 172.21.15.42:0/3225714512 >> v2:172.21.15.42:3300/0 conn(0x7f2d2813c6d0 0x7f2d2813eac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:01:39.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:39.296+0000 7f2d2d023640 1 --2- 172.21.15.42:0/3225714512 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2d281340c0 0x7f2d2813c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:01:39.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:39.296+0000 7f2d2e025640 1 --2- 172.21.15.42:0/3225714512 >> v2:172.21.15.42:3300/0 conn(0x7f2d2813c6d0 0x7f2d2813eac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:01:39.502 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:39.500+0000 7f2d2d023640 1 --2- 172.21.15.42:0/3225714512 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2d281340c0 0x7f2d2813c190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:01:39.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:39.500+0000 7f2d2e025640 1 --2- 172.21.15.42:0/3225714512 >> v2:172.21.15.42:3300/0 conn(0x7f2d2813c6d0 0x7f2d2813eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:01:39.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:39.500+0000 7f2d2d023640 10 monclient(hunting): get_auth_request con 0x7f2d281340c0 auth_method 0 2024-09-16T04:01:39.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:39.500+0000 7f2d2d023640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:01:39.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:39.500+0000 7f2d2d023640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:01:39.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:39.500+0000 7f2d2d023640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:01:39.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:39.500+0000 7f2d2e025640 10 monclient(hunting): get_auth_request con 0x7f2d2813c6d0 auth_method 0 2024-09-16T04:01:39.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:39.500+0000 7f2d2e025640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:01:39.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:39.500+0000 7f2d2e025640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:01:39.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:39.500+0000 7f2d2e025640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:01:39.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:39.500+0000 7f2d2d023640 1 -- 172.21.15.42:0/3225714512 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2d281340c0 msgr2=0x7f2d2813c190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:01:39.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:39.500+0000 7f2d2d023640 1 -- 172.21.15.42:0/3225714512 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2d281340c0 msgr2=0x7f2d2813c190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:01:39.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:39.500+0000 7f2d2d023640 1 --2- 172.21.15.42:0/3225714512 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2d281340c0 0x7f2d2813c190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:01:39.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:39.500+0000 7f2d2e025640 1 -- 172.21.15.42:0/3225714512 >> v2:172.21.15.42:3300/0 conn(0x7f2d2813c6d0 msgr2=0x7f2d2813eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T04:01:39.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:39.500+0000 7f2d2e025640 1 -- 172.21.15.42:0/3225714512 >> v2:172.21.15.42:3300/0 conn(0x7f2d2813c6d0 msgr2=0x7f2d2813eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:01:39.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:39.500+0000 7f2d2e025640 1 --2- 172.21.15.42:0/3225714512 >> v2:172.21.15.42:3300/0 conn(0x7f2d2813c6d0 0x7f2d2813eac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:01:39.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:39.500+0000 7f2d2d023640 1 --2- 172.21.15.42:0/3225714512 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2d281340c0 0x7f2d2813c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T04:01:39.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:39.500+0000 7f2d2e025640 1 --2- 172.21.15.42:0/3225714512 >> v2:172.21.15.42:3300/0 conn(0x7f2d2813c6d0 0x7f2d2813eac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T04:01:39.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:39.904+0000 7f2d2e025640 1 --2- 172.21.15.42:0/3225714512 >> v2:172.21.15.42:3300/0 conn(0x7f2d2813c6d0 0x7f2d2813eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:01:39.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:39.904+0000 7f2d2d023640 1 --2- 172.21.15.42:0/3225714512 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2d281340c0 0x7f2d2813c190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:01:39.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:39.904+0000 7f2d2e025640 10 monclient(hunting): get_auth_request con 0x7f2d2813c6d0 auth_method 0 2024-09-16T04:01:39.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:39.904+0000 7f2d2e025640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:01:39.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:39.904+0000 7f2d2e025640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:01:39.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:39.904+0000 7f2d2e025640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:01:39.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:39.904+0000 7f2d2d023640 10 monclient(hunting): get_auth_request con 0x7f2d281340c0 auth_method 0 2024-09-16T04:01:39.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:39.904+0000 7f2d2d023640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:01:39.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:39.904+0000 7f2d2d023640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:01:39.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:39.904+0000 7f2d2d023640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:01:39.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:39.904+0000 7f2d2e025640 1 -- 172.21.15.42:0/3225714512 >> v2:172.21.15.42:3300/0 conn(0x7f2d2813c6d0 msgr2=0x7f2d2813eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:01:39.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:39.904+0000 7f2d2e025640 1 -- 172.21.15.42:0/3225714512 >> v2:172.21.15.42:3300/0 conn(0x7f2d2813c6d0 msgr2=0x7f2d2813eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:01:39.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:39.904+0000 7f2d2d023640 1 -- 172.21.15.42:0/3225714512 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2d281340c0 msgr2=0x7f2d2813c190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T04:01:39.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:39.904+0000 7f2d2e025640 1 --2- 172.21.15.42:0/3225714512 >> v2:172.21.15.42:3300/0 conn(0x7f2d2813c6d0 0x7f2d2813eac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:01:39.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:39.904+0000 7f2d2d023640 1 -- 172.21.15.42:0/3225714512 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2d281340c0 msgr2=0x7f2d2813c190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:01:39.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:39.904+0000 7f2d2d023640 1 --2- 172.21.15.42:0/3225714512 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2d281340c0 0x7f2d2813c190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:01:39.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:39.904+0000 7f2d2e025640 1 --2- 172.21.15.42:0/3225714512 >> v2:172.21.15.42:3300/0 conn(0x7f2d2813c6d0 0x7f2d2813eac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T04:01:39.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:39.904+0000 7f2d2d023640 1 --2- 172.21.15.42:0/3225714512 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2d281340c0 0x7f2d2813c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T04:01:40.714 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:40.708+0000 7f2d2d023640 1 --2- 172.21.15.42:0/3225714512 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2d281340c0 0x7f2d2813c190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:01:40.714 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:40.708+0000 7f2d2e025640 1 --2- 172.21.15.42:0/3225714512 >> v2:172.21.15.42:3300/0 conn(0x7f2d2813c6d0 0x7f2d2813eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:01:40.714 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:40.708+0000 7f2d2d023640 10 monclient(hunting): get_auth_request con 0x7f2d281340c0 auth_method 0 2024-09-16T04:01:40.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:40.708+0000 7f2d2d023640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:01:40.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:40.708+0000 7f2d2d023640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:01:40.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:40.708+0000 7f2d2d023640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:01:40.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:40.708+0000 7f2d2e025640 10 monclient(hunting): get_auth_request con 0x7f2d2813c6d0 auth_method 0 2024-09-16T04:01:40.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:40.708+0000 7f2d2e025640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:01:40.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:40.708+0000 7f2d2e025640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:01:40.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:40.708+0000 7f2d2e025640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:01:40.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:40.708+0000 7f2d2d023640 1 -- 172.21.15.42:0/3225714512 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2d281340c0 msgr2=0x7f2d2813c190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:01:40.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:40.708+0000 7f2d2e025640 1 -- 172.21.15.42:0/3225714512 >> v2:172.21.15.42:3300/0 conn(0x7f2d2813c6d0 msgr2=0x7f2d2813eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T04:01:40.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:40.708+0000 7f2d2d023640 1 -- 172.21.15.42:0/3225714512 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2d281340c0 msgr2=0x7f2d2813c190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:01:40.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:40.708+0000 7f2d2e025640 1 -- 172.21.15.42:0/3225714512 >> v2:172.21.15.42:3300/0 conn(0x7f2d2813c6d0 msgr2=0x7f2d2813eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:01:40.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:40.708+0000 7f2d2e025640 1 --2- 172.21.15.42:0/3225714512 >> v2:172.21.15.42:3300/0 conn(0x7f2d2813c6d0 0x7f2d2813eac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:01:40.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:40.708+0000 7f2d2d023640 1 --2- 172.21.15.42:0/3225714512 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2d281340c0 0x7f2d2813c190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:01:40.716 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:40.708+0000 7f2d2e025640 1 --2- 172.21.15.42:0/3225714512 >> v2:172.21.15.42:3300/0 conn(0x7f2d2813c6d0 0x7f2d2813eac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T04:01:40.716 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:40.708+0000 7f2d2d023640 1 --2- 172.21.15.42:0/3225714512 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2d281340c0 0x7f2d2813c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T04:01:42.301 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.296+0000 7f2d1f7fe640 10 monclient(hunting): tick 2024-09-16T04:01:42.301 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.296+0000 7f2d1f7fe640 10 monclient(hunting): _check_auth_tickets 2024-09-16T04:01:42.301 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.296+0000 7f2d1f7fe640 1 monclient(hunting): continuing hunt 2024-09-16T04:01:42.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.296+0000 7f2d1f7fe640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T04:01:42.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.296+0000 7f2d1f7fe640 1 -- 172.21.15.42:0/3225714512 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2d281340c0 msgr2=0x7f2d2813c190 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:01:42.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.296+0000 7f2d1f7fe640 1 --2- 172.21.15.42:0/3225714512 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2d281340c0 0x7f2d2813c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:42.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.296+0000 7f2d1f7fe640 1 -- 172.21.15.42:0/3225714512 >> v2:172.21.15.42:3300/0 conn(0x7f2d2813c6d0 msgr2=0x7f2d2813eac0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:01:42.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.296+0000 7f2d1f7fe640 1 --2- 172.21.15.42:0/3225714512 >> v2:172.21.15.42:3300/0 conn(0x7f2d2813c6d0 0x7f2d2813eac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:42.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.296+0000 7f2d1f7fe640 1 -- 172.21.15.42:0/3225714512 >> v1:172.21.15.154:6789/0 conn(0x7f2d2806db90 legacy=0x7f2d280a37f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:01:42.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.296+0000 7f2d1f7fe640 10 monclient: _add_conns ranks=[4,0] 2024-09-16T04:01:42.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.296+0000 7f2d1f7fe640 1 --2- 172.21.15.42:0/3225714512 >> v2:172.21.15.154:3300/0 conn(0x7f2d0c0036b0 0x7f2d0c005b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:01:42.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.296+0000 7f2d1f7fe640 10 monclient(hunting): picked mon.noname-c con 0x7f2d0c0036b0 addr v2:172.21.15.154:3300/0 2024-09-16T04:01:42.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.296+0000 7f2d1f7fe640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2d0c006220 addr v1:172.21.15.42:6789/0 2024-09-16T04:01:42.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.296+0000 7f2d1f7fe640 1 -- 172.21.15.42:0/3225714512 --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2d281344d0 con 0x7f2d0c006220 2024-09-16T04:01:42.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.296+0000 7f2d1f7fe640 10 monclient(hunting): start opening mon connection 2024-09-16T04:01:42.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.296+0000 7f2d1f7fe640 1 -- 172.21.15.42:0/3225714512 --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f2d2806df70 con 0x7f2d0c0036b0 2024-09-16T04:01:42.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.296+0000 7f2d1f7fe640 10 monclient(hunting): _renew_subs 2024-09-16T04:01:42.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.296+0000 7f2d2d824640 1 --2- 172.21.15.42:0/3225714512 >> v2:172.21.15.154:3300/0 conn(0x7f2d0c0036b0 0x7f2d0c005b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:01:42.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.296+0000 7f2d2d824640 10 monclient(hunting): get_auth_request con 0x7f2d0c0036b0 auth_method 0 2024-09-16T04:01:42.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.296+0000 7f2d2d824640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:01:42.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.296+0000 7f2d2d824640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:01:42.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.296+0000 7f2d2d824640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:01:42.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.296+0000 7f2d2d824640 1 -- 172.21.15.42:0/3225714512 >> v2:172.21.15.154:3300/0 conn(0x7f2d0c0036b0 msgr2=0x7f2d0c005b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:01:42.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.296+0000 7f2d2d824640 1 -- 172.21.15.42:0/3225714512 >> v2:172.21.15.154:3300/0 conn(0x7f2d0c0036b0 msgr2=0x7f2d0c005b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:01:42.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.296+0000 7f2d2d824640 1 --2- 172.21.15.42:0/3225714512 >> v2:172.21.15.154:3300/0 conn(0x7f2d0c0036b0 0x7f2d0c005b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:01:42.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.296+0000 7f2d2d824640 1 --2- 172.21.15.42:0/3225714512 >> v2:172.21.15.154:3300/0 conn(0x7f2d0c0036b0 0x7f2d0c005b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:01:42.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.500+0000 7f2d2d824640 1 --2- 172.21.15.42:0/3225714512 >> v2:172.21.15.154:3300/0 conn(0x7f2d0c0036b0 0x7f2d0c005b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:01:42.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.500+0000 7f2d2d824640 10 monclient(hunting): get_auth_request con 0x7f2d0c0036b0 auth_method 0 2024-09-16T04:01:42.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.500+0000 7f2d2d824640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:01:42.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.500+0000 7f2d2d824640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:01:42.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.500+0000 7f2d2d824640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:01:42.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.500+0000 7f2d2d824640 1 -- 172.21.15.42:0/3225714512 >> v2:172.21.15.154:3300/0 conn(0x7f2d0c0036b0 msgr2=0x7f2d0c005b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:01:42.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.500+0000 7f2d2d824640 1 -- 172.21.15.42:0/3225714512 >> v2:172.21.15.154:3300/0 conn(0x7f2d0c0036b0 msgr2=0x7f2d0c005b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:01:42.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.500+0000 7f2d2d824640 1 --2- 172.21.15.42:0/3225714512 >> v2:172.21.15.154:3300/0 conn(0x7f2d0c0036b0 0x7f2d0c005b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:01:42.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.500+0000 7f2d2d824640 1 --2- 172.21.15.42:0/3225714512 >> v2:172.21.15.154:3300/0 conn(0x7f2d0c0036b0 0x7f2d0c005b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T04:01:42.644 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.636+0000 7f2d2c822640 1 -- 172.21.15.42:0/3225714512 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 81924418 0 0) 0x7f2d281344d0 con 0x7f2d0c006220 2024-09-16T04:01:42.644 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.636+0000 7f2d2c822640 10 monclient(hunting): _negotiate 2024-09-16T04:01:42.644 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.636+0000 7f2d2c822640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:01:42.644 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.636+0000 7f2d2c822640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:01:42.644 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.636+0000 7f2d2c822640 10 monclient(hunting): my global_id is 825378 2024-09-16T04:01:42.644 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.636+0000 7f2d2c822640 1 -- 172.21.15.42:0/3225714512 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2d14003530 con 0x7f2d0c006220 2024-09-16T04:01:42.652 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.644+0000 7f2d2c822640 1 -- 172.21.15.42:0/3225714512 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3954418833 0 0) 0x7f2d14003530 con 0x7f2d0c006220 2024-09-16T04:01:42.652 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.644+0000 7f2d2c822640 1 -- 172.21.15.42:0/3225714512 >> v2:172.21.15.154:3300/0 conn(0x7f2d0c0036b0 msgr2=0x7f2d0c005b30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:01:42.652 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.644+0000 7f2d2c822640 1 --2- 172.21.15.42:0/3225714512 >> v2:172.21.15.154:3300/0 conn(0x7f2d0c0036b0 0x7f2d0c005b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:42.652 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.644+0000 7f2d2c822640 10 monclient: _finish_hunting 0 2024-09-16T04:01:42.652 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.644+0000 7f2d2c822640 1 monclient: found mon.noname-a-legacy 2024-09-16T04:01:42.652 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.644+0000 7f2d2c822640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:01:42.652 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.644+0000 7f2d2c822640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.42:6789/0 2024-09-16T04:01:42.653 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.644+0000 7f2d2c822640 1 -- 172.21.15.42:0/3225714512 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d0c00b030 con 0x7f2d0c006220 2024-09-16T04:01:42.653 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.644+0000 7f2d2c822640 10 monclient: _finish_auth 0 2024-09-16T04:01:42.653 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.644+0000 7f2d2c822640 10 monclient: _check_auth_tickets 2024-09-16T04:01:42.653 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.644+0000 7f2d2c822640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:01:42.653 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.644+0000 7f2d2c822640 1 -- 172.21.15.42:0/3225714512 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2177413734 0 0) 0x7f2d1800f040 con 0x7f2d0c006220 2024-09-16T04:01:42.653 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.644+0000 7f2d2fa88640 5 monclient: authenticate success, global_id 825378 2024-09-16T04:01:42.653 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.644+0000 7f2d2fa88640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:01:42.653 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.644+0000 7f2d2c822640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:01:42.653 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.644+0000 7f2d2c822640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T04:01:42.653 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.644+0000 7f2d2c822640 10 monclient: dump: 2024-09-16T04:01:42.653 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:01:42.653 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:01:42.653 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:42.654 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:42.654 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:01:42.654 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:01:42.654 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:01:42.654 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:01:42.654 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:01:42.654 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:01:42.654 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.644+0000 7f2d2c822640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] but i'm connected to v1:172.21.15.42:6789/0, reconnecting 2024-09-16T04:01:42.654 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.644+0000 7f2d2c822640 10 monclient: _reopen_session rank -1 2024-09-16T04:01:42.654 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.644+0000 7f2d2c822640 1 -- 172.21.15.42:0/3225714512 >> v1:172.21.15.42:6789/0 conn(0x7f2d0c006220 legacy=0x7f2d0c0086e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:01:42.655 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.644+0000 7f2d2e025640 1 -- 172.21.15.42:0/3225714512 reap_dead start 2024-09-16T04:01:42.655 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.644+0000 7f2d2c822640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:01:42.655 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.644+0000 7f2d2c822640 1 --2- 172.21.15.42:0/3225714512 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2d14003a10 0x7f2d14003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:01:42.655 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.644+0000 7f2d2c822640 10 monclient(hunting): picked mon.c con 0x7f2d14003a10 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:01:42.655 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.644+0000 7f2d2c822640 1 --2- 172.21.15.42:0/3225714512 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d14004810 0x7f2d14004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:01:42.656 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.644+0000 7f2d2c822640 10 monclient(hunting): picked mon.b con 0x7f2d14004810 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:01:42.656 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.644+0000 7f2d2c822640 1 --2- 172.21.15.42:0/3225714512 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2d140053d0 0x7f2d1400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:01:42.656 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.644+0000 7f2d2c822640 10 monclient(hunting): picked mon.a con 0x7f2d140053d0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:01:42.656 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.644+0000 7f2d2c822640 10 monclient(hunting): start opening mon connection 2024-09-16T04:01:42.656 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.644+0000 7f2d2c822640 1 -- 172.21.15.42:0/3225714512 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f2d2806df70 con 0x7f2d140053d0 2024-09-16T04:01:42.656 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.644+0000 7f2d2c822640 10 monclient(hunting): start opening mon connection 2024-09-16T04:01:42.656 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.644+0000 7f2d2c822640 1 -- 172.21.15.42:0/3225714512 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f2d1400c290 con 0x7f2d14004810 2024-09-16T04:01:42.656 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.644+0000 7f2d2c822640 10 monclient(hunting): start opening mon connection 2024-09-16T04:01:42.656 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.644+0000 7f2d2c822640 1 -- 172.21.15.42:0/3225714512 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f2d1400c490 con 0x7f2d14003a10 2024-09-16T04:01:42.656 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.644+0000 7f2d2c822640 10 monclient(hunting): _renew_subs 2024-09-16T04:01:42.656 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.644+0000 7f2d2c822640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:01:42.656 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.644+0000 7f2d2fa88640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-16T04:01:42.656 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.644+0000 7f2d2d824640 1 --2- 172.21.15.42:0/3225714512 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2d14003a10 0x7f2d14003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:01:42.657 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.644+0000 7f2d2e025640 1 --2- 172.21.15.42:0/3225714512 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2d140053d0 0x7f2d1400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:01:42.657 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.644+0000 7f2d2d023640 1 --2- 172.21.15.42:0/3225714512 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d14004810 0x7f2d14004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:01:42.657 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.644+0000 7f2d2e025640 10 monclient(hunting): get_auth_request con 0x7f2d140053d0 auth_method 0 2024-09-16T04:01:42.657 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.644+0000 7f2d2e025640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:01:42.657 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.644+0000 7f2d2e025640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:01:42.657 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.644+0000 7f2d2e025640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:01:42.658 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.644+0000 7f2d2d824640 10 monclient(hunting): get_auth_request con 0x7f2d14003a10 auth_method 0 2024-09-16T04:01:42.658 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.644+0000 7f2d2d824640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:01:42.658 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.644+0000 7f2d2d824640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:01:42.658 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.644+0000 7f2d2d824640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:01:42.658 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.644+0000 7f2d2d023640 10 monclient(hunting): get_auth_request con 0x7f2d14004810 auth_method 0 2024-09-16T04:01:42.658 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.644+0000 7f2d2d023640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:01:42.658 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.644+0000 7f2d2d023640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:01:42.660 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.644+0000 7f2d2d023640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:01:42.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.644+0000 7f2d2e025640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:01:42.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.644+0000 7f2d2e025640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:01:42.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.644+0000 7f2d2e025640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:01:42.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.644+0000 7f2d2d023640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:01:42.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.644+0000 7f2d2d023640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:01:42.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.644+0000 7f2d2d023640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:01:42.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.644+0000 7f2d2d824640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:01:42.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.644+0000 7f2d2d824640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:01:42.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.644+0000 7f2d2d824640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:01:42.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.644+0000 7f2d2e025640 10 monclient(hunting): handle_auth_done global_id 825378 payload 293 2024-09-16T04:01:42.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.644+0000 7f2d2e025640 1 -- 172.21.15.42:0/3225714512 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2d14003a10 msgr2=0x7f2d14003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:01:42.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.644+0000 7f2d2e025640 1 --2- 172.21.15.42:0/3225714512 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2d14003a10 0x7f2d14003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:42.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.644+0000 7f2d2e025640 1 -- 172.21.15.42:0/3225714512 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d14004810 msgr2=0x7f2d14004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:01:42.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.644+0000 7f2d2e025640 1 --2- 172.21.15.42:0/3225714512 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d14004810 0x7f2d14004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:42.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.644+0000 7f2d2e025640 10 monclient: _finish_hunting 0 2024-09-16T04:01:42.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.644+0000 7f2d2e025640 1 monclient: found mon.a 2024-09-16T04:01:42.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.644+0000 7f2d2e025640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:01:42.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.644+0000 7f2d2e025640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:01:42.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.644+0000 7f2d2e025640 1 -- 172.21.15.42:0/3225714512 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d14003530 con 0x7f2d140053d0 2024-09-16T04:01:42.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.644+0000 7f2d2e025640 10 monclient: _finish_auth 0 2024-09-16T04:01:42.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.644+0000 7f2d2e025640 10 monclient: _check_auth_tickets 2024-09-16T04:01:42.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.644+0000 7f2d2e025640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:01:42.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.644+0000 7f2d2d023640 1 --2- 172.21.15.42:0/3225714512 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d14004810 0x7f2d14004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:01:42.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.652+0000 7f2d2e025640 1 --2- 172.21.15.42:0/3225714512 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2d140053d0 0x7f2d1400b9d0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f2d28069180 tx=0x7f2d2001d140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81080542e704dc9e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:01:42.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.652+0000 7f2d2c822640 1 -- 172.21.15.42:0/3225714512 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d200b3070 con 0x7f2d140053d0 2024-09-16T04:01:42.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.652+0000 7f2d2c822640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:01:42.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.652+0000 7f2d2c822640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:01:42.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.652+0000 7f2d2c822640 10 monclient: dump: 2024-09-16T04:01:42.671 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:01:42.671 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:01:42.671 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:42.672 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:42.672 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:01:42.672 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:01:42.672 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:01:42.672 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:01:42.672 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:01:42.672 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:01:42.672 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.652+0000 7f2d2c822640 1 -- 172.21.15.42:0/3225714512 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d200ae410 con 0x7f2d140053d0 2024-09-16T04:01:42.672 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.652+0000 7f2d2c822640 10 monclient: handle_config config(0 keys) 2024-09-16T04:01:42.672 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.652+0000 7f2d2c822640 1 -- 172.21.15.42:0/3225714512 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d20003de0 con 0x7f2d140053d0 2024-09-16T04:01:42.672 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.652+0000 7f2d2fa88640 10 monclient: get_monmap_and_config success 2024-09-16T04:01:42.672 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.652+0000 7f2d2fa88640 4 set_mon_vals no callback set 2024-09-16T04:01:42.672 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.652+0000 7f2d2c822640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:01:42.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.652+0000 7f2d2c822640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:01:42.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.652+0000 7f2d2c822640 10 monclient: dump: 2024-09-16T04:01:42.673 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:01:42.673 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:01:42.673 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:42.673 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:42.673 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:01:42.673 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:01:42.673 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:01:42.673 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:01:42.673 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:01:42.673 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:01:42.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.652+0000 7f2d2fa88640 10 monclient: shutdown 2024-09-16T04:01:42.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.652+0000 7f2d2fa88640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:01:42.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.652+0000 7f2d2fa88640 1 -- 172.21.15.42:0/3225714512 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2d140053d0 msgr2=0x7f2d1400b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:01:42.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.652+0000 7f2d2fa88640 1 --2- 172.21.15.42:0/3225714512 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2d140053d0 0x7f2d1400b9d0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f2d28069180 tx=0x7f2d2001d140 comp rx=0 tx=0).stop 2024-09-16T04:01:42.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.652+0000 7f2d2fa88640 1 -- 172.21.15.42:0/3225714512 shutdown_connections 2024-09-16T04:01:42.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.652+0000 7f2d2fa88640 1 --2- 172.21.15.42:0/3225714512 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2d140053d0 0x7f2d1400b9d0 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:42.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.652+0000 7f2d2fa88640 1 --2- 172.21.15.42:0/3225714512 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d14004810 0x7f2d14004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:42.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.652+0000 7f2d2fa88640 1 --2- 172.21.15.42:0/3225714512 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2d14003a10 0x7f2d14003e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:42.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.652+0000 7f2d2fa88640 1 -- 172.21.15.42:0/3225714512 >> 172.21.15.42:0/3225714512 conn(0x7f2d280a9fb0 msgr2=0x7f2d280aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:01:42.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.652+0000 7f2d2fa88640 1 -- 172.21.15.42:0/3225714512 shutdown_connections 2024-09-16T04:01:42.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.652+0000 7f2d2fa88640 1 -- 172.21.15.42:0/3225714512 wait complete. 2024-09-16T04:01:42.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.656+0000 7f2d2fa88640 10 monclient: build_initial_monmap 2024-09-16T04:01:42.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.656+0000 7f2d2fa88640 10 monclient: monmap: 2024-09-16T04:01:42.675 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:01:42.675 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:01:42.675 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:01:42.675 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:01:42.675 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:01:42.675 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:01:42.675 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:01:42.675 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:01:42.675 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:01:42.675 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:01:42.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.656+0000 7f2d2fa88640 1 librados: starting msgr at 2024-09-16T04:01:42.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.656+0000 7f2d2fa88640 1 librados: starting objecter 2024-09-16T04:01:42.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.656+0000 7f2d2fa88640 1 Processor -- start 2024-09-16T04:01:42.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.656+0000 7f2d2fa88640 1 -- start start 2024-09-16T04:01:42.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.656+0000 7f2d2fa88640 1 librados: setting wanted keys 2024-09-16T04:01:42.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.656+0000 7f2d2fa88640 1 librados: calling monclient init 2024-09-16T04:01:42.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.656+0000 7f2d2fa88640 10 monclient: init 2024-09-16T04:01:42.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.656+0000 7f2d2fa88640 10 monclient: _reopen_session rank -1 2024-09-16T04:01:42.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.656+0000 7f2d2fa88640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:01:42.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.656+0000 7f2d2fa88640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d14003a10 0x7f2d2809cc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:01:42.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.660+0000 7f2d2fa88640 10 monclient(hunting): picked mon.noname-b con 0x7f2d14003a10 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:01:42.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.660+0000 7f2d2fa88640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2d14004810 0x7f2d2809d170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:01:42.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.660+0000 7f2d2fa88640 10 monclient(hunting): picked mon.noname-a con 0x7f2d14004810 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:01:42.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.660+0000 7f2d2fa88640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2d140053d0 0x7f2d281cbbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:01:42.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.660+0000 7f2d2fa88640 10 monclient(hunting): picked mon.noname-c con 0x7f2d140053d0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:01:42.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.660+0000 7f2d2fa88640 10 monclient(hunting): start opening mon connection 2024-09-16T04:01:42.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.660+0000 7f2d2fa88640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f2d280a37f0 con 0x7f2d14004810 2024-09-16T04:01:42.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.660+0000 7f2d2fa88640 10 monclient(hunting): start opening mon connection 2024-09-16T04:01:42.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.660+0000 7f2d2fa88640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f2d2806e9d0 con 0x7f2d14003a10 2024-09-16T04:01:42.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.660+0000 7f2d2fa88640 10 monclient(hunting): start opening mon connection 2024-09-16T04:01:42.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.660+0000 7f2d2fa88640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f2d280a3970 con 0x7f2d140053d0 2024-09-16T04:01:42.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.660+0000 7f2d2fa88640 10 monclient(hunting): _renew_subs 2024-09-16T04:01:42.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.660+0000 7f2d2fa88640 10 monclient(hunting): authenticate will time out at 2149.632239s 2024-09-16T04:01:42.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.660+0000 7f2d2d023640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2d14004810 0x7f2d2809d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:01:42.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.660+0000 7f2d2e025640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2d140053d0 0x7f2d281cbbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:01:42.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.660+0000 7f2d2d023640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2d14004810 0x7f2d2809d170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:40986/0 (socket says 172.21.15.42:40986) 2024-09-16T04:01:42.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.660+0000 7f2d2d824640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d14003a10 0x7f2d2809cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:01:42.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.660+0000 7f2d2d023640 1 -- 172.21.15.42:0/175429037 learned_addr learned my addr 172.21.15.42:0/175429037 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:01:42.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.660+0000 7f2d2d023640 10 monclient(hunting): get_auth_request con 0x7f2d14004810 auth_method 0 2024-09-16T04:01:42.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.660+0000 7f2d2d023640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:01:42.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.660+0000 7f2d2d023640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:01:42.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.660+0000 7f2d2d023640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:01:42.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.660+0000 7f2d2e025640 10 monclient(hunting): get_auth_request con 0x7f2d140053d0 auth_method 0 2024-09-16T04:01:42.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.660+0000 7f2d2e025640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:01:42.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.660+0000 7f2d2e025640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:01:42.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.660+0000 7f2d2e025640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:01:42.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.660+0000 7f2d2d824640 10 monclient(hunting): get_auth_request con 0x7f2d14003a10 auth_method 0 2024-09-16T04:01:42.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.660+0000 7f2d2d824640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:01:42.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.660+0000 7f2d2d824640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:01:42.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.660+0000 7f2d2d824640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:01:42.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.660+0000 7f2d2d023640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:01:42.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.660+0000 7f2d2d023640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:01:42.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.660+0000 7f2d2d023640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:01:42.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.660+0000 7f2d2e025640 1 -- 172.21.15.42:0/175429037 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2d140053d0 msgr2=0x7f2d281cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-16T04:01:42.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.660+0000 7f2d2e025640 1 -- 172.21.15.42:0/175429037 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2d140053d0 msgr2=0x7f2d281cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:01:42.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.660+0000 7f2d2e025640 1 --2- 172.21.15.42:0/175429037 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2d140053d0 0x7f2d281cbbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:01:42.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.660+0000 7f2d2e025640 1 --2- 172.21.15.42:0/175429037 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2d140053d0 0x7f2d281cbbe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:01:42.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.660+0000 7f2d2d824640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:01:42.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.660+0000 7f2d2d824640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:01:42.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.660+0000 7f2d2d824640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:01:42.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.660+0000 7f2d2d023640 10 monclient(hunting): handle_auth_done global_id 825381 payload 824 2024-09-16T04:01:42.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.660+0000 7f2d2d023640 1 -- 172.21.15.42:0/175429037 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2d140053d0 msgr2=0x7f2d281cbbe0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:01:42.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.660+0000 7f2d2d023640 1 --2- 172.21.15.42:0/175429037 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2d140053d0 0x7f2d281cbbe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:42.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.660+0000 7f2d2d023640 1 -- 172.21.15.42:0/175429037 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d14003a10 msgr2=0x7f2d2809cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:01:42.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.660+0000 7f2d2d023640 1 --2- 172.21.15.42:0/175429037 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d14003a10 0x7f2d2809cc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:42.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.660+0000 7f2d2d023640 10 monclient: _finish_hunting 0 2024-09-16T04:01:42.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.660+0000 7f2d2d023640 1 monclient: found mon.noname-a 2024-09-16T04:01:42.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.660+0000 7f2d2d023640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:01:42.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.660+0000 7f2d2d023640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:01:42.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.660+0000 7f2d2d023640 1 -- 172.21.15.42:0/175429037 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d2809f610 con 0x7f2d14004810 2024-09-16T04:01:42.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.660+0000 7f2d2d023640 1 --2- 172.21.15.42:0/175429037 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2d14004810 0x7f2d2809d170 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f2d18004660 tx=0x7f2d180039f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:01:42.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.660+0000 7f2d1e7fc640 10 client.?.objecter ms_handle_connect 0x7f2d14004810 2024-09-16T04:01:42.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.660+0000 7f2d1e7fc640 10 client.?.objecter resend_mon_ops 2024-09-16T04:01:42.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.660+0000 7f2d1e7fc640 1 -- 172.21.15.42:0/175429037 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d1801d070 con 0x7f2d14004810 2024-09-16T04:01:42.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.660+0000 7f2d1e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:01:42.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.660+0000 7f2d1e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:01:42.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.660+0000 7f2d1e7fc640 10 monclient: dump: 2024-09-16T04:01:42.680 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:01:42.680 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:01:42.681 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:42.681 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:42.681 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:01:42.681 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:01:42.681 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:01:42.681 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:01:42.681 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:01:42.681 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:01:42.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.660+0000 7f2d1e7fc640 10 monclient: _finish_auth 0 2024-09-16T04:01:42.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.660+0000 7f2d1e7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:01:42.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.660+0000 7f2d1e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:01:42.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.660+0000 7f2d1e7fc640 1 -- 172.21.15.42:0/175429037 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d18000e50 con 0x7f2d14004810 2024-09-16T04:01:42.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.660+0000 7f2d1e7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:01:42.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.660+0000 7f2d1e7fc640 1 -- 172.21.15.42:0/175429037 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d180199e0 con 0x7f2d14004810 2024-09-16T04:01:42.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.660+0000 7f2d2fa88640 5 monclient: authenticate success, global_id 825381 2024-09-16T04:01:42.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.660+0000 7f2d2fa88640 10 monclient: _renew_subs 2024-09-16T04:01:42.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.660+0000 7f2d2fa88640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:01:42.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.660+0000 7f2d1ffff640 4 set_mon_vals no callback set 2024-09-16T04:01:42.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.660+0000 7f2d2fa88640 1 -- 172.21.15.42:0/175429037 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d2806f6a0 con 0x7f2d14004810 2024-09-16T04:01:42.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.660+0000 7f2d1e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:01:42.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.660+0000 7f2d1e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:01:42.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.660+0000 7f2d1e7fc640 10 monclient: dump: 2024-09-16T04:01:42.682 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:01:42.682 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:01:42.682 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:42.682 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:42.682 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:01:42.682 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:01:42.683 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:01:42.683 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:01:42.683 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:01:42.683 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:01:42.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.660+0000 7f2d2fa88640 10 client.825381.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:01:42.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.660+0000 7f2d2fa88640 10 monclient: _renew_subs 2024-09-16T04:01:42.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.660+0000 7f2d2fa88640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:01:42.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.660+0000 7f2d2fa88640 1 -- 172.21.15.42:0/175429037 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d2809f330 con 0x7f2d14004810 2024-09-16T04:01:42.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.660+0000 7f2d2fa88640 1 librados: init done 2024-09-16T04:01:42.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.660+0000 7f2d1e7fc640 1 -- 172.21.15.42:0/175429037 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f2d18020070 con 0x7f2d14004810 2024-09-16T04:01:42.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.660+0000 7f2d1e7fc640 1 --2- 172.21.15.42:0/175429037 >> v2:172.21.15.154:6812/254845342 conn(0x7f2d0c037ef0 0x7f2d0c03a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:01:42.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.660+0000 7f2d2fa88640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:01:42.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.660+0000 7f2d1e7fc640 1 -- 172.21.15.42:0/175429037 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(594..594 src has 451..594) ==== 6252+0+0 (secure 0 0 0) 0x7f2d1804c900 con 0x7f2d14004810 2024-09-16T04:01:42.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.660+0000 7f2d1e7fc640 10 client.825381.objecter ms_dispatch 0x7f2d28000ba0 osd_map(594..594 src has 451..594) 2024-09-16T04:01:42.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.660+0000 7f2d1e7fc640 3 client.825381.objecter handle_osd_map got epochs [594,594] > 0 2024-09-16T04:01:42.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.660+0000 7f2d1e7fc640 3 client.825381.objecter handle_osd_map decoding full epoch 594 2024-09-16T04:01:42.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.660+0000 7f2d2fa88640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:01:42.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.660+0000 7f2d2fa88640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:01:42.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.660+0000 7f2d2fa88640 1 -- 172.21.15.42:0/175429037 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2d281344d0 con 0x7f2d14004810 2024-09-16T04:01:42.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.660+0000 7f2d1e7fc640 20 client.825381.objecter dump_active .. 0 homeless 2024-09-16T04:01:42.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.660+0000 7f2d2d824640 1 --2- 172.21.15.42:0/175429037 >> v2:172.21.15.154:6812/254845342 conn(0x7f2d0c037ef0 0x7f2d0c03a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:01:42.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.660+0000 7f2d2d824640 10 monclient: get_auth_request con 0x7f2d0c037ef0 auth_method 0 2024-09-16T04:01:42.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.664+0000 7f2d1e7fc640 10 client.825381.objecter ms_handle_connect 0x7f2d0c037ef0 2024-09-16T04:01:42.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.664+0000 7f2d2d824640 1 --2- 172.21.15.42:0/175429037 >> v2:172.21.15.154:6812/254845342 conn(0x7f2d0c037ef0 0x7f2d0c03a3b0 secure :-1 s=READY pgs=3763 cs=0 l=1 rev1=1 crypto rx=0x7f2d2806d6e0 tx=0x7f2d24009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:01:42.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.668+0000 7f2d1e7fc640 1 -- 172.21.15.42:0/175429037 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f2d18014520 con 0x7f2d14004810 2024-09-16T04:01:42.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.668+0000 7f2d1e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:01:42.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.668+0000 7f2d1e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:01:42.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.900+0000 7f2d2fa88640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:01:42.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.900+0000 7f2d2fa88640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:01:42.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.900+0000 7f2d2fa88640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:01:42.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.900+0000 7f2d2fa88640 1 -- 172.21.15.42:0/175429037 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f2d28068e40 con 0x7f2d14004810 2024-09-16T04:01:42.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.900+0000 7f2d1e7fc640 1 -- 172.21.15.42:0/175429037 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1555 (secure 0 0 0) 0x7f2d18014780 con 0x7f2d14004810 2024-09-16T04:01:42.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.900+0000 7f2d1e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:01:42.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.900+0000 7f2d1e7fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:01:42.909 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":176,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:01:42.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.904+0000 7f2d2fa88640 10 librados: watch_flush enter 2024-09-16T04:01:42.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.904+0000 7f2d2fa88640 10 librados: watch_flush exit 2024-09-16T04:01:42.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.904+0000 7f2d2fa88640 20 client.825381.objecter shutdown clearing up homeless session... 2024-09-16T04:01:42.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.904+0000 7f2d2fa88640 10 client.825381.objecter successfully canceled tick 2024-09-16T04:01:42.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.904+0000 7f2d2fa88640 1 -- 172.21.15.42:0/175429037 >> v2:172.21.15.154:6812/254845342 conn(0x7f2d0c037ef0 msgr2=0x7f2d0c03a3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:01:42.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.904+0000 7f2d2fa88640 1 --2- 172.21.15.42:0/175429037 >> v2:172.21.15.154:6812/254845342 conn(0x7f2d0c037ef0 0x7f2d0c03a3b0 secure :-1 s=READY pgs=3763 cs=0 l=1 rev1=1 crypto rx=0x7f2d2806d6e0 tx=0x7f2d24009040 comp rx=0 tx=0).stop 2024-09-16T04:01:42.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.904+0000 7f2d2fa88640 10 monclient: shutdown 2024-09-16T04:01:42.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.904+0000 7f2d2fa88640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:01:42.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.904+0000 7f2d2fa88640 1 -- 172.21.15.42:0/175429037 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2d14004810 msgr2=0x7f2d2809d170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:01:42.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.904+0000 7f2d2fa88640 1 --2- 172.21.15.42:0/175429037 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2d14004810 0x7f2d2809d170 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f2d18004660 tx=0x7f2d180039f0 comp rx=0 tx=0).stop 2024-09-16T04:01:42.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.904+0000 7f2d2fa88640 1 -- 172.21.15.42:0/175429037 shutdown_connections 2024-09-16T04:01:42.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.904+0000 7f2d2fa88640 1 --2- 172.21.15.42:0/175429037 >> v2:172.21.15.154:6812/254845342 conn(0x7f2d0c037ef0 0x7f2d0c03a3b0 unknown :-1 s=CLOSED pgs=3763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:42.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.904+0000 7f2d2fa88640 1 --2- 172.21.15.42:0/175429037 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2d140053d0 0x7f2d281cbbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:42.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.904+0000 7f2d2fa88640 1 --2- 172.21.15.42:0/175429037 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2d14004810 0x7f2d2809d170 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:42.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.904+0000 7f2d2fa88640 1 --2- 172.21.15.42:0/175429037 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d14003a10 0x7f2d2809cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:42.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.904+0000 7f2d2fa88640 1 -- 172.21.15.42:0/175429037 >> 172.21.15.42:0/175429037 conn(0x7f2d280a9fb0 msgr2=0x7f2d28131fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:01:42.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.904+0000 7f2d2fa88640 1 -- 172.21.15.42:0/175429037 shutdown_connections 2024-09-16T04:01:42.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.904+0000 7f2d2fa88640 1 -- 172.21.15.42:0/175429037 wait complete. 2024-09-16T04:01:42.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:42.904+0000 7f2d2fa88640 1 librados: shutdown 2024-09-16T04:01:45.929 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:01:46.063 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.060+0000 7f2bebb7b640 10 monclient: get_monmap_and_config 2024-09-16T04:01:46.063 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.060+0000 7f2bebb7b640 10 monclient: build_initial_monmap 2024-09-16T04:01:46.063 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.060+0000 7f2bebb7b640 10 monclient: monmap: 2024-09-16T04:01:46.063 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:01:46.063 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:01:46.063 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:01:46.061460+0000 2024-09-16T04:01:46.063 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:01:46.061460+0000 2024-09-16T04:01:46.063 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:01:46.064 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:01:46.064 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:01:46.064 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:01:46.064 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:01:46.064 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:01:46.064 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:01:46.064 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:01:46.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.060+0000 7f2bebb7b640 1 Processor -- start 2024-09-16T04:01:46.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.060+0000 7f2bebb7b640 1 -- start start 2024-09-16T04:01:46.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.060+0000 7f2bebb7b640 10 monclient: init 2024-09-16T04:01:46.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.060+0000 7f2bebb7b640 10 monclient: _reopen_session rank -1 2024-09-16T04:01:46.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.060+0000 7f2bebb7b640 10 monclient: _add_conns ranks=[1,0,2,4,3] 2024-09-16T04:01:46.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.060+0000 7f2bebb7b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2be406db90 addr v1:172.21.15.154:6789/0 2024-09-16T04:01:46.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.060+0000 7f2bebb7b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2be41340c0 addr v1:172.21.15.42:6789/0 2024-09-16T04:01:46.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.060+0000 7f2bebb7b640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2be413d8b0 0x7f2be413fca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:01:46.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.060+0000 7f2bebb7b640 10 monclient(hunting): picked mon.noname-b con 0x7f2be413d8b0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:01:46.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.060+0000 7f2bebb7b640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2be406ece0 con 0x7f2be41340c0 2024-09-16T04:01:46.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.060+0000 7f2bebb7b640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2be409bf80 con 0x7f2be406db90 2024-09-16T04:01:46.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.060+0000 7f2bebb7b640 10 monclient(hunting): start opening mon connection 2024-09-16T04:01:46.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.060+0000 7f2bebb7b640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f2be406df70 con 0x7f2be413d8b0 2024-09-16T04:01:46.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.060+0000 7f2bebb7b640 10 monclient(hunting): _renew_subs 2024-09-16T04:01:46.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.060+0000 7f2bebb7b640 10 monclient(hunting): authenticate will time out at 2153.031257s 2024-09-16T04:01:46.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.060+0000 7f2bea118640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2be413d8b0 0x7f2be413fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:01:46.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.060+0000 7f2bea118640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2be413d8b0 0x7f2be413fca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:49404/0 (socket says 172.21.15.42:49404) 2024-09-16T04:01:46.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.060+0000 7f2bea118640 1 -- 172.21.15.42:0/2243025311 learned_addr learned my addr 172.21.15.42:0/2243025311 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:01:46.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.060+0000 7f2bea118640 10 monclient(hunting): get_auth_request con 0x7f2be413d8b0 auth_method 0 2024-09-16T04:01:46.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.060+0000 7f2bea118640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:01:46.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.060+0000 7f2bea118640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:01:46.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.060+0000 7f2bea118640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:01:46.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.060+0000 7f2bea118640 1 -- 172.21.15.42:0/2243025311 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2be413d8b0 msgr2=0x7f2be413fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:01:46.077 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.060+0000 7f2bea118640 1 -- 172.21.15.42:0/2243025311 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2be413d8b0 msgr2=0x7f2be413fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:01:46.077 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.060+0000 7f2bea118640 1 --2- 172.21.15.42:0/2243025311 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2be413d8b0 0x7f2be413fca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:01:46.077 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.060+0000 7f2bea118640 1 --2- 172.21.15.42:0/2243025311 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2be413d8b0 0x7f2be413fca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:01:46.077 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.060+0000 7f2be8915640 1 -- 172.21.15.42:0/2243025311 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2796281413 0 0) 0x7f2be406ece0 con 0x7f2be41340c0 2024-09-16T04:01:46.077 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.060+0000 7f2be8915640 10 monclient(hunting): _negotiate 2024-09-16T04:01:46.077 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.060+0000 7f2be8915640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:01:46.077 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.060+0000 7f2be8915640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:01:46.077 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.060+0000 7f2be8915640 10 monclient(hunting): my global_id is 825384 2024-09-16T04:01:46.077 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.060+0000 7f2be8915640 1 -- 172.21.15.42:0/2243025311 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2bd0003530 con 0x7f2be41340c0 2024-09-16T04:01:46.077 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.060+0000 7f2be8915640 1 -- 172.21.15.42:0/2243025311 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2477477270 0 0) 0x7f2bd4001120 con 0x7f2be406db90 2024-09-16T04:01:46.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.060+0000 7f2be8915640 10 monclient(hunting): _negotiate 2024-09-16T04:01:46.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.060+0000 7f2be8915640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:01:46.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.060+0000 7f2be8915640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:01:46.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.060+0000 7f2be8915640 10 monclient(hunting): my global_id is 904495 2024-09-16T04:01:46.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.060+0000 7f2be8915640 1 -- 172.21.15.42:0/2243025311 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2be406ece0 con 0x7f2be406db90 2024-09-16T04:01:46.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.060+0000 7f2be8915640 1 -- 172.21.15.42:0/2243025311 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4269621774 0 0) 0x7f2bd0003530 con 0x7f2be41340c0 2024-09-16T04:01:46.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.060+0000 7f2be8915640 1 -- 172.21.15.42:0/2243025311 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2be413d8b0 msgr2=0x7f2be413fca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:01:46.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.060+0000 7f2be8915640 1 --2- 172.21.15.42:0/2243025311 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2be413d8b0 0x7f2be413fca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:46.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.060+0000 7f2be8915640 1 -- 172.21.15.42:0/2243025311 >> v1:172.21.15.154:6789/0 conn(0x7f2be406db90 legacy=0x7f2be40a37f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:01:46.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.060+0000 7f2be8915640 10 monclient: _finish_hunting 0 2024-09-16T04:01:46.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.060+0000 7f2be8915640 1 monclient: found mon.noname-a-legacy 2024-09-16T04:01:46.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.060+0000 7f2be8915640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:01:46.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.060+0000 7f2be8915640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.42:6789/0 2024-09-16T04:01:46.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.060+0000 7f2be8915640 1 -- 172.21.15.42:0/2243025311 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2be41344d0 con 0x7f2be41340c0 2024-09-16T04:01:46.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.060+0000 7f2be8915640 10 monclient: _finish_auth 0 2024-09-16T04:01:46.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.060+0000 7f2be8915640 10 monclient: _check_auth_tickets 2024-09-16T04:01:46.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.060+0000 7f2be8915640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:01:46.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.060+0000 7f2bebb7b640 5 monclient: authenticate success, global_id 825384 2024-09-16T04:01:46.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.060+0000 7f2bebb7b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:01:46.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.060+0000 7f2be8915640 1 -- 172.21.15.42:0/2243025311 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2177413734 0 0) 0x7f2be0001a00 con 0x7f2be41340c0 2024-09-16T04:01:46.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.060+0000 7f2be8915640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:01:46.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.060+0000 7f2be8915640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T04:01:46.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.060+0000 7f2be8915640 10 monclient: dump: 2024-09-16T04:01:46.080 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:01:46.080 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:01:46.080 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:46.080 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:46.080 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:01:46.080 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:01:46.080 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:01:46.080 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:01:46.080 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:01:46.080 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:01:46.080 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.060+0000 7f2be8915640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] but i'm connected to v1:172.21.15.42:6789/0, reconnecting 2024-09-16T04:01:46.081 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.060+0000 7f2be8915640 10 monclient: _reopen_session rank -1 2024-09-16T04:01:46.081 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.060+0000 7f2be8915640 1 -- 172.21.15.42:0/2243025311 >> v1:172.21.15.42:6789/0 conn(0x7f2be41340c0 legacy=0x7f2be413c190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:01:46.081 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.060+0000 7f2be8915640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:01:46.081 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.060+0000 7f2be8915640 1 --2- 172.21.15.42:0/2243025311 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2bd0003b80 0x7f2bd0003f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:01:46.081 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.060+0000 7f2be8915640 10 monclient(hunting): picked mon.b con 0x7f2bd0003b80 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:01:46.081 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.060+0000 7f2be8915640 1 --2- 172.21.15.42:0/2243025311 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2bd0004810 0x7f2bd0004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:01:46.081 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.060+0000 7f2be8915640 10 monclient(hunting): picked mon.a con 0x7f2bd0004810 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:01:46.081 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.060+0000 7f2be8915640 1 --2- 172.21.15.42:0/2243025311 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2be413d8b0 0x7f2bd000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:01:46.081 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.060+0000 7f2be8915640 10 monclient(hunting): picked mon.c con 0x7f2be413d8b0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:01:46.081 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.060+0000 7f2be8915640 10 monclient(hunting): start opening mon connection 2024-09-16T04:01:46.081 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.060+0000 7f2be8915640 1 -- 172.21.15.42:0/2243025311 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f2be406df70 con 0x7f2bd0004810 2024-09-16T04:01:46.081 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.060+0000 7f2be8915640 10 monclient(hunting): start opening mon connection 2024-09-16T04:01:46.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.060+0000 7f2be8915640 1 -- 172.21.15.42:0/2243025311 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f2bd0005590 con 0x7f2bd0003b80 2024-09-16T04:01:46.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.060+0000 7f2be8915640 10 monclient(hunting): start opening mon connection 2024-09-16T04:01:46.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.060+0000 7f2be8915640 1 -- 172.21.15.42:0/2243025311 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f2bd000bf10 con 0x7f2be413d8b0 2024-09-16T04:01:46.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.060+0000 7f2be8915640 10 monclient(hunting): _renew_subs 2024-09-16T04:01:46.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.060+0000 7f2be8915640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:01:46.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.060+0000 7f2bebb7b640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-16T04:01:46.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.060+0000 7f2bea118640 1 --2- 172.21.15.42:0/2243025311 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2be413d8b0 0x7f2bd000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:01:46.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.060+0000 7f2be9116640 1 --2- 172.21.15.42:0/2243025311 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2bd0004810 0x7f2bd0004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:01:46.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.060+0000 7f2bea118640 10 monclient(hunting): get_auth_request con 0x7f2be413d8b0 auth_method 0 2024-09-16T04:01:46.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.060+0000 7f2bea118640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:01:46.083 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.060+0000 7f2bea118640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:01:46.083 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.060+0000 7f2bea118640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:01:46.083 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.060+0000 7f2be9116640 10 monclient(hunting): get_auth_request con 0x7f2bd0004810 auth_method 0 2024-09-16T04:01:46.083 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.060+0000 7f2be9116640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:01:46.083 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.060+0000 7f2be9116640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:01:46.083 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.060+0000 7f2be9116640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:01:46.083 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.060+0000 7f2be9116640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:01:46.083 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.060+0000 7f2be9116640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:01:46.083 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.060+0000 7f2be9917640 1 --2- 172.21.15.42:0/2243025311 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2bd0003b80 0x7f2bd0003f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:01:46.083 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.060+0000 7f2be9116640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:01:46.083 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.060+0000 7f2bea118640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:01:46.083 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.060+0000 7f2bea118640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:01:46.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.060+0000 7f2bea118640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:01:46.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.060+0000 7f2be9917640 10 monclient(hunting): get_auth_request con 0x7f2bd0003b80 auth_method 0 2024-09-16T04:01:46.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.060+0000 7f2be9917640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:01:46.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.060+0000 7f2be9917640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:01:46.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.060+0000 7f2be9917640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:01:46.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.060+0000 7f2be9116640 10 monclient(hunting): handle_auth_done global_id 825384 payload 293 2024-09-16T04:01:46.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.060+0000 7f2be9116640 1 -- 172.21.15.42:0/2243025311 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2be413d8b0 msgr2=0x7f2bd000b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:01:46.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.060+0000 7f2be9116640 1 --2- 172.21.15.42:0/2243025311 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2be413d8b0 0x7f2bd000b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:46.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.060+0000 7f2be9116640 1 -- 172.21.15.42:0/2243025311 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2bd0003b80 msgr2=0x7f2bd0003f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:01:46.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.060+0000 7f2be9116640 1 --2- 172.21.15.42:0/2243025311 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2bd0003b80 0x7f2bd0003f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:46.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.060+0000 7f2be9116640 10 monclient: _finish_hunting 0 2024-09-16T04:01:46.085 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.060+0000 7f2be9116640 1 monclient: found mon.a 2024-09-16T04:01:46.085 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.060+0000 7f2be9116640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:01:46.085 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.060+0000 7f2be9116640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:01:46.085 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.060+0000 7f2be9116640 1 -- 172.21.15.42:0/2243025311 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2bd0003530 con 0x7f2bd0004810 2024-09-16T04:01:46.085 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.060+0000 7f2be9116640 10 monclient: _finish_auth 0 2024-09-16T04:01:46.085 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.060+0000 7f2be9116640 10 monclient: _check_auth_tickets 2024-09-16T04:01:46.085 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.060+0000 7f2be9116640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:01:46.085 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.060+0000 7f2be9917640 1 --2- 172.21.15.42:0/2243025311 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2bd0003b80 0x7f2bd0003f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T04:01:46.085 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.060+0000 7f2be9116640 1 --2- 172.21.15.42:0/2243025311 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2bd0004810 0x7f2bd0004c90 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f2be000c4f0 tx=0x7f2be001cf00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d18ca0140893e937 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:01:46.085 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.060+0000 7f2be8915640 1 -- 172.21.15.42:0/2243025311 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2be000c0e0 con 0x7f2bd0004810 2024-09-16T04:01:46.085 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.060+0000 7f2be8915640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:01:46.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.060+0000 7f2be8915640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:01:46.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.060+0000 7f2be8915640 10 monclient: dump: 2024-09-16T04:01:46.086 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:01:46.086 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:01:46.086 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:46.086 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:46.086 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:01:46.086 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:01:46.086 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:01:46.086 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:01:46.086 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:01:46.086 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:01:46.087 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.060+0000 7f2be8915640 1 -- 172.21.15.42:0/2243025311 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2be0006d80 con 0x7f2bd0004810 2024-09-16T04:01:46.087 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.060+0000 7f2be8915640 10 monclient: handle_config config(0 keys) 2024-09-16T04:01:46.087 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.060+0000 7f2be8915640 1 -- 172.21.15.42:0/2243025311 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2be00a5d90 con 0x7f2bd0004810 2024-09-16T04:01:46.087 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.060+0000 7f2bebb7b640 10 monclient: get_monmap_and_config success 2024-09-16T04:01:46.087 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.060+0000 7f2bebb7b640 4 set_mon_vals no callback set 2024-09-16T04:01:46.087 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.060+0000 7f2be8915640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:01:46.087 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.060+0000 7f2be8915640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:01:46.087 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.060+0000 7f2be8915640 10 monclient: dump: 2024-09-16T04:01:46.087 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:01:46.087 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:01:46.087 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:46.087 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:46.088 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:01:46.088 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:01:46.088 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:01:46.088 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:01:46.088 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:01:46.088 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:01:46.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.060+0000 7f2bebb7b640 10 monclient: shutdown 2024-09-16T04:01:46.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.060+0000 7f2bebb7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:01:46.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.060+0000 7f2bebb7b640 1 -- 172.21.15.42:0/2243025311 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2bd0004810 msgr2=0x7f2bd0004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:01:46.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.060+0000 7f2bebb7b640 1 --2- 172.21.15.42:0/2243025311 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2bd0004810 0x7f2bd0004c90 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f2be000c4f0 tx=0x7f2be001cf00 comp rx=0 tx=0).stop 2024-09-16T04:01:46.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.060+0000 7f2bea118640 1 -- 172.21.15.42:0/2243025311 reap_dead start 2024-09-16T04:01:46.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.064+0000 7f2bebb7b640 1 -- 172.21.15.42:0/2243025311 shutdown_connections 2024-09-16T04:01:46.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.064+0000 7f2bebb7b640 1 -- 172.21.15.42:0/2243025311 >> 172.21.15.42:0/2243025311 conn(0x7f2be40a9fb0 msgr2=0x7f2be40aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:01:46.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.064+0000 7f2bebb7b640 1 -- 172.21.15.42:0/2243025311 shutdown_connections 2024-09-16T04:01:46.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.064+0000 7f2bebb7b640 1 -- 172.21.15.42:0/2243025311 wait complete. 2024-09-16T04:01:46.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.064+0000 7f2bebb7b640 10 monclient: build_initial_monmap 2024-09-16T04:01:46.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.064+0000 7f2bebb7b640 10 monclient: monmap: 2024-09-16T04:01:46.089 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:01:46.089 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:01:46.089 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:01:46.089 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:01:46.090 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:01:46.090 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:01:46.090 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:01:46.090 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:01:46.090 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:01:46.090 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:01:46.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.064+0000 7f2bebb7b640 1 librados: starting msgr at 2024-09-16T04:01:46.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.064+0000 7f2bebb7b640 1 librados: starting objecter 2024-09-16T04:01:46.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.064+0000 7f2bebb7b640 1 Processor -- start 2024-09-16T04:01:46.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.064+0000 7f2bebb7b640 1 -- start start 2024-09-16T04:01:46.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.064+0000 7f2bebb7b640 1 librados: setting wanted keys 2024-09-16T04:01:46.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.064+0000 7f2bebb7b640 1 librados: calling monclient init 2024-09-16T04:01:46.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.064+0000 7f2bebb7b640 10 monclient: init 2024-09-16T04:01:46.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.064+0000 7f2bebb7b640 10 monclient: _reopen_session rank -1 2024-09-16T04:01:46.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.064+0000 7f2bebb7b640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:01:46.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.064+0000 7f2bebb7b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2bd0004810 0x7f2be41cb820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:01:46.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.064+0000 7f2bebb7b640 10 monclient(hunting): picked mon.noname-b con 0x7f2bd0004810 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:01:46.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.064+0000 7f2bebb7b640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2be4137400 0x7f2be413c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:01:46.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.064+0000 7f2bebb7b640 10 monclient(hunting): picked mon.noname-a con 0x7f2be4137400 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:01:46.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.064+0000 7f2bebb7b640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2be413c620 0x7f2be41ced60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:01:46.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.064+0000 7f2bebb7b640 10 monclient(hunting): picked mon.noname-c con 0x7f2be413c620 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:01:46.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.064+0000 7f2bebb7b640 10 monclient(hunting): start opening mon connection 2024-09-16T04:01:46.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.064+0000 7f2bebb7b640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f2be41404b0 con 0x7f2be4137400 2024-09-16T04:01:46.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.064+0000 7f2bebb7b640 10 monclient(hunting): start opening mon connection 2024-09-16T04:01:46.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.064+0000 7f2bebb7b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f2be4140630 con 0x7f2bd0004810 2024-09-16T04:01:46.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.064+0000 7f2bebb7b640 10 monclient(hunting): start opening mon connection 2024-09-16T04:01:46.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.064+0000 7f2bebb7b640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f2be40a9710 con 0x7f2be413c620 2024-09-16T04:01:46.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.064+0000 7f2bebb7b640 10 monclient(hunting): _renew_subs 2024-09-16T04:01:46.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.064+0000 7f2bebb7b640 10 monclient(hunting): authenticate will time out at 2153.037774s 2024-09-16T04:01:46.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.064+0000 7f2bea118640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2be413c620 0x7f2be41ced60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:01:46.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.064+0000 7f2be9116640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2be4137400 0x7f2be413c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:01:46.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.064+0000 7f2bea118640 1 -- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2be413c620 msgr2=0x7f2be41ced60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-16T04:01:46.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.064+0000 7f2bea118640 1 -- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2be413c620 msgr2=0x7f2be41ced60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:01:46.093 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.064+0000 7f2bea118640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2be413c620 0x7f2be41ced60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:01:46.093 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.064+0000 7f2bea118640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2be413c620 0x7f2be41ced60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:01:46.093 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.064+0000 7f2be9116640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2be4137400 0x7f2be413c0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:35240/0 (socket says 172.21.15.42:35240) 2024-09-16T04:01:46.093 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.064+0000 7f2be9116640 1 -- 172.21.15.42:0/1117834727 learned_addr learned my addr 172.21.15.42:0/1117834727 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:01:46.093 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.064+0000 7f2be9116640 10 monclient(hunting): get_auth_request con 0x7f2be4137400 auth_method 0 2024-09-16T04:01:46.093 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.064+0000 7f2be9116640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:01:46.093 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.064+0000 7f2be9116640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:01:46.093 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.064+0000 7f2be9116640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:01:46.093 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.064+0000 7f2be9116640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:01:46.093 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.064+0000 7f2be9116640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:01:46.093 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.064+0000 7f2be9116640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:01:46.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.064+0000 7f2be9917640 1 --2- 172.21.15.42:0/1117834727 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2bd0004810 0x7f2be41cb820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:01:46.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.064+0000 7f2be9917640 10 monclient(hunting): get_auth_request con 0x7f2bd0004810 auth_method 0 2024-09-16T04:01:46.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.064+0000 7f2be9917640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:01:46.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.064+0000 7f2be9917640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:01:46.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.064+0000 7f2be9917640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:01:46.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.064+0000 7f2be9116640 10 monclient(hunting): handle_auth_done global_id 825387 payload 824 2024-09-16T04:01:46.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.064+0000 7f2be9116640 1 -- 172.21.15.42:0/1117834727 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2be413c620 msgr2=0x7f2be41ced60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:01:46.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.064+0000 7f2be9116640 1 --2- 172.21.15.42:0/1117834727 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2be413c620 0x7f2be41ced60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:46.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.064+0000 7f2be9116640 1 -- 172.21.15.42:0/1117834727 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2bd0004810 msgr2=0x7f2be41cb820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:01:46.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.064+0000 7f2be9116640 1 --2- 172.21.15.42:0/1117834727 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2bd0004810 0x7f2be41cb820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:46.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.064+0000 7f2be9116640 10 monclient: _finish_hunting 0 2024-09-16T04:01:46.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.064+0000 7f2be9116640 1 monclient: found mon.noname-a 2024-09-16T04:01:46.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.064+0000 7f2be9116640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:01:46.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.064+0000 7f2be9116640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:01:46.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.064+0000 7f2be9116640 1 -- 172.21.15.42:0/1117834727 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2be4137b90 con 0x7f2be4137400 2024-09-16T04:01:46.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.064+0000 7f2be9116640 1 --2- 172.21.15.42:0/1117834727 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2be4137400 0x7f2be413c0e0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f2be00af040 tx=0x7f2bd0005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:01:46.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.064+0000 7f2bda7fc640 10 client.?.objecter ms_handle_connect 0x7f2be4137400 2024-09-16T04:01:46.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.064+0000 7f2bda7fc640 10 client.?.objecter resend_mon_ops 2024-09-16T04:01:46.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.064+0000 7f2bda7fc640 1 -- 172.21.15.42:0/1117834727 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2be00ba020 con 0x7f2be4137400 2024-09-16T04:01:46.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.064+0000 7f2bda7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:01:46.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.064+0000 7f2bda7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:01:46.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.064+0000 7f2bda7fc640 10 monclient: dump: 2024-09-16T04:01:46.095 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:01:46.096 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:01:46.096 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:46.096 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:46.096 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:01:46.096 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:01:46.096 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:01:46.096 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:01:46.096 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:01:46.096 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:01:46.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.064+0000 7f2bda7fc640 10 monclient: _finish_auth 0 2024-09-16T04:01:46.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.064+0000 7f2bda7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:01:46.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.064+0000 7f2bda7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:01:46.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.064+0000 7f2bebb7b640 5 monclient: authenticate success, global_id 825387 2024-09-16T04:01:46.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.064+0000 7f2bda7fc640 1 -- 172.21.15.42:0/1117834727 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2be0003980 con 0x7f2be4137400 2024-09-16T04:01:46.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.064+0000 7f2bda7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:01:46.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.064+0000 7f2bebb7b640 10 monclient: _renew_subs 2024-09-16T04:01:46.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.064+0000 7f2bda7fc640 1 -- 172.21.15.42:0/1117834727 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2be0003d70 con 0x7f2be4137400 2024-09-16T04:01:46.100 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.064+0000 7f2bebb7b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:01:46.100 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.064+0000 7f2bebb7b640 1 -- 172.21.15.42:0/1117834727 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2be41388b0 con 0x7f2be4137400 2024-09-16T04:01:46.100 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.064+0000 7f2bda7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:01:46.101 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.064+0000 7f2bda7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:01:46.101 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.064+0000 7f2bdbfff640 4 set_mon_vals no callback set 2024-09-16T04:01:46.101 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.064+0000 7f2bda7fc640 10 monclient: dump: 2024-09-16T04:01:46.101 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:01:46.101 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:01:46.101 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:46.101 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:46.101 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:01:46.101 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:01:46.101 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:01:46.101 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:01:46.101 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:01:46.102 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:01:46.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.064+0000 7f2bebb7b640 10 client.825387.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:01:46.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.064+0000 7f2bebb7b640 10 monclient: _renew_subs 2024-09-16T04:01:46.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.064+0000 7f2bebb7b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:01:46.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.064+0000 7f2bebb7b640 1 -- 172.21.15.42:0/1117834727 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2be41cf470 con 0x7f2be4137400 2024-09-16T04:01:46.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.064+0000 7f2bebb7b640 1 librados: init done 2024-09-16T04:01:46.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.064+0000 7f2bebb7b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:01:46.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.064+0000 7f2bebb7b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:01:46.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.064+0000 7f2bebb7b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:01:46.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.064+0000 7f2bebb7b640 1 -- 172.21.15.42:0/1117834727 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2be406f680 con 0x7f2be4137400 2024-09-16T04:01:46.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.068+0000 7f2bda7fc640 1 -- 172.21.15.42:0/1117834727 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f2be00bd020 con 0x7f2be4137400 2024-09-16T04:01:46.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.068+0000 7f2bda7fc640 1 --2- 172.21.15.42:0/1117834727 >> v2:172.21.15.154:6812/254845342 conn(0x7f2bc8037810 0x7f2bc8039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:01:46.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.068+0000 7f2bda7fc640 1 -- 172.21.15.42:0/1117834727 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(597..597 src has 451..597) ==== 5871+0+0 (secure 0 0 0) 0x7f2be00e3830 con 0x7f2be4137400 2024-09-16T04:01:46.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.068+0000 7f2bda7fc640 10 client.825387.objecter ms_dispatch 0x7f2be4000ba0 osd_map(597..597 src has 451..597) 2024-09-16T04:01:46.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.068+0000 7f2bda7fc640 3 client.825387.objecter handle_osd_map got epochs [597,597] > 0 2024-09-16T04:01:46.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.068+0000 7f2bda7fc640 3 client.825387.objecter handle_osd_map decoding full epoch 597 2024-09-16T04:01:46.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.068+0000 7f2bda7fc640 20 client.825387.objecter dump_active .. 0 homeless 2024-09-16T04:01:46.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.068+0000 7f2be9917640 1 --2- 172.21.15.42:0/1117834727 >> v2:172.21.15.154:6812/254845342 conn(0x7f2bc8037810 0x7f2bc8039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:01:46.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.068+0000 7f2be9917640 10 monclient: get_auth_request con 0x7f2bc8037810 auth_method 0 2024-09-16T04:01:46.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.068+0000 7f2bda7fc640 10 client.825387.objecter ms_handle_connect 0x7f2bc8037810 2024-09-16T04:01:46.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.068+0000 7f2be9917640 1 --2- 172.21.15.42:0/1117834727 >> v2:172.21.15.154:6812/254845342 conn(0x7f2bc8037810 0x7f2bc8039cd0 secure :-1 s=READY pgs=3764 cs=0 l=1 rev1=1 crypto rx=0x7f2bd4000f70 tx=0x7f2bd4009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:01:46.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.072+0000 7f2bda7fc640 1 -- 172.21.15.42:0/1117834727 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f2be00c4030 con 0x7f2be4137400 2024-09-16T04:01:46.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.072+0000 7f2bda7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:01:46.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.072+0000 7f2bda7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:01:46.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.296+0000 7f2bebb7b640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:01:46.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.296+0000 7f2bebb7b640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:01:46.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.296+0000 7f2bebb7b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:01:46.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.296+0000 7f2bebb7b640 1 -- 172.21.15.42:0/1117834727 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f2be40a37f0 con 0x7f2be4137400 2024-09-16T04:01:46.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.300+0000 7f2bda7fc640 1 -- 172.21.15.42:0/1117834727 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7f2be0016c10 con 0x7f2be4137400 2024-09-16T04:01:46.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.300+0000 7f2bda7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:01:46.306 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.300+0000 7f2bda7fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:01:46.306 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":178,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:01:46.308 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.300+0000 7f2bebb7b640 10 librados: watch_flush enter 2024-09-16T04:01:46.308 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.300+0000 7f2bebb7b640 10 librados: watch_flush exit 2024-09-16T04:01:46.308 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.300+0000 7f2bebb7b640 20 client.825387.objecter shutdown clearing up homeless session... 2024-09-16T04:01:46.309 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.300+0000 7f2bebb7b640 10 client.825387.objecter successfully canceled tick 2024-09-16T04:01:46.309 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.300+0000 7f2bebb7b640 1 -- 172.21.15.42:0/1117834727 >> v2:172.21.15.154:6812/254845342 conn(0x7f2bc8037810 msgr2=0x7f2bc8039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:01:46.309 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.300+0000 7f2bebb7b640 1 --2- 172.21.15.42:0/1117834727 >> v2:172.21.15.154:6812/254845342 conn(0x7f2bc8037810 0x7f2bc8039cd0 secure :-1 s=READY pgs=3764 cs=0 l=1 rev1=1 crypto rx=0x7f2bd4000f70 tx=0x7f2bd4009040 comp rx=0 tx=0).stop 2024-09-16T04:01:46.309 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.300+0000 7f2bebb7b640 10 monclient: shutdown 2024-09-16T04:01:46.309 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.300+0000 7f2bebb7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:01:46.309 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.300+0000 7f2bebb7b640 1 -- 172.21.15.42:0/1117834727 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2be4137400 msgr2=0x7f2be413c0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:01:46.309 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.300+0000 7f2bebb7b640 1 --2- 172.21.15.42:0/1117834727 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2be4137400 0x7f2be413c0e0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f2be00af040 tx=0x7f2bd0005840 comp rx=0 tx=0).stop 2024-09-16T04:01:46.309 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.300+0000 7f2bebb7b640 1 -- 172.21.15.42:0/1117834727 shutdown_connections 2024-09-16T04:01:46.309 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.300+0000 7f2bebb7b640 1 --2- 172.21.15.42:0/1117834727 >> v2:172.21.15.154:6812/254845342 conn(0x7f2bc8037810 0x7f2bc8039cd0 unknown :-1 s=CLOSED pgs=3764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:46.309 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.300+0000 7f2bebb7b640 1 --2- 172.21.15.42:0/1117834727 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2be413c620 0x7f2be41ced60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:46.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.300+0000 7f2bebb7b640 1 --2- 172.21.15.42:0/1117834727 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2be4137400 0x7f2be413c0e0 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:46.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.300+0000 7f2bebb7b640 1 --2- 172.21.15.42:0/1117834727 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2bd0004810 0x7f2be41cb820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:46.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.300+0000 7f2bebb7b640 1 -- 172.21.15.42:0/1117834727 >> 172.21.15.42:0/1117834727 conn(0x7f2be40a9fb0 msgr2=0x7f2be413cc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:01:46.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.300+0000 7f2bebb7b640 1 -- 172.21.15.42:0/1117834727 shutdown_connections 2024-09-16T04:01:46.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.300+0000 7f2bebb7b640 1 -- 172.21.15.42:0/1117834727 wait complete. 2024-09-16T04:01:46.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.300+0000 7f2bebb7b640 1 librados: shutdown 2024-09-16T04:01:46.325 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T04:01:46.326 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T04:01:46.326 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T04:01:46.445 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.444+0000 7f45a8834640 10 monclient: get_monmap_and_config 2024-09-16T04:01:46.446 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.444+0000 7f45a8834640 10 monclient: build_initial_monmap 2024-09-16T04:01:46.446 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.444+0000 7f45a8834640 10 monclient: monmap: 2024-09-16T04:01:46.446 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:01:46.446 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:01:46.446 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:01:46.444395+0000 2024-09-16T04:01:46.446 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:01:46.444395+0000 2024-09-16T04:01:46.446 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:01:46.446 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:01:46.446 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:01:46.446 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:01:46.447 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:01:46.447 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:01:46.447 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:01:46.447 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:01:46.449 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.444+0000 7f45a8834640 1 Processor -- start 2024-09-16T04:01:46.449 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.444+0000 7f45a8834640 1 -- start start 2024-09-16T04:01:46.449 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.444+0000 7f45a8834640 10 monclient: init 2024-09-16T04:01:46.449 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.444+0000 7f45a8834640 10 monclient: _reopen_session rank -1 2024-09-16T04:01:46.449 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.444+0000 7f45a8834640 10 monclient: _add_conns ranks=[4,3,0,2,1] 2024-09-16T04:01:46.450 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.444+0000 7f45a8834640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f45a406db90 0x7f45a41340c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:01:46.450 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.444+0000 7f45a8834640 10 monclient(hunting): picked mon.noname-c con 0x7f45a406db90 addr v2:172.21.15.154:3300/0 2024-09-16T04:01:46.450 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.444+0000 7f45a8834640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f45a40a37f0 0x7f45a40a3c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:01:46.450 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.444+0000 7f45a8834640 10 monclient(hunting): picked mon.noname-a con 0x7f45a40a37f0 addr v2:172.21.15.42:3300/0 2024-09-16T04:01:46.450 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.444+0000 7f45a8834640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f45a4136e10 addr v1:172.21.15.42:6789/0 2024-09-16T04:01:46.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.444+0000 7f45a8834640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f45a406ece0 con 0x7f45a4136e10 2024-09-16T04:01:46.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.444+0000 7f45a8834640 10 monclient(hunting): start opening mon connection 2024-09-16T04:01:46.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.444+0000 7f45a8834640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f45a406df70 con 0x7f45a40a37f0 2024-09-16T04:01:46.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.444+0000 7f45a8834640 10 monclient(hunting): start opening mon connection 2024-09-16T04:01:46.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.444+0000 7f45a8834640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f45a409bf80 con 0x7f45a406db90 2024-09-16T04:01:46.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.444+0000 7f45a8834640 10 monclient(hunting): _renew_subs 2024-09-16T04:01:46.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.444+0000 7f45a8834640 10 monclient(hunting): authenticate will time out at 2153.413789s 2024-09-16T04:01:46.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.444+0000 7f45a1d9b640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f45a40a37f0 0x7f45a40a3c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:01:46.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.444+0000 7f45a2d9d640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f45a4136e10 0x7f45a4137220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:58934/0 (socket says 172.21.15.42:58934) 2024-09-16T04:01:46.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.444+0000 7f45a1d9b640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f45a40a37f0 0x7f45a40a3c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:35252/0 (socket says 172.21.15.42:35252) 2024-09-16T04:01:46.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.444+0000 7f45a2d9d640 1 -- 172.21.15.42:0/3024599887 learned_addr learned my addr 172.21.15.42:0/3024599887 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T04:01:46.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.444+0000 7f45a1d9b640 10 monclient(hunting): get_auth_request con 0x7f45a40a37f0 auth_method 0 2024-09-16T04:01:46.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.444+0000 7f45a1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:01:46.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.444+0000 7f45a1d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:01:46.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.444+0000 7f45a1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:01:46.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.444+0000 7f45a1d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:01:46.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.444+0000 7f45a1d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:01:46.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.444+0000 7f45a1d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:01:46.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.444+0000 7f45a259c640 1 --2- 172.21.15.42:0/3024599887 >> v2:172.21.15.154:3300/0 conn(0x7f45a406db90 0x7f45a41340c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:01:46.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.444+0000 7f45a259c640 10 monclient(hunting): get_auth_request con 0x7f45a406db90 auth_method 0 2024-09-16T04:01:46.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.444+0000 7f45a259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:01:46.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.444+0000 7f45a259c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:01:46.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.444+0000 7f45a259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:01:46.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.444+0000 7f45a1d9b640 10 monclient(hunting): handle_auth_done global_id 825390 payload 274 2024-09-16T04:01:46.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.444+0000 7f45a1d9b640 1 -- 172.21.15.42:0/3024599887 >> v2:172.21.15.154:3300/0 conn(0x7f45a406db90 msgr2=0x7f45a41340c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:01:46.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.444+0000 7f45a1d9b640 1 --2- 172.21.15.42:0/3024599887 >> v2:172.21.15.154:3300/0 conn(0x7f45a406db90 0x7f45a41340c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:46.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.444+0000 7f45a1d9b640 1 -- 172.21.15.42:0/3024599887 >> v1:172.21.15.42:6789/0 conn(0x7f45a4136e10 legacy=0x7f45a4137220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:01:46.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.444+0000 7f45a1d9b640 10 monclient: _finish_hunting 0 2024-09-16T04:01:46.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.444+0000 7f45a1d9b640 1 monclient: found mon.noname-a 2024-09-16T04:01:46.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.444+0000 7f45a159a640 1 -- 172.21.15.42:0/3024599887 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1444018846 0 0) 0x7f45a406ece0 con 0x7f45a4136e10 2024-09-16T04:01:46.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.444+0000 7f45a1d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:01:46.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.444+0000 7f45a1d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:01:46.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.444+0000 7f45a1d9b640 1 -- 172.21.15.42:0/3024599887 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45a4134780 con 0x7f45a40a37f0 2024-09-16T04:01:46.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.444+0000 7f45a159a640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T04:01:46.460 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.444+0000 7f45a1d9b640 1 --2- 172.21.15.42:0/3024599887 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f45a40a37f0 0x7f45a40a3c00 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f45a40a76f0 tx=0x7f458c01cda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb7b3399bd078604 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:01:46.460 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.444+0000 7f45a159a640 1 -- 172.21.15.42:0/3024599887 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f458c0b0070 con 0x7f45a40a37f0 2024-09-16T04:01:46.460 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.444+0000 7f45a159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:01:46.460 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.444+0000 7f45a159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:01:46.460 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.444+0000 7f45a159a640 10 monclient: dump: 2024-09-16T04:01:46.460 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:01:46.460 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:01:46.461 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:46.461 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:46.461 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:01:46.461 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:01:46.461 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:01:46.461 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:01:46.461 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:01:46.461 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:01:46.461 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.444+0000 7f45a159a640 10 monclient: _finish_auth 0 2024-09-16T04:01:46.461 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.444+0000 7f45a159a640 10 monclient: _check_auth_tickets 2024-09-16T04:01:46.461 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.444+0000 7f45a159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:01:46.461 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.444+0000 7f45a159a640 1 -- 172.21.15.42:0/3024599887 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f458c0ab600 con 0x7f45a40a37f0 2024-09-16T04:01:46.461 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.444+0000 7f45a159a640 10 monclient: handle_config config(0 keys) 2024-09-16T04:01:46.462 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.444+0000 7f45a8834640 5 monclient: authenticate success, global_id 825390 2024-09-16T04:01:46.462 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.444+0000 7f45a159a640 1 -- 172.21.15.42:0/3024599887 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f458c0ab9d0 con 0x7f45a40a37f0 2024-09-16T04:01:46.462 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.444+0000 7f45a159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:01:46.462 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.444+0000 7f45a159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:01:46.462 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.444+0000 7f45a159a640 10 monclient: dump: 2024-09-16T04:01:46.462 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:01:46.462 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:01:46.462 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:46.462 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:46.462 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:01:46.462 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:01:46.462 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:01:46.462 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:01:46.462 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:01:46.463 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:01:46.463 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.444+0000 7f45a8834640 10 monclient: get_monmap_and_config success 2024-09-16T04:01:46.463 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.444+0000 7f45a8834640 4 set_mon_vals no callback set 2024-09-16T04:01:46.463 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.444+0000 7f45a8834640 10 monclient: shutdown 2024-09-16T04:01:46.463 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.444+0000 7f45a8834640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:01:46.463 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.444+0000 7f45a8834640 1 -- 172.21.15.42:0/3024599887 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f45a40a37f0 msgr2=0x7f45a40a3c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:01:46.463 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.444+0000 7f45a8834640 1 --2- 172.21.15.42:0/3024599887 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f45a40a37f0 0x7f45a40a3c00 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f45a40a76f0 tx=0x7f458c01cda0 comp rx=0 tx=0).stop 2024-09-16T04:01:46.463 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.444+0000 7f45a8834640 1 -- 172.21.15.42:0/3024599887 shutdown_connections 2024-09-16T04:01:46.463 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.444+0000 7f45a8834640 1 --2- 172.21.15.42:0/3024599887 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f45a40a37f0 0x7f45a40a3c00 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:46.463 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.444+0000 7f45a8834640 1 --2- 172.21.15.42:0/3024599887 >> v2:172.21.15.154:3300/0 conn(0x7f45a406db90 0x7f45a41340c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:46.463 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.444+0000 7f45a8834640 1 -- 172.21.15.42:0/3024599887 >> 172.21.15.42:0/3024599887 conn(0x7f45a40a9fb0 msgr2=0x7f45a40aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:01:46.463 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.444+0000 7f45a8834640 1 -- 172.21.15.42:0/3024599887 shutdown_connections 2024-09-16T04:01:46.463 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.444+0000 7f45a8834640 1 -- 172.21.15.42:0/3024599887 wait complete. 2024-09-16T04:01:46.463 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.444+0000 7f45a8834640 10 monclient: build_initial_monmap 2024-09-16T04:01:46.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.444+0000 7f45a8834640 10 monclient: monmap: 2024-09-16T04:01:46.464 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:01:46.464 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:01:46.464 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:01:46.464 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:01:46.464 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:01:46.464 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:01:46.464 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:01:46.464 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:01:46.464 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:01:46.464 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:01:46.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.444+0000 7f45a8834640 1 librados: starting msgr at 2024-09-16T04:01:46.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.444+0000 7f45a8834640 1 librados: starting objecter 2024-09-16T04:01:46.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.444+0000 7f45a8834640 1 Processor -- start 2024-09-16T04:01:46.465 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.444+0000 7f45a8834640 1 -- start start 2024-09-16T04:01:46.465 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.444+0000 7f45a8834640 1 librados: setting wanted keys 2024-09-16T04:01:46.465 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.444+0000 7f45a8834640 1 librados: calling monclient init 2024-09-16T04:01:46.465 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.444+0000 7f45a8834640 10 monclient: init 2024-09-16T04:01:46.465 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.444+0000 7f45a8834640 10 monclient: _reopen_session rank -1 2024-09-16T04:01:46.465 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.444+0000 7f45a8834640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:01:46.465 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.444+0000 7f45a8834640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f45a406db90 0x7f45a40a1610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:01:46.465 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.444+0000 7f45a8834640 10 monclient(hunting): picked mon.noname-c con 0x7f45a406db90 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:01:46.465 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.444+0000 7f45a8834640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f45a40a37f0 0x7f45a40a1b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:01:46.465 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.444+0000 7f45a8834640 10 monclient(hunting): picked mon.noname-a con 0x7f45a40a37f0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:01:46.465 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.444+0000 7f45a8834640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f45a4136e10 0x7f45a40a2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:01:46.465 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.444+0000 7f45a8834640 10 monclient(hunting): picked mon.noname-b con 0x7f45a4136e10 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:01:46.465 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.444+0000 7f45a8834640 10 monclient(hunting): start opening mon connection 2024-09-16T04:01:46.465 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.444+0000 7f45a8834640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f45a4141d60 con 0x7f45a40a37f0 2024-09-16T04:01:46.466 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.444+0000 7f45a8834640 10 monclient(hunting): start opening mon connection 2024-09-16T04:01:46.466 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.444+0000 7f45a8834640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f45a4141be0 con 0x7f45a4136e10 2024-09-16T04:01:46.466 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.444+0000 7f45a8834640 10 monclient(hunting): start opening mon connection 2024-09-16T04:01:46.466 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.444+0000 7f45a8834640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f45a4141ee0 con 0x7f45a406db90 2024-09-16T04:01:46.466 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.444+0000 7f45a8834640 10 monclient(hunting): _renew_subs 2024-09-16T04:01:46.466 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.444+0000 7f45a8834640 10 monclient(hunting): authenticate will time out at 2153.419286s 2024-09-16T04:01:46.466 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.444+0000 7f45a1d9b640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f45a40a37f0 0x7f45a40a1b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:01:46.466 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.444+0000 7f45a259c640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f45a406db90 0x7f45a40a1610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:01:46.466 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.444+0000 7f45a1d9b640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f45a40a37f0 0x7f45a40a1b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:35254/0 (socket says 172.21.15.42:35254) 2024-09-16T04:01:46.466 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.444+0000 7f45a1d9b640 1 -- 172.21.15.42:0/1275743078 learned_addr learned my addr 172.21.15.42:0/1275743078 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:01:46.466 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.444+0000 7f45a1d9b640 10 monclient(hunting): get_auth_request con 0x7f45a40a37f0 auth_method 0 2024-09-16T04:01:46.466 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.444+0000 7f45a1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:01:46.466 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.444+0000 7f45a1d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:01:46.467 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.444+0000 7f45a1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:01:46.467 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.444+0000 7f45a259c640 10 monclient(hunting): get_auth_request con 0x7f45a406db90 auth_method 0 2024-09-16T04:01:46.467 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.444+0000 7f45a259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:01:46.467 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.444+0000 7f45a259c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:01:46.467 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.444+0000 7f45a259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:01:46.467 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.444+0000 7f45a1d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:01:46.467 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.444+0000 7f45a1d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:01:46.467 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.444+0000 7f45a1d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:01:46.467 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.444+0000 7f45a259c640 1 -- 172.21.15.42:0/1275743078 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f45a406db90 msgr2=0x7f45a40a1610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-16T04:01:46.467 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.444+0000 7f45a259c640 1 -- 172.21.15.42:0/1275743078 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f45a406db90 msgr2=0x7f45a40a1610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:01:46.467 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.444+0000 7f45a259c640 1 --2- 172.21.15.42:0/1275743078 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f45a406db90 0x7f45a40a1610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:01:46.467 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.444+0000 7f45a259c640 1 --2- 172.21.15.42:0/1275743078 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f45a406db90 0x7f45a40a1610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:01:46.467 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.444+0000 7f45a2d9d640 1 --2- 172.21.15.42:0/1275743078 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f45a4136e10 0x7f45a40a2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:01:46.467 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.444+0000 7f45a1d9b640 10 monclient(hunting): handle_auth_done global_id 825396 payload 824 2024-09-16T04:01:46.468 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.444+0000 7f45a1d9b640 1 -- 172.21.15.42:0/1275743078 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f45a406db90 msgr2=0x7f45a40a1610 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:01:46.468 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.444+0000 7f45a1d9b640 1 --2- 172.21.15.42:0/1275743078 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f45a406db90 0x7f45a40a1610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:46.468 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.444+0000 7f45a1d9b640 1 -- 172.21.15.42:0/1275743078 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f45a4136e10 msgr2=0x7f45a40a2090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:01:46.468 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.444+0000 7f45a1d9b640 1 --2- 172.21.15.42:0/1275743078 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f45a4136e10 0x7f45a40a2090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:46.468 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.444+0000 7f45a1d9b640 10 monclient: _finish_hunting 0 2024-09-16T04:01:46.468 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.444+0000 7f45a1d9b640 1 monclient: found mon.noname-a 2024-09-16T04:01:46.468 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.444+0000 7f45a1d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:01:46.468 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.444+0000 7f45a1d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:01:46.468 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.444+0000 7f45a1d9b640 1 -- 172.21.15.42:0/1275743078 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45a40a8c30 con 0x7f45a40a37f0 2024-09-16T04:01:46.468 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.444+0000 7f45a2d9d640 10 monclient: get_auth_request con 0x7f45a4136e10 auth_method 0 2024-09-16T04:01:46.468 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.444+0000 7f45a2d9d640 1 --2- 172.21.15.42:0/1275743078 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f45a4136e10 0x7f45a40a2090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T04:01:46.468 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.448+0000 7f45a1d9b640 1 --2- 172.21.15.42:0/1275743078 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f45a40a37f0 0x7f45a40a1b50 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f458c019f70 tx=0x7f458c01ab80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:01:46.468 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.448+0000 7f459affd640 10 client.?.objecter ms_handle_connect 0x7f45a40a37f0 2024-09-16T04:01:46.469 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.448+0000 7f459affd640 10 client.?.objecter resend_mon_ops 2024-09-16T04:01:46.469 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.448+0000 7f459affd640 1 -- 172.21.15.42:0/1275743078 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f458c0b0050 con 0x7f45a40a37f0 2024-09-16T04:01:46.469 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.448+0000 7f459affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:01:46.469 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.448+0000 7f459affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:01:46.469 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.448+0000 7f459affd640 10 monclient: dump: 2024-09-16T04:01:46.469 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:01:46.469 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:01:46.469 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:46.469 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:46.469 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:01:46.469 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:01:46.469 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:01:46.469 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:01:46.469 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:01:46.469 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:01:46.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.448+0000 7f459affd640 10 monclient: _finish_auth 0 2024-09-16T04:01:46.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.448+0000 7f459affd640 10 monclient: _check_auth_tickets 2024-09-16T04:01:46.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.448+0000 7f459affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:01:46.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.448+0000 7f459affd640 1 -- 172.21.15.42:0/1275743078 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f458c0b1040 con 0x7f45a40a37f0 2024-09-16T04:01:46.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.448+0000 7f45a8834640 5 monclient: authenticate success, global_id 825396 2024-09-16T04:01:46.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.448+0000 7f45a8834640 10 monclient: _renew_subs 2024-09-16T04:01:46.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.448+0000 7f45a8834640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:01:46.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.448+0000 7f45a8834640 1 -- 172.21.15.42:0/1275743078 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45a406f6a0 con 0x7f45a40a37f0 2024-09-16T04:01:46.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.448+0000 7f459affd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:01:46.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.448+0000 7f459affd640 1 -- 172.21.15.42:0/1275743078 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f458c004240 con 0x7f45a40a37f0 2024-09-16T04:01:46.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.448+0000 7f459affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:01:46.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.448+0000 7f459affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:01:46.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.448+0000 7f459affd640 10 monclient: dump: 2024-09-16T04:01:46.471 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:01:46.471 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:01:46.471 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:46.471 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:46.471 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:01:46.471 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:01:46.471 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:01:46.471 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:01:46.471 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:01:46.471 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:01:46.471 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.448+0000 7f45a8834640 10 client.825396.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:01:46.471 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.448+0000 7f459bfff640 4 set_mon_vals no callback set 2024-09-16T04:01:46.471 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.448+0000 7f45a8834640 10 monclient: _renew_subs 2024-09-16T04:01:46.471 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.448+0000 7f45a8834640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:01:46.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.448+0000 7f45a8834640 1 -- 172.21.15.42:0/1275743078 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45a4139a70 con 0x7f45a40a37f0 2024-09-16T04:01:46.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.448+0000 7f45a8834640 1 librados: init done 2024-09-16T04:01:46.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.448+0000 7f45a8834640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:01:46.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.448+0000 7f45a8834640 1 --2- 172.21.15.42:0/1275743078 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f45a41385c0 0x7f45a41cf0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:01:46.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.448+0000 7f45a8834640 10 monclient(hunting): start opening mon connection 2024-09-16T04:01:46.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.448+0000 7f45a8834640 1 -- 172.21.15.42:0/1275743078 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f45a40a5200 con 0x7f45a41385c0 2024-09-16T04:01:46.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.448+0000 7f45a259c640 1 --2- 172.21.15.42:0/1275743078 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f45a41385c0 0x7f45a41cf0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:01:46.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.448+0000 7f45a259c640 10 monclient: get_auth_request con 0x7f45a41385c0 auth_method 0 2024-09-16T04:01:46.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.448+0000 7f45a259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:01:46.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.448+0000 7f45a259c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:01:46.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.448+0000 7f45a259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:01:46.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.448+0000 7f45a259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:01:46.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.448+0000 7f45a259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:01:46.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.448+0000 7f45a259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:01:46.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.448+0000 7f459affd640 1 -- 172.21.15.42:0/1275743078 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f458c0be060 con 0x7f45a40a37f0 2024-09-16T04:01:46.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.448+0000 7f45a259c640 10 monclient(hunting): handle_auth_done global_id 825399 payload 824 2024-09-16T04:01:46.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.448+0000 7f45a259c640 1 -- 172.21.15.42:0/1275743078 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f45a41340c0 con 0x7f45a41385c0 2024-09-16T04:01:46.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.448+0000 7f459affd640 1 --2- 172.21.15.42:0/1275743078 >> v2:172.21.15.154:6812/254845342 conn(0x7f4584037810 0x7f4584039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:01:46.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.448+0000 7f459affd640 1 -- 172.21.15.42:0/1275743078 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(597..597 src has 451..597) ==== 5871+0+0 (secure 0 0 0) 0x7f458c0e2930 con 0x7f45a40a37f0 2024-09-16T04:01:46.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.448+0000 7f459affd640 10 client.825396.objecter ms_dispatch 0x7f45a4000ba0 osd_map(597..597 src has 451..597) 2024-09-16T04:01:46.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.448+0000 7f459affd640 3 client.825396.objecter handle_osd_map got epochs [597,597] > 0 2024-09-16T04:01:46.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.448+0000 7f459affd640 3 client.825396.objecter handle_osd_map decoding full epoch 597 2024-09-16T04:01:46.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.448+0000 7f45a259c640 1 --2- 172.21.15.42:0/1275743078 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f45a41385c0 0x7f45a41cf0e0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f459400f3f0 tx=0x7f459400f8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:01:46.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.448+0000 7f459affd640 20 client.825396.objecter dump_active .. 0 homeless 2024-09-16T04:01:46.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.448+0000 7f459affd640 10 client.825396.objecter ms_handle_connect 0x7f45a41385c0 2024-09-16T04:01:46.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.448+0000 7f459affd640 10 client.825396.objecter resend_mon_ops 2024-09-16T04:01:46.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.448+0000 7f459affd640 1 -- 172.21.15.42:0/1275743078 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4594019020 con 0x7f45a41385c0 2024-09-16T04:01:46.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.448+0000 7f459affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:01:46.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.448+0000 7f459affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:01:46.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.448+0000 7f459affd640 10 monclient: dump: 2024-09-16T04:01:46.474 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:01:46.474 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:01:46.474 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:46.474 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:46.474 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:01:46.474 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:01:46.474 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:01:46.474 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:01:46.474 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:01:46.474 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:01:46.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.448+0000 7f45a2d9d640 1 --2- 172.21.15.42:0/1275743078 >> v2:172.21.15.154:6812/254845342 conn(0x7f4584037810 0x7f4584039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:01:46.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.448+0000 7f45a2d9d640 10 monclient: get_auth_request con 0x7f4584037810 auth_method 0 2024-09-16T04:01:46.475 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.448+0000 7f459affd640 1 -- 172.21.15.42:0/1275743078 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f4594014870 con 0x7f45a41385c0 2024-09-16T04:01:46.475 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.448+0000 7f459affd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:01:46.475 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.448+0000 7f459affd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:01:46.475 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.448+0000 7f459affd640 1 -- 172.21.15.42:0/1275743078 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f45a41385c0 msgr2=0x7f45a41cf0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:01:46.475 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.448+0000 7f459affd640 1 --2- 172.21.15.42:0/1275743078 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f45a41385c0 0x7f45a41cf0e0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f459400f3f0 tx=0x7f459400f8b0 comp rx=0 tx=0).stop 2024-09-16T04:01:46.475 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.448+0000 7f459affd640 1 -- 172.21.15.42:0/1275743078 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f45a41385c0 msgr2=0x7f45a41cf0e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:01:46.475 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.448+0000 7f459affd640 1 --2- 172.21.15.42:0/1275743078 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f45a41385c0 0x7f45a41cf0e0 secure :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f459400f3f0 tx=0x7f459400f8b0 comp rx=0 tx=0).stop 2024-09-16T04:01:46.475 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.448+0000 7f459affd640 10 client.825396.objecter ms_handle_connect 0x7f4584037810 2024-09-16T04:01:46.475 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.448+0000 7f45a2d9d640 1 --2- 172.21.15.42:0/1275743078 >> v2:172.21.15.154:6812/254845342 conn(0x7f4584037810 0x7f4584039cd0 secure :-1 s=READY pgs=3765 cs=0 l=1 rev1=1 crypto rx=0x7f45a4139560 tx=0x7f459c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:01:46.477 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.460+0000 7f45a8834640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:01:46.477 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.460+0000 7f45a8834640 1 --2- 172.21.15.42:0/1275743078 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f45a41cf620 0x7f45a41cfa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:01:46.477 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.460+0000 7f45a8834640 10 monclient(hunting): start opening mon connection 2024-09-16T04:01:46.477 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.460+0000 7f45a8834640 1 -- 172.21.15.42:0/1275743078 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f45a41cff70 con 0x7f45a41cf620 2024-09-16T04:01:46.477 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.460+0000 7f45a259c640 1 --2- 172.21.15.42:0/1275743078 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f45a41cf620 0x7f45a41cfa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:01:46.477 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.460+0000 7f45a259c640 10 monclient: get_auth_request con 0x7f45a41cf620 auth_method 0 2024-09-16T04:01:46.477 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.460+0000 7f45a259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:01:46.477 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.460+0000 7f45a259c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:01:46.477 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.460+0000 7f45a259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:01:46.477 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.460+0000 7f45a259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:01:46.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.460+0000 7f45a259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:01:46.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.460+0000 7f45a259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:01:46.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.460+0000 7f45a259c640 10 monclient(hunting): handle_auth_done global_id 825402 payload 824 2024-09-16T04:01:46.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.460+0000 7f45a259c640 1 -- 172.21.15.42:0/1275743078 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f45a41da430 con 0x7f45a41cf620 2024-09-16T04:01:46.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.460+0000 7f45a259c640 1 --2- 172.21.15.42:0/1275743078 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f45a41cf620 0x7f45a41cfa30 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f459401a000 tx=0x7f45940046d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:01:46.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.460+0000 7f459affd640 10 client.825396.objecter ms_handle_connect 0x7f45a41cf620 2024-09-16T04:01:46.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.460+0000 7f459affd640 10 client.825396.objecter resend_mon_ops 2024-09-16T04:01:46.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.460+0000 7f459affd640 1 -- 172.21.15.42:0/1275743078 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4594019040 con 0x7f45a41cf620 2024-09-16T04:01:46.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.460+0000 7f459affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:01:46.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.460+0000 7f459affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:01:46.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.460+0000 7f459affd640 10 monclient: dump: 2024-09-16T04:01:46.478 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:01:46.478 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:01:46.478 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:46.479 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:46.479 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:01:46.479 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:01:46.479 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:01:46.479 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:01:46.479 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:01:46.479 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:01:46.479 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.460+0000 7f459affd640 1 -- 172.21.15.42:0/1275743078 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4715 (secure 0 0 0) 0x7f459401e420 con 0x7f45a41cf620 2024-09-16T04:01:46.479 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.460+0000 7f459affd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:01:46.479 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.460+0000 7f459affd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:01:46.479 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.460+0000 7f459affd640 1 -- 172.21.15.42:0/1275743078 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f45a41cf620 msgr2=0x7f45a41cfa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:01:46.479 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.460+0000 7f459affd640 1 --2- 172.21.15.42:0/1275743078 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f45a41cf620 0x7f45a41cfa30 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f459401a000 tx=0x7f45940046d0 comp rx=0 tx=0).stop 2024-09-16T04:01:46.479 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.460+0000 7f459affd640 1 -- 172.21.15.42:0/1275743078 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f45a41cf620 msgr2=0x7f45a41cfa30 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:01:46.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.460+0000 7f459affd640 1 --2- 172.21.15.42:0/1275743078 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f45a41cf620 0x7f45a41cfa30 secure :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f459401a000 tx=0x7f45940046d0 comp rx=0 tx=0).stop 2024-09-16T04:01:46.480 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T04:01:46.480 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:01:46.480 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:01:46.480 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "leader", 2024-09-16T04:01:46.480 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 178, 2024-09-16T04:01:46.480 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 596314, 2024-09-16T04:01:46.480 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T04:01:46.481 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T04:01:46.481 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T04:01:46.481 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T04:01:46.481 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:01:46.481 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 2, 2024-09-16T04:01:46.481 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:01:46.481 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T04:01:46.481 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T04:01:46.481 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:01:46.481 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:01:46.481 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:01:46.481 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:01:46.481 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:01:46.482 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:01:46.482 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:01:46.482 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:01:46.482 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:01:46.482 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:01:46.482 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:01:46.482 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:01:46.482 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:01:46.482 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T04:01:46.482 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:01:46.482 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:01:46.482 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:01:46.482 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:01:46.482 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:01:46.483 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:01:46.483 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:01:46.483 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:01:46.483 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:01:46.483 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:01:46.483 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:01:46.483 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:01:46.483 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:01:46.483 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T04:01:46.483 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T04:01:46.483 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T04:01:46.483 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T04:01:46.483 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T04:01:46.483 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T04:01:46.484 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:01:46.484 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:01:46.484 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T04:01:46.484 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T04:01:46.484 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T04:01:46.484 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T04:01:46.484 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T04:01:46.484 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T04:01:46.484 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T04:01:46.484 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:01:46.484 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T04:01:46.484 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:01:46.484 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:01:46.484 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:01:46.485 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:01:46.485 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:01:46.485 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:01:46.485 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:01:46.485 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:01:46.485 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:01:46.485 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:01:46.485 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:01:46.485 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:01:46.485 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T04:01:46.485 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:01:46.485 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T04:01:46.485 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:01:46.485 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:01:46.486 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:01:46.486 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:01:46.486 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:01:46.486 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:01:46.486 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:01:46.486 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T04:01:46.486 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:01:46.486 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:01:46.486 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:01:46.486 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:01:46.486 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T04:01:46.486 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:01:46.486 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:01:46.486 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:01:46.487 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:01:46.487 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T04:01:46.487 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T04:01:46.487 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:01:46.487 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:01:46.487 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:01:46.487 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:01:46.487 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:01:46.487 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:01:46.487 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:01:46.487 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:01:46.487 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:01:46.487 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:01:46.488 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:01:46.488 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T04:01:46.488 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:01:46.488 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:01:46.488 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:01:46.488 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:01:46.488 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T04:01:46.488 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:01:46.488 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:01:46.488 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:01:46.488 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:01:46.488 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T04:01:46.488 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T04:01:46.488 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:01:46.489 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:01:46.489 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:01:46.489 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:01:46.489 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:01:46.489 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:01:46.489 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:01:46.489 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:01:46.489 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:01:46.489 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:01:46.489 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:01:46.489 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T04:01:46.489 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:01:46.489 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:01:46.490 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:01:46.490 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:01:46.490 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T04:01:46.490 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:01:46.490 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:01:46.490 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:01:46.490 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:01:46.490 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T04:01:46.490 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T04:01:46.490 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:01:46.490 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:01:46.490 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:01:46.490 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:01:46.490 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:01:46.491 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:01:46.491 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T04:01:46.491 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T04:01:46.491 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:01:46.491 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:01:46.491 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:01:46.491 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:01:46.491 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:01:46.491 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:01:46.491 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T04:01:46.491 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:01:46.491 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:01:46.491 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:01:46.491 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 5 2024-09-16T04:01:46.492 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:01:46.492 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:01:46.492 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T04:01:46.492 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:01:46.492 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:01:46.492 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:01:46.492 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 7 2024-09-16T04:01:46.492 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:01:46.492 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:01:46.492 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:01:46.492 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T04:01:46.492 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T04:01:46.494 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.460+0000 7f45a8834640 10 librados: watch_flush enter 2024-09-16T04:01:46.494 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.460+0000 7f45a8834640 10 librados: watch_flush exit 2024-09-16T04:01:46.494 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.460+0000 7f45a8834640 20 client.825396.objecter shutdown clearing up homeless session... 2024-09-16T04:01:46.494 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.460+0000 7f45a8834640 10 client.825396.objecter successfully canceled tick 2024-09-16T04:01:46.494 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.460+0000 7f45a8834640 1 -- 172.21.15.42:0/1275743078 >> v2:172.21.15.154:6812/254845342 conn(0x7f4584037810 msgr2=0x7f4584039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:01:46.494 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.460+0000 7f45a8834640 1 --2- 172.21.15.42:0/1275743078 >> v2:172.21.15.154:6812/254845342 conn(0x7f4584037810 0x7f4584039cd0 secure :-1 s=READY pgs=3765 cs=0 l=1 rev1=1 crypto rx=0x7f45a4139560 tx=0x7f459c009040 comp rx=0 tx=0).stop 2024-09-16T04:01:46.494 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.460+0000 7f45a8834640 10 monclient: shutdown 2024-09-16T04:01:46.494 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.460+0000 7f45a8834640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:01:46.494 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.460+0000 7f45a8834640 1 -- 172.21.15.42:0/1275743078 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f45a40a37f0 msgr2=0x7f45a40a1b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:01:46.494 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.460+0000 7f45a8834640 1 --2- 172.21.15.42:0/1275743078 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f45a40a37f0 0x7f45a40a1b50 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f458c019f70 tx=0x7f458c01ab80 comp rx=0 tx=0).stop 2024-09-16T04:01:46.495 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.460+0000 7f45a2d9d640 1 -- 172.21.15.42:0/1275743078 reap_dead start 2024-09-16T04:01:46.495 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.460+0000 7f45a2d9d640 1 -- 172.21.15.42:0/1275743078 reap_dead start 2024-09-16T04:01:46.495 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.460+0000 7f45a8834640 1 -- 172.21.15.42:0/1275743078 shutdown_connections 2024-09-16T04:01:46.495 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.460+0000 7f45a8834640 1 -- 172.21.15.42:0/1275743078 >> 172.21.15.42:0/1275743078 conn(0x7f45a40a9fb0 msgr2=0x7f45a4131900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:01:46.495 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.460+0000 7f45a8834640 1 -- 172.21.15.42:0/1275743078 shutdown_connections 2024-09-16T04:01:46.495 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.460+0000 7f45a8834640 1 -- 172.21.15.42:0/1275743078 wait complete. 2024-09-16T04:01:46.495 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.460+0000 7f45a8834640 1 librados: shutdown 2024-09-16T04:01:46.495 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:01:46.495 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T04:01:46.495 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T04:01:46.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.616+0000 7eff71d9f640 10 monclient: get_monmap_and_config 2024-09-16T04:01:46.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.616+0000 7eff71d9f640 10 monclient: build_initial_monmap 2024-09-16T04:01:46.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.616+0000 7eff71d9f640 10 monclient: monmap: 2024-09-16T04:01:46.619 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:01:46.619 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:01:46.619 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:01:46.617051+0000 2024-09-16T04:01:46.619 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:01:46.617051+0000 2024-09-16T04:01:46.619 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:01:46.619 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:01:46.619 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:01:46.619 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:01:46.619 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:01:46.619 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:01:46.619 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:01:46.619 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:01:46.622 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.616+0000 7eff71d9f640 1 Processor -- start 2024-09-16T04:01:46.622 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.616+0000 7eff71d9f640 1 -- start start 2024-09-16T04:01:46.622 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.616+0000 7eff71d9f640 10 monclient: init 2024-09-16T04:01:46.622 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.616+0000 7eff71d9f640 10 monclient: _reopen_session rank -1 2024-09-16T04:01:46.622 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.616+0000 7eff71d9f640 10 monclient: _add_conns ranks=[4,1,2,0,3] 2024-09-16T04:01:46.622 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.616+0000 7eff71d9f640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7eff6c0a39b0 0x7eff6c133b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:01:46.622 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.616+0000 7eff71d9f640 10 monclient(hunting): picked mon.noname-c con 0x7eff6c0a39b0 addr v2:172.21.15.154:3300/0 2024-09-16T04:01:46.622 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.616+0000 7eff71d9f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7eff6c0a3d90 addr v1:172.21.15.154:6789/0 2024-09-16T04:01:46.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.616+0000 7eff71d9f640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7eff6c137710 0x7eff6c13f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:01:46.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.616+0000 7eff71d9f640 10 monclient(hunting): picked mon.noname-b con 0x7eff6c137710 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:01:46.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.616+0000 7eff71d9f640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7eff6c09c140 con 0x7eff6c0a3d90 2024-09-16T04:01:46.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.616+0000 7eff71d9f640 10 monclient(hunting): start opening mon connection 2024-09-16T04:01:46.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.616+0000 7eff71d9f640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7eff6c09c420 con 0x7eff6c0a39b0 2024-09-16T04:01:46.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.616+0000 7eff71d9f640 10 monclient(hunting): start opening mon connection 2024-09-16T04:01:46.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.616+0000 7eff71d9f640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7eff6c06dd60 con 0x7eff6c137710 2024-09-16T04:01:46.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.616+0000 7eff71d9f640 10 monclient(hunting): _renew_subs 2024-09-16T04:01:46.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.616+0000 7eff71d9f640 10 monclient(hunting): authenticate will time out at 2153.587215s 2024-09-16T04:01:46.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.616+0000 7eff6bfff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7eff6c137710 0x7eff6c13f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:01:46.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.616+0000 7eff6bfff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7eff6c137710 0x7eff6c13f3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:49448/0 (socket says 172.21.15.42:49448) 2024-09-16T04:01:46.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.616+0000 7eff6bfff640 1 -- 172.21.15.42:0/3493156804 learned_addr learned my addr 172.21.15.42:0/3493156804 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:01:46.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.616+0000 7eff6bfff640 10 monclient(hunting): get_auth_request con 0x7eff6c137710 auth_method 0 2024-09-16T04:01:46.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.616+0000 7eff6bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:01:46.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.616+0000 7eff6bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:01:46.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.616+0000 7eff6bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:01:46.625 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.616+0000 7eff6b7fe640 1 --2- 172.21.15.42:0/3493156804 >> v2:172.21.15.154:3300/0 conn(0x7eff6c0a39b0 0x7eff6c133b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:01:46.625 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.616+0000 7eff6bfff640 1 -- 172.21.15.42:0/3493156804 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7eff6c137710 msgr2=0x7eff6c13f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T04:01:46.625 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.616+0000 7eff6b7fe640 10 monclient(hunting): get_auth_request con 0x7eff6c0a39b0 auth_method 0 2024-09-16T04:01:46.625 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.616+0000 7eff6bfff640 1 -- 172.21.15.42:0/3493156804 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7eff6c137710 msgr2=0x7eff6c13f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:01:46.625 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.616+0000 7eff6b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:01:46.625 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.616+0000 7eff6b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:01:46.625 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.616+0000 7eff6b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:01:46.625 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.616+0000 7eff6bfff640 1 --2- 172.21.15.42:0/3493156804 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7eff6c137710 0x7eff6c13f3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:01:46.625 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.616+0000 7eff6bfff640 1 --2- 172.21.15.42:0/3493156804 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7eff6c137710 0x7eff6c13f3a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:01:46.625 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.616+0000 7eff6b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:01:46.625 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.616+0000 7eff6b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:01:46.625 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.616+0000 7eff6b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:01:46.625 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.616+0000 7eff6b7fe640 10 monclient(hunting): handle_auth_done global_id 904504 payload 274 2024-09-16T04:01:46.625 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.616+0000 7eff6b7fe640 1 -- 172.21.15.42:0/3493156804 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7eff6c137710 msgr2=0x7eff6c13f3a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:01:46.626 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.616+0000 7eff6b7fe640 1 --2- 172.21.15.42:0/3493156804 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7eff6c137710 0x7eff6c13f3a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:46.626 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.616+0000 7eff6b7fe640 1 -- 172.21.15.42:0/3493156804 >> v1:172.21.15.154:6789/0 conn(0x7eff6c0a3d90 legacy=0x7eff6c137000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:01:46.626 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.616+0000 7eff6b7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:01:46.626 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.616+0000 7eff6b7fe640 1 monclient: found mon.noname-c 2024-09-16T04:01:46.626 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.616+0000 7eff6b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:01:46.626 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.616+0000 7eff6b7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.154:3300/0 2024-09-16T04:01:46.626 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.616+0000 7eff6b7fe640 1 -- 172.21.15.42:0/3493156804 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff6c0a5e20 con 0x7eff6c0a39b0 2024-09-16T04:01:46.631 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.616+0000 7eff6b7fe640 1 --2- 172.21.15.42:0/3493156804 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7eff6c0a39b0 0x7eff6c133b80 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7eff5c009d90 tx=0x7eff5c01c830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2e144943335b28ee server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:01:46.631 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.616+0000 7eff6a7fc640 1 -- 172.21.15.42:0/3493156804 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff5c0a6700 con 0x7eff6c0a39b0 2024-09-16T04:01:46.631 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.616+0000 7eff6a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:01:46.631 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.616+0000 7eff6a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:01:46.631 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.616+0000 7eff6a7fc640 10 monclient: dump: 2024-09-16T04:01:46.631 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:01:46.631 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:01:46.632 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:46.632 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:46.632 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:01:46.632 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:01:46.632 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:01:46.632 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:01:46.632 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:01:46.632 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:01:46.632 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.616+0000 7eff6a7fc640 10 monclient: _finish_auth 0 2024-09-16T04:01:46.632 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.616+0000 7eff6a7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:01:46.632 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.616+0000 7eff6a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:01:46.632 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.616+0000 7eff71d9f640 5 monclient: authenticate success, global_id 904504 2024-09-16T04:01:46.633 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.616+0000 7eff71d9f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:01:46.633 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.616+0000 7eff6a7fc640 1 -- 172.21.15.42:0/3493156804 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff5c0a68a0 con 0x7eff6c0a39b0 2024-09-16T04:01:46.633 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.616+0000 7eff6a7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:01:46.633 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.616+0000 7eff71d9f640 10 monclient: get_monmap_and_config success 2024-09-16T04:01:46.633 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.616+0000 7eff71d9f640 4 set_mon_vals no callback set 2024-09-16T04:01:46.633 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.616+0000 7eff6a7fc640 1 -- 172.21.15.42:0/3493156804 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff5c0a6c70 con 0x7eff6c0a39b0 2024-09-16T04:01:46.633 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.616+0000 7eff6a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:01:46.633 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.616+0000 7eff6a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:01:46.633 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.616+0000 7eff6a7fc640 10 monclient: dump: 2024-09-16T04:01:46.633 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:01:46.633 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:01:46.633 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:46.633 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:46.633 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:01:46.634 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:01:46.634 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:01:46.634 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:01:46.634 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:01:46.634 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:01:46.634 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.616+0000 7eff71d9f640 10 monclient: shutdown 2024-09-16T04:01:46.634 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.616+0000 7eff71d9f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:01:46.634 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.616+0000 7eff71d9f640 1 -- 172.21.15.42:0/3493156804 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7eff6c0a39b0 msgr2=0x7eff6c133b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:01:46.634 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.616+0000 7eff71d9f640 1 --2- 172.21.15.42:0/3493156804 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7eff6c0a39b0 0x7eff6c133b80 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7eff5c009d90 tx=0x7eff5c01c830 comp rx=0 tx=0).stop 2024-09-16T04:01:46.634 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.616+0000 7eff71d9f640 1 -- 172.21.15.42:0/3493156804 shutdown_connections 2024-09-16T04:01:46.634 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.616+0000 7eff71d9f640 1 --2- 172.21.15.42:0/3493156804 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7eff6c137710 0x7eff6c13f3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:46.634 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.616+0000 7eff71d9f640 1 --2- 172.21.15.42:0/3493156804 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7eff6c0a39b0 0x7eff6c133b80 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:46.634 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.616+0000 7eff71d9f640 1 -- 172.21.15.42:0/3493156804 >> 172.21.15.42:0/3493156804 conn(0x7eff6c0a9fb0 msgr2=0x7eff6c0aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:01:46.634 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.616+0000 7eff71d9f640 1 -- 172.21.15.42:0/3493156804 shutdown_connections 2024-09-16T04:01:46.634 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.616+0000 7eff71d9f640 1 -- 172.21.15.42:0/3493156804 wait complete. 2024-09-16T04:01:46.635 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.620+0000 7eff71d9f640 10 monclient: build_initial_monmap 2024-09-16T04:01:46.635 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.620+0000 7eff71d9f640 10 monclient: monmap: 2024-09-16T04:01:46.635 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:01:46.635 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:01:46.635 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:01:46.635 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:01:46.635 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:01:46.635 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:01:46.635 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:01:46.635 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:01:46.635 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:01:46.635 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:01:46.635 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.620+0000 7eff71d9f640 1 librados: starting msgr at 2024-09-16T04:01:46.635 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.620+0000 7eff71d9f640 1 librados: starting objecter 2024-09-16T04:01:46.636 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.620+0000 7eff71d9f640 1 Processor -- start 2024-09-16T04:01:46.636 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.620+0000 7eff71d9f640 1 -- start start 2024-09-16T04:01:46.636 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.620+0000 7eff71d9f640 1 librados: setting wanted keys 2024-09-16T04:01:46.636 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.620+0000 7eff71d9f640 1 librados: calling monclient init 2024-09-16T04:01:46.636 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.620+0000 7eff71d9f640 10 monclient: init 2024-09-16T04:01:46.636 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.620+0000 7eff71d9f640 10 monclient: _reopen_session rank -1 2024-09-16T04:01:46.636 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.620+0000 7eff71d9f640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:01:46.636 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.620+0000 7eff71d9f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7eff6c0a39b0 0x7eff6c1306f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:01:46.636 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.620+0000 7eff71d9f640 10 monclient(hunting): picked mon.noname-b con 0x7eff6c0a39b0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:01:46.636 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.620+0000 7eff71d9f640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7eff6c0a3d90 0x7eff6c13e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:01:46.636 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.620+0000 7eff71d9f640 10 monclient(hunting): picked mon.noname-c con 0x7eff6c0a3d90 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:01:46.636 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.620+0000 7eff71d9f640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7eff6c137710 0x7eff6c13e900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:01:46.636 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.620+0000 7eff71d9f640 10 monclient(hunting): picked mon.noname-a con 0x7eff6c137710 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:01:46.636 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.620+0000 7eff71d9f640 10 monclient(hunting): start opening mon connection 2024-09-16T04:01:46.636 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.620+0000 7eff71d9f640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7eff6c142310 con 0x7eff6c137710 2024-09-16T04:01:46.637 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.620+0000 7eff71d9f640 10 monclient(hunting): start opening mon connection 2024-09-16T04:01:46.637 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.620+0000 7eff71d9f640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7eff6c142490 con 0x7eff6c0a39b0 2024-09-16T04:01:46.637 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.620+0000 7eff71d9f640 10 monclient(hunting): start opening mon connection 2024-09-16T04:01:46.637 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.620+0000 7eff71d9f640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7eff6c142610 con 0x7eff6c0a3d90 2024-09-16T04:01:46.637 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.620+0000 7eff71d9f640 10 monclient(hunting): _renew_subs 2024-09-16T04:01:46.637 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.620+0000 7eff71d9f640 10 monclient(hunting): authenticate will time out at 2153.593072s 2024-09-16T04:01:46.637 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.620+0000 7eff6affd640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7eff6c0a3d90 0x7eff6c13e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:01:46.637 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.620+0000 7eff6bfff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7eff6c137710 0x7eff6c13e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:01:46.637 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.620+0000 7eff6affd640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7eff6c0a3d90 0x7eff6c13e3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:49450/0 (socket says 172.21.15.42:49450) 2024-09-16T04:01:46.637 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.620+0000 7eff6affd640 1 -- 172.21.15.42:0/1831764666 learned_addr learned my addr 172.21.15.42:0/1831764666 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:01:46.637 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.620+0000 7eff6bfff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7eff6c137710 0x7eff6c13e900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:35268/0 (socket says 172.21.15.42:35268) 2024-09-16T04:01:46.637 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.620+0000 7eff6affd640 10 monclient(hunting): get_auth_request con 0x7eff6c0a3d90 auth_method 0 2024-09-16T04:01:46.637 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.620+0000 7eff6affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:01:46.637 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.620+0000 7eff6affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:01:46.638 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.620+0000 7eff6affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:01:46.638 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.620+0000 7eff6bfff640 10 monclient(hunting): get_auth_request con 0x7eff6c137710 auth_method 0 2024-09-16T04:01:46.638 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.620+0000 7eff6bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:01:46.638 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.620+0000 7eff6bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:01:46.638 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.620+0000 7eff6bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:01:46.638 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.620+0000 7eff6affd640 1 -- 172.21.15.42:0/1831764666 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7eff6c0a3d90 msgr2=0x7eff6c13e3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-16T04:01:46.638 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.620+0000 7eff6affd640 1 -- 172.21.15.42:0/1831764666 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7eff6c0a3d90 msgr2=0x7eff6c13e3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:01:46.638 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.620+0000 7eff6affd640 1 --2- 172.21.15.42:0/1831764666 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7eff6c0a3d90 0x7eff6c13e3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:01:46.638 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.620+0000 7eff6bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:01:46.638 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.620+0000 7eff6bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:01:46.638 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.620+0000 7eff6bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:01:46.638 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.620+0000 7eff6affd640 1 --2- 172.21.15.42:0/1831764666 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7eff6c0a3d90 0x7eff6c13e3c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:01:46.638 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.620+0000 7eff6b7fe640 1 --2- 172.21.15.42:0/1831764666 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7eff6c0a39b0 0x7eff6c1306f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:01:46.638 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.620+0000 7eff6b7fe640 10 monclient(hunting): get_auth_request con 0x7eff6c0a39b0 auth_method 0 2024-09-16T04:01:46.639 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.620+0000 7eff6b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:01:46.639 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.620+0000 7eff6b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:01:46.639 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.620+0000 7eff6b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:01:46.639 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.620+0000 7eff6bfff640 10 monclient(hunting): handle_auth_done global_id 825405 payload 824 2024-09-16T04:01:46.639 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.620+0000 7eff6bfff640 1 -- 172.21.15.42:0/1831764666 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7eff6c0a3d90 msgr2=0x7eff6c13e3c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:01:46.639 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.620+0000 7eff6bfff640 1 --2- 172.21.15.42:0/1831764666 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7eff6c0a3d90 0x7eff6c13e3c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:46.639 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.620+0000 7eff6bfff640 1 -- 172.21.15.42:0/1831764666 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7eff6c0a39b0 msgr2=0x7eff6c1306f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:01:46.639 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.620+0000 7eff6bfff640 1 --2- 172.21.15.42:0/1831764666 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7eff6c0a39b0 0x7eff6c1306f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:46.639 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.620+0000 7eff6bfff640 10 monclient: _finish_hunting 0 2024-09-16T04:01:46.639 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.620+0000 7eff6bfff640 1 monclient: found mon.noname-a 2024-09-16T04:01:46.639 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.620+0000 7eff6bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:01:46.639 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.620+0000 7eff6bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:01:46.639 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.620+0000 7eff6bfff640 1 -- 172.21.15.42:0/1831764666 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff6c1312c0 con 0x7eff6c137710 2024-09-16T04:01:46.639 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.620+0000 7eff6bfff640 1 --2- 172.21.15.42:0/1831764666 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7eff6c137710 0x7eff6c13e900 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7eff64009100 tx=0x7eff6400da30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:01:46.639 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.620+0000 7eff53fff640 10 client.?.objecter ms_handle_connect 0x7eff6c137710 2024-09-16T04:01:46.640 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.620+0000 7eff53fff640 10 client.?.objecter resend_mon_ops 2024-09-16T04:01:46.640 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.620+0000 7eff53fff640 1 -- 172.21.15.42:0/1831764666 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff64004030 con 0x7eff6c137710 2024-09-16T04:01:46.640 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.620+0000 7eff53fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:01:46.640 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.620+0000 7eff53fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:01:46.640 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.620+0000 7eff53fff640 10 monclient: dump: 2024-09-16T04:01:46.640 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:01:46.640 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:01:46.640 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:46.640 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:46.640 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:01:46.640 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:01:46.640 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:01:46.640 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:01:46.640 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:01:46.640 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:01:46.641 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.620+0000 7eff53fff640 10 monclient: _finish_auth 0 2024-09-16T04:01:46.641 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.620+0000 7eff53fff640 10 monclient: _check_auth_tickets 2024-09-16T04:01:46.641 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.620+0000 7eff53fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:01:46.641 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.620+0000 7eff53fff640 1 -- 172.21.15.42:0/1831764666 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff640041f0 con 0x7eff6c137710 2024-09-16T04:01:46.641 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.620+0000 7eff53fff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:01:46.641 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.620+0000 7eff53fff640 1 -- 172.21.15.42:0/1831764666 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff64003620 con 0x7eff6c137710 2024-09-16T04:01:46.641 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.620+0000 7eff71d9f640 5 monclient: authenticate success, global_id 825405 2024-09-16T04:01:46.641 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.620+0000 7eff71d9f640 10 monclient: _renew_subs 2024-09-16T04:01:46.641 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.620+0000 7eff71d9f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:01:46.641 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.620+0000 7eff71d9f640 1 -- 172.21.15.42:0/1831764666 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff6c13f180 con 0x7eff6c137710 2024-09-16T04:01:46.641 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.620+0000 7eff69ffb640 4 set_mon_vals no callback set 2024-09-16T04:01:46.641 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.620+0000 7eff53fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:01:46.641 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.620+0000 7eff53fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:01:46.641 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.620+0000 7eff53fff640 10 monclient: dump: 2024-09-16T04:01:46.642 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:01:46.642 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:01:46.642 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:46.642 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:46.642 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:01:46.642 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:01:46.642 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:01:46.642 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:01:46.642 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:01:46.642 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:01:46.642 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.620+0000 7eff71d9f640 10 client.825405.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:01:46.642 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.620+0000 7eff71d9f640 10 monclient: _renew_subs 2024-09-16T04:01:46.642 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.620+0000 7eff71d9f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:01:46.642 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.620+0000 7eff71d9f640 1 -- 172.21.15.42:0/1831764666 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff6c1cfd40 con 0x7eff6c137710 2024-09-16T04:01:46.643 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.620+0000 7eff71d9f640 1 librados: init done 2024-09-16T04:01:46.643 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.620+0000 7eff71d9f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:01:46.643 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.620+0000 7eff71d9f640 1 --2- 172.21.15.42:0/1831764666 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7eff6c138130 0x7eff6c138510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:01:46.643 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.620+0000 7eff71d9f640 10 monclient(hunting): start opening mon connection 2024-09-16T04:01:46.643 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.620+0000 7eff71d9f640 1 -- 172.21.15.42:0/1831764666 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7eff6c1342d0 con 0x7eff6c138130 2024-09-16T04:01:46.643 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.620+0000 7eff6b7fe640 1 --2- 172.21.15.42:0/1831764666 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7eff6c138130 0x7eff6c138510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:01:46.643 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.620+0000 7eff6b7fe640 10 monclient: get_auth_request con 0x7eff6c138130 auth_method 0 2024-09-16T04:01:46.643 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.620+0000 7eff6b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:01:46.643 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.620+0000 7eff6b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:01:46.643 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.620+0000 7eff6b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:01:46.643 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.620+0000 7eff6b7fe640 1 -- 172.21.15.42:0/1831764666 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7eff6c138130 msgr2=0x7eff6c138510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-16T04:01:46.643 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.620+0000 7eff6b7fe640 1 -- 172.21.15.42:0/1831764666 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7eff6c138130 msgr2=0x7eff6c138510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:01:46.643 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.620+0000 7eff6b7fe640 1 --2- 172.21.15.42:0/1831764666 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7eff6c138130 0x7eff6c138510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:01:46.643 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.620+0000 7eff53fff640 1 -- 172.21.15.42:0/1831764666 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7eff640037c0 con 0x7eff6c137710 2024-09-16T04:01:46.644 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.620+0000 7eff6b7fe640 1 --2- 172.21.15.42:0/1831764666 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7eff6c138130 0x7eff6c138510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:01:46.644 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.620+0000 7eff53fff640 1 --2- 172.21.15.42:0/1831764666 >> v2:172.21.15.154:6812/254845342 conn(0x7eff58037810 0x7eff58039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:01:46.644 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.620+0000 7eff53fff640 1 -- 172.21.15.42:0/1831764666 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(597..597 src has 451..597) ==== 5871+0+0 (secure 0 0 0) 0x7eff640496b0 con 0x7eff6c137710 2024-09-16T04:01:46.644 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.620+0000 7eff53fff640 10 client.825405.objecter ms_dispatch 0x7eff6c000ba0 osd_map(597..597 src has 451..597) 2024-09-16T04:01:46.644 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.620+0000 7eff53fff640 3 client.825405.objecter handle_osd_map got epochs [597,597] > 0 2024-09-16T04:01:46.644 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.620+0000 7eff53fff640 3 client.825405.objecter handle_osd_map decoding full epoch 597 2024-09-16T04:01:46.644 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.620+0000 7eff53fff640 20 client.825405.objecter dump_active .. 0 homeless 2024-09-16T04:01:46.644 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.620+0000 7eff6affd640 1 --2- 172.21.15.42:0/1831764666 >> v2:172.21.15.154:6812/254845342 conn(0x7eff58037810 0x7eff58039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:01:46.644 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.620+0000 7eff6affd640 10 monclient: get_auth_request con 0x7eff58037810 auth_method 0 2024-09-16T04:01:46.644 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.624+0000 7eff53fff640 10 client.825405.objecter ms_handle_connect 0x7eff58037810 2024-09-16T04:01:46.644 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.624+0000 7eff6affd640 1 --2- 172.21.15.42:0/1831764666 >> v2:172.21.15.154:6812/254845342 conn(0x7eff58037810 0x7eff58039cd0 secure :-1 s=READY pgs=3766 cs=0 l=1 rev1=1 crypto rx=0x7eff60007d40 tx=0x7eff60009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:01:46.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.824+0000 7eff6b7fe640 1 --2- 172.21.15.42:0/1831764666 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7eff6c138130 0x7eff6c138510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:01:46.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.824+0000 7eff6b7fe640 10 monclient: get_auth_request con 0x7eff6c138130 auth_method 0 2024-09-16T04:01:46.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.824+0000 7eff6b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:01:46.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.824+0000 7eff6b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:01:46.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.824+0000 7eff6b7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:01:46.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.824+0000 7eff6b7fe640 1 -- 172.21.15.42:0/1831764666 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7eff6c138130 msgr2=0x7eff6c138510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-16T04:01:46.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.824+0000 7eff6b7fe640 1 -- 172.21.15.42:0/1831764666 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7eff6c138130 msgr2=0x7eff6c138510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:01:46.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.824+0000 7eff6b7fe640 1 --2- 172.21.15.42:0/1831764666 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7eff6c138130 0x7eff6c138510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:01:46.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:46.824+0000 7eff6b7fe640 1 --2- 172.21.15.42:0/1831764666 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7eff6c138130 0x7eff6c138510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T04:01:47.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:47.228+0000 7eff6b7fe640 1 --2- 172.21.15.42:0/1831764666 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7eff6c138130 0x7eff6c138510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:01:47.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:47.228+0000 7eff6b7fe640 10 monclient: get_auth_request con 0x7eff6c138130 auth_method 0 2024-09-16T04:01:47.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:47.228+0000 7eff6b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:01:47.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:47.228+0000 7eff6b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:01:47.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:47.228+0000 7eff6b7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:01:47.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:47.228+0000 7eff6b7fe640 1 -- 172.21.15.42:0/1831764666 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7eff6c138130 msgr2=0x7eff6c138510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-16T04:01:47.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:47.228+0000 7eff6b7fe640 1 -- 172.21.15.42:0/1831764666 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7eff6c138130 msgr2=0x7eff6c138510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:01:47.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:47.228+0000 7eff6b7fe640 1 --2- 172.21.15.42:0/1831764666 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7eff6c138130 0x7eff6c138510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:01:47.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:47.228+0000 7eff6b7fe640 1 --2- 172.21.15.42:0/1831764666 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7eff6c138130 0x7eff6c138510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T04:01:48.034 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.032+0000 7eff6b7fe640 1 --2- 172.21.15.42:0/1831764666 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7eff6c138130 0x7eff6c138510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:01:48.034 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.032+0000 7eff6b7fe640 10 monclient: get_auth_request con 0x7eff6c138130 auth_method 0 2024-09-16T04:01:48.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.032+0000 7eff6b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:01:48.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.032+0000 7eff6b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:01:48.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.032+0000 7eff6b7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:01:48.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.032+0000 7eff6b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:01:48.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.032+0000 7eff6b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:01:48.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.032+0000 7eff6b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:01:48.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.032+0000 7eff6b7fe640 10 monclient(hunting): handle_auth_done global_id 954101 payload 824 2024-09-16T04:01:48.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.032+0000 7eff6b7fe640 1 -- 172.21.15.42:0/1831764666 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7eff6c133b80 con 0x7eff6c138130 2024-09-16T04:01:48.036 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.032+0000 7eff6b7fe640 1 --2- 172.21.15.42:0/1831764666 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7eff6c138130 0x7eff6c138510 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7eff6c0a56c0 tx=0x7eff5c0195d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:01:48.036 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.032+0000 7eff53fff640 10 client.825405.objecter ms_handle_connect 0x7eff6c138130 2024-09-16T04:01:48.036 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.032+0000 7eff53fff640 10 client.825405.objecter resend_mon_ops 2024-09-16T04:01:48.036 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.032+0000 7eff53fff640 1 -- 172.21.15.42:0/1831764666 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff5c0a6e40 con 0x7eff6c138130 2024-09-16T04:01:48.036 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.032+0000 7eff53fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:01:48.036 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.032+0000 7eff53fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:01:48.036 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.032+0000 7eff53fff640 10 monclient: dump: 2024-09-16T04:01:48.036 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:01:48.037 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:01:48.037 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:48.037 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:48.037 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:01:48.037 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:01:48.037 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:01:48.037 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:01:48.037 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:01:48.037 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:01:48.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.032+0000 7eff53fff640 1 -- 172.21.15.42:0/1831764666 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7eff5c0a6e40 con 0x7eff6c138130 2024-09-16T04:01:48.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.032+0000 7eff53fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:01:48.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.032+0000 7eff53fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:01:48.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.032+0000 7eff53fff640 1 -- 172.21.15.42:0/1831764666 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7eff6c138130 msgr2=0x7eff6c138510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:01:48.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.032+0000 7eff53fff640 1 --2- 172.21.15.42:0/1831764666 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7eff6c138130 0x7eff6c138510 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7eff6c0a56c0 tx=0x7eff5c0195d0 comp rx=0 tx=0).stop 2024-09-16T04:01:48.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.032+0000 7eff53fff640 1 -- 172.21.15.42:0/1831764666 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7eff6c138130 msgr2=0x7eff6c138510 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:01:48.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.032+0000 7eff53fff640 1 --2- 172.21.15.42:0/1831764666 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7eff6c138130 0x7eff6c138510 secure :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7eff6c0a56c0 tx=0x7eff5c0195d0 comp rx=0 tx=0).stop 2024-09-16T04:01:48.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.040+0000 7eff71d9f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:01:48.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.040+0000 7eff71d9f640 1 --2- 172.21.15.42:0/1831764666 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7eff6c138a50 0x7eff6c135230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:01:48.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.040+0000 7eff71d9f640 10 monclient(hunting): start opening mon connection 2024-09-16T04:01:48.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.040+0000 7eff71d9f640 1 -- 172.21.15.42:0/1831764666 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7eff6c13af30 con 0x7eff6c138a50 2024-09-16T04:01:48.045 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.040+0000 7eff6b7fe640 1 --2- 172.21.15.42:0/1831764666 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7eff6c138a50 0x7eff6c135230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:01:48.045 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.040+0000 7eff6b7fe640 10 monclient: get_auth_request con 0x7eff6c138a50 auth_method 0 2024-09-16T04:01:48.045 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.040+0000 7eff6b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:01:48.045 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.040+0000 7eff6b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:01:48.045 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.040+0000 7eff6b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:01:48.045 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.040+0000 7eff6b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:01:48.045 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.040+0000 7eff6b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:01:48.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.040+0000 7eff6b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:01:48.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.040+0000 7eff6b7fe640 10 monclient(hunting): handle_auth_done global_id 954104 payload 824 2024-09-16T04:01:48.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.040+0000 7eff6b7fe640 1 -- 172.21.15.42:0/1831764666 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7eff6c0a5750 con 0x7eff6c138a50 2024-09-16T04:01:48.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.044+0000 7eff6b7fe640 1 --2- 172.21.15.42:0/1831764666 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7eff6c138a50 0x7eff6c135230 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7eff5c01aee0 tx=0x7eff5c0a9000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:01:48.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.044+0000 7eff53fff640 10 client.825405.objecter ms_handle_connect 0x7eff6c138a50 2024-09-16T04:01:48.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.044+0000 7eff53fff640 10 client.825405.objecter resend_mon_ops 2024-09-16T04:01:48.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.044+0000 7eff53fff640 1 -- 172.21.15.42:0/1831764666 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff5c0ae820 con 0x7eff6c138a50 2024-09-16T04:01:48.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.044+0000 7eff53fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:01:48.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.044+0000 7eff53fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:01:48.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.044+0000 7eff53fff640 10 monclient: dump: 2024-09-16T04:01:48.047 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:01:48.047 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:01:48.047 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:48.047 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:48.048 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:01:48.048 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:01:48.048 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:01:48.048 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:01:48.048 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:01:48.048 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:01:48.048 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.044+0000 7eff53fff640 1 -- 172.21.15.42:0/1831764666 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4545 (secure 0 0 0) 0x7eff5c0ae820 con 0x7eff6c138a50 2024-09-16T04:01:48.048 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.044+0000 7eff53fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:01:48.048 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.044+0000 7eff53fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:01:48.048 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.044+0000 7eff53fff640 1 -- 172.21.15.42:0/1831764666 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7eff6c138a50 msgr2=0x7eff6c135230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:01:48.048 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.044+0000 7eff53fff640 1 --2- 172.21.15.42:0/1831764666 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7eff6c138a50 0x7eff6c135230 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7eff5c01aee0 tx=0x7eff5c0a9000 comp rx=0 tx=0).stop 2024-09-16T04:01:48.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.044+0000 7eff53fff640 1 -- 172.21.15.42:0/1831764666 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7eff6c138a50 msgr2=0x7eff6c135230 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:01:48.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.044+0000 7eff53fff640 1 --2- 172.21.15.42:0/1831764666 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7eff6c138a50 0x7eff6c135230 secure :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7eff5c01aee0 tx=0x7eff5c0a9000 comp rx=0 tx=0).stop 2024-09-16T04:01:48.049 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T04:01:48.049 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:01:48.049 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:01:48.049 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T04:01:48.050 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 178, 2024-09-16T04:01:48.050 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 15603, 2024-09-16T04:01:48.050 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T04:01:48.050 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T04:01:48.050 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T04:01:48.050 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T04:01:48.050 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:01:48.050 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 4, 2024-09-16T04:01:48.050 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:01:48.050 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T04:01:48.051 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T04:01:48.051 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:01:48.051 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:01:48.051 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:01:48.051 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:01:48.051 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:01:48.051 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:01:48.051 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:01:48.051 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:01:48.051 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:01:48.051 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:01:48.051 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:01:48.051 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:01:48.052 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:01:48.052 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T04:01:48.052 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:01:48.052 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:01:48.052 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:01:48.052 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:01:48.052 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:01:48.052 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:01:48.052 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:01:48.052 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:01:48.052 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:01:48.052 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:01:48.052 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:01:48.053 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:01:48.053 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:01:48.053 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T04:01:48.053 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T04:01:48.053 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T04:01:48.053 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T04:01:48.053 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T04:01:48.053 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T04:01:48.054 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:01:48.054 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:01:48.054 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T04:01:48.054 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T04:01:48.054 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T04:01:48.054 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T04:01:48.054 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T04:01:48.054 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T04:01:48.054 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T04:01:48.055 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:01:48.055 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T04:01:48.055 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:01:48.055 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:01:48.055 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:01:48.055 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:01:48.055 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:01:48.055 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:01:48.055 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:01:48.055 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:01:48.055 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:01:48.055 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:01:48.056 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:01:48.056 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:01:48.056 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T04:01:48.056 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:01:48.056 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T04:01:48.056 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:01:48.056 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:01:48.056 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:01:48.056 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:01:48.056 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:01:48.056 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:01:48.056 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:01:48.056 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T04:01:48.057 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:01:48.057 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:01:48.057 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:01:48.057 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:01:48.057 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T04:01:48.057 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:01:48.057 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:01:48.057 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:01:48.057 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:01:48.057 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T04:01:48.058 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T04:01:48.058 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:01:48.058 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:01:48.058 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:01:48.058 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:01:48.058 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:01:48.058 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:01:48.058 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:01:48.059 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:01:48.059 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:01:48.059 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:01:48.059 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:01:48.059 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T04:01:48.059 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:01:48.059 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:01:48.059 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:01:48.059 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:01:48.059 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T04:01:48.059 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:01:48.059 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:01:48.059 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:01:48.060 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:01:48.060 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T04:01:48.060 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T04:01:48.060 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:01:48.060 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:01:48.060 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:01:48.060 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:01:48.060 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:01:48.060 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:01:48.061 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:01:48.061 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:01:48.061 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:01:48.061 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:01:48.061 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:01:48.061 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T04:01:48.061 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:01:48.061 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:01:48.061 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:01:48.061 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:01:48.061 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T04:01:48.061 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:01:48.061 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:01:48.062 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:01:48.062 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:01:48.062 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T04:01:48.062 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T04:01:48.062 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:01:48.062 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:01:48.062 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:01:48.062 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:01:48.062 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:01:48.062 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:01:48.062 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T04:01:48.062 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T04:01:48.062 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:01:48.063 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:01:48.063 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:01:48.063 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:01:48.063 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:01:48.063 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:01:48.063 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T04:01:48.063 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:01:48.063 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:01:48.063 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:01:48.063 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:01:48.063 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:01:48.063 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:01:48.064 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:01:48.064 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T04:01:48.064 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T04:01:48.065 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.044+0000 7eff71d9f640 10 librados: watch_flush enter 2024-09-16T04:01:48.065 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.044+0000 7eff71d9f640 10 librados: watch_flush exit 2024-09-16T04:01:48.065 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.044+0000 7eff71d9f640 20 client.825405.objecter shutdown clearing up homeless session... 2024-09-16T04:01:48.065 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.044+0000 7eff71d9f640 10 client.825405.objecter successfully canceled tick 2024-09-16T04:01:48.065 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.044+0000 7eff71d9f640 1 -- 172.21.15.42:0/1831764666 >> v2:172.21.15.154:6812/254845342 conn(0x7eff58037810 msgr2=0x7eff58039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:01:48.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.044+0000 7eff71d9f640 1 --2- 172.21.15.42:0/1831764666 >> v2:172.21.15.154:6812/254845342 conn(0x7eff58037810 0x7eff58039cd0 secure :-1 s=READY pgs=3766 cs=0 l=1 rev1=1 crypto rx=0x7eff60007d40 tx=0x7eff60009040 comp rx=0 tx=0).stop 2024-09-16T04:01:48.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.044+0000 7eff71d9f640 10 monclient: shutdown 2024-09-16T04:01:48.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.044+0000 7eff71d9f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:01:48.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.044+0000 7eff71d9f640 1 -- 172.21.15.42:0/1831764666 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7eff6c137710 msgr2=0x7eff6c13e900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:01:48.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.044+0000 7eff71d9f640 1 --2- 172.21.15.42:0/1831764666 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7eff6c137710 0x7eff6c13e900 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7eff64009100 tx=0x7eff6400da30 comp rx=0 tx=0).stop 2024-09-16T04:01:48.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.044+0000 7eff6bfff640 1 -- 172.21.15.42:0/1831764666 reap_dead start 2024-09-16T04:01:48.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.044+0000 7eff6bfff640 1 -- 172.21.15.42:0/1831764666 reap_dead start 2024-09-16T04:01:48.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.044+0000 7eff71d9f640 1 -- 172.21.15.42:0/1831764666 shutdown_connections 2024-09-16T04:01:48.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.044+0000 7eff71d9f640 1 -- 172.21.15.42:0/1831764666 >> 172.21.15.42:0/1831764666 conn(0x7eff6c0a9fb0 msgr2=0x7eff6c12d440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:01:48.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.044+0000 7eff71d9f640 1 -- 172.21.15.42:0/1831764666 shutdown_connections 2024-09-16T04:01:48.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.044+0000 7eff71d9f640 1 -- 172.21.15.42:0/1831764666 wait complete. 2024-09-16T04:01:48.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.044+0000 7eff71d9f640 1 librados: shutdown 2024-09-16T04:01:48.075 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:01:48.075 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T04:01:48.076 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T04:01:48.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.208+0000 7eff04fa1640 10 monclient: get_monmap_and_config 2024-09-16T04:01:48.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.208+0000 7eff04fa1640 10 monclient: build_initial_monmap 2024-09-16T04:01:48.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.208+0000 7eff04fa1640 10 monclient: monmap: 2024-09-16T04:01:48.211 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:01:48.211 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:01:48.211 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:01:48.209819+0000 2024-09-16T04:01:48.211 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:01:48.209819+0000 2024-09-16T04:01:48.212 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:01:48.212 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:01:48.212 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:01:48.212 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:01:48.212 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:01:48.212 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:01:48.212 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:01:48.212 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:01:48.214 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.208+0000 7eff04fa1640 1 Processor -- start 2024-09-16T04:01:48.214 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.208+0000 7eff04fa1640 1 -- start start 2024-09-16T04:01:48.214 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.208+0000 7eff04fa1640 10 monclient: init 2024-09-16T04:01:48.214 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.208+0000 7eff04fa1640 10 monclient: _reopen_session rank -1 2024-09-16T04:01:48.214 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.208+0000 7eff04fa1640 10 monclient: _add_conns ranks=[1,2,4,0,3] 2024-09-16T04:01:48.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.208+0000 7eff04fa1640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7eff000a39b0 addr v1:172.21.15.154:6789/0 2024-09-16T04:01:48.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.208+0000 7eff04fa1640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7eff000a3d90 0x7eff0013c380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:01:48.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.208+0000 7eff04fa1640 10 monclient(hunting): picked mon.noname-b con 0x7eff000a3d90 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:01:48.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.208+0000 7eff04fa1640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7eff0013c8c0 0x7eff0013ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:01:48.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.208+0000 7eff04fa1640 10 monclient(hunting): picked mon.noname-c con 0x7eff0013c8c0 addr v2:172.21.15.154:3300/0 2024-09-16T04:01:48.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.208+0000 7eff04fa1640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7eff00134290 con 0x7eff000a39b0 2024-09-16T04:01:48.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.208+0000 7eff04fa1640 10 monclient(hunting): start opening mon connection 2024-09-16T04:01:48.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.208+0000 7eff04fa1640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7eff0009c140 con 0x7eff0013c8c0 2024-09-16T04:01:48.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.208+0000 7eff04fa1640 10 monclient(hunting): start opening mon connection 2024-09-16T04:01:48.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.208+0000 7eff04fa1640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7eff0009c420 con 0x7eff000a3d90 2024-09-16T04:01:48.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.208+0000 7eff04fa1640 10 monclient(hunting): _renew_subs 2024-09-16T04:01:48.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.208+0000 7eff04fa1640 10 monclient(hunting): authenticate will time out at 2155.179483s 2024-09-16T04:01:48.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.208+0000 7efefdd9b640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7eff000a3d90 0x7eff0013c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:01:48.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.208+0000 7efefdd9b640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7eff000a3d90 0x7eff0013c380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:49526/0 (socket says 172.21.15.42:49526) 2024-09-16T04:01:48.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.208+0000 7efefdd9b640 1 -- 172.21.15.42:0/1733496674 learned_addr learned my addr 172.21.15.42:0/1733496674 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:01:48.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.208+0000 7efefdd9b640 10 monclient(hunting): get_auth_request con 0x7eff000a3d90 auth_method 0 2024-09-16T04:01:48.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.208+0000 7efefdd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:01:48.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.208+0000 7efefdd9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:01:48.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.208+0000 7efefdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:01:48.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.208+0000 7efefdd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:01:48.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.208+0000 7efefdd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:01:48.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.208+0000 7efefdd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:01:48.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.212+0000 7efefed9d640 1 --2- 172.21.15.42:0/1733496674 >> v2:172.21.15.154:3300/0 conn(0x7eff0013c8c0 0x7eff0013ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:01:48.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.212+0000 7efefed9d640 10 monclient(hunting): get_auth_request con 0x7eff0013c8c0 auth_method 0 2024-09-16T04:01:48.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.212+0000 7efefed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:01:48.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.212+0000 7efefed9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:01:48.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.212+0000 7efefed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:01:48.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.212+0000 7efefdd9b640 10 monclient(hunting): handle_auth_done global_id 954107 payload 274 2024-09-16T04:01:48.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.212+0000 7efefdd9b640 1 -- 172.21.15.42:0/1733496674 >> v2:172.21.15.154:3300/0 conn(0x7eff0013c8c0 msgr2=0x7eff0013ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:01:48.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.212+0000 7efefdd9b640 1 --2- 172.21.15.42:0/1733496674 >> v2:172.21.15.154:3300/0 conn(0x7eff0013c8c0 0x7eff0013ecb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:48.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.212+0000 7efefdd9b640 1 -- 172.21.15.42:0/1733496674 >> v1:172.21.15.154:6789/0 conn(0x7eff000a39b0 legacy=0x7eff00133b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:01:48.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.212+0000 7efefdd9b640 10 monclient: _finish_hunting 0 2024-09-16T04:01:48.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.212+0000 7efefdd9b640 1 monclient: found mon.noname-b 2024-09-16T04:01:48.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.212+0000 7efefdd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:01:48.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.212+0000 7efefdd9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T04:01:48.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.212+0000 7efefdd9b640 1 -- 172.21.15.42:0/1733496674 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff0006dd60 con 0x7eff000a3d90 2024-09-16T04:01:48.223 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.212+0000 7efefdd9b640 1 --2- 172.21.15.42:0/1733496674 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7eff000a3d90 0x7eff0013c380 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7efee8009870 tx=0x7efee801c3e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fb2b2d183292665b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:01:48.223 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.212+0000 7efefd59a640 1 -- 172.21.15.42:0/1733496674 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efee80b0070 con 0x7eff000a3d90 2024-09-16T04:01:48.223 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.212+0000 7efefd59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:01:48.223 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.212+0000 7efefd59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:01:48.223 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.212+0000 7efefd59a640 10 monclient: dump: 2024-09-16T04:01:48.224 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:01:48.224 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:01:48.224 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:48.224 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:48.224 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:01:48.224 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:01:48.224 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:01:48.224 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:01:48.224 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:01:48.224 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:01:48.224 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.212+0000 7efefd59a640 10 monclient: _finish_auth 0 2024-09-16T04:01:48.224 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.212+0000 7efefd59a640 10 monclient: _check_auth_tickets 2024-09-16T04:01:48.224 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.212+0000 7efefd59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:01:48.225 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.212+0000 7efefd59a640 1 -- 172.21.15.42:0/1733496674 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efee80ab600 con 0x7eff000a3d90 2024-09-16T04:01:48.225 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.212+0000 7eff04fa1640 5 monclient: authenticate success, global_id 954107 2024-09-16T04:01:48.225 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.212+0000 7eff04fa1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:01:48.225 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.212+0000 7efefd59a640 10 monclient: handle_config config(0 keys) 2024-09-16T04:01:48.225 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.212+0000 7eff04fa1640 10 monclient: get_monmap_and_config success 2024-09-16T04:01:48.225 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.212+0000 7eff04fa1640 4 set_mon_vals no callback set 2024-09-16T04:01:48.225 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.212+0000 7efefd59a640 1 -- 172.21.15.42:0/1733496674 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efee80ab9d0 con 0x7eff000a3d90 2024-09-16T04:01:48.225 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.212+0000 7efefd59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:01:48.225 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.212+0000 7efefd59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:01:48.225 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.212+0000 7efefd59a640 10 monclient: dump: 2024-09-16T04:01:48.225 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:01:48.225 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:01:48.225 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:48.225 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:48.226 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:01:48.226 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:01:48.226 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:01:48.226 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:01:48.226 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:01:48.226 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:01:48.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.212+0000 7eff04fa1640 10 monclient: shutdown 2024-09-16T04:01:48.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.212+0000 7eff04fa1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:01:48.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.212+0000 7eff04fa1640 1 -- 172.21.15.42:0/1733496674 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7eff000a3d90 msgr2=0x7eff0013c380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:01:48.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.212+0000 7eff04fa1640 1 --2- 172.21.15.42:0/1733496674 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7eff000a3d90 0x7eff0013c380 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7efee8009870 tx=0x7efee801c3e0 comp rx=0 tx=0).stop 2024-09-16T04:01:48.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.212+0000 7eff04fa1640 1 -- 172.21.15.42:0/1733496674 shutdown_connections 2024-09-16T04:01:48.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.212+0000 7eff04fa1640 1 --2- 172.21.15.42:0/1733496674 >> v2:172.21.15.154:3300/0 conn(0x7eff0013c8c0 0x7eff0013ecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:48.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.212+0000 7eff04fa1640 1 --2- 172.21.15.42:0/1733496674 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7eff000a3d90 0x7eff0013c380 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:48.227 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.212+0000 7eff04fa1640 1 -- 172.21.15.42:0/1733496674 >> 172.21.15.42:0/1733496674 conn(0x7eff000a9fb0 msgr2=0x7eff000aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:01:48.227 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.216+0000 7eff04fa1640 1 -- 172.21.15.42:0/1733496674 shutdown_connections 2024-09-16T04:01:48.227 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.216+0000 7eff04fa1640 1 -- 172.21.15.42:0/1733496674 wait complete. 2024-09-16T04:01:48.227 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.216+0000 7eff04fa1640 10 monclient: build_initial_monmap 2024-09-16T04:01:48.227 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.216+0000 7eff04fa1640 10 monclient: monmap: 2024-09-16T04:01:48.227 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:01:48.227 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:01:48.227 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:01:48.227 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:01:48.227 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:01:48.227 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:01:48.227 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:01:48.227 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:01:48.227 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:01:48.228 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:01:48.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.216+0000 7eff04fa1640 1 librados: starting msgr at 2024-09-16T04:01:48.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.216+0000 7eff04fa1640 1 librados: starting objecter 2024-09-16T04:01:48.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.216+0000 7eff04fa1640 1 Processor -- start 2024-09-16T04:01:48.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.216+0000 7eff04fa1640 1 -- start start 2024-09-16T04:01:48.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.216+0000 7eff04fa1640 1 librados: setting wanted keys 2024-09-16T04:01:48.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.216+0000 7eff04fa1640 1 librados: calling monclient init 2024-09-16T04:01:48.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.216+0000 7eff04fa1640 10 monclient: init 2024-09-16T04:01:48.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.216+0000 7eff04fa1640 10 monclient: _reopen_session rank -1 2024-09-16T04:01:48.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.216+0000 7eff04fa1640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:01:48.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.216+0000 7eff04fa1640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7eff000a39b0 0x7eff0012d620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:01:48.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.216+0000 7eff04fa1640 10 monclient(hunting): picked mon.noname-b con 0x7eff000a39b0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:01:48.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.216+0000 7eff04fa1640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7eff000a3d90 0x7eff0012db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:01:48.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.216+0000 7eff04fa1640 10 monclient(hunting): picked mon.noname-a con 0x7eff000a3d90 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:01:48.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.216+0000 7eff04fa1640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7eff0013c8c0 0x7eff001cf5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:01:48.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.216+0000 7eff04fa1640 10 monclient(hunting): picked mon.noname-c con 0x7eff0013c8c0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:01:48.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.216+0000 7eff04fa1640 10 monclient(hunting): start opening mon connection 2024-09-16T04:01:48.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.216+0000 7eff04fa1640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7eff00141da0 con 0x7eff000a3d90 2024-09-16T04:01:48.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.216+0000 7eff04fa1640 10 monclient(hunting): start opening mon connection 2024-09-16T04:01:48.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.216+0000 7eff04fa1640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7eff00141c20 con 0x7eff000a39b0 2024-09-16T04:01:48.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.216+0000 7eff04fa1640 10 monclient(hunting): start opening mon connection 2024-09-16T04:01:48.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.216+0000 7eff04fa1640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7eff00141f20 con 0x7eff0013c8c0 2024-09-16T04:01:48.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.216+0000 7eff04fa1640 10 monclient(hunting): _renew_subs 2024-09-16T04:01:48.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.216+0000 7eff04fa1640 10 monclient(hunting): authenticate will time out at 2155.185497s 2024-09-16T04:01:48.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.216+0000 7efefed9d640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7eff0013c8c0 0x7eff001cf5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:01:48.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.216+0000 7efefdd9b640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7eff000a3d90 0x7eff0012db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:01:48.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.216+0000 7efefed9d640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7eff0013c8c0 0x7eff001cf5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:49542/0 (socket says 172.21.15.42:49542) 2024-09-16T04:01:48.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.216+0000 7efefed9d640 1 -- 172.21.15.42:0/3741971663 learned_addr learned my addr 172.21.15.42:0/3741971663 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:01:48.230 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.216+0000 7efefed9d640 10 monclient(hunting): get_auth_request con 0x7eff0013c8c0 auth_method 0 2024-09-16T04:01:48.230 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.216+0000 7efefed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:01:48.230 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.216+0000 7efefed9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:01:48.230 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.216+0000 7efefed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:01:48.230 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.216+0000 7efefdd9b640 10 monclient(hunting): get_auth_request con 0x7eff000a3d90 auth_method 0 2024-09-16T04:01:48.230 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.216+0000 7efefdd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:01:48.230 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.216+0000 7efefdd9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:01:48.230 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.216+0000 7efefdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:01:48.230 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.216+0000 7efefed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:01:48.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.216+0000 7efefed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:01:48.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.216+0000 7efefed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:01:48.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.216+0000 7efefdd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:01:48.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.216+0000 7efefdd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:01:48.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.216+0000 7efefe59c640 1 --2- 172.21.15.42:0/3741971663 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7eff000a39b0 0x7eff0012d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:01:48.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.216+0000 7efefdd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:01:48.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.216+0000 7efefe59c640 10 monclient(hunting): get_auth_request con 0x7eff000a39b0 auth_method 0 2024-09-16T04:01:48.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.216+0000 7efefe59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:01:48.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.216+0000 7efefe59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:01:48.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.216+0000 7efefe59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:01:48.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.216+0000 7efefed9d640 10 monclient(hunting): handle_auth_done global_id 954110 payload 824 2024-09-16T04:01:48.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.216+0000 7efefed9d640 1 -- 172.21.15.42:0/3741971663 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7eff000a39b0 msgr2=0x7eff0012d620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:01:48.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.216+0000 7efefed9d640 1 --2- 172.21.15.42:0/3741971663 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7eff000a39b0 0x7eff0012d620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:48.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.216+0000 7efefed9d640 1 -- 172.21.15.42:0/3741971663 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7eff000a3d90 msgr2=0x7eff0012db60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:01:48.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.216+0000 7efefed9d640 1 --2- 172.21.15.42:0/3741971663 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7eff000a3d90 0x7eff0012db60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:48.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.216+0000 7efefed9d640 10 monclient: _finish_hunting 0 2024-09-16T04:01:48.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.216+0000 7efefed9d640 1 monclient: found mon.noname-c 2024-09-16T04:01:48.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.216+0000 7efefed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:01:48.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.216+0000 7efefed9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T04:01:48.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.216+0000 7efefed9d640 1 -- 172.21.15.42:0/3741971663 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff0013ecc0 con 0x7eff0013c8c0 2024-09-16T04:01:48.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.216+0000 7efefed9d640 1 --2- 172.21.15.42:0/3741971663 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7eff0013c8c0 0x7eff001cf5e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7efef0007e60 tx=0x7efef000e4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:01:48.233 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.216+0000 7efef6ffd640 10 client.?.objecter ms_handle_connect 0x7eff0013c8c0 2024-09-16T04:01:48.233 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.216+0000 7efef6ffd640 10 client.?.objecter resend_mon_ops 2024-09-16T04:01:48.233 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.216+0000 7efef6ffd640 1 -- 172.21.15.42:0/3741971663 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efef0014620 con 0x7eff0013c8c0 2024-09-16T04:01:48.233 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.216+0000 7efef6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:01:48.233 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.216+0000 7efef6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:01:48.233 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.216+0000 7efef6ffd640 10 monclient: dump: 2024-09-16T04:01:48.233 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:01:48.233 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:01:48.233 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:48.234 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:48.234 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:01:48.234 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:01:48.234 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:01:48.234 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:01:48.234 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:01:48.234 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:01:48.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.216+0000 7efef6ffd640 10 monclient: _finish_auth 0 2024-09-16T04:01:48.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.216+0000 7efef6ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:01:48.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.216+0000 7efef6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:01:48.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.216+0000 7efef6ffd640 1 -- 172.21.15.42:0/3741971663 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efef0019070 con 0x7eff0013c8c0 2024-09-16T04:01:48.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.216+0000 7efef6ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:01:48.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.216+0000 7efef6ffd640 1 -- 172.21.15.42:0/3741971663 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efef0014a00 con 0x7eff0013c8c0 2024-09-16T04:01:48.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.216+0000 7eff04fa1640 5 monclient: authenticate success, global_id 954110 2024-09-16T04:01:48.235 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.216+0000 7eff04fa1640 10 monclient: _renew_subs 2024-09-16T04:01:48.235 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.216+0000 7eff04fa1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:01:48.235 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.216+0000 7eff04fa1640 1 -- 172.21.15.42:0/3741971663 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff0006f680 con 0x7eff0013c8c0 2024-09-16T04:01:48.235 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.216+0000 7efefcd99640 4 set_mon_vals no callback set 2024-09-16T04:01:48.235 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.216+0000 7efef6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:01:48.235 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.216+0000 7efef6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:01:48.235 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.216+0000 7efef6ffd640 10 monclient: dump: 2024-09-16T04:01:48.235 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:01:48.235 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:01:48.235 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:48.235 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:48.235 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:01:48.235 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:01:48.235 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:01:48.236 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:01:48.236 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:01:48.236 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:01:48.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.216+0000 7eff04fa1640 10 client.954110.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:01:48.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.216+0000 7eff04fa1640 10 monclient: _renew_subs 2024-09-16T04:01:48.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.216+0000 7eff04fa1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:01:48.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.216+0000 7eff04fa1640 1 -- 172.21.15.42:0/3741971663 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7eff0012e0a0 con 0x7eff0013c8c0 2024-09-16T04:01:48.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.216+0000 7eff04fa1640 1 librados: init done 2024-09-16T04:01:48.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.216+0000 7eff04fa1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:01:48.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.216+0000 7eff04fa1640 1 --2- 172.21.15.42:0/3741971663 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7eff0013bda0 0x7eff001cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:01:48.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.216+0000 7eff04fa1640 10 monclient(hunting): start opening mon connection 2024-09-16T04:01:48.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.216+0000 7eff04fa1640 1 -- 172.21.15.42:0/3741971663 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7eff00133b80 con 0x7eff0013bda0 2024-09-16T04:01:48.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.216+0000 7efef6ffd640 1 -- 172.21.15.42:0/3741971663 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7efef0014ba0 con 0x7eff0013c8c0 2024-09-16T04:01:48.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.216+0000 7efef6ffd640 1 --2- 172.21.15.42:0/3741971663 >> v2:172.21.15.154:6812/254845342 conn(0x7efee0037810 0x7efee0039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:01:48.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.216+0000 7efef6ffd640 1 -- 172.21.15.42:0/3741971663 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(599..599 src has 451..599) ==== 6257+0+0 (secure 0 0 0) 0x7efef0049980 con 0x7eff0013c8c0 2024-09-16T04:01:48.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.216+0000 7efef6ffd640 10 client.954110.objecter ms_dispatch 0x7eff00000ba0 osd_map(599..599 src has 451..599) 2024-09-16T04:01:48.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.216+0000 7efef6ffd640 3 client.954110.objecter handle_osd_map got epochs [599,599] > 0 2024-09-16T04:01:48.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.216+0000 7efef6ffd640 3 client.954110.objecter handle_osd_map decoding full epoch 599 2024-09-16T04:01:48.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.216+0000 7efefe59c640 1 --2- 172.21.15.42:0/3741971663 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7eff0013bda0 0x7eff001cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:01:48.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.216+0000 7efefe59c640 10 monclient: get_auth_request con 0x7eff0013bda0 auth_method 0 2024-09-16T04:01:48.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.216+0000 7efefe59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:01:48.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.216+0000 7efefe59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:01:48.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.216+0000 7efefe59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:01:48.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.216+0000 7efef6ffd640 20 client.954110.objecter dump_active .. 0 homeless 2024-09-16T04:01:48.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.216+0000 7efefe59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:01:48.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.216+0000 7efefe59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:01:48.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.216+0000 7efefe59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:01:48.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.220+0000 7efefdd9b640 1 --2- 172.21.15.42:0/3741971663 >> v2:172.21.15.154:6812/254845342 conn(0x7efee0037810 0x7efee0039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:01:48.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.220+0000 7efefdd9b640 10 monclient: get_auth_request con 0x7efee0037810 auth_method 0 2024-09-16T04:01:48.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.220+0000 7efefe59c640 10 monclient(hunting): handle_auth_done global_id 904516 payload 824 2024-09-16T04:01:48.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.220+0000 7efefe59c640 1 -- 172.21.15.42:0/3741971663 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7eff000a5750 con 0x7eff0013bda0 2024-09-16T04:01:48.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.220+0000 7efefe59c640 1 --2- 172.21.15.42:0/3741971663 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7eff0013bda0 0x7eff001cfb20 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7efef800d500 tx=0x7efef800d9c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:01:48.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.220+0000 7efef6ffd640 10 client.954110.objecter ms_handle_connect 0x7eff0013bda0 2024-09-16T04:01:48.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.220+0000 7efef6ffd640 10 client.954110.objecter resend_mon_ops 2024-09-16T04:01:48.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.220+0000 7efef6ffd640 10 client.954110.objecter ms_handle_connect 0x7efee0037810 2024-09-16T04:01:48.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.220+0000 7efefdd9b640 1 --2- 172.21.15.42:0/3741971663 >> v2:172.21.15.154:6812/254845342 conn(0x7efee0037810 0x7efee0039cd0 secure :-1 s=READY pgs=3767 cs=0 l=1 rev1=1 crypto rx=0x7efee80ad000 tx=0x7efee8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:01:48.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.220+0000 7efef6ffd640 1 -- 172.21.15.42:0/3741971663 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efef8018020 con 0x7eff0013bda0 2024-09-16T04:01:48.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.220+0000 7efef6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:01:48.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.220+0000 7efef6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:01:48.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.220+0000 7efef6ffd640 10 monclient: dump: 2024-09-16T04:01:48.239 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:01:48.239 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:01:48.239 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:48.239 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:48.239 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:01:48.239 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:01:48.239 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:01:48.239 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:01:48.239 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:01:48.239 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:01:48.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.220+0000 7efef6ffd640 1 -- 172.21.15.42:0/3741971663 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7efef8014620 con 0x7eff0013bda0 2024-09-16T04:01:48.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.220+0000 7efef6ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:01:48.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.220+0000 7efef6ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:01:48.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.220+0000 7efef6ffd640 1 -- 172.21.15.42:0/3741971663 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7eff0013bda0 msgr2=0x7eff001cfb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:01:48.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.220+0000 7efef6ffd640 1 --2- 172.21.15.42:0/3741971663 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7eff0013bda0 0x7eff001cfb20 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7efef800d500 tx=0x7efef800d9c0 comp rx=0 tx=0).stop 2024-09-16T04:01:48.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.220+0000 7efef6ffd640 1 -- 172.21.15.42:0/3741971663 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7eff0013bda0 msgr2=0x7eff001cfb20 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:01:48.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.220+0000 7efef6ffd640 1 --2- 172.21.15.42:0/3741971663 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7eff0013bda0 0x7eff001cfb20 secure :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7efef800d500 tx=0x7efef800d9c0 comp rx=0 tx=0).stop 2024-09-16T04:01:48.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.228+0000 7eff04fa1640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:01:48.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.228+0000 7eff04fa1640 1 --2- 172.21.15.42:0/3741971663 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7eff00139ca0 0x7eff001d0060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:01:48.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.228+0000 7eff04fa1640 10 monclient(hunting): start opening mon connection 2024-09-16T04:01:48.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.228+0000 7eff04fa1640 1 -- 172.21.15.42:0/3741971663 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7eff0013af50 con 0x7eff00139ca0 2024-09-16T04:01:48.242 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.228+0000 7efefe59c640 1 --2- 172.21.15.42:0/3741971663 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7eff00139ca0 0x7eff001d0060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:01:48.242 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.228+0000 7efefe59c640 10 monclient: get_auth_request con 0x7eff00139ca0 auth_method 0 2024-09-16T04:01:48.242 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.228+0000 7efefe59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:01:48.242 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.228+0000 7efefe59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:01:48.242 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.228+0000 7efefe59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:01:48.242 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.228+0000 7efefe59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:01:48.242 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.228+0000 7efefe59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:01:48.242 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.228+0000 7efefe59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:01:48.242 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.228+0000 7efefe59c640 10 monclient(hunting): handle_auth_done global_id 904519 payload 824 2024-09-16T04:01:48.242 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.228+0000 7efefe59c640 1 -- 172.21.15.42:0/3741971663 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7eff00068e40 con 0x7eff00139ca0 2024-09-16T04:01:48.242 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.228+0000 7efefe59c640 1 --2- 172.21.15.42:0/3741971663 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7eff00139ca0 0x7eff001d0060 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7efef800d500 tx=0x7efef8004700 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:01:48.242 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.228+0000 7efef6ffd640 10 client.954110.objecter ms_handle_connect 0x7eff00139ca0 2024-09-16T04:01:48.242 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.228+0000 7efef6ffd640 10 client.954110.objecter resend_mon_ops 2024-09-16T04:01:48.243 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.228+0000 7efef6ffd640 1 -- 172.21.15.42:0/3741971663 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efef8026070 con 0x7eff00139ca0 2024-09-16T04:01:48.243 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.228+0000 7efef6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:01:48.243 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.228+0000 7efef6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:01:48.243 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.228+0000 7efef6ffd640 10 monclient: dump: 2024-09-16T04:01:48.243 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:01:48.243 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:01:48.243 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:48.243 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:48.243 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:01:48.243 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:01:48.243 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:01:48.243 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:01:48.243 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:01:48.243 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:01:48.243 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.228+0000 7efef6ffd640 1 -- 172.21.15.42:0/3741971663 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4880 (secure 0 0 0) 0x7efef801fa60 con 0x7eff00139ca0 2024-09-16T04:01:48.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.228+0000 7efef6ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:01:48.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.228+0000 7efef6ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:01:48.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.228+0000 7efef6ffd640 1 -- 172.21.15.42:0/3741971663 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7eff00139ca0 msgr2=0x7eff001d0060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:01:48.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.228+0000 7efef6ffd640 1 --2- 172.21.15.42:0/3741971663 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7eff00139ca0 0x7eff001d0060 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7efef800d500 tx=0x7efef8004700 comp rx=0 tx=0).stop 2024-09-16T04:01:48.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.228+0000 7efef6ffd640 1 -- 172.21.15.42:0/3741971663 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7eff00139ca0 msgr2=0x7eff001d0060 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:01:48.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.228+0000 7efef6ffd640 1 --2- 172.21.15.42:0/3741971663 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7eff00139ca0 0x7eff001d0060 secure :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7efef800d500 tx=0x7efef8004700 comp rx=0 tx=0).stop 2024-09-16T04:01:48.244 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T04:01:48.244 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:01:48.244 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:01:48.244 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T04:01:48.245 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 178, 2024-09-16T04:01:48.245 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 236408, 2024-09-16T04:01:48.245 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T04:01:48.245 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T04:01:48.245 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T04:01:48.245 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T04:01:48.245 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:01:48.245 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 4, 2024-09-16T04:01:48.245 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:01:48.245 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T04:01:48.245 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T04:01:48.245 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:01:48.245 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:01:48.245 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:01:48.246 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:01:48.246 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:01:48.246 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:01:48.246 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:01:48.246 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:01:48.246 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:01:48.246 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:01:48.246 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:01:48.246 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:01:48.246 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:01:48.246 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T04:01:48.246 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:01:48.246 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:01:48.246 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:01:48.247 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:01:48.247 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:01:48.247 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:01:48.247 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:01:48.247 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:01:48.247 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:01:48.247 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:01:48.247 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:01:48.247 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:01:48.247 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:01:48.247 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T04:01:48.247 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T04:01:48.247 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T04:01:48.247 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T04:01:48.248 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T04:01:48.248 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T04:01:48.248 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:01:48.248 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:01:48.248 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T04:01:48.248 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T04:01:48.248 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T04:01:48.248 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T04:01:48.248 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T04:01:48.248 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T04:01:48.248 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T04:01:48.248 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:01:48.248 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T04:01:48.249 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:01:48.249 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:01:48.249 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:01:48.249 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:01:48.249 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:01:48.249 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:01:48.249 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:01:48.249 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:01:48.249 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:01:48.249 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:01:48.249 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:01:48.249 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:01:48.249 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T04:01:48.250 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:01:48.250 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T04:01:48.250 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:01:48.250 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:01:48.250 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:01:48.250 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:01:48.250 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:01:48.250 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:01:48.250 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:01:48.250 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T04:01:48.250 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:01:48.250 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:01:48.250 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:01:48.250 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:01:48.251 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T04:01:48.251 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:01:48.251 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:01:48.251 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:01:48.251 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:01:48.251 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T04:01:48.251 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T04:01:48.251 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:01:48.251 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:01:48.251 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:01:48.251 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:01:48.251 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:01:48.251 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:01:48.251 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:01:48.252 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:01:48.252 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:01:48.252 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:01:48.252 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:01:48.252 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T04:01:48.252 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:01:48.252 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:01:48.252 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:01:48.252 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:01:48.252 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T04:01:48.252 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:01:48.252 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:01:48.252 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:01:48.252 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:01:48.253 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T04:01:48.253 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T04:01:48.253 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:01:48.253 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:01:48.253 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:01:48.253 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:01:48.253 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:01:48.253 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:01:48.253 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:01:48.253 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:01:48.253 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:01:48.253 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:01:48.253 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:01:48.253 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T04:01:48.253 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:01:48.254 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:01:48.254 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:01:48.254 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:01:48.254 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T04:01:48.254 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:01:48.254 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:01:48.254 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:01:48.254 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:01:48.254 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T04:01:48.254 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T04:01:48.254 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:01:48.254 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:01:48.254 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:01:48.255 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:01:48.255 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:01:48.255 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:01:48.255 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T04:01:48.255 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T04:01:48.255 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:01:48.255 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:01:48.255 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:01:48.255 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:01:48.255 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:01:48.255 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:01:48.255 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T04:01:48.255 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:01:48.255 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:01:48.256 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:01:48.256 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:01:48.256 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:01:48.256 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:01:48.256 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T04:01:48.256 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:01:48.256 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:01:48.256 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:01:48.256 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 2 2024-09-16T04:01:48.256 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:01:48.256 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:01:48.256 INFO:teuthology.orchestra.run.smithi042.stdout: "mgr": [ 2024-09-16T04:01:48.257 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:01:48.257 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:01:48.257 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:01:48.257 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:01:48.257 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:01:48.257 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:01:48.257 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:01:48.257 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T04:01:48.257 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T04:01:48.259 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.228+0000 7eff04fa1640 10 librados: watch_flush enter 2024-09-16T04:01:48.259 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.228+0000 7eff04fa1640 10 librados: watch_flush exit 2024-09-16T04:01:48.259 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.228+0000 7eff04fa1640 20 client.954110.objecter shutdown clearing up homeless session... 2024-09-16T04:01:48.259 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.228+0000 7eff04fa1640 10 client.954110.objecter successfully canceled tick 2024-09-16T04:01:48.259 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.228+0000 7eff04fa1640 1 -- 172.21.15.42:0/3741971663 >> v2:172.21.15.154:6812/254845342 conn(0x7efee0037810 msgr2=0x7efee0039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:01:48.259 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.228+0000 7eff04fa1640 1 --2- 172.21.15.42:0/3741971663 >> v2:172.21.15.154:6812/254845342 conn(0x7efee0037810 0x7efee0039cd0 secure :-1 s=READY pgs=3767 cs=0 l=1 rev1=1 crypto rx=0x7efee80ad000 tx=0x7efee8008040 comp rx=0 tx=0).stop 2024-09-16T04:01:48.259 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.228+0000 7eff04fa1640 10 monclient: shutdown 2024-09-16T04:01:48.259 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.228+0000 7eff04fa1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:01:48.259 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.228+0000 7eff04fa1640 1 -- 172.21.15.42:0/3741971663 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7eff0013c8c0 msgr2=0x7eff001cf5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:01:48.260 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.228+0000 7eff04fa1640 1 --2- 172.21.15.42:0/3741971663 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7eff0013c8c0 0x7eff001cf5e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7efef0007e60 tx=0x7efef000e4b0 comp rx=0 tx=0).stop 2024-09-16T04:01:48.260 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.228+0000 7efefed9d640 1 -- 172.21.15.42:0/3741971663 reap_dead start 2024-09-16T04:01:48.260 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.228+0000 7efefed9d640 1 -- 172.21.15.42:0/3741971663 reap_dead start 2024-09-16T04:01:48.260 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.228+0000 7eff04fa1640 1 -- 172.21.15.42:0/3741971663 shutdown_connections 2024-09-16T04:01:48.260 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.228+0000 7eff04fa1640 1 -- 172.21.15.42:0/3741971663 >> 172.21.15.42:0/3741971663 conn(0x7eff000a9fb0 msgr2=0x7eff000a8a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:01:48.260 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.228+0000 7eff04fa1640 1 -- 172.21.15.42:0/3741971663 shutdown_connections 2024-09-16T04:01:48.260 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.228+0000 7eff04fa1640 1 -- 172.21.15.42:0/3741971663 wait complete. 2024-09-16T04:01:48.260 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.228+0000 7eff04fa1640 1 librados: shutdown 2024-09-16T04:01:48.260 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:01:48.260 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T04:01:48.261 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T04:01:48.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.372+0000 7f6183140640 10 monclient: get_monmap_and_config 2024-09-16T04:01:48.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.372+0000 7f6183140640 10 monclient: build_initial_monmap 2024-09-16T04:01:48.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.372+0000 7f6183140640 10 monclient: monmap: 2024-09-16T04:01:48.377 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:01:48.377 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:01:48.377 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:01:48.375042+0000 2024-09-16T04:01:48.377 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:01:48.375042+0000 2024-09-16T04:01:48.377 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:01:48.377 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:01:48.377 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:01:48.377 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:01:48.377 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:01:48.378 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:01:48.378 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:01:48.378 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:01:48.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.372+0000 7f6183140640 1 Processor -- start 2024-09-16T04:01:48.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.372+0000 7f6183140640 1 -- start start 2024-09-16T04:01:48.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.372+0000 7f6183140640 10 monclient: init 2024-09-16T04:01:48.382 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.376+0000 7f6183140640 10 monclient: _reopen_session rank -1 2024-09-16T04:01:48.382 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.376+0000 7f6183140640 10 monclient: _add_conns ranks=[3,1,0,2,4] 2024-09-16T04:01:48.382 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.376+0000 7f6183140640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f617c0a1330 0x7f617c0a1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:01:48.382 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.376+0000 7f6183140640 10 monclient(hunting): picked mon.noname-a con 0x7f617c0a1330 addr v2:172.21.15.42:3300/0 2024-09-16T04:01:48.382 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.376+0000 7f6183140640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f617c06dac0 addr v1:172.21.15.154:6789/0 2024-09-16T04:01:48.382 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.376+0000 7f6183140640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f617c0a1c50 addr v1:172.21.15.42:6789/0 2024-09-16T04:01:48.382 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.376+0000 7f6183140640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f617c135f20 con 0x7f617c0a1c50 2024-09-16T04:01:48.382 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.376+0000 7f6183140640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f617c09c090 con 0x7f617c06dac0 2024-09-16T04:01:48.382 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.376+0000 7f6183140640 10 monclient(hunting): start opening mon connection 2024-09-16T04:01:48.382 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.376+0000 7f6183140640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f617c09f610 con 0x7f617c0a1330 2024-09-16T04:01:48.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.376+0000 7f6183140640 10 monclient(hunting): _renew_subs 2024-09-16T04:01:48.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.376+0000 7f6183140640 10 monclient(hunting): authenticate will time out at 2155.344453s 2024-09-16T04:01:48.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.376+0000 7f6180edc640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f617c0a1330 0x7f617c0a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:01:48.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.376+0000 7f61816dd640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f617c0a1c50 0x7f617c135810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:58938/0 (socket says 172.21.15.42:58938) 2024-09-16T04:01:48.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.376+0000 7f61816dd640 1 -- 172.21.15.42:0/3492108512 learned_addr learned my addr 172.21.15.42:0/3492108512 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T04:01:48.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.376+0000 7f6180edc640 10 monclient(hunting): get_auth_request con 0x7f617c0a1330 auth_method 0 2024-09-16T04:01:48.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.376+0000 7f6180edc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:01:48.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.376+0000 7f6180edc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:01:48.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.376+0000 7f6180edc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:01:48.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.376+0000 7f6180edc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:01:48.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.376+0000 7f6180edc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:01:48.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.376+0000 7f6180edc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:01:48.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.376+0000 7f6180edc640 10 monclient(hunting): handle_auth_done global_id 825411 payload 274 2024-09-16T04:01:48.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.376+0000 7f6180edc640 1 -- 172.21.15.42:0/3492108512 >> v1:172.21.15.154:6789/0 conn(0x7f617c06dac0 legacy=0x7f617c135100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:01:48.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.376+0000 7f617b7fe640 1 -- 172.21.15.42:0/3492108512 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 434321573 0 0) 0x7f617c135f20 con 0x7f617c0a1c50 2024-09-16T04:01:48.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.376+0000 7f6180edc640 1 -- 172.21.15.42:0/3492108512 >> v1:172.21.15.42:6789/0 conn(0x7f617c0a1c50 legacy=0x7f617c135810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:01:48.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.376+0000 7f6180edc640 10 monclient: _finish_hunting 0 2024-09-16T04:01:48.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.376+0000 7f6180edc640 1 monclient: found mon.noname-a 2024-09-16T04:01:48.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.376+0000 7f6180edc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:01:48.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.376+0000 7f6180edc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:01:48.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.376+0000 7f6180edc640 1 -- 172.21.15.42:0/3492108512 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f617c09f8f0 con 0x7f617c0a1330 2024-09-16T04:01:48.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.376+0000 7f617b7fe640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T04:01:48.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.376+0000 7f6180edc640 1 --2- 172.21.15.42:0/3492108512 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f617c0a1330 0x7f617c0a1710 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f61700097c0 tx=0x7f617001c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97ef5ba5e718d760 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:01:48.385 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.376+0000 7f617b7fe640 1 -- 172.21.15.42:0/3492108512 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61700b1070 con 0x7f617c0a1330 2024-09-16T04:01:48.385 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.376+0000 7f617b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:01:48.385 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.376+0000 7f617b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:01:48.385 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.376+0000 7f617b7fe640 10 monclient: dump: 2024-09-16T04:01:48.385 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:01:48.385 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:01:48.385 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:48.385 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:48.385 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:01:48.385 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:01:48.385 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:01:48.386 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:01:48.386 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:01:48.386 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:01:48.386 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.376+0000 7f617b7fe640 10 monclient: _finish_auth 0 2024-09-16T04:01:48.386 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.376+0000 7f617b7fe640 10 monclient: _check_auth_tickets 2024-09-16T04:01:48.386 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.376+0000 7f617b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:01:48.386 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.376+0000 7f617b7fe640 1 -- 172.21.15.42:0/3492108512 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f61700a8050 con 0x7f617c0a1330 2024-09-16T04:01:48.386 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.376+0000 7f617b7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:01:48.386 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.376+0000 7f617b7fe640 1 -- 172.21.15.42:0/3492108512 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61700ac860 con 0x7f617c0a1330 2024-09-16T04:01:48.386 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.376+0000 7f6183140640 5 monclient: authenticate success, global_id 825411 2024-09-16T04:01:48.386 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.376+0000 7f6183140640 10 monclient: get_monmap_and_config success 2024-09-16T04:01:48.386 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.376+0000 7f6183140640 4 set_mon_vals no callback set 2024-09-16T04:01:48.387 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.376+0000 7f617b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:01:48.387 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.376+0000 7f617b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:01:48.387 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.376+0000 7f617b7fe640 10 monclient: dump: 2024-09-16T04:01:48.387 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:01:48.387 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:01:48.387 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:48.387 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:48.387 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:01:48.387 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:01:48.387 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:01:48.387 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:01:48.388 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:01:48.388 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:01:48.388 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.380+0000 7f6183140640 10 monclient: shutdown 2024-09-16T04:01:48.388 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.380+0000 7f6183140640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:01:48.388 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.380+0000 7f6183140640 1 -- 172.21.15.42:0/3492108512 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f617c0a1330 msgr2=0x7f617c0a1710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:01:48.388 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.380+0000 7f6183140640 1 --2- 172.21.15.42:0/3492108512 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f617c0a1330 0x7f617c0a1710 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f61700097c0 tx=0x7f617001c3c0 comp rx=0 tx=0).stop 2024-09-16T04:01:48.388 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.380+0000 7f6183140640 1 -- 172.21.15.42:0/3492108512 shutdown_connections 2024-09-16T04:01:48.388 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.380+0000 7f6183140640 1 --2- 172.21.15.42:0/3492108512 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f617c0a1330 0x7f617c0a1710 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:48.388 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.380+0000 7f6183140640 1 -- 172.21.15.42:0/3492108512 >> 172.21.15.42:0/3492108512 conn(0x7f617c0a2420 msgr2=0x7f617c0a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:01:48.388 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.380+0000 7f6183140640 1 -- 172.21.15.42:0/3492108512 shutdown_connections 2024-09-16T04:01:48.388 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.380+0000 7f6183140640 1 -- 172.21.15.42:0/3492108512 wait complete. 2024-09-16T04:01:48.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.380+0000 7f6183140640 10 monclient: build_initial_monmap 2024-09-16T04:01:48.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.380+0000 7f6183140640 10 monclient: monmap: 2024-09-16T04:01:48.393 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:01:48.393 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:01:48.393 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:01:48.393 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:01:48.394 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:01:48.394 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:01:48.394 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:01:48.394 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:01:48.394 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:01:48.394 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:01:48.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.380+0000 7f6183140640 1 librados: starting msgr at 2024-09-16T04:01:48.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.380+0000 7f6183140640 1 librados: starting objecter 2024-09-16T04:01:48.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.380+0000 7f6183140640 1 Processor -- start 2024-09-16T04:01:48.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.380+0000 7f6183140640 1 -- start start 2024-09-16T04:01:48.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.380+0000 7f6183140640 1 librados: setting wanted keys 2024-09-16T04:01:48.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.380+0000 7f6183140640 1 librados: calling monclient init 2024-09-16T04:01:48.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.380+0000 7f6183140640 10 monclient: init 2024-09-16T04:01:48.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.380+0000 7f6183140640 10 monclient: _reopen_session rank -1 2024-09-16T04:01:48.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.380+0000 7f6183140640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:01:48.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.380+0000 7f6183140640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f617c0a1330 0x7f617c13ce10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:01:48.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.380+0000 7f6183140640 10 monclient(hunting): picked mon.noname-c con 0x7f617c0a1330 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:01:48.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.380+0000 7f6183140640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f617c0a1c50 0x7f617c1351b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:01:48.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.380+0000 7f6183140640 10 monclient(hunting): picked mon.noname-b con 0x7f617c0a1c50 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:01:48.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.380+0000 7f6183140640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f617c1356f0 0x7f617c1cec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:01:48.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.380+0000 7f6183140640 10 monclient(hunting): picked mon.noname-a con 0x7f617c1356f0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:01:48.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.380+0000 7f6183140640 10 monclient(hunting): start opening mon connection 2024-09-16T04:01:48.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.380+0000 7f6183140640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f617c09eef0 con 0x7f617c1356f0 2024-09-16T04:01:48.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.380+0000 7f6183140640 10 monclient(hunting): start opening mon connection 2024-09-16T04:01:48.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.380+0000 7f6183140640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f617c09ed70 con 0x7f617c0a1c50 2024-09-16T04:01:48.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.380+0000 7f6183140640 10 monclient(hunting): start opening mon connection 2024-09-16T04:01:48.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.380+0000 7f6183140640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f617c140b90 con 0x7f617c0a1330 2024-09-16T04:01:48.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.380+0000 7f6183140640 10 monclient(hunting): _renew_subs 2024-09-16T04:01:48.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.380+0000 7f6183140640 10 monclient(hunting): authenticate will time out at 2155.349850s 2024-09-16T04:01:48.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.380+0000 7f6180edc640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f617c0a1330 0x7f617c13ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:01:48.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.380+0000 7f6180edc640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f617c0a1330 0x7f617c13ce10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:49546/0 (socket says 172.21.15.42:49546) 2024-09-16T04:01:48.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.380+0000 7f6180edc640 1 -- 172.21.15.42:0/130010394 learned_addr learned my addr 172.21.15.42:0/130010394 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:01:48.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.380+0000 7f6180edc640 10 monclient(hunting): get_auth_request con 0x7f617c0a1330 auth_method 0 2024-09-16T04:01:48.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.380+0000 7f6180edc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:01:48.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.380+0000 7f6180edc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:01:48.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.380+0000 7f6180edc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:01:48.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.380+0000 7f61816dd640 1 --2- 172.21.15.42:0/130010394 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f617c1356f0 0x7f617c1cec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:01:48.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.380+0000 7f61816dd640 10 monclient(hunting): get_auth_request con 0x7f617c1356f0 auth_method 0 2024-09-16T04:01:48.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.380+0000 7f61816dd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:01:48.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.380+0000 7f61816dd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:01:48.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.380+0000 7f61816dd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:01:48.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.380+0000 7f6180edc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:01:48.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.380+0000 7f6180edc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:01:48.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.380+0000 7f6180edc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:01:48.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.380+0000 7f61816dd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:01:48.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.380+0000 7f61816dd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:01:48.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.380+0000 7f617bfff640 1 --2- 172.21.15.42:0/130010394 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f617c0a1c50 0x7f617c1351b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:01:48.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.380+0000 7f61816dd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:01:48.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.380+0000 7f617bfff640 10 monclient(hunting): get_auth_request con 0x7f617c0a1c50 auth_method 0 2024-09-16T04:01:48.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.380+0000 7f617bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:01:48.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.380+0000 7f617bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:01:48.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.380+0000 7f617bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:01:48.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.380+0000 7f6180edc640 10 monclient(hunting): handle_auth_done global_id 954113 payload 824 2024-09-16T04:01:48.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.380+0000 7f6180edc640 1 -- 172.21.15.42:0/130010394 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f617c0a1c50 msgr2=0x7f617c1351b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:01:48.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.380+0000 7f6180edc640 1 --2- 172.21.15.42:0/130010394 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f617c0a1c50 0x7f617c1351b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:48.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.380+0000 7f6180edc640 1 -- 172.21.15.42:0/130010394 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f617c1356f0 msgr2=0x7f617c1cec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:01:48.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.380+0000 7f6180edc640 1 --2- 172.21.15.42:0/130010394 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f617c1356f0 0x7f617c1cec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:48.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.380+0000 7f6180edc640 10 monclient: _finish_hunting 0 2024-09-16T04:01:48.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.380+0000 7f6180edc640 1 monclient: found mon.noname-c 2024-09-16T04:01:48.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.380+0000 7f6180edc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:01:48.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.380+0000 7f6180edc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T04:01:48.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.380+0000 7f6180edc640 1 -- 172.21.15.42:0/130010394 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f617c09ddb0 con 0x7f617c0a1330 2024-09-16T04:01:48.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.380+0000 7f6180edc640 1 --2- 172.21.15.42:0/130010394 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f617c0a1330 0x7f617c13ce10 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f6170018fd0 tx=0x7f6170019160 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:01:48.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.380+0000 7f61797fa640 10 client.?.objecter ms_handle_connect 0x7f617c0a1330 2024-09-16T04:01:48.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.380+0000 7f61797fa640 10 client.?.objecter resend_mon_ops 2024-09-16T04:01:48.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.380+0000 7f61797fa640 1 -- 172.21.15.42:0/130010394 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61700b9070 con 0x7f617c0a1330 2024-09-16T04:01:48.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.380+0000 7f61797fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:01:48.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.380+0000 7f61797fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:01:48.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.380+0000 7f61797fa640 10 monclient: dump: 2024-09-16T04:01:48.399 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:01:48.399 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:01:48.399 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:48.399 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:48.399 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:01:48.399 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:01:48.399 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:01:48.399 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:01:48.399 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:01:48.399 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:01:48.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.380+0000 7f61797fa640 10 monclient: _finish_auth 0 2024-09-16T04:01:48.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.380+0000 7f61797fa640 10 monclient: _check_auth_tickets 2024-09-16T04:01:48.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.380+0000 7f61797fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:01:48.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.380+0000 7f6183140640 5 monclient: authenticate success, global_id 954113 2024-09-16T04:01:48.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.380+0000 7f61797fa640 1 -- 172.21.15.42:0/130010394 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f61700b5410 con 0x7f617c0a1330 2024-09-16T04:01:48.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.380+0000 7f61797fa640 10 monclient: handle_config config(0 keys) 2024-09-16T04:01:48.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.380+0000 7f61797fa640 1 -- 172.21.15.42:0/130010394 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61700b1040 con 0x7f617c0a1330 2024-09-16T04:01:48.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.380+0000 7f61797fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:01:48.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.380+0000 7f61797fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:01:48.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.380+0000 7f61797fa640 10 monclient: dump: 2024-09-16T04:01:48.400 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:01:48.400 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:01:48.400 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:48.400 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:48.401 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:01:48.401 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:01:48.401 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:01:48.401 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:01:48.401 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:01:48.401 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:01:48.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.380+0000 7f617affd640 4 set_mon_vals no callback set 2024-09-16T04:01:48.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.380+0000 7f6183140640 10 monclient: _renew_subs 2024-09-16T04:01:48.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.380+0000 7f6183140640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:01:48.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.380+0000 7f6183140640 1 -- 172.21.15.42:0/130010394 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f617c120870 con 0x7f617c0a1330 2024-09-16T04:01:48.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.380+0000 7f6183140640 10 client.954113.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:01:48.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.380+0000 7f6183140640 10 monclient: _renew_subs 2024-09-16T04:01:48.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.380+0000 7f6183140640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:01:48.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.380+0000 7f6183140640 1 -- 172.21.15.42:0/130010394 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f617c09dfd0 con 0x7f617c0a1330 2024-09-16T04:01:48.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.380+0000 7f6183140640 1 librados: init done 2024-09-16T04:01:48.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.380+0000 7f6183140640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:01:48.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.380+0000 7f6183140640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:01:48.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.380+0000 7f6183140640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:01:48.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.380+0000 7f6183140640 1 -- 172.21.15.42:0/130010394 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f617c13def0 con 0x7f617c0a1330 2024-09-16T04:01:48.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.380+0000 7f61797fa640 1 -- 172.21.15.42:0/130010394 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f61700bc070 con 0x7f617c0a1330 2024-09-16T04:01:48.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.380+0000 7f61797fa640 1 --2- 172.21.15.42:0/130010394 >> v2:172.21.15.154:6812/254845342 conn(0x7f6168037810 0x7f6168039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:01:48.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.380+0000 7f61797fa640 1 -- 172.21.15.42:0/130010394 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(599..599 src has 451..599) ==== 6257+0+0 (secure 0 0 0) 0x7f61700e2ec0 con 0x7f617c0a1330 2024-09-16T04:01:48.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.380+0000 7f61797fa640 10 client.954113.objecter ms_dispatch 0x7f617c000ba0 osd_map(599..599 src has 451..599) 2024-09-16T04:01:48.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.380+0000 7f61797fa640 3 client.954113.objecter handle_osd_map got epochs [599,599] > 0 2024-09-16T04:01:48.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.380+0000 7f61797fa640 3 client.954113.objecter handle_osd_map decoding full epoch 599 2024-09-16T04:01:48.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.384+0000 7f61797fa640 20 client.954113.objecter dump_active .. 0 homeless 2024-09-16T04:01:48.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.384+0000 7f617bfff640 1 --2- 172.21.15.42:0/130010394 >> v2:172.21.15.154:6812/254845342 conn(0x7f6168037810 0x7f6168039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:01:48.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.384+0000 7f617bfff640 10 monclient: get_auth_request con 0x7f6168037810 auth_method 0 2024-09-16T04:01:48.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.384+0000 7f61797fa640 10 client.954113.objecter ms_handle_connect 0x7f6168037810 2024-09-16T04:01:48.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.384+0000 7f617bfff640 1 --2- 172.21.15.42:0/130010394 >> v2:172.21.15.154:6812/254845342 conn(0x7f6168037810 0x7f6168039cd0 secure :-1 s=READY pgs=3768 cs=0 l=1 rev1=1 crypto rx=0x7f617c13d650 tx=0x7f616c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:01:48.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.388+0000 7f61797fa640 1 -- 172.21.15.42:0/130010394 <== mon.2 v2:172.21.15.42:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f6170016070 con 0x7f617c0a1330 2024-09-16T04:01:48.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.388+0000 7f61797fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:01:48.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.388+0000 7f61797fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:01:48.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.616+0000 7f6183140640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T04:01:48.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.616+0000 7f6183140640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T04:01:48.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.616+0000 7f6183140640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:01:48.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.616+0000 7f6183140640 1 -- 172.21.15.42:0/130010394 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f617c071290 con 0x7f617c0a1330 2024-09-16T04:01:48.621 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.620+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.621 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.620+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=27,config=2,health=10,kv=10,logm=51} crc {auth=509458627,config=692619927,health=3063526734,kv=3446188498,logm=4248756239}) 2024-09-16T04:01:48.622 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.620+0000 7f61797fa640 1 -- 172.21.15.42:0/130010394 <== mon.2 v2:172.21.15.42:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f61700b5800 con 0x7f617c0a1330 2024-09-16T04:01:48.622 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.620+0000 7f61797fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T04:01:48.622 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.620+0000 7f61797fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:01:48.622 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.620+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=29,config=2,health=12,kv=10,logm=47} crc {auth=3015858920,config=692619927,health=3035128946,kv=3446188498,logm=3071473657}) 2024-09-16T04:01:48.622 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.620+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.622 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.620+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=27,config=2,health=10,kv=10,logm=51} crc {auth=509458627,config=692619927,health=3063526734,kv=3446188498,logm=4248756239}) 2024-09-16T04:01:48.622 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.620+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=29,config=2,health=11,kv=9,logm=49} crc {auth=737989904,config=692619927,health=1608548740,kv=2818795783,logm=1927731964}) 2024-09-16T04:01:48.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.620+0000 7f6183140640 10 librados: watch_flush enter 2024-09-16T04:01:48.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.620+0000 7f6183140640 10 librados: watch_flush exit 2024-09-16T04:01:48.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.620+0000 7f6183140640 20 client.954113.objecter shutdown clearing up homeless session... 2024-09-16T04:01:48.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.620+0000 7f6183140640 10 client.954113.objecter successfully canceled tick 2024-09-16T04:01:48.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.620+0000 7f6183140640 1 -- 172.21.15.42:0/130010394 >> v2:172.21.15.154:6812/254845342 conn(0x7f6168037810 msgr2=0x7f6168039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:01:48.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.620+0000 7f6183140640 1 --2- 172.21.15.42:0/130010394 >> v2:172.21.15.154:6812/254845342 conn(0x7f6168037810 0x7f6168039cd0 secure :-1 s=READY pgs=3768 cs=0 l=1 rev1=1 crypto rx=0x7f617c13d650 tx=0x7f616c009040 comp rx=0 tx=0).stop 2024-09-16T04:01:48.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.620+0000 7f6183140640 10 monclient: shutdown 2024-09-16T04:01:48.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.620+0000 7f6183140640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:01:48.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.620+0000 7f6183140640 1 -- 172.21.15.42:0/130010394 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f617c0a1330 msgr2=0x7f617c13ce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:01:48.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.620+0000 7f6183140640 1 --2- 172.21.15.42:0/130010394 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f617c0a1330 0x7f617c13ce10 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f6170018fd0 tx=0x7f6170019160 comp rx=0 tx=0).stop 2024-09-16T04:01:48.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.620+0000 7f6183140640 1 -- 172.21.15.42:0/130010394 shutdown_connections 2024-09-16T04:01:48.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.620+0000 7f6183140640 1 --2- 172.21.15.42:0/130010394 >> v2:172.21.15.154:6812/254845342 conn(0x7f6168037810 0x7f6168039cd0 unknown :-1 s=CLOSED pgs=3768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:48.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.620+0000 7f6183140640 1 --2- 172.21.15.42:0/130010394 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f617c1356f0 0x7f617c1cec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:48.625 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.620+0000 7f6183140640 1 --2- 172.21.15.42:0/130010394 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f617c0a1c50 0x7f617c1351b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:48.625 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.620+0000 7f6183140640 1 --2- 172.21.15.42:0/130010394 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f617c0a1330 0x7f617c13ce10 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:48.625 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.620+0000 7f6183140640 1 -- 172.21.15.42:0/130010394 >> 172.21.15.42:0/130010394 conn(0x7f617c0a2420 msgr2=0x7f617c06f980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:01:48.625 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.620+0000 7f6183140640 1 -- 172.21.15.42:0/130010394 shutdown_connections 2024-09-16T04:01:48.625 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.620+0000 7f6183140640 1 -- 172.21.15.42:0/130010394 wait complete. 2024-09-16T04:01:48.625 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:48.620+0000 7f6183140640 1 librados: shutdown 2024-09-16T04:01:48.626 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.620+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.626 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.620+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3358346890}) 2024-09-16T04:01:48.626 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.620+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4159106063}) 2024-09-16T04:01:48.626 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.620+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.626 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.620+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3358346890}) 2024-09-16T04:01:48.626 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.620+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1608600957}) 2024-09-16T04:01:48.628 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.624+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.628 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.624+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=48869497}) 2024-09-16T04:01:48.628 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.624+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=608341869}) 2024-09-16T04:01:48.628 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.624+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.628 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.624+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=48869497}) 2024-09-16T04:01:48.628 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.624+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1465476157}) 2024-09-16T04:01:48.631 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.628+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.632 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.628+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2406614277}) 2024-09-16T04:01:48.632 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.628+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1011296493}) 2024-09-16T04:01:48.632 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.628+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.632 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.628+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2406614277}) 2024-09-16T04:01:48.632 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.628+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3210153209}) 2024-09-16T04:01:48.636 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.632+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.636 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.632+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3920040982}) 2024-09-16T04:01:48.636 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.632+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3679398145}) 2024-09-16T04:01:48.636 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.632+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.636 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.632+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3920040982}) 2024-09-16T04:01:48.636 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.632+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=829731220}) 2024-09-16T04:01:48.638 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.636+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.638 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.636+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=394650886}) 2024-09-16T04:01:48.638 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.636+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3191105099}) 2024-09-16T04:01:48.638 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.636+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.638 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.636+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=394650886}) 2024-09-16T04:01:48.638 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.636+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2785864603}) 2024-09-16T04:01:48.640 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.636+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.640 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.636+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1087171282}) 2024-09-16T04:01:48.640 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.636+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=756844621}) 2024-09-16T04:01:48.640 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.636+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.640 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.636+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1087171282}) 2024-09-16T04:01:48.640 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.636+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2572833118}) 2024-09-16T04:01:48.642 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T04:01:48.642 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.640+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.643 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.640+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4017404763}) 2024-09-16T04:01:48.643 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.640+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2304015205}) 2024-09-16T04:01:48.643 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.640+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.643 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.640+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4017404763}) 2024-09-16T04:01:48.643 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.640+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=666412333}) 2024-09-16T04:01:48.644 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.640+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.644 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.640+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2810579325}) 2024-09-16T04:01:48.644 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.640+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3722689208}) 2024-09-16T04:01:48.644 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.640+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.644 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.640+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2810579325}) 2024-09-16T04:01:48.645 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.640+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3232674899}) 2024-09-16T04:01:48.646 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.644+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.646 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.644+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2846602641}) 2024-09-16T04:01:48.646 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.644+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2042349849}) 2024-09-16T04:01:48.646 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.644+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.646 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.644+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2846602641}) 2024-09-16T04:01:48.647 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.644+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3433612073}) 2024-09-16T04:01:48.648 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.644+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.648 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.644+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=72775493}) 2024-09-16T04:01:48.648 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.644+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=328684210}) 2024-09-16T04:01:48.648 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.644+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.648 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.644+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=72775493}) 2024-09-16T04:01:48.648 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.644+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2394629505}) 2024-09-16T04:01:48.650 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.648+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.650 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.648+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2940157003}) 2024-09-16T04:01:48.650 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.648+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1020482771}) 2024-09-16T04:01:48.650 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.648+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.650 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.648+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2940157003}) 2024-09-16T04:01:48.650 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.648+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3735903354}) 2024-09-16T04:01:48.652 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.648+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.652 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.648+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3498557701}) 2024-09-16T04:01:48.652 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.648+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3232066975}) 2024-09-16T04:01:48.653 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.648+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.653 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.648+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3498557701}) 2024-09-16T04:01:48.653 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.648+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=323994176}) 2024-09-16T04:01:48.654 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.652+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.654 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.652+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3965304776}) 2024-09-16T04:01:48.654 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.652+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2561281146}) 2024-09-16T04:01:48.654 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.652+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.654 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.652+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3965304776}) 2024-09-16T04:01:48.654 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.652+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3010905496}) 2024-09-16T04:01:48.656 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.652+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.656 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.652+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1228331619}) 2024-09-16T04:01:48.656 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.652+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2334955203}) 2024-09-16T04:01:48.656 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.652+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.656 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.652+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1228331619}) 2024-09-16T04:01:48.657 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.652+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=772275938}) 2024-09-16T04:01:48.658 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.656+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.658 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.656+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3037948185}) 2024-09-16T04:01:48.658 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.656+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3122400373}) 2024-09-16T04:01:48.658 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.656+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.659 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.656+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3037948185}) 2024-09-16T04:01:48.659 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.656+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4019728249}) 2024-09-16T04:01:48.660 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.656+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.660 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.656+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4145801240}) 2024-09-16T04:01:48.660 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.656+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2708364300}) 2024-09-16T04:01:48.660 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.656+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.661 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.656+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4145801240}) 2024-09-16T04:01:48.661 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.656+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3499922266}) 2024-09-16T04:01:48.662 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.660+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.662 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.660+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1961495355}) 2024-09-16T04:01:48.662 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.660+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3888142715}) 2024-09-16T04:01:48.662 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.660+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.662 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.660+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1961495355}) 2024-09-16T04:01:48.663 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.660+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2796430211}) 2024-09-16T04:01:48.664 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.660+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.664 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.660+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2661712370}) 2024-09-16T04:01:48.664 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.660+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=5259778}) 2024-09-16T04:01:48.664 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.660+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.664 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.660+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2661712370}) 2024-09-16T04:01:48.664 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.660+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1071904759}) 2024-09-16T04:01:48.666 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.664+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.666 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.664+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1020010118}) 2024-09-16T04:01:48.666 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.664+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=927294707}) 2024-09-16T04:01:48.666 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.664+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.666 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.664+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1020010118}) 2024-09-16T04:01:48.666 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.664+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3876220264}) 2024-09-16T04:01:48.667 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.664+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.668 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.664+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3746333932}) 2024-09-16T04:01:48.668 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.664+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3692688621}) 2024-09-16T04:01:48.668 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.664+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.668 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.664+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3746333932}) 2024-09-16T04:01:48.668 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.664+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1536440590}) 2024-09-16T04:01:48.669 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.668+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.669 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.668+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=147307943}) 2024-09-16T04:01:48.669 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.668+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1837631497}) 2024-09-16T04:01:48.669 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.668+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.670 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.668+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=147307943}) 2024-09-16T04:01:48.670 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.668+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2531454366}) 2024-09-16T04:01:48.671 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.668+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.671 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.668+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1875206793}) 2024-09-16T04:01:48.671 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.668+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3437424220}) 2024-09-16T04:01:48.671 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.668+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.671 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.668+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1875206793}) 2024-09-16T04:01:48.671 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.668+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1911802351}) 2024-09-16T04:01:48.673 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.668+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.673 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.668+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3967561955}) 2024-09-16T04:01:48.673 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.668+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3396403632}) 2024-09-16T04:01:48.673 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.668+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.673 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.668+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3967561955}) 2024-09-16T04:01:48.673 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.668+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3976257963}) 2024-09-16T04:01:48.675 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.672+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.675 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.672+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=51894082}) 2024-09-16T04:01:48.675 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.672+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3367937043}) 2024-09-16T04:01:48.675 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.672+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.675 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.672+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=51894082}) 2024-09-16T04:01:48.675 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.672+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3012052309}) 2024-09-16T04:01:48.677 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.672+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.677 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.672+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2636907412}) 2024-09-16T04:01:48.677 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.672+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2454214850}) 2024-09-16T04:01:48.677 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.672+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.677 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.672+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2636907412}) 2024-09-16T04:01:48.677 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.672+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1978687542}) 2024-09-16T04:01:48.678 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.676+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.679 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.676+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3060987066}) 2024-09-16T04:01:48.679 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.676+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2317131982}) 2024-09-16T04:01:48.679 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.676+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.679 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.676+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3060987066}) 2024-09-16T04:01:48.679 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.676+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2157404248}) 2024-09-16T04:01:48.680 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.676+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.680 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.676+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=792274063}) 2024-09-16T04:01:48.680 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.676+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2683022352}) 2024-09-16T04:01:48.681 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.676+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.681 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.676+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=792274063}) 2024-09-16T04:01:48.681 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.676+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1714701984}) 2024-09-16T04:01:48.682 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.680+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.682 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.680+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1332278230}) 2024-09-16T04:01:48.682 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.680+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1613308837}) 2024-09-16T04:01:48.682 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.680+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.682 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.680+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1332278230}) 2024-09-16T04:01:48.683 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.680+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=929052991}) 2024-09-16T04:01:48.684 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.680+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.684 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.680+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1713225370}) 2024-09-16T04:01:48.684 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.680+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1594858605}) 2024-09-16T04:01:48.684 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.680+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.684 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.680+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1713225370}) 2024-09-16T04:01:48.685 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.680+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3274029547}) 2024-09-16T04:01:48.686 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.684+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.686 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.684+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2673275267}) 2024-09-16T04:01:48.686 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.684+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=369079873}) 2024-09-16T04:01:48.686 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.684+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.686 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.684+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2673275267}) 2024-09-16T04:01:48.686 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.684+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1397636738}) 2024-09-16T04:01:48.688 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.684+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.688 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.684+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1204385170}) 2024-09-16T04:01:48.688 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.684+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2259187186}) 2024-09-16T04:01:48.688 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.684+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.689 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.684+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1204385170}) 2024-09-16T04:01:48.689 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.684+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3415021722}) 2024-09-16T04:01:48.690 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.688+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.690 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.688+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=450277306}) 2024-09-16T04:01:48.690 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.688+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2537607275}) 2024-09-16T04:01:48.690 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.688+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.691 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.688+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=450277306}) 2024-09-16T04:01:48.691 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.688+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2343955075}) 2024-09-16T04:01:48.692 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.688+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.692 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.688+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4012030614}) 2024-09-16T04:01:48.692 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.688+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1865610030}) 2024-09-16T04:01:48.692 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.688+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.692 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.688+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4012030614}) 2024-09-16T04:01:48.693 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.688+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1556942963}) 2024-09-16T04:01:48.694 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.692+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.694 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.692+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2985772687}) 2024-09-16T04:01:48.694 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.692+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4199168096}) 2024-09-16T04:01:48.694 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.692+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.694 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.692+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2985772687}) 2024-09-16T04:01:48.695 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.692+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2483410882}) 2024-09-16T04:01:48.696 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.692+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.696 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.692+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1839870049}) 2024-09-16T04:01:48.696 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.692+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3292211986}) 2024-09-16T04:01:48.696 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.692+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.696 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.692+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1839870049}) 2024-09-16T04:01:48.697 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.692+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3885664508}) 2024-09-16T04:01:48.698 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.696+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.698 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.696+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=581714986}) 2024-09-16T04:01:48.698 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.696+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=583312702}) 2024-09-16T04:01:48.699 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.696+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.699 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.696+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=581714986}) 2024-09-16T04:01:48.699 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.696+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=315867853}) 2024-09-16T04:01:48.700 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.696+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.700 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.696+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1968715958}) 2024-09-16T04:01:48.700 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.696+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4234673849}) 2024-09-16T04:01:48.700 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.696+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.701 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.696+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1968715958}) 2024-09-16T04:01:48.701 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.696+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3630497214}) 2024-09-16T04:01:48.702 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.700+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.702 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.700+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=230143255}) 2024-09-16T04:01:48.702 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.700+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3401456905}) 2024-09-16T04:01:48.702 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.700+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.702 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.700+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=230143255}) 2024-09-16T04:01:48.703 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.700+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=810266276}) 2024-09-16T04:01:48.704 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.700+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.704 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.700+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1872876989}) 2024-09-16T04:01:48.705 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.700+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3853872339}) 2024-09-16T04:01:48.705 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.700+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.705 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.700+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1872876989}) 2024-09-16T04:01:48.705 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.700+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2754273611}) 2024-09-16T04:01:48.706 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.704+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.706 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.704+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3286082394}) 2024-09-16T04:01:48.706 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.704+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=569235289}) 2024-09-16T04:01:48.706 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.704+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.706 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.704+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3286082394}) 2024-09-16T04:01:48.706 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.704+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2568271327}) 2024-09-16T04:01:48.708 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.704+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.708 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.704+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4057945196}) 2024-09-16T04:01:48.708 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.704+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2375551179}) 2024-09-16T04:01:48.708 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.704+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.708 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.704+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4057945196}) 2024-09-16T04:01:48.708 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.704+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4006903529}) 2024-09-16T04:01:48.710 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.708+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.710 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.708+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=882405340}) 2024-09-16T04:01:48.710 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.708+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2397544311}) 2024-09-16T04:01:48.710 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.708+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.710 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.708+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=882405340}) 2024-09-16T04:01:48.710 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.708+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1319096719}) 2024-09-16T04:01:48.712 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.708+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.712 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.708+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=68282279}) 2024-09-16T04:01:48.712 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.708+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1774874946}) 2024-09-16T04:01:48.712 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.708+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.712 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.708+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=68282279}) 2024-09-16T04:01:48.712 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.708+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2597935200}) 2024-09-16T04:01:48.714 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.712+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.714 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.712+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1002630041}) 2024-09-16T04:01:48.714 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.712+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=43782111}) 2024-09-16T04:01:48.714 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.712+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.714 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.712+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1002630041}) 2024-09-16T04:01:48.714 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.712+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2286848973}) 2024-09-16T04:01:48.716 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.712+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.716 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.712+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4062672896}) 2024-09-16T04:01:48.716 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.712+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3010367161}) 2024-09-16T04:01:48.716 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.712+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.716 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.712+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4062672896}) 2024-09-16T04:01:48.716 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.712+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2725926209}) 2024-09-16T04:01:48.718 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.716+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.718 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.716+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=19593068}) 2024-09-16T04:01:48.718 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.716+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=776096644}) 2024-09-16T04:01:48.718 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.716+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.718 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.716+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=19593068}) 2024-09-16T04:01:48.718 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.716+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3901845344}) 2024-09-16T04:01:48.720 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.716+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.720 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.716+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=421184859}) 2024-09-16T04:01:48.720 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.716+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2477635547}) 2024-09-16T04:01:48.720 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.716+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.721 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.716+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=421184859}) 2024-09-16T04:01:48.721 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.716+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4050200729}) 2024-09-16T04:01:48.722 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.720+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.722 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.720+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1494705458}) 2024-09-16T04:01:48.722 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.720+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=134419037}) 2024-09-16T04:01:48.722 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.720+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.722 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.720+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1494705458}) 2024-09-16T04:01:48.722 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.720+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=245244416}) 2024-09-16T04:01:48.724 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.720+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.724 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.720+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2723080991}) 2024-09-16T04:01:48.724 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.720+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3160005179}) 2024-09-16T04:01:48.724 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.720+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.724 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.720+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2723080991}) 2024-09-16T04:01:48.724 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.720+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3104575887}) 2024-09-16T04:01:48.725 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.724+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.725 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.724+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2959485984}) 2024-09-16T04:01:48.726 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.724+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3401123518}) 2024-09-16T04:01:48.726 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.724+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.726 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.724+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2959485984}) 2024-09-16T04:01:48.726 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.724+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=313569868}) 2024-09-16T04:01:48.727 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.724+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.728 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.724+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=699164722}) 2024-09-16T04:01:48.728 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.724+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3267442465}) 2024-09-16T04:01:48.728 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.724+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.728 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.724+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=699164722}) 2024-09-16T04:01:48.728 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.724+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3491940845}) 2024-09-16T04:01:48.729 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.728+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.729 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.728+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4204994083}) 2024-09-16T04:01:48.729 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.728+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2335349287}) 2024-09-16T04:01:48.730 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.728+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.730 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.728+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4204994083}) 2024-09-16T04:01:48.730 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.728+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3929065948}) 2024-09-16T04:01:48.731 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.728+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.731 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.728+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=736235422}) 2024-09-16T04:01:48.731 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.728+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2713145080}) 2024-09-16T04:01:48.732 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.728+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.732 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.728+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=736235422}) 2024-09-16T04:01:48.732 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.728+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1956278577}) 2024-09-16T04:01:48.733 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.728+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.733 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.728+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=329867651}) 2024-09-16T04:01:48.733 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.728+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1627580350}) 2024-09-16T04:01:48.733 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.728+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.733 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.728+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=329867651}) 2024-09-16T04:01:48.733 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.728+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=965520317}) 2024-09-16T04:01:48.738 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.736+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.738 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.736+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3732950642}) 2024-09-16T04:01:48.738 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.736+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=914915674}) 2024-09-16T04:01:48.738 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.736+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.738 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.736+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3732950642}) 2024-09-16T04:01:48.738 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.736+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1515082311}) 2024-09-16T04:01:48.743 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.740+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.743 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.740+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3805054293}) 2024-09-16T04:01:48.743 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.740+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=410707517}) 2024-09-16T04:01:48.744 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.740+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.744 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.740+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3805054293}) 2024-09-16T04:01:48.744 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.740+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4079297418}) 2024-09-16T04:01:48.760 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.756+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.760 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.756+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3977770949}) 2024-09-16T04:01:48.760 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.756+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2115829953}) 2024-09-16T04:01:48.760 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.756+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.761 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.756+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3977770949}) 2024-09-16T04:01:48.761 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.756+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3826901479}) 2024-09-16T04:01:48.767 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.764+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.767 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.764+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1413187996}) 2024-09-16T04:01:48.767 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.764+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2701351819}) 2024-09-16T04:01:48.767 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.764+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.767 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.764+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1413187996}) 2024-09-16T04:01:48.767 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.764+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2691248991}) 2024-09-16T04:01:48.769 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.768+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.769 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.768+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3030737141}) 2024-09-16T04:01:48.769 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.768+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=950533849}) 2024-09-16T04:01:48.769 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.768+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.770 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.768+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3030737141}) 2024-09-16T04:01:48.770 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.768+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=16412915}) 2024-09-16T04:01:48.771 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.768+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.771 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.768+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3407931187}) 2024-09-16T04:01:48.771 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.768+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=154314177}) 2024-09-16T04:01:48.771 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.768+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.771 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.768+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3407931187}) 2024-09-16T04:01:48.772 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.768+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=354384964}) 2024-09-16T04:01:48.773 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.768+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.773 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.768+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1790625458}) 2024-09-16T04:01:48.773 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.768+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=647591176}) 2024-09-16T04:01:48.773 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.768+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.773 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.768+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1790625458}) 2024-09-16T04:01:48.773 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.768+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4000297200}) 2024-09-16T04:01:48.774 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.772+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.774 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.772+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1085560068}) 2024-09-16T04:01:48.774 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.772+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1657498161}) 2024-09-16T04:01:48.774 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.772+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.774 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.772+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1085560068}) 2024-09-16T04:01:48.775 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.772+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=785911278}) 2024-09-16T04:01:48.776 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.772+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.776 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.772+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2897715653}) 2024-09-16T04:01:48.776 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.772+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2988841397}) 2024-09-16T04:01:48.776 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.772+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.776 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.772+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2897715653}) 2024-09-16T04:01:48.776 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.772+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3808519497}) 2024-09-16T04:01:48.777 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.776+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.778 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.776+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4234383702}) 2024-09-16T04:01:48.778 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.776+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2610037715}) 2024-09-16T04:01:48.778 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.776+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.778 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.776+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4234383702}) 2024-09-16T04:01:48.778 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.776+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3364973449}) 2024-09-16T04:01:48.779 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.776+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.779 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.776+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=157135973}) 2024-09-16T04:01:48.779 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.776+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1104367256}) 2024-09-16T04:01:48.779 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.776+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.780 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.776+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=157135973}) 2024-09-16T04:01:48.780 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.776+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1233102864}) 2024-09-16T04:01:48.781 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.776+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.781 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.776+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1315741870}) 2024-09-16T04:01:48.781 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.776+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=542336213}) 2024-09-16T04:01:48.781 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.776+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.781 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.776+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1315741870}) 2024-09-16T04:01:48.781 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.776+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3797976002}) 2024-09-16T04:01:48.782 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.780+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.783 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.780+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2193536566}) 2024-09-16T04:01:48.783 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.780+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3970455317}) 2024-09-16T04:01:48.783 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.780+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.783 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.780+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2193536566}) 2024-09-16T04:01:48.783 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.780+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1041702264}) 2024-09-16T04:01:48.784 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.780+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.784 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.780+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3348163312}) 2024-09-16T04:01:48.784 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.780+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=585047474}) 2024-09-16T04:01:48.785 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.780+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.785 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.780+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3348163312}) 2024-09-16T04:01:48.785 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.780+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1506643136}) 2024-09-16T04:01:48.786 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.784+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.786 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.784+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2538155766}) 2024-09-16T04:01:48.786 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.784+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3221398503}) 2024-09-16T04:01:48.786 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.784+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.786 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.784+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2538155766}) 2024-09-16T04:01:48.786 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.784+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1177043917}) 2024-09-16T04:01:48.788 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.784+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.788 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.784+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=125230935}) 2024-09-16T04:01:48.788 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.784+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1580098856}) 2024-09-16T04:01:48.788 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.784+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.788 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.784+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=125230935}) 2024-09-16T04:01:48.788 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.784+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4012465909}) 2024-09-16T04:01:48.789 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.788+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.789 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.788+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4269450722}) 2024-09-16T04:01:48.789 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.788+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2835723687}) 2024-09-16T04:01:48.790 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.788+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.790 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.788+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4269450722}) 2024-09-16T04:01:48.790 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.788+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=804599998}) 2024-09-16T04:01:48.791 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.788+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.792 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.788+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2564149418}) 2024-09-16T04:01:48.792 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.788+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=558568180}) 2024-09-16T04:01:48.792 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.788+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.792 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.788+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2564149418}) 2024-09-16T04:01:48.792 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.788+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2230786837}) 2024-09-16T04:01:48.793 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.792+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.793 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.792+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=42202494}) 2024-09-16T04:01:48.793 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.792+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2168894180}) 2024-09-16T04:01:48.794 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.792+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.794 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.792+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=42202494}) 2024-09-16T04:01:48.794 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.792+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3505898567}) 2024-09-16T04:01:48.796 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.792+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.796 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.792+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2026732696}) 2024-09-16T04:01:48.796 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.792+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3544527638}) 2024-09-16T04:01:48.796 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.792+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.796 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.792+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2026732696}) 2024-09-16T04:01:48.796 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.792+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3483846090}) 2024-09-16T04:01:48.797 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.792+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.797 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.792+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4200202413}) 2024-09-16T04:01:48.797 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.796+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=444417349}) 2024-09-16T04:01:48.797 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.796+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.798 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.796+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4200202413}) 2024-09-16T04:01:48.798 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.796+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1389617391}) 2024-09-16T04:01:48.799 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.796+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.799 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.796+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=187021520}) 2024-09-16T04:01:48.800 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.796+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2966432086}) 2024-09-16T04:01:48.800 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.796+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.800 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.796+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=187021520}) 2024-09-16T04:01:48.800 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.796+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2013747723}) 2024-09-16T04:01:48.801 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.796+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.801 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.800+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1270467849}) 2024-09-16T04:01:48.801 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.800+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1970335870}) 2024-09-16T04:01:48.802 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.800+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.802 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.800+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1270467849}) 2024-09-16T04:01:48.802 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.800+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2372889668}) 2024-09-16T04:01:48.803 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.800+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.804 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.800+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3494055723}) 2024-09-16T04:01:48.804 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.800+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3729022400}) 2024-09-16T04:01:48.804 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.800+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.804 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.800+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3494055723}) 2024-09-16T04:01:48.804 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.800+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1864833264}) 2024-09-16T04:01:48.805 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.800+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.805 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.804+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2923475879}) 2024-09-16T04:01:48.805 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.804+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3381124099}) 2024-09-16T04:01:48.805 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.804+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.806 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.804+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2923475879}) 2024-09-16T04:01:48.806 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.804+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2533762863}) 2024-09-16T04:01:48.807 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.804+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.807 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.804+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=150923249}) 2024-09-16T04:01:48.807 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.804+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2463585121}) 2024-09-16T04:01:48.808 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.804+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.808 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.804+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=150923249}) 2024-09-16T04:01:48.808 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.804+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2854224178}) 2024-09-16T04:01:48.809 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.804+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.810 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.808+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3580613872}) 2024-09-16T04:01:48.810 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.808+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2184739290}) 2024-09-16T04:01:48.810 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.808+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.810 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.808+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3580613872}) 2024-09-16T04:01:48.810 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.808+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4109163325}) 2024-09-16T04:01:48.811 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.808+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.811 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.808+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3377469806}) 2024-09-16T04:01:48.811 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.808+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3086532901}) 2024-09-16T04:01:48.811 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.808+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.811 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.808+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3377469806}) 2024-09-16T04:01:48.811 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.808+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=705063223}) 2024-09-16T04:01:48.813 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.808+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.813 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.808+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3833375693}) 2024-09-16T04:01:48.813 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.808+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2256747243}) 2024-09-16T04:01:48.813 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.808+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.813 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.808+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3833375693}) 2024-09-16T04:01:48.813 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.808+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1123230727}) 2024-09-16T04:01:48.815 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.812+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.815 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.812+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2462058942}) 2024-09-16T04:01:48.815 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.812+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=12759420}) 2024-09-16T04:01:48.815 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.812+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.815 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.812+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2462058942}) 2024-09-16T04:01:48.815 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.812+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3817541199}) 2024-09-16T04:01:48.816 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.812+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.816 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.812+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=859724935}) 2024-09-16T04:01:48.817 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.812+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=73482678}) 2024-09-16T04:01:48.817 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.812+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.817 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.812+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=859724935}) 2024-09-16T04:01:48.817 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.812+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2658130217}) 2024-09-16T04:01:48.818 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.816+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.818 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.816+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4169749305}) 2024-09-16T04:01:48.818 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.816+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1311805126}) 2024-09-16T04:01:48.819 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.816+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.819 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.816+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4169749305}) 2024-09-16T04:01:48.819 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.816+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1702098571}) 2024-09-16T04:01:48.821 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.816+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.821 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.816+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3002711925}) 2024-09-16T04:01:48.821 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.816+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1184787917}) 2024-09-16T04:01:48.821 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.816+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.821 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.816+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3002711925}) 2024-09-16T04:01:48.821 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.816+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2386073272}) 2024-09-16T04:01:48.822 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.820+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.822 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.820+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4207548088}) 2024-09-16T04:01:48.823 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.820+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3263479586}) 2024-09-16T04:01:48.823 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.820+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.823 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.820+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4207548088}) 2024-09-16T04:01:48.823 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.820+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=128983969}) 2024-09-16T04:01:48.824 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.820+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.824 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.820+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1467115474}) 2024-09-16T04:01:48.824 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.820+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=969526025}) 2024-09-16T04:01:48.824 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.820+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.824 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.820+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1467115474}) 2024-09-16T04:01:48.824 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.820+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3966489558}) 2024-09-16T04:01:48.826 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.824+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.826 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.824+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4167630657}) 2024-09-16T04:01:48.826 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.824+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1345584900}) 2024-09-16T04:01:48.826 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.824+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.826 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.824+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4167630657}) 2024-09-16T04:01:48.826 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.824+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2133957991}) 2024-09-16T04:01:48.828 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.824+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.828 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.824+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=928951428}) 2024-09-16T04:01:48.828 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.824+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1098774703}) 2024-09-16T04:01:48.828 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.824+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.828 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.824+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=928951428}) 2024-09-16T04:01:48.828 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.824+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3103869727}) 2024-09-16T04:01:48.829 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.828+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.830 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.828+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=892802221}) 2024-09-16T04:01:48.830 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.828+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1203892909}) 2024-09-16T04:01:48.830 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.828+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.830 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.828+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=892802221}) 2024-09-16T04:01:48.830 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.828+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=890402601}) 2024-09-16T04:01:48.831 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.828+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.831 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.828+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3734217401}) 2024-09-16T04:01:48.831 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.828+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2922092820}) 2024-09-16T04:01:48.831 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.828+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.832 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.828+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3734217401}) 2024-09-16T04:01:48.832 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.828+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=870527257}) 2024-09-16T04:01:48.833 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.828+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.833 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.828+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2053267284}) 2024-09-16T04:01:48.833 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.828+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3645345785}) 2024-09-16T04:01:48.834 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.828+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.834 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.828+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2053267284}) 2024-09-16T04:01:48.834 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.828+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1930127230}) 2024-09-16T04:01:48.834 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.832+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.834 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.832+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=53389191}) 2024-09-16T04:01:48.835 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.832+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2702864963}) 2024-09-16T04:01:48.835 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.832+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.835 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.832+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=53389191}) 2024-09-16T04:01:48.835 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.832+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3983305615}) 2024-09-16T04:01:48.836 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.832+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.836 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.832+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2278835982}) 2024-09-16T04:01:48.836 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.832+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3735735234}) 2024-09-16T04:01:48.836 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.832+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.836 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.832+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2278835982}) 2024-09-16T04:01:48.836 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.832+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4289543775}) 2024-09-16T04:01:48.838 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.836+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.838 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.836+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1443859638}) 2024-09-16T04:01:48.838 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.836+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1984371521}) 2024-09-16T04:01:48.838 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.836+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.838 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.836+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1443859638}) 2024-09-16T04:01:48.838 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.836+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=655299109}) 2024-09-16T04:01:48.839 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.836+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.839 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.836+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3727217516}) 2024-09-16T04:01:48.840 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.836+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=622951631}) 2024-09-16T04:01:48.840 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.836+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.840 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.836+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3727217516}) 2024-09-16T04:01:48.840 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.836+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3093036788}) 2024-09-16T04:01:48.841 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.836+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.841 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.836+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4209111211}) 2024-09-16T04:01:48.841 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.836+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=621136187}) 2024-09-16T04:01:48.841 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.836+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.841 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.836+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4209111211}) 2024-09-16T04:01:48.841 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.836+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2752095253}) 2024-09-16T04:01:48.843 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.840+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.843 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.840+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2353822125}) 2024-09-16T04:01:48.843 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.840+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3103808772}) 2024-09-16T04:01:48.843 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.840+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.843 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.840+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2353822125}) 2024-09-16T04:01:48.843 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.840+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4147122464}) 2024-09-16T04:01:48.844 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.840+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.844 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.840+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=784280732}) 2024-09-16T04:01:48.844 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.840+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2220303539}) 2024-09-16T04:01:48.845 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.840+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.845 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.840+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=784280732}) 2024-09-16T04:01:48.845 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.840+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1244368457}) 2024-09-16T04:01:48.846 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.844+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.846 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.844+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3630923629}) 2024-09-16T04:01:48.846 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.844+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3913833566}) 2024-09-16T04:01:48.846 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.844+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.847 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.844+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3630923629}) 2024-09-16T04:01:48.847 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.844+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1824861001}) 2024-09-16T04:01:48.848 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.844+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.848 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.844+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3433531147}) 2024-09-16T04:01:48.848 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.844+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2877525733}) 2024-09-16T04:01:48.848 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.844+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.848 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.844+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3433531147}) 2024-09-16T04:01:48.848 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.844+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=490989821}) 2024-09-16T04:01:48.850 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.848+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.850 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.848+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2815558960}) 2024-09-16T04:01:48.850 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.848+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1199779448}) 2024-09-16T04:01:48.850 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.848+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.850 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.848+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2815558960}) 2024-09-16T04:01:48.850 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.848+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3381619954}) 2024-09-16T04:01:48.852 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.848+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.852 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.848+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1228321762}) 2024-09-16T04:01:48.852 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.848+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3346559745}) 2024-09-16T04:01:48.852 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.848+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.852 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.848+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1228321762}) 2024-09-16T04:01:48.852 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.848+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1354726958}) 2024-09-16T04:01:48.854 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.852+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.854 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.852+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=524492622}) 2024-09-16T04:01:48.854 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.852+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2858708592}) 2024-09-16T04:01:48.854 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.852+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.854 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.852+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=524492622}) 2024-09-16T04:01:48.854 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.852+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1030899028}) 2024-09-16T04:01:48.855 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.852+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.855 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.852+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3931651443}) 2024-09-16T04:01:48.855 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.852+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=602274324}) 2024-09-16T04:01:48.855 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.852+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.855 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.852+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3931651443}) 2024-09-16T04:01:48.856 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.852+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=646989345}) 2024-09-16T04:01:48.857 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.852+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.857 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.852+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2795027948}) 2024-09-16T04:01:48.857 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.852+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3577452082}) 2024-09-16T04:01:48.857 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.852+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.857 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.852+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2795027948}) 2024-09-16T04:01:48.857 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.852+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=508218619}) 2024-09-16T04:01:48.859 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.856+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.859 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.856+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=523371572}) 2024-09-16T04:01:48.859 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.856+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3736271668}) 2024-09-16T04:01:48.859 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.856+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.859 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.856+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=523371572}) 2024-09-16T04:01:48.859 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.856+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3919952166}) 2024-09-16T04:01:48.860 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.856+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.860 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.856+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4252910489}) 2024-09-16T04:01:48.860 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.856+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2165062540}) 2024-09-16T04:01:48.861 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.856+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.861 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.856+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4252910489}) 2024-09-16T04:01:48.861 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.856+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2232594026}) 2024-09-16T04:01:48.863 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.860+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.863 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.860+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3169615415}) 2024-09-16T04:01:48.863 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.860+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2336940349}) 2024-09-16T04:01:48.864 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.860+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.864 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.860+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3169615415}) 2024-09-16T04:01:48.864 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.860+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=396705013}) 2024-09-16T04:01:48.864 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.860+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.864 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.860+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3000936143}) 2024-09-16T04:01:48.864 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.860+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=94076306}) 2024-09-16T04:01:48.864 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.860+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.864 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.860+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3000936143}) 2024-09-16T04:01:48.864 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.860+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1474570901}) 2024-09-16T04:01:48.865 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.860+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.865 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.860+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=918550366}) 2024-09-16T04:01:48.865 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.860+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2149568763}) 2024-09-16T04:01:48.865 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.860+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.865 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.860+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=918550366}) 2024-09-16T04:01:48.865 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.860+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3846718303}) 2024-09-16T04:01:48.866 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.864+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.866 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.864+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1544365667}) 2024-09-16T04:01:48.867 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.864+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=906849069}) 2024-09-16T04:01:48.867 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.864+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.867 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.864+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1544365667}) 2024-09-16T04:01:48.867 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.864+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=572340754}) 2024-09-16T04:01:48.868 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.864+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.868 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.864+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1913021880}) 2024-09-16T04:01:48.868 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.864+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1752069290}) 2024-09-16T04:01:48.868 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.864+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.868 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.864+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1913021880}) 2024-09-16T04:01:48.868 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.864+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3424736609}) 2024-09-16T04:01:48.870 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.868+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.870 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.868+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3438748727}) 2024-09-16T04:01:48.870 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.868+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2846869485}) 2024-09-16T04:01:48.870 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.868+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.870 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.868+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3438748727}) 2024-09-16T04:01:48.870 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.868+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=909158729}) 2024-09-16T04:01:48.871 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.868+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.872 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.868+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=507055639}) 2024-09-16T04:01:48.872 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.868+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2999534825}) 2024-09-16T04:01:48.872 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.868+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.872 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.868+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=507055639}) 2024-09-16T04:01:48.872 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.868+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3556066353}) 2024-09-16T04:01:48.873 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.868+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.873 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.872+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2610156756}) 2024-09-16T04:01:48.873 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.872+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4156344}) 2024-09-16T04:01:48.873 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.872+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.873 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.872+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2610156756}) 2024-09-16T04:01:48.873 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.872+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4273999152}) 2024-09-16T04:01:48.875 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.872+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.875 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.872+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1502499314}) 2024-09-16T04:01:48.875 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.872+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3648163238}) 2024-09-16T04:01:48.875 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.872+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.875 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.872+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1502499314}) 2024-09-16T04:01:48.875 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.872+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1477705062}) 2024-09-16T04:01:48.876 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.872+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.877 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.872+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3837744017}) 2024-09-16T04:01:48.877 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.872+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=540938663}) 2024-09-16T04:01:48.877 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.872+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.877 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.872+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3837744017}) 2024-09-16T04:01:48.877 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.872+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=300325317}) 2024-09-16T04:01:48.878 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.876+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.878 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.876+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=612878857}) 2024-09-16T04:01:48.878 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.876+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2843301443}) 2024-09-16T04:01:48.878 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.876+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.878 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.876+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=612878857}) 2024-09-16T04:01:48.879 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.876+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3884279447}) 2024-09-16T04:01:48.880 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.876+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.880 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.876+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3161143883}) 2024-09-16T04:01:48.880 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.876+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1595795510}) 2024-09-16T04:01:48.880 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.876+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.880 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.876+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3161143883}) 2024-09-16T04:01:48.880 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.876+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3404649986}) 2024-09-16T04:01:48.883 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.880+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.883 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.880+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3059081989}) 2024-09-16T04:01:48.883 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.880+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=372612296}) 2024-09-16T04:01:48.883 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.880+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.883 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.880+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3059081989}) 2024-09-16T04:01:48.883 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.880+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3420375431}) 2024-09-16T04:01:48.885 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.880+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.885 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.880+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1026938951}) 2024-09-16T04:01:48.885 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.880+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1958496902}) 2024-09-16T04:01:48.885 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.880+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.885 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.880+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1026938951}) 2024-09-16T04:01:48.885 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.880+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=970293804}) 2024-09-16T04:01:48.887 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.884+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.887 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.884+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3663624748}) 2024-09-16T04:01:48.887 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.884+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3824446419}) 2024-09-16T04:01:48.887 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.884+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.887 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.884+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3663624748}) 2024-09-16T04:01:48.887 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.884+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1833905302}) 2024-09-16T04:01:48.889 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.884+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.889 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.884+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1569198531}) 2024-09-16T04:01:48.889 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.884+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3137413128}) 2024-09-16T04:01:48.889 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.884+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.889 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.884+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1569198531}) 2024-09-16T04:01:48.889 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.884+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=484153068}) 2024-09-16T04:01:48.890 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.888+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.890 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.888+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=718966176}) 2024-09-16T04:01:48.891 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.888+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1112351424}) 2024-09-16T04:01:48.891 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.888+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.891 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.888+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=718966176}) 2024-09-16T04:01:48.891 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.888+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2874078592}) 2024-09-16T04:01:48.892 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.888+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.892 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.888+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1465499132}) 2024-09-16T04:01:48.892 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.888+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=794608332}) 2024-09-16T04:01:48.892 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.888+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.892 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.888+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1465499132}) 2024-09-16T04:01:48.892 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.888+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=80124150}) 2024-09-16T04:01:48.893 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.888+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.893 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.888+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2266920}) 2024-09-16T04:01:48.893 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.888+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1367379194}) 2024-09-16T04:01:48.893 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.888+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.893 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.888+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2266920}) 2024-09-16T04:01:48.893 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.888+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=280411417}) 2024-09-16T04:01:48.894 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.892+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.894 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.892+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1633369462}) 2024-09-16T04:01:48.894 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.892+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=745035633}) 2024-09-16T04:01:48.894 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.892+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.894 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.892+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1633369462}) 2024-09-16T04:01:48.894 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.892+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1158370235}) 2024-09-16T04:01:48.896 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.892+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.896 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.892+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=77,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=1} crc {logm=2281975853,mdsmap=4088524775,mgr=460109776,mgr_command_descs=2107673600,mgr_metadata=1367272202,mgrstat=3577819238,mon_config_key=4067097899}) 2024-09-16T04:01:48.896 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.892+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=75,mdsmap=3,mgr=7,mgr_metadata=1,mgrstat=10,mon_config_key=4} crc {logm=351123849,mdsmap=4088524775,mgr=1483503728,mgr_metadata=1367272202,mgrstat=2094374808,mon_config_key=1804595475}) 2024-09-16T04:01:48.896 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.892+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.896 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.892+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=77,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=1} crc {logm=2281975853,mdsmap=4088524775,mgr=460109776,mgr_command_descs=2107673600,mgr_metadata=1367272202,mgrstat=3577819238,mon_config_key=4067097899}) 2024-09-16T04:01:48.896 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.892+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=74,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=4} crc {logm=448336530,mdsmap=4088524775,mgr=366503433,mgr_command_descs=2107673600,mgr_metadata=1367272202,mgrstat=2094374808,mon_config_key=2532506080}) 2024-09-16T04:01:48.898 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.896+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.898 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.896+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mon_config_key=9,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=40,osdmap=41} crc {mon_config_key=3254350870,monmap=3937513981,osd_metadata=4182959540,osd_pg_creating=1987194041,osd_snap=2227794264,osdmap=4182494446}) 2024-09-16T04:01:48.898 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.896+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mon_config_key=10,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=41,osdmap=39} crc {mon_config_key=1439144890,monmap=3937513981,osd_metadata=4182959540,osd_pg_creating=1987194041,osd_snap=2805642534,osdmap=3383809920}) 2024-09-16T04:01:48.898 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.896+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.898 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.896+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mon_config_key=9,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=40,osdmap=41} crc {mon_config_key=3254350870,monmap=3937513981,osd_metadata=4182959540,osd_pg_creating=1987194041,osd_snap=2227794264,osdmap=4182494446}) 2024-09-16T04:01:48.898 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.896+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mon_config_key=10,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=39,osdmap=41} crc {mon_config_key=1439144890,monmap=3937513981,osd_metadata=4182959540,osd_pg_creating=1987194041,osd_snap=2565802976,osdmap=148324913}) 2024-09-16T04:01:48.900 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.896+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.900 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.896+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=383197862}) 2024-09-16T04:01:48.900 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.896+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3459125385}) 2024-09-16T04:01:48.900 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.896+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.900 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.896+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=383197862}) 2024-09-16T04:01:48.900 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.896+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2138546492}) 2024-09-16T04:01:48.915 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.912+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.915 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.912+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=90} crc {osdmap=3804324808}) 2024-09-16T04:01:48.915 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.912+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=90} crc {osdmap=2324012545}) 2024-09-16T04:01:48.915 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.912+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:01:48.915 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.912+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=90} crc {osdmap=3804324808}) 2024-09-16T04:01:48.915 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:01:48.912+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=91} crc {osdmap=3003223437}) 2024-09-16T04:01:49.643 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T04:01:49.643 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:01:49.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.772+0000 7f3069ef3640 10 monclient: get_monmap_and_config 2024-09-16T04:01:49.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.772+0000 7f3069ef3640 10 monclient: build_initial_monmap 2024-09-16T04:01:49.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.772+0000 7f3069ef3640 10 monclient: monmap: 2024-09-16T04:01:49.776 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:01:49.776 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:01:49.776 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:01:49.774816+0000 2024-09-16T04:01:49.777 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:01:49.774816+0000 2024-09-16T04:01:49.777 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:01:49.777 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:01:49.777 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:01:49.777 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:01:49.777 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:01:49.777 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:01:49.777 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:01:49.777 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:01:49.779 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.772+0000 7f3069ef3640 1 Processor -- start 2024-09-16T04:01:49.779 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.772+0000 7f3069ef3640 1 -- start start 2024-09-16T04:01:49.779 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.772+0000 7f3069ef3640 10 monclient: init 2024-09-16T04:01:49.779 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.772+0000 7f3069ef3640 10 monclient: _reopen_session rank -1 2024-09-16T04:01:49.779 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.772+0000 7f3069ef3640 10 monclient: _add_conns ranks=[3,1,2,0,4] 2024-09-16T04:01:49.779 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.776+0000 7f3069ef3640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f30640a39b0 0x7f3064133b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:01:49.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.776+0000 7f3069ef3640 10 monclient(hunting): picked mon.noname-a con 0x7f30640a39b0 addr v2:172.21.15.42:3300/0 2024-09-16T04:01:49.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.776+0000 7f3069ef3640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f30640a3d90 addr v1:172.21.15.154:6789/0 2024-09-16T04:01:49.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.776+0000 7f3069ef3640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3064137710 0x7f306413f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:01:49.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.776+0000 7f3069ef3640 10 monclient(hunting): picked mon.noname-b con 0x7f3064137710 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:01:49.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.776+0000 7f3069ef3640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f306409c140 con 0x7f30640a3d90 2024-09-16T04:01:49.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.776+0000 7f3069ef3640 10 monclient(hunting): start opening mon connection 2024-09-16T04:01:49.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.776+0000 7f3069ef3640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f306409c420 con 0x7f30640a39b0 2024-09-16T04:01:49.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.776+0000 7f3069ef3640 10 monclient(hunting): start opening mon connection 2024-09-16T04:01:49.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.776+0000 7f3069ef3640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f306406dd60 con 0x7f3064137710 2024-09-16T04:01:49.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.776+0000 7f3069ef3640 10 monclient(hunting): _renew_subs 2024-09-16T04:01:49.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.776+0000 7f3069ef3640 10 monclient(hunting): authenticate will time out at 2156.744171s 2024-09-16T04:01:49.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.776+0000 7f30637fe640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f30640a39b0 0x7f3064133b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:01:49.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.776+0000 7f3063fff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3064137710 0x7f306413f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:01:49.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.776+0000 7f30637fe640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f30640a39b0 0x7f3064133b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:35322/0 (socket says 172.21.15.42:35322) 2024-09-16T04:01:49.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.776+0000 7f30637fe640 1 -- 172.21.15.42:0/3895957644 learned_addr learned my addr 172.21.15.42:0/3895957644 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:01:49.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.776+0000 7f30637fe640 10 monclient(hunting): get_auth_request con 0x7f30640a39b0 auth_method 0 2024-09-16T04:01:49.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.776+0000 7f30637fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:01:49.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.776+0000 7f30637fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:01:49.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.776+0000 7f30637fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:01:49.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.776+0000 7f3063fff640 10 monclient(hunting): get_auth_request con 0x7f3064137710 auth_method 0 2024-09-16T04:01:49.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.776+0000 7f3063fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:01:49.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.776+0000 7f3063fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:01:49.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.776+0000 7f3063fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:01:49.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.776+0000 7f3063fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:01:49.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.776+0000 7f3063fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:01:49.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.776+0000 7f3063fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:01:49.782 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.776+0000 7f30637fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:01:49.782 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.776+0000 7f30637fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:01:49.782 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.776+0000 7f30637fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:01:49.782 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.776+0000 7f3063fff640 10 monclient(hunting): handle_auth_done global_id 954116 payload 274 2024-09-16T04:01:49.782 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.776+0000 7f3063fff640 1 -- 172.21.15.42:0/3895957644 >> v2:172.21.15.42:3300/0 conn(0x7f30640a39b0 msgr2=0x7f3064133b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:01:49.782 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.776+0000 7f3063fff640 1 --2- 172.21.15.42:0/3895957644 >> v2:172.21.15.42:3300/0 conn(0x7f30640a39b0 0x7f3064133b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:49.783 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.776+0000 7f3063fff640 1 -- 172.21.15.42:0/3895957644 >> v1:172.21.15.154:6789/0 conn(0x7f30640a3d90 legacy=0x7f3064137000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:01:49.783 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.776+0000 7f3063fff640 10 monclient: _finish_hunting 0 2024-09-16T04:01:49.783 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.776+0000 7f3063fff640 1 monclient: found mon.noname-b 2024-09-16T04:01:49.783 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.776+0000 7f3063fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:01:49.783 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.776+0000 7f3063fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T04:01:49.783 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.776+0000 7f3063fff640 1 -- 172.21.15.42:0/3895957644 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30640a5e20 con 0x7f3064137710 2024-09-16T04:01:49.783 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.776+0000 7f3063fff640 1 --2- 172.21.15.42:0/3895957644 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3064137710 0x7f306413f3a0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f30540097c0 tx=0x7f305401c320 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6aa31a4ac99b6ea9 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:01:49.795 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.792+0000 7f30627fc640 1 -- 172.21.15.42:0/3895957644 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30540b0070 con 0x7f3064137710 2024-09-16T04:01:49.795 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.792+0000 7f30627fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:01:49.795 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.792+0000 7f30627fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:01:49.796 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.792+0000 7f30627fc640 10 monclient: dump: 2024-09-16T04:01:49.796 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:01:49.796 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:01:49.796 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:49.796 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:49.796 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:01:49.796 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:01:49.796 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:01:49.796 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:01:49.796 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:01:49.796 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:01:49.796 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.792+0000 7f30627fc640 10 monclient: _finish_auth 0 2024-09-16T04:01:49.797 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.792+0000 7f30627fc640 10 monclient: _check_auth_tickets 2024-09-16T04:01:49.797 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.792+0000 7f30627fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:01:49.797 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.792+0000 7f30627fc640 1 -- 172.21.15.42:0/3895957644 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f30540ab600 con 0x7f3064137710 2024-09-16T04:01:49.797 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.792+0000 7f30627fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:01:49.797 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.792+0000 7f30627fc640 1 -- 172.21.15.42:0/3895957644 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30540ab9d0 con 0x7f3064137710 2024-09-16T04:01:49.797 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.792+0000 7f3069ef3640 5 monclient: authenticate success, global_id 954116 2024-09-16T04:01:49.797 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.792+0000 7f3069ef3640 10 monclient: get_monmap_and_config success 2024-09-16T04:01:49.797 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.792+0000 7f3069ef3640 4 set_mon_vals no callback set 2024-09-16T04:01:49.797 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.792+0000 7f30627fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:01:49.797 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.792+0000 7f30627fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:01:49.797 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.792+0000 7f30627fc640 10 monclient: dump: 2024-09-16T04:01:49.797 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:01:49.797 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:01:49.797 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:49.798 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:49.798 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:01:49.798 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:01:49.798 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:01:49.798 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:01:49.798 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:01:49.798 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:01:49.798 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.792+0000 7f3069ef3640 10 monclient: shutdown 2024-09-16T04:01:49.798 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.792+0000 7f3069ef3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:01:49.798 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.792+0000 7f3069ef3640 1 -- 172.21.15.42:0/3895957644 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3064137710 msgr2=0x7f306413f3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:01:49.798 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.792+0000 7f3069ef3640 1 --2- 172.21.15.42:0/3895957644 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3064137710 0x7f306413f3a0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f30540097c0 tx=0x7f305401c320 comp rx=0 tx=0).stop 2024-09-16T04:01:49.798 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.792+0000 7f3069ef3640 1 -- 172.21.15.42:0/3895957644 shutdown_connections 2024-09-16T04:01:49.798 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.792+0000 7f3069ef3640 1 --2- 172.21.15.42:0/3895957644 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3064137710 0x7f306413f3a0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:49.799 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.792+0000 7f3069ef3640 1 --2- 172.21.15.42:0/3895957644 >> v2:172.21.15.42:3300/0 conn(0x7f30640a39b0 0x7f3064133b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:49.799 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.792+0000 7f3069ef3640 1 -- 172.21.15.42:0/3895957644 >> 172.21.15.42:0/3895957644 conn(0x7f30640a9fb0 msgr2=0x7f30640aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:01:49.799 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.792+0000 7f3069ef3640 1 -- 172.21.15.42:0/3895957644 shutdown_connections 2024-09-16T04:01:49.799 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.792+0000 7f3069ef3640 1 -- 172.21.15.42:0/3895957644 wait complete. 2024-09-16T04:01:49.803 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.792+0000 7f3069ef3640 10 monclient: build_initial_monmap 2024-09-16T04:01:49.803 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.792+0000 7f3069ef3640 10 monclient: monmap: 2024-09-16T04:01:49.803 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:01:49.803 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:01:49.803 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:01:49.803 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:01:49.803 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:01:49.803 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:01:49.803 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:01:49.803 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:01:49.803 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:01:49.803 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:01:49.803 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.792+0000 7f3069ef3640 1 librados: starting msgr at 2024-09-16T04:01:49.804 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.792+0000 7f3069ef3640 1 librados: starting objecter 2024-09-16T04:01:49.804 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.792+0000 7f3069ef3640 1 Processor -- start 2024-09-16T04:01:49.804 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.792+0000 7f3069ef3640 1 -- start start 2024-09-16T04:01:49.804 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.792+0000 7f3069ef3640 1 librados: setting wanted keys 2024-09-16T04:01:49.804 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.792+0000 7f3069ef3640 1 librados: calling monclient init 2024-09-16T04:01:49.804 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.792+0000 7f3069ef3640 10 monclient: init 2024-09-16T04:01:49.804 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.792+0000 7f3069ef3640 10 monclient: _reopen_session rank -1 2024-09-16T04:01:49.804 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.792+0000 7f3069ef3640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:01:49.804 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.792+0000 7f3069ef3640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f30640a39b0 0x7f30640a2d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:01:49.804 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.792+0000 7f3069ef3640 10 monclient(hunting): picked mon.noname-c con 0x7f30640a39b0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:01:49.804 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.792+0000 7f3069ef3640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f30640a3d90 0x7f306409e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:01:49.804 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.792+0000 7f3069ef3640 10 monclient(hunting): picked mon.noname-b con 0x7f30640a3d90 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:01:49.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.792+0000 7f3069ef3640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3064137710 0x7f306409ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:01:49.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.792+0000 7f3069ef3640 10 monclient(hunting): picked mon.noname-a con 0x7f3064137710 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:01:49.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.792+0000 7f3069ef3640 10 monclient(hunting): start opening mon connection 2024-09-16T04:01:49.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.792+0000 7f3069ef3640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f3064142310 con 0x7f3064137710 2024-09-16T04:01:49.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.792+0000 7f3069ef3640 10 monclient(hunting): start opening mon connection 2024-09-16T04:01:49.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.792+0000 7f3069ef3640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f3064142490 con 0x7f30640a3d90 2024-09-16T04:01:49.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.792+0000 7f3069ef3640 10 monclient(hunting): start opening mon connection 2024-09-16T04:01:49.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.792+0000 7f3069ef3640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f3064142610 con 0x7f30640a39b0 2024-09-16T04:01:49.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.792+0000 7f3069ef3640 10 monclient(hunting): _renew_subs 2024-09-16T04:01:49.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.792+0000 7f3069ef3640 10 monclient(hunting): authenticate will time out at 2156.763780s 2024-09-16T04:01:49.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.792+0000 7f30637fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f30640a39b0 0x7f30640a2d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:01:49.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.792+0000 7f30637fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f30640a39b0 0x7f30640a2d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:49564/0 (socket says 172.21.15.42:49564) 2024-09-16T04:01:49.806 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.792+0000 7f30637fe640 1 -- 172.21.15.42:0/3882607458 learned_addr learned my addr 172.21.15.42:0/3882607458 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:01:49.806 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.792+0000 7f30637fe640 10 monclient(hunting): get_auth_request con 0x7f30640a39b0 auth_method 0 2024-09-16T04:01:49.806 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.792+0000 7f30637fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:01:49.806 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.792+0000 7f30637fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:01:49.806 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.792+0000 7f30637fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:01:49.806 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.792+0000 7f30637fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:01:49.806 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.792+0000 7f30637fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:01:49.806 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.796+0000 7f30637fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:01:49.806 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.796+0000 7f3062ffd640 1 --2- 172.21.15.42:0/3882607458 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f30640a3d90 0x7f306409e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:01:49.806 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.796+0000 7f3063fff640 1 --2- 172.21.15.42:0/3882607458 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3064137710 0x7f306409ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:01:49.806 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.796+0000 7f3062ffd640 10 monclient(hunting): get_auth_request con 0x7f30640a3d90 auth_method 0 2024-09-16T04:01:49.806 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.796+0000 7f3062ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:01:49.806 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.796+0000 7f3062ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:01:49.807 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.796+0000 7f3062ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:01:49.807 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.796+0000 7f30637fe640 10 monclient(hunting): handle_auth_done global_id 954119 payload 824 2024-09-16T04:01:49.807 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.796+0000 7f30637fe640 1 -- 172.21.15.42:0/3882607458 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f30640a3d90 msgr2=0x7f306409e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:01:49.807 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.796+0000 7f30637fe640 1 --2- 172.21.15.42:0/3882607458 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f30640a3d90 0x7f306409e670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:49.807 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.796+0000 7f30637fe640 1 -- 172.21.15.42:0/3882607458 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3064137710 msgr2=0x7f306409ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:01:49.807 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.796+0000 7f30637fe640 1 --2- 172.21.15.42:0/3882607458 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3064137710 0x7f306409ebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:49.807 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.796+0000 7f30637fe640 10 monclient: _finish_hunting 0 2024-09-16T04:01:49.807 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.796+0000 7f30637fe640 1 monclient: found mon.noname-c 2024-09-16T04:01:49.807 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.796+0000 7f30637fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:01:49.807 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.796+0000 7f30637fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T04:01:49.807 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.796+0000 7f30637fe640 1 -- 172.21.15.42:0/3882607458 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f306406f680 con 0x7f30640a39b0 2024-09-16T04:01:49.807 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.796+0000 7f3063fff640 10 monclient: get_auth_request con 0x7f3064137710 auth_method 0 2024-09-16T04:01:49.807 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.796+0000 7f3063fff640 1 --2- 172.21.15.42:0/3882607458 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3064137710 0x7f306409ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T04:01:49.808 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.796+0000 7f30637fe640 1 --2- 172.21.15.42:0/3882607458 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f30640a39b0 0x7f30640a2d80 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f305c00efc0 tx=0x7f305c00c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:01:49.808 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.796+0000 7f305bfff640 10 client.?.objecter ms_handle_connect 0x7f30640a39b0 2024-09-16T04:01:49.808 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.796+0000 7f305bfff640 10 client.?.objecter resend_mon_ops 2024-09-16T04:01:49.808 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.796+0000 7f305bfff640 1 -- 172.21.15.42:0/3882607458 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f305c00cda0 con 0x7f30640a39b0 2024-09-16T04:01:49.808 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.796+0000 7f305bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:01:49.808 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.796+0000 7f305bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:01:49.808 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.796+0000 7f305bfff640 10 monclient: dump: 2024-09-16T04:01:49.808 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:01:49.808 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:01:49.808 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:49.808 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:49.808 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:01:49.808 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:01:49.809 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:01:49.809 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:01:49.809 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:01:49.809 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:01:49.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.796+0000 7f305bfff640 10 monclient: _finish_auth 0 2024-09-16T04:01:49.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.796+0000 7f305bfff640 10 monclient: _check_auth_tickets 2024-09-16T04:01:49.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.796+0000 7f305bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:01:49.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.796+0000 7f3069ef3640 5 monclient: authenticate success, global_id 954119 2024-09-16T04:01:49.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.796+0000 7f305bfff640 1 -- 172.21.15.42:0/3882607458 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f305c019070 con 0x7f30640a39b0 2024-09-16T04:01:49.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.796+0000 7f3069ef3640 10 monclient: _renew_subs 2024-09-16T04:01:49.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.796+0000 7f3069ef3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:01:49.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.796+0000 7f3069ef3640 1 -- 172.21.15.42:0/3882607458 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3064132ae0 con 0x7f30640a39b0 2024-09-16T04:01:49.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.796+0000 7f305bfff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:01:49.810 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.796+0000 7f305bfff640 1 -- 172.21.15.42:0/3882607458 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f305c014680 con 0x7f30640a39b0 2024-09-16T04:01:49.810 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.796+0000 7f305bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:01:49.810 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.796+0000 7f305bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:01:49.810 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.796+0000 7f305bfff640 10 monclient: dump: 2024-09-16T04:01:49.810 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:01:49.810 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:01:49.810 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:49.810 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:49.810 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:01:49.810 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:01:49.810 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:01:49.810 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:01:49.810 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:01:49.811 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:01:49.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.796+0000 7f3061ffb640 4 set_mon_vals no callback set 2024-09-16T04:01:49.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.796+0000 7f3069ef3640 10 client.954119.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:01:49.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.796+0000 7f3069ef3640 10 monclient: _renew_subs 2024-09-16T04:01:49.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.796+0000 7f3069ef3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:01:49.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.796+0000 7f3069ef3640 1 -- 172.21.15.42:0/3882607458 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f30640a2600 con 0x7f30640a39b0 2024-09-16T04:01:49.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.796+0000 7f3069ef3640 1 librados: init done 2024-09-16T04:01:49.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.796+0000 7f305bfff640 1 -- 172.21.15.42:0/3882607458 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f305c012050 con 0x7f30640a39b0 2024-09-16T04:01:49.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.796+0000 7f3069ef3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:01:49.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.796+0000 7f3069ef3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:01:49.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.796+0000 7f3069ef3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:01:49.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.796+0000 7f3069ef3640 1 -- 172.21.15.42:0/3882607458 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3064133b80 con 0x7f30640a39b0 2024-09-16T04:01:49.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.796+0000 7f305bfff640 1 --2- 172.21.15.42:0/3882607458 >> v2:172.21.15.154:6812/254845342 conn(0x7f3044037810 0x7f3044039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:01:49.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.796+0000 7f305bfff640 1 -- 172.21.15.42:0/3882607458 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(601..601 src has 451..601) ==== 6252+0+0 (secure 0 0 0) 0x7f305c049720 con 0x7f30640a39b0 2024-09-16T04:01:49.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.796+0000 7f305bfff640 10 client.954119.objecter ms_dispatch 0x7f3064000ba0 osd_map(601..601 src has 451..601) 2024-09-16T04:01:49.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.796+0000 7f305bfff640 3 client.954119.objecter handle_osd_map got epochs [601,601] > 0 2024-09-16T04:01:49.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.796+0000 7f305bfff640 3 client.954119.objecter handle_osd_map decoding full epoch 601 2024-09-16T04:01:49.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.796+0000 7f305bfff640 20 client.954119.objecter dump_active .. 0 homeless 2024-09-16T04:01:49.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.796+0000 7f3062ffd640 1 --2- 172.21.15.42:0/3882607458 >> v2:172.21.15.154:6812/254845342 conn(0x7f3044037810 0x7f3044039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:01:49.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.796+0000 7f3062ffd640 10 monclient: get_auth_request con 0x7f3044037810 auth_method 0 2024-09-16T04:01:49.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.796+0000 7f3062ffd640 1 --2- 172.21.15.42:0/3882607458 >> v2:172.21.15.154:6812/254845342 conn(0x7f3044037810 0x7f3044039cd0 secure :-1 s=READY pgs=3769 cs=0 l=1 rev1=1 crypto rx=0x7f3064131950 tx=0x7f304c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:01:49.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.796+0000 7f305bfff640 10 client.954119.objecter ms_handle_connect 0x7f3044037810 2024-09-16T04:01:49.813 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.800+0000 7f305bfff640 1 -- 172.21.15.42:0/3882607458 <== mon.2 v2:172.21.15.42:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f305c010040 con 0x7f30640a39b0 2024-09-16T04:01:49.813 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.800+0000 7f305bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:01:49.813 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:49.800+0000 7f305bfff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:01:50.029 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.024+0000 7f3069ef3640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:01:50.029 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.024+0000 7f3069ef3640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:01:50.029 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.024+0000 7f3069ef3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:01:50.029 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.024+0000 7f3069ef3640 1 -- 172.21.15.42:0/3882607458 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f30640a5750 con 0x7f30640a39b0 2024-09-16T04:01:50.030 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.028+0000 7f305bfff640 1 -- 172.21.15.42:0/3882607458 <== mon.2 v2:172.21.15.42:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7f305c0137e0 con 0x7f30640a39b0 2024-09-16T04:01:50.030 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.028+0000 7f305bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:01:50.030 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.028+0000 7f305bfff640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:01:50.030 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":178,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:01:50.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.028+0000 7f3069ef3640 10 librados: watch_flush enter 2024-09-16T04:01:50.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.028+0000 7f3069ef3640 10 librados: watch_flush exit 2024-09-16T04:01:50.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.028+0000 7f3069ef3640 20 client.954119.objecter shutdown clearing up homeless session... 2024-09-16T04:01:50.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.028+0000 7f3069ef3640 10 client.954119.objecter successfully canceled tick 2024-09-16T04:01:50.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.028+0000 7f3069ef3640 1 -- 172.21.15.42:0/3882607458 >> v2:172.21.15.154:6812/254845342 conn(0x7f3044037810 msgr2=0x7f3044039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:01:50.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.028+0000 7f3069ef3640 1 --2- 172.21.15.42:0/3882607458 >> v2:172.21.15.154:6812/254845342 conn(0x7f3044037810 0x7f3044039cd0 secure :-1 s=READY pgs=3769 cs=0 l=1 rev1=1 crypto rx=0x7f3064131950 tx=0x7f304c009040 comp rx=0 tx=0).stop 2024-09-16T04:01:50.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.028+0000 7f3069ef3640 10 monclient: shutdown 2024-09-16T04:01:50.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.028+0000 7f3069ef3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:01:50.034 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.028+0000 7f3069ef3640 1 -- 172.21.15.42:0/3882607458 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f30640a39b0 msgr2=0x7f30640a2d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:01:50.034 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.028+0000 7f3069ef3640 1 --2- 172.21.15.42:0/3882607458 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f30640a39b0 0x7f30640a2d80 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f305c00efc0 tx=0x7f305c00c490 comp rx=0 tx=0).stop 2024-09-16T04:01:50.034 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.028+0000 7f3069ef3640 1 -- 172.21.15.42:0/3882607458 shutdown_connections 2024-09-16T04:01:50.034 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.028+0000 7f3069ef3640 1 --2- 172.21.15.42:0/3882607458 >> v2:172.21.15.154:6812/254845342 conn(0x7f3044037810 0x7f3044039cd0 unknown :-1 s=CLOSED pgs=3769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:50.034 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.028+0000 7f3069ef3640 1 --2- 172.21.15.42:0/3882607458 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3064137710 0x7f306409ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:50.034 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.028+0000 7f3069ef3640 1 --2- 172.21.15.42:0/3882607458 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f30640a3d90 0x7f306409e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:50.034 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.028+0000 7f3069ef3640 1 --2- 172.21.15.42:0/3882607458 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f30640a39b0 0x7f30640a2d80 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:50.034 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.028+0000 7f3069ef3640 1 -- 172.21.15.42:0/3882607458 >> 172.21.15.42:0/3882607458 conn(0x7f30640a9fb0 msgr2=0x7f30640a89d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:01:50.034 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.028+0000 7f3069ef3640 1 -- 172.21.15.42:0/3882607458 shutdown_connections 2024-09-16T04:01:50.034 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.028+0000 7f3069ef3640 1 -- 172.21.15.42:0/3882607458 wait complete. 2024-09-16T04:01:50.034 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.028+0000 7f3069ef3640 1 librados: shutdown 2024-09-16T04:01:50.050 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T04:01:50.050 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T04:01:50.050 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T04:01:50.050 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T04:01:50.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.164+0000 7fed22139640 10 monclient: get_monmap_and_config 2024-09-16T04:01:50.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.164+0000 7fed22139640 10 monclient: build_initial_monmap 2024-09-16T04:01:50.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.164+0000 7fed22139640 10 monclient: monmap: 2024-09-16T04:01:50.167 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:01:50.167 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:01:50.167 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:01:50.165374+0000 2024-09-16T04:01:50.167 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:01:50.165374+0000 2024-09-16T04:01:50.167 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:01:50.167 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:01:50.167 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:01:50.167 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:01:50.167 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:01:50.167 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:01:50.168 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:01:50.168 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:01:50.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.164+0000 7fed22139640 1 Processor -- start 2024-09-16T04:01:50.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.164+0000 7fed22139640 1 -- start start 2024-09-16T04:01:50.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.164+0000 7fed22139640 10 monclient: init 2024-09-16T04:01:50.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.164+0000 7fed22139640 10 monclient: _reopen_session rank -1 2024-09-16T04:01:50.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.164+0000 7fed22139640 10 monclient: _add_conns ranks=[0,3,4,2,1] 2024-09-16T04:01:50.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.164+0000 7fed22139640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fed1c0a1330 addr v1:172.21.15.42:6789/0 2024-09-16T04:01:50.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.164+0000 7fed22139640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fed1c0a1710 0x7fed1c0a1b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:01:50.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.164+0000 7fed22139640 10 monclient(hunting): picked mon.noname-a con 0x7fed1c0a1710 addr v2:172.21.15.42:3300/0 2024-09-16T04:01:50.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.164+0000 7fed22139640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fed1c1393a0 0x7fed1c13b790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:01:50.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.164+0000 7fed22139640 10 monclient(hunting): picked mon.noname-c con 0x7fed1c1393a0 addr v2:172.21.15.154:3300/0 2024-09-16T04:01:50.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.164+0000 7fed22139640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fed1c09e4c0 con 0x7fed1c0a1330 2024-09-16T04:01:50.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.164+0000 7fed22139640 10 monclient(hunting): start opening mon connection 2024-09-16T04:01:50.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.164+0000 7fed22139640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7fed1c09edf0 con 0x7fed1c0a1710 2024-09-16T04:01:50.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.164+0000 7fed22139640 10 monclient(hunting): start opening mon connection 2024-09-16T04:01:50.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.164+0000 7fed22139640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fed1c09cd60 con 0x7fed1c1393a0 2024-09-16T04:01:50.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.164+0000 7fed22139640 10 monclient(hunting): _renew_subs 2024-09-16T04:01:50.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.164+0000 7fed22139640 10 monclient(hunting): authenticate will time out at 2157.134699s 2024-09-16T04:01:50.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.164+0000 7fed1affd640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fed1c0a1710 0x7fed1c0a1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:01:50.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.164+0000 7fed1b7fe640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fed1c0a1330 0x7fed1c09f180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:58948/0 (socket says 172.21.15.42:58948) 2024-09-16T04:01:50.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.164+0000 7fed1b7fe640 1 -- 172.21.15.42:0/3459537963 learned_addr learned my addr 172.21.15.42:0/3459537963 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T04:01:50.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.164+0000 7fed1affd640 10 monclient(hunting): get_auth_request con 0x7fed1c0a1710 auth_method 0 2024-09-16T04:01:50.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.164+0000 7fed1affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:01:50.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.164+0000 7fed1affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:01:50.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.164+0000 7fed1affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:01:50.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.164+0000 7fed1affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:01:50.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.164+0000 7fed1affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:01:50.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.164+0000 7fed1affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:01:50.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.164+0000 7fed1bfff640 1 --2- 172.21.15.42:0/3459537963 >> v2:172.21.15.154:3300/0 conn(0x7fed1c1393a0 0x7fed1c13b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:01:50.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.164+0000 7fed1bfff640 10 monclient(hunting): get_auth_request con 0x7fed1c1393a0 auth_method 0 2024-09-16T04:01:50.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.164+0000 7fed1bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:01:50.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.164+0000 7fed1bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:01:50.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.164+0000 7fed1bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:01:50.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.164+0000 7fed1a7fc640 1 -- 172.21.15.42:0/3459537963 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2727406340 0 0) 0x7fed1c09e4c0 con 0x7fed1c0a1330 2024-09-16T04:01:50.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.164+0000 7fed1affd640 10 monclient(hunting): handle_auth_done global_id 825423 payload 274 2024-09-16T04:01:50.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.164+0000 7fed1affd640 1 -- 172.21.15.42:0/3459537963 >> v2:172.21.15.154:3300/0 conn(0x7fed1c1393a0 msgr2=0x7fed1c13b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:01:50.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.164+0000 7fed1affd640 1 --2- 172.21.15.42:0/3459537963 >> v2:172.21.15.154:3300/0 conn(0x7fed1c1393a0 0x7fed1c13b790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:50.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.164+0000 7fed1affd640 1 -- 172.21.15.42:0/3459537963 >> v1:172.21.15.42:6789/0 conn(0x7fed1c0a1330 legacy=0x7fed1c09f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:01:50.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.164+0000 7fed1affd640 10 monclient: _finish_hunting 0 2024-09-16T04:01:50.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.164+0000 7fed1affd640 1 monclient: found mon.noname-a 2024-09-16T04:01:50.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.164+0000 7fed1affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:01:50.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.164+0000 7fed1affd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:01:50.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.164+0000 7fed1affd640 1 -- 172.21.15.42:0/3459537963 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed1c0a2080 con 0x7fed1c0a1710 2024-09-16T04:01:50.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.164+0000 7fed1a7fc640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T04:01:50.181 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.168+0000 7fed1affd640 1 --2- 172.21.15.42:0/3459537963 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fed1c0a1710 0x7fed1c0a1b40 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fed1c09df50 tx=0x7fed1001c340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6bff85aaf944db74 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:01:50.181 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.168+0000 7fed1a7fc640 1 -- 172.21.15.42:0/3459537963 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed100b0070 con 0x7fed1c0a1710 2024-09-16T04:01:50.181 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.168+0000 7fed1a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:01:50.181 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.168+0000 7fed1a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:01:50.181 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.168+0000 7fed1a7fc640 10 monclient: dump: 2024-09-16T04:01:50.181 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:01:50.181 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:01:50.181 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:50.181 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:50.181 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:01:50.181 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:01:50.182 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:01:50.182 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:01:50.182 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:01:50.182 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:01:50.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.168+0000 7fed1a7fc640 10 monclient: _finish_auth 0 2024-09-16T04:01:50.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.168+0000 7fed1a7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:01:50.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.168+0000 7fed1a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:01:50.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.168+0000 7fed22139640 5 monclient: authenticate success, global_id 825423 2024-09-16T04:01:50.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.168+0000 7fed22139640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:01:50.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.168+0000 7fed1a7fc640 1 -- 172.21.15.42:0/3459537963 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fed100ab600 con 0x7fed1c0a1710 2024-09-16T04:01:50.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.168+0000 7fed1a7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:01:50.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.168+0000 7fed22139640 10 monclient: get_monmap_and_config success 2024-09-16T04:01:50.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.168+0000 7fed22139640 4 set_mon_vals no callback set 2024-09-16T04:01:50.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.168+0000 7fed1a7fc640 1 -- 172.21.15.42:0/3459537963 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed100ab9d0 con 0x7fed1c0a1710 2024-09-16T04:01:50.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.168+0000 7fed1a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:01:50.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.168+0000 7fed1a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:01:50.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.168+0000 7fed1a7fc640 10 monclient: dump: 2024-09-16T04:01:50.183 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:01:50.183 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:01:50.183 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:50.183 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:50.183 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:01:50.183 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:01:50.184 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:01:50.184 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:01:50.184 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:01:50.184 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:01:50.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.168+0000 7fed22139640 10 monclient: shutdown 2024-09-16T04:01:50.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.168+0000 7fed22139640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:01:50.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.168+0000 7fed22139640 1 -- 172.21.15.42:0/3459537963 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fed1c0a1710 msgr2=0x7fed1c0a1b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:01:50.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.168+0000 7fed22139640 1 --2- 172.21.15.42:0/3459537963 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fed1c0a1710 0x7fed1c0a1b40 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fed1c09df50 tx=0x7fed1001c340 comp rx=0 tx=0).stop 2024-09-16T04:01:50.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.168+0000 7fed22139640 1 -- 172.21.15.42:0/3459537963 shutdown_connections 2024-09-16T04:01:50.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.168+0000 7fed22139640 1 --2- 172.21.15.42:0/3459537963 >> v2:172.21.15.154:3300/0 conn(0x7fed1c1393a0 0x7fed1c13b790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:50.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.168+0000 7fed22139640 1 --2- 172.21.15.42:0/3459537963 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fed1c0a1710 0x7fed1c0a1b40 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:50.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.168+0000 7fed22139640 1 -- 172.21.15.42:0/3459537963 >> 172.21.15.42:0/3459537963 conn(0x7fed1c0a2420 msgr2=0x7fed1c0a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:01:50.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.168+0000 7fed22139640 1 -- 172.21.15.42:0/3459537963 shutdown_connections 2024-09-16T04:01:50.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.168+0000 7fed22139640 1 -- 172.21.15.42:0/3459537963 wait complete. 2024-09-16T04:01:50.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.168+0000 7fed22139640 10 monclient: build_initial_monmap 2024-09-16T04:01:50.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.168+0000 7fed22139640 10 monclient: monmap: 2024-09-16T04:01:50.185 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:01:50.185 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:01:50.185 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:01:50.185 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:01:50.185 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:01:50.185 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:01:50.186 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:01:50.186 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:01:50.186 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:01:50.186 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:01:50.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.168+0000 7fed22139640 1 librados: starting msgr at 2024-09-16T04:01:50.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.168+0000 7fed22139640 1 librados: starting objecter 2024-09-16T04:01:50.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.168+0000 7fed22139640 1 Processor -- start 2024-09-16T04:01:50.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.168+0000 7fed22139640 1 -- start start 2024-09-16T04:01:50.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.168+0000 7fed22139640 1 librados: setting wanted keys 2024-09-16T04:01:50.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.168+0000 7fed22139640 1 librados: calling monclient init 2024-09-16T04:01:50.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.168+0000 7fed22139640 10 monclient: init 2024-09-16T04:01:50.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.168+0000 7fed22139640 10 monclient: _reopen_session rank -1 2024-09-16T04:01:50.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.168+0000 7fed22139640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:01:50.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.168+0000 7fed22139640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fed1c0a1330 0x7fed1c1ca8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:01:50.187 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.168+0000 7fed22139640 10 monclient(hunting): picked mon.noname-a con 0x7fed1c0a1330 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:01:50.187 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.168+0000 7fed22139640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fed1c1393a0 0x7fed1c072050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:01:50.187 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.168+0000 7fed22139640 10 monclient(hunting): picked mon.noname-b con 0x7fed1c1393a0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:01:50.187 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.168+0000 7fed22139640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fed1c072590 0x7fed1c1c7320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:01:50.187 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.168+0000 7fed22139640 10 monclient(hunting): picked mon.noname-c con 0x7fed1c072590 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:01:50.187 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.168+0000 7fed22139640 10 monclient(hunting): start opening mon connection 2024-09-16T04:01:50.187 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.168+0000 7fed22139640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fed1c09bde0 con 0x7fed1c0a1330 2024-09-16T04:01:50.187 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.168+0000 7fed22139640 10 monclient(hunting): start opening mon connection 2024-09-16T04:01:50.187 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.168+0000 7fed22139640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fed1c09bc60 con 0x7fed1c1393a0 2024-09-16T04:01:50.187 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.168+0000 7fed22139640 10 monclient(hunting): start opening mon connection 2024-09-16T04:01:50.187 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.168+0000 7fed22139640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fed1c09bf60 con 0x7fed1c072590 2024-09-16T04:01:50.188 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.168+0000 7fed22139640 10 monclient(hunting): _renew_subs 2024-09-16T04:01:50.188 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.168+0000 7fed22139640 10 monclient(hunting): authenticate will time out at 2157.139766s 2024-09-16T04:01:50.188 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.168+0000 7fed1b7fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fed1c0a1330 0x7fed1c1ca8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:01:50.188 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.168+0000 7fed1bfff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fed1c072590 0x7fed1c1c7320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:01:50.188 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.168+0000 7fed1b7fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fed1c0a1330 0x7fed1c1ca8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:35360/0 (socket says 172.21.15.42:35360) 2024-09-16T04:01:50.188 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.168+0000 7fed1b7fe640 1 -- 172.21.15.42:0/1813515815 learned_addr learned my addr 172.21.15.42:0/1813515815 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:01:50.188 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.168+0000 7fed1b7fe640 10 monclient(hunting): get_auth_request con 0x7fed1c0a1330 auth_method 0 2024-09-16T04:01:50.188 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.168+0000 7fed1b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:01:50.189 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.168+0000 7fed1b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:01:50.189 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.168+0000 7fed1b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:01:50.189 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.168+0000 7fed1bfff640 10 monclient(hunting): get_auth_request con 0x7fed1c072590 auth_method 0 2024-09-16T04:01:50.189 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.168+0000 7fed1bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:01:50.189 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.168+0000 7fed1bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:01:50.189 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.168+0000 7fed1bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:01:50.189 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.172+0000 7fed1b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:01:50.190 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.172+0000 7fed1b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:01:50.190 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.172+0000 7fed1b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:01:50.190 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.172+0000 7fed1bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:01:50.190 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.172+0000 7fed1bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:01:50.190 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.172+0000 7fed1bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:01:50.190 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.172+0000 7fed1affd640 1 --2- 172.21.15.42:0/1813515815 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fed1c1393a0 0x7fed1c072050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:01:50.190 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.172+0000 7fed1affd640 10 monclient(hunting): get_auth_request con 0x7fed1c1393a0 auth_method 0 2024-09-16T04:01:50.191 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.172+0000 7fed1affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:01:50.191 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.172+0000 7fed1affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:01:50.191 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.172+0000 7fed1affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:01:50.191 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.172+0000 7fed1b7fe640 10 monclient(hunting): handle_auth_done global_id 825429 payload 824 2024-09-16T04:01:50.191 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.172+0000 7fed1b7fe640 1 -- 172.21.15.42:0/1813515815 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fed1c072590 msgr2=0x7fed1c1c7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:01:50.191 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.172+0000 7fed1b7fe640 1 --2- 172.21.15.42:0/1813515815 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fed1c072590 0x7fed1c1c7320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:50.191 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.172+0000 7fed1b7fe640 1 -- 172.21.15.42:0/1813515815 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fed1c1393a0 msgr2=0x7fed1c072050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:01:50.191 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.172+0000 7fed1b7fe640 1 --2- 172.21.15.42:0/1813515815 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fed1c1393a0 0x7fed1c072050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:50.191 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.172+0000 7fed1b7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:01:50.192 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.172+0000 7fed1b7fe640 1 monclient: found mon.noname-a 2024-09-16T04:01:50.192 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.172+0000 7fed1b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:01:50.192 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.172+0000 7fed1b7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:01:50.192 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.172+0000 7fed1b7fe640 1 -- 172.21.15.42:0/1813515815 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed1c133da0 con 0x7fed1c0a1330 2024-09-16T04:01:50.192 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.172+0000 7fed1affd640 1 --2- 172.21.15.42:0/1813515815 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fed1c1393a0 0x7fed1c072050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T04:01:50.192 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.172+0000 7fed1bfff640 1 --2- 172.21.15.42:0/1813515815 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fed1c072590 0x7fed1c1c7320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:01:50.192 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.172+0000 7fed1b7fe640 1 --2- 172.21.15.42:0/1813515815 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fed1c0a1330 0x7fed1c1ca8a0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fed0c00c550 tx=0x7fed0c00ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:01:50.192 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.172+0000 7fed03fff640 10 client.?.objecter ms_handle_connect 0x7fed1c0a1330 2024-09-16T04:01:50.193 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.172+0000 7fed03fff640 10 client.?.objecter resend_mon_ops 2024-09-16T04:01:50.193 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.172+0000 7fed03fff640 1 -- 172.21.15.42:0/1813515815 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed0c018070 con 0x7fed1c0a1330 2024-09-16T04:01:50.193 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.172+0000 7fed03fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:01:50.193 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.172+0000 7fed03fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:01:50.193 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.172+0000 7fed03fff640 10 monclient: dump: 2024-09-16T04:01:50.193 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:01:50.193 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:01:50.193 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:50.194 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:50.194 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:01:50.194 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:01:50.194 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:01:50.194 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:01:50.194 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:01:50.194 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:01:50.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.172+0000 7fed03fff640 10 monclient: _finish_auth 0 2024-09-16T04:01:50.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.172+0000 7fed03fff640 10 monclient: _check_auth_tickets 2024-09-16T04:01:50.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.172+0000 7fed03fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:01:50.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.172+0000 7fed03fff640 1 -- 172.21.15.42:0/1813515815 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fed0c0074e0 con 0x7fed1c0a1330 2024-09-16T04:01:50.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.172+0000 7fed03fff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:01:50.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.172+0000 7fed03fff640 1 -- 172.21.15.42:0/1813515815 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed0c0049f0 con 0x7fed1c0a1330 2024-09-16T04:01:50.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.172+0000 7fed03fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:01:50.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.172+0000 7fed03fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:01:50.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.172+0000 7fed03fff640 10 monclient: dump: 2024-09-16T04:01:50.195 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:01:50.196 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:01:50.196 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:50.196 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:50.196 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:01:50.196 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:01:50.196 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:01:50.196 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:01:50.196 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:01:50.197 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:01:50.197 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.172+0000 7fed19ffb640 4 set_mon_vals no callback set 2024-09-16T04:01:50.197 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.172+0000 7fed22139640 5 monclient: authenticate success, global_id 825429 2024-09-16T04:01:50.197 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.172+0000 7fed22139640 10 monclient: _renew_subs 2024-09-16T04:01:50.197 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.172+0000 7fed22139640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:01:50.197 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.172+0000 7fed22139640 1 -- 172.21.15.42:0/1813515815 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed1c0a51f0 con 0x7fed1c0a1330 2024-09-16T04:01:50.197 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.172+0000 7fed22139640 10 client.825429.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:01:50.200 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.172+0000 7fed22139640 10 monclient: _renew_subs 2024-09-16T04:01:50.200 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.172+0000 7fed22139640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:01:50.200 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.172+0000 7fed22139640 1 -- 172.21.15.42:0/1813515815 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed1c133020 con 0x7fed1c0a1330 2024-09-16T04:01:50.200 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.172+0000 7fed22139640 1 librados: init done 2024-09-16T04:01:50.201 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.172+0000 7fed22139640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:01:50.201 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.172+0000 7fed22139640 1 --2- 172.21.15.42:0/1813515815 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fed1c1c7860 0x7fed1c1c7c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:01:50.201 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.172+0000 7fed22139640 10 monclient(hunting): start opening mon connection 2024-09-16T04:01:50.201 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.172+0000 7fed22139640 1 -- 172.21.15.42:0/1813515815 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fed1c1c8180 con 0x7fed1c1c7860 2024-09-16T04:01:50.201 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.172+0000 7fed1affd640 1 --2- 172.21.15.42:0/1813515815 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fed1c1c7860 0x7fed1c1c7c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:01:50.201 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.172+0000 7fed1affd640 10 monclient: get_auth_request con 0x7fed1c1c7860 auth_method 0 2024-09-16T04:01:50.201 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.172+0000 7fed1affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:01:50.201 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.172+0000 7fed1affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:01:50.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.172+0000 7fed1affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:01:50.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.172+0000 7fed03fff640 1 -- 172.21.15.42:0/1813515815 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fed0c004b90 con 0x7fed1c0a1330 2024-09-16T04:01:50.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.172+0000 7fed1affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:01:50.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.172+0000 7fed1affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:01:50.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.172+0000 7fed1affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:01:50.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.172+0000 7fed03fff640 1 --2- 172.21.15.42:0/1813515815 >> v2:172.21.15.154:6812/254845342 conn(0x7fed08037840 0x7fed08039d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:01:50.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.172+0000 7fed1affd640 10 monclient(hunting): handle_auth_done global_id 825432 payload 824 2024-09-16T04:01:50.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.172+0000 7fed03fff640 1 -- 172.21.15.42:0/1813515815 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(601..601 src has 451..601) ==== 6252+0+0 (secure 0 0 0) 0x7fed0c0484b0 con 0x7fed1c0a1330 2024-09-16T04:01:50.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.172+0000 7fed03fff640 10 client.825429.objecter ms_dispatch 0x7fed1c000ba0 osd_map(601..601 src has 451..601) 2024-09-16T04:01:50.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.172+0000 7fed03fff640 3 client.825429.objecter handle_osd_map got epochs [601,601] > 0 2024-09-16T04:01:50.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.172+0000 7fed03fff640 3 client.825429.objecter handle_osd_map decoding full epoch 601 2024-09-16T04:01:50.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.172+0000 7fed1affd640 1 -- 172.21.15.42:0/1813515815 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fed1c0a1b40 con 0x7fed1c1c7860 2024-09-16T04:01:50.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.172+0000 7fed03fff640 20 client.825429.objecter dump_active .. 0 homeless 2024-09-16T04:01:50.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.172+0000 7fed1affd640 1 --2- 172.21.15.42:0/1813515815 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fed1c1c7860 0x7fed1c1c7c40 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7fed1c06add0 tx=0x7fed100190e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:01:50.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.172+0000 7fed03fff640 10 client.825429.objecter ms_handle_connect 0x7fed1c1c7860 2024-09-16T04:01:50.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.172+0000 7fed03fff640 10 client.825429.objecter resend_mon_ops 2024-09-16T04:01:50.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.172+0000 7fed1bfff640 1 --2- 172.21.15.42:0/1813515815 >> v2:172.21.15.154:6812/254845342 conn(0x7fed08037840 0x7fed08039d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:01:50.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.172+0000 7fed03fff640 1 -- 172.21.15.42:0/1813515815 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed100bb030 con 0x7fed1c1c7860 2024-09-16T04:01:50.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.172+0000 7fed03fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:01:50.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.172+0000 7fed03fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:01:50.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.172+0000 7fed03fff640 10 monclient: dump: 2024-09-16T04:01:50.204 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:01:50.204 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:01:50.204 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:50.204 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:50.205 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:01:50.205 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:01:50.205 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:01:50.205 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:01:50.205 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:01:50.205 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:01:50.205 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.172+0000 7fed1bfff640 10 monclient: get_auth_request con 0x7fed08037840 auth_method 0 2024-09-16T04:01:50.205 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.172+0000 7fed03fff640 1 -- 172.21.15.42:0/1813515815 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fed100b1040 con 0x7fed1c1c7860 2024-09-16T04:01:50.205 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.172+0000 7fed03fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:01:50.205 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.172+0000 7fed03fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:01:50.205 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.172+0000 7fed03fff640 1 -- 172.21.15.42:0/1813515815 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fed1c1c7860 msgr2=0x7fed1c1c7c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:01:50.205 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.172+0000 7fed03fff640 1 --2- 172.21.15.42:0/1813515815 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fed1c1c7860 0x7fed1c1c7c40 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7fed1c06add0 tx=0x7fed100190e0 comp rx=0 tx=0).stop 2024-09-16T04:01:50.205 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.172+0000 7fed03fff640 1 -- 172.21.15.42:0/1813515815 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fed1c1c7860 msgr2=0x7fed1c1c7c40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:01:50.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.172+0000 7fed03fff640 1 --2- 172.21.15.42:0/1813515815 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fed1c1c7860 0x7fed1c1c7c40 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:50.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.172+0000 7fed03fff640 10 client.825429.objecter ms_handle_connect 0x7fed08037840 2024-09-16T04:01:50.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.172+0000 7fed1bfff640 1 --2- 172.21.15.42:0/1813515815 >> v2:172.21.15.154:6812/254845342 conn(0x7fed08037840 0x7fed08039d00 secure :-1 s=READY pgs=3770 cs=0 l=1 rev1=1 crypto rx=0x7fed140077c0 tx=0x7fed14009820 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:01:50.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.180+0000 7fed22139640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:01:50.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.180+0000 7fed22139640 1 --2- 172.21.15.42:0/1813515815 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fed1c13a230 0x7fed1c13a610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:01:50.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.180+0000 7fed22139640 10 monclient(hunting): start opening mon connection 2024-09-16T04:01:50.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.180+0000 7fed22139640 1 -- 172.21.15.42:0/1813515815 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fed1c1cb300 con 0x7fed1c13a230 2024-09-16T04:01:50.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.180+0000 7fed1affd640 1 --2- 172.21.15.42:0/1813515815 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fed1c13a230 0x7fed1c13a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:01:50.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.180+0000 7fed1affd640 10 monclient: get_auth_request con 0x7fed1c13a230 auth_method 0 2024-09-16T04:01:50.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.180+0000 7fed1affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:01:50.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.180+0000 7fed1affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:01:50.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.180+0000 7fed1affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:01:50.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.180+0000 7fed1affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:01:50.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.180+0000 7fed1affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:01:50.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.180+0000 7fed1affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:01:50.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.180+0000 7fed1affd640 10 monclient(hunting): handle_auth_done global_id 825435 payload 824 2024-09-16T04:01:50.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.180+0000 7fed1affd640 1 -- 172.21.15.42:0/1813515815 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fed1c134d20 con 0x7fed1c13a230 2024-09-16T04:01:50.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.180+0000 7fed1affd640 1 --2- 172.21.15.42:0/1813515815 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fed1c13a230 0x7fed1c13a610 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fed1001c380 tx=0x7fed10009190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:01:50.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.180+0000 7fed03fff640 10 client.825429.objecter ms_handle_connect 0x7fed1c13a230 2024-09-16T04:01:50.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.180+0000 7fed03fff640 10 client.825429.objecter resend_mon_ops 2024-09-16T04:01:50.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.180+0000 7fed03fff640 1 -- 172.21.15.42:0/1813515815 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed100bb070 con 0x7fed1c13a230 2024-09-16T04:01:50.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.180+0000 7fed03fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:01:50.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.180+0000 7fed03fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:01:50.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.180+0000 7fed03fff640 10 monclient: dump: 2024-09-16T04:01:50.209 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:01:50.209 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:01:50.209 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:50.209 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:50.209 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:01:50.209 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:01:50.210 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:01:50.210 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:01:50.210 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:01:50.210 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:01:50.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.180+0000 7fed03fff640 1 -- 172.21.15.42:0/1813515815 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4715 (secure 0 0 0) 0x7fed100b7b40 con 0x7fed1c13a230 2024-09-16T04:01:50.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.180+0000 7fed03fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:01:50.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.180+0000 7fed03fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:01:50.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.180+0000 7fed03fff640 1 -- 172.21.15.42:0/1813515815 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fed1c13a230 msgr2=0x7fed1c13a610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:01:50.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.180+0000 7fed03fff640 1 --2- 172.21.15.42:0/1813515815 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fed1c13a230 0x7fed1c13a610 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fed1001c380 tx=0x7fed10009190 comp rx=0 tx=0).stop 2024-09-16T04:01:50.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.180+0000 7fed03fff640 1 -- 172.21.15.42:0/1813515815 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fed1c13a230 msgr2=0x7fed1c13a610 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:01:50.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.180+0000 7fed03fff640 1 --2- 172.21.15.42:0/1813515815 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fed1c13a230 0x7fed1c13a610 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:50.211 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T04:01:50.211 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:01:50.211 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:01:50.211 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "leader", 2024-09-16T04:01:50.211 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 178, 2024-09-16T04:01:50.211 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 600034, 2024-09-16T04:01:50.211 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T04:01:50.211 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T04:01:50.211 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T04:01:50.211 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T04:01:50.211 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:01:50.211 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 6, 2024-09-16T04:01:50.211 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:01:50.212 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T04:01:50.212 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T04:01:50.212 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:01:50.212 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:01:50.212 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:01:50.212 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:01:50.212 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:01:50.212 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:01:50.212 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:01:50.212 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:01:50.212 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:01:50.212 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:01:50.212 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:01:50.213 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:01:50.213 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:01:50.213 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T04:01:50.213 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:01:50.213 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:01:50.213 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:01:50.213 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:01:50.213 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:01:50.213 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:01:50.213 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:01:50.213 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:01:50.213 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:01:50.213 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:01:50.213 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:01:50.214 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:01:50.214 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:01:50.214 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T04:01:50.214 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T04:01:50.214 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T04:01:50.214 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T04:01:50.214 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T04:01:50.214 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T04:01:50.214 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:01:50.214 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:01:50.214 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T04:01:50.214 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T04:01:50.214 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T04:01:50.214 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T04:01:50.215 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T04:01:50.215 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T04:01:50.215 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T04:01:50.215 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:01:50.215 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T04:01:50.215 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:01:50.215 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:01:50.215 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:01:50.215 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:01:50.215 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:01:50.215 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:01:50.215 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:01:50.215 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:01:50.215 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:01:50.216 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:01:50.216 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:01:50.216 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:01:50.216 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T04:01:50.216 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:01:50.216 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T04:01:50.216 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:01:50.216 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:01:50.216 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:01:50.216 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:01:50.216 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:01:50.216 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:01:50.216 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:01:50.217 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T04:01:50.217 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:01:50.217 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:01:50.217 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:01:50.217 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:01:50.217 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T04:01:50.217 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:01:50.217 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:01:50.217 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:01:50.217 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:01:50.217 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T04:01:50.217 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T04:01:50.217 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:01:50.217 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:01:50.218 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:01:50.218 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:01:50.218 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:01:50.218 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:01:50.218 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:01:50.218 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:01:50.218 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:01:50.218 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:01:50.218 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:01:50.218 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T04:01:50.218 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:01:50.218 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:01:50.218 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:01:50.218 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:01:50.219 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T04:01:50.219 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:01:50.219 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:01:50.219 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:01:50.219 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:01:50.219 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T04:01:50.219 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T04:01:50.219 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:01:50.219 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:01:50.219 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:01:50.219 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:01:50.219 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:01:50.219 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:01:50.219 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:01:50.220 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:01:50.220 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:01:50.220 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:01:50.220 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:01:50.220 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T04:01:50.220 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:01:50.220 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:01:50.220 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:01:50.220 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:01:50.220 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T04:01:50.220 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:01:50.220 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:01:50.220 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:01:50.220 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:01:50.221 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T04:01:50.221 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T04:01:50.221 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:01:50.221 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:01:50.221 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:01:50.221 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:01:50.221 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:01:50.221 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:01:50.221 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T04:01:50.222 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T04:01:50.222 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:01:50.222 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:01:50.222 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:01:50.222 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:01:50.222 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:01:50.222 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:01:50.222 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T04:01:50.222 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:01:50.222 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:01:50.222 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:01:50.222 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 5 2024-09-16T04:01:50.222 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:01:50.222 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:01:50.222 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T04:01:50.223 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:01:50.223 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:01:50.223 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:01:50.223 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 7 2024-09-16T04:01:50.223 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:01:50.223 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:01:50.223 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:01:50.223 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T04:01:50.223 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T04:01:50.225 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.180+0000 7fed22139640 10 librados: watch_flush enter 2024-09-16T04:01:50.225 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.180+0000 7fed22139640 10 librados: watch_flush exit 2024-09-16T04:01:50.225 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.180+0000 7fed22139640 20 client.825429.objecter shutdown clearing up homeless session... 2024-09-16T04:01:50.225 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.180+0000 7fed22139640 10 client.825429.objecter successfully canceled tick 2024-09-16T04:01:50.225 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.180+0000 7fed22139640 1 -- 172.21.15.42:0/1813515815 >> v2:172.21.15.154:6812/254845342 conn(0x7fed08037840 msgr2=0x7fed08039d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:01:50.225 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.180+0000 7fed22139640 1 --2- 172.21.15.42:0/1813515815 >> v2:172.21.15.154:6812/254845342 conn(0x7fed08037840 0x7fed08039d00 secure :-1 s=READY pgs=3770 cs=0 l=1 rev1=1 crypto rx=0x7fed140077c0 tx=0x7fed14009820 comp rx=0 tx=0).stop 2024-09-16T04:01:50.225 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.180+0000 7fed22139640 10 monclient: shutdown 2024-09-16T04:01:50.225 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.180+0000 7fed22139640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:01:50.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.180+0000 7fed22139640 1 -- 172.21.15.42:0/1813515815 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fed1c0a1330 msgr2=0x7fed1c1ca8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:01:50.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.180+0000 7fed22139640 1 --2- 172.21.15.42:0/1813515815 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fed1c0a1330 0x7fed1c1ca8a0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fed0c00c550 tx=0x7fed0c00ca10 comp rx=0 tx=0).stop 2024-09-16T04:01:50.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.180+0000 7fed1bfff640 1 -- 172.21.15.42:0/1813515815 reap_dead start 2024-09-16T04:01:50.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.180+0000 7fed1bfff640 1 -- 172.21.15.42:0/1813515815 reap_dead start 2024-09-16T04:01:50.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.180+0000 7fed22139640 1 -- 172.21.15.42:0/1813515815 shutdown_connections 2024-09-16T04:01:50.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.180+0000 7fed22139640 1 -- 172.21.15.42:0/1813515815 >> 172.21.15.42:0/1813515815 conn(0x7fed1c0a2420 msgr2=0x7fed1c0a3800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:01:50.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.180+0000 7fed22139640 1 -- 172.21.15.42:0/1813515815 shutdown_connections 2024-09-16T04:01:50.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.180+0000 7fed22139640 1 -- 172.21.15.42:0/1813515815 wait complete. 2024-09-16T04:01:50.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.180+0000 7fed22139640 1 librados: shutdown 2024-09-16T04:01:50.226 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:01:50.227 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T04:01:50.227 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T04:01:50.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.340+0000 7f1ae5269640 10 monclient: get_monmap_and_config 2024-09-16T04:01:50.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.340+0000 7f1ae5269640 10 monclient: build_initial_monmap 2024-09-16T04:01:50.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.340+0000 7f1ae5269640 10 monclient: monmap: 2024-09-16T04:01:50.343 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:01:50.343 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:01:50.343 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:01:50.341261+0000 2024-09-16T04:01:50.343 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:01:50.341261+0000 2024-09-16T04:01:50.343 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:01:50.343 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:01:50.343 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:01:50.343 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:01:50.343 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:01:50.343 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:01:50.343 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:01:50.343 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:01:50.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.340+0000 7f1ae5269640 1 Processor -- start 2024-09-16T04:01:50.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.340+0000 7f1ae5269640 1 -- start start 2024-09-16T04:01:50.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.340+0000 7f1ae5269640 10 monclient: init 2024-09-16T04:01:50.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.340+0000 7f1ae5269640 10 monclient: _reopen_session rank -1 2024-09-16T04:01:50.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.340+0000 7f1ae5269640 10 monclient: _add_conns ranks=[4,0,1,2,3] 2024-09-16T04:01:50.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.340+0000 7f1ae5269640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f1ae006db90 0x7f1ae01340c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:01:50.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.340+0000 7f1ae5269640 10 monclient(hunting): picked mon.noname-c con 0x7f1ae006db90 addr v2:172.21.15.154:3300/0 2024-09-16T04:01:50.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.340+0000 7f1ae5269640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1ae00a37f0 addr v1:172.21.15.42:6789/0 2024-09-16T04:01:50.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.340+0000 7f1ae5269640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1ae00a3c00 addr v1:172.21.15.154:6789/0 2024-09-16T04:01:50.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.340+0000 7f1ae5269640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1ae006ece0 con 0x7f1ae00a37f0 2024-09-16T04:01:50.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.340+0000 7f1ae5269640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1ae009bf80 con 0x7f1ae00a3c00 2024-09-16T04:01:50.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.340+0000 7f1ae5269640 10 monclient(hunting): start opening mon connection 2024-09-16T04:01:50.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.340+0000 7f1ae5269640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f1ae006df70 con 0x7f1ae006db90 2024-09-16T04:01:50.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.340+0000 7f1ae5269640 10 monclient(hunting): _renew_subs 2024-09-16T04:01:50.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.340+0000 7f1ae5269640 10 monclient(hunting): authenticate will time out at 2157.311510s 2024-09-16T04:01:50.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.340+0000 7f1ade7fc640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f1ae00a37f0 0x7f1ae0136e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:58950/0 (socket says 172.21.15.42:58950) 2024-09-16T04:01:50.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.340+0000 7f1ade7fc640 1 -- 172.21.15.42:0/1813616114 learned_addr learned my addr 172.21.15.42:0/1813616114 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T04:01:50.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.340+0000 7f1adeffd640 1 --2- 172.21.15.42:0/1813616114 >> v2:172.21.15.154:3300/0 conn(0x7f1ae006db90 0x7f1ae01340c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:01:50.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.344+0000 7f1adeffd640 10 monclient(hunting): get_auth_request con 0x7f1ae006db90 auth_method 0 2024-09-16T04:01:50.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.344+0000 7f1adeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:01:50.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.344+0000 7f1adeffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:01:50.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.344+0000 7f1adeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:01:50.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.344+0000 7f1addffb640 1 -- 172.21.15.42:0/1813616114 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2956464779 0 0) 0x7f1ae006ece0 con 0x7f1ae00a37f0 2024-09-16T04:01:50.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.344+0000 7f1addffb640 10 monclient(hunting): _negotiate 2024-09-16T04:01:50.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.344+0000 7f1addffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:01:50.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.344+0000 7f1addffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:01:50.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.344+0000 7f1addffb640 10 monclient(hunting): my global_id is 825438 2024-09-16T04:01:50.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.344+0000 7f1addffb640 1 -- 172.21.15.42:0/1813616114 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1ac8003530 con 0x7f1ae00a37f0 2024-09-16T04:01:50.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.344+0000 7f1adeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:01:50.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.344+0000 7f1adeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:01:50.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.344+0000 7f1adeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:01:50.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.344+0000 7f1addffb640 1 -- 172.21.15.42:0/1813616114 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3820446654 0 0) 0x7f1ac8003530 con 0x7f1ae00a37f0 2024-09-16T04:01:50.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.344+0000 7f1addffb640 1 -- 172.21.15.42:0/1813616114 >> v2:172.21.15.154:3300/0 conn(0x7f1ae006db90 msgr2=0x7f1ae01340c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:01:50.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.344+0000 7f1addffb640 1 --2- 172.21.15.42:0/1813616114 >> v2:172.21.15.154:3300/0 conn(0x7f1ae006db90 0x7f1ae01340c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:50.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.344+0000 7f1addffb640 1 -- 172.21.15.42:0/1813616114 >> v1:172.21.15.154:6789/0 conn(0x7f1ae00a3c00 legacy=0x7f1ae0137520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:01:50.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.344+0000 7f1addffb640 10 monclient: _finish_hunting 0 2024-09-16T04:01:50.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.344+0000 7f1addffb640 1 monclient: found mon.noname-a-legacy 2024-09-16T04:01:50.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.344+0000 7f1addffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:01:50.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.344+0000 7f1addffb640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.42:6789/0 2024-09-16T04:01:50.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.344+0000 7f1addffb640 1 -- 172.21.15.42:0/1813616114 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ae0137c30 con 0x7f1ae00a37f0 2024-09-16T04:01:50.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.344+0000 7f1addffb640 10 monclient: _finish_auth 0 2024-09-16T04:01:50.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.344+0000 7f1addffb640 10 monclient: _check_auth_tickets 2024-09-16T04:01:50.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.344+0000 7f1addffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:01:50.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.344+0000 7f1addffb640 1 -- 172.21.15.42:0/1813616114 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2177413734 0 0) 0x7f1ad4001680 con 0x7f1ae00a37f0 2024-09-16T04:01:50.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.344+0000 7f1addffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:01:50.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.344+0000 7f1addffb640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T04:01:50.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.344+0000 7f1addffb640 10 monclient: dump: 2024-09-16T04:01:50.351 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:01:50.352 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:01:50.352 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:50.352 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:50.352 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:01:50.352 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:01:50.352 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:01:50.352 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:01:50.352 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:01:50.352 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:01:50.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.344+0000 7f1addffb640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] but i'm connected to v1:172.21.15.42:6789/0, reconnecting 2024-09-16T04:01:50.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.344+0000 7f1addffb640 10 monclient: _reopen_session rank -1 2024-09-16T04:01:50.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.344+0000 7f1addffb640 1 -- 172.21.15.42:0/1813616114 >> v1:172.21.15.42:6789/0 conn(0x7f1ae00a37f0 legacy=0x7f1ae0136e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:01:50.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.344+0000 7f1addffb640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:01:50.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.344+0000 7f1addffb640 1 --2- 172.21.15.42:0/1813616114 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1ac80039c0 0x7f1ac8003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:01:50.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.344+0000 7f1addffb640 10 monclient(hunting): picked mon.c con 0x7f1ac80039c0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:01:50.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.344+0000 7f1addffb640 1 --2- 172.21.15.42:0/1813616114 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1ac8004810 0x7f1ac8004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:01:50.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.344+0000 7f1addffb640 10 monclient(hunting): picked mon.a con 0x7f1ac8004810 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:01:50.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.344+0000 7f1addffb640 1 --2- 172.21.15.42:0/1813616114 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1ac80052f0 0x7f1ac800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:01:50.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.344+0000 7f1addffb640 10 monclient(hunting): picked mon.b con 0x7f1ac80052f0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:01:50.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.344+0000 7f1addffb640 10 monclient(hunting): start opening mon connection 2024-09-16T04:01:50.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.344+0000 7f1addffb640 1 -- 172.21.15.42:0/1813616114 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f1ae006df70 con 0x7f1ac8004810 2024-09-16T04:01:50.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.344+0000 7f1addffb640 10 monclient(hunting): start opening mon connection 2024-09-16T04:01:50.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.344+0000 7f1addffb640 1 -- 172.21.15.42:0/1813616114 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f1ac800c1c0 con 0x7f1ac80052f0 2024-09-16T04:01:50.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.344+0000 7f1addffb640 10 monclient(hunting): start opening mon connection 2024-09-16T04:01:50.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.344+0000 7f1addffb640 1 -- 172.21.15.42:0/1813616114 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f1ac800c3c0 con 0x7f1ac80039c0 2024-09-16T04:01:50.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.344+0000 7f1addffb640 10 monclient(hunting): _renew_subs 2024-09-16T04:01:50.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.344+0000 7f1addffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:01:50.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.344+0000 7f1adeffd640 1 --2- 172.21.15.42:0/1813616114 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1ac80039c0 0x7f1ac8003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:01:50.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.344+0000 7f1ade7fc640 1 --2- 172.21.15.42:0/1813616114 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1ac8004810 0x7f1ac8004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:01:50.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.344+0000 7f1adeffd640 10 monclient(hunting): get_auth_request con 0x7f1ac80039c0 auth_method 0 2024-09-16T04:01:50.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.344+0000 7f1adeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:01:50.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.344+0000 7f1adeffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:01:50.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.344+0000 7f1adeffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:01:50.355 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.344+0000 7f1ade7fc640 10 monclient(hunting): get_auth_request con 0x7f1ac8004810 auth_method 0 2024-09-16T04:01:50.355 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.344+0000 7f1ade7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:01:50.355 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.344+0000 7f1ade7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:01:50.355 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.344+0000 7f1ade7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:01:50.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.344+0000 7f1adeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:01:50.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.344+0000 7f1adeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:01:50.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.344+0000 7f1adeffd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:01:50.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.344+0000 7f1ade7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:01:50.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.344+0000 7f1ade7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:01:50.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.344+0000 7f1ade7fc640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:01:50.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.344+0000 7f1ade7fc640 10 monclient(hunting): handle_auth_done global_id 825438 payload 293 2024-09-16T04:01:50.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.344+0000 7f1ade7fc640 1 -- 172.21.15.42:0/1813616114 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1ac80039c0 msgr2=0x7f1ac8003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:01:50.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.344+0000 7f1ade7fc640 1 --2- 172.21.15.42:0/1813616114 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1ac80039c0 0x7f1ac8003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:50.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.344+0000 7f1ade7fc640 1 -- 172.21.15.42:0/1813616114 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1ac80052f0 msgr2=0x7f1ac800b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:01:50.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.344+0000 7f1adf7fe640 1 --2- 172.21.15.42:0/1813616114 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1ac80052f0 0x7f1ac800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:01:50.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.344+0000 7f1ade7fc640 1 --2- 172.21.15.42:0/1813616114 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1ac80052f0 0x7f1ac800b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:50.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.344+0000 7f1ade7fc640 10 monclient: _finish_hunting 0 2024-09-16T04:01:50.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.344+0000 7f1ade7fc640 1 monclient: found mon.a 2024-09-16T04:01:50.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.344+0000 7f1ade7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:01:50.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.344+0000 7f1ade7fc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:01:50.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.344+0000 7f1ade7fc640 1 -- 172.21.15.42:0/1813616114 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ac8003530 con 0x7f1ac8004810 2024-09-16T04:01:50.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.344+0000 7f1ade7fc640 10 monclient: _finish_auth 0 2024-09-16T04:01:50.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.344+0000 7f1ade7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:01:50.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.344+0000 7f1ade7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:01:50.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.344+0000 7f1ae5269640 5 monclient: authenticate success, global_id 825438 2024-09-16T04:01:50.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.344+0000 7f1ae5269640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:01:50.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.344+0000 7f1adf7fe640 10 monclient: get_auth_request con 0x7f1ac80052f0 auth_method 0 2024-09-16T04:01:50.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.344+0000 7f1adf7fe640 1 --2- 172.21.15.42:0/1813616114 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1ac80052f0 0x7f1ac800b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T04:01:50.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.344+0000 7f1adeffd640 1 --2- 172.21.15.42:0/1813616114 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1ac80039c0 0x7f1ac8003de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:01:50.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.344+0000 7f1adf7fe640 1 -- 172.21.15.42:0/1813616114 reap_dead start 2024-09-16T04:01:50.366 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.344+0000 7f1ade7fc640 1 --2- 172.21.15.42:0/1813616114 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1ac8004810 0x7f1ac8004c90 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f1ad4001ec0 tx=0x7f1ad401cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db8074902ecb14ea server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:01:50.366 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.344+0000 7f1addffb640 1 -- 172.21.15.42:0/1813616114 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ad4007d40 con 0x7f1ac8004810 2024-09-16T04:01:50.366 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.344+0000 7f1addffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:01:50.366 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.344+0000 7f1addffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:01:50.366 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.344+0000 7f1addffb640 10 monclient: dump: 2024-09-16T04:01:50.366 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:01:50.366 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:01:50.366 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:50.366 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:50.366 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:01:50.367 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:01:50.367 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:01:50.367 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:01:50.367 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:01:50.367 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:01:50.367 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.344+0000 7f1addffb640 1 -- 172.21.15.42:0/1813616114 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ad40a9070 con 0x7f1ac8004810 2024-09-16T04:01:50.367 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.344+0000 7f1ae5269640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:01:50.367 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.344+0000 7f1addffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:01:50.367 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.344+0000 7f1ae5269640 10 monclient: get_monmap_and_config success 2024-09-16T04:01:50.367 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.344+0000 7f1ae5269640 4 set_mon_vals no callback set 2024-09-16T04:01:50.367 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.344+0000 7f1addffb640 1 -- 172.21.15.42:0/1813616114 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ad4005d80 con 0x7f1ac8004810 2024-09-16T04:01:50.367 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.344+0000 7f1addffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:01:50.367 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.344+0000 7f1addffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:01:50.368 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.344+0000 7f1addffb640 10 monclient: dump: 2024-09-16T04:01:50.368 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:01:50.368 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:01:50.368 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:50.368 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:50.368 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:01:50.368 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:01:50.368 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:01:50.368 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:01:50.368 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:01:50.368 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:01:50.368 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.348+0000 7f1ae5269640 10 monclient: shutdown 2024-09-16T04:01:50.368 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.348+0000 7f1ae5269640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:01:50.368 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.348+0000 7f1ae5269640 1 -- 172.21.15.42:0/1813616114 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1ac8004810 msgr2=0x7f1ac8004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:01:50.369 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.348+0000 7f1ae5269640 1 --2- 172.21.15.42:0/1813616114 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1ac8004810 0x7f1ac8004c90 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f1ad4001ec0 tx=0x7f1ad401cc40 comp rx=0 tx=0).stop 2024-09-16T04:01:50.369 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.348+0000 7f1ae5269640 1 -- 172.21.15.42:0/1813616114 shutdown_connections 2024-09-16T04:01:50.369 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.348+0000 7f1ae5269640 1 --2- 172.21.15.42:0/1813616114 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1ac8004810 0x7f1ac8004c90 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:50.369 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.348+0000 7f1ae5269640 1 -- 172.21.15.42:0/1813616114 >> 172.21.15.42:0/1813616114 conn(0x7f1ae00a9fb0 msgr2=0x7f1ae00aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:01:50.369 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.348+0000 7f1ae5269640 1 -- 172.21.15.42:0/1813616114 shutdown_connections 2024-09-16T04:01:50.369 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.348+0000 7f1ae5269640 1 -- 172.21.15.42:0/1813616114 wait complete. 2024-09-16T04:01:50.369 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.348+0000 7f1ae5269640 10 monclient: build_initial_monmap 2024-09-16T04:01:50.369 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.348+0000 7f1ae5269640 10 monclient: monmap: 2024-09-16T04:01:50.369 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:01:50.369 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:01:50.369 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:01:50.369 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:01:50.369 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:01:50.370 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:01:50.370 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:01:50.370 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:01:50.370 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:01:50.370 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:01:50.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.348+0000 7f1ae5269640 1 librados: starting msgr at 2024-09-16T04:01:50.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.348+0000 7f1ae5269640 1 librados: starting objecter 2024-09-16T04:01:50.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.348+0000 7f1ae5269640 1 Processor -- start 2024-09-16T04:01:50.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.348+0000 7f1ae5269640 1 -- start start 2024-09-16T04:01:50.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.348+0000 7f1ae5269640 1 librados: setting wanted keys 2024-09-16T04:01:50.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.348+0000 7f1ae5269640 1 librados: calling monclient init 2024-09-16T04:01:50.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.348+0000 7f1ae5269640 10 monclient: init 2024-09-16T04:01:50.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.348+0000 7f1ae5269640 10 monclient: _reopen_session rank -1 2024-09-16T04:01:50.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.348+0000 7f1ae5269640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:01:50.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.348+0000 7f1ae5269640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1ac8004810 0x7f1ae0120200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:01:50.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.348+0000 7f1ae5269640 10 monclient(hunting): picked mon.noname-b con 0x7f1ac8004810 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:01:50.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.348+0000 7f1ae5269640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1ae0120740 0x7f1ae0120b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:01:50.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.348+0000 7f1ae5269640 10 monclient(hunting): picked mon.noname-a con 0x7f1ae0120740 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:01:50.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.348+0000 7f1ae5269640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1ae01cf2c0 0x7f1ae01cf740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:01:50.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.348+0000 7f1ae5269640 10 monclient(hunting): picked mon.noname-c con 0x7f1ae01cf2c0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:01:50.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.348+0000 7f1ae5269640 10 monclient(hunting): start opening mon connection 2024-09-16T04:01:50.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.348+0000 7f1ae5269640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f1ae0137620 con 0x7f1ae0120740 2024-09-16T04:01:50.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.348+0000 7f1ae5269640 10 monclient(hunting): start opening mon connection 2024-09-16T04:01:50.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.348+0000 7f1ae5269640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f1ae01377a0 con 0x7f1ac8004810 2024-09-16T04:01:50.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.348+0000 7f1ae5269640 10 monclient(hunting): start opening mon connection 2024-09-16T04:01:50.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.348+0000 7f1ae5269640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f1ae0137920 con 0x7f1ae01cf2c0 2024-09-16T04:01:50.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.348+0000 7f1ae5269640 10 monclient(hunting): _renew_subs 2024-09-16T04:01:50.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.348+0000 7f1ae5269640 10 monclient(hunting): authenticate will time out at 2157.317835s 2024-09-16T04:01:50.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.348+0000 7f1ade7fc640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1ae0120740 0x7f1ae0120b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:01:50.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.348+0000 7f1ade7fc640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1ae0120740 0x7f1ae0120b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:35378/0 (socket says 172.21.15.42:35378) 2024-09-16T04:01:50.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.348+0000 7f1ade7fc640 1 -- 172.21.15.42:0/4011149115 learned_addr learned my addr 172.21.15.42:0/4011149115 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:01:50.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.348+0000 7f1ade7fc640 10 monclient(hunting): get_auth_request con 0x7f1ae0120740 auth_method 0 2024-09-16T04:01:50.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.348+0000 7f1ade7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:01:50.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.348+0000 7f1ade7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:01:50.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.348+0000 7f1ade7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:01:50.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.348+0000 7f1adf7fe640 1 --2- 172.21.15.42:0/4011149115 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1ae01cf2c0 0x7f1ae01cf740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:01:50.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.348+0000 7f1adf7fe640 10 monclient(hunting): get_auth_request con 0x7f1ae01cf2c0 auth_method 0 2024-09-16T04:01:50.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.348+0000 7f1adf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:01:50.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.348+0000 7f1adf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:01:50.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.348+0000 7f1adf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:01:50.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.348+0000 7f1ade7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:01:50.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.348+0000 7f1ade7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:01:50.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.348+0000 7f1adeffd640 1 --2- 172.21.15.42:0/4011149115 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1ac8004810 0x7f1ae0120200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:01:50.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.348+0000 7f1ade7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:01:50.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.348+0000 7f1adeffd640 10 monclient(hunting): get_auth_request con 0x7f1ac8004810 auth_method 0 2024-09-16T04:01:50.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.348+0000 7f1adeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:01:50.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.348+0000 7f1adeffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:01:50.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.348+0000 7f1adeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:01:50.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.348+0000 7f1adf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:01:50.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.348+0000 7f1adf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:01:50.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.348+0000 7f1adf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:01:50.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.348+0000 7f1adeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:01:50.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.348+0000 7f1adeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:01:50.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.348+0000 7f1adeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:01:50.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.348+0000 7f1ade7fc640 10 monclient(hunting): handle_auth_done global_id 825441 payload 824 2024-09-16T04:01:50.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.348+0000 7f1ade7fc640 1 -- 172.21.15.42:0/4011149115 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1ae01cf2c0 msgr2=0x7f1ae01cf740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:01:50.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.348+0000 7f1ade7fc640 1 --2- 172.21.15.42:0/4011149115 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1ae01cf2c0 0x7f1ae01cf740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:50.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.348+0000 7f1ade7fc640 1 -- 172.21.15.42:0/4011149115 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1ac8004810 msgr2=0x7f1ae0120200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:01:50.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.348+0000 7f1ade7fc640 1 --2- 172.21.15.42:0/4011149115 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1ac8004810 0x7f1ae0120200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:50.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.348+0000 7f1ade7fc640 10 monclient: _finish_hunting 0 2024-09-16T04:01:50.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.348+0000 7f1ade7fc640 1 monclient: found mon.noname-a 2024-09-16T04:01:50.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.348+0000 7f1ade7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:01:50.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.348+0000 7f1ade7fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:01:50.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.348+0000 7f1ade7fc640 1 -- 172.21.15.42:0/4011149115 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ae01223f0 con 0x7f1ae0120740 2024-09-16T04:01:50.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.348+0000 7f1adf7fe640 1 --2- 172.21.15.42:0/4011149115 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1ae01cf2c0 0x7f1ae01cf740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:01:50.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.348+0000 7f1ade7fc640 1 --2- 172.21.15.42:0/4011149115 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1ae0120740 0x7f1ae0120b80 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f1ad401ca70 tx=0x7f1ac8005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:01:50.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.348+0000 7f1ac77fe640 10 client.?.objecter ms_handle_connect 0x7f1ae0120740 2024-09-16T04:01:50.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.348+0000 7f1ac77fe640 10 client.?.objecter resend_mon_ops 2024-09-16T04:01:50.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.348+0000 7f1ac77fe640 1 -- 172.21.15.42:0/4011149115 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ad4000f60 con 0x7f1ae0120740 2024-09-16T04:01:50.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.348+0000 7f1ac77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:01:50.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.348+0000 7f1ac77fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:01:50.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.348+0000 7f1ac77fe640 10 monclient: dump: 2024-09-16T04:01:50.375 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:01:50.375 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:01:50.375 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:50.375 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:50.375 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:01:50.375 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:01:50.375 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:01:50.376 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:01:50.376 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:01:50.376 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:01:50.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.348+0000 7f1ac77fe640 10 monclient: _finish_auth 0 2024-09-16T04:01:50.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.348+0000 7f1ac77fe640 10 monclient: _check_auth_tickets 2024-09-16T04:01:50.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.348+0000 7f1ac77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:01:50.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.348+0000 7f1ac77fe640 1 -- 172.21.15.42:0/4011149115 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ad4007390 con 0x7f1ae0120740 2024-09-16T04:01:50.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.348+0000 7f1ae5269640 5 monclient: authenticate success, global_id 825441 2024-09-16T04:01:50.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.348+0000 7f1ae5269640 10 monclient: _renew_subs 2024-09-16T04:01:50.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.348+0000 7f1ae5269640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:01:50.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.348+0000 7f1ae5269640 1 -- 172.21.15.42:0/4011149115 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ae01226d0 con 0x7f1ae0120740 2024-09-16T04:01:50.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.348+0000 7f1ac77fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:01:50.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.348+0000 7f1ac77fe640 1 -- 172.21.15.42:0/4011149115 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ad4007740 con 0x7f1ae0120740 2024-09-16T04:01:50.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.348+0000 7f1ac77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:01:50.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.348+0000 7f1ac77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:01:50.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.348+0000 7f1ac77fe640 10 monclient: dump: 2024-09-16T04:01:50.377 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:01:50.377 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:01:50.377 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:50.377 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:50.377 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:01:50.377 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:01:50.377 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:01:50.378 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:01:50.378 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:01:50.378 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:01:50.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.348+0000 7f1add7fa640 4 set_mon_vals no callback set 2024-09-16T04:01:50.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.348+0000 7f1ae5269640 10 client.825441.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:01:50.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.348+0000 7f1ae5269640 10 monclient: _renew_subs 2024-09-16T04:01:50.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.348+0000 7f1ae5269640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:01:50.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.348+0000 7f1ae5269640 1 -- 172.21.15.42:0/4011149115 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ae01229b0 con 0x7f1ae0120740 2024-09-16T04:01:50.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.348+0000 7f1ae5269640 1 librados: init done 2024-09-16T04:01:50.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.348+0000 7f1ae5269640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:01:50.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.348+0000 7f1ae5269640 1 --2- 172.21.15.42:0/4011149115 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1ae00a2520 0x7f1ae01cfc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:01:50.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.348+0000 7f1ae5269640 10 monclient(hunting): start opening mon connection 2024-09-16T04:01:50.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.348+0000 7f1ae5269640 1 -- 172.21.15.42:0/4011149115 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f1ae006eaa0 con 0x7f1ae00a2520 2024-09-16T04:01:50.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.348+0000 7f1adeffd640 1 --2- 172.21.15.42:0/4011149115 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1ae00a2520 0x7f1ae01cfc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:01:50.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.348+0000 7f1adeffd640 10 monclient: get_auth_request con 0x7f1ae00a2520 auth_method 0 2024-09-16T04:01:50.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.348+0000 7f1adeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:01:50.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.348+0000 7f1adeffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:01:50.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.348+0000 7f1adeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:01:50.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.348+0000 7f1adeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:01:50.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.348+0000 7f1adeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:01:50.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.348+0000 7f1adeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:01:50.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.348+0000 7f1ac77fe640 1 -- 172.21.15.42:0/4011149115 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f1ad40a5830 con 0x7f1ae0120740 2024-09-16T04:01:50.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.352+0000 7f1adeffd640 10 monclient(hunting): handle_auth_done global_id 954128 payload 824 2024-09-16T04:01:50.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.352+0000 7f1adeffd640 1 -- 172.21.15.42:0/4011149115 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1ae0130c80 con 0x7f1ae00a2520 2024-09-16T04:01:50.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.352+0000 7f1ac77fe640 1 --2- 172.21.15.42:0/4011149115 >> v2:172.21.15.154:6812/254845342 conn(0x7f1acc037840 0x7f1acc039d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:01:50.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.352+0000 7f1ac77fe640 1 -- 172.21.15.42:0/4011149115 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(601..601 src has 451..601) ==== 6252+0+0 (secure 0 0 0) 0x7f1ad40e29f0 con 0x7f1ae0120740 2024-09-16T04:01:50.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.352+0000 7f1ac77fe640 10 client.825441.objecter ms_dispatch 0x7f1ae0000ba0 osd_map(601..601 src has 451..601) 2024-09-16T04:01:50.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.352+0000 7f1ac77fe640 3 client.825441.objecter handle_osd_map got epochs [601,601] > 0 2024-09-16T04:01:50.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.352+0000 7f1ac77fe640 3 client.825441.objecter handle_osd_map decoding full epoch 601 2024-09-16T04:01:50.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.352+0000 7f1adeffd640 1 --2- 172.21.15.42:0/4011149115 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1ae00a2520 0x7f1ae01cfc80 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f1ae00745f0 tx=0x7f1ad0012960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:01:50.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.352+0000 7f1ac77fe640 20 client.825441.objecter dump_active .. 0 homeless 2024-09-16T04:01:50.380 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.352+0000 7f1ac77fe640 10 client.825441.objecter ms_handle_connect 0x7f1ae00a2520 2024-09-16T04:01:50.380 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.352+0000 7f1ac77fe640 10 client.825441.objecter resend_mon_ops 2024-09-16T04:01:50.380 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.352+0000 7f1ac77fe640 1 -- 172.21.15.42:0/4011149115 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ad0009c10 con 0x7f1ae00a2520 2024-09-16T04:01:50.380 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.352+0000 7f1ac77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:01:50.380 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.352+0000 7f1ac77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:01:50.380 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.352+0000 7f1ac77fe640 10 monclient: dump: 2024-09-16T04:01:50.380 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:01:50.380 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:01:50.380 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:50.380 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:50.380 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:01:50.380 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:01:50.380 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:01:50.381 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:01:50.381 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:01:50.381 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:01:50.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.352+0000 7f1adf7fe640 1 --2- 172.21.15.42:0/4011149115 >> v2:172.21.15.154:6812/254845342 conn(0x7f1acc037840 0x7f1acc039d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:01:50.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.352+0000 7f1adf7fe640 10 monclient: get_auth_request con 0x7f1acc037840 auth_method 0 2024-09-16T04:01:50.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.352+0000 7f1ac77fe640 1 -- 172.21.15.42:0/4011149115 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f1ad00168d0 con 0x7f1ae00a2520 2024-09-16T04:01:50.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.352+0000 7f1ac77fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:01:50.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.352+0000 7f1ac77fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:01:50.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.352+0000 7f1ac77fe640 1 -- 172.21.15.42:0/4011149115 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1ae00a2520 msgr2=0x7f1ae01cfc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:01:50.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.352+0000 7f1ac77fe640 1 --2- 172.21.15.42:0/4011149115 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1ae00a2520 0x7f1ae01cfc80 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f1ae00745f0 tx=0x7f1ad0012960 comp rx=0 tx=0).stop 2024-09-16T04:01:50.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.352+0000 7f1ac77fe640 1 -- 172.21.15.42:0/4011149115 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1ae00a2520 msgr2=0x7f1ae01cfc80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:01:50.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.352+0000 7f1ac77fe640 1 --2- 172.21.15.42:0/4011149115 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1ae00a2520 0x7f1ae01cfc80 secure :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f1ae00745f0 tx=0x7f1ad0012960 comp rx=0 tx=0).stop 2024-09-16T04:01:50.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.352+0000 7f1adf7fe640 1 --2- 172.21.15.42:0/4011149115 >> v2:172.21.15.154:6812/254845342 conn(0x7f1acc037840 0x7f1acc039d00 secure :-1 s=READY pgs=3771 cs=0 l=1 rev1=1 crypto rx=0x7f1ae00a3bd0 tx=0x7f1ad8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:01:50.382 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.352+0000 7f1ac77fe640 10 client.825441.objecter ms_handle_connect 0x7f1acc037840 2024-09-16T04:01:50.382 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.356+0000 7f1ae5269640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:01:50.382 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.356+0000 7f1ae5269640 1 --2- 172.21.15.42:0/4011149115 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1ae009e720 0x7f1ae009eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:01:50.382 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.356+0000 7f1ae5269640 10 monclient(hunting): start opening mon connection 2024-09-16T04:01:50.382 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.356+0000 7f1ae5269640 1 -- 172.21.15.42:0/4011149115 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f1ae006f680 con 0x7f1ae009e720 2024-09-16T04:01:50.382 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.356+0000 7f1adeffd640 1 --2- 172.21.15.42:0/4011149115 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1ae009e720 0x7f1ae009eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:01:50.382 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.356+0000 7f1adeffd640 10 monclient: get_auth_request con 0x7f1ae009e720 auth_method 0 2024-09-16T04:01:50.382 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.356+0000 7f1adeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:01:50.382 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.356+0000 7f1adeffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:01:50.382 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.356+0000 7f1adeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:01:50.382 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.356+0000 7f1adeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:01:50.382 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.356+0000 7f1adeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:01:50.382 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.356+0000 7f1adeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:01:50.382 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.356+0000 7f1adeffd640 10 monclient(hunting): handle_auth_done global_id 954131 payload 824 2024-09-16T04:01:50.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.356+0000 7f1adeffd640 1 -- 172.21.15.42:0/4011149115 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f1ae013a2a0 con 0x7f1ae009e720 2024-09-16T04:01:50.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.356+0000 7f1adeffd640 1 --2- 172.21.15.42:0/4011149115 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1ae009e720 0x7f1ae009eb30 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f1ae00745f0 tx=0x7f1ad0004ac0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:01:50.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.356+0000 7f1ac77fe640 10 client.825441.objecter ms_handle_connect 0x7f1ae009e720 2024-09-16T04:01:50.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.356+0000 7f1ac77fe640 10 client.825441.objecter resend_mon_ops 2024-09-16T04:01:50.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.356+0000 7f1ac77fe640 1 -- 172.21.15.42:0/4011149115 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ad0026be0 con 0x7f1ae009e720 2024-09-16T04:01:50.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.356+0000 7f1ac77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:01:50.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.356+0000 7f1ac77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:01:50.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.356+0000 7f1ac77fe640 10 monclient: dump: 2024-09-16T04:01:50.383 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:01:50.383 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:01:50.383 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:50.383 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:50.383 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:01:50.383 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:01:50.384 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:01:50.384 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:01:50.384 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:01:50.384 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:01:50.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.356+0000 7f1ac77fe640 1 -- 172.21.15.42:0/4011149115 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4545 (secure 0 0 0) 0x7f1ad0026be0 con 0x7f1ae009e720 2024-09-16T04:01:50.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.356+0000 7f1ac77fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:01:50.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.356+0000 7f1ac77fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:01:50.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.356+0000 7f1ac77fe640 1 -- 172.21.15.42:0/4011149115 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1ae009e720 msgr2=0x7f1ae009eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:01:50.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.356+0000 7f1ac77fe640 1 --2- 172.21.15.42:0/4011149115 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1ae009e720 0x7f1ae009eb30 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f1ae00745f0 tx=0x7f1ad0004ac0 comp rx=0 tx=0).stop 2024-09-16T04:01:50.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.356+0000 7f1ac77fe640 1 -- 172.21.15.42:0/4011149115 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1ae009e720 msgr2=0x7f1ae009eb30 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:01:50.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.356+0000 7f1ac77fe640 1 --2- 172.21.15.42:0/4011149115 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1ae009e720 0x7f1ae009eb30 secure :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f1ae00745f0 tx=0x7f1ad0004ac0 comp rx=0 tx=0).stop 2024-09-16T04:01:50.385 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T04:01:50.385 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:01:50.385 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:01:50.385 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T04:01:50.385 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 178, 2024-09-16T04:01:50.385 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 17915, 2024-09-16T04:01:50.385 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T04:01:50.385 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T04:01:50.385 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T04:01:50.385 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T04:01:50.385 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:01:50.385 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 6, 2024-09-16T04:01:50.386 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:01:50.386 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T04:01:50.386 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T04:01:50.386 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:01:50.386 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:01:50.386 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:01:50.386 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:01:50.386 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:01:50.386 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:01:50.386 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:01:50.386 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:01:50.386 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:01:50.386 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:01:50.386 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:01:50.387 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:01:50.387 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:01:50.387 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T04:01:50.387 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:01:50.387 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:01:50.387 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:01:50.387 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:01:50.387 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:01:50.387 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:01:50.387 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:01:50.387 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:01:50.387 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:01:50.387 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:01:50.387 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:01:50.388 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:01:50.388 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:01:50.388 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T04:01:50.388 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T04:01:50.388 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T04:01:50.388 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T04:01:50.388 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T04:01:50.388 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T04:01:50.388 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:01:50.388 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:01:50.388 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T04:01:50.388 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T04:01:50.388 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T04:01:50.389 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T04:01:50.389 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T04:01:50.389 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T04:01:50.389 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T04:01:50.389 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:01:50.389 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T04:01:50.389 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:01:50.389 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:01:50.389 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:01:50.389 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:01:50.390 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:01:50.390 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:01:50.390 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:01:50.390 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:01:50.390 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:01:50.390 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:01:50.390 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:01:50.390 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:01:50.390 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T04:01:50.390 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:01:50.390 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T04:01:50.390 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:01:50.390 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:01:50.391 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:01:50.391 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:01:50.391 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:01:50.391 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:01:50.391 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:01:50.391 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T04:01:50.391 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:01:50.391 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:01:50.391 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:01:50.391 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:01:50.391 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T04:01:50.391 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:01:50.391 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:01:50.391 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:01:50.392 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:01:50.392 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T04:01:50.392 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T04:01:50.392 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:01:50.392 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:01:50.392 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:01:50.392 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:01:50.392 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:01:50.392 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:01:50.392 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:01:50.392 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:01:50.392 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:01:50.392 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:01:50.392 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:01:50.393 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T04:01:50.393 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:01:50.393 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:01:50.393 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:01:50.393 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:01:50.393 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T04:01:50.393 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:01:50.393 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:01:50.393 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:01:50.393 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:01:50.393 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T04:01:50.393 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T04:01:50.393 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:01:50.393 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:01:50.394 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:01:50.394 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:01:50.394 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:01:50.394 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:01:50.394 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:01:50.394 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:01:50.394 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:01:50.394 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:01:50.394 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:01:50.394 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T04:01:50.394 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:01:50.394 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:01:50.394 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:01:50.394 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:01:50.395 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T04:01:50.395 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:01:50.395 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:01:50.395 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:01:50.395 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:01:50.395 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T04:01:50.395 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T04:01:50.395 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:01:50.395 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:01:50.395 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:01:50.395 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:01:50.396 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:01:50.396 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:01:50.396 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T04:01:50.396 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T04:01:50.396 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:01:50.396 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:01:50.396 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:01:50.396 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:01:50.396 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:01:50.396 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:01:50.396 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T04:01:50.396 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:01:50.396 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:01:50.396 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:01:50.397 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:01:50.397 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:01:50.397 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:01:50.397 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:01:50.397 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T04:01:50.397 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T04:01:50.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.360+0000 7f1ae5269640 10 librados: watch_flush enter 2024-09-16T04:01:50.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.360+0000 7f1ae5269640 10 librados: watch_flush exit 2024-09-16T04:01:50.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.360+0000 7f1ae5269640 20 client.825441.objecter shutdown clearing up homeless session... 2024-09-16T04:01:50.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.360+0000 7f1ae5269640 10 client.825441.objecter successfully canceled tick 2024-09-16T04:01:50.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.360+0000 7f1ae5269640 1 -- 172.21.15.42:0/4011149115 >> v2:172.21.15.154:6812/254845342 conn(0x7f1acc037840 msgr2=0x7f1acc039d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:01:50.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.360+0000 7f1ae5269640 1 --2- 172.21.15.42:0/4011149115 >> v2:172.21.15.154:6812/254845342 conn(0x7f1acc037840 0x7f1acc039d00 secure :-1 s=READY pgs=3771 cs=0 l=1 rev1=1 crypto rx=0x7f1ae00a3bd0 tx=0x7f1ad8009040 comp rx=0 tx=0).stop 2024-09-16T04:01:50.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.360+0000 7f1ae5269640 10 monclient: shutdown 2024-09-16T04:01:50.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.360+0000 7f1ae5269640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:01:50.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.360+0000 7f1ae5269640 1 -- 172.21.15.42:0/4011149115 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1ae0120740 msgr2=0x7f1ae0120b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:01:50.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.360+0000 7f1ae5269640 1 --2- 172.21.15.42:0/4011149115 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1ae0120740 0x7f1ae0120b80 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f1ad401ca70 tx=0x7f1ac8005840 comp rx=0 tx=0).stop 2024-09-16T04:01:50.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.360+0000 7f1adf7fe640 1 -- 172.21.15.42:0/4011149115 reap_dead start 2024-09-16T04:01:50.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.360+0000 7f1ae5269640 1 -- 172.21.15.42:0/4011149115 shutdown_connections 2024-09-16T04:01:50.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.360+0000 7f1ae5269640 1 --2- 172.21.15.42:0/4011149115 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1ae0120740 0x7f1ae0120b80 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:50.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.360+0000 7f1ae5269640 1 -- 172.21.15.42:0/4011149115 >> 172.21.15.42:0/4011149115 conn(0x7f1ae00a9fb0 msgr2=0x7f1ae00a8970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:01:50.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.360+0000 7f1ae5269640 1 -- 172.21.15.42:0/4011149115 shutdown_connections 2024-09-16T04:01:50.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.360+0000 7f1ae5269640 1 -- 172.21.15.42:0/4011149115 wait complete. 2024-09-16T04:01:50.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.360+0000 7f1ae5269640 1 librados: shutdown 2024-09-16T04:01:50.400 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:01:50.400 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T04:01:50.400 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T04:01:50.529 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.524+0000 7fe206c96640 10 monclient: get_monmap_and_config 2024-09-16T04:01:50.529 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.524+0000 7fe206c96640 10 monclient: build_initial_monmap 2024-09-16T04:01:50.529 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.524+0000 7fe206c96640 10 monclient: monmap: 2024-09-16T04:01:50.529 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:01:50.529 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:01:50.529 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:01:50.527662+0000 2024-09-16T04:01:50.530 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:01:50.527662+0000 2024-09-16T04:01:50.530 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:01:50.530 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:01:50.530 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:01:50.530 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:01:50.530 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:01:50.530 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:01:50.530 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:01:50.530 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:01:50.531 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.528+0000 7fe206c96640 1 Processor -- start 2024-09-16T04:01:50.532 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.528+0000 7fe206c96640 1 -- start start 2024-09-16T04:01:50.532 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.528+0000 7fe206c96640 10 monclient: init 2024-09-16T04:01:50.532 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.528+0000 7fe206c96640 10 monclient: _reopen_session rank -1 2024-09-16T04:01:50.532 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.528+0000 7fe206c96640 10 monclient: _add_conns ranks=[2,4,3,1,0] 2024-09-16T04:01:50.532 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.528+0000 7fe206c96640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe2000a39b0 0x7fe200133b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:01:50.532 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.528+0000 7fe206c96640 10 monclient(hunting): picked mon.noname-b con 0x7fe2000a39b0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:01:50.532 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.528+0000 7fe206c96640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fe2000a3d90 0x7fe200137000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:01:50.532 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.528+0000 7fe206c96640 10 monclient(hunting): picked mon.noname-c con 0x7fe2000a3d90 addr v2:172.21.15.154:3300/0 2024-09-16T04:01:50.532 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.528+0000 7fe206c96640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe200137540 0x7fe200137950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:01:50.532 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.528+0000 7fe206c96640 10 monclient(hunting): picked mon.noname-a con 0x7fe200137540 addr v2:172.21.15.42:3300/0 2024-09-16T04:01:50.532 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.528+0000 7fe206c96640 10 monclient(hunting): start opening mon connection 2024-09-16T04:01:50.533 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.528+0000 7fe206c96640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7fe200137e90 con 0x7fe200137540 2024-09-16T04:01:50.533 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.528+0000 7fe206c96640 10 monclient(hunting): start opening mon connection 2024-09-16T04:01:50.533 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.528+0000 7fe206c96640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fe20009c140 con 0x7fe2000a3d90 2024-09-16T04:01:50.533 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.528+0000 7fe206c96640 10 monclient(hunting): start opening mon connection 2024-09-16T04:01:50.533 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.528+0000 7fe206c96640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fe20009c420 con 0x7fe2000a39b0 2024-09-16T04:01:50.533 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.528+0000 7fe206c96640 10 monclient(hunting): _renew_subs 2024-09-16T04:01:50.533 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.528+0000 7fe206c96640 10 monclient(hunting): authenticate will time out at 2157.497085s 2024-09-16T04:01:50.533 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.528+0000 7fe205233640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe200137540 0x7fe200137950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:01:50.533 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.528+0000 7fe204a32640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe2000a39b0 0x7fe200133b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:01:50.533 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.528+0000 7fe205233640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe200137540 0x7fe200137950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:35394/0 (socket says 172.21.15.42:35394) 2024-09-16T04:01:50.533 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.528+0000 7fe205233640 1 -- 172.21.15.42:0/240343164 learned_addr learned my addr 172.21.15.42:0/240343164 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:01:50.533 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.528+0000 7fe205233640 10 monclient(hunting): get_auth_request con 0x7fe200137540 auth_method 0 2024-09-16T04:01:50.533 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.528+0000 7fe205233640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:01:50.534 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.528+0000 7fe205233640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:01:50.534 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.528+0000 7fe205233640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:01:50.534 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.528+0000 7fe204a32640 10 monclient(hunting): get_auth_request con 0x7fe2000a39b0 auth_method 0 2024-09-16T04:01:50.534 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.528+0000 7fe204a32640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:01:50.534 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.528+0000 7fe204a32640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:01:50.534 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.528+0000 7fe204a32640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:01:50.534 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.528+0000 7fe205233640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:01:50.534 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.528+0000 7fe205233640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:01:50.534 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.528+0000 7fe205233640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:01:50.534 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.528+0000 7fe204a32640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:01:50.534 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.528+0000 7fe204a32640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:01:50.534 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.528+0000 7fe204a32640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:01:50.534 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.528+0000 7fe1fffff640 1 --2- 172.21.15.42:0/240343164 >> v2:172.21.15.154:3300/0 conn(0x7fe2000a3d90 0x7fe200137000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:01:50.534 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.528+0000 7fe1fffff640 10 monclient(hunting): get_auth_request con 0x7fe2000a3d90 auth_method 0 2024-09-16T04:01:50.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.528+0000 7fe1fffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:01:50.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.528+0000 7fe1fffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:01:50.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.528+0000 7fe1fffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:01:50.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.528+0000 7fe205233640 10 monclient(hunting): handle_auth_done global_id 825444 payload 274 2024-09-16T04:01:50.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.528+0000 7fe205233640 1 -- 172.21.15.42:0/240343164 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe2000a39b0 msgr2=0x7fe200133b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:01:50.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.528+0000 7fe205233640 1 --2- 172.21.15.42:0/240343164 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe2000a39b0 0x7fe200133b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:50.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.528+0000 7fe205233640 1 -- 172.21.15.42:0/240343164 >> v2:172.21.15.154:3300/0 conn(0x7fe2000a3d90 msgr2=0x7fe200137000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:01:50.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.528+0000 7fe205233640 1 --2- 172.21.15.42:0/240343164 >> v2:172.21.15.154:3300/0 conn(0x7fe2000a3d90 0x7fe200137000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:50.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.528+0000 7fe205233640 10 monclient: _finish_hunting 0 2024-09-16T04:01:50.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.528+0000 7fe205233640 1 monclient: found mon.noname-a 2024-09-16T04:01:50.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.528+0000 7fe205233640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:01:50.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.528+0000 7fe205233640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:01:50.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.528+0000 7fe205233640 1 -- 172.21.15.42:0/240343164 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe20006dd60 con 0x7fe200137540 2024-09-16T04:01:50.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.528+0000 7fe204a32640 1 --2- 172.21.15.42:0/240343164 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe2000a39b0 0x7fe200133b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:01:50.541 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.528+0000 7fe205233640 1 --2- 172.21.15.42:0/240343164 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe200137540 0x7fe200137950 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fe20006e3c0 tx=0x7fe1f801c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82a6b6ea13ccef33 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:01:50.541 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.532+0000 7fe1ff7fe640 1 -- 172.21.15.42:0/240343164 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1f80a5530 con 0x7fe200137540 2024-09-16T04:01:50.541 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.532+0000 7fe1ff7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:01:50.541 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.532+0000 7fe1ff7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:01:50.541 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.532+0000 7fe1ff7fe640 10 monclient: dump: 2024-09-16T04:01:50.541 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:01:50.542 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:01:50.542 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:50.542 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:50.542 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:01:50.542 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:01:50.542 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:01:50.542 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:01:50.542 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:01:50.542 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:01:50.542 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.532+0000 7fe1ff7fe640 10 monclient: _finish_auth 0 2024-09-16T04:01:50.542 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.532+0000 7fe1ff7fe640 10 monclient: _check_auth_tickets 2024-09-16T04:01:50.542 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.532+0000 7fe1ff7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:01:50.542 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.532+0000 7fe1ff7fe640 1 -- 172.21.15.42:0/240343164 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe1f80a56d0 con 0x7fe200137540 2024-09-16T04:01:50.543 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.532+0000 7fe206c96640 5 monclient: authenticate success, global_id 825444 2024-09-16T04:01:50.543 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.532+0000 7fe206c96640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:01:50.543 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.532+0000 7fe1ff7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:01:50.543 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.532+0000 7fe206c96640 10 monclient: get_monmap_and_config success 2024-09-16T04:01:50.543 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.532+0000 7fe1ff7fe640 1 -- 172.21.15.42:0/240343164 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1f80a5aa0 con 0x7fe200137540 2024-09-16T04:01:50.543 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.532+0000 7fe1ff7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:01:50.543 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.532+0000 7fe206c96640 4 set_mon_vals no callback set 2024-09-16T04:01:50.543 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.532+0000 7fe1ff7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:01:50.543 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.532+0000 7fe1ff7fe640 10 monclient: dump: 2024-09-16T04:01:50.543 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:01:50.543 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:01:50.543 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:50.543 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:50.543 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:01:50.544 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:01:50.544 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:01:50.544 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:01:50.544 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:01:50.544 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:01:50.544 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.532+0000 7fe206c96640 10 monclient: shutdown 2024-09-16T04:01:50.544 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.532+0000 7fe206c96640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:01:50.544 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.532+0000 7fe206c96640 1 -- 172.21.15.42:0/240343164 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe200137540 msgr2=0x7fe200137950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:01:50.544 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.532+0000 7fe206c96640 1 --2- 172.21.15.42:0/240343164 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe200137540 0x7fe200137950 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fe20006e3c0 tx=0x7fe1f801c8b0 comp rx=0 tx=0).stop 2024-09-16T04:01:50.544 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.532+0000 7fe206c96640 1 -- 172.21.15.42:0/240343164 shutdown_connections 2024-09-16T04:01:50.544 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.532+0000 7fe206c96640 1 --2- 172.21.15.42:0/240343164 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe200137540 0x7fe200137950 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:50.544 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.532+0000 7fe206c96640 1 --2- 172.21.15.42:0/240343164 >> v2:172.21.15.154:3300/0 conn(0x7fe2000a3d90 0x7fe200137000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:50.544 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.532+0000 7fe206c96640 1 --2- 172.21.15.42:0/240343164 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe2000a39b0 0x7fe200133b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:50.545 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.532+0000 7fe206c96640 1 -- 172.21.15.42:0/240343164 >> 172.21.15.42:0/240343164 conn(0x7fe2000a9fb0 msgr2=0x7fe2000aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:01:50.545 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.532+0000 7fe206c96640 1 -- 172.21.15.42:0/240343164 shutdown_connections 2024-09-16T04:01:50.545 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.532+0000 7fe206c96640 1 -- 172.21.15.42:0/240343164 wait complete. 2024-09-16T04:01:50.545 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.532+0000 7fe206c96640 10 monclient: build_initial_monmap 2024-09-16T04:01:50.545 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.532+0000 7fe206c96640 10 monclient: monmap: 2024-09-16T04:01:50.545 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:01:50.545 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:01:50.545 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:01:50.545 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:01:50.545 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:01:50.545 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:01:50.545 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:01:50.545 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:01:50.546 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:01:50.546 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:01:50.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.532+0000 7fe206c96640 1 librados: starting msgr at 2024-09-16T04:01:50.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.532+0000 7fe206c96640 1 librados: starting objecter 2024-09-16T04:01:50.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.532+0000 7fe206c96640 1 Processor -- start 2024-09-16T04:01:50.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.532+0000 7fe206c96640 1 -- start start 2024-09-16T04:01:50.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.532+0000 7fe206c96640 1 librados: setting wanted keys 2024-09-16T04:01:50.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.532+0000 7fe206c96640 1 librados: calling monclient init 2024-09-16T04:01:50.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.532+0000 7fe206c96640 10 monclient: init 2024-09-16T04:01:50.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.532+0000 7fe206c96640 10 monclient: _reopen_session rank -1 2024-09-16T04:01:50.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.532+0000 7fe206c96640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:01:50.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.532+0000 7fe206c96640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe2000a39b0 0x7fe20013b530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:01:50.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.532+0000 7fe206c96640 10 monclient(hunting): picked mon.noname-a con 0x7fe2000a39b0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:01:50.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.532+0000 7fe206c96640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe2000a3d90 0x7fe20013ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:01:50.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.532+0000 7fe206c96640 10 monclient(hunting): picked mon.noname-b con 0x7fe2000a3d90 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:01:50.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.532+0000 7fe206c96640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe200137540 0x7fe2001cf1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:01:50.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.532+0000 7fe206c96640 10 monclient(hunting): picked mon.noname-c con 0x7fe200137540 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:01:50.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.532+0000 7fe206c96640 10 monclient(hunting): start opening mon connection 2024-09-16T04:01:50.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.532+0000 7fe206c96640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fe200140dc0 con 0x7fe2000a39b0 2024-09-16T04:01:50.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.532+0000 7fe206c96640 10 monclient(hunting): start opening mon connection 2024-09-16T04:01:50.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.532+0000 7fe206c96640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe200140f40 con 0x7fe2000a3d90 2024-09-16T04:01:50.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.532+0000 7fe206c96640 10 monclient(hunting): start opening mon connection 2024-09-16T04:01:50.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.532+0000 7fe206c96640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fe2001410c0 con 0x7fe200137540 2024-09-16T04:01:50.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.532+0000 7fe206c96640 10 monclient(hunting): _renew_subs 2024-09-16T04:01:50.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.532+0000 7fe206c96640 10 monclient(hunting): authenticate will time out at 2157.502532s 2024-09-16T04:01:50.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.532+0000 7fe204a32640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe2000a39b0 0x7fe20013b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:01:50.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.532+0000 7fe204a32640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe2000a39b0 0x7fe20013b530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:35408/0 (socket says 172.21.15.42:35408) 2024-09-16T04:01:50.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.532+0000 7fe204a32640 1 -- 172.21.15.42:0/3301785913 learned_addr learned my addr 172.21.15.42:0/3301785913 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:01:50.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.532+0000 7fe204a32640 10 monclient(hunting): get_auth_request con 0x7fe2000a39b0 auth_method 0 2024-09-16T04:01:50.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.532+0000 7fe204a32640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:01:50.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.532+0000 7fe204a32640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:01:50.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.532+0000 7fe204a32640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:01:50.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.532+0000 7fe205233640 1 --2- 172.21.15.42:0/3301785913 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe200137540 0x7fe2001cf1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:01:50.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.532+0000 7fe205233640 10 monclient(hunting): get_auth_request con 0x7fe200137540 auth_method 0 2024-09-16T04:01:50.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.532+0000 7fe205233640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:01:50.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.532+0000 7fe205233640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:01:50.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.532+0000 7fe205233640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:01:50.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.532+0000 7fe205233640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:01:50.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.532+0000 7fe205233640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:01:50.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.532+0000 7fe205233640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:01:50.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.532+0000 7fe204a32640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:01:50.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.532+0000 7fe204a32640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:01:50.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.532+0000 7fe204a32640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:01:50.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.532+0000 7fe1fffff640 1 --2- 172.21.15.42:0/3301785913 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe2000a3d90 0x7fe20013ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:01:50.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.532+0000 7fe205233640 10 monclient(hunting): handle_auth_done global_id 954137 payload 824 2024-09-16T04:01:50.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.532+0000 7fe205233640 1 -- 172.21.15.42:0/3301785913 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe2000a3d90 msgr2=0x7fe20013ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:01:50.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.532+0000 7fe205233640 1 --2- 172.21.15.42:0/3301785913 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe2000a3d90 0x7fe20013ba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:50.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.532+0000 7fe205233640 1 -- 172.21.15.42:0/3301785913 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe2000a39b0 msgr2=0x7fe20013b530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:01:50.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.532+0000 7fe205233640 1 --2- 172.21.15.42:0/3301785913 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe2000a39b0 0x7fe20013b530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:50.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.532+0000 7fe205233640 10 monclient: _finish_hunting 0 2024-09-16T04:01:50.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.532+0000 7fe205233640 1 monclient: found mon.noname-c 2024-09-16T04:01:50.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.532+0000 7fe205233640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:01:50.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.532+0000 7fe205233640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T04:01:50.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.532+0000 7fe205233640 1 -- 172.21.15.42:0/3301785913 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2000a4e20 con 0x7fe200137540 2024-09-16T04:01:50.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.532+0000 7fe1fffff640 10 monclient: get_auth_request con 0x7fe2000a3d90 auth_method 0 2024-09-16T04:01:50.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.532+0000 7fe1fffff640 1 --2- 172.21.15.42:0/3301785913 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe2000a3d90 0x7fe20013ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T04:01:50.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.532+0000 7fe205233640 1 --2- 172.21.15.42:0/3301785913 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe200137540 0x7fe2001cf1e0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fe2000766a0 tx=0x7fe1f8019650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:01:50.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.532+0000 7fe1fd7fa640 10 client.?.objecter ms_handle_connect 0x7fe200137540 2024-09-16T04:01:50.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.532+0000 7fe1fd7fa640 10 client.?.objecter resend_mon_ops 2024-09-16T04:01:50.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.532+0000 7fe1fd7fa640 1 -- 172.21.15.42:0/3301785913 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1f80bc030 con 0x7fe200137540 2024-09-16T04:01:50.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.532+0000 7fe1fd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:01:50.551 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.532+0000 7fe1fd7fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:01:50.551 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.532+0000 7fe1fd7fa640 10 monclient: dump: 2024-09-16T04:01:50.551 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:01:50.551 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:01:50.551 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:50.551 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:50.551 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:01:50.551 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:01:50.551 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:01:50.551 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:01:50.551 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:01:50.551 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:01:50.551 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.532+0000 7fe1fd7fa640 10 monclient: _finish_auth 0 2024-09-16T04:01:50.552 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.532+0000 7fe1fd7fa640 10 monclient: _check_auth_tickets 2024-09-16T04:01:50.552 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.532+0000 7fe1fd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:01:50.552 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.532+0000 7fe1fd7fa640 1 -- 172.21.15.42:0/3301785913 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe1f80b8470 con 0x7fe200137540 2024-09-16T04:01:50.552 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.532+0000 7fe1fd7fa640 10 monclient: handle_config config(0 keys) 2024-09-16T04:01:50.552 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.532+0000 7fe1fd7fa640 1 -- 172.21.15.42:0/3301785913 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1f8009510 con 0x7fe200137540 2024-09-16T04:01:50.552 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.532+0000 7fe1fd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:01:50.552 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.532+0000 7fe1fd7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:01:50.552 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.532+0000 7fe1fd7fa640 10 monclient: dump: 2024-09-16T04:01:50.552 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:01:50.552 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:01:50.552 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:50.553 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:50.553 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:01:50.553 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:01:50.553 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:01:50.553 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:01:50.553 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:01:50.553 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:01:50.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.532+0000 7fe1feffd640 4 set_mon_vals no callback set 2024-09-16T04:01:50.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.536+0000 7fe206c96640 5 monclient: authenticate success, global_id 954137 2024-09-16T04:01:50.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.536+0000 7fe206c96640 10 monclient: _renew_subs 2024-09-16T04:01:50.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.536+0000 7fe206c96640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:01:50.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.536+0000 7fe206c96640 1 -- 172.21.15.42:0/3301785913 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe20006f680 con 0x7fe200137540 2024-09-16T04:01:50.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.536+0000 7fe206c96640 10 client.954137.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:01:50.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.536+0000 7fe206c96640 10 monclient: _renew_subs 2024-09-16T04:01:50.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.536+0000 7fe206c96640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:01:50.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.536+0000 7fe206c96640 1 -- 172.21.15.42:0/3301785913 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fe20009cd50 con 0x7fe200137540 2024-09-16T04:01:50.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.536+0000 7fe206c96640 1 librados: init done 2024-09-16T04:01:50.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.536+0000 7fe1fd7fa640 1 -- 172.21.15.42:0/3301785913 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fe1f80bf070 con 0x7fe200137540 2024-09-16T04:01:50.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.536+0000 7fe206c96640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:01:50.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.536+0000 7fe206c96640 1 --2- 172.21.15.42:0/3301785913 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe2001cfee0 0x7fe20002c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:01:50.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.536+0000 7fe206c96640 10 monclient(hunting): start opening mon connection 2024-09-16T04:01:50.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.536+0000 7fe206c96640 1 -- 172.21.15.42:0/3301785913 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe2001342d0 con 0x7fe2001cfee0 2024-09-16T04:01:50.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.536+0000 7fe1fd7fa640 1 --2- 172.21.15.42:0/3301785913 >> v2:172.21.15.154:6812/254845342 conn(0x7fe1e4037810 0x7fe1e4039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:01:50.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.536+0000 7fe1fd7fa640 1 -- 172.21.15.42:0/3301785913 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(602..602 src has 451..602) ==== 6252+0+0 (secure 0 0 0) 0x7fe1f80e3a90 con 0x7fe200137540 2024-09-16T04:01:50.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.536+0000 7fe1fd7fa640 10 client.954137.objecter ms_dispatch 0x7fe200000ba0 osd_map(602..602 src has 451..602) 2024-09-16T04:01:50.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.536+0000 7fe1fd7fa640 3 client.954137.objecter handle_osd_map got epochs [602,602] > 0 2024-09-16T04:01:50.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.536+0000 7fe1fd7fa640 3 client.954137.objecter handle_osd_map decoding full epoch 602 2024-09-16T04:01:50.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.536+0000 7fe1fd7fa640 20 client.954137.objecter dump_active .. 0 homeless 2024-09-16T04:01:50.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.536+0000 7fe204a32640 1 --2- 172.21.15.42:0/3301785913 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe2001cfee0 0x7fe20002c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:01:50.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.536+0000 7fe204a32640 10 monclient: get_auth_request con 0x7fe2001cfee0 auth_method 0 2024-09-16T04:01:50.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.536+0000 7fe204a32640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:01:50.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.536+0000 7fe204a32640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:01:50.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.536+0000 7fe204a32640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:01:50.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.536+0000 7fe1fffff640 1 --2- 172.21.15.42:0/3301785913 >> v2:172.21.15.154:6812/254845342 conn(0x7fe1e4037810 0x7fe1e4039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:01:50.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.536+0000 7fe1fffff640 10 monclient: get_auth_request con 0x7fe1e4037810 auth_method 0 2024-09-16T04:01:50.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.536+0000 7fe204a32640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:01:50.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.536+0000 7fe204a32640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:01:50.556 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.536+0000 7fe204a32640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:01:50.556 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.536+0000 7fe204a32640 10 monclient(hunting): handle_auth_done global_id 904546 payload 824 2024-09-16T04:01:50.556 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.536+0000 7fe204a32640 1 -- 172.21.15.42:0/3301785913 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe200137000 con 0x7fe2001cfee0 2024-09-16T04:01:50.556 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.536+0000 7fe1fd7fa640 10 client.954137.objecter ms_handle_connect 0x7fe2001cfee0 2024-09-16T04:01:50.556 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.536+0000 7fe1fd7fa640 10 client.954137.objecter resend_mon_ops 2024-09-16T04:01:50.557 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.536+0000 7fe204a32640 1 --2- 172.21.15.42:0/3301785913 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe2001cfee0 0x7fe20002c650 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fe1f4011c00 tx=0x7fe1f400f610 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:01:50.557 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.536+0000 7fe1fd7fa640 10 client.954137.objecter ms_handle_connect 0x7fe1e4037810 2024-09-16T04:01:50.557 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.536+0000 7fe1fffff640 1 --2- 172.21.15.42:0/3301785913 >> v2:172.21.15.154:6812/254845342 conn(0x7fe1e4037810 0x7fe1e4039cd0 secure :-1 s=READY pgs=3772 cs=0 l=1 rev1=1 crypto rx=0x7fe200133090 tx=0x7fe1ec008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:01:50.557 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.536+0000 7fe1fd7fa640 1 -- 172.21.15.42:0/3301785913 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1f4011cf0 con 0x7fe2001cfee0 2024-09-16T04:01:50.557 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.536+0000 7fe1fd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:01:50.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.536+0000 7fe1fd7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:01:50.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.536+0000 7fe1fd7fa640 10 monclient: dump: 2024-09-16T04:01:50.558 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:01:50.558 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:01:50.558 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:50.558 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:50.558 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:01:50.558 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:01:50.558 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:01:50.558 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:01:50.558 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:01:50.558 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:01:50.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.536+0000 7fe1fd7fa640 1 -- 172.21.15.42:0/3301785913 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fe1f4011cf0 con 0x7fe2001cfee0 2024-09-16T04:01:50.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.536+0000 7fe1fd7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:01:50.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.536+0000 7fe1fd7fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:01:50.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.536+0000 7fe1fd7fa640 1 -- 172.21.15.42:0/3301785913 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe2001cfee0 msgr2=0x7fe20002c650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:01:50.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.536+0000 7fe1fd7fa640 1 --2- 172.21.15.42:0/3301785913 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe2001cfee0 0x7fe20002c650 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fe1f4011c00 tx=0x7fe1f400f610 comp rx=0 tx=0).stop 2024-09-16T04:01:50.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.536+0000 7fe1fd7fa640 1 -- 172.21.15.42:0/3301785913 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe2001cfee0 msgr2=0x7fe20002c650 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:01:50.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.536+0000 7fe1fd7fa640 1 --2- 172.21.15.42:0/3301785913 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe2001cfee0 0x7fe20002c650 secure :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fe1f4011c00 tx=0x7fe1f400f610 comp rx=0 tx=0).stop 2024-09-16T04:01:50.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.544+0000 7fe206c96640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:01:50.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.544+0000 7fe206c96640 1 --2- 172.21.15.42:0/3301785913 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe20002cb90 0x7fe20002cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:01:50.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.544+0000 7fe206c96640 10 monclient(hunting): start opening mon connection 2024-09-16T04:01:50.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.544+0000 7fe206c96640 1 -- 172.21.15.42:0/3301785913 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe200130c10 con 0x7fe20002cb90 2024-09-16T04:01:50.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.544+0000 7fe204a32640 1 --2- 172.21.15.42:0/3301785913 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe20002cb90 0x7fe20002cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:01:50.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.544+0000 7fe204a32640 10 monclient: get_auth_request con 0x7fe20002cb90 auth_method 0 2024-09-16T04:01:50.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.544+0000 7fe204a32640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:01:50.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.544+0000 7fe204a32640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:01:50.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.544+0000 7fe204a32640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:01:50.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.544+0000 7fe204a32640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:01:50.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.544+0000 7fe204a32640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:01:50.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.544+0000 7fe204a32640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:01:50.562 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.544+0000 7fe204a32640 10 monclient(hunting): handle_auth_done global_id 904549 payload 824 2024-09-16T04:01:50.562 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.544+0000 7fe204a32640 1 -- 172.21.15.42:0/3301785913 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fe200068e40 con 0x7fe20002cb90 2024-09-16T04:01:50.562 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.544+0000 7fe204a32640 1 --2- 172.21.15.42:0/3301785913 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe20002cb90 0x7fe20002cfa0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fe1f4011c00 tx=0x7fe1f40046d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:01:50.562 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.544+0000 7fe1fd7fa640 10 client.954137.objecter ms_handle_connect 0x7fe20002cb90 2024-09-16T04:01:50.562 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.544+0000 7fe1fd7fa640 10 client.954137.objecter resend_mon_ops 2024-09-16T04:01:50.562 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.544+0000 7fe1fd7fa640 1 -- 172.21.15.42:0/3301785913 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1f4004020 con 0x7fe20002cb90 2024-09-16T04:01:50.562 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.544+0000 7fe1fd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:01:50.562 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.544+0000 7fe1fd7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:01:50.562 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.544+0000 7fe1fd7fa640 10 monclient: dump: 2024-09-16T04:01:50.562 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:01:50.563 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:01:50.563 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:50.563 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:50.563 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:01:50.563 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:01:50.563 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:01:50.563 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:01:50.563 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:01:50.563 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:01:50.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.548+0000 7fe1fd7fa640 1 -- 172.21.15.42:0/3301785913 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4880 (secure 0 0 0) 0x7fe1f4004020 con 0x7fe20002cb90 2024-09-16T04:01:50.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.548+0000 7fe1fd7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:01:50.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.548+0000 7fe1fd7fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:01:50.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.548+0000 7fe1fd7fa640 1 -- 172.21.15.42:0/3301785913 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe20002cb90 msgr2=0x7fe20002cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:01:50.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.548+0000 7fe1fd7fa640 1 --2- 172.21.15.42:0/3301785913 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe20002cb90 0x7fe20002cfa0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fe1f4011c00 tx=0x7fe1f40046d0 comp rx=0 tx=0).stop 2024-09-16T04:01:50.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.548+0000 7fe1fd7fa640 1 -- 172.21.15.42:0/3301785913 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe20002cb90 msgr2=0x7fe20002cfa0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:01:50.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.548+0000 7fe1fd7fa640 1 --2- 172.21.15.42:0/3301785913 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe20002cb90 0x7fe20002cfa0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:50.564 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T04:01:50.564 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:01:50.564 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:01:50.564 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T04:01:50.564 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 178, 2024-09-16T04:01:50.564 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 238728, 2024-09-16T04:01:50.564 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T04:01:50.564 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T04:01:50.565 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T04:01:50.565 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T04:01:50.565 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:01:50.565 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 6, 2024-09-16T04:01:50.565 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:01:50.565 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T04:01:50.565 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T04:01:50.565 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:01:50.565 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:01:50.565 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:01:50.565 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:01:50.565 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:01:50.565 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:01:50.566 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:01:50.566 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:01:50.566 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:01:50.566 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:01:50.566 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:01:50.566 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:01:50.566 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:01:50.566 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T04:01:50.566 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:01:50.566 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:01:50.566 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:01:50.567 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:01:50.567 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:01:50.567 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:01:50.567 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:01:50.567 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:01:50.567 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:01:50.567 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:01:50.567 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:01:50.567 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:01:50.567 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:01:50.568 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T04:01:50.568 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T04:01:50.568 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T04:01:50.568 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T04:01:50.568 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T04:01:50.568 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T04:01:50.568 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:01:50.568 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:01:50.568 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T04:01:50.568 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T04:01:50.568 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T04:01:50.568 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T04:01:50.568 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T04:01:50.568 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T04:01:50.569 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T04:01:50.569 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:01:50.569 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T04:01:50.569 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:01:50.569 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:01:50.569 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:01:50.569 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:01:50.569 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:01:50.569 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:01:50.569 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:01:50.569 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:01:50.569 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:01:50.569 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:01:50.570 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:01:50.570 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:01:50.570 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T04:01:50.570 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:01:50.570 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T04:01:50.570 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:01:50.570 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:01:50.570 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:01:50.570 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:01:50.570 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:01:50.570 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:01:50.570 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:01:50.570 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T04:01:50.570 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:01:50.571 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:01:50.571 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:01:50.571 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:01:50.571 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T04:01:50.571 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:01:50.571 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:01:50.571 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:01:50.571 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:01:50.571 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T04:01:50.571 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T04:01:50.571 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:01:50.571 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:01:50.571 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:01:50.571 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:01:50.572 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:01:50.572 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:01:50.572 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:01:50.572 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:01:50.572 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:01:50.572 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:01:50.572 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:01:50.572 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T04:01:50.572 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:01:50.572 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:01:50.572 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:01:50.572 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:01:50.572 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T04:01:50.572 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:01:50.573 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:01:50.573 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:01:50.573 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:01:50.573 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T04:01:50.573 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T04:01:50.573 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:01:50.573 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:01:50.573 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:01:50.573 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:01:50.573 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:01:50.573 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:01:50.573 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:01:50.573 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:01:50.573 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:01:50.574 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:01:50.574 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:01:50.574 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T04:01:50.574 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:01:50.574 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:01:50.574 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:01:50.574 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:01:50.574 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T04:01:50.574 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:01:50.574 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:01:50.574 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:01:50.574 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:01:50.574 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T04:01:50.575 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T04:01:50.575 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:01:50.575 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:01:50.575 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:01:50.575 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:01:50.575 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:01:50.575 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:01:50.575 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T04:01:50.575 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T04:01:50.575 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:01:50.575 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:01:50.575 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:01:50.575 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:01:50.575 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:01:50.576 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:01:50.576 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T04:01:50.576 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:01:50.576 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:01:50.576 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:01:50.576 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:01:50.576 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:01:50.576 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:01:50.576 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T04:01:50.576 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:01:50.576 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:01:50.576 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:01:50.576 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 2 2024-09-16T04:01:50.576 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:01:50.577 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:01:50.577 INFO:teuthology.orchestra.run.smithi042.stdout: "mgr": [ 2024-09-16T04:01:50.577 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:01:50.577 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:01:50.577 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:01:50.577 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:01:50.578 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:01:50.578 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:01:50.578 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:01:50.578 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T04:01:50.578 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T04:01:50.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.548+0000 7fe206c96640 10 librados: watch_flush enter 2024-09-16T04:01:50.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.548+0000 7fe206c96640 10 librados: watch_flush exit 2024-09-16T04:01:50.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.548+0000 7fe206c96640 20 client.954137.objecter shutdown clearing up homeless session... 2024-09-16T04:01:50.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.548+0000 7fe206c96640 10 client.954137.objecter successfully canceled tick 2024-09-16T04:01:50.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.548+0000 7fe206c96640 1 -- 172.21.15.42:0/3301785913 >> v2:172.21.15.154:6812/254845342 conn(0x7fe1e4037810 msgr2=0x7fe1e4039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:01:50.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.548+0000 7fe206c96640 1 --2- 172.21.15.42:0/3301785913 >> v2:172.21.15.154:6812/254845342 conn(0x7fe1e4037810 0x7fe1e4039cd0 secure :-1 s=READY pgs=3772 cs=0 l=1 rev1=1 crypto rx=0x7fe200133090 tx=0x7fe1ec008040 comp rx=0 tx=0).stop 2024-09-16T04:01:50.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.548+0000 7fe206c96640 10 monclient: shutdown 2024-09-16T04:01:50.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.548+0000 7fe206c96640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:01:50.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.548+0000 7fe206c96640 1 -- 172.21.15.42:0/3301785913 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe200137540 msgr2=0x7fe2001cf1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:01:50.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.548+0000 7fe205233640 1 -- 172.21.15.42:0/3301785913 reap_dead start 2024-09-16T04:01:50.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.548+0000 7fe206c96640 1 --2- 172.21.15.42:0/3301785913 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe200137540 0x7fe2001cf1e0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fe2000766a0 tx=0x7fe1f8019650 comp rx=0 tx=0).stop 2024-09-16T04:01:50.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.548+0000 7fe206c96640 1 -- 172.21.15.42:0/3301785913 shutdown_connections 2024-09-16T04:01:50.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.548+0000 7fe206c96640 1 --2- 172.21.15.42:0/3301785913 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe200137540 0x7fe2001cf1e0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:50.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.548+0000 7fe206c96640 1 -- 172.21.15.42:0/3301785913 >> 172.21.15.42:0/3301785913 conn(0x7fe2000a9fb0 msgr2=0x7fe200131980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:01:50.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.548+0000 7fe206c96640 1 -- 172.21.15.42:0/3301785913 shutdown_connections 2024-09-16T04:01:50.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.548+0000 7fe206c96640 1 -- 172.21.15.42:0/3301785913 wait complete. 2024-09-16T04:01:50.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.548+0000 7fe206c96640 1 librados: shutdown 2024-09-16T04:01:50.581 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:01:50.581 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-09-16T04:01:50.581 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-16T04:01:50.581 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-09-16T04:01:50.581 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-16T04:01:50.582 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-16T04:01:50.582 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T04:01:50.595 INFO:tasks.ceph.mon.c:Stopped 2024-09-16T04:01:50.595 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T04:01:50.595 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:01:50.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.748+0000 7f47309a2640 10 monclient: get_monmap_and_config 2024-09-16T04:01:50.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.748+0000 7f47309a2640 10 monclient: build_initial_monmap 2024-09-16T04:01:50.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.748+0000 7f47309a2640 10 monclient: monmap: 2024-09-16T04:01:50.752 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:01:50.752 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:01:50.753 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:01:50.750253+0000 2024-09-16T04:01:50.753 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:01:50.750253+0000 2024-09-16T04:01:50.753 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:01:50.753 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:01:50.753 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:01:50.753 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:01:50.753 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:01:50.753 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:01:50.753 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:01:50.753 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:01:50.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.748+0000 7f47309a2640 1 Processor -- start 2024-09-16T04:01:50.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.748+0000 7f47309a2640 1 -- start start 2024-09-16T04:01:50.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.748+0000 7f47309a2640 10 monclient: init 2024-09-16T04:01:50.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.748+0000 7f47309a2640 10 monclient: _reopen_session rank -1 2024-09-16T04:01:50.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.748+0000 7f47309a2640 10 monclient: _add_conns ranks=[1,0,3,4,2] 2024-09-16T04:01:50.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.748+0000 7f47309a2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f472c06db90 addr v1:172.21.15.154:6789/0 2024-09-16T04:01:50.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.748+0000 7f47309a2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f472c1340c0 addr v1:172.21.15.42:6789/0 2024-09-16T04:01:50.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.748+0000 7f47309a2640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f472c13d8b0 0x7f472c13fca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:01:50.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.748+0000 7f47309a2640 10 monclient(hunting): picked mon.noname-a con 0x7f472c13d8b0 addr v2:172.21.15.42:3300/0 2024-09-16T04:01:50.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.748+0000 7f47309a2640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f472c06ece0 con 0x7f472c1340c0 2024-09-16T04:01:50.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.748+0000 7f47309a2640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f472c09bf80 con 0x7f472c06db90 2024-09-16T04:01:50.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.748+0000 7f47309a2640 10 monclient(hunting): start opening mon connection 2024-09-16T04:01:50.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.748+0000 7f47309a2640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f472c06df70 con 0x7f472c13d8b0 2024-09-16T04:01:50.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.748+0000 7f47309a2640 10 monclient(hunting): _renew_subs 2024-09-16T04:01:50.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.748+0000 7f47309a2640 10 monclient(hunting): authenticate will time out at 2157.719840s 2024-09-16T04:01:50.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.752+0000 7f472ad9d640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f472c13d8b0 0x7f472c13fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:01:50.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.752+0000 7f4729d9b640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f472c1340c0 0x7f472c13c190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:58958/0 (socket says 172.21.15.42:58958) 2024-09-16T04:01:50.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.752+0000 7f4729d9b640 1 -- 172.21.15.42:0/858301576 learned_addr learned my addr 172.21.15.42:0/858301576 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T04:01:50.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.752+0000 7f472ad9d640 10 monclient(hunting): get_auth_request con 0x7f472c13d8b0 auth_method 0 2024-09-16T04:01:50.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.752+0000 7f472ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:01:50.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.752+0000 7f472ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:01:50.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.752+0000 7f472ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:01:50.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.752+0000 7f472ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:01:50.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.752+0000 7f472ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:01:50.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.752+0000 7f472ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:01:50.761 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.752+0000 7f472ad9d640 10 monclient(hunting): handle_auth_done global_id 825450 payload 274 2024-09-16T04:01:50.761 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.752+0000 7f472ad9d640 1 -- 172.21.15.42:0/858301576 >> v1:172.21.15.154:6789/0 conn(0x7f472c06db90 legacy=0x7f472c0a37f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:01:50.761 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.752+0000 7f472ad9d640 1 -- 172.21.15.42:0/858301576 >> v1:172.21.15.42:6789/0 conn(0x7f472c1340c0 legacy=0x7f472c13c190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:01:50.761 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.752+0000 7f472ad9d640 10 monclient: _finish_hunting 0 2024-09-16T04:01:50.761 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.752+0000 7f472ad9d640 1 monclient: found mon.noname-a 2024-09-16T04:01:50.761 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.752+0000 7f472ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:01:50.761 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.752+0000 7f472ad9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:01:50.761 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.752+0000 7f472ad9d640 1 -- 172.21.15.42:0/858301576 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f472c1344d0 con 0x7f472c13d8b0 2024-09-16T04:01:50.761 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.752+0000 7f472959a640 1 -- 172.21.15.42:0/858301576 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2427948450 0 0) 0x7f472c06ece0 con 0x7f472c1340c0 2024-09-16T04:01:50.761 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.752+0000 7f472959a640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T04:01:50.761 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.752+0000 7f472ad9d640 1 --2- 172.21.15.42:0/858301576 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f472c13d8b0 0x7f472c13fca0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f4724009d90 tx=0x7f472401d290 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bef8a2f06f55313 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:01:50.761 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.752+0000 7f472959a640 1 -- 172.21.15.42:0/858301576 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47240b2070 con 0x7f472c13d8b0 2024-09-16T04:01:50.761 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.752+0000 7f472959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:01:50.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.752+0000 7f472959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:01:50.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.752+0000 7f472959a640 10 monclient: dump: 2024-09-16T04:01:50.762 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:01:50.762 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:01:50.762 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:50.762 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:50.762 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:01:50.762 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:01:50.762 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:01:50.762 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:01:50.762 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:01:50.762 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:01:50.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.752+0000 7f472959a640 10 monclient: _finish_auth 0 2024-09-16T04:01:50.763 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.752+0000 7f472959a640 10 monclient: _check_auth_tickets 2024-09-16T04:01:50.763 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.752+0000 7f472959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:01:50.763 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.752+0000 7f47309a2640 5 monclient: authenticate success, global_id 825450 2024-09-16T04:01:50.763 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.752+0000 7f47309a2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:01:50.763 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.752+0000 7f472959a640 1 -- 172.21.15.42:0/858301576 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f47240a5800 con 0x7f472c13d8b0 2024-09-16T04:01:50.763 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.752+0000 7f472959a640 10 monclient: handle_config config(0 keys) 2024-09-16T04:01:50.764 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.752+0000 7f472959a640 1 -- 172.21.15.42:0/858301576 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47240a5bd0 con 0x7f472c13d8b0 2024-09-16T04:01:50.764 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.752+0000 7f472959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:01:50.764 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.752+0000 7f472959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:01:50.764 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.752+0000 7f472959a640 10 monclient: dump: 2024-09-16T04:01:50.764 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:01:50.764 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:01:50.764 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:50.764 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:50.764 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:01:50.764 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:01:50.765 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:01:50.765 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:01:50.765 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:01:50.765 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:01:50.765 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.752+0000 7f47309a2640 10 monclient: get_monmap_and_config success 2024-09-16T04:01:50.765 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.752+0000 7f47309a2640 4 set_mon_vals no callback set 2024-09-16T04:01:50.765 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.752+0000 7f47309a2640 10 monclient: shutdown 2024-09-16T04:01:50.765 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.752+0000 7f47309a2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:01:50.765 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.752+0000 7f47309a2640 1 -- 172.21.15.42:0/858301576 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f472c13d8b0 msgr2=0x7f472c13fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:01:50.765 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.752+0000 7f47309a2640 1 --2- 172.21.15.42:0/858301576 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f472c13d8b0 0x7f472c13fca0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f4724009d90 tx=0x7f472401d290 comp rx=0 tx=0).stop 2024-09-16T04:01:50.765 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.752+0000 7f47309a2640 1 -- 172.21.15.42:0/858301576 shutdown_connections 2024-09-16T04:01:50.765 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.752+0000 7f47309a2640 1 --2- 172.21.15.42:0/858301576 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f472c13d8b0 0x7f472c13fca0 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:50.765 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.752+0000 7f47309a2640 1 -- 172.21.15.42:0/858301576 >> 172.21.15.42:0/858301576 conn(0x7f472c0a9fb0 msgr2=0x7f472c0aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:01:50.765 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.752+0000 7f47309a2640 1 -- 172.21.15.42:0/858301576 shutdown_connections 2024-09-16T04:01:50.766 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.752+0000 7f47309a2640 1 -- 172.21.15.42:0/858301576 wait complete. 2024-09-16T04:01:50.766 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.756+0000 7f47309a2640 10 monclient: build_initial_monmap 2024-09-16T04:01:50.766 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.756+0000 7f47309a2640 10 monclient: monmap: 2024-09-16T04:01:50.766 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:01:50.766 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:01:50.766 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:01:50.766 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:01:50.766 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:01:50.766 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:01:50.766 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:01:50.766 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:01:50.766 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:01:50.766 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:01:50.767 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.756+0000 7f47309a2640 1 librados: starting msgr at 2024-09-16T04:01:50.767 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.756+0000 7f47309a2640 1 librados: starting objecter 2024-09-16T04:01:50.767 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.756+0000 7f47309a2640 1 Processor -- start 2024-09-16T04:01:50.767 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.756+0000 7f47309a2640 1 -- start start 2024-09-16T04:01:50.767 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.756+0000 7f47309a2640 1 librados: setting wanted keys 2024-09-16T04:01:50.767 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.756+0000 7f47309a2640 1 librados: calling monclient init 2024-09-16T04:01:50.767 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.756+0000 7f47309a2640 10 monclient: init 2024-09-16T04:01:50.771 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.756+0000 7f47309a2640 10 monclient: _reopen_session rank -1 2024-09-16T04:01:50.771 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.756+0000 7f47309a2640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:01:50.771 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.756+0000 7f47309a2640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f472c06db90 0x7f472c139d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:01:50.771 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.756+0000 7f47309a2640 10 monclient(hunting): picked mon.noname-b con 0x7f472c06db90 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:01:50.771 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.756+0000 7f47309a2640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f472c1340c0 0x7f472c13a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:01:50.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.756+0000 7f47309a2640 10 monclient(hunting): picked mon.noname-c con 0x7f472c1340c0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:01:50.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.756+0000 7f47309a2640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f472c13d8b0 0x7f472c13a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:01:50.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.756+0000 7f47309a2640 10 monclient(hunting): picked mon.noname-a con 0x7f472c13d8b0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:01:50.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.756+0000 7f47309a2640 10 monclient(hunting): start opening mon connection 2024-09-16T04:01:50.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.756+0000 7f47309a2640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f472c142b80 con 0x7f472c13d8b0 2024-09-16T04:01:50.773 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.756+0000 7f47309a2640 10 monclient(hunting): start opening mon connection 2024-09-16T04:01:50.773 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.756+0000 7f47309a2640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f472c142a00 con 0x7f472c06db90 2024-09-16T04:01:50.773 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.756+0000 7f47309a2640 10 monclient(hunting): start opening mon connection 2024-09-16T04:01:50.773 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.756+0000 7f47309a2640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f472c142d00 con 0x7f472c1340c0 2024-09-16T04:01:50.774 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.756+0000 7f47309a2640 10 monclient(hunting): _renew_subs 2024-09-16T04:01:50.774 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.756+0000 7f47309a2640 10 monclient(hunting): authenticate will time out at 2157.725219s 2024-09-16T04:01:50.774 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.756+0000 7f4729d9b640 1 -- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f472c1340c0 msgr2=0x7f472c13a2b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T04:01:50.774 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.756+0000 7f4729d9b640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f472c1340c0 0x7f472c13a2b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:01:50.774 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.756+0000 7f472ad9d640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f472c13d8b0 0x7f472c13a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:01:50.774 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.756+0000 7f472ad9d640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f472c13d8b0 0x7f472c13a7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:35438/0 (socket says 172.21.15.42:35438) 2024-09-16T04:01:50.774 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.756+0000 7f472ad9d640 1 -- 172.21.15.42:0/2261922374 learned_addr learned my addr 172.21.15.42:0/2261922374 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:01:50.774 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.756+0000 7f472ad9d640 10 monclient(hunting): get_auth_request con 0x7f472c13d8b0 auth_method 0 2024-09-16T04:01:50.774 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.756+0000 7f472ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:01:50.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.756+0000 7f472ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:01:50.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.756+0000 7f472ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:01:50.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.756+0000 7f472ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:01:50.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.756+0000 7f472ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:01:50.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.756+0000 7f472ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:01:50.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.756+0000 7f472a59c640 1 --2- 172.21.15.42:0/2261922374 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f472c06db90 0x7f472c139d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:01:50.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.756+0000 7f472a59c640 10 monclient(hunting): get_auth_request con 0x7f472c06db90 auth_method 0 2024-09-16T04:01:50.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.756+0000 7f472a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:01:50.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.756+0000 7f472a59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:01:50.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.756+0000 7f472a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:01:50.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.756+0000 7f472ad9d640 10 monclient(hunting): handle_auth_done global_id 825456 payload 824 2024-09-16T04:01:50.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.756+0000 7f472ad9d640 1 -- 172.21.15.42:0/2261922374 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f472c1340c0 msgr2=0x7f472c13a2b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:01:50.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.756+0000 7f472ad9d640 1 --2- 172.21.15.42:0/2261922374 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f472c1340c0 0x7f472c13a2b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:50.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.756+0000 7f472ad9d640 1 -- 172.21.15.42:0/2261922374 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f472c06db90 msgr2=0x7f472c139d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:01:50.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.756+0000 7f472ad9d640 1 --2- 172.21.15.42:0/2261922374 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f472c06db90 0x7f472c139d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:50.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.756+0000 7f472ad9d640 10 monclient: _finish_hunting 0 2024-09-16T04:01:50.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.756+0000 7f472ad9d640 1 monclient: found mon.noname-a 2024-09-16T04:01:50.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.756+0000 7f472ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:01:50.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.756+0000 7f472ad9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:01:50.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.756+0000 7f472ad9d640 1 -- 172.21.15.42:0/2261922374 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f472c09d480 con 0x7f472c13d8b0 2024-09-16T04:01:50.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.756+0000 7f472ad9d640 1 --2- 172.21.15.42:0/2261922374 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f472c13d8b0 0x7f472c13a7f0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f4724009d90 tx=0x7f47240195d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:01:50.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.756+0000 7f4722ffd640 10 client.?.objecter ms_handle_connect 0x7f472c13d8b0 2024-09-16T04:01:50.777 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.756+0000 7f4722ffd640 10 client.?.objecter resend_mon_ops 2024-09-16T04:01:50.777 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.756+0000 7f4722ffd640 1 -- 172.21.15.42:0/2261922374 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47240bc070 con 0x7f472c13d8b0 2024-09-16T04:01:50.777 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.756+0000 7f4722ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:01:50.777 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.756+0000 7f4722ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:01:50.777 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.756+0000 7f4722ffd640 10 monclient: dump: 2024-09-16T04:01:50.777 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:01:50.777 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:01:50.777 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:50.777 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:50.778 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:01:50.778 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:01:50.778 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:01:50.778 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:01:50.778 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:01:50.778 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:01:50.778 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.756+0000 7f4722ffd640 10 monclient: _finish_auth 0 2024-09-16T04:01:50.778 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.756+0000 7f4722ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:01:50.778 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.756+0000 7f4722ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:01:50.778 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.756+0000 7f4722ffd640 1 -- 172.21.15.42:0/2261922374 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4724003ba0 con 0x7f472c13d8b0 2024-09-16T04:01:50.778 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.756+0000 7f4722ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:01:50.778 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.756+0000 7f4722ffd640 1 -- 172.21.15.42:0/2261922374 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47240b2040 con 0x7f472c13d8b0 2024-09-16T04:01:50.778 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.756+0000 7f47309a2640 5 monclient: authenticate success, global_id 825456 2024-09-16T04:01:50.779 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.756+0000 7f47309a2640 10 monclient: _renew_subs 2024-09-16T04:01:50.779 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.756+0000 7f47309a2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:01:50.779 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.756+0000 7f47309a2640 1 -- 172.21.15.42:0/2261922374 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f472c13d1b0 con 0x7f472c13d8b0 2024-09-16T04:01:50.779 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.756+0000 7f4728d99640 4 set_mon_vals no callback set 2024-09-16T04:01:50.779 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.756+0000 7f4722ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:01:50.779 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.756+0000 7f4722ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:01:50.779 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.756+0000 7f4722ffd640 10 monclient: dump: 2024-09-16T04:01:50.779 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:01:50.779 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:01:50.779 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:50.780 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:50.780 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:01:50.780 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:01:50.780 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:01:50.780 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:01:50.780 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:01:50.780 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:01:50.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.756+0000 7f47309a2640 10 client.825456.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:01:50.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.756+0000 7f47309a2640 10 monclient: _renew_subs 2024-09-16T04:01:50.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.756+0000 7f47309a2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:01:50.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.756+0000 7f47309a2640 1 -- 172.21.15.42:0/2261922374 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f472c09d1a0 con 0x7f472c13d8b0 2024-09-16T04:01:50.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.756+0000 7f47309a2640 1 librados: init done 2024-09-16T04:01:50.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.756+0000 7f47309a2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:01:50.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.756+0000 7f47309a2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:01:50.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.756+0000 7f47309a2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:01:50.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.756+0000 7f47309a2640 1 -- 172.21.15.42:0/2261922374 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f472c09cfa0 con 0x7f472c13d8b0 2024-09-16T04:01:50.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.756+0000 7f4722ffd640 1 -- 172.21.15.42:0/2261922374 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f4724004030 con 0x7f472c13d8b0 2024-09-16T04:01:50.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.756+0000 7f4722ffd640 1 --2- 172.21.15.42:0/2261922374 >> v2:172.21.15.154:6812/254845342 conn(0x7f470c0377c0 0x7f470c039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:01:50.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.756+0000 7f4722ffd640 1 -- 172.21.15.42:0/2261922374 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(602..602 src has 451..602) ==== 6252+0+0 (secure 0 0 0) 0x7f47240e3da0 con 0x7f472c13d8b0 2024-09-16T04:01:50.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.756+0000 7f4722ffd640 10 client.825456.objecter ms_dispatch 0x7f472c000ba0 osd_map(602..602 src has 451..602) 2024-09-16T04:01:50.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.756+0000 7f4722ffd640 3 client.825456.objecter handle_osd_map got epochs [602,602] > 0 2024-09-16T04:01:50.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.756+0000 7f4722ffd640 3 client.825456.objecter handle_osd_map decoding full epoch 602 2024-09-16T04:01:50.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.756+0000 7f4722ffd640 20 client.825456.objecter dump_active .. 0 homeless 2024-09-16T04:01:50.782 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.756+0000 7f472a59c640 1 --2- 172.21.15.42:0/2261922374 >> v2:172.21.15.154:6812/254845342 conn(0x7f470c0377c0 0x7f470c039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:01:50.782 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.756+0000 7f472a59c640 10 monclient: get_auth_request con 0x7f470c0377c0 auth_method 0 2024-09-16T04:01:50.782 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.760+0000 7f4722ffd640 10 client.825456.objecter ms_handle_connect 0x7f470c0377c0 2024-09-16T04:01:50.782 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.760+0000 7f472a59c640 1 --2- 172.21.15.42:0/2261922374 >> v2:172.21.15.154:6812/254845342 conn(0x7f470c0377c0 0x7f470c039c80 secure :-1 s=READY pgs=3773 cs=0 l=1 rev1=1 crypto rx=0x7f4714000f70 tx=0x7f4714009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:01:50.782 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.764+0000 7f4722ffd640 1 -- 172.21.15.42:0/2261922374 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f47240c6020 con 0x7f472c13d8b0 2024-09-16T04:01:50.782 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.764+0000 7f4722ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:01:50.782 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.764+0000 7f4722ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:01:50.992 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.988+0000 7f47309a2640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:01:50.992 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.988+0000 7f47309a2640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:01:50.992 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.988+0000 7f47309a2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:01:50.992 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.988+0000 7f47309a2640 1 -- 172.21.15.42:0/2261922374 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f472c0a37f0 con 0x7f472c13d8b0 2024-09-16T04:01:50.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.988+0000 7f4722ffd640 1 -- 172.21.15.42:0/2261922374 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7f4724016540 con 0x7f472c13d8b0 2024-09-16T04:01:50.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.988+0000 7f4722ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:01:50.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.988+0000 7f4722ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:01:50.993 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":178,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:01:50.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.992+0000 7f47309a2640 10 librados: watch_flush enter 2024-09-16T04:01:50.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.992+0000 7f47309a2640 10 librados: watch_flush exit 2024-09-16T04:01:50.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.992+0000 7f47309a2640 20 client.825456.objecter shutdown clearing up homeless session... 2024-09-16T04:01:50.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.992+0000 7f47309a2640 10 client.825456.objecter successfully canceled tick 2024-09-16T04:01:50.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.992+0000 7f47309a2640 1 -- 172.21.15.42:0/2261922374 >> v2:172.21.15.154:6812/254845342 conn(0x7f470c0377c0 msgr2=0x7f470c039c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:01:50.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.992+0000 7f47309a2640 1 --2- 172.21.15.42:0/2261922374 >> v2:172.21.15.154:6812/254845342 conn(0x7f470c0377c0 0x7f470c039c80 secure :-1 s=READY pgs=3773 cs=0 l=1 rev1=1 crypto rx=0x7f4714000f70 tx=0x7f4714009040 comp rx=0 tx=0).stop 2024-09-16T04:01:50.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.992+0000 7f47309a2640 10 monclient: shutdown 2024-09-16T04:01:50.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.992+0000 7f47309a2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:01:50.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.992+0000 7f47309a2640 1 -- 172.21.15.42:0/2261922374 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f472c13d8b0 msgr2=0x7f472c13a7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:01:50.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.992+0000 7f47309a2640 1 --2- 172.21.15.42:0/2261922374 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f472c13d8b0 0x7f472c13a7f0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f4724009d90 tx=0x7f47240195d0 comp rx=0 tx=0).stop 2024-09-16T04:01:50.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.992+0000 7f47309a2640 1 -- 172.21.15.42:0/2261922374 shutdown_connections 2024-09-16T04:01:50.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.992+0000 7f47309a2640 1 --2- 172.21.15.42:0/2261922374 >> v2:172.21.15.154:6812/254845342 conn(0x7f470c0377c0 0x7f470c039c80 unknown :-1 s=CLOSED pgs=3773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:50.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.992+0000 7f47309a2640 1 --2- 172.21.15.42:0/2261922374 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f472c13d8b0 0x7f472c13a7f0 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:50.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.992+0000 7f47309a2640 1 --2- 172.21.15.42:0/2261922374 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f472c1340c0 0x7f472c13a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:50.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.992+0000 7f47309a2640 1 --2- 172.21.15.42:0/2261922374 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f472c06db90 0x7f472c139d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:50.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.992+0000 7f47309a2640 1 -- 172.21.15.42:0/2261922374 >> 172.21.15.42:0/2261922374 conn(0x7f472c0a9fb0 msgr2=0x7f472c0a82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:01:50.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.992+0000 7f47309a2640 1 -- 172.21.15.42:0/2261922374 shutdown_connections 2024-09-16T04:01:50.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.992+0000 7f47309a2640 1 -- 172.21.15.42:0/2261922374 wait complete. 2024-09-16T04:01:50.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:50.992+0000 7f47309a2640 1 librados: shutdown 2024-09-16T04:01:54.013 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:01:54.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.144+0000 7f744de70640 10 monclient: get_monmap_and_config 2024-09-16T04:01:54.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.144+0000 7f744de70640 10 monclient: build_initial_monmap 2024-09-16T04:01:54.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.144+0000 7f744de70640 10 monclient: monmap: 2024-09-16T04:01:54.147 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:01:54.147 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:01:54.147 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:01:54.145691+0000 2024-09-16T04:01:54.147 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:01:54.145691+0000 2024-09-16T04:01:54.147 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:01:54.147 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:01:54.148 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:01:54.148 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:01:54.148 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:01:54.148 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:01:54.148 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:01:54.148 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:01:54.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.144+0000 7f744de70640 1 Processor -- start 2024-09-16T04:01:54.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.144+0000 7f744de70640 1 -- start start 2024-09-16T04:01:54.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.144+0000 7f744de70640 10 monclient: init 2024-09-16T04:01:54.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.144+0000 7f744de70640 10 monclient: _reopen_session rank -1 2024-09-16T04:01:54.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.144+0000 7f744de70640 10 monclient: _add_conns ranks=[2,4,1,0,3] 2024-09-16T04:01:54.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.144+0000 7f744de70640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f74480a39b0 0x7f7448133b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:01:54.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.144+0000 7f744de70640 10 monclient(hunting): picked mon.noname-b con 0x7f74480a39b0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:01:54.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.144+0000 7f744de70640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f74480a3d90 0x7f7448137000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:01:54.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.144+0000 7f744de70640 10 monclient(hunting): picked mon.noname-c con 0x7f74480a3d90 addr v2:172.21.15.154:3300/0 2024-09-16T04:01:54.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.144+0000 7f744de70640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7448137540 addr v1:172.21.15.154:6789/0 2024-09-16T04:01:54.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.144+0000 7f744de70640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f744809c140 con 0x7f7448137540 2024-09-16T04:01:54.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.144+0000 7f744de70640 10 monclient(hunting): start opening mon connection 2024-09-16T04:01:54.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.144+0000 7f744de70640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f744809c420 con 0x7f74480a3d90 2024-09-16T04:01:54.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.144+0000 7f744de70640 10 monclient(hunting): start opening mon connection 2024-09-16T04:01:54.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.144+0000 7f744de70640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f744806dd60 con 0x7f74480a39b0 2024-09-16T04:01:54.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.144+0000 7f744de70640 10 monclient(hunting): _renew_subs 2024-09-16T04:01:54.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.144+0000 7f744de70640 10 monclient(hunting): authenticate will time out at 2161.115240s 2024-09-16T04:01:54.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.144+0000 7f74477fe640 1 -- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f74480a39b0 msgr2=0x7f7448133b80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T04:01:54.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.144+0000 7f74477fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f74480a39b0 0x7f7448133b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:01:54.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.144+0000 7f7446ffd640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f74480a3d90 0x7f7448137000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:01:54.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.144+0000 7f7446ffd640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f74480a3d90 0x7f7448137000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.42:35920/0 (socket says 172.21.15.42:35920) 2024-09-16T04:01:54.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.144+0000 7f7446ffd640 1 -- 172.21.15.42:0/3832894253 learned_addr learned my addr 172.21.15.42:0/3832894253 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:01:54.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.144+0000 7f7446ffd640 10 monclient(hunting): get_auth_request con 0x7f74480a3d90 auth_method 0 2024-09-16T04:01:54.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.144+0000 7f7446ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:01:54.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.144+0000 7f7446ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:01:54.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.144+0000 7f7446ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:01:54.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.144+0000 7f7446ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:01:54.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.144+0000 7f7446ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:01:54.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.144+0000 7f7446ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:01:54.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.148+0000 7f7446ffd640 10 monclient(hunting): handle_auth_done global_id 904555 payload 274 2024-09-16T04:01:54.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.148+0000 7f7446ffd640 1 -- 172.21.15.42:0/3832894253 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f74480a39b0 msgr2=0x7f7448133b80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:01:54.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.148+0000 7f7446ffd640 1 --2- 172.21.15.42:0/3832894253 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f74480a39b0 0x7f7448133b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:54.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.148+0000 7f7446ffd640 1 -- 172.21.15.42:0/3832894253 >> v1:172.21.15.154:6789/0 conn(0x7f7448137540 legacy=0x7f744813e390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:01:54.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.148+0000 7f7446ffd640 10 monclient: _finish_hunting 0 2024-09-16T04:01:54.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.148+0000 7f7446ffd640 1 monclient: found mon.noname-c 2024-09-16T04:01:54.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.148+0000 7f7446ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:01:54.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.148+0000 7f7446ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.154:3300/0 2024-09-16T04:01:54.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.148+0000 7f7446ffd640 1 -- 172.21.15.42:0/3832894253 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74480a5e20 con 0x7f74480a3d90 2024-09-16T04:01:54.157 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.148+0000 7f7446ffd640 1 --2- 172.21.15.42:0/3832894253 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f74480a3d90 0x7f7448137000 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f7430009e50 tx=0x7f743001c910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f19ef753b087029d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:01:54.157 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.148+0000 7f74467fc640 1 -- 172.21.15.42:0/3832894253 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74300a55d0 con 0x7f74480a3d90 2024-09-16T04:01:54.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.148+0000 7f74467fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:01:54.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.148+0000 7f74467fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:01:54.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.148+0000 7f74467fc640 10 monclient: dump: 2024-09-16T04:01:54.158 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:01:54.158 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:01:54.158 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:54.158 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:54.158 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:01:54.158 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:01:54.158 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:01:54.158 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:01:54.158 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:01:54.158 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:01:54.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.148+0000 7f74467fc640 10 monclient: _finish_auth 0 2024-09-16T04:01:54.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.148+0000 7f74467fc640 10 monclient: _check_auth_tickets 2024-09-16T04:01:54.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.148+0000 7f74467fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:01:54.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.148+0000 7f74467fc640 1 -- 172.21.15.42:0/3832894253 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f74300a5770 con 0x7f74480a3d90 2024-09-16T04:01:54.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.148+0000 7f74467fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:01:54.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.148+0000 7f74467fc640 1 -- 172.21.15.42:0/3832894253 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74300a5b40 con 0x7f74480a3d90 2024-09-16T04:01:54.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.148+0000 7f744de70640 5 monclient: authenticate success, global_id 904555 2024-09-16T04:01:54.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.148+0000 7f744de70640 10 monclient: get_monmap_and_config success 2024-09-16T04:01:54.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.148+0000 7f744de70640 4 set_mon_vals no callback set 2024-09-16T04:01:54.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.148+0000 7f74467fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:01:54.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.148+0000 7f74467fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:01:54.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.148+0000 7f74467fc640 10 monclient: dump: 2024-09-16T04:01:54.160 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:01:54.160 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:01:54.160 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:54.160 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:54.160 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:01:54.160 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:01:54.160 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:01:54.160 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:01:54.160 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:01:54.160 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:01:54.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.148+0000 7f744de70640 10 monclient: shutdown 2024-09-16T04:01:54.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.148+0000 7f744de70640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:01:54.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.148+0000 7f744de70640 1 -- 172.21.15.42:0/3832894253 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f74480a3d90 msgr2=0x7f7448137000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:01:54.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.148+0000 7f744de70640 1 --2- 172.21.15.42:0/3832894253 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f74480a3d90 0x7f7448137000 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f7430009e50 tx=0x7f743001c910 comp rx=0 tx=0).stop 2024-09-16T04:01:54.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.148+0000 7f744de70640 1 -- 172.21.15.42:0/3832894253 shutdown_connections 2024-09-16T04:01:54.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.148+0000 7f744de70640 1 --2- 172.21.15.42:0/3832894253 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f74480a3d90 0x7f7448137000 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:54.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.148+0000 7f744de70640 1 --2- 172.21.15.42:0/3832894253 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f74480a39b0 0x7f7448133b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:54.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.148+0000 7f744de70640 1 -- 172.21.15.42:0/3832894253 >> 172.21.15.42:0/3832894253 conn(0x7f74480a9fb0 msgr2=0x7f74480aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:01:54.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.148+0000 7f744de70640 1 -- 172.21.15.42:0/3832894253 shutdown_connections 2024-09-16T04:01:54.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.148+0000 7f744de70640 1 -- 172.21.15.42:0/3832894253 wait complete. 2024-09-16T04:01:54.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.152+0000 7f744de70640 10 monclient: build_initial_monmap 2024-09-16T04:01:54.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.152+0000 7f744de70640 10 monclient: monmap: 2024-09-16T04:01:54.161 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:01:54.161 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:01:54.161 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:01:54.161 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:01:54.161 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:01:54.162 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:01:54.162 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:01:54.162 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:01:54.162 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:01:54.162 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:01:54.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.152+0000 7f744de70640 1 librados: starting msgr at 2024-09-16T04:01:54.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.152+0000 7f744de70640 1 librados: starting objecter 2024-09-16T04:01:54.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.152+0000 7f744de70640 1 Processor -- start 2024-09-16T04:01:54.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.152+0000 7f744de70640 1 -- start start 2024-09-16T04:01:54.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.152+0000 7f744de70640 1 librados: setting wanted keys 2024-09-16T04:01:54.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.152+0000 7f744de70640 1 librados: calling monclient init 2024-09-16T04:01:54.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.152+0000 7f744de70640 10 monclient: init 2024-09-16T04:01:54.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.152+0000 7f744de70640 10 monclient: _reopen_session rank -1 2024-09-16T04:01:54.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.152+0000 7f744de70640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:01:54.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.152+0000 7f744de70640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f74480a39b0 0x7f744813d130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:01:54.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.152+0000 7f744de70640 10 monclient(hunting): picked mon.noname-b con 0x7f74480a39b0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:01:54.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.152+0000 7f744de70640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f74480a3d90 0x7f744813d670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:01:54.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.152+0000 7f744de70640 10 monclient(hunting): picked mon.noname-a con 0x7f74480a3d90 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:01:54.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.152+0000 7f744de70640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7448137540 0x7f744813dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:01:54.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.152+0000 7f744de70640 10 monclient(hunting): picked mon.noname-c con 0x7f7448137540 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:01:54.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.152+0000 7f744de70640 10 monclient(hunting): start opening mon connection 2024-09-16T04:01:54.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.152+0000 7f744de70640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f74481424e0 con 0x7f74480a3d90 2024-09-16T04:01:54.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.152+0000 7f744de70640 10 monclient(hunting): start opening mon connection 2024-09-16T04:01:54.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.152+0000 7f744de70640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f7448142660 con 0x7f74480a39b0 2024-09-16T04:01:54.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.152+0000 7f744de70640 10 monclient(hunting): start opening mon connection 2024-09-16T04:01:54.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.152+0000 7f744de70640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f744809d070 con 0x7f7448137540 2024-09-16T04:01:54.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.152+0000 7f744de70640 10 monclient(hunting): _renew_subs 2024-09-16T04:01:54.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.152+0000 7f744de70640 10 monclient(hunting): authenticate will time out at 2161.120931s 2024-09-16T04:01:54.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.152+0000 7f7447fff640 1 -- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7448137540 msgr2=0x7f744813dbb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T04:01:54.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.152+0000 7f7447fff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7448137540 0x7f744813dbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:01:54.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.152+0000 7f7446ffd640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f74480a3d90 0x7f744813d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:01:54.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.152+0000 7f7446ffd640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f74480a3d90 0x7f744813d670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:37328/0 (socket says 172.21.15.42:37328) 2024-09-16T04:01:54.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.152+0000 7f7446ffd640 1 -- 172.21.15.42:0/2580528234 learned_addr learned my addr 172.21.15.42:0/2580528234 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:01:54.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.152+0000 7f7446ffd640 10 monclient(hunting): get_auth_request con 0x7f74480a3d90 auth_method 0 2024-09-16T04:01:54.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.152+0000 7f7446ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:01:54.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.152+0000 7f7446ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:01:54.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.152+0000 7f7446ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:01:54.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.152+0000 7f7446ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:01:54.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.152+0000 7f7446ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:01:54.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.152+0000 7f7446ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:01:54.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.152+0000 7f74477fe640 1 --2- 172.21.15.42:0/2580528234 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f74480a39b0 0x7f744813d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:01:54.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.152+0000 7f74477fe640 10 monclient(hunting): get_auth_request con 0x7f74480a39b0 auth_method 0 2024-09-16T04:01:54.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.152+0000 7f74477fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:01:54.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.152+0000 7f74477fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:01:54.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.152+0000 7f74477fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:01:54.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.152+0000 7f7446ffd640 10 monclient(hunting): handle_auth_done global_id 825459 payload 824 2024-09-16T04:01:54.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.152+0000 7f7446ffd640 1 -- 172.21.15.42:0/2580528234 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7448137540 msgr2=0x7f744813dbb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:01:54.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.152+0000 7f7446ffd640 1 --2- 172.21.15.42:0/2580528234 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7448137540 0x7f744813dbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:54.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.152+0000 7f7446ffd640 1 -- 172.21.15.42:0/2580528234 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f74480a39b0 msgr2=0x7f744813d130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:01:54.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.152+0000 7f7446ffd640 1 --2- 172.21.15.42:0/2580528234 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f74480a39b0 0x7f744813d130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:54.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.152+0000 7f7446ffd640 10 monclient: _finish_hunting 0 2024-09-16T04:01:54.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.152+0000 7f7446ffd640 1 monclient: found mon.noname-a 2024-09-16T04:01:54.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.152+0000 7f7446ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:01:54.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.152+0000 7f7446ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:01:54.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.152+0000 7f7446ffd640 1 -- 172.21.15.42:0/2580528234 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7448136e00 con 0x7f74480a3d90 2024-09-16T04:01:54.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.152+0000 7f7446ffd640 1 --2- 172.21.15.42:0/2580528234 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f74480a3d90 0x7f744813d670 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f74300af040 tx=0x7f74300196b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:01:54.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.152+0000 7f743ffff640 10 client.?.objecter ms_handle_connect 0x7f74480a3d90 2024-09-16T04:01:54.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.152+0000 7f743ffff640 10 client.?.objecter resend_mon_ops 2024-09-16T04:01:54.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.152+0000 7f743ffff640 1 -- 172.21.15.42:0/2580528234 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74300bc020 con 0x7f74480a3d90 2024-09-16T04:01:54.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.152+0000 7f743ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:01:54.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.152+0000 7f743ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:01:54.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.152+0000 7f743ffff640 10 monclient: dump: 2024-09-16T04:01:54.167 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:01:54.167 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:01:54.167 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:54.167 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:54.167 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:01:54.167 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:01:54.167 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:01:54.167 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:01:54.167 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:01:54.167 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:01:54.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.152+0000 7f743ffff640 10 monclient: _finish_auth 0 2024-09-16T04:01:54.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.152+0000 7f743ffff640 10 monclient: _check_auth_tickets 2024-09-16T04:01:54.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.152+0000 7f743ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:01:54.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.152+0000 7f743ffff640 1 -- 172.21.15.42:0/2580528234 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f74300b8690 con 0x7f74480a3d90 2024-09-16T04:01:54.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.152+0000 7f744de70640 5 monclient: authenticate success, global_id 825459 2024-09-16T04:01:54.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.152+0000 7f744de70640 10 monclient: _renew_subs 2024-09-16T04:01:54.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.152+0000 7f744de70640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:01:54.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.152+0000 7f744de70640 1 -- 172.21.15.42:0/2580528234 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7448133340 con 0x7f74480a3d90 2024-09-16T04:01:54.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.152+0000 7f743ffff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:01:54.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.152+0000 7f743ffff640 1 -- 172.21.15.42:0/2580528234 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74300b8a80 con 0x7f74480a3d90 2024-09-16T04:01:54.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.152+0000 7f743ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:01:54.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.152+0000 7f743ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:01:54.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.152+0000 7f743ffff640 10 monclient: dump: 2024-09-16T04:01:54.168 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:01:54.168 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:01:54.169 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:54.169 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:54.169 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:01:54.169 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:01:54.169 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:01:54.169 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:01:54.169 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:01:54.169 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:01:54.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.152+0000 7f744de70640 10 client.825459.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:01:54.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.152+0000 7f744de70640 10 monclient: _renew_subs 2024-09-16T04:01:54.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.152+0000 7f7445ffb640 4 set_mon_vals no callback set 2024-09-16T04:01:54.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.152+0000 7f744de70640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:01:54.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.152+0000 7f744de70640 1 -- 172.21.15.42:0/2580528234 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f744809e240 con 0x7f74480a3d90 2024-09-16T04:01:54.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.152+0000 7f744de70640 1 librados: init done 2024-09-16T04:01:54.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.152+0000 7f744de70640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:01:54.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.152+0000 7f744de70640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:01:54.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.152+0000 7f744de70640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:01:54.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.152+0000 7f744de70640 1 -- 172.21.15.42:0/2580528234 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f744813ed80 con 0x7f74480a3d90 2024-09-16T04:01:54.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.152+0000 7f743ffff640 1 -- 172.21.15.42:0/2580528234 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f74300bf020 con 0x7f74480a3d90 2024-09-16T04:01:54.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.152+0000 7f743ffff640 1 --2- 172.21.15.42:0/2580528234 >> v2:172.21.15.154:6812/254845342 conn(0x7f742c0377c0 0x7f742c039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:01:54.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.152+0000 7f743ffff640 1 -- 172.21.15.42:0/2580528234 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(602..602 src has 451..602) ==== 6252+0+0 (secure 0 0 0) 0x7f74300e4200 con 0x7f74480a3d90 2024-09-16T04:01:54.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.152+0000 7f743ffff640 10 client.825459.objecter ms_dispatch 0x7f7448000ba0 osd_map(602..602 src has 451..602) 2024-09-16T04:01:54.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.152+0000 7f743ffff640 3 client.825459.objecter handle_osd_map got epochs [602,602] > 0 2024-09-16T04:01:54.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.152+0000 7f743ffff640 3 client.825459.objecter handle_osd_map decoding full epoch 602 2024-09-16T04:01:54.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.152+0000 7f743ffff640 20 client.825459.objecter dump_active .. 0 homeless 2024-09-16T04:01:54.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.152+0000 7f74477fe640 1 --2- 172.21.15.42:0/2580528234 >> v2:172.21.15.154:6812/254845342 conn(0x7f742c0377c0 0x7f742c039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:01:54.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.152+0000 7f74477fe640 10 monclient: get_auth_request con 0x7f742c0377c0 auth_method 0 2024-09-16T04:01:54.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.156+0000 7f743ffff640 10 client.825459.objecter ms_handle_connect 0x7f742c0377c0 2024-09-16T04:01:54.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.156+0000 7f74477fe640 1 --2- 172.21.15.42:0/2580528234 >> v2:172.21.15.154:6812/254845342 conn(0x7f742c0377c0 0x7f742c039c80 secure :-1 s=READY pgs=3774 cs=0 l=1 rev1=1 crypto rx=0x7f7438006fd0 tx=0x7f7438008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:01:54.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.156+0000 7f743ffff640 1 -- 172.21.15.42:0/2580528234 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f744813ed80 con 0x7f74480a3d90 2024-09-16T04:01:54.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.156+0000 7f743ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:01:54.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.156+0000 7f743ffff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:01:54.380 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.376+0000 7f744de70640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:01:54.380 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.376+0000 7f744de70640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:01:54.380 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.376+0000 7f744de70640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:01:54.380 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.376+0000 7f744de70640 1 -- 172.21.15.42:0/2580528234 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f7448133b80 con 0x7f74480a3d90 2024-09-16T04:01:54.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.376+0000 7f743ffff640 1 -- 172.21.15.42:0/2580528234 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1562 (secure 0 0 0) 0x7f74300c6030 con 0x7f74480a3d90 2024-09-16T04:01:54.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.376+0000 7f743ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:01:54.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.376+0000 7f743ffff640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:01:54.381 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":178,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:01:54.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.380+0000 7f744de70640 10 librados: watch_flush enter 2024-09-16T04:01:54.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.380+0000 7f744de70640 10 librados: watch_flush exit 2024-09-16T04:01:54.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.380+0000 7f744de70640 20 client.825459.objecter shutdown clearing up homeless session... 2024-09-16T04:01:54.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.380+0000 7f744de70640 10 client.825459.objecter successfully canceled tick 2024-09-16T04:01:54.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.380+0000 7f744de70640 1 -- 172.21.15.42:0/2580528234 >> v2:172.21.15.154:6812/254845342 conn(0x7f742c0377c0 msgr2=0x7f742c039c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:01:54.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.380+0000 7f744de70640 1 --2- 172.21.15.42:0/2580528234 >> v2:172.21.15.154:6812/254845342 conn(0x7f742c0377c0 0x7f742c039c80 secure :-1 s=READY pgs=3774 cs=0 l=1 rev1=1 crypto rx=0x7f7438006fd0 tx=0x7f7438008040 comp rx=0 tx=0).stop 2024-09-16T04:01:54.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.380+0000 7f744de70640 10 monclient: shutdown 2024-09-16T04:01:54.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.380+0000 7f744de70640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:01:54.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.380+0000 7f744de70640 1 -- 172.21.15.42:0/2580528234 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f74480a3d90 msgr2=0x7f744813d670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:01:54.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.380+0000 7f744de70640 1 --2- 172.21.15.42:0/2580528234 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f74480a3d90 0x7f744813d670 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f74300af040 tx=0x7f74300196b0 comp rx=0 tx=0).stop 2024-09-16T04:01:54.385 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.380+0000 7f744de70640 1 -- 172.21.15.42:0/2580528234 shutdown_connections 2024-09-16T04:01:54.385 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.380+0000 7f744de70640 1 --2- 172.21.15.42:0/2580528234 >> v2:172.21.15.154:6812/254845342 conn(0x7f742c0377c0 0x7f742c039c80 unknown :-1 s=CLOSED pgs=3774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:54.385 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.380+0000 7f744de70640 1 --2- 172.21.15.42:0/2580528234 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7448137540 0x7f744813dbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:54.385 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.380+0000 7f744de70640 1 --2- 172.21.15.42:0/2580528234 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f74480a3d90 0x7f744813d670 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:54.385 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.380+0000 7f744de70640 1 --2- 172.21.15.42:0/2580528234 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f74480a39b0 0x7f744813d130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:54.385 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.380+0000 7f744de70640 1 -- 172.21.15.42:0/2580528234 >> 172.21.15.42:0/2580528234 conn(0x7f74480a9fb0 msgr2=0x7f74480a8a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:01:54.385 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.380+0000 7f744de70640 1 -- 172.21.15.42:0/2580528234 shutdown_connections 2024-09-16T04:01:54.386 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.380+0000 7f744de70640 1 -- 172.21.15.42:0/2580528234 wait complete. 2024-09-16T04:01:54.386 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:54.380+0000 7f744de70640 1 librados: shutdown 2024-09-16T04:01:57.398 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:01:57.527 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.524+0000 7f74382c7640 10 monclient: get_monmap_and_config 2024-09-16T04:01:57.527 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.524+0000 7f74382c7640 10 monclient: build_initial_monmap 2024-09-16T04:01:57.528 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.524+0000 7f74382c7640 10 monclient: monmap: 2024-09-16T04:01:57.528 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:01:57.528 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:01:57.528 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:01:57.526007+0000 2024-09-16T04:01:57.528 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:01:57.526007+0000 2024-09-16T04:01:57.528 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:01:57.528 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:01:57.528 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:01:57.528 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:01:57.528 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:01:57.528 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:01:57.529 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:01:57.529 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:01:57.530 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.524+0000 7f74382c7640 1 Processor -- start 2024-09-16T04:01:57.530 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.524+0000 7f74382c7640 1 -- start start 2024-09-16T04:01:57.531 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.524+0000 7f74382c7640 10 monclient: init 2024-09-16T04:01:57.531 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.524+0000 7f74382c7640 10 monclient: _reopen_session rank -1 2024-09-16T04:01:57.531 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.524+0000 7f74382c7640 10 monclient: _add_conns ranks=[4,0,1,2,3] 2024-09-16T04:01:57.531 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.524+0000 7f74382c7640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f74300a1330 0x7f743009f180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:01:57.531 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.524+0000 7f74382c7640 10 monclient(hunting): picked mon.noname-c con 0x7f74300a1330 addr v2:172.21.15.154:3300/0 2024-09-16T04:01:57.531 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.524+0000 7f74382c7640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f74300a1710 addr v1:172.21.15.42:6789/0 2024-09-16T04:01:57.531 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.524+0000 7f74382c7640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f74301393a0 addr v1:172.21.15.154:6789/0 2024-09-16T04:01:57.531 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.524+0000 7f74382c7640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f743009e4c0 con 0x7f74300a1710 2024-09-16T04:01:57.531 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.524+0000 7f74382c7640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f743009f6c0 con 0x7f74301393a0 2024-09-16T04:01:57.531 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.524+0000 7f74382c7640 10 monclient(hunting): start opening mon connection 2024-09-16T04:01:57.531 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.524+0000 7f74382c7640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f743009edf0 con 0x7f74300a1330 2024-09-16T04:01:57.531 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.524+0000 7f74382c7640 10 monclient(hunting): _renew_subs 2024-09-16T04:01:57.531 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.524+0000 7f74382c7640 10 monclient(hunting): authenticate will time out at 2164.495607s 2024-09-16T04:01:57.532 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.524+0000 7f7435862640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f74300a1710 0x7f74300a1b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:47338/0 (socket says 172.21.15.42:47338) 2024-09-16T04:01:57.532 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.524+0000 7f7435862640 1 -- 172.21.15.42:0/104314699 learned_addr learned my addr 172.21.15.42:0/104314699 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T04:01:57.532 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.528+0000 7f7436063640 1 --2- 172.21.15.42:0/104314699 >> v2:172.21.15.154:3300/0 conn(0x7f74300a1330 0x7f743009f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:01:57.532 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.528+0000 7f7436063640 10 monclient(hunting): get_auth_request con 0x7f74300a1330 auth_method 0 2024-09-16T04:01:57.532 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.528+0000 7f7436063640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:01:57.532 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.528+0000 7f7436063640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:01:57.532 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.528+0000 7f7436063640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:01:57.532 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.528+0000 7f7436063640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:01:57.532 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.528+0000 7f7436063640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:01:57.532 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.528+0000 7f7436063640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:01:57.532 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.528+0000 7f7436063640 10 monclient(hunting): handle_auth_done global_id 904561 payload 274 2024-09-16T04:01:57.532 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.528+0000 7f7436063640 1 -- 172.21.15.42:0/104314699 >> v1:172.21.15.154:6789/0 conn(0x7f74301393a0 legacy=0x7f743013b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:01:57.532 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.528+0000 7f7436063640 1 -- 172.21.15.42:0/104314699 >> v1:172.21.15.42:6789/0 conn(0x7f74300a1710 legacy=0x7f74300a1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:01:57.533 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.528+0000 7f7436063640 10 monclient: _finish_hunting 0 2024-09-16T04:01:57.533 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.528+0000 7f7436063640 1 monclient: found mon.noname-c 2024-09-16T04:01:57.533 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.528+0000 7f7436063640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:01:57.533 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.528+0000 7f7436063640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.154:3300/0 2024-09-16T04:01:57.533 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.528+0000 7f7436063640 1 -- 172.21.15.42:0/104314699 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74300a3170 con 0x7f74300a1330 2024-09-16T04:01:57.538 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.528+0000 7f7436063640 1 --2- 172.21.15.42:0/104314699 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f74300a1330 0x7f743009f180 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f741c0097c0 tx=0x7f741c01c360 comp rx=0 tx=0).ready entity=mon.1 client_cookie=82eb0b8e9edd9486 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:01:57.538 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.528+0000 7f7435061640 1 -- 172.21.15.42:0/104314699 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f741c0b0070 con 0x7f74300a1330 2024-09-16T04:01:57.538 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.528+0000 7f7435061640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:01:57.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.528+0000 7f7435061640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:01:57.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.528+0000 7f7435061640 10 monclient: dump: 2024-09-16T04:01:57.539 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:01:57.539 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:01:57.539 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:57.539 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:57.539 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:01:57.539 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:01:57.539 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:01:57.539 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:01:57.539 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:01:57.539 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:01:57.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.528+0000 7f7435061640 10 monclient: _finish_auth 0 2024-09-16T04:01:57.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.528+0000 7f7435061640 10 monclient: _check_auth_tickets 2024-09-16T04:01:57.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.528+0000 7f7435061640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:01:57.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.528+0000 7f74382c7640 5 monclient: authenticate success, global_id 904561 2024-09-16T04:01:57.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.528+0000 7f74382c7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:01:57.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.528+0000 7f7435061640 1 -- 172.21.15.42:0/104314699 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f741c0ab600 con 0x7f74300a1330 2024-09-16T04:01:57.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.528+0000 7f7435061640 10 monclient: handle_config config(0 keys) 2024-09-16T04:01:57.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.528+0000 7f7435061640 1 -- 172.21.15.42:0/104314699 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f741c0ab9d0 con 0x7f74300a1330 2024-09-16T04:01:57.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.528+0000 7f7435061640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:01:57.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.528+0000 7f7435061640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:01:57.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.528+0000 7f7435061640 10 monclient: dump: 2024-09-16T04:01:57.540 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:01:57.540 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:01:57.541 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:57.541 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:57.541 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:01:57.541 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:01:57.541 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:01:57.541 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:01:57.541 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:01:57.541 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:01:57.541 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.528+0000 7f74382c7640 10 monclient: get_monmap_and_config success 2024-09-16T04:01:57.541 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.528+0000 7f74382c7640 4 set_mon_vals no callback set 2024-09-16T04:01:57.541 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.528+0000 7f74382c7640 10 monclient: shutdown 2024-09-16T04:01:57.541 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.528+0000 7f74382c7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:01:57.542 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.528+0000 7f74382c7640 1 -- 172.21.15.42:0/104314699 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f74300a1330 msgr2=0x7f743009f180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:01:57.542 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.528+0000 7f74382c7640 1 --2- 172.21.15.42:0/104314699 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f74300a1330 0x7f743009f180 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f741c0097c0 tx=0x7f741c01c360 comp rx=0 tx=0).stop 2024-09-16T04:01:57.542 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.528+0000 7f74382c7640 1 -- 172.21.15.42:0/104314699 shutdown_connections 2024-09-16T04:01:57.542 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.528+0000 7f74382c7640 1 --2- 172.21.15.42:0/104314699 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f74300a1330 0x7f743009f180 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:57.542 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.528+0000 7f74382c7640 1 -- 172.21.15.42:0/104314699 >> 172.21.15.42:0/104314699 conn(0x7f74300a2420 msgr2=0x7f74300a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:01:57.542 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.528+0000 7f74382c7640 1 -- 172.21.15.42:0/104314699 shutdown_connections 2024-09-16T04:01:57.542 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.532+0000 7f74382c7640 1 -- 172.21.15.42:0/104314699 wait complete. 2024-09-16T04:01:57.542 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.532+0000 7f74382c7640 10 monclient: build_initial_monmap 2024-09-16T04:01:57.542 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.532+0000 7f74382c7640 10 monclient: monmap: 2024-09-16T04:01:57.542 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:01:57.542 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:01:57.542 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:01:57.543 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:01:57.543 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:01:57.543 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:01:57.543 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:01:57.543 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:01:57.543 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:01:57.543 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:01:57.543 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.532+0000 7f74382c7640 1 librados: starting msgr at 2024-09-16T04:01:57.543 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.532+0000 7f74382c7640 1 librados: starting objecter 2024-09-16T04:01:57.544 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.532+0000 7f74382c7640 1 Processor -- start 2024-09-16T04:01:57.544 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.532+0000 7f74382c7640 1 -- start start 2024-09-16T04:01:57.544 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.532+0000 7f74382c7640 1 librados: setting wanted keys 2024-09-16T04:01:57.544 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.532+0000 7f74382c7640 1 librados: calling monclient init 2024-09-16T04:01:57.544 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.532+0000 7f74382c7640 10 monclient: init 2024-09-16T04:01:57.544 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.532+0000 7f74382c7640 10 monclient: _reopen_session rank -1 2024-09-16T04:01:57.544 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.532+0000 7f74382c7640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:01:57.544 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.532+0000 7f74382c7640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f74300a1330 0x7f74301338e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:01:57.544 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.532+0000 7f74382c7640 10 monclient(hunting): picked mon.noname-c con 0x7f74300a1330 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:01:57.544 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.532+0000 7f74382c7640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f74300a1710 0x7f7430133e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:01:57.544 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.532+0000 7f74382c7640 10 monclient(hunting): picked mon.noname-b con 0x7f74300a1710 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:01:57.544 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.532+0000 7f74382c7640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f74301393a0 0x7f743006f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:01:57.544 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.532+0000 7f74382c7640 10 monclient(hunting): picked mon.noname-a con 0x7f74301393a0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:01:57.545 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.532+0000 7f74382c7640 10 monclient(hunting): start opening mon connection 2024-09-16T04:01:57.545 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.532+0000 7f74382c7640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f743013f8e0 con 0x7f74301393a0 2024-09-16T04:01:57.545 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.532+0000 7f74382c7640 10 monclient(hunting): start opening mon connection 2024-09-16T04:01:57.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.532+0000 7f74382c7640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f743009cd60 con 0x7f74300a1710 2024-09-16T04:01:57.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.532+0000 7f74382c7640 10 monclient(hunting): start opening mon connection 2024-09-16T04:01:57.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.532+0000 7f74382c7640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f743013fa60 con 0x7f74300a1330 2024-09-16T04:01:57.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.532+0000 7f74382c7640 10 monclient(hunting): _renew_subs 2024-09-16T04:01:57.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.532+0000 7f7436063640 1 -- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f74300a1330 msgr2=0x7f74301338e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T04:01:57.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.532+0000 7f74382c7640 10 monclient(hunting): authenticate will time out at 2164.501654s 2024-09-16T04:01:57.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.532+0000 7f7436063640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f74300a1330 0x7f74301338e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:01:57.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.532+0000 7f7436864640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f74301393a0 0x7f743006f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:01:57.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.532+0000 7f7436864640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f74301393a0 0x7f743006f3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:37342/0 (socket says 172.21.15.42:37342) 2024-09-16T04:01:57.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.532+0000 7f7436864640 1 -- 172.21.15.42:0/3326389143 learned_addr learned my addr 172.21.15.42:0/3326389143 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:01:57.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.532+0000 7f7436864640 10 monclient(hunting): get_auth_request con 0x7f74301393a0 auth_method 0 2024-09-16T04:01:57.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.532+0000 7f7436864640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:01:57.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.532+0000 7f7436864640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:01:57.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.532+0000 7f7436864640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:01:57.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.532+0000 7f7435862640 1 --2- 172.21.15.42:0/3326389143 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f74300a1710 0x7f7430133e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:01:57.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.532+0000 7f7435862640 10 monclient(hunting): get_auth_request con 0x7f74300a1710 auth_method 0 2024-09-16T04:01:57.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.532+0000 7f7435862640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:01:57.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.532+0000 7f7435862640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:01:57.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.532+0000 7f7435862640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:01:57.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.532+0000 7f7436864640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:01:57.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.532+0000 7f7436864640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:01:57.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.532+0000 7f7436864640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:01:57.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.532+0000 7f7436864640 10 monclient(hunting): handle_auth_done global_id 825462 payload 824 2024-09-16T04:01:57.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.532+0000 7f7436864640 1 -- 172.21.15.42:0/3326389143 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f74300a1330 msgr2=0x7f74301338e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:01:57.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.532+0000 7f7436864640 1 --2- 172.21.15.42:0/3326389143 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f74300a1330 0x7f74301338e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:57.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.532+0000 7f7436864640 1 -- 172.21.15.42:0/3326389143 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f74300a1710 msgr2=0x7f7430133e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:01:57.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.532+0000 7f7436864640 1 --2- 172.21.15.42:0/3326389143 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f74300a1710 0x7f7430133e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:57.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.532+0000 7f7436864640 10 monclient: _finish_hunting 0 2024-09-16T04:01:57.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.532+0000 7f7436864640 1 monclient: found mon.noname-a 2024-09-16T04:01:57.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.532+0000 7f7436864640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:01:57.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.532+0000 7f7436864640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:01:57.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.532+0000 7f7436864640 1 -- 172.21.15.42:0/3326389143 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f743011c2f0 con 0x7f74301393a0 2024-09-16T04:01:57.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.532+0000 7f7436864640 1 --2- 172.21.15.42:0/3326389143 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f74301393a0 0x7f743006f3c0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f742800c430 tx=0x7f742800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:01:57.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.532+0000 7f742effd640 10 client.?.objecter ms_handle_connect 0x7f74301393a0 2024-09-16T04:01:57.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.532+0000 7f742effd640 10 client.?.objecter resend_mon_ops 2024-09-16T04:01:57.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.532+0000 7f742effd640 1 -- 172.21.15.42:0/3326389143 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7428005220 con 0x7f74301393a0 2024-09-16T04:01:57.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.532+0000 7f742effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:01:57.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.532+0000 7f742effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:01:57.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.532+0000 7f742effd640 10 monclient: dump: 2024-09-16T04:01:57.549 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:01:57.549 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:01:57.549 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:57.550 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:57.550 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:01:57.550 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:01:57.550 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:01:57.550 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:01:57.550 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:01:57.550 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:01:57.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.532+0000 7f742effd640 10 monclient: _finish_auth 0 2024-09-16T04:01:57.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.532+0000 7f742effd640 10 monclient: _check_auth_tickets 2024-09-16T04:01:57.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.532+0000 7f742effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:01:57.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.532+0000 7f74382c7640 5 monclient: authenticate success, global_id 825462 2024-09-16T04:01:57.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.532+0000 7f74382c7640 10 monclient: _renew_subs 2024-09-16T04:01:57.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.532+0000 7f74382c7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:01:57.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.532+0000 7f742effd640 1 -- 172.21.15.42:0/3326389143 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f74280053c0 con 0x7f74301393a0 2024-09-16T04:01:57.551 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.532+0000 7f74382c7640 1 -- 172.21.15.42:0/3326389143 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7430071b20 con 0x7f74301393a0 2024-09-16T04:01:57.551 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.532+0000 7f742effd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:01:57.551 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.532+0000 7f742effd640 1 -- 172.21.15.42:0/3326389143 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7428007270 con 0x7f74301393a0 2024-09-16T04:01:57.551 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.532+0000 7f742effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:01:57.551 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.532+0000 7f742effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:01:57.551 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.532+0000 7f742effd640 10 monclient: dump: 2024-09-16T04:01:57.551 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:01:57.551 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:01:57.551 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:57.551 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:01:57.551 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:01:57.551 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:01:57.551 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:01:57.552 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:01:57.552 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:01:57.552 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:01:57.552 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.532+0000 7f7434860640 4 set_mon_vals no callback set 2024-09-16T04:01:57.552 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.532+0000 7f74382c7640 10 client.825462.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:01:57.552 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.532+0000 7f74382c7640 10 monclient: _renew_subs 2024-09-16T04:01:57.552 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.532+0000 7f74382c7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:01:57.552 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.532+0000 7f74382c7640 1 -- 172.21.15.42:0/3326389143 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f743013c2e0 con 0x7f74301393a0 2024-09-16T04:01:57.552 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.532+0000 7f74382c7640 1 librados: init done 2024-09-16T04:01:57.552 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.532+0000 7f74382c7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:01:57.552 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.532+0000 7f74382c7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:01:57.552 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.532+0000 7f74382c7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:01:57.552 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.532+0000 7f74382c7640 1 -- 172.21.15.42:0/3326389143 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f743009f180 con 0x7f74301393a0 2024-09-16T04:01:57.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.532+0000 7f742effd640 1 -- 172.21.15.42:0/3326389143 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f7428019020 con 0x7f74301393a0 2024-09-16T04:01:57.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.532+0000 7f742effd640 1 --2- 172.21.15.42:0/3326389143 >> v2:172.21.15.154:6812/254845342 conn(0x7f74140377c0 0x7f7414039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:01:57.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.536+0000 7f742effd640 1 -- 172.21.15.42:0/3326389143 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(602..602 src has 451..602) ==== 6252+0+0 (secure 0 0 0) 0x7f7428048470 con 0x7f74301393a0 2024-09-16T04:01:57.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.536+0000 7f742effd640 10 client.825462.objecter ms_dispatch 0x7f7430000ba0 osd_map(602..602 src has 451..602) 2024-09-16T04:01:57.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.536+0000 7f742effd640 3 client.825462.objecter handle_osd_map got epochs [602,602] > 0 2024-09-16T04:01:57.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.536+0000 7f742effd640 3 client.825462.objecter handle_osd_map decoding full epoch 602 2024-09-16T04:01:57.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.536+0000 7f742effd640 20 client.825462.objecter dump_active .. 0 homeless 2024-09-16T04:01:57.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.536+0000 7f7436063640 1 --2- 172.21.15.42:0/3326389143 >> v2:172.21.15.154:6812/254845342 conn(0x7f74140377c0 0x7f7414039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:01:57.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.536+0000 7f7436063640 10 monclient: get_auth_request con 0x7f74140377c0 auth_method 0 2024-09-16T04:01:57.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.536+0000 7f742effd640 10 client.825462.objecter ms_handle_connect 0x7f74140377c0 2024-09-16T04:01:57.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.536+0000 7f7436063640 1 --2- 172.21.15.42:0/3326389143 >> v2:172.21.15.154:6812/254845342 conn(0x7f74140377c0 0x7f7414039c80 secure :-1 s=READY pgs=3775 cs=0 l=1 rev1=1 crypto rx=0x7f741c0ad000 tx=0x7f741c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:01:57.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.540+0000 7f742effd640 1 -- 172.21.15.42:0/3326389143 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f7428006d00 con 0x7f74301393a0 2024-09-16T04:01:57.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.540+0000 7f742effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:01:57.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.540+0000 7f742effd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:01:57.766 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.764+0000 7f74382c7640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:01:57.767 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.764+0000 7f74382c7640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:01:57.767 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.764+0000 7f74382c7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:01:57.767 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.764+0000 7f74382c7640 1 -- 172.21.15.42:0/3326389143 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f7430069040 con 0x7f74301393a0 2024-09-16T04:01:57.767 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.764+0000 7f742effd640 1 -- 172.21.15.42:0/3326389143 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1562 (secure 0 0 0) 0x7f74280067e0 con 0x7f74301393a0 2024-09-16T04:01:57.767 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.764+0000 7f742effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:01:57.767 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.764+0000 7f742effd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:01:57.768 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":178,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":14,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:01:57.771 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.768+0000 7f74382c7640 10 librados: watch_flush enter 2024-09-16T04:01:57.771 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.768+0000 7f74382c7640 10 librados: watch_flush exit 2024-09-16T04:01:57.771 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.768+0000 7f74382c7640 20 client.825462.objecter shutdown clearing up homeless session... 2024-09-16T04:01:57.771 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.768+0000 7f74382c7640 10 client.825462.objecter successfully canceled tick 2024-09-16T04:01:57.771 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.768+0000 7f74382c7640 1 -- 172.21.15.42:0/3326389143 >> v2:172.21.15.154:6812/254845342 conn(0x7f74140377c0 msgr2=0x7f7414039c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:01:57.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.768+0000 7f74382c7640 1 --2- 172.21.15.42:0/3326389143 >> v2:172.21.15.154:6812/254845342 conn(0x7f74140377c0 0x7f7414039c80 secure :-1 s=READY pgs=3775 cs=0 l=1 rev1=1 crypto rx=0x7f741c0ad000 tx=0x7f741c008040 comp rx=0 tx=0).stop 2024-09-16T04:01:57.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.768+0000 7f74382c7640 10 monclient: shutdown 2024-09-16T04:01:57.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.768+0000 7f74382c7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:01:57.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.768+0000 7f74382c7640 1 -- 172.21.15.42:0/3326389143 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f74301393a0 msgr2=0x7f743006f3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:01:57.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.768+0000 7f74382c7640 1 --2- 172.21.15.42:0/3326389143 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f74301393a0 0x7f743006f3c0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f742800c430 tx=0x7f742800c8f0 comp rx=0 tx=0).stop 2024-09-16T04:01:57.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.768+0000 7f74382c7640 1 -- 172.21.15.42:0/3326389143 shutdown_connections 2024-09-16T04:01:57.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.768+0000 7f74382c7640 1 --2- 172.21.15.42:0/3326389143 >> v2:172.21.15.154:6812/254845342 conn(0x7f74140377c0 0x7f7414039c80 unknown :-1 s=CLOSED pgs=3775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:57.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.768+0000 7f74382c7640 1 --2- 172.21.15.42:0/3326389143 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f74301393a0 0x7f743006f3c0 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:57.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.768+0000 7f74382c7640 1 --2- 172.21.15.42:0/3326389143 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f74300a1710 0x7f7430133e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:57.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.768+0000 7f74382c7640 1 --2- 172.21.15.42:0/3326389143 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f74300a1330 0x7f74301338e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:01:57.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.768+0000 7f74382c7640 1 -- 172.21.15.42:0/3326389143 >> 172.21.15.42:0/3326389143 conn(0x7f74300a2420 msgr2=0x7f743009c270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:01:57.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.768+0000 7f74382c7640 1 -- 172.21.15.42:0/3326389143 shutdown_connections 2024-09-16T04:01:57.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.768+0000 7f74382c7640 1 -- 172.21.15.42:0/3326389143 wait complete. 2024-09-16T04:01:57.773 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:01:57.768+0000 7f74382c7640 1 librados: shutdown 2024-09-16T04:02:00.788 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:02:00.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:00.915+0000 7f7ce172e640 10 monclient: get_monmap_and_config 2024-09-16T04:02:00.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:00.915+0000 7f7ce172e640 10 monclient: build_initial_monmap 2024-09-16T04:02:00.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:00.915+0000 7f7ce172e640 10 monclient: monmap: 2024-09-16T04:02:00.919 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:02:00.919 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:02:00.919 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:02:00.917917+0000 2024-09-16T04:02:00.920 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:02:00.917917+0000 2024-09-16T04:02:00.920 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:02:00.920 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:02:00.920 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:02:00.920 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:02:00.920 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:02:00.920 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:02:00.920 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:02:00.920 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:02:00.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:00.915+0000 7f7ce172e640 1 Processor -- start 2024-09-16T04:02:00.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:00.915+0000 7f7ce172e640 1 -- start start 2024-09-16T04:02:00.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:00.915+0000 7f7ce172e640 10 monclient: init 2024-09-16T04:02:00.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:00.915+0000 7f7ce172e640 10 monclient: _reopen_session rank -1 2024-09-16T04:02:00.923 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:00.915+0000 7f7ce172e640 10 monclient: _add_conns ranks=[4,2,3,1,0] 2024-09-16T04:02:00.923 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:00.915+0000 7f7ce172e640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f7cdc0a1330 0x7f7cdc0a1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:00.923 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:00.915+0000 7f7ce172e640 10 monclient(hunting): picked mon.noname-c con 0x7f7cdc0a1330 addr v2:172.21.15.154:3300/0 2024-09-16T04:02:00.923 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:00.915+0000 7f7ce172e640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7cdc06dac0 0x7f7cdc0a1c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:00.923 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:00.915+0000 7f7ce172e640 10 monclient(hunting): picked mon.noname-b con 0x7f7cdc06dac0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:02:00.923 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:00.915+0000 7f7ce172e640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f7cdc135100 0x7f7cdc135550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:00.923 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:00.915+0000 7f7ce172e640 10 monclient(hunting): picked mon.noname-a con 0x7f7cdc135100 addr v2:172.21.15.42:3300/0 2024-09-16T04:02:00.923 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:00.915+0000 7f7ce172e640 10 monclient(hunting): start opening mon connection 2024-09-16T04:02:00.923 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:00.915+0000 7f7ce172e640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f7cdc09c7d0 con 0x7f7cdc135100 2024-09-16T04:02:00.923 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:00.915+0000 7f7ce172e640 10 monclient(hunting): start opening mon connection 2024-09-16T04:02:00.923 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:00.915+0000 7f7ce172e640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f7cdc09c090 con 0x7f7cdc0a1330 2024-09-16T04:02:00.923 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:00.915+0000 7f7ce172e640 10 monclient(hunting): start opening mon connection 2024-09-16T04:02:00.923 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:00.915+0000 7f7ce172e640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f7cdc09f610 con 0x7f7cdc06dac0 2024-09-16T04:02:00.924 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:00.915+0000 7f7ce172e640 10 monclient(hunting): _renew_subs 2024-09-16T04:02:00.924 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:00.915+0000 7f7ce172e640 10 monclient(hunting): authenticate will time out at 2167.887317s 2024-09-16T04:02:00.924 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:00.915+0000 7f7cda7fc640 1 -- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7cdc06dac0 msgr2=0x7f7cdc0a1c50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T04:02:00.924 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:00.915+0000 7f7cda7fc640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7cdc06dac0 0x7f7cdc0a1c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:02:00.924 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:00.915+0000 7f7cdb7fe640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f7cdc135100 0x7f7cdc135550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:02:00.924 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:00.915+0000 7f7cdb7fe640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f7cdc135100 0x7f7cdc135550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:37350/0 (socket says 172.21.15.42:37350) 2024-09-16T04:02:00.924 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:00.915+0000 7f7cdb7fe640 1 -- 172.21.15.42:0/2425161035 learned_addr learned my addr 172.21.15.42:0/2425161035 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:02:00.924 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:00.915+0000 7f7cdb7fe640 10 monclient(hunting): get_auth_request con 0x7f7cdc135100 auth_method 0 2024-09-16T04:02:00.924 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:00.915+0000 7f7cdb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:02:00.924 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:00.915+0000 7f7cdb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:02:00.924 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:00.915+0000 7f7cdb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:02:00.924 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:00.915+0000 7f7cdb7fe640 1 -- 172.21.15.42:0/2425161035 >> v2:172.21.15.42:3300/0 conn(0x7f7cdc135100 msgr2=0x7f7cdc135550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:02:00.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:00.915+0000 7f7cdb7fe640 1 -- 172.21.15.42:0/2425161035 >> v2:172.21.15.42:3300/0 conn(0x7f7cdc135100 msgr2=0x7f7cdc135550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:02:00.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:00.915+0000 7f7cdb7fe640 1 --2- 172.21.15.42:0/2425161035 >> v2:172.21.15.42:3300/0 conn(0x7f7cdc135100 0x7f7cdc135550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:02:00.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:00.915+0000 7f7cdb7fe640 1 --2- 172.21.15.42:0/2425161035 >> v2:172.21.15.42:3300/0 conn(0x7f7cdc135100 0x7f7cdc135550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:02:00.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:00.915+0000 7f7cdaffd640 1 --2- 172.21.15.42:0/2425161035 >> v2:172.21.15.154:3300/0 conn(0x7f7cdc0a1330 0x7f7cdc0a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:02:00.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:00.915+0000 7f7cdaffd640 10 monclient(hunting): get_auth_request con 0x7f7cdc0a1330 auth_method 0 2024-09-16T04:02:00.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:00.915+0000 7f7cdaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:02:00.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:00.915+0000 7f7cdaffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:02:00.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:00.915+0000 7f7cdaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:02:00.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:00.915+0000 7f7cdaffd640 1 -- 172.21.15.42:0/2425161035 >> v2:172.21.15.154:3300/0 conn(0x7f7cdc0a1330 msgr2=0x7f7cdc0a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:02:00.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:00.915+0000 7f7cdaffd640 1 -- 172.21.15.42:0/2425161035 >> v2:172.21.15.154:3300/0 conn(0x7f7cdc0a1330 msgr2=0x7f7cdc0a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:02:00.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:00.915+0000 7f7cdaffd640 1 --2- 172.21.15.42:0/2425161035 >> v2:172.21.15.154:3300/0 conn(0x7f7cdc0a1330 0x7f7cdc0a1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:02:00.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:00.919+0000 7f7cdaffd640 1 --2- 172.21.15.42:0/2425161035 >> v2:172.21.15.154:3300/0 conn(0x7f7cdc0a1330 0x7f7cdc0a1710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:02:01.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:01.119+0000 7f7cda7fc640 1 -- 172.21.15.42:0/2425161035 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7cdc06dac0 msgr2=0x7f7cdc0a1c50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T04:02:01.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:01.119+0000 7f7cda7fc640 1 --2- 172.21.15.42:0/2425161035 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7cdc06dac0 0x7f7cdc0a1c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T04:02:01.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:01.119+0000 7f7cdb7fe640 1 --2- 172.21.15.42:0/2425161035 >> v2:172.21.15.42:3300/0 conn(0x7f7cdc135100 0x7f7cdc135550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:02:01.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:01.119+0000 7f7cdb7fe640 10 monclient(hunting): get_auth_request con 0x7f7cdc135100 auth_method 0 2024-09-16T04:02:01.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:01.119+0000 7f7cdb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:02:01.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:01.119+0000 7f7cdb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:02:01.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:01.119+0000 7f7cdb7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:02:01.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:01.119+0000 7f7cdb7fe640 1 -- 172.21.15.42:0/2425161035 >> v2:172.21.15.42:3300/0 conn(0x7f7cdc135100 msgr2=0x7f7cdc135550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:02:01.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:01.119+0000 7f7cdb7fe640 1 -- 172.21.15.42:0/2425161035 >> v2:172.21.15.42:3300/0 conn(0x7f7cdc135100 msgr2=0x7f7cdc135550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:02:01.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:01.119+0000 7f7cdb7fe640 1 --2- 172.21.15.42:0/2425161035 >> v2:172.21.15.42:3300/0 conn(0x7f7cdc135100 0x7f7cdc135550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:02:01.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:01.119+0000 7f7cdb7fe640 1 --2- 172.21.15.42:0/2425161035 >> v2:172.21.15.42:3300/0 conn(0x7f7cdc135100 0x7f7cdc135550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T04:02:01.126 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:01.123+0000 7f7cdaffd640 1 --2- 172.21.15.42:0/2425161035 >> v2:172.21.15.154:3300/0 conn(0x7f7cdc0a1330 0x7f7cdc0a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:02:01.126 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:01.123+0000 7f7cdaffd640 10 monclient(hunting): get_auth_request con 0x7f7cdc0a1330 auth_method 0 2024-09-16T04:02:01.126 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:01.123+0000 7f7cdaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:02:01.126 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:01.123+0000 7f7cdaffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:02:01.126 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:01.123+0000 7f7cdaffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:02:01.126 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:01.123+0000 7f7cdaffd640 1 -- 172.21.15.42:0/2425161035 >> v2:172.21.15.154:3300/0 conn(0x7f7cdc0a1330 msgr2=0x7f7cdc0a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:02:01.126 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:01.123+0000 7f7cdaffd640 1 -- 172.21.15.42:0/2425161035 >> v2:172.21.15.154:3300/0 conn(0x7f7cdc0a1330 msgr2=0x7f7cdc0a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:02:01.126 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:01.123+0000 7f7cdaffd640 1 --2- 172.21.15.42:0/2425161035 >> v2:172.21.15.154:3300/0 conn(0x7f7cdc0a1330 0x7f7cdc0a1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:02:01.126 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:01.123+0000 7f7cdaffd640 1 --2- 172.21.15.42:0/2425161035 >> v2:172.21.15.154:3300/0 conn(0x7f7cdc0a1330 0x7f7cdc0a1710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T04:02:01.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:01.523+0000 7f7cda7fc640 1 -- 172.21.15.42:0/2425161035 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7cdc06dac0 msgr2=0x7f7cdc0a1c50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T04:02:01.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:01.523+0000 7f7cda7fc640 1 --2- 172.21.15.42:0/2425161035 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7cdc06dac0 0x7f7cdc0a1c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T04:02:01.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:01.523+0000 7f7cdb7fe640 1 --2- 172.21.15.42:0/2425161035 >> v2:172.21.15.42:3300/0 conn(0x7f7cdc135100 0x7f7cdc135550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:02:01.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:01.523+0000 7f7cdb7fe640 10 monclient(hunting): get_auth_request con 0x7f7cdc135100 auth_method 0 2024-09-16T04:02:01.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:01.523+0000 7f7cdb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:02:01.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:01.523+0000 7f7cdb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:02:01.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:01.523+0000 7f7cdb7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:02:01.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:01.523+0000 7f7cdb7fe640 1 -- 172.21.15.42:0/2425161035 >> v2:172.21.15.42:3300/0 conn(0x7f7cdc135100 msgr2=0x7f7cdc135550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:02:01.527 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:01.523+0000 7f7cdb7fe640 1 -- 172.21.15.42:0/2425161035 >> v2:172.21.15.42:3300/0 conn(0x7f7cdc135100 msgr2=0x7f7cdc135550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:02:01.527 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:01.523+0000 7f7cdb7fe640 1 --2- 172.21.15.42:0/2425161035 >> v2:172.21.15.42:3300/0 conn(0x7f7cdc135100 0x7f7cdc135550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:02:01.527 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:01.523+0000 7f7cdb7fe640 1 --2- 172.21.15.42:0/2425161035 >> v2:172.21.15.42:3300/0 conn(0x7f7cdc135100 0x7f7cdc135550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T04:02:01.530 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:01.527+0000 7f7cdaffd640 1 --2- 172.21.15.42:0/2425161035 >> v2:172.21.15.154:3300/0 conn(0x7f7cdc0a1330 0x7f7cdc0a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:02:01.530 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:01.527+0000 7f7cdaffd640 10 monclient(hunting): get_auth_request con 0x7f7cdc0a1330 auth_method 0 2024-09-16T04:02:01.530 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:01.527+0000 7f7cdaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:02:01.530 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:01.527+0000 7f7cdaffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:02:01.530 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:01.527+0000 7f7cdaffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:02:01.530 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:01.527+0000 7f7cdaffd640 1 -- 172.21.15.42:0/2425161035 >> v2:172.21.15.154:3300/0 conn(0x7f7cdc0a1330 msgr2=0x7f7cdc0a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:02:01.530 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:01.527+0000 7f7cdaffd640 1 -- 172.21.15.42:0/2425161035 >> v2:172.21.15.154:3300/0 conn(0x7f7cdc0a1330 msgr2=0x7f7cdc0a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:02:01.530 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:01.527+0000 7f7cdaffd640 1 --2- 172.21.15.42:0/2425161035 >> v2:172.21.15.154:3300/0 conn(0x7f7cdc0a1330 0x7f7cdc0a1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:02:01.530 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:01.527+0000 7f7cdaffd640 1 --2- 172.21.15.42:0/2425161035 >> v2:172.21.15.154:3300/0 conn(0x7f7cdc0a1330 0x7f7cdc0a1710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T04:02:02.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:02.327+0000 7f7cda7fc640 1 -- 172.21.15.42:0/2425161035 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7cdc06dac0 msgr2=0x7f7cdc0a1c50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T04:02:02.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:02.327+0000 7f7cda7fc640 1 --2- 172.21.15.42:0/2425161035 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7cdc06dac0 0x7f7cdc0a1c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T04:02:02.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:02.327+0000 7f7cdb7fe640 1 --2- 172.21.15.42:0/2425161035 >> v2:172.21.15.42:3300/0 conn(0x7f7cdc135100 0x7f7cdc135550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:02:02.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:02.327+0000 7f7cdb7fe640 10 monclient(hunting): get_auth_request con 0x7f7cdc135100 auth_method 0 2024-09-16T04:02:02.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:02.327+0000 7f7cdb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:02:02.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:02.327+0000 7f7cdb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:02:02.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:02.327+0000 7f7cdb7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:02:02.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:02.327+0000 7f7cdb7fe640 1 -- 172.21.15.42:0/2425161035 >> v2:172.21.15.42:3300/0 conn(0x7f7cdc135100 msgr2=0x7f7cdc135550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:02:02.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:02.327+0000 7f7cdb7fe640 1 -- 172.21.15.42:0/2425161035 >> v2:172.21.15.42:3300/0 conn(0x7f7cdc135100 msgr2=0x7f7cdc135550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:02:02.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:02.327+0000 7f7cdb7fe640 1 --2- 172.21.15.42:0/2425161035 >> v2:172.21.15.42:3300/0 conn(0x7f7cdc135100 0x7f7cdc135550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:02:02.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:02.327+0000 7f7cdb7fe640 1 --2- 172.21.15.42:0/2425161035 >> v2:172.21.15.42:3300/0 conn(0x7f7cdc135100 0x7f7cdc135550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T04:02:02.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:02.331+0000 7f7cdaffd640 1 --2- 172.21.15.42:0/2425161035 >> v2:172.21.15.154:3300/0 conn(0x7f7cdc0a1330 0x7f7cdc0a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:02:02.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:02.331+0000 7f7cdaffd640 10 monclient(hunting): get_auth_request con 0x7f7cdc0a1330 auth_method 0 2024-09-16T04:02:02.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:02.331+0000 7f7cdaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:02:02.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:02.331+0000 7f7cdaffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:02:02.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:02.331+0000 7f7cdaffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:02:02.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:02.331+0000 7f7cdaffd640 1 -- 172.21.15.42:0/2425161035 >> v2:172.21.15.154:3300/0 conn(0x7f7cdc0a1330 msgr2=0x7f7cdc0a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:02:02.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:02.331+0000 7f7cdaffd640 1 -- 172.21.15.42:0/2425161035 >> v2:172.21.15.154:3300/0 conn(0x7f7cdc0a1330 msgr2=0x7f7cdc0a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:02:02.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:02.331+0000 7f7cdaffd640 1 --2- 172.21.15.42:0/2425161035 >> v2:172.21.15.154:3300/0 conn(0x7f7cdc0a1330 0x7f7cdc0a1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:02:02.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:02.331+0000 7f7cdaffd640 1 --2- 172.21.15.42:0/2425161035 >> v2:172.21.15.154:3300/0 conn(0x7f7cdc0a1330 0x7f7cdc0a1710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T04:02:03.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:03.915+0000 7f7cd8ff9640 10 monclient(hunting): tick 2024-09-16T04:02:03.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:03.915+0000 7f7cd8ff9640 10 monclient(hunting): _check_auth_tickets 2024-09-16T04:02:03.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:03.915+0000 7f7cd8ff9640 1 monclient(hunting): continuing hunt 2024-09-16T04:02:03.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:03.915+0000 7f7cd8ff9640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T04:02:03.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:03.915+0000 7f7cd8ff9640 1 -- 172.21.15.42:0/2425161035 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7cdc06dac0 msgr2=0x7f7cdc0a1c50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:02:03.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:03.915+0000 7f7cd8ff9640 1 --2- 172.21.15.42:0/2425161035 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7cdc06dac0 0x7f7cdc0a1c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:03.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:03.915+0000 7f7cd8ff9640 1 -- 172.21.15.42:0/2425161035 >> v2:172.21.15.154:3300/0 conn(0x7f7cdc0a1330 msgr2=0x7f7cdc0a1710 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:02:03.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:03.915+0000 7f7cd8ff9640 1 --2- 172.21.15.42:0/2425161035 >> v2:172.21.15.154:3300/0 conn(0x7f7cdc0a1330 0x7f7cdc0a1710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:03.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:03.915+0000 7f7cd8ff9640 1 -- 172.21.15.42:0/2425161035 >> v2:172.21.15.42:3300/0 conn(0x7f7cdc135100 msgr2=0x7f7cdc135550 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:02:03.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:03.915+0000 7f7cd8ff9640 1 --2- 172.21.15.42:0/2425161035 >> v2:172.21.15.42:3300/0 conn(0x7f7cdc135100 0x7f7cdc135550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:03.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:03.915+0000 7f7cd8ff9640 10 monclient: _add_conns ranks=[1,0] 2024-09-16T04:02:03.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:03.915+0000 7f7cd8ff9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7cc80036a0 addr v1:172.21.15.154:6789/0 2024-09-16T04:02:03.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:03.915+0000 7f7cd8ff9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7cc8007330 addr v1:172.21.15.42:6789/0 2024-09-16T04:02:03.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:03.915+0000 7f7cd8ff9640 1 -- 172.21.15.42:0/2425161035 --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7cdc09f8f0 con 0x7f7cc8007330 2024-09-16T04:02:03.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:03.915+0000 7f7cd8ff9640 1 -- 172.21.15.42:0/2425161035 --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7cc800c0b0 con 0x7f7cc80036a0 2024-09-16T04:02:03.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:03.915+0000 7f7cd8ff9640 10 monclient(hunting): _renew_subs 2024-09-16T04:02:05.562 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.559+0000 7f7cd9ffb640 1 -- 172.21.15.42:0/2425161035 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4064910651 0 0) 0x7f7cc800c0b0 con 0x7f7cc80036a0 2024-09-16T04:02:05.562 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.559+0000 7f7cd9ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:02:05.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.559+0000 7f7cd9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:02:05.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.559+0000 7f7cd9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:02:05.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.559+0000 7f7cd9ffb640 10 monclient(hunting): my global_id is 904567 2024-09-16T04:02:05.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.559+0000 7f7cd9ffb640 1 -- 172.21.15.42:0/2425161035 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7cc4003530 con 0x7f7cc80036a0 2024-09-16T04:02:05.565 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.559+0000 7f7cd9ffb640 1 -- 172.21.15.42:0/2425161035 <== mon.1 v1:172.21.15.154:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1790769427 0 0) 0x7f7cc4003530 con 0x7f7cc80036a0 2024-09-16T04:02:05.565 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.559+0000 7f7cd9ffb640 1 -- 172.21.15.42:0/2425161035 >> v1:172.21.15.42:6789/0 conn(0x7f7cc8007330 legacy=0x7f7cc80097f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:02:05.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.559+0000 7f7cd9ffb640 10 monclient: _finish_hunting 0 2024-09-16T04:02:05.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.559+0000 7f7cd9ffb640 1 monclient: found mon.noname-c-legacy 2024-09-16T04:02:05.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.559+0000 7f7cd9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:02:05.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.559+0000 7f7cd9ffb640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.154:6789/0 2024-09-16T04:02:05.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.559+0000 7f7cd9ffb640 1 -- 172.21.15.42:0/2425161035 --> v1:172.21.15.154:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7cc800d350 con 0x7f7cc80036a0 2024-09-16T04:02:05.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.559+0000 7f7cd9ffb640 10 monclient: _finish_auth 0 2024-09-16T04:02:05.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.559+0000 7f7cd9ffb640 10 monclient: _check_auth_tickets 2024-09-16T04:02:05.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.559+0000 7f7cd9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:02:05.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.559+0000 7f7cd9ffb640 1 -- 172.21.15.42:0/2425161035 <== mon.1 v1:172.21.15.154:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2177413734 0 0) 0x7f7ccc00f040 con 0x7f7cc80036a0 2024-09-16T04:02:05.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.559+0000 7f7cd9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:02:05.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.559+0000 7f7cd9ffb640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-16T04:02:05.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.559+0000 7f7cd9ffb640 10 monclient: dump: 2024-09-16T04:02:05.567 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:02:05.567 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:02:05.567 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:05.567 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:05.567 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:02:05.567 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:02:05.567 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:02:05.567 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:02:05.567 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:02:05.567 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:02:05.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.559+0000 7f7cd9ffb640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] but i'm connected to v1:172.21.15.154:6789/0, reconnecting 2024-09-16T04:02:05.568 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.559+0000 7f7cd9ffb640 10 monclient: _reopen_session rank -1 2024-09-16T04:02:05.568 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.559+0000 7f7cd9ffb640 1 -- 172.21.15.42:0/2425161035 >> v1:172.21.15.154:6789/0 conn(0x7f7cc80036a0 legacy=0x7f7cc8005b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:02:05.568 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.559+0000 7f7cdb7fe640 1 -- 172.21.15.42:0/2425161035 reap_dead start 2024-09-16T04:02:05.568 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.559+0000 7f7cd9ffb640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:02:05.568 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.559+0000 7f7cd9ffb640 1 --2- 172.21.15.42:0/2425161035 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7cc40039c0 0x7f7cc4003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:05.568 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.559+0000 7f7cd9ffb640 10 monclient(hunting): picked mon.c con 0x7f7cc40039c0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:02:05.568 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.559+0000 7f7cd9ffb640 1 --2- 172.21.15.42:0/2425161035 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7cc4004810 0x7f7cc4004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:05.568 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.559+0000 7f7cd9ffb640 10 monclient(hunting): picked mon.a con 0x7f7cc4004810 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:02:05.568 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.559+0000 7f7cd9ffb640 1 --2- 172.21.15.42:0/2425161035 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7cc40053d0 0x7f7cc400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:05.568 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.559+0000 7f7cdaffd640 1 -- 172.21.15.42:0/2425161035 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7cc40039c0 msgr2=0x7f7cc4003dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T04:02:05.568 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.559+0000 7f7cd9ffb640 10 monclient(hunting): picked mon.b con 0x7f7cc40053d0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:02:05.569 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.559+0000 7f7cd9ffb640 10 monclient(hunting): start opening mon connection 2024-09-16T04:02:05.569 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.559+0000 7f7cd9ffb640 1 -- 172.21.15.42:0/2425161035 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f7cc400c240 con 0x7f7cc4004810 2024-09-16T04:02:05.569 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.559+0000 7f7cdaffd640 1 --2- 172.21.15.42:0/2425161035 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7cc40039c0 0x7f7cc4003dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:02:05.569 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.559+0000 7f7cd9ffb640 10 monclient(hunting): start opening mon connection 2024-09-16T04:02:05.569 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.559+0000 7f7cd9ffb640 1 -- 172.21.15.42:0/2425161035 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f7cc400c3f0 con 0x7f7cc40053d0 2024-09-16T04:02:05.569 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.559+0000 7f7cd9ffb640 10 monclient(hunting): start opening mon connection 2024-09-16T04:02:05.569 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.559+0000 7f7cd9ffb640 1 -- 172.21.15.42:0/2425161035 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f7cc400c5f0 con 0x7f7cc40039c0 2024-09-16T04:02:05.569 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.559+0000 7f7cd9ffb640 10 monclient(hunting): _renew_subs 2024-09-16T04:02:05.569 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.559+0000 7f7cd9ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:02:05.569 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.559+0000 7f7cdb7fe640 1 --2- 172.21.15.42:0/2425161035 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7cc40053d0 0x7f7cc400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:02:05.570 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.559+0000 7f7cdb7fe640 10 monclient(hunting): get_auth_request con 0x7f7cc40053d0 auth_method 0 2024-09-16T04:02:05.570 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.559+0000 7f7cdb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:02:05.570 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.559+0000 7f7cdb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:02:05.570 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.559+0000 7f7cdb7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:02:05.570 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.559+0000 7f7cdb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:02:05.570 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.559+0000 7f7cdb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:02:05.570 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.559+0000 7f7cdb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:02:05.570 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.559+0000 7f7cdb7fe640 10 monclient(hunting): handle_auth_done global_id 904567 payload 293 2024-09-16T04:02:05.570 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.559+0000 7f7cdb7fe640 1 -- 172.21.15.42:0/2425161035 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7cc40039c0 msgr2=0x7f7cc4003dc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:02:05.570 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.559+0000 7f7cdb7fe640 1 --2- 172.21.15.42:0/2425161035 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7cc40039c0 0x7f7cc4003dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:05.570 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.559+0000 7f7cdb7fe640 1 -- 172.21.15.42:0/2425161035 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7cc4004810 msgr2=0x7f7cc4004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:02:05.570 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.559+0000 7f7cdb7fe640 1 --2- 172.21.15.42:0/2425161035 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7cc4004810 0x7f7cc4004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:05.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.559+0000 7f7cdb7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:02:05.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.559+0000 7f7cdb7fe640 1 monclient: found mon.b 2024-09-16T04:02:05.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.559+0000 7f7cdb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:02:05.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.559+0000 7f7cdb7fe640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2024-09-16T04:02:05.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.559+0000 7f7cdb7fe640 1 -- 172.21.15.42:0/2425161035 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7cc4003530 con 0x7f7cc40053d0 2024-09-16T04:02:05.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.559+0000 7f7cdb7fe640 10 monclient: _finish_auth 0 2024-09-16T04:02:05.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.559+0000 7f7cdb7fe640 10 monclient: _check_auth_tickets 2024-09-16T04:02:05.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.559+0000 7f7cdb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:02:05.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.559+0000 7f7ce172e640 5 monclient: authenticate success, global_id 904567 2024-09-16T04:02:05.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.559+0000 7f7ce172e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:02:05.572 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.567+0000 7f7cdb7fe640 1 --2- 172.21.15.42:0/2425161035 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7cc40053d0 0x7f7cc400b9d0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f7cdc0a2190 tx=0x7f7cd401d130 comp rx=0 tx=0).ready entity=mon.1 client_cookie=941514d117c15ada server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:02:05.572 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.567+0000 7f7cd9ffb640 1 -- 172.21.15.42:0/2425161035 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cd40ab070 con 0x7f7cc40053d0 2024-09-16T04:02:05.572 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.567+0000 7f7cd9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:02:05.573 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.567+0000 7f7cd9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:02:05.573 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.567+0000 7f7cd9ffb640 10 monclient: dump: 2024-09-16T04:02:05.573 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:02:05.573 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:02:05.573 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:05.573 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:05.573 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:02:05.573 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:02:05.573 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:02:05.573 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:02:05.573 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:02:05.573 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:02:05.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.567+0000 7f7ce172e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:02:05.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.567+0000 7f7cd9ffb640 1 -- 172.21.15.42:0/2425161035 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7cd40a5cf0 con 0x7f7cc40053d0 2024-09-16T04:02:05.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.567+0000 7f7cd9ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:02:05.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.567+0000 7f7ce172e640 10 monclient: get_monmap_and_config success 2024-09-16T04:02:05.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.567+0000 7f7ce172e640 4 set_mon_vals no callback set 2024-09-16T04:02:05.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.567+0000 7f7cd9ffb640 1 -- 172.21.15.42:0/2425161035 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cd4009700 con 0x7f7cc40053d0 2024-09-16T04:02:05.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.567+0000 7f7cd9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:02:05.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.567+0000 7f7cd9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:02:05.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.567+0000 7f7cd9ffb640 10 monclient: dump: 2024-09-16T04:02:05.574 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:02:05.574 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:02:05.575 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:05.575 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:05.575 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:02:05.575 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:02:05.575 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:02:05.575 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:02:05.575 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:02:05.575 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:02:05.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.571+0000 7f7ce172e640 10 monclient: shutdown 2024-09-16T04:02:05.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.571+0000 7f7ce172e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:02:05.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.571+0000 7f7ce172e640 1 -- 172.21.15.42:0/2425161035 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7cc40053d0 msgr2=0x7f7cc400b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:02:05.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.571+0000 7f7ce172e640 1 --2- 172.21.15.42:0/2425161035 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7cc40053d0 0x7f7cc400b9d0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f7cdc0a2190 tx=0x7f7cd401d130 comp rx=0 tx=0).stop 2024-09-16T04:02:05.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.571+0000 7f7ce172e640 1 -- 172.21.15.42:0/2425161035 shutdown_connections 2024-09-16T04:02:05.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.571+0000 7f7ce172e640 1 --2- 172.21.15.42:0/2425161035 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7cc40053d0 0x7f7cc400b9d0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:05.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.571+0000 7f7ce172e640 1 --2- 172.21.15.42:0/2425161035 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7cc4004810 0x7f7cc4004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:05.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.571+0000 7f7ce172e640 1 --2- 172.21.15.42:0/2425161035 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7cc40039c0 0x7f7cc4003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:05.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.571+0000 7f7ce172e640 1 -- 172.21.15.42:0/2425161035 >> 172.21.15.42:0/2425161035 conn(0x7f7cdc0a2420 msgr2=0x7f7cdc0a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:02:05.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.571+0000 7f7ce172e640 1 -- 172.21.15.42:0/2425161035 shutdown_connections 2024-09-16T04:02:05.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.571+0000 7f7ce172e640 1 -- 172.21.15.42:0/2425161035 wait complete. 2024-09-16T04:02:05.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.571+0000 7f7ce172e640 10 monclient: build_initial_monmap 2024-09-16T04:02:05.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.571+0000 7f7ce172e640 10 monclient: monmap: 2024-09-16T04:02:05.581 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:02:05.581 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:02:05.581 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:02:05.581 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:02:05.581 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:02:05.581 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:02:05.581 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:02:05.581 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:02:05.581 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:02:05.582 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:02:05.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.571+0000 7f7ce172e640 1 librados: starting msgr at 2024-09-16T04:02:05.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.571+0000 7f7ce172e640 1 librados: starting objecter 2024-09-16T04:02:05.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.571+0000 7f7ce172e640 1 Processor -- start 2024-09-16T04:02:05.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.571+0000 7f7ce172e640 1 -- start start 2024-09-16T04:02:05.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.571+0000 7f7ce172e640 1 librados: setting wanted keys 2024-09-16T04:02:05.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.571+0000 7f7ce172e640 1 librados: calling monclient init 2024-09-16T04:02:05.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.571+0000 7f7ce172e640 10 monclient: init 2024-09-16T04:02:05.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.571+0000 7f7ce172e640 10 monclient: _reopen_session rank -1 2024-09-16T04:02:05.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.571+0000 7f7ce172e640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:02:05.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.571+0000 7f7ce172e640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7cc40039c0 0x7f7cdc1c99e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:05.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.571+0000 7f7ce172e640 10 monclient(hunting): picked mon.noname-a con 0x7f7cc40039c0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:02:05.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.571+0000 7f7ce172e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7cc4004810 0x7f7cdc133430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:05.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.571+0000 7f7ce172e640 10 monclient(hunting): picked mon.noname-b con 0x7f7cc4004810 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:02:05.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.571+0000 7f7ce172e640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7cc40053d0 0x7f7cdc133970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:05.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.571+0000 7f7ce172e640 10 monclient(hunting): picked mon.noname-c con 0x7f7cc40053d0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:02:05.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.571+0000 7f7ce172e640 10 monclient(hunting): start opening mon connection 2024-09-16T04:02:05.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.571+0000 7f7ce172e640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f7cdc09ecf0 con 0x7f7cc40039c0 2024-09-16T04:02:05.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.571+0000 7f7ce172e640 10 monclient(hunting): start opening mon connection 2024-09-16T04:02:05.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.571+0000 7f7ce172e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f7cdc09eb70 con 0x7f7cc4004810 2024-09-16T04:02:05.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.571+0000 7f7ce172e640 10 monclient(hunting): start opening mon connection 2024-09-16T04:02:05.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.571+0000 7f7ce172e640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f7cdc09ee70 con 0x7f7cc40053d0 2024-09-16T04:02:05.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.571+0000 7f7ce172e640 10 monclient(hunting): _renew_subs 2024-09-16T04:02:05.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.571+0000 7f7ce172e640 10 monclient(hunting): authenticate will time out at 2172.543087s 2024-09-16T04:02:05.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.571+0000 7f7cdb7fe640 1 -- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7cc40053d0 msgr2=0x7f7cdc133970 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T04:02:05.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.571+0000 7f7cdb7fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7cc40053d0 0x7f7cdc133970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:02:05.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.571+0000 7f7cdaffd640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7cc40039c0 0x7f7cdc1c99e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:02:05.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.571+0000 7f7cdaffd640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7cc40039c0 0x7f7cdc1c99e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:38906/0 (socket says 172.21.15.42:38906) 2024-09-16T04:02:05.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.571+0000 7f7cdaffd640 1 -- 172.21.15.42:0/1915416744 learned_addr learned my addr 172.21.15.42:0/1915416744 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:02:05.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.571+0000 7f7cdaffd640 10 monclient(hunting): get_auth_request con 0x7f7cc40039c0 auth_method 0 2024-09-16T04:02:05.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.571+0000 7f7cdaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:02:05.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.571+0000 7f7cdaffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:02:05.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.571+0000 7f7cdaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:02:05.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.571+0000 7f7cda7fc640 1 --2- 172.21.15.42:0/1915416744 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7cc4004810 0x7f7cdc133430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:02:05.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.571+0000 7f7cdaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:02:05.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.571+0000 7f7cdaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:02:05.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.571+0000 7f7cdaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:02:05.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.571+0000 7f7cda7fc640 10 monclient(hunting): get_auth_request con 0x7f7cc4004810 auth_method 0 2024-09-16T04:02:05.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.571+0000 7f7cda7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:02:05.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.571+0000 7f7cda7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:02:05.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.571+0000 7f7cda7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:02:05.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.571+0000 7f7cda7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:02:05.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.571+0000 7f7cda7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:02:05.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.571+0000 7f7cda7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:02:05.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.571+0000 7f7cdaffd640 10 monclient(hunting): handle_auth_done global_id 825468 payload 824 2024-09-16T04:02:05.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.575+0000 7f7cdaffd640 1 -- 172.21.15.42:0/1915416744 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7cc40053d0 msgr2=0x7f7cdc133970 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:02:05.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.575+0000 7f7cdaffd640 1 --2- 172.21.15.42:0/1915416744 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7cc40053d0 0x7f7cdc133970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:05.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.575+0000 7f7cdaffd640 1 -- 172.21.15.42:0/1915416744 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7cc4004810 msgr2=0x7f7cdc133430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:02:05.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.575+0000 7f7cdaffd640 1 --2- 172.21.15.42:0/1915416744 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7cc4004810 0x7f7cdc133430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:05.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.575+0000 7f7cdaffd640 10 monclient: _finish_hunting 0 2024-09-16T04:02:05.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.575+0000 7f7cdaffd640 1 monclient: found mon.noname-a 2024-09-16T04:02:05.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.575+0000 7f7cdaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:02:05.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.575+0000 7f7cdaffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:02:05.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.575+0000 7f7cdaffd640 1 -- 172.21.15.42:0/1915416744 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7cdc120b10 con 0x7f7cc40039c0 2024-09-16T04:02:05.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.575+0000 7f7cc37fe640 10 client.?.objecter ms_handle_connect 0x7f7cc40039c0 2024-09-16T04:02:05.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.575+0000 7f7cc37fe640 10 client.?.objecter resend_mon_ops 2024-09-16T04:02:05.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.575+0000 7f7cdaffd640 1 --2- 172.21.15.42:0/1915416744 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7cc40039c0 0x7f7cdc1c99e0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f7ccc000eb0 tx=0x7f7ccc013ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:02:05.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.575+0000 7f7cc37fe640 1 -- 172.21.15.42:0/1915416744 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ccc01b070 con 0x7f7cc40039c0 2024-09-16T04:02:05.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.575+0000 7f7cc37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:02:05.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.575+0000 7f7cc37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:02:05.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.575+0000 7f7cc37fe640 10 monclient: dump: 2024-09-16T04:02:05.586 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:02:05.587 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:02:05.587 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:05.587 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:05.587 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:02:05.587 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:02:05.587 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:02:05.587 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:02:05.587 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:02:05.587 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:02:05.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.575+0000 7f7cc37fe640 10 monclient: _finish_auth 0 2024-09-16T04:02:05.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.575+0000 7f7cc37fe640 10 monclient: _check_auth_tickets 2024-09-16T04:02:05.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.575+0000 7f7cc37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:02:05.588 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.575+0000 7f7cc37fe640 1 -- 172.21.15.42:0/1915416744 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ccc0158c0 con 0x7f7cc40039c0 2024-09-16T04:02:05.588 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.575+0000 7f7cc37fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:02:05.588 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.575+0000 7f7cc37fe640 1 -- 172.21.15.42:0/1915416744 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ccc015c70 con 0x7f7cc40039c0 2024-09-16T04:02:05.588 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.575+0000 7f7ce172e640 5 monclient: authenticate success, global_id 825468 2024-09-16T04:02:05.588 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.575+0000 7f7ce172e640 10 monclient: _renew_subs 2024-09-16T04:02:05.588 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.575+0000 7f7ce172e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:02:05.588 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.575+0000 7f7ce172e640 1 -- 172.21.15.42:0/1915416744 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7cdc11f660 con 0x7f7cc40039c0 2024-09-16T04:02:05.588 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.575+0000 7f7cd97fa640 4 set_mon_vals no callback set 2024-09-16T04:02:05.588 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.575+0000 7f7cc37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:02:05.588 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.575+0000 7f7cc37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:02:05.588 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.575+0000 7f7cc37fe640 10 monclient: dump: 2024-09-16T04:02:05.588 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:02:05.588 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:02:05.589 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:05.589 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:05.589 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:02:05.589 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:02:05.589 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:02:05.589 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:02:05.589 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:02:05.589 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:02:05.589 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.575+0000 7f7ce172e640 10 client.825468.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:02:05.589 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.575+0000 7f7ce172e640 10 monclient: _renew_subs 2024-09-16T04:02:05.589 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.575+0000 7f7ce172e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:02:05.589 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.575+0000 7f7ce172e640 1 -- 172.21.15.42:0/1915416744 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7cdc0a4b00 con 0x7f7cc40039c0 2024-09-16T04:02:05.589 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.575+0000 7f7ce172e640 1 librados: init done 2024-09-16T04:02:05.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.575+0000 7f7ce172e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:02:05.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.575+0000 7f7ce172e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:02:05.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.575+0000 7f7ce172e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:02:05.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.575+0000 7f7ce172e640 1 -- 172.21.15.42:0/1915416744 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7cdc073a30 con 0x7f7cc40039c0 2024-09-16T04:02:05.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.579+0000 7f7cc37fe640 1 -- 172.21.15.42:0/1915416744 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f7ccc01e070 con 0x7f7cc40039c0 2024-09-16T04:02:05.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.579+0000 7f7cc37fe640 1 --2- 172.21.15.42:0/1915416744 >> v2:172.21.15.154:6812/254845342 conn(0x7f7cc8037fc0 0x7f7cc803a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:05.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.579+0000 7f7cc37fe640 1 -- 172.21.15.42:0/1915416744 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(602..602 src has 451..602) ==== 6252+0+0 (secure 0 0 0) 0x7f7ccc049f70 con 0x7f7cc40039c0 2024-09-16T04:02:05.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.579+0000 7f7cc37fe640 10 client.825468.objecter ms_dispatch 0x7f7cdc000ba0 osd_map(602..602 src has 451..602) 2024-09-16T04:02:05.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.579+0000 7f7cc37fe640 3 client.825468.objecter handle_osd_map got epochs [602,602] > 0 2024-09-16T04:02:05.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.579+0000 7f7cc37fe640 3 client.825468.objecter handle_osd_map decoding full epoch 602 2024-09-16T04:02:05.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.579+0000 7f7cc37fe640 20 client.825468.objecter dump_active .. 0 homeless 2024-09-16T04:02:05.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.579+0000 7f7cda7fc640 1 --2- 172.21.15.42:0/1915416744 >> v2:172.21.15.154:6812/254845342 conn(0x7f7cc8037fc0 0x7f7cc803a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:02:05.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.579+0000 7f7cda7fc640 10 monclient: get_auth_request con 0x7f7cc8037fc0 auth_method 0 2024-09-16T04:02:05.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.579+0000 7f7cc37fe640 10 client.825468.objecter ms_handle_connect 0x7f7cc8037fc0 2024-09-16T04:02:05.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.579+0000 7f7cda7fc640 1 --2- 172.21.15.42:0/1915416744 >> v2:172.21.15.154:6812/254845342 conn(0x7f7cc8037fc0 0x7f7cc803a480 secure :-1 s=READY pgs=3776 cs=0 l=1 rev1=1 crypto rx=0x7f7cdc1342d0 tx=0x7f7cd0000bd0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:02:05.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.583+0000 7f7cc37fe640 1 -- 172.21.15.42:0/1915416744 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f7ccc015e10 con 0x7f7cc40039c0 2024-09-16T04:02:05.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.583+0000 7f7cc37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:02:05.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.583+0000 7f7cc37fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:02:05.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.819+0000 7f7ce172e640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:02:05.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.819+0000 7f7ce172e640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:02:05.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.819+0000 7f7ce172e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:02:05.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.819+0000 7f7ce172e640 1 -- 172.21.15.42:0/1915416744 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f7cdc11f3e0 con 0x7f7cc40039c0 2024-09-16T04:02:05.823 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.819+0000 7f7cc37fe640 1 -- 172.21.15.42:0/1915416744 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1555 (secure 0 0 0) 0x7f7ccc016980 con 0x7f7cc40039c0 2024-09-16T04:02:05.823 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.819+0000 7f7cc37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:02:05.823 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.819+0000 7f7cc37fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:02:05.823 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":182,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:02:05.825 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.819+0000 7f7ce172e640 10 librados: watch_flush enter 2024-09-16T04:02:05.826 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.819+0000 7f7ce172e640 10 librados: watch_flush exit 2024-09-16T04:02:05.826 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.819+0000 7f7ce172e640 20 client.825468.objecter shutdown clearing up homeless session... 2024-09-16T04:02:05.826 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.819+0000 7f7ce172e640 10 client.825468.objecter successfully canceled tick 2024-09-16T04:02:05.826 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.819+0000 7f7ce172e640 1 -- 172.21.15.42:0/1915416744 >> v2:172.21.15.154:6812/254845342 conn(0x7f7cc8037fc0 msgr2=0x7f7cc803a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:02:05.826 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.819+0000 7f7ce172e640 1 --2- 172.21.15.42:0/1915416744 >> v2:172.21.15.154:6812/254845342 conn(0x7f7cc8037fc0 0x7f7cc803a480 secure :-1 s=READY pgs=3776 cs=0 l=1 rev1=1 crypto rx=0x7f7cdc1342d0 tx=0x7f7cd0000bd0 comp rx=0 tx=0).stop 2024-09-16T04:02:05.826 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.819+0000 7f7ce172e640 10 monclient: shutdown 2024-09-16T04:02:05.826 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.819+0000 7f7ce172e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:02:05.826 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.819+0000 7f7ce172e640 1 -- 172.21.15.42:0/1915416744 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7cc40039c0 msgr2=0x7f7cdc1c99e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:02:05.826 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.819+0000 7f7ce172e640 1 --2- 172.21.15.42:0/1915416744 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7cc40039c0 0x7f7cdc1c99e0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f7ccc000eb0 tx=0x7f7ccc013ae0 comp rx=0 tx=0).stop 2024-09-16T04:02:05.826 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.819+0000 7f7ce172e640 1 -- 172.21.15.42:0/1915416744 shutdown_connections 2024-09-16T04:02:05.826 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.819+0000 7f7ce172e640 1 --2- 172.21.15.42:0/1915416744 >> v2:172.21.15.154:6812/254845342 conn(0x7f7cc8037fc0 0x7f7cc803a480 unknown :-1 s=CLOSED pgs=3776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:05.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.823+0000 7f7ce172e640 1 --2- 172.21.15.42:0/1915416744 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7cc40053d0 0x7f7cdc133970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:05.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.823+0000 7f7ce172e640 1 --2- 172.21.15.42:0/1915416744 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7cc4004810 0x7f7cdc133430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:05.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.823+0000 7f7ce172e640 1 --2- 172.21.15.42:0/1915416744 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7cc40039c0 0x7f7cdc1c99e0 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:05.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.823+0000 7f7ce172e640 1 -- 172.21.15.42:0/1915416744 >> 172.21.15.42:0/1915416744 conn(0x7f7cdc0a2420 msgr2=0x7f7cdc0a3720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:02:05.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.823+0000 7f7ce172e640 1 -- 172.21.15.42:0/1915416744 shutdown_connections 2024-09-16T04:02:05.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.823+0000 7f7ce172e640 1 -- 172.21.15.42:0/1915416744 wait complete. 2024-09-16T04:02:05.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.823+0000 7f7ce172e640 1 librados: shutdown 2024-09-16T04:02:05.843 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T04:02:05.843 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-16T04:02:05.843 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T04:02:05.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.967+0000 7f97e5c13640 10 monclient: get_monmap_and_config 2024-09-16T04:02:05.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.967+0000 7f97e5c13640 10 monclient: build_initial_monmap 2024-09-16T04:02:05.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.967+0000 7f97e5c13640 10 monclient: monmap: 2024-09-16T04:02:05.971 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:02:05.971 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:02:05.971 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:02:05.969017+0000 2024-09-16T04:02:05.971 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:02:05.969017+0000 2024-09-16T04:02:05.971 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:02:05.971 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:02:05.971 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:02:05.971 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:02:05.971 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:02:05.971 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:02:05.971 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:02:05.971 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:02:05.974 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.967+0000 7f97e5c13640 1 Processor -- start 2024-09-16T04:02:05.974 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.967+0000 7f97e5c13640 1 -- start start 2024-09-16T04:02:05.974 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.967+0000 7f97e5c13640 10 monclient: init 2024-09-16T04:02:05.974 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.967+0000 7f97e5c13640 10 monclient: _reopen_session rank -1 2024-09-16T04:02:05.974 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.967+0000 7f97e5c13640 10 monclient: _add_conns ranks=[1,3,4,0,2] 2024-09-16T04:02:05.974 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.967+0000 7f97e5c13640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f97e00a67e0 addr v1:172.21.15.154:6789/0 2024-09-16T04:02:05.974 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.967+0000 7f97e5c13640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f97e00a72d0 0x7f97e01408b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:05.974 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.967+0000 7f97e5c13640 10 monclient(hunting): picked mon.noname-a con 0x7f97e00a72d0 addr v2:172.21.15.42:3300/0 2024-09-16T04:02:05.974 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.967+0000 7f97e5c13640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f97e0140df0 0x7f97e01431e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:05.975 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.967+0000 7f97e5c13640 10 monclient(hunting): picked mon.noname-c con 0x7f97e0140df0 addr v2:172.21.15.154:3300/0 2024-09-16T04:02:05.975 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.967+0000 7f97e5c13640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f97e006f3b0 con 0x7f97e00a67e0 2024-09-16T04:02:05.975 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.967+0000 7f97e5c13640 10 monclient(hunting): start opening mon connection 2024-09-16T04:02:05.975 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.967+0000 7f97e5c13640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f97e006e030 con 0x7f97e00a72d0 2024-09-16T04:02:05.975 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.967+0000 7f97e5c13640 10 monclient(hunting): start opening mon connection 2024-09-16T04:02:05.975 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.967+0000 7f97e5c13640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f97e006f0d0 con 0x7f97e0140df0 2024-09-16T04:02:05.975 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.967+0000 7f97e5c13640 10 monclient(hunting): _renew_subs 2024-09-16T04:02:05.975 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.967+0000 7f97e5c13640 10 monclient(hunting): authenticate will time out at 2172.938913s 2024-09-16T04:02:05.975 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.967+0000 7f97deffd640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f97e00a72d0 0x7f97e01408b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:02:05.975 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.967+0000 7f97deffd640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f97e00a72d0 0x7f97e01408b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:38908/0 (socket says 172.21.15.42:38908) 2024-09-16T04:02:05.975 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.967+0000 7f97deffd640 1 -- 172.21.15.42:0/460718500 learned_addr learned my addr 172.21.15.42:0/460718500 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:02:05.975 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.967+0000 7f97deffd640 10 monclient(hunting): get_auth_request con 0x7f97e00a72d0 auth_method 0 2024-09-16T04:02:05.976 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.967+0000 7f97deffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:02:05.976 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.967+0000 7f97deffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:02:05.976 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.967+0000 7f97deffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:02:05.976 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.967+0000 7f97deffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:02:05.976 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.967+0000 7f97deffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:02:05.976 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.967+0000 7f97deffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:02:05.976 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.967+0000 7f97dffff640 1 --2- 172.21.15.42:0/460718500 >> v2:172.21.15.154:3300/0 conn(0x7f97e0140df0 0x7f97e01431e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:02:05.976 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.967+0000 7f97dffff640 10 monclient(hunting): get_auth_request con 0x7f97e0140df0 auth_method 0 2024-09-16T04:02:05.976 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.967+0000 7f97dffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:02:05.976 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.967+0000 7f97dffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:02:05.976 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.967+0000 7f97dffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:02:05.976 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.967+0000 7f97deffd640 10 monclient(hunting): handle_auth_done global_id 825471 payload 274 2024-09-16T04:02:05.976 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.967+0000 7f97deffd640 1 -- 172.21.15.42:0/460718500 >> v2:172.21.15.154:3300/0 conn(0x7f97e0140df0 msgr2=0x7f97e01431e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:02:05.976 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.967+0000 7f97deffd640 1 --2- 172.21.15.42:0/460718500 >> v2:172.21.15.154:3300/0 conn(0x7f97e0140df0 0x7f97e01431e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:05.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.967+0000 7f97deffd640 1 -- 172.21.15.42:0/460718500 >> v1:172.21.15.154:6789/0 conn(0x7f97e00a67e0 legacy=0x7f97e00a6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:02:05.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.967+0000 7f97deffd640 10 monclient: _finish_hunting 0 2024-09-16T04:02:05.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.967+0000 7f97deffd640 1 monclient: found mon.noname-a 2024-09-16T04:02:05.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.967+0000 7f97deffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:02:05.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.967+0000 7f97deffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:02:05.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.967+0000 7f97deffd640 1 -- 172.21.15.42:0/460718500 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97e0073110 con 0x7f97e00a72d0 2024-09-16T04:02:05.983 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.971+0000 7f97deffd640 1 --2- 172.21.15.42:0/460718500 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f97e00a72d0 0x7f97e01408b0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f97e00a5780 tx=0x7f97d401c250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad8cd72cbcd4b792 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:02:05.983 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.971+0000 7f97de7fc640 1 -- 172.21.15.42:0/460718500 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97d40b0070 con 0x7f97e00a72d0 2024-09-16T04:02:05.983 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.971+0000 7f97de7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:02:05.983 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.971+0000 7f97de7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:02:05.983 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.971+0000 7f97de7fc640 10 monclient: dump: 2024-09-16T04:02:05.983 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:02:05.983 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:02:05.983 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:05.983 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:05.983 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:02:05.984 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:02:05.984 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:02:05.984 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:02:05.984 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:02:05.984 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:02:05.984 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.971+0000 7f97de7fc640 10 monclient: _finish_auth 0 2024-09-16T04:02:05.984 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.971+0000 7f97de7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:02:05.984 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.971+0000 7f97de7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:02:05.984 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.971+0000 7f97e5c13640 5 monclient: authenticate success, global_id 825471 2024-09-16T04:02:05.984 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.971+0000 7f97de7fc640 1 -- 172.21.15.42:0/460718500 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f97d40ab600 con 0x7f97e00a72d0 2024-09-16T04:02:05.984 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.971+0000 7f97e5c13640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:02:05.984 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.971+0000 7f97de7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:02:05.984 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.971+0000 7f97e5c13640 10 monclient: get_monmap_and_config success 2024-09-16T04:02:05.985 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.971+0000 7f97e5c13640 4 set_mon_vals no callback set 2024-09-16T04:02:05.985 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.971+0000 7f97de7fc640 1 -- 172.21.15.42:0/460718500 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97d40ab9d0 con 0x7f97e00a72d0 2024-09-16T04:02:05.985 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.971+0000 7f97de7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:02:05.985 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.971+0000 7f97de7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:02:05.985 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.971+0000 7f97de7fc640 10 monclient: dump: 2024-09-16T04:02:05.985 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:02:05.985 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:02:05.985 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:05.985 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:05.985 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:02:05.985 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:02:05.985 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:02:05.985 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:02:05.985 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:02:05.986 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:02:05.986 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.971+0000 7f97e5c13640 10 monclient: shutdown 2024-09-16T04:02:05.986 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.971+0000 7f97e5c13640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:02:05.986 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.971+0000 7f97e5c13640 1 -- 172.21.15.42:0/460718500 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f97e00a72d0 msgr2=0x7f97e01408b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:02:05.986 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.971+0000 7f97e5c13640 1 --2- 172.21.15.42:0/460718500 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f97e00a72d0 0x7f97e01408b0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f97e00a5780 tx=0x7f97d401c250 comp rx=0 tx=0).stop 2024-09-16T04:02:05.986 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.971+0000 7f97e5c13640 1 -- 172.21.15.42:0/460718500 shutdown_connections 2024-09-16T04:02:05.986 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.971+0000 7f97e5c13640 1 --2- 172.21.15.42:0/460718500 >> v2:172.21.15.154:3300/0 conn(0x7f97e0140df0 0x7f97e01431e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:05.986 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.971+0000 7f97e5c13640 1 --2- 172.21.15.42:0/460718500 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f97e00a72d0 0x7f97e01408b0 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:05.986 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.971+0000 7f97e5c13640 1 -- 172.21.15.42:0/460718500 >> 172.21.15.42:0/460718500 conn(0x7f97e00a7ed0 msgr2=0x7f97e00a43d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:02:05.986 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.971+0000 7f97e5c13640 1 -- 172.21.15.42:0/460718500 shutdown_connections 2024-09-16T04:02:05.986 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.971+0000 7f97e5c13640 1 -- 172.21.15.42:0/460718500 wait complete. 2024-09-16T04:02:05.986 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.971+0000 7f97e5c13640 10 monclient: build_initial_monmap 2024-09-16T04:02:05.986 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.971+0000 7f97e5c13640 10 monclient: monmap: 2024-09-16T04:02:05.987 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:02:05.987 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:02:05.987 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:02:05.987 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:02:05.987 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:02:05.987 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:02:05.987 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:02:05.987 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:02:05.987 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:02:05.987 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:02:05.987 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.971+0000 7f97e5c13640 1 librados: starting msgr at 2024-09-16T04:02:05.987 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.971+0000 7f97e5c13640 1 librados: starting objecter 2024-09-16T04:02:05.987 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.971+0000 7f97e5c13640 1 Processor -- start 2024-09-16T04:02:05.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.971+0000 7f97e5c13640 1 -- start start 2024-09-16T04:02:05.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.971+0000 7f97e5c13640 1 librados: setting wanted keys 2024-09-16T04:02:05.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.971+0000 7f97e5c13640 1 librados: calling monclient init 2024-09-16T04:02:05.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.971+0000 7f97e5c13640 10 monclient: init 2024-09-16T04:02:05.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.975+0000 7f97e5c13640 10 monclient: _reopen_session rank -1 2024-09-16T04:02:05.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.975+0000 7f97e5c13640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:02:05.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.975+0000 7f97e5c13640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f97e00a67e0 0x7f97e01d3230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:05.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.975+0000 7f97e5c13640 10 monclient(hunting): picked mon.noname-a con 0x7f97e00a67e0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:02:05.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.975+0000 7f97e5c13640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f97e00a72d0 0x7f97e013bc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:05.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.975+0000 7f97e5c13640 10 monclient(hunting): picked mon.noname-b con 0x7f97e00a72d0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:02:05.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.975+0000 7f97e5c13640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f97e0140df0 0x7f97e009db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:05.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.975+0000 7f97e5c13640 10 monclient(hunting): picked mon.noname-c con 0x7f97e0140df0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:02:05.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.975+0000 7f97e5c13640 10 monclient(hunting): start opening mon connection 2024-09-16T04:02:05.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.975+0000 7f97e5c13640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f97e01460c0 con 0x7f97e00a67e0 2024-09-16T04:02:05.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.975+0000 7f97e5c13640 10 monclient(hunting): start opening mon connection 2024-09-16T04:02:05.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.975+0000 7f97e5c13640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f97e0145f40 con 0x7f97e00a72d0 2024-09-16T04:02:05.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.975+0000 7f97e5c13640 10 monclient(hunting): start opening mon connection 2024-09-16T04:02:05.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.975+0000 7f97e5c13640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f97e0146240 con 0x7f97e0140df0 2024-09-16T04:02:05.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.975+0000 7f97e5c13640 10 monclient(hunting): _renew_subs 2024-09-16T04:02:05.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.975+0000 7f97e5c13640 10 monclient(hunting): authenticate will time out at 2172.944273s 2024-09-16T04:02:05.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.975+0000 7f97dffff640 1 -- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f97e0140df0 msgr2=0x7f97e009db90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T04:02:05.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.975+0000 7f97dffff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f97e0140df0 0x7f97e009db90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:02:05.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.975+0000 7f97df7fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f97e00a67e0 0x7f97e01d3230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:02:05.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.975+0000 7f97df7fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f97e00a67e0 0x7f97e01d3230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:38924/0 (socket says 172.21.15.42:38924) 2024-09-16T04:02:05.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.975+0000 7f97df7fe640 1 -- 172.21.15.42:0/2362404850 learned_addr learned my addr 172.21.15.42:0/2362404850 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:02:05.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.975+0000 7f97df7fe640 10 monclient(hunting): get_auth_request con 0x7f97e00a67e0 auth_method 0 2024-09-16T04:02:05.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.975+0000 7f97df7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:02:05.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.975+0000 7f97df7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:02:05.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.975+0000 7f97df7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:02:05.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.975+0000 7f97df7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:02:05.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.975+0000 7f97df7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:02:05.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.975+0000 7f97df7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:02:05.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.975+0000 7f97deffd640 1 --2- 172.21.15.42:0/2362404850 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f97e00a72d0 0x7f97e013bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:02:05.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.975+0000 7f97deffd640 10 monclient(hunting): get_auth_request con 0x7f97e00a72d0 auth_method 0 2024-09-16T04:02:05.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.975+0000 7f97deffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:02:05.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.975+0000 7f97deffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:02:05.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.975+0000 7f97deffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:02:05.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.975+0000 7f97df7fe640 10 monclient(hunting): handle_auth_done global_id 825474 payload 824 2024-09-16T04:02:05.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.975+0000 7f97df7fe640 1 -- 172.21.15.42:0/2362404850 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f97e0140df0 msgr2=0x7f97e009db90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:02:05.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.975+0000 7f97df7fe640 1 --2- 172.21.15.42:0/2362404850 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f97e0140df0 0x7f97e009db90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:05.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.975+0000 7f97df7fe640 1 -- 172.21.15.42:0/2362404850 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f97e00a72d0 msgr2=0x7f97e013bc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:02:05.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.975+0000 7f97df7fe640 1 --2- 172.21.15.42:0/2362404850 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f97e00a72d0 0x7f97e013bc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:05.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.975+0000 7f97df7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:02:05.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.975+0000 7f97df7fe640 1 monclient: found mon.noname-a 2024-09-16T04:02:05.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.975+0000 7f97df7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:02:05.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.975+0000 7f97df7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:02:05.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.975+0000 7f97df7fe640 1 -- 172.21.15.42:0/2362404850 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97e0070b00 con 0x7f97e00a67e0 2024-09-16T04:02:05.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.975+0000 7f97deffd640 1 --2- 172.21.15.42:0/2362404850 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f97e00a72d0 0x7f97e013bc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T04:02:05.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.975+0000 7f97df7fe640 1 --2- 172.21.15.42:0/2362404850 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f97e00a67e0 0x7f97e01d3230 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f97d000c480 tx=0x7f97d000c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:02:05.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.975+0000 7f97c7fff640 10 client.?.objecter ms_handle_connect 0x7f97e00a67e0 2024-09-16T04:02:05.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.975+0000 7f97c7fff640 10 client.?.objecter resend_mon_ops 2024-09-16T04:02:05.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.975+0000 7f97c7fff640 1 -- 172.21.15.42:0/2362404850 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97d0012830 con 0x7f97e00a67e0 2024-09-16T04:02:05.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.975+0000 7f97c7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:02:05.992 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.975+0000 7f97c7fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:02:05.992 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.975+0000 7f97c7fff640 10 monclient: dump: 2024-09-16T04:02:05.992 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:02:05.992 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:02:05.992 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:05.992 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:05.992 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:02:05.992 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:02:05.992 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:02:05.992 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:02:05.992 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:02:05.992 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:02:05.992 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.975+0000 7f97c7fff640 10 monclient: _finish_auth 0 2024-09-16T04:02:05.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.975+0000 7f97c7fff640 10 monclient: _check_auth_tickets 2024-09-16T04:02:05.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.975+0000 7f97c7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:02:05.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.975+0000 7f97c7fff640 1 -- 172.21.15.42:0/2362404850 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f97d0017070 con 0x7f97e00a67e0 2024-09-16T04:02:05.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.975+0000 7f97c7fff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:02:05.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.975+0000 7f97c7fff640 1 -- 172.21.15.42:0/2362404850 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97d0012c30 con 0x7f97e00a67e0 2024-09-16T04:02:05.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.975+0000 7f97e5c13640 5 monclient: authenticate success, global_id 825474 2024-09-16T04:02:05.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.975+0000 7f97e5c13640 10 monclient: _renew_subs 2024-09-16T04:02:05.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.975+0000 7f97e5c13640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:02:05.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.975+0000 7f97e5c13640 1 -- 172.21.15.42:0/2362404850 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97e0073970 con 0x7f97e00a67e0 2024-09-16T04:02:05.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.975+0000 7f97c7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:02:05.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.975+0000 7f97ddffb640 4 set_mon_vals no callback set 2024-09-16T04:02:05.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.975+0000 7f97c7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:02:05.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.975+0000 7f97c7fff640 10 monclient: dump: 2024-09-16T04:02:05.994 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:02:05.994 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:02:05.994 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:05.994 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:05.994 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:02:05.994 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:02:05.994 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:02:05.994 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:02:05.994 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:02:05.994 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:02:05.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.975+0000 7f97e5c13640 10 client.825474.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:02:05.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.975+0000 7f97e5c13640 10 monclient: _renew_subs 2024-09-16T04:02:05.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.975+0000 7f97e5c13640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:02:05.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.975+0000 7f97e5c13640 1 -- 172.21.15.42:0/2362404850 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97e00a1560 con 0x7f97e00a67e0 2024-09-16T04:02:05.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.975+0000 7f97e5c13640 1 librados: init done 2024-09-16T04:02:05.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.975+0000 7f97e5c13640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:02:05.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.975+0000 7f97e5c13640 1 --2- 172.21.15.42:0/2362404850 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f97e013c180 0x7f97e002c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:05.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.975+0000 7f97e5c13640 10 monclient(hunting): start opening mon connection 2024-09-16T04:02:05.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.975+0000 7f97e5c13640 1 -- 172.21.15.42:0/2362404850 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f97e00a1360 con 0x7f97e013c180 2024-09-16T04:02:05.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.975+0000 7f97deffd640 1 --2- 172.21.15.42:0/2362404850 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f97e013c180 0x7f97e002c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:02:05.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.975+0000 7f97c7fff640 1 -- 172.21.15.42:0/2362404850 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f97d001c460 con 0x7f97e00a67e0 2024-09-16T04:02:05.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.975+0000 7f97deffd640 10 monclient: get_auth_request con 0x7f97e013c180 auth_method 0 2024-09-16T04:02:05.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.975+0000 7f97deffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:02:05.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.975+0000 7f97deffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:02:05.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.975+0000 7f97deffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:02:05.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.975+0000 7f97deffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:02:05.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.975+0000 7f97deffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:02:05.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.975+0000 7f97deffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:02:05.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.975+0000 7f97c7fff640 1 --2- 172.21.15.42:0/2362404850 >> v2:172.21.15.154:6812/254845342 conn(0x7f97cc0377c0 0x7f97cc039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:05.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.975+0000 7f97c7fff640 1 -- 172.21.15.42:0/2362404850 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(602..602 src has 451..602) ==== 6252+0+0 (secure 0 0 0) 0x7f97d0048f70 con 0x7f97e00a67e0 2024-09-16T04:02:05.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.975+0000 7f97c7fff640 10 client.825474.objecter ms_dispatch 0x7f97e0000ba0 osd_map(602..602 src has 451..602) 2024-09-16T04:02:05.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.975+0000 7f97c7fff640 3 client.825474.objecter handle_osd_map got epochs [602,602] > 0 2024-09-16T04:02:05.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.975+0000 7f97c7fff640 3 client.825474.objecter handle_osd_map decoding full epoch 602 2024-09-16T04:02:05.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.975+0000 7f97deffd640 10 monclient(hunting): handle_auth_done global_id 825477 payload 824 2024-09-16T04:02:05.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.975+0000 7f97deffd640 1 -- 172.21.15.42:0/2362404850 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f97e00a6bc0 con 0x7f97e013c180 2024-09-16T04:02:05.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.975+0000 7f97c7fff640 20 client.825474.objecter dump_active .. 0 homeless 2024-09-16T04:02:05.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.975+0000 7f97c7fff640 10 client.825474.objecter ms_handle_connect 0x7f97e013c180 2024-09-16T04:02:05.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.975+0000 7f97c7fff640 10 client.825474.objecter resend_mon_ops 2024-09-16T04:02:05.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.975+0000 7f97deffd640 1 --2- 172.21.15.42:0/2362404850 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f97e013c180 0x7f97e002c650 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f97e00765c0 tx=0x7f97d401aa90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:02:05.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.975+0000 7f97dffff640 1 --2- 172.21.15.42:0/2362404850 >> v2:172.21.15.154:6812/254845342 conn(0x7f97cc0377c0 0x7f97cc039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:02:05.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.975+0000 7f97c7fff640 1 -- 172.21.15.42:0/2362404850 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97d40ba030 con 0x7f97e013c180 2024-09-16T04:02:05.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.975+0000 7f97c7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:02:05.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.975+0000 7f97c7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:02:05.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.975+0000 7f97c7fff640 10 monclient: dump: 2024-09-16T04:02:05.997 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:02:05.997 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:02:05.997 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:05.997 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:05.997 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:02:05.997 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:02:05.997 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:02:05.997 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:02:05.997 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:02:05.998 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:02:05.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.975+0000 7f97dffff640 10 monclient: get_auth_request con 0x7f97cc0377c0 auth_method 0 2024-09-16T04:02:05.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.975+0000 7f97c7fff640 1 -- 172.21.15.42:0/2362404850 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f97d4009490 con 0x7f97e013c180 2024-09-16T04:02:05.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.975+0000 7f97c7fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:02:05.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.975+0000 7f97c7fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:02:05.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.975+0000 7f97c7fff640 1 -- 172.21.15.42:0/2362404850 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f97e013c180 msgr2=0x7f97e002c650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:02:05.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.975+0000 7f97c7fff640 1 --2- 172.21.15.42:0/2362404850 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f97e013c180 0x7f97e002c650 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f97e00765c0 tx=0x7f97d401aa90 comp rx=0 tx=0).stop 2024-09-16T04:02:05.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.975+0000 7f97c7fff640 1 -- 172.21.15.42:0/2362404850 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f97e013c180 msgr2=0x7f97e002c650 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:02:05.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.975+0000 7f97c7fff640 1 --2- 172.21.15.42:0/2362404850 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f97e013c180 0x7f97e002c650 secure :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f97e00765c0 tx=0x7f97d401aa90 comp rx=0 tx=0).stop 2024-09-16T04:02:05.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.975+0000 7f97dffff640 1 --2- 172.21.15.42:0/2362404850 >> v2:172.21.15.154:6812/254845342 conn(0x7f97cc0377c0 0x7f97cc039c80 secure :-1 s=READY pgs=3777 cs=0 l=1 rev1=1 crypto rx=0x7f97d8009660 tx=0x7f97d80098b0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:02:05.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.975+0000 7f97c7fff640 10 client.825474.objecter ms_handle_connect 0x7f97cc0377c0 2024-09-16T04:02:06.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.983+0000 7f97e5c13640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:02:06.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.983+0000 7f97e5c13640 1 --2- 172.21.15.42:0/2362404850 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f97e002cb90 0x7f97e002cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:06.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.983+0000 7f97e5c13640 10 monclient(hunting): start opening mon connection 2024-09-16T04:02:06.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.983+0000 7f97e5c13640 1 -- 172.21.15.42:0/2362404850 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f97e013f2d0 con 0x7f97e002cb90 2024-09-16T04:02:06.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.983+0000 7f97deffd640 1 --2- 172.21.15.42:0/2362404850 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f97e002cb90 0x7f97e002cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:02:06.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.983+0000 7f97deffd640 10 monclient: get_auth_request con 0x7f97e002cb90 auth_method 0 2024-09-16T04:02:06.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.983+0000 7f97deffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:02:06.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.983+0000 7f97deffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:02:06.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.983+0000 7f97deffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:02:06.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.983+0000 7f97deffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:02:06.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.983+0000 7f97deffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:02:06.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.983+0000 7f97deffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:02:06.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.983+0000 7f97deffd640 10 monclient(hunting): handle_auth_done global_id 825480 payload 824 2024-09-16T04:02:06.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.983+0000 7f97deffd640 1 -- 172.21.15.42:0/2362404850 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f97e00a0110 con 0x7f97e002cb90 2024-09-16T04:02:06.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.983+0000 7f97deffd640 1 --2- 172.21.15.42:0/2362404850 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f97e002cb90 0x7f97e002cfa0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f97d4019e80 tx=0x7f97d4018ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:02:06.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.983+0000 7f97c7fff640 10 client.825474.objecter ms_handle_connect 0x7f97e002cb90 2024-09-16T04:02:06.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.983+0000 7f97c7fff640 10 client.825474.objecter resend_mon_ops 2024-09-16T04:02:06.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.983+0000 7f97c7fff640 1 -- 172.21.15.42:0/2362404850 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97d40ba070 con 0x7f97e002cb90 2024-09-16T04:02:06.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.983+0000 7f97c7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:02:06.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.983+0000 7f97c7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:02:06.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.983+0000 7f97c7fff640 10 monclient: dump: 2024-09-16T04:02:06.002 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:02:06.002 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:02:06.002 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:06.002 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:06.002 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:02:06.002 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:02:06.002 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:02:06.002 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:02:06.002 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:02:06.003 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:02:06.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.983+0000 7f97c7fff640 1 -- 172.21.15.42:0/2362404850 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4704 (secure 0 0 0) 0x7f97d4003620 con 0x7f97e002cb90 2024-09-16T04:02:06.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.983+0000 7f97c7fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:02:06.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.983+0000 7f97c7fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:02:06.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.983+0000 7f97c7fff640 1 -- 172.21.15.42:0/2362404850 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f97e002cb90 msgr2=0x7f97e002cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:02:06.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.983+0000 7f97c7fff640 1 --2- 172.21.15.42:0/2362404850 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f97e002cb90 0x7f97e002cfa0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f97d4019e80 tx=0x7f97d4018ff0 comp rx=0 tx=0).stop 2024-09-16T04:02:06.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.983+0000 7f97c7fff640 1 -- 172.21.15.42:0/2362404850 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f97e002cb90 msgr2=0x7f97e002cfa0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:02:06.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.983+0000 7f97c7fff640 1 --2- 172.21.15.42:0/2362404850 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f97e002cb90 0x7f97e002cfa0 secure :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f97d4019e80 tx=0x7f97d4018ff0 comp rx=0 tx=0).stop 2024-09-16T04:02:06.003 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T04:02:06.003 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:02:06.004 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:02:06.004 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "leader", 2024-09-16T04:02:06.004 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 182, 2024-09-16T04:02:06.004 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 615838, 2024-09-16T04:02:06.004 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T04:02:06.004 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T04:02:06.004 INFO:teuthology.orchestra.run.smithi042.stdout: 1 2024-09-16T04:02:06.004 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:02:06.004 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 0, 2024-09-16T04:02:06.004 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:02:06.004 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T04:02:06.004 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T04:02:06.004 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:02:06.005 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:02:06.005 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:02:06.005 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:02:06.005 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:02:06.005 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:02:06.005 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:02:06.005 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:02:06.005 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:02:06.005 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:02:06.005 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:02:06.005 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:02:06.005 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:02:06.005 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T04:02:06.006 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:02:06.006 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:02:06.006 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:02:06.006 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:02:06.006 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:02:06.006 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:02:06.006 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:02:06.006 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:02:06.006 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:02:06.006 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:02:06.006 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:02:06.006 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:02:06.006 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:02:06.007 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T04:02:06.007 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T04:02:06.007 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T04:02:06.007 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T04:02:06.007 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T04:02:06.007 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T04:02:06.007 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:02:06.007 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:02:06.007 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T04:02:06.007 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T04:02:06.007 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T04:02:06.007 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T04:02:06.007 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T04:02:06.008 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T04:02:06.008 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T04:02:06.008 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:02:06.008 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T04:02:06.008 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:02:06.008 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:02:06.008 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:02:06.008 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:02:06.008 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:02:06.008 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:02:06.008 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:02:06.008 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:02:06.008 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:02:06.009 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:02:06.009 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:02:06.009 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:02:06.009 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T04:02:06.009 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:02:06.009 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T04:02:06.009 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:02:06.009 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:02:06.009 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:02:06.009 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:02:06.009 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:02:06.009 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:02:06.009 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:02:06.009 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T04:02:06.010 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:02:06.010 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:02:06.010 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:02:06.010 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:02:06.010 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T04:02:06.010 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:02:06.010 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:02:06.010 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:02:06.010 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:02:06.010 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T04:02:06.010 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T04:02:06.010 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:02:06.010 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:02:06.011 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:02:06.011 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:02:06.011 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:02:06.011 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:02:06.011 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:02:06.011 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:02:06.011 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:02:06.011 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:02:06.011 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:02:06.011 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T04:02:06.011 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:02:06.011 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:02:06.011 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:02:06.011 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:02:06.012 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T04:02:06.012 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:02:06.012 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:02:06.012 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:02:06.012 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:02:06.012 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T04:02:06.012 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T04:02:06.012 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:02:06.012 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:02:06.012 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:02:06.012 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:02:06.012 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:02:06.012 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:02:06.013 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:02:06.013 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:02:06.013 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:02:06.013 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:02:06.013 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:02:06.013 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T04:02:06.013 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:02:06.013 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:02:06.013 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:02:06.013 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:02:06.013 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T04:02:06.013 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:02:06.013 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:02:06.013 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:02:06.014 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:02:06.014 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T04:02:06.014 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T04:02:06.014 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:02:06.014 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:02:06.014 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:02:06.014 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:02:06.014 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:02:06.014 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:02:06.014 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T04:02:06.014 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T04:02:06.014 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:02:06.014 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:02:06.014 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:02:06.015 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:02:06.015 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:02:06.015 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:02:06.015 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T04:02:06.015 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:02:06.015 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:02:06.015 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:02:06.015 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 5 2024-09-16T04:02:06.015 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:02:06.015 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:02:06.015 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T04:02:06.015 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:02:06.015 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:02:06.016 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:02:06.016 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 7 2024-09-16T04:02:06.016 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:02:06.016 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:02:06.016 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:02:06.016 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T04:02:06.016 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T04:02:06.018 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.983+0000 7f97e5c13640 10 librados: watch_flush enter 2024-09-16T04:02:06.018 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.983+0000 7f97e5c13640 10 librados: watch_flush exit 2024-09-16T04:02:06.018 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.983+0000 7f97e5c13640 20 client.825474.objecter shutdown clearing up homeless session... 2024-09-16T04:02:06.018 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.983+0000 7f97e5c13640 10 client.825474.objecter successfully canceled tick 2024-09-16T04:02:06.018 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.983+0000 7f97e5c13640 1 -- 172.21.15.42:0/2362404850 >> v2:172.21.15.154:6812/254845342 conn(0x7f97cc0377c0 msgr2=0x7f97cc039c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:02:06.018 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.983+0000 7f97e5c13640 1 --2- 172.21.15.42:0/2362404850 >> v2:172.21.15.154:6812/254845342 conn(0x7f97cc0377c0 0x7f97cc039c80 secure :-1 s=READY pgs=3777 cs=0 l=1 rev1=1 crypto rx=0x7f97d8009660 tx=0x7f97d80098b0 comp rx=0 tx=0).stop 2024-09-16T04:02:06.018 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.983+0000 7f97e5c13640 10 monclient: shutdown 2024-09-16T04:02:06.018 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.983+0000 7f97e5c13640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:02:06.018 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.983+0000 7f97e5c13640 1 -- 172.21.15.42:0/2362404850 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f97e00a67e0 msgr2=0x7f97e01d3230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:02:06.018 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.983+0000 7f97e5c13640 1 --2- 172.21.15.42:0/2362404850 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f97e00a67e0 0x7f97e01d3230 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f97d000c480 tx=0x7f97d000c940 comp rx=0 tx=0).stop 2024-09-16T04:02:06.018 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.983+0000 7f97dffff640 1 -- 172.21.15.42:0/2362404850 reap_dead start 2024-09-16T04:02:06.018 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.983+0000 7f97dffff640 1 -- 172.21.15.42:0/2362404850 reap_dead start 2024-09-16T04:02:06.019 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.983+0000 7f97e5c13640 1 -- 172.21.15.42:0/2362404850 shutdown_connections 2024-09-16T04:02:06.019 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.983+0000 7f97e5c13640 1 -- 172.21.15.42:0/2362404850 >> 172.21.15.42:0/2362404850 conn(0x7f97e00a7ed0 msgr2=0x7f97e0072000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:02:06.019 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.983+0000 7f97e5c13640 1 -- 172.21.15.42:0/2362404850 shutdown_connections 2024-09-16T04:02:06.019 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.983+0000 7f97e5c13640 1 -- 172.21.15.42:0/2362404850 wait complete. 2024-09-16T04:02:06.019 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:05.983+0000 7f97e5c13640 1 librados: shutdown 2024-09-16T04:02:06.019 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T04:02:06.019 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-16T04:02:06.019 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T04:02:06.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.131+0000 7f94c7a53640 10 monclient: get_monmap_and_config 2024-09-16T04:02:06.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.131+0000 7f94c7a53640 10 monclient: build_initial_monmap 2024-09-16T04:02:06.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.131+0000 7f94c7a53640 10 monclient: monmap: 2024-09-16T04:02:06.135 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:02:06.135 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:02:06.135 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:02:06.133625+0000 2024-09-16T04:02:06.135 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:02:06.133625+0000 2024-09-16T04:02:06.135 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:02:06.135 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:02:06.135 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:02:06.136 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:02:06.136 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:02:06.136 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:02:06.136 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:02:06.136 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:02:06.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.131+0000 7f94c7a53640 1 Processor -- start 2024-09-16T04:02:06.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.131+0000 7f94c7a53640 1 -- start start 2024-09-16T04:02:06.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.131+0000 7f94c7a53640 10 monclient: init 2024-09-16T04:02:06.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.131+0000 7f94c7a53640 10 monclient: _reopen_session rank -1 2024-09-16T04:02:06.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.131+0000 7f94c7a53640 10 monclient: _add_conns ranks=[4,3,1,2,0] 2024-09-16T04:02:06.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.131+0000 7f94c7a53640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f94c00a1330 0x7f94c009f180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:06.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.131+0000 7f94c7a53640 10 monclient(hunting): picked mon.noname-c con 0x7f94c00a1330 addr v2:172.21.15.154:3300/0 2024-09-16T04:02:06.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.131+0000 7f94c7a53640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f94c00a1710 0x7f94c00a1b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:06.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.131+0000 7f94c7a53640 10 monclient(hunting): picked mon.noname-a con 0x7f94c00a1710 addr v2:172.21.15.42:3300/0 2024-09-16T04:02:06.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.131+0000 7f94c7a53640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f94c01340f0 addr v1:172.21.15.154:6789/0 2024-09-16T04:02:06.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.131+0000 7f94c7a53640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f94c009e4c0 con 0x7f94c01340f0 2024-09-16T04:02:06.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.131+0000 7f94c7a53640 10 monclient(hunting): start opening mon connection 2024-09-16T04:02:06.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.131+0000 7f94c7a53640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f94c009edf0 con 0x7f94c00a1710 2024-09-16T04:02:06.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.131+0000 7f94c7a53640 10 monclient(hunting): start opening mon connection 2024-09-16T04:02:06.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.131+0000 7f94c7a53640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f94c009cd60 con 0x7f94c00a1330 2024-09-16T04:02:06.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.131+0000 7f94c7a53640 10 monclient(hunting): _renew_subs 2024-09-16T04:02:06.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.131+0000 7f94c7a53640 10 monclient(hunting): authenticate will time out at 2173.103199s 2024-09-16T04:02:06.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.131+0000 7f94c4fee640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f94c00a1710 0x7f94c00a1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:02:06.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.131+0000 7f94c4fee640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f94c00a1710 0x7f94c00a1b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:38958/0 (socket says 172.21.15.42:38958) 2024-09-16T04:02:06.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.131+0000 7f94c4fee640 1 -- 172.21.15.42:0/3659398682 learned_addr learned my addr 172.21.15.42:0/3659398682 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:02:06.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.131+0000 7f94c4fee640 10 monclient(hunting): get_auth_request con 0x7f94c00a1710 auth_method 0 2024-09-16T04:02:06.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.131+0000 7f94c4fee640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:02:06.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.131+0000 7f94c4fee640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:02:06.140 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.131+0000 7f94c4fee640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:02:06.140 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.131+0000 7f94c4fee640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:02:06.140 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.131+0000 7f94c4fee640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:02:06.140 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.131+0000 7f94c4fee640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:02:06.140 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.131+0000 7f94c4fee640 10 monclient(hunting): handle_auth_done global_id 825483 payload 274 2024-09-16T04:02:06.140 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.131+0000 7f94c4fee640 1 -- 172.21.15.42:0/3659398682 >> v2:172.21.15.154:3300/0 conn(0x7f94c00a1330 msgr2=0x7f94c009f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:02:06.140 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.131+0000 7f94c4fee640 1 --2- 172.21.15.42:0/3659398682 >> v2:172.21.15.154:3300/0 conn(0x7f94c00a1330 0x7f94c009f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:06.140 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.131+0000 7f94c4fee640 1 -- 172.21.15.42:0/3659398682 >> v1:172.21.15.154:6789/0 conn(0x7f94c01340f0 legacy=0x7f94c0134540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:02:06.140 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.131+0000 7f94c4fee640 10 monclient: _finish_hunting 0 2024-09-16T04:02:06.140 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.131+0000 7f94c4fee640 1 monclient: found mon.noname-a 2024-09-16T04:02:06.140 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.131+0000 7f94c4fee640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:02:06.140 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.131+0000 7f94c4fee640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:02:06.140 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.131+0000 7f94c4fee640 1 -- 172.21.15.42:0/3659398682 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94c00a2080 con 0x7f94c00a1710 2024-09-16T04:02:06.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.135+0000 7f94c4fee640 1 --2- 172.21.15.42:0/3659398682 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f94c00a1710 0x7f94c00a1b40 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f94c006add0 tx=0x7f94b001e640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7962aa33139f107f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:02:06.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.135+0000 7f94b7fff640 1 -- 172.21.15.42:0/3659398682 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94b00ad660 con 0x7f94c00a1710 2024-09-16T04:02:06.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.135+0000 7f94b7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:02:06.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.135+0000 7f94b7fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:02:06.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.135+0000 7f94b7fff640 10 monclient: dump: 2024-09-16T04:02:06.147 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:02:06.147 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:02:06.147 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:06.147 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:06.147 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:02:06.147 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:02:06.148 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:02:06.148 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:02:06.148 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:02:06.148 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:02:06.148 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.135+0000 7f94b7fff640 10 monclient: _finish_auth 0 2024-09-16T04:02:06.148 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.135+0000 7f94b7fff640 10 monclient: _check_auth_tickets 2024-09-16T04:02:06.148 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.135+0000 7f94b7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:02:06.148 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.135+0000 7f94b7fff640 1 -- 172.21.15.42:0/3659398682 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f94b00b2070 con 0x7f94c00a1710 2024-09-16T04:02:06.148 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.135+0000 7f94b7fff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:02:06.148 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.135+0000 7f94b7fff640 1 -- 172.21.15.42:0/3659398682 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94b00ada20 con 0x7f94c00a1710 2024-09-16T04:02:06.148 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.135+0000 7f94c7a53640 5 monclient: authenticate success, global_id 825483 2024-09-16T04:02:06.148 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.135+0000 7f94c7a53640 10 monclient: get_monmap_and_config success 2024-09-16T04:02:06.148 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.135+0000 7f94c7a53640 4 set_mon_vals no callback set 2024-09-16T04:02:06.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.135+0000 7f94b7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:02:06.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.135+0000 7f94b7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:02:06.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.135+0000 7f94b7fff640 10 monclient: dump: 2024-09-16T04:02:06.149 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:02:06.149 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:02:06.149 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:06.149 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:06.149 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:02:06.149 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:02:06.149 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:02:06.149 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:02:06.149 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:02:06.149 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:02:06.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.135+0000 7f94c7a53640 10 monclient: shutdown 2024-09-16T04:02:06.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.135+0000 7f94c7a53640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:02:06.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.135+0000 7f94c7a53640 1 -- 172.21.15.42:0/3659398682 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f94c00a1710 msgr2=0x7f94c00a1b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:02:06.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.135+0000 7f94c7a53640 1 --2- 172.21.15.42:0/3659398682 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f94c00a1710 0x7f94c00a1b40 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f94c006add0 tx=0x7f94b001e640 comp rx=0 tx=0).stop 2024-09-16T04:02:06.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.135+0000 7f94c7a53640 1 -- 172.21.15.42:0/3659398682 shutdown_connections 2024-09-16T04:02:06.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.135+0000 7f94c7a53640 1 --2- 172.21.15.42:0/3659398682 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f94c00a1710 0x7f94c00a1b40 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:06.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.135+0000 7f94c7a53640 1 --2- 172.21.15.42:0/3659398682 >> v2:172.21.15.154:3300/0 conn(0x7f94c00a1330 0x7f94c009f180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:06.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.135+0000 7f94c7a53640 1 -- 172.21.15.42:0/3659398682 >> 172.21.15.42:0/3659398682 conn(0x7f94c00a2420 msgr2=0x7f94c00a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:02:06.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.135+0000 7f94c7a53640 1 -- 172.21.15.42:0/3659398682 shutdown_connections 2024-09-16T04:02:06.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.135+0000 7f94c7a53640 1 -- 172.21.15.42:0/3659398682 wait complete. 2024-09-16T04:02:06.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.135+0000 7f94c7a53640 10 monclient: build_initial_monmap 2024-09-16T04:02:06.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.135+0000 7f94c7a53640 10 monclient: monmap: 2024-09-16T04:02:06.150 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:02:06.150 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:02:06.151 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:02:06.151 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:02:06.151 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:02:06.151 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:02:06.151 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:02:06.151 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:02:06.151 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:02:06.151 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:02:06.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.135+0000 7f94c7a53640 1 librados: starting msgr at 2024-09-16T04:02:06.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.135+0000 7f94c7a53640 1 librados: starting objecter 2024-09-16T04:02:06.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.139+0000 7f94c7a53640 1 Processor -- start 2024-09-16T04:02:06.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.139+0000 7f94c7a53640 1 -- start start 2024-09-16T04:02:06.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.139+0000 7f94c7a53640 1 librados: setting wanted keys 2024-09-16T04:02:06.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.139+0000 7f94c7a53640 1 librados: calling monclient init 2024-09-16T04:02:06.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.139+0000 7f94c7a53640 10 monclient: init 2024-09-16T04:02:06.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.139+0000 7f94c7a53640 10 monclient: _reopen_session rank -1 2024-09-16T04:02:06.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.139+0000 7f94c7a53640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:02:06.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.139+0000 7f94c7a53640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f94c00a1330 0x7f94c0132e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:06.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.139+0000 7f94c7a53640 10 monclient(hunting): picked mon.noname-a con 0x7f94c00a1330 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:02:06.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.139+0000 7f94c7a53640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f94c00a1710 0x7f94c0133380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:06.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.139+0000 7f94c7a53640 10 monclient(hunting): picked mon.noname-b con 0x7f94c00a1710 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:02:06.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.139+0000 7f94c7a53640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f94c01340f0 0x7f94c0139e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:06.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.139+0000 7f94c7a53640 10 monclient(hunting): picked mon.noname-c con 0x7f94c01340f0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:02:06.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.139+0000 7f94c7a53640 10 monclient(hunting): start opening mon connection 2024-09-16T04:02:06.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.139+0000 7f94c7a53640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f94c009c3e0 con 0x7f94c00a1330 2024-09-16T04:02:06.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.139+0000 7f94c7a53640 10 monclient(hunting): start opening mon connection 2024-09-16T04:02:06.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.139+0000 7f94c7a53640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f94c009c260 con 0x7f94c00a1710 2024-09-16T04:02:06.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.139+0000 7f94c7a53640 10 monclient(hunting): start opening mon connection 2024-09-16T04:02:06.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.139+0000 7f94c7a53640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f94c009c560 con 0x7f94c01340f0 2024-09-16T04:02:06.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.139+0000 7f94c7a53640 10 monclient(hunting): _renew_subs 2024-09-16T04:02:06.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.139+0000 7f94c7a53640 10 monclient(hunting): authenticate will time out at 2173.108822s 2024-09-16T04:02:06.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.139+0000 7f94c5ff0640 1 -- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f94c01340f0 msgr2=0x7f94c0139e00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T04:02:06.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.139+0000 7f94c5ff0640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f94c01340f0 0x7f94c0139e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:02:06.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.139+0000 7f94c57ef640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f94c00a1330 0x7f94c0132e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:02:06.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.139+0000 7f94c57ef640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f94c00a1330 0x7f94c0132e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:38968/0 (socket says 172.21.15.42:38968) 2024-09-16T04:02:06.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.139+0000 7f94c57ef640 1 -- 172.21.15.42:0/1757951818 learned_addr learned my addr 172.21.15.42:0/1757951818 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:02:06.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.139+0000 7f94c57ef640 10 monclient(hunting): get_auth_request con 0x7f94c00a1330 auth_method 0 2024-09-16T04:02:06.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.139+0000 7f94c57ef640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:02:06.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.139+0000 7f94c57ef640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:02:06.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.139+0000 7f94c57ef640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:02:06.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.139+0000 7f94c57ef640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:02:06.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.139+0000 7f94c57ef640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:02:06.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.139+0000 7f94c57ef640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:02:06.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.139+0000 7f94c57ef640 10 monclient(hunting): handle_auth_done global_id 825486 payload 824 2024-09-16T04:02:06.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.139+0000 7f94c57ef640 1 -- 172.21.15.42:0/1757951818 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f94c01340f0 msgr2=0x7f94c0139e00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:02:06.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.139+0000 7f94c57ef640 1 --2- 172.21.15.42:0/1757951818 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f94c01340f0 0x7f94c0139e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:06.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.139+0000 7f94c57ef640 1 -- 172.21.15.42:0/1757951818 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f94c00a1710 msgr2=0x7f94c0133380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:02:06.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.139+0000 7f94c4fee640 1 --2- 172.21.15.42:0/1757951818 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f94c00a1710 0x7f94c0133380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:02:06.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.139+0000 7f94c57ef640 1 --2- 172.21.15.42:0/1757951818 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f94c00a1710 0x7f94c0133380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:06.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.139+0000 7f94c57ef640 10 monclient: _finish_hunting 0 2024-09-16T04:02:06.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.139+0000 7f94c57ef640 1 monclient: found mon.noname-a 2024-09-16T04:02:06.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.139+0000 7f94c57ef640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:02:06.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.139+0000 7f94c57ef640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:02:06.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.139+0000 7f94c57ef640 1 -- 172.21.15.42:0/1757951818 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94c011d940 con 0x7f94c00a1330 2024-09-16T04:02:06.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.139+0000 7f94c57ef640 1 --2- 172.21.15.42:0/1757951818 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f94c00a1330 0x7f94c0132e40 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f94bc00ba50 tx=0x7f94bc00bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:02:06.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.139+0000 7f94b5ffb640 10 client.?.objecter ms_handle_connect 0x7f94c00a1330 2024-09-16T04:02:06.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.139+0000 7f94b5ffb640 10 client.?.objecter resend_mon_ops 2024-09-16T04:02:06.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.139+0000 7f94b5ffb640 1 -- 172.21.15.42:0/1757951818 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94bc019070 con 0x7f94c00a1330 2024-09-16T04:02:06.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.139+0000 7f94b5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:02:06.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.139+0000 7f94b5ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:02:06.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.139+0000 7f94b5ffb640 10 monclient: dump: 2024-09-16T04:02:06.155 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:02:06.155 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:02:06.155 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:06.155 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:06.155 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:02:06.155 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:02:06.155 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:02:06.156 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:02:06.156 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:02:06.156 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:02:06.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.139+0000 7f94b5ffb640 10 monclient: _finish_auth 0 2024-09-16T04:02:06.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.139+0000 7f94b5ffb640 10 monclient: _check_auth_tickets 2024-09-16T04:02:06.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.139+0000 7f94b5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:02:06.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.139+0000 7f94b5ffb640 1 -- 172.21.15.42:0/1757951818 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f94bc00cc00 con 0x7f94c00a1330 2024-09-16T04:02:06.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.139+0000 7f94b5ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:02:06.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.139+0000 7f94b5ffb640 1 -- 172.21.15.42:0/1757951818 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94bc014440 con 0x7f94c00a1330 2024-09-16T04:02:06.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.139+0000 7f94b5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:02:06.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.139+0000 7f94b5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:02:06.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.139+0000 7f94b5ffb640 10 monclient: dump: 2024-09-16T04:02:06.156 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:02:06.156 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:02:06.157 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:06.157 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:06.157 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:02:06.157 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:02:06.157 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:02:06.157 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:02:06.157 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:02:06.157 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:02:06.157 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.139+0000 7f94c7a53640 5 monclient: authenticate success, global_id 825486 2024-09-16T04:02:06.157 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.139+0000 7f94c7a53640 10 monclient: _renew_subs 2024-09-16T04:02:06.157 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.139+0000 7f94b77fe640 4 set_mon_vals no callback set 2024-09-16T04:02:06.157 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.139+0000 7f94c7a53640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:02:06.157 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.139+0000 7f94c7a53640 1 -- 172.21.15.42:0/1757951818 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94c0071e00 con 0x7f94c00a1330 2024-09-16T04:02:06.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.139+0000 7f94c7a53640 10 client.825486.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:02:06.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.139+0000 7f94c7a53640 10 monclient: _renew_subs 2024-09-16T04:02:06.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.139+0000 7f94c7a53640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:02:06.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.139+0000 7f94c7a53640 1 -- 172.21.15.42:0/1757951818 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94c0073a30 con 0x7f94c00a1330 2024-09-16T04:02:06.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.139+0000 7f94c7a53640 1 librados: init done 2024-09-16T04:02:06.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.139+0000 7f94c7a53640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:02:06.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.139+0000 7f94c7a53640 1 --2- 172.21.15.42:0/1757951818 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f94c011d210 0x7f94c013ab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:06.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.139+0000 7f94c7a53640 10 monclient(hunting): start opening mon connection 2024-09-16T04:02:06.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.139+0000 7f94c7a53640 1 -- 172.21.15.42:0/1757951818 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f94c011bf70 con 0x7f94c011d210 2024-09-16T04:02:06.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.139+0000 7f94b5ffb640 1 -- 172.21.15.42:0/1757951818 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f94bc01c070 con 0x7f94c00a1330 2024-09-16T04:02:06.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.139+0000 7f94c4fee640 1 --2- 172.21.15.42:0/1757951818 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f94c011d210 0x7f94c013ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:02:06.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.139+0000 7f94c4fee640 10 monclient: get_auth_request con 0x7f94c011d210 auth_method 0 2024-09-16T04:02:06.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.139+0000 7f94c4fee640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:02:06.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.139+0000 7f94c4fee640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:02:06.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.139+0000 7f94c4fee640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:02:06.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.139+0000 7f94b5ffb640 1 --2- 172.21.15.42:0/1757951818 >> v2:172.21.15.154:6812/254845342 conn(0x7f94a4037870 0x7f94a4039d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:06.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.139+0000 7f94b5ffb640 1 -- 172.21.15.42:0/1757951818 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(602..602 src has 451..602) ==== 6252+0+0 (secure 0 0 0) 0x7f94bc048e10 con 0x7f94c00a1330 2024-09-16T04:02:06.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.139+0000 7f94b5ffb640 10 client.825486.objecter ms_dispatch 0x7f94c0000ba0 osd_map(602..602 src has 451..602) 2024-09-16T04:02:06.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.139+0000 7f94c4fee640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:02:06.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.139+0000 7f94b5ffb640 3 client.825486.objecter handle_osd_map got epochs [602,602] > 0 2024-09-16T04:02:06.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.139+0000 7f94b5ffb640 3 client.825486.objecter handle_osd_map decoding full epoch 602 2024-09-16T04:02:06.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.139+0000 7f94c4fee640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:02:06.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.139+0000 7f94c4fee640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:02:06.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.139+0000 7f94b5ffb640 20 client.825486.objecter dump_active .. 0 homeless 2024-09-16T04:02:06.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.139+0000 7f94c4fee640 10 monclient(hunting): handle_auth_done global_id 904579 payload 824 2024-09-16T04:02:06.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.139+0000 7f94c4fee640 1 -- 172.21.15.42:0/1757951818 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f94c009f180 con 0x7f94c011d210 2024-09-16T04:02:06.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.139+0000 7f94c5ff0640 1 --2- 172.21.15.42:0/1757951818 >> v2:172.21.15.154:6812/254845342 conn(0x7f94a4037870 0x7f94a4039d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:02:06.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.139+0000 7f94c5ff0640 10 monclient: get_auth_request con 0x7f94a4037870 auth_method 0 2024-09-16T04:02:06.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.139+0000 7f94b5ffb640 10 client.825486.objecter ms_handle_connect 0x7f94c011d210 2024-09-16T04:02:06.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.139+0000 7f94b5ffb640 10 client.825486.objecter resend_mon_ops 2024-09-16T04:02:06.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.139+0000 7f94c4fee640 1 --2- 172.21.15.42:0/1757951818 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f94c011d210 0x7f94c013ab00 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f94c009df50 tx=0x7f94b001ce80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:02:06.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.143+0000 7f94b5ffb640 1 -- 172.21.15.42:0/1757951818 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94b00c05f0 con 0x7f94c011d210 2024-09-16T04:02:06.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.143+0000 7f94b5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:02:06.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.143+0000 7f94b5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:02:06.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.143+0000 7f94b5ffb640 10 monclient: dump: 2024-09-16T04:02:06.161 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:02:06.161 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:02:06.161 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:06.161 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:06.161 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:02:06.161 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:02:06.161 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:02:06.161 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:02:06.161 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:02:06.161 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:02:06.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.143+0000 7f94b5ffb640 10 client.825486.objecter ms_handle_connect 0x7f94a4037870 2024-09-16T04:02:06.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.143+0000 7f94c5ff0640 1 --2- 172.21.15.42:0/1757951818 >> v2:172.21.15.154:6812/254845342 conn(0x7f94a4037870 0x7f94a4039d30 secure :-1 s=READY pgs=3778 cs=0 l=1 rev1=1 crypto rx=0x7f94b8007fd0 tx=0x7f94b80079a0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:02:06.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.143+0000 7f94b5ffb640 1 -- 172.21.15.42:0/1757951818 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f94b00b2040 con 0x7f94c011d210 2024-09-16T04:02:06.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.143+0000 7f94b5ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:02:06.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.143+0000 7f94b5ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:02:06.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.143+0000 7f94b5ffb640 1 -- 172.21.15.42:0/1757951818 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f94c011d210 msgr2=0x7f94c013ab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:02:06.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.143+0000 7f94b5ffb640 1 --2- 172.21.15.42:0/1757951818 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f94c011d210 0x7f94c013ab00 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f94c009df50 tx=0x7f94b001ce80 comp rx=0 tx=0).stop 2024-09-16T04:02:06.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.143+0000 7f94b5ffb640 1 -- 172.21.15.42:0/1757951818 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f94c011d210 msgr2=0x7f94c013ab00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:02:06.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.143+0000 7f94b5ffb640 1 --2- 172.21.15.42:0/1757951818 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f94c011d210 0x7f94c013ab00 secure :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f94c009df50 tx=0x7f94b001ce80 comp rx=0 tx=0).stop 2024-09-16T04:02:06.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.147+0000 7f94c7a53640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:02:06.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.147+0000 7f94c7a53640 1 --2- 172.21.15.42:0/1757951818 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f94c002c650 0x7f94c002ca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:06.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.147+0000 7f94c7a53640 10 monclient(hunting): start opening mon connection 2024-09-16T04:02:06.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.147+0000 7f94c7a53640 1 -- 172.21.15.42:0/1757951818 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f94c0136a90 con 0x7f94c002c650 2024-09-16T04:02:06.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.151+0000 7f94c4fee640 1 --2- 172.21.15.42:0/1757951818 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f94c002c650 0x7f94c002ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:02:06.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.151+0000 7f94c4fee640 10 monclient: get_auth_request con 0x7f94c002c650 auth_method 0 2024-09-16T04:02:06.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.151+0000 7f94c4fee640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:02:06.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.151+0000 7f94c4fee640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:02:06.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.151+0000 7f94c4fee640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:02:06.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.151+0000 7f94c4fee640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:02:06.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.151+0000 7f94c4fee640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:02:06.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.151+0000 7f94c4fee640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:02:06.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.151+0000 7f94c4fee640 10 monclient(hunting): handle_auth_done global_id 904582 payload 824 2024-09-16T04:02:06.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.151+0000 7f94c4fee640 1 -- 172.21.15.42:0/1757951818 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f94c0069040 con 0x7f94c002c650 2024-09-16T04:02:06.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.151+0000 7f94c4fee640 1 --2- 172.21.15.42:0/1757951818 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f94c002c650 0x7f94c002ca30 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f94b001c270 tx=0x7f94b001b3e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:02:06.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.151+0000 7f94b5ffb640 10 client.825486.objecter ms_handle_connect 0x7f94c002c650 2024-09-16T04:02:06.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.151+0000 7f94b5ffb640 10 client.825486.objecter resend_mon_ops 2024-09-16T04:02:06.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.151+0000 7f94b5ffb640 1 -- 172.21.15.42:0/1757951818 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94b00cc070 con 0x7f94c002c650 2024-09-16T04:02:06.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.151+0000 7f94b5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:02:06.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.151+0000 7f94b5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:02:06.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.151+0000 7f94b5ffb640 10 monclient: dump: 2024-09-16T04:02:06.165 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:02:06.165 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:02:06.166 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:06.166 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:06.166 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:02:06.166 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:02:06.166 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:02:06.166 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:02:06.166 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:02:06.166 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:02:06.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.151+0000 7f94b5ffb640 1 -- 172.21.15.42:0/1757951818 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4869 (secure 0 0 0) 0x7f94b00c0d30 con 0x7f94c002c650 2024-09-16T04:02:06.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.151+0000 7f94b5ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:02:06.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.151+0000 7f94b5ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:02:06.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.151+0000 7f94b5ffb640 1 -- 172.21.15.42:0/1757951818 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f94c002c650 msgr2=0x7f94c002ca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:02:06.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.151+0000 7f94b5ffb640 1 --2- 172.21.15.42:0/1757951818 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f94c002c650 0x7f94c002ca30 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f94b001c270 tx=0x7f94b001b3e0 comp rx=0 tx=0).stop 2024-09-16T04:02:06.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.151+0000 7f94b5ffb640 1 -- 172.21.15.42:0/1757951818 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f94c002c650 msgr2=0x7f94c002ca30 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:02:06.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.151+0000 7f94b5ffb640 1 --2- 172.21.15.42:0/1757951818 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f94c002c650 0x7f94c002ca30 secure :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f94b001c270 tx=0x7f94b001b3e0 comp rx=0 tx=0).stop 2024-09-16T04:02:06.167 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T04:02:06.167 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:02:06.167 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:02:06.167 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T04:02:06.167 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 182, 2024-09-16T04:02:06.167 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 254332, 2024-09-16T04:02:06.167 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T04:02:06.167 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T04:02:06.168 INFO:teuthology.orchestra.run.smithi042.stdout: 1 2024-09-16T04:02:06.168 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:02:06.168 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 0, 2024-09-16T04:02:06.168 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:02:06.168 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T04:02:06.168 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T04:02:06.168 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:02:06.168 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:02:06.168 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:02:06.168 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:02:06.168 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:02:06.168 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:02:06.168 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:02:06.169 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:02:06.169 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:02:06.169 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:02:06.169 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:02:06.169 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:02:06.169 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:02:06.169 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T04:02:06.169 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:02:06.169 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:02:06.169 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:02:06.169 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:02:06.169 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:02:06.169 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:02:06.169 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:02:06.170 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:02:06.170 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:02:06.170 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:02:06.170 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:02:06.170 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:02:06.170 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:02:06.170 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T04:02:06.170 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T04:02:06.170 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T04:02:06.170 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T04:02:06.170 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T04:02:06.170 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T04:02:06.170 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:02:06.171 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:02:06.171 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T04:02:06.171 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T04:02:06.171 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T04:02:06.171 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T04:02:06.171 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T04:02:06.171 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T04:02:06.171 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T04:02:06.171 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:02:06.171 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T04:02:06.171 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:02:06.171 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:02:06.171 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:02:06.171 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:02:06.172 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:02:06.172 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:02:06.172 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:02:06.172 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:02:06.172 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:02:06.172 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:02:06.172 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:02:06.172 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:02:06.172 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T04:02:06.172 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:02:06.172 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T04:02:06.172 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:02:06.172 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:02:06.172 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:02:06.173 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:02:06.173 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:02:06.173 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:02:06.173 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:02:06.173 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T04:02:06.173 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:02:06.173 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:02:06.173 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:02:06.173 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:02:06.173 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T04:02:06.173 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:02:06.173 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:02:06.173 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:02:06.173 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:02:06.174 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T04:02:06.174 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T04:02:06.174 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:02:06.174 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:02:06.174 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:02:06.174 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:02:06.174 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:02:06.174 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:02:06.174 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:02:06.174 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:02:06.174 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:02:06.174 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:02:06.174 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:02:06.174 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T04:02:06.175 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:02:06.175 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:02:06.175 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:02:06.175 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:02:06.175 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T04:02:06.175 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:02:06.175 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:02:06.175 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:02:06.175 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:02:06.175 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T04:02:06.175 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T04:02:06.175 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:02:06.175 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:02:06.175 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:02:06.176 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:02:06.176 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:02:06.176 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:02:06.176 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:02:06.176 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:02:06.176 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:02:06.176 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:02:06.176 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:02:06.176 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T04:02:06.176 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:02:06.176 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:02:06.176 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:02:06.176 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:02:06.176 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T04:02:06.177 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:02:06.177 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:02:06.177 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:02:06.177 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:02:06.177 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T04:02:06.177 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T04:02:06.177 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:02:06.177 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:02:06.177 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:02:06.177 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:02:06.177 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:02:06.177 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:02:06.177 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T04:02:06.178 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T04:02:06.178 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:02:06.178 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:02:06.178 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:02:06.178 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:02:06.178 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:02:06.178 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:02:06.178 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T04:02:06.178 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:02:06.178 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:02:06.178 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:02:06.178 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:02:06.178 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:02:06.178 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:02:06.179 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T04:02:06.179 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:02:06.179 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:02:06.179 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:02:06.179 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 2 2024-09-16T04:02:06.179 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:02:06.179 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:02:06.179 INFO:teuthology.orchestra.run.smithi042.stdout: "mgr": [ 2024-09-16T04:02:06.179 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:02:06.179 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:02:06.179 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:02:06.179 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:02:06.179 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:02:06.179 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:02:06.179 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:02:06.180 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T04:02:06.180 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T04:02:06.181 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.151+0000 7f94c7a53640 10 librados: watch_flush enter 2024-09-16T04:02:06.181 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.151+0000 7f94c7a53640 10 librados: watch_flush exit 2024-09-16T04:02:06.181 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.151+0000 7f94c7a53640 20 client.825486.objecter shutdown clearing up homeless session... 2024-09-16T04:02:06.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.151+0000 7f94c7a53640 10 client.825486.objecter successfully canceled tick 2024-09-16T04:02:06.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.151+0000 7f94c7a53640 1 -- 172.21.15.42:0/1757951818 >> v2:172.21.15.154:6812/254845342 conn(0x7f94a4037870 msgr2=0x7f94a4039d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:02:06.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.151+0000 7f94c7a53640 1 --2- 172.21.15.42:0/1757951818 >> v2:172.21.15.154:6812/254845342 conn(0x7f94a4037870 0x7f94a4039d30 secure :-1 s=READY pgs=3778 cs=0 l=1 rev1=1 crypto rx=0x7f94b8007fd0 tx=0x7f94b80079a0 comp rx=0 tx=0).stop 2024-09-16T04:02:06.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.151+0000 7f94c7a53640 10 monclient: shutdown 2024-09-16T04:02:06.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.151+0000 7f94c7a53640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:02:06.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.151+0000 7f94c7a53640 1 -- 172.21.15.42:0/1757951818 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f94c00a1330 msgr2=0x7f94c0132e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:02:06.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.151+0000 7f94c7a53640 1 --2- 172.21.15.42:0/1757951818 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f94c00a1330 0x7f94c0132e40 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f94bc00ba50 tx=0x7f94bc00bf10 comp rx=0 tx=0).stop 2024-09-16T04:02:06.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.151+0000 7f94c5ff0640 1 -- 172.21.15.42:0/1757951818 reap_dead start 2024-09-16T04:02:06.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.151+0000 7f94c5ff0640 1 -- 172.21.15.42:0/1757951818 reap_dead start 2024-09-16T04:02:06.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.151+0000 7f94c7a53640 1 -- 172.21.15.42:0/1757951818 shutdown_connections 2024-09-16T04:02:06.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.151+0000 7f94c7a53640 1 -- 172.21.15.42:0/1757951818 >> 172.21.15.42:0/1757951818 conn(0x7f94c00a2420 msgr2=0x7f94c00a4360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:02:06.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.151+0000 7f94c7a53640 1 -- 172.21.15.42:0/1757951818 shutdown_connections 2024-09-16T04:02:06.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.151+0000 7f94c7a53640 1 -- 172.21.15.42:0/1757951818 wait complete. 2024-09-16T04:02:06.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:06.151+0000 7f94c7a53640 1 librados: shutdown 2024-09-16T04:02:06.183 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T04:02:06.183 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-16T04:02:26.183 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-16T04:02:26.184 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-09-16T04:02:26.184 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-16T04:02:26.184 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-16T04:02:26.187 INFO:tasks.ceph.mon.c:Started 2024-09-16T04:02:26.187 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T04:02:26.187 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:02:26.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.307+0000 7f3ea3c01640 10 monclient: get_monmap_and_config 2024-09-16T04:02:26.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.307+0000 7f3ea3c01640 10 monclient: build_initial_monmap 2024-09-16T04:02:26.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.307+0000 7f3ea3c01640 10 monclient: monmap: 2024-09-16T04:02:26.311 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:02:26.311 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:02:26.312 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:02:26.309997+0000 2024-09-16T04:02:26.312 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:02:26.309997+0000 2024-09-16T04:02:26.312 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:02:26.312 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:02:26.312 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:02:26.312 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:02:26.312 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:02:26.312 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:02:26.312 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:02:26.312 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:02:26.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.307+0000 7f3ea3c01640 1 Processor -- start 2024-09-16T04:02:26.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.307+0000 7f3ea3c01640 1 -- start start 2024-09-16T04:02:26.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.307+0000 7f3ea3c01640 10 monclient: init 2024-09-16T04:02:26.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.307+0000 7f3ea3c01640 10 monclient: _reopen_session rank -1 2024-09-16T04:02:26.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.307+0000 7f3ea3c01640 10 monclient: _add_conns ranks=[3,2,0,4,1] 2024-09-16T04:02:26.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.307+0000 7f3ea3c01640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f3e9c0a39b0 0x7f3e9c133b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:26.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.307+0000 7f3ea3c01640 10 monclient(hunting): picked mon.noname-a con 0x7f3e9c0a39b0 addr v2:172.21.15.42:3300/0 2024-09-16T04:02:26.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.307+0000 7f3ea3c01640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3e9c0a3d90 0x7f3e9c137000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:26.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.307+0000 7f3ea3c01640 10 monclient(hunting): picked mon.noname-b con 0x7f3e9c0a3d90 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:02:26.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.307+0000 7f3ea3c01640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3e9c137540 addr v1:172.21.15.42:6789/0 2024-09-16T04:02:26.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.307+0000 7f3ea3c01640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3e9c09c140 con 0x7f3e9c137540 2024-09-16T04:02:26.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.307+0000 7f3ea3c01640 10 monclient(hunting): start opening mon connection 2024-09-16T04:02:26.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.307+0000 7f3ea3c01640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f3e9c09c420 con 0x7f3e9c0a39b0 2024-09-16T04:02:26.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.307+0000 7f3ea3c01640 10 monclient(hunting): start opening mon connection 2024-09-16T04:02:26.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.307+0000 7f3ea3c01640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f3e9c06dd60 con 0x7f3e9c0a3d90 2024-09-16T04:02:26.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.307+0000 7f3ea3c01640 10 monclient(hunting): _renew_subs 2024-09-16T04:02:26.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.307+0000 7f3ea3c01640 10 monclient(hunting): authenticate will time out at 2193.279045s 2024-09-16T04:02:26.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.307+0000 7f3ea119c640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3e9c0a3d90 0x7f3e9c137000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:02:26.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.307+0000 7f3ea199d640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f3e9c0a39b0 0x7f3e9c133b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:02:26.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.307+0000 7f3ea219e640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f3e9c137540 0x7f3e9c13e390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:41266/0 (socket says 172.21.15.42:41266) 2024-09-16T04:02:26.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.307+0000 7f3ea119c640 10 monclient(hunting): get_auth_request con 0x7f3e9c0a3d90 auth_method 0 2024-09-16T04:02:26.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.307+0000 7f3ea119c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:02:26.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.307+0000 7f3ea119c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:02:26.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.307+0000 7f3ea119c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:02:26.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.307+0000 7f3ea219e640 1 -- 172.21.15.42:0/3955966857 learned_addr learned my addr 172.21.15.42:0/3955966857 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T04:02:26.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.307+0000 7f3ea199d640 10 monclient(hunting): get_auth_request con 0x7f3e9c0a39b0 auth_method 0 2024-09-16T04:02:26.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.307+0000 7f3ea199d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:02:26.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.307+0000 7f3ea199d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:02:26.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.307+0000 7f3ea199d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:02:26.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.307+0000 7f3ea199d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:02:26.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.307+0000 7f3ea199d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:02:26.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.307+0000 7f3ea119c640 1 -- 172.21.15.42:0/3955966857 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3e9c0a3d90 msgr2=0x7f3e9c137000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:02:26.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.307+0000 7f3ea119c640 1 -- 172.21.15.42:0/3955966857 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3e9c0a3d90 msgr2=0x7f3e9c137000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:02:26.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.307+0000 7f3ea119c640 1 --2- 172.21.15.42:0/3955966857 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3e9c0a3d90 0x7f3e9c137000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:02:26.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.307+0000 7f3ea199d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:02:26.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.307+0000 7f3ea119c640 1 --2- 172.21.15.42:0/3955966857 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3e9c0a3d90 0x7f3e9c137000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:02:26.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.307+0000 7f3ea199d640 10 monclient(hunting): handle_auth_done global_id 825489 payload 274 2024-09-16T04:02:26.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.307+0000 7f3ea199d640 1 -- 172.21.15.42:0/3955966857 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3e9c0a3d90 msgr2=0x7f3e9c137000 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:02:26.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.307+0000 7f3ea199d640 1 --2- 172.21.15.42:0/3955966857 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3e9c0a3d90 0x7f3e9c137000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:26.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.307+0000 7f3ea199d640 1 -- 172.21.15.42:0/3955966857 >> v1:172.21.15.42:6789/0 conn(0x7f3e9c137540 legacy=0x7f3e9c13e390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:02:26.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.307+0000 7f3ea199d640 10 monclient: _finish_hunting 0 2024-09-16T04:02:26.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.307+0000 7f3ea199d640 1 monclient: found mon.noname-a 2024-09-16T04:02:26.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.307+0000 7f3ea199d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:02:26.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.307+0000 7f3ea199d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:02:26.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.307+0000 7f3ea099b640 1 -- 172.21.15.42:0/3955966857 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2553467096 0 0) 0x7f3e9c09c140 con 0x7f3e9c137540 2024-09-16T04:02:26.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.307+0000 7f3ea199d640 1 -- 172.21.15.42:0/3955966857 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e9c0a5e20 con 0x7f3e9c0a39b0 2024-09-16T04:02:26.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.307+0000 7f3ea099b640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T04:02:26.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.311+0000 7f3ea199d640 1 --2- 172.21.15.42:0/3955966857 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3e9c0a39b0 0x7f3e9c133b80 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f3e980097c0 tx=0x7f3e9801c360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=692243c6ec98209c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:02:26.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.311+0000 7f3ea099b640 1 -- 172.21.15.42:0/3955966857 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e980b0070 con 0x7f3e9c0a39b0 2024-09-16T04:02:26.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.311+0000 7f3ea099b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:02:26.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.311+0000 7f3ea099b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:02:26.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.311+0000 7f3ea099b640 10 monclient: dump: 2024-09-16T04:02:26.319 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:02:26.319 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:02:26.319 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:26.319 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:26.319 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:02:26.319 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:02:26.320 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:02:26.320 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:02:26.320 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:02:26.320 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:02:26.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.311+0000 7f3ea099b640 10 monclient: _finish_auth 0 2024-09-16T04:02:26.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.311+0000 7f3ea099b640 10 monclient: _check_auth_tickets 2024-09-16T04:02:26.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.311+0000 7f3ea099b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:02:26.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.311+0000 7f3ea3c01640 5 monclient: authenticate success, global_id 825489 2024-09-16T04:02:26.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.311+0000 7f3ea3c01640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:02:26.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.311+0000 7f3ea099b640 1 -- 172.21.15.42:0/3955966857 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e980ab600 con 0x7f3e9c0a39b0 2024-09-16T04:02:26.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.311+0000 7f3ea099b640 10 monclient: handle_config config(0 keys) 2024-09-16T04:02:26.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.311+0000 7f3ea3c01640 10 monclient: get_monmap_and_config success 2024-09-16T04:02:26.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.311+0000 7f3ea3c01640 4 set_mon_vals no callback set 2024-09-16T04:02:26.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.311+0000 7f3ea099b640 1 -- 172.21.15.42:0/3955966857 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e980ab9d0 con 0x7f3e9c0a39b0 2024-09-16T04:02:26.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.311+0000 7f3ea099b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:02:26.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.311+0000 7f3ea099b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:02:26.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.311+0000 7f3ea099b640 10 monclient: dump: 2024-09-16T04:02:26.321 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:02:26.321 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:02:26.321 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:26.321 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:26.321 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:02:26.321 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:02:26.321 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:02:26.321 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:02:26.321 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:02:26.321 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:02:26.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.311+0000 7f3ea3c01640 10 monclient: shutdown 2024-09-16T04:02:26.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.311+0000 7f3ea3c01640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:02:26.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.311+0000 7f3ea3c01640 1 -- 172.21.15.42:0/3955966857 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3e9c0a39b0 msgr2=0x7f3e9c133b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:02:26.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.311+0000 7f3ea3c01640 1 --2- 172.21.15.42:0/3955966857 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3e9c0a39b0 0x7f3e9c133b80 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f3e980097c0 tx=0x7f3e9801c360 comp rx=0 tx=0).stop 2024-09-16T04:02:26.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.311+0000 7f3ea3c01640 1 -- 172.21.15.42:0/3955966857 shutdown_connections 2024-09-16T04:02:26.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.311+0000 7f3ea3c01640 1 --2- 172.21.15.42:0/3955966857 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3e9c0a3d90 0x7f3e9c137000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:26.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.311+0000 7f3ea3c01640 1 --2- 172.21.15.42:0/3955966857 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3e9c0a39b0 0x7f3e9c133b80 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:26.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.311+0000 7f3ea3c01640 1 -- 172.21.15.42:0/3955966857 >> 172.21.15.42:0/3955966857 conn(0x7f3e9c0a9fb0 msgr2=0x7f3e9c0aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:02:26.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.311+0000 7f3ea3c01640 1 -- 172.21.15.42:0/3955966857 shutdown_connections 2024-09-16T04:02:26.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.311+0000 7f3ea3c01640 1 -- 172.21.15.42:0/3955966857 wait complete. 2024-09-16T04:02:26.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.311+0000 7f3ea3c01640 10 monclient: build_initial_monmap 2024-09-16T04:02:26.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.311+0000 7f3ea3c01640 10 monclient: monmap: 2024-09-16T04:02:26.328 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:02:26.328 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:02:26.328 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:02:26.328 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:02:26.328 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:02:26.328 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:02:26.328 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:02:26.329 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:02:26.329 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:02:26.329 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:02:26.329 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.311+0000 7f3ea3c01640 1 librados: starting msgr at 2024-09-16T04:02:26.329 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.311+0000 7f3ea3c01640 1 librados: starting objecter 2024-09-16T04:02:26.329 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.311+0000 7f3ea3c01640 1 Processor -- start 2024-09-16T04:02:26.329 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.311+0000 7f3ea3c01640 1 -- start start 2024-09-16T04:02:26.329 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.311+0000 7f3ea3c01640 1 librados: setting wanted keys 2024-09-16T04:02:26.329 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.311+0000 7f3ea3c01640 1 librados: calling monclient init 2024-09-16T04:02:26.329 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.311+0000 7f3ea3c01640 10 monclient: init 2024-09-16T04:02:26.329 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.315+0000 7f3ea3c01640 10 monclient: _reopen_session rank -1 2024-09-16T04:02:26.329 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.315+0000 7f3ea3c01640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:02:26.329 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.315+0000 7f3ea3c01640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3e9c0a39b0 0x7f3e9c130560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:26.329 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.315+0000 7f3ea3c01640 10 monclient(hunting): picked mon.noname-b con 0x7f3e9c0a39b0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:02:26.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.315+0000 7f3ea3c01640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3e9c0a3d90 0x7f3e9c130aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:26.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.315+0000 7f3ea3c01640 10 monclient(hunting): picked mon.noname-c con 0x7f3e9c0a3d90 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:02:26.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.315+0000 7f3ea3c01640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3e9c137540 0x7f3e9c13cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:26.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.315+0000 7f3ea3c01640 10 monclient(hunting): picked mon.noname-a con 0x7f3e9c137540 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:02:26.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.315+0000 7f3ea3c01640 10 monclient(hunting): start opening mon connection 2024-09-16T04:02:26.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.315+0000 7f3ea3c01640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f3e9c1424e0 con 0x7f3e9c137540 2024-09-16T04:02:26.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.315+0000 7f3ea3c01640 10 monclient(hunting): start opening mon connection 2024-09-16T04:02:26.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.315+0000 7f3ea3c01640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f3e9c142660 con 0x7f3e9c0a39b0 2024-09-16T04:02:26.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.315+0000 7f3ea3c01640 10 monclient(hunting): start opening mon connection 2024-09-16T04:02:26.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.315+0000 7f3ea3c01640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f3e9c1427e0 con 0x7f3e9c0a3d90 2024-09-16T04:02:26.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.315+0000 7f3ea3c01640 10 monclient(hunting): _renew_subs 2024-09-16T04:02:26.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.315+0000 7f3ea3c01640 10 monclient(hunting): authenticate will time out at 2193.284201s 2024-09-16T04:02:26.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.315+0000 7f3ea119c640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3e9c0a3d90 0x7f3e9c130aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:02:26.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.315+0000 7f3ea219e640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3e9c137540 0x7f3e9c13cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:02:26.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.315+0000 7f3ea119c640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3e9c0a3d90 0x7f3e9c130aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:36238/0 (socket says 172.21.15.42:36238) 2024-09-16T04:02:26.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.315+0000 7f3ea119c640 1 -- 172.21.15.42:0/3288904204 learned_addr learned my addr 172.21.15.42:0/3288904204 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:02:26.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.315+0000 7f3ea119c640 10 monclient(hunting): get_auth_request con 0x7f3e9c0a3d90 auth_method 0 2024-09-16T04:02:26.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.315+0000 7f3ea119c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:02:26.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.315+0000 7f3ea119c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:02:26.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.315+0000 7f3ea119c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:02:26.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.315+0000 7f3ea219e640 10 monclient(hunting): get_auth_request con 0x7f3e9c137540 auth_method 0 2024-09-16T04:02:26.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.315+0000 7f3ea219e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:02:26.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.315+0000 7f3ea219e640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:02:26.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.315+0000 7f3ea219e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:02:26.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.315+0000 7f3ea119c640 1 -- 172.21.15.42:0/3288904204 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3e9c0a3d90 msgr2=0x7f3e9c130aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-16T04:02:26.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.315+0000 7f3ea119c640 1 -- 172.21.15.42:0/3288904204 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3e9c0a3d90 msgr2=0x7f3e9c130aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:02:26.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.315+0000 7f3ea119c640 1 --2- 172.21.15.42:0/3288904204 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3e9c0a3d90 0x7f3e9c130aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:02:26.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.315+0000 7f3ea219e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:02:26.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.315+0000 7f3ea219e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:02:26.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.315+0000 7f3ea219e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:02:26.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.315+0000 7f3ea119c640 1 --2- 172.21.15.42:0/3288904204 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3e9c0a3d90 0x7f3e9c130aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:02:26.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.315+0000 7f3ea219e640 10 monclient(hunting): handle_auth_done global_id 825495 payload 824 2024-09-16T04:02:26.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.315+0000 7f3ea219e640 1 -- 172.21.15.42:0/3288904204 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3e9c0a3d90 msgr2=0x7f3e9c130aa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:02:26.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.315+0000 7f3ea219e640 1 --2- 172.21.15.42:0/3288904204 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3e9c0a3d90 0x7f3e9c130aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:26.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.315+0000 7f3ea219e640 1 -- 172.21.15.42:0/3288904204 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3e9c0a39b0 msgr2=0x7f3e9c130560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:02:26.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.315+0000 7f3ea219e640 1 --2- 172.21.15.42:0/3288904204 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3e9c0a39b0 0x7f3e9c130560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:26.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.315+0000 7f3ea219e640 10 monclient: _finish_hunting 0 2024-09-16T04:02:26.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.315+0000 7f3ea219e640 1 monclient: found mon.noname-a 2024-09-16T04:02:26.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.315+0000 7f3ea219e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:02:26.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.315+0000 7f3ea219e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:02:26.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.315+0000 7f3ea219e640 1 -- 172.21.15.42:0/3288904204 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e9c09e4a0 con 0x7f3e9c137540 2024-09-16T04:02:26.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.315+0000 7f3ea219e640 1 --2- 172.21.15.42:0/3288904204 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3e9c137540 0x7f3e9c13cd80 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f3e9400c6e0 tx=0x7f3e9400cba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:02:26.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.315+0000 7f3e927fc640 10 client.?.objecter ms_handle_connect 0x7f3e9c137540 2024-09-16T04:02:26.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.315+0000 7f3e927fc640 10 client.?.objecter resend_mon_ops 2024-09-16T04:02:26.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.315+0000 7f3e927fc640 1 -- 172.21.15.42:0/3288904204 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e94018070 con 0x7f3e9c137540 2024-09-16T04:02:26.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.315+0000 7f3e927fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:02:26.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.315+0000 7f3e927fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:02:26.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.315+0000 7f3e927fc640 10 monclient: dump: 2024-09-16T04:02:26.333 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:02:26.333 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:02:26.333 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:26.333 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:26.333 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:02:26.333 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:02:26.334 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:02:26.334 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:02:26.334 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:02:26.334 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:02:26.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.315+0000 7f3e927fc640 10 monclient: _finish_auth 0 2024-09-16T04:02:26.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.315+0000 7f3e927fc640 10 monclient: _check_auth_tickets 2024-09-16T04:02:26.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.315+0000 7f3e927fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:02:26.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.315+0000 7f3e927fc640 1 -- 172.21.15.42:0/3288904204 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e94004d90 con 0x7f3e9c137540 2024-09-16T04:02:26.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.315+0000 7f3ea3c01640 5 monclient: authenticate success, global_id 825495 2024-09-16T04:02:26.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.315+0000 7f3ea3c01640 10 monclient: _renew_subs 2024-09-16T04:02:26.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.315+0000 7f3ea3c01640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:02:26.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.315+0000 7f3ea3c01640 1 -- 172.21.15.42:0/3288904204 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e9c137240 con 0x7f3e9c137540 2024-09-16T04:02:26.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.315+0000 7f3e927fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:02:26.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.315+0000 7f3e927fc640 1 -- 172.21.15.42:0/3288904204 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e9400a880 con 0x7f3e9c137540 2024-09-16T04:02:26.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.315+0000 7f3e927fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:02:26.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.315+0000 7f3e927fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:02:26.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.315+0000 7f3e927fc640 10 monclient: dump: 2024-09-16T04:02:26.335 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:02:26.335 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:02:26.335 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:26.335 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:26.335 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:02:26.335 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:02:26.335 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:02:26.335 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:02:26.335 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:02:26.335 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:02:26.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.315+0000 7f3e93fff640 4 set_mon_vals no callback set 2024-09-16T04:02:26.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.315+0000 7f3ea3c01640 10 client.825495.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:02:26.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.315+0000 7f3ea3c01640 10 monclient: _renew_subs 2024-09-16T04:02:26.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.315+0000 7f3ea3c01640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:02:26.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.315+0000 7f3ea3c01640 1 -- 172.21.15.42:0/3288904204 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e9c13f430 con 0x7f3e9c137540 2024-09-16T04:02:26.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.315+0000 7f3ea3c01640 1 librados: init done 2024-09-16T04:02:26.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.315+0000 7f3e927fc640 1 -- 172.21.15.42:0/3288904204 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f3e9401b070 con 0x7f3e9c137540 2024-09-16T04:02:26.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.315+0000 7f3ea3c01640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:02:26.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.315+0000 7f3ea3c01640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:02:26.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.315+0000 7f3ea3c01640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:02:26.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.315+0000 7f3ea3c01640 1 -- 172.21.15.42:0/3288904204 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3e9c133b80 con 0x7f3e9c137540 2024-09-16T04:02:26.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.315+0000 7f3e927fc640 1 --2- 172.21.15.42:0/3288904204 >> v2:172.21.15.154:6812/254845342 conn(0x7f3e80037840 0x7f3e80039d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:26.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.315+0000 7f3e927fc640 1 -- 172.21.15.42:0/3288904204 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(622..622 src has 451..622) ==== 6308+0+0 (secure 0 0 0) 0x7f3e94048630 con 0x7f3e9c137540 2024-09-16T04:02:26.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.315+0000 7f3e927fc640 10 client.825495.objecter ms_dispatch 0x7f3e9c000ba0 osd_map(622..622 src has 451..622) 2024-09-16T04:02:26.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.315+0000 7f3e927fc640 3 client.825495.objecter handle_osd_map got epochs [622,622] > 0 2024-09-16T04:02:26.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.315+0000 7f3e927fc640 3 client.825495.objecter handle_osd_map decoding full epoch 622 2024-09-16T04:02:26.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.315+0000 7f3e927fc640 20 client.825495.objecter dump_active .. 0 homeless 2024-09-16T04:02:26.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.315+0000 7f3ea199d640 1 --2- 172.21.15.42:0/3288904204 >> v2:172.21.15.154:6812/254845342 conn(0x7f3e80037840 0x7f3e80039d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:02:26.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.315+0000 7f3ea199d640 10 monclient: get_auth_request con 0x7f3e80037840 auth_method 0 2024-09-16T04:02:26.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.315+0000 7f3e927fc640 10 client.825495.objecter ms_handle_connect 0x7f3e80037840 2024-09-16T04:02:26.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.315+0000 7f3ea199d640 1 --2- 172.21.15.42:0/3288904204 >> v2:172.21.15.154:6812/254845342 conn(0x7f3e80037840 0x7f3e80039d00 secure :-1 s=READY pgs=3780 cs=0 l=1 rev1=1 crypto rx=0x7f3e980ad000 tx=0x7f3e98008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:02:26.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.319+0000 7f3e927fc640 1 -- 172.21.15.42:0/3288904204 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f3e9400ac00 con 0x7f3e9c137540 2024-09-16T04:02:26.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.319+0000 7f3e927fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:02:26.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.319+0000 7f3e927fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:02:26.560 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.555+0000 7f3ea3c01640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:02:26.560 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.555+0000 7f3ea3c01640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:02:26.560 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.555+0000 7f3ea3c01640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:02:26.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.555+0000 7f3ea3c01640 1 -- 172.21.15.42:0/3288904204 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f3e9c0a5750 con 0x7f3e9c137540 2024-09-16T04:02:26.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.555+0000 7f3e927fc640 1 -- 172.21.15.42:0/3288904204 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7f3e9401bbf0 con 0x7f3e9c137540 2024-09-16T04:02:26.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.555+0000 7f3e927fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:02:26.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.555+0000 7f3e927fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:02:26.561 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":182,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":21,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:02:26.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.559+0000 7f3ea3c01640 10 librados: watch_flush enter 2024-09-16T04:02:26.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.559+0000 7f3ea3c01640 10 librados: watch_flush exit 2024-09-16T04:02:26.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.559+0000 7f3ea3c01640 20 client.825495.objecter shutdown clearing up homeless session... 2024-09-16T04:02:26.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.559+0000 7f3ea3c01640 10 client.825495.objecter successfully canceled tick 2024-09-16T04:02:26.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.559+0000 7f3ea3c01640 1 -- 172.21.15.42:0/3288904204 >> v2:172.21.15.154:6812/254845342 conn(0x7f3e80037840 msgr2=0x7f3e80039d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:02:26.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.559+0000 7f3ea3c01640 1 --2- 172.21.15.42:0/3288904204 >> v2:172.21.15.154:6812/254845342 conn(0x7f3e80037840 0x7f3e80039d00 secure :-1 s=READY pgs=3780 cs=0 l=1 rev1=1 crypto rx=0x7f3e980ad000 tx=0x7f3e98008040 comp rx=0 tx=0).stop 2024-09-16T04:02:26.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.559+0000 7f3ea3c01640 10 monclient: shutdown 2024-09-16T04:02:26.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.559+0000 7f3ea3c01640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:02:26.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.559+0000 7f3ea3c01640 1 -- 172.21.15.42:0/3288904204 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3e9c137540 msgr2=0x7f3e9c13cd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:02:26.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.559+0000 7f3ea3c01640 1 --2- 172.21.15.42:0/3288904204 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3e9c137540 0x7f3e9c13cd80 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f3e9400c6e0 tx=0x7f3e9400cba0 comp rx=0 tx=0).stop 2024-09-16T04:02:26.565 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.559+0000 7f3ea3c01640 1 -- 172.21.15.42:0/3288904204 shutdown_connections 2024-09-16T04:02:26.565 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.559+0000 7f3ea3c01640 1 --2- 172.21.15.42:0/3288904204 >> v2:172.21.15.154:6812/254845342 conn(0x7f3e80037840 0x7f3e80039d00 unknown :-1 s=CLOSED pgs=3780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:26.565 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.559+0000 7f3ea3c01640 1 --2- 172.21.15.42:0/3288904204 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3e9c137540 0x7f3e9c13cd80 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:26.565 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.559+0000 7f3ea3c01640 1 --2- 172.21.15.42:0/3288904204 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3e9c0a3d90 0x7f3e9c130aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:26.565 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.559+0000 7f3ea3c01640 1 --2- 172.21.15.42:0/3288904204 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3e9c0a39b0 0x7f3e9c130560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:26.565 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.559+0000 7f3ea3c01640 1 -- 172.21.15.42:0/3288904204 >> 172.21.15.42:0/3288904204 conn(0x7f3e9c0a9fb0 msgr2=0x7f3e9c0a7a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:02:26.565 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.559+0000 7f3ea3c01640 1 -- 172.21.15.42:0/3288904204 shutdown_connections 2024-09-16T04:02:26.565 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.559+0000 7f3ea3c01640 1 -- 172.21.15.42:0/3288904204 wait complete. 2024-09-16T04:02:26.565 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:26.559+0000 7f3ea3c01640 1 librados: shutdown 2024-09-16T04:02:29.585 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:02:29.719 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.715+0000 7f4c987c2640 10 monclient: get_monmap_and_config 2024-09-16T04:02:29.719 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.715+0000 7f4c987c2640 10 monclient: build_initial_monmap 2024-09-16T04:02:29.719 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.715+0000 7f4c987c2640 10 monclient: monmap: 2024-09-16T04:02:29.719 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:02:29.719 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:02:29.719 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:02:29.717609+0000 2024-09-16T04:02:29.719 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:02:29.717609+0000 2024-09-16T04:02:29.719 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:02:29.719 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:02:29.720 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:02:29.720 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:02:29.720 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:02:29.720 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:02:29.720 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:02:29.720 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:02:29.722 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.715+0000 7f4c987c2640 1 Processor -- start 2024-09-16T04:02:29.722 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.715+0000 7f4c987c2640 1 -- start start 2024-09-16T04:02:29.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.715+0000 7f4c987c2640 10 monclient: init 2024-09-16T04:02:29.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.715+0000 7f4c987c2640 10 monclient: _reopen_session rank -1 2024-09-16T04:02:29.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.715+0000 7f4c987c2640 10 monclient: _add_conns ranks=[1,4,2,0,3] 2024-09-16T04:02:29.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.715+0000 7f4c987c2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4c900a1330 addr v1:172.21.15.154:6789/0 2024-09-16T04:02:29.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.715+0000 7f4c987c2640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f4c9006dac0 0x7f4c9013a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:29.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.715+0000 7f4c987c2640 10 monclient(hunting): picked mon.noname-c con 0x7f4c9006dac0 addr v2:172.21.15.154:3300/0 2024-09-16T04:02:29.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.715+0000 7f4c987c2640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4c900a1e20 0x7f4c9013c890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:29.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.715+0000 7f4c987c2640 10 monclient(hunting): picked mon.noname-b con 0x7f4c900a1e20 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:02:29.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.715+0000 7f4c987c2640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4c9009c090 con 0x7f4c900a1330 2024-09-16T04:02:29.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.715+0000 7f4c987c2640 10 monclient(hunting): start opening mon connection 2024-09-16T04:02:29.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.715+0000 7f4c987c2640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f4c9009c890 con 0x7f4c9006dac0 2024-09-16T04:02:29.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.715+0000 7f4c987c2640 10 monclient(hunting): start opening mon connection 2024-09-16T04:02:29.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.715+0000 7f4c987c2640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f4c9009f610 con 0x7f4c900a1e20 2024-09-16T04:02:29.724 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.715+0000 7f4c987c2640 10 monclient(hunting): _renew_subs 2024-09-16T04:02:29.724 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.715+0000 7f4c987c2640 10 monclient(hunting): authenticate will time out at 2196.687267s 2024-09-16T04:02:29.724 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.715+0000 7f4c96d5f640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4c900a1e20 0x7f4c9013c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:02:29.724 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.715+0000 7f4c96d5f640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4c900a1e20 0x7f4c9013c890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:36250/0 (socket says 172.21.15.42:36250) 2024-09-16T04:02:29.724 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.715+0000 7f4c96d5f640 1 -- 172.21.15.42:0/2397978909 learned_addr learned my addr 172.21.15.42:0/2397978909 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:02:29.724 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.715+0000 7f4c96d5f640 10 monclient(hunting): get_auth_request con 0x7f4c900a1e20 auth_method 0 2024-09-16T04:02:29.724 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.715+0000 7f4c96d5f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:02:29.724 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.715+0000 7f4c96d5f640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:02:29.724 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.715+0000 7f4c96d5f640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:02:29.724 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.715+0000 7f4c96d5f640 1 -- 172.21.15.42:0/2397978909 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4c900a1e20 msgr2=0x7f4c9013c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T04:02:29.724 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.715+0000 7f4c96d5f640 1 -- 172.21.15.42:0/2397978909 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4c900a1e20 msgr2=0x7f4c9013c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:02:29.724 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.715+0000 7f4c96d5f640 1 --2- 172.21.15.42:0/2397978909 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4c900a1e20 0x7f4c9013c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:02:29.725 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.715+0000 7f4c96d5f640 1 --2- 172.21.15.42:0/2397978909 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4c900a1e20 0x7f4c9013c890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:02:29.725 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.715+0000 7f4c95d5d640 1 --2- 172.21.15.42:0/2397978909 >> v2:172.21.15.154:3300/0 conn(0x7f4c9006dac0 0x7f4c9013a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:02:29.725 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.715+0000 7f4c95d5d640 10 monclient(hunting): get_auth_request con 0x7f4c9006dac0 auth_method 0 2024-09-16T04:02:29.725 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.715+0000 7f4c95d5d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:02:29.725 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.715+0000 7f4c95d5d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:02:29.725 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.715+0000 7f4c95d5d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:02:29.725 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.715+0000 7f4c9655e640 1 -- 172.21.15.42:0/2397978909 >> v1:172.21.15.154:6789/0 conn(0x7f4c900a1330 legacy=0x7f4c900a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:02:29.725 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.715+0000 7f4c9655e640 1 -- 172.21.15.42:0/2397978909 >> v1:172.21.15.154:6789/0 conn(0x7f4c900a1330 legacy=0x7f4c900a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:02:29.725 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.715+0000 7f4c9655e640 1 --1- 172.21.15.42:0/2397978909 >> v1:172.21.15.154:6789/0 conn(0x7f4c900a1330 0x7f4c900a1710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify read banner and identify addresses failed 2024-09-16T04:02:29.725 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.715+0000 7f4c95d5d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:02:29.725 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.715+0000 7f4c95d5d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:02:29.725 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.715+0000 7f4c95d5d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:02:29.725 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.719+0000 7f4c95d5d640 10 monclient(hunting): handle_auth_done global_id 904585 payload 274 2024-09-16T04:02:29.725 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.719+0000 7f4c95d5d640 1 -- 172.21.15.42:0/2397978909 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4c900a1e20 msgr2=0x7f4c9013c890 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:02:29.726 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.719+0000 7f4c95d5d640 1 --2- 172.21.15.42:0/2397978909 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4c900a1e20 0x7f4c9013c890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:29.726 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.719+0000 7f4c95d5d640 1 -- 172.21.15.42:0/2397978909 >> v1:172.21.15.154:6789/0 conn(0x7f4c900a1330 legacy=0x7f4c900a1710 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:02:29.726 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.719+0000 7f4c95d5d640 10 monclient: _finish_hunting 0 2024-09-16T04:02:29.726 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.719+0000 7f4c95d5d640 1 monclient: found mon.noname-c 2024-09-16T04:02:29.726 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.719+0000 7f4c95d5d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:02:29.726 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.719+0000 7f4c95d5d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.154:3300/0 2024-09-16T04:02:29.726 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.719+0000 7f4c95d5d640 1 -- 172.21.15.42:0/2397978909 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c9009f8f0 con 0x7f4c9006dac0 2024-09-16T04:02:29.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.719+0000 7f4c95d5d640 1 --2- 172.21.15.42:0/2397978909 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4c9006dac0 0x7f4c9013a340 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f4c840097c0 tx=0x7f4c8401c380 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1ea51105536d1d2d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:02:29.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.719+0000 7f4c9555c640 1 -- 172.21.15.42:0/2397978909 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c840b2070 con 0x7f4c9006dac0 2024-09-16T04:02:29.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.719+0000 7f4c9555c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:02:29.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.719+0000 7f4c9555c640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:02:29.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.719+0000 7f4c9555c640 10 monclient: dump: 2024-09-16T04:02:29.731 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:02:29.731 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:02:29.731 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:29.731 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:29.731 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:02:29.732 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:02:29.732 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:02:29.732 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:02:29.732 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:02:29.732 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:02:29.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.719+0000 7f4c9555c640 10 monclient: _finish_auth 0 2024-09-16T04:02:29.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.719+0000 7f4c9555c640 10 monclient: _check_auth_tickets 2024-09-16T04:02:29.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.719+0000 7f4c9555c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:02:29.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.719+0000 7f4c9555c640 1 -- 172.21.15.42:0/2397978909 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4c840a9070 con 0x7f4c9006dac0 2024-09-16T04:02:29.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.719+0000 7f4c9555c640 10 monclient: handle_config config(0 keys) 2024-09-16T04:02:29.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.719+0000 7f4c9555c640 1 -- 172.21.15.42:0/2397978909 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c840ad880 con 0x7f4c9006dac0 2024-09-16T04:02:29.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.719+0000 7f4c987c2640 5 monclient: authenticate success, global_id 904585 2024-09-16T04:02:29.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.719+0000 7f4c987c2640 10 monclient: get_monmap_and_config success 2024-09-16T04:02:29.733 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.719+0000 7f4c9555c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:02:29.733 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.719+0000 7f4c987c2640 4 set_mon_vals no callback set 2024-09-16T04:02:29.733 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.719+0000 7f4c9555c640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:02:29.733 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.719+0000 7f4c9555c640 10 monclient: dump: 2024-09-16T04:02:29.733 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:02:29.733 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:02:29.733 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:29.733 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:29.734 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:02:29.734 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:02:29.734 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:02:29.734 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:02:29.734 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:02:29.734 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:02:29.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.723+0000 7f4c987c2640 10 monclient: shutdown 2024-09-16T04:02:29.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.723+0000 7f4c987c2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:02:29.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.723+0000 7f4c987c2640 1 -- 172.21.15.42:0/2397978909 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4c9006dac0 msgr2=0x7f4c9013a340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:02:29.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.723+0000 7f4c987c2640 1 --2- 172.21.15.42:0/2397978909 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4c9006dac0 0x7f4c9013a340 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f4c840097c0 tx=0x7f4c8401c380 comp rx=0 tx=0).stop 2024-09-16T04:02:29.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.723+0000 7f4c987c2640 1 -- 172.21.15.42:0/2397978909 shutdown_connections 2024-09-16T04:02:29.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.723+0000 7f4c987c2640 1 --2- 172.21.15.42:0/2397978909 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4c900a1e20 0x7f4c9013c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:29.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.723+0000 7f4c987c2640 1 --2- 172.21.15.42:0/2397978909 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4c9006dac0 0x7f4c9013a340 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:29.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.723+0000 7f4c987c2640 1 -- 172.21.15.42:0/2397978909 >> 172.21.15.42:0/2397978909 conn(0x7f4c900a2420 msgr2=0x7f4c900a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:02:29.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.723+0000 7f4c987c2640 1 -- 172.21.15.42:0/2397978909 shutdown_connections 2024-09-16T04:02:29.736 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.723+0000 7f4c987c2640 1 -- 172.21.15.42:0/2397978909 wait complete. 2024-09-16T04:02:29.736 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.723+0000 7f4c987c2640 10 monclient: build_initial_monmap 2024-09-16T04:02:29.736 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.723+0000 7f4c987c2640 10 monclient: monmap: 2024-09-16T04:02:29.736 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:02:29.736 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:02:29.736 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:02:29.736 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:02:29.736 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:02:29.736 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:02:29.737 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:02:29.737 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:02:29.737 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:02:29.737 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:02:29.737 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.723+0000 7f4c987c2640 1 librados: starting msgr at 2024-09-16T04:02:29.737 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.723+0000 7f4c987c2640 1 librados: starting objecter 2024-09-16T04:02:29.737 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.723+0000 7f4c987c2640 1 Processor -- start 2024-09-16T04:02:29.737 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.723+0000 7f4c987c2640 1 -- start start 2024-09-16T04:02:29.737 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.723+0000 7f4c987c2640 1 librados: setting wanted keys 2024-09-16T04:02:29.737 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.723+0000 7f4c987c2640 1 librados: calling monclient init 2024-09-16T04:02:29.737 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.723+0000 7f4c987c2640 10 monclient: init 2024-09-16T04:02:29.737 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.723+0000 7f4c987c2640 10 monclient: _reopen_session rank -1 2024-09-16T04:02:29.737 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.723+0000 7f4c987c2640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:02:29.738 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.723+0000 7f4c987c2640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4c900a1330 0x7f4c901ccd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:29.738 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.723+0000 7f4c987c2640 10 monclient(hunting): picked mon.noname-c con 0x7f4c900a1330 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:02:29.738 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.723+0000 7f4c987c2640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4c900a1e20 0x7f4c90132550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:29.738 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.723+0000 7f4c987c2640 10 monclient(hunting): picked mon.noname-b con 0x7f4c900a1e20 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:02:29.738 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.723+0000 7f4c987c2640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4c90132a90 0x7f4c90139650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:29.738 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.723+0000 7f4c987c2640 10 monclient(hunting): picked mon.noname-a con 0x7f4c90132a90 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:02:29.738 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.723+0000 7f4c987c2640 10 monclient(hunting): start opening mon connection 2024-09-16T04:02:29.738 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.723+0000 7f4c987c2640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f4c9013f770 con 0x7f4c90132a90 2024-09-16T04:02:29.738 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.723+0000 7f4c987c2640 10 monclient(hunting): start opening mon connection 2024-09-16T04:02:29.738 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.723+0000 7f4c987c2640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f4c9013f5f0 con 0x7f4c900a1e20 2024-09-16T04:02:29.738 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.723+0000 7f4c987c2640 10 monclient(hunting): start opening mon connection 2024-09-16T04:02:29.738 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.723+0000 7f4c987c2640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f4c9013f8f0 con 0x7f4c900a1330 2024-09-16T04:02:29.738 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.723+0000 7f4c987c2640 10 monclient(hunting): _renew_subs 2024-09-16T04:02:29.738 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.723+0000 7f4c987c2640 10 monclient(hunting): authenticate will time out at 2196.694474s 2024-09-16T04:02:29.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.723+0000 7f4c9655e640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4c900a1330 0x7f4c901ccd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:02:29.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.723+0000 7f4c96d5f640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4c90132a90 0x7f4c90139650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:02:29.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.723+0000 7f4c96d5f640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4c90132a90 0x7f4c90139650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:42568/0 (socket says 172.21.15.42:42568) 2024-09-16T04:02:29.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.723+0000 7f4c96d5f640 1 -- 172.21.15.42:0/3426924444 learned_addr learned my addr 172.21.15.42:0/3426924444 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:02:29.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.723+0000 7f4c96d5f640 10 monclient(hunting): get_auth_request con 0x7f4c90132a90 auth_method 0 2024-09-16T04:02:29.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.723+0000 7f4c96d5f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:02:29.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.723+0000 7f4c96d5f640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:02:29.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.723+0000 7f4c96d5f640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:02:29.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.723+0000 7f4c9655e640 10 monclient(hunting): get_auth_request con 0x7f4c900a1330 auth_method 0 2024-09-16T04:02:29.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.723+0000 7f4c9655e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:02:29.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.723+0000 7f4c9655e640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:02:29.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.723+0000 7f4c9655e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:02:29.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.723+0000 7f4c96d5f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:02:29.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.723+0000 7f4c96d5f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:02:29.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.723+0000 7f4c9655e640 1 -- 172.21.15.42:0/3426924444 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4c900a1330 msgr2=0x7f4c901ccd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-16T04:02:29.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.723+0000 7f4c9655e640 1 -- 172.21.15.42:0/3426924444 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4c900a1330 msgr2=0x7f4c901ccd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:02:29.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.723+0000 7f4c96d5f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:02:29.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.723+0000 7f4c9655e640 1 --2- 172.21.15.42:0/3426924444 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4c900a1330 0x7f4c901ccd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:02:29.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.723+0000 7f4c9655e640 1 --2- 172.21.15.42:0/3426924444 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4c900a1330 0x7f4c901ccd40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:02:29.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.723+0000 7f4c95d5d640 1 --2- 172.21.15.42:0/3426924444 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4c900a1e20 0x7f4c90132550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:02:29.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.723+0000 7f4c95d5d640 10 monclient(hunting): get_auth_request con 0x7f4c900a1e20 auth_method 0 2024-09-16T04:02:29.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.723+0000 7f4c95d5d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:02:29.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.723+0000 7f4c95d5d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:02:29.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.723+0000 7f4c95d5d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:02:29.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.723+0000 7f4c95d5d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:02:29.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.723+0000 7f4c95d5d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:02:29.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.723+0000 7f4c95d5d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:02:29.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.723+0000 7f4c96d5f640 10 monclient(hunting): handle_auth_done global_id 825498 payload 824 2024-09-16T04:02:29.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.723+0000 7f4c96d5f640 1 -- 172.21.15.42:0/3426924444 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4c900a1330 msgr2=0x7f4c901ccd40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:02:29.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.723+0000 7f4c96d5f640 1 --2- 172.21.15.42:0/3426924444 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4c900a1330 0x7f4c901ccd40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:29.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.723+0000 7f4c96d5f640 1 -- 172.21.15.42:0/3426924444 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4c900a1e20 msgr2=0x7f4c90132550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:02:29.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.723+0000 7f4c96d5f640 1 --2- 172.21.15.42:0/3426924444 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4c900a1e20 0x7f4c90132550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:29.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.723+0000 7f4c96d5f640 10 monclient: _finish_hunting 0 2024-09-16T04:02:29.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.723+0000 7f4c96d5f640 1 monclient: found mon.noname-a 2024-09-16T04:02:29.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.723+0000 7f4c96d5f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:02:29.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.723+0000 7f4c96d5f640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:02:29.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.723+0000 7f4c96d5f640 1 -- 172.21.15.42:0/3426924444 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c9011c940 con 0x7f4c90132a90 2024-09-16T04:02:29.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.723+0000 7f4c96d5f640 1 --2- 172.21.15.42:0/3426924444 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4c90132a90 0x7f4c90139650 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f4c8000d5f0 tx=0x7f4c8000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:02:29.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.723+0000 7f4c8effd640 10 client.?.objecter ms_handle_connect 0x7f4c90132a90 2024-09-16T04:02:29.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.723+0000 7f4c8effd640 10 client.?.objecter resend_mon_ops 2024-09-16T04:02:29.742 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.723+0000 7f4c8effd640 1 -- 172.21.15.42:0/3426924444 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c80018030 con 0x7f4c90132a90 2024-09-16T04:02:29.742 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.723+0000 7f4c8effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:02:29.742 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.723+0000 7f4c8effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:02:29.742 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.723+0000 7f4c8effd640 10 monclient: dump: 2024-09-16T04:02:29.742 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:02:29.742 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:02:29.742 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:29.742 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:29.742 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:02:29.742 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:02:29.742 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:02:29.742 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:02:29.742 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:02:29.742 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:02:29.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.723+0000 7f4c8effd640 10 monclient: _finish_auth 0 2024-09-16T04:02:29.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.723+0000 7f4c8effd640 10 monclient: _check_auth_tickets 2024-09-16T04:02:29.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.723+0000 7f4c8effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:02:29.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.723+0000 7f4c8effd640 1 -- 172.21.15.42:0/3426924444 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4c80004020 con 0x7f4c90132a90 2024-09-16T04:02:29.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.723+0000 7f4c987c2640 5 monclient: authenticate success, global_id 825498 2024-09-16T04:02:29.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.723+0000 7f4c987c2640 10 monclient: _renew_subs 2024-09-16T04:02:29.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.723+0000 7f4c987c2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:02:29.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.723+0000 7f4c987c2640 1 -- 172.21.15.42:0/3426924444 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c9011cc20 con 0x7f4c90132a90 2024-09-16T04:02:29.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.723+0000 7f4c8effd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:02:29.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.723+0000 7f4c8effd640 1 -- 172.21.15.42:0/3426924444 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c80004210 con 0x7f4c90132a90 2024-09-16T04:02:29.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.723+0000 7f4c8effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:02:29.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.723+0000 7f4c8effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:02:29.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.723+0000 7f4c8effd640 10 monclient: dump: 2024-09-16T04:02:29.743 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:02:29.743 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:02:29.744 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:29.744 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:29.744 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:02:29.744 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:02:29.744 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:02:29.744 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:02:29.744 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:02:29.744 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:02:29.744 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.723+0000 7f4c987c2640 10 client.825498.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:02:29.744 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.723+0000 7f4c94d5b640 4 set_mon_vals no callback set 2024-09-16T04:02:29.744 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.723+0000 7f4c987c2640 10 monclient: _renew_subs 2024-09-16T04:02:29.744 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.723+0000 7f4c987c2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:02:29.744 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.723+0000 7f4c987c2640 1 -- 172.21.15.42:0/3426924444 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c9011cf00 con 0x7f4c90132a90 2024-09-16T04:02:29.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.723+0000 7f4c987c2640 1 librados: init done 2024-09-16T04:02:29.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.727+0000 7f4c987c2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:02:29.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.727+0000 7f4c987c2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:02:29.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.727+0000 7f4c987c2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:02:29.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.727+0000 7f4c987c2640 1 -- 172.21.15.42:0/3426924444 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4c900a1710 con 0x7f4c90132a90 2024-09-16T04:02:29.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.727+0000 7f4c8effd640 1 -- 172.21.15.42:0/3426924444 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f4c8001b070 con 0x7f4c90132a90 2024-09-16T04:02:29.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.727+0000 7f4c8effd640 1 --2- 172.21.15.42:0/3426924444 >> v2:172.21.15.154:6812/254845342 conn(0x7f4c7c037810 0x7f4c7c039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:29.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.727+0000 7f4c8effd640 1 -- 172.21.15.42:0/3426924444 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(626..626 src has 451..626) ==== 5899+0+0 (secure 0 0 0) 0x7f4c80049570 con 0x7f4c90132a90 2024-09-16T04:02:29.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.727+0000 7f4c8effd640 10 client.825498.objecter ms_dispatch 0x7f4c90000ba0 osd_map(626..626 src has 451..626) 2024-09-16T04:02:29.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.727+0000 7f4c8effd640 3 client.825498.objecter handle_osd_map got epochs [626,626] > 0 2024-09-16T04:02:29.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.727+0000 7f4c8effd640 3 client.825498.objecter handle_osd_map decoding full epoch 626 2024-09-16T04:02:29.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.727+0000 7f4c8effd640 20 client.825498.objecter dump_active .. 0 homeless 2024-09-16T04:02:29.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.727+0000 7f4c9655e640 1 --2- 172.21.15.42:0/3426924444 >> v2:172.21.15.154:6812/254845342 conn(0x7f4c7c037810 0x7f4c7c039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:02:29.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.727+0000 7f4c9655e640 10 monclient: get_auth_request con 0x7f4c7c037810 auth_method 0 2024-09-16T04:02:29.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.727+0000 7f4c9655e640 1 --2- 172.21.15.42:0/3426924444 >> v2:172.21.15.154:6812/254845342 conn(0x7f4c7c037810 0x7f4c7c039cd0 secure :-1 s=READY pgs=3781 cs=0 l=1 rev1=1 crypto rx=0x7f4c88001350 tx=0x7f4c88005b50 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:02:29.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.727+0000 7f4c8effd640 10 client.825498.objecter ms_handle_connect 0x7f4c7c037810 2024-09-16T04:02:29.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.731+0000 7f4c8effd640 1 -- 172.21.15.42:0/3426924444 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f4c80014620 con 0x7f4c90132a90 2024-09-16T04:02:29.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.731+0000 7f4c8effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:02:29.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.731+0000 7f4c8effd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:02:29.961 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.955+0000 7f4c987c2640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:02:29.961 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.955+0000 7f4c987c2640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:02:29.961 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.955+0000 7f4c987c2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:02:29.961 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.955+0000 7f4c987c2640 1 -- 172.21.15.42:0/3426924444 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f4c90069040 con 0x7f4c90132a90 2024-09-16T04:02:29.961 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.959+0000 7f4c8effd640 1 -- 172.21.15.42:0/3426924444 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7f4c80014d80 con 0x7f4c90132a90 2024-09-16T04:02:29.962 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.959+0000 7f4c8effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:02:29.962 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.959+0000 7f4c8effd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:02:29.962 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":182,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":24,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:02:29.964 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.959+0000 7f4c987c2640 10 librados: watch_flush enter 2024-09-16T04:02:29.964 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.959+0000 7f4c987c2640 10 librados: watch_flush exit 2024-09-16T04:02:29.964 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.959+0000 7f4c987c2640 20 client.825498.objecter shutdown clearing up homeless session... 2024-09-16T04:02:29.964 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.959+0000 7f4c987c2640 10 client.825498.objecter successfully canceled tick 2024-09-16T04:02:29.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.959+0000 7f4c987c2640 1 -- 172.21.15.42:0/3426924444 >> v2:172.21.15.154:6812/254845342 conn(0x7f4c7c037810 msgr2=0x7f4c7c039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:02:29.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.959+0000 7f4c987c2640 1 --2- 172.21.15.42:0/3426924444 >> v2:172.21.15.154:6812/254845342 conn(0x7f4c7c037810 0x7f4c7c039cd0 secure :-1 s=READY pgs=3781 cs=0 l=1 rev1=1 crypto rx=0x7f4c88001350 tx=0x7f4c88005b50 comp rx=0 tx=0).stop 2024-09-16T04:02:29.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.959+0000 7f4c987c2640 10 monclient: shutdown 2024-09-16T04:02:29.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.959+0000 7f4c987c2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:02:29.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.959+0000 7f4c987c2640 1 -- 172.21.15.42:0/3426924444 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4c90132a90 msgr2=0x7f4c90139650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:02:29.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.959+0000 7f4c987c2640 1 --2- 172.21.15.42:0/3426924444 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4c90132a90 0x7f4c90139650 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f4c8000d5f0 tx=0x7f4c8000dab0 comp rx=0 tx=0).stop 2024-09-16T04:02:29.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.959+0000 7f4c987c2640 1 -- 172.21.15.42:0/3426924444 shutdown_connections 2024-09-16T04:02:29.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.959+0000 7f4c987c2640 1 --2- 172.21.15.42:0/3426924444 >> v2:172.21.15.154:6812/254845342 conn(0x7f4c7c037810 0x7f4c7c039cd0 unknown :-1 s=CLOSED pgs=3781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:29.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.959+0000 7f4c987c2640 1 --2- 172.21.15.42:0/3426924444 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4c90132a90 0x7f4c90139650 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:29.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.959+0000 7f4c987c2640 1 --2- 172.21.15.42:0/3426924444 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4c900a1e20 0x7f4c90132550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:29.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.959+0000 7f4c987c2640 1 --2- 172.21.15.42:0/3426924444 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4c900a1330 0x7f4c901ccd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:29.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.959+0000 7f4c987c2640 1 -- 172.21.15.42:0/3426924444 >> 172.21.15.42:0/3426924444 conn(0x7f4c900a2420 msgr2=0x7f4c90135390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:02:29.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.959+0000 7f4c987c2640 1 -- 172.21.15.42:0/3426924444 shutdown_connections 2024-09-16T04:02:29.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.959+0000 7f4c987c2640 1 -- 172.21.15.42:0/3426924444 wait complete. 2024-09-16T04:02:29.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:29.959+0000 7f4c987c2640 1 librados: shutdown 2024-09-16T04:02:32.981 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:02:33.112 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.107+0000 7f8a68142640 10 monclient: get_monmap_and_config 2024-09-16T04:02:33.112 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.107+0000 7f8a68142640 10 monclient: build_initial_monmap 2024-09-16T04:02:33.112 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.107+0000 7f8a68142640 10 monclient: monmap: 2024-09-16T04:02:33.112 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:02:33.112 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:02:33.112 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:02:33.110453+0000 2024-09-16T04:02:33.113 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:02:33.110453+0000 2024-09-16T04:02:33.113 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:02:33.113 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:02:33.113 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:02:33.113 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:02:33.113 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:02:33.113 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:02:33.113 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:02:33.113 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:02:33.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.107+0000 7f8a68142640 1 Processor -- start 2024-09-16T04:02:33.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.107+0000 7f8a68142640 1 -- start start 2024-09-16T04:02:33.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.107+0000 7f8a68142640 10 monclient: init 2024-09-16T04:02:33.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.107+0000 7f8a68142640 10 monclient: _reopen_session rank -1 2024-09-16T04:02:33.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.107+0000 7f8a68142640 10 monclient: _add_conns ranks=[2,3,1,4,0] 2024-09-16T04:02:33.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.107+0000 7f8a68142640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f8a6006db90 0x7f8a601340c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:33.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.107+0000 7f8a68142640 10 monclient(hunting): picked mon.noname-b con 0x7f8a6006db90 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:02:33.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.107+0000 7f8a68142640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f8a600a37f0 0x7f8a600a3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:33.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.107+0000 7f8a68142640 10 monclient(hunting): picked mon.noname-a con 0x7f8a600a37f0 addr v2:172.21.15.42:3300/0 2024-09-16T04:02:33.125 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.111+0000 7f8a68142640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8a60136e10 addr v1:172.21.15.154:6789/0 2024-09-16T04:02:33.125 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.111+0000 7f8a68142640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8a6006ece0 con 0x7f8a60136e10 2024-09-16T04:02:33.125 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.111+0000 7f8a68142640 10 monclient(hunting): start opening mon connection 2024-09-16T04:02:33.125 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.111+0000 7f8a68142640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f8a600a4110 con 0x7f8a600a37f0 2024-09-16T04:02:33.125 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.111+0000 7f8a68142640 10 monclient(hunting): start opening mon connection 2024-09-16T04:02:33.125 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.111+0000 7f8a68142640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f8a6006df70 con 0x7f8a6006db90 2024-09-16T04:02:33.125 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.111+0000 7f8a68142640 10 monclient(hunting): _renew_subs 2024-09-16T04:02:33.125 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.111+0000 7f8a68142640 10 monclient(hunting): authenticate will time out at 2200.080079s 2024-09-16T04:02:33.125 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.111+0000 7f8a656dd640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f8a600a37f0 0x7f8a600a3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:02:33.125 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.111+0000 7f8a656dd640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f8a600a37f0 0x7f8a600a3bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:42580/0 (socket says 172.21.15.42:42580) 2024-09-16T04:02:33.125 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.111+0000 7f8a65ede640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f8a6006db90 0x7f8a601340c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:02:33.125 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.111+0000 7f8a656dd640 1 -- 172.21.15.42:0/547736908 learned_addr learned my addr 172.21.15.42:0/547736908 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:02:33.125 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.111+0000 7f8a656dd640 10 monclient(hunting): get_auth_request con 0x7f8a600a37f0 auth_method 0 2024-09-16T04:02:33.126 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.111+0000 7f8a656dd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:02:33.126 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.111+0000 7f8a656dd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:02:33.126 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.111+0000 7f8a656dd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:02:33.126 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.111+0000 7f8a65ede640 10 monclient(hunting): get_auth_request con 0x7f8a6006db90 auth_method 0 2024-09-16T04:02:33.126 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.111+0000 7f8a65ede640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:02:33.126 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.111+0000 7f8a65ede640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:02:33.126 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.111+0000 7f8a65ede640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:02:33.126 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.111+0000 7f8a656dd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:02:33.126 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.111+0000 7f8a656dd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:02:33.126 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.111+0000 7f8a656dd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:02:33.126 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.111+0000 7f8a65ede640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:02:33.126 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.111+0000 7f8a65ede640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:02:33.126 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.111+0000 7f8a65ede640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:02:33.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.111+0000 7f8a656dd640 10 monclient(hunting): handle_auth_done global_id 825501 payload 274 2024-09-16T04:02:33.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.111+0000 7f8a656dd640 1 -- 172.21.15.42:0/547736908 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f8a6006db90 msgr2=0x7f8a601340c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:02:33.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.111+0000 7f8a656dd640 1 --2- 172.21.15.42:0/547736908 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f8a6006db90 0x7f8a601340c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:33.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.111+0000 7f8a656dd640 1 -- 172.21.15.42:0/547736908 >> v1:172.21.15.154:6789/0 conn(0x7f8a60136e10 legacy=0x7f8a60137220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:02:33.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.111+0000 7f8a656dd640 10 monclient: _finish_hunting 0 2024-09-16T04:02:33.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.111+0000 7f8a656dd640 1 monclient: found mon.noname-a 2024-09-16T04:02:33.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.111+0000 7f8a656dd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:02:33.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.111+0000 7f8a656dd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:02:33.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.111+0000 7f8a656dd640 1 -- 172.21.15.42:0/547736908 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a6009bf80 con 0x7f8a600a37f0 2024-09-16T04:02:33.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.111+0000 7f8a65ede640 1 --2- 172.21.15.42:0/547736908 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f8a6006db90 0x7f8a601340c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:02:33.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.111+0000 7f8a656dd640 1 --2- 172.21.15.42:0/547736908 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8a600a37f0 0x7f8a600a3bd0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f8a5c009870 tx=0x7f8a5c01c420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a32f33d85341afc server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:02:33.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.111+0000 7f8a64edc640 1 -- 172.21.15.42:0/547736908 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a5c0b1070 con 0x7f8a600a37f0 2024-09-16T04:02:33.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.111+0000 7f8a64edc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:02:33.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.111+0000 7f8a64edc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:02:33.128 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.111+0000 7f8a64edc640 10 monclient: dump: 2024-09-16T04:02:33.128 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:02:33.128 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:02:33.128 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:33.128 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:33.128 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:02:33.128 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:02:33.128 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:02:33.128 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:02:33.128 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:02:33.128 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:02:33.128 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.111+0000 7f8a64edc640 10 monclient: _finish_auth 0 2024-09-16T04:02:33.128 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.111+0000 7f8a64edc640 10 monclient: _check_auth_tickets 2024-09-16T04:02:33.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.111+0000 7f8a64edc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:02:33.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.111+0000 7f8a64edc640 1 -- 172.21.15.42:0/547736908 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a5c0ac640 con 0x7f8a600a37f0 2024-09-16T04:02:33.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.111+0000 7f8a64edc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:02:33.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.111+0000 7f8a64edc640 1 -- 172.21.15.42:0/547736908 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a5c0aca10 con 0x7f8a600a37f0 2024-09-16T04:02:33.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.111+0000 7f8a68142640 5 monclient: authenticate success, global_id 825501 2024-09-16T04:02:33.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.111+0000 7f8a68142640 10 monclient: get_monmap_and_config success 2024-09-16T04:02:33.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.111+0000 7f8a68142640 4 set_mon_vals no callback set 2024-09-16T04:02:33.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.111+0000 7f8a64edc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:02:33.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.111+0000 7f8a64edc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:02:33.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.111+0000 7f8a64edc640 10 monclient: dump: 2024-09-16T04:02:33.129 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:02:33.129 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:02:33.129 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:33.129 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:33.130 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:02:33.130 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:02:33.130 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:02:33.130 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:02:33.130 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:02:33.130 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:02:33.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.111+0000 7f8a68142640 10 monclient: shutdown 2024-09-16T04:02:33.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.111+0000 7f8a68142640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:02:33.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.111+0000 7f8a68142640 1 -- 172.21.15.42:0/547736908 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8a600a37f0 msgr2=0x7f8a600a3bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:02:33.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.111+0000 7f8a68142640 1 --2- 172.21.15.42:0/547736908 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8a600a37f0 0x7f8a600a3bd0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f8a5c009870 tx=0x7f8a5c01c420 comp rx=0 tx=0).stop 2024-09-16T04:02:33.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.111+0000 7f8a68142640 1 -- 172.21.15.42:0/547736908 shutdown_connections 2024-09-16T04:02:33.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.111+0000 7f8a68142640 1 --2- 172.21.15.42:0/547736908 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8a600a37f0 0x7f8a600a3bd0 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:33.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.111+0000 7f8a68142640 1 --2- 172.21.15.42:0/547736908 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f8a6006db90 0x7f8a601340c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:33.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.111+0000 7f8a68142640 1 -- 172.21.15.42:0/547736908 >> 172.21.15.42:0/547736908 conn(0x7f8a600a9fb0 msgr2=0x7f8a600aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:02:33.131 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.111+0000 7f8a68142640 1 -- 172.21.15.42:0/547736908 shutdown_connections 2024-09-16T04:02:33.131 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.111+0000 7f8a68142640 1 -- 172.21.15.42:0/547736908 wait complete. 2024-09-16T04:02:33.131 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.115+0000 7f8a68142640 10 monclient: build_initial_monmap 2024-09-16T04:02:33.131 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.115+0000 7f8a68142640 10 monclient: monmap: 2024-09-16T04:02:33.131 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:02:33.131 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:02:33.131 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:02:33.131 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:02:33.131 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:02:33.131 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:02:33.131 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:02:33.131 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:02:33.131 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:02:33.132 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:02:33.132 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.115+0000 7f8a68142640 1 librados: starting msgr at 2024-09-16T04:02:33.132 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.115+0000 7f8a68142640 1 librados: starting objecter 2024-09-16T04:02:33.132 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.115+0000 7f8a68142640 1 Processor -- start 2024-09-16T04:02:33.132 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.115+0000 7f8a68142640 1 -- start start 2024-09-16T04:02:33.132 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.115+0000 7f8a68142640 1 librados: setting wanted keys 2024-09-16T04:02:33.132 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.115+0000 7f8a68142640 1 librados: calling monclient init 2024-09-16T04:02:33.132 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.115+0000 7f8a68142640 10 monclient: init 2024-09-16T04:02:33.132 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.115+0000 7f8a68142640 10 monclient: _reopen_session rank -1 2024-09-16T04:02:33.132 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.115+0000 7f8a68142640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:02:33.132 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.115+0000 7f8a68142640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8a6006db90 0x7f8a600a17a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:33.132 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.115+0000 7f8a68142640 10 monclient(hunting): picked mon.noname-a con 0x7f8a6006db90 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:02:33.132 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.115+0000 7f8a68142640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f8a600a37f0 0x7f8a600a1ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:33.132 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.115+0000 7f8a68142640 10 monclient(hunting): picked mon.noname-c con 0x7f8a600a37f0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:02:33.133 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.115+0000 7f8a68142640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8a60136e10 0x7f8a600a2220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:33.133 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.115+0000 7f8a68142640 10 monclient(hunting): picked mon.noname-b con 0x7f8a60136e10 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:02:33.133 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.115+0000 7f8a68142640 10 monclient(hunting): start opening mon connection 2024-09-16T04:02:33.133 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.115+0000 7f8a68142640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f8a60141be0 con 0x7f8a6006db90 2024-09-16T04:02:33.133 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.115+0000 7f8a68142640 10 monclient(hunting): start opening mon connection 2024-09-16T04:02:33.133 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.115+0000 7f8a68142640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f8a6006eab0 con 0x7f8a60136e10 2024-09-16T04:02:33.133 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.115+0000 7f8a68142640 10 monclient(hunting): start opening mon connection 2024-09-16T04:02:33.133 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.115+0000 7f8a68142640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f8a60141d60 con 0x7f8a600a37f0 2024-09-16T04:02:33.133 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.115+0000 7f8a68142640 10 monclient(hunting): _renew_subs 2024-09-16T04:02:33.133 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.115+0000 7f8a68142640 10 monclient(hunting): authenticate will time out at 2200.085223s 2024-09-16T04:02:33.133 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.115+0000 7f8a65ede640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8a6006db90 0x7f8a600a17a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:02:33.133 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.115+0000 7f8a656dd640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f8a600a37f0 0x7f8a600a1ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:02:33.133 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.115+0000 7f8a65ede640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8a6006db90 0x7f8a600a17a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:42592/0 (socket says 172.21.15.42:42592) 2024-09-16T04:02:33.133 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.115+0000 7f8a65ede640 1 -- 172.21.15.42:0/2773256211 learned_addr learned my addr 172.21.15.42:0/2773256211 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:02:33.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.115+0000 7f8a656dd640 1 --2- 172.21.15.42:0/2773256211 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f8a600a37f0 0x7f8a600a1ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:36294/0 (socket says 172.21.15.42:36294) 2024-09-16T04:02:33.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.115+0000 7f8a65ede640 10 monclient(hunting): get_auth_request con 0x7f8a6006db90 auth_method 0 2024-09-16T04:02:33.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.115+0000 7f8a65ede640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:02:33.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.115+0000 7f8a65ede640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:02:33.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.115+0000 7f8a65ede640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:02:33.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.115+0000 7f8a656dd640 10 monclient(hunting): get_auth_request con 0x7f8a600a37f0 auth_method 0 2024-09-16T04:02:33.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.115+0000 7f8a656dd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:02:33.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.115+0000 7f8a656dd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:02:33.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.115+0000 7f8a656dd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:02:33.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.115+0000 7f8a65ede640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:02:33.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.115+0000 7f8a65ede640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:02:33.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.115+0000 7f8a65ede640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:02:33.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.115+0000 7f8a656dd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:02:33.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.115+0000 7f8a656dd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:02:33.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.115+0000 7f8a656dd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:02:33.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.115+0000 7f8a666df640 1 --2- 172.21.15.42:0/2773256211 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8a60136e10 0x7f8a600a2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:02:33.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.115+0000 7f8a666df640 10 monclient(hunting): get_auth_request con 0x7f8a60136e10 auth_method 0 2024-09-16T04:02:33.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.115+0000 7f8a666df640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:02:33.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.115+0000 7f8a666df640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:02:33.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.115+0000 7f8a666df640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:02:33.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.115+0000 7f8a65ede640 10 monclient(hunting): handle_auth_done global_id 825504 payload 824 2024-09-16T04:02:33.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.115+0000 7f8a65ede640 1 -- 172.21.15.42:0/2773256211 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f8a600a37f0 msgr2=0x7f8a600a1ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:02:33.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.115+0000 7f8a65ede640 1 --2- 172.21.15.42:0/2773256211 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f8a600a37f0 0x7f8a600a1ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:33.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.115+0000 7f8a65ede640 1 -- 172.21.15.42:0/2773256211 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8a60136e10 msgr2=0x7f8a600a2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:02:33.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.115+0000 7f8a65ede640 1 --2- 172.21.15.42:0/2773256211 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8a60136e10 0x7f8a600a2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:33.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.115+0000 7f8a65ede640 10 monclient: _finish_hunting 0 2024-09-16T04:02:33.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.115+0000 7f8a65ede640 1 monclient: found mon.noname-a 2024-09-16T04:02:33.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.115+0000 7f8a65ede640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:02:33.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.115+0000 7f8a65ede640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:02:33.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.115+0000 7f8a65ede640 1 -- 172.21.15.42:0/2773256211 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a60139b60 con 0x7f8a6006db90 2024-09-16T04:02:33.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.115+0000 7f8a656dd640 1 --2- 172.21.15.42:0/2773256211 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f8a600a37f0 0x7f8a600a1ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:02:33.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.115+0000 7f8a65ede640 1 --2- 172.21.15.42:0/2773256211 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8a6006db90 0x7f8a600a17a0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f8a5800efc0 tx=0x7f8a5800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:02:33.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.115+0000 7f8a567fc640 10 client.?.objecter ms_handle_connect 0x7f8a6006db90 2024-09-16T04:02:33.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.115+0000 7f8a567fc640 10 client.?.objecter resend_mon_ops 2024-09-16T04:02:33.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.115+0000 7f8a567fc640 1 -- 172.21.15.42:0/2773256211 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a58018020 con 0x7f8a6006db90 2024-09-16T04:02:33.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.115+0000 7f8a567fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:02:33.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.115+0000 7f8a567fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:02:33.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.115+0000 7f8a567fc640 10 monclient: dump: 2024-09-16T04:02:33.136 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:02:33.137 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:02:33.137 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:33.137 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:33.137 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:02:33.137 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:02:33.137 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:02:33.137 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:02:33.137 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:02:33.137 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:02:33.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.115+0000 7f8a567fc640 10 monclient: _finish_auth 0 2024-09-16T04:02:33.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.115+0000 7f8a567fc640 10 monclient: _check_auth_tickets 2024-09-16T04:02:33.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.115+0000 7f8a567fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:02:33.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.115+0000 7f8a567fc640 1 -- 172.21.15.42:0/2773256211 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a58014680 con 0x7f8a6006db90 2024-09-16T04:02:33.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.115+0000 7f8a68142640 5 monclient: authenticate success, global_id 825504 2024-09-16T04:02:33.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.115+0000 7f8a68142640 10 monclient: _renew_subs 2024-09-16T04:02:33.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.115+0000 7f8a68142640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:02:33.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.115+0000 7f8a68142640 1 -- 172.21.15.42:0/2773256211 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a60139e40 con 0x7f8a6006db90 2024-09-16T04:02:33.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.115+0000 7f8a567fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:02:33.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.115+0000 7f8a567fc640 1 -- 172.21.15.42:0/2773256211 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a58014a50 con 0x7f8a6006db90 2024-09-16T04:02:33.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.115+0000 7f8a567fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:02:33.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.115+0000 7f8a567fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:02:33.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.115+0000 7f8a567fc640 10 monclient: dump: 2024-09-16T04:02:33.138 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:02:33.138 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:02:33.138 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:33.138 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:33.138 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:02:33.139 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:02:33.139 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:02:33.139 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:02:33.139 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:02:33.139 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:02:33.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.115+0000 7f8a57fff640 4 set_mon_vals no callback set 2024-09-16T04:02:33.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.115+0000 7f8a68142640 10 client.825504.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:02:33.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.115+0000 7f8a68142640 10 monclient: _renew_subs 2024-09-16T04:02:33.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.115+0000 7f8a68142640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:02:33.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.115+0000 7f8a68142640 1 -- 172.21.15.42:0/2773256211 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a600a8aa0 con 0x7f8a6006db90 2024-09-16T04:02:33.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.115+0000 7f8a68142640 1 librados: init done 2024-09-16T04:02:33.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.115+0000 7f8a68142640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:02:33.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.115+0000 7f8a68142640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:02:33.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.115+0000 7f8a68142640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:02:33.140 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.115+0000 7f8a68142640 1 -- 172.21.15.42:0/2773256211 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a601340c0 con 0x7f8a6006db90 2024-09-16T04:02:33.140 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.115+0000 7f8a567fc640 1 -- 172.21.15.42:0/2773256211 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f8a5801b020 con 0x7f8a6006db90 2024-09-16T04:02:33.140 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.115+0000 7f8a567fc640 1 --2- 172.21.15.42:0/2773256211 >> v2:172.21.15.154:6812/254845342 conn(0x7f8a4c037810 0x7f8a4c039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:33.140 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.115+0000 7f8a567fc640 1 -- 172.21.15.42:0/2773256211 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(629..629 src has 451..629) ==== 6280+0+0 (secure 0 0 0) 0x7f8a58049de0 con 0x7f8a6006db90 2024-09-16T04:02:33.140 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.115+0000 7f8a567fc640 10 client.825504.objecter ms_dispatch 0x7f8a60000ba0 osd_map(629..629 src has 451..629) 2024-09-16T04:02:33.140 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.115+0000 7f8a567fc640 3 client.825504.objecter handle_osd_map got epochs [629,629] > 0 2024-09-16T04:02:33.140 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.115+0000 7f8a567fc640 3 client.825504.objecter handle_osd_map decoding full epoch 629 2024-09-16T04:02:33.140 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.115+0000 7f8a567fc640 20 client.825504.objecter dump_active .. 0 homeless 2024-09-16T04:02:33.140 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.115+0000 7f8a656dd640 1 --2- 172.21.15.42:0/2773256211 >> v2:172.21.15.154:6812/254845342 conn(0x7f8a4c037810 0x7f8a4c039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:02:33.140 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.115+0000 7f8a656dd640 10 monclient: get_auth_request con 0x7f8a4c037810 auth_method 0 2024-09-16T04:02:33.140 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.119+0000 7f8a567fc640 10 client.825504.objecter ms_handle_connect 0x7f8a4c037810 2024-09-16T04:02:33.140 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.119+0000 7f8a656dd640 1 --2- 172.21.15.42:0/2773256211 >> v2:172.21.15.154:6812/254845342 conn(0x7f8a4c037810 0x7f8a4c039cd0 secure :-1 s=READY pgs=3782 cs=0 l=1 rev1=1 crypto rx=0x7f8a5c0ae000 tx=0x7f8a5c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:02:33.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.123+0000 7f8a567fc640 1 -- 172.21.15.42:0/2773256211 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f8a58022030 con 0x7f8a6006db90 2024-09-16T04:02:33.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.123+0000 7f8a567fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:02:33.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.123+0000 7f8a567fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:02:33.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.347+0000 7f8a68142640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:02:33.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.347+0000 7f8a68142640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:02:33.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.347+0000 7f8a68142640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:02:33.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.347+0000 7f8a68142640 1 -- 172.21.15.42:0/2773256211 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f8a601398e0 con 0x7f8a6006db90 2024-09-16T04:02:33.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.351+0000 7f8a567fc640 1 -- 172.21.15.42:0/2773256211 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7f8a58010040 con 0x7f8a6006db90 2024-09-16T04:02:33.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.351+0000 7f8a567fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:02:33.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.351+0000 7f8a567fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:02:33.354 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":184,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:02:33.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.351+0000 7f8a68142640 10 librados: watch_flush enter 2024-09-16T04:02:33.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.351+0000 7f8a68142640 10 librados: watch_flush exit 2024-09-16T04:02:33.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.351+0000 7f8a68142640 20 client.825504.objecter shutdown clearing up homeless session... 2024-09-16T04:02:33.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.351+0000 7f8a68142640 10 client.825504.objecter successfully canceled tick 2024-09-16T04:02:33.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.351+0000 7f8a68142640 1 -- 172.21.15.42:0/2773256211 >> v2:172.21.15.154:6812/254845342 conn(0x7f8a4c037810 msgr2=0x7f8a4c039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:02:33.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.351+0000 7f8a68142640 1 --2- 172.21.15.42:0/2773256211 >> v2:172.21.15.154:6812/254845342 conn(0x7f8a4c037810 0x7f8a4c039cd0 secure :-1 s=READY pgs=3782 cs=0 l=1 rev1=1 crypto rx=0x7f8a5c0ae000 tx=0x7f8a5c008040 comp rx=0 tx=0).stop 2024-09-16T04:02:33.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.351+0000 7f8a68142640 10 monclient: shutdown 2024-09-16T04:02:33.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.351+0000 7f8a68142640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:02:33.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.351+0000 7f8a68142640 1 -- 172.21.15.42:0/2773256211 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8a6006db90 msgr2=0x7f8a600a17a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:02:33.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.351+0000 7f8a68142640 1 --2- 172.21.15.42:0/2773256211 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8a6006db90 0x7f8a600a17a0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f8a5800efc0 tx=0x7f8a5800c490 comp rx=0 tx=0).stop 2024-09-16T04:02:33.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.351+0000 7f8a68142640 1 -- 172.21.15.42:0/2773256211 shutdown_connections 2024-09-16T04:02:33.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.351+0000 7f8a68142640 1 --2- 172.21.15.42:0/2773256211 >> v2:172.21.15.154:6812/254845342 conn(0x7f8a4c037810 0x7f8a4c039cd0 unknown :-1 s=CLOSED pgs=3782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:33.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.351+0000 7f8a68142640 1 --2- 172.21.15.42:0/2773256211 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8a60136e10 0x7f8a600a2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:33.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.351+0000 7f8a68142640 1 --2- 172.21.15.42:0/2773256211 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f8a600a37f0 0x7f8a600a1ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:33.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.351+0000 7f8a68142640 1 --2- 172.21.15.42:0/2773256211 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8a6006db90 0x7f8a600a17a0 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:33.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.351+0000 7f8a68142640 1 -- 172.21.15.42:0/2773256211 >> 172.21.15.42:0/2773256211 conn(0x7f8a600a9fb0 msgr2=0x7f8a60131f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:02:33.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.351+0000 7f8a68142640 1 -- 172.21.15.42:0/2773256211 shutdown_connections 2024-09-16T04:02:33.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.351+0000 7f8a68142640 1 -- 172.21.15.42:0/2773256211 wait complete. 2024-09-16T04:02:33.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.351+0000 7f8a68142640 1 librados: shutdown 2024-09-16T04:02:33.373 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T04:02:33.373 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T04:02:33.373 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T04:02:33.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.491+0000 7f58db011640 10 monclient: get_monmap_and_config 2024-09-16T04:02:33.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.491+0000 7f58db011640 10 monclient: build_initial_monmap 2024-09-16T04:02:33.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.491+0000 7f58db011640 10 monclient: monmap: 2024-09-16T04:02:33.496 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:02:33.496 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:02:33.497 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:02:33.494598+0000 2024-09-16T04:02:33.497 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:02:33.494598+0000 2024-09-16T04:02:33.497 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:02:33.497 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:02:33.498 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:02:33.498 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:02:33.498 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:02:33.498 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:02:33.498 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:02:33.498 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:02:33.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.491+0000 7f58db011640 1 Processor -- start 2024-09-16T04:02:33.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.491+0000 7f58db011640 1 -- start start 2024-09-16T04:02:33.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.491+0000 7f58db011640 10 monclient: init 2024-09-16T04:02:33.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.495+0000 7f58db011640 10 monclient: _reopen_session rank -1 2024-09-16T04:02:33.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.495+0000 7f58db011640 10 monclient: _add_conns ranks=[4,0,3,2,1] 2024-09-16T04:02:33.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.495+0000 7f58db011640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f58d40a1330 0x7f58d409f180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:33.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.495+0000 7f58db011640 10 monclient(hunting): picked mon.noname-c con 0x7f58d40a1330 addr v2:172.21.15.154:3300/0 2024-09-16T04:02:33.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.495+0000 7f58db011640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f58d40a1710 addr v1:172.21.15.42:6789/0 2024-09-16T04:02:33.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.495+0000 7f58db011640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f58d41393a0 0x7f58d413b790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:33.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.495+0000 7f58db011640 10 monclient(hunting): picked mon.noname-a con 0x7f58d41393a0 addr v2:172.21.15.42:3300/0 2024-09-16T04:02:33.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.495+0000 7f58db011640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f58d409e4c0 con 0x7f58d40a1710 2024-09-16T04:02:33.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.495+0000 7f58db011640 10 monclient(hunting): start opening mon connection 2024-09-16T04:02:33.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.495+0000 7f58db011640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f58d409edf0 con 0x7f58d41393a0 2024-09-16T04:02:33.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.495+0000 7f58db011640 10 monclient(hunting): start opening mon connection 2024-09-16T04:02:33.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.495+0000 7f58db011640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f58d409cd60 con 0x7f58d40a1330 2024-09-16T04:02:33.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.495+0000 7f58db011640 10 monclient(hunting): _renew_subs 2024-09-16T04:02:33.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.495+0000 7f58db011640 10 monclient(hunting): authenticate will time out at 2200.464695s 2024-09-16T04:02:33.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.495+0000 7f58d95ae640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f58d41393a0 0x7f58d413b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:02:33.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.495+0000 7f58d3fff640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f58d40a1710 0x7f58d40a1b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:41270/0 (socket says 172.21.15.42:41270) 2024-09-16T04:02:33.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.495+0000 7f58d3fff640 1 -- 172.21.15.42:0/3544448406 learned_addr learned my addr 172.21.15.42:0/3544448406 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T04:02:33.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.495+0000 7f58d95ae640 10 monclient(hunting): get_auth_request con 0x7f58d41393a0 auth_method 0 2024-09-16T04:02:33.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.495+0000 7f58d95ae640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:02:33.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.495+0000 7f58d95ae640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:02:33.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.495+0000 7f58d95ae640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:02:33.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.495+0000 7f58d95ae640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:02:33.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.495+0000 7f58d95ae640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:02:33.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.495+0000 7f58d95ae640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:02:33.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.495+0000 7f58d8dad640 1 --2- 172.21.15.42:0/3544448406 >> v2:172.21.15.154:3300/0 conn(0x7f58d40a1330 0x7f58d409f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:02:33.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.495+0000 7f58d8dad640 10 monclient(hunting): get_auth_request con 0x7f58d40a1330 auth_method 0 2024-09-16T04:02:33.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.495+0000 7f58d8dad640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:02:33.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.495+0000 7f58d8dad640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:02:33.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.495+0000 7f58d8dad640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:02:33.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.495+0000 7f58d95ae640 10 monclient(hunting): handle_auth_done global_id 825507 payload 274 2024-09-16T04:02:33.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.495+0000 7f58d95ae640 1 -- 172.21.15.42:0/3544448406 >> v2:172.21.15.154:3300/0 conn(0x7f58d40a1330 msgr2=0x7f58d409f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:02:33.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.495+0000 7f58d95ae640 1 --2- 172.21.15.42:0/3544448406 >> v2:172.21.15.154:3300/0 conn(0x7f58d40a1330 0x7f58d409f180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:33.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.495+0000 7f58d95ae640 1 -- 172.21.15.42:0/3544448406 >> v1:172.21.15.42:6789/0 conn(0x7f58d40a1710 legacy=0x7f58d40a1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:02:33.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.495+0000 7f58d95ae640 10 monclient: _finish_hunting 0 2024-09-16T04:02:33.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.495+0000 7f58d95ae640 1 monclient: found mon.noname-a 2024-09-16T04:02:33.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.495+0000 7f58d95ae640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:02:33.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.495+0000 7f58d95ae640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:02:33.513 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.495+0000 7f58d95ae640 1 -- 172.21.15.42:0/3544448406 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58d409f6c0 con 0x7f58d41393a0 2024-09-16T04:02:33.513 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.495+0000 7f58d95ae640 1 --2- 172.21.15.42:0/3544448406 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f58d41393a0 0x7f58d413b790 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f58d406add0 tx=0x7f58cc01c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76f18400f85e6d8a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:02:33.513 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.495+0000 7f58d37fe640 1 -- 172.21.15.42:0/3544448406 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58cc0b2070 con 0x7f58d41393a0 2024-09-16T04:02:33.513 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.495+0000 7f58d37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:02:33.513 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.495+0000 7f58d37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:02:33.513 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.495+0000 7f58d37fe640 10 monclient: dump: 2024-09-16T04:02:33.513 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:02:33.513 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:02:33.513 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:33.513 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:33.514 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:02:33.514 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:02:33.514 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:02:33.514 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:02:33.514 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:02:33.514 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:02:33.514 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.495+0000 7f58d37fe640 10 monclient: _finish_auth 0 2024-09-16T04:02:33.514 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.495+0000 7f58d37fe640 10 monclient: _check_auth_tickets 2024-09-16T04:02:33.514 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.495+0000 7f58d37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:02:33.514 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.495+0000 7f58db011640 5 monclient: authenticate success, global_id 825507 2024-09-16T04:02:33.514 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.495+0000 7f58d37fe640 1 -- 172.21.15.42:0/3544448406 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f58cc0a57c0 con 0x7f58d41393a0 2024-09-16T04:02:33.514 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.495+0000 7f58db011640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:02:33.514 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.495+0000 7f58d37fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:02:33.514 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.495+0000 7f58db011640 10 monclient: get_monmap_and_config success 2024-09-16T04:02:33.514 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.495+0000 7f58d37fe640 1 -- 172.21.15.42:0/3544448406 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58cc0a5b90 con 0x7f58d41393a0 2024-09-16T04:02:33.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.495+0000 7f58db011640 4 set_mon_vals no callback set 2024-09-16T04:02:33.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.495+0000 7f58d37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:02:33.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.495+0000 7f58d37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:02:33.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.495+0000 7f58d37fe640 10 monclient: dump: 2024-09-16T04:02:33.515 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:02:33.515 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:02:33.515 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:33.515 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:33.515 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:02:33.515 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:02:33.515 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:02:33.515 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:02:33.515 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:02:33.515 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:02:33.516 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.499+0000 7f58db011640 10 monclient: shutdown 2024-09-16T04:02:33.516 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.499+0000 7f58db011640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:02:33.516 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.499+0000 7f58db011640 1 -- 172.21.15.42:0/3544448406 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f58d41393a0 msgr2=0x7f58d413b790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:02:33.516 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.499+0000 7f58db011640 1 --2- 172.21.15.42:0/3544448406 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f58d41393a0 0x7f58d413b790 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f58d406add0 tx=0x7f58cc01c7f0 comp rx=0 tx=0).stop 2024-09-16T04:02:33.516 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.499+0000 7f58db011640 1 -- 172.21.15.42:0/3544448406 shutdown_connections 2024-09-16T04:02:33.516 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.499+0000 7f58db011640 1 --2- 172.21.15.42:0/3544448406 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f58d41393a0 0x7f58d413b790 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:33.516 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.499+0000 7f58db011640 1 --2- 172.21.15.42:0/3544448406 >> v2:172.21.15.154:3300/0 conn(0x7f58d40a1330 0x7f58d409f180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:33.516 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.499+0000 7f58db011640 1 -- 172.21.15.42:0/3544448406 >> 172.21.15.42:0/3544448406 conn(0x7f58d40a2420 msgr2=0x7f58d40a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:02:33.516 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.499+0000 7f58db011640 1 -- 172.21.15.42:0/3544448406 shutdown_connections 2024-09-16T04:02:33.516 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.499+0000 7f58db011640 1 -- 172.21.15.42:0/3544448406 wait complete. 2024-09-16T04:02:33.516 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.499+0000 7f58db011640 10 monclient: build_initial_monmap 2024-09-16T04:02:33.516 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.499+0000 7f58db011640 10 monclient: monmap: 2024-09-16T04:02:33.516 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:02:33.516 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:02:33.517 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:02:33.517 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:02:33.517 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:02:33.517 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:02:33.517 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:02:33.517 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:02:33.517 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:02:33.517 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:02:33.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.499+0000 7f58db011640 1 librados: starting msgr at 2024-09-16T04:02:33.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.499+0000 7f58db011640 1 librados: starting objecter 2024-09-16T04:02:33.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.499+0000 7f58db011640 1 Processor -- start 2024-09-16T04:02:33.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.499+0000 7f58db011640 1 -- start start 2024-09-16T04:02:33.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.499+0000 7f58db011640 1 librados: setting wanted keys 2024-09-16T04:02:33.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.499+0000 7f58db011640 1 librados: calling monclient init 2024-09-16T04:02:33.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.499+0000 7f58db011640 10 monclient: init 2024-09-16T04:02:33.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.499+0000 7f58db011640 10 monclient: _reopen_session rank -1 2024-09-16T04:02:33.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.499+0000 7f58db011640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:02:33.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.499+0000 7f58db011640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f58d40a1330 0x7f58d41cede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:33.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.499+0000 7f58db011640 10 monclient(hunting): picked mon.noname-b con 0x7f58d40a1330 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:02:33.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.499+0000 7f58db011640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f58d40a1710 0x7f58d41344e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:33.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.499+0000 7f58db011640 10 monclient(hunting): picked mon.noname-a con 0x7f58d40a1710 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:02:33.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.499+0000 7f58db011640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f58d41393a0 0x7f58d4139780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:33.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.499+0000 7f58db011640 10 monclient(hunting): picked mon.noname-c con 0x7f58d41393a0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:02:33.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.499+0000 7f58db011640 10 monclient(hunting): start opening mon connection 2024-09-16T04:02:33.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.499+0000 7f58db011640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f58d413e880 con 0x7f58d40a1710 2024-09-16T04:02:33.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.499+0000 7f58db011640 10 monclient(hunting): start opening mon connection 2024-09-16T04:02:33.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.499+0000 7f58db011640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f58d413e700 con 0x7f58d40a1330 2024-09-16T04:02:33.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.499+0000 7f58db011640 10 monclient(hunting): start opening mon connection 2024-09-16T04:02:33.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.499+0000 7f58db011640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f58d413ea00 con 0x7f58d41393a0 2024-09-16T04:02:33.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.499+0000 7f58db011640 10 monclient(hunting): _renew_subs 2024-09-16T04:02:33.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.499+0000 7f58db011640 10 monclient(hunting): authenticate will time out at 2200.469827s 2024-09-16T04:02:33.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.499+0000 7f58d3fff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f58d40a1710 0x7f58d41344e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:02:33.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.499+0000 7f58d95ae640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f58d41393a0 0x7f58d4139780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:02:33.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.499+0000 7f58d3fff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f58d40a1710 0x7f58d41344e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:42598/0 (socket says 172.21.15.42:42598) 2024-09-16T04:02:33.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.499+0000 7f58d3fff640 1 -- 172.21.15.42:0/613471690 learned_addr learned my addr 172.21.15.42:0/613471690 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:02:33.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.499+0000 7f58d3fff640 10 monclient(hunting): get_auth_request con 0x7f58d40a1710 auth_method 0 2024-09-16T04:02:33.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.499+0000 7f58d3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:02:33.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.499+0000 7f58d3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:02:33.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.499+0000 7f58d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:02:33.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.499+0000 7f58d95ae640 10 monclient(hunting): get_auth_request con 0x7f58d41393a0 auth_method 0 2024-09-16T04:02:33.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.499+0000 7f58d95ae640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:02:33.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.499+0000 7f58d95ae640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:02:33.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.499+0000 7f58d95ae640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:02:33.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.499+0000 7f58d95ae640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:02:33.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.499+0000 7f58d95ae640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:02:33.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.499+0000 7f58d95ae640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:02:33.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.499+0000 7f58d3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:02:33.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.499+0000 7f58d3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:02:33.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.499+0000 7f58d3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:02:33.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.499+0000 7f58d8dad640 1 --2- 172.21.15.42:0/613471690 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f58d40a1330 0x7f58d41cede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:02:33.521 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.499+0000 7f58d95ae640 10 monclient(hunting): handle_auth_done global_id 964106 payload 824 2024-09-16T04:02:33.521 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.499+0000 7f58d95ae640 1 -- 172.21.15.42:0/613471690 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f58d40a1330 msgr2=0x7f58d41cede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:02:33.521 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.499+0000 7f58d95ae640 1 --2- 172.21.15.42:0/613471690 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f58d40a1330 0x7f58d41cede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:33.521 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.499+0000 7f58d95ae640 1 -- 172.21.15.42:0/613471690 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f58d40a1710 msgr2=0x7f58d41344e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:02:33.521 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.499+0000 7f58d95ae640 1 --2- 172.21.15.42:0/613471690 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f58d40a1710 0x7f58d41344e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:33.521 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.499+0000 7f58d95ae640 10 monclient: _finish_hunting 0 2024-09-16T04:02:33.521 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.499+0000 7f58d95ae640 1 monclient: found mon.noname-c 2024-09-16T04:02:33.521 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.499+0000 7f58d95ae640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:02:33.521 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.499+0000 7f58d95ae640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T04:02:33.521 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.499+0000 7f58d95ae640 1 -- 172.21.15.42:0/613471690 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58d413b150 con 0x7f58d41393a0 2024-09-16T04:02:33.521 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.499+0000 7f58d8dad640 10 monclient: get_auth_request con 0x7f58d40a1330 auth_method 0 2024-09-16T04:02:33.521 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.499+0000 7f58d8dad640 1 --2- 172.21.15.42:0/613471690 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f58d40a1330 0x7f58d41cede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T04:02:33.521 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.499+0000 7f58d95ae640 1 --2- 172.21.15.42:0/613471690 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f58d41393a0 0x7f58d4139780 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f58d40a6900 tx=0x7f58cc019590 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:02:33.522 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.499+0000 7f58d17fa640 10 client.?.objecter ms_handle_connect 0x7f58d41393a0 2024-09-16T04:02:33.522 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.499+0000 7f58d17fa640 10 client.?.objecter resend_mon_ops 2024-09-16T04:02:33.522 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.499+0000 7f58d17fa640 1 -- 172.21.15.42:0/613471690 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58cc0b2050 con 0x7f58d41393a0 2024-09-16T04:02:33.522 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.499+0000 7f58d17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:02:33.522 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.499+0000 7f58d17fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:02:33.522 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.499+0000 7f58d17fa640 10 monclient: dump: 2024-09-16T04:02:33.522 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:02:33.522 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:02:33.522 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:33.522 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:33.522 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:02:33.522 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:02:33.522 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:02:33.522 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:02:33.523 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:02:33.523 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:02:33.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.499+0000 7f58d17fa640 10 monclient: _finish_auth 0 2024-09-16T04:02:33.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.499+0000 7f58d17fa640 10 monclient: _check_auth_tickets 2024-09-16T04:02:33.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.499+0000 7f58d17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:02:33.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.499+0000 7f58d17fa640 1 -- 172.21.15.42:0/613471690 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f58cc0be070 con 0x7f58d41393a0 2024-09-16T04:02:33.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.499+0000 7f58d17fa640 10 monclient: handle_config config(0 keys) 2024-09-16T04:02:33.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.499+0000 7f58d17fa640 1 -- 172.21.15.42:0/613471690 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58cc0ada90 con 0x7f58d41393a0 2024-09-16T04:02:33.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.499+0000 7f58d17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:02:33.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.499+0000 7f58d17fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:02:33.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.499+0000 7f58d17fa640 10 monclient: dump: 2024-09-16T04:02:33.523 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:02:33.523 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:02:33.523 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:33.524 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:33.524 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:02:33.524 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:02:33.524 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:02:33.524 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:02:33.524 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:02:33.524 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:02:33.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.499+0000 7f58db011640 5 monclient: authenticate success, global_id 964106 2024-09-16T04:02:33.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.499+0000 7f58db011640 10 monclient: _renew_subs 2024-09-16T04:02:33.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.499+0000 7f58d2ffd640 4 set_mon_vals no callback set 2024-09-16T04:02:33.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.499+0000 7f58db011640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:02:33.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.499+0000 7f58db011640 1 -- 172.21.15.42:0/613471690 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58d4073a30 con 0x7f58d41393a0 2024-09-16T04:02:33.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.499+0000 7f58db011640 10 client.964106.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:02:33.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.499+0000 7f58db011640 10 monclient: _renew_subs 2024-09-16T04:02:33.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.499+0000 7f58db011640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:02:33.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.499+0000 7f58db011640 1 -- 172.21.15.42:0/613471690 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f58d4121600 con 0x7f58d41393a0 2024-09-16T04:02:33.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.499+0000 7f58db011640 1 librados: init done 2024-09-16T04:02:33.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.499+0000 7f58db011640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:02:33.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.499+0000 7f58db011640 1 --2- 172.21.15.42:0/613471690 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f58d4134a20 0x7f58d402c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:33.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.499+0000 7f58db011640 10 monclient(hunting): start opening mon connection 2024-09-16T04:02:33.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.499+0000 7f58db011640 1 -- 172.21.15.42:0/613471690 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f58d409f180 con 0x7f58d4134a20 2024-09-16T04:02:33.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.499+0000 7f58d8dad640 1 --2- 172.21.15.42:0/613471690 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f58d4134a20 0x7f58d402c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:02:33.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.499+0000 7f58d8dad640 10 monclient: get_auth_request con 0x7f58d4134a20 auth_method 0 2024-09-16T04:02:33.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.499+0000 7f58d8dad640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:02:33.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.499+0000 7f58d8dad640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:02:33.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.499+0000 7f58d8dad640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:02:33.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.499+0000 7f58d17fa640 1 -- 172.21.15.42:0/613471690 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f58cc0a5640 con 0x7f58d41393a0 2024-09-16T04:02:33.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.499+0000 7f58d8dad640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:02:33.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.499+0000 7f58d8dad640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:02:33.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.499+0000 7f58d8dad640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:02:33.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.499+0000 7f58d8dad640 10 monclient(hunting): handle_auth_done global_id 825516 payload 824 2024-09-16T04:02:33.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.499+0000 7f58d17fa640 1 --2- 172.21.15.42:0/613471690 >> v2:172.21.15.154:6812/254845342 conn(0x7f58c0037810 0x7f58c0039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:33.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.499+0000 7f58d8dad640 1 -- 172.21.15.42:0/613471690 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f58d406f010 con 0x7f58d4134a20 2024-09-16T04:02:33.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.499+0000 7f58d17fa640 1 -- 172.21.15.42:0/613471690 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(629..629 src has 451..629) ==== 6280+0+0 (secure 0 0 0) 0x7f58cc0e40b0 con 0x7f58d41393a0 2024-09-16T04:02:33.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.499+0000 7f58d17fa640 10 client.964106.objecter ms_dispatch 0x7f58d4000ba0 osd_map(629..629 src has 451..629) 2024-09-16T04:02:33.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.499+0000 7f58d17fa640 3 client.964106.objecter handle_osd_map got epochs [629,629] > 0 2024-09-16T04:02:33.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.499+0000 7f58d17fa640 3 client.964106.objecter handle_osd_map decoding full epoch 629 2024-09-16T04:02:33.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.503+0000 7f58d17fa640 20 client.964106.objecter dump_active .. 0 homeless 2024-09-16T04:02:33.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.503+0000 7f58d17fa640 10 client.964106.objecter ms_handle_connect 0x7f58d4134a20 2024-09-16T04:02:33.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.503+0000 7f58d17fa640 10 client.964106.objecter resend_mon_ops 2024-09-16T04:02:33.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.503+0000 7f58d8dad640 1 --2- 172.21.15.42:0/613471690 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f58d4134a20 0x7f58d402c650 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f58c400e740 tx=0x7f58c400ec00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:02:33.527 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.503+0000 7f58d17fa640 1 -- 172.21.15.42:0/613471690 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58c4017070 con 0x7f58d4134a20 2024-09-16T04:02:33.527 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.503+0000 7f58d17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:02:33.527 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.503+0000 7f58d17fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:02:33.527 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.503+0000 7f58d17fa640 10 monclient: dump: 2024-09-16T04:02:33.527 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:02:33.527 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:02:33.527 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:33.527 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:33.527 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:02:33.527 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:02:33.527 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:02:33.528 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:02:33.528 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:02:33.528 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:02:33.528 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.503+0000 7f58d3fff640 1 --2- 172.21.15.42:0/613471690 >> v2:172.21.15.154:6812/254845342 conn(0x7f58c0037810 0x7f58c0039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:02:33.528 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.503+0000 7f58d3fff640 10 monclient: get_auth_request con 0x7f58c0037810 auth_method 0 2024-09-16T04:02:33.528 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.503+0000 7f58d17fa640 1 -- 172.21.15.42:0/613471690 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f58c4004020 con 0x7f58d4134a20 2024-09-16T04:02:33.528 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.503+0000 7f58d17fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:02:33.528 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.503+0000 7f58d17fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:02:33.528 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.503+0000 7f58d17fa640 1 -- 172.21.15.42:0/613471690 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f58d4134a20 msgr2=0x7f58d402c650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:02:33.528 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.503+0000 7f58d17fa640 1 --2- 172.21.15.42:0/613471690 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f58d4134a20 0x7f58d402c650 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f58c400e740 tx=0x7f58c400ec00 comp rx=0 tx=0).stop 2024-09-16T04:02:33.528 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.503+0000 7f58d17fa640 1 -- 172.21.15.42:0/613471690 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f58d4134a20 msgr2=0x7f58d402c650 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:02:33.528 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.503+0000 7f58d17fa640 1 --2- 172.21.15.42:0/613471690 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f58d4134a20 0x7f58d402c650 secure :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f58c400e740 tx=0x7f58c400ec00 comp rx=0 tx=0).stop 2024-09-16T04:02:33.528 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.503+0000 7f58d17fa640 10 client.964106.objecter ms_handle_connect 0x7f58c0037810 2024-09-16T04:02:33.528 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.503+0000 7f58d3fff640 1 --2- 172.21.15.42:0/613471690 >> v2:172.21.15.154:6812/254845342 conn(0x7f58c0037810 0x7f58c0039cd0 secure :-1 s=READY pgs=3783 cs=0 l=1 rev1=1 crypto rx=0x7f58d4072a20 tx=0x7f58c8007f20 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:02:33.530 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.507+0000 7f58db011640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:02:33.530 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.507+0000 7f58db011640 1 --2- 172.21.15.42:0/613471690 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f58d402cb90 0x7f58d402cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:33.530 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.507+0000 7f58db011640 10 monclient(hunting): start opening mon connection 2024-09-16T04:02:33.530 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.507+0000 7f58db011640 1 -- 172.21.15.42:0/613471690 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f58d4069040 con 0x7f58d402cb90 2024-09-16T04:02:33.530 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.507+0000 7f58d8dad640 1 --2- 172.21.15.42:0/613471690 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f58d402cb90 0x7f58d402cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:02:33.531 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.507+0000 7f58d8dad640 10 monclient: get_auth_request con 0x7f58d402cb90 auth_method 0 2024-09-16T04:02:33.531 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.507+0000 7f58d8dad640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:02:33.531 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.507+0000 7f58d8dad640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:02:33.531 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.507+0000 7f58d8dad640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:02:33.531 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.507+0000 7f58d8dad640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:02:33.531 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.507+0000 7f58d8dad640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:02:33.531 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.507+0000 7f58d8dad640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:02:33.531 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.507+0000 7f58d8dad640 10 monclient(hunting): handle_auth_done global_id 825519 payload 824 2024-09-16T04:02:33.531 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.507+0000 7f58d8dad640 1 -- 172.21.15.42:0/613471690 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f58d413a480 con 0x7f58d402cb90 2024-09-16T04:02:33.531 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.507+0000 7f58d8dad640 1 --2- 172.21.15.42:0/613471690 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f58d402cb90 0x7f58d402cfa0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f58c400e740 tx=0x7f58c40046d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:02:33.531 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.507+0000 7f58d17fa640 10 client.964106.objecter ms_handle_connect 0x7f58d402cb90 2024-09-16T04:02:33.531 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.507+0000 7f58d17fa640 10 client.964106.objecter resend_mon_ops 2024-09-16T04:02:33.531 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.507+0000 7f58d17fa640 1 -- 172.21.15.42:0/613471690 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58c400f040 con 0x7f58d402cb90 2024-09-16T04:02:33.531 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.507+0000 7f58d17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:02:33.532 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.507+0000 7f58d17fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:02:33.532 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.507+0000 7f58d17fa640 10 monclient: dump: 2024-09-16T04:02:33.532 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:02:33.532 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:02:33.532 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:33.532 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:33.532 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:02:33.532 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:02:33.532 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:02:33.532 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:02:33.532 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:02:33.532 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:02:33.532 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.507+0000 7f58d17fa640 1 -- 172.21.15.42:0/613471690 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4715 (secure 0 0 0) 0x7f58c4003950 con 0x7f58d402cb90 2024-09-16T04:02:33.532 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.507+0000 7f58d17fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:02:33.533 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.507+0000 7f58d17fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:02:33.533 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.507+0000 7f58d17fa640 1 -- 172.21.15.42:0/613471690 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f58d402cb90 msgr2=0x7f58d402cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:02:33.533 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.507+0000 7f58d17fa640 1 --2- 172.21.15.42:0/613471690 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f58d402cb90 0x7f58d402cfa0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f58c400e740 tx=0x7f58c40046d0 comp rx=0 tx=0).stop 2024-09-16T04:02:33.533 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.511+0000 7f58d17fa640 1 -- 172.21.15.42:0/613471690 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f58d402cb90 msgr2=0x7f58d402cfa0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:02:33.533 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.511+0000 7f58d17fa640 1 --2- 172.21.15.42:0/613471690 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f58d402cb90 0x7f58d402cfa0 secure :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f58c400e740 tx=0x7f58c40046d0 comp rx=0 tx=0).stop 2024-09-16T04:02:33.533 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T04:02:33.533 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:02:33.533 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:02:33.533 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "leader", 2024-09-16T04:02:33.533 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 184, 2024-09-16T04:02:33.534 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 643362, 2024-09-16T04:02:33.534 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T04:02:33.534 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T04:02:33.534 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T04:02:33.534 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T04:02:33.534 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:02:33.534 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 2, 2024-09-16T04:02:33.534 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:02:33.534 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T04:02:33.534 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T04:02:33.534 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:02:33.534 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:02:33.534 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:02:33.534 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:02:33.535 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:02:33.535 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:02:33.535 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:02:33.535 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:02:33.535 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:02:33.535 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:02:33.535 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:02:33.535 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:02:33.535 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:02:33.535 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T04:02:33.535 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:02:33.535 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:02:33.535 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:02:33.535 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:02:33.535 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:02:33.536 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:02:33.536 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:02:33.536 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:02:33.536 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:02:33.536 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:02:33.536 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:02:33.536 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:02:33.536 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:02:33.536 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T04:02:33.536 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T04:02:33.536 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T04:02:33.536 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T04:02:33.536 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T04:02:33.536 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T04:02:33.537 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:02:33.537 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:02:33.537 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T04:02:33.537 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T04:02:33.537 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T04:02:33.537 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T04:02:33.538 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T04:02:33.538 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T04:02:33.538 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T04:02:33.538 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:02:33.538 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T04:02:33.538 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:02:33.538 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:02:33.538 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:02:33.538 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:02:33.538 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:02:33.538 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:02:33.538 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:02:33.538 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:02:33.538 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:02:33.539 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:02:33.539 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:02:33.539 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:02:33.539 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T04:02:33.539 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:02:33.539 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T04:02:33.539 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:02:33.539 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:02:33.539 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:02:33.539 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:02:33.539 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:02:33.539 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:02:33.539 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:02:33.539 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T04:02:33.540 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:02:33.540 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:02:33.540 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:02:33.540 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:02:33.540 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T04:02:33.540 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:02:33.540 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:02:33.540 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:02:33.540 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:02:33.540 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T04:02:33.540 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T04:02:33.540 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:02:33.540 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:02:33.540 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:02:33.541 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:02:33.541 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:02:33.541 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:02:33.541 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:02:33.541 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:02:33.541 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:02:33.541 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:02:33.541 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:02:33.541 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T04:02:33.541 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:02:33.541 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:02:33.541 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:02:33.541 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:02:33.541 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T04:02:33.542 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:02:33.542 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:02:33.542 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:02:33.542 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:02:33.542 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T04:02:33.542 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T04:02:33.542 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:02:33.542 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:02:33.542 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:02:33.542 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:02:33.542 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:02:33.542 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:02:33.542 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:02:33.542 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:02:33.542 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:02:33.543 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:02:33.543 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:02:33.543 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T04:02:33.543 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:02:33.543 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:02:33.543 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:02:33.543 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:02:33.543 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T04:02:33.543 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:02:33.543 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:02:33.543 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:02:33.543 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:02:33.543 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T04:02:33.543 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T04:02:33.544 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:02:33.544 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:02:33.544 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:02:33.544 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:02:33.544 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:02:33.544 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:02:33.544 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T04:02:33.544 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T04:02:33.544 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:02:33.544 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:02:33.544 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:02:33.544 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:02:33.544 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:02:33.544 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:02:33.544 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T04:02:33.545 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:02:33.545 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:02:33.545 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:02:33.545 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 5 2024-09-16T04:02:33.545 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:02:33.545 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:02:33.545 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T04:02:33.545 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:02:33.545 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:02:33.545 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:02:33.545 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 6 2024-09-16T04:02:33.545 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:02:33.545 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:02:33.545 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:02:33.546 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T04:02:33.546 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T04:02:33.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.511+0000 7f58db011640 10 librados: watch_flush enter 2024-09-16T04:02:33.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.511+0000 7f58db011640 10 librados: watch_flush exit 2024-09-16T04:02:33.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.511+0000 7f58db011640 20 client.964106.objecter shutdown clearing up homeless session... 2024-09-16T04:02:33.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.511+0000 7f58db011640 10 client.964106.objecter successfully canceled tick 2024-09-16T04:02:33.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.511+0000 7f58db011640 1 -- 172.21.15.42:0/613471690 >> v2:172.21.15.154:6812/254845342 conn(0x7f58c0037810 msgr2=0x7f58c0039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:02:33.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.511+0000 7f58db011640 1 --2- 172.21.15.42:0/613471690 >> v2:172.21.15.154:6812/254845342 conn(0x7f58c0037810 0x7f58c0039cd0 secure :-1 s=READY pgs=3783 cs=0 l=1 rev1=1 crypto rx=0x7f58d4072a20 tx=0x7f58c8007f20 comp rx=0 tx=0).stop 2024-09-16T04:02:33.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.511+0000 7f58db011640 10 monclient: shutdown 2024-09-16T04:02:33.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.511+0000 7f58db011640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:02:33.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.511+0000 7f58db011640 1 -- 172.21.15.42:0/613471690 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f58d41393a0 msgr2=0x7f58d4139780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:02:33.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.511+0000 7f58db011640 1 --2- 172.21.15.42:0/613471690 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f58d41393a0 0x7f58d4139780 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f58d40a6900 tx=0x7f58cc019590 comp rx=0 tx=0).stop 2024-09-16T04:02:33.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.511+0000 7f58d95ae640 1 -- 172.21.15.42:0/613471690 reap_dead start 2024-09-16T04:02:33.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.511+0000 7f58d95ae640 1 -- 172.21.15.42:0/613471690 reap_dead start 2024-09-16T04:02:33.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.511+0000 7f58db011640 1 -- 172.21.15.42:0/613471690 shutdown_connections 2024-09-16T04:02:33.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.511+0000 7f58db011640 1 -- 172.21.15.42:0/613471690 >> 172.21.15.42:0/613471690 conn(0x7f58d40a2420 msgr2=0x7f58d409c270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:02:33.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.511+0000 7f58db011640 1 -- 172.21.15.42:0/613471690 shutdown_connections 2024-09-16T04:02:33.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.511+0000 7f58db011640 1 -- 172.21.15.42:0/613471690 wait complete. 2024-09-16T04:02:33.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.511+0000 7f58db011640 1 librados: shutdown 2024-09-16T04:02:33.549 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:02:33.549 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T04:02:33.549 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T04:02:33.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.663+0000 7f5d50fe9640 10 monclient: get_monmap_and_config 2024-09-16T04:02:33.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.663+0000 7f5d50fe9640 10 monclient: build_initial_monmap 2024-09-16T04:02:33.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.663+0000 7f5d50fe9640 10 monclient: monmap: 2024-09-16T04:02:33.667 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:02:33.667 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:02:33.667 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:02:33.665372+0000 2024-09-16T04:02:33.667 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:02:33.665372+0000 2024-09-16T04:02:33.667 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:02:33.667 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:02:33.667 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:02:33.667 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:02:33.667 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:02:33.667 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:02:33.667 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:02:33.668 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:02:33.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.663+0000 7f5d50fe9640 1 Processor -- start 2024-09-16T04:02:33.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.663+0000 7f5d50fe9640 1 -- start start 2024-09-16T04:02:33.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.663+0000 7f5d50fe9640 10 monclient: init 2024-09-16T04:02:33.670 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.663+0000 7f5d50fe9640 10 monclient: _reopen_session rank -1 2024-09-16T04:02:33.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.663+0000 7f5d50fe9640 10 monclient: _add_conns ranks=[4,1,2,3,0] 2024-09-16T04:02:33.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.663+0000 7f5d50fe9640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f5d4c09ca60 0x7f5d4c09f310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:33.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.663+0000 7f5d50fe9640 10 monclient(hunting): picked mon.noname-c con 0x7f5d4c09ca60 addr v2:172.21.15.154:3300/0 2024-09-16T04:02:33.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.663+0000 7f5d50fe9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5d4c09f850 addr v1:172.21.15.154:6789/0 2024-09-16T04:02:33.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.663+0000 7f5d50fe9640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5d4c09fc60 0x7f5d4c0a1150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:33.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.663+0000 7f5d50fe9640 10 monclient(hunting): picked mon.noname-b con 0x7f5d4c09fc60 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:02:33.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.663+0000 7f5d50fe9640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5d4c09eee0 con 0x7f5d4c09f850 2024-09-16T04:02:33.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.663+0000 7f5d50fe9640 10 monclient(hunting): start opening mon connection 2024-09-16T04:02:33.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.663+0000 7f5d50fe9640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f5d4c09de20 con 0x7f5d4c09ca60 2024-09-16T04:02:33.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.663+0000 7f5d50fe9640 10 monclient(hunting): start opening mon connection 2024-09-16T04:02:33.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.663+0000 7f5d50fe9640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f5d4c09ce40 con 0x7f5d4c09fc60 2024-09-16T04:02:33.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.663+0000 7f5d50fe9640 10 monclient(hunting): _renew_subs 2024-09-16T04:02:33.672 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.663+0000 7f5d50fe9640 10 monclient(hunting): authenticate will time out at 2200.635991s 2024-09-16T04:02:33.672 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.667+0000 7f5d4ad9d640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5d4c09fc60 0x7f5d4c0a1150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:02:33.672 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.667+0000 7f5d4ad9d640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5d4c09fc60 0x7f5d4c0a1150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:36310/0 (socket says 172.21.15.42:36310) 2024-09-16T04:02:33.672 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.667+0000 7f5d4ad9d640 1 -- 172.21.15.42:0/2951580318 learned_addr learned my addr 172.21.15.42:0/2951580318 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:02:33.672 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.667+0000 7f5d4ad9d640 10 monclient(hunting): get_auth_request con 0x7f5d4c09fc60 auth_method 0 2024-09-16T04:02:33.672 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.667+0000 7f5d4ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:02:33.672 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.667+0000 7f5d4ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:02:33.672 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.667+0000 7f5d4ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:02:33.672 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.667+0000 7f5d4a59c640 1 --2- 172.21.15.42:0/2951580318 >> v2:172.21.15.154:3300/0 conn(0x7f5d4c09ca60 0x7f5d4c09f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:02:33.672 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.667+0000 7f5d4a59c640 10 monclient(hunting): get_auth_request con 0x7f5d4c09ca60 auth_method 0 2024-09-16T04:02:33.672 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.667+0000 7f5d4a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:02:33.672 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.667+0000 7f5d4a59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:02:33.672 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.667+0000 7f5d4a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:02:33.672 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.667+0000 7f5d4ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:02:33.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.667+0000 7f5d4ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:02:33.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.667+0000 7f5d4ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:02:33.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.667+0000 7f5d4a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:02:33.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.667+0000 7f5d4a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:02:33.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.667+0000 7f5d4a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:02:33.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.667+0000 7f5d4ad9d640 10 monclient(hunting): handle_auth_done global_id 964109 payload 274 2024-09-16T04:02:33.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.667+0000 7f5d4ad9d640 1 -- 172.21.15.42:0/2951580318 >> v2:172.21.15.154:3300/0 conn(0x7f5d4c09ca60 msgr2=0x7f5d4c09f310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:02:33.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.667+0000 7f5d4ad9d640 1 --2- 172.21.15.42:0/2951580318 >> v2:172.21.15.154:3300/0 conn(0x7f5d4c09ca60 0x7f5d4c09f310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:33.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.667+0000 7f5d4ad9d640 1 -- 172.21.15.42:0/2951580318 >> v1:172.21.15.154:6789/0 conn(0x7f5d4c09f850 legacy=0x7f5d4c0a0a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:02:33.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.667+0000 7f5d4ad9d640 10 monclient: _finish_hunting 0 2024-09-16T04:02:33.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.667+0000 7f5d4ad9d640 1 monclient: found mon.noname-b 2024-09-16T04:02:33.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.667+0000 7f5d4ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:02:33.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.667+0000 7f5d4ad9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T04:02:33.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.667+0000 7f5d4ad9d640 1 -- 172.21.15.42:0/2951580318 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d4c0a38b0 con 0x7f5d4c09fc60 2024-09-16T04:02:33.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.667+0000 7f5d4ad9d640 1 --2- 172.21.15.42:0/2951580318 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5d4c09fc60 0x7f5d4c0a1150 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f5d44009e50 tx=0x7f5d4401c8c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7320e6209d46e53b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:02:33.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.667+0000 7f5d4959a640 1 -- 172.21.15.42:0/2951580318 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d440a5700 con 0x7f5d4c09fc60 2024-09-16T04:02:33.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.667+0000 7f5d4959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:02:33.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.667+0000 7f5d4959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:02:33.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.667+0000 7f5d4959a640 10 monclient: dump: 2024-09-16T04:02:33.680 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:02:33.680 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:02:33.680 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:33.680 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:33.680 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:02:33.680 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:02:33.681 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:02:33.681 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:02:33.681 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:02:33.681 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:02:33.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.667+0000 7f5d4959a640 10 monclient: _finish_auth 0 2024-09-16T04:02:33.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.667+0000 7f5d4959a640 10 monclient: _check_auth_tickets 2024-09-16T04:02:33.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.667+0000 7f5d4959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:02:33.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.667+0000 7f5d4959a640 1 -- 172.21.15.42:0/2951580318 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5d440a58a0 con 0x7f5d4c09fc60 2024-09-16T04:02:33.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.667+0000 7f5d4959a640 10 monclient: handle_config config(0 keys) 2024-09-16T04:02:33.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.667+0000 7f5d4959a640 1 -- 172.21.15.42:0/2951580318 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d440a5c70 con 0x7f5d4c09fc60 2024-09-16T04:02:33.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.667+0000 7f5d50fe9640 5 monclient: authenticate success, global_id 964109 2024-09-16T04:02:33.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.667+0000 7f5d50fe9640 10 monclient: get_monmap_and_config success 2024-09-16T04:02:33.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.667+0000 7f5d50fe9640 4 set_mon_vals no callback set 2024-09-16T04:02:33.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.667+0000 7f5d4959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:02:33.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.667+0000 7f5d4959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:02:33.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.667+0000 7f5d4959a640 10 monclient: dump: 2024-09-16T04:02:33.682 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:02:33.682 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:02:33.682 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:33.682 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:33.682 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:02:33.682 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:02:33.682 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:02:33.682 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:02:33.682 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:02:33.682 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:02:33.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.667+0000 7f5d50fe9640 10 monclient: shutdown 2024-09-16T04:02:33.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.667+0000 7f5d50fe9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:02:33.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.667+0000 7f5d50fe9640 1 -- 172.21.15.42:0/2951580318 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5d4c09fc60 msgr2=0x7f5d4c0a1150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:02:33.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.667+0000 7f5d50fe9640 1 --2- 172.21.15.42:0/2951580318 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5d4c09fc60 0x7f5d4c0a1150 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f5d44009e50 tx=0x7f5d4401c8c0 comp rx=0 tx=0).stop 2024-09-16T04:02:33.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.667+0000 7f5d50fe9640 1 -- 172.21.15.42:0/2951580318 shutdown_connections 2024-09-16T04:02:33.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.667+0000 7f5d50fe9640 1 --2- 172.21.15.42:0/2951580318 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5d4c09fc60 0x7f5d4c0a1150 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:33.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.667+0000 7f5d50fe9640 1 --2- 172.21.15.42:0/2951580318 >> v2:172.21.15.154:3300/0 conn(0x7f5d4c09ca60 0x7f5d4c09f310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:33.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.667+0000 7f5d50fe9640 1 -- 172.21.15.42:0/2951580318 >> 172.21.15.42:0/2951580318 conn(0x7f5d4c0a6640 msgr2=0x7f5d4c0a2b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:02:33.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.667+0000 7f5d50fe9640 1 -- 172.21.15.42:0/2951580318 shutdown_connections 2024-09-16T04:02:33.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.667+0000 7f5d50fe9640 1 -- 172.21.15.42:0/2951580318 wait complete. 2024-09-16T04:02:33.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.671+0000 7f5d50fe9640 10 monclient: build_initial_monmap 2024-09-16T04:02:33.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.671+0000 7f5d50fe9640 10 monclient: monmap: 2024-09-16T04:02:33.683 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:02:33.683 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:02:33.684 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:02:33.684 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:02:33.684 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:02:33.684 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:02:33.684 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:02:33.684 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:02:33.684 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:02:33.684 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:02:33.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.671+0000 7f5d50fe9640 1 librados: starting msgr at 2024-09-16T04:02:33.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.671+0000 7f5d50fe9640 1 librados: starting objecter 2024-09-16T04:02:33.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.671+0000 7f5d50fe9640 1 Processor -- start 2024-09-16T04:02:33.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.671+0000 7f5d50fe9640 1 -- start start 2024-09-16T04:02:33.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.671+0000 7f5d50fe9640 1 librados: setting wanted keys 2024-09-16T04:02:33.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.671+0000 7f5d50fe9640 1 librados: calling monclient init 2024-09-16T04:02:33.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.671+0000 7f5d50fe9640 10 monclient: init 2024-09-16T04:02:33.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.671+0000 7f5d50fe9640 10 monclient: _reopen_session rank -1 2024-09-16T04:02:33.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.671+0000 7f5d50fe9640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:02:33.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.671+0000 7f5d50fe9640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5d4c09ca60 0x7f5d4c1cedc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:33.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.671+0000 7f5d50fe9640 10 monclient(hunting): picked mon.noname-a con 0x7f5d4c09ca60 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:02:33.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.671+0000 7f5d50fe9640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5d4c09f850 0x7f5d4c137db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:33.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.671+0000 7f5d50fe9640 10 monclient(hunting): picked mon.noname-b con 0x7f5d4c09f850 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:02:33.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.671+0000 7f5d50fe9640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5d4c09fc60 0x7f5d4c1382f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:33.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.671+0000 7f5d50fe9640 10 monclient(hunting): picked mon.noname-c con 0x7f5d4c09fc60 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:02:33.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.671+0000 7f5d50fe9640 10 monclient(hunting): start opening mon connection 2024-09-16T04:02:33.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.671+0000 7f5d50fe9640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f5d4c0a1690 con 0x7f5d4c09ca60 2024-09-16T04:02:33.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.671+0000 7f5d50fe9640 10 monclient(hunting): start opening mon connection 2024-09-16T04:02:33.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.671+0000 7f5d50fe9640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f5d4c0a1ea0 con 0x7f5d4c09f850 2024-09-16T04:02:33.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.671+0000 7f5d50fe9640 10 monclient(hunting): start opening mon connection 2024-09-16T04:02:33.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.671+0000 7f5d50fe9640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f5d4c0a1810 con 0x7f5d4c09fc60 2024-09-16T04:02:33.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.671+0000 7f5d50fe9640 10 monclient(hunting): _renew_subs 2024-09-16T04:02:33.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.671+0000 7f5d50fe9640 10 monclient(hunting): authenticate will time out at 2200.640992s 2024-09-16T04:02:33.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.671+0000 7f5d4a59c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5d4c09ca60 0x7f5d4c1cedc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:02:33.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.671+0000 7f5d4ad9d640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5d4c09fc60 0x7f5d4c1382f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:02:33.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.671+0000 7f5d4a59c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5d4c09ca60 0x7f5d4c1cedc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:42622/0 (socket says 172.21.15.42:42622) 2024-09-16T04:02:33.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.671+0000 7f5d4a59c640 1 -- 172.21.15.42:0/3411136308 learned_addr learned my addr 172.21.15.42:0/3411136308 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:02:33.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.671+0000 7f5d4a59c640 10 monclient(hunting): get_auth_request con 0x7f5d4c09ca60 auth_method 0 2024-09-16T04:02:33.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.671+0000 7f5d4a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:02:33.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.671+0000 7f5d4a59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:02:33.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.671+0000 7f5d4a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:02:33.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.671+0000 7f5d4ad9d640 10 monclient(hunting): get_auth_request con 0x7f5d4c09fc60 auth_method 0 2024-09-16T04:02:33.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.671+0000 7f5d4ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:02:33.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.671+0000 7f5d4ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:02:33.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.671+0000 7f5d4ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:02:33.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.671+0000 7f5d4a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:02:33.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.671+0000 7f5d4a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:02:33.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.671+0000 7f5d4a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:02:33.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.671+0000 7f5d4ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:02:33.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.671+0000 7f5d4ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:02:33.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.671+0000 7f5d49d9b640 1 --2- 172.21.15.42:0/3411136308 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5d4c09f850 0x7f5d4c137db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:02:33.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.671+0000 7f5d4ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:02:33.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.671+0000 7f5d49d9b640 10 monclient(hunting): get_auth_request con 0x7f5d4c09f850 auth_method 0 2024-09-16T04:02:33.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.671+0000 7f5d49d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:02:33.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.671+0000 7f5d49d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:02:33.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.671+0000 7f5d49d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:02:33.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.671+0000 7f5d4a59c640 10 monclient(hunting): handle_auth_done global_id 825522 payload 824 2024-09-16T04:02:33.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.671+0000 7f5d4a59c640 1 -- 172.21.15.42:0/3411136308 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5d4c09fc60 msgr2=0x7f5d4c1382f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:02:33.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.671+0000 7f5d4a59c640 1 --2- 172.21.15.42:0/3411136308 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5d4c09fc60 0x7f5d4c1382f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:33.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.671+0000 7f5d4a59c640 1 -- 172.21.15.42:0/3411136308 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5d4c09f850 msgr2=0x7f5d4c137db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:02:33.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.671+0000 7f5d4a59c640 1 --2- 172.21.15.42:0/3411136308 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5d4c09f850 0x7f5d4c137db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:33.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.671+0000 7f5d4a59c640 10 monclient: _finish_hunting 0 2024-09-16T04:02:33.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.671+0000 7f5d4a59c640 1 monclient: found mon.noname-a 2024-09-16T04:02:33.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.671+0000 7f5d4a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:02:33.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.671+0000 7f5d4a59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:02:33.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.671+0000 7f5d4a59c640 1 -- 172.21.15.42:0/3411136308 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d4c13d660 con 0x7f5d4c09ca60 2024-09-16T04:02:33.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.671+0000 7f5d49d9b640 1 --2- 172.21.15.42:0/3411136308 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5d4c09f850 0x7f5d4c137db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T04:02:33.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.671+0000 7f5d4ad9d640 1 --2- 172.21.15.42:0/3411136308 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5d4c09fc60 0x7f5d4c1382f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:02:33.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.671+0000 7f5d4a59c640 1 --2- 172.21.15.42:0/3411136308 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5d4c09ca60 0x7f5d4c1cedc0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f5d3c00e870 tx=0x7f5d3c00ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:02:33.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.671+0000 7f5d3affd640 10 client.?.objecter ms_handle_connect 0x7f5d4c09ca60 2024-09-16T04:02:33.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.671+0000 7f5d3affd640 10 client.?.objecter resend_mon_ops 2024-09-16T04:02:33.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.671+0000 7f5d3affd640 1 -- 172.21.15.42:0/3411136308 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d3c017070 con 0x7f5d4c09ca60 2024-09-16T04:02:33.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.671+0000 7f5d3affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:02:33.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.671+0000 7f5d3affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:02:33.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.671+0000 7f5d3affd640 10 monclient: dump: 2024-09-16T04:02:33.689 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:02:33.689 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:02:33.689 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:33.689 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:33.689 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:02:33.689 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:02:33.689 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:02:33.689 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:02:33.689 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:02:33.689 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:02:33.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.671+0000 7f5d3affd640 10 monclient: _finish_auth 0 2024-09-16T04:02:33.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.671+0000 7f5d3affd640 10 monclient: _check_auth_tickets 2024-09-16T04:02:33.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.671+0000 7f5d3affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:02:33.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.671+0000 7f5d3affd640 1 -- 172.21.15.42:0/3411136308 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5d3c00c940 con 0x7f5d4c09ca60 2024-09-16T04:02:33.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.671+0000 7f5d50fe9640 5 monclient: authenticate success, global_id 825522 2024-09-16T04:02:33.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.671+0000 7f5d50fe9640 10 monclient: _renew_subs 2024-09-16T04:02:33.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.671+0000 7f5d50fe9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:02:33.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.671+0000 7f5d50fe9640 1 -- 172.21.15.42:0/3411136308 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d4c123640 con 0x7f5d4c09ca60 2024-09-16T04:02:33.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.671+0000 7f5d3affd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:02:33.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.671+0000 7f5d3affd640 1 -- 172.21.15.42:0/3411136308 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d3c00ccf0 con 0x7f5d4c09ca60 2024-09-16T04:02:33.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.671+0000 7f5d3affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:02:33.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.671+0000 7f5d3affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:02:33.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.671+0000 7f5d3affd640 10 monclient: dump: 2024-09-16T04:02:33.691 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:02:33.691 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:02:33.691 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:33.691 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:33.691 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:02:33.691 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:02:33.691 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:02:33.691 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:02:33.691 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:02:33.691 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:02:33.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.671+0000 7f5d48d99640 4 set_mon_vals no callback set 2024-09-16T04:02:33.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.671+0000 7f5d50fe9640 10 client.825522.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:02:33.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.671+0000 7f5d50fe9640 10 monclient: _renew_subs 2024-09-16T04:02:33.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.671+0000 7f5d50fe9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:02:33.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.671+0000 7f5d50fe9640 1 -- 172.21.15.42:0/3411136308 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d4c124480 con 0x7f5d4c09ca60 2024-09-16T04:02:33.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.671+0000 7f5d50fe9640 1 librados: init done 2024-09-16T04:02:33.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.671+0000 7f5d50fe9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:02:33.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.671+0000 7f5d50fe9640 1 --2- 172.21.15.42:0/3411136308 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5d4c06f060 0x7f5d4c02c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:33.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.671+0000 7f5d50fe9640 10 monclient(hunting): start opening mon connection 2024-09-16T04:02:33.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.671+0000 7f5d50fe9640 1 -- 172.21.15.42:0/3411136308 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f5d4c1cfd20 con 0x7f5d4c06f060 2024-09-16T04:02:33.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.671+0000 7f5d49d9b640 1 --2- 172.21.15.42:0/3411136308 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5d4c06f060 0x7f5d4c02c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:02:33.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.671+0000 7f5d49d9b640 10 monclient: get_auth_request con 0x7f5d4c06f060 auth_method 0 2024-09-16T04:02:33.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.671+0000 7f5d49d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:02:33.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.671+0000 7f5d49d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:02:33.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.671+0000 7f5d49d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:02:33.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.671+0000 7f5d3affd640 1 -- 172.21.15.42:0/3411136308 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f5d3c01a070 con 0x7f5d4c09ca60 2024-09-16T04:02:33.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.671+0000 7f5d49d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:02:33.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.671+0000 7f5d49d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:02:33.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.671+0000 7f5d49d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:02:33.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.671+0000 7f5d49d9b640 10 monclient(hunting): handle_auth_done global_id 964115 payload 824 2024-09-16T04:02:33.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.671+0000 7f5d3affd640 1 --2- 172.21.15.42:0/3411136308 >> v2:172.21.15.154:6812/254845342 conn(0x7f5d2c037840 0x7f5d2c039d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:33.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.671+0000 7f5d49d9b640 1 -- 172.21.15.42:0/3411136308 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5d4c09d230 con 0x7f5d4c06f060 2024-09-16T04:02:33.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.671+0000 7f5d3affd640 1 -- 172.21.15.42:0/3411136308 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(630..630 src has 451..630) ==== 6280+0+0 (secure 0 0 0) 0x7f5d3c0490d0 con 0x7f5d4c09ca60 2024-09-16T04:02:33.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.671+0000 7f5d3affd640 10 client.825522.objecter ms_dispatch 0x7f5d4c000ba0 osd_map(630..630 src has 451..630) 2024-09-16T04:02:33.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.671+0000 7f5d3affd640 3 client.825522.objecter handle_osd_map got epochs [630,630] > 0 2024-09-16T04:02:33.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.671+0000 7f5d3affd640 3 client.825522.objecter handle_osd_map decoding full epoch 630 2024-09-16T04:02:33.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.671+0000 7f5d3affd640 20 client.825522.objecter dump_active .. 0 homeless 2024-09-16T04:02:33.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.671+0000 7f5d3affd640 10 client.825522.objecter ms_handle_connect 0x7f5d4c06f060 2024-09-16T04:02:33.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.671+0000 7f5d3affd640 10 client.825522.objecter resend_mon_ops 2024-09-16T04:02:33.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.671+0000 7f5d49d9b640 1 --2- 172.21.15.42:0/3411136308 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5d4c06f060 0x7f5d4c02c650 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f5d4c122f60 tx=0x7f5d4000e940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:02:33.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.671+0000 7f5d4ad9d640 1 --2- 172.21.15.42:0/3411136308 >> v2:172.21.15.154:6812/254845342 conn(0x7f5d2c037840 0x7f5d2c039d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:02:33.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.671+0000 7f5d3affd640 1 -- 172.21.15.42:0/3411136308 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d400052e0 con 0x7f5d4c06f060 2024-09-16T04:02:33.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.671+0000 7f5d3affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:02:33.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.671+0000 7f5d3affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:02:33.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.671+0000 7f5d3affd640 10 monclient: dump: 2024-09-16T04:02:33.694 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:02:33.694 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:02:33.694 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:33.694 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:33.694 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:02:33.694 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:02:33.694 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:02:33.694 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:02:33.694 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:02:33.695 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:02:33.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.671+0000 7f5d4ad9d640 10 monclient: get_auth_request con 0x7f5d2c037840 auth_method 0 2024-09-16T04:02:33.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.671+0000 7f5d3affd640 1 -- 172.21.15.42:0/3411136308 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f5d400052e0 con 0x7f5d4c06f060 2024-09-16T04:02:33.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.671+0000 7f5d3affd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:02:33.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.671+0000 7f5d3affd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:02:33.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.671+0000 7f5d3affd640 1 -- 172.21.15.42:0/3411136308 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5d4c06f060 msgr2=0x7f5d4c02c650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:02:33.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.671+0000 7f5d3affd640 1 --2- 172.21.15.42:0/3411136308 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5d4c06f060 0x7f5d4c02c650 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f5d4c122f60 tx=0x7f5d4000e940 comp rx=0 tx=0).stop 2024-09-16T04:02:33.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.671+0000 7f5d3affd640 1 -- 172.21.15.42:0/3411136308 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5d4c06f060 msgr2=0x7f5d4c02c650 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:02:33.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.671+0000 7f5d3affd640 1 --2- 172.21.15.42:0/3411136308 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5d4c06f060 0x7f5d4c02c650 secure :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f5d4c122f60 tx=0x7f5d4000e940 comp rx=0 tx=0).stop 2024-09-16T04:02:33.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.675+0000 7f5d4ad9d640 1 --2- 172.21.15.42:0/3411136308 >> v2:172.21.15.154:6812/254845342 conn(0x7f5d2c037840 0x7f5d2c039d00 secure :-1 s=READY pgs=3784 cs=0 l=1 rev1=1 crypto rx=0x7f5d440ae000 tx=0x7f5d440a7070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:02:33.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.675+0000 7f5d3affd640 10 client.825522.objecter ms_handle_connect 0x7f5d2c037840 2024-09-16T04:02:33.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.679+0000 7f5d50fe9640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:02:33.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.679+0000 7f5d50fe9640 1 --2- 172.21.15.42:0/3411136308 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5d4c06f440 0x7f5d4c02cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:33.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.679+0000 7f5d50fe9640 10 monclient(hunting): start opening mon connection 2024-09-16T04:02:33.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.679+0000 7f5d50fe9640 1 -- 172.21.15.42:0/3411136308 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f5d4c13a170 con 0x7f5d4c06f440 2024-09-16T04:02:33.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.679+0000 7f5d49d9b640 1 --2- 172.21.15.42:0/3411136308 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5d4c06f440 0x7f5d4c02cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:02:33.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.679+0000 7f5d49d9b640 10 monclient: get_auth_request con 0x7f5d4c06f440 auth_method 0 2024-09-16T04:02:33.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.679+0000 7f5d49d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:02:33.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.679+0000 7f5d49d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:02:33.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.679+0000 7f5d49d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:02:33.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.679+0000 7f5d49d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:02:33.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.679+0000 7f5d49d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:02:33.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.679+0000 7f5d49d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:02:33.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.679+0000 7f5d49d9b640 10 monclient(hunting): handle_auth_done global_id 964118 payload 824 2024-09-16T04:02:33.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.679+0000 7f5d49d9b640 1 -- 172.21.15.42:0/3411136308 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f5d4c06b420 con 0x7f5d4c06f440 2024-09-16T04:02:33.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.679+0000 7f5d49d9b640 1 --2- 172.21.15.42:0/3411136308 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5d4c06f440 0x7f5d4c02cb90 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f5d4c122f60 tx=0x7f5d40000ee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:02:33.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.679+0000 7f5d3affd640 10 client.825522.objecter ms_handle_connect 0x7f5d4c06f440 2024-09-16T04:02:33.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.679+0000 7f5d3affd640 10 client.825522.objecter resend_mon_ops 2024-09-16T04:02:33.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.679+0000 7f5d3affd640 1 -- 172.21.15.42:0/3411136308 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d40007240 con 0x7f5d4c06f440 2024-09-16T04:02:33.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.679+0000 7f5d3affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:02:33.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.679+0000 7f5d3affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:02:33.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.679+0000 7f5d3affd640 10 monclient: dump: 2024-09-16T04:02:33.699 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:02:33.699 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:02:33.699 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:33.699 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:33.699 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:02:33.699 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:02:33.699 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:02:33.699 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:02:33.699 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:02:33.699 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:02:33.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.683+0000 7f5d3affd640 1 -- 172.21.15.42:0/3411136308 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4544 (secure 0 0 0) 0x7f5d40007240 con 0x7f5d4c06f440 2024-09-16T04:02:33.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.683+0000 7f5d3affd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:02:33.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.683+0000 7f5d3affd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:02:33.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.683+0000 7f5d3affd640 1 -- 172.21.15.42:0/3411136308 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5d4c06f440 msgr2=0x7f5d4c02cb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:02:33.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.683+0000 7f5d3affd640 1 --2- 172.21.15.42:0/3411136308 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5d4c06f440 0x7f5d4c02cb90 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f5d4c122f60 tx=0x7f5d40000ee0 comp rx=0 tx=0).stop 2024-09-16T04:02:33.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.683+0000 7f5d3affd640 1 -- 172.21.15.42:0/3411136308 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5d4c06f440 msgr2=0x7f5d4c02cb90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:02:33.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.683+0000 7f5d3affd640 1 --2- 172.21.15.42:0/3411136308 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5d4c06f440 0x7f5d4c02cb90 secure :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f5d4c122f60 tx=0x7f5d40000ee0 comp rx=0 tx=0).stop 2024-09-16T04:02:33.700 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T04:02:33.700 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:02:33.700 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:02:33.700 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T04:02:33.700 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 184, 2024-09-16T04:02:33.700 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 7415, 2024-09-16T04:02:33.700 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T04:02:33.701 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T04:02:33.701 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T04:02:33.701 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T04:02:33.701 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:02:33.701 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 2, 2024-09-16T04:02:33.701 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:02:33.701 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T04:02:33.701 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T04:02:33.701 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:02:33.701 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:02:33.701 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:02:33.701 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:02:33.701 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:02:33.702 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:02:33.702 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:02:33.702 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:02:33.702 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:02:33.702 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:02:33.702 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:02:33.702 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:02:33.702 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:02:33.702 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T04:02:33.702 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:02:33.702 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:02:33.702 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:02:33.702 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:02:33.702 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:02:33.703 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:02:33.703 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:02:33.703 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:02:33.703 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:02:33.703 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:02:33.703 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:02:33.703 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:02:33.703 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:02:33.703 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T04:02:33.703 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T04:02:33.703 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T04:02:33.703 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T04:02:33.703 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T04:02:33.704 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T04:02:33.704 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:02:33.704 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:02:33.704 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T04:02:33.704 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T04:02:33.704 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T04:02:33.704 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T04:02:33.704 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T04:02:33.704 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T04:02:33.704 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T04:02:33.704 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:02:33.704 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T04:02:33.704 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:02:33.704 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:02:33.705 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:02:33.705 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:02:33.705 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:02:33.705 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:02:33.705 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:02:33.705 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:02:33.705 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:02:33.705 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:02:33.705 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:02:33.705 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:02:33.705 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T04:02:33.705 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:02:33.705 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T04:02:33.706 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:02:33.706 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:02:33.706 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:02:33.706 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:02:33.706 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:02:33.706 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:02:33.706 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:02:33.706 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T04:02:33.706 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:02:33.706 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:02:33.706 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:02:33.706 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:02:33.706 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T04:02:33.706 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:02:33.707 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:02:33.707 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:02:33.707 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:02:33.707 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T04:02:33.707 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T04:02:33.707 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:02:33.707 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:02:33.707 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:02:33.707 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:02:33.707 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:02:33.707 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:02:33.707 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:02:33.707 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:02:33.707 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:02:33.707 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:02:33.708 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:02:33.708 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T04:02:33.708 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:02:33.708 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:02:33.708 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:02:33.708 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:02:33.708 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T04:02:33.708 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:02:33.708 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:02:33.708 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:02:33.708 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:02:33.708 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T04:02:33.708 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T04:02:33.709 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:02:33.709 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:02:33.709 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:02:33.709 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:02:33.709 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:02:33.709 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:02:33.709 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:02:33.709 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:02:33.709 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:02:33.709 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:02:33.709 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:02:33.709 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T04:02:33.709 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:02:33.709 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:02:33.710 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:02:33.710 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:02:33.710 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T04:02:33.710 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:02:33.710 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:02:33.710 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:02:33.710 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:02:33.710 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T04:02:33.710 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T04:02:33.710 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:02:33.710 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:02:33.710 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:02:33.710 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:02:33.710 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:02:33.711 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:02:33.711 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T04:02:33.711 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T04:02:33.711 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:02:33.711 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:02:33.711 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:02:33.711 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:02:33.711 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:02:33.711 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:02:33.711 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T04:02:33.711 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:02:33.711 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:02:33.711 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:02:33.711 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:02:33.712 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:02:33.712 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:02:33.712 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:02:33.712 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T04:02:33.712 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T04:02:33.714 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.683+0000 7f5d50fe9640 10 librados: watch_flush enter 2024-09-16T04:02:33.714 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.683+0000 7f5d50fe9640 10 librados: watch_flush exit 2024-09-16T04:02:33.714 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.683+0000 7f5d50fe9640 20 client.825522.objecter shutdown clearing up homeless session... 2024-09-16T04:02:33.714 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.683+0000 7f5d50fe9640 10 client.825522.objecter successfully canceled tick 2024-09-16T04:02:33.714 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.683+0000 7f5d50fe9640 1 -- 172.21.15.42:0/3411136308 >> v2:172.21.15.154:6812/254845342 conn(0x7f5d2c037840 msgr2=0x7f5d2c039d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:02:33.714 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.683+0000 7f5d50fe9640 1 --2- 172.21.15.42:0/3411136308 >> v2:172.21.15.154:6812/254845342 conn(0x7f5d2c037840 0x7f5d2c039d00 secure :-1 s=READY pgs=3784 cs=0 l=1 rev1=1 crypto rx=0x7f5d440ae000 tx=0x7f5d440a7070 comp rx=0 tx=0).stop 2024-09-16T04:02:33.714 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.683+0000 7f5d50fe9640 10 monclient: shutdown 2024-09-16T04:02:33.714 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.683+0000 7f5d50fe9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:02:33.714 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.683+0000 7f5d50fe9640 1 -- 172.21.15.42:0/3411136308 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5d4c09ca60 msgr2=0x7f5d4c1cedc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:02:33.714 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.683+0000 7f5d50fe9640 1 --2- 172.21.15.42:0/3411136308 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5d4c09ca60 0x7f5d4c1cedc0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f5d3c00e870 tx=0x7f5d3c00ec70 comp rx=0 tx=0).stop 2024-09-16T04:02:33.714 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.683+0000 7f5d4ad9d640 1 -- 172.21.15.42:0/3411136308 reap_dead start 2024-09-16T04:02:33.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.683+0000 7f5d4ad9d640 1 -- 172.21.15.42:0/3411136308 reap_dead start 2024-09-16T04:02:33.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.683+0000 7f5d50fe9640 1 -- 172.21.15.42:0/3411136308 shutdown_connections 2024-09-16T04:02:33.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.683+0000 7f5d50fe9640 1 -- 172.21.15.42:0/3411136308 >> 172.21.15.42:0/3411136308 conn(0x7f5d4c0a6640 msgr2=0x7f5d4c072a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:02:33.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.683+0000 7f5d50fe9640 1 -- 172.21.15.42:0/3411136308 shutdown_connections 2024-09-16T04:02:33.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.683+0000 7f5d50fe9640 1 -- 172.21.15.42:0/3411136308 wait complete. 2024-09-16T04:02:33.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.683+0000 7f5d50fe9640 1 librados: shutdown 2024-09-16T04:02:33.715 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:02:33.715 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T04:02:33.715 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T04:02:33.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.819+0000 7f31a03ab640 10 monclient: get_monmap_and_config 2024-09-16T04:02:33.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.819+0000 7f31a03ab640 10 monclient: build_initial_monmap 2024-09-16T04:02:33.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.819+0000 7f31a03ab640 10 monclient: monmap: 2024-09-16T04:02:33.822 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:02:33.822 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:02:33.822 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:02:33.820417+0000 2024-09-16T04:02:33.822 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:02:33.820417+0000 2024-09-16T04:02:33.822 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:02:33.822 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:02:33.822 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:02:33.822 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:02:33.822 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:02:33.822 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:02:33.822 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:02:33.822 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:02:33.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.819+0000 7f31a03ab640 1 Processor -- start 2024-09-16T04:02:33.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.819+0000 7f31a03ab640 1 -- start start 2024-09-16T04:02:33.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.819+0000 7f31a03ab640 10 monclient: init 2024-09-16T04:02:33.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.819+0000 7f31a03ab640 10 monclient: _reopen_session rank -1 2024-09-16T04:02:33.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.819+0000 7f31a03ab640 10 monclient: _add_conns ranks=[4,0,1,3,2] 2024-09-16T04:02:33.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.819+0000 7f31a03ab640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f31980a39b0 0x7f3198133b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:33.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.819+0000 7f31a03ab640 10 monclient(hunting): picked mon.noname-c con 0x7f31980a39b0 addr v2:172.21.15.154:3300/0 2024-09-16T04:02:33.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.819+0000 7f31a03ab640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f31980a3d90 addr v1:172.21.15.42:6789/0 2024-09-16T04:02:33.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.819+0000 7f31a03ab640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3198137710 addr v1:172.21.15.154:6789/0 2024-09-16T04:02:33.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.819+0000 7f31a03ab640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f319809c140 con 0x7f31980a3d90 2024-09-16T04:02:33.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.819+0000 7f31a03ab640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f319809c420 con 0x7f3198137710 2024-09-16T04:02:33.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.819+0000 7f31a03ab640 10 monclient(hunting): start opening mon connection 2024-09-16T04:02:33.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.819+0000 7f31a03ab640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f319806dd60 con 0x7f31980a39b0 2024-09-16T04:02:33.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.819+0000 7f31a03ab640 10 monclient(hunting): _renew_subs 2024-09-16T04:02:33.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.819+0000 7f31a03ab640 10 monclient(hunting): authenticate will time out at 2200.789843s 2024-09-16T04:02:33.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.819+0000 7f319d946640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f31980a3d90 0x7f3198137000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:41276/0 (socket says 172.21.15.42:41276) 2024-09-16T04:02:33.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.819+0000 7f319d946640 1 -- 172.21.15.42:0/218189115 learned_addr learned my addr 172.21.15.42:0/218189115 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T04:02:33.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.819+0000 7f319e147640 1 --2- 172.21.15.42:0/218189115 >> v2:172.21.15.154:3300/0 conn(0x7f31980a39b0 0x7f3198133b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:02:33.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.819+0000 7f319e147640 10 monclient(hunting): get_auth_request con 0x7f31980a39b0 auth_method 0 2024-09-16T04:02:33.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.819+0000 7f319e147640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:02:33.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.819+0000 7f319e147640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:02:33.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.819+0000 7f319e147640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:02:33.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.819+0000 7f319d145640 1 -- 172.21.15.42:0/218189115 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1471256340 0 0) 0x7f319809c140 con 0x7f31980a3d90 2024-09-16T04:02:33.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.819+0000 7f319d145640 10 monclient(hunting): _negotiate 2024-09-16T04:02:33.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.819+0000 7f319d145640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:02:33.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.819+0000 7f319d145640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:02:33.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.819+0000 7f319d145640 10 monclient(hunting): my global_id is 825525 2024-09-16T04:02:33.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.819+0000 7f319d145640 1 -- 172.21.15.42:0/218189115 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3180003530 con 0x7f31980a3d90 2024-09-16T04:02:33.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.819+0000 7f319e147640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:02:33.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.819+0000 7f319e147640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:02:33.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.819+0000 7f319e147640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:02:33.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.819+0000 7f319d145640 1 -- 172.21.15.42:0/218189115 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2367689951 0 0) 0x7f3180003530 con 0x7f31980a3d90 2024-09-16T04:02:33.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.819+0000 7f319d145640 1 -- 172.21.15.42:0/218189115 >> v2:172.21.15.154:3300/0 conn(0x7f31980a39b0 msgr2=0x7f3198133b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:02:33.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.819+0000 7f319d145640 1 --2- 172.21.15.42:0/218189115 >> v2:172.21.15.154:3300/0 conn(0x7f31980a39b0 0x7f3198133b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:33.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.819+0000 7f319d145640 1 -- 172.21.15.42:0/218189115 >> v1:172.21.15.154:6789/0 conn(0x7f3198137710 legacy=0x7f319813f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:02:33.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.819+0000 7f319d145640 10 monclient: _finish_hunting 0 2024-09-16T04:02:33.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.819+0000 7f319d145640 1 monclient: found mon.noname-a-legacy 2024-09-16T04:02:33.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.819+0000 7f319d145640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:02:33.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.819+0000 7f319d145640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.42:6789/0 2024-09-16T04:02:33.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.819+0000 7f319d145640 1 -- 172.21.15.42:0/218189115 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31980a5e20 con 0x7f31980a3d90 2024-09-16T04:02:33.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.819+0000 7f319d145640 10 monclient: _finish_auth 0 2024-09-16T04:02:33.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.819+0000 7f319d145640 10 monclient: _check_auth_tickets 2024-09-16T04:02:33.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.819+0000 7f319d145640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:02:33.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.819+0000 7f319d145640 1 -- 172.21.15.42:0/218189115 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2177413734 0 0) 0x7f318c001680 con 0x7f31980a3d90 2024-09-16T04:02:33.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.819+0000 7f319d145640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:02:33.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.819+0000 7f319d145640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T04:02:33.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.819+0000 7f319d145640 10 monclient: dump: 2024-09-16T04:02:33.837 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:02:33.837 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:02:33.837 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:33.837 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:33.837 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:02:33.837 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:02:33.837 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:02:33.837 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:02:33.837 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:02:33.838 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:02:33.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.819+0000 7f319d145640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] but i'm connected to v1:172.21.15.42:6789/0, reconnecting 2024-09-16T04:02:33.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.819+0000 7f319d145640 10 monclient: _reopen_session rank -1 2024-09-16T04:02:33.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.819+0000 7f319d145640 1 -- 172.21.15.42:0/218189115 >> v1:172.21.15.42:6789/0 conn(0x7f31980a3d90 legacy=0x7f3198137000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:02:33.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.819+0000 7f319d145640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:02:33.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.819+0000 7f319d145640 1 --2- 172.21.15.42:0/218189115 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f31800039c0 0x7f3180003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:33.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.819+0000 7f319d145640 10 monclient(hunting): picked mon.b con 0x7f31800039c0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:02:33.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.819+0000 7f319d145640 1 --2- 172.21.15.42:0/218189115 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3180004810 0x7f3180004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:33.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.819+0000 7f319d145640 10 monclient(hunting): picked mon.c con 0x7f3180004810 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:02:33.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.819+0000 7f319d145640 1 --2- 172.21.15.42:0/218189115 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f31800052f0 0x7f318000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:33.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.819+0000 7f319d145640 10 monclient(hunting): picked mon.a con 0x7f31800052f0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:02:33.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.819+0000 7f319d145640 10 monclient(hunting): start opening mon connection 2024-09-16T04:02:33.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.819+0000 7f319d145640 1 -- 172.21.15.42:0/218189115 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f319806dd60 con 0x7f31800052f0 2024-09-16T04:02:33.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.819+0000 7f319d145640 10 monclient(hunting): start opening mon connection 2024-09-16T04:02:33.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.819+0000 7f319d145640 1 -- 172.21.15.42:0/218189115 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f318000c1c0 con 0x7f31800039c0 2024-09-16T04:02:33.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.819+0000 7f319d145640 10 monclient(hunting): start opening mon connection 2024-09-16T04:02:33.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.819+0000 7f319d145640 1 -- 172.21.15.42:0/218189115 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f318000c3c0 con 0x7f3180004810 2024-09-16T04:02:33.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.819+0000 7f319d145640 10 monclient(hunting): _renew_subs 2024-09-16T04:02:33.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.819+0000 7f319d145640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:02:33.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.819+0000 7f319e948640 1 --2- 172.21.15.42:0/218189115 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f31800052f0 0x7f318000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:02:33.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.819+0000 7f319d946640 1 --2- 172.21.15.42:0/218189115 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3180004810 0x7f3180004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:02:33.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.819+0000 7f319e948640 10 monclient(hunting): get_auth_request con 0x7f31800052f0 auth_method 0 2024-09-16T04:02:33.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.819+0000 7f319e948640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:02:33.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.819+0000 7f319e948640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:02:33.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.819+0000 7f319e948640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:02:33.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.819+0000 7f319d946640 10 monclient(hunting): get_auth_request con 0x7f3180004810 auth_method 0 2024-09-16T04:02:33.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.819+0000 7f319d946640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:02:33.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.819+0000 7f319d946640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:02:33.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.819+0000 7f319d946640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:02:33.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.819+0000 7f319e948640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:02:33.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.819+0000 7f319e948640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:02:33.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.819+0000 7f319e948640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:02:33.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.819+0000 7f319d946640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:02:33.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.819+0000 7f319d946640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:02:33.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.819+0000 7f319d946640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:02:33.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.819+0000 7f319e147640 1 --2- 172.21.15.42:0/218189115 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f31800039c0 0x7f3180003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:02:33.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.819+0000 7f319e147640 10 monclient(hunting): get_auth_request con 0x7f31800039c0 auth_method 0 2024-09-16T04:02:33.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.819+0000 7f319e147640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:02:33.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.819+0000 7f319e147640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:02:33.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.819+0000 7f319e147640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:02:33.841 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.819+0000 7f319e948640 10 monclient(hunting): handle_auth_done global_id 825525 payload 293 2024-09-16T04:02:33.841 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.819+0000 7f319e948640 1 -- 172.21.15.42:0/218189115 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3180004810 msgr2=0x7f3180004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:02:33.841 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.819+0000 7f319e948640 1 --2- 172.21.15.42:0/218189115 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3180004810 0x7f3180004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:33.841 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.819+0000 7f319e948640 1 -- 172.21.15.42:0/218189115 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f31800039c0 msgr2=0x7f3180003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:02:33.841 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.819+0000 7f319e948640 1 --2- 172.21.15.42:0/218189115 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f31800039c0 0x7f3180003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:33.841 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.819+0000 7f319e948640 10 monclient: _finish_hunting 0 2024-09-16T04:02:33.841 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.819+0000 7f319e948640 1 monclient: found mon.a 2024-09-16T04:02:33.841 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.819+0000 7f319e948640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:02:33.841 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.819+0000 7f319e948640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:02:33.841 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.819+0000 7f319e948640 1 -- 172.21.15.42:0/218189115 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3180003530 con 0x7f31800052f0 2024-09-16T04:02:33.841 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.819+0000 7f319e948640 10 monclient: _finish_auth 0 2024-09-16T04:02:33.841 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.819+0000 7f319e948640 10 monclient: _check_auth_tickets 2024-09-16T04:02:33.841 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.819+0000 7f319e948640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:02:33.841 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.819+0000 7f31a03ab640 5 monclient: authenticate success, global_id 825525 2024-09-16T04:02:33.842 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.819+0000 7f31a03ab640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:02:33.842 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.819+0000 7f319d946640 1 --2- 172.21.15.42:0/218189115 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3180004810 0x7f3180004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:02:33.842 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.823+0000 7f319e948640 1 -- 172.21.15.42:0/218189115 reap_dead start 2024-09-16T04:02:33.842 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.823+0000 7f319e948640 1 --2- 172.21.15.42:0/218189115 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f31800052f0 0x7f318000b9d0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f3190001750 tx=0x7f319001c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62cd4e791e824791 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:02:33.842 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.823+0000 7f319d145640 1 -- 172.21.15.42:0/218189115 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f318c007d40 con 0x7f31800052f0 2024-09-16T04:02:33.842 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.823+0000 7f319d145640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:02:33.842 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.823+0000 7f319d145640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:02:33.842 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.823+0000 7f319d145640 10 monclient: dump: 2024-09-16T04:02:33.842 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:02:33.842 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:02:33.842 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:33.842 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:33.842 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:02:33.842 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:02:33.843 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:02:33.843 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:02:33.843 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:02:33.843 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:02:33.843 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.823+0000 7f31a03ab640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:02:33.843 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.823+0000 7f319d145640 1 -- 172.21.15.42:0/218189115 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f31900052d0 con 0x7f31800052f0 2024-09-16T04:02:33.843 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.823+0000 7f319d145640 10 monclient: handle_config config(0 keys) 2024-09-16T04:02:33.843 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.823+0000 7f31a03ab640 10 monclient: get_monmap_and_config success 2024-09-16T04:02:33.843 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.823+0000 7f31a03ab640 4 set_mon_vals no callback set 2024-09-16T04:02:33.843 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.823+0000 7f319d145640 1 -- 172.21.15.42:0/218189115 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31900013a0 con 0x7f31800052f0 2024-09-16T04:02:33.843 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.823+0000 7f319d145640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:02:33.843 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.823+0000 7f319d145640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:02:33.843 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.823+0000 7f319d145640 10 monclient: dump: 2024-09-16T04:02:33.843 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:02:33.844 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:02:33.844 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:33.844 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:33.844 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:02:33.844 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:02:33.844 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:02:33.844 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:02:33.844 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:02:33.844 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:02:33.844 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.823+0000 7f31a03ab640 10 monclient: shutdown 2024-09-16T04:02:33.844 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.823+0000 7f31a03ab640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:02:33.844 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.823+0000 7f31a03ab640 1 -- 172.21.15.42:0/218189115 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f31800052f0 msgr2=0x7f318000b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:02:33.844 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.823+0000 7f31a03ab640 1 --2- 172.21.15.42:0/218189115 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f31800052f0 0x7f318000b9d0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f3190001750 tx=0x7f319001c3c0 comp rx=0 tx=0).stop 2024-09-16T04:02:33.845 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.823+0000 7f31a03ab640 1 -- 172.21.15.42:0/218189115 shutdown_connections 2024-09-16T04:02:33.845 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.823+0000 7f31a03ab640 1 --2- 172.21.15.42:0/218189115 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f31800052f0 0x7f318000b9d0 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:33.845 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.823+0000 7f31a03ab640 1 -- 172.21.15.42:0/218189115 >> 172.21.15.42:0/218189115 conn(0x7f31980a9fb0 msgr2=0x7f31980aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:02:33.845 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.823+0000 7f31a03ab640 1 -- 172.21.15.42:0/218189115 shutdown_connections 2024-09-16T04:02:33.845 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.823+0000 7f31a03ab640 1 -- 172.21.15.42:0/218189115 wait complete. 2024-09-16T04:02:33.845 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.823+0000 7f31a03ab640 10 monclient: build_initial_monmap 2024-09-16T04:02:33.845 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.823+0000 7f31a03ab640 10 monclient: monmap: 2024-09-16T04:02:33.845 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:02:33.845 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:02:33.845 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:02:33.845 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:02:33.845 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:02:33.845 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:02:33.845 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:02:33.846 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:02:33.846 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:02:33.846 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:02:33.846 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.823+0000 7f31a03ab640 1 librados: starting msgr at 2024-09-16T04:02:33.846 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.823+0000 7f31a03ab640 1 librados: starting objecter 2024-09-16T04:02:33.846 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.823+0000 7f31a03ab640 1 Processor -- start 2024-09-16T04:02:33.846 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.823+0000 7f31a03ab640 1 -- start start 2024-09-16T04:02:33.846 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.823+0000 7f31a03ab640 1 librados: setting wanted keys 2024-09-16T04:02:33.846 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.823+0000 7f31a03ab640 1 librados: calling monclient init 2024-09-16T04:02:33.846 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.823+0000 7f31a03ab640 10 monclient: init 2024-09-16T04:02:33.846 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.823+0000 7f31a03ab640 10 monclient: _reopen_session rank -1 2024-09-16T04:02:33.846 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.823+0000 7f31a03ab640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:02:33.846 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.823+0000 7f31a03ab640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f31800052f0 0x7f3198130a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:33.846 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.823+0000 7f31a03ab640 10 monclient(hunting): picked mon.noname-c con 0x7f31800052f0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:02:33.847 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.823+0000 7f31a03ab640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3198130f60 0x7f31980a1b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:33.847 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.823+0000 7f31a03ab640 10 monclient(hunting): picked mon.noname-b con 0x7f3198130f60 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:02:33.847 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.823+0000 7f31a03ab640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f31980a20c0 0x7f31980a2540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:33.847 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.823+0000 7f31a03ab640 10 monclient(hunting): picked mon.noname-a con 0x7f31980a20c0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:02:33.847 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.823+0000 7f31a03ab640 10 monclient(hunting): start opening mon connection 2024-09-16T04:02:33.847 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.823+0000 7f31a03ab640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f3198133f10 con 0x7f31980a20c0 2024-09-16T04:02:33.847 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.823+0000 7f31a03ab640 10 monclient(hunting): start opening mon connection 2024-09-16T04:02:33.847 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.823+0000 7f31a03ab640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f3198133d90 con 0x7f3198130f60 2024-09-16T04:02:33.847 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.823+0000 7f31a03ab640 10 monclient(hunting): start opening mon connection 2024-09-16T04:02:33.847 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.823+0000 7f31a03ab640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f319806dee0 con 0x7f31800052f0 2024-09-16T04:02:33.847 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.823+0000 7f31a03ab640 10 monclient(hunting): _renew_subs 2024-09-16T04:02:33.847 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.823+0000 7f31a03ab640 10 monclient(hunting): authenticate will time out at 2200.795341s 2024-09-16T04:02:33.847 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.823+0000 7f319e147640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f31800052f0 0x7f3198130a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:02:33.847 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.823+0000 7f319e948640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f31980a20c0 0x7f31980a2540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:02:33.848 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.823+0000 7f319e147640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f31800052f0 0x7f3198130a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:36350/0 (socket says 172.21.15.42:36350) 2024-09-16T04:02:33.848 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.823+0000 7f319e147640 1 -- 172.21.15.42:0/3685414743 learned_addr learned my addr 172.21.15.42:0/3685414743 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:02:33.848 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.823+0000 7f319e147640 10 monclient(hunting): get_auth_request con 0x7f31800052f0 auth_method 0 2024-09-16T04:02:33.848 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.823+0000 7f319e147640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:02:33.848 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.823+0000 7f319e147640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:02:33.848 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.823+0000 7f319e147640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:02:33.848 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.823+0000 7f319e948640 10 monclient(hunting): get_auth_request con 0x7f31980a20c0 auth_method 0 2024-09-16T04:02:33.848 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.823+0000 7f319e948640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:02:33.848 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.823+0000 7f319e948640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:02:33.848 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.823+0000 7f319e948640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:02:33.848 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.823+0000 7f319e147640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:02:33.848 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.823+0000 7f319e147640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:02:33.848 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.823+0000 7f319e147640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:02:33.848 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.823+0000 7f319e948640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:02:33.849 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.823+0000 7f319e948640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:02:33.849 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.823+0000 7f319e948640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:02:33.849 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.823+0000 7f319e147640 10 monclient(hunting): handle_auth_done global_id 964121 payload 824 2024-09-16T04:02:33.849 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.823+0000 7f319e147640 1 -- 172.21.15.42:0/3685414743 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3198130f60 msgr2=0x7f31980a1b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:02:33.849 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.823+0000 7f319e147640 1 --2- 172.21.15.42:0/3685414743 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3198130f60 0x7f31980a1b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:33.849 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.823+0000 7f319e147640 1 -- 172.21.15.42:0/3685414743 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f31980a20c0 msgr2=0x7f31980a2540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:02:33.849 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.823+0000 7f319e147640 1 --2- 172.21.15.42:0/3685414743 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f31980a20c0 0x7f31980a2540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:33.849 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.823+0000 7f319e147640 10 monclient: _finish_hunting 0 2024-09-16T04:02:33.849 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.823+0000 7f319e147640 1 monclient: found mon.noname-c 2024-09-16T04:02:33.849 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.823+0000 7f319e147640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:02:33.849 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.823+0000 7f319e147640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T04:02:33.849 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.823+0000 7f319e147640 1 -- 172.21.15.42:0/3685414743 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f319806f680 con 0x7f31800052f0 2024-09-16T04:02:33.849 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.823+0000 7f319e147640 1 --2- 172.21.15.42:0/3685414743 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f31800052f0 0x7f3198130a20 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f3188009fb0 tx=0x7f31880115a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:02:33.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.823+0000 7f3196ffd640 10 client.?.objecter ms_handle_connect 0x7f31800052f0 2024-09-16T04:02:33.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.823+0000 7f3196ffd640 10 client.?.objecter resend_mon_ops 2024-09-16T04:02:33.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.827+0000 7f3196ffd640 1 -- 172.21.15.42:0/3685414743 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f318801a020 con 0x7f31800052f0 2024-09-16T04:02:33.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.827+0000 7f3196ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:02:33.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.827+0000 7f3196ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:02:33.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.827+0000 7f3196ffd640 10 monclient: dump: 2024-09-16T04:02:33.850 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:02:33.850 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:02:33.850 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:33.850 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:33.850 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:02:33.850 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:02:33.850 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:02:33.850 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:02:33.851 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:02:33.851 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:02:33.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.827+0000 7f3196ffd640 10 monclient: _finish_auth 0 2024-09-16T04:02:33.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.827+0000 7f3196ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:02:33.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.827+0000 7f3196ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:02:33.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.827+0000 7f3196ffd640 1 -- 172.21.15.42:0/3685414743 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3188016470 con 0x7f31800052f0 2024-09-16T04:02:33.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.827+0000 7f31a03ab640 5 monclient: authenticate success, global_id 964121 2024-09-16T04:02:33.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.827+0000 7f31a03ab640 10 monclient: _renew_subs 2024-09-16T04:02:33.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.827+0000 7f31a03ab640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:02:33.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.827+0000 7f31a03ab640 1 -- 172.21.15.42:0/3685414743 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f319813a140 con 0x7f31800052f0 2024-09-16T04:02:33.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.827+0000 7f3196ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:02:33.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.827+0000 7f3196ffd640 1 -- 172.21.15.42:0/3685414743 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3188016860 con 0x7f31800052f0 2024-09-16T04:02:33.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.827+0000 7f3196ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:02:33.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.827+0000 7f3196ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:02:33.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.827+0000 7f3196ffd640 10 monclient: dump: 2024-09-16T04:02:33.852 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:02:33.852 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:02:33.852 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:33.852 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:33.852 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:02:33.852 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:02:33.852 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:02:33.852 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:02:33.852 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:02:33.852 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:02:33.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.827+0000 7f319c944640 4 set_mon_vals no callback set 2024-09-16T04:02:33.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.827+0000 7f31a03ab640 10 client.964121.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:02:33.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.827+0000 7f31a03ab640 10 monclient: _renew_subs 2024-09-16T04:02:33.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.827+0000 7f31a03ab640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:02:33.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.827+0000 7f31a03ab640 1 -- 172.21.15.42:0/3685414743 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f31980a3650 con 0x7f31800052f0 2024-09-16T04:02:33.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.827+0000 7f31a03ab640 1 librados: init done 2024-09-16T04:02:33.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.827+0000 7f31a03ab640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:02:33.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.827+0000 7f31a03ab640 1 --2- 172.21.15.42:0/3685414743 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f319802d7d0 0x7f31980a2a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:33.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.827+0000 7f31a03ab640 10 monclient(hunting): start opening mon connection 2024-09-16T04:02:33.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.827+0000 7f31a03ab640 1 -- 172.21.15.42:0/3685414743 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f319802dbb0 con 0x7f319802d7d0 2024-09-16T04:02:33.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.827+0000 7f319d946640 1 --2- 172.21.15.42:0/3685414743 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f319802d7d0 0x7f31980a2a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:02:33.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.827+0000 7f319d946640 10 monclient: get_auth_request con 0x7f319802d7d0 auth_method 0 2024-09-16T04:02:33.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.827+0000 7f319d946640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:02:33.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.827+0000 7f319d946640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:02:33.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.827+0000 7f319d946640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:02:33.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.827+0000 7f3196ffd640 1 -- 172.21.15.42:0/3685414743 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f318801d020 con 0x7f31800052f0 2024-09-16T04:02:33.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.827+0000 7f319d946640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:02:33.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.827+0000 7f319d946640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:02:33.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.827+0000 7f319d946640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:02:33.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.827+0000 7f3196ffd640 1 --2- 172.21.15.42:0/3685414743 >> v2:172.21.15.154:6812/254845342 conn(0x7f3184037810 0x7f3184039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:33.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.827+0000 7f3196ffd640 1 -- 172.21.15.42:0/3685414743 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(630..630 src has 451..630) ==== 6280+0+0 (secure 0 0 0) 0x7f318804a0b0 con 0x7f31800052f0 2024-09-16T04:02:33.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.827+0000 7f3196ffd640 10 client.964121.objecter ms_dispatch 0x7f3198000ba0 osd_map(630..630 src has 451..630) 2024-09-16T04:02:33.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.827+0000 7f3196ffd640 3 client.964121.objecter handle_osd_map got epochs [630,630] > 0 2024-09-16T04:02:33.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.827+0000 7f3196ffd640 3 client.964121.objecter handle_osd_map decoding full epoch 630 2024-09-16T04:02:33.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.827+0000 7f3196ffd640 20 client.964121.objecter dump_active .. 0 homeless 2024-09-16T04:02:33.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.827+0000 7f319d946640 10 monclient(hunting): handle_auth_done global_id 904612 payload 824 2024-09-16T04:02:33.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.827+0000 7f319d946640 1 -- 172.21.15.42:0/3685414743 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3198137000 con 0x7f319802d7d0 2024-09-16T04:02:33.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.827+0000 7f319e948640 1 --2- 172.21.15.42:0/3685414743 >> v2:172.21.15.154:6812/254845342 conn(0x7f3184037810 0x7f3184039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:02:33.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.827+0000 7f319e948640 10 monclient: get_auth_request con 0x7f3184037810 auth_method 0 2024-09-16T04:02:33.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.827+0000 7f3196ffd640 10 client.964121.objecter ms_handle_connect 0x7f319802d7d0 2024-09-16T04:02:33.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.827+0000 7f3196ffd640 10 client.964121.objecter resend_mon_ops 2024-09-16T04:02:33.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.827+0000 7f319d946640 1 --2- 172.21.15.42:0/3685414743 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f319802d7d0 0x7f31980a2a80 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f318c000c10 tx=0x7f318c004f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:02:33.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.827+0000 7f3196ffd640 1 -- 172.21.15.42:0/3685414743 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f318c00f600 con 0x7f319802d7d0 2024-09-16T04:02:33.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.827+0000 7f3196ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:02:33.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.827+0000 7f3196ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:02:33.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.827+0000 7f3196ffd640 10 monclient: dump: 2024-09-16T04:02:33.855 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:02:33.855 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:02:33.855 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:33.855 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:33.855 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:02:33.855 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:02:33.855 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:02:33.855 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:02:33.856 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:02:33.856 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:02:33.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.827+0000 7f3196ffd640 10 client.964121.objecter ms_handle_connect 0x7f3184037810 2024-09-16T04:02:33.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.827+0000 7f319e948640 1 --2- 172.21.15.42:0/3685414743 >> v2:172.21.15.154:6812/254845342 conn(0x7f3184037810 0x7f3184039cd0 secure :-1 s=READY pgs=3785 cs=0 l=1 rev1=1 crypto rx=0x7f318000c190 tx=0x7f31900ad040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:02:33.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.827+0000 7f3196ffd640 1 -- 172.21.15.42:0/3685414743 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f318c018070 con 0x7f319802d7d0 2024-09-16T04:02:33.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.827+0000 7f3196ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:02:33.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.827+0000 7f3196ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:02:33.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.827+0000 7f3196ffd640 1 -- 172.21.15.42:0/3685414743 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f319802d7d0 msgr2=0x7f31980a2a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:02:33.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.827+0000 7f3196ffd640 1 --2- 172.21.15.42:0/3685414743 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f319802d7d0 0x7f31980a2a80 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f318c000c10 tx=0x7f318c004f90 comp rx=0 tx=0).stop 2024-09-16T04:02:33.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.827+0000 7f3196ffd640 1 -- 172.21.15.42:0/3685414743 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f319802d7d0 msgr2=0x7f31980a2a80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:02:33.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.827+0000 7f3196ffd640 1 --2- 172.21.15.42:0/3685414743 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f319802d7d0 0x7f31980a2a80 secure :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f318c000c10 tx=0x7f318c004f90 comp rx=0 tx=0).stop 2024-09-16T04:02:33.858 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.835+0000 7f31a03ab640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:02:33.858 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.835+0000 7f31a03ab640 1 --2- 172.21.15.42:0/3685414743 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3198062ea0 0x7f31980632b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:33.858 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.835+0000 7f31a03ab640 10 monclient(hunting): start opening mon connection 2024-09-16T04:02:33.858 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.835+0000 7f31a03ab640 1 -- 172.21.15.42:0/3685414743 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f31980a5750 con 0x7f3198062ea0 2024-09-16T04:02:33.858 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.835+0000 7f319d946640 1 --2- 172.21.15.42:0/3685414743 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3198062ea0 0x7f31980632b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:02:33.858 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.835+0000 7f319d946640 10 monclient: get_auth_request con 0x7f3198062ea0 auth_method 0 2024-09-16T04:02:33.858 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.835+0000 7f319d946640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:02:33.858 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.835+0000 7f319d946640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:02:33.859 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.835+0000 7f319d946640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:02:33.859 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.835+0000 7f319d946640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:02:33.859 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.835+0000 7f319d946640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:02:33.859 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.835+0000 7f319d946640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:02:33.859 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.835+0000 7f319d946640 10 monclient(hunting): handle_auth_done global_id 904615 payload 824 2024-09-16T04:02:33.859 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.835+0000 7f319d946640 1 -- 172.21.15.42:0/3685414743 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f319809dfe0 con 0x7f3198062ea0 2024-09-16T04:02:33.859 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.835+0000 7f319d946640 1 --2- 172.21.15.42:0/3685414743 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3198062ea0 0x7f31980632b0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f318c001ce0 tx=0x7f318c007900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:02:33.859 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.835+0000 7f3196ffd640 10 client.964121.objecter ms_handle_connect 0x7f3198062ea0 2024-09-16T04:02:33.859 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.835+0000 7f3196ffd640 10 client.964121.objecter resend_mon_ops 2024-09-16T04:02:33.859 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.835+0000 7f3196ffd640 1 -- 172.21.15.42:0/3685414743 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f318c027410 con 0x7f3198062ea0 2024-09-16T04:02:33.859 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.835+0000 7f3196ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:02:33.859 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.835+0000 7f3196ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:02:33.859 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.835+0000 7f3196ffd640 10 monclient: dump: 2024-09-16T04:02:33.860 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:02:33.860 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:02:33.860 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:33.860 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:33.860 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:02:33.860 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:02:33.860 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:02:33.860 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:02:33.860 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:02:33.860 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:02:33.861 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.835+0000 7f3196ffd640 1 -- 172.21.15.42:0/3685414743 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4880 (secure 0 0 0) 0x7f318c0133f0 con 0x7f3198062ea0 2024-09-16T04:02:33.861 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.835+0000 7f3196ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:02:33.861 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.835+0000 7f3196ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:02:33.861 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.835+0000 7f3196ffd640 1 -- 172.21.15.42:0/3685414743 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3198062ea0 msgr2=0x7f31980632b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:02:33.861 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.835+0000 7f3196ffd640 1 --2- 172.21.15.42:0/3685414743 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3198062ea0 0x7f31980632b0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f318c001ce0 tx=0x7f318c007900 comp rx=0 tx=0).stop 2024-09-16T04:02:33.861 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.835+0000 7f3196ffd640 1 -- 172.21.15.42:0/3685414743 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3198062ea0 msgr2=0x7f31980632b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:02:33.861 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.835+0000 7f3196ffd640 1 --2- 172.21.15.42:0/3685414743 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3198062ea0 0x7f31980632b0 secure :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f318c001ce0 tx=0x7f318c007900 comp rx=0 tx=0).stop 2024-09-16T04:02:33.861 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T04:02:33.861 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:02:33.861 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:02:33.861 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T04:02:33.862 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 184, 2024-09-16T04:02:33.862 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 282014, 2024-09-16T04:02:33.862 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T04:02:33.862 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T04:02:33.862 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T04:02:33.862 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T04:02:33.862 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:02:33.862 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 2, 2024-09-16T04:02:33.862 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:02:33.862 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T04:02:33.862 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T04:02:33.862 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:02:33.862 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:02:33.862 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:02:33.863 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:02:33.863 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:02:33.863 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:02:33.863 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:02:33.863 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:02:33.863 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:02:33.863 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:02:33.863 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:02:33.863 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:02:33.863 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:02:33.863 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T04:02:33.863 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:02:33.863 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:02:33.863 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:02:33.864 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:02:33.864 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:02:33.864 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:02:33.864 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:02:33.864 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:02:33.864 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:02:33.864 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:02:33.864 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:02:33.864 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:02:33.864 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:02:33.864 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T04:02:33.864 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T04:02:33.864 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T04:02:33.865 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T04:02:33.865 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T04:02:33.865 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T04:02:33.865 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:02:33.865 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:02:33.865 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T04:02:33.865 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T04:02:33.865 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T04:02:33.865 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T04:02:33.865 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T04:02:33.865 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T04:02:33.865 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T04:02:33.865 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:02:33.865 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T04:02:33.866 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:02:33.866 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:02:33.866 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:02:33.866 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:02:33.866 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:02:33.866 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:02:33.866 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:02:33.866 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:02:33.866 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:02:33.866 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:02:33.866 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:02:33.866 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:02:33.866 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T04:02:33.866 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:02:33.867 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T04:02:33.867 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:02:33.867 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:02:33.867 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:02:33.867 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:02:33.867 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:02:33.867 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:02:33.867 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:02:33.867 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T04:02:33.867 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:02:33.867 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:02:33.867 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:02:33.867 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:02:33.867 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T04:02:33.867 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:02:33.868 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:02:33.868 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:02:33.868 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:02:33.868 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T04:02:33.868 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T04:02:33.868 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:02:33.868 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:02:33.868 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:02:33.868 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:02:33.868 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:02:33.868 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:02:33.868 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:02:33.868 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:02:33.869 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:02:33.869 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:02:33.869 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:02:33.869 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T04:02:33.869 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:02:33.869 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:02:33.869 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:02:33.869 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:02:33.869 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T04:02:33.869 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:02:33.869 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:02:33.869 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:02:33.869 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:02:33.869 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T04:02:33.870 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T04:02:33.870 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:02:33.870 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:02:33.870 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:02:33.870 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:02:33.870 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:02:33.870 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:02:33.870 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:02:33.870 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:02:33.870 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:02:33.870 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:02:33.870 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:02:33.870 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T04:02:33.870 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:02:33.871 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:02:33.871 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:02:33.871 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:02:33.871 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T04:02:33.871 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:02:33.871 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:02:33.871 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:02:33.871 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:02:33.871 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T04:02:33.871 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T04:02:33.871 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:02:33.871 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:02:33.871 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:02:33.871 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:02:33.872 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:02:33.872 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:02:33.872 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T04:02:33.872 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T04:02:33.872 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:02:33.872 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:02:33.872 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:02:33.872 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:02:33.872 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:02:33.872 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:02:33.872 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T04:02:33.872 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:02:33.872 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:02:33.872 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:02:33.873 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:02:33.873 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:02:33.873 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:02:33.873 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T04:02:33.873 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:02:33.873 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:02:33.873 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:02:33.873 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 2 2024-09-16T04:02:33.873 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:02:33.873 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:02:33.873 INFO:teuthology.orchestra.run.smithi042.stdout: "mgr": [ 2024-09-16T04:02:33.873 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:02:33.873 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:02:33.873 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:02:33.874 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:02:33.874 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:02:33.874 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:02:33.874 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:02:33.874 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T04:02:33.874 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T04:02:33.875 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.839+0000 7f31a03ab640 10 librados: watch_flush enter 2024-09-16T04:02:33.876 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.839+0000 7f31a03ab640 10 librados: watch_flush exit 2024-09-16T04:02:33.876 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.839+0000 7f31a03ab640 20 client.964121.objecter shutdown clearing up homeless session... 2024-09-16T04:02:33.876 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.839+0000 7f31a03ab640 10 client.964121.objecter successfully canceled tick 2024-09-16T04:02:33.876 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.839+0000 7f31a03ab640 1 -- 172.21.15.42:0/3685414743 >> v2:172.21.15.154:6812/254845342 conn(0x7f3184037810 msgr2=0x7f3184039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:02:33.876 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.839+0000 7f31a03ab640 1 --2- 172.21.15.42:0/3685414743 >> v2:172.21.15.154:6812/254845342 conn(0x7f3184037810 0x7f3184039cd0 secure :-1 s=READY pgs=3785 cs=0 l=1 rev1=1 crypto rx=0x7f318000c190 tx=0x7f31900ad040 comp rx=0 tx=0).stop 2024-09-16T04:02:33.876 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.839+0000 7f31a03ab640 10 monclient: shutdown 2024-09-16T04:02:33.876 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.839+0000 7f31a03ab640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:02:33.876 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.839+0000 7f31a03ab640 1 -- 172.21.15.42:0/3685414743 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f31800052f0 msgr2=0x7f3198130a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:02:33.876 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.839+0000 7f31a03ab640 1 --2- 172.21.15.42:0/3685414743 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f31800052f0 0x7f3198130a20 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f3188009fb0 tx=0x7f31880115a0 comp rx=0 tx=0).stop 2024-09-16T04:02:33.876 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.839+0000 7f319e948640 1 -- 172.21.15.42:0/3685414743 reap_dead start 2024-09-16T04:02:33.876 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.839+0000 7f319e948640 1 -- 172.21.15.42:0/3685414743 reap_dead start 2024-09-16T04:02:33.876 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.839+0000 7f31a03ab640 1 -- 172.21.15.42:0/3685414743 shutdown_connections 2024-09-16T04:02:33.876 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.839+0000 7f31a03ab640 1 -- 172.21.15.42:0/3685414743 >> 172.21.15.42:0/3685414743 conn(0x7f31980a9fb0 msgr2=0x7f31980a9080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:02:33.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.839+0000 7f31a03ab640 1 -- 172.21.15.42:0/3685414743 shutdown_connections 2024-09-16T04:02:33.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.839+0000 7f31a03ab640 1 -- 172.21.15.42:0/3685414743 wait complete. 2024-09-16T04:02:33.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.839+0000 7f31a03ab640 1 librados: shutdown 2024-09-16T04:02:33.877 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:02:33.877 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T04:02:33.877 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T04:02:33.986 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.983+0000 7ff81cd75640 10 monclient: get_monmap_and_config 2024-09-16T04:02:33.986 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.983+0000 7ff81cd75640 10 monclient: build_initial_monmap 2024-09-16T04:02:33.986 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.983+0000 7ff81cd75640 10 monclient: monmap: 2024-09-16T04:02:33.986 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:02:33.987 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:02:33.987 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:02:33.985083+0000 2024-09-16T04:02:33.987 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:02:33.985083+0000 2024-09-16T04:02:33.987 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:02:33.987 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:02:33.987 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:02:33.987 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:02:33.987 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:02:33.987 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:02:33.987 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:02:33.987 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:02:33.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.983+0000 7ff81cd75640 1 Processor -- start 2024-09-16T04:02:33.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.983+0000 7ff81cd75640 1 -- start start 2024-09-16T04:02:33.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.983+0000 7ff81cd75640 10 monclient: init 2024-09-16T04:02:33.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.983+0000 7ff81cd75640 10 monclient: _reopen_session rank -1 2024-09-16T04:02:33.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.983+0000 7ff81cd75640 10 monclient: _add_conns ranks=[1,3,4,2,0] 2024-09-16T04:02:33.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.983+0000 7ff81cd75640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff8180a39b0 addr v1:172.21.15.154:6789/0 2024-09-16T04:02:33.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.983+0000 7ff81cd75640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff8180a3d90 0x7ff81813c380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:33.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.983+0000 7ff81cd75640 10 monclient(hunting): picked mon.noname-a con 0x7ff8180a3d90 addr v2:172.21.15.42:3300/0 2024-09-16T04:02:33.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.983+0000 7ff81cd75640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7ff81813c8c0 0x7ff81813ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:33.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.983+0000 7ff81cd75640 10 monclient(hunting): picked mon.noname-c con 0x7ff81813c8c0 addr v2:172.21.15.154:3300/0 2024-09-16T04:02:33.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.983+0000 7ff81cd75640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff818134290 con 0x7ff8180a39b0 2024-09-16T04:02:33.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.983+0000 7ff81cd75640 10 monclient(hunting): start opening mon connection 2024-09-16T04:02:33.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.983+0000 7ff81cd75640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7ff81809c140 con 0x7ff8180a3d90 2024-09-16T04:02:33.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.983+0000 7ff81cd75640 10 monclient(hunting): start opening mon connection 2024-09-16T04:02:33.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.983+0000 7ff81cd75640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7ff81809c420 con 0x7ff81813c8c0 2024-09-16T04:02:33.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.983+0000 7ff81cd75640 10 monclient(hunting): _renew_subs 2024-09-16T04:02:33.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.983+0000 7ff81cd75640 10 monclient(hunting): authenticate will time out at 2200.954441s 2024-09-16T04:02:33.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.983+0000 7ff815d9b640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff8180a3d90 0x7ff81813c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:02:33.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.983+0000 7ff815d9b640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff8180a3d90 0x7ff81813c380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:42638/0 (socket says 172.21.15.42:42638) 2024-09-16T04:02:33.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.983+0000 7ff815d9b640 1 -- 172.21.15.42:0/3773536381 learned_addr learned my addr 172.21.15.42:0/3773536381 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:02:33.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.983+0000 7ff815d9b640 10 monclient(hunting): get_auth_request con 0x7ff8180a3d90 auth_method 0 2024-09-16T04:02:33.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.983+0000 7ff815d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:02:33.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.983+0000 7ff815d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:02:33.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.983+0000 7ff815d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:02:33.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.983+0000 7ff815d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:02:33.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.983+0000 7ff815d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:02:33.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.983+0000 7ff815d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:02:33.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.983+0000 7ff816d9d640 1 --2- 172.21.15.42:0/3773536381 >> v2:172.21.15.154:3300/0 conn(0x7ff81813c8c0 0x7ff81813ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:02:33.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.983+0000 7ff815d9b640 10 monclient(hunting): handle_auth_done global_id 825531 payload 274 2024-09-16T04:02:33.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.983+0000 7ff815d9b640 1 -- 172.21.15.42:0/3773536381 >> v2:172.21.15.154:3300/0 conn(0x7ff81813c8c0 msgr2=0x7ff81813ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:02:33.992 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.983+0000 7ff815d9b640 1 --2- 172.21.15.42:0/3773536381 >> v2:172.21.15.154:3300/0 conn(0x7ff81813c8c0 0x7ff81813ecb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:33.992 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.983+0000 7ff815d9b640 1 -- 172.21.15.42:0/3773536381 >> v1:172.21.15.154:6789/0 conn(0x7ff8180a39b0 legacy=0x7ff818133b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:02:33.992 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.983+0000 7ff815d9b640 10 monclient: _finish_hunting 0 2024-09-16T04:02:33.992 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.983+0000 7ff815d9b640 1 monclient: found mon.noname-a 2024-09-16T04:02:33.992 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.983+0000 7ff815d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:02:33.992 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.983+0000 7ff815d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:02:33.992 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.983+0000 7ff815d9b640 1 -- 172.21.15.42:0/3773536381 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff81806dd60 con 0x7ff8180a3d90 2024-09-16T04:02:33.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.987+0000 7ff815d9b640 1 --2- 172.21.15.42:0/3773536381 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff8180a3d90 0x7ff81813c380 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7ff81806b540 tx=0x7ff80001c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=713712f768a1de6b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:02:33.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.987+0000 7ff81559a640 1 -- 172.21.15.42:0/3773536381 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8000b0070 con 0x7ff8180a3d90 2024-09-16T04:02:33.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.987+0000 7ff81559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:02:33.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.987+0000 7ff81559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:02:33.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.987+0000 7ff81559a640 10 monclient: dump: 2024-09-16T04:02:33.997 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:02:33.997 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:02:33.997 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:33.997 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:33.997 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:02:33.997 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:02:33.997 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:02:34.000 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:02:34.000 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:02:34.001 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:02:34.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.987+0000 7ff81559a640 10 monclient: _finish_auth 0 2024-09-16T04:02:34.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.987+0000 7ff81559a640 10 monclient: _check_auth_tickets 2024-09-16T04:02:34.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.987+0000 7ff81559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:02:34.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.987+0000 7ff81559a640 1 -- 172.21.15.42:0/3773536381 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff8000ab600 con 0x7ff8180a3d90 2024-09-16T04:02:34.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.987+0000 7ff81559a640 10 monclient: handle_config config(0 keys) 2024-09-16T04:02:34.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.987+0000 7ff81cd75640 5 monclient: authenticate success, global_id 825531 2024-09-16T04:02:34.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.987+0000 7ff81559a640 1 -- 172.21.15.42:0/3773536381 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8000ab9d0 con 0x7ff8180a3d90 2024-09-16T04:02:34.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.987+0000 7ff81cd75640 10 monclient: get_monmap_and_config success 2024-09-16T04:02:34.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.987+0000 7ff81559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:02:34.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.987+0000 7ff81cd75640 4 set_mon_vals no callback set 2024-09-16T04:02:34.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.987+0000 7ff81559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:02:34.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.987+0000 7ff81559a640 10 monclient: dump: 2024-09-16T04:02:34.002 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:02:34.002 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:02:34.002 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:34.002 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:34.003 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:02:34.003 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:02:34.003 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:02:34.004 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:02:34.004 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:02:34.004 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:02:34.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.987+0000 7ff81cd75640 10 monclient: shutdown 2024-09-16T04:02:34.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.987+0000 7ff81cd75640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:02:34.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.987+0000 7ff81cd75640 1 -- 172.21.15.42:0/3773536381 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff8180a3d90 msgr2=0x7ff81813c380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:02:34.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.987+0000 7ff81cd75640 1 --2- 172.21.15.42:0/3773536381 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff8180a3d90 0x7ff81813c380 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7ff81806b540 tx=0x7ff80001c3a0 comp rx=0 tx=0).stop 2024-09-16T04:02:34.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.987+0000 7ff81cd75640 1 -- 172.21.15.42:0/3773536381 shutdown_connections 2024-09-16T04:02:34.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.987+0000 7ff81cd75640 1 --2- 172.21.15.42:0/3773536381 >> v2:172.21.15.154:3300/0 conn(0x7ff81813c8c0 0x7ff81813ecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:34.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.987+0000 7ff81cd75640 1 --2- 172.21.15.42:0/3773536381 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff8180a3d90 0x7ff81813c380 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:34.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.987+0000 7ff81cd75640 1 -- 172.21.15.42:0/3773536381 >> 172.21.15.42:0/3773536381 conn(0x7ff8180a9fb0 msgr2=0x7ff8180aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:02:34.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.987+0000 7ff81cd75640 1 -- 172.21.15.42:0/3773536381 shutdown_connections 2024-09-16T04:02:34.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.987+0000 7ff81cd75640 1 -- 172.21.15.42:0/3773536381 wait complete. 2024-09-16T04:02:34.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.987+0000 7ff81cd75640 10 monclient: build_initial_monmap 2024-09-16T04:02:34.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.987+0000 7ff81cd75640 10 monclient: monmap: 2024-09-16T04:02:34.005 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:02:34.005 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:02:34.006 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:02:34.006 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:02:34.006 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:02:34.006 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:02:34.006 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:02:34.006 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:02:34.006 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:02:34.006 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:02:34.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.987+0000 7ff81cd75640 1 librados: starting msgr at 2024-09-16T04:02:34.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.987+0000 7ff81cd75640 1 librados: starting objecter 2024-09-16T04:02:34.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.987+0000 7ff81cd75640 1 Processor -- start 2024-09-16T04:02:34.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.991+0000 7ff81cd75640 1 -- start start 2024-09-16T04:02:34.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.991+0000 7ff81cd75640 1 librados: setting wanted keys 2024-09-16T04:02:34.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.991+0000 7ff81cd75640 1 librados: calling monclient init 2024-09-16T04:02:34.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.991+0000 7ff81cd75640 10 monclient: init 2024-09-16T04:02:34.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.991+0000 7ff81cd75640 10 monclient: _reopen_session rank -1 2024-09-16T04:02:34.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.991+0000 7ff81cd75640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:02:34.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.991+0000 7ff81cd75640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff8180a39b0 0x7ff81812d620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:34.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.991+0000 7ff81cd75640 10 monclient(hunting): picked mon.noname-a con 0x7ff8180a39b0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:02:34.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.991+0000 7ff81cd75640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff8180a3d90 0x7ff81812db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:34.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.991+0000 7ff81cd75640 10 monclient(hunting): picked mon.noname-b con 0x7ff8180a3d90 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:02:34.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.991+0000 7ff81cd75640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff81813c8c0 0x7ff818138130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:34.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.991+0000 7ff81cd75640 10 monclient(hunting): picked mon.noname-c con 0x7ff81813c8c0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:02:34.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.991+0000 7ff81cd75640 10 monclient(hunting): start opening mon connection 2024-09-16T04:02:34.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.991+0000 7ff81cd75640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7ff818141da0 con 0x7ff8180a39b0 2024-09-16T04:02:34.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.991+0000 7ff81cd75640 10 monclient(hunting): start opening mon connection 2024-09-16T04:02:34.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.991+0000 7ff81cd75640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff818141c20 con 0x7ff8180a3d90 2024-09-16T04:02:34.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.991+0000 7ff81cd75640 10 monclient(hunting): start opening mon connection 2024-09-16T04:02:34.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.991+0000 7ff81cd75640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7ff818141f20 con 0x7ff81813c8c0 2024-09-16T04:02:34.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.991+0000 7ff81cd75640 10 monclient(hunting): _renew_subs 2024-09-16T04:02:34.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.991+0000 7ff81cd75640 10 monclient(hunting): authenticate will time out at 2200.960817s 2024-09-16T04:02:34.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.991+0000 7ff816d9d640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff81813c8c0 0x7ff818138130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:02:34.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.991+0000 7ff816d9d640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff81813c8c0 0x7ff818138130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:36352/0 (socket says 172.21.15.42:36352) 2024-09-16T04:02:34.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.991+0000 7ff816d9d640 1 -- 172.21.15.42:0/3266201353 learned_addr learned my addr 172.21.15.42:0/3266201353 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:02:34.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.991+0000 7ff816d9d640 10 monclient(hunting): get_auth_request con 0x7ff81813c8c0 auth_method 0 2024-09-16T04:02:34.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.991+0000 7ff816d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:02:34.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.991+0000 7ff816d9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:02:34.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.991+0000 7ff816d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:02:34.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.991+0000 7ff81659c640 1 --2- 172.21.15.42:0/3266201353 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff8180a39b0 0x7ff81812d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:02:34.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.991+0000 7ff81659c640 10 monclient(hunting): get_auth_request con 0x7ff8180a39b0 auth_method 0 2024-09-16T04:02:34.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.991+0000 7ff81659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:02:34.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.991+0000 7ff81659c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:02:34.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.991+0000 7ff81659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:02:34.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.991+0000 7ff815d9b640 1 --2- 172.21.15.42:0/3266201353 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff8180a3d90 0x7ff81812db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:02:34.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.991+0000 7ff816d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:02:34.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.991+0000 7ff816d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:02:34.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.991+0000 7ff816d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:02:34.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.991+0000 7ff81659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:02:34.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.991+0000 7ff81659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:02:34.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.991+0000 7ff81659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:02:34.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.991+0000 7ff815d9b640 10 monclient(hunting): get_auth_request con 0x7ff8180a3d90 auth_method 0 2024-09-16T04:02:34.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.991+0000 7ff815d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:02:34.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.991+0000 7ff815d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:02:34.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.991+0000 7ff815d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:02:34.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.991+0000 7ff816d9d640 10 monclient(hunting): handle_auth_done global_id 964124 payload 824 2024-09-16T04:02:34.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.991+0000 7ff816d9d640 1 -- 172.21.15.42:0/3266201353 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff8180a3d90 msgr2=0x7ff81812db60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:02:34.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.991+0000 7ff816d9d640 1 --2- 172.21.15.42:0/3266201353 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff8180a3d90 0x7ff81812db60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:34.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.991+0000 7ff816d9d640 1 -- 172.21.15.42:0/3266201353 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff8180a39b0 msgr2=0x7ff81812d620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:02:34.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.991+0000 7ff816d9d640 1 --2- 172.21.15.42:0/3266201353 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff8180a39b0 0x7ff81812d620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:34.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.991+0000 7ff816d9d640 10 monclient: _finish_hunting 0 2024-09-16T04:02:34.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.991+0000 7ff816d9d640 1 monclient: found mon.noname-c 2024-09-16T04:02:34.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.991+0000 7ff816d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:02:34.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.991+0000 7ff816d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T04:02:34.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.991+0000 7ff816d9d640 1 -- 172.21.15.42:0/3266201353 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff818139260 con 0x7ff81813c8c0 2024-09-16T04:02:34.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.991+0000 7ff81659c640 1 --2- 172.21.15.42:0/3266201353 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff8180a39b0 0x7ff81812d620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:02:34.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.991+0000 7ff815d9b640 1 --2- 172.21.15.42:0/3266201353 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff8180a3d90 0x7ff81812db60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T04:02:34.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.991+0000 7ff816d9d640 1 -- 172.21.15.42:0/3266201353 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff81813c8c0 msgr2=0x7ff818138130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-16T04:02:34.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.991+0000 7ff816d9d640 1 -- 172.21.15.42:0/3266201353 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff81813c8c0 msgr2=0x7ff818138130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:02:34.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.991+0000 7ff816d9d640 1 --2- 172.21.15.42:0/3266201353 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff81813c8c0 0x7ff818138130 secure :-1 s=COMPRESSION_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0x7ff81000cda0 tx=0x7ff81000a660 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:02:34.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:33.991+0000 7ff816d9d640 1 --2- 172.21.15.42:0/3266201353 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff81813c8c0 0x7ff818138130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:02:34.200 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:34.195+0000 7ff816d9d640 1 --2- 172.21.15.42:0/3266201353 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff81813c8c0 0x7ff818138130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:02:34.200 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:34.195+0000 7ff816d9d640 10 monclient: get_auth_request con 0x7ff81813c8c0 auth_method 0 2024-09-16T04:02:34.200 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:34.195+0000 7ff816d9d640 0 --2- 172.21.15.42:0/3266201353 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff81813c8c0 0x7ff818138130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-09-16T04:02:34.200 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:34.195+0000 7ff816d9d640 1 --2- 172.21.15.42:0/3266201353 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff81813c8c0 0x7ff818138130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:34.200 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:34.195+0000 7ff80effd640 10 monclient: ms_handle_reset current mon [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:02:34.200 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:34.195+0000 7ff80effd640 10 monclient: _reopen_session rank -1 2024-09-16T04:02:34.201 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:34.195+0000 7ff80effd640 1 -- 172.21.15.42:0/3266201353 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff81813c8c0 msgr2=0x7ff818138130 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:02:34.201 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:34.195+0000 7ff80effd640 1 --2- 172.21.15.42:0/3266201353 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff81813c8c0 0x7ff818138130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:34.201 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:34.195+0000 7ff80effd640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:02:34.201 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:34.195+0000 7ff80effd640 1 --2- 172.21.15.42:0/3266201353 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff7f80034b0 0x7ff7f8005900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:34.201 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:34.195+0000 7ff80effd640 10 monclient(hunting): picked mon.noname-c con 0x7ff7f80034b0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:02:34.201 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:34.195+0000 7ff80effd640 1 --2- 172.21.15.42:0/3266201353 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff8180a3d90 0x7ff7f8008480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:34.201 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:34.195+0000 7ff80effd640 10 monclient(hunting): picked mon.noname-b con 0x7ff8180a3d90 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:02:34.201 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:34.195+0000 7ff80effd640 1 --2- 172.21.15.42:0/3266201353 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff8180a39b0 0x7ff7f800af10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:34.201 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:34.195+0000 7ff80effd640 10 monclient(hunting): picked mon.noname-a con 0x7ff8180a39b0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:02:34.201 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:34.195+0000 7ff80effd640 10 monclient(hunting): start opening mon connection 2024-09-16T04:02:34.201 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:34.195+0000 7ff80effd640 1 -- 172.21.15.42:0/3266201353 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7ff7f800b920 con 0x7ff8180a39b0 2024-09-16T04:02:34.201 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:34.195+0000 7ff80effd640 10 monclient(hunting): start opening mon connection 2024-09-16T04:02:34.201 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:34.195+0000 7ff80effd640 1 -- 172.21.15.42:0/3266201353 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff7f800baa0 con 0x7ff8180a3d90 2024-09-16T04:02:34.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:34.195+0000 7ff80effd640 10 monclient(hunting): start opening mon connection 2024-09-16T04:02:34.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:34.195+0000 7ff80effd640 1 -- 172.21.15.42:0/3266201353 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7ff7f800bca0 con 0x7ff7f80034b0 2024-09-16T04:02:34.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:34.195+0000 7ff80effd640 10 monclient(hunting): _renew_subs 2024-09-16T04:02:34.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:34.195+0000 7ff81659c640 1 --2- 172.21.15.42:0/3266201353 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff7f80034b0 0x7ff7f8005900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:02:34.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:34.195+0000 7ff81659c640 10 monclient(hunting): get_auth_request con 0x7ff7f80034b0 auth_method 0 2024-09-16T04:02:34.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:34.195+0000 7ff81659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:02:34.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:34.195+0000 7ff81659c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:02:34.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:34.195+0000 7ff81659c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:02:34.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:34.195+0000 7ff816d9d640 1 --2- 172.21.15.42:0/3266201353 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff8180a39b0 0x7ff7f800af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:02:34.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:34.195+0000 7ff81659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:02:34.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:34.195+0000 7ff81659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:02:34.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:34.195+0000 7ff81659c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:02:34.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:34.195+0000 7ff816d9d640 10 monclient(hunting): get_auth_request con 0x7ff8180a39b0 auth_method 0 2024-09-16T04:02:34.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:34.195+0000 7ff816d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:02:34.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:34.195+0000 7ff816d9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:02:34.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:34.195+0000 7ff816d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:02:34.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:34.195+0000 7ff815d9b640 1 --2- 172.21.15.42:0/3266201353 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff8180a3d90 0x7ff7f8008480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:02:34.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:34.195+0000 7ff815d9b640 10 monclient(hunting): get_auth_request con 0x7ff8180a3d90 auth_method 0 2024-09-16T04:02:34.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:34.195+0000 7ff815d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:02:34.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:34.195+0000 7ff815d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:02:34.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:34.195+0000 7ff815d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:02:34.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:34.195+0000 7ff816d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:02:34.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:34.195+0000 7ff816d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:02:34.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:34.195+0000 7ff816d9d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:02:34.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:34.195+0000 7ff81659c640 10 monclient(hunting): handle_auth_done global_id 964124 payload 293 2024-09-16T04:02:34.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:34.195+0000 7ff81659c640 1 -- 172.21.15.42:0/3266201353 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff8180a3d90 msgr2=0x7ff7f8008480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:02:34.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:34.195+0000 7ff81659c640 1 --2- 172.21.15.42:0/3266201353 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff8180a3d90 0x7ff7f8008480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:34.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:34.195+0000 7ff81659c640 1 -- 172.21.15.42:0/3266201353 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff8180a39b0 msgr2=0x7ff7f800af10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:02:34.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:34.195+0000 7ff81659c640 1 --2- 172.21.15.42:0/3266201353 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff8180a39b0 0x7ff7f800af10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:34.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:34.195+0000 7ff81659c640 10 monclient: _finish_hunting 0 2024-09-16T04:02:34.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:34.195+0000 7ff81659c640 1 monclient: found mon.noname-c 2024-09-16T04:02:34.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:34.195+0000 7ff81659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:02:34.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:34.195+0000 7ff81659c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T04:02:34.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:34.195+0000 7ff81659c640 1 -- 172.21.15.42:0/3266201353 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7f800bf60 con 0x7ff7f80034b0 2024-09-16T04:02:34.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:34.195+0000 7ff815d9b640 1 --2- 172.21.15.42:0/3266201353 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff8180a3d90 0x7ff7f8008480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T04:02:34.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:34.195+0000 7ff81659c640 1 --2- 172.21.15.42:0/3266201353 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff7f80034b0 0x7ff7f8005900 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7ff8080043d0 tx=0x7ff8100077b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:02:34.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:34.195+0000 7ff80effd640 10 client.?.objecter ms_handle_connect 0x7ff7f80034b0 2024-09-16T04:02:34.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:34.195+0000 7ff80effd640 10 client.?.objecter resend_mon_ops 2024-09-16T04:02:34.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:34.195+0000 7ff80effd640 1 -- 172.21.15.42:0/3266201353 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff808018070 con 0x7ff7f80034b0 2024-09-16T04:02:34.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:34.195+0000 7ff80effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:02:34.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:34.195+0000 7ff80effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:02:34.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:34.195+0000 7ff80effd640 10 monclient: dump: 2024-09-16T04:02:34.205 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:02:34.205 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:02:34.205 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:34.205 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:34.208 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:02:34.208 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:02:34.208 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:02:34.208 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:02:34.208 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:02:34.208 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:02:34.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:34.195+0000 7ff80effd640 10 monclient: _finish_auth 0 2024-09-16T04:02:34.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:34.195+0000 7ff80effd640 10 monclient: _check_auth_tickets 2024-09-16T04:02:34.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:34.195+0000 7ff80effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:02:34.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:34.195+0000 7ff81cd75640 5 monclient: authenticate success, global_id 964124 2024-09-16T04:02:34.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:34.195+0000 7ff81cd75640 10 monclient: _renew_subs 2024-09-16T04:02:34.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:34.195+0000 7ff80effd640 1 -- 172.21.15.42:0/3266201353 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff80800fc40 con 0x7ff7f80034b0 2024-09-16T04:02:34.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:34.195+0000 7ff81cd75640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:02:34.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:34.195+0000 7ff81cd75640 1 -- 172.21.15.42:0/3266201353 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff81813c200 con 0x7ff7f80034b0 2024-09-16T04:02:34.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:34.195+0000 7ff80effd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:02:34.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:34.195+0000 7ff80effd640 1 -- 172.21.15.42:0/3266201353 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff808003fd0 con 0x7ff7f80034b0 2024-09-16T04:02:34.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:34.195+0000 7ff80effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:02:34.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:34.195+0000 7ff80effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:02:34.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:34.195+0000 7ff81cd75640 10 client.964124.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:02:34.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:34.195+0000 7ff80effd640 10 monclient: dump: 2024-09-16T04:02:34.209 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:02:34.209 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:02:34.209 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:34.209 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:34.210 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:02:34.210 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:02:34.210 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:02:34.210 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:02:34.210 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:02:34.210 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:02:34.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:34.195+0000 7ff814d99640 4 set_mon_vals no callback set 2024-09-16T04:02:34.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:34.195+0000 7ff81cd75640 10 monclient: _renew_subs 2024-09-16T04:02:34.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:34.195+0000 7ff81cd75640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:02:34.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:34.195+0000 7ff81cd75640 1 -- 172.21.15.42:0/3266201353 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff81813aaf0 con 0x7ff7f80034b0 2024-09-16T04:02:34.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:34.195+0000 7ff81cd75640 1 librados: init done 2024-09-16T04:02:34.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:34.195+0000 7ff81cd75640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:02:34.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:34.199+0000 7ff81cd75640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:02:34.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:34.199+0000 7ff81cd75640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:02:34.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:34.199+0000 7ff81cd75640 1 -- 172.21.15.42:0/3266201353 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff818133b80 con 0x7ff7f80034b0 2024-09-16T04:02:34.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:34.199+0000 7ff80effd640 1 -- 172.21.15.42:0/3266201353 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7ff808005020 con 0x7ff7f80034b0 2024-09-16T04:02:34.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:34.199+0000 7ff80effd640 1 --2- 172.21.15.42:0/3266201353 >> v2:172.21.15.154:6812/254845342 conn(0x7ff81813c8c0 0x7ff7f8042110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:34.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:34.199+0000 7ff80effd640 1 -- 172.21.15.42:0/3266201353 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(630..630 src has 451..630) ==== 6280+0+0 (secure 0 0 0) 0x7ff808048410 con 0x7ff7f80034b0 2024-09-16T04:02:34.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:34.199+0000 7ff80effd640 10 client.964124.objecter ms_dispatch 0x7ff818000ba0 osd_map(630..630 src has 451..630) 2024-09-16T04:02:34.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:34.199+0000 7ff80effd640 3 client.964124.objecter handle_osd_map got epochs [630,630] > 0 2024-09-16T04:02:34.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:34.199+0000 7ff80effd640 3 client.964124.objecter handle_osd_map decoding full epoch 630 2024-09-16T04:02:34.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:34.199+0000 7ff80effd640 20 client.964124.objecter dump_active .. 0 homeless 2024-09-16T04:02:34.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:34.199+0000 7ff815d9b640 1 --2- 172.21.15.42:0/3266201353 >> v2:172.21.15.154:6812/254845342 conn(0x7ff81813c8c0 0x7ff7f8042110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:02:34.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:34.199+0000 7ff815d9b640 10 monclient: get_auth_request con 0x7ff81813c8c0 auth_method 0 2024-09-16T04:02:34.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:34.199+0000 7ff80effd640 10 client.964124.objecter ms_handle_connect 0x7ff81813c8c0 2024-09-16T04:02:34.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:34.199+0000 7ff815d9b640 1 --2- 172.21.15.42:0/3266201353 >> v2:172.21.15.154:6812/254845342 conn(0x7ff81813c8c0 0x7ff7f8042110 secure :-1 s=READY pgs=3786 cs=0 l=1 rev1=1 crypto rx=0x7ff8000ad000 tx=0x7ff8000a6000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:02:34.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:34.203+0000 7ff80effd640 1 -- 172.21.15.42:0/3266201353 <== mon.2 v2:172.21.15.42:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7ff808005310 con 0x7ff7f80034b0 2024-09-16T04:02:34.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:34.203+0000 7ff80effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:02:34.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:34.203+0000 7ff80effd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:02:34.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:34.431+0000 7ff81cd75640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T04:02:34.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:34.431+0000 7ff81cd75640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T04:02:34.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:34.431+0000 7ff81cd75640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:02:34.437 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:34.431+0000 7ff81cd75640 1 -- 172.21.15.42:0/3266201353 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7ff8180a5750 con 0x7ff7f80034b0 2024-09-16T04:02:34.440 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:34.435+0000 7ff80effd640 1 -- 172.21.15.42:0/3266201353 <== mon.2 v2:172.21.15.42:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7ff808007550 con 0x7ff7f80034b0 2024-09-16T04:02:34.440 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:34.435+0000 7ff80effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T04:02:34.440 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:34.435+0000 7ff80effd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:02:34.441 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.435+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.441 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.435+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=28,config=2,health=11,kv=10,logm=49} crc {auth=503507790,config=692619927,health=1210547175,kv=3446188498,logm=358916099}) 2024-09-16T04:02:34.441 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.435+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=25,config=2,health=11,kv=9,logm=53} crc {auth=3576762235,config=692619927,health=47271155,kv=2323084916,logm=3412696305}) 2024-09-16T04:02:34.441 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.435+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.441 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.435+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=28,config=2,health=11,kv=10,logm=49} crc {auth=503507790,config=692619927,health=1210547175,kv=3446188498,logm=358916099}) 2024-09-16T04:02:34.441 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.435+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=24,config=2,health=12,kv=9,logm=53} crc {auth=1359465095,config=692619927,health=2940166115,kv=2818795783,logm=3698725183}) 2024-09-16T04:02:34.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:34.439+0000 7ff81cd75640 10 librados: watch_flush enter 2024-09-16T04:02:34.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:34.439+0000 7ff81cd75640 10 librados: watch_flush exit 2024-09-16T04:02:34.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:34.439+0000 7ff81cd75640 20 client.964124.objecter shutdown clearing up homeless session... 2024-09-16T04:02:34.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:34.439+0000 7ff81cd75640 10 client.964124.objecter successfully canceled tick 2024-09-16T04:02:34.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:34.439+0000 7ff81cd75640 1 -- 172.21.15.42:0/3266201353 >> v2:172.21.15.154:6812/254845342 conn(0x7ff81813c8c0 msgr2=0x7ff7f8042110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:02:34.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:34.439+0000 7ff81cd75640 1 --2- 172.21.15.42:0/3266201353 >> v2:172.21.15.154:6812/254845342 conn(0x7ff81813c8c0 0x7ff7f8042110 secure :-1 s=READY pgs=3786 cs=0 l=1 rev1=1 crypto rx=0x7ff8000ad000 tx=0x7ff8000a6000 comp rx=0 tx=0).stop 2024-09-16T04:02:34.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:34.439+0000 7ff81cd75640 10 monclient: shutdown 2024-09-16T04:02:34.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:34.439+0000 7ff81cd75640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:02:34.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:34.439+0000 7ff81cd75640 1 -- 172.21.15.42:0/3266201353 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff7f80034b0 msgr2=0x7ff7f8005900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:02:34.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:34.439+0000 7ff81cd75640 1 --2- 172.21.15.42:0/3266201353 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff7f80034b0 0x7ff7f8005900 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7ff8080043d0 tx=0x7ff8100077b0 comp rx=0 tx=0).stop 2024-09-16T04:02:34.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:34.439+0000 7ff81cd75640 1 -- 172.21.15.42:0/3266201353 shutdown_connections 2024-09-16T04:02:34.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:34.439+0000 7ff81cd75640 1 --2- 172.21.15.42:0/3266201353 >> v2:172.21.15.154:6812/254845342 conn(0x7ff81813c8c0 0x7ff7f8042110 unknown :-1 s=CLOSED pgs=3786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:34.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:34.439+0000 7ff81cd75640 1 --2- 172.21.15.42:0/3266201353 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff8180a39b0 0x7ff7f800af10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:34.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:34.439+0000 7ff81cd75640 1 --2- 172.21.15.42:0/3266201353 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff8180a3d90 0x7ff7f8008480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:34.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:34.439+0000 7ff81cd75640 1 --2- 172.21.15.42:0/3266201353 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff7f80034b0 0x7ff7f8005900 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:34.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:34.439+0000 7ff81cd75640 1 -- 172.21.15.42:0/3266201353 >> 172.21.15.42:0/3266201353 conn(0x7ff8180a9fb0 msgr2=0x7ff818131980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:02:34.445 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:34.439+0000 7ff81cd75640 1 -- 172.21.15.42:0/3266201353 shutdown_connections 2024-09-16T04:02:34.445 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:34.439+0000 7ff81cd75640 1 -- 172.21.15.42:0/3266201353 wait complete. 2024-09-16T04:02:34.446 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:34.439+0000 7ff81cd75640 1 librados: shutdown 2024-09-16T04:02:34.447 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.439+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.447 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.439+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3802875106}) 2024-09-16T04:02:34.447 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.439+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1088047628}) 2024-09-16T04:02:34.447 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.439+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.447 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.439+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3802875106}) 2024-09-16T04:02:34.447 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.439+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1401701447}) 2024-09-16T04:02:34.447 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.443+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.447 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.443+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4131005154}) 2024-09-16T04:02:34.448 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.443+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=349490618}) 2024-09-16T04:02:34.448 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.443+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.448 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.443+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4131005154}) 2024-09-16T04:02:34.448 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.443+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1249824534}) 2024-09-16T04:02:34.450 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.447+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.450 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.447+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2068007388}) 2024-09-16T04:02:34.450 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.447+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=43681962}) 2024-09-16T04:02:34.450 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.447+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.450 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.447+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2068007388}) 2024-09-16T04:02:34.450 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.447+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=23247404}) 2024-09-16T04:02:34.453 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.451+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.454 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.451+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1515901066}) 2024-09-16T04:02:34.454 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.451+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1629958193}) 2024-09-16T04:02:34.454 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.451+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.454 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.451+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1515901066}) 2024-09-16T04:02:34.454 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.451+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3273949262}) 2024-09-16T04:02:34.456 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.451+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.456 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.451+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=348229046}) 2024-09-16T04:02:34.456 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.451+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4248257276}) 2024-09-16T04:02:34.456 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.451+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.456 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.451+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=348229046}) 2024-09-16T04:02:34.456 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.451+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2864038354}) 2024-09-16T04:02:34.458 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.455+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.458 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.455+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4119522658}) 2024-09-16T04:02:34.458 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.455+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4159115701}) 2024-09-16T04:02:34.458 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.455+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.458 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.455+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4119522658}) 2024-09-16T04:02:34.458 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.455+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4216853219}) 2024-09-16T04:02:34.460 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.455+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.460 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.455+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3413460651}) 2024-09-16T04:02:34.460 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.455+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=104309741}) 2024-09-16T04:02:34.460 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.455+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.460 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.455+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3413460651}) 2024-09-16T04:02:34.461 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.455+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2805486302}) 2024-09-16T04:02:34.462 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.459+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.462 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.459+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3439438912}) 2024-09-16T04:02:34.462 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.459+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2455896824}) 2024-09-16T04:02:34.462 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.459+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.463 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.459+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3439438912}) 2024-09-16T04:02:34.463 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.459+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=980622599}) 2024-09-16T04:02:34.464 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T04:02:34.465 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.459+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.465 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.459+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2541574413}) 2024-09-16T04:02:34.465 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.459+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=294329675}) 2024-09-16T04:02:34.465 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.459+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.465 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.459+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2541574413}) 2024-09-16T04:02:34.465 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.459+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=324441045}) 2024-09-16T04:02:34.467 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.463+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.467 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.463+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3152074499}) 2024-09-16T04:02:34.467 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.463+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2532137114}) 2024-09-16T04:02:34.467 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.463+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.467 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.463+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3152074499}) 2024-09-16T04:02:34.468 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.463+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1274307736}) 2024-09-16T04:02:34.469 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.463+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.469 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.463+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=128015035}) 2024-09-16T04:02:34.469 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.463+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2321108795}) 2024-09-16T04:02:34.469 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.463+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.469 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.463+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=128015035}) 2024-09-16T04:02:34.469 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.463+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=88424469}) 2024-09-16T04:02:34.471 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.467+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.471 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.467+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2893021680}) 2024-09-16T04:02:34.471 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.467+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2194732504}) 2024-09-16T04:02:34.471 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.467+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.471 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.467+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2893021680}) 2024-09-16T04:02:34.472 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.467+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=709964939}) 2024-09-16T04:02:34.473 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.471+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.474 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.471+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4264209395}) 2024-09-16T04:02:34.474 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.471+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=691125489}) 2024-09-16T04:02:34.474 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.471+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.474 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.471+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4264209395}) 2024-09-16T04:02:34.474 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.471+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1664204715}) 2024-09-16T04:02:34.476 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.471+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.476 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.471+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3742815862}) 2024-09-16T04:02:34.476 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.471+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2939188990}) 2024-09-16T04:02:34.476 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.471+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.476 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.471+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3742815862}) 2024-09-16T04:02:34.476 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.471+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1174568928}) 2024-09-16T04:02:34.478 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.475+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.478 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.475+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1199660067}) 2024-09-16T04:02:34.478 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.475+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3794579299}) 2024-09-16T04:02:34.478 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.475+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.478 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.475+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1199660067}) 2024-09-16T04:02:34.478 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.475+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3620576714}) 2024-09-16T04:02:34.480 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.475+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.480 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.475+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2929002109}) 2024-09-16T04:02:34.480 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.475+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3830009623}) 2024-09-16T04:02:34.480 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.475+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.481 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.475+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2929002109}) 2024-09-16T04:02:34.481 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.475+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=178809467}) 2024-09-16T04:02:34.482 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.479+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.483 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.479+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1772457287}) 2024-09-16T04:02:34.483 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.479+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1953249652}) 2024-09-16T04:02:34.483 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.479+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.483 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.479+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1772457287}) 2024-09-16T04:02:34.483 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.479+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4285665817}) 2024-09-16T04:02:34.485 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.479+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.485 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.479+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2226281007}) 2024-09-16T04:02:34.485 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.479+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=417759482}) 2024-09-16T04:02:34.485 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.479+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.485 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.479+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2226281007}) 2024-09-16T04:02:34.485 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.479+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1718864718}) 2024-09-16T04:02:34.487 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.483+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.487 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.483+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3436448081}) 2024-09-16T04:02:34.487 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.483+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1150832899}) 2024-09-16T04:02:34.487 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.483+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.487 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.483+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3436448081}) 2024-09-16T04:02:34.487 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.483+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=358972924}) 2024-09-16T04:02:34.489 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.483+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.489 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.483+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2145287523}) 2024-09-16T04:02:34.489 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.483+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1507445594}) 2024-09-16T04:02:34.489 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.483+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.489 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.483+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2145287523}) 2024-09-16T04:02:34.489 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.483+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=358714338}) 2024-09-16T04:02:34.491 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.487+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.491 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.487+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3095025907}) 2024-09-16T04:02:34.491 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.487+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1775760408}) 2024-09-16T04:02:34.491 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.487+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.491 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.487+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3095025907}) 2024-09-16T04:02:34.492 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.487+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3871248441}) 2024-09-16T04:02:34.493 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.487+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.493 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.487+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=257648899}) 2024-09-16T04:02:34.493 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.487+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1493599086}) 2024-09-16T04:02:34.494 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.487+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.494 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.487+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=257648899}) 2024-09-16T04:02:34.494 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.487+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=398072956}) 2024-09-16T04:02:34.495 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.491+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.496 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.491+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=828728129}) 2024-09-16T04:02:34.496 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.491+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4199925086}) 2024-09-16T04:02:34.496 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.491+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.496 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.491+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=828728129}) 2024-09-16T04:02:34.496 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.491+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2025803019}) 2024-09-16T04:02:34.497 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.491+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.497 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.491+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3178669234}) 2024-09-16T04:02:34.497 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.491+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=491724700}) 2024-09-16T04:02:34.497 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.491+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.498 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.491+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3178669234}) 2024-09-16T04:02:34.498 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.491+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3645487015}) 2024-09-16T04:02:34.499 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.495+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.499 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.495+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4036960236}) 2024-09-16T04:02:34.499 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.495+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2478991749}) 2024-09-16T04:02:34.500 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.495+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.500 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.495+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4036960236}) 2024-09-16T04:02:34.500 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.495+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3883407133}) 2024-09-16T04:02:34.501 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.495+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.501 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.495+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1156974185}) 2024-09-16T04:02:34.501 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.495+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4249783009}) 2024-09-16T04:02:34.501 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.495+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.502 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.495+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1156974185}) 2024-09-16T04:02:34.502 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.495+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2122069556}) 2024-09-16T04:02:34.503 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.499+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.503 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.499+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1415508865}) 2024-09-16T04:02:34.503 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.499+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3433116570}) 2024-09-16T04:02:34.503 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.499+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.504 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.499+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1415508865}) 2024-09-16T04:02:34.504 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.499+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2740285785}) 2024-09-16T04:02:34.505 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.499+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.505 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.499+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4071972942}) 2024-09-16T04:02:34.505 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.499+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1333459118}) 2024-09-16T04:02:34.506 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.499+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.506 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.499+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4071972942}) 2024-09-16T04:02:34.506 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.499+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1630512182}) 2024-09-16T04:02:34.507 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.503+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.507 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.503+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4065253387}) 2024-09-16T04:02:34.507 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.503+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3216622193}) 2024-09-16T04:02:34.507 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.503+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.508 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.503+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4065253387}) 2024-09-16T04:02:34.508 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.503+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3100543513}) 2024-09-16T04:02:34.509 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.503+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.509 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.503+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2654861979}) 2024-09-16T04:02:34.509 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.503+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2287898485}) 2024-09-16T04:02:34.509 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.503+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.509 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.503+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2654861979}) 2024-09-16T04:02:34.510 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.503+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2923259225}) 2024-09-16T04:02:34.511 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.507+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.511 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.507+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2832094088}) 2024-09-16T04:02:34.511 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.507+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2557691206}) 2024-09-16T04:02:34.511 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.507+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.512 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.507+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2832094088}) 2024-09-16T04:02:34.512 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.507+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3953665947}) 2024-09-16T04:02:34.513 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.507+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.513 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.507+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2742035058}) 2024-09-16T04:02:34.513 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.507+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2433401850}) 2024-09-16T04:02:34.514 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.507+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.514 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.511+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2742035058}) 2024-09-16T04:02:34.514 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.511+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1479553689}) 2024-09-16T04:02:34.515 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.511+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.516 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.511+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2991578585}) 2024-09-16T04:02:34.516 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.511+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3603041435}) 2024-09-16T04:02:34.516 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.511+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.516 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.511+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2991578585}) 2024-09-16T04:02:34.516 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.511+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1472501595}) 2024-09-16T04:02:34.517 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.515+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.518 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.515+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1480772280}) 2024-09-16T04:02:34.518 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.515+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2676278650}) 2024-09-16T04:02:34.518 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.515+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.518 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.515+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1480772280}) 2024-09-16T04:02:34.518 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.515+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=568207652}) 2024-09-16T04:02:34.520 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.515+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.520 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.515+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1205609941}) 2024-09-16T04:02:34.520 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.515+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1327502781}) 2024-09-16T04:02:34.520 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.515+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.520 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.515+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1205609941}) 2024-09-16T04:02:34.520 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.515+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3754568039}) 2024-09-16T04:02:34.521 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.519+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.522 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.519+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3129432389}) 2024-09-16T04:02:34.522 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.519+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3299933807}) 2024-09-16T04:02:34.522 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.519+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.522 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.519+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3129432389}) 2024-09-16T04:02:34.522 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.519+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=575515297}) 2024-09-16T04:02:34.524 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.519+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.524 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.519+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3457435019}) 2024-09-16T04:02:34.524 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.519+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4243684770}) 2024-09-16T04:02:34.524 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.519+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.524 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.519+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3457435019}) 2024-09-16T04:02:34.524 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.519+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2087166231}) 2024-09-16T04:02:34.526 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.523+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.526 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.523+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1976946852}) 2024-09-16T04:02:34.526 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.523+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4097045166}) 2024-09-16T04:02:34.526 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.523+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.526 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.523+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1976946852}) 2024-09-16T04:02:34.526 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.523+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3866671302}) 2024-09-16T04:02:34.528 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.523+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.528 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.523+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=146985743}) 2024-09-16T04:02:34.528 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.523+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4163721418}) 2024-09-16T04:02:34.528 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.523+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.528 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.523+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=146985743}) 2024-09-16T04:02:34.528 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.523+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=978482488}) 2024-09-16T04:02:34.530 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.527+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.530 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.527+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1045099551}) 2024-09-16T04:02:34.530 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.527+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3649295717}) 2024-09-16T04:02:34.530 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.527+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.530 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.527+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1045099551}) 2024-09-16T04:02:34.530 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.527+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=709757087}) 2024-09-16T04:02:34.532 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.527+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.532 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.527+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1485297248}) 2024-09-16T04:02:34.532 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.527+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2476829522}) 2024-09-16T04:02:34.532 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.527+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.532 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.527+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1485297248}) 2024-09-16T04:02:34.532 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.527+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2672023550}) 2024-09-16T04:02:34.534 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.531+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.534 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.531+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1865026170}) 2024-09-16T04:02:34.534 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.531+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1481757354}) 2024-09-16T04:02:34.534 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.531+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.534 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.531+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1865026170}) 2024-09-16T04:02:34.534 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.531+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=579135965}) 2024-09-16T04:02:34.536 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.531+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.536 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.531+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3270504216}) 2024-09-16T04:02:34.536 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.531+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1818421270}) 2024-09-16T04:02:34.536 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.531+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.536 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.531+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3270504216}) 2024-09-16T04:02:34.536 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.531+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=309272950}) 2024-09-16T04:02:34.538 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.535+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.538 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.535+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2541848087}) 2024-09-16T04:02:34.538 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.535+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=455077182}) 2024-09-16T04:02:34.538 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.535+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.538 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.535+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2541848087}) 2024-09-16T04:02:34.538 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.535+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=200624086}) 2024-09-16T04:02:34.540 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.535+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.540 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.535+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=668161211}) 2024-09-16T04:02:34.540 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.535+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2390733898}) 2024-09-16T04:02:34.540 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.535+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.540 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.535+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=668161211}) 2024-09-16T04:02:34.540 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.535+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1474618699}) 2024-09-16T04:02:34.542 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.539+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.542 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.539+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=655313029}) 2024-09-16T04:02:34.542 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.539+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4100525961}) 2024-09-16T04:02:34.542 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.539+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.542 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.539+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=655313029}) 2024-09-16T04:02:34.542 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.539+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=772836395}) 2024-09-16T04:02:34.544 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.539+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.544 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.539+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3434664422}) 2024-09-16T04:02:34.544 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.539+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2820551771}) 2024-09-16T04:02:34.544 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.539+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.545 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.539+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3434664422}) 2024-09-16T04:02:34.545 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.539+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1722399367}) 2024-09-16T04:02:34.546 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.543+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.546 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.543+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=196469908}) 2024-09-16T04:02:34.546 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.543+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2722752475}) 2024-09-16T04:02:34.546 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.543+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.546 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.543+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=196469908}) 2024-09-16T04:02:34.547 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.543+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1744543367}) 2024-09-16T04:02:34.548 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.543+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.548 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.543+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3761367994}) 2024-09-16T04:02:34.548 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.543+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1214102319}) 2024-09-16T04:02:34.548 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.543+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.549 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.543+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3761367994}) 2024-09-16T04:02:34.549 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.543+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2057807351}) 2024-09-16T04:02:34.550 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.547+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.551 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.547+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2997612179}) 2024-09-16T04:02:34.551 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.547+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4271024574}) 2024-09-16T04:02:34.551 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.547+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.551 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.547+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2997612179}) 2024-09-16T04:02:34.551 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.547+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1511855422}) 2024-09-16T04:02:34.552 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.547+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.552 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.547+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3410844468}) 2024-09-16T04:02:34.553 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.547+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1593898184}) 2024-09-16T04:02:34.553 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.547+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.553 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.547+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3410844468}) 2024-09-16T04:02:34.553 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.547+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1044882618}) 2024-09-16T04:02:34.554 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.551+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.555 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.551+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3345726215}) 2024-09-16T04:02:34.555 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.551+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=651444218}) 2024-09-16T04:02:34.555 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.551+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.555 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.551+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3345726215}) 2024-09-16T04:02:34.555 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.551+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=825725640}) 2024-09-16T04:02:34.556 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.551+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.557 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.551+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=645947994}) 2024-09-16T04:02:34.557 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.551+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2128407054}) 2024-09-16T04:02:34.557 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.551+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.557 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.551+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=645947994}) 2024-09-16T04:02:34.557 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.551+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=687409970}) 2024-09-16T04:02:34.558 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.555+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.558 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.555+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2770451729}) 2024-09-16T04:02:34.558 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.555+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1836678667}) 2024-09-16T04:02:34.558 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.555+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.558 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.555+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2770451729}) 2024-09-16T04:02:34.559 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.555+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3872621594}) 2024-09-16T04:02:34.560 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.555+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.560 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.555+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=177104359}) 2024-09-16T04:02:34.560 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.555+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3314143766}) 2024-09-16T04:02:34.561 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.555+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.561 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.555+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=177104359}) 2024-09-16T04:02:34.561 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.555+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3229888977}) 2024-09-16T04:02:34.562 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.559+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.563 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.559+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=607086762}) 2024-09-16T04:02:34.563 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.559+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1302556707}) 2024-09-16T04:02:34.563 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.559+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.563 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.559+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=607086762}) 2024-09-16T04:02:34.563 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.559+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=14615477}) 2024-09-16T04:02:34.565 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.559+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.565 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.559+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=352273880}) 2024-09-16T04:02:34.565 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.559+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1142872731}) 2024-09-16T04:02:34.565 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.559+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.565 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.559+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=352273880}) 2024-09-16T04:02:34.566 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.559+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2606776108}) 2024-09-16T04:02:34.566 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.563+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.567 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.563+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3543449108}) 2024-09-16T04:02:34.567 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.563+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3313056640}) 2024-09-16T04:02:34.567 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.563+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.567 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.563+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3543449108}) 2024-09-16T04:02:34.567 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.563+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1385623433}) 2024-09-16T04:02:34.568 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.563+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.568 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.563+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2630417086}) 2024-09-16T04:02:34.569 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.563+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=596427545}) 2024-09-16T04:02:34.569 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.563+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.569 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.563+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2630417086}) 2024-09-16T04:02:34.569 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.563+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1417431228}) 2024-09-16T04:02:34.570 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.567+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.570 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.567+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1127243758}) 2024-09-16T04:02:34.570 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.567+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2923406096}) 2024-09-16T04:02:34.570 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.567+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.571 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.567+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1127243758}) 2024-09-16T04:02:34.571 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.567+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=588198880}) 2024-09-16T04:02:34.572 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.567+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.572 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.567+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=611887752}) 2024-09-16T04:02:34.572 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.567+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=267986435}) 2024-09-16T04:02:34.572 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.567+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.573 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.567+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=611887752}) 2024-09-16T04:02:34.573 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.567+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=669106757}) 2024-09-16T04:02:34.574 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.571+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.574 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.571+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2604444340}) 2024-09-16T04:02:34.574 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.571+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2975452787}) 2024-09-16T04:02:34.574 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.571+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.574 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.571+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2604444340}) 2024-09-16T04:02:34.574 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.571+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1076651863}) 2024-09-16T04:02:34.576 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.571+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.577 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.571+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1366444226}) 2024-09-16T04:02:34.577 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.571+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1660129558}) 2024-09-16T04:02:34.577 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.571+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.578 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.571+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1366444226}) 2024-09-16T04:02:34.578 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.571+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3459283257}) 2024-09-16T04:02:34.579 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.575+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.579 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.575+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3085423028}) 2024-09-16T04:02:34.579 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.575+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3746214629}) 2024-09-16T04:02:34.579 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.575+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.580 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.575+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3085423028}) 2024-09-16T04:02:34.580 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.575+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4159069346}) 2024-09-16T04:02:34.581 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.575+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.581 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.575+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1223834607}) 2024-09-16T04:02:34.582 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.575+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3711618389}) 2024-09-16T04:02:34.583 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.575+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.583 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.575+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1223834607}) 2024-09-16T04:02:34.583 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.575+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1004059427}) 2024-09-16T04:02:34.585 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.579+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.585 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.579+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3708154487}) 2024-09-16T04:02:34.585 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.579+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3707206893}) 2024-09-16T04:02:34.586 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.579+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.586 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.579+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3708154487}) 2024-09-16T04:02:34.586 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.579+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4213828401}) 2024-09-16T04:02:34.586 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.579+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.586 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.579+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1750792141}) 2024-09-16T04:02:34.586 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.579+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1992445459}) 2024-09-16T04:02:34.586 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.579+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.586 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.579+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1750792141}) 2024-09-16T04:02:34.586 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.579+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=728798194}) 2024-09-16T04:02:34.587 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.583+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.587 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.583+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=333238376}) 2024-09-16T04:02:34.588 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.583+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3506270843}) 2024-09-16T04:02:34.588 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.583+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.588 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.583+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=333238376}) 2024-09-16T04:02:34.588 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.583+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2156651672}) 2024-09-16T04:02:34.589 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.583+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.589 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.583+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2559809754}) 2024-09-16T04:02:34.589 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.583+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1070614048}) 2024-09-16T04:02:34.589 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.583+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.589 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.583+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2559809754}) 2024-09-16T04:02:34.589 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.583+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3746995550}) 2024-09-16T04:02:34.590 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.587+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.590 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.587+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2045724323}) 2024-09-16T04:02:34.590 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.587+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3637344963}) 2024-09-16T04:02:34.590 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.587+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.590 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.587+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2045724323}) 2024-09-16T04:02:34.590 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.587+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=420563805}) 2024-09-16T04:02:34.592 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.587+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.592 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.587+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3920267532}) 2024-09-16T04:02:34.592 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.587+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=657384744}) 2024-09-16T04:02:34.593 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.587+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.593 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.587+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3920267532}) 2024-09-16T04:02:34.593 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.587+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3041448098}) 2024-09-16T04:02:34.594 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.591+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.594 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.591+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4062565528}) 2024-09-16T04:02:34.594 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.591+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3900368340}) 2024-09-16T04:02:34.594 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.591+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.594 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.591+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4062565528}) 2024-09-16T04:02:34.594 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.591+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=323452391}) 2024-09-16T04:02:34.596 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.591+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.596 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.591+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3436534792}) 2024-09-16T04:02:34.596 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.591+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1086858342}) 2024-09-16T04:02:34.596 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.591+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.596 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.591+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3436534792}) 2024-09-16T04:02:34.596 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.591+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2623765738}) 2024-09-16T04:02:34.598 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.595+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.598 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.595+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1084761893}) 2024-09-16T04:02:34.598 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.595+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2508969374}) 2024-09-16T04:02:34.598 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.595+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.598 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.595+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1084761893}) 2024-09-16T04:02:34.598 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.595+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1662026034}) 2024-09-16T04:02:34.599 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.595+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.600 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.595+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=221832920}) 2024-09-16T04:02:34.600 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.595+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4221421637}) 2024-09-16T04:02:34.600 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.595+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.600 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.595+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=221832920}) 2024-09-16T04:02:34.600 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.595+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3250176930}) 2024-09-16T04:02:34.601 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.599+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.601 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.599+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=269422738}) 2024-09-16T04:02:34.601 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.599+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2650962945}) 2024-09-16T04:02:34.602 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.599+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.602 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.599+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=269422738}) 2024-09-16T04:02:34.602 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.599+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2870752243}) 2024-09-16T04:02:34.603 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.599+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.603 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.599+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=666046902}) 2024-09-16T04:02:34.603 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.599+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3085957763}) 2024-09-16T04:02:34.603 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.599+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.603 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.599+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=666046902}) 2024-09-16T04:02:34.604 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.599+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2866846417}) 2024-09-16T04:02:34.605 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.603+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.605 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.603+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=743255145}) 2024-09-16T04:02:34.606 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.603+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3909507132}) 2024-09-16T04:02:34.606 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.603+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.606 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.603+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=743255145}) 2024-09-16T04:02:34.606 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.603+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2448088449}) 2024-09-16T04:02:34.608 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.603+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.608 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.603+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3391828637}) 2024-09-16T04:02:34.608 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.603+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=901631783}) 2024-09-16T04:02:34.608 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.603+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.609 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.603+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3391828637}) 2024-09-16T04:02:34.609 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.603+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3350876325}) 2024-09-16T04:02:34.610 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.603+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.610 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.603+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=139086494}) 2024-09-16T04:02:34.610 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.603+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2065117107}) 2024-09-16T04:02:34.610 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.607+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.610 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.607+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=139086494}) 2024-09-16T04:02:34.610 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.607+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4162369368}) 2024-09-16T04:02:34.611 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.607+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.611 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.607+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3866537804}) 2024-09-16T04:02:34.611 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.607+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=398329522}) 2024-09-16T04:02:34.611 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.607+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.611 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.607+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3866537804}) 2024-09-16T04:02:34.611 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.607+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2500223557}) 2024-09-16T04:02:34.613 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.607+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.613 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.607+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=61748423}) 2024-09-16T04:02:34.613 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.607+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2189442836}) 2024-09-16T04:02:34.617 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.607+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.617 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.607+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=61748423}) 2024-09-16T04:02:34.617 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.607+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4014178012}) 2024-09-16T04:02:34.619 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.611+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.619 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.611+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2936574729}) 2024-09-16T04:02:34.619 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.611+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=915143479}) 2024-09-16T04:02:34.619 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.611+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.619 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.611+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2936574729}) 2024-09-16T04:02:34.619 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.611+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3357443218}) 2024-09-16T04:02:34.619 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.611+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.619 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.611+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1261186415}) 2024-09-16T04:02:34.620 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.611+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2461168597}) 2024-09-16T04:02:34.620 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.611+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.620 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.611+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1261186415}) 2024-09-16T04:02:34.620 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.611+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4277332486}) 2024-09-16T04:02:34.620 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.615+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.620 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.615+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=215287286}) 2024-09-16T04:02:34.620 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.615+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=640462012}) 2024-09-16T04:02:34.620 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.615+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.620 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.615+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=215287286}) 2024-09-16T04:02:34.620 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.615+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3716977191}) 2024-09-16T04:02:34.621 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.615+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.622 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.615+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1618623443}) 2024-09-16T04:02:34.622 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.615+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2612683094}) 2024-09-16T04:02:34.622 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.615+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.622 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.615+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1618623443}) 2024-09-16T04:02:34.622 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.615+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2379881745}) 2024-09-16T04:02:34.622 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.619+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.622 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.619+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2822635061}) 2024-09-16T04:02:34.622 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.619+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2128521238}) 2024-09-16T04:02:34.622 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.619+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.622 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.619+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2822635061}) 2024-09-16T04:02:34.622 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.619+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3448091543}) 2024-09-16T04:02:34.624 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.619+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.624 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.619+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3526247599}) 2024-09-16T04:02:34.624 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.619+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=352354336}) 2024-09-16T04:02:34.624 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.619+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.624 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.619+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3526247599}) 2024-09-16T04:02:34.624 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.619+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1832702555}) 2024-09-16T04:02:34.626 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.623+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.626 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.623+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=644562757}) 2024-09-16T04:02:34.626 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.623+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3786984025}) 2024-09-16T04:02:34.626 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.623+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.626 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.623+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=644562757}) 2024-09-16T04:02:34.626 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.623+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3006081824}) 2024-09-16T04:02:34.627 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.623+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.627 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.623+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1300835542}) 2024-09-16T04:02:34.628 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.623+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1125702676}) 2024-09-16T04:02:34.628 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.623+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.628 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.623+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1300835542}) 2024-09-16T04:02:34.628 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.623+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3279463799}) 2024-09-16T04:02:34.629 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.627+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.629 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.627+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2273586995}) 2024-09-16T04:02:34.629 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.627+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=521522575}) 2024-09-16T04:02:34.630 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.627+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.630 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.627+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2273586995}) 2024-09-16T04:02:34.630 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.627+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3066165106}) 2024-09-16T04:02:34.631 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.627+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.631 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.627+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1860116804}) 2024-09-16T04:02:34.631 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.627+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1834258531}) 2024-09-16T04:02:34.631 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.627+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.632 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.627+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1860116804}) 2024-09-16T04:02:34.632 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.627+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2841060146}) 2024-09-16T04:02:34.633 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.631+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.634 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.631+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2151265823}) 2024-09-16T04:02:34.634 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.631+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2878811798}) 2024-09-16T04:02:34.634 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.631+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.634 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.631+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2151265823}) 2024-09-16T04:02:34.634 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.631+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2201663962}) 2024-09-16T04:02:34.636 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.631+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.636 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.631+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1861487289}) 2024-09-16T04:02:34.636 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.631+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1251808949}) 2024-09-16T04:02:34.636 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.631+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.636 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.631+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1861487289}) 2024-09-16T04:02:34.636 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.631+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4195529050}) 2024-09-16T04:02:34.638 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.635+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.639 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.635+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=604915087}) 2024-09-16T04:02:34.639 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.635+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2647190692}) 2024-09-16T04:02:34.639 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.635+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.639 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.635+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=604915087}) 2024-09-16T04:02:34.639 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.635+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2042515063}) 2024-09-16T04:02:34.641 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.639+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.641 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.639+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=787467827}) 2024-09-16T04:02:34.642 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.639+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3155715187}) 2024-09-16T04:02:34.642 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.639+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.642 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.639+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=787467827}) 2024-09-16T04:02:34.642 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.639+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2631687652}) 2024-09-16T04:02:34.643 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.639+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.644 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.639+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3500826715}) 2024-09-16T04:02:34.644 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.639+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=788860359}) 2024-09-16T04:02:34.644 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.639+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.644 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.639+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3500826715}) 2024-09-16T04:02:34.644 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.639+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2462703490}) 2024-09-16T04:02:34.646 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.643+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.646 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.643+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2285999546}) 2024-09-16T04:02:34.646 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.643+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3052830358}) 2024-09-16T04:02:34.646 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.643+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.646 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.643+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2285999546}) 2024-09-16T04:02:34.646 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.643+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1690734006}) 2024-09-16T04:02:34.648 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.643+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.648 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.643+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3336609525}) 2024-09-16T04:02:34.648 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.643+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1230962724}) 2024-09-16T04:02:34.648 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.643+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.648 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.643+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3336609525}) 2024-09-16T04:02:34.648 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.643+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1374020111}) 2024-09-16T04:02:34.650 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.647+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.650 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.647+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4203606210}) 2024-09-16T04:02:34.650 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.647+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4131171648}) 2024-09-16T04:02:34.650 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.647+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.650 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.647+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4203606210}) 2024-09-16T04:02:34.650 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.647+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2014293242}) 2024-09-16T04:02:34.652 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.647+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.652 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.647+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1045065719}) 2024-09-16T04:02:34.652 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.647+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2899629362}) 2024-09-16T04:02:34.652 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.647+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.652 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.647+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1045065719}) 2024-09-16T04:02:34.652 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.647+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1942771756}) 2024-09-16T04:02:34.654 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.651+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.654 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.651+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=710404315}) 2024-09-16T04:02:34.654 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.651+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4196140380}) 2024-09-16T04:02:34.655 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.651+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.655 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.651+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=710404315}) 2024-09-16T04:02:34.655 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.651+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=958429176}) 2024-09-16T04:02:34.656 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.651+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.656 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.651+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2419137947}) 2024-09-16T04:02:34.656 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.651+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1960845658}) 2024-09-16T04:02:34.657 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.651+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.657 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.651+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2419137947}) 2024-09-16T04:02:34.657 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.651+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3613637209}) 2024-09-16T04:02:34.658 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.655+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.659 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.655+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1755681076}) 2024-09-16T04:02:34.659 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.655+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2770290717}) 2024-09-16T04:02:34.659 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.655+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.659 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.655+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1755681076}) 2024-09-16T04:02:34.659 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.655+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=248367872}) 2024-09-16T04:02:34.660 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.655+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.661 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.655+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2599143846}) 2024-09-16T04:02:34.661 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.655+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=167462506}) 2024-09-16T04:02:34.661 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.655+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.661 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.655+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2599143846}) 2024-09-16T04:02:34.661 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.655+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2376339184}) 2024-09-16T04:02:34.662 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.659+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.663 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.659+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4044548287}) 2024-09-16T04:02:34.663 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.659+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4194032559}) 2024-09-16T04:02:34.663 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.659+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.663 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.659+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4044548287}) 2024-09-16T04:02:34.663 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.659+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=738665004}) 2024-09-16T04:02:34.671 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.667+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.671 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.667+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2987839164}) 2024-09-16T04:02:34.671 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.667+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2204123721}) 2024-09-16T04:02:34.671 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.667+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.671 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.667+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2987839164}) 2024-09-16T04:02:34.671 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.667+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1569139436}) 2024-09-16T04:02:34.677 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.671+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.677 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.671+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2408540682}) 2024-09-16T04:02:34.677 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.671+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1771762577}) 2024-09-16T04:02:34.677 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.675+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.678 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.675+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2408540682}) 2024-09-16T04:02:34.678 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.675+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3135230013}) 2024-09-16T04:02:34.721 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.715+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.721 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.715+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2589249027}) 2024-09-16T04:02:34.722 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.715+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2856186166}) 2024-09-16T04:02:34.722 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.715+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.722 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.715+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2589249027}) 2024-09-16T04:02:34.722 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.715+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3036022292}) 2024-09-16T04:02:34.723 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.719+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.724 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.719+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1387304435}) 2024-09-16T04:02:34.724 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.719+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=740272897}) 2024-09-16T04:02:34.724 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.719+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.724 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.719+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1387304435}) 2024-09-16T04:02:34.724 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.719+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2298677087}) 2024-09-16T04:02:34.725 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.723+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.726 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.723+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4062696894}) 2024-09-16T04:02:34.726 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.723+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1096394199}) 2024-09-16T04:02:34.726 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.723+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.726 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.723+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4062696894}) 2024-09-16T04:02:34.726 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.723+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1589959142}) 2024-09-16T04:02:34.728 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.723+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.728 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.723+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3502083549}) 2024-09-16T04:02:34.728 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.723+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1466621803}) 2024-09-16T04:02:34.728 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.723+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.728 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.723+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3502083549}) 2024-09-16T04:02:34.728 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.723+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=579082463}) 2024-09-16T04:02:34.730 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.727+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.731 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.727+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2797368459}) 2024-09-16T04:02:34.731 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.727+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4096563682}) 2024-09-16T04:02:34.731 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.727+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.731 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.727+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2797368459}) 2024-09-16T04:02:34.731 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.727+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2305373352}) 2024-09-16T04:02:34.732 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.727+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.733 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.727+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3475634640}) 2024-09-16T04:02:34.733 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.727+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2757848986}) 2024-09-16T04:02:34.733 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.727+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.733 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.727+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3475634640}) 2024-09-16T04:02:34.733 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.727+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2222489987}) 2024-09-16T04:02:34.735 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.731+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.735 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.731+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1447432934}) 2024-09-16T04:02:34.735 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.731+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2545999348}) 2024-09-16T04:02:34.735 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.731+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.735 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.731+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1447432934}) 2024-09-16T04:02:34.735 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.731+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=30997791}) 2024-09-16T04:02:34.743 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.739+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.743 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.739+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1439555348}) 2024-09-16T04:02:34.743 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.739+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1749168844}) 2024-09-16T04:02:34.743 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.739+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.743 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.739+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1439555348}) 2024-09-16T04:02:34.743 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.739+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=832244689}) 2024-09-16T04:02:34.746 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.743+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.746 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.743+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=951835914}) 2024-09-16T04:02:34.746 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.743+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2542333209}) 2024-09-16T04:02:34.746 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.743+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.746 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.743+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=951835914}) 2024-09-16T04:02:34.746 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.743+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1165553742}) 2024-09-16T04:02:34.748 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.743+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.748 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.743+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4109843454}) 2024-09-16T04:02:34.748 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.743+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3076545400}) 2024-09-16T04:02:34.749 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.743+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.749 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.743+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4109843454}) 2024-09-16T04:02:34.749 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.743+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2735721}) 2024-09-16T04:02:34.751 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.747+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.751 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.747+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4079108221}) 2024-09-16T04:02:34.751 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.747+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1356217684}) 2024-09-16T04:02:34.751 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.747+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.751 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.747+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4079108221}) 2024-09-16T04:02:34.751 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.747+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4124456383}) 2024-09-16T04:02:34.757 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.751+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.757 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.751+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2665956991}) 2024-09-16T04:02:34.757 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.751+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4118514299}) 2024-09-16T04:02:34.757 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.751+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.757 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.751+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2665956991}) 2024-09-16T04:02:34.757 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.751+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2043435139}) 2024-09-16T04:02:34.762 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.759+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.762 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.759+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=952966617}) 2024-09-16T04:02:34.763 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.759+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=371164575}) 2024-09-16T04:02:34.763 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.759+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.763 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.759+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=952966617}) 2024-09-16T04:02:34.763 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.759+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3573711379}) 2024-09-16T04:02:34.765 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.759+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.765 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.759+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3936515662}) 2024-09-16T04:02:34.765 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.759+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3696984431}) 2024-09-16T04:02:34.765 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.759+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.765 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.759+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3936515662}) 2024-09-16T04:02:34.765 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.759+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=256908219}) 2024-09-16T04:02:34.767 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.763+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.767 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.763+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=267192569}) 2024-09-16T04:02:34.767 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.763+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1314941162}) 2024-09-16T04:02:34.767 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.763+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.767 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.763+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=267192569}) 2024-09-16T04:02:34.767 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.763+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3517010987}) 2024-09-16T04:02:34.769 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.763+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.769 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.763+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3391488700}) 2024-09-16T04:02:34.769 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.763+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=502394345}) 2024-09-16T04:02:34.769 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.763+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.769 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.763+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3391488700}) 2024-09-16T04:02:34.769 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.763+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4288177152}) 2024-09-16T04:02:34.772 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.767+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.772 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.767+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3868926003}) 2024-09-16T04:02:34.772 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.767+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2957444021}) 2024-09-16T04:02:34.772 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.767+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.772 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.767+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3868926003}) 2024-09-16T04:02:34.772 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.767+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=337063570}) 2024-09-16T04:02:34.774 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.771+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.774 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.771+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=585854686}) 2024-09-16T04:02:34.774 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.771+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2516309710}) 2024-09-16T04:02:34.774 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.771+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.774 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.771+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=585854686}) 2024-09-16T04:02:34.774 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.771+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2287429246}) 2024-09-16T04:02:34.776 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.771+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.776 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.771+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4251684268}) 2024-09-16T04:02:34.776 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.771+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3082808653}) 2024-09-16T04:02:34.776 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.771+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.776 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.771+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4251684268}) 2024-09-16T04:02:34.776 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.771+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1774339992}) 2024-09-16T04:02:34.778 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.775+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.778 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.775+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=472607731}) 2024-09-16T04:02:34.778 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.775+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3226712529}) 2024-09-16T04:02:34.778 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.775+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.778 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.775+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=472607731}) 2024-09-16T04:02:34.778 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.775+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=935552574}) 2024-09-16T04:02:34.780 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.775+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.780 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.775+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2359620327}) 2024-09-16T04:02:34.780 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.775+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3243212496}) 2024-09-16T04:02:34.780 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.775+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.780 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.775+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2359620327}) 2024-09-16T04:02:34.780 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.775+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3969485062}) 2024-09-16T04:02:34.781 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.775+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.781 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.775+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4082948178}) 2024-09-16T04:02:34.781 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.775+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=982665242}) 2024-09-16T04:02:34.781 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.775+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.781 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.775+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4082948178}) 2024-09-16T04:02:34.781 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.775+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1077605245}) 2024-09-16T04:02:34.782 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.779+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.782 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.779+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3260041463}) 2024-09-16T04:02:34.782 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.779+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1850020565}) 2024-09-16T04:02:34.783 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.779+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.783 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.779+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3260041463}) 2024-09-16T04:02:34.783 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.779+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2909397777}) 2024-09-16T04:02:34.784 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.779+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.784 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.779+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2093444168}) 2024-09-16T04:02:34.784 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.779+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3909616919}) 2024-09-16T04:02:34.784 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.779+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.784 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.779+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2093444168}) 2024-09-16T04:02:34.784 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.779+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1914084463}) 2024-09-16T04:02:34.785 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.779+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.785 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.779+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1482860129}) 2024-09-16T04:02:34.785 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.779+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=575702024}) 2024-09-16T04:02:34.785 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.779+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.785 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.779+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1482860129}) 2024-09-16T04:02:34.785 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.779+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2907392352}) 2024-09-16T04:02:34.787 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.783+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.787 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.783+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1345180431}) 2024-09-16T04:02:34.787 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.783+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1090127933}) 2024-09-16T04:02:34.787 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.783+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.787 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.783+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1345180431}) 2024-09-16T04:02:34.787 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.783+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=122436967}) 2024-09-16T04:02:34.790 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.787+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.790 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.787+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=32,mdsmap=2,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=27} crc {logm=3513844295,mdsmap=3140514009,mgr=911650295,mgr_command_descs=2107673600,mgr_metadata=1367272202,mgrstat=4215487596,mon_config_key=85057112,monmap=3937513981,osd_metadata=4182959540,osd_pg_creating=2926369545,osd_snap=2411386450}) 2024-09-16T04:02:34.790 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.787+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=33,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=11,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=24} crc {logm=3873268632,mdsmap=4088524775,mgr=1669371261,mgr_command_descs=2107673600,mgr_metadata=1367272202,mgrstat=3451165199,mon_config_key=85057112,monmap=3937513981,osd_metadata=191738038,osd_pg_creating=2926369545,osd_snap=3841933698}) 2024-09-16T04:02:34.790 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.787+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.790 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.787+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=32,mdsmap=2,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=27} crc {logm=3513844295,mdsmap=3140514009,mgr=911650295,mgr_command_descs=2107673600,mgr_metadata=1367272202,mgrstat=4215487596,mon_config_key=85057112,monmap=3937513981,osd_metadata=4182959540,osd_pg_creating=2926369545,osd_snap=2411386450}) 2024-09-16T04:02:34.790 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.787+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=32,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=11,monmap=3,osd_metadata=4,osd_pg_creating=1,osd_snap=27} crc {logm=3360200827,mdsmap=4088524775,mgr=2858448270,mgr_command_descs=2107673600,mgr_metadata=1367272202,mgrstat=1940538598,mon_config_key=85057112,monmap=3937513981,osd_metadata=3653513097,osd_pg_creating=2926369545,osd_snap=1272183722}) 2024-09-16T04:02:34.792 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.787+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.792 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.787+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=14,osdmap=86} crc {osd_snap=3608842349,osdmap=756551940}) 2024-09-16T04:02:34.792 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.787+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=14,osdmap=86} crc {osd_snap=3533574614,osdmap=2982847822}) 2024-09-16T04:02:34.792 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.787+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.792 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.787+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=14,osdmap=86} crc {osd_snap=3608842349,osdmap=756551940}) 2024-09-16T04:02:34.792 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.787+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=14,osdmap=86} crc {osd_snap=3608842349,osdmap=2738740009}) 2024-09-16T04:02:34.794 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.791+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.794 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.791+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2772393176}) 2024-09-16T04:02:34.794 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.791+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1177419939}) 2024-09-16T04:02:34.794 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.791+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.795 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.791+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2772393176}) 2024-09-16T04:02:34.795 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.791+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1517296214}) 2024-09-16T04:02:34.812 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.807+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.812 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.807+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=89} crc {osdmap=1145377641}) 2024-09-16T04:02:34.812 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.807+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=89} crc {osdmap=1263039298}) 2024-09-16T04:02:34.812 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.807+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:02:34.812 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.807+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=89} crc {osdmap=1145377641}) 2024-09-16T04:02:34.812 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:02:34.807+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=88} crc {osdmap=578367426}) 2024-09-16T04:02:35.465 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T04:02:35.466 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:02:35.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.607+0000 7f4e55dcf640 10 monclient: get_monmap_and_config 2024-09-16T04:02:35.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.607+0000 7f4e55dcf640 10 monclient: build_initial_monmap 2024-09-16T04:02:35.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.607+0000 7f4e55dcf640 10 monclient: monmap: 2024-09-16T04:02:35.611 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:02:35.611 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:02:35.611 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:02:35.609376+0000 2024-09-16T04:02:35.611 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:02:35.609376+0000 2024-09-16T04:02:35.611 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:02:35.611 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:02:35.611 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:02:35.612 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:02:35.612 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:02:35.612 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:02:35.612 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:02:35.612 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:02:35.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.607+0000 7f4e55dcf640 1 Processor -- start 2024-09-16T04:02:35.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.607+0000 7f4e55dcf640 1 -- start start 2024-09-16T04:02:35.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.607+0000 7f4e55dcf640 10 monclient: init 2024-09-16T04:02:35.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.607+0000 7f4e55dcf640 10 monclient: _reopen_session rank -1 2024-09-16T04:02:35.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.607+0000 7f4e55dcf640 10 monclient: _add_conns ranks=[1,3,0,2,4] 2024-09-16T04:02:35.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.607+0000 7f4e55dcf640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4e500a39b0 addr v1:172.21.15.154:6789/0 2024-09-16T04:02:35.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.607+0000 7f4e55dcf640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f4e500a3d90 0x7f4e5013c380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:35.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.607+0000 7f4e55dcf640 10 monclient(hunting): picked mon.noname-a con 0x7f4e500a3d90 addr v2:172.21.15.42:3300/0 2024-09-16T04:02:35.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.607+0000 7f4e55dcf640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4e5013c8c0 addr v1:172.21.15.42:6789/0 2024-09-16T04:02:35.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.607+0000 7f4e55dcf640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4e50134290 con 0x7f4e5013c8c0 2024-09-16T04:02:35.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.607+0000 7f4e55dcf640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4e5009c140 con 0x7f4e500a39b0 2024-09-16T04:02:35.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.607+0000 7f4e55dcf640 10 monclient(hunting): start opening mon connection 2024-09-16T04:02:35.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.607+0000 7f4e55dcf640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f4e5009c420 con 0x7f4e500a3d90 2024-09-16T04:02:35.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.607+0000 7f4e55dcf640 10 monclient(hunting): _renew_subs 2024-09-16T04:02:35.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.607+0000 7f4e55dcf640 10 monclient(hunting): authenticate will time out at 2202.578713s 2024-09-16T04:02:35.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.607+0000 7f4e4effd640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f4e500a3d90 0x7f4e5013c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:02:35.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.607+0000 7f4e4ffff640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f4e5013c8c0 0x7f4e5013ecb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:36434/0 (socket says 172.21.15.42:36434) 2024-09-16T04:02:35.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.607+0000 7f4e4ffff640 1 -- 172.21.15.42:0/3571390455 learned_addr learned my addr 172.21.15.42:0/3571390455 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T04:02:35.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.607+0000 7f4e4effd640 10 monclient(hunting): get_auth_request con 0x7f4e500a3d90 auth_method 0 2024-09-16T04:02:35.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.607+0000 7f4e4effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:02:35.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.607+0000 7f4e4effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:02:35.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.607+0000 7f4e4effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:02:35.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.607+0000 7f4e4effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:02:35.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.607+0000 7f4e4effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:02:35.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.607+0000 7f4e4effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:02:35.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.607+0000 7f4e4effd640 10 monclient(hunting): handle_auth_done global_id 825537 payload 274 2024-09-16T04:02:35.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.607+0000 7f4e4effd640 1 -- 172.21.15.42:0/3571390455 >> v1:172.21.15.154:6789/0 conn(0x7f4e500a39b0 legacy=0x7f4e50133b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:02:35.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.607+0000 7f4e4effd640 1 -- 172.21.15.42:0/3571390455 >> v1:172.21.15.42:6789/0 conn(0x7f4e5013c8c0 legacy=0x7f4e5013ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:02:35.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.607+0000 7f4e4effd640 10 monclient: _finish_hunting 0 2024-09-16T04:02:35.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.607+0000 7f4e4effd640 1 monclient: found mon.noname-a 2024-09-16T04:02:35.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.607+0000 7f4e4effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:02:35.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.607+0000 7f4e4effd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:02:35.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.607+0000 7f4e4effd640 1 -- 172.21.15.42:0/3571390455 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e5006dd60 con 0x7f4e500a3d90 2024-09-16T04:02:35.622 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.611+0000 7f4e4effd640 1 --2- 172.21.15.42:0/3571390455 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4e500a3d90 0x7f4e5013c380 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f4e440097c0 tx=0x7f4e4401c320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44723e4c3ee832c6 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:02:35.622 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.611+0000 7f4e4e7fc640 1 -- 172.21.15.42:0/3571390455 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e440b0070 con 0x7f4e500a3d90 2024-09-16T04:02:35.622 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.611+0000 7f4e4e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:02:35.622 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.611+0000 7f4e4e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:02:35.622 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.611+0000 7f4e4e7fc640 10 monclient: dump: 2024-09-16T04:02:35.622 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:02:35.622 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:02:35.622 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:35.622 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:35.623 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:02:35.623 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:02:35.623 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:02:35.623 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:02:35.623 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:02:35.623 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:02:35.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.611+0000 7f4e4e7fc640 10 monclient: _finish_auth 0 2024-09-16T04:02:35.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.611+0000 7f4e4e7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:02:35.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.611+0000 7f4e4e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:02:35.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.611+0000 7f4e4e7fc640 1 -- 172.21.15.42:0/3571390455 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e440ab600 con 0x7f4e500a3d90 2024-09-16T04:02:35.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.611+0000 7f4e55dcf640 5 monclient: authenticate success, global_id 825537 2024-09-16T04:02:35.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.611+0000 7f4e55dcf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:02:35.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.611+0000 7f4e4e7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:02:35.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.611+0000 7f4e55dcf640 10 monclient: get_monmap_and_config success 2024-09-16T04:02:35.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.611+0000 7f4e4e7fc640 1 -- 172.21.15.42:0/3571390455 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e440ab9d0 con 0x7f4e500a3d90 2024-09-16T04:02:35.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.611+0000 7f4e4e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:02:35.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.611+0000 7f4e55dcf640 4 set_mon_vals no callback set 2024-09-16T04:02:35.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.611+0000 7f4e4e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:02:35.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.611+0000 7f4e4e7fc640 10 monclient: dump: 2024-09-16T04:02:35.624 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:02:35.624 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:02:35.624 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:35.624 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:35.624 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:02:35.624 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:02:35.624 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:02:35.624 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:02:35.625 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:02:35.625 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:02:35.625 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.611+0000 7f4e55dcf640 10 monclient: shutdown 2024-09-16T04:02:35.625 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.611+0000 7f4e55dcf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:02:35.625 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.611+0000 7f4e55dcf640 1 -- 172.21.15.42:0/3571390455 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4e500a3d90 msgr2=0x7f4e5013c380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:02:35.625 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.611+0000 7f4e55dcf640 1 --2- 172.21.15.42:0/3571390455 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4e500a3d90 0x7f4e5013c380 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f4e440097c0 tx=0x7f4e4401c320 comp rx=0 tx=0).stop 2024-09-16T04:02:35.625 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.611+0000 7f4e55dcf640 1 -- 172.21.15.42:0/3571390455 shutdown_connections 2024-09-16T04:02:35.625 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.611+0000 7f4e55dcf640 1 --2- 172.21.15.42:0/3571390455 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4e500a3d90 0x7f4e5013c380 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:35.625 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.611+0000 7f4e55dcf640 1 -- 172.21.15.42:0/3571390455 >> 172.21.15.42:0/3571390455 conn(0x7f4e500a9fb0 msgr2=0x7f4e500aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:02:35.625 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.611+0000 7f4e55dcf640 1 -- 172.21.15.42:0/3571390455 shutdown_connections 2024-09-16T04:02:35.625 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.611+0000 7f4e55dcf640 1 -- 172.21.15.42:0/3571390455 wait complete. 2024-09-16T04:02:35.625 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.611+0000 7f4e55dcf640 10 monclient: build_initial_monmap 2024-09-16T04:02:35.625 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.611+0000 7f4e55dcf640 10 monclient: monmap: 2024-09-16T04:02:35.626 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:02:35.626 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:02:35.626 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:02:35.626 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:02:35.626 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:02:35.626 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:02:35.626 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:02:35.626 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:02:35.626 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:02:35.626 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:02:35.626 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.611+0000 7f4e55dcf640 1 librados: starting msgr at 2024-09-16T04:02:35.626 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.611+0000 7f4e55dcf640 1 librados: starting objecter 2024-09-16T04:02:35.626 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.611+0000 7f4e55dcf640 1 Processor -- start 2024-09-16T04:02:35.626 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.611+0000 7f4e55dcf640 1 -- start start 2024-09-16T04:02:35.627 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.611+0000 7f4e55dcf640 1 librados: setting wanted keys 2024-09-16T04:02:35.627 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.611+0000 7f4e55dcf640 1 librados: calling monclient init 2024-09-16T04:02:35.627 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.611+0000 7f4e55dcf640 10 monclient: init 2024-09-16T04:02:35.627 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.611+0000 7f4e55dcf640 10 monclient: _reopen_session rank -1 2024-09-16T04:02:35.627 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.611+0000 7f4e55dcf640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:02:35.627 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.611+0000 7f4e55dcf640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4e500a39b0 0x7f4e50139580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:35.627 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.611+0000 7f4e55dcf640 10 monclient(hunting): picked mon.noname-c con 0x7f4e500a39b0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:02:35.627 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.611+0000 7f4e55dcf640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4e500a3d90 0x7f4e50139ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:35.627 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.611+0000 7f4e55dcf640 10 monclient(hunting): picked mon.noname-a con 0x7f4e500a3d90 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:02:35.627 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.611+0000 7f4e55dcf640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4e5013c8c0 0x7f4e5013a000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:35.627 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.611+0000 7f4e55dcf640 10 monclient(hunting): picked mon.noname-b con 0x7f4e5013c8c0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:02:35.627 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.611+0000 7f4e55dcf640 10 monclient(hunting): start opening mon connection 2024-09-16T04:02:35.627 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.611+0000 7f4e55dcf640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f4e50142f80 con 0x7f4e500a3d90 2024-09-16T04:02:35.627 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.611+0000 7f4e55dcf640 10 monclient(hunting): start opening mon connection 2024-09-16T04:02:35.628 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.611+0000 7f4e55dcf640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f4e50142e00 con 0x7f4e5013c8c0 2024-09-16T04:02:35.628 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.611+0000 7f4e55dcf640 10 monclient(hunting): start opening mon connection 2024-09-16T04:02:35.628 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.611+0000 7f4e55dcf640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f4e50143100 con 0x7f4e500a39b0 2024-09-16T04:02:35.628 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.611+0000 7f4e55dcf640 10 monclient(hunting): _renew_subs 2024-09-16T04:02:35.628 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.611+0000 7f4e55dcf640 10 monclient(hunting): authenticate will time out at 2202.583753s 2024-09-16T04:02:35.628 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.611+0000 7f4e4effd640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4e500a3d90 0x7f4e50139ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:02:35.628 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.611+0000 7f4e4f7fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4e500a39b0 0x7f4e50139580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:02:35.628 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.611+0000 7f4e4effd640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4e500a3d90 0x7f4e50139ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:53742/0 (socket says 172.21.15.42:53742) 2024-09-16T04:02:35.628 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.611+0000 7f4e4effd640 1 -- 172.21.15.42:0/2507671103 learned_addr learned my addr 172.21.15.42:0/2507671103 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:02:35.628 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.611+0000 7f4e4effd640 10 monclient(hunting): get_auth_request con 0x7f4e500a3d90 auth_method 0 2024-09-16T04:02:35.628 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.611+0000 7f4e4f7fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4e500a39b0 0x7f4e50139580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:33882/0 (socket says 172.21.15.42:33882) 2024-09-16T04:02:35.628 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.611+0000 7f4e4effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:02:35.628 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.611+0000 7f4e4effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:02:35.629 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.611+0000 7f4e4effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:02:35.629 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.611+0000 7f4e4f7fe640 10 monclient(hunting): get_auth_request con 0x7f4e500a39b0 auth_method 0 2024-09-16T04:02:35.629 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.611+0000 7f4e4f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:02:35.629 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.611+0000 7f4e4f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:02:35.629 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.611+0000 7f4e4f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:02:35.629 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.615+0000 7f4e4effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:02:35.629 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.615+0000 7f4e4effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:02:35.629 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.615+0000 7f4e4effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:02:35.629 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.615+0000 7f4e4f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:02:35.629 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.615+0000 7f4e4f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:02:35.629 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.615+0000 7f4e4ffff640 1 --2- 172.21.15.42:0/2507671103 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4e5013c8c0 0x7f4e5013a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:02:35.629 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.615+0000 7f4e4f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:02:35.629 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.615+0000 7f4e4ffff640 10 monclient(hunting): get_auth_request con 0x7f4e5013c8c0 auth_method 0 2024-09-16T04:02:35.630 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.615+0000 7f4e4ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:02:35.630 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.615+0000 7f4e4ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:02:35.630 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.615+0000 7f4e4ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:02:35.630 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.615+0000 7f4e4effd640 10 monclient(hunting): handle_auth_done global_id 825543 payload 824 2024-09-16T04:02:35.630 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.615+0000 7f4e4effd640 1 -- 172.21.15.42:0/2507671103 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4e500a39b0 msgr2=0x7f4e50139580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:02:35.630 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.615+0000 7f4e4effd640 1 --2- 172.21.15.42:0/2507671103 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4e500a39b0 0x7f4e50139580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:35.630 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.615+0000 7f4e4effd640 1 -- 172.21.15.42:0/2507671103 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4e5013c8c0 msgr2=0x7f4e5013a000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:02:35.630 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.615+0000 7f4e4effd640 1 --2- 172.21.15.42:0/2507671103 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4e5013c8c0 0x7f4e5013a000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:35.630 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.615+0000 7f4e4effd640 10 monclient: _finish_hunting 0 2024-09-16T04:02:35.630 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.615+0000 7f4e4effd640 1 monclient: found mon.noname-a 2024-09-16T04:02:35.630 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.615+0000 7f4e4effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:02:35.630 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.615+0000 7f4e4effd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:02:35.630 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.615+0000 7f4e4effd640 1 -- 172.21.15.42:0/2507671103 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e5013b430 con 0x7f4e500a3d90 2024-09-16T04:02:35.630 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.615+0000 7f4e4f7fe640 1 --2- 172.21.15.42:0/2507671103 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4e500a39b0 0x7f4e50139580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:02:35.631 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.615+0000 7f4e4effd640 1 --2- 172.21.15.42:0/2507671103 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4e500a3d90 0x7f4e50139ac0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f4e44018f30 tx=0x7f4e4401ab60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:02:35.631 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.615+0000 7f4e37fff640 10 client.?.objecter ms_handle_connect 0x7f4e500a3d90 2024-09-16T04:02:35.631 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.615+0000 7f4e37fff640 10 client.?.objecter resend_mon_ops 2024-09-16T04:02:35.631 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.615+0000 7f4e37fff640 1 -- 172.21.15.42:0/2507671103 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e440b0050 con 0x7f4e500a3d90 2024-09-16T04:02:35.631 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.615+0000 7f4e37fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:02:35.631 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.615+0000 7f4e37fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:02:35.631 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.615+0000 7f4e37fff640 10 monclient: dump: 2024-09-16T04:02:35.631 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:02:35.631 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:02:35.631 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:35.631 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:35.631 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:02:35.631 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:02:35.632 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:02:35.632 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:02:35.632 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:02:35.632 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:02:35.632 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.615+0000 7f4e37fff640 10 monclient: _finish_auth 0 2024-09-16T04:02:35.632 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.615+0000 7f4e37fff640 10 monclient: _check_auth_tickets 2024-09-16T04:02:35.632 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.615+0000 7f4e37fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:02:35.632 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.615+0000 7f4e37fff640 1 -- 172.21.15.42:0/2507671103 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e440bc070 con 0x7f4e500a3d90 2024-09-16T04:02:35.632 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.615+0000 7f4e55dcf640 5 monclient: authenticate success, global_id 825543 2024-09-16T04:02:35.632 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.615+0000 7f4e55dcf640 10 monclient: _renew_subs 2024-09-16T04:02:35.632 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.615+0000 7f4e55dcf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:02:35.632 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.615+0000 7f4e55dcf640 1 -- 172.21.15.42:0/2507671103 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e50140120 con 0x7f4e500a3d90 2024-09-16T04:02:35.632 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.615+0000 7f4e37fff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:02:35.633 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.615+0000 7f4e37fff640 1 -- 172.21.15.42:0/2507671103 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e440aac00 con 0x7f4e500a3d90 2024-09-16T04:02:35.633 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.615+0000 7f4e37fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:02:35.633 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.615+0000 7f4e37fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:02:35.633 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.615+0000 7f4e37fff640 10 monclient: dump: 2024-09-16T04:02:35.633 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:02:35.633 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:02:35.633 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:35.633 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:35.633 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:02:35.633 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:02:35.633 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:02:35.633 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:02:35.633 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:02:35.633 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:02:35.634 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.615+0000 7f4e55dcf640 10 client.825543.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:02:35.634 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.615+0000 7f4e4dffb640 4 set_mon_vals no callback set 2024-09-16T04:02:35.634 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.615+0000 7f4e55dcf640 10 monclient: _renew_subs 2024-09-16T04:02:35.634 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.615+0000 7f4e55dcf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:02:35.636 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.615+0000 7f4e55dcf640 1 -- 172.21.15.42:0/2507671103 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e5006f680 con 0x7f4e500a3d90 2024-09-16T04:02:35.636 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.615+0000 7f4e55dcf640 1 librados: init done 2024-09-16T04:02:35.636 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.615+0000 7f4e37fff640 1 -- 172.21.15.42:0/2507671103 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f4e440c1460 con 0x7f4e500a3d90 2024-09-16T04:02:35.636 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.615+0000 7f4e55dcf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:02:35.636 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.615+0000 7f4e55dcf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:02:35.637 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.615+0000 7f4e55dcf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:02:35.637 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.615+0000 7f4e55dcf640 1 -- 172.21.15.42:0/2507671103 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e5013af90 con 0x7f4e500a3d90 2024-09-16T04:02:35.637 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.615+0000 7f4e37fff640 1 --2- 172.21.15.42:0/2507671103 >> v2:172.21.15.154:6812/254845342 conn(0x7f4e3c037840 0x7f4e3c039d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:35.637 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.615+0000 7f4e37fff640 1 -- 172.21.15.42:0/2507671103 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(632..632 src has 451..632) ==== 6367+0+0 (secure 0 0 0) 0x7f4e440e3e10 con 0x7f4e500a3d90 2024-09-16T04:02:35.637 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.615+0000 7f4e37fff640 10 client.825543.objecter ms_dispatch 0x7f4e50000ba0 osd_map(632..632 src has 451..632) 2024-09-16T04:02:35.637 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.615+0000 7f4e37fff640 3 client.825543.objecter handle_osd_map got epochs [632,632] > 0 2024-09-16T04:02:35.637 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.615+0000 7f4e37fff640 3 client.825543.objecter handle_osd_map decoding full epoch 632 2024-09-16T04:02:35.637 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.615+0000 7f4e37fff640 20 client.825543.objecter dump_active .. 0 homeless 2024-09-16T04:02:35.637 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.615+0000 7f4e4f7fe640 1 --2- 172.21.15.42:0/2507671103 >> v2:172.21.15.154:6812/254845342 conn(0x7f4e3c037840 0x7f4e3c039d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:02:35.637 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.615+0000 7f4e4f7fe640 10 monclient: get_auth_request con 0x7f4e3c037840 auth_method 0 2024-09-16T04:02:35.637 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.615+0000 7f4e4f7fe640 1 --2- 172.21.15.42:0/2507671103 >> v2:172.21.15.154:6812/254845342 conn(0x7f4e3c037840 0x7f4e3c039d00 secure :-1 s=READY pgs=3787 cs=0 l=1 rev1=1 crypto rx=0x7f4e40009cb0 tx=0x7f4e40009060 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:02:35.637 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.615+0000 7f4e37fff640 10 client.825543.objecter ms_handle_connect 0x7f4e3c037840 2024-09-16T04:02:35.638 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.619+0000 7f4e37fff640 1 -- 172.21.15.42:0/2507671103 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f4e44016030 con 0x7f4e500a3d90 2024-09-16T04:02:35.638 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.619+0000 7f4e37fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:02:35.638 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.619+0000 7f4e37fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:02:35.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.847+0000 7f4e55dcf640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:02:35.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.847+0000 7f4e55dcf640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:02:35.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.847+0000 7f4e55dcf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:02:35.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.847+0000 7f4e55dcf640 1 -- 172.21.15.42:0/2507671103 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f4e50133b80 con 0x7f4e500a3d90 2024-09-16T04:02:35.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.847+0000 7f4e37fff640 1 -- 172.21.15.42:0/2507671103 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7f4e440c03c0 con 0x7f4e500a3d90 2024-09-16T04:02:35.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.847+0000 7f4e37fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:02:35.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.847+0000 7f4e37fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:02:35.851 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":184,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":4,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:02:35.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.847+0000 7f4e55dcf640 10 librados: watch_flush enter 2024-09-16T04:02:35.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.847+0000 7f4e55dcf640 10 librados: watch_flush exit 2024-09-16T04:02:35.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.847+0000 7f4e55dcf640 20 client.825543.objecter shutdown clearing up homeless session... 2024-09-16T04:02:35.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.847+0000 7f4e55dcf640 10 client.825543.objecter successfully canceled tick 2024-09-16T04:02:35.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.847+0000 7f4e55dcf640 1 -- 172.21.15.42:0/2507671103 >> v2:172.21.15.154:6812/254845342 conn(0x7f4e3c037840 msgr2=0x7f4e3c039d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:02:35.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.847+0000 7f4e55dcf640 1 --2- 172.21.15.42:0/2507671103 >> v2:172.21.15.154:6812/254845342 conn(0x7f4e3c037840 0x7f4e3c039d00 secure :-1 s=READY pgs=3787 cs=0 l=1 rev1=1 crypto rx=0x7f4e40009cb0 tx=0x7f4e40009060 comp rx=0 tx=0).stop 2024-09-16T04:02:35.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.847+0000 7f4e55dcf640 10 monclient: shutdown 2024-09-16T04:02:35.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.847+0000 7f4e55dcf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:02:35.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.847+0000 7f4e55dcf640 1 -- 172.21.15.42:0/2507671103 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4e500a3d90 msgr2=0x7f4e50139ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:02:35.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.847+0000 7f4e55dcf640 1 --2- 172.21.15.42:0/2507671103 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4e500a3d90 0x7f4e50139ac0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f4e44018f30 tx=0x7f4e4401ab60 comp rx=0 tx=0).stop 2024-09-16T04:02:35.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.847+0000 7f4e55dcf640 1 -- 172.21.15.42:0/2507671103 shutdown_connections 2024-09-16T04:02:35.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.847+0000 7f4e55dcf640 1 --2- 172.21.15.42:0/2507671103 >> v2:172.21.15.154:6812/254845342 conn(0x7f4e3c037840 0x7f4e3c039d00 unknown :-1 s=CLOSED pgs=3787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:35.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.847+0000 7f4e55dcf640 1 --2- 172.21.15.42:0/2507671103 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4e5013c8c0 0x7f4e5013a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:35.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.847+0000 7f4e55dcf640 1 --2- 172.21.15.42:0/2507671103 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4e500a3d90 0x7f4e50139ac0 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:35.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.847+0000 7f4e55dcf640 1 --2- 172.21.15.42:0/2507671103 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4e500a39b0 0x7f4e50139580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:35.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.847+0000 7f4e55dcf640 1 -- 172.21.15.42:0/2507671103 >> 172.21.15.42:0/2507671103 conn(0x7f4e500a9fb0 msgr2=0x7f4e50131980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:02:35.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.847+0000 7f4e55dcf640 1 -- 172.21.15.42:0/2507671103 shutdown_connections 2024-09-16T04:02:35.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.847+0000 7f4e55dcf640 1 -- 172.21.15.42:0/2507671103 wait complete. 2024-09-16T04:02:35.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.847+0000 7f4e55dcf640 1 librados: shutdown 2024-09-16T04:02:35.870 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T04:02:35.870 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T04:02:35.870 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T04:02:35.870 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T04:02:35.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.987+0000 7f2a50ee2640 10 monclient: get_monmap_and_config 2024-09-16T04:02:35.992 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.987+0000 7f2a50ee2640 10 monclient: build_initial_monmap 2024-09-16T04:02:35.992 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.987+0000 7f2a50ee2640 10 monclient: monmap: 2024-09-16T04:02:35.992 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:02:35.992 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:02:35.992 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:02:35.990478+0000 2024-09-16T04:02:35.992 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:02:35.990478+0000 2024-09-16T04:02:35.992 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:02:35.992 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:02:35.992 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:02:35.992 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:02:35.992 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:02:35.992 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:02:35.993 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:02:35.993 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:02:35.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.987+0000 7f2a50ee2640 1 Processor -- start 2024-09-16T04:02:35.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.987+0000 7f2a50ee2640 1 -- start start 2024-09-16T04:02:35.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.987+0000 7f2a50ee2640 10 monclient: init 2024-09-16T04:02:35.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.991+0000 7f2a50ee2640 10 monclient: _reopen_session rank -1 2024-09-16T04:02:35.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.991+0000 7f2a50ee2640 10 monclient: _add_conns ranks=[2,1,3,4,0] 2024-09-16T04:02:35.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.991+0000 7f2a50ee2640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2a4c0a1330 0x7f2a4c09f180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:35.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.991+0000 7f2a50ee2640 10 monclient(hunting): picked mon.noname-b con 0x7f2a4c0a1330 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:02:35.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.991+0000 7f2a50ee2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2a4c0a1710 addr v1:172.21.15.154:6789/0 2024-09-16T04:02:35.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.991+0000 7f2a50ee2640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f2a4c1393a0 0x7f2a4c13b790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:35.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.991+0000 7f2a50ee2640 10 monclient(hunting): picked mon.noname-a con 0x7f2a4c1393a0 addr v2:172.21.15.42:3300/0 2024-09-16T04:02:35.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.991+0000 7f2a50ee2640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2a4c09e4c0 con 0x7f2a4c0a1710 2024-09-16T04:02:35.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.991+0000 7f2a50ee2640 10 monclient(hunting): start opening mon connection 2024-09-16T04:02:35.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.991+0000 7f2a50ee2640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f2a4c09edf0 con 0x7f2a4c1393a0 2024-09-16T04:02:35.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.991+0000 7f2a50ee2640 10 monclient(hunting): start opening mon connection 2024-09-16T04:02:35.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.991+0000 7f2a50ee2640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f2a4c09cd60 con 0x7f2a4c0a1330 2024-09-16T04:02:35.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.991+0000 7f2a50ee2640 10 monclient(hunting): _renew_subs 2024-09-16T04:02:35.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.991+0000 7f2a50ee2640 10 monclient(hunting): authenticate will time out at 2202.960425s 2024-09-16T04:02:35.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.991+0000 7f2a4ad9d640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f2a4c1393a0 0x7f2a4c13b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:02:35.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.991+0000 7f2a4a59c640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2a4c0a1330 0x7f2a4c09f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:02:35.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.991+0000 7f2a4ad9d640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f2a4c1393a0 0x7f2a4c13b790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:53748/0 (socket says 172.21.15.42:53748) 2024-09-16T04:02:35.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.991+0000 7f2a4ad9d640 1 -- 172.21.15.42:0/29013624 learned_addr learned my addr 172.21.15.42:0/29013624 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:02:35.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.991+0000 7f2a4ad9d640 10 monclient(hunting): get_auth_request con 0x7f2a4c1393a0 auth_method 0 2024-09-16T04:02:35.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.991+0000 7f2a4ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:02:35.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.991+0000 7f2a4ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:02:35.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.991+0000 7f2a4ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:02:35.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.991+0000 7f2a4a59c640 10 monclient(hunting): get_auth_request con 0x7f2a4c0a1330 auth_method 0 2024-09-16T04:02:35.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.991+0000 7f2a4a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:02:35.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.991+0000 7f2a4a59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:02:35.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.991+0000 7f2a4a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:02:35.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.991+0000 7f2a4ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:02:35.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.991+0000 7f2a4ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:02:35.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.991+0000 7f2a4ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:02:35.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.991+0000 7f2a4a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:02:35.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.991+0000 7f2a4a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:02:35.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.991+0000 7f2a4a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:02:35.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.991+0000 7f2a4ad9d640 10 monclient(hunting): handle_auth_done global_id 825546 payload 274 2024-09-16T04:02:35.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.991+0000 7f2a4ad9d640 1 -- 172.21.15.42:0/29013624 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2a4c0a1330 msgr2=0x7f2a4c09f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:02:35.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.991+0000 7f2a4ad9d640 1 --2- 172.21.15.42:0/29013624 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2a4c0a1330 0x7f2a4c09f180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:35.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.991+0000 7f2a4ad9d640 1 -- 172.21.15.42:0/29013624 >> v1:172.21.15.154:6789/0 conn(0x7f2a4c0a1710 legacy=0x7f2a4c0a1b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:02:35.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.991+0000 7f2a4ad9d640 10 monclient: _finish_hunting 0 2024-09-16T04:02:35.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.991+0000 7f2a4ad9d640 1 monclient: found mon.noname-a 2024-09-16T04:02:35.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.991+0000 7f2a4ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:02:35.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.991+0000 7f2a4ad9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:02:35.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.991+0000 7f2a4ad9d640 1 -- 172.21.15.42:0/29013624 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a4c09f6c0 con 0x7f2a4c1393a0 2024-09-16T04:02:36.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.991+0000 7f2a4ad9d640 1 --2- 172.21.15.42:0/29013624 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2a4c1393a0 0x7f2a4c13b790 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f2a44009e50 tx=0x7f2a4401c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=429ea632f0e92d07 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:02:36.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.991+0000 7f2a49d9b640 1 -- 172.21.15.42:0/29013624 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a440a5640 con 0x7f2a4c1393a0 2024-09-16T04:02:36.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.991+0000 7f2a49d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:02:36.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.991+0000 7f2a49d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:02:36.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.991+0000 7f2a49d9b640 10 monclient: dump: 2024-09-16T04:02:36.005 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:02:36.005 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:02:36.005 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:36.005 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:36.005 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:02:36.005 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:02:36.005 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:02:36.005 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:02:36.006 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:02:36.006 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:02:36.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.991+0000 7f2a49d9b640 10 monclient: _finish_auth 0 2024-09-16T04:02:36.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.991+0000 7f2a49d9b640 10 monclient: _check_auth_tickets 2024-09-16T04:02:36.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.991+0000 7f2a49d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:02:36.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.991+0000 7f2a49d9b640 1 -- 172.21.15.42:0/29013624 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a440a57e0 con 0x7f2a4c1393a0 2024-09-16T04:02:36.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.991+0000 7f2a49d9b640 10 monclient: handle_config config(0 keys) 2024-09-16T04:02:36.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.991+0000 7f2a49d9b640 1 -- 172.21.15.42:0/29013624 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a440a5bb0 con 0x7f2a4c1393a0 2024-09-16T04:02:36.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.991+0000 7f2a49d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:02:36.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.991+0000 7f2a49d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:02:36.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.991+0000 7f2a49d9b640 10 monclient: dump: 2024-09-16T04:02:36.006 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:02:36.006 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:02:36.006 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:36.007 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:36.007 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:02:36.007 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:02:36.007 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:02:36.007 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:02:36.007 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:02:36.007 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:02:36.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.991+0000 7f2a50ee2640 5 monclient: authenticate success, global_id 825546 2024-09-16T04:02:36.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.991+0000 7f2a50ee2640 10 monclient: get_monmap_and_config success 2024-09-16T04:02:36.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.991+0000 7f2a50ee2640 4 set_mon_vals no callback set 2024-09-16T04:02:36.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.991+0000 7f2a50ee2640 10 monclient: shutdown 2024-09-16T04:02:36.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.991+0000 7f2a50ee2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:02:36.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.991+0000 7f2a50ee2640 1 -- 172.21.15.42:0/29013624 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2a4c1393a0 msgr2=0x7f2a4c13b790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:02:36.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.991+0000 7f2a50ee2640 1 --2- 172.21.15.42:0/29013624 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2a4c1393a0 0x7f2a4c13b790 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f2a44009e50 tx=0x7f2a4401c8b0 comp rx=0 tx=0).stop 2024-09-16T04:02:36.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.991+0000 7f2a50ee2640 1 -- 172.21.15.42:0/29013624 shutdown_connections 2024-09-16T04:02:36.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.991+0000 7f2a50ee2640 1 --2- 172.21.15.42:0/29013624 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2a4c1393a0 0x7f2a4c13b790 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:36.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.991+0000 7f2a50ee2640 1 --2- 172.21.15.42:0/29013624 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2a4c0a1330 0x7f2a4c09f180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:36.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.991+0000 7f2a50ee2640 1 -- 172.21.15.42:0/29013624 >> 172.21.15.42:0/29013624 conn(0x7f2a4c0a2420 msgr2=0x7f2a4c0a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:02:36.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.995+0000 7f2a50ee2640 1 -- 172.21.15.42:0/29013624 shutdown_connections 2024-09-16T04:02:36.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.995+0000 7f2a50ee2640 1 -- 172.21.15.42:0/29013624 wait complete. 2024-09-16T04:02:36.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.995+0000 7f2a50ee2640 10 monclient: build_initial_monmap 2024-09-16T04:02:36.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.995+0000 7f2a50ee2640 10 monclient: monmap: 2024-09-16T04:02:36.008 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:02:36.008 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:02:36.008 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:02:36.008 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:02:36.008 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:02:36.009 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:02:36.009 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:02:36.009 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:02:36.009 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:02:36.009 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:02:36.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.995+0000 7f2a50ee2640 1 librados: starting msgr at 2024-09-16T04:02:36.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.995+0000 7f2a50ee2640 1 librados: starting objecter 2024-09-16T04:02:36.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.995+0000 7f2a50ee2640 1 Processor -- start 2024-09-16T04:02:36.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.995+0000 7f2a50ee2640 1 -- start start 2024-09-16T04:02:36.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.995+0000 7f2a50ee2640 1 librados: setting wanted keys 2024-09-16T04:02:36.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.995+0000 7f2a50ee2640 1 librados: calling monclient init 2024-09-16T04:02:36.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.995+0000 7f2a50ee2640 10 monclient: init 2024-09-16T04:02:36.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.995+0000 7f2a50ee2640 10 monclient: _reopen_session rank -1 2024-09-16T04:02:36.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.995+0000 7f2a50ee2640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:02:36.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.995+0000 7f2a50ee2640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2a4c0a1330 0x7f2a4c1cee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:36.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.995+0000 7f2a50ee2640 10 monclient(hunting): picked mon.noname-a con 0x7f2a4c0a1330 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:02:36.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.995+0000 7f2a50ee2640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2a4c0a1710 0x7f2a4c1344c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:36.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.995+0000 7f2a50ee2640 10 monclient(hunting): picked mon.noname-b con 0x7f2a4c0a1710 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:02:36.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.995+0000 7f2a50ee2640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2a4c1393a0 0x7f2a4c13cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:36.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.995+0000 7f2a50ee2640 10 monclient(hunting): picked mon.noname-c con 0x7f2a4c1393a0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:02:36.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.995+0000 7f2a50ee2640 10 monclient(hunting): start opening mon connection 2024-09-16T04:02:36.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.995+0000 7f2a50ee2640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f2a4c13c870 con 0x7f2a4c0a1330 2024-09-16T04:02:36.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.995+0000 7f2a50ee2640 10 monclient(hunting): start opening mon connection 2024-09-16T04:02:36.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.995+0000 7f2a50ee2640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f2a4c13c6f0 con 0x7f2a4c0a1710 2024-09-16T04:02:36.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.995+0000 7f2a50ee2640 10 monclient(hunting): start opening mon connection 2024-09-16T04:02:36.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.995+0000 7f2a50ee2640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f2a4c13c9f0 con 0x7f2a4c1393a0 2024-09-16T04:02:36.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.995+0000 7f2a50ee2640 10 monclient(hunting): _renew_subs 2024-09-16T04:02:36.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.995+0000 7f2a50ee2640 10 monclient(hunting): authenticate will time out at 2202.965658s 2024-09-16T04:02:36.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.995+0000 7f2a4ad9d640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2a4c1393a0 0x7f2a4c13cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:02:36.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.995+0000 7f2a4a59c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2a4c0a1330 0x7f2a4c1cee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:02:36.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.995+0000 7f2a4ad9d640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2a4c1393a0 0x7f2a4c13cce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:33910/0 (socket says 172.21.15.42:33910) 2024-09-16T04:02:36.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.995+0000 7f2a4ad9d640 1 -- 172.21.15.42:0/1954462771 learned_addr learned my addr 172.21.15.42:0/1954462771 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:02:36.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.995+0000 7f2a4ad9d640 10 monclient(hunting): get_auth_request con 0x7f2a4c1393a0 auth_method 0 2024-09-16T04:02:36.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.995+0000 7f2a4ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:02:36.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.995+0000 7f2a4ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:02:36.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.995+0000 7f2a4ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:02:36.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.995+0000 7f2a4a59c640 10 monclient(hunting): get_auth_request con 0x7f2a4c0a1330 auth_method 0 2024-09-16T04:02:36.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.995+0000 7f2a4a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:02:36.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.995+0000 7f2a4a59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:02:36.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.995+0000 7f2a4a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:02:36.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.995+0000 7f2a4ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:02:36.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.995+0000 7f2a4ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:02:36.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.995+0000 7f2a4ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:02:36.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.995+0000 7f2a4a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:02:36.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.995+0000 7f2a4a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:02:36.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.995+0000 7f2a4a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:02:36.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.995+0000 7f2a4ad9d640 10 monclient(hunting): handle_auth_done global_id 964133 payload 824 2024-09-16T04:02:36.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.995+0000 7f2a4ad9d640 1 -- 172.21.15.42:0/1954462771 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2a4c0a1710 msgr2=0x7f2a4c1344c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:02:36.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.995+0000 7f2a4ad9d640 1 --2- 172.21.15.42:0/1954462771 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2a4c0a1710 0x7f2a4c1344c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:36.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.995+0000 7f2a4ad9d640 1 -- 172.21.15.42:0/1954462771 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2a4c0a1330 msgr2=0x7f2a4c1cee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:02:36.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.995+0000 7f2a4ad9d640 1 --2- 172.21.15.42:0/1954462771 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2a4c0a1330 0x7f2a4c1cee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:36.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.995+0000 7f2a4ad9d640 10 monclient: _finish_hunting 0 2024-09-16T04:02:36.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.995+0000 7f2a4ad9d640 1 monclient: found mon.noname-c 2024-09-16T04:02:36.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.995+0000 7f2a4ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:02:36.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.995+0000 7f2a4ad9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T04:02:36.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.995+0000 7f2a4ad9d640 1 -- 172.21.15.42:0/1954462771 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a4c0722f0 con 0x7f2a4c1393a0 2024-09-16T04:02:36.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.995+0000 7f2a4ad9d640 1 --2- 172.21.15.42:0/1954462771 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2a4c1393a0 0x7f2a4c13cce0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f2a440af040 tx=0x7f2a44019650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:02:36.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.995+0000 7f2a42ffd640 10 client.?.objecter ms_handle_connect 0x7f2a4c1393a0 2024-09-16T04:02:36.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.995+0000 7f2a42ffd640 10 client.?.objecter resend_mon_ops 2024-09-16T04:02:36.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.995+0000 7f2a42ffd640 1 -- 172.21.15.42:0/1954462771 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a440bc020 con 0x7f2a4c1393a0 2024-09-16T04:02:36.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.995+0000 7f2a42ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:02:36.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.995+0000 7f2a42ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:02:36.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.995+0000 7f2a42ffd640 10 monclient: dump: 2024-09-16T04:02:36.013 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:02:36.014 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:02:36.014 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:36.014 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:36.014 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:02:36.014 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:02:36.014 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:02:36.014 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:02:36.014 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:02:36.014 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:02:36.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.995+0000 7f2a42ffd640 10 monclient: _finish_auth 0 2024-09-16T04:02:36.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.995+0000 7f2a42ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:02:36.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.995+0000 7f2a42ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:02:36.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.995+0000 7f2a42ffd640 1 -- 172.21.15.42:0/1954462771 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a440b8670 con 0x7f2a4c1393a0 2024-09-16T04:02:36.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.995+0000 7f2a50ee2640 5 monclient: authenticate success, global_id 964133 2024-09-16T04:02:36.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.995+0000 7f2a42ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:02:36.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.995+0000 7f2a50ee2640 10 monclient: _renew_subs 2024-09-16T04:02:36.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.995+0000 7f2a42ffd640 1 -- 172.21.15.42:0/1954462771 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a44009510 con 0x7f2a4c1393a0 2024-09-16T04:02:36.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.995+0000 7f2a50ee2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:02:36.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.995+0000 7f2a50ee2640 1 -- 172.21.15.42:0/1954462771 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a4c070b60 con 0x7f2a4c1393a0 2024-09-16T04:02:36.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.995+0000 7f2a42ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:02:36.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.995+0000 7f2a4959a640 4 set_mon_vals no callback set 2024-09-16T04:02:36.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.995+0000 7f2a42ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:02:36.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.995+0000 7f2a42ffd640 10 monclient: dump: 2024-09-16T04:02:36.015 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:02:36.015 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:02:36.015 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:36.015 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:36.016 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:02:36.016 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:02:36.016 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:02:36.016 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:02:36.016 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:02:36.016 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:02:36.016 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.995+0000 7f2a50ee2640 10 client.964133.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:02:36.016 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.995+0000 7f2a50ee2640 10 monclient: _renew_subs 2024-09-16T04:02:36.016 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.995+0000 7f2a50ee2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:02:36.016 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.995+0000 7f2a50ee2640 1 -- 172.21.15.42:0/1954462771 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a4c13e800 con 0x7f2a4c1393a0 2024-09-16T04:02:36.016 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.995+0000 7f2a50ee2640 1 librados: init done 2024-09-16T04:02:36.016 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.995+0000 7f2a50ee2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:02:36.016 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.995+0000 7f2a50ee2640 1 --2- 172.21.15.42:0/1954462771 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2a4c071c90 0x7f2a4c02c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:36.016 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.995+0000 7f2a50ee2640 10 monclient(hunting): start opening mon connection 2024-09-16T04:02:36.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.995+0000 7f2a50ee2640 1 -- 172.21.15.42:0/1954462771 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f2a4c09f180 con 0x7f2a4c071c90 2024-09-16T04:02:36.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.995+0000 7f2a42ffd640 1 -- 172.21.15.42:0/1954462771 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f2a440bf020 con 0x7f2a4c1393a0 2024-09-16T04:02:36.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.995+0000 7f2a4a59c640 1 --2- 172.21.15.42:0/1954462771 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2a4c071c90 0x7f2a4c02c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:02:36.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.995+0000 7f2a4a59c640 10 monclient: get_auth_request con 0x7f2a4c071c90 auth_method 0 2024-09-16T04:02:36.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.995+0000 7f2a4a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:02:36.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.995+0000 7f2a4a59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:02:36.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.995+0000 7f2a4a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:02:36.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.995+0000 7f2a4a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:02:36.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.995+0000 7f2a4a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:02:36.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.995+0000 7f2a4a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:02:36.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.995+0000 7f2a42ffd640 1 --2- 172.21.15.42:0/1954462771 >> v2:172.21.15.154:6812/254845342 conn(0x7f2a2c037810 0x7f2a2c039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:36.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.995+0000 7f2a4a59c640 10 monclient(hunting): handle_auth_done global_id 825552 payload 824 2024-09-16T04:02:36.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.995+0000 7f2a42ffd640 1 -- 172.21.15.42:0/1954462771 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(632..632 src has 451..632) ==== 6367+0+0 (secure 0 0 0) 0x7f2a440e3ef0 con 0x7f2a4c1393a0 2024-09-16T04:02:36.018 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.995+0000 7f2a42ffd640 10 client.964133.objecter ms_dispatch 0x7f2a4c000ba0 osd_map(632..632 src has 451..632) 2024-09-16T04:02:36.018 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.995+0000 7f2a42ffd640 3 client.964133.objecter handle_osd_map got epochs [632,632] > 0 2024-09-16T04:02:36.018 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.995+0000 7f2a42ffd640 3 client.964133.objecter handle_osd_map decoding full epoch 632 2024-09-16T04:02:36.018 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.995+0000 7f2a4a59c640 1 -- 172.21.15.42:0/1954462771 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2a4c071a50 con 0x7f2a4c071c90 2024-09-16T04:02:36.018 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.995+0000 7f2a42ffd640 20 client.964133.objecter dump_active .. 0 homeless 2024-09-16T04:02:36.018 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.995+0000 7f2a42ffd640 10 client.964133.objecter ms_handle_connect 0x7f2a4c071c90 2024-09-16T04:02:36.018 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.995+0000 7f2a4a59c640 1 --2- 172.21.15.42:0/1954462771 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2a4c071c90 0x7f2a4c02c650 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f2a3c0119d0 tx=0x7f2a3c011e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:02:36.018 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.995+0000 7f2a42ffd640 10 client.964133.objecter resend_mon_ops 2024-09-16T04:02:36.018 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.999+0000 7f2a43fff640 1 --2- 172.21.15.42:0/1954462771 >> v2:172.21.15.154:6812/254845342 conn(0x7f2a2c037810 0x7f2a2c039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:02:36.018 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.999+0000 7f2a43fff640 10 monclient: get_auth_request con 0x7f2a2c037810 auth_method 0 2024-09-16T04:02:36.018 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.999+0000 7f2a42ffd640 1 -- 172.21.15.42:0/1954462771 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a3c019070 con 0x7f2a4c071c90 2024-09-16T04:02:36.018 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.999+0000 7f2a42ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:02:36.018 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.999+0000 7f2a42ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:02:36.019 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.999+0000 7f2a42ffd640 10 monclient: dump: 2024-09-16T04:02:36.019 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:02:36.019 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:02:36.019 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:36.019 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:36.019 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:02:36.019 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:02:36.019 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:02:36.019 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:02:36.019 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:02:36.019 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:02:36.019 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.999+0000 7f2a42ffd640 1 -- 172.21.15.42:0/1954462771 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f2a3c004300 con 0x7f2a4c071c90 2024-09-16T04:02:36.019 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.999+0000 7f2a42ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:02:36.020 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.999+0000 7f2a42ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:02:36.020 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.999+0000 7f2a42ffd640 1 -- 172.21.15.42:0/1954462771 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2a4c071c90 msgr2=0x7f2a4c02c650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:02:36.020 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.999+0000 7f2a42ffd640 1 --2- 172.21.15.42:0/1954462771 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2a4c071c90 0x7f2a4c02c650 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f2a3c0119d0 tx=0x7f2a3c011e90 comp rx=0 tx=0).stop 2024-09-16T04:02:36.020 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.999+0000 7f2a42ffd640 1 -- 172.21.15.42:0/1954462771 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2a4c071c90 msgr2=0x7f2a4c02c650 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:02:36.020 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.999+0000 7f2a42ffd640 1 --2- 172.21.15.42:0/1954462771 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2a4c071c90 0x7f2a4c02c650 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:36.020 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.999+0000 7f2a42ffd640 10 client.964133.objecter ms_handle_connect 0x7f2a2c037810 2024-09-16T04:02:36.020 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:35.999+0000 7f2a43fff640 1 --2- 172.21.15.42:0/1954462771 >> v2:172.21.15.154:6812/254845342 conn(0x7f2a2c037810 0x7f2a2c039cd0 secure :-1 s=READY pgs=3788 cs=0 l=1 rev1=1 crypto rx=0x7f2a34007fd0 tx=0x7f2a340079a0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:02:36.022 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.003+0000 7f2a50ee2640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:02:36.022 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.003+0000 7f2a50ee2640 1 --2- 172.21.15.42:0/1954462771 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2a4c134a00 0x7f2a4c02cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:36.022 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.003+0000 7f2a50ee2640 10 monclient(hunting): start opening mon connection 2024-09-16T04:02:36.022 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.003+0000 7f2a50ee2640 1 -- 172.21.15.42:0/1954462771 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f2a4c138aa0 con 0x7f2a4c134a00 2024-09-16T04:02:36.022 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.003+0000 7f2a4a59c640 1 --2- 172.21.15.42:0/1954462771 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2a4c134a00 0x7f2a4c02cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:02:36.022 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.003+0000 7f2a4a59c640 10 monclient: get_auth_request con 0x7f2a4c134a00 auth_method 0 2024-09-16T04:02:36.022 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.003+0000 7f2a4a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:02:36.022 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.003+0000 7f2a4a59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:02:36.022 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.003+0000 7f2a4a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:02:36.023 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.003+0000 7f2a4a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:02:36.023 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.003+0000 7f2a4a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:02:36.023 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.003+0000 7f2a4a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:02:36.023 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.003+0000 7f2a4a59c640 10 monclient(hunting): handle_auth_done global_id 825555 payload 824 2024-09-16T04:02:36.023 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.003+0000 7f2a4a59c640 1 -- 172.21.15.42:0/1954462771 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f2a4c0008e0 con 0x7f2a4c134a00 2024-09-16T04:02:36.023 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.003+0000 7f2a42ffd640 10 client.964133.objecter ms_handle_connect 0x7f2a4c134a00 2024-09-16T04:02:36.023 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.003+0000 7f2a42ffd640 10 client.964133.objecter resend_mon_ops 2024-09-16T04:02:36.023 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.003+0000 7f2a4a59c640 1 --2- 172.21.15.42:0/1954462771 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2a4c134a00 0x7f2a4c02cb90 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f2a3c0119d0 tx=0x7f2a3c0046d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:02:36.023 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.003+0000 7f2a42ffd640 1 -- 172.21.15.42:0/1954462771 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a3c019040 con 0x7f2a4c134a00 2024-09-16T04:02:36.023 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.003+0000 7f2a42ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:02:36.023 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.003+0000 7f2a42ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:02:36.023 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.003+0000 7f2a42ffd640 10 monclient: dump: 2024-09-16T04:02:36.023 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:02:36.024 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:02:36.024 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:36.024 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:36.024 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:02:36.024 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:02:36.024 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:02:36.024 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:02:36.024 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:02:36.024 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:02:36.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.003+0000 7f2a42ffd640 1 -- 172.21.15.42:0/1954462771 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4715 (secure 0 0 0) 0x7f2a3c00fe40 con 0x7f2a4c134a00 2024-09-16T04:02:36.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.003+0000 7f2a42ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:02:36.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.003+0000 7f2a42ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:02:36.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.003+0000 7f2a42ffd640 1 -- 172.21.15.42:0/1954462771 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2a4c134a00 msgr2=0x7f2a4c02cb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:02:36.025 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.003+0000 7f2a42ffd640 1 --2- 172.21.15.42:0/1954462771 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2a4c134a00 0x7f2a4c02cb90 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f2a3c0119d0 tx=0x7f2a3c0046d0 comp rx=0 tx=0).stop 2024-09-16T04:02:36.025 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.003+0000 7f2a42ffd640 1 -- 172.21.15.42:0/1954462771 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2a4c134a00 msgr2=0x7f2a4c02cb90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:02:36.025 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.003+0000 7f2a42ffd640 1 --2- 172.21.15.42:0/1954462771 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2a4c134a00 0x7f2a4c02cb90 secure :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f2a3c0119d0 tx=0x7f2a3c0046d0 comp rx=0 tx=0).stop 2024-09-16T04:02:36.025 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T04:02:36.025 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:02:36.025 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:02:36.025 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "leader", 2024-09-16T04:02:36.025 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 184, 2024-09-16T04:02:36.025 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 645858, 2024-09-16T04:02:36.026 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T04:02:36.026 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T04:02:36.026 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T04:02:36.026 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T04:02:36.026 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:02:36.026 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 4, 2024-09-16T04:02:36.026 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:02:36.026 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T04:02:36.026 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T04:02:36.026 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:02:36.026 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:02:36.026 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:02:36.026 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:02:36.026 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:02:36.027 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:02:36.027 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:02:36.027 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:02:36.027 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:02:36.027 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:02:36.027 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:02:36.027 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:02:36.027 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:02:36.027 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T04:02:36.027 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:02:36.027 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:02:36.027 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:02:36.027 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:02:36.027 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:02:36.027 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:02:36.028 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:02:36.028 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:02:36.028 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:02:36.028 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:02:36.028 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:02:36.028 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:02:36.028 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:02:36.028 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T04:02:36.028 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T04:02:36.028 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T04:02:36.028 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T04:02:36.028 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T04:02:36.028 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T04:02:36.029 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:02:36.029 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:02:36.029 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T04:02:36.029 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T04:02:36.029 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T04:02:36.029 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T04:02:36.029 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T04:02:36.029 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T04:02:36.029 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T04:02:36.029 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:02:36.029 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T04:02:36.029 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:02:36.029 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:02:36.029 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:02:36.030 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:02:36.030 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:02:36.030 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:02:36.030 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:02:36.030 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:02:36.030 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:02:36.030 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:02:36.030 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:02:36.030 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:02:36.030 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T04:02:36.030 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:02:36.032 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T04:02:36.032 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:02:36.032 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:02:36.032 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:02:36.032 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:02:36.032 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:02:36.032 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:02:36.033 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:02:36.033 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T04:02:36.033 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:02:36.033 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:02:36.033 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:02:36.033 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:02:36.033 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T04:02:36.033 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:02:36.033 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:02:36.033 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:02:36.033 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:02:36.033 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T04:02:36.033 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T04:02:36.033 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:02:36.034 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:02:36.034 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:02:36.034 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:02:36.034 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:02:36.034 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:02:36.034 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:02:36.034 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:02:36.034 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:02:36.034 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:02:36.034 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:02:36.034 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T04:02:36.034 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:02:36.034 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:02:36.034 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:02:36.035 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:02:36.035 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T04:02:36.035 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:02:36.035 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:02:36.035 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:02:36.035 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:02:36.035 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T04:02:36.035 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T04:02:36.035 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:02:36.035 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:02:36.035 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:02:36.035 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:02:36.035 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:02:36.035 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:02:36.036 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:02:36.036 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:02:36.036 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:02:36.036 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:02:36.036 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:02:36.036 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T04:02:36.036 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:02:36.036 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:02:36.036 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:02:36.036 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:02:36.036 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T04:02:36.036 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:02:36.036 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:02:36.036 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:02:36.037 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:02:36.037 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T04:02:36.037 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T04:02:36.037 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:02:36.037 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:02:36.037 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:02:36.037 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:02:36.037 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:02:36.037 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:02:36.037 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T04:02:36.037 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T04:02:36.037 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:02:36.037 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:02:36.037 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:02:36.038 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:02:36.038 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:02:36.038 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:02:36.038 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T04:02:36.038 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:02:36.038 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:02:36.038 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:02:36.038 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 5 2024-09-16T04:02:36.038 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:02:36.038 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:02:36.038 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T04:02:36.038 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:02:36.038 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:02:36.038 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:02:36.039 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 6 2024-09-16T04:02:36.039 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:02:36.039 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:02:36.039 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:02:36.039 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T04:02:36.039 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T04:02:36.040 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.003+0000 7f2a50ee2640 10 librados: watch_flush enter 2024-09-16T04:02:36.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.003+0000 7f2a50ee2640 10 librados: watch_flush exit 2024-09-16T04:02:36.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.007+0000 7f2a50ee2640 20 client.964133.objecter shutdown clearing up homeless session... 2024-09-16T04:02:36.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.007+0000 7f2a50ee2640 10 client.964133.objecter successfully canceled tick 2024-09-16T04:02:36.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.007+0000 7f2a50ee2640 1 -- 172.21.15.42:0/1954462771 >> v2:172.21.15.154:6812/254845342 conn(0x7f2a2c037810 msgr2=0x7f2a2c039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:02:36.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.007+0000 7f2a50ee2640 1 --2- 172.21.15.42:0/1954462771 >> v2:172.21.15.154:6812/254845342 conn(0x7f2a2c037810 0x7f2a2c039cd0 secure :-1 s=READY pgs=3788 cs=0 l=1 rev1=1 crypto rx=0x7f2a34007fd0 tx=0x7f2a340079a0 comp rx=0 tx=0).stop 2024-09-16T04:02:36.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.007+0000 7f2a50ee2640 10 monclient: shutdown 2024-09-16T04:02:36.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.007+0000 7f2a50ee2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:02:36.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.007+0000 7f2a50ee2640 1 -- 172.21.15.42:0/1954462771 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2a4c1393a0 msgr2=0x7f2a4c13cce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:02:36.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.007+0000 7f2a4ad9d640 1 -- 172.21.15.42:0/1954462771 reap_dead start 2024-09-16T04:02:36.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.007+0000 7f2a50ee2640 1 --2- 172.21.15.42:0/1954462771 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2a4c1393a0 0x7f2a4c13cce0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f2a440af040 tx=0x7f2a44019650 comp rx=0 tx=0).stop 2024-09-16T04:02:36.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.007+0000 7f2a50ee2640 1 -- 172.21.15.42:0/1954462771 shutdown_connections 2024-09-16T04:02:36.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.007+0000 7f2a50ee2640 1 --2- 172.21.15.42:0/1954462771 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2a4c1393a0 0x7f2a4c13cce0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:36.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.007+0000 7f2a50ee2640 1 -- 172.21.15.42:0/1954462771 >> 172.21.15.42:0/1954462771 conn(0x7f2a4c0a2420 msgr2=0x7f2a4c13a000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:02:36.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.007+0000 7f2a50ee2640 1 -- 172.21.15.42:0/1954462771 shutdown_connections 2024-09-16T04:02:36.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.007+0000 7f2a50ee2640 1 -- 172.21.15.42:0/1954462771 wait complete. 2024-09-16T04:02:36.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.007+0000 7f2a50ee2640 1 librados: shutdown 2024-09-16T04:02:36.042 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:02:36.042 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T04:02:36.042 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T04:02:36.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.155+0000 7f4062835640 10 monclient: get_monmap_and_config 2024-09-16T04:02:36.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.155+0000 7f4062835640 10 monclient: build_initial_monmap 2024-09-16T04:02:36.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.155+0000 7f4062835640 10 monclient: monmap: 2024-09-16T04:02:36.159 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:02:36.159 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:02:36.159 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:02:36.157186+0000 2024-09-16T04:02:36.159 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:02:36.157186+0000 2024-09-16T04:02:36.159 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:02:36.159 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:02:36.159 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:02:36.159 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:02:36.159 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:02:36.159 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:02:36.159 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:02:36.159 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:02:36.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.155+0000 7f4062835640 1 Processor -- start 2024-09-16T04:02:36.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.155+0000 7f4062835640 1 -- start start 2024-09-16T04:02:36.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.155+0000 7f4062835640 10 monclient: init 2024-09-16T04:02:36.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.155+0000 7f4062835640 10 monclient: _reopen_session rank -1 2024-09-16T04:02:36.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.155+0000 7f4062835640 10 monclient: _add_conns ranks=[1,4,0,3,2] 2024-09-16T04:02:36.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.155+0000 7f4062835640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f405c0a39b0 addr v1:172.21.15.154:6789/0 2024-09-16T04:02:36.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.155+0000 7f4062835640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f405c0a3d90 0x7f405c13c380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:36.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.155+0000 7f4062835640 10 monclient(hunting): picked mon.noname-c con 0x7f405c0a3d90 addr v2:172.21.15.154:3300/0 2024-09-16T04:02:36.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.155+0000 7f4062835640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f405c13c8c0 addr v1:172.21.15.42:6789/0 2024-09-16T04:02:36.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.155+0000 7f4062835640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f405c134290 con 0x7f405c13c8c0 2024-09-16T04:02:36.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.155+0000 7f4062835640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f405c09c140 con 0x7f405c0a39b0 2024-09-16T04:02:36.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.155+0000 7f4062835640 10 monclient(hunting): start opening mon connection 2024-09-16T04:02:36.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.155+0000 7f4062835640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f405c09c420 con 0x7f405c0a3d90 2024-09-16T04:02:36.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.155+0000 7f4062835640 10 monclient(hunting): _renew_subs 2024-09-16T04:02:36.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.155+0000 7f4062835640 10 monclient(hunting): authenticate will time out at 2203.126569s 2024-09-16T04:02:36.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.155+0000 7f4060dd2640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f405c13c8c0 0x7f405c13ecb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:36446/0 (socket says 172.21.15.42:36446) 2024-09-16T04:02:36.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.155+0000 7f4060dd2640 1 -- 172.21.15.42:0/1023426340 learned_addr learned my addr 172.21.15.42:0/1023426340 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T04:02:36.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.155+0000 7f405b7fe640 1 --2- 172.21.15.42:0/1023426340 >> v2:172.21.15.154:3300/0 conn(0x7f405c0a3d90 0x7f405c13c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:02:36.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.155+0000 7f405b7fe640 10 monclient(hunting): get_auth_request con 0x7f405c0a3d90 auth_method 0 2024-09-16T04:02:36.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.155+0000 7f405b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:02:36.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.155+0000 7f405b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:02:36.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.155+0000 7f405b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:02:36.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.155+0000 7f405affd640 1 -- 172.21.15.42:0/1023426340 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 351036053 0 0) 0x7f405c134290 con 0x7f405c13c8c0 2024-09-16T04:02:36.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.155+0000 7f405affd640 10 monclient(hunting): _negotiate 2024-09-16T04:02:36.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.155+0000 7f405affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:02:36.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.155+0000 7f405affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:02:36.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.155+0000 7f405affd640 10 monclient(hunting): my global_id is 825558 2024-09-16T04:02:36.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.155+0000 7f405affd640 1 -- 172.21.15.42:0/1023426340 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4044003530 con 0x7f405c13c8c0 2024-09-16T04:02:36.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.155+0000 7f405b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:02:36.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.155+0000 7f405b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:02:36.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.155+0000 7f405b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:02:36.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.155+0000 7f405affd640 1 -- 172.21.15.42:0/1023426340 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4204719477 0 0) 0x7f405c09c140 con 0x7f405c0a39b0 2024-09-16T04:02:36.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.155+0000 7f405affd640 10 monclient(hunting): _negotiate 2024-09-16T04:02:36.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.155+0000 7f405affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:02:36.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.155+0000 7f405affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:02:36.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.155+0000 7f405affd640 10 monclient(hunting): my global_id is 904627 2024-09-16T04:02:36.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.155+0000 7f405affd640 1 -- 172.21.15.42:0/1023426340 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f405c134290 con 0x7f405c0a39b0 2024-09-16T04:02:36.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.155+0000 7f405affd640 1 -- 172.21.15.42:0/1023426340 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3168580080 0 0) 0x7f4044003530 con 0x7f405c13c8c0 2024-09-16T04:02:36.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.155+0000 7f405affd640 1 -- 172.21.15.42:0/1023426340 >> v2:172.21.15.154:3300/0 conn(0x7f405c0a3d90 msgr2=0x7f405c13c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:02:36.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.155+0000 7f405affd640 1 --2- 172.21.15.42:0/1023426340 >> v2:172.21.15.154:3300/0 conn(0x7f405c0a3d90 0x7f405c13c380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:36.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.155+0000 7f405affd640 1 -- 172.21.15.42:0/1023426340 >> v1:172.21.15.154:6789/0 conn(0x7f405c0a39b0 legacy=0x7f405c133b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:02:36.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.155+0000 7f405affd640 10 monclient: _finish_hunting 0 2024-09-16T04:02:36.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.155+0000 7f405affd640 1 monclient: found mon.noname-a-legacy 2024-09-16T04:02:36.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.155+0000 7f405affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:02:36.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.155+0000 7f405affd640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.42:6789/0 2024-09-16T04:02:36.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.155+0000 7f405affd640 1 -- 172.21.15.42:0/1023426340 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f405c06dd60 con 0x7f405c13c8c0 2024-09-16T04:02:36.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.155+0000 7f405affd640 10 monclient: _finish_auth 0 2024-09-16T04:02:36.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.155+0000 7f405affd640 10 monclient: _check_auth_tickets 2024-09-16T04:02:36.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.155+0000 7f405affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:02:36.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.155+0000 7f4062835640 5 monclient: authenticate success, global_id 825558 2024-09-16T04:02:36.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.155+0000 7f4062835640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:02:36.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.155+0000 7f405affd640 1 -- 172.21.15.42:0/1023426340 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2177413734 0 0) 0x7f4054001a00 con 0x7f405c13c8c0 2024-09-16T04:02:36.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.155+0000 7f405affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:02:36.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.155+0000 7f405affd640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T04:02:36.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.155+0000 7f405affd640 10 monclient: dump: 2024-09-16T04:02:36.170 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:02:36.170 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:02:36.170 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:36.170 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:36.170 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:02:36.170 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:02:36.170 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:02:36.170 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:02:36.170 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:02:36.170 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:02:36.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.155+0000 7f405affd640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] but i'm connected to v1:172.21.15.42:6789/0, reconnecting 2024-09-16T04:02:36.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.155+0000 7f405affd640 10 monclient: _reopen_session rank -1 2024-09-16T04:02:36.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.155+0000 7f405affd640 1 -- 172.21.15.42:0/1023426340 >> v1:172.21.15.42:6789/0 conn(0x7f405c13c8c0 legacy=0x7f405c13ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:02:36.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.155+0000 7f405affd640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:02:36.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.155+0000 7f405affd640 1 --2- 172.21.15.42:0/1023426340 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4044003b50 0x7f4044003f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:36.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.155+0000 7f405affd640 10 monclient(hunting): picked mon.b con 0x7f4044003b50 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:02:36.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.155+0000 7f405affd640 1 --2- 172.21.15.42:0/1023426340 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4044004810 0x7f4044004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:36.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.155+0000 7f405affd640 10 monclient(hunting): picked mon.a con 0x7f4044004810 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:02:36.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.155+0000 7f405affd640 1 --2- 172.21.15.42:0/1023426340 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4044005220 0x7f404400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:36.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.155+0000 7f405affd640 10 monclient(hunting): picked mon.c con 0x7f4044005220 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:02:36.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.155+0000 7f405affd640 10 monclient(hunting): start opening mon connection 2024-09-16T04:02:36.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.155+0000 7f405affd640 1 -- 172.21.15.42:0/1023426340 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f405c09c420 con 0x7f4044004810 2024-09-16T04:02:36.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.155+0000 7f405affd640 10 monclient(hunting): start opening mon connection 2024-09-16T04:02:36.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.155+0000 7f405affd640 1 -- 172.21.15.42:0/1023426340 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f404400c150 con 0x7f4044003b50 2024-09-16T04:02:36.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.155+0000 7f405affd640 10 monclient(hunting): start opening mon connection 2024-09-16T04:02:36.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.155+0000 7f405affd640 1 -- 172.21.15.42:0/1023426340 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f404400c350 con 0x7f4044005220 2024-09-16T04:02:36.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.155+0000 7f405affd640 10 monclient(hunting): _renew_subs 2024-09-16T04:02:36.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.155+0000 7f405affd640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:02:36.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.155+0000 7f4062835640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-16T04:02:36.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.155+0000 7f405b7fe640 1 --2- 172.21.15.42:0/1023426340 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4044004810 0x7f4044004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:02:36.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.159+0000 7f405b7fe640 10 monclient(hunting): get_auth_request con 0x7f4044004810 auth_method 0 2024-09-16T04:02:36.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.159+0000 7f405b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:02:36.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.159+0000 7f405b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:02:36.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.159+0000 7f405b7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:02:36.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.159+0000 7f4060dd2640 1 --2- 172.21.15.42:0/1023426340 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4044005220 0x7f404400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:02:36.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.159+0000 7f4060dd2640 10 monclient(hunting): get_auth_request con 0x7f4044005220 auth_method 0 2024-09-16T04:02:36.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.159+0000 7f4060dd2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:02:36.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.159+0000 7f4060dd2640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:02:36.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.159+0000 7f4060dd2640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:02:36.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.159+0000 7f405b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:02:36.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.159+0000 7f405b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:02:36.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.159+0000 7f405bfff640 1 --2- 172.21.15.42:0/1023426340 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4044003b50 0x7f4044003f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:02:36.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.159+0000 7f405b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:02:36.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.159+0000 7f405bfff640 10 monclient(hunting): get_auth_request con 0x7f4044003b50 auth_method 0 2024-09-16T04:02:36.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.159+0000 7f405bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:02:36.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.159+0000 7f405bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:02:36.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.159+0000 7f405bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:02:36.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.159+0000 7f4060dd2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:02:36.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.159+0000 7f4060dd2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:02:36.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.159+0000 7f4060dd2640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:02:36.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.159+0000 7f405b7fe640 10 monclient(hunting): handle_auth_done global_id 825558 payload 293 2024-09-16T04:02:36.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.159+0000 7f405b7fe640 1 -- 172.21.15.42:0/1023426340 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4044005220 msgr2=0x7f404400b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:02:36.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.159+0000 7f405b7fe640 1 --2- 172.21.15.42:0/1023426340 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4044005220 0x7f404400b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:36.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.159+0000 7f405b7fe640 1 -- 172.21.15.42:0/1023426340 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4044003b50 msgr2=0x7f4044003f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:02:36.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.159+0000 7f405b7fe640 1 --2- 172.21.15.42:0/1023426340 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4044003b50 0x7f4044003f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:36.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.159+0000 7f405b7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:02:36.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.159+0000 7f405b7fe640 1 monclient: found mon.a 2024-09-16T04:02:36.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.159+0000 7f405b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:02:36.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.159+0000 7f405b7fe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:02:36.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.159+0000 7f405b7fe640 1 -- 172.21.15.42:0/1023426340 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4044003530 con 0x7f4044004810 2024-09-16T04:02:36.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.159+0000 7f405b7fe640 10 monclient: _finish_auth 0 2024-09-16T04:02:36.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.159+0000 7f405b7fe640 10 monclient: _check_auth_tickets 2024-09-16T04:02:36.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.159+0000 7f405b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:02:36.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.159+0000 7f4060dd2640 1 -- 172.21.15.42:0/1023426340 reap_dead start 2024-09-16T04:02:36.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.159+0000 7f405b7fe640 1 --2- 172.21.15.42:0/1023426340 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4044004810 0x7f4044004c90 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f4050009ab0 tx=0x7f405001ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bacfe8c6643c7be server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:02:36.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.159+0000 7f405affd640 1 -- 172.21.15.42:0/1023426340 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40500096b0 con 0x7f4044004810 2024-09-16T04:02:36.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.159+0000 7f405affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:02:36.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.159+0000 7f405affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:02:36.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.159+0000 7f405affd640 10 monclient: dump: 2024-09-16T04:02:36.175 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:02:36.175 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:02:36.175 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:36.175 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:36.175 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:02:36.175 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:02:36.175 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:02:36.175 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:02:36.175 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:02:36.176 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:02:36.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.159+0000 7f4062835640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:02:36.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.159+0000 7f405affd640 1 -- 172.21.15.42:0/1023426340 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40500a8420 con 0x7f4044004810 2024-09-16T04:02:36.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.159+0000 7f405affd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:02:36.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.159+0000 7f4062835640 10 monclient: get_monmap_and_config success 2024-09-16T04:02:36.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.159+0000 7f4062835640 4 set_mon_vals no callback set 2024-09-16T04:02:36.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.159+0000 7f405affd640 1 -- 172.21.15.42:0/1023426340 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40500a8810 con 0x7f4044004810 2024-09-16T04:02:36.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.159+0000 7f405affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:02:36.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.159+0000 7f405affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:02:36.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.159+0000 7f405affd640 10 monclient: dump: 2024-09-16T04:02:36.176 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:02:36.176 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:02:36.176 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:36.177 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:36.177 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:02:36.177 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:02:36.177 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:02:36.177 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:02:36.177 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:02:36.177 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:02:36.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.159+0000 7f4062835640 10 monclient: shutdown 2024-09-16T04:02:36.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.159+0000 7f4062835640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:02:36.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.159+0000 7f4062835640 1 -- 172.21.15.42:0/1023426340 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4044004810 msgr2=0x7f4044004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:02:36.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.159+0000 7f4062835640 1 --2- 172.21.15.42:0/1023426340 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4044004810 0x7f4044004c90 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f4050009ab0 tx=0x7f405001ca10 comp rx=0 tx=0).stop 2024-09-16T04:02:36.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.159+0000 7f4062835640 1 -- 172.21.15.42:0/1023426340 shutdown_connections 2024-09-16T04:02:36.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.159+0000 7f4062835640 1 --2- 172.21.15.42:0/1023426340 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4044004810 0x7f4044004c90 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:36.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.159+0000 7f4062835640 1 -- 172.21.15.42:0/1023426340 >> 172.21.15.42:0/1023426340 conn(0x7f405c0a9fb0 msgr2=0x7f405c0aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:02:36.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.159+0000 7f4062835640 1 -- 172.21.15.42:0/1023426340 shutdown_connections 2024-09-16T04:02:36.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.159+0000 7f4062835640 1 -- 172.21.15.42:0/1023426340 wait complete. 2024-09-16T04:02:36.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.159+0000 7f4062835640 10 monclient: build_initial_monmap 2024-09-16T04:02:36.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.159+0000 7f4062835640 10 monclient: monmap: 2024-09-16T04:02:36.178 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:02:36.178 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:02:36.178 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:02:36.178 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:02:36.178 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:02:36.178 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:02:36.178 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:02:36.178 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:02:36.178 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:02:36.179 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:02:36.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.163+0000 7f4062835640 1 librados: starting msgr at 2024-09-16T04:02:36.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.163+0000 7f4062835640 1 librados: starting objecter 2024-09-16T04:02:36.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.163+0000 7f4062835640 1 Processor -- start 2024-09-16T04:02:36.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.163+0000 7f4062835640 1 -- start start 2024-09-16T04:02:36.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.163+0000 7f4062835640 1 librados: setting wanted keys 2024-09-16T04:02:36.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.163+0000 7f4062835640 1 librados: calling monclient init 2024-09-16T04:02:36.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.163+0000 7f4062835640 10 monclient: init 2024-09-16T04:02:36.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.163+0000 7f4062835640 10 monclient: _reopen_session rank -1 2024-09-16T04:02:36.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.163+0000 7f4062835640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:02:36.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.163+0000 7f4062835640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4044004810 0x7f405c1cfcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:36.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.163+0000 7f4062835640 10 monclient(hunting): picked mon.noname-b con 0x7f4044004810 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:02:36.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.163+0000 7f4062835640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f405c1d0230 0x7f405c0a00d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:36.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.163+0000 7f4062835640 10 monclient(hunting): picked mon.noname-c con 0x7f405c1d0230 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:02:36.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.163+0000 7f4062835640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f405c1d0690 0x7f405c0a0610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:36.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.163+0000 7f4062835640 10 monclient(hunting): picked mon.noname-a con 0x7f405c1d0690 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:02:36.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.163+0000 7f4062835640 10 monclient(hunting): start opening mon connection 2024-09-16T04:02:36.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.163+0000 7f4062835640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f405c137ba0 con 0x7f405c1d0690 2024-09-16T04:02:36.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.163+0000 7f4062835640 10 monclient(hunting): start opening mon connection 2024-09-16T04:02:36.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.163+0000 7f4062835640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f405c137a20 con 0x7f4044004810 2024-09-16T04:02:36.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.163+0000 7f4062835640 10 monclient(hunting): start opening mon connection 2024-09-16T04:02:36.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.163+0000 7f4062835640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f405c137d20 con 0x7f405c1d0230 2024-09-16T04:02:36.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.163+0000 7f4062835640 10 monclient(hunting): _renew_subs 2024-09-16T04:02:36.187 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.163+0000 7f4062835640 10 monclient(hunting): authenticate will time out at 2203.132718s 2024-09-16T04:02:36.187 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.163+0000 7f405b7fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f405c1d0230 0x7f405c0a00d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:02:36.187 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.163+0000 7f4060dd2640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f405c1d0690 0x7f405c0a0610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:02:36.187 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.163+0000 7f405b7fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f405c1d0230 0x7f405c0a00d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:33924/0 (socket says 172.21.15.42:33924) 2024-09-16T04:02:36.187 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.163+0000 7f405b7fe640 1 -- 172.21.15.42:0/3389020659 learned_addr learned my addr 172.21.15.42:0/3389020659 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:02:36.187 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.163+0000 7f405b7fe640 10 monclient(hunting): get_auth_request con 0x7f405c1d0230 auth_method 0 2024-09-16T04:02:36.187 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.163+0000 7f405b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:02:36.187 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.163+0000 7f405b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:02:36.187 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.163+0000 7f405b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:02:36.187 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.163+0000 7f4060dd2640 10 monclient(hunting): get_auth_request con 0x7f405c1d0690 auth_method 0 2024-09-16T04:02:36.187 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.163+0000 7f4060dd2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:02:36.187 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.163+0000 7f4060dd2640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:02:36.187 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.163+0000 7f4060dd2640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:02:36.187 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.163+0000 7f405b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:02:36.188 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.163+0000 7f405b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:02:36.188 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.163+0000 7f405b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:02:36.188 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.163+0000 7f4060dd2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:02:36.188 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.163+0000 7f4060dd2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:02:36.188 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.163+0000 7f4060dd2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:02:36.188 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.163+0000 7f405bfff640 1 --2- 172.21.15.42:0/3389020659 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4044004810 0x7f405c1cfcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:02:36.188 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.163+0000 7f405bfff640 10 monclient(hunting): get_auth_request con 0x7f4044004810 auth_method 0 2024-09-16T04:02:36.188 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.163+0000 7f405bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:02:36.188 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.163+0000 7f405bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:02:36.188 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.163+0000 7f405bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:02:36.188 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.163+0000 7f4060dd2640 10 monclient(hunting): handle_auth_done global_id 825561 payload 824 2024-09-16T04:02:36.188 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.163+0000 7f4060dd2640 1 -- 172.21.15.42:0/3389020659 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f405c1d0230 msgr2=0x7f405c0a00d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:02:36.188 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.163+0000 7f4060dd2640 1 --2- 172.21.15.42:0/3389020659 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f405c1d0230 0x7f405c0a00d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:36.188 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.163+0000 7f4060dd2640 1 -- 172.21.15.42:0/3389020659 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4044004810 msgr2=0x7f405c1cfcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:02:36.188 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.163+0000 7f4060dd2640 1 --2- 172.21.15.42:0/3389020659 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4044004810 0x7f405c1cfcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:36.189 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.163+0000 7f4060dd2640 10 monclient: _finish_hunting 0 2024-09-16T04:02:36.189 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.163+0000 7f4060dd2640 1 monclient: found mon.noname-a 2024-09-16T04:02:36.189 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.163+0000 7f4060dd2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:02:36.189 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.163+0000 7f4060dd2640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:02:36.189 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.163+0000 7f4060dd2640 1 -- 172.21.15.42:0/3389020659 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f405c0a2800 con 0x7f405c1d0690 2024-09-16T04:02:36.189 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.163+0000 7f405b7fe640 1 --2- 172.21.15.42:0/3389020659 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f405c1d0230 0x7f405c0a00d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:02:36.189 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.163+0000 7f4060dd2640 1 --2- 172.21.15.42:0/3389020659 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f405c1d0690 0x7f405c0a0610 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f405c0a6280 tx=0x7f4054001cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:02:36.189 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.163+0000 7f4058ff9640 10 client.?.objecter ms_handle_connect 0x7f405c1d0690 2024-09-16T04:02:36.189 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.163+0000 7f4058ff9640 10 client.?.objecter resend_mon_ops 2024-09-16T04:02:36.189 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.163+0000 7f4058ff9640 1 -- 172.21.15.42:0/3389020659 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4054007800 con 0x7f405c1d0690 2024-09-16T04:02:36.189 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.163+0000 7f4058ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:02:36.189 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.163+0000 7f4058ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:02:36.189 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.163+0000 7f4058ff9640 10 monclient: dump: 2024-09-16T04:02:36.190 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:02:36.190 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:02:36.190 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:36.190 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:36.190 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:02:36.190 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:02:36.190 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:02:36.190 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:02:36.190 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:02:36.190 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:02:36.190 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.163+0000 7f4058ff9640 10 monclient: _finish_auth 0 2024-09-16T04:02:36.190 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.163+0000 7f4058ff9640 10 monclient: _check_auth_tickets 2024-09-16T04:02:36.190 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.163+0000 7f4058ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:02:36.190 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.163+0000 7f4058ff9640 1 -- 172.21.15.42:0/3389020659 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4054018070 con 0x7f405c1d0690 2024-09-16T04:02:36.191 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.163+0000 7f4058ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T04:02:36.191 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.163+0000 7f4058ff9640 1 -- 172.21.15.42:0/3389020659 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40540133f0 con 0x7f405c1d0690 2024-09-16T04:02:36.191 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.163+0000 7f4058ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:02:36.191 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.163+0000 7f4058ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:02:36.191 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.163+0000 7f4058ff9640 10 monclient: dump: 2024-09-16T04:02:36.191 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:02:36.191 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:02:36.191 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:36.191 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:36.191 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:02:36.191 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:02:36.191 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:02:36.191 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:02:36.191 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:02:36.192 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:02:36.192 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.163+0000 7f4062835640 5 monclient: authenticate success, global_id 825561 2024-09-16T04:02:36.192 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.163+0000 7f4062835640 10 monclient: _renew_subs 2024-09-16T04:02:36.192 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.163+0000 7f405a7fc640 4 set_mon_vals no callback set 2024-09-16T04:02:36.192 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.163+0000 7f4062835640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:02:36.192 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.163+0000 7f4062835640 1 -- 172.21.15.42:0/3389020659 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f405c06f680 con 0x7f405c1d0690 2024-09-16T04:02:36.192 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.163+0000 7f4062835640 10 client.825561.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:02:36.192 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.163+0000 7f4062835640 10 monclient: _renew_subs 2024-09-16T04:02:36.192 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.163+0000 7f4062835640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:02:36.192 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.163+0000 7f4062835640 1 -- 172.21.15.42:0/3389020659 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f405c120580 con 0x7f405c1d0690 2024-09-16T04:02:36.192 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.163+0000 7f4062835640 1 librados: init done 2024-09-16T04:02:36.192 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.163+0000 7f4062835640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:02:36.192 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.163+0000 7f4062835640 1 --2- 172.21.15.42:0/3389020659 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f405c0a0b50 0x7f405c0a0f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:36.192 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.163+0000 7f4062835640 10 monclient(hunting): start opening mon connection 2024-09-16T04:02:36.193 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.163+0000 7f4062835640 1 -- 172.21.15.42:0/3389020659 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f405c132fc0 con 0x7f405c0a0b50 2024-09-16T04:02:36.193 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.163+0000 7f405bfff640 1 --2- 172.21.15.42:0/3389020659 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f405c0a0b50 0x7f405c0a0f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:02:36.193 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.163+0000 7f405bfff640 10 monclient: get_auth_request con 0x7f405c0a0b50 auth_method 0 2024-09-16T04:02:36.193 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.163+0000 7f405bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:02:36.193 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.163+0000 7f405bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:02:36.193 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.163+0000 7f405bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:02:36.193 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.163+0000 7f405bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:02:36.193 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.163+0000 7f405bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:02:36.193 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.163+0000 7f405bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:02:36.193 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.163+0000 7f4058ff9640 1 -- 172.21.15.42:0/3389020659 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f4054013590 con 0x7f405c1d0690 2024-09-16T04:02:36.193 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.163+0000 7f405bfff640 10 monclient(hunting): handle_auth_done global_id 964139 payload 824 2024-09-16T04:02:36.193 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.163+0000 7f405bfff640 1 -- 172.21.15.42:0/3389020659 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f405c0a37b0 con 0x7f405c0a0b50 2024-09-16T04:02:36.193 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.163+0000 7f4058ff9640 1 --2- 172.21.15.42:0/3389020659 >> v2:172.21.15.154:6812/254845342 conn(0x7f4048037810 0x7f4048039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:36.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.163+0000 7f405bfff640 1 --2- 172.21.15.42:0/3389020659 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f405c0a0b50 0x7f405c0a0f30 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f404c007860 tx=0x7f404c0017b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:02:36.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.163+0000 7f4058ff9640 10 client.825561.objecter ms_handle_connect 0x7f405c0a0b50 2024-09-16T04:02:36.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.163+0000 7f4058ff9640 10 client.825561.objecter resend_mon_ops 2024-09-16T04:02:36.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.163+0000 7f4058ff9640 1 -- 172.21.15.42:0/3389020659 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(632..632 src has 451..632) ==== 6367+0+0 (secure 0 0 0) 0x7f4054049610 con 0x7f405c1d0690 2024-09-16T04:02:36.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.163+0000 7f4058ff9640 10 client.825561.objecter ms_dispatch 0x7f405c000ba0 osd_map(632..632 src has 451..632) 2024-09-16T04:02:36.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.163+0000 7f4058ff9640 3 client.825561.objecter handle_osd_map got epochs [632,632] > 0 2024-09-16T04:02:36.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.163+0000 7f4058ff9640 3 client.825561.objecter handle_osd_map decoding full epoch 632 2024-09-16T04:02:36.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.163+0000 7f4058ff9640 20 client.825561.objecter dump_active .. 0 homeless 2024-09-16T04:02:36.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.163+0000 7f4058ff9640 1 -- 172.21.15.42:0/3389020659 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f404c01a070 con 0x7f405c0a0b50 2024-09-16T04:02:36.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.163+0000 7f4058ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:02:36.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.163+0000 7f4058ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:02:36.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.163+0000 7f4058ff9640 10 monclient: dump: 2024-09-16T04:02:36.194 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:02:36.194 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:02:36.195 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:36.195 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:36.195 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:02:36.195 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:02:36.195 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:02:36.195 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:02:36.195 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:02:36.195 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:02:36.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.163+0000 7f4058ff9640 1 -- 172.21.15.42:0/3389020659 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f404c012590 con 0x7f405c0a0b50 2024-09-16T04:02:36.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.163+0000 7f4058ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:02:36.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.163+0000 7f4058ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:02:36.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.163+0000 7f4058ff9640 1 -- 172.21.15.42:0/3389020659 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f405c0a0b50 msgr2=0x7f405c0a0f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:02:36.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.163+0000 7f4058ff9640 1 --2- 172.21.15.42:0/3389020659 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f405c0a0b50 0x7f405c0a0f30 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f404c007860 tx=0x7f404c0017b0 comp rx=0 tx=0).stop 2024-09-16T04:02:36.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.163+0000 7f4058ff9640 1 -- 172.21.15.42:0/3389020659 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f405c0a0b50 msgr2=0x7f405c0a0f30 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:02:36.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.163+0000 7f4058ff9640 1 --2- 172.21.15.42:0/3389020659 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f405c0a0b50 0x7f405c0a0f30 secure :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f404c007860 tx=0x7f404c0017b0 comp rx=0 tx=0).stop 2024-09-16T04:02:36.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.163+0000 7f405b7fe640 1 --2- 172.21.15.42:0/3389020659 >> v2:172.21.15.154:6812/254845342 conn(0x7f4048037810 0x7f4048039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:02:36.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.163+0000 7f405b7fe640 10 monclient: get_auth_request con 0x7f4048037810 auth_method 0 2024-09-16T04:02:36.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.167+0000 7f4058ff9640 10 client.825561.objecter ms_handle_connect 0x7f4048037810 2024-09-16T04:02:36.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.167+0000 7f405b7fe640 1 --2- 172.21.15.42:0/3389020659 >> v2:172.21.15.154:6812/254845342 conn(0x7f4048037810 0x7f4048039cd0 secure :-1 s=READY pgs=3789 cs=0 l=1 rev1=1 crypto rx=0x7f405001ca50 tx=0x7f40500b3040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:02:36.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.171+0000 7f4062835640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:02:36.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.171+0000 7f4062835640 1 --2- 172.21.15.42:0/3389020659 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f405c09c940 0x7f405c09cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:36.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.171+0000 7f4062835640 10 monclient(hunting): start opening mon connection 2024-09-16T04:02:36.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.171+0000 7f4062835640 1 -- 172.21.15.42:0/3389020659 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f405c0a1470 con 0x7f405c09c940 2024-09-16T04:02:36.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.171+0000 7f405bfff640 1 --2- 172.21.15.42:0/3389020659 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f405c09c940 0x7f405c09cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:02:36.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.171+0000 7f405bfff640 10 monclient: get_auth_request con 0x7f405c09c940 auth_method 0 2024-09-16T04:02:36.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.171+0000 7f405bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:02:36.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.171+0000 7f405bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:02:36.197 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.171+0000 7f405bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:02:36.197 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.171+0000 7f405bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:02:36.197 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.171+0000 7f405bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:02:36.197 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.171+0000 7f405bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:02:36.197 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.171+0000 7f405bfff640 10 monclient(hunting): handle_auth_done global_id 964142 payload 824 2024-09-16T04:02:36.197 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.171+0000 7f405bfff640 1 -- 172.21.15.42:0/3389020659 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f405c13ab30 con 0x7f405c09c940 2024-09-16T04:02:36.197 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.171+0000 7f405bfff640 1 --2- 172.21.15.42:0/3389020659 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f405c09c940 0x7f405c09cd20 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f404c007860 tx=0x7f404c0077f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:02:36.197 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.171+0000 7f4058ff9640 10 client.825561.objecter ms_handle_connect 0x7f405c09c940 2024-09-16T04:02:36.197 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.171+0000 7f4058ff9640 10 client.825561.objecter resend_mon_ops 2024-09-16T04:02:36.197 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.171+0000 7f4058ff9640 1 -- 172.21.15.42:0/3389020659 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f404c01a040 con 0x7f405c09c940 2024-09-16T04:02:36.197 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.171+0000 7f4058ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:02:36.197 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.171+0000 7f4058ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:02:36.197 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.171+0000 7f4058ff9640 10 monclient: dump: 2024-09-16T04:02:36.198 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:02:36.198 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:02:36.198 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:36.198 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:36.198 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:02:36.198 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:02:36.198 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:02:36.198 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:02:36.198 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:02:36.198 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:02:36.198 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.171+0000 7f4058ff9640 1 -- 172.21.15.42:0/3389020659 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4544 (secure 0 0 0) 0x7f404c015460 con 0x7f405c09c940 2024-09-16T04:02:36.198 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.171+0000 7f4058ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:02:36.198 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.171+0000 7f4058ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:02:36.198 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.171+0000 7f4058ff9640 1 -- 172.21.15.42:0/3389020659 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f405c09c940 msgr2=0x7f405c09cd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:02:36.199 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.171+0000 7f4058ff9640 1 --2- 172.21.15.42:0/3389020659 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f405c09c940 0x7f405c09cd20 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f404c007860 tx=0x7f404c0077f0 comp rx=0 tx=0).stop 2024-09-16T04:02:36.199 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.171+0000 7f4058ff9640 1 -- 172.21.15.42:0/3389020659 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f405c09c940 msgr2=0x7f405c09cd20 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:02:36.199 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.171+0000 7f4058ff9640 1 --2- 172.21.15.42:0/3389020659 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f405c09c940 0x7f405c09cd20 secure :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f404c007860 tx=0x7f404c0077f0 comp rx=0 tx=0).stop 2024-09-16T04:02:36.199 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T04:02:36.199 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:02:36.199 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:02:36.199 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T04:02:36.199 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 184, 2024-09-16T04:02:36.199 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 9907, 2024-09-16T04:02:36.200 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T04:02:36.200 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T04:02:36.200 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T04:02:36.200 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T04:02:36.200 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:02:36.200 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 4, 2024-09-16T04:02:36.200 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:02:36.200 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T04:02:36.200 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T04:02:36.200 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:02:36.200 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:02:36.200 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:02:36.200 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:02:36.201 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:02:36.201 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:02:36.201 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:02:36.201 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:02:36.201 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:02:36.201 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:02:36.201 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:02:36.201 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:02:36.201 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:02:36.201 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T04:02:36.201 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:02:36.201 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:02:36.201 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:02:36.201 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:02:36.202 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:02:36.202 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:02:36.202 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:02:36.202 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:02:36.202 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:02:36.202 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:02:36.202 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:02:36.202 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:02:36.202 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:02:36.202 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T04:02:36.202 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T04:02:36.202 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T04:02:36.202 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T04:02:36.203 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T04:02:36.203 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T04:02:36.203 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:02:36.203 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:02:36.203 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T04:02:36.203 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T04:02:36.203 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T04:02:36.203 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T04:02:36.203 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T04:02:36.203 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T04:02:36.203 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T04:02:36.203 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:02:36.203 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T04:02:36.203 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:02:36.204 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:02:36.204 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:02:36.204 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:02:36.204 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:02:36.204 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:02:36.204 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:02:36.204 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:02:36.204 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:02:36.204 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:02:36.204 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:02:36.205 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:02:36.205 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T04:02:36.205 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:02:36.205 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T04:02:36.205 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:02:36.205 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:02:36.205 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:02:36.205 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:02:36.205 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:02:36.205 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:02:36.205 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:02:36.205 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T04:02:36.205 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:02:36.205 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:02:36.206 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:02:36.206 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:02:36.206 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T04:02:36.206 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:02:36.206 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:02:36.206 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:02:36.206 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:02:36.206 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T04:02:36.206 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T04:02:36.206 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:02:36.206 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:02:36.206 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:02:36.206 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:02:36.206 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:02:36.207 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:02:36.207 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:02:36.207 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:02:36.207 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:02:36.207 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:02:36.207 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:02:36.207 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T04:02:36.207 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:02:36.207 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:02:36.207 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:02:36.207 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:02:36.207 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T04:02:36.207 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:02:36.207 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:02:36.208 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:02:36.208 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:02:36.208 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T04:02:36.208 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T04:02:36.208 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:02:36.208 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:02:36.208 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:02:36.208 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:02:36.208 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:02:36.208 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:02:36.208 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:02:36.208 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:02:36.208 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:02:36.209 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:02:36.209 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:02:36.209 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T04:02:36.209 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:02:36.209 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:02:36.209 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:02:36.209 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:02:36.209 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T04:02:36.209 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:02:36.209 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:02:36.209 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:02:36.209 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:02:36.209 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T04:02:36.209 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T04:02:36.209 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:02:36.210 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:02:36.210 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:02:36.210 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:02:36.210 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:02:36.210 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:02:36.210 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T04:02:36.210 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T04:02:36.210 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:02:36.210 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:02:36.210 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:02:36.210 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:02:36.210 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:02:36.210 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:02:36.211 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T04:02:36.211 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:02:36.211 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:02:36.211 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:02:36.211 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:02:36.211 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:02:36.211 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:02:36.211 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:02:36.211 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T04:02:36.211 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T04:02:36.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.171+0000 7f4062835640 10 librados: watch_flush enter 2024-09-16T04:02:36.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.171+0000 7f4062835640 10 librados: watch_flush exit 2024-09-16T04:02:36.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.171+0000 7f4062835640 20 client.825561.objecter shutdown clearing up homeless session... 2024-09-16T04:02:36.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.171+0000 7f4062835640 10 client.825561.objecter successfully canceled tick 2024-09-16T04:02:36.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.171+0000 7f4062835640 1 -- 172.21.15.42:0/3389020659 >> v2:172.21.15.154:6812/254845342 conn(0x7f4048037810 msgr2=0x7f4048039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:02:36.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.171+0000 7f4062835640 1 --2- 172.21.15.42:0/3389020659 >> v2:172.21.15.154:6812/254845342 conn(0x7f4048037810 0x7f4048039cd0 secure :-1 s=READY pgs=3789 cs=0 l=1 rev1=1 crypto rx=0x7f405001ca50 tx=0x7f40500b3040 comp rx=0 tx=0).stop 2024-09-16T04:02:36.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.171+0000 7f4062835640 10 monclient: shutdown 2024-09-16T04:02:36.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.171+0000 7f4062835640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:02:36.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.171+0000 7f4062835640 1 -- 172.21.15.42:0/3389020659 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f405c1d0690 msgr2=0x7f405c0a0610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:02:36.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.171+0000 7f4062835640 1 --2- 172.21.15.42:0/3389020659 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f405c1d0690 0x7f405c0a0610 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f405c0a6280 tx=0x7f4054001cb0 comp rx=0 tx=0).stop 2024-09-16T04:02:36.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.171+0000 7f4060dd2640 1 -- 172.21.15.42:0/3389020659 reap_dead start 2024-09-16T04:02:36.214 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.171+0000 7f4060dd2640 1 -- 172.21.15.42:0/3389020659 reap_dead start 2024-09-16T04:02:36.214 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.171+0000 7f4062835640 1 -- 172.21.15.42:0/3389020659 shutdown_connections 2024-09-16T04:02:36.214 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.171+0000 7f4062835640 1 -- 172.21.15.42:0/3389020659 >> 172.21.15.42:0/3389020659 conn(0x7f405c0a9fb0 msgr2=0x7f405c0a9030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:02:36.214 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.171+0000 7f4062835640 1 -- 172.21.15.42:0/3389020659 shutdown_connections 2024-09-16T04:02:36.214 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.171+0000 7f4062835640 1 -- 172.21.15.42:0/3389020659 wait complete. 2024-09-16T04:02:36.214 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.171+0000 7f4062835640 1 librados: shutdown 2024-09-16T04:02:36.214 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:02:36.214 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T04:02:36.215 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T04:02:36.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.327+0000 7fc113b93640 10 monclient: get_monmap_and_config 2024-09-16T04:02:36.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.327+0000 7fc113b93640 10 monclient: build_initial_monmap 2024-09-16T04:02:36.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.327+0000 7fc113b93640 10 monclient: monmap: 2024-09-16T04:02:36.330 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:02:36.330 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:02:36.330 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:02:36.328431+0000 2024-09-16T04:02:36.330 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:02:36.328431+0000 2024-09-16T04:02:36.330 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:02:36.330 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:02:36.330 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:02:36.331 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:02:36.331 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:02:36.331 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:02:36.331 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:02:36.331 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:02:36.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.327+0000 7fc113b93640 1 Processor -- start 2024-09-16T04:02:36.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.327+0000 7fc113b93640 1 -- start start 2024-09-16T04:02:36.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.327+0000 7fc113b93640 10 monclient: init 2024-09-16T04:02:36.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.327+0000 7fc113b93640 10 monclient: _reopen_session rank -1 2024-09-16T04:02:36.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.327+0000 7fc113b93640 10 monclient: _add_conns ranks=[4,2,0,3,1] 2024-09-16T04:02:36.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.327+0000 7fc113b93640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fc10c09ca60 0x7fc10c09d460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:36.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.327+0000 7fc113b93640 10 monclient(hunting): picked mon.noname-c con 0x7fc10c09ca60 addr v2:172.21.15.154:3300/0 2024-09-16T04:02:36.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.327+0000 7fc113b93640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc10c09f260 0x7fc10c09d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:36.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.327+0000 7fc113b93640 10 monclient(hunting): picked mon.noname-b con 0x7fc10c09f260 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:02:36.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.327+0000 7fc113b93640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc10c09dee0 addr v1:172.21.15.42:6789/0 2024-09-16T04:02:36.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.327+0000 7fc113b93640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc10c0a0050 con 0x7fc10c09dee0 2024-09-16T04:02:36.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.327+0000 7fc113b93640 10 monclient(hunting): start opening mon connection 2024-09-16T04:02:36.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.327+0000 7fc113b93640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fc10c09ebf0 con 0x7fc10c09ca60 2024-09-16T04:02:36.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.327+0000 7fc113b93640 10 monclient(hunting): start opening mon connection 2024-09-16T04:02:36.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.327+0000 7fc113b93640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fc10c09ce40 con 0x7fc10c09f260 2024-09-16T04:02:36.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.327+0000 7fc113b93640 10 monclient(hunting): _renew_subs 2024-09-16T04:02:36.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.327+0000 7fc113b93640 10 monclient(hunting): authenticate will time out at 2203.298758s 2024-09-16T04:02:36.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.327+0000 7fc11112e640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc10c09f260 0x7fc10c09d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:02:36.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.327+0000 7fc112130640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fc10c09dee0 0x7fc10c13a760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:36458/0 (socket says 172.21.15.42:36458) 2024-09-16T04:02:36.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.327+0000 7fc11112e640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc10c09f260 0x7fc10c09d9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:33956/0 (socket says 172.21.15.42:33956) 2024-09-16T04:02:36.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.327+0000 7fc112130640 1 -- 172.21.15.42:0/2463818749 learned_addr learned my addr 172.21.15.42:0/2463818749 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T04:02:36.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.327+0000 7fc11112e640 10 monclient(hunting): get_auth_request con 0x7fc10c09f260 auth_method 0 2024-09-16T04:02:36.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.327+0000 7fc11112e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:02:36.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.327+0000 7fc11112e640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:02:36.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.327+0000 7fc11112e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:02:36.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.327+0000 7fc11112e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:02:36.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.327+0000 7fc11112e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:02:36.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.327+0000 7fc11112e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:02:36.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.327+0000 7fc11192f640 1 --2- 172.21.15.42:0/2463818749 >> v2:172.21.15.154:3300/0 conn(0x7fc10c09ca60 0x7fc10c09d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:02:36.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.327+0000 7fc11192f640 10 monclient(hunting): get_auth_request con 0x7fc10c09ca60 auth_method 0 2024-09-16T04:02:36.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.327+0000 7fc11192f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:02:36.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.327+0000 7fc11192f640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:02:36.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.327+0000 7fc11192f640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:02:36.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.327+0000 7fc11112e640 10 monclient(hunting): handle_auth_done global_id 964145 payload 274 2024-09-16T04:02:36.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.327+0000 7fc11112e640 1 -- 172.21.15.42:0/2463818749 >> v2:172.21.15.154:3300/0 conn(0x7fc10c09ca60 msgr2=0x7fc10c09d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:02:36.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.327+0000 7fc11112e640 1 --2- 172.21.15.42:0/2463818749 >> v2:172.21.15.154:3300/0 conn(0x7fc10c09ca60 0x7fc10c09d460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:36.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.327+0000 7fc11112e640 1 -- 172.21.15.42:0/2463818749 >> v1:172.21.15.42:6789/0 conn(0x7fc10c09dee0 legacy=0x7fc10c13a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:02:36.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.327+0000 7fc11112e640 10 monclient: _finish_hunting 0 2024-09-16T04:02:36.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.327+0000 7fc11112e640 1 monclient: found mon.noname-b 2024-09-16T04:02:36.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.327+0000 7fc11112e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:02:36.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.327+0000 7fc11112e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T04:02:36.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.327+0000 7fc11112e640 1 -- 172.21.15.42:0/2463818749 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc10c09e5a0 con 0x7fc10c09f260 2024-09-16T04:02:36.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.327+0000 7fc11092d640 1 -- 172.21.15.42:0/2463818749 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2163134318 0 0) 0x7fc10c0a0050 con 0x7fc10c09dee0 2024-09-16T04:02:36.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.327+0000 7fc11092d640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T04:02:36.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.331+0000 7fc11112e640 1 --2- 172.21.15.42:0/2463818749 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc10c09f260 0x7fc10c09d9a0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fc0fc0097f0 tx=0x7fc0fc01c360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b490fdd3e3cfa1b0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:02:36.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.331+0000 7fc11092d640 1 -- 172.21.15.42:0/2463818749 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0fc0b0070 con 0x7fc10c09f260 2024-09-16T04:02:36.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.331+0000 7fc11092d640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:02:36.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.331+0000 7fc11092d640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:02:36.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.331+0000 7fc11092d640 10 monclient: dump: 2024-09-16T04:02:36.343 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:02:36.343 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:02:36.343 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:36.343 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:36.343 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:02:36.343 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:02:36.344 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:02:36.344 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:02:36.344 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:02:36.344 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:02:36.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.331+0000 7fc11092d640 10 monclient: _finish_auth 0 2024-09-16T04:02:36.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.331+0000 7fc11092d640 10 monclient: _check_auth_tickets 2024-09-16T04:02:36.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.331+0000 7fc11092d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:02:36.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.331+0000 7fc11092d640 1 -- 172.21.15.42:0/2463818749 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc0fc0ab600 con 0x7fc10c09f260 2024-09-16T04:02:36.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.331+0000 7fc11092d640 10 monclient: handle_config config(0 keys) 2024-09-16T04:02:36.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.331+0000 7fc11092d640 1 -- 172.21.15.42:0/2463818749 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0fc0ab9d0 con 0x7fc10c09f260 2024-09-16T04:02:36.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.331+0000 7fc113b93640 5 monclient: authenticate success, global_id 964145 2024-09-16T04:02:36.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.331+0000 7fc113b93640 10 monclient: get_monmap_and_config success 2024-09-16T04:02:36.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.331+0000 7fc113b93640 4 set_mon_vals no callback set 2024-09-16T04:02:36.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.331+0000 7fc11092d640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:02:36.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.331+0000 7fc11092d640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:02:36.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.331+0000 7fc11092d640 10 monclient: dump: 2024-09-16T04:02:36.345 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:02:36.345 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:02:36.345 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:36.345 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:36.345 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:02:36.345 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:02:36.345 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:02:36.345 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:02:36.345 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:02:36.345 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:02:36.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.331+0000 7fc113b93640 10 monclient: shutdown 2024-09-16T04:02:36.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.331+0000 7fc113b93640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:02:36.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.331+0000 7fc113b93640 1 -- 172.21.15.42:0/2463818749 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc10c09f260 msgr2=0x7fc10c09d9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:02:36.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.331+0000 7fc113b93640 1 --2- 172.21.15.42:0/2463818749 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc10c09f260 0x7fc10c09d9a0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fc0fc0097f0 tx=0x7fc0fc01c360 comp rx=0 tx=0).stop 2024-09-16T04:02:36.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.331+0000 7fc113b93640 1 -- 172.21.15.42:0/2463818749 shutdown_connections 2024-09-16T04:02:36.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.331+0000 7fc113b93640 1 --2- 172.21.15.42:0/2463818749 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc10c09f260 0x7fc10c09d9a0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:36.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.331+0000 7fc113b93640 1 --2- 172.21.15.42:0/2463818749 >> v2:172.21.15.154:3300/0 conn(0x7fc10c09ca60 0x7fc10c09d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:36.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.331+0000 7fc113b93640 1 -- 172.21.15.42:0/2463818749 >> 172.21.15.42:0/2463818749 conn(0x7fc10c06bc40 msgr2=0x7fc10c06c050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:02:36.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.331+0000 7fc113b93640 1 -- 172.21.15.42:0/2463818749 shutdown_connections 2024-09-16T04:02:36.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.331+0000 7fc113b93640 1 -- 172.21.15.42:0/2463818749 wait complete. 2024-09-16T04:02:36.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.331+0000 7fc113b93640 10 monclient: build_initial_monmap 2024-09-16T04:02:36.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.331+0000 7fc113b93640 10 monclient: monmap: 2024-09-16T04:02:36.346 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:02:36.346 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:02:36.347 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:02:36.347 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:02:36.347 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:02:36.347 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:02:36.347 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:02:36.347 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:02:36.347 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:02:36.347 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:02:36.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.331+0000 7fc113b93640 1 librados: starting msgr at 2024-09-16T04:02:36.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.331+0000 7fc113b93640 1 librados: starting objecter 2024-09-16T04:02:36.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.331+0000 7fc113b93640 1 Processor -- start 2024-09-16T04:02:36.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.331+0000 7fc113b93640 1 -- start start 2024-09-16T04:02:36.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.331+0000 7fc113b93640 1 librados: setting wanted keys 2024-09-16T04:02:36.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.331+0000 7fc113b93640 1 librados: calling monclient init 2024-09-16T04:02:36.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.331+0000 7fc113b93640 10 monclient: init 2024-09-16T04:02:36.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.331+0000 7fc113b93640 10 monclient: _reopen_session rank -1 2024-09-16T04:02:36.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.331+0000 7fc113b93640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:02:36.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.331+0000 7fc113b93640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc10c09ca60 0x7fc10c1cca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:36.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.331+0000 7fc113b93640 10 monclient(hunting): picked mon.noname-b con 0x7fc10c09ca60 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:02:36.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.331+0000 7fc113b93640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc10c09dee0 0x7fc10c13b350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:36.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.331+0000 7fc113b93640 10 monclient(hunting): picked mon.noname-c con 0x7fc10c09dee0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:02:36.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.331+0000 7fc113b93640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc10c09f260 0x7fc10c13b890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:36.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.331+0000 7fc113b93640 10 monclient(hunting): picked mon.noname-a con 0x7fc10c09f260 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:02:36.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.331+0000 7fc113b93640 10 monclient(hunting): start opening mon connection 2024-09-16T04:02:36.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.331+0000 7fc113b93640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fc10c0a0ae0 con 0x7fc10c09f260 2024-09-16T04:02:36.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.331+0000 7fc113b93640 10 monclient(hunting): start opening mon connection 2024-09-16T04:02:36.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.331+0000 7fc113b93640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc10c0a0960 con 0x7fc10c09ca60 2024-09-16T04:02:36.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.331+0000 7fc113b93640 10 monclient(hunting): start opening mon connection 2024-09-16T04:02:36.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.331+0000 7fc113b93640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fc10c0a0c60 con 0x7fc10c09dee0 2024-09-16T04:02:36.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.331+0000 7fc113b93640 10 monclient(hunting): _renew_subs 2024-09-16T04:02:36.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.331+0000 7fc113b93640 10 monclient(hunting): authenticate will time out at 2203.303967s 2024-09-16T04:02:36.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.335+0000 7fc112130640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc10c09f260 0x7fc10c13b890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:02:36.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.335+0000 7fc11112e640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc10c09dee0 0x7fc10c13b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:02:36.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.335+0000 7fc11112e640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc10c09dee0 0x7fc10c13b350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:33958/0 (socket says 172.21.15.42:33958) 2024-09-16T04:02:36.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.335+0000 7fc112130640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc10c09f260 0x7fc10c13b890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:53794/0 (socket says 172.21.15.42:53794) 2024-09-16T04:02:36.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.335+0000 7fc11112e640 1 -- 172.21.15.42:0/2388550465 learned_addr learned my addr 172.21.15.42:0/2388550465 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:02:36.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.335+0000 7fc11112e640 10 monclient(hunting): get_auth_request con 0x7fc10c09dee0 auth_method 0 2024-09-16T04:02:36.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.335+0000 7fc11112e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:02:36.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.335+0000 7fc11112e640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:02:36.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.335+0000 7fc11112e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:02:36.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.335+0000 7fc112130640 10 monclient(hunting): get_auth_request con 0x7fc10c09f260 auth_method 0 2024-09-16T04:02:36.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.335+0000 7fc112130640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:02:36.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.335+0000 7fc112130640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:02:36.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.335+0000 7fc112130640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:02:36.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.335+0000 7fc11112e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:02:36.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.335+0000 7fc11112e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:02:36.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.335+0000 7fc11112e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:02:36.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.335+0000 7fc112130640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:02:36.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.335+0000 7fc112130640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:02:36.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.335+0000 7fc112130640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:02:36.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.335+0000 7fc11192f640 1 --2- 172.21.15.42:0/2388550465 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc10c09ca60 0x7fc10c1cca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:02:36.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.335+0000 7fc11192f640 10 monclient(hunting): get_auth_request con 0x7fc10c09ca60 auth_method 0 2024-09-16T04:02:36.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.335+0000 7fc11192f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:02:36.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.335+0000 7fc11192f640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:02:36.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.335+0000 7fc11192f640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:02:36.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.335+0000 7fc112130640 10 monclient(hunting): handle_auth_done global_id 825567 payload 824 2024-09-16T04:02:36.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.335+0000 7fc112130640 1 -- 172.21.15.42:0/2388550465 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc10c09dee0 msgr2=0x7fc10c13b350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:02:36.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.335+0000 7fc112130640 1 --2- 172.21.15.42:0/2388550465 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc10c09dee0 0x7fc10c13b350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:36.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.335+0000 7fc112130640 1 -- 172.21.15.42:0/2388550465 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc10c09ca60 msgr2=0x7fc10c1cca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:02:36.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.335+0000 7fc112130640 1 --2- 172.21.15.42:0/2388550465 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc10c09ca60 0x7fc10c1cca90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:36.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.335+0000 7fc112130640 10 monclient: _finish_hunting 0 2024-09-16T04:02:36.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.335+0000 7fc112130640 1 monclient: found mon.noname-a 2024-09-16T04:02:36.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.335+0000 7fc112130640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:02:36.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.335+0000 7fc112130640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:02:36.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.335+0000 7fc112130640 1 -- 172.21.15.42:0/2388550465 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc10c1229e0 con 0x7fc10c09f260 2024-09-16T04:02:36.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.335+0000 7fc11112e640 1 --2- 172.21.15.42:0/2388550465 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc10c09dee0 0x7fc10c13b350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:02:36.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.335+0000 7fc112130640 1 --2- 172.21.15.42:0/2388550465 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc10c09f260 0x7fc10c13b890 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fc10400c710 tx=0x7fc10400cbd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:02:36.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.335+0000 7fc1027fc640 10 client.?.objecter ms_handle_connect 0x7fc10c09f260 2024-09-16T04:02:36.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.335+0000 7fc1027fc640 10 client.?.objecter resend_mon_ops 2024-09-16T04:02:36.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.335+0000 7fc1027fc640 1 -- 172.21.15.42:0/2388550465 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc104018070 con 0x7fc10c09f260 2024-09-16T04:02:36.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.335+0000 7fc1027fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:02:36.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.335+0000 7fc1027fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:02:36.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.335+0000 7fc1027fc640 10 monclient: dump: 2024-09-16T04:02:36.353 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:02:36.353 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:02:36.353 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:36.353 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:36.353 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:02:36.353 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:02:36.353 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:02:36.353 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:02:36.353 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:02:36.353 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:02:36.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.335+0000 7fc1027fc640 10 monclient: _finish_auth 0 2024-09-16T04:02:36.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.335+0000 7fc1027fc640 10 monclient: _check_auth_tickets 2024-09-16T04:02:36.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.335+0000 7fc1027fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:02:36.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.335+0000 7fc1027fc640 1 -- 172.21.15.42:0/2388550465 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc10400ce20 con 0x7fc10c09f260 2024-09-16T04:02:36.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.335+0000 7fc1027fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:02:36.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.335+0000 7fc1027fc640 1 -- 172.21.15.42:0/2388550465 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc10400a880 con 0x7fc10c09f260 2024-09-16T04:02:36.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.335+0000 7fc1027fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:02:36.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.335+0000 7fc1027fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:02:36.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.335+0000 7fc1027fc640 10 monclient: dump: 2024-09-16T04:02:36.354 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:02:36.354 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:02:36.354 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:36.354 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:36.354 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:02:36.354 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:02:36.354 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:02:36.354 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:02:36.355 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:02:36.355 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:02:36.355 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.335+0000 7fc103fff640 4 set_mon_vals no callback set 2024-09-16T04:02:36.355 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.335+0000 7fc113b93640 5 monclient: authenticate success, global_id 825567 2024-09-16T04:02:36.355 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.335+0000 7fc113b93640 10 monclient: _renew_subs 2024-09-16T04:02:36.355 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.335+0000 7fc113b93640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:02:36.355 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.335+0000 7fc113b93640 1 -- 172.21.15.42:0/2388550465 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc10c0a5620 con 0x7fc10c09f260 2024-09-16T04:02:36.355 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.335+0000 7fc113b93640 10 client.825567.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:02:36.355 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.335+0000 7fc113b93640 10 monclient: _renew_subs 2024-09-16T04:02:36.355 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.335+0000 7fc113b93640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:02:36.355 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.335+0000 7fc113b93640 1 -- 172.21.15.42:0/2388550465 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc10c123800 con 0x7fc10c09f260 2024-09-16T04:02:36.355 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.335+0000 7fc113b93640 1 librados: init done 2024-09-16T04:02:36.355 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.335+0000 7fc113b93640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:02:36.355 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.335+0000 7fc113b93640 1 --2- 172.21.15.42:0/2388550465 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc10c120e20 0x7fc10c121200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:36.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.335+0000 7fc113b93640 10 monclient(hunting): start opening mon connection 2024-09-16T04:02:36.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.335+0000 7fc113b93640 1 -- 172.21.15.42:0/2388550465 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc10c0a1040 con 0x7fc10c120e20 2024-09-16T04:02:36.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.335+0000 7fc1027fc640 1 -- 172.21.15.42:0/2388550465 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fc10400aa20 con 0x7fc10c09f260 2024-09-16T04:02:36.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.335+0000 7fc11192f640 1 --2- 172.21.15.42:0/2388550465 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc10c120e20 0x7fc10c121200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:02:36.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.335+0000 7fc1027fc640 1 --2- 172.21.15.42:0/2388550465 >> v2:172.21.15.154:6812/254845342 conn(0x7fc0f0037840 0x7fc0f0039d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:36.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.335+0000 7fc11192f640 10 monclient: get_auth_request con 0x7fc10c120e20 auth_method 0 2024-09-16T04:02:36.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.335+0000 7fc11192f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:02:36.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.335+0000 7fc11192f640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:02:36.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.335+0000 7fc11192f640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:02:36.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.335+0000 7fc1027fc640 1 -- 172.21.15.42:0/2388550465 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(632..632 src has 451..632) ==== 6367+0+0 (secure 0 0 0) 0x7fc104048710 con 0x7fc10c09f260 2024-09-16T04:02:36.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.335+0000 7fc1027fc640 10 client.825567.objecter ms_dispatch 0x7fc10c000ba0 osd_map(632..632 src has 451..632) 2024-09-16T04:02:36.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.335+0000 7fc1027fc640 3 client.825567.objecter handle_osd_map got epochs [632,632] > 0 2024-09-16T04:02:36.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.335+0000 7fc1027fc640 3 client.825567.objecter handle_osd_map decoding full epoch 632 2024-09-16T04:02:36.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.335+0000 7fc1027fc640 20 client.825567.objecter dump_active .. 0 homeless 2024-09-16T04:02:36.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.335+0000 7fc11192f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:02:36.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.335+0000 7fc11192f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:02:36.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.335+0000 7fc11192f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:02:36.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.335+0000 7fc11112e640 1 --2- 172.21.15.42:0/2388550465 >> v2:172.21.15.154:6812/254845342 conn(0x7fc0f0037840 0x7fc0f0039d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:02:36.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.335+0000 7fc11112e640 10 monclient: get_auth_request con 0x7fc0f0037840 auth_method 0 2024-09-16T04:02:36.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.335+0000 7fc11192f640 10 monclient(hunting): handle_auth_done global_id 904639 payload 824 2024-09-16T04:02:36.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.335+0000 7fc11192f640 1 -- 172.21.15.42:0/2388550465 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc10c0008e0 con 0x7fc10c120e20 2024-09-16T04:02:36.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.335+0000 7fc11192f640 1 --2- 172.21.15.42:0/2388550465 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc10c120e20 0x7fc10c121200 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fc10800f4b0 tx=0x7fc10800f970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:02:36.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.335+0000 7fc1027fc640 10 client.825567.objecter ms_handle_connect 0x7fc10c120e20 2024-09-16T04:02:36.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.335+0000 7fc1027fc640 10 client.825567.objecter resend_mon_ops 2024-09-16T04:02:36.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.335+0000 7fc1027fc640 1 -- 172.21.15.42:0/2388550465 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc108019020 con 0x7fc10c120e20 2024-09-16T04:02:36.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.335+0000 7fc1027fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:02:36.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.339+0000 7fc1027fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:02:36.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.339+0000 7fc1027fc640 10 monclient: dump: 2024-09-16T04:02:36.358 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:02:36.358 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:02:36.358 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:36.358 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:36.358 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:02:36.358 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:02:36.358 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:02:36.358 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:02:36.358 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:02:36.358 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:02:36.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.339+0000 7fc1027fc640 10 client.825567.objecter ms_handle_connect 0x7fc0f0037840 2024-09-16T04:02:36.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.339+0000 7fc11112e640 1 --2- 172.21.15.42:0/2388550465 >> v2:172.21.15.154:6812/254845342 conn(0x7fc0f0037840 0x7fc0f0039d00 secure :-1 s=READY pgs=3790 cs=0 l=1 rev1=1 crypto rx=0x7fc0fc0097f0 tx=0x7fc0fc008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:02:36.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.339+0000 7fc1027fc640 1 -- 172.21.15.42:0/2388550465 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fc108014870 con 0x7fc10c120e20 2024-09-16T04:02:36.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.339+0000 7fc1027fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:02:36.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.339+0000 7fc1027fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:02:36.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.339+0000 7fc1027fc640 1 -- 172.21.15.42:0/2388550465 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc10c120e20 msgr2=0x7fc10c121200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:02:36.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.339+0000 7fc1027fc640 1 --2- 172.21.15.42:0/2388550465 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc10c120e20 0x7fc10c121200 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fc10800f4b0 tx=0x7fc10800f970 comp rx=0 tx=0).stop 2024-09-16T04:02:36.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.339+0000 7fc1027fc640 1 -- 172.21.15.42:0/2388550465 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc10c120e20 msgr2=0x7fc10c121200 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:02:36.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.339+0000 7fc1027fc640 1 --2- 172.21.15.42:0/2388550465 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc10c120e20 0x7fc10c121200 secure :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fc10800f4b0 tx=0x7fc10800f970 comp rx=0 tx=0).stop 2024-09-16T04:02:36.361 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.343+0000 7fc113b93640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:02:36.361 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.343+0000 7fc113b93640 1 --2- 172.21.15.42:0/2388550465 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc10c121740 0x7fc10c11c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:36.361 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.343+0000 7fc113b93640 10 monclient(hunting): start opening mon connection 2024-09-16T04:02:36.361 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.343+0000 7fc113b93640 1 -- 172.21.15.42:0/2388550465 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc10c121b50 con 0x7fc10c121740 2024-09-16T04:02:36.361 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.343+0000 7fc11192f640 1 --2- 172.21.15.42:0/2388550465 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc10c121740 0x7fc10c11c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:02:36.361 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.343+0000 7fc11192f640 10 monclient: get_auth_request con 0x7fc10c121740 auth_method 0 2024-09-16T04:02:36.361 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.343+0000 7fc11192f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:02:36.361 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.343+0000 7fc11192f640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:02:36.361 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.343+0000 7fc11192f640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:02:36.361 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.343+0000 7fc11192f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:02:36.361 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.343+0000 7fc11192f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:02:36.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.343+0000 7fc11192f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:02:36.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.347+0000 7fc11192f640 10 monclient(hunting): handle_auth_done global_id 904642 payload 824 2024-09-16T04:02:36.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.347+0000 7fc11192f640 1 -- 172.21.15.42:0/2388550465 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fc10c1ccfd0 con 0x7fc10c121740 2024-09-16T04:02:36.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.347+0000 7fc11192f640 1 --2- 172.21.15.42:0/2388550465 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc10c121740 0x7fc10c11c5b0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fc10800f4b0 tx=0x7fc1080046d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:02:36.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.347+0000 7fc1027fc640 10 client.825567.objecter ms_handle_connect 0x7fc10c121740 2024-09-16T04:02:36.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.347+0000 7fc1027fc640 10 client.825567.objecter resend_mon_ops 2024-09-16T04:02:36.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.347+0000 7fc1027fc640 1 -- 172.21.15.42:0/2388550465 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc108019030 con 0x7fc10c121740 2024-09-16T04:02:36.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.347+0000 7fc1027fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:02:36.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.347+0000 7fc1027fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:02:36.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.347+0000 7fc1027fc640 10 monclient: dump: 2024-09-16T04:02:36.362 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:02:36.362 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:02:36.362 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:36.363 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:36.363 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:02:36.363 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:02:36.363 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:02:36.363 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:02:36.363 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:02:36.363 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:02:36.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.347+0000 7fc1027fc640 1 -- 172.21.15.42:0/2388550465 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4880 (secure 0 0 0) 0x7fc108004250 con 0x7fc10c121740 2024-09-16T04:02:36.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.347+0000 7fc1027fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:02:36.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.347+0000 7fc1027fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:02:36.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.347+0000 7fc1027fc640 1 -- 172.21.15.42:0/2388550465 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc10c121740 msgr2=0x7fc10c11c5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:02:36.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.347+0000 7fc1027fc640 1 --2- 172.21.15.42:0/2388550465 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc10c121740 0x7fc10c11c5b0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fc10800f4b0 tx=0x7fc1080046d0 comp rx=0 tx=0).stop 2024-09-16T04:02:36.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.347+0000 7fc1027fc640 1 -- 172.21.15.42:0/2388550465 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc10c121740 msgr2=0x7fc10c11c5b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:02:36.364 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.347+0000 7fc1027fc640 1 --2- 172.21.15.42:0/2388550465 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc10c121740 0x7fc10c11c5b0 secure :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fc10800f4b0 tx=0x7fc1080046d0 comp rx=0 tx=0).stop 2024-09-16T04:02:36.364 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T04:02:36.364 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:02:36.364 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:02:36.364 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T04:02:36.364 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 184, 2024-09-16T04:02:36.364 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 284526, 2024-09-16T04:02:36.364 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T04:02:36.364 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T04:02:36.364 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T04:02:36.365 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T04:02:36.365 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:02:36.365 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 5, 2024-09-16T04:02:36.365 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:02:36.365 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T04:02:36.365 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T04:02:36.365 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:02:36.365 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:02:36.365 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:02:36.365 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:02:36.365 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:02:36.365 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:02:36.366 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:02:36.366 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:02:36.366 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:02:36.366 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:02:36.366 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:02:36.366 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:02:36.366 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:02:36.366 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T04:02:36.366 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:02:36.366 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:02:36.366 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:02:36.366 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:02:36.366 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:02:36.367 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:02:36.367 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:02:36.367 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:02:36.367 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:02:36.367 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:02:36.367 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:02:36.367 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:02:36.367 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:02:36.367 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T04:02:36.367 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T04:02:36.367 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T04:02:36.367 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T04:02:36.367 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T04:02:36.368 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T04:02:36.368 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:02:36.368 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:02:36.368 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T04:02:36.368 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T04:02:36.368 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T04:02:36.368 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T04:02:36.368 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T04:02:36.368 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T04:02:36.368 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T04:02:36.368 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:02:36.368 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T04:02:36.369 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:02:36.369 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:02:36.369 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:02:36.369 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:02:36.369 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:02:36.369 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:02:36.369 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:02:36.369 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:02:36.369 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:02:36.369 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:02:36.369 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:02:36.369 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:02:36.369 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T04:02:36.369 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:02:36.370 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T04:02:36.370 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:02:36.370 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:02:36.370 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:02:36.370 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:02:36.370 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:02:36.370 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:02:36.370 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:02:36.370 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T04:02:36.370 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:02:36.370 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:02:36.370 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:02:36.370 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:02:36.370 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T04:02:36.371 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:02:36.371 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:02:36.371 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:02:36.371 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:02:36.371 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T04:02:36.371 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T04:02:36.371 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:02:36.371 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:02:36.371 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:02:36.371 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:02:36.371 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:02:36.371 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:02:36.371 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:02:36.371 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:02:36.372 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:02:36.372 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:02:36.372 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:02:36.372 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T04:02:36.372 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:02:36.372 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:02:36.372 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:02:36.372 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:02:36.372 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T04:02:36.372 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:02:36.372 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:02:36.372 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:02:36.372 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:02:36.373 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T04:02:36.373 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T04:02:36.373 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:02:36.373 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:02:36.373 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:02:36.373 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:02:36.373 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:02:36.373 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:02:36.373 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:02:36.373 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:02:36.373 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:02:36.373 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:02:36.373 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:02:36.373 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T04:02:36.374 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:02:36.374 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:02:36.374 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:02:36.374 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:02:36.374 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T04:02:36.374 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:02:36.374 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:02:36.374 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:02:36.374 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:02:36.374 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T04:02:36.374 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T04:02:36.374 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:02:36.374 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:02:36.374 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:02:36.375 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:02:36.375 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:02:36.375 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:02:36.375 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T04:02:36.375 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T04:02:36.375 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:02:36.375 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:02:36.375 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:02:36.375 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:02:36.375 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:02:36.375 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:02:36.375 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T04:02:36.375 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:02:36.375 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:02:36.376 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:02:36.376 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:02:36.376 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:02:36.376 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:02:36.376 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T04:02:36.376 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:02:36.376 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:02:36.376 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:02:36.376 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 2 2024-09-16T04:02:36.376 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:02:36.376 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:02:36.376 INFO:teuthology.orchestra.run.smithi042.stdout: "mgr": [ 2024-09-16T04:02:36.376 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:02:36.376 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:02:36.377 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:02:36.377 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:02:36.377 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:02:36.377 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:02:36.377 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:02:36.377 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T04:02:36.377 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T04:02:36.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.347+0000 7fc113b93640 10 librados: watch_flush enter 2024-09-16T04:02:36.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.347+0000 7fc113b93640 10 librados: watch_flush exit 2024-09-16T04:02:36.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.347+0000 7fc113b93640 20 client.825567.objecter shutdown clearing up homeless session... 2024-09-16T04:02:36.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.347+0000 7fc113b93640 10 client.825567.objecter successfully canceled tick 2024-09-16T04:02:36.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.347+0000 7fc113b93640 1 -- 172.21.15.42:0/2388550465 >> v2:172.21.15.154:6812/254845342 conn(0x7fc0f0037840 msgr2=0x7fc0f0039d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:02:36.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.347+0000 7fc113b93640 1 --2- 172.21.15.42:0/2388550465 >> v2:172.21.15.154:6812/254845342 conn(0x7fc0f0037840 0x7fc0f0039d00 secure :-1 s=READY pgs=3790 cs=0 l=1 rev1=1 crypto rx=0x7fc0fc0097f0 tx=0x7fc0fc008040 comp rx=0 tx=0).stop 2024-09-16T04:02:36.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.347+0000 7fc113b93640 10 monclient: shutdown 2024-09-16T04:02:36.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.347+0000 7fc113b93640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:02:36.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.347+0000 7fc113b93640 1 -- 172.21.15.42:0/2388550465 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc10c09f260 msgr2=0x7fc10c13b890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:02:36.380 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.347+0000 7fc112130640 1 -- 172.21.15.42:0/2388550465 reap_dead start 2024-09-16T04:02:36.380 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.347+0000 7fc113b93640 1 --2- 172.21.15.42:0/2388550465 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc10c09f260 0x7fc10c13b890 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fc10400c710 tx=0x7fc10400cbd0 comp rx=0 tx=0).stop 2024-09-16T04:02:36.380 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.347+0000 7fc113b93640 1 -- 172.21.15.42:0/2388550465 shutdown_connections 2024-09-16T04:02:36.380 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.347+0000 7fc113b93640 1 --2- 172.21.15.42:0/2388550465 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc10c09f260 0x7fc10c13b890 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:36.380 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.347+0000 7fc113b93640 1 -- 172.21.15.42:0/2388550465 >> 172.21.15.42:0/2388550465 conn(0x7fc10c06bc40 msgr2=0x7fc10c071b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:02:36.380 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.347+0000 7fc113b93640 1 -- 172.21.15.42:0/2388550465 shutdown_connections 2024-09-16T04:02:36.380 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.347+0000 7fc113b93640 1 -- 172.21.15.42:0/2388550465 wait complete. 2024-09-16T04:02:36.380 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.347+0000 7fc113b93640 1 librados: shutdown 2024-09-16T04:02:36.380 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:02:36.381 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-09-16T04:02:36.381 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-16T04:02:36.381 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-09-16T04:02:36.381 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-16T04:02:36.381 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-16T04:02:36.381 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T04:02:36.392 INFO:tasks.ceph.mon.c:Stopped 2024-09-16T04:02:36.393 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T04:02:36.393 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:02:36.551 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.547+0000 7f9e9ef39640 10 monclient: get_monmap_and_config 2024-09-16T04:02:36.551 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.547+0000 7f9e9ef39640 10 monclient: build_initial_monmap 2024-09-16T04:02:36.551 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.547+0000 7f9e9ef39640 10 monclient: monmap: 2024-09-16T04:02:36.551 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:02:36.551 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:02:36.551 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:02:36.549748+0000 2024-09-16T04:02:36.551 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:02:36.549748+0000 2024-09-16T04:02:36.552 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:02:36.552 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:02:36.552 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:02:36.552 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:02:36.552 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:02:36.552 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:02:36.552 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:02:36.552 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:02:36.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.547+0000 7f9e9ef39640 1 Processor -- start 2024-09-16T04:02:36.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.547+0000 7f9e9ef39640 1 -- start start 2024-09-16T04:02:36.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.547+0000 7f9e9ef39640 10 monclient: init 2024-09-16T04:02:36.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.547+0000 7f9e9ef39640 10 monclient: _reopen_session rank -1 2024-09-16T04:02:36.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.547+0000 7f9e9ef39640 10 monclient: _add_conns ranks=[2,3,1,0,4] 2024-09-16T04:02:36.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.547+0000 7f9e9ef39640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9e980a1330 0x7f9e9809f180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:36.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.547+0000 7f9e9ef39640 10 monclient(hunting): picked mon.noname-b con 0x7f9e980a1330 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:02:36.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.547+0000 7f9e9ef39640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f9e980a1710 0x7f9e980a1b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:36.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.547+0000 7f9e9ef39640 10 monclient(hunting): picked mon.noname-a con 0x7f9e980a1710 addr v2:172.21.15.42:3300/0 2024-09-16T04:02:36.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.547+0000 7f9e9ef39640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9e981340f0 addr v1:172.21.15.154:6789/0 2024-09-16T04:02:36.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.547+0000 7f9e9ef39640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9e9809e4c0 con 0x7f9e981340f0 2024-09-16T04:02:36.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.547+0000 7f9e9ef39640 10 monclient(hunting): start opening mon connection 2024-09-16T04:02:36.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.547+0000 7f9e9ef39640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f9e9809edf0 con 0x7f9e980a1710 2024-09-16T04:02:36.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.547+0000 7f9e9ef39640 10 monclient(hunting): start opening mon connection 2024-09-16T04:02:36.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.547+0000 7f9e9ef39640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f9e9809cd60 con 0x7f9e980a1330 2024-09-16T04:02:36.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.547+0000 7f9e9ef39640 10 monclient(hunting): _renew_subs 2024-09-16T04:02:36.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.547+0000 7f9e9ef39640 10 monclient(hunting): authenticate will time out at 2203.519180s 2024-09-16T04:02:36.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.547+0000 7f9e9ccd5640 1 -- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9e980a1330 msgr2=0x7f9e9809f180 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T04:02:36.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.547+0000 7f9e9ccd5640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9e980a1330 0x7f9e9809f180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:02:36.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.547+0000 7f9e97fff640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f9e980a1710 0x7f9e980a1b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:02:36.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.547+0000 7f9e97fff640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f9e980a1710 0x7f9e980a1b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:53808/0 (socket says 172.21.15.42:53808) 2024-09-16T04:02:36.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.547+0000 7f9e97fff640 1 -- 172.21.15.42:0/2470823351 learned_addr learned my addr 172.21.15.42:0/2470823351 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:02:36.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.547+0000 7f9e97fff640 10 monclient(hunting): get_auth_request con 0x7f9e980a1710 auth_method 0 2024-09-16T04:02:36.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.547+0000 7f9e97fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:02:36.556 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.547+0000 7f9e97fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:02:36.556 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.547+0000 7f9e97fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:02:36.556 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.547+0000 7f9e97fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:02:36.556 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.547+0000 7f9e97fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:02:36.556 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.547+0000 7f9e97fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:02:36.556 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.547+0000 7f9e97fff640 10 monclient(hunting): handle_auth_done global_id 825570 payload 274 2024-09-16T04:02:36.556 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.547+0000 7f9e97fff640 1 -- 172.21.15.42:0/2470823351 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9e980a1330 msgr2=0x7f9e9809f180 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:02:36.556 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.547+0000 7f9e97fff640 1 --2- 172.21.15.42:0/2470823351 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9e980a1330 0x7f9e9809f180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:36.556 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.547+0000 7f9e97fff640 1 -- 172.21.15.42:0/2470823351 >> v1:172.21.15.154:6789/0 conn(0x7f9e981340f0 legacy=0x7f9e981345a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:02:36.556 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.547+0000 7f9e97fff640 10 monclient: _finish_hunting 0 2024-09-16T04:02:36.556 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.547+0000 7f9e97fff640 1 monclient: found mon.noname-a 2024-09-16T04:02:36.556 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.547+0000 7f9e97fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:02:36.556 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.547+0000 7f9e97fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:02:36.557 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.547+0000 7f9e97fff640 1 -- 172.21.15.42:0/2470823351 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e980a2050 con 0x7f9e980a1710 2024-09-16T04:02:36.557 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.551+0000 7f9e97fff640 1 -- 172.21.15.42:0/2470823351 >> v2:172.21.15.42:3300/0 conn(0x7f9e980a1710 msgr2=0x7f9e980a1b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:02:36.557 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.551+0000 7f9e97fff640 1 -- 172.21.15.42:0/2470823351 >> v2:172.21.15.42:3300/0 conn(0x7f9e980a1710 msgr2=0x7f9e980a1b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:02:36.557 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.551+0000 7f9e97fff640 1 --2- 172.21.15.42:0/2470823351 >> v2:172.21.15.42:3300/0 conn(0x7f9e980a1710 0x7f9e980a1b10 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0x7f9e800098a0 tx=0x7f9e8001c3e0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:02:36.557 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.551+0000 7f9e97fff640 1 --2- 172.21.15.42:0/2470823351 >> v2:172.21.15.42:3300/0 conn(0x7f9e980a1710 0x7f9e980a1b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:02:36.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.755+0000 7f9e97fff640 1 --2- 172.21.15.42:0/2470823351 >> v2:172.21.15.42:3300/0 conn(0x7f9e980a1710 0x7f9e980a1b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:02:36.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.755+0000 7f9e97fff640 10 monclient: get_auth_request con 0x7f9e980a1710 auth_method 0 2024-09-16T04:02:36.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.755+0000 7f9e97fff640 0 --2- 172.21.15.42:0/2470823351 >> v2:172.21.15.42:3300/0 conn(0x7f9e980a1710 0x7f9e980a1b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-09-16T04:02:36.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.755+0000 7f9e97fff640 1 --2- 172.21.15.42:0/2470823351 >> v2:172.21.15.42:3300/0 conn(0x7f9e980a1710 0x7f9e980a1b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:36.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.755+0000 7f9e977fe640 10 monclient: ms_handle_reset current mon v2:172.21.15.42:3300/0 2024-09-16T04:02:36.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.755+0000 7f9e977fe640 10 monclient: _reopen_session rank -1 2024-09-16T04:02:36.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.755+0000 7f9e977fe640 1 -- 172.21.15.42:0/2470823351 >> v2:172.21.15.42:3300/0 conn(0x7f9e980a1710 msgr2=0x7f9e980a1b10 unknown :-1 s=STATE_CLOSED l=0).mark_down 2024-09-16T04:02:36.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.755+0000 7f9e977fe640 1 --2- 172.21.15.42:0/2470823351 >> v2:172.21.15.42:3300/0 conn(0x7f9e980a1710 0x7f9e980a1b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:36.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.755+0000 7f9e977fe640 10 monclient: _add_conns ranks=[0,4] 2024-09-16T04:02:36.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.755+0000 7f9e977fe640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9e84003510 addr v1:172.21.15.42:6789/0 2024-09-16T04:02:36.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.755+0000 7f9e977fe640 1 --2- 172.21.15.42:0/2470823351 >> v2:172.21.15.154:3300/0 conn(0x7f9e840074a0 0x7f9e84009960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:36.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.755+0000 7f9e977fe640 10 monclient(hunting): picked mon.noname-c con 0x7f9e840074a0 addr v2:172.21.15.154:3300/0 2024-09-16T04:02:36.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.755+0000 7f9e977fe640 1 -- 172.21.15.42:0/2470823351 --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9e8400a230 con 0x7f9e84003510 2024-09-16T04:02:36.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.755+0000 7f9e977fe640 10 monclient(hunting): start opening mon connection 2024-09-16T04:02:36.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.755+0000 7f9e977fe640 1 -- 172.21.15.42:0/2470823351 --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f9e8400b470 con 0x7f9e840074a0 2024-09-16T04:02:36.761 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.755+0000 7f9e977fe640 10 monclient(hunting): _renew_subs 2024-09-16T04:02:36.761 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.755+0000 7f9e977fe640 10 monclient(hunting): ms_handle_reset stray mon v2:172.21.15.42:3300/0 2024-09-16T04:02:36.761 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.755+0000 7f9e97fff640 1 --2- 172.21.15.42:0/2470823351 >> v2:172.21.15.154:3300/0 conn(0x7f9e840074a0 0x7f9e84009960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:02:36.761 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.755+0000 7f9e97fff640 10 monclient(hunting): get_auth_request con 0x7f9e840074a0 auth_method 0 2024-09-16T04:02:36.761 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.755+0000 7f9e97fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:02:36.761 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.755+0000 7f9e97fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:02:36.761 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.755+0000 7f9e97fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:02:36.761 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.755+0000 7f9e977fe640 1 -- 172.21.15.42:0/2470823351 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3023590144 0 0) 0x7f9e8400a230 con 0x7f9e84003510 2024-09-16T04:02:36.761 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.755+0000 7f9e977fe640 10 monclient(hunting): _negotiate 2024-09-16T04:02:36.761 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.755+0000 7f9e977fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:02:36.761 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.755+0000 7f9e977fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:02:36.761 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.755+0000 7f9e977fe640 1 -- 172.21.15.42:0/2470823351 --> v1:172.21.15.42:6789/0 -- auth(proto 2 132 bytes epoch 0) -- 0x7f9e8400a400 con 0x7f9e84003510 2024-09-16T04:02:36.761 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.755+0000 7f9e97fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:02:36.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.755+0000 7f9e97fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:02:36.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.755+0000 7f9e97fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:02:36.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.755+0000 7f9e97fff640 10 monclient(hunting): handle_auth_done global_id 825570 payload 293 2024-09-16T04:02:36.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.755+0000 7f9e97fff640 1 -- 172.21.15.42:0/2470823351 >> v1:172.21.15.42:6789/0 conn(0x7f9e84003510 legacy=0x7f9e84005990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:02:36.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.755+0000 7f9e977fe640 1 -- 172.21.15.42:0/2470823351 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (unknown 4102054118 0 0) 0x7f9e8400a400 con 0x7f9e84003510 2024-09-16T04:02:36.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.755+0000 7f9e97fff640 10 monclient: _finish_hunting 0 2024-09-16T04:02:36.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.755+0000 7f9e97fff640 1 monclient: found mon.noname-c 2024-09-16T04:02:36.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.755+0000 7f9e97fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:02:36.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.755+0000 7f9e97fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.154:3300/0 2024-09-16T04:02:36.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.755+0000 7f9e97fff640 1 -- 172.21.15.42:0/2470823351 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e8400b6e0 con 0x7f9e840074a0 2024-09-16T04:02:36.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.755+0000 7f9e977fe640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T04:02:36.768 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.755+0000 7f9e97fff640 1 --2- 172.21.15.42:0/2470823351 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9e840074a0 0x7f9e84009960 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f9e800098a0 tx=0x7f9e80009240 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f362d2ac6c62f727 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:02:36.768 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.755+0000 7f9e977fe640 1 -- 172.21.15.42:0/2470823351 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e800ab6e0 con 0x7f9e840074a0 2024-09-16T04:02:36.768 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.755+0000 7f9e977fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:02:36.768 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.755+0000 7f9e977fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:02:36.768 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.755+0000 7f9e977fe640 10 monclient: dump: 2024-09-16T04:02:36.768 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:02:36.768 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:02:36.768 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:36.768 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:36.769 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:02:36.769 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:02:36.769 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:02:36.769 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:02:36.769 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:02:36.769 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:02:36.769 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.755+0000 7f9e977fe640 10 monclient: _finish_auth 0 2024-09-16T04:02:36.769 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.755+0000 7f9e977fe640 10 monclient: _check_auth_tickets 2024-09-16T04:02:36.769 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.755+0000 7f9e977fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:02:36.769 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.755+0000 7f9e977fe640 1 -- 172.21.15.42:0/2470823351 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9e800aba70 con 0x7f9e840074a0 2024-09-16T04:02:36.769 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.755+0000 7f9e977fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:02:36.769 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.755+0000 7f9e977fe640 1 -- 172.21.15.42:0/2470823351 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e88004030 con 0x7f9e840074a0 2024-09-16T04:02:36.769 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.755+0000 7f9e9ef39640 5 monclient: authenticate success, global_id 825570 2024-09-16T04:02:36.770 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.755+0000 7f9e9ef39640 10 monclient: get_monmap_and_config success 2024-09-16T04:02:36.770 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.755+0000 7f9e9ef39640 4 set_mon_vals no callback set 2024-09-16T04:02:36.770 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.755+0000 7f9e977fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:02:36.770 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.755+0000 7f9e977fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:02:36.770 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.755+0000 7f9e977fe640 10 monclient: dump: 2024-09-16T04:02:36.770 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:02:36.770 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:02:36.770 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:36.770 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:36.770 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:02:36.770 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:02:36.770 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:02:36.770 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:02:36.770 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:02:36.771 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:02:36.771 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.755+0000 7f9e9ef39640 10 monclient: shutdown 2024-09-16T04:02:36.771 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.755+0000 7f9e9ef39640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:02:36.771 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.755+0000 7f9e9ef39640 1 -- 172.21.15.42:0/2470823351 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9e840074a0 msgr2=0x7f9e84009960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:02:36.771 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.755+0000 7f9e9ef39640 1 --2- 172.21.15.42:0/2470823351 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9e840074a0 0x7f9e84009960 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f9e800098a0 tx=0x7f9e80009240 comp rx=0 tx=0).stop 2024-09-16T04:02:36.771 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.759+0000 7f9e9d4d6640 1 -- 172.21.15.42:0/2470823351 reap_dead start 2024-09-16T04:02:36.771 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.759+0000 7f9e9ef39640 1 -- 172.21.15.42:0/2470823351 shutdown_connections 2024-09-16T04:02:36.771 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.759+0000 7f9e9ef39640 1 --2- 172.21.15.42:0/2470823351 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9e840074a0 0x7f9e84009960 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:36.771 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.759+0000 7f9e9ef39640 1 -- 172.21.15.42:0/2470823351 >> 172.21.15.42:0/2470823351 conn(0x7f9e980a2420 msgr2=0x7f9e980a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:02:36.771 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.759+0000 7f9e9ef39640 1 -- 172.21.15.42:0/2470823351 shutdown_connections 2024-09-16T04:02:36.771 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.759+0000 7f9e9ef39640 1 -- 172.21.15.42:0/2470823351 wait complete. 2024-09-16T04:02:36.771 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.759+0000 7f9e9ef39640 10 monclient: build_initial_monmap 2024-09-16T04:02:36.771 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.759+0000 7f9e9ef39640 10 monclient: monmap: 2024-09-16T04:02:36.772 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:02:36.772 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:02:36.772 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:02:36.772 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:02:36.772 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:02:36.772 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:02:36.772 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:02:36.772 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:02:36.772 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:02:36.772 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:02:36.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.759+0000 7f9e9ef39640 1 librados: starting msgr at 2024-09-16T04:02:36.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.759+0000 7f9e9ef39640 1 librados: starting objecter 2024-09-16T04:02:36.773 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.759+0000 7f9e9ef39640 1 Processor -- start 2024-09-16T04:02:36.773 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.759+0000 7f9e9ef39640 1 -- start start 2024-09-16T04:02:36.773 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.759+0000 7f9e9ef39640 1 librados: setting wanted keys 2024-09-16T04:02:36.773 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.759+0000 7f9e9ef39640 1 librados: calling monclient init 2024-09-16T04:02:36.773 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.759+0000 7f9e9ef39640 10 monclient: init 2024-09-16T04:02:36.773 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.759+0000 7f9e9ef39640 10 monclient: _reopen_session rank -1 2024-09-16T04:02:36.773 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.759+0000 7f9e9ef39640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:02:36.773 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.759+0000 7f9e9ef39640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9e840074a0 0x7f9e981cf420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:36.773 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.759+0000 7f9e9ef39640 10 monclient(hunting): picked mon.noname-c con 0x7f9e840074a0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:02:36.773 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.759+0000 7f9e9ef39640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9e981314f0 0x7f9e98131950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:36.773 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.759+0000 7f9e9ef39640 10 monclient(hunting): picked mon.noname-a con 0x7f9e981314f0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:02:36.773 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.759+0000 7f9e9ef39640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9e981383d0 0x7f9e98138880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:36.773 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.759+0000 7f9e9ef39640 10 monclient(hunting): picked mon.noname-b con 0x7f9e981383d0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:02:36.774 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.759+0000 7f9e9ef39640 10 monclient(hunting): start opening mon connection 2024-09-16T04:02:36.774 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.759+0000 7f9e9ef39640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f9e9813ed60 con 0x7f9e981314f0 2024-09-16T04:02:36.774 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.759+0000 7f9e9ef39640 10 monclient(hunting): start opening mon connection 2024-09-16T04:02:36.774 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.759+0000 7f9e9ef39640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f9e9813ebe0 con 0x7f9e981383d0 2024-09-16T04:02:36.774 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.759+0000 7f9e9ef39640 10 monclient(hunting): start opening mon connection 2024-09-16T04:02:36.774 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.759+0000 7f9e9ef39640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f9e9813eee0 con 0x7f9e840074a0 2024-09-16T04:02:36.774 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.759+0000 7f9e9ef39640 10 monclient(hunting): _renew_subs 2024-09-16T04:02:36.774 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.759+0000 7f9e9ccd5640 1 -- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9e840074a0 msgr2=0x7f9e981cf420 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T04:02:36.774 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.759+0000 7f9e9ef39640 10 monclient(hunting): authenticate will time out at 2203.730561s 2024-09-16T04:02:36.774 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.759+0000 7f9e9ccd5640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9e840074a0 0x7f9e981cf420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:02:36.774 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.759+0000 7f9e97fff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9e981314f0 0x7f9e98131950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:02:36.774 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.759+0000 7f9e97fff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9e981314f0 0x7f9e98131950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:53828/0 (socket says 172.21.15.42:53828) 2024-09-16T04:02:36.774 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.759+0000 7f9e97fff640 1 -- 172.21.15.42:0/1520753507 learned_addr learned my addr 172.21.15.42:0/1520753507 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:02:36.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.759+0000 7f9e97fff640 10 monclient(hunting): get_auth_request con 0x7f9e981314f0 auth_method 0 2024-09-16T04:02:36.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.759+0000 7f9e97fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:02:36.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.759+0000 7f9e97fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:02:36.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.759+0000 7f9e97fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:02:36.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.759+0000 7f9e97fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:02:36.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.759+0000 7f9e97fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:02:36.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.759+0000 7f9e97fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:02:36.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.759+0000 7f9e9d4d6640 1 --2- 172.21.15.42:0/1520753507 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9e981383d0 0x7f9e98138880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:02:36.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.759+0000 7f9e9d4d6640 10 monclient(hunting): get_auth_request con 0x7f9e981383d0 auth_method 0 2024-09-16T04:02:36.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.759+0000 7f9e9d4d6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:02:36.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.759+0000 7f9e9d4d6640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:02:36.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.759+0000 7f9e9d4d6640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:02:36.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.759+0000 7f9e97fff640 10 monclient(hunting): handle_auth_done global_id 825573 payload 824 2024-09-16T04:02:36.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.759+0000 7f9e97fff640 1 -- 172.21.15.42:0/1520753507 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9e840074a0 msgr2=0x7f9e981cf420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:02:36.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.759+0000 7f9e97fff640 1 --2- 172.21.15.42:0/1520753507 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9e840074a0 0x7f9e981cf420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:36.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.759+0000 7f9e97fff640 1 -- 172.21.15.42:0/1520753507 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9e981383d0 msgr2=0x7f9e98138880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:02:36.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.759+0000 7f9e97fff640 1 --2- 172.21.15.42:0/1520753507 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9e981383d0 0x7f9e98138880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:36.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.759+0000 7f9e97fff640 10 monclient: _finish_hunting 0 2024-09-16T04:02:36.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.759+0000 7f9e97fff640 1 monclient: found mon.noname-a 2024-09-16T04:02:36.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.759+0000 7f9e97fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:02:36.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.759+0000 7f9e97fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:02:36.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.759+0000 7f9e97fff640 1 -- 172.21.15.42:0/1520753507 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e9811cbb0 con 0x7f9e981314f0 2024-09-16T04:02:36.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.759+0000 7f9e9d4d6640 1 --2- 172.21.15.42:0/1520753507 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9e981383d0 0x7f9e98138880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T04:02:36.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.759+0000 7f9e97fff640 1 --2- 172.21.15.42:0/1520753507 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9e981314f0 0x7f9e98131950 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f9e80097d80 tx=0x7f9e80019180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:02:36.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.759+0000 7f9e957fa640 10 client.?.objecter ms_handle_connect 0x7f9e981314f0 2024-09-16T04:02:36.777 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.759+0000 7f9e957fa640 10 client.?.objecter resend_mon_ops 2024-09-16T04:02:36.777 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.759+0000 7f9e957fa640 1 -- 172.21.15.42:0/1520753507 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e800bb040 con 0x7f9e981314f0 2024-09-16T04:02:36.777 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.759+0000 7f9e957fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:02:36.777 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.759+0000 7f9e957fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:02:36.777 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.759+0000 7f9e957fa640 10 monclient: dump: 2024-09-16T04:02:36.777 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:02:36.777 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:02:36.777 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:36.777 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:36.777 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:02:36.777 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:02:36.777 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:02:36.777 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:02:36.778 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:02:36.778 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:02:36.778 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.759+0000 7f9e957fa640 10 monclient: _finish_auth 0 2024-09-16T04:02:36.778 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.759+0000 7f9e957fa640 10 monclient: _check_auth_tickets 2024-09-16T04:02:36.778 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.759+0000 7f9e957fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:02:36.778 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.759+0000 7f9e957fa640 1 -- 172.21.15.42:0/1520753507 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9e80004250 con 0x7f9e981314f0 2024-09-16T04:02:36.778 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.759+0000 7f9e9ef39640 5 monclient: authenticate success, global_id 825573 2024-09-16T04:02:36.778 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.759+0000 7f9e9ef39640 10 monclient: _renew_subs 2024-09-16T04:02:36.778 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.759+0000 7f9e9ef39640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:02:36.778 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.759+0000 7f9e9ef39640 1 -- 172.21.15.42:0/1520753507 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e981cf960 con 0x7f9e981314f0 2024-09-16T04:02:36.778 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.759+0000 7f9e957fa640 10 monclient: handle_config config(0 keys) 2024-09-16T04:02:36.778 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.759+0000 7f9e957fa640 1 -- 172.21.15.42:0/1520753507 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e800059a0 con 0x7f9e981314f0 2024-09-16T04:02:36.778 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.759+0000 7f9e957fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:02:36.778 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.759+0000 7f9e957fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:02:36.779 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.759+0000 7f9e957fa640 10 monclient: dump: 2024-09-16T04:02:36.779 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:02:36.779 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:02:36.779 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:36.779 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:36.779 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:02:36.779 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:02:36.779 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:02:36.779 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:02:36.779 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:02:36.779 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:02:36.779 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.759+0000 7f9e967fc640 4 set_mon_vals no callback set 2024-09-16T04:02:36.779 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.759+0000 7f9e9ef39640 10 client.825573.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:02:36.779 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.759+0000 7f9e9ef39640 10 monclient: _renew_subs 2024-09-16T04:02:36.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.759+0000 7f9e9ef39640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:02:36.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.759+0000 7f9e9ef39640 1 -- 172.21.15.42:0/1520753507 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e980a0a70 con 0x7f9e981314f0 2024-09-16T04:02:36.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.759+0000 7f9e9ef39640 1 librados: init done 2024-09-16T04:02:36.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.763+0000 7f9e9ef39640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:02:36.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.763+0000 7f9e9ef39640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:02:36.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.763+0000 7f9e9ef39640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:02:36.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.763+0000 7f9e9ef39640 1 -- 172.21.15.42:0/1520753507 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e98073a30 con 0x7f9e981314f0 2024-09-16T04:02:36.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.763+0000 7f9e957fa640 1 -- 172.21.15.42:0/1520753507 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f9e80005b40 con 0x7f9e981314f0 2024-09-16T04:02:36.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.763+0000 7f9e957fa640 1 --2- 172.21.15.42:0/1520753507 >> v2:172.21.15.154:6812/254845342 conn(0x7f9e780377f0 0x7f9e78039cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:36.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.763+0000 7f9e957fa640 1 -- 172.21.15.42:0/1520753507 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(632..632 src has 451..632) ==== 6367+0+0 (secure 0 0 0) 0x7f9e800e4510 con 0x7f9e981314f0 2024-09-16T04:02:36.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.763+0000 7f9e957fa640 10 client.825573.objecter ms_dispatch 0x7f9e98000ba0 osd_map(632..632 src has 451..632) 2024-09-16T04:02:36.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.763+0000 7f9e957fa640 3 client.825573.objecter handle_osd_map got epochs [632,632] > 0 2024-09-16T04:02:36.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.763+0000 7f9e957fa640 3 client.825573.objecter handle_osd_map decoding full epoch 632 2024-09-16T04:02:36.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.763+0000 7f9e957fa640 20 client.825573.objecter dump_active .. 0 homeless 2024-09-16T04:02:36.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.763+0000 7f9e9ccd5640 1 --2- 172.21.15.42:0/1520753507 >> v2:172.21.15.154:6812/254845342 conn(0x7f9e780377f0 0x7f9e78039cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:02:36.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.763+0000 7f9e9ccd5640 10 monclient: get_auth_request con 0x7f9e780377f0 auth_method 0 2024-09-16T04:02:36.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.763+0000 7f9e9ccd5640 1 --2- 172.21.15.42:0/1520753507 >> v2:172.21.15.154:6812/254845342 conn(0x7f9e780377f0 0x7f9e78039cb0 secure :-1 s=READY pgs=3791 cs=0 l=1 rev1=1 crypto rx=0x7f9e88003d00 tx=0x7f9e88003e30 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:02:36.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.763+0000 7f9e957fa640 10 client.825573.objecter ms_handle_connect 0x7f9e780377f0 2024-09-16T04:02:36.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.767+0000 7f9e957fa640 1 -- 172.21.15.42:0/1520753507 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f9e800b2070 con 0x7f9e981314f0 2024-09-16T04:02:36.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.767+0000 7f9e957fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:02:36.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.767+0000 7f9e957fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:02:37.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.999+0000 7f9e9ef39640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:02:37.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.999+0000 7f9e9ef39640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:02:37.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.999+0000 7f9e9ef39640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:02:37.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.999+0000 7f9e9ef39640 1 -- 172.21.15.42:0/1520753507 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f9e9809f140 con 0x7f9e981314f0 2024-09-16T04:02:37.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.999+0000 7f9e957fa640 1 -- 172.21.15.42:0/1520753507 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7f9e800160f0 con 0x7f9e981314f0 2024-09-16T04:02:37.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.999+0000 7f9e957fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:02:37.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:36.999+0000 7f9e957fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:02:37.005 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":184,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:02:37.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:37.003+0000 7f9e9ef39640 10 librados: watch_flush enter 2024-09-16T04:02:37.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:37.003+0000 7f9e9ef39640 10 librados: watch_flush exit 2024-09-16T04:02:37.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:37.003+0000 7f9e9ef39640 20 client.825573.objecter shutdown clearing up homeless session... 2024-09-16T04:02:37.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:37.003+0000 7f9e9ef39640 10 client.825573.objecter successfully canceled tick 2024-09-16T04:02:37.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:37.003+0000 7f9e9ef39640 1 -- 172.21.15.42:0/1520753507 >> v2:172.21.15.154:6812/254845342 conn(0x7f9e780377f0 msgr2=0x7f9e78039cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:02:37.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:37.003+0000 7f9e9ef39640 1 --2- 172.21.15.42:0/1520753507 >> v2:172.21.15.154:6812/254845342 conn(0x7f9e780377f0 0x7f9e78039cb0 secure :-1 s=READY pgs=3791 cs=0 l=1 rev1=1 crypto rx=0x7f9e88003d00 tx=0x7f9e88003e30 comp rx=0 tx=0).stop 2024-09-16T04:02:37.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:37.003+0000 7f9e9ef39640 10 monclient: shutdown 2024-09-16T04:02:37.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:37.003+0000 7f9e9ef39640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:02:37.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:37.003+0000 7f9e9ef39640 1 -- 172.21.15.42:0/1520753507 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9e981314f0 msgr2=0x7f9e98131950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:02:37.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:37.003+0000 7f9e9ef39640 1 --2- 172.21.15.42:0/1520753507 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9e981314f0 0x7f9e98131950 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f9e80097d80 tx=0x7f9e80019180 comp rx=0 tx=0).stop 2024-09-16T04:02:37.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:37.003+0000 7f9e9ef39640 1 -- 172.21.15.42:0/1520753507 shutdown_connections 2024-09-16T04:02:37.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:37.003+0000 7f9e9ef39640 1 --2- 172.21.15.42:0/1520753507 >> v2:172.21.15.154:6812/254845342 conn(0x7f9e780377f0 0x7f9e78039cb0 unknown :-1 s=CLOSED pgs=3791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:37.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:37.003+0000 7f9e9ef39640 1 --2- 172.21.15.42:0/1520753507 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9e981383d0 0x7f9e98138880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:37.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:37.003+0000 7f9e9ef39640 1 --2- 172.21.15.42:0/1520753507 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9e981314f0 0x7f9e98131950 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:37.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:37.003+0000 7f9e9ef39640 1 --2- 172.21.15.42:0/1520753507 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9e840074a0 0x7f9e981cf420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:37.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:37.003+0000 7f9e9ef39640 1 -- 172.21.15.42:0/1520753507 >> 172.21.15.42:0/1520753507 conn(0x7f9e980a2420 msgr2=0x7f9e9813b780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:02:37.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:37.003+0000 7f9e9ef39640 1 -- 172.21.15.42:0/1520753507 shutdown_connections 2024-09-16T04:02:37.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:37.003+0000 7f9e9ef39640 1 -- 172.21.15.42:0/1520753507 wait complete. 2024-09-16T04:02:37.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:37.003+0000 7f9e9ef39640 1 librados: shutdown 2024-09-16T04:02:40.024 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:02:40.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.155+0000 7f07a7b40640 10 monclient: get_monmap_and_config 2024-09-16T04:02:40.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.155+0000 7f07a7b40640 10 monclient: build_initial_monmap 2024-09-16T04:02:40.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.155+0000 7f07a7b40640 10 monclient: monmap: 2024-09-16T04:02:40.159 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:02:40.160 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:02:40.160 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:02:40.157553+0000 2024-09-16T04:02:40.160 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:02:40.157553+0000 2024-09-16T04:02:40.160 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:02:40.160 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:02:40.160 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:02:40.160 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:02:40.161 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:02:40.161 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:02:40.161 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:02:40.161 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:02:40.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.155+0000 7f07a7b40640 1 Processor -- start 2024-09-16T04:02:40.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.155+0000 7f07a7b40640 1 -- start start 2024-09-16T04:02:40.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.155+0000 7f07a7b40640 10 monclient: init 2024-09-16T04:02:40.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.155+0000 7f07a7b40640 10 monclient: _reopen_session rank -1 2024-09-16T04:02:40.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.155+0000 7f07a7b40640 10 monclient: _add_conns ranks=[1,4,3,0,2] 2024-09-16T04:02:40.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.155+0000 7f07a7b40640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f07a00a1330 addr v1:172.21.15.154:6789/0 2024-09-16T04:02:40.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.155+0000 7f07a7b40640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f07a006dac0 0x7f07a013a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:40.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.155+0000 7f07a7b40640 10 monclient(hunting): picked mon.noname-c con 0x7f07a006dac0 addr v2:172.21.15.154:3300/0 2024-09-16T04:02:40.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.155+0000 7f07a7b40640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f07a00a1e20 0x7f07a013c890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:40.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.155+0000 7f07a7b40640 10 monclient(hunting): picked mon.noname-a con 0x7f07a00a1e20 addr v2:172.21.15.42:3300/0 2024-09-16T04:02:40.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.155+0000 7f07a7b40640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f07a009c090 con 0x7f07a00a1330 2024-09-16T04:02:40.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.155+0000 7f07a7b40640 10 monclient(hunting): start opening mon connection 2024-09-16T04:02:40.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.155+0000 7f07a7b40640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f07a009f610 con 0x7f07a00a1e20 2024-09-16T04:02:40.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.155+0000 7f07a7b40640 10 monclient(hunting): start opening mon connection 2024-09-16T04:02:40.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.155+0000 7f07a7b40640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f07a009f8f0 con 0x7f07a006dac0 2024-09-16T04:02:40.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.155+0000 7f07a7b40640 10 monclient(hunting): _renew_subs 2024-09-16T04:02:40.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.155+0000 7f07a7b40640 10 monclient(hunting): authenticate will time out at 2207.127015s 2024-09-16T04:02:40.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.155+0000 7f07a60dd640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f07a00a1e20 0x7f07a013c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:02:40.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.155+0000 7f07a60dd640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f07a00a1e20 0x7f07a013c890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:53840/0 (socket says 172.21.15.42:53840) 2024-09-16T04:02:40.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.155+0000 7f07a60dd640 1 -- 172.21.15.42:0/811847738 learned_addr learned my addr 172.21.15.42:0/811847738 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:02:40.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.155+0000 7f07a60dd640 10 monclient(hunting): get_auth_request con 0x7f07a00a1e20 auth_method 0 2024-09-16T04:02:40.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.155+0000 7f07a60dd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:02:40.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.155+0000 7f07a60dd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:02:40.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.155+0000 7f07a60dd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:02:40.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.155+0000 7f07a60dd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:02:40.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.155+0000 7f07a60dd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:02:40.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.155+0000 7f07a50db640 1 --2- 172.21.15.42:0/811847738 >> v2:172.21.15.154:3300/0 conn(0x7f07a006dac0 0x7f07a013a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:02:40.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.155+0000 7f07a60dd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:02:40.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.155+0000 7f07a50db640 10 monclient(hunting): get_auth_request con 0x7f07a006dac0 auth_method 0 2024-09-16T04:02:40.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.155+0000 7f07a50db640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:02:40.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.155+0000 7f07a50db640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:02:40.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.155+0000 7f07a50db640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:02:40.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.155+0000 7f07a60dd640 10 monclient(hunting): handle_auth_done global_id 825576 payload 274 2024-09-16T04:02:40.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.155+0000 7f07a60dd640 1 -- 172.21.15.42:0/811847738 >> v2:172.21.15.154:3300/0 conn(0x7f07a006dac0 msgr2=0x7f07a013a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:02:40.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.155+0000 7f07a60dd640 1 --2- 172.21.15.42:0/811847738 >> v2:172.21.15.154:3300/0 conn(0x7f07a006dac0 0x7f07a013a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:40.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.155+0000 7f07a60dd640 1 -- 172.21.15.42:0/811847738 >> v1:172.21.15.154:6789/0 conn(0x7f07a00a1330 legacy=0x7f07a00a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:02:40.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.155+0000 7f07a60dd640 10 monclient: _finish_hunting 0 2024-09-16T04:02:40.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.155+0000 7f07a60dd640 1 monclient: found mon.noname-a 2024-09-16T04:02:40.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.155+0000 7f07a60dd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:02:40.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.155+0000 7f07a60dd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:02:40.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.155+0000 7f07a60dd640 1 -- 172.21.15.42:0/811847738 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07a009fbd0 con 0x7f07a00a1e20 2024-09-16T04:02:40.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.159+0000 7f07a60dd640 1 --2- 172.21.15.42:0/811847738 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f07a00a1e20 0x7f07a013c890 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f07a006da10 tx=0x7f079801c9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ded360c893aa71c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:02:40.180 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.159+0000 7f07a48da640 1 -- 172.21.15.42:0/811847738 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07980a5690 con 0x7f07a00a1e20 2024-09-16T04:02:40.180 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.159+0000 7f07a48da640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:02:40.180 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.159+0000 7f07a48da640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:02:40.180 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.159+0000 7f07a48da640 10 monclient: dump: 2024-09-16T04:02:40.180 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:02:40.180 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:02:40.181 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:40.181 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:40.181 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:02:40.181 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:02:40.181 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:02:40.181 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:02:40.181 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:02:40.182 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:02:40.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.159+0000 7f07a48da640 10 monclient: _finish_auth 0 2024-09-16T04:02:40.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.159+0000 7f07a48da640 10 monclient: _check_auth_tickets 2024-09-16T04:02:40.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.159+0000 7f07a48da640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:02:40.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.159+0000 7f07a48da640 1 -- 172.21.15.42:0/811847738 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f07980a5830 con 0x7f07a00a1e20 2024-09-16T04:02:40.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.159+0000 7f07a48da640 10 monclient: handle_config config(0 keys) 2024-09-16T04:02:40.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.159+0000 7f07a48da640 1 -- 172.21.15.42:0/811847738 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07980a5c00 con 0x7f07a00a1e20 2024-09-16T04:02:40.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.159+0000 7f07a7b40640 5 monclient: authenticate success, global_id 825576 2024-09-16T04:02:40.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.159+0000 7f07a7b40640 10 monclient: get_monmap_and_config success 2024-09-16T04:02:40.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.159+0000 7f07a7b40640 4 set_mon_vals no callback set 2024-09-16T04:02:40.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.159+0000 7f07a48da640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:02:40.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.159+0000 7f07a48da640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:02:40.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.159+0000 7f07a48da640 10 monclient: dump: 2024-09-16T04:02:40.183 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:02:40.183 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:02:40.184 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:40.184 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:40.184 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:02:40.184 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:02:40.184 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:02:40.184 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:02:40.184 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:02:40.184 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:02:40.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.163+0000 7f07a7b40640 10 monclient: shutdown 2024-09-16T04:02:40.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.163+0000 7f07a7b40640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:02:40.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.163+0000 7f07a7b40640 1 -- 172.21.15.42:0/811847738 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f07a00a1e20 msgr2=0x7f07a013c890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:02:40.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.163+0000 7f07a7b40640 1 --2- 172.21.15.42:0/811847738 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f07a00a1e20 0x7f07a013c890 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f07a006da10 tx=0x7f079801c9d0 comp rx=0 tx=0).stop 2024-09-16T04:02:40.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.163+0000 7f07a7b40640 1 -- 172.21.15.42:0/811847738 shutdown_connections 2024-09-16T04:02:40.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.163+0000 7f07a7b40640 1 --2- 172.21.15.42:0/811847738 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f07a00a1e20 0x7f07a013c890 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:40.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.163+0000 7f07a7b40640 1 --2- 172.21.15.42:0/811847738 >> v2:172.21.15.154:3300/0 conn(0x7f07a006dac0 0x7f07a013a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:40.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.163+0000 7f07a7b40640 1 -- 172.21.15.42:0/811847738 >> 172.21.15.42:0/811847738 conn(0x7f07a00a2420 msgr2=0x7f07a00a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:02:40.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.163+0000 7f07a7b40640 1 -- 172.21.15.42:0/811847738 shutdown_connections 2024-09-16T04:02:40.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.163+0000 7f07a7b40640 1 -- 172.21.15.42:0/811847738 wait complete. 2024-09-16T04:02:40.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.163+0000 7f07a7b40640 10 monclient: build_initial_monmap 2024-09-16T04:02:40.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.163+0000 7f07a7b40640 10 monclient: monmap: 2024-09-16T04:02:40.186 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:02:40.186 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:02:40.187 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:02:40.187 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:02:40.187 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:02:40.187 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:02:40.187 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:02:40.187 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:02:40.187 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:02:40.188 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:02:40.188 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.163+0000 7f07a7b40640 1 librados: starting msgr at 2024-09-16T04:02:40.188 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.163+0000 7f07a7b40640 1 librados: starting objecter 2024-09-16T04:02:40.188 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.163+0000 7f07a7b40640 1 Processor -- start 2024-09-16T04:02:40.188 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.163+0000 7f07a7b40640 1 -- start start 2024-09-16T04:02:40.188 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.163+0000 7f07a7b40640 1 librados: setting wanted keys 2024-09-16T04:02:40.188 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.163+0000 7f07a7b40640 1 librados: calling monclient init 2024-09-16T04:02:40.189 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.163+0000 7f07a7b40640 10 monclient: init 2024-09-16T04:02:40.189 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.163+0000 7f07a7b40640 10 monclient: _reopen_session rank -1 2024-09-16T04:02:40.189 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.163+0000 7f07a7b40640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:02:40.189 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.163+0000 7f07a7b40640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f07a00a1330 0x7f07a01cedb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:40.189 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.163+0000 7f07a7b40640 10 monclient(hunting): picked mon.noname-b con 0x7f07a00a1330 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:02:40.189 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.163+0000 7f07a7b40640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f07a00a1e20 0x7f07a0135340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:40.189 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.163+0000 7f07a7b40640 10 monclient(hunting): picked mon.noname-a con 0x7f07a00a1e20 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:02:40.189 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.163+0000 7f07a7b40640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f07a0135880 0x7f07a009cb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:40.190 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.163+0000 7f07a7b40640 10 monclient(hunting): picked mon.noname-c con 0x7f07a0135880 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:02:40.190 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.163+0000 7f07a7b40640 10 monclient(hunting): start opening mon connection 2024-09-16T04:02:40.190 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.163+0000 7f07a7b40640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f07a013f800 con 0x7f07a00a1e20 2024-09-16T04:02:40.190 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.163+0000 7f07a7b40640 10 monclient(hunting): start opening mon connection 2024-09-16T04:02:40.190 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.163+0000 7f07a7b40640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f07a009ef80 con 0x7f07a00a1330 2024-09-16T04:02:40.190 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.163+0000 7f07a7b40640 10 monclient(hunting): start opening mon connection 2024-09-16T04:02:40.190 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.163+0000 7f07a7b40640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f07a013f980 con 0x7f07a0135880 2024-09-16T04:02:40.190 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.163+0000 7f07a7b40640 10 monclient(hunting): _renew_subs 2024-09-16T04:02:40.191 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.163+0000 7f07a7b40640 10 monclient(hunting): authenticate will time out at 2207.134292s 2024-09-16T04:02:40.191 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.163+0000 7f07a60dd640 1 -- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f07a0135880 msgr2=0x7f07a009cb60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T04:02:40.191 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.163+0000 7f07a60dd640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f07a0135880 0x7f07a009cb60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:02:40.191 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.163+0000 7f07a50db640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f07a00a1e20 0x7f07a0135340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:02:40.191 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.163+0000 7f07a50db640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f07a00a1e20 0x7f07a0135340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:53844/0 (socket says 172.21.15.42:53844) 2024-09-16T04:02:40.191 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.163+0000 7f07a50db640 1 -- 172.21.15.42:0/4241874723 learned_addr learned my addr 172.21.15.42:0/4241874723 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:02:40.192 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.163+0000 7f07a50db640 10 monclient(hunting): get_auth_request con 0x7f07a00a1e20 auth_method 0 2024-09-16T04:02:40.192 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.163+0000 7f07a50db640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:02:40.192 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.163+0000 7f07a50db640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:02:40.192 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.163+0000 7f07a50db640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:02:40.192 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.163+0000 7f07a50db640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:02:40.192 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.163+0000 7f07a50db640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:02:40.192 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.163+0000 7f07a50db640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:02:40.193 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.163+0000 7f07a58dc640 1 --2- 172.21.15.42:0/4241874723 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f07a00a1330 0x7f07a01cedb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:02:40.193 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.163+0000 7f07a58dc640 10 monclient(hunting): get_auth_request con 0x7f07a00a1330 auth_method 0 2024-09-16T04:02:40.193 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.163+0000 7f07a58dc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:02:40.193 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.163+0000 7f07a58dc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:02:40.193 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.163+0000 7f07a58dc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:02:40.193 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.163+0000 7f07a50db640 10 monclient(hunting): handle_auth_done global_id 825579 payload 824 2024-09-16T04:02:40.193 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.163+0000 7f07a50db640 1 -- 172.21.15.42:0/4241874723 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f07a0135880 msgr2=0x7f07a009cb60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:02:40.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.163+0000 7f07a50db640 1 --2- 172.21.15.42:0/4241874723 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f07a0135880 0x7f07a009cb60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:40.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.163+0000 7f07a50db640 1 -- 172.21.15.42:0/4241874723 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f07a00a1330 msgr2=0x7f07a01cedb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:02:40.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.163+0000 7f07a50db640 1 --2- 172.21.15.42:0/4241874723 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f07a00a1330 0x7f07a01cedb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:40.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.163+0000 7f07a50db640 10 monclient: _finish_hunting 0 2024-09-16T04:02:40.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.163+0000 7f07a50db640 1 monclient: found mon.noname-a 2024-09-16T04:02:40.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.163+0000 7f07a50db640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:02:40.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.163+0000 7f07a50db640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:02:40.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.163+0000 7f07a50db640 1 -- 172.21.15.42:0/4241874723 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07a0121cf0 con 0x7f07a00a1e20 2024-09-16T04:02:40.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.163+0000 7f07a50db640 1 --2- 172.21.15.42:0/4241874723 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f07a00a1e20 0x7f07a0135340 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f079000d800 tx=0x7f079000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:02:40.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.163+0000 7f07967fc640 10 client.?.objecter ms_handle_connect 0x7f07a00a1e20 2024-09-16T04:02:40.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.163+0000 7f07967fc640 10 client.?.objecter resend_mon_ops 2024-09-16T04:02:40.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.163+0000 7f07967fc640 1 -- 172.21.15.42:0/4241874723 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f079000b660 con 0x7f07a00a1e20 2024-09-16T04:02:40.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.163+0000 7f07967fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:02:40.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.163+0000 7f07967fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:02:40.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.163+0000 7f07967fc640 10 monclient: dump: 2024-09-16T04:02:40.196 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:02:40.196 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:02:40.196 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:40.196 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:40.197 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:02:40.197 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:02:40.197 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:02:40.197 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:02:40.197 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:02:40.197 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:02:40.197 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.163+0000 7f07967fc640 10 monclient: _finish_auth 0 2024-09-16T04:02:40.197 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.163+0000 7f07967fc640 10 monclient: _check_auth_tickets 2024-09-16T04:02:40.198 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.163+0000 7f07967fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:02:40.198 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.163+0000 7f07967fc640 1 -- 172.21.15.42:0/4241874723 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0790019070 con 0x7f07a00a1e20 2024-09-16T04:02:40.198 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.163+0000 7f07967fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:02:40.198 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.163+0000 7f07967fc640 1 -- 172.21.15.42:0/4241874723 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f079000ba00 con 0x7f07a00a1e20 2024-09-16T04:02:40.198 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.163+0000 7f07967fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:02:40.198 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.163+0000 7f07967fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:02:40.198 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.163+0000 7f07967fc640 10 monclient: dump: 2024-09-16T04:02:40.198 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:02:40.199 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:02:40.199 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:40.199 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:40.199 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:02:40.199 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:02:40.199 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:02:40.199 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:02:40.199 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:02:40.200 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:02:40.200 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.163+0000 7f07a7b40640 5 monclient: authenticate success, global_id 825579 2024-09-16T04:02:40.200 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.163+0000 7f0797fff640 4 set_mon_vals no callback set 2024-09-16T04:02:40.200 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.163+0000 7f07a7b40640 10 monclient: _renew_subs 2024-09-16T04:02:40.200 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.163+0000 7f07a7b40640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:02:40.200 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.163+0000 7f07a7b40640 1 -- 172.21.15.42:0/4241874723 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07a006e140 con 0x7f07a00a1e20 2024-09-16T04:02:40.200 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.163+0000 7f07a7b40640 10 client.825579.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:02:40.201 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.163+0000 7f07a7b40640 10 monclient: _renew_subs 2024-09-16T04:02:40.201 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.163+0000 7f07a7b40640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:02:40.201 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.163+0000 7f07a7b40640 1 -- 172.21.15.42:0/4241874723 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07a0073a30 con 0x7f07a00a1e20 2024-09-16T04:02:40.201 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.163+0000 7f07a7b40640 1 librados: init done 2024-09-16T04:02:40.201 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.167+0000 7f07a7b40640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:02:40.201 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.167+0000 7f07a7b40640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:02:40.201 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.167+0000 7f07a7b40640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:02:40.201 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.167+0000 7f07a7b40640 1 -- 172.21.15.42:0/4241874723 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f07a00a1710 con 0x7f07a00a1e20 2024-09-16T04:02:40.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.167+0000 7f07967fc640 1 -- 172.21.15.42:0/4241874723 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f079000bba0 con 0x7f07a00a1e20 2024-09-16T04:02:40.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.167+0000 7f07967fc640 1 --2- 172.21.15.42:0/4241874723 >> v2:172.21.15.154:6812/254845342 conn(0x7f07840377c0 0x7f0784039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:40.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.167+0000 7f07967fc640 1 -- 172.21.15.42:0/4241874723 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(632..632 src has 451..632) ==== 6367+0+0 (secure 0 0 0) 0x7f0790049630 con 0x7f07a00a1e20 2024-09-16T04:02:40.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.167+0000 7f07967fc640 10 client.825579.objecter ms_dispatch 0x7f07a0000ba0 osd_map(632..632 src has 451..632) 2024-09-16T04:02:40.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.167+0000 7f07967fc640 3 client.825579.objecter handle_osd_map got epochs [632,632] > 0 2024-09-16T04:02:40.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.167+0000 7f07967fc640 3 client.825579.objecter handle_osd_map decoding full epoch 632 2024-09-16T04:02:40.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.167+0000 7f07967fc640 20 client.825579.objecter dump_active .. 0 homeless 2024-09-16T04:02:40.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.167+0000 7f07a58dc640 1 --2- 172.21.15.42:0/4241874723 >> v2:172.21.15.154:6812/254845342 conn(0x7f07840377c0 0x7f0784039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:02:40.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.167+0000 7f07a58dc640 10 monclient: get_auth_request con 0x7f07840377c0 auth_method 0 2024-09-16T04:02:40.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.167+0000 7f07967fc640 10 client.825579.objecter ms_handle_connect 0x7f07840377c0 2024-09-16T04:02:40.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.167+0000 7f07a58dc640 1 --2- 172.21.15.42:0/4241874723 >> v2:172.21.15.154:6812/254845342 conn(0x7f07840377c0 0x7f0784039c80 secure :-1 s=READY pgs=3792 cs=0 l=1 rev1=1 crypto rx=0x7f079c000f70 tx=0x7f079c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:02:40.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.171+0000 7f07967fc640 1 -- 172.21.15.42:0/4241874723 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f079001d580 con 0x7f07a00a1e20 2024-09-16T04:02:40.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.171+0000 7f07967fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:02:40.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.171+0000 7f07967fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:02:40.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.395+0000 7f07a7b40640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:02:40.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.395+0000 7f07a7b40640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:02:40.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.395+0000 7f07a7b40640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:02:40.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.395+0000 7f07a7b40640 1 -- 172.21.15.42:0/4241874723 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f07a0069040 con 0x7f07a00a1e20 2024-09-16T04:02:40.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.395+0000 7f07967fc640 1 -- 172.21.15.42:0/4241874723 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7f079004b6e0 con 0x7f07a00a1e20 2024-09-16T04:02:40.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.395+0000 7f07967fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:02:40.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.395+0000 7f07967fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:02:40.399 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":184,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":9,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:02:40.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.395+0000 7f07a7b40640 10 librados: watch_flush enter 2024-09-16T04:02:40.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.399+0000 7f07a7b40640 10 librados: watch_flush exit 2024-09-16T04:02:40.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.399+0000 7f07a7b40640 20 client.825579.objecter shutdown clearing up homeless session... 2024-09-16T04:02:40.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.399+0000 7f07a7b40640 10 client.825579.objecter successfully canceled tick 2024-09-16T04:02:40.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.399+0000 7f07a7b40640 1 -- 172.21.15.42:0/4241874723 >> v2:172.21.15.154:6812/254845342 conn(0x7f07840377c0 msgr2=0x7f0784039c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:02:40.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.399+0000 7f07a7b40640 1 --2- 172.21.15.42:0/4241874723 >> v2:172.21.15.154:6812/254845342 conn(0x7f07840377c0 0x7f0784039c80 secure :-1 s=READY pgs=3792 cs=0 l=1 rev1=1 crypto rx=0x7f079c000f70 tx=0x7f079c009040 comp rx=0 tx=0).stop 2024-09-16T04:02:40.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.399+0000 7f07a7b40640 10 monclient: shutdown 2024-09-16T04:02:40.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.399+0000 7f07a7b40640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:02:40.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.399+0000 7f07a7b40640 1 -- 172.21.15.42:0/4241874723 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f07a00a1e20 msgr2=0x7f07a0135340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:02:40.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.399+0000 7f07a7b40640 1 --2- 172.21.15.42:0/4241874723 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f07a00a1e20 0x7f07a0135340 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f079000d800 tx=0x7f079000dcc0 comp rx=0 tx=0).stop 2024-09-16T04:02:40.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.399+0000 7f07a7b40640 1 -- 172.21.15.42:0/4241874723 shutdown_connections 2024-09-16T04:02:40.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.399+0000 7f07a7b40640 1 --2- 172.21.15.42:0/4241874723 >> v2:172.21.15.154:6812/254845342 conn(0x7f07840377c0 0x7f0784039c80 secure :-1 s=CLOSED pgs=3792 cs=0 l=1 rev1=1 crypto rx=0x7f079c000f70 tx=0x7f079c009040 comp rx=0 tx=0).stop 2024-09-16T04:02:40.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.399+0000 7f07a7b40640 1 --2- 172.21.15.42:0/4241874723 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f07a0135880 0x7f07a009cb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:40.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.399+0000 7f07a7b40640 1 --2- 172.21.15.42:0/4241874723 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f07a00a1e20 0x7f07a0135340 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:40.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.399+0000 7f07a7b40640 1 --2- 172.21.15.42:0/4241874723 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f07a00a1330 0x7f07a01cedb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:40.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.399+0000 7f07a7b40640 1 -- 172.21.15.42:0/4241874723 >> 172.21.15.42:0/4241874723 conn(0x7f07a00a2420 msgr2=0x7f07a006e690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:02:40.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.399+0000 7f07a7b40640 1 -- 172.21.15.42:0/4241874723 shutdown_connections 2024-09-16T04:02:40.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.399+0000 7f07a7b40640 1 -- 172.21.15.42:0/4241874723 wait complete. 2024-09-16T04:02:40.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:40.399+0000 7f07a7b40640 1 librados: shutdown 2024-09-16T04:02:43.420 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:02:43.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.551+0000 7fdac090e640 10 monclient: get_monmap_and_config 2024-09-16T04:02:43.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.551+0000 7fdac090e640 10 monclient: build_initial_monmap 2024-09-16T04:02:43.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.551+0000 7fdac090e640 10 monclient: monmap: 2024-09-16T04:02:43.555 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:02:43.555 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:02:43.556 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:02:43.553729+0000 2024-09-16T04:02:43.556 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:02:43.553729+0000 2024-09-16T04:02:43.556 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:02:43.556 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:02:43.556 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:02:43.556 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:02:43.556 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:02:43.556 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:02:43.556 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:02:43.556 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:02:43.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.551+0000 7fdac090e640 1 Processor -- start 2024-09-16T04:02:43.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.551+0000 7fdac090e640 1 -- start start 2024-09-16T04:02:43.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.551+0000 7fdac090e640 10 monclient: init 2024-09-16T04:02:43.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.551+0000 7fdac090e640 10 monclient: _reopen_session rank -1 2024-09-16T04:02:43.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.551+0000 7fdac090e640 10 monclient: _add_conns ranks=[4,3,1,2,0] 2024-09-16T04:02:43.560 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.551+0000 7fdac090e640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fdabc0a1330 0x7fdabc0a1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:43.560 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.551+0000 7fdac090e640 10 monclient(hunting): picked mon.noname-c con 0x7fdabc0a1330 addr v2:172.21.15.154:3300/0 2024-09-16T04:02:43.560 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.551+0000 7fdac090e640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fdabc06dac0 0x7fdabc0a1c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:43.560 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.551+0000 7fdac090e640 10 monclient(hunting): picked mon.noname-a con 0x7fdabc06dac0 addr v2:172.21.15.42:3300/0 2024-09-16T04:02:43.560 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.551+0000 7fdac090e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fdabc135100 addr v1:172.21.15.154:6789/0 2024-09-16T04:02:43.560 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.551+0000 7fdac090e640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdabc09c7d0 con 0x7fdabc135100 2024-09-16T04:02:43.560 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.551+0000 7fdac090e640 10 monclient(hunting): start opening mon connection 2024-09-16T04:02:43.560 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.551+0000 7fdac090e640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7fdabc09c090 con 0x7fdabc06dac0 2024-09-16T04:02:43.560 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.551+0000 7fdac090e640 10 monclient(hunting): start opening mon connection 2024-09-16T04:02:43.560 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.551+0000 7fdac090e640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fdabc09f610 con 0x7fdabc0a1330 2024-09-16T04:02:43.560 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.551+0000 7fdac090e640 10 monclient(hunting): _renew_subs 2024-09-16T04:02:43.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.551+0000 7fdac090e640 10 monclient(hunting): authenticate will time out at 2210.523205s 2024-09-16T04:02:43.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.551+0000 7fdab9d9b640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fdabc06dac0 0x7fdabc0a1c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:02:43.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.551+0000 7fdab9d9b640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fdabc06dac0 0x7fdabc0a1c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:53850/0 (socket says 172.21.15.42:53850) 2024-09-16T04:02:43.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.551+0000 7fdab9d9b640 1 -- 172.21.15.42:0/2285361572 learned_addr learned my addr 172.21.15.42:0/2285361572 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:02:43.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.551+0000 7fdab9d9b640 10 monclient(hunting): get_auth_request con 0x7fdabc06dac0 auth_method 0 2024-09-16T04:02:43.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.551+0000 7fdab9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:02:43.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.551+0000 7fdab9d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:02:43.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.551+0000 7fdab9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:02:43.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.551+0000 7fdab9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:02:43.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.551+0000 7fdab9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:02:43.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.551+0000 7fdab9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:02:43.562 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.551+0000 7fdaba59c640 1 --2- 172.21.15.42:0/2285361572 >> v2:172.21.15.154:3300/0 conn(0x7fdabc0a1330 0x7fdabc0a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:02:43.562 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.551+0000 7fdab9d9b640 10 monclient(hunting): handle_auth_done global_id 825582 payload 274 2024-09-16T04:02:43.562 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.551+0000 7fdab9d9b640 1 -- 172.21.15.42:0/2285361572 >> v2:172.21.15.154:3300/0 conn(0x7fdabc0a1330 msgr2=0x7fdabc0a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:02:43.562 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.551+0000 7fdab9d9b640 1 --2- 172.21.15.42:0/2285361572 >> v2:172.21.15.154:3300/0 conn(0x7fdabc0a1330 0x7fdabc0a1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:43.562 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.551+0000 7fdab9d9b640 1 -- 172.21.15.42:0/2285361572 >> v1:172.21.15.154:6789/0 conn(0x7fdabc135100 legacy=0x7fdabc135550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:02:43.562 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.551+0000 7fdab9d9b640 10 monclient: _finish_hunting 0 2024-09-16T04:02:43.562 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.551+0000 7fdab9d9b640 1 monclient: found mon.noname-a 2024-09-16T04:02:43.562 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.551+0000 7fdab9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:02:43.562 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.551+0000 7fdab9d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:02:43.562 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.551+0000 7fdab9d9b640 1 -- 172.21.15.42:0/2285361572 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdabc09f8f0 con 0x7fdabc06dac0 2024-09-16T04:02:43.562 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.551+0000 7fdaba59c640 10 monclient: get_auth_request con 0x7fdabc0a1330 auth_method 0 2024-09-16T04:02:43.562 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.551+0000 7fdaba59c640 1 --2- 172.21.15.42:0/2285361572 >> v2:172.21.15.154:3300/0 conn(0x7fdabc0a1330 0x7fdabc0a1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T04:02:43.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.555+0000 7fdab9d9b640 1 --2- 172.21.15.42:0/2285361572 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdabc06dac0 0x7fdabc0a1c50 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fdabc06add0 tx=0x7fdaac01c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d570c8f66973a62e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:02:43.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.555+0000 7fdab959a640 1 -- 172.21.15.42:0/2285361572 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaac0aa610 con 0x7fdabc06dac0 2024-09-16T04:02:43.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.555+0000 7fdab959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:02:43.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.555+0000 7fdab959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:02:43.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.555+0000 7fdab959a640 10 monclient: dump: 2024-09-16T04:02:43.563 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:02:43.563 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:02:43.563 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:43.563 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:43.563 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:02:43.563 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:02:43.563 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:02:43.564 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:02:43.564 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:02:43.564 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:02:43.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.555+0000 7fdab959a640 10 monclient: _finish_auth 0 2024-09-16T04:02:43.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.555+0000 7fdab959a640 10 monclient: _check_auth_tickets 2024-09-16T04:02:43.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.555+0000 7fdab959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:02:43.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.555+0000 7fdab959a640 1 -- 172.21.15.42:0/2285361572 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdaac0af070 con 0x7fdabc06dac0 2024-09-16T04:02:43.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.555+0000 7fdac090e640 5 monclient: authenticate success, global_id 825582 2024-09-16T04:02:43.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.555+0000 7fdac090e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:02:43.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.555+0000 7fdab959a640 10 monclient: handle_config config(0 keys) 2024-09-16T04:02:43.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.555+0000 7fdac090e640 10 monclient: get_monmap_and_config success 2024-09-16T04:02:43.565 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.555+0000 7fdab959a640 1 -- 172.21.15.42:0/2285361572 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaac0aa9d0 con 0x7fdabc06dac0 2024-09-16T04:02:43.565 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.555+0000 7fdab959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:02:43.565 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.555+0000 7fdac090e640 4 set_mon_vals no callback set 2024-09-16T04:02:43.565 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.555+0000 7fdab959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:02:43.565 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.555+0000 7fdab959a640 10 monclient: dump: 2024-09-16T04:02:43.565 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:02:43.565 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:02:43.565 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:43.565 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:43.565 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:02:43.565 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:02:43.565 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:02:43.566 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:02:43.566 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:02:43.566 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:02:43.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.555+0000 7fdac090e640 10 monclient: shutdown 2024-09-16T04:02:43.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.555+0000 7fdac090e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:02:43.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.555+0000 7fdac090e640 1 -- 172.21.15.42:0/2285361572 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdabc06dac0 msgr2=0x7fdabc0a1c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:02:43.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.555+0000 7fdac090e640 1 --2- 172.21.15.42:0/2285361572 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdabc06dac0 0x7fdabc0a1c50 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fdabc06add0 tx=0x7fdaac01c490 comp rx=0 tx=0).stop 2024-09-16T04:02:43.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.555+0000 7fdac090e640 1 -- 172.21.15.42:0/2285361572 shutdown_connections 2024-09-16T04:02:43.572 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.555+0000 7fdac090e640 1 --2- 172.21.15.42:0/2285361572 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdabc06dac0 0x7fdabc0a1c50 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:43.572 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.555+0000 7fdac090e640 1 --2- 172.21.15.42:0/2285361572 >> v2:172.21.15.154:3300/0 conn(0x7fdabc0a1330 0x7fdabc0a1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:43.572 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.555+0000 7fdac090e640 1 -- 172.21.15.42:0/2285361572 >> 172.21.15.42:0/2285361572 conn(0x7fdabc0a2420 msgr2=0x7fdabc0a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:02:43.572 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.555+0000 7fdac090e640 1 -- 172.21.15.42:0/2285361572 shutdown_connections 2024-09-16T04:02:43.572 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.555+0000 7fdac090e640 1 -- 172.21.15.42:0/2285361572 wait complete. 2024-09-16T04:02:43.572 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.555+0000 7fdac090e640 10 monclient: build_initial_monmap 2024-09-16T04:02:43.572 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.555+0000 7fdac090e640 10 monclient: monmap: 2024-09-16T04:02:43.573 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:02:43.574 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:02:43.574 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:02:43.574 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:02:43.574 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:02:43.574 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:02:43.574 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:02:43.574 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:02:43.574 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:02:43.574 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:02:43.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.555+0000 7fdac090e640 1 librados: starting msgr at 2024-09-16T04:02:43.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.555+0000 7fdac090e640 1 librados: starting objecter 2024-09-16T04:02:43.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.555+0000 7fdac090e640 1 Processor -- start 2024-09-16T04:02:43.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.555+0000 7fdac090e640 1 -- start start 2024-09-16T04:02:43.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.555+0000 7fdac090e640 1 librados: setting wanted keys 2024-09-16T04:02:43.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.555+0000 7fdac090e640 1 librados: calling monclient init 2024-09-16T04:02:43.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.555+0000 7fdac090e640 10 monclient: init 2024-09-16T04:02:43.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.559+0000 7fdac090e640 10 monclient: _reopen_session rank -1 2024-09-16T04:02:43.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.559+0000 7fdac090e640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:02:43.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.559+0000 7fdac090e640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fdabc0a1330 0x7fdabc13c100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:43.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.559+0000 7fdac090e640 10 monclient(hunting): picked mon.noname-c con 0x7fdabc0a1330 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:02:43.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.559+0000 7fdac090e640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdabc135100 0x7fdabc134530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:43.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.559+0000 7fdac090e640 10 monclient(hunting): picked mon.noname-a con 0x7fdabc135100 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:02:43.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.559+0000 7fdac090e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdabc134a70 0x7fdabc09cb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:43.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.559+0000 7fdac090e640 10 monclient(hunting): picked mon.noname-b con 0x7fdabc134a70 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:02:43.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.559+0000 7fdac090e640 10 monclient(hunting): start opening mon connection 2024-09-16T04:02:43.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.559+0000 7fdac090e640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fdabc13fb80 con 0x7fdabc135100 2024-09-16T04:02:43.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.559+0000 7fdac090e640 10 monclient(hunting): start opening mon connection 2024-09-16T04:02:43.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.559+0000 7fdac090e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fdabc09ef80 con 0x7fdabc134a70 2024-09-16T04:02:43.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.559+0000 7fdac090e640 10 monclient(hunting): start opening mon connection 2024-09-16T04:02:43.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.559+0000 7fdac090e640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fdabc13fd00 con 0x7fdabc0a1330 2024-09-16T04:02:43.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.559+0000 7fdac090e640 10 monclient(hunting): _renew_subs 2024-09-16T04:02:43.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.559+0000 7fdac090e640 10 monclient(hunting): authenticate will time out at 2210.528193s 2024-09-16T04:02:43.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.559+0000 7fdaba59c640 1 -- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fdabc0a1330 msgr2=0x7fdabc13c100 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T04:02:43.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.559+0000 7fdaba59c640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fdabc0a1330 0x7fdabc13c100 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:02:43.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.559+0000 7fdab9d9b640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdabc135100 0x7fdabc134530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:02:43.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.559+0000 7fdab9d9b640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdabc135100 0x7fdabc134530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:53862/0 (socket says 172.21.15.42:53862) 2024-09-16T04:02:43.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.559+0000 7fdab9d9b640 1 -- 172.21.15.42:0/783258411 learned_addr learned my addr 172.21.15.42:0/783258411 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:02:43.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.559+0000 7fdab9d9b640 10 monclient(hunting): get_auth_request con 0x7fdabc135100 auth_method 0 2024-09-16T04:02:43.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.559+0000 7fdab9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:02:43.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.559+0000 7fdab9d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:02:43.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.559+0000 7fdab9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:02:43.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.559+0000 7fdab9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:02:43.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.559+0000 7fdab9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:02:43.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.559+0000 7fdab9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:02:43.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.559+0000 7fdabad9d640 1 --2- 172.21.15.42:0/783258411 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdabc134a70 0x7fdabc09cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:02:43.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.559+0000 7fdabad9d640 10 monclient(hunting): get_auth_request con 0x7fdabc134a70 auth_method 0 2024-09-16T04:02:43.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.559+0000 7fdabad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:02:43.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.559+0000 7fdabad9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:02:43.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.559+0000 7fdabad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:02:43.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.559+0000 7fdab9d9b640 10 monclient(hunting): handle_auth_done global_id 825585 payload 824 2024-09-16T04:02:43.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.559+0000 7fdab9d9b640 1 -- 172.21.15.42:0/783258411 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fdabc0a1330 msgr2=0x7fdabc13c100 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:02:43.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.559+0000 7fdab9d9b640 1 --2- 172.21.15.42:0/783258411 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fdabc0a1330 0x7fdabc13c100 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:43.578 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.559+0000 7fdab9d9b640 1 -- 172.21.15.42:0/783258411 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdabc134a70 msgr2=0x7fdabc09cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:02:43.578 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.559+0000 7fdab9d9b640 1 --2- 172.21.15.42:0/783258411 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdabc134a70 0x7fdabc09cb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:43.578 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.559+0000 7fdab9d9b640 10 monclient: _finish_hunting 0 2024-09-16T04:02:43.578 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.559+0000 7fdab9d9b640 1 monclient: found mon.noname-a 2024-09-16T04:02:43.578 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.559+0000 7fdab9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:02:43.578 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.559+0000 7fdab9d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:02:43.578 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.559+0000 7fdab9d9b640 1 -- 172.21.15.42:0/783258411 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdabc1cf690 con 0x7fdabc135100 2024-09-16T04:02:43.578 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.559+0000 7fdab9d9b640 1 --2- 172.21.15.42:0/783258411 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdabc135100 0x7fdabc134530 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7fdaac0ac040 tx=0x7fdaac01acd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:02:43.578 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.559+0000 7fdab2ffd640 10 client.?.objecter ms_handle_connect 0x7fdabc135100 2024-09-16T04:02:43.578 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.559+0000 7fdab2ffd640 10 client.?.objecter resend_mon_ops 2024-09-16T04:02:43.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.559+0000 7fdab2ffd640 1 -- 172.21.15.42:0/783258411 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaac0b3600 con 0x7fdabc135100 2024-09-16T04:02:43.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.559+0000 7fdab2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:02:43.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.559+0000 7fdab2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:02:43.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.559+0000 7fdab2ffd640 10 monclient: dump: 2024-09-16T04:02:43.579 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:02:43.579 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:02:43.579 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:43.579 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:43.579 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:02:43.579 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:02:43.579 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:02:43.579 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:02:43.579 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:02:43.580 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:02:43.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.559+0000 7fdab2ffd640 10 monclient: _finish_auth 0 2024-09-16T04:02:43.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.559+0000 7fdab2ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:02:43.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.559+0000 7fdab2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:02:43.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.559+0000 7fdab2ffd640 1 -- 172.21.15.42:0/783258411 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdaac0b37c0 con 0x7fdabc135100 2024-09-16T04:02:43.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.559+0000 7fdab2ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:02:43.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.559+0000 7fdab2ffd640 1 -- 172.21.15.42:0/783258411 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaac0b3bb0 con 0x7fdabc135100 2024-09-16T04:02:43.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.559+0000 7fdac090e640 5 monclient: authenticate success, global_id 825585 2024-09-16T04:02:43.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.559+0000 7fdac090e640 10 monclient: _renew_subs 2024-09-16T04:02:43.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.559+0000 7fdac090e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:02:43.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.559+0000 7fdac090e640 1 -- 172.21.15.42:0/783258411 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdabc0a4fd0 con 0x7fdabc135100 2024-09-16T04:02:43.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.559+0000 7fdab8d99640 4 set_mon_vals no callback set 2024-09-16T04:02:43.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.559+0000 7fdab2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:02:43.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.559+0000 7fdab2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:02:43.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.559+0000 7fdab2ffd640 10 monclient: dump: 2024-09-16T04:02:43.581 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:02:43.581 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:02:43.581 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:43.581 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:43.581 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:02:43.581 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:02:43.581 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:02:43.581 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:02:43.581 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:02:43.581 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:02:43.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.559+0000 7fdac090e640 10 client.825585.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:02:43.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.559+0000 7fdac090e640 10 monclient: _renew_subs 2024-09-16T04:02:43.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.559+0000 7fdac090e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:02:43.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.559+0000 7fdac090e640 1 -- 172.21.15.42:0/783258411 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdabc073a30 con 0x7fdabc135100 2024-09-16T04:02:43.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.559+0000 7fdac090e640 1 librados: init done 2024-09-16T04:02:43.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.559+0000 7fdac090e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:02:43.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.559+0000 7fdac090e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:02:43.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.559+0000 7fdac090e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:02:43.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.559+0000 7fdac090e640 1 -- 172.21.15.42:0/783258411 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdabc135550 con 0x7fdabc135100 2024-09-16T04:02:43.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.559+0000 7fdab2ffd640 1 -- 172.21.15.42:0/783258411 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fdaac0b2380 con 0x7fdabc135100 2024-09-16T04:02:43.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.559+0000 7fdab2ffd640 1 --2- 172.21.15.42:0/783258411 >> v2:172.21.15.154:6812/254845342 conn(0x7fda9c0377c0 0x7fda9c039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:43.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.559+0000 7fdab2ffd640 1 -- 172.21.15.42:0/783258411 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(632..632 src has 451..632) ==== 6367+0+0 (secure 0 0 0) 0x7fdaac0e3840 con 0x7fdabc135100 2024-09-16T04:02:43.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.559+0000 7fdab2ffd640 10 client.825585.objecter ms_dispatch 0x7fdabc000ba0 osd_map(632..632 src has 451..632) 2024-09-16T04:02:43.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.559+0000 7fdab2ffd640 3 client.825585.objecter handle_osd_map got epochs [632,632] > 0 2024-09-16T04:02:43.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.559+0000 7fdab2ffd640 3 client.825585.objecter handle_osd_map decoding full epoch 632 2024-09-16T04:02:43.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.559+0000 7fdab2ffd640 20 client.825585.objecter dump_active .. 0 homeless 2024-09-16T04:02:43.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.559+0000 7fdaba59c640 1 --2- 172.21.15.42:0/783258411 >> v2:172.21.15.154:6812/254845342 conn(0x7fda9c0377c0 0x7fda9c039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:02:43.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.559+0000 7fdaba59c640 10 monclient: get_auth_request con 0x7fda9c0377c0 auth_method 0 2024-09-16T04:02:43.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.563+0000 7fdaba59c640 1 --2- 172.21.15.42:0/783258411 >> v2:172.21.15.154:6812/254845342 conn(0x7fda9c0377c0 0x7fda9c039c80 secure :-1 s=READY pgs=3793 cs=0 l=1 rev1=1 crypto rx=0x7fdaa4006fd0 tx=0x7fdaa4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:02:43.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.563+0000 7fdab2ffd640 10 client.825585.objecter ms_handle_connect 0x7fda9c0377c0 2024-09-16T04:02:43.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.563+0000 7fdab2ffd640 1 -- 172.21.15.42:0/783258411 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7fdaac0c3060 con 0x7fdabc135100 2024-09-16T04:02:43.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.563+0000 7fdab2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:02:43.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.563+0000 7fdab2ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:02:43.800 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.795+0000 7fdac090e640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:02:43.800 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.795+0000 7fdac090e640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:02:43.800 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.795+0000 7fdac090e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:02:43.800 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.795+0000 7fdac090e640 1 -- 172.21.15.42:0/783258411 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fdabc0741a0 con 0x7fdabc135100 2024-09-16T04:02:43.800 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.795+0000 7fdab2ffd640 1 -- 172.21.15.42:0/783258411 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1562 (secure 0 0 0) 0x7fdabc1cf690 con 0x7fdabc135100 2024-09-16T04:02:43.800 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.795+0000 7fdab2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:02:43.800 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.795+0000 7fdab2ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:02:43.801 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":184,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:02:43.803 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.799+0000 7fdac090e640 10 librados: watch_flush enter 2024-09-16T04:02:43.803 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.799+0000 7fdac090e640 10 librados: watch_flush exit 2024-09-16T04:02:43.803 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.799+0000 7fdac090e640 20 client.825585.objecter shutdown clearing up homeless session... 2024-09-16T04:02:43.803 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.799+0000 7fdac090e640 10 client.825585.objecter successfully canceled tick 2024-09-16T04:02:43.803 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.799+0000 7fdac090e640 1 -- 172.21.15.42:0/783258411 >> v2:172.21.15.154:6812/254845342 conn(0x7fda9c0377c0 msgr2=0x7fda9c039c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:02:43.803 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.799+0000 7fdac090e640 1 --2- 172.21.15.42:0/783258411 >> v2:172.21.15.154:6812/254845342 conn(0x7fda9c0377c0 0x7fda9c039c80 secure :-1 s=READY pgs=3793 cs=0 l=1 rev1=1 crypto rx=0x7fdaa4006fd0 tx=0x7fdaa4008040 comp rx=0 tx=0).stop 2024-09-16T04:02:43.803 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.799+0000 7fdac090e640 10 monclient: shutdown 2024-09-16T04:02:43.803 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.799+0000 7fdac090e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:02:43.803 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.799+0000 7fdac090e640 1 -- 172.21.15.42:0/783258411 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdabc135100 msgr2=0x7fdabc134530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:02:43.803 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.799+0000 7fdac090e640 1 --2- 172.21.15.42:0/783258411 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdabc135100 0x7fdabc134530 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7fdaac0ac040 tx=0x7fdaac01acd0 comp rx=0 tx=0).stop 2024-09-16T04:02:43.803 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.799+0000 7fdac090e640 1 -- 172.21.15.42:0/783258411 shutdown_connections 2024-09-16T04:02:43.803 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.799+0000 7fdac090e640 1 --2- 172.21.15.42:0/783258411 >> v2:172.21.15.154:6812/254845342 conn(0x7fda9c0377c0 0x7fda9c039c80 unknown :-1 s=CLOSED pgs=3793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:43.803 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.799+0000 7fdac090e640 1 --2- 172.21.15.42:0/783258411 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdabc134a70 0x7fdabc09cb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:43.804 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.799+0000 7fdac090e640 1 --2- 172.21.15.42:0/783258411 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdabc135100 0x7fdabc134530 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:43.804 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.799+0000 7fdac090e640 1 --2- 172.21.15.42:0/783258411 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fdabc0a1330 0x7fdabc13c100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:43.804 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.799+0000 7fdac090e640 1 -- 172.21.15.42:0/783258411 >> 172.21.15.42:0/783258411 conn(0x7fdabc0a2420 msgr2=0x7fdabc0a3720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:02:43.804 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.799+0000 7fdac090e640 1 -- 172.21.15.42:0/783258411 shutdown_connections 2024-09-16T04:02:43.804 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.799+0000 7fdac090e640 1 -- 172.21.15.42:0/783258411 wait complete. 2024-09-16T04:02:43.804 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:43.799+0000 7fdac090e640 1 librados: shutdown 2024-09-16T04:02:46.819 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:02:46.952 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:46.947+0000 7efc4c8cc640 10 monclient: get_monmap_and_config 2024-09-16T04:02:46.952 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:46.947+0000 7efc4c8cc640 10 monclient: build_initial_monmap 2024-09-16T04:02:46.952 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:46.947+0000 7efc4c8cc640 10 monclient: monmap: 2024-09-16T04:02:46.952 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:02:46.952 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:02:46.952 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:02:46.950451+0000 2024-09-16T04:02:46.952 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:02:46.950451+0000 2024-09-16T04:02:46.952 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:02:46.952 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:02:46.952 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:02:46.952 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:02:46.953 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:02:46.953 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:02:46.953 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:02:46.953 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:02:46.954 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:46.947+0000 7efc4c8cc640 1 Processor -- start 2024-09-16T04:02:46.955 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:46.947+0000 7efc4c8cc640 1 -- start start 2024-09-16T04:02:46.955 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:46.947+0000 7efc4c8cc640 10 monclient: init 2024-09-16T04:02:46.955 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:46.947+0000 7efc4c8cc640 10 monclient: _reopen_session rank -1 2024-09-16T04:02:46.955 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:46.947+0000 7efc4c8cc640 10 monclient: _add_conns ranks=[4,0,2,3,1] 2024-09-16T04:02:46.955 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:46.947+0000 7efc4c8cc640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7efc480a39b0 0x7efc48133b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:46.955 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:46.947+0000 7efc4c8cc640 10 monclient(hunting): picked mon.noname-c con 0x7efc480a39b0 addr v2:172.21.15.154:3300/0 2024-09-16T04:02:46.955 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:46.947+0000 7efc4c8cc640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7efc480a3d90 addr v1:172.21.15.42:6789/0 2024-09-16T04:02:46.955 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:46.947+0000 7efc4c8cc640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7efc48137710 0x7efc4813f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:46.955 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:46.947+0000 7efc4c8cc640 10 monclient(hunting): picked mon.noname-b con 0x7efc48137710 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:02:46.955 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:46.947+0000 7efc4c8cc640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efc4809c140 con 0x7efc480a3d90 2024-09-16T04:02:46.955 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:46.951+0000 7efc4c8cc640 10 monclient(hunting): start opening mon connection 2024-09-16T04:02:46.955 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:46.951+0000 7efc4c8cc640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7efc4809c420 con 0x7efc480a39b0 2024-09-16T04:02:46.955 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:46.951+0000 7efc4c8cc640 10 monclient(hunting): start opening mon connection 2024-09-16T04:02:46.956 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:46.951+0000 7efc4c8cc640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7efc4806dd60 con 0x7efc48137710 2024-09-16T04:02:46.956 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:46.951+0000 7efc4c8cc640 10 monclient(hunting): _renew_subs 2024-09-16T04:02:46.956 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:46.951+0000 7efc4c8cc640 10 monclient(hunting): authenticate will time out at 2213.920034s 2024-09-16T04:02:46.956 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:46.951+0000 7efc46d9d640 1 -- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7efc48137710 msgr2=0x7efc4813f3a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T04:02:46.956 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:46.951+0000 7efc46d9d640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7efc48137710 0x7efc4813f3a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:02:46.956 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:46.951+0000 7efc45d9b640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7efc480a3d90 0x7efc48137000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:42476/0 (socket says 172.21.15.42:42476) 2024-09-16T04:02:46.956 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:46.951+0000 7efc45d9b640 1 -- 172.21.15.42:0/525815344 learned_addr learned my addr 172.21.15.42:0/525815344 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T04:02:46.956 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:46.951+0000 7efc4659c640 1 --2- 172.21.15.42:0/525815344 >> v2:172.21.15.154:3300/0 conn(0x7efc480a39b0 0x7efc48133b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:02:46.957 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:46.951+0000 7efc4659c640 10 monclient(hunting): get_auth_request con 0x7efc480a39b0 auth_method 0 2024-09-16T04:02:46.957 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:46.951+0000 7efc4659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:02:46.957 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:46.951+0000 7efc4659c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:02:46.957 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:46.951+0000 7efc4659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:02:46.957 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:46.951+0000 7efc4659c640 1 -- 172.21.15.42:0/525815344 >> v2:172.21.15.154:3300/0 conn(0x7efc480a39b0 msgr2=0x7efc48133b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:02:46.957 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:46.951+0000 7efc4659c640 1 -- 172.21.15.42:0/525815344 >> v2:172.21.15.154:3300/0 conn(0x7efc480a39b0 msgr2=0x7efc48133b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:02:46.957 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:46.951+0000 7efc4659c640 1 --2- 172.21.15.42:0/525815344 >> v2:172.21.15.154:3300/0 conn(0x7efc480a39b0 0x7efc48133b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:02:46.957 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:46.951+0000 7efc4659c640 1 --2- 172.21.15.42:0/525815344 >> v2:172.21.15.154:3300/0 conn(0x7efc480a39b0 0x7efc48133b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:02:47.157 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:47.155+0000 7efc46d9d640 1 -- 172.21.15.42:0/525815344 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7efc48137710 msgr2=0x7efc4813f3a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T04:02:47.157 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:47.155+0000 7efc46d9d640 1 --2- 172.21.15.42:0/525815344 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7efc48137710 0x7efc4813f3a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T04:02:47.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:47.155+0000 7efc4659c640 1 --2- 172.21.15.42:0/525815344 >> v2:172.21.15.154:3300/0 conn(0x7efc480a39b0 0x7efc48133b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:02:47.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:47.155+0000 7efc4659c640 10 monclient(hunting): get_auth_request con 0x7efc480a39b0 auth_method 0 2024-09-16T04:02:47.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:47.155+0000 7efc4659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:02:47.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:47.155+0000 7efc4659c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:02:47.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:47.155+0000 7efc4659c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:02:47.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:47.155+0000 7efc4659c640 1 -- 172.21.15.42:0/525815344 >> v2:172.21.15.154:3300/0 conn(0x7efc480a39b0 msgr2=0x7efc48133b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T04:02:47.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:47.155+0000 7efc4659c640 1 -- 172.21.15.42:0/525815344 >> v2:172.21.15.154:3300/0 conn(0x7efc480a39b0 msgr2=0x7efc48133b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:02:47.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:47.155+0000 7efc4659c640 1 --2- 172.21.15.42:0/525815344 >> v2:172.21.15.154:3300/0 conn(0x7efc480a39b0 0x7efc48133b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:02:47.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:47.155+0000 7efc4659c640 1 --2- 172.21.15.42:0/525815344 >> v2:172.21.15.154:3300/0 conn(0x7efc480a39b0 0x7efc48133b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T04:02:47.562 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:47.559+0000 7efc46d9d640 1 -- 172.21.15.42:0/525815344 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7efc48137710 msgr2=0x7efc4813f3a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T04:02:47.562 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:47.559+0000 7efc46d9d640 1 --2- 172.21.15.42:0/525815344 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7efc48137710 0x7efc4813f3a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T04:02:47.562 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:47.559+0000 7efc4659c640 1 --2- 172.21.15.42:0/525815344 >> v2:172.21.15.154:3300/0 conn(0x7efc480a39b0 0x7efc48133b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:02:47.562 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:47.559+0000 7efc4659c640 10 monclient(hunting): get_auth_request con 0x7efc480a39b0 auth_method 0 2024-09-16T04:02:47.562 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:47.559+0000 7efc4659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:02:47.562 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:47.559+0000 7efc4659c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:02:47.562 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:47.559+0000 7efc4659c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:02:47.562 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:47.559+0000 7efc4659c640 1 -- 172.21.15.42:0/525815344 >> v2:172.21.15.154:3300/0 conn(0x7efc480a39b0 msgr2=0x7efc48133b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T04:02:47.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:47.559+0000 7efc4659c640 1 -- 172.21.15.42:0/525815344 >> v2:172.21.15.154:3300/0 conn(0x7efc480a39b0 msgr2=0x7efc48133b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:02:47.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:47.559+0000 7efc4659c640 1 --2- 172.21.15.42:0/525815344 >> v2:172.21.15.154:3300/0 conn(0x7efc480a39b0 0x7efc48133b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:02:47.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:47.559+0000 7efc4659c640 1 --2- 172.21.15.42:0/525815344 >> v2:172.21.15.154:3300/0 conn(0x7efc480a39b0 0x7efc48133b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T04:02:48.365 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:48.363+0000 7efc46d9d640 1 -- 172.21.15.42:0/525815344 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7efc48137710 msgr2=0x7efc4813f3a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T04:02:48.365 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:48.363+0000 7efc46d9d640 1 --2- 172.21.15.42:0/525815344 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7efc48137710 0x7efc4813f3a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T04:02:48.366 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:48.363+0000 7efc4659c640 1 --2- 172.21.15.42:0/525815344 >> v2:172.21.15.154:3300/0 conn(0x7efc480a39b0 0x7efc48133b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:02:48.366 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:48.363+0000 7efc4659c640 10 monclient(hunting): get_auth_request con 0x7efc480a39b0 auth_method 0 2024-09-16T04:02:48.366 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:48.363+0000 7efc4659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:02:48.366 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:48.363+0000 7efc4659c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:02:48.366 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:48.363+0000 7efc4659c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:02:48.367 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:48.363+0000 7efc4659c640 1 -- 172.21.15.42:0/525815344 >> v2:172.21.15.154:3300/0 conn(0x7efc480a39b0 msgr2=0x7efc48133b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T04:02:48.367 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:48.363+0000 7efc4659c640 1 -- 172.21.15.42:0/525815344 >> v2:172.21.15.154:3300/0 conn(0x7efc480a39b0 msgr2=0x7efc48133b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:02:48.367 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:48.363+0000 7efc4659c640 1 --2- 172.21.15.42:0/525815344 >> v2:172.21.15.154:3300/0 conn(0x7efc480a39b0 0x7efc48133b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:02:48.367 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:48.363+0000 7efc4659c640 1 --2- 172.21.15.42:0/525815344 >> v2:172.21.15.154:3300/0 conn(0x7efc480a39b0 0x7efc48133b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T04:02:49.954 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:49.951+0000 7efc3ffff640 10 monclient(hunting): tick 2024-09-16T04:02:49.954 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:49.951+0000 7efc3ffff640 10 monclient(hunting): _check_auth_tickets 2024-09-16T04:02:49.954 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:49.951+0000 7efc3ffff640 1 monclient(hunting): continuing hunt 2024-09-16T04:02:49.954 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:49.951+0000 7efc3ffff640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T04:02:49.954 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:49.951+0000 7efc3ffff640 1 -- 172.21.15.42:0/525815344 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7efc48137710 msgr2=0x7efc4813f3a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:02:49.955 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:49.951+0000 7efc3ffff640 1 --2- 172.21.15.42:0/525815344 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7efc48137710 0x7efc4813f3a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:49.955 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:49.951+0000 7efc3ffff640 1 -- 172.21.15.42:0/525815344 >> v2:172.21.15.154:3300/0 conn(0x7efc480a39b0 msgr2=0x7efc48133b80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:02:49.955 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:49.951+0000 7efc3ffff640 1 --2- 172.21.15.42:0/525815344 >> v2:172.21.15.154:3300/0 conn(0x7efc480a39b0 0x7efc48133b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:49.955 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:49.951+0000 7efc3ffff640 1 -- 172.21.15.42:0/525815344 >> v1:172.21.15.42:6789/0 conn(0x7efc480a3d90 legacy=0x7efc48137000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:02:49.955 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:49.951+0000 7efc3ffff640 10 monclient: _add_conns ranks=[1,3] 2024-09-16T04:02:49.955 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:49.951+0000 7efc3ffff640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7efc280036d0 addr v1:172.21.15.154:6789/0 2024-09-16T04:02:49.955 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:49.951+0000 7efc3ffff640 1 --2- 172.21.15.42:0/525815344 >> v2:172.21.15.42:3300/0 conn(0x7efc28007420 0x7efc280098e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:49.955 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:49.951+0000 7efc3ffff640 10 monclient(hunting): picked mon.noname-a con 0x7efc28007420 addr v2:172.21.15.42:3300/0 2024-09-16T04:02:49.955 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:49.951+0000 7efc3ffff640 1 -- 172.21.15.42:0/525815344 --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efc480a5e20 con 0x7efc280036d0 2024-09-16T04:02:49.955 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:49.951+0000 7efc3ffff640 10 monclient(hunting): start opening mon connection 2024-09-16T04:02:49.955 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:49.951+0000 7efc3ffff640 1 -- 172.21.15.42:0/525815344 --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7efc4809c420 con 0x7efc28007420 2024-09-16T04:02:49.955 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:49.951+0000 7efc3ffff640 10 monclient(hunting): _renew_subs 2024-09-16T04:02:49.955 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:49.951+0000 7efc45d9b640 1 --2- 172.21.15.42:0/525815344 >> v2:172.21.15.42:3300/0 conn(0x7efc28007420 0x7efc280098e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:02:49.956 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:49.951+0000 7efc45d9b640 10 monclient(hunting): get_auth_request con 0x7efc28007420 auth_method 0 2024-09-16T04:02:49.956 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:49.951+0000 7efc45d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:02:49.956 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:49.951+0000 7efc45d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:02:49.956 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:49.951+0000 7efc45d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:02:49.956 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:49.951+0000 7efc45d9b640 1 -- 172.21.15.42:0/525815344 >> v2:172.21.15.42:3300/0 conn(0x7efc28007420 msgr2=0x7efc280098e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:02:49.956 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:49.951+0000 7efc45d9b640 1 -- 172.21.15.42:0/525815344 >> v2:172.21.15.42:3300/0 conn(0x7efc28007420 msgr2=0x7efc280098e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:02:49.956 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:49.951+0000 7efc45d9b640 1 --2- 172.21.15.42:0/525815344 >> v2:172.21.15.42:3300/0 conn(0x7efc28007420 0x7efc280098e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:02:49.956 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:49.951+0000 7efc45d9b640 1 --2- 172.21.15.42:0/525815344 >> v2:172.21.15.42:3300/0 conn(0x7efc28007420 0x7efc280098e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:02:50.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.155+0000 7efc45d9b640 1 --2- 172.21.15.42:0/525815344 >> v2:172.21.15.42:3300/0 conn(0x7efc28007420 0x7efc280098e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:02:50.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.155+0000 7efc45d9b640 10 monclient(hunting): get_auth_request con 0x7efc28007420 auth_method 0 2024-09-16T04:02:50.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.155+0000 7efc45d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:02:50.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.155+0000 7efc45d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:02:50.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.155+0000 7efc45d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:02:50.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.155+0000 7efc45d9b640 1 -- 172.21.15.42:0/525815344 >> v2:172.21.15.42:3300/0 conn(0x7efc28007420 msgr2=0x7efc280098e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:02:50.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.155+0000 7efc45d9b640 1 -- 172.21.15.42:0/525815344 >> v2:172.21.15.42:3300/0 conn(0x7efc28007420 msgr2=0x7efc280098e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:02:50.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.155+0000 7efc45d9b640 1 --2- 172.21.15.42:0/525815344 >> v2:172.21.15.42:3300/0 conn(0x7efc28007420 0x7efc280098e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:02:50.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.155+0000 7efc45d9b640 1 --2- 172.21.15.42:0/525815344 >> v2:172.21.15.42:3300/0 conn(0x7efc28007420 0x7efc280098e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T04:02:50.562 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.559+0000 7efc45d9b640 1 --2- 172.21.15.42:0/525815344 >> v2:172.21.15.42:3300/0 conn(0x7efc28007420 0x7efc280098e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:02:50.562 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.559+0000 7efc45d9b640 10 monclient(hunting): get_auth_request con 0x7efc28007420 auth_method 0 2024-09-16T04:02:50.562 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.559+0000 7efc45d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:02:50.562 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.559+0000 7efc45d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:02:50.562 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.559+0000 7efc45d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:02:50.562 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.559+0000 7efc45d9b640 1 -- 172.21.15.42:0/525815344 >> v2:172.21.15.42:3300/0 conn(0x7efc28007420 msgr2=0x7efc280098e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:02:50.562 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.559+0000 7efc45d9b640 1 -- 172.21.15.42:0/525815344 >> v2:172.21.15.42:3300/0 conn(0x7efc28007420 msgr2=0x7efc280098e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:02:50.562 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.559+0000 7efc45d9b640 1 --2- 172.21.15.42:0/525815344 >> v2:172.21.15.42:3300/0 conn(0x7efc28007420 0x7efc280098e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:02:50.562 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.559+0000 7efc45d9b640 1 --2- 172.21.15.42:0/525815344 >> v2:172.21.15.42:3300/0 conn(0x7efc28007420 0x7efc280098e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T04:02:50.768 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.763+0000 7efc4559a640 1 -- 172.21.15.42:0/525815344 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1184122506 0 0) 0x7efc480a5e20 con 0x7efc280036d0 2024-09-16T04:02:50.768 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.763+0000 7efc4559a640 10 monclient(hunting): _negotiate 2024-09-16T04:02:50.768 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.763+0000 7efc4559a640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:02:50.768 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.763+0000 7efc4559a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:02:50.768 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.763+0000 7efc4559a640 10 monclient(hunting): my global_id is 904657 2024-09-16T04:02:50.768 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.763+0000 7efc4559a640 1 -- 172.21.15.42:0/525815344 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efc34003530 con 0x7efc280036d0 2024-09-16T04:02:50.782 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.775+0000 7efc4559a640 1 -- 172.21.15.42:0/525815344 <== mon.1 v1:172.21.15.154:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3871782261 0 0) 0x7efc34003530 con 0x7efc280036d0 2024-09-16T04:02:50.782 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.775+0000 7efc4559a640 1 -- 172.21.15.42:0/525815344 >> v2:172.21.15.42:3300/0 conn(0x7efc28007420 msgr2=0x7efc280098e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:02:50.782 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.775+0000 7efc4559a640 1 --2- 172.21.15.42:0/525815344 >> v2:172.21.15.42:3300/0 conn(0x7efc28007420 0x7efc280098e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:50.782 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.775+0000 7efc4559a640 10 monclient: _finish_hunting 0 2024-09-16T04:02:50.782 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.775+0000 7efc4559a640 1 monclient: found mon.noname-c-legacy 2024-09-16T04:02:50.782 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.775+0000 7efc4559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:02:50.782 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.775+0000 7efc4559a640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.154:6789/0 2024-09-16T04:02:50.782 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.775+0000 7efc4559a640 1 -- 172.21.15.42:0/525815344 --> v1:172.21.15.154:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc2800aff0 con 0x7efc280036d0 2024-09-16T04:02:50.782 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.775+0000 7efc4559a640 10 monclient: _finish_auth 0 2024-09-16T04:02:50.782 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.775+0000 7efc4559a640 10 monclient: _check_auth_tickets 2024-09-16T04:02:50.782 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.775+0000 7efc4559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:02:50.782 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.775+0000 7efc4559a640 1 -- 172.21.15.42:0/525815344 <== mon.1 v1:172.21.15.154:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2177413734 0 0) 0x7efc38009a00 con 0x7efc280036d0 2024-09-16T04:02:50.783 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.775+0000 7efc4559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:02:50.783 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.775+0000 7efc4559a640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-16T04:02:50.783 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.775+0000 7efc4559a640 10 monclient: dump: 2024-09-16T04:02:50.783 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:02:50.783 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:02:50.783 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:50.783 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:50.783 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:02:50.783 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:02:50.783 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:02:50.783 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:02:50.783 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:02:50.783 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:02:50.784 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.775+0000 7efc4559a640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] but i'm connected to v1:172.21.15.154:6789/0, reconnecting 2024-09-16T04:02:50.784 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.775+0000 7efc4559a640 10 monclient: _reopen_session rank -1 2024-09-16T04:02:50.784 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.775+0000 7efc4559a640 1 -- 172.21.15.42:0/525815344 >> v1:172.21.15.154:6789/0 conn(0x7efc280036d0 legacy=0x7efc28005b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:02:50.784 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.775+0000 7efc4559a640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:02:50.784 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.775+0000 7efc46d9d640 1 -- 172.21.15.42:0/525815344 reap_dead start 2024-09-16T04:02:50.784 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.775+0000 7efc4559a640 1 --2- 172.21.15.42:0/525815344 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7efc340039c0 0x7efc34003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:50.784 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.775+0000 7efc4559a640 10 monclient(hunting): picked mon.c con 0x7efc340039c0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:02:50.784 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.775+0000 7efc4559a640 1 --2- 172.21.15.42:0/525815344 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efc34004810 0x7efc34004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:50.784 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.775+0000 7efc4559a640 10 monclient(hunting): picked mon.a con 0x7efc34004810 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:02:50.784 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.775+0000 7efc4559a640 1 --2- 172.21.15.42:0/525815344 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efc34005220 0x7efc3400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:50.784 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.775+0000 7efc4559a640 10 monclient(hunting): picked mon.b con 0x7efc34005220 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:02:50.784 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.775+0000 7efc4559a640 10 monclient(hunting): start opening mon connection 2024-09-16T04:02:50.785 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.775+0000 7efc4559a640 1 -- 172.21.15.42:0/525815344 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7efc4809c420 con 0x7efc34004810 2024-09-16T04:02:50.785 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.775+0000 7efc4559a640 10 monclient(hunting): start opening mon connection 2024-09-16T04:02:50.785 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.775+0000 7efc4559a640 1 -- 172.21.15.42:0/525815344 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7efc3400c150 con 0x7efc34005220 2024-09-16T04:02:50.785 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.775+0000 7efc4559a640 10 monclient(hunting): start opening mon connection 2024-09-16T04:02:50.785 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.775+0000 7efc4559a640 1 -- 172.21.15.42:0/525815344 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7efc3400c350 con 0x7efc340039c0 2024-09-16T04:02:50.785 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.775+0000 7efc4559a640 10 monclient(hunting): _renew_subs 2024-09-16T04:02:50.785 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.775+0000 7efc4559a640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:02:50.785 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.775+0000 7efc4659c640 1 -- 172.21.15.42:0/525815344 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7efc340039c0 msgr2=0x7efc34003dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T04:02:50.785 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.775+0000 7efc4659c640 1 --2- 172.21.15.42:0/525815344 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7efc340039c0 0x7efc34003dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:02:50.785 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.775+0000 7efc45d9b640 1 --2- 172.21.15.42:0/525815344 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efc34004810 0x7efc34004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:02:50.785 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.775+0000 7efc45d9b640 10 monclient(hunting): get_auth_request con 0x7efc34004810 auth_method 0 2024-09-16T04:02:50.785 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.775+0000 7efc45d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:02:50.785 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.775+0000 7efc45d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:02:50.786 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.775+0000 7efc45d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:02:50.786 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.775+0000 7efc46d9d640 1 --2- 172.21.15.42:0/525815344 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efc34005220 0x7efc3400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:02:50.786 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.775+0000 7efc45d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:02:50.786 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.775+0000 7efc45d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:02:50.786 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.775+0000 7efc45d9b640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:02:50.786 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.775+0000 7efc46d9d640 10 monclient(hunting): get_auth_request con 0x7efc34005220 auth_method 0 2024-09-16T04:02:50.786 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.775+0000 7efc46d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:02:50.786 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.775+0000 7efc46d9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:02:50.786 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.775+0000 7efc46d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:02:50.786 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.775+0000 7efc46d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:02:50.786 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.775+0000 7efc46d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:02:50.786 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.775+0000 7efc46d9d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:02:50.786 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.775+0000 7efc45d9b640 10 monclient(hunting): handle_auth_done global_id 904657 payload 293 2024-09-16T04:02:50.786 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.779+0000 7efc45d9b640 1 -- 172.21.15.42:0/525815344 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7efc340039c0 msgr2=0x7efc34003dc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:02:50.787 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.779+0000 7efc45d9b640 1 --2- 172.21.15.42:0/525815344 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7efc340039c0 0x7efc34003dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:50.787 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.779+0000 7efc45d9b640 1 -- 172.21.15.42:0/525815344 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efc34005220 msgr2=0x7efc3400b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:02:50.787 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.779+0000 7efc45d9b640 1 --2- 172.21.15.42:0/525815344 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efc34005220 0x7efc3400b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:50.787 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.779+0000 7efc45d9b640 10 monclient: _finish_hunting 0 2024-09-16T04:02:50.787 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.779+0000 7efc45d9b640 1 monclient: found mon.a 2024-09-16T04:02:50.787 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.779+0000 7efc45d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:02:50.787 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.779+0000 7efc45d9b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:02:50.787 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.779+0000 7efc45d9b640 1 -- 172.21.15.42:0/525815344 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc34003530 con 0x7efc34004810 2024-09-16T04:02:50.787 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.779+0000 7efc45d9b640 10 monclient: _finish_auth 0 2024-09-16T04:02:50.787 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.779+0000 7efc45d9b640 10 monclient: _check_auth_tickets 2024-09-16T04:02:50.787 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.779+0000 7efc45d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:02:50.787 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.779+0000 7efc4c8cc640 5 monclient: authenticate success, global_id 904657 2024-09-16T04:02:50.787 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.779+0000 7efc4c8cc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:02:50.790 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.787+0000 7efc45d9b640 1 --2- 172.21.15.42:0/525815344 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efc34004810 0x7efc34004c90 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7efc30007fd0 tx=0x7efc3001cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63cf2b8f5d2e72f5 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:02:50.791 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.787+0000 7efc4559a640 1 -- 172.21.15.42:0/525815344 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc300ab070 con 0x7efc34004810 2024-09-16T04:02:50.791 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.787+0000 7efc4559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:02:50.791 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.787+0000 7efc4559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:02:50.791 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.787+0000 7efc4559a640 10 monclient: dump: 2024-09-16T04:02:50.791 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:02:50.791 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:02:50.791 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:50.791 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:50.791 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:02:50.791 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:02:50.791 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:02:50.792 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:02:50.792 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:02:50.792 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:02:50.792 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.787+0000 7efc4c8cc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:02:50.792 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.787+0000 7efc4559a640 1 -- 172.21.15.42:0/525815344 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efc300a5850 con 0x7efc34004810 2024-09-16T04:02:50.792 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.787+0000 7efc4559a640 10 monclient: handle_config config(0 keys) 2024-09-16T04:02:50.792 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.787+0000 7efc4c8cc640 10 monclient: get_monmap_and_config success 2024-09-16T04:02:50.792 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.787+0000 7efc4c8cc640 4 set_mon_vals no callback set 2024-09-16T04:02:50.792 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.787+0000 7efc4559a640 1 -- 172.21.15.42:0/525815344 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc30005bb0 con 0x7efc34004810 2024-09-16T04:02:50.792 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.787+0000 7efc4559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:02:50.792 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.787+0000 7efc4559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:02:50.792 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.787+0000 7efc4559a640 10 monclient: dump: 2024-09-16T04:02:50.792 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:02:50.793 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:02:50.793 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:50.793 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:50.793 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:02:50.793 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:02:50.793 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:02:50.793 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:02:50.793 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:02:50.793 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:02:50.794 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.787+0000 7efc4c8cc640 10 monclient: shutdown 2024-09-16T04:02:50.794 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.787+0000 7efc4c8cc640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:02:50.794 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.787+0000 7efc4c8cc640 1 -- 172.21.15.42:0/525815344 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efc34004810 msgr2=0x7efc34004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:02:50.795 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.787+0000 7efc4c8cc640 1 --2- 172.21.15.42:0/525815344 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efc34004810 0x7efc34004c90 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7efc30007fd0 tx=0x7efc3001cca0 comp rx=0 tx=0).stop 2024-09-16T04:02:50.795 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.787+0000 7efc4c8cc640 1 -- 172.21.15.42:0/525815344 shutdown_connections 2024-09-16T04:02:50.795 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.787+0000 7efc4c8cc640 1 --2- 172.21.15.42:0/525815344 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efc34005220 0x7efc3400b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:50.795 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.787+0000 7efc4c8cc640 1 --2- 172.21.15.42:0/525815344 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efc34004810 0x7efc34004c90 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:50.795 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.787+0000 7efc4c8cc640 1 --2- 172.21.15.42:0/525815344 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7efc340039c0 0x7efc34003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:50.795 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.787+0000 7efc4c8cc640 1 -- 172.21.15.42:0/525815344 >> 172.21.15.42:0/525815344 conn(0x7efc480a9fb0 msgr2=0x7efc480aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:02:50.795 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.787+0000 7efc4c8cc640 1 -- 172.21.15.42:0/525815344 shutdown_connections 2024-09-16T04:02:50.795 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.787+0000 7efc4c8cc640 1 -- 172.21.15.42:0/525815344 wait complete. 2024-09-16T04:02:50.795 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.791+0000 7efc4c8cc640 10 monclient: build_initial_monmap 2024-09-16T04:02:50.795 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.791+0000 7efc4c8cc640 10 monclient: monmap: 2024-09-16T04:02:50.795 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:02:50.795 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:02:50.795 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:02:50.796 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:02:50.796 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:02:50.796 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:02:50.796 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:02:50.796 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:02:50.796 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:02:50.796 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:02:50.796 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.791+0000 7efc4c8cc640 1 librados: starting msgr at 2024-09-16T04:02:50.796 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.791+0000 7efc4c8cc640 1 librados: starting objecter 2024-09-16T04:02:50.796 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.791+0000 7efc4c8cc640 1 Processor -- start 2024-09-16T04:02:50.796 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.791+0000 7efc4c8cc640 1 -- start start 2024-09-16T04:02:50.796 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.791+0000 7efc4c8cc640 1 librados: setting wanted keys 2024-09-16T04:02:50.796 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.791+0000 7efc4c8cc640 1 librados: calling monclient init 2024-09-16T04:02:50.797 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.791+0000 7efc4c8cc640 10 monclient: init 2024-09-16T04:02:50.801 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.791+0000 7efc4c8cc640 10 monclient: _reopen_session rank -1 2024-09-16T04:02:50.801 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.791+0000 7efc4c8cc640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:02:50.802 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.791+0000 7efc4c8cc640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efc340039c0 0x7efc481cfb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:50.802 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.791+0000 7efc4c8cc640 10 monclient(hunting): picked mon.noname-a con 0x7efc340039c0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:02:50.802 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.791+0000 7efc4c8cc640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efc34004810 0x7efc481d0040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:50.802 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.791+0000 7efc4c8cc640 10 monclient(hunting): picked mon.noname-b con 0x7efc34004810 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:02:50.802 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.791+0000 7efc4c8cc640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7efc34005220 0x7efc480a22e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:50.802 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.791+0000 7efc4c8cc640 10 monclient(hunting): picked mon.noname-c con 0x7efc34005220 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:02:50.802 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.791+0000 7efc4c8cc640 10 monclient(hunting): start opening mon connection 2024-09-16T04:02:50.802 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.791+0000 7efc4c8cc640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7efc4813a580 con 0x7efc340039c0 2024-09-16T04:02:50.802 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.791+0000 7efc4c8cc640 10 monclient(hunting): start opening mon connection 2024-09-16T04:02:50.802 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.791+0000 7efc4c8cc640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7efc4806dee0 con 0x7efc34004810 2024-09-16T04:02:50.802 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.791+0000 7efc4c8cc640 10 monclient(hunting): start opening mon connection 2024-09-16T04:02:50.802 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.791+0000 7efc4c8cc640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7efc4813a700 con 0x7efc34005220 2024-09-16T04:02:50.802 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.791+0000 7efc4c8cc640 10 monclient(hunting): _renew_subs 2024-09-16T04:02:50.803 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.791+0000 7efc4c8cc640 10 monclient(hunting): authenticate will time out at 2217.762043s 2024-09-16T04:02:50.803 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.791+0000 7efc46d9d640 1 -- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7efc34005220 msgr2=0x7efc480a22e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T04:02:50.803 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.791+0000 7efc46d9d640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7efc34005220 0x7efc480a22e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:02:50.803 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.791+0000 7efc4659c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efc340039c0 0x7efc481cfb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:02:50.803 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.791+0000 7efc4659c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efc340039c0 0x7efc481cfb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:43596/0 (socket says 172.21.15.42:43596) 2024-09-16T04:02:50.803 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.791+0000 7efc4659c640 1 -- 172.21.15.42:0/3522271810 learned_addr learned my addr 172.21.15.42:0/3522271810 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:02:50.803 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.791+0000 7efc4659c640 10 monclient(hunting): get_auth_request con 0x7efc340039c0 auth_method 0 2024-09-16T04:02:50.803 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.791+0000 7efc4659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:02:50.803 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.791+0000 7efc4659c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:02:50.803 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.791+0000 7efc4659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:02:50.803 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.791+0000 7efc4659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:02:50.803 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.791+0000 7efc4659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:02:50.803 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.791+0000 7efc4659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:02:50.804 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.791+0000 7efc45d9b640 1 --2- 172.21.15.42:0/3522271810 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efc34004810 0x7efc481d0040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:02:50.804 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.791+0000 7efc45d9b640 10 monclient(hunting): get_auth_request con 0x7efc34004810 auth_method 0 2024-09-16T04:02:50.804 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.791+0000 7efc45d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:02:50.804 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.791+0000 7efc45d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:02:50.804 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.791+0000 7efc45d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:02:50.804 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.791+0000 7efc45d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:02:50.804 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.791+0000 7efc45d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:02:50.804 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.791+0000 7efc45d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:02:50.804 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.791+0000 7efc4659c640 10 monclient(hunting): handle_auth_done global_id 825588 payload 824 2024-09-16T04:02:50.804 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.791+0000 7efc4659c640 1 -- 172.21.15.42:0/3522271810 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7efc34005220 msgr2=0x7efc480a22e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:02:50.804 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.791+0000 7efc4659c640 1 --2- 172.21.15.42:0/3522271810 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7efc34005220 0x7efc480a22e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:50.804 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.791+0000 7efc4659c640 1 -- 172.21.15.42:0/3522271810 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efc34004810 msgr2=0x7efc481d0040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:02:50.804 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.791+0000 7efc4659c640 1 --2- 172.21.15.42:0/3522271810 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efc34004810 0x7efc481d0040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:50.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.791+0000 7efc4659c640 10 monclient: _finish_hunting 0 2024-09-16T04:02:50.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.791+0000 7efc4659c640 1 monclient: found mon.noname-a 2024-09-16T04:02:50.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.791+0000 7efc4659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:02:50.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.791+0000 7efc4659c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:02:50.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.791+0000 7efc4659c640 1 -- 172.21.15.42:0/3522271810 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc48122430 con 0x7efc340039c0 2024-09-16T04:02:50.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.791+0000 7efc4659c640 1 --2- 172.21.15.42:0/3522271810 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efc340039c0 0x7efc481cfb00 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7efc38003f30 tx=0x7efc38011600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:02:50.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.791+0000 7efc3effd640 10 client.?.objecter ms_handle_connect 0x7efc340039c0 2024-09-16T04:02:50.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.791+0000 7efc3effd640 10 client.?.objecter resend_mon_ops 2024-09-16T04:02:50.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.791+0000 7efc3effd640 1 -- 172.21.15.42:0/3522271810 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc3801b020 con 0x7efc340039c0 2024-09-16T04:02:50.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.791+0000 7efc3effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:02:50.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.791+0000 7efc3effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:02:50.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.791+0000 7efc3effd640 10 monclient: dump: 2024-09-16T04:02:50.805 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:02:50.805 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:02:50.806 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:50.806 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:50.806 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:02:50.806 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:02:50.806 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:02:50.806 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:02:50.806 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:02:50.806 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:02:50.806 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.791+0000 7efc3effd640 10 monclient: _finish_auth 0 2024-09-16T04:02:50.806 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.791+0000 7efc3effd640 10 monclient: _check_auth_tickets 2024-09-16T04:02:50.806 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.791+0000 7efc3effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:02:50.806 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.791+0000 7efc4c8cc640 5 monclient: authenticate success, global_id 825588 2024-09-16T04:02:50.807 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.791+0000 7efc3effd640 1 -- 172.21.15.42:0/3522271810 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efc38017660 con 0x7efc340039c0 2024-09-16T04:02:50.807 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.791+0000 7efc4c8cc640 10 monclient: _renew_subs 2024-09-16T04:02:50.807 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.791+0000 7efc4c8cc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:02:50.807 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.791+0000 7efc4c8cc640 1 -- 172.21.15.42:0/3522271810 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc48121260 con 0x7efc340039c0 2024-09-16T04:02:50.807 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.791+0000 7efc3effd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:02:50.807 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.791+0000 7efc3effd640 1 -- 172.21.15.42:0/3522271810 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc38017a50 con 0x7efc340039c0 2024-09-16T04:02:50.807 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.791+0000 7efc3effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:02:50.807 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.791+0000 7efc3effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:02:50.807 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.791+0000 7efc3effd640 10 monclient: dump: 2024-09-16T04:02:50.807 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:02:50.807 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:02:50.807 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:50.807 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:50.808 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:02:50.808 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:02:50.808 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:02:50.808 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:02:50.808 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:02:50.808 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:02:50.808 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.791+0000 7efc3ffff640 4 set_mon_vals no callback set 2024-09-16T04:02:50.808 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.791+0000 7efc4c8cc640 10 client.825588.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:02:50.808 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.791+0000 7efc4c8cc640 10 monclient: _renew_subs 2024-09-16T04:02:50.808 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.791+0000 7efc4c8cc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:02:50.808 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.791+0000 7efc4c8cc640 1 -- 172.21.15.42:0/3522271810 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc48122150 con 0x7efc340039c0 2024-09-16T04:02:50.808 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.791+0000 7efc4c8cc640 1 librados: init done 2024-09-16T04:02:50.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.795+0000 7efc3effd640 1 -- 172.21.15.42:0/3522271810 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7efc3801e020 con 0x7efc340039c0 2024-09-16T04:02:50.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.795+0000 7efc4c8cc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:02:50.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.795+0000 7efc4c8cc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:02:50.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.795+0000 7efc4c8cc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:02:50.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.795+0000 7efc4c8cc640 1 -- 172.21.15.42:0/3522271810 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efc4813a880 con 0x7efc340039c0 2024-09-16T04:02:50.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.795+0000 7efc3effd640 1 --2- 172.21.15.42:0/3522271810 >> v2:172.21.15.154:6812/254845342 conn(0x7efc280380d0 0x7efc2803a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:50.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.795+0000 7efc3effd640 1 -- 172.21.15.42:0/3522271810 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(633..633 src has 451..633) ==== 6359+0+0 (secure 0 0 0) 0x7efc38049f70 con 0x7efc340039c0 2024-09-16T04:02:50.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.795+0000 7efc3effd640 10 client.825588.objecter ms_dispatch 0x7efc48000ba0 osd_map(633..633 src has 451..633) 2024-09-16T04:02:50.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.795+0000 7efc3effd640 3 client.825588.objecter handle_osd_map got epochs [633,633] > 0 2024-09-16T04:02:50.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.795+0000 7efc3effd640 3 client.825588.objecter handle_osd_map decoding full epoch 633 2024-09-16T04:02:50.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.795+0000 7efc3effd640 20 client.825588.objecter dump_active .. 0 homeless 2024-09-16T04:02:50.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.795+0000 7efc45d9b640 1 --2- 172.21.15.42:0/3522271810 >> v2:172.21.15.154:6812/254845342 conn(0x7efc280380d0 0x7efc2803a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:02:50.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.795+0000 7efc45d9b640 10 monclient: get_auth_request con 0x7efc280380d0 auth_method 0 2024-09-16T04:02:50.810 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.795+0000 7efc45d9b640 1 --2- 172.21.15.42:0/3522271810 >> v2:172.21.15.154:6812/254845342 conn(0x7efc280380d0 0x7efc2803a590 secure :-1 s=READY pgs=3794 cs=0 l=1 rev1=1 crypto rx=0x7efc300b0000 tx=0x7efc30001130 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:02:50.810 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.795+0000 7efc3effd640 10 client.825588.objecter ms_handle_connect 0x7efc280380d0 2024-09-16T04:02:50.810 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.799+0000 7efc3effd640 1 -- 172.21.15.42:0/3522271810 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7efc38025070 con 0x7efc340039c0 2024-09-16T04:02:50.810 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.799+0000 7efc3effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:02:50.810 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:50.799+0000 7efc3effd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:02:51.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.047+0000 7efc4c8cc640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:02:51.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.047+0000 7efc4c8cc640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:02:51.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.047+0000 7efc4c8cc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:02:51.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.047+0000 7efc4c8cc640 1 -- 172.21.15.42:0/3522271810 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7efc480a5750 con 0x7efc340039c0 2024-09-16T04:02:51.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.047+0000 7efc3effd640 1 -- 172.21.15.42:0/3522271810 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1555 (secure 0 0 0) 0x7efc38010040 con 0x7efc340039c0 2024-09-16T04:02:51.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.047+0000 7efc3effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:02:51.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.047+0000 7efc3effd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:02:51.052 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":188,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:02:51.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.051+0000 7efc4c8cc640 10 librados: watch_flush enter 2024-09-16T04:02:51.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.051+0000 7efc4c8cc640 10 librados: watch_flush exit 2024-09-16T04:02:51.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.051+0000 7efc4c8cc640 20 client.825588.objecter shutdown clearing up homeless session... 2024-09-16T04:02:51.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.051+0000 7efc4c8cc640 10 client.825588.objecter successfully canceled tick 2024-09-16T04:02:51.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.051+0000 7efc4c8cc640 1 -- 172.21.15.42:0/3522271810 >> v2:172.21.15.154:6812/254845342 conn(0x7efc280380d0 msgr2=0x7efc2803a590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:02:51.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.051+0000 7efc4c8cc640 1 --2- 172.21.15.42:0/3522271810 >> v2:172.21.15.154:6812/254845342 conn(0x7efc280380d0 0x7efc2803a590 secure :-1 s=READY pgs=3794 cs=0 l=1 rev1=1 crypto rx=0x7efc300b0000 tx=0x7efc30001130 comp rx=0 tx=0).stop 2024-09-16T04:02:51.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.051+0000 7efc4c8cc640 10 monclient: shutdown 2024-09-16T04:02:51.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.051+0000 7efc4c8cc640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:02:51.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.051+0000 7efc4c8cc640 1 -- 172.21.15.42:0/3522271810 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efc340039c0 msgr2=0x7efc481cfb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:02:51.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.051+0000 7efc4c8cc640 1 --2- 172.21.15.42:0/3522271810 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efc340039c0 0x7efc481cfb00 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7efc38003f30 tx=0x7efc38011600 comp rx=0 tx=0).stop 2024-09-16T04:02:51.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.051+0000 7efc4c8cc640 1 -- 172.21.15.42:0/3522271810 shutdown_connections 2024-09-16T04:02:51.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.051+0000 7efc4c8cc640 1 --2- 172.21.15.42:0/3522271810 >> v2:172.21.15.154:6812/254845342 conn(0x7efc280380d0 0x7efc2803a590 unknown :-1 s=CLOSED pgs=3794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:51.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.051+0000 7efc4c8cc640 1 --2- 172.21.15.42:0/3522271810 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7efc34005220 0x7efc480a22e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:51.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.051+0000 7efc4c8cc640 1 --2- 172.21.15.42:0/3522271810 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7efc34004810 0x7efc481d0040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:51.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.051+0000 7efc4c8cc640 1 --2- 172.21.15.42:0/3522271810 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7efc340039c0 0x7efc481cfb00 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:51.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.051+0000 7efc4c8cc640 1 -- 172.21.15.42:0/3522271810 >> 172.21.15.42:0/3522271810 conn(0x7efc480a9fb0 msgr2=0x7efc48131980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:02:51.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.051+0000 7efc4c8cc640 1 -- 172.21.15.42:0/3522271810 shutdown_connections 2024-09-16T04:02:51.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.051+0000 7efc4c8cc640 1 -- 172.21.15.42:0/3522271810 wait complete. 2024-09-16T04:02:51.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.051+0000 7efc4c8cc640 1 librados: shutdown 2024-09-16T04:02:51.071 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T04:02:51.071 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-16T04:02:51.071 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T04:02:51.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.191+0000 7ff0bcefe640 10 monclient: get_monmap_and_config 2024-09-16T04:02:51.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.191+0000 7ff0bcefe640 10 monclient: build_initial_monmap 2024-09-16T04:02:51.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.191+0000 7ff0bcefe640 10 monclient: monmap: 2024-09-16T04:02:51.195 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:02:51.195 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:02:51.195 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:02:51.193111+0000 2024-09-16T04:02:51.195 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:02:51.193111+0000 2024-09-16T04:02:51.195 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:02:51.195 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:02:51.195 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:02:51.195 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:02:51.195 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:02:51.195 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:02:51.195 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:02:51.195 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:02:51.198 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.191+0000 7ff0bcefe640 1 Processor -- start 2024-09-16T04:02:51.198 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.191+0000 7ff0bcefe640 1 -- start start 2024-09-16T04:02:51.198 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.191+0000 7ff0bcefe640 10 monclient: init 2024-09-16T04:02:51.198 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.191+0000 7ff0bcefe640 10 monclient: _reopen_session rank -1 2024-09-16T04:02:51.198 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.191+0000 7ff0bcefe640 10 monclient: _add_conns ranks=[3,4,2,0,1] 2024-09-16T04:02:51.198 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.191+0000 7ff0bcefe640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff0b80a3860 0x7ff0b80a3c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:51.199 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.191+0000 7ff0bcefe640 10 monclient(hunting): picked mon.noname-a con 0x7ff0b80a3860 addr v2:172.21.15.42:3300/0 2024-09-16T04:02:51.199 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.191+0000 7ff0bcefe640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7ff0b8137020 0x7ff0b80a4540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:51.199 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.191+0000 7ff0bcefe640 10 monclient(hunting): picked mon.noname-c con 0x7ff0b8137020 addr v2:172.21.15.154:3300/0 2024-09-16T04:02:51.199 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.191+0000 7ff0bcefe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff0b8137460 0x7ff0b80a4a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:51.199 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.191+0000 7ff0bcefe640 10 monclient(hunting): picked mon.noname-b con 0x7ff0b8137460 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:02:51.199 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.191+0000 7ff0bcefe640 10 monclient(hunting): start opening mon connection 2024-09-16T04:02:51.199 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.191+0000 7ff0bcefe640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7ff0b80a5e80 con 0x7ff0b80a3860 2024-09-16T04:02:51.199 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.191+0000 7ff0bcefe640 10 monclient(hunting): start opening mon connection 2024-09-16T04:02:51.199 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.191+0000 7ff0bcefe640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7ff0b809c010 con 0x7ff0b8137020 2024-09-16T04:02:51.200 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.191+0000 7ff0bcefe640 10 monclient(hunting): start opening mon connection 2024-09-16T04:02:51.200 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.191+0000 7ff0bcefe640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7ff0b809c2f0 con 0x7ff0b8137460 2024-09-16T04:02:51.200 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.191+0000 7ff0bcefe640 10 monclient(hunting): _renew_subs 2024-09-16T04:02:51.200 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.191+0000 7ff0bcefe640 10 monclient(hunting): authenticate will time out at 2218.162967s 2024-09-16T04:02:51.200 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.191+0000 7ff0b6d9d640 1 -- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff0b8137460 msgr2=0x7ff0b80a4a80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T04:02:51.200 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.191+0000 7ff0b6d9d640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff0b8137460 0x7ff0b80a4a80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:02:51.200 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.191+0000 7ff0b659c640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff0b80a3860 0x7ff0b80a3c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:02:51.201 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.191+0000 7ff0b659c640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff0b80a3860 0x7ff0b80a3c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:43606/0 (socket says 172.21.15.42:43606) 2024-09-16T04:02:51.201 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.191+0000 7ff0b659c640 1 -- 172.21.15.42:0/3929937554 learned_addr learned my addr 172.21.15.42:0/3929937554 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:02:51.201 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.191+0000 7ff0b659c640 10 monclient(hunting): get_auth_request con 0x7ff0b80a3860 auth_method 0 2024-09-16T04:02:51.201 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.191+0000 7ff0b659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:02:51.201 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.191+0000 7ff0b659c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:02:51.201 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.191+0000 7ff0b659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:02:51.201 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.191+0000 7ff0b659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:02:51.201 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.191+0000 7ff0b659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:02:51.201 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.191+0000 7ff0b659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:02:51.201 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.191+0000 7ff0b5d9b640 1 --2- 172.21.15.42:0/3929937554 >> v2:172.21.15.154:3300/0 conn(0x7ff0b8137020 0x7ff0b80a4540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:02:51.201 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.191+0000 7ff0b5d9b640 10 monclient(hunting): get_auth_request con 0x7ff0b8137020 auth_method 0 2024-09-16T04:02:51.201 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.191+0000 7ff0b5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:02:51.201 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.191+0000 7ff0b5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:02:51.201 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.191+0000 7ff0b5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:02:51.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.191+0000 7ff0b659c640 10 monclient(hunting): handle_auth_done global_id 825591 payload 274 2024-09-16T04:02:51.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.191+0000 7ff0b659c640 1 -- 172.21.15.42:0/3929937554 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff0b8137460 msgr2=0x7ff0b80a4a80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:02:51.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.191+0000 7ff0b659c640 1 --2- 172.21.15.42:0/3929937554 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff0b8137460 0x7ff0b80a4a80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:51.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.191+0000 7ff0b659c640 1 -- 172.21.15.42:0/3929937554 >> v2:172.21.15.154:3300/0 conn(0x7ff0b8137020 msgr2=0x7ff0b80a4540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:02:51.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.191+0000 7ff0b659c640 1 --2- 172.21.15.42:0/3929937554 >> v2:172.21.15.154:3300/0 conn(0x7ff0b8137020 0x7ff0b80a4540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:51.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.191+0000 7ff0b659c640 10 monclient: _finish_hunting 0 2024-09-16T04:02:51.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.191+0000 7ff0b659c640 1 monclient: found mon.noname-a 2024-09-16T04:02:51.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.191+0000 7ff0b659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:02:51.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.191+0000 7ff0b659c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:02:51.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.191+0000 7ff0b659c640 1 -- 172.21.15.42:0/3929937554 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0b80a5c20 con 0x7ff0b80a3860 2024-09-16T04:02:51.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.195+0000 7ff0b659c640 1 --2- 172.21.15.42:0/3929937554 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff0b80a3860 0x7ff0b80a3c40 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7ff0b8131530 tx=0x7ff0b001c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6dd30e01bf2e89d4 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:02:51.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.195+0000 7ff0b559a640 1 -- 172.21.15.42:0/3929937554 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0b00a5550 con 0x7ff0b80a3860 2024-09-16T04:02:51.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.195+0000 7ff0b559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:02:51.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.195+0000 7ff0b559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:02:51.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.195+0000 7ff0b559a640 10 monclient: dump: 2024-09-16T04:02:51.210 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:02:51.210 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:02:51.210 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:51.210 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:51.210 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:02:51.210 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:02:51.210 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:02:51.210 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:02:51.210 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:02:51.210 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:02:51.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.195+0000 7ff0b559a640 10 monclient: _finish_auth 0 2024-09-16T04:02:51.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.195+0000 7ff0b559a640 10 monclient: _check_auth_tickets 2024-09-16T04:02:51.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.195+0000 7ff0b559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:02:51.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.195+0000 7ff0b559a640 1 -- 172.21.15.42:0/3929937554 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff0b00a56f0 con 0x7ff0b80a3860 2024-09-16T04:02:51.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.195+0000 7ff0bcefe640 5 monclient: authenticate success, global_id 825591 2024-09-16T04:02:51.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.195+0000 7ff0bcefe640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:02:51.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.195+0000 7ff0b559a640 10 monclient: handle_config config(0 keys) 2024-09-16T04:02:51.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.195+0000 7ff0bcefe640 10 monclient: get_monmap_and_config success 2024-09-16T04:02:51.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.195+0000 7ff0bcefe640 4 set_mon_vals no callback set 2024-09-16T04:02:51.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.195+0000 7ff0b559a640 1 -- 172.21.15.42:0/3929937554 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0b00a5ac0 con 0x7ff0b80a3860 2024-09-16T04:02:51.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.195+0000 7ff0b559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:02:51.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.195+0000 7ff0b559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:02:51.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.195+0000 7ff0b559a640 10 monclient: dump: 2024-09-16T04:02:51.211 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:02:51.211 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:02:51.211 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:51.211 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:51.212 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:02:51.212 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:02:51.212 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:02:51.212 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:02:51.212 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:02:51.212 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:02:51.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.195+0000 7ff0bcefe640 10 monclient: shutdown 2024-09-16T04:02:51.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.195+0000 7ff0bcefe640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:02:51.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.195+0000 7ff0bcefe640 1 -- 172.21.15.42:0/3929937554 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff0b80a3860 msgr2=0x7ff0b80a3c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:02:51.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.195+0000 7ff0bcefe640 1 --2- 172.21.15.42:0/3929937554 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff0b80a3860 0x7ff0b80a3c40 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7ff0b8131530 tx=0x7ff0b001c8d0 comp rx=0 tx=0).stop 2024-09-16T04:02:51.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.195+0000 7ff0bcefe640 1 -- 172.21.15.42:0/3929937554 shutdown_connections 2024-09-16T04:02:51.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.195+0000 7ff0bcefe640 1 --2- 172.21.15.42:0/3929937554 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff0b8137460 0x7ff0b80a4a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:51.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.195+0000 7ff0bcefe640 1 --2- 172.21.15.42:0/3929937554 >> v2:172.21.15.154:3300/0 conn(0x7ff0b8137020 0x7ff0b80a4540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:51.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.195+0000 7ff0bcefe640 1 --2- 172.21.15.42:0/3929937554 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff0b80a3860 0x7ff0b80a3c40 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:51.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.195+0000 7ff0bcefe640 1 -- 172.21.15.42:0/3929937554 >> 172.21.15.42:0/3929937554 conn(0x7ff0b80a9f40 msgr2=0x7ff0b80aa350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:02:51.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.195+0000 7ff0bcefe640 1 -- 172.21.15.42:0/3929937554 shutdown_connections 2024-09-16T04:02:51.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.195+0000 7ff0bcefe640 1 -- 172.21.15.42:0/3929937554 wait complete. 2024-09-16T04:02:51.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.195+0000 7ff0bcefe640 10 monclient: build_initial_monmap 2024-09-16T04:02:51.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.199+0000 7ff0bcefe640 10 monclient: monmap: 2024-09-16T04:02:51.213 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:02:51.213 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:02:51.213 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:02:51.213 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:02:51.213 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:02:51.213 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:02:51.213 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:02:51.213 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:02:51.213 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:02:51.214 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:02:51.214 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.199+0000 7ff0bcefe640 1 librados: starting msgr at 2024-09-16T04:02:51.214 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.199+0000 7ff0bcefe640 1 librados: starting objecter 2024-09-16T04:02:51.214 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.199+0000 7ff0bcefe640 1 Processor -- start 2024-09-16T04:02:51.214 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.199+0000 7ff0bcefe640 1 -- start start 2024-09-16T04:02:51.214 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.199+0000 7ff0bcefe640 1 librados: setting wanted keys 2024-09-16T04:02:51.214 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.199+0000 7ff0bcefe640 1 librados: calling monclient init 2024-09-16T04:02:51.214 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.199+0000 7ff0bcefe640 10 monclient: init 2024-09-16T04:02:51.214 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.199+0000 7ff0bcefe640 10 monclient: _reopen_session rank -1 2024-09-16T04:02:51.214 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.199+0000 7ff0bcefe640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:02:51.214 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.199+0000 7ff0bcefe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff0b80a3860 0x7ff0b813d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:51.214 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.199+0000 7ff0bcefe640 10 monclient(hunting): picked mon.noname-a con 0x7ff0b80a3860 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:02:51.214 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.199+0000 7ff0bcefe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff0b8137020 0x7ff0b8137f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:51.214 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.199+0000 7ff0bcefe640 10 monclient(hunting): picked mon.noname-c con 0x7ff0b8137020 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:02:51.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.199+0000 7ff0bcefe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff0b8137460 0x7ff0b81384a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:51.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.199+0000 7ff0bcefe640 10 monclient(hunting): picked mon.noname-b con 0x7ff0b8137460 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:02:51.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.199+0000 7ff0bcefe640 10 monclient(hunting): start opening mon connection 2024-09-16T04:02:51.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.199+0000 7ff0bcefe640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7ff0b8140c10 con 0x7ff0b80a3860 2024-09-16T04:02:51.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.199+0000 7ff0bcefe640 10 monclient(hunting): start opening mon connection 2024-09-16T04:02:51.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.199+0000 7ff0bcefe640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff0b80a53d0 con 0x7ff0b8137460 2024-09-16T04:02:51.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.199+0000 7ff0bcefe640 10 monclient(hunting): start opening mon connection 2024-09-16T04:02:51.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.199+0000 7ff0bcefe640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7ff0b8140d90 con 0x7ff0b8137020 2024-09-16T04:02:51.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.199+0000 7ff0bcefe640 10 monclient(hunting): _renew_subs 2024-09-16T04:02:51.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.199+0000 7ff0bcefe640 10 monclient(hunting): authenticate will time out at 2218.168911s 2024-09-16T04:02:51.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.199+0000 7ff0b5d9b640 1 -- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff0b8137020 msgr2=0x7ff0b8137f60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T04:02:51.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.199+0000 7ff0b5d9b640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff0b8137020 0x7ff0b8137f60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:02:51.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.199+0000 7ff0b659c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff0b80a3860 0x7ff0b813d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:02:51.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.199+0000 7ff0b659c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff0b80a3860 0x7ff0b813d9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:43610/0 (socket says 172.21.15.42:43610) 2024-09-16T04:02:51.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.199+0000 7ff0b659c640 1 -- 172.21.15.42:0/3546973438 learned_addr learned my addr 172.21.15.42:0/3546973438 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:02:51.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.199+0000 7ff0b659c640 10 monclient(hunting): get_auth_request con 0x7ff0b80a3860 auth_method 0 2024-09-16T04:02:51.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.199+0000 7ff0b659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:02:51.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.199+0000 7ff0b659c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:02:51.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.199+0000 7ff0b659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:02:51.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.199+0000 7ff0b659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:02:51.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.199+0000 7ff0b659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:02:51.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.199+0000 7ff0b659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:02:51.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.199+0000 7ff0b6d9d640 1 --2- 172.21.15.42:0/3546973438 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff0b8137460 0x7ff0b81384a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:02:51.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.199+0000 7ff0b6d9d640 10 monclient(hunting): get_auth_request con 0x7ff0b8137460 auth_method 0 2024-09-16T04:02:51.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.199+0000 7ff0b6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:02:51.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.199+0000 7ff0b6d9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:02:51.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.199+0000 7ff0b6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:02:51.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.199+0000 7ff0b6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:02:51.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.199+0000 7ff0b6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:02:51.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.199+0000 7ff0b6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:02:51.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.199+0000 7ff0b659c640 10 monclient(hunting): handle_auth_done global_id 825594 payload 824 2024-09-16T04:02:51.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.199+0000 7ff0b659c640 1 -- 172.21.15.42:0/3546973438 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff0b8137020 msgr2=0x7ff0b8137f60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:02:51.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.199+0000 7ff0b659c640 1 --2- 172.21.15.42:0/3546973438 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff0b8137020 0x7ff0b8137f60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:51.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.199+0000 7ff0b659c640 1 -- 172.21.15.42:0/3546973438 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff0b8137460 msgr2=0x7ff0b81384a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:02:51.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.199+0000 7ff0b659c640 1 --2- 172.21.15.42:0/3546973438 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff0b8137460 0x7ff0b81384a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:51.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.199+0000 7ff0b659c640 10 monclient: _finish_hunting 0 2024-09-16T04:02:51.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.199+0000 7ff0b659c640 1 monclient: found mon.noname-a 2024-09-16T04:02:51.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.199+0000 7ff0b659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:02:51.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.199+0000 7ff0b659c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:02:51.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.199+0000 7ff0b659c640 1 -- 172.21.15.42:0/3546973438 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0b813afe0 con 0x7ff0b80a3860 2024-09-16T04:02:51.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.199+0000 7ff0b659c640 1 --2- 172.21.15.42:0/3546973438 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff0b80a3860 0x7ff0b813d9c0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7ff0b00af040 tx=0x7ff0b0019670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:02:51.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.199+0000 7ff0aeffd640 10 client.?.objecter ms_handle_connect 0x7ff0b80a3860 2024-09-16T04:02:51.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.199+0000 7ff0aeffd640 10 client.?.objecter resend_mon_ops 2024-09-16T04:02:51.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.199+0000 7ff0aeffd640 1 -- 172.21.15.42:0/3546973438 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0b00bc020 con 0x7ff0b80a3860 2024-09-16T04:02:51.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.199+0000 7ff0aeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:02:51.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.199+0000 7ff0aeffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:02:51.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.199+0000 7ff0aeffd640 10 monclient: dump: 2024-09-16T04:02:51.218 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:02:51.218 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:02:51.218 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:51.218 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:51.218 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:02:51.218 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:02:51.218 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:02:51.218 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:02:51.219 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:02:51.219 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:02:51.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.199+0000 7ff0aeffd640 10 monclient: _finish_auth 0 2024-09-16T04:02:51.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.199+0000 7ff0aeffd640 10 monclient: _check_auth_tickets 2024-09-16T04:02:51.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.199+0000 7ff0aeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:02:51.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.199+0000 7ff0aeffd640 1 -- 172.21.15.42:0/3546973438 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff0b00b88a0 con 0x7ff0b80a3860 2024-09-16T04:02:51.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.199+0000 7ff0bcefe640 5 monclient: authenticate success, global_id 825594 2024-09-16T04:02:51.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.199+0000 7ff0bcefe640 10 monclient: _renew_subs 2024-09-16T04:02:51.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.199+0000 7ff0bcefe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:02:51.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.199+0000 7ff0bcefe640 1 -- 172.21.15.42:0/3546973438 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0b81353a0 con 0x7ff0b80a3860 2024-09-16T04:02:51.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.199+0000 7ff0aeffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:02:51.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.199+0000 7ff0aeffd640 1 -- 172.21.15.42:0/3546973438 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0b0009510 con 0x7ff0b80a3860 2024-09-16T04:02:51.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.199+0000 7ff0aeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:02:51.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.199+0000 7ff0aeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:02:51.220 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.199+0000 7ff0aeffd640 10 monclient: dump: 2024-09-16T04:02:51.220 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:02:51.220 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:02:51.220 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:51.220 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:51.220 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:02:51.220 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:02:51.220 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:02:51.220 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:02:51.220 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:02:51.220 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:02:51.220 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.199+0000 7ff0b4d99640 4 set_mon_vals no callback set 2024-09-16T04:02:51.220 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.199+0000 7ff0bcefe640 10 client.825594.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:02:51.221 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.199+0000 7ff0bcefe640 10 monclient: _renew_subs 2024-09-16T04:02:51.221 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.199+0000 7ff0bcefe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:02:51.221 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.199+0000 7ff0bcefe640 1 -- 172.21.15.42:0/3546973438 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0b813aa20 con 0x7ff0b80a3860 2024-09-16T04:02:51.221 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.199+0000 7ff0bcefe640 1 librados: init done 2024-09-16T04:02:51.221 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.199+0000 7ff0bcefe640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:02:51.221 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.199+0000 7ff0bcefe640 1 --2- 172.21.15.42:0/3546973438 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff0b802d7d0 0x7ff0b81389e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:51.221 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.199+0000 7ff0bcefe640 10 monclient(hunting): start opening mon connection 2024-09-16T04:02:51.221 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.199+0000 7ff0bcefe640 1 -- 172.21.15.42:0/3546973438 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7ff0b802dbb0 con 0x7ff0b802d7d0 2024-09-16T04:02:51.221 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.199+0000 7ff0b5d9b640 1 --2- 172.21.15.42:0/3546973438 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff0b802d7d0 0x7ff0b81389e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:02:51.221 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.199+0000 7ff0b5d9b640 10 monclient: get_auth_request con 0x7ff0b802d7d0 auth_method 0 2024-09-16T04:02:51.221 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.199+0000 7ff0b5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:02:51.221 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.199+0000 7ff0b5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:02:51.221 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.199+0000 7ff0b5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:02:51.222 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.199+0000 7ff0b5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:02:51.222 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.199+0000 7ff0b5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:02:51.222 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.199+0000 7ff0b5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:02:51.222 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.199+0000 7ff0aeffd640 1 -- 172.21.15.42:0/3546973438 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7ff0b00bf020 con 0x7ff0b80a3860 2024-09-16T04:02:51.222 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.199+0000 7ff0b5d9b640 10 monclient(hunting): handle_auth_done global_id 825597 payload 824 2024-09-16T04:02:51.222 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.199+0000 7ff0b5d9b640 1 -- 172.21.15.42:0/3546973438 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff0b80a3c40 con 0x7ff0b802d7d0 2024-09-16T04:02:51.222 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.199+0000 7ff0aeffd640 1 --2- 172.21.15.42:0/3546973438 >> v2:172.21.15.154:6812/254845342 conn(0x7ff0980377c0 0x7ff098039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:51.222 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.199+0000 7ff0aeffd640 1 -- 172.21.15.42:0/3546973438 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(633..633 src has 451..633) ==== 6359+0+0 (secure 0 0 0) 0x7ff0b00e4210 con 0x7ff0b80a3860 2024-09-16T04:02:51.222 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.199+0000 7ff0aeffd640 10 client.825594.objecter ms_dispatch 0x7ff0b8000ba0 osd_map(633..633 src has 451..633) 2024-09-16T04:02:51.222 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.199+0000 7ff0aeffd640 3 client.825594.objecter handle_osd_map got epochs [633,633] > 0 2024-09-16T04:02:51.222 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.199+0000 7ff0aeffd640 3 client.825594.objecter handle_osd_map decoding full epoch 633 2024-09-16T04:02:51.222 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.199+0000 7ff0b5d9b640 1 --2- 172.21.15.42:0/3546973438 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff0b802d7d0 0x7ff0b81389e0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7ff0b8130280 tx=0x7ff0a000eba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:02:51.222 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.199+0000 7ff0aeffd640 20 client.825594.objecter dump_active .. 0 homeless 2024-09-16T04:02:51.223 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.199+0000 7ff0aeffd640 10 client.825594.objecter ms_handle_connect 0x7ff0b802d7d0 2024-09-16T04:02:51.223 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.199+0000 7ff0aeffd640 10 client.825594.objecter resend_mon_ops 2024-09-16T04:02:51.223 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.199+0000 7ff0aeffd640 1 -- 172.21.15.42:0/3546973438 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0a0017020 con 0x7ff0b802d7d0 2024-09-16T04:02:51.223 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.199+0000 7ff0aeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:02:51.223 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.199+0000 7ff0aeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:02:51.223 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.199+0000 7ff0aeffd640 10 monclient: dump: 2024-09-16T04:02:51.223 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:02:51.223 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:02:51.223 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:51.223 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:51.223 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:02:51.223 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:02:51.223 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:02:51.223 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:02:51.224 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:02:51.224 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:02:51.224 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.199+0000 7ff0aeffd640 1 -- 172.21.15.42:0/3546973438 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7ff0a000c710 con 0x7ff0b802d7d0 2024-09-16T04:02:51.224 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.199+0000 7ff0aeffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:02:51.224 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.199+0000 7ff0aeffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:02:51.224 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.199+0000 7ff0aeffd640 1 -- 172.21.15.42:0/3546973438 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff0b802d7d0 msgr2=0x7ff0b81389e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:02:51.224 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.199+0000 7ff0aeffd640 1 --2- 172.21.15.42:0/3546973438 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff0b802d7d0 0x7ff0b81389e0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7ff0b8130280 tx=0x7ff0a000eba0 comp rx=0 tx=0).stop 2024-09-16T04:02:51.224 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.199+0000 7ff0aeffd640 1 -- 172.21.15.42:0/3546973438 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff0b802d7d0 msgr2=0x7ff0b81389e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:02:51.224 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.199+0000 7ff0aeffd640 1 --2- 172.21.15.42:0/3546973438 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff0b802d7d0 0x7ff0b81389e0 secure :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7ff0b8130280 tx=0x7ff0a000eba0 comp rx=0 tx=0).stop 2024-09-16T04:02:51.224 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.203+0000 7ff0b6d9d640 1 --2- 172.21.15.42:0/3546973438 >> v2:172.21.15.154:6812/254845342 conn(0x7ff0980377c0 0x7ff098039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:02:51.224 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.203+0000 7ff0b6d9d640 10 monclient: get_auth_request con 0x7ff0980377c0 auth_method 0 2024-09-16T04:02:51.224 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.203+0000 7ff0b6d9d640 1 --2- 172.21.15.42:0/3546973438 >> v2:172.21.15.154:6812/254845342 conn(0x7ff0980377c0 0x7ff098039c80 secure :-1 s=READY pgs=3795 cs=0 l=1 rev1=1 crypto rx=0x7ff0b813df30 tx=0x7ff0a8006f90 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:02:51.224 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.203+0000 7ff0aeffd640 10 client.825594.objecter ms_handle_connect 0x7ff0980377c0 2024-09-16T04:02:51.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.207+0000 7ff0bcefe640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:02:51.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.207+0000 7ff0bcefe640 1 --2- 172.21.15.42:0/3546973438 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff0b8062ea0 0x7ff0b80632b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:51.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.207+0000 7ff0bcefe640 10 monclient(hunting): start opening mon connection 2024-09-16T04:02:51.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.207+0000 7ff0bcefe640 1 -- 172.21.15.42:0/3546973438 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7ff0b813dfc0 con 0x7ff0b8062ea0 2024-09-16T04:02:51.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.207+0000 7ff0b5d9b640 1 --2- 172.21.15.42:0/3546973438 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff0b8062ea0 0x7ff0b80632b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:02:51.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.207+0000 7ff0b5d9b640 10 monclient: get_auth_request con 0x7ff0b8062ea0 auth_method 0 2024-09-16T04:02:51.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.207+0000 7ff0b5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:02:51.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.207+0000 7ff0b5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:02:51.227 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.207+0000 7ff0b5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:02:51.227 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.207+0000 7ff0b5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:02:51.227 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.207+0000 7ff0b5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:02:51.227 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.207+0000 7ff0b5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:02:51.227 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.207+0000 7ff0b5d9b640 10 monclient(hunting): handle_auth_done global_id 825600 payload 824 2024-09-16T04:02:51.227 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.207+0000 7ff0b5d9b640 1 -- 172.21.15.42:0/3546973438 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ff0b80a5550 con 0x7ff0b8062ea0 2024-09-16T04:02:51.227 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.207+0000 7ff0aeffd640 10 client.825594.objecter ms_handle_connect 0x7ff0b8062ea0 2024-09-16T04:02:51.227 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.207+0000 7ff0aeffd640 10 client.825594.objecter resend_mon_ops 2024-09-16T04:02:51.227 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.207+0000 7ff0b5d9b640 1 --2- 172.21.15.42:0/3546973438 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff0b8062ea0 0x7ff0b80632b0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7ff0b8130280 tx=0x7ff0a0004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:02:51.227 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.207+0000 7ff0aeffd640 1 -- 172.21.15.42:0/3546973438 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0a0017040 con 0x7ff0b8062ea0 2024-09-16T04:02:51.227 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.207+0000 7ff0aeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:02:51.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.207+0000 7ff0aeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:02:51.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.207+0000 7ff0aeffd640 10 monclient: dump: 2024-09-16T04:02:51.228 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:02:51.228 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:02:51.228 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:51.228 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:51.228 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:02:51.228 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:02:51.228 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:02:51.228 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:02:51.228 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:02:51.228 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:02:51.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.207+0000 7ff0aeffd640 1 -- 172.21.15.42:0/3546973438 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4704 (secure 0 0 0) 0x7ff0a0013640 con 0x7ff0b8062ea0 2024-09-16T04:02:51.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.207+0000 7ff0aeffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:02:51.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.207+0000 7ff0aeffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:02:51.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.207+0000 7ff0aeffd640 1 -- 172.21.15.42:0/3546973438 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff0b8062ea0 msgr2=0x7ff0b80632b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:02:51.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.207+0000 7ff0aeffd640 1 --2- 172.21.15.42:0/3546973438 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff0b8062ea0 0x7ff0b80632b0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7ff0b8130280 tx=0x7ff0a0004740 comp rx=0 tx=0).stop 2024-09-16T04:02:51.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.207+0000 7ff0aeffd640 1 -- 172.21.15.42:0/3546973438 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff0b8062ea0 msgr2=0x7ff0b80632b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:02:51.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.207+0000 7ff0aeffd640 1 --2- 172.21.15.42:0/3546973438 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff0b8062ea0 0x7ff0b80632b0 secure :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7ff0b8130280 tx=0x7ff0a0004740 comp rx=0 tx=0).stop 2024-09-16T04:02:51.229 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T04:02:51.229 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:02:51.229 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:02:51.230 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "leader", 2024-09-16T04:02:51.230 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 188, 2024-09-16T04:02:51.230 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 661062, 2024-09-16T04:02:51.230 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T04:02:51.230 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T04:02:51.230 INFO:teuthology.orchestra.run.smithi042.stdout: 1 2024-09-16T04:02:51.230 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:02:51.230 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 0, 2024-09-16T04:02:51.230 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:02:51.230 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T04:02:51.230 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T04:02:51.230 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:02:51.230 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:02:51.231 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:02:51.231 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:02:51.231 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:02:51.231 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:02:51.231 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:02:51.231 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:02:51.231 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:02:51.231 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:02:51.231 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:02:51.231 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:02:51.231 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:02:51.231 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T04:02:51.231 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:02:51.232 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:02:51.232 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:02:51.232 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:02:51.232 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:02:51.232 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:02:51.232 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:02:51.232 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:02:51.232 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:02:51.232 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:02:51.232 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:02:51.232 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:02:51.232 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:02:51.232 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T04:02:51.233 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T04:02:51.233 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T04:02:51.233 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T04:02:51.233 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T04:02:51.233 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T04:02:51.233 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:02:51.233 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:02:51.233 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T04:02:51.233 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T04:02:51.233 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T04:02:51.233 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T04:02:51.233 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T04:02:51.233 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T04:02:51.233 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T04:02:51.234 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:02:51.234 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T04:02:51.234 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:02:51.234 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:02:51.234 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:02:51.234 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:02:51.234 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:02:51.234 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:02:51.234 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:02:51.234 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:02:51.234 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:02:51.234 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:02:51.234 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:02:51.234 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:02:51.234 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T04:02:51.235 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:02:51.235 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T04:02:51.235 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:02:51.235 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:02:51.235 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:02:51.235 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:02:51.235 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:02:51.235 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:02:51.235 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:02:51.235 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T04:02:51.235 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:02:51.235 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:02:51.235 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:02:51.235 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:02:51.236 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T04:02:51.236 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:02:51.236 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:02:51.236 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:02:51.236 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:02:51.236 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T04:02:51.236 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T04:02:51.236 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:02:51.236 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:02:51.236 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:02:51.236 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:02:51.236 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:02:51.236 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:02:51.236 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:02:51.236 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:02:51.237 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:02:51.237 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:02:51.237 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:02:51.237 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T04:02:51.237 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:02:51.237 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:02:51.237 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:02:51.237 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:02:51.237 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T04:02:51.237 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:02:51.237 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:02:51.237 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:02:51.237 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:02:51.237 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T04:02:51.238 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T04:02:51.238 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:02:51.238 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:02:51.238 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:02:51.238 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:02:51.238 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:02:51.238 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:02:51.238 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:02:51.238 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:02:51.238 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:02:51.238 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:02:51.238 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:02:51.238 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T04:02:51.238 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:02:51.238 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:02:51.239 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:02:51.239 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:02:51.239 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T04:02:51.239 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:02:51.239 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:02:51.239 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:02:51.239 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:02:51.239 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T04:02:51.239 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T04:02:51.239 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:02:51.239 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:02:51.239 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:02:51.239 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:02:51.239 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:02:51.240 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:02:51.240 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T04:02:51.240 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T04:02:51.240 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:02:51.240 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:02:51.240 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:02:51.240 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:02:51.240 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:02:51.240 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:02:51.240 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T04:02:51.240 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:02:51.240 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:02:51.240 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:02:51.240 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 5 2024-09-16T04:02:51.240 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:02:51.241 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:02:51.241 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T04:02:51.241 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:02:51.241 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:02:51.241 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:02:51.241 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 7 2024-09-16T04:02:51.241 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:02:51.241 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:02:51.241 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:02:51.241 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T04:02:51.241 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T04:02:51.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.207+0000 7ff0bcefe640 10 librados: watch_flush enter 2024-09-16T04:02:51.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.207+0000 7ff0bcefe640 10 librados: watch_flush exit 2024-09-16T04:02:51.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.207+0000 7ff0bcefe640 20 client.825594.objecter shutdown clearing up homeless session... 2024-09-16T04:02:51.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.207+0000 7ff0bcefe640 10 client.825594.objecter successfully canceled tick 2024-09-16T04:02:51.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.207+0000 7ff0bcefe640 1 -- 172.21.15.42:0/3546973438 >> v2:172.21.15.154:6812/254845342 conn(0x7ff0980377c0 msgr2=0x7ff098039c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:02:51.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.207+0000 7ff0bcefe640 1 --2- 172.21.15.42:0/3546973438 >> v2:172.21.15.154:6812/254845342 conn(0x7ff0980377c0 0x7ff098039c80 secure :-1 s=READY pgs=3795 cs=0 l=1 rev1=1 crypto rx=0x7ff0b813df30 tx=0x7ff0a8006f90 comp rx=0 tx=0).stop 2024-09-16T04:02:51.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.207+0000 7ff0bcefe640 10 monclient: shutdown 2024-09-16T04:02:51.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.207+0000 7ff0bcefe640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:02:51.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.207+0000 7ff0bcefe640 1 -- 172.21.15.42:0/3546973438 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff0b80a3860 msgr2=0x7ff0b813d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:02:51.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.207+0000 7ff0bcefe640 1 --2- 172.21.15.42:0/3546973438 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff0b80a3860 0x7ff0b813d9c0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7ff0b00af040 tx=0x7ff0b0019670 comp rx=0 tx=0).stop 2024-09-16T04:02:51.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.207+0000 7ff0b6d9d640 1 -- 172.21.15.42:0/3546973438 reap_dead start 2024-09-16T04:02:51.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.207+0000 7ff0b6d9d640 1 -- 172.21.15.42:0/3546973438 reap_dead start 2024-09-16T04:02:51.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.207+0000 7ff0bcefe640 1 -- 172.21.15.42:0/3546973438 shutdown_connections 2024-09-16T04:02:51.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.207+0000 7ff0bcefe640 1 -- 172.21.15.42:0/3546973438 >> 172.21.15.42:0/3546973438 conn(0x7ff0b80a9f40 msgr2=0x7ff0b81307d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:02:51.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.207+0000 7ff0bcefe640 1 -- 172.21.15.42:0/3546973438 shutdown_connections 2024-09-16T04:02:51.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.207+0000 7ff0bcefe640 1 -- 172.21.15.42:0/3546973438 wait complete. 2024-09-16T04:02:51.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.207+0000 7ff0bcefe640 1 librados: shutdown 2024-09-16T04:02:51.246 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T04:02:51.246 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-16T04:02:51.246 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T04:02:51.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.367+0000 7fc9f9412640 10 monclient: get_monmap_and_config 2024-09-16T04:02:51.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.367+0000 7fc9f9412640 10 monclient: build_initial_monmap 2024-09-16T04:02:51.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.367+0000 7fc9f9412640 10 monclient: monmap: 2024-09-16T04:02:51.371 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:02:51.371 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:02:51.371 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:02:51.369094+0000 2024-09-16T04:02:51.371 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:02:51.369094+0000 2024-09-16T04:02:51.371 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:02:51.371 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:02:51.371 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:02:51.371 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:02:51.371 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:02:51.371 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:02:51.371 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:02:51.371 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:02:51.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.367+0000 7fc9f9412640 1 Processor -- start 2024-09-16T04:02:51.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.367+0000 7fc9f9412640 1 -- start start 2024-09-16T04:02:51.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.367+0000 7fc9f9412640 10 monclient: init 2024-09-16T04:02:51.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.367+0000 7fc9f9412640 10 monclient: _reopen_session rank -1 2024-09-16T04:02:51.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.367+0000 7fc9f9412640 10 monclient: _add_conns ranks=[4,2,1,3,0] 2024-09-16T04:02:51.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.367+0000 7fc9f9412640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fc9f40a1330 0x7fc9f40a1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:51.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.367+0000 7fc9f9412640 10 monclient(hunting): picked mon.noname-c con 0x7fc9f40a1330 addr v2:172.21.15.154:3300/0 2024-09-16T04:02:51.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.367+0000 7fc9f9412640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc9f406dac0 0x7fc9f40a1c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:51.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.367+0000 7fc9f9412640 10 monclient(hunting): picked mon.noname-b con 0x7fc9f406dac0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:02:51.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.367+0000 7fc9f9412640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc9f4135100 addr v1:172.21.15.154:6789/0 2024-09-16T04:02:51.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.367+0000 7fc9f9412640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc9f409c7d0 con 0x7fc9f4135100 2024-09-16T04:02:51.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.367+0000 7fc9f9412640 10 monclient(hunting): start opening mon connection 2024-09-16T04:02:51.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.367+0000 7fc9f9412640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fc9f409c090 con 0x7fc9f40a1330 2024-09-16T04:02:51.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.367+0000 7fc9f9412640 10 monclient(hunting): start opening mon connection 2024-09-16T04:02:51.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.367+0000 7fc9f9412640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fc9f409f610 con 0x7fc9f406dac0 2024-09-16T04:02:51.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.367+0000 7fc9f9412640 10 monclient(hunting): _renew_subs 2024-09-16T04:02:51.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.367+0000 7fc9f9412640 10 monclient(hunting): authenticate will time out at 2218.338852s 2024-09-16T04:02:51.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.367+0000 7fc9f27fc640 1 -- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc9f406dac0 msgr2=0x7fc9f40a1c50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T04:02:51.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.367+0000 7fc9f27fc640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc9f406dac0 0x7fc9f40a1c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:02:51.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.367+0000 7fc9f2ffd640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fc9f40a1330 0x7fc9f40a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:02:51.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.367+0000 7fc9f37fe640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7fc9f4135100 0x7fc9f4135550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.42:56548/0 (socket says 172.21.15.42:56548) 2024-09-16T04:02:51.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.367+0000 7fc9f2ffd640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fc9f40a1330 0x7fc9f40a1710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.42:37786/0 (socket says 172.21.15.42:37786) 2024-09-16T04:02:51.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.367+0000 7fc9f37fe640 1 -- 172.21.15.42:0/3205144889 learned_addr learned my addr 172.21.15.42:0/3205144889 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T04:02:51.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.367+0000 7fc9f2ffd640 10 monclient(hunting): get_auth_request con 0x7fc9f40a1330 auth_method 0 2024-09-16T04:02:51.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.367+0000 7fc9f2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:02:51.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.367+0000 7fc9f2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:02:51.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.367+0000 7fc9f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:02:51.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.367+0000 7fc9f2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:02:51.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.367+0000 7fc9f2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:02:51.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.367+0000 7fc9f2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:02:51.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.367+0000 7fc9f2ffd640 10 monclient(hunting): handle_auth_done global_id 904669 payload 274 2024-09-16T04:02:51.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.367+0000 7fc9f2ffd640 1 -- 172.21.15.42:0/3205144889 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc9f406dac0 msgr2=0x7fc9f40a1c50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:02:51.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.367+0000 7fc9f2ffd640 1 --2- 172.21.15.42:0/3205144889 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc9f406dac0 0x7fc9f40a1c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:51.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.367+0000 7fc9f2ffd640 1 -- 172.21.15.42:0/3205144889 >> v1:172.21.15.154:6789/0 conn(0x7fc9f4135100 legacy=0x7fc9f4135550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:02:51.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.367+0000 7fc9f2ffd640 10 monclient: _finish_hunting 0 2024-09-16T04:02:51.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.367+0000 7fc9f2ffd640 1 monclient: found mon.noname-c 2024-09-16T04:02:51.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.367+0000 7fc9f2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:02:51.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.367+0000 7fc9f2ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.154:3300/0 2024-09-16T04:02:51.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.367+0000 7fc9f2ffd640 1 -- 172.21.15.42:0/3205144889 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9f409f8f0 con 0x7fc9f40a1330 2024-09-16T04:02:51.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.371+0000 7fc9f2ffd640 1 --2- 172.21.15.42:0/3205144889 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc9f40a1330 0x7fc9f40a1710 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fc9dc0097c0 tx=0x7fc9dc01c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b153948e87309fc6 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:02:51.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.371+0000 7fc9f1ffb640 1 -- 172.21.15.42:0/3205144889 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9dc0b1070 con 0x7fc9f40a1330 2024-09-16T04:02:51.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.371+0000 7fc9f1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:02:51.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.371+0000 7fc9f1ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:02:51.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.371+0000 7fc9f1ffb640 10 monclient: dump: 2024-09-16T04:02:51.383 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:02:51.383 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:02:51.383 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:51.383 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:51.384 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:02:51.384 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:02:51.384 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:02:51.384 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:02:51.384 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:02:51.384 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:02:51.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.371+0000 7fc9f1ffb640 10 monclient: _finish_auth 0 2024-09-16T04:02:51.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.371+0000 7fc9f1ffb640 10 monclient: _check_auth_tickets 2024-09-16T04:02:51.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.371+0000 7fc9f1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:02:51.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.371+0000 7fc9f1ffb640 1 -- 172.21.15.42:0/3205144889 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc9dc0ac410 con 0x7fc9f40a1330 2024-09-16T04:02:51.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.371+0000 7fc9f1ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:02:51.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.371+0000 7fc9f9412640 5 monclient: authenticate success, global_id 904669 2024-09-16T04:02:51.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.371+0000 7fc9f9412640 10 monclient: get_monmap_and_config success 2024-09-16T04:02:51.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.371+0000 7fc9f1ffb640 1 -- 172.21.15.42:0/3205144889 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9dc0ac7e0 con 0x7fc9f40a1330 2024-09-16T04:02:51.385 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.371+0000 7fc9f9412640 4 set_mon_vals no callback set 2024-09-16T04:02:51.385 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.371+0000 7fc9f1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:02:51.385 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.371+0000 7fc9f1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:02:51.385 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.371+0000 7fc9f1ffb640 10 monclient: dump: 2024-09-16T04:02:51.385 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:02:51.385 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:02:51.385 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:51.385 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:51.385 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:02:51.385 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:02:51.385 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:02:51.385 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:02:51.385 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:02:51.385 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:02:51.386 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.371+0000 7fc9f9412640 10 monclient: shutdown 2024-09-16T04:02:51.386 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.371+0000 7fc9f9412640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:02:51.386 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.371+0000 7fc9f9412640 1 -- 172.21.15.42:0/3205144889 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc9f40a1330 msgr2=0x7fc9f40a1710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:02:51.386 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.371+0000 7fc9f9412640 1 --2- 172.21.15.42:0/3205144889 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc9f40a1330 0x7fc9f40a1710 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fc9dc0097c0 tx=0x7fc9dc01c4a0 comp rx=0 tx=0).stop 2024-09-16T04:02:51.386 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.371+0000 7fc9f9412640 1 -- 172.21.15.42:0/3205144889 shutdown_connections 2024-09-16T04:02:51.386 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.371+0000 7fc9f9412640 1 --2- 172.21.15.42:0/3205144889 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc9f406dac0 0x7fc9f40a1c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:51.386 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.371+0000 7fc9f9412640 1 --2- 172.21.15.42:0/3205144889 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc9f40a1330 0x7fc9f40a1710 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:51.386 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.371+0000 7fc9f9412640 1 -- 172.21.15.42:0/3205144889 >> 172.21.15.42:0/3205144889 conn(0x7fc9f40a2420 msgr2=0x7fc9f40a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:02:51.386 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.371+0000 7fc9f9412640 1 -- 172.21.15.42:0/3205144889 shutdown_connections 2024-09-16T04:02:51.386 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.371+0000 7fc9f9412640 1 -- 172.21.15.42:0/3205144889 wait complete. 2024-09-16T04:02:51.386 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.371+0000 7fc9f9412640 10 monclient: build_initial_monmap 2024-09-16T04:02:51.386 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.371+0000 7fc9f9412640 10 monclient: monmap: 2024-09-16T04:02:51.386 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:02:51.386 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:02:51.387 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:02:51.387 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:02:51.387 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:02:51.387 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:02:51.387 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:02:51.387 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:02:51.387 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:02:51.387 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:02:51.387 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.371+0000 7fc9f9412640 1 librados: starting msgr at 2024-09-16T04:02:51.387 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.371+0000 7fc9f9412640 1 librados: starting objecter 2024-09-16T04:02:51.387 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.375+0000 7fc9f9412640 1 Processor -- start 2024-09-16T04:02:51.387 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.375+0000 7fc9f9412640 1 -- start start 2024-09-16T04:02:51.387 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.375+0000 7fc9f9412640 1 librados: setting wanted keys 2024-09-16T04:02:51.387 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.375+0000 7fc9f9412640 1 librados: calling monclient init 2024-09-16T04:02:51.388 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.375+0000 7fc9f9412640 10 monclient: init 2024-09-16T04:02:51.388 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.375+0000 7fc9f9412640 10 monclient: _reopen_session rank -1 2024-09-16T04:02:51.388 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.375+0000 7fc9f9412640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:02:51.388 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.375+0000 7fc9f9412640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc9f40a1330 0x7fc9f41317c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:51.388 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.375+0000 7fc9f9412640 10 monclient(hunting): picked mon.noname-b con 0x7fc9f40a1330 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:02:51.388 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.375+0000 7fc9f9412640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc9f4135100 0x7fc9f409cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:51.388 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.375+0000 7fc9f9412640 10 monclient(hunting): picked mon.noname-c con 0x7fc9f4135100 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:02:51.388 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.375+0000 7fc9f9412640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc9f409d0d0 0x7fc9f409d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:51.388 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.375+0000 7fc9f9412640 10 monclient(hunting): picked mon.noname-a con 0x7fc9f409d0d0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:02:51.388 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.375+0000 7fc9f9412640 10 monclient(hunting): start opening mon connection 2024-09-16T04:02:51.388 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.375+0000 7fc9f9412640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fc9f413fd00 con 0x7fc9f409d0d0 2024-09-16T04:02:51.388 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.375+0000 7fc9f9412640 10 monclient(hunting): start opening mon connection 2024-09-16T04:02:51.388 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.375+0000 7fc9f9412640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc9f413fb80 con 0x7fc9f40a1330 2024-09-16T04:02:51.389 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.375+0000 7fc9f9412640 10 monclient(hunting): start opening mon connection 2024-09-16T04:02:51.389 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.375+0000 7fc9f9412640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fc9f413fe80 con 0x7fc9f4135100 2024-09-16T04:02:51.389 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.375+0000 7fc9f9412640 10 monclient(hunting): _renew_subs 2024-09-16T04:02:51.389 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.375+0000 7fc9f9412640 10 monclient(hunting): authenticate will time out at 2218.344664s 2024-09-16T04:02:51.389 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.375+0000 7fc9f27fc640 1 -- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc9f4135100 msgr2=0x7fc9f409cb90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T04:02:51.389 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.375+0000 7fc9f27fc640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc9f4135100 0x7fc9f409cb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:02:51.389 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.375+0000 7fc9f37fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc9f409d0d0 0x7fc9f409d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:02:51.389 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.375+0000 7fc9f37fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc9f409d0d0 0x7fc9f409d530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:43636/0 (socket says 172.21.15.42:43636) 2024-09-16T04:02:51.389 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.375+0000 7fc9f37fe640 1 -- 172.21.15.42:0/2578116790 learned_addr learned my addr 172.21.15.42:0/2578116790 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:02:51.389 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.375+0000 7fc9f37fe640 10 monclient(hunting): get_auth_request con 0x7fc9f409d0d0 auth_method 0 2024-09-16T04:02:51.389 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.375+0000 7fc9f37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:02:51.389 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.375+0000 7fc9f37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:02:51.389 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.375+0000 7fc9f37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:02:51.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.375+0000 7fc9f37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:02:51.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.375+0000 7fc9f37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:02:51.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.375+0000 7fc9f2ffd640 1 --2- 172.21.15.42:0/2578116790 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc9f40a1330 0x7fc9f41317c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:02:51.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.375+0000 7fc9f37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:02:51.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.375+0000 7fc9f2ffd640 10 monclient(hunting): get_auth_request con 0x7fc9f40a1330 auth_method 0 2024-09-16T04:02:51.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.375+0000 7fc9f2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:02:51.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.375+0000 7fc9f2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:02:51.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.375+0000 7fc9f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:02:51.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.375+0000 7fc9f2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:02:51.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.375+0000 7fc9f2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:02:51.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.375+0000 7fc9f2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:02:51.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.375+0000 7fc9f37fe640 10 monclient(hunting): handle_auth_done global_id 825603 payload 824 2024-09-16T04:02:51.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.375+0000 7fc9f37fe640 1 -- 172.21.15.42:0/2578116790 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc9f4135100 msgr2=0x7fc9f409cb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:02:51.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.375+0000 7fc9f37fe640 1 --2- 172.21.15.42:0/2578116790 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc9f4135100 0x7fc9f409cb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:51.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.375+0000 7fc9f37fe640 1 -- 172.21.15.42:0/2578116790 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc9f40a1330 msgr2=0x7fc9f41317c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:02:51.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.375+0000 7fc9f37fe640 1 --2- 172.21.15.42:0/2578116790 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc9f40a1330 0x7fc9f41317c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:02:51.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.375+0000 7fc9f37fe640 10 monclient: _finish_hunting 0 2024-09-16T04:02:51.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.375+0000 7fc9f37fe640 1 monclient: found mon.noname-a 2024-09-16T04:02:51.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.375+0000 7fc9f37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:02:51.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.375+0000 7fc9f37fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:02:51.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.375+0000 7fc9f37fe640 1 -- 172.21.15.42:0/2578116790 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9f409e830 con 0x7fc9f409d0d0 2024-09-16T04:02:51.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.375+0000 7fc9f37fe640 1 --2- 172.21.15.42:0/2578116790 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc9f409d0d0 0x7fc9f409d530 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7fc9ec00cbc0 tx=0x7fc9ec00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:02:51.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.375+0000 7fc9eb7fe640 10 client.?.objecter ms_handle_connect 0x7fc9f409d0d0 2024-09-16T04:02:51.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.375+0000 7fc9eb7fe640 10 client.?.objecter resend_mon_ops 2024-09-16T04:02:51.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.375+0000 7fc9eb7fe640 1 -- 172.21.15.42:0/2578116790 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9ec017070 con 0x7fc9f409d0d0 2024-09-16T04:02:51.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.375+0000 7fc9eb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:02:51.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.375+0000 7fc9eb7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:02:51.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.375+0000 7fc9eb7fe640 10 monclient: dump: 2024-09-16T04:02:51.392 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:02:51.392 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:02:51.392 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:51.392 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:51.392 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:02:51.392 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:02:51.392 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:02:51.392 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:02:51.392 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:02:51.392 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:02:51.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.375+0000 7fc9eb7fe640 10 monclient: _finish_auth 0 2024-09-16T04:02:51.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.375+0000 7fc9eb7fe640 10 monclient: _check_auth_tickets 2024-09-16T04:02:51.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.375+0000 7fc9eb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:02:51.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.375+0000 7fc9f9412640 5 monclient: authenticate success, global_id 825603 2024-09-16T04:02:51.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.375+0000 7fc9eb7fe640 1 -- 172.21.15.42:0/2578116790 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc9ec00ad40 con 0x7fc9f409d0d0 2024-09-16T04:02:51.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.375+0000 7fc9f9412640 10 monclient: _renew_subs 2024-09-16T04:02:51.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.375+0000 7fc9f9412640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:02:51.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.375+0000 7fc9f9412640 1 -- 172.21.15.42:0/2578116790 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9f40a5210 con 0x7fc9f409d0d0 2024-09-16T04:02:51.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.375+0000 7fc9eb7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:02:51.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.375+0000 7fc9eb7fe640 1 -- 172.21.15.42:0/2578116790 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9ec012600 con 0x7fc9f409d0d0 2024-09-16T04:02:51.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.375+0000 7fc9eb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:02:51.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.375+0000 7fc9eb7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:02:51.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.375+0000 7fc9eb7fe640 10 monclient: dump: 2024-09-16T04:02:51.393 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:02:51.393 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:02:51.394 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:51.394 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:51.394 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:02:51.394 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:02:51.394 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:02:51.394 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:02:51.394 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:02:51.394 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:02:51.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.375+0000 7fc9f0ff9640 4 set_mon_vals no callback set 2024-09-16T04:02:51.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.375+0000 7fc9f9412640 10 client.825603.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:02:51.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.375+0000 7fc9f9412640 10 monclient: _renew_subs 2024-09-16T04:02:51.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.375+0000 7fc9f9412640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:02:51.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.375+0000 7fc9f9412640 1 -- 172.21.15.42:0/2578116790 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9f41389a0 con 0x7fc9f409d0d0 2024-09-16T04:02:51.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.375+0000 7fc9f9412640 1 librados: init done 2024-09-16T04:02:51.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.375+0000 7fc9f9412640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:02:51.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.375+0000 7fc9f9412640 1 --2- 172.21.15.42:0/2578116790 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc9f406dac0 0x7fc9f41332f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:51.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.375+0000 7fc9f9412640 10 monclient(hunting): start opening mon connection 2024-09-16T04:02:51.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.375+0000 7fc9f9412640 1 -- 172.21.15.42:0/2578116790 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc9f4135510 con 0x7fc9f406dac0 2024-09-16T04:02:51.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.375+0000 7fc9eb7fe640 1 -- 172.21.15.42:0/2578116790 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fc9ec01a070 con 0x7fc9f409d0d0 2024-09-16T04:02:51.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.375+0000 7fc9f2ffd640 1 --2- 172.21.15.42:0/2578116790 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc9f406dac0 0x7fc9f41332f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:02:51.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.375+0000 7fc9f2ffd640 10 monclient: get_auth_request con 0x7fc9f406dac0 auth_method 0 2024-09-16T04:02:51.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.375+0000 7fc9f2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:02:51.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.375+0000 7fc9f2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:02:51.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.375+0000 7fc9f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:02:51.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.375+0000 7fc9eb7fe640 1 --2- 172.21.15.42:0/2578116790 >> v2:172.21.15.154:6812/254845342 conn(0x7fc9d40377c0 0x7fc9d4039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:51.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.375+0000 7fc9eb7fe640 1 -- 172.21.15.42:0/2578116790 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(633..633 src has 451..633) ==== 6359+0+0 (secure 0 0 0) 0x7fc9ec048ea0 con 0x7fc9f409d0d0 2024-09-16T04:02:51.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.375+0000 7fc9eb7fe640 10 client.825603.objecter ms_dispatch 0x7fc9f4000ba0 osd_map(633..633 src has 451..633) 2024-09-16T04:02:51.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.375+0000 7fc9eb7fe640 3 client.825603.objecter handle_osd_map got epochs [633,633] > 0 2024-09-16T04:02:51.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.375+0000 7fc9eb7fe640 3 client.825603.objecter handle_osd_map decoding full epoch 633 2024-09-16T04:02:51.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.375+0000 7fc9f2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:02:51.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.375+0000 7fc9f2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:02:51.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.375+0000 7fc9f2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:02:51.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.375+0000 7fc9eb7fe640 20 client.825603.objecter dump_active .. 0 homeless 2024-09-16T04:02:51.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.375+0000 7fc9f27fc640 1 --2- 172.21.15.42:0/2578116790 >> v2:172.21.15.154:6812/254845342 conn(0x7fc9d40377c0 0x7fc9d4039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:02:51.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.375+0000 7fc9f27fc640 10 monclient: get_auth_request con 0x7fc9d40377c0 auth_method 0 2024-09-16T04:02:51.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.375+0000 7fc9f2ffd640 10 monclient(hunting): handle_auth_done global_id 904678 payload 824 2024-09-16T04:02:51.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.375+0000 7fc9f2ffd640 1 -- 172.21.15.42:0/2578116790 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc9f4139990 con 0x7fc9f406dac0 2024-09-16T04:02:51.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.375+0000 7fc9f2ffd640 1 --2- 172.21.15.42:0/2578116790 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc9f406dac0 0x7fc9f41332f0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fc9f409bb00 tx=0x7fc9dc019240 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:02:51.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.375+0000 7fc9eb7fe640 10 client.825603.objecter ms_handle_connect 0x7fc9f406dac0 2024-09-16T04:02:51.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.375+0000 7fc9eb7fe640 10 client.825603.objecter resend_mon_ops 2024-09-16T04:02:51.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.375+0000 7fc9eb7fe640 1 -- 172.21.15.42:0/2578116790 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9dc0bc070 con 0x7fc9f406dac0 2024-09-16T04:02:51.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.375+0000 7fc9eb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:02:51.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.375+0000 7fc9eb7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:02:51.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.375+0000 7fc9eb7fe640 10 monclient: dump: 2024-09-16T04:02:51.397 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:02:51.397 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:02:51.397 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:51.397 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:51.397 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:02:51.397 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:02:51.397 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:02:51.397 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:02:51.397 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:02:51.397 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:02:51.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.379+0000 7fc9eb7fe640 10 client.825603.objecter ms_handle_connect 0x7fc9d40377c0 2024-09-16T04:02:51.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.379+0000 7fc9f27fc640 1 --2- 172.21.15.42:0/2578116790 >> v2:172.21.15.154:6812/254845342 conn(0x7fc9d40377c0 0x7fc9d4039c80 secure :-1 s=READY pgs=3796 cs=0 l=1 rev1=1 crypto rx=0x7fc9f41385d0 tx=0x7fc9e4006d90 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:02:51.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.379+0000 7fc9eb7fe640 1 -- 172.21.15.42:0/2578116790 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fc9dc0acc60 con 0x7fc9f406dac0 2024-09-16T04:02:51.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.379+0000 7fc9eb7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:02:51.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.379+0000 7fc9eb7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:02:51.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.379+0000 7fc9eb7fe640 1 -- 172.21.15.42:0/2578116790 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc9f406dac0 msgr2=0x7fc9f41332f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:02:51.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.379+0000 7fc9eb7fe640 1 --2- 172.21.15.42:0/2578116790 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc9f406dac0 0x7fc9f41332f0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fc9f409bb00 tx=0x7fc9dc019240 comp rx=0 tx=0).stop 2024-09-16T04:02:51.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.379+0000 7fc9eb7fe640 1 -- 172.21.15.42:0/2578116790 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc9f406dac0 msgr2=0x7fc9f41332f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:02:51.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.379+0000 7fc9eb7fe640 1 --2- 172.21.15.42:0/2578116790 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc9f406dac0 0x7fc9f41332f0 secure :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fc9f409bb00 tx=0x7fc9dc019240 comp rx=0 tx=0).stop 2024-09-16T04:02:51.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.383+0000 7fc9f9412640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:02:51.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.383+0000 7fc9f9412640 1 --2- 172.21.15.42:0/2578116790 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc9f4133830 0x7fc9f4133c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:02:51.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.383+0000 7fc9f9412640 10 monclient(hunting): start opening mon connection 2024-09-16T04:02:51.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.383+0000 7fc9f9412640 1 -- 172.21.15.42:0/2578116790 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc9f4134180 con 0x7fc9f4133830 2024-09-16T04:02:51.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.383+0000 7fc9f2ffd640 1 --2- 172.21.15.42:0/2578116790 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc9f4133830 0x7fc9f4133c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:02:51.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.383+0000 7fc9f2ffd640 10 monclient: get_auth_request con 0x7fc9f4133830 auth_method 0 2024-09-16T04:02:51.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.383+0000 7fc9f2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:02:51.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.383+0000 7fc9f2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:02:51.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.383+0000 7fc9f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:02:51.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.383+0000 7fc9f2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:02:51.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.383+0000 7fc9f2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:02:51.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.383+0000 7fc9f2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:02:51.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.383+0000 7fc9f2ffd640 10 monclient(hunting): handle_auth_done global_id 904681 payload 824 2024-09-16T04:02:51.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.383+0000 7fc9f2ffd640 1 -- 172.21.15.42:0/2578116790 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fc9f4139db0 con 0x7fc9f4133830 2024-09-16T04:02:51.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.387+0000 7fc9f2ffd640 1 --2- 172.21.15.42:0/2578116790 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc9f4133830 0x7fc9f4133c40 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fc9dc01a0d0 tx=0x7fc9dc0a6040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:02:51.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.387+0000 7fc9eb7fe640 10 client.825603.objecter ms_handle_connect 0x7fc9f4133830 2024-09-16T04:02:51.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.387+0000 7fc9eb7fe640 10 client.825603.objecter resend_mon_ops 2024-09-16T04:02:51.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.387+0000 7fc9eb7fe640 1 -- 172.21.15.42:0/2578116790 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9dc0cc020 con 0x7fc9f4133830 2024-09-16T04:02:51.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.387+0000 7fc9eb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:02:51.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.387+0000 7fc9eb7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:02:51.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.387+0000 7fc9eb7fe640 10 monclient: dump: 2024-09-16T04:02:51.401 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:02:51.401 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:02:51.401 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:51.402 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:02:51.402 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:02:51.402 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:02:51.402 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:02:51.402 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:02:51.402 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:02:51.402 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:02:51.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.387+0000 7fc9eb7fe640 1 -- 172.21.15.42:0/2578116790 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4869 (secure 0 0 0) 0x7fc9dc0b63e0 con 0x7fc9f4133830 2024-09-16T04:02:51.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.387+0000 7fc9eb7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:02:51.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.387+0000 7fc9eb7fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:02:51.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.387+0000 7fc9eb7fe640 1 -- 172.21.15.42:0/2578116790 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc9f4133830 msgr2=0x7fc9f4133c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:02:51.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.387+0000 7fc9eb7fe640 1 --2- 172.21.15.42:0/2578116790 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc9f4133830 0x7fc9f4133c40 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fc9dc01a0d0 tx=0x7fc9dc0a6040 comp rx=0 tx=0).stop 2024-09-16T04:02:51.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.387+0000 7fc9eb7fe640 1 -- 172.21.15.42:0/2578116790 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc9f4133830 msgr2=0x7fc9f4133c40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:02:51.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.387+0000 7fc9eb7fe640 1 --2- 172.21.15.42:0/2578116790 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc9f4133830 0x7fc9f4133c40 secure :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fc9dc01a0d0 tx=0x7fc9dc0a6040 comp rx=0 tx=0).stop 2024-09-16T04:02:51.403 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T04:02:51.403 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:02:51.403 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:02:51.403 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T04:02:51.403 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 188, 2024-09-16T04:02:51.403 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 299562, 2024-09-16T04:02:51.403 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T04:02:51.403 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T04:02:51.403 INFO:teuthology.orchestra.run.smithi042.stdout: 1 2024-09-16T04:02:51.404 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:02:51.404 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 0, 2024-09-16T04:02:51.404 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:02:51.404 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T04:02:51.404 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T04:02:51.404 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:02:51.404 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:02:51.404 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:02:51.404 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:02:51.404 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:02:51.404 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:02:51.404 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:02:51.404 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:02:51.404 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:02:51.405 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:02:51.405 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:02:51.405 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:02:51.405 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:02:51.405 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T04:02:51.405 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:02:51.405 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:02:51.405 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:02:51.405 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:02:51.405 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:02:51.405 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:02:51.405 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:02:51.405 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:02:51.406 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:02:51.406 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:02:51.406 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:02:51.406 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:02:51.406 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:02:51.406 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T04:02:51.406 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T04:02:51.406 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T04:02:51.406 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T04:02:51.406 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T04:02:51.406 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T04:02:51.406 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:02:51.406 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:02:51.406 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T04:02:51.407 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T04:02:51.407 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T04:02:51.407 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T04:02:51.407 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T04:02:51.407 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T04:02:51.407 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T04:02:51.407 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:02:51.407 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T04:02:51.407 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:02:51.407 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:02:51.407 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:02:51.407 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:02:51.407 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:02:51.408 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:02:51.408 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:02:51.408 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:02:51.408 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:02:51.408 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:02:51.408 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:02:51.408 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:02:51.408 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T04:02:51.408 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:02:51.408 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T04:02:51.408 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:02:51.408 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:02:51.408 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:02:51.409 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:02:51.409 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:02:51.409 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:02:51.409 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:02:51.409 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T04:02:51.409 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:02:51.409 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:02:51.409 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:02:51.409 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:02:51.409 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T04:02:51.409 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:02:51.409 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:02:51.409 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:02:51.410 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:02:51.410 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T04:02:51.410 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T04:02:51.410 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:02:51.410 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:02:51.410 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:02:51.410 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:02:51.410 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:02:51.410 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:02:51.410 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:02:51.410 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:02:51.410 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:02:51.410 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:02:51.411 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:02:51.411 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T04:02:51.411 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:02:51.411 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:02:51.411 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:02:51.411 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:02:51.411 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T04:02:51.411 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:02:51.411 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:02:51.411 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:02:51.411 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:02:51.411 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T04:02:51.411 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T04:02:51.411 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:02:51.412 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:02:51.412 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:02:51.412 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:02:51.412 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:02:51.412 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:02:51.412 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:02:51.412 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:02:51.412 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:02:51.412 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:02:51.412 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:02:51.412 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T04:02:51.412 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:02:51.412 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:02:51.412 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:02:51.413 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:02:51.413 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T04:02:51.413 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:02:51.413 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:02:51.413 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:02:51.413 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:02:51.413 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T04:02:51.413 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T04:02:51.413 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:02:51.413 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:02:51.413 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:02:51.413 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:02:51.413 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:02:51.413 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:02:51.414 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T04:02:51.414 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T04:02:51.414 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:02:51.414 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:02:51.414 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:02:51.414 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:02:51.414 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:02:51.414 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:02:51.414 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T04:02:51.414 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:02:51.414 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:02:51.414 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:02:51.414 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:02:51.415 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:02:51.415 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:02:51.415 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T04:02:51.415 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:02:51.415 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:02:51.415 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:02:51.415 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 2 2024-09-16T04:02:51.415 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:02:51.415 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:02:51.415 INFO:teuthology.orchestra.run.smithi042.stdout: "mgr": [ 2024-09-16T04:02:51.415 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:02:51.415 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:02:51.415 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:02:51.415 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:02:51.416 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:02:51.416 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:02:51.416 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:02:51.416 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T04:02:51.416 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T04:02:51.418 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.387+0000 7fc9f9412640 10 librados: watch_flush enter 2024-09-16T04:02:51.418 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.387+0000 7fc9f9412640 10 librados: watch_flush exit 2024-09-16T04:02:51.418 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.387+0000 7fc9f9412640 20 client.825603.objecter shutdown clearing up homeless session... 2024-09-16T04:02:51.418 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.387+0000 7fc9f9412640 10 client.825603.objecter successfully canceled tick 2024-09-16T04:02:51.418 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.387+0000 7fc9f9412640 1 -- 172.21.15.42:0/2578116790 >> v2:172.21.15.154:6812/254845342 conn(0x7fc9d40377c0 msgr2=0x7fc9d4039c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:02:51.418 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.387+0000 7fc9f9412640 1 --2- 172.21.15.42:0/2578116790 >> v2:172.21.15.154:6812/254845342 conn(0x7fc9d40377c0 0x7fc9d4039c80 secure :-1 s=READY pgs=3796 cs=0 l=1 rev1=1 crypto rx=0x7fc9f41385d0 tx=0x7fc9e4006d90 comp rx=0 tx=0).stop 2024-09-16T04:02:51.418 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.387+0000 7fc9f9412640 10 monclient: shutdown 2024-09-16T04:02:51.418 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.387+0000 7fc9f9412640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:02:51.418 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.387+0000 7fc9f9412640 1 -- 172.21.15.42:0/2578116790 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc9f409d0d0 msgr2=0x7fc9f409d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:02:51.418 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.387+0000 7fc9f9412640 1 --2- 172.21.15.42:0/2578116790 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc9f409d0d0 0x7fc9f409d530 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7fc9ec00cbc0 tx=0x7fc9ec00a430 comp rx=0 tx=0).stop 2024-09-16T04:02:51.418 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.387+0000 7fc9f37fe640 1 -- 172.21.15.42:0/2578116790 reap_dead start 2024-09-16T04:02:51.418 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.387+0000 7fc9f37fe640 1 -- 172.21.15.42:0/2578116790 reap_dead start 2024-09-16T04:02:51.419 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.387+0000 7fc9f9412640 1 -- 172.21.15.42:0/2578116790 shutdown_connections 2024-09-16T04:02:51.419 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.387+0000 7fc9f9412640 1 -- 172.21.15.42:0/2578116790 >> 172.21.15.42:0/2578116790 conn(0x7fc9f40a2420 msgr2=0x7fc9f40a3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:02:51.419 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.387+0000 7fc9f9412640 1 -- 172.21.15.42:0/2578116790 shutdown_connections 2024-09-16T04:02:51.419 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.387+0000 7fc9f9412640 1 -- 172.21.15.42:0/2578116790 wait complete. 2024-09-16T04:02:51.419 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:02:51.387+0000 7fc9f9412640 1 librados: shutdown 2024-09-16T04:02:51.419 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T04:02:51.419 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-16T04:03:11.420 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-16T04:03:11.420 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-09-16T04:03:11.420 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-16T04:03:11.421 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-16T04:03:11.424 INFO:tasks.ceph.mon.c:Started 2024-09-16T04:03:11.424 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T04:03:11.424 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:03:11.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.543+0000 7f264d61f640 10 monclient: get_monmap_and_config 2024-09-16T04:03:11.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.543+0000 7f264d61f640 10 monclient: build_initial_monmap 2024-09-16T04:03:11.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.543+0000 7f264d61f640 10 monclient: monmap: 2024-09-16T04:03:11.546 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:03:11.546 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:03:11.546 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:03:11.544861+0000 2024-09-16T04:03:11.547 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:03:11.544861+0000 2024-09-16T04:03:11.547 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:03:11.547 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:03:11.547 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:03:11.547 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:03:11.547 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:03:11.547 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:03:11.547 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:03:11.547 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:03:11.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.543+0000 7f264d61f640 1 Processor -- start 2024-09-16T04:03:11.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.543+0000 7f264d61f640 1 -- start start 2024-09-16T04:03:11.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.543+0000 7f264d61f640 10 monclient: init 2024-09-16T04:03:11.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.543+0000 7f264d61f640 10 monclient: _reopen_session rank -1 2024-09-16T04:03:11.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.543+0000 7f264d61f640 10 monclient: _add_conns ranks=[3,4,0,1,2] 2024-09-16T04:03:11.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.543+0000 7f264d61f640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f26480a39b0 0x7f2648133b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:03:11.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.543+0000 7f264d61f640 10 monclient(hunting): picked mon.noname-a con 0x7f26480a39b0 addr v2:172.21.15.42:3300/0 2024-09-16T04:03:11.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.543+0000 7f264d61f640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f26480a3d90 0x7f2648137000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:03:11.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.543+0000 7f264d61f640 10 monclient(hunting): picked mon.noname-c con 0x7f26480a3d90 addr v2:172.21.15.154:3300/0 2024-09-16T04:03:11.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.543+0000 7f264d61f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2648137540 addr v1:172.21.15.42:6789/0 2024-09-16T04:03:11.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.543+0000 7f264d61f640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f264809c140 con 0x7f2648137540 2024-09-16T04:03:11.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.543+0000 7f264d61f640 10 monclient(hunting): start opening mon connection 2024-09-16T04:03:11.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.543+0000 7f264d61f640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f264809c420 con 0x7f26480a39b0 2024-09-16T04:03:11.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.543+0000 7f264d61f640 10 monclient(hunting): start opening mon connection 2024-09-16T04:03:11.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.543+0000 7f264d61f640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f264806dd60 con 0x7f26480a3d90 2024-09-16T04:03:11.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.543+0000 7f264d61f640 10 monclient(hunting): _renew_subs 2024-09-16T04:03:11.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.543+0000 7f264d61f640 10 monclient(hunting): authenticate will time out at 2238.513955s 2024-09-16T04:03:11.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.543+0000 7f2646ffd640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f26480a39b0 0x7f2648133b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:03:11.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.543+0000 7f26477fe640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f2648137540 0x7f264813e390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:60474/0 (socket says 172.21.15.42:60474) 2024-09-16T04:03:11.551 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.543+0000 7f26477fe640 1 -- 172.21.15.42:0/1184188948 learned_addr learned my addr 172.21.15.42:0/1184188948 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T04:03:11.551 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.543+0000 7f2646ffd640 10 monclient(hunting): get_auth_request con 0x7f26480a39b0 auth_method 0 2024-09-16T04:03:11.551 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.543+0000 7f2646ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:03:11.551 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.543+0000 7f2646ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:03:11.551 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.543+0000 7f2646ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:03:11.551 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.543+0000 7f2646ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:03:11.551 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.543+0000 7f2646ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:03:11.551 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.543+0000 7f2646ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:03:11.551 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.543+0000 7f2646ffd640 10 monclient(hunting): handle_auth_done global_id 825606 payload 274 2024-09-16T04:03:11.551 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.543+0000 7f2646ffd640 1 -- 172.21.15.42:0/1184188948 >> v2:172.21.15.154:3300/0 conn(0x7f26480a3d90 msgr2=0x7f2648137000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:03:11.551 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.543+0000 7f2646ffd640 1 --2- 172.21.15.42:0/1184188948 >> v2:172.21.15.154:3300/0 conn(0x7f26480a3d90 0x7f2648137000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:03:11.551 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.543+0000 7f2646ffd640 1 -- 172.21.15.42:0/1184188948 >> v1:172.21.15.42:6789/0 conn(0x7f2648137540 legacy=0x7f264813e390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:03:11.551 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.543+0000 7f2646ffd640 10 monclient: _finish_hunting 0 2024-09-16T04:03:11.551 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.543+0000 7f2646ffd640 1 monclient: found mon.noname-a 2024-09-16T04:03:11.552 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.543+0000 7f2646ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:03:11.552 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.543+0000 7f2646ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:03:11.552 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.543+0000 7f2646ffd640 1 -- 172.21.15.42:0/1184188948 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26480a5e20 con 0x7f26480a39b0 2024-09-16T04:03:11.557 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.547+0000 7f2646ffd640 1 --2- 172.21.15.42:0/1184188948 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f26480a39b0 0x7f2648133b80 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f26480a56c0 tx=0x7f263801c340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54b1c157f15348e9 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:03:11.557 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.547+0000 7f2645ffb640 1 -- 172.21.15.42:0/1184188948 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26380b0070 con 0x7f26480a39b0 2024-09-16T04:03:11.557 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.547+0000 7f2645ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:03:11.557 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.547+0000 7f2645ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:03:11.557 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.547+0000 7f2645ffb640 10 monclient: dump: 2024-09-16T04:03:11.557 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:03:11.557 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:03:11.558 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:11.558 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:11.558 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:03:11.558 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:03:11.558 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:03:11.558 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:03:11.558 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:03:11.558 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:03:11.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.547+0000 7f2645ffb640 10 monclient: _finish_auth 0 2024-09-16T04:03:11.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.547+0000 7f2645ffb640 10 monclient: _check_auth_tickets 2024-09-16T04:03:11.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.547+0000 7f2645ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:03:11.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.547+0000 7f2645ffb640 1 -- 172.21.15.42:0/1184188948 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f26380ab600 con 0x7f26480a39b0 2024-09-16T04:03:11.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.547+0000 7f264d61f640 5 monclient: authenticate success, global_id 825606 2024-09-16T04:03:11.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.547+0000 7f264d61f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:03:11.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.547+0000 7f2645ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:03:11.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.547+0000 7f264d61f640 10 monclient: get_monmap_and_config success 2024-09-16T04:03:11.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.547+0000 7f2645ffb640 1 -- 172.21.15.42:0/1184188948 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26380ab9d0 con 0x7f26480a39b0 2024-09-16T04:03:11.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.547+0000 7f2645ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:03:11.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.547+0000 7f264d61f640 4 set_mon_vals no callback set 2024-09-16T04:03:11.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.547+0000 7f2645ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:03:11.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.547+0000 7f2645ffb640 10 monclient: dump: 2024-09-16T04:03:11.559 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:03:11.559 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:03:11.559 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:11.559 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:11.559 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:03:11.559 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:03:11.560 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:03:11.560 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:03:11.560 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:03:11.560 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:03:11.560 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.547+0000 7f264d61f640 10 monclient: shutdown 2024-09-16T04:03:11.560 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.547+0000 7f264d61f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:03:11.560 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.547+0000 7f264d61f640 1 -- 172.21.15.42:0/1184188948 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f26480a39b0 msgr2=0x7f2648133b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:03:11.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.547+0000 7f264d61f640 1 --2- 172.21.15.42:0/1184188948 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f26480a39b0 0x7f2648133b80 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f26480a56c0 tx=0x7f263801c340 comp rx=0 tx=0).stop 2024-09-16T04:03:11.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.547+0000 7f264d61f640 1 -- 172.21.15.42:0/1184188948 shutdown_connections 2024-09-16T04:03:11.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.547+0000 7f264d61f640 1 --2- 172.21.15.42:0/1184188948 >> v2:172.21.15.154:3300/0 conn(0x7f26480a3d90 0x7f2648137000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:03:11.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.547+0000 7f264d61f640 1 --2- 172.21.15.42:0/1184188948 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f26480a39b0 0x7f2648133b80 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:03:11.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.547+0000 7f264d61f640 1 -- 172.21.15.42:0/1184188948 >> 172.21.15.42:0/1184188948 conn(0x7f26480a9fb0 msgr2=0x7f26480aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:03:11.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.547+0000 7f264d61f640 1 -- 172.21.15.42:0/1184188948 shutdown_connections 2024-09-16T04:03:11.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.547+0000 7f264d61f640 1 -- 172.21.15.42:0/1184188948 wait complete. 2024-09-16T04:03:11.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.547+0000 7f264d61f640 10 monclient: build_initial_monmap 2024-09-16T04:03:11.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.547+0000 7f264d61f640 10 monclient: monmap: 2024-09-16T04:03:11.561 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:03:11.561 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:03:11.561 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:03:11.561 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:03:11.562 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:03:11.562 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:03:11.562 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:03:11.562 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:03:11.562 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:03:11.562 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:03:11.562 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.547+0000 7f264d61f640 1 librados: starting msgr at 2024-09-16T04:03:11.562 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.547+0000 7f264d61f640 1 librados: starting objecter 2024-09-16T04:03:11.562 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.547+0000 7f264d61f640 1 Processor -- start 2024-09-16T04:03:11.562 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.547+0000 7f264d61f640 1 -- start start 2024-09-16T04:03:11.562 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.547+0000 7f264d61f640 1 librados: setting wanted keys 2024-09-16T04:03:11.562 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.547+0000 7f264d61f640 1 librados: calling monclient init 2024-09-16T04:03:11.562 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.547+0000 7f264d61f640 10 monclient: init 2024-09-16T04:03:11.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.547+0000 7f264d61f640 10 monclient: _reopen_session rank -1 2024-09-16T04:03:11.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.547+0000 7f264d61f640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:03:11.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.547+0000 7f264d61f640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f26480a39b0 0x7f26480a20d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:03:11.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.547+0000 7f264d61f640 10 monclient(hunting): picked mon.noname-c con 0x7f26480a39b0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:03:11.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.547+0000 7f264d61f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f26480a3d90 0x7f264813d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:03:11.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.547+0000 7f264d61f640 10 monclient(hunting): picked mon.noname-b con 0x7f26480a3d90 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:03:11.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.547+0000 7f264d61f640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2648137540 0x7f264813daf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:03:11.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.547+0000 7f264d61f640 10 monclient(hunting): picked mon.noname-a con 0x7f2648137540 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:03:11.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.547+0000 7f264d61f640 10 monclient(hunting): start opening mon connection 2024-09-16T04:03:11.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.547+0000 7f264d61f640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f26481424e0 con 0x7f2648137540 2024-09-16T04:03:11.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.547+0000 7f264d61f640 10 monclient(hunting): start opening mon connection 2024-09-16T04:03:11.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.547+0000 7f264d61f640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f2648142660 con 0x7f26480a3d90 2024-09-16T04:03:11.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.547+0000 7f264d61f640 10 monclient(hunting): start opening mon connection 2024-09-16T04:03:11.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.547+0000 7f264d61f640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f26481427e0 con 0x7f26480a39b0 2024-09-16T04:03:11.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.547+0000 7f264d61f640 10 monclient(hunting): _renew_subs 2024-09-16T04:03:11.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.547+0000 7f264d61f640 10 monclient(hunting): authenticate will time out at 2238.518831s 2024-09-16T04:03:11.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.547+0000 7f2646ffd640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f26480a39b0 0x7f26480a20d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:03:11.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.547+0000 7f26477fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2648137540 0x7f264813daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:03:11.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.547+0000 7f2646ffd640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f26480a39b0 0x7f26480a20d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:51284/0 (socket says 172.21.15.42:51284) 2024-09-16T04:03:11.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.547+0000 7f26477fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2648137540 0x7f264813daf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:51914/0 (socket says 172.21.15.42:51914) 2024-09-16T04:03:11.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.547+0000 7f2646ffd640 1 -- 172.21.15.42:0/1799016047 learned_addr learned my addr 172.21.15.42:0/1799016047 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:03:11.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.547+0000 7f2646ffd640 10 monclient(hunting): get_auth_request con 0x7f26480a39b0 auth_method 0 2024-09-16T04:03:11.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.547+0000 7f2646ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:03:11.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.547+0000 7f2646ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:03:11.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.547+0000 7f2646ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:03:11.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.547+0000 7f26477fe640 10 monclient(hunting): get_auth_request con 0x7f2648137540 auth_method 0 2024-09-16T04:03:11.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.547+0000 7f26477fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:03:11.565 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.547+0000 7f26477fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:03:11.565 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.547+0000 7f26477fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:03:11.565 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.547+0000 7f26477fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:03:11.565 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.547+0000 7f26477fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:03:11.565 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.547+0000 7f2646ffd640 1 -- 172.21.15.42:0/1799016047 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f26480a39b0 msgr2=0x7f26480a20d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-16T04:03:11.565 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.547+0000 7f2646ffd640 1 -- 172.21.15.42:0/1799016047 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f26480a39b0 msgr2=0x7f26480a20d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:03:11.565 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.547+0000 7f2646ffd640 1 --2- 172.21.15.42:0/1799016047 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f26480a39b0 0x7f26480a20d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:03:11.565 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.547+0000 7f26477fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:03:11.565 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.547+0000 7f2646ffd640 1 --2- 172.21.15.42:0/1799016047 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f26480a39b0 0x7f26480a20d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:03:11.565 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.547+0000 7f26467fc640 1 --2- 172.21.15.42:0/1799016047 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f26480a3d90 0x7f264813d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:03:11.565 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.547+0000 7f26467fc640 10 monclient(hunting): get_auth_request con 0x7f26480a3d90 auth_method 0 2024-09-16T04:03:11.565 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.547+0000 7f26467fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:03:11.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.547+0000 7f26467fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:03:11.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.547+0000 7f26467fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:03:11.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.547+0000 7f26477fe640 10 monclient(hunting): handle_auth_done global_id 825612 payload 824 2024-09-16T04:03:11.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.547+0000 7f26477fe640 1 -- 172.21.15.42:0/1799016047 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f26480a39b0 msgr2=0x7f26480a20d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:03:11.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.547+0000 7f26477fe640 1 --2- 172.21.15.42:0/1799016047 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f26480a39b0 0x7f26480a20d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:03:11.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.547+0000 7f26477fe640 1 -- 172.21.15.42:0/1799016047 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f26480a3d90 msgr2=0x7f264813d5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:03:11.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.547+0000 7f26477fe640 1 --2- 172.21.15.42:0/1799016047 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f26480a3d90 0x7f264813d5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:03:11.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.547+0000 7f26477fe640 10 monclient: _finish_hunting 0 2024-09-16T04:03:11.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.547+0000 7f26477fe640 1 monclient: found mon.noname-a 2024-09-16T04:03:11.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.547+0000 7f26477fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:03:11.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.547+0000 7f26477fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:03:11.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.547+0000 7f26477fe640 1 -- 172.21.15.42:0/1799016047 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f264809e8a0 con 0x7f2648137540 2024-09-16T04:03:11.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.547+0000 7f26477fe640 1 --2- 172.21.15.42:0/1799016047 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2648137540 0x7f264813daf0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f264000c3c0 tx=0x7f264000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:03:11.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.547+0000 7f262f7fe640 10 client.?.objecter ms_handle_connect 0x7f2648137540 2024-09-16T04:03:11.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.547+0000 7f262f7fe640 10 client.?.objecter resend_mon_ops 2024-09-16T04:03:11.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.547+0000 7f262f7fe640 1 -- 172.21.15.42:0/1799016047 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2640017070 con 0x7f2648137540 2024-09-16T04:03:11.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.547+0000 7f262f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:03:11.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.547+0000 7f262f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:03:11.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.547+0000 7f262f7fe640 10 monclient: dump: 2024-09-16T04:03:11.567 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:03:11.567 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:03:11.567 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:11.567 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:11.567 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:03:11.567 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:03:11.567 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:03:11.567 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:03:11.568 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:03:11.568 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:03:11.568 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.547+0000 7f262f7fe640 10 monclient: _finish_auth 0 2024-09-16T04:03:11.568 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.547+0000 7f262f7fe640 10 monclient: _check_auth_tickets 2024-09-16T04:03:11.568 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.547+0000 7f262f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:03:11.568 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.547+0000 7f264d61f640 5 monclient: authenticate success, global_id 825612 2024-09-16T04:03:11.568 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.547+0000 7f262f7fe640 1 -- 172.21.15.42:0/1799016047 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2640005cf0 con 0x7f2648137540 2024-09-16T04:03:11.568 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.547+0000 7f262f7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:03:11.568 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.547+0000 7f264d61f640 10 monclient: _renew_subs 2024-09-16T04:03:11.568 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.547+0000 7f262f7fe640 1 -- 172.21.15.42:0/1799016047 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26400072d0 con 0x7f2648137540 2024-09-16T04:03:11.568 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.547+0000 7f264d61f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:03:11.568 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.547+0000 7f264d61f640 1 -- 172.21.15.42:0/1799016047 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f264809eb80 con 0x7f2648137540 2024-09-16T04:03:11.568 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.547+0000 7f262f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:03:11.568 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.547+0000 7f262f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:03:11.569 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.547+0000 7f262f7fe640 10 monclient: dump: 2024-09-16T04:03:11.569 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:03:11.569 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:03:11.569 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:11.569 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:11.569 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:03:11.569 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:03:11.569 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:03:11.569 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:03:11.569 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:03:11.569 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:03:11.569 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.547+0000 7f26457fa640 4 set_mon_vals no callback set 2024-09-16T04:03:11.569 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.547+0000 7f264d61f640 10 client.825612.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:03:11.570 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.547+0000 7f264d61f640 10 monclient: _renew_subs 2024-09-16T04:03:11.570 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.547+0000 7f264d61f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:03:11.570 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.547+0000 7f264d61f640 1 -- 172.21.15.42:0/1799016047 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f264809e5c0 con 0x7f2648137540 2024-09-16T04:03:11.570 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.547+0000 7f264d61f640 1 librados: init done 2024-09-16T04:03:11.570 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.551+0000 7f264d61f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:03:11.570 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.551+0000 7f264d61f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:03:11.570 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.551+0000 7f264d61f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:03:11.570 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.551+0000 7f264d61f640 1 -- 172.21.15.42:0/1799016047 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2648133b80 con 0x7f2648137540 2024-09-16T04:03:11.570 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.551+0000 7f262f7fe640 1 -- 172.21.15.42:0/1799016047 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f264001a070 con 0x7f2648137540 2024-09-16T04:03:11.570 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.551+0000 7f262f7fe640 1 --2- 172.21.15.42:0/1799016047 >> v2:172.21.15.154:6812/254845342 conn(0x7f2634037810 0x7f2634039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:03:11.570 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.551+0000 7f262f7fe640 1 -- 172.21.15.42:0/1799016047 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(650..650 src has 451..650) ==== 6306+0+0 (secure 0 0 0) 0x7f2640048440 con 0x7f2648137540 2024-09-16T04:03:11.570 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.551+0000 7f262f7fe640 10 client.825612.objecter ms_dispatch 0x7f2648000ba0 osd_map(650..650 src has 451..650) 2024-09-16T04:03:11.570 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.551+0000 7f262f7fe640 3 client.825612.objecter handle_osd_map got epochs [650,650] > 0 2024-09-16T04:03:11.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.551+0000 7f262f7fe640 3 client.825612.objecter handle_osd_map decoding full epoch 650 2024-09-16T04:03:11.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.551+0000 7f262f7fe640 20 client.825612.objecter dump_active .. 0 homeless 2024-09-16T04:03:11.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.551+0000 7f2646ffd640 1 --2- 172.21.15.42:0/1799016047 >> v2:172.21.15.154:6812/254845342 conn(0x7f2634037810 0x7f2634039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:03:11.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.551+0000 7f2646ffd640 10 monclient: get_auth_request con 0x7f2634037810 auth_method 0 2024-09-16T04:03:11.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.551+0000 7f2646ffd640 1 --2- 172.21.15.42:0/1799016047 >> v2:172.21.15.154:6812/254845342 conn(0x7f2634037810 0x7f2634039cd0 secure :-1 s=READY pgs=3798 cs=0 l=1 rev1=1 crypto rx=0x7f2638000c10 tx=0x7f26380a6000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:03:11.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.551+0000 7f262f7fe640 10 client.825612.objecter ms_handle_connect 0x7f2634037810 2024-09-16T04:03:11.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.555+0000 7f262f7fe640 1 -- 172.21.15.42:0/1799016047 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f26400076b0 con 0x7f2648137540 2024-09-16T04:03:11.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.555+0000 7f262f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:03:11.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.555+0000 7f262f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:03:11.786 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.783+0000 7f264d61f640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:03:11.786 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.783+0000 7f264d61f640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:03:11.786 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.783+0000 7f264d61f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:03:11.786 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.783+0000 7f264d61f640 1 -- 172.21.15.42:0/1799016047 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f26480a5750 con 0x7f2648137540 2024-09-16T04:03:11.786 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.783+0000 7f262f7fe640 1 -- 172.21.15.42:0/1799016047 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7f2640007470 con 0x7f2648137540 2024-09-16T04:03:11.786 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.783+0000 7f262f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:03:11.787 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.783+0000 7f262f7fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:03:11.787 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":188,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":21,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:03:11.789 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.783+0000 7f264d61f640 10 librados: watch_flush enter 2024-09-16T04:03:11.789 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.783+0000 7f264d61f640 10 librados: watch_flush exit 2024-09-16T04:03:11.789 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.783+0000 7f264d61f640 20 client.825612.objecter shutdown clearing up homeless session... 2024-09-16T04:03:11.789 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.783+0000 7f264d61f640 10 client.825612.objecter successfully canceled tick 2024-09-16T04:03:11.789 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.783+0000 7f264d61f640 1 -- 172.21.15.42:0/1799016047 >> v2:172.21.15.154:6812/254845342 conn(0x7f2634037810 msgr2=0x7f2634039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:03:11.790 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.783+0000 7f264d61f640 1 --2- 172.21.15.42:0/1799016047 >> v2:172.21.15.154:6812/254845342 conn(0x7f2634037810 0x7f2634039cd0 secure :-1 s=READY pgs=3798 cs=0 l=1 rev1=1 crypto rx=0x7f2638000c10 tx=0x7f26380a6000 comp rx=0 tx=0).stop 2024-09-16T04:03:11.790 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.783+0000 7f264d61f640 10 monclient: shutdown 2024-09-16T04:03:11.790 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.783+0000 7f264d61f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:03:11.790 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.783+0000 7f264d61f640 1 -- 172.21.15.42:0/1799016047 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2648137540 msgr2=0x7f264813daf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:03:11.790 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.783+0000 7f264d61f640 1 --2- 172.21.15.42:0/1799016047 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2648137540 0x7f264813daf0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f264000c3c0 tx=0x7f264000c880 comp rx=0 tx=0).stop 2024-09-16T04:03:11.790 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.783+0000 7f264d61f640 1 -- 172.21.15.42:0/1799016047 shutdown_connections 2024-09-16T04:03:11.790 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.783+0000 7f264d61f640 1 --2- 172.21.15.42:0/1799016047 >> v2:172.21.15.154:6812/254845342 conn(0x7f2634037810 0x7f2634039cd0 unknown :-1 s=CLOSED pgs=3798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:03:11.790 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.783+0000 7f264d61f640 1 --2- 172.21.15.42:0/1799016047 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2648137540 0x7f264813daf0 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:03:11.790 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.783+0000 7f264d61f640 1 --2- 172.21.15.42:0/1799016047 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f26480a3d90 0x7f264813d5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:03:11.790 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.783+0000 7f264d61f640 1 --2- 172.21.15.42:0/1799016047 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f26480a39b0 0x7f26480a20d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:03:11.790 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.783+0000 7f264d61f640 1 -- 172.21.15.42:0/1799016047 >> 172.21.15.42:0/1799016047 conn(0x7f26480a9fb0 msgr2=0x7f2648132170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:03:11.790 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.787+0000 7f264d61f640 1 -- 172.21.15.42:0/1799016047 shutdown_connections 2024-09-16T04:03:11.790 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.787+0000 7f264d61f640 1 -- 172.21.15.42:0/1799016047 wait complete. 2024-09-16T04:03:11.791 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:11.787+0000 7f264d61f640 1 librados: shutdown 2024-09-16T04:03:14.806 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:03:14.939 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.935+0000 7f2da6598640 10 monclient: get_monmap_and_config 2024-09-16T04:03:14.939 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.935+0000 7f2da6598640 10 monclient: build_initial_monmap 2024-09-16T04:03:14.939 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.935+0000 7f2da6598640 10 monclient: monmap: 2024-09-16T04:03:14.939 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:03:14.939 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:03:14.939 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:03:14.937757+0000 2024-09-16T04:03:14.939 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:03:14.937757+0000 2024-09-16T04:03:14.939 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:03:14.940 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:03:14.940 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:03:14.940 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:03:14.940 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:03:14.940 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:03:14.940 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:03:14.940 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:03:14.944 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.935+0000 7f2da6598640 1 Processor -- start 2024-09-16T04:03:14.944 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.935+0000 7f2da6598640 1 -- start start 2024-09-16T04:03:14.944 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.935+0000 7f2da6598640 10 monclient: init 2024-09-16T04:03:14.944 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.935+0000 7f2da6598640 10 monclient: _reopen_session rank -1 2024-09-16T04:03:14.944 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.935+0000 7f2da6598640 10 monclient: _add_conns ranks=[4,0,1,2,3] 2024-09-16T04:03:14.944 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.935+0000 7f2da6598640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f2da006db90 0x7f2da01340c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:03:14.944 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.935+0000 7f2da6598640 10 monclient(hunting): picked mon.noname-c con 0x7f2da006db90 addr v2:172.21.15.154:3300/0 2024-09-16T04:03:14.944 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.935+0000 7f2da6598640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2da00a37f0 addr v1:172.21.15.42:6789/0 2024-09-16T04:03:14.945 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.935+0000 7f2da6598640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2da00a3c00 addr v1:172.21.15.154:6789/0 2024-09-16T04:03:14.945 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.935+0000 7f2da6598640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2da006ece0 con 0x7f2da00a37f0 2024-09-16T04:03:14.945 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.935+0000 7f2da6598640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2da009bf80 con 0x7f2da00a3c00 2024-09-16T04:03:14.945 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.935+0000 7f2da6598640 10 monclient(hunting): start opening mon connection 2024-09-16T04:03:14.945 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.935+0000 7f2da6598640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f2da006df70 con 0x7f2da006db90 2024-09-16T04:03:14.945 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.935+0000 7f2da6598640 10 monclient(hunting): _renew_subs 2024-09-16T04:03:14.945 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.935+0000 7f2da6598640 10 monclient(hunting): authenticate will time out at 2241.907947s 2024-09-16T04:03:14.945 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.939+0000 7f2d9f7fe640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f2da00a37f0 0x7f2da0136e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:42998/0 (socket says 172.21.15.42:42998) 2024-09-16T04:03:14.945 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.939+0000 7f2d9f7fe640 1 -- 172.21.15.42:0/163364092 learned_addr learned my addr 172.21.15.42:0/163364092 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T04:03:14.945 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.939+0000 7f2d9ffff640 1 --2- 172.21.15.42:0/163364092 >> v2:172.21.15.154:3300/0 conn(0x7f2da006db90 0x7f2da01340c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:03:14.945 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.939+0000 7f2d9ffff640 10 monclient(hunting): get_auth_request con 0x7f2da006db90 auth_method 0 2024-09-16T04:03:14.945 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.939+0000 7f2d9ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:03:14.946 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.939+0000 7f2d9ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:03:14.946 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.939+0000 7f2d9ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:03:14.946 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.939+0000 7f2d9effd640 1 -- 172.21.15.42:0/163364092 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2793368492 0 0) 0x7f2da006ece0 con 0x7f2da00a37f0 2024-09-16T04:03:14.946 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.939+0000 7f2d9effd640 10 monclient(hunting): _negotiate 2024-09-16T04:03:14.946 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.939+0000 7f2d9effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:03:14.946 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.939+0000 7f2d9effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:03:14.946 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.939+0000 7f2d9effd640 10 monclient(hunting): my global_id is 825615 2024-09-16T04:03:14.946 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.939+0000 7f2d9effd640 1 -- 172.21.15.42:0/163364092 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2d88003530 con 0x7f2da00a37f0 2024-09-16T04:03:14.946 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.939+0000 7f2d9ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:03:14.946 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.939+0000 7f2d9ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:03:14.946 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.939+0000 7f2d9ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:03:14.946 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.939+0000 7f2d9effd640 1 -- 172.21.15.42:0/163364092 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2417951896 0 0) 0x7f2d88003530 con 0x7f2da00a37f0 2024-09-16T04:03:14.946 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.939+0000 7f2d9effd640 1 -- 172.21.15.42:0/163364092 >> v2:172.21.15.154:3300/0 conn(0x7f2da006db90 msgr2=0x7f2da01340c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:03:14.947 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.939+0000 7f2d9effd640 1 --2- 172.21.15.42:0/163364092 >> v2:172.21.15.154:3300/0 conn(0x7f2da006db90 0x7f2da01340c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:03:14.947 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.939+0000 7f2d9effd640 1 -- 172.21.15.42:0/163364092 >> v1:172.21.15.154:6789/0 conn(0x7f2da00a3c00 legacy=0x7f2da0137520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:03:14.947 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.939+0000 7f2d9effd640 10 monclient: _finish_hunting 0 2024-09-16T04:03:14.947 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.939+0000 7f2d9effd640 1 monclient: found mon.noname-a-legacy 2024-09-16T04:03:14.947 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.939+0000 7f2d9effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:03:14.947 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.939+0000 7f2d9effd640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.42:6789/0 2024-09-16T04:03:14.947 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.939+0000 7f2d9effd640 1 -- 172.21.15.42:0/163364092 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2da0137c30 con 0x7f2da00a37f0 2024-09-16T04:03:14.947 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.939+0000 7f2d9effd640 10 monclient: _finish_auth 0 2024-09-16T04:03:14.947 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.939+0000 7f2d9effd640 10 monclient: _check_auth_tickets 2024-09-16T04:03:14.947 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.939+0000 7f2d9effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:03:14.947 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.939+0000 7f2d9effd640 1 -- 172.21.15.42:0/163364092 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2177413734 0 0) 0x7f2d94001680 con 0x7f2da00a37f0 2024-09-16T04:03:14.948 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.939+0000 7f2d9effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:03:14.948 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.939+0000 7f2d9effd640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T04:03:14.948 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.939+0000 7f2d9effd640 10 monclient: dump: 2024-09-16T04:03:14.948 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:03:14.948 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:03:14.948 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:14.948 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:14.948 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:03:14.948 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:03:14.948 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:03:14.948 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:03:14.948 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:03:14.948 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:03:14.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.939+0000 7f2d9effd640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] but i'm connected to v1:172.21.15.42:6789/0, reconnecting 2024-09-16T04:03:14.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.939+0000 7f2d9effd640 10 monclient: _reopen_session rank -1 2024-09-16T04:03:14.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.939+0000 7f2d9effd640 1 -- 172.21.15.42:0/163364092 >> v1:172.21.15.42:6789/0 conn(0x7f2da00a37f0 legacy=0x7f2da0136e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:03:14.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.939+0000 7f2d9effd640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:03:14.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.939+0000 7f2d9effd640 1 --2- 172.21.15.42:0/163364092 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d880039c0 0x7f2d88003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:03:14.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.939+0000 7f2d9effd640 10 monclient(hunting): picked mon.b con 0x7f2d880039c0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:03:14.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.939+0000 7f2d9effd640 1 --2- 172.21.15.42:0/163364092 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2d88004810 0x7f2d88004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:03:14.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.939+0000 7f2d9effd640 10 monclient(hunting): picked mon.c con 0x7f2d88004810 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:03:14.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.939+0000 7f2d9effd640 1 --2- 172.21.15.42:0/163364092 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2d880052f0 0x7f2d8800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:03:14.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.939+0000 7f2d9effd640 10 monclient(hunting): picked mon.a con 0x7f2d880052f0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:03:14.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.939+0000 7f2d9effd640 10 monclient(hunting): start opening mon connection 2024-09-16T04:03:14.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.939+0000 7f2d9effd640 1 -- 172.21.15.42:0/163364092 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f2da006df70 con 0x7f2d880052f0 2024-09-16T04:03:14.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.939+0000 7f2d9effd640 10 monclient(hunting): start opening mon connection 2024-09-16T04:03:14.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.939+0000 7f2d9effd640 1 -- 172.21.15.42:0/163364092 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f2d8800c1c0 con 0x7f2d880039c0 2024-09-16T04:03:14.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.939+0000 7f2d9effd640 10 monclient(hunting): start opening mon connection 2024-09-16T04:03:14.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.939+0000 7f2d9effd640 1 -- 172.21.15.42:0/163364092 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f2d8800c3c0 con 0x7f2d88004810 2024-09-16T04:03:14.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.939+0000 7f2d9effd640 10 monclient(hunting): _renew_subs 2024-09-16T04:03:14.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.939+0000 7f2d9effd640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:03:14.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.939+0000 7f2d9f7fe640 1 --2- 172.21.15.42:0/163364092 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2d88004810 0x7f2d88004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:03:14.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.939+0000 7f2da4b35640 1 --2- 172.21.15.42:0/163364092 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2d880052f0 0x7f2d8800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:03:14.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.939+0000 7f2da4b35640 10 monclient(hunting): get_auth_request con 0x7f2d880052f0 auth_method 0 2024-09-16T04:03:14.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.939+0000 7f2da4b35640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:03:14.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.939+0000 7f2da4b35640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:03:14.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.939+0000 7f2da4b35640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:03:14.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.939+0000 7f2d9f7fe640 10 monclient(hunting): get_auth_request con 0x7f2d88004810 auth_method 0 2024-09-16T04:03:14.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.939+0000 7f2d9f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:03:14.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.939+0000 7f2d9f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:03:14.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.939+0000 7f2d9f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:03:14.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.939+0000 7f2d9ffff640 1 --2- 172.21.15.42:0/163364092 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d880039c0 0x7f2d88003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:03:14.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.939+0000 7f2d9ffff640 10 monclient(hunting): get_auth_request con 0x7f2d880039c0 auth_method 0 2024-09-16T04:03:14.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.939+0000 7f2d9ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:03:14.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.939+0000 7f2d9ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:03:14.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.939+0000 7f2d9ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:03:14.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.939+0000 7f2d9f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:03:14.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.939+0000 7f2d9f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:03:14.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.939+0000 7f2d9f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:03:14.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.939+0000 7f2da4b35640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:03:14.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.939+0000 7f2da4b35640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:03:14.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.939+0000 7f2da4b35640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:03:14.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.939+0000 7f2d9ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:03:14.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.939+0000 7f2d9ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:03:14.952 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.939+0000 7f2d9ffff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:03:14.952 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.939+0000 7f2d9f7fe640 10 monclient(hunting): handle_auth_done global_id 825615 payload 293 2024-09-16T04:03:14.952 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.939+0000 7f2d9f7fe640 1 -- 172.21.15.42:0/163364092 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d880039c0 msgr2=0x7f2d88003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:03:14.952 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.939+0000 7f2d9f7fe640 1 --2- 172.21.15.42:0/163364092 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d880039c0 0x7f2d88003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:03:14.952 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.939+0000 7f2d9f7fe640 1 -- 172.21.15.42:0/163364092 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2d880052f0 msgr2=0x7f2d8800b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:03:14.952 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.939+0000 7f2d9f7fe640 1 --2- 172.21.15.42:0/163364092 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2d880052f0 0x7f2d8800b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:03:14.952 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.939+0000 7f2d9f7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:03:14.952 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.939+0000 7f2d9f7fe640 1 monclient: found mon.c 2024-09-16T04:03:14.952 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.939+0000 7f2d9f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:03:14.952 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.939+0000 7f2d9f7fe640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:03:14.952 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.939+0000 7f2d9f7fe640 1 -- 172.21.15.42:0/163364092 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d88003530 con 0x7f2d88004810 2024-09-16T04:03:14.952 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.939+0000 7f2d9f7fe640 10 monclient: _finish_auth 0 2024-09-16T04:03:14.952 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.939+0000 7f2d9f7fe640 10 monclient: _check_auth_tickets 2024-09-16T04:03:14.952 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.939+0000 7f2d9f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:03:14.953 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.939+0000 7f2da4b35640 1 --2- 172.21.15.42:0/163364092 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2d880052f0 0x7f2d8800b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:03:14.953 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.939+0000 7f2da4b35640 1 -- 172.21.15.42:0/163364092 reap_dead start 2024-09-16T04:03:14.953 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.939+0000 7f2da6598640 5 monclient: authenticate success, global_id 825615 2024-09-16T04:03:14.953 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.939+0000 7f2da6598640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:03:14.958 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.939+0000 7f2d9f7fe640 1 --2- 172.21.15.42:0/163364092 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2d88004810 0x7f2d88004c90 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f2d94001ec0 tx=0x7f2d9401cc40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ef7144ec2d0677f6 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:03:14.958 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.939+0000 7f2d9effd640 1 -- 172.21.15.42:0/163364092 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d94007d40 con 0x7f2d88004810 2024-09-16T04:03:14.958 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.939+0000 7f2d9effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:03:14.958 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.939+0000 7f2d9effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:03:14.958 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.939+0000 7f2d9effd640 10 monclient: dump: 2024-09-16T04:03:14.958 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:03:14.958 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:03:14.958 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:14.958 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:14.959 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:03:14.959 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:03:14.959 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:03:14.959 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:03:14.959 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:03:14.959 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:03:14.959 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.939+0000 7f2da6598640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:03:14.959 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.939+0000 7f2d9effd640 1 -- 172.21.15.42:0/163364092 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d940a9070 con 0x7f2d88004810 2024-09-16T04:03:14.959 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.939+0000 7f2d9effd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:03:14.959 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.939+0000 7f2da6598640 10 monclient: get_monmap_and_config success 2024-09-16T04:03:14.959 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.939+0000 7f2da6598640 4 set_mon_vals no callback set 2024-09-16T04:03:14.959 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.939+0000 7f2d9effd640 1 -- 172.21.15.42:0/163364092 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d94005d80 con 0x7f2d88004810 2024-09-16T04:03:14.959 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.939+0000 7f2d9effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:03:14.960 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.939+0000 7f2d9effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:03:14.960 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.939+0000 7f2d9effd640 10 monclient: dump: 2024-09-16T04:03:14.960 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:03:14.960 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:03:14.960 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:14.960 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:14.960 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:03:14.960 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:03:14.960 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:03:14.960 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:03:14.960 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:03:14.960 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:03:14.960 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.943+0000 7f2da6598640 10 monclient: shutdown 2024-09-16T04:03:14.961 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.943+0000 7f2da6598640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:03:14.961 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.943+0000 7f2da6598640 1 -- 172.21.15.42:0/163364092 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2d88004810 msgr2=0x7f2d88004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:03:14.961 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.943+0000 7f2da6598640 1 --2- 172.21.15.42:0/163364092 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2d88004810 0x7f2d88004c90 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f2d94001ec0 tx=0x7f2d9401cc40 comp rx=0 tx=0).stop 2024-09-16T04:03:14.961 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.943+0000 7f2da6598640 1 -- 172.21.15.42:0/163364092 shutdown_connections 2024-09-16T04:03:14.961 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.943+0000 7f2da6598640 1 --2- 172.21.15.42:0/163364092 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2d88004810 0x7f2d88004c90 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:03:14.961 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.943+0000 7f2da6598640 1 -- 172.21.15.42:0/163364092 >> 172.21.15.42:0/163364092 conn(0x7f2da00a9fb0 msgr2=0x7f2da00aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:03:14.961 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.943+0000 7f2da6598640 1 -- 172.21.15.42:0/163364092 shutdown_connections 2024-09-16T04:03:14.961 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.943+0000 7f2da6598640 1 -- 172.21.15.42:0/163364092 wait complete. 2024-09-16T04:03:14.961 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.943+0000 7f2da6598640 10 monclient: build_initial_monmap 2024-09-16T04:03:14.961 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.943+0000 7f2da6598640 10 monclient: monmap: 2024-09-16T04:03:14.961 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:03:14.961 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:03:14.961 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:03:14.961 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:03:14.962 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:03:14.962 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:03:14.962 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:03:14.962 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:03:14.962 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:03:14.962 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:03:14.962 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.943+0000 7f2da6598640 1 librados: starting msgr at 2024-09-16T04:03:14.962 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.943+0000 7f2da6598640 1 librados: starting objecter 2024-09-16T04:03:14.962 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.943+0000 7f2da6598640 1 Processor -- start 2024-09-16T04:03:14.962 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.943+0000 7f2da6598640 1 -- start start 2024-09-16T04:03:14.962 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.943+0000 7f2da6598640 1 librados: setting wanted keys 2024-09-16T04:03:14.962 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.943+0000 7f2da6598640 1 librados: calling monclient init 2024-09-16T04:03:14.962 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.943+0000 7f2da6598640 10 monclient: init 2024-09-16T04:03:14.962 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.943+0000 7f2da6598640 10 monclient: _reopen_session rank -1 2024-09-16T04:03:14.963 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.943+0000 7f2da6598640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:03:14.963 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.943+0000 7f2da6598640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d88004810 0x7f2da01cf1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:03:14.963 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.943+0000 7f2da6598640 10 monclient(hunting): picked mon.noname-b con 0x7f2d88004810 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:03:14.963 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.943+0000 7f2da6598640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2da01cf6e0 0x7f2da01cfae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:03:14.963 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.943+0000 7f2da6598640 10 monclient(hunting): picked mon.noname-a con 0x7f2da01cf6e0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:03:14.963 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.943+0000 7f2da6598640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2da0120200 0x7f2da0120680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:03:14.963 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.943+0000 7f2da6598640 10 monclient(hunting): picked mon.noname-c con 0x7f2da0120200 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:03:14.963 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.943+0000 7f2da6598640 10 monclient(hunting): start opening mon connection 2024-09-16T04:03:14.963 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.943+0000 7f2da6598640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f2da0137620 con 0x7f2da01cf6e0 2024-09-16T04:03:14.963 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.943+0000 7f2da6598640 10 monclient(hunting): start opening mon connection 2024-09-16T04:03:14.963 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.943+0000 7f2da6598640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f2da006ea70 con 0x7f2d88004810 2024-09-16T04:03:14.963 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.943+0000 7f2da6598640 10 monclient(hunting): start opening mon connection 2024-09-16T04:03:14.963 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.943+0000 7f2da6598640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f2da01377a0 con 0x7f2da0120200 2024-09-16T04:03:14.963 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.943+0000 7f2da6598640 10 monclient(hunting): _renew_subs 2024-09-16T04:03:14.964 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.943+0000 7f2da6598640 10 monclient(hunting): authenticate will time out at 2241.914238s 2024-09-16T04:03:14.964 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.943+0000 7f2d9f7fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2da01cf6e0 0x7f2da01cfae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:03:14.964 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.943+0000 7f2d9f7fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2da01cf6e0 0x7f2da01cfae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:42342/0 (socket says 172.21.15.42:42342) 2024-09-16T04:03:14.964 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.943+0000 7f2da4b35640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2da0120200 0x7f2da0120680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:03:14.964 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.943+0000 7f2d9f7fe640 1 -- 172.21.15.42:0/3945200452 learned_addr learned my addr 172.21.15.42:0/3945200452 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:03:14.964 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.943+0000 7f2d9f7fe640 10 monclient(hunting): get_auth_request con 0x7f2da01cf6e0 auth_method 0 2024-09-16T04:03:14.964 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.943+0000 7f2d9f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:03:14.964 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.943+0000 7f2d9f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:03:14.964 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.943+0000 7f2d9f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:03:14.964 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.943+0000 7f2da4b35640 10 monclient(hunting): get_auth_request con 0x7f2da0120200 auth_method 0 2024-09-16T04:03:14.964 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.943+0000 7f2da4b35640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:03:14.964 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.943+0000 7f2da4b35640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:03:14.964 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.943+0000 7f2da4b35640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:03:14.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.943+0000 7f2d9f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:03:14.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.943+0000 7f2d9f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:03:14.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.943+0000 7f2d9f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:03:14.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.943+0000 7f2da4b35640 1 -- 172.21.15.42:0/3945200452 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2da0120200 msgr2=0x7f2da0120680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-16T04:03:14.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.943+0000 7f2da4b35640 1 -- 172.21.15.42:0/3945200452 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2da0120200 msgr2=0x7f2da0120680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:03:14.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.943+0000 7f2da4b35640 1 --2- 172.21.15.42:0/3945200452 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2da0120200 0x7f2da0120680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:03:14.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.943+0000 7f2d9ffff640 1 --2- 172.21.15.42:0/3945200452 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d88004810 0x7f2da01cf1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:03:14.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.943+0000 7f2da4b35640 1 --2- 172.21.15.42:0/3945200452 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2da0120200 0x7f2da0120680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:03:14.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.943+0000 7f2d9ffff640 10 monclient(hunting): get_auth_request con 0x7f2d88004810 auth_method 0 2024-09-16T04:03:14.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.943+0000 7f2d9ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:03:14.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.943+0000 7f2d9ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:03:14.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.943+0000 7f2d9ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:03:14.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.943+0000 7f2d9f7fe640 10 monclient(hunting): handle_auth_done global_id 825618 payload 824 2024-09-16T04:03:14.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.943+0000 7f2d9f7fe640 1 -- 172.21.15.42:0/3945200452 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2da0120200 msgr2=0x7f2da0120680 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:03:14.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.943+0000 7f2d9f7fe640 1 --2- 172.21.15.42:0/3945200452 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2da0120200 0x7f2da0120680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:03:14.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.943+0000 7f2d9f7fe640 1 -- 172.21.15.42:0/3945200452 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d88004810 msgr2=0x7f2da01cf1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:03:14.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.943+0000 7f2d9f7fe640 1 --2- 172.21.15.42:0/3945200452 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d88004810 0x7f2da01cf1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:03:14.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.943+0000 7f2d9f7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:03:14.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.943+0000 7f2d9f7fe640 1 monclient: found mon.noname-a 2024-09-16T04:03:14.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.943+0000 7f2d9f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:03:14.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.943+0000 7f2d9f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:03:14.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.943+0000 7f2d9f7fe640 1 -- 172.21.15.42:0/3945200452 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2da0122640 con 0x7f2da01cf6e0 2024-09-16T04:03:14.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.943+0000 7f2d9f7fe640 1 --2- 172.21.15.42:0/3945200452 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2da01cf6e0 0x7f2da01cfae0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f2d9401ca70 tx=0x7f2d88005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:03:14.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.943+0000 7f2d9cff9640 10 client.?.objecter ms_handle_connect 0x7f2da01cf6e0 2024-09-16T04:03:14.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.943+0000 7f2d9cff9640 10 client.?.objecter resend_mon_ops 2024-09-16T04:03:14.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.943+0000 7f2d9cff9640 1 -- 172.21.15.42:0/3945200452 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d94007390 con 0x7f2da01cf6e0 2024-09-16T04:03:14.967 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.943+0000 7f2d9cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:03:14.967 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.943+0000 7f2d9cff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:03:14.967 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.943+0000 7f2d9cff9640 10 monclient: dump: 2024-09-16T04:03:14.967 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:03:14.967 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:03:14.967 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:14.967 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:14.967 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:03:14.967 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:03:14.967 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:03:14.967 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:03:14.967 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:03:14.967 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:03:14.967 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.943+0000 7f2d9cff9640 10 monclient: _finish_auth 0 2024-09-16T04:03:14.967 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.943+0000 7f2d9cff9640 10 monclient: _check_auth_tickets 2024-09-16T04:03:14.968 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.943+0000 7f2d9cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:03:14.968 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.943+0000 7f2d9cff9640 1 -- 172.21.15.42:0/3945200452 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d94007530 con 0x7f2da01cf6e0 2024-09-16T04:03:14.968 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.943+0000 7f2d9cff9640 10 monclient: handle_config config(0 keys) 2024-09-16T04:03:14.968 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.943+0000 7f2d9cff9640 1 -- 172.21.15.42:0/3945200452 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d940a5830 con 0x7f2da01cf6e0 2024-09-16T04:03:14.968 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.943+0000 7f2d9cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:03:14.968 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.943+0000 7f2d9cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:03:14.968 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.943+0000 7f2d9cff9640 10 monclient: dump: 2024-09-16T04:03:14.968 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:03:14.968 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:03:14.968 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:14.968 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:14.968 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:03:14.968 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:03:14.969 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:03:14.969 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:03:14.969 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:03:14.969 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:03:14.969 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.943+0000 7f2d9dffb640 4 set_mon_vals no callback set 2024-09-16T04:03:14.969 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.943+0000 7f2da6598640 5 monclient: authenticate success, global_id 825618 2024-09-16T04:03:14.969 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.943+0000 7f2da6598640 10 monclient: _renew_subs 2024-09-16T04:03:14.969 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.943+0000 7f2da6598640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:03:14.969 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.943+0000 7f2da6598640 1 -- 172.21.15.42:0/3945200452 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2da0122920 con 0x7f2da01cf6e0 2024-09-16T04:03:14.969 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.943+0000 7f2da6598640 10 client.825618.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:03:14.969 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.943+0000 7f2da6598640 10 monclient: _renew_subs 2024-09-16T04:03:14.969 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.943+0000 7f2da6598640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:03:14.969 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.943+0000 7f2da6598640 1 -- 172.21.15.42:0/3945200452 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2da00a94a0 con 0x7f2da01cf6e0 2024-09-16T04:03:14.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.943+0000 7f2da6598640 1 librados: init done 2024-09-16T04:03:14.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.947+0000 7f2da6598640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:03:14.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.947+0000 7f2da6598640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:03:14.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.947+0000 7f2da6598640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:03:14.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.947+0000 7f2da6598640 1 -- 172.21.15.42:0/3945200452 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2da013a180 con 0x7f2da01cf6e0 2024-09-16T04:03:14.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.947+0000 7f2d9cff9640 1 -- 172.21.15.42:0/3945200452 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f2d940a59d0 con 0x7f2da01cf6e0 2024-09-16T04:03:14.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.947+0000 7f2d9cff9640 1 --2- 172.21.15.42:0/3945200452 >> v2:172.21.15.154:6812/254845342 conn(0x7f2d8c037840 0x7f2d8c039d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:03:14.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.947+0000 7f2d9cff9640 1 -- 172.21.15.42:0/3945200452 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(650..650 src has 451..650) ==== 6306+0+0 (secure 0 0 0) 0x7f2d940e2bb0 con 0x7f2da01cf6e0 2024-09-16T04:03:14.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.947+0000 7f2d9cff9640 10 client.825618.objecter ms_dispatch 0x7f2da0000ba0 osd_map(650..650 src has 451..650) 2024-09-16T04:03:14.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.947+0000 7f2d9cff9640 3 client.825618.objecter handle_osd_map got epochs [650,650] > 0 2024-09-16T04:03:14.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.947+0000 7f2d9cff9640 3 client.825618.objecter handle_osd_map decoding full epoch 650 2024-09-16T04:03:14.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.947+0000 7f2d9cff9640 20 client.825618.objecter dump_active .. 0 homeless 2024-09-16T04:03:14.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.947+0000 7f2d9ffff640 1 --2- 172.21.15.42:0/3945200452 >> v2:172.21.15.154:6812/254845342 conn(0x7f2d8c037840 0x7f2d8c039d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:03:14.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.947+0000 7f2d9ffff640 10 monclient: get_auth_request con 0x7f2d8c037840 auth_method 0 2024-09-16T04:03:14.971 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.947+0000 7f2d9cff9640 10 client.825618.objecter ms_handle_connect 0x7f2d8c037840 2024-09-16T04:03:14.971 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.947+0000 7f2d9ffff640 1 --2- 172.21.15.42:0/3945200452 >> v2:172.21.15.154:6812/254845342 conn(0x7f2d8c037840 0x7f2d8c039d00 secure :-1 s=READY pgs=3799 cs=0 l=1 rev1=1 crypto rx=0x7f2d90007520 tx=0x7f2d9000b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:03:14.971 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.951+0000 7f2d9cff9640 1 -- 172.21.15.42:0/3945200452 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f2d94016950 con 0x7f2da01cf6e0 2024-09-16T04:03:14.971 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.951+0000 7f2d9cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:03:14.971 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:14.951+0000 7f2d9cff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:03:15.181 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:15.175+0000 7f2da6598640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:03:15.181 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:15.175+0000 7f2da6598640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:03:15.181 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:15.175+0000 7f2da6598640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:03:15.181 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:15.175+0000 7f2da6598640 1 -- 172.21.15.42:0/3945200452 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f2da0068e40 con 0x7f2da01cf6e0 2024-09-16T04:03:15.181 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:15.179+0000 7f2d9cff9640 1 -- 172.21.15.42:0/3945200452 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7f2d940b7380 con 0x7f2da01cf6e0 2024-09-16T04:03:15.181 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:15.179+0000 7f2d9cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:03:15.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:15.179+0000 7f2d9cff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:03:15.182 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":188,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":24,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:03:15.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:15.179+0000 7f2da6598640 10 librados: watch_flush enter 2024-09-16T04:03:15.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:15.179+0000 7f2da6598640 10 librados: watch_flush exit 2024-09-16T04:03:15.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:15.179+0000 7f2da6598640 20 client.825618.objecter shutdown clearing up homeless session... 2024-09-16T04:03:15.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:15.179+0000 7f2da6598640 10 client.825618.objecter successfully canceled tick 2024-09-16T04:03:15.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:15.179+0000 7f2da6598640 1 -- 172.21.15.42:0/3945200452 >> v2:172.21.15.154:6812/254845342 conn(0x7f2d8c037840 msgr2=0x7f2d8c039d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:03:15.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:15.179+0000 7f2da6598640 1 --2- 172.21.15.42:0/3945200452 >> v2:172.21.15.154:6812/254845342 conn(0x7f2d8c037840 0x7f2d8c039d00 secure :-1 s=READY pgs=3799 cs=0 l=1 rev1=1 crypto rx=0x7f2d90007520 tx=0x7f2d9000b040 comp rx=0 tx=0).stop 2024-09-16T04:03:15.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:15.179+0000 7f2da6598640 10 monclient: shutdown 2024-09-16T04:03:15.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:15.179+0000 7f2da6598640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:03:15.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:15.179+0000 7f2da6598640 1 -- 172.21.15.42:0/3945200452 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2da01cf6e0 msgr2=0x7f2da01cfae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:03:15.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:15.179+0000 7f2da6598640 1 --2- 172.21.15.42:0/3945200452 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2da01cf6e0 0x7f2da01cfae0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f2d9401ca70 tx=0x7f2d88005840 comp rx=0 tx=0).stop 2024-09-16T04:03:15.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:15.179+0000 7f2da6598640 1 -- 172.21.15.42:0/3945200452 shutdown_connections 2024-09-16T04:03:15.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:15.179+0000 7f2da6598640 1 --2- 172.21.15.42:0/3945200452 >> v2:172.21.15.154:6812/254845342 conn(0x7f2d8c037840 0x7f2d8c039d00 unknown :-1 s=CLOSED pgs=3799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:03:15.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:15.179+0000 7f2da6598640 1 --2- 172.21.15.42:0/3945200452 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2da0120200 0x7f2da0120680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:03:15.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:15.179+0000 7f2da6598640 1 --2- 172.21.15.42:0/3945200452 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2da01cf6e0 0x7f2da01cfae0 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:03:15.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:15.179+0000 7f2da6598640 1 --2- 172.21.15.42:0/3945200452 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2d88004810 0x7f2da01cf1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:03:15.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:15.179+0000 7f2da6598640 1 -- 172.21.15.42:0/3945200452 >> 172.21.15.42:0/3945200452 conn(0x7f2da00a9fb0 msgr2=0x7f2da0131790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:03:15.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:15.179+0000 7f2da6598640 1 -- 172.21.15.42:0/3945200452 shutdown_connections 2024-09-16T04:03:15.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:15.179+0000 7f2da6598640 1 -- 172.21.15.42:0/3945200452 wait complete. 2024-09-16T04:03:15.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:15.179+0000 7f2da6598640 1 librados: shutdown 2024-09-16T04:03:18.202 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:03:18.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.331+0000 7fd320084640 10 monclient: get_monmap_and_config 2024-09-16T04:03:18.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.331+0000 7fd320084640 10 monclient: build_initial_monmap 2024-09-16T04:03:18.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.331+0000 7fd320084640 10 monclient: monmap: 2024-09-16T04:03:18.336 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:03:18.336 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:03:18.336 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:03:18.334125+0000 2024-09-16T04:03:18.336 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:03:18.334125+0000 2024-09-16T04:03:18.336 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:03:18.336 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:03:18.336 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:03:18.336 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:03:18.336 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:03:18.336 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:03:18.336 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:03:18.337 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:03:18.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.331+0000 7fd320084640 1 Processor -- start 2024-09-16T04:03:18.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.331+0000 7fd320084640 1 -- start start 2024-09-16T04:03:18.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.331+0000 7fd320084640 10 monclient: init 2024-09-16T04:03:18.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.331+0000 7fd320084640 10 monclient: _reopen_session rank -1 2024-09-16T04:03:18.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.331+0000 7fd320084640 10 monclient: _add_conns ranks=[0,2,3,4,1] 2024-09-16T04:03:18.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.331+0000 7fd320084640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd31806db90 addr v1:172.21.15.42:6789/0 2024-09-16T04:03:18.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.331+0000 7fd320084640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd3181340c0 0x7fd31813c190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:03:18.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.331+0000 7fd320084640 10 monclient(hunting): picked mon.noname-b con 0x7fd3181340c0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:03:18.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.331+0000 7fd320084640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd31813c6d0 0x7fd31813eac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:03:18.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.331+0000 7fd320084640 10 monclient(hunting): picked mon.noname-a con 0x7fd31813c6d0 addr v2:172.21.15.42:3300/0 2024-09-16T04:03:18.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.331+0000 7fd320084640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd31806ece0 con 0x7fd31806db90 2024-09-16T04:03:18.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.331+0000 7fd320084640 10 monclient(hunting): start opening mon connection 2024-09-16T04:03:18.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.331+0000 7fd320084640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7fd31806df70 con 0x7fd31813c6d0 2024-09-16T04:03:18.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.331+0000 7fd320084640 10 monclient(hunting): start opening mon connection 2024-09-16T04:03:18.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.331+0000 7fd320084640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fd31809bf80 con 0x7fd3181340c0 2024-09-16T04:03:18.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.331+0000 7fd320084640 10 monclient(hunting): _renew_subs 2024-09-16T04:03:18.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.331+0000 7fd320084640 10 monclient(hunting): authenticate will time out at 2245.303364s 2024-09-16T04:03:18.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.331+0000 7fd31de20640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fd31806db90 0x7fd3180a37f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:43000/0 (socket says 172.21.15.42:43000) 2024-09-16T04:03:18.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.331+0000 7fd31de20640 1 -- 172.21.15.42:0/3732920700 learned_addr learned my addr 172.21.15.42:0/3732920700 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T04:03:18.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.331+0000 7fd31e621640 1 --2- 172.21.15.42:0/3732920700 >> v2:172.21.15.42:3300/0 conn(0x7fd31813c6d0 0x7fd31813eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:03:18.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.331+0000 7fd31e621640 10 monclient(hunting): get_auth_request con 0x7fd31813c6d0 auth_method 0 2024-09-16T04:03:18.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.331+0000 7fd31e621640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:03:18.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.331+0000 7fd31e621640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:03:18.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.331+0000 7fd31e621640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:03:18.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.331+0000 7fd31d61f640 1 --2- 172.21.15.42:0/3732920700 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd3181340c0 0x7fd31813c190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:03:18.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.331+0000 7fd31d61f640 10 monclient(hunting): get_auth_request con 0x7fd3181340c0 auth_method 0 2024-09-16T04:03:18.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.331+0000 7fd31d61f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:03:18.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.331+0000 7fd31d61f640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:03:18.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.331+0000 7fd31d61f640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:03:18.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.331+0000 7fd31e621640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:03:18.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.331+0000 7fd31e621640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:03:18.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.331+0000 7fd31e621640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:03:18.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.331+0000 7fd31d61f640 1 -- 172.21.15.42:0/3732920700 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd3181340c0 msgr2=0x7fd31813c190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:03:18.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.331+0000 7fd31d61f640 1 -- 172.21.15.42:0/3732920700 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd3181340c0 msgr2=0x7fd31813c190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:03:18.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.331+0000 7fd31d61f640 1 --2- 172.21.15.42:0/3732920700 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd3181340c0 0x7fd31813c190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:03:18.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.331+0000 7fd31d61f640 1 --2- 172.21.15.42:0/3732920700 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd3181340c0 0x7fd31813c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:03:18.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.331+0000 7fd31e621640 10 monclient(hunting): handle_auth_done global_id 825621 payload 274 2024-09-16T04:03:18.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.331+0000 7fd31e621640 1 -- 172.21.15.42:0/3732920700 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd3181340c0 msgr2=0x7fd31813c190 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:03:18.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.331+0000 7fd31e621640 1 --2- 172.21.15.42:0/3732920700 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd3181340c0 0x7fd31813c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:03:18.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.331+0000 7fd31e621640 1 -- 172.21.15.42:0/3732920700 >> v1:172.21.15.42:6789/0 conn(0x7fd31806db90 legacy=0x7fd3180a37f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:03:18.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.331+0000 7fd31e621640 10 monclient: _finish_hunting 0 2024-09-16T04:03:18.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.331+0000 7fd31e621640 1 monclient: found mon.noname-a 2024-09-16T04:03:18.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.335+0000 7fd31e621640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:03:18.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.335+0000 7fd31e621640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:03:18.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.335+0000 7fd31e621640 1 -- 172.21.15.42:0/3732920700 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3181344d0 con 0x7fd31813c6d0 2024-09-16T04:03:18.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.335+0000 7fd31e621640 1 --2- 172.21.15.42:0/3732920700 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd31813c6d0 0x7fd31813eac0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7fd310009d90 tx=0x7fd31001c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5343748b16242d2f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:03:18.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.335+0000 7fd31ce1e640 1 -- 172.21.15.42:0/3732920700 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3100b2070 con 0x7fd31813c6d0 2024-09-16T04:03:18.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.335+0000 7fd31ce1e640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:03:18.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.335+0000 7fd31ce1e640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:03:18.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.335+0000 7fd31ce1e640 10 monclient: dump: 2024-09-16T04:03:18.349 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:03:18.349 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:03:18.349 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:18.349 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:18.349 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:03:18.349 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:03:18.349 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:03:18.349 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:03:18.349 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:03:18.349 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:03:18.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.335+0000 7fd31ce1e640 10 monclient: _finish_auth 0 2024-09-16T04:03:18.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.335+0000 7fd31ce1e640 10 monclient: _check_auth_tickets 2024-09-16T04:03:18.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.335+0000 7fd31ce1e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:03:18.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.335+0000 7fd31ce1e640 1 -- 172.21.15.42:0/3732920700 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd3100a58f0 con 0x7fd31813c6d0 2024-09-16T04:03:18.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.335+0000 7fd320084640 5 monclient: authenticate success, global_id 825621 2024-09-16T04:03:18.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.335+0000 7fd320084640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:03:18.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.335+0000 7fd31ce1e640 10 monclient: handle_config config(0 keys) 2024-09-16T04:03:18.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.335+0000 7fd320084640 10 monclient: get_monmap_and_config success 2024-09-16T04:03:18.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.335+0000 7fd31ce1e640 1 -- 172.21.15.42:0/3732920700 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3100a5cc0 con 0x7fd31813c6d0 2024-09-16T04:03:18.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.335+0000 7fd320084640 4 set_mon_vals no callback set 2024-09-16T04:03:18.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.335+0000 7fd31ce1e640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:03:18.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.335+0000 7fd31ce1e640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:03:18.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.335+0000 7fd31ce1e640 10 monclient: dump: 2024-09-16T04:03:18.350 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:03:18.350 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:03:18.350 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:18.351 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:18.351 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:03:18.351 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:03:18.351 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:03:18.351 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:03:18.351 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:03:18.351 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:03:18.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.335+0000 7fd320084640 10 monclient: shutdown 2024-09-16T04:03:18.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.335+0000 7fd320084640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:03:18.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.335+0000 7fd320084640 1 -- 172.21.15.42:0/3732920700 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd31813c6d0 msgr2=0x7fd31813eac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:03:18.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.335+0000 7fd320084640 1 --2- 172.21.15.42:0/3732920700 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd31813c6d0 0x7fd31813eac0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7fd310009d90 tx=0x7fd31001c830 comp rx=0 tx=0).stop 2024-09-16T04:03:18.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.335+0000 7fd320084640 1 -- 172.21.15.42:0/3732920700 shutdown_connections 2024-09-16T04:03:18.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.335+0000 7fd320084640 1 --2- 172.21.15.42:0/3732920700 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd31813c6d0 0x7fd31813eac0 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:03:18.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.335+0000 7fd320084640 1 --2- 172.21.15.42:0/3732920700 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd3181340c0 0x7fd31813c190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:03:18.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.335+0000 7fd320084640 1 -- 172.21.15.42:0/3732920700 >> 172.21.15.42:0/3732920700 conn(0x7fd3180a9fb0 msgr2=0x7fd3180aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:03:18.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.335+0000 7fd320084640 1 -- 172.21.15.42:0/3732920700 shutdown_connections 2024-09-16T04:03:18.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.335+0000 7fd320084640 1 -- 172.21.15.42:0/3732920700 wait complete. 2024-09-16T04:03:18.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.335+0000 7fd320084640 10 monclient: build_initial_monmap 2024-09-16T04:03:18.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.335+0000 7fd320084640 10 monclient: monmap: 2024-09-16T04:03:18.352 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:03:18.352 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:03:18.352 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:03:18.352 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:03:18.352 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:03:18.352 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:03:18.352 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:03:18.352 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:03:18.353 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:03:18.353 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:03:18.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.335+0000 7fd320084640 1 librados: starting msgr at 2024-09-16T04:03:18.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.335+0000 7fd320084640 1 librados: starting objecter 2024-09-16T04:03:18.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.335+0000 7fd320084640 1 Processor -- start 2024-09-16T04:03:18.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.339+0000 7fd320084640 1 -- start start 2024-09-16T04:03:18.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.339+0000 7fd320084640 1 librados: setting wanted keys 2024-09-16T04:03:18.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.339+0000 7fd320084640 1 librados: calling monclient init 2024-09-16T04:03:18.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.339+0000 7fd320084640 10 monclient: init 2024-09-16T04:03:18.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.339+0000 7fd320084640 10 monclient: _reopen_session rank -1 2024-09-16T04:03:18.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.339+0000 7fd320084640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:03:18.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.339+0000 7fd320084640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd31806db90 0x7fd3181ceeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:03:18.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.339+0000 7fd320084640 10 monclient(hunting): picked mon.noname-b con 0x7fd31806db90 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:03:18.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.339+0000 7fd320084640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd3181340c0 0x7fd3180a2ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:03:18.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.339+0000 7fd320084640 10 monclient(hunting): picked mon.noname-c con 0x7fd3181340c0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:03:18.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.339+0000 7fd320084640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd31813c6d0 0x7fd3180a30e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:03:18.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.339+0000 7fd320084640 10 monclient(hunting): picked mon.noname-a con 0x7fd31813c6d0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:03:18.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.339+0000 7fd320084640 10 monclient(hunting): start opening mon connection 2024-09-16T04:03:18.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.339+0000 7fd320084640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fd318141820 con 0x7fd31813c6d0 2024-09-16T04:03:18.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.339+0000 7fd320084640 10 monclient(hunting): start opening mon connection 2024-09-16T04:03:18.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.339+0000 7fd320084640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd3181419a0 con 0x7fd31806db90 2024-09-16T04:03:18.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.339+0000 7fd320084640 10 monclient(hunting): start opening mon connection 2024-09-16T04:03:18.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.339+0000 7fd320084640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fd318141b20 con 0x7fd3181340c0 2024-09-16T04:03:18.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.339+0000 7fd320084640 10 monclient(hunting): _renew_subs 2024-09-16T04:03:18.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.339+0000 7fd320084640 10 monclient(hunting): authenticate will time out at 2245.308591s 2024-09-16T04:03:18.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.339+0000 7fd31d61f640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd3181340c0 0x7fd3180a2ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:03:18.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.339+0000 7fd31d61f640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd3181340c0 0x7fd3180a2ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:52362/0 (socket says 172.21.15.42:52362) 2024-09-16T04:03:18.355 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.339+0000 7fd31e621640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd31813c6d0 0x7fd3180a30e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:03:18.355 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.339+0000 7fd31d61f640 1 -- 172.21.15.42:0/2115685877 learned_addr learned my addr 172.21.15.42:0/2115685877 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:03:18.355 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.339+0000 7fd31d61f640 10 monclient(hunting): get_auth_request con 0x7fd3181340c0 auth_method 0 2024-09-16T04:03:18.355 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.339+0000 7fd31d61f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:03:18.355 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.339+0000 7fd31d61f640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:03:18.355 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.339+0000 7fd31d61f640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:03:18.355 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.339+0000 7fd31e621640 10 monclient(hunting): get_auth_request con 0x7fd31813c6d0 auth_method 0 2024-09-16T04:03:18.355 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.339+0000 7fd31e621640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:03:18.355 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.339+0000 7fd31e621640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:03:18.355 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.339+0000 7fd31e621640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:03:18.355 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.339+0000 7fd31e621640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:03:18.355 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.339+0000 7fd31e621640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:03:18.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.339+0000 7fd31d61f640 1 -- 172.21.15.42:0/2115685877 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd3181340c0 msgr2=0x7fd3180a2ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-16T04:03:18.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.339+0000 7fd31d61f640 1 -- 172.21.15.42:0/2115685877 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd3181340c0 msgr2=0x7fd3180a2ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:03:18.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.339+0000 7fd31d61f640 1 --2- 172.21.15.42:0/2115685877 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd3181340c0 0x7fd3180a2ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:03:18.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.339+0000 7fd31e621640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:03:18.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.339+0000 7fd31d61f640 1 --2- 172.21.15.42:0/2115685877 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd3181340c0 0x7fd3180a2ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:03:18.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.339+0000 7fd31e621640 10 monclient(hunting): handle_auth_done global_id 825627 payload 824 2024-09-16T04:03:18.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.339+0000 7fd31e621640 1 -- 172.21.15.42:0/2115685877 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd3181340c0 msgr2=0x7fd3180a2ba0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:03:18.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.339+0000 7fd31e621640 1 --2- 172.21.15.42:0/2115685877 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd3181340c0 0x7fd3180a2ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:03:18.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.339+0000 7fd31e621640 1 -- 172.21.15.42:0/2115685877 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd31806db90 msgr2=0x7fd3181ceeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:03:18.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.339+0000 7fd31de20640 1 --2- 172.21.15.42:0/2115685877 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd31806db90 0x7fd3181ceeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:03:18.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.339+0000 7fd31e621640 1 --2- 172.21.15.42:0/2115685877 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd31806db90 0x7fd3181ceeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:03:18.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.339+0000 7fd31e621640 10 monclient: _finish_hunting 0 2024-09-16T04:03:18.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.339+0000 7fd31e621640 1 monclient: found mon.noname-a 2024-09-16T04:03:18.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.339+0000 7fd31e621640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:03:18.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.339+0000 7fd31e621640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:03:18.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.339+0000 7fd31e621640 1 -- 172.21.15.42:0/2115685877 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd318133380 con 0x7fd31813c6d0 2024-09-16T04:03:18.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.339+0000 7fd31de20640 10 monclient: get_auth_request con 0x7fd31806db90 auth_method 0 2024-09-16T04:03:18.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.339+0000 7fd31de20640 1 --2- 172.21.15.42:0/2115685877 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd31806db90 0x7fd3181ceeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T04:03:18.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.339+0000 7fd31e621640 1 --2- 172.21.15.42:0/2115685877 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd31813c6d0 0x7fd3180a30e0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7fd3100ac000 tx=0x7fd3100195d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:03:18.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.339+0000 7fd30e7fc640 10 client.?.objecter ms_handle_connect 0x7fd31813c6d0 2024-09-16T04:03:18.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.339+0000 7fd30e7fc640 10 client.?.objecter resend_mon_ops 2024-09-16T04:03:18.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.339+0000 7fd30e7fc640 1 -- 172.21.15.42:0/2115685877 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3100b2050 con 0x7fd31813c6d0 2024-09-16T04:03:18.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.339+0000 7fd30e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:03:18.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.339+0000 7fd30e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:03:18.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.339+0000 7fd30e7fc640 10 monclient: dump: 2024-09-16T04:03:18.357 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:03:18.358 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:03:18.358 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:18.358 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:18.358 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:03:18.358 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:03:18.358 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:03:18.358 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:03:18.358 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:03:18.358 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:03:18.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.339+0000 7fd30e7fc640 10 monclient: _finish_auth 0 2024-09-16T04:03:18.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.339+0000 7fd30e7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:03:18.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.339+0000 7fd30e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:03:18.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.339+0000 7fd30e7fc640 1 -- 172.21.15.42:0/2115685877 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd3100b3040 con 0x7fd31813c6d0 2024-09-16T04:03:18.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.339+0000 7fd30e7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:03:18.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.339+0000 7fd30e7fc640 1 -- 172.21.15.42:0/2115685877 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3100ad980 con 0x7fd31813c6d0 2024-09-16T04:03:18.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.339+0000 7fd30e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:03:18.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.339+0000 7fd30e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:03:18.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.339+0000 7fd30e7fc640 10 monclient: dump: 2024-09-16T04:03:18.359 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:03:18.359 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:03:18.359 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:18.359 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:18.359 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:03:18.359 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:03:18.359 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:03:18.359 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:03:18.360 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:03:18.360 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:03:18.360 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.339+0000 7fd320084640 5 monclient: authenticate success, global_id 825627 2024-09-16T04:03:18.360 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.339+0000 7fd320084640 10 monclient: _renew_subs 2024-09-16T04:03:18.360 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.339+0000 7fd320084640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:03:18.360 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.339+0000 7fd320084640 1 -- 172.21.15.42:0/2115685877 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd31806f6a0 con 0x7fd31813c6d0 2024-09-16T04:03:18.360 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.339+0000 7fd30f7fe640 4 set_mon_vals no callback set 2024-09-16T04:03:18.360 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.339+0000 7fd320084640 10 client.825627.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:03:18.360 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.339+0000 7fd320084640 10 monclient: _renew_subs 2024-09-16T04:03:18.360 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.339+0000 7fd320084640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:03:18.360 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.339+0000 7fd320084640 1 -- 172.21.15.42:0/2115685877 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd31809dda0 con 0x7fd31813c6d0 2024-09-16T04:03:18.360 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.339+0000 7fd320084640 1 librados: init done 2024-09-16T04:03:18.360 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.339+0000 7fd320084640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:03:18.361 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.339+0000 7fd320084640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:03:18.361 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.339+0000 7fd320084640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:03:18.361 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.339+0000 7fd30e7fc640 1 -- 172.21.15.42:0/2115685877 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fd3100c0020 con 0x7fd31813c6d0 2024-09-16T04:03:18.361 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.339+0000 7fd320084640 1 -- 172.21.15.42:0/2115685877 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd3180a37f0 con 0x7fd31813c6d0 2024-09-16T04:03:18.361 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.339+0000 7fd30e7fc640 1 --2- 172.21.15.42:0/2115685877 >> v2:172.21.15.154:6812/254845342 conn(0x7fd2fc037840 0x7fd2fc039d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:03:18.361 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.339+0000 7fd30e7fc640 1 -- 172.21.15.42:0/2115685877 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(650..650 src has 451..650) ==== 6306+0+0 (secure 0 0 0) 0x7fd3100e3e80 con 0x7fd31813c6d0 2024-09-16T04:03:18.361 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.339+0000 7fd30e7fc640 10 client.825627.objecter ms_dispatch 0x7fd318000ba0 osd_map(650..650 src has 451..650) 2024-09-16T04:03:18.361 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.339+0000 7fd30e7fc640 3 client.825627.objecter handle_osd_map got epochs [650,650] > 0 2024-09-16T04:03:18.361 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.339+0000 7fd30e7fc640 3 client.825627.objecter handle_osd_map decoding full epoch 650 2024-09-16T04:03:18.361 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.339+0000 7fd30e7fc640 20 client.825627.objecter dump_active .. 0 homeless 2024-09-16T04:03:18.361 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.339+0000 7fd31de20640 1 --2- 172.21.15.42:0/2115685877 >> v2:172.21.15.154:6812/254845342 conn(0x7fd2fc037840 0x7fd2fc039d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:03:18.361 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.339+0000 7fd31de20640 10 monclient: get_auth_request con 0x7fd2fc037840 auth_method 0 2024-09-16T04:03:18.361 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.343+0000 7fd31de20640 1 --2- 172.21.15.42:0/2115685877 >> v2:172.21.15.154:6812/254845342 conn(0x7fd2fc037840 0x7fd2fc039d00 secure :-1 s=READY pgs=3800 cs=0 l=1 rev1=1 crypto rx=0x7fd314007fd0 tx=0x7fd314009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:03:18.361 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.343+0000 7fd30e7fc640 10 client.825627.objecter ms_handle_connect 0x7fd2fc037840 2024-09-16T04:03:18.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.343+0000 7fd30e7fc640 1 -- 172.21.15.42:0/2115685877 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7fd3100c7020 con 0x7fd31813c6d0 2024-09-16T04:03:18.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.343+0000 7fd30e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:03:18.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.343+0000 7fd30e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:03:18.578 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.575+0000 7fd320084640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:03:18.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.575+0000 7fd320084640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:03:18.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.575+0000 7fd320084640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:03:18.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.575+0000 7fd320084640 1 -- 172.21.15.42:0/2115685877 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fd31813b6a0 con 0x7fd31813c6d0 2024-09-16T04:03:18.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.575+0000 7fd30e7fc640 1 -- 172.21.15.42:0/2115685877 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7fd310016540 con 0x7fd31813c6d0 2024-09-16T04:03:18.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.575+0000 7fd30e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:03:18.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.575+0000 7fd30e7fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:03:18.580 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":190,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:03:18.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.575+0000 7fd320084640 10 librados: watch_flush enter 2024-09-16T04:03:18.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.579+0000 7fd320084640 10 librados: watch_flush exit 2024-09-16T04:03:18.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.579+0000 7fd320084640 20 client.825627.objecter shutdown clearing up homeless session... 2024-09-16T04:03:18.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.579+0000 7fd320084640 10 client.825627.objecter successfully canceled tick 2024-09-16T04:03:18.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.579+0000 7fd320084640 1 -- 172.21.15.42:0/2115685877 >> v2:172.21.15.154:6812/254845342 conn(0x7fd2fc037840 msgr2=0x7fd2fc039d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:03:18.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.579+0000 7fd320084640 1 --2- 172.21.15.42:0/2115685877 >> v2:172.21.15.154:6812/254845342 conn(0x7fd2fc037840 0x7fd2fc039d00 secure :-1 s=READY pgs=3800 cs=0 l=1 rev1=1 crypto rx=0x7fd314007fd0 tx=0x7fd314009040 comp rx=0 tx=0).stop 2024-09-16T04:03:18.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.579+0000 7fd320084640 10 monclient: shutdown 2024-09-16T04:03:18.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.579+0000 7fd320084640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:03:18.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.579+0000 7fd320084640 1 -- 172.21.15.42:0/2115685877 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd31813c6d0 msgr2=0x7fd3180a30e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:03:18.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.579+0000 7fd320084640 1 --2- 172.21.15.42:0/2115685877 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd31813c6d0 0x7fd3180a30e0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7fd3100ac000 tx=0x7fd3100195d0 comp rx=0 tx=0).stop 2024-09-16T04:03:18.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.579+0000 7fd320084640 1 -- 172.21.15.42:0/2115685877 shutdown_connections 2024-09-16T04:03:18.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.579+0000 7fd320084640 1 --2- 172.21.15.42:0/2115685877 >> v2:172.21.15.154:6812/254845342 conn(0x7fd2fc037840 0x7fd2fc039d00 unknown :-1 s=CLOSED pgs=3800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:03:18.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.579+0000 7fd320084640 1 --2- 172.21.15.42:0/2115685877 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd31813c6d0 0x7fd3180a30e0 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:03:18.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.579+0000 7fd320084640 1 --2- 172.21.15.42:0/2115685877 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd3181340c0 0x7fd3180a2ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:03:18.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.579+0000 7fd320084640 1 --2- 172.21.15.42:0/2115685877 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd31806db90 0x7fd3181ceeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:03:18.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.579+0000 7fd320084640 1 -- 172.21.15.42:0/2115685877 >> 172.21.15.42:0/2115685877 conn(0x7fd3180a9fb0 msgr2=0x7fd3180a9180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:03:18.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.579+0000 7fd320084640 1 -- 172.21.15.42:0/2115685877 shutdown_connections 2024-09-16T04:03:18.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.579+0000 7fd320084640 1 -- 172.21.15.42:0/2115685877 wait complete. 2024-09-16T04:03:18.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.579+0000 7fd320084640 1 librados: shutdown 2024-09-16T04:03:18.598 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T04:03:18.598 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T04:03:18.598 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T04:03:18.717 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.715+0000 7fbc35880640 10 monclient: get_monmap_and_config 2024-09-16T04:03:18.718 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.715+0000 7fbc35880640 10 monclient: build_initial_monmap 2024-09-16T04:03:18.718 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.715+0000 7fbc35880640 10 monclient: monmap: 2024-09-16T04:03:18.718 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:03:18.718 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:03:18.718 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:03:18.716368+0000 2024-09-16T04:03:18.718 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:03:18.716368+0000 2024-09-16T04:03:18.718 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:03:18.718 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:03:18.718 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:03:18.718 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:03:18.718 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:03:18.718 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:03:18.718 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:03:18.718 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:03:18.724 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.715+0000 7fbc35880640 1 Processor -- start 2024-09-16T04:03:18.724 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.715+0000 7fbc35880640 1 -- start start 2024-09-16T04:03:18.724 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.715+0000 7fbc35880640 10 monclient: init 2024-09-16T04:03:18.724 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.715+0000 7fbc35880640 10 monclient: _reopen_session rank -1 2024-09-16T04:03:18.724 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.715+0000 7fbc35880640 10 monclient: _add_conns ranks=[2,0,1,3,4] 2024-09-16T04:03:18.725 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.715+0000 7fbc35880640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbc3006f920 0x7fbc30071a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:03:18.725 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.715+0000 7fbc35880640 10 monclient(hunting): picked mon.noname-b con 0x7fbc3006f920 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:03:18.725 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.715+0000 7fbc35880640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbc3006fd00 addr v1:172.21.15.42:6789/0 2024-09-16T04:03:18.725 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.715+0000 7fbc35880640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbc30139330 addr v1:172.21.15.154:6789/0 2024-09-16T04:03:18.725 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.715+0000 7fbc35880640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbc30070e90 con 0x7fbc3006fd00 2024-09-16T04:03:18.725 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.715+0000 7fbc35880640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbc300712d0 con 0x7fbc30139330 2024-09-16T04:03:18.725 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.715+0000 7fbc35880640 10 monclient(hunting): start opening mon connection 2024-09-16T04:03:18.725 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.715+0000 7fbc35880640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fbc30072f00 con 0x7fbc3006f920 2024-09-16T04:03:18.725 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.715+0000 7fbc35880640 10 monclient(hunting): _renew_subs 2024-09-16T04:03:18.725 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.715+0000 7fbc35880640 10 monclient(hunting): authenticate will time out at 2245.685623s 2024-09-16T04:03:18.725 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.715+0000 7fbc2e7fc640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fbc3006fd00 0x7fbc300700e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:43002/0 (socket says 172.21.15.42:43002) 2024-09-16T04:03:18.725 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.715+0000 7fbc2e7fc640 1 -- 172.21.15.42:0/999041649 learned_addr learned my addr 172.21.15.42:0/999041649 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T04:03:18.725 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.715+0000 7fbc2effd640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbc3006f920 0x7fbc30071a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:03:18.725 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.715+0000 7fbc2effd640 10 monclient(hunting): get_auth_request con 0x7fbc3006f920 auth_method 0 2024-09-16T04:03:18.726 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.715+0000 7fbc2effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:03:18.726 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.715+0000 7fbc2effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:03:18.726 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.715+0000 7fbc2effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:03:18.726 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.715+0000 7fbc2effd640 1 -- 172.21.15.42:0/999041649 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbc3006f920 msgr2=0x7fbc30071a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:03:18.726 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.715+0000 7fbc2effd640 1 -- 172.21.15.42:0/999041649 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbc3006f920 msgr2=0x7fbc30071a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:03:18.726 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.715+0000 7fbc2effd640 1 --2- 172.21.15.42:0/999041649 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbc3006f920 0x7fbc30071a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:03:18.726 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.715+0000 7fbc2effd640 1 --2- 172.21.15.42:0/999041649 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbc3006f920 0x7fbc30071a60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:03:18.726 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.715+0000 7fbc2dffb640 1 -- 172.21.15.42:0/999041649 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4200112932 0 0) 0x7fbc30070e90 con 0x7fbc3006fd00 2024-09-16T04:03:18.726 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.715+0000 7fbc2dffb640 10 monclient(hunting): _negotiate 2024-09-16T04:03:18.726 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.715+0000 7fbc2dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:03:18.726 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.715+0000 7fbc2dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:03:18.726 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.715+0000 7fbc2dffb640 10 monclient(hunting): my global_id is 825630 2024-09-16T04:03:18.726 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.715+0000 7fbc2dffb640 1 -- 172.21.15.42:0/999041649 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbc18005740 con 0x7fbc3006fd00 2024-09-16T04:03:18.726 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.715+0000 7fbc2dffb640 1 -- 172.21.15.42:0/999041649 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 387075985 0 0) 0x7fbc18005740 con 0x7fbc3006fd00 2024-09-16T04:03:18.727 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.715+0000 7fbc2dffb640 1 -- 172.21.15.42:0/999041649 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbc3006f920 msgr2=0x7fbc30071a60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:03:18.727 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.715+0000 7fbc2dffb640 1 --2- 172.21.15.42:0/999041649 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbc3006f920 0x7fbc30071a60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:03:18.727 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.715+0000 7fbc2dffb640 1 -- 172.21.15.42:0/999041649 >> v1:172.21.15.154:6789/0 conn(0x7fbc30139330 legacy=0x7fbc3013b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:03:18.727 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.715+0000 7fbc2dffb640 10 monclient: _finish_hunting 0 2024-09-16T04:03:18.727 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.715+0000 7fbc2dffb640 1 monclient: found mon.noname-a-legacy 2024-09-16T04:03:18.727 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.715+0000 7fbc2dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:03:18.727 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.715+0000 7fbc2dffb640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.42:6789/0 2024-09-16T04:03:18.727 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.715+0000 7fbc2dffb640 1 -- 172.21.15.42:0/999041649 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc300725f0 con 0x7fbc3006fd00 2024-09-16T04:03:18.727 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.715+0000 7fbc2dffb640 10 monclient: _finish_auth 0 2024-09-16T04:03:18.727 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.715+0000 7fbc2dffb640 10 monclient: _check_auth_tickets 2024-09-16T04:03:18.727 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.715+0000 7fbc2dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:03:18.727 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.715+0000 7fbc35880640 5 monclient: authenticate success, global_id 825630 2024-09-16T04:03:18.727 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.715+0000 7fbc2dffb640 1 -- 172.21.15.42:0/999041649 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2177413734 0 0) 0x7fbc24001680 con 0x7fbc3006fd00 2024-09-16T04:03:18.728 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.715+0000 7fbc35880640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:03:18.728 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.715+0000 7fbc2dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:03:18.728 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.715+0000 7fbc2dffb640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T04:03:18.728 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.715+0000 7fbc2dffb640 10 monclient: dump: 2024-09-16T04:03:18.728 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:03:18.728 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:03:18.728 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:18.728 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:18.728 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:03:18.728 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:03:18.728 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:03:18.728 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:03:18.728 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:03:18.728 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:03:18.729 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.715+0000 7fbc2dffb640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] but i'm connected to v1:172.21.15.42:6789/0, reconnecting 2024-09-16T04:03:18.729 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.715+0000 7fbc2dffb640 10 monclient: _reopen_session rank -1 2024-09-16T04:03:18.729 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.715+0000 7fbc2dffb640 1 -- 172.21.15.42:0/999041649 >> v1:172.21.15.42:6789/0 conn(0x7fbc3006fd00 legacy=0x7fbc300700e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:03:18.729 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.715+0000 7fbc2dffb640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:03:18.729 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.715+0000 7fbc2dffb640 1 --2- 172.21.15.42:0/999041649 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbc3006f920 0x7fbc18005c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:03:18.729 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.715+0000 7fbc2dffb640 10 monclient(hunting): picked mon.c con 0x7fbc3006f920 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:03:18.729 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.715+0000 7fbc2dffb640 1 --2- 172.21.15.42:0/999041649 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbc180061f0 0x7fbc18006a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:03:18.729 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.715+0000 7fbc2dffb640 10 monclient(hunting): picked mon.b con 0x7fbc180061f0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:03:18.729 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.715+0000 7fbc2dffb640 1 --2- 172.21.15.42:0/999041649 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbc18006f60 0x7fbc180073e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:03:18.729 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.715+0000 7fbc2dffb640 10 monclient(hunting): picked mon.a con 0x7fbc18006f60 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:03:18.729 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.715+0000 7fbc2dffb640 10 monclient(hunting): start opening mon connection 2024-09-16T04:03:18.729 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.715+0000 7fbc2dffb640 1 -- 172.21.15.42:0/999041649 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fbc30072f00 con 0x7fbc18006f60 2024-09-16T04:03:18.729 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.715+0000 7fbc2dffb640 10 monclient(hunting): start opening mon connection 2024-09-16T04:03:18.729 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.715+0000 7fbc2dffb640 1 -- 172.21.15.42:0/999041649 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fbc1800de20 con 0x7fbc180061f0 2024-09-16T04:03:18.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.715+0000 7fbc2dffb640 10 monclient(hunting): start opening mon connection 2024-09-16T04:03:18.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.715+0000 7fbc2dffb640 1 -- 172.21.15.42:0/999041649 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fbc1800e020 con 0x7fbc3006f920 2024-09-16T04:03:18.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.715+0000 7fbc2dffb640 10 monclient(hunting): _renew_subs 2024-09-16T04:03:18.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.715+0000 7fbc2dffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:03:18.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.715+0000 7fbc35880640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-16T04:03:18.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.715+0000 7fbc2effd640 1 --2- 172.21.15.42:0/999041649 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbc3006f920 0x7fbc18005c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:03:18.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.715+0000 7fbc2effd640 10 monclient(hunting): get_auth_request con 0x7fbc3006f920 auth_method 0 2024-09-16T04:03:18.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.715+0000 7fbc2effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:03:18.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.715+0000 7fbc2effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:03:18.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.715+0000 7fbc2effd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:03:18.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.715+0000 7fbc2f7fe640 1 --2- 172.21.15.42:0/999041649 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbc18006f60 0x7fbc180073e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:03:18.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.715+0000 7fbc2f7fe640 10 monclient(hunting): get_auth_request con 0x7fbc18006f60 auth_method 0 2024-09-16T04:03:18.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.715+0000 7fbc2f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:03:18.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.715+0000 7fbc2f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:03:18.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.715+0000 7fbc2f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:03:18.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.715+0000 7fbc2effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:03:18.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.715+0000 7fbc2effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:03:18.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.715+0000 7fbc2effd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:03:18.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.715+0000 7fbc2f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:03:18.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.715+0000 7fbc2f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:03:18.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.715+0000 7fbc2f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:03:18.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.715+0000 7fbc2e7fc640 1 --2- 172.21.15.42:0/999041649 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbc180061f0 0x7fbc18006a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:03:18.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.715+0000 7fbc2e7fc640 10 monclient(hunting): get_auth_request con 0x7fbc180061f0 auth_method 0 2024-09-16T04:03:18.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.715+0000 7fbc2e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:03:18.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.715+0000 7fbc2e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:03:18.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.715+0000 7fbc2e7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:03:18.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.715+0000 7fbc2effd640 10 monclient(hunting): handle_auth_done global_id 825630 payload 293 2024-09-16T04:03:18.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.715+0000 7fbc2effd640 1 -- 172.21.15.42:0/999041649 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbc180061f0 msgr2=0x7fbc18006a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:03:18.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.715+0000 7fbc2effd640 1 --2- 172.21.15.42:0/999041649 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbc180061f0 0x7fbc18006a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:03:18.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.715+0000 7fbc2effd640 1 -- 172.21.15.42:0/999041649 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbc18006f60 msgr2=0x7fbc180073e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:03:18.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.715+0000 7fbc2effd640 1 --2- 172.21.15.42:0/999041649 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbc18006f60 0x7fbc180073e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:03:18.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.715+0000 7fbc2effd640 10 monclient: _finish_hunting 0 2024-09-16T04:03:18.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.715+0000 7fbc2effd640 1 monclient: found mon.c 2024-09-16T04:03:18.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.715+0000 7fbc2effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:03:18.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.715+0000 7fbc2effd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:03:18.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.715+0000 7fbc2effd640 1 -- 172.21.15.42:0/999041649 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc18005740 con 0x7fbc3006f920 2024-09-16T04:03:18.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.715+0000 7fbc2effd640 10 monclient: _finish_auth 0 2024-09-16T04:03:18.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.715+0000 7fbc2effd640 10 monclient: _check_auth_tickets 2024-09-16T04:03:18.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.715+0000 7fbc2effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:03:18.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.719+0000 7fbc2effd640 1 --2- 172.21.15.42:0/999041649 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbc3006f920 0x7fbc18005c00 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fbc2000c820 tx=0x7fbc2001c510 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7f8bd5a66cc1c0b4 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:03:18.733 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.719+0000 7fbc2dffb640 1 -- 172.21.15.42:0/999041649 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc2000f040 con 0x7fbc3006f920 2024-09-16T04:03:18.733 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.719+0000 7fbc2dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:03:18.733 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.719+0000 7fbc2dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:03:18.733 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.719+0000 7fbc2dffb640 10 monclient: dump: 2024-09-16T04:03:18.733 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:03:18.733 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:03:18.733 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:18.733 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:18.733 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:03:18.733 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:03:18.733 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:03:18.733 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:03:18.733 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:03:18.734 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:03:18.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.719+0000 7fbc35880640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:03:18.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.719+0000 7fbc2dffb640 1 -- 172.21.15.42:0/999041649 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc200a9620 con 0x7fbc3006f920 2024-09-16T04:03:18.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.719+0000 7fbc2dffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:03:18.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.719+0000 7fbc35880640 10 monclient: get_monmap_and_config success 2024-09-16T04:03:18.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.719+0000 7fbc35880640 4 set_mon_vals no callback set 2024-09-16T04:03:18.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.719+0000 7fbc2dffb640 1 -- 172.21.15.42:0/999041649 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc200a9a10 con 0x7fbc3006f920 2024-09-16T04:03:18.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.719+0000 7fbc2dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:03:18.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.719+0000 7fbc2dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:03:18.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.719+0000 7fbc2dffb640 10 monclient: dump: 2024-09-16T04:03:18.734 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:03:18.734 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:03:18.734 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:18.734 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:18.734 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:03:18.735 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:03:18.735 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:03:18.735 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:03:18.735 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:03:18.735 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:03:18.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.719+0000 7fbc35880640 10 monclient: shutdown 2024-09-16T04:03:18.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.719+0000 7fbc35880640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:03:18.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.719+0000 7fbc35880640 1 -- 172.21.15.42:0/999041649 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbc3006f920 msgr2=0x7fbc18005c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:03:18.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.719+0000 7fbc35880640 1 --2- 172.21.15.42:0/999041649 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbc3006f920 0x7fbc18005c00 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fbc2000c820 tx=0x7fbc2001c510 comp rx=0 tx=0).stop 2024-09-16T04:03:18.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.719+0000 7fbc2f7fe640 1 -- 172.21.15.42:0/999041649 reap_dead start 2024-09-16T04:03:18.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.719+0000 7fbc35880640 1 -- 172.21.15.42:0/999041649 shutdown_connections 2024-09-16T04:03:18.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.719+0000 7fbc35880640 1 -- 172.21.15.42:0/999041649 >> 172.21.15.42:0/999041649 conn(0x7fbc300a3220 msgr2=0x7fbc300a3630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:03:18.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.719+0000 7fbc35880640 1 -- 172.21.15.42:0/999041649 shutdown_connections 2024-09-16T04:03:18.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.719+0000 7fbc35880640 1 -- 172.21.15.42:0/999041649 wait complete. 2024-09-16T04:03:18.736 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.719+0000 7fbc35880640 10 monclient: build_initial_monmap 2024-09-16T04:03:18.736 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.719+0000 7fbc35880640 10 monclient: monmap: 2024-09-16T04:03:18.736 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:03:18.736 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:03:18.736 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:03:18.736 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:03:18.736 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:03:18.736 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:03:18.736 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:03:18.736 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:03:18.736 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:03:18.736 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:03:18.736 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.719+0000 7fbc35880640 1 librados: starting msgr at 2024-09-16T04:03:18.737 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.719+0000 7fbc35880640 1 librados: starting objecter 2024-09-16T04:03:18.737 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.719+0000 7fbc35880640 1 Processor -- start 2024-09-16T04:03:18.737 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.719+0000 7fbc35880640 1 -- start start 2024-09-16T04:03:18.737 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.719+0000 7fbc35880640 1 librados: setting wanted keys 2024-09-16T04:03:18.737 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.719+0000 7fbc35880640 1 librados: calling monclient init 2024-09-16T04:03:18.737 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.719+0000 7fbc35880640 10 monclient: init 2024-09-16T04:03:18.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.719+0000 7fbc35880640 10 monclient: _reopen_session rank -1 2024-09-16T04:03:18.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.719+0000 7fbc35880640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:03:18.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.719+0000 7fbc35880640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbc3006f920 0x7fbc301319d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:03:18.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.719+0000 7fbc35880640 10 monclient(hunting): picked mon.noname-a con 0x7fbc3006f920 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:03:18.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.719+0000 7fbc35880640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbc3009dbf0 0x7fbc3009e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:03:18.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.719+0000 7fbc35880640 10 monclient(hunting): picked mon.noname-b con 0x7fbc3009dbf0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:03:18.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.719+0000 7fbc35880640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbc3009e5b0 0x7fbc301d2530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:03:18.744 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.719+0000 7fbc35880640 10 monclient(hunting): picked mon.noname-c con 0x7fbc3009e5b0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:03:18.744 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.719+0000 7fbc35880640 10 monclient(hunting): start opening mon connection 2024-09-16T04:03:18.744 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.719+0000 7fbc35880640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fbc30071a60 con 0x7fbc3006f920 2024-09-16T04:03:18.744 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.719+0000 7fbc35880640 10 monclient(hunting): start opening mon connection 2024-09-16T04:03:18.744 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.719+0000 7fbc35880640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fbc300728d0 con 0x7fbc3009dbf0 2024-09-16T04:03:18.744 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.719+0000 7fbc35880640 10 monclient(hunting): start opening mon connection 2024-09-16T04:03:18.744 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.719+0000 7fbc35880640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fbc30071be0 con 0x7fbc3009e5b0 2024-09-16T04:03:18.744 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.719+0000 7fbc35880640 10 monclient(hunting): _renew_subs 2024-09-16T04:03:18.744 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.719+0000 7fbc35880640 10 monclient(hunting): authenticate will time out at 2245.691685s 2024-09-16T04:03:18.744 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.719+0000 7fbc2effd640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbc3006f920 0x7fbc301319d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:03:18.744 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.719+0000 7fbc2f7fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbc3009e5b0 0x7fbc301d2530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:03:18.744 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.719+0000 7fbc2effd640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbc3006f920 0x7fbc301319d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:42370/0 (socket says 172.21.15.42:42370) 2024-09-16T04:03:18.744 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.719+0000 7fbc2effd640 1 -- 172.21.15.42:0/1599494071 learned_addr learned my addr 172.21.15.42:0/1599494071 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:03:18.744 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.719+0000 7fbc2effd640 10 monclient(hunting): get_auth_request con 0x7fbc3006f920 auth_method 0 2024-09-16T04:03:18.744 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.719+0000 7fbc2effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:03:18.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.719+0000 7fbc2effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:03:18.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.719+0000 7fbc2effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:03:18.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.719+0000 7fbc2f7fe640 10 monclient(hunting): get_auth_request con 0x7fbc3009e5b0 auth_method 0 2024-09-16T04:03:18.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.719+0000 7fbc2f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:03:18.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.719+0000 7fbc2f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:03:18.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.719+0000 7fbc2f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:03:18.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.719+0000 7fbc2effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:03:18.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.719+0000 7fbc2effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:03:18.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.723+0000 7fbc2f7fe640 1 -- 172.21.15.42:0/1599494071 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbc3009e5b0 msgr2=0x7fbc301d2530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-16T04:03:18.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.723+0000 7fbc2f7fe640 1 -- 172.21.15.42:0/1599494071 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbc3009e5b0 msgr2=0x7fbc301d2530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:03:18.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.723+0000 7fbc2effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:03:18.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.723+0000 7fbc2f7fe640 1 --2- 172.21.15.42:0/1599494071 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbc3009e5b0 0x7fbc301d2530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:03:18.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.723+0000 7fbc2e7fc640 1 --2- 172.21.15.42:0/1599494071 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbc3009dbf0 0x7fbc3009e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:03:18.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.723+0000 7fbc2f7fe640 1 --2- 172.21.15.42:0/1599494071 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbc3009e5b0 0x7fbc301d2530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:03:18.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.723+0000 7fbc2e7fc640 10 monclient(hunting): get_auth_request con 0x7fbc3009dbf0 auth_method 0 2024-09-16T04:03:18.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.723+0000 7fbc2e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:03:18.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.723+0000 7fbc2e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:03:18.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.723+0000 7fbc2e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:03:18.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.723+0000 7fbc2effd640 10 monclient(hunting): handle_auth_done global_id 825633 payload 824 2024-09-16T04:03:18.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.723+0000 7fbc2effd640 1 -- 172.21.15.42:0/1599494071 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbc3009e5b0 msgr2=0x7fbc301d2530 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:03:18.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.723+0000 7fbc2effd640 1 --2- 172.21.15.42:0/1599494071 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbc3009e5b0 0x7fbc301d2530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:03:18.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.723+0000 7fbc2effd640 1 -- 172.21.15.42:0/1599494071 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbc3009dbf0 msgr2=0x7fbc3009e070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:03:18.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.723+0000 7fbc2effd640 1 --2- 172.21.15.42:0/1599494071 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbc3009dbf0 0x7fbc3009e070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:03:18.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.723+0000 7fbc2effd640 10 monclient: _finish_hunting 0 2024-09-16T04:03:18.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.723+0000 7fbc2effd640 1 monclient: found mon.noname-a 2024-09-16T04:03:18.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.723+0000 7fbc2effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:03:18.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.723+0000 7fbc2effd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:03:18.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.723+0000 7fbc2effd640 1 -- 172.21.15.42:0/1599494071 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc3009ea30 con 0x7fbc3006f920 2024-09-16T04:03:18.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.723+0000 7fbc2e7fc640 1 --2- 172.21.15.42:0/1599494071 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbc3009dbf0 0x7fbc3009e070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T04:03:18.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.723+0000 7fbc2effd640 1 --2- 172.21.15.42:0/1599494071 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbc3006f920 0x7fbc301319d0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fbc200ab040 tx=0x7fbc200192b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:03:18.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.723+0000 7fbc17fff640 10 client.?.objecter ms_handle_connect 0x7fbc3006f920 2024-09-16T04:03:18.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.723+0000 7fbc17fff640 10 client.?.objecter resend_mon_ops 2024-09-16T04:03:18.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.723+0000 7fbc17fff640 1 -- 172.21.15.42:0/1599494071 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc200c1070 con 0x7fbc3006f920 2024-09-16T04:03:18.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.723+0000 7fbc17fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:03:18.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.723+0000 7fbc17fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:03:18.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.723+0000 7fbc17fff640 10 monclient: dump: 2024-09-16T04:03:18.747 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:03:18.747 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:03:18.748 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:18.748 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:18.748 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:03:18.748 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:03:18.748 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:03:18.748 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:03:18.748 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:03:18.748 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:03:18.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.723+0000 7fbc17fff640 10 monclient: _finish_auth 0 2024-09-16T04:03:18.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.723+0000 7fbc17fff640 10 monclient: _check_auth_tickets 2024-09-16T04:03:18.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.723+0000 7fbc17fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:03:18.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.723+0000 7fbc17fff640 1 -- 172.21.15.42:0/1599494071 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc200bc3f0 con 0x7fbc3006f920 2024-09-16T04:03:18.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.723+0000 7fbc35880640 5 monclient: authenticate success, global_id 825633 2024-09-16T04:03:18.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.723+0000 7fbc35880640 10 monclient: _renew_subs 2024-09-16T04:03:18.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.723+0000 7fbc35880640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:03:18.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.723+0000 7fbc35880640 1 -- 172.21.15.42:0/1599494071 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc300a1810 con 0x7fbc3006f920 2024-09-16T04:03:18.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.723+0000 7fbc17fff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:03:18.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.723+0000 7fbc17fff640 1 -- 172.21.15.42:0/1599494071 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc200bc7a0 con 0x7fbc3006f920 2024-09-16T04:03:18.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.723+0000 7fbc17fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:03:18.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.723+0000 7fbc17fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:03:18.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.723+0000 7fbc17fff640 10 monclient: dump: 2024-09-16T04:03:18.749 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:03:18.749 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:03:18.749 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:18.749 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:18.749 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:03:18.749 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:03:18.750 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:03:18.750 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:03:18.750 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:03:18.750 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:03:18.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.723+0000 7fbc2d7fa640 4 set_mon_vals no callback set 2024-09-16T04:03:18.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.723+0000 7fbc35880640 10 client.825633.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:03:18.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.723+0000 7fbc35880640 10 monclient: _renew_subs 2024-09-16T04:03:18.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.723+0000 7fbc35880640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:03:18.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.723+0000 7fbc35880640 1 -- 172.21.15.42:0/1599494071 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc300a1250 con 0x7fbc3006f920 2024-09-16T04:03:18.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.723+0000 7fbc35880640 1 librados: init done 2024-09-16T04:03:18.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.723+0000 7fbc35880640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:03:18.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.723+0000 7fbc35880640 1 --2- 172.21.15.42:0/1599494071 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbc3002d7d0 0x7fbc30062ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:03:18.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.723+0000 7fbc35880640 10 monclient(hunting): start opening mon connection 2024-09-16T04:03:18.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.723+0000 7fbc35880640 1 -- 172.21.15.42:0/1599494071 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fbc300a5530 con 0x7fbc3002d7d0 2024-09-16T04:03:18.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.723+0000 7fbc2e7fc640 1 --2- 172.21.15.42:0/1599494071 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbc3002d7d0 0x7fbc30062ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:03:18.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.723+0000 7fbc2e7fc640 10 monclient: get_auth_request con 0x7fbc3002d7d0 auth_method 0 2024-09-16T04:03:18.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.723+0000 7fbc2e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:03:18.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.723+0000 7fbc2e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:03:18.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.723+0000 7fbc2e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:03:18.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.723+0000 7fbc17fff640 1 -- 172.21.15.42:0/1599494071 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fbc200c4070 con 0x7fbc3006f920 2024-09-16T04:03:18.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.723+0000 7fbc2e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:03:18.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.723+0000 7fbc2e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:03:18.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.723+0000 7fbc2e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:03:18.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.723+0000 7fbc2e7fc640 10 monclient(hunting): handle_auth_done global_id 825636 payload 824 2024-09-16T04:03:18.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.723+0000 7fbc17fff640 1 --2- 172.21.15.42:0/1599494071 >> v2:172.21.15.154:6812/254845342 conn(0x7fbc1c037870 0x7fbc1c039d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:03:18.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.723+0000 7fbc2e7fc640 1 -- 172.21.15.42:0/1599494071 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbc300a0360 con 0x7fbc3002d7d0 2024-09-16T04:03:18.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.723+0000 7fbc17fff640 1 -- 172.21.15.42:0/1599494071 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(650..650 src has 451..650) ==== 6306+0+0 (secure 0 0 0) 0x7fbc200e3940 con 0x7fbc3006f920 2024-09-16T04:03:18.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.723+0000 7fbc17fff640 10 client.825633.objecter ms_dispatch 0x7fbc30000ba0 osd_map(650..650 src has 451..650) 2024-09-16T04:03:18.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.723+0000 7fbc17fff640 3 client.825633.objecter handle_osd_map got epochs [650,650] > 0 2024-09-16T04:03:18.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.723+0000 7fbc17fff640 3 client.825633.objecter handle_osd_map decoding full epoch 650 2024-09-16T04:03:18.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.723+0000 7fbc17fff640 20 client.825633.objecter dump_active .. 0 homeless 2024-09-16T04:03:18.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.723+0000 7fbc17fff640 10 client.825633.objecter ms_handle_connect 0x7fbc3002d7d0 2024-09-16T04:03:18.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.723+0000 7fbc2e7fc640 1 --2- 172.21.15.42:0/1599494071 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbc3002d7d0 0x7fbc30062ea0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fbc24001210 tx=0x7fbc24010c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:03:18.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.723+0000 7fbc17fff640 10 client.825633.objecter resend_mon_ops 2024-09-16T04:03:18.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.723+0000 7fbc17fff640 1 -- 172.21.15.42:0/1599494071 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc24018020 con 0x7fbc3002d7d0 2024-09-16T04:03:18.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.723+0000 7fbc17fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:03:18.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.723+0000 7fbc17fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:03:18.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.723+0000 7fbc2f7fe640 1 --2- 172.21.15.42:0/1599494071 >> v2:172.21.15.154:6812/254845342 conn(0x7fbc1c037870 0x7fbc1c039d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:03:18.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.723+0000 7fbc17fff640 10 monclient: dump: 2024-09-16T04:03:18.752 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:03:18.752 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:03:18.752 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:18.753 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:18.753 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:03:18.753 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:03:18.753 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:03:18.753 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:03:18.753 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:03:18.753 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:03:18.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.723+0000 7fbc2f7fe640 10 monclient: get_auth_request con 0x7fbc1c037870 auth_method 0 2024-09-16T04:03:18.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.723+0000 7fbc17fff640 1 -- 172.21.15.42:0/1599494071 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fbc2400e760 con 0x7fbc3002d7d0 2024-09-16T04:03:18.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.723+0000 7fbc17fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:03:18.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.723+0000 7fbc17fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:03:18.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.723+0000 7fbc17fff640 1 -- 172.21.15.42:0/1599494071 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbc3002d7d0 msgr2=0x7fbc30062ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:03:18.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.723+0000 7fbc17fff640 1 --2- 172.21.15.42:0/1599494071 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbc3002d7d0 0x7fbc30062ea0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fbc24001210 tx=0x7fbc24010c10 comp rx=0 tx=0).stop 2024-09-16T04:03:18.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.723+0000 7fbc17fff640 1 -- 172.21.15.42:0/1599494071 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbc3002d7d0 msgr2=0x7fbc30062ea0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:03:18.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.723+0000 7fbc17fff640 1 --2- 172.21.15.42:0/1599494071 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbc3002d7d0 0x7fbc30062ea0 secure :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fbc24001210 tx=0x7fbc24010c10 comp rx=0 tx=0).stop 2024-09-16T04:03:18.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.723+0000 7fbc17fff640 10 client.825633.objecter ms_handle_connect 0x7fbc1c037870 2024-09-16T04:03:18.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.723+0000 7fbc2f7fe640 1 --2- 172.21.15.42:0/1599494071 >> v2:172.21.15.154:6812/254845342 conn(0x7fbc1c037870 0x7fbc1c039d30 secure :-1 s=READY pgs=3801 cs=0 l=1 rev1=1 crypto rx=0x7fbc18005bd0 tx=0x7fbc28007c80 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:03:18.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.731+0000 7fbc35880640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:03:18.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.731+0000 7fbc35880640 1 --2- 172.21.15.42:0/1599494071 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbc300633e0 0x7fbc301d6a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:03:18.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.731+0000 7fbc35880640 10 monclient(hunting): start opening mon connection 2024-09-16T04:03:18.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.731+0000 7fbc35880640 1 -- 172.21.15.42:0/1599494071 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fbc3009fca0 con 0x7fbc300633e0 2024-09-16T04:03:18.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.731+0000 7fbc2e7fc640 1 --2- 172.21.15.42:0/1599494071 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbc300633e0 0x7fbc301d6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:03:18.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.731+0000 7fbc2e7fc640 10 monclient: get_auth_request con 0x7fbc300633e0 auth_method 0 2024-09-16T04:03:18.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.731+0000 7fbc2e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:03:18.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.731+0000 7fbc2e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:03:18.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.731+0000 7fbc2e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:03:18.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.731+0000 7fbc2e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:03:18.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.731+0000 7fbc2e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:03:18.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.731+0000 7fbc2e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:03:18.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.731+0000 7fbc2e7fc640 10 monclient(hunting): handle_auth_done global_id 825639 payload 824 2024-09-16T04:03:18.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.731+0000 7fbc2e7fc640 1 -- 172.21.15.42:0/1599494071 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fbc3002dbb0 con 0x7fbc300633e0 2024-09-16T04:03:18.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.731+0000 7fbc2e7fc640 1 --2- 172.21.15.42:0/1599494071 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbc300633e0 0x7fbc301d6a90 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fbc24019000 tx=0x7fbc24001160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:03:18.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.731+0000 7fbc17fff640 10 client.825633.objecter ms_handle_connect 0x7fbc300633e0 2024-09-16T04:03:18.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.731+0000 7fbc17fff640 10 client.825633.objecter resend_mon_ops 2024-09-16T04:03:18.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.731+0000 7fbc17fff640 1 -- 172.21.15.42:0/1599494071 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc24018070 con 0x7fbc300633e0 2024-09-16T04:03:18.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.731+0000 7fbc17fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:03:18.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.731+0000 7fbc17fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:03:18.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.731+0000 7fbc17fff640 10 monclient: dump: 2024-09-16T04:03:18.755 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:03:18.755 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:03:18.755 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:18.756 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:18.756 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:03:18.756 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:03:18.756 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:03:18.756 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:03:18.756 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:03:18.756 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:03:18.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.731+0000 7fbc17fff640 1 -- 172.21.15.42:0/1599494071 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4715 (secure 0 0 0) 0x7fbc2401d620 con 0x7fbc300633e0 2024-09-16T04:03:18.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.731+0000 7fbc17fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:03:18.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.731+0000 7fbc17fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:03:18.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.731+0000 7fbc17fff640 1 -- 172.21.15.42:0/1599494071 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbc300633e0 msgr2=0x7fbc301d6a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:03:18.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.731+0000 7fbc17fff640 1 --2- 172.21.15.42:0/1599494071 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbc300633e0 0x7fbc301d6a90 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fbc24019000 tx=0x7fbc24001160 comp rx=0 tx=0).stop 2024-09-16T04:03:18.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.731+0000 7fbc17fff640 1 -- 172.21.15.42:0/1599494071 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbc300633e0 msgr2=0x7fbc301d6a90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:03:18.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.731+0000 7fbc17fff640 1 --2- 172.21.15.42:0/1599494071 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbc300633e0 0x7fbc301d6a90 secure :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fbc24019000 tx=0x7fbc24001160 comp rx=0 tx=0).stop 2024-09-16T04:03:18.757 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T04:03:18.757 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:03:18.757 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:03:18.757 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "leader", 2024-09-16T04:03:18.757 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 190, 2024-09-16T04:03:18.757 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 688585, 2024-09-16T04:03:18.757 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T04:03:18.757 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T04:03:18.757 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T04:03:18.758 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T04:03:18.758 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:03:18.758 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 1, 2024-09-16T04:03:18.758 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:03:18.758 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T04:03:18.758 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T04:03:18.758 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:03:18.758 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:03:18.758 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:03:18.758 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:03:18.758 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:03:18.758 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:03:18.758 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:03:18.758 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:03:18.759 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:03:18.759 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:03:18.759 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:03:18.759 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:03:18.759 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:03:18.759 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T04:03:18.759 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:03:18.759 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:03:18.759 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:03:18.759 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:03:18.759 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:03:18.759 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:03:18.759 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:03:18.759 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:03:18.760 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:03:18.760 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:03:18.760 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:03:18.760 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:03:18.760 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:03:18.760 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T04:03:18.760 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T04:03:18.760 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T04:03:18.760 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T04:03:18.760 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T04:03:18.760 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T04:03:18.760 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:03:18.760 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:03:18.760 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T04:03:18.761 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T04:03:18.761 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T04:03:18.761 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T04:03:18.761 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T04:03:18.761 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T04:03:18.761 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T04:03:18.761 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:03:18.761 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T04:03:18.761 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:03:18.761 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:03:18.761 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:03:18.761 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:03:18.761 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:03:18.761 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:03:18.762 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:03:18.762 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:03:18.762 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:03:18.762 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:03:18.762 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:03:18.762 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:03:18.762 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T04:03:18.762 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:03:18.762 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T04:03:18.762 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:03:18.762 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:03:18.762 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:03:18.762 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:03:18.762 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:03:18.763 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:03:18.763 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:03:18.763 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T04:03:18.763 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:03:18.763 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:03:18.763 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:03:18.763 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:03:18.763 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T04:03:18.763 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:03:18.763 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:03:18.763 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:03:18.763 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:03:18.763 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T04:03:18.764 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T04:03:18.764 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:03:18.764 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:03:18.764 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:03:18.764 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:03:18.764 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:03:18.764 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:03:18.764 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:03:18.764 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:03:18.764 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:03:18.764 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:03:18.764 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:03:18.764 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T04:03:18.765 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:03:18.765 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:03:18.765 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:03:18.765 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:03:18.765 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T04:03:18.765 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:03:18.765 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:03:18.765 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:03:18.765 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:03:18.765 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T04:03:18.765 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T04:03:18.765 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:03:18.765 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:03:18.766 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:03:18.766 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:03:18.766 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:03:18.766 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:03:18.766 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:03:18.766 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:03:18.766 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:03:18.766 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:03:18.766 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:03:18.766 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T04:03:18.766 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:03:18.766 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:03:18.766 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:03:18.766 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:03:18.767 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T04:03:18.767 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:03:18.767 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:03:18.767 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:03:18.767 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:03:18.767 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T04:03:18.767 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T04:03:18.767 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:03:18.767 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:03:18.767 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:03:18.767 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:03:18.767 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:03:18.767 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:03:18.767 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T04:03:18.768 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T04:03:18.768 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:03:18.768 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:03:18.768 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:03:18.768 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:03:18.768 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:03:18.768 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:03:18.768 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T04:03:18.768 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:03:18.768 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:03:18.768 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:03:18.768 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 4 2024-09-16T04:03:18.768 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:03:18.768 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:03:18.769 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T04:03:18.769 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:03:18.769 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:03:18.769 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:03:18.769 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 7 2024-09-16T04:03:18.769 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:03:18.769 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:03:18.769 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:03:18.769 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T04:03:18.769 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T04:03:18.771 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.731+0000 7fbc35880640 10 librados: watch_flush enter 2024-09-16T04:03:18.771 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.731+0000 7fbc35880640 10 librados: watch_flush exit 2024-09-16T04:03:18.771 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.731+0000 7fbc35880640 20 client.825633.objecter shutdown clearing up homeless session... 2024-09-16T04:03:18.771 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.731+0000 7fbc35880640 10 client.825633.objecter successfully canceled tick 2024-09-16T04:03:18.771 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.731+0000 7fbc35880640 1 -- 172.21.15.42:0/1599494071 >> v2:172.21.15.154:6812/254845342 conn(0x7fbc1c037870 msgr2=0x7fbc1c039d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:03:18.771 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.731+0000 7fbc35880640 1 --2- 172.21.15.42:0/1599494071 >> v2:172.21.15.154:6812/254845342 conn(0x7fbc1c037870 0x7fbc1c039d30 secure :-1 s=READY pgs=3801 cs=0 l=1 rev1=1 crypto rx=0x7fbc18005bd0 tx=0x7fbc28007c80 comp rx=0 tx=0).stop 2024-09-16T04:03:18.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.731+0000 7fbc35880640 10 monclient: shutdown 2024-09-16T04:03:18.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.731+0000 7fbc35880640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:03:18.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.731+0000 7fbc35880640 1 -- 172.21.15.42:0/1599494071 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbc3006f920 msgr2=0x7fbc301319d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:03:18.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.731+0000 7fbc35880640 1 --2- 172.21.15.42:0/1599494071 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbc3006f920 0x7fbc301319d0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fbc200ab040 tx=0x7fbc200192b0 comp rx=0 tx=0).stop 2024-09-16T04:03:18.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.731+0000 7fbc2f7fe640 1 -- 172.21.15.42:0/1599494071 reap_dead start 2024-09-16T04:03:18.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.731+0000 7fbc2f7fe640 1 -- 172.21.15.42:0/1599494071 reap_dead start 2024-09-16T04:03:18.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.731+0000 7fbc35880640 1 -- 172.21.15.42:0/1599494071 shutdown_connections 2024-09-16T04:03:18.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.731+0000 7fbc35880640 1 -- 172.21.15.42:0/1599494071 >> 172.21.15.42:0/1599494071 conn(0x7fbc300a3220 msgr2=0x7fbc300a5d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:03:18.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.731+0000 7fbc35880640 1 -- 172.21.15.42:0/1599494071 shutdown_connections 2024-09-16T04:03:18.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.731+0000 7fbc35880640 1 -- 172.21.15.42:0/1599494071 wait complete. 2024-09-16T04:03:18.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.731+0000 7fbc35880640 1 librados: shutdown 2024-09-16T04:03:18.773 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:03:18.773 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T04:03:18.773 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T04:03:18.889 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.883+0000 7fa1857be640 10 monclient: get_monmap_and_config 2024-09-16T04:03:18.889 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.883+0000 7fa1857be640 10 monclient: build_initial_monmap 2024-09-16T04:03:18.889 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.883+0000 7fa1857be640 10 monclient: monmap: 2024-09-16T04:03:18.889 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:03:18.889 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:03:18.889 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:03:18.887769+0000 2024-09-16T04:03:18.889 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:03:18.887769+0000 2024-09-16T04:03:18.889 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:03:18.890 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:03:18.890 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:03:18.890 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:03:18.890 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:03:18.890 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:03:18.890 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:03:18.890 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:03:18.892 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.887+0000 7fa1857be640 1 Processor -- start 2024-09-16T04:03:18.892 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.887+0000 7fa1857be640 1 -- start start 2024-09-16T04:03:18.893 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.887+0000 7fa1857be640 10 monclient: init 2024-09-16T04:03:18.893 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.887+0000 7fa1857be640 10 monclient: _reopen_session rank -1 2024-09-16T04:03:18.893 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.887+0000 7fa1857be640 10 monclient: _add_conns ranks=[2,4,3,1,0] 2024-09-16T04:03:18.893 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.887+0000 7fa1857be640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa1800a39b0 0x7fa180133b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:03:18.893 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.887+0000 7fa1857be640 10 monclient(hunting): picked mon.noname-b con 0x7fa1800a39b0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:03:18.893 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.887+0000 7fa1857be640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fa1800a3d90 0x7fa180137000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:03:18.893 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.887+0000 7fa1857be640 10 monclient(hunting): picked mon.noname-c con 0x7fa1800a3d90 addr v2:172.21.15.154:3300/0 2024-09-16T04:03:18.893 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.887+0000 7fa1857be640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fa180137540 0x7fa180137950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:03:18.893 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.887+0000 7fa1857be640 10 monclient(hunting): picked mon.noname-a con 0x7fa180137540 addr v2:172.21.15.42:3300/0 2024-09-16T04:03:18.893 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.887+0000 7fa1857be640 10 monclient(hunting): start opening mon connection 2024-09-16T04:03:18.893 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.887+0000 7fa1857be640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7fa180137e90 con 0x7fa180137540 2024-09-16T04:03:18.893 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.887+0000 7fa1857be640 10 monclient(hunting): start opening mon connection 2024-09-16T04:03:18.893 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.887+0000 7fa1857be640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fa18009c140 con 0x7fa1800a3d90 2024-09-16T04:03:18.894 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.887+0000 7fa1857be640 10 monclient(hunting): start opening mon connection 2024-09-16T04:03:18.894 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.887+0000 7fa1857be640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fa18009c420 con 0x7fa1800a39b0 2024-09-16T04:03:18.894 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.887+0000 7fa1857be640 10 monclient(hunting): _renew_subs 2024-09-16T04:03:18.894 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.887+0000 7fa1857be640 10 monclient(hunting): authenticate will time out at 2245.857138s 2024-09-16T04:03:18.894 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.887+0000 7fa17f7fe640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fa180137540 0x7fa180137950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:03:18.894 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.887+0000 7fa17effd640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa1800a39b0 0x7fa180133b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:03:18.894 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.887+0000 7fa17f7fe640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fa180137540 0x7fa180137950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:42396/0 (socket says 172.21.15.42:42396) 2024-09-16T04:03:18.894 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.887+0000 7fa17f7fe640 1 -- 172.21.15.42:0/1519305096 learned_addr learned my addr 172.21.15.42:0/1519305096 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:03:18.894 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.887+0000 7fa17effd640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa1800a39b0 0x7fa180133b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:52402/0 (socket says 172.21.15.42:52402) 2024-09-16T04:03:18.894 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.887+0000 7fa17f7fe640 10 monclient(hunting): get_auth_request con 0x7fa180137540 auth_method 0 2024-09-16T04:03:18.894 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.887+0000 7fa17f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:03:18.894 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.887+0000 7fa17f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:03:18.894 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.887+0000 7fa17f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:03:18.894 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.887+0000 7fa17effd640 10 monclient(hunting): get_auth_request con 0x7fa1800a39b0 auth_method 0 2024-09-16T04:03:18.895 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.887+0000 7fa17effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:03:18.895 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.887+0000 7fa17effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:03:18.895 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.887+0000 7fa17effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:03:18.895 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.887+0000 7fa17f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:03:18.895 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.887+0000 7fa17f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:03:18.895 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.887+0000 7fa17f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:03:18.895 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.887+0000 7fa17effd640 1 -- 172.21.15.42:0/1519305096 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa1800a39b0 msgr2=0x7fa180133b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:03:18.895 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.887+0000 7fa17effd640 1 -- 172.21.15.42:0/1519305096 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa1800a39b0 msgr2=0x7fa180133b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:03:18.895 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.887+0000 7fa17effd640 1 --2- 172.21.15.42:0/1519305096 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa1800a39b0 0x7fa180133b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:03:18.895 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.887+0000 7fa17e7fc640 1 --2- 172.21.15.42:0/1519305096 >> v2:172.21.15.154:3300/0 conn(0x7fa1800a3d90 0x7fa180137000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:03:18.895 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.887+0000 7fa17e7fc640 10 monclient(hunting): get_auth_request con 0x7fa1800a3d90 auth_method 0 2024-09-16T04:03:18.895 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.887+0000 7fa17e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:03:18.895 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.887+0000 7fa17e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:03:18.895 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.887+0000 7fa17e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:03:18.896 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.887+0000 7fa17effd640 1 --2- 172.21.15.42:0/1519305096 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa1800a39b0 0x7fa180133b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:03:18.896 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.887+0000 7fa17f7fe640 10 monclient(hunting): handle_auth_done global_id 825642 payload 274 2024-09-16T04:03:18.896 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.887+0000 7fa17f7fe640 1 -- 172.21.15.42:0/1519305096 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa1800a39b0 msgr2=0x7fa180133b80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:03:18.896 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.887+0000 7fa17f7fe640 1 --2- 172.21.15.42:0/1519305096 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa1800a39b0 0x7fa180133b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:03:18.896 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.887+0000 7fa17f7fe640 1 -- 172.21.15.42:0/1519305096 >> v2:172.21.15.154:3300/0 conn(0x7fa1800a3d90 msgr2=0x7fa180137000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:03:18.896 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.887+0000 7fa17f7fe640 1 --2- 172.21.15.42:0/1519305096 >> v2:172.21.15.154:3300/0 conn(0x7fa1800a3d90 0x7fa180137000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:03:18.896 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.887+0000 7fa17f7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:03:18.896 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.887+0000 7fa17f7fe640 1 monclient: found mon.noname-a 2024-09-16T04:03:18.896 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.887+0000 7fa17f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:03:18.896 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.887+0000 7fa17f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:03:18.896 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.887+0000 7fa17f7fe640 1 -- 172.21.15.42:0/1519305096 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa18006dd60 con 0x7fa180137540 2024-09-16T04:03:18.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.887+0000 7fa17f7fe640 1 --2- 172.21.15.42:0/1519305096 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa180137540 0x7fa180137950 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fa18006e3c0 tx=0x7fa17801c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec22ae3f01345f90 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:03:18.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.887+0000 7fa17dffb640 1 -- 172.21.15.42:0/1519305096 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1780a5530 con 0x7fa180137540 2024-09-16T04:03:18.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.891+0000 7fa17dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:03:18.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.891+0000 7fa17dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:03:18.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.891+0000 7fa17dffb640 10 monclient: dump: 2024-09-16T04:03:18.901 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:03:18.902 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:03:18.902 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:18.902 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:18.902 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:03:18.902 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:03:18.902 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:03:18.902 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:03:18.902 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:03:18.902 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:03:18.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.891+0000 7fa17dffb640 10 monclient: _finish_auth 0 2024-09-16T04:03:18.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.891+0000 7fa17dffb640 10 monclient: _check_auth_tickets 2024-09-16T04:03:18.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.891+0000 7fa17dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:03:18.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.891+0000 7fa17dffb640 1 -- 172.21.15.42:0/1519305096 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa1780a56d0 con 0x7fa180137540 2024-09-16T04:03:18.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.891+0000 7fa17dffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:03:18.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.891+0000 7fa17dffb640 1 -- 172.21.15.42:0/1519305096 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1780a5aa0 con 0x7fa180137540 2024-09-16T04:03:18.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.891+0000 7fa1857be640 5 monclient: authenticate success, global_id 825642 2024-09-16T04:03:18.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.891+0000 7fa1857be640 10 monclient: get_monmap_and_config success 2024-09-16T04:03:18.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.891+0000 7fa1857be640 4 set_mon_vals no callback set 2024-09-16T04:03:18.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.891+0000 7fa17dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:03:18.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.891+0000 7fa17dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:03:18.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.891+0000 7fa17dffb640 10 monclient: dump: 2024-09-16T04:03:18.903 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:03:18.903 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:03:18.903 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:18.903 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:18.903 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:03:18.903 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:03:18.903 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:03:18.904 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:03:18.904 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:03:18.904 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:03:18.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.891+0000 7fa1857be640 10 monclient: shutdown 2024-09-16T04:03:18.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.891+0000 7fa1857be640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:03:18.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.891+0000 7fa1857be640 1 -- 172.21.15.42:0/1519305096 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa180137540 msgr2=0x7fa180137950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:03:18.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.891+0000 7fa1857be640 1 --2- 172.21.15.42:0/1519305096 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa180137540 0x7fa180137950 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fa18006e3c0 tx=0x7fa17801c8b0 comp rx=0 tx=0).stop 2024-09-16T04:03:18.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.891+0000 7fa1857be640 1 -- 172.21.15.42:0/1519305096 shutdown_connections 2024-09-16T04:03:18.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.891+0000 7fa1857be640 1 --2- 172.21.15.42:0/1519305096 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa180137540 0x7fa180137950 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:03:18.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.891+0000 7fa1857be640 1 --2- 172.21.15.42:0/1519305096 >> v2:172.21.15.154:3300/0 conn(0x7fa1800a3d90 0x7fa180137000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:03:18.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.891+0000 7fa1857be640 1 --2- 172.21.15.42:0/1519305096 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa1800a39b0 0x7fa180133b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:03:18.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.891+0000 7fa1857be640 1 -- 172.21.15.42:0/1519305096 >> 172.21.15.42:0/1519305096 conn(0x7fa1800a9fb0 msgr2=0x7fa1800aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:03:18.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.891+0000 7fa1857be640 1 -- 172.21.15.42:0/1519305096 shutdown_connections 2024-09-16T04:03:18.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.891+0000 7fa1857be640 1 -- 172.21.15.42:0/1519305096 wait complete. 2024-09-16T04:03:18.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.891+0000 7fa1857be640 10 monclient: build_initial_monmap 2024-09-16T04:03:18.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.891+0000 7fa1857be640 10 monclient: monmap: 2024-09-16T04:03:18.905 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:03:18.905 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:03:18.905 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:03:18.905 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:03:18.905 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:03:18.905 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:03:18.905 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:03:18.905 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:03:18.905 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:03:18.905 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:03:18.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.891+0000 7fa1857be640 1 librados: starting msgr at 2024-09-16T04:03:18.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.891+0000 7fa1857be640 1 librados: starting objecter 2024-09-16T04:03:18.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.891+0000 7fa1857be640 1 Processor -- start 2024-09-16T04:03:18.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.891+0000 7fa1857be640 1 -- start start 2024-09-16T04:03:18.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.891+0000 7fa1857be640 1 librados: setting wanted keys 2024-09-16T04:03:18.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.891+0000 7fa1857be640 1 librados: calling monclient init 2024-09-16T04:03:18.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.891+0000 7fa1857be640 10 monclient: init 2024-09-16T04:03:18.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.891+0000 7fa1857be640 10 monclient: _reopen_session rank -1 2024-09-16T04:03:18.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.891+0000 7fa1857be640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:03:18.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.891+0000 7fa1857be640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa1800a39b0 0x7fa18013b320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:03:18.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.891+0000 7fa1857be640 10 monclient(hunting): picked mon.noname-c con 0x7fa1800a39b0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:03:18.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.891+0000 7fa1857be640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa1800a3d90 0x7fa18013b860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:03:18.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.891+0000 7fa1857be640 10 monclient(hunting): picked mon.noname-b con 0x7fa1800a3d90 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:03:18.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.891+0000 7fa1857be640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa180137540 0x7fa1801cf170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:03:18.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.891+0000 7fa1857be640 10 monclient(hunting): picked mon.noname-a con 0x7fa180137540 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:03:18.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.891+0000 7fa1857be640 10 monclient(hunting): start opening mon connection 2024-09-16T04:03:18.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.891+0000 7fa1857be640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fa180140dc0 con 0x7fa180137540 2024-09-16T04:03:18.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.891+0000 7fa1857be640 10 monclient(hunting): start opening mon connection 2024-09-16T04:03:18.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.891+0000 7fa1857be640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa180140f40 con 0x7fa1800a3d90 2024-09-16T04:03:18.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.891+0000 7fa1857be640 10 monclient(hunting): start opening mon connection 2024-09-16T04:03:18.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.891+0000 7fa1857be640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fa1801410c0 con 0x7fa1800a39b0 2024-09-16T04:03:18.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.891+0000 7fa1857be640 10 monclient(hunting): _renew_subs 2024-09-16T04:03:18.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.891+0000 7fa1857be640 10 monclient(hunting): authenticate will time out at 2245.862042s 2024-09-16T04:03:18.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.891+0000 7fa17f7fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa180137540 0x7fa1801cf170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:03:18.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.891+0000 7fa17effd640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa1800a39b0 0x7fa18013b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:03:18.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.891+0000 7fa17f7fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa180137540 0x7fa1801cf170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:42400/0 (socket says 172.21.15.42:42400) 2024-09-16T04:03:18.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.891+0000 7fa17f7fe640 1 -- 172.21.15.42:0/3123468107 learned_addr learned my addr 172.21.15.42:0/3123468107 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:03:18.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.891+0000 7fa17effd640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa1800a39b0 0x7fa18013b320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:52418/0 (socket says 172.21.15.42:52418) 2024-09-16T04:03:18.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.891+0000 7fa17f7fe640 10 monclient(hunting): get_auth_request con 0x7fa180137540 auth_method 0 2024-09-16T04:03:18.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.891+0000 7fa17f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:03:18.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.891+0000 7fa17f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:03:18.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.891+0000 7fa17f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:03:18.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.891+0000 7fa17effd640 10 monclient(hunting): get_auth_request con 0x7fa1800a39b0 auth_method 0 2024-09-16T04:03:18.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.891+0000 7fa17effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:03:18.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.891+0000 7fa17effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:03:18.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.891+0000 7fa17effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:03:18.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.891+0000 7fa17f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:03:18.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.891+0000 7fa17f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:03:18.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.891+0000 7fa17f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:03:18.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.891+0000 7fa17effd640 1 -- 172.21.15.42:0/3123468107 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa1800a39b0 msgr2=0x7fa18013b320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-16T04:03:18.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.891+0000 7fa17effd640 1 -- 172.21.15.42:0/3123468107 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa1800a39b0 msgr2=0x7fa18013b320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:03:18.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.891+0000 7fa17effd640 1 --2- 172.21.15.42:0/3123468107 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa1800a39b0 0x7fa18013b320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:03:18.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.891+0000 7fa17effd640 1 --2- 172.21.15.42:0/3123468107 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa1800a39b0 0x7fa18013b320 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:03:18.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.891+0000 7fa17e7fc640 1 --2- 172.21.15.42:0/3123468107 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa1800a3d90 0x7fa18013b860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:03:18.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.891+0000 7fa17e7fc640 10 monclient(hunting): get_auth_request con 0x7fa1800a3d90 auth_method 0 2024-09-16T04:03:18.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.891+0000 7fa17e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:03:18.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.891+0000 7fa17e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:03:18.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.891+0000 7fa17e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:03:18.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.891+0000 7fa17f7fe640 10 monclient(hunting): handle_auth_done global_id 825645 payload 824 2024-09-16T04:03:18.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.891+0000 7fa17f7fe640 1 -- 172.21.15.42:0/3123468107 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa1800a39b0 msgr2=0x7fa18013b320 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:03:18.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.891+0000 7fa17f7fe640 1 --2- 172.21.15.42:0/3123468107 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa1800a39b0 0x7fa18013b320 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:03:18.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.891+0000 7fa17f7fe640 1 -- 172.21.15.42:0/3123468107 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa1800a3d90 msgr2=0x7fa18013b860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:03:18.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.891+0000 7fa17f7fe640 1 --2- 172.21.15.42:0/3123468107 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa1800a3d90 0x7fa18013b860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:03:18.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.891+0000 7fa17f7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:03:18.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.891+0000 7fa17f7fe640 1 monclient: found mon.noname-a 2024-09-16T04:03:18.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.891+0000 7fa17f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:03:18.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.891+0000 7fa17f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:03:18.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.891+0000 7fa17f7fe640 1 -- 172.21.15.42:0/3123468107 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa18013d6d0 con 0x7fa180137540 2024-09-16T04:03:18.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.891+0000 7fa17f7fe640 1 --2- 172.21.15.42:0/3123468107 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa180137540 0x7fa1801cf170 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7fa1780af040 tx=0x7fa178019650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:03:18.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.891+0000 7fa1677fe640 10 client.?.objecter ms_handle_connect 0x7fa180137540 2024-09-16T04:03:18.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.891+0000 7fa1677fe640 10 client.?.objecter resend_mon_ops 2024-09-16T04:03:18.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.891+0000 7fa1677fe640 1 -- 172.21.15.42:0/3123468107 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1780bc020 con 0x7fa180137540 2024-09-16T04:03:18.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.891+0000 7fa1677fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:03:18.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.891+0000 7fa1677fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:03:18.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.891+0000 7fa1677fe640 10 monclient: dump: 2024-09-16T04:03:18.910 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:03:18.910 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:03:18.911 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:18.911 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:18.911 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:03:18.911 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:03:18.911 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:03:18.911 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:03:18.911 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:03:18.911 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:03:18.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.891+0000 7fa1677fe640 10 monclient: _finish_auth 0 2024-09-16T04:03:18.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.891+0000 7fa1677fe640 10 monclient: _check_auth_tickets 2024-09-16T04:03:18.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.891+0000 7fa1677fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:03:18.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.891+0000 7fa1857be640 5 monclient: authenticate success, global_id 825645 2024-09-16T04:03:18.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.891+0000 7fa1677fe640 1 -- 172.21.15.42:0/3123468107 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa1780b8670 con 0x7fa180137540 2024-09-16T04:03:18.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.891+0000 7fa1677fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:03:18.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.891+0000 7fa1677fe640 1 -- 172.21.15.42:0/3123468107 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa178009510 con 0x7fa180137540 2024-09-16T04:03:18.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.891+0000 7fa1677fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:03:18.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.891+0000 7fa1677fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:03:18.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.891+0000 7fa1677fe640 10 monclient: dump: 2024-09-16T04:03:18.912 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:03:18.912 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:03:18.912 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:18.912 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:18.912 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:03:18.912 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:03:18.912 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:03:18.912 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:03:18.912 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:03:18.912 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:03:18.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.891+0000 7fa17d7fa640 4 set_mon_vals no callback set 2024-09-16T04:03:18.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.891+0000 7fa1857be640 10 monclient: _renew_subs 2024-09-16T04:03:18.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.891+0000 7fa1857be640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:03:18.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.891+0000 7fa1857be640 1 -- 172.21.15.42:0/3123468107 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa18009d2a0 con 0x7fa180137540 2024-09-16T04:03:18.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.891+0000 7fa1857be640 10 client.825645.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:03:18.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.891+0000 7fa1857be640 10 monclient: _renew_subs 2024-09-16T04:03:18.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.891+0000 7fa1857be640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:03:18.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.891+0000 7fa1857be640 1 -- 172.21.15.42:0/3123468107 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa18009cfc0 con 0x7fa180137540 2024-09-16T04:03:18.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.891+0000 7fa1857be640 1 librados: init done 2024-09-16T04:03:18.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.891+0000 7fa1857be640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:03:18.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.891+0000 7fa1857be640 1 --2- 172.21.15.42:0/3123468107 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa1801cf6b0 0x7fa1801cfa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:03:18.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.891+0000 7fa1857be640 10 monclient(hunting): start opening mon connection 2024-09-16T04:03:18.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.891+0000 7fa1857be640 1 -- 172.21.15.42:0/3123468107 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fa18006f710 con 0x7fa1801cf6b0 2024-09-16T04:03:18.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.891+0000 7fa17effd640 1 --2- 172.21.15.42:0/3123468107 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa1801cf6b0 0x7fa1801cfa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:03:18.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.891+0000 7fa17effd640 10 monclient: get_auth_request con 0x7fa1801cf6b0 auth_method 0 2024-09-16T04:03:18.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.891+0000 7fa17effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:03:18.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.891+0000 7fa17effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:03:18.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.891+0000 7fa17effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:03:18.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.895+0000 7fa17effd640 1 -- 172.21.15.42:0/3123468107 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa1801cf6b0 msgr2=0x7fa1801cfa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-16T04:03:18.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.895+0000 7fa17effd640 1 -- 172.21.15.42:0/3123468107 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa1801cf6b0 msgr2=0x7fa1801cfa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:03:18.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.895+0000 7fa17effd640 1 --2- 172.21.15.42:0/3123468107 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa1801cf6b0 0x7fa1801cfa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:03:18.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.895+0000 7fa17effd640 1 --2- 172.21.15.42:0/3123468107 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa1801cf6b0 0x7fa1801cfa90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:03:18.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.895+0000 7fa1677fe640 1 -- 172.21.15.42:0/3123468107 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fa1780bf020 con 0x7fa180137540 2024-09-16T04:03:18.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.895+0000 7fa1677fe640 1 --2- 172.21.15.42:0/3123468107 >> v2:172.21.15.154:6812/254845342 conn(0x7fa16c037810 0x7fa16c039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:03:18.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.895+0000 7fa1677fe640 1 -- 172.21.15.42:0/3123468107 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(650..650 src has 451..650) ==== 6306+0+0 (secure 0 0 0) 0x7fa1780e4210 con 0x7fa180137540 2024-09-16T04:03:18.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.895+0000 7fa1677fe640 10 client.825645.objecter ms_dispatch 0x7fa180000ba0 osd_map(650..650 src has 451..650) 2024-09-16T04:03:18.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.895+0000 7fa1677fe640 3 client.825645.objecter handle_osd_map got epochs [650,650] > 0 2024-09-16T04:03:18.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.895+0000 7fa1677fe640 3 client.825645.objecter handle_osd_map decoding full epoch 650 2024-09-16T04:03:18.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.895+0000 7fa1677fe640 20 client.825645.objecter dump_active .. 0 homeless 2024-09-16T04:03:18.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.895+0000 7fa17e7fc640 1 --2- 172.21.15.42:0/3123468107 >> v2:172.21.15.154:6812/254845342 conn(0x7fa16c037810 0x7fa16c039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:03:18.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.895+0000 7fa17e7fc640 10 monclient: get_auth_request con 0x7fa16c037810 auth_method 0 2024-09-16T04:03:18.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.895+0000 7fa1677fe640 10 client.825645.objecter ms_handle_connect 0x7fa16c037810 2024-09-16T04:03:18.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:18.895+0000 7fa17e7fc640 1 --2- 172.21.15.42:0/3123468107 >> v2:172.21.15.154:6812/254845342 conn(0x7fa16c037810 0x7fa16c039cd0 secure :-1 s=READY pgs=3802 cs=0 l=1 rev1=1 crypto rx=0x7fa1800a3860 tx=0x7fa174008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:03:19.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:19.099+0000 7fa17effd640 1 --2- 172.21.15.42:0/3123468107 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa1801cf6b0 0x7fa1801cfa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:03:19.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:19.099+0000 7fa17effd640 10 monclient: get_auth_request con 0x7fa1801cf6b0 auth_method 0 2024-09-16T04:03:19.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:19.099+0000 7fa17effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:03:19.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:19.099+0000 7fa17effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:03:19.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:19.099+0000 7fa17effd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:03:19.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:19.099+0000 7fa17effd640 1 -- 172.21.15.42:0/3123468107 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa1801cf6b0 msgr2=0x7fa1801cfa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-16T04:03:19.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:19.099+0000 7fa17effd640 1 -- 172.21.15.42:0/3123468107 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa1801cf6b0 msgr2=0x7fa1801cfa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:03:19.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:19.099+0000 7fa17effd640 1 --2- 172.21.15.42:0/3123468107 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa1801cf6b0 0x7fa1801cfa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:03:19.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:19.099+0000 7fa17effd640 1 --2- 172.21.15.42:0/3123468107 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa1801cf6b0 0x7fa1801cfa90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T04:03:19.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:19.503+0000 7fa17effd640 1 --2- 172.21.15.42:0/3123468107 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa1801cf6b0 0x7fa1801cfa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:03:19.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:19.503+0000 7fa17effd640 10 monclient: get_auth_request con 0x7fa1801cf6b0 auth_method 0 2024-09-16T04:03:19.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:19.503+0000 7fa17effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:03:19.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:19.503+0000 7fa17effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:03:19.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:19.503+0000 7fa17effd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:03:19.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:19.503+0000 7fa17effd640 1 -- 172.21.15.42:0/3123468107 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa1801cf6b0 msgr2=0x7fa1801cfa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-16T04:03:19.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:19.503+0000 7fa17effd640 1 -- 172.21.15.42:0/3123468107 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa1801cf6b0 msgr2=0x7fa1801cfa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:03:19.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:19.503+0000 7fa17effd640 1 --2- 172.21.15.42:0/3123468107 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa1801cf6b0 0x7fa1801cfa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:03:19.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:19.503+0000 7fa17effd640 1 --2- 172.21.15.42:0/3123468107 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa1801cf6b0 0x7fa1801cfa90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T04:03:20.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:20.307+0000 7fa17effd640 1 --2- 172.21.15.42:0/3123468107 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa1801cf6b0 0x7fa1801cfa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:03:20.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:20.307+0000 7fa17effd640 10 monclient: get_auth_request con 0x7fa1801cf6b0 auth_method 0 2024-09-16T04:03:20.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:20.307+0000 7fa17effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:03:20.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:20.307+0000 7fa17effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:03:20.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:20.307+0000 7fa17effd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:03:20.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:20.307+0000 7fa17effd640 1 -- 172.21.15.42:0/3123468107 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa1801cf6b0 msgr2=0x7fa1801cfa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-16T04:03:20.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:20.307+0000 7fa17effd640 1 -- 172.21.15.42:0/3123468107 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa1801cf6b0 msgr2=0x7fa1801cfa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:03:20.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:20.307+0000 7fa17effd640 1 --2- 172.21.15.42:0/3123468107 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa1801cf6b0 0x7fa1801cfa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:03:20.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:20.307+0000 7fa17effd640 1 --2- 172.21.15.42:0/3123468107 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa1801cf6b0 0x7fa1801cfa90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T04:03:21.896 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:21.891+0000 7fa1667fc640 10 monclient: tick 2024-09-16T04:03:21.896 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:21.891+0000 7fa1667fc640 10 monclient: _check_auth_tickets 2024-09-16T04:03:21.896 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:21.891+0000 7fa1667fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:03:21.896 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:21.891+0000 7fa1667fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:03:21.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:21.911+0000 7fa17effd640 1 --2- 172.21.15.42:0/3123468107 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa1801cf6b0 0x7fa1801cfa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:03:21.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:21.911+0000 7fa17effd640 10 monclient: get_auth_request con 0x7fa1801cf6b0 auth_method 0 2024-09-16T04:03:21.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:21.911+0000 7fa17effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:03:21.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:21.911+0000 7fa17effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:03:21.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:21.911+0000 7fa17effd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:03:21.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:21.911+0000 7fa17effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:03:21.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:21.911+0000 7fa17effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:03:21.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:21.911+0000 7fa17effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:03:21.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:21.911+0000 7fa17effd640 10 monclient(hunting): handle_auth_done global_id 974099 payload 824 2024-09-16T04:03:21.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:21.911+0000 7fa17effd640 1 -- 172.21.15.42:0/3123468107 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa18013cbe0 con 0x7fa1801cf6b0 2024-09-16T04:03:21.916 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:21.911+0000 7fa17effd640 1 --2- 172.21.15.42:0/3123468107 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa1801cf6b0 0x7fa1801cfa90 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fa170000ee0 tx=0x7fa170018e20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:03:21.916 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:21.911+0000 7fa1677fe640 10 client.825645.objecter ms_handle_connect 0x7fa1801cf6b0 2024-09-16T04:03:21.916 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:21.911+0000 7fa1677fe640 10 client.825645.objecter resend_mon_ops 2024-09-16T04:03:21.916 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:21.911+0000 7fa1677fe640 1 -- 172.21.15.42:0/3123468107 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa17000f040 con 0x7fa1801cf6b0 2024-09-16T04:03:21.916 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:21.911+0000 7fa1677fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:03:21.916 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:21.911+0000 7fa1677fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:03:21.916 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:21.911+0000 7fa1677fe640 10 monclient: dump: 2024-09-16T04:03:21.916 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:03:21.916 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:03:21.916 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:21.916 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:21.917 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:03:21.917 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:03:21.917 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:03:21.917 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:03:21.917 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:03:21.917 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:03:21.917 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:21.911+0000 7fa1677fe640 1 -- 172.21.15.42:0/3123468107 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fa170014d50 con 0x7fa1801cf6b0 2024-09-16T04:03:21.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:21.911+0000 7fa1677fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:03:21.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:21.911+0000 7fa1677fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:03:21.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:21.911+0000 7fa1677fe640 1 -- 172.21.15.42:0/3123468107 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa1801cf6b0 msgr2=0x7fa1801cfa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:03:21.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:21.911+0000 7fa1677fe640 1 --2- 172.21.15.42:0/3123468107 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa1801cf6b0 0x7fa1801cfa90 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fa170000ee0 tx=0x7fa170018e20 comp rx=0 tx=0).stop 2024-09-16T04:03:21.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:21.911+0000 7fa1677fe640 1 -- 172.21.15.42:0/3123468107 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa1801cf6b0 msgr2=0x7fa1801cfa90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:03:21.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:21.911+0000 7fa1677fe640 1 --2- 172.21.15.42:0/3123468107 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa1801cf6b0 0x7fa1801cfa90 secure :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fa170000ee0 tx=0x7fa170018e20 comp rx=0 tx=0).stop 2024-09-16T04:03:21.923 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:21.919+0000 7fa1857be640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:03:21.924 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:21.919+0000 7fa1857be640 1 --2- 172.21.15.42:0/3123468107 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa180135bc0 0x7fa180135fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:03:21.924 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:21.919+0000 7fa1857be640 10 monclient(hunting): start opening mon connection 2024-09-16T04:03:21.924 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:21.919+0000 7fa1857be640 1 -- 172.21.15.42:0/3123468107 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fa180132c90 con 0x7fa180135bc0 2024-09-16T04:03:21.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:21.919+0000 7fa17effd640 1 --2- 172.21.15.42:0/3123468107 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa180135bc0 0x7fa180135fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:03:21.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:21.919+0000 7fa17effd640 10 monclient: get_auth_request con 0x7fa180135bc0 auth_method 0 2024-09-16T04:03:21.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:21.919+0000 7fa17effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:03:21.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:21.919+0000 7fa17effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:03:21.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:21.919+0000 7fa17effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:03:21.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:21.919+0000 7fa17effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:03:21.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:21.919+0000 7fa17effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:03:21.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:21.919+0000 7fa17effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:03:21.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:21.919+0000 7fa17effd640 10 monclient(hunting): handle_auth_done global_id 974102 payload 824 2024-09-16T04:03:21.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:21.919+0000 7fa17effd640 1 -- 172.21.15.42:0/3123468107 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fa180137000 con 0x7fa180135bc0 2024-09-16T04:03:21.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:21.923+0000 7fa1677fe640 10 client.825645.objecter ms_handle_connect 0x7fa180135bc0 2024-09-16T04:03:21.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:21.923+0000 7fa1677fe640 10 client.825645.objecter resend_mon_ops 2024-09-16T04:03:21.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:21.923+0000 7fa17effd640 1 --2- 172.21.15.42:0/3123468107 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa180135bc0 0x7fa180135fd0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fa170000ee0 tx=0x7fa1700046d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:03:21.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:21.923+0000 7fa1677fe640 1 -- 172.21.15.42:0/3123468107 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1700272c0 con 0x7fa180135bc0 2024-09-16T04:03:21.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:21.923+0000 7fa1677fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:03:21.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:21.923+0000 7fa1677fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:03:21.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:21.923+0000 7fa1677fe640 10 monclient: dump: 2024-09-16T04:03:21.927 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:03:21.927 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:03:21.927 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:21.928 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:21.928 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:03:21.928 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:03:21.928 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:03:21.928 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:03:21.928 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:03:21.928 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:03:21.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:21.923+0000 7fa1677fe640 1 -- 172.21.15.42:0/3123468107 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4712 (secure 0 0 0) 0x7fa170026050 con 0x7fa180135bc0 2024-09-16T04:03:21.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:21.923+0000 7fa1677fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:03:21.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:21.923+0000 7fa1677fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:03:21.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:21.923+0000 7fa1677fe640 1 -- 172.21.15.42:0/3123468107 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa180135bc0 msgr2=0x7fa180135fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:03:21.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:21.923+0000 7fa1677fe640 1 --2- 172.21.15.42:0/3123468107 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa180135bc0 0x7fa180135fd0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fa170000ee0 tx=0x7fa1700046d0 comp rx=0 tx=0).stop 2024-09-16T04:03:21.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:21.923+0000 7fa1677fe640 1 -- 172.21.15.42:0/3123468107 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa180135bc0 msgr2=0x7fa180135fd0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:03:21.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:21.923+0000 7fa1677fe640 1 --2- 172.21.15.42:0/3123468107 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa180135bc0 0x7fa180135fd0 secure :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fa170000ee0 tx=0x7fa1700046d0 comp rx=0 tx=0).stop 2024-09-16T04:03:21.929 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T04:03:21.929 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:03:21.929 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:03:21.929 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T04:03:21.929 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 190, 2024-09-16T04:03:21.929 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 10423, 2024-09-16T04:03:21.929 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T04:03:21.929 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T04:03:21.929 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T04:03:21.930 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T04:03:21.930 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:03:21.930 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 4, 2024-09-16T04:03:21.930 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:03:21.930 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T04:03:21.930 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T04:03:21.930 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:03:21.930 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:03:21.930 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:03:21.930 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:03:21.930 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:03:21.930 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:03:21.930 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:03:21.930 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:03:21.931 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:03:21.931 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:03:21.931 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:03:21.931 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:03:21.931 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:03:21.931 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T04:03:21.931 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:03:21.931 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:03:21.931 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:03:21.931 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:03:21.931 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:03:21.931 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:03:21.931 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:03:21.931 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:03:21.932 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:03:21.932 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:03:21.932 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:03:21.932 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:03:21.932 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:03:21.932 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T04:03:21.932 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T04:03:21.932 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T04:03:21.932 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T04:03:21.932 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T04:03:21.932 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T04:03:21.932 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:03:21.932 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:03:21.932 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T04:03:21.932 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T04:03:21.933 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T04:03:21.933 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T04:03:21.933 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T04:03:21.933 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T04:03:21.933 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T04:03:21.933 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:03:21.933 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T04:03:21.933 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:03:21.933 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:03:21.933 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:03:21.933 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:03:21.933 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:03:21.933 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:03:21.933 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:03:21.934 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:03:21.934 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:03:21.934 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:03:21.934 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:03:21.934 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:03:21.934 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T04:03:21.934 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:03:21.935 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T04:03:21.935 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:03:21.935 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:03:21.935 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:03:21.935 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:03:21.935 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:03:21.935 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:03:21.935 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:03:21.935 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T04:03:21.935 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:03:21.935 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:03:21.935 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:03:21.935 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:03:21.935 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T04:03:21.936 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:03:21.936 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:03:21.936 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:03:21.936 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:03:21.936 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T04:03:21.936 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T04:03:21.936 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:03:21.936 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:03:21.936 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:03:21.936 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:03:21.936 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:03:21.936 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:03:21.936 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:03:21.937 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:03:21.937 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:03:21.937 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:03:21.937 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:03:21.937 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T04:03:21.937 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:03:21.937 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:03:21.937 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:03:21.937 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:03:21.937 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T04:03:21.937 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:03:21.938 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:03:21.938 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:03:21.938 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:03:21.938 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T04:03:21.938 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T04:03:21.938 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:03:21.938 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:03:21.938 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:03:21.938 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:03:21.938 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:03:21.938 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:03:21.938 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:03:21.938 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:03:21.939 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:03:21.939 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:03:21.939 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:03:21.939 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T04:03:21.939 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:03:21.939 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:03:21.939 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:03:21.939 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:03:21.939 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T04:03:21.939 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:03:21.939 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:03:21.939 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:03:21.939 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:03:21.940 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T04:03:21.940 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T04:03:21.940 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:03:21.940 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:03:21.940 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:03:21.940 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:03:21.940 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:03:21.940 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:03:21.940 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T04:03:21.940 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T04:03:21.940 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:03:21.940 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:03:21.940 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:03:21.940 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:03:21.940 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:03:21.941 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:03:21.941 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T04:03:21.941 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:03:21.941 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:03:21.941 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:03:21.941 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:03:21.941 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:03:21.941 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:03:21.941 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T04:03:21.941 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:03:21.941 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:03:21.941 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:03:21.941 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:03:21.942 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:03:21.942 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:03:21.942 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:03:21.942 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T04:03:21.942 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T04:03:21.943 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:21.923+0000 7fa1857be640 10 librados: watch_flush enter 2024-09-16T04:03:21.943 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:21.923+0000 7fa1857be640 10 librados: watch_flush exit 2024-09-16T04:03:21.943 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:21.923+0000 7fa1857be640 20 client.825645.objecter shutdown clearing up homeless session... 2024-09-16T04:03:21.943 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:21.923+0000 7fa1857be640 10 client.825645.objecter successfully canceled tick 2024-09-16T04:03:21.943 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:21.923+0000 7fa1857be640 1 -- 172.21.15.42:0/3123468107 >> v2:172.21.15.154:6812/254845342 conn(0x7fa16c037810 msgr2=0x7fa16c039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:03:21.943 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:21.923+0000 7fa1857be640 1 --2- 172.21.15.42:0/3123468107 >> v2:172.21.15.154:6812/254845342 conn(0x7fa16c037810 0x7fa16c039cd0 secure :-1 s=READY pgs=3802 cs=0 l=1 rev1=1 crypto rx=0x7fa1800a3860 tx=0x7fa174008040 comp rx=0 tx=0).stop 2024-09-16T04:03:21.944 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:21.923+0000 7fa1857be640 10 monclient: shutdown 2024-09-16T04:03:21.944 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:21.923+0000 7fa1857be640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:03:21.944 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:21.923+0000 7fa1857be640 1 -- 172.21.15.42:0/3123468107 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa180137540 msgr2=0x7fa1801cf170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:03:21.944 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:21.923+0000 7fa1857be640 1 --2- 172.21.15.42:0/3123468107 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa180137540 0x7fa1801cf170 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7fa1780af040 tx=0x7fa178019650 comp rx=0 tx=0).stop 2024-09-16T04:03:21.944 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:21.923+0000 7fa17f7fe640 1 -- 172.21.15.42:0/3123468107 reap_dead start 2024-09-16T04:03:21.944 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:21.923+0000 7fa17f7fe640 1 -- 172.21.15.42:0/3123468107 reap_dead start 2024-09-16T04:03:21.944 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:21.923+0000 7fa1857be640 1 -- 172.21.15.42:0/3123468107 shutdown_connections 2024-09-16T04:03:21.944 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:21.923+0000 7fa1857be640 1 -- 172.21.15.42:0/3123468107 >> 172.21.15.42:0/3123468107 conn(0x7fa1800a9fb0 msgr2=0x7fa1800a8070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:03:21.944 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:21.923+0000 7fa1857be640 1 -- 172.21.15.42:0/3123468107 shutdown_connections 2024-09-16T04:03:21.945 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:21.923+0000 7fa1857be640 1 -- 172.21.15.42:0/3123468107 wait complete. 2024-09-16T04:03:21.945 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:21.923+0000 7fa1857be640 1 librados: shutdown 2024-09-16T04:03:21.953 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:03:21.953 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T04:03:21.954 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T04:03:22.085 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.079+0000 7f922361e640 10 monclient: get_monmap_and_config 2024-09-16T04:03:22.085 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.079+0000 7f922361e640 10 monclient: build_initial_monmap 2024-09-16T04:03:22.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.083+0000 7f922361e640 10 monclient: monmap: 2024-09-16T04:03:22.086 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:03:22.086 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:03:22.086 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:03:22.084125+0000 2024-09-16T04:03:22.086 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:03:22.084125+0000 2024-09-16T04:03:22.086 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:03:22.086 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:03:22.086 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:03:22.086 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:03:22.086 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:03:22.087 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:03:22.087 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:03:22.087 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:03:22.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.083+0000 7f922361e640 1 Processor -- start 2024-09-16T04:03:22.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.083+0000 7f922361e640 1 -- start start 2024-09-16T04:03:22.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.083+0000 7f922361e640 10 monclient: init 2024-09-16T04:03:22.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.083+0000 7f922361e640 10 monclient: _reopen_session rank -1 2024-09-16T04:03:22.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.083+0000 7f922361e640 10 monclient: _add_conns ranks=[2,1,3,0,4] 2024-09-16T04:03:22.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.083+0000 7f922361e640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f921c0a39b0 0x7f921c133b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:03:22.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.083+0000 7f922361e640 10 monclient(hunting): picked mon.noname-b con 0x7f921c0a39b0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:03:22.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.083+0000 7f922361e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f921c0a3d90 addr v1:172.21.15.154:6789/0 2024-09-16T04:03:22.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.083+0000 7f922361e640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f921c137710 0x7f921c13f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:03:22.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.083+0000 7f922361e640 10 monclient(hunting): picked mon.noname-a con 0x7f921c137710 addr v2:172.21.15.42:3300/0 2024-09-16T04:03:22.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.083+0000 7f922361e640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f921c09c140 con 0x7f921c0a3d90 2024-09-16T04:03:22.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.083+0000 7f922361e640 10 monclient(hunting): start opening mon connection 2024-09-16T04:03:22.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.083+0000 7f922361e640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f921c09c420 con 0x7f921c137710 2024-09-16T04:03:22.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.083+0000 7f922361e640 10 monclient(hunting): start opening mon connection 2024-09-16T04:03:22.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.083+0000 7f922361e640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f921c06dd60 con 0x7f921c0a39b0 2024-09-16T04:03:22.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.083+0000 7f922361e640 10 monclient(hunting): _renew_subs 2024-09-16T04:03:22.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.083+0000 7f922361e640 10 monclient(hunting): authenticate will time out at 2249.053713s 2024-09-16T04:03:22.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.083+0000 7f9221bbb640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f921c137710 0x7f921c13f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:03:22.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.083+0000 7f92213ba640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f921c0a39b0 0x7f921c133b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:03:22.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.083+0000 7f9221bbb640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f921c137710 0x7f921c13f3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:42416/0 (socket says 172.21.15.42:42416) 2024-09-16T04:03:22.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.083+0000 7f92213ba640 10 monclient(hunting): get_auth_request con 0x7f921c0a39b0 auth_method 0 2024-09-16T04:03:22.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.083+0000 7f92213ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:03:22.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.083+0000 7f9221bbb640 1 -- 172.21.15.42:0/3492987072 learned_addr learned my addr 172.21.15.42:0/3492987072 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:03:22.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.083+0000 7f92213ba640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:03:22.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.083+0000 7f92213ba640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:03:22.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.083+0000 7f9221bbb640 10 monclient(hunting): get_auth_request con 0x7f921c137710 auth_method 0 2024-09-16T04:03:22.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.083+0000 7f9221bbb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:03:22.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.083+0000 7f9221bbb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:03:22.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.083+0000 7f9221bbb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:03:22.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.083+0000 7f9221bbb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:03:22.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.083+0000 7f9221bbb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:03:22.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.083+0000 7f9221bbb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:03:22.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.083+0000 7f92213ba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:03:22.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.083+0000 7f92213ba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:03:22.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.083+0000 7f92213ba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:03:22.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.083+0000 7f9221bbb640 10 monclient(hunting): handle_auth_done global_id 825648 payload 274 2024-09-16T04:03:22.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.083+0000 7f9221bbb640 1 -- 172.21.15.42:0/3492987072 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f921c0a39b0 msgr2=0x7f921c133b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:03:22.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.083+0000 7f9221bbb640 1 --2- 172.21.15.42:0/3492987072 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f921c0a39b0 0x7f921c133b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:03:22.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.083+0000 7f9221bbb640 1 -- 172.21.15.42:0/3492987072 >> v1:172.21.15.154:6789/0 conn(0x7f921c0a3d90 legacy=0x7f921c137000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:03:22.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.083+0000 7f9221bbb640 10 monclient: _finish_hunting 0 2024-09-16T04:03:22.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.083+0000 7f9221bbb640 1 monclient: found mon.noname-a 2024-09-16T04:03:22.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.083+0000 7f9221bbb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:03:22.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.083+0000 7f9221bbb640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:03:22.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.083+0000 7f9221bbb640 1 -- 172.21.15.42:0/3492987072 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f921c0a5e20 con 0x7f921c137710 2024-09-16T04:03:22.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.083+0000 7f92213ba640 1 --2- 172.21.15.42:0/3492987072 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f921c0a39b0 0x7f921c133b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:03:22.099 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.087+0000 7f9221bbb640 1 --2- 172.21.15.42:0/3492987072 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f921c137710 0x7f921c13f3a0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f9214009e50 tx=0x7f921401c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f39c603ff0785e85 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:03:22.099 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.087+0000 7f9213fff640 1 -- 172.21.15.42:0/3492987072 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92140a5530 con 0x7f921c137710 2024-09-16T04:03:22.099 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.087+0000 7f9213fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:03:22.099 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.087+0000 7f9213fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:03:22.099 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.087+0000 7f9213fff640 10 monclient: dump: 2024-09-16T04:03:22.099 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:03:22.099 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:03:22.099 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:22.099 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:22.100 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:03:22.100 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:03:22.100 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:03:22.100 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:03:22.101 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:03:22.101 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:03:22.101 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.087+0000 7f9213fff640 10 monclient: _finish_auth 0 2024-09-16T04:03:22.101 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.087+0000 7f9213fff640 10 monclient: _check_auth_tickets 2024-09-16T04:03:22.101 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.087+0000 7f9213fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:03:22.101 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.087+0000 7f9213fff640 1 -- 172.21.15.42:0/3492987072 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f92140a56d0 con 0x7f921c137710 2024-09-16T04:03:22.101 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.087+0000 7f9213fff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:03:22.101 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.087+0000 7f9213fff640 1 -- 172.21.15.42:0/3492987072 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92140a5aa0 con 0x7f921c137710 2024-09-16T04:03:22.101 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.087+0000 7f922361e640 5 monclient: authenticate success, global_id 825648 2024-09-16T04:03:22.101 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.087+0000 7f922361e640 10 monclient: get_monmap_and_config success 2024-09-16T04:03:22.101 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.087+0000 7f922361e640 4 set_mon_vals no callback set 2024-09-16T04:03:22.101 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.087+0000 7f9213fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:03:22.101 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.087+0000 7f9213fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:03:22.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.087+0000 7f9213fff640 10 monclient: dump: 2024-09-16T04:03:22.102 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:03:22.102 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:03:22.102 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:22.102 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:22.102 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:03:22.102 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:03:22.102 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:03:22.102 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:03:22.102 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:03:22.102 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:03:22.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.087+0000 7f922361e640 10 monclient: shutdown 2024-09-16T04:03:22.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.087+0000 7f922361e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:03:22.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.087+0000 7f922361e640 1 -- 172.21.15.42:0/3492987072 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f921c137710 msgr2=0x7f921c13f3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:03:22.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.087+0000 7f922361e640 1 --2- 172.21.15.42:0/3492987072 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f921c137710 0x7f921c13f3a0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f9214009e50 tx=0x7f921401c8b0 comp rx=0 tx=0).stop 2024-09-16T04:03:22.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.087+0000 7f922361e640 1 -- 172.21.15.42:0/3492987072 shutdown_connections 2024-09-16T04:03:22.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.087+0000 7f922361e640 1 --2- 172.21.15.42:0/3492987072 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f921c137710 0x7f921c13f3a0 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:03:22.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.087+0000 7f922361e640 1 --2- 172.21.15.42:0/3492987072 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f921c0a39b0 0x7f921c133b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:03:22.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.087+0000 7f922361e640 1 -- 172.21.15.42:0/3492987072 >> 172.21.15.42:0/3492987072 conn(0x7f921c0a9fb0 msgr2=0x7f921c0aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:03:22.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.087+0000 7f922361e640 1 -- 172.21.15.42:0/3492987072 shutdown_connections 2024-09-16T04:03:22.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.087+0000 7f922361e640 1 -- 172.21.15.42:0/3492987072 wait complete. 2024-09-16T04:03:22.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.087+0000 7f922361e640 10 monclient: build_initial_monmap 2024-09-16T04:03:22.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.087+0000 7f922361e640 10 monclient: monmap: 2024-09-16T04:03:22.103 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:03:22.103 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:03:22.103 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:03:22.104 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:03:22.104 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:03:22.104 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:03:22.104 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:03:22.104 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:03:22.104 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:03:22.104 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:03:22.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.087+0000 7f922361e640 1 librados: starting msgr at 2024-09-16T04:03:22.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.087+0000 7f922361e640 1 librados: starting objecter 2024-09-16T04:03:22.105 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.087+0000 7f922361e640 1 Processor -- start 2024-09-16T04:03:22.105 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.087+0000 7f922361e640 1 -- start start 2024-09-16T04:03:22.105 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.087+0000 7f922361e640 1 librados: setting wanted keys 2024-09-16T04:03:22.105 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.087+0000 7f922361e640 1 librados: calling monclient init 2024-09-16T04:03:22.105 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.087+0000 7f922361e640 10 monclient: init 2024-09-16T04:03:22.105 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.087+0000 7f922361e640 10 monclient: _reopen_session rank -1 2024-09-16T04:03:22.105 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.087+0000 7f922361e640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:03:22.105 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.087+0000 7f922361e640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f921c0a39b0 0x7f921c12d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:03:22.105 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.087+0000 7f922361e640 10 monclient(hunting): picked mon.noname-a con 0x7f921c0a39b0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:03:22.105 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.087+0000 7f922361e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f921c0a3d90 0x7f921c12dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:03:22.106 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.087+0000 7f922361e640 10 monclient(hunting): picked mon.noname-b con 0x7f921c0a3d90 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:03:22.106 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.087+0000 7f922361e640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f921c137710 0x7f921c1cf890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:03:22.106 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.087+0000 7f922361e640 10 monclient(hunting): picked mon.noname-c con 0x7f921c137710 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:03:22.106 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.087+0000 7f922361e640 10 monclient(hunting): start opening mon connection 2024-09-16T04:03:22.106 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.087+0000 7f922361e640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f921c142310 con 0x7f921c0a39b0 2024-09-16T04:03:22.106 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.087+0000 7f922361e640 10 monclient(hunting): start opening mon connection 2024-09-16T04:03:22.106 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.087+0000 7f922361e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f921c142490 con 0x7f921c0a3d90 2024-09-16T04:03:22.106 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.087+0000 7f922361e640 10 monclient(hunting): start opening mon connection 2024-09-16T04:03:22.106 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.087+0000 7f922361e640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f921c0a0e50 con 0x7f921c137710 2024-09-16T04:03:22.106 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.087+0000 7f922361e640 10 monclient(hunting): _renew_subs 2024-09-16T04:03:22.106 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.087+0000 7f922361e640 10 monclient(hunting): authenticate will time out at 2249.058791s 2024-09-16T04:03:22.107 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.087+0000 7f92213ba640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f921c0a39b0 0x7f921c12d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:03:22.107 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.087+0000 7f9221bbb640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f921c137710 0x7f921c1cf890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:03:22.107 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.087+0000 7f92213ba640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f921c0a39b0 0x7f921c12d690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:42430/0 (socket says 172.21.15.42:42430) 2024-09-16T04:03:22.107 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.087+0000 7f92213ba640 1 -- 172.21.15.42:0/3122238470 learned_addr learned my addr 172.21.15.42:0/3122238470 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:03:22.107 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.087+0000 7f92213ba640 10 monclient(hunting): get_auth_request con 0x7f921c0a39b0 auth_method 0 2024-09-16T04:03:22.107 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.087+0000 7f92213ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:03:22.107 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.087+0000 7f92213ba640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:03:22.107 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.087+0000 7f92213ba640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:03:22.107 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.087+0000 7f9221bbb640 10 monclient(hunting): get_auth_request con 0x7f921c137710 auth_method 0 2024-09-16T04:03:22.107 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.087+0000 7f9221bbb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:03:22.107 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.087+0000 7f9221bbb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:03:22.108 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.087+0000 7f9221bbb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:03:22.108 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.087+0000 7f9221bbb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:03:22.108 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.087+0000 7f9221bbb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:03:22.108 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.087+0000 7f9221bbb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:03:22.108 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.087+0000 7f92213ba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:03:22.108 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.087+0000 7f92213ba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:03:22.108 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.087+0000 7f92213ba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:03:22.108 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.087+0000 7f9220bb9640 1 --2- 172.21.15.42:0/3122238470 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f921c0a3d90 0x7f921c12dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:03:22.108 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.087+0000 7f9220bb9640 10 monclient(hunting): get_auth_request con 0x7f921c0a3d90 auth_method 0 2024-09-16T04:03:22.108 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.087+0000 7f9220bb9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:03:22.108 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.087+0000 7f9220bb9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:03:22.108 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.087+0000 7f9220bb9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:03:22.109 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.087+0000 7f9221bbb640 10 monclient(hunting): handle_auth_done global_id 974108 payload 824 2024-09-16T04:03:22.109 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.087+0000 7f9221bbb640 1 -- 172.21.15.42:0/3122238470 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f921c0a3d90 msgr2=0x7f921c12dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:03:22.109 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.087+0000 7f9221bbb640 1 --2- 172.21.15.42:0/3122238470 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f921c0a3d90 0x7f921c12dbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:03:22.109 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.087+0000 7f9221bbb640 1 -- 172.21.15.42:0/3122238470 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f921c0a39b0 msgr2=0x7f921c12d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:03:22.109 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.087+0000 7f9221bbb640 1 --2- 172.21.15.42:0/3122238470 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f921c0a39b0 0x7f921c12d690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:03:22.109 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.087+0000 7f9221bbb640 10 monclient: _finish_hunting 0 2024-09-16T04:03:22.109 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.087+0000 7f9221bbb640 1 monclient: found mon.noname-c 2024-09-16T04:03:22.109 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.087+0000 7f9221bbb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:03:22.109 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.087+0000 7f9221bbb640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T04:03:22.109 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.087+0000 7f9221bbb640 1 -- 172.21.15.42:0/3122238470 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f921c12e110 con 0x7f921c137710 2024-09-16T04:03:22.109 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.087+0000 7f92213ba640 1 --2- 172.21.15.42:0/3122238470 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f921c0a39b0 0x7f921c12d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:03:22.109 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.087+0000 7f9221bbb640 1 --2- 172.21.15.42:0/3122238470 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f921c137710 0x7f921c1cf890 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f921c0766a0 tx=0x7f9214019650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:03:22.109 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.087+0000 7f9211ffb640 10 client.?.objecter ms_handle_connect 0x7f921c137710 2024-09-16T04:03:22.110 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.087+0000 7f9211ffb640 10 client.?.objecter resend_mon_ops 2024-09-16T04:03:22.110 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.087+0000 7f9211ffb640 1 -- 172.21.15.42:0/3122238470 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92140a5530 con 0x7f921c137710 2024-09-16T04:03:22.110 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.087+0000 7f9211ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:03:22.110 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.087+0000 7f9211ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:03:22.110 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.087+0000 7f9211ffb640 10 monclient: dump: 2024-09-16T04:03:22.110 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:03:22.110 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:03:22.110 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:22.110 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:22.110 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:03:22.110 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:03:22.110 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:03:22.111 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:03:22.111 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:03:22.111 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:03:22.111 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.087+0000 7f9211ffb640 10 monclient: _finish_auth 0 2024-09-16T04:03:22.111 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.087+0000 7f9211ffb640 10 monclient: _check_auth_tickets 2024-09-16T04:03:22.111 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.087+0000 7f9211ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:03:22.111 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.087+0000 7f922361e640 5 monclient: authenticate success, global_id 974108 2024-09-16T04:03:22.111 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.087+0000 7f9211ffb640 1 -- 172.21.15.42:0/3122238470 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f92140b6690 con 0x7f921c137710 2024-09-16T04:03:22.111 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.087+0000 7f922361e640 10 monclient: _renew_subs 2024-09-16T04:03:22.111 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.087+0000 7f922361e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:03:22.111 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.087+0000 7f922361e640 1 -- 172.21.15.42:0/3122238470 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f921c09eb80 con 0x7f921c137710 2024-09-16T04:03:22.111 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.087+0000 7f9211ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:03:22.111 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.087+0000 7f9211ffb640 1 -- 172.21.15.42:0/3122238470 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9214009510 con 0x7f921c137710 2024-09-16T04:03:22.111 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.087+0000 7f9211ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:03:22.112 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.087+0000 7f9211ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:03:22.112 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.087+0000 7f9211ffb640 10 monclient: dump: 2024-09-16T04:03:22.112 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:03:22.112 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:03:22.112 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:22.112 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:22.112 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:03:22.112 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:03:22.112 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:03:22.112 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:03:22.113 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:03:22.113 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:03:22.113 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.087+0000 7f92137fe640 4 set_mon_vals no callback set 2024-09-16T04:03:22.113 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.087+0000 7f922361e640 10 client.974108.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:03:22.113 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.087+0000 7f922361e640 10 monclient: _renew_subs 2024-09-16T04:03:22.113 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.087+0000 7f922361e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:03:22.113 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.087+0000 7f922361e640 1 -- 172.21.15.42:0/3122238470 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f921c0a2300 con 0x7f921c137710 2024-09-16T04:03:22.113 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.087+0000 7f922361e640 1 librados: init done 2024-09-16T04:03:22.113 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.091+0000 7f922361e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:03:22.113 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.091+0000 7f922361e640 1 --2- 172.21.15.42:0/3122238470 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f921c1cfdd0 0x7f921c1d01b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:03:22.113 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.091+0000 7f922361e640 10 monclient(hunting): start opening mon connection 2024-09-16T04:03:22.113 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.091+0000 7f922361e640 1 -- 172.21.15.42:0/3122238470 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f921c1342d0 con 0x7f921c1cfdd0 2024-09-16T04:03:22.113 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.091+0000 7f9211ffb640 1 -- 172.21.15.42:0/3122238470 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f92140be020 con 0x7f921c137710 2024-09-16T04:03:22.113 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.091+0000 7f92213ba640 1 --2- 172.21.15.42:0/3122238470 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f921c1cfdd0 0x7f921c1d01b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:03:22.114 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.091+0000 7f92213ba640 10 monclient: get_auth_request con 0x7f921c1cfdd0 auth_method 0 2024-09-16T04:03:22.114 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.091+0000 7f92213ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:03:22.114 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.091+0000 7f92213ba640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:03:22.114 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.091+0000 7f92213ba640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:03:22.114 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.091+0000 7f9211ffb640 1 --2- 172.21.15.42:0/3122238470 >> v2:172.21.15.154:6812/254845342 conn(0x7f9200037810 0x7f9200039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:03:22.114 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.091+0000 7f92213ba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:03:22.114 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.091+0000 7f92213ba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:03:22.114 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.091+0000 7f92213ba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:03:22.114 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.091+0000 7f9211ffb640 1 -- 172.21.15.42:0/3122238470 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(652..652 src has 451..652) ==== 5899+0+0 (secure 0 0 0) 0x7f92140e3320 con 0x7f921c137710 2024-09-16T04:03:22.114 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.091+0000 7f9211ffb640 10 client.974108.objecter ms_dispatch 0x7f921c000ba0 osd_map(652..652 src has 451..652) 2024-09-16T04:03:22.114 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.091+0000 7f9211ffb640 3 client.974108.objecter handle_osd_map got epochs [652,652] > 0 2024-09-16T04:03:22.114 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.091+0000 7f9211ffb640 3 client.974108.objecter handle_osd_map decoding full epoch 652 2024-09-16T04:03:22.114 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.091+0000 7f9211ffb640 20 client.974108.objecter dump_active .. 0 homeless 2024-09-16T04:03:22.115 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.091+0000 7f9220bb9640 1 --2- 172.21.15.42:0/3122238470 >> v2:172.21.15.154:6812/254845342 conn(0x7f9200037810 0x7f9200039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:03:22.115 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.091+0000 7f9220bb9640 10 monclient: get_auth_request con 0x7f9200037810 auth_method 0 2024-09-16T04:03:22.115 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.091+0000 7f92213ba640 10 monclient(hunting): handle_auth_done global_id 904711 payload 824 2024-09-16T04:03:22.115 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.091+0000 7f92213ba640 1 -- 172.21.15.42:0/3122238470 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f921c1d06f0 con 0x7f921c1cfdd0 2024-09-16T04:03:22.115 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.091+0000 7f9211ffb640 10 client.974108.objecter ms_handle_connect 0x7f921c1cfdd0 2024-09-16T04:03:22.115 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.091+0000 7f9211ffb640 10 client.974108.objecter resend_mon_ops 2024-09-16T04:03:22.115 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.091+0000 7f92213ba640 1 --2- 172.21.15.42:0/3122238470 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f921c1cfdd0 0x7f921c1d01b0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f9218011d80 tx=0x7f921800f610 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:03:22.115 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.091+0000 7f9211ffb640 1 -- 172.21.15.42:0/3122238470 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9218019070 con 0x7f921c1cfdd0 2024-09-16T04:03:22.115 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.091+0000 7f9211ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:03:22.115 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.091+0000 7f9211ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:03:22.115 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.091+0000 7f9211ffb640 10 monclient: dump: 2024-09-16T04:03:22.115 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:03:22.115 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:03:22.115 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:22.115 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:22.116 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:03:22.116 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:03:22.116 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:03:22.116 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:03:22.116 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:03:22.116 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:03:22.116 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.091+0000 7f9211ffb640 10 client.974108.objecter ms_handle_connect 0x7f9200037810 2024-09-16T04:03:22.116 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.091+0000 7f9220bb9640 1 --2- 172.21.15.42:0/3122238470 >> v2:172.21.15.154:6812/254845342 conn(0x7f9200037810 0x7f9200039cd0 secure :-1 s=READY pgs=3803 cs=0 l=1 rev1=1 crypto rx=0x7f920c004e30 tx=0x7f920c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:03:22.116 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.091+0000 7f9211ffb640 1 -- 172.21.15.42:0/3122238470 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f9218004030 con 0x7f921c1cfdd0 2024-09-16T04:03:22.116 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.091+0000 7f9211ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:03:22.116 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.091+0000 7f9211ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:03:22.116 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.091+0000 7f9211ffb640 1 -- 172.21.15.42:0/3122238470 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f921c1cfdd0 msgr2=0x7f921c1d01b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:03:22.116 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.091+0000 7f9211ffb640 1 --2- 172.21.15.42:0/3122238470 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f921c1cfdd0 0x7f921c1d01b0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f9218011d80 tx=0x7f921800f610 comp rx=0 tx=0).stop 2024-09-16T04:03:22.116 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.091+0000 7f9211ffb640 1 -- 172.21.15.42:0/3122238470 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f921c1cfdd0 msgr2=0x7f921c1d01b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:03:22.117 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.091+0000 7f9211ffb640 1 --2- 172.21.15.42:0/3122238470 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f921c1cfdd0 0x7f921c1d01b0 secure :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f9218011d80 tx=0x7f921800f610 comp rx=0 tx=0).stop 2024-09-16T04:03:22.118 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.099+0000 7f922361e640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:03:22.118 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.099+0000 7f922361e640 1 --2- 172.21.15.42:0/3122238470 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f921c0a2540 0x7f921c12f500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:03:22.118 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.099+0000 7f922361e640 10 monclient(hunting): start opening mon connection 2024-09-16T04:03:22.118 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.099+0000 7f922361e640 1 -- 172.21.15.42:0/3122238470 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f921c06df10 con 0x7f921c0a2540 2024-09-16T04:03:22.118 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.099+0000 7f92213ba640 1 --2- 172.21.15.42:0/3122238470 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f921c0a2540 0x7f921c12f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:03:22.118 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.099+0000 7f92213ba640 10 monclient: get_auth_request con 0x7f921c0a2540 auth_method 0 2024-09-16T04:03:22.119 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.099+0000 7f92213ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:03:22.119 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.099+0000 7f92213ba640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:03:22.119 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.099+0000 7f92213ba640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:03:22.119 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.099+0000 7f92213ba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:03:22.119 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.099+0000 7f92213ba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:03:22.119 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.099+0000 7f92213ba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:03:22.119 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.099+0000 7f92213ba640 10 monclient(hunting): handle_auth_done global_id 904714 payload 824 2024-09-16T04:03:22.119 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.099+0000 7f92213ba640 1 -- 172.21.15.42:0/3122238470 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f921c133b80 con 0x7f921c0a2540 2024-09-16T04:03:22.119 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.099+0000 7f92213ba640 1 --2- 172.21.15.42:0/3122238470 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f921c0a2540 0x7f921c12f500 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f9218011d80 tx=0x7f92180046d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:03:22.119 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.099+0000 7f9211ffb640 10 client.974108.objecter ms_handle_connect 0x7f921c0a2540 2024-09-16T04:03:22.119 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.099+0000 7f9211ffb640 10 client.974108.objecter resend_mon_ops 2024-09-16T04:03:22.119 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.099+0000 7f9211ffb640 1 -- 172.21.15.42:0/3122238470 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9218019040 con 0x7f921c0a2540 2024-09-16T04:03:22.119 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.099+0000 7f9211ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:03:22.119 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.099+0000 7f9211ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:03:22.120 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.099+0000 7f9211ffb640 10 monclient: dump: 2024-09-16T04:03:22.120 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:03:22.120 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:03:22.120 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:22.120 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:22.120 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:03:22.120 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:03:22.120 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:03:22.120 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:03:22.120 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:03:22.120 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:03:22.120 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.099+0000 7f9211ffb640 1 -- 172.21.15.42:0/3122238470 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4713 (secure 0 0 0) 0x7f921801e660 con 0x7f921c0a2540 2024-09-16T04:03:22.120 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.099+0000 7f9211ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:03:22.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.099+0000 7f9211ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:03:22.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.099+0000 7f9211ffb640 1 -- 172.21.15.42:0/3122238470 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f921c0a2540 msgr2=0x7f921c12f500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:03:22.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.099+0000 7f9211ffb640 1 --2- 172.21.15.42:0/3122238470 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f921c0a2540 0x7f921c12f500 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f9218011d80 tx=0x7f92180046d0 comp rx=0 tx=0).stop 2024-09-16T04:03:22.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.099+0000 7f9211ffb640 1 -- 172.21.15.42:0/3122238470 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f921c0a2540 msgr2=0x7f921c12f500 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:03:22.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.099+0000 7f9211ffb640 1 --2- 172.21.15.42:0/3122238470 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f921c0a2540 0x7f921c12f500 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:03:22.121 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T04:03:22.121 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:03:22.121 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:03:22.121 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T04:03:22.121 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 190, 2024-09-16T04:03:22.122 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 330280, 2024-09-16T04:03:22.122 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T04:03:22.122 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T04:03:22.122 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T04:03:22.122 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T04:03:22.122 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:03:22.122 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 5, 2024-09-16T04:03:22.122 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:03:22.122 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T04:03:22.122 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T04:03:22.122 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:03:22.122 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:03:22.122 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:03:22.122 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:03:22.123 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:03:22.123 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:03:22.123 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:03:22.123 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:03:22.123 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:03:22.123 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:03:22.123 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:03:22.123 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:03:22.123 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:03:22.123 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T04:03:22.123 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:03:22.123 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:03:22.123 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:03:22.124 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:03:22.124 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:03:22.124 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:03:22.124 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:03:22.124 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:03:22.124 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:03:22.124 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:03:22.124 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:03:22.124 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:03:22.124 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:03:22.124 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T04:03:22.124 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T04:03:22.124 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T04:03:22.125 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T04:03:22.125 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T04:03:22.125 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T04:03:22.125 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:03:22.125 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:03:22.125 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T04:03:22.125 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T04:03:22.125 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T04:03:22.125 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T04:03:22.125 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T04:03:22.125 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T04:03:22.125 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T04:03:22.125 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:03:22.125 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T04:03:22.126 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:03:22.126 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:03:22.126 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:03:22.126 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:03:22.126 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:03:22.126 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:03:22.126 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:03:22.126 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:03:22.126 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:03:22.126 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:03:22.126 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:03:22.126 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:03:22.126 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T04:03:22.127 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:03:22.127 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T04:03:22.127 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:03:22.127 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:03:22.127 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:03:22.127 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:03:22.127 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:03:22.127 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:03:22.127 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:03:22.127 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T04:03:22.127 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:03:22.127 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:03:22.127 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:03:22.127 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:03:22.128 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T04:03:22.128 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:03:22.128 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:03:22.128 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:03:22.128 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:03:22.128 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T04:03:22.128 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T04:03:22.128 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:03:22.128 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:03:22.128 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:03:22.128 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:03:22.128 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:03:22.128 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:03:22.129 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:03:22.129 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:03:22.129 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:03:22.129 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:03:22.129 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:03:22.129 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T04:03:22.129 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:03:22.129 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:03:22.129 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:03:22.129 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:03:22.129 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T04:03:22.129 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:03:22.129 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:03:22.130 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:03:22.130 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:03:22.130 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T04:03:22.130 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T04:03:22.130 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:03:22.130 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:03:22.130 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:03:22.130 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:03:22.130 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:03:22.130 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:03:22.130 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:03:22.130 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:03:22.130 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:03:22.131 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:03:22.131 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:03:22.131 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T04:03:22.131 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:03:22.131 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:03:22.131 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:03:22.131 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:03:22.131 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T04:03:22.131 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:03:22.131 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:03:22.131 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:03:22.131 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:03:22.131 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T04:03:22.131 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T04:03:22.132 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:03:22.132 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:03:22.132 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:03:22.132 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:03:22.132 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:03:22.132 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:03:22.132 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T04:03:22.132 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T04:03:22.132 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:03:22.132 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:03:22.132 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:03:22.132 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:03:22.133 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:03:22.133 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:03:22.133 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T04:03:22.133 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:03:22.133 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:03:22.133 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:03:22.133 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:03:22.133 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:03:22.133 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:03:22.133 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T04:03:22.133 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:03:22.133 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:03:22.133 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:03:22.133 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 2 2024-09-16T04:03:22.134 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:03:22.134 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:03:22.134 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:03:22.134 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T04:03:22.134 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T04:03:22.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.099+0000 7f922361e640 10 librados: watch_flush enter 2024-09-16T04:03:22.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.099+0000 7f922361e640 10 librados: watch_flush exit 2024-09-16T04:03:22.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.099+0000 7f922361e640 20 client.974108.objecter shutdown clearing up homeless session... 2024-09-16T04:03:22.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.099+0000 7f922361e640 10 client.974108.objecter successfully canceled tick 2024-09-16T04:03:22.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.103+0000 7f922361e640 1 -- 172.21.15.42:0/3122238470 >> v2:172.21.15.154:6812/254845342 conn(0x7f9200037810 msgr2=0x7f9200039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:03:22.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.103+0000 7f922361e640 1 --2- 172.21.15.42:0/3122238470 >> v2:172.21.15.154:6812/254845342 conn(0x7f9200037810 0x7f9200039cd0 secure :-1 s=READY pgs=3803 cs=0 l=1 rev1=1 crypto rx=0x7f920c004e30 tx=0x7f920c009040 comp rx=0 tx=0).stop 2024-09-16T04:03:22.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.103+0000 7f922361e640 10 monclient: shutdown 2024-09-16T04:03:22.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.103+0000 7f922361e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:03:22.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.103+0000 7f922361e640 1 -- 172.21.15.42:0/3122238470 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f921c137710 msgr2=0x7f921c1cf890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:03:22.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.103+0000 7f922361e640 1 --2- 172.21.15.42:0/3122238470 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f921c137710 0x7f921c1cf890 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f921c0766a0 tx=0x7f9214019650 comp rx=0 tx=0).stop 2024-09-16T04:03:22.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.103+0000 7f9221bbb640 1 -- 172.21.15.42:0/3122238470 reap_dead start 2024-09-16T04:03:22.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.103+0000 7f9221bbb640 1 -- 172.21.15.42:0/3122238470 reap_dead start 2024-09-16T04:03:22.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.103+0000 7f922361e640 1 -- 172.21.15.42:0/3122238470 shutdown_connections 2024-09-16T04:03:22.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.103+0000 7f922361e640 1 -- 172.21.15.42:0/3122238470 >> 172.21.15.42:0/3122238470 conn(0x7f921c0a9fb0 msgr2=0x7f921c0a8a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:03:22.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.103+0000 7f922361e640 1 -- 172.21.15.42:0/3122238470 shutdown_connections 2024-09-16T04:03:22.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.103+0000 7f922361e640 1 -- 172.21.15.42:0/3122238470 wait complete. 2024-09-16T04:03:22.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.103+0000 7f922361e640 1 librados: shutdown 2024-09-16T04:03:22.137 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:03:22.138 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T04:03:22.138 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T04:03:22.259 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.255+0000 7fd9ea5a0640 10 monclient: get_monmap_and_config 2024-09-16T04:03:22.259 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.255+0000 7fd9ea5a0640 10 monclient: build_initial_monmap 2024-09-16T04:03:22.260 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.255+0000 7fd9ea5a0640 10 monclient: monmap: 2024-09-16T04:03:22.260 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:03:22.260 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:03:22.260 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:03:22.258208+0000 2024-09-16T04:03:22.260 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:03:22.258208+0000 2024-09-16T04:03:22.260 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:03:22.260 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:03:22.260 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:03:22.260 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:03:22.260 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:03:22.260 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:03:22.260 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:03:22.261 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:03:22.263 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.255+0000 7fd9ea5a0640 1 Processor -- start 2024-09-16T04:03:22.263 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.255+0000 7fd9ea5a0640 1 -- start start 2024-09-16T04:03:22.263 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.255+0000 7fd9ea5a0640 10 monclient: init 2024-09-16T04:03:22.264 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.255+0000 7fd9ea5a0640 10 monclient: _reopen_session rank -1 2024-09-16T04:03:22.264 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.255+0000 7fd9ea5a0640 10 monclient: _add_conns ranks=[0,3,4,2,1] 2024-09-16T04:03:22.264 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.255+0000 7fd9ea5a0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd9e40a1330 addr v1:172.21.15.42:6789/0 2024-09-16T04:03:22.264 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.255+0000 7fd9ea5a0640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd9e406dac0 0x7fd9e413a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:03:22.264 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.255+0000 7fd9ea5a0640 10 monclient(hunting): picked mon.noname-a con 0x7fd9e406dac0 addr v2:172.21.15.42:3300/0 2024-09-16T04:03:22.264 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.255+0000 7fd9ea5a0640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fd9e40a1e20 0x7fd9e413c890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:03:22.264 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.255+0000 7fd9ea5a0640 10 monclient(hunting): picked mon.noname-c con 0x7fd9e40a1e20 addr v2:172.21.15.154:3300/0 2024-09-16T04:03:22.264 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.255+0000 7fd9ea5a0640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd9e409c090 con 0x7fd9e40a1330 2024-09-16T04:03:22.264 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.255+0000 7fd9ea5a0640 10 monclient(hunting): start opening mon connection 2024-09-16T04:03:22.264 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.255+0000 7fd9ea5a0640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7fd9e409f610 con 0x7fd9e406dac0 2024-09-16T04:03:22.264 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.255+0000 7fd9ea5a0640 10 monclient(hunting): start opening mon connection 2024-09-16T04:03:22.264 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.255+0000 7fd9ea5a0640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fd9e409f8f0 con 0x7fd9e40a1e20 2024-09-16T04:03:22.264 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.255+0000 7fd9ea5a0640 10 monclient(hunting): _renew_subs 2024-09-16T04:03:22.265 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.255+0000 7fd9ea5a0640 10 monclient(hunting): authenticate will time out at 2249.227974s 2024-09-16T04:03:22.265 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.259+0000 7fd9dbfff640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd9e406dac0 0x7fd9e413a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:03:22.265 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.259+0000 7fd9dbfff640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd9e406dac0 0x7fd9e413a340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:42446/0 (socket says 172.21.15.42:42446) 2024-09-16T04:03:22.265 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.259+0000 7fd9dbfff640 1 -- 172.21.15.42:0/4071478541 learned_addr learned my addr 172.21.15.42:0/4071478541 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:03:22.265 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.259+0000 7fd9dbfff640 10 monclient(hunting): get_auth_request con 0x7fd9e406dac0 auth_method 0 2024-09-16T04:03:22.265 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.259+0000 7fd9dbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:03:22.265 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.259+0000 7fd9dbfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:03:22.265 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.259+0000 7fd9dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:03:22.265 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.259+0000 7fd9e3fff640 1 --1- 172.21.15.42:0/4071478541 >> v1:172.21.15.42:6789/0 conn(0x7fd9e40a1330 0x7fd9e40a1710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:43012/0 (socket says 172.21.15.42:43012) 2024-09-16T04:03:22.265 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.259+0000 7fd9dbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:03:22.265 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.259+0000 7fd9dbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:03:22.265 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.259+0000 7fd9e8b3d640 1 --2- 172.21.15.42:0/4071478541 >> v2:172.21.15.154:3300/0 conn(0x7fd9e40a1e20 0x7fd9e413c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:03:22.266 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.259+0000 7fd9dbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:03:22.266 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.259+0000 7fd9e8b3d640 10 monclient(hunting): get_auth_request con 0x7fd9e40a1e20 auth_method 0 2024-09-16T04:03:22.266 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.259+0000 7fd9e8b3d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:03:22.266 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.259+0000 7fd9e8b3d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:03:22.266 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.259+0000 7fd9e8b3d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:03:22.266 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.259+0000 7fd9dbfff640 10 monclient(hunting): handle_auth_done global_id 825654 payload 274 2024-09-16T04:03:22.266 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.259+0000 7fd9dbfff640 1 -- 172.21.15.42:0/4071478541 >> v2:172.21.15.154:3300/0 conn(0x7fd9e40a1e20 msgr2=0x7fd9e413c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:03:22.266 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.259+0000 7fd9dbfff640 1 --2- 172.21.15.42:0/4071478541 >> v2:172.21.15.154:3300/0 conn(0x7fd9e40a1e20 0x7fd9e413c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:03:22.266 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.259+0000 7fd9dbfff640 1 -- 172.21.15.42:0/4071478541 >> v1:172.21.15.42:6789/0 conn(0x7fd9e40a1330 legacy=0x7fd9e40a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:03:22.266 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.259+0000 7fd9dbfff640 10 monclient: _finish_hunting 0 2024-09-16T04:03:22.266 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.259+0000 7fd9dbfff640 1 monclient: found mon.noname-a 2024-09-16T04:03:22.266 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.259+0000 7fd9dbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:03:22.266 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.259+0000 7fd9dbfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:03:22.266 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.259+0000 7fd9dbfff640 1 -- 172.21.15.42:0/4071478541 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9e409fbd0 con 0x7fd9e406dac0 2024-09-16T04:03:22.267 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.259+0000 7fd9e37fe640 1 -- 172.21.15.42:0/4071478541 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2737527331 0 0) 0x7fd9e409c090 con 0x7fd9e40a1330 2024-09-16T04:03:22.267 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.259+0000 7fd9e37fe640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T04:03:22.267 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.259+0000 7fd9dbfff640 1 --2- 172.21.15.42:0/4071478541 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd9e406dac0 0x7fd9e413a340 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7fd9e409c800 tx=0x7fd9cc01c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e7cc01106598f05 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:03:22.271 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.259+0000 7fd9e37fe640 1 -- 172.21.15.42:0/4071478541 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9cc0b0070 con 0x7fd9e406dac0 2024-09-16T04:03:22.271 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.259+0000 7fd9e37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:03:22.272 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.259+0000 7fd9e37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:03:22.272 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.259+0000 7fd9e37fe640 10 monclient: dump: 2024-09-16T04:03:22.272 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:03:22.272 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:03:22.272 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:22.272 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:22.272 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:03:22.272 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:03:22.272 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:03:22.272 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:03:22.272 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:03:22.272 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:03:22.273 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.259+0000 7fd9e37fe640 10 monclient: _finish_auth 0 2024-09-16T04:03:22.273 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.259+0000 7fd9e37fe640 10 monclient: _check_auth_tickets 2024-09-16T04:03:22.273 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.259+0000 7fd9e37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:03:22.273 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.259+0000 7fd9e37fe640 1 -- 172.21.15.42:0/4071478541 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd9cc0ab600 con 0x7fd9e406dac0 2024-09-16T04:03:22.273 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.259+0000 7fd9e37fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:03:22.273 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.259+0000 7fd9e37fe640 1 -- 172.21.15.42:0/4071478541 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9cc0ab9d0 con 0x7fd9e406dac0 2024-09-16T04:03:22.273 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.259+0000 7fd9ea5a0640 5 monclient: authenticate success, global_id 825654 2024-09-16T04:03:22.273 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.259+0000 7fd9ea5a0640 10 monclient: get_monmap_and_config success 2024-09-16T04:03:22.273 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.259+0000 7fd9ea5a0640 4 set_mon_vals no callback set 2024-09-16T04:03:22.273 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.259+0000 7fd9e37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:03:22.273 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.259+0000 7fd9e37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:03:22.273 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.259+0000 7fd9e37fe640 10 monclient: dump: 2024-09-16T04:03:22.273 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:03:22.273 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:03:22.274 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:22.274 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:22.274 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:03:22.274 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:03:22.274 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:03:22.274 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:03:22.274 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:03:22.274 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:03:22.274 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.259+0000 7fd9ea5a0640 10 monclient: shutdown 2024-09-16T04:03:22.274 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.259+0000 7fd9ea5a0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:03:22.274 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.259+0000 7fd9ea5a0640 1 -- 172.21.15.42:0/4071478541 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd9e406dac0 msgr2=0x7fd9e413a340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:03:22.274 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.259+0000 7fd9ea5a0640 1 --2- 172.21.15.42:0/4071478541 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd9e406dac0 0x7fd9e413a340 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7fd9e409c800 tx=0x7fd9cc01c470 comp rx=0 tx=0).stop 2024-09-16T04:03:22.275 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.259+0000 7fd9ea5a0640 1 -- 172.21.15.42:0/4071478541 shutdown_connections 2024-09-16T04:03:22.275 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.259+0000 7fd9ea5a0640 1 --2- 172.21.15.42:0/4071478541 >> v2:172.21.15.154:3300/0 conn(0x7fd9e40a1e20 0x7fd9e413c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:03:22.275 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.259+0000 7fd9ea5a0640 1 --2- 172.21.15.42:0/4071478541 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd9e406dac0 0x7fd9e413a340 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:03:22.275 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.259+0000 7fd9ea5a0640 1 -- 172.21.15.42:0/4071478541 >> 172.21.15.42:0/4071478541 conn(0x7fd9e40a2420 msgr2=0x7fd9e40a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:03:22.275 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.259+0000 7fd9ea5a0640 1 -- 172.21.15.42:0/4071478541 shutdown_connections 2024-09-16T04:03:22.275 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.259+0000 7fd9ea5a0640 1 -- 172.21.15.42:0/4071478541 wait complete. 2024-09-16T04:03:22.275 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.263+0000 7fd9ea5a0640 10 monclient: build_initial_monmap 2024-09-16T04:03:22.275 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.263+0000 7fd9ea5a0640 10 monclient: monmap: 2024-09-16T04:03:22.275 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:03:22.275 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:03:22.275 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:03:22.275 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:03:22.275 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:03:22.275 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:03:22.275 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:03:22.276 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:03:22.276 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:03:22.276 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:03:22.276 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.263+0000 7fd9ea5a0640 1 librados: starting msgr at 2024-09-16T04:03:22.276 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.263+0000 7fd9ea5a0640 1 librados: starting objecter 2024-09-16T04:03:22.276 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.263+0000 7fd9ea5a0640 1 Processor -- start 2024-09-16T04:03:22.276 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.263+0000 7fd9ea5a0640 1 -- start start 2024-09-16T04:03:22.276 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.263+0000 7fd9ea5a0640 1 librados: setting wanted keys 2024-09-16T04:03:22.276 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.263+0000 7fd9ea5a0640 1 librados: calling monclient init 2024-09-16T04:03:22.276 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.263+0000 7fd9ea5a0640 10 monclient: init 2024-09-16T04:03:22.276 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.263+0000 7fd9ea5a0640 10 monclient: _reopen_session rank -1 2024-09-16T04:03:22.276 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.263+0000 7fd9ea5a0640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:03:22.276 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.263+0000 7fd9ea5a0640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd9e40a1330 0x7fd9e41ceba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:03:22.276 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.263+0000 7fd9ea5a0640 10 monclient(hunting): picked mon.noname-c con 0x7fd9e40a1330 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:03:22.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.263+0000 7fd9ea5a0640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd9e40a1e20 0x7fd9e4135260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:03:22.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.263+0000 7fd9ea5a0640 10 monclient(hunting): picked mon.noname-a con 0x7fd9e40a1e20 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:03:22.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.263+0000 7fd9ea5a0640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd9e41357a0 0x7fd9e411dc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:03:22.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.263+0000 7fd9ea5a0640 10 monclient(hunting): picked mon.noname-b con 0x7fd9e41357a0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:03:22.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.263+0000 7fd9ea5a0640 10 monclient(hunting): start opening mon connection 2024-09-16T04:03:22.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.263+0000 7fd9ea5a0640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fd9e413f800 con 0x7fd9e40a1e20 2024-09-16T04:03:22.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.263+0000 7fd9ea5a0640 10 monclient(hunting): start opening mon connection 2024-09-16T04:03:22.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.263+0000 7fd9ea5a0640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd9e409ef80 con 0x7fd9e41357a0 2024-09-16T04:03:22.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.263+0000 7fd9ea5a0640 10 monclient(hunting): start opening mon connection 2024-09-16T04:03:22.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.263+0000 7fd9ea5a0640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fd9e413f980 con 0x7fd9e40a1330 2024-09-16T04:03:22.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.263+0000 7fd9ea5a0640 10 monclient(hunting): _renew_subs 2024-09-16T04:03:22.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.263+0000 7fd9ea5a0640 10 monclient(hunting): authenticate will time out at 2249.233162s 2024-09-16T04:03:22.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.263+0000 7fd9dbfff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd9e40a1e20 0x7fd9e4135260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:03:22.278 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.263+0000 7fd9e3fff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd9e40a1330 0x7fd9e41ceba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:03:22.278 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.263+0000 7fd9dbfff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd9e40a1e20 0x7fd9e4135260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:42448/0 (socket says 172.21.15.42:42448) 2024-09-16T04:03:22.278 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.263+0000 7fd9dbfff640 1 -- 172.21.15.42:0/1584040263 learned_addr learned my addr 172.21.15.42:0/1584040263 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:03:22.278 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.263+0000 7fd9dbfff640 10 monclient(hunting): get_auth_request con 0x7fd9e40a1e20 auth_method 0 2024-09-16T04:03:22.278 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.263+0000 7fd9dbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:03:22.278 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.263+0000 7fd9dbfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:03:22.278 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.263+0000 7fd9dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:03:22.278 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.263+0000 7fd9e3fff640 10 monclient(hunting): get_auth_request con 0x7fd9e40a1330 auth_method 0 2024-09-16T04:03:22.278 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.263+0000 7fd9e3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:03:22.278 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.263+0000 7fd9e3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:03:22.278 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.263+0000 7fd9e3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:03:22.278 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.263+0000 7fd9dbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:03:22.278 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.263+0000 7fd9dbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:03:22.279 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.263+0000 7fd9dbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:03:22.279 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.263+0000 7fd9e3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:03:22.279 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.263+0000 7fd9e3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:03:22.279 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.263+0000 7fd9e3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:03:22.279 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.263+0000 7fd9dbfff640 10 monclient(hunting): handle_auth_done global_id 825660 payload 824 2024-09-16T04:03:22.279 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.263+0000 7fd9dbfff640 1 -- 172.21.15.42:0/1584040263 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd9e40a1330 msgr2=0x7fd9e41ceba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:03:22.279 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.263+0000 7fd9dbfff640 1 --2- 172.21.15.42:0/1584040263 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd9e40a1330 0x7fd9e41ceba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:03:22.279 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.263+0000 7fd9dbfff640 1 -- 172.21.15.42:0/1584040263 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd9e41357a0 msgr2=0x7fd9e411dc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:03:22.279 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.263+0000 7fd9dbfff640 1 --2- 172.21.15.42:0/1584040263 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd9e41357a0 0x7fd9e411dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:03:22.279 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.263+0000 7fd9dbfff640 10 monclient: _finish_hunting 0 2024-09-16T04:03:22.279 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.263+0000 7fd9dbfff640 1 monclient: found mon.noname-a 2024-09-16T04:03:22.279 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.263+0000 7fd9dbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:03:22.279 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.263+0000 7fd9dbfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:03:22.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.263+0000 7fd9dbfff640 1 -- 172.21.15.42:0/1584040263 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9e413c950 con 0x7fd9e40a1e20 2024-09-16T04:03:22.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.263+0000 7fd9e3fff640 1 --2- 172.21.15.42:0/1584040263 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd9e40a1330 0x7fd9e41ceba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:03:22.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.263+0000 7fd9dbfff640 1 --2- 172.21.15.42:0/1584040263 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd9e40a1e20 0x7fd9e4135260 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7fd9cc01a0a0 tx=0x7fd9cc019210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:03:22.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.263+0000 7fd9e17fa640 10 client.?.objecter ms_handle_connect 0x7fd9e40a1e20 2024-09-16T04:03:22.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.263+0000 7fd9e17fa640 10 client.?.objecter resend_mon_ops 2024-09-16T04:03:22.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.263+0000 7fd9e17fa640 1 -- 172.21.15.42:0/1584040263 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9cc0b0040 con 0x7fd9e40a1e20 2024-09-16T04:03:22.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.263+0000 7fd9e17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:03:22.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.263+0000 7fd9e17fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:03:22.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.263+0000 7fd9e17fa640 10 monclient: dump: 2024-09-16T04:03:22.280 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:03:22.280 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:03:22.280 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:22.280 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:22.281 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:03:22.281 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:03:22.281 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:03:22.281 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:03:22.281 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:03:22.281 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:03:22.281 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.263+0000 7fd9e17fa640 10 monclient: _finish_auth 0 2024-09-16T04:03:22.281 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.263+0000 7fd9e17fa640 10 monclient: _check_auth_tickets 2024-09-16T04:03:22.281 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.263+0000 7fd9e17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:03:22.281 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.263+0000 7fd9e17fa640 1 -- 172.21.15.42:0/1584040263 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd9cc0b4420 con 0x7fd9e40a1e20 2024-09-16T04:03:22.281 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.263+0000 7fd9e17fa640 10 monclient: handle_config config(0 keys) 2024-09-16T04:03:22.281 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.263+0000 7fd9e17fa640 1 -- 172.21.15.42:0/1584040263 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9cc0b4810 con 0x7fd9e40a1e20 2024-09-16T04:03:22.281 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.263+0000 7fd9ea5a0640 5 monclient: authenticate success, global_id 825660 2024-09-16T04:03:22.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.263+0000 7fd9ea5a0640 10 monclient: _renew_subs 2024-09-16T04:03:22.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.263+0000 7fd9ea5a0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:03:22.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.263+0000 7fd9ea5a0640 1 -- 172.21.15.42:0/1584040263 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9e409cc80 con 0x7fd9e40a1e20 2024-09-16T04:03:22.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.263+0000 7fd9e2ffd640 4 set_mon_vals no callback set 2024-09-16T04:03:22.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.263+0000 7fd9e17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:03:22.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.263+0000 7fd9e17fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:03:22.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.263+0000 7fd9e17fa640 10 monclient: dump: 2024-09-16T04:03:22.282 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:03:22.282 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:03:22.282 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:22.282 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:22.282 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:03:22.282 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:03:22.282 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:03:22.283 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:03:22.283 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:03:22.283 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:03:22.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.263+0000 7fd9ea5a0640 10 client.825660.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:03:22.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.263+0000 7fd9ea5a0640 10 monclient: _renew_subs 2024-09-16T04:03:22.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.263+0000 7fd9ea5a0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:03:22.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.263+0000 7fd9ea5a0640 1 -- 172.21.15.42:0/1584040263 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9e411f920 con 0x7fd9e40a1e20 2024-09-16T04:03:22.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.263+0000 7fd9ea5a0640 1 librados: init done 2024-09-16T04:03:22.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.263+0000 7fd9ea5a0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:03:22.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.263+0000 7fd9ea5a0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:03:22.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.263+0000 7fd9ea5a0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:03:22.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.263+0000 7fd9ea5a0640 1 -- 172.21.15.42:0/1584040263 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd9e4121370 con 0x7fd9e40a1e20 2024-09-16T04:03:22.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.263+0000 7fd9e17fa640 1 -- 172.21.15.42:0/1584040263 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fd9cc0b49b0 con 0x7fd9e40a1e20 2024-09-16T04:03:22.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.263+0000 7fd9e17fa640 1 --2- 172.21.15.42:0/1584040263 >> v2:172.21.15.154:6812/254845342 conn(0x7fd9c4037860 0x7fd9c4039d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:03:22.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.263+0000 7fd9e17fa640 1 -- 172.21.15.42:0/1584040263 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(652..652 src has 451..652) ==== 5899+0+0 (secure 0 0 0) 0x7fd9cc0e2c20 con 0x7fd9e40a1e20 2024-09-16T04:03:22.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.263+0000 7fd9e17fa640 10 client.825660.objecter ms_dispatch 0x7fd9e4000ba0 osd_map(652..652 src has 451..652) 2024-09-16T04:03:22.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.263+0000 7fd9e17fa640 3 client.825660.objecter handle_osd_map got epochs [652,652] > 0 2024-09-16T04:03:22.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.263+0000 7fd9e17fa640 3 client.825660.objecter handle_osd_map decoding full epoch 652 2024-09-16T04:03:22.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.263+0000 7fd9e17fa640 20 client.825660.objecter dump_active .. 0 homeless 2024-09-16T04:03:22.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.263+0000 7fd9e3fff640 1 --2- 172.21.15.42:0/1584040263 >> v2:172.21.15.154:6812/254845342 conn(0x7fd9c4037860 0x7fd9c4039d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:03:22.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.267+0000 7fd9e3fff640 10 monclient: get_auth_request con 0x7fd9c4037860 auth_method 0 2024-09-16T04:03:22.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.267+0000 7fd9e17fa640 10 client.825660.objecter ms_handle_connect 0x7fd9c4037860 2024-09-16T04:03:22.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.267+0000 7fd9e3fff640 1 --2- 172.21.15.42:0/1584040263 >> v2:172.21.15.154:6812/254845342 conn(0x7fd9c4037860 0x7fd9c4039d20 secure :-1 s=READY pgs=3804 cs=0 l=1 rev1=1 crypto rx=0x7fd9d4007fd0 tx=0x7fd9d4007d60 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:03:22.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.271+0000 7fd9e17fa640 1 -- 172.21.15.42:0/1584040263 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7fd9cc0160f0 con 0x7fd9e40a1e20 2024-09-16T04:03:22.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.271+0000 7fd9e17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:03:22.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.271+0000 7fd9e17fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:03:22.499 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.495+0000 7fd9ea5a0640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T04:03:22.499 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.495+0000 7fd9ea5a0640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T04:03:22.499 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.495+0000 7fd9ea5a0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:03:22.499 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.495+0000 7fd9ea5a0640 1 -- 172.21.15.42:0/1584040263 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7fd9e40a1710 con 0x7fd9e40a1e20 2024-09-16T04:03:22.503 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.499+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.503 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.499+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=29,config=2,health=12,kv=9,logm=48} crc {auth=4022793450,config=692619927,health=3650494571,kv=867007345,logm=2806734730}) 2024-09-16T04:03:22.503 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.499+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=29,config=2,health=12,kv=9,logm=48} crc {auth=4022793450,config=692619927,health=3650494571,kv=3915008329,logm=2466837344}) 2024-09-16T04:03:22.503 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.499+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.503 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.499+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=29,config=2,health=12,kv=9,logm=48} crc {auth=4022793450,config=692619927,health=3650494571,kv=867007345,logm=2806734730}) 2024-09-16T04:03:22.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.499+0000 7fd9e17fa640 1 -- 172.21.15.42:0/1584040263 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7fd9cc0bd920 con 0x7fd9e40a1e20 2024-09-16T04:03:22.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.499+0000 7fd9e17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T04:03:22.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.499+0000 7fd9e17fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:03:22.504 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.499+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=28,config=2,health=12,kv=10,logm=48} crc {auth=2389484051,config=692619927,health=3650494571,kv=1063560236,logm=159635438}) 2024-09-16T04:03:22.505 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.499+0000 7fd9ea5a0640 10 librados: watch_flush enter 2024-09-16T04:03:22.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.499+0000 7fd9ea5a0640 10 librados: watch_flush exit 2024-09-16T04:03:22.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.499+0000 7fd9ea5a0640 20 client.825660.objecter shutdown clearing up homeless session... 2024-09-16T04:03:22.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.499+0000 7fd9ea5a0640 10 client.825660.objecter successfully canceled tick 2024-09-16T04:03:22.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.499+0000 7fd9ea5a0640 1 -- 172.21.15.42:0/1584040263 >> v2:172.21.15.154:6812/254845342 conn(0x7fd9c4037860 msgr2=0x7fd9c4039d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:03:22.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.499+0000 7fd9ea5a0640 1 --2- 172.21.15.42:0/1584040263 >> v2:172.21.15.154:6812/254845342 conn(0x7fd9c4037860 0x7fd9c4039d20 secure :-1 s=READY pgs=3804 cs=0 l=1 rev1=1 crypto rx=0x7fd9d4007fd0 tx=0x7fd9d4007d60 comp rx=0 tx=0).stop 2024-09-16T04:03:22.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.499+0000 7fd9ea5a0640 10 monclient: shutdown 2024-09-16T04:03:22.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.499+0000 7fd9ea5a0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:03:22.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.499+0000 7fd9ea5a0640 1 -- 172.21.15.42:0/1584040263 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd9e40a1e20 msgr2=0x7fd9e4135260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:03:22.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.499+0000 7fd9ea5a0640 1 --2- 172.21.15.42:0/1584040263 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd9e40a1e20 0x7fd9e4135260 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7fd9cc01a0a0 tx=0x7fd9cc019210 comp rx=0 tx=0).stop 2024-09-16T04:03:22.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.499+0000 7fd9ea5a0640 1 -- 172.21.15.42:0/1584040263 shutdown_connections 2024-09-16T04:03:22.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.499+0000 7fd9ea5a0640 1 --2- 172.21.15.42:0/1584040263 >> v2:172.21.15.154:6812/254845342 conn(0x7fd9c4037860 0x7fd9c4039d20 unknown :-1 s=CLOSED pgs=3804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:03:22.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.503+0000 7fd9ea5a0640 1 --2- 172.21.15.42:0/1584040263 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd9e41357a0 0x7fd9e411dc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:03:22.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.503+0000 7fd9ea5a0640 1 --2- 172.21.15.42:0/1584040263 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd9e40a1e20 0x7fd9e4135260 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:03:22.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.503+0000 7fd9ea5a0640 1 --2- 172.21.15.42:0/1584040263 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd9e40a1330 0x7fd9e41ceba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:03:22.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.503+0000 7fd9ea5a0640 1 -- 172.21.15.42:0/1584040263 >> 172.21.15.42:0/1584040263 conn(0x7fd9e40a2420 msgr2=0x7fd9e40a45b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:03:22.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.503+0000 7fd9ea5a0640 1 -- 172.21.15.42:0/1584040263 shutdown_connections 2024-09-16T04:03:22.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.503+0000 7fd9ea5a0640 1 -- 172.21.15.42:0/1584040263 wait complete. 2024-09-16T04:03:22.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:22.503+0000 7fd9ea5a0640 1 librados: shutdown 2024-09-16T04:03:22.508 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.503+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.508 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.503+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2937526117}) 2024-09-16T04:03:22.508 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.503+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2225337669}) 2024-09-16T04:03:22.508 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.503+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.508 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.503+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2937526117}) 2024-09-16T04:03:22.508 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.503+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3831183717}) 2024-09-16T04:03:22.513 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.507+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.513 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.507+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=462623004}) 2024-09-16T04:03:22.513 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.507+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1027660863}) 2024-09-16T04:03:22.513 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.507+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.513 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.507+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=462623004}) 2024-09-16T04:03:22.513 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.507+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3958549268}) 2024-09-16T04:03:22.518 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.515+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.518 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.515+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2710587115}) 2024-09-16T04:03:22.518 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.515+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1650213552}) 2024-09-16T04:03:22.518 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.515+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.518 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.515+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2710587115}) 2024-09-16T04:03:22.519 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.515+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1567516544}) 2024-09-16T04:03:22.523 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T04:03:22.524 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.519+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.524 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.519+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2516238893}) 2024-09-16T04:03:22.524 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.519+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1167438603}) 2024-09-16T04:03:22.524 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.519+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.524 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.519+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2516238893}) 2024-09-16T04:03:22.524 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.519+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3330331234}) 2024-09-16T04:03:22.526 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.523+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.527 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.523+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3572452438}) 2024-09-16T04:03:22.527 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.523+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1296475073}) 2024-09-16T04:03:22.527 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.523+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.527 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.523+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3572452438}) 2024-09-16T04:03:22.527 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.523+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1698274964}) 2024-09-16T04:03:22.529 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.523+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.529 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.523+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3690464955}) 2024-09-16T04:03:22.529 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.523+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2716543731}) 2024-09-16T04:03:22.529 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.523+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.529 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.523+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3690464955}) 2024-09-16T04:03:22.529 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.523+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2611465567}) 2024-09-16T04:03:22.531 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.527+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.531 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.527+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3192934108}) 2024-09-16T04:03:22.531 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.527+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3534892703}) 2024-09-16T04:03:22.531 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.527+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.531 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.527+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3192934108}) 2024-09-16T04:03:22.531 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.527+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2766120644}) 2024-09-16T04:03:22.533 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.527+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.533 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.527+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1729044589}) 2024-09-16T04:03:22.533 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.527+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=591291077}) 2024-09-16T04:03:22.533 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.527+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.533 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.527+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1729044589}) 2024-09-16T04:03:22.533 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.527+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4216398637}) 2024-09-16T04:03:22.535 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.531+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.535 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.531+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1809125264}) 2024-09-16T04:03:22.535 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.531+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1958488749}) 2024-09-16T04:03:22.535 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.531+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.535 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.531+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1809125264}) 2024-09-16T04:03:22.536 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.531+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1087914613}) 2024-09-16T04:03:22.537 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.535+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.538 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.535+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2128614292}) 2024-09-16T04:03:22.538 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.535+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2010886453}) 2024-09-16T04:03:22.538 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.535+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.538 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.535+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2128614292}) 2024-09-16T04:03:22.538 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.535+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=933557310}) 2024-09-16T04:03:22.540 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.535+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.540 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.535+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2930542887}) 2024-09-16T04:03:22.540 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.535+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2338196220}) 2024-09-16T04:03:22.540 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.535+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.540 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.535+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2930542887}) 2024-09-16T04:03:22.540 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.535+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4008442700}) 2024-09-16T04:03:22.542 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.539+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.542 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.539+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=559193694}) 2024-09-16T04:03:22.542 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.539+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1412299436}) 2024-09-16T04:03:22.542 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.539+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.542 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.539+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=559193694}) 2024-09-16T04:03:22.542 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.539+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2663213507}) 2024-09-16T04:03:22.544 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.539+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.544 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.539+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1266955354}) 2024-09-16T04:03:22.544 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.539+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=345977069}) 2024-09-16T04:03:22.544 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.539+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.544 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.539+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1266955354}) 2024-09-16T04:03:22.544 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.539+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1428640730}) 2024-09-16T04:03:22.546 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.543+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.546 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.543+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4086729846}) 2024-09-16T04:03:22.546 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.543+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=979533896}) 2024-09-16T04:03:22.546 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.543+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.546 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.543+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4086729846}) 2024-09-16T04:03:22.547 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.543+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=466772147}) 2024-09-16T04:03:22.548 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.543+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.548 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.543+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=36025446}) 2024-09-16T04:03:22.549 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.543+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3575148868}) 2024-09-16T04:03:22.549 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.543+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.549 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.543+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=36025446}) 2024-09-16T04:03:22.549 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.543+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=251773949}) 2024-09-16T04:03:22.551 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.547+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.551 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.547+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3962129707}) 2024-09-16T04:03:22.551 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.547+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3156439129}) 2024-09-16T04:03:22.551 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.547+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.551 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.547+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3962129707}) 2024-09-16T04:03:22.551 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.547+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1881590153}) 2024-09-16T04:03:22.552 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.547+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.553 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.547+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=134122335}) 2024-09-16T04:03:22.553 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.547+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=696112471}) 2024-09-16T04:03:22.553 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.547+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.553 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.547+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=134122335}) 2024-09-16T04:03:22.553 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.547+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4043597869}) 2024-09-16T04:03:22.554 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.551+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.555 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.551+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2787002531}) 2024-09-16T04:03:22.555 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.551+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1870852762}) 2024-09-16T04:03:22.555 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.551+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.555 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.551+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2787002531}) 2024-09-16T04:03:22.555 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.551+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2233580069}) 2024-09-16T04:03:22.557 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.551+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.557 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.551+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2835510931}) 2024-09-16T04:03:22.557 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.551+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1333734048}) 2024-09-16T04:03:22.557 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.551+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.557 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.551+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2835510931}) 2024-09-16T04:03:22.557 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.551+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1220744874}) 2024-09-16T04:03:22.559 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.555+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.559 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.555+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1759650584}) 2024-09-16T04:03:22.559 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.555+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3474727194}) 2024-09-16T04:03:22.559 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.555+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.559 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.555+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1759650584}) 2024-09-16T04:03:22.559 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.555+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3910228954}) 2024-09-16T04:03:22.561 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.555+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.561 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.555+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2762744776}) 2024-09-16T04:03:22.561 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.555+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=721064866}) 2024-09-16T04:03:22.561 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.555+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.561 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.555+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2762744776}) 2024-09-16T04:03:22.561 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.555+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3046687736}) 2024-09-16T04:03:22.563 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.559+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.563 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.559+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3924784514}) 2024-09-16T04:03:22.563 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.559+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2316897025}) 2024-09-16T04:03:22.563 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.559+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.564 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.559+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3924784514}) 2024-09-16T04:03:22.564 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.559+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=685068276}) 2024-09-16T04:03:22.565 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.563+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.566 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.563+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2122436300}) 2024-09-16T04:03:22.566 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.563+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=530977133}) 2024-09-16T04:03:22.566 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.563+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.566 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.563+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2122436300}) 2024-09-16T04:03:22.566 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.563+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3782200124}) 2024-09-16T04:03:22.568 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.563+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.568 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.563+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3989713019}) 2024-09-16T04:03:22.568 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.563+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4064523960}) 2024-09-16T04:03:22.568 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.563+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.568 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.563+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3989713019}) 2024-09-16T04:03:22.568 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.563+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1010550055}) 2024-09-16T04:03:22.570 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.567+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.570 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.567+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=551858096}) 2024-09-16T04:03:22.570 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.567+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1047420424}) 2024-09-16T04:03:22.570 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.567+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.570 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.567+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=551858096}) 2024-09-16T04:03:22.570 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.567+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1832725056}) 2024-09-16T04:03:22.572 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.567+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.572 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.567+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2902402989}) 2024-09-16T04:03:22.572 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.567+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3384721477}) 2024-09-16T04:03:22.572 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.567+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.572 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.567+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2902402989}) 2024-09-16T04:03:22.572 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.567+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2156927202}) 2024-09-16T04:03:22.574 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.571+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.574 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.571+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3059373721}) 2024-09-16T04:03:22.574 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.571+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2410903535}) 2024-09-16T04:03:22.575 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.571+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.575 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.571+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3059373721}) 2024-09-16T04:03:22.575 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.571+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2530553514}) 2024-09-16T04:03:22.576 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.571+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.576 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.571+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1450836506}) 2024-09-16T04:03:22.577 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.571+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1290179002}) 2024-09-16T04:03:22.577 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.571+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.577 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.571+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1450836506}) 2024-09-16T04:03:22.577 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.571+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2088729642}) 2024-09-16T04:03:22.579 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.575+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.579 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.575+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3964744894}) 2024-09-16T04:03:22.579 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.575+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2302161070}) 2024-09-16T04:03:22.579 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.575+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.579 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.575+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3964744894}) 2024-09-16T04:03:22.579 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.575+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3575331401}) 2024-09-16T04:03:22.580 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.575+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.581 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.575+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1577546261}) 2024-09-16T04:03:22.581 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.575+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2422004522}) 2024-09-16T04:03:22.581 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.575+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.581 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.575+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1577546261}) 2024-09-16T04:03:22.581 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.575+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2570314141}) 2024-09-16T04:03:22.583 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.579+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.583 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.579+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1900841697}) 2024-09-16T04:03:22.583 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.579+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3479960015}) 2024-09-16T04:03:22.583 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.579+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.583 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.579+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1900841697}) 2024-09-16T04:03:22.583 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.579+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1458952107}) 2024-09-16T04:03:22.585 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.579+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.585 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.579+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2016520770}) 2024-09-16T04:03:22.585 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.579+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2783319633}) 2024-09-16T04:03:22.585 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.579+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.585 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.579+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2016520770}) 2024-09-16T04:03:22.585 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.579+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1380191206}) 2024-09-16T04:03:22.587 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.583+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.587 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.583+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3330068027}) 2024-09-16T04:03:22.587 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.583+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1173821660}) 2024-09-16T04:03:22.587 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.583+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.587 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.583+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3330068027}) 2024-09-16T04:03:22.587 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.583+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=512146696}) 2024-09-16T04:03:22.589 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.583+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.589 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.583+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4086089176}) 2024-09-16T04:03:22.589 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.583+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=907694044}) 2024-09-16T04:03:22.589 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.583+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.589 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.583+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4086089176}) 2024-09-16T04:03:22.589 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.583+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2033120033}) 2024-09-16T04:03:22.591 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.587+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.591 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.587+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2046018160}) 2024-09-16T04:03:22.591 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.587+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2264587721}) 2024-09-16T04:03:22.591 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.587+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.591 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.587+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2046018160}) 2024-09-16T04:03:22.591 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.587+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3714029974}) 2024-09-16T04:03:22.593 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.587+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.593 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.587+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4147619558}) 2024-09-16T04:03:22.593 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.587+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=78677958}) 2024-09-16T04:03:22.593 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.587+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.593 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.587+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4147619558}) 2024-09-16T04:03:22.593 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.587+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1658466672}) 2024-09-16T04:03:22.595 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.591+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.595 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.591+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=479757952}) 2024-09-16T04:03:22.595 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.591+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3344105659}) 2024-09-16T04:03:22.595 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.591+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.595 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.591+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=479757952}) 2024-09-16T04:03:22.595 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.591+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3165014567}) 2024-09-16T04:03:22.597 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.591+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.597 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.591+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4254564328}) 2024-09-16T04:03:22.597 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.591+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2636263996}) 2024-09-16T04:03:22.597 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.591+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.597 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.591+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4254564328}) 2024-09-16T04:03:22.597 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.595+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3822076887}) 2024-09-16T04:03:22.599 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.595+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.600 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.595+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3573955818}) 2024-09-16T04:03:22.600 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.595+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=433243920}) 2024-09-16T04:03:22.600 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.595+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.600 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.595+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3573955818}) 2024-09-16T04:03:22.600 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.595+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=939333457}) 2024-09-16T04:03:22.601 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.595+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.601 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.599+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1607098697}) 2024-09-16T04:03:22.602 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.599+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1309797529}) 2024-09-16T04:03:22.602 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.599+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.602 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.599+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1607098697}) 2024-09-16T04:03:22.602 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.599+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2956909148}) 2024-09-16T04:03:22.603 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.599+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.603 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.599+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3280442440}) 2024-09-16T04:03:22.603 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.599+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4222983224}) 2024-09-16T04:03:22.603 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.599+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.604 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.599+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3280442440}) 2024-09-16T04:03:22.604 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.599+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1858797479}) 2024-09-16T04:03:22.605 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.603+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.605 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.603+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4219460551}) 2024-09-16T04:03:22.606 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.603+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3120067634}) 2024-09-16T04:03:22.606 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.603+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.606 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.603+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4219460551}) 2024-09-16T04:03:22.606 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.603+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2189113292}) 2024-09-16T04:03:22.607 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.603+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.607 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.603+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1194892786}) 2024-09-16T04:03:22.607 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.603+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3501907039}) 2024-09-16T04:03:22.608 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.603+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.608 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.603+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1194892786}) 2024-09-16T04:03:22.608 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.603+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=997498193}) 2024-09-16T04:03:22.610 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.607+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.610 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.607+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2073464289}) 2024-09-16T04:03:22.610 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.607+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2736195453}) 2024-09-16T04:03:22.610 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.607+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.610 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.607+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2073464289}) 2024-09-16T04:03:22.610 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.607+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1906590298}) 2024-09-16T04:03:22.612 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.607+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.612 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.607+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2286282596}) 2024-09-16T04:03:22.612 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.607+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1241142142}) 2024-09-16T04:03:22.612 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.607+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.612 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.607+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2286282596}) 2024-09-16T04:03:22.612 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.607+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=370472417}) 2024-09-16T04:03:22.614 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.611+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.614 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.611+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1851317955}) 2024-09-16T04:03:22.614 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.611+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1143634221}) 2024-09-16T04:03:22.614 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.611+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.614 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.611+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1851317955}) 2024-09-16T04:03:22.614 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.611+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3485633054}) 2024-09-16T04:03:22.616 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.611+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.616 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.611+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2252571595}) 2024-09-16T04:03:22.616 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.611+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1563237595}) 2024-09-16T04:03:22.616 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.611+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.616 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.611+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2252571595}) 2024-09-16T04:03:22.616 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.611+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=20376917}) 2024-09-16T04:03:22.618 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.615+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.618 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.615+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3707192778}) 2024-09-16T04:03:22.618 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.615+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3379144640}) 2024-09-16T04:03:22.618 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.615+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.618 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.615+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3707192778}) 2024-09-16T04:03:22.618 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.615+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2460353564}) 2024-09-16T04:03:22.620 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.615+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.620 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.615+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=182079116}) 2024-09-16T04:03:22.620 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.615+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=653646385}) 2024-09-16T04:03:22.620 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.615+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.620 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.615+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=182079116}) 2024-09-16T04:03:22.620 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.615+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3738855071}) 2024-09-16T04:03:22.622 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.619+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.622 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.619+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=129047582}) 2024-09-16T04:03:22.622 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.619+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3591627671}) 2024-09-16T04:03:22.622 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.619+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.622 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.619+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=129047582}) 2024-09-16T04:03:22.622 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.619+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2401339397}) 2024-09-16T04:03:22.624 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.619+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.624 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.619+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3189443395}) 2024-09-16T04:03:22.624 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.619+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1210459909}) 2024-09-16T04:03:22.624 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.619+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.625 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.619+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3189443395}) 2024-09-16T04:03:22.625 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.619+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1536148079}) 2024-09-16T04:03:22.626 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.623+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.626 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.623+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3045322455}) 2024-09-16T04:03:22.626 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.623+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2035899939}) 2024-09-16T04:03:22.626 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.623+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.626 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.623+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3045322455}) 2024-09-16T04:03:22.627 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.623+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1939439909}) 2024-09-16T04:03:22.628 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.623+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.628 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.623+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3358242020}) 2024-09-16T04:03:22.628 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.623+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3279015454}) 2024-09-16T04:03:22.628 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.623+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.629 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.623+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3358242020}) 2024-09-16T04:03:22.629 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.623+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3407089522}) 2024-09-16T04:03:22.630 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.627+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.630 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.627+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1472452413}) 2024-09-16T04:03:22.631 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.627+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3658684752}) 2024-09-16T04:03:22.631 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.627+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.631 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.627+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1472452413}) 2024-09-16T04:03:22.631 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.627+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3214841424}) 2024-09-16T04:03:22.632 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.627+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.632 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.627+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=82650638}) 2024-09-16T04:03:22.632 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.627+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1874169067}) 2024-09-16T04:03:22.633 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.627+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.633 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.627+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=82650638}) 2024-09-16T04:03:22.633 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.627+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=838970928}) 2024-09-16T04:03:22.634 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.631+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.634 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.631+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=833711723}) 2024-09-16T04:03:22.634 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.631+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2642350339}) 2024-09-16T04:03:22.634 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.631+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.635 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.631+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=833711723}) 2024-09-16T04:03:22.635 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.631+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=962263606}) 2024-09-16T04:03:22.636 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.631+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.636 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.631+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=316926417}) 2024-09-16T04:03:22.636 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.631+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3059751372}) 2024-09-16T04:03:22.637 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.631+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.637 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.631+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=316926417}) 2024-09-16T04:03:22.637 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.631+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=260260249}) 2024-09-16T04:03:22.638 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.635+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.638 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.635+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3936939861}) 2024-09-16T04:03:22.638 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.635+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1882284088}) 2024-09-16T04:03:22.638 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.635+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.638 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.635+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3936939861}) 2024-09-16T04:03:22.639 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.635+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4073548507}) 2024-09-16T04:03:22.640 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.635+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.640 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.635+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=924838222}) 2024-09-16T04:03:22.640 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.635+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=669380813}) 2024-09-16T04:03:22.640 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.635+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.640 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.635+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=924838222}) 2024-09-16T04:03:22.640 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.635+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3117375328}) 2024-09-16T04:03:22.642 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.639+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.642 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.639+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=558196160}) 2024-09-16T04:03:22.642 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.639+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2241735939}) 2024-09-16T04:03:22.642 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.639+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.643 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.639+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=558196160}) 2024-09-16T04:03:22.643 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.639+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3926873792}) 2024-09-16T04:03:22.644 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.639+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.644 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.639+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3583857499}) 2024-09-16T04:03:22.644 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.639+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3496821072}) 2024-09-16T04:03:22.644 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.639+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.645 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.639+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3583857499}) 2024-09-16T04:03:22.645 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.639+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=464082040}) 2024-09-16T04:03:22.646 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.643+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.646 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.643+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2588235554}) 2024-09-16T04:03:22.646 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.643+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=521257211}) 2024-09-16T04:03:22.647 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.643+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.647 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.643+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2588235554}) 2024-09-16T04:03:22.647 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.643+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=730487531}) 2024-09-16T04:03:22.648 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.643+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.648 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.643+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3610193075}) 2024-09-16T04:03:22.649 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.643+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2703668772}) 2024-09-16T04:03:22.649 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.643+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.649 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.643+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3610193075}) 2024-09-16T04:03:22.649 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.643+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1726025572}) 2024-09-16T04:03:22.650 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.647+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.650 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.647+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=999181194}) 2024-09-16T04:03:22.651 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.647+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=475490234}) 2024-09-16T04:03:22.651 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.647+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.651 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.647+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=999181194}) 2024-09-16T04:03:22.651 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.647+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3334197014}) 2024-09-16T04:03:22.652 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.647+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.652 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.647+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3400405759}) 2024-09-16T04:03:22.653 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.647+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4272039703}) 2024-09-16T04:03:22.653 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.647+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.653 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.647+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3400405759}) 2024-09-16T04:03:22.653 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.647+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3196345893}) 2024-09-16T04:03:22.658 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.655+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.658 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.655+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=706645113}) 2024-09-16T04:03:22.658 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.655+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1175609932}) 2024-09-16T04:03:22.658 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.655+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.658 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.655+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=706645113}) 2024-09-16T04:03:22.658 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.655+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4151969049}) 2024-09-16T04:03:22.667 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.663+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.667 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.663+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1481652432}) 2024-09-16T04:03:22.667 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.663+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2186497919}) 2024-09-16T04:03:22.667 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.663+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.668 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.663+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1481652432}) 2024-09-16T04:03:22.668 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.663+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3511351453}) 2024-09-16T04:03:22.672 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.667+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.672 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.667+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2816514715}) 2024-09-16T04:03:22.672 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.667+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2820390886}) 2024-09-16T04:03:22.672 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.667+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.672 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.667+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2816514715}) 2024-09-16T04:03:22.673 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.667+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1543490097}) 2024-09-16T04:03:22.704 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.699+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.704 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.699+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2316123674}) 2024-09-16T04:03:22.704 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.699+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3312075297}) 2024-09-16T04:03:22.704 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.699+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.704 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.699+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2316123674}) 2024-09-16T04:03:22.704 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.699+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1945618877}) 2024-09-16T04:03:22.710 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.707+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.710 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.707+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3220920319}) 2024-09-16T04:03:22.710 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.707+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2370800584}) 2024-09-16T04:03:22.711 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.707+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.711 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.707+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3220920319}) 2024-09-16T04:03:22.711 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.707+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1721609985}) 2024-09-16T04:03:22.711 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.707+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.712 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.707+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3658108444}) 2024-09-16T04:03:22.712 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.707+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1768111179}) 2024-09-16T04:03:22.712 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.707+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.712 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.707+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3658108444}) 2024-09-16T04:03:22.712 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.707+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2615074726}) 2024-09-16T04:03:22.713 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.707+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.713 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.707+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2870055014}) 2024-09-16T04:03:22.713 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.707+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1699940790}) 2024-09-16T04:03:22.713 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.707+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.713 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.707+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2870055014}) 2024-09-16T04:03:22.713 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.707+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1466035723}) 2024-09-16T04:03:22.715 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.711+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.715 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.711+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2787743698}) 2024-09-16T04:03:22.715 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.711+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1821945385}) 2024-09-16T04:03:22.715 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.711+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.715 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.711+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2787743698}) 2024-09-16T04:03:22.715 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.711+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1614440287}) 2024-09-16T04:03:22.717 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.711+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.717 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.711+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1291687425}) 2024-09-16T04:03:22.717 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.711+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2613079709}) 2024-09-16T04:03:22.717 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.711+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.717 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.711+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1291687425}) 2024-09-16T04:03:22.717 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.711+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2561275772}) 2024-09-16T04:03:22.719 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.715+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.719 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.715+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2484860389}) 2024-09-16T04:03:22.719 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.715+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1625415516}) 2024-09-16T04:03:22.719 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.715+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.719 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.715+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2484860389}) 2024-09-16T04:03:22.719 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.715+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=346822292}) 2024-09-16T04:03:22.721 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.715+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.721 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.715+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2374240416}) 2024-09-16T04:03:22.721 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.715+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1327489247}) 2024-09-16T04:03:22.721 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.715+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.721 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.715+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2374240416}) 2024-09-16T04:03:22.721 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.715+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1409991827}) 2024-09-16T04:03:22.723 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.719+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.723 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.719+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1297224686}) 2024-09-16T04:03:22.723 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.719+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1223332702}) 2024-09-16T04:03:22.723 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.719+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.723 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.719+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1297224686}) 2024-09-16T04:03:22.723 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.719+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3924445516}) 2024-09-16T04:03:22.725 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.719+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.725 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.719+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=423257689}) 2024-09-16T04:03:22.725 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.719+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=774460820}) 2024-09-16T04:03:22.725 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.719+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.725 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.719+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=423257689}) 2024-09-16T04:03:22.725 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.723+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=739012109}) 2024-09-16T04:03:22.727 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.723+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.727 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.723+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1141482557}) 2024-09-16T04:03:22.727 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.723+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=880148922}) 2024-09-16T04:03:22.727 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.723+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.727 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.723+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1141482557}) 2024-09-16T04:03:22.727 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.723+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=905418338}) 2024-09-16T04:03:22.729 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.723+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.729 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.723+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2071745403}) 2024-09-16T04:03:22.729 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.727+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3796405978}) 2024-09-16T04:03:22.729 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.727+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.729 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.727+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2071745403}) 2024-09-16T04:03:22.730 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.727+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=208611204}) 2024-09-16T04:03:22.731 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.727+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.731 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.727+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=27080482}) 2024-09-16T04:03:22.731 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.727+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1198878231}) 2024-09-16T04:03:22.731 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.727+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.731 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.727+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=27080482}) 2024-09-16T04:03:22.731 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.727+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2074408501}) 2024-09-16T04:03:22.733 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.727+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.733 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.727+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3763136808}) 2024-09-16T04:03:22.733 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.727+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2256470182}) 2024-09-16T04:03:22.733 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.727+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.733 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.731+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3763136808}) 2024-09-16T04:03:22.733 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.731+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=334955139}) 2024-09-16T04:03:22.735 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.731+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.735 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.731+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=367336684}) 2024-09-16T04:03:22.735 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.731+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3850724150}) 2024-09-16T04:03:22.735 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.731+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.735 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.731+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=367336684}) 2024-09-16T04:03:22.735 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.731+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=995525047}) 2024-09-16T04:03:22.737 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.731+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.737 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.731+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=983243988}) 2024-09-16T04:03:22.737 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.731+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=499677860}) 2024-09-16T04:03:22.738 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.731+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.738 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.735+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=983243988}) 2024-09-16T04:03:22.738 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.735+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2351461441}) 2024-09-16T04:03:22.739 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.735+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.739 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.735+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1527591925}) 2024-09-16T04:03:22.739 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.735+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=503578702}) 2024-09-16T04:03:22.739 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.735+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.739 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.735+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1527591925}) 2024-09-16T04:03:22.739 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.735+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1652925664}) 2024-09-16T04:03:22.741 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.735+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.741 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.735+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3196967152}) 2024-09-16T04:03:22.741 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.735+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=303631346}) 2024-09-16T04:03:22.741 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.735+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.741 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.735+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3196967152}) 2024-09-16T04:03:22.741 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.735+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1578174470}) 2024-09-16T04:03:22.743 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.739+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.743 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.739+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3153792528}) 2024-09-16T04:03:22.743 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.739+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3262849011}) 2024-09-16T04:03:22.743 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.739+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.743 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.739+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3153792528}) 2024-09-16T04:03:22.743 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.739+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3750540965}) 2024-09-16T04:03:22.744 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.739+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.745 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.739+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4240755289}) 2024-09-16T04:03:22.745 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.739+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=625196472}) 2024-09-16T04:03:22.745 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.739+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.745 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.739+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4240755289}) 2024-09-16T04:03:22.745 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.739+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4052598049}) 2024-09-16T04:03:22.746 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.743+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.746 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.743+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1871956313}) 2024-09-16T04:03:22.746 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.743+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2042279287}) 2024-09-16T04:03:22.747 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.743+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.747 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.743+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1871956313}) 2024-09-16T04:03:22.747 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.743+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1232428194}) 2024-09-16T04:03:22.748 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.743+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.748 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.743+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3286228753}) 2024-09-16T04:03:22.748 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.743+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2515398689}) 2024-09-16T04:03:22.748 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.743+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.748 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.743+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3286228753}) 2024-09-16T04:03:22.748 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.743+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2438863283}) 2024-09-16T04:03:22.750 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.747+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.750 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.747+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3018823143}) 2024-09-16T04:03:22.750 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.747+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3960005389}) 2024-09-16T04:03:22.750 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.747+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.750 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.747+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3018823143}) 2024-09-16T04:03:22.750 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.747+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1198913990}) 2024-09-16T04:03:22.752 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.747+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.752 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.747+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=179391482}) 2024-09-16T04:03:22.752 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.747+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2410490583}) 2024-09-16T04:03:22.752 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.747+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.752 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.747+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=179391482}) 2024-09-16T04:03:22.752 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.747+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2111891219}) 2024-09-16T04:03:22.754 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.751+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.754 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.751+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1268060841}) 2024-09-16T04:03:22.754 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.751+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=863194484}) 2024-09-16T04:03:22.754 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.751+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.754 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.751+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1268060841}) 2024-09-16T04:03:22.754 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.751+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3181624251}) 2024-09-16T04:03:22.756 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.751+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.756 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.751+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3078072571}) 2024-09-16T04:03:22.756 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.751+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1494921629}) 2024-09-16T04:03:22.756 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.751+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.757 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.751+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3078072571}) 2024-09-16T04:03:22.757 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.751+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1100300647}) 2024-09-16T04:03:22.758 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.755+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.758 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.755+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2054592532}) 2024-09-16T04:03:22.759 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.755+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2055369331}) 2024-09-16T04:03:22.759 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.755+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.759 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.755+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2054592532}) 2024-09-16T04:03:22.759 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.755+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3548839737}) 2024-09-16T04:03:22.760 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.755+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.761 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.755+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1735386001}) 2024-09-16T04:03:22.761 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.755+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2559819244}) 2024-09-16T04:03:22.761 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.755+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.761 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.755+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1735386001}) 2024-09-16T04:03:22.761 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.755+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=949356996}) 2024-09-16T04:03:22.763 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.759+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.763 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.759+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=589686099}) 2024-09-16T04:03:22.763 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.759+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=246234153}) 2024-09-16T04:03:22.763 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.759+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.763 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.759+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=589686099}) 2024-09-16T04:03:22.763 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.759+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=501421141}) 2024-09-16T04:03:22.765 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.759+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.765 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.759+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3479519243}) 2024-09-16T04:03:22.765 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.759+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2621300637}) 2024-09-16T04:03:22.765 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.759+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.765 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.759+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3479519243}) 2024-09-16T04:03:22.765 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.759+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=125788416}) 2024-09-16T04:03:22.767 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.763+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.767 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.763+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2369031469}) 2024-09-16T04:03:22.767 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.763+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=138257060}) 2024-09-16T04:03:22.767 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.763+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.767 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.763+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2369031469}) 2024-09-16T04:03:22.768 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.763+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4085678136}) 2024-09-16T04:03:22.769 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.763+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.769 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.763+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=52929231}) 2024-09-16T04:03:22.769 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.763+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2315259273}) 2024-09-16T04:03:22.769 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.763+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.769 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.763+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=52929231}) 2024-09-16T04:03:22.769 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.763+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2464212990}) 2024-09-16T04:03:22.771 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.767+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.771 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.767+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1965812043}) 2024-09-16T04:03:22.771 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.767+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=700085014}) 2024-09-16T04:03:22.771 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.767+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.771 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.767+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1965812043}) 2024-09-16T04:03:22.771 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.767+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3345327701}) 2024-09-16T04:03:22.773 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.767+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.773 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.767+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=244600042}) 2024-09-16T04:03:22.773 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.767+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2382879234}) 2024-09-16T04:03:22.773 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.767+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.773 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.767+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=244600042}) 2024-09-16T04:03:22.773 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.767+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4164986050}) 2024-09-16T04:03:22.775 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.771+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.775 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.771+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3565465312}) 2024-09-16T04:03:22.775 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.771+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3500775200}) 2024-09-16T04:03:22.775 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.771+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.775 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.771+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3565465312}) 2024-09-16T04:03:22.775 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.771+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3834320860}) 2024-09-16T04:03:22.777 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.775+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.777 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.775+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1750565579}) 2024-09-16T04:03:22.778 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.775+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=113638995}) 2024-09-16T04:03:22.778 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.775+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.778 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.775+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1750565579}) 2024-09-16T04:03:22.778 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.775+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=19879915}) 2024-09-16T04:03:22.779 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.775+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.780 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.775+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2721647696}) 2024-09-16T04:03:22.780 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.775+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3673351445}) 2024-09-16T04:03:22.780 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.775+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.780 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.775+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2721647696}) 2024-09-16T04:03:22.780 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.775+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=645288996}) 2024-09-16T04:03:22.782 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.779+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.782 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.779+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2705107854}) 2024-09-16T04:03:22.782 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.779+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2717227798}) 2024-09-16T04:03:22.782 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.779+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.782 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.779+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2705107854}) 2024-09-16T04:03:22.782 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.779+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1341580379}) 2024-09-16T04:03:22.784 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.779+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.784 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.779+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=347231199}) 2024-09-16T04:03:22.784 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.779+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2093431047}) 2024-09-16T04:03:22.785 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.779+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.785 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.779+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=347231199}) 2024-09-16T04:03:22.785 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.779+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1254222742}) 2024-09-16T04:03:22.786 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.783+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.787 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.783+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2125906508}) 2024-09-16T04:03:22.787 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.783+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1862848290}) 2024-09-16T04:03:22.787 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.783+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.787 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.783+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2125906508}) 2024-09-16T04:03:22.787 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.783+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1993581231}) 2024-09-16T04:03:22.788 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.783+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.788 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.783+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2149684041}) 2024-09-16T04:03:22.788 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.783+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1338935519}) 2024-09-16T04:03:22.788 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.783+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.788 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.783+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2149684041}) 2024-09-16T04:03:22.788 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.783+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1338206393}) 2024-09-16T04:03:22.790 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.787+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.790 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.787+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=661091329}) 2024-09-16T04:03:22.790 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.787+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=392634976}) 2024-09-16T04:03:22.790 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.787+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.790 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.787+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=661091329}) 2024-09-16T04:03:22.790 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.787+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3212388590}) 2024-09-16T04:03:22.792 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.787+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.792 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.787+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1826381819}) 2024-09-16T04:03:22.792 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.787+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3444924123}) 2024-09-16T04:03:22.792 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.787+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.792 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.787+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1826381819}) 2024-09-16T04:03:22.793 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.787+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3837044600}) 2024-09-16T04:03:22.794 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.791+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.794 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.791+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1807354637}) 2024-09-16T04:03:22.794 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.791+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=91179715}) 2024-09-16T04:03:22.794 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.791+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.794 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.791+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1807354637}) 2024-09-16T04:03:22.794 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.791+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2985578681}) 2024-09-16T04:03:22.795 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.791+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.795 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.791+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3562511877}) 2024-09-16T04:03:22.796 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.791+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=896413533}) 2024-09-16T04:03:22.796 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.791+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.796 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.791+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3562511877}) 2024-09-16T04:03:22.796 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.791+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=809945070}) 2024-09-16T04:03:22.797 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.791+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.797 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.791+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=951858435}) 2024-09-16T04:03:22.797 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.791+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3102412062}) 2024-09-16T04:03:22.797 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.791+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.797 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.791+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=951858435}) 2024-09-16T04:03:22.797 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.791+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4034292673}) 2024-09-16T04:03:22.799 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.795+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.799 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.795+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1576122508}) 2024-09-16T04:03:22.799 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.795+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2257986480}) 2024-09-16T04:03:22.799 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.795+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.799 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.795+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1576122508}) 2024-09-16T04:03:22.799 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.795+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3844957351}) 2024-09-16T04:03:22.800 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.795+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.800 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.795+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3301188791}) 2024-09-16T04:03:22.800 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.795+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1084774058}) 2024-09-16T04:03:22.800 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.795+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.800 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.795+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3301188791}) 2024-09-16T04:03:22.800 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.795+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2816108996}) 2024-09-16T04:03:22.802 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.799+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.802 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.799+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=187921245}) 2024-09-16T04:03:22.802 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.799+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=108743204}) 2024-09-16T04:03:22.802 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.799+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.802 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.799+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=187921245}) 2024-09-16T04:03:22.802 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.799+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2701679133}) 2024-09-16T04:03:22.803 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.799+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.803 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.799+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3832362965}) 2024-09-16T04:03:22.803 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.799+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1108316082}) 2024-09-16T04:03:22.803 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.799+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.804 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.799+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3832362965}) 2024-09-16T04:03:22.804 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.799+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=650126230}) 2024-09-16T04:03:22.805 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.799+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.805 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.799+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2439014964}) 2024-09-16T04:03:22.805 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.799+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2550840027}) 2024-09-16T04:03:22.805 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.799+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.805 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.799+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2439014964}) 2024-09-16T04:03:22.805 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.803+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2288266769}) 2024-09-16T04:03:22.807 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.803+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.807 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.803+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1743962020}) 2024-09-16T04:03:22.807 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.803+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3334374129}) 2024-09-16T04:03:22.808 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.803+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.808 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.803+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1743962020}) 2024-09-16T04:03:22.808 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.803+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1578551020}) 2024-09-16T04:03:22.809 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.803+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.809 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.803+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2922715416}) 2024-09-16T04:03:22.809 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.803+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3371080107}) 2024-09-16T04:03:22.809 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.803+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.809 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.803+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2922715416}) 2024-09-16T04:03:22.810 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.803+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1182426752}) 2024-09-16T04:03:22.810 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.807+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.811 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.807+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3841015551}) 2024-09-16T04:03:22.811 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.807+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1947146861}) 2024-09-16T04:03:22.811 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.807+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.811 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.807+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3841015551}) 2024-09-16T04:03:22.811 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.807+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2878907045}) 2024-09-16T04:03:22.812 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.807+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.812 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.807+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=318455284}) 2024-09-16T04:03:22.812 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.807+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1109670132}) 2024-09-16T04:03:22.812 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.807+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.813 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.807+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=318455284}) 2024-09-16T04:03:22.813 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.807+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3393650645}) 2024-09-16T04:03:22.814 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.811+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.814 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.811+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=55228100}) 2024-09-16T04:03:22.814 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.811+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=556982511}) 2024-09-16T04:03:22.814 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.811+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.814 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.811+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=55228100}) 2024-09-16T04:03:22.814 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.811+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2729516689}) 2024-09-16T04:03:22.816 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.811+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.816 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.811+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3233605350}) 2024-09-16T04:03:22.816 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.811+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=507791583}) 2024-09-16T04:03:22.816 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.811+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.816 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.811+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3233605350}) 2024-09-16T04:03:22.816 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.811+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3472191059}) 2024-09-16T04:03:22.818 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.815+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.818 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.815+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=133178111}) 2024-09-16T04:03:22.818 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.815+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=253273044}) 2024-09-16T04:03:22.818 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.815+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.818 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.815+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=133178111}) 2024-09-16T04:03:22.818 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.815+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=705425723}) 2024-09-16T04:03:22.820 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.815+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.820 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.815+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3891195568}) 2024-09-16T04:03:22.820 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.815+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=551268567}) 2024-09-16T04:03:22.820 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.815+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.820 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.815+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3891195568}) 2024-09-16T04:03:22.820 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.815+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3828680531}) 2024-09-16T04:03:22.822 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.819+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.822 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.819+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2338599618}) 2024-09-16T04:03:22.822 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.819+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3691697092}) 2024-09-16T04:03:22.822 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.819+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.822 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.819+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2338599618}) 2024-09-16T04:03:22.822 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.819+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3644314095}) 2024-09-16T04:03:22.824 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.819+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.824 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.819+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1991652387}) 2024-09-16T04:03:22.824 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.819+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1397444057}) 2024-09-16T04:03:22.824 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.819+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.824 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.819+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1991652387}) 2024-09-16T04:03:22.824 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.819+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3071867735}) 2024-09-16T04:03:22.825 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.823+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.826 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.823+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3008114014}) 2024-09-16T04:03:22.826 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.823+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1113868803}) 2024-09-16T04:03:22.826 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.823+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.826 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.823+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3008114014}) 2024-09-16T04:03:22.826 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.823+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2703406640}) 2024-09-16T04:03:22.827 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.823+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.827 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.823+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2430524174}) 2024-09-16T04:03:22.827 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.823+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1307643234}) 2024-09-16T04:03:22.827 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.823+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.827 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.823+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2430524174}) 2024-09-16T04:03:22.827 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.823+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1130869749}) 2024-09-16T04:03:22.828 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.823+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.828 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.823+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2512124009}) 2024-09-16T04:03:22.828 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.823+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4280682264}) 2024-09-16T04:03:22.828 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.823+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.828 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.823+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2512124009}) 2024-09-16T04:03:22.828 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.823+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=701398889}) 2024-09-16T04:03:22.829 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.823+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.829 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.823+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2770266670}) 2024-09-16T04:03:22.829 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.823+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2595272538}) 2024-09-16T04:03:22.829 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.823+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.829 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.823+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2770266670}) 2024-09-16T04:03:22.829 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.827+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3837012979}) 2024-09-16T04:03:22.832 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.827+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.832 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.827+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=46,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=8} crc {logm=94017003,mdsmap=4088524775,mgr=460109776,mgr_command_descs=2107673600,mgr_metadata=1367272202,mgrstat=3081966865,mon_config_key=85057112,monmap=3937513981,osd_metadata=4182959540,osd_pg_creating=209625309,osd_snap=2544150413}) 2024-09-16T04:03:22.832 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.827+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=46,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=9} crc {logm=1201597618,mdsmap=4088524775,mgr=460109776,mgr_command_descs=2107673600,mgr_metadata=1367272202,mgrstat=3925298562,mon_config_key=85057112,monmap=3937513981,osd_metadata=4182959540,osd_pg_creating=209625309,osd_snap=1840073672}) 2024-09-16T04:03:22.832 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.827+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.832 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.827+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=46,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=8} crc {logm=94017003,mdsmap=4088524775,mgr=460109776,mgr_command_descs=2107673600,mgr_metadata=1367272202,mgrstat=3081966865,mon_config_key=85057112,monmap=3937513981,osd_metadata=4182959540,osd_pg_creating=209625309,osd_snap=2544150413}) 2024-09-16T04:03:22.832 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.827+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=45,mdsmap=3,mgr=7,mgr_metadata=1,mgrstat=12,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=11} crc {logm=1648101466,mdsmap=4088524775,mgr=911650295,mgr_metadata=1367272202,mgrstat=3081966865,mon_config_key=1896270695,monmap=3937513981,osd_metadata=4182959540,osd_pg_creating=209625309,osd_snap=179356103}) 2024-09-16T04:03:22.833 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.831+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.834 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.831+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=29,osdmap=71} crc {osd_snap=3390229901,osdmap=1017725179}) 2024-09-16T04:03:22.834 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.831+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=31,osdmap=69} crc {osd_snap=974893570,osdmap=1380471646}) 2024-09-16T04:03:22.834 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.831+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.834 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.831+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=29,osdmap=71} crc {osd_snap=3390229901,osdmap=1017725179}) 2024-09-16T04:03:22.834 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.831+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=32,osdmap=68} crc {osd_snap=1991768330,osdmap=1834836678}) 2024-09-16T04:03:22.835 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.831+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.835 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.831+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2692741420}) 2024-09-16T04:03:22.836 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.831+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2177369455}) 2024-09-16T04:03:22.836 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.831+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.836 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.831+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2692741420}) 2024-09-16T04:03:22.836 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.831+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2866900605}) 2024-09-16T04:03:22.839 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.835+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.839 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.835+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2345387067}) 2024-09-16T04:03:22.839 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.835+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3492303168}) 2024-09-16T04:03:22.839 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.835+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.839 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.835+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2345387067}) 2024-09-16T04:03:22.839 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.835+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3628706881}) 2024-09-16T04:03:22.850 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.847+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.850 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.847+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=32} crc {osdmap=779538899}) 2024-09-16T04:03:22.850 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.847+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=32} crc {osdmap=2362310059}) 2024-09-16T04:03:22.850 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.847+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:03:22.850 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.847+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=32} crc {osdmap=779538899}) 2024-09-16T04:03:22.850 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:03:22.847+0000 7efd60e61640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=34} crc {osdmap=1443749458}) 2024-09-16T04:03:23.525 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T04:03:23.525 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:03:23.655 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.651+0000 7f4e105ff640 10 monclient: get_monmap_and_config 2024-09-16T04:03:23.655 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.651+0000 7f4e105ff640 10 monclient: build_initial_monmap 2024-09-16T04:03:23.655 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.651+0000 7f4e105ff640 10 monclient: monmap: 2024-09-16T04:03:23.656 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:03:23.656 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:03:23.656 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:03:23.654068+0000 2024-09-16T04:03:23.656 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:03:23.654068+0000 2024-09-16T04:03:23.656 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:03:23.656 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:03:23.656 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:03:23.656 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:03:23.656 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:03:23.656 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:03:23.656 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:03:23.656 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:03:23.659 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.651+0000 7f4e105ff640 1 Processor -- start 2024-09-16T04:03:23.659 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.651+0000 7f4e105ff640 1 -- start start 2024-09-16T04:03:23.659 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.651+0000 7f4e105ff640 10 monclient: init 2024-09-16T04:03:23.659 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.651+0000 7f4e105ff640 10 monclient: _reopen_session rank -1 2024-09-16T04:03:23.659 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.651+0000 7f4e105ff640 10 monclient: _add_conns ranks=[4,1,2,0,3] 2024-09-16T04:03:23.659 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.651+0000 7f4e105ff640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f4e080a39b0 0x7f4e08133b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:03:23.659 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.651+0000 7f4e105ff640 10 monclient(hunting): picked mon.noname-c con 0x7f4e080a39b0 addr v2:172.21.15.154:3300/0 2024-09-16T04:03:23.659 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.651+0000 7f4e105ff640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4e080a3d90 addr v1:172.21.15.154:6789/0 2024-09-16T04:03:23.659 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.651+0000 7f4e105ff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4e08137710 0x7f4e0813f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:03:23.660 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.651+0000 7f4e105ff640 10 monclient(hunting): picked mon.noname-b con 0x7f4e08137710 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:03:23.660 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.651+0000 7f4e105ff640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4e0809c140 con 0x7f4e080a3d90 2024-09-16T04:03:23.660 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.651+0000 7f4e105ff640 10 monclient(hunting): start opening mon connection 2024-09-16T04:03:23.660 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.651+0000 7f4e105ff640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f4e0809c420 con 0x7f4e080a39b0 2024-09-16T04:03:23.660 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.651+0000 7f4e105ff640 10 monclient(hunting): start opening mon connection 2024-09-16T04:03:23.660 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.651+0000 7f4e105ff640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f4e0806dd60 con 0x7f4e08137710 2024-09-16T04:03:23.660 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.651+0000 7f4e105ff640 10 monclient(hunting): _renew_subs 2024-09-16T04:03:23.660 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.651+0000 7f4e105ff640 10 monclient(hunting): authenticate will time out at 2250.623609s 2024-09-16T04:03:23.660 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.651+0000 7f4e0eb9c640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4e08137710 0x7f4e0813f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:03:23.660 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.651+0000 7f4e0eb9c640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4e08137710 0x7f4e0813f3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:52498/0 (socket says 172.21.15.42:52498) 2024-09-16T04:03:23.660 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.651+0000 7f4e0eb9c640 1 -- 172.21.15.42:0/653316154 learned_addr learned my addr 172.21.15.42:0/653316154 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:03:23.660 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.651+0000 7f4e0eb9c640 10 monclient(hunting): get_auth_request con 0x7f4e08137710 auth_method 0 2024-09-16T04:03:23.660 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.651+0000 7f4e0eb9c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:03:23.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.651+0000 7f4e0eb9c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:03:23.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.651+0000 7f4e0eb9c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:03:23.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.651+0000 7f4e0eb9c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:03:23.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.651+0000 7f4e0eb9c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:03:23.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.655+0000 7f4e0eb9c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:03:23.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.655+0000 7f4e0e39b640 1 --2- 172.21.15.42:0/653316154 >> v2:172.21.15.154:3300/0 conn(0x7f4e080a39b0 0x7f4e08133b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:03:23.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.655+0000 7f4e0e39b640 10 monclient(hunting): get_auth_request con 0x7f4e080a39b0 auth_method 0 2024-09-16T04:03:23.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.655+0000 7f4e0e39b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:03:23.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.655+0000 7f4e0e39b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:03:23.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.655+0000 7f4e0e39b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:03:23.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.655+0000 7f4e0eb9c640 10 monclient(hunting): handle_auth_done global_id 974114 payload 274 2024-09-16T04:03:23.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.655+0000 7f4e0eb9c640 1 -- 172.21.15.42:0/653316154 >> v2:172.21.15.154:3300/0 conn(0x7f4e080a39b0 msgr2=0x7f4e08133b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:03:23.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.655+0000 7f4e0eb9c640 1 --2- 172.21.15.42:0/653316154 >> v2:172.21.15.154:3300/0 conn(0x7f4e080a39b0 0x7f4e08133b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:03:23.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.655+0000 7f4e0eb9c640 1 -- 172.21.15.42:0/653316154 >> v1:172.21.15.154:6789/0 conn(0x7f4e080a3d90 legacy=0x7f4e08137000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:03:23.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.655+0000 7f4e0eb9c640 10 monclient: _finish_hunting 0 2024-09-16T04:03:23.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.655+0000 7f4e0eb9c640 1 monclient: found mon.noname-b 2024-09-16T04:03:23.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.655+0000 7f4e0eb9c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:03:23.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.655+0000 7f4e0eb9c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T04:03:23.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.655+0000 7f4e0eb9c640 1 -- 172.21.15.42:0/653316154 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e080a5e20 con 0x7f4e08137710 2024-09-16T04:03:23.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.655+0000 7f4e0eb9c640 1 --2- 172.21.15.42:0/653316154 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4e08137710 0x7f4e0813f3a0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f4e00009e50 tx=0x7f4e0001c8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b1ecaf858c3ed0c9 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:03:23.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.655+0000 7f4e0d399640 1 -- 172.21.15.42:0/653316154 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e000a5530 con 0x7f4e08137710 2024-09-16T04:03:23.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.655+0000 7f4e0d399640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:03:23.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.655+0000 7f4e0d399640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:03:23.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.655+0000 7f4e0d399640 10 monclient: dump: 2024-09-16T04:03:23.668 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:03:23.668 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:03:23.669 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:23.669 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:23.669 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:03:23.669 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:03:23.669 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:03:23.669 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:03:23.669 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:03:23.669 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:03:23.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.655+0000 7f4e0d399640 10 monclient: _finish_auth 0 2024-09-16T04:03:23.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.655+0000 7f4e0d399640 10 monclient: _check_auth_tickets 2024-09-16T04:03:23.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.655+0000 7f4e0d399640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:03:23.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.655+0000 7f4e0d399640 1 -- 172.21.15.42:0/653316154 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e000a56d0 con 0x7f4e08137710 2024-09-16T04:03:23.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.655+0000 7f4e0d399640 10 monclient: handle_config config(0 keys) 2024-09-16T04:03:23.670 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.655+0000 7f4e0d399640 1 -- 172.21.15.42:0/653316154 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e000a5aa0 con 0x7f4e08137710 2024-09-16T04:03:23.670 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.655+0000 7f4e0d399640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:03:23.670 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.655+0000 7f4e0d399640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:03:23.670 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.655+0000 7f4e0d399640 10 monclient: dump: 2024-09-16T04:03:23.670 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:03:23.670 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:03:23.670 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:23.670 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:23.670 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:03:23.670 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:03:23.670 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:03:23.670 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:03:23.670 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:03:23.670 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:03:23.670 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.655+0000 7f4e105ff640 5 monclient: authenticate success, global_id 974114 2024-09-16T04:03:23.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.655+0000 7f4e105ff640 10 monclient: get_monmap_and_config success 2024-09-16T04:03:23.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.655+0000 7f4e105ff640 4 set_mon_vals no callback set 2024-09-16T04:03:23.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.655+0000 7f4e105ff640 10 monclient: shutdown 2024-09-16T04:03:23.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.655+0000 7f4e105ff640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:03:23.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.655+0000 7f4e105ff640 1 -- 172.21.15.42:0/653316154 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4e08137710 msgr2=0x7f4e0813f3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:03:23.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.655+0000 7f4e105ff640 1 --2- 172.21.15.42:0/653316154 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4e08137710 0x7f4e0813f3a0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f4e00009e50 tx=0x7f4e0001c8b0 comp rx=0 tx=0).stop 2024-09-16T04:03:23.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.655+0000 7f4e105ff640 1 -- 172.21.15.42:0/653316154 shutdown_connections 2024-09-16T04:03:23.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.655+0000 7f4e105ff640 1 --2- 172.21.15.42:0/653316154 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4e08137710 0x7f4e0813f3a0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:03:23.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.655+0000 7f4e105ff640 1 --2- 172.21.15.42:0/653316154 >> v2:172.21.15.154:3300/0 conn(0x7f4e080a39b0 0x7f4e08133b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:03:23.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.655+0000 7f4e105ff640 1 -- 172.21.15.42:0/653316154 >> 172.21.15.42:0/653316154 conn(0x7f4e080a9fb0 msgr2=0x7f4e080aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:03:23.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.655+0000 7f4e105ff640 1 -- 172.21.15.42:0/653316154 shutdown_connections 2024-09-16T04:03:23.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.655+0000 7f4e105ff640 1 -- 172.21.15.42:0/653316154 wait complete. 2024-09-16T04:03:23.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.655+0000 7f4e105ff640 10 monclient: build_initial_monmap 2024-09-16T04:03:23.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.655+0000 7f4e105ff640 10 monclient: monmap: 2024-09-16T04:03:23.672 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:03:23.672 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:03:23.672 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:03:23.672 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:03:23.672 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:03:23.672 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:03:23.672 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:03:23.672 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:03:23.672 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:03:23.672 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:03:23.672 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.659+0000 7f4e105ff640 1 librados: starting msgr at 2024-09-16T04:03:23.672 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.659+0000 7f4e105ff640 1 librados: starting objecter 2024-09-16T04:03:23.672 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.659+0000 7f4e105ff640 1 Processor -- start 2024-09-16T04:03:23.672 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.659+0000 7f4e105ff640 1 -- start start 2024-09-16T04:03:23.672 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.659+0000 7f4e105ff640 1 librados: setting wanted keys 2024-09-16T04:03:23.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.659+0000 7f4e105ff640 1 librados: calling monclient init 2024-09-16T04:03:23.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.659+0000 7f4e105ff640 10 monclient: init 2024-09-16T04:03:23.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.659+0000 7f4e105ff640 10 monclient: _reopen_session rank -1 2024-09-16T04:03:23.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.659+0000 7f4e105ff640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:03:23.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.659+0000 7f4e105ff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4e080a39b0 0x7f4e081cf620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:03:23.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.659+0000 7f4e105ff640 10 monclient(hunting): picked mon.noname-c con 0x7f4e080a39b0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:03:23.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.659+0000 7f4e105ff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4e080a3d90 0x7f4e0813e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:03:23.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.659+0000 7f4e105ff640 10 monclient(hunting): picked mon.noname-b con 0x7f4e080a3d90 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:03:23.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.659+0000 7f4e105ff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4e08137710 0x7f4e0813eb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:03:23.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.659+0000 7f4e105ff640 10 monclient(hunting): picked mon.noname-a con 0x7f4e08137710 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:03:23.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.659+0000 7f4e105ff640 10 monclient(hunting): start opening mon connection 2024-09-16T04:03:23.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.659+0000 7f4e105ff640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f4e08142310 con 0x7f4e08137710 2024-09-16T04:03:23.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.659+0000 7f4e105ff640 10 monclient(hunting): start opening mon connection 2024-09-16T04:03:23.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.659+0000 7f4e105ff640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f4e08142490 con 0x7f4e080a3d90 2024-09-16T04:03:23.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.659+0000 7f4e105ff640 10 monclient(hunting): start opening mon connection 2024-09-16T04:03:23.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.659+0000 7f4e105ff640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f4e08142610 con 0x7f4e080a39b0 2024-09-16T04:03:23.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.659+0000 7f4e105ff640 10 monclient(hunting): _renew_subs 2024-09-16T04:03:23.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.659+0000 7f4e105ff640 10 monclient(hunting): authenticate will time out at 2250.628879s 2024-09-16T04:03:23.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.659+0000 7f4e0eb9c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4e08137710 0x7f4e0813eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:03:23.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.659+0000 7f4e0eb9c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4e08137710 0x7f4e0813eb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:42456/0 (socket says 172.21.15.42:42456) 2024-09-16T04:03:23.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.659+0000 7f4e0e39b640 1 --2- 172.21.15.42:0/987660882 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4e080a39b0 0x7f4e081cf620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:03:23.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.659+0000 7f4e0eb9c640 1 -- 172.21.15.42:0/987660882 learned_addr learned my addr 172.21.15.42:0/987660882 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:03:23.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.659+0000 7f4e0eb9c640 10 monclient(hunting): get_auth_request con 0x7f4e08137710 auth_method 0 2024-09-16T04:03:23.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.659+0000 7f4e0eb9c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:03:23.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.659+0000 7f4e0eb9c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:03:23.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.659+0000 7f4e0eb9c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:03:23.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.659+0000 7f4e0e39b640 10 monclient(hunting): get_auth_request con 0x7f4e080a39b0 auth_method 0 2024-09-16T04:03:23.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.659+0000 7f4e0e39b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:03:23.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.659+0000 7f4e0e39b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:03:23.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.659+0000 7f4e0e39b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:03:23.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.659+0000 7f4e0eb9c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:03:23.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.659+0000 7f4e0eb9c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:03:23.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.659+0000 7f4e0eb9c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:03:23.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.659+0000 7f4e0e39b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:03:23.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.659+0000 7f4e0e39b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:03:23.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.659+0000 7f4e0e39b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:03:23.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.659+0000 7f4e0db9a640 1 --2- 172.21.15.42:0/987660882 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4e080a3d90 0x7f4e0813e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:03:23.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.659+0000 7f4e0db9a640 10 monclient(hunting): get_auth_request con 0x7f4e080a3d90 auth_method 0 2024-09-16T04:03:23.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.659+0000 7f4e0db9a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:03:23.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.659+0000 7f4e0db9a640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:03:23.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.659+0000 7f4e0db9a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:03:23.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.659+0000 7f4e0e39b640 10 monclient(hunting): handle_auth_done global_id 974117 payload 824 2024-09-16T04:03:23.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.659+0000 7f4e0e39b640 1 -- 172.21.15.42:0/987660882 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4e080a3d90 msgr2=0x7f4e0813e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:03:23.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.659+0000 7f4e0e39b640 1 --2- 172.21.15.42:0/987660882 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4e080a3d90 0x7f4e0813e5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:03:23.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.659+0000 7f4e0e39b640 1 -- 172.21.15.42:0/987660882 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4e08137710 msgr2=0x7f4e0813eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:03:23.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.659+0000 7f4e0e39b640 1 --2- 172.21.15.42:0/987660882 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4e08137710 0x7f4e0813eb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:03:23.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.659+0000 7f4e0e39b640 10 monclient: _finish_hunting 0 2024-09-16T04:03:23.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.659+0000 7f4e0e39b640 1 monclient: found mon.noname-c 2024-09-16T04:03:23.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.659+0000 7f4e0e39b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:03:23.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.659+0000 7f4e0e39b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T04:03:23.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.659+0000 7f4e0e39b640 1 -- 172.21.15.42:0/987660882 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e081cfb60 con 0x7f4e080a39b0 2024-09-16T04:03:23.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.659+0000 7f4e0eb9c640 1 --2- 172.21.15.42:0/987660882 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4e08137710 0x7f4e0813eb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:03:23.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.659+0000 7f4e0e39b640 1 --2- 172.21.15.42:0/987660882 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4e080a39b0 0x7f4e081cf620 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f4e0400d5f0 tx=0x7f4e0400dab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:03:23.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.659+0000 7f4dfeffd640 10 client.?.objecter ms_handle_connect 0x7f4e080a39b0 2024-09-16T04:03:23.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.659+0000 7f4dfeffd640 10 client.?.objecter resend_mon_ops 2024-09-16T04:03:23.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.659+0000 7f4dfeffd640 1 -- 172.21.15.42:0/987660882 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e040125f0 con 0x7f4e080a39b0 2024-09-16T04:03:23.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.659+0000 7f4dfeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:03:23.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.659+0000 7f4dfeffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:03:23.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.659+0000 7f4dfeffd640 10 monclient: dump: 2024-09-16T04:03:23.677 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:03:23.677 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:03:23.677 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:23.677 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:23.677 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:03:23.677 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:03:23.677 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:03:23.677 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:03:23.677 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:03:23.677 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:03:23.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.659+0000 7f4dfeffd640 10 monclient: _finish_auth 0 2024-09-16T04:03:23.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.659+0000 7f4dfeffd640 10 monclient: _check_auth_tickets 2024-09-16T04:03:23.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.659+0000 7f4dfeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:03:23.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.659+0000 7f4dfeffd640 1 -- 172.21.15.42:0/987660882 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e04019070 con 0x7f4e080a39b0 2024-09-16T04:03:23.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.659+0000 7f4dfeffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:03:23.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.659+0000 7f4dfeffd640 1 -- 172.21.15.42:0/987660882 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e040129b0 con 0x7f4e080a39b0 2024-09-16T04:03:23.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.659+0000 7f4e105ff640 5 monclient: authenticate success, global_id 974117 2024-09-16T04:03:23.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.659+0000 7f4e105ff640 10 monclient: _renew_subs 2024-09-16T04:03:23.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.659+0000 7f4e105ff640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:03:23.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.659+0000 7f4e105ff640 1 -- 172.21.15.42:0/987660882 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e0813a080 con 0x7f4e080a39b0 2024-09-16T04:03:23.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.659+0000 7f4dfffff640 4 set_mon_vals no callback set 2024-09-16T04:03:23.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.659+0000 7f4dfeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:03:23.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.659+0000 7f4dfeffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:03:23.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.659+0000 7f4dfeffd640 10 monclient: dump: 2024-09-16T04:03:23.678 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:03:23.678 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:03:23.678 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:23.678 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:23.679 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:03:23.679 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:03:23.679 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:03:23.679 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:03:23.679 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:03:23.679 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:03:23.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.659+0000 7f4e105ff640 10 client.974117.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:03:23.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.659+0000 7f4e105ff640 10 monclient: _renew_subs 2024-09-16T04:03:23.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.659+0000 7f4e105ff640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:03:23.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.659+0000 7f4e105ff640 1 -- 172.21.15.42:0/987660882 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e08133240 con 0x7f4e080a39b0 2024-09-16T04:03:23.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.659+0000 7f4e105ff640 1 librados: init done 2024-09-16T04:03:23.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.659+0000 7f4dfeffd640 1 -- 172.21.15.42:0/987660882 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f4e04012b50 con 0x7f4e080a39b0 2024-09-16T04:03:23.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.659+0000 7f4e105ff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:03:23.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.659+0000 7f4e105ff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:03:23.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.659+0000 7f4dfeffd640 1 --2- 172.21.15.42:0/987660882 >> v2:172.21.15.154:6812/254845342 conn(0x7f4dec037810 0x7f4dec039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:03:23.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.659+0000 7f4e105ff640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:03:23.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.659+0000 7f4e105ff640 1 -- 172.21.15.42:0/987660882 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e08133b80 con 0x7f4e080a39b0 2024-09-16T04:03:23.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.659+0000 7f4dfeffd640 1 -- 172.21.15.42:0/987660882 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(653..653 src has 451..653) ==== 6272+0+0 (secure 0 0 0) 0x7f4e040494f0 con 0x7f4e080a39b0 2024-09-16T04:03:23.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.659+0000 7f4dfeffd640 10 client.974117.objecter ms_dispatch 0x7f4e08000ba0 osd_map(653..653 src has 451..653) 2024-09-16T04:03:23.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.659+0000 7f4dfeffd640 3 client.974117.objecter handle_osd_map got epochs [653,653] > 0 2024-09-16T04:03:23.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.659+0000 7f4dfeffd640 3 client.974117.objecter handle_osd_map decoding full epoch 653 2024-09-16T04:03:23.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.659+0000 7f4dfeffd640 20 client.974117.objecter dump_active .. 0 homeless 2024-09-16T04:03:23.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.659+0000 7f4e0db9a640 1 --2- 172.21.15.42:0/987660882 >> v2:172.21.15.154:6812/254845342 conn(0x7f4dec037810 0x7f4dec039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:03:23.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.659+0000 7f4e0db9a640 10 monclient: get_auth_request con 0x7f4dec037810 auth_method 0 2024-09-16T04:03:23.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.663+0000 7f4dfeffd640 10 client.974117.objecter ms_handle_connect 0x7f4dec037810 2024-09-16T04:03:23.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.663+0000 7f4e0db9a640 1 --2- 172.21.15.42:0/987660882 >> v2:172.21.15.154:6812/254845342 conn(0x7f4dec037810 0x7f4dec039cd0 secure :-1 s=READY pgs=3805 cs=0 l=1 rev1=1 crypto rx=0x7f4e080a3210 tx=0x7f4df8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:03:23.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.667+0000 7f4dfeffd640 1 -- 172.21.15.42:0/987660882 <== mon.2 v2:172.21.15.42:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f4e0401e460 con 0x7f4e080a39b0 2024-09-16T04:03:23.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.667+0000 7f4dfeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:03:23.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.667+0000 7f4dfeffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:03:23.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.919+0000 7f4e105ff640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:03:23.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.919+0000 7f4e105ff640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:03:23.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.919+0000 7f4e105ff640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:03:23.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.919+0000 7f4e105ff640 1 -- 172.21.15.42:0/987660882 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f4e080a5750 con 0x7f4e080a39b0 2024-09-16T04:03:23.923 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.919+0000 7f4dfeffd640 1 -- 172.21.15.42:0/987660882 <== mon.2 v2:172.21.15.42:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7f4e04046630 con 0x7f4e080a39b0 2024-09-16T04:03:23.923 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.919+0000 7f4dfeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:03:23.923 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.919+0000 7f4dfeffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:03:23.923 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":190,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:03:23.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.919+0000 7f4e105ff640 10 librados: watch_flush enter 2024-09-16T04:03:23.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.919+0000 7f4e105ff640 10 librados: watch_flush exit 2024-09-16T04:03:23.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.919+0000 7f4e105ff640 20 client.974117.objecter shutdown clearing up homeless session... 2024-09-16T04:03:23.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.919+0000 7f4e105ff640 10 client.974117.objecter successfully canceled tick 2024-09-16T04:03:23.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.919+0000 7f4e105ff640 1 -- 172.21.15.42:0/987660882 >> v2:172.21.15.154:6812/254845342 conn(0x7f4dec037810 msgr2=0x7f4dec039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:03:23.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.919+0000 7f4e105ff640 1 --2- 172.21.15.42:0/987660882 >> v2:172.21.15.154:6812/254845342 conn(0x7f4dec037810 0x7f4dec039cd0 secure :-1 s=READY pgs=3805 cs=0 l=1 rev1=1 crypto rx=0x7f4e080a3210 tx=0x7f4df8009040 comp rx=0 tx=0).stop 2024-09-16T04:03:23.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.919+0000 7f4e105ff640 10 monclient: shutdown 2024-09-16T04:03:23.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.919+0000 7f4e105ff640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:03:23.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.919+0000 7f4e105ff640 1 -- 172.21.15.42:0/987660882 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4e080a39b0 msgr2=0x7f4e081cf620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:03:23.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.919+0000 7f4e105ff640 1 --2- 172.21.15.42:0/987660882 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4e080a39b0 0x7f4e081cf620 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f4e0400d5f0 tx=0x7f4e0400dab0 comp rx=0 tx=0).stop 2024-09-16T04:03:23.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.923+0000 7f4e105ff640 1 -- 172.21.15.42:0/987660882 shutdown_connections 2024-09-16T04:03:23.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.923+0000 7f4e105ff640 1 --2- 172.21.15.42:0/987660882 >> v2:172.21.15.154:6812/254845342 conn(0x7f4dec037810 0x7f4dec039cd0 unknown :-1 s=CLOSED pgs=3805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:03:23.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.923+0000 7f4e105ff640 1 --2- 172.21.15.42:0/987660882 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4e08137710 0x7f4e0813eb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:03:23.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.923+0000 7f4e105ff640 1 --2- 172.21.15.42:0/987660882 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4e080a3d90 0x7f4e0813e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:03:23.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.923+0000 7f4e105ff640 1 --2- 172.21.15.42:0/987660882 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4e080a39b0 0x7f4e081cf620 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:03:23.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.923+0000 7f4e105ff640 1 -- 172.21.15.42:0/987660882 >> 172.21.15.42:0/987660882 conn(0x7f4e080a9fb0 msgr2=0x7f4e08131f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:03:23.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.923+0000 7f4e105ff640 1 -- 172.21.15.42:0/987660882 shutdown_connections 2024-09-16T04:03:23.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.923+0000 7f4e105ff640 1 -- 172.21.15.42:0/987660882 wait complete. 2024-09-16T04:03:23.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:23.923+0000 7f4e105ff640 1 librados: shutdown 2024-09-16T04:03:23.942 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T04:03:23.943 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T04:03:23.943 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T04:03:23.943 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T04:03:24.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.059+0000 7f6ba5453640 10 monclient: get_monmap_and_config 2024-09-16T04:03:24.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.059+0000 7f6ba5453640 10 monclient: build_initial_monmap 2024-09-16T04:03:24.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.059+0000 7f6ba5453640 10 monclient: monmap: 2024-09-16T04:03:24.062 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:03:24.062 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:03:24.062 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:03:24.060698+0000 2024-09-16T04:03:24.062 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:03:24.060698+0000 2024-09-16T04:03:24.062 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:03:24.062 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:03:24.063 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:03:24.063 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:03:24.063 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:03:24.063 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:03:24.063 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:03:24.063 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:03:24.065 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.059+0000 7f6ba5453640 1 Processor -- start 2024-09-16T04:03:24.065 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.059+0000 7f6ba5453640 1 -- start start 2024-09-16T04:03:24.065 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.059+0000 7f6ba5453640 10 monclient: init 2024-09-16T04:03:24.065 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.059+0000 7f6ba5453640 10 monclient: _reopen_session rank -1 2024-09-16T04:03:24.065 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.059+0000 7f6ba5453640 10 monclient: _add_conns ranks=[3,2,1,0,4] 2024-09-16T04:03:24.065 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.059+0000 7f6ba5453640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f6ba00a1330 0x7f6ba00a1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:03:24.065 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.059+0000 7f6ba5453640 10 monclient(hunting): picked mon.noname-a con 0x7f6ba00a1330 addr v2:172.21.15.42:3300/0 2024-09-16T04:03:24.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.059+0000 7f6ba5453640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f6ba006dac0 0x7f6ba00a1c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:03:24.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.059+0000 7f6ba5453640 10 monclient(hunting): picked mon.noname-b con 0x7f6ba006dac0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:03:24.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.059+0000 7f6ba5453640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6ba0135100 addr v1:172.21.15.154:6789/0 2024-09-16T04:03:24.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.059+0000 7f6ba5453640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6ba009c7d0 con 0x7f6ba0135100 2024-09-16T04:03:24.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.059+0000 7f6ba5453640 10 monclient(hunting): start opening mon connection 2024-09-16T04:03:24.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.059+0000 7f6ba5453640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f6ba009c090 con 0x7f6ba00a1330 2024-09-16T04:03:24.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.059+0000 7f6ba5453640 10 monclient(hunting): start opening mon connection 2024-09-16T04:03:24.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.059+0000 7f6ba5453640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f6ba009f610 con 0x7f6ba006dac0 2024-09-16T04:03:24.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.059+0000 7f6ba5453640 10 monclient(hunting): _renew_subs 2024-09-16T04:03:24.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.059+0000 7f6ba5453640 10 monclient(hunting): authenticate will time out at 2251.030366s 2024-09-16T04:03:24.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.059+0000 7f6b9e7fc640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f6ba006dac0 0x7f6ba00a1c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:03:24.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.059+0000 7f6b9effd640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f6ba00a1330 0x7f6ba00a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:03:24.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.059+0000 7f6b9e7fc640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f6ba006dac0 0x7f6ba00a1c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:60498/0 (socket says 172.21.15.42:60498) 2024-09-16T04:03:24.067 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.059+0000 7f6b9e7fc640 1 -- 172.21.15.42:0/1557241630 learned_addr learned my addr 172.21.15.42:0/1557241630 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:03:24.067 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.059+0000 7f6b9e7fc640 10 monclient(hunting): get_auth_request con 0x7f6ba006dac0 auth_method 0 2024-09-16T04:03:24.067 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.059+0000 7f6b9e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:03:24.067 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.059+0000 7f6b9e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:03:24.067 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.059+0000 7f6b9e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:03:24.067 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.059+0000 7f6b9effd640 1 --2- 172.21.15.42:0/1557241630 >> v2:172.21.15.42:3300/0 conn(0x7f6ba00a1330 0x7f6ba00a1710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:42504/0 (socket says 172.21.15.42:42504) 2024-09-16T04:03:24.067 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.059+0000 7f6b9effd640 10 monclient(hunting): get_auth_request con 0x7f6ba00a1330 auth_method 0 2024-09-16T04:03:24.067 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.059+0000 7f6b9effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:03:24.067 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.059+0000 7f6b9effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:03:24.067 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.059+0000 7f6b9effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:03:24.067 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.059+0000 7f6b9e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:03:24.067 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.059+0000 7f6b9e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:03:24.068 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.059+0000 7f6b9e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:03:24.068 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.059+0000 7f6b9effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:03:24.068 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.059+0000 7f6b9effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:03:24.068 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.059+0000 7f6b9effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:03:24.068 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.059+0000 7f6b9effd640 10 monclient(hunting): handle_auth_done global_id 825666 payload 274 2024-09-16T04:03:24.068 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.059+0000 7f6b9effd640 1 -- 172.21.15.42:0/1557241630 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f6ba006dac0 msgr2=0x7f6ba00a1c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:03:24.068 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.059+0000 7f6b9effd640 1 --2- 172.21.15.42:0/1557241630 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f6ba006dac0 0x7f6ba00a1c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:03:24.068 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.059+0000 7f6b9effd640 1 -- 172.21.15.42:0/1557241630 >> v1:172.21.15.154:6789/0 conn(0x7f6ba0135100 legacy=0x7f6ba0135550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:03:24.068 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.059+0000 7f6b9effd640 10 monclient: _finish_hunting 0 2024-09-16T04:03:24.068 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.059+0000 7f6b9effd640 1 monclient: found mon.noname-a 2024-09-16T04:03:24.068 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.059+0000 7f6b9effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:03:24.068 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.059+0000 7f6b9effd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:03:24.068 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.059+0000 7f6b9effd640 1 -- 172.21.15.42:0/1557241630 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ba009f8f0 con 0x7f6ba00a1330 2024-09-16T04:03:24.069 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.059+0000 7f6b9e7fc640 1 --2- 172.21.15.42:0/1557241630 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f6ba006dac0 0x7f6ba00a1c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:03:24.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.063+0000 7f6b9effd640 1 --2- 172.21.15.42:0/1557241630 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6ba00a1330 0x7f6ba00a1710 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f6b98009d90 tx=0x7f6b9801c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34c8b279f558beb5 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:03:24.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.063+0000 7f6b9dffb640 1 -- 172.21.15.42:0/1557241630 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b980b2070 con 0x7f6ba00a1330 2024-09-16T04:03:24.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.063+0000 7f6b9dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:03:24.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.063+0000 7f6b9dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:03:24.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.063+0000 7f6b9dffb640 10 monclient: dump: 2024-09-16T04:03:24.075 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:03:24.075 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:03:24.075 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:24.075 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:24.075 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:03:24.075 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:03:24.075 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:03:24.076 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:03:24.076 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:03:24.076 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:03:24.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.063+0000 7f6b9dffb640 10 monclient: _finish_auth 0 2024-09-16T04:03:24.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.063+0000 7f6b9dffb640 10 monclient: _check_auth_tickets 2024-09-16T04:03:24.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.063+0000 7f6b9dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:03:24.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.063+0000 7f6b9dffb640 1 -- 172.21.15.42:0/1557241630 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6b980a58a0 con 0x7f6ba00a1330 2024-09-16T04:03:24.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.063+0000 7f6b9dffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:03:24.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.063+0000 7f6b9dffb640 1 -- 172.21.15.42:0/1557241630 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b980a5c70 con 0x7f6ba00a1330 2024-09-16T04:03:24.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.063+0000 7f6ba5453640 5 monclient: authenticate success, global_id 825666 2024-09-16T04:03:24.077 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.063+0000 7f6ba5453640 10 monclient: get_monmap_and_config success 2024-09-16T04:03:24.077 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.063+0000 7f6ba5453640 4 set_mon_vals no callback set 2024-09-16T04:03:24.077 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.063+0000 7f6b9dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:03:24.077 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.063+0000 7f6b9dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:03:24.077 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.063+0000 7f6b9dffb640 10 monclient: dump: 2024-09-16T04:03:24.077 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:03:24.077 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:03:24.077 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:24.077 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:24.077 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:03:24.077 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:03:24.077 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:03:24.078 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:03:24.078 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:03:24.078 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:03:24.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.063+0000 7f6ba5453640 10 monclient: shutdown 2024-09-16T04:03:24.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.063+0000 7f6ba5453640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:03:24.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.063+0000 7f6ba5453640 1 -- 172.21.15.42:0/1557241630 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6ba00a1330 msgr2=0x7f6ba00a1710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:03:24.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.063+0000 7f6ba5453640 1 --2- 172.21.15.42:0/1557241630 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6ba00a1330 0x7f6ba00a1710 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f6b98009d90 tx=0x7f6b9801c910 comp rx=0 tx=0).stop 2024-09-16T04:03:24.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.063+0000 7f6ba5453640 1 -- 172.21.15.42:0/1557241630 shutdown_connections 2024-09-16T04:03:24.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.063+0000 7f6ba5453640 1 --2- 172.21.15.42:0/1557241630 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f6ba006dac0 0x7f6ba00a1c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:03:24.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.063+0000 7f6ba5453640 1 --2- 172.21.15.42:0/1557241630 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6ba00a1330 0x7f6ba00a1710 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:03:24.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.063+0000 7f6ba5453640 1 -- 172.21.15.42:0/1557241630 >> 172.21.15.42:0/1557241630 conn(0x7f6ba00a2420 msgr2=0x7f6ba00a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:03:24.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.063+0000 7f6ba5453640 1 -- 172.21.15.42:0/1557241630 shutdown_connections 2024-09-16T04:03:24.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.063+0000 7f6ba5453640 1 -- 172.21.15.42:0/1557241630 wait complete. 2024-09-16T04:03:24.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.063+0000 7f6ba5453640 10 monclient: build_initial_monmap 2024-09-16T04:03:24.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.063+0000 7f6ba5453640 10 monclient: monmap: 2024-09-16T04:03:24.079 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:03:24.079 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:03:24.079 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:03:24.079 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:03:24.079 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:03:24.079 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:03:24.079 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:03:24.079 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:03:24.079 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:03:24.080 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:03:24.080 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.063+0000 7f6ba5453640 1 librados: starting msgr at 2024-09-16T04:03:24.080 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.063+0000 7f6ba5453640 1 librados: starting objecter 2024-09-16T04:03:24.080 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.063+0000 7f6ba5453640 1 Processor -- start 2024-09-16T04:03:24.080 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.063+0000 7f6ba5453640 1 -- start start 2024-09-16T04:03:24.080 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.063+0000 7f6ba5453640 1 librados: setting wanted keys 2024-09-16T04:03:24.080 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.063+0000 7f6ba5453640 1 librados: calling monclient init 2024-09-16T04:03:24.080 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.063+0000 7f6ba5453640 10 monclient: init 2024-09-16T04:03:24.080 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.063+0000 7f6ba5453640 10 monclient: _reopen_session rank -1 2024-09-16T04:03:24.080 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.063+0000 7f6ba5453640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:03:24.080 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.063+0000 7f6ba5453640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6ba00a1330 0x7f6ba013b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:03:24.080 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.063+0000 7f6ba5453640 10 monclient(hunting): picked mon.noname-a con 0x7f6ba00a1330 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:03:24.080 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.063+0000 7f6ba5453640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f6ba0135100 0x7f6ba0134530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:03:24.081 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.063+0000 7f6ba5453640 10 monclient(hunting): picked mon.noname-c con 0x7f6ba0135100 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:03:24.081 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.063+0000 7f6ba5453640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6ba0134a70 0x7f6ba01cf330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:03:24.081 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.063+0000 7f6ba5453640 10 monclient(hunting): picked mon.noname-b con 0x7f6ba0134a70 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:03:24.081 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.063+0000 7f6ba5453640 10 monclient(hunting): start opening mon connection 2024-09-16T04:03:24.081 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.063+0000 7f6ba5453640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f6ba013fd00 con 0x7f6ba00a1330 2024-09-16T04:03:24.081 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.063+0000 7f6ba5453640 10 monclient(hunting): start opening mon connection 2024-09-16T04:03:24.081 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.063+0000 7f6ba5453640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f6ba013fb80 con 0x7f6ba0134a70 2024-09-16T04:03:24.081 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.063+0000 7f6ba5453640 10 monclient(hunting): start opening mon connection 2024-09-16T04:03:24.081 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.063+0000 7f6ba5453640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f6ba013fe80 con 0x7f6ba0135100 2024-09-16T04:03:24.081 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.063+0000 7f6ba5453640 10 monclient(hunting): _renew_subs 2024-09-16T04:03:24.081 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.063+0000 7f6ba5453640 10 monclient(hunting): authenticate will time out at 2251.035881s 2024-09-16T04:03:24.081 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.067+0000 7f6b9e7fc640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f6ba0135100 0x7f6ba0134530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:03:24.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.067+0000 7f6b9effd640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6ba00a1330 0x7f6ba013b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:03:24.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.067+0000 7f6b9e7fc640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f6ba0135100 0x7f6ba0134530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:60512/0 (socket says 172.21.15.42:60512) 2024-09-16T04:03:24.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.067+0000 7f6b9e7fc640 1 -- 172.21.15.42:0/2181017677 learned_addr learned my addr 172.21.15.42:0/2181017677 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:03:24.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.067+0000 7f6b9e7fc640 10 monclient(hunting): get_auth_request con 0x7f6ba0135100 auth_method 0 2024-09-16T04:03:24.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.067+0000 7f6b9e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:03:24.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.067+0000 7f6b9e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:03:24.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.067+0000 7f6b9e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:03:24.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.067+0000 7f6b9effd640 10 monclient(hunting): get_auth_request con 0x7f6ba00a1330 auth_method 0 2024-09-16T04:03:24.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.067+0000 7f6b9effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:03:24.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.067+0000 7f6b9effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:03:24.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.067+0000 7f6b9effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:03:24.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.067+0000 7f6b9effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:03:24.083 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.067+0000 7f6b9effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:03:24.083 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.067+0000 7f6b9effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:03:24.083 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.067+0000 7f6b9e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:03:24.083 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.067+0000 7f6b9e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:03:24.083 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.067+0000 7f6b9e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:03:24.083 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.067+0000 7f6b9f7fe640 1 --2- 172.21.15.42:0/2181017677 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6ba0134a70 0x7f6ba01cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:03:24.083 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.067+0000 7f6b9effd640 10 monclient(hunting): handle_auth_done global_id 825669 payload 824 2024-09-16T04:03:24.083 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.067+0000 7f6b9effd640 1 -- 172.21.15.42:0/2181017677 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f6ba0135100 msgr2=0x7f6ba0134530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:03:24.083 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.067+0000 7f6b9effd640 1 --2- 172.21.15.42:0/2181017677 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f6ba0135100 0x7f6ba0134530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:03:24.083 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.067+0000 7f6b9effd640 1 -- 172.21.15.42:0/2181017677 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6ba0134a70 msgr2=0x7f6ba01cf330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:03:24.083 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.067+0000 7f6b9effd640 1 --2- 172.21.15.42:0/2181017677 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6ba0134a70 0x7f6ba01cf330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:03:24.083 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.067+0000 7f6b9effd640 10 monclient: _finish_hunting 0 2024-09-16T04:03:24.083 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.067+0000 7f6b9effd640 1 monclient: found mon.noname-a 2024-09-16T04:03:24.083 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.067+0000 7f6b9effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:03:24.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.067+0000 7f6b9effd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:03:24.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.067+0000 7f6b9effd640 1 -- 172.21.15.42:0/2181017677 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ba0120c00 con 0x7f6ba00a1330 2024-09-16T04:03:24.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.067+0000 7f6b9f7fe640 10 monclient: get_auth_request con 0x7f6ba0134a70 auth_method 0 2024-09-16T04:03:24.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.067+0000 7f6b9f7fe640 1 --2- 172.21.15.42:0/2181017677 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6ba0134a70 0x7f6ba01cf330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T04:03:24.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.067+0000 7f6b9e7fc640 1 --2- 172.21.15.42:0/2181017677 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f6ba0135100 0x7f6ba0134530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:03:24.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.067+0000 7f6b9effd640 1 --2- 172.21.15.42:0/2181017677 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6ba00a1330 0x7f6ba013b170 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f6b980ac000 tx=0x7f6ba00a22d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:03:24.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.067+0000 7f6b977fe640 10 client.?.objecter ms_handle_connect 0x7f6ba00a1330 2024-09-16T04:03:24.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.067+0000 7f6b977fe640 10 client.?.objecter resend_mon_ops 2024-09-16T04:03:24.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.067+0000 7f6b977fe640 1 -- 172.21.15.42:0/2181017677 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b980bc030 con 0x7f6ba00a1330 2024-09-16T04:03:24.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.067+0000 7f6b977fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:03:24.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.067+0000 7f6b977fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:03:24.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.067+0000 7f6b977fe640 10 monclient: dump: 2024-09-16T04:03:24.084 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:03:24.085 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:03:24.085 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:24.085 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:24.085 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:03:24.085 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:03:24.085 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:03:24.085 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:03:24.085 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:03:24.085 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:03:24.085 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.067+0000 7f6b977fe640 10 monclient: _finish_auth 0 2024-09-16T04:03:24.085 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.067+0000 7f6b977fe640 10 monclient: _check_auth_tickets 2024-09-16T04:03:24.085 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.067+0000 7f6b977fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:03:24.085 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.067+0000 7f6ba5453640 5 monclient: authenticate success, global_id 825669 2024-09-16T04:03:24.085 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.067+0000 7f6b977fe640 1 -- 172.21.15.42:0/2181017677 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6b980ad440 con 0x7f6ba00a1330 2024-09-16T04:03:24.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.067+0000 7f6b977fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:03:24.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.067+0000 7f6b977fe640 1 -- 172.21.15.42:0/2181017677 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b980b2040 con 0x7f6ba00a1330 2024-09-16T04:03:24.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.067+0000 7f6ba5453640 10 monclient: _renew_subs 2024-09-16T04:03:24.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.067+0000 7f6ba5453640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:03:24.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.067+0000 7f6ba5453640 1 -- 172.21.15.42:0/2181017677 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ba009dcf0 con 0x7f6ba00a1330 2024-09-16T04:03:24.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.067+0000 7f6b977fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:03:24.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.067+0000 7f6b977fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:03:24.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.067+0000 7f6b977fe640 10 monclient: dump: 2024-09-16T04:03:24.086 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:03:24.086 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:03:24.086 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:24.086 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:24.086 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:03:24.086 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:03:24.087 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:03:24.087 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:03:24.087 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:03:24.087 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:03:24.087 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.067+0000 7f6b9cff9640 4 set_mon_vals no callback set 2024-09-16T04:03:24.087 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.067+0000 7f6ba5453640 10 client.825669.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:03:24.087 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.067+0000 7f6ba5453640 10 monclient: _renew_subs 2024-09-16T04:03:24.087 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.067+0000 7f6ba5453640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:03:24.087 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.067+0000 7f6ba5453640 1 -- 172.21.15.42:0/2181017677 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ba00a5110 con 0x7f6ba00a1330 2024-09-16T04:03:24.087 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.067+0000 7f6ba5453640 1 librados: init done 2024-09-16T04:03:24.087 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.067+0000 7f6ba5453640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:03:24.087 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.067+0000 7f6ba5453640 1 --2- 172.21.15.42:0/2181017677 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6ba006dac0 0x7f6ba002c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:03:24.087 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.067+0000 7f6ba5453640 10 monclient(hunting): start opening mon connection 2024-09-16T04:03:24.087 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.067+0000 7f6ba5453640 1 -- 172.21.15.42:0/2181017677 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f6ba0134e80 con 0x7f6ba006dac0 2024-09-16T04:03:24.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.067+0000 7f6b977fe640 1 -- 172.21.15.42:0/2181017677 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f6b980bf070 con 0x7f6ba00a1330 2024-09-16T04:03:24.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.067+0000 7f6b9e7fc640 1 --2- 172.21.15.42:0/2181017677 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6ba006dac0 0x7f6ba002c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:03:24.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.067+0000 7f6b9e7fc640 10 monclient: get_auth_request con 0x7f6ba006dac0 auth_method 0 2024-09-16T04:03:24.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.067+0000 7f6b9e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:03:24.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.067+0000 7f6b9e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:03:24.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.067+0000 7f6b9e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:03:24.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.067+0000 7f6b9e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:03:24.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.067+0000 7f6b9e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:03:24.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.067+0000 7f6b9e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:03:24.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.067+0000 7f6b977fe640 1 --2- 172.21.15.42:0/2181017677 >> v2:172.21.15.154:6812/254845342 conn(0x7f6b80037810 0x7f6b80039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:03:24.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.067+0000 7f6b9e7fc640 10 monclient(hunting): handle_auth_done global_id 825672 payload 824 2024-09-16T04:03:24.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.067+0000 7f6b977fe640 1 -- 172.21.15.42:0/2181017677 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(654..654 src has 451..654) ==== 6285+0+0 (secure 0 0 0) 0x7f6b980e3a30 con 0x7f6ba00a1330 2024-09-16T04:03:24.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.067+0000 7f6b977fe640 10 client.825669.objecter ms_dispatch 0x7f6ba0000ba0 osd_map(654..654 src has 451..654) 2024-09-16T04:03:24.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.067+0000 7f6b977fe640 3 client.825669.objecter handle_osd_map got epochs [654,654] > 0 2024-09-16T04:03:24.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.067+0000 7f6b977fe640 3 client.825669.objecter handle_osd_map decoding full epoch 654 2024-09-16T04:03:24.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.067+0000 7f6b9e7fc640 1 -- 172.21.15.42:0/2181017677 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6ba0071b30 con 0x7f6ba006dac0 2024-09-16T04:03:24.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.067+0000 7f6b977fe640 20 client.825669.objecter dump_active .. 0 homeless 2024-09-16T04:03:24.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.067+0000 7f6b977fe640 10 client.825669.objecter ms_handle_connect 0x7f6ba006dac0 2024-09-16T04:03:24.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.067+0000 7f6b977fe640 10 client.825669.objecter resend_mon_ops 2024-09-16T04:03:24.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.067+0000 7f6b9e7fc640 1 --2- 172.21.15.42:0/2181017677 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6ba006dac0 0x7f6ba002c650 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f6ba0072ae0 tx=0x7f6b8800f610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:03:24.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.067+0000 7f6b977fe640 1 -- 172.21.15.42:0/2181017677 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b88019070 con 0x7f6ba006dac0 2024-09-16T04:03:24.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.067+0000 7f6b977fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:03:24.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.067+0000 7f6b9f7fe640 1 --2- 172.21.15.42:0/2181017677 >> v2:172.21.15.154:6812/254845342 conn(0x7f6b80037810 0x7f6b80039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:03:24.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.067+0000 7f6b977fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:03:24.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.067+0000 7f6b977fe640 10 monclient: dump: 2024-09-16T04:03:24.090 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:03:24.090 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:03:24.090 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:24.090 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:24.090 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:03:24.090 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:03:24.090 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:03:24.090 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:03:24.090 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:03:24.090 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:03:24.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.067+0000 7f6b9f7fe640 10 monclient: get_auth_request con 0x7f6b80037810 auth_method 0 2024-09-16T04:03:24.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.067+0000 7f6b977fe640 1 -- 172.21.15.42:0/2181017677 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f6b88004030 con 0x7f6ba006dac0 2024-09-16T04:03:24.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.067+0000 7f6b977fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:03:24.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.067+0000 7f6b977fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:03:24.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.067+0000 7f6b977fe640 1 -- 172.21.15.42:0/2181017677 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6ba006dac0 msgr2=0x7f6ba002c650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:03:24.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.067+0000 7f6b977fe640 1 --2- 172.21.15.42:0/2181017677 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6ba006dac0 0x7f6ba002c650 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f6ba0072ae0 tx=0x7f6b8800f610 comp rx=0 tx=0).stop 2024-09-16T04:03:24.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.067+0000 7f6b977fe640 1 -- 172.21.15.42:0/2181017677 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6ba006dac0 msgr2=0x7f6ba002c650 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:03:24.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.067+0000 7f6b977fe640 1 --2- 172.21.15.42:0/2181017677 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6ba006dac0 0x7f6ba002c650 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:03:24.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.067+0000 7f6b9f7fe640 1 --2- 172.21.15.42:0/2181017677 >> v2:172.21.15.154:6812/254845342 conn(0x7f6b80037810 0x7f6b80039cd0 secure :-1 s=READY pgs=3806 cs=0 l=1 rev1=1 crypto rx=0x7f6b90003dc0 tx=0x7f6b90009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:03:24.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.067+0000 7f6b977fe640 10 client.825669.objecter ms_handle_connect 0x7f6b80037810 2024-09-16T04:03:24.093 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.075+0000 7f6ba5453640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:03:24.093 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.075+0000 7f6ba5453640 1 --2- 172.21.15.42:0/2181017677 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6ba009d000 0x7f6ba002cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:03:24.093 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.075+0000 7f6ba5453640 10 monclient(hunting): start opening mon connection 2024-09-16T04:03:24.093 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.075+0000 7f6ba5453640 1 -- 172.21.15.42:0/2181017677 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f6ba0135550 con 0x7f6ba009d000 2024-09-16T04:03:24.093 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.075+0000 7f6b9e7fc640 1 --2- 172.21.15.42:0/2181017677 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6ba009d000 0x7f6ba002cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:03:24.093 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.075+0000 7f6b9e7fc640 10 monclient: get_auth_request con 0x7f6ba009d000 auth_method 0 2024-09-16T04:03:24.093 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.075+0000 7f6b9e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:03:24.093 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.075+0000 7f6b9e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:03:24.093 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.075+0000 7f6b9e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:03:24.093 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.075+0000 7f6b9e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:03:24.093 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.075+0000 7f6b9e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:03:24.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.075+0000 7f6b9e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:03:24.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.075+0000 7f6b9e7fc640 10 monclient(hunting): handle_auth_done global_id 825675 payload 824 2024-09-16T04:03:24.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.075+0000 7f6b9e7fc640 1 -- 172.21.15.42:0/2181017677 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f6ba0070eb0 con 0x7f6ba009d000 2024-09-16T04:03:24.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.075+0000 7f6b9e7fc640 1 --2- 172.21.15.42:0/2181017677 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6ba009d000 0x7f6ba002cb90 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f6ba0072ae0 tx=0x7f6b880046d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:03:24.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.075+0000 7f6b977fe640 10 client.825669.objecter ms_handle_connect 0x7f6ba009d000 2024-09-16T04:03:24.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.075+0000 7f6b977fe640 10 client.825669.objecter resend_mon_ops 2024-09-16T04:03:24.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.075+0000 7f6b977fe640 1 -- 172.21.15.42:0/2181017677 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b88019040 con 0x7f6ba009d000 2024-09-16T04:03:24.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.075+0000 7f6b977fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:03:24.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.075+0000 7f6b977fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:03:24.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.075+0000 7f6b977fe640 10 monclient: dump: 2024-09-16T04:03:24.094 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:03:24.094 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:03:24.094 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:24.095 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:24.095 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:03:24.095 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:03:24.095 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:03:24.095 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:03:24.095 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:03:24.095 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:03:24.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.075+0000 7f6b977fe640 1 -- 172.21.15.42:0/2181017677 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4882 (secure 0 0 0) 0x7f6b8801e450 con 0x7f6ba009d000 2024-09-16T04:03:24.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.075+0000 7f6b977fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:03:24.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.075+0000 7f6b977fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:03:24.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.075+0000 7f6b977fe640 1 -- 172.21.15.42:0/2181017677 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6ba009d000 msgr2=0x7f6ba002cb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:03:24.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.075+0000 7f6b977fe640 1 --2- 172.21.15.42:0/2181017677 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6ba009d000 0x7f6ba002cb90 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f6ba0072ae0 tx=0x7f6b880046d0 comp rx=0 tx=0).stop 2024-09-16T04:03:24.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.075+0000 7f6b977fe640 1 -- 172.21.15.42:0/2181017677 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6ba009d000 msgr2=0x7f6ba002cb90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:03:24.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.075+0000 7f6b977fe640 1 --2- 172.21.15.42:0/2181017677 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6ba009d000 0x7f6ba002cb90 secure :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f6ba0072ae0 tx=0x7f6b880046d0 comp rx=0 tx=0).stop 2024-09-16T04:03:24.096 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T04:03:24.096 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:03:24.096 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:03:24.096 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "leader", 2024-09-16T04:03:24.096 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 190, 2024-09-16T04:03:24.096 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 693929, 2024-09-16T04:03:24.096 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T04:03:24.096 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T04:03:24.096 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T04:03:24.096 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T04:03:24.097 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:03:24.097 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 7, 2024-09-16T04:03:24.097 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:03:24.097 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T04:03:24.097 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T04:03:24.097 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:03:24.097 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:03:24.097 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:03:24.097 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:03:24.097 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:03:24.097 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:03:24.097 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:03:24.097 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:03:24.097 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:03:24.098 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:03:24.098 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:03:24.098 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:03:24.098 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:03:24.098 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T04:03:24.098 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:03:24.098 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:03:24.098 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:03:24.098 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:03:24.098 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:03:24.098 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:03:24.098 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:03:24.098 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:03:24.098 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:03:24.099 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:03:24.099 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:03:24.099 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:03:24.099 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:03:24.099 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T04:03:24.099 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T04:03:24.099 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T04:03:24.099 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T04:03:24.099 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T04:03:24.099 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T04:03:24.099 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:03:24.099 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:03:24.099 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T04:03:24.099 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T04:03:24.099 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T04:03:24.100 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T04:03:24.100 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T04:03:24.100 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T04:03:24.100 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T04:03:24.100 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:03:24.100 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T04:03:24.100 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:03:24.100 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:03:24.100 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:03:24.100 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:03:24.100 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:03:24.100 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:03:24.101 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:03:24.101 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:03:24.101 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:03:24.101 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:03:24.101 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:03:24.101 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:03:24.101 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T04:03:24.101 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:03:24.101 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T04:03:24.101 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:03:24.101 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:03:24.101 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:03:24.101 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:03:24.101 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:03:24.102 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:03:24.102 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:03:24.102 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T04:03:24.102 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:03:24.102 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:03:24.102 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:03:24.102 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:03:24.102 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T04:03:24.102 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:03:24.102 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:03:24.102 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:03:24.102 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:03:24.102 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T04:03:24.103 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T04:03:24.103 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:03:24.103 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:03:24.103 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:03:24.103 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:03:24.103 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:03:24.103 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:03:24.103 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:03:24.103 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:03:24.103 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:03:24.103 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:03:24.103 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:03:24.103 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T04:03:24.104 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:03:24.104 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:03:24.104 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:03:24.104 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:03:24.104 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T04:03:24.104 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:03:24.104 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:03:24.104 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:03:24.104 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:03:24.104 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T04:03:24.104 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T04:03:24.104 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:03:24.104 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:03:24.105 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:03:24.105 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:03:24.105 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:03:24.105 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:03:24.105 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:03:24.105 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:03:24.105 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:03:24.105 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:03:24.105 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:03:24.105 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T04:03:24.105 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:03:24.105 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:03:24.105 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:03:24.106 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:03:24.106 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T04:03:24.106 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:03:24.106 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:03:24.106 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:03:24.106 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:03:24.106 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T04:03:24.106 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T04:03:24.106 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:03:24.106 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:03:24.106 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:03:24.106 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:03:24.106 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:03:24.106 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:03:24.107 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T04:03:24.107 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T04:03:24.107 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:03:24.107 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:03:24.107 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:03:24.107 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:03:24.107 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:03:24.107 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:03:24.107 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T04:03:24.107 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:03:24.107 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:03:24.107 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:03:24.107 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 3 2024-09-16T04:03:24.107 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:03:24.107 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:03:24.108 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T04:03:24.108 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:03:24.108 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:03:24.108 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:03:24.108 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 7 2024-09-16T04:03:24.108 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:03:24.108 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:03:24.108 INFO:teuthology.orchestra.run.smithi042.stdout: "mgr": [ 2024-09-16T04:03:24.108 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:03:24.108 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:03:24.108 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:03:24.108 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:03:24.109 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:03:24.109 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:03:24.109 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:03:24.109 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T04:03:24.109 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T04:03:24.111 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.075+0000 7f6ba5453640 10 librados: watch_flush enter 2024-09-16T04:03:24.111 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.075+0000 7f6ba5453640 10 librados: watch_flush exit 2024-09-16T04:03:24.111 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.075+0000 7f6ba5453640 20 client.825669.objecter shutdown clearing up homeless session... 2024-09-16T04:03:24.111 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.075+0000 7f6ba5453640 10 client.825669.objecter successfully canceled tick 2024-09-16T04:03:24.111 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.075+0000 7f6ba5453640 1 -- 172.21.15.42:0/2181017677 >> v2:172.21.15.154:6812/254845342 conn(0x7f6b80037810 msgr2=0x7f6b80039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:03:24.111 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.075+0000 7f6ba5453640 1 --2- 172.21.15.42:0/2181017677 >> v2:172.21.15.154:6812/254845342 conn(0x7f6b80037810 0x7f6b80039cd0 secure :-1 s=READY pgs=3806 cs=0 l=1 rev1=1 crypto rx=0x7f6b90003dc0 tx=0x7f6b90009040 comp rx=0 tx=0).stop 2024-09-16T04:03:24.111 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.075+0000 7f6ba5453640 10 monclient: shutdown 2024-09-16T04:03:24.111 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.075+0000 7f6ba5453640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:03:24.111 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.075+0000 7f6ba5453640 1 -- 172.21.15.42:0/2181017677 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6ba00a1330 msgr2=0x7f6ba013b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:03:24.111 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.075+0000 7f6ba5453640 1 --2- 172.21.15.42:0/2181017677 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6ba00a1330 0x7f6ba013b170 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f6b980ac000 tx=0x7f6ba00a22d0 comp rx=0 tx=0).stop 2024-09-16T04:03:24.112 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.075+0000 7f6b9f7fe640 1 -- 172.21.15.42:0/2181017677 reap_dead start 2024-09-16T04:03:24.112 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.075+0000 7f6b9f7fe640 1 -- 172.21.15.42:0/2181017677 reap_dead start 2024-09-16T04:03:24.112 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.075+0000 7f6ba5453640 1 -- 172.21.15.42:0/2181017677 shutdown_connections 2024-09-16T04:03:24.112 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.075+0000 7f6ba5453640 1 -- 172.21.15.42:0/2181017677 >> 172.21.15.42:0/2181017677 conn(0x7f6ba00a2420 msgr2=0x7f6ba013c590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:03:24.112 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.075+0000 7f6ba5453640 1 -- 172.21.15.42:0/2181017677 shutdown_connections 2024-09-16T04:03:24.112 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.075+0000 7f6ba5453640 1 -- 172.21.15.42:0/2181017677 wait complete. 2024-09-16T04:03:24.112 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.075+0000 7f6ba5453640 1 librados: shutdown 2024-09-16T04:03:24.112 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:03:24.112 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T04:03:24.113 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T04:03:24.233 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.227+0000 7f3226f58640 10 monclient: get_monmap_and_config 2024-09-16T04:03:24.233 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.227+0000 7f3226f58640 10 monclient: build_initial_monmap 2024-09-16T04:03:24.233 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.227+0000 7f3226f58640 10 monclient: monmap: 2024-09-16T04:03:24.233 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:03:24.234 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:03:24.234 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:03:24.231834+0000 2024-09-16T04:03:24.234 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:03:24.231834+0000 2024-09-16T04:03:24.234 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:03:24.234 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:03:24.234 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:03:24.234 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:03:24.234 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:03:24.234 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:03:24.234 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:03:24.235 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:03:24.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.231+0000 7f3226f58640 1 Processor -- start 2024-09-16T04:03:24.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.231+0000 7f3226f58640 1 -- start start 2024-09-16T04:03:24.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.231+0000 7f3226f58640 10 monclient: init 2024-09-16T04:03:24.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.231+0000 7f3226f58640 10 monclient: _reopen_session rank -1 2024-09-16T04:03:24.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.231+0000 7f3226f58640 10 monclient: _add_conns ranks=[2,0,4,1,3] 2024-09-16T04:03:24.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.231+0000 7f3226f58640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f322009ca60 0x7f322009f240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:03:24.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.231+0000 7f3226f58640 10 monclient(hunting): picked mon.noname-b con 0x7f322009ca60 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:03:24.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.231+0000 7f3226f58640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f322009d460 addr v1:172.21.15.42:6789/0 2024-09-16T04:03:24.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.231+0000 7f3226f58640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f322009df50 0x7f322013b770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:03:24.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.231+0000 7f3226f58640 10 monclient(hunting): picked mon.noname-c con 0x7f322009df50 addr v2:172.21.15.154:3300/0 2024-09-16T04:03:24.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.231+0000 7f3226f58640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f322009e580 con 0x7f322009d460 2024-09-16T04:03:24.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.231+0000 7f3226f58640 10 monclient(hunting): start opening mon connection 2024-09-16T04:03:24.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.231+0000 7f3226f58640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f322009ce40 con 0x7f322009df50 2024-09-16T04:03:24.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.231+0000 7f3226f58640 10 monclient(hunting): start opening mon connection 2024-09-16T04:03:24.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.231+0000 7f3226f58640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f322009e860 con 0x7f322009ca60 2024-09-16T04:03:24.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.231+0000 7f3226f58640 10 monclient(hunting): _renew_subs 2024-09-16T04:03:24.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.231+0000 7f3226f58640 10 monclient(hunting): authenticate will time out at 2251.201467s 2024-09-16T04:03:24.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.231+0000 7f3224cf4640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f322009ca60 0x7f322009f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:03:24.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.231+0000 7f3224cf4640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f322009ca60 0x7f322009f240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:60522/0 (socket says 172.21.15.42:60522) 2024-09-16T04:03:24.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.231+0000 7f3224cf4640 1 -- 172.21.15.42:0/2260828618 learned_addr learned my addr 172.21.15.42:0/2260828618 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:03:24.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.231+0000 7f3224cf4640 10 monclient(hunting): get_auth_request con 0x7f322009ca60 auth_method 0 2024-09-16T04:03:24.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.231+0000 7f3224cf4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:03:24.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.231+0000 7f3224cf4640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:03:24.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.231+0000 7f3224cf4640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:03:24.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.231+0000 7f321ffff640 1 --1- 172.21.15.42:0/2260828618 >> v1:172.21.15.42:6789/0 conn(0x7f322009d460 0x7f322009d840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:35514/0 (socket says 172.21.15.42:35514) 2024-09-16T04:03:24.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.231+0000 7f3224cf4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:03:24.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.231+0000 7f3224cf4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:03:24.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.231+0000 7f3224cf4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:03:24.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.231+0000 7f32254f5640 1 --2- 172.21.15.42:0/2260828618 >> v2:172.21.15.154:3300/0 conn(0x7f322009df50 0x7f322013b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:03:24.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.231+0000 7f32254f5640 10 monclient(hunting): get_auth_request con 0x7f322009df50 auth_method 0 2024-09-16T04:03:24.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.231+0000 7f32254f5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:03:24.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.231+0000 7f32254f5640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:03:24.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.231+0000 7f32254f5640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:03:24.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.231+0000 7f3224cf4640 10 monclient(hunting): handle_auth_done global_id 974126 payload 274 2024-09-16T04:03:24.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.231+0000 7f3224cf4640 1 -- 172.21.15.42:0/2260828618 >> v2:172.21.15.154:3300/0 conn(0x7f322009df50 msgr2=0x7f322013b770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:03:24.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.231+0000 7f3224cf4640 1 --2- 172.21.15.42:0/2260828618 >> v2:172.21.15.154:3300/0 conn(0x7f322009df50 0x7f322013b770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:03:24.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.231+0000 7f3224cf4640 1 -- 172.21.15.42:0/2260828618 >> v1:172.21.15.42:6789/0 conn(0x7f322009d460 legacy=0x7f322009d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:03:24.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.231+0000 7f3224cf4640 10 monclient: _finish_hunting 0 2024-09-16T04:03:24.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.231+0000 7f3224cf4640 1 monclient: found mon.noname-b 2024-09-16T04:03:24.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.231+0000 7f321f7fe640 1 -- 172.21.15.42:0/2260828618 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3572197437 0 0) 0x7f322009e580 con 0x7f322009d460 2024-09-16T04:03:24.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.231+0000 7f3224cf4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:03:24.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.231+0000 7f3224cf4640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T04:03:24.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.231+0000 7f3224cf4640 1 -- 172.21.15.42:0/2260828618 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f322009eb40 con 0x7f322009ca60 2024-09-16T04:03:24.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.231+0000 7f321f7fe640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T04:03:24.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.235+0000 7f3224cf4640 1 --2- 172.21.15.42:0/2260828618 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f322009ca60 0x7f322009f240 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f3210009870 tx=0x7f321001c420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2e3692d7c8ebad24 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:03:24.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.235+0000 7f321f7fe640 1 -- 172.21.15.42:0/2260828618 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32100b0070 con 0x7f322009ca60 2024-09-16T04:03:24.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.235+0000 7f321f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:03:24.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.235+0000 7f321f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:03:24.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.235+0000 7f321f7fe640 10 monclient: dump: 2024-09-16T04:03:24.246 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:03:24.246 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:03:24.246 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:24.246 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:24.246 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:03:24.247 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:03:24.247 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:03:24.247 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:03:24.247 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:03:24.247 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:03:24.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.235+0000 7f321f7fe640 10 monclient: _finish_auth 0 2024-09-16T04:03:24.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.235+0000 7f321f7fe640 10 monclient: _check_auth_tickets 2024-09-16T04:03:24.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.235+0000 7f321f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:03:24.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.235+0000 7f321f7fe640 1 -- 172.21.15.42:0/2260828618 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f32100ab600 con 0x7f322009ca60 2024-09-16T04:03:24.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.235+0000 7f321f7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:03:24.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.235+0000 7f321f7fe640 1 -- 172.21.15.42:0/2260828618 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32100ab9d0 con 0x7f322009ca60 2024-09-16T04:03:24.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.235+0000 7f3226f58640 5 monclient: authenticate success, global_id 974126 2024-09-16T04:03:24.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.235+0000 7f3226f58640 10 monclient: get_monmap_and_config success 2024-09-16T04:03:24.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.235+0000 7f3226f58640 4 set_mon_vals no callback set 2024-09-16T04:03:24.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.235+0000 7f321f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:03:24.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.235+0000 7f321f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:03:24.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.235+0000 7f321f7fe640 10 monclient: dump: 2024-09-16T04:03:24.248 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:03:24.248 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:03:24.248 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:24.248 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:24.248 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:03:24.248 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:03:24.248 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:03:24.249 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:03:24.249 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:03:24.249 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:03:24.249 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.235+0000 7f3226f58640 10 monclient: shutdown 2024-09-16T04:03:24.249 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.235+0000 7f3226f58640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:03:24.249 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.235+0000 7f3226f58640 1 -- 172.21.15.42:0/2260828618 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f322009ca60 msgr2=0x7f322009f240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:03:24.249 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.235+0000 7f3226f58640 1 --2- 172.21.15.42:0/2260828618 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f322009ca60 0x7f322009f240 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f3210009870 tx=0x7f321001c420 comp rx=0 tx=0).stop 2024-09-16T04:03:24.249 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.235+0000 7f3226f58640 1 -- 172.21.15.42:0/2260828618 shutdown_connections 2024-09-16T04:03:24.249 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.235+0000 7f3226f58640 1 --2- 172.21.15.42:0/2260828618 >> v2:172.21.15.154:3300/0 conn(0x7f322009df50 0x7f322013b770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:03:24.249 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.235+0000 7f3226f58640 1 --2- 172.21.15.42:0/2260828618 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f322009ca60 0x7f322009f240 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:03:24.249 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.235+0000 7f3226f58640 1 -- 172.21.15.42:0/2260828618 >> 172.21.15.42:0/2260828618 conn(0x7f322006bc40 msgr2=0x7f322006c050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:03:24.249 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.235+0000 7f3226f58640 1 -- 172.21.15.42:0/2260828618 shutdown_connections 2024-09-16T04:03:24.249 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.235+0000 7f3226f58640 1 -- 172.21.15.42:0/2260828618 wait complete. 2024-09-16T04:03:24.249 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.235+0000 7f3226f58640 10 monclient: build_initial_monmap 2024-09-16T04:03:24.250 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.235+0000 7f3226f58640 10 monclient: monmap: 2024-09-16T04:03:24.250 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:03:24.250 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:03:24.250 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:03:24.250 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:03:24.250 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:03:24.250 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:03:24.250 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:03:24.250 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:03:24.251 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:03:24.251 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:03:24.251 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.235+0000 7f3226f58640 1 librados: starting msgr at 2024-09-16T04:03:24.251 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.235+0000 7f3226f58640 1 librados: starting objecter 2024-09-16T04:03:24.251 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.235+0000 7f3226f58640 1 Processor -- start 2024-09-16T04:03:24.251 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.235+0000 7f3226f58640 1 -- start start 2024-09-16T04:03:24.251 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.235+0000 7f3226f58640 1 librados: setting wanted keys 2024-09-16T04:03:24.251 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.235+0000 7f3226f58640 1 librados: calling monclient init 2024-09-16T04:03:24.251 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.235+0000 7f3226f58640 10 monclient: init 2024-09-16T04:03:24.251 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.235+0000 7f3226f58640 10 monclient: _reopen_session rank -1 2024-09-16T04:03:24.251 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.235+0000 7f3226f58640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:03:24.251 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.235+0000 7f3226f58640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f322009ca60 0x7f32201ca640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:03:24.251 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.235+0000 7f3226f58640 10 monclient(hunting): picked mon.noname-c con 0x7f322009ca60 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:03:24.251 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.235+0000 7f3226f58640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f322009d460 0x7f3220132b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:03:24.251 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.235+0000 7f3226f58640 10 monclient(hunting): picked mon.noname-b con 0x7f322009d460 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:03:24.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.235+0000 7f3226f58640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f322009df50 0x7f32201330a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:03:24.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.235+0000 7f3226f58640 10 monclient(hunting): picked mon.noname-a con 0x7f322009df50 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:03:24.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.235+0000 7f3226f58640 10 monclient(hunting): start opening mon connection 2024-09-16T04:03:24.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.235+0000 7f3226f58640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f32200a0cf0 con 0x7f322009df50 2024-09-16T04:03:24.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.235+0000 7f3226f58640 10 monclient(hunting): start opening mon connection 2024-09-16T04:03:24.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.235+0000 7f3226f58640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f32200a0b70 con 0x7f322009d460 2024-09-16T04:03:24.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.235+0000 7f3226f58640 10 monclient(hunting): start opening mon connection 2024-09-16T04:03:24.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.235+0000 7f3226f58640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f32200a0e70 con 0x7f322009ca60 2024-09-16T04:03:24.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.235+0000 7f3226f58640 10 monclient(hunting): _renew_subs 2024-09-16T04:03:24.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.235+0000 7f3226f58640 10 monclient(hunting): authenticate will time out at 2251.206778s 2024-09-16T04:03:24.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.235+0000 7f3224cf4640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f322009ca60 0x7f32201ca640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:03:24.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.235+0000 7f3224cf4640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f322009ca60 0x7f32201ca640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:60538/0 (socket says 172.21.15.42:60538) 2024-09-16T04:03:24.253 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.235+0000 7f3224cf4640 1 -- 172.21.15.42:0/3764897876 learned_addr learned my addr 172.21.15.42:0/3764897876 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:03:24.253 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.235+0000 7f3224cf4640 10 monclient(hunting): get_auth_request con 0x7f322009ca60 auth_method 0 2024-09-16T04:03:24.253 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.235+0000 7f3224cf4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:03:24.253 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.235+0000 7f3224cf4640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:03:24.253 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.235+0000 7f3224cf4640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:03:24.253 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.235+0000 7f32254f5640 1 --2- 172.21.15.42:0/3764897876 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f322009df50 0x7f32201330a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:03:24.253 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.235+0000 7f32254f5640 10 monclient(hunting): get_auth_request con 0x7f322009df50 auth_method 0 2024-09-16T04:03:24.253 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.235+0000 7f32254f5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:03:24.253 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.235+0000 7f32254f5640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:03:24.253 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.235+0000 7f32254f5640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:03:24.253 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.235+0000 7f3224cf4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:03:24.254 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.235+0000 7f3224cf4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:03:24.254 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.235+0000 7f3224cf4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:03:24.254 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.235+0000 7f32254f5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:03:24.254 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.235+0000 7f32254f5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:03:24.254 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.235+0000 7f32254f5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:03:24.254 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.235+0000 7f3224cf4640 10 monclient(hunting): handle_auth_done global_id 974129 payload 824 2024-09-16T04:03:24.254 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.235+0000 7f321ffff640 1 --2- 172.21.15.42:0/3764897876 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f322009d460 0x7f3220132b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:03:24.254 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.235+0000 7f3224cf4640 1 -- 172.21.15.42:0/3764897876 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f322009d460 msgr2=0x7f3220132b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:03:24.254 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.235+0000 7f3224cf4640 1 --2- 172.21.15.42:0/3764897876 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f322009d460 0x7f3220132b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:03:24.254 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.235+0000 7f3224cf4640 1 -- 172.21.15.42:0/3764897876 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f322009df50 msgr2=0x7f32201330a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:03:24.254 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.235+0000 7f3224cf4640 1 --2- 172.21.15.42:0/3764897876 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f322009df50 0x7f32201330a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:03:24.254 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.235+0000 7f3224cf4640 10 monclient: _finish_hunting 0 2024-09-16T04:03:24.254 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.235+0000 7f3224cf4640 1 monclient: found mon.noname-c 2024-09-16T04:03:24.254 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.235+0000 7f3224cf4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:03:24.254 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.235+0000 7f3224cf4640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T04:03:24.255 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.235+0000 7f3224cf4640 1 -- 172.21.15.42:0/3764897876 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f322009d150 con 0x7f322009ca60 2024-09-16T04:03:24.255 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.235+0000 7f32254f5640 1 --2- 172.21.15.42:0/3764897876 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f322009df50 0x7f32201330a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:03:24.255 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.235+0000 7f3224cf4640 1 --2- 172.21.15.42:0/3764897876 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f322009ca60 0x7f32201ca640 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f32100ae040 tx=0x7f32100191c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:03:24.255 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.235+0000 7f321d7fa640 10 client.?.objecter ms_handle_connect 0x7f322009ca60 2024-09-16T04:03:24.255 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.235+0000 7f321d7fa640 10 client.?.objecter resend_mon_ops 2024-09-16T04:03:24.255 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.235+0000 7f321d7fa640 1 -- 172.21.15.42:0/3764897876 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32100b0040 con 0x7f322009ca60 2024-09-16T04:03:24.255 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.235+0000 7f321d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:03:24.255 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.235+0000 7f321d7fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:03:24.255 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.235+0000 7f321d7fa640 10 monclient: dump: 2024-09-16T04:03:24.255 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:03:24.255 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:03:24.255 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:24.255 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:24.255 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:03:24.256 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:03:24.256 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:03:24.256 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:03:24.256 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:03:24.256 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:03:24.256 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.235+0000 7f321d7fa640 10 monclient: _finish_auth 0 2024-09-16T04:03:24.256 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.235+0000 7f321d7fa640 10 monclient: _check_auth_tickets 2024-09-16T04:03:24.256 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.235+0000 7f321d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:03:24.256 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.235+0000 7f321d7fa640 1 -- 172.21.15.42:0/3764897876 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f32100b9070 con 0x7f322009ca60 2024-09-16T04:03:24.257 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.235+0000 7f321d7fa640 10 monclient: handle_config config(0 keys) 2024-09-16T04:03:24.257 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.235+0000 7f321d7fa640 1 -- 172.21.15.42:0/3764897876 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32100b4860 con 0x7f322009ca60 2024-09-16T04:03:24.257 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.235+0000 7f321d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:03:24.257 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.235+0000 7f321d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:03:24.257 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.235+0000 7f321d7fa640 10 monclient: dump: 2024-09-16T04:03:24.257 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:03:24.257 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:03:24.257 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:24.257 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:24.257 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:03:24.257 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:03:24.257 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:03:24.258 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:03:24.258 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:03:24.258 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:03:24.258 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.235+0000 7f3226f58640 5 monclient: authenticate success, global_id 974129 2024-09-16T04:03:24.259 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.235+0000 7f3226f58640 10 monclient: _renew_subs 2024-09-16T04:03:24.259 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.235+0000 7f3226f58640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:03:24.259 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.235+0000 7f321effd640 4 set_mon_vals no callback set 2024-09-16T04:03:24.259 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.235+0000 7f3226f58640 1 -- 172.21.15.42:0/3764897876 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32200a51e0 con 0x7f322009ca60 2024-09-16T04:03:24.259 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.235+0000 7f3226f58640 10 client.974129.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:03:24.259 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.235+0000 7f3226f58640 10 monclient: _renew_subs 2024-09-16T04:03:24.259 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.235+0000 7f3226f58640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:03:24.259 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.235+0000 7f3226f58640 1 -- 172.21.15.42:0/3764897876 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f322013a1d0 con 0x7f322009ca60 2024-09-16T04:03:24.259 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.235+0000 7f3226f58640 1 librados: init done 2024-09-16T04:03:24.259 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.239+0000 7f3226f58640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:03:24.259 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.239+0000 7f3226f58640 1 --2- 172.21.15.42:0/3764897876 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3220137ca0 0x7f3220138080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:03:24.259 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.239+0000 7f3226f58640 10 monclient(hunting): start opening mon connection 2024-09-16T04:03:24.259 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.239+0000 7f3226f58640 1 -- 172.21.15.42:0/3764897876 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f322013b560 con 0x7f3220137ca0 2024-09-16T04:03:24.260 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.239+0000 7f321d7fa640 1 -- 172.21.15.42:0/3764897876 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f32100b4a00 con 0x7f322009ca60 2024-09-16T04:03:24.260 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.239+0000 7f321ffff640 1 --2- 172.21.15.42:0/3764897876 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3220137ca0 0x7f3220138080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:03:24.260 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.239+0000 7f321ffff640 10 monclient: get_auth_request con 0x7f3220137ca0 auth_method 0 2024-09-16T04:03:24.260 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.239+0000 7f321ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:03:24.260 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.239+0000 7f321ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:03:24.260 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.239+0000 7f321ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:03:24.260 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.239+0000 7f321ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:03:24.260 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.239+0000 7f321ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:03:24.260 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.239+0000 7f321ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:03:24.260 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.239+0000 7f321d7fa640 1 --2- 172.21.15.42:0/3764897876 >> v2:172.21.15.154:6812/254845342 conn(0x7f320c037810 0x7f320c039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:03:24.260 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.239+0000 7f321ffff640 10 monclient(hunting): handle_auth_done global_id 974132 payload 824 2024-09-16T04:03:24.260 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.239+0000 7f321d7fa640 1 -- 172.21.15.42:0/3764897876 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(654..654 src has 451..654) ==== 6285+0+0 (secure 0 0 0) 0x7f32100e2fe0 con 0x7f322009ca60 2024-09-16T04:03:24.260 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.239+0000 7f321d7fa640 10 client.974129.objecter ms_dispatch 0x7f3220000ba0 osd_map(654..654 src has 451..654) 2024-09-16T04:03:24.261 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.239+0000 7f321d7fa640 3 client.974129.objecter handle_osd_map got epochs [654,654] > 0 2024-09-16T04:03:24.261 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.239+0000 7f321d7fa640 3 client.974129.objecter handle_osd_map decoding full epoch 654 2024-09-16T04:03:24.261 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.239+0000 7f321ffff640 1 -- 172.21.15.42:0/3764897876 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f322009f240 con 0x7f3220137ca0 2024-09-16T04:03:24.261 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.239+0000 7f321d7fa640 20 client.974129.objecter dump_active .. 0 homeless 2024-09-16T04:03:24.261 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.239+0000 7f321d7fa640 10 client.974129.objecter ms_handle_connect 0x7f3220137ca0 2024-09-16T04:03:24.261 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.239+0000 7f321ffff640 1 --2- 172.21.15.42:0/3764897876 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3220137ca0 0x7f3220138080 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f32201394f0 tx=0x7f321400d8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:03:24.261 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.239+0000 7f321d7fa640 10 client.974129.objecter resend_mon_ops 2024-09-16T04:03:24.261 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.239+0000 7f32254f5640 1 --2- 172.21.15.42:0/3764897876 >> v2:172.21.15.154:6812/254845342 conn(0x7f320c037810 0x7f320c039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:03:24.261 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.239+0000 7f321d7fa640 1 -- 172.21.15.42:0/3764897876 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32140075a0 con 0x7f3220137ca0 2024-09-16T04:03:24.261 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.239+0000 7f321d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:03:24.261 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.239+0000 7f321d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:03:24.261 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.239+0000 7f321d7fa640 10 monclient: dump: 2024-09-16T04:03:24.261 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:03:24.262 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:03:24.262 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:24.262 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:24.262 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:03:24.262 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:03:24.262 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:03:24.262 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:03:24.262 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:03:24.262 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:03:24.262 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.239+0000 7f32254f5640 10 monclient: get_auth_request con 0x7f320c037810 auth_method 0 2024-09-16T04:03:24.262 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.239+0000 7f321d7fa640 1 -- 172.21.15.42:0/3764897876 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f32140075a0 con 0x7f3220137ca0 2024-09-16T04:03:24.262 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.239+0000 7f321d7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:03:24.262 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.239+0000 7f321d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:03:24.262 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.239+0000 7f321d7fa640 1 -- 172.21.15.42:0/3764897876 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3220137ca0 msgr2=0x7f3220138080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:03:24.263 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.239+0000 7f321d7fa640 1 --2- 172.21.15.42:0/3764897876 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3220137ca0 0x7f3220138080 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f32201394f0 tx=0x7f321400d8a0 comp rx=0 tx=0).stop 2024-09-16T04:03:24.263 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.239+0000 7f321d7fa640 1 -- 172.21.15.42:0/3764897876 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3220137ca0 msgr2=0x7f3220138080 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:03:24.263 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.239+0000 7f321d7fa640 1 --2- 172.21.15.42:0/3764897876 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3220137ca0 0x7f3220138080 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:03:24.263 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.239+0000 7f321d7fa640 10 client.974129.objecter ms_handle_connect 0x7f320c037810 2024-09-16T04:03:24.263 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.239+0000 7f32254f5640 1 --2- 172.21.15.42:0/3764897876 >> v2:172.21.15.154:6812/254845342 conn(0x7f320c037810 0x7f320c039cd0 secure :-1 s=READY pgs=3807 cs=0 l=1 rev1=1 crypto rx=0x7f32201395e0 tx=0x7f3218009820 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:03:24.265 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.243+0000 7f3226f58640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:03:24.265 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.243+0000 7f3226f58640 1 --2- 172.21.15.42:0/3764897876 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f32201385c0 0x7f322006f690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:03:24.265 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.243+0000 7f3226f58640 10 monclient(hunting): start opening mon connection 2024-09-16T04:03:24.265 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.243+0000 7f3226f58640 1 -- 172.21.15.42:0/3764897876 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f32201398f0 con 0x7f32201385c0 2024-09-16T04:03:24.265 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.243+0000 7f321ffff640 1 --2- 172.21.15.42:0/3764897876 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f32201385c0 0x7f322006f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:03:24.265 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.247+0000 7f321ffff640 10 monclient: get_auth_request con 0x7f32201385c0 auth_method 0 2024-09-16T04:03:24.265 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.247+0000 7f321ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:03:24.265 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.247+0000 7f321ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:03:24.265 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.247+0000 7f321ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:03:24.266 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.247+0000 7f321ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:03:24.266 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.247+0000 7f321ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:03:24.266 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.247+0000 7f321ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:03:24.266 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.247+0000 7f321ffff640 10 monclient(hunting): handle_auth_done global_id 974135 payload 824 2024-09-16T04:03:24.266 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.247+0000 7f321ffff640 1 -- 172.21.15.42:0/3764897876 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f322006b3e0 con 0x7f32201385c0 2024-09-16T04:03:24.266 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.247+0000 7f321ffff640 1 --2- 172.21.15.42:0/3764897876 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f32201385c0 0x7f322006f690 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f3214014000 tx=0x7f3214001090 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:03:24.266 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.247+0000 7f321d7fa640 10 client.974129.objecter ms_handle_connect 0x7f32201385c0 2024-09-16T04:03:24.266 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.247+0000 7f321d7fa640 10 client.974129.objecter resend_mon_ops 2024-09-16T04:03:24.266 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.247+0000 7f321d7fa640 1 -- 172.21.15.42:0/3764897876 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f321401bc40 con 0x7f32201385c0 2024-09-16T04:03:24.266 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.247+0000 7f321d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:03:24.266 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.247+0000 7f321d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:03:24.266 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.247+0000 7f321d7fa640 10 monclient: dump: 2024-09-16T04:03:24.266 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:03:24.266 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:03:24.267 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:24.267 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:24.267 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:03:24.267 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:03:24.267 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:03:24.267 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:03:24.267 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:03:24.267 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:03:24.267 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.247+0000 7f321d7fa640 1 -- 172.21.15.42:0/3764897876 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4712 (secure 0 0 0) 0x7f3214013420 con 0x7f32201385c0 2024-09-16T04:03:24.267 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.247+0000 7f321d7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:03:24.267 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.247+0000 7f321d7fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:03:24.267 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.247+0000 7f321d7fa640 1 -- 172.21.15.42:0/3764897876 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f32201385c0 msgr2=0x7f322006f690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:03:24.267 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.247+0000 7f321d7fa640 1 --2- 172.21.15.42:0/3764897876 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f32201385c0 0x7f322006f690 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f3214014000 tx=0x7f3214001090 comp rx=0 tx=0).stop 2024-09-16T04:03:24.267 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.247+0000 7f321d7fa640 1 -- 172.21.15.42:0/3764897876 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f32201385c0 msgr2=0x7f322006f690 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:03:24.268 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.247+0000 7f321d7fa640 1 --2- 172.21.15.42:0/3764897876 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f32201385c0 0x7f322006f690 secure :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f3214014000 tx=0x7f3214001090 comp rx=0 tx=0).stop 2024-09-16T04:03:24.268 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T04:03:24.268 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:03:24.268 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:03:24.268 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T04:03:24.268 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 190, 2024-09-16T04:03:24.268 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 12747, 2024-09-16T04:03:24.268 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T04:03:24.268 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T04:03:24.268 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T04:03:24.269 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T04:03:24.269 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:03:24.269 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 7, 2024-09-16T04:03:24.269 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:03:24.269 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T04:03:24.269 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T04:03:24.269 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:03:24.269 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:03:24.269 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:03:24.269 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:03:24.269 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:03:24.269 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:03:24.269 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:03:24.270 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:03:24.270 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:03:24.270 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:03:24.270 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:03:24.270 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:03:24.270 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:03:24.270 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T04:03:24.270 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:03:24.270 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:03:24.270 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:03:24.270 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:03:24.270 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:03:24.270 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:03:24.270 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:03:24.270 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:03:24.271 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:03:24.271 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:03:24.271 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:03:24.271 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:03:24.271 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:03:24.271 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T04:03:24.271 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T04:03:24.271 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T04:03:24.271 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T04:03:24.271 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T04:03:24.271 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T04:03:24.271 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:03:24.271 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:03:24.271 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T04:03:24.272 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T04:03:24.272 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T04:03:24.272 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T04:03:24.272 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T04:03:24.272 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T04:03:24.272 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T04:03:24.272 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:03:24.272 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T04:03:24.272 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:03:24.272 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:03:24.272 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:03:24.272 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:03:24.272 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:03:24.272 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:03:24.273 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:03:24.273 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:03:24.273 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:03:24.273 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:03:24.273 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:03:24.273 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:03:24.273 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T04:03:24.273 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:03:24.273 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T04:03:24.273 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:03:24.273 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:03:24.273 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:03:24.273 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:03:24.273 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:03:24.274 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:03:24.274 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:03:24.274 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T04:03:24.274 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:03:24.274 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:03:24.274 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:03:24.274 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:03:24.274 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T04:03:24.274 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:03:24.274 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:03:24.274 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:03:24.274 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:03:24.274 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T04:03:24.274 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T04:03:24.275 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:03:24.275 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:03:24.275 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:03:24.275 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:03:24.275 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:03:24.275 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:03:24.275 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:03:24.275 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:03:24.275 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:03:24.275 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:03:24.275 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:03:24.275 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T04:03:24.275 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:03:24.276 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:03:24.276 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:03:24.276 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:03:24.276 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T04:03:24.276 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:03:24.276 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:03:24.276 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:03:24.276 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:03:24.276 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T04:03:24.276 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T04:03:24.276 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:03:24.276 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:03:24.276 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:03:24.276 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:03:24.277 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:03:24.277 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:03:24.277 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:03:24.277 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:03:24.277 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:03:24.277 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:03:24.277 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:03:24.277 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T04:03:24.277 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:03:24.277 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:03:24.277 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:03:24.277 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:03:24.277 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T04:03:24.277 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:03:24.278 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:03:24.278 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:03:24.278 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:03:24.278 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T04:03:24.278 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T04:03:24.278 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:03:24.278 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:03:24.278 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:03:24.278 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:03:24.278 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:03:24.278 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:03:24.278 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T04:03:24.278 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T04:03:24.278 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:03:24.279 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:03:24.279 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:03:24.279 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:03:24.279 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:03:24.279 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:03:24.279 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T04:03:24.279 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:03:24.279 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:03:24.279 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:03:24.279 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 2 2024-09-16T04:03:24.279 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:03:24.279 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:03:24.279 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T04:03:24.280 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:03:24.280 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:03:24.280 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:03:24.280 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 2 2024-09-16T04:03:24.280 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:03:24.280 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:03:24.280 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:03:24.280 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T04:03:24.280 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T04:03:24.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.247+0000 7f3226f58640 10 librados: watch_flush enter 2024-09-16T04:03:24.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.247+0000 7f3226f58640 10 librados: watch_flush exit 2024-09-16T04:03:24.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.247+0000 7f3226f58640 20 client.974129.objecter shutdown clearing up homeless session... 2024-09-16T04:03:24.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.247+0000 7f3226f58640 10 client.974129.objecter successfully canceled tick 2024-09-16T04:03:24.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.247+0000 7f3226f58640 1 -- 172.21.15.42:0/3764897876 >> v2:172.21.15.154:6812/254845342 conn(0x7f320c037810 msgr2=0x7f320c039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:03:24.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.247+0000 7f3226f58640 1 --2- 172.21.15.42:0/3764897876 >> v2:172.21.15.154:6812/254845342 conn(0x7f320c037810 0x7f320c039cd0 secure :-1 s=READY pgs=3807 cs=0 l=1 rev1=1 crypto rx=0x7f32201395e0 tx=0x7f3218009820 comp rx=0 tx=0).stop 2024-09-16T04:03:24.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.247+0000 7f3226f58640 10 monclient: shutdown 2024-09-16T04:03:24.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.247+0000 7f3226f58640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:03:24.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.247+0000 7f3226f58640 1 -- 172.21.15.42:0/3764897876 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f322009ca60 msgr2=0x7f32201ca640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:03:24.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.247+0000 7f3226f58640 1 --2- 172.21.15.42:0/3764897876 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f322009ca60 0x7f32201ca640 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f32100ae040 tx=0x7f32100191c0 comp rx=0 tx=0).stop 2024-09-16T04:03:24.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.247+0000 7f32254f5640 1 -- 172.21.15.42:0/3764897876 reap_dead start 2024-09-16T04:03:24.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.247+0000 7f32254f5640 1 -- 172.21.15.42:0/3764897876 reap_dead start 2024-09-16T04:03:24.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.247+0000 7f3226f58640 1 -- 172.21.15.42:0/3764897876 shutdown_connections 2024-09-16T04:03:24.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.247+0000 7f3226f58640 1 -- 172.21.15.42:0/3764897876 >> 172.21.15.42:0/3764897876 conn(0x7f322006bc40 msgr2=0x7f32200a3e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:03:24.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.247+0000 7f3226f58640 1 -- 172.21.15.42:0/3764897876 shutdown_connections 2024-09-16T04:03:24.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.247+0000 7f3226f58640 1 -- 172.21.15.42:0/3764897876 wait complete. 2024-09-16T04:03:24.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.247+0000 7f3226f58640 1 librados: shutdown 2024-09-16T04:03:24.283 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:03:24.283 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T04:03:24.284 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T04:03:24.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.395+0000 7fb19ddca640 10 monclient: get_monmap_and_config 2024-09-16T04:03:24.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.395+0000 7fb19ddca640 10 monclient: build_initial_monmap 2024-09-16T04:03:24.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.395+0000 7fb19ddca640 10 monclient: monmap: 2024-09-16T04:03:24.399 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:03:24.399 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:03:24.399 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:03:24.397115+0000 2024-09-16T04:03:24.399 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:03:24.397115+0000 2024-09-16T04:03:24.399 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:03:24.399 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:03:24.399 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:03:24.399 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:03:24.399 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:03:24.399 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:03:24.399 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:03:24.399 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:03:24.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.395+0000 7fb19ddca640 1 Processor -- start 2024-09-16T04:03:24.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.395+0000 7fb19ddca640 1 -- start start 2024-09-16T04:03:24.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.395+0000 7fb19ddca640 10 monclient: init 2024-09-16T04:03:24.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.395+0000 7fb19ddca640 10 monclient: _reopen_session rank -1 2024-09-16T04:03:24.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.395+0000 7fb19ddca640 10 monclient: _add_conns ranks=[1,3,0,2,4] 2024-09-16T04:03:24.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.395+0000 7fb19ddca640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb1980a1330 addr v1:172.21.15.154:6789/0 2024-09-16T04:03:24.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.395+0000 7fb19ddca640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb1980a1710 0x7fb1980a1b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:03:24.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.395+0000 7fb19ddca640 10 monclient(hunting): picked mon.noname-a con 0x7fb1980a1710 addr v2:172.21.15.42:3300/0 2024-09-16T04:03:24.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.395+0000 7fb19ddca640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb1981393a0 addr v1:172.21.15.42:6789/0 2024-09-16T04:03:24.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.395+0000 7fb19ddca640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb19809e4c0 con 0x7fb1981393a0 2024-09-16T04:03:24.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.395+0000 7fb19ddca640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1980a2080 con 0x7fb1980a1330 2024-09-16T04:03:24.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.395+0000 7fb19ddca640 10 monclient(hunting): start opening mon connection 2024-09-16T04:03:24.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.395+0000 7fb19ddca640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7fb19809edf0 con 0x7fb1980a1710 2024-09-16T04:03:24.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.395+0000 7fb19ddca640 10 monclient(hunting): _renew_subs 2024-09-16T04:03:24.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.395+0000 7fb19ddca640 10 monclient(hunting): authenticate will time out at 2251.366646s 2024-09-16T04:03:24.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.395+0000 7fb196ffd640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb1980a1710 0x7fb1980a1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:03:24.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.395+0000 7fb196ffd640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb1980a1710 0x7fb1980a1b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:42530/0 (socket says 172.21.15.42:42530) 2024-09-16T04:03:24.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.395+0000 7fb196ffd640 1 -- 172.21.15.42:0/2501197276 learned_addr learned my addr 172.21.15.42:0/2501197276 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:03:24.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.395+0000 7fb196ffd640 10 monclient(hunting): get_auth_request con 0x7fb1980a1710 auth_method 0 2024-09-16T04:03:24.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.395+0000 7fb196ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:03:24.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.395+0000 7fb196ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:03:24.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.395+0000 7fb196ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:03:24.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.395+0000 7fb196ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:03:24.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.395+0000 7fb196ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:03:24.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.395+0000 7fb196ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:03:24.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.395+0000 7fb196ffd640 10 monclient(hunting): handle_auth_done global_id 825684 payload 274 2024-09-16T04:03:24.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.395+0000 7fb196ffd640 1 -- 172.21.15.42:0/2501197276 >> v1:172.21.15.154:6789/0 conn(0x7fb1980a1330 legacy=0x7fb19809f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:03:24.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.395+0000 7fb196ffd640 1 -- 172.21.15.42:0/2501197276 >> v1:172.21.15.42:6789/0 conn(0x7fb1981393a0 legacy=0x7fb19813b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:03:24.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.395+0000 7fb196ffd640 10 monclient: _finish_hunting 0 2024-09-16T04:03:24.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.395+0000 7fb196ffd640 1 monclient: found mon.noname-a 2024-09-16T04:03:24.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.395+0000 7fb196ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:03:24.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.395+0000 7fb196ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:03:24.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.395+0000 7fb196ffd640 1 -- 172.21.15.42:0/2501197276 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1980a3170 con 0x7fb1980a1710 2024-09-16T04:03:24.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.399+0000 7fb196ffd640 1 --2- 172.21.15.42:0/2501197276 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb1980a1710 0x7fb1980a1b40 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fb188009870 tx=0x7fb18801c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7de8926fe82b6dc5 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:03:24.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.399+0000 7fb1967fc640 1 -- 172.21.15.42:0/2501197276 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1880b2070 con 0x7fb1980a1710 2024-09-16T04:03:24.413 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.399+0000 7fb1967fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:03:24.413 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.399+0000 7fb1967fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:03:24.413 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.399+0000 7fb1967fc640 10 monclient: dump: 2024-09-16T04:03:24.413 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:03:24.413 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:03:24.413 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:24.413 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:24.413 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:03:24.413 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:03:24.413 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:03:24.413 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:03:24.413 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:03:24.413 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:03:24.414 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.399+0000 7fb1967fc640 10 monclient: _finish_auth 0 2024-09-16T04:03:24.414 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.399+0000 7fb1967fc640 10 monclient: _check_auth_tickets 2024-09-16T04:03:24.414 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.399+0000 7fb1967fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:03:24.414 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.399+0000 7fb1967fc640 1 -- 172.21.15.42:0/2501197276 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb1880ad680 con 0x7fb1980a1710 2024-09-16T04:03:24.414 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.399+0000 7fb1967fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:03:24.414 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.399+0000 7fb1967fc640 1 -- 172.21.15.42:0/2501197276 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1880ada50 con 0x7fb1980a1710 2024-09-16T04:03:24.414 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.399+0000 7fb1967fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:03:24.414 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.399+0000 7fb1967fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:03:24.414 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.399+0000 7fb1967fc640 10 monclient: dump: 2024-09-16T04:03:24.414 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:03:24.414 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:03:24.414 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:24.414 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:24.415 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:03:24.415 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:03:24.415 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:03:24.415 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:03:24.415 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:03:24.415 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:03:24.415 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.399+0000 7fb19ddca640 5 monclient: authenticate success, global_id 825684 2024-09-16T04:03:24.415 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.399+0000 7fb19ddca640 10 monclient: get_monmap_and_config success 2024-09-16T04:03:24.415 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.399+0000 7fb19ddca640 4 set_mon_vals no callback set 2024-09-16T04:03:24.415 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.399+0000 7fb19ddca640 10 monclient: shutdown 2024-09-16T04:03:24.415 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.399+0000 7fb19ddca640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:03:24.415 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.399+0000 7fb19ddca640 1 -- 172.21.15.42:0/2501197276 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb1980a1710 msgr2=0x7fb1980a1b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:03:24.416 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.399+0000 7fb19ddca640 1 --2- 172.21.15.42:0/2501197276 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb1980a1710 0x7fb1980a1b40 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fb188009870 tx=0x7fb18801c470 comp rx=0 tx=0).stop 2024-09-16T04:03:24.416 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.399+0000 7fb19ddca640 1 -- 172.21.15.42:0/2501197276 shutdown_connections 2024-09-16T04:03:24.416 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.399+0000 7fb19ddca640 1 --2- 172.21.15.42:0/2501197276 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb1980a1710 0x7fb1980a1b40 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:03:24.416 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.399+0000 7fb19ddca640 1 -- 172.21.15.42:0/2501197276 >> 172.21.15.42:0/2501197276 conn(0x7fb1980a2420 msgr2=0x7fb1980a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:03:24.416 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.399+0000 7fb19ddca640 1 -- 172.21.15.42:0/2501197276 shutdown_connections 2024-09-16T04:03:24.416 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.399+0000 7fb19ddca640 1 -- 172.21.15.42:0/2501197276 wait complete. 2024-09-16T04:03:24.416 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.399+0000 7fb19ddca640 10 monclient: build_initial_monmap 2024-09-16T04:03:24.416 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.399+0000 7fb19ddca640 10 monclient: monmap: 2024-09-16T04:03:24.416 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:03:24.416 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:03:24.416 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:03:24.416 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:03:24.416 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:03:24.417 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:03:24.417 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:03:24.417 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:03:24.417 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:03:24.417 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:03:24.417 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.399+0000 7fb19ddca640 1 librados: starting msgr at 2024-09-16T04:03:24.417 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.399+0000 7fb19ddca640 1 librados: starting objecter 2024-09-16T04:03:24.417 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.399+0000 7fb19ddca640 1 Processor -- start 2024-09-16T04:03:24.417 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.399+0000 7fb19ddca640 1 -- start start 2024-09-16T04:03:24.417 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.399+0000 7fb19ddca640 1 librados: setting wanted keys 2024-09-16T04:03:24.417 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.399+0000 7fb19ddca640 1 librados: calling monclient init 2024-09-16T04:03:24.417 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.399+0000 7fb19ddca640 10 monclient: init 2024-09-16T04:03:24.417 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.399+0000 7fb19ddca640 10 monclient: _reopen_session rank -1 2024-09-16T04:03:24.418 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.399+0000 7fb19ddca640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:03:24.418 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.399+0000 7fb19ddca640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb1980a1330 0x7fb1981c74f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:03:24.418 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.399+0000 7fb19ddca640 10 monclient(hunting): picked mon.noname-b con 0x7fb1980a1330 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:03:24.418 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.399+0000 7fb19ddca640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb1980a1710 0x7fb198072050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:03:24.418 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.399+0000 7fb19ddca640 10 monclient(hunting): picked mon.noname-c con 0x7fb1980a1710 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:03:24.418 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.399+0000 7fb19ddca640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb1981393a0 0x7fb1981caa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:03:24.418 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.399+0000 7fb19ddca640 10 monclient(hunting): picked mon.noname-a con 0x7fb1981393a0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:03:24.418 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.399+0000 7fb19ddca640 10 monclient(hunting): start opening mon connection 2024-09-16T04:03:24.418 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.399+0000 7fb19ddca640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fb19809be60 con 0x7fb1981393a0 2024-09-16T04:03:24.418 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.399+0000 7fb19ddca640 10 monclient(hunting): start opening mon connection 2024-09-16T04:03:24.418 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.399+0000 7fb19ddca640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fb19809bce0 con 0x7fb1980a1330 2024-09-16T04:03:24.418 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.399+0000 7fb19ddca640 10 monclient(hunting): start opening mon connection 2024-09-16T04:03:24.418 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.399+0000 7fb19ddca640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fb19809bfe0 con 0x7fb1980a1710 2024-09-16T04:03:24.419 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.399+0000 7fb19ddca640 10 monclient(hunting): _renew_subs 2024-09-16T04:03:24.419 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.399+0000 7fb19ddca640 10 monclient(hunting): authenticate will time out at 2251.371820s 2024-09-16T04:03:24.419 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.399+0000 7fb196ffd640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb1980a1710 0x7fb198072050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:03:24.419 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.399+0000 7fb197fff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb1981393a0 0x7fb1981caa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:03:24.419 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.403+0000 7fb196ffd640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb1980a1710 0x7fb198072050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:60576/0 (socket says 172.21.15.42:60576) 2024-09-16T04:03:24.419 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.403+0000 7fb197fff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb1981393a0 0x7fb1981caa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:42540/0 (socket says 172.21.15.42:42540) 2024-09-16T04:03:24.419 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.403+0000 7fb196ffd640 1 -- 172.21.15.42:0/2637093347 learned_addr learned my addr 172.21.15.42:0/2637093347 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:03:24.419 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.403+0000 7fb196ffd640 10 monclient(hunting): get_auth_request con 0x7fb1980a1710 auth_method 0 2024-09-16T04:03:24.419 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.403+0000 7fb196ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:03:24.419 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.403+0000 7fb196ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:03:24.419 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.403+0000 7fb196ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:03:24.419 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.403+0000 7fb197fff640 10 monclient(hunting): get_auth_request con 0x7fb1981393a0 auth_method 0 2024-09-16T04:03:24.420 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.403+0000 7fb197fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:03:24.420 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.403+0000 7fb197fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:03:24.420 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.403+0000 7fb197fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:03:24.420 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.403+0000 7fb196ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:03:24.420 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.403+0000 7fb196ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:03:24.420 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.403+0000 7fb196ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:03:24.420 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.403+0000 7fb197fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:03:24.420 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.403+0000 7fb197fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:03:24.420 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.403+0000 7fb197fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:03:24.420 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.403+0000 7fb1977fe640 1 --2- 172.21.15.42:0/2637093347 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb1980a1330 0x7fb1981c74f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:03:24.420 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.403+0000 7fb1977fe640 10 monclient(hunting): get_auth_request con 0x7fb1980a1330 auth_method 0 2024-09-16T04:03:24.420 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.403+0000 7fb1977fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:03:24.420 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.403+0000 7fb1977fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:03:24.421 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.403+0000 7fb1977fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:03:24.421 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.403+0000 7fb197fff640 10 monclient(hunting): handle_auth_done global_id 825687 payload 824 2024-09-16T04:03:24.421 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.403+0000 7fb197fff640 1 -- 172.21.15.42:0/2637093347 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb1980a1710 msgr2=0x7fb198072050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:03:24.421 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.403+0000 7fb197fff640 1 --2- 172.21.15.42:0/2637093347 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb1980a1710 0x7fb198072050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:03:24.421 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.403+0000 7fb197fff640 1 -- 172.21.15.42:0/2637093347 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb1980a1330 msgr2=0x7fb1981c74f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:03:24.421 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.403+0000 7fb197fff640 1 --2- 172.21.15.42:0/2637093347 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb1980a1330 0x7fb1981c74f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:03:24.421 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.403+0000 7fb197fff640 10 monclient: _finish_hunting 0 2024-09-16T04:03:24.421 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.403+0000 7fb197fff640 1 monclient: found mon.noname-a 2024-09-16T04:03:24.421 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.403+0000 7fb197fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:03:24.421 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.403+0000 7fb197fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:03:24.421 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.403+0000 7fb197fff640 1 -- 172.21.15.42:0/2637093347 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb198073a30 con 0x7fb1981393a0 2024-09-16T04:03:24.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.403+0000 7fb197fff640 1 --2- 172.21.15.42:0/2637093347 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb1981393a0 0x7fb1981caa10 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7fb19000c700 tx=0x7fb19000cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:03:24.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.403+0000 7fb18ffff640 10 client.?.objecter ms_handle_connect 0x7fb1981393a0 2024-09-16T04:03:24.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.403+0000 7fb18ffff640 10 client.?.objecter resend_mon_ops 2024-09-16T04:03:24.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.403+0000 7fb18ffff640 1 -- 172.21.15.42:0/2637093347 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb190003b80 con 0x7fb1981393a0 2024-09-16T04:03:24.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.403+0000 7fb18ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:03:24.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.403+0000 7fb18ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:03:24.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.403+0000 7fb18ffff640 10 monclient: dump: 2024-09-16T04:03:24.422 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:03:24.422 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:03:24.422 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:24.422 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:24.422 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:03:24.422 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:03:24.422 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:03:24.423 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:03:24.423 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:03:24.423 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:03:24.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.403+0000 7fb18ffff640 10 monclient: _finish_auth 0 2024-09-16T04:03:24.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.403+0000 7fb18ffff640 10 monclient: _check_auth_tickets 2024-09-16T04:03:24.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.403+0000 7fb18ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:03:24.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.403+0000 7fb18ffff640 1 -- 172.21.15.42:0/2637093347 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb190003d20 con 0x7fb1981393a0 2024-09-16T04:03:24.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.403+0000 7fb19ddca640 5 monclient: authenticate success, global_id 825687 2024-09-16T04:03:24.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.403+0000 7fb19ddca640 10 monclient: _renew_subs 2024-09-16T04:03:24.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.403+0000 7fb19ddca640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:03:24.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.403+0000 7fb19ddca640 1 -- 172.21.15.42:0/2637093347 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb198134020 con 0x7fb1981393a0 2024-09-16T04:03:24.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.403+0000 7fb18ffff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:03:24.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.403+0000 7fb18ffff640 1 -- 172.21.15.42:0/2637093347 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb19000aa30 con 0x7fb1981393a0 2024-09-16T04:03:24.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.403+0000 7fb18ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:03:24.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.403+0000 7fb18ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:03:24.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.403+0000 7fb19ddca640 10 client.825687.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:03:24.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.403+0000 7fb18ffff640 10 monclient: dump: 2024-09-16T04:03:24.424 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:03:24.424 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:03:24.424 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:24.424 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:24.424 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:03:24.424 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:03:24.424 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:03:24.424 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:03:24.424 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:03:24.424 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:03:24.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.403+0000 7fb1957fa640 4 set_mon_vals no callback set 2024-09-16T04:03:24.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.403+0000 7fb19ddca640 10 monclient: _renew_subs 2024-09-16T04:03:24.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.403+0000 7fb19ddca640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:03:24.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.403+0000 7fb19ddca640 1 -- 172.21.15.42:0/2637093347 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1981c7f50 con 0x7fb1981393a0 2024-09-16T04:03:24.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.403+0000 7fb19ddca640 1 librados: init done 2024-09-16T04:03:24.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.403+0000 7fb19ddca640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:03:24.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.403+0000 7fb19ddca640 1 --2- 172.21.15.42:0/2637093347 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb198072590 0x7fb1981caf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:03:24.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.403+0000 7fb19ddca640 10 monclient(hunting): start opening mon connection 2024-09-16T04:03:24.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.403+0000 7fb19ddca640 1 -- 172.21.15.42:0/2637093347 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fb19806e780 con 0x7fb198072590 2024-09-16T04:03:24.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.403+0000 7fb18ffff640 1 -- 172.21.15.42:0/2637093347 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fb19000abd0 con 0x7fb1981393a0 2024-09-16T04:03:24.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.403+0000 7fb1977fe640 1 --2- 172.21.15.42:0/2637093347 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb198072590 0x7fb1981caf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:03:24.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.403+0000 7fb1977fe640 10 monclient: get_auth_request con 0x7fb198072590 auth_method 0 2024-09-16T04:03:24.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.403+0000 7fb1977fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:03:24.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.403+0000 7fb1977fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:03:24.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.403+0000 7fb1977fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:03:24.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.403+0000 7fb18ffff640 1 --2- 172.21.15.42:0/2637093347 >> v2:172.21.15.154:6812/254845342 conn(0x7fb178037840 0x7fb178039d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:03:24.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.403+0000 7fb18ffff640 1 -- 172.21.15.42:0/2637093347 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(654..654 src has 451..654) ==== 6285+0+0 (secure 0 0 0) 0x7fb190048f40 con 0x7fb1981393a0 2024-09-16T04:03:24.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.403+0000 7fb18ffff640 10 client.825687.objecter ms_dispatch 0x7fb198000ba0 osd_map(654..654 src has 451..654) 2024-09-16T04:03:24.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.403+0000 7fb18ffff640 3 client.825687.objecter handle_osd_map got epochs [654,654] > 0 2024-09-16T04:03:24.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.403+0000 7fb18ffff640 3 client.825687.objecter handle_osd_map decoding full epoch 654 2024-09-16T04:03:24.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.403+0000 7fb1977fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:03:24.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.403+0000 7fb1977fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:03:24.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.403+0000 7fb18ffff640 20 client.825687.objecter dump_active .. 0 homeless 2024-09-16T04:03:24.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.403+0000 7fb1977fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:03:24.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.403+0000 7fb196ffd640 1 --2- 172.21.15.42:0/2637093347 >> v2:172.21.15.154:6812/254845342 conn(0x7fb178037840 0x7fb178039d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:03:24.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.403+0000 7fb196ffd640 10 monclient: get_auth_request con 0x7fb178037840 auth_method 0 2024-09-16T04:03:24.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.403+0000 7fb1977fe640 10 monclient(hunting): handle_auth_done global_id 904732 payload 824 2024-09-16T04:03:24.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.403+0000 7fb1977fe640 1 -- 172.21.15.42:0/2637093347 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb1980a4930 con 0x7fb198072590 2024-09-16T04:03:24.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.403+0000 7fb18ffff640 10 client.825687.objecter ms_handle_connect 0x7fb198072590 2024-09-16T04:03:24.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.403+0000 7fb18ffff640 10 client.825687.objecter resend_mon_ops 2024-09-16T04:03:24.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.403+0000 7fb1977fe640 1 --2- 172.21.15.42:0/2637093347 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb198072590 0x7fb1981caf50 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fb18000e450 tx=0x7fb18000e910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:03:24.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.403+0000 7fb18ffff640 1 -- 172.21.15.42:0/2637093347 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb180007240 con 0x7fb198072590 2024-09-16T04:03:24.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.403+0000 7fb18ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:03:24.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.403+0000 7fb18ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:03:24.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.403+0000 7fb18ffff640 10 monclient: dump: 2024-09-16T04:03:24.427 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:03:24.427 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:03:24.427 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:24.427 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:24.427 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:03:24.427 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:03:24.427 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:03:24.428 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:03:24.428 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:03:24.428 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:03:24.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.403+0000 7fb18ffff640 10 client.825687.objecter ms_handle_connect 0x7fb178037840 2024-09-16T04:03:24.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.403+0000 7fb196ffd640 1 --2- 172.21.15.42:0/2637093347 >> v2:172.21.15.154:6812/254845342 conn(0x7fb178037840 0x7fb178039d00 secure :-1 s=READY pgs=3808 cs=0 l=1 rev1=1 crypto rx=0x7fb1880b0040 tx=0x7fb1880a7000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:03:24.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.407+0000 7fb18ffff640 1 -- 172.21.15.42:0/2637093347 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fb180007240 con 0x7fb198072590 2024-09-16T04:03:24.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.407+0000 7fb18ffff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:03:24.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.407+0000 7fb18ffff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:03:24.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.407+0000 7fb18ffff640 1 -- 172.21.15.42:0/2637093347 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb198072590 msgr2=0x7fb1981caf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:03:24.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.407+0000 7fb18ffff640 1 --2- 172.21.15.42:0/2637093347 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb198072590 0x7fb1981caf50 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fb18000e450 tx=0x7fb18000e910 comp rx=0 tx=0).stop 2024-09-16T04:03:24.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.407+0000 7fb18ffff640 1 -- 172.21.15.42:0/2637093347 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb198072590 msgr2=0x7fb1981caf50 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:03:24.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.407+0000 7fb18ffff640 1 --2- 172.21.15.42:0/2637093347 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb198072590 0x7fb1981caf50 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:03:24.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.411+0000 7fb19ddca640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:03:24.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.411+0000 7fb19ddca640 1 --2- 172.21.15.42:0/2637093347 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb1981cb490 0x7fb198136390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:03:24.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.411+0000 7fb19ddca640 10 monclient(hunting): start opening mon connection 2024-09-16T04:03:24.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.411+0000 7fb19ddca640 1 -- 172.21.15.42:0/2637093347 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fb1981cb8a0 con 0x7fb1981cb490 2024-09-16T04:03:24.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.411+0000 7fb1977fe640 1 --2- 172.21.15.42:0/2637093347 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb1981cb490 0x7fb198136390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:03:24.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.411+0000 7fb1977fe640 10 monclient: get_auth_request con 0x7fb1981cb490 auth_method 0 2024-09-16T04:03:24.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.411+0000 7fb1977fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:03:24.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.411+0000 7fb1977fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:03:24.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.411+0000 7fb1977fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:03:24.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.411+0000 7fb1977fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:03:24.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.411+0000 7fb1977fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:03:24.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.411+0000 7fb1977fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:03:24.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.411+0000 7fb1977fe640 10 monclient(hunting): handle_auth_done global_id 904735 payload 824 2024-09-16T04:03:24.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.411+0000 7fb1977fe640 1 -- 172.21.15.42:0/2637093347 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fb1980a1b40 con 0x7fb1981cb490 2024-09-16T04:03:24.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.411+0000 7fb18ffff640 10 client.825687.objecter ms_handle_connect 0x7fb1981cb490 2024-09-16T04:03:24.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.411+0000 7fb18ffff640 10 client.825687.objecter resend_mon_ops 2024-09-16T04:03:24.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.411+0000 7fb1977fe640 1 --2- 172.21.15.42:0/2637093347 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb1981cb490 0x7fb198136390 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fb18000efd0 tx=0x7fb180003770 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:03:24.432 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.411+0000 7fb18ffff640 1 -- 172.21.15.42:0/2637093347 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb18001ea40 con 0x7fb1981cb490 2024-09-16T04:03:24.432 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.411+0000 7fb18ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:03:24.432 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.415+0000 7fb18ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:03:24.432 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.415+0000 7fb18ffff640 10 monclient: dump: 2024-09-16T04:03:24.432 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:03:24.432 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:03:24.432 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:24.432 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:24.432 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:03:24.432 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:03:24.432 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:03:24.432 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:03:24.432 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:03:24.433 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:03:24.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.415+0000 7fb18ffff640 1 -- 172.21.15.42:0/2637093347 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4713 (secure 0 0 0) 0x7fb18001ea20 con 0x7fb1981cb490 2024-09-16T04:03:24.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.415+0000 7fb18ffff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:03:24.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.415+0000 7fb18ffff640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:03:24.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.415+0000 7fb18ffff640 1 -- 172.21.15.42:0/2637093347 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb1981cb490 msgr2=0x7fb198136390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:03:24.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.415+0000 7fb18ffff640 1 --2- 172.21.15.42:0/2637093347 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb1981cb490 0x7fb198136390 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fb18000efd0 tx=0x7fb180003770 comp rx=0 tx=0).stop 2024-09-16T04:03:24.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.415+0000 7fb18ffff640 1 -- 172.21.15.42:0/2637093347 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb1981cb490 msgr2=0x7fb198136390 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:03:24.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.415+0000 7fb18ffff640 1 --2- 172.21.15.42:0/2637093347 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb1981cb490 0x7fb198136390 secure :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fb18000efd0 tx=0x7fb180003770 comp rx=0 tx=0).stop 2024-09-16T04:03:24.433 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T04:03:24.433 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:03:24.434 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:03:24.434 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T04:03:24.434 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 190, 2024-09-16T04:03:24.434 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 332592, 2024-09-16T04:03:24.434 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T04:03:24.434 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T04:03:24.434 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T04:03:24.434 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T04:03:24.434 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:03:24.434 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 7, 2024-09-16T04:03:24.434 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:03:24.434 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T04:03:24.434 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T04:03:24.435 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:03:24.435 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:03:24.435 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:03:24.435 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:03:24.435 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:03:24.435 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:03:24.435 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:03:24.435 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:03:24.435 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:03:24.435 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:03:24.435 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:03:24.435 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:03:24.435 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:03:24.436 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T04:03:24.436 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:03:24.436 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:03:24.436 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:03:24.436 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:03:24.436 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:03:24.436 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:03:24.436 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:03:24.436 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:03:24.436 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:03:24.436 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:03:24.436 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:03:24.436 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:03:24.436 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:03:24.437 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T04:03:24.437 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T04:03:24.437 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T04:03:24.437 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T04:03:24.437 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T04:03:24.437 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T04:03:24.437 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:03:24.437 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:03:24.437 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T04:03:24.437 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T04:03:24.437 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T04:03:24.437 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T04:03:24.437 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T04:03:24.438 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T04:03:24.438 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T04:03:24.438 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:03:24.438 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T04:03:24.438 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:03:24.438 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:03:24.438 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:03:24.438 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:03:24.438 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:03:24.438 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:03:24.438 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:03:24.438 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:03:24.438 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:03:24.438 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:03:24.439 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:03:24.439 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:03:24.439 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T04:03:24.439 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:03:24.439 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T04:03:24.439 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:03:24.439 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:03:24.439 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:03:24.439 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:03:24.439 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:03:24.439 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:03:24.439 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:03:24.439 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T04:03:24.439 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:03:24.440 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:03:24.440 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:03:24.440 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:03:24.440 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T04:03:24.440 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:03:24.440 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:03:24.440 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:03:24.440 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:03:24.440 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T04:03:24.440 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T04:03:24.440 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:03:24.440 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:03:24.440 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:03:24.441 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:03:24.441 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:03:24.441 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:03:24.441 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:03:24.441 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:03:24.441 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:03:24.441 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:03:24.441 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:03:24.441 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T04:03:24.441 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:03:24.441 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:03:24.441 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:03:24.441 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:03:24.442 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T04:03:24.442 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:03:24.442 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:03:24.442 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:03:24.442 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:03:24.442 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T04:03:24.442 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T04:03:24.442 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:03:24.442 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:03:24.442 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:03:24.442 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:03:24.442 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:03:24.442 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:03:24.442 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:03:24.443 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:03:24.443 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:03:24.443 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:03:24.443 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:03:24.443 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T04:03:24.443 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:03:24.443 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:03:24.443 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:03:24.443 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:03:24.443 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T04:03:24.443 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:03:24.443 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:03:24.443 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:03:24.443 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:03:24.444 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T04:03:24.444 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T04:03:24.444 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:03:24.444 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:03:24.444 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:03:24.444 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:03:24.444 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:03:24.444 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:03:24.444 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T04:03:24.444 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T04:03:24.444 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:03:24.444 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:03:24.444 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:03:24.445 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:03:24.445 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:03:24.445 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:03:24.445 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T04:03:24.445 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:03:24.445 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:03:24.445 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:03:24.445 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:03:24.445 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:03:24.445 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:03:24.445 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T04:03:24.445 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:03:24.445 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:03:24.446 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:03:24.446 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 2 2024-09-16T04:03:24.446 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:03:24.446 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:03:24.446 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:03:24.446 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T04:03:24.446 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T04:03:24.448 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.415+0000 7fb19ddca640 10 librados: watch_flush enter 2024-09-16T04:03:24.448 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.415+0000 7fb19ddca640 10 librados: watch_flush exit 2024-09-16T04:03:24.448 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.415+0000 7fb19ddca640 20 client.825687.objecter shutdown clearing up homeless session... 2024-09-16T04:03:24.448 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.415+0000 7fb19ddca640 10 client.825687.objecter successfully canceled tick 2024-09-16T04:03:24.448 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.415+0000 7fb19ddca640 1 -- 172.21.15.42:0/2637093347 >> v2:172.21.15.154:6812/254845342 conn(0x7fb178037840 msgr2=0x7fb178039d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:03:24.448 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.415+0000 7fb19ddca640 1 --2- 172.21.15.42:0/2637093347 >> v2:172.21.15.154:6812/254845342 conn(0x7fb178037840 0x7fb178039d00 secure :-1 s=READY pgs=3808 cs=0 l=1 rev1=1 crypto rx=0x7fb1880b0040 tx=0x7fb1880a7000 comp rx=0 tx=0).stop 2024-09-16T04:03:24.448 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.415+0000 7fb19ddca640 10 monclient: shutdown 2024-09-16T04:03:24.448 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.415+0000 7fb19ddca640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:03:24.448 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.415+0000 7fb19ddca640 1 -- 172.21.15.42:0/2637093347 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb1981393a0 msgr2=0x7fb1981caa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:03:24.449 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.415+0000 7fb19ddca640 1 --2- 172.21.15.42:0/2637093347 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb1981393a0 0x7fb1981caa10 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7fb19000c700 tx=0x7fb19000cbc0 comp rx=0 tx=0).stop 2024-09-16T04:03:24.449 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.415+0000 7fb197fff640 1 -- 172.21.15.42:0/2637093347 reap_dead start 2024-09-16T04:03:24.449 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.415+0000 7fb19ddca640 1 -- 172.21.15.42:0/2637093347 shutdown_connections 2024-09-16T04:03:24.449 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.415+0000 7fb19ddca640 1 --2- 172.21.15.42:0/2637093347 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb1981393a0 0x7fb1981caa10 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:03:24.449 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.415+0000 7fb19ddca640 1 -- 172.21.15.42:0/2637093347 >> 172.21.15.42:0/2637093347 conn(0x7fb1980a2420 msgr2=0x7fb1980a3720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:03:24.449 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.415+0000 7fb19ddca640 1 -- 172.21.15.42:0/2637093347 shutdown_connections 2024-09-16T04:03:24.449 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.415+0000 7fb19ddca640 1 -- 172.21.15.42:0/2637093347 wait complete. 2024-09-16T04:03:24.449 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.415+0000 7fb19ddca640 1 librados: shutdown 2024-09-16T04:03:24.449 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:03:24.450 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-09-16T04:03:24.450 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-16T04:03:24.450 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-09-16T04:03:24.450 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-16T04:03:24.450 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-16T04:03:24.450 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T04:03:24.474 INFO:tasks.ceph.mon.a:Stopped 2024-09-16T04:03:24.474 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T04:03:24.474 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:03:24.643 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.639+0000 7f6b0a716640 10 monclient: get_monmap_and_config 2024-09-16T04:03:24.643 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.639+0000 7f6b0a716640 10 monclient: build_initial_monmap 2024-09-16T04:03:24.643 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.639+0000 7f6b0a716640 10 monclient: monmap: 2024-09-16T04:03:24.643 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:03:24.644 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:03:24.644 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:03:24.641802+0000 2024-09-16T04:03:24.644 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:03:24.641802+0000 2024-09-16T04:03:24.644 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:03:24.644 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:03:24.644 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:03:24.644 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:03:24.644 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:03:24.644 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:03:24.644 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:03:24.644 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:03:24.646 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.639+0000 7f6b0a716640 1 Processor -- start 2024-09-16T04:03:24.647 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.639+0000 7f6b0a716640 1 -- start start 2024-09-16T04:03:24.647 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.639+0000 7f6b0a716640 10 monclient: init 2024-09-16T04:03:24.647 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.639+0000 7f6b0a716640 10 monclient: _reopen_session rank -1 2024-09-16T04:03:24.647 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.639+0000 7f6b0a716640 10 monclient: _add_conns ranks=[2,0,4,3,1] 2024-09-16T04:03:24.647 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.639+0000 7f6b0a716640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f6b040a1330 0x7f6b040a1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:03:24.647 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.639+0000 7f6b0a716640 10 monclient(hunting): picked mon.noname-b con 0x7f6b040a1330 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:03:24.647 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.639+0000 7f6b0a716640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6b0406dac0 addr v1:172.21.15.42:6789/0 2024-09-16T04:03:24.647 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.639+0000 7f6b0a716640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f6b040a1c50 0x7f6b04135810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:03:24.647 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.639+0000 7f6b0a716640 10 monclient(hunting): picked mon.noname-c con 0x7f6b040a1c50 addr v2:172.21.15.154:3300/0 2024-09-16T04:03:24.647 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.639+0000 7f6b0a716640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6b0409c090 con 0x7f6b0406dac0 2024-09-16T04:03:24.647 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.639+0000 7f6b0a716640 10 monclient(hunting): start opening mon connection 2024-09-16T04:03:24.647 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.639+0000 7f6b0a716640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f6b0409c890 con 0x7f6b040a1c50 2024-09-16T04:03:24.647 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.639+0000 7f6b0a716640 10 monclient(hunting): start opening mon connection 2024-09-16T04:03:24.648 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.639+0000 7f6b0a716640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f6b0409f610 con 0x7f6b040a1330 2024-09-16T04:03:24.648 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.639+0000 7f6b0a716640 10 monclient(hunting): _renew_subs 2024-09-16T04:03:24.648 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.639+0000 7f6b0a716640 10 monclient(hunting): authenticate will time out at 2251.611223s 2024-09-16T04:03:24.648 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.639+0000 7f6b037fe640 1 -- >> v1:172.21.15.42:6789/0 conn(0x7f6b0406dac0 legacy=0x7f6b04135100 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.42:6789/0 2024-09-16T04:03:24.648 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.639+0000 7f6b03fff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f6b040a1330 0x7f6b040a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:03:24.648 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.639+0000 7f6b03fff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f6b040a1330 0x7f6b040a1710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:60590/0 (socket says 172.21.15.42:60590) 2024-09-16T04:03:24.648 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.639+0000 7f6b03fff640 1 -- 172.21.15.42:0/567602364 learned_addr learned my addr 172.21.15.42:0/567602364 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:03:24.648 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.639+0000 7f6b03fff640 10 monclient(hunting): get_auth_request con 0x7f6b040a1330 auth_method 0 2024-09-16T04:03:24.648 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.639+0000 7f6b03fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:03:24.648 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.639+0000 7f6b03fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:03:24.648 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.639+0000 7f6b03fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:03:24.648 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.639+0000 7f6b03fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:03:24.648 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.639+0000 7f6b03fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:03:24.649 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.639+0000 7f6b03fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:03:24.649 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.639+0000 7f6b08cb3640 1 --2- 172.21.15.42:0/567602364 >> v2:172.21.15.154:3300/0 conn(0x7f6b040a1c50 0x7f6b04135810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:03:24.649 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.639+0000 7f6b08cb3640 10 monclient(hunting): get_auth_request con 0x7f6b040a1c50 auth_method 0 2024-09-16T04:03:24.649 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.639+0000 7f6b08cb3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:03:24.649 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.639+0000 7f6b08cb3640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:03:24.649 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.639+0000 7f6b08cb3640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:03:24.649 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.639+0000 7f6b03fff640 10 monclient(hunting): handle_auth_done global_id 974141 payload 274 2024-09-16T04:03:24.649 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.639+0000 7f6b03fff640 1 -- 172.21.15.42:0/567602364 >> v2:172.21.15.154:3300/0 conn(0x7f6b040a1c50 msgr2=0x7f6b04135810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:03:24.649 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.639+0000 7f6b03fff640 1 --2- 172.21.15.42:0/567602364 >> v2:172.21.15.154:3300/0 conn(0x7f6b040a1c50 0x7f6b04135810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:03:24.649 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.639+0000 7f6b03fff640 1 -- 172.21.15.42:0/567602364 >> v1:172.21.15.42:6789/0 conn(0x7f6b0406dac0 legacy=0x7f6b04135100 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:03:24.649 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.639+0000 7f6b03fff640 10 monclient: _finish_hunting 0 2024-09-16T04:03:24.649 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.639+0000 7f6b03fff640 1 monclient: found mon.noname-b 2024-09-16T04:03:24.649 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.639+0000 7f6b03fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:03:24.650 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.639+0000 7f6b03fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T04:03:24.650 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.639+0000 7f6b03fff640 1 -- 172.21.15.42:0/567602364 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b0409f8f0 con 0x7f6b040a1330 2024-09-16T04:03:24.655 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.643+0000 7f6b03fff640 1 --2- 172.21.15.42:0/567602364 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f6b040a1330 0x7f6b040a1710 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f6af4009870 tx=0x7f6af401c440 comp rx=0 tx=0).ready entity=mon.2 client_cookie=feb4a26c8f5891e9 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:03:24.655 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.643+0000 7f6b02ffd640 1 -- 172.21.15.42:0/567602364 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6af40ab620 con 0x7f6b040a1330 2024-09-16T04:03:24.656 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.643+0000 7f6b02ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:03:24.656 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.643+0000 7f6b02ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:03:24.656 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.643+0000 7f6b02ffd640 10 monclient: dump: 2024-09-16T04:03:24.656 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:03:24.656 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:03:24.656 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:24.656 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:24.656 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:03:24.656 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:03:24.656 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:03:24.656 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:03:24.657 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:03:24.657 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:03:24.657 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.643+0000 7f6b02ffd640 10 monclient: _finish_auth 0 2024-09-16T04:03:24.657 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.643+0000 7f6b02ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:03:24.657 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.643+0000 7f6b02ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:03:24.657 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.643+0000 7f6b02ffd640 1 -- 172.21.15.42:0/567602364 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6af40b0070 con 0x7f6b040a1330 2024-09-16T04:03:24.657 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.643+0000 7f6b0a716640 5 monclient: authenticate success, global_id 974141 2024-09-16T04:03:24.657 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.643+0000 7f6b0a716640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:03:24.657 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.643+0000 7f6b02ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:03:24.657 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.643+0000 7f6b0a716640 10 monclient: get_monmap_and_config success 2024-09-16T04:03:24.657 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.643+0000 7f6b02ffd640 1 -- 172.21.15.42:0/567602364 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6af40ab9e0 con 0x7f6b040a1330 2024-09-16T04:03:24.657 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.643+0000 7f6b02ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:03:24.657 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.643+0000 7f6b0a716640 4 set_mon_vals no callback set 2024-09-16T04:03:24.658 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.643+0000 7f6b02ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:03:24.658 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.643+0000 7f6b02ffd640 10 monclient: dump: 2024-09-16T04:03:24.658 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:03:24.658 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:03:24.658 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:24.658 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:24.658 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:03:24.658 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:03:24.658 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:03:24.659 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:03:24.659 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:03:24.659 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:03:24.659 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.643+0000 7f6b0a716640 10 monclient: shutdown 2024-09-16T04:03:24.659 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.643+0000 7f6b0a716640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:03:24.659 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.643+0000 7f6b0a716640 1 -- 172.21.15.42:0/567602364 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f6b040a1330 msgr2=0x7f6b040a1710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:03:24.659 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.643+0000 7f6b0a716640 1 --2- 172.21.15.42:0/567602364 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f6b040a1330 0x7f6b040a1710 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f6af4009870 tx=0x7f6af401c440 comp rx=0 tx=0).stop 2024-09-16T04:03:24.659 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.643+0000 7f6b0a716640 1 -- 172.21.15.42:0/567602364 shutdown_connections 2024-09-16T04:03:24.659 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.643+0000 7f6b0a716640 1 --2- 172.21.15.42:0/567602364 >> v2:172.21.15.154:3300/0 conn(0x7f6b040a1c50 0x7f6b04135810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:03:24.659 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.643+0000 7f6b0a716640 1 --2- 172.21.15.42:0/567602364 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f6b040a1330 0x7f6b040a1710 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:03:24.659 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.643+0000 7f6b0a716640 1 -- 172.21.15.42:0/567602364 >> 172.21.15.42:0/567602364 conn(0x7f6b040a2420 msgr2=0x7f6b040a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:03:24.659 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.643+0000 7f6b0a716640 1 -- 172.21.15.42:0/567602364 shutdown_connections 2024-09-16T04:03:24.659 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.643+0000 7f6b0a716640 1 -- 172.21.15.42:0/567602364 wait complete. 2024-09-16T04:03:24.660 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.647+0000 7f6b0a716640 10 monclient: build_initial_monmap 2024-09-16T04:03:24.660 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.647+0000 7f6b0a716640 10 monclient: monmap: 2024-09-16T04:03:24.660 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:03:24.660 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:03:24.660 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:03:24.660 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:03:24.660 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:03:24.660 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:03:24.660 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:03:24.660 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:03:24.660 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:03:24.660 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:03:24.660 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.647+0000 7f6b0a716640 1 librados: starting msgr at 2024-09-16T04:03:24.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.647+0000 7f6b0a716640 1 librados: starting objecter 2024-09-16T04:03:24.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.647+0000 7f6b0a716640 1 Processor -- start 2024-09-16T04:03:24.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.647+0000 7f6b0a716640 1 -- start start 2024-09-16T04:03:24.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.647+0000 7f6b0a716640 1 librados: setting wanted keys 2024-09-16T04:03:24.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.647+0000 7f6b0a716640 1 librados: calling monclient init 2024-09-16T04:03:24.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.647+0000 7f6b0a716640 10 monclient: init 2024-09-16T04:03:24.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.647+0000 7f6b0a716640 10 monclient: _reopen_session rank -1 2024-09-16T04:03:24.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.647+0000 7f6b0a716640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:03:24.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.647+0000 7f6b0a716640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f6b040a1330 0x7f6b04134bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:03:24.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.647+0000 7f6b0a716640 10 monclient(hunting): picked mon.noname-c con 0x7f6b040a1330 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:03:24.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.647+0000 7f6b0a716640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6b040a1c50 0x7f6b041350f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:03:24.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.647+0000 7f6b0a716640 10 monclient(hunting): picked mon.noname-a con 0x7f6b040a1c50 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:03:24.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.647+0000 7f6b0a716640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6b0411bf70 0x7f6b0411c420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:03:24.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.647+0000 7f6b0a716640 10 monclient(hunting): picked mon.noname-b con 0x7f6b0411bf70 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:03:24.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.647+0000 7f6b0a716640 10 monclient(hunting): start opening mon connection 2024-09-16T04:03:24.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.647+0000 7f6b0a716640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f6b0409ef30 con 0x7f6b040a1c50 2024-09-16T04:03:24.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.647+0000 7f6b0a716640 10 monclient(hunting): start opening mon connection 2024-09-16T04:03:24.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.647+0000 7f6b0a716640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f6b0409edb0 con 0x7f6b0411bf70 2024-09-16T04:03:24.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.647+0000 7f6b0a716640 10 monclient(hunting): start opening mon connection 2024-09-16T04:03:24.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.647+0000 7f6b0a716640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f6b04135d50 con 0x7f6b040a1330 2024-09-16T04:03:24.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.647+0000 7f6b0a716640 10 monclient(hunting): _renew_subs 2024-09-16T04:03:24.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.647+0000 7f6b0a716640 10 monclient(hunting): authenticate will time out at 2251.617092s 2024-09-16T04:03:24.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.647+0000 7f6b037fe640 1 -- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6b040a1c50 msgr2=0x7f6b041350f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.42:3300/0 2024-09-16T04:03:24.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.647+0000 7f6b037fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6b040a1c50 0x7f6b041350f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:03:24.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.647+0000 7f6b03fff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f6b040a1330 0x7f6b04134bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:03:24.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.647+0000 7f6b03fff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f6b040a1330 0x7f6b04134bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:60592/0 (socket says 172.21.15.42:60592) 2024-09-16T04:03:24.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.647+0000 7f6b03fff640 1 -- 172.21.15.42:0/1803124504 learned_addr learned my addr 172.21.15.42:0/1803124504 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:03:24.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.647+0000 7f6b03fff640 10 monclient(hunting): get_auth_request con 0x7f6b040a1330 auth_method 0 2024-09-16T04:03:24.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.647+0000 7f6b03fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:03:24.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.647+0000 7f6b03fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:03:24.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.647+0000 7f6b03fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:03:24.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.647+0000 7f6b03fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:03:24.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.647+0000 7f6b03fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:03:24.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.647+0000 7f6b03fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:03:24.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.647+0000 7f6b08cb3640 1 --2- 172.21.15.42:0/1803124504 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6b0411bf70 0x7f6b0411c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:03:24.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.647+0000 7f6b08cb3640 10 monclient(hunting): get_auth_request con 0x7f6b0411bf70 auth_method 0 2024-09-16T04:03:24.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.647+0000 7f6b08cb3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:03:24.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.647+0000 7f6b08cb3640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:03:24.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.647+0000 7f6b08cb3640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:03:24.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.647+0000 7f6b03fff640 10 monclient(hunting): handle_auth_done global_id 974144 payload 824 2024-09-16T04:03:24.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.647+0000 7f6b03fff640 1 -- 172.21.15.42:0/1803124504 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6b0411bf70 msgr2=0x7f6b0411c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:03:24.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.647+0000 7f6b03fff640 1 --2- 172.21.15.42:0/1803124504 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6b0411bf70 0x7f6b0411c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:03:24.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.647+0000 7f6b03fff640 1 -- 172.21.15.42:0/1803124504 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6b040a1c50 msgr2=0x7f6b041350f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:03:24.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.647+0000 7f6b03fff640 1 --2- 172.21.15.42:0/1803124504 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6b040a1c50 0x7f6b041350f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:03:24.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.647+0000 7f6b03fff640 10 monclient: _finish_hunting 0 2024-09-16T04:03:24.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.647+0000 7f6b03fff640 1 monclient: found mon.noname-c 2024-09-16T04:03:24.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.647+0000 7f6b03fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:03:24.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.647+0000 7f6b03fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T04:03:24.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.647+0000 7f6b03fff640 1 -- 172.21.15.42:0/1803124504 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b0411f6b0 con 0x7f6b040a1330 2024-09-16T04:03:24.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.647+0000 7f6b03fff640 1 --2- 172.21.15.42:0/1803124504 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f6b040a1330 0x7f6b04134bb0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f6af401aaf0 tx=0x7f6af401ac80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:03:24.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.647+0000 7f6b00ff9640 10 client.?.objecter ms_handle_connect 0x7f6b040a1330 2024-09-16T04:03:24.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.647+0000 7f6b00ff9640 10 client.?.objecter resend_mon_ops 2024-09-16T04:03:24.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.647+0000 7f6b00ff9640 1 -- 172.21.15.42:0/1803124504 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6af40abdb0 con 0x7f6b040a1330 2024-09-16T04:03:24.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.647+0000 7f6b00ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:03:24.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.647+0000 7f6b00ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:03:24.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.647+0000 7f6b00ff9640 10 monclient: dump: 2024-09-16T04:03:24.666 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:03:24.666 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:03:24.666 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:24.666 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:24.666 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:03:24.666 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:03:24.667 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:03:24.667 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:03:24.667 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:03:24.667 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:03:24.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.647+0000 7f6b00ff9640 10 monclient: _finish_auth 0 2024-09-16T04:03:24.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.647+0000 7f6b00ff9640 10 monclient: _check_auth_tickets 2024-09-16T04:03:24.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.647+0000 7f6b00ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:03:24.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.647+0000 7f6b00ff9640 1 -- 172.21.15.42:0/1803124504 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6af40be3f0 con 0x7f6b040a1330 2024-09-16T04:03:24.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.647+0000 7f6b0a716640 5 monclient: authenticate success, global_id 974144 2024-09-16T04:03:24.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.647+0000 7f6b0a716640 10 monclient: _renew_subs 2024-09-16T04:03:24.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.647+0000 7f6b0a716640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:03:24.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.647+0000 7f6b0a716640 1 -- 172.21.15.42:0/1803124504 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b0411e200 con 0x7f6b040a1330 2024-09-16T04:03:24.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.647+0000 7f6b00ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T04:03:24.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.647+0000 7f6b00ff9640 1 -- 172.21.15.42:0/1803124504 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6af40be7e0 con 0x7f6b040a1330 2024-09-16T04:03:24.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.647+0000 7f6b00ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:03:24.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.647+0000 7f6b00ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:03:24.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.647+0000 7f6b00ff9640 10 monclient: dump: 2024-09-16T04:03:24.668 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:03:24.668 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:03:24.668 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:24.668 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:24.668 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:03:24.668 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:03:24.668 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:03:24.668 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:03:24.668 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:03:24.668 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:03:24.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.647+0000 7f6b0a716640 10 client.974144.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:03:24.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.647+0000 7f6b0a716640 10 monclient: _renew_subs 2024-09-16T04:03:24.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.647+0000 7f6b027fc640 4 set_mon_vals no callback set 2024-09-16T04:03:24.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.647+0000 7f6b0a716640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:03:24.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.647+0000 7f6b0a716640 1 -- 172.21.15.42:0/1803124504 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b0411f3d0 con 0x7f6b040a1330 2024-09-16T04:03:24.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.647+0000 7f6b0a716640 1 librados: init done 2024-09-16T04:03:24.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.647+0000 7f6b00ff9640 1 -- 172.21.15.42:0/1803124504 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f6af40c7440 con 0x7f6b040a1330 2024-09-16T04:03:24.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.647+0000 7f6b0a716640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:03:24.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.647+0000 7f6b0a716640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:03:24.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.647+0000 7f6b0a716640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:03:24.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.647+0000 7f6b0a716640 1 -- 172.21.15.42:0/1803124504 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b040a1710 con 0x7f6b040a1330 2024-09-16T04:03:24.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.647+0000 7f6b00ff9640 1 --2- 172.21.15.42:0/1803124504 >> v2:172.21.15.154:6812/254845342 conn(0x7f6af00377c0 0x7f6af0039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:03:24.670 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.647+0000 7f6b00ff9640 1 -- 172.21.15.42:0/1803124504 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(654..654 src has 451..654) ==== 6285+0+0 (secure 0 0 0) 0x7f6af40e3850 con 0x7f6b040a1330 2024-09-16T04:03:24.670 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.647+0000 7f6b00ff9640 10 client.974144.objecter ms_dispatch 0x7f6b04000ba0 osd_map(654..654 src has 451..654) 2024-09-16T04:03:24.670 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.647+0000 7f6b00ff9640 3 client.974144.objecter handle_osd_map got epochs [654,654] > 0 2024-09-16T04:03:24.670 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.647+0000 7f6b00ff9640 3 client.974144.objecter handle_osd_map decoding full epoch 654 2024-09-16T04:03:24.670 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.647+0000 7f6b00ff9640 20 client.974144.objecter dump_active .. 0 homeless 2024-09-16T04:03:24.670 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.647+0000 7f6b037fe640 1 --2- 172.21.15.42:0/1803124504 >> v2:172.21.15.154:6812/254845342 conn(0x7f6af00377c0 0x7f6af0039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:03:24.670 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.647+0000 7f6b037fe640 10 monclient: get_auth_request con 0x7f6af00377c0 auth_method 0 2024-09-16T04:03:24.670 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.651+0000 7f6b037fe640 1 --2- 172.21.15.42:0/1803124504 >> v2:172.21.15.154:6812/254845342 conn(0x7f6af00377c0 0x7f6af0039c80 secure :-1 s=READY pgs=3809 cs=0 l=1 rev1=1 crypto rx=0x7f6af8006fd0 tx=0x7f6af8006d40 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:03:24.670 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.651+0000 7f6b00ff9640 10 client.974144.objecter ms_handle_connect 0x7f6af00377c0 2024-09-16T04:03:24.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.655+0000 7f6b00ff9640 1 -- 172.21.15.42:0/1803124504 <== mon.2 v2:172.21.15.42:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f6af40160f0 con 0x7f6b040a1330 2024-09-16T04:03:24.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.655+0000 7f6b00ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:03:24.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.655+0000 7f6b00ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:03:24.883 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.879+0000 7f6b0a716640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:03:24.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.879+0000 7f6b0a716640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:03:24.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.879+0000 7f6b0a716640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:03:24.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.879+0000 7f6b0a716640 1 -- 172.21.15.42:0/1803124504 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f6b040a1710 con 0x7f6b040a1330 2024-09-16T04:03:24.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.879+0000 7f6b00ff9640 1 -- 172.21.15.42:0/1803124504 <== mon.2 v2:172.21.15.42:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7f6af40bd790 con 0x7f6b040a1330 2024-09-16T04:03:24.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.879+0000 7f6b00ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:03:24.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.879+0000 7f6b00ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:03:24.885 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":190,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:03:24.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.883+0000 7f6b0a716640 10 librados: watch_flush enter 2024-09-16T04:03:24.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.883+0000 7f6b0a716640 10 librados: watch_flush exit 2024-09-16T04:03:24.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.883+0000 7f6b0a716640 20 client.974144.objecter shutdown clearing up homeless session... 2024-09-16T04:03:24.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.883+0000 7f6b0a716640 10 client.974144.objecter successfully canceled tick 2024-09-16T04:03:24.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.883+0000 7f6b0a716640 1 -- 172.21.15.42:0/1803124504 >> v2:172.21.15.154:6812/254845342 conn(0x7f6af00377c0 msgr2=0x7f6af0039c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:03:24.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.883+0000 7f6b0a716640 1 --2- 172.21.15.42:0/1803124504 >> v2:172.21.15.154:6812/254845342 conn(0x7f6af00377c0 0x7f6af0039c80 secure :-1 s=READY pgs=3809 cs=0 l=1 rev1=1 crypto rx=0x7f6af8006fd0 tx=0x7f6af8006d40 comp rx=0 tx=0).stop 2024-09-16T04:03:24.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.883+0000 7f6b0a716640 10 monclient: shutdown 2024-09-16T04:03:24.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.883+0000 7f6b0a716640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:03:24.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.883+0000 7f6b0a716640 1 -- 172.21.15.42:0/1803124504 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f6b040a1330 msgr2=0x7f6b04134bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:03:24.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.883+0000 7f6b0a716640 1 --2- 172.21.15.42:0/1803124504 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f6b040a1330 0x7f6b04134bb0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f6af401aaf0 tx=0x7f6af401ac80 comp rx=0 tx=0).stop 2024-09-16T04:03:24.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.883+0000 7f6b0a716640 1 -- 172.21.15.42:0/1803124504 shutdown_connections 2024-09-16T04:03:24.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.883+0000 7f6b0a716640 1 --2- 172.21.15.42:0/1803124504 >> v2:172.21.15.154:6812/254845342 conn(0x7f6af00377c0 0x7f6af0039c80 unknown :-1 s=CLOSED pgs=3809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:03:24.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.883+0000 7f6b0a716640 1 --2- 172.21.15.42:0/1803124504 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6b0411bf70 0x7f6b0411c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:03:24.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.883+0000 7f6b0a716640 1 --2- 172.21.15.42:0/1803124504 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6b040a1c50 0x7f6b041350f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:03:24.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.883+0000 7f6b0a716640 1 --2- 172.21.15.42:0/1803124504 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f6b040a1330 0x7f6b04134bb0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:03:24.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.883+0000 7f6b0a716640 1 -- 172.21.15.42:0/1803124504 >> 172.21.15.42:0/1803124504 conn(0x7f6b040a2420 msgr2=0x7f6b0406f920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:03:24.888 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.883+0000 7f6b0a716640 1 -- 172.21.15.42:0/1803124504 shutdown_connections 2024-09-16T04:03:24.888 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.883+0000 7f6b0a716640 1 -- 172.21.15.42:0/1803124504 wait complete. 2024-09-16T04:03:24.888 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:24.883+0000 7f6b0a716640 1 librados: shutdown 2024-09-16T04:03:27.904 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:03:28.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.035+0000 7f668366b640 10 monclient: get_monmap_and_config 2024-09-16T04:03:28.039 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.035+0000 7f668366b640 10 monclient: build_initial_monmap 2024-09-16T04:03:28.039 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.035+0000 7f668366b640 10 monclient: monmap: 2024-09-16T04:03:28.039 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:03:28.039 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:03:28.039 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:03:28.037316+0000 2024-09-16T04:03:28.039 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:03:28.037316+0000 2024-09-16T04:03:28.039 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:03:28.040 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:03:28.040 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:03:28.040 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:03:28.040 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:03:28.040 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:03:28.040 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:03:28.040 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:03:28.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.035+0000 7f668366b640 1 Processor -- start 2024-09-16T04:03:28.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.035+0000 7f668366b640 1 -- start start 2024-09-16T04:03:28.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.035+0000 7f668366b640 10 monclient: init 2024-09-16T04:03:28.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.035+0000 7f668366b640 10 monclient: _reopen_session rank -1 2024-09-16T04:03:28.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.035+0000 7f668366b640 10 monclient: _add_conns ranks=[0,4,1,2,3] 2024-09-16T04:03:28.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.035+0000 7f668366b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f667c06db90 addr v1:172.21.15.42:6789/0 2024-09-16T04:03:28.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.035+0000 7f668366b640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f667c1340c0 0x7f667c13c190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:03:28.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.035+0000 7f668366b640 10 monclient(hunting): picked mon.noname-c con 0x7f667c1340c0 addr v2:172.21.15.154:3300/0 2024-09-16T04:03:28.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.035+0000 7f668366b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f667c13c6d0 addr v1:172.21.15.154:6789/0 2024-09-16T04:03:28.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.035+0000 7f668366b640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f667c06ece0 con 0x7f667c06db90 2024-09-16T04:03:28.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.035+0000 7f668366b640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f667c09bf80 con 0x7f667c13c6d0 2024-09-16T04:03:28.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.035+0000 7f668366b640 10 monclient(hunting): start opening mon connection 2024-09-16T04:03:28.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.035+0000 7f668366b640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f667c06df70 con 0x7f667c1340c0 2024-09-16T04:03:28.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.035+0000 7f668366b640 10 monclient(hunting): _renew_subs 2024-09-16T04:03:28.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.035+0000 7f668366b640 10 monclient(hunting): authenticate will time out at 2255.007526s 2024-09-16T04:03:28.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.035+0000 7f6681407640 1 -- >> v1:172.21.15.42:6789/0 conn(0x7f667c06db90 legacy=0x7f667c0a37f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.42:6789/0 2024-09-16T04:03:28.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.035+0000 7f6680c06640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f667c1340c0 0x7f667c13c190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:03:28.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.039+0000 7f6680c06640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f667c1340c0 0x7f667c13c190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.42:33106/0 (socket says 172.21.15.42:33106) 2024-09-16T04:03:28.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.039+0000 7f6680c06640 1 -- 172.21.15.42:0/1146408142 learned_addr learned my addr 172.21.15.42:0/1146408142 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:03:28.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.039+0000 7f6680c06640 10 monclient(hunting): get_auth_request con 0x7f667c1340c0 auth_method 0 2024-09-16T04:03:28.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.039+0000 7f6680c06640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:03:28.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.039+0000 7f6680c06640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:03:28.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.039+0000 7f6680c06640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:03:28.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.039+0000 7f6680c06640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:03:28.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.039+0000 7f6680c06640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:03:28.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.039+0000 7f6680c06640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:03:28.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.039+0000 7f6680c06640 10 monclient(hunting): handle_auth_done global_id 904744 payload 274 2024-09-16T04:03:28.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.039+0000 7f6680c06640 1 -- 172.21.15.42:0/1146408142 >> v1:172.21.15.154:6789/0 conn(0x7f667c13c6d0 legacy=0x7f667c13eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:03:28.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.039+0000 7f6680c06640 1 -- 172.21.15.42:0/1146408142 >> v1:172.21.15.42:6789/0 conn(0x7f667c06db90 legacy=0x7f667c0a37f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:03:28.045 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.039+0000 7f6680c06640 10 monclient: _finish_hunting 0 2024-09-16T04:03:28.045 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.039+0000 7f6680c06640 1 monclient: found mon.noname-c 2024-09-16T04:03:28.045 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.039+0000 7f6680c06640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:03:28.045 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.039+0000 7f6680c06640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.154:3300/0 2024-09-16T04:03:28.045 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.039+0000 7f6680c06640 1 -- 172.21.15.42:0/1146408142 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f667c1344d0 con 0x7f667c1340c0 2024-09-16T04:03:28.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.039+0000 7f6680c06640 1 --2- 172.21.15.42:0/1146408142 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f667c1340c0 0x7f667c13c190 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f666c009870 tx=0x7f666c01c3e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2361a57131e34d41 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:03:28.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.039+0000 7f6673fff640 1 -- 172.21.15.42:0/1146408142 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f666c0b0070 con 0x7f667c1340c0 2024-09-16T04:03:28.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.039+0000 7f6673fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:03:28.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.039+0000 7f6673fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:03:28.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.039+0000 7f6673fff640 10 monclient: dump: 2024-09-16T04:03:28.052 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:03:28.052 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:03:28.052 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:28.052 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:28.052 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:03:28.052 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:03:28.052 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:03:28.052 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:03:28.052 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:03:28.052 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:03:28.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.039+0000 7f6673fff640 10 monclient: _finish_auth 0 2024-09-16T04:03:28.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.039+0000 7f6673fff640 10 monclient: _check_auth_tickets 2024-09-16T04:03:28.053 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.039+0000 7f6673fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:03:28.053 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.039+0000 7f6673fff640 1 -- 172.21.15.42:0/1146408142 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f666c0ab600 con 0x7f667c1340c0 2024-09-16T04:03:28.053 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.039+0000 7f6673fff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:03:28.053 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.039+0000 7f6673fff640 1 -- 172.21.15.42:0/1146408142 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f666c0ab9d0 con 0x7f667c1340c0 2024-09-16T04:03:28.053 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.039+0000 7f6673fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:03:28.053 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.039+0000 7f6673fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:03:28.053 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.039+0000 7f6673fff640 10 monclient: dump: 2024-09-16T04:03:28.053 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:03:28.053 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:03:28.053 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:28.053 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:28.054 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:03:28.054 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:03:28.054 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:03:28.054 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:03:28.054 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:03:28.054 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:03:28.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.039+0000 7f668366b640 5 monclient: authenticate success, global_id 904744 2024-09-16T04:03:28.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.039+0000 7f668366b640 10 monclient: get_monmap_and_config success 2024-09-16T04:03:28.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.039+0000 7f668366b640 4 set_mon_vals no callback set 2024-09-16T04:03:28.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.043+0000 7f668366b640 10 monclient: shutdown 2024-09-16T04:03:28.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.043+0000 7f668366b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:03:28.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.043+0000 7f668366b640 1 -- 172.21.15.42:0/1146408142 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f667c1340c0 msgr2=0x7f667c13c190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:03:28.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.043+0000 7f668366b640 1 --2- 172.21.15.42:0/1146408142 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f667c1340c0 0x7f667c13c190 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f666c009870 tx=0x7f666c01c3e0 comp rx=0 tx=0).stop 2024-09-16T04:03:28.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.043+0000 7f668366b640 1 -- 172.21.15.42:0/1146408142 shutdown_connections 2024-09-16T04:03:28.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.043+0000 7f668366b640 1 --2- 172.21.15.42:0/1146408142 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f667c1340c0 0x7f667c13c190 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:03:28.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.043+0000 7f668366b640 1 -- 172.21.15.42:0/1146408142 >> 172.21.15.42:0/1146408142 conn(0x7f667c0a9fb0 msgr2=0x7f667c0aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:03:28.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.043+0000 7f668366b640 1 -- 172.21.15.42:0/1146408142 shutdown_connections 2024-09-16T04:03:28.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.043+0000 7f668366b640 1 -- 172.21.15.42:0/1146408142 wait complete. 2024-09-16T04:03:28.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.043+0000 7f668366b640 10 monclient: build_initial_monmap 2024-09-16T04:03:28.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.043+0000 7f668366b640 10 monclient: monmap: 2024-09-16T04:03:28.055 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:03:28.055 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:03:28.055 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:03:28.055 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:03:28.056 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:03:28.056 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:03:28.056 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:03:28.056 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:03:28.056 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:03:28.056 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:03:28.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.043+0000 7f668366b640 1 librados: starting msgr at 2024-09-16T04:03:28.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.043+0000 7f668366b640 1 librados: starting objecter 2024-09-16T04:03:28.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.043+0000 7f668366b640 1 Processor -- start 2024-09-16T04:03:28.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.043+0000 7f668366b640 1 -- start start 2024-09-16T04:03:28.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.043+0000 7f668366b640 1 librados: setting wanted keys 2024-09-16T04:03:28.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.043+0000 7f668366b640 1 librados: calling monclient init 2024-09-16T04:03:28.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.043+0000 7f668366b640 10 monclient: init 2024-09-16T04:03:28.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.043+0000 7f668366b640 10 monclient: _reopen_session rank -1 2024-09-16T04:03:28.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.043+0000 7f668366b640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:03:28.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.043+0000 7f668366b640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f667c06db90 0x7f667c1cf0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:03:28.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.043+0000 7f668366b640 10 monclient(hunting): picked mon.noname-c con 0x7f667c06db90 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:03:28.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.043+0000 7f668366b640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f667c1340c0 0x7f667c0a2c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:03:28.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.043+0000 7f668366b640 10 monclient(hunting): picked mon.noname-a con 0x7f667c1340c0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:03:28.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.043+0000 7f668366b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f667c13c6d0 0x7f667c0a3150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:03:28.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.043+0000 7f668366b640 10 monclient(hunting): picked mon.noname-b con 0x7f667c13c6d0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:03:28.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.043+0000 7f668366b640 10 monclient(hunting): start opening mon connection 2024-09-16T04:03:28.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.043+0000 7f668366b640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f667c142b80 con 0x7f667c1340c0 2024-09-16T04:03:28.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.043+0000 7f668366b640 10 monclient(hunting): start opening mon connection 2024-09-16T04:03:28.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.043+0000 7f668366b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f667c142a00 con 0x7f667c13c6d0 2024-09-16T04:03:28.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.043+0000 7f668366b640 10 monclient(hunting): start opening mon connection 2024-09-16T04:03:28.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.043+0000 7f668366b640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f667c142d00 con 0x7f667c06db90 2024-09-16T04:03:28.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.043+0000 7f668366b640 10 monclient(hunting): _renew_subs 2024-09-16T04:03:28.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.043+0000 7f668366b640 10 monclient(hunting): authenticate will time out at 2255.013713s 2024-09-16T04:03:28.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.043+0000 7f6680c06640 1 -- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f667c1340c0 msgr2=0x7f667c0a2c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.42:3300/0 2024-09-16T04:03:28.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.043+0000 7f6680c06640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f667c1340c0 0x7f667c0a2c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:03:28.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.043+0000 7f6681407640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f667c06db90 0x7f667c1cf0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:03:28.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.043+0000 7f6681407640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f667c06db90 0x7f667c1cf0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:60596/0 (socket says 172.21.15.42:60596) 2024-09-16T04:03:28.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.043+0000 7f6681407640 1 -- 172.21.15.42:0/4229012647 learned_addr learned my addr 172.21.15.42:0/4229012647 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:03:28.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.043+0000 7f6681407640 10 monclient(hunting): get_auth_request con 0x7f667c06db90 auth_method 0 2024-09-16T04:03:28.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.043+0000 7f6681407640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:03:28.059 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.043+0000 7f6681407640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:03:28.059 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.043+0000 7f6681407640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:03:28.059 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.043+0000 7f6681407640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:03:28.059 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.043+0000 7f6681407640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:03:28.059 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.043+0000 7f6681407640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:03:28.059 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.043+0000 7f6681c08640 1 --2- 172.21.15.42:0/4229012647 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f667c13c6d0 0x7f667c0a3150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:03:28.059 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.043+0000 7f6681c08640 10 monclient(hunting): get_auth_request con 0x7f667c13c6d0 auth_method 0 2024-09-16T04:03:28.059 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.043+0000 7f6681c08640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:03:28.059 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.043+0000 7f6681c08640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:03:28.059 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.043+0000 7f6681c08640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:03:28.059 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.043+0000 7f6681407640 10 monclient(hunting): handle_auth_done global_id 974147 payload 824 2024-09-16T04:03:28.059 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.043+0000 7f6681407640 1 -- 172.21.15.42:0/4229012647 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f667c13c6d0 msgr2=0x7f667c0a3150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:03:28.059 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.043+0000 7f6681407640 1 --2- 172.21.15.42:0/4229012647 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f667c13c6d0 0x7f667c0a3150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:03:28.060 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.043+0000 7f6681407640 1 -- 172.21.15.42:0/4229012647 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f667c1340c0 msgr2=0x7f667c0a2c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:03:28.060 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.043+0000 7f6681407640 1 --2- 172.21.15.42:0/4229012647 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f667c1340c0 0x7f667c0a2c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:03:28.060 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.043+0000 7f6681407640 10 monclient: _finish_hunting 0 2024-09-16T04:03:28.060 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.043+0000 7f6681407640 1 monclient: found mon.noname-c 2024-09-16T04:03:28.060 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.043+0000 7f6681407640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:03:28.060 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.043+0000 7f6681407640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T04:03:28.060 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.043+0000 7f6681407640 1 -- 172.21.15.42:0/4229012647 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f667c13fca0 con 0x7f667c06db90 2024-09-16T04:03:28.060 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.043+0000 7f6681407640 1 --2- 172.21.15.42:0/4229012647 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f667c06db90 0x7f667c1cf0a0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f667800b550 tx=0x7f667800ba10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:03:28.060 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.043+0000 7f6671ffb640 10 client.?.objecter ms_handle_connect 0x7f667c06db90 2024-09-16T04:03:28.060 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.043+0000 7f6671ffb640 10 client.?.objecter resend_mon_ops 2024-09-16T04:03:28.060 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.043+0000 7f6671ffb640 1 -- 172.21.15.42:0/4229012647 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6678003dd0 con 0x7f667c06db90 2024-09-16T04:03:28.061 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.043+0000 7f6671ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:03:28.061 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.043+0000 7f6671ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:03:28.061 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.043+0000 7f6671ffb640 10 monclient: dump: 2024-09-16T04:03:28.061 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:03:28.061 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:03:28.061 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:28.061 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:28.061 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:03:28.061 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:03:28.061 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:03:28.062 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:03:28.062 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:03:28.062 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:03:28.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.043+0000 7f6671ffb640 10 monclient: _finish_auth 0 2024-09-16T04:03:28.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.043+0000 7f6671ffb640 10 monclient: _check_auth_tickets 2024-09-16T04:03:28.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.043+0000 7f6671ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:03:28.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.043+0000 7f6671ffb640 1 -- 172.21.15.42:0/4229012647 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6678015070 con 0x7f667c06db90 2024-09-16T04:03:28.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.043+0000 7f6671ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:03:28.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.043+0000 7f6671ffb640 1 -- 172.21.15.42:0/4229012647 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6678005330 con 0x7f667c06db90 2024-09-16T04:03:28.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.043+0000 7f668366b640 5 monclient: authenticate success, global_id 974147 2024-09-16T04:03:28.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.043+0000 7f668366b640 10 monclient: _renew_subs 2024-09-16T04:03:28.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.043+0000 7f668366b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:03:28.063 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.043+0000 7f668366b640 1 -- 172.21.15.42:0/4229012647 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f667c1349e0 con 0x7f667c06db90 2024-09-16T04:03:28.063 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.043+0000 7f6671ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:03:28.063 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.043+0000 7f66737fe640 4 set_mon_vals no callback set 2024-09-16T04:03:28.063 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.043+0000 7f6671ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:03:28.063 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.043+0000 7f6671ffb640 10 monclient: dump: 2024-09-16T04:03:28.063 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:03:28.063 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:03:28.063 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:28.063 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:28.063 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:03:28.063 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:03:28.063 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:03:28.063 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:03:28.064 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:03:28.064 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:03:28.064 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.043+0000 7f668366b640 10 client.974147.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:03:28.064 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.043+0000 7f668366b640 10 monclient: _renew_subs 2024-09-16T04:03:28.064 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.043+0000 7f668366b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:03:28.064 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.043+0000 7f668366b640 1 -- 172.21.15.42:0/4229012647 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f667c09d0e0 con 0x7f667c06db90 2024-09-16T04:03:28.064 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.043+0000 7f668366b640 1 librados: init done 2024-09-16T04:03:28.064 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.043+0000 7f6671ffb640 1 -- 172.21.15.42:0/4229012647 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f66780054d0 con 0x7f667c06db90 2024-09-16T04:03:28.064 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.043+0000 7f668366b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:03:28.064 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.043+0000 7f668366b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:03:28.064 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.043+0000 7f668366b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:03:28.064 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.043+0000 7f668366b640 1 -- 172.21.15.42:0/4229012647 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f667c06f680 con 0x7f667c06db90 2024-09-16T04:03:28.065 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.043+0000 7f6671ffb640 1 --2- 172.21.15.42:0/4229012647 >> v2:172.21.15.154:6812/254845342 conn(0x7f66600377c0 0x7f6660039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:03:28.065 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.043+0000 7f6671ffb640 1 -- 172.21.15.42:0/4229012647 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(654..654 src has 451..654) ==== 6285+0+0 (secure 0 0 0) 0x7f6678049120 con 0x7f667c06db90 2024-09-16T04:03:28.065 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.043+0000 7f6671ffb640 10 client.974147.objecter ms_dispatch 0x7f667c000ba0 osd_map(654..654 src has 451..654) 2024-09-16T04:03:28.065 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.043+0000 7f6671ffb640 3 client.974147.objecter handle_osd_map got epochs [654,654] > 0 2024-09-16T04:03:28.065 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.043+0000 7f6671ffb640 3 client.974147.objecter handle_osd_map decoding full epoch 654 2024-09-16T04:03:28.065 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.047+0000 7f6671ffb640 20 client.974147.objecter dump_active .. 0 homeless 2024-09-16T04:03:28.065 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.047+0000 7f6680c06640 1 --2- 172.21.15.42:0/4229012647 >> v2:172.21.15.154:6812/254845342 conn(0x7f66600377c0 0x7f6660039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:03:28.065 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.047+0000 7f6680c06640 10 monclient: get_auth_request con 0x7f66600377c0 auth_method 0 2024-09-16T04:03:28.065 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.047+0000 7f6680c06640 1 --2- 172.21.15.42:0/4229012647 >> v2:172.21.15.154:6812/254845342 conn(0x7f66600377c0 0x7f6660039c80 secure :-1 s=READY pgs=3810 cs=0 l=1 rev1=1 crypto rx=0x7f666c0ad000 tx=0x7f666c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:03:28.065 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.047+0000 7f6671ffb640 10 client.974147.objecter ms_handle_connect 0x7f66600377c0 2024-09-16T04:03:28.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.051+0000 7f6671ffb640 1 -- 172.21.15.42:0/4229012647 <== mon.2 v2:172.21.15.42:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f667c1349e0 con 0x7f667c06db90 2024-09-16T04:03:28.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.051+0000 7f6671ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:03:28.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.051+0000 7f6671ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:03:28.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.279+0000 7f668366b640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:03:28.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.279+0000 7f668366b640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:03:28.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.279+0000 7f668366b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:03:28.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.279+0000 7f668366b640 1 -- 172.21.15.42:0/4229012647 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f667c0a37f0 con 0x7f667c06db90 2024-09-16T04:03:28.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.279+0000 7f6671ffb640 1 -- 172.21.15.42:0/4229012647 <== mon.2 v2:172.21.15.42:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1562 (secure 0 0 0) 0x7f667c13fca0 con 0x7f667c06db90 2024-09-16T04:03:28.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.279+0000 7f6671ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:03:28.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.279+0000 7f6671ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:03:28.284 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":190,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:03:28.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.283+0000 7f668366b640 10 librados: watch_flush enter 2024-09-16T04:03:28.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.283+0000 7f668366b640 10 librados: watch_flush exit 2024-09-16T04:03:28.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.283+0000 7f668366b640 20 client.974147.objecter shutdown clearing up homeless session... 2024-09-16T04:03:28.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.283+0000 7f668366b640 10 client.974147.objecter successfully canceled tick 2024-09-16T04:03:28.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.283+0000 7f668366b640 1 -- 172.21.15.42:0/4229012647 >> v2:172.21.15.154:6812/254845342 conn(0x7f66600377c0 msgr2=0x7f6660039c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:03:28.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.283+0000 7f668366b640 1 --2- 172.21.15.42:0/4229012647 >> v2:172.21.15.154:6812/254845342 conn(0x7f66600377c0 0x7f6660039c80 secure :-1 s=READY pgs=3810 cs=0 l=1 rev1=1 crypto rx=0x7f666c0ad000 tx=0x7f666c008040 comp rx=0 tx=0).stop 2024-09-16T04:03:28.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.283+0000 7f668366b640 10 monclient: shutdown 2024-09-16T04:03:28.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.283+0000 7f668366b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:03:28.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.283+0000 7f668366b640 1 -- 172.21.15.42:0/4229012647 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f667c06db90 msgr2=0x7f667c1cf0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:03:28.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.283+0000 7f668366b640 1 --2- 172.21.15.42:0/4229012647 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f667c06db90 0x7f667c1cf0a0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f667800b550 tx=0x7f667800ba10 comp rx=0 tx=0).stop 2024-09-16T04:03:28.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.283+0000 7f668366b640 1 -- 172.21.15.42:0/4229012647 shutdown_connections 2024-09-16T04:03:28.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.283+0000 7f668366b640 1 --2- 172.21.15.42:0/4229012647 >> v2:172.21.15.154:6812/254845342 conn(0x7f66600377c0 0x7f6660039c80 unknown :-1 s=CLOSED pgs=3810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:03:28.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.283+0000 7f668366b640 1 --2- 172.21.15.42:0/4229012647 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f667c13c6d0 0x7f667c0a3150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:03:28.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.283+0000 7f668366b640 1 --2- 172.21.15.42:0/4229012647 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f667c1340c0 0x7f667c0a2c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:03:28.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.283+0000 7f668366b640 1 --2- 172.21.15.42:0/4229012647 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f667c06db90 0x7f667c1cf0a0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:03:28.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.283+0000 7f668366b640 1 -- 172.21.15.42:0/4229012647 >> 172.21.15.42:0/4229012647 conn(0x7f667c0a9fb0 msgr2=0x7f667c0a80e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:03:28.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.283+0000 7f668366b640 1 -- 172.21.15.42:0/4229012647 shutdown_connections 2024-09-16T04:03:28.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.283+0000 7f668366b640 1 -- 172.21.15.42:0/4229012647 wait complete. 2024-09-16T04:03:28.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:28.283+0000 7f668366b640 1 librados: shutdown 2024-09-16T04:03:31.306 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:03:31.439 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.435+0000 7f149fc38640 10 monclient: get_monmap_and_config 2024-09-16T04:03:31.439 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.435+0000 7f149fc38640 10 monclient: build_initial_monmap 2024-09-16T04:03:31.439 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.435+0000 7f149fc38640 10 monclient: monmap: 2024-09-16T04:03:31.439 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:03:31.439 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:03:31.439 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:03:31.437706+0000 2024-09-16T04:03:31.439 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:03:31.437706+0000 2024-09-16T04:03:31.440 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:03:31.440 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:03:31.440 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:03:31.440 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:03:31.440 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:03:31.440 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:03:31.440 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:03:31.440 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:03:31.442 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.435+0000 7f149fc38640 1 Processor -- start 2024-09-16T04:03:31.442 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.435+0000 7f149fc38640 1 -- start start 2024-09-16T04:03:31.442 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.435+0000 7f149fc38640 10 monclient: init 2024-09-16T04:03:31.442 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.435+0000 7f149fc38640 10 monclient: _reopen_session rank -1 2024-09-16T04:03:31.442 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.435+0000 7f149fc38640 10 monclient: _add_conns ranks=[2,4,1,3,0] 2024-09-16T04:03:31.442 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.435+0000 7f149fc38640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f149806db90 0x7f14981340c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:03:31.442 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.435+0000 7f149fc38640 10 monclient(hunting): picked mon.noname-b con 0x7f149806db90 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:03:31.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.435+0000 7f149fc38640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f14980a37f0 0x7f14980a3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:03:31.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.435+0000 7f149fc38640 10 monclient(hunting): picked mon.noname-c con 0x7f14980a37f0 addr v2:172.21.15.154:3300/0 2024-09-16T04:03:31.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.435+0000 7f149fc38640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1498136e10 addr v1:172.21.15.154:6789/0 2024-09-16T04:03:31.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.435+0000 7f149fc38640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f149806ece0 con 0x7f1498136e10 2024-09-16T04:03:31.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.435+0000 7f149fc38640 10 monclient(hunting): start opening mon connection 2024-09-16T04:03:31.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.435+0000 7f149fc38640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f14980a4110 con 0x7f14980a37f0 2024-09-16T04:03:31.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.435+0000 7f149fc38640 10 monclient(hunting): start opening mon connection 2024-09-16T04:03:31.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.435+0000 7f149fc38640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f149806df70 con 0x7f149806db90 2024-09-16T04:03:31.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.435+0000 7f149fc38640 10 monclient(hunting): _renew_subs 2024-09-16T04:03:31.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.435+0000 7f149fc38640 10 monclient(hunting): authenticate will time out at 2258.407243s 2024-09-16T04:03:31.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.435+0000 7f149d9d4640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f149806db90 0x7f14981340c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:03:31.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.435+0000 7f149d9d4640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f149806db90 0x7f14981340c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:60604/0 (socket says 172.21.15.42:60604) 2024-09-16T04:03:31.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.435+0000 7f149d9d4640 1 -- 172.21.15.42:0/622109731 learned_addr learned my addr 172.21.15.42:0/622109731 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:03:31.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.435+0000 7f149d9d4640 10 monclient(hunting): get_auth_request con 0x7f149806db90 auth_method 0 2024-09-16T04:03:31.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.435+0000 7f149d9d4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:03:31.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.435+0000 7f149d9d4640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:03:31.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.435+0000 7f149d9d4640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:03:31.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.435+0000 7f149d9d4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:03:31.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.435+0000 7f149d9d4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:03:31.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.435+0000 7f149d9d4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:03:31.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.435+0000 7f149d9d4640 10 monclient(hunting): handle_auth_done global_id 974150 payload 274 2024-09-16T04:03:31.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.435+0000 7f149d9d4640 1 -- 172.21.15.42:0/622109731 >> v2:172.21.15.154:3300/0 conn(0x7f14980a37f0 msgr2=0x7f14980a3bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:03:31.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.435+0000 7f149d9d4640 1 --2- 172.21.15.42:0/622109731 >> v2:172.21.15.154:3300/0 conn(0x7f14980a37f0 0x7f14980a3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:03:31.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.435+0000 7f149d9d4640 1 -- 172.21.15.42:0/622109731 >> v1:172.21.15.154:6789/0 conn(0x7f1498136e10 legacy=0x7f1498137220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:03:31.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.435+0000 7f149d9d4640 10 monclient: _finish_hunting 0 2024-09-16T04:03:31.445 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.435+0000 7f149d9d4640 1 monclient: found mon.noname-b 2024-09-16T04:03:31.445 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.435+0000 7f149d9d4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:03:31.445 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.435+0000 7f149d9d4640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T04:03:31.445 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.435+0000 7f149d9d4640 1 -- 172.21.15.42:0/622109731 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f149809bf80 con 0x7f149806db90 2024-09-16T04:03:31.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.439+0000 7f149d9d4640 1 --2- 172.21.15.42:0/622109731 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f149806db90 0x7f14981340c0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f1494009870 tx=0x7f149401c3e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=68fe8c0a1b3f4cd5 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:03:31.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.439+0000 7f149c9d2640 1 -- 172.21.15.42:0/622109731 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14940b0070 con 0x7f149806db90 2024-09-16T04:03:31.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.439+0000 7f149c9d2640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:03:31.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.439+0000 7f149c9d2640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:03:31.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.439+0000 7f149c9d2640 10 monclient: dump: 2024-09-16T04:03:31.451 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:03:31.451 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:03:31.452 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:31.452 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:31.452 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:03:31.452 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:03:31.452 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:03:31.452 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:03:31.452 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:03:31.452 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:03:31.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.439+0000 7f149c9d2640 10 monclient: _finish_auth 0 2024-09-16T04:03:31.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.439+0000 7f149c9d2640 10 monclient: _check_auth_tickets 2024-09-16T04:03:31.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.439+0000 7f149c9d2640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:03:31.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.439+0000 7f149c9d2640 1 -- 172.21.15.42:0/622109731 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f14940ab600 con 0x7f149806db90 2024-09-16T04:03:31.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.439+0000 7f149c9d2640 10 monclient: handle_config config(0 keys) 2024-09-16T04:03:31.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.439+0000 7f149c9d2640 1 -- 172.21.15.42:0/622109731 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14940ab9d0 con 0x7f149806db90 2024-09-16T04:03:31.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.439+0000 7f149fc38640 5 monclient: authenticate success, global_id 974150 2024-09-16T04:03:31.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.439+0000 7f149fc38640 10 monclient: get_monmap_and_config success 2024-09-16T04:03:31.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.439+0000 7f149fc38640 4 set_mon_vals no callback set 2024-09-16T04:03:31.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.439+0000 7f149c9d2640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:03:31.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.439+0000 7f149c9d2640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:03:31.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.439+0000 7f149c9d2640 10 monclient: dump: 2024-09-16T04:03:31.453 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:03:31.453 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:03:31.453 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:31.453 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:31.453 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:03:31.453 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:03:31.454 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:03:31.454 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:03:31.454 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:03:31.454 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:03:31.454 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.439+0000 7f149fc38640 10 monclient: shutdown 2024-09-16T04:03:31.454 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.439+0000 7f149fc38640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:03:31.454 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.439+0000 7f149fc38640 1 -- 172.21.15.42:0/622109731 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f149806db90 msgr2=0x7f14981340c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:03:31.454 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.439+0000 7f149fc38640 1 --2- 172.21.15.42:0/622109731 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f149806db90 0x7f14981340c0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f1494009870 tx=0x7f149401c3e0 comp rx=0 tx=0).stop 2024-09-16T04:03:31.454 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.439+0000 7f149fc38640 1 -- 172.21.15.42:0/622109731 shutdown_connections 2024-09-16T04:03:31.454 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.439+0000 7f149fc38640 1 --2- 172.21.15.42:0/622109731 >> v2:172.21.15.154:3300/0 conn(0x7f14980a37f0 0x7f14980a3bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:03:31.454 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.439+0000 7f149fc38640 1 --2- 172.21.15.42:0/622109731 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f149806db90 0x7f14981340c0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:03:31.454 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.439+0000 7f149fc38640 1 -- 172.21.15.42:0/622109731 >> 172.21.15.42:0/622109731 conn(0x7f14980a9fb0 msgr2=0x7f14980aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:03:31.454 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.439+0000 7f149fc38640 1 -- 172.21.15.42:0/622109731 shutdown_connections 2024-09-16T04:03:31.455 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.439+0000 7f149fc38640 1 -- 172.21.15.42:0/622109731 wait complete. 2024-09-16T04:03:31.455 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.439+0000 7f149fc38640 10 monclient: build_initial_monmap 2024-09-16T04:03:31.455 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.439+0000 7f149fc38640 10 monclient: monmap: 2024-09-16T04:03:31.455 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:03:31.455 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:03:31.455 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:03:31.455 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:03:31.455 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:03:31.455 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:03:31.455 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:03:31.455 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:03:31.455 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:03:31.455 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:03:31.455 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.439+0000 7f149fc38640 1 librados: starting msgr at 2024-09-16T04:03:31.455 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.439+0000 7f149fc38640 1 librados: starting objecter 2024-09-16T04:03:31.456 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.443+0000 7f149fc38640 1 Processor -- start 2024-09-16T04:03:31.456 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.443+0000 7f149fc38640 1 -- start start 2024-09-16T04:03:31.456 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.443+0000 7f149fc38640 1 librados: setting wanted keys 2024-09-16T04:03:31.456 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.443+0000 7f149fc38640 1 librados: calling monclient init 2024-09-16T04:03:31.456 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.443+0000 7f149fc38640 10 monclient: init 2024-09-16T04:03:31.456 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.443+0000 7f149fc38640 10 monclient: _reopen_session rank -1 2024-09-16T04:03:31.456 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.443+0000 7f149fc38640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:03:31.456 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.443+0000 7f149fc38640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f149806db90 0x7f14981cf190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:03:31.456 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.443+0000 7f149fc38640 10 monclient(hunting): picked mon.noname-b con 0x7f149806db90 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:03:31.456 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.443+0000 7f149fc38640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f14980a37f0 0x7f14980a2ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:03:31.456 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.443+0000 7f149fc38640 10 monclient(hunting): picked mon.noname-c con 0x7f14980a37f0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:03:31.456 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.443+0000 7f149fc38640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1498136e10 0x7f14980a3020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:03:31.456 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.443+0000 7f149fc38640 10 monclient(hunting): picked mon.noname-a con 0x7f1498136e10 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:03:31.456 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.443+0000 7f149fc38640 10 monclient(hunting): start opening mon connection 2024-09-16T04:03:31.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.443+0000 7f149fc38640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f1498141d60 con 0x7f1498136e10 2024-09-16T04:03:31.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.443+0000 7f149fc38640 10 monclient(hunting): start opening mon connection 2024-09-16T04:03:31.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.443+0000 7f149fc38640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f1498141be0 con 0x7f149806db90 2024-09-16T04:03:31.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.443+0000 7f149fc38640 10 monclient(hunting): start opening mon connection 2024-09-16T04:03:31.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.443+0000 7f149fc38640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f1498141ee0 con 0x7f14980a37f0 2024-09-16T04:03:31.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.443+0000 7f149fc38640 10 monclient(hunting): _renew_subs 2024-09-16T04:03:31.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.443+0000 7f149fc38640 10 monclient(hunting): authenticate will time out at 2258.412632s 2024-09-16T04:03:31.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.443+0000 7f149e1d5640 1 -- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1498136e10 msgr2=0x7f14980a3020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.42:3300/0 2024-09-16T04:03:31.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.443+0000 7f149e1d5640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1498136e10 0x7f14980a3020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:03:31.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.443+0000 7f149d1d3640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f14980a37f0 0x7f14980a2ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:03:31.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.443+0000 7f149d1d3640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f14980a37f0 0x7f14980a2ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:60620/0 (socket says 172.21.15.42:60620) 2024-09-16T04:03:31.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.443+0000 7f149d1d3640 1 -- 172.21.15.42:0/379819495 learned_addr learned my addr 172.21.15.42:0/379819495 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:03:31.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.443+0000 7f149d1d3640 10 monclient(hunting): get_auth_request con 0x7f14980a37f0 auth_method 0 2024-09-16T04:03:31.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.443+0000 7f149d1d3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:03:31.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.443+0000 7f149d1d3640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:03:31.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.443+0000 7f149d1d3640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:03:31.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.443+0000 7f149d1d3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:03:31.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.443+0000 7f149d1d3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:03:31.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.443+0000 7f149d1d3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:03:31.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.443+0000 7f149d9d4640 1 --2- 172.21.15.42:0/379819495 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f149806db90 0x7f14981cf190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:03:31.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.443+0000 7f149d9d4640 10 monclient(hunting): get_auth_request con 0x7f149806db90 auth_method 0 2024-09-16T04:03:31.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.443+0000 7f149d9d4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:03:31.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.443+0000 7f149d9d4640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:03:31.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.443+0000 7f149d9d4640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:03:31.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.443+0000 7f149d1d3640 10 monclient(hunting): handle_auth_done global_id 974153 payload 824 2024-09-16T04:03:31.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.443+0000 7f149d1d3640 1 -- 172.21.15.42:0/379819495 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f149806db90 msgr2=0x7f14981cf190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:03:31.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.443+0000 7f149d1d3640 1 --2- 172.21.15.42:0/379819495 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f149806db90 0x7f14981cf190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:03:31.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.443+0000 7f149d1d3640 1 -- 172.21.15.42:0/379819495 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1498136e10 msgr2=0x7f14980a3020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:03:31.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.443+0000 7f149d1d3640 1 --2- 172.21.15.42:0/379819495 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1498136e10 0x7f14980a3020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:03:31.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.443+0000 7f149d1d3640 10 monclient: _finish_hunting 0 2024-09-16T04:03:31.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.443+0000 7f149d1d3640 1 monclient: found mon.noname-c 2024-09-16T04:03:31.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.443+0000 7f149d1d3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:03:31.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.443+0000 7f149d1d3640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T04:03:31.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.443+0000 7f149d1d3640 1 -- 172.21.15.42:0/379819495 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f149809e2a0 con 0x7f14980a37f0 2024-09-16T04:03:31.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.443+0000 7f149d1d3640 1 --2- 172.21.15.42:0/379819495 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f14980a37f0 0x7f14980a2ae0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f148800b550 tx=0x7f148800ba10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:03:31.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.443+0000 7f148e7fc640 10 client.?.objecter ms_handle_connect 0x7f14980a37f0 2024-09-16T04:03:31.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.443+0000 7f148e7fc640 10 client.?.objecter resend_mon_ops 2024-09-16T04:03:31.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.443+0000 7f148e7fc640 1 -- 172.21.15.42:0/379819495 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1488014070 con 0x7f14980a37f0 2024-09-16T04:03:31.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.443+0000 7f148e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:03:31.460 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.443+0000 7f148e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:03:31.460 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.443+0000 7f148e7fc640 10 monclient: dump: 2024-09-16T04:03:31.460 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:03:31.460 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:03:31.460 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:31.460 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:31.460 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:03:31.460 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:03:31.460 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:03:31.460 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:03:31.460 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:03:31.460 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:03:31.460 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.443+0000 7f148e7fc640 10 monclient: _finish_auth 0 2024-09-16T04:03:31.461 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.443+0000 7f148e7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:03:31.461 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.443+0000 7f148e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:03:31.461 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.443+0000 7f148e7fc640 1 -- 172.21.15.42:0/379819495 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1488010890 con 0x7f14980a37f0 2024-09-16T04:03:31.461 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.443+0000 7f149fc38640 5 monclient: authenticate success, global_id 974153 2024-09-16T04:03:31.461 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.443+0000 7f149fc38640 10 monclient: _renew_subs 2024-09-16T04:03:31.461 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.443+0000 7f149fc38640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:03:31.461 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.443+0000 7f149fc38640 1 -- 172.21.15.42:0/379819495 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14981cf6d0 con 0x7f14980a37f0 2024-09-16T04:03:31.461 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.443+0000 7f148e7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:03:31.461 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.443+0000 7f148e7fc640 1 -- 172.21.15.42:0/379819495 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1488010c80 con 0x7f14980a37f0 2024-09-16T04:03:31.461 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.443+0000 7f148e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:03:31.461 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.443+0000 7f148e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:03:31.461 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.443+0000 7f148e7fc640 10 monclient: dump: 2024-09-16T04:03:31.461 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:03:31.462 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:03:31.462 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:31.462 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:31.462 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:03:31.462 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:03:31.462 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:03:31.462 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:03:31.462 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:03:31.462 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:03:31.462 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.443+0000 7f149fc38640 10 client.974153.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:03:31.462 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.443+0000 7f149fc38640 10 monclient: _renew_subs 2024-09-16T04:03:31.462 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.443+0000 7f148ffff640 4 set_mon_vals no callback set 2024-09-16T04:03:31.462 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.443+0000 7f149fc38640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:03:31.462 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.443+0000 7f149fc38640 1 -- 172.21.15.42:0/379819495 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f149809dfc0 con 0x7f14980a37f0 2024-09-16T04:03:31.463 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.443+0000 7f149fc38640 1 librados: init done 2024-09-16T04:03:31.463 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.443+0000 7f149fc38640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:03:31.463 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.443+0000 7f149fc38640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:03:31.463 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.443+0000 7f149fc38640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:03:31.463 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.443+0000 7f149fc38640 1 -- 172.21.15.42:0/379819495 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f14981340c0 con 0x7f14980a37f0 2024-09-16T04:03:31.463 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.443+0000 7f148e7fc640 1 -- 172.21.15.42:0/379819495 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f1488017070 con 0x7f14980a37f0 2024-09-16T04:03:31.463 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.443+0000 7f148e7fc640 1 --2- 172.21.15.42:0/379819495 >> v2:172.21.15.154:6812/254845342 conn(0x7f147c0377c0 0x7f147c039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:03:31.463 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.443+0000 7f148e7fc640 1 -- 172.21.15.42:0/379819495 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(654..654 src has 451..654) ==== 6285+0+0 (secure 0 0 0) 0x7f1488048f10 con 0x7f14980a37f0 2024-09-16T04:03:31.463 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.443+0000 7f148e7fc640 10 client.974153.objecter ms_dispatch 0x7f1498000ba0 osd_map(654..654 src has 451..654) 2024-09-16T04:03:31.463 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.443+0000 7f148e7fc640 3 client.974153.objecter handle_osd_map got epochs [654,654] > 0 2024-09-16T04:03:31.463 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.443+0000 7f148e7fc640 3 client.974153.objecter handle_osd_map decoding full epoch 654 2024-09-16T04:03:31.463 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.443+0000 7f148e7fc640 20 client.974153.objecter dump_active .. 0 homeless 2024-09-16T04:03:31.463 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.443+0000 7f149d9d4640 1 --2- 172.21.15.42:0/379819495 >> v2:172.21.15.154:6812/254845342 conn(0x7f147c0377c0 0x7f147c039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:03:31.463 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.443+0000 7f149d9d4640 10 monclient: get_auth_request con 0x7f147c0377c0 auth_method 0 2024-09-16T04:03:31.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.447+0000 7f148e7fc640 10 client.974153.objecter ms_handle_connect 0x7f147c0377c0 2024-09-16T04:03:31.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.447+0000 7f149d9d4640 1 --2- 172.21.15.42:0/379819495 >> v2:172.21.15.154:6812/254845342 conn(0x7f147c0377c0 0x7f147c039c80 secure :-1 s=READY pgs=3811 cs=0 l=1 rev1=1 crypto rx=0x7f14940ad000 tx=0x7f14940a6000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:03:31.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.447+0000 7f148e7fc640 1 -- 172.21.15.42:0/379819495 <== mon.2 v2:172.21.15.42:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f1488018cb0 con 0x7f14980a37f0 2024-09-16T04:03:31.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.447+0000 7f148e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:03:31.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.447+0000 7f148e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:03:31.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.671+0000 7f149fc38640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:03:31.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.671+0000 7f149fc38640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:03:31.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.671+0000 7f149fc38640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:03:31.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.671+0000 7f149fc38640 1 -- 172.21.15.42:0/379819495 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f149809cb70 con 0x7f14980a37f0 2024-09-16T04:03:31.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.671+0000 7f148e7fc640 1 -- 172.21.15.42:0/379819495 <== mon.2 v2:172.21.15.42:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1562 (secure 0 0 0) 0x7f1488021be0 con 0x7f14980a37f0 2024-09-16T04:03:31.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.671+0000 7f148e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:03:31.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.671+0000 7f148e7fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:03:31.676 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":190,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":14,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:03:31.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.671+0000 7f149fc38640 10 librados: watch_flush enter 2024-09-16T04:03:31.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.671+0000 7f149fc38640 10 librados: watch_flush exit 2024-09-16T04:03:31.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.671+0000 7f149fc38640 20 client.974153.objecter shutdown clearing up homeless session... 2024-09-16T04:03:31.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.671+0000 7f149fc38640 10 client.974153.objecter successfully canceled tick 2024-09-16T04:03:31.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.675+0000 7f149fc38640 1 -- 172.21.15.42:0/379819495 >> v2:172.21.15.154:6812/254845342 conn(0x7f147c0377c0 msgr2=0x7f147c039c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:03:31.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.675+0000 7f149fc38640 1 --2- 172.21.15.42:0/379819495 >> v2:172.21.15.154:6812/254845342 conn(0x7f147c0377c0 0x7f147c039c80 secure :-1 s=READY pgs=3811 cs=0 l=1 rev1=1 crypto rx=0x7f14940ad000 tx=0x7f14940a6000 comp rx=0 tx=0).stop 2024-09-16T04:03:31.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.675+0000 7f149fc38640 10 monclient: shutdown 2024-09-16T04:03:31.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.675+0000 7f149fc38640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:03:31.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.675+0000 7f149fc38640 1 -- 172.21.15.42:0/379819495 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f14980a37f0 msgr2=0x7f14980a2ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:03:31.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.675+0000 7f149fc38640 1 --2- 172.21.15.42:0/379819495 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f14980a37f0 0x7f14980a2ae0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f148800b550 tx=0x7f148800ba10 comp rx=0 tx=0).stop 2024-09-16T04:03:31.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.675+0000 7f149fc38640 1 -- 172.21.15.42:0/379819495 shutdown_connections 2024-09-16T04:03:31.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.675+0000 7f149fc38640 1 --2- 172.21.15.42:0/379819495 >> v2:172.21.15.154:6812/254845342 conn(0x7f147c0377c0 0x7f147c039c80 unknown :-1 s=CLOSED pgs=3811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:03:31.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.675+0000 7f149fc38640 1 --2- 172.21.15.42:0/379819495 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1498136e10 0x7f14980a3020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:03:31.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.675+0000 7f149fc38640 1 --2- 172.21.15.42:0/379819495 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f14980a37f0 0x7f14980a2ae0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:03:31.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.675+0000 7f149fc38640 1 --2- 172.21.15.42:0/379819495 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f149806db90 0x7f14981cf190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:03:31.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.675+0000 7f149fc38640 1 -- 172.21.15.42:0/379819495 >> 172.21.15.42:0/379819495 conn(0x7f14980a9fb0 msgr2=0x7f1498132a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:03:31.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.675+0000 7f149fc38640 1 -- 172.21.15.42:0/379819495 shutdown_connections 2024-09-16T04:03:31.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.675+0000 7f149fc38640 1 -- 172.21.15.42:0/379819495 wait complete. 2024-09-16T04:03:31.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:31.675+0000 7f149fc38640 1 librados: shutdown 2024-09-16T04:03:34.694 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:03:34.826 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:34.823+0000 7fc370813640 10 monclient: get_monmap_and_config 2024-09-16T04:03:34.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:34.823+0000 7fc370813640 10 monclient: build_initial_monmap 2024-09-16T04:03:34.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:34.823+0000 7fc370813640 10 monclient: monmap: 2024-09-16T04:03:34.827 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:03:34.827 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:03:34.827 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:03:34.825459+0000 2024-09-16T04:03:34.827 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:03:34.825459+0000 2024-09-16T04:03:34.827 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:03:34.827 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:03:34.827 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:03:34.828 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:03:34.828 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:03:34.828 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:03:34.828 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:03:34.828 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:03:34.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:34.823+0000 7fc370813640 1 Processor -- start 2024-09-16T04:03:34.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:34.823+0000 7fc370813640 1 -- start start 2024-09-16T04:03:34.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:34.823+0000 7fc370813640 10 monclient: init 2024-09-16T04:03:34.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:34.823+0000 7fc370813640 10 monclient: _reopen_session rank -1 2024-09-16T04:03:34.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:34.823+0000 7fc370813640 10 monclient: _add_conns ranks=[0,2,1,3,4] 2024-09-16T04:03:34.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:34.823+0000 7fc370813640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc36c0a1330 addr v1:172.21.15.42:6789/0 2024-09-16T04:03:34.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:34.823+0000 7fc370813640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc36c0a1710 0x7fc36c0a1b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:03:34.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:34.823+0000 7fc370813640 10 monclient(hunting): picked mon.noname-b con 0x7fc36c0a1710 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:03:34.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:34.823+0000 7fc370813640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc36c1393a0 addr v1:172.21.15.154:6789/0 2024-09-16T04:03:34.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:34.823+0000 7fc370813640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc36c09e4c0 con 0x7fc36c0a1330 2024-09-16T04:03:34.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:34.823+0000 7fc370813640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc36c0a2050 con 0x7fc36c1393a0 2024-09-16T04:03:34.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:34.823+0000 7fc370813640 10 monclient(hunting): start opening mon connection 2024-09-16T04:03:34.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:34.823+0000 7fc370813640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fc36c09edf0 con 0x7fc36c0a1710 2024-09-16T04:03:34.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:34.823+0000 7fc370813640 10 monclient(hunting): _renew_subs 2024-09-16T04:03:34.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:34.823+0000 7fc370813640 10 monclient(hunting): authenticate will time out at 2261.794914s 2024-09-16T04:03:34.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:34.823+0000 7fc36a59c640 1 -- >> v1:172.21.15.42:6789/0 conn(0x7fc36c0a1330 legacy=0x7fc36c09f180 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.42:6789/0 2024-09-16T04:03:34.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:34.823+0000 7fc369d9b640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc36c0a1710 0x7fc36c0a1b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:03:34.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:34.823+0000 7fc369d9b640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc36c0a1710 0x7fc36c0a1b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:36798/0 (socket says 172.21.15.42:36798) 2024-09-16T04:03:34.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:34.823+0000 7fc369d9b640 1 -- 172.21.15.42:0/820556166 learned_addr learned my addr 172.21.15.42:0/820556166 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:03:34.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:34.823+0000 7fc369d9b640 10 monclient(hunting): get_auth_request con 0x7fc36c0a1710 auth_method 0 2024-09-16T04:03:34.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:34.823+0000 7fc369d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:03:34.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:34.823+0000 7fc369d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:03:34.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:34.823+0000 7fc369d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:03:34.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:34.823+0000 7fc369d9b640 1 -- 172.21.15.42:0/820556166 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc36c0a1710 msgr2=0x7fc36c0a1b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:03:34.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:34.823+0000 7fc369d9b640 1 -- 172.21.15.42:0/820556166 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc36c0a1710 msgr2=0x7fc36c0a1b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:03:34.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:34.823+0000 7fc369d9b640 1 --2- 172.21.15.42:0/820556166 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc36c0a1710 0x7fc36c0a1b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:03:34.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:34.823+0000 7fc369d9b640 1 --2- 172.21.15.42:0/820556166 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc36c0a1710 0x7fc36c0a1b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:03:35.029 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:35.027+0000 7fc36a59c640 1 -- 172.21.15.42:0/820556166 >> v1:172.21.15.42:6789/0 conn(0x7fc36c0a1330 legacy=0x7fc36c09f180 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.42:6789/0 2024-09-16T04:03:35.029 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:35.027+0000 7fc369d9b640 1 --2- 172.21.15.42:0/820556166 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc36c0a1710 0x7fc36c0a1b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:03:35.030 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:35.027+0000 7fc369d9b640 10 monclient(hunting): get_auth_request con 0x7fc36c0a1710 auth_method 0 2024-09-16T04:03:35.030 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:35.027+0000 7fc369d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:03:35.030 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:35.027+0000 7fc369d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:03:35.030 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:35.027+0000 7fc369d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:03:35.030 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:35.027+0000 7fc369d9b640 1 -- 172.21.15.42:0/820556166 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc36c0a1710 msgr2=0x7fc36c0a1b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:03:35.030 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:35.027+0000 7fc369d9b640 1 -- 172.21.15.42:0/820556166 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc36c0a1710 msgr2=0x7fc36c0a1b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:03:35.030 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:35.027+0000 7fc369d9b640 1 --2- 172.21.15.42:0/820556166 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc36c0a1710 0x7fc36c0a1b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:03:35.030 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:35.027+0000 7fc369d9b640 1 --2- 172.21.15.42:0/820556166 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc36c0a1710 0x7fc36c0a1b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T04:03:35.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:35.431+0000 7fc36a59c640 1 -- 172.21.15.42:0/820556166 >> v1:172.21.15.42:6789/0 conn(0x7fc36c0a1330 legacy=0x7fc36c09f180 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.42:6789/0 2024-09-16T04:03:35.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:35.431+0000 7fc369d9b640 1 --2- 172.21.15.42:0/820556166 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc36c0a1710 0x7fc36c0a1b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:03:35.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:35.431+0000 7fc369d9b640 10 monclient(hunting): get_auth_request con 0x7fc36c0a1710 auth_method 0 2024-09-16T04:03:35.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:35.431+0000 7fc369d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:03:35.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:35.431+0000 7fc369d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:03:35.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:35.431+0000 7fc369d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:03:35.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:35.431+0000 7fc369d9b640 1 -- 172.21.15.42:0/820556166 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc36c0a1710 msgr2=0x7fc36c0a1b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:03:35.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:35.431+0000 7fc369d9b640 1 -- 172.21.15.42:0/820556166 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc36c0a1710 msgr2=0x7fc36c0a1b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:03:35.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:35.431+0000 7fc369d9b640 1 --2- 172.21.15.42:0/820556166 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc36c0a1710 0x7fc36c0a1b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:03:35.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:35.431+0000 7fc369d9b640 1 --2- 172.21.15.42:0/820556166 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc36c0a1710 0x7fc36c0a1b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T04:03:36.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:36.235+0000 7fc36a59c640 1 -- 172.21.15.42:0/820556166 >> v1:172.21.15.42:6789/0 conn(0x7fc36c0a1330 legacy=0x7fc36c09f180 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.42:6789/0 2024-09-16T04:03:36.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:36.235+0000 7fc369d9b640 1 --2- 172.21.15.42:0/820556166 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc36c0a1710 0x7fc36c0a1b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:03:36.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:36.235+0000 7fc369d9b640 10 monclient(hunting): get_auth_request con 0x7fc36c0a1710 auth_method 0 2024-09-16T04:03:36.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:36.235+0000 7fc369d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:03:36.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:36.235+0000 7fc369d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:03:36.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:36.235+0000 7fc369d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:03:36.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:36.235+0000 7fc369d9b640 1 -- 172.21.15.42:0/820556166 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc36c0a1710 msgr2=0x7fc36c0a1b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:03:36.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:36.235+0000 7fc369d9b640 1 -- 172.21.15.42:0/820556166 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc36c0a1710 msgr2=0x7fc36c0a1b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:03:36.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:36.235+0000 7fc369d9b640 1 --2- 172.21.15.42:0/820556166 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc36c0a1710 0x7fc36c0a1b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:03:36.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:36.235+0000 7fc369d9b640 1 --2- 172.21.15.42:0/820556166 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc36c0a1710 0x7fc36c0a1b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T04:03:37.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:37.823+0000 7fc363fff640 10 monclient(hunting): tick 2024-09-16T04:03:37.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:37.823+0000 7fc363fff640 10 monclient(hunting): _check_auth_tickets 2024-09-16T04:03:37.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:37.823+0000 7fc363fff640 1 monclient(hunting): continuing hunt 2024-09-16T04:03:37.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:37.823+0000 7fc363fff640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T04:03:37.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:37.823+0000 7fc363fff640 1 -- 172.21.15.42:0/820556166 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc36c0a1710 msgr2=0x7fc36c0a1b10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:03:37.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:37.823+0000 7fc363fff640 1 --2- 172.21.15.42:0/820556166 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc36c0a1710 0x7fc36c0a1b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:03:37.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:37.823+0000 7fc363fff640 1 -- 172.21.15.42:0/820556166 >> v1:172.21.15.154:6789/0 conn(0x7fc36c1393a0 legacy=0x7fc36c13b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:03:37.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:37.823+0000 7fc363fff640 1 -- 172.21.15.42:0/820556166 >> v1:172.21.15.42:6789/0 conn(0x7fc36c0a1330 legacy=0x7fc36c09f180 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:03:37.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:37.823+0000 7fc363fff640 10 monclient: _add_conns ranks=[4,3] 2024-09-16T04:03:37.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:37.823+0000 7fc363fff640 1 --2- 172.21.15.42:0/820556166 >> v2:172.21.15.154:3300/0 conn(0x7fc34c0036e0 0x7fc34c005b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:03:37.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:37.823+0000 7fc363fff640 10 monclient(hunting): picked mon.noname-c con 0x7fc34c0036e0 addr v2:172.21.15.154:3300/0 2024-09-16T04:03:37.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:37.823+0000 7fc363fff640 1 --2- 172.21.15.42:0/820556166 >> v2:172.21.15.42:3300/0 conn(0x7fc34c006250 0x7fc34c008710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:03:37.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:37.823+0000 7fc363fff640 10 monclient(hunting): picked mon.noname-a con 0x7fc34c006250 addr v2:172.21.15.42:3300/0 2024-09-16T04:03:37.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:37.823+0000 7fc363fff640 10 monclient(hunting): start opening mon connection 2024-09-16T04:03:37.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:37.823+0000 7fc363fff640 1 -- 172.21.15.42:0/820556166 --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7fc36c09edf0 con 0x7fc34c006250 2024-09-16T04:03:37.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:37.823+0000 7fc363fff640 10 monclient(hunting): start opening mon connection 2024-09-16T04:03:37.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:37.823+0000 7fc363fff640 1 -- 172.21.15.42:0/820556166 --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fc34c008dd0 con 0x7fc34c0036e0 2024-09-16T04:03:37.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:37.823+0000 7fc363fff640 10 monclient(hunting): _renew_subs 2024-09-16T04:03:37.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:37.823+0000 7fc369d9b640 1 -- 172.21.15.42:0/820556166 >> v2:172.21.15.42:3300/0 conn(0x7fc34c006250 msgr2=0x7fc34c008710 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3300/0 2024-09-16T04:03:37.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:37.823+0000 7fc369d9b640 1 --2- 172.21.15.42:0/820556166 >> v2:172.21.15.42:3300/0 conn(0x7fc34c006250 0x7fc34c008710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:03:37.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:37.823+0000 7fc36a59c640 1 --2- 172.21.15.42:0/820556166 >> v2:172.21.15.154:3300/0 conn(0x7fc34c0036e0 0x7fc34c005b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:03:37.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:37.823+0000 7fc36a59c640 10 monclient(hunting): get_auth_request con 0x7fc34c0036e0 auth_method 0 2024-09-16T04:03:37.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:37.823+0000 7fc36a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:03:37.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:37.823+0000 7fc36a59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:03:37.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:37.823+0000 7fc36a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:03:37.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:37.827+0000 7fc36a59c640 1 -- 172.21.15.42:0/820556166 >> v2:172.21.15.154:3300/0 conn(0x7fc34c0036e0 msgr2=0x7fc34c005b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:03:37.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:37.827+0000 7fc36a59c640 1 -- 172.21.15.42:0/820556166 >> v2:172.21.15.154:3300/0 conn(0x7fc34c0036e0 msgr2=0x7fc34c005b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:03:37.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:37.827+0000 7fc36a59c640 1 --2- 172.21.15.42:0/820556166 >> v2:172.21.15.154:3300/0 conn(0x7fc34c0036e0 0x7fc34c005b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:03:37.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:37.827+0000 7fc36a59c640 1 --2- 172.21.15.42:0/820556166 >> v2:172.21.15.154:3300/0 conn(0x7fc34c0036e0 0x7fc34c005b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:03:38.029 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:38.027+0000 7fc369d9b640 1 -- 172.21.15.42:0/820556166 >> v2:172.21.15.42:3300/0 conn(0x7fc34c006250 msgr2=0x7fc34c008710 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3300/0 2024-09-16T04:03:38.029 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:38.027+0000 7fc369d9b640 1 --2- 172.21.15.42:0/820556166 >> v2:172.21.15.42:3300/0 conn(0x7fc34c006250 0x7fc34c008710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T04:03:38.034 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:38.031+0000 7fc36a59c640 1 --2- 172.21.15.42:0/820556166 >> v2:172.21.15.154:3300/0 conn(0x7fc34c0036e0 0x7fc34c005b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:03:38.034 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:38.031+0000 7fc36a59c640 10 monclient(hunting): get_auth_request con 0x7fc34c0036e0 auth_method 0 2024-09-16T04:03:38.034 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:38.031+0000 7fc36a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:03:38.034 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:38.031+0000 7fc36a59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:03:38.034 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:38.031+0000 7fc36a59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:03:38.034 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:38.031+0000 7fc36a59c640 1 -- 172.21.15.42:0/820556166 >> v2:172.21.15.154:3300/0 conn(0x7fc34c0036e0 msgr2=0x7fc34c005b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:03:38.034 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:38.031+0000 7fc36a59c640 1 -- 172.21.15.42:0/820556166 >> v2:172.21.15.154:3300/0 conn(0x7fc34c0036e0 msgr2=0x7fc34c005b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:03:38.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:38.031+0000 7fc36a59c640 1 --2- 172.21.15.42:0/820556166 >> v2:172.21.15.154:3300/0 conn(0x7fc34c0036e0 0x7fc34c005b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:03:38.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:38.031+0000 7fc36a59c640 1 --2- 172.21.15.42:0/820556166 >> v2:172.21.15.154:3300/0 conn(0x7fc34c0036e0 0x7fc34c005b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T04:03:38.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:38.431+0000 7fc369d9b640 1 -- 172.21.15.42:0/820556166 >> v2:172.21.15.42:3300/0 conn(0x7fc34c006250 msgr2=0x7fc34c008710 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3300/0 2024-09-16T04:03:38.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:38.431+0000 7fc369d9b640 1 --2- 172.21.15.42:0/820556166 >> v2:172.21.15.42:3300/0 conn(0x7fc34c006250 0x7fc34c008710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T04:03:38.438 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:38.435+0000 7fc36a59c640 1 --2- 172.21.15.42:0/820556166 >> v2:172.21.15.154:3300/0 conn(0x7fc34c0036e0 0x7fc34c005b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:03:38.438 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:38.435+0000 7fc36a59c640 10 monclient(hunting): get_auth_request con 0x7fc34c0036e0 auth_method 0 2024-09-16T04:03:38.438 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:38.435+0000 7fc36a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:03:38.438 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:38.435+0000 7fc36a59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:03:38.439 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:38.435+0000 7fc36a59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:03:38.439 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:38.435+0000 7fc36a59c640 1 -- 172.21.15.42:0/820556166 >> v2:172.21.15.154:3300/0 conn(0x7fc34c0036e0 msgr2=0x7fc34c005b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:03:38.439 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:38.435+0000 7fc36a59c640 1 -- 172.21.15.42:0/820556166 >> v2:172.21.15.154:3300/0 conn(0x7fc34c0036e0 msgr2=0x7fc34c005b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:03:38.439 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:38.435+0000 7fc36a59c640 1 --2- 172.21.15.42:0/820556166 >> v2:172.21.15.154:3300/0 conn(0x7fc34c0036e0 0x7fc34c005b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:03:38.439 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:38.435+0000 7fc36a59c640 1 --2- 172.21.15.42:0/820556166 >> v2:172.21.15.154:3300/0 conn(0x7fc34c0036e0 0x7fc34c005b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T04:03:39.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.235+0000 7fc369d9b640 1 -- 172.21.15.42:0/820556166 >> v2:172.21.15.42:3300/0 conn(0x7fc34c006250 msgr2=0x7fc34c008710 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3300/0 2024-09-16T04:03:39.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.235+0000 7fc369d9b640 1 --2- 172.21.15.42:0/820556166 >> v2:172.21.15.42:3300/0 conn(0x7fc34c006250 0x7fc34c008710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T04:03:39.242 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.239+0000 7fc36a59c640 1 --2- 172.21.15.42:0/820556166 >> v2:172.21.15.154:3300/0 conn(0x7fc34c0036e0 0x7fc34c005b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:03:39.242 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.239+0000 7fc36a59c640 10 monclient(hunting): get_auth_request con 0x7fc34c0036e0 auth_method 0 2024-09-16T04:03:39.242 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.239+0000 7fc36a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:03:39.242 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.239+0000 7fc36a59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:03:39.242 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.239+0000 7fc36a59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:03:39.242 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.239+0000 7fc36a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:03:39.242 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.239+0000 7fc36a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:03:39.242 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.239+0000 7fc36a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:03:39.243 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.239+0000 7fc36a59c640 10 monclient(hunting): handle_auth_done global_id 904753 payload 274 2024-09-16T04:03:39.243 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.239+0000 7fc36a59c640 1 -- 172.21.15.42:0/820556166 >> v2:172.21.15.42:3300/0 conn(0x7fc34c006250 msgr2=0x7fc34c008710 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:03:39.243 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.239+0000 7fc36a59c640 1 --2- 172.21.15.42:0/820556166 >> v2:172.21.15.42:3300/0 conn(0x7fc34c006250 0x7fc34c008710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:03:39.243 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.239+0000 7fc36a59c640 10 monclient: _finish_hunting 0 2024-09-16T04:03:39.243 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.239+0000 7fc36a59c640 1 monclient: found mon.noname-c 2024-09-16T04:03:39.243 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.239+0000 7fc36a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:03:39.243 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.239+0000 7fc36a59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.154:3300/0 2024-09-16T04:03:39.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.239+0000 7fc36a59c640 1 -- 172.21.15.42:0/820556166 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc36c0a3170 con 0x7fc34c0036e0 2024-09-16T04:03:39.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.243+0000 7fc36a59c640 1 --2- 172.21.15.42:0/820556166 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc34c0036e0 0x7fc34c005b60 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fc34c003690 tx=0x7fc35401d180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7b8693cb9289e3ec server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:03:39.249 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.243+0000 7fc36959a640 1 -- 172.21.15.42:0/820556166 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3540b3070 con 0x7fc34c0036e0 2024-09-16T04:03:39.249 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.243+0000 7fc36959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:03:39.249 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.243+0000 7fc36959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:03:39.249 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.243+0000 7fc36959a640 10 monclient: dump: 2024-09-16T04:03:39.249 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:03:39.249 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:03:39.249 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:39.249 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:39.250 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:03:39.250 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:03:39.250 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:03:39.250 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:03:39.250 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:03:39.250 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:03:39.250 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.243+0000 7fc36959a640 10 monclient: _finish_auth 0 2024-09-16T04:03:39.250 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.243+0000 7fc36959a640 10 monclient: _check_auth_tickets 2024-09-16T04:03:39.250 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.243+0000 7fc36959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:03:39.250 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.243+0000 7fc36959a640 1 -- 172.21.15.42:0/820556166 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc3540ae410 con 0x7fc34c0036e0 2024-09-16T04:03:39.250 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.243+0000 7fc36959a640 10 monclient: handle_config config(0 keys) 2024-09-16T04:03:39.250 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.243+0000 7fc36959a640 1 -- 172.21.15.42:0/820556166 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3540ae800 con 0x7fc34c0036e0 2024-09-16T04:03:39.250 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.243+0000 7fc370813640 5 monclient: authenticate success, global_id 904753 2024-09-16T04:03:39.251 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.243+0000 7fc370813640 10 monclient: get_monmap_and_config success 2024-09-16T04:03:39.251 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.243+0000 7fc370813640 4 set_mon_vals no callback set 2024-09-16T04:03:39.251 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.243+0000 7fc36959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:03:39.251 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.243+0000 7fc36959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:03:39.251 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.243+0000 7fc36959a640 10 monclient: dump: 2024-09-16T04:03:39.251 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:03:39.251 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:03:39.251 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:39.251 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:39.251 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:03:39.251 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:03:39.251 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:03:39.251 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:03:39.251 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:03:39.252 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:03:39.255 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.247+0000 7fc370813640 10 monclient: shutdown 2024-09-16T04:03:39.255 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.247+0000 7fc370813640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:03:39.255 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.247+0000 7fc370813640 1 -- 172.21.15.42:0/820556166 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc34c0036e0 msgr2=0x7fc34c005b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:03:39.255 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.247+0000 7fc370813640 1 --2- 172.21.15.42:0/820556166 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc34c0036e0 0x7fc34c005b60 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fc34c003690 tx=0x7fc35401d180 comp rx=0 tx=0).stop 2024-09-16T04:03:39.255 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.247+0000 7fc36ad9d640 1 -- 172.21.15.42:0/820556166 reap_dead start 2024-09-16T04:03:39.255 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.247+0000 7fc370813640 1 -- 172.21.15.42:0/820556166 shutdown_connections 2024-09-16T04:03:39.255 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.247+0000 7fc370813640 1 --2- 172.21.15.42:0/820556166 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc34c0036e0 0x7fc34c005b60 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:03:39.256 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.247+0000 7fc370813640 1 -- 172.21.15.42:0/820556166 >> 172.21.15.42:0/820556166 conn(0x7fc36c0a2420 msgr2=0x7fc36c0a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:03:39.256 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.247+0000 7fc370813640 1 -- 172.21.15.42:0/820556166 shutdown_connections 2024-09-16T04:03:39.256 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.247+0000 7fc370813640 1 -- 172.21.15.42:0/820556166 wait complete. 2024-09-16T04:03:39.256 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.247+0000 7fc370813640 10 monclient: build_initial_monmap 2024-09-16T04:03:39.256 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.247+0000 7fc370813640 10 monclient: monmap: 2024-09-16T04:03:39.256 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:03:39.256 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:03:39.256 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:03:39.256 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:03:39.256 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:03:39.256 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:03:39.256 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:03:39.256 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:03:39.256 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:03:39.257 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:03:39.257 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.247+0000 7fc370813640 1 librados: starting msgr at 2024-09-16T04:03:39.257 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.247+0000 7fc370813640 1 librados: starting objecter 2024-09-16T04:03:39.257 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.247+0000 7fc370813640 1 Processor -- start 2024-09-16T04:03:39.257 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.247+0000 7fc370813640 1 -- start start 2024-09-16T04:03:39.257 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.247+0000 7fc370813640 1 librados: setting wanted keys 2024-09-16T04:03:39.257 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.247+0000 7fc370813640 1 librados: calling monclient init 2024-09-16T04:03:39.257 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.247+0000 7fc370813640 10 monclient: init 2024-09-16T04:03:39.257 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.247+0000 7fc370813640 10 monclient: _reopen_session rank -1 2024-09-16T04:03:39.257 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.247+0000 7fc370813640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:03:39.257 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.247+0000 7fc370813640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc34c0036e0 0x7fc36c1ceda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:03:39.257 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.247+0000 7fc370813640 10 monclient(hunting): picked mon.noname-b con 0x7fc34c0036e0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:03:39.258 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.247+0000 7fc370813640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc36c1344e0 0x7fc36c134960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:03:39.258 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.251+0000 7fc370813640 10 monclient(hunting): picked mon.noname-a con 0x7fc36c1344e0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:03:39.258 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.251+0000 7fc370813640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc36c06f8a0 0x7fc36c06fd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:03:39.258 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.251+0000 7fc370813640 10 monclient(hunting): picked mon.noname-c con 0x7fc36c06f8a0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:03:39.258 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.251+0000 7fc370813640 10 monclient(hunting): start opening mon connection 2024-09-16T04:03:39.258 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.251+0000 7fc369d9b640 1 -- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc36c1344e0 msgr2=0x7fc36c134960 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.42:3300/0 2024-09-16T04:03:39.258 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.251+0000 7fc370813640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fc36c09bde0 con 0x7fc36c1344e0 2024-09-16T04:03:39.258 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.251+0000 7fc369d9b640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc36c1344e0 0x7fc36c134960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:03:39.258 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.251+0000 7fc370813640 10 monclient(hunting): start opening mon connection 2024-09-16T04:03:39.258 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.251+0000 7fc370813640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc36c09bc60 con 0x7fc34c0036e0 2024-09-16T04:03:39.258 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.251+0000 7fc370813640 10 monclient(hunting): start opening mon connection 2024-09-16T04:03:39.258 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.251+0000 7fc370813640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fc36c09bf60 con 0x7fc36c06f8a0 2024-09-16T04:03:39.258 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.251+0000 7fc370813640 10 monclient(hunting): _renew_subs 2024-09-16T04:03:39.258 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.251+0000 7fc370813640 10 monclient(hunting): authenticate will time out at 2266.220180s 2024-09-16T04:03:39.259 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.251+0000 7fc36ad9d640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc36c06f8a0 0x7fc36c06fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:03:39.259 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.251+0000 7fc36ad9d640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc36c06f8a0 0x7fc36c06fd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:36836/0 (socket says 172.21.15.42:36836) 2024-09-16T04:03:39.259 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.251+0000 7fc36ad9d640 1 -- 172.21.15.42:0/1885260067 learned_addr learned my addr 172.21.15.42:0/1885260067 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:03:39.259 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.251+0000 7fc36ad9d640 10 monclient(hunting): get_auth_request con 0x7fc36c06f8a0 auth_method 0 2024-09-16T04:03:39.259 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.251+0000 7fc36ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:03:39.259 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.251+0000 7fc36ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:03:39.259 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.251+0000 7fc36ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:03:39.259 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.251+0000 7fc36ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:03:39.259 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.251+0000 7fc36ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:03:39.259 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.251+0000 7fc36ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:03:39.259 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.251+0000 7fc36a59c640 1 --2- 172.21.15.42:0/1885260067 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc34c0036e0 0x7fc36c1ceda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:03:39.259 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.251+0000 7fc36a59c640 10 monclient(hunting): get_auth_request con 0x7fc34c0036e0 auth_method 0 2024-09-16T04:03:39.259 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.251+0000 7fc36a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:03:39.259 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.251+0000 7fc36a59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:03:39.260 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.251+0000 7fc36a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:03:39.260 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.251+0000 7fc36a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:03:39.260 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.251+0000 7fc36a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:03:39.260 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.251+0000 7fc36a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:03:39.260 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.251+0000 7fc36ad9d640 10 monclient(hunting): handle_auth_done global_id 974156 payload 824 2024-09-16T04:03:39.260 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.251+0000 7fc36ad9d640 1 -- 172.21.15.42:0/1885260067 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc34c0036e0 msgr2=0x7fc36c1ceda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:03:39.260 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.251+0000 7fc36ad9d640 1 --2- 172.21.15.42:0/1885260067 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc34c0036e0 0x7fc36c1ceda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:03:39.260 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.251+0000 7fc36ad9d640 1 -- 172.21.15.42:0/1885260067 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc36c1344e0 msgr2=0x7fc36c134960 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:03:39.260 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.251+0000 7fc36ad9d640 1 --2- 172.21.15.42:0/1885260067 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc36c1344e0 0x7fc36c134960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:03:39.260 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.251+0000 7fc36ad9d640 10 monclient: _finish_hunting 0 2024-09-16T04:03:39.260 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.251+0000 7fc36ad9d640 1 monclient: found mon.noname-c 2024-09-16T04:03:39.260 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.251+0000 7fc36ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:03:39.260 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.251+0000 7fc36ad9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T04:03:39.261 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.251+0000 7fc36ad9d640 1 -- 172.21.15.42:0/1885260067 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc36c0a4730 con 0x7fc36c06f8a0 2024-09-16T04:03:39.261 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.251+0000 7fc36ad9d640 1 --2- 172.21.15.42:0/1885260067 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc36c06f8a0 0x7fc36c06fd20 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fc36c09df70 tx=0x7fc36400cb90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:03:39.261 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.251+0000 7fc362ffd640 10 client.?.objecter ms_handle_connect 0x7fc36c06f8a0 2024-09-16T04:03:39.261 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.251+0000 7fc362ffd640 10 client.?.objecter resend_mon_ops 2024-09-16T04:03:39.261 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.251+0000 7fc362ffd640 1 -- 172.21.15.42:0/1885260067 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc364017070 con 0x7fc36c06f8a0 2024-09-16T04:03:39.261 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.251+0000 7fc362ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:03:39.261 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.251+0000 7fc362ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:03:39.261 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.251+0000 7fc362ffd640 10 monclient: dump: 2024-09-16T04:03:39.261 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:03:39.261 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:03:39.261 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:39.261 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:39.261 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:03:39.261 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:03:39.262 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:03:39.262 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:03:39.262 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:03:39.262 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:03:39.262 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.251+0000 7fc362ffd640 10 monclient: _finish_auth 0 2024-09-16T04:03:39.262 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.251+0000 7fc362ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:03:39.262 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.251+0000 7fc362ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:03:39.262 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.251+0000 7fc370813640 5 monclient: authenticate success, global_id 974156 2024-09-16T04:03:39.262 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.251+0000 7fc370813640 10 monclient: _renew_subs 2024-09-16T04:03:39.262 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.251+0000 7fc362ffd640 1 -- 172.21.15.42:0/1885260067 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc364005d50 con 0x7fc36c06f8a0 2024-09-16T04:03:39.262 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.251+0000 7fc370813640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:03:39.262 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.251+0000 7fc370813640 1 -- 172.21.15.42:0/1885260067 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc36c11cb70 con 0x7fc36c06f8a0 2024-09-16T04:03:39.262 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.251+0000 7fc362ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:03:39.262 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.251+0000 7fc362ffd640 1 -- 172.21.15.42:0/1885260067 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc36400a850 con 0x7fc36c06f8a0 2024-09-16T04:03:39.263 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.251+0000 7fc362ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:03:39.263 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.251+0000 7fc368d99640 4 set_mon_vals no callback set 2024-09-16T04:03:39.263 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.251+0000 7fc362ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:03:39.263 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.251+0000 7fc370813640 10 client.974156.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:03:39.263 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.251+0000 7fc362ffd640 10 monclient: dump: 2024-09-16T04:03:39.263 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:03:39.263 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:03:39.263 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:39.263 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:39.263 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:03:39.263 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:03:39.263 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:03:39.263 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:03:39.263 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:03:39.263 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:03:39.264 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.251+0000 7fc370813640 10 monclient: _renew_subs 2024-09-16T04:03:39.264 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.251+0000 7fc370813640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:03:39.264 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.251+0000 7fc370813640 1 -- 172.21.15.42:0/1885260067 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc36c11bf70 con 0x7fc36c06f8a0 2024-09-16T04:03:39.264 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.251+0000 7fc370813640 1 librados: init done 2024-09-16T04:03:39.265 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.251+0000 7fc370813640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:03:39.265 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.251+0000 7fc370813640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:03:39.265 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.251+0000 7fc370813640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:03:39.265 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.251+0000 7fc370813640 1 -- 172.21.15.42:0/1885260067 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc36c073a30 con 0x7fc36c06f8a0 2024-09-16T04:03:39.265 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.251+0000 7fc362ffd640 1 -- 172.21.15.42:0/1885260067 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fc36401a070 con 0x7fc36c06f8a0 2024-09-16T04:03:39.266 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.251+0000 7fc362ffd640 1 --2- 172.21.15.42:0/1885260067 >> v2:172.21.15.154:6812/254845342 conn(0x7fc34c038650 0x7fc34c03ab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:03:39.266 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.251+0000 7fc362ffd640 1 -- 172.21.15.42:0/1885260067 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(655..655 src has 451..655) ==== 6338+0+0 (secure 0 0 0) 0x7fc364048400 con 0x7fc36c06f8a0 2024-09-16T04:03:39.266 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.251+0000 7fc362ffd640 10 client.974156.objecter ms_dispatch 0x7fc36c000ba0 osd_map(655..655 src has 451..655) 2024-09-16T04:03:39.266 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.251+0000 7fc362ffd640 3 client.974156.objecter handle_osd_map got epochs [655,655] > 0 2024-09-16T04:03:39.266 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.251+0000 7fc362ffd640 3 client.974156.objecter handle_osd_map decoding full epoch 655 2024-09-16T04:03:39.266 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.251+0000 7fc362ffd640 20 client.974156.objecter dump_active .. 0 homeless 2024-09-16T04:03:39.266 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.251+0000 7fc36a59c640 1 --2- 172.21.15.42:0/1885260067 >> v2:172.21.15.154:6812/254845342 conn(0x7fc34c038650 0x7fc34c03ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:03:39.266 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.251+0000 7fc36a59c640 10 monclient: get_auth_request con 0x7fc34c038650 auth_method 0 2024-09-16T04:03:39.266 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.255+0000 7fc362ffd640 10 client.974156.objecter ms_handle_connect 0x7fc34c038650 2024-09-16T04:03:39.266 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.255+0000 7fc36a59c640 1 --2- 172.21.15.42:0/1885260067 >> v2:172.21.15.154:6812/254845342 conn(0x7fc34c038650 0x7fc34c03ab10 secure :-1 s=READY pgs=3812 cs=0 l=1 rev1=1 crypto rx=0x7fc3540b0000 tx=0x7fc3540a7000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:03:39.266 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.259+0000 7fc362ffd640 1 -- 172.21.15.42:0/1885260067 <== mon.2 v2:172.21.15.42:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7fc36c0a4730 con 0x7fc36c06f8a0 2024-09-16T04:03:39.266 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.259+0000 7fc362ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:03:39.266 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.259+0000 7fc362ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:03:39.492 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.487+0000 7fc370813640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:03:39.493 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.487+0000 7fc370813640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:03:39.493 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.487+0000 7fc370813640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:03:39.493 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.487+0000 7fc370813640 1 -- 172.21.15.42:0/1885260067 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fc36c0a1b10 con 0x7fc36c06f8a0 2024-09-16T04:03:39.493 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.491+0000 7fc362ffd640 1 -- 172.21.15.42:0/1885260067 <== mon.2 v2:172.21.15.42:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1555 (secure 0 0 0) 0x7fc36401abb0 con 0x7fc36c06f8a0 2024-09-16T04:03:39.493 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.491+0000 7fc362ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:03:39.493 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.491+0000 7fc362ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:03:39.494 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":194,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:03:39.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.491+0000 7fc370813640 10 librados: watch_flush enter 2024-09-16T04:03:39.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.491+0000 7fc370813640 10 librados: watch_flush exit 2024-09-16T04:03:39.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.491+0000 7fc370813640 20 client.974156.objecter shutdown clearing up homeless session... 2024-09-16T04:03:39.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.491+0000 7fc370813640 10 client.974156.objecter successfully canceled tick 2024-09-16T04:03:39.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.491+0000 7fc370813640 1 -- 172.21.15.42:0/1885260067 >> v2:172.21.15.154:6812/254845342 conn(0x7fc34c038650 msgr2=0x7fc34c03ab10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:03:39.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.491+0000 7fc370813640 1 --2- 172.21.15.42:0/1885260067 >> v2:172.21.15.154:6812/254845342 conn(0x7fc34c038650 0x7fc34c03ab10 secure :-1 s=READY pgs=3812 cs=0 l=1 rev1=1 crypto rx=0x7fc3540b0000 tx=0x7fc3540a7000 comp rx=0 tx=0).stop 2024-09-16T04:03:39.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.491+0000 7fc370813640 10 monclient: shutdown 2024-09-16T04:03:39.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.491+0000 7fc370813640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:03:39.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.491+0000 7fc370813640 1 -- 172.21.15.42:0/1885260067 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc36c06f8a0 msgr2=0x7fc36c06fd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:03:39.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.491+0000 7fc370813640 1 --2- 172.21.15.42:0/1885260067 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc36c06f8a0 0x7fc36c06fd20 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fc36c09df70 tx=0x7fc36400cb90 comp rx=0 tx=0).stop 2024-09-16T04:03:39.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.491+0000 7fc370813640 1 -- 172.21.15.42:0/1885260067 shutdown_connections 2024-09-16T04:03:39.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.491+0000 7fc370813640 1 --2- 172.21.15.42:0/1885260067 >> v2:172.21.15.154:6812/254845342 conn(0x7fc34c038650 0x7fc34c03ab10 unknown :-1 s=CLOSED pgs=3812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:03:39.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.491+0000 7fc370813640 1 --2- 172.21.15.42:0/1885260067 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc36c06f8a0 0x7fc36c06fd20 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:03:39.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.491+0000 7fc370813640 1 --2- 172.21.15.42:0/1885260067 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc36c1344e0 0x7fc36c134960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:03:39.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.491+0000 7fc370813640 1 --2- 172.21.15.42:0/1885260067 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc34c0036e0 0x7fc36c1ceda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:03:39.498 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.491+0000 7fc370813640 1 -- 172.21.15.42:0/1885260067 >> 172.21.15.42:0/1885260067 conn(0x7fc36c0a2420 msgr2=0x7fc36c1385c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:03:39.498 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.491+0000 7fc370813640 1 -- 172.21.15.42:0/1885260067 shutdown_connections 2024-09-16T04:03:39.498 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.491+0000 7fc370813640 1 -- 172.21.15.42:0/1885260067 wait complete. 2024-09-16T04:03:39.498 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.491+0000 7fc370813640 1 librados: shutdown 2024-09-16T04:03:39.513 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T04:03:39.513 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-16T04:03:39.513 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T04:03:39.628 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.623+0000 7f55c30aa640 10 monclient: get_monmap_and_config 2024-09-16T04:03:39.628 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.623+0000 7f55c30aa640 10 monclient: build_initial_monmap 2024-09-16T04:03:39.628 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.623+0000 7f55c30aa640 10 monclient: monmap: 2024-09-16T04:03:39.628 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:03:39.628 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:03:39.628 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:03:39.626440+0000 2024-09-16T04:03:39.628 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:03:39.626440+0000 2024-09-16T04:03:39.628 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:03:39.628 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:03:39.629 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:03:39.629 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:03:39.629 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:03:39.629 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:03:39.629 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:03:39.629 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:03:39.633 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.623+0000 7f55c30aa640 1 Processor -- start 2024-09-16T04:03:39.633 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.623+0000 7f55c30aa640 1 -- start start 2024-09-16T04:03:39.633 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.623+0000 7f55c30aa640 10 monclient: init 2024-09-16T04:03:39.633 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.623+0000 7f55c30aa640 10 monclient: _reopen_session rank -1 2024-09-16T04:03:39.633 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.623+0000 7f55c30aa640 10 monclient: _add_conns ranks=[0,1,3,4,2] 2024-09-16T04:03:39.633 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.623+0000 7f55c30aa640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f55bc0a2f10 addr v1:172.21.15.42:6789/0 2024-09-16T04:03:39.634 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.623+0000 7f55c30aa640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f55bc09e070 addr v1:172.21.15.154:6789/0 2024-09-16T04:03:39.634 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.627+0000 7f55c30aa640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f55bc141b40 0x7f55bc143f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:03:39.634 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.627+0000 7f55c30aa640 10 monclient(hunting): picked mon.noname-a con 0x7f55bc141b40 addr v2:172.21.15.42:3300/0 2024-09-16T04:03:39.634 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.627+0000 7f55c30aa640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f55bc09dc50 con 0x7f55bc0a2f10 2024-09-16T04:03:39.634 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.627+0000 7f55c30aa640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f55bc0a2150 con 0x7f55bc09e070 2024-09-16T04:03:39.634 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.627+0000 7f55c30aa640 10 monclient(hunting): start opening mon connection 2024-09-16T04:03:39.634 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.627+0000 7f55c30aa640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f55bc0a32f0 con 0x7f55bc141b40 2024-09-16T04:03:39.634 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.627+0000 7f55c0e46640 1 -- >> v1:172.21.15.42:6789/0 conn(0x7f55bc0a2f10 legacy=0x7f55bc0a1330 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.42:6789/0 2024-09-16T04:03:39.634 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.627+0000 7f55c30aa640 10 monclient(hunting): _renew_subs 2024-09-16T04:03:39.634 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.627+0000 7f55c1647640 1 -- >> v2:172.21.15.42:3300/0 conn(0x7f55bc141b40 msgr2=0x7f55bc143f30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3300/0 2024-09-16T04:03:39.634 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.627+0000 7f55c30aa640 10 monclient(hunting): authenticate will time out at 2266.596093s 2024-09-16T04:03:39.634 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.627+0000 7f55c1647640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f55bc141b40 0x7f55bc143f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:03:39.635 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.627+0000 7f55bbfff640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f55bc09e070 0x7f55bc0a1a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.42:33130/0 (socket says 172.21.15.42:33130) 2024-09-16T04:03:39.635 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.627+0000 7f55bbfff640 1 -- 172.21.15.42:0/2267957407 learned_addr learned my addr 172.21.15.42:0/2267957407 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T04:03:39.635 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.627+0000 7f55bb7fe640 1 -- 172.21.15.42:0/2267957407 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 915180084 0 0) 0x7f55bc0a2150 con 0x7f55bc09e070 2024-09-16T04:03:39.635 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.627+0000 7f55bb7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:03:39.635 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.627+0000 7f55bb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:03:39.635 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.627+0000 7f55bb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:03:39.635 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.627+0000 7f55bb7fe640 10 monclient(hunting): my global_id is 904759 2024-09-16T04:03:39.635 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.627+0000 7f55bb7fe640 1 -- 172.21.15.42:0/2267957407 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f55a4003530 con 0x7f55bc09e070 2024-09-16T04:03:39.635 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.627+0000 7f55bb7fe640 1 -- 172.21.15.42:0/2267957407 <== mon.1 v1:172.21.15.154:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 92045097 0 0) 0x7f55a4003530 con 0x7f55bc09e070 2024-09-16T04:03:39.636 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.627+0000 7f55bb7fe640 1 -- 172.21.15.42:0/2267957407 >> v2:172.21.15.42:3300/0 conn(0x7f55bc141b40 msgr2=0x7f55bc143f30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:03:39.636 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.627+0000 7f55bb7fe640 1 --2- 172.21.15.42:0/2267957407 >> v2:172.21.15.42:3300/0 conn(0x7f55bc141b40 0x7f55bc143f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:03:39.636 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.627+0000 7f55bb7fe640 1 -- 172.21.15.42:0/2267957407 >> v1:172.21.15.42:6789/0 conn(0x7f55bc0a2f10 legacy=0x7f55bc0a1330 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:03:39.636 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.627+0000 7f55bb7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:03:39.636 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.627+0000 7f55bb7fe640 1 monclient: found mon.noname-c-legacy 2024-09-16T04:03:39.636 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.627+0000 7f55bb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:03:39.636 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.627+0000 7f55bb7fe640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.154:6789/0 2024-09-16T04:03:39.636 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.627+0000 7f55bb7fe640 1 -- 172.21.15.42:0/2267957407 --> v1:172.21.15.154:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55bc073290 con 0x7f55bc09e070 2024-09-16T04:03:39.636 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.627+0000 7f55bb7fe640 10 monclient: _finish_auth 0 2024-09-16T04:03:39.636 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.627+0000 7f55bb7fe640 10 monclient: _check_auth_tickets 2024-09-16T04:03:39.636 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.627+0000 7f55bb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:03:39.637 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.627+0000 7f55bb7fe640 1 -- 172.21.15.42:0/2267957407 <== mon.1 v1:172.21.15.154:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2177413734 0 0) 0x7f55b0001a00 con 0x7f55bc09e070 2024-09-16T04:03:39.637 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.627+0000 7f55bb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:03:39.637 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.627+0000 7f55bb7fe640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-16T04:03:39.637 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.627+0000 7f55bb7fe640 10 monclient: dump: 2024-09-16T04:03:39.637 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:03:39.637 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:03:39.637 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:39.637 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:39.637 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:03:39.637 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:03:39.637 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:03:39.637 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:03:39.638 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:03:39.638 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:03:39.638 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.627+0000 7f55bb7fe640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] but i'm connected to v1:172.21.15.154:6789/0, reconnecting 2024-09-16T04:03:39.638 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.627+0000 7f55bb7fe640 10 monclient: _reopen_session rank -1 2024-09-16T04:03:39.638 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.627+0000 7f55bb7fe640 1 -- 172.21.15.42:0/2267957407 >> v1:172.21.15.154:6789/0 conn(0x7f55bc09e070 legacy=0x7f55bc0a1a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:03:39.638 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.627+0000 7f55bb7fe640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:03:39.638 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.627+0000 7f55bb7fe640 1 --2- 172.21.15.42:0/2267957407 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f55a40039c0 0x7f55a4003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:03:39.638 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.627+0000 7f55bb7fe640 10 monclient(hunting): picked mon.b con 0x7f55a40039c0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:03:39.638 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.627+0000 7f55bb7fe640 1 --2- 172.21.15.42:0/2267957407 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f55a4004810 0x7f55a4004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:03:39.638 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.627+0000 7f55bb7fe640 10 monclient(hunting): picked mon.c con 0x7f55a4004810 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:03:39.638 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.627+0000 7f55bb7fe640 1 --2- 172.21.15.42:0/2267957407 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f55a40053d0 0x7f55a400ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:03:39.638 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.627+0000 7f55bb7fe640 10 monclient(hunting): picked mon.a con 0x7f55a40053d0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:03:39.638 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.627+0000 7f55bb7fe640 10 monclient(hunting): start opening mon connection 2024-09-16T04:03:39.638 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.627+0000 7f55bb7fe640 1 -- 172.21.15.42:0/2267957407 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f55bc0a32f0 con 0x7f55a40053d0 2024-09-16T04:03:39.639 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.627+0000 7f55bb7fe640 10 monclient(hunting): start opening mon connection 2024-09-16T04:03:39.639 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.627+0000 7f55bb7fe640 1 -- 172.21.15.42:0/2267957407 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f55a400c310 con 0x7f55a40039c0 2024-09-16T04:03:39.639 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.627+0000 7f55bb7fe640 10 monclient(hunting): start opening mon connection 2024-09-16T04:03:39.639 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.627+0000 7f55bb7fe640 1 -- 172.21.15.42:0/2267957407 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f55a400c510 con 0x7f55a4004810 2024-09-16T04:03:39.639 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.627+0000 7f55bb7fe640 10 monclient(hunting): _renew_subs 2024-09-16T04:03:39.639 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.627+0000 7f55bb7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:03:39.639 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.627+0000 7f55c1647640 1 -- 172.21.15.42:0/2267957407 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f55a40053d0 msgr2=0x7f55a400ba30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3300/0 2024-09-16T04:03:39.639 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.627+0000 7f55c1647640 1 --2- 172.21.15.42:0/2267957407 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f55a40053d0 0x7f55a400ba30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:03:39.639 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.627+0000 7f55bbfff640 1 --2- 172.21.15.42:0/2267957407 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f55a4004810 0x7f55a4004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:03:39.639 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.627+0000 7f55bbfff640 10 monclient(hunting): get_auth_request con 0x7f55a4004810 auth_method 0 2024-09-16T04:03:39.639 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.627+0000 7f55bbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:03:39.639 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.627+0000 7f55bbfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:03:39.639 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.627+0000 7f55bbfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:03:39.640 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.627+0000 7f55c0e46640 1 --2- 172.21.15.42:0/2267957407 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f55a40039c0 0x7f55a4003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:03:39.640 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.627+0000 7f55c0e46640 10 monclient(hunting): get_auth_request con 0x7f55a40039c0 auth_method 0 2024-09-16T04:03:39.640 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.627+0000 7f55c0e46640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:03:39.640 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.627+0000 7f55c0e46640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:03:39.640 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.627+0000 7f55c0e46640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:03:39.640 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.627+0000 7f55bbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:03:39.640 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.627+0000 7f55bbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:03:39.640 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.627+0000 7f55bbfff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:03:39.640 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.627+0000 7f55c0e46640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:03:39.640 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.627+0000 7f55c0e46640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:03:39.640 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.627+0000 7f55c0e46640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:03:39.640 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.627+0000 7f55bbfff640 10 monclient(hunting): handle_auth_done global_id 904759 payload 293 2024-09-16T04:03:39.640 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.627+0000 7f55bbfff640 1 -- 172.21.15.42:0/2267957407 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f55a40039c0 msgr2=0x7f55a4003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:03:39.640 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.627+0000 7f55bbfff640 1 --2- 172.21.15.42:0/2267957407 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f55a40039c0 0x7f55a4003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:03:39.641 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.627+0000 7f55bbfff640 1 -- 172.21.15.42:0/2267957407 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f55a40053d0 msgr2=0x7f55a400ba30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:03:39.641 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.627+0000 7f55bbfff640 1 --2- 172.21.15.42:0/2267957407 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f55a40053d0 0x7f55a400ba30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:03:39.641 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.627+0000 7f55bbfff640 10 monclient: _finish_hunting 0 2024-09-16T04:03:39.641 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.627+0000 7f55bbfff640 1 monclient: found mon.c 2024-09-16T04:03:39.641 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.627+0000 7f55bbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:03:39.641 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.627+0000 7f55bbfff640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:03:39.641 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.627+0000 7f55bbfff640 1 -- 172.21.15.42:0/2267957407 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55a4003530 con 0x7f55a4004810 2024-09-16T04:03:39.641 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.627+0000 7f55bbfff640 10 monclient: _finish_auth 0 2024-09-16T04:03:39.641 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.627+0000 7f55bbfff640 10 monclient: _check_auth_tickets 2024-09-16T04:03:39.641 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.627+0000 7f55bbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:03:39.641 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.627+0000 7f55c30aa640 5 monclient: authenticate success, global_id 904759 2024-09-16T04:03:39.641 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.627+0000 7f55c30aa640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:03:39.641 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.627+0000 7f55c1647640 1 -- 172.21.15.42:0/2267957407 reap_dead start 2024-09-16T04:03:39.649 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.631+0000 7f55bbfff640 1 --2- 172.21.15.42:0/2267957407 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f55a4004810 0x7f55a4004c90 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f55b000a7b0 tx=0x7f55b001d280 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a12b35c43303892a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:03:39.650 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.631+0000 7f55bb7fe640 1 -- 172.21.15.42:0/2267957407 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55b000a3d0 con 0x7f55a4004810 2024-09-16T04:03:39.650 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.631+0000 7f55bb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:03:39.650 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.631+0000 7f55bb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:03:39.650 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.631+0000 7f55bb7fe640 10 monclient: dump: 2024-09-16T04:03:39.650 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:03:39.650 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:03:39.650 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:39.650 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:39.650 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:03:39.651 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:03:39.651 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:03:39.651 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:03:39.651 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:03:39.651 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:03:39.651 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.631+0000 7f55bb7fe640 1 -- 172.21.15.42:0/2267957407 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f55b00ad630 con 0x7f55a4004810 2024-09-16T04:03:39.651 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.631+0000 7f55c30aa640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:03:39.651 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.631+0000 7f55bb7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:03:39.651 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.631+0000 7f55c30aa640 10 monclient: get_monmap_and_config success 2024-09-16T04:03:39.651 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.631+0000 7f55bb7fe640 1 -- 172.21.15.42:0/2267957407 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55b00ada60 con 0x7f55a4004810 2024-09-16T04:03:39.651 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.631+0000 7f55bb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:03:39.651 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.631+0000 7f55c30aa640 4 set_mon_vals no callback set 2024-09-16T04:03:39.651 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.631+0000 7f55bb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:03:39.651 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.631+0000 7f55bb7fe640 10 monclient: dump: 2024-09-16T04:03:39.652 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:03:39.652 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:03:39.652 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:39.652 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:39.652 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:03:39.652 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:03:39.652 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:03:39.652 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:03:39.652 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:03:39.652 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:03:39.652 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.631+0000 7f55c30aa640 10 monclient: shutdown 2024-09-16T04:03:39.652 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.631+0000 7f55c30aa640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:03:39.652 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.631+0000 7f55c30aa640 1 -- 172.21.15.42:0/2267957407 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f55a4004810 msgr2=0x7f55a4004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:03:39.652 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.631+0000 7f55c30aa640 1 --2- 172.21.15.42:0/2267957407 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f55a4004810 0x7f55a4004c90 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f55b000a7b0 tx=0x7f55b001d280 comp rx=0 tx=0).stop 2024-09-16T04:03:39.653 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.631+0000 7f55c30aa640 1 -- 172.21.15.42:0/2267957407 shutdown_connections 2024-09-16T04:03:39.653 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.631+0000 7f55c30aa640 1 --2- 172.21.15.42:0/2267957407 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f55a4004810 0x7f55a4004c90 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:03:39.653 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.631+0000 7f55c30aa640 1 -- 172.21.15.42:0/2267957407 >> 172.21.15.42:0/2267957407 conn(0x7f55bc0a2420 msgr2=0x7f55bc0a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:03:39.653 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.631+0000 7f55c30aa640 1 -- 172.21.15.42:0/2267957407 shutdown_connections 2024-09-16T04:03:39.653 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.631+0000 7f55c30aa640 1 -- 172.21.15.42:0/2267957407 wait complete. 2024-09-16T04:03:39.653 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.631+0000 7f55c30aa640 10 monclient: build_initial_monmap 2024-09-16T04:03:39.653 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.631+0000 7f55c30aa640 10 monclient: monmap: 2024-09-16T04:03:39.653 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:03:39.653 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:03:39.653 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:03:39.653 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:03:39.653 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:03:39.653 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:03:39.654 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:03:39.654 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:03:39.654 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:03:39.654 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:03:39.654 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.631+0000 7f55c30aa640 1 librados: starting msgr at 2024-09-16T04:03:39.654 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.631+0000 7f55c30aa640 1 librados: starting objecter 2024-09-16T04:03:39.654 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.631+0000 7f55c30aa640 1 Processor -- start 2024-09-16T04:03:39.654 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.631+0000 7f55c30aa640 1 -- start start 2024-09-16T04:03:39.654 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.631+0000 7f55c30aa640 1 librados: setting wanted keys 2024-09-16T04:03:39.654 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.631+0000 7f55c30aa640 1 librados: calling monclient init 2024-09-16T04:03:39.654 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.631+0000 7f55c30aa640 10 monclient: init 2024-09-16T04:03:39.654 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.631+0000 7f55c30aa640 10 monclient: _reopen_session rank -1 2024-09-16T04:03:39.655 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.631+0000 7f55c30aa640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:03:39.655 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.631+0000 7f55c30aa640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f55a4004810 0x7f55bc125ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:03:39.655 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.631+0000 7f55c30aa640 10 monclient(hunting): picked mon.noname-b con 0x7f55a4004810 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:03:39.655 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.631+0000 7f55c30aa640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f55bc126410 0x7f55bc126890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:03:39.655 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.631+0000 7f55c30aa640 10 monclient(hunting): picked mon.noname-c con 0x7f55bc126410 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:03:39.655 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.631+0000 7f55c30aa640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f55bc126dd0 0x7f55bc1d9d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:03:39.655 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.631+0000 7f55c30aa640 10 monclient(hunting): picked mon.noname-a con 0x7f55bc126dd0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:03:39.655 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.631+0000 7f55c30aa640 10 monclient(hunting): start opening mon connection 2024-09-16T04:03:39.655 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.631+0000 7f55c30aa640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f55bc0a1cc0 con 0x7f55bc126dd0 2024-09-16T04:03:39.655 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.631+0000 7f55c30aa640 10 monclient(hunting): start opening mon connection 2024-09-16T04:03:39.655 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.631+0000 7f55c30aa640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f55bc0a1b40 con 0x7f55a4004810 2024-09-16T04:03:39.655 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.631+0000 7f55c30aa640 10 monclient(hunting): start opening mon connection 2024-09-16T04:03:39.655 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.631+0000 7f55c30aa640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f55bc0a1e40 con 0x7f55bc126410 2024-09-16T04:03:39.655 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.631+0000 7f55c30aa640 10 monclient(hunting): _renew_subs 2024-09-16T04:03:39.656 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.631+0000 7f55c30aa640 10 monclient(hunting): authenticate will time out at 2266.603133s 2024-09-16T04:03:39.656 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.631+0000 7f55c1647640 1 -- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f55bc126dd0 msgr2=0x7f55bc1d9d30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.42:3300/0 2024-09-16T04:03:39.656 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.631+0000 7f55c1647640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f55bc126dd0 0x7f55bc1d9d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:03:39.656 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.631+0000 7f55bbfff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f55bc126410 0x7f55bc126890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:03:39.656 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.631+0000 7f55bbfff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f55bc126410 0x7f55bc126890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:36862/0 (socket says 172.21.15.42:36862) 2024-09-16T04:03:39.656 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.631+0000 7f55bbfff640 1 -- 172.21.15.42:0/1422434323 learned_addr learned my addr 172.21.15.42:0/1422434323 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:03:39.656 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.631+0000 7f55bbfff640 10 monclient(hunting): get_auth_request con 0x7f55bc126410 auth_method 0 2024-09-16T04:03:39.656 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.631+0000 7f55bbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:03:39.656 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.631+0000 7f55bbfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:03:39.656 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.631+0000 7f55bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:03:39.656 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.631+0000 7f55bbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:03:39.656 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.631+0000 7f55bbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:03:39.656 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.631+0000 7f55bbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:03:39.656 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.631+0000 7f55c0e46640 1 --2- 172.21.15.42:0/1422434323 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f55a4004810 0x7f55bc125ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:03:39.657 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.631+0000 7f55bbfff640 10 monclient(hunting): handle_auth_done global_id 974159 payload 824 2024-09-16T04:03:39.657 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.631+0000 7f55bbfff640 1 -- 172.21.15.42:0/1422434323 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f55a4004810 msgr2=0x7f55bc125ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:03:39.657 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.631+0000 7f55bbfff640 1 --2- 172.21.15.42:0/1422434323 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f55a4004810 0x7f55bc125ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:03:39.657 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.631+0000 7f55bbfff640 1 -- 172.21.15.42:0/1422434323 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f55bc126dd0 msgr2=0x7f55bc1d9d30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:03:39.657 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.631+0000 7f55bbfff640 1 --2- 172.21.15.42:0/1422434323 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f55bc126dd0 0x7f55bc1d9d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:03:39.657 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.631+0000 7f55bbfff640 10 monclient: _finish_hunting 0 2024-09-16T04:03:39.657 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.631+0000 7f55bbfff640 1 monclient: found mon.noname-c 2024-09-16T04:03:39.657 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.631+0000 7f55bbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:03:39.657 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.631+0000 7f55bbfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T04:03:39.657 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.631+0000 7f55bbfff640 1 -- 172.21.15.42:0/1422434323 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55bc1281f0 con 0x7f55bc126410 2024-09-16T04:03:39.657 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.631+0000 7f55c0e46640 10 monclient: get_auth_request con 0x7f55a4004810 auth_method 0 2024-09-16T04:03:39.657 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.631+0000 7f55c0e46640 1 --2- 172.21.15.42:0/1422434323 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f55a4004810 0x7f55bc125ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T04:03:39.657 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.631+0000 7f55bbfff640 1 --2- 172.21.15.42:0/1422434323 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f55bc126410 0x7f55bc126890 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f55b000a7b0 tx=0x7f55a4005840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:03:39.658 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.631+0000 7f55b97fa640 10 client.?.objecter ms_handle_connect 0x7f55bc126410 2024-09-16T04:03:39.658 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.631+0000 7f55b97fa640 10 client.?.objecter resend_mon_ops 2024-09-16T04:03:39.658 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.631+0000 7f55b97fa640 1 -- 172.21.15.42:0/1422434323 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55b0001800 con 0x7f55bc126410 2024-09-16T04:03:39.658 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.631+0000 7f55b97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:03:39.658 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.631+0000 7f55b97fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:03:39.658 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.631+0000 7f55b97fa640 10 monclient: dump: 2024-09-16T04:03:39.658 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:03:39.658 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:03:39.658 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:39.658 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:39.658 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:03:39.658 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:03:39.658 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:03:39.659 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:03:39.659 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:03:39.659 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:03:39.659 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.631+0000 7f55b97fa640 10 monclient: _finish_auth 0 2024-09-16T04:03:39.659 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.631+0000 7f55b97fa640 10 monclient: _check_auth_tickets 2024-09-16T04:03:39.659 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.631+0000 7f55b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:03:39.659 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.631+0000 7f55c30aa640 5 monclient: authenticate success, global_id 974159 2024-09-16T04:03:39.659 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.631+0000 7f55b97fa640 1 -- 172.21.15.42:0/1422434323 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f55b00bb070 con 0x7f55bc126410 2024-09-16T04:03:39.659 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.631+0000 7f55b97fa640 10 monclient: handle_config config(0 keys) 2024-09-16T04:03:39.659 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.631+0000 7f55c30aa640 10 monclient: _renew_subs 2024-09-16T04:03:39.659 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.631+0000 7f55b97fa640 1 -- 172.21.15.42:0/1422434323 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55b00ad7e0 con 0x7f55bc126410 2024-09-16T04:03:39.659 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.631+0000 7f55c30aa640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:03:39.660 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.631+0000 7f55c30aa640 1 -- 172.21.15.42:0/1422434323 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55bc129c00 con 0x7f55bc126410 2024-09-16T04:03:39.660 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.631+0000 7f55baffd640 4 set_mon_vals no callback set 2024-09-16T04:03:39.660 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.635+0000 7f55b97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:03:39.660 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.635+0000 7f55b97fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:03:39.660 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.635+0000 7f55b97fa640 10 monclient: dump: 2024-09-16T04:03:39.660 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:03:39.660 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:03:39.660 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:39.660 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:39.660 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:03:39.660 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:03:39.660 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:03:39.660 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:03:39.661 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:03:39.661 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:03:39.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.635+0000 7f55c30aa640 10 client.974159.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:03:39.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.635+0000 7f55c30aa640 10 monclient: _renew_subs 2024-09-16T04:03:39.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.635+0000 7f55c30aa640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:03:39.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.635+0000 7f55c30aa640 1 -- 172.21.15.42:0/1422434323 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f55bc0a4040 con 0x7f55bc126410 2024-09-16T04:03:39.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.635+0000 7f55c30aa640 1 librados: init done 2024-09-16T04:03:39.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.635+0000 7f55c30aa640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:03:39.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.635+0000 7f55c30aa640 1 --2- 172.21.15.42:0/1422434323 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f55bc129550 0x7f55bc06f910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:03:39.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.635+0000 7f55c30aa640 10 monclient(hunting): start opening mon connection 2024-09-16T04:03:39.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.635+0000 7f55c30aa640 1 -- 172.21.15.42:0/1422434323 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f55bc128690 con 0x7f55bc129550 2024-09-16T04:03:39.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.635+0000 7f55c0e46640 1 --2- 172.21.15.42:0/1422434323 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f55bc129550 0x7f55bc06f910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:03:39.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.635+0000 7f55c0e46640 10 monclient: get_auth_request con 0x7f55bc129550 auth_method 0 2024-09-16T04:03:39.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.635+0000 7f55c0e46640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:03:39.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.635+0000 7f55c0e46640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:03:39.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.635+0000 7f55c0e46640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:03:39.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.635+0000 7f55b97fa640 1 -- 172.21.15.42:0/1422434323 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f55b00b5070 con 0x7f55bc126410 2024-09-16T04:03:39.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.635+0000 7f55c0e46640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:03:39.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.635+0000 7f55c0e46640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:03:39.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.635+0000 7f55c0e46640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:03:39.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.635+0000 7f55c0e46640 10 monclient(hunting): handle_auth_done global_id 974162 payload 824 2024-09-16T04:03:39.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.635+0000 7f55b97fa640 1 --2- 172.21.15.42:0/1422434323 >> v2:172.21.15.154:6812/254845342 conn(0x7f55a80377c0 0x7f55a8039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:03:39.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.635+0000 7f55c0e46640 1 -- 172.21.15.42:0/1422434323 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f55bc073a30 con 0x7f55bc129550 2024-09-16T04:03:39.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.635+0000 7f55b97fa640 1 -- 172.21.15.42:0/1422434323 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(655..655 src has 451..655) ==== 6338+0+0 (secure 0 0 0) 0x7f55b00e3a00 con 0x7f55bc126410 2024-09-16T04:03:39.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.635+0000 7f55b97fa640 10 client.974159.objecter ms_dispatch 0x7f55bc000ba0 osd_map(655..655 src has 451..655) 2024-09-16T04:03:39.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.635+0000 7f55b97fa640 3 client.974159.objecter handle_osd_map got epochs [655,655] > 0 2024-09-16T04:03:39.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.635+0000 7f55b97fa640 3 client.974159.objecter handle_osd_map decoding full epoch 655 2024-09-16T04:03:39.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.635+0000 7f55b97fa640 20 client.974159.objecter dump_active .. 0 homeless 2024-09-16T04:03:39.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.635+0000 7f55c0e46640 1 --2- 172.21.15.42:0/1422434323 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f55bc129550 0x7f55bc06f910 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f55ac003990 tx=0x7f55ac009ac0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:03:39.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.635+0000 7f55b97fa640 10 client.974159.objecter ms_handle_connect 0x7f55bc129550 2024-09-16T04:03:39.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.635+0000 7f55b97fa640 10 client.974159.objecter resend_mon_ops 2024-09-16T04:03:39.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.635+0000 7f55b97fa640 1 -- 172.21.15.42:0/1422434323 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55ac018020 con 0x7f55bc129550 2024-09-16T04:03:39.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.635+0000 7f55b97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:03:39.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.635+0000 7f55b97fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:03:39.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.635+0000 7f55b97fa640 10 monclient: dump: 2024-09-16T04:03:39.663 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:03:39.663 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:03:39.663 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:39.663 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:39.663 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:03:39.663 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:03:39.664 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:03:39.664 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:03:39.664 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:03:39.664 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:03:39.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.635+0000 7f55c1647640 1 --2- 172.21.15.42:0/1422434323 >> v2:172.21.15.154:6812/254845342 conn(0x7f55a80377c0 0x7f55a8039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:03:39.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.635+0000 7f55c1647640 10 monclient: get_auth_request con 0x7f55a80377c0 auth_method 0 2024-09-16T04:03:39.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.635+0000 7f55b97fa640 1 -- 172.21.15.42:0/1422434323 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f55ac014820 con 0x7f55bc129550 2024-09-16T04:03:39.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.635+0000 7f55b97fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:03:39.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.635+0000 7f55b97fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:03:39.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.635+0000 7f55b97fa640 1 -- 172.21.15.42:0/1422434323 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f55bc129550 msgr2=0x7f55bc06f910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:03:39.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.635+0000 7f55b97fa640 1 --2- 172.21.15.42:0/1422434323 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f55bc129550 0x7f55bc06f910 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f55ac003990 tx=0x7f55ac009ac0 comp rx=0 tx=0).stop 2024-09-16T04:03:39.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.635+0000 7f55b97fa640 1 -- 172.21.15.42:0/1422434323 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f55bc129550 msgr2=0x7f55bc06f910 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:03:39.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.635+0000 7f55b97fa640 1 --2- 172.21.15.42:0/1422434323 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f55bc129550 0x7f55bc06f910 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:03:39.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.635+0000 7f55c1647640 1 --2- 172.21.15.42:0/1422434323 >> v2:172.21.15.154:6812/254845342 conn(0x7f55a80377c0 0x7f55a8039c80 secure :-1 s=READY pgs=3813 cs=0 l=1 rev1=1 crypto rx=0x7f55bc0a2340 tx=0x7f55b4006b40 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:03:39.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.635+0000 7f55b97fa640 10 client.974159.objecter ms_handle_connect 0x7f55a80377c0 2024-09-16T04:03:39.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.639+0000 7f55c30aa640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:03:39.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.639+0000 7f55c30aa640 1 --2- 172.21.15.42:0/1422434323 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f55bc06fe50 0x7f55bc070260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:03:39.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.639+0000 7f55c30aa640 10 monclient(hunting): start opening mon connection 2024-09-16T04:03:39.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.639+0000 7f55c30aa640 1 -- 172.21.15.42:0/1422434323 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f55bc0a20f0 con 0x7f55bc06fe50 2024-09-16T04:03:39.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.639+0000 7f55c0e46640 1 --2- 172.21.15.42:0/1422434323 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f55bc06fe50 0x7f55bc070260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:03:39.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.639+0000 7f55c0e46640 10 monclient: get_auth_request con 0x7f55bc06fe50 auth_method 0 2024-09-16T04:03:39.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.639+0000 7f55c0e46640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:03:39.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.639+0000 7f55c0e46640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:03:39.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.639+0000 7f55c0e46640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:03:39.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.639+0000 7f55c0e46640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:03:39.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.639+0000 7f55c0e46640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:03:39.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.639+0000 7f55c0e46640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:03:39.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.639+0000 7f55c0e46640 10 monclient(hunting): handle_auth_done global_id 974165 payload 824 2024-09-16T04:03:39.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.643+0000 7f55c0e46640 1 -- 172.21.15.42:0/1422434323 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f55bc067c70 con 0x7f55bc06fe50 2024-09-16T04:03:39.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.643+0000 7f55c0e46640 1 --2- 172.21.15.42:0/1422434323 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f55bc06fe50 0x7f55bc070260 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f55ac019000 tx=0x7f55ac005a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:03:39.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.643+0000 7f55b97fa640 10 client.974159.objecter ms_handle_connect 0x7f55bc06fe50 2024-09-16T04:03:39.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.643+0000 7f55b97fa640 10 client.974159.objecter resend_mon_ops 2024-09-16T04:03:39.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.643+0000 7f55b97fa640 1 -- 172.21.15.42:0/1422434323 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55ac018030 con 0x7f55bc06fe50 2024-09-16T04:03:39.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.643+0000 7f55b97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:03:39.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.643+0000 7f55b97fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:03:39.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.643+0000 7f55b97fa640 10 monclient: dump: 2024-09-16T04:03:39.667 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:03:39.667 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:03:39.667 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:39.667 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:39.667 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:03:39.667 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:03:39.667 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:03:39.667 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:03:39.667 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:03:39.667 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:03:39.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.643+0000 7f55b97fa640 1 -- 172.21.15.42:0/1422434323 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4701 (secure 0 0 0) 0x7f55ac01c6b0 con 0x7f55bc06fe50 2024-09-16T04:03:39.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.643+0000 7f55b97fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:03:39.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.643+0000 7f55b97fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:03:39.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.643+0000 7f55b97fa640 1 -- 172.21.15.42:0/1422434323 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f55bc06fe50 msgr2=0x7f55bc070260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:03:39.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.643+0000 7f55b97fa640 1 --2- 172.21.15.42:0/1422434323 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f55bc06fe50 0x7f55bc070260 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f55ac019000 tx=0x7f55ac005a20 comp rx=0 tx=0).stop 2024-09-16T04:03:39.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.643+0000 7f55b97fa640 1 -- 172.21.15.42:0/1422434323 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f55bc06fe50 msgr2=0x7f55bc070260 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:03:39.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.643+0000 7f55b97fa640 1 --2- 172.21.15.42:0/1422434323 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f55bc06fe50 0x7f55bc070260 secure :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f55ac019000 tx=0x7f55ac005a20 comp rx=0 tx=0).stop 2024-09-16T04:03:39.668 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T04:03:39.668 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:03:39.668 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:03:39.668 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T04:03:39.668 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 194, 2024-09-16T04:03:39.668 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 28143, 2024-09-16T04:03:39.669 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T04:03:39.669 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T04:03:39.669 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T04:03:39.669 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:03:39.669 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 0, 2024-09-16T04:03:39.669 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:03:39.669 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T04:03:39.669 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T04:03:39.669 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:03:39.669 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:03:39.669 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:03:39.669 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:03:39.669 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:03:39.670 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:03:39.670 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:03:39.670 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:03:39.670 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:03:39.670 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:03:39.670 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:03:39.670 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:03:39.670 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:03:39.670 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T04:03:39.670 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:03:39.670 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:03:39.670 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:03:39.670 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:03:39.671 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:03:39.671 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:03:39.671 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:03:39.671 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:03:39.671 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:03:39.671 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:03:39.671 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:03:39.671 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:03:39.671 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:03:39.671 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T04:03:39.671 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T04:03:39.671 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T04:03:39.671 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T04:03:39.671 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T04:03:39.672 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T04:03:39.672 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:03:39.672 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:03:39.672 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T04:03:39.672 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T04:03:39.672 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T04:03:39.672 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T04:03:39.672 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T04:03:39.672 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T04:03:39.672 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T04:03:39.672 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:03:39.672 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T04:03:39.672 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:03:39.672 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:03:39.672 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:03:39.673 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:03:39.673 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:03:39.673 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:03:39.673 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:03:39.673 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:03:39.673 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:03:39.673 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:03:39.673 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:03:39.673 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:03:39.673 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T04:03:39.673 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:03:39.673 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T04:03:39.673 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:03:39.674 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:03:39.674 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:03:39.674 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:03:39.674 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:03:39.674 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:03:39.674 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:03:39.674 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T04:03:39.674 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:03:39.674 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:03:39.674 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:03:39.674 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:03:39.674 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T04:03:39.674 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:03:39.674 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:03:39.675 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:03:39.675 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:03:39.675 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T04:03:39.675 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T04:03:39.675 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:03:39.675 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:03:39.675 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:03:39.675 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:03:39.675 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:03:39.675 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:03:39.675 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:03:39.675 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:03:39.675 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:03:39.676 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:03:39.676 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:03:39.676 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T04:03:39.676 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:03:39.676 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:03:39.676 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:03:39.676 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:03:39.676 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T04:03:39.676 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:03:39.676 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:03:39.676 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:03:39.676 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:03:39.676 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T04:03:39.676 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T04:03:39.677 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:03:39.677 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:03:39.677 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:03:39.677 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:03:39.677 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:03:39.677 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:03:39.677 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:03:39.677 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:03:39.677 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:03:39.677 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:03:39.677 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:03:39.677 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T04:03:39.677 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:03:39.677 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:03:39.678 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:03:39.678 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:03:39.678 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T04:03:39.678 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:03:39.678 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:03:39.678 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:03:39.678 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:03:39.678 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T04:03:39.678 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T04:03:39.678 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:03:39.678 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:03:39.678 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:03:39.678 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:03:39.678 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:03:39.679 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:03:39.679 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T04:03:39.679 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T04:03:39.679 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:03:39.679 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:03:39.679 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:03:39.679 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:03:39.679 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:03:39.679 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:03:39.679 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T04:03:39.679 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:03:39.679 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:03:39.679 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:03:39.680 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 5 2024-09-16T04:03:39.680 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:03:39.680 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:03:39.680 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T04:03:39.680 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:03:39.680 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:03:39.680 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:03:39.680 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 6 2024-09-16T04:03:39.680 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:03:39.680 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:03:39.680 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:03:39.680 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T04:03:39.680 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T04:03:39.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.643+0000 7f55c30aa640 10 librados: watch_flush enter 2024-09-16T04:03:39.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.643+0000 7f55c30aa640 10 librados: watch_flush exit 2024-09-16T04:03:39.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.643+0000 7f55c30aa640 20 client.974159.objecter shutdown clearing up homeless session... 2024-09-16T04:03:39.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.643+0000 7f55c30aa640 10 client.974159.objecter successfully canceled tick 2024-09-16T04:03:39.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.643+0000 7f55c30aa640 1 -- 172.21.15.42:0/1422434323 >> v2:172.21.15.154:6812/254845342 conn(0x7f55a80377c0 msgr2=0x7f55a8039c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:03:39.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.643+0000 7f55c30aa640 1 --2- 172.21.15.42:0/1422434323 >> v2:172.21.15.154:6812/254845342 conn(0x7f55a80377c0 0x7f55a8039c80 secure :-1 s=READY pgs=3813 cs=0 l=1 rev1=1 crypto rx=0x7f55bc0a2340 tx=0x7f55b4006b40 comp rx=0 tx=0).stop 2024-09-16T04:03:39.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.643+0000 7f55c30aa640 10 monclient: shutdown 2024-09-16T04:03:39.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.643+0000 7f55c30aa640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:03:39.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.643+0000 7f55c30aa640 1 -- 172.21.15.42:0/1422434323 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f55bc126410 msgr2=0x7f55bc126890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:03:39.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.643+0000 7f55c30aa640 1 --2- 172.21.15.42:0/1422434323 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f55bc126410 0x7f55bc126890 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f55b000a7b0 tx=0x7f55a4005840 comp rx=0 tx=0).stop 2024-09-16T04:03:39.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.643+0000 7f55c1647640 1 -- 172.21.15.42:0/1422434323 reap_dead start 2024-09-16T04:03:39.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.643+0000 7f55c1647640 1 -- 172.21.15.42:0/1422434323 reap_dead start 2024-09-16T04:03:39.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.643+0000 7f55c30aa640 1 -- 172.21.15.42:0/1422434323 shutdown_connections 2024-09-16T04:03:39.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.643+0000 7f55c30aa640 1 -- 172.21.15.42:0/1422434323 >> 172.21.15.42:0/1422434323 conn(0x7f55bc0a2420 msgr2=0x7f55bc09e7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:03:39.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.643+0000 7f55c30aa640 1 -- 172.21.15.42:0/1422434323 shutdown_connections 2024-09-16T04:03:39.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.643+0000 7f55c30aa640 1 -- 172.21.15.42:0/1422434323 wait complete. 2024-09-16T04:03:39.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.643+0000 7f55c30aa640 1 librados: shutdown 2024-09-16T04:03:39.684 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T04:03:39.684 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-16T04:03:39.684 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T04:03:39.798 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.795+0000 7f8baa666640 10 monclient: get_monmap_and_config 2024-09-16T04:03:39.799 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.795+0000 7f8baa666640 10 monclient: build_initial_monmap 2024-09-16T04:03:39.799 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.795+0000 7f8baa666640 10 monclient: monmap: 2024-09-16T04:03:39.799 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:03:39.799 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:03:39.799 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:03:39.797279+0000 2024-09-16T04:03:39.799 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:03:39.797279+0000 2024-09-16T04:03:39.799 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:03:39.799 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:03:39.799 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:03:39.799 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:03:39.799 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:03:39.799 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:03:39.799 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:03:39.800 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:03:39.802 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.795+0000 7f8baa666640 1 Processor -- start 2024-09-16T04:03:39.802 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.795+0000 7f8baa666640 1 -- start start 2024-09-16T04:03:39.802 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.795+0000 7f8baa666640 10 monclient: init 2024-09-16T04:03:39.802 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.795+0000 7f8baa666640 10 monclient: _reopen_session rank -1 2024-09-16T04:03:39.802 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.795+0000 7f8baa666640 10 monclient: _add_conns ranks=[0,2,4,3,1] 2024-09-16T04:03:39.802 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.795+0000 7f8baa666640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8ba40a1330 addr v1:172.21.15.42:6789/0 2024-09-16T04:03:39.802 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.795+0000 7f8baa666640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f8ba40a1710 0x7f8ba40a1b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:03:39.802 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.795+0000 7f8baa666640 10 monclient(hunting): picked mon.noname-b con 0x7f8ba40a1710 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:03:39.803 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.795+0000 7f8baa666640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f8ba41393a0 0x7f8ba413b790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:03:39.803 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.795+0000 7f8baa666640 10 monclient(hunting): picked mon.noname-c con 0x7f8ba41393a0 addr v2:172.21.15.154:3300/0 2024-09-16T04:03:39.803 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.795+0000 7f8ba3fff640 1 -- >> v1:172.21.15.42:6789/0 conn(0x7f8ba40a1330 legacy=0x7f8ba409f180 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.42:6789/0 2024-09-16T04:03:39.803 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.795+0000 7f8baa666640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8ba409e4c0 con 0x7f8ba40a1330 2024-09-16T04:03:39.803 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.795+0000 7f8baa666640 10 monclient(hunting): start opening mon connection 2024-09-16T04:03:39.803 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.795+0000 7f8baa666640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f8ba409edf0 con 0x7f8ba41393a0 2024-09-16T04:03:39.803 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.795+0000 7f8baa666640 10 monclient(hunting): start opening mon connection 2024-09-16T04:03:39.803 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.795+0000 7f8baa666640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f8ba409cd60 con 0x7f8ba40a1710 2024-09-16T04:03:39.803 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.795+0000 7f8baa666640 10 monclient(hunting): _renew_subs 2024-09-16T04:03:39.803 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.795+0000 7f8baa666640 10 monclient(hunting): authenticate will time out at 2266.766908s 2024-09-16T04:03:39.803 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.795+0000 7f8ba37fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f8ba40a1710 0x7f8ba40a1b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:03:39.803 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.795+0000 7f8ba37fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f8ba40a1710 0x7f8ba40a1b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:36890/0 (socket says 172.21.15.42:36890) 2024-09-16T04:03:39.803 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.795+0000 7f8ba37fe640 1 -- 172.21.15.42:0/2780992404 learned_addr learned my addr 172.21.15.42:0/2780992404 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:03:39.803 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.795+0000 7f8ba37fe640 10 monclient(hunting): get_auth_request con 0x7f8ba40a1710 auth_method 0 2024-09-16T04:03:39.804 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.795+0000 7f8ba37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:03:39.804 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.795+0000 7f8ba37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:03:39.804 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.795+0000 7f8ba37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:03:39.804 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.795+0000 7f8ba37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:03:39.804 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.795+0000 7f8ba37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:03:39.804 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.795+0000 7f8ba37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:03:39.804 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.795+0000 7f8ba8c03640 1 --2- 172.21.15.42:0/2780992404 >> v2:172.21.15.154:3300/0 conn(0x7f8ba41393a0 0x7f8ba413b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:03:39.804 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.795+0000 7f8ba37fe640 10 monclient(hunting): handle_auth_done global_id 974168 payload 274 2024-09-16T04:03:39.804 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.795+0000 7f8ba37fe640 1 -- 172.21.15.42:0/2780992404 >> v2:172.21.15.154:3300/0 conn(0x7f8ba41393a0 msgr2=0x7f8ba413b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:03:39.804 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.795+0000 7f8ba37fe640 1 --2- 172.21.15.42:0/2780992404 >> v2:172.21.15.154:3300/0 conn(0x7f8ba41393a0 0x7f8ba413b790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:03:39.804 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.795+0000 7f8ba37fe640 1 -- 172.21.15.42:0/2780992404 >> v1:172.21.15.42:6789/0 conn(0x7f8ba40a1330 legacy=0x7f8ba409f180 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:03:39.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.795+0000 7f8ba37fe640 10 monclient: _finish_hunting 0 2024-09-16T04:03:39.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.795+0000 7f8ba37fe640 1 monclient: found mon.noname-b 2024-09-16T04:03:39.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.795+0000 7f8ba37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:03:39.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.795+0000 7f8ba37fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T04:03:39.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.795+0000 7f8ba37fe640 1 -- 172.21.15.42:0/2780992404 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ba40a2050 con 0x7f8ba40a1710 2024-09-16T04:03:39.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.799+0000 7f8ba37fe640 1 --2- 172.21.15.42:0/2780992404 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f8ba40a1710 0x7f8ba40a1b10 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f8b98009870 tx=0x7f8b9801c3e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=246caf4e25203f35 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:03:39.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.799+0000 7f8ba2ffd640 1 -- 172.21.15.42:0/2780992404 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b980b0070 con 0x7f8ba40a1710 2024-09-16T04:03:39.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.799+0000 7f8ba2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:03:39.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.799+0000 7f8ba2ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:03:39.813 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.799+0000 7f8ba2ffd640 10 monclient: dump: 2024-09-16T04:03:39.813 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:03:39.813 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:03:39.813 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:39.813 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:39.813 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:03:39.813 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:03:39.813 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:03:39.813 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:03:39.813 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:03:39.814 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:03:39.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.799+0000 7f8ba2ffd640 10 monclient: _finish_auth 0 2024-09-16T04:03:39.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.799+0000 7f8ba2ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:03:39.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.799+0000 7f8ba2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:03:39.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.799+0000 7f8ba2ffd640 1 -- 172.21.15.42:0/2780992404 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8b980ab600 con 0x7f8ba40a1710 2024-09-16T04:03:39.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.799+0000 7f8ba2ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:03:39.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.799+0000 7f8ba2ffd640 1 -- 172.21.15.42:0/2780992404 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b980ab9d0 con 0x7f8ba40a1710 2024-09-16T04:03:39.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.799+0000 7f8ba2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:03:39.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.799+0000 7f8ba2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:03:39.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.799+0000 7f8ba2ffd640 10 monclient: dump: 2024-09-16T04:03:39.814 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:03:39.815 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:03:39.815 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:39.815 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:39.815 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:03:39.815 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:03:39.815 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:03:39.815 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:03:39.815 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:03:39.815 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:03:39.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.799+0000 7f8baa666640 5 monclient: authenticate success, global_id 974168 2024-09-16T04:03:39.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.799+0000 7f8baa666640 10 monclient: get_monmap_and_config success 2024-09-16T04:03:39.816 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.799+0000 7f8baa666640 4 set_mon_vals no callback set 2024-09-16T04:03:39.816 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.799+0000 7f8baa666640 10 monclient: shutdown 2024-09-16T04:03:39.816 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.799+0000 7f8baa666640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:03:39.816 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.799+0000 7f8baa666640 1 -- 172.21.15.42:0/2780992404 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f8ba40a1710 msgr2=0x7f8ba40a1b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:03:39.816 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.799+0000 7f8baa666640 1 --2- 172.21.15.42:0/2780992404 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f8ba40a1710 0x7f8ba40a1b10 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f8b98009870 tx=0x7f8b9801c3e0 comp rx=0 tx=0).stop 2024-09-16T04:03:39.816 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.799+0000 7f8baa666640 1 -- 172.21.15.42:0/2780992404 shutdown_connections 2024-09-16T04:03:39.816 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.799+0000 7f8baa666640 1 --2- 172.21.15.42:0/2780992404 >> v2:172.21.15.154:3300/0 conn(0x7f8ba41393a0 0x7f8ba413b790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:03:39.816 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.799+0000 7f8baa666640 1 --2- 172.21.15.42:0/2780992404 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f8ba40a1710 0x7f8ba40a1b10 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:03:39.816 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.799+0000 7f8baa666640 1 -- 172.21.15.42:0/2780992404 >> 172.21.15.42:0/2780992404 conn(0x7f8ba40a2420 msgr2=0x7f8ba40a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:03:39.816 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.799+0000 7f8baa666640 1 -- 172.21.15.42:0/2780992404 shutdown_connections 2024-09-16T04:03:39.817 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.799+0000 7f8baa666640 1 -- 172.21.15.42:0/2780992404 wait complete. 2024-09-16T04:03:39.817 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.799+0000 7f8baa666640 10 monclient: build_initial_monmap 2024-09-16T04:03:39.817 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.799+0000 7f8baa666640 10 monclient: monmap: 2024-09-16T04:03:39.817 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:03:39.817 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:03:39.817 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:03:39.817 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:03:39.817 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:03:39.817 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:03:39.817 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:03:39.817 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:03:39.817 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:03:39.817 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:03:39.818 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.799+0000 7f8baa666640 1 librados: starting msgr at 2024-09-16T04:03:39.818 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.799+0000 7f8baa666640 1 librados: starting objecter 2024-09-16T04:03:39.818 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.799+0000 7f8baa666640 1 Processor -- start 2024-09-16T04:03:39.818 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.799+0000 7f8baa666640 1 -- start start 2024-09-16T04:03:39.818 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.799+0000 7f8baa666640 1 librados: setting wanted keys 2024-09-16T04:03:39.818 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.799+0000 7f8baa666640 1 librados: calling monclient init 2024-09-16T04:03:39.818 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.799+0000 7f8baa666640 10 monclient: init 2024-09-16T04:03:39.818 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.803+0000 7f8baa666640 10 monclient: _reopen_session rank -1 2024-09-16T04:03:39.818 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.803+0000 7f8baa666640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:03:39.818 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.803+0000 7f8baa666640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8ba40a1330 0x7f8ba41ccc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:03:39.818 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.803+0000 7f8baa666640 10 monclient(hunting): picked mon.noname-a con 0x7f8ba40a1330 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:03:39.818 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.803+0000 7f8baa666640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f8ba40a1710 0x7f8ba4131600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:03:39.818 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.803+0000 7f8baa666640 10 monclient(hunting): picked mon.noname-c con 0x7f8ba40a1710 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:03:39.819 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.803+0000 7f8baa666640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8ba41393a0 0x7f8ba406f9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:03:39.819 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.803+0000 7f8ba3fff640 1 -- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8ba40a1330 msgr2=0x7f8ba41ccc90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.42:3300/0 2024-09-16T04:03:39.819 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.803+0000 7f8baa666640 10 monclient(hunting): picked mon.noname-b con 0x7f8ba41393a0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:03:39.819 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.803+0000 7f8baa666640 10 monclient(hunting): start opening mon connection 2024-09-16T04:03:39.819 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.803+0000 7f8baa666640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f8ba413e880 con 0x7f8ba40a1330 2024-09-16T04:03:39.819 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.803+0000 7f8ba3fff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8ba40a1330 0x7f8ba41ccc90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:03:39.819 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.803+0000 7f8baa666640 10 monclient(hunting): start opening mon connection 2024-09-16T04:03:39.819 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.803+0000 7f8baa666640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f8ba413e700 con 0x7f8ba41393a0 2024-09-16T04:03:39.819 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.803+0000 7f8baa666640 10 monclient(hunting): start opening mon connection 2024-09-16T04:03:39.819 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.803+0000 7f8baa666640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f8ba413ea00 con 0x7f8ba40a1710 2024-09-16T04:03:39.819 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.803+0000 7f8baa666640 10 monclient(hunting): _renew_subs 2024-09-16T04:03:39.819 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.803+0000 7f8baa666640 10 monclient(hunting): authenticate will time out at 2266.772273s 2024-09-16T04:03:39.819 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.803+0000 7f8ba37fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f8ba40a1710 0x7f8ba4131600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:03:39.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.803+0000 7f8ba37fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f8ba40a1710 0x7f8ba4131600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:36892/0 (socket says 172.21.15.42:36892) 2024-09-16T04:03:39.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.803+0000 7f8ba37fe640 1 -- 172.21.15.42:0/814102774 learned_addr learned my addr 172.21.15.42:0/814102774 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:03:39.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.803+0000 7f8ba37fe640 10 monclient(hunting): get_auth_request con 0x7f8ba40a1710 auth_method 0 2024-09-16T04:03:39.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.803+0000 7f8ba37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:03:39.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.803+0000 7f8ba37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:03:39.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.803+0000 7f8ba37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:03:39.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.803+0000 7f8ba37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:03:39.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.803+0000 7f8ba37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:03:39.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.803+0000 7f8ba8c03640 1 --2- 172.21.15.42:0/814102774 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8ba41393a0 0x7f8ba406f9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:03:39.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.803+0000 7f8ba37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:03:39.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.803+0000 7f8ba8c03640 10 monclient(hunting): get_auth_request con 0x7f8ba41393a0 auth_method 0 2024-09-16T04:03:39.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.803+0000 7f8ba8c03640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:03:39.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.803+0000 7f8ba8c03640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:03:39.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.803+0000 7f8ba8c03640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:03:39.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.803+0000 7f8ba8c03640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:03:39.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.803+0000 7f8ba8c03640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:03:39.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.803+0000 7f8ba8c03640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:03:39.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.803+0000 7f8ba37fe640 10 monclient(hunting): handle_auth_done global_id 974171 payload 824 2024-09-16T04:03:39.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.803+0000 7f8ba37fe640 1 -- 172.21.15.42:0/814102774 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8ba41393a0 msgr2=0x7f8ba406f9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:03:39.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.803+0000 7f8ba37fe640 1 --2- 172.21.15.42:0/814102774 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8ba41393a0 0x7f8ba406f9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:03:39.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.803+0000 7f8ba37fe640 1 -- 172.21.15.42:0/814102774 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8ba40a1330 msgr2=0x7f8ba41ccc90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:03:39.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.803+0000 7f8ba37fe640 1 --2- 172.21.15.42:0/814102774 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8ba40a1330 0x7f8ba41ccc90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:03:39.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.803+0000 7f8ba37fe640 10 monclient: _finish_hunting 0 2024-09-16T04:03:39.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.803+0000 7f8ba37fe640 1 monclient: found mon.noname-c 2024-09-16T04:03:39.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.803+0000 7f8ba37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:03:39.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.803+0000 7f8ba37fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T04:03:39.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.803+0000 7f8ba37fe640 1 -- 172.21.15.42:0/814102774 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ba411ccc0 con 0x7f8ba40a1710 2024-09-16T04:03:39.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.803+0000 7f8ba37fe640 1 --2- 172.21.15.42:0/814102774 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f8ba40a1710 0x7f8ba4131600 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f8b980ae040 tx=0x7f8b9801ac20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:03:39.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.803+0000 7f8ba0ff9640 10 client.?.objecter ms_handle_connect 0x7f8ba40a1710 2024-09-16T04:03:39.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.803+0000 7f8ba0ff9640 10 client.?.objecter resend_mon_ops 2024-09-16T04:03:39.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.803+0000 7f8ba0ff9640 1 -- 172.21.15.42:0/814102774 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b980ba020 con 0x7f8ba40a1710 2024-09-16T04:03:39.823 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.803+0000 7f8ba0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:03:39.823 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.803+0000 7f8ba0ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:03:39.823 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.803+0000 7f8ba0ff9640 10 monclient: dump: 2024-09-16T04:03:39.823 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:03:39.823 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:03:39.823 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:39.823 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:39.823 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:03:39.823 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:03:39.823 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:03:39.824 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:03:39.824 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:03:39.824 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:03:39.824 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.803+0000 7f8ba0ff9640 10 monclient: _finish_auth 0 2024-09-16T04:03:39.824 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.803+0000 7f8ba0ff9640 10 monclient: _check_auth_tickets 2024-09-16T04:03:39.824 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.803+0000 7f8ba0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:03:39.825 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.803+0000 7f8ba0ff9640 1 -- 172.21.15.42:0/814102774 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8b980b6640 con 0x7f8ba40a1710 2024-09-16T04:03:39.825 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.803+0000 7f8ba0ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T04:03:39.825 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.803+0000 7f8ba0ff9640 1 -- 172.21.15.42:0/814102774 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b980b0040 con 0x7f8ba40a1710 2024-09-16T04:03:39.825 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.803+0000 7f8baa666640 5 monclient: authenticate success, global_id 974171 2024-09-16T04:03:39.825 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.803+0000 7f8baa666640 10 monclient: _renew_subs 2024-09-16T04:03:39.825 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.803+0000 7f8baa666640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:03:39.825 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.803+0000 7f8baa666640 1 -- 172.21.15.42:0/814102774 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ba411cfa0 con 0x7f8ba40a1710 2024-09-16T04:03:39.826 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.803+0000 7f8ba0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:03:39.826 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.803+0000 7f8ba0ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:03:39.826 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.803+0000 7f8ba0ff9640 10 monclient: dump: 2024-09-16T04:03:39.826 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:03:39.826 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:03:39.826 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:39.826 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:39.826 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:03:39.826 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:03:39.826 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:03:39.826 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:03:39.826 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:03:39.826 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:03:39.826 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.803+0000 7f8ba27fc640 4 set_mon_vals no callback set 2024-09-16T04:03:39.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.803+0000 7f8baa666640 10 client.974171.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:03:39.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.803+0000 7f8baa666640 10 monclient: _renew_subs 2024-09-16T04:03:39.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.803+0000 7f8baa666640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:03:39.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.803+0000 7f8baa666640 1 -- 172.21.15.42:0/814102774 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ba40a4010 con 0x7f8ba40a1710 2024-09-16T04:03:39.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.803+0000 7f8baa666640 1 librados: init done 2024-09-16T04:03:39.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.803+0000 7f8baa666640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:03:39.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.803+0000 7f8baa666640 1 --2- 172.21.15.42:0/814102774 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8ba406ff30 0x7f8ba4070310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:03:39.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.803+0000 7f8baa666640 10 monclient(hunting): start opening mon connection 2024-09-16T04:03:39.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.803+0000 7f8baa666640 1 -- 172.21.15.42:0/814102774 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f8ba4070850 con 0x7f8ba406ff30 2024-09-16T04:03:39.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.803+0000 7f8ba0ff9640 1 -- 172.21.15.42:0/814102774 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f8b980bd020 con 0x7f8ba40a1710 2024-09-16T04:03:39.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.803+0000 7f8ba0ff9640 1 --2- 172.21.15.42:0/814102774 >> v2:172.21.15.154:6812/254845342 conn(0x7f8b900377c0 0x7f8b90039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:03:39.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.803+0000 7f8ba0ff9640 1 -- 172.21.15.42:0/814102774 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(655..655 src has 451..655) ==== 6338+0+0 (secure 0 0 0) 0x7f8b980e34a0 con 0x7f8ba40a1710 2024-09-16T04:03:39.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.803+0000 7f8ba0ff9640 10 client.974171.objecter ms_dispatch 0x7f8ba4000ba0 osd_map(655..655 src has 451..655) 2024-09-16T04:03:39.828 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.803+0000 7f8ba0ff9640 3 client.974171.objecter handle_osd_map got epochs [655,655] > 0 2024-09-16T04:03:39.828 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.803+0000 7f8ba0ff9640 3 client.974171.objecter handle_osd_map decoding full epoch 655 2024-09-16T04:03:39.828 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.803+0000 7f8ba3fff640 1 --2- 172.21.15.42:0/814102774 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8ba406ff30 0x7f8ba4070310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:03:39.828 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.803+0000 7f8ba3fff640 10 monclient: get_auth_request con 0x7f8ba406ff30 auth_method 0 2024-09-16T04:03:39.828 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.803+0000 7f8ba3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:03:39.828 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.803+0000 7f8ba3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:03:39.828 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.803+0000 7f8ba3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:03:39.828 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.803+0000 7f8ba0ff9640 20 client.974171.objecter dump_active .. 0 homeless 2024-09-16T04:03:39.828 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.803+0000 7f8ba3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:03:39.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.803+0000 7f8ba3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:03:39.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.803+0000 7f8ba3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:03:39.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.803+0000 7f8ba8c03640 1 --2- 172.21.15.42:0/814102774 >> v2:172.21.15.154:6812/254845342 conn(0x7f8b900377c0 0x7f8b90039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:03:39.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.803+0000 7f8ba8c03640 10 monclient: get_auth_request con 0x7f8b900377c0 auth_method 0 2024-09-16T04:03:39.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.803+0000 7f8ba3fff640 10 monclient(hunting): handle_auth_done global_id 904765 payload 824 2024-09-16T04:03:39.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.803+0000 7f8ba3fff640 1 -- 172.21.15.42:0/814102774 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8ba4073a30 con 0x7f8ba406ff30 2024-09-16T04:03:39.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.803+0000 7f8ba0ff9640 10 client.974171.objecter ms_handle_connect 0x7f8ba406ff30 2024-09-16T04:03:39.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.803+0000 7f8ba3fff640 1 --2- 172.21.15.42:0/814102774 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8ba406ff30 0x7f8ba4070310 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f8b9400b450 tx=0x7f8b9400b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:03:39.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.803+0000 7f8ba0ff9640 10 client.974171.objecter resend_mon_ops 2024-09-16T04:03:39.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.803+0000 7f8ba0ff9640 10 client.974171.objecter ms_handle_connect 0x7f8b900377c0 2024-09-16T04:03:39.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.803+0000 7f8ba8c03640 1 --2- 172.21.15.42:0/814102774 >> v2:172.21.15.154:6812/254845342 conn(0x7f8b900377c0 0x7f8b90039c80 secure :-1 s=READY pgs=3814 cs=0 l=1 rev1=1 crypto rx=0x7f8ba409df70 tx=0x7f8b9c007710 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:03:39.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.803+0000 7f8ba0ff9640 1 -- 172.21.15.42:0/814102774 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b94003dd0 con 0x7f8ba406ff30 2024-09-16T04:03:39.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.803+0000 7f8ba0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:03:39.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.803+0000 7f8ba0ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:03:39.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.803+0000 7f8ba0ff9640 10 monclient: dump: 2024-09-16T04:03:39.830 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:03:39.830 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:03:39.830 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:39.830 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:39.830 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:03:39.830 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:03:39.830 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:03:39.830 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:03:39.830 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:03:39.831 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:03:39.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.807+0000 7f8ba0ff9640 1 -- 172.21.15.42:0/814102774 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f8b94015070 con 0x7f8ba406ff30 2024-09-16T04:03:39.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.807+0000 7f8ba0ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:03:39.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.807+0000 7f8ba0ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:03:39.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.807+0000 7f8ba0ff9640 1 -- 172.21.15.42:0/814102774 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8ba406ff30 msgr2=0x7f8ba4070310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:03:39.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.807+0000 7f8ba0ff9640 1 --2- 172.21.15.42:0/814102774 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8ba406ff30 0x7f8ba4070310 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f8b9400b450 tx=0x7f8b9400b910 comp rx=0 tx=0).stop 2024-09-16T04:03:39.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.807+0000 7f8ba0ff9640 1 -- 172.21.15.42:0/814102774 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8ba406ff30 msgr2=0x7f8ba4070310 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:03:39.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.807+0000 7f8ba0ff9640 1 --2- 172.21.15.42:0/814102774 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8ba406ff30 0x7f8ba4070310 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:03:39.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.811+0000 7f8baa666640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:03:39.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.811+0000 7f8baa666640 1 --2- 172.21.15.42:0/814102774 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8ba4136390 0x7f8ba41367a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:03:39.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.811+0000 7f8baa666640 10 monclient(hunting): start opening mon connection 2024-09-16T04:03:39.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.811+0000 7f8baa666640 1 -- 172.21.15.42:0/814102774 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f8ba41cdb10 con 0x7f8ba4136390 2024-09-16T04:03:39.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.811+0000 7f8ba3fff640 1 --2- 172.21.15.42:0/814102774 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8ba4136390 0x7f8ba41367a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:03:39.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.811+0000 7f8ba3fff640 10 monclient: get_auth_request con 0x7f8ba4136390 auth_method 0 2024-09-16T04:03:39.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.811+0000 7f8ba3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:03:39.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.811+0000 7f8ba3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:03:39.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.811+0000 7f8ba3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:03:39.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.811+0000 7f8ba3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:03:39.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.811+0000 7f8ba3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:03:39.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.811+0000 7f8ba3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:03:39.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.811+0000 7f8ba3fff640 10 monclient(hunting): handle_auth_done global_id 904768 payload 824 2024-09-16T04:03:39.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.811+0000 7f8ba3fff640 1 -- 172.21.15.42:0/814102774 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f8ba4072830 con 0x7f8ba4136390 2024-09-16T04:03:39.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.815+0000 7f8ba3fff640 1 --2- 172.21.15.42:0/814102774 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8ba4136390 0x7f8ba41367a0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f8b9400b450 tx=0x7f8b940059e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:03:39.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.815+0000 7f8ba0ff9640 10 client.974171.objecter ms_handle_connect 0x7f8ba4136390 2024-09-16T04:03:39.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.815+0000 7f8ba0ff9640 10 client.974171.objecter resend_mon_ops 2024-09-16T04:03:39.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.815+0000 7f8ba0ff9640 1 -- 172.21.15.42:0/814102774 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b94010480 con 0x7f8ba4136390 2024-09-16T04:03:39.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.815+0000 7f8ba0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:03:39.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.815+0000 7f8ba0ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:03:39.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.815+0000 7f8ba0ff9640 10 monclient: dump: 2024-09-16T04:03:39.835 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:03:39.835 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:03:39.835 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:39.835 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:03:39.835 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:03:39.835 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:03:39.835 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:03:39.835 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:03:39.835 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:03:39.835 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:03:39.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.815+0000 7f8ba0ff9640 1 -- 172.21.15.42:0/814102774 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4871 (secure 0 0 0) 0x7f8b94010480 con 0x7f8ba4136390 2024-09-16T04:03:39.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.815+0000 7f8ba0ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:03:39.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.815+0000 7f8ba0ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:03:39.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.815+0000 7f8ba0ff9640 1 -- 172.21.15.42:0/814102774 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8ba4136390 msgr2=0x7f8ba41367a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:03:39.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.815+0000 7f8ba0ff9640 1 --2- 172.21.15.42:0/814102774 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8ba4136390 0x7f8ba41367a0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f8b9400b450 tx=0x7f8b940059e0 comp rx=0 tx=0).stop 2024-09-16T04:03:39.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.815+0000 7f8ba0ff9640 1 -- 172.21.15.42:0/814102774 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8ba4136390 msgr2=0x7f8ba41367a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:03:39.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.815+0000 7f8ba0ff9640 1 --2- 172.21.15.42:0/814102774 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8ba4136390 0x7f8ba41367a0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:03:39.836 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T04:03:39.836 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:03:39.836 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:03:39.836 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "leader", 2024-09-16T04:03:39.836 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 194, 2024-09-16T04:03:39.836 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 347992, 2024-09-16T04:03:39.837 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T04:03:39.837 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T04:03:39.837 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T04:03:39.837 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:03:39.837 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 1, 2024-09-16T04:03:39.837 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:03:39.837 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T04:03:39.838 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T04:03:39.838 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:03:39.838 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:03:39.838 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:03:39.838 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:03:39.838 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:03:39.838 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:03:39.838 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:03:39.839 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:03:39.839 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:03:39.839 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:03:39.839 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:03:39.839 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:03:39.839 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:03:39.839 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T04:03:39.839 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:03:39.839 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:03:39.840 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:03:39.840 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:03:39.840 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:03:39.840 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:03:39.840 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:03:39.840 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:03:39.840 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:03:39.840 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:03:39.840 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:03:39.840 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:03:39.841 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:03:39.841 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T04:03:39.841 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T04:03:39.841 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T04:03:39.841 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T04:03:39.841 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T04:03:39.841 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T04:03:39.841 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:03:39.841 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:03:39.841 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T04:03:39.842 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T04:03:39.842 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T04:03:39.842 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T04:03:39.842 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T04:03:39.842 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T04:03:39.842 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T04:03:39.842 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:03:39.842 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T04:03:39.842 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:03:39.843 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:03:39.843 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:03:39.843 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:03:39.843 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:03:39.843 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:03:39.843 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:03:39.843 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:03:39.843 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:03:39.843 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:03:39.843 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:03:39.843 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:03:39.843 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T04:03:39.844 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:03:39.844 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T04:03:39.844 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:03:39.844 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:03:39.844 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:03:39.844 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:03:39.844 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:03:39.844 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:03:39.844 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:03:39.844 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T04:03:39.844 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:03:39.845 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:03:39.845 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:03:39.845 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:03:39.845 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T04:03:39.845 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:03:39.845 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:03:39.845 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:03:39.845 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:03:39.845 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T04:03:39.845 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T04:03:39.845 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:03:39.845 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:03:39.846 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:03:39.846 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:03:39.846 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:03:39.846 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:03:39.846 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:03:39.846 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:03:39.846 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:03:39.846 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:03:39.846 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:03:39.846 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T04:03:39.846 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:03:39.847 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:03:39.847 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:03:39.847 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:03:39.847 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T04:03:39.847 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:03:39.847 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:03:39.847 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:03:39.847 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:03:39.848 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T04:03:39.848 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T04:03:39.848 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:03:39.848 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:03:39.848 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:03:39.848 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:03:39.848 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:03:39.848 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:03:39.848 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:03:39.848 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:03:39.848 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:03:39.848 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:03:39.849 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:03:39.849 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T04:03:39.849 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:03:39.849 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:03:39.849 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:03:39.849 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:03:39.849 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T04:03:39.850 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:03:39.850 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:03:39.850 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:03:39.850 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:03:39.850 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T04:03:39.851 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T04:03:39.851 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:03:39.851 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:03:39.851 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:03:39.851 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:03:39.851 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:03:39.851 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:03:39.851 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T04:03:39.851 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T04:03:39.851 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:03:39.851 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:03:39.851 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:03:39.852 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:03:39.852 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:03:39.852 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:03:39.852 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T04:03:39.852 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:03:39.852 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:03:39.852 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:03:39.852 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:03:39.852 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:03:39.852 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:03:39.853 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T04:03:39.853 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:03:39.853 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:03:39.853 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:03:39.853 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 3 2024-09-16T04:03:39.853 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:03:39.853 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:03:39.853 INFO:teuthology.orchestra.run.smithi042.stdout: "mgr": [ 2024-09-16T04:03:39.853 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:03:39.853 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:03:39.853 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:03:39.853 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:03:39.853 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:03:39.854 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:03:39.854 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:03:39.854 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T04:03:39.854 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T04:03:39.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.815+0000 7f8baa666640 10 librados: watch_flush enter 2024-09-16T04:03:39.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.815+0000 7f8baa666640 10 librados: watch_flush exit 2024-09-16T04:03:39.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.815+0000 7f8baa666640 20 client.974171.objecter shutdown clearing up homeless session... 2024-09-16T04:03:39.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.815+0000 7f8baa666640 10 client.974171.objecter successfully canceled tick 2024-09-16T04:03:39.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.815+0000 7f8baa666640 1 -- 172.21.15.42:0/814102774 >> v2:172.21.15.154:6812/254845342 conn(0x7f8b900377c0 msgr2=0x7f8b90039c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:03:39.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.815+0000 7f8baa666640 1 --2- 172.21.15.42:0/814102774 >> v2:172.21.15.154:6812/254845342 conn(0x7f8b900377c0 0x7f8b90039c80 secure :-1 s=READY pgs=3814 cs=0 l=1 rev1=1 crypto rx=0x7f8ba409df70 tx=0x7f8b9c007710 comp rx=0 tx=0).stop 2024-09-16T04:03:39.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.815+0000 7f8baa666640 10 monclient: shutdown 2024-09-16T04:03:39.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.815+0000 7f8baa666640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:03:39.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.815+0000 7f8baa666640 1 -- 172.21.15.42:0/814102774 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f8ba40a1710 msgr2=0x7f8ba4131600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:03:39.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.815+0000 7f8baa666640 1 --2- 172.21.15.42:0/814102774 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f8ba40a1710 0x7f8ba4131600 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f8b980ae040 tx=0x7f8b9801ac20 comp rx=0 tx=0).stop 2024-09-16T04:03:39.857 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.815+0000 7f8ba8c03640 1 -- 172.21.15.42:0/814102774 reap_dead start 2024-09-16T04:03:39.857 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.815+0000 7f8baa666640 1 -- 172.21.15.42:0/814102774 shutdown_connections 2024-09-16T04:03:39.857 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.815+0000 7f8baa666640 1 --2- 172.21.15.42:0/814102774 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f8ba40a1710 0x7f8ba4131600 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:03:39.857 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.815+0000 7f8baa666640 1 -- 172.21.15.42:0/814102774 >> 172.21.15.42:0/814102774 conn(0x7f8ba40a2420 msgr2=0x7f8ba409ba50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:03:39.857 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.815+0000 7f8baa666640 1 -- 172.21.15.42:0/814102774 shutdown_connections 2024-09-16T04:03:39.857 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.815+0000 7f8baa666640 1 -- 172.21.15.42:0/814102774 wait complete. 2024-09-16T04:03:39.857 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:39.815+0000 7f8baa666640 1 librados: shutdown 2024-09-16T04:03:39.857 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T04:03:39.857 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-16T04:03:59.858 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-16T04:03:59.858 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-09-16T04:03:59.859 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-16T04:03:59.859 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-16T04:03:59.862 INFO:tasks.ceph.mon.a:Started 2024-09-16T04:03:59.862 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T04:03:59.862 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:03:59.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.990+0000 7fac1cf1f640 10 monclient: get_monmap_and_config 2024-09-16T04:03:59.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.990+0000 7fac1cf1f640 10 monclient: build_initial_monmap 2024-09-16T04:03:59.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.990+0000 7fac1cf1f640 10 monclient: monmap: 2024-09-16T04:03:59.995 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:03:59.995 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:03:59.995 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:03:59.993305+0000 2024-09-16T04:03:59.995 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:03:59.993305+0000 2024-09-16T04:03:59.995 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:03:59.995 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:03:59.996 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:03:59.996 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:03:59.996 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:03:59.996 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:03:59.996 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:03:59.996 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:03:59.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.990+0000 7fac1cf1f640 1 Processor -- start 2024-09-16T04:03:59.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.990+0000 7fac1cf1f640 1 -- start start 2024-09-16T04:03:59.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.990+0000 7fac1cf1f640 10 monclient: init 2024-09-16T04:03:59.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.990+0000 7fac1cf1f640 10 monclient: _reopen_session rank -1 2024-09-16T04:03:59.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.990+0000 7fac1cf1f640 10 monclient: _add_conns ranks=[2,1,4,0,3] 2024-09-16T04:03:59.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.990+0000 7fac1cf1f640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fac1806db90 0x7fac181340c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:03:59.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.990+0000 7fac1cf1f640 10 monclient(hunting): picked mon.noname-b con 0x7fac1806db90 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:03:59.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.990+0000 7fac1cf1f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fac180a37f0 addr v1:172.21.15.154:6789/0 2024-09-16T04:04:00.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.990+0000 7fac1cf1f640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fac180a3bd0 0x7fac18137520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:04:00.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.990+0000 7fac1cf1f640 10 monclient(hunting): picked mon.noname-c con 0x7fac180a3bd0 addr v2:172.21.15.154:3300/0 2024-09-16T04:04:00.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.990+0000 7fac1cf1f640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fac1806ece0 con 0x7fac180a37f0 2024-09-16T04:04:00.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.990+0000 7fac1cf1f640 10 monclient(hunting): start opening mon connection 2024-09-16T04:04:00.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.990+0000 7fac1cf1f640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fac1806df70 con 0x7fac180a3bd0 2024-09-16T04:04:00.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.990+0000 7fac1cf1f640 10 monclient(hunting): start opening mon connection 2024-09-16T04:04:00.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.990+0000 7fac1cf1f640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fac1809bf80 con 0x7fac1806db90 2024-09-16T04:04:00.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.990+0000 7fac1cf1f640 10 monclient(hunting): _renew_subs 2024-09-16T04:04:00.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.990+0000 7fac1cf1f640 10 monclient(hunting): authenticate will time out at 2286.962519s 2024-09-16T04:04:00.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.990+0000 7fac1659c640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fac1806db90 0x7fac181340c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:00.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.990+0000 7fac1659c640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fac1806db90 0x7fac181340c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:33258/0 (socket says 172.21.15.42:33258) 2024-09-16T04:04:00.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.990+0000 7fac1659c640 1 -- 172.21.15.42:0/2049615248 learned_addr learned my addr 172.21.15.42:0/2049615248 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:04:00.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.990+0000 7fac1659c640 10 monclient(hunting): get_auth_request con 0x7fac1806db90 auth_method 0 2024-09-16T04:04:00.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.990+0000 7fac1659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:04:00.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.990+0000 7fac1659c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:04:00.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.990+0000 7fac1659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:04:00.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.990+0000 7fac1659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:04:00.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.990+0000 7fac1659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:04:00.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.990+0000 7fac1659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:04:00.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.990+0000 7fac16d9d640 1 --2- 172.21.15.42:0/2049615248 >> v2:172.21.15.154:3300/0 conn(0x7fac180a3bd0 0x7fac18137520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:00.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.990+0000 7fac1659c640 10 monclient(hunting): handle_auth_done global_id 974174 payload 274 2024-09-16T04:04:00.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.990+0000 7fac1659c640 1 -- 172.21.15.42:0/2049615248 >> v2:172.21.15.154:3300/0 conn(0x7fac180a3bd0 msgr2=0x7fac18137520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:04:00.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.990+0000 7fac1659c640 1 --2- 172.21.15.42:0/2049615248 >> v2:172.21.15.154:3300/0 conn(0x7fac180a3bd0 0x7fac18137520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:00.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.990+0000 7fac1659c640 1 -- 172.21.15.42:0/2049615248 >> v1:172.21.15.154:6789/0 conn(0x7fac180a37f0 legacy=0x7fac18136e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:04:00.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.990+0000 7fac1659c640 10 monclient: _finish_hunting 0 2024-09-16T04:04:00.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.990+0000 7fac1659c640 1 monclient: found mon.noname-b 2024-09-16T04:04:00.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.990+0000 7fac1659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:04:00.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.990+0000 7fac1659c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T04:04:00.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.990+0000 7fac1659c640 1 -- 172.21.15.42:0/2049615248 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac18134780 con 0x7fac1806db90 2024-09-16T04:04:00.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.990+0000 7fac16d9d640 10 monclient: get_auth_request con 0x7fac180a3bd0 auth_method 0 2024-09-16T04:04:00.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.990+0000 7fac16d9d640 1 --2- 172.21.15.42:0/2049615248 >> v2:172.21.15.154:3300/0 conn(0x7fac180a3bd0 0x7fac18137520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T04:04:00.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.994+0000 7fac1659c640 1 --2- 172.21.15.42:0/2049615248 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fac1806db90 0x7fac181340c0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fac08009870 tx=0x7fac0801c3e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ea519db8ff5104f7 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:04:00.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.994+0000 7fac1559a640 1 -- 172.21.15.42:0/2049615248 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac080b0070 con 0x7fac1806db90 2024-09-16T04:04:00.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.994+0000 7fac1559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:04:00.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.994+0000 7fac1559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:04:00.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.994+0000 7fac1559a640 10 monclient: dump: 2024-09-16T04:04:00.003 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:04:00.003 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:04:00.003 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:00.003 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:00.003 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:04:00.003 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:04:00.003 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:04:00.003 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:04:00.003 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:04:00.004 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:04:00.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.994+0000 7fac1559a640 10 monclient: _finish_auth 0 2024-09-16T04:04:00.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.994+0000 7fac1559a640 10 monclient: _check_auth_tickets 2024-09-16T04:04:00.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.994+0000 7fac1559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:04:00.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.994+0000 7fac1559a640 1 -- 172.21.15.42:0/2049615248 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac080ab600 con 0x7fac1806db90 2024-09-16T04:04:00.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.994+0000 7fac1559a640 10 monclient: handle_config config(0 keys) 2024-09-16T04:04:00.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.994+0000 7fac1559a640 1 -- 172.21.15.42:0/2049615248 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac080ab9d0 con 0x7fac1806db90 2024-09-16T04:04:00.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.994+0000 7fac1cf1f640 5 monclient: authenticate success, global_id 974174 2024-09-16T04:04:00.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.994+0000 7fac1cf1f640 10 monclient: get_monmap_and_config success 2024-09-16T04:04:00.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.994+0000 7fac1cf1f640 4 set_mon_vals no callback set 2024-09-16T04:04:00.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.994+0000 7fac1559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:04:00.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.994+0000 7fac1559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:04:00.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.994+0000 7fac1559a640 10 monclient: dump: 2024-09-16T04:04:00.005 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:04:00.005 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:04:00.005 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:00.005 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:00.005 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:04:00.005 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:04:00.005 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:04:00.005 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:04:00.005 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:04:00.005 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:04:00.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.994+0000 7fac1cf1f640 10 monclient: shutdown 2024-09-16T04:04:00.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.994+0000 7fac1cf1f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:04:00.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.994+0000 7fac1cf1f640 1 -- 172.21.15.42:0/2049615248 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fac1806db90 msgr2=0x7fac181340c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:04:00.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.994+0000 7fac1cf1f640 1 --2- 172.21.15.42:0/2049615248 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fac1806db90 0x7fac181340c0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fac08009870 tx=0x7fac0801c3e0 comp rx=0 tx=0).stop 2024-09-16T04:04:00.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.994+0000 7fac1cf1f640 1 -- 172.21.15.42:0/2049615248 shutdown_connections 2024-09-16T04:04:00.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.994+0000 7fac1cf1f640 1 --2- 172.21.15.42:0/2049615248 >> v2:172.21.15.154:3300/0 conn(0x7fac180a3bd0 0x7fac18137520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:00.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.994+0000 7fac1cf1f640 1 --2- 172.21.15.42:0/2049615248 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fac1806db90 0x7fac181340c0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:00.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.994+0000 7fac1cf1f640 1 -- 172.21.15.42:0/2049615248 >> 172.21.15.42:0/2049615248 conn(0x7fac180a9fb0 msgr2=0x7fac180aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:04:00.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.994+0000 7fac1cf1f640 1 -- 172.21.15.42:0/2049615248 shutdown_connections 2024-09-16T04:04:00.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.994+0000 7fac1cf1f640 1 -- 172.21.15.42:0/2049615248 wait complete. 2024-09-16T04:04:00.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.994+0000 7fac1cf1f640 10 monclient: build_initial_monmap 2024-09-16T04:04:00.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.994+0000 7fac1cf1f640 10 monclient: monmap: 2024-09-16T04:04:00.012 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:04:00.012 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:04:00.012 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:04:00.012 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:04:00.012 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:04:00.012 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:04:00.012 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:04:00.012 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:04:00.012 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:04:00.013 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:04:00.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.994+0000 7fac1cf1f640 1 librados: starting msgr at 2024-09-16T04:04:00.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.994+0000 7fac1cf1f640 1 librados: starting objecter 2024-09-16T04:04:00.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.994+0000 7fac1cf1f640 1 Processor -- start 2024-09-16T04:04:00.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.994+0000 7fac1cf1f640 1 -- start start 2024-09-16T04:04:00.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.994+0000 7fac1cf1f640 1 librados: setting wanted keys 2024-09-16T04:04:00.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.994+0000 7fac1cf1f640 1 librados: calling monclient init 2024-09-16T04:04:00.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.994+0000 7fac1cf1f640 10 monclient: init 2024-09-16T04:04:00.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.994+0000 7fac1cf1f640 10 monclient: _reopen_session rank -1 2024-09-16T04:04:00.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.994+0000 7fac1cf1f640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:04:00.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.994+0000 7fac1cf1f640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fac1806db90 0x7fac181cee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:04:00.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.994+0000 7fac1cf1f640 10 monclient(hunting): picked mon.noname-a con 0x7fac1806db90 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:04:00.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.994+0000 7fac1cf1f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fac180a37f0 0x7fac180a2b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:04:00.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.994+0000 7fac1cf1f640 10 monclient(hunting): picked mon.noname-b con 0x7fac180a37f0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:04:00.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.994+0000 7fac1cf1f640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fac180a3bd0 0x7fac180a30b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:04:00.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.994+0000 7fac1cf1f640 10 monclient(hunting): picked mon.noname-c con 0x7fac180a3bd0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:04:00.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.994+0000 7fac1cf1f640 10 monclient(hunting): start opening mon connection 2024-09-16T04:04:00.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.994+0000 7fac1cf1f640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fac181419d0 con 0x7fac1806db90 2024-09-16T04:04:00.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.994+0000 7fac1cf1f640 10 monclient(hunting): start opening mon connection 2024-09-16T04:04:00.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.994+0000 7fac1cf1f640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fac180a4100 con 0x7fac180a37f0 2024-09-16T04:04:00.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.994+0000 7fac1cf1f640 10 monclient(hunting): start opening mon connection 2024-09-16T04:04:00.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.994+0000 7fac1cf1f640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fac18141b50 con 0x7fac180a3bd0 2024-09-16T04:04:00.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.994+0000 7fac1cf1f640 10 monclient(hunting): _renew_subs 2024-09-16T04:04:00.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.994+0000 7fac1cf1f640 10 monclient(hunting): authenticate will time out at 2286.967715s 2024-09-16T04:04:00.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.994+0000 7fac16d9d640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fac180a3bd0 0x7fac180a30b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:00.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.994+0000 7fac1659c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fac1806db90 0x7fac181cee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:00.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.994+0000 7fac16d9d640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fac180a3bd0 0x7fac180a30b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:33272/0 (socket says 172.21.15.42:33272) 2024-09-16T04:04:00.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.994+0000 7fac16d9d640 1 -- 172.21.15.42:0/3380304550 learned_addr learned my addr 172.21.15.42:0/3380304550 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:04:00.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.994+0000 7fac16d9d640 10 monclient(hunting): get_auth_request con 0x7fac180a3bd0 auth_method 0 2024-09-16T04:04:00.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.994+0000 7fac16d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:04:00.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.994+0000 7fac16d9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:04:00.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.994+0000 7fac16d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:04:00.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.994+0000 7fac1659c640 10 monclient(hunting): get_auth_request con 0x7fac1806db90 auth_method 0 2024-09-16T04:04:00.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.994+0000 7fac1659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:04:00.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.994+0000 7fac1659c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:04:00.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.994+0000 7fac1659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:04:00.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.994+0000 7fac16d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:04:00.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.994+0000 7fac16d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:04:00.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.994+0000 7fac16d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:04:00.016 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.994+0000 7fac1659c640 1 -- 172.21.15.42:0/3380304550 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fac1806db90 msgr2=0x7fac181cee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-16T04:04:00.016 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.998+0000 7fac1659c640 1 -- 172.21.15.42:0/3380304550 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fac1806db90 msgr2=0x7fac181cee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:04:00.016 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.998+0000 7fac1659c640 1 --2- 172.21.15.42:0/3380304550 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fac1806db90 0x7fac181cee00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:04:00.016 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.998+0000 7fac1659c640 1 --2- 172.21.15.42:0/3380304550 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fac1806db90 0x7fac181cee00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:04:00.016 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.998+0000 7fac15d9b640 1 --2- 172.21.15.42:0/3380304550 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fac180a37f0 0x7fac180a2b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:00.016 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.998+0000 7fac15d9b640 10 monclient(hunting): get_auth_request con 0x7fac180a37f0 auth_method 0 2024-09-16T04:04:00.016 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.998+0000 7fac15d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:04:00.016 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.998+0000 7fac15d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:04:00.016 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.998+0000 7fac15d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:04:00.016 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.998+0000 7fac16d9d640 10 monclient(hunting): handle_auth_done global_id 974177 payload 824 2024-09-16T04:04:00.016 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.998+0000 7fac16d9d640 1 -- 172.21.15.42:0/3380304550 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fac180a37f0 msgr2=0x7fac180a2b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:04:00.016 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.998+0000 7fac16d9d640 1 --2- 172.21.15.42:0/3380304550 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fac180a37f0 0x7fac180a2b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:00.016 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.998+0000 7fac16d9d640 1 -- 172.21.15.42:0/3380304550 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fac1806db90 msgr2=0x7fac181cee00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:04:00.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.998+0000 7fac16d9d640 1 --2- 172.21.15.42:0/3380304550 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fac1806db90 0x7fac181cee00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:00.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.998+0000 7fac16d9d640 10 monclient: _finish_hunting 0 2024-09-16T04:04:00.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.998+0000 7fac16d9d640 1 monclient: found mon.noname-c 2024-09-16T04:04:00.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.998+0000 7fac16d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:04:00.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.998+0000 7fac16d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T04:04:00.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.998+0000 7fac16d9d640 1 -- 172.21.15.42:0/3380304550 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac1806f6a0 con 0x7fac180a3bd0 2024-09-16T04:04:00.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.998+0000 7fac16d9d640 1 --2- 172.21.15.42:0/3380304550 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fac180a3bd0 0x7fac180a30b0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fac1000cef0 tx=0x7fac1000a710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:04:00.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.998+0000 7fac06ffd640 10 client.?.objecter ms_handle_connect 0x7fac180a3bd0 2024-09-16T04:04:00.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.998+0000 7fac06ffd640 10 client.?.objecter resend_mon_ops 2024-09-16T04:04:00.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.998+0000 7fac06ffd640 1 -- 172.21.15.42:0/3380304550 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac1001a070 con 0x7fac180a3bd0 2024-09-16T04:04:00.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.998+0000 7fac06ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:04:00.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.998+0000 7fac06ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:04:00.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.998+0000 7fac06ffd640 10 monclient: dump: 2024-09-16T04:04:00.017 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:04:00.017 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:04:00.018 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:00.018 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:00.018 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:04:00.018 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:04:00.018 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:04:00.018 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:04:00.018 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:04:00.018 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:04:00.018 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.998+0000 7fac06ffd640 10 monclient: _finish_auth 0 2024-09-16T04:04:00.018 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.998+0000 7fac06ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:04:00.018 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.998+0000 7fac06ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:04:00.018 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.998+0000 7fac1cf1f640 5 monclient: authenticate success, global_id 974177 2024-09-16T04:04:00.019 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.998+0000 7fac06ffd640 1 -- 172.21.15.42:0/3380304550 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac1000d860 con 0x7fac180a3bd0 2024-09-16T04:04:00.019 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.998+0000 7fac06ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:04:00.019 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.998+0000 7fac1cf1f640 10 monclient: _renew_subs 2024-09-16T04:04:00.019 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.998+0000 7fac06ffd640 1 -- 172.21.15.42:0/3380304550 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac1000dc50 con 0x7fac180a3bd0 2024-09-16T04:04:00.019 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.998+0000 7fac1cf1f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:04:00.019 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.998+0000 7fac1cf1f640 1 -- 172.21.15.42:0/3380304550 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac1813eb00 con 0x7fac180a3bd0 2024-09-16T04:04:00.019 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.998+0000 7fac06ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:04:00.019 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.998+0000 7fac06ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:04:00.019 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.998+0000 7fac06ffd640 10 monclient: dump: 2024-09-16T04:04:00.019 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:04:00.019 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:04:00.020 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:00.021 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:00.021 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:04:00.021 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:04:00.021 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:04:00.022 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:04:00.022 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:04:00.022 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:04:00.023 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.998+0000 7fac14d99640 4 set_mon_vals no callback set 2024-09-16T04:04:00.023 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.998+0000 7fac1cf1f640 10 client.974177.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:04:00.023 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.998+0000 7fac1cf1f640 10 monclient: _renew_subs 2024-09-16T04:04:00.023 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.998+0000 7fac1cf1f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:04:00.023 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.998+0000 7fac1cf1f640 1 -- 172.21.15.42:0/3380304550 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fac1809e330 con 0x7fac180a3bd0 2024-09-16T04:04:00.023 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.998+0000 7fac1cf1f640 1 librados: init done 2024-09-16T04:04:00.023 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.998+0000 7fac1cf1f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:04:00.023 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.998+0000 7fac1cf1f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:04:00.023 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.998+0000 7fac1cf1f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:04:00.023 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.998+0000 7fac1cf1f640 1 -- 172.21.15.42:0/3380304550 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fac1809df70 con 0x7fac180a3bd0 2024-09-16T04:04:00.023 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.998+0000 7fac06ffd640 1 -- 172.21.15.42:0/3380304550 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fac1001d070 con 0x7fac180a3bd0 2024-09-16T04:04:00.023 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.998+0000 7fac06ffd640 1 --2- 172.21.15.42:0/3380304550 >> v2:172.21.15.154:6812/254845342 conn(0x7fabf8037810 0x7fabf8039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:04:00.023 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.998+0000 7fac06ffd640 1 -- 172.21.15.42:0/3380304550 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(672..672 src has 451..672) ==== 6280+0+0 (secure 0 0 0) 0x7fac10049aa0 con 0x7fac180a3bd0 2024-09-16T04:04:00.023 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.998+0000 7fac06ffd640 10 client.974177.objecter ms_dispatch 0x7fac18000ba0 osd_map(672..672 src has 451..672) 2024-09-16T04:04:00.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.998+0000 7fac06ffd640 3 client.974177.objecter handle_osd_map got epochs [672,672] > 0 2024-09-16T04:04:00.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.998+0000 7fac06ffd640 3 client.974177.objecter handle_osd_map decoding full epoch 672 2024-09-16T04:04:00.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.998+0000 7fac06ffd640 20 client.974177.objecter dump_active .. 0 homeless 2024-09-16T04:04:00.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.998+0000 7fac1659c640 1 --2- 172.21.15.42:0/3380304550 >> v2:172.21.15.154:6812/254845342 conn(0x7fabf8037810 0x7fabf8039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:00.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.998+0000 7fac1659c640 10 monclient: get_auth_request con 0x7fabf8037810 auth_method 0 2024-09-16T04:04:00.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.998+0000 7fac06ffd640 10 client.974177.objecter ms_handle_connect 0x7fabf8037810 2024-09-16T04:04:00.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:03:59.998+0000 7fac1659c640 1 --2- 172.21.15.42:0/3380304550 >> v2:172.21.15.154:6812/254845342 conn(0x7fabf8037810 0x7fabf8039cd0 secure :-1 s=READY pgs=3816 cs=0 l=1 rev1=1 crypto rx=0x7fac080ad000 tx=0x7fac080a6000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:04:00.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:00.002+0000 7fac06ffd640 1 -- 172.21.15.42:0/3380304550 <== mon.2 v2:172.21.15.42:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7fac1809df70 con 0x7fac180a3bd0 2024-09-16T04:04:00.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:00.002+0000 7fac06ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:04:00.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:00.002+0000 7fac06ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:04:00.233 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:00.230+0000 7fac1cf1f640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:04:00.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:00.230+0000 7fac1cf1f640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:04:00.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:00.230+0000 7fac1cf1f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:04:00.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:00.230+0000 7fac1cf1f640 1 -- 172.21.15.42:0/3380304550 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fac1809cc00 con 0x7fac180a3bd0 2024-09-16T04:04:00.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:00.230+0000 7fac06ffd640 1 -- 172.21.15.42:0/3380304550 <== mon.2 v2:172.21.15.42:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7fac1001e640 con 0x7fac180a3bd0 2024-09-16T04:04:00.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:00.230+0000 7fac06ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:04:00.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:00.230+0000 7fac06ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:04:00.234 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":194,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":21,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:04:00.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:00.230+0000 7fac1cf1f640 10 librados: watch_flush enter 2024-09-16T04:04:00.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:00.230+0000 7fac1cf1f640 10 librados: watch_flush exit 2024-09-16T04:04:00.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:00.230+0000 7fac1cf1f640 20 client.974177.objecter shutdown clearing up homeless session... 2024-09-16T04:04:00.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:00.230+0000 7fac1cf1f640 10 client.974177.objecter successfully canceled tick 2024-09-16T04:04:00.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:00.230+0000 7fac1cf1f640 1 -- 172.21.15.42:0/3380304550 >> v2:172.21.15.154:6812/254845342 conn(0x7fabf8037810 msgr2=0x7fabf8039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:04:00.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:00.230+0000 7fac1cf1f640 1 --2- 172.21.15.42:0/3380304550 >> v2:172.21.15.154:6812/254845342 conn(0x7fabf8037810 0x7fabf8039cd0 secure :-1 s=READY pgs=3816 cs=0 l=1 rev1=1 crypto rx=0x7fac080ad000 tx=0x7fac080a6000 comp rx=0 tx=0).stop 2024-09-16T04:04:00.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:00.230+0000 7fac1cf1f640 10 monclient: shutdown 2024-09-16T04:04:00.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:00.230+0000 7fac1cf1f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:04:00.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:00.230+0000 7fac1cf1f640 1 -- 172.21.15.42:0/3380304550 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fac180a3bd0 msgr2=0x7fac180a30b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:04:00.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:00.230+0000 7fac1cf1f640 1 --2- 172.21.15.42:0/3380304550 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fac180a3bd0 0x7fac180a30b0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fac1000cef0 tx=0x7fac1000a710 comp rx=0 tx=0).stop 2024-09-16T04:04:00.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:00.230+0000 7fac1cf1f640 1 -- 172.21.15.42:0/3380304550 shutdown_connections 2024-09-16T04:04:00.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:00.230+0000 7fac1cf1f640 1 --2- 172.21.15.42:0/3380304550 >> v2:172.21.15.154:6812/254845342 conn(0x7fabf8037810 0x7fabf8039cd0 unknown :-1 s=CLOSED pgs=3816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:00.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:00.230+0000 7fac1cf1f640 1 --2- 172.21.15.42:0/3380304550 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fac180a3bd0 0x7fac180a30b0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:00.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:00.230+0000 7fac1cf1f640 1 --2- 172.21.15.42:0/3380304550 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fac180a37f0 0x7fac180a2b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:00.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:00.230+0000 7fac1cf1f640 1 --2- 172.21.15.42:0/3380304550 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fac1806db90 0x7fac181cee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:00.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:00.230+0000 7fac1cf1f640 1 -- 172.21.15.42:0/3380304550 >> 172.21.15.42:0/3380304550 conn(0x7fac180a9fb0 msgr2=0x7fac18136b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:04:00.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:00.230+0000 7fac1cf1f640 1 -- 172.21.15.42:0/3380304550 shutdown_connections 2024-09-16T04:04:00.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:00.230+0000 7fac1cf1f640 1 -- 172.21.15.42:0/3380304550 wait complete. 2024-09-16T04:04:00.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:00.230+0000 7fac1cf1f640 1 librados: shutdown 2024-09-16T04:04:03.254 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:04:03.388 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.382+0000 7fae6851e640 10 monclient: get_monmap_and_config 2024-09-16T04:04:03.388 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.382+0000 7fae6851e640 10 monclient: build_initial_monmap 2024-09-16T04:04:03.388 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.382+0000 7fae6851e640 10 monclient: monmap: 2024-09-16T04:04:03.388 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:04:03.388 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:04:03.388 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:04:03.386442+0000 2024-09-16T04:04:03.388 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:04:03.386442+0000 2024-09-16T04:04:03.388 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:04:03.388 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:04:03.388 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:04:03.388 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:04:03.389 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:04:03.389 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:04:03.389 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:04:03.389 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:04:03.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.382+0000 7fae6851e640 1 Processor -- start 2024-09-16T04:04:03.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.382+0000 7fae6851e640 1 -- start start 2024-09-16T04:04:03.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.382+0000 7fae6851e640 10 monclient: init 2024-09-16T04:04:03.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.386+0000 7fae6851e640 10 monclient: _reopen_session rank -1 2024-09-16T04:04:03.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.386+0000 7fae6851e640 10 monclient: _add_conns ranks=[3,2,0,4,1] 2024-09-16T04:04:03.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.386+0000 7fae6851e640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fae6006db90 0x7fae601340c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:04:03.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.386+0000 7fae6851e640 10 monclient(hunting): picked mon.noname-a con 0x7fae6006db90 addr v2:172.21.15.42:3300/0 2024-09-16T04:04:03.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.386+0000 7fae6851e640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fae600a37f0 0x7fae600a3c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:04:03.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.386+0000 7fae6851e640 10 monclient(hunting): picked mon.noname-b con 0x7fae600a37f0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:04:03.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.386+0000 7fae6851e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fae60136e10 addr v1:172.21.15.42:6789/0 2024-09-16T04:04:03.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.386+0000 7fae6851e640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fae6006ece0 con 0x7fae60136e10 2024-09-16T04:04:03.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.386+0000 7fae6851e640 10 monclient(hunting): start opening mon connection 2024-09-16T04:04:03.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.386+0000 7fae6851e640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7fae6006df70 con 0x7fae6006db90 2024-09-16T04:04:03.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.386+0000 7fae6851e640 10 monclient(hunting): start opening mon connection 2024-09-16T04:04:03.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.386+0000 7fae6851e640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fae6009bf80 con 0x7fae600a37f0 2024-09-16T04:04:03.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.386+0000 7fae6851e640 10 monclient(hunting): _renew_subs 2024-09-16T04:04:03.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.386+0000 7fae6851e640 10 monclient(hunting): authenticate will time out at 2290.356687s 2024-09-16T04:04:03.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.386+0000 7fae662ba640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fae6006db90 0x7fae601340c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:03.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.386+0000 7fae66abb640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fae60136e10 0x7fae60137240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:34986/0 (socket says 172.21.15.42:34986) 2024-09-16T04:04:03.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.386+0000 7fae65ab9640 1 --2- 172.21.15.42:0/1259476989 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fae600a37f0 0x7fae600a3c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:03.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.386+0000 7fae66abb640 1 -- 172.21.15.42:0/1259476989 learned_addr learned my addr 172.21.15.42:0/1259476989 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T04:04:03.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.386+0000 7fae662ba640 10 monclient(hunting): get_auth_request con 0x7fae6006db90 auth_method 0 2024-09-16T04:04:03.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.386+0000 7fae662ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:04:03.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.386+0000 7fae662ba640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:04:03.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.386+0000 7fae662ba640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:04:03.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.386+0000 7fae65ab9640 10 monclient(hunting): get_auth_request con 0x7fae600a37f0 auth_method 0 2024-09-16T04:04:03.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.386+0000 7fae65ab9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:04:03.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.386+0000 7fae65ab9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:04:03.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.386+0000 7fae65ab9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:04:03.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.386+0000 7fae662ba640 1 -- 172.21.15.42:0/1259476989 >> v2:172.21.15.42:3300/0 conn(0x7fae6006db90 msgr2=0x7fae601340c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:04:03.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.386+0000 7fae662ba640 1 -- 172.21.15.42:0/1259476989 >> v2:172.21.15.42:3300/0 conn(0x7fae6006db90 msgr2=0x7fae601340c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:04:03.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.386+0000 7fae662ba640 1 --2- 172.21.15.42:0/1259476989 >> v2:172.21.15.42:3300/0 conn(0x7fae6006db90 0x7fae601340c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:04:03.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.386+0000 7fae65ab9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:04:03.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.386+0000 7fae65ab9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:04:03.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.386+0000 7fae65ab9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:04:03.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.386+0000 7fae662ba640 1 --2- 172.21.15.42:0/1259476989 >> v2:172.21.15.42:3300/0 conn(0x7fae6006db90 0x7fae601340c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:04:03.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.386+0000 7fae65ab9640 10 monclient(hunting): handle_auth_done global_id 974180 payload 274 2024-09-16T04:04:03.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.386+0000 7fae65ab9640 1 -- 172.21.15.42:0/1259476989 >> v2:172.21.15.42:3300/0 conn(0x7fae6006db90 msgr2=0x7fae601340c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:04:03.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.386+0000 7fae65ab9640 1 --2- 172.21.15.42:0/1259476989 >> v2:172.21.15.42:3300/0 conn(0x7fae6006db90 0x7fae601340c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:03.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.386+0000 7fae65ab9640 1 -- 172.21.15.42:0/1259476989 >> v1:172.21.15.42:6789/0 conn(0x7fae60136e10 legacy=0x7fae60137240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:04:03.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.386+0000 7fae65ab9640 10 monclient: _finish_hunting 0 2024-09-16T04:04:03.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.386+0000 7fae65ab9640 1 monclient: found mon.noname-b 2024-09-16T04:04:03.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.386+0000 7fae65ab9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:04:03.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.386+0000 7fae65ab9640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T04:04:03.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.386+0000 7fae65ab9640 1 -- 172.21.15.42:0/1259476989 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae60134780 con 0x7fae600a37f0 2024-09-16T04:04:03.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.386+0000 7fae65ab9640 1 --2- 172.21.15.42:0/1259476989 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fae600a37f0 0x7fae600a3c00 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fae5c0097c0 tx=0x7fae5c01c360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a019fa401293c7ea server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:04:03.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.386+0000 7fae652b8640 1 -- 172.21.15.42:0/1259476989 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae5c0b0070 con 0x7fae600a37f0 2024-09-16T04:04:03.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.386+0000 7fae652b8640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:04:03.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.386+0000 7fae652b8640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:04:03.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.386+0000 7fae652b8640 10 monclient: dump: 2024-09-16T04:04:03.402 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:04:03.402 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:04:03.402 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:03.402 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:03.402 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:04:03.403 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:04:03.403 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:04:03.403 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:04:03.403 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:04:03.403 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:04:03.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.386+0000 7fae652b8640 10 monclient: _finish_auth 0 2024-09-16T04:04:03.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.386+0000 7fae652b8640 10 monclient: _check_auth_tickets 2024-09-16T04:04:03.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.386+0000 7fae652b8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:04:03.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.386+0000 7fae652b8640 1 -- 172.21.15.42:0/1259476989 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fae5c0ab600 con 0x7fae600a37f0 2024-09-16T04:04:03.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.386+0000 7fae6851e640 5 monclient: authenticate success, global_id 974180 2024-09-16T04:04:03.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.386+0000 7fae6851e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:04:03.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.386+0000 7fae652b8640 10 monclient: handle_config config(0 keys) 2024-09-16T04:04:03.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.386+0000 7fae6851e640 10 monclient: get_monmap_and_config success 2024-09-16T04:04:03.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.386+0000 7fae652b8640 1 -- 172.21.15.42:0/1259476989 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae5c0ab9d0 con 0x7fae600a37f0 2024-09-16T04:04:03.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.386+0000 7fae6851e640 4 set_mon_vals no callback set 2024-09-16T04:04:03.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.386+0000 7fae652b8640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:04:03.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.386+0000 7fae652b8640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:04:03.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.386+0000 7fae652b8640 10 monclient: dump: 2024-09-16T04:04:03.404 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:04:03.404 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:04:03.404 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:03.404 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:03.404 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:04:03.404 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:04:03.404 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:04:03.404 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:04:03.405 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:04:03.405 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:04:03.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.390+0000 7fae6851e640 10 monclient: shutdown 2024-09-16T04:04:03.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.390+0000 7fae6851e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:04:03.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.390+0000 7fae6851e640 1 -- 172.21.15.42:0/1259476989 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fae600a37f0 msgr2=0x7fae600a3c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:04:03.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.390+0000 7fae6851e640 1 --2- 172.21.15.42:0/1259476989 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fae600a37f0 0x7fae600a3c00 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fae5c0097c0 tx=0x7fae5c01c360 comp rx=0 tx=0).stop 2024-09-16T04:04:03.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.390+0000 7fae6851e640 1 -- 172.21.15.42:0/1259476989 shutdown_connections 2024-09-16T04:04:03.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.390+0000 7fae6851e640 1 --2- 172.21.15.42:0/1259476989 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fae600a37f0 0x7fae600a3c00 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:03.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.390+0000 7fae6851e640 1 --2- 172.21.15.42:0/1259476989 >> v2:172.21.15.42:3300/0 conn(0x7fae6006db90 0x7fae601340c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:03.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.390+0000 7fae6851e640 1 -- 172.21.15.42:0/1259476989 >> 172.21.15.42:0/1259476989 conn(0x7fae600a9fb0 msgr2=0x7fae600aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:04:03.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.390+0000 7fae6851e640 1 -- 172.21.15.42:0/1259476989 shutdown_connections 2024-09-16T04:04:03.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.390+0000 7fae6851e640 1 -- 172.21.15.42:0/1259476989 wait complete. 2024-09-16T04:04:03.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.390+0000 7fae6851e640 10 monclient: build_initial_monmap 2024-09-16T04:04:03.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.390+0000 7fae6851e640 10 monclient: monmap: 2024-09-16T04:04:03.406 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:04:03.406 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:04:03.406 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:04:03.406 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:04:03.406 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:04:03.406 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:04:03.406 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:04:03.406 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:04:03.406 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:04:03.406 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:04:03.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.390+0000 7fae6851e640 1 librados: starting msgr at 2024-09-16T04:04:03.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.390+0000 7fae6851e640 1 librados: starting objecter 2024-09-16T04:04:03.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.390+0000 7fae6851e640 1 Processor -- start 2024-09-16T04:04:03.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.390+0000 7fae6851e640 1 -- start start 2024-09-16T04:04:03.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.390+0000 7fae6851e640 1 librados: setting wanted keys 2024-09-16T04:04:03.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.390+0000 7fae6851e640 1 librados: calling monclient init 2024-09-16T04:04:03.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.390+0000 7fae6851e640 10 monclient: init 2024-09-16T04:04:03.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.390+0000 7fae6851e640 10 monclient: _reopen_session rank -1 2024-09-16T04:04:03.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.390+0000 7fae6851e640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:04:03.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.390+0000 7fae6851e640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fae6006db90 0x7fae601cf3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:04:03.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.390+0000 7fae6851e640 10 monclient(hunting): picked mon.noname-a con 0x7fae6006db90 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:04:03.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.390+0000 7fae6851e640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fae600a37f0 0x7fae6013e200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:04:03.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.390+0000 7fae6851e640 10 monclient(hunting): picked mon.noname-c con 0x7fae600a37f0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:04:03.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.390+0000 7fae6851e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fae60136e10 0x7fae6013e740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:04:03.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.390+0000 7fae6851e640 10 monclient(hunting): picked mon.noname-b con 0x7fae60136e10 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:04:03.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.390+0000 7fae6851e640 10 monclient(hunting): start opening mon connection 2024-09-16T04:04:03.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.390+0000 7fae6851e640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fae60141e40 con 0x7fae6006db90 2024-09-16T04:04:03.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.390+0000 7fae6851e640 10 monclient(hunting): start opening mon connection 2024-09-16T04:04:03.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.390+0000 7fae6851e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fae60141cc0 con 0x7fae60136e10 2024-09-16T04:04:03.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.390+0000 7fae6851e640 10 monclient(hunting): start opening mon connection 2024-09-16T04:04:03.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.390+0000 7fae6851e640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fae60141fc0 con 0x7fae600a37f0 2024-09-16T04:04:03.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.390+0000 7fae6851e640 10 monclient(hunting): _renew_subs 2024-09-16T04:04:03.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.390+0000 7fae6851e640 10 monclient(hunting): authenticate will time out at 2290.362010s 2024-09-16T04:04:03.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.390+0000 7fae65ab9640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fae600a37f0 0x7fae6013e200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:03.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.390+0000 7fae662ba640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fae6006db90 0x7fae601cf3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:03.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.390+0000 7fae65ab9640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fae600a37f0 0x7fae6013e200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:33332/0 (socket says 172.21.15.42:33332) 2024-09-16T04:04:03.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.390+0000 7fae662ba640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fae6006db90 0x7fae601cf3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:46056/0 (socket says 172.21.15.42:46056) 2024-09-16T04:04:03.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.390+0000 7fae65ab9640 1 -- 172.21.15.42:0/3204262098 learned_addr learned my addr 172.21.15.42:0/3204262098 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:04:03.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.390+0000 7fae65ab9640 10 monclient(hunting): get_auth_request con 0x7fae600a37f0 auth_method 0 2024-09-16T04:04:03.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.390+0000 7fae65ab9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:04:03.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.390+0000 7fae65ab9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:04:03.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.390+0000 7fae65ab9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:04:03.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.390+0000 7fae662ba640 10 monclient(hunting): get_auth_request con 0x7fae6006db90 auth_method 0 2024-09-16T04:04:03.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.390+0000 7fae662ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:04:03.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.390+0000 7fae662ba640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:04:03.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.390+0000 7fae662ba640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:04:03.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.390+0000 7fae65ab9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:04:03.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.390+0000 7fae65ab9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:04:03.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.390+0000 7fae65ab9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:04:03.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.390+0000 7fae662ba640 1 -- 172.21.15.42:0/3204262098 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fae6006db90 msgr2=0x7fae601cf3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-16T04:04:03.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.390+0000 7fae662ba640 1 -- 172.21.15.42:0/3204262098 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fae6006db90 msgr2=0x7fae601cf3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:04:03.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.390+0000 7fae662ba640 1 --2- 172.21.15.42:0/3204262098 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fae6006db90 0x7fae601cf3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:04:03.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.390+0000 7fae662ba640 1 --2- 172.21.15.42:0/3204262098 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fae6006db90 0x7fae601cf3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:04:03.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.390+0000 7fae66abb640 1 --2- 172.21.15.42:0/3204262098 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fae60136e10 0x7fae6013e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:03.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.390+0000 7fae66abb640 10 monclient(hunting): get_auth_request con 0x7fae60136e10 auth_method 0 2024-09-16T04:04:03.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.390+0000 7fae66abb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:04:03.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.390+0000 7fae66abb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:04:03.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.390+0000 7fae66abb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:04:03.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.390+0000 7fae65ab9640 10 monclient(hunting): handle_auth_done global_id 974183 payload 824 2024-09-16T04:04:03.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.390+0000 7fae65ab9640 1 -- 172.21.15.42:0/3204262098 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fae60136e10 msgr2=0x7fae6013e740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:04:03.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.390+0000 7fae65ab9640 1 --2- 172.21.15.42:0/3204262098 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fae60136e10 0x7fae6013e740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:03.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.390+0000 7fae65ab9640 1 -- 172.21.15.42:0/3204262098 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fae6006db90 msgr2=0x7fae601cf3e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:04:03.411 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.390+0000 7fae65ab9640 1 --2- 172.21.15.42:0/3204262098 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fae6006db90 0x7fae601cf3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:03.411 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.390+0000 7fae65ab9640 10 monclient: _finish_hunting 0 2024-09-16T04:04:03.411 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.390+0000 7fae65ab9640 1 monclient: found mon.noname-c 2024-09-16T04:04:03.411 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.390+0000 7fae65ab9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:04:03.411 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.390+0000 7fae65ab9640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T04:04:03.411 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.390+0000 7fae65ab9640 1 -- 172.21.15.42:0/3204262098 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae6006f6a0 con 0x7fae600a37f0 2024-09-16T04:04:03.411 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.390+0000 7fae65ab9640 1 --2- 172.21.15.42:0/3204262098 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fae600a37f0 0x7fae6013e200 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fae5c0ae040 tx=0x7fae5c019100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:04:03.411 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.390+0000 7fae52ffd640 10 client.?.objecter ms_handle_connect 0x7fae600a37f0 2024-09-16T04:04:03.411 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.390+0000 7fae52ffd640 10 client.?.objecter resend_mon_ops 2024-09-16T04:04:03.411 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.390+0000 7fae52ffd640 1 -- 172.21.15.42:0/3204262098 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae5c0b8030 con 0x7fae600a37f0 2024-09-16T04:04:03.411 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.390+0000 7fae52ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:04:03.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.390+0000 7fae52ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:04:03.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.390+0000 7fae52ffd640 10 monclient: dump: 2024-09-16T04:04:03.412 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:04:03.412 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:04:03.412 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:03.412 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:03.412 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:04:03.412 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:04:03.412 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:04:03.412 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:04:03.412 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:04:03.412 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:04:03.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.390+0000 7fae52ffd640 10 monclient: _finish_auth 0 2024-09-16T04:04:03.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.390+0000 7fae52ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:04:03.413 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.390+0000 7fae52ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:04:03.413 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.390+0000 7fae6851e640 5 monclient: authenticate success, global_id 974183 2024-09-16T04:04:03.413 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.390+0000 7fae52ffd640 1 -- 172.21.15.42:0/3204262098 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fae5c0ab400 con 0x7fae600a37f0 2024-09-16T04:04:03.413 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.390+0000 7fae6851e640 10 monclient: _renew_subs 2024-09-16T04:04:03.413 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.390+0000 7fae6851e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:04:03.413 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.390+0000 7fae6851e640 1 -- 172.21.15.42:0/3204262098 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae60133540 con 0x7fae600a37f0 2024-09-16T04:04:03.413 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.390+0000 7fae52ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:04:03.413 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.390+0000 7fae52ffd640 1 -- 172.21.15.42:0/3204262098 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae5c0b0040 con 0x7fae600a37f0 2024-09-16T04:04:03.413 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.390+0000 7fae52ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:04:03.413 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.390+0000 7fae52ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:04:03.413 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.390+0000 7fae52ffd640 10 monclient: dump: 2024-09-16T04:04:03.413 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:04:03.413 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:04:03.414 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:03.414 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:03.414 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:04:03.414 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:04:03.414 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:04:03.414 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:04:03.414 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:04:03.414 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:04:03.414 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.390+0000 7fae6851e640 10 client.974183.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:04:03.414 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.390+0000 7fae6851e640 10 monclient: _renew_subs 2024-09-16T04:04:03.414 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.390+0000 7fae64ab7640 4 set_mon_vals no callback set 2024-09-16T04:04:03.414 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.390+0000 7fae6851e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:04:03.414 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.390+0000 7fae6851e640 1 -- 172.21.15.42:0/3204262098 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fae60139ca0 con 0x7fae600a37f0 2024-09-16T04:04:03.414 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.390+0000 7fae6851e640 1 librados: init done 2024-09-16T04:04:03.415 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.390+0000 7fae6851e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:04:03.415 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.390+0000 7fae6851e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:04:03.415 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.390+0000 7fae6851e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:04:03.415 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.390+0000 7fae6851e640 1 -- 172.21.15.42:0/3204262098 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fae601340c0 con 0x7fae600a37f0 2024-09-16T04:04:03.415 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.390+0000 7fae52ffd640 1 -- 172.21.15.42:0/3204262098 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fae5c0bb070 con 0x7fae600a37f0 2024-09-16T04:04:03.415 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.394+0000 7fae52ffd640 1 --2- 172.21.15.42:0/3204262098 >> v2:172.21.15.154:6812/254845342 conn(0x7fae44037810 0x7fae44039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:04:03.415 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.394+0000 7fae52ffd640 1 -- 172.21.15.42:0/3204262098 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(672..672 src has 451..672) ==== 6280+0+0 (secure 0 0 0) 0x7fae5c0e2a00 con 0x7fae600a37f0 2024-09-16T04:04:03.415 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.394+0000 7fae52ffd640 10 client.974183.objecter ms_dispatch 0x7fae60000ba0 osd_map(672..672 src has 451..672) 2024-09-16T04:04:03.415 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.394+0000 7fae52ffd640 3 client.974183.objecter handle_osd_map got epochs [672,672] > 0 2024-09-16T04:04:03.415 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.394+0000 7fae52ffd640 3 client.974183.objecter handle_osd_map decoding full epoch 672 2024-09-16T04:04:03.415 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.394+0000 7fae52ffd640 20 client.974183.objecter dump_active .. 0 homeless 2024-09-16T04:04:03.415 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.394+0000 7fae662ba640 1 --2- 172.21.15.42:0/3204262098 >> v2:172.21.15.154:6812/254845342 conn(0x7fae44037810 0x7fae44039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:03.415 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.394+0000 7fae662ba640 10 monclient: get_auth_request con 0x7fae44037810 auth_method 0 2024-09-16T04:04:03.415 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.394+0000 7fae662ba640 1 --2- 172.21.15.42:0/3204262098 >> v2:172.21.15.154:6812/254845342 conn(0x7fae44037810 0x7fae44039cd0 secure :-1 s=READY pgs=3817 cs=0 l=1 rev1=1 crypto rx=0x7fae54006aa0 tx=0x7fae54006a30 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:04:03.416 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.394+0000 7fae52ffd640 10 client.974183.objecter ms_handle_connect 0x7fae44037810 2024-09-16T04:04:03.416 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.398+0000 7fae52ffd640 1 -- 172.21.15.42:0/3204262098 <== mon.2 v2:172.21.15.42:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7fae5c0c2070 con 0x7fae600a37f0 2024-09-16T04:04:03.416 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.398+0000 7fae52ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:04:03.416 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.398+0000 7fae52ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:04:03.626 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.622+0000 7fae6851e640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:04:03.627 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.622+0000 7fae6851e640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:04:03.627 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.622+0000 7fae6851e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:04:03.627 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.622+0000 7fae6851e640 1 -- 172.21.15.42:0/3204262098 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fae60068e40 con 0x7fae600a37f0 2024-09-16T04:04:03.627 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.622+0000 7fae52ffd640 1 -- 172.21.15.42:0/3204262098 <== mon.2 v2:172.21.15.42:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7fae5c016070 con 0x7fae600a37f0 2024-09-16T04:04:03.627 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.622+0000 7fae52ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:04:03.627 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.622+0000 7fae52ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:04:03.628 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":194,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":24,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:04:03.629 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.622+0000 7fae6851e640 10 librados: watch_flush enter 2024-09-16T04:04:03.629 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.622+0000 7fae6851e640 10 librados: watch_flush exit 2024-09-16T04:04:03.629 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.622+0000 7fae6851e640 20 client.974183.objecter shutdown clearing up homeless session... 2024-09-16T04:04:03.630 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.622+0000 7fae6851e640 10 client.974183.objecter successfully canceled tick 2024-09-16T04:04:03.630 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.622+0000 7fae6851e640 1 -- 172.21.15.42:0/3204262098 >> v2:172.21.15.154:6812/254845342 conn(0x7fae44037810 msgr2=0x7fae44039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:04:03.630 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.622+0000 7fae6851e640 1 --2- 172.21.15.42:0/3204262098 >> v2:172.21.15.154:6812/254845342 conn(0x7fae44037810 0x7fae44039cd0 secure :-1 s=READY pgs=3817 cs=0 l=1 rev1=1 crypto rx=0x7fae54006aa0 tx=0x7fae54006a30 comp rx=0 tx=0).stop 2024-09-16T04:04:03.630 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.622+0000 7fae6851e640 10 monclient: shutdown 2024-09-16T04:04:03.630 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.622+0000 7fae6851e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:04:03.630 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.622+0000 7fae6851e640 1 -- 172.21.15.42:0/3204262098 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fae600a37f0 msgr2=0x7fae6013e200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:04:03.630 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.622+0000 7fae6851e640 1 --2- 172.21.15.42:0/3204262098 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fae600a37f0 0x7fae6013e200 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fae5c0ae040 tx=0x7fae5c019100 comp rx=0 tx=0).stop 2024-09-16T04:04:03.630 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.626+0000 7fae6851e640 1 -- 172.21.15.42:0/3204262098 shutdown_connections 2024-09-16T04:04:03.630 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.626+0000 7fae6851e640 1 --2- 172.21.15.42:0/3204262098 >> v2:172.21.15.154:6812/254845342 conn(0x7fae44037810 0x7fae44039cd0 unknown :-1 s=CLOSED pgs=3817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:03.630 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.626+0000 7fae6851e640 1 --2- 172.21.15.42:0/3204262098 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fae60136e10 0x7fae6013e740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:03.630 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.626+0000 7fae6851e640 1 --2- 172.21.15.42:0/3204262098 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fae600a37f0 0x7fae6013e200 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:03.630 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.626+0000 7fae6851e640 1 --2- 172.21.15.42:0/3204262098 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fae6006db90 0x7fae601cf3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:03.630 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.626+0000 7fae6851e640 1 -- 172.21.15.42:0/3204262098 >> 172.21.15.42:0/3204262098 conn(0x7fae600a9fb0 msgr2=0x7fae601327d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:04:03.631 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.626+0000 7fae6851e640 1 -- 172.21.15.42:0/3204262098 shutdown_connections 2024-09-16T04:04:03.631 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.626+0000 7fae6851e640 1 -- 172.21.15.42:0/3204262098 wait complete. 2024-09-16T04:04:03.631 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:03.626+0000 7fae6851e640 1 librados: shutdown 2024-09-16T04:04:06.646 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:04:06.779 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:06.774+0000 7f5f9ba56640 10 monclient: get_monmap_and_config 2024-09-16T04:04:06.779 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:06.774+0000 7f5f9ba56640 10 monclient: build_initial_monmap 2024-09-16T04:04:06.779 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:06.774+0000 7f5f9ba56640 10 monclient: monmap: 2024-09-16T04:04:06.779 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:04:06.779 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:04:06.779 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:04:06.777654+0000 2024-09-16T04:04:06.779 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:04:06.777654+0000 2024-09-16T04:04:06.779 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:04:06.780 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:04:06.780 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:04:06.780 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:04:06.780 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:04:06.780 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:04:06.780 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:04:06.780 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:04:06.782 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:06.774+0000 7f5f9ba56640 1 Processor -- start 2024-09-16T04:04:06.782 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:06.774+0000 7f5f9ba56640 1 -- start start 2024-09-16T04:04:06.782 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:06.774+0000 7f5f9ba56640 10 monclient: init 2024-09-16T04:04:06.782 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:06.774+0000 7f5f9ba56640 10 monclient: _reopen_session rank -1 2024-09-16T04:04:06.782 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:06.774+0000 7f5f9ba56640 10 monclient: _add_conns ranks=[1,2,4,3,0] 2024-09-16T04:04:06.782 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:06.774+0000 7f5f9ba56640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5f940a1330 addr v1:172.21.15.154:6789/0 2024-09-16T04:04:06.782 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:06.774+0000 7f5f9ba56640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5f940a1710 0x7f5f940a1b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:04:06.782 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:06.774+0000 7f5f9ba56640 10 monclient(hunting): picked mon.noname-b con 0x7f5f940a1710 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:04:06.782 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:06.774+0000 7f5f9ba56640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f5f941393a0 0x7f5f9413b790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:04:06.782 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:06.774+0000 7f5f9ba56640 10 monclient(hunting): picked mon.noname-c con 0x7f5f941393a0 addr v2:172.21.15.154:3300/0 2024-09-16T04:04:06.783 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:06.774+0000 7f5f9ba56640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5f9409e4c0 con 0x7f5f940a1330 2024-09-16T04:04:06.783 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:06.774+0000 7f5f9ba56640 10 monclient(hunting): start opening mon connection 2024-09-16T04:04:06.783 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:06.774+0000 7f5f9ba56640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f5f9409edf0 con 0x7f5f941393a0 2024-09-16T04:04:06.783 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:06.774+0000 7f5f9ba56640 10 monclient(hunting): start opening mon connection 2024-09-16T04:04:06.783 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:06.774+0000 7f5f9ba56640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f5f9409cd60 con 0x7f5f940a1710 2024-09-16T04:04:06.783 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:06.774+0000 7f5f9ba56640 10 monclient(hunting): _renew_subs 2024-09-16T04:04:06.783 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:06.774+0000 7f5f9ba56640 10 monclient(hunting): authenticate will time out at 2293.747701s 2024-09-16T04:04:06.783 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:06.778+0000 7f5f98ff1640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5f940a1710 0x7f5f940a1b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:06.783 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:06.778+0000 7f5f98ff1640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5f940a1710 0x7f5f940a1b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:36966/0 (socket says 172.21.15.42:36966) 2024-09-16T04:04:06.783 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:06.778+0000 7f5f98ff1640 1 -- 172.21.15.42:0/4213657315 learned_addr learned my addr 172.21.15.42:0/4213657315 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:04:06.783 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:06.778+0000 7f5f98ff1640 10 monclient(hunting): get_auth_request con 0x7f5f940a1710 auth_method 0 2024-09-16T04:04:06.783 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:06.778+0000 7f5f98ff1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:04:06.783 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:06.778+0000 7f5f98ff1640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:04:06.784 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:06.778+0000 7f5f98ff1640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:04:06.784 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:06.778+0000 7f5f99ff3640 1 --2- 172.21.15.42:0/4213657315 >> v2:172.21.15.154:3300/0 conn(0x7f5f941393a0 0x7f5f9413b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:06.784 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:06.778+0000 7f5f98ff1640 1 -- 172.21.15.42:0/4213657315 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5f940a1710 msgr2=0x7f5f940a1b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T04:04:06.784 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:06.778+0000 7f5f98ff1640 1 -- 172.21.15.42:0/4213657315 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5f940a1710 msgr2=0x7f5f940a1b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:04:06.784 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:06.778+0000 7f5f98ff1640 1 --2- 172.21.15.42:0/4213657315 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5f940a1710 0x7f5f940a1b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:04:06.784 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:06.778+0000 7f5f99ff3640 10 monclient(hunting): get_auth_request con 0x7f5f941393a0 auth_method 0 2024-09-16T04:04:06.784 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:06.778+0000 7f5f99ff3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:04:06.784 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:06.778+0000 7f5f99ff3640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:04:06.784 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:06.778+0000 7f5f99ff3640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:04:06.784 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:06.778+0000 7f5f98ff1640 1 --2- 172.21.15.42:0/4213657315 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5f940a1710 0x7f5f940a1b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:04:06.784 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:06.778+0000 7f5f99ff3640 1 -- 172.21.15.42:0/4213657315 >> v2:172.21.15.154:3300/0 conn(0x7f5f941393a0 msgr2=0x7f5f9413b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:04:06.784 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:06.778+0000 7f5f99ff3640 1 -- 172.21.15.42:0/4213657315 >> v2:172.21.15.154:3300/0 conn(0x7f5f941393a0 msgr2=0x7f5f9413b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:04:06.784 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:06.778+0000 7f5f99ff3640 1 --2- 172.21.15.42:0/4213657315 >> v2:172.21.15.154:3300/0 conn(0x7f5f941393a0 0x7f5f9413b790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:04:06.784 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:06.778+0000 7f5f99ff3640 1 --2- 172.21.15.42:0/4213657315 >> v2:172.21.15.154:3300/0 conn(0x7f5f941393a0 0x7f5f9413b790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:04:06.987 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:06.982+0000 7f5f98ff1640 1 --2- 172.21.15.42:0/4213657315 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5f940a1710 0x7f5f940a1b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:06.987 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:06.982+0000 7f5f98ff1640 10 monclient(hunting): get_auth_request con 0x7f5f940a1710 auth_method 0 2024-09-16T04:04:06.987 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:06.982+0000 7f5f98ff1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:04:06.987 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:06.982+0000 7f5f98ff1640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:04:06.987 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:06.982+0000 7f5f98ff1640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:04:06.987 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:06.982+0000 7f5f98ff1640 1 -- 172.21.15.42:0/4213657315 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5f940a1710 msgr2=0x7f5f940a1b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:04:06.987 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:06.982+0000 7f5f98ff1640 1 -- 172.21.15.42:0/4213657315 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5f940a1710 msgr2=0x7f5f940a1b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:04:06.987 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:06.982+0000 7f5f98ff1640 1 --2- 172.21.15.42:0/4213657315 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5f940a1710 0x7f5f940a1b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:04:06.987 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:06.982+0000 7f5f99ff3640 1 --2- 172.21.15.42:0/4213657315 >> v2:172.21.15.154:3300/0 conn(0x7f5f941393a0 0x7f5f9413b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:06.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:06.982+0000 7f5f98ff1640 1 --2- 172.21.15.42:0/4213657315 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5f940a1710 0x7f5f940a1b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T04:04:06.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:06.982+0000 7f5f99ff3640 10 monclient(hunting): get_auth_request con 0x7f5f941393a0 auth_method 0 2024-09-16T04:04:06.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:06.982+0000 7f5f99ff3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:04:06.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:06.982+0000 7f5f99ff3640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:04:06.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:06.982+0000 7f5f99ff3640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:04:06.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:06.982+0000 7f5f99ff3640 1 -- 172.21.15.42:0/4213657315 >> v2:172.21.15.154:3300/0 conn(0x7f5f941393a0 msgr2=0x7f5f9413b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T04:04:06.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:06.982+0000 7f5f99ff3640 1 -- 172.21.15.42:0/4213657315 >> v2:172.21.15.154:3300/0 conn(0x7f5f941393a0 msgr2=0x7f5f9413b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:04:06.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:06.982+0000 7f5f99ff3640 1 --2- 172.21.15.42:0/4213657315 >> v2:172.21.15.154:3300/0 conn(0x7f5f941393a0 0x7f5f9413b790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:04:06.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:06.982+0000 7f5f99ff3640 1 --2- 172.21.15.42:0/4213657315 >> v2:172.21.15.154:3300/0 conn(0x7f5f941393a0 0x7f5f9413b790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T04:04:07.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:07.386+0000 7f5f98ff1640 1 --2- 172.21.15.42:0/4213657315 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5f940a1710 0x7f5f940a1b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:07.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:07.386+0000 7f5f98ff1640 10 monclient(hunting): get_auth_request con 0x7f5f940a1710 auth_method 0 2024-09-16T04:04:07.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:07.386+0000 7f5f98ff1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:04:07.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:07.386+0000 7f5f98ff1640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:04:07.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:07.386+0000 7f5f98ff1640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:04:07.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:07.386+0000 7f5f99ff3640 1 --2- 172.21.15.42:0/4213657315 >> v2:172.21.15.154:3300/0 conn(0x7f5f941393a0 0x7f5f9413b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:07.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:07.386+0000 7f5f98ff1640 1 -- 172.21.15.42:0/4213657315 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5f940a1710 msgr2=0x7f5f940a1b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T04:04:07.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:07.386+0000 7f5f99ff3640 10 monclient(hunting): get_auth_request con 0x7f5f941393a0 auth_method 0 2024-09-16T04:04:07.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:07.386+0000 7f5f98ff1640 1 -- 172.21.15.42:0/4213657315 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5f940a1710 msgr2=0x7f5f940a1b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:04:07.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:07.386+0000 7f5f99ff3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:04:07.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:07.386+0000 7f5f99ff3640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:04:07.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:07.386+0000 7f5f98ff1640 1 --2- 172.21.15.42:0/4213657315 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5f940a1710 0x7f5f940a1b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:04:07.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:07.386+0000 7f5f99ff3640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:04:07.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:07.386+0000 7f5f98ff1640 1 --2- 172.21.15.42:0/4213657315 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5f940a1710 0x7f5f940a1b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T04:04:07.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:07.386+0000 7f5f99ff3640 1 -- 172.21.15.42:0/4213657315 >> v2:172.21.15.154:3300/0 conn(0x7f5f941393a0 msgr2=0x7f5f9413b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:04:07.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:07.386+0000 7f5f99ff3640 1 -- 172.21.15.42:0/4213657315 >> v2:172.21.15.154:3300/0 conn(0x7f5f941393a0 msgr2=0x7f5f9413b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:04:07.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:07.386+0000 7f5f99ff3640 1 --2- 172.21.15.42:0/4213657315 >> v2:172.21.15.154:3300/0 conn(0x7f5f941393a0 0x7f5f9413b790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:04:07.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:07.386+0000 7f5f99ff3640 1 --2- 172.21.15.42:0/4213657315 >> v2:172.21.15.154:3300/0 conn(0x7f5f941393a0 0x7f5f9413b790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T04:04:08.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:08.190+0000 7f5f98ff1640 1 --2- 172.21.15.42:0/4213657315 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5f940a1710 0x7f5f940a1b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:08.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:08.190+0000 7f5f98ff1640 10 monclient(hunting): get_auth_request con 0x7f5f940a1710 auth_method 0 2024-09-16T04:04:08.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:08.190+0000 7f5f98ff1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:04:08.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:08.190+0000 7f5f98ff1640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:04:08.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:08.190+0000 7f5f98ff1640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:04:08.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:08.190+0000 7f5f99ff3640 1 --2- 172.21.15.42:0/4213657315 >> v2:172.21.15.154:3300/0 conn(0x7f5f941393a0 0x7f5f9413b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:08.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:08.190+0000 7f5f98ff1640 1 -- 172.21.15.42:0/4213657315 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5f940a1710 msgr2=0x7f5f940a1b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T04:04:08.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:08.190+0000 7f5f98ff1640 1 -- 172.21.15.42:0/4213657315 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5f940a1710 msgr2=0x7f5f940a1b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:04:08.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:08.190+0000 7f5f98ff1640 1 --2- 172.21.15.42:0/4213657315 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5f940a1710 0x7f5f940a1b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:04:08.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:08.190+0000 7f5f99ff3640 10 monclient(hunting): get_auth_request con 0x7f5f941393a0 auth_method 0 2024-09-16T04:04:08.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:08.190+0000 7f5f99ff3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:04:08.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:08.190+0000 7f5f99ff3640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:04:08.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:08.190+0000 7f5f99ff3640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:04:08.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:08.190+0000 7f5f98ff1640 1 --2- 172.21.15.42:0/4213657315 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5f940a1710 0x7f5f940a1b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T04:04:08.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:08.190+0000 7f5f99ff3640 1 -- 172.21.15.42:0/4213657315 >> v2:172.21.15.154:3300/0 conn(0x7f5f941393a0 msgr2=0x7f5f9413b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:04:08.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:08.190+0000 7f5f99ff3640 1 -- 172.21.15.42:0/4213657315 >> v2:172.21.15.154:3300/0 conn(0x7f5f941393a0 msgr2=0x7f5f9413b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:04:08.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:08.190+0000 7f5f99ff3640 1 --2- 172.21.15.42:0/4213657315 >> v2:172.21.15.154:3300/0 conn(0x7f5f941393a0 0x7f5f9413b790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:04:08.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:08.190+0000 7f5f99ff3640 1 --2- 172.21.15.42:0/4213657315 >> v2:172.21.15.154:3300/0 conn(0x7f5f941393a0 0x7f5f9413b790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T04:04:09.782 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:09.774+0000 7f5f8affd640 10 monclient(hunting): tick 2024-09-16T04:04:09.782 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:09.774+0000 7f5f8affd640 10 monclient(hunting): _check_auth_tickets 2024-09-16T04:04:09.782 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:09.774+0000 7f5f8affd640 1 monclient(hunting): continuing hunt 2024-09-16T04:04:09.782 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:09.774+0000 7f5f8affd640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T04:04:09.783 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:09.774+0000 7f5f8affd640 1 -- 172.21.15.42:0/4213657315 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5f940a1710 msgr2=0x7f5f940a1b10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:04:09.783 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:09.774+0000 7f5f8affd640 1 --2- 172.21.15.42:0/4213657315 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5f940a1710 0x7f5f940a1b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:09.783 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:09.774+0000 7f5f8affd640 1 -- 172.21.15.42:0/4213657315 >> v2:172.21.15.154:3300/0 conn(0x7f5f941393a0 msgr2=0x7f5f9413b790 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:04:09.783 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:09.774+0000 7f5f8affd640 1 --2- 172.21.15.42:0/4213657315 >> v2:172.21.15.154:3300/0 conn(0x7f5f941393a0 0x7f5f9413b790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:09.783 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:09.774+0000 7f5f8affd640 1 -- 172.21.15.42:0/4213657315 >> v1:172.21.15.154:6789/0 conn(0x7f5f940a1330 legacy=0x7f5f9409f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:04:09.783 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:09.774+0000 7f5f8affd640 10 monclient: _add_conns ranks=[3,0] 2024-09-16T04:04:09.783 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:09.778+0000 7f5f8affd640 1 --2- 172.21.15.42:0/4213657315 >> v2:172.21.15.42:3300/0 conn(0x7f5f780058e0 0x7f5f78007d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:04:09.783 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:09.778+0000 7f5f8affd640 10 monclient(hunting): picked mon.noname-a con 0x7f5f780058e0 addr v2:172.21.15.42:3300/0 2024-09-16T04:04:09.783 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:09.778+0000 7f5f8affd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5f78008430 addr v1:172.21.15.42:6789/0 2024-09-16T04:04:09.783 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:09.778+0000 7f5f8affd640 1 -- 172.21.15.42:0/4213657315 --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5f940a2050 con 0x7f5f78008430 2024-09-16T04:04:09.783 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:09.778+0000 7f5f8affd640 10 monclient(hunting): start opening mon connection 2024-09-16T04:04:09.783 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:09.778+0000 7f5f8affd640 1 -- 172.21.15.42:0/4213657315 --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f5f9409edf0 con 0x7f5f780058e0 2024-09-16T04:04:09.784 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:09.778+0000 7f5f8affd640 10 monclient(hunting): _renew_subs 2024-09-16T04:04:09.784 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:09.778+0000 7f5f997f2640 1 --2- 172.21.15.42:0/4213657315 >> v2:172.21.15.42:3300/0 conn(0x7f5f780058e0 0x7f5f78007d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:09.784 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:09.778+0000 7f5f997f2640 10 monclient(hunting): get_auth_request con 0x7f5f780058e0 auth_method 0 2024-09-16T04:04:09.784 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:09.778+0000 7f5f997f2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:04:09.784 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:09.778+0000 7f5f997f2640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:04:09.784 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:09.778+0000 7f5f997f2640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:04:09.784 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:09.778+0000 7f5f997f2640 1 -- 172.21.15.42:0/4213657315 >> v2:172.21.15.42:3300/0 conn(0x7f5f780058e0 msgr2=0x7f5f78007d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:04:09.784 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:09.778+0000 7f5f997f2640 1 -- 172.21.15.42:0/4213657315 >> v2:172.21.15.42:3300/0 conn(0x7f5f780058e0 msgr2=0x7f5f78007d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:04:09.784 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:09.778+0000 7f5f997f2640 1 --2- 172.21.15.42:0/4213657315 >> v2:172.21.15.42:3300/0 conn(0x7f5f780058e0 0x7f5f78007d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:04:09.784 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:09.778+0000 7f5f997f2640 1 --2- 172.21.15.42:0/4213657315 >> v2:172.21.15.42:3300/0 conn(0x7f5f780058e0 0x7f5f78007d40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:04:09.954 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:09.950+0000 7f5f98ff1640 1 -- 172.21.15.42:0/4213657315 >> v1:172.21.15.42:6789/0 conn(0x7f5f78008430 legacy=0x7f5f7800a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-09-16T04:04:09.954 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:09.950+0000 7f5f98ff1640 1 -- 172.21.15.42:0/4213657315 >> v1:172.21.15.42:6789/0 conn(0x7f5f78008430 legacy=0x7f5f7800a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:04:09.954 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:09.950+0000 7f5f98ff1640 1 --1- 172.21.15.42:0/4213657315 >> v1:172.21.15.42:6789/0 conn(0x7f5f78008430 0x7f5f7800a8f0 :-1 s=OPENED pgs=25 cs=1 l=1).handle_message read tag failed 2024-09-16T04:04:09.954 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:09.950+0000 7f5f98ff1640 1 --1- 172.21.15.42:0/4213657315 >> v1:172.21.15.42:6789/0 conn(0x7f5f78008430 0x7f5f7800a8f0 :-1 s=OPENED pgs=25 cs=1 l=1).fault on lossy channel, failing 2024-09-16T04:04:09.954 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:09.950+0000 7f5f8bfff640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.42:6789/0 2024-09-16T04:04:09.986 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:09.982+0000 7f5f997f2640 1 --2- 172.21.15.42:0/4213657315 >> v2:172.21.15.42:3300/0 conn(0x7f5f780058e0 0x7f5f78007d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:09.986 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:09.982+0000 7f5f997f2640 10 monclient(hunting): get_auth_request con 0x7f5f780058e0 auth_method 0 2024-09-16T04:04:09.986 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:09.982+0000 7f5f997f2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:04:09.986 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:09.982+0000 7f5f997f2640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:04:09.986 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:09.982+0000 7f5f997f2640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:04:09.986 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:09.982+0000 7f5f997f2640 1 -- 172.21.15.42:0/4213657315 >> v2:172.21.15.42:3300/0 conn(0x7f5f780058e0 msgr2=0x7f5f78007d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:04:09.986 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:09.982+0000 7f5f997f2640 1 -- 172.21.15.42:0/4213657315 >> v2:172.21.15.42:3300/0 conn(0x7f5f780058e0 msgr2=0x7f5f78007d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:04:09.987 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:09.982+0000 7f5f997f2640 1 --2- 172.21.15.42:0/4213657315 >> v2:172.21.15.42:3300/0 conn(0x7f5f780058e0 0x7f5f78007d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:04:09.987 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:09.982+0000 7f5f997f2640 1 --2- 172.21.15.42:0/4213657315 >> v2:172.21.15.42:3300/0 conn(0x7f5f780058e0 0x7f5f78007d40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T04:04:10.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:10.386+0000 7f5f997f2640 1 --2- 172.21.15.42:0/4213657315 >> v2:172.21.15.42:3300/0 conn(0x7f5f780058e0 0x7f5f78007d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:10.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:10.386+0000 7f5f997f2640 10 monclient(hunting): get_auth_request con 0x7f5f780058e0 auth_method 0 2024-09-16T04:04:10.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:10.386+0000 7f5f997f2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:04:10.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:10.386+0000 7f5f997f2640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:04:10.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:10.386+0000 7f5f997f2640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:04:10.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:10.386+0000 7f5f997f2640 1 -- 172.21.15.42:0/4213657315 >> v2:172.21.15.42:3300/0 conn(0x7f5f780058e0 msgr2=0x7f5f78007d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:04:10.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:10.386+0000 7f5f997f2640 1 -- 172.21.15.42:0/4213657315 >> v2:172.21.15.42:3300/0 conn(0x7f5f780058e0 msgr2=0x7f5f78007d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:04:10.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:10.386+0000 7f5f997f2640 1 --2- 172.21.15.42:0/4213657315 >> v2:172.21.15.42:3300/0 conn(0x7f5f780058e0 0x7f5f78007d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:04:10.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:10.386+0000 7f5f997f2640 1 --2- 172.21.15.42:0/4213657315 >> v2:172.21.15.42:3300/0 conn(0x7f5f780058e0 0x7f5f78007d40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T04:04:11.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:11.190+0000 7f5f997f2640 1 --2- 172.21.15.42:0/4213657315 >> v2:172.21.15.42:3300/0 conn(0x7f5f780058e0 0x7f5f78007d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:11.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:11.190+0000 7f5f997f2640 10 monclient(hunting): get_auth_request con 0x7f5f780058e0 auth_method 0 2024-09-16T04:04:11.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:11.190+0000 7f5f997f2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:04:11.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:11.190+0000 7f5f997f2640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:04:11.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:11.190+0000 7f5f997f2640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:04:11.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:11.190+0000 7f5f997f2640 1 -- 172.21.15.42:0/4213657315 >> v2:172.21.15.42:3300/0 conn(0x7f5f780058e0 msgr2=0x7f5f78007d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:04:11.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:11.190+0000 7f5f997f2640 1 -- 172.21.15.42:0/4213657315 >> v2:172.21.15.42:3300/0 conn(0x7f5f780058e0 msgr2=0x7f5f78007d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:04:11.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:11.190+0000 7f5f997f2640 1 --2- 172.21.15.42:0/4213657315 >> v2:172.21.15.42:3300/0 conn(0x7f5f780058e0 0x7f5f78007d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:04:11.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:11.190+0000 7f5f997f2640 1 --2- 172.21.15.42:0/4213657315 >> v2:172.21.15.42:3300/0 conn(0x7f5f780058e0 0x7f5f78007d40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T04:04:12.783 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:12.778+0000 7f5f8affd640 10 monclient(hunting): tick 2024-09-16T04:04:12.783 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:12.778+0000 7f5f8affd640 10 monclient(hunting): _check_auth_tickets 2024-09-16T04:04:12.783 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:12.778+0000 7f5f8affd640 1 monclient(hunting): continuing hunt 2024-09-16T04:04:12.783 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:12.778+0000 7f5f8affd640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T04:04:12.783 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:12.778+0000 7f5f8affd640 1 -- 172.21.15.42:0/4213657315 >> v2:172.21.15.42:3300/0 conn(0x7f5f780058e0 msgr2=0x7f5f78007d40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:04:12.783 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:12.778+0000 7f5f8affd640 1 --2- 172.21.15.42:0/4213657315 >> v2:172.21.15.42:3300/0 conn(0x7f5f780058e0 0x7f5f78007d40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:12.783 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:12.778+0000 7f5f8affd640 1 -- 172.21.15.42:0/4213657315 >> v1:172.21.15.42:6789/0 conn(0x7f5f78008430 legacy=0x7f5f7800a8f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:04:12.784 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:12.778+0000 7f5f8affd640 10 monclient: _add_conns ranks=[1,3,4,0,2] 2024-09-16T04:04:12.784 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:12.778+0000 7f5f99ff3640 1 -- 172.21.15.42:0/4213657315 reap_dead start 2024-09-16T04:04:12.784 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:12.778+0000 7f5f8affd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5f940a1330 addr v1:172.21.15.154:6789/0 2024-09-16T04:04:12.784 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:12.778+0000 7f5f8affd640 1 --2- 172.21.15.42:0/4213657315 >> v2:172.21.15.42:3300/0 conn(0x7f5f7800c8d0 0x7f5f78007cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:04:12.784 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:12.778+0000 7f5f8affd640 10 monclient(hunting): picked mon.noname-a con 0x7f5f7800c8d0 addr v2:172.21.15.42:3300/0 2024-09-16T04:04:12.784 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:12.778+0000 7f5f8affd640 1 --2- 172.21.15.42:0/4213657315 >> v2:172.21.15.154:3300/0 conn(0x7f5f7800b0a0 0x7f5f7800a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:04:12.784 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:12.778+0000 7f5f8affd640 10 monclient(hunting): picked mon.noname-c con 0x7f5f7800b0a0 addr v2:172.21.15.154:3300/0 2024-09-16T04:04:12.784 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:12.778+0000 7f5f8affd640 1 -- 172.21.15.42:0/4213657315 --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5f7800d1c0 con 0x7f5f940a1330 2024-09-16T04:04:12.784 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:12.778+0000 7f5f8affd640 10 monclient(hunting): start opening mon connection 2024-09-16T04:04:12.784 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:12.778+0000 7f5f8affd640 1 -- 172.21.15.42:0/4213657315 --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f5f9409edf0 con 0x7f5f7800c8d0 2024-09-16T04:04:12.784 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:12.778+0000 7f5f8affd640 10 monclient(hunting): start opening mon connection 2024-09-16T04:04:12.784 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:12.778+0000 7f5f8affd640 1 -- 172.21.15.42:0/4213657315 --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f5f9409cd60 con 0x7f5f7800b0a0 2024-09-16T04:04:12.784 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:12.778+0000 7f5f8affd640 10 monclient(hunting): _renew_subs 2024-09-16T04:04:12.785 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:12.778+0000 7f5f98ff1640 1 --2- 172.21.15.42:0/4213657315 >> v2:172.21.15.42:3300/0 conn(0x7f5f7800c8d0 0x7f5f78007cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:12.785 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:12.778+0000 7f5f98ff1640 10 monclient(hunting): get_auth_request con 0x7f5f7800c8d0 auth_method 0 2024-09-16T04:04:12.785 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:12.778+0000 7f5f98ff1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:04:12.785 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:12.778+0000 7f5f98ff1640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:04:12.785 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:12.778+0000 7f5f98ff1640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:04:12.785 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:12.778+0000 7f5f99ff3640 1 -- 172.21.15.42:0/4213657315 >> v2:172.21.15.154:3300/0 conn(0x7f5f7800b0a0 msgr2=0x7f5f7800a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:04:12.785 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:12.778+0000 7f5f99ff3640 1 -- 172.21.15.42:0/4213657315 >> v2:172.21.15.154:3300/0 conn(0x7f5f7800b0a0 msgr2=0x7f5f7800a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:04:12.785 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:12.778+0000 7f5f99ff3640 1 --2- 172.21.15.42:0/4213657315 >> v2:172.21.15.154:3300/0 conn(0x7f5f7800b0a0 0x7f5f7800a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner read peer banner failed r=-1 ((1) Operation not permitted) 2024-09-16T04:04:12.785 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:12.778+0000 7f5f99ff3640 1 --2- 172.21.15.42:0/4213657315 >> v2:172.21.15.154:3300/0 conn(0x7f5f7800b0a0 0x7f5f7800a8f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:04:12.785 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:12.778+0000 7f5f98ff1640 1 -- 172.21.15.42:0/4213657315 >> v2:172.21.15.42:3300/0 conn(0x7f5f7800c8d0 msgr2=0x7f5f78007cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T04:04:12.785 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:12.778+0000 7f5f98ff1640 1 -- 172.21.15.42:0/4213657315 >> v2:172.21.15.42:3300/0 conn(0x7f5f7800c8d0 msgr2=0x7f5f78007cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:04:12.785 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:12.778+0000 7f5f98ff1640 1 --2- 172.21.15.42:0/4213657315 >> v2:172.21.15.42:3300/0 conn(0x7f5f7800c8d0 0x7f5f78007cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:04:12.785 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:12.778+0000 7f5f98ff1640 1 --2- 172.21.15.42:0/4213657315 >> v2:172.21.15.42:3300/0 conn(0x7f5f7800c8d0 0x7f5f78007cd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:04:12.986 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:12.982+0000 7f5f98ff1640 1 --2- 172.21.15.42:0/4213657315 >> v2:172.21.15.42:3300/0 conn(0x7f5f7800c8d0 0x7f5f78007cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:12.986 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:12.982+0000 7f5f98ff1640 10 monclient(hunting): get_auth_request con 0x7f5f7800c8d0 auth_method 0 2024-09-16T04:04:12.987 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:12.982+0000 7f5f98ff1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:04:12.987 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:12.982+0000 7f5f98ff1640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:04:12.987 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:12.982+0000 7f5f98ff1640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:04:12.987 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:12.982+0000 7f5f99ff3640 1 --2- 172.21.15.42:0/4213657315 >> v2:172.21.15.154:3300/0 conn(0x7f5f7800b0a0 0x7f5f7800a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:12.987 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:12.982+0000 7f5f99ff3640 10 monclient(hunting): get_auth_request con 0x7f5f7800b0a0 auth_method 0 2024-09-16T04:04:12.987 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:12.982+0000 7f5f99ff3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:04:12.987 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:12.982+0000 7f5f99ff3640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:04:12.987 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:12.982+0000 7f5f99ff3640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:04:12.987 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:12.982+0000 7f5f98ff1640 1 -- 172.21.15.42:0/4213657315 >> v2:172.21.15.42:3300/0 conn(0x7f5f7800c8d0 msgr2=0x7f5f78007cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T04:04:12.987 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:12.982+0000 7f5f98ff1640 1 -- 172.21.15.42:0/4213657315 >> v2:172.21.15.42:3300/0 conn(0x7f5f7800c8d0 msgr2=0x7f5f78007cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:04:12.987 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:12.982+0000 7f5f98ff1640 1 --2- 172.21.15.42:0/4213657315 >> v2:172.21.15.42:3300/0 conn(0x7f5f7800c8d0 0x7f5f78007cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:04:12.987 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:12.982+0000 7f5f99ff3640 1 -- 172.21.15.42:0/4213657315 >> v2:172.21.15.154:3300/0 conn(0x7f5f7800b0a0 msgr2=0x7f5f7800a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:04:12.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:12.982+0000 7f5f98ff1640 1 --2- 172.21.15.42:0/4213657315 >> v2:172.21.15.42:3300/0 conn(0x7f5f7800c8d0 0x7f5f78007cd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T04:04:12.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:12.982+0000 7f5f99ff3640 1 -- 172.21.15.42:0/4213657315 >> v2:172.21.15.154:3300/0 conn(0x7f5f7800b0a0 msgr2=0x7f5f7800a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:04:12.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:12.982+0000 7f5f99ff3640 1 --2- 172.21.15.42:0/4213657315 >> v2:172.21.15.154:3300/0 conn(0x7f5f7800b0a0 0x7f5f7800a8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:04:12.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:12.982+0000 7f5f99ff3640 1 --2- 172.21.15.42:0/4213657315 >> v2:172.21.15.154:3300/0 conn(0x7f5f7800b0a0 0x7f5f7800a8f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T04:04:13.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:13.386+0000 7f5f98ff1640 1 --2- 172.21.15.42:0/4213657315 >> v2:172.21.15.42:3300/0 conn(0x7f5f7800c8d0 0x7f5f78007cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:13.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:13.386+0000 7f5f98ff1640 10 monclient(hunting): get_auth_request con 0x7f5f7800c8d0 auth_method 0 2024-09-16T04:04:13.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:13.386+0000 7f5f98ff1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:04:13.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:13.386+0000 7f5f98ff1640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:04:13.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:13.386+0000 7f5f98ff1640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:04:13.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:13.386+0000 7f5f99ff3640 1 --2- 172.21.15.42:0/4213657315 >> v2:172.21.15.154:3300/0 conn(0x7f5f7800b0a0 0x7f5f7800a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:13.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:13.386+0000 7f5f99ff3640 10 monclient(hunting): get_auth_request con 0x7f5f7800b0a0 auth_method 0 2024-09-16T04:04:13.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:13.386+0000 7f5f99ff3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:04:13.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:13.386+0000 7f5f99ff3640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:04:13.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:13.386+0000 7f5f99ff3640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:04:13.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:13.386+0000 7f5f98ff1640 1 -- 172.21.15.42:0/4213657315 >> v2:172.21.15.42:3300/0 conn(0x7f5f7800c8d0 msgr2=0x7f5f78007cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T04:04:13.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:13.386+0000 7f5f98ff1640 1 -- 172.21.15.42:0/4213657315 >> v2:172.21.15.42:3300/0 conn(0x7f5f7800c8d0 msgr2=0x7f5f78007cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:04:13.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:13.386+0000 7f5f98ff1640 1 --2- 172.21.15.42:0/4213657315 >> v2:172.21.15.42:3300/0 conn(0x7f5f7800c8d0 0x7f5f78007cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:04:13.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:13.386+0000 7f5f99ff3640 1 -- 172.21.15.42:0/4213657315 >> v2:172.21.15.154:3300/0 conn(0x7f5f7800b0a0 msgr2=0x7f5f7800a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:04:13.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:13.386+0000 7f5f99ff3640 1 -- 172.21.15.42:0/4213657315 >> v2:172.21.15.154:3300/0 conn(0x7f5f7800b0a0 msgr2=0x7f5f7800a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:04:13.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:13.386+0000 7f5f99ff3640 1 --2- 172.21.15.42:0/4213657315 >> v2:172.21.15.154:3300/0 conn(0x7f5f7800b0a0 0x7f5f7800a8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:04:13.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:13.386+0000 7f5f99ff3640 1 --2- 172.21.15.42:0/4213657315 >> v2:172.21.15.154:3300/0 conn(0x7f5f7800b0a0 0x7f5f7800a8f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T04:04:13.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:13.386+0000 7f5f98ff1640 1 --2- 172.21.15.42:0/4213657315 >> v2:172.21.15.42:3300/0 conn(0x7f5f7800c8d0 0x7f5f78007cd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T04:04:14.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:14.190+0000 7f5f98ff1640 1 --2- 172.21.15.42:0/4213657315 >> v2:172.21.15.42:3300/0 conn(0x7f5f7800c8d0 0x7f5f78007cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:14.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:14.190+0000 7f5f98ff1640 10 monclient(hunting): get_auth_request con 0x7f5f7800c8d0 auth_method 0 2024-09-16T04:04:14.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:14.190+0000 7f5f98ff1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:04:14.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:14.190+0000 7f5f98ff1640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:04:14.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:14.190+0000 7f5f98ff1640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:04:14.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:14.190+0000 7f5f99ff3640 1 --2- 172.21.15.42:0/4213657315 >> v2:172.21.15.154:3300/0 conn(0x7f5f7800b0a0 0x7f5f7800a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:14.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:14.190+0000 7f5f99ff3640 10 monclient(hunting): get_auth_request con 0x7f5f7800b0a0 auth_method 0 2024-09-16T04:04:14.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:14.190+0000 7f5f99ff3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:04:14.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:14.190+0000 7f5f99ff3640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:04:14.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:14.190+0000 7f5f99ff3640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:04:14.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:14.190+0000 7f5f98ff1640 1 -- 172.21.15.42:0/4213657315 >> v2:172.21.15.42:3300/0 conn(0x7f5f7800c8d0 msgr2=0x7f5f78007cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:04:14.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:14.190+0000 7f5f98ff1640 1 -- 172.21.15.42:0/4213657315 >> v2:172.21.15.42:3300/0 conn(0x7f5f7800c8d0 msgr2=0x7f5f78007cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:04:14.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:14.190+0000 7f5f98ff1640 1 --2- 172.21.15.42:0/4213657315 >> v2:172.21.15.42:3300/0 conn(0x7f5f7800c8d0 0x7f5f78007cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:04:14.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:14.190+0000 7f5f99ff3640 1 -- 172.21.15.42:0/4213657315 >> v2:172.21.15.154:3300/0 conn(0x7f5f7800b0a0 msgr2=0x7f5f7800a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T04:04:14.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:14.190+0000 7f5f99ff3640 1 -- 172.21.15.42:0/4213657315 >> v2:172.21.15.154:3300/0 conn(0x7f5f7800b0a0 msgr2=0x7f5f7800a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:04:14.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:14.190+0000 7f5f99ff3640 1 --2- 172.21.15.42:0/4213657315 >> v2:172.21.15.154:3300/0 conn(0x7f5f7800b0a0 0x7f5f7800a8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:04:14.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:14.190+0000 7f5f98ff1640 1 --2- 172.21.15.42:0/4213657315 >> v2:172.21.15.42:3300/0 conn(0x7f5f7800c8d0 0x7f5f78007cd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T04:04:14.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:14.190+0000 7f5f99ff3640 1 --2- 172.21.15.42:0/4213657315 >> v2:172.21.15.154:3300/0 conn(0x7f5f7800b0a0 0x7f5f7800a8f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T04:04:15.783 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:15.778+0000 7f5f8affd640 10 monclient(hunting): tick 2024-09-16T04:04:15.783 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:15.778+0000 7f5f8affd640 10 monclient(hunting): _check_auth_tickets 2024-09-16T04:04:15.783 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:15.778+0000 7f5f8affd640 1 monclient(hunting): continuing hunt 2024-09-16T04:04:15.783 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:15.778+0000 7f5f8affd640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T04:04:15.783 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:15.778+0000 7f5f8affd640 1 -- 172.21.15.42:0/4213657315 >> v2:172.21.15.154:3300/0 conn(0x7f5f7800b0a0 msgr2=0x7f5f7800a8f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:04:15.783 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:15.778+0000 7f5f8affd640 1 --2- 172.21.15.42:0/4213657315 >> v2:172.21.15.154:3300/0 conn(0x7f5f7800b0a0 0x7f5f7800a8f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:15.784 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:15.778+0000 7f5f8affd640 1 -- 172.21.15.42:0/4213657315 >> v2:172.21.15.42:3300/0 conn(0x7f5f7800c8d0 msgr2=0x7f5f78007cd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:04:15.784 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:15.778+0000 7f5f8affd640 1 --2- 172.21.15.42:0/4213657315 >> v2:172.21.15.42:3300/0 conn(0x7f5f7800c8d0 0x7f5f78007cd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:15.784 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:15.778+0000 7f5f8affd640 1 -- 172.21.15.42:0/4213657315 >> v1:172.21.15.154:6789/0 conn(0x7f5f940a1330 legacy=0x7f5f7800c1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:04:15.784 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:15.778+0000 7f5f8affd640 10 monclient: _add_conns ranks=[0,2] 2024-09-16T04:04:15.784 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:15.778+0000 7f5f8affd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5f7800b650 addr v1:172.21.15.42:6789/0 2024-09-16T04:04:15.784 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:15.778+0000 7f5f8affd640 1 --2- 172.21.15.42:0/4213657315 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5f7800bb00 0x7f5f78015c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:04:15.784 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:15.778+0000 7f5f8affd640 10 monclient(hunting): picked mon.noname-b con 0x7f5f7800bb00 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:04:15.784 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:15.778+0000 7f5f8affd640 1 -- 172.21.15.42:0/4213657315 --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5f7800ce60 con 0x7f5f7800b650 2024-09-16T04:04:15.784 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:15.778+0000 7f5f8affd640 10 monclient(hunting): start opening mon connection 2024-09-16T04:04:15.784 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:15.778+0000 7f5f8affd640 1 -- 172.21.15.42:0/4213657315 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f5f9409edf0 con 0x7f5f7800bb00 2024-09-16T04:04:15.784 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:15.778+0000 7f5f8affd640 10 monclient(hunting): _renew_subs 2024-09-16T04:04:15.784 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:15.778+0000 7f5f98ff1640 1 --2- 172.21.15.42:0/4213657315 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5f7800bb00 0x7f5f78015c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:15.785 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:15.778+0000 7f5f98ff1640 10 monclient(hunting): get_auth_request con 0x7f5f7800bb00 auth_method 0 2024-09-16T04:04:15.785 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:15.778+0000 7f5f98ff1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:04:15.785 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:15.778+0000 7f5f98ff1640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:04:15.785 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:15.778+0000 7f5f98ff1640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:04:15.785 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:15.778+0000 7f5f98ff1640 1 -- 172.21.15.42:0/4213657315 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5f7800bb00 msgr2=0x7f5f78015c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:04:15.785 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:15.778+0000 7f5f98ff1640 1 -- 172.21.15.42:0/4213657315 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5f7800bb00 msgr2=0x7f5f78015c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:04:15.785 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:15.778+0000 7f5f98ff1640 1 --2- 172.21.15.42:0/4213657315 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5f7800bb00 0x7f5f78015c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:04:15.785 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:15.778+0000 7f5f98ff1640 1 --2- 172.21.15.42:0/4213657315 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5f7800bb00 0x7f5f78015c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:04:15.986 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:15.982+0000 7f5f98ff1640 1 --2- 172.21.15.42:0/4213657315 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5f7800bb00 0x7f5f78015c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:15.986 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:15.982+0000 7f5f98ff1640 10 monclient(hunting): get_auth_request con 0x7f5f7800bb00 auth_method 0 2024-09-16T04:04:15.986 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:15.982+0000 7f5f98ff1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:04:15.986 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:15.982+0000 7f5f98ff1640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:04:15.986 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:15.982+0000 7f5f98ff1640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:04:15.986 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:15.982+0000 7f5f98ff1640 1 -- 172.21.15.42:0/4213657315 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5f7800bb00 msgr2=0x7f5f78015c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:04:15.986 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:15.982+0000 7f5f98ff1640 1 -- 172.21.15.42:0/4213657315 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5f7800bb00 msgr2=0x7f5f78015c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:04:15.987 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:15.982+0000 7f5f98ff1640 1 --2- 172.21.15.42:0/4213657315 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5f7800bb00 0x7f5f78015c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:04:15.987 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:15.982+0000 7f5f98ff1640 1 --2- 172.21.15.42:0/4213657315 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5f7800bb00 0x7f5f78015c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T04:04:16.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:16.386+0000 7f5f98ff1640 1 --2- 172.21.15.42:0/4213657315 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5f7800bb00 0x7f5f78015c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:16.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:16.386+0000 7f5f98ff1640 10 monclient(hunting): get_auth_request con 0x7f5f7800bb00 auth_method 0 2024-09-16T04:04:16.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:16.386+0000 7f5f98ff1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:04:16.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:16.386+0000 7f5f98ff1640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:04:16.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:16.386+0000 7f5f98ff1640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:04:16.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:16.386+0000 7f5f98ff1640 1 -- 172.21.15.42:0/4213657315 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5f7800bb00 msgr2=0x7f5f78015c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:04:16.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:16.386+0000 7f5f98ff1640 1 -- 172.21.15.42:0/4213657315 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5f7800bb00 msgr2=0x7f5f78015c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:04:16.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:16.386+0000 7f5f98ff1640 1 --2- 172.21.15.42:0/4213657315 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5f7800bb00 0x7f5f78015c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:04:16.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:16.386+0000 7f5f98ff1640 1 --2- 172.21.15.42:0/4213657315 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5f7800bb00 0x7f5f78015c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T04:04:17.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.190+0000 7f5f98ff1640 1 --2- 172.21.15.42:0/4213657315 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5f7800bb00 0x7f5f78015c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:17.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.190+0000 7f5f98ff1640 10 monclient(hunting): get_auth_request con 0x7f5f7800bb00 auth_method 0 2024-09-16T04:04:17.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.190+0000 7f5f98ff1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:04:17.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.190+0000 7f5f98ff1640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:04:17.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.190+0000 7f5f98ff1640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:04:17.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.190+0000 7f5f98ff1640 1 -- 172.21.15.42:0/4213657315 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5f7800bb00 msgr2=0x7f5f78015c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:04:17.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.190+0000 7f5f98ff1640 1 -- 172.21.15.42:0/4213657315 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5f7800bb00 msgr2=0x7f5f78015c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:04:17.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.190+0000 7f5f98ff1640 1 --2- 172.21.15.42:0/4213657315 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5f7800bb00 0x7f5f78015c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:04:17.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.190+0000 7f5f98ff1640 1 --2- 172.21.15.42:0/4213657315 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5f7800bb00 0x7f5f78015c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T04:04:17.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.282+0000 7f5f8bfff640 1 -- 172.21.15.42:0/4213657315 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 557707830 0 0) 0x7f5f7800ce60 con 0x7f5f7800b650 2024-09-16T04:04:17.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.282+0000 7f5f8bfff640 10 monclient(hunting): _negotiate 2024-09-16T04:04:17.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.282+0000 7f5f8bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:04:17.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.282+0000 7f5f8bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:04:17.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.282+0000 7f5f8bfff640 10 monclient(hunting): my global_id is 984099 2024-09-16T04:04:17.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.282+0000 7f5f8bfff640 1 -- 172.21.15.42:0/4213657315 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5f80003530 con 0x7f5f7800b650 2024-09-16T04:04:17.294 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.286+0000 7f5f8bfff640 1 -- 172.21.15.42:0/4213657315 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 897925736 0 0) 0x7f5f80003530 con 0x7f5f7800b650 2024-09-16T04:04:17.294 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.286+0000 7f5f8bfff640 1 -- 172.21.15.42:0/4213657315 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5f7800bb00 msgr2=0x7f5f78015c50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:04:17.294 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.286+0000 7f5f8bfff640 1 --2- 172.21.15.42:0/4213657315 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5f7800bb00 0x7f5f78015c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:17.295 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.286+0000 7f5f8bfff640 10 monclient: _finish_hunting 0 2024-09-16T04:04:17.295 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.286+0000 7f5f8bfff640 1 monclient: found mon.noname-a-legacy 2024-09-16T04:04:17.295 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.286+0000 7f5f8bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:04:17.295 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.286+0000 7f5f8bfff640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.42:6789/0 2024-09-16T04:04:17.295 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.286+0000 7f5f8bfff640 1 -- 172.21.15.42:0/4213657315 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f78017220 con 0x7f5f7800b650 2024-09-16T04:04:17.295 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.286+0000 7f5f8bfff640 10 monclient: _finish_auth 0 2024-09-16T04:04:17.295 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.286+0000 7f5f8bfff640 10 monclient: _check_auth_tickets 2024-09-16T04:04:17.295 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.286+0000 7f5f8bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:04:17.295 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.286+0000 7f5f8bfff640 1 -- 172.21.15.42:0/4213657315 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2177413734 0 0) 0x7f5f900013e0 con 0x7f5f7800b650 2024-09-16T04:04:17.295 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.286+0000 7f5f8bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:04:17.295 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.286+0000 7f5f8bfff640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T04:04:17.295 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.286+0000 7f5f8bfff640 10 monclient: dump: 2024-09-16T04:04:17.295 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:04:17.296 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:04:17.296 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:17.296 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:17.296 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:04:17.296 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:04:17.296 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:04:17.296 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:04:17.296 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:04:17.296 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:04:17.296 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.286+0000 7f5f8bfff640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] but i'm connected to v1:172.21.15.42:6789/0, reconnecting 2024-09-16T04:04:17.296 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.286+0000 7f5f8bfff640 10 monclient: _reopen_session rank -1 2024-09-16T04:04:17.296 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.286+0000 7f5f8bfff640 1 -- 172.21.15.42:0/4213657315 >> v1:172.21.15.42:6789/0 conn(0x7f5f7800b650 legacy=0x7f5f78012520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:04:17.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.286+0000 7f5f8bfff640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:04:17.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.286+0000 7f5f99ff3640 1 -- 172.21.15.42:0/4213657315 reap_dead start 2024-09-16T04:04:17.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.290+0000 7f5f8bfff640 1 --2- 172.21.15.42:0/4213657315 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5f800039c0 0x7f5f80003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:04:17.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.290+0000 7f5f8bfff640 10 monclient(hunting): picked mon.c con 0x7f5f800039c0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:04:17.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.290+0000 7f5f8bfff640 1 --2- 172.21.15.42:0/4213657315 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5f80004810 0x7f5f80004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:04:17.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.290+0000 7f5f8bfff640 10 monclient(hunting): picked mon.b con 0x7f5f80004810 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:04:17.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.290+0000 7f5f8bfff640 1 --2- 172.21.15.42:0/4213657315 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5f80005220 0x7f5f8000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:04:17.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.290+0000 7f5f8bfff640 10 monclient(hunting): picked mon.a con 0x7f5f80005220 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:04:17.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.290+0000 7f5f8bfff640 10 monclient(hunting): start opening mon connection 2024-09-16T04:04:17.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.290+0000 7f5f8bfff640 1 -- 172.21.15.42:0/4213657315 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f5f9409edf0 con 0x7f5f80005220 2024-09-16T04:04:17.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.290+0000 7f5f8bfff640 10 monclient(hunting): start opening mon connection 2024-09-16T04:04:17.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.290+0000 7f5f8bfff640 1 -- 172.21.15.42:0/4213657315 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f5f8000c150 con 0x7f5f80004810 2024-09-16T04:04:17.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.290+0000 7f5f8bfff640 10 monclient(hunting): start opening mon connection 2024-09-16T04:04:17.298 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.290+0000 7f5f8bfff640 1 -- 172.21.15.42:0/4213657315 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f5f8000c350 con 0x7f5f800039c0 2024-09-16T04:04:17.298 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.290+0000 7f5f8bfff640 10 monclient(hunting): _renew_subs 2024-09-16T04:04:17.298 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.290+0000 7f5f8bfff640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:04:17.298 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.290+0000 7f5f997f2640 1 --2- 172.21.15.42:0/4213657315 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5f800039c0 0x7f5f80003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:17.298 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.290+0000 7f5f997f2640 10 monclient(hunting): get_auth_request con 0x7f5f800039c0 auth_method 0 2024-09-16T04:04:17.298 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.290+0000 7f5f99ff3640 1 --2- 172.21.15.42:0/4213657315 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5f80005220 0x7f5f8000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:17.298 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.290+0000 7f5f997f2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:04:17.298 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.290+0000 7f5f997f2640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:04:17.298 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.290+0000 7f5f997f2640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:04:17.298 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.290+0000 7f5f99ff3640 10 monclient(hunting): get_auth_request con 0x7f5f80005220 auth_method 0 2024-09-16T04:04:17.298 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.290+0000 7f5f99ff3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:04:17.298 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.290+0000 7f5f99ff3640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:04:17.298 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.290+0000 7f5f99ff3640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:04:17.298 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.290+0000 7f5f997f2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:04:17.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.290+0000 7f5f997f2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:04:17.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.290+0000 7f5f997f2640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:04:17.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.290+0000 7f5f99ff3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:04:17.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.290+0000 7f5f99ff3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:04:17.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.290+0000 7f5f98ff1640 1 --2- 172.21.15.42:0/4213657315 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5f80004810 0x7f5f80004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:17.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.290+0000 7f5f99ff3640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:04:17.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.290+0000 7f5f98ff1640 10 monclient(hunting): get_auth_request con 0x7f5f80004810 auth_method 0 2024-09-16T04:04:17.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.290+0000 7f5f98ff1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:04:17.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.290+0000 7f5f98ff1640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:04:17.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.290+0000 7f5f98ff1640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:04:17.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.290+0000 7f5f99ff3640 10 monclient(hunting): handle_auth_done global_id 984099 payload 293 2024-09-16T04:04:17.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.290+0000 7f5f99ff3640 1 -- 172.21.15.42:0/4213657315 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5f800039c0 msgr2=0x7f5f80003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:04:17.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.290+0000 7f5f99ff3640 1 --2- 172.21.15.42:0/4213657315 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5f800039c0 0x7f5f80003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:17.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.290+0000 7f5f99ff3640 1 -- 172.21.15.42:0/4213657315 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5f80004810 msgr2=0x7f5f80004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:04:17.300 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.290+0000 7f5f99ff3640 1 --2- 172.21.15.42:0/4213657315 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5f80004810 0x7f5f80004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:17.300 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.290+0000 7f5f99ff3640 10 monclient: _finish_hunting 0 2024-09-16T04:04:17.300 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.290+0000 7f5f99ff3640 1 monclient: found mon.a 2024-09-16T04:04:17.300 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.290+0000 7f5f99ff3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:04:17.300 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.290+0000 7f5f99ff3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:04:17.300 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.290+0000 7f5f99ff3640 1 -- 172.21.15.42:0/4213657315 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f80003530 con 0x7f5f80005220 2024-09-16T04:04:17.300 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.290+0000 7f5f99ff3640 10 monclient: _finish_auth 0 2024-09-16T04:04:17.300 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.290+0000 7f5f99ff3640 10 monclient: _check_auth_tickets 2024-09-16T04:04:17.300 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.290+0000 7f5f99ff3640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:04:17.300 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.290+0000 7f5f9ba56640 5 monclient: authenticate success, global_id 984099 2024-09-16T04:04:17.300 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.290+0000 7f5f9ba56640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:04:17.300 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.290+0000 7f5f997f2640 1 --2- 172.21.15.42:0/4213657315 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5f800039c0 0x7f5f80003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:04:17.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.294+0000 7f5f99ff3640 1 --2- 172.21.15.42:0/4213657315 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5f80005220 0x7f5f8000b9d0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f5f9409e800 tx=0x7f5f8c01dee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca613ca9422963ee server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:04:17.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.294+0000 7f5f8bfff640 1 -- 172.21.15.42:0/4213657315 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f8c0b4070 con 0x7f5f80005220 2024-09-16T04:04:17.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.294+0000 7f5f8bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:04:17.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.294+0000 7f5f8bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:04:17.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.294+0000 7f5f8bfff640 10 monclient: dump: 2024-09-16T04:04:17.302 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:04:17.302 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:04:17.302 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:17.303 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:17.303 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:04:17.303 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:04:17.303 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:04:17.303 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:04:17.303 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:04:17.303 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:04:17.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.294+0000 7f5f9ba56640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:04:17.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.294+0000 7f5f8bfff640 1 -- 172.21.15.42:0/4213657315 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5f8c0a6a20 con 0x7f5f80005220 2024-09-16T04:04:17.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.294+0000 7f5f8bfff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:04:17.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.294+0000 7f5f9ba56640 10 monclient: get_monmap_and_config success 2024-09-16T04:04:17.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.294+0000 7f5f9ba56640 4 set_mon_vals no callback set 2024-09-16T04:04:17.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.294+0000 7f5f8bfff640 1 -- 172.21.15.42:0/4213657315 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f8c0ac070 con 0x7f5f80005220 2024-09-16T04:04:17.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.294+0000 7f5f8bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:04:17.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.294+0000 7f5f8bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:04:17.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.294+0000 7f5f8bfff640 10 monclient: dump: 2024-09-16T04:04:17.304 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:04:17.304 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:04:17.304 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:17.304 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:17.304 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:04:17.304 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:04:17.304 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:04:17.304 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:04:17.304 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:04:17.304 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:04:17.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.294+0000 7f5f9ba56640 10 monclient: shutdown 2024-09-16T04:04:17.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.294+0000 7f5f9ba56640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:04:17.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.294+0000 7f5f9ba56640 1 -- 172.21.15.42:0/4213657315 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5f80005220 msgr2=0x7f5f8000b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:04:17.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.294+0000 7f5f9ba56640 1 --2- 172.21.15.42:0/4213657315 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5f80005220 0x7f5f8000b9d0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f5f9409e800 tx=0x7f5f8c01dee0 comp rx=0 tx=0).stop 2024-09-16T04:04:17.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.294+0000 7f5f9ba56640 1 -- 172.21.15.42:0/4213657315 shutdown_connections 2024-09-16T04:04:17.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.294+0000 7f5f9ba56640 1 --2- 172.21.15.42:0/4213657315 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5f80005220 0x7f5f8000b9d0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:17.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.294+0000 7f5f9ba56640 1 --2- 172.21.15.42:0/4213657315 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5f80004810 0x7f5f80004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:17.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.294+0000 7f5f9ba56640 1 --2- 172.21.15.42:0/4213657315 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5f800039c0 0x7f5f80003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:17.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.294+0000 7f5f9ba56640 1 -- 172.21.15.42:0/4213657315 >> 172.21.15.42:0/4213657315 conn(0x7f5f940a2420 msgr2=0x7f5f940a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:04:17.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.294+0000 7f5f9ba56640 1 -- 172.21.15.42:0/4213657315 shutdown_connections 2024-09-16T04:04:17.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.294+0000 7f5f9ba56640 1 -- 172.21.15.42:0/4213657315 wait complete. 2024-09-16T04:04:17.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.294+0000 7f5f9ba56640 10 monclient: build_initial_monmap 2024-09-16T04:04:17.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.294+0000 7f5f9ba56640 10 monclient: monmap: 2024-09-16T04:04:17.305 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:04:17.306 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:04:17.306 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:04:17.306 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:04:17.306 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:04:17.306 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:04:17.306 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:04:17.306 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:04:17.306 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:04:17.306 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:04:17.306 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.298+0000 7f5f9ba56640 1 librados: starting msgr at 2024-09-16T04:04:17.306 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.298+0000 7f5f9ba56640 1 librados: starting objecter 2024-09-16T04:04:17.306 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.298+0000 7f5f9ba56640 1 Processor -- start 2024-09-16T04:04:17.306 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.298+0000 7f5f9ba56640 1 -- start start 2024-09-16T04:04:17.306 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.298+0000 7f5f9ba56640 1 librados: setting wanted keys 2024-09-16T04:04:17.306 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.298+0000 7f5f9ba56640 1 librados: calling monclient init 2024-09-16T04:04:17.307 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.298+0000 7f5f9ba56640 10 monclient: init 2024-09-16T04:04:17.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.298+0000 7f5f9ba56640 10 monclient: _reopen_session rank -1 2024-09-16T04:04:17.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.298+0000 7f5f9ba56640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:04:17.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.298+0000 7f5f9ba56640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5f800039c0 0x7f5f94133c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:04:17.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.298+0000 7f5f9ba56640 10 monclient(hunting): picked mon.noname-c con 0x7f5f800039c0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:04:17.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.298+0000 7f5f9ba56640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5f80004810 0x7f5f9411f400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:04:17.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.298+0000 7f5f9ba56640 10 monclient(hunting): picked mon.noname-a con 0x7f5f80004810 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:04:17.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.298+0000 7f5f9ba56640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5f80005220 0x7f5f9411f940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:04:17.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.298+0000 7f5f9ba56640 10 monclient(hunting): picked mon.noname-b con 0x7f5f80005220 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:04:17.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.298+0000 7f5f9ba56640 10 monclient(hunting): start opening mon connection 2024-09-16T04:04:17.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.298+0000 7f5f9ba56640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f5f9409d1b0 con 0x7f5f80004810 2024-09-16T04:04:17.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.298+0000 7f5f9ba56640 10 monclient(hunting): start opening mon connection 2024-09-16T04:04:17.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.298+0000 7f5f9ba56640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f5f9409d030 con 0x7f5f80005220 2024-09-16T04:04:17.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.298+0000 7f5f9ba56640 10 monclient(hunting): start opening mon connection 2024-09-16T04:04:17.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.298+0000 7f5f9ba56640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f5f940a1b10 con 0x7f5f800039c0 2024-09-16T04:04:17.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.298+0000 7f5f9ba56640 10 monclient(hunting): _renew_subs 2024-09-16T04:04:17.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.298+0000 7f5f9ba56640 10 monclient(hunting): authenticate will time out at 2304.269522s 2024-09-16T04:04:17.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.298+0000 7f5f997f2640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5f800039c0 0x7f5f94133c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:17.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.298+0000 7f5f98ff1640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5f80004810 0x7f5f9411f400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:17.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.298+0000 7f5f98ff1640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5f80004810 0x7f5f9411f400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:46326/0 (socket says 172.21.15.42:46326) 2024-09-16T04:04:17.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.298+0000 7f5f98ff1640 1 -- 172.21.15.42:0/3255213260 learned_addr learned my addr 172.21.15.42:0/3255213260 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:04:17.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.298+0000 7f5f98ff1640 10 monclient(hunting): get_auth_request con 0x7f5f80004810 auth_method 0 2024-09-16T04:04:17.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.298+0000 7f5f98ff1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:04:17.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.298+0000 7f5f98ff1640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:04:17.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.298+0000 7f5f98ff1640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:04:17.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.298+0000 7f5f997f2640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5f800039c0 0x7f5f94133c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:48716/0 (socket says 172.21.15.42:48716) 2024-09-16T04:04:17.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.298+0000 7f5f997f2640 10 monclient(hunting): get_auth_request con 0x7f5f800039c0 auth_method 0 2024-09-16T04:04:17.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.298+0000 7f5f997f2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:04:17.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.298+0000 7f5f997f2640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:04:17.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.298+0000 7f5f997f2640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:04:17.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.298+0000 7f5f98ff1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:04:17.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.298+0000 7f5f98ff1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:04:17.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.298+0000 7f5f98ff1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:04:17.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.298+0000 7f5f997f2640 1 -- 172.21.15.42:0/3255213260 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5f800039c0 msgr2=0x7f5f94133c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-16T04:04:17.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.298+0000 7f5f997f2640 1 -- 172.21.15.42:0/3255213260 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5f800039c0 msgr2=0x7f5f94133c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:04:17.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.298+0000 7f5f997f2640 1 --2- 172.21.15.42:0/3255213260 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5f800039c0 0x7f5f94133c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:04:17.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.298+0000 7f5f997f2640 1 --2- 172.21.15.42:0/3255213260 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5f800039c0 0x7f5f94133c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:04:17.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.298+0000 7f5f98ff1640 10 monclient(hunting): handle_auth_done global_id 984102 payload 824 2024-09-16T04:04:17.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.298+0000 7f5f98ff1640 1 -- 172.21.15.42:0/3255213260 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5f800039c0 msgr2=0x7f5f94133c70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:04:17.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.298+0000 7f5f98ff1640 1 --2- 172.21.15.42:0/3255213260 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5f800039c0 0x7f5f94133c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:17.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.298+0000 7f5f98ff1640 1 -- 172.21.15.42:0/3255213260 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5f80005220 msgr2=0x7f5f9411f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:04:17.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.298+0000 7f5f98ff1640 1 --2- 172.21.15.42:0/3255213260 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5f80005220 0x7f5f9411f940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:17.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.298+0000 7f5f98ff1640 10 monclient: _finish_hunting 0 2024-09-16T04:04:17.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.298+0000 7f5f98ff1640 1 monclient: found mon.noname-a 2024-09-16T04:04:17.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.298+0000 7f5f98ff1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:04:17.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.298+0000 7f5f98ff1640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:04:17.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.298+0000 7f5f98ff1640 1 -- 172.21.15.42:0/3255213260 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f941230b0 con 0x7f5f80004810 2024-09-16T04:04:17.314 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.298+0000 7f5f98ff1640 1 --2- 172.21.15.42:0/3255213260 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5f80004810 0x7f5f9411f400 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f5f84006b20 tx=0x7f5f84018d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:04:17.314 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.298+0000 7f5f89ffb640 10 client.?.objecter ms_handle_connect 0x7f5f80004810 2024-09-16T04:04:17.314 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.298+0000 7f5f89ffb640 10 client.?.objecter resend_mon_ops 2024-09-16T04:04:17.314 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.298+0000 7f5f89ffb640 1 -- 172.21.15.42:0/3255213260 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f8400f040 con 0x7f5f80004810 2024-09-16T04:04:17.314 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.298+0000 7f5f89ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:04:17.314 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.298+0000 7f5f89ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:04:17.314 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.298+0000 7f5f89ffb640 10 monclient: dump: 2024-09-16T04:04:17.314 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:04:17.314 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:04:17.314 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:17.314 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:17.314 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:04:17.314 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:04:17.315 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:04:17.315 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:04:17.315 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:04:17.315 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:04:17.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.298+0000 7f5f89ffb640 10 monclient: _finish_auth 0 2024-09-16T04:04:17.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.298+0000 7f5f89ffb640 10 monclient: _check_auth_tickets 2024-09-16T04:04:17.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.298+0000 7f5f89ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:04:17.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.298+0000 7f5f89ffb640 1 -- 172.21.15.42:0/3255213260 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5f84017dd0 con 0x7f5f80004810 2024-09-16T04:04:17.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.298+0000 7f5f9ba56640 5 monclient: authenticate success, global_id 984102 2024-09-16T04:04:17.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.298+0000 7f5f9ba56640 10 monclient: _renew_subs 2024-09-16T04:04:17.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.298+0000 7f5f9ba56640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:04:17.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.298+0000 7f5f9ba56640 1 -- 172.21.15.42:0/3255213260 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f94121540 con 0x7f5f80004810 2024-09-16T04:04:17.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.298+0000 7f5f89ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:04:17.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.298+0000 7f5f89ffb640 1 -- 172.21.15.42:0/3255213260 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f84025660 con 0x7f5f80004810 2024-09-16T04:04:17.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.298+0000 7f5f89ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:04:17.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.298+0000 7f5f89ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:04:17.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.298+0000 7f5f89ffb640 10 monclient: dump: 2024-09-16T04:04:17.316 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:04:17.316 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:04:17.316 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:17.316 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:17.316 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:04:17.316 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:04:17.316 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:04:17.316 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:04:17.316 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:04:17.317 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:04:17.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.298+0000 7f5f8b7fe640 4 set_mon_vals no callback set 2024-09-16T04:04:17.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.298+0000 7f5f9ba56640 10 client.984102.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:04:17.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.298+0000 7f5f9ba56640 10 monclient: _renew_subs 2024-09-16T04:04:17.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.298+0000 7f5f9ba56640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:04:17.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.298+0000 7f5f9ba56640 1 -- 172.21.15.42:0/3255213260 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f940a3800 con 0x7f5f80004810 2024-09-16T04:04:17.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.298+0000 7f5f9ba56640 1 librados: init done 2024-09-16T04:04:17.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.298+0000 7f5f89ffb640 1 -- 172.21.15.42:0/3255213260 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f5f84025800 con 0x7f5f80004810 2024-09-16T04:04:17.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.298+0000 7f5f89ffb640 1 --2- 172.21.15.42:0/3255213260 >> v2:172.21.15.154:6812/254845342 conn(0x7f5f780391d0 0x7f5f7803b690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:04:17.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.298+0000 7f5f9ba56640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:04:17.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.298+0000 7f5f9ba56640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:04:17.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.298+0000 7f5f89ffb640 1 -- 172.21.15.42:0/3255213260 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(674..674 src has 451..674) ==== 6272+0+0 (secure 0 0 0) 0x7f5f8404bb10 con 0x7f5f80004810 2024-09-16T04:04:17.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.298+0000 7f5f9ba56640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:04:17.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.298+0000 7f5f89ffb640 10 client.984102.objecter ms_dispatch 0x7f5f94000ba0 osd_map(674..674 src has 451..674) 2024-09-16T04:04:17.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.298+0000 7f5f89ffb640 3 client.984102.objecter handle_osd_map got epochs [674,674] > 0 2024-09-16T04:04:17.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.298+0000 7f5f89ffb640 3 client.984102.objecter handle_osd_map decoding full epoch 674 2024-09-16T04:04:17.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.298+0000 7f5f9ba56640 1 -- 172.21.15.42:0/3255213260 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5f94073a30 con 0x7f5f80004810 2024-09-16T04:04:17.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.302+0000 7f5f89ffb640 20 client.984102.objecter dump_active .. 0 homeless 2024-09-16T04:04:17.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.302+0000 7f5f997f2640 1 --2- 172.21.15.42:0/3255213260 >> v2:172.21.15.154:6812/254845342 conn(0x7f5f780391d0 0x7f5f7803b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:17.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.302+0000 7f5f997f2640 10 monclient: get_auth_request con 0x7f5f780391d0 auth_method 0 2024-09-16T04:04:17.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.302+0000 7f5f89ffb640 10 client.984102.objecter ms_handle_connect 0x7f5f780391d0 2024-09-16T04:04:17.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.302+0000 7f5f997f2640 1 --2- 172.21.15.42:0/3255213260 >> v2:172.21.15.154:6812/254845342 conn(0x7f5f780391d0 0x7f5f7803b690 secure :-1 s=READY pgs=3818 cs=0 l=1 rev1=1 crypto rx=0x7f5f90005630 tx=0x7f5f90009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:04:17.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.306+0000 7f5f89ffb640 1 -- 172.21.15.42:0/3255213260 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f5f8401f070 con 0x7f5f80004810 2024-09-16T04:04:17.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.306+0000 7f5f89ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:04:17.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.306+0000 7f5f89ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:04:17.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.542+0000 7f5f9ba56640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:04:17.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.542+0000 7f5f9ba56640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:04:17.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.542+0000 7f5f9ba56640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:04:17.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.542+0000 7f5f9ba56640 1 -- 172.21.15.42:0/3255213260 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f5f940a1e40 con 0x7f5f80004810 2024-09-16T04:04:17.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.542+0000 7f5f89ffb640 1 -- 172.21.15.42:0/3255213260 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1555 (secure 0 0 0) 0x7f5f84023dd0 con 0x7f5f80004810 2024-09-16T04:04:17.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.542+0000 7f5f89ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:04:17.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.542+0000 7f5f89ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:04:17.549 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":202,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:04:17.551 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.546+0000 7f5f9ba56640 10 librados: watch_flush enter 2024-09-16T04:04:17.551 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.546+0000 7f5f9ba56640 10 librados: watch_flush exit 2024-09-16T04:04:17.551 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.546+0000 7f5f9ba56640 20 client.984102.objecter shutdown clearing up homeless session... 2024-09-16T04:04:17.551 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.546+0000 7f5f9ba56640 10 client.984102.objecter successfully canceled tick 2024-09-16T04:04:17.551 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.546+0000 7f5f9ba56640 1 -- 172.21.15.42:0/3255213260 >> v2:172.21.15.154:6812/254845342 conn(0x7f5f780391d0 msgr2=0x7f5f7803b690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:04:17.551 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.546+0000 7f5f9ba56640 1 --2- 172.21.15.42:0/3255213260 >> v2:172.21.15.154:6812/254845342 conn(0x7f5f780391d0 0x7f5f7803b690 secure :-1 s=READY pgs=3818 cs=0 l=1 rev1=1 crypto rx=0x7f5f90005630 tx=0x7f5f90009040 comp rx=0 tx=0).stop 2024-09-16T04:04:17.551 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.546+0000 7f5f9ba56640 10 monclient: shutdown 2024-09-16T04:04:17.552 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.546+0000 7f5f9ba56640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:04:17.552 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.546+0000 7f5f9ba56640 1 -- 172.21.15.42:0/3255213260 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5f80004810 msgr2=0x7f5f9411f400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:04:17.552 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.546+0000 7f5f9ba56640 1 --2- 172.21.15.42:0/3255213260 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5f80004810 0x7f5f9411f400 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f5f84006b20 tx=0x7f5f84018d20 comp rx=0 tx=0).stop 2024-09-16T04:04:17.552 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.546+0000 7f5f9ba56640 1 -- 172.21.15.42:0/3255213260 shutdown_connections 2024-09-16T04:04:17.552 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.546+0000 7f5f9ba56640 1 --2- 172.21.15.42:0/3255213260 >> v2:172.21.15.154:6812/254845342 conn(0x7f5f780391d0 0x7f5f7803b690 unknown :-1 s=CLOSED pgs=3818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:17.552 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.546+0000 7f5f9ba56640 1 --2- 172.21.15.42:0/3255213260 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5f80005220 0x7f5f9411f940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:17.552 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.546+0000 7f5f9ba56640 1 --2- 172.21.15.42:0/3255213260 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5f80004810 0x7f5f9411f400 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:17.552 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.546+0000 7f5f9ba56640 1 --2- 172.21.15.42:0/3255213260 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5f800039c0 0x7f5f94133c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:17.552 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.546+0000 7f5f9ba56640 1 -- 172.21.15.42:0/3255213260 >> 172.21.15.42:0/3255213260 conn(0x7f5f940a2420 msgr2=0x7f5f941385b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:04:17.552 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.546+0000 7f5f9ba56640 1 -- 172.21.15.42:0/3255213260 shutdown_connections 2024-09-16T04:04:17.552 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.546+0000 7f5f9ba56640 1 -- 172.21.15.42:0/3255213260 wait complete. 2024-09-16T04:04:17.552 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:17.546+0000 7f5f9ba56640 1 librados: shutdown 2024-09-16T04:04:20.569 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:04:20.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:20.698+0000 7f2e0b039640 10 monclient: get_monmap_and_config 2024-09-16T04:04:20.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:20.698+0000 7f2e0b039640 10 monclient: build_initial_monmap 2024-09-16T04:04:20.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:20.698+0000 7f2e0b039640 10 monclient: monmap: 2024-09-16T04:04:20.703 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:04:20.703 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:04:20.703 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:04:20.701749+0000 2024-09-16T04:04:20.703 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:04:20.701749+0000 2024-09-16T04:04:20.704 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:04:20.704 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:04:20.704 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:04:20.704 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:04:20.704 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:04:20.704 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:04:20.704 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:04:20.704 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:04:20.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:20.698+0000 7f2e0b039640 1 Processor -- start 2024-09-16T04:04:20.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:20.698+0000 7f2e0b039640 1 -- start start 2024-09-16T04:04:20.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:20.698+0000 7f2e0b039640 10 monclient: init 2024-09-16T04:04:20.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:20.698+0000 7f2e0b039640 10 monclient: _reopen_session rank -1 2024-09-16T04:04:20.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:20.698+0000 7f2e0b039640 10 monclient: _add_conns ranks=[3,0,1,4,2] 2024-09-16T04:04:20.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:20.698+0000 7f2e0b039640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f2e0406db90 0x7f2e041340c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:04:20.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:20.698+0000 7f2e0b039640 10 monclient(hunting): picked mon.noname-a con 0x7f2e0406db90 addr v2:172.21.15.42:3300/0 2024-09-16T04:04:20.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:20.698+0000 7f2e0b039640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2e040a37f0 addr v1:172.21.15.42:6789/0 2024-09-16T04:04:20.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:20.698+0000 7f2e0b039640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2e040a3c00 addr v1:172.21.15.154:6789/0 2024-09-16T04:04:20.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:20.698+0000 7f2e0b039640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2e0406ece0 con 0x7f2e040a37f0 2024-09-16T04:04:20.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:20.698+0000 7f2e0b039640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2e0409bf80 con 0x7f2e040a3c00 2024-09-16T04:04:20.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:20.698+0000 7f2e0b039640 10 monclient(hunting): start opening mon connection 2024-09-16T04:04:20.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:20.698+0000 7f2e0b039640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f2e0406df70 con 0x7f2e0406db90 2024-09-16T04:04:20.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:20.698+0000 7f2e0b039640 10 monclient(hunting): _renew_subs 2024-09-16T04:04:20.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:20.698+0000 7f2e0b039640 10 monclient(hunting): authenticate will time out at 2307.671288s 2024-09-16T04:04:20.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:20.698+0000 7f2e08dd5640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f2e0406db90 0x7f2e041340c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:20.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:20.698+0000 7f2e08dd5640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f2e0406db90 0x7f2e041340c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:46330/0 (socket says 172.21.15.42:46330) 2024-09-16T04:04:20.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:20.698+0000 7f2e08dd5640 1 -- 172.21.15.42:0/3126749755 learned_addr learned my addr 172.21.15.42:0/3126749755 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:04:20.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:20.698+0000 7f2e08dd5640 10 monclient(hunting): get_auth_request con 0x7f2e0406db90 auth_method 0 2024-09-16T04:04:20.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:20.698+0000 7f2e08dd5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:04:20.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:20.698+0000 7f2e08dd5640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:04:20.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:20.698+0000 7f2e08dd5640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:04:20.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:20.698+0000 7f2e08dd5640 1 -- 172.21.15.42:0/3126749755 >> v2:172.21.15.42:3300/0 conn(0x7f2e0406db90 msgr2=0x7f2e041340c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:04:20.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:20.698+0000 7f2e08dd5640 1 -- 172.21.15.42:0/3126749755 >> v2:172.21.15.42:3300/0 conn(0x7f2e0406db90 msgr2=0x7f2e041340c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:04:20.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:20.698+0000 7f2e08dd5640 1 --2- 172.21.15.42:0/3126749755 >> v2:172.21.15.42:3300/0 conn(0x7f2e0406db90 0x7f2e041340c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:04:20.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:20.698+0000 7f2e08dd5640 1 --2- 172.21.15.42:0/3126749755 >> v2:172.21.15.42:3300/0 conn(0x7f2e0406db90 0x7f2e041340c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:04:20.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:20.902+0000 7f2e08dd5640 1 --2- 172.21.15.42:0/3126749755 >> v2:172.21.15.42:3300/0 conn(0x7f2e0406db90 0x7f2e041340c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:20.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:20.902+0000 7f2e08dd5640 10 monclient(hunting): get_auth_request con 0x7f2e0406db90 auth_method 0 2024-09-16T04:04:20.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:20.902+0000 7f2e08dd5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:04:20.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:20.902+0000 7f2e08dd5640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:04:20.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:20.902+0000 7f2e08dd5640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:04:20.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:20.902+0000 7f2e08dd5640 1 -- 172.21.15.42:0/3126749755 >> v2:172.21.15.42:3300/0 conn(0x7f2e0406db90 msgr2=0x7f2e041340c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:04:20.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:20.902+0000 7f2e08dd5640 1 -- 172.21.15.42:0/3126749755 >> v2:172.21.15.42:3300/0 conn(0x7f2e0406db90 msgr2=0x7f2e041340c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:04:20.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:20.902+0000 7f2e08dd5640 1 --2- 172.21.15.42:0/3126749755 >> v2:172.21.15.42:3300/0 conn(0x7f2e0406db90 0x7f2e041340c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:04:20.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:20.902+0000 7f2e08dd5640 1 --2- 172.21.15.42:0/3126749755 >> v2:172.21.15.42:3300/0 conn(0x7f2e0406db90 0x7f2e041340c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T04:04:21.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:21.306+0000 7f2e08dd5640 1 --2- 172.21.15.42:0/3126749755 >> v2:172.21.15.42:3300/0 conn(0x7f2e0406db90 0x7f2e041340c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:21.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:21.306+0000 7f2e08dd5640 10 monclient(hunting): get_auth_request con 0x7f2e0406db90 auth_method 0 2024-09-16T04:04:21.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:21.306+0000 7f2e08dd5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:04:21.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:21.306+0000 7f2e08dd5640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:04:21.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:21.306+0000 7f2e08dd5640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:04:21.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:21.306+0000 7f2e08dd5640 1 -- 172.21.15.42:0/3126749755 >> v2:172.21.15.42:3300/0 conn(0x7f2e0406db90 msgr2=0x7f2e041340c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:04:21.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:21.306+0000 7f2e08dd5640 1 -- 172.21.15.42:0/3126749755 >> v2:172.21.15.42:3300/0 conn(0x7f2e0406db90 msgr2=0x7f2e041340c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:04:21.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:21.306+0000 7f2e08dd5640 1 --2- 172.21.15.42:0/3126749755 >> v2:172.21.15.42:3300/0 conn(0x7f2e0406db90 0x7f2e041340c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:04:21.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:21.306+0000 7f2e08dd5640 1 --2- 172.21.15.42:0/3126749755 >> v2:172.21.15.42:3300/0 conn(0x7f2e0406db90 0x7f2e041340c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T04:04:22.114 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:22.110+0000 7f2e08dd5640 1 --2- 172.21.15.42:0/3126749755 >> v2:172.21.15.42:3300/0 conn(0x7f2e0406db90 0x7f2e041340c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:22.114 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:22.110+0000 7f2e08dd5640 10 monclient(hunting): get_auth_request con 0x7f2e0406db90 auth_method 0 2024-09-16T04:04:22.114 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:22.110+0000 7f2e08dd5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:04:22.114 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:22.110+0000 7f2e08dd5640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:04:22.115 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:22.110+0000 7f2e08dd5640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:04:22.115 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:22.110+0000 7f2e08dd5640 1 -- 172.21.15.42:0/3126749755 >> v2:172.21.15.42:3300/0 conn(0x7f2e0406db90 msgr2=0x7f2e041340c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:04:22.115 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:22.110+0000 7f2e08dd5640 1 -- 172.21.15.42:0/3126749755 >> v2:172.21.15.42:3300/0 conn(0x7f2e0406db90 msgr2=0x7f2e041340c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:04:22.115 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:22.110+0000 7f2e08dd5640 1 --2- 172.21.15.42:0/3126749755 >> v2:172.21.15.42:3300/0 conn(0x7f2e0406db90 0x7f2e041340c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:04:22.115 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:22.110+0000 7f2e08dd5640 1 --2- 172.21.15.42:0/3126749755 >> v2:172.21.15.42:3300/0 conn(0x7f2e0406db90 0x7f2e041340c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T04:04:23.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.322+0000 7f2e037fe640 1 -- 172.21.15.42:0/3126749755 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1141413268 0 0) 0x7f2dfc001120 con 0x7f2e040a3c00 2024-09-16T04:04:23.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.322+0000 7f2e037fe640 10 monclient(hunting): _negotiate 2024-09-16T04:04:23.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.322+0000 7f2e037fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:04:23.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.322+0000 7f2e037fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:04:23.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.322+0000 7f2e037fe640 10 monclient(hunting): my global_id is 904777 2024-09-16T04:04:23.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.322+0000 7f2e037fe640 1 -- 172.21.15.42:0/3126749755 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2df0003530 con 0x7f2e040a3c00 2024-09-16T04:04:23.329 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.322+0000 7f2e037fe640 1 -- 172.21.15.42:0/3126749755 <== mon.1 v1:172.21.15.154:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1601085193 0 0) 0x7f2df0003530 con 0x7f2e040a3c00 2024-09-16T04:04:23.329 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.322+0000 7f2e037fe640 1 -- 172.21.15.42:0/3126749755 >> v2:172.21.15.42:3300/0 conn(0x7f2e0406db90 msgr2=0x7f2e041340c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:04:23.329 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.322+0000 7f2e037fe640 1 --2- 172.21.15.42:0/3126749755 >> v2:172.21.15.42:3300/0 conn(0x7f2e0406db90 0x7f2e041340c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:23.329 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.322+0000 7f2e037fe640 1 -- 172.21.15.42:0/3126749755 >> v1:172.21.15.42:6789/0 conn(0x7f2e040a37f0 legacy=0x7f2e04136e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:04:23.329 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.322+0000 7f2e037fe640 10 monclient: _finish_hunting 0 2024-09-16T04:04:23.329 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.322+0000 7f2e037fe640 1 monclient: found mon.noname-c-legacy 2024-09-16T04:04:23.329 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.322+0000 7f2e037fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:04:23.329 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.322+0000 7f2e037fe640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.154:6789/0 2024-09-16T04:04:23.329 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.322+0000 7f2e037fe640 1 -- 172.21.15.42:0/3126749755 --> v1:172.21.15.154:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e04137c30 con 0x7f2e040a3c00 2024-09-16T04:04:23.329 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.322+0000 7f2e037fe640 10 monclient: _finish_auth 0 2024-09-16T04:04:23.329 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.322+0000 7f2e037fe640 10 monclient: _check_auth_tickets 2024-09-16T04:04:23.329 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.322+0000 7f2e037fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:04:23.329 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.322+0000 7f2e037fe640 1 -- 172.21.15.42:0/3126749755 <== mon.1 v1:172.21.15.154:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2177413734 0 0) 0x7f2dfc001850 con 0x7f2e040a3c00 2024-09-16T04:04:23.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.322+0000 7f2e037fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:04:23.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.322+0000 7f2e037fe640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-16T04:04:23.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.322+0000 7f2e037fe640 10 monclient: dump: 2024-09-16T04:04:23.330 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:04:23.330 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:04:23.330 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:23.330 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:23.330 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:04:23.330 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:04:23.330 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:04:23.330 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:04:23.330 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:04:23.330 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:04:23.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.322+0000 7f2e037fe640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] but i'm connected to v1:172.21.15.154:6789/0, reconnecting 2024-09-16T04:04:23.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.322+0000 7f2e037fe640 10 monclient: _reopen_session rank -1 2024-09-16T04:04:23.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.322+0000 7f2e037fe640 1 -- 172.21.15.42:0/3126749755 >> v1:172.21.15.154:6789/0 conn(0x7f2e040a3c00 legacy=0x7f2e04137520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:04:23.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.322+0000 7f2e037fe640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:04:23.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.322+0000 7f2e037fe640 1 --2- 172.21.15.42:0/3126749755 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2df0003a10 0x7f2df0003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:04:23.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.322+0000 7f2e037fe640 10 monclient(hunting): picked mon.c con 0x7f2df0003a10 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:04:23.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.322+0000 7f2e037fe640 1 --2- 172.21.15.42:0/3126749755 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2df0004810 0x7f2df0004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:04:23.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.322+0000 7f2e037fe640 10 monclient(hunting): picked mon.b con 0x7f2df0004810 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:04:23.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.322+0000 7f2e037fe640 1 --2- 172.21.15.42:0/3126749755 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2df00053d0 0x7f2df000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:04:23.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.322+0000 7f2e037fe640 10 monclient(hunting): picked mon.a con 0x7f2df00053d0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:04:23.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.322+0000 7f2e037fe640 10 monclient(hunting): start opening mon connection 2024-09-16T04:04:23.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.322+0000 7f2e037fe640 1 -- 172.21.15.42:0/3126749755 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f2e0406df70 con 0x7f2df00053d0 2024-09-16T04:04:23.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.322+0000 7f2e037fe640 10 monclient(hunting): start opening mon connection 2024-09-16T04:04:23.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.322+0000 7f2e037fe640 1 -- 172.21.15.42:0/3126749755 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f2df000c2b0 con 0x7f2df0004810 2024-09-16T04:04:23.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.322+0000 7f2e037fe640 10 monclient(hunting): start opening mon connection 2024-09-16T04:04:23.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.322+0000 7f2e037fe640 1 -- 172.21.15.42:0/3126749755 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f2df000c4b0 con 0x7f2df0003a10 2024-09-16T04:04:23.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.322+0000 7f2e037fe640 10 monclient(hunting): _renew_subs 2024-09-16T04:04:23.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.322+0000 7f2e037fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:04:23.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.322+0000 7f2e08dd5640 1 --2- 172.21.15.42:0/3126749755 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2df0003a10 0x7f2df0003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:23.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.322+0000 7f2e095d6640 1 --2- 172.21.15.42:0/3126749755 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2df00053d0 0x7f2df000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:23.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.322+0000 7f2e08dd5640 10 monclient(hunting): get_auth_request con 0x7f2df0003a10 auth_method 0 2024-09-16T04:04:23.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.322+0000 7f2e08dd5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:04:23.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.322+0000 7f2e08dd5640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:04:23.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.322+0000 7f2e08dd5640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:04:23.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.322+0000 7f2e095d6640 10 monclient(hunting): get_auth_request con 0x7f2df00053d0 auth_method 0 2024-09-16T04:04:23.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.322+0000 7f2e095d6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:04:23.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.322+0000 7f2e095d6640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:04:23.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.322+0000 7f2e095d6640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:04:23.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.322+0000 7f2e03fff640 1 --2- 172.21.15.42:0/3126749755 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2df0004810 0x7f2df0004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:23.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.322+0000 7f2e08dd5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:04:23.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.322+0000 7f2e08dd5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:04:23.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.322+0000 7f2e08dd5640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:04:23.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.322+0000 7f2e03fff640 10 monclient(hunting): get_auth_request con 0x7f2df0004810 auth_method 0 2024-09-16T04:04:23.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.322+0000 7f2e03fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:04:23.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.322+0000 7f2e03fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:04:23.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.322+0000 7f2e03fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:04:23.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.322+0000 7f2e095d6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:04:23.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.322+0000 7f2e095d6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:04:23.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.322+0000 7f2e095d6640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:04:23.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.322+0000 7f2e03fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:04:23.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.322+0000 7f2e03fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:04:23.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.322+0000 7f2e03fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:04:23.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.322+0000 7f2e08dd5640 10 monclient(hunting): handle_auth_done global_id 904777 payload 293 2024-09-16T04:04:23.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.322+0000 7f2e08dd5640 1 -- 172.21.15.42:0/3126749755 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2df0004810 msgr2=0x7f2df0004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:04:23.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.322+0000 7f2e08dd5640 1 --2- 172.21.15.42:0/3126749755 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2df0004810 0x7f2df0004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:23.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.322+0000 7f2e08dd5640 1 -- 172.21.15.42:0/3126749755 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2df00053d0 msgr2=0x7f2df000b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:04:23.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.322+0000 7f2e08dd5640 1 --2- 172.21.15.42:0/3126749755 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2df00053d0 0x7f2df000b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:23.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.322+0000 7f2e08dd5640 10 monclient: _finish_hunting 0 2024-09-16T04:04:23.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.322+0000 7f2e08dd5640 1 monclient: found mon.c 2024-09-16T04:04:23.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.322+0000 7f2e08dd5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:04:23.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.322+0000 7f2e08dd5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:04:23.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.322+0000 7f2e095d6640 1 -- 172.21.15.42:0/3126749755 reap_dead start 2024-09-16T04:04:23.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.322+0000 7f2e08dd5640 1 -- 172.21.15.42:0/3126749755 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2df0003530 con 0x7f2df0003a10 2024-09-16T04:04:23.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.322+0000 7f2e08dd5640 10 monclient: _finish_auth 0 2024-09-16T04:04:23.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.322+0000 7f2e08dd5640 10 monclient: _check_auth_tickets 2024-09-16T04:04:23.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.322+0000 7f2e08dd5640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:04:23.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.322+0000 7f2e0b039640 5 monclient: authenticate success, global_id 904777 2024-09-16T04:04:23.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.322+0000 7f2e0b039640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:04:23.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.326+0000 7f2e08dd5640 1 --2- 172.21.15.42:0/3126749755 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2df0003a10 0x7f2df0003e30 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f2df40129f0 tx=0x7f2df401d270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9cc2d555837ed78e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:04:23.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.326+0000 7f2e037fe640 1 -- 172.21.15.42:0/3126749755 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2df40b3070 con 0x7f2df0003a10 2024-09-16T04:04:23.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.326+0000 7f2e037fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:04:23.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.326+0000 7f2e037fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:04:23.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.326+0000 7f2e037fe640 10 monclient: dump: 2024-09-16T04:04:23.339 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:04:23.339 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:04:23.339 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:23.339 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:23.339 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:04:23.339 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:04:23.339 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:04:23.339 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:04:23.339 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:04:23.339 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:04:23.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.326+0000 7f2e037fe640 1 -- 172.21.15.42:0/3126749755 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2df40ae410 con 0x7f2df0003a10 2024-09-16T04:04:23.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.326+0000 7f2e0b039640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:04:23.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.326+0000 7f2e037fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:04:23.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.326+0000 7f2e0b039640 10 monclient: get_monmap_and_config success 2024-09-16T04:04:23.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.326+0000 7f2e0b039640 4 set_mon_vals no callback set 2024-09-16T04:04:23.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.326+0000 7f2e037fe640 1 -- 172.21.15.42:0/3126749755 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2df400f040 con 0x7f2df0003a10 2024-09-16T04:04:23.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.326+0000 7f2e037fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:04:23.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.326+0000 7f2e037fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:04:23.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.326+0000 7f2e037fe640 10 monclient: dump: 2024-09-16T04:04:23.340 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:04:23.340 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:04:23.340 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:23.340 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:23.340 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:04:23.340 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:04:23.341 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:04:23.341 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:04:23.341 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:04:23.341 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:04:23.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.326+0000 7f2e0b039640 10 monclient: shutdown 2024-09-16T04:04:23.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.326+0000 7f2e0b039640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:04:23.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.326+0000 7f2e0b039640 1 -- 172.21.15.42:0/3126749755 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2df0003a10 msgr2=0x7f2df0003e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:04:23.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.326+0000 7f2e0b039640 1 --2- 172.21.15.42:0/3126749755 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2df0003a10 0x7f2df0003e30 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f2df40129f0 tx=0x7f2df401d270 comp rx=0 tx=0).stop 2024-09-16T04:04:23.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.326+0000 7f2e0b039640 1 -- 172.21.15.42:0/3126749755 shutdown_connections 2024-09-16T04:04:23.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.326+0000 7f2e0b039640 1 --2- 172.21.15.42:0/3126749755 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2df0003a10 0x7f2df0003e30 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:23.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.326+0000 7f2e0b039640 1 -- 172.21.15.42:0/3126749755 >> 172.21.15.42:0/3126749755 conn(0x7f2e040a9fb0 msgr2=0x7f2e040aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:04:23.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.326+0000 7f2e0b039640 1 -- 172.21.15.42:0/3126749755 shutdown_connections 2024-09-16T04:04:23.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.326+0000 7f2e0b039640 1 -- 172.21.15.42:0/3126749755 wait complete. 2024-09-16T04:04:23.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.330+0000 7f2e0b039640 10 monclient: build_initial_monmap 2024-09-16T04:04:23.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.330+0000 7f2e0b039640 10 monclient: monmap: 2024-09-16T04:04:23.342 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:04:23.342 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:04:23.342 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:04:23.342 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:04:23.342 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:04:23.342 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:04:23.342 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:04:23.342 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:04:23.342 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:04:23.342 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:04:23.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.330+0000 7f2e0b039640 1 librados: starting msgr at 2024-09-16T04:04:23.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.330+0000 7f2e0b039640 1 librados: starting objecter 2024-09-16T04:04:23.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.330+0000 7f2e0b039640 1 Processor -- start 2024-09-16T04:04:23.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.330+0000 7f2e0b039640 1 -- start start 2024-09-16T04:04:23.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.330+0000 7f2e0b039640 1 librados: setting wanted keys 2024-09-16T04:04:23.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.330+0000 7f2e0b039640 1 librados: calling monclient init 2024-09-16T04:04:23.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.330+0000 7f2e0b039640 10 monclient: init 2024-09-16T04:04:23.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.330+0000 7f2e0b039640 10 monclient: _reopen_session rank -1 2024-09-16T04:04:23.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.330+0000 7f2e0b039640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:04:23.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.330+0000 7f2e0b039640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2df0003a10 0x7f2e040a2510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:04:23.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.330+0000 7f2e0b039640 10 monclient(hunting): picked mon.noname-a con 0x7f2df0003a10 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:04:23.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.330+0000 7f2e0b039640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2e041304a0 0x7f2e04130880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:04:23.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.330+0000 7f2e0b039640 10 monclient(hunting): picked mon.noname-b con 0x7f2e041304a0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:04:23.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.330+0000 7f2e0b039640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2e04120200 0x7f2e04120680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:04:23.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.330+0000 7f2e0b039640 10 monclient(hunting): picked mon.noname-c con 0x7f2e04120200 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:04:23.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.330+0000 7f2e0b039640 10 monclient(hunting): start opening mon connection 2024-09-16T04:04:23.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.330+0000 7f2e0b039640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f2e04137620 con 0x7f2df0003a10 2024-09-16T04:04:23.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.330+0000 7f2e0b039640 10 monclient(hunting): start opening mon connection 2024-09-16T04:04:23.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.330+0000 7f2e0b039640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f2e0406ea70 con 0x7f2e041304a0 2024-09-16T04:04:23.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.330+0000 7f2e0b039640 10 monclient(hunting): start opening mon connection 2024-09-16T04:04:23.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.330+0000 7f2e0b039640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f2e041377a0 con 0x7f2e04120200 2024-09-16T04:04:23.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.330+0000 7f2e0b039640 10 monclient(hunting): _renew_subs 2024-09-16T04:04:23.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.330+0000 7f2e0b039640 10 monclient(hunting): authenticate will time out at 2310.302222s 2024-09-16T04:04:23.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.330+0000 7f2e095d6640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2e04120200 0x7f2e04120680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:23.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.330+0000 7f2e095d6640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2e04120200 0x7f2e04120680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:48740/0 (socket says 172.21.15.42:48740) 2024-09-16T04:04:23.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.330+0000 7f2e095d6640 1 -- 172.21.15.42:0/1552318376 learned_addr learned my addr 172.21.15.42:0/1552318376 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:04:23.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.330+0000 7f2e095d6640 10 monclient(hunting): get_auth_request con 0x7f2e04120200 auth_method 0 2024-09-16T04:04:23.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.330+0000 7f2e095d6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:04:23.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.330+0000 7f2e095d6640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:04:23.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.330+0000 7f2e095d6640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:04:23.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.330+0000 7f2e08dd5640 1 --2- 172.21.15.42:0/1552318376 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2df0003a10 0x7f2e040a2510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:23.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.330+0000 7f2e08dd5640 10 monclient(hunting): get_auth_request con 0x7f2df0003a10 auth_method 0 2024-09-16T04:04:23.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.330+0000 7f2e08dd5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:04:23.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.330+0000 7f2e08dd5640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:04:23.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.330+0000 7f2e08dd5640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:04:23.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.330+0000 7f2e095d6640 1 -- 172.21.15.42:0/1552318376 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2e04120200 msgr2=0x7f2e04120680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-16T04:04:23.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.330+0000 7f2e095d6640 1 -- 172.21.15.42:0/1552318376 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2e04120200 msgr2=0x7f2e04120680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:04:23.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.330+0000 7f2e095d6640 1 --2- 172.21.15.42:0/1552318376 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2e04120200 0x7f2e04120680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:04:23.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.330+0000 7f2e03fff640 1 --2- 172.21.15.42:0/1552318376 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2e041304a0 0x7f2e04130880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:23.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.330+0000 7f2e095d6640 1 --2- 172.21.15.42:0/1552318376 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2e04120200 0x7f2e04120680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:04:23.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.330+0000 7f2e08dd5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:04:23.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.330+0000 7f2e08dd5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:04:23.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.330+0000 7f2e08dd5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:04:23.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.330+0000 7f2e03fff640 10 monclient(hunting): get_auth_request con 0x7f2e041304a0 auth_method 0 2024-09-16T04:04:23.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.330+0000 7f2e03fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:04:23.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.330+0000 7f2e03fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:04:23.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.330+0000 7f2e03fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:04:23.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.330+0000 7f2e03fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:04:23.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.330+0000 7f2e03fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:04:23.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.330+0000 7f2e03fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:04:23.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.330+0000 7f2e08dd5640 10 monclient(hunting): handle_auth_done global_id 984105 payload 824 2024-09-16T04:04:23.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.330+0000 7f2e08dd5640 1 -- 172.21.15.42:0/1552318376 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2e04120200 msgr2=0x7f2e04120680 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:04:23.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.330+0000 7f2e08dd5640 1 --2- 172.21.15.42:0/1552318376 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2e04120200 0x7f2e04120680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:23.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.330+0000 7f2e08dd5640 1 -- 172.21.15.42:0/1552318376 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2e041304a0 msgr2=0x7f2e04130880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:04:23.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.330+0000 7f2e08dd5640 1 --2- 172.21.15.42:0/1552318376 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2e041304a0 0x7f2e04130880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:23.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.330+0000 7f2e08dd5640 10 monclient: _finish_hunting 0 2024-09-16T04:04:23.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.330+0000 7f2e08dd5640 1 monclient: found mon.noname-a 2024-09-16T04:04:23.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.330+0000 7f2e08dd5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:04:23.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.330+0000 7f2e08dd5640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:04:23.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.330+0000 7f2e08dd5640 1 -- 172.21.15.42:0/1552318376 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e041cf740 con 0x7f2df0003a10 2024-09-16T04:04:23.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.330+0000 7f2e08dd5640 1 --2- 172.21.15.42:0/1552318376 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2df0003a10 0x7f2e040a2510 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f2df40ad000 tx=0x7f2df0005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:04:23.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.330+0000 7f2e017fa640 10 client.?.objecter ms_handle_connect 0x7f2df0003a10 2024-09-16T04:04:23.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.330+0000 7f2e017fa640 10 client.?.objecter resend_mon_ops 2024-09-16T04:04:23.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.370+0000 7f2e017fa640 1 -- 172.21.15.42:0/1552318376 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2df40bd020 con 0x7f2df0003a10 2024-09-16T04:04:23.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.370+0000 7f2e017fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:04:23.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.370+0000 7f2e017fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:04:23.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.370+0000 7f2e017fa640 10 monclient: dump: 2024-09-16T04:04:23.376 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:04:23.377 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:04:23.377 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:23.377 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:23.377 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:04:23.377 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:04:23.377 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:04:23.377 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:04:23.377 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:04:23.377 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:04:23.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.370+0000 7f2e017fa640 10 monclient: _finish_auth 0 2024-09-16T04:04:23.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.370+0000 7f2e017fa640 10 monclient: _check_auth_tickets 2024-09-16T04:04:23.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.370+0000 7f2e017fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:04:23.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.370+0000 7f2e0b039640 5 monclient: authenticate success, global_id 984105 2024-09-16T04:04:23.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.370+0000 7f2e0b039640 10 monclient: _renew_subs 2024-09-16T04:04:23.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.370+0000 7f2e0b039640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:04:23.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.370+0000 7f2e0b039640 1 -- 172.21.15.42:0/1552318376 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e04122eb0 con 0x7f2df0003a10 2024-09-16T04:04:23.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.370+0000 7f2e0b039640 10 client.984105.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:04:23.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.370+0000 7f2e0b039640 10 monclient: _renew_subs 2024-09-16T04:04:23.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.370+0000 7f2e0b039640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:04:23.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.370+0000 7f2e0b039640 1 -- 172.21.15.42:0/1552318376 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e0406f6a0 con 0x7f2df0003a10 2024-09-16T04:04:23.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.370+0000 7f2e0b039640 1 librados: init done 2024-09-16T04:04:23.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.370+0000 7f2e017fa640 1 -- 172.21.15.42:0/1552318376 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2df40ae410 con 0x7f2df0003a10 2024-09-16T04:04:23.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.370+0000 7f2e017fa640 10 monclient: handle_config config(0 keys) 2024-09-16T04:04:23.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.370+0000 7f2e017fa640 1 -- 172.21.15.42:0/1552318376 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2df40b3040 con 0x7f2df0003a10 2024-09-16T04:04:23.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.370+0000 7f2e017fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:04:23.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.370+0000 7f2e017fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:04:23.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.370+0000 7f2e02ffd640 4 set_mon_vals no callback set 2024-09-16T04:04:23.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.370+0000 7f2e017fa640 10 monclient: dump: 2024-09-16T04:04:23.379 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:04:23.379 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:04:23.379 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:23.379 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:23.379 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:04:23.379 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:04:23.379 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:04:23.379 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:04:23.379 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:04:23.379 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:04:23.380 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.370+0000 7f2e0b039640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:04:23.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.374+0000 7f2e0b039640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:04:23.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.374+0000 7f2e0b039640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:04:23.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.374+0000 7f2e0b039640 1 -- 172.21.15.42:0/1552318376 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2de8005090 con 0x7f2df0003a10 2024-09-16T04:04:23.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.374+0000 7f2e017fa640 1 -- 172.21.15.42:0/1552318376 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f2df40c0020 con 0x7f2df0003a10 2024-09-16T04:04:23.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.374+0000 7f2e017fa640 1 --2- 172.21.15.42:0/1552318376 >> v2:172.21.15.154:6812/254845342 conn(0x7f2de4037810 0x7f2de4039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:04:23.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.374+0000 7f2e017fa640 1 -- 172.21.15.42:0/1552318376 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(675..675 src has 451..675) ==== 5899+0+0 (secure 0 0 0) 0x7f2df40e3f60 con 0x7f2df0003a10 2024-09-16T04:04:23.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.374+0000 7f2e017fa640 10 client.984105.objecter ms_dispatch 0x7f2e04000ba0 osd_map(675..675 src has 451..675) 2024-09-16T04:04:23.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.374+0000 7f2e017fa640 3 client.984105.objecter handle_osd_map got epochs [675,675] > 0 2024-09-16T04:04:23.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.374+0000 7f2e017fa640 3 client.984105.objecter handle_osd_map decoding full epoch 675 2024-09-16T04:04:23.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.374+0000 7f2e017fa640 20 client.984105.objecter dump_active .. 0 homeless 2024-09-16T04:04:23.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.374+0000 7f2e03fff640 1 --2- 172.21.15.42:0/1552318376 >> v2:172.21.15.154:6812/254845342 conn(0x7f2de4037810 0x7f2de4039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:23.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.374+0000 7f2e03fff640 10 monclient: get_auth_request con 0x7f2de4037810 auth_method 0 2024-09-16T04:04:23.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.374+0000 7f2e03fff640 1 --2- 172.21.15.42:0/1552318376 >> v2:172.21.15.154:6812/254845342 conn(0x7f2de4037810 0x7f2de4039cd0 secure :-1 s=READY pgs=3819 cs=0 l=1 rev1=1 crypto rx=0x7f2e040a2da0 tx=0x7f2dec009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:04:23.382 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.374+0000 7f2e017fa640 10 client.984105.objecter ms_handle_connect 0x7f2de4037810 2024-09-16T04:04:23.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.386+0000 7f2e017fa640 1 -- 172.21.15.42:0/1552318376 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f2df40c7030 con 0x7f2df0003a10 2024-09-16T04:04:23.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.386+0000 7f2e017fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:04:23.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.386+0000 7f2e017fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:04:23.637 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.630+0000 7f2e0b039640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:04:23.637 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.630+0000 7f2e0b039640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:04:23.637 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.630+0000 7f2e0b039640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:04:23.638 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.630+0000 7f2e0b039640 1 -- 172.21.15.42:0/1552318376 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f2de8005380 con 0x7f2df0003a10 2024-09-16T04:04:23.638 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.634+0000 7f2e017fa640 1 -- 172.21.15.42:0/1552318376 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1555 (secure 0 0 0) 0x7f2e041cf740 con 0x7f2df0003a10 2024-09-16T04:04:23.638 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.634+0000 7f2e017fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:04:23.638 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.634+0000 7f2e017fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:04:23.638 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":206,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:04:23.641 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.634+0000 7f2e0b039640 10 librados: watch_flush enter 2024-09-16T04:04:23.641 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.634+0000 7f2e0b039640 10 librados: watch_flush exit 2024-09-16T04:04:23.641 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.634+0000 7f2e0b039640 20 client.984105.objecter shutdown clearing up homeless session... 2024-09-16T04:04:23.641 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.634+0000 7f2e0b039640 10 client.984105.objecter successfully canceled tick 2024-09-16T04:04:23.641 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.634+0000 7f2e0b039640 1 -- 172.21.15.42:0/1552318376 >> v2:172.21.15.154:6812/254845342 conn(0x7f2de4037810 msgr2=0x7f2de4039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:04:23.642 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.634+0000 7f2e0b039640 1 --2- 172.21.15.42:0/1552318376 >> v2:172.21.15.154:6812/254845342 conn(0x7f2de4037810 0x7f2de4039cd0 secure :-1 s=READY pgs=3819 cs=0 l=1 rev1=1 crypto rx=0x7f2e040a2da0 tx=0x7f2dec009040 comp rx=0 tx=0).stop 2024-09-16T04:04:23.642 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.634+0000 7f2e0b039640 10 monclient: shutdown 2024-09-16T04:04:23.642 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.634+0000 7f2e0b039640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:04:23.642 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.634+0000 7f2e0b039640 1 -- 172.21.15.42:0/1552318376 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2df0003a10 msgr2=0x7f2e040a2510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:04:23.642 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.634+0000 7f2e0b039640 1 --2- 172.21.15.42:0/1552318376 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2df0003a10 0x7f2e040a2510 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f2df40ad000 tx=0x7f2df0005840 comp rx=0 tx=0).stop 2024-09-16T04:04:23.642 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.634+0000 7f2e0b039640 1 -- 172.21.15.42:0/1552318376 shutdown_connections 2024-09-16T04:04:23.642 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.634+0000 7f2e0b039640 1 --2- 172.21.15.42:0/1552318376 >> v2:172.21.15.154:6812/254845342 conn(0x7f2de4037810 0x7f2de4039cd0 unknown :-1 s=CLOSED pgs=3819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:23.642 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.634+0000 7f2e0b039640 1 --2- 172.21.15.42:0/1552318376 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2e04120200 0x7f2e04120680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:23.642 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.634+0000 7f2e0b039640 1 --2- 172.21.15.42:0/1552318376 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2e041304a0 0x7f2e04130880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:23.642 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.634+0000 7f2e0b039640 1 --2- 172.21.15.42:0/1552318376 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2df0003a10 0x7f2e040a2510 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:23.642 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.634+0000 7f2e0b039640 1 -- 172.21.15.42:0/1552318376 >> 172.21.15.42:0/1552318376 conn(0x7f2e040a9fb0 msgr2=0x7f2e04131790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:04:23.643 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.634+0000 7f2e0b039640 1 -- 172.21.15.42:0/1552318376 shutdown_connections 2024-09-16T04:04:23.643 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.634+0000 7f2e0b039640 1 -- 172.21.15.42:0/1552318376 wait complete. 2024-09-16T04:04:23.643 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:23.634+0000 7f2e0b039640 1 librados: shutdown 2024-09-16T04:04:26.659 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:04:26.795 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.790+0000 7ff9c2b36640 10 monclient: get_monmap_and_config 2024-09-16T04:04:26.795 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.790+0000 7ff9c2b36640 10 monclient: build_initial_monmap 2024-09-16T04:04:26.795 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.790+0000 7ff9c2b36640 10 monclient: monmap: 2024-09-16T04:04:26.795 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:04:26.795 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:04:26.795 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:04:26.793497+0000 2024-09-16T04:04:26.795 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:04:26.793497+0000 2024-09-16T04:04:26.796 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:04:26.796 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:04:26.796 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:04:26.796 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:04:26.796 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:04:26.796 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:04:26.796 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:04:26.796 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:04:26.798 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.790+0000 7ff9c2b36640 1 Processor -- start 2024-09-16T04:04:26.798 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.790+0000 7ff9c2b36640 1 -- start start 2024-09-16T04:04:26.798 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.790+0000 7ff9c2b36640 10 monclient: init 2024-09-16T04:04:26.798 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.790+0000 7ff9c2b36640 10 monclient: _reopen_session rank -1 2024-09-16T04:04:26.799 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.790+0000 7ff9c2b36640 10 monclient: _add_conns ranks=[1,2,4,3,0] 2024-09-16T04:04:26.799 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.790+0000 7ff9c2b36640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff9bc0a1330 addr v1:172.21.15.154:6789/0 2024-09-16T04:04:26.799 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.790+0000 7ff9c2b36640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff9bc06dac0 0x7ff9bc13a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:04:26.799 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.790+0000 7ff9c2b36640 10 monclient(hunting): picked mon.noname-b con 0x7ff9bc06dac0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:04:26.799 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.790+0000 7ff9c2b36640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7ff9bc0a1e20 0x7ff9bc13c890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:04:26.799 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.790+0000 7ff9c2b36640 10 monclient(hunting): picked mon.noname-c con 0x7ff9bc0a1e20 addr v2:172.21.15.154:3300/0 2024-09-16T04:04:26.799 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.790+0000 7ff9c2b36640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff9bc09c7d0 con 0x7ff9bc0a1330 2024-09-16T04:04:26.799 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.790+0000 7ff9c2b36640 10 monclient(hunting): start opening mon connection 2024-09-16T04:04:26.799 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.790+0000 7ff9c2b36640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7ff9bc09c090 con 0x7ff9bc0a1e20 2024-09-16T04:04:26.799 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.790+0000 7ff9c2b36640 10 monclient(hunting): start opening mon connection 2024-09-16T04:04:26.799 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.790+0000 7ff9c2b36640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7ff9bc09f610 con 0x7ff9bc06dac0 2024-09-16T04:04:26.800 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.790+0000 7ff9c2b36640 10 monclient(hunting): _renew_subs 2024-09-16T04:04:26.800 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.790+0000 7ff9c2b36640 10 monclient(hunting): authenticate will time out at 2313.763168s 2024-09-16T04:04:26.800 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.790+0000 7ff9bbfff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff9bc06dac0 0x7ff9bc13a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:26.800 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.790+0000 7ff9bbfff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff9bc06dac0 0x7ff9bc13a340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:52312/0 (socket says 172.21.15.42:52312) 2024-09-16T04:04:26.800 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.790+0000 7ff9bbfff640 1 -- 172.21.15.42:0/2369103327 learned_addr learned my addr 172.21.15.42:0/2369103327 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:04:26.800 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.790+0000 7ff9bbfff640 10 monclient(hunting): get_auth_request con 0x7ff9bc06dac0 auth_method 0 2024-09-16T04:04:26.800 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.790+0000 7ff9bbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:04:26.800 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.790+0000 7ff9bbfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:04:26.800 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.790+0000 7ff9bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:04:26.800 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.790+0000 7ff9c10d3640 1 --2- 172.21.15.42:0/2369103327 >> v2:172.21.15.154:3300/0 conn(0x7ff9bc0a1e20 0x7ff9bc13c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:26.800 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.790+0000 7ff9c10d3640 10 monclient(hunting): get_auth_request con 0x7ff9bc0a1e20 auth_method 0 2024-09-16T04:04:26.800 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.790+0000 7ff9c10d3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:04:26.801 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.790+0000 7ff9c10d3640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:04:26.801 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.790+0000 7ff9c10d3640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:04:26.801 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.790+0000 7ff9bbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:04:26.801 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.790+0000 7ff9bbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:04:26.801 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.790+0000 7ff9bbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:04:26.801 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.790+0000 7ff9c10d3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:04:26.801 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.790+0000 7ff9c10d3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:04:26.801 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.790+0000 7ff9c10d3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:04:26.801 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.790+0000 7ff9bbfff640 10 monclient(hunting): handle_auth_done global_id 974186 payload 274 2024-09-16T04:04:26.801 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.790+0000 7ff9bbfff640 1 -- 172.21.15.42:0/2369103327 >> v2:172.21.15.154:3300/0 conn(0x7ff9bc0a1e20 msgr2=0x7ff9bc13c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:04:26.801 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.790+0000 7ff9bbfff640 1 --2- 172.21.15.42:0/2369103327 >> v2:172.21.15.154:3300/0 conn(0x7ff9bc0a1e20 0x7ff9bc13c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:26.801 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.790+0000 7ff9bbfff640 1 -- 172.21.15.42:0/2369103327 >> v1:172.21.15.154:6789/0 conn(0x7ff9bc0a1330 legacy=0x7ff9bc0a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:04:26.801 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.790+0000 7ff9bbfff640 10 monclient: _finish_hunting 0 2024-09-16T04:04:26.802 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.790+0000 7ff9bbfff640 1 monclient: found mon.noname-b 2024-09-16T04:04:26.802 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.790+0000 7ff9bbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:04:26.802 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.790+0000 7ff9bbfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T04:04:26.802 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.790+0000 7ff9bbfff640 1 -- 172.21.15.42:0/2369103327 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9bc09f8f0 con 0x7ff9bc06dac0 2024-09-16T04:04:26.807 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.794+0000 7ff9bbfff640 1 --2- 172.21.15.42:0/2369103327 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff9bc06dac0 0x7ff9bc13a340 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7ff9a4009870 tx=0x7ff9a401c510 comp rx=0 tx=0).ready entity=mon.2 client_cookie=73c5afe72029ccc5 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:04:26.807 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.794+0000 7ff9bb7fe640 1 -- 172.21.15.42:0/2369103327 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9a40a95f0 con 0x7ff9bc06dac0 2024-09-16T04:04:26.807 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.794+0000 7ff9bb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:04:26.808 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.794+0000 7ff9bb7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:04:26.808 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.794+0000 7ff9bb7fe640 10 monclient: dump: 2024-09-16T04:04:26.808 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:04:26.808 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:04:26.808 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:26.808 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:26.808 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:04:26.808 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:04:26.808 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:04:26.808 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:04:26.808 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:04:26.808 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:04:26.808 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.794+0000 7ff9bb7fe640 10 monclient: _finish_auth 0 2024-09-16T04:04:26.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.794+0000 7ff9bb7fe640 10 monclient: _check_auth_tickets 2024-09-16T04:04:26.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.794+0000 7ff9bb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:04:26.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.794+0000 7ff9c2b36640 5 monclient: authenticate success, global_id 974186 2024-09-16T04:04:26.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.794+0000 7ff9c2b36640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:04:26.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.794+0000 7ff9bb7fe640 1 -- 172.21.15.42:0/2369103327 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff9a40a9790 con 0x7ff9bc06dac0 2024-09-16T04:04:26.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.794+0000 7ff9bb7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:04:26.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.794+0000 7ff9c2b36640 10 monclient: get_monmap_and_config success 2024-09-16T04:04:26.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.794+0000 7ff9c2b36640 4 set_mon_vals no callback set 2024-09-16T04:04:26.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.794+0000 7ff9bb7fe640 1 -- 172.21.15.42:0/2369103327 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9a40a9b60 con 0x7ff9bc06dac0 2024-09-16T04:04:26.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.794+0000 7ff9bb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:04:26.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.794+0000 7ff9bb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:04:26.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.794+0000 7ff9bb7fe640 10 monclient: dump: 2024-09-16T04:04:26.809 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:04:26.810 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:04:26.810 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:26.810 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:26.810 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:04:26.810 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:04:26.810 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:04:26.810 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:04:26.810 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:04:26.811 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:04:26.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.794+0000 7ff9c2b36640 10 monclient: shutdown 2024-09-16T04:04:26.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.794+0000 7ff9c2b36640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:04:26.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.794+0000 7ff9c2b36640 1 -- 172.21.15.42:0/2369103327 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff9bc06dac0 msgr2=0x7ff9bc13a340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:04:26.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.794+0000 7ff9c2b36640 1 --2- 172.21.15.42:0/2369103327 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff9bc06dac0 0x7ff9bc13a340 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7ff9a4009870 tx=0x7ff9a401c510 comp rx=0 tx=0).stop 2024-09-16T04:04:26.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.794+0000 7ff9c2b36640 1 -- 172.21.15.42:0/2369103327 shutdown_connections 2024-09-16T04:04:26.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.794+0000 7ff9c2b36640 1 --2- 172.21.15.42:0/2369103327 >> v2:172.21.15.154:3300/0 conn(0x7ff9bc0a1e20 0x7ff9bc13c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:26.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.794+0000 7ff9c2b36640 1 --2- 172.21.15.42:0/2369103327 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff9bc06dac0 0x7ff9bc13a340 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:26.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.794+0000 7ff9c2b36640 1 -- 172.21.15.42:0/2369103327 >> 172.21.15.42:0/2369103327 conn(0x7ff9bc0a2420 msgr2=0x7ff9bc0a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:04:26.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.794+0000 7ff9c2b36640 1 -- 172.21.15.42:0/2369103327 shutdown_connections 2024-09-16T04:04:26.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.794+0000 7ff9c2b36640 1 -- 172.21.15.42:0/2369103327 wait complete. 2024-09-16T04:04:26.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.794+0000 7ff9c2b36640 10 monclient: build_initial_monmap 2024-09-16T04:04:26.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.798+0000 7ff9c2b36640 10 monclient: monmap: 2024-09-16T04:04:26.811 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:04:26.812 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:04:26.812 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:04:26.812 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:04:26.812 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:04:26.812 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:04:26.812 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:04:26.812 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:04:26.812 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:04:26.812 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:04:26.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.798+0000 7ff9c2b36640 1 librados: starting msgr at 2024-09-16T04:04:26.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.798+0000 7ff9c2b36640 1 librados: starting objecter 2024-09-16T04:04:26.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.798+0000 7ff9c2b36640 1 Processor -- start 2024-09-16T04:04:26.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.798+0000 7ff9c2b36640 1 -- start start 2024-09-16T04:04:26.813 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.798+0000 7ff9c2b36640 1 librados: setting wanted keys 2024-09-16T04:04:26.813 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.798+0000 7ff9c2b36640 1 librados: calling monclient init 2024-09-16T04:04:26.813 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.798+0000 7ff9c2b36640 10 monclient: init 2024-09-16T04:04:26.813 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.798+0000 7ff9c2b36640 10 monclient: _reopen_session rank -1 2024-09-16T04:04:26.813 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.798+0000 7ff9c2b36640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:04:26.813 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.798+0000 7ff9c2b36640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff9bc0a1330 0x7ff9bc1ccb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:04:26.813 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.798+0000 7ff9c2b36640 10 monclient(hunting): picked mon.noname-b con 0x7ff9bc0a1330 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:04:26.813 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.798+0000 7ff9c2b36640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff9bc0a1e20 0x7ff9bc132320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:04:26.813 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.798+0000 7ff9c2b36640 10 monclient(hunting): picked mon.noname-c con 0x7ff9bc0a1e20 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:04:26.813 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.798+0000 7ff9c2b36640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff9bc1392c0 0x7ff9bc139770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:04:26.813 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.798+0000 7ff9c2b36640 10 monclient(hunting): picked mon.noname-a con 0x7ff9bc1392c0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:04:26.813 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.798+0000 7ff9c2b36640 10 monclient(hunting): start opening mon connection 2024-09-16T04:04:26.813 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.798+0000 7ff9c2b36640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7ff9bc13f770 con 0x7ff9bc1392c0 2024-09-16T04:04:26.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.798+0000 7ff9c2b36640 10 monclient(hunting): start opening mon connection 2024-09-16T04:04:26.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.798+0000 7ff9c2b36640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff9bc13f5f0 con 0x7ff9bc0a1330 2024-09-16T04:04:26.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.798+0000 7ff9c2b36640 10 monclient(hunting): start opening mon connection 2024-09-16T04:04:26.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.798+0000 7ff9c2b36640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7ff9bc13f8f0 con 0x7ff9bc0a1e20 2024-09-16T04:04:26.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.798+0000 7ff9c2b36640 10 monclient(hunting): _renew_subs 2024-09-16T04:04:26.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.798+0000 7ff9c2b36640 10 monclient(hunting): authenticate will time out at 2313.769316s 2024-09-16T04:04:26.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.798+0000 7ff9c10d3640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff9bc1392c0 0x7ff9bc139770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:26.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.798+0000 7ff9bbfff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff9bc0a1e20 0x7ff9bc132320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:26.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.798+0000 7ff9bbfff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff9bc0a1e20 0x7ff9bc132320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:52316/0 (socket says 172.21.15.42:52316) 2024-09-16T04:04:26.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.798+0000 7ff9c10d3640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff9bc1392c0 0x7ff9bc139770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:39584/0 (socket says 172.21.15.42:39584) 2024-09-16T04:04:26.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.798+0000 7ff9bbfff640 1 -- 172.21.15.42:0/453178422 learned_addr learned my addr 172.21.15.42:0/453178422 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:04:26.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.798+0000 7ff9bbfff640 10 monclient(hunting): get_auth_request con 0x7ff9bc0a1e20 auth_method 0 2024-09-16T04:04:26.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.798+0000 7ff9bbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:04:26.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.798+0000 7ff9bbfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:04:26.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.798+0000 7ff9bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:04:26.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.798+0000 7ff9c10d3640 10 monclient(hunting): get_auth_request con 0x7ff9bc1392c0 auth_method 0 2024-09-16T04:04:26.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.798+0000 7ff9c10d3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:04:26.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.798+0000 7ff9c10d3640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:04:26.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.798+0000 7ff9c10d3640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:04:26.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.798+0000 7ff9bbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:04:26.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.798+0000 7ff9bbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:04:26.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.798+0000 7ff9bbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:04:26.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.798+0000 7ff9c10d3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:04:26.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.798+0000 7ff9c10d3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:04:26.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.798+0000 7ff9c10d3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:04:26.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.798+0000 7ff9c08d2640 1 --2- 172.21.15.42:0/453178422 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff9bc0a1330 0x7ff9bc1ccb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:26.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.798+0000 7ff9c08d2640 10 monclient(hunting): get_auth_request con 0x7ff9bc0a1330 auth_method 0 2024-09-16T04:04:26.816 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.798+0000 7ff9c08d2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:04:26.816 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.798+0000 7ff9c08d2640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:04:26.816 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.798+0000 7ff9c08d2640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:04:26.816 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.798+0000 7ff9c10d3640 10 monclient(hunting): handle_auth_done global_id 984108 payload 824 2024-09-16T04:04:26.816 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.798+0000 7ff9c10d3640 1 -- 172.21.15.42:0/453178422 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff9bc0a1e20 msgr2=0x7ff9bc132320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:04:26.816 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.798+0000 7ff9c10d3640 1 --2- 172.21.15.42:0/453178422 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff9bc0a1e20 0x7ff9bc132320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:26.816 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.798+0000 7ff9c10d3640 1 -- 172.21.15.42:0/453178422 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff9bc0a1330 msgr2=0x7ff9bc1ccb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:04:26.816 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.798+0000 7ff9c10d3640 1 --2- 172.21.15.42:0/453178422 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff9bc0a1330 0x7ff9bc1ccb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:26.816 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.798+0000 7ff9c10d3640 10 monclient: _finish_hunting 0 2024-09-16T04:04:26.816 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.798+0000 7ff9c10d3640 1 monclient: found mon.noname-a 2024-09-16T04:04:26.816 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.798+0000 7ff9c10d3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:04:26.816 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.798+0000 7ff9c10d3640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:04:26.816 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.798+0000 7ff9c10d3640 1 -- 172.21.15.42:0/453178422 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9bc11c6f0 con 0x7ff9bc1392c0 2024-09-16T04:04:26.817 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.798+0000 7ff9bbfff640 1 --2- 172.21.15.42:0/453178422 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff9bc0a1e20 0x7ff9bc132320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:04:26.817 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.798+0000 7ff9c10d3640 1 --2- 172.21.15.42:0/453178422 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff9bc1392c0 0x7ff9bc139770 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7ff9b400e7d0 tx=0x7ff9b400ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:04:26.817 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.798+0000 7ff9b97fa640 10 client.?.objecter ms_handle_connect 0x7ff9bc1392c0 2024-09-16T04:04:26.817 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.798+0000 7ff9b97fa640 10 client.?.objecter resend_mon_ops 2024-09-16T04:04:26.817 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.798+0000 7ff9b97fa640 1 -- 172.21.15.42:0/453178422 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9b4017070 con 0x7ff9bc1392c0 2024-09-16T04:04:26.817 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.798+0000 7ff9b97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:04:26.817 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.798+0000 7ff9b97fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:04:26.817 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.798+0000 7ff9b97fa640 10 monclient: dump: 2024-09-16T04:04:26.817 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:04:26.817 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:04:26.817 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:26.817 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:26.818 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:04:26.818 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:04:26.818 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:04:26.818 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:04:26.818 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:04:26.818 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:04:26.818 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.798+0000 7ff9b97fa640 10 monclient: _finish_auth 0 2024-09-16T04:04:26.818 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.798+0000 7ff9b97fa640 10 monclient: _check_auth_tickets 2024-09-16T04:04:26.818 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.798+0000 7ff9b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:04:26.818 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.798+0000 7ff9b97fa640 1 -- 172.21.15.42:0/453178422 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff9b400c940 con 0x7ff9bc1392c0 2024-09-16T04:04:26.818 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.798+0000 7ff9c2b36640 5 monclient: authenticate success, global_id 984108 2024-09-16T04:04:26.818 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.798+0000 7ff9c2b36640 10 monclient: _renew_subs 2024-09-16T04:04:26.818 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.798+0000 7ff9c2b36640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:04:26.819 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.798+0000 7ff9c2b36640 1 -- 172.21.15.42:0/453178422 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9bc11c9d0 con 0x7ff9bc1392c0 2024-09-16T04:04:26.819 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.798+0000 7ff9b97fa640 10 monclient: handle_config config(0 keys) 2024-09-16T04:04:26.819 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.798+0000 7ff9b97fa640 1 -- 172.21.15.42:0/453178422 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9b400cd30 con 0x7ff9bc1392c0 2024-09-16T04:04:26.819 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.798+0000 7ff9b97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:04:26.819 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.798+0000 7ff9b97fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:04:26.819 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.798+0000 7ff9b97fa640 10 monclient: dump: 2024-09-16T04:04:26.819 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:04:26.819 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:04:26.819 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:26.819 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:26.819 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:04:26.819 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:04:26.819 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:04:26.819 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:04:26.820 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:04:26.820 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:04:26.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.798+0000 7ff9c2b36640 10 client.984108.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:04:26.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.798+0000 7ff9c2b36640 10 monclient: _renew_subs 2024-09-16T04:04:26.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.798+0000 7ff9c2b36640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:04:26.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.798+0000 7ff9ba7fc640 4 set_mon_vals no callback set 2024-09-16T04:04:26.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.798+0000 7ff9c2b36640 1 -- 172.21.15.42:0/453178422 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9bc06e060 con 0x7ff9bc1392c0 2024-09-16T04:04:26.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.798+0000 7ff9c2b36640 1 librados: init done 2024-09-16T04:04:26.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.798+0000 7ff9b97fa640 1 -- 172.21.15.42:0/453178422 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7ff9b401a070 con 0x7ff9bc1392c0 2024-09-16T04:04:26.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.798+0000 7ff9c2b36640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:04:26.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.798+0000 7ff9c2b36640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:04:26.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.798+0000 7ff9c2b36640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:04:26.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.798+0000 7ff9c2b36640 1 -- 172.21.15.42:0/453178422 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff9bc13c900 con 0x7ff9bc1392c0 2024-09-16T04:04:26.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.798+0000 7ff9b97fa640 1 --2- 172.21.15.42:0/453178422 >> v2:172.21.15.154:6812/254845342 conn(0x7ff99c037810 0x7ff99c039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:04:26.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.798+0000 7ff9b97fa640 1 -- 172.21.15.42:0/453178422 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(678..678 src has 451..678) ==== 6280+0+0 (secure 0 0 0) 0x7ff9b40491a0 con 0x7ff9bc1392c0 2024-09-16T04:04:26.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.798+0000 7ff9b97fa640 10 client.984108.objecter ms_dispatch 0x7ff9bc000ba0 osd_map(678..678 src has 451..678) 2024-09-16T04:04:26.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.798+0000 7ff9b97fa640 3 client.984108.objecter handle_osd_map got epochs [678,678] > 0 2024-09-16T04:04:26.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.798+0000 7ff9b97fa640 3 client.984108.objecter handle_osd_map decoding full epoch 678 2024-09-16T04:04:26.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.798+0000 7ff9b97fa640 20 client.984108.objecter dump_active .. 0 homeless 2024-09-16T04:04:26.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.802+0000 7ff9c08d2640 1 --2- 172.21.15.42:0/453178422 >> v2:172.21.15.154:6812/254845342 conn(0x7ff99c037810 0x7ff99c039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:26.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.802+0000 7ff9c08d2640 10 monclient: get_auth_request con 0x7ff99c037810 auth_method 0 2024-09-16T04:04:26.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.802+0000 7ff9b97fa640 10 client.984108.objecter ms_handle_connect 0x7ff99c037810 2024-09-16T04:04:26.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.802+0000 7ff9c08d2640 1 --2- 172.21.15.42:0/453178422 >> v2:172.21.15.154:6812/254845342 conn(0x7ff99c037810 0x7ff99c039cd0 secure :-1 s=READY pgs=3820 cs=0 l=1 rev1=1 crypto rx=0x7ff9ac001d50 tx=0x7ff9ac009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:04:26.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.806+0000 7ff9b97fa640 1 -- 172.21.15.42:0/453178422 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7ff9b4013470 con 0x7ff9bc1392c0 2024-09-16T04:04:26.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.806+0000 7ff9b97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:04:26.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:26.806+0000 7ff9b97fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:04:27.040 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.034+0000 7ff9c2b36640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:04:27.040 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.034+0000 7ff9c2b36640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:04:27.040 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.034+0000 7ff9c2b36640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:04:27.040 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.034+0000 7ff9c2b36640 1 -- 172.21.15.42:0/453178422 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7ff9bc0a1710 con 0x7ff9bc1392c0 2024-09-16T04:04:27.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.034+0000 7ff9b97fa640 1 -- 172.21.15.42:0/453178422 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7ff9b4011e40 con 0x7ff9bc1392c0 2024-09-16T04:04:27.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.034+0000 7ff9b97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:04:27.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.034+0000 7ff9b97fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:04:27.041 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":210,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:04:27.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.038+0000 7ff9c2b36640 10 librados: watch_flush enter 2024-09-16T04:04:27.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.038+0000 7ff9c2b36640 10 librados: watch_flush exit 2024-09-16T04:04:27.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.038+0000 7ff9c2b36640 20 client.984108.objecter shutdown clearing up homeless session... 2024-09-16T04:04:27.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.038+0000 7ff9c2b36640 10 client.984108.objecter successfully canceled tick 2024-09-16T04:04:27.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.038+0000 7ff9c2b36640 1 -- 172.21.15.42:0/453178422 >> v2:172.21.15.154:6812/254845342 conn(0x7ff99c037810 msgr2=0x7ff99c039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:04:27.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.038+0000 7ff9c2b36640 1 --2- 172.21.15.42:0/453178422 >> v2:172.21.15.154:6812/254845342 conn(0x7ff99c037810 0x7ff99c039cd0 secure :-1 s=READY pgs=3820 cs=0 l=1 rev1=1 crypto rx=0x7ff9ac001d50 tx=0x7ff9ac009040 comp rx=0 tx=0).stop 2024-09-16T04:04:27.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.038+0000 7ff9c2b36640 10 monclient: shutdown 2024-09-16T04:04:27.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.038+0000 7ff9c2b36640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:04:27.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.038+0000 7ff9c2b36640 1 -- 172.21.15.42:0/453178422 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff9bc1392c0 msgr2=0x7ff9bc139770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:04:27.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.038+0000 7ff9c2b36640 1 --2- 172.21.15.42:0/453178422 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff9bc1392c0 0x7ff9bc139770 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7ff9b400e7d0 tx=0x7ff9b400ec90 comp rx=0 tx=0).stop 2024-09-16T04:04:27.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.038+0000 7ff9c2b36640 1 -- 172.21.15.42:0/453178422 shutdown_connections 2024-09-16T04:04:27.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.038+0000 7ff9c2b36640 1 --2- 172.21.15.42:0/453178422 >> v2:172.21.15.154:6812/254845342 conn(0x7ff99c037810 0x7ff99c039cd0 unknown :-1 s=CLOSED pgs=3820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:27.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.038+0000 7ff9c2b36640 1 --2- 172.21.15.42:0/453178422 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff9bc1392c0 0x7ff9bc139770 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:27.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.038+0000 7ff9c2b36640 1 --2- 172.21.15.42:0/453178422 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff9bc0a1e20 0x7ff9bc132320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:27.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.038+0000 7ff9c2b36640 1 --2- 172.21.15.42:0/453178422 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff9bc0a1330 0x7ff9bc1ccb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:27.045 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.038+0000 7ff9c2b36640 1 -- 172.21.15.42:0/453178422 >> 172.21.15.42:0/453178422 conn(0x7ff9bc0a2420 msgr2=0x7ff9bc135300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:04:27.045 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.038+0000 7ff9c2b36640 1 -- 172.21.15.42:0/453178422 shutdown_connections 2024-09-16T04:04:27.045 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.038+0000 7ff9c2b36640 1 -- 172.21.15.42:0/453178422 wait complete. 2024-09-16T04:04:27.045 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.038+0000 7ff9c2b36640 1 librados: shutdown 2024-09-16T04:04:27.060 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T04:04:27.061 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T04:04:27.061 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T04:04:27.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.174+0000 7f5b0491b640 10 monclient: get_monmap_and_config 2024-09-16T04:04:27.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.174+0000 7f5b0491b640 10 monclient: build_initial_monmap 2024-09-16T04:04:27.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.174+0000 7f5b0491b640 10 monclient: monmap: 2024-09-16T04:04:27.178 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:04:27.179 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:04:27.179 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:04:27.176954+0000 2024-09-16T04:04:27.179 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:04:27.176954+0000 2024-09-16T04:04:27.179 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:04:27.179 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:04:27.179 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:04:27.179 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:04:27.179 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:04:27.179 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:04:27.180 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:04:27.180 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:04:27.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.174+0000 7f5b0491b640 1 Processor -- start 2024-09-16T04:04:27.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.174+0000 7f5b0491b640 1 -- start start 2024-09-16T04:04:27.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.174+0000 7f5b0491b640 10 monclient: init 2024-09-16T04:04:27.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.174+0000 7f5b0491b640 10 monclient: _reopen_session rank -1 2024-09-16T04:04:27.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.174+0000 7f5b0491b640 10 monclient: _add_conns ranks=[2,0,3,4,1] 2024-09-16T04:04:27.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.174+0000 7f5b0491b640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5b0006db90 0x7f5b001340c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:04:27.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.174+0000 7f5b0491b640 10 monclient(hunting): picked mon.noname-b con 0x7f5b0006db90 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:04:27.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.174+0000 7f5b0491b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5b000a37f0 addr v1:172.21.15.42:6789/0 2024-09-16T04:04:27.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.174+0000 7f5b0491b640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f5b000a3bd0 0x7f5b00137520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:04:27.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.174+0000 7f5b0491b640 10 monclient(hunting): picked mon.noname-a con 0x7f5b000a3bd0 addr v2:172.21.15.42:3300/0 2024-09-16T04:04:27.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.174+0000 7f5b0491b640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5b0006ece0 con 0x7f5b000a37f0 2024-09-16T04:04:27.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.174+0000 7f5b0491b640 10 monclient(hunting): start opening mon connection 2024-09-16T04:04:27.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.174+0000 7f5b0491b640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f5b0006df70 con 0x7f5b000a3bd0 2024-09-16T04:04:27.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.174+0000 7f5b0491b640 10 monclient(hunting): start opening mon connection 2024-09-16T04:04:27.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.174+0000 7f5b0491b640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f5b0009bf80 con 0x7f5b0006db90 2024-09-16T04:04:27.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.174+0000 7f5b0491b640 10 monclient(hunting): _renew_subs 2024-09-16T04:04:27.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.174+0000 7f5b0491b640 10 monclient(hunting): authenticate will time out at 2314.146364s 2024-09-16T04:04:27.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.174+0000 7f5afed9d640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f5b000a3bd0 0x7f5b00137520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:27.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.174+0000 7f5afdd9b640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f5b000a37f0 0x7f5b00136e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:39726/0 (socket says 172.21.15.42:39726) 2024-09-16T04:04:27.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.174+0000 7f5afdd9b640 1 -- 172.21.15.42:0/2062191425 learned_addr learned my addr 172.21.15.42:0/2062191425 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T04:04:27.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.174+0000 7f5afed9d640 10 monclient(hunting): get_auth_request con 0x7f5b000a3bd0 auth_method 0 2024-09-16T04:04:27.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.174+0000 7f5afed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:04:27.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.174+0000 7f5afed9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:04:27.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.174+0000 7f5afed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:04:27.187 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.174+0000 7f5afe59c640 1 --2- 172.21.15.42:0/2062191425 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5b0006db90 0x7f5b001340c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:27.187 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.174+0000 7f5afe59c640 10 monclient(hunting): get_auth_request con 0x7f5b0006db90 auth_method 0 2024-09-16T04:04:27.187 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.174+0000 7f5afe59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:04:27.187 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.174+0000 7f5afe59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:04:27.187 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.174+0000 7f5afe59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:04:27.187 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.174+0000 7f5afed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:04:27.187 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.174+0000 7f5afed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:04:27.187 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.174+0000 7f5afed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:04:27.187 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.174+0000 7f5afe59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:04:27.187 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.174+0000 7f5afe59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:04:27.187 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.174+0000 7f5afe59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:04:27.187 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.174+0000 7f5afd59a640 1 -- 172.21.15.42:0/2062191425 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4028502289 0 0) 0x7f5b0006ece0 con 0x7f5b000a37f0 2024-09-16T04:04:27.187 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.174+0000 7f5afd59a640 10 monclient(hunting): _negotiate 2024-09-16T04:04:27.188 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.174+0000 7f5afd59a640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:04:27.188 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.174+0000 7f5afd59a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:04:27.188 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.174+0000 7f5afd59a640 10 monclient(hunting): my global_id is 984114 2024-09-16T04:04:27.188 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.174+0000 7f5afd59a640 1 -- 172.21.15.42:0/2062191425 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5ae8003530 con 0x7f5b000a37f0 2024-09-16T04:04:27.188 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.174+0000 7f5afed9d640 10 monclient(hunting): handle_auth_done global_id 984111 payload 274 2024-09-16T04:04:27.188 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.174+0000 7f5afed9d640 1 -- 172.21.15.42:0/2062191425 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5b0006db90 msgr2=0x7f5b001340c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:04:27.188 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.174+0000 7f5afed9d640 1 --2- 172.21.15.42:0/2062191425 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5b0006db90 0x7f5b001340c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:27.188 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.174+0000 7f5afed9d640 1 -- 172.21.15.42:0/2062191425 >> v1:172.21.15.42:6789/0 conn(0x7f5b000a37f0 legacy=0x7f5b00136e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:04:27.188 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.174+0000 7f5afed9d640 10 monclient: _finish_hunting 0 2024-09-16T04:04:27.188 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.174+0000 7f5afed9d640 1 monclient: found mon.noname-a 2024-09-16T04:04:27.188 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.174+0000 7f5afed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:04:27.188 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.174+0000 7f5afed9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:04:27.188 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.174+0000 7f5afed9d640 1 -- 172.21.15.42:0/2062191425 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b00134780 con 0x7f5b000a3bd0 2024-09-16T04:04:27.189 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.174+0000 7f5afe59c640 1 --2- 172.21.15.42:0/2062191425 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5b0006db90 0x7f5b001340c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:04:27.189 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.178+0000 7f5afed9d640 1 --2- 172.21.15.42:0/2062191425 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5b000a3bd0 0x7f5b00137520 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f5af0009d90 tx=0x7f5af001d290 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14d4e8cc78ac47a2 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:04:27.189 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.178+0000 7f5afd59a640 1 -- 172.21.15.42:0/2062191425 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5af00b2070 con 0x7f5b000a3bd0 2024-09-16T04:04:27.189 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.178+0000 7f5afd59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:04:27.189 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.178+0000 7f5afd59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:04:27.189 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.178+0000 7f5afd59a640 10 monclient: dump: 2024-09-16T04:04:27.189 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:04:27.189 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:04:27.189 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:27.189 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:27.189 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:04:27.189 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:04:27.189 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:04:27.190 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:04:27.190 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:04:27.190 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:04:27.190 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.178+0000 7f5afd59a640 10 monclient: _finish_auth 0 2024-09-16T04:04:27.190 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.178+0000 7f5afd59a640 10 monclient: _check_auth_tickets 2024-09-16T04:04:27.190 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.178+0000 7f5afd59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:04:27.190 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.178+0000 7f5afd59a640 1 -- 172.21.15.42:0/2062191425 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5af00a5800 con 0x7f5b000a3bd0 2024-09-16T04:04:27.190 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.178+0000 7f5afd59a640 10 monclient: handle_config config(0 keys) 2024-09-16T04:04:27.190 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.178+0000 7f5afd59a640 1 -- 172.21.15.42:0/2062191425 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5af00a5bd0 con 0x7f5b000a3bd0 2024-09-16T04:04:27.190 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.178+0000 7f5b0491b640 5 monclient: authenticate success, global_id 984111 2024-09-16T04:04:27.190 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.178+0000 7f5b0491b640 10 monclient: get_monmap_and_config success 2024-09-16T04:04:27.190 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.178+0000 7f5b0491b640 4 set_mon_vals no callback set 2024-09-16T04:04:27.191 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.178+0000 7f5afd59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:04:27.191 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.178+0000 7f5afd59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:04:27.191 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.178+0000 7f5afd59a640 10 monclient: dump: 2024-09-16T04:04:27.191 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:04:27.191 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:04:27.191 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:27.191 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:27.192 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:04:27.192 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:04:27.192 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:04:27.192 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:04:27.192 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:04:27.192 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:04:27.192 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.178+0000 7f5b0491b640 10 monclient: shutdown 2024-09-16T04:04:27.192 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.178+0000 7f5b0491b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:04:27.192 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.178+0000 7f5b0491b640 1 -- 172.21.15.42:0/2062191425 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5b000a3bd0 msgr2=0x7f5b00137520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:04:27.192 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.178+0000 7f5b0491b640 1 --2- 172.21.15.42:0/2062191425 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5b000a3bd0 0x7f5b00137520 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f5af0009d90 tx=0x7f5af001d290 comp rx=0 tx=0).stop 2024-09-16T04:04:27.192 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.178+0000 7f5b0491b640 1 -- 172.21.15.42:0/2062191425 shutdown_connections 2024-09-16T04:04:27.193 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.178+0000 7f5b0491b640 1 --2- 172.21.15.42:0/2062191425 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5b000a3bd0 0x7f5b00137520 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:27.193 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.178+0000 7f5b0491b640 1 --2- 172.21.15.42:0/2062191425 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5b0006db90 0x7f5b001340c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:27.193 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.178+0000 7f5b0491b640 1 -- 172.21.15.42:0/2062191425 >> 172.21.15.42:0/2062191425 conn(0x7f5b000a9fb0 msgr2=0x7f5b000aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:04:27.193 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.178+0000 7f5b0491b640 1 -- 172.21.15.42:0/2062191425 shutdown_connections 2024-09-16T04:04:27.193 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.178+0000 7f5b0491b640 1 -- 172.21.15.42:0/2062191425 wait complete. 2024-09-16T04:04:27.193 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.178+0000 7f5b0491b640 10 monclient: build_initial_monmap 2024-09-16T04:04:27.193 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.178+0000 7f5b0491b640 10 monclient: monmap: 2024-09-16T04:04:27.193 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:04:27.193 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:04:27.193 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:04:27.193 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:04:27.193 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:04:27.194 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:04:27.194 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:04:27.194 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:04:27.194 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:04:27.194 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:04:27.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.178+0000 7f5b0491b640 1 librados: starting msgr at 2024-09-16T04:04:27.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.178+0000 7f5b0491b640 1 librados: starting objecter 2024-09-16T04:04:27.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.178+0000 7f5b0491b640 1 Processor -- start 2024-09-16T04:04:27.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.178+0000 7f5b0491b640 1 -- start start 2024-09-16T04:04:27.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.178+0000 7f5b0491b640 1 librados: setting wanted keys 2024-09-16T04:04:27.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.178+0000 7f5b0491b640 1 librados: calling monclient init 2024-09-16T04:04:27.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.178+0000 7f5b0491b640 10 monclient: init 2024-09-16T04:04:27.201 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.178+0000 7f5b0491b640 10 monclient: _reopen_session rank -1 2024-09-16T04:04:27.201 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.178+0000 7f5b0491b640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:04:27.201 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.178+0000 7f5b0491b640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5b0006db90 0x7f5b00130560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:04:27.201 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.178+0000 7f5b0491b640 10 monclient(hunting): picked mon.noname-a con 0x7f5b0006db90 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:04:27.201 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.178+0000 7f5b0491b640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5b000a37f0 0x7f5b00130aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:04:27.201 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.178+0000 7f5b0491b640 10 monclient(hunting): picked mon.noname-c con 0x7f5b000a37f0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:04:27.201 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.178+0000 7f5b0491b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5b000a3bd0 0x7f5b000a1590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:04:27.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.178+0000 7f5b0491b640 10 monclient(hunting): picked mon.noname-b con 0x7f5b000a3bd0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:04:27.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.178+0000 7f5b0491b640 10 monclient(hunting): start opening mon connection 2024-09-16T04:04:27.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.178+0000 7f5b0491b640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f5b001419d0 con 0x7f5b0006db90 2024-09-16T04:04:27.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.178+0000 7f5b0491b640 10 monclient(hunting): start opening mon connection 2024-09-16T04:04:27.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.178+0000 7f5b0491b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f5b000a4100 con 0x7f5b000a3bd0 2024-09-16T04:04:27.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.178+0000 7f5b0491b640 10 monclient(hunting): start opening mon connection 2024-09-16T04:04:27.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.178+0000 7f5b0491b640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f5b00141b50 con 0x7f5b000a37f0 2024-09-16T04:04:27.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.178+0000 7f5b0491b640 10 monclient(hunting): _renew_subs 2024-09-16T04:04:27.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.178+0000 7f5b0491b640 10 monclient(hunting): authenticate will time out at 2314.151598s 2024-09-16T04:04:27.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.178+0000 7f5afe59c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5b0006db90 0x7f5b00130560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:27.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.178+0000 7f5afe59c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5b0006db90 0x7f5b00130560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:39606/0 (socket says 172.21.15.42:39606) 2024-09-16T04:04:27.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.178+0000 7f5afdd9b640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5b000a37f0 0x7f5b00130aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:27.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.178+0000 7f5afe59c640 1 -- 172.21.15.42:0/3872559743 learned_addr learned my addr 172.21.15.42:0/3872559743 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:04:27.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.178+0000 7f5afe59c640 10 monclient(hunting): get_auth_request con 0x7f5b0006db90 auth_method 0 2024-09-16T04:04:27.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.178+0000 7f5afe59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:04:27.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.178+0000 7f5afe59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:04:27.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.178+0000 7f5afe59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:04:27.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.178+0000 7f5afdd9b640 10 monclient(hunting): get_auth_request con 0x7f5b000a37f0 auth_method 0 2024-09-16T04:04:27.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.178+0000 7f5afdd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:04:27.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.178+0000 7f5afdd9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:04:27.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.178+0000 7f5afdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:04:27.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.178+0000 7f5afdd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:04:27.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.178+0000 7f5afdd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:04:27.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.178+0000 7f5afdd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:04:27.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.178+0000 7f5afe59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:04:27.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.178+0000 7f5afe59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:04:27.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.178+0000 7f5afe59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:04:27.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.182+0000 7f5afe59c640 10 monclient(hunting): handle_auth_done global_id 984117 payload 824 2024-09-16T04:04:27.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.182+0000 7f5afe59c640 1 -- 172.21.15.42:0/3872559743 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5b000a37f0 msgr2=0x7f5b00130aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:04:27.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.182+0000 7f5afe59c640 1 --2- 172.21.15.42:0/3872559743 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5b000a37f0 0x7f5b00130aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:27.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.182+0000 7f5afe59c640 1 -- 172.21.15.42:0/3872559743 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5b000a3bd0 msgr2=0x7f5b000a1590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:04:27.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.182+0000 7f5afe59c640 1 --2- 172.21.15.42:0/3872559743 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5b000a3bd0 0x7f5b000a1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:27.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.182+0000 7f5afe59c640 10 monclient: _finish_hunting 0 2024-09-16T04:04:27.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.182+0000 7f5afe59c640 1 monclient: found mon.noname-a 2024-09-16T04:04:27.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.182+0000 7f5afe59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:04:27.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.182+0000 7f5afe59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:04:27.205 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.182+0000 7f5afe59c640 1 -- 172.21.15.42:0/3872559743 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b0013a510 con 0x7f5b0006db90 2024-09-16T04:04:27.205 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.182+0000 7f5afdd9b640 1 --2- 172.21.15.42:0/3872559743 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5b000a37f0 0x7f5b00130aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:04:27.205 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.182+0000 7f5afe59c640 1 --2- 172.21.15.42:0/3872559743 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5b0006db90 0x7f5b00130560 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f5af800e980 tx=0x7f5af800ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:04:27.205 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.182+0000 7f5aeeffd640 10 client.?.objecter ms_handle_connect 0x7f5b0006db90 2024-09-16T04:04:27.205 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.182+0000 7f5aeeffd640 10 client.?.objecter resend_mon_ops 2024-09-16T04:04:27.205 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.182+0000 7f5aeeffd640 1 -- 172.21.15.42:0/3872559743 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5af8017070 con 0x7f5b0006db90 2024-09-16T04:04:27.205 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.182+0000 7f5aeeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:04:27.205 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.182+0000 7f5aeeffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:04:27.205 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.182+0000 7f5aeeffd640 10 monclient: dump: 2024-09-16T04:04:27.205 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:04:27.205 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:04:27.205 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:27.205 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:27.206 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:04:27.206 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:04:27.206 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:04:27.206 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:04:27.206 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:04:27.206 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:04:27.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.182+0000 7f5aeeffd640 10 monclient: _finish_auth 0 2024-09-16T04:04:27.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.182+0000 7f5aeeffd640 10 monclient: _check_auth_tickets 2024-09-16T04:04:27.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.182+0000 7f5aeeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:04:27.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.182+0000 7f5aeeffd640 1 -- 172.21.15.42:0/3872559743 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5af800ca00 con 0x7f5b0006db90 2024-09-16T04:04:27.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.182+0000 7f5b0491b640 5 monclient: authenticate success, global_id 984117 2024-09-16T04:04:27.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.182+0000 7f5aeeffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:04:27.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.182+0000 7f5aeeffd640 1 -- 172.21.15.42:0/3872559743 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5af800cdd0 con 0x7f5b0006db90 2024-09-16T04:04:27.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.182+0000 7f5b0491b640 10 monclient: _renew_subs 2024-09-16T04:04:27.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.182+0000 7f5b0491b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:04:27.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.182+0000 7f5b0491b640 1 -- 172.21.15.42:0/3872559743 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b00133150 con 0x7f5b0006db90 2024-09-16T04:04:27.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.182+0000 7f5afcd99640 4 set_mon_vals no callback set 2024-09-16T04:04:27.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.182+0000 7f5aeeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:04:27.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.182+0000 7f5aeeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:04:27.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.182+0000 7f5aeeffd640 10 monclient: dump: 2024-09-16T04:04:27.207 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:04:27.207 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:04:27.207 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:27.207 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:27.208 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:04:27.208 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:04:27.208 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:04:27.208 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:04:27.208 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:04:27.208 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:04:27.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.182+0000 7f5b0491b640 10 client.984117.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:04:27.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.182+0000 7f5b0491b640 10 monclient: _renew_subs 2024-09-16T04:04:27.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.182+0000 7f5b0491b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:04:27.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.182+0000 7f5b0491b640 1 -- 172.21.15.42:0/3872559743 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b00130fe0 con 0x7f5b0006db90 2024-09-16T04:04:27.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.182+0000 7f5b0491b640 1 librados: init done 2024-09-16T04:04:27.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.182+0000 7f5b0491b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:04:27.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.182+0000 7f5b0491b640 1 --2- 172.21.15.42:0/3872559743 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5b000a2290 0x7f5b0002c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:04:27.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.182+0000 7f5b0491b640 10 monclient(hunting): start opening mon connection 2024-09-16T04:04:27.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.182+0000 7f5b0491b640 1 -- 172.21.15.42:0/3872559743 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f5b000a9010 con 0x7f5b000a2290 2024-09-16T04:04:27.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.182+0000 7f5afdd9b640 1 --2- 172.21.15.42:0/3872559743 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5b000a2290 0x7f5b0002c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:27.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.182+0000 7f5afdd9b640 10 monclient: get_auth_request con 0x7f5b000a2290 auth_method 0 2024-09-16T04:04:27.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.182+0000 7f5afdd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:04:27.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.182+0000 7f5afdd9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:04:27.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.182+0000 7f5afdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:04:27.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.182+0000 7f5afdd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:04:27.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.182+0000 7f5afdd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:04:27.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.182+0000 7f5afdd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:04:27.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.182+0000 7f5afdd9b640 10 monclient(hunting): handle_auth_done global_id 984120 payload 824 2024-09-16T04:04:27.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.182+0000 7f5afdd9b640 1 -- 172.21.15.42:0/3872559743 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5b001340c0 con 0x7f5b000a2290 2024-09-16T04:04:27.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.182+0000 7f5afdd9b640 1 --2- 172.21.15.42:0/3872559743 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5b000a2290 0x7f5b0002c650 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f5b000a2db0 tx=0x7f5af400fe40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:04:27.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.182+0000 7f5aeeffd640 10 client.984117.objecter ms_handle_connect 0x7f5b000a2290 2024-09-16T04:04:27.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.182+0000 7f5aeeffd640 10 client.984117.objecter resend_mon_ops 2024-09-16T04:04:27.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.182+0000 7f5aeeffd640 1 -- 172.21.15.42:0/3872559743 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f5af801a070 con 0x7f5b0006db90 2024-09-16T04:04:27.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.182+0000 7f5aeeffd640 1 --2- 172.21.15.42:0/3872559743 >> v2:172.21.15.154:6812/254845342 conn(0x7f5ae0037810 0x7f5ae0039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:04:27.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.182+0000 7f5aeeffd640 1 -- 172.21.15.42:0/3872559743 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(678..678 src has 451..678) ==== 6280+0+0 (secure 0 0 0) 0x7f5af8049060 con 0x7f5b0006db90 2024-09-16T04:04:27.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.182+0000 7f5aeeffd640 10 client.984117.objecter ms_dispatch 0x7f5b00000ba0 osd_map(678..678 src has 451..678) 2024-09-16T04:04:27.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.182+0000 7f5aeeffd640 3 client.984117.objecter handle_osd_map got epochs [678,678] > 0 2024-09-16T04:04:27.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.182+0000 7f5aeeffd640 3 client.984117.objecter handle_osd_map decoding full epoch 678 2024-09-16T04:04:27.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.182+0000 7f5aeeffd640 20 client.984117.objecter dump_active .. 0 homeless 2024-09-16T04:04:27.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.182+0000 7f5aeeffd640 1 -- 172.21.15.42:0/3872559743 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5af4005ad0 con 0x7f5b000a2290 2024-09-16T04:04:27.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.182+0000 7f5aeeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:04:27.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.182+0000 7f5aeeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:04:27.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.182+0000 7f5aeeffd640 10 monclient: dump: 2024-09-16T04:04:27.211 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:04:27.211 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:04:27.211 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:27.211 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:27.211 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:04:27.211 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:04:27.211 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:04:27.211 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:04:27.211 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:04:27.211 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:04:27.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.182+0000 7f5aeeffd640 1 -- 172.21.15.42:0/3872559743 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f5af4018070 con 0x7f5b000a2290 2024-09-16T04:04:27.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.182+0000 7f5aeeffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:04:27.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.182+0000 7f5aeeffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:04:27.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.182+0000 7f5aeeffd640 1 -- 172.21.15.42:0/3872559743 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5b000a2290 msgr2=0x7f5b0002c650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:04:27.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.182+0000 7f5aeeffd640 1 --2- 172.21.15.42:0/3872559743 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5b000a2290 0x7f5b0002c650 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f5b000a2db0 tx=0x7f5af400fe40 comp rx=0 tx=0).stop 2024-09-16T04:04:27.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.182+0000 7f5aeeffd640 1 -- 172.21.15.42:0/3872559743 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5b000a2290 msgr2=0x7f5b0002c650 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:04:27.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.182+0000 7f5aeeffd640 1 --2- 172.21.15.42:0/3872559743 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5b000a2290 0x7f5b0002c650 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:27.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.182+0000 7f5afed9d640 1 --2- 172.21.15.42:0/3872559743 >> v2:172.21.15.154:6812/254845342 conn(0x7f5ae0037810 0x7f5ae0039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:27.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.182+0000 7f5afed9d640 10 monclient: get_auth_request con 0x7f5ae0037810 auth_method 0 2024-09-16T04:04:27.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.182+0000 7f5afed9d640 1 --2- 172.21.15.42:0/3872559743 >> v2:172.21.15.154:6812/254845342 conn(0x7f5ae0037810 0x7f5ae0039cd0 secure :-1 s=READY pgs=3821 cs=0 l=1 rev1=1 crypto rx=0x7f5af00af000 tx=0x7f5af0004a50 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:04:27.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.182+0000 7f5aeeffd640 10 client.984117.objecter ms_handle_connect 0x7f5ae0037810 2024-09-16T04:04:27.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.190+0000 7f5b0491b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:04:27.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.190+0000 7f5b0491b640 1 --2- 172.21.15.42:0/3872559743 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5b000a2670 0x7f5b0002cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:04:27.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.190+0000 7f5b0491b640 10 monclient(hunting): start opening mon connection 2024-09-16T04:04:27.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.190+0000 7f5b0491b640 1 -- 172.21.15.42:0/3872559743 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f5b0009de20 con 0x7f5b000a2670 2024-09-16T04:04:27.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.190+0000 7f5afdd9b640 1 --2- 172.21.15.42:0/3872559743 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5b000a2670 0x7f5b0002cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:27.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.190+0000 7f5afdd9b640 10 monclient: get_auth_request con 0x7f5b000a2670 auth_method 0 2024-09-16T04:04:27.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.190+0000 7f5afdd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:04:27.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.190+0000 7f5afdd9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:04:27.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.190+0000 7f5afdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:04:27.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.190+0000 7f5afdd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:04:27.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.190+0000 7f5afdd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:04:27.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.190+0000 7f5afdd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:04:27.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.190+0000 7f5afdd9b640 10 monclient(hunting): handle_auth_done global_id 984123 payload 824 2024-09-16T04:04:27.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.190+0000 7f5afdd9b640 1 -- 172.21.15.42:0/3872559743 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f5b000008e0 con 0x7f5b000a2670 2024-09-16T04:04:27.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.190+0000 7f5aeeffd640 10 client.984117.objecter ms_handle_connect 0x7f5b000a2670 2024-09-16T04:04:27.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.190+0000 7f5aeeffd640 10 client.984117.objecter resend_mon_ops 2024-09-16T04:04:27.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.190+0000 7f5afdd9b640 1 --2- 172.21.15.42:0/3872559743 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5b000a2670 0x7f5b0002cb90 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f5af4017000 tx=0x7f5af40010b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:04:27.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.190+0000 7f5aeeffd640 1 -- 172.21.15.42:0/3872559743 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5af40075b0 con 0x7f5b000a2670 2024-09-16T04:04:27.214 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.190+0000 7f5aeeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:04:27.214 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.190+0000 7f5aeeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:04:27.214 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.190+0000 7f5aeeffd640 10 monclient: dump: 2024-09-16T04:04:27.214 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:04:27.214 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:04:27.214 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:27.214 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:27.214 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:04:27.214 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:04:27.214 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:04:27.214 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:04:27.214 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:04:27.214 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:04:27.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.190+0000 7f5aeeffd640 1 -- 172.21.15.42:0/3872559743 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4714 (secure 0 0 0) 0x7f5af40075b0 con 0x7f5b000a2670 2024-09-16T04:04:27.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.190+0000 7f5aeeffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:04:27.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.190+0000 7f5aeeffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:04:27.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.190+0000 7f5aeeffd640 1 -- 172.21.15.42:0/3872559743 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5b000a2670 msgr2=0x7f5b0002cb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:04:27.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.190+0000 7f5aeeffd640 1 --2- 172.21.15.42:0/3872559743 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5b000a2670 0x7f5b0002cb90 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f5af4017000 tx=0x7f5af40010b0 comp rx=0 tx=0).stop 2024-09-16T04:04:27.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.190+0000 7f5aeeffd640 1 -- 172.21.15.42:0/3872559743 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5b000a2670 msgr2=0x7f5b0002cb90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:04:27.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.190+0000 7f5aeeffd640 1 --2- 172.21.15.42:0/3872559743 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5b000a2670 0x7f5b0002cb90 secure :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f5af4017000 tx=0x7f5af40010b0 comp rx=0 tx=0).stop 2024-09-16T04:04:27.215 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T04:04:27.215 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:04:27.216 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:04:27.216 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "leader", 2024-09-16T04:04:27.216 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 210, 2024-09-16T04:04:27.216 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 27247, 2024-09-16T04:04:27.216 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T04:04:27.216 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T04:04:27.216 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T04:04:27.216 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T04:04:27.216 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:04:27.216 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 1, 2024-09-16T04:04:27.216 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:04:27.216 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T04:04:27.217 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T04:04:27.217 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:04:27.217 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:04:27.217 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:04:27.217 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:04:27.217 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:04:27.217 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:04:27.217 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:04:27.217 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:04:27.217 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:04:27.217 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:04:27.217 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:04:27.217 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:04:27.217 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:04:27.218 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T04:04:27.218 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:04:27.218 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:04:27.218 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:04:27.218 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:04:27.218 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:04:27.218 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:04:27.218 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:04:27.218 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:04:27.218 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:04:27.218 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:04:27.219 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:04:27.219 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:04:27.219 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:04:27.219 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T04:04:27.219 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T04:04:27.219 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T04:04:27.219 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T04:04:27.219 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T04:04:27.219 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T04:04:27.219 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:04:27.220 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:04:27.220 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T04:04:27.220 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T04:04:27.220 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T04:04:27.220 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T04:04:27.220 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T04:04:27.220 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T04:04:27.220 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T04:04:27.220 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:04:27.220 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T04:04:27.220 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:04:27.220 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:04:27.221 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:04:27.221 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:04:27.221 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:04:27.221 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:04:27.221 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:04:27.221 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:04:27.221 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:04:27.221 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:04:27.221 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:04:27.221 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:04:27.221 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T04:04:27.221 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:04:27.221 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T04:04:27.222 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:04:27.222 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:04:27.222 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:04:27.222 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:04:27.222 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:04:27.222 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:04:27.222 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:04:27.222 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T04:04:27.222 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:04:27.222 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:04:27.222 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:04:27.223 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:04:27.223 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T04:04:27.223 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:04:27.223 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:04:27.223 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:04:27.223 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:04:27.223 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T04:04:27.223 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T04:04:27.223 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:04:27.224 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:04:27.224 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:04:27.224 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:04:27.224 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:04:27.224 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:04:27.224 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:04:27.224 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:04:27.224 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:04:27.224 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:04:27.224 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:04:27.224 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T04:04:27.224 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:04:27.224 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:04:27.225 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:04:27.225 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:04:27.225 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T04:04:27.225 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:04:27.225 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:04:27.225 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:04:27.225 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:04:27.225 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T04:04:27.225 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T04:04:27.225 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:04:27.225 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:04:27.225 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:04:27.225 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:04:27.226 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:04:27.226 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:04:27.226 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:04:27.226 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:04:27.226 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:04:27.226 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:04:27.226 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:04:27.226 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T04:04:27.226 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:04:27.226 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:04:27.226 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:04:27.226 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:04:27.227 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T04:04:27.227 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:04:27.227 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:04:27.227 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:04:27.227 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:04:27.227 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T04:04:27.227 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T04:04:27.227 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:04:27.227 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:04:27.227 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:04:27.227 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:04:27.227 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:04:27.227 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:04:27.228 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T04:04:27.228 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T04:04:27.228 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:04:27.228 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:04:27.228 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:04:27.228 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:04:27.228 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:04:27.228 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:04:27.228 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T04:04:27.228 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:04:27.228 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:04:27.228 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:04:27.229 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 3 2024-09-16T04:04:27.229 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:04:27.229 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:04:27.229 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T04:04:27.229 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:04:27.229 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:04:27.229 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:04:27.229 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 3 2024-09-16T04:04:27.229 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:04:27.229 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:04:27.229 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:04:27.229 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T04:04:27.230 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T04:04:27.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.190+0000 7f5b0491b640 10 librados: watch_flush enter 2024-09-16T04:04:27.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.190+0000 7f5b0491b640 10 librados: watch_flush exit 2024-09-16T04:04:27.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.190+0000 7f5b0491b640 20 client.984117.objecter shutdown clearing up homeless session... 2024-09-16T04:04:27.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.190+0000 7f5b0491b640 10 client.984117.objecter successfully canceled tick 2024-09-16T04:04:27.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.190+0000 7f5b0491b640 1 -- 172.21.15.42:0/3872559743 >> v2:172.21.15.154:6812/254845342 conn(0x7f5ae0037810 msgr2=0x7f5ae0039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:04:27.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.190+0000 7f5b0491b640 1 --2- 172.21.15.42:0/3872559743 >> v2:172.21.15.154:6812/254845342 conn(0x7f5ae0037810 0x7f5ae0039cd0 secure :-1 s=READY pgs=3821 cs=0 l=1 rev1=1 crypto rx=0x7f5af00af000 tx=0x7f5af0004a50 comp rx=0 tx=0).stop 2024-09-16T04:04:27.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.190+0000 7f5b0491b640 10 monclient: shutdown 2024-09-16T04:04:27.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.190+0000 7f5b0491b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:04:27.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.190+0000 7f5afed9d640 1 -- 172.21.15.42:0/3872559743 reap_dead start 2024-09-16T04:04:27.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.190+0000 7f5b0491b640 1 -- 172.21.15.42:0/3872559743 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5b0006db90 msgr2=0x7f5b00130560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:04:27.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.190+0000 7f5b0491b640 1 --2- 172.21.15.42:0/3872559743 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5b0006db90 0x7f5b00130560 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f5af800e980 tx=0x7f5af800ee40 comp rx=0 tx=0).stop 2024-09-16T04:04:27.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.190+0000 7f5b0491b640 1 -- 172.21.15.42:0/3872559743 shutdown_connections 2024-09-16T04:04:27.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.190+0000 7f5b0491b640 1 --2- 172.21.15.42:0/3872559743 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5b0006db90 0x7f5b00130560 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:27.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.190+0000 7f5b0491b640 1 -- 172.21.15.42:0/3872559743 >> 172.21.15.42:0/3872559743 conn(0x7f5b000a9fb0 msgr2=0x7f5b001369f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:04:27.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.190+0000 7f5b0491b640 1 -- 172.21.15.42:0/3872559743 shutdown_connections 2024-09-16T04:04:27.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.190+0000 7f5b0491b640 1 -- 172.21.15.42:0/3872559743 wait complete. 2024-09-16T04:04:27.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.190+0000 7f5b0491b640 1 librados: shutdown 2024-09-16T04:04:27.232 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:04:27.232 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T04:04:27.233 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T04:04:27.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.342+0000 7fd429c5e640 10 monclient: get_monmap_and_config 2024-09-16T04:04:27.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.342+0000 7fd429c5e640 10 monclient: build_initial_monmap 2024-09-16T04:04:27.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.342+0000 7fd429c5e640 10 monclient: monmap: 2024-09-16T04:04:27.358 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:04:27.359 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:04:27.359 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:04:27.345108+0000 2024-09-16T04:04:27.359 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:04:27.345108+0000 2024-09-16T04:04:27.359 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:04:27.359 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:04:27.359 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:04:27.359 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:04:27.359 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:04:27.359 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:04:27.359 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:04:27.359 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:04:27.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.342+0000 7fd429c5e640 1 Processor -- start 2024-09-16T04:04:27.360 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.342+0000 7fd429c5e640 1 -- start start 2024-09-16T04:04:27.360 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.342+0000 7fd429c5e640 10 monclient: init 2024-09-16T04:04:27.360 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.342+0000 7fd429c5e640 10 monclient: _reopen_session rank -1 2024-09-16T04:04:27.360 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.342+0000 7fd429c5e640 10 monclient: _add_conns ranks=[3,2,1,4,0] 2024-09-16T04:04:27.360 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.342+0000 7fd429c5e640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd4240a1330 0x7fd4240a1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:04:27.360 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.342+0000 7fd429c5e640 10 monclient(hunting): picked mon.noname-a con 0x7fd4240a1330 addr v2:172.21.15.42:3300/0 2024-09-16T04:04:27.360 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.342+0000 7fd429c5e640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd42406dac0 0x7fd4240a1c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:04:27.360 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.342+0000 7fd429c5e640 10 monclient(hunting): picked mon.noname-b con 0x7fd42406dac0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:04:27.360 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.342+0000 7fd429c5e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd424135100 addr v1:172.21.15.154:6789/0 2024-09-16T04:04:27.360 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.342+0000 7fd429c5e640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd42409c7d0 con 0x7fd424135100 2024-09-16T04:04:27.360 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.342+0000 7fd429c5e640 10 monclient(hunting): start opening mon connection 2024-09-16T04:04:27.360 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.342+0000 7fd429c5e640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7fd42409c090 con 0x7fd4240a1330 2024-09-16T04:04:27.361 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.342+0000 7fd429c5e640 10 monclient(hunting): start opening mon connection 2024-09-16T04:04:27.361 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.342+0000 7fd429c5e640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fd42409f610 con 0x7fd42406dac0 2024-09-16T04:04:27.361 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.342+0000 7fd429c5e640 10 monclient(hunting): _renew_subs 2024-09-16T04:04:27.361 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.342+0000 7fd429c5e640 10 monclient(hunting): authenticate will time out at 2314.314538s 2024-09-16T04:04:27.361 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.342+0000 7fd422ffd640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd42406dac0 0x7fd4240a1c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:27.361 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.342+0000 7fd4237fe640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd4240a1330 0x7fd4240a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:27.361 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.342+0000 7fd422ffd640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd42406dac0 0x7fd4240a1c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:52340/0 (socket says 172.21.15.42:52340) 2024-09-16T04:04:27.361 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.342+0000 7fd422ffd640 1 -- 172.21.15.42:0/318280906 learned_addr learned my addr 172.21.15.42:0/318280906 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:04:27.361 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.342+0000 7fd422ffd640 10 monclient(hunting): get_auth_request con 0x7fd42406dac0 auth_method 0 2024-09-16T04:04:27.361 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.342+0000 7fd422ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:04:27.361 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.342+0000 7fd422ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:04:27.361 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.342+0000 7fd422ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:04:27.361 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.342+0000 7fd4237fe640 10 monclient(hunting): get_auth_request con 0x7fd4240a1330 auth_method 0 2024-09-16T04:04:27.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.342+0000 7fd4237fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:04:27.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.342+0000 7fd4237fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:04:27.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.342+0000 7fd4237fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:04:27.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.342+0000 7fd422ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:04:27.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.342+0000 7fd422ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:04:27.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.342+0000 7fd422ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:04:27.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.342+0000 7fd4237fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:04:27.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.342+0000 7fd4237fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:04:27.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.342+0000 7fd4237fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:04:27.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.342+0000 7fd4237fe640 10 monclient(hunting): handle_auth_done global_id 984126 payload 274 2024-09-16T04:04:27.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.342+0000 7fd4237fe640 1 -- 172.21.15.42:0/318280906 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd42406dac0 msgr2=0x7fd4240a1c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:04:27.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.342+0000 7fd4237fe640 1 --2- 172.21.15.42:0/318280906 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd42406dac0 0x7fd4240a1c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:27.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.342+0000 7fd4237fe640 1 -- 172.21.15.42:0/318280906 >> v1:172.21.15.154:6789/0 conn(0x7fd424135100 legacy=0x7fd424135550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:04:27.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.342+0000 7fd4237fe640 10 monclient: _finish_hunting 0 2024-09-16T04:04:27.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.342+0000 7fd4237fe640 1 monclient: found mon.noname-a 2024-09-16T04:04:27.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.342+0000 7fd4237fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:04:27.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.342+0000 7fd4237fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:04:27.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.342+0000 7fd4237fe640 1 -- 172.21.15.42:0/318280906 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd42409f8f0 con 0x7fd4240a1330 2024-09-16T04:04:27.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.342+0000 7fd422ffd640 1 --2- 172.21.15.42:0/318280906 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd42406dac0 0x7fd4240a1c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:04:27.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.346+0000 7fd4237fe640 1 --2- 172.21.15.42:0/318280906 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd4240a1330 0x7fd4240a1710 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fd41c009d90 tx=0x7fd41c01c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6f1594ea27a6928 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:04:27.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.346+0000 7fd4227fc640 1 -- 172.21.15.42:0/318280906 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd41c0b2070 con 0x7fd4240a1330 2024-09-16T04:04:27.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.346+0000 7fd4227fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:04:27.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.346+0000 7fd4227fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:04:27.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.346+0000 7fd4227fc640 10 monclient: dump: 2024-09-16T04:04:27.363 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:04:27.363 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:04:27.363 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:27.364 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:27.364 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:04:27.364 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:04:27.364 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:04:27.364 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:04:27.364 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:04:27.364 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:04:27.364 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.346+0000 7fd4227fc640 10 monclient: _finish_auth 0 2024-09-16T04:04:27.364 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.346+0000 7fd4227fc640 10 monclient: _check_auth_tickets 2024-09-16T04:04:27.364 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.346+0000 7fd4227fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:04:27.364 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.346+0000 7fd429c5e640 5 monclient: authenticate success, global_id 984126 2024-09-16T04:04:27.364 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.346+0000 7fd4227fc640 1 -- 172.21.15.42:0/318280906 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd41c0a5920 con 0x7fd4240a1330 2024-09-16T04:04:27.365 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.346+0000 7fd429c5e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:04:27.365 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.346+0000 7fd4227fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:04:27.365 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.346+0000 7fd429c5e640 10 monclient: get_monmap_and_config success 2024-09-16T04:04:27.365 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.346+0000 7fd429c5e640 4 set_mon_vals no callback set 2024-09-16T04:04:27.365 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.346+0000 7fd4227fc640 1 -- 172.21.15.42:0/318280906 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd41c0a5cf0 con 0x7fd4240a1330 2024-09-16T04:04:27.365 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.346+0000 7fd4227fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:04:27.365 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.346+0000 7fd4227fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:04:27.365 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.346+0000 7fd4227fc640 10 monclient: dump: 2024-09-16T04:04:27.365 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:04:27.365 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:04:27.365 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:27.365 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:27.365 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:04:27.365 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:04:27.366 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:04:27.366 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:04:27.366 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:04:27.366 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:04:27.366 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.346+0000 7fd429c5e640 10 monclient: shutdown 2024-09-16T04:04:27.366 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.346+0000 7fd429c5e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:04:27.366 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.346+0000 7fd429c5e640 1 -- 172.21.15.42:0/318280906 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd4240a1330 msgr2=0x7fd4240a1710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:04:27.366 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.346+0000 7fd429c5e640 1 --2- 172.21.15.42:0/318280906 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd4240a1330 0x7fd4240a1710 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fd41c009d90 tx=0x7fd41c01c950 comp rx=0 tx=0).stop 2024-09-16T04:04:27.366 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.346+0000 7fd429c5e640 1 -- 172.21.15.42:0/318280906 shutdown_connections 2024-09-16T04:04:27.366 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.346+0000 7fd429c5e640 1 --2- 172.21.15.42:0/318280906 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd42406dac0 0x7fd4240a1c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:27.366 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.346+0000 7fd429c5e640 1 --2- 172.21.15.42:0/318280906 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd4240a1330 0x7fd4240a1710 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:27.366 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.346+0000 7fd429c5e640 1 -- 172.21.15.42:0/318280906 >> 172.21.15.42:0/318280906 conn(0x7fd4240a2420 msgr2=0x7fd4240a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:04:27.366 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.346+0000 7fd429c5e640 1 -- 172.21.15.42:0/318280906 shutdown_connections 2024-09-16T04:04:27.366 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.346+0000 7fd429c5e640 1 -- 172.21.15.42:0/318280906 wait complete. 2024-09-16T04:04:27.367 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.346+0000 7fd429c5e640 10 monclient: build_initial_monmap 2024-09-16T04:04:27.367 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.346+0000 7fd429c5e640 10 monclient: monmap: 2024-09-16T04:04:27.367 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:04:27.367 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:04:27.367 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:04:27.367 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:04:27.367 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:04:27.367 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:04:27.367 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:04:27.367 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:04:27.367 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:04:27.367 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:04:27.367 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.346+0000 7fd429c5e640 1 librados: starting msgr at 2024-09-16T04:04:27.367 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.346+0000 7fd429c5e640 1 librados: starting objecter 2024-09-16T04:04:27.368 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.346+0000 7fd429c5e640 1 Processor -- start 2024-09-16T04:04:27.368 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.346+0000 7fd429c5e640 1 -- start start 2024-09-16T04:04:27.368 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.346+0000 7fd429c5e640 1 librados: setting wanted keys 2024-09-16T04:04:27.368 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.346+0000 7fd429c5e640 1 librados: calling monclient init 2024-09-16T04:04:27.368 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.346+0000 7fd429c5e640 10 monclient: init 2024-09-16T04:04:27.368 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.346+0000 7fd429c5e640 10 monclient: _reopen_session rank -1 2024-09-16T04:04:27.368 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.346+0000 7fd429c5e640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:04:27.368 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.346+0000 7fd429c5e640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd4240a1330 0x7fd42413b180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:04:27.368 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.346+0000 7fd429c5e640 10 monclient(hunting): picked mon.noname-c con 0x7fd4240a1330 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:04:27.368 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.346+0000 7fd429c5e640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd424135100 0x7fd424134530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:04:27.368 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.346+0000 7fd429c5e640 10 monclient(hunting): picked mon.noname-a con 0x7fd424135100 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:04:27.368 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.346+0000 7fd429c5e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd424134a70 0x7fd42409cb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:04:27.368 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.346+0000 7fd429c5e640 10 monclient(hunting): picked mon.noname-b con 0x7fd424134a70 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:04:27.368 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.346+0000 7fd429c5e640 10 monclient(hunting): start opening mon connection 2024-09-16T04:04:27.369 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.346+0000 7fd429c5e640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fd42413fd00 con 0x7fd424135100 2024-09-16T04:04:27.369 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.346+0000 7fd429c5e640 10 monclient(hunting): start opening mon connection 2024-09-16T04:04:27.369 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.346+0000 7fd429c5e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd42413fb80 con 0x7fd424134a70 2024-09-16T04:04:27.369 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.346+0000 7fd429c5e640 10 monclient(hunting): start opening mon connection 2024-09-16T04:04:27.369 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.346+0000 7fd429c5e640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fd42413fe80 con 0x7fd4240a1330 2024-09-16T04:04:27.369 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.346+0000 7fd429c5e640 10 monclient(hunting): _renew_subs 2024-09-16T04:04:27.369 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.346+0000 7fd429c5e640 10 monclient(hunting): authenticate will time out at 2314.319897s 2024-09-16T04:04:27.369 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.350+0000 7fd4237fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd4240a1330 0x7fd42413b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:27.369 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.350+0000 7fd422ffd640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd424135100 0x7fd424134530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:27.369 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.350+0000 7fd4237fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd4240a1330 0x7fd42413b180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:52356/0 (socket says 172.21.15.42:52356) 2024-09-16T04:04:27.369 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.350+0000 7fd422ffd640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd424135100 0x7fd424134530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:39644/0 (socket says 172.21.15.42:39644) 2024-09-16T04:04:27.369 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.350+0000 7fd4237fe640 1 -- 172.21.15.42:0/3803304700 learned_addr learned my addr 172.21.15.42:0/3803304700 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:04:27.369 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.350+0000 7fd4237fe640 10 monclient(hunting): get_auth_request con 0x7fd4240a1330 auth_method 0 2024-09-16T04:04:27.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.350+0000 7fd4237fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:04:27.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.350+0000 7fd4237fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:04:27.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.350+0000 7fd4237fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:04:27.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.350+0000 7fd422ffd640 10 monclient(hunting): get_auth_request con 0x7fd424135100 auth_method 0 2024-09-16T04:04:27.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.350+0000 7fd422ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:04:27.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.350+0000 7fd422ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:04:27.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.350+0000 7fd422ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:04:27.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.350+0000 7fd4237fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:04:27.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.350+0000 7fd4237fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:04:27.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.350+0000 7fd4237fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:04:27.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.350+0000 7fd422ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:04:27.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.350+0000 7fd422ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:04:27.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.350+0000 7fd422ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:04:27.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.350+0000 7fd423fff640 1 --2- 172.21.15.42:0/3803304700 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd424134a70 0x7fd42409cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:27.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.350+0000 7fd423fff640 10 monclient(hunting): get_auth_request con 0x7fd424134a70 auth_method 0 2024-09-16T04:04:27.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.350+0000 7fd423fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:04:27.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.350+0000 7fd423fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:04:27.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.350+0000 7fd423fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:04:27.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.350+0000 7fd4237fe640 10 monclient(hunting): handle_auth_done global_id 974201 payload 824 2024-09-16T04:04:27.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.350+0000 7fd4237fe640 1 -- 172.21.15.42:0/3803304700 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd424134a70 msgr2=0x7fd42409cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:04:27.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.350+0000 7fd4237fe640 1 --2- 172.21.15.42:0/3803304700 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd424134a70 0x7fd42409cb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:27.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.350+0000 7fd4237fe640 1 -- 172.21.15.42:0/3803304700 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd424135100 msgr2=0x7fd424134530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:04:27.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.350+0000 7fd4237fe640 1 --2- 172.21.15.42:0/3803304700 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd424135100 0x7fd424134530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:27.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.350+0000 7fd4237fe640 10 monclient: _finish_hunting 0 2024-09-16T04:04:27.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.350+0000 7fd4237fe640 1 monclient: found mon.noname-c 2024-09-16T04:04:27.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.350+0000 7fd4237fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:04:27.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.350+0000 7fd4237fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T04:04:27.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.350+0000 7fd4237fe640 1 -- 172.21.15.42:0/3803304700 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4240a5060 con 0x7fd4240a1330 2024-09-16T04:04:27.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.350+0000 7fd4237fe640 1 --2- 172.21.15.42:0/3803304700 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd4240a1330 0x7fd42413b180 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fd41c0b0040 tx=0x7fd4240a22d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:04:27.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.350+0000 7fd41bfff640 10 client.?.objecter ms_handle_connect 0x7fd4240a1330 2024-09-16T04:04:27.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.350+0000 7fd41bfff640 10 client.?.objecter resend_mon_ops 2024-09-16T04:04:27.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.350+0000 7fd41bfff640 1 -- 172.21.15.42:0/3803304700 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd41c0bc020 con 0x7fd4240a1330 2024-09-16T04:04:27.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.350+0000 7fd41bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:04:27.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.350+0000 7fd41bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:04:27.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.350+0000 7fd41bfff640 10 monclient: dump: 2024-09-16T04:04:27.372 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:04:27.372 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:04:27.372 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:27.372 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:27.373 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:04:27.373 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:04:27.373 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:04:27.373 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:04:27.373 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:04:27.373 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:04:27.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.350+0000 7fd41bfff640 10 monclient: _finish_auth 0 2024-09-16T04:04:27.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.350+0000 7fd41bfff640 10 monclient: _check_auth_tickets 2024-09-16T04:04:27.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.350+0000 7fd41bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:04:27.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.350+0000 7fd429c5e640 5 monclient: authenticate success, global_id 974201 2024-09-16T04:04:27.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.350+0000 7fd41bfff640 1 -- 172.21.15.42:0/3803304700 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd41c0ad640 con 0x7fd4240a1330 2024-09-16T04:04:27.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.350+0000 7fd41bfff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:04:27.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.350+0000 7fd41bfff640 1 -- 172.21.15.42:0/3803304700 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd41c0b2040 con 0x7fd4240a1330 2024-09-16T04:04:27.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.350+0000 7fd41bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:04:27.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.350+0000 7fd41bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:04:27.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.350+0000 7fd421ffb640 4 set_mon_vals no callback set 2024-09-16T04:04:27.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.350+0000 7fd41bfff640 10 monclient: dump: 2024-09-16T04:04:27.374 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:04:27.374 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:04:27.374 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:27.374 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:27.374 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:04:27.374 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:04:27.374 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:04:27.374 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:04:27.374 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:04:27.374 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:04:27.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.350+0000 7fd429c5e640 10 monclient: _renew_subs 2024-09-16T04:04:27.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.350+0000 7fd429c5e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:04:27.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.350+0000 7fd429c5e640 1 -- 172.21.15.42:0/3803304700 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd424120920 con 0x7fd4240a1330 2024-09-16T04:04:27.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.350+0000 7fd429c5e640 10 client.974201.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:04:27.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.350+0000 7fd429c5e640 10 monclient: _renew_subs 2024-09-16T04:04:27.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.350+0000 7fd429c5e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:04:27.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.350+0000 7fd429c5e640 1 -- 172.21.15.42:0/3803304700 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd424071ae0 con 0x7fd4240a1330 2024-09-16T04:04:27.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.350+0000 7fd429c5e640 1 librados: init done 2024-09-16T04:04:27.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.350+0000 7fd429c5e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:04:27.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.350+0000 7fd429c5e640 1 --2- 172.21.15.42:0/3803304700 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd42406dac0 0x7fd42409d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:04:27.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.350+0000 7fd429c5e640 10 monclient(hunting): start opening mon connection 2024-09-16T04:04:27.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.350+0000 7fd429c5e640 1 -- 172.21.15.42:0/3803304700 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fd4241cf7a0 con 0x7fd42406dac0 2024-09-16T04:04:27.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.350+0000 7fd422ffd640 1 --2- 172.21.15.42:0/3803304700 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd42406dac0 0x7fd42409d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:27.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.350+0000 7fd422ffd640 10 monclient: get_auth_request con 0x7fd42406dac0 auth_method 0 2024-09-16T04:04:27.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.350+0000 7fd422ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:04:27.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.350+0000 7fd422ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:04:27.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.350+0000 7fd422ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:04:27.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.350+0000 7fd41bfff640 1 -- 172.21.15.42:0/3803304700 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fd41c0bf020 con 0x7fd4240a1330 2024-09-16T04:04:27.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.350+0000 7fd422ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:04:27.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.350+0000 7fd422ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:04:27.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.350+0000 7fd422ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:04:27.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.350+0000 7fd422ffd640 10 monclient(hunting): handle_auth_done global_id 974204 payload 824 2024-09-16T04:04:27.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.350+0000 7fd422ffd640 1 -- 172.21.15.42:0/3803304700 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd424073a30 con 0x7fd42406dac0 2024-09-16T04:04:27.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.350+0000 7fd41bfff640 1 --2- 172.21.15.42:0/3803304700 >> v2:172.21.15.154:6812/254845342 conn(0x7fd404037810 0x7fd404039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:04:27.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.350+0000 7fd41bfff640 1 -- 172.21.15.42:0/3803304700 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(678..678 src has 451..678) ==== 6280+0+0 (secure 0 0 0) 0x7fd41c0e3d10 con 0x7fd4240a1330 2024-09-16T04:04:27.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.350+0000 7fd41bfff640 10 client.974201.objecter ms_dispatch 0x7fd424000ba0 osd_map(678..678 src has 451..678) 2024-09-16T04:04:27.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.350+0000 7fd41bfff640 3 client.974201.objecter handle_osd_map got epochs [678,678] > 0 2024-09-16T04:04:27.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.350+0000 7fd41bfff640 3 client.974201.objecter handle_osd_map decoding full epoch 678 2024-09-16T04:04:27.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.350+0000 7fd422ffd640 1 --2- 172.21.15.42:0/3803304700 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd42406dac0 0x7fd42409d0a0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fd42413ba10 tx=0x7fd40c011d80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:04:27.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.350+0000 7fd41bfff640 20 client.974201.objecter dump_active .. 0 homeless 2024-09-16T04:04:27.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.350+0000 7fd41bfff640 10 client.974201.objecter ms_handle_connect 0x7fd42406dac0 2024-09-16T04:04:27.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.350+0000 7fd41bfff640 10 client.974201.objecter resend_mon_ops 2024-09-16T04:04:27.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.350+0000 7fd41bfff640 1 -- 172.21.15.42:0/3803304700 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd40c00fa70 con 0x7fd42406dac0 2024-09-16T04:04:27.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.350+0000 7fd41bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:04:27.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.350+0000 7fd41bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:04:27.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.350+0000 7fd41bfff640 10 monclient: dump: 2024-09-16T04:04:27.377 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:04:27.377 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:04:27.377 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:27.377 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:27.378 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:04:27.378 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:04:27.378 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:04:27.378 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:04:27.378 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:04:27.378 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:04:27.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.350+0000 7fd423fff640 1 --2- 172.21.15.42:0/3803304700 >> v2:172.21.15.154:6812/254845342 conn(0x7fd404037810 0x7fd404039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:27.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.350+0000 7fd423fff640 10 monclient: get_auth_request con 0x7fd404037810 auth_method 0 2024-09-16T04:04:27.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.350+0000 7fd41bfff640 1 -- 172.21.15.42:0/3803304700 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fd40c00fa70 con 0x7fd42406dac0 2024-09-16T04:04:27.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.350+0000 7fd41bfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:04:27.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.350+0000 7fd41bfff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:04:27.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.350+0000 7fd41bfff640 1 -- 172.21.15.42:0/3803304700 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd42406dac0 msgr2=0x7fd42409d0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:04:27.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.350+0000 7fd41bfff640 1 --2- 172.21.15.42:0/3803304700 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd42406dac0 0x7fd42409d0a0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fd42413ba10 tx=0x7fd40c011d80 comp rx=0 tx=0).stop 2024-09-16T04:04:27.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.350+0000 7fd41bfff640 1 -- 172.21.15.42:0/3803304700 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd42406dac0 msgr2=0x7fd42409d0a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:04:27.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.350+0000 7fd41bfff640 1 --2- 172.21.15.42:0/3803304700 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd42406dac0 0x7fd42409d0a0 secure :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fd42413ba10 tx=0x7fd40c011d80 comp rx=0 tx=0).stop 2024-09-16T04:04:27.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.354+0000 7fd41bfff640 10 client.974201.objecter ms_handle_connect 0x7fd404037810 2024-09-16T04:04:27.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.354+0000 7fd423fff640 1 --2- 172.21.15.42:0/3803304700 >> v2:172.21.15.154:6812/254845342 conn(0x7fd404037810 0x7fd404039cd0 secure :-1 s=READY pgs=3822 cs=0 l=1 rev1=1 crypto rx=0x7fd414000f70 tx=0x7fd414009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:04:27.380 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.358+0000 7fd429c5e640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:04:27.380 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.358+0000 7fd429c5e640 1 --2- 172.21.15.42:0/3803304700 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd42409d5e0 0x7fd424138330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:04:27.380 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.358+0000 7fd429c5e640 10 monclient(hunting): start opening mon connection 2024-09-16T04:04:27.380 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.358+0000 7fd429c5e640 1 -- 172.21.15.42:0/3803304700 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fd42409d9f0 con 0x7fd42409d5e0 2024-09-16T04:04:27.380 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.358+0000 7fd422ffd640 1 --2- 172.21.15.42:0/3803304700 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd42409d5e0 0x7fd424138330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:27.380 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.358+0000 7fd422ffd640 10 monclient: get_auth_request con 0x7fd42409d5e0 auth_method 0 2024-09-16T04:04:27.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.358+0000 7fd422ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:04:27.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.358+0000 7fd422ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:04:27.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.358+0000 7fd422ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:04:27.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.358+0000 7fd422ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:04:27.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.358+0000 7fd422ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:04:27.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.358+0000 7fd422ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:04:27.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.358+0000 7fd422ffd640 10 monclient(hunting): handle_auth_done global_id 974207 payload 824 2024-09-16T04:04:27.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.358+0000 7fd422ffd640 1 -- 172.21.15.42:0/3803304700 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fd42409e5d0 con 0x7fd42409d5e0 2024-09-16T04:04:27.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.358+0000 7fd41bfff640 10 client.974201.objecter ms_handle_connect 0x7fd42409d5e0 2024-09-16T04:04:27.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.358+0000 7fd41bfff640 10 client.974201.objecter resend_mon_ops 2024-09-16T04:04:27.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.358+0000 7fd422ffd640 1 --2- 172.21.15.42:0/3803304700 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd42409d5e0 0x7fd424138330 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fd42409e350 tx=0x7fd40c0046d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:04:27.382 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.358+0000 7fd41bfff640 1 -- 172.21.15.42:0/3803304700 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd40c014640 con 0x7fd42409d5e0 2024-09-16T04:04:27.382 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.358+0000 7fd41bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:04:27.382 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.358+0000 7fd41bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:04:27.382 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.358+0000 7fd41bfff640 10 monclient: dump: 2024-09-16T04:04:27.382 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:04:27.382 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:04:27.382 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:27.382 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:27.382 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:04:27.382 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:04:27.383 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:04:27.383 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:04:27.383 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:04:27.383 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:04:27.383 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T04:04:27.383 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:04:27.383 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:04:27.383 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T04:04:27.383 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 210, 2024-09-16T04:04:27.384 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 75859, 2024-09-16T04:04:27.384 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T04:04:27.384 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T04:04:27.384 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T04:04:27.384 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T04:04:27.384 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:04:27.384 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 1, 2024-09-16T04:04:27.384 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:04:27.384 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T04:04:27.384 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T04:04:27.384 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:04:27.385 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:04:27.385 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:04:27.385 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:04:27.385 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:04:27.385 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:04:27.385 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:04:27.385 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:04:27.385 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:04:27.385 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:04:27.385 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:04:27.385 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:04:27.385 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:04:27.386 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T04:04:27.386 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:04:27.386 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:04:27.386 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:04:27.386 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:04:27.386 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:04:27.386 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:04:27.386 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:04:27.386 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:04:27.386 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:04:27.386 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:04:27.386 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:04:27.386 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:04:27.387 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:04:27.387 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T04:04:27.387 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T04:04:27.387 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T04:04:27.387 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T04:04:27.387 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T04:04:27.387 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T04:04:27.387 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:04:27.387 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:04:27.387 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T04:04:27.387 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T04:04:27.387 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T04:04:27.387 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T04:04:27.388 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T04:04:27.388 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T04:04:27.388 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T04:04:27.388 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:04:27.388 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T04:04:27.388 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:04:27.388 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:04:27.388 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:04:27.388 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:04:27.388 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:04:27.388 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:04:27.388 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:04:27.388 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:04:27.389 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:04:27.389 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:04:27.389 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:04:27.389 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:04:27.389 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T04:04:27.389 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:04:27.389 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T04:04:27.389 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:04:27.389 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:04:27.389 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:04:27.389 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:04:27.389 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:04:27.389 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:04:27.390 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:04:27.390 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T04:04:27.390 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:04:27.390 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:04:27.390 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:04:27.390 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:04:27.390 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T04:04:27.390 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:04:27.390 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:04:27.391 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:04:27.391 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:04:27.391 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T04:04:27.391 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T04:04:27.391 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:04:27.391 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:04:27.391 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:04:27.391 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:04:27.391 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:04:27.391 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:04:27.391 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:04:27.391 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:04:27.391 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:04:27.392 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:04:27.392 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:04:27.392 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T04:04:27.392 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:04:27.392 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:04:27.392 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:04:27.392 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:04:27.392 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T04:04:27.392 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:04:27.392 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:04:27.392 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:04:27.392 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:04:27.392 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T04:04:27.393 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T04:04:27.393 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:04:27.393 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:04:27.393 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:04:27.393 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:04:27.393 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:04:27.393 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:04:27.393 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:04:27.393 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:04:27.393 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:04:27.393 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:04:27.393 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:04:27.393 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T04:04:27.394 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:04:27.394 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:04:27.394 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:04:27.394 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:04:27.394 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T04:04:27.394 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:04:27.394 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:04:27.394 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:04:27.394 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:04:27.394 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T04:04:27.394 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T04:04:27.394 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:04:27.394 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:04:27.394 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:04:27.394 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:04:27.395 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:04:27.395 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:04:27.395 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T04:04:27.395 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T04:04:27.395 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:04:27.395 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:04:27.395 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:04:27.395 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:04:27.395 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:04:27.395 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:04:27.395 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T04:04:27.395 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:04:27.395 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:04:27.396 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:04:27.396 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 2 2024-09-16T04:04:27.396 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:04:27.396 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:04:27.396 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:04:27.396 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T04:04:27.396 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T04:04:27.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.358+0000 7fd41bfff640 1 -- 172.21.15.42:0/3803304700 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4545 (secure 0 0 0) 0x7fd40c02a420 con 0x7fd42409d5e0 2024-09-16T04:04:27.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.358+0000 7fd41bfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:04:27.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.358+0000 7fd41bfff640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:04:27.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.358+0000 7fd41bfff640 1 -- 172.21.15.42:0/3803304700 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd42409d5e0 msgr2=0x7fd424138330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:04:27.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.358+0000 7fd41bfff640 1 --2- 172.21.15.42:0/3803304700 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd42409d5e0 0x7fd424138330 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fd42409e350 tx=0x7fd40c0046d0 comp rx=0 tx=0).stop 2024-09-16T04:04:27.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.358+0000 7fd41bfff640 1 -- 172.21.15.42:0/3803304700 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd42409d5e0 msgr2=0x7fd424138330 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:04:27.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.358+0000 7fd41bfff640 1 --2- 172.21.15.42:0/3803304700 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd42409d5e0 0x7fd424138330 secure :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fd42409e350 tx=0x7fd40c0046d0 comp rx=0 tx=0).stop 2024-09-16T04:04:27.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.358+0000 7fd429c5e640 10 librados: watch_flush enter 2024-09-16T04:04:27.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.358+0000 7fd429c5e640 10 librados: watch_flush exit 2024-09-16T04:04:27.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.358+0000 7fd429c5e640 20 client.974201.objecter shutdown clearing up homeless session... 2024-09-16T04:04:27.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.358+0000 7fd429c5e640 10 client.974201.objecter successfully canceled tick 2024-09-16T04:04:27.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.358+0000 7fd429c5e640 1 -- 172.21.15.42:0/3803304700 >> v2:172.21.15.154:6812/254845342 conn(0x7fd404037810 msgr2=0x7fd404039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:04:27.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.358+0000 7fd429c5e640 1 --2- 172.21.15.42:0/3803304700 >> v2:172.21.15.154:6812/254845342 conn(0x7fd404037810 0x7fd404039cd0 secure :-1 s=READY pgs=3822 cs=0 l=1 rev1=1 crypto rx=0x7fd414000f70 tx=0x7fd414009040 comp rx=0 tx=0).stop 2024-09-16T04:04:27.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.358+0000 7fd429c5e640 10 monclient: shutdown 2024-09-16T04:04:27.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.358+0000 7fd429c5e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:04:27.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.358+0000 7fd429c5e640 1 -- 172.21.15.42:0/3803304700 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd4240a1330 msgr2=0x7fd42413b180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:04:27.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.358+0000 7fd429c5e640 1 --2- 172.21.15.42:0/3803304700 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd4240a1330 0x7fd42413b180 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fd41c0b0040 tx=0x7fd4240a22d0 comp rx=0 tx=0).stop 2024-09-16T04:04:27.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.358+0000 7fd423fff640 1 -- 172.21.15.42:0/3803304700 reap_dead start 2024-09-16T04:04:27.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.358+0000 7fd423fff640 1 -- 172.21.15.42:0/3803304700 reap_dead start 2024-09-16T04:04:27.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.358+0000 7fd429c5e640 1 -- 172.21.15.42:0/3803304700 shutdown_connections 2024-09-16T04:04:27.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.358+0000 7fd429c5e640 1 -- 172.21.15.42:0/3803304700 >> 172.21.15.42:0/3803304700 conn(0x7fd4240a2420 msgr2=0x7fd42413c590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:04:27.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.358+0000 7fd429c5e640 1 -- 172.21.15.42:0/3803304700 shutdown_connections 2024-09-16T04:04:27.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.358+0000 7fd429c5e640 1 -- 172.21.15.42:0/3803304700 wait complete. 2024-09-16T04:04:27.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.358+0000 7fd429c5e640 1 librados: shutdown 2024-09-16T04:04:27.400 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:04:27.400 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T04:04:27.400 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T04:04:27.522 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.518+0000 7f8211399640 10 monclient: get_monmap_and_config 2024-09-16T04:04:27.522 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.518+0000 7f8211399640 10 monclient: build_initial_monmap 2024-09-16T04:04:27.522 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.518+0000 7f8211399640 10 monclient: monmap: 2024-09-16T04:04:27.522 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:04:27.523 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:04:27.523 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:04:27.520820+0000 2024-09-16T04:04:27.523 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:04:27.520820+0000 2024-09-16T04:04:27.523 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:04:27.523 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:04:27.523 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:04:27.523 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:04:27.523 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:04:27.523 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:04:27.523 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:04:27.523 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:04:27.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.518+0000 7f8211399640 1 Processor -- start 2024-09-16T04:04:27.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.518+0000 7f8211399640 1 -- start start 2024-09-16T04:04:27.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.518+0000 7f8211399640 10 monclient: init 2024-09-16T04:04:27.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.518+0000 7f8211399640 10 monclient: _reopen_session rank -1 2024-09-16T04:04:27.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.518+0000 7f8211399640 10 monclient: _add_conns ranks=[1,4,2,3,0] 2024-09-16T04:04:27.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.518+0000 7f8211399640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f820c073080 addr v1:172.21.15.154:6789/0 2024-09-16T04:04:27.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.518+0000 7f8211399640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f820c073460 0x7f820c13a100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:04:27.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.518+0000 7f8211399640 10 monclient(hunting): picked mon.noname-c con 0x7f820c073460 addr v2:172.21.15.154:3300/0 2024-09-16T04:04:27.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.518+0000 7f8211399640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f820c073890 0x7f820c13c650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:04:27.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.518+0000 7f8211399640 10 monclient(hunting): picked mon.noname-b con 0x7f820c073890 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:04:27.527 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.518+0000 7f8211399640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f820c1355f0 con 0x7f820c073080 2024-09-16T04:04:27.527 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.518+0000 7f8211399640 10 monclient(hunting): start opening mon connection 2024-09-16T04:04:27.527 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.518+0000 7f8211399640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f820c0a1a10 con 0x7f820c073460 2024-09-16T04:04:27.527 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.518+0000 7f8211399640 10 monclient(hunting): start opening mon connection 2024-09-16T04:04:27.527 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.518+0000 7f8211399640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f820c0a1cd0 con 0x7f820c073890 2024-09-16T04:04:27.527 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.518+0000 7f8211399640 10 monclient(hunting): _renew_subs 2024-09-16T04:04:27.527 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.518+0000 7f8211399640 10 monclient(hunting): authenticate will time out at 2314.490376s 2024-09-16T04:04:27.527 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.518+0000 7f820b7fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f820c073890 0x7f820c13c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:27.527 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.518+0000 7f820b7fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f820c073890 0x7f820c13c650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:52400/0 (socket says 172.21.15.42:52400) 2024-09-16T04:04:27.527 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.518+0000 7f820b7fe640 1 -- 172.21.15.42:0/3193614762 learned_addr learned my addr 172.21.15.42:0/3193614762 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:04:27.527 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.518+0000 7f820b7fe640 10 monclient(hunting): get_auth_request con 0x7f820c073890 auth_method 0 2024-09-16T04:04:27.528 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.518+0000 7f820b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:04:27.528 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.518+0000 7f820b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:04:27.528 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.518+0000 7f820b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:04:27.528 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.518+0000 7f820a7fc640 1 --2- 172.21.15.42:0/3193614762 >> v2:172.21.15.154:3300/0 conn(0x7f820c073460 0x7f820c13a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:27.528 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.518+0000 7f820b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:04:27.528 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.518+0000 7f820b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:04:27.528 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.518+0000 7f820b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:04:27.528 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.518+0000 7f820a7fc640 10 monclient(hunting): get_auth_request con 0x7f820c073460 auth_method 0 2024-09-16T04:04:27.528 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.518+0000 7f820a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:04:27.528 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.518+0000 7f820a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:04:27.528 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.518+0000 7f820a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:04:27.529 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.518+0000 7f820b7fe640 10 monclient(hunting): handle_auth_done global_id 974210 payload 274 2024-09-16T04:04:27.529 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.518+0000 7f820b7fe640 1 -- 172.21.15.42:0/3193614762 >> v2:172.21.15.154:3300/0 conn(0x7f820c073460 msgr2=0x7f820c13a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:04:27.529 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.518+0000 7f820b7fe640 1 --2- 172.21.15.42:0/3193614762 >> v2:172.21.15.154:3300/0 conn(0x7f820c073460 0x7f820c13a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:27.529 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.518+0000 7f820b7fe640 1 -- 172.21.15.42:0/3193614762 >> v1:172.21.15.154:6789/0 conn(0x7f820c073080 legacy=0x7f820c134ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:04:27.529 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.518+0000 7f820b7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:04:27.529 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.518+0000 7f820b7fe640 1 monclient: found mon.noname-b 2024-09-16T04:04:27.529 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.518+0000 7f820b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:04:27.529 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.518+0000 7f820b7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T04:04:27.529 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.518+0000 7f820b7fe640 1 -- 172.21.15.42:0/3193614762 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f820c0a1fb0 con 0x7f820c073890 2024-09-16T04:04:27.538 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.522+0000 7f820b7fe640 1 --2- 172.21.15.42:0/3193614762 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f820c073890 0x7f820c13c650 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f8204009e50 tx=0x7f820401c8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1b9ac15e2baf9ec9 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:04:27.538 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.522+0000 7f8209ffb640 1 -- 172.21.15.42:0/3193614762 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82040a5640 con 0x7f820c073890 2024-09-16T04:04:27.538 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.522+0000 7f8209ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:04:27.538 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.522+0000 7f8209ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:04:27.538 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.522+0000 7f8209ffb640 10 monclient: dump: 2024-09-16T04:04:27.538 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:04:27.538 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:04:27.538 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:27.538 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:27.538 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:04:27.539 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:04:27.539 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:04:27.539 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:04:27.539 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:04:27.539 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:04:27.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.522+0000 7f8209ffb640 10 monclient: _finish_auth 0 2024-09-16T04:04:27.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.522+0000 7f8209ffb640 10 monclient: _check_auth_tickets 2024-09-16T04:04:27.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.522+0000 7f8209ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:04:27.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.522+0000 7f8209ffb640 1 -- 172.21.15.42:0/3193614762 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f82040a57e0 con 0x7f820c073890 2024-09-16T04:04:27.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.522+0000 7f8209ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:04:27.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.522+0000 7f8209ffb640 1 -- 172.21.15.42:0/3193614762 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82040a5bb0 con 0x7f820c073890 2024-09-16T04:04:27.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.522+0000 7f8211399640 5 monclient: authenticate success, global_id 974210 2024-09-16T04:04:27.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.522+0000 7f8211399640 10 monclient: get_monmap_and_config success 2024-09-16T04:04:27.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.522+0000 7f8211399640 4 set_mon_vals no callback set 2024-09-16T04:04:27.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.522+0000 7f8209ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:04:27.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.522+0000 7f8209ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:04:27.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.522+0000 7f8209ffb640 10 monclient: dump: 2024-09-16T04:04:27.540 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:04:27.540 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:04:27.541 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:27.541 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:27.541 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:04:27.541 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:04:27.541 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:04:27.541 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:04:27.541 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:04:27.541 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:04:27.541 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.522+0000 7f8211399640 10 monclient: shutdown 2024-09-16T04:04:27.541 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.522+0000 7f8211399640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:04:27.541 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.522+0000 7f8211399640 1 -- 172.21.15.42:0/3193614762 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f820c073890 msgr2=0x7f820c13c650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:04:27.541 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.522+0000 7f8211399640 1 --2- 172.21.15.42:0/3193614762 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f820c073890 0x7f820c13c650 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f8204009e50 tx=0x7f820401c8b0 comp rx=0 tx=0).stop 2024-09-16T04:04:27.542 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.522+0000 7f8211399640 1 -- 172.21.15.42:0/3193614762 shutdown_connections 2024-09-16T04:04:27.542 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.522+0000 7f8211399640 1 --2- 172.21.15.42:0/3193614762 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f820c073890 0x7f820c13c650 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:27.542 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.522+0000 7f8211399640 1 --2- 172.21.15.42:0/3193614762 >> v2:172.21.15.154:3300/0 conn(0x7f820c073460 0x7f820c13a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:27.542 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.522+0000 7f8211399640 1 -- 172.21.15.42:0/3193614762 >> 172.21.15.42:0/3193614762 conn(0x7f820c06bc40 msgr2=0x7f820c06c050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:04:27.542 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.522+0000 7f8211399640 1 -- 172.21.15.42:0/3193614762 shutdown_connections 2024-09-16T04:04:27.542 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.522+0000 7f8211399640 1 -- 172.21.15.42:0/3193614762 wait complete. 2024-09-16T04:04:27.542 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.522+0000 7f8211399640 10 monclient: build_initial_monmap 2024-09-16T04:04:27.542 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.522+0000 7f8211399640 10 monclient: monmap: 2024-09-16T04:04:27.542 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:04:27.542 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:04:27.543 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:04:27.543 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:04:27.543 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:04:27.543 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:04:27.543 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:04:27.543 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:04:27.543 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:04:27.544 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:04:27.544 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.526+0000 7f8211399640 1 librados: starting msgr at 2024-09-16T04:04:27.544 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.526+0000 7f8211399640 1 librados: starting objecter 2024-09-16T04:04:27.544 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.526+0000 7f8211399640 1 Processor -- start 2024-09-16T04:04:27.544 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.526+0000 7f8211399640 1 -- start start 2024-09-16T04:04:27.544 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.526+0000 7f8211399640 1 librados: setting wanted keys 2024-09-16T04:04:27.544 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.526+0000 7f8211399640 1 librados: calling monclient init 2024-09-16T04:04:27.544 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.526+0000 7f8211399640 10 monclient: init 2024-09-16T04:04:27.544 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.526+0000 7f8211399640 10 monclient: _reopen_session rank -1 2024-09-16T04:04:27.544 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.526+0000 7f8211399640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:04:27.544 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.526+0000 7f8211399640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f820c073080 0x7f820c12f130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:04:27.545 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.526+0000 7f8211399640 10 monclient(hunting): picked mon.noname-b con 0x7f820c073080 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:04:27.545 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.526+0000 7f8211399640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f820c073460 0x7f820c137e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:04:27.545 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.526+0000 7f8211399640 10 monclient(hunting): picked mon.noname-a con 0x7f820c073460 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:04:27.545 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.526+0000 7f8211399640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f820c073890 0x7f820c138350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:04:27.545 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.526+0000 7f8211399640 10 monclient(hunting): picked mon.noname-c con 0x7f820c073890 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:04:27.545 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.526+0000 7f8211399640 10 monclient(hunting): start opening mon connection 2024-09-16T04:04:27.545 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.526+0000 7f8211399640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f820c13f740 con 0x7f820c073460 2024-09-16T04:04:27.545 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.526+0000 7f8211399640 10 monclient(hunting): start opening mon connection 2024-09-16T04:04:27.545 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.526+0000 7f8211399640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f820c13f5c0 con 0x7f820c073080 2024-09-16T04:04:27.545 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.526+0000 7f8211399640 10 monclient(hunting): start opening mon connection 2024-09-16T04:04:27.545 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.526+0000 7f8211399640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f820c13f8c0 con 0x7f820c073890 2024-09-16T04:04:27.545 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.526+0000 7f8211399640 10 monclient(hunting): _renew_subs 2024-09-16T04:04:27.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.526+0000 7f8211399640 10 monclient(hunting): authenticate will time out at 2314.496783s 2024-09-16T04:04:27.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.526+0000 7f820b7fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f820c073890 0x7f820c138350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:27.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.526+0000 7f820a7fc640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f820c073460 0x7f820c137e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:27.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.526+0000 7f820b7fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f820c073890 0x7f820c138350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:52410/0 (socket says 172.21.15.42:52410) 2024-09-16T04:04:27.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.526+0000 7f820b7fe640 1 -- 172.21.15.42:0/3595300856 learned_addr learned my addr 172.21.15.42:0/3595300856 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:04:27.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.526+0000 7f820b7fe640 10 monclient(hunting): get_auth_request con 0x7f820c073890 auth_method 0 2024-09-16T04:04:27.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.526+0000 7f820b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:04:27.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.526+0000 7f820b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:04:27.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.526+0000 7f820b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:04:27.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.526+0000 7f820a7fc640 10 monclient(hunting): get_auth_request con 0x7f820c073460 auth_method 0 2024-09-16T04:04:27.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.526+0000 7f820a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:04:27.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.526+0000 7f820a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:04:27.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.526+0000 7f820a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:04:27.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.526+0000 7f820b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:04:27.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.526+0000 7f820b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:04:27.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.526+0000 7f820b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:04:27.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.526+0000 7f820a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:04:27.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.526+0000 7f820a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:04:27.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.526+0000 7f820a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:04:27.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.526+0000 7f820affd640 1 --2- 172.21.15.42:0/3595300856 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f820c073080 0x7f820c12f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:27.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.526+0000 7f820affd640 10 monclient(hunting): get_auth_request con 0x7f820c073080 auth_method 0 2024-09-16T04:04:27.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.526+0000 7f820affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:04:27.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.526+0000 7f820affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:04:27.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.526+0000 7f820affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:04:27.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.526+0000 7f820b7fe640 10 monclient(hunting): handle_auth_done global_id 974213 payload 824 2024-09-16T04:04:27.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.526+0000 7f820b7fe640 1 -- 172.21.15.42:0/3595300856 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f820c073080 msgr2=0x7f820c12f130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:04:27.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.526+0000 7f820b7fe640 1 --2- 172.21.15.42:0/3595300856 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f820c073080 0x7f820c12f130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:27.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.526+0000 7f820b7fe640 1 -- 172.21.15.42:0/3595300856 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f820c073460 msgr2=0x7f820c137e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:04:27.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.526+0000 7f820b7fe640 1 --2- 172.21.15.42:0/3595300856 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f820c073460 0x7f820c137e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:27.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.526+0000 7f820b7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:04:27.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.526+0000 7f820b7fe640 1 monclient: found mon.noname-c 2024-09-16T04:04:27.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.526+0000 7f820b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:04:27.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.526+0000 7f820b7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T04:04:27.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.526+0000 7f820b7fe640 1 -- 172.21.15.42:0/3595300856 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f820c13a390 con 0x7f820c073890 2024-09-16T04:04:27.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.526+0000 7f820b7fe640 1 --2- 172.21.15.42:0/3595300856 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f820c073890 0x7f820c138350 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f820401af60 tx=0x7f8204019650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:04:27.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.526+0000 7f81f37fe640 10 client.?.objecter ms_handle_connect 0x7f820c073890 2024-09-16T04:04:27.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.526+0000 7f81f37fe640 10 client.?.objecter resend_mon_ops 2024-09-16T04:04:27.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.526+0000 7f81f37fe640 1 -- 172.21.15.42:0/3595300856 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82040bc020 con 0x7f820c073890 2024-09-16T04:04:27.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.526+0000 7f81f37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:04:27.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.526+0000 7f81f37fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:04:27.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.526+0000 7f81f37fe640 10 monclient: dump: 2024-09-16T04:04:27.549 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:04:27.549 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:04:27.549 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:27.549 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:27.550 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:04:27.550 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:04:27.550 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:04:27.550 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:04:27.550 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:04:27.550 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:04:27.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.526+0000 7f81f37fe640 10 monclient: _finish_auth 0 2024-09-16T04:04:27.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.526+0000 7f81f37fe640 10 monclient: _check_auth_tickets 2024-09-16T04:04:27.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.526+0000 7f81f37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:04:27.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.526+0000 7f81f37fe640 1 -- 172.21.15.42:0/3595300856 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f82040b88c0 con 0x7f820c073890 2024-09-16T04:04:27.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.526+0000 7f8211399640 5 monclient: authenticate success, global_id 974213 2024-09-16T04:04:27.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.526+0000 7f8211399640 10 monclient: _renew_subs 2024-09-16T04:04:27.551 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.526+0000 7f8211399640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:04:27.551 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.526+0000 7f8211399640 1 -- 172.21.15.42:0/3595300856 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f820c13bda0 con 0x7f820c073890 2024-09-16T04:04:27.551 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.526+0000 7f81f37fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:04:27.551 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.526+0000 7f81f37fe640 1 -- 172.21.15.42:0/3595300856 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8204009510 con 0x7f820c073890 2024-09-16T04:04:27.551 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.526+0000 7f81f37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:04:27.551 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.526+0000 7f81f37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:04:27.551 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.526+0000 7f81f37fe640 10 monclient: dump: 2024-09-16T04:04:27.551 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:04:27.551 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:04:27.551 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:27.551 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:27.551 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:04:27.552 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:04:27.552 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:04:27.552 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:04:27.552 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:04:27.552 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:04:27.552 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.526+0000 7f8211399640 10 client.974213.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:04:27.552 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.526+0000 7f8211399640 10 monclient: _renew_subs 2024-09-16T04:04:27.552 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.526+0000 7f8211399640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:04:27.552 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.526+0000 7f8211399640 1 -- 172.21.15.42:0/3595300856 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f820c0729d0 con 0x7f820c073890 2024-09-16T04:04:27.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.526+0000 7f8211399640 1 librados: init done 2024-09-16T04:04:27.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.526+0000 7f82097fa640 4 set_mon_vals no callback set 2024-09-16T04:04:27.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.526+0000 7f8211399640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:04:27.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.526+0000 7f8211399640 1 --2- 172.21.15.42:0/3595300856 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f820c138890 0x7f820c13b740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:04:27.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.526+0000 7f8211399640 10 monclient(hunting): start opening mon connection 2024-09-16T04:04:27.557 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.526+0000 7f8211399640 1 -- 172.21.15.42:0/3595300856 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f820c138c70 con 0x7f820c138890 2024-09-16T04:04:27.557 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.526+0000 7f81f37fe640 1 -- 172.21.15.42:0/3595300856 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f82040bf020 con 0x7f820c073890 2024-09-16T04:04:27.557 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.526+0000 7f81f37fe640 1 --2- 172.21.15.42:0/3595300856 >> v2:172.21.15.154:6812/254845342 conn(0x7f81f8037810 0x7f81f8039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:04:27.557 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.526+0000 7f81f37fe640 1 -- 172.21.15.42:0/3595300856 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(679..679 src has 451..679) ==== 6280+0+0 (secure 0 0 0) 0x7f82040e3e30 con 0x7f820c073890 2024-09-16T04:04:27.557 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.526+0000 7f81f37fe640 10 client.974213.objecter ms_dispatch 0x7f820c000ba0 osd_map(679..679 src has 451..679) 2024-09-16T04:04:27.557 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.526+0000 7f81f37fe640 3 client.974213.objecter handle_osd_map got epochs [679,679] > 0 2024-09-16T04:04:27.557 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.526+0000 7f81f37fe640 3 client.974213.objecter handle_osd_map decoding full epoch 679 2024-09-16T04:04:27.557 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.526+0000 7f820affd640 1 --2- 172.21.15.42:0/3595300856 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f820c138890 0x7f820c13b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:27.557 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.526+0000 7f81f37fe640 20 client.974213.objecter dump_active .. 0 homeless 2024-09-16T04:04:27.557 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.526+0000 7f820affd640 10 monclient: get_auth_request con 0x7f820c138890 auth_method 0 2024-09-16T04:04:27.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.526+0000 7f820affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:04:27.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.526+0000 7f820affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:04:27.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.526+0000 7f820affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:04:27.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.526+0000 7f820a7fc640 1 --2- 172.21.15.42:0/3595300856 >> v2:172.21.15.154:6812/254845342 conn(0x7f81f8037810 0x7f81f8039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:27.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.526+0000 7f820affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:04:27.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.526+0000 7f820affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:04:27.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.526+0000 7f820affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:04:27.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.526+0000 7f820a7fc640 10 monclient: get_auth_request con 0x7f81f8037810 auth_method 0 2024-09-16T04:04:27.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.526+0000 7f820affd640 10 monclient(hunting): handle_auth_done global_id 904798 payload 824 2024-09-16T04:04:27.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.526+0000 7f820affd640 1 -- 172.21.15.42:0/3595300856 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f820c09d150 con 0x7f820c138890 2024-09-16T04:04:27.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.530+0000 7f820a7fc640 1 --2- 172.21.15.42:0/3595300856 >> v2:172.21.15.154:6812/254845342 conn(0x7f81f8037810 0x7f81f8039cd0 secure :-1 s=READY pgs=3823 cs=0 l=1 rev1=1 crypto rx=0x7f820c139480 tx=0x7f8200008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:04:27.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.530+0000 7f81f37fe640 10 client.974213.objecter ms_handle_connect 0x7f81f8037810 2024-09-16T04:04:27.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.530+0000 7f81f37fe640 10 client.974213.objecter ms_handle_connect 0x7f820c138890 2024-09-16T04:04:27.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.530+0000 7f81f37fe640 10 client.974213.objecter resend_mon_ops 2024-09-16T04:04:27.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.530+0000 7f820affd640 1 --2- 172.21.15.42:0/3595300856 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f820c138890 0x7f820c13b740 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f81fc00d500 tx=0x7f81fc00d9c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:04:27.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.530+0000 7f81f37fe640 1 -- 172.21.15.42:0/3595300856 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81fc018020 con 0x7f820c138890 2024-09-16T04:04:27.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.530+0000 7f81f37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:04:27.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.530+0000 7f81f37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:04:27.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.530+0000 7f81f37fe640 10 monclient: dump: 2024-09-16T04:04:27.559 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:04:27.559 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:04:27.559 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:27.559 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:27.560 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:04:27.560 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:04:27.560 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:04:27.560 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:04:27.560 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:04:27.560 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:04:27.560 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.530+0000 7f81f37fe640 1 -- 172.21.15.42:0/3595300856 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f81fc014620 con 0x7f820c138890 2024-09-16T04:04:27.560 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.530+0000 7f81f37fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:04:27.560 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.530+0000 7f81f37fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:04:27.560 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.530+0000 7f81f37fe640 1 -- 172.21.15.42:0/3595300856 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f820c138890 msgr2=0x7f820c13b740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:04:27.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.530+0000 7f81f37fe640 1 --2- 172.21.15.42:0/3595300856 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f820c138890 0x7f820c13b740 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f81fc00d500 tx=0x7f81fc00d9c0 comp rx=0 tx=0).stop 2024-09-16T04:04:27.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.530+0000 7f81f37fe640 1 -- 172.21.15.42:0/3595300856 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f820c138890 msgr2=0x7f820c13b740 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:04:27.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.530+0000 7f81f37fe640 1 --2- 172.21.15.42:0/3595300856 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f820c138890 0x7f820c13b740 secure :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f81fc00d500 tx=0x7f81fc00d9c0 comp rx=0 tx=0).stop 2024-09-16T04:04:27.562 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.534+0000 7f8211399640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:04:27.562 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.534+0000 7f8211399640 1 --2- 172.21.15.42:0/3595300856 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f820c1307f0 0x7f820c130c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:04:27.562 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.534+0000 7f8211399640 10 monclient(hunting): start opening mon connection 2024-09-16T04:04:27.562 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.534+0000 7f8211399640 1 -- 172.21.15.42:0/3595300856 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f820c13a830 con 0x7f820c1307f0 2024-09-16T04:04:27.562 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.538+0000 7f820affd640 1 --2- 172.21.15.42:0/3595300856 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f820c1307f0 0x7f820c130c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:27.562 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.538+0000 7f820affd640 10 monclient: get_auth_request con 0x7f820c1307f0 auth_method 0 2024-09-16T04:04:27.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.538+0000 7f820affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:04:27.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.538+0000 7f820affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:04:27.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.538+0000 7f820affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:04:27.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.538+0000 7f820affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:04:27.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.538+0000 7f820affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:04:27.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.538+0000 7f820affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:04:27.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.538+0000 7f820affd640 10 monclient(hunting): handle_auth_done global_id 904801 payload 824 2024-09-16T04:04:27.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.538+0000 7f820affd640 1 -- 172.21.15.42:0/3595300856 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f820c134ee0 con 0x7f820c1307f0 2024-09-16T04:04:27.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.538+0000 7f81f37fe640 10 client.974213.objecter ms_handle_connect 0x7f820c1307f0 2024-09-16T04:04:27.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.538+0000 7f81f37fe640 10 client.974213.objecter resend_mon_ops 2024-09-16T04:04:27.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.538+0000 7f820affd640 1 --2- 172.21.15.42:0/3595300856 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f820c1307f0 0x7f820c130c00 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f81fc00d500 tx=0x7f81fc004700 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:04:27.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.538+0000 7f81f37fe640 1 -- 172.21.15.42:0/3595300856 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81fc026070 con 0x7f820c1307f0 2024-09-16T04:04:27.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.538+0000 7f81f37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:04:27.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.538+0000 7f81f37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:04:27.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.538+0000 7f81f37fe640 10 monclient: dump: 2024-09-16T04:04:27.564 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:04:27.564 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:04:27.564 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:27.565 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:27.565 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:04:27.565 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:04:27.565 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:04:27.565 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:04:27.565 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:04:27.565 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:04:27.565 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.538+0000 7f81f37fe640 1 -- 172.21.15.42:0/3595300856 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4880 (secure 0 0 0) 0x7f81fc01fa60 con 0x7f820c1307f0 2024-09-16T04:04:27.565 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.538+0000 7f81f37fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:04:27.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.538+0000 7f81f37fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:04:27.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.538+0000 7f81f37fe640 1 -- 172.21.15.42:0/3595300856 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f820c1307f0 msgr2=0x7f820c130c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:04:27.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.538+0000 7f81f37fe640 1 --2- 172.21.15.42:0/3595300856 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f820c1307f0 0x7f820c130c00 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f81fc00d500 tx=0x7f81fc004700 comp rx=0 tx=0).stop 2024-09-16T04:04:27.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.538+0000 7f81f37fe640 1 -- 172.21.15.42:0/3595300856 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f820c1307f0 msgr2=0x7f820c130c00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:04:27.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.538+0000 7f81f37fe640 1 --2- 172.21.15.42:0/3595300856 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f820c1307f0 0x7f820c130c00 secure :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f81fc00d500 tx=0x7f81fc004700 comp rx=0 tx=0).stop 2024-09-16T04:04:27.566 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T04:04:27.566 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:04:27.566 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:04:27.567 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T04:04:27.567 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 210, 2024-09-16T04:04:27.567 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 395714, 2024-09-16T04:04:27.567 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T04:04:27.567 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T04:04:27.567 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T04:04:27.567 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T04:04:27.567 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:04:27.567 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 2, 2024-09-16T04:04:27.567 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:04:27.567 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T04:04:27.567 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T04:04:27.567 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:04:27.568 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:04:27.568 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:04:27.568 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:04:27.568 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:04:27.568 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:04:27.568 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:04:27.568 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:04:27.568 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:04:27.568 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:04:27.568 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:04:27.568 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:04:27.569 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:04:27.569 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T04:04:27.569 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:04:27.569 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:04:27.569 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:04:27.569 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:04:27.569 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:04:27.569 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:04:27.569 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:04:27.569 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:04:27.569 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:04:27.569 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:04:27.570 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:04:27.570 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:04:27.570 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:04:27.570 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T04:04:27.570 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T04:04:27.570 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T04:04:27.570 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T04:04:27.570 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T04:04:27.570 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T04:04:27.570 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:04:27.570 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:04:27.571 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T04:04:27.571 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T04:04:27.571 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T04:04:27.571 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T04:04:27.571 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T04:04:27.571 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T04:04:27.571 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T04:04:27.571 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:04:27.571 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T04:04:27.571 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:04:27.571 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:04:27.572 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:04:27.572 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:04:27.572 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:04:27.572 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:04:27.572 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:04:27.572 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:04:27.572 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:04:27.572 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:04:27.572 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:04:27.572 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:04:27.572 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T04:04:27.573 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:04:27.573 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T04:04:27.573 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:04:27.573 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:04:27.573 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:04:27.573 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:04:27.573 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:04:27.573 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:04:27.573 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:04:27.573 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T04:04:27.573 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:04:27.574 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:04:27.574 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:04:27.574 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:04:27.574 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T04:04:27.574 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:04:27.574 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:04:27.574 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:04:27.574 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:04:27.574 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T04:04:27.574 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T04:04:27.574 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:04:27.574 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:04:27.575 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:04:27.575 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:04:27.575 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:04:27.575 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:04:27.575 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:04:27.575 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:04:27.575 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:04:27.575 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:04:27.575 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:04:27.575 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T04:04:27.575 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:04:27.576 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:04:27.576 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:04:27.576 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:04:27.576 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T04:04:27.576 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:04:27.576 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:04:27.576 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:04:27.576 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:04:27.576 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T04:04:27.576 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T04:04:27.576 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:04:27.577 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:04:27.577 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:04:27.577 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:04:27.577 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:04:27.577 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:04:27.577 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:04:27.577 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:04:27.577 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:04:27.577 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:04:27.578 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:04:27.578 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T04:04:27.578 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:04:27.578 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:04:27.578 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:04:27.578 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:04:27.578 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T04:04:27.578 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:04:27.578 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:04:27.578 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:04:27.578 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:04:27.579 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T04:04:27.579 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T04:04:27.579 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:04:27.579 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:04:27.579 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:04:27.579 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:04:27.579 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:04:27.579 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:04:27.579 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T04:04:27.579 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T04:04:27.579 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:04:27.579 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:04:27.580 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:04:27.580 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:04:27.580 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:04:27.580 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:04:27.580 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T04:04:27.580 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:04:27.580 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:04:27.580 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:04:27.580 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 3 2024-09-16T04:04:27.580 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:04:27.581 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:04:27.581 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T04:04:27.581 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:04:27.581 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:04:27.581 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:04:27.581 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 6 2024-09-16T04:04:27.581 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:04:27.581 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:04:27.581 INFO:teuthology.orchestra.run.smithi042.stdout: "mgr": [ 2024-09-16T04:04:27.581 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:04:27.581 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:04:27.581 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:04:27.582 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:04:27.582 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:04:27.582 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:04:27.582 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:04:27.582 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T04:04:27.582 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T04:04:27.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.538+0000 7f8211399640 10 librados: watch_flush enter 2024-09-16T04:04:27.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.538+0000 7f8211399640 10 librados: watch_flush exit 2024-09-16T04:04:27.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.538+0000 7f8211399640 20 client.974213.objecter shutdown clearing up homeless session... 2024-09-16T04:04:27.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.538+0000 7f8211399640 10 client.974213.objecter successfully canceled tick 2024-09-16T04:04:27.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.538+0000 7f8211399640 1 -- 172.21.15.42:0/3595300856 >> v2:172.21.15.154:6812/254845342 conn(0x7f81f8037810 msgr2=0x7f81f8039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:04:27.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.538+0000 7f8211399640 1 --2- 172.21.15.42:0/3595300856 >> v2:172.21.15.154:6812/254845342 conn(0x7f81f8037810 0x7f81f8039cd0 secure :-1 s=READY pgs=3823 cs=0 l=1 rev1=1 crypto rx=0x7f820c139480 tx=0x7f8200008040 comp rx=0 tx=0).stop 2024-09-16T04:04:27.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.538+0000 7f8211399640 10 monclient: shutdown 2024-09-16T04:04:27.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.538+0000 7f8211399640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:04:27.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.538+0000 7f8211399640 1 -- 172.21.15.42:0/3595300856 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f820c073890 msgr2=0x7f820c138350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:04:27.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.538+0000 7f820b7fe640 1 -- 172.21.15.42:0/3595300856 reap_dead start 2024-09-16T04:04:27.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.538+0000 7f8211399640 1 --2- 172.21.15.42:0/3595300856 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f820c073890 0x7f820c138350 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f820401af60 tx=0x7f8204019650 comp rx=0 tx=0).stop 2024-09-16T04:04:27.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.538+0000 7f8211399640 1 -- 172.21.15.42:0/3595300856 shutdown_connections 2024-09-16T04:04:27.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.538+0000 7f8211399640 1 --2- 172.21.15.42:0/3595300856 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f820c073890 0x7f820c138350 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:27.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.538+0000 7f8211399640 1 -- 172.21.15.42:0/3595300856 >> 172.21.15.42:0/3595300856 conn(0x7f820c06bc40 msgr2=0x7f820c132ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:04:27.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.538+0000 7f8211399640 1 -- 172.21.15.42:0/3595300856 shutdown_connections 2024-09-16T04:04:27.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.538+0000 7f8211399640 1 -- 172.21.15.42:0/3595300856 wait complete. 2024-09-16T04:04:27.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.538+0000 7f8211399640 1 librados: shutdown 2024-09-16T04:04:27.586 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:04:27.586 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T04:04:27.586 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T04:04:27.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.698+0000 7ff0ee5e7640 10 monclient: get_monmap_and_config 2024-09-16T04:04:27.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.698+0000 7ff0ee5e7640 10 monclient: build_initial_monmap 2024-09-16T04:04:27.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.698+0000 7ff0ee5e7640 10 monclient: monmap: 2024-09-16T04:04:27.704 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:04:27.704 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:04:27.704 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:04:27.702755+0000 2024-09-16T04:04:27.705 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:04:27.702755+0000 2024-09-16T04:04:27.705 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:04:27.705 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:04:27.705 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:04:27.705 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:04:27.705 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:04:27.705 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:04:27.705 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:04:27.705 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:04:27.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.698+0000 7ff0ee5e7640 1 Processor -- start 2024-09-16T04:04:27.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.702+0000 7ff0ee5e7640 1 -- start start 2024-09-16T04:04:27.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.702+0000 7ff0ee5e7640 10 monclient: init 2024-09-16T04:04:27.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.702+0000 7ff0ee5e7640 10 monclient: _reopen_session rank -1 2024-09-16T04:04:27.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.702+0000 7ff0ee5e7640 10 monclient: _add_conns ranks=[1,4,2,0,3] 2024-09-16T04:04:27.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.702+0000 7ff0ee5e7640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff0e809fb00 addr v1:172.21.15.154:6789/0 2024-09-16T04:04:27.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.702+0000 7ff0ee5e7640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7ff0e8137210 0x7ff0e8139600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:04:27.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.702+0000 7ff0ee5e7640 10 monclient(hunting): picked mon.noname-c con 0x7ff0e8137210 addr v2:172.21.15.154:3300/0 2024-09-16T04:04:27.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.702+0000 7ff0ee5e7640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff0e8139b40 0x7ff0e813bf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:04:27.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.702+0000 7ff0ee5e7640 10 monclient(hunting): picked mon.noname-b con 0x7ff0e8139b40 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:04:27.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.702+0000 7ff0ee5e7640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff0e80a1e20 con 0x7ff0e809fb00 2024-09-16T04:04:27.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.702+0000 7ff0ee5e7640 10 monclient(hunting): start opening mon connection 2024-09-16T04:04:27.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.702+0000 7ff0ee5e7640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7ff0e80a1a40 con 0x7ff0e8137210 2024-09-16T04:04:27.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.702+0000 7ff0ee5e7640 10 monclient(hunting): start opening mon connection 2024-09-16T04:04:27.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.702+0000 7ff0ee5e7640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7ff0e809e250 con 0x7ff0e8139b40 2024-09-16T04:04:27.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.702+0000 7ff0ee5e7640 10 monclient(hunting): _renew_subs 2024-09-16T04:04:27.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.702+0000 7ff0ee5e7640 10 monclient(hunting): authenticate will time out at 2314.672916s 2024-09-16T04:04:27.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.702+0000 7ff0ecb84640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff0e8139b40 0x7ff0e813bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:27.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.702+0000 7ff0ecb84640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff0e8139b40 0x7ff0e813bf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:52426/0 (socket says 172.21.15.42:52426) 2024-09-16T04:04:27.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.702+0000 7ff0ecb84640 1 -- 172.21.15.42:0/230604046 learned_addr learned my addr 172.21.15.42:0/230604046 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:04:27.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.702+0000 7ff0ecb84640 10 monclient(hunting): get_auth_request con 0x7ff0e8139b40 auth_method 0 2024-09-16T04:04:27.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.702+0000 7ff0ecb84640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:04:27.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.702+0000 7ff0ecb84640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:04:27.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.702+0000 7ff0ecb84640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:04:27.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.702+0000 7ff0dffff640 1 --2- 172.21.15.42:0/230604046 >> v2:172.21.15.154:3300/0 conn(0x7ff0e8137210 0x7ff0e8139600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:27.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.702+0000 7ff0ecb84640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:04:27.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.702+0000 7ff0ecb84640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:04:27.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.702+0000 7ff0ecb84640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:04:27.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.702+0000 7ff0dffff640 10 monclient(hunting): get_auth_request con 0x7ff0e8137210 auth_method 0 2024-09-16T04:04:27.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.702+0000 7ff0dffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:04:27.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.702+0000 7ff0dffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:04:27.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.702+0000 7ff0dffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:04:27.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.702+0000 7ff0ecb84640 10 monclient(hunting): handle_auth_done global_id 974216 payload 274 2024-09-16T04:04:27.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.702+0000 7ff0ecb84640 1 -- 172.21.15.42:0/230604046 >> v2:172.21.15.154:3300/0 conn(0x7ff0e8137210 msgr2=0x7ff0e8139600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:04:27.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.702+0000 7ff0ecb84640 1 --2- 172.21.15.42:0/230604046 >> v2:172.21.15.154:3300/0 conn(0x7ff0e8137210 0x7ff0e8139600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:27.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.702+0000 7ff0ecb84640 1 -- 172.21.15.42:0/230604046 >> v1:172.21.15.154:6789/0 conn(0x7ff0e809fb00 legacy=0x7ff0e80a1330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:04:27.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.702+0000 7ff0ecb84640 10 monclient: _finish_hunting 0 2024-09-16T04:04:27.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.702+0000 7ff0ecb84640 1 monclient: found mon.noname-b 2024-09-16T04:04:27.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.702+0000 7ff0ecb84640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:04:27.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.702+0000 7ff0ecb84640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T04:04:27.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.702+0000 7ff0ecb84640 1 -- 172.21.15.42:0/230604046 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0e809ea30 con 0x7ff0e8139b40 2024-09-16T04:04:27.717 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.702+0000 7ff0ecb84640 1 --2- 172.21.15.42:0/230604046 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff0e8139b40 0x7ff0e813bf30 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7ff0e0009e50 tx=0x7ff0e001c890 comp rx=0 tx=0).ready entity=mon.2 client_cookie=db8bf33313eb1914 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:04:27.717 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.706+0000 7ff0e77fe640 1 -- 172.21.15.42:0/230604046 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0e00a5710 con 0x7ff0e8139b40 2024-09-16T04:04:27.717 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.706+0000 7ff0e77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:04:27.717 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.706+0000 7ff0e77fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:04:27.717 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.706+0000 7ff0e77fe640 10 monclient: dump: 2024-09-16T04:04:27.717 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:04:27.718 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:04:27.718 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:27.718 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:27.718 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:04:27.718 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:04:27.718 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:04:27.718 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:04:27.718 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:04:27.718 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:04:27.718 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.706+0000 7ff0e77fe640 10 monclient: _finish_auth 0 2024-09-16T04:04:27.718 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.706+0000 7ff0e77fe640 10 monclient: _check_auth_tickets 2024-09-16T04:04:27.718 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.706+0000 7ff0e77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:04:27.718 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.706+0000 7ff0e77fe640 1 -- 172.21.15.42:0/230604046 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff0e00a58b0 con 0x7ff0e8139b40 2024-09-16T04:04:27.719 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.706+0000 7ff0e77fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:04:27.719 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.706+0000 7ff0e77fe640 1 -- 172.21.15.42:0/230604046 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0e00a5c80 con 0x7ff0e8139b40 2024-09-16T04:04:27.719 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.706+0000 7ff0e77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:04:27.719 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.706+0000 7ff0e77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:04:27.719 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.706+0000 7ff0e77fe640 10 monclient: dump: 2024-09-16T04:04:27.719 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:04:27.719 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:04:27.719 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:27.719 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:27.719 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:04:27.719 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:04:27.719 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:04:27.720 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:04:27.720 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:04:27.720 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:04:27.720 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.706+0000 7ff0ee5e7640 5 monclient: authenticate success, global_id 974216 2024-09-16T04:04:27.720 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.706+0000 7ff0ee5e7640 10 monclient: get_monmap_and_config success 2024-09-16T04:04:27.720 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.706+0000 7ff0ee5e7640 4 set_mon_vals no callback set 2024-09-16T04:04:27.720 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.706+0000 7ff0ee5e7640 10 monclient: shutdown 2024-09-16T04:04:27.720 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.706+0000 7ff0ee5e7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:04:27.720 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.706+0000 7ff0ee5e7640 1 -- 172.21.15.42:0/230604046 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff0e8139b40 msgr2=0x7ff0e813bf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:04:27.720 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.706+0000 7ff0ee5e7640 1 --2- 172.21.15.42:0/230604046 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff0e8139b40 0x7ff0e813bf30 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7ff0e0009e50 tx=0x7ff0e001c890 comp rx=0 tx=0).stop 2024-09-16T04:04:27.720 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.706+0000 7ff0ee5e7640 1 -- 172.21.15.42:0/230604046 shutdown_connections 2024-09-16T04:04:27.720 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.706+0000 7ff0ee5e7640 1 --2- 172.21.15.42:0/230604046 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff0e8139b40 0x7ff0e813bf30 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:27.720 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.706+0000 7ff0ee5e7640 1 --2- 172.21.15.42:0/230604046 >> v2:172.21.15.154:3300/0 conn(0x7ff0e8137210 0x7ff0e8139600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:27.721 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.706+0000 7ff0ee5e7640 1 -- 172.21.15.42:0/230604046 >> 172.21.15.42:0/230604046 conn(0x7ff0e80a2420 msgr2=0x7ff0e80a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:04:27.721 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.706+0000 7ff0ee5e7640 1 -- 172.21.15.42:0/230604046 shutdown_connections 2024-09-16T04:04:27.721 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.706+0000 7ff0ee5e7640 1 -- 172.21.15.42:0/230604046 wait complete. 2024-09-16T04:04:27.721 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.706+0000 7ff0ee5e7640 10 monclient: build_initial_monmap 2024-09-16T04:04:27.721 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.706+0000 7ff0ee5e7640 10 monclient: monmap: 2024-09-16T04:04:27.721 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:04:27.721 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:04:27.721 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:04:27.721 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:04:27.721 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:04:27.721 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:04:27.721 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:04:27.721 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:04:27.722 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:04:27.722 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:04:27.722 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.706+0000 7ff0ee5e7640 1 librados: starting msgr at 2024-09-16T04:04:27.722 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.706+0000 7ff0ee5e7640 1 librados: starting objecter 2024-09-16T04:04:27.722 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.706+0000 7ff0ee5e7640 1 Processor -- start 2024-09-16T04:04:27.722 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.706+0000 7ff0ee5e7640 1 -- start start 2024-09-16T04:04:27.722 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.706+0000 7ff0ee5e7640 1 librados: setting wanted keys 2024-09-16T04:04:27.722 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.706+0000 7ff0ee5e7640 1 librados: calling monclient init 2024-09-16T04:04:27.722 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.706+0000 7ff0ee5e7640 10 monclient: init 2024-09-16T04:04:27.722 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.706+0000 7ff0ee5e7640 10 monclient: _reopen_session rank -1 2024-09-16T04:04:27.722 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.706+0000 7ff0ee5e7640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:04:27.722 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.706+0000 7ff0ee5e7640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff0e809fb00 0x7ff0e81ced40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:04:27.722 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.706+0000 7ff0ee5e7640 10 monclient(hunting): picked mon.noname-a con 0x7ff0e809fb00 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:04:27.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.706+0000 7ff0ee5e7640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff0e8137210 0x7ff0e8136550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:04:27.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.706+0000 7ff0ee5e7640 10 monclient(hunting): picked mon.noname-b con 0x7ff0e8137210 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:04:27.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.706+0000 7ff0ee5e7640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff0e8139b40 0x7ff0e8136a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:04:27.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.706+0000 7ff0ee5e7640 10 monclient(hunting): picked mon.noname-c con 0x7ff0e8139b40 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:04:27.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.706+0000 7ff0ee5e7640 10 monclient(hunting): start opening mon connection 2024-09-16T04:04:27.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.706+0000 7ff0ee5e7640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7ff0e80a3170 con 0x7ff0e809fb00 2024-09-16T04:04:27.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.706+0000 7ff0ee5e7640 10 monclient(hunting): start opening mon connection 2024-09-16T04:04:27.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.706+0000 7ff0ee5e7640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff0e809d0b0 con 0x7ff0e8137210 2024-09-16T04:04:27.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.706+0000 7ff0ee5e7640 10 monclient(hunting): start opening mon connection 2024-09-16T04:04:27.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.706+0000 7ff0ee5e7640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7ff0e80a32f0 con 0x7ff0e8139b40 2024-09-16T04:04:27.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.706+0000 7ff0ee5e7640 10 monclient(hunting): _renew_subs 2024-09-16T04:04:27.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.706+0000 7ff0ee5e7640 10 monclient(hunting): authenticate will time out at 2314.678458s 2024-09-16T04:04:27.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.706+0000 7ff0ecb84640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff0e8139b40 0x7ff0e8136a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:27.724 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.706+0000 7ff0e7fff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff0e809fb00 0x7ff0e81ced40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:27.724 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.706+0000 7ff0ecb84640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff0e8139b40 0x7ff0e8136a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:52436/0 (socket says 172.21.15.42:52436) 2024-09-16T04:04:27.724 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.706+0000 7ff0ecb84640 1 -- 172.21.15.42:0/4222162500 learned_addr learned my addr 172.21.15.42:0/4222162500 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:04:27.724 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.706+0000 7ff0e7fff640 10 monclient(hunting): get_auth_request con 0x7ff0e809fb00 auth_method 0 2024-09-16T04:04:27.724 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.706+0000 7ff0e7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:04:27.724 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.706+0000 7ff0e7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:04:27.724 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.706+0000 7ff0e7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:04:27.724 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.706+0000 7ff0ecb84640 10 monclient(hunting): get_auth_request con 0x7ff0e8139b40 auth_method 0 2024-09-16T04:04:27.724 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.706+0000 7ff0ecb84640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:04:27.724 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.706+0000 7ff0ecb84640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:04:27.724 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.706+0000 7ff0ecb84640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:04:27.725 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.706+0000 7ff0e7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:04:27.725 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.706+0000 7ff0e7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:04:27.725 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.706+0000 7ff0e7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:04:27.725 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.706+0000 7ff0ecb84640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:04:27.725 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.706+0000 7ff0ecb84640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:04:27.725 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.706+0000 7ff0ecb84640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:04:27.725 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.706+0000 7ff0dffff640 1 --2- 172.21.15.42:0/4222162500 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff0e8137210 0x7ff0e8136550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:27.725 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.706+0000 7ff0dffff640 10 monclient(hunting): get_auth_request con 0x7ff0e8137210 auth_method 0 2024-09-16T04:04:27.725 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.706+0000 7ff0dffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:04:27.725 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.706+0000 7ff0dffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:04:27.725 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.706+0000 7ff0dffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:04:27.725 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.706+0000 7ff0e7fff640 10 monclient(hunting): handle_auth_done global_id 984135 payload 824 2024-09-16T04:04:27.725 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.706+0000 7ff0e7fff640 1 -- 172.21.15.42:0/4222162500 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff0e8139b40 msgr2=0x7ff0e8136a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:04:27.726 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.706+0000 7ff0e7fff640 1 --2- 172.21.15.42:0/4222162500 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff0e8139b40 0x7ff0e8136a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:27.726 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.706+0000 7ff0e7fff640 1 -- 172.21.15.42:0/4222162500 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff0e8137210 msgr2=0x7ff0e8136550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:04:27.726 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.706+0000 7ff0e7fff640 1 --2- 172.21.15.42:0/4222162500 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff0e8137210 0x7ff0e8136550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:27.726 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.706+0000 7ff0e7fff640 10 monclient: _finish_hunting 0 2024-09-16T04:04:27.726 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.706+0000 7ff0e7fff640 1 monclient: found mon.noname-a 2024-09-16T04:04:27.726 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.706+0000 7ff0e7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:04:27.726 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.706+0000 7ff0e7fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:04:27.726 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.706+0000 7ff0e7fff640 1 -- 172.21.15.42:0/4222162500 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0e8123c30 con 0x7ff0e809fb00 2024-09-16T04:04:27.726 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.706+0000 7ff0ecb84640 1 --2- 172.21.15.42:0/4222162500 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff0e8139b40 0x7ff0e8136a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:04:27.726 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.706+0000 7ff0e7fff640 1 --2- 172.21.15.42:0/4222162500 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff0e809fb00 0x7ff0e81ced40 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ff0d800c3e0 tx=0x7ff0d800c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:04:27.726 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.706+0000 7ff0e57fa640 10 client.?.objecter ms_handle_connect 0x7ff0e809fb00 2024-09-16T04:04:27.726 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.706+0000 7ff0e57fa640 10 client.?.objecter resend_mon_ops 2024-09-16T04:04:27.726 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.706+0000 7ff0e57fa640 1 -- 172.21.15.42:0/4222162500 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0d8011790 con 0x7ff0e809fb00 2024-09-16T04:04:27.727 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.706+0000 7ff0e57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:04:27.727 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.706+0000 7ff0e57fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:04:27.727 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.706+0000 7ff0e57fa640 10 monclient: dump: 2024-09-16T04:04:27.727 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:04:27.727 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:04:27.727 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:27.727 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:27.727 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:04:27.727 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:04:27.727 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:04:27.727 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:04:27.728 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:04:27.728 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:04:27.728 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.706+0000 7ff0e57fa640 10 monclient: _finish_auth 0 2024-09-16T04:04:27.728 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.706+0000 7ff0e57fa640 10 monclient: _check_auth_tickets 2024-09-16T04:04:27.728 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.706+0000 7ff0e57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:04:27.728 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.706+0000 7ff0e57fa640 1 -- 172.21.15.42:0/4222162500 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff0d8017070 con 0x7ff0e809fb00 2024-09-16T04:04:27.728 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.706+0000 7ff0e57fa640 10 monclient: handle_config config(0 keys) 2024-09-16T04:04:27.728 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.706+0000 7ff0e57fa640 1 -- 172.21.15.42:0/4222162500 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0d8011bb0 con 0x7ff0e809fb00 2024-09-16T04:04:27.728 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.706+0000 7ff0ee5e7640 5 monclient: authenticate success, global_id 984135 2024-09-16T04:04:27.728 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.706+0000 7ff0ee5e7640 10 monclient: _renew_subs 2024-09-16T04:04:27.728 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.706+0000 7ff0ee5e7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:04:27.728 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.706+0000 7ff0ee5e7640 1 -- 172.21.15.42:0/4222162500 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0e8122780 con 0x7ff0e809fb00 2024-09-16T04:04:27.728 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.706+0000 7ff0e57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:04:27.729 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.706+0000 7ff0e57fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:04:27.729 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.706+0000 7ff0e57fa640 10 monclient: dump: 2024-09-16T04:04:27.729 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:04:27.729 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:04:27.729 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:27.729 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:27.729 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:04:27.729 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:04:27.729 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:04:27.729 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:04:27.729 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:04:27.729 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:04:27.729 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.706+0000 7ff0e6ffd640 4 set_mon_vals no callback set 2024-09-16T04:04:27.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.706+0000 7ff0ee5e7640 10 client.984135.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:04:27.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.706+0000 7ff0ee5e7640 10 monclient: _renew_subs 2024-09-16T04:04:27.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.706+0000 7ff0ee5e7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:04:27.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.706+0000 7ff0ee5e7640 1 -- 172.21.15.42:0/4222162500 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0e8123950 con 0x7ff0e809fb00 2024-09-16T04:04:27.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.706+0000 7ff0ee5e7640 1 librados: init done 2024-09-16T04:04:27.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.710+0000 7ff0ee5e7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:04:27.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.710+0000 7ff0ee5e7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:04:27.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.710+0000 7ff0ee5e7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:04:27.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.710+0000 7ff0e57fa640 1 -- 172.21.15.42:0/4222162500 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7ff0d801c460 con 0x7ff0e809fb00 2024-09-16T04:04:27.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.710+0000 7ff0ee5e7640 1 -- 172.21.15.42:0/4222162500 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff0e80a1330 con 0x7ff0e809fb00 2024-09-16T04:04:27.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.710+0000 7ff0e57fa640 1 --2- 172.21.15.42:0/4222162500 >> v2:172.21.15.154:6812/254845342 conn(0x7ff0c8037810 0x7ff0c8039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:04:27.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.710+0000 7ff0e57fa640 1 -- 172.21.15.42:0/4222162500 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(679..679 src has 451..679) ==== 6280+0+0 (secure 0 0 0) 0x7ff0d8048f10 con 0x7ff0e809fb00 2024-09-16T04:04:27.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.710+0000 7ff0e57fa640 10 client.984135.objecter ms_dispatch 0x7ff0e8000ba0 osd_map(679..679 src has 451..679) 2024-09-16T04:04:27.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.710+0000 7ff0e57fa640 3 client.984135.objecter handle_osd_map got epochs [679,679] > 0 2024-09-16T04:04:27.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.710+0000 7ff0e57fa640 3 client.984135.objecter handle_osd_map decoding full epoch 679 2024-09-16T04:04:27.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.710+0000 7ff0e57fa640 20 client.984135.objecter dump_active .. 0 homeless 2024-09-16T04:04:27.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.710+0000 7ff0dffff640 1 --2- 172.21.15.42:0/4222162500 >> v2:172.21.15.154:6812/254845342 conn(0x7ff0c8037810 0x7ff0c8039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:27.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.710+0000 7ff0dffff640 10 monclient: get_auth_request con 0x7ff0c8037810 auth_method 0 2024-09-16T04:04:27.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.710+0000 7ff0dffff640 1 --2- 172.21.15.42:0/4222162500 >> v2:172.21.15.154:6812/254845342 conn(0x7ff0c8037810 0x7ff0c8039cd0 secure :-1 s=READY pgs=3824 cs=0 l=1 rev1=1 crypto rx=0x7ff0e81390a0 tx=0x7ff0d0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:04:27.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.710+0000 7ff0e57fa640 10 client.984135.objecter ms_handle_connect 0x7ff0c8037810 2024-09-16T04:04:27.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.714+0000 7ff0e57fa640 1 -- 172.21.15.42:0/4222162500 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7ff0d801a520 con 0x7ff0e809fb00 2024-09-16T04:04:27.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.714+0000 7ff0e57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:04:27.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.714+0000 7ff0e57fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:04:27.942 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.938+0000 7ff0ee5e7640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T04:04:27.943 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.938+0000 7ff0ee5e7640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T04:04:27.943 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.938+0000 7ff0ee5e7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:04:27.943 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.938+0000 7ff0ee5e7640 1 -- 172.21.15.42:0/4222162500 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7ff0e813bb80 con 0x7ff0e809fb00 2024-09-16T04:04:27.944 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.938+0000 7ff0e57fa640 1 -- 172.21.15.42:0/4222162500 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7ff0d801a780 con 0x7ff0e809fb00 2024-09-16T04:04:27.945 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.938+0000 7ff0e57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T04:04:27.945 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.938+0000 7ff0e57fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:04:27.945 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.942+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:27.946 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.942+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=30,config=2,health=12,kv=9,logm=47} crc {auth=1102481265,config=692619927,health=2904421822,kv=2094645990,logm=572169001}) 2024-09-16T04:04:27.946 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.942+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=30,config=2,health=12,kv=9,logm=47} crc {auth=3781283846,config=692619927,health=2904421822,kv=2236749221,logm=1106345168}) 2024-09-16T04:04:27.946 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.942+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:27.946 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.942+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=30,config=2,health=12,kv=9,logm=47} crc {auth=1102481265,config=692619927,health=2904421822,kv=2094645990,logm=572169001}) 2024-09-16T04:04:27.946 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.942+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=29,config=2,health=11,kv=8,logm=50} crc {auth=258387587,config=692619928,health=4278022219,kv=3251691281,logm=2437624776}) 2024-09-16T04:04:27.948 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.942+0000 7ff0ee5e7640 10 librados: watch_flush enter 2024-09-16T04:04:27.948 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.942+0000 7ff0ee5e7640 10 librados: watch_flush exit 2024-09-16T04:04:27.948 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.942+0000 7ff0ee5e7640 20 client.984135.objecter shutdown clearing up homeless session... 2024-09-16T04:04:27.948 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.942+0000 7ff0ee5e7640 10 client.984135.objecter successfully canceled tick 2024-09-16T04:04:27.948 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.942+0000 7ff0ee5e7640 1 -- 172.21.15.42:0/4222162500 >> v2:172.21.15.154:6812/254845342 conn(0x7ff0c8037810 msgr2=0x7ff0c8039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:04:27.948 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.942+0000 7ff0ee5e7640 1 --2- 172.21.15.42:0/4222162500 >> v2:172.21.15.154:6812/254845342 conn(0x7ff0c8037810 0x7ff0c8039cd0 secure :-1 s=READY pgs=3824 cs=0 l=1 rev1=1 crypto rx=0x7ff0e81390a0 tx=0x7ff0d0008040 comp rx=0 tx=0).stop 2024-09-16T04:04:27.948 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.942+0000 7ff0ee5e7640 10 monclient: shutdown 2024-09-16T04:04:27.948 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.942+0000 7ff0ee5e7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:04:27.948 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.942+0000 7ff0ee5e7640 1 -- 172.21.15.42:0/4222162500 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff0e809fb00 msgr2=0x7ff0e81ced40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:04:27.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.942+0000 7ff0ee5e7640 1 --2- 172.21.15.42:0/4222162500 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff0e809fb00 0x7ff0e81ced40 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ff0d800c3e0 tx=0x7ff0d800c8a0 comp rx=0 tx=0).stop 2024-09-16T04:04:27.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.942+0000 7ff0ee5e7640 1 -- 172.21.15.42:0/4222162500 shutdown_connections 2024-09-16T04:04:27.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.942+0000 7ff0ee5e7640 1 --2- 172.21.15.42:0/4222162500 >> v2:172.21.15.154:6812/254845342 conn(0x7ff0c8037810 0x7ff0c8039cd0 unknown :-1 s=CLOSED pgs=3824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:27.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.942+0000 7ff0ee5e7640 1 --2- 172.21.15.42:0/4222162500 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff0e8139b40 0x7ff0e8136a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:27.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.942+0000 7ff0ee5e7640 1 --2- 172.21.15.42:0/4222162500 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff0e8137210 0x7ff0e8136550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:27.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.942+0000 7ff0ee5e7640 1 --2- 172.21.15.42:0/4222162500 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff0e809fb00 0x7ff0e81ced40 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:27.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.942+0000 7ff0ee5e7640 1 -- 172.21.15.42:0/4222162500 >> 172.21.15.42:0/4222162500 conn(0x7ff0e80a2420 msgr2=0x7ff0e80a4000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:04:27.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.942+0000 7ff0ee5e7640 1 -- 172.21.15.42:0/4222162500 shutdown_connections 2024-09-16T04:04:27.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.942+0000 7ff0ee5e7640 1 -- 172.21.15.42:0/4222162500 wait complete. 2024-09-16T04:04:27.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:27.942+0000 7ff0ee5e7640 1 librados: shutdown 2024-09-16T04:04:27.950 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.942+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:27.950 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.942+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2990921706}) 2024-09-16T04:04:27.950 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.942+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1307106058}) 2024-09-16T04:04:27.950 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.942+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:27.951 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.942+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2990921706}) 2024-09-16T04:04:27.951 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.942+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2518115128}) 2024-09-16T04:04:27.952 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.946+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:27.952 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.946+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=547044612}) 2024-09-16T04:04:27.952 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.946+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3876383300}) 2024-09-16T04:04:27.952 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.946+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:27.952 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.946+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=547044612}) 2024-09-16T04:04:27.952 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.946+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3744102761}) 2024-09-16T04:04:27.956 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.950+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:27.956 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.950+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=651827149}) 2024-09-16T04:04:27.956 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.950+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2921269673}) 2024-09-16T04:04:27.956 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.950+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:27.956 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.950+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=651827149}) 2024-09-16T04:04:27.956 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.950+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2467005568}) 2024-09-16T04:04:27.959 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.954+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:27.959 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.954+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1825944926}) 2024-09-16T04:04:27.960 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.954+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1642671213}) 2024-09-16T04:04:27.960 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.954+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:27.960 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.954+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1825944926}) 2024-09-16T04:04:27.960 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.954+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=607030702}) 2024-09-16T04:04:27.962 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.958+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:27.962 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.958+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1832413325}) 2024-09-16T04:04:27.962 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.958+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4260327967}) 2024-09-16T04:04:27.962 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.958+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:27.962 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.958+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1832413325}) 2024-09-16T04:04:27.962 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.958+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2638421722}) 2024-09-16T04:04:27.964 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.958+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:27.964 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.958+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1810487065}) 2024-09-16T04:04:27.964 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.958+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3284101074}) 2024-09-16T04:04:27.965 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.958+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:27.965 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.958+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1810487065}) 2024-09-16T04:04:27.965 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.958+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4027204267}) 2024-09-16T04:04:27.966 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T04:04:27.966 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.962+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:27.967 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.962+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2253927516}) 2024-09-16T04:04:27.967 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.962+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3818341741}) 2024-09-16T04:04:27.967 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.962+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:27.967 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.962+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2253927516}) 2024-09-16T04:04:27.967 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.962+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2150018613}) 2024-09-16T04:04:27.968 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.962+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:27.969 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.962+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2440865589}) 2024-09-16T04:04:27.969 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.962+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1875052087}) 2024-09-16T04:04:27.969 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.962+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:27.969 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.962+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2440865589}) 2024-09-16T04:04:27.969 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.962+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1813206443}) 2024-09-16T04:04:27.971 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.966+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:27.971 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.966+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=828279725}) 2024-09-16T04:04:27.971 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.966+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1112124676}) 2024-09-16T04:04:27.971 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.966+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:27.971 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.966+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=828279725}) 2024-09-16T04:04:27.971 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.966+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3279825877}) 2024-09-16T04:04:27.973 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.966+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:27.973 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.966+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3748590202}) 2024-09-16T04:04:27.973 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.966+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=631296395}) 2024-09-16T04:04:27.973 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.966+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:27.973 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.966+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3748590202}) 2024-09-16T04:04:27.973 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.966+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2993669391}) 2024-09-16T04:04:27.974 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.970+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:27.975 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.970+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2143043529}) 2024-09-16T04:04:27.975 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.970+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1097158380}) 2024-09-16T04:04:27.975 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.970+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:27.975 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.970+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2143043529}) 2024-09-16T04:04:27.975 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.970+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=659832555}) 2024-09-16T04:04:27.977 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.970+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:27.977 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.970+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4159077929}) 2024-09-16T04:04:27.977 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.970+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=184550568}) 2024-09-16T04:04:27.977 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.970+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:27.977 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.970+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4159077929}) 2024-09-16T04:04:27.977 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.970+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3204497859}) 2024-09-16T04:04:27.978 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.974+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:27.978 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.974+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2749410388}) 2024-09-16T04:04:27.979 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.974+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3339997971}) 2024-09-16T04:04:27.979 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.974+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:27.979 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.974+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2749410388}) 2024-09-16T04:04:27.979 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.974+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2210688053}) 2024-09-16T04:04:27.980 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.974+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:27.980 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.974+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2813975019}) 2024-09-16T04:04:27.980 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.974+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3026294256}) 2024-09-16T04:04:27.981 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.974+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:27.981 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.974+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2813975019}) 2024-09-16T04:04:27.981 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.974+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3362001577}) 2024-09-16T04:04:27.982 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.978+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:27.982 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.978+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3798060444}) 2024-09-16T04:04:27.982 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.978+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2520562449}) 2024-09-16T04:04:27.983 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.978+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:27.983 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.978+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3798060444}) 2024-09-16T04:04:27.983 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.978+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1568776044}) 2024-09-16T04:04:27.984 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.978+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:27.984 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.978+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3086185990}) 2024-09-16T04:04:27.985 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.978+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=900112975}) 2024-09-16T04:04:27.985 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.978+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:27.985 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.978+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3086185990}) 2024-09-16T04:04:27.985 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.978+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2553196771}) 2024-09-16T04:04:27.986 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.982+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:27.986 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.982+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2462161326}) 2024-09-16T04:04:27.987 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.982+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2394012599}) 2024-09-16T04:04:27.987 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.982+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:27.987 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.982+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2462161326}) 2024-09-16T04:04:27.987 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.982+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1018836296}) 2024-09-16T04:04:27.988 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.982+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:27.988 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.982+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=684634810}) 2024-09-16T04:04:27.988 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.982+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2244118411}) 2024-09-16T04:04:27.989 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.982+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:27.989 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.982+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=684634810}) 2024-09-16T04:04:27.989 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.982+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=403211922}) 2024-09-16T04:04:27.991 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.986+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:27.991 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.986+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1551408496}) 2024-09-16T04:04:27.991 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.986+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3622648062}) 2024-09-16T04:04:27.991 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.986+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:27.991 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.986+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1551408496}) 2024-09-16T04:04:27.991 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.986+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2327922103}) 2024-09-16T04:04:27.992 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.986+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:27.992 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.986+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=694236720}) 2024-09-16T04:04:27.992 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.986+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=292255622}) 2024-09-16T04:04:27.992 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.986+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:27.992 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.986+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=694236720}) 2024-09-16T04:04:27.992 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.986+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1639229875}) 2024-09-16T04:04:27.994 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.990+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:27.994 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.990+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2445561253}) 2024-09-16T04:04:27.994 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.990+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1248372796}) 2024-09-16T04:04:27.994 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.990+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:27.994 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.990+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2445561253}) 2024-09-16T04:04:27.994 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.990+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=790852204}) 2024-09-16T04:04:27.996 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.990+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:27.996 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.990+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3613668052}) 2024-09-16T04:04:27.996 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.990+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3766390349}) 2024-09-16T04:04:27.996 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.990+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:27.996 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.990+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3613668052}) 2024-09-16T04:04:27.996 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.990+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1211855213}) 2024-09-16T04:04:27.998 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.994+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:27.998 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.994+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4019072789}) 2024-09-16T04:04:27.998 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.994+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1702113205}) 2024-09-16T04:04:27.998 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.994+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:27.998 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.994+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4019072789}) 2024-09-16T04:04:27.998 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.994+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1095433489}) 2024-09-16T04:04:28.000 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.994+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.000 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.994+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1546319843}) 2024-09-16T04:04:28.000 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.994+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3049871138}) 2024-09-16T04:04:28.000 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.994+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.000 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.994+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1546319843}) 2024-09-16T04:04:28.000 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.994+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2201967824}) 2024-09-16T04:04:28.002 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.998+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.002 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.998+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1970933716}) 2024-09-16T04:04:28.002 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.998+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2820850604}) 2024-09-16T04:04:28.002 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.998+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.002 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.998+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1970933716}) 2024-09-16T04:04:28.002 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.998+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2945451055}) 2024-09-16T04:04:28.004 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.998+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.004 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.998+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=206201291}) 2024-09-16T04:04:28.004 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.998+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3660514734}) 2024-09-16T04:04:28.004 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.998+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.004 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.998+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=206201291}) 2024-09-16T04:04:28.004 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:27.998+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1845345294}) 2024-09-16T04:04:28.005 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.002+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.006 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.002+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=293216466}) 2024-09-16T04:04:28.006 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.002+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=253212512}) 2024-09-16T04:04:28.006 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.002+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.006 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.002+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=293216466}) 2024-09-16T04:04:28.006 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.002+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3558699757}) 2024-09-16T04:04:28.007 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.002+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.008 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.002+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1311952610}) 2024-09-16T04:04:28.008 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.002+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3807606280}) 2024-09-16T04:04:28.008 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.002+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.008 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.002+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1311952610}) 2024-09-16T04:04:28.008 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.002+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3593828138}) 2024-09-16T04:04:28.010 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.006+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.010 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.006+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2481803647}) 2024-09-16T04:04:28.010 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.006+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3509826353}) 2024-09-16T04:04:28.010 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.006+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.010 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.006+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2481803647}) 2024-09-16T04:04:28.010 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.006+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=144286951}) 2024-09-16T04:04:28.012 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.006+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.012 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.006+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2822933307}) 2024-09-16T04:04:28.012 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.006+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1091186612}) 2024-09-16T04:04:28.012 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.006+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.012 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.006+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2822933307}) 2024-09-16T04:04:28.013 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.006+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=532066412}) 2024-09-16T04:04:28.014 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.010+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.014 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.010+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=372033785}) 2024-09-16T04:04:28.014 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.010+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=620196391}) 2024-09-16T04:04:28.014 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.010+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.014 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.010+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=372033785}) 2024-09-16T04:04:28.014 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.010+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2577545078}) 2024-09-16T04:04:28.016 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.010+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.016 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.010+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=345710933}) 2024-09-16T04:04:28.016 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.010+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3979714603}) 2024-09-16T04:04:28.016 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.010+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.016 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.010+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=345710933}) 2024-09-16T04:04:28.017 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.010+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2592853735}) 2024-09-16T04:04:28.018 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.014+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.018 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.014+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=562501775}) 2024-09-16T04:04:28.018 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.014+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=366938042}) 2024-09-16T04:04:28.018 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.014+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.018 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.014+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=562501775}) 2024-09-16T04:04:28.018 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.014+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=933380538}) 2024-09-16T04:04:28.020 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.014+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.020 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.014+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3855576660}) 2024-09-16T04:04:28.020 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.014+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1529330970}) 2024-09-16T04:04:28.020 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.014+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.020 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.014+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3855576660}) 2024-09-16T04:04:28.021 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.014+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1692395857}) 2024-09-16T04:04:28.022 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.018+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.022 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.018+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4096063390}) 2024-09-16T04:04:28.022 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.018+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4240246665}) 2024-09-16T04:04:28.022 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.018+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.022 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.018+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4096063390}) 2024-09-16T04:04:28.022 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.018+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=626850699}) 2024-09-16T04:04:28.024 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.018+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.024 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.018+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1616482882}) 2024-09-16T04:04:28.025 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.018+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2254102706}) 2024-09-16T04:04:28.025 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.018+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.025 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.018+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1616482882}) 2024-09-16T04:04:28.025 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.018+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3298568114}) 2024-09-16T04:04:28.026 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.022+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.026 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.022+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2483675801}) 2024-09-16T04:04:28.026 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.022+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1510064764}) 2024-09-16T04:04:28.026 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.022+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.027 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.022+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2483675801}) 2024-09-16T04:04:28.027 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.022+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=582473032}) 2024-09-16T04:04:28.028 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.022+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.028 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.022+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3331689653}) 2024-09-16T04:04:28.029 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.022+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1335717681}) 2024-09-16T04:04:28.029 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.022+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.029 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.022+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3331689653}) 2024-09-16T04:04:28.029 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.022+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1686406208}) 2024-09-16T04:04:28.030 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.026+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.031 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.026+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2088387527}) 2024-09-16T04:04:28.031 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.026+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=273675585}) 2024-09-16T04:04:28.031 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.026+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.031 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.026+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2088387527}) 2024-09-16T04:04:28.031 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.026+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4003432052}) 2024-09-16T04:04:28.033 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.026+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.033 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.026+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3596424256}) 2024-09-16T04:04:28.033 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.026+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3479419629}) 2024-09-16T04:04:28.033 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.026+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.033 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.026+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3596424256}) 2024-09-16T04:04:28.033 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.026+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2961599635}) 2024-09-16T04:04:28.035 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.030+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.035 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.030+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3569497947}) 2024-09-16T04:04:28.035 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.030+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3515767678}) 2024-09-16T04:04:28.035 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.030+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.035 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.030+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3569497947}) 2024-09-16T04:04:28.035 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.030+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=550927936}) 2024-09-16T04:04:28.037 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.030+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.037 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.030+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2420861}) 2024-09-16T04:04:28.037 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.030+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2628390308}) 2024-09-16T04:04:28.037 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.030+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.037 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.030+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2420861}) 2024-09-16T04:04:28.037 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.030+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=750031036}) 2024-09-16T04:04:28.039 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.034+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.039 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.034+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1574626980}) 2024-09-16T04:04:28.039 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.034+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=278011781}) 2024-09-16T04:04:28.039 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.034+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.039 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.034+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1574626980}) 2024-09-16T04:04:28.039 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.034+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=455291131}) 2024-09-16T04:04:28.041 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.034+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.041 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.034+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3340728123}) 2024-09-16T04:04:28.041 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.034+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=819987926}) 2024-09-16T04:04:28.041 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.034+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.041 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.034+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3340728123}) 2024-09-16T04:04:28.041 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.034+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=974457919}) 2024-09-16T04:04:28.043 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.038+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.043 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.038+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2605839261}) 2024-09-16T04:04:28.043 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.038+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3628227084}) 2024-09-16T04:04:28.043 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.038+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.043 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.038+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2605839261}) 2024-09-16T04:04:28.043 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.038+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3861284581}) 2024-09-16T04:04:28.045 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.042+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.045 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.042+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1063299144}) 2024-09-16T04:04:28.045 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.042+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=748550260}) 2024-09-16T04:04:28.046 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.042+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.046 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.042+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1063299144}) 2024-09-16T04:04:28.046 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.042+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3990814396}) 2024-09-16T04:04:28.047 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.042+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.047 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.042+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3809831326}) 2024-09-16T04:04:28.048 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.042+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2517919705}) 2024-09-16T04:04:28.048 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.042+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.048 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.042+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3809831326}) 2024-09-16T04:04:28.048 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.042+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1841005201}) 2024-09-16T04:04:28.050 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.046+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.050 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.046+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4153678561}) 2024-09-16T04:04:28.050 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.046+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1532370312}) 2024-09-16T04:04:28.050 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.046+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.050 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.046+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4153678561}) 2024-09-16T04:04:28.050 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.046+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=610828520}) 2024-09-16T04:04:28.052 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.046+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.052 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.046+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2381852209}) 2024-09-16T04:04:28.052 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.046+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2256103156}) 2024-09-16T04:04:28.052 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.046+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.052 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.046+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2381852209}) 2024-09-16T04:04:28.052 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.046+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2942448237}) 2024-09-16T04:04:28.054 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.050+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.054 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.050+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2000827795}) 2024-09-16T04:04:28.054 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.050+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3335929337}) 2024-09-16T04:04:28.054 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.050+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.054 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.050+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2000827795}) 2024-09-16T04:04:28.054 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.050+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2158867961}) 2024-09-16T04:04:28.056 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.050+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.056 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.050+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2138490164}) 2024-09-16T04:04:28.056 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.050+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1092497855}) 2024-09-16T04:04:28.056 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.050+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.057 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.050+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2138490164}) 2024-09-16T04:04:28.057 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.050+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=638749056}) 2024-09-16T04:04:28.058 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.054+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.058 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.054+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3835285087}) 2024-09-16T04:04:28.058 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.054+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1363381359}) 2024-09-16T04:04:28.059 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.054+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.059 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.054+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3835285087}) 2024-09-16T04:04:28.059 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.054+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=70629821}) 2024-09-16T04:04:28.060 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.054+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.060 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.054+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3503395099}) 2024-09-16T04:04:28.061 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.054+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=942470643}) 2024-09-16T04:04:28.061 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.054+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.061 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.054+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3503395099}) 2024-09-16T04:04:28.061 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.054+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2750973861}) 2024-09-16T04:04:28.063 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.058+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.063 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.058+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2605922371}) 2024-09-16T04:04:28.063 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.058+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2609512469}) 2024-09-16T04:04:28.063 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.058+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.063 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.058+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2605922371}) 2024-09-16T04:04:28.063 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.058+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3765451570}) 2024-09-16T04:04:28.065 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.058+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.065 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.058+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4149674962}) 2024-09-16T04:04:28.065 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.058+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=266624370}) 2024-09-16T04:04:28.065 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.058+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.065 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.058+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4149674962}) 2024-09-16T04:04:28.065 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.058+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2166460977}) 2024-09-16T04:04:28.067 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.062+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.067 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.062+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3995147390}) 2024-09-16T04:04:28.067 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.062+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=666324351}) 2024-09-16T04:04:28.067 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.062+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.067 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.062+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3995147390}) 2024-09-16T04:04:28.067 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.062+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1626939402}) 2024-09-16T04:04:28.069 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.062+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.069 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.062+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3965097474}) 2024-09-16T04:04:28.069 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.062+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=594172965}) 2024-09-16T04:04:28.069 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.062+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.069 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.066+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3965097474}) 2024-09-16T04:04:28.069 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.066+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2879066685}) 2024-09-16T04:04:28.071 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.066+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.072 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.066+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=614008940}) 2024-09-16T04:04:28.072 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.066+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=564798924}) 2024-09-16T04:04:28.072 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.066+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.072 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.066+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=614008940}) 2024-09-16T04:04:28.072 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.066+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3313548233}) 2024-09-16T04:04:28.073 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.070+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.073 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.070+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3634208854}) 2024-09-16T04:04:28.074 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.070+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=931876728}) 2024-09-16T04:04:28.074 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.070+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.074 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.070+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3634208854}) 2024-09-16T04:04:28.074 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.070+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3782019836}) 2024-09-16T04:04:28.076 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.070+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.076 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.070+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3920583939}) 2024-09-16T04:04:28.076 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.070+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3719094047}) 2024-09-16T04:04:28.076 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.070+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.076 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.070+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3920583939}) 2024-09-16T04:04:28.076 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.070+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4082446466}) 2024-09-16T04:04:28.078 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.074+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.078 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.074+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1167110214}) 2024-09-16T04:04:28.078 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.074+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2113018540}) 2024-09-16T04:04:28.078 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.074+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.078 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.074+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1167110214}) 2024-09-16T04:04:28.078 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.074+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2876492063}) 2024-09-16T04:04:28.080 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.074+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.080 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.074+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1230474789}) 2024-09-16T04:04:28.080 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.074+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=123478042}) 2024-09-16T04:04:28.080 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.074+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.080 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.074+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1230474789}) 2024-09-16T04:04:28.080 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.074+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2633989980}) 2024-09-16T04:04:28.082 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.078+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.082 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.078+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1536341886}) 2024-09-16T04:04:28.082 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.078+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2118505048}) 2024-09-16T04:04:28.082 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.078+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.082 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.078+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1536341886}) 2024-09-16T04:04:28.082 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.078+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3019256769}) 2024-09-16T04:04:28.084 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.078+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.085 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.078+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3786537354}) 2024-09-16T04:04:28.085 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.078+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=224682967}) 2024-09-16T04:04:28.085 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.078+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.085 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.078+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3786537354}) 2024-09-16T04:04:28.085 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.078+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4100806513}) 2024-09-16T04:04:28.087 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.082+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.087 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.082+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1447536040}) 2024-09-16T04:04:28.087 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.082+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=179769714}) 2024-09-16T04:04:28.087 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.082+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.087 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.082+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1447536040}) 2024-09-16T04:04:28.087 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.082+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2761902662}) 2024-09-16T04:04:28.089 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.082+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.089 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.082+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4115738669}) 2024-09-16T04:04:28.089 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.082+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=446421909}) 2024-09-16T04:04:28.089 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.082+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.089 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.082+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4115738669}) 2024-09-16T04:04:28.089 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.082+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2688168470}) 2024-09-16T04:04:28.091 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.086+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.091 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.086+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=888024103}) 2024-09-16T04:04:28.091 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.086+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=794882550}) 2024-09-16T04:04:28.091 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.086+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.091 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.086+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=888024103}) 2024-09-16T04:04:28.091 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.086+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3538336935}) 2024-09-16T04:04:28.093 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.086+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.093 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.086+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1123931104}) 2024-09-16T04:04:28.093 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.086+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=364429175}) 2024-09-16T04:04:28.093 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.086+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.093 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.086+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1123931104}) 2024-09-16T04:04:28.093 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.086+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=47760323}) 2024-09-16T04:04:28.095 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.090+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.095 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.090+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2735800053}) 2024-09-16T04:04:28.095 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.090+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4111978229}) 2024-09-16T04:04:28.095 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.090+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.095 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.090+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2735800053}) 2024-09-16T04:04:28.095 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.090+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4060330181}) 2024-09-16T04:04:28.097 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.094+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.097 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.094+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1469505930}) 2024-09-16T04:04:28.097 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.094+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3187839452}) 2024-09-16T04:04:28.098 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.094+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.098 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.094+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1469505930}) 2024-09-16T04:04:28.098 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.094+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1392936238}) 2024-09-16T04:04:28.099 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.094+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.099 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.094+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3530462024}) 2024-09-16T04:04:28.100 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.094+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=540595289}) 2024-09-16T04:04:28.100 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.094+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.100 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.094+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3530462024}) 2024-09-16T04:04:28.100 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.094+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1867749510}) 2024-09-16T04:04:28.101 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.098+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.102 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.098+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1710209648}) 2024-09-16T04:04:28.102 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.098+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2584374130}) 2024-09-16T04:04:28.102 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.098+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.102 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.098+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1710209648}) 2024-09-16T04:04:28.102 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.098+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3667705755}) 2024-09-16T04:04:28.104 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.098+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.104 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.098+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1680523576}) 2024-09-16T04:04:28.104 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.098+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2230996357}) 2024-09-16T04:04:28.104 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.098+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.104 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.098+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1680523576}) 2024-09-16T04:04:28.104 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.098+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3412585287}) 2024-09-16T04:04:28.106 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.102+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.106 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.102+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1579255861}) 2024-09-16T04:04:28.106 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.102+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1489047846}) 2024-09-16T04:04:28.106 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.102+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.106 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.102+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1579255861}) 2024-09-16T04:04:28.106 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.102+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1496859127}) 2024-09-16T04:04:28.108 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.102+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.108 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.102+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3268033594}) 2024-09-16T04:04:28.108 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.102+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3083315354}) 2024-09-16T04:04:28.108 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.102+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.108 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.102+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3268033594}) 2024-09-16T04:04:28.108 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.102+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=338307612}) 2024-09-16T04:04:28.110 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.106+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.110 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.106+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2211719099}) 2024-09-16T04:04:28.110 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.106+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1004817536}) 2024-09-16T04:04:28.110 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.106+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.110 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.106+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2211719099}) 2024-09-16T04:04:28.110 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.106+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=556052179}) 2024-09-16T04:04:28.112 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.106+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.112 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.106+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2846095586}) 2024-09-16T04:04:28.112 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.106+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=718871304}) 2024-09-16T04:04:28.113 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.106+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.113 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.106+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2846095586}) 2024-09-16T04:04:28.113 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.106+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1337416628}) 2024-09-16T04:04:28.116 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.110+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.116 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.110+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4089379968}) 2024-09-16T04:04:28.116 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.110+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=701300756}) 2024-09-16T04:04:28.116 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.110+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.116 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.110+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4089379968}) 2024-09-16T04:04:28.116 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.110+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2729772213}) 2024-09-16T04:04:28.118 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.114+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.118 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.114+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1842359156}) 2024-09-16T04:04:28.118 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.114+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4204207815}) 2024-09-16T04:04:28.118 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.114+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.118 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.114+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1842359156}) 2024-09-16T04:04:28.118 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.114+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3996724094}) 2024-09-16T04:04:28.120 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.114+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.120 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.114+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1871119051}) 2024-09-16T04:04:28.120 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.114+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4198404224}) 2024-09-16T04:04:28.120 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.114+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.120 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.114+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1871119051}) 2024-09-16T04:04:28.121 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.114+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=283901244}) 2024-09-16T04:04:28.122 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.118+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.122 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.118+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=232037901}) 2024-09-16T04:04:28.122 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.118+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1313321491}) 2024-09-16T04:04:28.122 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.118+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.123 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.118+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=232037901}) 2024-09-16T04:04:28.123 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.118+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=414542105}) 2024-09-16T04:04:28.124 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.118+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.124 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.118+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3810192080}) 2024-09-16T04:04:28.124 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.118+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1996137247}) 2024-09-16T04:04:28.125 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.118+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.125 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.118+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3810192080}) 2024-09-16T04:04:28.125 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.118+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3831206975}) 2024-09-16T04:04:28.126 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.122+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.127 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.122+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2632712411}) 2024-09-16T04:04:28.127 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.122+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2310924955}) 2024-09-16T04:04:28.127 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.122+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.127 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.122+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2632712411}) 2024-09-16T04:04:28.127 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.122+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2963626800}) 2024-09-16T04:04:28.129 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.122+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.129 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.122+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=882173918}) 2024-09-16T04:04:28.129 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.122+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=253103766}) 2024-09-16T04:04:28.129 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.122+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.129 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.122+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=882173918}) 2024-09-16T04:04:28.129 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.122+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4230918103}) 2024-09-16T04:04:28.131 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.126+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.131 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.126+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=991323616}) 2024-09-16T04:04:28.131 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.126+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1339304156}) 2024-09-16T04:04:28.131 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.126+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.131 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.126+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=991323616}) 2024-09-16T04:04:28.131 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.126+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2930356098}) 2024-09-16T04:04:28.133 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.126+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.133 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.126+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3014903573}) 2024-09-16T04:04:28.133 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.126+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1478687883}) 2024-09-16T04:04:28.133 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.126+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.133 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.126+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3014903573}) 2024-09-16T04:04:28.133 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.126+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2626227954}) 2024-09-16T04:04:28.135 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.130+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.135 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.130+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2111622382}) 2024-09-16T04:04:28.135 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.130+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1803818061}) 2024-09-16T04:04:28.135 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.130+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.135 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.130+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2111622382}) 2024-09-16T04:04:28.135 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.130+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=83914715}) 2024-09-16T04:04:28.137 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.130+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.137 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.130+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2809790222}) 2024-09-16T04:04:28.137 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.130+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2427529154}) 2024-09-16T04:04:28.137 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.130+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.137 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.130+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2809790222}) 2024-09-16T04:04:28.137 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.130+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=734908510}) 2024-09-16T04:04:28.139 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.134+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.139 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.134+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=863653864}) 2024-09-16T04:04:28.139 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.134+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2342869816}) 2024-09-16T04:04:28.139 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.134+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.139 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.134+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=863653864}) 2024-09-16T04:04:28.139 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.134+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=787968640}) 2024-09-16T04:04:28.141 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.134+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.141 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.134+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=313897657}) 2024-09-16T04:04:28.141 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.134+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3318705376}) 2024-09-16T04:04:28.141 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.134+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.141 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.134+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=313897657}) 2024-09-16T04:04:28.141 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.134+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3112308791}) 2024-09-16T04:04:28.143 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.138+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.143 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.138+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1584032144}) 2024-09-16T04:04:28.143 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.138+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4221549686}) 2024-09-16T04:04:28.143 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.138+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.143 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.138+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1584032144}) 2024-09-16T04:04:28.143 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.138+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1950628817}) 2024-09-16T04:04:28.145 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.138+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.145 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.138+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3268849042}) 2024-09-16T04:04:28.145 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.138+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2694494659}) 2024-09-16T04:04:28.145 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.138+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.145 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.138+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3268849042}) 2024-09-16T04:04:28.145 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.138+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2805823436}) 2024-09-16T04:04:28.147 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.142+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.147 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.142+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3452706851}) 2024-09-16T04:04:28.147 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.142+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2328140377}) 2024-09-16T04:04:28.147 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.142+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.147 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.142+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3452706851}) 2024-09-16T04:04:28.147 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.142+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3708744855}) 2024-09-16T04:04:28.149 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.142+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.149 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.142+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4252110987}) 2024-09-16T04:04:28.149 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.142+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=602469584}) 2024-09-16T04:04:28.149 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.142+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.149 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.142+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4252110987}) 2024-09-16T04:04:28.149 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.142+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=236596102}) 2024-09-16T04:04:28.151 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.146+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.151 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.146+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2537593671}) 2024-09-16T04:04:28.151 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.146+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=635122095}) 2024-09-16T04:04:28.151 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.146+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.151 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.146+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2537593671}) 2024-09-16T04:04:28.151 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.146+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=152728357}) 2024-09-16T04:04:28.153 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.146+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.153 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.146+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2480908175}) 2024-09-16T04:04:28.153 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.146+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3189526724}) 2024-09-16T04:04:28.153 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.146+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.153 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.146+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2480908175}) 2024-09-16T04:04:28.153 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.146+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1715436317}) 2024-09-16T04:04:28.155 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.150+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.155 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.150+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=409461257}) 2024-09-16T04:04:28.155 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.150+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=989204166}) 2024-09-16T04:04:28.155 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.150+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.155 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.150+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=409461257}) 2024-09-16T04:04:28.155 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.150+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2712776422}) 2024-09-16T04:04:28.157 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.150+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.157 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.150+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1726628955}) 2024-09-16T04:04:28.157 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.150+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2856761377}) 2024-09-16T04:04:28.157 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.150+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.157 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.150+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1726628955}) 2024-09-16T04:04:28.157 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.150+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1236501348}) 2024-09-16T04:04:28.159 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.154+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.159 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.154+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3962818205}) 2024-09-16T04:04:28.159 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.154+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=864682038}) 2024-09-16T04:04:28.159 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.154+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.159 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.154+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3962818205}) 2024-09-16T04:04:28.159 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.154+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2853354526}) 2024-09-16T04:04:28.161 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.154+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.161 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.154+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2408824338}) 2024-09-16T04:04:28.161 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.154+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=823249003}) 2024-09-16T04:04:28.161 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.154+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.161 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.154+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2408824338}) 2024-09-16T04:04:28.161 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.154+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1746575292}) 2024-09-16T04:04:28.163 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.158+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.163 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.158+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=135414927}) 2024-09-16T04:04:28.163 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.158+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2463675260}) 2024-09-16T04:04:28.163 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.158+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.163 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.158+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=135414927}) 2024-09-16T04:04:28.163 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.158+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3224697065}) 2024-09-16T04:04:28.165 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.158+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.165 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.158+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=733612872}) 2024-09-16T04:04:28.165 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.158+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2027765620}) 2024-09-16T04:04:28.165 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.158+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.165 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.158+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=733612872}) 2024-09-16T04:04:28.165 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.158+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2969641690}) 2024-09-16T04:04:28.167 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.162+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.167 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.162+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3344725942}) 2024-09-16T04:04:28.167 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.162+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3644038593}) 2024-09-16T04:04:28.167 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.162+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.167 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.162+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3344725942}) 2024-09-16T04:04:28.167 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.162+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3105062673}) 2024-09-16T04:04:28.169 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.162+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.169 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.162+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2054159939}) 2024-09-16T04:04:28.169 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.162+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1057874485}) 2024-09-16T04:04:28.169 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.162+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.169 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.162+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2054159939}) 2024-09-16T04:04:28.169 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.162+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3398362222}) 2024-09-16T04:04:28.171 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.166+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.171 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.166+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2886430279}) 2024-09-16T04:04:28.171 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.166+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2629664613}) 2024-09-16T04:04:28.171 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.166+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.171 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.166+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2886430279}) 2024-09-16T04:04:28.171 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.166+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=903340721}) 2024-09-16T04:04:28.173 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.166+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.173 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.166+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2240401349}) 2024-09-16T04:04:28.173 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.166+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3852570022}) 2024-09-16T04:04:28.173 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.166+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.173 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.166+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2240401349}) 2024-09-16T04:04:28.173 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.166+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3194424001}) 2024-09-16T04:04:28.175 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.170+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.175 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.170+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=834805845}) 2024-09-16T04:04:28.175 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.170+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1089290994}) 2024-09-16T04:04:28.175 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.170+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.175 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.170+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=834805845}) 2024-09-16T04:04:28.175 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.170+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2977041252}) 2024-09-16T04:04:28.177 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.170+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.177 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.170+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2087860067}) 2024-09-16T04:04:28.177 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.170+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2558958267}) 2024-09-16T04:04:28.177 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.170+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.177 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.170+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2087860067}) 2024-09-16T04:04:28.177 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.170+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2955273116}) 2024-09-16T04:04:28.179 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.174+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.179 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.174+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=848443400}) 2024-09-16T04:04:28.179 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.174+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3662684141}) 2024-09-16T04:04:28.179 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.174+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.179 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.174+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=848443400}) 2024-09-16T04:04:28.179 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.174+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3179944813}) 2024-09-16T04:04:28.181 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.174+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.181 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.174+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3402371629}) 2024-09-16T04:04:28.181 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.174+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1331473277}) 2024-09-16T04:04:28.181 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.174+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.181 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.174+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3402371629}) 2024-09-16T04:04:28.181 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.174+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3646991692}) 2024-09-16T04:04:28.183 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.178+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.183 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.178+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1704715033}) 2024-09-16T04:04:28.183 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.178+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2391476596}) 2024-09-16T04:04:28.183 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.178+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.183 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.178+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1704715033}) 2024-09-16T04:04:28.183 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.178+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1408567422}) 2024-09-16T04:04:28.185 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.178+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.185 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.178+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=591491896}) 2024-09-16T04:04:28.185 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.178+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1555273573}) 2024-09-16T04:04:28.185 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.178+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.185 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.178+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=591491896}) 2024-09-16T04:04:28.185 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.178+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1790009769}) 2024-09-16T04:04:28.187 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.182+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.187 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.182+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3900374768}) 2024-09-16T04:04:28.187 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.182+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2178776723}) 2024-09-16T04:04:28.187 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.182+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.187 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.182+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3900374768}) 2024-09-16T04:04:28.187 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.182+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1546432345}) 2024-09-16T04:04:28.189 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.182+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.189 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.182+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1190396978}) 2024-09-16T04:04:28.189 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.182+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2463088696}) 2024-09-16T04:04:28.189 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.182+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.189 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.182+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1190396978}) 2024-09-16T04:04:28.189 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.182+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=111129294}) 2024-09-16T04:04:28.191 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.186+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.191 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.186+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=186577071}) 2024-09-16T04:04:28.191 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.186+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=92671796}) 2024-09-16T04:04:28.191 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.186+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.191 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.186+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=186577071}) 2024-09-16T04:04:28.191 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.186+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2256954360}) 2024-09-16T04:04:28.193 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.186+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.193 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.190+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2343237348}) 2024-09-16T04:04:28.193 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.190+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3718760931}) 2024-09-16T04:04:28.193 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.190+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.193 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.190+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2343237348}) 2024-09-16T04:04:28.194 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.190+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2538914035}) 2024-09-16T04:04:28.195 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.190+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.195 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.190+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3012673120}) 2024-09-16T04:04:28.195 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.190+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=945598627}) 2024-09-16T04:04:28.195 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.190+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.196 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.190+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3012673120}) 2024-09-16T04:04:28.196 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.190+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3608083846}) 2024-09-16T04:04:28.197 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.190+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.197 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.190+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=34435055}) 2024-09-16T04:04:28.197 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.190+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=882635852}) 2024-09-16T04:04:28.197 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.194+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.198 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.194+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=34435055}) 2024-09-16T04:04:28.198 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.194+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3647975559}) 2024-09-16T04:04:28.199 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.194+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.199 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.194+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3841941996}) 2024-09-16T04:04:28.199 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.194+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=818358346}) 2024-09-16T04:04:28.199 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.194+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.199 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.194+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3841941996}) 2024-09-16T04:04:28.200 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.194+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=308151652}) 2024-09-16T04:04:28.201 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.198+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.201 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.198+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3947289844}) 2024-09-16T04:04:28.202 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.198+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=242967238}) 2024-09-16T04:04:28.202 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.198+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.202 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.198+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3947289844}) 2024-09-16T04:04:28.202 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.198+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1587693227}) 2024-09-16T04:04:28.203 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.198+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.203 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.198+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1791708949}) 2024-09-16T04:04:28.204 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.198+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1224396833}) 2024-09-16T04:04:28.204 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.198+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.204 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.198+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1791708949}) 2024-09-16T04:04:28.204 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.198+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3233655292}) 2024-09-16T04:04:28.205 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.202+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.206 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.202+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2140691956}) 2024-09-16T04:04:28.206 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.202+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4037864899}) 2024-09-16T04:04:28.206 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.202+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.206 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.202+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2140691956}) 2024-09-16T04:04:28.206 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.202+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3443099345}) 2024-09-16T04:04:28.207 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.202+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.208 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.202+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2058192477}) 2024-09-16T04:04:28.208 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.202+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2579258406}) 2024-09-16T04:04:28.208 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.202+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.208 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.202+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2058192477}) 2024-09-16T04:04:28.208 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.202+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2423893876}) 2024-09-16T04:04:28.209 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.206+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.209 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.206+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=319141380}) 2024-09-16T04:04:28.210 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.206+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2485611761}) 2024-09-16T04:04:28.210 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.206+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.210 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.206+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=319141380}) 2024-09-16T04:04:28.210 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.206+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1231722447}) 2024-09-16T04:04:28.212 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.206+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.212 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.206+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1744570958}) 2024-09-16T04:04:28.212 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.206+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=557967551}) 2024-09-16T04:04:28.212 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.206+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.212 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.206+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1744570958}) 2024-09-16T04:04:28.212 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.206+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2494796623}) 2024-09-16T04:04:28.214 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.210+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.214 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.210+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3840742521}) 2024-09-16T04:04:28.214 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.210+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3833234863}) 2024-09-16T04:04:28.214 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.210+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.214 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.210+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3840742521}) 2024-09-16T04:04:28.214 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.210+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2755796823}) 2024-09-16T04:04:28.216 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.210+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.216 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.210+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=419338415}) 2024-09-16T04:04:28.216 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.210+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1237692516}) 2024-09-16T04:04:28.216 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.210+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.216 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.210+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=419338415}) 2024-09-16T04:04:28.216 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.210+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4100214349}) 2024-09-16T04:04:28.218 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.214+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.218 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.214+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2291303989}) 2024-09-16T04:04:28.218 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.214+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1461015343}) 2024-09-16T04:04:28.218 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.214+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.219 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.214+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2291303989}) 2024-09-16T04:04:28.219 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.214+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1196073201}) 2024-09-16T04:04:28.220 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.214+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.221 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.214+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1018477273}) 2024-09-16T04:04:28.221 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.214+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=43324369}) 2024-09-16T04:04:28.221 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.214+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.221 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.214+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1018477273}) 2024-09-16T04:04:28.221 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.214+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3428902808}) 2024-09-16T04:04:28.222 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.218+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.222 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.218+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=261134353}) 2024-09-16T04:04:28.222 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.218+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1223163332}) 2024-09-16T04:04:28.222 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.218+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.223 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.218+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=261134353}) 2024-09-16T04:04:28.223 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.218+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2720199141}) 2024-09-16T04:04:28.225 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.218+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.225 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.222+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=8,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=10,monmap=2,osd_metadata=6,osd_pg_creating=1,osd_snap=41,osdmap=8} crc {logm=2503630280,mdsmap=4088524775,mgr=366503433,mgr_command_descs=2107673600,mgr_metadata=1367272202,mgrstat=3040880244,mon_config_key=1481991774,monmap=268092033,osd_metadata=4182959540,osd_pg_creating=3036340958,osd_snap=3148683592,osdmap=666449599}) 2024-09-16T04:04:28.225 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.222+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=9,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=11,monmap=3,osd_metadata=4,osd_pg_creating=1,osd_snap=40,osdmap=8} crc {logm=2041910544,mdsmap=4088524775,mgr=460109776,mgr_command_descs=2107673600,mgr_metadata=1367272202,mgrstat=1258138986,mon_config_key=85057112,monmap=3937513981,osd_metadata=3306448908,osd_pg_creating=3036340958,osd_snap=2570991635,osdmap=2676022504}) 2024-09-16T04:04:28.225 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.222+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.226 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.222+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=8,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=10,monmap=2,osd_metadata=6,osd_pg_creating=1,osd_snap=41,osdmap=8} crc {logm=2503630280,mdsmap=4088524775,mgr=366503433,mgr_command_descs=2107673600,mgr_metadata=1367272202,mgrstat=3040880244,mon_config_key=1481991774,monmap=268092033,osd_metadata=4182959540,osd_pg_creating=3036340958,osd_snap=3148683592,osdmap=666449599}) 2024-09-16T04:04:28.226 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.222+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=9,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=40,osdmap=5} crc {logm=2041910544,mdsmap=4088524775,mgr=460109776,mgr_command_descs=2107673600,mgr_metadata=1367272202,mgrstat=3040880244,mon_config_key=85057112,monmap=3937513981,osd_metadata=4182959540,osd_pg_creating=3036340958,osd_snap=4133070595,osdmap=1259260822}) 2024-09-16T04:04:28.227 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.222+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.228 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.222+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3209665950}) 2024-09-16T04:04:28.228 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.222+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1524798418}) 2024-09-16T04:04:28.228 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.222+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.228 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.222+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3209665950}) 2024-09-16T04:04:28.228 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.222+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2508566168}) 2024-09-16T04:04:28.230 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.226+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.230 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.226+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=15530851}) 2024-09-16T04:04:28.230 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.226+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1143916394}) 2024-09-16T04:04:28.230 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.226+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.230 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.226+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=15530851}) 2024-09-16T04:04:28.230 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.226+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=997934202}) 2024-09-16T04:04:28.234 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.230+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.234 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.230+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3843125298}) 2024-09-16T04:04:28.234 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.230+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3738423784}) 2024-09-16T04:04:28.235 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.230+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.235 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.230+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3843125298}) 2024-09-16T04:04:28.235 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.230+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=115617320}) 2024-09-16T04:04:28.248 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.242+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.248 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.242+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=43} crc {osdmap=1834279067}) 2024-09-16T04:04:28.248 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.242+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=42} crc {osdmap=884568460}) 2024-09-16T04:04:28.248 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.242+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:04:28.248 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.242+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=43} crc {osdmap=1834279067}) 2024-09-16T04:04:28.248 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:04:28.242+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=42} crc {osdmap=753722844}) 2024-09-16T04:04:28.967 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T04:04:28.967 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:04:29.099 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.094+0000 7f6229011640 10 monclient: get_monmap_and_config 2024-09-16T04:04:29.099 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.094+0000 7f6229011640 10 monclient: build_initial_monmap 2024-09-16T04:04:29.099 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.094+0000 7f6229011640 10 monclient: monmap: 2024-09-16T04:04:29.099 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:04:29.100 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:04:29.100 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:04:29.097943+0000 2024-09-16T04:04:29.100 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:04:29.097943+0000 2024-09-16T04:04:29.100 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:04:29.100 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:04:29.100 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:04:29.100 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:04:29.100 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:04:29.100 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:04:29.100 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:04:29.100 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:04:29.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.094+0000 7f6229011640 1 Processor -- start 2024-09-16T04:04:29.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.094+0000 7f6229011640 1 -- start start 2024-09-16T04:04:29.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.094+0000 7f6229011640 10 monclient: init 2024-09-16T04:04:29.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.094+0000 7f6229011640 10 monclient: _reopen_session rank -1 2024-09-16T04:04:29.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.094+0000 7f6229011640 10 monclient: _add_conns ranks=[0,3,2,4,1] 2024-09-16T04:04:29.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.094+0000 7f6229011640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f62240a1330 addr v1:172.21.15.42:6789/0 2024-09-16T04:04:29.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.094+0000 7f6229011640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f622406dac0 0x7f622413a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:04:29.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.094+0000 7f6229011640 10 monclient(hunting): picked mon.noname-a con 0x7f622406dac0 addr v2:172.21.15.42:3300/0 2024-09-16T04:04:29.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.094+0000 7f6229011640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f62240a1e20 0x7f622413c890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:04:29.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.094+0000 7f6229011640 10 monclient(hunting): picked mon.noname-b con 0x7f62240a1e20 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:04:29.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.094+0000 7f6229011640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f622409c090 con 0x7f62240a1330 2024-09-16T04:04:29.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.094+0000 7f6229011640 10 monclient(hunting): start opening mon connection 2024-09-16T04:04:29.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.094+0000 7f6229011640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f622409c890 con 0x7f622406dac0 2024-09-16T04:04:29.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.094+0000 7f6229011640 10 monclient(hunting): start opening mon connection 2024-09-16T04:04:29.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.094+0000 7f6229011640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f622409f610 con 0x7f62240a1e20 2024-09-16T04:04:29.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.094+0000 7f6229011640 10 monclient(hunting): _renew_subs 2024-09-16T04:04:29.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.094+0000 7f6229011640 10 monclient(hunting): authenticate will time out at 2316.067601s 2024-09-16T04:04:29.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.094+0000 7f622259c640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f622406dac0 0x7f622413a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:29.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.094+0000 7f622359e640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f62240a1e20 0x7f622413c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:29.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.094+0000 7f622259c640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f622406dac0 0x7f622413a340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:39690/0 (socket says 172.21.15.42:39690) 2024-09-16T04:04:29.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.094+0000 7f6222d9d640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f62240a1330 0x7f62240a1710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:39728/0 (socket says 172.21.15.42:39728) 2024-09-16T04:04:29.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.094+0000 7f622259c640 1 -- 172.21.15.42:0/2504784814 learned_addr learned my addr 172.21.15.42:0/2504784814 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:04:29.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.094+0000 7f622259c640 10 monclient(hunting): get_auth_request con 0x7f622406dac0 auth_method 0 2024-09-16T04:04:29.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.094+0000 7f622259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:04:29.105 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.094+0000 7f622259c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:04:29.105 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.094+0000 7f622259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:04:29.105 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.094+0000 7f622359e640 10 monclient(hunting): get_auth_request con 0x7f62240a1e20 auth_method 0 2024-09-16T04:04:29.105 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.094+0000 7f622359e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:04:29.105 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.094+0000 7f622359e640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:04:29.105 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.094+0000 7f622359e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:04:29.105 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.094+0000 7f622259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:04:29.105 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.094+0000 7f622259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:04:29.105 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.094+0000 7f622259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:04:29.105 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.094+0000 7f622359e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:04:29.105 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.094+0000 7f622359e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:04:29.105 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.094+0000 7f622359e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:04:29.105 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.098+0000 7f622259c640 10 monclient(hunting): handle_auth_done global_id 984138 payload 274 2024-09-16T04:04:29.106 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.098+0000 7f622259c640 1 -- 172.21.15.42:0/2504784814 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f62240a1e20 msgr2=0x7f622413c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:04:29.106 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.098+0000 7f622259c640 1 --2- 172.21.15.42:0/2504784814 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f62240a1e20 0x7f622413c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:29.106 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.098+0000 7f622259c640 1 -- 172.21.15.42:0/2504784814 >> v1:172.21.15.42:6789/0 conn(0x7f62240a1330 legacy=0x7f62240a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:04:29.106 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.098+0000 7f622259c640 10 monclient: _finish_hunting 0 2024-09-16T04:04:29.106 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.098+0000 7f622259c640 1 monclient: found mon.noname-a 2024-09-16T04:04:29.106 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.098+0000 7f622259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:04:29.106 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.098+0000 7f622259c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:04:29.107 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.098+0000 7f622259c640 1 -- 172.21.15.42:0/2504784814 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f622409f8f0 con 0x7f622406dac0 2024-09-16T04:04:29.114 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.098+0000 7f622259c640 1 --2- 172.21.15.42:0/2504784814 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f622406dac0 0x7f622413a340 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f620c009870 tx=0x7f620c01c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56e6d516bdc5bd63 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:04:29.114 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.098+0000 7f6221d9b640 1 -- 172.21.15.42:0/2504784814 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f620c0aa600 con 0x7f622406dac0 2024-09-16T04:04:29.114 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.098+0000 7f6221d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:04:29.114 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.098+0000 7f6221d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:04:29.114 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.098+0000 7f6221d9b640 10 monclient: dump: 2024-09-16T04:04:29.115 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:04:29.115 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:04:29.115 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:29.115 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:29.115 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:04:29.115 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:04:29.115 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:04:29.115 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:04:29.115 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:04:29.115 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:04:29.115 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.098+0000 7f6221d9b640 10 monclient: _finish_auth 0 2024-09-16T04:04:29.115 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.098+0000 7f6221d9b640 10 monclient: _check_auth_tickets 2024-09-16T04:04:29.116 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.098+0000 7f6221d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:04:29.116 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.098+0000 7f6221d9b640 1 -- 172.21.15.42:0/2504784814 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f620c0af070 con 0x7f622406dac0 2024-09-16T04:04:29.116 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.098+0000 7f6229011640 5 monclient: authenticate success, global_id 984138 2024-09-16T04:04:29.116 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.098+0000 7f6229011640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:04:29.116 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.098+0000 7f6221d9b640 10 monclient: handle_config config(0 keys) 2024-09-16T04:04:29.116 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.098+0000 7f6229011640 10 monclient: get_monmap_and_config success 2024-09-16T04:04:29.116 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.098+0000 7f6229011640 4 set_mon_vals no callback set 2024-09-16T04:04:29.116 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.098+0000 7f6221d9b640 1 -- 172.21.15.42:0/2504784814 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f620c0aa9c0 con 0x7f622406dac0 2024-09-16T04:04:29.116 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.098+0000 7f6221d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:04:29.116 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.098+0000 7f6221d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:04:29.117 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.098+0000 7f6221d9b640 10 monclient: dump: 2024-09-16T04:04:29.117 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:04:29.117 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:04:29.117 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:29.117 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:29.117 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:04:29.117 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:04:29.117 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:04:29.117 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:04:29.117 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:04:29.117 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:04:29.118 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.098+0000 7f6229011640 10 monclient: shutdown 2024-09-16T04:04:29.118 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.098+0000 7f6229011640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:04:29.118 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.098+0000 7f6229011640 1 -- 172.21.15.42:0/2504784814 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f622406dac0 msgr2=0x7f622413a340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:04:29.118 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.098+0000 7f6229011640 1 --2- 172.21.15.42:0/2504784814 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f622406dac0 0x7f622413a340 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f620c009870 tx=0x7f620c01c480 comp rx=0 tx=0).stop 2024-09-16T04:04:29.118 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.098+0000 7f6229011640 1 -- 172.21.15.42:0/2504784814 shutdown_connections 2024-09-16T04:04:29.118 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.098+0000 7f6229011640 1 --2- 172.21.15.42:0/2504784814 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f62240a1e20 0x7f622413c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:29.118 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.098+0000 7f6229011640 1 --2- 172.21.15.42:0/2504784814 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f622406dac0 0x7f622413a340 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:29.118 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.098+0000 7f6229011640 1 -- 172.21.15.42:0/2504784814 >> 172.21.15.42:0/2504784814 conn(0x7f62240a2420 msgr2=0x7f62240a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:04:29.119 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.098+0000 7f6229011640 1 -- 172.21.15.42:0/2504784814 shutdown_connections 2024-09-16T04:04:29.119 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.098+0000 7f6229011640 1 -- 172.21.15.42:0/2504784814 wait complete. 2024-09-16T04:04:29.119 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.098+0000 7f6229011640 10 monclient: build_initial_monmap 2024-09-16T04:04:29.119 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.102+0000 7f6229011640 10 monclient: monmap: 2024-09-16T04:04:29.119 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:04:29.119 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:04:29.119 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:04:29.119 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:04:29.119 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:04:29.119 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:04:29.119 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:04:29.119 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:04:29.120 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:04:29.120 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:04:29.120 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.102+0000 7f6229011640 1 librados: starting msgr at 2024-09-16T04:04:29.120 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.102+0000 7f6229011640 1 librados: starting objecter 2024-09-16T04:04:29.120 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.102+0000 7f6229011640 1 Processor -- start 2024-09-16T04:04:29.120 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.102+0000 7f6229011640 1 -- start start 2024-09-16T04:04:29.120 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.102+0000 7f6229011640 1 librados: setting wanted keys 2024-09-16T04:04:29.120 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.102+0000 7f6229011640 1 librados: calling monclient init 2024-09-16T04:04:29.120 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.102+0000 7f6229011640 10 monclient: init 2024-09-16T04:04:29.120 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.102+0000 7f6229011640 10 monclient: _reopen_session rank -1 2024-09-16T04:04:29.120 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.102+0000 7f6229011640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:04:29.120 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.102+0000 7f6229011640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f62240a1330 0x7f62241cf070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:04:29.120 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.102+0000 7f6229011640 10 monclient(hunting): picked mon.noname-b con 0x7f62240a1330 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:04:29.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.102+0000 7f6229011640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f62240a1e20 0x7f6224135570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:04:29.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.102+0000 7f6229011640 10 monclient(hunting): picked mon.noname-c con 0x7f62240a1e20 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:04:29.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.102+0000 7f6229011640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6224135ab0 0x7f622411df90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:04:29.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.102+0000 7f6229011640 10 monclient(hunting): picked mon.noname-a con 0x7f6224135ab0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:04:29.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.102+0000 7f6229011640 10 monclient(hunting): start opening mon connection 2024-09-16T04:04:29.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.102+0000 7f6229011640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f622413f770 con 0x7f6224135ab0 2024-09-16T04:04:29.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.102+0000 7f6229011640 10 monclient(hunting): start opening mon connection 2024-09-16T04:04:29.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.102+0000 7f6229011640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f622413f5f0 con 0x7f62240a1330 2024-09-16T04:04:29.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.102+0000 7f6229011640 10 monclient(hunting): start opening mon connection 2024-09-16T04:04:29.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.102+0000 7f6229011640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f622413f8f0 con 0x7f62240a1e20 2024-09-16T04:04:29.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.102+0000 7f6229011640 10 monclient(hunting): _renew_subs 2024-09-16T04:04:29.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.102+0000 7f6229011640 10 monclient(hunting): authenticate will time out at 2316.073184s 2024-09-16T04:04:29.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.102+0000 7f622259c640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f62240a1e20 0x7f6224135570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:29.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.102+0000 7f622359e640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6224135ab0 0x7f622411df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:29.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.102+0000 7f622359e640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6224135ab0 0x7f622411df90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:39702/0 (socket says 172.21.15.42:39702) 2024-09-16T04:04:29.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.102+0000 7f622259c640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f62240a1e20 0x7f6224135570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:52474/0 (socket says 172.21.15.42:52474) 2024-09-16T04:04:29.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.102+0000 7f622359e640 1 -- 172.21.15.42:0/2410462616 learned_addr learned my addr 172.21.15.42:0/2410462616 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:04:29.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.102+0000 7f622359e640 10 monclient(hunting): get_auth_request con 0x7f6224135ab0 auth_method 0 2024-09-16T04:04:29.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.102+0000 7f622359e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:04:29.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.102+0000 7f622359e640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:04:29.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.102+0000 7f622359e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:04:29.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.102+0000 7f622259c640 10 monclient(hunting): get_auth_request con 0x7f62240a1e20 auth_method 0 2024-09-16T04:04:29.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.102+0000 7f622259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:04:29.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.102+0000 7f622259c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:04:29.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.102+0000 7f622259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:04:29.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.102+0000 7f622359e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:04:29.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.102+0000 7f622359e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:04:29.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.102+0000 7f622359e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:04:29.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.102+0000 7f622259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:04:29.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.102+0000 7f622259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:04:29.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.102+0000 7f622259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:04:29.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.102+0000 7f6222d9d640 1 --2- 172.21.15.42:0/2410462616 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f62240a1330 0x7f62241cf070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:29.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.102+0000 7f6222d9d640 10 monclient(hunting): get_auth_request con 0x7f62240a1330 auth_method 0 2024-09-16T04:04:29.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.102+0000 7f6222d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:04:29.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.102+0000 7f6222d9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:04:29.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.102+0000 7f6222d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:04:29.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.102+0000 7f622359e640 10 monclient(hunting): handle_auth_done global_id 984144 payload 824 2024-09-16T04:04:29.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.102+0000 7f622359e640 1 -- 172.21.15.42:0/2410462616 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f62240a1e20 msgr2=0x7f6224135570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:04:29.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.102+0000 7f622359e640 1 --2- 172.21.15.42:0/2410462616 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f62240a1e20 0x7f6224135570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:29.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.102+0000 7f622359e640 1 -- 172.21.15.42:0/2410462616 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f62240a1330 msgr2=0x7f62241cf070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:04:29.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.102+0000 7f622359e640 1 --2- 172.21.15.42:0/2410462616 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f62240a1330 0x7f62241cf070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:29.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.102+0000 7f622359e640 10 monclient: _finish_hunting 0 2024-09-16T04:04:29.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.102+0000 7f622359e640 1 monclient: found mon.noname-a 2024-09-16T04:04:29.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.102+0000 7f622359e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:04:29.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.102+0000 7f622359e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:04:29.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.102+0000 7f622359e640 1 -- 172.21.15.42:0/2410462616 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6224073a30 con 0x7f6224135ab0 2024-09-16T04:04:29.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.102+0000 7f622359e640 1 --2- 172.21.15.42:0/2410462616 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6224135ab0 0x7f622411df90 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f621c00efc0 tx=0x7f621c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:04:29.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.102+0000 7f621b7fe640 10 client.?.objecter ms_handle_connect 0x7f6224135ab0 2024-09-16T04:04:29.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.102+0000 7f621b7fe640 10 client.?.objecter resend_mon_ops 2024-09-16T04:04:29.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.102+0000 7f621b7fe640 1 -- 172.21.15.42:0/2410462616 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f621c018020 con 0x7f6224135ab0 2024-09-16T04:04:29.125 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.102+0000 7f621b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:04:29.125 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.102+0000 7f621b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:04:29.125 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.102+0000 7f621b7fe640 10 monclient: dump: 2024-09-16T04:04:29.125 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:04:29.125 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:04:29.125 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:29.125 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:29.125 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:04:29.125 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:04:29.125 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:04:29.125 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:04:29.125 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:04:29.125 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:04:29.125 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.102+0000 7f621b7fe640 10 monclient: _finish_auth 0 2024-09-16T04:04:29.126 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.102+0000 7f621b7fe640 10 monclient: _check_auth_tickets 2024-09-16T04:04:29.126 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.102+0000 7f621b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:04:29.126 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.102+0000 7f621b7fe640 1 -- 172.21.15.42:0/2410462616 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f621c014480 con 0x7f6224135ab0 2024-09-16T04:04:29.126 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.102+0000 7f621b7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:04:29.126 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.102+0000 7f621b7fe640 1 -- 172.21.15.42:0/2410462616 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f621c014870 con 0x7f6224135ab0 2024-09-16T04:04:29.126 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.102+0000 7f621b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:04:29.126 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.102+0000 7f621b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:04:29.126 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.102+0000 7f621b7fe640 10 monclient: dump: 2024-09-16T04:04:29.126 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:04:29.126 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:04:29.126 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:29.126 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:29.126 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:04:29.126 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:04:29.127 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:04:29.127 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:04:29.127 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:04:29.127 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:04:29.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.102+0000 7f622159a640 4 set_mon_vals no callback set 2024-09-16T04:04:29.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.102+0000 7f6229011640 5 monclient: authenticate success, global_id 984144 2024-09-16T04:04:29.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.102+0000 7f6229011640 10 monclient: _renew_subs 2024-09-16T04:04:29.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.102+0000 7f6229011640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:04:29.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.102+0000 7f6229011640 1 -- 172.21.15.42:0/2410462616 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6224121960 con 0x7f6224135ab0 2024-09-16T04:04:29.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.102+0000 7f6229011640 10 client.984144.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:04:29.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.102+0000 7f6229011640 10 monclient: _renew_subs 2024-09-16T04:04:29.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.102+0000 7f6229011640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:04:29.128 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.102+0000 7f6229011640 1 -- 172.21.15.42:0/2410462616 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f622413c9f0 con 0x7f6224135ab0 2024-09-16T04:04:29.128 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.102+0000 7f6229011640 1 librados: init done 2024-09-16T04:04:29.128 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.102+0000 7f621b7fe640 1 -- 172.21.15.42:0/2410462616 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f621c01b020 con 0x7f6224135ab0 2024-09-16T04:04:29.128 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.102+0000 7f6229011640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:04:29.128 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.102+0000 7f6229011640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:04:29.128 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.102+0000 7f6229011640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:04:29.128 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.102+0000 7f6229011640 1 -- 172.21.15.42:0/2410462616 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f62240a1710 con 0x7f6224135ab0 2024-09-16T04:04:29.128 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.102+0000 7f621b7fe640 1 --2- 172.21.15.42:0/2410462616 >> v2:172.21.15.154:6812/254845342 conn(0x7f6208037810 0x7f6208039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:04:29.128 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.102+0000 7f621b7fe640 1 -- 172.21.15.42:0/2410462616 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(680..680 src has 451..680) ==== 6280+0+0 (secure 0 0 0) 0x7f621c049b80 con 0x7f6224135ab0 2024-09-16T04:04:29.128 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.102+0000 7f621b7fe640 10 client.984144.objecter ms_dispatch 0x7f6224000ba0 osd_map(680..680 src has 451..680) 2024-09-16T04:04:29.128 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.102+0000 7f621b7fe640 3 client.984144.objecter handle_osd_map got epochs [680,680] > 0 2024-09-16T04:04:29.128 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.102+0000 7f621b7fe640 3 client.984144.objecter handle_osd_map decoding full epoch 680 2024-09-16T04:04:29.128 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.102+0000 7f621b7fe640 20 client.984144.objecter dump_active .. 0 homeless 2024-09-16T04:04:29.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.106+0000 7f6222d9d640 1 --2- 172.21.15.42:0/2410462616 >> v2:172.21.15.154:6812/254845342 conn(0x7f6208037810 0x7f6208039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:29.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.106+0000 7f6222d9d640 10 monclient: get_auth_request con 0x7f6208037810 auth_method 0 2024-09-16T04:04:29.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.106+0000 7f6222d9d640 1 --2- 172.21.15.42:0/2410462616 >> v2:172.21.15.154:6812/254845342 conn(0x7f6208037810 0x7f6208039cd0 secure :-1 s=READY pgs=3825 cs=0 l=1 rev1=1 crypto rx=0x7f6214000f70 tx=0x7f6214007cd0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:04:29.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.106+0000 7f621b7fe640 10 client.984144.objecter ms_handle_connect 0x7f6208037810 2024-09-16T04:04:29.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.110+0000 7f621b7fe640 1 -- 172.21.15.42:0/2410462616 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f621c022070 con 0x7f6224135ab0 2024-09-16T04:04:29.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.110+0000 7f621b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:04:29.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.110+0000 7f621b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:04:29.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.334+0000 7f6229011640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:04:29.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.334+0000 7f6229011640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:04:29.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.334+0000 7f6229011640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:04:29.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.334+0000 7f6229011640 1 -- 172.21.15.42:0/2410462616 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f6224120510 con 0x7f6224135ab0 2024-09-16T04:04:29.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.334+0000 7f621b7fe640 1 -- 172.21.15.42:0/2410462616 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7f6224073a30 con 0x7f6224135ab0 2024-09-16T04:04:29.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.334+0000 7f621b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:04:29.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.334+0000 7f621b7fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:04:29.340 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":210,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:04:29.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.338+0000 7f6229011640 10 librados: watch_flush enter 2024-09-16T04:04:29.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.338+0000 7f6229011640 10 librados: watch_flush exit 2024-09-16T04:04:29.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.338+0000 7f6229011640 20 client.984144.objecter shutdown clearing up homeless session... 2024-09-16T04:04:29.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.338+0000 7f6229011640 10 client.984144.objecter successfully canceled tick 2024-09-16T04:04:29.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.338+0000 7f6229011640 1 -- 172.21.15.42:0/2410462616 >> v2:172.21.15.154:6812/254845342 conn(0x7f6208037810 msgr2=0x7f6208039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:04:29.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.338+0000 7f6229011640 1 --2- 172.21.15.42:0/2410462616 >> v2:172.21.15.154:6812/254845342 conn(0x7f6208037810 0x7f6208039cd0 secure :-1 s=READY pgs=3825 cs=0 l=1 rev1=1 crypto rx=0x7f6214000f70 tx=0x7f6214007cd0 comp rx=0 tx=0).stop 2024-09-16T04:04:29.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.338+0000 7f6229011640 10 monclient: shutdown 2024-09-16T04:04:29.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.338+0000 7f6229011640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:04:29.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.338+0000 7f6229011640 1 -- 172.21.15.42:0/2410462616 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6224135ab0 msgr2=0x7f622411df90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:04:29.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.338+0000 7f6229011640 1 --2- 172.21.15.42:0/2410462616 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6224135ab0 0x7f622411df90 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f621c00efc0 tx=0x7f621c00c490 comp rx=0 tx=0).stop 2024-09-16T04:04:29.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.338+0000 7f6229011640 1 -- 172.21.15.42:0/2410462616 shutdown_connections 2024-09-16T04:04:29.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.338+0000 7f6229011640 1 --2- 172.21.15.42:0/2410462616 >> v2:172.21.15.154:6812/254845342 conn(0x7f6208037810 0x7f6208039cd0 unknown :-1 s=CLOSED pgs=3825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:29.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.338+0000 7f6229011640 1 --2- 172.21.15.42:0/2410462616 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6224135ab0 0x7f622411df90 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:29.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.338+0000 7f6229011640 1 --2- 172.21.15.42:0/2410462616 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f62240a1e20 0x7f6224135570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:29.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.338+0000 7f6229011640 1 --2- 172.21.15.42:0/2410462616 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f62240a1330 0x7f62241cf070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:29.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.338+0000 7f6229011640 1 -- 172.21.15.42:0/2410462616 >> 172.21.15.42:0/2410462616 conn(0x7f62240a2420 msgr2=0x7f62240a45b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:04:29.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.338+0000 7f6229011640 1 -- 172.21.15.42:0/2410462616 shutdown_connections 2024-09-16T04:04:29.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.338+0000 7f6229011640 1 -- 172.21.15.42:0/2410462616 wait complete. 2024-09-16T04:04:29.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.338+0000 7f6229011640 1 librados: shutdown 2024-09-16T04:04:29.360 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T04:04:29.360 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T04:04:29.360 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T04:04:29.360 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T04:04:29.479 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.474+0000 7f21ba8bc640 10 monclient: get_monmap_and_config 2024-09-16T04:04:29.479 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.474+0000 7f21ba8bc640 10 monclient: build_initial_monmap 2024-09-16T04:04:29.479 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.474+0000 7f21ba8bc640 10 monclient: monmap: 2024-09-16T04:04:29.479 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:04:29.479 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:04:29.479 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:04:29.477430+0000 2024-09-16T04:04:29.480 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:04:29.477430+0000 2024-09-16T04:04:29.480 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:04:29.480 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:04:29.480 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:04:29.480 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:04:29.480 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:04:29.480 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:04:29.480 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:04:29.480 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:04:29.485 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.474+0000 7f21ba8bc640 1 Processor -- start 2024-09-16T04:04:29.485 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.474+0000 7f21ba8bc640 1 -- start start 2024-09-16T04:04:29.485 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.474+0000 7f21ba8bc640 10 monclient: init 2024-09-16T04:04:29.485 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.474+0000 7f21ba8bc640 10 monclient: _reopen_session rank -1 2024-09-16T04:04:29.485 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.474+0000 7f21ba8bc640 10 monclient: _add_conns ranks=[0,3,2,1,4] 2024-09-16T04:04:29.485 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.474+0000 7f21ba8bc640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f21b40a67e0 addr v1:172.21.15.42:6789/0 2024-09-16T04:04:29.485 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.474+0000 7f21ba8bc640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f21b4072560 0x7f21b413a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:04:29.485 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.474+0000 7f21ba8bc640 10 monclient(hunting): picked mon.noname-a con 0x7f21b4072560 addr v2:172.21.15.42:3300/0 2024-09-16T04:04:29.485 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.474+0000 7f21ba8bc640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f21b40a72d0 0x7f21b413c820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:04:29.486 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.474+0000 7f21ba8bc640 10 monclient(hunting): picked mon.noname-b con 0x7f21b40a72d0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:04:29.486 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.474+0000 7f21ba8bc640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f21b4071b80 con 0x7f21b40a67e0 2024-09-16T04:04:29.486 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.474+0000 7f21ba8bc640 10 monclient(hunting): start opening mon connection 2024-09-16T04:04:29.486 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.474+0000 7f21ba8bc640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f21b4072ed0 con 0x7f21b4072560 2024-09-16T04:04:29.486 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.474+0000 7f21ba8bc640 10 monclient(hunting): start opening mon connection 2024-09-16T04:04:29.486 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.474+0000 7f21ba8bc640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f21b4070030 con 0x7f21b40a72d0 2024-09-16T04:04:29.486 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.474+0000 7f21ba8bc640 10 monclient(hunting): _renew_subs 2024-09-16T04:04:29.486 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.474+0000 7f21ba8bc640 10 monclient(hunting): authenticate will time out at 2316.447196s 2024-09-16T04:04:29.486 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.474+0000 7f21b37fe640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f21b4072560 0x7f21b413a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:29.486 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.474+0000 7f21b37fe640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f21b4072560 0x7f21b413a2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:39718/0 (socket says 172.21.15.42:39718) 2024-09-16T04:04:29.486 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.474+0000 7f21b37fe640 1 -- 172.21.15.42:0/4180304720 learned_addr learned my addr 172.21.15.42:0/4180304720 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:04:29.486 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.474+0000 7f21b37fe640 10 monclient(hunting): get_auth_request con 0x7f21b4072560 auth_method 0 2024-09-16T04:04:29.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.474+0000 7f21b37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:04:29.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.474+0000 7f21b37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:04:29.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.474+0000 7f21b37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:04:29.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.474+0000 7f21b3fff640 1 --1- 172.21.15.42:0/4180304720 >> v1:172.21.15.42:6789/0 conn(0x7f21b40a67e0 0x7f21b40a6bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:39742/0 (socket says 172.21.15.42:39742) 2024-09-16T04:04:29.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.474+0000 7f21b8e59640 1 --2- 172.21.15.42:0/4180304720 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f21b40a72d0 0x7f21b413c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:29.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.474+0000 7f21b8e59640 10 monclient(hunting): get_auth_request con 0x7f21b40a72d0 auth_method 0 2024-09-16T04:04:29.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.474+0000 7f21b8e59640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:04:29.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.474+0000 7f21b8e59640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:04:29.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.474+0000 7f21b8e59640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:04:29.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.474+0000 7f21b37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:04:29.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.474+0000 7f21b37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:04:29.488 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.474+0000 7f21b37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:04:29.488 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.474+0000 7f21b8e59640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:04:29.488 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.474+0000 7f21b8e59640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:04:29.488 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.474+0000 7f21b8e59640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:04:29.488 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.474+0000 7f21b37fe640 10 monclient(hunting): handle_auth_done global_id 984147 payload 274 2024-09-16T04:04:29.488 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.474+0000 7f21b37fe640 1 -- 172.21.15.42:0/4180304720 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f21b40a72d0 msgr2=0x7f21b413c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:04:29.488 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.474+0000 7f21b37fe640 1 --2- 172.21.15.42:0/4180304720 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f21b40a72d0 0x7f21b413c820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:29.488 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.474+0000 7f21b37fe640 1 -- 172.21.15.42:0/4180304720 >> v1:172.21.15.42:6789/0 conn(0x7f21b40a67e0 legacy=0x7f21b40a6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:04:29.488 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.474+0000 7f21b37fe640 10 monclient: _finish_hunting 0 2024-09-16T04:04:29.488 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.474+0000 7f21b37fe640 1 monclient: found mon.noname-a 2024-09-16T04:04:29.488 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.474+0000 7f21b37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:04:29.488 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.474+0000 7f21b37fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:04:29.488 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.474+0000 7f21b37fe640 1 -- 172.21.15.42:0/4180304720 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21b40712e0 con 0x7f21b4072560 2024-09-16T04:04:29.489 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.478+0000 7f21b37fe640 1 --2- 172.21.15.42:0/4180304720 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f21b4072560 0x7f21b413a2d0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f21a4009e50 tx=0x7f21a401c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b488d25d33356de7 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:04:29.489 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.478+0000 7f21b2ffd640 1 -- 172.21.15.42:0/4180304720 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21a40a5690 con 0x7f21b4072560 2024-09-16T04:04:29.489 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.478+0000 7f21b2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:04:29.489 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.478+0000 7f21b2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:04:29.489 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.478+0000 7f21b2ffd640 10 monclient: dump: 2024-09-16T04:04:29.489 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:04:29.489 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:04:29.489 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:29.489 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:29.489 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:04:29.489 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:04:29.489 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:04:29.489 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:04:29.490 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:04:29.490 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:04:29.490 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.478+0000 7f21b2ffd640 10 monclient: _finish_auth 0 2024-09-16T04:04:29.490 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.478+0000 7f21b2ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:04:29.490 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.478+0000 7f21b2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:04:29.490 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.478+0000 7f21ba8bc640 5 monclient: authenticate success, global_id 984147 2024-09-16T04:04:29.490 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.478+0000 7f21b2ffd640 1 -- 172.21.15.42:0/4180304720 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f21a40a5830 con 0x7f21b4072560 2024-09-16T04:04:29.490 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.478+0000 7f21b2ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:04:29.490 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.478+0000 7f21ba8bc640 10 monclient: get_monmap_and_config success 2024-09-16T04:04:29.490 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.478+0000 7f21b2ffd640 1 -- 172.21.15.42:0/4180304720 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21a40a5c00 con 0x7f21b4072560 2024-09-16T04:04:29.490 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.478+0000 7f21b2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:04:29.490 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.478+0000 7f21ba8bc640 4 set_mon_vals no callback set 2024-09-16T04:04:29.490 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.478+0000 7f21b2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:04:29.491 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.478+0000 7f21b2ffd640 10 monclient: dump: 2024-09-16T04:04:29.491 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:04:29.491 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:04:29.491 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:29.491 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:29.491 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:04:29.491 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:04:29.491 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:04:29.491 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:04:29.491 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:04:29.491 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:04:29.491 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.478+0000 7f21ba8bc640 10 monclient: shutdown 2024-09-16T04:04:29.492 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.478+0000 7f21ba8bc640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:04:29.492 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.478+0000 7f21ba8bc640 1 -- 172.21.15.42:0/4180304720 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f21b4072560 msgr2=0x7f21b413a2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:04:29.492 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.478+0000 7f21ba8bc640 1 --2- 172.21.15.42:0/4180304720 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f21b4072560 0x7f21b413a2d0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f21a4009e50 tx=0x7f21a401c8c0 comp rx=0 tx=0).stop 2024-09-16T04:04:29.492 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.478+0000 7f21ba8bc640 1 -- 172.21.15.42:0/4180304720 shutdown_connections 2024-09-16T04:04:29.492 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.478+0000 7f21ba8bc640 1 --2- 172.21.15.42:0/4180304720 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f21b40a72d0 0x7f21b413c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:29.492 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.478+0000 7f21ba8bc640 1 --2- 172.21.15.42:0/4180304720 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f21b4072560 0x7f21b413a2d0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:29.492 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.478+0000 7f21ba8bc640 1 -- 172.21.15.42:0/4180304720 >> 172.21.15.42:0/4180304720 conn(0x7f21b40a7ed0 msgr2=0x7f21b40a43d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:04:29.492 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.478+0000 7f21ba8bc640 1 -- 172.21.15.42:0/4180304720 shutdown_connections 2024-09-16T04:04:29.492 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.478+0000 7f21ba8bc640 1 -- 172.21.15.42:0/4180304720 wait complete. 2024-09-16T04:04:29.492 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.478+0000 7f21ba8bc640 10 monclient: build_initial_monmap 2024-09-16T04:04:29.492 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.478+0000 7f21ba8bc640 10 monclient: monmap: 2024-09-16T04:04:29.492 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:04:29.492 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:04:29.493 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:04:29.493 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:04:29.493 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:04:29.493 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:04:29.493 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:04:29.493 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:04:29.493 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:04:29.493 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:04:29.493 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.478+0000 7f21ba8bc640 1 librados: starting msgr at 2024-09-16T04:04:29.493 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.478+0000 7f21ba8bc640 1 librados: starting objecter 2024-09-16T04:04:29.493 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.478+0000 7f21ba8bc640 1 Processor -- start 2024-09-16T04:04:29.493 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.478+0000 7f21ba8bc640 1 -- start start 2024-09-16T04:04:29.493 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.478+0000 7f21ba8bc640 1 librados: setting wanted keys 2024-09-16T04:04:29.493 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.478+0000 7f21ba8bc640 1 librados: calling monclient init 2024-09-16T04:04:29.493 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.478+0000 7f21ba8bc640 10 monclient: init 2024-09-16T04:04:29.499 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.482+0000 7f21ba8bc640 10 monclient: _reopen_session rank -1 2024-09-16T04:04:29.499 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.482+0000 7f21ba8bc640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:04:29.499 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.482+0000 7f21ba8bc640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f21b4072560 0x7f21b413b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:04:29.500 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.482+0000 7f21ba8bc640 10 monclient(hunting): picked mon.noname-a con 0x7f21b4072560 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:04:29.500 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.482+0000 7f21ba8bc640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f21b40a67e0 0x7f21b41325e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:04:29.500 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.482+0000 7f21ba8bc640 10 monclient(hunting): picked mon.noname-b con 0x7f21b40a67e0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:04:29.500 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.482+0000 7f21ba8bc640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f21b40a72d0 0x7f21b41ced30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:04:29.500 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.482+0000 7f21ba8bc640 10 monclient(hunting): picked mon.noname-c con 0x7f21b40a72d0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:04:29.500 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.482+0000 7f21ba8bc640 10 monclient(hunting): start opening mon connection 2024-09-16T04:04:29.500 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.482+0000 7f21ba8bc640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f21b413f700 con 0x7f21b4072560 2024-09-16T04:04:29.500 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.482+0000 7f21ba8bc640 10 monclient(hunting): start opening mon connection 2024-09-16T04:04:29.500 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.482+0000 7f21ba8bc640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f21b413f580 con 0x7f21b40a67e0 2024-09-16T04:04:29.500 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.482+0000 7f21ba8bc640 10 monclient(hunting): start opening mon connection 2024-09-16T04:04:29.500 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.482+0000 7f21ba8bc640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f21b413f880 con 0x7f21b40a72d0 2024-09-16T04:04:29.500 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.482+0000 7f21ba8bc640 10 monclient(hunting): _renew_subs 2024-09-16T04:04:29.500 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.482+0000 7f21ba8bc640 10 monclient(hunting): authenticate will time out at 2316.452250s 2024-09-16T04:04:29.501 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.482+0000 7f21b3fff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f21b4072560 0x7f21b413b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:29.501 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.482+0000 7f21b3fff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f21b4072560 0x7f21b413b910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:39728/0 (socket says 172.21.15.42:39728) 2024-09-16T04:04:29.501 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.482+0000 7f21b8e59640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f21b40a72d0 0x7f21b41ced30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:29.501 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.482+0000 7f21b3fff640 1 -- 172.21.15.42:0/3735189196 learned_addr learned my addr 172.21.15.42:0/3735189196 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:04:29.501 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.482+0000 7f21b3fff640 10 monclient(hunting): get_auth_request con 0x7f21b4072560 auth_method 0 2024-09-16T04:04:29.501 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.482+0000 7f21b3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:04:29.501 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.482+0000 7f21b3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:04:29.501 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.482+0000 7f21b3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:04:29.501 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.482+0000 7f21b8e59640 10 monclient(hunting): get_auth_request con 0x7f21b40a72d0 auth_method 0 2024-09-16T04:04:29.501 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.482+0000 7f21b8e59640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:04:29.501 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.482+0000 7f21b8e59640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:04:29.501 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.482+0000 7f21b8e59640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:04:29.501 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.482+0000 7f21b3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:04:29.501 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.482+0000 7f21b3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:04:29.502 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.482+0000 7f21b3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:04:29.502 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.482+0000 7f21b8e59640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:04:29.502 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.482+0000 7f21b8e59640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:04:29.502 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.482+0000 7f21b8e59640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:04:29.502 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.482+0000 7f21b3fff640 10 monclient(hunting): handle_auth_done global_id 984153 payload 824 2024-09-16T04:04:29.502 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.482+0000 7f21b3fff640 1 -- 172.21.15.42:0/3735189196 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f21b40a72d0 msgr2=0x7f21b41ced30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:04:29.502 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.482+0000 7f21b37fe640 1 --2- 172.21.15.42:0/3735189196 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f21b40a67e0 0x7f21b41325e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:29.502 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.482+0000 7f21b3fff640 1 --2- 172.21.15.42:0/3735189196 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f21b40a72d0 0x7f21b41ced30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:29.502 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.482+0000 7f21b3fff640 1 -- 172.21.15.42:0/3735189196 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f21b40a67e0 msgr2=0x7f21b41325e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:04:29.502 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.482+0000 7f21b3fff640 1 --2- 172.21.15.42:0/3735189196 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f21b40a67e0 0x7f21b41325e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:29.502 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.482+0000 7f21b3fff640 10 monclient: _finish_hunting 0 2024-09-16T04:04:29.502 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.482+0000 7f21b3fff640 1 monclient: found mon.noname-a 2024-09-16T04:04:29.502 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.482+0000 7f21b3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:04:29.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.482+0000 7f21b3fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:04:29.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.482+0000 7f21b3fff640 1 -- 172.21.15.42:0/3735189196 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21b409bd20 con 0x7f21b4072560 2024-09-16T04:04:29.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.482+0000 7f21b8e59640 1 --2- 172.21.15.42:0/3735189196 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f21b40a72d0 0x7f21b41ced30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:04:29.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.482+0000 7f21b37fe640 10 monclient: get_auth_request con 0x7f21b40a67e0 auth_method 0 2024-09-16T04:04:29.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.482+0000 7f21b37fe640 1 --2- 172.21.15.42:0/3735189196 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f21b40a67e0 0x7f21b41325e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T04:04:29.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.482+0000 7f21b3fff640 1 --2- 172.21.15.42:0/3735189196 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f21b4072560 0x7f21b413b910 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f219c00c450 tx=0x7f219c00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:04:29.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.482+0000 7f21b0ff9640 10 client.?.objecter ms_handle_connect 0x7f21b4072560 2024-09-16T04:04:29.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.482+0000 7f21b0ff9640 10 client.?.objecter resend_mon_ops 2024-09-16T04:04:29.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.482+0000 7f21b0ff9640 1 -- 172.21.15.42:0/3735189196 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f219c018070 con 0x7f21b4072560 2024-09-16T04:04:29.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.482+0000 7f21b0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:04:29.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.482+0000 7f21b0ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:04:29.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.482+0000 7f21b0ff9640 10 monclient: dump: 2024-09-16T04:04:29.503 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:04:29.503 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:04:29.504 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:29.504 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:29.504 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:04:29.504 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:04:29.504 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:04:29.504 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:04:29.504 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:04:29.504 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:04:29.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.482+0000 7f21b0ff9640 10 monclient: _finish_auth 0 2024-09-16T04:04:29.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.482+0000 7f21b0ff9640 10 monclient: _check_auth_tickets 2024-09-16T04:04:29.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.482+0000 7f21b0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:04:29.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.482+0000 7f21b0ff9640 1 -- 172.21.15.42:0/3735189196 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f219c007480 con 0x7f21b4072560 2024-09-16T04:04:29.505 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.482+0000 7f21ba8bc640 5 monclient: authenticate success, global_id 984153 2024-09-16T04:04:29.505 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.482+0000 7f21ba8bc640 10 monclient: _renew_subs 2024-09-16T04:04:29.505 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.482+0000 7f21ba8bc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:04:29.505 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.482+0000 7f21ba8bc640 1 -- 172.21.15.42:0/3735189196 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21b413c990 con 0x7f21b4072560 2024-09-16T04:04:29.505 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.482+0000 7f21b0ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T04:04:29.505 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.482+0000 7f21b0ff9640 1 -- 172.21.15.42:0/3735189196 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f219c0047f0 con 0x7f21b4072560 2024-09-16T04:04:29.505 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.482+0000 7f21b0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:04:29.505 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.482+0000 7f21b0ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:04:29.505 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.482+0000 7f21b0ff9640 10 monclient: dump: 2024-09-16T04:04:29.505 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:04:29.505 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:04:29.506 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:29.506 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:29.506 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:04:29.506 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:04:29.506 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:04:29.506 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:04:29.506 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:04:29.506 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:04:29.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.482+0000 7f21b27fc640 4 set_mon_vals no callback set 2024-09-16T04:04:29.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.482+0000 7f21ba8bc640 10 client.984153.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:04:29.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.482+0000 7f21ba8bc640 10 monclient: _renew_subs 2024-09-16T04:04:29.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.482+0000 7f21ba8bc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:04:29.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.482+0000 7f21ba8bc640 1 -- 172.21.15.42:0/3735189196 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21b411e9d0 con 0x7f21b4072560 2024-09-16T04:04:29.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.482+0000 7f21ba8bc640 1 librados: init done 2024-09-16T04:04:29.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.482+0000 7f21ba8bc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:04:29.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.482+0000 7f21ba8bc640 1 --2- 172.21.15.42:0/3735189196 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f21b4132b20 0x7f21b41cf270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:04:29.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.482+0000 7f21ba8bc640 10 monclient(hunting): start opening mon connection 2024-09-16T04:04:29.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.482+0000 7f21ba8bc640 1 -- 172.21.15.42:0/3735189196 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f21b406f120 con 0x7f21b4132b20 2024-09-16T04:04:29.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.482+0000 7f21b37fe640 1 --2- 172.21.15.42:0/3735189196 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f21b4132b20 0x7f21b41cf270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:29.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.482+0000 7f21b0ff9640 1 -- 172.21.15.42:0/3735189196 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f219c01b070 con 0x7f21b4072560 2024-09-16T04:04:29.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.482+0000 7f21b37fe640 10 monclient: get_auth_request con 0x7f21b4132b20 auth_method 0 2024-09-16T04:04:29.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.482+0000 7f21b37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:04:29.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.482+0000 7f21b37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:04:29.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.482+0000 7f21b37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:04:29.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.482+0000 7f21b37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:04:29.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.482+0000 7f21b37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:04:29.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.482+0000 7f21b37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:04:29.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.482+0000 7f21b0ff9640 1 --2- 172.21.15.42:0/3735189196 >> v2:172.21.15.154:6812/254845342 conn(0x7f2194037810 0x7f2194039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:04:29.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.482+0000 7f21b0ff9640 1 -- 172.21.15.42:0/3735189196 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(680..680 src has 451..680) ==== 6280+0+0 (secure 0 0 0) 0x7f219c0482f0 con 0x7f21b4072560 2024-09-16T04:04:29.508 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.482+0000 7f21b0ff9640 10 client.984153.objecter ms_dispatch 0x7f21b4000ba0 osd_map(680..680 src has 451..680) 2024-09-16T04:04:29.508 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.482+0000 7f21b0ff9640 3 client.984153.objecter handle_osd_map got epochs [680,680] > 0 2024-09-16T04:04:29.508 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.482+0000 7f21b37fe640 10 monclient(hunting): handle_auth_done global_id 984156 payload 824 2024-09-16T04:04:29.508 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.482+0000 7f21b0ff9640 3 client.984153.objecter handle_osd_map decoding full epoch 680 2024-09-16T04:04:29.508 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.482+0000 7f21b37fe640 1 -- 172.21.15.42:0/3735189196 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f21b406dd50 con 0x7f21b4132b20 2024-09-16T04:04:29.508 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.482+0000 7f21b0ff9640 20 client.984153.objecter dump_active .. 0 homeless 2024-09-16T04:04:29.508 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.482+0000 7f21b0ff9640 10 client.984153.objecter ms_handle_connect 0x7f21b4132b20 2024-09-16T04:04:29.508 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.482+0000 7f21b0ff9640 10 client.984153.objecter resend_mon_ops 2024-09-16T04:04:29.508 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.482+0000 7f21b37fe640 1 --2- 172.21.15.42:0/3735189196 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f21b4132b20 0x7f21b41cf270 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f21b40a77f0 tx=0x7f21a4019660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:04:29.508 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.482+0000 7f21b8e59640 1 --2- 172.21.15.42:0/3735189196 >> v2:172.21.15.154:6812/254845342 conn(0x7f2194037810 0x7f2194039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:29.508 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.482+0000 7f21b0ff9640 1 -- 172.21.15.42:0/3735189196 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21a40bd030 con 0x7f21b4132b20 2024-09-16T04:04:29.508 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.482+0000 7f21b0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:04:29.508 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.482+0000 7f21b0ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:04:29.508 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.482+0000 7f21b0ff9640 10 monclient: dump: 2024-09-16T04:04:29.509 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:04:29.509 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:04:29.509 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:29.509 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:29.509 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:04:29.509 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:04:29.509 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:04:29.509 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:04:29.509 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:04:29.509 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:04:29.509 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.482+0000 7f21b8e59640 10 monclient: get_auth_request con 0x7f2194037810 auth_method 0 2024-09-16T04:04:29.509 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.482+0000 7f21b0ff9640 1 -- 172.21.15.42:0/3735189196 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f21a40b9470 con 0x7f21b4132b20 2024-09-16T04:04:29.509 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.482+0000 7f21b0ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:04:29.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.482+0000 7f21b0ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:04:29.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.482+0000 7f21b0ff9640 1 -- 172.21.15.42:0/3735189196 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f21b4132b20 msgr2=0x7f21b41cf270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:04:29.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.482+0000 7f21b0ff9640 1 --2- 172.21.15.42:0/3735189196 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f21b4132b20 0x7f21b41cf270 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f21b40a77f0 tx=0x7f21a4019660 comp rx=0 tx=0).stop 2024-09-16T04:04:29.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.482+0000 7f21b0ff9640 1 -- 172.21.15.42:0/3735189196 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f21b4132b20 msgr2=0x7f21b41cf270 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:04:29.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.482+0000 7f21b0ff9640 1 --2- 172.21.15.42:0/3735189196 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f21b4132b20 0x7f21b41cf270 secure :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f21b40a77f0 tx=0x7f21a4019660 comp rx=0 tx=0).stop 2024-09-16T04:04:29.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.482+0000 7f21b8e59640 1 --2- 172.21.15.42:0/3735189196 >> v2:172.21.15.154:6812/254845342 conn(0x7f2194037810 0x7f2194039cd0 secure :-1 s=READY pgs=3826 cs=0 l=1 rev1=1 crypto rx=0x7f21b41cfe10 tx=0x7f21ac009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:04:29.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.482+0000 7f21b0ff9640 10 client.984153.objecter ms_handle_connect 0x7f2194037810 2024-09-16T04:04:29.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.490+0000 7f21ba8bc640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:04:29.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.490+0000 7f21ba8bc640 1 --2- 172.21.15.42:0/3735189196 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f21b411ec10 0x7f21b41cf7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:04:29.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.490+0000 7f21ba8bc640 10 monclient(hunting): start opening mon connection 2024-09-16T04:04:29.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.490+0000 7f21ba8bc640 1 -- 172.21.15.42:0/3735189196 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f21b40a6bc0 con 0x7f21b411ec10 2024-09-16T04:04:29.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.490+0000 7f21b37fe640 1 --2- 172.21.15.42:0/3735189196 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f21b411ec10 0x7f21b41cf7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:29.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.490+0000 7f21b37fe640 10 monclient: get_auth_request con 0x7f21b411ec10 auth_method 0 2024-09-16T04:04:29.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.490+0000 7f21b37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:04:29.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.490+0000 7f21b37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:04:29.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.490+0000 7f21b37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:04:29.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.490+0000 7f21b37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:04:29.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.490+0000 7f21b37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:04:29.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.490+0000 7f21b37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:04:29.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.490+0000 7f21b37fe640 10 monclient(hunting): handle_auth_done global_id 984159 payload 824 2024-09-16T04:04:29.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.490+0000 7f21b37fe640 1 -- 172.21.15.42:0/3735189196 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f21b406f2f0 con 0x7f21b411ec10 2024-09-16T04:04:29.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.490+0000 7f21b0ff9640 10 client.984153.objecter ms_handle_connect 0x7f21b411ec10 2024-09-16T04:04:29.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.490+0000 7f21b37fe640 1 --2- 172.21.15.42:0/3735189196 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f21b411ec10 0x7f21b41cf7b0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f21a401a4f0 tx=0x7f21a40a7000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:04:29.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.490+0000 7f21b0ff9640 10 client.984153.objecter resend_mon_ops 2024-09-16T04:04:29.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.490+0000 7f21b0ff9640 1 -- 172.21.15.42:0/3735189196 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21a40c6070 con 0x7f21b411ec10 2024-09-16T04:04:29.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.490+0000 7f21b0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:04:29.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.490+0000 7f21b0ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:04:29.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.490+0000 7f21b0ff9640 10 monclient: dump: 2024-09-16T04:04:29.512 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:04:29.512 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:04:29.512 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:29.512 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:29.512 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:04:29.512 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:04:29.512 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:04:29.512 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:04:29.513 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:04:29.513 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:04:29.513 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.490+0000 7f21b0ff9640 1 -- 172.21.15.42:0/3735189196 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4714 (secure 0 0 0) 0x7f21a40a5e50 con 0x7f21b411ec10 2024-09-16T04:04:29.513 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.490+0000 7f21b0ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:04:29.513 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.490+0000 7f21b0ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:04:29.513 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.490+0000 7f21b0ff9640 1 -- 172.21.15.42:0/3735189196 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f21b411ec10 msgr2=0x7f21b41cf7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:04:29.513 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.490+0000 7f21b0ff9640 1 --2- 172.21.15.42:0/3735189196 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f21b411ec10 0x7f21b41cf7b0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f21a401a4f0 tx=0x7f21a40a7000 comp rx=0 tx=0).stop 2024-09-16T04:04:29.513 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.490+0000 7f21b0ff9640 1 -- 172.21.15.42:0/3735189196 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f21b411ec10 msgr2=0x7f21b41cf7b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:04:29.513 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.490+0000 7f21b0ff9640 1 --2- 172.21.15.42:0/3735189196 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f21b411ec10 0x7f21b41cf7b0 secure :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f21a401a4f0 tx=0x7f21a40a7000 comp rx=0 tx=0).stop 2024-09-16T04:04:29.513 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T04:04:29.514 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:04:29.514 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:04:29.514 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "leader", 2024-09-16T04:04:29.514 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 210, 2024-09-16T04:04:29.514 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 29547, 2024-09-16T04:04:29.514 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T04:04:29.514 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T04:04:29.514 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T04:04:29.514 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T04:04:29.514 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:04:29.514 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 4, 2024-09-16T04:04:29.514 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:04:29.514 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T04:04:29.515 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T04:04:29.515 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:04:29.515 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:04:29.515 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:04:29.515 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:04:29.515 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:04:29.515 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:04:29.515 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:04:29.515 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:04:29.515 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:04:29.515 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:04:29.515 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:04:29.515 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:04:29.516 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:04:29.516 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T04:04:29.516 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:04:29.516 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:04:29.516 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:04:29.516 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:04:29.516 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:04:29.516 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:04:29.516 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:04:29.516 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:04:29.516 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:04:29.516 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:04:29.516 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:04:29.517 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:04:29.517 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:04:29.517 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T04:04:29.517 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T04:04:29.517 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T04:04:29.517 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T04:04:29.517 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T04:04:29.517 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T04:04:29.517 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:04:29.517 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:04:29.517 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T04:04:29.517 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T04:04:29.518 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T04:04:29.518 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T04:04:29.518 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T04:04:29.518 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T04:04:29.518 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T04:04:29.518 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:04:29.518 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T04:04:29.518 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:04:29.518 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:04:29.519 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:04:29.519 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:04:29.519 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:04:29.519 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:04:29.519 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:04:29.519 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:04:29.519 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:04:29.519 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:04:29.519 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:04:29.519 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:04:29.519 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T04:04:29.519 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:04:29.519 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T04:04:29.519 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:04:29.520 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:04:29.520 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:04:29.520 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:04:29.520 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:04:29.520 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:04:29.520 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:04:29.520 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T04:04:29.520 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:04:29.520 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:04:29.520 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:04:29.520 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:04:29.520 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T04:04:29.520 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:04:29.520 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:04:29.520 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:04:29.520 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:04:29.521 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T04:04:29.521 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T04:04:29.521 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:04:29.521 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:04:29.521 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:04:29.521 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:04:29.521 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:04:29.521 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:04:29.521 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:04:29.521 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:04:29.521 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:04:29.521 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:04:29.521 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:04:29.521 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T04:04:29.521 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:04:29.522 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:04:29.522 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:04:29.522 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:04:29.522 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T04:04:29.522 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:04:29.522 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:04:29.522 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:04:29.522 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:04:29.522 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T04:04:29.522 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T04:04:29.522 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:04:29.522 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:04:29.523 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:04:29.523 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:04:29.523 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:04:29.523 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:04:29.523 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:04:29.523 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:04:29.523 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:04:29.523 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:04:29.523 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:04:29.523 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T04:04:29.523 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:04:29.523 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:04:29.523 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:04:29.524 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:04:29.524 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T04:04:29.524 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:04:29.524 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:04:29.524 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:04:29.524 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:04:29.524 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T04:04:29.524 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T04:04:29.524 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:04:29.524 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:04:29.524 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:04:29.524 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:04:29.524 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:04:29.525 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:04:29.525 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T04:04:29.525 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T04:04:29.525 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:04:29.525 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:04:29.525 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:04:29.525 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:04:29.525 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:04:29.525 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:04:29.525 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T04:04:29.525 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:04:29.525 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:04:29.525 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:04:29.525 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 3 2024-09-16T04:04:29.526 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:04:29.526 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:04:29.526 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T04:04:29.526 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:04:29.526 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:04:29.526 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:04:29.526 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 3 2024-09-16T04:04:29.526 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:04:29.526 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:04:29.526 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:04:29.526 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T04:04:29.526 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T04:04:29.528 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.490+0000 7f21ba8bc640 10 librados: watch_flush enter 2024-09-16T04:04:29.528 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.490+0000 7f21ba8bc640 10 librados: watch_flush exit 2024-09-16T04:04:29.528 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.490+0000 7f21ba8bc640 20 client.984153.objecter shutdown clearing up homeless session... 2024-09-16T04:04:29.528 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.490+0000 7f21ba8bc640 10 client.984153.objecter successfully canceled tick 2024-09-16T04:04:29.528 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.490+0000 7f21ba8bc640 1 -- 172.21.15.42:0/3735189196 >> v2:172.21.15.154:6812/254845342 conn(0x7f2194037810 msgr2=0x7f2194039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:04:29.528 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.490+0000 7f21ba8bc640 1 --2- 172.21.15.42:0/3735189196 >> v2:172.21.15.154:6812/254845342 conn(0x7f2194037810 0x7f2194039cd0 secure :-1 s=READY pgs=3826 cs=0 l=1 rev1=1 crypto rx=0x7f21b41cfe10 tx=0x7f21ac009190 comp rx=0 tx=0).stop 2024-09-16T04:04:29.528 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.490+0000 7f21ba8bc640 10 monclient: shutdown 2024-09-16T04:04:29.528 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.490+0000 7f21ba8bc640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:04:29.529 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.490+0000 7f21ba8bc640 1 -- 172.21.15.42:0/3735189196 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f21b4072560 msgr2=0x7f21b413b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:04:29.529 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.490+0000 7f21ba8bc640 1 --2- 172.21.15.42:0/3735189196 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f21b4072560 0x7f21b413b910 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f219c00c450 tx=0x7f219c00c910 comp rx=0 tx=0).stop 2024-09-16T04:04:29.529 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.490+0000 7f21b8e59640 1 -- 172.21.15.42:0/3735189196 reap_dead start 2024-09-16T04:04:29.529 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.490+0000 7f21b8e59640 1 -- 172.21.15.42:0/3735189196 reap_dead start 2024-09-16T04:04:29.529 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.490+0000 7f21ba8bc640 1 -- 172.21.15.42:0/3735189196 shutdown_connections 2024-09-16T04:04:29.529 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.490+0000 7f21ba8bc640 1 -- 172.21.15.42:0/3735189196 >> 172.21.15.42:0/3735189196 conn(0x7f21b40a7ed0 msgr2=0x7f21b4135290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:04:29.529 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.490+0000 7f21ba8bc640 1 -- 172.21.15.42:0/3735189196 shutdown_connections 2024-09-16T04:04:29.529 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.490+0000 7f21ba8bc640 1 -- 172.21.15.42:0/3735189196 wait complete. 2024-09-16T04:04:29.529 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.490+0000 7f21ba8bc640 1 librados: shutdown 2024-09-16T04:04:29.529 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:04:29.530 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T04:04:29.530 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T04:04:29.658 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.654+0000 7fa5c5062640 10 monclient: get_monmap_and_config 2024-09-16T04:04:29.658 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.654+0000 7fa5c5062640 10 monclient: build_initial_monmap 2024-09-16T04:04:29.658 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.654+0000 7fa5c5062640 10 monclient: monmap: 2024-09-16T04:04:29.658 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:04:29.658 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:04:29.658 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:04:29.656499+0000 2024-09-16T04:04:29.658 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:04:29.656499+0000 2024-09-16T04:04:29.658 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:04:29.658 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:04:29.658 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:04:29.658 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:04:29.659 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:04:29.659 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:04:29.659 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:04:29.659 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:04:29.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.654+0000 7fa5c5062640 1 Processor -- start 2024-09-16T04:04:29.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.654+0000 7fa5c5062640 1 -- start start 2024-09-16T04:04:29.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.654+0000 7fa5c5062640 10 monclient: init 2024-09-16T04:04:29.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.654+0000 7fa5c5062640 10 monclient: _reopen_session rank -1 2024-09-16T04:04:29.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.654+0000 7fa5c5062640 10 monclient: _add_conns ranks=[1,3,4,2,0] 2024-09-16T04:04:29.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.654+0000 7fa5c5062640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa5c00a1330 addr v1:172.21.15.154:6789/0 2024-09-16T04:04:29.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.654+0000 7fa5c5062640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fa5c006dac0 0x7fa5c013a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:04:29.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.654+0000 7fa5c5062640 10 monclient(hunting): picked mon.noname-a con 0x7fa5c006dac0 addr v2:172.21.15.42:3300/0 2024-09-16T04:04:29.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.654+0000 7fa5c5062640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fa5c00a1e20 0x7fa5c013c890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:04:29.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.654+0000 7fa5c5062640 10 monclient(hunting): picked mon.noname-c con 0x7fa5c00a1e20 addr v2:172.21.15.154:3300/0 2024-09-16T04:04:29.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.654+0000 7fa5c5062640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa5c009c090 con 0x7fa5c00a1330 2024-09-16T04:04:29.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.654+0000 7fa5c5062640 10 monclient(hunting): start opening mon connection 2024-09-16T04:04:29.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.654+0000 7fa5c5062640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7fa5c009f610 con 0x7fa5c006dac0 2024-09-16T04:04:29.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.654+0000 7fa5c5062640 10 monclient(hunting): start opening mon connection 2024-09-16T04:04:29.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.654+0000 7fa5c5062640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fa5c009f8f0 con 0x7fa5c00a1e20 2024-09-16T04:04:29.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.654+0000 7fa5c5062640 10 monclient(hunting): _renew_subs 2024-09-16T04:04:29.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.654+0000 7fa5c5062640 10 monclient(hunting): authenticate will time out at 2316.626398s 2024-09-16T04:04:29.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.654+0000 7fa5be59c640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fa5c006dac0 0x7fa5c013a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:29.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.654+0000 7fa5be59c640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fa5c006dac0 0x7fa5c013a340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:39750/0 (socket says 172.21.15.42:39750) 2024-09-16T04:04:29.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.654+0000 7fa5be59c640 1 -- 172.21.15.42:0/385520961 learned_addr learned my addr 172.21.15.42:0/385520961 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:04:29.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.654+0000 7fa5be59c640 10 monclient(hunting): get_auth_request con 0x7fa5c006dac0 auth_method 0 2024-09-16T04:04:29.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.654+0000 7fa5be59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:04:29.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.654+0000 7fa5be59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:04:29.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.654+0000 7fa5be59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:04:29.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.654+0000 7fa5be59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:04:29.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.654+0000 7fa5be59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:04:29.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.654+0000 7fa5be59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:04:29.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.654+0000 7fa5bf59e640 1 --2- 172.21.15.42:0/385520961 >> v2:172.21.15.154:3300/0 conn(0x7fa5c00a1e20 0x7fa5c013c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:29.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.654+0000 7fa5bf59e640 10 monclient(hunting): get_auth_request con 0x7fa5c00a1e20 auth_method 0 2024-09-16T04:04:29.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.654+0000 7fa5bf59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:04:29.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.654+0000 7fa5bf59e640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:04:29.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.654+0000 7fa5bf59e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:04:29.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.654+0000 7fa5be59c640 10 monclient(hunting): handle_auth_done global_id 984162 payload 274 2024-09-16T04:04:29.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.654+0000 7fa5be59c640 1 -- 172.21.15.42:0/385520961 >> v2:172.21.15.154:3300/0 conn(0x7fa5c00a1e20 msgr2=0x7fa5c013c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:04:29.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.654+0000 7fa5be59c640 1 --2- 172.21.15.42:0/385520961 >> v2:172.21.15.154:3300/0 conn(0x7fa5c00a1e20 0x7fa5c013c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:29.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.654+0000 7fa5be59c640 1 -- 172.21.15.42:0/385520961 >> v1:172.21.15.154:6789/0 conn(0x7fa5c00a1330 legacy=0x7fa5c00a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:04:29.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.654+0000 7fa5be59c640 10 monclient: _finish_hunting 0 2024-09-16T04:04:29.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.654+0000 7fa5be59c640 1 monclient: found mon.noname-a 2024-09-16T04:04:29.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.654+0000 7fa5be59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:04:29.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.654+0000 7fa5be59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:04:29.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.654+0000 7fa5be59c640 1 -- 172.21.15.42:0/385520961 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5c009fbd0 con 0x7fa5c006dac0 2024-09-16T04:04:29.670 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.658+0000 7fa5be59c640 1 --2- 172.21.15.42:0/385520961 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa5c006dac0 0x7fa5c013a340 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fa5c009c800 tx=0x7fa5b401c430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d9dfff069a8043 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:04:29.670 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.658+0000 7fa5bdd9b640 1 -- 172.21.15.42:0/385520961 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5b40b0070 con 0x7fa5c006dac0 2024-09-16T04:04:29.670 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.658+0000 7fa5bdd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:04:29.670 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.658+0000 7fa5bdd9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:04:29.670 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.658+0000 7fa5bdd9b640 10 monclient: dump: 2024-09-16T04:04:29.670 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:04:29.670 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:04:29.670 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:29.670 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:29.670 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:04:29.671 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:04:29.671 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:04:29.671 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:04:29.671 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:04:29.671 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:04:29.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.658+0000 7fa5bdd9b640 10 monclient: _finish_auth 0 2024-09-16T04:04:29.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.658+0000 7fa5bdd9b640 10 monclient: _check_auth_tickets 2024-09-16T04:04:29.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.658+0000 7fa5bdd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:04:29.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.658+0000 7fa5c5062640 5 monclient: authenticate success, global_id 984162 2024-09-16T04:04:29.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.658+0000 7fa5c5062640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:04:29.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.658+0000 7fa5bdd9b640 1 -- 172.21.15.42:0/385520961 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa5b40ab600 con 0x7fa5c006dac0 2024-09-16T04:04:29.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.658+0000 7fa5bdd9b640 10 monclient: handle_config config(0 keys) 2024-09-16T04:04:29.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.658+0000 7fa5c5062640 10 monclient: get_monmap_and_config success 2024-09-16T04:04:29.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.658+0000 7fa5c5062640 4 set_mon_vals no callback set 2024-09-16T04:04:29.672 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.658+0000 7fa5bdd9b640 1 -- 172.21.15.42:0/385520961 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5b40ab9d0 con 0x7fa5c006dac0 2024-09-16T04:04:29.672 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.658+0000 7fa5bdd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:04:29.672 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.658+0000 7fa5bdd9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:04:29.672 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.658+0000 7fa5bdd9b640 10 monclient: dump: 2024-09-16T04:04:29.672 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:04:29.672 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:04:29.672 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:29.672 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:29.672 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:04:29.672 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:04:29.672 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:04:29.672 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:04:29.672 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:04:29.672 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:04:29.672 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.658+0000 7fa5c5062640 10 monclient: shutdown 2024-09-16T04:04:29.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.658+0000 7fa5c5062640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:04:29.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.658+0000 7fa5c5062640 1 -- 172.21.15.42:0/385520961 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa5c006dac0 msgr2=0x7fa5c013a340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:04:29.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.658+0000 7fa5c5062640 1 --2- 172.21.15.42:0/385520961 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa5c006dac0 0x7fa5c013a340 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fa5c009c800 tx=0x7fa5b401c430 comp rx=0 tx=0).stop 2024-09-16T04:04:29.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.658+0000 7fa5c5062640 1 -- 172.21.15.42:0/385520961 shutdown_connections 2024-09-16T04:04:29.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.658+0000 7fa5c5062640 1 --2- 172.21.15.42:0/385520961 >> v2:172.21.15.154:3300/0 conn(0x7fa5c00a1e20 0x7fa5c013c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:29.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.658+0000 7fa5c5062640 1 --2- 172.21.15.42:0/385520961 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa5c006dac0 0x7fa5c013a340 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:29.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.658+0000 7fa5c5062640 1 -- 172.21.15.42:0/385520961 >> 172.21.15.42:0/385520961 conn(0x7fa5c00a2420 msgr2=0x7fa5c00a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:04:29.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.658+0000 7fa5c5062640 1 -- 172.21.15.42:0/385520961 shutdown_connections 2024-09-16T04:04:29.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.658+0000 7fa5c5062640 1 -- 172.21.15.42:0/385520961 wait complete. 2024-09-16T04:04:29.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.658+0000 7fa5c5062640 10 monclient: build_initial_monmap 2024-09-16T04:04:29.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.658+0000 7fa5c5062640 10 monclient: monmap: 2024-09-16T04:04:29.673 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:04:29.673 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:04:29.674 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:04:29.674 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:04:29.674 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:04:29.674 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:04:29.674 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:04:29.674 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:04:29.674 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:04:29.674 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:04:29.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.658+0000 7fa5c5062640 1 librados: starting msgr at 2024-09-16T04:04:29.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.658+0000 7fa5c5062640 1 librados: starting objecter 2024-09-16T04:04:29.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.658+0000 7fa5c5062640 1 Processor -- start 2024-09-16T04:04:29.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.658+0000 7fa5c5062640 1 -- start start 2024-09-16T04:04:29.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.658+0000 7fa5c5062640 1 librados: setting wanted keys 2024-09-16T04:04:29.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.658+0000 7fa5c5062640 1 librados: calling monclient init 2024-09-16T04:04:29.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.658+0000 7fa5c5062640 10 monclient: init 2024-09-16T04:04:29.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.658+0000 7fa5c5062640 10 monclient: _reopen_session rank -1 2024-09-16T04:04:29.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.658+0000 7fa5c5062640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:04:29.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.658+0000 7fa5c5062640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa5c00a1330 0x7fa5c01cbb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:04:29.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.658+0000 7fa5c5062640 10 monclient(hunting): picked mon.noname-b con 0x7fa5c00a1330 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:04:29.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.658+0000 7fa5c5062640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa5c00a1e20 0x7fa5c0135770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:04:29.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.658+0000 7fa5c5062640 10 monclient(hunting): picked mon.noname-a con 0x7fa5c00a1e20 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:04:29.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.658+0000 7fa5c5062640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa5c0135cb0 0x7fa5c01cf070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:04:29.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.658+0000 7fa5c5062640 10 monclient(hunting): picked mon.noname-c con 0x7fa5c0135cb0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:04:29.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.658+0000 7fa5c5062640 10 monclient(hunting): start opening mon connection 2024-09-16T04:04:29.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.658+0000 7fa5c5062640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fa5c013f800 con 0x7fa5c00a1e20 2024-09-16T04:04:29.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.658+0000 7fa5c5062640 10 monclient(hunting): start opening mon connection 2024-09-16T04:04:29.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.658+0000 7fa5c5062640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa5c009ef80 con 0x7fa5c00a1330 2024-09-16T04:04:29.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.658+0000 7fa5c5062640 10 monclient(hunting): start opening mon connection 2024-09-16T04:04:29.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.658+0000 7fa5c5062640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fa5c013f980 con 0x7fa5c0135cb0 2024-09-16T04:04:29.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.658+0000 7fa5c5062640 10 monclient(hunting): _renew_subs 2024-09-16T04:04:29.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.658+0000 7fa5c5062640 10 monclient(hunting): authenticate will time out at 2316.631451s 2024-09-16T04:04:29.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.658+0000 7fa5be59c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa5c00a1e20 0x7fa5c0135770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:29.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.658+0000 7fa5be59c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa5c00a1e20 0x7fa5c0135770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:39762/0 (socket says 172.21.15.42:39762) 2024-09-16T04:04:29.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.658+0000 7fa5be59c640 1 -- 172.21.15.42:0/1301032055 learned_addr learned my addr 172.21.15.42:0/1301032055 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:04:29.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.658+0000 7fa5be59c640 10 monclient(hunting): get_auth_request con 0x7fa5c00a1e20 auth_method 0 2024-09-16T04:04:29.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.658+0000 7fa5be59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:04:29.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.658+0000 7fa5be59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:04:29.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.658+0000 7fa5be59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:04:29.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.658+0000 7fa5bf59e640 1 --2- 172.21.15.42:0/1301032055 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa5c0135cb0 0x7fa5c01cf070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:29.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.658+0000 7fa5bf59e640 10 monclient(hunting): get_auth_request con 0x7fa5c0135cb0 auth_method 0 2024-09-16T04:04:29.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.658+0000 7fa5bf59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:04:29.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.658+0000 7fa5bf59e640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:04:29.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.658+0000 7fa5bf59e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:04:29.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.658+0000 7fa5be59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:04:29.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.658+0000 7fa5be59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:04:29.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.658+0000 7fa5be59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:04:29.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.658+0000 7fa5bed9d640 1 --2- 172.21.15.42:0/1301032055 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa5c00a1330 0x7fa5c01cbb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:29.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.658+0000 7fa5bf59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:04:29.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.658+0000 7fa5bf59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:04:29.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.658+0000 7fa5bf59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:04:29.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.658+0000 7fa5bed9d640 10 monclient(hunting): get_auth_request con 0x7fa5c00a1330 auth_method 0 2024-09-16T04:04:29.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.658+0000 7fa5bed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:04:29.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.658+0000 7fa5bed9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:04:29.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.658+0000 7fa5bed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:04:29.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.658+0000 7fa5be59c640 10 monclient(hunting): handle_auth_done global_id 984165 payload 824 2024-09-16T04:04:29.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.662+0000 7fa5be59c640 1 -- 172.21.15.42:0/1301032055 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa5c0135cb0 msgr2=0x7fa5c01cf070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:04:29.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.662+0000 7fa5be59c640 1 --2- 172.21.15.42:0/1301032055 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa5c0135cb0 0x7fa5c01cf070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:29.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.662+0000 7fa5be59c640 1 -- 172.21.15.42:0/1301032055 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa5c00a1330 msgr2=0x7fa5c01cbb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:04:29.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.662+0000 7fa5be59c640 1 --2- 172.21.15.42:0/1301032055 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa5c00a1330 0x7fa5c01cbb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:29.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.662+0000 7fa5be59c640 10 monclient: _finish_hunting 0 2024-09-16T04:04:29.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.662+0000 7fa5be59c640 1 monclient: found mon.noname-a 2024-09-16T04:04:29.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.662+0000 7fa5be59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:04:29.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.662+0000 7fa5be59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:04:29.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.662+0000 7fa5be59c640 1 -- 172.21.15.42:0/1301032055 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5c0122120 con 0x7fa5c00a1e20 2024-09-16T04:04:29.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.662+0000 7fa5bed9d640 1 --2- 172.21.15.42:0/1301032055 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa5c00a1330 0x7fa5c01cbb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T04:04:29.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.662+0000 7fa5be59c640 1 --2- 172.21.15.42:0/1301032055 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa5c00a1e20 0x7fa5c0135770 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fa5b40ae040 tx=0x7fa5b401ac70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:04:29.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.662+0000 7fa5a77fe640 10 client.?.objecter ms_handle_connect 0x7fa5c00a1e20 2024-09-16T04:04:29.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.662+0000 7fa5a77fe640 10 client.?.objecter resend_mon_ops 2024-09-16T04:04:29.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.662+0000 7fa5a77fe640 1 -- 172.21.15.42:0/1301032055 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5b40ba020 con 0x7fa5c00a1e20 2024-09-16T04:04:29.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.662+0000 7fa5a77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:04:29.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.662+0000 7fa5a77fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:04:29.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.662+0000 7fa5a77fe640 10 monclient: dump: 2024-09-16T04:04:29.679 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:04:29.679 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:04:29.679 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:29.679 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:29.679 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:04:29.679 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:04:29.679 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:04:29.679 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:04:29.679 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:04:29.679 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:04:29.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.662+0000 7fa5a77fe640 10 monclient: _finish_auth 0 2024-09-16T04:04:29.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.662+0000 7fa5a77fe640 10 monclient: _check_auth_tickets 2024-09-16T04:04:29.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.662+0000 7fa5a77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:04:29.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.662+0000 7fa5a77fe640 1 -- 172.21.15.42:0/1301032055 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa5b40b6620 con 0x7fa5c00a1e20 2024-09-16T04:04:29.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.662+0000 7fa5c5062640 5 monclient: authenticate success, global_id 984165 2024-09-16T04:04:29.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.662+0000 7fa5c5062640 10 monclient: _renew_subs 2024-09-16T04:04:29.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.662+0000 7fa5c5062640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:04:29.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.662+0000 7fa5c5062640 1 -- 172.21.15.42:0/1301032055 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5c006e160 con 0x7fa5c00a1e20 2024-09-16T04:04:29.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.662+0000 7fa5a77fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:04:29.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.662+0000 7fa5a77fe640 1 -- 172.21.15.42:0/1301032055 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5b40b0040 con 0x7fa5c00a1e20 2024-09-16T04:04:29.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.662+0000 7fa5a77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:04:29.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.662+0000 7fa5a77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:04:29.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.662+0000 7fa5a77fe640 10 monclient: dump: 2024-09-16T04:04:29.680 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:04:29.681 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:04:29.681 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:29.681 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:29.681 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:04:29.681 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:04:29.681 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:04:29.681 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:04:29.681 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:04:29.681 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:04:29.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.662+0000 7fa5c5062640 10 client.984165.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:04:29.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.662+0000 7fa5c5062640 10 monclient: _renew_subs 2024-09-16T04:04:29.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.662+0000 7fa5c5062640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:04:29.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.662+0000 7fa5c5062640 1 -- 172.21.15.42:0/1301032055 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5c0073a30 con 0x7fa5c00a1e20 2024-09-16T04:04:29.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.662+0000 7fa5c5062640 1 librados: init done 2024-09-16T04:04:29.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.662+0000 7fa5bd59a640 4 set_mon_vals no callback set 2024-09-16T04:04:29.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.662+0000 7fa5c5062640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:04:29.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.662+0000 7fa5c5062640 1 --2- 172.21.15.42:0/1301032055 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa5c006dac0 0x7fa5c01cf5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:04:29.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.662+0000 7fa5c5062640 10 monclient(hunting): start opening mon connection 2024-09-16T04:04:29.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.662+0000 7fa5c5062640 1 -- 172.21.15.42:0/1301032055 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fa5c0121ac0 con 0x7fa5c006dac0 2024-09-16T04:04:29.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.662+0000 7fa5a77fe640 1 -- 172.21.15.42:0/1301032055 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fa5b40bd020 con 0x7fa5c00a1e20 2024-09-16T04:04:29.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.662+0000 7fa5bed9d640 1 --2- 172.21.15.42:0/1301032055 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa5c006dac0 0x7fa5c01cf5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:29.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.662+0000 7fa5bed9d640 10 monclient: get_auth_request con 0x7fa5c006dac0 auth_method 0 2024-09-16T04:04:29.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.662+0000 7fa5bed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:04:29.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.662+0000 7fa5bed9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:04:29.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.662+0000 7fa5bed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:04:29.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.662+0000 7fa5bed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:04:29.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.662+0000 7fa5bed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:04:29.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.662+0000 7fa5bed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:04:29.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.662+0000 7fa5a77fe640 1 --2- 172.21.15.42:0/1301032055 >> v2:172.21.15.154:6812/254845342 conn(0x7fa5ac037810 0x7fa5ac039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:04:29.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.662+0000 7fa5a77fe640 1 -- 172.21.15.42:0/1301032055 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(681..681 src has 451..681) ==== 6280+0+0 (secure 0 0 0) 0x7fa5b40e3450 con 0x7fa5c00a1e20 2024-09-16T04:04:29.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.662+0000 7fa5a77fe640 10 client.984165.objecter ms_dispatch 0x7fa5c0000ba0 osd_map(681..681 src has 451..681) 2024-09-16T04:04:29.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.662+0000 7fa5a77fe640 3 client.984165.objecter handle_osd_map got epochs [681,681] > 0 2024-09-16T04:04:29.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.662+0000 7fa5a77fe640 3 client.984165.objecter handle_osd_map decoding full epoch 681 2024-09-16T04:04:29.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.662+0000 7fa5bed9d640 10 monclient(hunting): handle_auth_done global_id 974237 payload 824 2024-09-16T04:04:29.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.662+0000 7fa5bed9d640 1 -- 172.21.15.42:0/1301032055 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa5c01218e0 con 0x7fa5c006dac0 2024-09-16T04:04:29.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.662+0000 7fa5a77fe640 20 client.984165.objecter dump_active .. 0 homeless 2024-09-16T04:04:29.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.662+0000 7fa5bed9d640 1 --2- 172.21.15.42:0/1301032055 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa5c006dac0 0x7fa5c01cf5b0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fa5b000e4e0 tx=0x7fa5b000e9a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:04:29.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.662+0000 7fa5a77fe640 10 client.984165.objecter ms_handle_connect 0x7fa5c006dac0 2024-09-16T04:04:29.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.662+0000 7fa5a77fe640 10 client.984165.objecter resend_mon_ops 2024-09-16T04:04:29.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.662+0000 7fa5bf59e640 1 --2- 172.21.15.42:0/1301032055 >> v2:172.21.15.154:6812/254845342 conn(0x7fa5ac037810 0x7fa5ac039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:29.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.662+0000 7fa5a77fe640 1 -- 172.21.15.42:0/1301032055 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5b00052e0 con 0x7fa5c006dac0 2024-09-16T04:04:29.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.662+0000 7fa5a77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:04:29.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.662+0000 7fa5a77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:04:29.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.662+0000 7fa5a77fe640 10 monclient: dump: 2024-09-16T04:04:29.684 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:04:29.684 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:04:29.684 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:29.684 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:29.684 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:04:29.684 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:04:29.684 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:04:29.684 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:04:29.685 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:04:29.685 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:04:29.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.662+0000 7fa5bf59e640 10 monclient: get_auth_request con 0x7fa5ac037810 auth_method 0 2024-09-16T04:04:29.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.662+0000 7fa5a77fe640 1 -- 172.21.15.42:0/1301032055 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fa5b0018070 con 0x7fa5c006dac0 2024-09-16T04:04:29.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.662+0000 7fa5a77fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:04:29.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.662+0000 7fa5a77fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:04:29.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.662+0000 7fa5a77fe640 1 -- 172.21.15.42:0/1301032055 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa5c006dac0 msgr2=0x7fa5c01cf5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:04:29.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.662+0000 7fa5a77fe640 1 --2- 172.21.15.42:0/1301032055 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa5c006dac0 0x7fa5c01cf5b0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fa5b000e4e0 tx=0x7fa5b000e9a0 comp rx=0 tx=0).stop 2024-09-16T04:04:29.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.662+0000 7fa5a77fe640 1 -- 172.21.15.42:0/1301032055 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa5c006dac0 msgr2=0x7fa5c01cf5b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:04:29.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.662+0000 7fa5a77fe640 1 --2- 172.21.15.42:0/1301032055 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa5c006dac0 0x7fa5c01cf5b0 secure :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fa5b000e4e0 tx=0x7fa5b000e9a0 comp rx=0 tx=0).stop 2024-09-16T04:04:29.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.662+0000 7fa5bf59e640 1 --2- 172.21.15.42:0/1301032055 >> v2:172.21.15.154:6812/254845342 conn(0x7fa5ac037810 0x7fa5ac039cd0 secure :-1 s=READY pgs=3827 cs=0 l=1 rev1=1 crypto rx=0x7fa5c01360e0 tx=0x7fa5b8009820 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:04:29.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.662+0000 7fa5a77fe640 10 client.984165.objecter ms_handle_connect 0x7fa5ac037810 2024-09-16T04:04:29.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.670+0000 7fa5c5062640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:04:29.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.670+0000 7fa5c5062640 1 --2- 172.21.15.42:0/1301032055 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa5c0121d00 0x7fa5c01cfaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:04:29.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.670+0000 7fa5c5062640 10 monclient(hunting): start opening mon connection 2024-09-16T04:04:29.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.670+0000 7fa5c5062640 1 -- 172.21.15.42:0/1301032055 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fa5c009e7e0 con 0x7fa5c0121d00 2024-09-16T04:04:29.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.670+0000 7fa5bed9d640 1 --2- 172.21.15.42:0/1301032055 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa5c0121d00 0x7fa5c01cfaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:29.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.670+0000 7fa5bed9d640 10 monclient: get_auth_request con 0x7fa5c0121d00 auth_method 0 2024-09-16T04:04:29.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.670+0000 7fa5bed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:04:29.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.670+0000 7fa5bed9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:04:29.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.670+0000 7fa5bed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:04:29.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.670+0000 7fa5bed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:04:29.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.670+0000 7fa5bed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:04:29.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.670+0000 7fa5bed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:04:29.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.670+0000 7fa5bed9d640 10 monclient(hunting): handle_auth_done global_id 974240 payload 824 2024-09-16T04:04:29.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.670+0000 7fa5bed9d640 1 -- 172.21.15.42:0/1301032055 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fa5c00a1710 con 0x7fa5c0121d00 2024-09-16T04:04:29.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.670+0000 7fa5a77fe640 10 client.984165.objecter ms_handle_connect 0x7fa5c0121d00 2024-09-16T04:04:29.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.670+0000 7fa5a77fe640 10 client.984165.objecter resend_mon_ops 2024-09-16T04:04:29.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.670+0000 7fa5bed9d640 1 --2- 172.21.15.42:0/1301032055 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa5c0121d00 0x7fa5c01cfaf0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fa5b000e4e0 tx=0x7fa5b0001030 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:04:29.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.670+0000 7fa5a77fe640 1 -- 172.21.15.42:0/1301032055 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5b0003ad0 con 0x7fa5c0121d00 2024-09-16T04:04:29.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.670+0000 7fa5a77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:04:29.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.670+0000 7fa5a77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:04:29.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.670+0000 7fa5a77fe640 10 monclient: dump: 2024-09-16T04:04:29.688 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:04:29.689 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:04:29.689 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:29.689 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:29.689 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:04:29.689 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:04:29.689 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:04:29.689 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:04:29.689 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:04:29.689 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:04:29.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.670+0000 7fa5a77fe640 1 -- 172.21.15.42:0/1301032055 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4545 (secure 0 0 0) 0x7fa5b0003ad0 con 0x7fa5c0121d00 2024-09-16T04:04:29.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.670+0000 7fa5a77fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:04:29.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.670+0000 7fa5a77fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:04:29.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.670+0000 7fa5a77fe640 1 -- 172.21.15.42:0/1301032055 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa5c0121d00 msgr2=0x7fa5c01cfaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:04:29.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.670+0000 7fa5a77fe640 1 --2- 172.21.15.42:0/1301032055 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa5c0121d00 0x7fa5c01cfaf0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fa5b000e4e0 tx=0x7fa5b0001030 comp rx=0 tx=0).stop 2024-09-16T04:04:29.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.670+0000 7fa5a77fe640 1 -- 172.21.15.42:0/1301032055 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa5c0121d00 msgr2=0x7fa5c01cfaf0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:04:29.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.670+0000 7fa5a77fe640 1 --2- 172.21.15.42:0/1301032055 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa5c0121d00 0x7fa5c01cfaf0 secure :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fa5b000e4e0 tx=0x7fa5b0001030 comp rx=0 tx=0).stop 2024-09-16T04:04:29.690 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T04:04:29.690 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:04:29.690 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:04:29.690 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T04:04:29.690 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 210, 2024-09-16T04:04:29.690 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 78171, 2024-09-16T04:04:29.690 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T04:04:29.691 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T04:04:29.691 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T04:04:29.691 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T04:04:29.691 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:04:29.691 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 4, 2024-09-16T04:04:29.691 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:04:29.691 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T04:04:29.691 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T04:04:29.691 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:04:29.691 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:04:29.691 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:04:29.691 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:04:29.691 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:04:29.692 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:04:29.692 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:04:29.692 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:04:29.692 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:04:29.692 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:04:29.692 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:04:29.692 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:04:29.692 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:04:29.692 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T04:04:29.692 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:04:29.692 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:04:29.692 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:04:29.692 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:04:29.692 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:04:29.693 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:04:29.693 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:04:29.693 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:04:29.693 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:04:29.693 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:04:29.693 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:04:29.693 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:04:29.693 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:04:29.693 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T04:04:29.693 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T04:04:29.693 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T04:04:29.693 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T04:04:29.693 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T04:04:29.694 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T04:04:29.694 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:04:29.694 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:04:29.694 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T04:04:29.694 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T04:04:29.694 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T04:04:29.694 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T04:04:29.694 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T04:04:29.694 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T04:04:29.694 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T04:04:29.694 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:04:29.694 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T04:04:29.694 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:04:29.694 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:04:29.695 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:04:29.695 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:04:29.695 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:04:29.695 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:04:29.695 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:04:29.695 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:04:29.695 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:04:29.695 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:04:29.695 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:04:29.695 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:04:29.695 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T04:04:29.695 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:04:29.695 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T04:04:29.695 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:04:29.695 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:04:29.696 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:04:29.696 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:04:29.696 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:04:29.696 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:04:29.696 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:04:29.696 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T04:04:29.696 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:04:29.696 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:04:29.696 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:04:29.696 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:04:29.696 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T04:04:29.696 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:04:29.696 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:04:29.697 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:04:29.697 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:04:29.697 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T04:04:29.697 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T04:04:29.697 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:04:29.697 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:04:29.697 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:04:29.697 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:04:29.697 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:04:29.697 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:04:29.697 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:04:29.697 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:04:29.697 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:04:29.698 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:04:29.698 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:04:29.698 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T04:04:29.698 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:04:29.698 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:04:29.698 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:04:29.698 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:04:29.698 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T04:04:29.698 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:04:29.698 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:04:29.698 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:04:29.698 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:04:29.699 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T04:04:29.699 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T04:04:29.699 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:04:29.699 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:04:29.699 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:04:29.699 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:04:29.699 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:04:29.699 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:04:29.699 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:04:29.699 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:04:29.699 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:04:29.699 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:04:29.699 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:04:29.700 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T04:04:29.700 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:04:29.700 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:04:29.700 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:04:29.700 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:04:29.700 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T04:04:29.700 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:04:29.700 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:04:29.700 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:04:29.700 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:04:29.700 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T04:04:29.700 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T04:04:29.700 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:04:29.700 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:04:29.701 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:04:29.701 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:04:29.701 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:04:29.701 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:04:29.701 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T04:04:29.701 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T04:04:29.701 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:04:29.701 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:04:29.701 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:04:29.701 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:04:29.701 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:04:29.701 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:04:29.701 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T04:04:29.702 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:04:29.702 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:04:29.702 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:04:29.702 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:04:29.702 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:04:29.702 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:04:29.702 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:04:29.702 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T04:04:29.702 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T04:04:29.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.670+0000 7fa5c5062640 10 librados: watch_flush enter 2024-09-16T04:04:29.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.670+0000 7fa5c5062640 10 librados: watch_flush exit 2024-09-16T04:04:29.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.670+0000 7fa5c5062640 20 client.984165.objecter shutdown clearing up homeless session... 2024-09-16T04:04:29.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.670+0000 7fa5c5062640 10 client.984165.objecter successfully canceled tick 2024-09-16T04:04:29.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.670+0000 7fa5c5062640 1 -- 172.21.15.42:0/1301032055 >> v2:172.21.15.154:6812/254845342 conn(0x7fa5ac037810 msgr2=0x7fa5ac039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:04:29.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.670+0000 7fa5c5062640 1 --2- 172.21.15.42:0/1301032055 >> v2:172.21.15.154:6812/254845342 conn(0x7fa5ac037810 0x7fa5ac039cd0 secure :-1 s=READY pgs=3827 cs=0 l=1 rev1=1 crypto rx=0x7fa5c01360e0 tx=0x7fa5b8009820 comp rx=0 tx=0).stop 2024-09-16T04:04:29.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.670+0000 7fa5c5062640 10 monclient: shutdown 2024-09-16T04:04:29.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.670+0000 7fa5c5062640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:04:29.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.670+0000 7fa5c5062640 1 -- 172.21.15.42:0/1301032055 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa5c00a1e20 msgr2=0x7fa5c0135770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:04:29.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.670+0000 7fa5c5062640 1 --2- 172.21.15.42:0/1301032055 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa5c00a1e20 0x7fa5c0135770 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fa5b40ae040 tx=0x7fa5b401ac70 comp rx=0 tx=0).stop 2024-09-16T04:04:29.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.670+0000 7fa5bf59e640 1 -- 172.21.15.42:0/1301032055 reap_dead start 2024-09-16T04:04:29.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.670+0000 7fa5bf59e640 1 -- 172.21.15.42:0/1301032055 reap_dead start 2024-09-16T04:04:29.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.670+0000 7fa5c5062640 1 -- 172.21.15.42:0/1301032055 shutdown_connections 2024-09-16T04:04:29.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.670+0000 7fa5c5062640 1 -- 172.21.15.42:0/1301032055 >> 172.21.15.42:0/1301032055 conn(0x7fa5c00a2420 msgr2=0x7fa5c00a45b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:04:29.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.670+0000 7fa5c5062640 1 -- 172.21.15.42:0/1301032055 shutdown_connections 2024-09-16T04:04:29.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.670+0000 7fa5c5062640 1 -- 172.21.15.42:0/1301032055 wait complete. 2024-09-16T04:04:29.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.670+0000 7fa5c5062640 1 librados: shutdown 2024-09-16T04:04:29.705 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:04:29.705 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T04:04:29.705 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T04:04:29.825 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.822+0000 7fd1b9ee0640 10 monclient: get_monmap_and_config 2024-09-16T04:04:29.826 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.822+0000 7fd1b9ee0640 10 monclient: build_initial_monmap 2024-09-16T04:04:29.826 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.822+0000 7fd1b9ee0640 10 monclient: monmap: 2024-09-16T04:04:29.826 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:04:29.826 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:04:29.826 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:04:29.824450+0000 2024-09-16T04:04:29.826 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:04:29.824450+0000 2024-09-16T04:04:29.826 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:04:29.826 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:04:29.826 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:04:29.826 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:04:29.826 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:04:29.827 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:04:29.827 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:04:29.827 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:04:29.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.822+0000 7fd1b9ee0640 1 Processor -- start 2024-09-16T04:04:29.832 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.822+0000 7fd1b9ee0640 1 -- start start 2024-09-16T04:04:29.832 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.822+0000 7fd1b9ee0640 10 monclient: init 2024-09-16T04:04:29.832 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.822+0000 7fd1b9ee0640 10 monclient: _reopen_session rank -1 2024-09-16T04:04:29.832 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.822+0000 7fd1b9ee0640 10 monclient: _add_conns ranks=[1,4,0,3,2] 2024-09-16T04:04:29.832 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.822+0000 7fd1b9ee0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd1b40a2230 addr v1:172.21.15.154:6789/0 2024-09-16T04:04:29.832 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.822+0000 7fd1b9ee0640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fd1b4072740 0x7fd1b413a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:04:29.832 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.822+0000 7fd1b9ee0640 10 monclient(hunting): picked mon.noname-c con 0x7fd1b4072740 addr v2:172.21.15.154:3300/0 2024-09-16T04:04:29.832 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.822+0000 7fd1b9ee0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd1b40a2d20 addr v1:172.21.15.42:6789/0 2024-09-16T04:04:29.832 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.822+0000 7fd1b9ee0640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd1b406c050 con 0x7fd1b40a2d20 2024-09-16T04:04:29.832 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.822+0000 7fd1b9ee0640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd1b406c3b0 con 0x7fd1b40a2230 2024-09-16T04:04:29.832 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.822+0000 7fd1b9ee0640 10 monclient(hunting): start opening mon connection 2024-09-16T04:04:29.832 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.822+0000 7fd1b9ee0640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fd1b4071b10 con 0x7fd1b4072740 2024-09-16T04:04:29.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.822+0000 7fd1b9ee0640 10 monclient(hunting): _renew_subs 2024-09-16T04:04:29.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.822+0000 7fd1b9ee0640 10 monclient(hunting): authenticate will time out at 2316.794023s 2024-09-16T04:04:29.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.822+0000 7fd1b3fff640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fd1b40a2d20 0x7fd1b413ca00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:39744/0 (socket says 172.21.15.42:39744) 2024-09-16T04:04:29.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.822+0000 7fd1b3fff640 1 -- 172.21.15.42:0/3922824459 learned_addr learned my addr 172.21.15.42:0/3922824459 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T04:04:29.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.822+0000 7fd1b2ffd640 1 --2- 172.21.15.42:0/3922824459 >> v2:172.21.15.154:3300/0 conn(0x7fd1b4072740 0x7fd1b413a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:29.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.822+0000 7fd1b2ffd640 10 monclient(hunting): get_auth_request con 0x7fd1b4072740 auth_method 0 2024-09-16T04:04:29.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.822+0000 7fd1b2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:04:29.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.822+0000 7fd1b2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:04:29.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.822+0000 7fd1b2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:04:29.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.822+0000 7fd1b27fc640 1 -- 172.21.15.42:0/3922824459 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 39987554 0 0) 0x7fd1b406c050 con 0x7fd1b40a2d20 2024-09-16T04:04:29.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.822+0000 7fd1b27fc640 10 monclient(hunting): _negotiate 2024-09-16T04:04:29.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.822+0000 7fd1b27fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:04:29.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.822+0000 7fd1b27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:04:29.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.822+0000 7fd1b27fc640 10 monclient(hunting): my global_id is 984168 2024-09-16T04:04:29.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.822+0000 7fd1b27fc640 1 -- 172.21.15.42:0/3922824459 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd19c003530 con 0x7fd1b40a2d20 2024-09-16T04:04:29.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.822+0000 7fd1b2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:04:29.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.822+0000 7fd1b2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:04:29.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.822+0000 7fd1b2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:04:29.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.822+0000 7fd1b27fc640 1 -- 172.21.15.42:0/3922824459 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1738427072 0 0) 0x7fd19c003530 con 0x7fd1b40a2d20 2024-09-16T04:04:29.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.822+0000 7fd1b27fc640 1 -- 172.21.15.42:0/3922824459 >> v2:172.21.15.154:3300/0 conn(0x7fd1b4072740 msgr2=0x7fd1b413a4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:04:29.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.822+0000 7fd1b27fc640 1 --2- 172.21.15.42:0/3922824459 >> v2:172.21.15.154:3300/0 conn(0x7fd1b4072740 0x7fd1b413a4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:29.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.822+0000 7fd1b27fc640 1 -- 172.21.15.42:0/3922824459 >> v1:172.21.15.154:6789/0 conn(0x7fd1b40a2230 legacy=0x7fd1b40a2610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:04:29.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.822+0000 7fd1b27fc640 10 monclient: _finish_hunting 0 2024-09-16T04:04:29.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.822+0000 7fd1b27fc640 1 monclient: found mon.noname-a-legacy 2024-09-16T04:04:29.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.822+0000 7fd1b27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:04:29.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.822+0000 7fd1b27fc640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.42:6789/0 2024-09-16T04:04:29.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.822+0000 7fd1b27fc640 1 -- 172.21.15.42:0/3922824459 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1b40a3c90 con 0x7fd1b40a2d20 2024-09-16T04:04:29.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.822+0000 7fd1b27fc640 10 monclient: _finish_auth 0 2024-09-16T04:04:29.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.822+0000 7fd1b27fc640 10 monclient: _check_auth_tickets 2024-09-16T04:04:29.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.822+0000 7fd1b27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:04:29.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.822+0000 7fd1b27fc640 1 -- 172.21.15.42:0/3922824459 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2177413734 0 0) 0x7fd1a4001680 con 0x7fd1b40a2d20 2024-09-16T04:04:29.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.822+0000 7fd1b27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:04:29.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.822+0000 7fd1b27fc640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T04:04:29.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.822+0000 7fd1b27fc640 10 monclient: dump: 2024-09-16T04:04:29.835 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:04:29.835 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:04:29.835 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:29.835 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:29.835 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:04:29.835 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:04:29.836 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:04:29.836 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:04:29.836 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:04:29.836 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:04:29.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.822+0000 7fd1b27fc640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] but i'm connected to v1:172.21.15.42:6789/0, reconnecting 2024-09-16T04:04:29.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.822+0000 7fd1b27fc640 10 monclient: _reopen_session rank -1 2024-09-16T04:04:29.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.822+0000 7fd1b27fc640 1 -- 172.21.15.42:0/3922824459 >> v1:172.21.15.42:6789/0 conn(0x7fd1b40a2d20 legacy=0x7fd1b413ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:04:29.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.822+0000 7fd1b27fc640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:04:29.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.822+0000 7fd1b27fc640 1 --2- 172.21.15.42:0/3922824459 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd19c0039c0 0x7fd19c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:04:29.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.822+0000 7fd1b27fc640 10 monclient(hunting): picked mon.b con 0x7fd19c0039c0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:04:29.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.822+0000 7fd1b27fc640 1 --2- 172.21.15.42:0/3922824459 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd19c004810 0x7fd19c004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:04:29.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.822+0000 7fd1b27fc640 10 monclient(hunting): picked mon.c con 0x7fd19c004810 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:04:29.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.822+0000 7fd1b27fc640 1 --2- 172.21.15.42:0/3922824459 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd19c0052f0 0x7fd19c00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:04:29.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.822+0000 7fd1b27fc640 10 monclient(hunting): picked mon.a con 0x7fd19c0052f0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:04:29.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.822+0000 7fd1b27fc640 10 monclient(hunting): start opening mon connection 2024-09-16T04:04:29.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.822+0000 7fd1b27fc640 1 -- 172.21.15.42:0/3922824459 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fd1b4071b10 con 0x7fd19c0052f0 2024-09-16T04:04:29.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.822+0000 7fd1b27fc640 10 monclient(hunting): start opening mon connection 2024-09-16T04:04:29.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.822+0000 7fd1b27fc640 1 -- 172.21.15.42:0/3922824459 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd19c00c1c0 con 0x7fd19c0039c0 2024-09-16T04:04:29.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.822+0000 7fd1b27fc640 10 monclient(hunting): start opening mon connection 2024-09-16T04:04:29.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.822+0000 7fd1b27fc640 1 -- 172.21.15.42:0/3922824459 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fd19c00c3c0 con 0x7fd19c004810 2024-09-16T04:04:29.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.822+0000 7fd1b27fc640 10 monclient(hunting): _renew_subs 2024-09-16T04:04:29.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.822+0000 7fd1b27fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:04:29.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.822+0000 7fd1b3fff640 1 --2- 172.21.15.42:0/3922824459 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd19c0052f0 0x7fd19c00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:29.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.822+0000 7fd1b2ffd640 1 --2- 172.21.15.42:0/3922824459 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd19c004810 0x7fd19c004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:29.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.822+0000 7fd1b3fff640 10 monclient(hunting): get_auth_request con 0x7fd19c0052f0 auth_method 0 2024-09-16T04:04:29.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.822+0000 7fd1b3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:04:29.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.822+0000 7fd1b3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:04:29.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.822+0000 7fd1b3fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:04:29.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.822+0000 7fd1b2ffd640 10 monclient(hunting): get_auth_request con 0x7fd19c004810 auth_method 0 2024-09-16T04:04:29.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.822+0000 7fd1b2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:04:29.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.822+0000 7fd1b2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:04:29.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.822+0000 7fd1b2ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:04:29.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.822+0000 7fd1b3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:04:29.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.822+0000 7fd1b3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:04:29.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.822+0000 7fd1b3fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:04:29.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.822+0000 7fd1b2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:04:29.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.822+0000 7fd1b2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:04:29.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.822+0000 7fd1b2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:04:29.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.822+0000 7fd1b37fe640 1 --2- 172.21.15.42:0/3922824459 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd19c0039c0 0x7fd19c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:29.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.822+0000 7fd1b37fe640 10 monclient(hunting): get_auth_request con 0x7fd19c0039c0 auth_method 0 2024-09-16T04:04:29.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.822+0000 7fd1b37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:04:29.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.822+0000 7fd1b37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:04:29.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.822+0000 7fd1b37fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:04:29.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.822+0000 7fd1b3fff640 10 monclient(hunting): handle_auth_done global_id 984168 payload 293 2024-09-16T04:04:29.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.822+0000 7fd1b3fff640 1 -- 172.21.15.42:0/3922824459 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd19c004810 msgr2=0x7fd19c004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:04:29.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.822+0000 7fd1b3fff640 1 --2- 172.21.15.42:0/3922824459 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd19c004810 0x7fd19c004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:29.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.822+0000 7fd1b3fff640 1 -- 172.21.15.42:0/3922824459 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd19c0039c0 msgr2=0x7fd19c003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:04:29.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.822+0000 7fd1b3fff640 1 --2- 172.21.15.42:0/3922824459 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd19c0039c0 0x7fd19c003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:29.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.822+0000 7fd1b3fff640 10 monclient: _finish_hunting 0 2024-09-16T04:04:29.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.822+0000 7fd1b3fff640 1 monclient: found mon.a 2024-09-16T04:04:29.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.822+0000 7fd1b3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:04:29.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.822+0000 7fd1b3fff640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:04:29.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.822+0000 7fd1b3fff640 1 -- 172.21.15.42:0/3922824459 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd19c003530 con 0x7fd19c0052f0 2024-09-16T04:04:29.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.822+0000 7fd1b3fff640 10 monclient: _finish_auth 0 2024-09-16T04:04:29.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.822+0000 7fd1b3fff640 10 monclient: _check_auth_tickets 2024-09-16T04:04:29.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.822+0000 7fd1b3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:04:29.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.822+0000 7fd1b9ee0640 5 monclient: authenticate success, global_id 984168 2024-09-16T04:04:29.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.822+0000 7fd1b9ee0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:04:29.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.822+0000 7fd1b2ffd640 1 --2- 172.21.15.42:0/3922824459 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd19c004810 0x7fd19c004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:04:29.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.826+0000 7fd1b3fff640 1 -- 172.21.15.42:0/3922824459 reap_dead start 2024-09-16T04:04:29.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.826+0000 7fd1b3fff640 1 --2- 172.21.15.42:0/3922824459 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd19c0052f0 0x7fd19c00b9d0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fd1a4001ec0 tx=0x7fd1a401cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94881e5496e065e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:04:29.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.826+0000 7fd1b27fc640 1 -- 172.21.15.42:0/3922824459 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1a8009c10 con 0x7fd19c0052f0 2024-09-16T04:04:29.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.826+0000 7fd1b27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:04:29.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.826+0000 7fd1b27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:04:29.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.826+0000 7fd1b27fc640 10 monclient: dump: 2024-09-16T04:04:29.841 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:04:29.841 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:04:29.841 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:29.841 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:29.841 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:04:29.841 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:04:29.841 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:04:29.841 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:04:29.841 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:04:29.841 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:04:29.841 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.826+0000 7fd1b27fc640 1 -- 172.21.15.42:0/3922824459 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd1a40a9070 con 0x7fd19c0052f0 2024-09-16T04:04:29.841 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.826+0000 7fd1b27fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:04:29.841 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.826+0000 7fd1b27fc640 1 -- 172.21.15.42:0/3922824459 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1a4007d40 con 0x7fd19c0052f0 2024-09-16T04:04:29.841 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.826+0000 7fd1b9ee0640 10 monclient: get_monmap_and_config success 2024-09-16T04:04:29.842 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.826+0000 7fd1b9ee0640 4 set_mon_vals no callback set 2024-09-16T04:04:29.842 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.826+0000 7fd1b27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:04:29.842 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.826+0000 7fd1b27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:04:29.842 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.826+0000 7fd1b27fc640 10 monclient: dump: 2024-09-16T04:04:29.842 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:04:29.842 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:04:29.842 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:29.842 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:29.842 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:04:29.842 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:04:29.842 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:04:29.842 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:04:29.842 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:04:29.842 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:04:29.842 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.826+0000 7fd1b9ee0640 10 monclient: shutdown 2024-09-16T04:04:29.843 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.826+0000 7fd1b9ee0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:04:29.843 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.826+0000 7fd1b9ee0640 1 -- 172.21.15.42:0/3922824459 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd19c0052f0 msgr2=0x7fd19c00b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:04:29.843 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.826+0000 7fd1b9ee0640 1 --2- 172.21.15.42:0/3922824459 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd19c0052f0 0x7fd19c00b9d0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fd1a4001ec0 tx=0x7fd1a401cc40 comp rx=0 tx=0).stop 2024-09-16T04:04:29.843 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.826+0000 7fd1b9ee0640 1 -- 172.21.15.42:0/3922824459 shutdown_connections 2024-09-16T04:04:29.843 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.826+0000 7fd1b9ee0640 1 --2- 172.21.15.42:0/3922824459 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd19c0052f0 0x7fd19c00b9d0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:29.843 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.826+0000 7fd1b9ee0640 1 -- 172.21.15.42:0/3922824459 >> 172.21.15.42:0/3922824459 conn(0x7fd1b40a6e80 msgr2=0x7fd1b40a3320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:04:29.843 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.826+0000 7fd1b9ee0640 1 -- 172.21.15.42:0/3922824459 shutdown_connections 2024-09-16T04:04:29.843 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.826+0000 7fd1b9ee0640 1 -- 172.21.15.42:0/3922824459 wait complete. 2024-09-16T04:04:29.849 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.826+0000 7fd1b9ee0640 10 monclient: build_initial_monmap 2024-09-16T04:04:29.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.826+0000 7fd1b9ee0640 10 monclient: monmap: 2024-09-16T04:04:29.850 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:04:29.850 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:04:29.850 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:04:29.850 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:04:29.850 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:04:29.850 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:04:29.850 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:04:29.850 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:04:29.850 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:04:29.850 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:04:29.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.826+0000 7fd1b9ee0640 1 librados: starting msgr at 2024-09-16T04:04:29.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.826+0000 7fd1b9ee0640 1 librados: starting objecter 2024-09-16T04:04:29.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.826+0000 7fd1b9ee0640 1 Processor -- start 2024-09-16T04:04:29.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.826+0000 7fd1b9ee0640 1 -- start start 2024-09-16T04:04:29.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.826+0000 7fd1b9ee0640 1 librados: setting wanted keys 2024-09-16T04:04:29.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.826+0000 7fd1b9ee0640 1 librados: calling monclient init 2024-09-16T04:04:29.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.826+0000 7fd1b9ee0640 10 monclient: init 2024-09-16T04:04:29.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.830+0000 7fd1b9ee0640 10 monclient: _reopen_session rank -1 2024-09-16T04:04:29.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.830+0000 7fd1b9ee0640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:04:29.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.830+0000 7fd1b9ee0640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd19c0052f0 0x7fd1b41ccab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:04:29.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.830+0000 7fd1b9ee0640 10 monclient(hunting): picked mon.noname-c con 0x7fd19c0052f0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:04:29.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.830+0000 7fd1b9ee0640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd1b41344f0 0x7fd1b4134970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:04:29.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.830+0000 7fd1b9ee0640 10 monclient(hunting): picked mon.noname-a con 0x7fd1b41344f0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:04:29.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.830+0000 7fd1b9ee0640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd1b413a630 0x7fd1b413aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:04:29.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.830+0000 7fd1b9ee0640 10 monclient(hunting): picked mon.noname-b con 0x7fd1b413a630 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:04:29.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.830+0000 7fd1b9ee0640 10 monclient(hunting): start opening mon connection 2024-09-16T04:04:29.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.830+0000 7fd1b9ee0640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fd1b40a29a0 con 0x7fd1b41344f0 2024-09-16T04:04:29.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.830+0000 7fd1b9ee0640 10 monclient(hunting): start opening mon connection 2024-09-16T04:04:29.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.830+0000 7fd1b9ee0640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd1b40a2820 con 0x7fd1b413a630 2024-09-16T04:04:29.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.830+0000 7fd1b9ee0640 10 monclient(hunting): start opening mon connection 2024-09-16T04:04:29.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.830+0000 7fd1b9ee0640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fd1b40a2b20 con 0x7fd19c0052f0 2024-09-16T04:04:29.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.830+0000 7fd1b9ee0640 10 monclient(hunting): _renew_subs 2024-09-16T04:04:29.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.830+0000 7fd1b9ee0640 10 monclient(hunting): authenticate will time out at 2316.800199s 2024-09-16T04:04:29.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.830+0000 7fd1b37fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd19c0052f0 0x7fd1b41ccab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:29.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.830+0000 7fd1b2ffd640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd1b41344f0 0x7fd1b4134970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:29.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.830+0000 7fd1b37fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd19c0052f0 0x7fd1b41ccab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:52550/0 (socket says 172.21.15.42:52550) 2024-09-16T04:04:29.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.830+0000 7fd1b2ffd640 10 monclient(hunting): get_auth_request con 0x7fd1b41344f0 auth_method 0 2024-09-16T04:04:29.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.830+0000 7fd1b2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:04:29.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.830+0000 7fd1b2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:04:29.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.830+0000 7fd1b2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:04:29.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.830+0000 7fd1b37fe640 1 -- 172.21.15.42:0/2078795648 learned_addr learned my addr 172.21.15.42:0/2078795648 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:04:29.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.830+0000 7fd1b37fe640 10 monclient(hunting): get_auth_request con 0x7fd19c0052f0 auth_method 0 2024-09-16T04:04:29.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.830+0000 7fd1b37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:04:29.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.830+0000 7fd1b37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:04:29.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.830+0000 7fd1b37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:04:29.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.830+0000 7fd1b2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:04:29.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.830+0000 7fd1b2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:04:29.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.830+0000 7fd1b2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:04:29.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.830+0000 7fd1b37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:04:29.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.830+0000 7fd1b37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:04:29.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.830+0000 7fd1b37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:04:29.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.830+0000 7fd1b3fff640 1 --2- 172.21.15.42:0/2078795648 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd1b413a630 0x7fd1b413aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:29.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.830+0000 7fd1b3fff640 10 monclient(hunting): get_auth_request con 0x7fd1b413a630 auth_method 0 2024-09-16T04:04:29.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.830+0000 7fd1b3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:04:29.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.830+0000 7fd1b3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:04:29.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.830+0000 7fd1b3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:04:29.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.830+0000 7fd1b37fe640 10 monclient(hunting): handle_auth_done global_id 974243 payload 824 2024-09-16T04:04:29.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.830+0000 7fd1b37fe640 1 -- 172.21.15.42:0/2078795648 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd1b413a630 msgr2=0x7fd1b413aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:04:29.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.830+0000 7fd1b37fe640 1 --2- 172.21.15.42:0/2078795648 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd1b413a630 0x7fd1b413aa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:29.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.830+0000 7fd1b37fe640 1 -- 172.21.15.42:0/2078795648 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd1b41344f0 msgr2=0x7fd1b4134970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:04:29.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.830+0000 7fd1b37fe640 1 --2- 172.21.15.42:0/2078795648 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd1b41344f0 0x7fd1b4134970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:29.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.830+0000 7fd1b37fe640 10 monclient: _finish_hunting 0 2024-09-16T04:04:29.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.830+0000 7fd1b37fe640 1 monclient: found mon.noname-c 2024-09-16T04:04:29.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.830+0000 7fd1b37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:04:29.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.830+0000 7fd1b37fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T04:04:29.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.830+0000 7fd1b37fe640 1 -- 172.21.15.42:0/2078795648 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1b411de70 con 0x7fd19c0052f0 2024-09-16T04:04:29.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.830+0000 7fd1b2ffd640 1 --2- 172.21.15.42:0/2078795648 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd1b41344f0 0x7fd1b4134970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:04:29.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.830+0000 7fd1b37fe640 1 --2- 172.21.15.42:0/2078795648 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd19c0052f0 0x7fd1b41ccab0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fd1ac00e500 tx=0x7fd1ac00e9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:04:29.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.830+0000 7fd19bfff640 10 client.?.objecter ms_handle_connect 0x7fd19c0052f0 2024-09-16T04:04:29.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.830+0000 7fd19bfff640 10 client.?.objecter resend_mon_ops 2024-09-16T04:04:29.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.830+0000 7fd19bfff640 1 -- 172.21.15.42:0/2078795648 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1ac0052e0 con 0x7fd19c0052f0 2024-09-16T04:04:29.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.830+0000 7fd19bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:04:29.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.830+0000 7fd19bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:04:29.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.830+0000 7fd19bfff640 10 monclient: dump: 2024-09-16T04:04:29.855 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:04:29.855 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:04:29.855 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:29.855 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:29.855 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:04:29.856 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:04:29.856 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:04:29.856 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:04:29.856 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:04:29.856 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:04:29.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.830+0000 7fd19bfff640 10 monclient: _finish_auth 0 2024-09-16T04:04:29.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.830+0000 7fd19bfff640 10 monclient: _check_auth_tickets 2024-09-16T04:04:29.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.830+0000 7fd19bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:04:29.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.830+0000 7fd1b9ee0640 5 monclient: authenticate success, global_id 974243 2024-09-16T04:04:29.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.830+0000 7fd19bfff640 1 -- 172.21.15.42:0/2078795648 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd1ac018070 con 0x7fd19c0052f0 2024-09-16T04:04:29.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.830+0000 7fd1b9ee0640 10 monclient: _renew_subs 2024-09-16T04:04:29.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.830+0000 7fd1b9ee0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:04:29.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.830+0000 7fd1b9ee0640 1 -- 172.21.15.42:0/2078795648 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1b40a60e0 con 0x7fd19c0052f0 2024-09-16T04:04:29.857 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.830+0000 7fd19bfff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:04:29.857 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.830+0000 7fd19bfff640 1 -- 172.21.15.42:0/2078795648 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1ac003820 con 0x7fd19c0052f0 2024-09-16T04:04:29.857 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.830+0000 7fd19bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:04:29.857 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.830+0000 7fd19bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:04:29.857 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.830+0000 7fd1b1ffb640 4 set_mon_vals no callback set 2024-09-16T04:04:29.857 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.830+0000 7fd19bfff640 10 monclient: dump: 2024-09-16T04:04:29.857 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:04:29.857 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:04:29.857 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:29.857 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:29.857 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:04:29.857 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:04:29.857 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:04:29.857 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:04:29.858 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:04:29.858 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:04:29.858 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.830+0000 7fd1b9ee0640 10 client.974243.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:04:29.858 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.830+0000 7fd1b9ee0640 10 monclient: _renew_subs 2024-09-16T04:04:29.858 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.830+0000 7fd1b9ee0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:04:29.858 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.830+0000 7fd1b9ee0640 1 -- 172.21.15.42:0/2078795648 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1b40738a0 con 0x7fd19c0052f0 2024-09-16T04:04:29.858 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.830+0000 7fd1b9ee0640 1 librados: init done 2024-09-16T04:04:29.858 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.830+0000 7fd1b9ee0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:04:29.858 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.830+0000 7fd1b9ee0640 1 --2- 172.21.15.42:0/2078795648 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd1b413afd0 0x7fd1b4136490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:04:29.858 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.830+0000 7fd1b9ee0640 10 monclient(hunting): start opening mon connection 2024-09-16T04:04:29.858 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.830+0000 7fd1b9ee0640 1 -- 172.21.15.42:0/2078795648 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd1b4134ff0 con 0x7fd1b413afd0 2024-09-16T04:04:29.858 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.830+0000 7fd19bfff640 1 -- 172.21.15.42:0/2078795648 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fd1ac0039c0 con 0x7fd19c0052f0 2024-09-16T04:04:29.858 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.830+0000 7fd1b2ffd640 1 --2- 172.21.15.42:0/2078795648 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd1b413afd0 0x7fd1b4136490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:29.858 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.830+0000 7fd19bfff640 1 --2- 172.21.15.42:0/2078795648 >> v2:172.21.15.154:6812/254845342 conn(0x7fd1a0037810 0x7fd1a0039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:04:29.859 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.830+0000 7fd1b2ffd640 10 monclient: get_auth_request con 0x7fd1b413afd0 auth_method 0 2024-09-16T04:04:29.859 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.830+0000 7fd1b2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:04:29.859 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.830+0000 7fd1b2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:04:29.859 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.830+0000 7fd1b2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:04:29.859 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.830+0000 7fd19bfff640 1 -- 172.21.15.42:0/2078795648 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(681..681 src has 451..681) ==== 6280+0+0 (secure 0 0 0) 0x7fd1ac048d00 con 0x7fd19c0052f0 2024-09-16T04:04:29.859 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.830+0000 7fd19bfff640 10 client.974243.objecter ms_dispatch 0x7fd1b4000ba0 osd_map(681..681 src has 451..681) 2024-09-16T04:04:29.859 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.830+0000 7fd19bfff640 3 client.974243.objecter handle_osd_map got epochs [681,681] > 0 2024-09-16T04:04:29.859 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.830+0000 7fd19bfff640 3 client.974243.objecter handle_osd_map decoding full epoch 681 2024-09-16T04:04:29.859 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.830+0000 7fd19bfff640 20 client.974243.objecter dump_active .. 0 homeless 2024-09-16T04:04:29.859 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.830+0000 7fd1b2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:04:29.859 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.830+0000 7fd1b2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:04:29.859 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.830+0000 7fd1b2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:04:29.859 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.830+0000 7fd1b3fff640 1 --2- 172.21.15.42:0/2078795648 >> v2:172.21.15.154:6812/254845342 conn(0x7fd1a0037810 0x7fd1a0039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:29.859 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.830+0000 7fd1b3fff640 10 monclient: get_auth_request con 0x7fd1a0037810 auth_method 0 2024-09-16T04:04:29.860 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.830+0000 7fd1b2ffd640 10 monclient(hunting): handle_auth_done global_id 904828 payload 824 2024-09-16T04:04:29.860 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.830+0000 7fd1b2ffd640 1 -- 172.21.15.42:0/2078795648 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd1b413c230 con 0x7fd1b413afd0 2024-09-16T04:04:29.860 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.830+0000 7fd19bfff640 10 client.974243.objecter ms_handle_connect 0x7fd1b413afd0 2024-09-16T04:04:29.860 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.830+0000 7fd19bfff640 10 client.974243.objecter resend_mon_ops 2024-09-16T04:04:29.860 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.830+0000 7fd1b2ffd640 1 --2- 172.21.15.42:0/2078795648 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd1b413afd0 0x7fd1b4136490 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fd1b40a67d0 tx=0x7fd1a80149e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:04:29.860 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.830+0000 7fd19bfff640 10 client.974243.objecter ms_handle_connect 0x7fd1a0037810 2024-09-16T04:04:29.860 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.830+0000 7fd1b3fff640 1 --2- 172.21.15.42:0/2078795648 >> v2:172.21.15.154:6812/254845342 conn(0x7fd1a0037810 0x7fd1a0039cd0 secure :-1 s=READY pgs=3828 cs=0 l=1 rev1=1 crypto rx=0x7fd19c00c190 tx=0x7fd1a40a7070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:04:29.860 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.830+0000 7fd19bfff640 1 -- 172.21.15.42:0/2078795648 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1a8004030 con 0x7fd1b413afd0 2024-09-16T04:04:29.860 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.830+0000 7fd19bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:04:29.860 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.830+0000 7fd19bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:04:29.860 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.830+0000 7fd19bfff640 10 monclient: dump: 2024-09-16T04:04:29.860 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:04:29.860 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:04:29.861 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:29.861 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:29.861 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:04:29.861 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:04:29.861 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:04:29.861 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:04:29.861 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:04:29.861 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:04:29.861 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.834+0000 7fd19bfff640 1 -- 172.21.15.42:0/2078795648 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fd1a8004030 con 0x7fd1b413afd0 2024-09-16T04:04:29.861 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.834+0000 7fd19bfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:04:29.861 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.834+0000 7fd19bfff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:04:29.861 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.834+0000 7fd19bfff640 1 -- 172.21.15.42:0/2078795648 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd1b413afd0 msgr2=0x7fd1b4136490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:04:29.861 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.834+0000 7fd19bfff640 1 --2- 172.21.15.42:0/2078795648 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd1b413afd0 0x7fd1b4136490 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fd1b40a67d0 tx=0x7fd1a80149e0 comp rx=0 tx=0).stop 2024-09-16T04:04:29.861 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.834+0000 7fd19bfff640 1 -- 172.21.15.42:0/2078795648 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd1b413afd0 msgr2=0x7fd1b4136490 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:04:29.861 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.834+0000 7fd19bfff640 1 --2- 172.21.15.42:0/2078795648 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd1b413afd0 0x7fd1b4136490 secure :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fd1b40a67d0 tx=0x7fd1a80149e0 comp rx=0 tx=0).stop 2024-09-16T04:04:29.862 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.838+0000 7fd1b9ee0640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:04:29.862 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.838+0000 7fd1b9ee0640 1 --2- 172.21.15.42:0/2078795648 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd1b41369d0 0x7fd1b4136de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:04:29.862 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.838+0000 7fd1b9ee0640 10 monclient(hunting): start opening mon connection 2024-09-16T04:04:29.862 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.838+0000 7fd1b9ee0640 1 -- 172.21.15.42:0/2078795648 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd1b406b300 con 0x7fd1b41369d0 2024-09-16T04:04:29.862 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.838+0000 7fd1b2ffd640 1 --2- 172.21.15.42:0/2078795648 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd1b41369d0 0x7fd1b4136de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:29.862 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.838+0000 7fd1b2ffd640 10 monclient: get_auth_request con 0x7fd1b41369d0 auth_method 0 2024-09-16T04:04:29.862 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.838+0000 7fd1b2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:04:29.862 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.838+0000 7fd1b2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:04:29.862 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.838+0000 7fd1b2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:04:29.862 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.838+0000 7fd1b2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:04:29.862 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.838+0000 7fd1b2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:04:29.862 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.838+0000 7fd1b2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:04:29.862 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.838+0000 7fd1b2ffd640 10 monclient(hunting): handle_auth_done global_id 904831 payload 824 2024-09-16T04:04:29.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.838+0000 7fd1b2ffd640 1 -- 172.21.15.42:0/2078795648 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fd1b4137320 con 0x7fd1b41369d0 2024-09-16T04:04:29.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.842+0000 7fd1b2ffd640 1 --2- 172.21.15.42:0/2078795648 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd1b41369d0 0x7fd1b4136de0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fd1a801e000 tx=0x7fd1a8004ac0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:04:29.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.842+0000 7fd19bfff640 10 client.974243.objecter ms_handle_connect 0x7fd1b41369d0 2024-09-16T04:04:29.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.842+0000 7fd19bfff640 10 client.974243.objecter resend_mon_ops 2024-09-16T04:04:29.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.842+0000 7fd19bfff640 1 -- 172.21.15.42:0/2078795648 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1a8024980 con 0x7fd1b41369d0 2024-09-16T04:04:29.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.842+0000 7fd19bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:04:29.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.842+0000 7fd19bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:04:29.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.842+0000 7fd19bfff640 10 monclient: dump: 2024-09-16T04:04:29.863 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:04:29.863 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:04:29.863 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:29.863 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:29.863 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:04:29.864 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:04:29.864 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:04:29.864 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:04:29.864 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:04:29.864 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:04:29.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.842+0000 7fd19bfff640 1 -- 172.21.15.42:0/2078795648 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4880 (secure 0 0 0) 0x7fd1a8028070 con 0x7fd1b41369d0 2024-09-16T04:04:29.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.842+0000 7fd19bfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:04:29.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.842+0000 7fd19bfff640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:04:29.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.842+0000 7fd19bfff640 1 -- 172.21.15.42:0/2078795648 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd1b41369d0 msgr2=0x7fd1b4136de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:04:29.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.842+0000 7fd19bfff640 1 --2- 172.21.15.42:0/2078795648 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd1b41369d0 0x7fd1b4136de0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fd1a801e000 tx=0x7fd1a8004ac0 comp rx=0 tx=0).stop 2024-09-16T04:04:29.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.842+0000 7fd19bfff640 1 -- 172.21.15.42:0/2078795648 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd1b41369d0 msgr2=0x7fd1b4136de0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:04:29.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.842+0000 7fd19bfff640 1 --2- 172.21.15.42:0/2078795648 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd1b41369d0 0x7fd1b4136de0 secure :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fd1a801e000 tx=0x7fd1a8004ac0 comp rx=0 tx=0).stop 2024-09-16T04:04:29.865 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T04:04:29.865 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:04:29.865 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:04:29.865 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T04:04:29.865 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 210, 2024-09-16T04:04:29.865 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 398018, 2024-09-16T04:04:29.865 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T04:04:29.865 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T04:04:29.865 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T04:04:29.866 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T04:04:29.866 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:04:29.866 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 4, 2024-09-16T04:04:29.866 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:04:29.866 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T04:04:29.866 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T04:04:29.866 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:04:29.866 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:04:29.866 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:04:29.866 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:04:29.866 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:04:29.866 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:04:29.866 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:04:29.866 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:04:29.867 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:04:29.867 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:04:29.867 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:04:29.867 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:04:29.867 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:04:29.867 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T04:04:29.867 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:04:29.867 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:04:29.867 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:04:29.867 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:04:29.867 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:04:29.867 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:04:29.867 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:04:29.867 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:04:29.867 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:04:29.868 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:04:29.868 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:04:29.868 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:04:29.868 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:04:29.868 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T04:04:29.868 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T04:04:29.868 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T04:04:29.868 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T04:04:29.868 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T04:04:29.868 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T04:04:29.868 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:04:29.868 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:04:29.869 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T04:04:29.869 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T04:04:29.869 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T04:04:29.869 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T04:04:29.869 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T04:04:29.869 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T04:04:29.869 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T04:04:29.869 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:04:29.869 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T04:04:29.869 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:04:29.869 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:04:29.869 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:04:29.869 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:04:29.869 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:04:29.870 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:04:29.870 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:04:29.870 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:04:29.870 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:04:29.870 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:04:29.870 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:04:29.870 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:04:29.870 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T04:04:29.870 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:04:29.870 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T04:04:29.870 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:04:29.870 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:04:29.870 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:04:29.870 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:04:29.871 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:04:29.871 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:04:29.871 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:04:29.871 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T04:04:29.871 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:04:29.871 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:04:29.871 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:04:29.871 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:04:29.871 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T04:04:29.871 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:04:29.871 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:04:29.871 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:04:29.871 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:04:29.872 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T04:04:29.872 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T04:04:29.872 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:04:29.872 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:04:29.872 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:04:29.872 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:04:29.872 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:04:29.872 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:04:29.872 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:04:29.872 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:04:29.872 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:04:29.873 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:04:29.873 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:04:29.873 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T04:04:29.873 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:04:29.873 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:04:29.873 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:04:29.873 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:04:29.873 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T04:04:29.873 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:04:29.873 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:04:29.873 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:04:29.873 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:04:29.873 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T04:04:29.873 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T04:04:29.874 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:04:29.874 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:04:29.874 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:04:29.874 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:04:29.874 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:04:29.874 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:04:29.874 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:04:29.874 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:04:29.874 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:04:29.874 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:04:29.874 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:04:29.874 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T04:04:29.874 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:04:29.875 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:04:29.875 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:04:29.875 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:04:29.875 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T04:04:29.875 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:04:29.875 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:04:29.875 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:04:29.875 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:04:29.875 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T04:04:29.875 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T04:04:29.875 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:04:29.875 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:04:29.875 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:04:29.875 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:04:29.875 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:04:29.876 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:04:29.876 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T04:04:29.876 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T04:04:29.876 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:04:29.876 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:04:29.876 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:04:29.876 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:04:29.876 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:04:29.876 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:04:29.876 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T04:04:29.876 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:04:29.876 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:04:29.876 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:04:29.876 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 3 2024-09-16T04:04:29.877 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:04:29.877 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:04:29.877 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T04:04:29.877 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:04:29.877 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:04:29.877 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:04:29.877 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 6 2024-09-16T04:04:29.877 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:04:29.877 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:04:29.877 INFO:teuthology.orchestra.run.smithi042.stdout: "mgr": [ 2024-09-16T04:04:29.877 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:04:29.877 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:04:29.877 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:04:29.877 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:04:29.878 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:04:29.878 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:04:29.878 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:04:29.878 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T04:04:29.878 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T04:04:29.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.842+0000 7fd1b9ee0640 10 librados: watch_flush enter 2024-09-16T04:04:29.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.842+0000 7fd1b9ee0640 10 librados: watch_flush exit 2024-09-16T04:04:29.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.842+0000 7fd1b9ee0640 20 client.974243.objecter shutdown clearing up homeless session... 2024-09-16T04:04:29.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.842+0000 7fd1b9ee0640 10 client.974243.objecter successfully canceled tick 2024-09-16T04:04:29.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.842+0000 7fd1b9ee0640 1 -- 172.21.15.42:0/2078795648 >> v2:172.21.15.154:6812/254845342 conn(0x7fd1a0037810 msgr2=0x7fd1a0039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:04:29.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.842+0000 7fd1b9ee0640 1 --2- 172.21.15.42:0/2078795648 >> v2:172.21.15.154:6812/254845342 conn(0x7fd1a0037810 0x7fd1a0039cd0 secure :-1 s=READY pgs=3828 cs=0 l=1 rev1=1 crypto rx=0x7fd19c00c190 tx=0x7fd1a40a7070 comp rx=0 tx=0).stop 2024-09-16T04:04:29.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.842+0000 7fd1b9ee0640 10 monclient: shutdown 2024-09-16T04:04:29.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.842+0000 7fd1b9ee0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:04:29.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.842+0000 7fd1b9ee0640 1 -- 172.21.15.42:0/2078795648 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd19c0052f0 msgr2=0x7fd1b41ccab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:04:29.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.842+0000 7fd1b9ee0640 1 --2- 172.21.15.42:0/2078795648 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd19c0052f0 0x7fd1b41ccab0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fd1ac00e500 tx=0x7fd1ac00e9c0 comp rx=0 tx=0).stop 2024-09-16T04:04:29.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.842+0000 7fd1b3fff640 1 -- 172.21.15.42:0/2078795648 reap_dead start 2024-09-16T04:04:29.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.842+0000 7fd1b3fff640 1 -- 172.21.15.42:0/2078795648 reap_dead start 2024-09-16T04:04:29.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.842+0000 7fd1b9ee0640 1 -- 172.21.15.42:0/2078795648 shutdown_connections 2024-09-16T04:04:29.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.842+0000 7fd1b9ee0640 1 -- 172.21.15.42:0/2078795648 >> 172.21.15.42:0/2078795648 conn(0x7fd1b40a6e80 msgr2=0x7fd1b40a5180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:04:29.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.842+0000 7fd1b9ee0640 1 -- 172.21.15.42:0/2078795648 shutdown_connections 2024-09-16T04:04:29.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.842+0000 7fd1b9ee0640 1 -- 172.21.15.42:0/2078795648 wait complete. 2024-09-16T04:04:29.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:29.842+0000 7fd1b9ee0640 1 librados: shutdown 2024-09-16T04:04:29.881 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:04:29.881 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-09-16T04:04:29.881 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-16T04:04:29.881 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-09-16T04:04:29.881 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-16T04:04:29.882 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-16T04:04:29.882 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T04:04:29.894 INFO:tasks.ceph.mon.c:Stopped 2024-09-16T04:04:29.894 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T04:04:29.894 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:04:30.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.050+0000 7ffb66802640 10 monclient: get_monmap_and_config 2024-09-16T04:04:30.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.050+0000 7ffb66802640 10 monclient: build_initial_monmap 2024-09-16T04:04:30.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.050+0000 7ffb66802640 10 monclient: monmap: 2024-09-16T04:04:30.055 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:04:30.055 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:04:30.055 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:04:30.053389+0000 2024-09-16T04:04:30.055 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:04:30.053389+0000 2024-09-16T04:04:30.055 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:04:30.055 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:04:30.056 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:04:30.056 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:04:30.056 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:04:30.056 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:04:30.056 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:04:30.056 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:04:30.067 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.050+0000 7ffb66802640 1 Processor -- start 2024-09-16T04:04:30.067 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.050+0000 7ffb66802640 1 -- start start 2024-09-16T04:04:30.067 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.050+0000 7ffb66802640 10 monclient: init 2024-09-16T04:04:30.067 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.050+0000 7ffb66802640 10 monclient: _reopen_session rank -1 2024-09-16T04:04:30.067 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.050+0000 7ffb66802640 10 monclient: _add_conns ranks=[2,0,4,1,3] 2024-09-16T04:04:30.067 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.050+0000 7ffb66802640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ffb600a39b0 0x7ffb60133b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:04:30.067 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.050+0000 7ffb66802640 10 monclient(hunting): picked mon.noname-b con 0x7ffb600a39b0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:04:30.067 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.050+0000 7ffb66802640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ffb600a3d90 addr v1:172.21.15.42:6789/0 2024-09-16T04:04:30.067 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.050+0000 7ffb66802640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7ffb60137710 0x7ffb6013f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:04:30.068 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.050+0000 7ffb66802640 10 monclient(hunting): picked mon.noname-c con 0x7ffb60137710 addr v2:172.21.15.154:3300/0 2024-09-16T04:04:30.068 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.050+0000 7ffb66802640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffb6009c140 con 0x7ffb600a3d90 2024-09-16T04:04:30.068 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.050+0000 7ffb66802640 10 monclient(hunting): start opening mon connection 2024-09-16T04:04:30.068 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.050+0000 7ffb66802640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7ffb6009c420 con 0x7ffb60137710 2024-09-16T04:04:30.068 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.050+0000 7ffb66802640 10 monclient(hunting): start opening mon connection 2024-09-16T04:04:30.068 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.050+0000 7ffb66802640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7ffb6006dd60 con 0x7ffb600a39b0 2024-09-16T04:04:30.068 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.050+0000 7ffb66802640 10 monclient(hunting): _renew_subs 2024-09-16T04:04:30.068 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.050+0000 7ffb66802640 10 monclient(hunting): authenticate will time out at 2317.023582s 2024-09-16T04:04:30.068 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.050+0000 7ffb5ffff640 1 -- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ffb600a39b0 msgr2=0x7ffb60133b80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T04:04:30.068 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.050+0000 7ffb5ffff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ffb600a39b0 0x7ffb60133b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:04:30.068 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.050+0000 7ffb577fe640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7ffb600a3d90 0x7ffb60137000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:39752/0 (socket says 172.21.15.42:39752) 2024-09-16T04:04:30.069 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.050+0000 7ffb577fe640 1 -- 172.21.15.42:0/4244433155 learned_addr learned my addr 172.21.15.42:0/4244433155 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T04:04:30.069 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.054+0000 7ffb5f7fe640 1 -- 172.21.15.42:0/4244433155 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3506379177 0 0) 0x7ffb6009c140 con 0x7ffb600a3d90 2024-09-16T04:04:30.069 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.054+0000 7ffb5f7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:04:30.069 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.054+0000 7ffb5f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:04:30.069 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.054+0000 7ffb5f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:04:30.069 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.054+0000 7ffb5f7fe640 10 monclient(hunting): my global_id is 984174 2024-09-16T04:04:30.069 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.054+0000 7ffb5f7fe640 1 -- 172.21.15.42:0/4244433155 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffb4c003530 con 0x7ffb600a3d90 2024-09-16T04:04:30.069 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.054+0000 7ffb64d9f640 1 --2- 172.21.15.42:0/4244433155 >> v2:172.21.15.154:3300/0 conn(0x7ffb60137710 0x7ffb6013f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:30.069 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.054+0000 7ffb64d9f640 10 monclient(hunting): get_auth_request con 0x7ffb60137710 auth_method 0 2024-09-16T04:04:30.069 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.054+0000 7ffb64d9f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:04:30.069 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.054+0000 7ffb64d9f640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:04:30.070 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.054+0000 7ffb64d9f640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:04:30.070 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.054+0000 7ffb5f7fe640 1 -- 172.21.15.42:0/4244433155 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2249210068 0 0) 0x7ffb4c003530 con 0x7ffb600a3d90 2024-09-16T04:04:30.070 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.054+0000 7ffb5f7fe640 1 -- 172.21.15.42:0/4244433155 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ffb600a39b0 msgr2=0x7ffb60133b80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:04:30.070 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.054+0000 7ffb5f7fe640 1 --2- 172.21.15.42:0/4244433155 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ffb600a39b0 0x7ffb60133b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:30.070 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.054+0000 7ffb5f7fe640 1 -- 172.21.15.42:0/4244433155 >> v2:172.21.15.154:3300/0 conn(0x7ffb60137710 msgr2=0x7ffb6013f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:04:30.070 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.054+0000 7ffb5f7fe640 1 --2- 172.21.15.42:0/4244433155 >> v2:172.21.15.154:3300/0 conn(0x7ffb60137710 0x7ffb6013f3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:30.070 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.054+0000 7ffb5f7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:04:30.070 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.054+0000 7ffb5f7fe640 1 monclient: found mon.noname-a-legacy 2024-09-16T04:04:30.070 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.054+0000 7ffb5f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:04:30.070 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.054+0000 7ffb5f7fe640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.42:6789/0 2024-09-16T04:04:30.070 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.054+0000 7ffb5f7fe640 1 -- 172.21.15.42:0/4244433155 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb600a5e20 con 0x7ffb600a3d90 2024-09-16T04:04:30.070 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.054+0000 7ffb5f7fe640 10 monclient: _finish_auth 0 2024-09-16T04:04:30.071 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.054+0000 7ffb5f7fe640 10 monclient: _check_auth_tickets 2024-09-16T04:04:30.071 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.054+0000 7ffb5f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:04:30.071 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.054+0000 7ffb5f7fe640 1 -- 172.21.15.42:0/4244433155 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2177413734 0 0) 0x7ffb50001680 con 0x7ffb600a3d90 2024-09-16T04:04:30.071 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.054+0000 7ffb66802640 5 monclient: authenticate success, global_id 984174 2024-09-16T04:04:30.071 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.054+0000 7ffb5f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:04:30.071 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.054+0000 7ffb5f7fe640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T04:04:30.071 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.054+0000 7ffb5f7fe640 10 monclient: dump: 2024-09-16T04:04:30.071 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:04:30.071 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:04:30.071 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:30.071 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:30.071 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:04:30.072 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:04:30.072 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:04:30.072 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:04:30.072 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:04:30.072 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:04:30.072 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.054+0000 7ffb5f7fe640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] but i'm connected to v1:172.21.15.42:6789/0, reconnecting 2024-09-16T04:04:30.072 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.054+0000 7ffb5f7fe640 10 monclient: _reopen_session rank -1 2024-09-16T04:04:30.072 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.054+0000 7ffb5f7fe640 1 -- 172.21.15.42:0/4244433155 >> v1:172.21.15.42:6789/0 conn(0x7ffb600a3d90 legacy=0x7ffb60137000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:04:30.072 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.054+0000 7ffb5f7fe640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:04:30.072 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.054+0000 7ffb5f7fe640 1 --2- 172.21.15.42:0/4244433155 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ffb4c0039c0 0x7ffb4c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:04:30.072 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.054+0000 7ffb5f7fe640 10 monclient(hunting): picked mon.a con 0x7ffb4c0039c0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:04:30.072 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.054+0000 7ffb5f7fe640 1 --2- 172.21.15.42:0/4244433155 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ffb600a39b0 0x7ffb4c004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:04:30.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.054+0000 7ffb5f7fe640 10 monclient(hunting): picked mon.c con 0x7ffb600a39b0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:04:30.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.054+0000 7ffb5f7fe640 1 --2- 172.21.15.42:0/4244433155 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffb4c004d50 0x7ffb4c0051b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:04:30.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.054+0000 7ffb5f7fe640 10 monclient(hunting): picked mon.b con 0x7ffb4c004d50 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:04:30.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.054+0000 7ffb5f7fe640 10 monclient(hunting): start opening mon connection 2024-09-16T04:04:30.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.054+0000 7ffb5f7fe640 1 -- 172.21.15.42:0/4244433155 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7ffb6009c420 con 0x7ffb4c0039c0 2024-09-16T04:04:30.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.054+0000 7ffb5f7fe640 10 monclient(hunting): start opening mon connection 2024-09-16T04:04:30.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.054+0000 7ffb5f7fe640 1 -- 172.21.15.42:0/4244433155 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ffb6006dd60 con 0x7ffb4c004d50 2024-09-16T04:04:30.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.054+0000 7ffb5f7fe640 10 monclient(hunting): start opening mon connection 2024-09-16T04:04:30.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.054+0000 7ffb5f7fe640 1 -- 172.21.15.42:0/4244433155 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7ffb4c00bc60 con 0x7ffb600a39b0 2024-09-16T04:04:30.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.054+0000 7ffb5f7fe640 10 monclient(hunting): _renew_subs 2024-09-16T04:04:30.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.054+0000 7ffb5f7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:04:30.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.054+0000 7ffb66802640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-16T04:04:30.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.054+0000 7ffb577fe640 1 -- 172.21.15.42:0/4244433155 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ffb600a39b0 msgr2=0x7ffb4c004810 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T04:04:30.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.054+0000 7ffb577fe640 1 --2- 172.21.15.42:0/4244433155 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ffb600a39b0 0x7ffb4c004810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:04:30.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.054+0000 7ffb5ffff640 1 --2- 172.21.15.42:0/4244433155 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ffb4c0039c0 0x7ffb4c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:30.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.054+0000 7ffb5ffff640 10 monclient(hunting): get_auth_request con 0x7ffb4c0039c0 auth_method 0 2024-09-16T04:04:30.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.054+0000 7ffb5ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:04:30.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.054+0000 7ffb5ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:04:30.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.054+0000 7ffb5ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:04:30.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.054+0000 7ffb5ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:04:30.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.054+0000 7ffb5ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:04:30.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.054+0000 7ffb5ffff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:04:30.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.054+0000 7ffb64d9f640 1 --2- 172.21.15.42:0/4244433155 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffb4c004d50 0x7ffb4c0051b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:30.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.054+0000 7ffb64d9f640 10 monclient(hunting): get_auth_request con 0x7ffb4c004d50 auth_method 0 2024-09-16T04:04:30.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.054+0000 7ffb64d9f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:04:30.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.054+0000 7ffb64d9f640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:04:30.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.054+0000 7ffb64d9f640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:04:30.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.054+0000 7ffb5ffff640 10 monclient(hunting): handle_auth_done global_id 984174 payload 293 2024-09-16T04:04:30.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.054+0000 7ffb5ffff640 1 -- 172.21.15.42:0/4244433155 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ffb600a39b0 msgr2=0x7ffb4c004810 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:04:30.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.054+0000 7ffb5ffff640 1 --2- 172.21.15.42:0/4244433155 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ffb600a39b0 0x7ffb4c004810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:30.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.054+0000 7ffb5ffff640 1 -- 172.21.15.42:0/4244433155 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffb4c004d50 msgr2=0x7ffb4c0051b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:04:30.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.054+0000 7ffb5ffff640 1 --2- 172.21.15.42:0/4244433155 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffb4c004d50 0x7ffb4c0051b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:30.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.054+0000 7ffb5ffff640 10 monclient: _finish_hunting 0 2024-09-16T04:04:30.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.054+0000 7ffb5ffff640 1 monclient: found mon.a 2024-09-16T04:04:30.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.054+0000 7ffb5ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:04:30.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.054+0000 7ffb5ffff640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:04:30.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.054+0000 7ffb5ffff640 1 -- 172.21.15.42:0/4244433155 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb4c003530 con 0x7ffb4c0039c0 2024-09-16T04:04:30.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.054+0000 7ffb5ffff640 10 monclient: _finish_auth 0 2024-09-16T04:04:30.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.054+0000 7ffb5ffff640 10 monclient: _check_auth_tickets 2024-09-16T04:04:30.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.054+0000 7ffb5ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:04:30.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.054+0000 7ffb5ffff640 1 --2- 172.21.15.42:0/4244433155 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ffb4c0039c0 0x7ffb4c003de0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7ffb48003990 tx=0x7ffb4801c420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f227d370e0b9bc31 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:04:30.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.054+0000 7ffb5f7fe640 1 -- 172.21.15.42:0/4244433155 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb48009520 con 0x7ffb4c0039c0 2024-09-16T04:04:30.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.054+0000 7ffb5f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:04:30.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.054+0000 7ffb5f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:04:30.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.054+0000 7ffb5f7fe640 10 monclient: dump: 2024-09-16T04:04:30.076 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:04:30.076 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:04:30.076 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:30.077 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:30.077 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:04:30.077 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:04:30.077 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:04:30.077 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:04:30.077 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:04:30.077 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:04:30.077 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.054+0000 7ffb5f7fe640 1 -- 172.21.15.42:0/4244433155 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffb48003bc0 con 0x7ffb4c0039c0 2024-09-16T04:04:30.077 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.054+0000 7ffb66802640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:04:30.077 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.054+0000 7ffb5f7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:04:30.077 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.054+0000 7ffb66802640 10 monclient: get_monmap_and_config success 2024-09-16T04:04:30.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.054+0000 7ffb5f7fe640 1 -- 172.21.15.42:0/4244433155 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb48005070 con 0x7ffb4c0039c0 2024-09-16T04:04:30.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.054+0000 7ffb66802640 4 set_mon_vals no callback set 2024-09-16T04:04:30.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.054+0000 7ffb5f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:04:30.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.054+0000 7ffb5f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:04:30.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.054+0000 7ffb5f7fe640 10 monclient: dump: 2024-09-16T04:04:30.078 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:04:30.078 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:04:30.078 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:30.078 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:30.078 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:04:30.078 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:04:30.078 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:04:30.079 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:04:30.079 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:04:30.079 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:04:30.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.054+0000 7ffb66802640 10 monclient: shutdown 2024-09-16T04:04:30.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.054+0000 7ffb66802640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:04:30.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.054+0000 7ffb66802640 1 -- 172.21.15.42:0/4244433155 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ffb4c0039c0 msgr2=0x7ffb4c003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:04:30.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.054+0000 7ffb66802640 1 --2- 172.21.15.42:0/4244433155 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ffb4c0039c0 0x7ffb4c003de0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7ffb48003990 tx=0x7ffb4801c420 comp rx=0 tx=0).stop 2024-09-16T04:04:30.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.054+0000 7ffb64d9f640 1 -- 172.21.15.42:0/4244433155 reap_dead start 2024-09-16T04:04:30.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.058+0000 7ffb66802640 1 -- 172.21.15.42:0/4244433155 shutdown_connections 2024-09-16T04:04:30.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.058+0000 7ffb66802640 1 -- 172.21.15.42:0/4244433155 >> 172.21.15.42:0/4244433155 conn(0x7ffb600a9fb0 msgr2=0x7ffb600aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:04:30.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.058+0000 7ffb66802640 1 -- 172.21.15.42:0/4244433155 shutdown_connections 2024-09-16T04:04:30.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.058+0000 7ffb66802640 1 -- 172.21.15.42:0/4244433155 wait complete. 2024-09-16T04:04:30.080 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.058+0000 7ffb66802640 10 monclient: build_initial_monmap 2024-09-16T04:04:30.080 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.058+0000 7ffb66802640 10 monclient: monmap: 2024-09-16T04:04:30.080 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:04:30.080 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:04:30.080 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:04:30.080 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:04:30.080 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:04:30.080 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:04:30.080 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:04:30.080 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:04:30.080 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:04:30.081 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:04:30.081 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.058+0000 7ffb66802640 1 librados: starting msgr at 2024-09-16T04:04:30.081 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.058+0000 7ffb66802640 1 librados: starting objecter 2024-09-16T04:04:30.081 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.058+0000 7ffb66802640 1 Processor -- start 2024-09-16T04:04:30.081 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.058+0000 7ffb66802640 1 -- start start 2024-09-16T04:04:30.081 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.058+0000 7ffb66802640 1 librados: setting wanted keys 2024-09-16T04:04:30.081 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.058+0000 7ffb66802640 1 librados: calling monclient init 2024-09-16T04:04:30.081 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.058+0000 7ffb66802640 10 monclient: init 2024-09-16T04:04:30.081 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.058+0000 7ffb66802640 10 monclient: _reopen_session rank -1 2024-09-16T04:04:30.081 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.058+0000 7ffb66802640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:04:30.081 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.058+0000 7ffb66802640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ffb4c0039c0 0x7ffb600a33d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:04:30.081 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.058+0000 7ffb66802640 10 monclient(hunting): picked mon.noname-c con 0x7ffb4c0039c0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:04:30.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.058+0000 7ffb66802640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ffb6013b900 0x7ffb6013bd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:04:30.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.058+0000 7ffb66802640 10 monclient(hunting): picked mon.noname-a con 0x7ffb6013b900 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:04:30.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.058+0000 7ffb66802640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffb6013c2a0 0x7ffb601d5f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:04:30.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.058+0000 7ffb66802640 10 monclient(hunting): picked mon.noname-b con 0x7ffb6013c2a0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:04:30.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.058+0000 7ffb66802640 10 monclient(hunting): start opening mon connection 2024-09-16T04:04:30.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.058+0000 7ffb5ffff640 1 -- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ffb4c0039c0 msgr2=0x7ffb600a33d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T04:04:30.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.058+0000 7ffb66802640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7ffb6013faf0 con 0x7ffb6013b900 2024-09-16T04:04:30.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.058+0000 7ffb66802640 10 monclient(hunting): start opening mon connection 2024-09-16T04:04:30.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.058+0000 7ffb66802640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ffb6013fc70 con 0x7ffb6013c2a0 2024-09-16T04:04:30.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.058+0000 7ffb66802640 10 monclient(hunting): start opening mon connection 2024-09-16T04:04:30.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.058+0000 7ffb66802640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7ffb6013fdf0 con 0x7ffb4c0039c0 2024-09-16T04:04:30.083 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.058+0000 7ffb66802640 10 monclient(hunting): _renew_subs 2024-09-16T04:04:30.083 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.058+0000 7ffb5ffff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ffb4c0039c0 0x7ffb600a33d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:04:30.083 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.058+0000 7ffb66802640 10 monclient(hunting): authenticate will time out at 2317.029730s 2024-09-16T04:04:30.083 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.058+0000 7ffb577fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ffb6013b900 0x7ffb6013bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:30.083 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.058+0000 7ffb577fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ffb6013b900 0x7ffb6013bd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:39796/0 (socket says 172.21.15.42:39796) 2024-09-16T04:04:30.083 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.058+0000 7ffb577fe640 1 -- 172.21.15.42:0/3332449116 learned_addr learned my addr 172.21.15.42:0/3332449116 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:04:30.083 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.058+0000 7ffb577fe640 10 monclient(hunting): get_auth_request con 0x7ffb6013b900 auth_method 0 2024-09-16T04:04:30.083 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.058+0000 7ffb577fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:04:30.083 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.058+0000 7ffb577fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:04:30.083 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.058+0000 7ffb577fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:04:30.083 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.058+0000 7ffb577fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:04:30.083 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.058+0000 7ffb577fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:04:30.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.058+0000 7ffb577fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:04:30.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.058+0000 7ffb64d9f640 1 --2- 172.21.15.42:0/3332449116 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffb6013c2a0 0x7ffb601d5f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:30.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.058+0000 7ffb64d9f640 10 monclient(hunting): get_auth_request con 0x7ffb6013c2a0 auth_method 0 2024-09-16T04:04:30.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.058+0000 7ffb64d9f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:04:30.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.058+0000 7ffb64d9f640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:04:30.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.058+0000 7ffb64d9f640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:04:30.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.058+0000 7ffb577fe640 10 monclient(hunting): handle_auth_done global_id 984177 payload 824 2024-09-16T04:04:30.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.058+0000 7ffb577fe640 1 -- 172.21.15.42:0/3332449116 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ffb4c0039c0 msgr2=0x7ffb600a33d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:04:30.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.058+0000 7ffb577fe640 1 --2- 172.21.15.42:0/3332449116 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ffb4c0039c0 0x7ffb600a33d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:30.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.058+0000 7ffb577fe640 1 -- 172.21.15.42:0/3332449116 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffb6013c2a0 msgr2=0x7ffb601d5f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:04:30.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.058+0000 7ffb577fe640 1 --2- 172.21.15.42:0/3332449116 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffb6013c2a0 0x7ffb601d5f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:30.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.058+0000 7ffb577fe640 10 monclient: _finish_hunting 0 2024-09-16T04:04:30.085 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.058+0000 7ffb577fe640 1 monclient: found mon.noname-a 2024-09-16T04:04:30.085 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.058+0000 7ffb577fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:04:30.085 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.058+0000 7ffb577fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:04:30.085 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.058+0000 7ffb577fe640 1 -- 172.21.15.42:0/3332449116 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb6013df10 con 0x7ffb6013b900 2024-09-16T04:04:30.085 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.058+0000 7ffb577fe640 1 --2- 172.21.15.42:0/3332449116 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ffb6013b900 0x7ffb6013bd60 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7ffb5000c6e0 tx=0x7ffb5000cba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:04:30.085 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.058+0000 7ffb5d7fa640 10 client.?.objecter ms_handle_connect 0x7ffb6013b900 2024-09-16T04:04:30.085 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.058+0000 7ffb5d7fa640 10 client.?.objecter resend_mon_ops 2024-09-16T04:04:30.085 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.058+0000 7ffb5d7fa640 1 -- 172.21.15.42:0/3332449116 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb50018070 con 0x7ffb6013b900 2024-09-16T04:04:30.085 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.058+0000 7ffb5d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:04:30.085 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.058+0000 7ffb5d7fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:04:30.085 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.058+0000 7ffb5d7fa640 10 monclient: dump: 2024-09-16T04:04:30.085 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:04:30.086 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:04:30.086 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:30.086 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:30.086 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:04:30.086 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:04:30.086 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:04:30.086 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:04:30.086 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:04:30.086 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:04:30.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.058+0000 7ffb5d7fa640 10 monclient: _finish_auth 0 2024-09-16T04:04:30.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.058+0000 7ffb5d7fa640 10 monclient: _check_auth_tickets 2024-09-16T04:04:30.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.058+0000 7ffb5d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:04:30.087 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.058+0000 7ffb5d7fa640 1 -- 172.21.15.42:0/3332449116 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffb500072d0 con 0x7ffb6013b900 2024-09-16T04:04:30.087 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.058+0000 7ffb5d7fa640 10 monclient: handle_config config(0 keys) 2024-09-16T04:04:30.087 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.058+0000 7ffb5d7fa640 1 -- 172.21.15.42:0/3332449116 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb500047f0 con 0x7ffb6013b900 2024-09-16T04:04:30.087 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.058+0000 7ffb66802640 5 monclient: authenticate success, global_id 984177 2024-09-16T04:04:30.087 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.058+0000 7ffb66802640 10 monclient: _renew_subs 2024-09-16T04:04:30.087 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.058+0000 7ffb66802640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:04:30.087 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.058+0000 7ffb66802640 1 -- 172.21.15.42:0/3332449116 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffb6013e150 con 0x7ffb6013b900 2024-09-16T04:04:30.087 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.058+0000 7ffb5d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:04:30.087 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.058+0000 7ffb5d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:04:30.087 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.058+0000 7ffb5effd640 4 set_mon_vals no callback set 2024-09-16T04:04:30.087 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.058+0000 7ffb5d7fa640 10 monclient: dump: 2024-09-16T04:04:30.088 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:04:30.088 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:04:30.088 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:30.088 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:30.088 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:04:30.088 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:04:30.088 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:04:30.088 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:04:30.088 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:04:30.088 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:04:30.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.058+0000 7ffb66802640 10 client.984177.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:04:30.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.058+0000 7ffb66802640 10 monclient: _renew_subs 2024-09-16T04:04:30.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.058+0000 7ffb66802640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:04:30.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.058+0000 7ffb66802640 1 -- 172.21.15.42:0/3332449116 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb6013f050 con 0x7ffb6013b900 2024-09-16T04:04:30.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.058+0000 7ffb66802640 1 librados: init done 2024-09-16T04:04:30.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.058+0000 7ffb5d7fa640 1 -- 172.21.15.42:0/3332449116 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7ffb5001b070 con 0x7ffb6013b900 2024-09-16T04:04:30.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.058+0000 7ffb66802640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:04:30.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.058+0000 7ffb66802640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:04:30.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.058+0000 7ffb66802640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:04:30.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.058+0000 7ffb5d7fa640 1 --2- 172.21.15.42:0/3332449116 >> v2:172.21.15.154:6812/254845342 conn(0x7ffb400377c0 0x7ffb40039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:04:30.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.058+0000 7ffb66802640 1 -- 172.21.15.42:0/3332449116 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffb60133860 con 0x7ffb6013b900 2024-09-16T04:04:30.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.058+0000 7ffb5d7fa640 1 -- 172.21.15.42:0/3332449116 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(681..681 src has 451..681) ==== 6280+0+0 (secure 0 0 0) 0x7ffb50048630 con 0x7ffb6013b900 2024-09-16T04:04:30.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.058+0000 7ffb5d7fa640 10 client.984177.objecter ms_dispatch 0x7ffb60000ba0 osd_map(681..681 src has 451..681) 2024-09-16T04:04:30.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.058+0000 7ffb5d7fa640 3 client.984177.objecter handle_osd_map got epochs [681,681] > 0 2024-09-16T04:04:30.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.058+0000 7ffb5d7fa640 3 client.984177.objecter handle_osd_map decoding full epoch 681 2024-09-16T04:04:30.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.058+0000 7ffb5d7fa640 20 client.984177.objecter dump_active .. 0 homeless 2024-09-16T04:04:30.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.062+0000 7ffb5ffff640 1 --2- 172.21.15.42:0/3332449116 >> v2:172.21.15.154:6812/254845342 conn(0x7ffb400377c0 0x7ffb40039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:30.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.062+0000 7ffb5ffff640 10 monclient: get_auth_request con 0x7ffb400377c0 auth_method 0 2024-09-16T04:04:30.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.062+0000 7ffb5d7fa640 10 client.984177.objecter ms_handle_connect 0x7ffb400377c0 2024-09-16T04:04:30.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.062+0000 7ffb5ffff640 1 --2- 172.21.15.42:0/3332449116 >> v2:172.21.15.154:6812/254845342 conn(0x7ffb400377c0 0x7ffb40039c80 secure :-1 s=READY pgs=3829 cs=0 l=1 rev1=1 crypto rx=0x7ffb480aa000 tx=0x7ffb48005a90 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:04:30.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.066+0000 7ffb5d7fa640 1 -- 172.21.15.42:0/3332449116 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7ffb5000a960 con 0x7ffb6013b900 2024-09-16T04:04:30.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.066+0000 7ffb5d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:04:30.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.066+0000 7ffb5d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:04:30.294 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.290+0000 7ffb66802640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:04:30.295 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.290+0000 7ffb66802640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:04:30.295 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.290+0000 7ffb66802640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:04:30.295 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.290+0000 7ffb66802640 1 -- 172.21.15.42:0/3332449116 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7ffb60137000 con 0x7ffb6013b900 2024-09-16T04:04:30.295 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.290+0000 7ffb5d7fa640 1 -- 172.21.15.42:0/3332449116 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7ffb50004990 con 0x7ffb6013b900 2024-09-16T04:04:30.295 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.290+0000 7ffb5d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:04:30.296 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.290+0000 7ffb5d7fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:04:30.296 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":210,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":4,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:04:30.298 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.290+0000 7ffb66802640 10 librados: watch_flush enter 2024-09-16T04:04:30.298 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.290+0000 7ffb66802640 10 librados: watch_flush exit 2024-09-16T04:04:30.298 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.290+0000 7ffb66802640 20 client.984177.objecter shutdown clearing up homeless session... 2024-09-16T04:04:30.298 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.290+0000 7ffb66802640 10 client.984177.objecter successfully canceled tick 2024-09-16T04:04:30.298 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.294+0000 7ffb66802640 1 -- 172.21.15.42:0/3332449116 >> v2:172.21.15.154:6812/254845342 conn(0x7ffb400377c0 msgr2=0x7ffb40039c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:04:30.298 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.294+0000 7ffb66802640 1 --2- 172.21.15.42:0/3332449116 >> v2:172.21.15.154:6812/254845342 conn(0x7ffb400377c0 0x7ffb40039c80 secure :-1 s=READY pgs=3829 cs=0 l=1 rev1=1 crypto rx=0x7ffb480aa000 tx=0x7ffb48005a90 comp rx=0 tx=0).stop 2024-09-16T04:04:30.298 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.294+0000 7ffb66802640 10 monclient: shutdown 2024-09-16T04:04:30.298 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.294+0000 7ffb66802640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:04:30.298 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.294+0000 7ffb66802640 1 -- 172.21.15.42:0/3332449116 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ffb6013b900 msgr2=0x7ffb6013bd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:04:30.298 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.294+0000 7ffb66802640 1 --2- 172.21.15.42:0/3332449116 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ffb6013b900 0x7ffb6013bd60 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7ffb5000c6e0 tx=0x7ffb5000cba0 comp rx=0 tx=0).stop 2024-09-16T04:04:30.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.294+0000 7ffb66802640 1 -- 172.21.15.42:0/3332449116 shutdown_connections 2024-09-16T04:04:30.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.294+0000 7ffb66802640 1 --2- 172.21.15.42:0/3332449116 >> v2:172.21.15.154:6812/254845342 conn(0x7ffb400377c0 0x7ffb40039c80 unknown :-1 s=CLOSED pgs=3829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:30.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.294+0000 7ffb66802640 1 --2- 172.21.15.42:0/3332449116 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffb6013c2a0 0x7ffb601d5f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:30.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.294+0000 7ffb66802640 1 --2- 172.21.15.42:0/3332449116 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ffb6013b900 0x7ffb6013bd60 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:30.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.294+0000 7ffb66802640 1 --2- 172.21.15.42:0/3332449116 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ffb4c0039c0 0x7ffb600a33d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:30.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.294+0000 7ffb66802640 1 -- 172.21.15.42:0/3332449116 >> 172.21.15.42:0/3332449116 conn(0x7ffb600a9fb0 msgr2=0x7ffb601307a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:04:30.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.294+0000 7ffb66802640 1 -- 172.21.15.42:0/3332449116 shutdown_connections 2024-09-16T04:04:30.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.294+0000 7ffb66802640 1 -- 172.21.15.42:0/3332449116 wait complete. 2024-09-16T04:04:30.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:30.294+0000 7ffb66802640 1 librados: shutdown 2024-09-16T04:04:33.314 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:04:33.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.442+0000 7fd137e64640 10 monclient: get_monmap_and_config 2024-09-16T04:04:33.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.442+0000 7fd137e64640 10 monclient: build_initial_monmap 2024-09-16T04:04:33.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.442+0000 7fd137e64640 10 monclient: monmap: 2024-09-16T04:04:33.447 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:04:33.447 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:04:33.447 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:04:33.445579+0000 2024-09-16T04:04:33.447 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:04:33.445579+0000 2024-09-16T04:04:33.447 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:04:33.447 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:04:33.448 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:04:33.448 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:04:33.448 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:04:33.448 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:04:33.448 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:04:33.448 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:04:33.450 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.442+0000 7fd137e64640 1 Processor -- start 2024-09-16T04:04:33.450 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.442+0000 7fd137e64640 1 -- start start 2024-09-16T04:04:33.450 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.442+0000 7fd137e64640 10 monclient: init 2024-09-16T04:04:33.450 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.442+0000 7fd137e64640 10 monclient: _reopen_session rank -1 2024-09-16T04:04:33.450 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.442+0000 7fd137e64640 10 monclient: _add_conns ranks=[3,0,1,4,2] 2024-09-16T04:04:33.450 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.442+0000 7fd137e64640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd13006db90 0x7fd1301340c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:04:33.450 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.442+0000 7fd137e64640 10 monclient(hunting): picked mon.noname-a con 0x7fd13006db90 addr v2:172.21.15.42:3300/0 2024-09-16T04:04:33.450 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.442+0000 7fd137e64640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd1300a37f0 addr v1:172.21.15.42:6789/0 2024-09-16T04:04:33.450 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.442+0000 7fd137e64640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd1300a3c00 addr v1:172.21.15.154:6789/0 2024-09-16T04:04:33.450 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.442+0000 7fd137e64640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd13006ece0 con 0x7fd1300a37f0 2024-09-16T04:04:33.450 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.442+0000 7fd137e64640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd13009bf80 con 0x7fd1300a3c00 2024-09-16T04:04:33.450 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.442+0000 7fd137e64640 10 monclient(hunting): start opening mon connection 2024-09-16T04:04:33.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.442+0000 7fd137e64640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7fd13006df70 con 0x7fd13006db90 2024-09-16T04:04:33.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.442+0000 7fd137e64640 10 monclient(hunting): _renew_subs 2024-09-16T04:04:33.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.442+0000 7fd137e64640 10 monclient(hunting): authenticate will time out at 2320.415589s 2024-09-16T04:04:33.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.442+0000 7fd135c00640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd13006db90 0x7fd1301340c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:33.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.442+0000 7fd135c00640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd13006db90 0x7fd1301340c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:39812/0 (socket says 172.21.15.42:39812) 2024-09-16T04:04:33.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.442+0000 7fd135c00640 1 -- 172.21.15.42:0/2501910923 learned_addr learned my addr 172.21.15.42:0/2501910923 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:04:33.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.442+0000 7fd135c00640 10 monclient(hunting): get_auth_request con 0x7fd13006db90 auth_method 0 2024-09-16T04:04:33.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.442+0000 7fd135c00640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:04:33.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.442+0000 7fd135c00640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:04:33.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.442+0000 7fd135c00640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:04:33.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.442+0000 7fd1353ff640 1 --1- 172.21.15.42:0/2501910923 >> v1:172.21.15.42:6789/0 conn(0x7fd1300a37f0 0x7fd130136e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:39756/0 (socket says 172.21.15.42:39756) 2024-09-16T04:04:33.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.442+0000 7fd135c00640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:04:33.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.442+0000 7fd135c00640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:04:33.451 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.442+0000 7fd135c00640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:04:33.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.446+0000 7fd135c00640 10 monclient(hunting): handle_auth_done global_id 984180 payload 274 2024-09-16T04:04:33.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.446+0000 7fd135c00640 1 -- 172.21.15.42:0/2501910923 >> v1:172.21.15.154:6789/0 conn(0x7fd1300a3c00 legacy=0x7fd130137520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:04:33.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.446+0000 7fd135c00640 1 -- 172.21.15.42:0/2501910923 >> v1:172.21.15.42:6789/0 conn(0x7fd1300a37f0 legacy=0x7fd130136e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:04:33.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.446+0000 7fd135c00640 10 monclient: _finish_hunting 0 2024-09-16T04:04:33.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.446+0000 7fd135c00640 1 monclient: found mon.noname-a 2024-09-16T04:04:33.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.446+0000 7fd135c00640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:04:33.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.446+0000 7fd135c00640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:04:33.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.446+0000 7fd135c00640 1 -- 172.21.15.42:0/2501910923 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd130137c30 con 0x7fd13006db90 2024-09-16T04:04:33.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.446+0000 7fd135c00640 1 --2- 172.21.15.42:0/2501910923 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd13006db90 0x7fd1301340c0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fd12c009870 tx=0x7fd12c01bdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca97e815cbaf7703 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:04:33.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.446+0000 7fd134bfe640 1 -- 172.21.15.42:0/2501910923 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd12c0a8680 con 0x7fd13006db90 2024-09-16T04:04:33.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.446+0000 7fd134bfe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:04:33.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.446+0000 7fd134bfe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:04:33.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.446+0000 7fd134bfe640 10 monclient: dump: 2024-09-16T04:04:33.458 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:04:33.459 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:04:33.459 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:33.459 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:33.459 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:04:33.459 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:04:33.459 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:04:33.459 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:04:33.459 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:04:33.459 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:04:33.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.446+0000 7fd134bfe640 10 monclient: _finish_auth 0 2024-09-16T04:04:33.460 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.446+0000 7fd134bfe640 10 monclient: _check_auth_tickets 2024-09-16T04:04:33.460 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.446+0000 7fd134bfe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:04:33.460 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.446+0000 7fd134bfe640 1 -- 172.21.15.42:0/2501910923 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd12c0a8820 con 0x7fd13006db90 2024-09-16T04:04:33.460 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.446+0000 7fd137e64640 5 monclient: authenticate success, global_id 984180 2024-09-16T04:04:33.460 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.446+0000 7fd137e64640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:04:33.460 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.446+0000 7fd134bfe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:04:33.460 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.446+0000 7fd137e64640 10 monclient: get_monmap_and_config success 2024-09-16T04:04:33.460 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.446+0000 7fd137e64640 4 set_mon_vals no callback set 2024-09-16T04:04:33.460 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.446+0000 7fd134bfe640 1 -- 172.21.15.42:0/2501910923 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd12c0a8bf0 con 0x7fd13006db90 2024-09-16T04:04:33.460 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.446+0000 7fd134bfe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:04:33.460 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.446+0000 7fd134bfe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:04:33.460 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.446+0000 7fd134bfe640 10 monclient: dump: 2024-09-16T04:04:33.460 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:04:33.460 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:04:33.461 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:33.461 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:33.461 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:04:33.461 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:04:33.461 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:04:33.461 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:04:33.461 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:04:33.461 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:04:33.461 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.446+0000 7fd137e64640 10 monclient: shutdown 2024-09-16T04:04:33.461 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.446+0000 7fd137e64640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:04:33.462 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.446+0000 7fd137e64640 1 -- 172.21.15.42:0/2501910923 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd13006db90 msgr2=0x7fd1301340c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:04:33.462 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.446+0000 7fd137e64640 1 --2- 172.21.15.42:0/2501910923 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd13006db90 0x7fd1301340c0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fd12c009870 tx=0x7fd12c01bdd0 comp rx=0 tx=0).stop 2024-09-16T04:04:33.462 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.446+0000 7fd137e64640 1 -- 172.21.15.42:0/2501910923 shutdown_connections 2024-09-16T04:04:33.462 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.446+0000 7fd137e64640 1 --2- 172.21.15.42:0/2501910923 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd13006db90 0x7fd1301340c0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:33.462 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.446+0000 7fd137e64640 1 -- 172.21.15.42:0/2501910923 >> 172.21.15.42:0/2501910923 conn(0x7fd1300a9fb0 msgr2=0x7fd1300aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:04:33.462 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.446+0000 7fd137e64640 1 -- 172.21.15.42:0/2501910923 shutdown_connections 2024-09-16T04:04:33.462 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.446+0000 7fd137e64640 1 -- 172.21.15.42:0/2501910923 wait complete. 2024-09-16T04:04:33.462 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.446+0000 7fd137e64640 10 monclient: build_initial_monmap 2024-09-16T04:04:33.462 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.446+0000 7fd137e64640 10 monclient: monmap: 2024-09-16T04:04:33.462 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:04:33.462 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:04:33.462 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:04:33.462 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:04:33.462 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:04:33.463 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:04:33.463 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:04:33.463 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:04:33.463 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:04:33.463 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:04:33.463 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.446+0000 7fd137e64640 1 librados: starting msgr at 2024-09-16T04:04:33.463 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.446+0000 7fd137e64640 1 librados: starting objecter 2024-09-16T04:04:33.463 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.446+0000 7fd137e64640 1 Processor -- start 2024-09-16T04:04:33.463 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.450+0000 7fd137e64640 1 -- start start 2024-09-16T04:04:33.463 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.450+0000 7fd137e64640 1 librados: setting wanted keys 2024-09-16T04:04:33.463 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.450+0000 7fd137e64640 1 librados: calling monclient init 2024-09-16T04:04:33.463 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.450+0000 7fd137e64640 10 monclient: init 2024-09-16T04:04:33.463 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.450+0000 7fd137e64640 10 monclient: _reopen_session rank -1 2024-09-16T04:04:33.463 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.450+0000 7fd137e64640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:04:33.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.450+0000 7fd137e64640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd13006db90 0x7fd1300a31e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:04:33.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.450+0000 7fd137e64640 10 monclient(hunting): picked mon.noname-a con 0x7fd13006db90 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:04:33.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.450+0000 7fd137e64640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd1300a37f0 0x7fd13009db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:04:33.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.450+0000 7fd137e64640 10 monclient(hunting): picked mon.noname-c con 0x7fd1300a37f0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:04:33.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.450+0000 7fd137e64640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd1300a3c00 0x7fd13009e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:04:33.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.450+0000 7fd137e64640 10 monclient(hunting): picked mon.noname-b con 0x7fd1300a3c00 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:04:33.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.450+0000 7fd137e64640 10 monclient(hunting): start opening mon connection 2024-09-16T04:04:33.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.450+0000 7fd137e64640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fd130143a60 con 0x7fd13006db90 2024-09-16T04:04:33.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.450+0000 7fd137e64640 10 monclient(hunting): start opening mon connection 2024-09-16T04:04:33.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.450+0000 7fd137e64640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd1301438e0 con 0x7fd1300a3c00 2024-09-16T04:04:33.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.450+0000 7fd137e64640 10 monclient(hunting): start opening mon connection 2024-09-16T04:04:33.465 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.450+0000 7fd137e64640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fd130143be0 con 0x7fd1300a37f0 2024-09-16T04:04:33.465 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.450+0000 7fd137e64640 10 monclient(hunting): _renew_subs 2024-09-16T04:04:33.465 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.450+0000 7fd137e64640 10 monclient(hunting): authenticate will time out at 2320.420477s 2024-09-16T04:04:33.465 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.450+0000 7fd1353ff640 1 -- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd1300a37f0 msgr2=0x7fd13009db70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T04:04:33.465 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.450+0000 7fd1353ff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd1300a37f0 0x7fd13009db70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:04:33.465 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.450+0000 7fd135c00640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd13006db90 0x7fd1300a31e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:33.465 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.450+0000 7fd135c00640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd13006db90 0x7fd1300a31e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:39826/0 (socket says 172.21.15.42:39826) 2024-09-16T04:04:33.465 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.450+0000 7fd135c00640 1 -- 172.21.15.42:0/513890914 learned_addr learned my addr 172.21.15.42:0/513890914 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:04:33.465 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.450+0000 7fd135c00640 10 monclient(hunting): get_auth_request con 0x7fd13006db90 auth_method 0 2024-09-16T04:04:33.465 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.450+0000 7fd135c00640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:04:33.465 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.450+0000 7fd135c00640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:04:33.465 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.450+0000 7fd135c00640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:04:33.465 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.450+0000 7fd135c00640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:04:33.465 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.450+0000 7fd135c00640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:04:33.466 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.450+0000 7fd135c00640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:04:33.466 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.450+0000 7fd135c00640 10 monclient(hunting): handle_auth_done global_id 984186 payload 824 2024-09-16T04:04:33.466 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.450+0000 7fd136401640 1 --2- 172.21.15.42:0/513890914 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd1300a3c00 0x7fd13009e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:33.466 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.450+0000 7fd135c00640 1 -- 172.21.15.42:0/513890914 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd1300a37f0 msgr2=0x7fd13009db70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:04:33.466 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.450+0000 7fd135c00640 1 --2- 172.21.15.42:0/513890914 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd1300a37f0 0x7fd13009db70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:33.466 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.450+0000 7fd135c00640 1 -- 172.21.15.42:0/513890914 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd1300a3c00 msgr2=0x7fd13009e0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:04:33.466 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.450+0000 7fd135c00640 1 --2- 172.21.15.42:0/513890914 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd1300a3c00 0x7fd13009e0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:33.466 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.450+0000 7fd135c00640 10 monclient: _finish_hunting 0 2024-09-16T04:04:33.466 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.450+0000 7fd135c00640 1 monclient: found mon.noname-a 2024-09-16T04:04:33.466 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.450+0000 7fd135c00640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:04:33.466 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.450+0000 7fd135c00640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:04:33.466 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.450+0000 7fd135c00640 1 -- 172.21.15.42:0/513890914 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1300a19f0 con 0x7fd13006db90 2024-09-16T04:04:33.466 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.450+0000 7fd135c00640 1 --2- 172.21.15.42:0/513890914 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd13006db90 0x7fd1300a31e0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fd12c01a480 tx=0x7fd12c018ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:04:33.467 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.450+0000 7fd1267fc640 10 client.?.objecter ms_handle_connect 0x7fd13006db90 2024-09-16T04:04:33.467 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.450+0000 7fd1267fc640 10 client.?.objecter resend_mon_ops 2024-09-16T04:04:33.467 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.450+0000 7fd1267fc640 1 -- 172.21.15.42:0/513890914 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd12c0b08c0 con 0x7fd13006db90 2024-09-16T04:04:33.467 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.450+0000 7fd1267fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:04:33.467 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.450+0000 7fd1267fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:04:33.467 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.450+0000 7fd1267fc640 10 monclient: dump: 2024-09-16T04:04:33.467 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:04:33.467 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:04:33.467 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:33.467 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:33.467 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:04:33.467 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:04:33.467 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:04:33.468 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:04:33.468 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:04:33.468 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:04:33.468 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.450+0000 7fd1267fc640 10 monclient: _finish_auth 0 2024-09-16T04:04:33.468 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.450+0000 7fd1267fc640 10 monclient: _check_auth_tickets 2024-09-16T04:04:33.468 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.450+0000 7fd1267fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:04:33.468 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.450+0000 7fd1267fc640 1 -- 172.21.15.42:0/513890914 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd12c0b0a80 con 0x7fd13006db90 2024-09-16T04:04:33.468 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.450+0000 7fd137e64640 5 monclient: authenticate success, global_id 984186 2024-09-16T04:04:33.468 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.450+0000 7fd137e64640 10 monclient: _renew_subs 2024-09-16T04:04:33.468 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.450+0000 7fd137e64640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:04:33.468 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.450+0000 7fd137e64640 1 -- 172.21.15.42:0/513890914 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1300a8c60 con 0x7fd13006db90 2024-09-16T04:04:33.468 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.450+0000 7fd1267fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:04:33.468 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.450+0000 7fd1267fc640 1 -- 172.21.15.42:0/513890914 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd12c0ba3f0 con 0x7fd13006db90 2024-09-16T04:04:33.469 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.450+0000 7fd1267fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:04:33.469 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.450+0000 7fd1267fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:04:33.469 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.450+0000 7fd1267fc640 10 monclient: dump: 2024-09-16T04:04:33.469 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:04:33.469 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:04:33.469 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:33.469 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:33.469 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:04:33.469 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:04:33.469 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:04:33.469 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:04:33.469 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:04:33.469 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:04:33.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.450+0000 7fd137e64640 10 client.984186.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:04:33.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.450+0000 7fd137e64640 10 monclient: _renew_subs 2024-09-16T04:04:33.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.450+0000 7fd127fff640 4 set_mon_vals no callback set 2024-09-16T04:04:33.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.450+0000 7fd137e64640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:04:33.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.450+0000 7fd137e64640 1 -- 172.21.15.42:0/513890914 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1300a1430 con 0x7fd13006db90 2024-09-16T04:04:33.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.450+0000 7fd137e64640 1 librados: init done 2024-09-16T04:04:33.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.450+0000 7fd137e64640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:04:33.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.450+0000 7fd137e64640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:04:33.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.450+0000 7fd137e64640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:04:33.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.450+0000 7fd137e64640 1 -- 172.21.15.42:0/513890914 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd1301340c0 con 0x7fd13006db90 2024-09-16T04:04:33.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.450+0000 7fd1267fc640 1 -- 172.21.15.42:0/513890914 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fd12c0a6070 con 0x7fd13006db90 2024-09-16T04:04:33.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.450+0000 7fd1267fc640 1 --2- 172.21.15.42:0/513890914 >> v2:172.21.15.154:6812/254845342 conn(0x7fd1140377c0 0x7fd114039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:04:33.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.450+0000 7fd1267fc640 1 -- 172.21.15.42:0/513890914 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(681..681 src has 451..681) ==== 6280+0+0 (secure 0 0 0) 0x7fd12c0e2a10 con 0x7fd13006db90 2024-09-16T04:04:33.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.450+0000 7fd1267fc640 10 client.984186.objecter ms_dispatch 0x7fd130000ba0 osd_map(681..681 src has 451..681) 2024-09-16T04:04:33.471 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.450+0000 7fd1267fc640 3 client.984186.objecter handle_osd_map got epochs [681,681] > 0 2024-09-16T04:04:33.471 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.450+0000 7fd1267fc640 3 client.984186.objecter handle_osd_map decoding full epoch 681 2024-09-16T04:04:33.471 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.450+0000 7fd1267fc640 20 client.984186.objecter dump_active .. 0 homeless 2024-09-16T04:04:33.471 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.450+0000 7fd1353ff640 1 --2- 172.21.15.42:0/513890914 >> v2:172.21.15.154:6812/254845342 conn(0x7fd1140377c0 0x7fd114039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:33.471 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.450+0000 7fd1353ff640 10 monclient: get_auth_request con 0x7fd1140377c0 auth_method 0 2024-09-16T04:04:33.471 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.450+0000 7fd1353ff640 1 --2- 172.21.15.42:0/513890914 >> v2:172.21.15.154:6812/254845342 conn(0x7fd1140377c0 0x7fd114039c80 secure :-1 s=READY pgs=3830 cs=0 l=1 rev1=1 crypto rx=0x7fd13009f340 tx=0x7fd120007a50 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:04:33.471 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.450+0000 7fd1267fc640 10 client.984186.objecter ms_handle_connect 0x7fd1140377c0 2024-09-16T04:04:33.471 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.454+0000 7fd1267fc640 1 -- 172.21.15.42:0/513890914 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7fd12c0ac020 con 0x7fd13006db90 2024-09-16T04:04:33.471 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.454+0000 7fd1267fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:04:33.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.454+0000 7fd1267fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:04:33.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.678+0000 7fd137e64640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:04:33.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.678+0000 7fd137e64640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:04:33.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.678+0000 7fd137e64640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:04:33.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.678+0000 7fd137e64640 1 -- 172.21.15.42:0/513890914 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fd13009fb40 con 0x7fd13006db90 2024-09-16T04:04:33.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.678+0000 7fd1267fc640 1 -- 172.21.15.42:0/513890914 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7fd13009bf80 con 0x7fd13006db90 2024-09-16T04:04:33.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.678+0000 7fd1267fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:04:33.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.678+0000 7fd1267fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:04:33.684 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":210,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:04:33.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.678+0000 7fd137e64640 10 librados: watch_flush enter 2024-09-16T04:04:33.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.678+0000 7fd137e64640 10 librados: watch_flush exit 2024-09-16T04:04:33.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.678+0000 7fd137e64640 20 client.984186.objecter shutdown clearing up homeless session... 2024-09-16T04:04:33.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.678+0000 7fd137e64640 10 client.984186.objecter successfully canceled tick 2024-09-16T04:04:33.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.678+0000 7fd137e64640 1 -- 172.21.15.42:0/513890914 >> v2:172.21.15.154:6812/254845342 conn(0x7fd1140377c0 msgr2=0x7fd114039c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:04:33.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.678+0000 7fd137e64640 1 --2- 172.21.15.42:0/513890914 >> v2:172.21.15.154:6812/254845342 conn(0x7fd1140377c0 0x7fd114039c80 secure :-1 s=READY pgs=3830 cs=0 l=1 rev1=1 crypto rx=0x7fd13009f340 tx=0x7fd120007a50 comp rx=0 tx=0).stop 2024-09-16T04:04:33.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.678+0000 7fd137e64640 10 monclient: shutdown 2024-09-16T04:04:33.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.678+0000 7fd137e64640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:04:33.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.678+0000 7fd137e64640 1 -- 172.21.15.42:0/513890914 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd13006db90 msgr2=0x7fd1300a31e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:04:33.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.678+0000 7fd137e64640 1 --2- 172.21.15.42:0/513890914 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd13006db90 0x7fd1300a31e0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fd12c01a480 tx=0x7fd12c018ec0 comp rx=0 tx=0).stop 2024-09-16T04:04:33.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.678+0000 7fd137e64640 1 -- 172.21.15.42:0/513890914 shutdown_connections 2024-09-16T04:04:33.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.682+0000 7fd137e64640 1 --2- 172.21.15.42:0/513890914 >> v2:172.21.15.154:6812/254845342 conn(0x7fd1140377c0 0x7fd114039c80 unknown :-1 s=CLOSED pgs=3830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:33.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.682+0000 7fd137e64640 1 --2- 172.21.15.42:0/513890914 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd1300a3c00 0x7fd13009e0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:33.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.682+0000 7fd137e64640 1 --2- 172.21.15.42:0/513890914 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd1300a37f0 0x7fd13009db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:33.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.682+0000 7fd137e64640 1 --2- 172.21.15.42:0/513890914 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd13006db90 0x7fd1300a31e0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:33.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.682+0000 7fd137e64640 1 -- 172.21.15.42:0/513890914 >> 172.21.15.42:0/513890914 conn(0x7fd1300a9fb0 msgr2=0x7fd1300a7a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:04:33.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.682+0000 7fd137e64640 1 -- 172.21.15.42:0/513890914 shutdown_connections 2024-09-16T04:04:33.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.682+0000 7fd137e64640 1 -- 172.21.15.42:0/513890914 wait complete. 2024-09-16T04:04:33.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:33.682+0000 7fd137e64640 1 librados: shutdown 2024-09-16T04:04:36.701 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:04:36.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.826+0000 7fe00fcf1640 10 monclient: get_monmap_and_config 2024-09-16T04:04:36.832 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.826+0000 7fe00fcf1640 10 monclient: build_initial_monmap 2024-09-16T04:04:36.832 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.826+0000 7fe00fcf1640 10 monclient: monmap: 2024-09-16T04:04:36.832 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:04:36.832 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:04:36.832 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:04:36.830301+0000 2024-09-16T04:04:36.832 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:04:36.830301+0000 2024-09-16T04:04:36.832 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:04:36.832 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:04:36.832 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:04:36.832 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:04:36.832 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:04:36.832 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:04:36.833 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:04:36.833 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:04:36.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.826+0000 7fe00fcf1640 1 Processor -- start 2024-09-16T04:04:36.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.826+0000 7fe00fcf1640 1 -- start start 2024-09-16T04:04:36.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.826+0000 7fe00fcf1640 10 monclient: init 2024-09-16T04:04:36.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.826+0000 7fe00fcf1640 10 monclient: _reopen_session rank -1 2024-09-16T04:04:36.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.826+0000 7fe00fcf1640 10 monclient: _add_conns ranks=[3,1,4,2,0] 2024-09-16T04:04:36.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.826+0000 7fe00fcf1640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe0080a1330 0x7fe00809f180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:04:36.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.826+0000 7fe00fcf1640 10 monclient(hunting): picked mon.noname-a con 0x7fe0080a1330 addr v2:172.21.15.42:3300/0 2024-09-16T04:04:36.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.826+0000 7fe00fcf1640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe0080a1710 addr v1:172.21.15.154:6789/0 2024-09-16T04:04:36.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.826+0000 7fe00fcf1640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fe0081393a0 0x7fe00813b790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:04:36.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.826+0000 7fe00fcf1640 10 monclient(hunting): picked mon.noname-c con 0x7fe0081393a0 addr v2:172.21.15.154:3300/0 2024-09-16T04:04:36.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.826+0000 7fe00fcf1640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe00809e4c0 con 0x7fe0080a1710 2024-09-16T04:04:36.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.826+0000 7fe00fcf1640 10 monclient(hunting): start opening mon connection 2024-09-16T04:04:36.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.826+0000 7fe00fcf1640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7fe00809edf0 con 0x7fe0080a1330 2024-09-16T04:04:36.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.826+0000 7fe00fcf1640 10 monclient(hunting): start opening mon connection 2024-09-16T04:04:36.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.826+0000 7fe00fcf1640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fe00809cd60 con 0x7fe0081393a0 2024-09-16T04:04:36.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.826+0000 7fe00fcf1640 10 monclient(hunting): _renew_subs 2024-09-16T04:04:36.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.826+0000 7fe00fcf1640 10 monclient(hunting): authenticate will time out at 2323.799659s 2024-09-16T04:04:36.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.826+0000 7fe00da8d640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe0080a1330 0x7fe00809f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:36.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.826+0000 7fe00da8d640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe0080a1330 0x7fe00809f180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:53686/0 (socket says 172.21.15.42:53686) 2024-09-16T04:04:36.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.826+0000 7fe00da8d640 1 -- 172.21.15.42:0/1423264076 learned_addr learned my addr 172.21.15.42:0/1423264076 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:04:36.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.826+0000 7fe00da8d640 10 monclient(hunting): get_auth_request con 0x7fe0080a1330 auth_method 0 2024-09-16T04:04:36.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.826+0000 7fe00da8d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:04:36.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.826+0000 7fe00da8d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:04:36.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.826+0000 7fe00da8d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:04:36.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.826+0000 7fe00da8d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:04:36.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.826+0000 7fe00da8d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:04:36.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.826+0000 7fe00da8d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:04:36.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.830+0000 7fe00e28e640 1 --2- 172.21.15.42:0/1423264076 >> v2:172.21.15.154:3300/0 conn(0x7fe0081393a0 0x7fe00813b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:36.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.830+0000 7fe00da8d640 10 monclient(hunting): handle_auth_done global_id 984189 payload 274 2024-09-16T04:04:36.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.830+0000 7fe00da8d640 1 -- 172.21.15.42:0/1423264076 >> v2:172.21.15.154:3300/0 conn(0x7fe0081393a0 msgr2=0x7fe00813b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:04:36.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.830+0000 7fe00da8d640 1 --2- 172.21.15.42:0/1423264076 >> v2:172.21.15.154:3300/0 conn(0x7fe0081393a0 0x7fe00813b790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:36.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.830+0000 7fe00da8d640 1 -- 172.21.15.42:0/1423264076 >> v1:172.21.15.154:6789/0 conn(0x7fe0080a1710 legacy=0x7fe0080a1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:04:36.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.830+0000 7fe00da8d640 10 monclient: _finish_hunting 0 2024-09-16T04:04:36.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.830+0000 7fe00da8d640 1 monclient: found mon.noname-a 2024-09-16T04:04:36.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.830+0000 7fe00da8d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:04:36.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.830+0000 7fe00da8d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:04:36.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.830+0000 7fe00da8d640 1 -- 172.21.15.42:0/1423264076 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe00809f6c0 con 0x7fe0080a1330 2024-09-16T04:04:36.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.830+0000 7fe00e28e640 10 monclient: get_auth_request con 0x7fe0081393a0 auth_method 0 2024-09-16T04:04:36.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.830+0000 7fe00e28e640 1 --2- 172.21.15.42:0/1423264076 >> v2:172.21.15.154:3300/0 conn(0x7fe0081393a0 0x7fe00813b790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T04:04:36.843 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.830+0000 7fe00da8d640 1 --2- 172.21.15.42:0/1423264076 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe0080a1330 0x7fe00809f180 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fe00809df50 tx=0x7fe00401ea70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4fbc674c75541f0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:04:36.843 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.830+0000 7fe00ca8b640 1 -- 172.21.15.42:0/1423264076 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe00400b2d0 con 0x7fe0080a1330 2024-09-16T04:04:36.843 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.830+0000 7fe00ca8b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:04:36.843 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.830+0000 7fe00ca8b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:04:36.843 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.830+0000 7fe00ca8b640 10 monclient: dump: 2024-09-16T04:04:36.843 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:04:36.843 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:04:36.843 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:36.843 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:36.844 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:04:36.844 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:04:36.844 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:04:36.844 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:04:36.844 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:04:36.844 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:04:36.844 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.830+0000 7fe00ca8b640 10 monclient: _finish_auth 0 2024-09-16T04:04:36.844 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.830+0000 7fe00ca8b640 10 monclient: _check_auth_tickets 2024-09-16T04:04:36.844 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.830+0000 7fe00ca8b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:04:36.844 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.830+0000 7fe00ca8b640 1 -- 172.21.15.42:0/1423264076 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe00400b470 con 0x7fe0080a1330 2024-09-16T04:04:36.844 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.830+0000 7fe00ca8b640 10 monclient: handle_config config(0 keys) 2024-09-16T04:04:36.845 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.830+0000 7fe00ca8b640 1 -- 172.21.15.42:0/1423264076 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0040a7970 con 0x7fe0080a1330 2024-09-16T04:04:36.845 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.830+0000 7fe00fcf1640 5 monclient: authenticate success, global_id 984189 2024-09-16T04:04:36.845 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.830+0000 7fe00fcf1640 10 monclient: get_monmap_and_config success 2024-09-16T04:04:36.845 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.830+0000 7fe00fcf1640 4 set_mon_vals no callback set 2024-09-16T04:04:36.845 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.830+0000 7fe00ca8b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:04:36.845 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.830+0000 7fe00ca8b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:04:36.845 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.830+0000 7fe00ca8b640 10 monclient: dump: 2024-09-16T04:04:36.845 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:04:36.845 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:04:36.845 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:36.845 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:36.845 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:04:36.845 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:04:36.846 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:04:36.846 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:04:36.846 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:04:36.846 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:04:36.846 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.830+0000 7fe00fcf1640 10 monclient: shutdown 2024-09-16T04:04:36.846 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.830+0000 7fe00fcf1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:04:36.846 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.830+0000 7fe00fcf1640 1 -- 172.21.15.42:0/1423264076 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe0080a1330 msgr2=0x7fe00809f180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:04:36.846 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.830+0000 7fe00fcf1640 1 --2- 172.21.15.42:0/1423264076 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe0080a1330 0x7fe00809f180 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fe00809df50 tx=0x7fe00401ea70 comp rx=0 tx=0).stop 2024-09-16T04:04:36.846 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.830+0000 7fe00fcf1640 1 -- 172.21.15.42:0/1423264076 shutdown_connections 2024-09-16T04:04:36.846 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.830+0000 7fe00fcf1640 1 --2- 172.21.15.42:0/1423264076 >> v2:172.21.15.154:3300/0 conn(0x7fe0081393a0 0x7fe00813b790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:36.846 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.830+0000 7fe00fcf1640 1 --2- 172.21.15.42:0/1423264076 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe0080a1330 0x7fe00809f180 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:36.846 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.830+0000 7fe00fcf1640 1 -- 172.21.15.42:0/1423264076 >> 172.21.15.42:0/1423264076 conn(0x7fe0080a2420 msgr2=0x7fe0080a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:04:36.846 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.830+0000 7fe00fcf1640 1 -- 172.21.15.42:0/1423264076 shutdown_connections 2024-09-16T04:04:36.847 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.830+0000 7fe00fcf1640 1 -- 172.21.15.42:0/1423264076 wait complete. 2024-09-16T04:04:36.847 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.830+0000 7fe00fcf1640 10 monclient: build_initial_monmap 2024-09-16T04:04:36.847 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.830+0000 7fe00fcf1640 10 monclient: monmap: 2024-09-16T04:04:36.847 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:04:36.847 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:04:36.847 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:04:36.847 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:04:36.847 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:04:36.847 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:04:36.847 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:04:36.847 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:04:36.847 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:04:36.847 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:04:36.848 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.834+0000 7fe00fcf1640 1 librados: starting msgr at 2024-09-16T04:04:36.848 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.834+0000 7fe00fcf1640 1 librados: starting objecter 2024-09-16T04:04:36.848 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.834+0000 7fe00fcf1640 1 Processor -- start 2024-09-16T04:04:36.848 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.834+0000 7fe00fcf1640 1 -- start start 2024-09-16T04:04:36.848 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.834+0000 7fe00fcf1640 1 librados: setting wanted keys 2024-09-16T04:04:36.848 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.834+0000 7fe00fcf1640 1 librados: calling monclient init 2024-09-16T04:04:36.848 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.834+0000 7fe00fcf1640 10 monclient: init 2024-09-16T04:04:36.848 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.834+0000 7fe00fcf1640 10 monclient: _reopen_session rank -1 2024-09-16T04:04:36.848 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.834+0000 7fe00fcf1640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:04:36.848 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.834+0000 7fe00fcf1640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe0080a1330 0x7fe008134520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:04:36.848 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.834+0000 7fe00fcf1640 10 monclient(hunting): picked mon.noname-a con 0x7fe0080a1330 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:04:36.848 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.834+0000 7fe00fcf1640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe0080a1710 0x7fe008134a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:04:36.848 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.834+0000 7fe00fcf1640 10 monclient(hunting): picked mon.noname-b con 0x7fe0080a1710 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:04:36.849 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.834+0000 7fe00fcf1640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe0081393a0 0x7fe008120380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:04:36.849 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.834+0000 7fe00fcf1640 10 monclient(hunting): picked mon.noname-c con 0x7fe0081393a0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:04:36.849 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.834+0000 7fe00fcf1640 10 monclient(hunting): start opening mon connection 2024-09-16T04:04:36.849 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.834+0000 7fe00fcf1640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fe00809c5f0 con 0x7fe0080a1330 2024-09-16T04:04:36.849 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.834+0000 7fe00fcf1640 10 monclient(hunting): start opening mon connection 2024-09-16T04:04:36.849 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.834+0000 7fe00fcf1640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe00809c470 con 0x7fe0080a1710 2024-09-16T04:04:36.849 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.834+0000 7fe00fcf1640 10 monclient(hunting): start opening mon connection 2024-09-16T04:04:36.849 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.834+0000 7fe00fcf1640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fe00809c770 con 0x7fe0081393a0 2024-09-16T04:04:36.849 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.834+0000 7fe00fcf1640 10 monclient(hunting): _renew_subs 2024-09-16T04:04:36.849 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.834+0000 7fe00fcf1640 10 monclient(hunting): authenticate will time out at 2323.804782s 2024-09-16T04:04:36.849 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.834+0000 7fe00e28e640 1 -- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe0081393a0 msgr2=0x7fe008120380 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T04:04:36.849 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.834+0000 7fe00e28e640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe0081393a0 0x7fe008120380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:04:36.849 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.834+0000 7fe00da8d640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe0080a1330 0x7fe008134520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:36.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.834+0000 7fe00da8d640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe0080a1330 0x7fe008134520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:53688/0 (socket says 172.21.15.42:53688) 2024-09-16T04:04:36.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.834+0000 7fe00da8d640 1 -- 172.21.15.42:0/3109084507 learned_addr learned my addr 172.21.15.42:0/3109084507 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:04:36.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.834+0000 7fe00da8d640 10 monclient(hunting): get_auth_request con 0x7fe0080a1330 auth_method 0 2024-09-16T04:04:36.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.834+0000 7fe00da8d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:04:36.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.834+0000 7fe00da8d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:04:36.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.834+0000 7fe00da8d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:04:36.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.834+0000 7fe00da8d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:04:36.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.834+0000 7fe00da8d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:04:36.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.834+0000 7fe00da8d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:04:36.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.834+0000 7fe00da8d640 10 monclient(hunting): handle_auth_done global_id 984192 payload 824 2024-09-16T04:04:36.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.834+0000 7fe00d28c640 1 --2- 172.21.15.42:0/3109084507 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe0080a1710 0x7fe008134a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:36.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.834+0000 7fe00da8d640 1 -- 172.21.15.42:0/3109084507 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe0081393a0 msgr2=0x7fe008120380 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:04:36.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.834+0000 7fe00da8d640 1 --2- 172.21.15.42:0/3109084507 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe0081393a0 0x7fe008120380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:36.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.834+0000 7fe00da8d640 1 -- 172.21.15.42:0/3109084507 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe0080a1710 msgr2=0x7fe008134a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:04:36.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.834+0000 7fe00da8d640 1 --2- 172.21.15.42:0/3109084507 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe0080a1710 0x7fe008134a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:36.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.834+0000 7fe00da8d640 10 monclient: _finish_hunting 0 2024-09-16T04:04:36.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.834+0000 7fe00da8d640 1 monclient: found mon.noname-a 2024-09-16T04:04:36.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.834+0000 7fe00da8d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:04:36.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.834+0000 7fe00da8d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:04:36.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.834+0000 7fe00da8d640 1 -- 172.21.15.42:0/3109084507 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe008134fa0 con 0x7fe0080a1330 2024-09-16T04:04:36.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.834+0000 7fe00da8d640 1 --2- 172.21.15.42:0/3109084507 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe0080a1330 0x7fe008134520 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fe00401c6a0 tx=0x7fe00401b810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:04:36.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.834+0000 7fdffe7fc640 10 client.?.objecter ms_handle_connect 0x7fe0080a1330 2024-09-16T04:04:36.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.834+0000 7fdffe7fc640 10 client.?.objecter resend_mon_ops 2024-09-16T04:04:36.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.834+0000 7fdffe7fc640 1 -- 172.21.15.42:0/3109084507 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe004009800 con 0x7fe0080a1330 2024-09-16T04:04:36.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.834+0000 7fdffe7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:04:36.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.834+0000 7fdffe7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:04:36.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.834+0000 7fdffe7fc640 10 monclient: dump: 2024-09-16T04:04:36.852 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:04:36.852 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:04:36.852 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:36.852 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:36.852 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:04:36.852 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:04:36.852 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:04:36.852 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:04:36.852 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:04:36.853 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:04:36.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.834+0000 7fdffe7fc640 10 monclient: _finish_auth 0 2024-09-16T04:04:36.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.834+0000 7fdffe7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:04:36.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.834+0000 7fdffe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:04:36.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.834+0000 7fdffe7fc640 1 -- 172.21.15.42:0/3109084507 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe0040be070 con 0x7fe0080a1330 2024-09-16T04:04:36.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.834+0000 7fe00fcf1640 5 monclient: authenticate success, global_id 984192 2024-09-16T04:04:36.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.834+0000 7fe00fcf1640 10 monclient: _renew_subs 2024-09-16T04:04:36.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.834+0000 7fe00fcf1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:04:36.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.834+0000 7fe00fcf1640 1 -- 172.21.15.42:0/3109084507 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe008122ef0 con 0x7fe0080a1330 2024-09-16T04:04:36.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.834+0000 7fdffe7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:04:36.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.834+0000 7fdffe7fc640 1 -- 172.21.15.42:0/3109084507 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe004009c20 con 0x7fe0080a1330 2024-09-16T04:04:36.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.834+0000 7fdffe7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:04:36.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.834+0000 7fdffe7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:04:36.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.834+0000 7fdffe7fc640 10 monclient: dump: 2024-09-16T04:04:36.854 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:04:36.854 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:04:36.854 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:36.854 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:36.854 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:04:36.854 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:04:36.854 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:04:36.854 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:04:36.854 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:04:36.854 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:04:36.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.834+0000 7fe00fcf1640 10 client.984192.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:04:36.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.834+0000 7fdfff7fe640 4 set_mon_vals no callback set 2024-09-16T04:04:36.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.834+0000 7fe00fcf1640 10 monclient: _renew_subs 2024-09-16T04:04:36.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.834+0000 7fe00fcf1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:04:36.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.834+0000 7fe00fcf1640 1 -- 172.21.15.42:0/3109084507 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe00813b410 con 0x7fe0080a1330 2024-09-16T04:04:36.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.834+0000 7fe00fcf1640 1 librados: init done 2024-09-16T04:04:36.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.834+0000 7fdffe7fc640 1 -- 172.21.15.42:0/3109084507 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fe0040b6070 con 0x7fe0080a1330 2024-09-16T04:04:36.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.834+0000 7fe00fcf1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:04:36.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.834+0000 7fe00fcf1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:04:36.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.834+0000 7fe00fcf1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:04:36.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.834+0000 7fe00fcf1640 1 -- 172.21.15.42:0/3109084507 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe00809f180 con 0x7fe0080a1330 2024-09-16T04:04:36.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.834+0000 7fdffe7fc640 1 --2- 172.21.15.42:0/3109084507 >> v2:172.21.15.154:6812/254845342 conn(0x7fdfec0377c0 0x7fdfec039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:04:36.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.834+0000 7fdffe7fc640 1 -- 172.21.15.42:0/3109084507 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(681..681 src has 451..681) ==== 6280+0+0 (secure 0 0 0) 0x7fe0040e39f0 con 0x7fe0080a1330 2024-09-16T04:04:36.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.834+0000 7fdffe7fc640 10 client.984192.objecter ms_dispatch 0x7fe008000ba0 osd_map(681..681 src has 451..681) 2024-09-16T04:04:36.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.834+0000 7fdffe7fc640 3 client.984192.objecter handle_osd_map got epochs [681,681] > 0 2024-09-16T04:04:36.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.834+0000 7fdffe7fc640 3 client.984192.objecter handle_osd_map decoding full epoch 681 2024-09-16T04:04:36.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.834+0000 7fdffe7fc640 20 client.984192.objecter dump_active .. 0 homeless 2024-09-16T04:04:36.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.834+0000 7fe00d28c640 1 --2- 172.21.15.42:0/3109084507 >> v2:172.21.15.154:6812/254845342 conn(0x7fdfec0377c0 0x7fdfec039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:36.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.834+0000 7fe00d28c640 10 monclient: get_auth_request con 0x7fdfec0377c0 auth_method 0 2024-09-16T04:04:36.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.838+0000 7fdffe7fc640 10 client.984192.objecter ms_handle_connect 0x7fdfec0377c0 2024-09-16T04:04:36.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.838+0000 7fe00d28c640 1 --2- 172.21.15.42:0/3109084507 >> v2:172.21.15.154:6812/254845342 conn(0x7fdfec0377c0 0x7fdfec039c80 secure :-1 s=READY pgs=3831 cs=0 l=1 rev1=1 crypto rx=0x7fdff8007fd0 tx=0x7fdff8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:04:36.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.838+0000 7fdffe7fc640 1 -- 172.21.15.42:0/3109084507 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7fe0040e5e20 con 0x7fe0080a1330 2024-09-16T04:04:36.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.838+0000 7fdffe7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:04:36.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:36.838+0000 7fdffe7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:04:37.070 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:37.066+0000 7fe00fcf1640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:04:37.070 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:37.066+0000 7fe00fcf1640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:04:37.070 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:37.066+0000 7fe00fcf1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:04:37.070 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:37.066+0000 7fe00fcf1640 1 -- 172.21.15.42:0/3109084507 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fe008069040 con 0x7fe0080a1330 2024-09-16T04:04:37.070 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:37.066+0000 7fdffe7fc640 1 -- 172.21.15.42:0/3109084507 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1562 (secure 0 0 0) 0x7fe0040c7020 con 0x7fe0080a1330 2024-09-16T04:04:37.070 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:37.066+0000 7fdffe7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:04:37.070 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:37.066+0000 7fdffe7fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:04:37.071 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":210,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:04:37.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:37.066+0000 7fe00fcf1640 10 librados: watch_flush enter 2024-09-16T04:04:37.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:37.066+0000 7fe00fcf1640 10 librados: watch_flush exit 2024-09-16T04:04:37.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:37.066+0000 7fe00fcf1640 20 client.984192.objecter shutdown clearing up homeless session... 2024-09-16T04:04:37.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:37.066+0000 7fe00fcf1640 10 client.984192.objecter successfully canceled tick 2024-09-16T04:04:37.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:37.066+0000 7fe00fcf1640 1 -- 172.21.15.42:0/3109084507 >> v2:172.21.15.154:6812/254845342 conn(0x7fdfec0377c0 msgr2=0x7fdfec039c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:04:37.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:37.066+0000 7fe00fcf1640 1 --2- 172.21.15.42:0/3109084507 >> v2:172.21.15.154:6812/254845342 conn(0x7fdfec0377c0 0x7fdfec039c80 secure :-1 s=READY pgs=3831 cs=0 l=1 rev1=1 crypto rx=0x7fdff8007fd0 tx=0x7fdff8009040 comp rx=0 tx=0).stop 2024-09-16T04:04:37.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:37.066+0000 7fe00fcf1640 10 monclient: shutdown 2024-09-16T04:04:37.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:37.066+0000 7fe00fcf1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:04:37.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:37.066+0000 7fe00fcf1640 1 -- 172.21.15.42:0/3109084507 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe0080a1330 msgr2=0x7fe008134520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:04:37.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:37.066+0000 7fe00fcf1640 1 --2- 172.21.15.42:0/3109084507 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe0080a1330 0x7fe008134520 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fe00401c6a0 tx=0x7fe00401b810 comp rx=0 tx=0).stop 2024-09-16T04:04:37.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:37.066+0000 7fe00fcf1640 1 -- 172.21.15.42:0/3109084507 shutdown_connections 2024-09-16T04:04:37.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:37.066+0000 7fe00fcf1640 1 --2- 172.21.15.42:0/3109084507 >> v2:172.21.15.154:6812/254845342 conn(0x7fdfec0377c0 0x7fdfec039c80 unknown :-1 s=CLOSED pgs=3831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:37.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:37.066+0000 7fe00fcf1640 1 --2- 172.21.15.42:0/3109084507 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe0081393a0 0x7fe008120380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:37.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:37.066+0000 7fe00fcf1640 1 --2- 172.21.15.42:0/3109084507 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe0080a1710 0x7fe008134a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:37.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:37.066+0000 7fe00fcf1640 1 --2- 172.21.15.42:0/3109084507 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe0080a1330 0x7fe008134520 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:37.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:37.066+0000 7fe00fcf1640 1 -- 172.21.15.42:0/3109084507 >> 172.21.15.42:0/3109084507 conn(0x7fe0080a2420 msgr2=0x7fe0080a46b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:04:37.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:37.066+0000 7fe00fcf1640 1 -- 172.21.15.42:0/3109084507 shutdown_connections 2024-09-16T04:04:37.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:37.066+0000 7fe00fcf1640 1 -- 172.21.15.42:0/3109084507 wait complete. 2024-09-16T04:04:37.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:37.066+0000 7fe00fcf1640 1 librados: shutdown 2024-09-16T04:04:40.092 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:04:40.227 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:40.222+0000 7f98d3a14640 10 monclient: get_monmap_and_config 2024-09-16T04:04:40.227 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:40.222+0000 7f98d3a14640 10 monclient: build_initial_monmap 2024-09-16T04:04:40.227 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:40.222+0000 7f98d3a14640 10 monclient: monmap: 2024-09-16T04:04:40.227 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:04:40.227 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:04:40.227 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:04:40.225431+0000 2024-09-16T04:04:40.227 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:04:40.225431+0000 2024-09-16T04:04:40.227 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:04:40.227 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:04:40.227 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:04:40.228 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:04:40.228 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:04:40.228 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:04:40.228 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:04:40.228 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:04:40.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:40.222+0000 7f98d3a14640 1 Processor -- start 2024-09-16T04:04:40.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:40.222+0000 7f98d3a14640 1 -- start start 2024-09-16T04:04:40.230 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:40.222+0000 7f98d3a14640 10 monclient: init 2024-09-16T04:04:40.230 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:40.222+0000 7f98d3a14640 10 monclient: _reopen_session rank -1 2024-09-16T04:04:40.230 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:40.222+0000 7f98d3a14640 10 monclient: _add_conns ranks=[4,1,2,3,0] 2024-09-16T04:04:40.230 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:40.222+0000 7f98d3a14640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f98cc0a1330 0x7f98cc09f180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:04:40.230 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:40.222+0000 7f98d3a14640 10 monclient(hunting): picked mon.noname-c con 0x7f98cc0a1330 addr v2:172.21.15.154:3300/0 2024-09-16T04:04:40.230 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:40.222+0000 7f98d3a14640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f98cc0a1710 addr v1:172.21.15.154:6789/0 2024-09-16T04:04:40.230 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:40.222+0000 7f98d3a14640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f98cc1393a0 0x7f98cc13b790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:04:40.230 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:40.222+0000 7f98d3a14640 10 monclient(hunting): picked mon.noname-b con 0x7f98cc1393a0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:04:40.230 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:40.222+0000 7f98d3a14640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f98cc09e4c0 con 0x7f98cc0a1710 2024-09-16T04:04:40.230 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:40.222+0000 7f98d3a14640 10 monclient(hunting): start opening mon connection 2024-09-16T04:04:40.230 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:40.222+0000 7f98d3a14640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f98cc09edf0 con 0x7f98cc0a1330 2024-09-16T04:04:40.230 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:40.222+0000 7f98d3a14640 10 monclient(hunting): start opening mon connection 2024-09-16T04:04:40.230 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:40.222+0000 7f98d3a14640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f98cc09cd60 con 0x7f98cc1393a0 2024-09-16T04:04:40.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:40.222+0000 7f98d3a14640 10 monclient(hunting): _renew_subs 2024-09-16T04:04:40.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:40.222+0000 7f98d3a14640 10 monclient(hunting): authenticate will time out at 2327.194892s 2024-09-16T04:04:40.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:40.222+0000 7f98d1fb1640 1 -- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f98cc1393a0 msgr2=0x7f98cc13b790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T04:04:40.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:40.222+0000 7f98d1fb1640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f98cc1393a0 0x7f98cc13b790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:04:40.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:40.222+0000 7f98d17b0640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f98cc0a1330 0x7f98cc09f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:40.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:40.222+0000 7f98d0faf640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f98cc0a1710 0x7f98cc0a1b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.42:46152/0 (socket says 172.21.15.42:46152) 2024-09-16T04:04:40.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:40.222+0000 7f98d0faf640 1 -- 172.21.15.42:0/3220479930 learned_addr learned my addr 172.21.15.42:0/3220479930 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T04:04:40.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:40.222+0000 7f98d17b0640 10 monclient(hunting): get_auth_request con 0x7f98cc0a1330 auth_method 0 2024-09-16T04:04:40.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:40.222+0000 7f98d17b0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:04:40.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:40.222+0000 7f98d17b0640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:04:40.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:40.222+0000 7f98d17b0640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:04:40.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:40.222+0000 7f98d17b0640 1 -- 172.21.15.42:0/3220479930 >> v2:172.21.15.154:3300/0 conn(0x7f98cc0a1330 msgr2=0x7f98cc09f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:04:40.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:40.222+0000 7f98d17b0640 1 -- 172.21.15.42:0/3220479930 >> v2:172.21.15.154:3300/0 conn(0x7f98cc0a1330 msgr2=0x7f98cc09f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:04:40.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:40.222+0000 7f98d17b0640 1 --2- 172.21.15.42:0/3220479930 >> v2:172.21.15.154:3300/0 conn(0x7f98cc0a1330 0x7f98cc09f180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:04:40.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:40.222+0000 7f98d17b0640 1 --2- 172.21.15.42:0/3220479930 >> v2:172.21.15.154:3300/0 conn(0x7f98cc0a1330 0x7f98cc09f180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:04:40.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:40.426+0000 7f98d1fb1640 1 -- 172.21.15.42:0/3220479930 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f98cc1393a0 msgr2=0x7f98cc13b790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T04:04:40.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:40.426+0000 7f98d1fb1640 1 --2- 172.21.15.42:0/3220479930 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f98cc1393a0 0x7f98cc13b790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T04:04:40.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:40.426+0000 7f98d17b0640 1 --2- 172.21.15.42:0/3220479930 >> v2:172.21.15.154:3300/0 conn(0x7f98cc0a1330 0x7f98cc09f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:40.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:40.426+0000 7f98d17b0640 10 monclient(hunting): get_auth_request con 0x7f98cc0a1330 auth_method 0 2024-09-16T04:04:40.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:40.426+0000 7f98d17b0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:04:40.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:40.426+0000 7f98d17b0640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:04:40.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:40.426+0000 7f98d17b0640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:04:40.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:40.426+0000 7f98d17b0640 1 -- 172.21.15.42:0/3220479930 >> v2:172.21.15.154:3300/0 conn(0x7f98cc0a1330 msgr2=0x7f98cc09f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:04:40.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:40.426+0000 7f98d17b0640 1 -- 172.21.15.42:0/3220479930 >> v2:172.21.15.154:3300/0 conn(0x7f98cc0a1330 msgr2=0x7f98cc09f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:04:40.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:40.426+0000 7f98d17b0640 1 --2- 172.21.15.42:0/3220479930 >> v2:172.21.15.154:3300/0 conn(0x7f98cc0a1330 0x7f98cc09f180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:04:40.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:40.426+0000 7f98d17b0640 1 --2- 172.21.15.42:0/3220479930 >> v2:172.21.15.154:3300/0 conn(0x7f98cc0a1330 0x7f98cc09f180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T04:04:40.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:40.830+0000 7f98d1fb1640 1 -- 172.21.15.42:0/3220479930 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f98cc1393a0 msgr2=0x7f98cc13b790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T04:04:40.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:40.830+0000 7f98d1fb1640 1 --2- 172.21.15.42:0/3220479930 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f98cc1393a0 0x7f98cc13b790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T04:04:40.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:40.830+0000 7f98d17b0640 1 --2- 172.21.15.42:0/3220479930 >> v2:172.21.15.154:3300/0 conn(0x7f98cc0a1330 0x7f98cc09f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:40.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:40.830+0000 7f98d17b0640 10 monclient(hunting): get_auth_request con 0x7f98cc0a1330 auth_method 0 2024-09-16T04:04:40.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:40.830+0000 7f98d17b0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:04:40.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:40.830+0000 7f98d17b0640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:04:40.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:40.830+0000 7f98d17b0640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:04:40.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:40.830+0000 7f98d17b0640 1 -- 172.21.15.42:0/3220479930 >> v2:172.21.15.154:3300/0 conn(0x7f98cc0a1330 msgr2=0x7f98cc09f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T04:04:40.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:40.830+0000 7f98d17b0640 1 -- 172.21.15.42:0/3220479930 >> v2:172.21.15.154:3300/0 conn(0x7f98cc0a1330 msgr2=0x7f98cc09f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:04:40.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:40.830+0000 7f98d17b0640 1 --2- 172.21.15.42:0/3220479930 >> v2:172.21.15.154:3300/0 conn(0x7f98cc0a1330 0x7f98cc09f180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:04:40.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:40.830+0000 7f98d17b0640 1 --2- 172.21.15.42:0/3220479930 >> v2:172.21.15.154:3300/0 conn(0x7f98cc0a1330 0x7f98cc09f180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T04:04:41.638 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:41.634+0000 7f98d1fb1640 1 -- 172.21.15.42:0/3220479930 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f98cc1393a0 msgr2=0x7f98cc13b790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T04:04:41.638 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:41.634+0000 7f98d1fb1640 1 --2- 172.21.15.42:0/3220479930 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f98cc1393a0 0x7f98cc13b790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T04:04:41.638 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:41.634+0000 7f98d17b0640 1 --2- 172.21.15.42:0/3220479930 >> v2:172.21.15.154:3300/0 conn(0x7f98cc0a1330 0x7f98cc09f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:41.638 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:41.634+0000 7f98d17b0640 10 monclient(hunting): get_auth_request con 0x7f98cc0a1330 auth_method 0 2024-09-16T04:04:41.638 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:41.634+0000 7f98d17b0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:04:41.638 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:41.634+0000 7f98d17b0640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:04:41.638 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:41.634+0000 7f98d17b0640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:04:41.639 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:41.634+0000 7f98d17b0640 1 -- 172.21.15.42:0/3220479930 >> v2:172.21.15.154:3300/0 conn(0x7f98cc0a1330 msgr2=0x7f98cc09f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T04:04:41.639 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:41.634+0000 7f98d17b0640 1 -- 172.21.15.42:0/3220479930 >> v2:172.21.15.154:3300/0 conn(0x7f98cc0a1330 msgr2=0x7f98cc09f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:04:41.639 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:41.634+0000 7f98d17b0640 1 --2- 172.21.15.42:0/3220479930 >> v2:172.21.15.154:3300/0 conn(0x7f98cc0a1330 0x7f98cc09f180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:04:41.639 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:41.634+0000 7f98d17b0640 1 --2- 172.21.15.42:0/3220479930 >> v2:172.21.15.154:3300/0 conn(0x7f98cc0a1330 0x7f98cc09f180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T04:04:43.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:43.222+0000 7f98beffd640 10 monclient(hunting): tick 2024-09-16T04:04:43.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:43.222+0000 7f98beffd640 10 monclient(hunting): _check_auth_tickets 2024-09-16T04:04:43.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:43.222+0000 7f98beffd640 1 monclient(hunting): continuing hunt 2024-09-16T04:04:43.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:43.222+0000 7f98beffd640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T04:04:43.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:43.222+0000 7f98beffd640 1 -- 172.21.15.42:0/3220479930 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f98cc1393a0 msgr2=0x7f98cc13b790 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:04:43.230 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:43.222+0000 7f98beffd640 1 --2- 172.21.15.42:0/3220479930 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f98cc1393a0 0x7f98cc13b790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:43.230 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:43.222+0000 7f98beffd640 1 -- 172.21.15.42:0/3220479930 >> v2:172.21.15.154:3300/0 conn(0x7f98cc0a1330 msgr2=0x7f98cc09f180 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:04:43.230 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:43.222+0000 7f98beffd640 1 --2- 172.21.15.42:0/3220479930 >> v2:172.21.15.154:3300/0 conn(0x7f98cc0a1330 0x7f98cc09f180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:43.230 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:43.222+0000 7f98beffd640 1 -- 172.21.15.42:0/3220479930 >> v1:172.21.15.154:6789/0 conn(0x7f98cc0a1710 legacy=0x7f98cc0a1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:04:43.230 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:43.222+0000 7f98beffd640 10 monclient: _add_conns ranks=[0,3] 2024-09-16T04:04:43.230 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:43.222+0000 7f98beffd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f98b00036f0 addr v1:172.21.15.42:6789/0 2024-09-16T04:04:43.230 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:43.222+0000 7f98beffd640 1 --2- 172.21.15.42:0/3220479930 >> v2:172.21.15.42:3300/0 conn(0x7f98b0007420 0x7f98b00098e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:04:43.230 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:43.222+0000 7f98beffd640 10 monclient(hunting): picked mon.noname-a con 0x7f98b0007420 addr v2:172.21.15.42:3300/0 2024-09-16T04:04:43.230 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:43.222+0000 7f98beffd640 1 -- 172.21.15.42:0/3220479930 --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f98cc09f6c0 con 0x7f98b00036f0 2024-09-16T04:04:43.230 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:43.222+0000 7f98beffd640 10 monclient(hunting): start opening mon connection 2024-09-16T04:04:43.230 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:43.222+0000 7f98beffd640 1 -- 172.21.15.42:0/3220479930 --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f98cc09edf0 con 0x7f98b0007420 2024-09-16T04:04:43.230 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:43.222+0000 7f98beffd640 10 monclient(hunting): _renew_subs 2024-09-16T04:04:43.230 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:43.222+0000 7f98d0faf640 1 --2- 172.21.15.42:0/3220479930 >> v2:172.21.15.42:3300/0 conn(0x7f98b0007420 0x7f98b00098e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:43.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:43.222+0000 7f98d0faf640 10 monclient(hunting): get_auth_request con 0x7f98b0007420 auth_method 0 2024-09-16T04:04:43.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:43.222+0000 7f98d0faf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:04:43.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:43.222+0000 7f98d0faf640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:04:43.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:43.222+0000 7f98d0faf640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:04:43.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:43.222+0000 7f98d0faf640 1 -- 172.21.15.42:0/3220479930 >> v2:172.21.15.42:3300/0 conn(0x7f98b0007420 msgr2=0x7f98b00098e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:04:43.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:43.222+0000 7f98d0faf640 1 -- 172.21.15.42:0/3220479930 >> v2:172.21.15.42:3300/0 conn(0x7f98b0007420 msgr2=0x7f98b00098e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:04:43.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:43.222+0000 7f98d0faf640 1 --2- 172.21.15.42:0/3220479930 >> v2:172.21.15.42:3300/0 conn(0x7f98b0007420 0x7f98b00098e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:04:43.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:43.222+0000 7f98d0faf640 1 --2- 172.21.15.42:0/3220479930 >> v2:172.21.15.42:3300/0 conn(0x7f98b0007420 0x7f98b00098e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:04:43.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:43.426+0000 7f98d0faf640 1 --2- 172.21.15.42:0/3220479930 >> v2:172.21.15.42:3300/0 conn(0x7f98b0007420 0x7f98b00098e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:43.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:43.426+0000 7f98d0faf640 10 monclient(hunting): get_auth_request con 0x7f98b0007420 auth_method 0 2024-09-16T04:04:43.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:43.426+0000 7f98d0faf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:04:43.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:43.426+0000 7f98d0faf640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:04:43.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:43.426+0000 7f98d0faf640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:04:43.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:43.426+0000 7f98d0faf640 1 -- 172.21.15.42:0/3220479930 >> v2:172.21.15.42:3300/0 conn(0x7f98b0007420 msgr2=0x7f98b00098e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:04:43.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:43.426+0000 7f98d0faf640 1 -- 172.21.15.42:0/3220479930 >> v2:172.21.15.42:3300/0 conn(0x7f98b0007420 msgr2=0x7f98b00098e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:04:43.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:43.426+0000 7f98d0faf640 1 --2- 172.21.15.42:0/3220479930 >> v2:172.21.15.42:3300/0 conn(0x7f98b0007420 0x7f98b00098e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:04:43.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:43.426+0000 7f98d0faf640 1 --2- 172.21.15.42:0/3220479930 >> v2:172.21.15.42:3300/0 conn(0x7f98b0007420 0x7f98b00098e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T04:04:43.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:43.830+0000 7f98d0faf640 1 --2- 172.21.15.42:0/3220479930 >> v2:172.21.15.42:3300/0 conn(0x7f98b0007420 0x7f98b00098e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:43.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:43.830+0000 7f98d0faf640 10 monclient(hunting): get_auth_request con 0x7f98b0007420 auth_method 0 2024-09-16T04:04:43.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:43.830+0000 7f98d0faf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:04:43.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:43.830+0000 7f98d0faf640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:04:43.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:43.830+0000 7f98d0faf640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:04:43.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:43.830+0000 7f98d0faf640 1 -- 172.21.15.42:0/3220479930 >> v2:172.21.15.42:3300/0 conn(0x7f98b0007420 msgr2=0x7f98b00098e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:04:43.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:43.830+0000 7f98d0faf640 1 -- 172.21.15.42:0/3220479930 >> v2:172.21.15.42:3300/0 conn(0x7f98b0007420 msgr2=0x7f98b00098e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:04:43.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:43.830+0000 7f98d0faf640 1 --2- 172.21.15.42:0/3220479930 >> v2:172.21.15.42:3300/0 conn(0x7f98b0007420 0x7f98b00098e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:04:43.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:43.830+0000 7f98d0faf640 1 --2- 172.21.15.42:0/3220479930 >> v2:172.21.15.42:3300/0 conn(0x7f98b0007420 0x7f98b00098e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T04:04:44.620 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.614+0000 7f98bffff640 1 -- 172.21.15.42:0/3220479930 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1204799557 0 0) 0x7f98cc09f6c0 con 0x7f98b00036f0 2024-09-16T04:04:44.620 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.614+0000 7f98bffff640 10 monclient(hunting): _negotiate 2024-09-16T04:04:44.620 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.614+0000 7f98bffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:04:44.620 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.614+0000 7f98bffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:04:44.620 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.614+0000 7f98bffff640 10 monclient(hunting): my global_id is 984195 2024-09-16T04:04:44.620 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.614+0000 7f98bffff640 1 -- 172.21.15.42:0/3220479930 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f98b8003530 con 0x7f98b00036f0 2024-09-16T04:04:44.625 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.618+0000 7f98bffff640 1 -- 172.21.15.42:0/3220479930 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1304475255 0 0) 0x7f98b8003530 con 0x7f98b00036f0 2024-09-16T04:04:44.625 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.618+0000 7f98bffff640 1 -- 172.21.15.42:0/3220479930 >> v2:172.21.15.42:3300/0 conn(0x7f98b0007420 msgr2=0x7f98b00098e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:04:44.625 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.618+0000 7f98bffff640 1 --2- 172.21.15.42:0/3220479930 >> v2:172.21.15.42:3300/0 conn(0x7f98b0007420 0x7f98b00098e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:44.625 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.618+0000 7f98bffff640 10 monclient: _finish_hunting 0 2024-09-16T04:04:44.625 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.618+0000 7f98bffff640 1 monclient: found mon.noname-a-legacy 2024-09-16T04:04:44.625 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.618+0000 7f98bffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:04:44.625 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.618+0000 7f98bffff640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.42:6789/0 2024-09-16T04:04:44.625 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.618+0000 7f98bffff640 1 -- 172.21.15.42:0/3220479930 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98b000afd0 con 0x7f98b00036f0 2024-09-16T04:04:44.625 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.618+0000 7f98bffff640 10 monclient: _finish_auth 0 2024-09-16T04:04:44.625 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.618+0000 7f98bffff640 10 monclient: _check_auth_tickets 2024-09-16T04:04:44.626 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.618+0000 7f98bffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:04:44.626 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.618+0000 7f98bffff640 1 -- 172.21.15.42:0/3220479930 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2177413734 0 0) 0x7f98c000f040 con 0x7f98b00036f0 2024-09-16T04:04:44.626 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.618+0000 7f98bffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:04:44.626 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.618+0000 7f98bffff640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T04:04:44.626 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.618+0000 7f98bffff640 10 monclient: dump: 2024-09-16T04:04:44.626 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:04:44.627 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:04:44.627 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:44.627 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:44.627 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:04:44.627 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:04:44.627 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:04:44.627 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:04:44.627 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:04:44.627 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:04:44.627 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.618+0000 7f98bffff640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] but i'm connected to v1:172.21.15.42:6789/0, reconnecting 2024-09-16T04:04:44.627 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.618+0000 7f98bffff640 10 monclient: _reopen_session rank -1 2024-09-16T04:04:44.628 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.618+0000 7f98bffff640 1 -- 172.21.15.42:0/3220479930 >> v1:172.21.15.42:6789/0 conn(0x7f98b00036f0 legacy=0x7f98b0005b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:04:44.628 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.618+0000 7f98d1fb1640 1 -- 172.21.15.42:0/3220479930 reap_dead start 2024-09-16T04:04:44.628 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.618+0000 7f98bffff640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:04:44.628 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.618+0000 7f98bffff640 1 --2- 172.21.15.42:0/3220479930 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f98b80039c0 0x7f98b8003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:04:44.628 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.618+0000 7f98bffff640 10 monclient(hunting): picked mon.b con 0x7f98b80039c0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:04:44.628 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.618+0000 7f98bffff640 1 --2- 172.21.15.42:0/3220479930 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f98b8004810 0x7f98b8004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:04:44.628 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.618+0000 7f98bffff640 10 monclient(hunting): picked mon.c con 0x7f98b8004810 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:04:44.628 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.618+0000 7f98bffff640 1 --2- 172.21.15.42:0/3220479930 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f98b8005220 0x7f98b800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:04:44.628 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.618+0000 7f98bffff640 10 monclient(hunting): picked mon.a con 0x7f98b8005220 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:04:44.628 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.618+0000 7f98bffff640 10 monclient(hunting): start opening mon connection 2024-09-16T04:04:44.628 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.618+0000 7f98bffff640 1 -- 172.21.15.42:0/3220479930 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f98cc09edf0 con 0x7f98b8005220 2024-09-16T04:04:44.628 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.618+0000 7f98bffff640 10 monclient(hunting): start opening mon connection 2024-09-16T04:04:44.628 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.618+0000 7f98bffff640 1 -- 172.21.15.42:0/3220479930 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f98b800c150 con 0x7f98b80039c0 2024-09-16T04:04:44.629 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.618+0000 7f98bffff640 10 monclient(hunting): start opening mon connection 2024-09-16T04:04:44.629 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.618+0000 7f98bffff640 1 -- 172.21.15.42:0/3220479930 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f98b800c350 con 0x7f98b8004810 2024-09-16T04:04:44.629 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.618+0000 7f98bffff640 10 monclient(hunting): _renew_subs 2024-09-16T04:04:44.629 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.618+0000 7f98bffff640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:04:44.629 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.618+0000 7f98d0faf640 1 -- 172.21.15.42:0/3220479930 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f98b8004810 msgr2=0x7f98b8004c90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T04:04:44.629 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.618+0000 7f98d0faf640 1 --2- 172.21.15.42:0/3220479930 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f98b8004810 0x7f98b8004c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:04:44.629 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.618+0000 7f98d17b0640 1 --2- 172.21.15.42:0/3220479930 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f98b80039c0 0x7f98b8003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:44.629 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.618+0000 7f98d1fb1640 1 --2- 172.21.15.42:0/3220479930 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f98b8005220 0x7f98b800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:44.629 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.618+0000 7f98d17b0640 10 monclient(hunting): get_auth_request con 0x7f98b80039c0 auth_method 0 2024-09-16T04:04:44.629 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.618+0000 7f98d17b0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:04:44.629 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.618+0000 7f98d17b0640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:04:44.629 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.618+0000 7f98d17b0640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:04:44.629 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.618+0000 7f98d1fb1640 10 monclient(hunting): get_auth_request con 0x7f98b8005220 auth_method 0 2024-09-16T04:04:44.630 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.618+0000 7f98d1fb1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:04:44.630 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.618+0000 7f98d1fb1640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:04:44.630 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.618+0000 7f98d1fb1640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:04:44.630 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.618+0000 7f98d17b0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:04:44.630 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.618+0000 7f98d17b0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:04:44.630 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.618+0000 7f98d17b0640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:04:44.630 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.618+0000 7f98d1fb1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:04:44.630 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.618+0000 7f98d1fb1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:04:44.630 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.618+0000 7f98d1fb1640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:04:44.632 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.622+0000 7f98d17b0640 10 monclient(hunting): handle_auth_done global_id 984195 payload 293 2024-09-16T04:04:44.632 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.622+0000 7f98d17b0640 1 -- 172.21.15.42:0/3220479930 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f98b8004810 msgr2=0x7f98b8004c90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:04:44.633 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.622+0000 7f98d17b0640 1 --2- 172.21.15.42:0/3220479930 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f98b8004810 0x7f98b8004c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:44.633 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.622+0000 7f98d17b0640 1 -- 172.21.15.42:0/3220479930 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f98b8005220 msgr2=0x7f98b800b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:04:44.633 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.622+0000 7f98d17b0640 1 --2- 172.21.15.42:0/3220479930 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f98b8005220 0x7f98b800b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:44.633 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.622+0000 7f98d17b0640 10 monclient: _finish_hunting 0 2024-09-16T04:04:44.633 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.622+0000 7f98d17b0640 1 monclient: found mon.b 2024-09-16T04:04:44.633 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.622+0000 7f98d17b0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:04:44.633 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.622+0000 7f98d17b0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2024-09-16T04:04:44.633 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.622+0000 7f98d17b0640 1 -- 172.21.15.42:0/3220479930 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98b8003530 con 0x7f98b80039c0 2024-09-16T04:04:44.633 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.622+0000 7f98d17b0640 10 monclient: _finish_auth 0 2024-09-16T04:04:44.633 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.622+0000 7f98d17b0640 10 monclient: _check_auth_tickets 2024-09-16T04:04:44.633 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.622+0000 7f98d17b0640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:04:44.633 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.622+0000 7f98d3a14640 5 monclient: authenticate success, global_id 984195 2024-09-16T04:04:44.633 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.622+0000 7f98d3a14640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:04:44.634 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.626+0000 7f98d17b0640 1 --2- 172.21.15.42:0/3220479930 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f98b80039c0 0x7f98b8003dc0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f98c0000ee0 tx=0x7f98c00220d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=85890d89bb8d9ac3 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:04:44.634 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.626+0000 7f98bffff640 1 -- 172.21.15.42:0/3220479930 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98c00b1070 con 0x7f98b80039c0 2024-09-16T04:04:44.634 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.626+0000 7f98bffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:04:44.634 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.626+0000 7f98bffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:04:44.634 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.626+0000 7f98bffff640 10 monclient: dump: 2024-09-16T04:04:44.634 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:04:44.634 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:04:44.634 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:44.634 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:44.634 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:04:44.634 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:04:44.634 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:04:44.634 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:04:44.634 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:04:44.635 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:04:44.635 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.626+0000 7f98bffff640 1 -- 172.21.15.42:0/3220479930 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f98c00abc80 con 0x7f98b80039c0 2024-09-16T04:04:44.635 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.626+0000 7f98d3a14640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:04:44.635 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.626+0000 7f98bffff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:04:44.635 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.626+0000 7f98d3a14640 10 monclient: get_monmap_and_config success 2024-09-16T04:04:44.635 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.626+0000 7f98d3a14640 4 set_mon_vals no callback set 2024-09-16T04:04:44.635 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.626+0000 7f98bffff640 1 -- 172.21.15.42:0/3220479930 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98c000eb80 con 0x7f98b80039c0 2024-09-16T04:04:44.635 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.626+0000 7f98bffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:04:44.635 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.626+0000 7f98bffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:04:44.635 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.626+0000 7f98bffff640 10 monclient: dump: 2024-09-16T04:04:44.635 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:04:44.635 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:04:44.635 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:44.635 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:44.636 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:04:44.636 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:04:44.636 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:04:44.636 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:04:44.636 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:04:44.636 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:04:44.636 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.626+0000 7f98d3a14640 10 monclient: shutdown 2024-09-16T04:04:44.636 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.626+0000 7f98d3a14640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:04:44.636 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.626+0000 7f98d3a14640 1 -- 172.21.15.42:0/3220479930 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f98b80039c0 msgr2=0x7f98b8003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:04:44.636 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.626+0000 7f98d3a14640 1 --2- 172.21.15.42:0/3220479930 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f98b80039c0 0x7f98b8003dc0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f98c0000ee0 tx=0x7f98c00220d0 comp rx=0 tx=0).stop 2024-09-16T04:04:44.636 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.626+0000 7f98d3a14640 1 -- 172.21.15.42:0/3220479930 shutdown_connections 2024-09-16T04:04:44.636 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.626+0000 7f98d3a14640 1 --2- 172.21.15.42:0/3220479930 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f98b8005220 0x7f98b800b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:44.636 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.626+0000 7f98d3a14640 1 --2- 172.21.15.42:0/3220479930 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f98b8004810 0x7f98b8004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:44.637 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.626+0000 7f98d3a14640 1 --2- 172.21.15.42:0/3220479930 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f98b80039c0 0x7f98b8003dc0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:44.637 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.626+0000 7f98d3a14640 1 -- 172.21.15.42:0/3220479930 >> 172.21.15.42:0/3220479930 conn(0x7f98cc0a2420 msgr2=0x7f98cc0a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:04:44.637 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.626+0000 7f98d3a14640 1 -- 172.21.15.42:0/3220479930 shutdown_connections 2024-09-16T04:04:44.637 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.626+0000 7f98d3a14640 1 -- 172.21.15.42:0/3220479930 wait complete. 2024-09-16T04:04:44.641 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.630+0000 7f98d3a14640 10 monclient: build_initial_monmap 2024-09-16T04:04:44.641 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.630+0000 7f98d3a14640 10 monclient: monmap: 2024-09-16T04:04:44.642 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:04:44.642 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:04:44.642 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:04:44.642 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:04:44.642 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:04:44.642 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:04:44.642 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:04:44.642 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:04:44.642 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:04:44.642 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:04:44.642 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.630+0000 7f98d3a14640 1 librados: starting msgr at 2024-09-16T04:04:44.642 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.630+0000 7f98d3a14640 1 librados: starting objecter 2024-09-16T04:04:44.642 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.630+0000 7f98d3a14640 1 Processor -- start 2024-09-16T04:04:44.642 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.630+0000 7f98d3a14640 1 -- start start 2024-09-16T04:04:44.643 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.630+0000 7f98d3a14640 1 librados: setting wanted keys 2024-09-16T04:04:44.643 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.630+0000 7f98d3a14640 1 librados: calling monclient init 2024-09-16T04:04:44.643 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.630+0000 7f98d3a14640 10 monclient: init 2024-09-16T04:04:44.643 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.630+0000 7f98d3a14640 10 monclient: _reopen_session rank -1 2024-09-16T04:04:44.643 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.630+0000 7f98d3a14640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:04:44.643 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.630+0000 7f98d3a14640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f98b80039c0 0x7f98cc133690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:04:44.643 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.630+0000 7f98d3a14640 10 monclient(hunting): picked mon.noname-c con 0x7f98b80039c0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:04:44.643 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.630+0000 7f98d3a14640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f98b8004810 0x7f98cc133bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:04:44.643 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.630+0000 7f98d3a14640 10 monclient(hunting): picked mon.noname-b con 0x7f98b8004810 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:04:44.643 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.630+0000 7f98d3a14640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f98b8005220 0x7f98cc1c85a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:04:44.644 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.630+0000 7f98d3a14640 10 monclient(hunting): picked mon.noname-a con 0x7f98b8005220 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:04:44.644 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.630+0000 7f98d3a14640 10 monclient(hunting): start opening mon connection 2024-09-16T04:04:44.644 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.630+0000 7f98d3a14640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f98cc09d1b0 con 0x7f98b8005220 2024-09-16T04:04:44.644 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.630+0000 7f98d3a14640 10 monclient(hunting): start opening mon connection 2024-09-16T04:04:44.644 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.630+0000 7f98d3a14640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f98cc09d030 con 0x7f98b8004810 2024-09-16T04:04:44.644 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.630+0000 7f98d3a14640 10 monclient(hunting): start opening mon connection 2024-09-16T04:04:44.644 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.630+0000 7f98d3a14640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f98cc09f180 con 0x7f98b80039c0 2024-09-16T04:04:44.644 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.630+0000 7f98d3a14640 10 monclient(hunting): _renew_subs 2024-09-16T04:04:44.644 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.630+0000 7f98d3a14640 10 monclient(hunting): authenticate will time out at 2331.602515s 2024-09-16T04:04:44.644 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.630+0000 7f98d17b0640 1 -- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f98b80039c0 msgr2=0x7f98cc133690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T04:04:44.644 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.630+0000 7f98d17b0640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f98b80039c0 0x7f98cc133690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:04:44.644 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.630+0000 7f98d1fb1640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f98b8005220 0x7f98cc1c85a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:44.644 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.630+0000 7f98d1fb1640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f98b8005220 0x7f98cc1c85a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:32786/0 (socket says 172.21.15.42:32786) 2024-09-16T04:04:44.645 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.630+0000 7f98d1fb1640 1 -- 172.21.15.42:0/3380643574 learned_addr learned my addr 172.21.15.42:0/3380643574 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:04:44.645 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.630+0000 7f98d1fb1640 10 monclient(hunting): get_auth_request con 0x7f98b8005220 auth_method 0 2024-09-16T04:04:44.645 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.630+0000 7f98d1fb1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:04:44.645 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.630+0000 7f98d1fb1640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:04:44.645 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.630+0000 7f98d1fb1640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:04:44.645 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.630+0000 7f98d1fb1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:04:44.645 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.630+0000 7f98d1fb1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:04:44.645 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.630+0000 7f98d1fb1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:04:44.645 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.630+0000 7f98d0faf640 1 --2- 172.21.15.42:0/3380643574 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f98b8004810 0x7f98cc133bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:44.645 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.630+0000 7f98d0faf640 10 monclient(hunting): get_auth_request con 0x7f98b8004810 auth_method 0 2024-09-16T04:04:44.645 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.630+0000 7f98d0faf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:04:44.645 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.630+0000 7f98d0faf640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:04:44.645 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.630+0000 7f98d0faf640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:04:44.645 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.630+0000 7f98d0faf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:04:44.646 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.630+0000 7f98d0faf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:04:44.646 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.630+0000 7f98d0faf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:04:44.646 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.630+0000 7f98d1fb1640 10 monclient(hunting): handle_auth_done global_id 984198 payload 824 2024-09-16T04:04:44.646 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.630+0000 7f98d1fb1640 1 -- 172.21.15.42:0/3380643574 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f98b80039c0 msgr2=0x7f98cc133690 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:04:44.646 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.630+0000 7f98d1fb1640 1 --2- 172.21.15.42:0/3380643574 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f98b80039c0 0x7f98cc133690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:44.646 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.630+0000 7f98d1fb1640 1 -- 172.21.15.42:0/3380643574 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f98b8004810 msgr2=0x7f98cc133bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:04:44.646 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.630+0000 7f98d1fb1640 1 --2- 172.21.15.42:0/3380643574 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f98b8004810 0x7f98cc133bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:44.646 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.630+0000 7f98d1fb1640 10 monclient: _finish_hunting 0 2024-09-16T04:04:44.646 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.630+0000 7f98d1fb1640 1 monclient: found mon.noname-a 2024-09-16T04:04:44.646 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.630+0000 7f98d1fb1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:04:44.646 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.630+0000 7f98d1fb1640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:04:44.646 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.630+0000 7f98d1fb1640 1 -- 172.21.15.42:0/3380643574 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98cc0a3800 con 0x7f98b8005220 2024-09-16T04:04:44.646 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.630+0000 7f98d1fb1640 1 --2- 172.21.15.42:0/3380643574 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f98b8005220 0x7f98cc1c85a0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f98b0007390 tx=0x7f98c400ea00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:04:44.647 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.630+0000 7f98bdffb640 10 client.?.objecter ms_handle_connect 0x7f98b8005220 2024-09-16T04:04:44.647 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.630+0000 7f98bdffb640 10 client.?.objecter resend_mon_ops 2024-09-16T04:04:44.647 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.630+0000 7f98bdffb640 1 -- 172.21.15.42:0/3380643574 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98c400c780 con 0x7f98b8005220 2024-09-16T04:04:44.647 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.630+0000 7f98bdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:04:44.647 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.630+0000 7f98bdffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:04:44.647 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.630+0000 7f98bdffb640 10 monclient: dump: 2024-09-16T04:04:44.647 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:04:44.647 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:04:44.647 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:44.647 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:44.647 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:04:44.647 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:04:44.647 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:04:44.647 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:04:44.648 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:04:44.648 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:04:44.648 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.630+0000 7f98bdffb640 10 monclient: _finish_auth 0 2024-09-16T04:04:44.648 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.630+0000 7f98bdffb640 10 monclient: _check_auth_tickets 2024-09-16T04:04:44.648 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.630+0000 7f98bdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:04:44.648 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.630+0000 7f98bdffb640 1 -- 172.21.15.42:0/3380643574 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f98c400c940 con 0x7f98b8005220 2024-09-16T04:04:44.648 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.630+0000 7f98d3a14640 5 monclient: authenticate success, global_id 984198 2024-09-16T04:04:44.648 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.630+0000 7f98d3a14640 10 monclient: _renew_subs 2024-09-16T04:04:44.648 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.630+0000 7f98d3a14640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:04:44.648 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.630+0000 7f98d3a14640 1 -- 172.21.15.42:0/3380643574 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98cc073a30 con 0x7f98b8005220 2024-09-16T04:04:44.648 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.630+0000 7f98bdffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:04:44.648 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.630+0000 7f98bdffb640 1 -- 172.21.15.42:0/3380643574 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98c400cd30 con 0x7f98b8005220 2024-09-16T04:04:44.648 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.630+0000 7f98bdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:04:44.649 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.630+0000 7f98bdffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:04:44.649 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.630+0000 7f98bdffb640 10 monclient: dump: 2024-09-16T04:04:44.649 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:04:44.649 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:04:44.649 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:44.649 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:44.649 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:04:44.649 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:04:44.649 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:04:44.649 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:04:44.649 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:04:44.649 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:04:44.649 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.630+0000 7f98bf7fe640 4 set_mon_vals no callback set 2024-09-16T04:04:44.649 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.630+0000 7f98d3a14640 10 client.984198.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:04:44.650 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.634+0000 7f98d3a14640 10 monclient: _renew_subs 2024-09-16T04:04:44.650 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.634+0000 7f98d3a14640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:04:44.650 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.634+0000 7f98d3a14640 1 -- 172.21.15.42:0/3380643574 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98cc123070 con 0x7f98b8005220 2024-09-16T04:04:44.650 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.634+0000 7f98d3a14640 1 librados: init done 2024-09-16T04:04:44.650 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.634+0000 7f98bdffb640 1 -- 172.21.15.42:0/3380643574 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f98c4005070 con 0x7f98b8005220 2024-09-16T04:04:44.650 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.634+0000 7f98d3a14640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:04:44.650 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.634+0000 7f98d3a14640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:04:44.650 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.634+0000 7f98d3a14640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:04:44.650 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.634+0000 7f98d3a14640 1 -- 172.21.15.42:0/3380643574 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f98cc09f500 con 0x7f98b8005220 2024-09-16T04:04:44.650 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.634+0000 7f98bdffb640 1 --2- 172.21.15.42:0/3380643574 >> v2:172.21.15.154:6812/254845342 conn(0x7f98b00381a0 0x7f98b003a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:04:44.650 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.634+0000 7f98bdffb640 1 -- 172.21.15.42:0/3380643574 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(682..682 src has 451..682) ==== 6280+0+0 (secure 0 0 0) 0x7f98c4049670 con 0x7f98b8005220 2024-09-16T04:04:44.650 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.634+0000 7f98bdffb640 10 client.984198.objecter ms_dispatch 0x7f98cc000ba0 osd_map(682..682 src has 451..682) 2024-09-16T04:04:44.650 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.634+0000 7f98bdffb640 3 client.984198.objecter handle_osd_map got epochs [682,682] > 0 2024-09-16T04:04:44.650 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.634+0000 7f98bdffb640 3 client.984198.objecter handle_osd_map decoding full epoch 682 2024-09-16T04:04:44.651 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.634+0000 7f98bdffb640 20 client.984198.objecter dump_active .. 0 homeless 2024-09-16T04:04:44.651 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.634+0000 7f98d17b0640 1 --2- 172.21.15.42:0/3380643574 >> v2:172.21.15.154:6812/254845342 conn(0x7f98b00381a0 0x7f98b003a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:44.651 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.634+0000 7f98d17b0640 10 monclient: get_auth_request con 0x7f98b00381a0 auth_method 0 2024-09-16T04:04:44.651 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.634+0000 7f98bdffb640 10 client.984198.objecter ms_handle_connect 0x7f98b00381a0 2024-09-16T04:04:44.651 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.634+0000 7f98d17b0640 1 --2- 172.21.15.42:0/3380643574 >> v2:172.21.15.154:6812/254845342 conn(0x7f98b00381a0 0x7f98b003a660 secure :-1 s=READY pgs=3832 cs=0 l=1 rev1=1 crypto rx=0x7f98c00b6000 tx=0x7f98c0010040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:04:44.651 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.638+0000 7f98bdffb640 1 -- 172.21.15.42:0/3380643574 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f98c4003e20 con 0x7f98b8005220 2024-09-16T04:04:44.651 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.638+0000 7f98bdffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:04:44.651 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.638+0000 7f98bdffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:04:44.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.866+0000 7f98d3a14640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:04:44.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.866+0000 7f98d3a14640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:04:44.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.866+0000 7f98d3a14640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:04:44.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.866+0000 7f98d3a14640 1 -- 172.21.15.42:0/3380643574 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f98cc069040 con 0x7f98b8005220 2024-09-16T04:04:44.871 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.866+0000 7f98bdffb640 1 -- 172.21.15.42:0/3380643574 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1555 (secure 0 0 0) 0x7f98c4005360 con 0x7f98b8005220 2024-09-16T04:04:44.871 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.866+0000 7f98bdffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:04:44.871 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.866+0000 7f98bdffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:04:44.871 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":214,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:04:44.874 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.866+0000 7f98d3a14640 10 librados: watch_flush enter 2024-09-16T04:04:44.874 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.866+0000 7f98d3a14640 10 librados: watch_flush exit 2024-09-16T04:04:44.874 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.866+0000 7f98d3a14640 20 client.984198.objecter shutdown clearing up homeless session... 2024-09-16T04:04:44.874 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.866+0000 7f98d3a14640 10 client.984198.objecter successfully canceled tick 2024-09-16T04:04:44.874 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.866+0000 7f98d3a14640 1 -- 172.21.15.42:0/3380643574 >> v2:172.21.15.154:6812/254845342 conn(0x7f98b00381a0 msgr2=0x7f98b003a660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:04:44.874 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.866+0000 7f98d3a14640 1 --2- 172.21.15.42:0/3380643574 >> v2:172.21.15.154:6812/254845342 conn(0x7f98b00381a0 0x7f98b003a660 secure :-1 s=READY pgs=3832 cs=0 l=1 rev1=1 crypto rx=0x7f98c00b6000 tx=0x7f98c0010040 comp rx=0 tx=0).stop 2024-09-16T04:04:44.874 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.866+0000 7f98d3a14640 10 monclient: shutdown 2024-09-16T04:04:44.874 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.866+0000 7f98d3a14640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:04:44.874 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.866+0000 7f98d3a14640 1 -- 172.21.15.42:0/3380643574 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f98b8005220 msgr2=0x7f98cc1c85a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:04:44.874 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.866+0000 7f98d3a14640 1 --2- 172.21.15.42:0/3380643574 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f98b8005220 0x7f98cc1c85a0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f98b0007390 tx=0x7f98c400ea00 comp rx=0 tx=0).stop 2024-09-16T04:04:44.874 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.866+0000 7f98d3a14640 1 -- 172.21.15.42:0/3380643574 shutdown_connections 2024-09-16T04:04:44.875 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.866+0000 7f98d3a14640 1 --2- 172.21.15.42:0/3380643574 >> v2:172.21.15.154:6812/254845342 conn(0x7f98b00381a0 0x7f98b003a660 unknown :-1 s=CLOSED pgs=3832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:44.875 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.866+0000 7f98d3a14640 1 --2- 172.21.15.42:0/3380643574 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f98b8005220 0x7f98cc1c85a0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:44.875 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.870+0000 7f98d3a14640 1 --2- 172.21.15.42:0/3380643574 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f98b8004810 0x7f98cc133bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:44.875 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.870+0000 7f98d3a14640 1 --2- 172.21.15.42:0/3380643574 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f98b80039c0 0x7f98cc133690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:44.875 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.870+0000 7f98d3a14640 1 -- 172.21.15.42:0/3380643574 >> 172.21.15.42:0/3380643574 conn(0x7f98cc0a2420 msgr2=0x7f98cc138660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:04:44.875 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.870+0000 7f98d3a14640 1 -- 172.21.15.42:0/3380643574 shutdown_connections 2024-09-16T04:04:44.875 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.870+0000 7f98d3a14640 1 -- 172.21.15.42:0/3380643574 wait complete. 2024-09-16T04:04:44.875 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:44.870+0000 7f98d3a14640 1 librados: shutdown 2024-09-16T04:04:44.889 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T04:04:44.890 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-16T04:04:44.890 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T04:04:45.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.006+0000 7f84fa7a2640 10 monclient: get_monmap_and_config 2024-09-16T04:04:45.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.006+0000 7f84fa7a2640 10 monclient: build_initial_monmap 2024-09-16T04:04:45.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.006+0000 7f84fa7a2640 10 monclient: monmap: 2024-09-16T04:04:45.011 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:04:45.011 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:04:45.011 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:04:45.009186+0000 2024-09-16T04:04:45.011 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:04:45.009186+0000 2024-09-16T04:04:45.011 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:04:45.011 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:04:45.011 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:04:45.011 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:04:45.011 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:04:45.011 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:04:45.011 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:04:45.011 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:04:45.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.006+0000 7f84fa7a2640 1 Processor -- start 2024-09-16T04:04:45.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.006+0000 7f84fa7a2640 1 -- start start 2024-09-16T04:04:45.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.006+0000 7f84fa7a2640 10 monclient: init 2024-09-16T04:04:45.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.006+0000 7f84fa7a2640 10 monclient: _reopen_session rank -1 2024-09-16T04:04:45.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.006+0000 7f84fa7a2640 10 monclient: _add_conns ranks=[3,0,4,2,1] 2024-09-16T04:04:45.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.006+0000 7f84fa7a2640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f84f40a39b0 0x7f84f4133b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:04:45.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.006+0000 7f84fa7a2640 10 monclient(hunting): picked mon.noname-a con 0x7f84f40a39b0 addr v2:172.21.15.42:3300/0 2024-09-16T04:04:45.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.006+0000 7f84fa7a2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f84f40a3d90 addr v1:172.21.15.42:6789/0 2024-09-16T04:04:45.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.006+0000 7f84fa7a2640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f84f4137710 0x7f84f413f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:04:45.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.006+0000 7f84fa7a2640 10 monclient(hunting): picked mon.noname-c con 0x7f84f4137710 addr v2:172.21.15.154:3300/0 2024-09-16T04:04:45.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.006+0000 7f84fa7a2640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f84f409c140 con 0x7f84f40a3d90 2024-09-16T04:04:45.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.006+0000 7f84fa7a2640 10 monclient(hunting): start opening mon connection 2024-09-16T04:04:45.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.006+0000 7f84fa7a2640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f84f409c420 con 0x7f84f40a39b0 2024-09-16T04:04:45.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.006+0000 7f84fa7a2640 10 monclient(hunting): start opening mon connection 2024-09-16T04:04:45.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.006+0000 7f84fa7a2640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f84f406dd60 con 0x7f84f4137710 2024-09-16T04:04:45.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.006+0000 7f84fa7a2640 10 monclient(hunting): _renew_subs 2024-09-16T04:04:45.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.006+0000 7f84fa7a2640 10 monclient(hunting): authenticate will time out at 2331.978770s 2024-09-16T04:04:45.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.006+0000 7f84f3fff640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f84f40a39b0 0x7f84f4133b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:45.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.006+0000 7f84f37fe640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f84f40a3d90 0x7f84f4137000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:53152/0 (socket says 172.21.15.42:53152) 2024-09-16T04:04:45.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.006+0000 7f84f3fff640 10 monclient(hunting): get_auth_request con 0x7f84f40a39b0 auth_method 0 2024-09-16T04:04:45.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.006+0000 7f84f37fe640 1 -- 172.21.15.42:0/609316368 learned_addr learned my addr 172.21.15.42:0/609316368 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T04:04:45.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.006+0000 7f84f3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:04:45.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.006+0000 7f84f3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:04:45.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.006+0000 7f84f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:04:45.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.006+0000 7f84f3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:04:45.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.006+0000 7f84f3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:04:45.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.006+0000 7f84f3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:04:45.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.006+0000 7f84f8d3f640 1 --2- 172.21.15.42:0/609316368 >> v2:172.21.15.154:3300/0 conn(0x7f84f4137710 0x7f84f413f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:45.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.006+0000 7f84f3fff640 10 monclient(hunting): handle_auth_done global_id 984201 payload 274 2024-09-16T04:04:45.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.006+0000 7f84f3fff640 1 -- 172.21.15.42:0/609316368 >> v2:172.21.15.154:3300/0 conn(0x7f84f4137710 msgr2=0x7f84f413f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:04:45.016 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.006+0000 7f84f3fff640 1 --2- 172.21.15.42:0/609316368 >> v2:172.21.15.154:3300/0 conn(0x7f84f4137710 0x7f84f413f3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:45.016 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.006+0000 7f84f3fff640 1 -- 172.21.15.42:0/609316368 >> v1:172.21.15.42:6789/0 conn(0x7f84f40a3d90 legacy=0x7f84f4137000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:04:45.016 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.006+0000 7f84f3fff640 10 monclient: _finish_hunting 0 2024-09-16T04:04:45.016 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.006+0000 7f84f3fff640 1 monclient: found mon.noname-a 2024-09-16T04:04:45.016 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.006+0000 7f84f3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:04:45.016 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.006+0000 7f84f3fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:04:45.016 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.006+0000 7f84f3fff640 1 -- 172.21.15.42:0/609316368 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84f40a5e20 con 0x7f84f40a39b0 2024-09-16T04:04:45.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.010+0000 7f84f3fff640 1 --2- 172.21.15.42:0/609316368 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f84f40a39b0 0x7f84f4133b80 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f84f406b540 tx=0x7f84e401c340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eab8b0ce5be08a18 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:04:45.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.010+0000 7f84f2ffd640 1 -- 172.21.15.42:0/609316368 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84e40b0070 con 0x7f84f40a39b0 2024-09-16T04:04:45.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.010+0000 7f84f2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:04:45.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.010+0000 7f84f2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:04:45.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.010+0000 7f84f2ffd640 10 monclient: dump: 2024-09-16T04:04:45.024 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:04:45.024 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:04:45.024 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:45.024 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:45.024 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:04:45.024 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:04:45.024 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:04:45.025 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:04:45.025 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:04:45.025 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:04:45.025 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.010+0000 7f84f2ffd640 10 monclient: _finish_auth 0 2024-09-16T04:04:45.025 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.010+0000 7f84f2ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:04:45.025 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.010+0000 7f84f2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:04:45.025 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.010+0000 7f84f2ffd640 1 -- 172.21.15.42:0/609316368 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f84e40ab600 con 0x7f84f40a39b0 2024-09-16T04:04:45.025 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.010+0000 7f84fa7a2640 5 monclient: authenticate success, global_id 984201 2024-09-16T04:04:45.025 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.010+0000 7f84fa7a2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:04:45.025 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.010+0000 7f84f2ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:04:45.025 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.010+0000 7f84fa7a2640 10 monclient: get_monmap_and_config success 2024-09-16T04:04:45.025 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.010+0000 7f84fa7a2640 4 set_mon_vals no callback set 2024-09-16T04:04:45.025 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.010+0000 7f84f2ffd640 1 -- 172.21.15.42:0/609316368 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84e40ab9d0 con 0x7f84f40a39b0 2024-09-16T04:04:45.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.010+0000 7f84f2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:04:45.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.010+0000 7f84f2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:04:45.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.010+0000 7f84f2ffd640 10 monclient: dump: 2024-09-16T04:04:45.026 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:04:45.026 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:04:45.026 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:45.026 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:45.026 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:04:45.026 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:04:45.026 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:04:45.026 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:04:45.026 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:04:45.026 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:04:45.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.010+0000 7f84fa7a2640 10 monclient: shutdown 2024-09-16T04:04:45.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.010+0000 7f84fa7a2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:04:45.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.010+0000 7f84fa7a2640 1 -- 172.21.15.42:0/609316368 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f84f40a39b0 msgr2=0x7f84f4133b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:04:45.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.010+0000 7f84fa7a2640 1 --2- 172.21.15.42:0/609316368 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f84f40a39b0 0x7f84f4133b80 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f84f406b540 tx=0x7f84e401c340 comp rx=0 tx=0).stop 2024-09-16T04:04:45.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.010+0000 7f84fa7a2640 1 -- 172.21.15.42:0/609316368 shutdown_connections 2024-09-16T04:04:45.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.010+0000 7f84fa7a2640 1 --2- 172.21.15.42:0/609316368 >> v2:172.21.15.154:3300/0 conn(0x7f84f4137710 0x7f84f413f3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:45.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.010+0000 7f84fa7a2640 1 --2- 172.21.15.42:0/609316368 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f84f40a39b0 0x7f84f4133b80 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:45.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.010+0000 7f84fa7a2640 1 -- 172.21.15.42:0/609316368 >> 172.21.15.42:0/609316368 conn(0x7f84f40a9fb0 msgr2=0x7f84f40aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:04:45.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.010+0000 7f84fa7a2640 1 -- 172.21.15.42:0/609316368 shutdown_connections 2024-09-16T04:04:45.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.010+0000 7f84fa7a2640 1 -- 172.21.15.42:0/609316368 wait complete. 2024-09-16T04:04:45.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.010+0000 7f84fa7a2640 10 monclient: build_initial_monmap 2024-09-16T04:04:45.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.010+0000 7f84fa7a2640 10 monclient: monmap: 2024-09-16T04:04:45.028 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:04:45.028 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:04:45.028 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:04:45.028 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:04:45.028 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:04:45.028 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:04:45.028 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:04:45.028 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:04:45.028 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:04:45.028 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:04:45.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.010+0000 7f84fa7a2640 1 librados: starting msgr at 2024-09-16T04:04:45.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.010+0000 7f84fa7a2640 1 librados: starting objecter 2024-09-16T04:04:45.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.010+0000 7f84fa7a2640 1 Processor -- start 2024-09-16T04:04:45.029 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.010+0000 7f84fa7a2640 1 -- start start 2024-09-16T04:04:45.029 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.010+0000 7f84fa7a2640 1 librados: setting wanted keys 2024-09-16T04:04:45.029 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.010+0000 7f84fa7a2640 1 librados: calling monclient init 2024-09-16T04:04:45.029 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.010+0000 7f84fa7a2640 10 monclient: init 2024-09-16T04:04:45.029 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.014+0000 7f84fa7a2640 10 monclient: _reopen_session rank -1 2024-09-16T04:04:45.029 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.014+0000 7f84fa7a2640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:04:45.029 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.014+0000 7f84fa7a2640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f84f40a39b0 0x7f84f4138ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:04:45.029 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.014+0000 7f84fa7a2640 10 monclient(hunting): picked mon.noname-b con 0x7f84f40a39b0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:04:45.029 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.014+0000 7f84fa7a2640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f84f40a3d90 0x7f84f41393e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:04:45.029 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.014+0000 7f84fa7a2640 10 monclient(hunting): picked mon.noname-a con 0x7f84f40a3d90 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:04:45.029 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.014+0000 7f84fa7a2640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f84f4137710 0x7f84f4139920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:04:45.029 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.014+0000 7f84fa7a2640 10 monclient(hunting): picked mon.noname-c con 0x7f84f4137710 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:04:45.029 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.014+0000 7f84fa7a2640 10 monclient(hunting): start opening mon connection 2024-09-16T04:04:45.029 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.014+0000 7f84fa7a2640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f84f4142310 con 0x7f84f40a3d90 2024-09-16T04:04:45.030 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.014+0000 7f84fa7a2640 10 monclient(hunting): start opening mon connection 2024-09-16T04:04:45.030 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.014+0000 7f84fa7a2640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f84f4142490 con 0x7f84f40a39b0 2024-09-16T04:04:45.030 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.014+0000 7f84fa7a2640 10 monclient(hunting): start opening mon connection 2024-09-16T04:04:45.030 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.014+0000 7f84fa7a2640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f84f4142610 con 0x7f84f4137710 2024-09-16T04:04:45.030 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.014+0000 7f84fa7a2640 10 monclient(hunting): _renew_subs 2024-09-16T04:04:45.030 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.014+0000 7f84fa7a2640 10 monclient(hunting): authenticate will time out at 2331.984269s 2024-09-16T04:04:45.030 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.014+0000 7f84f8d3f640 1 -- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f84f4137710 msgr2=0x7f84f4139920 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T04:04:45.030 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.014+0000 7f84f8d3f640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f84f4137710 0x7f84f4139920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:04:45.030 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.014+0000 7f84f37fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f84f40a3d90 0x7f84f41393e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:45.030 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.014+0000 7f84f37fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f84f40a3d90 0x7f84f41393e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:32800/0 (socket says 172.21.15.42:32800) 2024-09-16T04:04:45.030 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.014+0000 7f84f37fe640 1 -- 172.21.15.42:0/3781502963 learned_addr learned my addr 172.21.15.42:0/3781502963 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:04:45.030 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.014+0000 7f84f37fe640 10 monclient(hunting): get_auth_request con 0x7f84f40a3d90 auth_method 0 2024-09-16T04:04:45.030 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.014+0000 7f84f37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:04:45.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.014+0000 7f84f37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:04:45.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.014+0000 7f84f37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:04:45.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.014+0000 7f84f37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:04:45.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.014+0000 7f84f37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:04:45.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.014+0000 7f84f37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:04:45.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.014+0000 7f84f3fff640 1 --2- 172.21.15.42:0/3781502963 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f84f40a39b0 0x7f84f4138ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:45.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.014+0000 7f84f3fff640 10 monclient(hunting): get_auth_request con 0x7f84f40a39b0 auth_method 0 2024-09-16T04:04:45.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.014+0000 7f84f3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:04:45.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.014+0000 7f84f3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:04:45.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.014+0000 7f84f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:04:45.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.014+0000 7f84f37fe640 10 monclient(hunting): handle_auth_done global_id 984207 payload 824 2024-09-16T04:04:45.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.014+0000 7f84f37fe640 1 -- 172.21.15.42:0/3781502963 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f84f4137710 msgr2=0x7f84f4139920 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:04:45.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.014+0000 7f84f37fe640 1 --2- 172.21.15.42:0/3781502963 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f84f4137710 0x7f84f4139920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:45.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.014+0000 7f84f37fe640 1 -- 172.21.15.42:0/3781502963 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f84f40a39b0 msgr2=0x7f84f4138ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:04:45.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.014+0000 7f84f37fe640 1 --2- 172.21.15.42:0/3781502963 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f84f40a39b0 0x7f84f4138ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:45.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.014+0000 7f84f37fe640 10 monclient: _finish_hunting 0 2024-09-16T04:04:45.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.014+0000 7f84f37fe640 1 monclient: found mon.noname-a 2024-09-16T04:04:45.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.014+0000 7f84f37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:04:45.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.014+0000 7f84f37fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:04:45.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.014+0000 7f84f37fe640 1 -- 172.21.15.42:0/3781502963 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84f406f680 con 0x7f84f40a3d90 2024-09-16T04:04:45.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.014+0000 7f84f37fe640 1 --2- 172.21.15.42:0/3781502963 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f84f40a3d90 0x7f84f41393e0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f84e800cb10 tx=0x7f84e800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:04:45.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.014+0000 7f84f0ff9640 10 client.?.objecter ms_handle_connect 0x7f84f40a3d90 2024-09-16T04:04:45.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.014+0000 7f84f0ff9640 10 client.?.objecter resend_mon_ops 2024-09-16T04:04:45.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.014+0000 7f84f0ff9640 1 -- 172.21.15.42:0/3781502963 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84e8017070 con 0x7f84f40a3d90 2024-09-16T04:04:45.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.014+0000 7f84f0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:04:45.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.014+0000 7f84f0ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:04:45.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.014+0000 7f84f0ff9640 10 monclient: dump: 2024-09-16T04:04:45.032 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:04:45.033 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:04:45.033 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:45.033 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:45.033 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:04:45.033 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:04:45.033 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:04:45.033 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:04:45.033 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:04:45.033 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:04:45.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.014+0000 7f84f0ff9640 10 monclient: _finish_auth 0 2024-09-16T04:04:45.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.014+0000 7f84f0ff9640 10 monclient: _check_auth_tickets 2024-09-16T04:04:45.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.014+0000 7f84f0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:04:45.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.014+0000 7f84f0ff9640 1 -- 172.21.15.42:0/3781502963 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f84e800ac70 con 0x7f84f40a3d90 2024-09-16T04:04:45.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.014+0000 7f84fa7a2640 5 monclient: authenticate success, global_id 984207 2024-09-16T04:04:45.034 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.014+0000 7f84fa7a2640 10 monclient: _renew_subs 2024-09-16T04:04:45.034 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.014+0000 7f84fa7a2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:04:45.034 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.014+0000 7f84fa7a2640 1 -- 172.21.15.42:0/3781502963 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84f413ca90 con 0x7f84f40a3d90 2024-09-16T04:04:45.034 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.014+0000 7f84f0ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T04:04:45.034 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.014+0000 7f84f0ff9640 1 -- 172.21.15.42:0/3781502963 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84e8012620 con 0x7f84f40a3d90 2024-09-16T04:04:45.034 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.014+0000 7f84f0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:04:45.034 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.014+0000 7f84f0ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:04:45.034 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.014+0000 7f84f0ff9640 10 monclient: dump: 2024-09-16T04:04:45.034 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:04:45.034 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:04:45.034 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:45.034 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:45.034 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:04:45.034 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:04:45.035 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:04:45.035 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:04:45.035 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:04:45.035 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:04:45.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.014+0000 7f84fa7a2640 10 client.984207.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:04:45.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.014+0000 7f84fa7a2640 10 monclient: _renew_subs 2024-09-16T04:04:45.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.014+0000 7f84f27fc640 4 set_mon_vals no callback set 2024-09-16T04:04:45.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.014+0000 7f84fa7a2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:04:45.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.014+0000 7f84fa7a2640 1 -- 172.21.15.42:0/3781502963 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84f41333b0 con 0x7f84f40a3d90 2024-09-16T04:04:45.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.014+0000 7f84fa7a2640 1 librados: init done 2024-09-16T04:04:45.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.014+0000 7f84fa7a2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:04:45.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.014+0000 7f84f0ff9640 1 -- 172.21.15.42:0/3781502963 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f84e801a070 con 0x7f84f40a3d90 2024-09-16T04:04:45.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.014+0000 7f84f0ff9640 1 --2- 172.21.15.42:0/3781502963 >> v2:172.21.15.154:6812/254845342 conn(0x7f84e00377c0 0x7f84e0039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:04:45.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.014+0000 7f84fa7a2640 1 --2- 172.21.15.42:0/3781502963 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f84f41cf620 0x7f84f41cfa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:04:45.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.014+0000 7f84fa7a2640 10 monclient(hunting): start opening mon connection 2024-09-16T04:04:45.036 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.014+0000 7f84fa7a2640 1 -- 172.21.15.42:0/3781502963 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f84f41342d0 con 0x7f84f41cf620 2024-09-16T04:04:45.036 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.014+0000 7f84f0ff9640 1 -- 172.21.15.42:0/3781502963 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(682..682 src has 451..682) ==== 6280+0+0 (secure 0 0 0) 0x7f84e8048e80 con 0x7f84f40a3d90 2024-09-16T04:04:45.036 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.014+0000 7f84f0ff9640 10 client.984207.objecter ms_dispatch 0x7f84f4000ba0 osd_map(682..682 src has 451..682) 2024-09-16T04:04:45.036 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.014+0000 7f84f0ff9640 3 client.984207.objecter handle_osd_map got epochs [682,682] > 0 2024-09-16T04:04:45.036 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.014+0000 7f84f0ff9640 3 client.984207.objecter handle_osd_map decoding full epoch 682 2024-09-16T04:04:45.036 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.014+0000 7f84f0ff9640 20 client.984207.objecter dump_active .. 0 homeless 2024-09-16T04:04:45.036 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.014+0000 7f84f8d3f640 1 --2- 172.21.15.42:0/3781502963 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f84f41cf620 0x7f84f41cfa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:45.036 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.014+0000 7f84f8d3f640 10 monclient: get_auth_request con 0x7f84f41cf620 auth_method 0 2024-09-16T04:04:45.036 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.014+0000 7f84f8d3f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:04:45.036 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.014+0000 7f84f8d3f640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:04:45.036 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.014+0000 7f84f8d3f640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:04:45.036 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.014+0000 7f84f8d3f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:04:45.036 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.014+0000 7f84f8d3f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:04:45.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.014+0000 7f84f8d3f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:04:45.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.014+0000 7f84f3fff640 1 --2- 172.21.15.42:0/3781502963 >> v2:172.21.15.154:6812/254845342 conn(0x7f84e00377c0 0x7f84e0039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:45.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.014+0000 7f84f3fff640 10 monclient: get_auth_request con 0x7f84e00377c0 auth_method 0 2024-09-16T04:04:45.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.014+0000 7f84f8d3f640 10 monclient(hunting): handle_auth_done global_id 984210 payload 824 2024-09-16T04:04:45.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.014+0000 7f84f8d3f640 1 -- 172.21.15.42:0/3781502963 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f84f4133b80 con 0x7f84f41cf620 2024-09-16T04:04:45.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.014+0000 7f84f0ff9640 10 client.984207.objecter ms_handle_connect 0x7f84f41cf620 2024-09-16T04:04:45.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.014+0000 7f84f0ff9640 10 client.984207.objecter resend_mon_ops 2024-09-16T04:04:45.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.014+0000 7f84f8d3f640 1 --2- 172.21.15.42:0/3781502963 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f84f41cf620 0x7f84f41cfa00 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f84ec006c40 tx=0x7f84ec00c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:04:45.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.014+0000 7f84f0ff9640 1 -- 172.21.15.42:0/3781502963 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84ec015070 con 0x7f84f41cf620 2024-09-16T04:04:45.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.014+0000 7f84f0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:04:45.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.014+0000 7f84f0ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:04:45.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.014+0000 7f84f0ff9640 10 monclient: dump: 2024-09-16T04:04:45.038 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:04:45.038 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:04:45.038 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:45.038 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:45.038 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:04:45.038 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:04:45.038 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:04:45.038 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:04:45.038 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:04:45.038 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:04:45.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.014+0000 7f84f0ff9640 1 -- 172.21.15.42:0/3781502963 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f84ec004220 con 0x7f84f41cf620 2024-09-16T04:04:45.039 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.014+0000 7f84f0ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:04:45.039 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.014+0000 7f84f0ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:04:45.039 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.014+0000 7f84f0ff9640 1 -- 172.21.15.42:0/3781502963 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f84f41cf620 msgr2=0x7f84f41cfa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:04:45.039 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.014+0000 7f84f0ff9640 1 --2- 172.21.15.42:0/3781502963 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f84f41cf620 0x7f84f41cfa00 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f84ec006c40 tx=0x7f84ec00c950 comp rx=0 tx=0).stop 2024-09-16T04:04:45.039 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.014+0000 7f84f0ff9640 1 -- 172.21.15.42:0/3781502963 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f84f41cf620 msgr2=0x7f84f41cfa00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:04:45.039 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.014+0000 7f84f0ff9640 1 --2- 172.21.15.42:0/3781502963 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f84f41cf620 0x7f84f41cfa00 secure :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f84ec006c40 tx=0x7f84ec00c950 comp rx=0 tx=0).stop 2024-09-16T04:04:45.039 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.014+0000 7f84f0ff9640 10 client.984207.objecter ms_handle_connect 0x7f84e00377c0 2024-09-16T04:04:45.039 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.014+0000 7f84f3fff640 1 --2- 172.21.15.42:0/3781502963 >> v2:172.21.15.154:6812/254845342 conn(0x7f84e00377c0 0x7f84e0039c80 secure :-1 s=READY pgs=3833 cs=0 l=1 rev1=1 crypto rx=0x7f84e40ad000 tx=0x7f84e4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:04:45.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.022+0000 7f84fa7a2640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:04:45.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.022+0000 7f84fa7a2640 1 --2- 172.21.15.42:0/3781502963 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f84f41cff40 0x7f84f412e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:04:45.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.022+0000 7f84fa7a2640 10 monclient(hunting): start opening mon connection 2024-09-16T04:04:45.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.022+0000 7f84fa7a2640 1 -- 172.21.15.42:0/3781502963 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f84f4068e40 con 0x7f84f41cff40 2024-09-16T04:04:45.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.022+0000 7f84f8d3f640 1 --2- 172.21.15.42:0/3781502963 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f84f41cff40 0x7f84f412e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:45.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.022+0000 7f84f8d3f640 10 monclient: get_auth_request con 0x7f84f41cff40 auth_method 0 2024-09-16T04:04:45.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.022+0000 7f84f8d3f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:04:45.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.022+0000 7f84f8d3f640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:04:45.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.022+0000 7f84f8d3f640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:04:45.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.022+0000 7f84f8d3f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:04:45.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.022+0000 7f84f8d3f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:04:45.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.022+0000 7f84f8d3f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:04:45.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.022+0000 7f84f8d3f640 10 monclient(hunting): handle_auth_done global_id 984213 payload 824 2024-09-16T04:04:45.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.022+0000 7f84f8d3f640 1 -- 172.21.15.42:0/3781502963 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f84f413c710 con 0x7f84f41cff40 2024-09-16T04:04:45.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.022+0000 7f84f8d3f640 1 --2- 172.21.15.42:0/3781502963 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f84f41cff40 0x7f84f412e680 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f84ec006c40 tx=0x7f84ec004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:04:45.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.022+0000 7f84f0ff9640 10 client.984207.objecter ms_handle_connect 0x7f84f41cff40 2024-09-16T04:04:45.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.022+0000 7f84f0ff9640 10 client.984207.objecter resend_mon_ops 2024-09-16T04:04:45.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.022+0000 7f84f0ff9640 1 -- 172.21.15.42:0/3781502963 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84ec011de0 con 0x7f84f41cff40 2024-09-16T04:04:45.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.022+0000 7f84f0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:04:45.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.022+0000 7f84f0ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:04:45.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.022+0000 7f84f0ff9640 10 monclient: dump: 2024-09-16T04:04:45.043 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:04:45.043 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:04:45.043 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:45.043 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:45.043 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:04:45.043 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:04:45.043 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:04:45.043 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:04:45.043 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:04:45.043 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:04:45.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.022+0000 7f84f0ff9640 1 -- 172.21.15.42:0/3781502963 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4703 (secure 0 0 0) 0x7f84ec003e70 con 0x7f84f41cff40 2024-09-16T04:04:45.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.022+0000 7f84f0ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:04:45.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.022+0000 7f84f0ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:04:45.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.022+0000 7f84f0ff9640 1 -- 172.21.15.42:0/3781502963 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f84f41cff40 msgr2=0x7f84f412e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:04:45.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.022+0000 7f84f0ff9640 1 --2- 172.21.15.42:0/3781502963 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f84f41cff40 0x7f84f412e680 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f84ec006c40 tx=0x7f84ec004740 comp rx=0 tx=0).stop 2024-09-16T04:04:45.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.022+0000 7f84f0ff9640 1 -- 172.21.15.42:0/3781502963 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f84f41cff40 msgr2=0x7f84f412e680 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:04:45.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.022+0000 7f84f0ff9640 1 --2- 172.21.15.42:0/3781502963 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f84f41cff40 0x7f84f412e680 secure :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f84ec006c40 tx=0x7f84ec004740 comp rx=0 tx=0).stop 2024-09-16T04:04:45.044 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T04:04:45.044 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:04:45.044 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:04:45.044 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "leader", 2024-09-16T04:04:45.044 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 214, 2024-09-16T04:04:45.045 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 45079, 2024-09-16T04:04:45.045 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T04:04:45.045 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T04:04:45.045 INFO:teuthology.orchestra.run.smithi042.stdout: 1 2024-09-16T04:04:45.045 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:04:45.045 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 0, 2024-09-16T04:04:45.045 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:04:45.045 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T04:04:45.045 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T04:04:45.045 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:04:45.045 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:04:45.045 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:04:45.045 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:04:45.046 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:04:45.046 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:04:45.046 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:04:45.046 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:04:45.046 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:04:45.046 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:04:45.046 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:04:45.046 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:04:45.046 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:04:45.046 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T04:04:45.046 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:04:45.046 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:04:45.046 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:04:45.046 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:04:45.046 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:04:45.046 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:04:45.047 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:04:45.047 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:04:45.047 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:04:45.047 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:04:45.047 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:04:45.047 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:04:45.047 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:04:45.047 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T04:04:45.047 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T04:04:45.047 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T04:04:45.047 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T04:04:45.047 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T04:04:45.047 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T04:04:45.048 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:04:45.048 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:04:45.048 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T04:04:45.048 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T04:04:45.048 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T04:04:45.048 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T04:04:45.048 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T04:04:45.048 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T04:04:45.048 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T04:04:45.048 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:04:45.048 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T04:04:45.048 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:04:45.048 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:04:45.048 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:04:45.049 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:04:45.049 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:04:45.049 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:04:45.049 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:04:45.049 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:04:45.049 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:04:45.049 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:04:45.049 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:04:45.049 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:04:45.049 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T04:04:45.049 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:04:45.049 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T04:04:45.049 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:04:45.049 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:04:45.050 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:04:45.050 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:04:45.050 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:04:45.050 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:04:45.050 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:04:45.050 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T04:04:45.050 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:04:45.050 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:04:45.050 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:04:45.050 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:04:45.050 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T04:04:45.050 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:04:45.050 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:04:45.050 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:04:45.051 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:04:45.051 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T04:04:45.051 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T04:04:45.051 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:04:45.051 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:04:45.051 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:04:45.051 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:04:45.051 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:04:45.051 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:04:45.051 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:04:45.051 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:04:45.051 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:04:45.051 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:04:45.051 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:04:45.052 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T04:04:45.052 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:04:45.052 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:04:45.052 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:04:45.052 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:04:45.052 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T04:04:45.052 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:04:45.052 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:04:45.052 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:04:45.052 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:04:45.052 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T04:04:45.052 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T04:04:45.052 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:04:45.053 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:04:45.053 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:04:45.053 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:04:45.053 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:04:45.053 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:04:45.053 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:04:45.053 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:04:45.053 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:04:45.053 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:04:45.053 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:04:45.053 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T04:04:45.053 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:04:45.053 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:04:45.054 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:04:45.054 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:04:45.054 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T04:04:45.054 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:04:45.054 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:04:45.055 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:04:45.055 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:04:45.055 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T04:04:45.055 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T04:04:45.056 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:04:45.056 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:04:45.056 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:04:45.056 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:04:45.056 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:04:45.056 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:04:45.056 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T04:04:45.056 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T04:04:45.056 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:04:45.056 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:04:45.056 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:04:45.056 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:04:45.056 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:04:45.057 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:04:45.057 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T04:04:45.057 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:04:45.057 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:04:45.057 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:04:45.057 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 3 2024-09-16T04:04:45.057 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:04:45.057 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:04:45.057 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T04:04:45.057 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:04:45.057 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:04:45.057 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:04:45.058 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 3 2024-09-16T04:04:45.058 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:04:45.058 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:04:45.058 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:04:45.058 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T04:04:45.058 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T04:04:45.060 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.022+0000 7f84fa7a2640 10 librados: watch_flush enter 2024-09-16T04:04:45.060 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.022+0000 7f84fa7a2640 10 librados: watch_flush exit 2024-09-16T04:04:45.060 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.022+0000 7f84fa7a2640 20 client.984207.objecter shutdown clearing up homeless session... 2024-09-16T04:04:45.061 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.022+0000 7f84fa7a2640 10 client.984207.objecter successfully canceled tick 2024-09-16T04:04:45.061 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.022+0000 7f84fa7a2640 1 -- 172.21.15.42:0/3781502963 >> v2:172.21.15.154:6812/254845342 conn(0x7f84e00377c0 msgr2=0x7f84e0039c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:04:45.061 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.022+0000 7f84fa7a2640 1 --2- 172.21.15.42:0/3781502963 >> v2:172.21.15.154:6812/254845342 conn(0x7f84e00377c0 0x7f84e0039c80 secure :-1 s=READY pgs=3833 cs=0 l=1 rev1=1 crypto rx=0x7f84e40ad000 tx=0x7f84e4008040 comp rx=0 tx=0).stop 2024-09-16T04:04:45.061 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.022+0000 7f84fa7a2640 10 monclient: shutdown 2024-09-16T04:04:45.061 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.022+0000 7f84fa7a2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:04:45.061 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.022+0000 7f84fa7a2640 1 -- 172.21.15.42:0/3781502963 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f84f40a3d90 msgr2=0x7f84f41393e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:04:45.061 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.022+0000 7f84f8d3f640 1 -- 172.21.15.42:0/3781502963 reap_dead start 2024-09-16T04:04:45.061 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.022+0000 7f84fa7a2640 1 --2- 172.21.15.42:0/3781502963 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f84f40a3d90 0x7f84f41393e0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f84e800cb10 tx=0x7f84e800a060 comp rx=0 tx=0).stop 2024-09-16T04:04:45.061 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.022+0000 7f84fa7a2640 1 -- 172.21.15.42:0/3781502963 shutdown_connections 2024-09-16T04:04:45.061 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.022+0000 7f84fa7a2640 1 --2- 172.21.15.42:0/3781502963 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f84f40a3d90 0x7f84f41393e0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:45.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.022+0000 7f84fa7a2640 1 -- 172.21.15.42:0/3781502963 >> 172.21.15.42:0/3781502963 conn(0x7f84f40a9fb0 msgr2=0x7f84f412d340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:04:45.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.022+0000 7f84fa7a2640 1 -- 172.21.15.42:0/3781502963 shutdown_connections 2024-09-16T04:04:45.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.022+0000 7f84fa7a2640 1 -- 172.21.15.42:0/3781502963 wait complete. 2024-09-16T04:04:45.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.022+0000 7f84fa7a2640 1 librados: shutdown 2024-09-16T04:04:45.062 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T04:04:45.062 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-16T04:04:45.062 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T04:04:45.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.174+0000 7f2db5a69640 10 monclient: get_monmap_and_config 2024-09-16T04:04:45.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.174+0000 7f2db5a69640 10 monclient: build_initial_monmap 2024-09-16T04:04:45.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.174+0000 7f2db5a69640 10 monclient: monmap: 2024-09-16T04:04:45.178 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:04:45.178 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:04:45.178 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:04:45.176327+0000 2024-09-16T04:04:45.178 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:04:45.176327+0000 2024-09-16T04:04:45.179 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:04:45.179 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:04:45.179 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:04:45.179 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:04:45.179 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:04:45.179 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:04:45.179 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:04:45.179 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:04:45.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.174+0000 7f2db5a69640 1 Processor -- start 2024-09-16T04:04:45.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.174+0000 7f2db5a69640 1 -- start start 2024-09-16T04:04:45.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.174+0000 7f2db5a69640 10 monclient: init 2024-09-16T04:04:45.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.174+0000 7f2db5a69640 10 monclient: _reopen_session rank -1 2024-09-16T04:04:45.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.174+0000 7f2db5a69640 10 monclient: _add_conns ranks=[4,1,3,0,2] 2024-09-16T04:04:45.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.174+0000 7f2db5a69640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f2db00a2f10 0x7f2db00a1330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:04:45.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.174+0000 7f2db5a69640 10 monclient(hunting): picked mon.noname-c con 0x7f2db00a2f10 addr v2:172.21.15.154:3300/0 2024-09-16T04:04:45.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.174+0000 7f2db5a69640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2db009e070 addr v1:172.21.15.154:6789/0 2024-09-16T04:04:45.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.174+0000 7f2db5a69640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f2db0140b30 0x7f2db0142f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:04:45.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.174+0000 7f2db5a69640 10 monclient(hunting): picked mon.noname-a con 0x7f2db0140b30 addr v2:172.21.15.42:3300/0 2024-09-16T04:04:45.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.174+0000 7f2db5a69640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2db009dc50 con 0x7f2db009e070 2024-09-16T04:04:45.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.174+0000 7f2db5a69640 10 monclient(hunting): start opening mon connection 2024-09-16T04:04:45.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.174+0000 7f2db5a69640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f2db00a32f0 con 0x7f2db0140b30 2024-09-16T04:04:45.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.174+0000 7f2db5a69640 10 monclient(hunting): start opening mon connection 2024-09-16T04:04:45.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.174+0000 7f2db5a69640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f2db0073290 con 0x7f2db00a2f10 2024-09-16T04:04:45.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.174+0000 7f2db5a69640 10 monclient(hunting): _renew_subs 2024-09-16T04:04:45.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.174+0000 7f2db5a69640 10 monclient(hunting): authenticate will time out at 2332.145547s 2024-09-16T04:04:45.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.174+0000 7f2daffff640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f2db0140b30 0x7f2db0142f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:45.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.174+0000 7f2daffff640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f2db0140b30 0x7f2db0142f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:32826/0 (socket says 172.21.15.42:32826) 2024-09-16T04:04:45.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.174+0000 7f2daffff640 1 -- 172.21.15.42:0/4275661647 learned_addr learned my addr 172.21.15.42:0/4275661647 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:04:45.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.174+0000 7f2daffff640 10 monclient(hunting): get_auth_request con 0x7f2db0140b30 auth_method 0 2024-09-16T04:04:45.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.174+0000 7f2daffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:04:45.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.174+0000 7f2daffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:04:45.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.174+0000 7f2daffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:04:45.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.174+0000 7f2daffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:04:45.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.174+0000 7f2daffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:04:45.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.174+0000 7f2daffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:04:45.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.174+0000 7f2daffff640 10 monclient(hunting): handle_auth_done global_id 984216 payload 274 2024-09-16T04:04:45.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.174+0000 7f2daffff640 1 -- 172.21.15.42:0/4275661647 >> v2:172.21.15.154:3300/0 conn(0x7f2db00a2f10 msgr2=0x7f2db00a1330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:04:45.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.174+0000 7f2daf7fe640 1 --2- 172.21.15.42:0/4275661647 >> v2:172.21.15.154:3300/0 conn(0x7f2db00a2f10 0x7f2db00a1330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:45.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.174+0000 7f2daffff640 1 --2- 172.21.15.42:0/4275661647 >> v2:172.21.15.154:3300/0 conn(0x7f2db00a2f10 0x7f2db00a1330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:45.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.174+0000 7f2daffff640 1 -- 172.21.15.42:0/4275661647 >> v1:172.21.15.154:6789/0 conn(0x7f2db009e070 legacy=0x7f2db00a1870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:04:45.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.174+0000 7f2daffff640 10 monclient: _finish_hunting 0 2024-09-16T04:04:45.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.174+0000 7f2daffff640 1 monclient: found mon.noname-a 2024-09-16T04:04:45.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.174+0000 7f2daffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:04:45.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.174+0000 7f2daffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:04:45.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.174+0000 7f2daffff640 1 -- 172.21.15.42:0/4275661647 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2db009e600 con 0x7f2db0140b30 2024-09-16T04:04:45.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.178+0000 7f2daffff640 1 --2- 172.21.15.42:0/4275661647 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2db0140b30 0x7f2db0142f20 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f2db006add0 tx=0x7f2da801c370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f85811a35a7f94a3 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:04:45.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.178+0000 7f2dae7fc640 1 -- 172.21.15.42:0/4275661647 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2da80b0070 con 0x7f2db0140b30 2024-09-16T04:04:45.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.178+0000 7f2dae7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:04:45.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.178+0000 7f2dae7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:04:45.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.178+0000 7f2dae7fc640 10 monclient: dump: 2024-09-16T04:04:45.185 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:04:45.185 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:04:45.185 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:45.185 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:45.185 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:04:45.185 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:04:45.186 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:04:45.186 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:04:45.186 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:04:45.186 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:04:45.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.178+0000 7f2dae7fc640 10 monclient: _finish_auth 0 2024-09-16T04:04:45.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.178+0000 7f2dae7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:04:45.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.178+0000 7f2dae7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:04:45.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.178+0000 7f2dae7fc640 1 -- 172.21.15.42:0/4275661647 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2da80ab600 con 0x7f2db0140b30 2024-09-16T04:04:45.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.178+0000 7f2dae7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:04:45.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.178+0000 7f2dae7fc640 1 -- 172.21.15.42:0/4275661647 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2da80ab9d0 con 0x7f2db0140b30 2024-09-16T04:04:45.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.178+0000 7f2db5a69640 5 monclient: authenticate success, global_id 984216 2024-09-16T04:04:45.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.178+0000 7f2db5a69640 10 monclient: get_monmap_and_config success 2024-09-16T04:04:45.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.178+0000 7f2db5a69640 4 set_mon_vals no callback set 2024-09-16T04:04:45.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.178+0000 7f2dae7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:04:45.187 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.178+0000 7f2dae7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:04:45.187 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.178+0000 7f2dae7fc640 10 monclient: dump: 2024-09-16T04:04:45.187 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:04:45.187 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:04:45.187 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:45.187 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:45.187 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:04:45.187 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:04:45.187 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:04:45.187 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:04:45.187 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:04:45.187 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:04:45.187 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.178+0000 7f2db5a69640 10 monclient: shutdown 2024-09-16T04:04:45.187 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.178+0000 7f2db5a69640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:04:45.188 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.178+0000 7f2db5a69640 1 -- 172.21.15.42:0/4275661647 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2db0140b30 msgr2=0x7f2db0142f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:04:45.188 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.178+0000 7f2db5a69640 1 --2- 172.21.15.42:0/4275661647 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2db0140b30 0x7f2db0142f20 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f2db006add0 tx=0x7f2da801c370 comp rx=0 tx=0).stop 2024-09-16T04:04:45.188 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.178+0000 7f2db5a69640 1 -- 172.21.15.42:0/4275661647 shutdown_connections 2024-09-16T04:04:45.188 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.178+0000 7f2db5a69640 1 --2- 172.21.15.42:0/4275661647 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2db0140b30 0x7f2db0142f20 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:45.188 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.178+0000 7f2db5a69640 1 --2- 172.21.15.42:0/4275661647 >> v2:172.21.15.154:3300/0 conn(0x7f2db00a2f10 0x7f2db00a1330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:45.188 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.178+0000 7f2db5a69640 1 -- 172.21.15.42:0/4275661647 >> 172.21.15.42:0/4275661647 conn(0x7f2db00a2420 msgr2=0x7f2db00a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:04:45.188 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.178+0000 7f2db5a69640 1 -- 172.21.15.42:0/4275661647 shutdown_connections 2024-09-16T04:04:45.188 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.178+0000 7f2db5a69640 1 -- 172.21.15.42:0/4275661647 wait complete. 2024-09-16T04:04:45.193 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.178+0000 7f2db5a69640 10 monclient: build_initial_monmap 2024-09-16T04:04:45.193 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.178+0000 7f2db5a69640 10 monclient: monmap: 2024-09-16T04:04:45.194 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:04:45.194 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:04:45.194 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:04:45.194 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:04:45.194 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:04:45.194 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:04:45.194 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:04:45.194 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:04:45.194 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:04:45.194 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:04:45.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.178+0000 7f2db5a69640 1 librados: starting msgr at 2024-09-16T04:04:45.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.178+0000 7f2db5a69640 1 librados: starting objecter 2024-09-16T04:04:45.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.178+0000 7f2db5a69640 1 Processor -- start 2024-09-16T04:04:45.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.178+0000 7f2db5a69640 1 -- start start 2024-09-16T04:04:45.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.178+0000 7f2db5a69640 1 librados: setting wanted keys 2024-09-16T04:04:45.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.178+0000 7f2db5a69640 1 librados: calling monclient init 2024-09-16T04:04:45.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.178+0000 7f2db5a69640 10 monclient: init 2024-09-16T04:04:45.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.178+0000 7f2db5a69640 10 monclient: _reopen_session rank -1 2024-09-16T04:04:45.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.178+0000 7f2db5a69640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:04:45.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.178+0000 7f2db5a69640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2db009e070 0x7f2db01d3110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:04:45.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.178+0000 7f2db5a69640 10 monclient(hunting): picked mon.noname-b con 0x7f2db009e070 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:04:45.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.178+0000 7f2db5a69640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2db00a2f10 0x7f2db013bc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:04:45.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.178+0000 7f2db5a69640 10 monclient(hunting): picked mon.noname-c con 0x7f2db00a2f10 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:04:45.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.178+0000 7f2db5a69640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2db0140b30 0x7f2db0070960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:04:45.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.178+0000 7f2db5a69640 10 monclient(hunting): picked mon.noname-a con 0x7f2db0140b30 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:04:45.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.178+0000 7f2db5a69640 10 monclient(hunting): start opening mon connection 2024-09-16T04:04:45.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.178+0000 7f2db5a69640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f2db0145e00 con 0x7f2db0140b30 2024-09-16T04:04:45.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.178+0000 7f2db5a69640 10 monclient(hunting): start opening mon connection 2024-09-16T04:04:45.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.178+0000 7f2db5a69640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f2db0145c80 con 0x7f2db009e070 2024-09-16T04:04:45.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.178+0000 7f2db5a69640 10 monclient(hunting): start opening mon connection 2024-09-16T04:04:45.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.178+0000 7f2db5a69640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f2db0145f80 con 0x7f2db00a2f10 2024-09-16T04:04:45.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.178+0000 7f2db5a69640 10 monclient(hunting): _renew_subs 2024-09-16T04:04:45.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.178+0000 7f2db5a69640 10 monclient(hunting): authenticate will time out at 2332.150846s 2024-09-16T04:04:45.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.178+0000 7f2daeffd640 1 -- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2db00a2f10 msgr2=0x7f2db013bc00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T04:04:45.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.178+0000 7f2daeffd640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2db00a2f10 0x7f2db013bc00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:04:45.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.178+0000 7f2daffff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2db0140b30 0x7f2db0070960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:45.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.178+0000 7f2daffff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2db0140b30 0x7f2db0070960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:32828/0 (socket says 172.21.15.42:32828) 2024-09-16T04:04:45.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.178+0000 7f2daffff640 1 -- 172.21.15.42:0/764418570 learned_addr learned my addr 172.21.15.42:0/764418570 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:04:45.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.178+0000 7f2daffff640 10 monclient(hunting): get_auth_request con 0x7f2db0140b30 auth_method 0 2024-09-16T04:04:45.197 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.178+0000 7f2daffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:04:45.197 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.178+0000 7f2daffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:04:45.197 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.178+0000 7f2daffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:04:45.197 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.178+0000 7f2daffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:04:45.197 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.178+0000 7f2daffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:04:45.197 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.178+0000 7f2daffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:04:45.197 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.178+0000 7f2daf7fe640 1 --2- 172.21.15.42:0/764418570 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2db009e070 0x7f2db01d3110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:45.197 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.178+0000 7f2daffff640 10 monclient(hunting): handle_auth_done global_id 984219 payload 824 2024-09-16T04:04:45.197 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.178+0000 7f2daffff640 1 -- 172.21.15.42:0/764418570 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2db00a2f10 msgr2=0x7f2db013bc00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:04:45.197 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.178+0000 7f2daffff640 1 --2- 172.21.15.42:0/764418570 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2db00a2f10 0x7f2db013bc00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:45.197 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.178+0000 7f2daffff640 1 -- 172.21.15.42:0/764418570 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2db009e070 msgr2=0x7f2db01d3110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:04:45.197 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.178+0000 7f2daffff640 1 --2- 172.21.15.42:0/764418570 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2db009e070 0x7f2db01d3110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:45.197 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.178+0000 7f2daffff640 10 monclient: _finish_hunting 0 2024-09-16T04:04:45.198 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.178+0000 7f2daffff640 1 monclient: found mon.noname-a 2024-09-16T04:04:45.198 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.178+0000 7f2daffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:04:45.198 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.178+0000 7f2daffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:04:45.198 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.178+0000 7f2daffff640 1 -- 172.21.15.42:0/764418570 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2db0072600 con 0x7f2db0140b30 2024-09-16T04:04:45.198 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.178+0000 7f2daffff640 1 --2- 172.21.15.42:0/764418570 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2db0140b30 0x7f2db0070960 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f2da80ae040 tx=0x7f2da8019110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:04:45.198 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.178+0000 7f2da7fff640 10 client.?.objecter ms_handle_connect 0x7f2db0140b30 2024-09-16T04:04:45.198 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.178+0000 7f2da7fff640 10 client.?.objecter resend_mon_ops 2024-09-16T04:04:45.198 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.178+0000 7f2da7fff640 1 -- 172.21.15.42:0/764418570 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2da80b0040 con 0x7f2db0140b30 2024-09-16T04:04:45.198 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.178+0000 7f2da7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:04:45.198 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.178+0000 7f2da7fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:04:45.198 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.178+0000 7f2da7fff640 10 monclient: dump: 2024-09-16T04:04:45.198 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:04:45.198 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:04:45.198 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:45.199 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:45.199 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:04:45.199 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:04:45.199 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:04:45.199 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:04:45.199 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:04:45.199 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:04:45.199 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.178+0000 7f2da7fff640 10 monclient: _finish_auth 0 2024-09-16T04:04:45.199 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.178+0000 7f2da7fff640 10 monclient: _check_auth_tickets 2024-09-16T04:04:45.199 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.178+0000 7f2da7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:04:45.199 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.178+0000 7f2da7fff640 1 -- 172.21.15.42:0/764418570 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2da80b9070 con 0x7f2db0140b30 2024-09-16T04:04:45.199 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.178+0000 7f2da7fff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:04:45.200 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.178+0000 7f2da7fff640 1 -- 172.21.15.42:0/764418570 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2da80b4640 con 0x7f2db0140b30 2024-09-16T04:04:45.200 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.178+0000 7f2db5a69640 5 monclient: authenticate success, global_id 984219 2024-09-16T04:04:45.200 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.178+0000 7f2db5a69640 10 monclient: _renew_subs 2024-09-16T04:04:45.200 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.178+0000 7f2db5a69640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:04:45.200 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.178+0000 7f2db5a69640 1 -- 172.21.15.42:0/764418570 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2db01265b0 con 0x7f2db0140b30 2024-09-16T04:04:45.200 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.178+0000 7f2da7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:04:45.200 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.178+0000 7f2da7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:04:45.200 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.178+0000 7f2da7fff640 10 monclient: dump: 2024-09-16T04:04:45.200 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:04:45.200 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:04:45.200 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:45.200 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:45.200 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:04:45.201 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:04:45.201 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:04:45.201 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:04:45.201 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:04:45.201 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:04:45.201 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.178+0000 7f2dadffb640 4 set_mon_vals no callback set 2024-09-16T04:04:45.201 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.178+0000 7f2db5a69640 10 client.984219.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:04:45.201 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.178+0000 7f2db5a69640 10 monclient: _renew_subs 2024-09-16T04:04:45.201 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.178+0000 7f2db5a69640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:04:45.201 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.178+0000 7f2db5a69640 1 -- 172.21.15.42:0/764418570 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2db0073a30 con 0x7f2db0140b30 2024-09-16T04:04:45.201 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.178+0000 7f2db5a69640 1 librados: init done 2024-09-16T04:04:45.201 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.182+0000 7f2db5a69640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:04:45.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.182+0000 7f2db5a69640 1 --2- 172.21.15.42:0/764418570 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2db013c140 0x7f2db0070ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:04:45.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.182+0000 7f2db5a69640 10 monclient(hunting): start opening mon connection 2024-09-16T04:04:45.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.182+0000 7f2db5a69640 1 -- 172.21.15.42:0/764418570 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f2db00a1330 con 0x7f2db013c140 2024-09-16T04:04:45.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.182+0000 7f2da7fff640 1 -- 172.21.15.42:0/764418570 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f2da80b47e0 con 0x7f2db0140b30 2024-09-16T04:04:45.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.182+0000 7f2da7fff640 1 --2- 172.21.15.42:0/764418570 >> v2:172.21.15.154:6812/254845342 conn(0x7f2d900377c0 0x7f2d90039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:04:45.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.182+0000 7f2da7fff640 1 -- 172.21.15.42:0/764418570 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(682..682 src has 451..682) ==== 6280+0+0 (secure 0 0 0) 0x7f2da80e2ef0 con 0x7f2db0140b30 2024-09-16T04:04:45.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.182+0000 7f2da7fff640 10 client.984219.objecter ms_dispatch 0x7f2db0000ba0 osd_map(682..682 src has 451..682) 2024-09-16T04:04:45.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.182+0000 7f2da7fff640 3 client.984219.objecter handle_osd_map got epochs [682,682] > 0 2024-09-16T04:04:45.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.182+0000 7f2da7fff640 3 client.984219.objecter handle_osd_map decoding full epoch 682 2024-09-16T04:04:45.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.182+0000 7f2da7fff640 20 client.984219.objecter dump_active .. 0 homeless 2024-09-16T04:04:45.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.182+0000 7f2daf7fe640 1 --2- 172.21.15.42:0/764418570 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2db013c140 0x7f2db0070ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:45.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.182+0000 7f2daf7fe640 10 monclient: get_auth_request con 0x7f2db013c140 auth_method 0 2024-09-16T04:04:45.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.182+0000 7f2daf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:04:45.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.182+0000 7f2daf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:04:45.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.182+0000 7f2daf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:04:45.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.182+0000 7f2daeffd640 1 --2- 172.21.15.42:0/764418570 >> v2:172.21.15.154:6812/254845342 conn(0x7f2d900377c0 0x7f2d90039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:45.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.182+0000 7f2daeffd640 10 monclient: get_auth_request con 0x7f2d900377c0 auth_method 0 2024-09-16T04:04:45.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.182+0000 7f2daf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:04:45.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.182+0000 7f2daf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:04:45.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.182+0000 7f2daf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:04:45.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.182+0000 7f2daf7fe640 10 monclient(hunting): handle_auth_done global_id 904846 payload 824 2024-09-16T04:04:45.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.182+0000 7f2daf7fe640 1 -- 172.21.15.42:0/764418570 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2db00a4880 con 0x7f2db013c140 2024-09-16T04:04:45.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.182+0000 7f2da7fff640 10 client.984219.objecter ms_handle_connect 0x7f2d900377c0 2024-09-16T04:04:45.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.182+0000 7f2daeffd640 1 --2- 172.21.15.42:0/764418570 >> v2:172.21.15.154:6812/254845342 conn(0x7f2d900377c0 0x7f2d90039c80 secure :-1 s=READY pgs=3834 cs=0 l=1 rev1=1 crypto rx=0x7f2db01267c0 tx=0x7f2d98007a20 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:04:45.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.182+0000 7f2da7fff640 10 client.984219.objecter ms_handle_connect 0x7f2db013c140 2024-09-16T04:04:45.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.182+0000 7f2da7fff640 10 client.984219.objecter resend_mon_ops 2024-09-16T04:04:45.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.182+0000 7f2daf7fe640 1 --2- 172.21.15.42:0/764418570 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2db013c140 0x7f2db0070ea0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f2da000d9e0 tx=0x7f2da000dea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:04:45.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.182+0000 7f2da7fff640 1 -- 172.21.15.42:0/764418570 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2da001b070 con 0x7f2db013c140 2024-09-16T04:04:45.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.182+0000 7f2da7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:04:45.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.182+0000 7f2da7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:04:45.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.182+0000 7f2da7fff640 10 monclient: dump: 2024-09-16T04:04:45.204 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:04:45.204 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:04:45.204 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:45.204 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:45.204 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:04:45.204 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:04:45.204 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:04:45.204 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:04:45.204 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:04:45.205 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:04:45.205 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.182+0000 7f2da7fff640 1 -- 172.21.15.42:0/764418570 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f2da000be50 con 0x7f2db013c140 2024-09-16T04:04:45.205 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.182+0000 7f2da7fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:04:45.205 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.182+0000 7f2da7fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:04:45.205 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.182+0000 7f2da7fff640 1 -- 172.21.15.42:0/764418570 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2db013c140 msgr2=0x7f2db0070ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:04:45.205 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.182+0000 7f2da7fff640 1 --2- 172.21.15.42:0/764418570 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2db013c140 0x7f2db0070ea0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f2da000d9e0 tx=0x7f2da000dea0 comp rx=0 tx=0).stop 2024-09-16T04:04:45.205 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.182+0000 7f2da7fff640 1 -- 172.21.15.42:0/764418570 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2db013c140 msgr2=0x7f2db0070ea0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:04:45.205 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.182+0000 7f2da7fff640 1 --2- 172.21.15.42:0/764418570 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2db013c140 0x7f2db0070ea0 secure :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f2da000d9e0 tx=0x7f2da000dea0 comp rx=0 tx=0).stop 2024-09-16T04:04:45.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.190+0000 7f2db5a69640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:04:45.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.190+0000 7f2db5a69640 1 --2- 172.21.15.42:0/764418570 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2db00713e0 0x7f2db006d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:04:45.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.190+0000 7f2db5a69640 10 monclient(hunting): start opening mon connection 2024-09-16T04:04:45.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.190+0000 7f2db5a69640 1 -- 172.21.15.42:0/764418570 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f2db00717f0 con 0x7f2db00713e0 2024-09-16T04:04:45.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.190+0000 7f2daf7fe640 1 --2- 172.21.15.42:0/764418570 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2db00713e0 0x7f2db006d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:04:45.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.190+0000 7f2daf7fe640 10 monclient: get_auth_request con 0x7f2db00713e0 auth_method 0 2024-09-16T04:04:45.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.190+0000 7f2daf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:04:45.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.190+0000 7f2daf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:04:45.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.190+0000 7f2daf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:04:45.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.190+0000 7f2daf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:04:45.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.190+0000 7f2daf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:04:45.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.190+0000 7f2daf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:04:45.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.190+0000 7f2daf7fe640 10 monclient(hunting): handle_auth_done global_id 904849 payload 824 2024-09-16T04:04:45.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.190+0000 7f2daf7fe640 1 -- 172.21.15.42:0/764418570 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f2db01251c0 con 0x7f2db00713e0 2024-09-16T04:04:45.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.190+0000 7f2daf7fe640 1 --2- 172.21.15.42:0/764418570 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2db00713e0 0x7f2db006d900 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f2da000d9e0 tx=0x7f2da0004a50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:04:45.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.190+0000 7f2da7fff640 10 client.984219.objecter ms_handle_connect 0x7f2db00713e0 2024-09-16T04:04:45.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.190+0000 7f2da7fff640 10 client.984219.objecter resend_mon_ops 2024-09-16T04:04:45.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.190+0000 7f2da7fff640 1 -- 172.21.15.42:0/764418570 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2da001b040 con 0x7f2db00713e0 2024-09-16T04:04:45.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.190+0000 7f2da7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:04:45.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.190+0000 7f2da7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:04:45.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.190+0000 7f2da7fff640 10 monclient: dump: 2024-09-16T04:04:45.208 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:04:45.208 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:04:45.208 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:45.208 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:04:45.209 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:04:45.209 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:04:45.209 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:04:45.209 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:04:45.209 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:04:45.209 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:04:45.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.190+0000 7f2da7fff640 1 -- 172.21.15.42:0/764418570 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4869 (secure 0 0 0) 0x7f2da000e3f0 con 0x7f2db00713e0 2024-09-16T04:04:45.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.190+0000 7f2da7fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:04:45.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.190+0000 7f2da7fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:04:45.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.190+0000 7f2da7fff640 1 -- 172.21.15.42:0/764418570 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2db00713e0 msgr2=0x7f2db006d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:04:45.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.190+0000 7f2da7fff640 1 --2- 172.21.15.42:0/764418570 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2db00713e0 0x7f2db006d900 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f2da000d9e0 tx=0x7f2da0004a50 comp rx=0 tx=0).stop 2024-09-16T04:04:45.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.190+0000 7f2da7fff640 1 -- 172.21.15.42:0/764418570 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2db00713e0 msgr2=0x7f2db006d900 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:04:45.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.190+0000 7f2da7fff640 1 --2- 172.21.15.42:0/764418570 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2db00713e0 0x7f2db006d900 secure :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f2da000d9e0 tx=0x7f2da0004a50 comp rx=0 tx=0).stop 2024-09-16T04:04:45.210 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T04:04:45.210 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:04:45.210 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:04:45.210 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T04:04:45.210 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 214, 2024-09-16T04:04:45.210 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 413370, 2024-09-16T04:04:45.210 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T04:04:45.210 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T04:04:45.210 INFO:teuthology.orchestra.run.smithi042.stdout: 1 2024-09-16T04:04:45.211 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:04:45.211 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 0, 2024-09-16T04:04:45.211 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:04:45.211 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T04:04:45.211 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T04:04:45.211 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:04:45.211 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:04:45.211 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:04:45.211 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:04:45.211 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:04:45.211 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:04:45.211 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:04:45.211 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:04:45.212 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:04:45.212 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:04:45.212 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:04:45.212 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:04:45.212 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:04:45.212 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T04:04:45.212 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:04:45.212 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:04:45.212 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:04:45.212 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:04:45.212 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:04:45.213 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:04:45.213 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:04:45.213 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:04:45.213 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:04:45.213 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:04:45.213 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:04:45.213 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:04:45.213 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:04:45.213 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T04:04:45.213 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T04:04:45.213 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T04:04:45.213 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T04:04:45.214 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T04:04:45.214 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T04:04:45.214 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:04:45.214 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:04:45.214 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T04:04:45.214 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T04:04:45.214 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T04:04:45.214 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T04:04:45.214 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T04:04:45.214 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T04:04:45.214 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T04:04:45.214 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:04:45.214 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T04:04:45.215 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:04:45.215 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:04:45.215 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:04:45.215 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:04:45.215 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:04:45.215 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:04:45.215 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:04:45.215 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:04:45.215 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:04:45.215 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:04:45.215 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:04:45.215 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:04:45.215 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T04:04:45.215 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:04:45.216 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T04:04:45.216 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:04:45.216 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:04:45.216 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:04:45.216 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:04:45.216 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:04:45.216 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:04:45.216 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:04:45.216 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T04:04:45.216 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:04:45.216 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:04:45.216 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:04:45.217 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:04:45.217 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T04:04:45.217 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:04:45.217 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:04:45.217 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:04:45.217 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:04:45.217 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T04:04:45.217 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T04:04:45.217 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:04:45.217 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:04:45.217 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:04:45.217 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:04:45.217 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:04:45.217 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:04:45.218 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:04:45.218 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:04:45.218 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:04:45.218 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:04:45.218 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:04:45.218 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T04:04:45.218 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:04:45.218 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:04:45.218 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:04:45.218 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:04:45.218 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T04:04:45.218 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:04:45.219 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:04:45.219 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:04:45.219 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:04:45.219 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T04:04:45.219 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T04:04:45.219 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:04:45.219 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:04:45.219 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:04:45.219 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:04:45.219 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:04:45.219 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:04:45.219 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:04:45.220 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:04:45.220 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:04:45.220 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:04:45.220 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:04:45.220 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T04:04:45.220 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:04:45.220 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:04:45.220 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:04:45.220 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:04:45.220 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T04:04:45.220 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:04:45.220 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:04:45.220 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:04:45.220 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:04:45.221 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T04:04:45.221 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T04:04:45.221 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:04:45.221 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:04:45.221 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:04:45.221 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:04:45.221 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:04:45.221 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:04:45.221 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T04:04:45.221 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T04:04:45.221 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:04:45.221 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:04:45.221 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:04:45.222 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:04:45.222 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:04:45.222 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:04:45.222 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T04:04:45.222 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:04:45.222 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:04:45.222 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:04:45.222 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 3 2024-09-16T04:04:45.222 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:04:45.222 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:04:45.222 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T04:04:45.222 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:04:45.222 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:04:45.222 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:04:45.223 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 6 2024-09-16T04:04:45.223 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:04:45.223 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:04:45.223 INFO:teuthology.orchestra.run.smithi042.stdout: "mgr": [ 2024-09-16T04:04:45.223 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:04:45.223 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:04:45.223 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:04:45.223 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:04:45.223 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:04:45.223 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:04:45.223 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:04:45.223 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T04:04:45.223 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T04:04:45.225 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.190+0000 7f2db5a69640 10 librados: watch_flush enter 2024-09-16T04:04:45.225 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.190+0000 7f2db5a69640 10 librados: watch_flush exit 2024-09-16T04:04:45.225 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.190+0000 7f2db5a69640 20 client.984219.objecter shutdown clearing up homeless session... 2024-09-16T04:04:45.225 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.190+0000 7f2db5a69640 10 client.984219.objecter successfully canceled tick 2024-09-16T04:04:45.225 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.190+0000 7f2db5a69640 1 -- 172.21.15.42:0/764418570 >> v2:172.21.15.154:6812/254845342 conn(0x7f2d900377c0 msgr2=0x7f2d90039c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:04:45.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.190+0000 7f2db5a69640 1 --2- 172.21.15.42:0/764418570 >> v2:172.21.15.154:6812/254845342 conn(0x7f2d900377c0 0x7f2d90039c80 secure :-1 s=READY pgs=3834 cs=0 l=1 rev1=1 crypto rx=0x7f2db01267c0 tx=0x7f2d98007a20 comp rx=0 tx=0).stop 2024-09-16T04:04:45.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.190+0000 7f2db5a69640 10 monclient: shutdown 2024-09-16T04:04:45.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.190+0000 7f2db5a69640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:04:45.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.190+0000 7f2db5a69640 1 -- 172.21.15.42:0/764418570 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2db0140b30 msgr2=0x7f2db0070960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:04:45.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.190+0000 7f2daffff640 1 -- 172.21.15.42:0/764418570 reap_dead start 2024-09-16T04:04:45.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.190+0000 7f2db5a69640 1 --2- 172.21.15.42:0/764418570 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2db0140b30 0x7f2db0070960 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f2da80ae040 tx=0x7f2da8019110 comp rx=0 tx=0).stop 2024-09-16T04:04:45.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.190+0000 7f2db5a69640 1 -- 172.21.15.42:0/764418570 shutdown_connections 2024-09-16T04:04:45.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.190+0000 7f2db5a69640 1 --2- 172.21.15.42:0/764418570 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2db0140b30 0x7f2db0070960 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:04:45.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.190+0000 7f2db5a69640 1 -- 172.21.15.42:0/764418570 >> 172.21.15.42:0/764418570 conn(0x7f2db00a2420 msgr2=0x7f2db0141f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:04:45.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.190+0000 7f2db5a69640 1 -- 172.21.15.42:0/764418570 shutdown_connections 2024-09-16T04:04:45.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.190+0000 7f2db5a69640 1 -- 172.21.15.42:0/764418570 wait complete. 2024-09-16T04:04:45.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:04:45.190+0000 7f2db5a69640 1 librados: shutdown 2024-09-16T04:04:45.227 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T04:04:45.227 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-16T04:05:05.228 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-16T04:05:05.228 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-09-16T04:05:05.228 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-16T04:05:05.228 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-16T04:05:05.231 INFO:tasks.ceph.mon.c:Started 2024-09-16T04:05:05.232 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T04:05:05.232 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:05:05.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.350+0000 7f79e1b98640 10 monclient: get_monmap_and_config 2024-09-16T04:05:05.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.350+0000 7f79e1b98640 10 monclient: build_initial_monmap 2024-09-16T04:05:05.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.350+0000 7f79e1b98640 10 monclient: monmap: 2024-09-16T04:05:05.356 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:05:05.357 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:05:05.357 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:05:05.355035+0000 2024-09-16T04:05:05.357 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:05:05.355035+0000 2024-09-16T04:05:05.357 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:05:05.357 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:05:05.357 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:05:05.357 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:05:05.357 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:05:05.357 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:05:05.357 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:05:05.357 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:05:05.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.350+0000 7f79e1b98640 1 Processor -- start 2024-09-16T04:05:05.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.350+0000 7f79e1b98640 1 -- start start 2024-09-16T04:05:05.360 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.350+0000 7f79e1b98640 10 monclient: init 2024-09-16T04:05:05.360 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.354+0000 7f79e1b98640 10 monclient: _reopen_session rank -1 2024-09-16T04:05:05.360 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.354+0000 7f79e1b98640 10 monclient: _add_conns ranks=[0,2,3,1,4] 2024-09-16T04:05:05.360 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.354+0000 7f79e1b98640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f79dc0a39b0 addr v1:172.21.15.42:6789/0 2024-09-16T04:05:05.360 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.354+0000 7f79e1b98640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f79dc0a3d90 0x7f79dc13c380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:05:05.360 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.354+0000 7f79e1b98640 10 monclient(hunting): picked mon.noname-b con 0x7f79dc0a3d90 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:05:05.360 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.354+0000 7f79e1b98640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f79dc13c8c0 0x7f79dc13ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:05:05.360 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.354+0000 7f79e1b98640 10 monclient(hunting): picked mon.noname-a con 0x7f79dc13c8c0 addr v2:172.21.15.42:3300/0 2024-09-16T04:05:05.360 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.354+0000 7f79e1b98640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f79dc134290 con 0x7f79dc0a39b0 2024-09-16T04:05:05.360 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.354+0000 7f79e1b98640 10 monclient(hunting): start opening mon connection 2024-09-16T04:05:05.360 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.354+0000 7f79e1b98640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f79dc09c140 con 0x7f79dc13c8c0 2024-09-16T04:05:05.361 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.354+0000 7f79e1b98640 10 monclient(hunting): start opening mon connection 2024-09-16T04:05:05.361 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.354+0000 7f79e1b98640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f79dc09c420 con 0x7f79dc0a3d90 2024-09-16T04:05:05.361 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.354+0000 7f79e1b98640 10 monclient(hunting): _renew_subs 2024-09-16T04:05:05.361 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.354+0000 7f79e1b98640 10 monclient(hunting): authenticate will time out at 2352.324160s 2024-09-16T04:05:05.361 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.354+0000 7f79dbfff640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f79dc13c8c0 0x7f79dc13ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:05:05.361 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.354+0000 7f79daffd640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f79dc0a3d90 0x7f79dc13c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:05:05.361 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.354+0000 7f79db7fe640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f79dc0a39b0 0x7f79dc133b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:50294/0 (socket says 172.21.15.42:50294) 2024-09-16T04:05:05.361 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.354+0000 7f79db7fe640 1 -- 172.21.15.42:0/1426499115 learned_addr learned my addr 172.21.15.42:0/1426499115 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T04:05:05.361 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.354+0000 7f79daffd640 10 monclient(hunting): get_auth_request con 0x7f79dc0a3d90 auth_method 0 2024-09-16T04:05:05.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.354+0000 7f79daffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:05:05.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.354+0000 7f79daffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:05:05.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.354+0000 7f79daffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:05:05.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.354+0000 7f79dbfff640 10 monclient(hunting): get_auth_request con 0x7f79dc13c8c0 auth_method 0 2024-09-16T04:05:05.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.354+0000 7f79dbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:05:05.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.354+0000 7f79dbfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:05:05.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.354+0000 7f79dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:05:05.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.354+0000 7f79dbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:05:05.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.354+0000 7f79dbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:05:05.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.354+0000 7f79daffd640 1 -- 172.21.15.42:0/1426499115 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f79dc0a3d90 msgr2=0x7f79dc13c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:05:05.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.354+0000 7f79daffd640 1 -- 172.21.15.42:0/1426499115 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f79dc0a3d90 msgr2=0x7f79dc13c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:05:05.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.354+0000 7f79daffd640 1 --2- 172.21.15.42:0/1426499115 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f79dc0a3d90 0x7f79dc13c380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:05:05.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.354+0000 7f79dbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:05:05.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.354+0000 7f79daffd640 1 --2- 172.21.15.42:0/1426499115 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f79dc0a3d90 0x7f79dc13c380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:05:05.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.354+0000 7f79dbfff640 10 monclient(hunting): handle_auth_done global_id 984222 payload 274 2024-09-16T04:05:05.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.354+0000 7f79dbfff640 1 -- 172.21.15.42:0/1426499115 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f79dc0a3d90 msgr2=0x7f79dc13c380 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:05:05.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.354+0000 7f79dbfff640 1 --2- 172.21.15.42:0/1426499115 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f79dc0a3d90 0x7f79dc13c380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:05.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.354+0000 7f79dbfff640 1 -- 172.21.15.42:0/1426499115 >> v1:172.21.15.42:6789/0 conn(0x7f79dc0a39b0 legacy=0x7f79dc133b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:05:05.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.354+0000 7f79dbfff640 10 monclient: _finish_hunting 0 2024-09-16T04:05:05.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.354+0000 7f79dbfff640 1 monclient: found mon.noname-a 2024-09-16T04:05:05.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.354+0000 7f79da7fc640 1 -- 172.21.15.42:0/1426499115 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2359429846 0 0) 0x7f79dc134290 con 0x7f79dc0a39b0 2024-09-16T04:05:05.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.354+0000 7f79dbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:05:05.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.354+0000 7f79dbfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:05:05.364 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.354+0000 7f79dbfff640 1 -- 172.21.15.42:0/1426499115 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79dc06dd60 con 0x7f79dc13c8c0 2024-09-16T04:05:05.364 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.354+0000 7f79da7fc640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T04:05:05.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.354+0000 7f79dbfff640 1 --2- 172.21.15.42:0/1426499115 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f79dc13c8c0 0x7f79dc13ecb0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f79d4009d90 tx=0x7f79d401c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56feec7598b21c0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:05:05.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.354+0000 7f79da7fc640 1 -- 172.21.15.42:0/1426499115 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79d40a6700 con 0x7f79dc13c8c0 2024-09-16T04:05:05.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.354+0000 7f79da7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:05:05.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.354+0000 7f79da7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:05:05.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.354+0000 7f79da7fc640 10 monclient: dump: 2024-09-16T04:05:05.370 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:05:05.370 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:05:05.370 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:05.371 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:05.371 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:05:05.371 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:05:05.371 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:05:05.371 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:05:05.371 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:05:05.371 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:05:05.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.354+0000 7f79da7fc640 10 monclient: _finish_auth 0 2024-09-16T04:05:05.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.354+0000 7f79da7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:05:05.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.354+0000 7f79da7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:05:05.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.354+0000 7f79da7fc640 1 -- 172.21.15.42:0/1426499115 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f79d40a68a0 con 0x7f79dc13c8c0 2024-09-16T04:05:05.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.354+0000 7f79e1b98640 5 monclient: authenticate success, global_id 984222 2024-09-16T04:05:05.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.354+0000 7f79e1b98640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:05:05.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.354+0000 7f79da7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:05:05.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.354+0000 7f79e1b98640 10 monclient: get_monmap_and_config success 2024-09-16T04:05:05.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.354+0000 7f79da7fc640 1 -- 172.21.15.42:0/1426499115 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79d40a6c70 con 0x7f79dc13c8c0 2024-09-16T04:05:05.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.354+0000 7f79da7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:05:05.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.354+0000 7f79e1b98640 4 set_mon_vals no callback set 2024-09-16T04:05:05.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.354+0000 7f79da7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:05:05.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.354+0000 7f79da7fc640 10 monclient: dump: 2024-09-16T04:05:05.372 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:05:05.372 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:05:05.372 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:05.373 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:05.373 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:05:05.373 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:05:05.373 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:05:05.373 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:05:05.373 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:05:05.373 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:05:05.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.354+0000 7f79e1b98640 10 monclient: shutdown 2024-09-16T04:05:05.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.354+0000 7f79e1b98640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:05:05.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.354+0000 7f79e1b98640 1 -- 172.21.15.42:0/1426499115 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f79dc13c8c0 msgr2=0x7f79dc13ecb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:05:05.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.354+0000 7f79e1b98640 1 --2- 172.21.15.42:0/1426499115 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f79dc13c8c0 0x7f79dc13ecb0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f79d4009d90 tx=0x7f79d401c830 comp rx=0 tx=0).stop 2024-09-16T04:05:05.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.354+0000 7f79e1b98640 1 -- 172.21.15.42:0/1426499115 shutdown_connections 2024-09-16T04:05:05.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.354+0000 7f79e1b98640 1 --2- 172.21.15.42:0/1426499115 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f79dc13c8c0 0x7f79dc13ecb0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:05.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.354+0000 7f79e1b98640 1 --2- 172.21.15.42:0/1426499115 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f79dc0a3d90 0x7f79dc13c380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:05.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.354+0000 7f79e1b98640 1 -- 172.21.15.42:0/1426499115 >> 172.21.15.42:0/1426499115 conn(0x7f79dc0a9fb0 msgr2=0x7f79dc0aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:05:05.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.354+0000 7f79e1b98640 1 -- 172.21.15.42:0/1426499115 shutdown_connections 2024-09-16T04:05:05.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.354+0000 7f79e1b98640 1 -- 172.21.15.42:0/1426499115 wait complete. 2024-09-16T04:05:05.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.358+0000 7f79e1b98640 10 monclient: build_initial_monmap 2024-09-16T04:05:05.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.358+0000 7f79e1b98640 10 monclient: monmap: 2024-09-16T04:05:05.374 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:05:05.374 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:05:05.374 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:05:05.374 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:05:05.374 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:05:05.374 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:05:05.374 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:05:05.375 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:05:05.375 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:05:05.375 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:05:05.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.358+0000 7f79e1b98640 1 librados: starting msgr at 2024-09-16T04:05:05.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.358+0000 7f79e1b98640 1 librados: starting objecter 2024-09-16T04:05:05.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.358+0000 7f79e1b98640 1 Processor -- start 2024-09-16T04:05:05.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.358+0000 7f79e1b98640 1 -- start start 2024-09-16T04:05:05.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.358+0000 7f79e1b98640 1 librados: setting wanted keys 2024-09-16T04:05:05.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.358+0000 7f79e1b98640 1 librados: calling monclient init 2024-09-16T04:05:05.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.358+0000 7f79e1b98640 10 monclient: init 2024-09-16T04:05:05.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.358+0000 7f79e1b98640 10 monclient: _reopen_session rank -1 2024-09-16T04:05:05.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.358+0000 7f79e1b98640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:05:05.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.358+0000 7f79e1b98640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f79dc0a39b0 0x7f79dc130560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:05:05.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.358+0000 7f79e1b98640 10 monclient(hunting): picked mon.noname-b con 0x7f79dc0a39b0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:05:05.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.358+0000 7f79e1b98640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f79dc0a3d90 0x7f79dc130aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:05:05.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.358+0000 7f79e1b98640 10 monclient(hunting): picked mon.noname-c con 0x7f79dc0a3d90 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:05:05.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.358+0000 7f79e1b98640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f79dc13c8c0 0x7f79dc139e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:05:05.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.358+0000 7f79e1b98640 10 monclient(hunting): picked mon.noname-a con 0x7f79dc13c8c0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:05:05.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.358+0000 7f79e1b98640 10 monclient(hunting): start opening mon connection 2024-09-16T04:05:05.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.358+0000 7f79e1b98640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f79dc141da0 con 0x7f79dc13c8c0 2024-09-16T04:05:05.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.358+0000 7f79e1b98640 10 monclient(hunting): start opening mon connection 2024-09-16T04:05:05.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.358+0000 7f79e1b98640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f79dc141c20 con 0x7f79dc0a39b0 2024-09-16T04:05:05.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.358+0000 7f79e1b98640 10 monclient(hunting): start opening mon connection 2024-09-16T04:05:05.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.358+0000 7f79e1b98640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f79dc141f20 con 0x7f79dc0a3d90 2024-09-16T04:05:05.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.358+0000 7f79e1b98640 10 monclient(hunting): _renew_subs 2024-09-16T04:05:05.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.358+0000 7f79e1b98640 10 monclient(hunting): authenticate will time out at 2352.329044s 2024-09-16T04:05:05.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.358+0000 7f79dbfff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f79dc13c8c0 0x7f79dc139e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:05:05.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.358+0000 7f79daffd640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f79dc0a3d90 0x7f79dc130aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:05:05.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.358+0000 7f79dbfff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f79dc13c8c0 0x7f79dc139e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:33310/0 (socket says 172.21.15.42:33310) 2024-09-16T04:05:05.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.358+0000 7f79daffd640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f79dc0a3d90 0x7f79dc130aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:40940/0 (socket says 172.21.15.42:40940) 2024-09-16T04:05:05.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.358+0000 7f79dbfff640 1 -- 172.21.15.42:0/2386428791 learned_addr learned my addr 172.21.15.42:0/2386428791 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:05:05.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.358+0000 7f79dbfff640 10 monclient(hunting): get_auth_request con 0x7f79dc13c8c0 auth_method 0 2024-09-16T04:05:05.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.358+0000 7f79dbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:05:05.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.358+0000 7f79dbfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:05:05.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.358+0000 7f79dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:05:05.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.358+0000 7f79daffd640 10 monclient(hunting): get_auth_request con 0x7f79dc0a3d90 auth_method 0 2024-09-16T04:05:05.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.358+0000 7f79daffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:05:05.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.358+0000 7f79daffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:05:05.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.358+0000 7f79daffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:05:05.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.358+0000 7f79dbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:05:05.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.358+0000 7f79dbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:05:05.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.358+0000 7f79daffd640 1 -- 172.21.15.42:0/2386428791 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f79dc0a3d90 msgr2=0x7f79dc130aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-16T04:05:05.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.358+0000 7f79daffd640 1 -- 172.21.15.42:0/2386428791 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f79dc0a3d90 msgr2=0x7f79dc130aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:05:05.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.358+0000 7f79daffd640 1 --2- 172.21.15.42:0/2386428791 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f79dc0a3d90 0x7f79dc130aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:05:05.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.358+0000 7f79dbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:05:05.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.358+0000 7f79daffd640 1 --2- 172.21.15.42:0/2386428791 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f79dc0a3d90 0x7f79dc130aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:05:05.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.358+0000 7f79db7fe640 1 --2- 172.21.15.42:0/2386428791 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f79dc0a39b0 0x7f79dc130560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:05:05.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.358+0000 7f79dbfff640 10 monclient(hunting): handle_auth_done global_id 984228 payload 824 2024-09-16T04:05:05.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.358+0000 7f79dbfff640 1 -- 172.21.15.42:0/2386428791 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f79dc0a3d90 msgr2=0x7f79dc130aa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:05:05.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.358+0000 7f79dbfff640 1 --2- 172.21.15.42:0/2386428791 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f79dc0a3d90 0x7f79dc130aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:05.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.358+0000 7f79dbfff640 1 -- 172.21.15.42:0/2386428791 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f79dc0a39b0 msgr2=0x7f79dc130560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:05:05.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.358+0000 7f79dbfff640 1 --2- 172.21.15.42:0/2386428791 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f79dc0a39b0 0x7f79dc130560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:05.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.358+0000 7f79dbfff640 10 monclient: _finish_hunting 0 2024-09-16T04:05:05.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.358+0000 7f79dbfff640 1 monclient: found mon.noname-a 2024-09-16T04:05:05.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.358+0000 7f79dbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:05:05.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.358+0000 7f79dbfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:05:05.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.358+0000 7f79dbfff640 1 -- 172.21.15.42:0/2386428791 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79dc09e600 con 0x7f79dc13c8c0 2024-09-16T04:05:05.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.358+0000 7f79db7fe640 10 monclient: get_auth_request con 0x7f79dc0a39b0 auth_method 0 2024-09-16T04:05:05.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.358+0000 7f79db7fe640 1 --2- 172.21.15.42:0/2386428791 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f79dc0a39b0 0x7f79dc130560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T04:05:05.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.358+0000 7f79dbfff640 1 --2- 172.21.15.42:0/2386428791 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f79dc13c8c0 0x7f79dc139e40 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f79d401aee0 tx=0x7f79d40195d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:05:05.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.358+0000 7f79c3fff640 10 client.?.objecter ms_handle_connect 0x7f79dc13c8c0 2024-09-16T04:05:05.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.358+0000 7f79c3fff640 10 client.?.objecter resend_mon_ops 2024-09-16T04:05:05.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.358+0000 7f79c3fff640 1 -- 172.21.15.42:0/2386428791 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79d40be070 con 0x7f79dc13c8c0 2024-09-16T04:05:05.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.358+0000 7f79c3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:05:05.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.358+0000 7f79c3fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:05:05.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.358+0000 7f79c3fff640 10 monclient: dump: 2024-09-16T04:05:05.379 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:05:05.380 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:05:05.380 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:05.380 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:05.380 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:05:05.380 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:05:05.380 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:05:05.380 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:05:05.380 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:05:05.380 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:05:05.380 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.358+0000 7f79c3fff640 10 monclient: _finish_auth 0 2024-09-16T04:05:05.380 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.358+0000 7f79c3fff640 10 monclient: _check_auth_tickets 2024-09-16T04:05:05.380 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.358+0000 7f79c3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:05:05.380 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.358+0000 7f79c3fff640 1 -- 172.21.15.42:0/2386428791 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f79d40b8900 con 0x7f79dc13c8c0 2024-09-16T04:05:05.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.358+0000 7f79e1b98640 5 monclient: authenticate success, global_id 984228 2024-09-16T04:05:05.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.358+0000 7f79e1b98640 10 monclient: _renew_subs 2024-09-16T04:05:05.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.358+0000 7f79e1b98640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:05:05.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.358+0000 7f79e1b98640 1 -- 172.21.15.42:0/2386428791 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79dc13c400 con 0x7f79dc13c8c0 2024-09-16T04:05:05.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.358+0000 7f79c3fff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:05:05.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.358+0000 7f79c3fff640 1 -- 172.21.15.42:0/2386428791 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79d40b8cd0 con 0x7f79dc13c8c0 2024-09-16T04:05:05.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.358+0000 7f79c3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:05:05.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.358+0000 7f79c3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:05:05.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.358+0000 7f79c3fff640 10 monclient: dump: 2024-09-16T04:05:05.381 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:05:05.381 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:05:05.381 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:05.381 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:05.381 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:05:05.382 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:05:05.382 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:05:05.382 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:05:05.382 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:05:05.382 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:05:05.382 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.358+0000 7f79e1b98640 10 client.984228.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:05:05.382 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.358+0000 7f79e1b98640 10 monclient: _renew_subs 2024-09-16T04:05:05.382 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.358+0000 7f79e1b98640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:05:05.382 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.358+0000 7f79e1b98640 1 -- 172.21.15.42:0/2386428791 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79dc09e040 con 0x7f79dc13c8c0 2024-09-16T04:05:05.382 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.358+0000 7f79e1b98640 1 librados: init done 2024-09-16T04:05:05.382 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.358+0000 7f79d9ffb640 4 set_mon_vals no callback set 2024-09-16T04:05:05.382 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.358+0000 7f79e1b98640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:05:05.382 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.358+0000 7f79e1b98640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:05:05.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.358+0000 7f79e1b98640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:05:05.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.358+0000 7f79c3fff640 1 -- 172.21.15.42:0/2386428791 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f79d40c1070 con 0x7f79dc13c8c0 2024-09-16T04:05:05.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.358+0000 7f79e1b98640 1 -- 172.21.15.42:0/2386428791 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f79dc133b80 con 0x7f79dc13c8c0 2024-09-16T04:05:05.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.358+0000 7f79c3fff640 1 --2- 172.21.15.42:0/2386428791 >> v2:172.21.15.154:6812/254845342 conn(0x7f79c8037810 0x7f79c8039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:05:05.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.358+0000 7f79c3fff640 1 -- 172.21.15.42:0/2386428791 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(693..693 src has 451..693) ==== 6272+0+0 (secure 0 0 0) 0x7f79d40e4a70 con 0x7f79dc13c8c0 2024-09-16T04:05:05.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.358+0000 7f79c3fff640 10 client.984228.objecter ms_dispatch 0x7f79dc000ba0 osd_map(693..693 src has 451..693) 2024-09-16T04:05:05.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.358+0000 7f79c3fff640 3 client.984228.objecter handle_osd_map got epochs [693,693] > 0 2024-09-16T04:05:05.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.358+0000 7f79c3fff640 3 client.984228.objecter handle_osd_map decoding full epoch 693 2024-09-16T04:05:05.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.358+0000 7f79c3fff640 20 client.984228.objecter dump_active .. 0 homeless 2024-09-16T04:05:05.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.358+0000 7f79db7fe640 1 --2- 172.21.15.42:0/2386428791 >> v2:172.21.15.154:6812/254845342 conn(0x7f79c8037810 0x7f79c8039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:05:05.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.358+0000 7f79db7fe640 10 monclient: get_auth_request con 0x7f79c8037810 auth_method 0 2024-09-16T04:05:05.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.362+0000 7f79db7fe640 1 --2- 172.21.15.42:0/2386428791 >> v2:172.21.15.154:6812/254845342 conn(0x7f79c8037810 0x7f79c8039cd0 secure :-1 s=READY pgs=3836 cs=0 l=1 rev1=1 crypto rx=0x7f79cc007fd0 tx=0x7f79cc009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:05:05.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.362+0000 7f79c3fff640 10 client.984228.objecter ms_handle_connect 0x7f79c8037810 2024-09-16T04:05:05.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.366+0000 7f79c3fff640 1 -- 172.21.15.42:0/2386428791 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f79d4016540 con 0x7f79dc13c8c0 2024-09-16T04:05:05.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.366+0000 7f79c3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:05:05.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.366+0000 7f79c3fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:05:05.596 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.590+0000 7f79e1b98640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:05:05.596 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.590+0000 7f79e1b98640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:05:05.596 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.590+0000 7f79e1b98640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:05:05.596 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.590+0000 7f79e1b98640 1 -- 172.21.15.42:0/2386428791 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f79dc0a5750 con 0x7f79dc13c8c0 2024-09-16T04:05:05.597 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.590+0000 7f79c3fff640 1 -- 172.21.15.42:0/2386428791 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7f79d40ad3d0 con 0x7f79dc13c8c0 2024-09-16T04:05:05.597 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.590+0000 7f79c3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:05:05.597 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.590+0000 7f79c3fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:05:05.597 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":214,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":21,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:05:05.599 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.594+0000 7f79e1b98640 10 librados: watch_flush enter 2024-09-16T04:05:05.599 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.594+0000 7f79e1b98640 10 librados: watch_flush exit 2024-09-16T04:05:05.599 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.594+0000 7f79e1b98640 20 client.984228.objecter shutdown clearing up homeless session... 2024-09-16T04:05:05.599 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.594+0000 7f79e1b98640 10 client.984228.objecter successfully canceled tick 2024-09-16T04:05:05.599 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.594+0000 7f79e1b98640 1 -- 172.21.15.42:0/2386428791 >> v2:172.21.15.154:6812/254845342 conn(0x7f79c8037810 msgr2=0x7f79c8039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:05:05.599 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.594+0000 7f79e1b98640 1 --2- 172.21.15.42:0/2386428791 >> v2:172.21.15.154:6812/254845342 conn(0x7f79c8037810 0x7f79c8039cd0 secure :-1 s=READY pgs=3836 cs=0 l=1 rev1=1 crypto rx=0x7f79cc007fd0 tx=0x7f79cc009040 comp rx=0 tx=0).stop 2024-09-16T04:05:05.599 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.594+0000 7f79e1b98640 10 monclient: shutdown 2024-09-16T04:05:05.600 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.594+0000 7f79e1b98640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:05:05.600 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.594+0000 7f79e1b98640 1 -- 172.21.15.42:0/2386428791 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f79dc13c8c0 msgr2=0x7f79dc139e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:05:05.600 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.594+0000 7f79e1b98640 1 --2- 172.21.15.42:0/2386428791 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f79dc13c8c0 0x7f79dc139e40 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f79d401aee0 tx=0x7f79d40195d0 comp rx=0 tx=0).stop 2024-09-16T04:05:05.600 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.594+0000 7f79e1b98640 1 -- 172.21.15.42:0/2386428791 shutdown_connections 2024-09-16T04:05:05.600 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.594+0000 7f79e1b98640 1 --2- 172.21.15.42:0/2386428791 >> v2:172.21.15.154:6812/254845342 conn(0x7f79c8037810 0x7f79c8039cd0 unknown :-1 s=CLOSED pgs=3836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:05.600 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.594+0000 7f79e1b98640 1 --2- 172.21.15.42:0/2386428791 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f79dc13c8c0 0x7f79dc139e40 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:05.600 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.594+0000 7f79e1b98640 1 --2- 172.21.15.42:0/2386428791 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f79dc0a3d90 0x7f79dc130aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:05.600 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.594+0000 7f79e1b98640 1 --2- 172.21.15.42:0/2386428791 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f79dc0a39b0 0x7f79dc130560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:05.600 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.594+0000 7f79e1b98640 1 -- 172.21.15.42:0/2386428791 >> 172.21.15.42:0/2386428791 conn(0x7f79dc0a9fb0 msgr2=0x7f79dc0a8df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:05:05.600 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.594+0000 7f79e1b98640 1 -- 172.21.15.42:0/2386428791 shutdown_connections 2024-09-16T04:05:05.600 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.594+0000 7f79e1b98640 1 -- 172.21.15.42:0/2386428791 wait complete. 2024-09-16T04:05:05.600 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:05.594+0000 7f79e1b98640 1 librados: shutdown 2024-09-16T04:05:08.617 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:05:08.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.742+0000 7fd5e942b640 10 monclient: get_monmap_and_config 2024-09-16T04:05:08.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.742+0000 7fd5e942b640 10 monclient: build_initial_monmap 2024-09-16T04:05:08.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.742+0000 7fd5e942b640 10 monclient: monmap: 2024-09-16T04:05:08.748 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:05:08.748 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:05:08.748 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:05:08.746369+0000 2024-09-16T04:05:08.748 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:05:08.746369+0000 2024-09-16T04:05:08.748 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:05:08.749 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:05:08.749 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:05:08.749 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:05:08.749 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:05:08.749 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:05:08.749 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:05:08.749 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:05:08.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.742+0000 7fd5e942b640 1 Processor -- start 2024-09-16T04:05:08.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.742+0000 7fd5e942b640 1 -- start start 2024-09-16T04:05:08.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.742+0000 7fd5e942b640 10 monclient: init 2024-09-16T04:05:08.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.742+0000 7fd5e942b640 10 monclient: _reopen_session rank -1 2024-09-16T04:05:08.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.742+0000 7fd5e942b640 10 monclient: _add_conns ranks=[0,2,3,4,1] 2024-09-16T04:05:08.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.742+0000 7fd5e942b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd5e409fb00 addr v1:172.21.15.42:6789/0 2024-09-16T04:05:08.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.742+0000 7fd5e942b640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd5e409c160 0x7fd5e4139120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:05:08.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.742+0000 7fd5e942b640 10 monclient(hunting): picked mon.noname-b con 0x7fd5e409c160 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:05:08.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.742+0000 7fd5e942b640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd5e409c5a0 0x7fd5e413b670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:05:08.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.742+0000 7fd5e942b640 10 monclient(hunting): picked mon.noname-a con 0x7fd5e409c5a0 addr v2:172.21.15.42:3300/0 2024-09-16T04:05:08.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.742+0000 7fd5e942b640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd5e40a1e20 con 0x7fd5e409fb00 2024-09-16T04:05:08.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.742+0000 7fd5e942b640 10 monclient(hunting): start opening mon connection 2024-09-16T04:05:08.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.742+0000 7fd5e942b640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7fd5e409e250 con 0x7fd5e409c5a0 2024-09-16T04:05:08.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.742+0000 7fd5e942b640 10 monclient(hunting): start opening mon connection 2024-09-16T04:05:08.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.742+0000 7fd5e942b640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fd5e409ea30 con 0x7fd5e409c160 2024-09-16T04:05:08.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.742+0000 7fd5e942b640 10 monclient(hunting): _renew_subs 2024-09-16T04:05:08.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.742+0000 7fd5e942b640 10 monclient(hunting): authenticate will time out at 2355.715996s 2024-09-16T04:05:08.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.746+0000 7fd5e27fc640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd5e409c160 0x7fd5e4139120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:05:08.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.746+0000 7fd5e2ffd640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fd5e409fb00 0x7fd5e409ba50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:50302/0 (socket says 172.21.15.42:50302) 2024-09-16T04:05:08.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.746+0000 7fd5e2ffd640 1 -- 172.21.15.42:0/2408826321 learned_addr learned my addr 172.21.15.42:0/2408826321 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T04:05:08.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.746+0000 7fd5e27fc640 10 monclient(hunting): get_auth_request con 0x7fd5e409c160 auth_method 0 2024-09-16T04:05:08.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.746+0000 7fd5e27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:05:08.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.746+0000 7fd5e27fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:05:08.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.746+0000 7fd5e27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:05:08.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.746+0000 7fd5e37fe640 1 --2- 172.21.15.42:0/2408826321 >> v2:172.21.15.42:3300/0 conn(0x7fd5e409c5a0 0x7fd5e413b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:05:08.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.746+0000 7fd5e27fc640 1 -- 172.21.15.42:0/2408826321 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd5e409c160 msgr2=0x7fd5e4139120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:05:08.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.746+0000 7fd5e27fc640 1 -- 172.21.15.42:0/2408826321 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd5e409c160 msgr2=0x7fd5e4139120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:05:08.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.746+0000 7fd5e27fc640 1 --2- 172.21.15.42:0/2408826321 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd5e409c160 0x7fd5e4139120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:05:08.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.746+0000 7fd5e27fc640 1 --2- 172.21.15.42:0/2408826321 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd5e409c160 0x7fd5e4139120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:05:08.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.746+0000 7fd5e37fe640 10 monclient(hunting): get_auth_request con 0x7fd5e409c5a0 auth_method 0 2024-09-16T04:05:08.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.746+0000 7fd5e37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:05:08.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.746+0000 7fd5e37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:05:08.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.746+0000 7fd5e37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:05:08.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.746+0000 7fd5e37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:05:08.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.746+0000 7fd5e37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:05:08.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.746+0000 7fd5e37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:05:08.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.746+0000 7fd5e1ffb640 1 -- 172.21.15.42:0/2408826321 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3270385076 0 0) 0x7fd5e40a1e20 con 0x7fd5e409fb00 2024-09-16T04:05:08.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.746+0000 7fd5e1ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:05:08.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.746+0000 7fd5e1ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:05:08.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.746+0000 7fd5e1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:05:08.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.746+0000 7fd5e1ffb640 10 monclient(hunting): my global_id is 984231 2024-09-16T04:05:08.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.746+0000 7fd5e1ffb640 1 -- 172.21.15.42:0/2408826321 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd5d0003530 con 0x7fd5e409fb00 2024-09-16T04:05:08.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.746+0000 7fd5e37fe640 10 monclient(hunting): handle_auth_done global_id 984234 payload 274 2024-09-16T04:05:08.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.746+0000 7fd5e37fe640 1 -- 172.21.15.42:0/2408826321 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd5e409c160 msgr2=0x7fd5e4139120 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:05:08.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.746+0000 7fd5e37fe640 1 --2- 172.21.15.42:0/2408826321 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd5e409c160 0x7fd5e4139120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:08.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.746+0000 7fd5e37fe640 1 -- 172.21.15.42:0/2408826321 >> v1:172.21.15.42:6789/0 conn(0x7fd5e409fb00 legacy=0x7fd5e409ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:05:08.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.746+0000 7fd5e37fe640 10 monclient: _finish_hunting 0 2024-09-16T04:05:08.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.746+0000 7fd5e37fe640 1 monclient: found mon.noname-a 2024-09-16T04:05:08.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.746+0000 7fd5e37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:05:08.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.746+0000 7fd5e37fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:05:08.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.746+0000 7fd5e37fe640 1 -- 172.21.15.42:0/2408826321 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5e409fee0 con 0x7fd5e409c5a0 2024-09-16T04:05:08.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.746+0000 7fd5e37fe640 1 --2- 172.21.15.42:0/2408826321 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd5e409c5a0 0x7fd5e413b670 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fd5dc0097c0 tx=0x7fd5dc01c360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4874ef6c4852ca23 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:05:08.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.746+0000 7fd5e1ffb640 1 -- 172.21.15.42:0/2408826321 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5dc0b0070 con 0x7fd5e409c5a0 2024-09-16T04:05:08.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.746+0000 7fd5e1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:05:08.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.746+0000 7fd5e1ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:05:08.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.746+0000 7fd5e1ffb640 10 monclient: dump: 2024-09-16T04:05:08.758 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:05:08.758 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:05:08.758 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:08.758 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:08.758 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:05:08.758 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:05:08.758 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:05:08.758 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:05:08.758 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:05:08.758 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:05:08.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.746+0000 7fd5e1ffb640 10 monclient: _finish_auth 0 2024-09-16T04:05:08.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.746+0000 7fd5e1ffb640 10 monclient: _check_auth_tickets 2024-09-16T04:05:08.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.746+0000 7fd5e1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:05:08.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.746+0000 7fd5e1ffb640 1 -- 172.21.15.42:0/2408826321 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd5dc0ab600 con 0x7fd5e409c5a0 2024-09-16T04:05:08.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.746+0000 7fd5e1ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:05:08.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.746+0000 7fd5e1ffb640 1 -- 172.21.15.42:0/2408826321 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5dc0a4e60 con 0x7fd5e409c5a0 2024-09-16T04:05:08.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.746+0000 7fd5e942b640 5 monclient: authenticate success, global_id 984234 2024-09-16T04:05:08.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.746+0000 7fd5e942b640 10 monclient: get_monmap_and_config success 2024-09-16T04:05:08.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.746+0000 7fd5e942b640 4 set_mon_vals no callback set 2024-09-16T04:05:08.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.746+0000 7fd5e1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:05:08.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.746+0000 7fd5e1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:05:08.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.746+0000 7fd5e1ffb640 10 monclient: dump: 2024-09-16T04:05:08.759 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:05:08.759 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:05:08.759 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:08.759 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:08.759 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:05:08.760 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:05:08.760 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:05:08.760 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:05:08.760 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:05:08.760 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:05:08.765 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.746+0000 7fd5e942b640 10 monclient: shutdown 2024-09-16T04:05:08.766 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.746+0000 7fd5e942b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:05:08.766 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.746+0000 7fd5e942b640 1 -- 172.21.15.42:0/2408826321 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd5e409c5a0 msgr2=0x7fd5e413b670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:05:08.766 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.746+0000 7fd5e942b640 1 --2- 172.21.15.42:0/2408826321 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd5e409c5a0 0x7fd5e413b670 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fd5dc0097c0 tx=0x7fd5dc01c360 comp rx=0 tx=0).stop 2024-09-16T04:05:08.766 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.746+0000 7fd5e942b640 1 -- 172.21.15.42:0/2408826321 shutdown_connections 2024-09-16T04:05:08.766 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.746+0000 7fd5e942b640 1 --2- 172.21.15.42:0/2408826321 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd5e409c5a0 0x7fd5e413b670 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:08.766 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.746+0000 7fd5e942b640 1 --2- 172.21.15.42:0/2408826321 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd5e409c160 0x7fd5e4139120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:08.766 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.746+0000 7fd5e942b640 1 -- 172.21.15.42:0/2408826321 >> 172.21.15.42:0/2408826321 conn(0x7fd5e40a2420 msgr2=0x7fd5e40a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:05:08.766 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.746+0000 7fd5e942b640 1 -- 172.21.15.42:0/2408826321 shutdown_connections 2024-09-16T04:05:08.766 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.746+0000 7fd5e942b640 1 -- 172.21.15.42:0/2408826321 wait complete. 2024-09-16T04:05:08.766 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.750+0000 7fd5e942b640 10 monclient: build_initial_monmap 2024-09-16T04:05:08.766 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.750+0000 7fd5e942b640 10 monclient: monmap: 2024-09-16T04:05:08.766 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:05:08.767 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:05:08.767 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:05:08.767 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:05:08.767 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:05:08.767 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:05:08.767 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:05:08.767 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:05:08.767 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:05:08.767 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:05:08.767 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.750+0000 7fd5e942b640 1 librados: starting msgr at 2024-09-16T04:05:08.767 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.750+0000 7fd5e942b640 1 librados: starting objecter 2024-09-16T04:05:08.767 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.750+0000 7fd5e942b640 1 Processor -- start 2024-09-16T04:05:08.767 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.750+0000 7fd5e942b640 1 -- start start 2024-09-16T04:05:08.767 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.750+0000 7fd5e942b640 1 librados: setting wanted keys 2024-09-16T04:05:08.768 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.750+0000 7fd5e942b640 1 librados: calling monclient init 2024-09-16T04:05:08.768 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.750+0000 7fd5e942b640 10 monclient: init 2024-09-16T04:05:08.768 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.750+0000 7fd5e942b640 10 monclient: _reopen_session rank -1 2024-09-16T04:05:08.768 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.750+0000 7fd5e942b640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:05:08.768 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.750+0000 7fd5e942b640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd5e409c160 0x7fd5e4134d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:05:08.768 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.750+0000 7fd5e942b640 10 monclient(hunting): picked mon.noname-c con 0x7fd5e409c160 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:05:08.768 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.750+0000 7fd5e942b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd5e409c5a0 0x7fd5e4135290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:05:08.768 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.750+0000 7fd5e942b640 10 monclient(hunting): picked mon.noname-b con 0x7fd5e409c5a0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:05:08.768 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.750+0000 7fd5e942b640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd5e409fb00 0x7fd5e411db50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:05:08.768 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.750+0000 7fd5e942b640 10 monclient(hunting): picked mon.noname-a con 0x7fd5e409fb00 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:05:08.768 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.750+0000 7fd5e942b640 10 monclient(hunting): start opening mon connection 2024-09-16T04:05:08.768 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.750+0000 7fd5e942b640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fd5e409d380 con 0x7fd5e409fb00 2024-09-16T04:05:08.768 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.750+0000 7fd5e942b640 10 monclient(hunting): start opening mon connection 2024-09-16T04:05:08.769 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.750+0000 7fd5e942b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd5e409d130 con 0x7fd5e409c5a0 2024-09-16T04:05:08.769 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.750+0000 7fd5e942b640 10 monclient(hunting): start opening mon connection 2024-09-16T04:05:08.769 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.750+0000 7fd5e942b640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fd5e409d500 con 0x7fd5e409c160 2024-09-16T04:05:08.769 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.750+0000 7fd5e942b640 10 monclient(hunting): _renew_subs 2024-09-16T04:05:08.769 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.750+0000 7fd5e942b640 10 monclient(hunting): authenticate will time out at 2355.721106s 2024-09-16T04:05:08.769 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.750+0000 7fd5e37fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd5e409fb00 0x7fd5e411db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:05:08.769 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.750+0000 7fd5e37fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd5e409fb00 0x7fd5e411db50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:33348/0 (socket says 172.21.15.42:33348) 2024-09-16T04:05:08.769 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.750+0000 7fd5e37fe640 1 -- 172.21.15.42:0/1677907631 learned_addr learned my addr 172.21.15.42:0/1677907631 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:05:08.769 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.750+0000 7fd5e37fe640 10 monclient(hunting): get_auth_request con 0x7fd5e409fb00 auth_method 0 2024-09-16T04:05:08.769 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.750+0000 7fd5e37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:05:08.769 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.750+0000 7fd5e37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:05:08.769 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.750+0000 7fd5e37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:05:08.769 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.750+0000 7fd5e2ffd640 1 --2- 172.21.15.42:0/1677907631 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd5e409c160 0x7fd5e4134d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:05:08.770 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.750+0000 7fd5e2ffd640 10 monclient(hunting): get_auth_request con 0x7fd5e409c160 auth_method 0 2024-09-16T04:05:08.770 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.750+0000 7fd5e2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:05:08.770 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.750+0000 7fd5e2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:05:08.770 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.750+0000 7fd5e2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:05:08.770 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.750+0000 7fd5e37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:05:08.770 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.750+0000 7fd5e37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:05:08.770 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.750+0000 7fd5e37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:05:08.770 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.750+0000 7fd5e2ffd640 1 -- 172.21.15.42:0/1677907631 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd5e409c160 msgr2=0x7fd5e4134d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-16T04:05:08.770 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.750+0000 7fd5e2ffd640 1 -- 172.21.15.42:0/1677907631 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd5e409c160 msgr2=0x7fd5e4134d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:05:08.770 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.750+0000 7fd5e2ffd640 1 --2- 172.21.15.42:0/1677907631 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd5e409c160 0x7fd5e4134d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:05:08.770 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.750+0000 7fd5e2ffd640 1 --2- 172.21.15.42:0/1677907631 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd5e409c160 0x7fd5e4134d50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:05:08.770 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.750+0000 7fd5e37fe640 10 monclient(hunting): handle_auth_done global_id 984237 payload 824 2024-09-16T04:05:08.770 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.750+0000 7fd5e37fe640 1 -- 172.21.15.42:0/1677907631 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd5e409c160 msgr2=0x7fd5e4134d50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:05:08.770 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.750+0000 7fd5e37fe640 1 --2- 172.21.15.42:0/1677907631 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd5e409c160 0x7fd5e4134d50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:08.771 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.750+0000 7fd5e37fe640 1 -- 172.21.15.42:0/1677907631 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd5e409c5a0 msgr2=0x7fd5e4135290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:05:08.771 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.750+0000 7fd5e37fe640 1 --2- 172.21.15.42:0/1677907631 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd5e409c5a0 0x7fd5e4135290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:08.771 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.750+0000 7fd5e37fe640 10 monclient: _finish_hunting 0 2024-09-16T04:05:08.771 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.750+0000 7fd5e37fe640 1 monclient: found mon.noname-a 2024-09-16T04:05:08.771 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.750+0000 7fd5e37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:05:08.771 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.750+0000 7fd5e37fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:05:08.771 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.750+0000 7fd5e37fe640 1 -- 172.21.15.42:0/1677907631 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5e4121200 con 0x7fd5e409fb00 2024-09-16T04:05:08.771 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.750+0000 7fd5e37fe640 1 --2- 172.21.15.42:0/1677907631 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd5e409fb00 0x7fd5e411db50 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fd5dc0ae040 tx=0x7fd5dc019100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:05:08.771 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.750+0000 7fd5db7fe640 10 client.?.objecter ms_handle_connect 0x7fd5e409fb00 2024-09-16T04:05:08.771 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.750+0000 7fd5db7fe640 10 client.?.objecter resend_mon_ops 2024-09-16T04:05:08.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.750+0000 7fd5db7fe640 1 -- 172.21.15.42:0/1677907631 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5dc0b8020 con 0x7fd5e409fb00 2024-09-16T04:05:08.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.750+0000 7fd5db7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:05:08.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.750+0000 7fd5db7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:05:08.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.750+0000 7fd5db7fe640 10 monclient: dump: 2024-09-16T04:05:08.772 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:05:08.772 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:05:08.772 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:08.772 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:08.772 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:05:08.772 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:05:08.772 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:05:08.772 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:05:08.772 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:05:08.773 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:05:08.773 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.750+0000 7fd5db7fe640 10 monclient: _finish_auth 0 2024-09-16T04:05:08.773 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.750+0000 7fd5db7fe640 10 monclient: _check_auth_tickets 2024-09-16T04:05:08.773 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.750+0000 7fd5db7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:05:08.773 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.750+0000 7fd5db7fe640 1 -- 172.21.15.42:0/1677907631 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd5dc0a4e60 con 0x7fd5e409fb00 2024-09-16T04:05:08.773 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.750+0000 7fd5db7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:05:08.773 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.750+0000 7fd5db7fe640 1 -- 172.21.15.42:0/1677907631 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5dc0b0040 con 0x7fd5e409fb00 2024-09-16T04:05:08.773 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.750+0000 7fd5db7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:05:08.773 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.750+0000 7fd5db7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:05:08.773 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.750+0000 7fd5db7fe640 10 monclient: dump: 2024-09-16T04:05:08.774 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:05:08.774 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:05:08.774 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:08.774 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:08.774 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:05:08.774 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:05:08.774 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:05:08.774 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:05:08.774 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:05:08.774 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:05:08.774 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.750+0000 7fd5e17fa640 4 set_mon_vals no callback set 2024-09-16T04:05:08.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.750+0000 7fd5e942b640 5 monclient: authenticate success, global_id 984237 2024-09-16T04:05:08.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.750+0000 7fd5e942b640 10 monclient: _renew_subs 2024-09-16T04:05:08.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.750+0000 7fd5e942b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:05:08.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.750+0000 7fd5e942b640 1 -- 172.21.15.42:0/1677907631 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5e41217c0 con 0x7fd5e409fb00 2024-09-16T04:05:08.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.750+0000 7fd5e942b640 10 client.984237.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:05:08.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.750+0000 7fd5e942b640 10 monclient: _renew_subs 2024-09-16T04:05:08.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.750+0000 7fd5e942b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:05:08.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.750+0000 7fd5e942b640 1 -- 172.21.15.42:0/1677907631 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5e4120310 con 0x7fd5e409fb00 2024-09-16T04:05:08.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.750+0000 7fd5e942b640 1 librados: init done 2024-09-16T04:05:08.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.750+0000 7fd5e942b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:05:08.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.750+0000 7fd5e942b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:05:08.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.750+0000 7fd5e942b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:05:08.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.750+0000 7fd5e942b640 1 -- 172.21.15.42:0/1677907631 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd5e409ba50 con 0x7fd5e409fb00 2024-09-16T04:05:08.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.750+0000 7fd5db7fe640 1 -- 172.21.15.42:0/1677907631 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fd5dc0bb020 con 0x7fd5e409fb00 2024-09-16T04:05:08.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.750+0000 7fd5db7fe640 1 --2- 172.21.15.42:0/1677907631 >> v2:172.21.15.154:6812/254845342 conn(0x7fd5c4037810 0x7fd5c4039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:05:08.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.750+0000 7fd5db7fe640 1 -- 172.21.15.42:0/1677907631 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(696..696 src has 451..696) ==== 6285+0+0 (secure 0 0 0) 0x7fd5dc0e2aa0 con 0x7fd5e409fb00 2024-09-16T04:05:08.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.750+0000 7fd5db7fe640 10 client.984237.objecter ms_dispatch 0x7fd5e4000ba0 osd_map(696..696 src has 451..696) 2024-09-16T04:05:08.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.750+0000 7fd5db7fe640 3 client.984237.objecter handle_osd_map got epochs [696,696] > 0 2024-09-16T04:05:08.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.750+0000 7fd5db7fe640 3 client.984237.objecter handle_osd_map decoding full epoch 696 2024-09-16T04:05:08.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.750+0000 7fd5db7fe640 20 client.984237.objecter dump_active .. 0 homeless 2024-09-16T04:05:08.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.754+0000 7fd5e2ffd640 1 --2- 172.21.15.42:0/1677907631 >> v2:172.21.15.154:6812/254845342 conn(0x7fd5c4037810 0x7fd5c4039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:05:08.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.754+0000 7fd5e2ffd640 10 monclient: get_auth_request con 0x7fd5c4037810 auth_method 0 2024-09-16T04:05:08.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.754+0000 7fd5e2ffd640 1 --2- 172.21.15.42:0/1677907631 >> v2:172.21.15.154:6812/254845342 conn(0x7fd5c4037810 0x7fd5c4039cd0 secure :-1 s=READY pgs=3837 cs=0 l=1 rev1=1 crypto rx=0x7fd5d4005890 tx=0x7fd5d4009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:05:08.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.754+0000 7fd5db7fe640 10 client.984237.objecter ms_handle_connect 0x7fd5c4037810 2024-09-16T04:05:08.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.758+0000 7fd5db7fe640 1 -- 172.21.15.42:0/1677907631 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7fd5dc0c2030 con 0x7fd5e409fb00 2024-09-16T04:05:08.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.758+0000 7fd5db7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:05:08.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.758+0000 7fd5db7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:05:08.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.986+0000 7fd5e942b640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:05:08.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.986+0000 7fd5e942b640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:05:08.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.986+0000 7fd5e942b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:05:08.992 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.986+0000 7fd5e942b640 1 -- 172.21.15.42:0/1677907631 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fd5e41215a0 con 0x7fd5e409fb00 2024-09-16T04:05:08.992 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.986+0000 7fd5db7fe640 1 -- 172.21.15.42:0/1677907631 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7fd5dc016070 con 0x7fd5e409fb00 2024-09-16T04:05:08.992 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.986+0000 7fd5db7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:05:08.992 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.986+0000 7fd5db7fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:05:08.992 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":214,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":24,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:05:08.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.990+0000 7fd5e942b640 10 librados: watch_flush enter 2024-09-16T04:05:08.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.990+0000 7fd5e942b640 10 librados: watch_flush exit 2024-09-16T04:05:08.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.990+0000 7fd5e942b640 20 client.984237.objecter shutdown clearing up homeless session... 2024-09-16T04:05:08.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.990+0000 7fd5e942b640 10 client.984237.objecter successfully canceled tick 2024-09-16T04:05:08.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.990+0000 7fd5e942b640 1 -- 172.21.15.42:0/1677907631 >> v2:172.21.15.154:6812/254845342 conn(0x7fd5c4037810 msgr2=0x7fd5c4039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:05:08.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.990+0000 7fd5e942b640 1 --2- 172.21.15.42:0/1677907631 >> v2:172.21.15.154:6812/254845342 conn(0x7fd5c4037810 0x7fd5c4039cd0 secure :-1 s=READY pgs=3837 cs=0 l=1 rev1=1 crypto rx=0x7fd5d4005890 tx=0x7fd5d4009040 comp rx=0 tx=0).stop 2024-09-16T04:05:08.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.990+0000 7fd5e942b640 10 monclient: shutdown 2024-09-16T04:05:08.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.990+0000 7fd5e942b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:05:08.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.990+0000 7fd5e942b640 1 -- 172.21.15.42:0/1677907631 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd5e409fb00 msgr2=0x7fd5e411db50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:05:08.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.990+0000 7fd5e942b640 1 --2- 172.21.15.42:0/1677907631 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd5e409fb00 0x7fd5e411db50 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fd5dc0ae040 tx=0x7fd5dc019100 comp rx=0 tx=0).stop 2024-09-16T04:05:08.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.990+0000 7fd5e942b640 1 -- 172.21.15.42:0/1677907631 shutdown_connections 2024-09-16T04:05:08.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.990+0000 7fd5e942b640 1 --2- 172.21.15.42:0/1677907631 >> v2:172.21.15.154:6812/254845342 conn(0x7fd5c4037810 0x7fd5c4039cd0 unknown :-1 s=CLOSED pgs=3837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:08.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.990+0000 7fd5e942b640 1 --2- 172.21.15.42:0/1677907631 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd5e409fb00 0x7fd5e411db50 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:08.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.990+0000 7fd5e942b640 1 --2- 172.21.15.42:0/1677907631 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd5e409c5a0 0x7fd5e4135290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:08.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.990+0000 7fd5e942b640 1 --2- 172.21.15.42:0/1677907631 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd5e409c160 0x7fd5e4134d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:08.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.990+0000 7fd5e942b640 1 -- 172.21.15.42:0/1677907631 >> 172.21.15.42:0/1677907631 conn(0x7fd5e40a2420 msgr2=0x7fd5e40a4fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:05:08.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.990+0000 7fd5e942b640 1 -- 172.21.15.42:0/1677907631 shutdown_connections 2024-09-16T04:05:08.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.990+0000 7fd5e942b640 1 -- 172.21.15.42:0/1677907631 wait complete. 2024-09-16T04:05:08.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:08.990+0000 7fd5e942b640 1 librados: shutdown 2024-09-16T04:05:12.012 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:05:12.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.142+0000 7f3459ad5640 10 monclient: get_monmap_and_config 2024-09-16T04:05:12.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.142+0000 7f3459ad5640 10 monclient: build_initial_monmap 2024-09-16T04:05:12.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.142+0000 7f3459ad5640 10 monclient: monmap: 2024-09-16T04:05:12.147 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:05:12.147 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:05:12.147 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:05:12.145506+0000 2024-09-16T04:05:12.147 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:05:12.145506+0000 2024-09-16T04:05:12.147 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:05:12.147 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:05:12.147 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:05:12.148 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:05:12.148 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:05:12.148 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:05:12.148 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:05:12.148 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:05:12.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.142+0000 7f3459ad5640 1 Processor -- start 2024-09-16T04:05:12.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.142+0000 7f3459ad5640 1 -- start start 2024-09-16T04:05:12.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.142+0000 7f3459ad5640 10 monclient: init 2024-09-16T04:05:12.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.142+0000 7f3459ad5640 10 monclient: _reopen_session rank -1 2024-09-16T04:05:12.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.142+0000 7f3459ad5640 10 monclient: _add_conns ranks=[0,2,4,3,1] 2024-09-16T04:05:12.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.142+0000 7f3459ad5640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f34540a1330 addr v1:172.21.15.42:6789/0 2024-09-16T04:05:12.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.142+0000 7f3459ad5640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f345406dac0 0x7f345413a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:05:12.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.142+0000 7f3459ad5640 10 monclient(hunting): picked mon.noname-b con 0x7f345406dac0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:05:12.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.142+0000 7f3459ad5640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f34540a1e20 0x7f345413c890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:05:12.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.142+0000 7f3459ad5640 10 monclient(hunting): picked mon.noname-c con 0x7f34540a1e20 addr v2:172.21.15.154:3300/0 2024-09-16T04:05:12.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.142+0000 7f3459ad5640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f345409c7d0 con 0x7f34540a1330 2024-09-16T04:05:12.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.142+0000 7f3459ad5640 10 monclient(hunting): start opening mon connection 2024-09-16T04:05:12.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.142+0000 7f3459ad5640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f345409c090 con 0x7f34540a1e20 2024-09-16T04:05:12.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.142+0000 7f3459ad5640 10 monclient(hunting): start opening mon connection 2024-09-16T04:05:12.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.142+0000 7f3459ad5640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f345409f610 con 0x7f345406dac0 2024-09-16T04:05:12.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.142+0000 7f3459ad5640 10 monclient(hunting): _renew_subs 2024-09-16T04:05:12.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.142+0000 7f3459ad5640 10 monclient(hunting): authenticate will time out at 2359.114849s 2024-09-16T04:05:12.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.142+0000 7f34537fe640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f34540a1330 0x7f34540a1710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:50304/0 (socket says 172.21.15.42:50304) 2024-09-16T04:05:12.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.142+0000 7f34537fe640 1 -- 172.21.15.42:0/554089026 learned_addr learned my addr 172.21.15.42:0/554089026 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T04:05:12.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.142+0000 7f3452ffd640 1 --2- 172.21.15.42:0/554089026 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f345406dac0 0x7f345413a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:05:12.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.142+0000 7f3452ffd640 10 monclient(hunting): get_auth_request con 0x7f345406dac0 auth_method 0 2024-09-16T04:05:12.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.142+0000 7f3452ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:05:12.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.142+0000 7f3452ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:05:12.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.142+0000 7f3452ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:05:12.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.142+0000 7f3452ffd640 1 -- 172.21.15.42:0/554089026 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f345406dac0 msgr2=0x7f345413a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:05:12.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.142+0000 7f3452ffd640 1 -- 172.21.15.42:0/554089026 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f345406dac0 msgr2=0x7f345413a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:05:12.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.142+0000 7f3452ffd640 1 --2- 172.21.15.42:0/554089026 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f345406dac0 0x7f345413a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:05:12.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.142+0000 7f3452ffd640 1 --2- 172.21.15.42:0/554089026 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f345406dac0 0x7f345413a340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:05:12.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.142+0000 7f3453fff640 1 --2- 172.21.15.42:0/554089026 >> v2:172.21.15.154:3300/0 conn(0x7f34540a1e20 0x7f345413c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:05:12.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.142+0000 7f3453fff640 10 monclient(hunting): get_auth_request con 0x7f34540a1e20 auth_method 0 2024-09-16T04:05:12.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.142+0000 7f3453fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:05:12.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.142+0000 7f3453fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:05:12.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.142+0000 7f3453fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:05:12.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.142+0000 7f34527fc640 1 -- 172.21.15.42:0/554089026 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1331821274 0 0) 0x7f345409c7d0 con 0x7f34540a1330 2024-09-16T04:05:12.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.142+0000 7f34527fc640 10 monclient(hunting): _negotiate 2024-09-16T04:05:12.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.142+0000 7f34527fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:05:12.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.142+0000 7f34527fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:05:12.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.142+0000 7f34527fc640 10 monclient(hunting): my global_id is 984240 2024-09-16T04:05:12.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.142+0000 7f34527fc640 1 -- 172.21.15.42:0/554089026 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f343c003530 con 0x7f34540a1330 2024-09-16T04:05:12.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.142+0000 7f3453fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:05:12.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.142+0000 7f3453fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:05:12.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.142+0000 7f3453fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:05:12.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.142+0000 7f34527fc640 1 -- 172.21.15.42:0/554089026 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 564193815 0 0) 0x7f343c003530 con 0x7f34540a1330 2024-09-16T04:05:12.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.142+0000 7f34527fc640 1 -- 172.21.15.42:0/554089026 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f345406dac0 msgr2=0x7f345413a340 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:05:12.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.142+0000 7f34527fc640 1 --2- 172.21.15.42:0/554089026 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f345406dac0 0x7f345413a340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:12.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.142+0000 7f34527fc640 1 -- 172.21.15.42:0/554089026 >> v2:172.21.15.154:3300/0 conn(0x7f34540a1e20 msgr2=0x7f345413c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:05:12.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.142+0000 7f34527fc640 1 --2- 172.21.15.42:0/554089026 >> v2:172.21.15.154:3300/0 conn(0x7f34540a1e20 0x7f345413c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:12.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.142+0000 7f34527fc640 10 monclient: _finish_hunting 0 2024-09-16T04:05:12.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.142+0000 7f34527fc640 1 monclient: found mon.noname-a-legacy 2024-09-16T04:05:12.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.142+0000 7f34527fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:05:12.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.142+0000 7f34527fc640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.42:6789/0 2024-09-16T04:05:12.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.142+0000 7f34527fc640 1 -- 172.21.15.42:0/554089026 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f345409f8f0 con 0x7f34540a1330 2024-09-16T04:05:12.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.142+0000 7f34527fc640 10 monclient: _finish_auth 0 2024-09-16T04:05:12.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.142+0000 7f34527fc640 10 monclient: _check_auth_tickets 2024-09-16T04:05:12.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.142+0000 7f34527fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:05:12.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.142+0000 7f34527fc640 1 -- 172.21.15.42:0/554089026 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2177413734 0 0) 0x7f3448001680 con 0x7f34540a1330 2024-09-16T04:05:12.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.142+0000 7f3459ad5640 5 monclient: authenticate success, global_id 984240 2024-09-16T04:05:12.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.142+0000 7f3459ad5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:05:12.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.142+0000 7f34527fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:05:12.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.142+0000 7f3453fff640 1 --2- 172.21.15.42:0/554089026 >> v2:172.21.15.154:3300/0 conn(0x7f34540a1e20 0x7f345413c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:05:12.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.142+0000 7f34527fc640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T04:05:12.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.142+0000 7f34527fc640 10 monclient: dump: 2024-09-16T04:05:12.162 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:05:12.162 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:05:12.162 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:12.162 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:12.163 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:05:12.163 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:05:12.163 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:05:12.163 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:05:12.163 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:05:12.163 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:05:12.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.142+0000 7f34527fc640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] but i'm connected to v1:172.21.15.42:6789/0, reconnecting 2024-09-16T04:05:12.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.142+0000 7f34527fc640 10 monclient: _reopen_session rank -1 2024-09-16T04:05:12.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.142+0000 7f34527fc640 1 -- 172.21.15.42:0/554089026 >> v1:172.21.15.42:6789/0 conn(0x7f34540a1330 legacy=0x7f34540a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:05:12.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.142+0000 7f34527fc640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:05:12.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.142+0000 7f34527fc640 1 --2- 172.21.15.42:0/554089026 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f343c0039c0 0x7f343c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:05:12.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.142+0000 7f34527fc640 10 monclient(hunting): picked mon.b con 0x7f343c0039c0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:05:12.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.142+0000 7f34527fc640 1 --2- 172.21.15.42:0/554089026 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f343c004810 0x7f343c004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:05:12.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.142+0000 7f34527fc640 10 monclient(hunting): picked mon.a con 0x7f343c004810 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:05:12.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.142+0000 7f34527fc640 1 --2- 172.21.15.42:0/554089026 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f343c0051d0 0x7f343c00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:05:12.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.142+0000 7f34527fc640 10 monclient(hunting): picked mon.c con 0x7f343c0051d0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:05:12.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.142+0000 7f34527fc640 10 monclient(hunting): start opening mon connection 2024-09-16T04:05:12.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.142+0000 7f34527fc640 1 -- 172.21.15.42:0/554089026 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f345409c090 con 0x7f343c004810 2024-09-16T04:05:12.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.142+0000 7f34527fc640 10 monclient(hunting): start opening mon connection 2024-09-16T04:05:12.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.142+0000 7f34527fc640 1 -- 172.21.15.42:0/554089026 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f345409f610 con 0x7f343c0039c0 2024-09-16T04:05:12.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.142+0000 7f34527fc640 10 monclient(hunting): start opening mon connection 2024-09-16T04:05:12.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.142+0000 7f34527fc640 1 -- 172.21.15.42:0/554089026 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f343c00bf10 con 0x7f343c0051d0 2024-09-16T04:05:12.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.142+0000 7f34527fc640 10 monclient(hunting): _renew_subs 2024-09-16T04:05:12.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.142+0000 7f34527fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:05:12.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.146+0000 7f3459ad5640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-16T04:05:12.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.146+0000 7f3452ffd640 1 --2- 172.21.15.42:0/554089026 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f343c004810 0x7f343c004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:05:12.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.146+0000 7f3453fff640 1 --2- 172.21.15.42:0/554089026 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f343c0051d0 0x7f343c00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:05:12.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.146+0000 7f3452ffd640 10 monclient(hunting): get_auth_request con 0x7f343c004810 auth_method 0 2024-09-16T04:05:12.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.146+0000 7f3452ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:05:12.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.146+0000 7f3452ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:05:12.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.146+0000 7f3452ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:05:12.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.146+0000 7f3453fff640 10 monclient(hunting): get_auth_request con 0x7f343c0051d0 auth_method 0 2024-09-16T04:05:12.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.146+0000 7f3453fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:05:12.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.146+0000 7f3453fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:05:12.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.146+0000 7f3453fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:05:12.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.146+0000 7f3452ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:05:12.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.146+0000 7f3452ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:05:12.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.146+0000 7f3452ffd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:05:12.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.146+0000 7f3453fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:05:12.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.146+0000 7f3453fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:05:12.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.146+0000 7f3453fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:05:12.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.146+0000 7f34537fe640 1 --2- 172.21.15.42:0/554089026 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f343c0039c0 0x7f343c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:05:12.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.146+0000 7f34537fe640 10 monclient(hunting): get_auth_request con 0x7f343c0039c0 auth_method 0 2024-09-16T04:05:12.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.146+0000 7f34537fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:05:12.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.146+0000 7f34537fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:05:12.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.146+0000 7f34537fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:05:12.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.146+0000 7f3453fff640 10 monclient(hunting): handle_auth_done global_id 984240 payload 293 2024-09-16T04:05:12.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.146+0000 7f3453fff640 1 -- 172.21.15.42:0/554089026 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f343c0039c0 msgr2=0x7f343c003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:05:12.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.146+0000 7f3453fff640 1 --2- 172.21.15.42:0/554089026 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f343c0039c0 0x7f343c003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:12.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.146+0000 7f3453fff640 1 -- 172.21.15.42:0/554089026 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f343c004810 msgr2=0x7f343c004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:05:12.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.146+0000 7f3453fff640 1 --2- 172.21.15.42:0/554089026 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f343c004810 0x7f343c004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:12.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.146+0000 7f3453fff640 10 monclient: _finish_hunting 0 2024-09-16T04:05:12.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.146+0000 7f3453fff640 1 monclient: found mon.c 2024-09-16T04:05:12.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.146+0000 7f3453fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:05:12.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.146+0000 7f3453fff640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:05:12.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.146+0000 7f3453fff640 1 -- 172.21.15.42:0/554089026 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f343c003530 con 0x7f343c0051d0 2024-09-16T04:05:12.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.146+0000 7f3453fff640 10 monclient: _finish_auth 0 2024-09-16T04:05:12.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.146+0000 7f3453fff640 10 monclient: _check_auth_tickets 2024-09-16T04:05:12.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.146+0000 7f3453fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:05:12.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.146+0000 7f3452ffd640 1 --2- 172.21.15.42:0/554089026 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f343c004810 0x7f343c004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:05:12.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.146+0000 7f3453fff640 1 --2- 172.21.15.42:0/554089026 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f343c0051d0 0x7f343c00b9d0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f344c00e350 tx=0x7f344c01ce60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=59d96b5f3acc4ade server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:05:12.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.146+0000 7f34527fc640 1 -- 172.21.15.42:0/554089026 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f344c0b2070 con 0x7f343c0051d0 2024-09-16T04:05:12.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.146+0000 7f34527fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:05:12.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.146+0000 7f34527fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:05:12.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.146+0000 7f34527fc640 10 monclient: dump: 2024-09-16T04:05:12.167 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:05:12.167 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:05:12.168 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:12.168 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:12.168 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:05:12.168 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:05:12.168 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:05:12.168 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:05:12.168 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:05:12.168 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:05:12.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.146+0000 7f3459ad5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:05:12.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.146+0000 7f34527fc640 1 -- 172.21.15.42:0/554089026 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f344c0a5d60 con 0x7f343c0051d0 2024-09-16T04:05:12.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.146+0000 7f34527fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:05:12.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.146+0000 7f34527fc640 1 -- 172.21.15.42:0/554089026 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f344c009de0 con 0x7f343c0051d0 2024-09-16T04:05:12.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.146+0000 7f3459ad5640 10 monclient: get_monmap_and_config success 2024-09-16T04:05:12.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.146+0000 7f3459ad5640 4 set_mon_vals no callback set 2024-09-16T04:05:12.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.146+0000 7f34527fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:05:12.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.146+0000 7f34527fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:05:12.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.146+0000 7f34527fc640 10 monclient: dump: 2024-09-16T04:05:12.169 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:05:12.169 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:05:12.169 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:12.169 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:12.169 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:05:12.169 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:05:12.169 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:05:12.169 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:05:12.169 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:05:12.169 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:05:12.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.146+0000 7f3459ad5640 10 monclient: shutdown 2024-09-16T04:05:12.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.146+0000 7f3459ad5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:05:12.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.146+0000 7f3459ad5640 1 -- 172.21.15.42:0/554089026 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f343c0051d0 msgr2=0x7f343c00b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:05:12.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.146+0000 7f3459ad5640 1 --2- 172.21.15.42:0/554089026 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f343c0051d0 0x7f343c00b9d0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f344c00e350 tx=0x7f344c01ce60 comp rx=0 tx=0).stop 2024-09-16T04:05:12.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.146+0000 7f3453fff640 1 -- 172.21.15.42:0/554089026 reap_dead start 2024-09-16T04:05:12.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.146+0000 7f3459ad5640 1 -- 172.21.15.42:0/554089026 shutdown_connections 2024-09-16T04:05:12.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.146+0000 7f3459ad5640 1 -- 172.21.15.42:0/554089026 >> 172.21.15.42:0/554089026 conn(0x7f34540a2420 msgr2=0x7f34540a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:05:12.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.146+0000 7f3459ad5640 1 -- 172.21.15.42:0/554089026 shutdown_connections 2024-09-16T04:05:12.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.146+0000 7f3459ad5640 1 -- 172.21.15.42:0/554089026 wait complete. 2024-09-16T04:05:12.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.150+0000 7f3459ad5640 10 monclient: build_initial_monmap 2024-09-16T04:05:12.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.150+0000 7f3459ad5640 10 monclient: monmap: 2024-09-16T04:05:12.170 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:05:12.170 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:05:12.170 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:05:12.171 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:05:12.171 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:05:12.171 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:05:12.171 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:05:12.171 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:05:12.171 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:05:12.171 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:05:12.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.150+0000 7f3459ad5640 1 librados: starting msgr at 2024-09-16T04:05:12.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.150+0000 7f3459ad5640 1 librados: starting objecter 2024-09-16T04:05:12.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.150+0000 7f3459ad5640 1 Processor -- start 2024-09-16T04:05:12.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.150+0000 7f3459ad5640 1 -- start start 2024-09-16T04:05:12.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.150+0000 7f3459ad5640 1 librados: setting wanted keys 2024-09-16T04:05:12.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.150+0000 7f3459ad5640 1 librados: calling monclient init 2024-09-16T04:05:12.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.150+0000 7f3459ad5640 10 monclient: init 2024-09-16T04:05:12.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.150+0000 7f3459ad5640 10 monclient: _reopen_session rank -1 2024-09-16T04:05:12.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.150+0000 7f3459ad5640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:05:12.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.150+0000 7f3459ad5640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3454134b80 0x7f345409cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:05:12.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.150+0000 7f3459ad5640 10 monclient(hunting): picked mon.noname-b con 0x7f3454134b80 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:05:12.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.150+0000 7f3459ad5640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f345409d0d0 0x7f345409d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:05:12.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.150+0000 7f3459ad5640 10 monclient(hunting): picked mon.noname-a con 0x7f345409d0d0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:05:12.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.150+0000 7f3459ad5640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f34541d06d0 0x7f34541d2ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:05:12.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.150+0000 7f3459ad5640 10 monclient(hunting): picked mon.noname-c con 0x7f34541d06d0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:05:12.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.150+0000 7f3459ad5640 10 monclient(hunting): start opening mon connection 2024-09-16T04:05:12.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.150+0000 7f3459ad5640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f34540a32f0 con 0x7f345409d0d0 2024-09-16T04:05:12.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.150+0000 7f3459ad5640 10 monclient(hunting): start opening mon connection 2024-09-16T04:05:12.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.150+0000 7f3459ad5640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f34540a3170 con 0x7f3454134b80 2024-09-16T04:05:12.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.150+0000 7f3459ad5640 10 monclient(hunting): start opening mon connection 2024-09-16T04:05:12.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.150+0000 7f3459ad5640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f345409ed70 con 0x7f34541d06d0 2024-09-16T04:05:12.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.150+0000 7f3459ad5640 10 monclient(hunting): _renew_subs 2024-09-16T04:05:12.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.150+0000 7f3459ad5640 10 monclient(hunting): authenticate will time out at 2359.121105s 2024-09-16T04:05:12.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.150+0000 7f3453fff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f34541d06d0 0x7f34541d2ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:05:12.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.150+0000 7f3452ffd640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f345409d0d0 0x7f345409d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:05:12.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.150+0000 7f3453fff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f34541d06d0 0x7f34541d2ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:40980/0 (socket says 172.21.15.42:40980) 2024-09-16T04:05:12.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.150+0000 7f3452ffd640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f345409d0d0 0x7f345409d4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:33366/0 (socket says 172.21.15.42:33366) 2024-09-16T04:05:12.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.150+0000 7f3453fff640 1 -- 172.21.15.42:0/3230355505 learned_addr learned my addr 172.21.15.42:0/3230355505 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:05:12.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.150+0000 7f3453fff640 10 monclient(hunting): get_auth_request con 0x7f34541d06d0 auth_method 0 2024-09-16T04:05:12.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.150+0000 7f3453fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:05:12.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.150+0000 7f3453fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:05:12.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.150+0000 7f3453fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:05:12.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.150+0000 7f3452ffd640 10 monclient(hunting): get_auth_request con 0x7f345409d0d0 auth_method 0 2024-09-16T04:05:12.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.150+0000 7f3452ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:05:12.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.150+0000 7f3452ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:05:12.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.150+0000 7f3452ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:05:12.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.150+0000 7f3452ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:05:12.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.150+0000 7f3452ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:05:12.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.150+0000 7f3452ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:05:12.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.150+0000 7f3453fff640 1 -- 172.21.15.42:0/3230355505 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f34541d06d0 msgr2=0x7f34541d2ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-16T04:05:12.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.150+0000 7f3453fff640 1 -- 172.21.15.42:0/3230355505 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f34541d06d0 msgr2=0x7f34541d2ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:05:12.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.150+0000 7f3453fff640 1 --2- 172.21.15.42:0/3230355505 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f34541d06d0 0x7f34541d2ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:05:12.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.150+0000 7f3453fff640 1 --2- 172.21.15.42:0/3230355505 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f34541d06d0 0x7f34541d2ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:05:12.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.150+0000 7f34537fe640 1 --2- 172.21.15.42:0/3230355505 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3454134b80 0x7f345409cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:05:12.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.150+0000 7f34537fe640 10 monclient(hunting): get_auth_request con 0x7f3454134b80 auth_method 0 2024-09-16T04:05:12.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.150+0000 7f34537fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:05:12.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.150+0000 7f34537fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:05:12.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.150+0000 7f34537fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:05:12.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.150+0000 7f3452ffd640 10 monclient(hunting): handle_auth_done global_id 984243 payload 824 2024-09-16T04:05:12.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.150+0000 7f3452ffd640 1 -- 172.21.15.42:0/3230355505 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f34541d06d0 msgr2=0x7f34541d2ac0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:05:12.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.150+0000 7f3452ffd640 1 --2- 172.21.15.42:0/3230355505 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f34541d06d0 0x7f34541d2ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:12.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.150+0000 7f3452ffd640 1 -- 172.21.15.42:0/3230355505 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3454134b80 msgr2=0x7f345409cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:05:12.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.150+0000 7f3452ffd640 1 --2- 172.21.15.42:0/3230355505 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3454134b80 0x7f345409cb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:12.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.150+0000 7f3452ffd640 10 monclient: _finish_hunting 0 2024-09-16T04:05:12.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.150+0000 7f3452ffd640 1 monclient: found mon.noname-a 2024-09-16T04:05:12.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.150+0000 7f3452ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:05:12.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.150+0000 7f3452ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:05:12.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.150+0000 7f3452ffd640 1 -- 172.21.15.42:0/3230355505 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3454123380 con 0x7f345409d0d0 2024-09-16T04:05:12.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.150+0000 7f3452ffd640 1 --2- 172.21.15.42:0/3230355505 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f345409d0d0 0x7f345409d4f0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f3444011560 tx=0x7f3444011a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:05:12.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.150+0000 7f343bfff640 10 client.?.objecter ms_handle_connect 0x7f345409d0d0 2024-09-16T04:05:12.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.150+0000 7f343bfff640 10 client.?.objecter resend_mon_ops 2024-09-16T04:05:12.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.150+0000 7f343bfff640 1 -- 172.21.15.42:0/3230355505 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3444019070 con 0x7f345409d0d0 2024-09-16T04:05:12.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.150+0000 7f343bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:05:12.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.150+0000 7f343bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:05:12.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.150+0000 7f343bfff640 10 monclient: dump: 2024-09-16T04:05:12.176 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:05:12.176 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:05:12.176 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:12.176 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:12.176 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:05:12.176 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:05:12.176 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:05:12.176 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:05:12.176 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:05:12.176 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:05:12.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.150+0000 7f343bfff640 10 monclient: _finish_auth 0 2024-09-16T04:05:12.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.150+0000 7f343bfff640 10 monclient: _check_auth_tickets 2024-09-16T04:05:12.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.150+0000 7f343bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:05:12.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.150+0000 7f343bfff640 1 -- 172.21.15.42:0/3230355505 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3444015890 con 0x7f345409d0d0 2024-09-16T04:05:12.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.150+0000 7f343bfff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:05:12.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.150+0000 7f343bfff640 1 -- 172.21.15.42:0/3230355505 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3444015c40 con 0x7f345409d0d0 2024-09-16T04:05:12.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.150+0000 7f343bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:05:12.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.150+0000 7f343bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:05:12.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.150+0000 7f343bfff640 10 monclient: dump: 2024-09-16T04:05:12.177 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:05:12.177 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:05:12.177 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:12.177 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:12.177 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:05:12.177 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:05:12.177 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:05:12.178 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:05:12.178 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:05:12.178 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:05:12.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.150+0000 7f3459ad5640 5 monclient: authenticate success, global_id 984243 2024-09-16T04:05:12.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.150+0000 7f3459ad5640 10 monclient: _renew_subs 2024-09-16T04:05:12.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.150+0000 7f3451ffb640 4 set_mon_vals no callback set 2024-09-16T04:05:12.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.150+0000 7f3459ad5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:05:12.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.150+0000 7f3459ad5640 1 -- 172.21.15.42:0/3230355505 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f345406dcd0 con 0x7f345409d0d0 2024-09-16T04:05:12.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.150+0000 7f3459ad5640 10 client.984243.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:05:12.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.150+0000 7f3459ad5640 10 monclient: _renew_subs 2024-09-16T04:05:12.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.150+0000 7f3459ad5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:05:12.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.150+0000 7f3459ad5640 1 -- 172.21.15.42:0/3230355505 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f345406e220 con 0x7f345409d0d0 2024-09-16T04:05:12.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.150+0000 7f3459ad5640 1 librados: init done 2024-09-16T04:05:12.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.150+0000 7f3459ad5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:05:12.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.150+0000 7f3459ad5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:05:12.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.150+0000 7f3459ad5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:05:12.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.150+0000 7f3459ad5640 1 -- 172.21.15.42:0/3230355505 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f345406dac0 con 0x7f345409d0d0 2024-09-16T04:05:12.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.150+0000 7f343bfff640 1 -- 172.21.15.42:0/3230355505 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f344401c070 con 0x7f345409d0d0 2024-09-16T04:05:12.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.150+0000 7f343bfff640 1 --2- 172.21.15.42:0/3230355505 >> v2:172.21.15.154:6812/254845342 conn(0x7f3440037810 0x7f3440039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:05:12.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.150+0000 7f343bfff640 1 -- 172.21.15.42:0/3230355505 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(700..700 src has 451..700) ==== 5899+0+0 (secure 0 0 0) 0x7f3444049230 con 0x7f345409d0d0 2024-09-16T04:05:12.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.150+0000 7f343bfff640 10 client.984243.objecter ms_dispatch 0x7f3454000ba0 osd_map(700..700 src has 451..700) 2024-09-16T04:05:12.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.150+0000 7f343bfff640 3 client.984243.objecter handle_osd_map got epochs [700,700] > 0 2024-09-16T04:05:12.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.150+0000 7f343bfff640 3 client.984243.objecter handle_osd_map decoding full epoch 700 2024-09-16T04:05:12.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.150+0000 7f343bfff640 20 client.984243.objecter dump_active .. 0 homeless 2024-09-16T04:05:12.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.150+0000 7f34537fe640 1 --2- 172.21.15.42:0/3230355505 >> v2:172.21.15.154:6812/254845342 conn(0x7f3440037810 0x7f3440039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:05:12.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.154+0000 7f34537fe640 10 monclient: get_auth_request con 0x7f3440037810 auth_method 0 2024-09-16T04:05:12.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.154+0000 7f343bfff640 10 client.984243.objecter ms_handle_connect 0x7f3440037810 2024-09-16T04:05:12.180 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.154+0000 7f34537fe640 1 --2- 172.21.15.42:0/3230355505 >> v2:172.21.15.154:6812/254845342 conn(0x7f3440037810 0x7f3440039cd0 secure :-1 s=READY pgs=3839 cs=0 l=1 rev1=1 crypto rx=0x7f3448000c10 tx=0x7f3448009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:05:12.180 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.158+0000 7f343bfff640 1 -- 172.21.15.42:0/3230355505 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f3444026ba0 con 0x7f345409d0d0 2024-09-16T04:05:12.180 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.158+0000 7f343bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:05:12.180 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.158+0000 7f343bfff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:05:12.389 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.382+0000 7f3459ad5640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:05:12.389 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.382+0000 7f3459ad5640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:05:12.389 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.382+0000 7f3459ad5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:05:12.389 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.382+0000 7f3459ad5640 1 -- 172.21.15.42:0/3230355505 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f345406ea40 con 0x7f345409d0d0 2024-09-16T04:05:12.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.386+0000 7f343bfff640 1 -- 172.21.15.42:0/3230355505 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7f344400f980 con 0x7f345409d0d0 2024-09-16T04:05:12.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.386+0000 7f343bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:05:12.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.386+0000 7f343bfff640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:05:12.390 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":216,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:05:12.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.386+0000 7f3459ad5640 10 librados: watch_flush enter 2024-09-16T04:05:12.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.386+0000 7f3459ad5640 10 librados: watch_flush exit 2024-09-16T04:05:12.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.386+0000 7f3459ad5640 20 client.984243.objecter shutdown clearing up homeless session... 2024-09-16T04:05:12.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.386+0000 7f3459ad5640 10 client.984243.objecter successfully canceled tick 2024-09-16T04:05:12.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.386+0000 7f3459ad5640 1 -- 172.21.15.42:0/3230355505 >> v2:172.21.15.154:6812/254845342 conn(0x7f3440037810 msgr2=0x7f3440039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:05:12.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.386+0000 7f3459ad5640 1 --2- 172.21.15.42:0/3230355505 >> v2:172.21.15.154:6812/254845342 conn(0x7f3440037810 0x7f3440039cd0 secure :-1 s=READY pgs=3839 cs=0 l=1 rev1=1 crypto rx=0x7f3448000c10 tx=0x7f3448009040 comp rx=0 tx=0).stop 2024-09-16T04:05:12.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.386+0000 7f3459ad5640 10 monclient: shutdown 2024-09-16T04:05:12.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.386+0000 7f3459ad5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:05:12.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.386+0000 7f3459ad5640 1 -- 172.21.15.42:0/3230355505 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f345409d0d0 msgr2=0x7f345409d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:05:12.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.386+0000 7f3459ad5640 1 --2- 172.21.15.42:0/3230355505 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f345409d0d0 0x7f345409d4f0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f3444011560 tx=0x7f3444011a20 comp rx=0 tx=0).stop 2024-09-16T04:05:12.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.386+0000 7f3459ad5640 1 -- 172.21.15.42:0/3230355505 shutdown_connections 2024-09-16T04:05:12.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.386+0000 7f3459ad5640 1 --2- 172.21.15.42:0/3230355505 >> v2:172.21.15.154:6812/254845342 conn(0x7f3440037810 0x7f3440039cd0 unknown :-1 s=CLOSED pgs=3839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:12.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.386+0000 7f3459ad5640 1 --2- 172.21.15.42:0/3230355505 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f34541d06d0 0x7f34541d2ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:12.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.386+0000 7f3459ad5640 1 --2- 172.21.15.42:0/3230355505 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f345409d0d0 0x7f345409d4f0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:12.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.386+0000 7f3459ad5640 1 --2- 172.21.15.42:0/3230355505 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3454134b80 0x7f345409cb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:12.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.386+0000 7f3459ad5640 1 -- 172.21.15.42:0/3230355505 >> 172.21.15.42:0/3230355505 conn(0x7f34540a2420 msgr2=0x7f34540a45b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:05:12.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.386+0000 7f3459ad5640 1 -- 172.21.15.42:0/3230355505 shutdown_connections 2024-09-16T04:05:12.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.386+0000 7f3459ad5640 1 -- 172.21.15.42:0/3230355505 wait complete. 2024-09-16T04:05:12.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.386+0000 7f3459ad5640 1 librados: shutdown 2024-09-16T04:05:12.409 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T04:05:12.409 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T04:05:12.409 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T04:05:12.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.530+0000 7f890410d640 10 monclient: get_monmap_and_config 2024-09-16T04:05:12.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.530+0000 7f890410d640 10 monclient: build_initial_monmap 2024-09-16T04:05:12.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.530+0000 7f890410d640 10 monclient: monmap: 2024-09-16T04:05:12.535 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:05:12.535 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:05:12.535 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:05:12.533538+0000 2024-09-16T04:05:12.535 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:05:12.533538+0000 2024-09-16T04:05:12.535 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:05:12.535 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:05:12.536 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:05:12.536 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:05:12.536 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:05:12.536 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:05:12.536 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:05:12.536 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:05:12.538 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.530+0000 7f890410d640 1 Processor -- start 2024-09-16T04:05:12.538 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.530+0000 7f890410d640 1 -- start start 2024-09-16T04:05:12.538 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.530+0000 7f890410d640 10 monclient: init 2024-09-16T04:05:12.538 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.530+0000 7f890410d640 10 monclient: _reopen_session rank -1 2024-09-16T04:05:12.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.530+0000 7f890410d640 10 monclient: _add_conns ranks=[1,4,2,3,0] 2024-09-16T04:05:12.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.530+0000 7f890410d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f88fc0a39b0 addr v1:172.21.15.154:6789/0 2024-09-16T04:05:12.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.530+0000 7f890410d640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f88fc0a3d90 0x7f88fc13c380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:05:12.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.530+0000 7f890410d640 10 monclient(hunting): picked mon.noname-c con 0x7f88fc0a3d90 addr v2:172.21.15.154:3300/0 2024-09-16T04:05:12.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.530+0000 7f890410d640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f88fc13c8c0 0x7f88fc13ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:05:12.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.530+0000 7f890410d640 10 monclient(hunting): picked mon.noname-b con 0x7f88fc13c8c0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:05:12.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.530+0000 7f890410d640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f88fc134290 con 0x7f88fc0a39b0 2024-09-16T04:05:12.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.530+0000 7f890410d640 10 monclient(hunting): start opening mon connection 2024-09-16T04:05:12.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.530+0000 7f890410d640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f88fc09c140 con 0x7f88fc0a3d90 2024-09-16T04:05:12.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.530+0000 7f890410d640 10 monclient(hunting): start opening mon connection 2024-09-16T04:05:12.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.530+0000 7f890410d640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f88fc09c420 con 0x7f88fc13c8c0 2024-09-16T04:05:12.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.530+0000 7f890410d640 10 monclient(hunting): _renew_subs 2024-09-16T04:05:12.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.530+0000 7f890410d640 10 monclient(hunting): authenticate will time out at 2359.503814s 2024-09-16T04:05:12.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.534+0000 7f89026aa640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f88fc13c8c0 0x7f88fc13ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:05:12.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.534+0000 7f89026aa640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f88fc13c8c0 0x7f88fc13ecb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:40982/0 (socket says 172.21.15.42:40982) 2024-09-16T04:05:12.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.534+0000 7f89026aa640 1 -- 172.21.15.42:0/1494643983 learned_addr learned my addr 172.21.15.42:0/1494643983 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:05:12.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.534+0000 7f89026aa640 10 monclient(hunting): get_auth_request con 0x7f88fc13c8c0 auth_method 0 2024-09-16T04:05:12.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.534+0000 7f89026aa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:05:12.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.534+0000 7f89026aa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:05:12.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.534+0000 7f89026aa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:05:12.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.534+0000 7f89016a8640 1 --2- 172.21.15.42:0/1494643983 >> v2:172.21.15.154:3300/0 conn(0x7f88fc0a3d90 0x7f88fc13c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:05:12.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.534+0000 7f89026aa640 1 -- 172.21.15.42:0/1494643983 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f88fc13c8c0 msgr2=0x7f88fc13ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T04:05:12.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.534+0000 7f89026aa640 1 -- 172.21.15.42:0/1494643983 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f88fc13c8c0 msgr2=0x7f88fc13ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:05:12.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.534+0000 7f89016a8640 10 monclient(hunting): get_auth_request con 0x7f88fc0a3d90 auth_method 0 2024-09-16T04:05:12.541 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.534+0000 7f89016a8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:05:12.541 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.534+0000 7f89016a8640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:05:12.541 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.534+0000 7f89016a8640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:05:12.541 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.534+0000 7f89026aa640 1 --2- 172.21.15.42:0/1494643983 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f88fc13c8c0 0x7f88fc13ecb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:05:12.541 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.534+0000 7f89026aa640 1 --2- 172.21.15.42:0/1494643983 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f88fc13c8c0 0x7f88fc13ecb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:05:12.541 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.534+0000 7f89016a8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:05:12.541 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.534+0000 7f89016a8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:05:12.541 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.534+0000 7f89016a8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:05:12.541 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.534+0000 7f89016a8640 10 monclient(hunting): handle_auth_done global_id 904858 payload 274 2024-09-16T04:05:12.541 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.534+0000 7f89016a8640 1 -- 172.21.15.42:0/1494643983 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f88fc13c8c0 msgr2=0x7f88fc13ecb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:05:12.541 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.534+0000 7f89016a8640 1 --2- 172.21.15.42:0/1494643983 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f88fc13c8c0 0x7f88fc13ecb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:12.541 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.534+0000 7f89016a8640 1 -- 172.21.15.42:0/1494643983 >> v1:172.21.15.154:6789/0 conn(0x7f88fc0a39b0 legacy=0x7f88fc133b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:05:12.541 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.534+0000 7f89016a8640 10 monclient: _finish_hunting 0 2024-09-16T04:05:12.542 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.534+0000 7f89016a8640 1 monclient: found mon.noname-c 2024-09-16T04:05:12.542 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.534+0000 7f89016a8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:05:12.542 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.534+0000 7f89016a8640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.154:3300/0 2024-09-16T04:05:12.542 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.534+0000 7f89016a8640 1 -- 172.21.15.42:0/1494643983 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88fc06dd60 con 0x7f88fc0a3d90 2024-09-16T04:05:12.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.534+0000 7f89016a8640 1 --2- 172.21.15.42:0/1494643983 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f88fc0a3d90 0x7f88fc13c380 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f88ec0097c0 tx=0x7f88ec01c360 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1c605076fd048628 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:05:12.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.534+0000 7f8900ea7640 1 -- 172.21.15.42:0/1494643983 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88ec0b0070 con 0x7f88fc0a3d90 2024-09-16T04:05:12.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.534+0000 7f8900ea7640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:05:12.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.534+0000 7f8900ea7640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:05:12.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.534+0000 7f8900ea7640 10 monclient: dump: 2024-09-16T04:05:12.549 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:05:12.549 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:05:12.549 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:12.549 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:12.549 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:05:12.549 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:05:12.549 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:05:12.549 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:05:12.550 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:05:12.550 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:05:12.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.534+0000 7f8900ea7640 10 monclient: _finish_auth 0 2024-09-16T04:05:12.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.534+0000 7f8900ea7640 10 monclient: _check_auth_tickets 2024-09-16T04:05:12.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.534+0000 7f8900ea7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:05:12.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.534+0000 7f890410d640 5 monclient: authenticate success, global_id 904858 2024-09-16T04:05:12.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.534+0000 7f890410d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:05:12.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.534+0000 7f8900ea7640 1 -- 172.21.15.42:0/1494643983 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f88ec0ab600 con 0x7f88fc0a3d90 2024-09-16T04:05:12.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.534+0000 7f8900ea7640 10 monclient: handle_config config(0 keys) 2024-09-16T04:05:12.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.534+0000 7f890410d640 10 monclient: get_monmap_and_config success 2024-09-16T04:05:12.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.534+0000 7f890410d640 4 set_mon_vals no callback set 2024-09-16T04:05:12.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.534+0000 7f8900ea7640 1 -- 172.21.15.42:0/1494643983 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88ec0ab9d0 con 0x7f88fc0a3d90 2024-09-16T04:05:12.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.534+0000 7f8900ea7640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:05:12.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.534+0000 7f8900ea7640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:05:12.551 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.534+0000 7f8900ea7640 10 monclient: dump: 2024-09-16T04:05:12.551 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:05:12.551 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:05:12.551 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:12.551 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:12.551 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:05:12.551 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:05:12.551 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:05:12.551 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:05:12.551 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:05:12.551 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:05:12.551 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.538+0000 7f890410d640 10 monclient: shutdown 2024-09-16T04:05:12.551 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.538+0000 7f890410d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:05:12.551 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.538+0000 7f890410d640 1 -- 172.21.15.42:0/1494643983 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f88fc0a3d90 msgr2=0x7f88fc13c380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:05:12.552 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.538+0000 7f890410d640 1 --2- 172.21.15.42:0/1494643983 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f88fc0a3d90 0x7f88fc13c380 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f88ec0097c0 tx=0x7f88ec01c360 comp rx=0 tx=0).stop 2024-09-16T04:05:12.552 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.538+0000 7f890410d640 1 -- 172.21.15.42:0/1494643983 shutdown_connections 2024-09-16T04:05:12.552 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.538+0000 7f890410d640 1 --2- 172.21.15.42:0/1494643983 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f88fc13c8c0 0x7f88fc13ecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:12.552 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.538+0000 7f890410d640 1 --2- 172.21.15.42:0/1494643983 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f88fc0a3d90 0x7f88fc13c380 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:12.552 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.538+0000 7f890410d640 1 -- 172.21.15.42:0/1494643983 >> 172.21.15.42:0/1494643983 conn(0x7f88fc0a9fb0 msgr2=0x7f88fc0aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:05:12.552 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.538+0000 7f890410d640 1 -- 172.21.15.42:0/1494643983 shutdown_connections 2024-09-16T04:05:12.552 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.538+0000 7f890410d640 1 -- 172.21.15.42:0/1494643983 wait complete. 2024-09-16T04:05:12.552 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.538+0000 7f890410d640 10 monclient: build_initial_monmap 2024-09-16T04:05:12.552 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.538+0000 7f890410d640 10 monclient: monmap: 2024-09-16T04:05:12.552 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:05:12.552 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:05:12.552 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:05:12.552 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:05:12.553 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:05:12.553 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:05:12.553 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:05:12.553 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:05:12.553 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:05:12.553 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:05:12.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.538+0000 7f890410d640 1 librados: starting msgr at 2024-09-16T04:05:12.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.538+0000 7f890410d640 1 librados: starting objecter 2024-09-16T04:05:12.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.538+0000 7f890410d640 1 Processor -- start 2024-09-16T04:05:12.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.538+0000 7f890410d640 1 -- start start 2024-09-16T04:05:12.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.538+0000 7f890410d640 1 librados: setting wanted keys 2024-09-16T04:05:12.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.538+0000 7f890410d640 1 librados: calling monclient init 2024-09-16T04:05:12.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.538+0000 7f890410d640 10 monclient: init 2024-09-16T04:05:12.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.538+0000 7f890410d640 10 monclient: _reopen_session rank -1 2024-09-16T04:05:12.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.538+0000 7f890410d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:05:12.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.538+0000 7f890410d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f88fc0a39b0 0x7f88fc13a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:05:12.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.538+0000 7f890410d640 10 monclient(hunting): picked mon.noname-b con 0x7f88fc0a39b0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:05:12.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.538+0000 7f890410d640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f88fc0a3d90 0x7f88fc13a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:05:12.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.538+0000 7f890410d640 10 monclient(hunting): picked mon.noname-a con 0x7f88fc0a3d90 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:05:12.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.538+0000 7f890410d640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f88fc13c8c0 0x7f88fc13ae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:05:12.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.538+0000 7f890410d640 10 monclient(hunting): picked mon.noname-c con 0x7f88fc13c8c0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:05:12.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.538+0000 7f890410d640 10 monclient(hunting): start opening mon connection 2024-09-16T04:05:12.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.538+0000 7f890410d640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f88fc141da0 con 0x7f88fc0a3d90 2024-09-16T04:05:12.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.538+0000 7f890410d640 10 monclient(hunting): start opening mon connection 2024-09-16T04:05:12.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.538+0000 7f890410d640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f88fc141c20 con 0x7f88fc0a39b0 2024-09-16T04:05:12.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.538+0000 7f890410d640 10 monclient(hunting): start opening mon connection 2024-09-16T04:05:12.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.538+0000 7f890410d640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f88fc141f20 con 0x7f88fc13c8c0 2024-09-16T04:05:12.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.538+0000 7f890410d640 10 monclient(hunting): _renew_subs 2024-09-16T04:05:12.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.538+0000 7f890410d640 10 monclient(hunting): authenticate will time out at 2359.509786s 2024-09-16T04:05:12.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.538+0000 7f89026aa640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f88fc13c8c0 0x7f88fc13ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:05:12.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.538+0000 7f89016a8640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f88fc0a3d90 0x7f88fc13a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:05:12.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.538+0000 7f89026aa640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f88fc13c8c0 0x7f88fc13ae60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:40996/0 (socket says 172.21.15.42:40996) 2024-09-16T04:05:12.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.538+0000 7f89026aa640 1 -- 172.21.15.42:0/3930694209 learned_addr learned my addr 172.21.15.42:0/3930694209 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:05:12.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.538+0000 7f89026aa640 10 monclient(hunting): get_auth_request con 0x7f88fc13c8c0 auth_method 0 2024-09-16T04:05:12.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.538+0000 7f89026aa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:05:12.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.538+0000 7f89026aa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:05:12.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.538+0000 7f89026aa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:05:12.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.538+0000 7f89016a8640 10 monclient(hunting): get_auth_request con 0x7f88fc0a3d90 auth_method 0 2024-09-16T04:05:12.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.538+0000 7f89016a8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:05:12.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.538+0000 7f89016a8640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:05:12.556 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.538+0000 7f89016a8640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:05:12.556 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.538+0000 7f89016a8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:05:12.556 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.538+0000 7f89016a8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:05:12.556 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.538+0000 7f89016a8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:05:12.556 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.538+0000 7f8901ea9640 1 --2- 172.21.15.42:0/3930694209 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f88fc0a39b0 0x7f88fc13a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:05:12.556 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.538+0000 7f89026aa640 1 -- 172.21.15.42:0/3930694209 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f88fc13c8c0 msgr2=0x7f88fc13ae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-16T04:05:12.556 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.538+0000 7f89026aa640 1 -- 172.21.15.42:0/3930694209 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f88fc13c8c0 msgr2=0x7f88fc13ae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:05:12.556 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.538+0000 7f89026aa640 1 --2- 172.21.15.42:0/3930694209 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f88fc13c8c0 0x7f88fc13ae60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:05:12.556 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.538+0000 7f8901ea9640 10 monclient(hunting): get_auth_request con 0x7f88fc0a39b0 auth_method 0 2024-09-16T04:05:12.556 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.538+0000 7f8901ea9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:05:12.556 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.538+0000 7f8901ea9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:05:12.556 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.538+0000 7f8901ea9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:05:12.557 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.538+0000 7f89026aa640 1 --2- 172.21.15.42:0/3930694209 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f88fc13c8c0 0x7f88fc13ae60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:05:12.557 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.538+0000 7f8901ea9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:05:12.557 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.538+0000 7f8901ea9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:05:12.557 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.538+0000 7f8901ea9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:05:12.557 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.538+0000 7f89016a8640 10 monclient(hunting): handle_auth_done global_id 984246 payload 824 2024-09-16T04:05:12.557 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.538+0000 7f89016a8640 1 -- 172.21.15.42:0/3930694209 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f88fc13c8c0 msgr2=0x7f88fc13ae60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:05:12.557 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.538+0000 7f89016a8640 1 --2- 172.21.15.42:0/3930694209 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f88fc13c8c0 0x7f88fc13ae60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:12.557 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.538+0000 7f89016a8640 1 -- 172.21.15.42:0/3930694209 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f88fc0a39b0 msgr2=0x7f88fc13a3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:05:12.557 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.538+0000 7f89016a8640 1 --2- 172.21.15.42:0/3930694209 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f88fc0a39b0 0x7f88fc13a3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:12.557 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.538+0000 7f89016a8640 10 monclient: _finish_hunting 0 2024-09-16T04:05:12.557 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.538+0000 7f89016a8640 1 monclient: found mon.noname-a 2024-09-16T04:05:12.557 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.538+0000 7f89016a8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:05:12.557 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.538+0000 7f89016a8640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:05:12.557 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.538+0000 7f89016a8640 1 -- 172.21.15.42:0/3930694209 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88fc132a70 con 0x7f88fc0a3d90 2024-09-16T04:05:12.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.538+0000 7f89016a8640 1 --2- 172.21.15.42:0/3930694209 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f88fc0a3d90 0x7f88fc13a920 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f88ec0ab000 tx=0x7f88ec019100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:05:12.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.538+0000 7f88f27fc640 10 client.?.objecter ms_handle_connect 0x7f88fc0a3d90 2024-09-16T04:05:12.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.538+0000 7f88f27fc640 10 client.?.objecter resend_mon_ops 2024-09-16T04:05:12.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.538+0000 7f88f27fc640 1 -- 172.21.15.42:0/3930694209 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88ec0b0040 con 0x7f88fc0a3d90 2024-09-16T04:05:12.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.538+0000 7f88f27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:05:12.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.538+0000 7f88f27fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:05:12.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.538+0000 7f88f27fc640 10 monclient: dump: 2024-09-16T04:05:12.558 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:05:12.558 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:05:12.558 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:12.558 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:12.558 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:05:12.558 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:05:12.559 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:05:12.559 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:05:12.559 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:05:12.559 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:05:12.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.538+0000 7f88f27fc640 10 monclient: _finish_auth 0 2024-09-16T04:05:12.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.538+0000 7f88f27fc640 10 monclient: _check_auth_tickets 2024-09-16T04:05:12.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.538+0000 7f88f27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:05:12.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.538+0000 7f88f27fc640 1 -- 172.21.15.42:0/3930694209 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f88ec0b9070 con 0x7f88fc0a3d90 2024-09-16T04:05:12.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.538+0000 7f88f27fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:05:12.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.538+0000 7f88f27fc640 1 -- 172.21.15.42:0/3930694209 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88ec0b4890 con 0x7f88fc0a3d90 2024-09-16T04:05:12.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.538+0000 7f88f27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:05:12.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.538+0000 7f88f27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:05:12.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.538+0000 7f88f27fc640 10 monclient: dump: 2024-09-16T04:05:12.560 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:05:12.560 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:05:12.560 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:12.560 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:12.560 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:05:12.560 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:05:12.560 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:05:12.560 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:05:12.560 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:05:12.560 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:05:12.560 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.538+0000 7f88f3fff640 4 set_mon_vals no callback set 2024-09-16T04:05:12.560 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.538+0000 7f890410d640 5 monclient: authenticate success, global_id 984246 2024-09-16T04:05:12.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.538+0000 7f890410d640 10 monclient: _renew_subs 2024-09-16T04:05:12.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.538+0000 7f890410d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:05:12.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.538+0000 7f890410d640 1 -- 172.21.15.42:0/3930694209 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88fc06f680 con 0x7f88fc0a3d90 2024-09-16T04:05:12.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.538+0000 7f890410d640 10 client.984246.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:05:12.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.538+0000 7f890410d640 10 monclient: _renew_subs 2024-09-16T04:05:12.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.538+0000 7f890410d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:05:12.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.538+0000 7f890410d640 1 -- 172.21.15.42:0/3930694209 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88fc09eb40 con 0x7f88fc0a3d90 2024-09-16T04:05:12.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.538+0000 7f890410d640 1 librados: init done 2024-09-16T04:05:12.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.538+0000 7f890410d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:05:12.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.538+0000 7f890410d640 1 --2- 172.21.15.42:0/3930694209 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f88fc1cf0c0 0x7f88fc1cf4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:05:12.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.538+0000 7f890410d640 10 monclient(hunting): start opening mon connection 2024-09-16T04:05:12.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.538+0000 7f890410d640 1 -- 172.21.15.42:0/3930694209 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f88fc133b80 con 0x7f88fc1cf0c0 2024-09-16T04:05:12.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.538+0000 7f8901ea9640 1 --2- 172.21.15.42:0/3930694209 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f88fc1cf0c0 0x7f88fc1cf4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:05:12.562 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.542+0000 7f8901ea9640 10 monclient: get_auth_request con 0x7f88fc1cf0c0 auth_method 0 2024-09-16T04:05:12.562 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.542+0000 7f8901ea9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:05:12.562 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.542+0000 7f8901ea9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:05:12.562 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.542+0000 7f8901ea9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:05:12.562 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.542+0000 7f8901ea9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:05:12.562 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.542+0000 7f8901ea9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:05:12.562 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.542+0000 7f8901ea9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:05:12.562 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.542+0000 7f88f27fc640 1 -- 172.21.15.42:0/3930694209 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f88ec0ab4c0 con 0x7f88fc0a3d90 2024-09-16T04:05:12.562 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.542+0000 7f8901ea9640 10 monclient(hunting): handle_auth_done global_id 984249 payload 824 2024-09-16T04:05:12.562 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.542+0000 7f88f27fc640 1 --2- 172.21.15.42:0/3930694209 >> v2:172.21.15.154:6812/254845342 conn(0x7f88e0037810 0x7f88e0039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:05:12.562 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.542+0000 7f8901ea9640 1 -- 172.21.15.42:0/3930694209 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f88fc0a5750 con 0x7f88fc1cf0c0 2024-09-16T04:05:12.562 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.542+0000 7f88f27fc640 1 -- 172.21.15.42:0/3930694209 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(700..700 src has 451..700) ==== 5899+0+0 (secure 0 0 0) 0x7f88ec0e2c50 con 0x7f88fc0a3d90 2024-09-16T04:05:12.562 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.542+0000 7f88f27fc640 10 client.984246.objecter ms_dispatch 0x7f88fc000ba0 osd_map(700..700 src has 451..700) 2024-09-16T04:05:12.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.542+0000 7f88f27fc640 3 client.984246.objecter handle_osd_map got epochs [700,700] > 0 2024-09-16T04:05:12.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.542+0000 7f88f27fc640 3 client.984246.objecter handle_osd_map decoding full epoch 700 2024-09-16T04:05:12.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.542+0000 7f8901ea9640 1 --2- 172.21.15.42:0/3930694209 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f88fc1cf0c0 0x7f88fc1cf4a0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f88f800f8c0 tx=0x7f88f800fd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:05:12.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.542+0000 7f88f27fc640 20 client.984246.objecter dump_active .. 0 homeless 2024-09-16T04:05:12.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.542+0000 7f88f27fc640 10 client.984246.objecter ms_handle_connect 0x7f88fc1cf0c0 2024-09-16T04:05:12.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.542+0000 7f88f27fc640 10 client.984246.objecter resend_mon_ops 2024-09-16T04:05:12.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.542+0000 7f89026aa640 1 --2- 172.21.15.42:0/3930694209 >> v2:172.21.15.154:6812/254845342 conn(0x7f88e0037810 0x7f88e0039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:05:12.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.542+0000 7f88f27fc640 1 -- 172.21.15.42:0/3930694209 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88f8019070 con 0x7f88fc1cf0c0 2024-09-16T04:05:12.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.542+0000 7f88f27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:05:12.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.542+0000 7f88f27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:05:12.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.542+0000 7f88f27fc640 10 monclient: dump: 2024-09-16T04:05:12.563 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:05:12.563 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:05:12.563 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:12.563 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:12.564 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:05:12.564 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:05:12.564 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:05:12.564 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:05:12.564 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:05:12.564 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:05:12.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.542+0000 7f89026aa640 10 monclient: get_auth_request con 0x7f88e0037810 auth_method 0 2024-09-16T04:05:12.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.542+0000 7f88f27fc640 1 -- 172.21.15.42:0/3930694209 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f88f8006d70 con 0x7f88fc1cf0c0 2024-09-16T04:05:12.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.542+0000 7f88f27fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:05:12.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.542+0000 7f88f27fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:05:12.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.542+0000 7f88f27fc640 1 -- 172.21.15.42:0/3930694209 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f88fc1cf0c0 msgr2=0x7f88fc1cf4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:05:12.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.542+0000 7f88f27fc640 1 --2- 172.21.15.42:0/3930694209 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f88fc1cf0c0 0x7f88fc1cf4a0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f88f800f8c0 tx=0x7f88f800fd80 comp rx=0 tx=0).stop 2024-09-16T04:05:12.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.542+0000 7f88f27fc640 1 -- 172.21.15.42:0/3930694209 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f88fc1cf0c0 msgr2=0x7f88fc1cf4a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:05:12.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.542+0000 7f88f27fc640 1 --2- 172.21.15.42:0/3930694209 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f88fc1cf0c0 0x7f88fc1cf4a0 secure :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f88f800f8c0 tx=0x7f88f800fd80 comp rx=0 tx=0).stop 2024-09-16T04:05:12.565 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.542+0000 7f89026aa640 1 --2- 172.21.15.42:0/3930694209 >> v2:172.21.15.154:6812/254845342 conn(0x7f88e0037810 0x7f88e0039cd0 secure :-1 s=READY pgs=3840 cs=0 l=1 rev1=1 crypto rx=0x7f88fc09d140 tx=0x7f88f4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:05:12.565 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.542+0000 7f88f27fc640 10 client.984246.objecter ms_handle_connect 0x7f88e0037810 2024-09-16T04:05:12.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.546+0000 7f890410d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:05:12.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.546+0000 7f890410d640 1 --2- 172.21.15.42:0/3930694209 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f88fc1cf9e0 0x7f88fc136bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:05:12.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.546+0000 7f890410d640 10 monclient(hunting): start opening mon connection 2024-09-16T04:05:12.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.546+0000 7f890410d640 1 -- 172.21.15.42:0/3930694209 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f88fc068e40 con 0x7f88fc1cf9e0 2024-09-16T04:05:12.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.546+0000 7f8901ea9640 1 --2- 172.21.15.42:0/3930694209 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f88fc1cf9e0 0x7f88fc136bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:05:12.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.546+0000 7f8901ea9640 10 monclient: get_auth_request con 0x7f88fc1cf9e0 auth_method 0 2024-09-16T04:05:12.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.546+0000 7f8901ea9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:05:12.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.546+0000 7f8901ea9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:05:12.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.546+0000 7f8901ea9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:05:12.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.546+0000 7f8901ea9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:05:12.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.546+0000 7f8901ea9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:05:12.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.546+0000 7f8901ea9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:05:12.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.546+0000 7f8901ea9640 10 monclient(hunting): handle_auth_done global_id 984252 payload 824 2024-09-16T04:05:12.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.546+0000 7f8901ea9640 1 -- 172.21.15.42:0/3930694209 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f88fc13c040 con 0x7f88fc1cf9e0 2024-09-16T04:05:12.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.546+0000 7f8901ea9640 1 --2- 172.21.15.42:0/3930694209 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f88fc1cf9e0 0x7f88fc136bc0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f88f800f8c0 tx=0x7f88f8001260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:05:12.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.546+0000 7f88f27fc640 10 client.984246.objecter ms_handle_connect 0x7f88fc1cf9e0 2024-09-16T04:05:12.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.546+0000 7f88f27fc640 10 client.984246.objecter resend_mon_ops 2024-09-16T04:05:12.568 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.546+0000 7f88f27fc640 1 -- 172.21.15.42:0/3930694209 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88f8019070 con 0x7f88fc1cf9e0 2024-09-16T04:05:12.568 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.546+0000 7f88f27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:05:12.568 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.546+0000 7f88f27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:05:12.568 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.546+0000 7f88f27fc640 10 monclient: dump: 2024-09-16T04:05:12.568 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:05:12.568 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:05:12.568 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:12.568 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:12.568 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:05:12.568 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:05:12.568 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:05:12.568 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:05:12.568 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:05:12.569 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:05:12.569 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.550+0000 7f88f27fc640 1 -- 172.21.15.42:0/3930694209 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4714 (secure 0 0 0) 0x7f88f80056a0 con 0x7f88fc1cf9e0 2024-09-16T04:05:12.569 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.550+0000 7f88f27fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:05:12.569 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.550+0000 7f88f27fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:05:12.569 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.550+0000 7f88f27fc640 1 -- 172.21.15.42:0/3930694209 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f88fc1cf9e0 msgr2=0x7f88fc136bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:05:12.569 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.550+0000 7f88f27fc640 1 --2- 172.21.15.42:0/3930694209 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f88fc1cf9e0 0x7f88fc136bc0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f88f800f8c0 tx=0x7f88f8001260 comp rx=0 tx=0).stop 2024-09-16T04:05:12.569 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.550+0000 7f88f27fc640 1 -- 172.21.15.42:0/3930694209 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f88fc1cf9e0 msgr2=0x7f88fc136bc0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:05:12.569 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.550+0000 7f88f27fc640 1 --2- 172.21.15.42:0/3930694209 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f88fc1cf9e0 0x7f88fc136bc0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:12.569 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T04:05:12.569 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:05:12.570 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:05:12.570 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "leader", 2024-09-16T04:05:12.570 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 216, 2024-09-16T04:05:12.570 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 72603, 2024-09-16T04:05:12.570 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T04:05:12.570 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T04:05:12.570 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T04:05:12.570 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T04:05:12.570 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:05:12.570 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 2, 2024-09-16T04:05:12.570 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:05:12.570 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T04:05:12.571 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T04:05:12.571 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:05:12.571 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:05:12.571 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:05:12.571 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:05:12.571 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:05:12.571 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:05:12.571 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:05:12.571 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:05:12.571 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:05:12.571 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:05:12.571 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:05:12.571 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:05:12.572 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:05:12.572 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T04:05:12.572 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:05:12.572 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:05:12.572 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:05:12.572 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:05:12.572 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:05:12.572 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:05:12.572 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:05:12.572 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:05:12.572 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:05:12.572 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:05:12.573 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:05:12.573 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:05:12.573 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:05:12.573 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T04:05:12.573 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T04:05:12.573 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T04:05:12.573 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T04:05:12.573 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T04:05:12.573 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T04:05:12.573 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:05:12.573 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:05:12.573 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T04:05:12.573 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T04:05:12.573 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T04:05:12.573 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T04:05:12.574 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T04:05:12.574 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T04:05:12.574 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T04:05:12.574 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:05:12.574 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T04:05:12.574 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:05:12.574 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:05:12.574 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:05:12.574 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:05:12.574 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:05:12.574 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:05:12.574 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:05:12.574 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:05:12.574 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:05:12.575 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:05:12.575 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:05:12.575 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:05:12.575 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T04:05:12.575 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:05:12.575 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T04:05:12.575 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:05:12.575 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:05:12.575 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:05:12.575 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:05:12.575 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:05:12.575 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:05:12.575 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:05:12.576 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T04:05:12.576 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:05:12.576 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:05:12.576 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:05:12.576 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:05:12.576 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T04:05:12.576 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:05:12.576 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:05:12.576 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:05:12.576 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:05:12.576 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T04:05:12.576 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T04:05:12.576 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:05:12.576 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:05:12.577 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:05:12.577 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:05:12.577 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:05:12.577 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:05:12.577 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:05:12.577 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:05:12.577 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:05:12.577 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:05:12.577 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:05:12.577 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T04:05:12.577 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:05:12.577 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:05:12.577 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:05:12.577 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:05:12.578 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T04:05:12.578 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:05:12.578 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:05:12.578 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:05:12.578 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:05:12.578 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T04:05:12.578 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T04:05:12.578 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:05:12.578 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:05:12.578 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:05:12.578 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:05:12.578 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:05:12.578 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:05:12.579 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:05:12.579 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:05:12.579 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:05:12.579 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:05:12.579 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:05:12.579 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T04:05:12.579 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:05:12.579 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:05:12.579 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:05:12.579 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:05:12.579 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T04:05:12.579 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:05:12.579 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:05:12.579 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:05:12.579 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:05:12.580 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T04:05:12.580 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T04:05:12.580 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:05:12.580 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:05:12.580 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:05:12.580 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:05:12.580 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:05:12.580 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:05:12.580 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T04:05:12.580 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T04:05:12.580 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:05:12.580 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:05:12.580 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:05:12.580 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:05:12.580 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:05:12.581 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:05:12.581 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T04:05:12.581 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:05:12.581 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:05:12.581 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:05:12.581 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 3 2024-09-16T04:05:12.581 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:05:12.581 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:05:12.581 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T04:05:12.581 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:05:12.581 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:05:12.581 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:05:12.581 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 3 2024-09-16T04:05:12.581 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:05:12.582 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:05:12.582 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:05:12.582 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T04:05:12.582 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T04:05:12.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.550+0000 7f890410d640 10 librados: watch_flush enter 2024-09-16T04:05:12.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.550+0000 7f890410d640 10 librados: watch_flush exit 2024-09-16T04:05:12.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.550+0000 7f890410d640 20 client.984246.objecter shutdown clearing up homeless session... 2024-09-16T04:05:12.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.550+0000 7f890410d640 10 client.984246.objecter successfully canceled tick 2024-09-16T04:05:12.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.550+0000 7f890410d640 1 -- 172.21.15.42:0/3930694209 >> v2:172.21.15.154:6812/254845342 conn(0x7f88e0037810 msgr2=0x7f88e0039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:05:12.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.550+0000 7f890410d640 1 --2- 172.21.15.42:0/3930694209 >> v2:172.21.15.154:6812/254845342 conn(0x7f88e0037810 0x7f88e0039cd0 secure :-1 s=READY pgs=3840 cs=0 l=1 rev1=1 crypto rx=0x7f88fc09d140 tx=0x7f88f4008040 comp rx=0 tx=0).stop 2024-09-16T04:05:12.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.550+0000 7f890410d640 10 monclient: shutdown 2024-09-16T04:05:12.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.550+0000 7f890410d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:05:12.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.550+0000 7f890410d640 1 -- 172.21.15.42:0/3930694209 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f88fc0a3d90 msgr2=0x7f88fc13a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:05:12.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.550+0000 7f890410d640 1 --2- 172.21.15.42:0/3930694209 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f88fc0a3d90 0x7f88fc13a920 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f88ec0ab000 tx=0x7f88ec019100 comp rx=0 tx=0).stop 2024-09-16T04:05:12.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.550+0000 7f89026aa640 1 -- 172.21.15.42:0/3930694209 reap_dead start 2024-09-16T04:05:12.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.550+0000 7f89026aa640 1 -- 172.21.15.42:0/3930694209 reap_dead start 2024-09-16T04:05:12.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.550+0000 7f890410d640 1 -- 172.21.15.42:0/3930694209 shutdown_connections 2024-09-16T04:05:12.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.550+0000 7f890410d640 1 -- 172.21.15.42:0/3930694209 >> 172.21.15.42:0/3930694209 conn(0x7f88fc0a9fb0 msgr2=0x7f88fc131980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:05:12.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.550+0000 7f890410d640 1 -- 172.21.15.42:0/3930694209 shutdown_connections 2024-09-16T04:05:12.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.550+0000 7f890410d640 1 -- 172.21.15.42:0/3930694209 wait complete. 2024-09-16T04:05:12.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.550+0000 7f890410d640 1 librados: shutdown 2024-09-16T04:05:12.585 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:05:12.585 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T04:05:12.585 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T04:05:12.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.702+0000 7f417a496640 10 monclient: get_monmap_and_config 2024-09-16T04:05:12.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.702+0000 7f417a496640 10 monclient: build_initial_monmap 2024-09-16T04:05:12.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.702+0000 7f417a496640 10 monclient: monmap: 2024-09-16T04:05:12.707 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:05:12.707 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:05:12.707 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:05:12.705312+0000 2024-09-16T04:05:12.707 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:05:12.705312+0000 2024-09-16T04:05:12.707 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:05:12.707 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:05:12.707 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:05:12.707 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:05:12.707 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:05:12.707 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:05:12.707 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:05:12.707 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:05:12.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.702+0000 7f417a496640 1 Processor -- start 2024-09-16T04:05:12.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.702+0000 7f417a496640 1 -- start start 2024-09-16T04:05:12.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.702+0000 7f417a496640 10 monclient: init 2024-09-16T04:05:12.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.702+0000 7f417a496640 10 monclient: _reopen_session rank -1 2024-09-16T04:05:12.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.702+0000 7f417a496640 10 monclient: _add_conns ranks=[2,4,3,0,1] 2024-09-16T04:05:12.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.702+0000 7f417a496640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4174130bb0 0x7f4174130f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:05:12.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.702+0000 7f417a496640 10 monclient(hunting): picked mon.noname-b con 0x7f4174130bb0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:05:12.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.702+0000 7f417a496640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f41740a2030 0x7f4174072c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:05:12.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.702+0000 7f417a496640 10 monclient(hunting): picked mon.noname-c con 0x7f41740a2030 addr v2:172.21.15.154:3300/0 2024-09-16T04:05:12.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.702+0000 7f417a496640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f41740a2410 0x7f4174073180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:05:12.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.702+0000 7f417a496640 10 monclient(hunting): picked mon.noname-a con 0x7f41740a2410 addr v2:172.21.15.42:3300/0 2024-09-16T04:05:12.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.702+0000 7f417a496640 10 monclient(hunting): start opening mon connection 2024-09-16T04:05:12.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.702+0000 7f417a496640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f41740a1470 con 0x7f41740a2410 2024-09-16T04:05:12.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.702+0000 7f417a496640 10 monclient(hunting): start opening mon connection 2024-09-16T04:05:12.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.702+0000 7f417a496640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f41740a1750 con 0x7f41740a2030 2024-09-16T04:05:12.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.702+0000 7f417a496640 10 monclient(hunting): start opening mon connection 2024-09-16T04:05:12.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.702+0000 7f417a496640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f41740a1a30 con 0x7f4174130bb0 2024-09-16T04:05:12.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.702+0000 7f417a496640 10 monclient(hunting): _renew_subs 2024-09-16T04:05:12.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.702+0000 7f417a496640 10 monclient(hunting): authenticate will time out at 2359.675507s 2024-09-16T04:05:12.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.702+0000 7f4173fff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4174130bb0 0x7f4174130f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:05:12.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.702+0000 7f4173fff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4174130bb0 0x7f4174130f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:41002/0 (socket says 172.21.15.42:41002) 2024-09-16T04:05:12.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.702+0000 7f4173fff640 1 -- 172.21.15.42:0/816176087 learned_addr learned my addr 172.21.15.42:0/816176087 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:05:12.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.702+0000 7f4173fff640 10 monclient(hunting): get_auth_request con 0x7f4174130bb0 auth_method 0 2024-09-16T04:05:12.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.702+0000 7f4173fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:05:12.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.702+0000 7f4173fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:05:12.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.702+0000 7f4173fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:05:12.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.702+0000 7f4178a33640 1 --2- 172.21.15.42:0/816176087 >> v2:172.21.15.42:3300/0 conn(0x7f41740a2410 0x7f4174073180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:05:12.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.702+0000 7f4178a33640 10 monclient(hunting): get_auth_request con 0x7f41740a2410 auth_method 0 2024-09-16T04:05:12.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.702+0000 7f4178a33640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:05:12.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.702+0000 7f4178a33640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:05:12.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.702+0000 7f4178a33640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:05:12.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.702+0000 7f4173fff640 1 -- 172.21.15.42:0/816176087 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4174130bb0 msgr2=0x7f4174130f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:05:12.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.702+0000 7f4173fff640 1 -- 172.21.15.42:0/816176087 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4174130bb0 msgr2=0x7f4174130f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:05:12.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.702+0000 7f4173fff640 1 --2- 172.21.15.42:0/816176087 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4174130bb0 0x7f4174130f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:05:12.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.702+0000 7f4178a33640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:05:12.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.702+0000 7f4178a33640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:05:12.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.702+0000 7f4173fff640 1 --2- 172.21.15.42:0/816176087 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4174130bb0 0x7f4174130f90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:05:12.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.702+0000 7f4178a33640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:05:12.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.706+0000 7f41737fe640 1 --2- 172.21.15.42:0/816176087 >> v2:172.21.15.154:3300/0 conn(0x7f41740a2030 0x7f4174072c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:05:12.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.706+0000 7f41737fe640 10 monclient(hunting): get_auth_request con 0x7f41740a2030 auth_method 0 2024-09-16T04:05:12.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.706+0000 7f41737fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:05:12.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.706+0000 7f41737fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:05:12.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.706+0000 7f41737fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:05:12.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.706+0000 7f4178a33640 10 monclient(hunting): handle_auth_done global_id 984255 payload 274 2024-09-16T04:05:12.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.706+0000 7f4178a33640 1 -- 172.21.15.42:0/816176087 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4174130bb0 msgr2=0x7f4174130f90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:05:12.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.706+0000 7f4178a33640 1 --2- 172.21.15.42:0/816176087 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4174130bb0 0x7f4174130f90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:12.714 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.706+0000 7f4178a33640 1 -- 172.21.15.42:0/816176087 >> v2:172.21.15.154:3300/0 conn(0x7f41740a2030 msgr2=0x7f4174072c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:05:12.714 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.706+0000 7f4178a33640 1 --2- 172.21.15.42:0/816176087 >> v2:172.21.15.154:3300/0 conn(0x7f41740a2030 0x7f4174072c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:12.714 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.706+0000 7f4178a33640 10 monclient: _finish_hunting 0 2024-09-16T04:05:12.714 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.706+0000 7f4178a33640 1 monclient: found mon.noname-a 2024-09-16T04:05:12.714 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.706+0000 7f4178a33640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:05:12.714 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.706+0000 7f4178a33640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:05:12.714 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.706+0000 7f4178a33640 1 -- 172.21.15.42:0/816176087 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41741314d0 con 0x7f41740a2410 2024-09-16T04:05:12.720 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.706+0000 7f4178a33640 1 --2- 172.21.15.42:0/816176087 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f41740a2410 0x7f4174073180 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f417406e2b0 tx=0x7f416c01c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4aa42a4d9557fcc server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:05:12.720 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.706+0000 7f4172ffd640 1 -- 172.21.15.42:0/816176087 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f416c0ac070 con 0x7f41740a2410 2024-09-16T04:05:12.720 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.706+0000 7f4172ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:05:12.720 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.706+0000 7f4172ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:05:12.720 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.706+0000 7f4172ffd640 10 monclient: dump: 2024-09-16T04:05:12.720 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:05:12.720 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:05:12.720 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:12.720 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:12.720 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:05:12.720 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:05:12.721 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:05:12.721 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:05:12.721 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:05:12.721 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:05:12.721 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.706+0000 7f4172ffd640 10 monclient: _finish_auth 0 2024-09-16T04:05:12.721 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.706+0000 7f4172ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:05:12.721 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.706+0000 7f4172ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:05:12.721 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.706+0000 7f4172ffd640 1 -- 172.21.15.42:0/816176087 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f416c0a6680 con 0x7f41740a2410 2024-09-16T04:05:12.722 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.706+0000 7f4172ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:05:12.722 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.706+0000 7f4172ffd640 1 -- 172.21.15.42:0/816176087 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f416c0a6a50 con 0x7f41740a2410 2024-09-16T04:05:12.722 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.706+0000 7f417a496640 5 monclient: authenticate success, global_id 984255 2024-09-16T04:05:12.722 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.706+0000 7f417a496640 10 monclient: get_monmap_and_config success 2024-09-16T04:05:12.722 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.706+0000 7f417a496640 4 set_mon_vals no callback set 2024-09-16T04:05:12.722 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.706+0000 7f4172ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:05:12.722 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.706+0000 7f4172ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:05:12.722 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.706+0000 7f4172ffd640 10 monclient: dump: 2024-09-16T04:05:12.722 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:05:12.722 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:05:12.722 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:12.722 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:12.722 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:05:12.723 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:05:12.723 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:05:12.723 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:05:12.723 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:05:12.723 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:05:12.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.706+0000 7f417a496640 10 monclient: shutdown 2024-09-16T04:05:12.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.706+0000 7f417a496640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:05:12.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.706+0000 7f417a496640 1 -- 172.21.15.42:0/816176087 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f41740a2410 msgr2=0x7f4174073180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:05:12.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.706+0000 7f417a496640 1 --2- 172.21.15.42:0/816176087 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f41740a2410 0x7f4174073180 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f417406e2b0 tx=0x7f416c01c7f0 comp rx=0 tx=0).stop 2024-09-16T04:05:12.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.706+0000 7f417a496640 1 -- 172.21.15.42:0/816176087 shutdown_connections 2024-09-16T04:05:12.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.706+0000 7f417a496640 1 --2- 172.21.15.42:0/816176087 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f41740a2410 0x7f4174073180 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:12.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.706+0000 7f417a496640 1 --2- 172.21.15.42:0/816176087 >> v2:172.21.15.154:3300/0 conn(0x7f41740a2030 0x7f4174072c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:12.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.706+0000 7f417a496640 1 --2- 172.21.15.42:0/816176087 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4174130bb0 0x7f4174130f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:12.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.706+0000 7f417a496640 1 -- 172.21.15.42:0/816176087 >> 172.21.15.42:0/816176087 conn(0x7f41740a2ad0 msgr2=0x7f41740a2ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:05:12.724 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.710+0000 7f417a496640 1 -- 172.21.15.42:0/816176087 shutdown_connections 2024-09-16T04:05:12.724 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.710+0000 7f417a496640 1 -- 172.21.15.42:0/816176087 wait complete. 2024-09-16T04:05:12.724 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.710+0000 7f417a496640 10 monclient: build_initial_monmap 2024-09-16T04:05:12.724 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.710+0000 7f417a496640 10 monclient: monmap: 2024-09-16T04:05:12.724 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:05:12.724 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:05:12.724 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:05:12.724 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:05:12.724 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:05:12.724 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:05:12.724 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:05:12.724 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:05:12.725 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:05:12.725 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:05:12.725 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.710+0000 7f417a496640 1 librados: starting msgr at 2024-09-16T04:05:12.725 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.710+0000 7f417a496640 1 librados: starting objecter 2024-09-16T04:05:12.725 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.710+0000 7f417a496640 1 Processor -- start 2024-09-16T04:05:12.725 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.710+0000 7f417a496640 1 -- start start 2024-09-16T04:05:12.725 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.710+0000 7f417a496640 1 librados: setting wanted keys 2024-09-16T04:05:12.725 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.710+0000 7f417a496640 1 librados: calling monclient init 2024-09-16T04:05:12.725 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.710+0000 7f417a496640 10 monclient: init 2024-09-16T04:05:12.725 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.710+0000 7f417a496640 10 monclient: _reopen_session rank -1 2024-09-16T04:05:12.725 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.710+0000 7f417a496640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:05:12.725 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.710+0000 7f417a496640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f41740a2030 0x7f41741cf390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:05:12.725 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.710+0000 7f417a496640 10 monclient(hunting): picked mon.noname-a con 0x7f41740a2030 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:05:12.725 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.710+0000 7f417a496640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f41740a2410 0x7f417413d720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:05:12.725 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.710+0000 7f417a496640 10 monclient(hunting): picked mon.noname-c con 0x7f41740a2410 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:05:12.726 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.710+0000 7f417a496640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4174130bb0 0x7f417413dc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:05:12.726 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.710+0000 7f417a496640 10 monclient(hunting): picked mon.noname-b con 0x7f4174130bb0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:05:12.726 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.710+0000 7f417a496640 10 monclient(hunting): start opening mon connection 2024-09-16T04:05:12.726 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.710+0000 7f417a496640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f4174140d90 con 0x7f41740a2030 2024-09-16T04:05:12.726 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.710+0000 7f417a496640 10 monclient(hunting): start opening mon connection 2024-09-16T04:05:12.726 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.710+0000 7f417a496640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f4174073ad0 con 0x7f4174130bb0 2024-09-16T04:05:12.726 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.710+0000 7f417a496640 10 monclient(hunting): start opening mon connection 2024-09-16T04:05:12.726 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.710+0000 7f417a496640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f4174140f10 con 0x7f41740a2410 2024-09-16T04:05:12.726 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.710+0000 7f417a496640 10 monclient(hunting): _renew_subs 2024-09-16T04:05:12.726 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.710+0000 7f417a496640 10 monclient(hunting): authenticate will time out at 2359.681932s 2024-09-16T04:05:12.726 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.710+0000 7f41737fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f41740a2410 0x7f417413d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:05:12.726 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.710+0000 7f4173fff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f41740a2030 0x7f41741cf390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:05:12.726 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.710+0000 7f41737fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f41740a2410 0x7f417413d720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:41010/0 (socket says 172.21.15.42:41010) 2024-09-16T04:05:12.727 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.710+0000 7f4173fff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f41740a2030 0x7f41741cf390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:33416/0 (socket says 172.21.15.42:33416) 2024-09-16T04:05:12.727 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.710+0000 7f41737fe640 1 -- 172.21.15.42:0/1972894322 learned_addr learned my addr 172.21.15.42:0/1972894322 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:05:12.727 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.710+0000 7f41737fe640 10 monclient(hunting): get_auth_request con 0x7f41740a2410 auth_method 0 2024-09-16T04:05:12.727 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.710+0000 7f41737fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:05:12.727 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.710+0000 7f41737fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:05:12.727 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.710+0000 7f41737fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:05:12.727 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.710+0000 7f4173fff640 10 monclient(hunting): get_auth_request con 0x7f41740a2030 auth_method 0 2024-09-16T04:05:12.727 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.710+0000 7f4173fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:05:12.727 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.710+0000 7f4173fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:05:12.727 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.710+0000 7f4173fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:05:12.727 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.710+0000 7f4173fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:05:12.727 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.710+0000 7f4173fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:05:12.727 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.710+0000 7f41737fe640 1 -- 172.21.15.42:0/1972894322 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f41740a2410 msgr2=0x7f417413d720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-16T04:05:12.728 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.710+0000 7f41737fe640 1 -- 172.21.15.42:0/1972894322 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f41740a2410 msgr2=0x7f417413d720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:05:12.728 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.710+0000 7f41737fe640 1 --2- 172.21.15.42:0/1972894322 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f41740a2410 0x7f417413d720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:05:12.728 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.710+0000 7f4173fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:05:12.728 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.710+0000 7f41737fe640 1 --2- 172.21.15.42:0/1972894322 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f41740a2410 0x7f417413d720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:05:12.728 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.710+0000 7f4173fff640 10 monclient(hunting): handle_auth_done global_id 984258 payload 824 2024-09-16T04:05:12.728 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.710+0000 7f4173fff640 1 -- 172.21.15.42:0/1972894322 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f41740a2410 msgr2=0x7f417413d720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:05:12.728 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.710+0000 7f4173fff640 1 --2- 172.21.15.42:0/1972894322 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f41740a2410 0x7f417413d720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:12.728 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.710+0000 7f4173fff640 1 -- 172.21.15.42:0/1972894322 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4174130bb0 msgr2=0x7f417413dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:05:12.728 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.710+0000 7f4173fff640 1 --2- 172.21.15.42:0/1972894322 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4174130bb0 0x7f417413dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:12.728 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.710+0000 7f4173fff640 10 monclient: _finish_hunting 0 2024-09-16T04:05:12.728 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.710+0000 7f4173fff640 1 monclient: found mon.noname-a 2024-09-16T04:05:12.728 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.710+0000 7f4173fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:05:12.728 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.710+0000 7f4173fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:05:12.728 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.710+0000 7f4173fff640 1 -- 172.21.15.42:0/1972894322 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41741d02a0 con 0x7f41740a2030 2024-09-16T04:05:12.729 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.710+0000 7f4173fff640 1 --2- 172.21.15.42:0/1972894322 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f41740a2030 0x7f41741cf390 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f416400e810 tx=0x7f416400ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:05:12.729 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.710+0000 7f4170ff9640 10 client.?.objecter ms_handle_connect 0x7f41740a2030 2024-09-16T04:05:12.729 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.710+0000 7f4170ff9640 10 client.?.objecter resend_mon_ops 2024-09-16T04:05:12.729 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.710+0000 7f4170ff9640 1 -- 172.21.15.42:0/1972894322 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4164017070 con 0x7f41740a2030 2024-09-16T04:05:12.729 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.710+0000 7f4170ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:05:12.729 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.710+0000 7f4170ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:05:12.729 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.710+0000 7f4170ff9640 10 monclient: dump: 2024-09-16T04:05:12.729 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:05:12.729 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:05:12.729 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:12.729 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:12.729 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:05:12.729 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:05:12.729 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:05:12.730 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:05:12.730 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:05:12.730 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:05:12.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.710+0000 7f4170ff9640 10 monclient: _finish_auth 0 2024-09-16T04:05:12.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.710+0000 7f4170ff9640 10 monclient: _check_auth_tickets 2024-09-16T04:05:12.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.710+0000 7f4170ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:05:12.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.710+0000 7f417a496640 5 monclient: authenticate success, global_id 984258 2024-09-16T04:05:12.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.710+0000 7f4170ff9640 1 -- 172.21.15.42:0/1972894322 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4164004020 con 0x7f41740a2030 2024-09-16T04:05:12.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.710+0000 7f417a496640 10 monclient: _renew_subs 2024-09-16T04:05:12.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.710+0000 7f417a496640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:05:12.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.710+0000 7f417a496640 1 -- 172.21.15.42:0/1972894322 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4174139ef0 con 0x7f41740a2030 2024-09-16T04:05:12.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.710+0000 7f4170ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T04:05:12.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.710+0000 7f4170ff9640 1 -- 172.21.15.42:0/1972894322 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f416400f040 con 0x7f41740a2030 2024-09-16T04:05:12.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.710+0000 7f4170ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:05:12.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.710+0000 7f4170ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:05:12.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.710+0000 7f4170ff9640 10 monclient: dump: 2024-09-16T04:05:12.731 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:05:12.731 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:05:12.731 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:12.731 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:12.731 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:05:12.731 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:05:12.731 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:05:12.731 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:05:12.731 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:05:12.731 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:05:12.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.710+0000 7f417a496640 10 client.984258.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:05:12.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.710+0000 7f417a496640 10 monclient: _renew_subs 2024-09-16T04:05:12.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.710+0000 7f417a496640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:05:12.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.710+0000 7f417a496640 1 -- 172.21.15.42:0/1972894322 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4174139930 con 0x7f41740a2030 2024-09-16T04:05:12.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.710+0000 7f417a496640 1 librados: init done 2024-09-16T04:05:12.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.710+0000 7f41727fc640 4 set_mon_vals no callback set 2024-09-16T04:05:12.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.710+0000 7f417a496640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:05:12.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.710+0000 7f417a496640 1 --2- 172.21.15.42:0/1972894322 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f417409e740 0x7f417409eb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:05:12.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.710+0000 7f417a496640 10 monclient(hunting): start opening mon connection 2024-09-16T04:05:12.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.710+0000 7f417a496640 1 -- 172.21.15.42:0/1972894322 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f4174130f90 con 0x7f417409e740 2024-09-16T04:05:12.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.710+0000 7f41737fe640 1 --2- 172.21.15.42:0/1972894322 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f417409e740 0x7f417409eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:05:12.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.710+0000 7f41737fe640 10 monclient: get_auth_request con 0x7f417409e740 auth_method 0 2024-09-16T04:05:12.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.710+0000 7f41737fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:05:12.733 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.710+0000 7f41737fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:05:12.733 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.710+0000 7f41737fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:05:12.733 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.710+0000 7f4170ff9640 1 -- 172.21.15.42:0/1972894322 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f416400c6d0 con 0x7f41740a2030 2024-09-16T04:05:12.733 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.710+0000 7f41737fe640 1 -- 172.21.15.42:0/1972894322 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f417409e740 msgr2=0x7f417409eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-16T04:05:12.733 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.710+0000 7f41737fe640 1 -- 172.21.15.42:0/1972894322 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f417409e740 msgr2=0x7f417409eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:05:12.733 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.710+0000 7f41737fe640 1 --2- 172.21.15.42:0/1972894322 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f417409e740 0x7f417409eb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:05:12.733 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.710+0000 7f41737fe640 1 --2- 172.21.15.42:0/1972894322 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f417409e740 0x7f417409eb20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:05:12.733 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.710+0000 7f4170ff9640 1 --2- 172.21.15.42:0/1972894322 >> v2:172.21.15.154:6812/254845342 conn(0x7f4160037810 0x7f4160039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:05:12.733 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.714+0000 7f4170ff9640 1 -- 172.21.15.42:0/1972894322 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(700..700 src has 451..700) ==== 5899+0+0 (secure 0 0 0) 0x7f41640489f0 con 0x7f41740a2030 2024-09-16T04:05:12.733 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.714+0000 7f4170ff9640 10 client.984258.objecter ms_dispatch 0x7f4174000ba0 osd_map(700..700 src has 451..700) 2024-09-16T04:05:12.733 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.714+0000 7f4170ff9640 3 client.984258.objecter handle_osd_map got epochs [700,700] > 0 2024-09-16T04:05:12.733 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.714+0000 7f4170ff9640 3 client.984258.objecter handle_osd_map decoding full epoch 700 2024-09-16T04:05:12.733 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.714+0000 7f4170ff9640 20 client.984258.objecter dump_active .. 0 homeless 2024-09-16T04:05:12.733 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.714+0000 7f4178a33640 1 --2- 172.21.15.42:0/1972894322 >> v2:172.21.15.154:6812/254845342 conn(0x7f4160037810 0x7f4160039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:05:12.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.714+0000 7f4178a33640 10 monclient: get_auth_request con 0x7f4160037810 auth_method 0 2024-09-16T04:05:12.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.714+0000 7f4170ff9640 10 client.984258.objecter ms_handle_connect 0x7f4160037810 2024-09-16T04:05:12.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.714+0000 7f4178a33640 1 --2- 172.21.15.42:0/1972894322 >> v2:172.21.15.154:6812/254845342 conn(0x7f4160037810 0x7f4160039cd0 secure :-1 s=READY pgs=3841 cs=0 l=1 rev1=1 crypto rx=0x7f416c0ae000 tx=0x7f416c004ac0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:05:12.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.914+0000 7f41737fe640 1 --2- 172.21.15.42:0/1972894322 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f417409e740 0x7f417409eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:05:12.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.914+0000 7f41737fe640 10 monclient: get_auth_request con 0x7f417409e740 auth_method 0 2024-09-16T04:05:12.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.914+0000 7f41737fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:05:12.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.914+0000 7f41737fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:05:12.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.914+0000 7f41737fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:05:12.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.914+0000 7f41737fe640 1 -- 172.21.15.42:0/1972894322 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f417409e740 msgr2=0x7f417409eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-16T04:05:12.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.914+0000 7f41737fe640 1 -- 172.21.15.42:0/1972894322 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f417409e740 msgr2=0x7f417409eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:05:12.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.914+0000 7f41737fe640 1 --2- 172.21.15.42:0/1972894322 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f417409e740 0x7f417409eb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:05:12.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:12.914+0000 7f41737fe640 1 --2- 172.21.15.42:0/1972894322 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f417409e740 0x7f417409eb20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T04:05:13.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:13.318+0000 7f41737fe640 1 --2- 172.21.15.42:0/1972894322 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f417409e740 0x7f417409eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:05:13.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:13.318+0000 7f41737fe640 10 monclient: get_auth_request con 0x7f417409e740 auth_method 0 2024-09-16T04:05:13.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:13.318+0000 7f41737fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:05:13.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:13.318+0000 7f41737fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:05:13.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:13.318+0000 7f41737fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:05:13.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:13.318+0000 7f41737fe640 1 -- 172.21.15.42:0/1972894322 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f417409e740 msgr2=0x7f417409eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-16T04:05:13.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:13.318+0000 7f41737fe640 1 -- 172.21.15.42:0/1972894322 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f417409e740 msgr2=0x7f417409eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:05:13.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:13.318+0000 7f41737fe640 1 --2- 172.21.15.42:0/1972894322 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f417409e740 0x7f417409eb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:05:13.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:13.318+0000 7f41737fe640 1 --2- 172.21.15.42:0/1972894322 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f417409e740 0x7f417409eb20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T04:05:14.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:14.122+0000 7f41737fe640 1 --2- 172.21.15.42:0/1972894322 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f417409e740 0x7f417409eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:05:14.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:14.122+0000 7f41737fe640 10 monclient: get_auth_request con 0x7f417409e740 auth_method 0 2024-09-16T04:05:14.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:14.122+0000 7f41737fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:05:14.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:14.122+0000 7f41737fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:05:14.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:14.122+0000 7f41737fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:05:14.128 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:14.122+0000 7f41737fe640 1 -- 172.21.15.42:0/1972894322 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f417409e740 msgr2=0x7f417409eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-16T04:05:14.128 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:14.122+0000 7f41737fe640 1 -- 172.21.15.42:0/1972894322 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f417409e740 msgr2=0x7f417409eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:05:14.128 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:14.122+0000 7f41737fe640 1 --2- 172.21.15.42:0/1972894322 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f417409e740 0x7f417409eb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:05:14.128 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:14.122+0000 7f41737fe640 1 --2- 172.21.15.42:0/1972894322 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f417409e740 0x7f417409eb20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T04:05:15.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.710+0000 7f415b7fe640 10 monclient: tick 2024-09-16T04:05:15.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.710+0000 7f415b7fe640 10 monclient: _check_auth_tickets 2024-09-16T04:05:15.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.710+0000 7f415b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:05:15.716 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.710+0000 7f415b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:05:15.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.726+0000 7f41737fe640 1 --2- 172.21.15.42:0/1972894322 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f417409e740 0x7f417409eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:05:15.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.726+0000 7f41737fe640 10 monclient: get_auth_request con 0x7f417409e740 auth_method 0 2024-09-16T04:05:15.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.726+0000 7f41737fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:05:15.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.726+0000 7f41737fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:05:15.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.726+0000 7f41737fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:05:15.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.726+0000 7f41737fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:05:15.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.726+0000 7f41737fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:05:15.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.726+0000 7f41737fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:05:15.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.726+0000 7f41737fe640 10 monclient(hunting): handle_auth_done global_id 994100 payload 824 2024-09-16T04:05:15.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.726+0000 7f41737fe640 1 -- 172.21.15.42:0/1972894322 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f417409cc20 con 0x7f417409e740 2024-09-16T04:05:15.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.726+0000 7f41737fe640 1 --2- 172.21.15.42:0/1972894322 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f417409e740 0x7f417409eb20 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f41740a00d0 tx=0x7f4168017e10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:05:15.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.726+0000 7f4170ff9640 10 client.984258.objecter ms_handle_connect 0x7f417409e740 2024-09-16T04:05:15.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.726+0000 7f4170ff9640 10 client.984258.objecter resend_mon_ops 2024-09-16T04:05:15.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.726+0000 7f4170ff9640 1 -- 172.21.15.42:0/1972894322 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f416801e070 con 0x7f417409e740 2024-09-16T04:05:15.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.726+0000 7f4170ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:05:15.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.726+0000 7f4170ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:05:15.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.726+0000 7f4170ff9640 10 monclient: dump: 2024-09-16T04:05:15.732 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:05:15.732 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:05:15.732 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:15.732 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:15.732 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:05:15.732 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:05:15.732 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:05:15.732 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:05:15.733 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:05:15.733 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:05:15.733 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.726+0000 7f4170ff9640 1 -- 172.21.15.42:0/1972894322 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f4168013b60 con 0x7f417409e740 2024-09-16T04:05:15.733 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.726+0000 7f4170ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:05:15.733 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.726+0000 7f4170ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:05:15.733 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.726+0000 7f4170ff9640 1 -- 172.21.15.42:0/1972894322 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f417409e740 msgr2=0x7f417409eb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:05:15.733 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.726+0000 7f4170ff9640 1 --2- 172.21.15.42:0/1972894322 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f417409e740 0x7f417409eb20 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f41740a00d0 tx=0x7f4168017e10 comp rx=0 tx=0).stop 2024-09-16T04:05:15.733 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.726+0000 7f4170ff9640 1 -- 172.21.15.42:0/1972894322 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f417409e740 msgr2=0x7f417409eb20 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:05:15.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.726+0000 7f4170ff9640 1 --2- 172.21.15.42:0/1972894322 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f417409e740 0x7f417409eb20 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:15.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.734+0000 7f417a496640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:05:15.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.734+0000 7f417a496640 1 --2- 172.21.15.42:0/1972894322 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f417409f060 0x7f417412eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:05:15.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.734+0000 7f417a496640 10 monclient(hunting): start opening mon connection 2024-09-16T04:05:15.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.734+0000 7f417a496640 1 -- 172.21.15.42:0/1972894322 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f41741cf8d0 con 0x7f417409f060 2024-09-16T04:05:15.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.734+0000 7f41737fe640 1 --2- 172.21.15.42:0/1972894322 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f417409f060 0x7f417412eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:05:15.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.734+0000 7f41737fe640 10 monclient: get_auth_request con 0x7f417409f060 auth_method 0 2024-09-16T04:05:15.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.734+0000 7f41737fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:05:15.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.734+0000 7f41737fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:05:15.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.734+0000 7f41737fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:05:15.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.734+0000 7f41737fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:05:15.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.734+0000 7f41737fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:05:15.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.734+0000 7f41737fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:05:15.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.734+0000 7f41737fe640 10 monclient(hunting): handle_auth_done global_id 994103 payload 824 2024-09-16T04:05:15.742 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.734+0000 7f41737fe640 1 -- 172.21.15.42:0/1972894322 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f417406b3e0 con 0x7f417409f060 2024-09-16T04:05:15.742 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.734+0000 7f4170ff9640 10 client.984258.objecter ms_handle_connect 0x7f417409f060 2024-09-16T04:05:15.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.734+0000 7f41737fe640 1 --2- 172.21.15.42:0/1972894322 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f417409f060 0x7f417412eb50 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f4168023000 tx=0x7f41680046d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:05:15.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.734+0000 7f4170ff9640 10 client.984258.objecter resend_mon_ops 2024-09-16T04:05:15.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.738+0000 7f4170ff9640 1 -- 172.21.15.42:0/1972894322 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4168029a80 con 0x7f417409f060 2024-09-16T04:05:15.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.738+0000 7f4170ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:05:15.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.738+0000 7f4170ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:05:15.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.738+0000 7f4170ff9640 10 monclient: dump: 2024-09-16T04:05:15.743 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:05:15.743 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:05:15.743 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:15.743 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:15.743 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:05:15.743 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:05:15.743 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:05:15.744 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:05:15.744 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:05:15.744 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:05:15.744 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.738+0000 7f4170ff9640 1 -- 172.21.15.42:0/1972894322 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4545 (secure 0 0 0) 0x7f41680292b0 con 0x7f417409f060 2024-09-16T04:05:15.744 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.738+0000 7f4170ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:05:15.744 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.738+0000 7f4170ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:05:15.744 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.738+0000 7f4170ff9640 1 -- 172.21.15.42:0/1972894322 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f417409f060 msgr2=0x7f417412eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:05:15.744 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.738+0000 7f4170ff9640 1 --2- 172.21.15.42:0/1972894322 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f417409f060 0x7f417412eb50 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f4168023000 tx=0x7f41680046d0 comp rx=0 tx=0).stop 2024-09-16T04:05:15.744 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.738+0000 7f4170ff9640 1 -- 172.21.15.42:0/1972894322 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f417409f060 msgr2=0x7f417412eb50 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:05:15.744 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.738+0000 7f4170ff9640 1 --2- 172.21.15.42:0/1972894322 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f417409f060 0x7f417412eb50 secure :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f4168023000 tx=0x7f41680046d0 comp rx=0 tx=0).stop 2024-09-16T04:05:15.745 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T04:05:15.745 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:05:15.745 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:05:15.745 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T04:05:15.745 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 216, 2024-09-16T04:05:15.745 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 10427, 2024-09-16T04:05:15.745 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T04:05:15.745 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T04:05:15.745 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T04:05:15.746 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T04:05:15.746 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:05:15.746 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 5, 2024-09-16T04:05:15.746 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:05:15.746 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T04:05:15.746 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T04:05:15.746 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:05:15.746 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:05:15.746 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:05:15.746 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:05:15.746 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:05:15.746 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:05:15.746 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:05:15.746 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:05:15.746 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:05:15.747 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:05:15.747 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:05:15.747 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:05:15.747 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:05:15.747 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T04:05:15.747 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:05:15.747 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:05:15.747 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:05:15.747 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:05:15.747 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:05:15.747 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:05:15.747 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:05:15.747 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:05:15.747 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:05:15.748 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:05:15.748 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:05:15.748 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:05:15.748 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:05:15.748 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T04:05:15.748 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T04:05:15.748 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T04:05:15.748 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T04:05:15.748 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T04:05:15.748 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T04:05:15.749 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:05:15.749 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:05:15.749 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T04:05:15.749 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T04:05:15.749 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T04:05:15.749 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T04:05:15.749 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T04:05:15.749 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T04:05:15.749 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T04:05:15.750 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:05:15.750 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T04:05:15.750 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:05:15.750 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:05:15.750 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:05:15.750 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:05:15.750 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:05:15.750 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:05:15.750 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:05:15.750 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:05:15.750 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:05:15.750 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:05:15.750 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:05:15.750 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:05:15.751 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T04:05:15.751 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:05:15.751 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T04:05:15.751 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:05:15.751 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:05:15.751 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:05:15.751 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:05:15.751 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:05:15.751 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:05:15.751 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:05:15.752 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T04:05:15.752 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:05:15.752 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:05:15.752 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:05:15.752 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:05:15.752 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T04:05:15.752 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:05:15.753 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:05:15.753 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:05:15.753 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:05:15.753 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T04:05:15.753 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T04:05:15.753 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:05:15.753 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:05:15.753 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:05:15.753 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:05:15.753 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:05:15.753 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:05:15.753 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:05:15.753 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:05:15.754 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:05:15.754 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:05:15.754 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:05:15.754 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T04:05:15.754 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:05:15.754 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:05:15.754 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:05:15.754 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:05:15.754 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T04:05:15.754 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:05:15.754 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:05:15.754 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:05:15.754 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:05:15.755 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T04:05:15.755 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T04:05:15.755 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:05:15.755 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:05:15.755 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:05:15.755 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:05:15.755 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:05:15.756 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:05:15.756 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:05:15.756 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:05:15.756 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:05:15.756 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:05:15.757 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:05:15.757 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T04:05:15.757 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:05:15.757 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:05:15.757 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:05:15.758 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:05:15.758 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T04:05:15.758 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:05:15.758 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:05:15.758 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:05:15.758 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:05:15.759 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T04:05:15.759 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T04:05:15.759 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:05:15.759 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:05:15.759 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:05:15.760 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:05:15.760 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:05:15.760 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:05:15.760 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T04:05:15.760 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T04:05:15.761 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:05:15.761 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:05:15.761 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:05:15.761 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:05:15.761 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:05:15.762 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:05:15.762 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T04:05:15.762 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:05:15.762 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:05:15.762 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:05:15.762 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:05:15.763 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:05:15.763 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:05:15.763 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:05:15.763 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T04:05:15.763 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T04:05:15.766 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.738+0000 7f417a496640 10 librados: watch_flush enter 2024-09-16T04:05:15.767 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.738+0000 7f417a496640 10 librados: watch_flush exit 2024-09-16T04:05:15.767 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.738+0000 7f417a496640 20 client.984258.objecter shutdown clearing up homeless session... 2024-09-16T04:05:15.767 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.738+0000 7f417a496640 10 client.984258.objecter successfully canceled tick 2024-09-16T04:05:15.767 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.738+0000 7f417a496640 1 -- 172.21.15.42:0/1972894322 >> v2:172.21.15.154:6812/254845342 conn(0x7f4160037810 msgr2=0x7f4160039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:05:15.767 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.738+0000 7f417a496640 1 --2- 172.21.15.42:0/1972894322 >> v2:172.21.15.154:6812/254845342 conn(0x7f4160037810 0x7f4160039cd0 secure :-1 s=READY pgs=3841 cs=0 l=1 rev1=1 crypto rx=0x7f416c0ae000 tx=0x7f416c004ac0 comp rx=0 tx=0).stop 2024-09-16T04:05:15.767 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.738+0000 7f417a496640 10 monclient: shutdown 2024-09-16T04:05:15.767 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.738+0000 7f417a496640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:05:15.767 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.738+0000 7f417a496640 1 -- 172.21.15.42:0/1972894322 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f41740a2030 msgr2=0x7f41741cf390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:05:15.767 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.738+0000 7f417a496640 1 --2- 172.21.15.42:0/1972894322 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f41740a2030 0x7f41741cf390 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f416400e810 tx=0x7f416400ecd0 comp rx=0 tx=0).stop 2024-09-16T04:05:15.767 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.738+0000 7f4178a33640 1 -- 172.21.15.42:0/1972894322 reap_dead start 2024-09-16T04:05:15.767 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.738+0000 7f4178a33640 1 -- 172.21.15.42:0/1972894322 reap_dead start 2024-09-16T04:05:15.768 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.738+0000 7f417a496640 1 -- 172.21.15.42:0/1972894322 shutdown_connections 2024-09-16T04:05:15.768 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.738+0000 7f417a496640 1 -- 172.21.15.42:0/1972894322 >> 172.21.15.42:0/1972894322 conn(0x7f41740a2ad0 msgr2=0x7f4174132640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:05:15.768 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.738+0000 7f417a496640 1 -- 172.21.15.42:0/1972894322 shutdown_connections 2024-09-16T04:05:15.768 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.738+0000 7f417a496640 1 -- 172.21.15.42:0/1972894322 wait complete. 2024-09-16T04:05:15.768 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.738+0000 7f417a496640 1 librados: shutdown 2024-09-16T04:05:15.768 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:05:15.768 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T04:05:15.769 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T04:05:15.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.894+0000 7f1346633640 10 monclient: get_monmap_and_config 2024-09-16T04:05:15.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.894+0000 7f1346633640 10 monclient: build_initial_monmap 2024-09-16T04:05:15.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.894+0000 7f1346633640 10 monclient: monmap: 2024-09-16T04:05:15.898 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:05:15.899 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:05:15.899 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:05:15.896932+0000 2024-09-16T04:05:15.899 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:05:15.896932+0000 2024-09-16T04:05:15.899 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:05:15.899 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:05:15.899 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:05:15.899 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:05:15.899 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:05:15.900 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:05:15.900 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:05:15.900 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:05:15.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.894+0000 7f1346633640 1 Processor -- start 2024-09-16T04:05:15.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.894+0000 7f1346633640 1 -- start start 2024-09-16T04:05:15.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.894+0000 7f1346633640 10 monclient: init 2024-09-16T04:05:15.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.894+0000 7f1346633640 10 monclient: _reopen_session rank -1 2024-09-16T04:05:15.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.894+0000 7f1346633640 10 monclient: _add_conns ranks=[3,4,0,1,2] 2024-09-16T04:05:15.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.894+0000 7f1346633640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f134006db90 0x7f13401340c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:05:15.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.894+0000 7f1346633640 10 monclient(hunting): picked mon.noname-a con 0x7f134006db90 addr v2:172.21.15.42:3300/0 2024-09-16T04:05:15.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.894+0000 7f1346633640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f13400a37f0 0x7f13400a3c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:05:15.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.894+0000 7f1346633640 10 monclient(hunting): picked mon.noname-c con 0x7f13400a37f0 addr v2:172.21.15.154:3300/0 2024-09-16T04:05:15.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.894+0000 7f1346633640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1340136e10 addr v1:172.21.15.42:6789/0 2024-09-16T04:05:15.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.894+0000 7f1346633640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f134006ece0 con 0x7f1340136e10 2024-09-16T04:05:15.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.894+0000 7f1346633640 10 monclient(hunting): start opening mon connection 2024-09-16T04:05:15.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.894+0000 7f1346633640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f134006df70 con 0x7f134006db90 2024-09-16T04:05:15.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.894+0000 7f1346633640 10 monclient(hunting): start opening mon connection 2024-09-16T04:05:15.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.894+0000 7f1346633640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f134009bf80 con 0x7f13400a37f0 2024-09-16T04:05:15.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.894+0000 7f1346633640 10 monclient(hunting): _renew_subs 2024-09-16T04:05:15.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.894+0000 7f1346633640 10 monclient(hunting): authenticate will time out at 2362.867112s 2024-09-16T04:05:15.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.894+0000 7f133ffff640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f134006db90 0x7f13401340c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:05:15.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.894+0000 7f1344bd0640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f1340136e10 0x7f1340137220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:40576/0 (socket says 172.21.15.42:40576) 2024-09-16T04:05:15.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.894+0000 7f1344bd0640 1 -- 172.21.15.42:0/2580518356 learned_addr learned my addr 172.21.15.42:0/2580518356 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T04:05:15.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.894+0000 7f133ffff640 10 monclient(hunting): get_auth_request con 0x7f134006db90 auth_method 0 2024-09-16T04:05:15.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.894+0000 7f133ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:05:15.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.894+0000 7f133ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:05:15.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.894+0000 7f133ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:05:15.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.894+0000 7f133ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:05:15.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.894+0000 7f133ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:05:15.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.894+0000 7f133ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:05:15.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.894+0000 7f1337fff640 1 --2- 172.21.15.42:0/2580518356 >> v2:172.21.15.154:3300/0 conn(0x7f13400a37f0 0x7f13400a3c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:05:15.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.894+0000 7f1337fff640 10 monclient(hunting): get_auth_request con 0x7f13400a37f0 auth_method 0 2024-09-16T04:05:15.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.894+0000 7f1337fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:05:15.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.894+0000 7f1337fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:05:15.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.894+0000 7f1337fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:05:15.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.894+0000 7f133ffff640 10 monclient(hunting): handle_auth_done global_id 984261 payload 274 2024-09-16T04:05:15.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.894+0000 7f133ffff640 1 -- 172.21.15.42:0/2580518356 >> v2:172.21.15.154:3300/0 conn(0x7f13400a37f0 msgr2=0x7f13400a3c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:05:15.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.894+0000 7f133ffff640 1 --2- 172.21.15.42:0/2580518356 >> v2:172.21.15.154:3300/0 conn(0x7f13400a37f0 0x7f13400a3c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:15.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.894+0000 7f133ffff640 1 -- 172.21.15.42:0/2580518356 >> v1:172.21.15.42:6789/0 conn(0x7f1340136e10 legacy=0x7f1340137220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:05:15.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.894+0000 7f133ffff640 10 monclient: _finish_hunting 0 2024-09-16T04:05:15.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.894+0000 7f133ffff640 1 monclient: found mon.noname-a 2024-09-16T04:05:15.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.894+0000 7f133ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:05:15.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.894+0000 7f133f7fe640 1 -- 172.21.15.42:0/2580518356 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3765094906 0 0) 0x7f134006ece0 con 0x7f1340136e10 2024-09-16T04:05:15.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.894+0000 7f133ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:05:15.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.894+0000 7f133ffff640 1 -- 172.21.15.42:0/2580518356 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1340134780 con 0x7f134006db90 2024-09-16T04:05:15.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.894+0000 7f133f7fe640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T04:05:15.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.898+0000 7f133ffff640 1 --2- 172.21.15.42:0/2580518356 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f134006db90 0x7f13401340c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f134006ce60 tx=0x7f133001c340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf08fc44f1b39fcd server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:05:15.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.898+0000 7f133f7fe640 1 -- 172.21.15.42:0/2580518356 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13300b0070 con 0x7f134006db90 2024-09-16T04:05:15.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.898+0000 7f133f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:05:15.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.898+0000 7f133f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:05:15.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.898+0000 7f133f7fe640 10 monclient: dump: 2024-09-16T04:05:15.908 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:05:15.908 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:05:15.908 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:15.908 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:15.908 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:05:15.908 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:05:15.909 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:05:15.909 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:05:15.909 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:05:15.909 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:05:15.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.898+0000 7f133f7fe640 10 monclient: _finish_auth 0 2024-09-16T04:05:15.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.898+0000 7f133f7fe640 10 monclient: _check_auth_tickets 2024-09-16T04:05:15.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.898+0000 7f133f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:05:15.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.898+0000 7f1346633640 5 monclient: authenticate success, global_id 984261 2024-09-16T04:05:15.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.898+0000 7f1346633640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:05:15.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.898+0000 7f133f7fe640 1 -- 172.21.15.42:0/2580518356 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f13300ab600 con 0x7f134006db90 2024-09-16T04:05:15.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.898+0000 7f133f7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:05:15.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.898+0000 7f133f7fe640 1 -- 172.21.15.42:0/2580518356 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13300ab9d0 con 0x7f134006db90 2024-09-16T04:05:15.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.898+0000 7f133f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:05:15.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.898+0000 7f133f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:05:15.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.898+0000 7f133f7fe640 10 monclient: dump: 2024-09-16T04:05:15.910 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:05:15.910 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:05:15.910 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:15.910 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:15.910 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:05:15.910 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:05:15.911 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:05:15.911 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:05:15.911 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:05:15.911 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:05:15.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.898+0000 7f1346633640 10 monclient: get_monmap_and_config success 2024-09-16T04:05:15.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.898+0000 7f1346633640 4 set_mon_vals no callback set 2024-09-16T04:05:15.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.898+0000 7f1346633640 10 monclient: shutdown 2024-09-16T04:05:15.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.898+0000 7f1346633640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:05:15.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.898+0000 7f1346633640 1 -- 172.21.15.42:0/2580518356 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f134006db90 msgr2=0x7f13401340c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:05:15.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.898+0000 7f1346633640 1 --2- 172.21.15.42:0/2580518356 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f134006db90 0x7f13401340c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f134006ce60 tx=0x7f133001c340 comp rx=0 tx=0).stop 2024-09-16T04:05:15.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.898+0000 7f1346633640 1 -- 172.21.15.42:0/2580518356 shutdown_connections 2024-09-16T04:05:15.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.898+0000 7f1346633640 1 --2- 172.21.15.42:0/2580518356 >> v2:172.21.15.154:3300/0 conn(0x7f13400a37f0 0x7f13400a3c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:15.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.898+0000 7f1346633640 1 --2- 172.21.15.42:0/2580518356 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f134006db90 0x7f13401340c0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:15.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.898+0000 7f1346633640 1 -- 172.21.15.42:0/2580518356 >> 172.21.15.42:0/2580518356 conn(0x7f13400a9fb0 msgr2=0x7f13400aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:05:15.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.898+0000 7f1346633640 1 -- 172.21.15.42:0/2580518356 shutdown_connections 2024-09-16T04:05:15.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.898+0000 7f1346633640 1 -- 172.21.15.42:0/2580518356 wait complete. 2024-09-16T04:05:15.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.898+0000 7f1346633640 10 monclient: build_initial_monmap 2024-09-16T04:05:15.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.898+0000 7f1346633640 10 monclient: monmap: 2024-09-16T04:05:15.919 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:05:15.919 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:05:15.919 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:05:15.919 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:05:15.919 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:05:15.919 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:05:15.919 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:05:15.919 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:05:15.920 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:05:15.920 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:05:15.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.898+0000 7f1346633640 1 librados: starting msgr at 2024-09-16T04:05:15.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.898+0000 7f1346633640 1 librados: starting objecter 2024-09-16T04:05:15.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.902+0000 7f1346633640 1 Processor -- start 2024-09-16T04:05:15.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.902+0000 7f1346633640 1 -- start start 2024-09-16T04:05:15.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.902+0000 7f1346633640 1 librados: setting wanted keys 2024-09-16T04:05:15.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.902+0000 7f1346633640 1 librados: calling monclient init 2024-09-16T04:05:15.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.902+0000 7f1346633640 10 monclient: init 2024-09-16T04:05:15.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.902+0000 7f1346633640 10 monclient: _reopen_session rank -1 2024-09-16T04:05:15.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.902+0000 7f1346633640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:05:15.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.902+0000 7f1346633640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f134006db90 0x7f1340130a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:05:15.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.902+0000 7f1346633640 10 monclient(hunting): picked mon.noname-c con 0x7f134006db90 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:05:15.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.902+0000 7f1346633640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f13400a37f0 0x7f134013cae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:05:15.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.902+0000 7f1346633640 10 monclient(hunting): picked mon.noname-b con 0x7f13400a37f0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:05:15.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.902+0000 7f1346633640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1340136e10 0x7f134013d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:05:15.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.902+0000 7f1346633640 10 monclient(hunting): picked mon.noname-a con 0x7f1340136e10 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:05:15.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.902+0000 7f1346633640 10 monclient(hunting): start opening mon connection 2024-09-16T04:05:15.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.902+0000 7f1346633640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f1340141d60 con 0x7f1340136e10 2024-09-16T04:05:15.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.902+0000 7f1346633640 10 monclient(hunting): start opening mon connection 2024-09-16T04:05:15.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.902+0000 7f1346633640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f1340141be0 con 0x7f13400a37f0 2024-09-16T04:05:15.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.902+0000 7f1346633640 10 monclient(hunting): start opening mon connection 2024-09-16T04:05:15.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.902+0000 7f1346633640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f1340141ee0 con 0x7f134006db90 2024-09-16T04:05:15.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.902+0000 7f1346633640 10 monclient(hunting): _renew_subs 2024-09-16T04:05:15.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.902+0000 7f1346633640 10 monclient(hunting): authenticate will time out at 2362.872655s 2024-09-16T04:05:15.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.902+0000 7f133ffff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f134006db90 0x7f1340130a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:05:15.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.902+0000 7f133ffff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f134006db90 0x7f1340130a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:39816/0 (socket says 172.21.15.42:39816) 2024-09-16T04:05:15.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.902+0000 7f133ffff640 1 -- 172.21.15.42:0/3181491931 learned_addr learned my addr 172.21.15.42:0/3181491931 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:05:15.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.902+0000 7f133ffff640 10 monclient(hunting): get_auth_request con 0x7f134006db90 auth_method 0 2024-09-16T04:05:15.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.902+0000 7f133ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:05:15.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.902+0000 7f133ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:05:15.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.902+0000 7f133ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:05:15.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.902+0000 7f1344bd0640 1 --2- 172.21.15.42:0/3181491931 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1340136e10 0x7f134013d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:05:15.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.902+0000 7f1344bd0640 10 monclient(hunting): get_auth_request con 0x7f1340136e10 auth_method 0 2024-09-16T04:05:15.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.902+0000 7f1344bd0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:05:15.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.902+0000 7f1344bd0640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:05:15.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.902+0000 7f1344bd0640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:05:15.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.902+0000 7f133ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:05:15.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.902+0000 7f133ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:05:15.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.902+0000 7f133ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:05:15.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.902+0000 7f1344bd0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:05:15.923 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.902+0000 7f1344bd0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:05:15.923 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.902+0000 7f1344bd0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:05:15.923 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.902+0000 7f1337fff640 1 --2- 172.21.15.42:0/3181491931 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f13400a37f0 0x7f134013cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:05:15.923 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.902+0000 7f1337fff640 10 monclient(hunting): get_auth_request con 0x7f13400a37f0 auth_method 0 2024-09-16T04:05:15.923 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.902+0000 7f1337fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:05:15.923 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.902+0000 7f1337fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:05:15.923 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.902+0000 7f1337fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:05:15.923 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.902+0000 7f133ffff640 10 monclient(hunting): handle_auth_done global_id 994106 payload 824 2024-09-16T04:05:15.923 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.902+0000 7f133ffff640 1 -- 172.21.15.42:0/3181491931 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f13400a37f0 msgr2=0x7f134013cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:05:15.923 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.902+0000 7f133ffff640 1 --2- 172.21.15.42:0/3181491931 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f13400a37f0 0x7f134013cae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:15.923 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.902+0000 7f133ffff640 1 -- 172.21.15.42:0/3181491931 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1340136e10 msgr2=0x7f134013d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:05:15.923 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.902+0000 7f133ffff640 1 --2- 172.21.15.42:0/3181491931 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1340136e10 0x7f134013d020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:15.923 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.902+0000 7f133ffff640 10 monclient: _finish_hunting 0 2024-09-16T04:05:15.923 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.902+0000 7f133ffff640 1 monclient: found mon.noname-c 2024-09-16T04:05:15.924 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.902+0000 7f133ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:05:15.924 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.902+0000 7f133ffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T04:05:15.924 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.902+0000 7f133ffff640 1 -- 172.21.15.42:0/3181491931 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f134013e730 con 0x7f134006db90 2024-09-16T04:05:15.924 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.902+0000 7f133ffff640 1 --2- 172.21.15.42:0/3181491931 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f134006db90 0x7f1340130a20 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f13300ae040 tx=0x7f133001ab80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:05:15.924 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.902+0000 7f133d7fa640 10 client.?.objecter ms_handle_connect 0x7f134006db90 2024-09-16T04:05:15.924 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.902+0000 7f133d7fa640 10 client.?.objecter resend_mon_ops 2024-09-16T04:05:15.924 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.902+0000 7f133d7fa640 1 -- 172.21.15.42:0/3181491931 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13300bc070 con 0x7f134006db90 2024-09-16T04:05:15.924 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.902+0000 7f133d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:05:15.924 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.902+0000 7f133d7fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:05:15.924 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.902+0000 7f133d7fa640 10 monclient: dump: 2024-09-16T04:05:15.924 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:05:15.924 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:05:15.925 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:15.925 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:15.925 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:05:15.925 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:05:15.925 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:05:15.925 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:05:15.925 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:05:15.925 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:05:15.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.902+0000 7f133d7fa640 10 monclient: _finish_auth 0 2024-09-16T04:05:15.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.902+0000 7f133d7fa640 10 monclient: _check_auth_tickets 2024-09-16T04:05:15.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.902+0000 7f133d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:05:15.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.902+0000 7f1346633640 5 monclient: authenticate success, global_id 994106 2024-09-16T04:05:15.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.902+0000 7f1346633640 10 monclient: _renew_subs 2024-09-16T04:05:15.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.902+0000 7f133d7fa640 1 -- 172.21.15.42:0/3181491931 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f13300b1040 con 0x7f134006db90 2024-09-16T04:05:15.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.902+0000 7f1346633640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:05:15.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.902+0000 7f1346633640 1 -- 172.21.15.42:0/3181491931 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f134009e320 con 0x7f134006db90 2024-09-16T04:05:15.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.902+0000 7f133d7fa640 10 monclient: handle_config config(0 keys) 2024-09-16T04:05:15.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.902+0000 7f133d7fa640 1 -- 172.21.15.42:0/3181491931 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13300b0050 con 0x7f134006db90 2024-09-16T04:05:15.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.902+0000 7f133d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:05:15.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.902+0000 7f133d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:05:15.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.902+0000 7f1346633640 10 client.994106.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:05:15.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.902+0000 7f133d7fa640 10 monclient: dump: 2024-09-16T04:05:15.926 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:05:15.926 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:05:15.926 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:15.926 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:15.926 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:05:15.927 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:05:15.927 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:05:15.927 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:05:15.927 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:05:15.927 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:05:15.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.902+0000 7f133effd640 4 set_mon_vals no callback set 2024-09-16T04:05:15.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.902+0000 7f1346633640 10 monclient: _renew_subs 2024-09-16T04:05:15.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.902+0000 7f1346633640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:05:15.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.902+0000 7f1346633640 1 -- 172.21.15.42:0/3181491931 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f1340133360 con 0x7f134006db90 2024-09-16T04:05:15.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.902+0000 7f1346633640 1 librados: init done 2024-09-16T04:05:15.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.902+0000 7f1346633640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:05:15.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.902+0000 7f1346633640 1 --2- 172.21.15.42:0/3181491931 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1340130f60 0x7f134002c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:05:15.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.902+0000 7f1346633640 10 monclient(hunting): start opening mon connection 2024-09-16T04:05:15.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.902+0000 7f1346633640 1 -- 172.21.15.42:0/3181491931 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f13401340c0 con 0x7f1340130f60 2024-09-16T04:05:15.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.902+0000 7f133d7fa640 1 -- 172.21.15.42:0/3181491931 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f13300bf070 con 0x7f134006db90 2024-09-16T04:05:15.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.902+0000 7f133d7fa640 1 --2- 172.21.15.42:0/3181491931 >> v2:172.21.15.154:6812/254845342 conn(0x7f1320037810 0x7f1320039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:05:15.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.902+0000 7f133d7fa640 1 -- 172.21.15.42:0/3181491931 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(703..703 src has 451..703) ==== 5899+0+0 (secure 0 0 0) 0x7f13300e2b60 con 0x7f134006db90 2024-09-16T04:05:15.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.902+0000 7f133d7fa640 10 client.994106.objecter ms_dispatch 0x7f1340000ba0 osd_map(703..703 src has 451..703) 2024-09-16T04:05:15.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.902+0000 7f133d7fa640 3 client.994106.objecter handle_osd_map got epochs [703,703] > 0 2024-09-16T04:05:15.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.902+0000 7f133d7fa640 3 client.994106.objecter handle_osd_map decoding full epoch 703 2024-09-16T04:05:15.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.902+0000 7f1337fff640 1 --2- 172.21.15.42:0/3181491931 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1340130f60 0x7f134002c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:05:15.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.902+0000 7f133d7fa640 20 client.994106.objecter dump_active .. 0 homeless 2024-09-16T04:05:15.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.902+0000 7f1337fff640 10 monclient: get_auth_request con 0x7f1340130f60 auth_method 0 2024-09-16T04:05:15.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.902+0000 7f1337fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:05:15.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.902+0000 7f1337fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:05:15.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.902+0000 7f1337fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:05:15.929 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.902+0000 7f1344bd0640 1 --2- 172.21.15.42:0/3181491931 >> v2:172.21.15.154:6812/254845342 conn(0x7f1320037810 0x7f1320039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:05:15.929 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.902+0000 7f1337fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:05:15.929 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.902+0000 7f1337fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:05:15.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.902+0000 7f1337fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:05:15.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.902+0000 7f1344bd0640 10 monclient: get_auth_request con 0x7f1320037810 auth_method 0 2024-09-16T04:05:15.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.902+0000 7f1337fff640 10 monclient(hunting): handle_auth_done global_id 904876 payload 824 2024-09-16T04:05:15.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.902+0000 7f1337fff640 1 -- 172.21.15.42:0/3181491931 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f13401394e0 con 0x7f1340130f60 2024-09-16T04:05:15.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.902+0000 7f133d7fa640 10 client.994106.objecter ms_handle_connect 0x7f1340130f60 2024-09-16T04:05:15.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.902+0000 7f133d7fa640 10 client.994106.objecter resend_mon_ops 2024-09-16T04:05:15.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.902+0000 7f1337fff640 1 --2- 172.21.15.42:0/3181491931 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1340130f60 0x7f134002c650 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f134013e210 tx=0x7f132800f910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:05:15.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.902+0000 7f133d7fa640 10 client.994106.objecter ms_handle_connect 0x7f1320037810 2024-09-16T04:05:15.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.902+0000 7f1344bd0640 1 --2- 172.21.15.42:0/3181491931 >> v2:172.21.15.154:6812/254845342 conn(0x7f1320037810 0x7f1320039cd0 secure :-1 s=READY pgs=3842 cs=0 l=1 rev1=1 crypto rx=0x7f1340131760 tx=0x7f13380016c0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:05:15.933 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.906+0000 7f133d7fa640 1 -- 172.21.15.42:0/3181491931 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1328019020 con 0x7f1340130f60 2024-09-16T04:05:15.933 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.906+0000 7f133d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:05:15.933 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.906+0000 7f133d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:05:15.933 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.906+0000 7f133d7fa640 10 monclient: dump: 2024-09-16T04:05:15.933 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:05:15.933 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:05:15.933 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:15.933 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:15.933 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:05:15.933 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:05:15.933 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:05:15.933 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:05:15.933 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:05:15.934 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:05:15.934 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.906+0000 7f133d7fa640 1 -- 172.21.15.42:0/3181491931 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f1328014870 con 0x7f1340130f60 2024-09-16T04:05:15.934 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.906+0000 7f133d7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:05:15.934 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.906+0000 7f133d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:05:15.934 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.906+0000 7f133d7fa640 1 -- 172.21.15.42:0/3181491931 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1340130f60 msgr2=0x7f134002c650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:05:15.934 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.906+0000 7f133d7fa640 1 --2- 172.21.15.42:0/3181491931 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1340130f60 0x7f134002c650 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f134013e210 tx=0x7f132800f910 comp rx=0 tx=0).stop 2024-09-16T04:05:15.934 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.906+0000 7f133d7fa640 1 -- 172.21.15.42:0/3181491931 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1340130f60 msgr2=0x7f134002c650 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:05:15.934 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.906+0000 7f133d7fa640 1 --2- 172.21.15.42:0/3181491931 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1340130f60 0x7f134002c650 secure :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f134013e210 tx=0x7f132800f910 comp rx=0 tx=0).stop 2024-09-16T04:05:15.934 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.910+0000 7f1346633640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:05:15.934 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.910+0000 7f1346633640 1 --2- 172.21.15.42:0/3181491931 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f134002cb90 0x7f134002cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:05:15.934 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.910+0000 7f1346633640 10 monclient(hunting): start opening mon connection 2024-09-16T04:05:15.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.910+0000 7f1346633640 1 -- 172.21.15.42:0/3181491931 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f134009db30 con 0x7f134002cb90 2024-09-16T04:05:15.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.910+0000 7f1337fff640 1 --2- 172.21.15.42:0/3181491931 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f134002cb90 0x7f134002cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:05:15.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.910+0000 7f1337fff640 10 monclient: get_auth_request con 0x7f134002cb90 auth_method 0 2024-09-16T04:05:15.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.910+0000 7f1337fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:05:15.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.910+0000 7f1337fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:05:15.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.910+0000 7f1337fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:05:15.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.910+0000 7f1337fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:05:15.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.910+0000 7f1337fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:05:15.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.910+0000 7f1337fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:05:15.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.910+0000 7f1337fff640 10 monclient(hunting): handle_auth_done global_id 904879 payload 824 2024-09-16T04:05:15.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.910+0000 7f1337fff640 1 -- 172.21.15.42:0/3181491931 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f134013dd20 con 0x7f134002cb90 2024-09-16T04:05:15.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.914+0000 7f1337fff640 1 --2- 172.21.15.42:0/3181491931 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f134002cb90 0x7f134002cfa0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f132801a000 tx=0x7f13280046d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:05:15.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.914+0000 7f133d7fa640 10 client.994106.objecter ms_handle_connect 0x7f134002cb90 2024-09-16T04:05:15.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.914+0000 7f133d7fa640 10 client.994106.objecter resend_mon_ops 2024-09-16T04:05:15.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.914+0000 7f133d7fa640 1 -- 172.21.15.42:0/3181491931 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1328019070 con 0x7f134002cb90 2024-09-16T04:05:15.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.914+0000 7f133d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:05:15.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.914+0000 7f133d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:05:15.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.914+0000 7f133d7fa640 10 monclient: dump: 2024-09-16T04:05:15.936 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:05:15.936 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:05:15.936 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:15.936 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:15.936 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:05:15.937 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:05:15.937 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:05:15.937 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:05:15.937 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:05:15.937 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:05:15.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.914+0000 7f133d7fa640 1 -- 172.21.15.42:0/3181491931 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4880 (secure 0 0 0) 0x7f1328004220 con 0x7f134002cb90 2024-09-16T04:05:15.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.914+0000 7f133d7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:05:15.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.914+0000 7f133d7fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:05:15.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.914+0000 7f133d7fa640 1 -- 172.21.15.42:0/3181491931 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f134002cb90 msgr2=0x7f134002cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:05:15.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.914+0000 7f133d7fa640 1 --2- 172.21.15.42:0/3181491931 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f134002cb90 0x7f134002cfa0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f132801a000 tx=0x7f13280046d0 comp rx=0 tx=0).stop 2024-09-16T04:05:15.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.914+0000 7f133d7fa640 1 -- 172.21.15.42:0/3181491931 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f134002cb90 msgr2=0x7f134002cfa0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:05:15.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.914+0000 7f133d7fa640 1 --2- 172.21.15.42:0/3181491931 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f134002cb90 0x7f134002cfa0 secure :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f132801a000 tx=0x7f13280046d0 comp rx=0 tx=0).stop 2024-09-16T04:05:15.938 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T04:05:15.938 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:05:15.938 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:05:15.938 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T04:05:15.938 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 216, 2024-09-16T04:05:15.938 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 444093, 2024-09-16T04:05:15.938 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T04:05:15.938 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T04:05:15.938 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T04:05:15.939 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T04:05:15.939 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:05:15.939 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 5, 2024-09-16T04:05:15.939 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:05:15.939 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T04:05:15.939 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T04:05:15.939 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:05:15.939 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:05:15.939 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:05:15.939 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:05:15.939 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:05:15.939 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:05:15.939 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:05:15.940 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:05:15.940 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:05:15.940 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:05:15.940 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:05:15.940 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:05:15.940 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:05:15.940 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T04:05:15.940 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:05:15.940 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:05:15.940 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:05:15.940 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:05:15.940 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:05:15.940 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:05:15.941 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:05:15.941 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:05:15.941 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:05:15.941 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:05:15.941 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:05:15.941 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:05:15.941 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:05:15.941 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T04:05:15.941 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T04:05:15.941 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T04:05:15.941 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T04:05:15.941 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T04:05:15.942 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T04:05:15.942 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:05:15.942 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:05:15.942 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T04:05:15.942 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T04:05:15.942 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T04:05:15.942 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T04:05:15.942 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T04:05:15.942 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T04:05:15.942 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T04:05:15.942 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:05:15.942 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T04:05:15.942 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:05:15.943 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:05:15.943 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:05:15.943 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:05:15.943 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:05:15.943 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:05:15.943 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:05:15.943 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:05:15.943 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:05:15.943 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:05:15.943 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:05:15.943 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:05:15.943 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T04:05:15.943 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:05:15.944 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T04:05:15.944 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:05:15.944 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:05:15.944 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:05:15.944 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:05:15.944 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:05:15.944 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:05:15.944 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:05:15.944 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T04:05:15.944 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:05:15.944 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:05:15.944 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:05:15.944 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:05:15.944 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T04:05:15.945 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:05:15.945 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:05:15.945 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:05:15.945 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:05:15.945 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T04:05:15.945 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T04:05:15.945 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:05:15.945 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:05:15.945 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:05:15.945 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:05:15.945 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:05:15.945 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:05:15.945 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:05:15.946 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:05:15.946 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:05:15.946 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:05:15.946 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:05:15.946 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T04:05:15.946 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:05:15.946 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:05:15.946 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:05:15.946 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:05:15.946 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T04:05:15.946 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:05:15.946 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:05:15.946 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:05:15.946 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:05:15.946 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T04:05:15.947 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T04:05:15.947 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:05:15.947 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:05:15.947 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:05:15.947 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:05:15.947 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:05:15.947 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:05:15.947 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:05:15.947 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:05:15.947 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:05:15.947 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:05:15.947 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:05:15.947 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T04:05:15.947 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:05:15.948 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:05:15.948 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:05:15.948 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:05:15.948 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T04:05:15.948 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:05:15.948 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:05:15.948 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:05:15.948 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:05:15.948 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T04:05:15.948 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T04:05:15.948 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:05:15.948 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:05:15.948 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:05:15.949 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:05:15.949 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:05:15.949 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:05:15.949 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T04:05:15.949 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T04:05:15.949 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:05:15.949 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:05:15.949 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:05:15.949 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:05:15.949 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:05:15.949 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:05:15.949 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T04:05:15.950 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:05:15.950 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:05:15.950 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:05:15.950 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 3 2024-09-16T04:05:15.950 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:05:15.950 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:05:15.950 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T04:05:15.950 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:05:15.950 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:05:15.950 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:05:15.950 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 6 2024-09-16T04:05:15.950 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:05:15.950 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:05:15.950 INFO:teuthology.orchestra.run.smithi042.stdout: "mgr": [ 2024-09-16T04:05:15.950 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:05:15.951 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:05:15.951 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:05:15.951 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:05:15.951 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:05:15.951 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:05:15.951 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:05:15.951 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T04:05:15.951 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T04:05:15.953 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.914+0000 7f1346633640 10 librados: watch_flush enter 2024-09-16T04:05:15.953 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.914+0000 7f1346633640 10 librados: watch_flush exit 2024-09-16T04:05:15.953 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.914+0000 7f1346633640 20 client.994106.objecter shutdown clearing up homeless session... 2024-09-16T04:05:15.953 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.914+0000 7f1346633640 10 client.994106.objecter successfully canceled tick 2024-09-16T04:05:15.953 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.914+0000 7f1346633640 1 -- 172.21.15.42:0/3181491931 >> v2:172.21.15.154:6812/254845342 conn(0x7f1320037810 msgr2=0x7f1320039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:05:15.953 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.914+0000 7f1346633640 1 --2- 172.21.15.42:0/3181491931 >> v2:172.21.15.154:6812/254845342 conn(0x7f1320037810 0x7f1320039cd0 secure :-1 s=READY pgs=3842 cs=0 l=1 rev1=1 crypto rx=0x7f1340131760 tx=0x7f13380016c0 comp rx=0 tx=0).stop 2024-09-16T04:05:15.953 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.914+0000 7f1346633640 10 monclient: shutdown 2024-09-16T04:05:15.953 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.914+0000 7f1346633640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:05:15.953 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.914+0000 7f1346633640 1 -- 172.21.15.42:0/3181491931 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f134006db90 msgr2=0x7f1340130a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:05:15.954 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.914+0000 7f1346633640 1 --2- 172.21.15.42:0/3181491931 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f134006db90 0x7f1340130a20 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f13300ae040 tx=0x7f133001ab80 comp rx=0 tx=0).stop 2024-09-16T04:05:15.954 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.914+0000 7f1344bd0640 1 -- 172.21.15.42:0/3181491931 reap_dead start 2024-09-16T04:05:15.954 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.914+0000 7f1344bd0640 1 -- 172.21.15.42:0/3181491931 reap_dead start 2024-09-16T04:05:15.954 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.914+0000 7f1346633640 1 -- 172.21.15.42:0/3181491931 shutdown_connections 2024-09-16T04:05:15.954 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.914+0000 7f1346633640 1 -- 172.21.15.42:0/3181491931 >> 172.21.15.42:0/3181491931 conn(0x7f13400a9fb0 msgr2=0x7f1340132350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:05:15.954 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.914+0000 7f1346633640 1 -- 172.21.15.42:0/3181491931 shutdown_connections 2024-09-16T04:05:15.954 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.914+0000 7f1346633640 1 -- 172.21.15.42:0/3181491931 wait complete. 2024-09-16T04:05:15.954 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:15.914+0000 7f1346633640 1 librados: shutdown 2024-09-16T04:05:15.954 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:05:15.954 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T04:05:15.955 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T04:05:16.067 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.062+0000 7f36a2a17640 10 monclient: get_monmap_and_config 2024-09-16T04:05:16.068 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.062+0000 7f36a2a17640 10 monclient: build_initial_monmap 2024-09-16T04:05:16.068 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.062+0000 7f36a2a17640 10 monclient: monmap: 2024-09-16T04:05:16.068 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:05:16.068 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:05:16.068 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:05:16.066366+0000 2024-09-16T04:05:16.068 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:05:16.066366+0000 2024-09-16T04:05:16.068 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:05:16.068 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:05:16.069 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:05:16.069 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:05:16.069 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:05:16.069 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:05:16.069 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:05:16.069 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:05:16.071 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.062+0000 7f36a2a17640 1 Processor -- start 2024-09-16T04:05:16.071 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.062+0000 7f36a2a17640 1 -- start start 2024-09-16T04:05:16.071 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.062+0000 7f36a2a17640 10 monclient: init 2024-09-16T04:05:16.071 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.062+0000 7f36a2a17640 10 monclient: _reopen_session rank -1 2024-09-16T04:05:16.072 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.062+0000 7f36a2a17640 10 monclient: _add_conns ranks=[1,4,3,0,2] 2024-09-16T04:05:16.072 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.062+0000 7f36a2a17640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f369c06db90 addr v1:172.21.15.154:6789/0 2024-09-16T04:05:16.072 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.062+0000 7f36a2a17640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f369c1340c0 0x7f369c13c190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:05:16.072 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.062+0000 7f36a2a17640 10 monclient(hunting): picked mon.noname-c con 0x7f369c1340c0 addr v2:172.21.15.154:3300/0 2024-09-16T04:05:16.072 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.062+0000 7f36a2a17640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f369c13c6d0 0x7f369c13eac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:05:16.072 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.062+0000 7f36a2a17640 10 monclient(hunting): picked mon.noname-a con 0x7f369c13c6d0 addr v2:172.21.15.42:3300/0 2024-09-16T04:05:16.072 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.062+0000 7f36a2a17640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f369c06ece0 con 0x7f369c06db90 2024-09-16T04:05:16.072 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.062+0000 7f36a2a17640 10 monclient(hunting): start opening mon connection 2024-09-16T04:05:16.072 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.062+0000 7f36a2a17640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f369c06df70 con 0x7f369c13c6d0 2024-09-16T04:05:16.072 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.062+0000 7f36a2a17640 10 monclient(hunting): start opening mon connection 2024-09-16T04:05:16.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.062+0000 7f36a2a17640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f369c09bf80 con 0x7f369c1340c0 2024-09-16T04:05:16.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.062+0000 7f36a2a17640 10 monclient(hunting): _renew_subs 2024-09-16T04:05:16.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.062+0000 7f36a2a17640 10 monclient(hunting): authenticate will time out at 2363.035444s 2024-09-16T04:05:16.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.062+0000 7f36a0fb4640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f369c13c6d0 0x7f369c13eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:05:16.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.062+0000 7f36a0fb4640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f369c13c6d0 0x7f369c13eac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:47506/0 (socket says 172.21.15.42:47506) 2024-09-16T04:05:16.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.062+0000 7f36a0fb4640 1 -- 172.21.15.42:0/3846767364 learned_addr learned my addr 172.21.15.42:0/3846767364 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:05:16.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.062+0000 7f36a0fb4640 10 monclient(hunting): get_auth_request con 0x7f369c13c6d0 auth_method 0 2024-09-16T04:05:16.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.062+0000 7f36a0fb4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:05:16.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.062+0000 7f36a0fb4640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:05:16.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.062+0000 7f36a0fb4640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:05:16.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.062+0000 7f36a0fb4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:05:16.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.062+0000 7f36a0fb4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:05:16.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.062+0000 7f36a0fb4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:05:16.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.062+0000 7f369b7fe640 1 --2- 172.21.15.42:0/3846767364 >> v2:172.21.15.154:3300/0 conn(0x7f369c1340c0 0x7f369c13c190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:05:16.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.062+0000 7f369b7fe640 10 monclient(hunting): get_auth_request con 0x7f369c1340c0 auth_method 0 2024-09-16T04:05:16.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.062+0000 7f369b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:05:16.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.062+0000 7f369b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:05:16.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.062+0000 7f369b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:05:16.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.066+0000 7f36a0fb4640 10 monclient(hunting): handle_auth_done global_id 984270 payload 274 2024-09-16T04:05:16.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.066+0000 7f36a0fb4640 1 -- 172.21.15.42:0/3846767364 >> v2:172.21.15.154:3300/0 conn(0x7f369c1340c0 msgr2=0x7f369c13c190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:05:16.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.066+0000 7f36a0fb4640 1 --2- 172.21.15.42:0/3846767364 >> v2:172.21.15.154:3300/0 conn(0x7f369c1340c0 0x7f369c13c190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:16.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.066+0000 7f36a0fb4640 1 -- 172.21.15.42:0/3846767364 >> v1:172.21.15.154:6789/0 conn(0x7f369c06db90 legacy=0x7f369c0a37f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:05:16.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.066+0000 7f36a0fb4640 10 monclient: _finish_hunting 0 2024-09-16T04:05:16.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.066+0000 7f36a0fb4640 1 monclient: found mon.noname-a 2024-09-16T04:05:16.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.066+0000 7f36a0fb4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:05:16.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.066+0000 7f36a0fb4640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:05:16.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.066+0000 7f36a0fb4640 1 -- 172.21.15.42:0/3846767364 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f369c1344d0 con 0x7f369c13c6d0 2024-09-16T04:05:16.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.066+0000 7f369b7fe640 1 --2- 172.21.15.42:0/3846767364 >> v2:172.21.15.154:3300/0 conn(0x7f369c1340c0 0x7f369c13c190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T04:05:16.080 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.066+0000 7f36a0fb4640 1 --2- 172.21.15.42:0/3846767364 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f369c13c6d0 0x7f369c13eac0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f369c06ce60 tx=0x7f369401ce40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=605711ace7380c6d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:05:16.080 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.066+0000 7f369affd640 1 -- 172.21.15.42:0/3846767364 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36940b0070 con 0x7f369c13c6d0 2024-09-16T04:05:16.081 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.066+0000 7f369affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:05:16.081 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.066+0000 7f369affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:05:16.081 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.066+0000 7f369affd640 10 monclient: dump: 2024-09-16T04:05:16.081 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:05:16.081 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:05:16.081 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:16.081 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:16.081 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:05:16.081 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:05:16.081 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:05:16.081 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:05:16.081 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:05:16.082 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:05:16.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.066+0000 7f369affd640 10 monclient: _finish_auth 0 2024-09-16T04:05:16.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.066+0000 7f369affd640 10 monclient: _check_auth_tickets 2024-09-16T04:05:16.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.066+0000 7f369affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:05:16.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.066+0000 7f369affd640 1 -- 172.21.15.42:0/3846767364 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f36940ab600 con 0x7f369c13c6d0 2024-09-16T04:05:16.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.066+0000 7f369affd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:05:16.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.066+0000 7f369affd640 1 -- 172.21.15.42:0/3846767364 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36940ab9d0 con 0x7f369c13c6d0 2024-09-16T04:05:16.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.066+0000 7f36a2a17640 5 monclient: authenticate success, global_id 984270 2024-09-16T04:05:16.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.066+0000 7f36a2a17640 10 monclient: get_monmap_and_config success 2024-09-16T04:05:16.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.066+0000 7f36a2a17640 4 set_mon_vals no callback set 2024-09-16T04:05:16.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.066+0000 7f369affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:05:16.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.066+0000 7f369affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:05:16.083 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.066+0000 7f369affd640 10 monclient: dump: 2024-09-16T04:05:16.083 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:05:16.083 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:05:16.083 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:16.083 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:16.083 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:05:16.083 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:05:16.083 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:05:16.083 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:05:16.083 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:05:16.083 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:05:16.083 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.066+0000 7f36a2a17640 10 monclient: shutdown 2024-09-16T04:05:16.083 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.066+0000 7f36a2a17640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:05:16.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.066+0000 7f36a2a17640 1 -- 172.21.15.42:0/3846767364 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f369c13c6d0 msgr2=0x7f369c13eac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:05:16.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.066+0000 7f36a2a17640 1 --2- 172.21.15.42:0/3846767364 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f369c13c6d0 0x7f369c13eac0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f369c06ce60 tx=0x7f369401ce40 comp rx=0 tx=0).stop 2024-09-16T04:05:16.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.066+0000 7f36a2a17640 1 -- 172.21.15.42:0/3846767364 shutdown_connections 2024-09-16T04:05:16.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.066+0000 7f36a2a17640 1 --2- 172.21.15.42:0/3846767364 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f369c13c6d0 0x7f369c13eac0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:16.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.066+0000 7f36a2a17640 1 --2- 172.21.15.42:0/3846767364 >> v2:172.21.15.154:3300/0 conn(0x7f369c1340c0 0x7f369c13c190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:16.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.066+0000 7f36a2a17640 1 -- 172.21.15.42:0/3846767364 >> 172.21.15.42:0/3846767364 conn(0x7f369c0a9fb0 msgr2=0x7f369c0aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:05:16.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.066+0000 7f36a2a17640 1 -- 172.21.15.42:0/3846767364 shutdown_connections 2024-09-16T04:05:16.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.066+0000 7f36a2a17640 1 -- 172.21.15.42:0/3846767364 wait complete. 2024-09-16T04:05:16.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.070+0000 7f36a2a17640 10 monclient: build_initial_monmap 2024-09-16T04:05:16.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.070+0000 7f36a2a17640 10 monclient: monmap: 2024-09-16T04:05:16.084 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:05:16.084 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:05:16.085 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:05:16.085 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:05:16.085 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:05:16.085 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:05:16.085 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:05:16.085 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:05:16.085 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:05:16.085 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:05:16.085 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.070+0000 7f36a2a17640 1 librados: starting msgr at 2024-09-16T04:05:16.085 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.070+0000 7f36a2a17640 1 librados: starting objecter 2024-09-16T04:05:16.085 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.070+0000 7f36a2a17640 1 Processor -- start 2024-09-16T04:05:16.085 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.070+0000 7f36a2a17640 1 -- start start 2024-09-16T04:05:16.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.070+0000 7f36a2a17640 1 librados: setting wanted keys 2024-09-16T04:05:16.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.070+0000 7f36a2a17640 1 librados: calling monclient init 2024-09-16T04:05:16.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.070+0000 7f36a2a17640 10 monclient: init 2024-09-16T04:05:16.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.070+0000 7f36a2a17640 10 monclient: _reopen_session rank -1 2024-09-16T04:05:16.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.070+0000 7f36a2a17640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:05:16.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.070+0000 7f36a2a17640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f369c06db90 0x7f369c1ced50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:05:16.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.070+0000 7f36a2a17640 10 monclient(hunting): picked mon.noname-b con 0x7f369c06db90 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:05:16.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.070+0000 7f36a2a17640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f369c1340c0 0x7f369c0a2a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:05:16.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.070+0000 7f36a2a17640 10 monclient(hunting): picked mon.noname-a con 0x7f369c1340c0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:05:16.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.070+0000 7f36a2a17640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f369c13c6d0 0x7f369c0a2f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:05:16.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.070+0000 7f36a2a17640 10 monclient(hunting): picked mon.noname-c con 0x7f369c13c6d0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:05:16.087 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.070+0000 7f36a2a17640 10 monclient(hunting): start opening mon connection 2024-09-16T04:05:16.087 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.070+0000 7f36a2a17640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f369c1419a0 con 0x7f369c1340c0 2024-09-16T04:05:16.087 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.070+0000 7f36a2a17640 10 monclient(hunting): start opening mon connection 2024-09-16T04:05:16.087 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.070+0000 7f36a2a17640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f369c141820 con 0x7f369c06db90 2024-09-16T04:05:16.087 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.070+0000 7f36a2a17640 10 monclient(hunting): start opening mon connection 2024-09-16T04:05:16.087 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.070+0000 7f36a2a17640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f369c141b20 con 0x7f369c13c6d0 2024-09-16T04:05:16.087 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.070+0000 7f36a2a17640 10 monclient(hunting): _renew_subs 2024-09-16T04:05:16.087 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.070+0000 7f36a2a17640 10 monclient(hunting): authenticate will time out at 2363.041315s 2024-09-16T04:05:16.087 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.070+0000 7f369b7fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f369c1340c0 0x7f369c0a2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:05:16.087 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.070+0000 7f36a0fb4640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f369c13c6d0 0x7f369c0a2f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:05:16.087 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.070+0000 7f36a0fb4640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f369c13c6d0 0x7f369c0a2f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:39828/0 (socket says 172.21.15.42:39828) 2024-09-16T04:05:16.087 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.070+0000 7f369b7fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f369c1340c0 0x7f369c0a2a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:47508/0 (socket says 172.21.15.42:47508) 2024-09-16T04:05:16.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.070+0000 7f36a0fb4640 1 -- 172.21.15.42:0/252569823 learned_addr learned my addr 172.21.15.42:0/252569823 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:05:16.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.070+0000 7f36a0fb4640 10 monclient(hunting): get_auth_request con 0x7f369c13c6d0 auth_method 0 2024-09-16T04:05:16.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.070+0000 7f36a0fb4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:05:16.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.070+0000 7f36a0fb4640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:05:16.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.070+0000 7f36a0fb4640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:05:16.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.070+0000 7f369b7fe640 10 monclient(hunting): get_auth_request con 0x7f369c1340c0 auth_method 0 2024-09-16T04:05:16.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.070+0000 7f369b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:05:16.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.070+0000 7f369b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:05:16.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.070+0000 7f369b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:05:16.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.070+0000 7f369bfff640 1 --2- 172.21.15.42:0/252569823 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f369c06db90 0x7f369c1ced50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:05:16.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.070+0000 7f36a0fb4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:05:16.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.070+0000 7f36a0fb4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:05:16.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.070+0000 7f36a0fb4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:05:16.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.070+0000 7f369bfff640 10 monclient(hunting): get_auth_request con 0x7f369c06db90 auth_method 0 2024-09-16T04:05:16.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.070+0000 7f369bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:05:16.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.070+0000 7f369bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:05:16.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.070+0000 7f369bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:05:16.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.070+0000 7f369b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:05:16.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.070+0000 7f369b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:05:16.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.070+0000 7f369b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:05:16.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.070+0000 7f369bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:05:16.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.070+0000 7f369bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:05:16.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.070+0000 7f369bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:05:16.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.070+0000 7f36a0fb4640 10 monclient(hunting): handle_auth_done global_id 994109 payload 824 2024-09-16T04:05:16.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.070+0000 7f36a0fb4640 1 -- 172.21.15.42:0/252569823 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f369c06db90 msgr2=0x7f369c1ced50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:05:16.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.070+0000 7f36a0fb4640 1 --2- 172.21.15.42:0/252569823 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f369c06db90 0x7f369c1ced50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:16.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.070+0000 7f36a0fb4640 1 -- 172.21.15.42:0/252569823 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f369c1340c0 msgr2=0x7f369c0a2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:05:16.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.070+0000 7f36a0fb4640 1 --2- 172.21.15.42:0/252569823 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f369c1340c0 0x7f369c0a2a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:16.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.070+0000 7f36a0fb4640 10 monclient: _finish_hunting 0 2024-09-16T04:05:16.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.070+0000 7f36a0fb4640 1 monclient: found mon.noname-c 2024-09-16T04:05:16.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.070+0000 7f36a0fb4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:05:16.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.070+0000 7f36a0fb4640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T04:05:16.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.070+0000 7f36a0fb4640 1 -- 172.21.15.42:0/252569823 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f369c09e4b0 con 0x7f369c13c6d0 2024-09-16T04:05:16.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.070+0000 7f369b7fe640 1 --2- 172.21.15.42:0/252569823 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f369c1340c0 0x7f369c0a2a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:05:16.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.070+0000 7f36a0fb4640 1 --2- 172.21.15.42:0/252569823 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f369c13c6d0 0x7f369c0a2f50 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f369401a010 tx=0x7f3694019180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:05:16.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.070+0000 7f3698ff9640 10 client.?.objecter ms_handle_connect 0x7f369c13c6d0 2024-09-16T04:05:16.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.070+0000 7f3698ff9640 10 client.?.objecter resend_mon_ops 2024-09-16T04:05:16.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.070+0000 7f3698ff9640 1 -- 172.21.15.42:0/252569823 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36940b0040 con 0x7f369c13c6d0 2024-09-16T04:05:16.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.070+0000 7f3698ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:05:16.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.070+0000 7f3698ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:05:16.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.070+0000 7f3698ff9640 10 monclient: dump: 2024-09-16T04:05:16.091 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:05:16.091 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:05:16.091 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:16.091 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:16.091 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:05:16.091 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:05:16.091 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:05:16.091 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:05:16.091 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:05:16.091 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:05:16.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.070+0000 7f3698ff9640 10 monclient: _finish_auth 0 2024-09-16T04:05:16.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.070+0000 7f3698ff9640 10 monclient: _check_auth_tickets 2024-09-16T04:05:16.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.070+0000 7f3698ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:05:16.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.070+0000 7f3698ff9640 1 -- 172.21.15.42:0/252569823 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f36940b4420 con 0x7f369c13c6d0 2024-09-16T04:05:16.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.070+0000 7f36a2a17640 5 monclient: authenticate success, global_id 994109 2024-09-16T04:05:16.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.070+0000 7f36a2a17640 10 monclient: _renew_subs 2024-09-16T04:05:16.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.070+0000 7f36a2a17640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:05:16.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.070+0000 7f36a2a17640 1 -- 172.21.15.42:0/252569823 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f369c1cf290 con 0x7f369c13c6d0 2024-09-16T04:05:16.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.070+0000 7f3698ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T04:05:16.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.070+0000 7f3698ff9640 1 -- 172.21.15.42:0/252569823 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36940b47f0 con 0x7f369c13c6d0 2024-09-16T04:05:16.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.070+0000 7f3698ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:05:16.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.070+0000 7f3698ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:05:16.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.070+0000 7f3698ff9640 10 monclient: dump: 2024-09-16T04:05:16.092 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:05:16.092 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:05:16.092 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:16.093 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:16.093 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:05:16.093 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:05:16.093 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:05:16.093 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:05:16.093 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:05:16.093 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:05:16.093 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.070+0000 7f36a2a17640 10 client.994109.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:05:16.093 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.070+0000 7f369a7fc640 4 set_mon_vals no callback set 2024-09-16T04:05:16.093 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.070+0000 7f36a2a17640 10 monclient: _renew_subs 2024-09-16T04:05:16.093 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.070+0000 7f36a2a17640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:05:16.093 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.070+0000 7f36a2a17640 1 -- 172.21.15.42:0/252569823 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f369c13b990 con 0x7f369c13c6d0 2024-09-16T04:05:16.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.070+0000 7f36a2a17640 1 librados: init done 2024-09-16T04:05:16.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.070+0000 7f3698ff9640 1 -- 172.21.15.42:0/252569823 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f36940b4990 con 0x7f369c13c6d0 2024-09-16T04:05:16.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.070+0000 7f36a2a17640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:05:16.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.070+0000 7f36a2a17640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:05:16.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.070+0000 7f36a2a17640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:05:16.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.070+0000 7f36a2a17640 1 -- 172.21.15.42:0/252569823 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f369c0a37f0 con 0x7f369c13c6d0 2024-09-16T04:05:16.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.070+0000 7f3698ff9640 1 --2- 172.21.15.42:0/252569823 >> v2:172.21.15.154:6812/254845342 conn(0x7f3688037840 0x7f3688039d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:05:16.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.070+0000 7f3698ff9640 1 -- 172.21.15.42:0/252569823 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(703..703 src has 451..703) ==== 5899+0+0 (secure 0 0 0) 0x7f36940e2b10 con 0x7f369c13c6d0 2024-09-16T04:05:16.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.070+0000 7f3698ff9640 10 client.994109.objecter ms_dispatch 0x7f369c000ba0 osd_map(703..703 src has 451..703) 2024-09-16T04:05:16.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.070+0000 7f3698ff9640 3 client.994109.objecter handle_osd_map got epochs [703,703] > 0 2024-09-16T04:05:16.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.070+0000 7f3698ff9640 3 client.994109.objecter handle_osd_map decoding full epoch 703 2024-09-16T04:05:16.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.070+0000 7f3698ff9640 20 client.994109.objecter dump_active .. 0 homeless 2024-09-16T04:05:16.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.070+0000 7f369bfff640 1 --2- 172.21.15.42:0/252569823 >> v2:172.21.15.154:6812/254845342 conn(0x7f3688037840 0x7f3688039d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:05:16.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.074+0000 7f369bfff640 10 monclient: get_auth_request con 0x7f3688037840 auth_method 0 2024-09-16T04:05:16.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.074+0000 7f369bfff640 1 --2- 172.21.15.42:0/252569823 >> v2:172.21.15.154:6812/254845342 conn(0x7f3688037840 0x7f3688039d00 secure :-1 s=READY pgs=3843 cs=0 l=1 rev1=1 crypto rx=0x7f3690007fd0 tx=0x7f3690009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:05:16.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.074+0000 7f3698ff9640 10 client.994109.objecter ms_handle_connect 0x7f3688037840 2024-09-16T04:05:16.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.078+0000 7f3698ff9640 1 -- 172.21.15.42:0/252569823 <== mon.2 v2:172.21.15.42:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f36940160f0 con 0x7f369c13c6d0 2024-09-16T04:05:16.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.078+0000 7f3698ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:05:16.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.078+0000 7f3698ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:05:16.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.314+0000 7f36a2a17640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T04:05:16.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.314+0000 7f36a2a17640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T04:05:16.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.314+0000 7f36a2a17640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:05:16.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.314+0000 7f36a2a17640 1 -- 172.21.15.42:0/252569823 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f369c068e40 con 0x7f369c13c6d0 2024-09-16T04:05:16.324 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.318+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.324 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.318+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=27,config=2,health=11,kv=9,logm=51} crc {auth=522666727,config=692619927,health=115915343,kv=2176084214,logm=386174061}) 2024-09-16T04:05:16.324 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.318+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=30,config=2,health=12,kv=10,logm=46} crc {auth=3119714929,config=692619927,health=961846530,kv=3446188498,logm=2385579120}) 2024-09-16T04:05:16.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.318+0000 7f3698ff9640 1 -- 172.21.15.42:0/252569823 <== mon.2 v2:172.21.15.42:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f36940b4c80 con 0x7f369c13c6d0 2024-09-16T04:05:16.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.318+0000 7f3698ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T04:05:16.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.318+0000 7f3698ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:05:16.325 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.318+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.325 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.318+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=27,config=2,health=11,kv=9,logm=51} crc {auth=522666727,config=692619927,health=115915343,kv=2176084214,logm=386174061}) 2024-09-16T04:05:16.325 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.318+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=30,config=2,health=12,kv=10,logm=46} crc {auth=2164148247,config=692619927,health=3826333626,kv=3446188498,logm=3142734165}) 2024-09-16T04:05:16.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.322+0000 7f36a2a17640 10 librados: watch_flush enter 2024-09-16T04:05:16.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.322+0000 7f36a2a17640 10 librados: watch_flush exit 2024-09-16T04:05:16.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.322+0000 7f36a2a17640 20 client.994109.objecter shutdown clearing up homeless session... 2024-09-16T04:05:16.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.322+0000 7f36a2a17640 10 client.994109.objecter successfully canceled tick 2024-09-16T04:05:16.328 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.322+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.328 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.322+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3281593680}) 2024-09-16T04:05:16.328 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.322+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4254770208}) 2024-09-16T04:05:16.328 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.322+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.328 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.322+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3281593680}) 2024-09-16T04:05:16.328 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.322+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=208543208}) 2024-09-16T04:05:16.329 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.322+0000 7f36a2a17640 1 -- 172.21.15.42:0/252569823 >> v2:172.21.15.154:6812/254845342 conn(0x7f3688037840 msgr2=0x7f3688039d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:05:16.329 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.322+0000 7f36a2a17640 1 --2- 172.21.15.42:0/252569823 >> v2:172.21.15.154:6812/254845342 conn(0x7f3688037840 0x7f3688039d00 secure :-1 s=READY pgs=3843 cs=0 l=1 rev1=1 crypto rx=0x7f3690007fd0 tx=0x7f3690009040 comp rx=0 tx=0).stop 2024-09-16T04:05:16.329 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.322+0000 7f36a2a17640 10 monclient: shutdown 2024-09-16T04:05:16.329 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.322+0000 7f36a2a17640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:05:16.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.322+0000 7f36a2a17640 1 -- 172.21.15.42:0/252569823 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f369c13c6d0 msgr2=0x7f369c0a2f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:05:16.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.322+0000 7f36a2a17640 1 --2- 172.21.15.42:0/252569823 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f369c13c6d0 0x7f369c0a2f50 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f369401a010 tx=0x7f3694019180 comp rx=0 tx=0).stop 2024-09-16T04:05:16.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.322+0000 7f36a2a17640 1 -- 172.21.15.42:0/252569823 shutdown_connections 2024-09-16T04:05:16.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.322+0000 7f36a2a17640 1 --2- 172.21.15.42:0/252569823 >> v2:172.21.15.154:6812/254845342 conn(0x7f3688037840 0x7f3688039d00 unknown :-1 s=CLOSED pgs=3843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:16.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.322+0000 7f36a2a17640 1 --2- 172.21.15.42:0/252569823 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f369c13c6d0 0x7f369c0a2f50 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:16.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.322+0000 7f36a2a17640 1 --2- 172.21.15.42:0/252569823 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f369c1340c0 0x7f369c0a2a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:16.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.322+0000 7f36a2a17640 1 --2- 172.21.15.42:0/252569823 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f369c06db90 0x7f369c1ced50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:16.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.322+0000 7f36a2a17640 1 -- 172.21.15.42:0/252569823 >> 172.21.15.42:0/252569823 conn(0x7f369c0a9fb0 msgr2=0x7f369c133300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:05:16.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.322+0000 7f36a2a17640 1 -- 172.21.15.42:0/252569823 shutdown_connections 2024-09-16T04:05:16.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.322+0000 7f36a2a17640 1 -- 172.21.15.42:0/252569823 wait complete. 2024-09-16T04:05:16.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:16.322+0000 7f36a2a17640 1 librados: shutdown 2024-09-16T04:05:16.331 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.326+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.331 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.326+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1124974963}) 2024-09-16T04:05:16.331 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.326+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3343623380}) 2024-09-16T04:05:16.331 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.326+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.331 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.326+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1124974963}) 2024-09-16T04:05:16.331 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.326+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1523780687}) 2024-09-16T04:05:16.336 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.330+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.336 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.330+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2835764495}) 2024-09-16T04:05:16.336 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.330+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3541166468}) 2024-09-16T04:05:16.336 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.330+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.337 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.330+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2835764495}) 2024-09-16T04:05:16.337 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.330+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4277055801}) 2024-09-16T04:05:16.341 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.334+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.341 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.334+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3200016488}) 2024-09-16T04:05:16.341 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.334+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4255100863}) 2024-09-16T04:05:16.341 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.334+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.341 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.334+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3200016488}) 2024-09-16T04:05:16.341 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.334+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2227101941}) 2024-09-16T04:05:16.342 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.338+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.343 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.338+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2498302764}) 2024-09-16T04:05:16.343 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.338+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3195898306}) 2024-09-16T04:05:16.343 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.338+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.343 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.338+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2498302764}) 2024-09-16T04:05:16.343 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.338+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3792588982}) 2024-09-16T04:05:16.344 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.338+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.345 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.338+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=613743400}) 2024-09-16T04:05:16.345 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.338+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2579975017}) 2024-09-16T04:05:16.345 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.338+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.345 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.338+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=613743400}) 2024-09-16T04:05:16.345 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.338+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=195956187}) 2024-09-16T04:05:16.347 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.342+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.347 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.342+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3702349723}) 2024-09-16T04:05:16.347 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.342+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2098721502}) 2024-09-16T04:05:16.347 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.342+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.347 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.342+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3702349723}) 2024-09-16T04:05:16.347 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.342+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=351904621}) 2024-09-16T04:05:16.348 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T04:05:16.349 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.342+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.349 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.342+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=769562013}) 2024-09-16T04:05:16.349 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.342+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1032227806}) 2024-09-16T04:05:16.349 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.342+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.349 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.342+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=769562013}) 2024-09-16T04:05:16.349 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.342+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2706948491}) 2024-09-16T04:05:16.350 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.346+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.350 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.346+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2645272055}) 2024-09-16T04:05:16.350 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.346+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1889881607}) 2024-09-16T04:05:16.351 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.346+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.351 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.346+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2645272055}) 2024-09-16T04:05:16.351 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.346+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1009624399}) 2024-09-16T04:05:16.352 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.346+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.352 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.346+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2594986668}) 2024-09-16T04:05:16.352 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.346+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2793067287}) 2024-09-16T04:05:16.352 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.346+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.352 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.346+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2594986668}) 2024-09-16T04:05:16.352 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.346+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2450546115}) 2024-09-16T04:05:16.354 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.350+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.354 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.350+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3222761717}) 2024-09-16T04:05:16.354 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.350+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=199824668}) 2024-09-16T04:05:16.354 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.350+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.354 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.350+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3222761717}) 2024-09-16T04:05:16.355 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.350+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2283197699}) 2024-09-16T04:05:16.356 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.350+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.357 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.350+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4100816402}) 2024-09-16T04:05:16.357 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.350+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1569765057}) 2024-09-16T04:05:16.357 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.350+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.357 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.350+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4100816402}) 2024-09-16T04:05:16.357 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.350+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3423730801}) 2024-09-16T04:05:16.359 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.354+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.359 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.354+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=592377081}) 2024-09-16T04:05:16.359 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.354+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3141515837}) 2024-09-16T04:05:16.359 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.354+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.359 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.354+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=592377081}) 2024-09-16T04:05:16.359 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.354+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3246828407}) 2024-09-16T04:05:16.361 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.354+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.361 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.354+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1132600110}) 2024-09-16T04:05:16.361 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.354+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2787091906}) 2024-09-16T04:05:16.361 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.354+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.361 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.354+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1132600110}) 2024-09-16T04:05:16.361 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.354+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1597125807}) 2024-09-16T04:05:16.363 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.358+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.363 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.358+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=533324775}) 2024-09-16T04:05:16.363 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.358+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3545229291}) 2024-09-16T04:05:16.363 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.358+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.363 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.358+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=533324775}) 2024-09-16T04:05:16.363 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.358+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1336434189}) 2024-09-16T04:05:16.365 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.358+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.365 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.358+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2515141668}) 2024-09-16T04:05:16.365 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.358+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=68060537}) 2024-09-16T04:05:16.365 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.358+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.365 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.362+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2515141668}) 2024-09-16T04:05:16.366 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.362+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2948955972}) 2024-09-16T04:05:16.367 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.362+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.367 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.362+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1119641063}) 2024-09-16T04:05:16.367 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.362+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1457448488}) 2024-09-16T04:05:16.367 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.362+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.368 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.362+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1119641063}) 2024-09-16T04:05:16.368 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.362+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2042747999}) 2024-09-16T04:05:16.369 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.362+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.369 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.362+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3167518959}) 2024-09-16T04:05:16.369 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.362+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1408144603}) 2024-09-16T04:05:16.369 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.366+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.369 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.366+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3167518959}) 2024-09-16T04:05:16.370 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.366+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=222672423}) 2024-09-16T04:05:16.371 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.366+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.371 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.366+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=488448417}) 2024-09-16T04:05:16.371 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.366+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2046240504}) 2024-09-16T04:05:16.371 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.366+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.371 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.366+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=488448417}) 2024-09-16T04:05:16.372 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.366+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3828866747}) 2024-09-16T04:05:16.373 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.366+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.373 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.366+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2181579814}) 2024-09-16T04:05:16.373 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.366+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=12462233}) 2024-09-16T04:05:16.373 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.366+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.373 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.366+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2181579814}) 2024-09-16T04:05:16.373 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.366+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4139984894}) 2024-09-16T04:05:16.375 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.370+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.375 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.370+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3623996308}) 2024-09-16T04:05:16.375 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.370+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3478330873}) 2024-09-16T04:05:16.375 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.370+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.375 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.370+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3623996308}) 2024-09-16T04:05:16.375 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.370+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=794284910}) 2024-09-16T04:05:16.377 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.370+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.377 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.370+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2581844363}) 2024-09-16T04:05:16.377 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.370+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1088908438}) 2024-09-16T04:05:16.377 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.370+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.377 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.370+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2581844363}) 2024-09-16T04:05:16.377 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.370+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3022961134}) 2024-09-16T04:05:16.378 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.374+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.379 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.374+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=798007177}) 2024-09-16T04:05:16.379 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.374+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1966406796}) 2024-09-16T04:05:16.379 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.374+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.379 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.374+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=798007177}) 2024-09-16T04:05:16.379 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.374+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=415660540}) 2024-09-16T04:05:16.381 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.374+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.381 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.374+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1127743085}) 2024-09-16T04:05:16.381 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.374+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3965951445}) 2024-09-16T04:05:16.381 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.374+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.381 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.374+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1127743085}) 2024-09-16T04:05:16.381 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.374+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1099208798}) 2024-09-16T04:05:16.383 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.378+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.383 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.378+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4145955888}) 2024-09-16T04:05:16.383 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.378+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1457744794}) 2024-09-16T04:05:16.383 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.378+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.383 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.378+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4145955888}) 2024-09-16T04:05:16.383 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.378+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3726409663}) 2024-09-16T04:05:16.384 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.378+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.385 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.378+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2245600104}) 2024-09-16T04:05:16.385 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.378+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3410024621}) 2024-09-16T04:05:16.385 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.378+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.385 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.378+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2245600104}) 2024-09-16T04:05:16.385 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.378+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=335420043}) 2024-09-16T04:05:16.386 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.382+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.386 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.382+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1463100194}) 2024-09-16T04:05:16.386 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.382+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2419329140}) 2024-09-16T04:05:16.387 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.382+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.387 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.382+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1463100194}) 2024-09-16T04:05:16.387 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.382+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1468865913}) 2024-09-16T04:05:16.388 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.382+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.388 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.382+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2565946230}) 2024-09-16T04:05:16.389 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.382+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3865760251}) 2024-09-16T04:05:16.389 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.382+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.389 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.382+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2565946230}) 2024-09-16T04:05:16.389 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.382+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3805896742}) 2024-09-16T04:05:16.391 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.386+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.391 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.386+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2057732776}) 2024-09-16T04:05:16.391 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.386+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=179185302}) 2024-09-16T04:05:16.391 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.386+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.391 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.386+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2057732776}) 2024-09-16T04:05:16.391 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.386+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1108028970}) 2024-09-16T04:05:16.393 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.386+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.393 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.386+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2886946494}) 2024-09-16T04:05:16.393 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.386+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2334858691}) 2024-09-16T04:05:16.393 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.386+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.393 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.386+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2886946494}) 2024-09-16T04:05:16.393 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.386+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1792116429}) 2024-09-16T04:05:16.394 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.390+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.395 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.390+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2533891438}) 2024-09-16T04:05:16.395 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.390+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1156165180}) 2024-09-16T04:05:16.395 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.390+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.395 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.390+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2533891438}) 2024-09-16T04:05:16.395 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.390+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2003521406}) 2024-09-16T04:05:16.397 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.390+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.397 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.390+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2842066210}) 2024-09-16T04:05:16.397 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.390+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3532200097}) 2024-09-16T04:05:16.397 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.390+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.397 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.390+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2842066210}) 2024-09-16T04:05:16.397 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.390+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2501051232}) 2024-09-16T04:05:16.398 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.394+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.399 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.394+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3430087451}) 2024-09-16T04:05:16.399 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.394+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2543422508}) 2024-09-16T04:05:16.399 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.394+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.399 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.394+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3430087451}) 2024-09-16T04:05:16.399 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.394+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=499524095}) 2024-09-16T04:05:16.400 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.394+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.400 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.394+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=566598330}) 2024-09-16T04:05:16.400 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.394+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=835814150}) 2024-09-16T04:05:16.400 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.394+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.401 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.394+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=566598330}) 2024-09-16T04:05:16.401 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.394+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1534262073}) 2024-09-16T04:05:16.402 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.398+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.402 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.398+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3516716716}) 2024-09-16T04:05:16.402 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.398+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=523806712}) 2024-09-16T04:05:16.402 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.398+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.402 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.398+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3516716716}) 2024-09-16T04:05:16.402 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.398+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2885199754}) 2024-09-16T04:05:16.404 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.398+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.404 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.398+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1241907477}) 2024-09-16T04:05:16.404 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.398+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2373568785}) 2024-09-16T04:05:16.404 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.398+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.404 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.398+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1241907477}) 2024-09-16T04:05:16.404 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.398+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2655163012}) 2024-09-16T04:05:16.406 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.402+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.406 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.402+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2460661863}) 2024-09-16T04:05:16.406 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.402+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1585217578}) 2024-09-16T04:05:16.406 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.402+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.406 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.402+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2460661863}) 2024-09-16T04:05:16.406 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.402+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2897643444}) 2024-09-16T04:05:16.408 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.402+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.408 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.402+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2839038625}) 2024-09-16T04:05:16.408 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.402+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1449368999}) 2024-09-16T04:05:16.408 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.402+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.408 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.402+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2839038625}) 2024-09-16T04:05:16.408 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.402+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1345253794}) 2024-09-16T04:05:16.410 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.406+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.410 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.406+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1277050905}) 2024-09-16T04:05:16.410 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.406+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2575878207}) 2024-09-16T04:05:16.410 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.406+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.410 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.406+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1277050905}) 2024-09-16T04:05:16.410 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.406+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4184455031}) 2024-09-16T04:05:16.411 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.406+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.411 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.406+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2543246040}) 2024-09-16T04:05:16.411 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.406+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3614564825}) 2024-09-16T04:05:16.411 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.406+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.412 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.406+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2543246040}) 2024-09-16T04:05:16.412 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.406+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1582321093}) 2024-09-16T04:05:16.413 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.406+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.413 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.406+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3230045442}) 2024-09-16T04:05:16.413 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.406+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=675156832}) 2024-09-16T04:05:16.413 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.406+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.413 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.406+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3230045442}) 2024-09-16T04:05:16.413 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.406+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2228676562}) 2024-09-16T04:05:16.415 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.410+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.415 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.410+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3412019540}) 2024-09-16T04:05:16.415 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.410+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=199091938}) 2024-09-16T04:05:16.415 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.410+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.415 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.410+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3412019540}) 2024-09-16T04:05:16.415 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.410+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2929424820}) 2024-09-16T04:05:16.417 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.410+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.417 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.410+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2121977078}) 2024-09-16T04:05:16.417 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.410+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1699425755}) 2024-09-16T04:05:16.417 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.410+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.417 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.410+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2121977078}) 2024-09-16T04:05:16.417 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.410+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3899454817}) 2024-09-16T04:05:16.419 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.414+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.419 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.414+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3251764095}) 2024-09-16T04:05:16.419 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.414+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2801137098}) 2024-09-16T04:05:16.419 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.414+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.419 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.414+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3251764095}) 2024-09-16T04:05:16.419 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.414+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2701949951}) 2024-09-16T04:05:16.421 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.414+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.421 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.414+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3420586619}) 2024-09-16T04:05:16.421 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.414+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3216125451}) 2024-09-16T04:05:16.421 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.414+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.421 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.414+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3420586619}) 2024-09-16T04:05:16.421 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.414+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3099974856}) 2024-09-16T04:05:16.423 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.418+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.423 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.418+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3584450476}) 2024-09-16T04:05:16.423 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.418+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=514679044}) 2024-09-16T04:05:16.423 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.418+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.423 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.418+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3584450476}) 2024-09-16T04:05:16.423 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.418+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=186374266}) 2024-09-16T04:05:16.425 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.418+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.425 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.418+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1884562926}) 2024-09-16T04:05:16.425 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.418+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1774865860}) 2024-09-16T04:05:16.425 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.418+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.425 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.418+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1884562926}) 2024-09-16T04:05:16.425 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.418+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3555170401}) 2024-09-16T04:05:16.426 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.422+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.426 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.422+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4022585792}) 2024-09-16T04:05:16.426 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.422+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4195949556}) 2024-09-16T04:05:16.427 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.422+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.427 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.422+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4022585792}) 2024-09-16T04:05:16.427 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.422+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3846457041}) 2024-09-16T04:05:16.429 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.422+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.429 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.422+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1626254421}) 2024-09-16T04:05:16.429 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.422+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3656643328}) 2024-09-16T04:05:16.429 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.422+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.429 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.422+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1626254421}) 2024-09-16T04:05:16.429 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.422+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3613282285}) 2024-09-16T04:05:16.430 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.426+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.430 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.426+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2756709398}) 2024-09-16T04:05:16.431 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.426+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3866812520}) 2024-09-16T04:05:16.431 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.426+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.431 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.426+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2756709398}) 2024-09-16T04:05:16.431 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.426+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1895194268}) 2024-09-16T04:05:16.432 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.426+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.433 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.426+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=314887510}) 2024-09-16T04:05:16.433 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.426+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1979198271}) 2024-09-16T04:05:16.433 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.426+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.433 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.426+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=314887510}) 2024-09-16T04:05:16.433 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.426+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1746920156}) 2024-09-16T04:05:16.434 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.430+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.434 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.430+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3137213521}) 2024-09-16T04:05:16.434 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.430+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3069105772}) 2024-09-16T04:05:16.435 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.430+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.435 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.430+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3137213521}) 2024-09-16T04:05:16.435 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.430+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4092260527}) 2024-09-16T04:05:16.436 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.430+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.436 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.430+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3709789371}) 2024-09-16T04:05:16.436 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.430+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1408389851}) 2024-09-16T04:05:16.436 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.430+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.436 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.430+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3709789371}) 2024-09-16T04:05:16.437 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.430+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=850407896}) 2024-09-16T04:05:16.438 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.434+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.438 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.434+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1736182134}) 2024-09-16T04:05:16.438 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.434+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3167169630}) 2024-09-16T04:05:16.438 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.434+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.438 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.434+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1736182134}) 2024-09-16T04:05:16.438 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.434+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3560235938}) 2024-09-16T04:05:16.440 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.434+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.440 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.434+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1743551839}) 2024-09-16T04:05:16.440 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.434+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3551608640}) 2024-09-16T04:05:16.440 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.434+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.440 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.434+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1743551839}) 2024-09-16T04:05:16.440 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.434+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=202744960}) 2024-09-16T04:05:16.442 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.438+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.442 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.438+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2813022588}) 2024-09-16T04:05:16.442 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.438+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2468022169}) 2024-09-16T04:05:16.442 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.438+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.442 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.438+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2813022588}) 2024-09-16T04:05:16.442 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.438+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2743180056}) 2024-09-16T04:05:16.443 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.438+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.443 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.438+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=706593441}) 2024-09-16T04:05:16.443 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.438+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2819739536}) 2024-09-16T04:05:16.443 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.438+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.443 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.438+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=706593441}) 2024-09-16T04:05:16.444 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.438+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4253619218}) 2024-09-16T04:05:16.445 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.438+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.445 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.438+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3678791388}) 2024-09-16T04:05:16.446 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.438+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=617947920}) 2024-09-16T04:05:16.446 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.438+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.446 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.438+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3678791388}) 2024-09-16T04:05:16.446 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.442+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2769728114}) 2024-09-16T04:05:16.447 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.442+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.448 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.442+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1343505770}) 2024-09-16T04:05:16.448 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.442+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1982114985}) 2024-09-16T04:05:16.448 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.442+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.448 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.442+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1343505770}) 2024-09-16T04:05:16.448 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.442+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3164167322}) 2024-09-16T04:05:16.449 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.446+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.450 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.446+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3067670581}) 2024-09-16T04:05:16.450 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.446+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=29154302}) 2024-09-16T04:05:16.450 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.446+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.450 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.446+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3067670581}) 2024-09-16T04:05:16.450 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.446+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=37144651}) 2024-09-16T04:05:16.451 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.446+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.451 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.446+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3508894508}) 2024-09-16T04:05:16.451 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.446+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3798964284}) 2024-09-16T04:05:16.451 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.446+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.452 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.446+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3508894508}) 2024-09-16T04:05:16.452 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.446+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3201862941}) 2024-09-16T04:05:16.453 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.446+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.453 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.446+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4257608623}) 2024-09-16T04:05:16.453 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.446+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4286175644}) 2024-09-16T04:05:16.453 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.446+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.453 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.446+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4257608623}) 2024-09-16T04:05:16.454 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.446+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1272926730}) 2024-09-16T04:05:16.455 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.450+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.455 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.450+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3832471410}) 2024-09-16T04:05:16.455 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.450+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2329091275}) 2024-09-16T04:05:16.456 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.450+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.456 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.450+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3832471410}) 2024-09-16T04:05:16.456 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.450+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4177476957}) 2024-09-16T04:05:16.457 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.450+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.457 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.450+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1654027448}) 2024-09-16T04:05:16.457 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.450+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3350894808}) 2024-09-16T04:05:16.457 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.450+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.457 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.450+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1654027448}) 2024-09-16T04:05:16.458 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.450+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=602280539}) 2024-09-16T04:05:16.459 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.454+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.459 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.454+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2570905974}) 2024-09-16T04:05:16.459 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.454+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2699117395}) 2024-09-16T04:05:16.459 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.454+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.459 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.454+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2570905974}) 2024-09-16T04:05:16.460 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.454+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2317151043}) 2024-09-16T04:05:16.461 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.454+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.461 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.454+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1523168271}) 2024-09-16T04:05:16.461 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.454+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3077837654}) 2024-09-16T04:05:16.461 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.458+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.462 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.458+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1523168271}) 2024-09-16T04:05:16.462 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.458+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1533043972}) 2024-09-16T04:05:16.463 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.458+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.463 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.458+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1475595903}) 2024-09-16T04:05:16.463 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.458+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=718271224}) 2024-09-16T04:05:16.464 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.458+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.464 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.458+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1475595903}) 2024-09-16T04:05:16.464 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.458+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1270010695}) 2024-09-16T04:05:16.465 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.462+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.465 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.462+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2353368901}) 2024-09-16T04:05:16.466 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.462+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1412935570}) 2024-09-16T04:05:16.466 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.462+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.466 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.462+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2353368901}) 2024-09-16T04:05:16.466 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.462+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2489452023}) 2024-09-16T04:05:16.467 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.462+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.467 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.462+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1443267569}) 2024-09-16T04:05:16.467 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.462+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=770764312}) 2024-09-16T04:05:16.468 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.462+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.468 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.462+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1443267569}) 2024-09-16T04:05:16.468 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.462+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2814566510}) 2024-09-16T04:05:16.469 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.466+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.470 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.466+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2365708633}) 2024-09-16T04:05:16.470 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.466+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4266312188}) 2024-09-16T04:05:16.470 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.466+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.470 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.466+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2365708633}) 2024-09-16T04:05:16.470 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.466+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1303316525}) 2024-09-16T04:05:16.472 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.466+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.472 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.466+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3212188373}) 2024-09-16T04:05:16.472 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.466+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1334521055}) 2024-09-16T04:05:16.472 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.466+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.472 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.466+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3212188373}) 2024-09-16T04:05:16.472 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.466+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=69875636}) 2024-09-16T04:05:16.474 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.470+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.474 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.470+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3948911882}) 2024-09-16T04:05:16.474 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.470+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2538064225}) 2024-09-16T04:05:16.474 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.470+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.474 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.470+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3948911882}) 2024-09-16T04:05:16.474 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.470+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2384814364}) 2024-09-16T04:05:16.476 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.470+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.476 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.470+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2942357797}) 2024-09-16T04:05:16.476 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.470+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1409822738}) 2024-09-16T04:05:16.476 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.470+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.476 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.470+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2942357797}) 2024-09-16T04:05:16.477 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.470+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2772983779}) 2024-09-16T04:05:16.478 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.474+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.478 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.474+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1956169280}) 2024-09-16T04:05:16.478 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.474+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2138386231}) 2024-09-16T04:05:16.479 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.474+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.479 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.474+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1956169280}) 2024-09-16T04:05:16.479 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.474+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3392872442}) 2024-09-16T04:05:16.480 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.474+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.480 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.474+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3489128288}) 2024-09-16T04:05:16.480 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.474+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2633495721}) 2024-09-16T04:05:16.481 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.474+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.481 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.474+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3489128288}) 2024-09-16T04:05:16.481 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.474+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3724030827}) 2024-09-16T04:05:16.483 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.478+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.483 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.478+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2842683846}) 2024-09-16T04:05:16.483 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.478+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3448714783}) 2024-09-16T04:05:16.483 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.478+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.483 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.478+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2842683846}) 2024-09-16T04:05:16.483 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.478+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1742275104}) 2024-09-16T04:05:16.485 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.478+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.485 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.478+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=308893921}) 2024-09-16T04:05:16.485 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.478+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1624399517}) 2024-09-16T04:05:16.485 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.478+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.485 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.478+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=308893921}) 2024-09-16T04:05:16.485 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.478+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2399723160}) 2024-09-16T04:05:16.487 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.482+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.487 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.482+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=718197780}) 2024-09-16T04:05:16.487 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.482+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4014395598}) 2024-09-16T04:05:16.487 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.482+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.487 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.482+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=718197780}) 2024-09-16T04:05:16.487 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.482+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3572664582}) 2024-09-16T04:05:16.489 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.486+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.490 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.486+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1889425194}) 2024-09-16T04:05:16.490 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.486+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1268992946}) 2024-09-16T04:05:16.490 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.486+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.490 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.486+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1889425194}) 2024-09-16T04:05:16.490 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.486+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=946714091}) 2024-09-16T04:05:16.491 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.486+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.491 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.486+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2428589801}) 2024-09-16T04:05:16.491 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.486+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3972770597}) 2024-09-16T04:05:16.492 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.486+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.492 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.486+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2428589801}) 2024-09-16T04:05:16.492 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.486+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3452305272}) 2024-09-16T04:05:16.494 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.490+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.494 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.490+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4008489135}) 2024-09-16T04:05:16.494 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.490+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3378719697}) 2024-09-16T04:05:16.494 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.490+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.494 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.490+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4008489135}) 2024-09-16T04:05:16.494 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.490+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1687274383}) 2024-09-16T04:05:16.496 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.490+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.496 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.490+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1068002823}) 2024-09-16T04:05:16.496 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.490+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=627509641}) 2024-09-16T04:05:16.496 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.490+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.496 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.490+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1068002823}) 2024-09-16T04:05:16.496 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.490+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=966971695}) 2024-09-16T04:05:16.498 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.494+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.498 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.494+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1402324856}) 2024-09-16T04:05:16.498 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.494+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2814776949}) 2024-09-16T04:05:16.498 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.494+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.499 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.494+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1402324856}) 2024-09-16T04:05:16.499 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.494+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2656335739}) 2024-09-16T04:05:16.500 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.494+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.500 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.494+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1233423581}) 2024-09-16T04:05:16.500 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.494+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1713998174}) 2024-09-16T04:05:16.501 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.494+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.501 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.494+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1233423581}) 2024-09-16T04:05:16.501 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.494+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1991482701}) 2024-09-16T04:05:16.503 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.498+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.503 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.498+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=131753320}) 2024-09-16T04:05:16.503 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.498+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=135575404}) 2024-09-16T04:05:16.503 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.498+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.503 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.498+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=131753320}) 2024-09-16T04:05:16.503 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.498+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=929942736}) 2024-09-16T04:05:16.505 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.498+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.505 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.498+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2420553539}) 2024-09-16T04:05:16.505 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.498+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3748911478}) 2024-09-16T04:05:16.505 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.498+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.505 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.498+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2420553539}) 2024-09-16T04:05:16.505 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.498+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1451580657}) 2024-09-16T04:05:16.507 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.502+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.507 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.502+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3535001483}) 2024-09-16T04:05:16.507 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.502+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2694767992}) 2024-09-16T04:05:16.507 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.502+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.507 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.502+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3535001483}) 2024-09-16T04:05:16.507 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.502+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1984232541}) 2024-09-16T04:05:16.509 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.502+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.509 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.502+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=654927906}) 2024-09-16T04:05:16.509 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.502+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1617811518}) 2024-09-16T04:05:16.509 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.502+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.510 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.502+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=654927906}) 2024-09-16T04:05:16.510 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.506+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3128456899}) 2024-09-16T04:05:16.511 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.506+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.511 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.506+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2895330608}) 2024-09-16T04:05:16.511 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.506+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2107629716}) 2024-09-16T04:05:16.511 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.506+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.512 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.506+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2895330608}) 2024-09-16T04:05:16.512 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.506+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=976476850}) 2024-09-16T04:05:16.513 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.506+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.513 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.510+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2119567874}) 2024-09-16T04:05:16.514 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.510+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4271441509}) 2024-09-16T04:05:16.514 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.510+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.514 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.510+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2119567874}) 2024-09-16T04:05:16.514 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.510+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1315895346}) 2024-09-16T04:05:16.515 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.510+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.515 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.510+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=201315874}) 2024-09-16T04:05:16.516 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.510+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2737413827}) 2024-09-16T04:05:16.516 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.510+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.516 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.510+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=201315874}) 2024-09-16T04:05:16.516 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.510+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=88064170}) 2024-09-16T04:05:16.517 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.514+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.518 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.514+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3204656898}) 2024-09-16T04:05:16.518 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.514+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=615810714}) 2024-09-16T04:05:16.518 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.514+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.518 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.514+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3204656898}) 2024-09-16T04:05:16.518 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.514+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4129237905}) 2024-09-16T04:05:16.520 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.514+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.520 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.514+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2952292157}) 2024-09-16T04:05:16.520 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.514+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1118644618}) 2024-09-16T04:05:16.520 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.514+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.520 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.514+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2952292157}) 2024-09-16T04:05:16.520 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.514+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=880325266}) 2024-09-16T04:05:16.522 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.518+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.522 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.518+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3442623672}) 2024-09-16T04:05:16.522 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.518+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=362614586}) 2024-09-16T04:05:16.522 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.518+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.522 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.518+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3442623672}) 2024-09-16T04:05:16.523 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.518+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4187299056}) 2024-09-16T04:05:16.524 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.518+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.524 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.518+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2780232024}) 2024-09-16T04:05:16.525 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.518+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3419197050}) 2024-09-16T04:05:16.525 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.518+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.525 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.518+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2780232024}) 2024-09-16T04:05:16.525 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.518+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4273254472}) 2024-09-16T04:05:16.528 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.522+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.528 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.522+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=66101715}) 2024-09-16T04:05:16.528 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.522+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2942701815}) 2024-09-16T04:05:16.528 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.522+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.528 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.522+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=66101715}) 2024-09-16T04:05:16.528 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.522+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1610296858}) 2024-09-16T04:05:16.529 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.522+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.530 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.522+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3438155240}) 2024-09-16T04:05:16.530 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.522+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2204132675}) 2024-09-16T04:05:16.530 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.522+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.530 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.522+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3438155240}) 2024-09-16T04:05:16.530 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.522+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3362669908}) 2024-09-16T04:05:16.532 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.526+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.532 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.526+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=970186530}) 2024-09-16T04:05:16.532 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.526+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3442279929}) 2024-09-16T04:05:16.532 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.526+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.533 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.526+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=970186530}) 2024-09-16T04:05:16.533 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.526+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3714921515}) 2024-09-16T04:05:16.534 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.530+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.534 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.530+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1095298814}) 2024-09-16T04:05:16.534 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.530+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=535444189}) 2024-09-16T04:05:16.534 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.530+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.534 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.530+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1095298814}) 2024-09-16T04:05:16.535 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.530+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1127799669}) 2024-09-16T04:05:16.536 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.530+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.536 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.530+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2158935755}) 2024-09-16T04:05:16.536 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.530+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2261399962}) 2024-09-16T04:05:16.536 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.530+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.536 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.530+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2158935755}) 2024-09-16T04:05:16.536 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.530+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=914257015}) 2024-09-16T04:05:16.538 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.534+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.538 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.534+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=550576843}) 2024-09-16T04:05:16.538 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.534+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3241394482}) 2024-09-16T04:05:16.538 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.534+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.538 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.534+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=550576843}) 2024-09-16T04:05:16.539 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.534+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3597014900}) 2024-09-16T04:05:16.540 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.534+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.541 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.534+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1153103071}) 2024-09-16T04:05:16.541 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.534+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2583978294}) 2024-09-16T04:05:16.544 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.534+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.544 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.534+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1153103071}) 2024-09-16T04:05:16.544 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.534+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1066414280}) 2024-09-16T04:05:16.545 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.538+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.546 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.538+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3845174304}) 2024-09-16T04:05:16.546 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.538+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2692502372}) 2024-09-16T04:05:16.546 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.538+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.546 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.538+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3845174304}) 2024-09-16T04:05:16.546 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.538+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2240904661}) 2024-09-16T04:05:16.547 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.538+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.547 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.538+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1752731483}) 2024-09-16T04:05:16.547 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.538+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3769475392}) 2024-09-16T04:05:16.547 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.538+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.548 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.542+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1752731483}) 2024-09-16T04:05:16.548 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.542+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2776958514}) 2024-09-16T04:05:16.549 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.542+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.549 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.542+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1562419390}) 2024-09-16T04:05:16.549 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.542+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2027641398}) 2024-09-16T04:05:16.549 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.542+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.549 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.542+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1562419390}) 2024-09-16T04:05:16.550 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.542+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4106851047}) 2024-09-16T04:05:16.550 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.546+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.550 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.546+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3551900781}) 2024-09-16T04:05:16.550 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.546+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=128384480}) 2024-09-16T04:05:16.550 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.546+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.551 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.546+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3551900781}) 2024-09-16T04:05:16.551 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.546+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2152112750}) 2024-09-16T04:05:16.552 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.546+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.552 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.546+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3377432750}) 2024-09-16T04:05:16.552 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.546+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=675582448}) 2024-09-16T04:05:16.552 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.546+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.552 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.546+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3377432750}) 2024-09-16T04:05:16.552 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.546+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=491446050}) 2024-09-16T04:05:16.554 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.550+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.555 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.550+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3612160772}) 2024-09-16T04:05:16.555 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.550+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1955055672}) 2024-09-16T04:05:16.555 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.550+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.555 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.550+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3612160772}) 2024-09-16T04:05:16.555 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.550+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=722236346}) 2024-09-16T04:05:16.557 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.550+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.557 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.550+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2081272780}) 2024-09-16T04:05:16.557 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.550+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2625746672}) 2024-09-16T04:05:16.557 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.550+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.557 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.550+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2081272780}) 2024-09-16T04:05:16.557 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.550+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=926673908}) 2024-09-16T04:05:16.559 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.554+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.559 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.554+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1836437737}) 2024-09-16T04:05:16.559 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.554+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2223990375}) 2024-09-16T04:05:16.559 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.554+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.559 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.554+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1836437737}) 2024-09-16T04:05:16.559 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.554+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4230605692}) 2024-09-16T04:05:16.561 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.554+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.561 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.554+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2424281976}) 2024-09-16T04:05:16.561 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.554+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1157658415}) 2024-09-16T04:05:16.561 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.554+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.561 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.554+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2424281976}) 2024-09-16T04:05:16.562 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.554+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=736993662}) 2024-09-16T04:05:16.563 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.558+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.563 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.558+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=679135853}) 2024-09-16T04:05:16.564 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.558+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=365864210}) 2024-09-16T04:05:16.564 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.558+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.564 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.558+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=679135853}) 2024-09-16T04:05:16.564 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.558+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2142671255}) 2024-09-16T04:05:16.566 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.562+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.566 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.562+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=605346764}) 2024-09-16T04:05:16.566 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.562+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2573778536}) 2024-09-16T04:05:16.566 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.562+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.566 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.562+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=605346764}) 2024-09-16T04:05:16.566 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.562+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2352675305}) 2024-09-16T04:05:16.568 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.562+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.568 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.562+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3106535212}) 2024-09-16T04:05:16.568 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.562+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1676665584}) 2024-09-16T04:05:16.568 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.562+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.568 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.562+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3106535212}) 2024-09-16T04:05:16.569 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.562+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3287800868}) 2024-09-16T04:05:16.571 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.566+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.571 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.566+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2162242142}) 2024-09-16T04:05:16.571 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.566+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1272769458}) 2024-09-16T04:05:16.571 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.566+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.571 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.566+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2162242142}) 2024-09-16T04:05:16.571 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.566+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4129923893}) 2024-09-16T04:05:16.573 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.566+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.573 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.566+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3642161958}) 2024-09-16T04:05:16.573 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.566+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3167290882}) 2024-09-16T04:05:16.573 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.566+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.573 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.566+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3642161958}) 2024-09-16T04:05:16.573 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.566+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3078538252}) 2024-09-16T04:05:16.576 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.570+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.576 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.570+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2607381062}) 2024-09-16T04:05:16.576 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.570+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=511474637}) 2024-09-16T04:05:16.576 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.570+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.576 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.570+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2607381062}) 2024-09-16T04:05:16.576 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.570+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2842026318}) 2024-09-16T04:05:16.578 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.574+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.578 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.574+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1452064599}) 2024-09-16T04:05:16.578 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.574+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1990522023}) 2024-09-16T04:05:16.578 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.574+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.578 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.574+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1452064599}) 2024-09-16T04:05:16.579 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.574+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1100665868}) 2024-09-16T04:05:16.580 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.574+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.580 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.574+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1199605304}) 2024-09-16T04:05:16.580 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.574+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3854966759}) 2024-09-16T04:05:16.580 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.574+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.580 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.574+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1199605304}) 2024-09-16T04:05:16.581 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.574+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1376571264}) 2024-09-16T04:05:16.582 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.578+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.582 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.578+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4232441365}) 2024-09-16T04:05:16.582 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.578+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=610675086}) 2024-09-16T04:05:16.582 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.578+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.583 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.578+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4232441365}) 2024-09-16T04:05:16.583 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.578+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2689963793}) 2024-09-16T04:05:16.584 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.578+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.584 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.578+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4243473633}) 2024-09-16T04:05:16.584 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.578+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=5592026}) 2024-09-16T04:05:16.585 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.578+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.585 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.578+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4243473633}) 2024-09-16T04:05:16.585 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.578+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1033416116}) 2024-09-16T04:05:16.586 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.582+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.587 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.582+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2386869703}) 2024-09-16T04:05:16.587 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.582+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1789506231}) 2024-09-16T04:05:16.587 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.582+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.587 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.582+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2386869703}) 2024-09-16T04:05:16.587 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.582+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3870295658}) 2024-09-16T04:05:16.589 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.582+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.590 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.582+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2445487079}) 2024-09-16T04:05:16.590 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.582+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=176864120}) 2024-09-16T04:05:16.590 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.582+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.590 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.582+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2445487079}) 2024-09-16T04:05:16.590 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.582+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3099366281}) 2024-09-16T04:05:16.592 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.586+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.592 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.586+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3260102383}) 2024-09-16T04:05:16.592 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.586+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1890967760}) 2024-09-16T04:05:16.592 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.586+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.592 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.586+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3260102383}) 2024-09-16T04:05:16.592 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.586+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=559995514}) 2024-09-16T04:05:16.594 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.590+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.594 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.590+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=673992284}) 2024-09-16T04:05:16.594 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.590+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2532831345}) 2024-09-16T04:05:16.594 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.590+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.594 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.590+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=673992284}) 2024-09-16T04:05:16.594 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.590+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=20127963}) 2024-09-16T04:05:16.596 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.590+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.596 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.590+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1990087500}) 2024-09-16T04:05:16.596 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.590+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4281504834}) 2024-09-16T04:05:16.596 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.590+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.597 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.590+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1990087500}) 2024-09-16T04:05:16.597 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.590+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4204122026}) 2024-09-16T04:05:16.598 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.594+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.598 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.594+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1828934543}) 2024-09-16T04:05:16.598 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.594+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=106732315}) 2024-09-16T04:05:16.598 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.594+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.598 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.594+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1828934543}) 2024-09-16T04:05:16.599 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.594+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3098275200}) 2024-09-16T04:05:16.601 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.594+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.601 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.594+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3074505163}) 2024-09-16T04:05:16.601 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.594+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2785133279}) 2024-09-16T04:05:16.601 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.594+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.601 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.594+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3074505163}) 2024-09-16T04:05:16.601 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.594+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3776856431}) 2024-09-16T04:05:16.603 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.598+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.603 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.598+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2480042650}) 2024-09-16T04:05:16.603 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.598+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3270802473}) 2024-09-16T04:05:16.603 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.598+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.603 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.598+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2480042650}) 2024-09-16T04:05:16.603 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.598+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2999319293}) 2024-09-16T04:05:16.605 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.598+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.606 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.598+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=34249773}) 2024-09-16T04:05:16.606 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.598+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=643975479}) 2024-09-16T04:05:16.606 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.598+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.606 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.598+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=34249773}) 2024-09-16T04:05:16.606 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.598+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4229768192}) 2024-09-16T04:05:16.608 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.602+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.608 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.602+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1843969953}) 2024-09-16T04:05:16.608 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.602+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=97,mdsmap=2,mgr=1} crc {logm=3821389335,mdsmap=210808725,mgr=3189685207}) 2024-09-16T04:05:16.608 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.602+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.608 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.602+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1843969953}) 2024-09-16T04:05:16.608 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.602+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2313374520}) 2024-09-16T04:05:16.611 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.606+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.611 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.606+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mdsmap=2,mgr=6,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=11,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=39,osdmap=20} crc {mdsmap=210808725,mgr=3983147178,mgr_command_descs=2107673600,mgr_metadata=1367272202,mgrstat=1613245471,mon_config_key=85057112,monmap=3937513982,osd_metadata=3450683029,osd_pg_creating=634549719,osd_snap=3520709557,osdmap=559579415}) 2024-09-16T04:05:16.611 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.606+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mdsmap=3,mgr=6,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=11,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=40,osdmap=20} crc {mdsmap=4088524775,mgr=566604210,mgr_command_descs=2107673600,mgr_metadata=1367272202,mgrstat=820730131,mon_config_key=85057112,monmap=3937513981,osd_metadata=498355664,osd_pg_creating=634549719,osd_snap=1687080213,osdmap=1717577178}) 2024-09-16T04:05:16.611 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.606+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.611 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.606+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mdsmap=2,mgr=6,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=11,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=39,osdmap=20} crc {mdsmap=210808725,mgr=3983147178,mgr_command_descs=2107673600,mgr_metadata=1367272202,mgrstat=1613245471,mon_config_key=85057112,monmap=3937513982,osd_metadata=3450683029,osd_pg_creating=634549719,osd_snap=3520709557,osdmap=559579415}) 2024-09-16T04:05:16.611 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.606+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mdsmap=3,mgr=6,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=10,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=43,osdmap=15} crc {mdsmap=4088524775,mgr=3699029739,mgr_command_descs=2107673600,mgr_metadata=1367272202,mgrstat=1613245471,mon_config_key=3969788172,monmap=3937513981,osd_metadata=4182959540,osd_pg_creating=634549719,osd_snap=1600545520,osdmap=3446153427}) 2024-09-16T04:05:16.613 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.606+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.613 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.606+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1432978466}) 2024-09-16T04:05:16.613 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.606+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2491129478}) 2024-09-16T04:05:16.613 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.606+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.613 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.606+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1432978466}) 2024-09-16T04:05:16.613 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.606+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=965357283}) 2024-09-16T04:05:16.615 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.610+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.615 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.610+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2935329197}) 2024-09-16T04:05:16.616 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.610+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=364399951}) 2024-09-16T04:05:16.616 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.610+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.616 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.610+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2935329197}) 2024-09-16T04:05:16.616 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.610+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=115156239}) 2024-09-16T04:05:16.620 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.614+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.620 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.614+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1191403509}) 2024-09-16T04:05:16.620 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.614+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1528472760}) 2024-09-16T04:05:16.620 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.614+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.620 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.614+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1191403509}) 2024-09-16T04:05:16.621 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.614+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3540385560}) 2024-09-16T04:05:16.638 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.630+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.638 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.630+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=59} crc {osdmap=1988785843}) 2024-09-16T04:05:16.638 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.630+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=59} crc {osdmap=253175424}) 2024-09-16T04:05:16.638 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.630+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:05:16.638 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.630+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=59} crc {osdmap=1988785843}) 2024-09-16T04:05:16.639 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:05:16.630+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=54} crc {osdmap=3700320937}) 2024-09-16T04:05:17.350 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T04:05:17.350 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:05:17.483 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.478+0000 7f743a6a7640 10 monclient: get_monmap_and_config 2024-09-16T04:05:17.483 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.478+0000 7f743a6a7640 10 monclient: build_initial_monmap 2024-09-16T04:05:17.483 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.478+0000 7f743a6a7640 10 monclient: monmap: 2024-09-16T04:05:17.484 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:05:17.484 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:05:17.484 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:05:17.481927+0000 2024-09-16T04:05:17.484 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:05:17.481927+0000 2024-09-16T04:05:17.484 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:05:17.484 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:05:17.484 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:05:17.484 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:05:17.484 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:05:17.484 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:05:17.484 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:05:17.485 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:05:17.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.478+0000 7f743a6a7640 1 Processor -- start 2024-09-16T04:05:17.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.478+0000 7f743a6a7640 1 -- start start 2024-09-16T04:05:17.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.478+0000 7f743a6a7640 10 monclient: init 2024-09-16T04:05:17.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.478+0000 7f743a6a7640 10 monclient: _reopen_session rank -1 2024-09-16T04:05:17.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.478+0000 7f743a6a7640 10 monclient: _add_conns ranks=[4,3,1,0,2] 2024-09-16T04:05:17.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.478+0000 7f743a6a7640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f74340a1330 0x7f743409f180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:05:17.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.478+0000 7f743a6a7640 10 monclient(hunting): picked mon.noname-c con 0x7f74340a1330 addr v2:172.21.15.154:3300/0 2024-09-16T04:05:17.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.478+0000 7f743a6a7640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f74340a1710 0x7f74340a1b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:05:17.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.478+0000 7f743a6a7640 10 monclient(hunting): picked mon.noname-a con 0x7f74340a1710 addr v2:172.21.15.42:3300/0 2024-09-16T04:05:17.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.478+0000 7f743a6a7640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f74341340f0 addr v1:172.21.15.154:6789/0 2024-09-16T04:05:17.488 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.478+0000 7f743a6a7640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f743409e4c0 con 0x7f74341340f0 2024-09-16T04:05:17.488 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.478+0000 7f743a6a7640 10 monclient(hunting): start opening mon connection 2024-09-16T04:05:17.488 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.478+0000 7f743a6a7640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f743409edf0 con 0x7f74340a1710 2024-09-16T04:05:17.488 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.478+0000 7f743a6a7640 10 monclient(hunting): start opening mon connection 2024-09-16T04:05:17.488 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.478+0000 7f743a6a7640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f743409cd60 con 0x7f74340a1330 2024-09-16T04:05:17.488 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.478+0000 7f743a6a7640 10 monclient(hunting): _renew_subs 2024-09-16T04:05:17.488 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.478+0000 7f743a6a7640 10 monclient(hunting): authenticate will time out at 2364.451617s 2024-09-16T04:05:17.488 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.478+0000 7f74337fe640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f74340a1710 0x7f74340a1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:05:17.488 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.478+0000 7f74337fe640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f74340a1710 0x7f74340a1b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:47532/0 (socket says 172.21.15.42:47532) 2024-09-16T04:05:17.488 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.478+0000 7f74337fe640 1 -- 172.21.15.42:0/2573737832 learned_addr learned my addr 172.21.15.42:0/2573737832 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:05:17.488 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.478+0000 7f74337fe640 10 monclient(hunting): get_auth_request con 0x7f74340a1710 auth_method 0 2024-09-16T04:05:17.489 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.478+0000 7f74337fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:05:17.489 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.478+0000 7f74337fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:05:17.489 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.478+0000 7f74337fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:05:17.489 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.478+0000 7f74337fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:05:17.489 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.478+0000 7f74337fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:05:17.489 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.478+0000 7f74337fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:05:17.489 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.482+0000 7f74337fe640 10 monclient(hunting): handle_auth_done global_id 984276 payload 274 2024-09-16T04:05:17.489 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.482+0000 7f74337fe640 1 -- 172.21.15.42:0/2573737832 >> v2:172.21.15.154:3300/0 conn(0x7f74340a1330 msgr2=0x7f743409f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:05:17.489 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.482+0000 7f74337fe640 1 --2- 172.21.15.42:0/2573737832 >> v2:172.21.15.154:3300/0 conn(0x7f74340a1330 0x7f743409f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:17.489 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.482+0000 7f74337fe640 1 -- 172.21.15.42:0/2573737832 >> v1:172.21.15.154:6789/0 conn(0x7f74341340f0 legacy=0x7f7434134540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:05:17.489 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.482+0000 7f74337fe640 10 monclient: _finish_hunting 0 2024-09-16T04:05:17.490 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.482+0000 7f74337fe640 1 monclient: found mon.noname-a 2024-09-16T04:05:17.490 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.482+0000 7f74337fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:05:17.490 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.482+0000 7f74337fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:05:17.490 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.482+0000 7f74337fe640 1 -- 172.21.15.42:0/2573737832 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74340a2080 con 0x7f74340a1710 2024-09-16T04:05:17.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.482+0000 7f74337fe640 1 --2- 172.21.15.42:0/2573737832 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f74340a1710 0x7f74340a1b40 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f743406add0 tx=0x7f741c01c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9fb01a79469ec77 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:05:17.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.482+0000 7f7432ffd640 1 -- 172.21.15.42:0/2573737832 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f741c0a56d0 con 0x7f74340a1710 2024-09-16T04:05:17.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.482+0000 7f7432ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:05:17.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.482+0000 7f7432ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:05:17.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.482+0000 7f7432ffd640 10 monclient: dump: 2024-09-16T04:05:17.497 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:05:17.497 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:05:17.497 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:17.497 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:17.497 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:05:17.497 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:05:17.497 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:05:17.497 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:05:17.497 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:05:17.497 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:05:17.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.482+0000 7f7432ffd640 10 monclient: _finish_auth 0 2024-09-16T04:05:17.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.482+0000 7f7432ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:05:17.498 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.482+0000 7f7432ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:05:17.498 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.482+0000 7f7432ffd640 1 -- 172.21.15.42:0/2573737832 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f741c0a5870 con 0x7f74340a1710 2024-09-16T04:05:17.498 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.482+0000 7f743a6a7640 5 monclient: authenticate success, global_id 984276 2024-09-16T04:05:17.498 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.482+0000 7f743a6a7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:05:17.498 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.482+0000 7f7432ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:05:17.498 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.482+0000 7f743a6a7640 10 monclient: get_monmap_and_config success 2024-09-16T04:05:17.498 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.482+0000 7f743a6a7640 4 set_mon_vals no callback set 2024-09-16T04:05:17.498 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.482+0000 7f7432ffd640 1 -- 172.21.15.42:0/2573737832 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f741c0a5c40 con 0x7f74340a1710 2024-09-16T04:05:17.498 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.482+0000 7f7432ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:05:17.498 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.482+0000 7f7432ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:05:17.498 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.482+0000 7f7432ffd640 10 monclient: dump: 2024-09-16T04:05:17.499 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:05:17.499 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:05:17.499 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:17.499 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:17.499 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:05:17.499 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:05:17.499 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:05:17.499 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:05:17.499 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:05:17.499 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:05:17.499 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.482+0000 7f743a6a7640 10 monclient: shutdown 2024-09-16T04:05:17.500 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.482+0000 7f743a6a7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:05:17.500 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.482+0000 7f743a6a7640 1 -- 172.21.15.42:0/2573737832 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f74340a1710 msgr2=0x7f74340a1b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:05:17.500 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.482+0000 7f743a6a7640 1 --2- 172.21.15.42:0/2573737832 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f74340a1710 0x7f74340a1b40 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f743406add0 tx=0x7f741c01c900 comp rx=0 tx=0).stop 2024-09-16T04:05:17.500 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.482+0000 7f743a6a7640 1 -- 172.21.15.42:0/2573737832 shutdown_connections 2024-09-16T04:05:17.500 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.482+0000 7f743a6a7640 1 --2- 172.21.15.42:0/2573737832 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f74340a1710 0x7f74340a1b40 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:17.500 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.482+0000 7f743a6a7640 1 --2- 172.21.15.42:0/2573737832 >> v2:172.21.15.154:3300/0 conn(0x7f74340a1330 0x7f743409f180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:17.500 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.482+0000 7f743a6a7640 1 -- 172.21.15.42:0/2573737832 >> 172.21.15.42:0/2573737832 conn(0x7f74340a2420 msgr2=0x7f74340a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:05:17.500 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.482+0000 7f743a6a7640 1 -- 172.21.15.42:0/2573737832 shutdown_connections 2024-09-16T04:05:17.500 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.482+0000 7f743a6a7640 1 -- 172.21.15.42:0/2573737832 wait complete. 2024-09-16T04:05:17.500 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.482+0000 7f743a6a7640 10 monclient: build_initial_monmap 2024-09-16T04:05:17.501 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.482+0000 7f743a6a7640 10 monclient: monmap: 2024-09-16T04:05:17.501 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:05:17.501 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:05:17.501 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:05:17.501 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:05:17.501 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:05:17.501 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:05:17.501 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:05:17.501 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:05:17.501 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:05:17.501 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:05:17.502 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.482+0000 7f743a6a7640 1 librados: starting msgr at 2024-09-16T04:05:17.502 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.482+0000 7f743a6a7640 1 librados: starting objecter 2024-09-16T04:05:17.502 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.482+0000 7f743a6a7640 1 Processor -- start 2024-09-16T04:05:17.502 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.486+0000 7f743a6a7640 1 -- start start 2024-09-16T04:05:17.502 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.486+0000 7f743a6a7640 1 librados: setting wanted keys 2024-09-16T04:05:17.502 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.486+0000 7f743a6a7640 1 librados: calling monclient init 2024-09-16T04:05:17.502 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.486+0000 7f743a6a7640 10 monclient: init 2024-09-16T04:05:17.502 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.486+0000 7f743a6a7640 10 monclient: _reopen_session rank -1 2024-09-16T04:05:17.502 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.486+0000 7f743a6a7640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:05:17.502 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.486+0000 7f743a6a7640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f74340a1330 0x7f74341cf330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:05:17.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.486+0000 7f743a6a7640 10 monclient(hunting): picked mon.noname-a con 0x7f74340a1330 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:05:17.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.486+0000 7f743a6a7640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f74340a1710 0x7f7434133530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:05:17.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.486+0000 7f743a6a7640 10 monclient(hunting): picked mon.noname-b con 0x7f74340a1710 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:05:17.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.486+0000 7f743a6a7640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f74341340f0 0x7f7434133a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:05:17.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.486+0000 7f743a6a7640 10 monclient(hunting): picked mon.noname-c con 0x7f74341340f0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:05:17.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.486+0000 7f743a6a7640 10 monclient(hunting): start opening mon connection 2024-09-16T04:05:17.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.486+0000 7f743a6a7640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f743413ed60 con 0x7f74340a1330 2024-09-16T04:05:17.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.486+0000 7f743a6a7640 10 monclient(hunting): start opening mon connection 2024-09-16T04:05:17.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.486+0000 7f743a6a7640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f743413ebe0 con 0x7f74340a1710 2024-09-16T04:05:17.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.486+0000 7f743a6a7640 10 monclient(hunting): start opening mon connection 2024-09-16T04:05:17.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.486+0000 7f743a6a7640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f743413eee0 con 0x7f74341340f0 2024-09-16T04:05:17.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.486+0000 7f743a6a7640 10 monclient(hunting): _renew_subs 2024-09-16T04:05:17.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.486+0000 7f743a6a7640 10 monclient(hunting): authenticate will time out at 2364.456521s 2024-09-16T04:05:17.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.486+0000 7f7433fff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f74340a1330 0x7f74341cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:05:17.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.486+0000 7f7433fff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f74340a1330 0x7f74341cf330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:47536/0 (socket says 172.21.15.42:47536) 2024-09-16T04:05:17.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.486+0000 7f7433fff640 1 -- 172.21.15.42:0/462636253 learned_addr learned my addr 172.21.15.42:0/462636253 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:05:17.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.486+0000 7f7433fff640 10 monclient(hunting): get_auth_request con 0x7f74340a1330 auth_method 0 2024-09-16T04:05:17.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.486+0000 7f7433fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:05:17.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.486+0000 7f7433fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:05:17.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.486+0000 7f7433fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:05:17.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.486+0000 7f7438c44640 1 --2- 172.21.15.42:0/462636253 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f74341340f0 0x7f7434133a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:05:17.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.486+0000 7f7438c44640 10 monclient(hunting): get_auth_request con 0x7f74341340f0 auth_method 0 2024-09-16T04:05:17.505 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.486+0000 7f7438c44640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:05:17.505 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.486+0000 7f7438c44640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:05:17.505 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.486+0000 7f7438c44640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:05:17.505 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.486+0000 7f7433fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:05:17.505 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.486+0000 7f7433fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:05:17.505 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.486+0000 7f7433fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:05:17.505 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.486+0000 7f7438c44640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:05:17.505 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.486+0000 7f7438c44640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:05:17.505 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.486+0000 7f7438c44640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:05:17.505 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.486+0000 7f74337fe640 1 --2- 172.21.15.42:0/462636253 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f74340a1710 0x7f7434133530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:05:17.505 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.486+0000 7f74337fe640 10 monclient(hunting): get_auth_request con 0x7f74340a1710 auth_method 0 2024-09-16T04:05:17.505 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.486+0000 7f74337fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:05:17.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.486+0000 7f74337fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:05:17.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.486+0000 7f74337fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:05:17.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.486+0000 7f7433fff640 10 monclient(hunting): handle_auth_done global_id 984279 payload 824 2024-09-16T04:05:17.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.486+0000 7f7433fff640 1 -- 172.21.15.42:0/462636253 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f74341340f0 msgr2=0x7f7434133a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:05:17.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.486+0000 7f7433fff640 1 --2- 172.21.15.42:0/462636253 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f74341340f0 0x7f7434133a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:17.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.486+0000 7f7433fff640 1 -- 172.21.15.42:0/462636253 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f74340a1710 msgr2=0x7f7434133530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:05:17.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.486+0000 7f7433fff640 1 --2- 172.21.15.42:0/462636253 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f74340a1710 0x7f7434133530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:17.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.486+0000 7f7433fff640 10 monclient: _finish_hunting 0 2024-09-16T04:05:17.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.486+0000 7f7433fff640 1 monclient: found mon.noname-a 2024-09-16T04:05:17.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.486+0000 7f7433fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:05:17.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.486+0000 7f7433fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:05:17.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.486+0000 7f7433fff640 1 -- 172.21.15.42:0/462636253 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74340a5030 con 0x7f74340a1330 2024-09-16T04:05:17.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.486+0000 7f7433fff640 1 --2- 172.21.15.42:0/462636253 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f74340a1330 0x7f74341cf330 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f742400b410 tx=0x7f742400b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:05:17.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.486+0000 7f7430ff9640 10 client.?.objecter ms_handle_connect 0x7f74340a1330 2024-09-16T04:05:17.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.486+0000 7f7430ff9640 10 client.?.objecter resend_mon_ops 2024-09-16T04:05:17.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.486+0000 7f7430ff9640 1 -- 172.21.15.42:0/462636253 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7424015020 con 0x7f74340a1330 2024-09-16T04:05:17.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.486+0000 7f7430ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:05:17.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.486+0000 7f7430ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:05:17.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.486+0000 7f7430ff9640 10 monclient: dump: 2024-09-16T04:05:17.507 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:05:17.507 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:05:17.508 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:17.508 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:17.508 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:05:17.508 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:05:17.508 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:05:17.508 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:05:17.508 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:05:17.508 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:05:17.508 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.486+0000 7f7430ff9640 10 monclient: _finish_auth 0 2024-09-16T04:05:17.508 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.486+0000 7f7430ff9640 10 monclient: _check_auth_tickets 2024-09-16T04:05:17.508 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.486+0000 7f7430ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:05:17.509 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.486+0000 7f7430ff9640 1 -- 172.21.15.42:0/462636253 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7424011830 con 0x7f74340a1330 2024-09-16T04:05:17.509 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.486+0000 7f7430ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T04:05:17.509 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.486+0000 7f7430ff9640 1 -- 172.21.15.42:0/462636253 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7424011c20 con 0x7f74340a1330 2024-09-16T04:05:17.509 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.486+0000 7f743a6a7640 5 monclient: authenticate success, global_id 984279 2024-09-16T04:05:17.509 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.486+0000 7f743a6a7640 10 monclient: _renew_subs 2024-09-16T04:05:17.509 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.486+0000 7f743a6a7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:05:17.509 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.486+0000 7f743a6a7640 1 -- 172.21.15.42:0/462636253 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74341cf870 con 0x7f74340a1330 2024-09-16T04:05:17.509 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.486+0000 7f7430ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:05:17.509 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.486+0000 7f7430ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:05:17.509 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.486+0000 7f7430ff9640 10 monclient: dump: 2024-09-16T04:05:17.509 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:05:17.510 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:05:17.510 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:17.510 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:17.510 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:05:17.510 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:05:17.510 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:05:17.510 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:05:17.510 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:05:17.510 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:05:17.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.486+0000 7f74327fc640 4 set_mon_vals no callback set 2024-09-16T04:05:17.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.486+0000 7f743a6a7640 10 client.984279.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:05:17.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.486+0000 7f743a6a7640 10 monclient: _renew_subs 2024-09-16T04:05:17.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.486+0000 7f743a6a7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:05:17.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.486+0000 7f743a6a7640 1 -- 172.21.15.42:0/462636253 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74340724f0 con 0x7f74340a1330 2024-09-16T04:05:17.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.486+0000 7f743a6a7640 1 librados: init done 2024-09-16T04:05:17.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.486+0000 7f743a6a7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:05:17.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.486+0000 7f743a6a7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:05:17.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.486+0000 7f743a6a7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:05:17.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.486+0000 7f743a6a7640 1 -- 172.21.15.42:0/462636253 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f743409f180 con 0x7f74340a1330 2024-09-16T04:05:17.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.486+0000 7f7430ff9640 1 -- 172.21.15.42:0/462636253 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f7424018020 con 0x7f74340a1330 2024-09-16T04:05:17.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.486+0000 7f7430ff9640 1 --2- 172.21.15.42:0/462636253 >> v2:172.21.15.154:6812/254845342 conn(0x7f7418037810 0x7f7418039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:05:17.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.486+0000 7f7430ff9640 1 -- 172.21.15.42:0/462636253 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(705..705 src has 451..705) ==== 5150+0+0 (secure 0 0 0) 0x7f7424048b40 con 0x7f74340a1330 2024-09-16T04:05:17.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.486+0000 7f7430ff9640 10 client.984279.objecter ms_dispatch 0x7f7434000ba0 osd_map(705..705 src has 451..705) 2024-09-16T04:05:17.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.486+0000 7f7430ff9640 3 client.984279.objecter handle_osd_map got epochs [705,705] > 0 2024-09-16T04:05:17.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.486+0000 7f7430ff9640 3 client.984279.objecter handle_osd_map decoding full epoch 705 2024-09-16T04:05:17.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.486+0000 7f7430ff9640 20 client.984279.objecter dump_active .. 0 homeless 2024-09-16T04:05:17.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.486+0000 7f74337fe640 1 --2- 172.21.15.42:0/462636253 >> v2:172.21.15.154:6812/254845342 conn(0x7f7418037810 0x7f7418039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:05:17.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.486+0000 7f74337fe640 10 monclient: get_auth_request con 0x7f7418037810 auth_method 0 2024-09-16T04:05:17.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.486+0000 7f7430ff9640 10 client.984279.objecter ms_handle_connect 0x7f7418037810 2024-09-16T04:05:17.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.486+0000 7f74337fe640 1 --2- 172.21.15.42:0/462636253 >> v2:172.21.15.154:6812/254845342 conn(0x7f7418037810 0x7f7418039cd0 secure :-1 s=READY pgs=3844 cs=0 l=1 rev1=1 crypto rx=0x7f741c0ae000 tx=0x7f741c0a7070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:05:17.513 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.490+0000 7f7430ff9640 1 -- 172.21.15.42:0/462636253 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f742401f020 con 0x7f74340a1330 2024-09-16T04:05:17.513 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.490+0000 7f7430ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:05:17.513 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.490+0000 7f7430ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:05:17.720 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.714+0000 7f743a6a7640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:05:17.720 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.714+0000 7f743a6a7640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:05:17.720 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.714+0000 7f743a6a7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:05:17.720 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.714+0000 7f743a6a7640 1 -- 172.21.15.42:0/462636253 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f7434069040 con 0x7f74340a1330 2024-09-16T04:05:17.721 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.714+0000 7f7430ff9640 1 -- 172.21.15.42:0/462636253 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7f74340a5030 con 0x7f74340a1330 2024-09-16T04:05:17.721 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.714+0000 7f7430ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:05:17.721 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.714+0000 7f7430ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:05:17.721 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":216,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:05:17.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.718+0000 7f743a6a7640 10 librados: watch_flush enter 2024-09-16T04:05:17.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.718+0000 7f743a6a7640 10 librados: watch_flush exit 2024-09-16T04:05:17.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.718+0000 7f743a6a7640 20 client.984279.objecter shutdown clearing up homeless session... 2024-09-16T04:05:17.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.718+0000 7f743a6a7640 10 client.984279.objecter successfully canceled tick 2024-09-16T04:05:17.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.718+0000 7f743a6a7640 1 -- 172.21.15.42:0/462636253 >> v2:172.21.15.154:6812/254845342 conn(0x7f7418037810 msgr2=0x7f7418039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:05:17.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.718+0000 7f743a6a7640 1 --2- 172.21.15.42:0/462636253 >> v2:172.21.15.154:6812/254845342 conn(0x7f7418037810 0x7f7418039cd0 secure :-1 s=READY pgs=3844 cs=0 l=1 rev1=1 crypto rx=0x7f741c0ae000 tx=0x7f741c0a7070 comp rx=0 tx=0).stop 2024-09-16T04:05:17.724 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.718+0000 7f743a6a7640 10 monclient: shutdown 2024-09-16T04:05:17.724 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.718+0000 7f743a6a7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:05:17.724 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.718+0000 7f743a6a7640 1 -- 172.21.15.42:0/462636253 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f74340a1330 msgr2=0x7f74341cf330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:05:17.724 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.718+0000 7f743a6a7640 1 --2- 172.21.15.42:0/462636253 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f74340a1330 0x7f74341cf330 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f742400b410 tx=0x7f742400b8d0 comp rx=0 tx=0).stop 2024-09-16T04:05:17.724 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.718+0000 7f743a6a7640 1 -- 172.21.15.42:0/462636253 shutdown_connections 2024-09-16T04:05:17.724 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.718+0000 7f743a6a7640 1 --2- 172.21.15.42:0/462636253 >> v2:172.21.15.154:6812/254845342 conn(0x7f7418037810 0x7f7418039cd0 unknown :-1 s=CLOSED pgs=3844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:17.724 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.718+0000 7f743a6a7640 1 --2- 172.21.15.42:0/462636253 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f74341340f0 0x7f7434133a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:17.724 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.718+0000 7f743a6a7640 1 --2- 172.21.15.42:0/462636253 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f74340a1710 0x7f7434133530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:17.724 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.718+0000 7f743a6a7640 1 --2- 172.21.15.42:0/462636253 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f74340a1330 0x7f74341cf330 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:17.724 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.718+0000 7f743a6a7640 1 -- 172.21.15.42:0/462636253 >> 172.21.15.42:0/462636253 conn(0x7f74340a2420 msgr2=0x7f74340a3800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:05:17.724 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.718+0000 7f743a6a7640 1 -- 172.21.15.42:0/462636253 shutdown_connections 2024-09-16T04:05:17.725 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.718+0000 7f743a6a7640 1 -- 172.21.15.42:0/462636253 wait complete. 2024-09-16T04:05:17.725 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.718+0000 7f743a6a7640 1 librados: shutdown 2024-09-16T04:05:17.740 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T04:05:17.740 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T04:05:17.740 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T04:05:17.741 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T04:05:17.859 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.854+0000 7fdffcaea640 10 monclient: get_monmap_and_config 2024-09-16T04:05:17.859 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.854+0000 7fdffcaea640 10 monclient: build_initial_monmap 2024-09-16T04:05:17.859 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.854+0000 7fdffcaea640 10 monclient: monmap: 2024-09-16T04:05:17.859 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:05:17.859 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:05:17.859 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:05:17.857216+0000 2024-09-16T04:05:17.859 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:05:17.857216+0000 2024-09-16T04:05:17.859 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:05:17.859 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:05:17.859 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:05:17.860 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:05:17.860 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:05:17.860 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:05:17.860 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:05:17.860 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:05:17.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.854+0000 7fdffcaea640 1 Processor -- start 2024-09-16T04:05:17.871 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.854+0000 7fdffcaea640 1 -- start start 2024-09-16T04:05:17.871 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.854+0000 7fdffcaea640 10 monclient: init 2024-09-16T04:05:17.871 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.854+0000 7fdffcaea640 10 monclient: _reopen_session rank -1 2024-09-16T04:05:17.871 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.854+0000 7fdffcaea640 10 monclient: _add_conns ranks=[1,4,0,3,2] 2024-09-16T04:05:17.871 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.854+0000 7fdffcaea640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fdff806db90 addr v1:172.21.15.154:6789/0 2024-09-16T04:05:17.871 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.854+0000 7fdffcaea640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fdff81340c0 0x7fdff813c190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:05:17.871 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.854+0000 7fdffcaea640 10 monclient(hunting): picked mon.noname-c con 0x7fdff81340c0 addr v2:172.21.15.154:3300/0 2024-09-16T04:05:17.871 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.854+0000 7fdffcaea640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fdff813c6d0 addr v1:172.21.15.42:6789/0 2024-09-16T04:05:17.871 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.854+0000 7fdffcaea640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdff806ece0 con 0x7fdff813c6d0 2024-09-16T04:05:17.871 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.854+0000 7fdffcaea640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdff809bf80 con 0x7fdff806db90 2024-09-16T04:05:17.871 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.854+0000 7fdffcaea640 10 monclient(hunting): start opening mon connection 2024-09-16T04:05:17.871 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.854+0000 7fdffcaea640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fdff806df70 con 0x7fdff81340c0 2024-09-16T04:05:17.871 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.854+0000 7fdffcaea640 10 monclient(hunting): _renew_subs 2024-09-16T04:05:17.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.854+0000 7fdffcaea640 10 monclient(hunting): authenticate will time out at 2364.827147s 2024-09-16T04:05:17.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.854+0000 7fdff6d9d640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fdff813c6d0 0x7fdff813eac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:40580/0 (socket says 172.21.15.42:40580) 2024-09-16T04:05:17.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.854+0000 7fdff6d9d640 1 -- 172.21.15.42:0/4103113180 learned_addr learned my addr 172.21.15.42:0/4103113180 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T04:05:17.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.854+0000 7fdff559a640 1 -- 172.21.15.42:0/4103113180 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3424979775 0 0) 0x7fdff806ece0 con 0x7fdff813c6d0 2024-09-16T04:05:17.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.854+0000 7fdff559a640 10 monclient(hunting): _negotiate 2024-09-16T04:05:17.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.854+0000 7fdff559a640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:05:17.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.854+0000 7fdff559a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:05:17.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.854+0000 7fdff559a640 10 monclient(hunting): my global_id is 984282 2024-09-16T04:05:17.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.854+0000 7fdff5d9b640 1 --2- 172.21.15.42:0/4103113180 >> v2:172.21.15.154:3300/0 conn(0x7fdff81340c0 0x7fdff813c190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:05:17.873 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.854+0000 7fdff559a640 1 -- 172.21.15.42:0/4103113180 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdfe0003530 con 0x7fdff813c6d0 2024-09-16T04:05:17.873 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.854+0000 7fdff5d9b640 10 monclient(hunting): get_auth_request con 0x7fdff81340c0 auth_method 0 2024-09-16T04:05:17.873 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.854+0000 7fdff5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:05:17.873 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.854+0000 7fdff5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:05:17.873 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.854+0000 7fdff5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:05:17.873 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.854+0000 7fdff559a640 1 -- 172.21.15.42:0/4103113180 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 114434845 0 0) 0x7fdfe0003530 con 0x7fdff813c6d0 2024-09-16T04:05:17.873 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.854+0000 7fdff559a640 1 -- 172.21.15.42:0/4103113180 >> v2:172.21.15.154:3300/0 conn(0x7fdff81340c0 msgr2=0x7fdff813c190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:05:17.874 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.854+0000 7fdff559a640 1 --2- 172.21.15.42:0/4103113180 >> v2:172.21.15.154:3300/0 conn(0x7fdff81340c0 0x7fdff813c190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:17.874 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.854+0000 7fdff559a640 1 -- 172.21.15.42:0/4103113180 >> v1:172.21.15.154:6789/0 conn(0x7fdff806db90 legacy=0x7fdff80a37f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:05:17.874 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.854+0000 7fdff559a640 10 monclient: _finish_hunting 0 2024-09-16T04:05:17.874 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.854+0000 7fdff559a640 1 monclient: found mon.noname-a-legacy 2024-09-16T04:05:17.874 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.854+0000 7fdff559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:05:17.874 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.854+0000 7fdff559a640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.42:6789/0 2024-09-16T04:05:17.874 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.854+0000 7fdff559a640 1 -- 172.21.15.42:0/4103113180 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdff81344d0 con 0x7fdff813c6d0 2024-09-16T04:05:17.874 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.854+0000 7fdff559a640 10 monclient: _finish_auth 0 2024-09-16T04:05:17.874 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.854+0000 7fdff559a640 10 monclient: _check_auth_tickets 2024-09-16T04:05:17.874 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.854+0000 7fdff559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:05:17.874 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.854+0000 7fdff559a640 1 -- 172.21.15.42:0/4103113180 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2177413734 0 0) 0x7fdfe8001680 con 0x7fdff813c6d0 2024-09-16T04:05:17.874 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.854+0000 7fdffcaea640 5 monclient: authenticate success, global_id 984282 2024-09-16T04:05:17.874 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.854+0000 7fdffcaea640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:05:17.875 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.854+0000 7fdff559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:05:17.875 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.854+0000 7fdff559a640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T04:05:17.875 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.858+0000 7fdff559a640 10 monclient: dump: 2024-09-16T04:05:17.875 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:05:17.875 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:05:17.875 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:17.875 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:17.875 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:05:17.875 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:05:17.875 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:05:17.875 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:05:17.875 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:05:17.875 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:05:17.876 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.858+0000 7fdff559a640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] but i'm connected to v1:172.21.15.42:6789/0, reconnecting 2024-09-16T04:05:17.876 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.858+0000 7fdff559a640 10 monclient: _reopen_session rank -1 2024-09-16T04:05:17.876 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.858+0000 7fdff559a640 1 -- 172.21.15.42:0/4103113180 >> v1:172.21.15.42:6789/0 conn(0x7fdff813c6d0 legacy=0x7fdff813eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:05:17.876 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.858+0000 7fdff559a640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:05:17.876 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.858+0000 7fdff559a640 1 --2- 172.21.15.42:0/4103113180 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdfe00039c0 0x7fdfe0003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:05:17.876 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.858+0000 7fdff559a640 10 monclient(hunting): picked mon.a con 0x7fdfe00039c0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:05:17.876 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.858+0000 7fdff559a640 1 --2- 172.21.15.42:0/4103113180 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fdfe0004810 0x7fdfe0004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:05:17.876 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.858+0000 7fdff559a640 10 monclient(hunting): picked mon.c con 0x7fdfe0004810 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:05:17.876 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.858+0000 7fdff559a640 1 --2- 172.21.15.42:0/4103113180 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdfe00053d0 0x7fdfe000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:05:17.876 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.858+0000 7fdff559a640 10 monclient(hunting): picked mon.b con 0x7fdfe00053d0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:05:17.876 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.858+0000 7fdff559a640 10 monclient(hunting): start opening mon connection 2024-09-16T04:05:17.876 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.858+0000 7fdff559a640 1 -- 172.21.15.42:0/4103113180 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fdff806df70 con 0x7fdfe00039c0 2024-09-16T04:05:17.876 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.858+0000 7fdff559a640 10 monclient(hunting): start opening mon connection 2024-09-16T04:05:17.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.858+0000 7fdff559a640 1 -- 172.21.15.42:0/4103113180 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fdfe000c260 con 0x7fdfe00053d0 2024-09-16T04:05:17.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.858+0000 7fdff559a640 10 monclient(hunting): start opening mon connection 2024-09-16T04:05:17.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.858+0000 7fdff559a640 1 -- 172.21.15.42:0/4103113180 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fdfe000c460 con 0x7fdfe0004810 2024-09-16T04:05:17.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.858+0000 7fdff559a640 10 monclient(hunting): _renew_subs 2024-09-16T04:05:17.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.858+0000 7fdff559a640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:05:17.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.858+0000 7fdffcaea640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-16T04:05:17.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.858+0000 7fdff659c640 1 --2- 172.21.15.42:0/4103113180 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdfe00039c0 0x7fdfe0003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:05:17.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.858+0000 7fdff5d9b640 1 --2- 172.21.15.42:0/4103113180 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fdfe0004810 0x7fdfe0004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:05:17.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.858+0000 7fdff659c640 10 monclient(hunting): get_auth_request con 0x7fdfe00039c0 auth_method 0 2024-09-16T04:05:17.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.858+0000 7fdff659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:05:17.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.858+0000 7fdff659c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:05:17.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.858+0000 7fdff659c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:05:17.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.858+0000 7fdff5d9b640 10 monclient(hunting): get_auth_request con 0x7fdfe0004810 auth_method 0 2024-09-16T04:05:17.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.858+0000 7fdff5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:05:17.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.858+0000 7fdff5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:05:17.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.858+0000 7fdff5d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:05:17.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.858+0000 7fdff659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:05:17.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.858+0000 7fdff659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:05:17.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.858+0000 7fdff659c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:05:17.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.858+0000 7fdff5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:05:17.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.858+0000 7fdff5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:05:17.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.858+0000 7fdff5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:05:17.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.858+0000 7fdff6d9d640 1 --2- 172.21.15.42:0/4103113180 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdfe00053d0 0x7fdfe000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:05:17.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.858+0000 7fdff6d9d640 10 monclient(hunting): get_auth_request con 0x7fdfe00053d0 auth_method 0 2024-09-16T04:05:17.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.858+0000 7fdff6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:05:17.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.858+0000 7fdff6d9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:05:17.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.858+0000 7fdff6d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:05:17.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.858+0000 7fdff659c640 10 monclient(hunting): handle_auth_done global_id 984282 payload 293 2024-09-16T04:05:17.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.858+0000 7fdff659c640 1 -- 172.21.15.42:0/4103113180 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fdfe0004810 msgr2=0x7fdfe0004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:05:17.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.858+0000 7fdff659c640 1 --2- 172.21.15.42:0/4103113180 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fdfe0004810 0x7fdfe0004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:17.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.858+0000 7fdff659c640 1 -- 172.21.15.42:0/4103113180 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdfe00053d0 msgr2=0x7fdfe000b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:05:17.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.858+0000 7fdff659c640 1 --2- 172.21.15.42:0/4103113180 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdfe00053d0 0x7fdfe000b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:17.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.858+0000 7fdff659c640 10 monclient: _finish_hunting 0 2024-09-16T04:05:17.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.858+0000 7fdff659c640 1 monclient: found mon.a 2024-09-16T04:05:17.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.858+0000 7fdff659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:05:17.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.858+0000 7fdff659c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:05:17.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.858+0000 7fdff659c640 1 -- 172.21.15.42:0/4103113180 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfe0003530 con 0x7fdfe00039c0 2024-09-16T04:05:17.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.858+0000 7fdff659c640 10 monclient: _finish_auth 0 2024-09-16T04:05:17.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.858+0000 7fdff659c640 10 monclient: _check_auth_tickets 2024-09-16T04:05:17.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.858+0000 7fdff659c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:05:17.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.858+0000 7fdff6d9d640 1 -- 172.21.15.42:0/4103113180 reap_dead start 2024-09-16T04:05:17.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.858+0000 7fdff5d9b640 1 --2- 172.21.15.42:0/4103113180 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fdfe0004810 0x7fdfe0004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:05:17.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.858+0000 7fdff659c640 1 --2- 172.21.15.42:0/4103113180 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdfe00039c0 0x7fdfe0003de0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fdff0001e70 tx=0x7fdff001cfc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2edebaff864fc32 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:05:17.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.858+0000 7fdff559a640 1 -- 172.21.15.42:0/4103113180 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdff0001a90 con 0x7fdfe00039c0 2024-09-16T04:05:17.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.858+0000 7fdff559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:05:17.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.858+0000 7fdff559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:05:17.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.858+0000 7fdff559a640 10 monclient: dump: 2024-09-16T04:05:17.880 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:05:17.880 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:05:17.881 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:17.881 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:17.881 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:05:17.881 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:05:17.881 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:05:17.881 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:05:17.881 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:05:17.882 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:05:17.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.858+0000 7fdffcaea640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:05:17.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.858+0000 7fdff559a640 1 -- 172.21.15.42:0/4103113180 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdff0003b80 con 0x7fdfe00039c0 2024-09-16T04:05:17.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.858+0000 7fdff559a640 10 monclient: handle_config config(0 keys) 2024-09-16T04:05:17.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.858+0000 7fdffcaea640 10 monclient: get_monmap_and_config success 2024-09-16T04:05:17.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.858+0000 7fdffcaea640 4 set_mon_vals no callback set 2024-09-16T04:05:17.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.858+0000 7fdff559a640 1 -- 172.21.15.42:0/4103113180 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdff00050d0 con 0x7fdfe00039c0 2024-09-16T04:05:17.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.858+0000 7fdff559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:05:17.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.858+0000 7fdff559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:05:17.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.858+0000 7fdff559a640 10 monclient: dump: 2024-09-16T04:05:17.882 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:05:17.882 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:05:17.883 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:17.883 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:17.883 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:05:17.883 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:05:17.883 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:05:17.883 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:05:17.883 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:05:17.883 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:05:17.883 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.858+0000 7fdffcaea640 10 monclient: shutdown 2024-09-16T04:05:17.883 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.858+0000 7fdffcaea640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:05:17.883 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.858+0000 7fdffcaea640 1 -- 172.21.15.42:0/4103113180 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdfe00039c0 msgr2=0x7fdfe0003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:05:17.883 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.858+0000 7fdffcaea640 1 --2- 172.21.15.42:0/4103113180 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdfe00039c0 0x7fdfe0003de0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fdff0001e70 tx=0x7fdff001cfc0 comp rx=0 tx=0).stop 2024-09-16T04:05:17.883 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.858+0000 7fdffcaea640 1 -- 172.21.15.42:0/4103113180 shutdown_connections 2024-09-16T04:05:17.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.858+0000 7fdffcaea640 1 --2- 172.21.15.42:0/4103113180 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdfe00039c0 0x7fdfe0003de0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:17.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.858+0000 7fdffcaea640 1 -- 172.21.15.42:0/4103113180 >> 172.21.15.42:0/4103113180 conn(0x7fdff80a9fb0 msgr2=0x7fdff80aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:05:17.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.858+0000 7fdffcaea640 1 -- 172.21.15.42:0/4103113180 shutdown_connections 2024-09-16T04:05:17.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.858+0000 7fdffcaea640 1 -- 172.21.15.42:0/4103113180 wait complete. 2024-09-16T04:05:17.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.862+0000 7fdffcaea640 10 monclient: build_initial_monmap 2024-09-16T04:05:17.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.862+0000 7fdffcaea640 10 monclient: monmap: 2024-09-16T04:05:17.884 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:05:17.884 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:05:17.884 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:05:17.884 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:05:17.884 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:05:17.884 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:05:17.884 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:05:17.885 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:05:17.885 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:05:17.885 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:05:17.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.862+0000 7fdffcaea640 1 librados: starting msgr at 2024-09-16T04:05:17.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.862+0000 7fdffcaea640 1 librados: starting objecter 2024-09-16T04:05:17.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.862+0000 7fdffcaea640 1 Processor -- start 2024-09-16T04:05:17.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.862+0000 7fdffcaea640 1 -- start start 2024-09-16T04:05:17.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.862+0000 7fdffcaea640 1 librados: setting wanted keys 2024-09-16T04:05:17.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.862+0000 7fdffcaea640 1 librados: calling monclient init 2024-09-16T04:05:17.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.862+0000 7fdffcaea640 10 monclient: init 2024-09-16T04:05:17.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.862+0000 7fdffcaea640 10 monclient: _reopen_session rank -1 2024-09-16T04:05:17.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.862+0000 7fdffcaea640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:05:17.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.862+0000 7fdffcaea640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fdfe00039c0 0x7fdff809fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:05:17.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.862+0000 7fdffcaea640 10 monclient(hunting): picked mon.noname-c con 0x7fdfe00039c0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:05:17.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.862+0000 7fdffcaea640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdff80a0100 0x7fdff80a0520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:05:17.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.862+0000 7fdffcaea640 10 monclient(hunting): picked mon.noname-b con 0x7fdff80a0100 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:05:17.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.862+0000 7fdffcaea640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdff80a0a60 0x7fdff81d58d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:05:17.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.862+0000 7fdffcaea640 10 monclient(hunting): picked mon.noname-a con 0x7fdff80a0a60 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:05:17.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.862+0000 7fdffcaea640 10 monclient(hunting): start opening mon connection 2024-09-16T04:05:17.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.862+0000 7fdffcaea640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fdff8137290 con 0x7fdff80a0a60 2024-09-16T04:05:17.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.862+0000 7fdffcaea640 10 monclient(hunting): start opening mon connection 2024-09-16T04:05:17.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.862+0000 7fdffcaea640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fdff8137110 con 0x7fdff80a0100 2024-09-16T04:05:17.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.862+0000 7fdffcaea640 10 monclient(hunting): start opening mon connection 2024-09-16T04:05:17.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.862+0000 7fdffcaea640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fdff8137410 con 0x7fdfe00039c0 2024-09-16T04:05:17.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.862+0000 7fdffcaea640 10 monclient(hunting): _renew_subs 2024-09-16T04:05:17.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.862+0000 7fdffcaea640 10 monclient(hunting): authenticate will time out at 2364.832977s 2024-09-16T04:05:17.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.862+0000 7fdff659c640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fdfe00039c0 0x7fdff809fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:05:17.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.862+0000 7fdff6d9d640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdff80a0a60 0x7fdff81d58d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:05:17.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.862+0000 7fdff659c640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fdfe00039c0 0x7fdff809fbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:39870/0 (socket says 172.21.15.42:39870) 2024-09-16T04:05:17.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.862+0000 7fdff6d9d640 10 monclient(hunting): get_auth_request con 0x7fdff80a0a60 auth_method 0 2024-09-16T04:05:17.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.862+0000 7fdff6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:05:17.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.862+0000 7fdff6d9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:05:17.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.862+0000 7fdff6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:05:17.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.862+0000 7fdff659c640 1 -- 172.21.15.42:0/2383215944 learned_addr learned my addr 172.21.15.42:0/2383215944 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:05:17.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.862+0000 7fdff659c640 10 monclient(hunting): get_auth_request con 0x7fdfe00039c0 auth_method 0 2024-09-16T04:05:17.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.862+0000 7fdff659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:05:17.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.862+0000 7fdff659c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:05:17.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.862+0000 7fdff659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:05:17.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.862+0000 7fdff6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:05:17.888 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.862+0000 7fdff6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:05:17.888 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.862+0000 7fdff6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:05:17.888 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.862+0000 7fdff659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:05:17.888 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.862+0000 7fdff659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:05:17.888 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.862+0000 7fdff659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:05:17.888 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.862+0000 7fdff5d9b640 1 --2- 172.21.15.42:0/2383215944 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdff80a0100 0x7fdff80a0520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:05:17.888 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.862+0000 7fdff5d9b640 10 monclient(hunting): get_auth_request con 0x7fdff80a0100 auth_method 0 2024-09-16T04:05:17.888 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.862+0000 7fdff5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:05:17.888 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.862+0000 7fdff5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:05:17.888 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.862+0000 7fdff5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:05:17.888 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.862+0000 7fdff659c640 10 monclient(hunting): handle_auth_done global_id 994115 payload 824 2024-09-16T04:05:17.888 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.862+0000 7fdff659c640 1 -- 172.21.15.42:0/2383215944 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdff80a0100 msgr2=0x7fdff80a0520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:05:17.888 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.862+0000 7fdff659c640 1 --2- 172.21.15.42:0/2383215944 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdff80a0100 0x7fdff80a0520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:17.889 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.862+0000 7fdff659c640 1 -- 172.21.15.42:0/2383215944 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdff80a0a60 msgr2=0x7fdff81d58d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:05:17.889 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.862+0000 7fdff659c640 1 --2- 172.21.15.42:0/2383215944 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdff80a0a60 0x7fdff81d58d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:17.889 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.862+0000 7fdff659c640 10 monclient: _finish_hunting 0 2024-09-16T04:05:17.889 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.862+0000 7fdff659c640 1 monclient: found mon.noname-c 2024-09-16T04:05:17.889 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.862+0000 7fdff659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:05:17.889 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.862+0000 7fdff659c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T04:05:17.889 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.862+0000 7fdff659c640 1 -- 172.21.15.42:0/2383215944 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdff81202a0 con 0x7fdfe00039c0 2024-09-16T04:05:17.889 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.862+0000 7fdff6d9d640 1 --2- 172.21.15.42:0/2383215944 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdff80a0a60 0x7fdff81d58d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:05:17.889 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.862+0000 7fdff659c640 1 --2- 172.21.15.42:0/2383215944 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fdfe00039c0 0x7fdff809fbc0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fdff0000f90 tx=0x7fdfe0005840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:05:17.889 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.862+0000 7fdfe6ffd640 10 client.?.objecter ms_handle_connect 0x7fdfe00039c0 2024-09-16T04:05:17.889 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.862+0000 7fdfe6ffd640 10 client.?.objecter resend_mon_ops 2024-09-16T04:05:17.889 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.862+0000 7fdfe6ffd640 1 -- 172.21.15.42:0/2383215944 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdff00bb070 con 0x7fdfe00039c0 2024-09-16T04:05:17.889 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.862+0000 7fdfe6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:05:17.890 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.862+0000 7fdfe6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:05:17.890 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.862+0000 7fdfe6ffd640 10 monclient: dump: 2024-09-16T04:05:17.890 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:05:17.890 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:05:17.890 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:17.890 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:17.890 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:05:17.890 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:05:17.890 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:05:17.890 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:05:17.890 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:05:17.890 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:05:17.890 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.862+0000 7fdfe6ffd640 10 monclient: _finish_auth 0 2024-09-16T04:05:17.890 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.862+0000 7fdfe6ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:05:17.891 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.862+0000 7fdfe6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:05:17.891 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.862+0000 7fdfe6ffd640 1 -- 172.21.15.42:0/2383215944 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdff0003980 con 0x7fdfe00039c0 2024-09-16T04:05:17.891 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.862+0000 7fdffcaea640 5 monclient: authenticate success, global_id 994115 2024-09-16T04:05:17.891 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.862+0000 7fdffcaea640 10 monclient: _renew_subs 2024-09-16T04:05:17.891 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.862+0000 7fdffcaea640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:05:17.891 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.862+0000 7fdffcaea640 1 -- 172.21.15.42:0/2383215944 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdff80a92f0 con 0x7fdfe00039c0 2024-09-16T04:05:17.891 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.862+0000 7fdfe6ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:05:17.891 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.862+0000 7fdfe6ffd640 1 -- 172.21.15.42:0/2383215944 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdff0001a90 con 0x7fdfe00039c0 2024-09-16T04:05:17.891 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.862+0000 7fdfe6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:05:17.891 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.862+0000 7fdfe6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:05:17.891 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.862+0000 7fdfe6ffd640 10 monclient: dump: 2024-09-16T04:05:17.891 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:05:17.891 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:05:17.891 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:17.892 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:17.892 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:05:17.892 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:05:17.892 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:05:17.892 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:05:17.892 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:05:17.892 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:05:17.892 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.862+0000 7fdffcaea640 10 client.994115.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:05:17.892 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.862+0000 7fdffcaea640 10 monclient: _renew_subs 2024-09-16T04:05:17.892 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.862+0000 7fdff4d99640 4 set_mon_vals no callback set 2024-09-16T04:05:17.892 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.862+0000 7fdffcaea640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:05:17.892 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.862+0000 7fdffcaea640 1 -- 172.21.15.42:0/2383215944 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fdff80a32b0 con 0x7fdfe00039c0 2024-09-16T04:05:17.893 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.862+0000 7fdffcaea640 1 librados: init done 2024-09-16T04:05:17.893 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.862+0000 7fdffcaea640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:05:17.893 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.862+0000 7fdffcaea640 1 --2- 172.21.15.42:0/2383215944 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdff813a180 0x7fdff813a560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:05:17.893 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.862+0000 7fdffcaea640 10 monclient(hunting): start opening mon connection 2024-09-16T04:05:17.893 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.862+0000 7fdffcaea640 1 -- 172.21.15.42:0/2383215944 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fdff80a25e0 con 0x7fdff813a180 2024-09-16T04:05:17.893 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.862+0000 7fdff5d9b640 1 --2- 172.21.15.42:0/2383215944 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdff813a180 0x7fdff813a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:05:17.893 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.862+0000 7fdff5d9b640 10 monclient: get_auth_request con 0x7fdff813a180 auth_method 0 2024-09-16T04:05:17.893 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.862+0000 7fdff5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:05:17.893 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.862+0000 7fdff5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:05:17.893 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.862+0000 7fdff5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:05:17.893 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.862+0000 7fdfe6ffd640 1 -- 172.21.15.42:0/2383215944 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fdff00be070 con 0x7fdfe00039c0 2024-09-16T04:05:17.893 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.862+0000 7fdff5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:05:17.893 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.862+0000 7fdff5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:05:17.893 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.862+0000 7fdff5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:05:17.894 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.862+0000 7fdff5d9b640 10 monclient(hunting): handle_auth_done global_id 984288 payload 824 2024-09-16T04:05:17.894 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.862+0000 7fdff5d9b640 1 -- 172.21.15.42:0/2383215944 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fdff806f680 con 0x7fdff813a180 2024-09-16T04:05:17.894 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.862+0000 7fdfe6ffd640 1 --2- 172.21.15.42:0/2383215944 >> v2:172.21.15.154:6812/254845342 conn(0x7fdfd8037840 0x7fdfd8039d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:05:17.894 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.862+0000 7fdfe6ffd640 1 -- 172.21.15.42:0/2383215944 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(705..705 src has 451..705) ==== 5150+0+0 (secure 0 0 0) 0x7fdff00e25e0 con 0x7fdfe00039c0 2024-09-16T04:05:17.894 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.862+0000 7fdfe6ffd640 10 client.994115.objecter ms_dispatch 0x7fdff8000ba0 osd_map(705..705 src has 451..705) 2024-09-16T04:05:17.894 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.862+0000 7fdfe6ffd640 3 client.994115.objecter handle_osd_map got epochs [705,705] > 0 2024-09-16T04:05:17.894 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.862+0000 7fdfe6ffd640 3 client.994115.objecter handle_osd_map decoding full epoch 705 2024-09-16T04:05:17.894 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.862+0000 7fdff5d9b640 1 --2- 172.21.15.42:0/2383215944 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdff813a180 0x7fdff813a560 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fdff80a1340 tx=0x7fdfec012d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:05:17.894 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.862+0000 7fdfe6ffd640 20 client.994115.objecter dump_active .. 0 homeless 2024-09-16T04:05:17.894 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.862+0000 7fdfe6ffd640 10 client.994115.objecter ms_handle_connect 0x7fdff813a180 2024-09-16T04:05:17.895 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.862+0000 7fdfe6ffd640 10 client.994115.objecter resend_mon_ops 2024-09-16T04:05:17.895 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.862+0000 7fdfe6ffd640 1 -- 172.21.15.42:0/2383215944 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfec01c070 con 0x7fdff813a180 2024-09-16T04:05:17.895 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.862+0000 7fdfe6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:05:17.895 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.862+0000 7fdfe6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:05:17.895 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.862+0000 7fdfe6ffd640 10 monclient: dump: 2024-09-16T04:05:17.895 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:05:17.895 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:05:17.895 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:17.895 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:17.895 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:05:17.895 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:05:17.895 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:05:17.895 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:05:17.895 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:05:17.896 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:05:17.896 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.862+0000 7fdff6d9d640 1 --2- 172.21.15.42:0/2383215944 >> v2:172.21.15.154:6812/254845342 conn(0x7fdfd8037840 0x7fdfd8039d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:05:17.896 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.862+0000 7fdfe6ffd640 1 -- 172.21.15.42:0/2383215944 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fdfec0109b0 con 0x7fdff813a180 2024-09-16T04:05:17.896 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.862+0000 7fdfe6ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:05:17.896 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.862+0000 7fdfe6ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:05:17.896 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.862+0000 7fdfe6ffd640 1 -- 172.21.15.42:0/2383215944 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdff813a180 msgr2=0x7fdff813a560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:05:17.896 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.862+0000 7fdfe6ffd640 1 --2- 172.21.15.42:0/2383215944 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdff813a180 0x7fdff813a560 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fdff80a1340 tx=0x7fdfec012d50 comp rx=0 tx=0).stop 2024-09-16T04:05:17.896 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.862+0000 7fdfe6ffd640 1 -- 172.21.15.42:0/2383215944 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdff813a180 msgr2=0x7fdff813a560 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:05:17.896 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.862+0000 7fdfe6ffd640 1 --2- 172.21.15.42:0/2383215944 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdff813a180 0x7fdff813a560 secure :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fdff80a1340 tx=0x7fdfec012d50 comp rx=0 tx=0).stop 2024-09-16T04:05:17.896 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.862+0000 7fdff6d9d640 10 monclient: get_auth_request con 0x7fdfd8037840 auth_method 0 2024-09-16T04:05:17.896 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.866+0000 7fdff6d9d640 1 --2- 172.21.15.42:0/2383215944 >> v2:172.21.15.154:6812/254845342 conn(0x7fdfd8037840 0x7fdfd8039d00 secure :-1 s=READY pgs=3845 cs=0 l=1 rev1=1 crypto rx=0x7fdff81344a0 tx=0x7fdfe8007040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:05:17.896 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.866+0000 7fdfe6ffd640 10 client.994115.objecter ms_handle_connect 0x7fdfd8037840 2024-09-16T04:05:17.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.870+0000 7fdffcaea640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:05:17.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.870+0000 7fdffcaea640 1 --2- 172.21.15.42:0/2383215944 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdff80a0e40 0x7fdff813aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:05:17.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.870+0000 7fdffcaea640 10 monclient(hunting): start opening mon connection 2024-09-16T04:05:17.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.870+0000 7fdffcaea640 1 -- 172.21.15.42:0/2383215944 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fdff81377a0 con 0x7fdff80a0e40 2024-09-16T04:05:17.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.870+0000 7fdff5d9b640 1 --2- 172.21.15.42:0/2383215944 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdff80a0e40 0x7fdff813aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:05:17.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.870+0000 7fdff5d9b640 10 monclient: get_auth_request con 0x7fdff80a0e40 auth_method 0 2024-09-16T04:05:17.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.870+0000 7fdff5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:05:17.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.870+0000 7fdff5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:05:17.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.870+0000 7fdff5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:05:17.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.870+0000 7fdff5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:05:17.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.870+0000 7fdff5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:05:17.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.870+0000 7fdff5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:05:17.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.870+0000 7fdff5d9b640 10 monclient(hunting): handle_auth_done global_id 984291 payload 824 2024-09-16T04:05:17.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.870+0000 7fdff5d9b640 1 -- 172.21.15.42:0/2383215944 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fdff80a1f40 con 0x7fdff80a0e40 2024-09-16T04:05:17.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.870+0000 7fdfe6ffd640 10 client.994115.objecter ms_handle_connect 0x7fdff80a0e40 2024-09-16T04:05:17.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.870+0000 7fdfe6ffd640 10 client.994115.objecter resend_mon_ops 2024-09-16T04:05:17.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.870+0000 7fdff5d9b640 1 --2- 172.21.15.42:0/2383215944 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdff80a0e40 0x7fdff813aaa0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fdfec01e000 tx=0x7fdfec0046d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:05:17.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.870+0000 7fdfe6ffd640 1 -- 172.21.15.42:0/2383215944 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfec00b5e0 con 0x7fdff80a0e40 2024-09-16T04:05:17.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.870+0000 7fdfe6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:05:17.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.870+0000 7fdfe6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:05:17.900 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.870+0000 7fdfe6ffd640 10 monclient: dump: 2024-09-16T04:05:17.900 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:05:17.900 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:05:17.900 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:17.900 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:17.900 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:05:17.900 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:05:17.900 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:05:17.900 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:05:17.900 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:05:17.900 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:05:17.900 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.870+0000 7fdfe6ffd640 1 -- 172.21.15.42:0/2383215944 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4714 (secure 0 0 0) 0x7fdfec00b5a0 con 0x7fdff80a0e40 2024-09-16T04:05:17.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.870+0000 7fdfe6ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:05:17.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.870+0000 7fdfe6ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:05:17.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.870+0000 7fdfe6ffd640 1 -- 172.21.15.42:0/2383215944 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdff80a0e40 msgr2=0x7fdff813aaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:05:17.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.870+0000 7fdfe6ffd640 1 --2- 172.21.15.42:0/2383215944 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdff80a0e40 0x7fdff813aaa0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fdfec01e000 tx=0x7fdfec0046d0 comp rx=0 tx=0).stop 2024-09-16T04:05:17.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.870+0000 7fdfe6ffd640 1 -- 172.21.15.42:0/2383215944 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdff80a0e40 msgr2=0x7fdff813aaa0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:05:17.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.870+0000 7fdfe6ffd640 1 --2- 172.21.15.42:0/2383215944 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdff80a0e40 0x7fdff813aaa0 secure :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fdfec01e000 tx=0x7fdfec0046d0 comp rx=0 tx=0).stop 2024-09-16T04:05:17.901 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T04:05:17.901 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:05:17.901 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:05:17.901 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "leader", 2024-09-16T04:05:17.902 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 216, 2024-09-16T04:05:17.902 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 77927, 2024-09-16T04:05:17.902 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T04:05:17.902 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T04:05:17.902 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T04:05:17.902 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T04:05:17.902 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:05:17.902 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 7, 2024-09-16T04:05:17.902 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:05:17.902 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T04:05:17.902 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T04:05:17.902 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:05:17.902 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:05:17.903 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:05:17.903 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:05:17.903 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:05:17.903 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:05:17.903 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:05:17.903 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:05:17.903 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:05:17.903 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:05:17.903 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:05:17.903 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:05:17.903 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:05:17.903 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T04:05:17.903 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:05:17.904 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:05:17.904 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:05:17.904 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:05:17.904 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:05:17.904 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:05:17.904 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:05:17.904 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:05:17.904 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:05:17.904 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:05:17.904 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:05:17.904 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:05:17.904 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:05:17.904 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T04:05:17.905 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T04:05:17.905 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T04:05:17.905 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T04:05:17.905 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T04:05:17.905 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T04:05:17.905 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:05:17.905 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:05:17.905 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T04:05:17.905 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T04:05:17.905 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T04:05:17.905 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T04:05:17.905 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T04:05:17.905 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T04:05:17.906 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T04:05:17.906 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:05:17.906 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T04:05:17.906 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:05:17.906 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:05:17.906 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:05:17.906 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:05:17.906 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:05:17.906 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:05:17.906 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:05:17.906 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:05:17.906 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:05:17.906 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:05:17.906 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:05:17.907 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:05:17.907 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T04:05:17.907 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:05:17.907 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T04:05:17.907 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:05:17.907 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:05:17.907 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:05:17.907 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:05:17.907 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:05:17.907 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:05:17.907 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:05:17.907 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T04:05:17.907 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:05:17.907 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:05:17.907 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:05:17.908 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:05:17.908 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T04:05:17.908 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:05:17.908 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:05:17.908 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:05:17.908 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:05:17.908 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T04:05:17.908 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T04:05:17.908 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:05:17.908 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:05:17.908 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:05:17.908 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:05:17.909 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:05:17.909 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:05:17.909 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:05:17.909 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:05:17.909 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:05:17.909 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:05:17.909 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:05:17.909 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T04:05:17.909 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:05:17.909 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:05:17.909 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:05:17.909 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:05:17.909 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T04:05:17.909 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:05:17.910 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:05:17.910 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:05:17.910 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:05:17.910 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T04:05:17.910 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T04:05:17.910 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:05:17.910 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:05:17.910 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:05:17.910 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:05:17.910 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:05:17.910 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:05:17.910 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:05:17.910 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:05:17.911 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:05:17.911 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:05:17.911 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:05:17.911 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T04:05:17.911 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:05:17.911 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:05:17.911 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:05:17.911 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:05:17.911 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T04:05:17.911 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:05:17.911 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:05:17.911 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:05:17.911 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:05:17.911 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T04:05:17.912 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T04:05:17.912 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:05:17.912 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:05:17.912 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:05:17.912 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:05:17.912 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:05:17.912 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:05:17.912 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T04:05:17.912 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T04:05:17.912 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:05:17.912 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:05:17.912 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:05:17.913 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:05:17.913 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:05:17.913 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:05:17.913 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T04:05:17.913 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:05:17.913 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:05:17.913 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:05:17.913 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 3 2024-09-16T04:05:17.913 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:05:17.913 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:05:17.913 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T04:05:17.913 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:05:17.914 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:05:17.914 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:05:17.914 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 2 2024-09-16T04:05:17.914 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:05:17.914 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:05:17.914 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:05:17.914 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T04:05:17.914 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T04:05:17.916 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.870+0000 7fdffcaea640 10 librados: watch_flush enter 2024-09-16T04:05:17.916 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.870+0000 7fdffcaea640 10 librados: watch_flush exit 2024-09-16T04:05:17.916 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.870+0000 7fdffcaea640 20 client.994115.objecter shutdown clearing up homeless session... 2024-09-16T04:05:17.916 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.870+0000 7fdffcaea640 10 client.994115.objecter successfully canceled tick 2024-09-16T04:05:17.916 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.870+0000 7fdffcaea640 1 -- 172.21.15.42:0/2383215944 >> v2:172.21.15.154:6812/254845342 conn(0x7fdfd8037840 msgr2=0x7fdfd8039d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:05:17.916 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.870+0000 7fdffcaea640 1 --2- 172.21.15.42:0/2383215944 >> v2:172.21.15.154:6812/254845342 conn(0x7fdfd8037840 0x7fdfd8039d00 secure :-1 s=READY pgs=3845 cs=0 l=1 rev1=1 crypto rx=0x7fdff81344a0 tx=0x7fdfe8007040 comp rx=0 tx=0).stop 2024-09-16T04:05:17.917 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.870+0000 7fdffcaea640 10 monclient: shutdown 2024-09-16T04:05:17.917 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.870+0000 7fdffcaea640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:05:17.917 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.870+0000 7fdffcaea640 1 -- 172.21.15.42:0/2383215944 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fdfe00039c0 msgr2=0x7fdff809fbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:05:17.917 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.870+0000 7fdffcaea640 1 --2- 172.21.15.42:0/2383215944 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fdfe00039c0 0x7fdff809fbc0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fdff0000f90 tx=0x7fdfe0005840 comp rx=0 tx=0).stop 2024-09-16T04:05:17.917 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.870+0000 7fdff6d9d640 1 -- 172.21.15.42:0/2383215944 reap_dead start 2024-09-16T04:05:17.917 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.870+0000 7fdff6d9d640 1 -- 172.21.15.42:0/2383215944 reap_dead start 2024-09-16T04:05:17.917 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.870+0000 7fdffcaea640 1 -- 172.21.15.42:0/2383215944 shutdown_connections 2024-09-16T04:05:17.917 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.870+0000 7fdffcaea640 1 -- 172.21.15.42:0/2383215944 >> 172.21.15.42:0/2383215944 conn(0x7fdff80a9fb0 msgr2=0x7fdff80a6d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:05:17.917 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.870+0000 7fdffcaea640 1 -- 172.21.15.42:0/2383215944 shutdown_connections 2024-09-16T04:05:17.917 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.870+0000 7fdffcaea640 1 -- 172.21.15.42:0/2383215944 wait complete. 2024-09-16T04:05:17.917 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:17.870+0000 7fdffcaea640 1 librados: shutdown 2024-09-16T04:05:17.918 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:05:17.918 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T04:05:17.918 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T04:05:18.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.030+0000 7fda1e93e640 10 monclient: get_monmap_and_config 2024-09-16T04:05:18.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.030+0000 7fda1e93e640 10 monclient: build_initial_monmap 2024-09-16T04:05:18.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.030+0000 7fda1e93e640 10 monclient: monmap: 2024-09-16T04:05:18.035 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:05:18.035 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:05:18.036 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:05:18.033628+0000 2024-09-16T04:05:18.036 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:05:18.033628+0000 2024-09-16T04:05:18.036 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:05:18.036 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:05:18.036 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:05:18.036 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:05:18.036 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:05:18.036 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:05:18.037 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:05:18.037 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:05:18.039 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.030+0000 7fda1e93e640 1 Processor -- start 2024-09-16T04:05:18.039 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.030+0000 7fda1e93e640 1 -- start start 2024-09-16T04:05:18.039 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.030+0000 7fda1e93e640 10 monclient: init 2024-09-16T04:05:18.039 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.030+0000 7fda1e93e640 10 monclient: _reopen_session rank -1 2024-09-16T04:05:18.039 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.030+0000 7fda1e93e640 10 monclient: _add_conns ranks=[1,3,2,0,4] 2024-09-16T04:05:18.039 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.030+0000 7fda1e93e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fda180a1330 addr v1:172.21.15.154:6789/0 2024-09-16T04:05:18.039 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.030+0000 7fda1e93e640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fda180a1710 0x7fda180a1b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:05:18.039 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.030+0000 7fda1e93e640 10 monclient(hunting): picked mon.noname-a con 0x7fda180a1710 addr v2:172.21.15.42:3300/0 2024-09-16T04:05:18.039 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.030+0000 7fda1e93e640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fda181393a0 0x7fda1813b790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:05:18.039 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.030+0000 7fda1e93e640 10 monclient(hunting): picked mon.noname-b con 0x7fda181393a0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:05:18.039 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.030+0000 7fda1e93e640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda1809e4c0 con 0x7fda180a1330 2024-09-16T04:05:18.040 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.030+0000 7fda1e93e640 10 monclient(hunting): start opening mon connection 2024-09-16T04:05:18.040 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.030+0000 7fda1e93e640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7fda1809edf0 con 0x7fda180a1710 2024-09-16T04:05:18.040 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.030+0000 7fda1e93e640 10 monclient(hunting): start opening mon connection 2024-09-16T04:05:18.040 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.030+0000 7fda1e93e640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fda1809cd60 con 0x7fda181393a0 2024-09-16T04:05:18.040 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.030+0000 7fda1e93e640 10 monclient(hunting): _renew_subs 2024-09-16T04:05:18.040 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.030+0000 7fda1e93e640 10 monclient(hunting): authenticate will time out at 2365.003838s 2024-09-16T04:05:18.040 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.034+0000 7fda0f7fe640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fda180a1710 0x7fda180a1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:05:18.040 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.034+0000 7fda1cedb640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fda181393a0 0x7fda1813b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:05:18.040 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.034+0000 7fda0f7fe640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fda180a1710 0x7fda180a1b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:47582/0 (socket says 172.21.15.42:47582) 2024-09-16T04:05:18.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.034+0000 7fda0f7fe640 1 -- 172.21.15.42:0/4108585361 learned_addr learned my addr 172.21.15.42:0/4108585361 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:05:18.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.034+0000 7fda0f7fe640 10 monclient(hunting): get_auth_request con 0x7fda180a1710 auth_method 0 2024-09-16T04:05:18.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.034+0000 7fda0f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:05:18.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.034+0000 7fda0f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:05:18.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.034+0000 7fda0f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:05:18.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.034+0000 7fda1cedb640 10 monclient(hunting): get_auth_request con 0x7fda181393a0 auth_method 0 2024-09-16T04:05:18.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.034+0000 7fda1cedb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:05:18.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.034+0000 7fda1cedb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:05:18.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.034+0000 7fda1cedb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:05:18.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.034+0000 7fda0f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:05:18.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.034+0000 7fda0f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:05:18.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.034+0000 7fda0f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:05:18.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.034+0000 7fda1cedb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:05:18.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.034+0000 7fda1cedb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:05:18.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.034+0000 7fda1cedb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:05:18.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.034+0000 7fda0f7fe640 10 monclient(hunting): handle_auth_done global_id 984294 payload 274 2024-09-16T04:05:18.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.034+0000 7fda0f7fe640 1 -- 172.21.15.42:0/4108585361 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fda181393a0 msgr2=0x7fda1813b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:05:18.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.034+0000 7fda0f7fe640 1 --2- 172.21.15.42:0/4108585361 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fda181393a0 0x7fda1813b790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:18.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.034+0000 7fda0f7fe640 1 -- 172.21.15.42:0/4108585361 >> v1:172.21.15.154:6789/0 conn(0x7fda180a1330 legacy=0x7fda1809f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:05:18.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.034+0000 7fda0f7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:05:18.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.034+0000 7fda0f7fe640 1 monclient: found mon.noname-a 2024-09-16T04:05:18.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.034+0000 7fda0f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:05:18.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.034+0000 7fda0f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:05:18.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.034+0000 7fda0f7fe640 1 -- 172.21.15.42:0/4108585361 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda180a2080 con 0x7fda180a1710 2024-09-16T04:05:18.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.034+0000 7fda1cedb640 1 --2- 172.21.15.42:0/4108585361 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fda181393a0 0x7fda1813b790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:05:18.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.034+0000 7fda0f7fe640 1 --2- 172.21.15.42:0/4108585361 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fda180a1710 0x7fda180a1b40 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fda0000bb10 tx=0x7fda0001eaf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23816713fb27af80 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:05:18.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.034+0000 7fda177fe640 1 -- 172.21.15.42:0/4108585361 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda000b4070 con 0x7fda180a1710 2024-09-16T04:05:18.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.034+0000 7fda177fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:05:18.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.034+0000 7fda177fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:05:18.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.034+0000 7fda177fe640 10 monclient: dump: 2024-09-16T04:05:18.054 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:05:18.054 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:05:18.054 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:18.054 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:18.055 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:05:18.055 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:05:18.055 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:05:18.055 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:05:18.055 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:05:18.055 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:05:18.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.034+0000 7fda177fe640 10 monclient: _finish_auth 0 2024-09-16T04:05:18.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.034+0000 7fda177fe640 10 monclient: _check_auth_tickets 2024-09-16T04:05:18.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.034+0000 7fda177fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:05:18.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.034+0000 7fda177fe640 1 -- 172.21.15.42:0/4108585361 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda000a7ad0 con 0x7fda180a1710 2024-09-16T04:05:18.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.034+0000 7fda1e93e640 5 monclient: authenticate success, global_id 984294 2024-09-16T04:05:18.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.034+0000 7fda1e93e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:05:18.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.034+0000 7fda177fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:05:18.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.034+0000 7fda1e93e640 10 monclient: get_monmap_and_config success 2024-09-16T04:05:18.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.034+0000 7fda177fe640 1 -- 172.21.15.42:0/4108585361 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda000af440 con 0x7fda180a1710 2024-09-16T04:05:18.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.034+0000 7fda1e93e640 4 set_mon_vals no callback set 2024-09-16T04:05:18.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.034+0000 7fda177fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:05:18.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.034+0000 7fda177fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:05:18.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.034+0000 7fda177fe640 10 monclient: dump: 2024-09-16T04:05:18.056 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:05:18.056 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:05:18.056 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:18.056 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:18.056 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:05:18.056 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:05:18.056 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:05:18.057 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:05:18.057 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:05:18.057 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:05:18.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.034+0000 7fda1e93e640 10 monclient: shutdown 2024-09-16T04:05:18.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.034+0000 7fda1e93e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:05:18.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.034+0000 7fda1e93e640 1 -- 172.21.15.42:0/4108585361 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fda180a1710 msgr2=0x7fda180a1b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:05:18.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.034+0000 7fda1e93e640 1 --2- 172.21.15.42:0/4108585361 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fda180a1710 0x7fda180a1b40 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fda0000bb10 tx=0x7fda0001eaf0 comp rx=0 tx=0).stop 2024-09-16T04:05:18.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.034+0000 7fda1e93e640 1 -- 172.21.15.42:0/4108585361 shutdown_connections 2024-09-16T04:05:18.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.034+0000 7fda1e93e640 1 --2- 172.21.15.42:0/4108585361 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fda181393a0 0x7fda1813b790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:18.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.034+0000 7fda1e93e640 1 --2- 172.21.15.42:0/4108585361 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fda180a1710 0x7fda180a1b40 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:18.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.034+0000 7fda1e93e640 1 -- 172.21.15.42:0/4108585361 >> 172.21.15.42:0/4108585361 conn(0x7fda180a2420 msgr2=0x7fda180a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:05:18.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.034+0000 7fda1e93e640 1 -- 172.21.15.42:0/4108585361 shutdown_connections 2024-09-16T04:05:18.059 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.034+0000 7fda1e93e640 1 -- 172.21.15.42:0/4108585361 wait complete. 2024-09-16T04:05:18.059 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.038+0000 7fda1e93e640 10 monclient: build_initial_monmap 2024-09-16T04:05:18.059 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.038+0000 7fda1e93e640 10 monclient: monmap: 2024-09-16T04:05:18.059 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:05:18.059 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:05:18.060 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:05:18.060 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:05:18.060 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:05:18.060 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:05:18.060 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:05:18.060 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:05:18.061 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:05:18.061 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:05:18.061 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.038+0000 7fda1e93e640 1 librados: starting msgr at 2024-09-16T04:05:18.061 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.038+0000 7fda1e93e640 1 librados: starting objecter 2024-09-16T04:05:18.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.038+0000 7fda1e93e640 1 Processor -- start 2024-09-16T04:05:18.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.038+0000 7fda1e93e640 1 -- start start 2024-09-16T04:05:18.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.038+0000 7fda1e93e640 1 librados: setting wanted keys 2024-09-16T04:05:18.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.038+0000 7fda1e93e640 1 librados: calling monclient init 2024-09-16T04:05:18.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.038+0000 7fda1e93e640 10 monclient: init 2024-09-16T04:05:18.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.038+0000 7fda1e93e640 10 monclient: _reopen_session rank -1 2024-09-16T04:05:18.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.038+0000 7fda1e93e640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:05:18.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.038+0000 7fda1e93e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fda180a1330 0x7fda18134340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:05:18.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.038+0000 7fda1e93e640 10 monclient(hunting): picked mon.noname-b con 0x7fda180a1330 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:05:18.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.038+0000 7fda1e93e640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fda180a1710 0x7fda18134880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:05:18.063 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.038+0000 7fda1e93e640 10 monclient(hunting): picked mon.noname-a con 0x7fda180a1710 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:05:18.063 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.038+0000 7fda1e93e640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fda181393a0 0x7fda1811dd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:05:18.063 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.038+0000 7fda1e93e640 10 monclient(hunting): picked mon.noname-c con 0x7fda181393a0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:05:18.063 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.038+0000 7fda1e93e640 10 monclient(hunting): start opening mon connection 2024-09-16T04:05:18.063 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.038+0000 7fda1e93e640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fda1809c630 con 0x7fda180a1710 2024-09-16T04:05:18.063 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.038+0000 7fda1e93e640 10 monclient(hunting): start opening mon connection 2024-09-16T04:05:18.063 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.038+0000 7fda1e93e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fda1809c4b0 con 0x7fda180a1330 2024-09-16T04:05:18.063 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.038+0000 7fda1e93e640 10 monclient(hunting): start opening mon connection 2024-09-16T04:05:18.063 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.038+0000 7fda1e93e640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fda1809c7b0 con 0x7fda181393a0 2024-09-16T04:05:18.063 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.038+0000 7fda1e93e640 10 monclient(hunting): _renew_subs 2024-09-16T04:05:18.063 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.038+0000 7fda1e93e640 10 monclient(hunting): authenticate will time out at 2365.009406s 2024-09-16T04:05:18.063 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.038+0000 7fda0f7fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fda180a1710 0x7fda18134880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:05:18.064 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.038+0000 7fda1cedb640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fda181393a0 0x7fda1811dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:05:18.064 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.038+0000 7fda0f7fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fda180a1710 0x7fda18134880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:47592/0 (socket says 172.21.15.42:47592) 2024-09-16T04:05:18.064 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.038+0000 7fda0f7fe640 1 -- 172.21.15.42:0/2426918681 learned_addr learned my addr 172.21.15.42:0/2426918681 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:05:18.064 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.038+0000 7fda1cedb640 10 monclient(hunting): get_auth_request con 0x7fda181393a0 auth_method 0 2024-09-16T04:05:18.064 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.038+0000 7fda1cedb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:05:18.064 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.038+0000 7fda1cedb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:05:18.064 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.038+0000 7fda1cedb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:05:18.064 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.038+0000 7fda0f7fe640 10 monclient(hunting): get_auth_request con 0x7fda180a1710 auth_method 0 2024-09-16T04:05:18.064 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.038+0000 7fda0f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:05:18.064 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.038+0000 7fda0f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:05:18.065 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.038+0000 7fda0f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:05:18.065 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.038+0000 7fda0f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:05:18.065 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.038+0000 7fda0f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:05:18.065 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.038+0000 7fda0f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:05:18.065 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.038+0000 7fda1cedb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:05:18.065 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.038+0000 7fda1cedb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:05:18.065 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.038+0000 7fda1cedb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:05:18.065 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.038+0000 7fda17fff640 1 --2- 172.21.15.42:0/2426918681 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fda180a1330 0x7fda18134340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:05:18.065 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.038+0000 7fda17fff640 10 monclient(hunting): get_auth_request con 0x7fda180a1330 auth_method 0 2024-09-16T04:05:18.065 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.038+0000 7fda17fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:05:18.065 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.038+0000 7fda17fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:05:18.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.038+0000 7fda17fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:05:18.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.038+0000 7fda0f7fe640 10 monclient(hunting): handle_auth_done global_id 984297 payload 824 2024-09-16T04:05:18.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.038+0000 7fda0f7fe640 1 -- 172.21.15.42:0/2426918681 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fda181393a0 msgr2=0x7fda1811dd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:05:18.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.038+0000 7fda0f7fe640 1 --2- 172.21.15.42:0/2426918681 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fda181393a0 0x7fda1811dd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:18.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.038+0000 7fda0f7fe640 1 -- 172.21.15.42:0/2426918681 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fda180a1330 msgr2=0x7fda18134340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:05:18.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.038+0000 7fda0f7fe640 1 --2- 172.21.15.42:0/2426918681 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fda180a1330 0x7fda18134340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:18.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.038+0000 7fda0f7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:05:18.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.038+0000 7fda0f7fe640 1 monclient: found mon.noname-a 2024-09-16T04:05:18.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.038+0000 7fda0f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:05:18.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.038+0000 7fda0f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:05:18.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.038+0000 7fda0f7fe640 1 -- 172.21.15.42:0/2426918681 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda18121d00 con 0x7fda180a1710 2024-09-16T04:05:18.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.038+0000 7fda1cedb640 1 --2- 172.21.15.42:0/2426918681 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fda181393a0 0x7fda1811dd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:05:18.067 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.038+0000 7fda17fff640 1 --2- 172.21.15.42:0/2426918681 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fda180a1330 0x7fda18134340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T04:05:18.067 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.038+0000 7fda0f7fe640 1 --2- 172.21.15.42:0/2426918681 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fda180a1710 0x7fda18134880 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fda0000b2c0 tx=0x7fda0001b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:05:18.067 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.038+0000 7fda157fa640 10 client.?.objecter ms_handle_connect 0x7fda180a1710 2024-09-16T04:05:18.067 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.038+0000 7fda157fa640 10 client.?.objecter resend_mon_ops 2024-09-16T04:05:18.067 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.038+0000 7fda157fa640 1 -- 172.21.15.42:0/2426918681 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda000b4030 con 0x7fda180a1710 2024-09-16T04:05:18.067 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.038+0000 7fda157fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:05:18.067 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.038+0000 7fda157fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:05:18.067 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.038+0000 7fda157fa640 10 monclient: dump: 2024-09-16T04:05:18.067 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:05:18.067 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:05:18.067 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:18.067 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:18.068 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:05:18.068 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:05:18.068 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:05:18.068 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:05:18.068 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:05:18.068 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:05:18.068 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.038+0000 7fda157fa640 10 monclient: _finish_auth 0 2024-09-16T04:05:18.068 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.038+0000 7fda157fa640 10 monclient: _check_auth_tickets 2024-09-16T04:05:18.068 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.038+0000 7fda157fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:05:18.068 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.038+0000 7fda1e93e640 5 monclient: authenticate success, global_id 984297 2024-09-16T04:05:18.068 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.038+0000 7fda157fa640 1 -- 172.21.15.42:0/2426918681 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda000b6070 con 0x7fda180a1710 2024-09-16T04:05:18.069 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.038+0000 7fda157fa640 10 monclient: handle_config config(0 keys) 2024-09-16T04:05:18.069 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.038+0000 7fda1e93e640 10 monclient: _renew_subs 2024-09-16T04:05:18.069 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.038+0000 7fda157fa640 1 -- 172.21.15.42:0/2426918681 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda000ad550 con 0x7fda180a1710 2024-09-16T04:05:18.069 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.038+0000 7fda1e93e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:05:18.069 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.038+0000 7fda1e93e640 1 -- 172.21.15.42:0/2426918681 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda180a4fb0 con 0x7fda180a1710 2024-09-16T04:05:18.069 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.038+0000 7fda16ffd640 4 set_mon_vals no callback set 2024-09-16T04:05:18.069 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.038+0000 7fda1e93e640 10 client.984297.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:05:18.069 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.038+0000 7fda1e93e640 10 monclient: _renew_subs 2024-09-16T04:05:18.069 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.038+0000 7fda1e93e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:05:18.069 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.038+0000 7fda1e93e640 1 -- 172.21.15.42:0/2426918681 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda18073a30 con 0x7fda180a1710 2024-09-16T04:05:18.069 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.038+0000 7fda1e93e640 1 librados: init done 2024-09-16T04:05:18.069 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.038+0000 7fda157fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:05:18.070 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.038+0000 7fda157fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:05:18.070 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.038+0000 7fda157fa640 10 monclient: dump: 2024-09-16T04:05:18.070 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:05:18.070 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:05:18.070 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:18.070 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:18.070 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:05:18.070 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:05:18.070 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:05:18.070 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:05:18.070 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:05:18.070 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:05:18.071 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.038+0000 7fda1e93e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:05:18.071 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.038+0000 7fda1e93e640 1 --2- 172.21.15.42:0/2426918681 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fda181cf300 0x7fda181216a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:05:18.071 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.038+0000 7fda1e93e640 10 monclient(hunting): start opening mon connection 2024-09-16T04:05:18.071 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.038+0000 7fda1e93e640 1 -- 172.21.15.42:0/2426918681 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fda180a1b40 con 0x7fda181cf300 2024-09-16T04:05:18.071 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.038+0000 7fda17fff640 1 --2- 172.21.15.42:0/2426918681 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fda181cf300 0x7fda181216a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:05:18.071 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.038+0000 7fda17fff640 10 monclient: get_auth_request con 0x7fda181cf300 auth_method 0 2024-09-16T04:05:18.071 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.038+0000 7fda17fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:05:18.071 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.038+0000 7fda17fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:05:18.071 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.038+0000 7fda17fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:05:18.071 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.038+0000 7fda17fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:05:18.071 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.038+0000 7fda17fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:05:18.071 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.038+0000 7fda17fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:05:18.072 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.038+0000 7fda157fa640 1 -- 172.21.15.42:0/2426918681 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fda000c1060 con 0x7fda180a1710 2024-09-16T04:05:18.072 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.038+0000 7fda17fff640 10 monclient(hunting): handle_auth_done global_id 994124 payload 824 2024-09-16T04:05:18.072 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.038+0000 7fda17fff640 1 -- 172.21.15.42:0/2426918681 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fda181202f0 con 0x7fda181cf300 2024-09-16T04:05:18.072 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.038+0000 7fda157fa640 1 --2- 172.21.15.42:0/2426918681 >> v2:172.21.15.154:6812/254845342 conn(0x7fd9f8037810 0x7fd9f8039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:05:18.072 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.038+0000 7fda157fa640 1 -- 172.21.15.42:0/2426918681 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(705..705 src has 451..705) ==== 5150+0+0 (secure 0 0 0) 0x7fda000e3250 con 0x7fda180a1710 2024-09-16T04:05:18.072 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.038+0000 7fda157fa640 10 client.984297.objecter ms_dispatch 0x7fda18000ba0 osd_map(705..705 src has 451..705) 2024-09-16T04:05:18.072 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.038+0000 7fda157fa640 3 client.984297.objecter handle_osd_map got epochs [705,705] > 0 2024-09-16T04:05:18.072 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.038+0000 7fda157fa640 3 client.984297.objecter handle_osd_map decoding full epoch 705 2024-09-16T04:05:18.072 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.038+0000 7fda17fff640 1 --2- 172.21.15.42:0/2426918681 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fda181cf300 0x7fda181216a0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fda0800e450 tx=0x7fda0800e910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:05:18.072 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.038+0000 7fda157fa640 20 client.984297.objecter dump_active .. 0 homeless 2024-09-16T04:05:18.072 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.038+0000 7fda157fa640 10 client.984297.objecter ms_handle_connect 0x7fda181cf300 2024-09-16T04:05:18.072 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.038+0000 7fda157fa640 10 client.984297.objecter resend_mon_ops 2024-09-16T04:05:18.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.042+0000 7fda157fa640 1 -- 172.21.15.42:0/2426918681 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda080052e0 con 0x7fda181cf300 2024-09-16T04:05:18.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.042+0000 7fda157fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:05:18.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.042+0000 7fda157fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:05:18.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.042+0000 7fda157fa640 10 monclient: dump: 2024-09-16T04:05:18.073 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:05:18.073 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:05:18.073 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:18.073 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:18.073 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:05:18.073 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:05:18.073 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:05:18.074 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:05:18.074 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:05:18.074 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:05:18.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.042+0000 7fda1cedb640 1 --2- 172.21.15.42:0/2426918681 >> v2:172.21.15.154:6812/254845342 conn(0x7fd9f8037810 0x7fd9f8039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:05:18.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.042+0000 7fda1cedb640 10 monclient: get_auth_request con 0x7fd9f8037810 auth_method 0 2024-09-16T04:05:18.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.042+0000 7fda157fa640 1 -- 172.21.15.42:0/2426918681 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fda080052e0 con 0x7fda181cf300 2024-09-16T04:05:18.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.042+0000 7fda157fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:05:18.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.042+0000 7fda157fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:05:18.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.042+0000 7fda157fa640 1 -- 172.21.15.42:0/2426918681 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fda181cf300 msgr2=0x7fda181216a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:05:18.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.042+0000 7fda157fa640 1 --2- 172.21.15.42:0/2426918681 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fda181cf300 0x7fda181216a0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fda0800e450 tx=0x7fda0800e910 comp rx=0 tx=0).stop 2024-09-16T04:05:18.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.042+0000 7fda157fa640 1 -- 172.21.15.42:0/2426918681 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fda181cf300 msgr2=0x7fda181216a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:05:18.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.042+0000 7fda157fa640 1 --2- 172.21.15.42:0/2426918681 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fda181cf300 0x7fda181216a0 secure :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fda0800e450 tx=0x7fda0800e910 comp rx=0 tx=0).stop 2024-09-16T04:05:18.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.042+0000 7fda1cedb640 1 --2- 172.21.15.42:0/2426918681 >> v2:172.21.15.154:6812/254845342 conn(0x7fd9f8037810 0x7fd9f8039cd0 secure :-1 s=READY pgs=3846 cs=0 l=1 rev1=1 crypto rx=0x7fda1809df70 tx=0x7fda1000b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:05:18.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.042+0000 7fda157fa640 10 client.984297.objecter ms_handle_connect 0x7fd9f8037810 2024-09-16T04:05:18.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.046+0000 7fda1e93e640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:05:18.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.046+0000 7fda1e93e640 1 --2- 172.21.15.42:0/2426918681 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fda1811e2b0 0x7fda1811e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:05:18.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.046+0000 7fda1e93e640 10 monclient(hunting): start opening mon connection 2024-09-16T04:05:18.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.046+0000 7fda1e93e640 1 -- 172.21.15.42:0/2426918681 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fda1811ec00 con 0x7fda1811e2b0 2024-09-16T04:05:18.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.046+0000 7fda17fff640 1 --2- 172.21.15.42:0/2426918681 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fda1811e2b0 0x7fda1811e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:05:18.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.046+0000 7fda17fff640 10 monclient: get_auth_request con 0x7fda1811e2b0 auth_method 0 2024-09-16T04:05:18.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.046+0000 7fda17fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:05:18.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.046+0000 7fda17fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:05:18.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.046+0000 7fda17fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:05:18.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.046+0000 7fda17fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:05:18.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.046+0000 7fda17fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:05:18.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.046+0000 7fda17fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:05:18.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.046+0000 7fda17fff640 10 monclient(hunting): handle_auth_done global_id 994127 payload 824 2024-09-16T04:05:18.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.046+0000 7fda17fff640 1 -- 172.21.15.42:0/2426918681 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fda18069040 con 0x7fda1811e2b0 2024-09-16T04:05:18.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.046+0000 7fda17fff640 1 --2- 172.21.15.42:0/2426918681 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fda1811e2b0 0x7fda1811e6c0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fda0800e450 tx=0x7fda08000ee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:05:18.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.046+0000 7fda157fa640 10 client.984297.objecter ms_handle_connect 0x7fda1811e2b0 2024-09-16T04:05:18.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.046+0000 7fda157fa640 10 client.984297.objecter resend_mon_ops 2024-09-16T04:05:18.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.046+0000 7fda157fa640 1 -- 172.21.15.42:0/2426918681 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda080069f0 con 0x7fda1811e2b0 2024-09-16T04:05:18.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.046+0000 7fda157fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:05:18.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.046+0000 7fda157fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:05:18.077 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.046+0000 7fda157fa640 10 monclient: dump: 2024-09-16T04:05:18.077 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:05:18.077 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:05:18.077 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:18.077 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:18.077 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:05:18.077 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:05:18.077 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:05:18.077 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:05:18.077 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:05:18.077 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:05:18.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.046+0000 7fda157fa640 1 -- 172.21.15.42:0/2426918681 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4545 (secure 0 0 0) 0x7fda080069f0 con 0x7fda1811e2b0 2024-09-16T04:05:18.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.046+0000 7fda157fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:05:18.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.046+0000 7fda157fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:05:18.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.046+0000 7fda157fa640 1 -- 172.21.15.42:0/2426918681 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fda1811e2b0 msgr2=0x7fda1811e6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:05:18.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.046+0000 7fda157fa640 1 --2- 172.21.15.42:0/2426918681 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fda1811e2b0 0x7fda1811e6c0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fda0800e450 tx=0x7fda08000ee0 comp rx=0 tx=0).stop 2024-09-16T04:05:18.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.046+0000 7fda157fa640 1 -- 172.21.15.42:0/2426918681 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fda1811e2b0 msgr2=0x7fda1811e6c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:05:18.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.046+0000 7fda157fa640 1 --2- 172.21.15.42:0/2426918681 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fda1811e2b0 0x7fda1811e6c0 secure :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fda0800e450 tx=0x7fda08000ee0 comp rx=0 tx=0).stop 2024-09-16T04:05:18.078 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T04:05:18.079 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:05:18.079 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:05:18.079 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T04:05:18.079 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 216, 2024-09-16T04:05:18.079 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 12735, 2024-09-16T04:05:18.079 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T04:05:18.079 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T04:05:18.079 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T04:05:18.079 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T04:05:18.079 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:05:18.079 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 7, 2024-09-16T04:05:18.080 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:05:18.080 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T04:05:18.080 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T04:05:18.080 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:05:18.080 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:05:18.080 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:05:18.080 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:05:18.080 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:05:18.080 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:05:18.080 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:05:18.080 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:05:18.080 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:05:18.081 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:05:18.081 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:05:18.081 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:05:18.081 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:05:18.081 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T04:05:18.081 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:05:18.081 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:05:18.081 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:05:18.081 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:05:18.081 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:05:18.081 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:05:18.081 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:05:18.082 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:05:18.082 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:05:18.082 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:05:18.082 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:05:18.082 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:05:18.082 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:05:18.082 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T04:05:18.082 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T04:05:18.082 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T04:05:18.082 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T04:05:18.082 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T04:05:18.082 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T04:05:18.083 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:05:18.083 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:05:18.083 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T04:05:18.083 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T04:05:18.083 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T04:05:18.083 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T04:05:18.083 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T04:05:18.083 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T04:05:18.083 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T04:05:18.083 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:05:18.083 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T04:05:18.084 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:05:18.084 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:05:18.084 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:05:18.084 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:05:18.084 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:05:18.084 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:05:18.084 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:05:18.084 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:05:18.084 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:05:18.084 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:05:18.084 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:05:18.084 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:05:18.085 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T04:05:18.085 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:05:18.085 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T04:05:18.085 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:05:18.085 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:05:18.085 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:05:18.085 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:05:18.085 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:05:18.085 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:05:18.085 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:05:18.085 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T04:05:18.086 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:05:18.086 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:05:18.086 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:05:18.086 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:05:18.086 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T04:05:18.086 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:05:18.086 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:05:18.086 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:05:18.086 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:05:18.086 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T04:05:18.086 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T04:05:18.086 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:05:18.087 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:05:18.087 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:05:18.087 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:05:18.087 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:05:18.087 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:05:18.087 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:05:18.087 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:05:18.087 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:05:18.087 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:05:18.087 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:05:18.087 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T04:05:18.088 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:05:18.088 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:05:18.088 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:05:18.088 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:05:18.088 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T04:05:18.088 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:05:18.088 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:05:18.088 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:05:18.088 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:05:18.088 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T04:05:18.088 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T04:05:18.088 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:05:18.089 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:05:18.089 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:05:18.089 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:05:18.089 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:05:18.089 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:05:18.089 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:05:18.089 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:05:18.089 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:05:18.089 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:05:18.089 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:05:18.089 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T04:05:18.090 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:05:18.090 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:05:18.090 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:05:18.090 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:05:18.090 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T04:05:18.090 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:05:18.090 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:05:18.090 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:05:18.090 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:05:18.090 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T04:05:18.090 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T04:05:18.090 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:05:18.091 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:05:18.091 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:05:18.091 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:05:18.091 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:05:18.091 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:05:18.091 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T04:05:18.091 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T04:05:18.091 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:05:18.091 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:05:18.091 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:05:18.091 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:05:18.091 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:05:18.092 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:05:18.092 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T04:05:18.092 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:05:18.092 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:05:18.092 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:05:18.092 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:05:18.092 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:05:18.092 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:05:18.092 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:05:18.092 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T04:05:18.092 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T04:05:18.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.050+0000 7fda1e93e640 10 librados: watch_flush enter 2024-09-16T04:05:18.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.050+0000 7fda1e93e640 10 librados: watch_flush exit 2024-09-16T04:05:18.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.050+0000 7fda1e93e640 20 client.984297.objecter shutdown clearing up homeless session... 2024-09-16T04:05:18.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.050+0000 7fda1e93e640 10 client.984297.objecter successfully canceled tick 2024-09-16T04:05:18.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.050+0000 7fda1e93e640 1 -- 172.21.15.42:0/2426918681 >> v2:172.21.15.154:6812/254845342 conn(0x7fd9f8037810 msgr2=0x7fd9f8039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:05:18.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.050+0000 7fda1e93e640 1 --2- 172.21.15.42:0/2426918681 >> v2:172.21.15.154:6812/254845342 conn(0x7fd9f8037810 0x7fd9f8039cd0 secure :-1 s=READY pgs=3846 cs=0 l=1 rev1=1 crypto rx=0x7fda1809df70 tx=0x7fda1000b040 comp rx=0 tx=0).stop 2024-09-16T04:05:18.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.050+0000 7fda1e93e640 10 monclient: shutdown 2024-09-16T04:05:18.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.050+0000 7fda1e93e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:05:18.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.050+0000 7fda1e93e640 1 -- 172.21.15.42:0/2426918681 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fda180a1710 msgr2=0x7fda18134880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:05:18.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.050+0000 7fda1e93e640 1 --2- 172.21.15.42:0/2426918681 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fda180a1710 0x7fda18134880 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fda0000b2c0 tx=0x7fda0001b890 comp rx=0 tx=0).stop 2024-09-16T04:05:18.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.050+0000 7fda1cedb640 1 -- 172.21.15.42:0/2426918681 reap_dead start 2024-09-16T04:05:18.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.050+0000 7fda1cedb640 1 -- 172.21.15.42:0/2426918681 reap_dead start 2024-09-16T04:05:18.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.050+0000 7fda1e93e640 1 -- 172.21.15.42:0/2426918681 shutdown_connections 2024-09-16T04:05:18.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.050+0000 7fda1e93e640 1 -- 172.21.15.42:0/2426918681 >> 172.21.15.42:0/2426918681 conn(0x7fda180a2420 msgr2=0x7fda180a3720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:05:18.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.050+0000 7fda1e93e640 1 -- 172.21.15.42:0/2426918681 shutdown_connections 2024-09-16T04:05:18.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.050+0000 7fda1e93e640 1 -- 172.21.15.42:0/2426918681 wait complete. 2024-09-16T04:05:18.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.050+0000 7fda1e93e640 1 librados: shutdown 2024-09-16T04:05:18.096 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:05:18.096 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T04:05:18.096 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T04:05:18.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.206+0000 7fe8bc868640 10 monclient: get_monmap_and_config 2024-09-16T04:05:18.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.206+0000 7fe8bc868640 10 monclient: build_initial_monmap 2024-09-16T04:05:18.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.206+0000 7fe8bc868640 10 monclient: monmap: 2024-09-16T04:05:18.213 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:05:18.213 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:05:18.213 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:05:18.211580+0000 2024-09-16T04:05:18.213 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:05:18.211580+0000 2024-09-16T04:05:18.213 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:05:18.213 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:05:18.213 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:05:18.213 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:05:18.214 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:05:18.214 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:05:18.214 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:05:18.214 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:05:18.214 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TryFlush (8072 ms) 2024-09-16T04:05:18.215 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FailedFlush 2024-09-16T04:05:18.215 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FailedFlush (16137 ms) 2024-09-16T04:05:18.215 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Flush 2024-09-16T04:05:18.215 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Flush (22853 ms) 2024-09-16T04:05:18.215 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FlushSnap 2024-09-16T04:05:18.215 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FlushSnap (12986 ms) 2024-09-16T04:05:18.215 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FlushTryFlushRaces 2024-09-16T04:05:18.215 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FlushTryFlushRaces (8063 ms) 2024-09-16T04:05:18.215 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TryFlushReadRace 2024-09-16T04:05:18.215 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TryFlushReadRace (22157 ms) 2024-09-16T04:05:18.215 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.HitSetRead 2024-09-16T04:05:18.215 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: hmm, no HitSet yet 2024-09-16T04:05:18.216 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: ok, hit_set contains 332:602f83fe:::foo:head 2024-09-16T04:05:18.216 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.HitSetRead (8729 ms) 2024-09-16T04:05:18.216 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.HitSetTrim 2024-09-16T04:05:18.216 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: got ls 1726459388,0 2024-09-16T04:05:18.216 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: first is 1726459388 2024-09-16T04:05:18.216 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: got ls 1726459388,0 2024-09-16T04:05:18.216 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: got ls 1726459388,0 2024-09-16T04:05:18.216 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: got ls 1726459388,1726459390,1726459391,0 2024-09-16T04:05:18.216 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: got ls 1726459388,1726459390,1726459391,0 2024-09-16T04:05:18.216 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: got ls 1726459388,1726459390,1726459391,0 2024-09-16T04:05:18.216 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: got ls 1726459388,1726459390,1726459391,1726459393,1726459394,0 2024-09-16T04:05:18.216 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: got ls 1726459388,1726459390,1726459391,1726459393,1726459394,0 2024-09-16T04:05:18.216 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: got ls 1726459388,1726459390,1726459391,1726459393,1726459394,0 2024-09-16T04:05:18.217 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: got ls 1726459388,1726459390,1726459391,1726459393,1726459394,1726459396,1726459397,0 2024-09-16T04:05:18.217 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: got ls 1726459388,1726459390,1726459391,1726459393,1726459394,1726459396,1726459397,0 2024-09-16T04:05:18.217 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: got ls 1726459388,1726459390,1726459391,1726459393,1726459394,1726459396,1726459397,0 2024-09-16T04:05:18.217 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: got ls 1726459391,1726459393,1726459394,1726459396,1726459397,1726459399,1726459400,0 2024-09-16T04:05:18.217 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: first now 1726459391, trimmed 2024-09-16T04:05:18.217 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.HitSetTrim (21134 ms) 2024-09-16T04:05:18.217 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteOn2ndRead 2024-09-16T04:05:18.217 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: foo0 2024-09-16T04:05:18.217 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: verifying foo0 is eventually promoted 2024-09-16T04:05:18.217 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteOn2ndRead (27519 ms) 2024-09-16T04:05:18.217 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.ProxyRead 2024-09-16T04:05:18.217 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.ProxyRead (28182 ms) 2024-09-16T04:05:18.217 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.CachePin 2024-09-16T04:05:18.217 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.CachePin (43640 ms) 2024-09-16T04:05:18.218 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.SetRedirectRead 2024-09-16T04:05:18.218 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.SetRedirectRead (5056 ms) 2024-09-16T04:05:18.218 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.SetChunkRead 2024-09-16T04:05:18.218 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.SetChunkRead (3016 ms) 2024-09-16T04:05:18.218 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.ManifestPromoteRead 2024-09-16T04:05:18.218 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.ManifestPromoteRead (3029 ms) 2024-09-16T04:05:18.218 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TrySetDedupTier 2024-09-16T04:05:18.218 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TrySetDedupTier (3032 ms) 2024-09-16T04:05:18.218 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [----------] 22 tests from LibRadosTwoPoolsECPP (358535 ms total) 2024-09-16T04:05:18.218 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: 2024-09-16T04:05:18.218 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [----------] Global test environment tear-down 2024-09-16T04:05:18.218 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [==========] 77 tests from 4 test suites ran. (1308574 ms total) 2024-09-16T04:05:18.218 INFO:tasks.workunit.client.0.smithi154.stdout: api_tier_pp: [ PASSED ] 77 tests. 2024-09-16T04:05:18.224 INFO:tasks.workunit.client.0.smithi154.stderr:+ for t in "${!pids[@]}" 2024-09-16T04:05:18.224 INFO:tasks.workunit.client.0.smithi154.stderr:+ pid=28110 2024-09-16T04:05:18.224 INFO:tasks.workunit.client.0.smithi154.stderr:+ wait 28110 2024-09-16T04:05:18.224 INFO:tasks.workunit.client.0.smithi154.stderr:+ for t in "${!pids[@]}" 2024-09-16T04:05:18.224 INFO:tasks.workunit.client.0.smithi154.stderr:+ pid=28227 2024-09-16T04:05:18.224 INFO:tasks.workunit.client.0.smithi154.stderr:+ wait 28227 2024-09-16T04:05:18.224 INFO:tasks.workunit.client.0.smithi154.stderr:+ for t in "${!pids[@]}" 2024-09-16T04:05:18.224 INFO:tasks.workunit.client.0.smithi154.stderr:+ pid=28423 2024-09-16T04:05:18.224 INFO:tasks.workunit.client.0.smithi154.stderr:+ wait 28423 2024-09-16T04:05:18.224 INFO:tasks.workunit.client.0.smithi154.stderr:+ for t in "${!pids[@]}" 2024-09-16T04:05:18.225 INFO:tasks.workunit.client.0.smithi154.stderr:+ pid=28346 2024-09-16T04:05:18.225 INFO:tasks.workunit.client.0.smithi154.stderr:+ wait 28346 2024-09-16T04:05:18.225 INFO:tasks.workunit.client.0.smithi154.stderr:+ for t in "${!pids[@]}" 2024-09-16T04:05:18.225 INFO:tasks.workunit.client.0.smithi154.stderr:+ pid=28260 2024-09-16T04:05:18.225 INFO:tasks.workunit.client.0.smithi154.stderr:+ wait 28260 2024-09-16T04:05:18.225 INFO:tasks.workunit.client.0.smithi154.stderr:+ for t in "${!pids[@]}" 2024-09-16T04:05:18.225 INFO:tasks.workunit.client.0.smithi154.stderr:+ pid=28181 2024-09-16T04:05:18.225 INFO:tasks.workunit.client.0.smithi154.stderr:+ wait 28181 2024-09-16T04:05:18.225 INFO:tasks.workunit.client.0.smithi154.stderr:+ for t in "${!pids[@]}" 2024-09-16T04:05:18.225 INFO:tasks.workunit.client.0.smithi154.stderr:+ pid=28280 2024-09-16T04:05:18.225 INFO:tasks.workunit.client.0.smithi154.stderr:+ wait 28280 2024-09-16T04:05:18.225 INFO:tasks.workunit.client.0.smithi154.stderr:+ for t in "${!pids[@]}" 2024-09-16T04:05:18.225 INFO:tasks.workunit.client.0.smithi154.stderr:+ pid=28543 2024-09-16T04:05:18.226 INFO:tasks.workunit.client.0.smithi154.stderr:+ wait 28543 2024-09-16T04:05:18.226 INFO:tasks.workunit.client.0.smithi154.stderr:+ for t in "${!pids[@]}" 2024-09-16T04:05:18.226 INFO:tasks.workunit.client.0.smithi154.stderr:+ pid=28552 2024-09-16T04:05:18.226 INFO:tasks.workunit.client.0.smithi154.stderr:+ wait 28552 2024-09-16T04:05:18.226 INFO:tasks.workunit.client.0.smithi154.stderr:+ for t in "${!pids[@]}" 2024-09-16T04:05:18.226 INFO:tasks.workunit.client.0.smithi154.stderr:+ pid=28196 2024-09-16T04:05:18.226 INFO:tasks.workunit.client.0.smithi154.stderr:+ wait 28196 2024-09-16T04:05:18.226 INFO:tasks.workunit.client.0.smithi154.stderr:+ for t in "${!pids[@]}" 2024-09-16T04:05:18.226 INFO:tasks.workunit.client.0.smithi154.stderr:+ pid=28461 2024-09-16T04:05:18.226 INFO:tasks.workunit.client.0.smithi154.stderr:+ wait 28461 2024-09-16T04:05:18.233 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.210+0000 7fe8bc868640 1 Processor -- start 2024-09-16T04:05:18.233 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.210+0000 7fe8bc868640 1 -- start start 2024-09-16T04:05:18.233 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.210+0000 7fe8bc868640 10 monclient: init 2024-09-16T04:05:18.233 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.210+0000 7fe8bc868640 10 monclient: _reopen_session rank -1 2024-09-16T04:05:18.233 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.210+0000 7fe8bc868640 10 monclient: _add_conns ranks=[4,0,1,3,2] 2024-09-16T04:05:18.233 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.210+0000 7fe8bc868640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fe8b80a2440 0x7fe8b80a2820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:05:18.233 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.210+0000 7fe8bc868640 10 monclient(hunting): picked mon.noname-c con 0x7fe8b80a2440 addr v2:172.21.15.154:3300/0 2024-09-16T04:05:18.233 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.210+0000 7fe8bc868640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe8b80a2d60 addr v1:172.21.15.42:6789/0 2024-09-16T04:05:18.233 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.210+0000 7fe8bc868640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe8b8135910 addr v1:172.21.15.154:6789/0 2024-09-16T04:05:18.233 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.210+0000 7fe8bc868640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe8b8071c60 con 0x7fe8b80a2d60 2024-09-16T04:05:18.233 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.210+0000 7fe8bc868640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe8b806c350 con 0x7fe8b8135910 2024-09-16T04:05:18.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.210+0000 7fe8bc868640 10 monclient(hunting): start opening mon connection 2024-09-16T04:05:18.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.210+0000 7fe8bc868640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fe8b806c5d0 con 0x7fe8b80a2440 2024-09-16T04:05:18.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.210+0000 7fe8bc868640 10 monclient(hunting): _renew_subs 2024-09-16T04:05:18.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.210+0000 7fe8bc868640 10 monclient(hunting): authenticate will time out at 2365.181408s 2024-09-16T04:05:18.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.210+0000 7fe8b5d9b640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fe8b80a2d60 0x7fe8b8135200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:40584/0 (socket says 172.21.15.42:40584) 2024-09-16T04:05:18.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.210+0000 7fe8b5d9b640 1 -- 172.21.15.42:0/1257332256 learned_addr learned my addr 172.21.15.42:0/1257332256 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T04:05:18.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.210+0000 7fe8b659c640 1 --2- 172.21.15.42:0/1257332256 >> v2:172.21.15.154:3300/0 conn(0x7fe8b80a2440 0x7fe8b80a2820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:05:18.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.210+0000 7fe8b559a640 1 -- 172.21.15.42:0/1257332256 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1992911518 0 0) 0x7fe8b8071c60 con 0x7fe8b80a2d60 2024-09-16T04:05:18.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.210+0000 7fe8b559a640 10 monclient(hunting): _negotiate 2024-09-16T04:05:18.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.210+0000 7fe8b559a640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:05:18.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.210+0000 7fe8b559a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:05:18.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.210+0000 7fe8b559a640 10 monclient(hunting): my global_id is 984300 2024-09-16T04:05:18.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.210+0000 7fe8b559a640 1 -- 172.21.15.42:0/1257332256 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe8a0003530 con 0x7fe8b80a2d60 2024-09-16T04:05:18.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.210+0000 7fe8b659c640 10 monclient(hunting): get_auth_request con 0x7fe8b80a2440 auth_method 0 2024-09-16T04:05:18.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.210+0000 7fe8b659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:05:18.235 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.210+0000 7fe8b659c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:05:18.235 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.210+0000 7fe8b659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:05:18.235 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.210+0000 7fe8b559a640 1 -- 172.21.15.42:0/1257332256 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1158036071 0 0) 0x7fe8a0003530 con 0x7fe8b80a2d60 2024-09-16T04:05:18.235 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.210+0000 7fe8b559a640 1 -- 172.21.15.42:0/1257332256 >> v2:172.21.15.154:3300/0 conn(0x7fe8b80a2440 msgr2=0x7fe8b80a2820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:05:18.235 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.210+0000 7fe8b559a640 1 --2- 172.21.15.42:0/1257332256 >> v2:172.21.15.154:3300/0 conn(0x7fe8b80a2440 0x7fe8b80a2820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:18.235 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.210+0000 7fe8b559a640 1 -- 172.21.15.42:0/1257332256 >> v1:172.21.15.154:6789/0 conn(0x7fe8b8135910 legacy=0x7fe8b813d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:05:18.235 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.210+0000 7fe8b559a640 10 monclient: _finish_hunting 0 2024-09-16T04:05:18.235 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.210+0000 7fe8b559a640 1 monclient: found mon.noname-a-legacy 2024-09-16T04:05:18.235 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.210+0000 7fe8b559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:05:18.235 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.210+0000 7fe8b559a640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.42:6789/0 2024-09-16T04:05:18.235 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.210+0000 7fe8b559a640 1 -- 172.21.15.42:0/1257332256 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8b80713c0 con 0x7fe8b80a2d60 2024-09-16T04:05:18.235 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.210+0000 7fe8b559a640 10 monclient: _finish_auth 0 2024-09-16T04:05:18.235 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.210+0000 7fe8b559a640 10 monclient: _check_auth_tickets 2024-09-16T04:05:18.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.210+0000 7fe8b559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:05:18.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.210+0000 7fe8b559a640 1 -- 172.21.15.42:0/1257332256 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2177413734 0 0) 0x7fe8ac001680 con 0x7fe8b80a2d60 2024-09-16T04:05:18.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.210+0000 7fe8b559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:05:18.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.210+0000 7fe8b659c640 1 --2- 172.21.15.42:0/1257332256 >> v2:172.21.15.154:3300/0 conn(0x7fe8b80a2440 0x7fe8b80a2820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T04:05:18.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.210+0000 7fe8b559a640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T04:05:18.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.210+0000 7fe8b559a640 10 monclient: dump: 2024-09-16T04:05:18.236 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:05:18.236 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:05:18.236 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:18.236 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:18.236 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:05:18.236 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:05:18.236 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:05:18.237 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:05:18.237 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:05:18.237 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:05:18.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.210+0000 7fe8b559a640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] but i'm connected to v1:172.21.15.42:6789/0, reconnecting 2024-09-16T04:05:18.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.210+0000 7fe8b559a640 10 monclient: _reopen_session rank -1 2024-09-16T04:05:18.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.210+0000 7fe8b559a640 1 -- 172.21.15.42:0/1257332256 >> v1:172.21.15.42:6789/0 conn(0x7fe8b80a2d60 legacy=0x7fe8b8135200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:05:18.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.210+0000 7fe8b559a640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:05:18.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.210+0000 7fe8b559a640 1 --2- 172.21.15.42:0/1257332256 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe8a0003a10 0x7fe8a0003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:05:18.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.210+0000 7fe8b559a640 10 monclient(hunting): picked mon.c con 0x7fe8a0003a10 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:05:18.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.210+0000 7fe8b559a640 1 --2- 172.21.15.42:0/1257332256 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe8a0004810 0x7fe8a0004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:05:18.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.210+0000 7fe8b559a640 10 monclient(hunting): picked mon.b con 0x7fe8a0004810 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:05:18.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.210+0000 7fe8b559a640 1 --2- 172.21.15.42:0/1257332256 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe8a00053d0 0x7fe8a000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:05:18.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.210+0000 7fe8b559a640 10 monclient(hunting): picked mon.a con 0x7fe8a00053d0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:05:18.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.210+0000 7fe8b559a640 10 monclient(hunting): start opening mon connection 2024-09-16T04:05:18.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.210+0000 7fe8b559a640 1 -- 172.21.15.42:0/1257332256 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fe8a000c280 con 0x7fe8a00053d0 2024-09-16T04:05:18.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.210+0000 7fe8b559a640 10 monclient(hunting): start opening mon connection 2024-09-16T04:05:18.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.210+0000 7fe8b559a640 1 -- 172.21.15.42:0/1257332256 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe8a000c430 con 0x7fe8a0004810 2024-09-16T04:05:18.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.210+0000 7fe8b559a640 10 monclient(hunting): start opening mon connection 2024-09-16T04:05:18.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.210+0000 7fe8b559a640 1 -- 172.21.15.42:0/1257332256 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fe8a000c630 con 0x7fe8a0003a10 2024-09-16T04:05:18.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.210+0000 7fe8b559a640 10 monclient(hunting): _renew_subs 2024-09-16T04:05:18.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.210+0000 7fe8b559a640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:05:18.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.210+0000 7fe8b659c640 1 --2- 172.21.15.42:0/1257332256 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe8a0003a10 0x7fe8a0003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:05:18.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.210+0000 7fe8b6d9d640 1 --2- 172.21.15.42:0/1257332256 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe8a00053d0 0x7fe8a000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:05:18.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.210+0000 7fe8b659c640 10 monclient(hunting): get_auth_request con 0x7fe8a0003a10 auth_method 0 2024-09-16T04:05:18.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.210+0000 7fe8b659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:05:18.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.210+0000 7fe8b659c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:05:18.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.210+0000 7fe8b659c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:05:18.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.210+0000 7fe8b6d9d640 10 monclient(hunting): get_auth_request con 0x7fe8a00053d0 auth_method 0 2024-09-16T04:05:18.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.210+0000 7fe8b6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:05:18.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.210+0000 7fe8b6d9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:05:18.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.210+0000 7fe8b6d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:05:18.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.210+0000 7fe8b5d9b640 1 --2- 172.21.15.42:0/1257332256 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe8a0004810 0x7fe8a0004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:05:18.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.210+0000 7fe8b659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:05:18.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.210+0000 7fe8b659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:05:18.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.210+0000 7fe8b659c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:05:18.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.210+0000 7fe8b5d9b640 10 monclient(hunting): get_auth_request con 0x7fe8a0004810 auth_method 0 2024-09-16T04:05:18.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.210+0000 7fe8b5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:05:18.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.210+0000 7fe8b5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:05:18.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.210+0000 7fe8b5d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:05:18.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.210+0000 7fe8b6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:05:18.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.210+0000 7fe8b6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:05:18.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.210+0000 7fe8b6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:05:18.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.210+0000 7fe8b659c640 10 monclient(hunting): handle_auth_done global_id 984300 payload 293 2024-09-16T04:05:18.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.210+0000 7fe8b659c640 1 -- 172.21.15.42:0/1257332256 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe8a0004810 msgr2=0x7fe8a0004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:05:18.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.210+0000 7fe8b659c640 1 --2- 172.21.15.42:0/1257332256 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe8a0004810 0x7fe8a0004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:18.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.210+0000 7fe8b659c640 1 -- 172.21.15.42:0/1257332256 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe8a00053d0 msgr2=0x7fe8a000b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:05:18.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.210+0000 7fe8b659c640 1 --2- 172.21.15.42:0/1257332256 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe8a00053d0 0x7fe8a000b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:18.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.210+0000 7fe8b659c640 10 monclient: _finish_hunting 0 2024-09-16T04:05:18.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.210+0000 7fe8b659c640 1 monclient: found mon.c 2024-09-16T04:05:18.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.210+0000 7fe8b659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:05:18.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.210+0000 7fe8b659c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:05:18.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.210+0000 7fe8b6d9d640 1 -- 172.21.15.42:0/1257332256 reap_dead start 2024-09-16T04:05:18.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.210+0000 7fe8b659c640 1 -- 172.21.15.42:0/1257332256 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8a0003530 con 0x7fe8a0003a10 2024-09-16T04:05:18.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.210+0000 7fe8b659c640 10 monclient: _finish_auth 0 2024-09-16T04:05:18.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.210+0000 7fe8b659c640 10 monclient: _check_auth_tickets 2024-09-16T04:05:18.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.210+0000 7fe8b659c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:05:18.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.210+0000 7fe8bc868640 5 monclient: authenticate success, global_id 984300 2024-09-16T04:05:18.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.210+0000 7fe8bc868640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:05:18.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.214+0000 7fe8b659c640 1 --2- 172.21.15.42:0/1257332256 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe8a0003a10 0x7fe8a0003e30 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fe8a80098d0 tx=0x7fe8a801c5b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=942670217e7ff1b1 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:05:18.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.214+0000 7fe8b559a640 1 -- 172.21.15.42:0/1257332256 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8a80b1070 con 0x7fe8a0003a10 2024-09-16T04:05:18.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.214+0000 7fe8b559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:05:18.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.214+0000 7fe8b559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:05:18.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.214+0000 7fe8b559a640 10 monclient: dump: 2024-09-16T04:05:18.241 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:05:18.241 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:05:18.241 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:18.241 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:18.242 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:05:18.242 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:05:18.242 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:05:18.242 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:05:18.242 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:05:18.242 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:05:18.242 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.214+0000 7fe8bc868640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:05:18.242 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.214+0000 7fe8b559a640 1 -- 172.21.15.42:0/1257332256 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe8a80a8050 con 0x7fe8a0003a10 2024-09-16T04:05:18.242 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.214+0000 7fe8b559a640 10 monclient: handle_config config(0 keys) 2024-09-16T04:05:18.242 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.214+0000 7fe8bc868640 10 monclient: get_monmap_and_config success 2024-09-16T04:05:18.242 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.214+0000 7fe8bc868640 4 set_mon_vals no callback set 2024-09-16T04:05:18.242 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.214+0000 7fe8b559a640 1 -- 172.21.15.42:0/1257332256 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8a800f040 con 0x7fe8a0003a10 2024-09-16T04:05:18.242 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.214+0000 7fe8b559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:05:18.242 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.214+0000 7fe8b559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:05:18.243 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.214+0000 7fe8b559a640 10 monclient: dump: 2024-09-16T04:05:18.243 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:05:18.243 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:05:18.243 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:18.243 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:18.243 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:05:18.243 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:05:18.243 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:05:18.243 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:05:18.243 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:05:18.243 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:05:18.243 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.214+0000 7fe8bc868640 10 monclient: shutdown 2024-09-16T04:05:18.243 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.214+0000 7fe8bc868640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:05:18.243 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.214+0000 7fe8bc868640 1 -- 172.21.15.42:0/1257332256 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe8a0003a10 msgr2=0x7fe8a0003e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:05:18.243 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.214+0000 7fe8bc868640 1 --2- 172.21.15.42:0/1257332256 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe8a0003a10 0x7fe8a0003e30 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fe8a80098d0 tx=0x7fe8a801c5b0 comp rx=0 tx=0).stop 2024-09-16T04:05:18.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.214+0000 7fe8bc868640 1 -- 172.21.15.42:0/1257332256 shutdown_connections 2024-09-16T04:05:18.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.214+0000 7fe8bc868640 1 --2- 172.21.15.42:0/1257332256 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe8a0003a10 0x7fe8a0003e30 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:18.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.214+0000 7fe8bc868640 1 -- 172.21.15.42:0/1257332256 >> 172.21.15.42:0/1257332256 conn(0x7fe8b80a3530 msgr2=0x7fe8b80a3910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:05:18.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.214+0000 7fe8bc868640 1 -- 172.21.15.42:0/1257332256 shutdown_connections 2024-09-16T04:05:18.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.214+0000 7fe8bc868640 1 -- 172.21.15.42:0/1257332256 wait complete. 2024-09-16T04:05:18.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.214+0000 7fe8bc868640 10 monclient: build_initial_monmap 2024-09-16T04:05:18.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.214+0000 7fe8bc868640 10 monclient: monmap: 2024-09-16T04:05:18.244 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:05:18.244 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:05:18.244 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:05:18.244 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:05:18.244 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:05:18.244 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:05:18.245 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:05:18.245 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:05:18.245 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:05:18.245 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:05:18.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.214+0000 7fe8bc868640 1 librados: starting msgr at 2024-09-16T04:05:18.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.214+0000 7fe8bc868640 1 librados: starting objecter 2024-09-16T04:05:18.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.214+0000 7fe8bc868640 1 Processor -- start 2024-09-16T04:05:18.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.214+0000 7fe8bc868640 1 -- start start 2024-09-16T04:05:18.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.214+0000 7fe8bc868640 1 librados: setting wanted keys 2024-09-16T04:05:18.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.214+0000 7fe8bc868640 1 librados: calling monclient init 2024-09-16T04:05:18.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.214+0000 7fe8bc868640 10 monclient: init 2024-09-16T04:05:18.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.214+0000 7fe8bc868640 10 monclient: _reopen_session rank -1 2024-09-16T04:05:18.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.214+0000 7fe8bc868640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:05:18.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.214+0000 7fe8bc868640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe8a0003a10 0x7fe8b8134620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:05:18.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.214+0000 7fe8bc868640 10 monclient(hunting): picked mon.noname-b con 0x7fe8a0003a10 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:05:18.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.214+0000 7fe8bc868640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe8b81cd590 0x7fe8b8134b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:05:18.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.214+0000 7fe8bc868640 10 monclient(hunting): picked mon.noname-a con 0x7fe8b81cd590 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:05:18.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.214+0000 7fe8bc868640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe8b811e0d0 0x7fe8b811e530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:05:18.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.214+0000 7fe8bc868640 10 monclient(hunting): picked mon.noname-c con 0x7fe8b811e0d0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:05:18.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.214+0000 7fe8bc868640 10 monclient(hunting): start opening mon connection 2024-09-16T04:05:18.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.214+0000 7fe8bc868640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fe8b8070c30 con 0x7fe8b81cd590 2024-09-16T04:05:18.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.214+0000 7fe8bc868640 10 monclient(hunting): start opening mon connection 2024-09-16T04:05:18.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.214+0000 7fe8bc868640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe8b8070ab0 con 0x7fe8a0003a10 2024-09-16T04:05:18.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.214+0000 7fe8bc868640 10 monclient(hunting): start opening mon connection 2024-09-16T04:05:18.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.214+0000 7fe8bc868640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fe8b8135200 con 0x7fe8b811e0d0 2024-09-16T04:05:18.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.214+0000 7fe8bc868640 10 monclient(hunting): _renew_subs 2024-09-16T04:05:18.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.214+0000 7fe8bc868640 10 monclient(hunting): authenticate will time out at 2365.187112s 2024-09-16T04:05:18.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.214+0000 7fe8b5d9b640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe8b81cd590 0x7fe8b8134b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:05:18.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.214+0000 7fe8b5d9b640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe8b81cd590 0x7fe8b8134b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:47618/0 (socket says 172.21.15.42:47618) 2024-09-16T04:05:18.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.214+0000 7fe8b5d9b640 1 -- 172.21.15.42:0/2291382600 learned_addr learned my addr 172.21.15.42:0/2291382600 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:05:18.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.214+0000 7fe8b5d9b640 10 monclient(hunting): get_auth_request con 0x7fe8b81cd590 auth_method 0 2024-09-16T04:05:18.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.214+0000 7fe8b5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:05:18.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.214+0000 7fe8b6d9d640 1 --2- 172.21.15.42:0/2291382600 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe8b811e0d0 0x7fe8b811e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:05:18.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.214+0000 7fe8b5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:05:18.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.214+0000 7fe8b5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:05:18.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.214+0000 7fe8b6d9d640 10 monclient(hunting): get_auth_request con 0x7fe8b811e0d0 auth_method 0 2024-09-16T04:05:18.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.214+0000 7fe8b6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:05:18.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.214+0000 7fe8b6d9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:05:18.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.214+0000 7fe8b6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:05:18.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.214+0000 7fe8b5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:05:18.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.214+0000 7fe8b5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:05:18.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.214+0000 7fe8b5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:05:18.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.214+0000 7fe8b6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:05:18.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.214+0000 7fe8b6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:05:18.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.214+0000 7fe8b6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:05:18.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.214+0000 7fe8b659c640 1 --2- 172.21.15.42:0/2291382600 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe8a0003a10 0x7fe8b8134620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:05:18.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.214+0000 7fe8b659c640 10 monclient(hunting): get_auth_request con 0x7fe8a0003a10 auth_method 0 2024-09-16T04:05:18.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.214+0000 7fe8b659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:05:18.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.214+0000 7fe8b659c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:05:18.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.214+0000 7fe8b659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:05:18.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.214+0000 7fe8b5d9b640 10 monclient(hunting): handle_auth_done global_id 984303 payload 824 2024-09-16T04:05:18.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.214+0000 7fe8b5d9b640 1 -- 172.21.15.42:0/2291382600 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe8b811e0d0 msgr2=0x7fe8b811e530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:05:18.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.214+0000 7fe8b5d9b640 1 --2- 172.21.15.42:0/2291382600 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe8b811e0d0 0x7fe8b811e530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:18.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.214+0000 7fe8b5d9b640 1 -- 172.21.15.42:0/2291382600 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe8a0003a10 msgr2=0x7fe8b8134620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:05:18.249 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.214+0000 7fe8b5d9b640 1 --2- 172.21.15.42:0/2291382600 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe8a0003a10 0x7fe8b8134620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:18.249 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.214+0000 7fe8b5d9b640 10 monclient: _finish_hunting 0 2024-09-16T04:05:18.249 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.214+0000 7fe8b5d9b640 1 monclient: found mon.noname-a 2024-09-16T04:05:18.249 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.214+0000 7fe8b5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:05:18.249 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.214+0000 7fe8b5d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:05:18.249 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.214+0000 7fe8b5d9b640 1 -- 172.21.15.42:0/2291382600 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8b8121d00 con 0x7fe8b81cd590 2024-09-16T04:05:18.249 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.214+0000 7fe8b6d9d640 1 --2- 172.21.15.42:0/2291382600 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe8b811e0d0 0x7fe8b811e530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:05:18.249 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.214+0000 7fe8b5d9b640 1 --2- 172.21.15.42:0/2291382600 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe8b81cd590 0x7fe8b8134b60 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fe8ac00e5a0 tx=0x7fe8ac00ea60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:05:18.249 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.214+0000 7fe8a6ffd640 10 client.?.objecter ms_handle_connect 0x7fe8b81cd590 2024-09-16T04:05:18.249 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.214+0000 7fe8a6ffd640 10 client.?.objecter resend_mon_ops 2024-09-16T04:05:18.249 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.214+0000 7fe8a6ffd640 1 -- 172.21.15.42:0/2291382600 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8ac012590 con 0x7fe8b81cd590 2024-09-16T04:05:18.249 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.214+0000 7fe8a6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:05:18.249 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.214+0000 7fe8a6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:05:18.250 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.214+0000 7fe8a6ffd640 10 monclient: dump: 2024-09-16T04:05:18.250 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:05:18.250 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:05:18.250 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:18.250 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:18.250 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:05:18.250 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:05:18.250 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:05:18.250 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:05:18.250 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:05:18.250 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:05:18.250 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.218+0000 7fe8a6ffd640 10 monclient: _finish_auth 0 2024-09-16T04:05:18.250 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.218+0000 7fe8a6ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:05:18.250 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.218+0000 7fe8a6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:05:18.251 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.218+0000 7fe8a6ffd640 1 -- 172.21.15.42:0/2291382600 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe8ac018070 con 0x7fe8b81cd590 2024-09-16T04:05:18.251 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.218+0000 7fe8bc868640 5 monclient: authenticate success, global_id 984303 2024-09-16T04:05:18.251 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.218+0000 7fe8bc868640 10 monclient: _renew_subs 2024-09-16T04:05:18.251 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.218+0000 7fe8bc868640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:05:18.251 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.218+0000 7fe8bc868640 1 -- 172.21.15.42:0/2291382600 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8b8121fe0 con 0x7fe8b81cd590 2024-09-16T04:05:18.251 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.218+0000 7fe8a6ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:05:18.251 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.218+0000 7fe8a6ffd640 1 -- 172.21.15.42:0/2291382600 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8ac012990 con 0x7fe8b81cd590 2024-09-16T04:05:18.251 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.218+0000 7fe8a6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:05:18.251 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.218+0000 7fe8a6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:05:18.251 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.218+0000 7fe8a6ffd640 10 monclient: dump: 2024-09-16T04:05:18.251 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:05:18.251 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:05:18.251 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:18.251 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:18.252 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:05:18.252 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:05:18.252 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:05:18.252 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:05:18.252 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:05:18.252 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:05:18.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.218+0000 7fe8b4d99640 4 set_mon_vals no callback set 2024-09-16T04:05:18.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.218+0000 7fe8bc868640 10 client.984303.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:05:18.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.218+0000 7fe8bc868640 10 monclient: _renew_subs 2024-09-16T04:05:18.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.218+0000 7fe8bc868640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:05:18.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.218+0000 7fe8bc868640 1 -- 172.21.15.42:0/2291382600 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8b8073ad0 con 0x7fe8b81cd590 2024-09-16T04:05:18.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.218+0000 7fe8bc868640 1 librados: init done 2024-09-16T04:05:18.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.218+0000 7fe8a6ffd640 1 -- 172.21.15.42:0/2291382600 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fe8ac012b30 con 0x7fe8b81cd590 2024-09-16T04:05:18.253 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.218+0000 7fe8bc868640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:05:18.253 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.218+0000 7fe8bc868640 1 --2- 172.21.15.42:0/2291382600 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe8b811ea70 0x7fe8b81bebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:05:18.253 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.218+0000 7fe8bc868640 10 monclient(hunting): start opening mon connection 2024-09-16T04:05:18.253 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.218+0000 7fe8bc868640 1 -- 172.21.15.42:0/2291382600 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe8b809fae0 con 0x7fe8b811ea70 2024-09-16T04:05:18.253 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.218+0000 7fe8a6ffd640 1 --2- 172.21.15.42:0/2291382600 >> v2:172.21.15.154:6812/254845342 conn(0x7fe898037840 0x7fe898039d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:05:18.253 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.218+0000 7fe8a6ffd640 1 -- 172.21.15.42:0/2291382600 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(706..706 src has 451..706) ==== 5040+0+0 (secure 0 0 0) 0x7fe8ac048880 con 0x7fe8b81cd590 2024-09-16T04:05:18.253 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.218+0000 7fe8a6ffd640 10 client.984303.objecter ms_dispatch 0x7fe8b8000ba0 osd_map(706..706 src has 451..706) 2024-09-16T04:05:18.253 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.218+0000 7fe8a6ffd640 3 client.984303.objecter handle_osd_map got epochs [706,706] > 0 2024-09-16T04:05:18.253 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.218+0000 7fe8a6ffd640 3 client.984303.objecter handle_osd_map decoding full epoch 706 2024-09-16T04:05:18.253 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.218+0000 7fe8a6ffd640 20 client.984303.objecter dump_active .. 0 homeless 2024-09-16T04:05:18.253 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.218+0000 7fe8b659c640 1 --2- 172.21.15.42:0/2291382600 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe8b811ea70 0x7fe8b81bebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:05:18.253 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.218+0000 7fe8b659c640 10 monclient: get_auth_request con 0x7fe8b811ea70 auth_method 0 2024-09-16T04:05:18.253 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.218+0000 7fe8b659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:05:18.254 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.218+0000 7fe8b659c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:05:18.254 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.218+0000 7fe8b659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:05:18.254 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.218+0000 7fe8b6d9d640 1 --2- 172.21.15.42:0/2291382600 >> v2:172.21.15.154:6812/254845342 conn(0x7fe898037840 0x7fe898039d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:05:18.254 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.218+0000 7fe8b6d9d640 10 monclient: get_auth_request con 0x7fe898037840 auth_method 0 2024-09-16T04:05:18.254 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.218+0000 7fe8b659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:05:18.254 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.218+0000 7fe8b659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:05:18.254 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.218+0000 7fe8b659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:05:18.254 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.218+0000 7fe8b659c640 10 monclient(hunting): handle_auth_done global_id 904915 payload 824 2024-09-16T04:05:18.254 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.218+0000 7fe8b659c640 1 -- 172.21.15.42:0/2291382600 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe8b8135590 con 0x7fe8b811ea70 2024-09-16T04:05:18.254 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.218+0000 7fe8a6ffd640 10 client.984303.objecter ms_handle_connect 0x7fe898037840 2024-09-16T04:05:18.254 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.218+0000 7fe8b6d9d640 1 --2- 172.21.15.42:0/2291382600 >> v2:172.21.15.154:6812/254845342 conn(0x7fe898037840 0x7fe898039d00 secure :-1 s=READY pgs=3847 cs=0 l=1 rev1=1 crypto rx=0x7fe8b80a3140 tx=0x7fe8b0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:05:18.254 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.218+0000 7fe8a6ffd640 10 client.984303.objecter ms_handle_connect 0x7fe8b811ea70 2024-09-16T04:05:18.254 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.218+0000 7fe8a6ffd640 10 client.984303.objecter resend_mon_ops 2024-09-16T04:05:18.255 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.218+0000 7fe8b659c640 1 --2- 172.21.15.42:0/2291382600 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe8b811ea70 0x7fe8b81bebe0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fe8a80098a0 tx=0x7fe8a0005840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:05:18.255 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.218+0000 7fe8a6ffd640 1 -- 172.21.15.42:0/2291382600 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8a80b1040 con 0x7fe8b811ea70 2024-09-16T04:05:18.255 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.218+0000 7fe8a6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:05:18.255 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.218+0000 7fe8a6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:05:18.255 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.218+0000 7fe8a6ffd640 10 monclient: dump: 2024-09-16T04:05:18.255 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:05:18.255 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:05:18.255 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:18.255 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:18.255 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:05:18.255 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:05:18.255 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:05:18.255 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:05:18.256 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:05:18.256 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:05:18.256 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.218+0000 7fe8a6ffd640 1 -- 172.21.15.42:0/2291382600 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fe8a80a8040 con 0x7fe8b811ea70 2024-09-16T04:05:18.256 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.218+0000 7fe8a6ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:05:18.256 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.218+0000 7fe8a6ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:05:18.256 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.218+0000 7fe8a6ffd640 1 -- 172.21.15.42:0/2291382600 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe8b811ea70 msgr2=0x7fe8b81bebe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:05:18.256 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.218+0000 7fe8a6ffd640 1 --2- 172.21.15.42:0/2291382600 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe8b811ea70 0x7fe8b81bebe0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fe8a80098a0 tx=0x7fe8a0005840 comp rx=0 tx=0).stop 2024-09-16T04:05:18.256 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.218+0000 7fe8a6ffd640 1 -- 172.21.15.42:0/2291382600 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe8b811ea70 msgr2=0x7fe8b81bebe0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:05:18.256 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.218+0000 7fe8a6ffd640 1 --2- 172.21.15.42:0/2291382600 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe8b811ea70 0x7fe8b81bebe0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:18.256 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.226+0000 7fe8bc868640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:05:18.256 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.226+0000 7fe8bc868640 1 --2- 172.21.15.42:0/2291382600 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe8b81bf120 0x7fe8b81bf530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:05:18.256 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.226+0000 7fe8bc868640 10 monclient(hunting): start opening mon connection 2024-09-16T04:05:18.256 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.226+0000 7fe8bc868640 1 -- 172.21.15.42:0/2291382600 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe8b81205d0 con 0x7fe8b81bf120 2024-09-16T04:05:18.257 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.226+0000 7fe8b659c640 1 --2- 172.21.15.42:0/2291382600 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe8b81bf120 0x7fe8b81bf530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:05:18.257 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.226+0000 7fe8b659c640 10 monclient: get_auth_request con 0x7fe8b81bf120 auth_method 0 2024-09-16T04:05:18.257 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.226+0000 7fe8b659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:05:18.257 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.226+0000 7fe8b659c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:05:18.257 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.226+0000 7fe8b659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:05:18.257 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.226+0000 7fe8b659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:05:18.257 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.226+0000 7fe8b659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:05:18.257 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.226+0000 7fe8b659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:05:18.257 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.226+0000 7fe8b659c640 10 monclient(hunting): handle_auth_done global_id 904918 payload 824 2024-09-16T04:05:18.257 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.226+0000 7fe8b659c640 1 -- 172.21.15.42:0/2291382600 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fe8b811ee50 con 0x7fe8b81bf120 2024-09-16T04:05:18.257 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.226+0000 7fe8b659c640 1 --2- 172.21.15.42:0/2291382600 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe8b81bf120 0x7fe8b81bf530 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fe8a80098d0 tx=0x7fe8a80a7000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:05:18.257 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.226+0000 7fe8a6ffd640 10 client.984303.objecter ms_handle_connect 0x7fe8b81bf120 2024-09-16T04:05:18.257 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.226+0000 7fe8a6ffd640 10 client.984303.objecter resend_mon_ops 2024-09-16T04:05:18.257 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.226+0000 7fe8a6ffd640 1 -- 172.21.15.42:0/2291382600 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8a80b1070 con 0x7fe8b81bf120 2024-09-16T04:05:18.258 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.226+0000 7fe8a6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:05:18.258 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.226+0000 7fe8a6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:05:18.258 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.226+0000 7fe8a6ffd640 10 monclient: dump: 2024-09-16T04:05:18.258 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:05:18.258 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:05:18.258 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:18.258 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:18.258 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:05:18.258 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:05:18.258 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:05:18.258 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:05:18.258 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:05:18.258 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:05:18.259 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.230+0000 7fe8a6ffd640 1 -- 172.21.15.42:0/2291382600 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4880 (secure 0 0 0) 0x7fe8a80b7c70 con 0x7fe8b81bf120 2024-09-16T04:05:18.259 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.230+0000 7fe8a6ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:05:18.259 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.230+0000 7fe8a6ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:05:18.259 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.230+0000 7fe8a6ffd640 1 -- 172.21.15.42:0/2291382600 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe8b81bf120 msgr2=0x7fe8b81bf530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:05:18.259 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.230+0000 7fe8a6ffd640 1 --2- 172.21.15.42:0/2291382600 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe8b81bf120 0x7fe8b81bf530 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fe8a80098d0 tx=0x7fe8a80a7000 comp rx=0 tx=0).stop 2024-09-16T04:05:18.259 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.230+0000 7fe8a6ffd640 1 -- 172.21.15.42:0/2291382600 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe8b81bf120 msgr2=0x7fe8b81bf530 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:05:18.260 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.230+0000 7fe8a6ffd640 1 --2- 172.21.15.42:0/2291382600 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe8b81bf120 0x7fe8b81bf530 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:18.260 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T04:05:18.260 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:05:18.260 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:05:18.260 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T04:05:18.260 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 216, 2024-09-16T04:05:18.260 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 446409, 2024-09-16T04:05:18.260 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T04:05:18.260 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T04:05:18.261 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T04:05:18.261 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T04:05:18.261 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:05:18.261 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 7, 2024-09-16T04:05:18.261 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:05:18.261 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T04:05:18.261 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T04:05:18.261 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:05:18.261 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:05:18.261 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:05:18.261 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:05:18.261 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:05:18.261 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:05:18.261 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:05:18.262 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:05:18.262 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:05:18.262 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:05:18.262 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:05:18.262 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:05:18.262 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:05:18.262 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T04:05:18.262 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:05:18.262 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:05:18.262 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:05:18.262 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:05:18.262 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:05:18.262 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:05:18.263 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:05:18.263 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:05:18.263 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:05:18.263 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:05:18.263 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:05:18.263 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:05:18.263 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:05:18.263 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T04:05:18.263 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T04:05:18.263 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T04:05:18.263 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T04:05:18.263 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T04:05:18.263 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T04:05:18.264 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:05:18.264 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:05:18.264 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T04:05:18.264 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T04:05:18.264 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T04:05:18.264 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T04:05:18.264 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T04:05:18.264 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T04:05:18.264 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T04:05:18.264 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:05:18.264 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T04:05:18.264 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:05:18.264 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:05:18.265 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:05:18.265 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:05:18.265 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:05:18.265 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:05:18.265 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:05:18.265 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:05:18.265 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:05:18.265 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:05:18.265 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:05:18.265 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:05:18.265 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T04:05:18.265 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:05:18.265 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T04:05:18.265 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:05:18.266 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:05:18.266 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:05:18.266 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:05:18.266 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:05:18.266 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:05:18.266 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:05:18.266 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T04:05:18.266 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:05:18.266 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:05:18.266 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:05:18.266 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:05:18.266 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T04:05:18.266 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:05:18.267 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:05:18.267 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:05:18.267 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:05:18.267 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T04:05:18.267 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T04:05:18.267 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:05:18.267 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:05:18.267 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:05:18.267 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:05:18.267 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:05:18.267 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:05:18.267 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:05:18.267 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:05:18.267 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:05:18.268 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:05:18.268 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:05:18.268 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T04:05:18.268 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:05:18.268 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:05:18.268 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:05:18.268 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:05:18.268 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T04:05:18.268 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:05:18.268 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:05:18.268 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:05:18.268 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:05:18.268 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T04:05:18.269 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T04:05:18.269 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:05:18.269 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:05:18.269 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:05:18.269 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:05:18.269 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:05:18.269 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:05:18.269 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:05:18.269 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:05:18.269 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:05:18.269 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:05:18.269 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:05:18.269 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T04:05:18.269 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:05:18.270 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:05:18.270 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:05:18.270 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:05:18.270 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T04:05:18.270 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:05:18.270 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:05:18.270 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:05:18.270 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:05:18.270 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T04:05:18.270 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T04:05:18.270 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:05:18.270 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:05:18.270 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:05:18.270 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:05:18.271 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:05:18.271 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:05:18.271 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T04:05:18.271 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T04:05:18.271 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:05:18.271 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:05:18.271 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:05:18.271 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:05:18.271 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:05:18.271 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:05:18.271 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T04:05:18.271 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:05:18.271 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:05:18.271 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:05:18.272 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 3 2024-09-16T04:05:18.272 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:05:18.272 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:05:18.272 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T04:05:18.272 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:05:18.272 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:05:18.272 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:05:18.272 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 5 2024-09-16T04:05:18.272 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:05:18.272 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:05:18.272 INFO:teuthology.orchestra.run.smithi042.stdout: "mgr": [ 2024-09-16T04:05:18.272 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:05:18.272 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:05:18.273 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:05:18.273 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:05:18.273 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:05:18.273 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:05:18.273 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:05:18.273 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T04:05:18.273 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T04:05:18.275 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.230+0000 7fe8bc868640 10 librados: watch_flush enter 2024-09-16T04:05:18.275 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.230+0000 7fe8bc868640 10 librados: watch_flush exit 2024-09-16T04:05:18.275 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.230+0000 7fe8bc868640 20 client.984303.objecter shutdown clearing up homeless session... 2024-09-16T04:05:18.275 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.230+0000 7fe8bc868640 10 client.984303.objecter successfully canceled tick 2024-09-16T04:05:18.275 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.230+0000 7fe8bc868640 1 -- 172.21.15.42:0/2291382600 >> v2:172.21.15.154:6812/254845342 conn(0x7fe898037840 msgr2=0x7fe898039d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:05:18.275 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.230+0000 7fe8bc868640 1 --2- 172.21.15.42:0/2291382600 >> v2:172.21.15.154:6812/254845342 conn(0x7fe898037840 0x7fe898039d00 secure :-1 s=READY pgs=3847 cs=0 l=1 rev1=1 crypto rx=0x7fe8b80a3140 tx=0x7fe8b0009040 comp rx=0 tx=0).stop 2024-09-16T04:05:18.275 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.230+0000 7fe8bc868640 10 monclient: shutdown 2024-09-16T04:05:18.275 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.230+0000 7fe8bc868640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:05:18.275 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.230+0000 7fe8bc868640 1 -- 172.21.15.42:0/2291382600 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe8b81cd590 msgr2=0x7fe8b8134b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:05:18.275 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.230+0000 7fe8bc868640 1 --2- 172.21.15.42:0/2291382600 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe8b81cd590 0x7fe8b8134b60 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fe8ac00e5a0 tx=0x7fe8ac00ea60 comp rx=0 tx=0).stop 2024-09-16T04:05:18.275 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.230+0000 7fe8b6d9d640 1 -- 172.21.15.42:0/2291382600 reap_dead start 2024-09-16T04:05:18.276 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.230+0000 7fe8bc868640 1 -- 172.21.15.42:0/2291382600 shutdown_connections 2024-09-16T04:05:18.276 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.230+0000 7fe8bc868640 1 --2- 172.21.15.42:0/2291382600 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe8b81cd590 0x7fe8b8134b60 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:18.276 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.230+0000 7fe8bc868640 1 -- 172.21.15.42:0/2291382600 >> 172.21.15.42:0/2291382600 conn(0x7fe8b80a3530 msgr2=0x7fe8b8072c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:05:18.276 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.230+0000 7fe8bc868640 1 -- 172.21.15.42:0/2291382600 shutdown_connections 2024-09-16T04:05:18.276 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.230+0000 7fe8bc868640 1 -- 172.21.15.42:0/2291382600 wait complete. 2024-09-16T04:05:18.276 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.230+0000 7fe8bc868640 1 librados: shutdown 2024-09-16T04:05:18.276 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:05:18.276 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-09-16T04:05:18.276 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-16T04:05:18.277 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-09-16T04:05:18.277 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-16T04:05:18.277 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-16T04:05:18.277 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T04:05:18.289 INFO:tasks.ceph.mon.c:Stopped 2024-09-16T04:05:18.290 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T04:05:18.290 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:05:18.447 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.442+0000 7fd48bc08640 10 monclient: get_monmap_and_config 2024-09-16T04:05:18.448 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.442+0000 7fd48bc08640 10 monclient: build_initial_monmap 2024-09-16T04:05:18.448 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.442+0000 7fd48bc08640 10 monclient: monmap: 2024-09-16T04:05:18.448 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:05:18.448 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:05:18.448 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:05:18.446350+0000 2024-09-16T04:05:18.448 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:05:18.446350+0000 2024-09-16T04:05:18.448 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:05:18.448 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:05:18.448 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:05:18.448 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:05:18.449 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:05:18.449 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:05:18.449 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:05:18.449 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:05:18.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.442+0000 7fd48bc08640 1 Processor -- start 2024-09-16T04:05:18.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.442+0000 7fd48bc08640 1 -- start start 2024-09-16T04:05:18.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.442+0000 7fd48bc08640 10 monclient: init 2024-09-16T04:05:18.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.446+0000 7fd48bc08640 10 monclient: _reopen_session rank -1 2024-09-16T04:05:18.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.446+0000 7fd48bc08640 10 monclient: _add_conns ranks=[1,4,2,0,3] 2024-09-16T04:05:18.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.446+0000 7fd48bc08640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd48409ca60 addr v1:172.21.15.154:6789/0 2024-09-16T04:05:18.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.446+0000 7fd48bc08640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fd48409f260 0x7fd48409db70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:05:18.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.446+0000 7fd48bc08640 10 monclient(hunting): picked mon.noname-c con 0x7fd48409f260 addr v2:172.21.15.154:3300/0 2024-09-16T04:05:18.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.446+0000 7fd48bc08640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd484139760 0x7fd48413bb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:05:18.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.446+0000 7fd48bc08640 10 monclient(hunting): picked mon.noname-b con 0x7fd484139760 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:05:18.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.446+0000 7fd48bc08640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd4840a0050 con 0x7fd48409ca60 2024-09-16T04:05:18.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.446+0000 7fd48bc08640 10 monclient(hunting): start opening mon connection 2024-09-16T04:05:18.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.446+0000 7fd48bc08640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fd48409ce40 con 0x7fd48409f260 2024-09-16T04:05:18.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.446+0000 7fd48bc08640 10 monclient(hunting): start opening mon connection 2024-09-16T04:05:18.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.446+0000 7fd48bc08640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fd48409e880 con 0x7fd484139760 2024-09-16T04:05:18.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.446+0000 7fd48bc08640 10 monclient(hunting): _renew_subs 2024-09-16T04:05:18.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.446+0000 7fd48bc08640 10 monclient(hunting): authenticate will time out at 2365.416624s 2024-09-16T04:05:18.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.446+0000 7fd48a1a5640 1 -- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd484139760 msgr2=0x7fd48413bb50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T04:05:18.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.446+0000 7fd48a1a5640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd484139760 0x7fd48413bb50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:05:18.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.446+0000 7fd4899a4640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7fd48409ca60 0x7fd48409d460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.42:51450/0 (socket says 172.21.15.42:51450) 2024-09-16T04:05:18.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.446+0000 7fd4899a4640 1 -- 172.21.15.42:0/3043580774 learned_addr learned my addr 172.21.15.42:0/3043580774 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T04:05:18.454 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.446+0000 7fd4891a3640 1 --2- 172.21.15.42:0/3043580774 >> v2:172.21.15.154:3300/0 conn(0x7fd48409f260 0x7fd48409db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:05:18.454 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.446+0000 7fd4891a3640 10 monclient(hunting): get_auth_request con 0x7fd48409f260 auth_method 0 2024-09-16T04:05:18.454 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.446+0000 7fd4891a3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:05:18.454 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.446+0000 7fd4891a3640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:05:18.454 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.446+0000 7fd4891a3640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:05:18.454 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.446+0000 7fd4889a2640 1 -- 172.21.15.42:0/3043580774 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 907126352 0 0) 0x7fd4840a0050 con 0x7fd48409ca60 2024-09-16T04:05:18.454 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.446+0000 7fd4889a2640 10 monclient(hunting): _negotiate 2024-09-16T04:05:18.454 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.446+0000 7fd4889a2640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:05:18.454 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.446+0000 7fd4889a2640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:05:18.454 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.446+0000 7fd4889a2640 10 monclient(hunting): my global_id is 904921 2024-09-16T04:05:18.454 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.446+0000 7fd4889a2640 1 -- 172.21.15.42:0/3043580774 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd470003530 con 0x7fd48409ca60 2024-09-16T04:05:18.455 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.446+0000 7fd4891a3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:05:18.455 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.446+0000 7fd4891a3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:05:18.455 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.446+0000 7fd4891a3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:05:18.455 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.446+0000 7fd4891a3640 10 monclient(hunting): handle_auth_done global_id 904924 payload 274 2024-09-16T04:05:18.455 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.446+0000 7fd4889a2640 1 -- 172.21.15.42:0/3043580774 <== mon.1 v1:172.21.15.154:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2478841351 0 0) 0x7fd470003530 con 0x7fd48409ca60 2024-09-16T04:05:18.455 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.446+0000 7fd4891a3640 1 -- 172.21.15.42:0/3043580774 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd484139760 msgr2=0x7fd48413bb50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:05:18.455 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.446+0000 7fd4891a3640 1 --2- 172.21.15.42:0/3043580774 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd484139760 0x7fd48413bb50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:18.455 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.446+0000 7fd4891a3640 1 -- 172.21.15.42:0/3043580774 >> v1:172.21.15.154:6789/0 conn(0x7fd48409ca60 legacy=0x7fd48409d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:05:18.455 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.446+0000 7fd4891a3640 10 monclient: _finish_hunting 0 2024-09-16T04:05:18.455 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.446+0000 7fd4891a3640 1 monclient: found mon.noname-c 2024-09-16T04:05:18.455 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.446+0000 7fd4891a3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:05:18.456 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.446+0000 7fd4891a3640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.154:3300/0 2024-09-16T04:05:18.456 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.446+0000 7fd4891a3640 1 -- 172.21.15.42:0/3043580774 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd48409e5a0 con 0x7fd48409f260 2024-09-16T04:05:18.456 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.446+0000 7fd4889a2640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T04:05:18.463 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.450+0000 7fd4891a3640 1 --2- 172.21.15.42:0/3043580774 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd48409f260 0x7fd48409db70 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fd4740097f0 tx=0x7fd47401c2a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b1b87b5fc6788d3d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:05:18.463 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.450+0000 7fd4889a2640 1 -- 172.21.15.42:0/3043580774 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4740b0070 con 0x7fd48409f260 2024-09-16T04:05:18.463 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.450+0000 7fd4889a2640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:05:18.463 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.450+0000 7fd4889a2640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:05:18.463 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.450+0000 7fd4889a2640 10 monclient: dump: 2024-09-16T04:05:18.463 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:05:18.463 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:05:18.464 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:18.464 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:18.464 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:05:18.464 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:05:18.464 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:05:18.464 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:05:18.464 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:05:18.464 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:05:18.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.450+0000 7fd4889a2640 10 monclient: _finish_auth 0 2024-09-16T04:05:18.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.450+0000 7fd4889a2640 10 monclient: _check_auth_tickets 2024-09-16T04:05:18.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.450+0000 7fd4889a2640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:05:18.465 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.450+0000 7fd48bc08640 5 monclient: authenticate success, global_id 904924 2024-09-16T04:05:18.465 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.450+0000 7fd48bc08640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:05:18.465 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.450+0000 7fd4889a2640 1 -- 172.21.15.42:0/3043580774 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd4740ab400 con 0x7fd48409f260 2024-09-16T04:05:18.465 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.450+0000 7fd4889a2640 10 monclient: handle_config config(0 keys) 2024-09-16T04:05:18.465 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.450+0000 7fd4889a2640 1 -- 172.21.15.42:0/3043580774 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd480001680 con 0x7fd48409f260 2024-09-16T04:05:18.465 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.450+0000 7fd48bc08640 10 monclient: get_monmap_and_config success 2024-09-16T04:05:18.465 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.450+0000 7fd48bc08640 4 set_mon_vals no callback set 2024-09-16T04:05:18.465 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.450+0000 7fd4889a2640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:05:18.465 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.450+0000 7fd4889a2640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:05:18.465 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.450+0000 7fd4889a2640 10 monclient: dump: 2024-09-16T04:05:18.465 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:05:18.466 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:05:18.466 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:18.466 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:18.466 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:05:18.466 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:05:18.466 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:05:18.466 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:05:18.466 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:05:18.466 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:05:18.466 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.450+0000 7fd48bc08640 10 monclient: shutdown 2024-09-16T04:05:18.466 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.450+0000 7fd48bc08640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:05:18.467 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.450+0000 7fd48bc08640 1 -- 172.21.15.42:0/3043580774 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd48409f260 msgr2=0x7fd48409db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:05:18.467 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.450+0000 7fd48bc08640 1 --2- 172.21.15.42:0/3043580774 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd48409f260 0x7fd48409db70 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fd4740097f0 tx=0x7fd47401c2a0 comp rx=0 tx=0).stop 2024-09-16T04:05:18.467 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.450+0000 7fd48bc08640 1 -- 172.21.15.42:0/3043580774 shutdown_connections 2024-09-16T04:05:18.467 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.450+0000 7fd48bc08640 1 --2- 172.21.15.42:0/3043580774 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd484139760 0x7fd48413bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:18.467 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.450+0000 7fd48bc08640 1 --2- 172.21.15.42:0/3043580774 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd48409f260 0x7fd48409db70 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:18.467 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.450+0000 7fd48bc08640 1 -- 172.21.15.42:0/3043580774 >> 172.21.15.42:0/3043580774 conn(0x7fd48406bc40 msgr2=0x7fd48406c050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:05:18.467 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.450+0000 7fd48bc08640 1 -- 172.21.15.42:0/3043580774 shutdown_connections 2024-09-16T04:05:18.467 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.450+0000 7fd48bc08640 1 -- 172.21.15.42:0/3043580774 wait complete. 2024-09-16T04:05:18.467 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.450+0000 7fd48bc08640 10 monclient: build_initial_monmap 2024-09-16T04:05:18.467 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.450+0000 7fd48bc08640 10 monclient: monmap: 2024-09-16T04:05:18.468 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:05:18.468 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:05:18.468 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:05:18.468 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:05:18.468 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:05:18.468 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:05:18.468 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:05:18.468 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:05:18.468 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:05:18.468 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:05:18.468 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.450+0000 7fd48bc08640 1 librados: starting msgr at 2024-09-16T04:05:18.469 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.450+0000 7fd48bc08640 1 librados: starting objecter 2024-09-16T04:05:18.469 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.450+0000 7fd48bc08640 1 Processor -- start 2024-09-16T04:05:18.469 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.450+0000 7fd48bc08640 1 -- start start 2024-09-16T04:05:18.469 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.450+0000 7fd48bc08640 1 librados: setting wanted keys 2024-09-16T04:05:18.469 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.450+0000 7fd48bc08640 1 librados: calling monclient init 2024-09-16T04:05:18.469 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.450+0000 7fd48bc08640 10 monclient: init 2024-09-16T04:05:18.469 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.450+0000 7fd48bc08640 10 monclient: _reopen_session rank -1 2024-09-16T04:05:18.469 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.450+0000 7fd48bc08640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:05:18.469 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.450+0000 7fd48bc08640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd48409ca60 0x7fd4841ca810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:05:18.469 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.450+0000 7fd48bc08640 10 monclient(hunting): picked mon.noname-b con 0x7fd48409ca60 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:05:18.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.450+0000 7fd48bc08640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd48409f260 0x7fd4841389d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:05:18.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.450+0000 7fd48bc08640 10 monclient(hunting): picked mon.noname-c con 0x7fd48409f260 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:05:18.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.450+0000 7fd48bc08640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd484139760 0x7fd484138f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:05:18.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.450+0000 7fd48bc08640 10 monclient(hunting): picked mon.noname-a con 0x7fd484139760 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:05:18.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.450+0000 7fd48bc08640 10 monclient(hunting): start opening mon connection 2024-09-16T04:05:18.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.450+0000 7fd48bc08640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fd48409e230 con 0x7fd484139760 2024-09-16T04:05:18.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.450+0000 7fd48bc08640 10 monclient(hunting): start opening mon connection 2024-09-16T04:05:18.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.450+0000 7fd48bc08640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd48409e0b0 con 0x7fd48409ca60 2024-09-16T04:05:18.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.450+0000 7fd48bc08640 10 monclient(hunting): start opening mon connection 2024-09-16T04:05:18.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.450+0000 7fd48bc08640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fd4840a0960 con 0x7fd48409f260 2024-09-16T04:05:18.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.450+0000 7fd48bc08640 10 monclient(hunting): _renew_subs 2024-09-16T04:05:18.471 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.450+0000 7fd48bc08640 10 monclient(hunting): authenticate will time out at 2365.423444s 2024-09-16T04:05:18.471 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.450+0000 7fd4891a3640 1 -- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd48409f260 msgr2=0x7fd4841389d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T04:05:18.471 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.450+0000 7fd4891a3640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd48409f260 0x7fd4841389d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:05:18.471 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.450+0000 7fd48a1a5640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd484139760 0x7fd484138f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:05:18.471 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.450+0000 7fd48a1a5640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd484139760 0x7fd484138f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:47624/0 (socket says 172.21.15.42:47624) 2024-09-16T04:05:18.471 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.450+0000 7fd48a1a5640 1 -- 172.21.15.42:0/883828911 learned_addr learned my addr 172.21.15.42:0/883828911 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:05:18.471 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.450+0000 7fd48a1a5640 10 monclient(hunting): get_auth_request con 0x7fd484139760 auth_method 0 2024-09-16T04:05:18.471 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.450+0000 7fd48a1a5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:05:18.471 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.450+0000 7fd48a1a5640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:05:18.471 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.450+0000 7fd48a1a5640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:05:18.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.450+0000 7fd48a1a5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:05:18.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.450+0000 7fd48a1a5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:05:18.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.450+0000 7fd4899a4640 1 --2- 172.21.15.42:0/883828911 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd48409ca60 0x7fd4841ca810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:05:18.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.450+0000 7fd48a1a5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:05:18.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.450+0000 7fd4899a4640 10 monclient(hunting): get_auth_request con 0x7fd48409ca60 auth_method 0 2024-09-16T04:05:18.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.450+0000 7fd4899a4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:05:18.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.450+0000 7fd4899a4640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:05:18.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.450+0000 7fd4899a4640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:05:18.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.454+0000 7fd4899a4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:05:18.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.454+0000 7fd4899a4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:05:18.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.454+0000 7fd4899a4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:05:18.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.454+0000 7fd48a1a5640 10 monclient(hunting): handle_auth_done global_id 984306 payload 824 2024-09-16T04:05:18.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.454+0000 7fd48a1a5640 1 -- 172.21.15.42:0/883828911 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd48409f260 msgr2=0x7fd4841389d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:05:18.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.454+0000 7fd48a1a5640 1 --2- 172.21.15.42:0/883828911 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd48409f260 0x7fd4841389d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:18.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.454+0000 7fd48a1a5640 1 -- 172.21.15.42:0/883828911 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd48409ca60 msgr2=0x7fd4841ca810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:05:18.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.454+0000 7fd48a1a5640 1 --2- 172.21.15.42:0/883828911 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd48409ca60 0x7fd4841ca810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:18.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.454+0000 7fd48a1a5640 10 monclient: _finish_hunting 0 2024-09-16T04:05:18.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.454+0000 7fd48a1a5640 1 monclient: found mon.noname-a 2024-09-16T04:05:18.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.454+0000 7fd48a1a5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:05:18.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.454+0000 7fd48a1a5640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:05:18.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.454+0000 7fd48a1a5640 1 -- 172.21.15.42:0/883828911 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd484122350 con 0x7fd484139760 2024-09-16T04:05:18.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.454+0000 7fd48a1a5640 1 --2- 172.21.15.42:0/883828911 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd484139760 0x7fd484138f10 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fd47c00be20 tx=0x7fd47c009f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:05:18.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.454+0000 7fd47a7fc640 10 client.?.objecter ms_handle_connect 0x7fd484139760 2024-09-16T04:05:18.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.454+0000 7fd47a7fc640 10 client.?.objecter resend_mon_ops 2024-09-16T04:05:18.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.454+0000 7fd47a7fc640 1 -- 172.21.15.42:0/883828911 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd47c016430 con 0x7fd484139760 2024-09-16T04:05:18.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.454+0000 7fd47a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:05:18.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.454+0000 7fd47a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:05:18.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.454+0000 7fd47a7fc640 10 monclient: dump: 2024-09-16T04:05:18.474 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:05:18.474 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:05:18.474 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:18.474 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:18.474 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:05:18.474 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:05:18.475 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:05:18.475 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:05:18.475 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:05:18.475 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:05:18.475 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.454+0000 7fd47a7fc640 10 monclient: _finish_auth 0 2024-09-16T04:05:18.475 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.454+0000 7fd47a7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:05:18.475 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.454+0000 7fd47a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:05:18.475 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.454+0000 7fd47a7fc640 1 -- 172.21.15.42:0/883828911 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd47c010030 con 0x7fd484139760 2024-09-16T04:05:18.475 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.454+0000 7fd47a7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:05:18.475 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.454+0000 7fd47a7fc640 1 -- 172.21.15.42:0/883828911 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd47c016870 con 0x7fd484139760 2024-09-16T04:05:18.475 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.454+0000 7fd47a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:05:18.476 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.454+0000 7fd47a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:05:18.476 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.454+0000 7fd47a7fc640 10 monclient: dump: 2024-09-16T04:05:18.476 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:05:18.476 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:05:18.476 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:18.476 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:18.476 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:05:18.476 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:05:18.476 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:05:18.476 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:05:18.476 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:05:18.477 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:05:18.477 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.454+0000 7fd48bc08640 5 monclient: authenticate success, global_id 984306 2024-09-16T04:05:18.477 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.454+0000 7fd48bc08640 10 monclient: _renew_subs 2024-09-16T04:05:18.477 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.454+0000 7fd47bfff640 4 set_mon_vals no callback set 2024-09-16T04:05:18.477 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.454+0000 7fd48bc08640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:05:18.477 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.454+0000 7fd48bc08640 1 -- 172.21.15.42:0/883828911 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4840a4f90 con 0x7fd484139760 2024-09-16T04:05:18.477 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.454+0000 7fd48bc08640 10 client.984306.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:05:18.477 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.454+0000 7fd48bc08640 10 monclient: _renew_subs 2024-09-16T04:05:18.477 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.454+0000 7fd48bc08640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:05:18.477 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.454+0000 7fd48bc08640 1 -- 172.21.15.42:0/883828911 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd48409d150 con 0x7fd484139760 2024-09-16T04:05:18.477 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.454+0000 7fd48bc08640 1 librados: init done 2024-09-16T04:05:18.477 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.454+0000 7fd47a7fc640 1 -- 172.21.15.42:0/883828911 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fd47c003d10 con 0x7fd484139760 2024-09-16T04:05:18.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.454+0000 7fd48bc08640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:05:18.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.454+0000 7fd48bc08640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:05:18.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.454+0000 7fd48bc08640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:05:18.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.454+0000 7fd48bc08640 1 -- 172.21.15.42:0/883828911 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd48409d460 con 0x7fd484139760 2024-09-16T04:05:18.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.454+0000 7fd47a7fc640 1 --2- 172.21.15.42:0/883828911 >> v2:172.21.15.154:6812/254845342 conn(0x7fd4680377c0 0x7fd468039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:05:18.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.454+0000 7fd47a7fc640 1 -- 172.21.15.42:0/883828911 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(706..706 src has 451..706) ==== 5040+0+0 (secure 0 0 0) 0x7fd47c048e70 con 0x7fd484139760 2024-09-16T04:05:18.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.454+0000 7fd47a7fc640 10 client.984306.objecter ms_dispatch 0x7fd484000ba0 osd_map(706..706 src has 451..706) 2024-09-16T04:05:18.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.454+0000 7fd47a7fc640 3 client.984306.objecter handle_osd_map got epochs [706,706] > 0 2024-09-16T04:05:18.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.454+0000 7fd47a7fc640 3 client.984306.objecter handle_osd_map decoding full epoch 706 2024-09-16T04:05:18.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.454+0000 7fd47a7fc640 20 client.984306.objecter dump_active .. 0 homeless 2024-09-16T04:05:18.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.454+0000 7fd4899a4640 1 --2- 172.21.15.42:0/883828911 >> v2:172.21.15.154:6812/254845342 conn(0x7fd4680377c0 0x7fd468039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:05:18.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.454+0000 7fd4899a4640 10 monclient: get_auth_request con 0x7fd4680377c0 auth_method 0 2024-09-16T04:05:18.479 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.454+0000 7fd4899a4640 1 --2- 172.21.15.42:0/883828911 >> v2:172.21.15.154:6812/254845342 conn(0x7fd4680377c0 0x7fd468039c80 secure :-1 s=READY pgs=3848 cs=0 l=1 rev1=1 crypto rx=0x7fd4800011f0 tx=0x7fd480001700 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:05:18.479 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.454+0000 7fd47a7fc640 10 client.984306.objecter ms_handle_connect 0x7fd4680377c0 2024-09-16T04:05:18.479 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.458+0000 7fd47a7fc640 1 -- 172.21.15.42:0/883828911 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7fd47c01d300 con 0x7fd484139760 2024-09-16T04:05:18.479 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.458+0000 7fd47a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:05:18.479 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.458+0000 7fd47a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:05:18.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.682+0000 7fd48bc08640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:05:18.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.682+0000 7fd48bc08640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:05:18.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.682+0000 7fd48bc08640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:05:18.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.682+0000 7fd48bc08640 1 -- 172.21.15.42:0/883828911 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fd484120060 con 0x7fd484139760 2024-09-16T04:05:18.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.682+0000 7fd47a7fc640 1 -- 172.21.15.42:0/883828911 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7fd47c0477d0 con 0x7fd484139760 2024-09-16T04:05:18.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.682+0000 7fd47a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:05:18.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.682+0000 7fd47a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:05:18.690 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":216,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:05:18.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.686+0000 7fd48bc08640 10 librados: watch_flush enter 2024-09-16T04:05:18.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.686+0000 7fd48bc08640 10 librados: watch_flush exit 2024-09-16T04:05:18.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.686+0000 7fd48bc08640 20 client.984306.objecter shutdown clearing up homeless session... 2024-09-16T04:05:18.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.686+0000 7fd48bc08640 10 client.984306.objecter successfully canceled tick 2024-09-16T04:05:18.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.686+0000 7fd48bc08640 1 -- 172.21.15.42:0/883828911 >> v2:172.21.15.154:6812/254845342 conn(0x7fd4680377c0 msgr2=0x7fd468039c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:05:18.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.686+0000 7fd48bc08640 1 --2- 172.21.15.42:0/883828911 >> v2:172.21.15.154:6812/254845342 conn(0x7fd4680377c0 0x7fd468039c80 secure :-1 s=READY pgs=3848 cs=0 l=1 rev1=1 crypto rx=0x7fd4800011f0 tx=0x7fd480001700 comp rx=0 tx=0).stop 2024-09-16T04:05:18.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.686+0000 7fd48bc08640 10 monclient: shutdown 2024-09-16T04:05:18.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.686+0000 7fd48bc08640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:05:18.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.686+0000 7fd48bc08640 1 -- 172.21.15.42:0/883828911 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd484139760 msgr2=0x7fd484138f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:05:18.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.686+0000 7fd48bc08640 1 --2- 172.21.15.42:0/883828911 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd484139760 0x7fd484138f10 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fd47c00be20 tx=0x7fd47c009f10 comp rx=0 tx=0).stop 2024-09-16T04:05:18.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.686+0000 7fd48bc08640 1 -- 172.21.15.42:0/883828911 shutdown_connections 2024-09-16T04:05:18.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.686+0000 7fd48bc08640 1 --2- 172.21.15.42:0/883828911 >> v2:172.21.15.154:6812/254845342 conn(0x7fd4680377c0 0x7fd468039c80 unknown :-1 s=CLOSED pgs=3848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:18.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.686+0000 7fd48bc08640 1 --2- 172.21.15.42:0/883828911 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd484139760 0x7fd484138f10 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:18.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.686+0000 7fd48bc08640 1 --2- 172.21.15.42:0/883828911 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd48409f260 0x7fd4841389d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:18.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.686+0000 7fd48bc08640 1 --2- 172.21.15.42:0/883828911 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd48409ca60 0x7fd4841ca810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:18.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.686+0000 7fd48bc08640 1 -- 172.21.15.42:0/883828911 >> 172.21.15.42:0/883828911 conn(0x7fd48406bc40 msgr2=0x7fd48413a600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:05:18.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.686+0000 7fd48bc08640 1 -- 172.21.15.42:0/883828911 shutdown_connections 2024-09-16T04:05:18.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.686+0000 7fd48bc08640 1 -- 172.21.15.42:0/883828911 wait complete. 2024-09-16T04:05:18.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:18.686+0000 7fd48bc08640 1 librados: shutdown 2024-09-16T04:05:21.709 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:05:21.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.834+0000 7f28b687b640 10 monclient: get_monmap_and_config 2024-09-16T04:05:21.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.834+0000 7f28b687b640 10 monclient: build_initial_monmap 2024-09-16T04:05:21.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.834+0000 7f28b687b640 10 monclient: monmap: 2024-09-16T04:05:21.839 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:05:21.839 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:05:21.839 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:05:21.837772+0000 2024-09-16T04:05:21.840 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:05:21.837772+0000 2024-09-16T04:05:21.840 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:05:21.840 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:05:21.840 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:05:21.840 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:05:21.840 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:05:21.840 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:05:21.840 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:05:21.840 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:05:21.843 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.834+0000 7f28b687b640 1 Processor -- start 2024-09-16T04:05:21.843 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.834+0000 7f28b687b640 1 -- start start 2024-09-16T04:05:21.843 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.834+0000 7f28b687b640 10 monclient: init 2024-09-16T04:05:21.843 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.834+0000 7f28b687b640 10 monclient: _reopen_session rank -1 2024-09-16T04:05:21.843 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.834+0000 7f28b687b640 10 monclient: _add_conns ranks=[4,1,0,3,2] 2024-09-16T04:05:21.843 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.834+0000 7f28b687b640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f28b0130b20 0x7f28b0130f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:05:21.843 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.834+0000 7f28b687b640 10 monclient(hunting): picked mon.noname-c con 0x7f28b0130b20 addr v2:172.21.15.154:3300/0 2024-09-16T04:05:21.844 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.834+0000 7f28b687b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f28b00a2250 addr v1:172.21.15.154:6789/0 2024-09-16T04:05:21.844 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.834+0000 7f28b687b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f28b0073420 addr v1:172.21.15.42:6789/0 2024-09-16T04:05:21.844 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.834+0000 7f28b687b640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f28b00a12b0 con 0x7f28b0073420 2024-09-16T04:05:21.844 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.834+0000 7f28b687b640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f28b00a1590 con 0x7f28b00a2250 2024-09-16T04:05:21.844 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.834+0000 7f28b687b640 10 monclient(hunting): start opening mon connection 2024-09-16T04:05:21.844 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.834+0000 7f28b687b640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f28b00a1070 con 0x7f28b0130b20 2024-09-16T04:05:21.844 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.834+0000 7f28b687b640 10 monclient(hunting): _renew_subs 2024-09-16T04:05:21.844 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.834+0000 7f28b687b640 10 monclient(hunting): authenticate will time out at 2368.807167s 2024-09-16T04:05:21.844 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.834+0000 7f28b4e18640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f28b0073420 0x7f28b013f240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:40586/0 (socket says 172.21.15.42:40586) 2024-09-16T04:05:21.844 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.834+0000 7f28b4e18640 1 -- 172.21.15.42:0/2920671596 learned_addr learned my addr 172.21.15.42:0/2920671596 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T04:05:21.844 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.834+0000 7f28aeffd640 1 -- 172.21.15.42:0/2920671596 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1539558452 0 0) 0x7f28b00a12b0 con 0x7f28b0073420 2024-09-16T04:05:21.844 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.834+0000 7f28aeffd640 10 monclient(hunting): _negotiate 2024-09-16T04:05:21.844 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.834+0000 7f28aeffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:05:21.845 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.834+0000 7f28aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:05:21.845 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.834+0000 7f28aeffd640 10 monclient(hunting): my global_id is 984309 2024-09-16T04:05:21.845 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.834+0000 7f28aeffd640 1 -- 172.21.15.42:0/2920671596 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2898003530 con 0x7f28b0073420 2024-09-16T04:05:21.845 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.834+0000 7f28affff640 1 --2- 172.21.15.42:0/2920671596 >> v2:172.21.15.154:3300/0 conn(0x7f28b0130b20 0x7f28b0130f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:05:21.845 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.834+0000 7f28affff640 10 monclient(hunting): get_auth_request con 0x7f28b0130b20 auth_method 0 2024-09-16T04:05:21.845 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.834+0000 7f28affff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:05:21.845 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.834+0000 7f28affff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:05:21.845 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.834+0000 7f28affff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:05:21.845 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.834+0000 7f28aeffd640 1 -- 172.21.15.42:0/2920671596 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1994888413 0 0) 0x7f2898003530 con 0x7f28b0073420 2024-09-16T04:05:21.845 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.834+0000 7f28aeffd640 1 -- 172.21.15.42:0/2920671596 >> v2:172.21.15.154:3300/0 conn(0x7f28b0130b20 msgr2=0x7f28b0130f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:05:21.845 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.834+0000 7f28aeffd640 1 --2- 172.21.15.42:0/2920671596 >> v2:172.21.15.154:3300/0 conn(0x7f28b0130b20 0x7f28b0130f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:21.845 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.834+0000 7f28aeffd640 1 -- 172.21.15.42:0/2920671596 >> v1:172.21.15.154:6789/0 conn(0x7f28b00a2250 legacy=0x7f28b0072d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:05:21.845 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.838+0000 7f28aeffd640 10 monclient: _finish_hunting 0 2024-09-16T04:05:21.846 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.838+0000 7f28aeffd640 1 monclient: found mon.noname-a-legacy 2024-09-16T04:05:21.846 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.838+0000 7f28aeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:05:21.846 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.838+0000 7f28aeffd640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.42:6789/0 2024-09-16T04:05:21.846 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.838+0000 7f28aeffd640 1 -- 172.21.15.42:0/2920671596 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28b00a1870 con 0x7f28b0073420 2024-09-16T04:05:21.846 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.838+0000 7f28aeffd640 10 monclient: _finish_auth 0 2024-09-16T04:05:21.846 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.838+0000 7f28aeffd640 10 monclient: _check_auth_tickets 2024-09-16T04:05:21.846 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.838+0000 7f28aeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:05:21.846 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.838+0000 7f28aeffd640 1 -- 172.21.15.42:0/2920671596 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2177413734 0 0) 0x7f28a8001a00 con 0x7f28b0073420 2024-09-16T04:05:21.846 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.838+0000 7f28b687b640 5 monclient: authenticate success, global_id 984309 2024-09-16T04:05:21.846 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.838+0000 7f28b687b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:05:21.846 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.838+0000 7f28aeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:05:21.846 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.838+0000 7f28aeffd640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T04:05:21.847 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.838+0000 7f28aeffd640 10 monclient: dump: 2024-09-16T04:05:21.847 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:05:21.847 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:05:21.847 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:21.847 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:21.847 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:05:21.847 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:05:21.847 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:05:21.847 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:05:21.847 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:05:21.847 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:05:21.847 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.838+0000 7f28aeffd640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] but i'm connected to v1:172.21.15.42:6789/0, reconnecting 2024-09-16T04:05:21.847 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.838+0000 7f28aeffd640 10 monclient: _reopen_session rank -1 2024-09-16T04:05:21.847 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.838+0000 7f28aeffd640 1 -- 172.21.15.42:0/2920671596 >> v1:172.21.15.42:6789/0 conn(0x7f28b0073420 legacy=0x7f28b013f240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:05:21.848 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.838+0000 7f28aeffd640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:05:21.848 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.838+0000 7f28aeffd640 1 --2- 172.21.15.42:0/2920671596 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f28980039c0 0x7f2898003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:05:21.848 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.838+0000 7f28aeffd640 10 monclient(hunting): picked mon.c con 0x7f28980039c0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:05:21.848 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.838+0000 7f28aeffd640 1 --2- 172.21.15.42:0/2920671596 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2898004810 0x7f2898004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:05:21.848 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.838+0000 7f28aeffd640 10 monclient(hunting): picked mon.a con 0x7f2898004810 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:05:21.848 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.838+0000 7f28aeffd640 1 --2- 172.21.15.42:0/2920671596 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f28980052f0 0x7f289800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:05:21.848 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.838+0000 7f28aeffd640 10 monclient(hunting): picked mon.b con 0x7f28980052f0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:05:21.848 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.838+0000 7f28aeffd640 10 monclient(hunting): start opening mon connection 2024-09-16T04:05:21.848 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.838+0000 7f28aeffd640 1 -- 172.21.15.42:0/2920671596 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f28b00a1070 con 0x7f2898004810 2024-09-16T04:05:21.848 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.838+0000 7f28aeffd640 10 monclient(hunting): start opening mon connection 2024-09-16T04:05:21.848 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.838+0000 7f28aeffd640 1 -- 172.21.15.42:0/2920671596 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f289800c1c0 con 0x7f28980052f0 2024-09-16T04:05:21.848 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.838+0000 7f28aeffd640 10 monclient(hunting): start opening mon connection 2024-09-16T04:05:21.849 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.838+0000 7f28aeffd640 1 -- 172.21.15.42:0/2920671596 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f289800c3c0 con 0x7f28980039c0 2024-09-16T04:05:21.849 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.838+0000 7f28aeffd640 10 monclient(hunting): _renew_subs 2024-09-16T04:05:21.849 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.838+0000 7f28affff640 1 -- 172.21.15.42:0/2920671596 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f28980039c0 msgr2=0x7f2898003de0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T04:05:21.849 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.838+0000 7f28aeffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:05:21.849 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.838+0000 7f28b687b640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-16T04:05:21.849 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.838+0000 7f28affff640 1 --2- 172.21.15.42:0/2920671596 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f28980039c0 0x7f2898003de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:05:21.849 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.838+0000 7f28af7fe640 1 --2- 172.21.15.42:0/2920671596 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2898004810 0x7f2898004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:05:21.849 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.838+0000 7f28af7fe640 10 monclient(hunting): get_auth_request con 0x7f2898004810 auth_method 0 2024-09-16T04:05:21.849 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.838+0000 7f28af7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:05:21.849 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.838+0000 7f28af7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:05:21.849 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.838+0000 7f28af7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:05:21.849 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.838+0000 7f28af7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:05:21.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.838+0000 7f28af7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:05:21.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.838+0000 7f28af7fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:05:21.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.838+0000 7f28af7fe640 10 monclient(hunting): handle_auth_done global_id 984309 payload 293 2024-09-16T04:05:21.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.838+0000 7f28af7fe640 1 -- 172.21.15.42:0/2920671596 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f28980039c0 msgr2=0x7f2898003de0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:05:21.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.838+0000 7f28af7fe640 1 --2- 172.21.15.42:0/2920671596 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f28980039c0 0x7f2898003de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:21.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.838+0000 7f28af7fe640 1 -- 172.21.15.42:0/2920671596 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f28980052f0 msgr2=0x7f289800b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:05:21.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.838+0000 7f28af7fe640 1 --2- 172.21.15.42:0/2920671596 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f28980052f0 0x7f289800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:21.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.838+0000 7f28af7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:05:21.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.838+0000 7f28af7fe640 1 monclient: found mon.a 2024-09-16T04:05:21.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.838+0000 7f28af7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:05:21.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.838+0000 7f28b4e18640 1 -- 172.21.15.42:0/2920671596 reap_dead start 2024-09-16T04:05:21.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.838+0000 7f28af7fe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:05:21.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.838+0000 7f28af7fe640 1 -- 172.21.15.42:0/2920671596 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2898003530 con 0x7f2898004810 2024-09-16T04:05:21.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.838+0000 7f28af7fe640 10 monclient: _finish_auth 0 2024-09-16T04:05:21.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.838+0000 7f28af7fe640 10 monclient: _check_auth_tickets 2024-09-16T04:05:21.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.838+0000 7f28af7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:05:21.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.838+0000 7f28af7fe640 1 --2- 172.21.15.42:0/2920671596 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2898004810 0x7f2898004c90 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f28b0137fa0 tx=0x7f28a401c320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a35098fcf39864d8 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:05:21.857 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.838+0000 7f28aeffd640 1 -- 172.21.15.42:0/2920671596 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28a40013a0 con 0x7f2898004810 2024-09-16T04:05:21.857 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.838+0000 7f28aeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:05:21.857 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.838+0000 7f28aeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:05:21.857 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.838+0000 7f28aeffd640 10 monclient: dump: 2024-09-16T04:05:21.857 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:05:21.857 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:05:21.857 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:21.857 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:21.857 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:05:21.857 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:05:21.858 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:05:21.858 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:05:21.858 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:05:21.858 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:05:21.858 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.838+0000 7f28aeffd640 1 -- 172.21.15.42:0/2920671596 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f28a40052f0 con 0x7f2898004810 2024-09-16T04:05:21.858 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.838+0000 7f28b687b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:05:21.858 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.838+0000 7f28aeffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:05:21.858 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.838+0000 7f28b687b640 10 monclient: get_monmap_and_config success 2024-09-16T04:05:21.858 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.838+0000 7f28b687b640 4 set_mon_vals no callback set 2024-09-16T04:05:21.858 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.838+0000 7f28aeffd640 1 -- 172.21.15.42:0/2920671596 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28a4003840 con 0x7f2898004810 2024-09-16T04:05:21.858 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.838+0000 7f28aeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:05:21.858 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.838+0000 7f28aeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:05:21.859 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.838+0000 7f28aeffd640 10 monclient: dump: 2024-09-16T04:05:21.859 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:05:21.859 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:05:21.859 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:21.859 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:21.859 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:05:21.859 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:05:21.859 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:05:21.859 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:05:21.859 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:05:21.859 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:05:21.859 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.838+0000 7f28b687b640 10 monclient: shutdown 2024-09-16T04:05:21.860 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.838+0000 7f28b687b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:05:21.860 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.838+0000 7f28b687b640 1 -- 172.21.15.42:0/2920671596 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2898004810 msgr2=0x7f2898004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:05:21.860 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.838+0000 7f28b687b640 1 --2- 172.21.15.42:0/2920671596 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2898004810 0x7f2898004c90 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f28b0137fa0 tx=0x7f28a401c320 comp rx=0 tx=0).stop 2024-09-16T04:05:21.860 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.838+0000 7f28b687b640 1 -- 172.21.15.42:0/2920671596 shutdown_connections 2024-09-16T04:05:21.860 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.838+0000 7f28b687b640 1 --2- 172.21.15.42:0/2920671596 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2898004810 0x7f2898004c90 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:21.860 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.838+0000 7f28b687b640 1 -- 172.21.15.42:0/2920671596 >> 172.21.15.42:0/2920671596 conn(0x7f28b00a6210 msgr2=0x7f28b00a6620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:05:21.860 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.838+0000 7f28b687b640 1 -- 172.21.15.42:0/2920671596 shutdown_connections 2024-09-16T04:05:21.860 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.838+0000 7f28b687b640 1 -- 172.21.15.42:0/2920671596 wait complete. 2024-09-16T04:05:21.860 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.842+0000 7f28b687b640 10 monclient: build_initial_monmap 2024-09-16T04:05:21.860 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.842+0000 7f28b687b640 10 monclient: monmap: 2024-09-16T04:05:21.861 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:05:21.861 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:05:21.861 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:05:21.861 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:05:21.861 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:05:21.861 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:05:21.861 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:05:21.861 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:05:21.861 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:05:21.861 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:05:21.861 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.842+0000 7f28b687b640 1 librados: starting msgr at 2024-09-16T04:05:21.861 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.842+0000 7f28b687b640 1 librados: starting objecter 2024-09-16T04:05:21.862 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.842+0000 7f28b687b640 1 Processor -- start 2024-09-16T04:05:21.862 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.842+0000 7f28b687b640 1 -- start start 2024-09-16T04:05:21.862 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.842+0000 7f28b687b640 1 librados: setting wanted keys 2024-09-16T04:05:21.862 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.842+0000 7f28b687b640 1 librados: calling monclient init 2024-09-16T04:05:21.862 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.842+0000 7f28b687b640 10 monclient: init 2024-09-16T04:05:21.862 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.842+0000 7f28b687b640 10 monclient: _reopen_session rank -1 2024-09-16T04:05:21.862 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.842+0000 7f28b687b640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:05:21.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.842+0000 7f28b687b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2898004810 0x7f28b01cf7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:05:21.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.842+0000 7f28b687b640 10 monclient(hunting): picked mon.noname-b con 0x7f2898004810 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:05:21.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.842+0000 7f28b687b640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f28b012fb60 0x7f28b012ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:05:21.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.842+0000 7f28b687b640 10 monclient(hunting): picked mon.noname-c con 0x7f28b012fb60 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:05:21.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.842+0000 7f28b687b640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f28b011e5e0 0x7f28b011ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:05:21.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.842+0000 7f28b687b640 10 monclient(hunting): picked mon.noname-a con 0x7f28b011e5e0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:05:21.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.842+0000 7f28b687b640 10 monclient(hunting): start opening mon connection 2024-09-16T04:05:21.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.842+0000 7f28b687b640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f28b0075ac0 con 0x7f28b011e5e0 2024-09-16T04:05:21.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.842+0000 7f28b687b640 10 monclient(hunting): start opening mon connection 2024-09-16T04:05:21.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.842+0000 7f28b687b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f28b0075940 con 0x7f2898004810 2024-09-16T04:05:21.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.842+0000 7f28b687b640 10 monclient(hunting): start opening mon connection 2024-09-16T04:05:21.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.842+0000 7f28b687b640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f28b013a210 con 0x7f28b012fb60 2024-09-16T04:05:21.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.842+0000 7f28b687b640 10 monclient(hunting): _renew_subs 2024-09-16T04:05:21.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.842+0000 7f28b687b640 10 monclient(hunting): authenticate will time out at 2368.813028s 2024-09-16T04:05:21.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.842+0000 7f28af7fe640 1 -- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f28b012fb60 msgr2=0x7f28b012ffa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T04:05:21.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.842+0000 7f28af7fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f28b012fb60 0x7f28b012ffa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:05:21.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.842+0000 7f28b4e18640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f28b011e5e0 0x7f28b011ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:05:21.865 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.842+0000 7f28b4e18640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f28b011e5e0 0x7f28b011ea60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:47648/0 (socket says 172.21.15.42:47648) 2024-09-16T04:05:21.865 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.842+0000 7f28b4e18640 1 -- 172.21.15.42:0/21052588 learned_addr learned my addr 172.21.15.42:0/21052588 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:05:21.865 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.842+0000 7f28b4e18640 10 monclient(hunting): get_auth_request con 0x7f28b011e5e0 auth_method 0 2024-09-16T04:05:21.865 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.842+0000 7f28b4e18640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:05:21.865 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.842+0000 7f28b4e18640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:05:21.865 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.842+0000 7f28b4e18640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:05:21.865 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.842+0000 7f28b4e18640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:05:21.865 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.842+0000 7f28b4e18640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:05:21.865 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.842+0000 7f28b4e18640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:05:21.865 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.842+0000 7f28b4e18640 10 monclient(hunting): handle_auth_done global_id 984312 payload 824 2024-09-16T04:05:21.865 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.842+0000 7f28b4e18640 1 -- 172.21.15.42:0/21052588 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f28b012fb60 msgr2=0x7f28b012ffa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:05:21.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.842+0000 7f28b4e18640 1 --2- 172.21.15.42:0/21052588 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f28b012fb60 0x7f28b012ffa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:21.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.842+0000 7f28b4e18640 1 -- 172.21.15.42:0/21052588 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2898004810 msgr2=0x7f28b01cf7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:05:21.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.842+0000 7f28b4e18640 1 --2- 172.21.15.42:0/21052588 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2898004810 0x7f28b01cf7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:21.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.842+0000 7f28b4e18640 10 monclient: _finish_hunting 0 2024-09-16T04:05:21.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.842+0000 7f28b4e18640 1 monclient: found mon.noname-a 2024-09-16T04:05:21.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.842+0000 7f28b4e18640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:05:21.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.842+0000 7f28b4e18640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:05:21.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.842+0000 7f28b4e18640 1 -- 172.21.15.42:0/21052588 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28b0121fb0 con 0x7f28b011e5e0 2024-09-16T04:05:21.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.842+0000 7f28b4e18640 1 --2- 172.21.15.42:0/21052588 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f28b011e5e0 0x7f28b011ea60 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f28a8001570 tx=0x7f28a800ccd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:05:21.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.842+0000 7f28acff9640 10 client.?.objecter ms_handle_connect 0x7f28b011e5e0 2024-09-16T04:05:21.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.842+0000 7f28acff9640 10 client.?.objecter resend_mon_ops 2024-09-16T04:05:21.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.842+0000 7f28acff9640 1 -- 172.21.15.42:0/21052588 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28a800a610 con 0x7f28b011e5e0 2024-09-16T04:05:21.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.842+0000 7f28acff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:05:21.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.842+0000 7f28acff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:05:21.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.842+0000 7f28acff9640 10 monclient: dump: 2024-09-16T04:05:21.867 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:05:21.867 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:05:21.867 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:21.867 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:21.867 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:05:21.867 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:05:21.868 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:05:21.868 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:05:21.868 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:05:21.868 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:05:21.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.842+0000 7f28acff9640 10 monclient: _finish_auth 0 2024-09-16T04:05:21.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.842+0000 7f28acff9640 10 monclient: _check_auth_tickets 2024-09-16T04:05:21.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.842+0000 7f28acff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:05:21.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.842+0000 7f28acff9640 1 -- 172.21.15.42:0/21052588 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f28a800a7b0 con 0x7f28b011e5e0 2024-09-16T04:05:21.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.842+0000 7f28acff9640 10 monclient: handle_config config(0 keys) 2024-09-16T04:05:21.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.842+0000 7f28acff9640 1 -- 172.21.15.42:0/21052588 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28a800ab60 con 0x7f28b011e5e0 2024-09-16T04:05:21.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.842+0000 7f28acff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:05:21.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.842+0000 7f28acff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:05:21.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.842+0000 7f28acff9640 10 monclient: dump: 2024-09-16T04:05:21.869 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:05:21.869 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:05:21.869 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:21.869 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:21.869 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:05:21.869 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:05:21.869 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:05:21.869 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:05:21.870 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:05:21.870 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:05:21.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.842+0000 7f28b687b640 5 monclient: authenticate success, global_id 984312 2024-09-16T04:05:21.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.842+0000 7f28b687b640 10 monclient: _renew_subs 2024-09-16T04:05:21.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.842+0000 7f28b687b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:05:21.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.842+0000 7f28b687b640 1 -- 172.21.15.42:0/21052588 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28b01304e0 con 0x7f28b011e5e0 2024-09-16T04:05:21.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.842+0000 7f28ae7fc640 4 set_mon_vals no callback set 2024-09-16T04:05:21.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.842+0000 7f28b687b640 10 client.984312.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:05:21.871 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.842+0000 7f28b687b640 10 monclient: _renew_subs 2024-09-16T04:05:21.871 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.842+0000 7f28b687b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:05:21.871 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.842+0000 7f28b687b640 1 -- 172.21.15.42:0/21052588 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28b00a3e40 con 0x7f28b011e5e0 2024-09-16T04:05:21.871 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.842+0000 7f28b687b640 1 librados: init done 2024-09-16T04:05:21.871 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.842+0000 7f28b687b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:05:21.871 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.842+0000 7f28b687b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:05:21.871 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.842+0000 7f28b687b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:05:21.871 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.842+0000 7f28b687b640 1 -- 172.21.15.42:0/21052588 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f28b00a5b00 con 0x7f28b011e5e0 2024-09-16T04:05:21.871 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.842+0000 7f28acff9640 1 -- 172.21.15.42:0/21052588 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f28a800ad00 con 0x7f28b011e5e0 2024-09-16T04:05:21.871 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.842+0000 7f28acff9640 1 --2- 172.21.15.42:0/21052588 >> v2:172.21.15.154:6812/254845342 conn(0x7f289c0377f0 0x7f289c039cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:05:21.871 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.842+0000 7f28acff9640 1 -- 172.21.15.42:0/21052588 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(706..706 src has 451..706) ==== 5040+0+0 (secure 0 0 0) 0x7f28a8047df0 con 0x7f28b011e5e0 2024-09-16T04:05:21.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.842+0000 7f28acff9640 10 client.984312.objecter ms_dispatch 0x7f28b0000ba0 osd_map(706..706 src has 451..706) 2024-09-16T04:05:21.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.842+0000 7f28acff9640 3 client.984312.objecter handle_osd_map got epochs [706,706] > 0 2024-09-16T04:05:21.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.842+0000 7f28acff9640 3 client.984312.objecter handle_osd_map decoding full epoch 706 2024-09-16T04:05:21.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.842+0000 7f28acff9640 20 client.984312.objecter dump_active .. 0 homeless 2024-09-16T04:05:21.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.842+0000 7f28affff640 1 --2- 172.21.15.42:0/21052588 >> v2:172.21.15.154:6812/254845342 conn(0x7f289c0377f0 0x7f289c039cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:05:21.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.842+0000 7f28affff640 10 monclient: get_auth_request con 0x7f289c0377f0 auth_method 0 2024-09-16T04:05:21.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.846+0000 7f28acff9640 10 client.984312.objecter ms_handle_connect 0x7f289c0377f0 2024-09-16T04:05:21.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.846+0000 7f28affff640 1 --2- 172.21.15.42:0/21052588 >> v2:172.21.15.154:6812/254845342 conn(0x7f289c0377f0 0x7f289c039cb0 secure :-1 s=READY pgs=3849 cs=0 l=1 rev1=1 crypto rx=0x7f28a0000f00 tx=0x7f28a0006a30 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:05:21.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.846+0000 7f28acff9640 1 -- 172.21.15.42:0/21052588 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f28a801a070 con 0x7f28b011e5e0 2024-09-16T04:05:21.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.846+0000 7f28acff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:05:21.873 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:21.846+0000 7f28acff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:05:22.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:22.074+0000 7f28b687b640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:05:22.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:22.074+0000 7f28b687b640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:05:22.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:22.074+0000 7f28b687b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:05:22.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:22.074+0000 7f28b687b640 1 -- 172.21.15.42:0/21052588 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f28b013a4b0 con 0x7f28b011e5e0 2024-09-16T04:05:22.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:22.074+0000 7f28acff9640 1 -- 172.21.15.42:0/21052588 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1562 (secure 0 0 0) 0x7f28a801a250 con 0x7f28b011e5e0 2024-09-16T04:05:22.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:22.074+0000 7f28acff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:05:22.080 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:22.074+0000 7f28acff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:05:22.080 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":216,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:05:22.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:22.074+0000 7f28b687b640 10 librados: watch_flush enter 2024-09-16T04:05:22.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:22.074+0000 7f28b687b640 10 librados: watch_flush exit 2024-09-16T04:05:22.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:22.074+0000 7f28b687b640 20 client.984312.objecter shutdown clearing up homeless session... 2024-09-16T04:05:22.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:22.074+0000 7f28b687b640 10 client.984312.objecter successfully canceled tick 2024-09-16T04:05:22.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:22.074+0000 7f28b687b640 1 -- 172.21.15.42:0/21052588 >> v2:172.21.15.154:6812/254845342 conn(0x7f289c0377f0 msgr2=0x7f289c039cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:05:22.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:22.074+0000 7f28b687b640 1 --2- 172.21.15.42:0/21052588 >> v2:172.21.15.154:6812/254845342 conn(0x7f289c0377f0 0x7f289c039cb0 secure :-1 s=READY pgs=3849 cs=0 l=1 rev1=1 crypto rx=0x7f28a0000f00 tx=0x7f28a0006a30 comp rx=0 tx=0).stop 2024-09-16T04:05:22.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:22.074+0000 7f28b687b640 10 monclient: shutdown 2024-09-16T04:05:22.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:22.074+0000 7f28b687b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:05:22.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:22.074+0000 7f28b687b640 1 -- 172.21.15.42:0/21052588 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f28b011e5e0 msgr2=0x7f28b011ea60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:05:22.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:22.074+0000 7f28b687b640 1 --2- 172.21.15.42:0/21052588 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f28b011e5e0 0x7f28b011ea60 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f28a8001570 tx=0x7f28a800ccd0 comp rx=0 tx=0).stop 2024-09-16T04:05:22.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:22.074+0000 7f28b687b640 1 -- 172.21.15.42:0/21052588 shutdown_connections 2024-09-16T04:05:22.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:22.074+0000 7f28b687b640 1 --2- 172.21.15.42:0/21052588 >> v2:172.21.15.154:6812/254845342 conn(0x7f289c0377f0 0x7f289c039cb0 unknown :-1 s=CLOSED pgs=3849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:22.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:22.078+0000 7f28b687b640 1 --2- 172.21.15.42:0/21052588 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f28b011e5e0 0x7f28b011ea60 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:22.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:22.078+0000 7f28b687b640 1 --2- 172.21.15.42:0/21052588 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f28b012fb60 0x7f28b012ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:22.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:22.078+0000 7f28b687b640 1 --2- 172.21.15.42:0/21052588 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2898004810 0x7f28b01cf7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:22.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:22.078+0000 7f28b687b640 1 -- 172.21.15.42:0/21052588 >> 172.21.15.42:0/21052588 conn(0x7f28b00a6210 msgr2=0x7f28b009f450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:05:22.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:22.078+0000 7f28b687b640 1 -- 172.21.15.42:0/21052588 shutdown_connections 2024-09-16T04:05:22.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:22.078+0000 7f28b687b640 1 -- 172.21.15.42:0/21052588 wait complete. 2024-09-16T04:05:22.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:22.078+0000 7f28b687b640 1 librados: shutdown 2024-09-16T04:05:25.100 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:05:25.235 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.230+0000 7fb5c0d58640 10 monclient: get_monmap_and_config 2024-09-16T04:05:25.235 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.230+0000 7fb5c0d58640 10 monclient: build_initial_monmap 2024-09-16T04:05:25.235 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.230+0000 7fb5c0d58640 10 monclient: monmap: 2024-09-16T04:05:25.235 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:05:25.235 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:05:25.235 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:05:25.233496+0000 2024-09-16T04:05:25.235 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:05:25.233496+0000 2024-09-16T04:05:25.235 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:05:25.236 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:05:25.236 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:05:25.236 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:05:25.236 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:05:25.236 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:05:25.236 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:05:25.236 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:05:25.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.230+0000 7fb5c0d58640 1 Processor -- start 2024-09-16T04:05:25.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.230+0000 7fb5c0d58640 1 -- start start 2024-09-16T04:05:25.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.230+0000 7fb5c0d58640 10 monclient: init 2024-09-16T04:05:25.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.230+0000 7fb5c0d58640 10 monclient: _reopen_session rank -1 2024-09-16T04:05:25.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.230+0000 7fb5c0d58640 10 monclient: _add_conns ranks=[3,0,4,1,2] 2024-09-16T04:05:25.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.230+0000 7fb5c0d58640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb5bc0a1330 0x7fb5bc0a1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:05:25.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.230+0000 7fb5c0d58640 10 monclient(hunting): picked mon.noname-a con 0x7fb5bc0a1330 addr v2:172.21.15.42:3300/0 2024-09-16T04:05:25.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.230+0000 7fb5c0d58640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb5bc09c920 addr v1:172.21.15.42:6789/0 2024-09-16T04:05:25.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.230+0000 7fb5c0d58640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fb5bc072d40 0x7fb5bc0a1c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:05:25.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.230+0000 7fb5c0d58640 10 monclient(hunting): picked mon.noname-c con 0x7fb5bc072d40 addr v2:172.21.15.154:3300/0 2024-09-16T04:05:25.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.230+0000 7fb5c0d58640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb5bc072350 con 0x7fb5bc09c920 2024-09-16T04:05:25.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.230+0000 7fb5c0d58640 10 monclient(hunting): start opening mon connection 2024-09-16T04:05:25.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.230+0000 7fb5c0d58640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7fb5bc072b50 con 0x7fb5bc0a1330 2024-09-16T04:05:25.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.230+0000 7fb5c0d58640 10 monclient(hunting): start opening mon connection 2024-09-16T04:05:25.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.230+0000 7fb5c0d58640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fb5bc0a2190 con 0x7fb5bc072d40 2024-09-16T04:05:25.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.230+0000 7fb5c0d58640 10 monclient(hunting): _renew_subs 2024-09-16T04:05:25.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.230+0000 7fb5c0d58640 10 monclient(hunting): authenticate will time out at 2372.202942s 2024-09-16T04:05:25.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.230+0000 7fb5b9d9b640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fb5bc09c920 0x7fb5bc137120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:33478/0 (socket says 172.21.15.42:33478) 2024-09-16T04:05:25.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.230+0000 7fb5b9d9b640 1 -- 172.21.15.42:0/3365102198 learned_addr learned my addr 172.21.15.42:0/3365102198 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T04:05:25.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.230+0000 7fb5ba59c640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb5bc0a1330 0x7fb5bc0a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:05:25.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.230+0000 7fb5ba59c640 10 monclient(hunting): get_auth_request con 0x7fb5bc0a1330 auth_method 0 2024-09-16T04:05:25.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.230+0000 7fb5ba59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:05:25.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.230+0000 7fb5ba59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:05:25.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.230+0000 7fb5ba59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:05:25.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.230+0000 7fb5ba59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:05:25.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.230+0000 7fb5ba59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:05:25.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.230+0000 7fb5ba59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:05:25.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.230+0000 7fb5bad9d640 1 --2- 172.21.15.42:0/3365102198 >> v2:172.21.15.154:3300/0 conn(0x7fb5bc072d40 0x7fb5bc0a1c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:05:25.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.230+0000 7fb5bad9d640 10 monclient(hunting): get_auth_request con 0x7fb5bc072d40 auth_method 0 2024-09-16T04:05:25.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.230+0000 7fb5bad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:05:25.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.230+0000 7fb5bad9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:05:25.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.230+0000 7fb5bad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:05:25.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.230+0000 7fb5ba59c640 10 monclient(hunting): handle_auth_done global_id 984315 payload 274 2024-09-16T04:05:25.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.230+0000 7fb5ba59c640 1 -- 172.21.15.42:0/3365102198 >> v2:172.21.15.154:3300/0 conn(0x7fb5bc072d40 msgr2=0x7fb5bc0a1c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:05:25.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.230+0000 7fb5ba59c640 1 --2- 172.21.15.42:0/3365102198 >> v2:172.21.15.154:3300/0 conn(0x7fb5bc072d40 0x7fb5bc0a1c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:25.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.230+0000 7fb5ba59c640 1 -- 172.21.15.42:0/3365102198 >> v1:172.21.15.42:6789/0 conn(0x7fb5bc09c920 legacy=0x7fb5bc137120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:05:25.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.230+0000 7fb5ba59c640 10 monclient: _finish_hunting 0 2024-09-16T04:05:25.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.230+0000 7fb5ba59c640 1 monclient: found mon.noname-a 2024-09-16T04:05:25.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.230+0000 7fb5ba59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:05:25.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.230+0000 7fb5ba59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:05:25.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.230+0000 7fb5ba59c640 1 -- 172.21.15.42:0/3365102198 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5bc09bc60 con 0x7fb5bc0a1330 2024-09-16T04:05:25.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.234+0000 7fb5ba59c640 1 --2- 172.21.15.42:0/3365102198 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb5bc0a1330 0x7fb5bc0a1710 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fb5bc068190 tx=0x7fb5ac01c310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7b69411a88f1ee7 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:05:25.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.234+0000 7fb5b959a640 1 -- 172.21.15.42:0/3365102198 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5ac0b0070 con 0x7fb5bc0a1330 2024-09-16T04:05:25.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.234+0000 7fb5b959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:05:25.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.234+0000 7fb5b959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:05:25.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.234+0000 7fb5b959a640 10 monclient: dump: 2024-09-16T04:05:25.247 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:05:25.247 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:05:25.247 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:25.248 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:25.248 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:05:25.248 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:05:25.248 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:05:25.248 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:05:25.248 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:05:25.248 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:05:25.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.234+0000 7fb5b959a640 10 monclient: _finish_auth 0 2024-09-16T04:05:25.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.234+0000 7fb5b959a640 10 monclient: _check_auth_tickets 2024-09-16T04:05:25.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.234+0000 7fb5b959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:05:25.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.234+0000 7fb5b959a640 1 -- 172.21.15.42:0/3365102198 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb5ac0ab600 con 0x7fb5bc0a1330 2024-09-16T04:05:25.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.234+0000 7fb5c0d58640 5 monclient: authenticate success, global_id 984315 2024-09-16T04:05:25.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.234+0000 7fb5c0d58640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:05:25.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.234+0000 7fb5b959a640 10 monclient: handle_config config(0 keys) 2024-09-16T04:05:25.249 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.234+0000 7fb5c0d58640 10 monclient: get_monmap_and_config success 2024-09-16T04:05:25.249 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.234+0000 7fb5b959a640 1 -- 172.21.15.42:0/3365102198 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5ac0ab9d0 con 0x7fb5bc0a1330 2024-09-16T04:05:25.249 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.234+0000 7fb5b959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:05:25.249 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.234+0000 7fb5c0d58640 4 set_mon_vals no callback set 2024-09-16T04:05:25.249 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.234+0000 7fb5b959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:05:25.249 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.234+0000 7fb5b959a640 10 monclient: dump: 2024-09-16T04:05:25.249 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:05:25.249 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:05:25.249 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:25.249 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:25.249 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:05:25.249 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:05:25.249 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:05:25.250 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:05:25.250 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:05:25.250 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:05:25.250 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.234+0000 7fb5c0d58640 10 monclient: shutdown 2024-09-16T04:05:25.250 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.234+0000 7fb5c0d58640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:05:25.250 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.234+0000 7fb5c0d58640 1 -- 172.21.15.42:0/3365102198 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb5bc0a1330 msgr2=0x7fb5bc0a1710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:05:25.250 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.234+0000 7fb5c0d58640 1 --2- 172.21.15.42:0/3365102198 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb5bc0a1330 0x7fb5bc0a1710 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fb5bc068190 tx=0x7fb5ac01c310 comp rx=0 tx=0).stop 2024-09-16T04:05:25.250 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.234+0000 7fb5c0d58640 1 -- 172.21.15.42:0/3365102198 shutdown_connections 2024-09-16T04:05:25.250 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.234+0000 7fb5c0d58640 1 --2- 172.21.15.42:0/3365102198 >> v2:172.21.15.154:3300/0 conn(0x7fb5bc072d40 0x7fb5bc0a1c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:25.250 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.234+0000 7fb5c0d58640 1 --2- 172.21.15.42:0/3365102198 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb5bc0a1330 0x7fb5bc0a1710 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:25.250 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.234+0000 7fb5c0d58640 1 -- 172.21.15.42:0/3365102198 >> 172.21.15.42:0/3365102198 conn(0x7fb5bc0a2420 msgr2=0x7fb5bc0a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:05:25.250 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.234+0000 7fb5c0d58640 1 -- 172.21.15.42:0/3365102198 shutdown_connections 2024-09-16T04:05:25.250 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.234+0000 7fb5c0d58640 1 -- 172.21.15.42:0/3365102198 wait complete. 2024-09-16T04:05:25.250 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.234+0000 7fb5c0d58640 10 monclient: build_initial_monmap 2024-09-16T04:05:25.251 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.234+0000 7fb5c0d58640 10 monclient: monmap: 2024-09-16T04:05:25.251 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:05:25.251 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:05:25.251 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:05:25.251 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:05:25.251 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:05:25.251 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:05:25.251 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:05:25.251 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:05:25.251 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:05:25.251 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:05:25.251 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.234+0000 7fb5c0d58640 1 librados: starting msgr at 2024-09-16T04:05:25.251 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.234+0000 7fb5c0d58640 1 librados: starting objecter 2024-09-16T04:05:25.251 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.234+0000 7fb5c0d58640 1 Processor -- start 2024-09-16T04:05:25.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.234+0000 7fb5c0d58640 1 -- start start 2024-09-16T04:05:25.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.234+0000 7fb5c0d58640 1 librados: setting wanted keys 2024-09-16T04:05:25.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.234+0000 7fb5c0d58640 1 librados: calling monclient init 2024-09-16T04:05:25.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.234+0000 7fb5c0d58640 10 monclient: init 2024-09-16T04:05:25.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.234+0000 7fb5c0d58640 10 monclient: _reopen_session rank -1 2024-09-16T04:05:25.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.234+0000 7fb5c0d58640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:05:25.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.234+0000 7fb5c0d58640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb5bc072d40 0x7fb5bc1cee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:05:25.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.238+0000 7fb5c0d58640 10 monclient(hunting): picked mon.noname-b con 0x7fb5bc072d40 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:05:25.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.238+0000 7fb5c0d58640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb5bc09c920 0x7fb5bc136350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:05:25.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.238+0000 7fb5c0d58640 10 monclient(hunting): picked mon.noname-c con 0x7fb5bc09c920 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:05:25.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.238+0000 7fb5c0d58640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb5bc0a1330 0x7fb5bc11ce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:05:25.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.238+0000 7fb5c0d58640 10 monclient(hunting): picked mon.noname-a con 0x7fb5bc0a1330 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:05:25.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.238+0000 7fb5c0d58640 10 monclient(hunting): start opening mon connection 2024-09-16T04:05:25.253 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.238+0000 7fb5c0d58640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fb5bc141a00 con 0x7fb5bc0a1330 2024-09-16T04:05:25.253 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.238+0000 7fb5c0d58640 10 monclient(hunting): start opening mon connection 2024-09-16T04:05:25.253 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.238+0000 7fb5c0d58640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fb5bc141880 con 0x7fb5bc072d40 2024-09-16T04:05:25.253 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.238+0000 7fb5c0d58640 10 monclient(hunting): start opening mon connection 2024-09-16T04:05:25.253 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.238+0000 7fb5c0d58640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fb5bc141b80 con 0x7fb5bc09c920 2024-09-16T04:05:25.253 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.238+0000 7fb5c0d58640 10 monclient(hunting): _renew_subs 2024-09-16T04:05:25.253 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.238+0000 7fb5c0d58640 10 monclient(hunting): authenticate will time out at 2372.208066s 2024-09-16T04:05:25.253 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.238+0000 7fb5b9d9b640 1 -- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb5bc09c920 msgr2=0x7fb5bc136350 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T04:05:25.253 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.238+0000 7fb5b9d9b640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb5bc09c920 0x7fb5bc136350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:05:25.253 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.238+0000 7fb5bad9d640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb5bc0a1330 0x7fb5bc11ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:05:25.253 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.238+0000 7fb5bad9d640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb5bc0a1330 0x7fb5bc11ce20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:56596/0 (socket says 172.21.15.42:56596) 2024-09-16T04:05:25.253 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.238+0000 7fb5bad9d640 1 -- 172.21.15.42:0/11537508 learned_addr learned my addr 172.21.15.42:0/11537508 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:05:25.254 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.238+0000 7fb5bad9d640 10 monclient(hunting): get_auth_request con 0x7fb5bc0a1330 auth_method 0 2024-09-16T04:05:25.254 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.238+0000 7fb5bad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:05:25.254 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.238+0000 7fb5bad9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:05:25.254 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.238+0000 7fb5bad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:05:25.254 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.238+0000 7fb5bad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:05:25.254 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.238+0000 7fb5bad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:05:25.254 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.238+0000 7fb5bad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:05:25.254 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.238+0000 7fb5ba59c640 1 --2- 172.21.15.42:0/11537508 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb5bc072d40 0x7fb5bc1cee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:05:25.254 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.238+0000 7fb5ba59c640 10 monclient(hunting): get_auth_request con 0x7fb5bc072d40 auth_method 0 2024-09-16T04:05:25.254 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.238+0000 7fb5ba59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:05:25.254 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.238+0000 7fb5ba59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:05:25.254 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.238+0000 7fb5ba59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:05:25.254 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.238+0000 7fb5bad9d640 10 monclient(hunting): handle_auth_done global_id 984318 payload 824 2024-09-16T04:05:25.254 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.238+0000 7fb5bad9d640 1 -- 172.21.15.42:0/11537508 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb5bc09c920 msgr2=0x7fb5bc136350 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:05:25.255 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.238+0000 7fb5bad9d640 1 --2- 172.21.15.42:0/11537508 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb5bc09c920 0x7fb5bc136350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:25.255 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.238+0000 7fb5bad9d640 1 -- 172.21.15.42:0/11537508 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb5bc072d40 msgr2=0x7fb5bc1cee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:05:25.255 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.238+0000 7fb5bad9d640 1 --2- 172.21.15.42:0/11537508 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb5bc072d40 0x7fb5bc1cee40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:25.255 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.238+0000 7fb5bad9d640 10 monclient: _finish_hunting 0 2024-09-16T04:05:25.255 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.238+0000 7fb5bad9d640 1 monclient: found mon.noname-a 2024-09-16T04:05:25.255 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.238+0000 7fb5bad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:05:25.255 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.238+0000 7fb5bad9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:05:25.255 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.238+0000 7fb5bad9d640 1 -- 172.21.15.42:0/11537508 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5bc0a5470 con 0x7fb5bc0a1330 2024-09-16T04:05:25.255 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.238+0000 7fb5bad9d640 1 --2- 172.21.15.42:0/11537508 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb5bc0a1330 0x7fb5bc11ce20 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fb5b400d800 tx=0x7fb5b400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:05:25.255 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.238+0000 7fb5b2ffd640 10 client.?.objecter ms_handle_connect 0x7fb5bc0a1330 2024-09-16T04:05:25.255 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.238+0000 7fb5b2ffd640 10 client.?.objecter resend_mon_ops 2024-09-16T04:05:25.255 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.238+0000 7fb5b2ffd640 1 -- 172.21.15.42:0/11537508 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5b400b6c0 con 0x7fb5bc0a1330 2024-09-16T04:05:25.255 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.238+0000 7fb5b2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:05:25.255 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.238+0000 7fb5b2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:05:25.256 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.238+0000 7fb5b2ffd640 10 monclient: dump: 2024-09-16T04:05:25.256 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:05:25.256 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:05:25.256 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:25.256 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:25.256 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:05:25.256 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:05:25.256 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:05:25.256 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:05:25.256 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:05:25.256 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:05:25.256 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.238+0000 7fb5b2ffd640 10 monclient: _finish_auth 0 2024-09-16T04:05:25.256 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.238+0000 7fb5b2ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:05:25.257 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.238+0000 7fb5b2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:05:25.257 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.238+0000 7fb5c0d58640 5 monclient: authenticate success, global_id 984318 2024-09-16T04:05:25.257 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.238+0000 7fb5c0d58640 10 monclient: _renew_subs 2024-09-16T04:05:25.257 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.238+0000 7fb5c0d58640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:05:25.257 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.238+0000 7fb5c0d58640 1 -- 172.21.15.42:0/11537508 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5bc073a30 con 0x7fb5bc0a1330 2024-09-16T04:05:25.257 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.238+0000 7fb5b2ffd640 1 -- 172.21.15.42:0/11537508 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb5b4019070 con 0x7fb5bc0a1330 2024-09-16T04:05:25.257 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.238+0000 7fb5b2ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:05:25.257 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.238+0000 7fb5b2ffd640 1 -- 172.21.15.42:0/11537508 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5b400baa0 con 0x7fb5bc0a1330 2024-09-16T04:05:25.257 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.238+0000 7fb5b2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:05:25.257 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.238+0000 7fb5b2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:05:25.257 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.238+0000 7fb5c0d58640 10 client.984318.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:05:25.257 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.238+0000 7fb5b2ffd640 10 monclient: dump: 2024-09-16T04:05:25.257 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:05:25.258 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:05:25.258 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:25.258 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:25.258 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:05:25.258 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:05:25.258 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:05:25.258 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:05:25.258 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:05:25.258 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:05:25.258 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.238+0000 7fb5c0d58640 10 monclient: _renew_subs 2024-09-16T04:05:25.258 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.238+0000 7fb5c0d58640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:05:25.258 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.238+0000 7fb5c0d58640 1 -- 172.21.15.42:0/11537508 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5bc120720 con 0x7fb5bc0a1330 2024-09-16T04:05:25.258 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.238+0000 7fb5b3fff640 4 set_mon_vals no callback set 2024-09-16T04:05:25.258 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.238+0000 7fb5c0d58640 1 librados: init done 2024-09-16T04:05:25.258 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.238+0000 7fb5b2ffd640 1 -- 172.21.15.42:0/11537508 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fb5b400bc40 con 0x7fb5bc0a1330 2024-09-16T04:05:25.259 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.238+0000 7fb5c0d58640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:05:25.259 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.238+0000 7fb5c0d58640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:05:25.259 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.238+0000 7fb5c0d58640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:05:25.259 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.238+0000 7fb5c0d58640 1 -- 172.21.15.42:0/11537508 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb5bc137120 con 0x7fb5bc0a1330 2024-09-16T04:05:25.259 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.238+0000 7fb5b2ffd640 1 --2- 172.21.15.42:0/11537508 >> v2:172.21.15.154:6812/254845342 conn(0x7fb59c0377c0 0x7fb59c039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:05:25.259 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.238+0000 7fb5b2ffd640 1 -- 172.21.15.42:0/11537508 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(706..706 src has 451..706) ==== 5040+0+0 (secure 0 0 0) 0x7fb5b40490c0 con 0x7fb5bc0a1330 2024-09-16T04:05:25.259 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.238+0000 7fb5b2ffd640 10 client.984318.objecter ms_dispatch 0x7fb5bc000ba0 osd_map(706..706 src has 451..706) 2024-09-16T04:05:25.259 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.238+0000 7fb5b2ffd640 3 client.984318.objecter handle_osd_map got epochs [706,706] > 0 2024-09-16T04:05:25.259 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.238+0000 7fb5b2ffd640 3 client.984318.objecter handle_osd_map decoding full epoch 706 2024-09-16T04:05:25.259 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.238+0000 7fb5b2ffd640 20 client.984318.objecter dump_active .. 0 homeless 2024-09-16T04:05:25.259 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.238+0000 7fb5ba59c640 1 --2- 172.21.15.42:0/11537508 >> v2:172.21.15.154:6812/254845342 conn(0x7fb59c0377c0 0x7fb59c039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:05:25.259 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.238+0000 7fb5ba59c640 10 monclient: get_auth_request con 0x7fb59c0377c0 auth_method 0 2024-09-16T04:05:25.259 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.238+0000 7fb5ba59c640 1 --2- 172.21.15.42:0/11537508 >> v2:172.21.15.154:6812/254845342 conn(0x7fb59c0377c0 0x7fb59c039c80 secure :-1 s=READY pgs=3850 cs=0 l=1 rev1=1 crypto rx=0x7fb5ac0ad000 tx=0x7fb5ac008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:05:25.260 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.238+0000 7fb5b2ffd640 10 client.984318.objecter ms_handle_connect 0x7fb59c0377c0 2024-09-16T04:05:25.260 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.242+0000 7fb5b2ffd640 1 -- 172.21.15.42:0/11537508 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7fb5b401d580 con 0x7fb5bc0a1330 2024-09-16T04:05:25.260 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.242+0000 7fb5b2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:05:25.260 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.242+0000 7fb5b2ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:05:25.469 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.466+0000 7fb5c0d58640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:05:25.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.466+0000 7fb5c0d58640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:05:25.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.466+0000 7fb5c0d58640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:05:25.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.466+0000 7fb5c0d58640 1 -- 172.21.15.42:0/11537508 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fb5bc11f260 con 0x7fb5bc0a1330 2024-09-16T04:05:25.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.466+0000 7fb5b2ffd640 1 -- 172.21.15.42:0/11537508 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1562 (secure 0 0 0) 0x7fb5b401e8e0 con 0x7fb5bc0a1330 2024-09-16T04:05:25.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.466+0000 7fb5b2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:05:25.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.466+0000 7fb5b2ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:05:25.471 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":216,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":15,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:05:25.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.466+0000 7fb5c0d58640 10 librados: watch_flush enter 2024-09-16T04:05:25.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.466+0000 7fb5c0d58640 10 librados: watch_flush exit 2024-09-16T04:05:25.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.466+0000 7fb5c0d58640 20 client.984318.objecter shutdown clearing up homeless session... 2024-09-16T04:05:25.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.466+0000 7fb5c0d58640 10 client.984318.objecter successfully canceled tick 2024-09-16T04:05:25.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.466+0000 7fb5c0d58640 1 -- 172.21.15.42:0/11537508 >> v2:172.21.15.154:6812/254845342 conn(0x7fb59c0377c0 msgr2=0x7fb59c039c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:05:25.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.466+0000 7fb5c0d58640 1 --2- 172.21.15.42:0/11537508 >> v2:172.21.15.154:6812/254845342 conn(0x7fb59c0377c0 0x7fb59c039c80 secure :-1 s=READY pgs=3850 cs=0 l=1 rev1=1 crypto rx=0x7fb5ac0ad000 tx=0x7fb5ac008040 comp rx=0 tx=0).stop 2024-09-16T04:05:25.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.466+0000 7fb5c0d58640 10 monclient: shutdown 2024-09-16T04:05:25.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.466+0000 7fb5c0d58640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:05:25.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.466+0000 7fb5c0d58640 1 -- 172.21.15.42:0/11537508 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb5bc0a1330 msgr2=0x7fb5bc11ce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:05:25.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.466+0000 7fb5c0d58640 1 --2- 172.21.15.42:0/11537508 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb5bc0a1330 0x7fb5bc11ce20 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fb5b400d800 tx=0x7fb5b400dcc0 comp rx=0 tx=0).stop 2024-09-16T04:05:25.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.466+0000 7fb5c0d58640 1 -- 172.21.15.42:0/11537508 shutdown_connections 2024-09-16T04:05:25.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.466+0000 7fb5c0d58640 1 --2- 172.21.15.42:0/11537508 >> v2:172.21.15.154:6812/254845342 conn(0x7fb59c0377c0 0x7fb59c039c80 unknown :-1 s=CLOSED pgs=3850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:25.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.466+0000 7fb5c0d58640 1 --2- 172.21.15.42:0/11537508 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb5bc0a1330 0x7fb5bc11ce20 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:25.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.466+0000 7fb5c0d58640 1 --2- 172.21.15.42:0/11537508 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb5bc09c920 0x7fb5bc136350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:25.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.466+0000 7fb5c0d58640 1 --2- 172.21.15.42:0/11537508 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb5bc072d40 0x7fb5bc1cee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:25.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.466+0000 7fb5c0d58640 1 -- 172.21.15.42:0/11537508 >> 172.21.15.42:0/11537508 conn(0x7fb5bc0a2420 msgr2=0x7fb5bc0a3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:05:25.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.466+0000 7fb5c0d58640 1 -- 172.21.15.42:0/11537508 shutdown_connections 2024-09-16T04:05:25.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.466+0000 7fb5c0d58640 1 -- 172.21.15.42:0/11537508 wait complete. 2024-09-16T04:05:25.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:25.466+0000 7fb5c0d58640 1 librados: shutdown 2024-09-16T04:05:28.489 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:05:28.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:28.618+0000 7f28a3480640 10 monclient: get_monmap_and_config 2024-09-16T04:05:28.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:28.618+0000 7f28a3480640 10 monclient: build_initial_monmap 2024-09-16T04:05:28.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:28.618+0000 7f28a3480640 10 monclient: monmap: 2024-09-16T04:05:28.623 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:05:28.623 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:05:28.623 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:05:28.621441+0000 2024-09-16T04:05:28.623 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:05:28.621441+0000 2024-09-16T04:05:28.623 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:05:28.624 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:05:28.624 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:05:28.624 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:05:28.624 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:05:28.624 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:05:28.624 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:05:28.624 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:05:28.626 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:28.618+0000 7f28a3480640 1 Processor -- start 2024-09-16T04:05:28.626 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:28.618+0000 7f28a3480640 1 -- start start 2024-09-16T04:05:28.626 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:28.618+0000 7f28a3480640 10 monclient: init 2024-09-16T04:05:28.626 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:28.618+0000 7f28a3480640 10 monclient: _reopen_session rank -1 2024-09-16T04:05:28.627 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:28.618+0000 7f28a3480640 10 monclient: _add_conns ranks=[0,3,4,1,2] 2024-09-16T04:05:28.627 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:28.618+0000 7f28a3480640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f289c0a1330 addr v1:172.21.15.42:6789/0 2024-09-16T04:05:28.627 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:28.618+0000 7f28a3480640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f289c06dac0 0x7f289c13a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:05:28.627 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:28.618+0000 7f28a3480640 10 monclient(hunting): picked mon.noname-a con 0x7f289c06dac0 addr v2:172.21.15.42:3300/0 2024-09-16T04:05:28.627 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:28.618+0000 7f28a3480640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f289c0a1e20 0x7f289c13c890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:05:28.627 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:28.618+0000 7f28a3480640 10 monclient(hunting): picked mon.noname-c con 0x7f289c0a1e20 addr v2:172.21.15.154:3300/0 2024-09-16T04:05:28.627 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:28.618+0000 7f28a3480640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f289c09c090 con 0x7f289c0a1330 2024-09-16T04:05:28.627 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:28.618+0000 7f28a3480640 10 monclient(hunting): start opening mon connection 2024-09-16T04:05:28.627 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:28.618+0000 7f28a3480640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f289c09f610 con 0x7f289c06dac0 2024-09-16T04:05:28.627 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:28.618+0000 7f28a3480640 10 monclient(hunting): start opening mon connection 2024-09-16T04:05:28.627 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:28.618+0000 7f28a3480640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f289c09f8f0 con 0x7f289c0a1e20 2024-09-16T04:05:28.628 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:28.618+0000 7f28a3480640 10 monclient(hunting): _renew_subs 2024-09-16T04:05:28.628 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:28.618+0000 7f28a3480640 10 monclient(hunting): authenticate will time out at 2375.590909s 2024-09-16T04:05:28.628 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:28.618+0000 7f28a121c640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f289c0a1330 0x7f289c0a1710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:33494/0 (socket says 172.21.15.42:33494) 2024-09-16T04:05:28.628 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:28.618+0000 7f28a121c640 1 -- 172.21.15.42:0/1624123751 learned_addr learned my addr 172.21.15.42:0/1624123751 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T04:05:28.628 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:28.618+0000 7f28a0a1b640 1 --2- 172.21.15.42:0/1624123751 >> v2:172.21.15.42:3300/0 conn(0x7f289c06dac0 0x7f289c13a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:05:28.628 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:28.618+0000 7f28a0a1b640 10 monclient(hunting): get_auth_request con 0x7f289c06dac0 auth_method 0 2024-09-16T04:05:28.628 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:28.618+0000 7f28a0a1b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:05:28.628 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:28.618+0000 7f28a0a1b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:05:28.628 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:28.618+0000 7f28a0a1b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:05:28.628 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:28.618+0000 7f28a0a1b640 1 -- 172.21.15.42:0/1624123751 >> v2:172.21.15.42:3300/0 conn(0x7f289c06dac0 msgr2=0x7f289c13a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T04:05:28.628 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:28.618+0000 7f28a0a1b640 1 -- 172.21.15.42:0/1624123751 >> v2:172.21.15.42:3300/0 conn(0x7f289c06dac0 msgr2=0x7f289c13a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:05:28.629 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:28.618+0000 7f28a0a1b640 1 --2- 172.21.15.42:0/1624123751 >> v2:172.21.15.42:3300/0 conn(0x7f289c06dac0 0x7f289c13a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:05:28.629 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:28.618+0000 7f28a0a1b640 1 --2- 172.21.15.42:0/1624123751 >> v2:172.21.15.42:3300/0 conn(0x7f289c06dac0 0x7f289c13a340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:05:28.629 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:28.618+0000 7f28a1a1d640 1 --2- 172.21.15.42:0/1624123751 >> v2:172.21.15.154:3300/0 conn(0x7f289c0a1e20 0x7f289c13c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:05:28.629 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:28.618+0000 7f28a1a1d640 10 monclient(hunting): get_auth_request con 0x7f289c0a1e20 auth_method 0 2024-09-16T04:05:28.629 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:28.618+0000 7f28a1a1d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:05:28.629 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:28.618+0000 7f28a1a1d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:05:28.629 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:28.618+0000 7f28a1a1d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:05:28.629 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:28.618+0000 7f28a1a1d640 1 -- 172.21.15.42:0/1624123751 >> v2:172.21.15.154:3300/0 conn(0x7f289c0a1e20 msgr2=0x7f289c13c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:05:28.629 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:28.622+0000 7f28a1a1d640 1 -- 172.21.15.42:0/1624123751 >> v2:172.21.15.154:3300/0 conn(0x7f289c0a1e20 msgr2=0x7f289c13c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:05:28.629 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:28.622+0000 7f28a1a1d640 1 --2- 172.21.15.42:0/1624123751 >> v2:172.21.15.154:3300/0 conn(0x7f289c0a1e20 0x7f289c13c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:05:28.629 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:28.622+0000 7f28a1a1d640 1 --2- 172.21.15.42:0/1624123751 >> v2:172.21.15.154:3300/0 conn(0x7f289c0a1e20 0x7f289c13c890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:05:28.826 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:28.822+0000 7f28a0a1b640 1 --2- 172.21.15.42:0/1624123751 >> v2:172.21.15.42:3300/0 conn(0x7f289c06dac0 0x7f289c13a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:05:28.826 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:28.822+0000 7f28a0a1b640 10 monclient(hunting): get_auth_request con 0x7f289c06dac0 auth_method 0 2024-09-16T04:05:28.826 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:28.822+0000 7f28a0a1b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:05:28.826 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:28.822+0000 7f28a0a1b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:05:28.826 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:28.822+0000 7f28a0a1b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:05:28.826 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:28.822+0000 7f28a0a1b640 1 -- 172.21.15.42:0/1624123751 >> v2:172.21.15.42:3300/0 conn(0x7f289c06dac0 msgr2=0x7f289c13a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:05:28.826 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:28.822+0000 7f28a0a1b640 1 -- 172.21.15.42:0/1624123751 >> v2:172.21.15.42:3300/0 conn(0x7f289c06dac0 msgr2=0x7f289c13a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:05:28.826 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:28.822+0000 7f28a0a1b640 1 --2- 172.21.15.42:0/1624123751 >> v2:172.21.15.42:3300/0 conn(0x7f289c06dac0 0x7f289c13a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:05:28.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:28.822+0000 7f28a0a1b640 1 --2- 172.21.15.42:0/1624123751 >> v2:172.21.15.42:3300/0 conn(0x7f289c06dac0 0x7f289c13a340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T04:05:28.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:28.826+0000 7f28a1a1d640 1 --2- 172.21.15.42:0/1624123751 >> v2:172.21.15.154:3300/0 conn(0x7f289c0a1e20 0x7f289c13c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:05:28.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:28.826+0000 7f28a1a1d640 10 monclient(hunting): get_auth_request con 0x7f289c0a1e20 auth_method 0 2024-09-16T04:05:28.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:28.826+0000 7f28a1a1d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:05:28.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:28.826+0000 7f28a1a1d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:05:28.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:28.826+0000 7f28a1a1d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:05:28.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:28.826+0000 7f28a1a1d640 1 -- 172.21.15.42:0/1624123751 >> v2:172.21.15.154:3300/0 conn(0x7f289c0a1e20 msgr2=0x7f289c13c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:05:28.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:28.826+0000 7f28a1a1d640 1 -- 172.21.15.42:0/1624123751 >> v2:172.21.15.154:3300/0 conn(0x7f289c0a1e20 msgr2=0x7f289c13c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:05:28.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:28.826+0000 7f28a1a1d640 1 --2- 172.21.15.42:0/1624123751 >> v2:172.21.15.154:3300/0 conn(0x7f289c0a1e20 0x7f289c13c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:05:28.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:28.826+0000 7f28a1a1d640 1 --2- 172.21.15.42:0/1624123751 >> v2:172.21.15.154:3300/0 conn(0x7f289c0a1e20 0x7f289c13c890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T04:05:29.230 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:29.226+0000 7f28a0a1b640 1 --2- 172.21.15.42:0/1624123751 >> v2:172.21.15.42:3300/0 conn(0x7f289c06dac0 0x7f289c13a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:05:29.230 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:29.226+0000 7f28a0a1b640 10 monclient(hunting): get_auth_request con 0x7f289c06dac0 auth_method 0 2024-09-16T04:05:29.230 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:29.226+0000 7f28a0a1b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:05:29.230 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:29.226+0000 7f28a0a1b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:05:29.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:29.226+0000 7f28a0a1b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:05:29.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:29.226+0000 7f28a0a1b640 1 -- 172.21.15.42:0/1624123751 >> v2:172.21.15.42:3300/0 conn(0x7f289c06dac0 msgr2=0x7f289c13a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:05:29.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:29.226+0000 7f28a0a1b640 1 -- 172.21.15.42:0/1624123751 >> v2:172.21.15.42:3300/0 conn(0x7f289c06dac0 msgr2=0x7f289c13a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:05:29.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:29.226+0000 7f28a0a1b640 1 --2- 172.21.15.42:0/1624123751 >> v2:172.21.15.42:3300/0 conn(0x7f289c06dac0 0x7f289c13a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:05:29.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:29.226+0000 7f28a0a1b640 1 --2- 172.21.15.42:0/1624123751 >> v2:172.21.15.42:3300/0 conn(0x7f289c06dac0 0x7f289c13a340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T04:05:29.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:29.230+0000 7f28a1a1d640 1 --2- 172.21.15.42:0/1624123751 >> v2:172.21.15.154:3300/0 conn(0x7f289c0a1e20 0x7f289c13c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:05:29.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:29.230+0000 7f28a1a1d640 10 monclient(hunting): get_auth_request con 0x7f289c0a1e20 auth_method 0 2024-09-16T04:05:29.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:29.230+0000 7f28a1a1d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:05:29.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:29.230+0000 7f28a1a1d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:05:29.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:29.230+0000 7f28a1a1d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:05:29.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:29.230+0000 7f28a1a1d640 1 -- 172.21.15.42:0/1624123751 >> v2:172.21.15.154:3300/0 conn(0x7f289c0a1e20 msgr2=0x7f289c13c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:05:29.235 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:29.230+0000 7f28a1a1d640 1 -- 172.21.15.42:0/1624123751 >> v2:172.21.15.154:3300/0 conn(0x7f289c0a1e20 msgr2=0x7f289c13c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:05:29.235 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:29.230+0000 7f28a1a1d640 1 --2- 172.21.15.42:0/1624123751 >> v2:172.21.15.154:3300/0 conn(0x7f289c0a1e20 0x7f289c13c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:05:29.235 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:29.230+0000 7f28a1a1d640 1 --2- 172.21.15.42:0/1624123751 >> v2:172.21.15.154:3300/0 conn(0x7f289c0a1e20 0x7f289c13c890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T04:05:30.034 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:30.030+0000 7f28a0a1b640 1 --2- 172.21.15.42:0/1624123751 >> v2:172.21.15.42:3300/0 conn(0x7f289c06dac0 0x7f289c13a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:05:30.034 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:30.030+0000 7f28a0a1b640 10 monclient(hunting): get_auth_request con 0x7f289c06dac0 auth_method 0 2024-09-16T04:05:30.034 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:30.030+0000 7f28a0a1b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:05:30.034 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:30.030+0000 7f28a0a1b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:05:30.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:30.030+0000 7f28a0a1b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:05:30.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:30.030+0000 7f28a0a1b640 1 -- 172.21.15.42:0/1624123751 >> v2:172.21.15.42:3300/0 conn(0x7f289c06dac0 msgr2=0x7f289c13a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:05:30.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:30.030+0000 7f28a0a1b640 1 -- 172.21.15.42:0/1624123751 >> v2:172.21.15.42:3300/0 conn(0x7f289c06dac0 msgr2=0x7f289c13a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:05:30.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:30.030+0000 7f28a0a1b640 1 --2- 172.21.15.42:0/1624123751 >> v2:172.21.15.42:3300/0 conn(0x7f289c06dac0 0x7f289c13a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:05:30.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:30.030+0000 7f28a0a1b640 1 --2- 172.21.15.42:0/1624123751 >> v2:172.21.15.42:3300/0 conn(0x7f289c06dac0 0x7f289c13a340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T04:05:30.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:30.034+0000 7f28a1a1d640 1 --2- 172.21.15.42:0/1624123751 >> v2:172.21.15.154:3300/0 conn(0x7f289c0a1e20 0x7f289c13c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:05:30.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:30.034+0000 7f28a1a1d640 10 monclient(hunting): get_auth_request con 0x7f289c0a1e20 auth_method 0 2024-09-16T04:05:30.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:30.034+0000 7f28a1a1d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:05:30.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:30.034+0000 7f28a1a1d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:05:30.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:30.034+0000 7f28a1a1d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:05:30.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:30.034+0000 7f28a1a1d640 1 -- 172.21.15.42:0/1624123751 >> v2:172.21.15.154:3300/0 conn(0x7f289c0a1e20 msgr2=0x7f289c13c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:05:30.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:30.034+0000 7f28a1a1d640 1 -- 172.21.15.42:0/1624123751 >> v2:172.21.15.154:3300/0 conn(0x7f289c0a1e20 msgr2=0x7f289c13c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:05:30.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:30.034+0000 7f28a1a1d640 1 --2- 172.21.15.42:0/1624123751 >> v2:172.21.15.154:3300/0 conn(0x7f289c0a1e20 0x7f289c13c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:05:30.039 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:30.034+0000 7f28a1a1d640 1 --2- 172.21.15.42:0/1624123751 >> v2:172.21.15.154:3300/0 conn(0x7f289c0a1e20 0x7f289c13c890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T04:05:31.625 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:31.622+0000 7f288effd640 10 monclient(hunting): tick 2024-09-16T04:05:31.625 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:31.622+0000 7f288effd640 10 monclient(hunting): _check_auth_tickets 2024-09-16T04:05:31.625 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:31.622+0000 7f288effd640 1 monclient(hunting): continuing hunt 2024-09-16T04:05:31.625 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:31.622+0000 7f288effd640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T04:05:31.625 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:31.622+0000 7f288effd640 1 -- 172.21.15.42:0/1624123751 >> v2:172.21.15.154:3300/0 conn(0x7f289c0a1e20 msgr2=0x7f289c13c890 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:05:31.625 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:31.622+0000 7f288effd640 1 --2- 172.21.15.42:0/1624123751 >> v2:172.21.15.154:3300/0 conn(0x7f289c0a1e20 0x7f289c13c890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:31.626 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:31.622+0000 7f288effd640 1 -- 172.21.15.42:0/1624123751 >> v2:172.21.15.42:3300/0 conn(0x7f289c06dac0 msgr2=0x7f289c13a340 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:05:31.626 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:31.622+0000 7f288effd640 1 --2- 172.21.15.42:0/1624123751 >> v2:172.21.15.42:3300/0 conn(0x7f289c06dac0 0x7f289c13a340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:31.626 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:31.622+0000 7f288effd640 1 -- 172.21.15.42:0/1624123751 >> v1:172.21.15.42:6789/0 conn(0x7f289c0a1330 legacy=0x7f289c0a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:05:31.626 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:31.622+0000 7f288effd640 10 monclient: _add_conns ranks=[2,1] 2024-09-16T04:05:31.626 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:31.622+0000 7f288effd640 1 --2- 172.21.15.42:0/1624123751 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f28800036c0 0x7f2880005af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:05:31.626 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:31.622+0000 7f288effd640 10 monclient(hunting): picked mon.noname-b con 0x7f28800036c0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:05:31.626 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:31.622+0000 7f288effd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2880006180 addr v1:172.21.15.154:6789/0 2024-09-16T04:05:31.626 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:31.622+0000 7f288effd640 1 -- 172.21.15.42:0/1624123751 --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f289c09fbd0 con 0x7f2880006180 2024-09-16T04:05:31.626 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:31.622+0000 7f288effd640 10 monclient(hunting): start opening mon connection 2024-09-16T04:05:31.626 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:31.622+0000 7f288effd640 1 -- 172.21.15.42:0/1624123751 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f289c09f610 con 0x7f28800036c0 2024-09-16T04:05:31.626 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:31.622+0000 7f288effd640 10 monclient(hunting): _renew_subs 2024-09-16T04:05:31.627 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:31.622+0000 7f28a121c640 1 -- 172.21.15.42:0/1624123751 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f28800036c0 msgr2=0x7f2880005af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T04:05:31.627 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:31.622+0000 7f28a121c640 1 --2- 172.21.15.42:0/1624123751 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f28800036c0 0x7f2880005af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:05:31.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:31.826+0000 7f28a121c640 1 -- 172.21.15.42:0/1624123751 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f28800036c0 msgr2=0x7f2880005af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T04:05:31.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:31.826+0000 7f28a121c640 1 --2- 172.21.15.42:0/1624123751 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f28800036c0 0x7f2880005af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T04:05:32.233 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:32.230+0000 7f28a121c640 1 -- 172.21.15.42:0/1624123751 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f28800036c0 msgr2=0x7f2880005af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T04:05:32.233 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:32.230+0000 7f28a121c640 1 --2- 172.21.15.42:0/1624123751 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f28800036c0 0x7f2880005af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T04:05:33.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.034+0000 7f28a121c640 1 -- 172.21.15.42:0/1624123751 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f28800036c0 msgr2=0x7f2880005af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T04:05:33.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.034+0000 7f28a121c640 1 --2- 172.21.15.42:0/1624123751 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f28800036c0 0x7f2880005af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T04:05:33.255 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.250+0000 7f288ffff640 1 -- 172.21.15.42:0/1624123751 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2971990015 0 0) 0x7f289c09fbd0 con 0x7f2880006180 2024-09-16T04:05:33.255 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.250+0000 7f288ffff640 10 monclient(hunting): _negotiate 2024-09-16T04:05:33.255 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.250+0000 7f288ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:05:33.256 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.250+0000 7f288ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:05:33.256 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.250+0000 7f288ffff640 10 monclient(hunting): my global_id is 904939 2024-09-16T04:05:33.256 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.250+0000 7f288ffff640 1 -- 172.21.15.42:0/1624123751 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2888003530 con 0x7f2880006180 2024-09-16T04:05:33.272 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.266+0000 7f288ffff640 1 -- 172.21.15.42:0/1624123751 <== mon.1 v1:172.21.15.154:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 308038478 0 0) 0x7f2888003530 con 0x7f2880006180 2024-09-16T04:05:33.273 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.266+0000 7f288ffff640 1 -- 172.21.15.42:0/1624123751 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f28800036c0 msgr2=0x7f2880005af0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:05:33.273 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.266+0000 7f288ffff640 1 --2- 172.21.15.42:0/1624123751 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f28800036c0 0x7f2880005af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:33.273 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.266+0000 7f288ffff640 10 monclient: _finish_hunting 0 2024-09-16T04:05:33.273 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.266+0000 7f288ffff640 1 monclient: found mon.noname-c-legacy 2024-09-16T04:05:33.273 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.266+0000 7f288ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:05:33.273 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.266+0000 7f288ffff640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.154:6789/0 2024-09-16T04:05:33.273 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.266+0000 7f288ffff640 1 -- 172.21.15.42:0/1624123751 --> v1:172.21.15.154:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f288000afc0 con 0x7f2880006180 2024-09-16T04:05:33.273 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.266+0000 7f288ffff640 10 monclient: _finish_auth 0 2024-09-16T04:05:33.273 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.266+0000 7f288ffff640 10 monclient: _check_auth_tickets 2024-09-16T04:05:33.273 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.266+0000 7f288ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:05:33.273 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.266+0000 7f288ffff640 1 -- 172.21.15.42:0/1624123751 <== mon.1 v1:172.21.15.154:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2177413734 0 0) 0x7f289800f040 con 0x7f2880006180 2024-09-16T04:05:33.273 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.266+0000 7f288ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:05:33.274 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.266+0000 7f288ffff640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-16T04:05:33.274 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.266+0000 7f288ffff640 10 monclient: dump: 2024-09-16T04:05:33.274 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:05:33.274 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:05:33.274 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:33.274 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:33.274 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:05:33.274 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:05:33.274 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:05:33.274 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:05:33.274 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:05:33.274 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:05:33.274 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.266+0000 7f288ffff640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] but i'm connected to v1:172.21.15.154:6789/0, reconnecting 2024-09-16T04:05:33.275 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.266+0000 7f288ffff640 10 monclient: _reopen_session rank -1 2024-09-16T04:05:33.275 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.266+0000 7f288ffff640 1 -- 172.21.15.42:0/1624123751 >> v1:172.21.15.154:6789/0 conn(0x7f2880006180 legacy=0x7f2880008640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:05:33.275 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.266+0000 7f28a1a1d640 1 -- 172.21.15.42:0/1624123751 reap_dead start 2024-09-16T04:05:33.275 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.266+0000 7f288ffff640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:05:33.275 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.266+0000 7f288ffff640 1 --2- 172.21.15.42:0/1624123751 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f28880039c0 0x7f2888003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:05:33.275 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.266+0000 7f288ffff640 10 monclient(hunting): picked mon.a con 0x7f28880039c0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:05:33.275 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.266+0000 7f288ffff640 1 --2- 172.21.15.42:0/1624123751 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2888004810 0x7f2888004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:05:33.275 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.266+0000 7f288ffff640 10 monclient(hunting): picked mon.c con 0x7f2888004810 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:05:33.275 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.266+0000 7f288ffff640 1 --2- 172.21.15.42:0/1624123751 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f28880052f0 0x7f288800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:05:33.275 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.266+0000 7f288ffff640 10 monclient(hunting): picked mon.b con 0x7f28880052f0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:05:33.276 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.266+0000 7f28a0a1b640 1 -- 172.21.15.42:0/1624123751 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2888004810 msgr2=0x7f2888004c90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T04:05:33.276 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.266+0000 7f288ffff640 10 monclient(hunting): start opening mon connection 2024-09-16T04:05:33.276 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.266+0000 7f28a0a1b640 1 --2- 172.21.15.42:0/1624123751 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2888004810 0x7f2888004c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:05:33.276 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.266+0000 7f28a121c640 1 --2- 172.21.15.42:0/1624123751 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f28880039c0 0x7f2888003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:05:33.276 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.266+0000 7f288ffff640 1 -- 172.21.15.42:0/1624123751 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f289c09f610 con 0x7f28880039c0 2024-09-16T04:05:33.276 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.266+0000 7f288ffff640 10 monclient(hunting): start opening mon connection 2024-09-16T04:05:33.276 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.266+0000 7f288ffff640 1 -- 172.21.15.42:0/1624123751 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f288800c240 con 0x7f28880052f0 2024-09-16T04:05:33.276 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.266+0000 7f288ffff640 10 monclient(hunting): start opening mon connection 2024-09-16T04:05:33.276 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.266+0000 7f288ffff640 1 -- 172.21.15.42:0/1624123751 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f288800c440 con 0x7f2888004810 2024-09-16T04:05:33.276 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.266+0000 7f288ffff640 10 monclient(hunting): _renew_subs 2024-09-16T04:05:33.276 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.266+0000 7f288ffff640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:05:33.276 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.266+0000 7f28a121c640 10 monclient(hunting): get_auth_request con 0x7f28880039c0 auth_method 0 2024-09-16T04:05:33.276 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.266+0000 7f28a121c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:05:33.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.266+0000 7f28a121c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:05:33.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.266+0000 7f28a121c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:05:33.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.266+0000 7f28a121c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:05:33.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.266+0000 7f28a121c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:05:33.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.266+0000 7f28a1a1d640 1 --2- 172.21.15.42:0/1624123751 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f28880052f0 0x7f288800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:05:33.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.266+0000 7f28a121c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:05:33.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.266+0000 7f28a1a1d640 10 monclient(hunting): get_auth_request con 0x7f28880052f0 auth_method 0 2024-09-16T04:05:33.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.266+0000 7f28a1a1d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:05:33.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.266+0000 7f28a1a1d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:05:33.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.266+0000 7f28a1a1d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:05:33.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.266+0000 7f28a1a1d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:05:33.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.266+0000 7f28a1a1d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:05:33.278 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.266+0000 7f28a1a1d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:05:33.278 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.266+0000 7f28a121c640 10 monclient(hunting): handle_auth_done global_id 904939 payload 293 2024-09-16T04:05:33.278 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.266+0000 7f28a121c640 1 -- 172.21.15.42:0/1624123751 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2888004810 msgr2=0x7f2888004c90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:05:33.278 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.266+0000 7f28a121c640 1 --2- 172.21.15.42:0/1624123751 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2888004810 0x7f2888004c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:33.278 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.266+0000 7f28a121c640 1 -- 172.21.15.42:0/1624123751 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f28880052f0 msgr2=0x7f288800b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:05:33.278 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.266+0000 7f28a121c640 1 --2- 172.21.15.42:0/1624123751 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f28880052f0 0x7f288800b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:33.278 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.266+0000 7f28a121c640 10 monclient: _finish_hunting 0 2024-09-16T04:05:33.278 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.266+0000 7f28a121c640 1 monclient: found mon.a 2024-09-16T04:05:33.278 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.266+0000 7f28a121c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:05:33.278 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.266+0000 7f28a121c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:05:33.278 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.266+0000 7f28a121c640 1 -- 172.21.15.42:0/1624123751 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2888003530 con 0x7f28880039c0 2024-09-16T04:05:33.278 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.266+0000 7f28a121c640 10 monclient: _finish_auth 0 2024-09-16T04:05:33.278 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.266+0000 7f28a121c640 10 monclient: _check_auth_tickets 2024-09-16T04:05:33.279 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.266+0000 7f28a121c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:05:33.279 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.266+0000 7f28a3480640 5 monclient: authenticate success, global_id 904939 2024-09-16T04:05:33.279 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.266+0000 7f28a3480640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:05:33.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.274+0000 7f28a121c640 1 --2- 172.21.15.42:0/1624123751 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f28880039c0 0x7f2888003dc0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f2890001130 tx=0x7f289001c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=118c4ccedddc747d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:05:33.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.274+0000 7f288ffff640 1 -- 172.21.15.42:0/1624123751 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28900ad070 con 0x7f28880039c0 2024-09-16T04:05:33.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.274+0000 7f288ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:05:33.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.274+0000 7f288ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:05:33.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.274+0000 7f288ffff640 10 monclient: dump: 2024-09-16T04:05:33.280 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:05:33.280 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:05:33.280 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:33.281 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:33.281 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:05:33.281 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:05:33.281 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:05:33.281 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:05:33.281 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:05:33.281 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:05:33.281 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.274+0000 7f28a3480640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:05:33.281 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.274+0000 7f288ffff640 1 -- 172.21.15.42:0/1624123751 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2890004ab0 con 0x7f28880039c0 2024-09-16T04:05:33.281 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.274+0000 7f288ffff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:05:33.281 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.274+0000 7f28a3480640 10 monclient: get_monmap_and_config success 2024-09-16T04:05:33.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.274+0000 7f28a3480640 4 set_mon_vals no callback set 2024-09-16T04:05:33.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.274+0000 7f288ffff640 1 -- 172.21.15.42:0/1624123751 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28900017f0 con 0x7f28880039c0 2024-09-16T04:05:33.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.274+0000 7f288ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:05:33.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.274+0000 7f288ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:05:33.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.274+0000 7f288ffff640 10 monclient: dump: 2024-09-16T04:05:33.282 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:05:33.282 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:05:33.282 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:33.282 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:33.282 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:05:33.282 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:05:33.283 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:05:33.283 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:05:33.283 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:05:33.283 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:05:33.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.274+0000 7f28a3480640 10 monclient: shutdown 2024-09-16T04:05:33.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.274+0000 7f28a3480640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:05:33.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.274+0000 7f28a3480640 1 -- 172.21.15.42:0/1624123751 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f28880039c0 msgr2=0x7f2888003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:05:33.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.274+0000 7f28a3480640 1 --2- 172.21.15.42:0/1624123751 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f28880039c0 0x7f2888003dc0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f2890001130 tx=0x7f289001c8d0 comp rx=0 tx=0).stop 2024-09-16T04:05:33.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.274+0000 7f28a3480640 1 -- 172.21.15.42:0/1624123751 shutdown_connections 2024-09-16T04:05:33.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.274+0000 7f28a3480640 1 --2- 172.21.15.42:0/1624123751 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f28880052f0 0x7f288800b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:33.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.274+0000 7f28a3480640 1 --2- 172.21.15.42:0/1624123751 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2888004810 0x7f2888004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:33.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.274+0000 7f28a3480640 1 --2- 172.21.15.42:0/1624123751 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f28880039c0 0x7f2888003dc0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:33.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.274+0000 7f28a3480640 1 -- 172.21.15.42:0/1624123751 >> 172.21.15.42:0/1624123751 conn(0x7f289c0a2420 msgr2=0x7f289c0a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:05:33.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.274+0000 7f28a3480640 1 -- 172.21.15.42:0/1624123751 shutdown_connections 2024-09-16T04:05:33.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.274+0000 7f28a3480640 1 -- 172.21.15.42:0/1624123751 wait complete. 2024-09-16T04:05:33.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.274+0000 7f28a3480640 10 monclient: build_initial_monmap 2024-09-16T04:05:33.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.274+0000 7f28a3480640 10 monclient: monmap: 2024-09-16T04:05:33.288 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:05:33.288 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:05:33.288 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:05:33.289 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:05:33.289 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:05:33.289 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:05:33.289 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:05:33.289 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:05:33.289 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:05:33.289 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:05:33.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.274+0000 7f28a3480640 1 librados: starting msgr at 2024-09-16T04:05:33.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.274+0000 7f28a3480640 1 librados: starting objecter 2024-09-16T04:05:33.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.274+0000 7f28a3480640 1 Processor -- start 2024-09-16T04:05:33.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.274+0000 7f28a3480640 1 -- start start 2024-09-16T04:05:33.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.274+0000 7f28a3480640 1 librados: setting wanted keys 2024-09-16T04:05:33.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.274+0000 7f28a3480640 1 librados: calling monclient init 2024-09-16T04:05:33.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.274+0000 7f28a3480640 10 monclient: init 2024-09-16T04:05:33.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.278+0000 7f28a3480640 10 monclient: _reopen_session rank -1 2024-09-16T04:05:33.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.278+0000 7f28a3480640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:05:33.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.278+0000 7f28a3480640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f28880039c0 0x7f289c1ce700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:05:33.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.278+0000 7f28a3480640 10 monclient(hunting): picked mon.noname-a con 0x7f28880039c0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:05:33.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.278+0000 7f28a3480640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2888004810 0x7f289c122f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:05:33.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.278+0000 7f28a3480640 10 monclient(hunting): picked mon.noname-b con 0x7f2888004810 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:05:33.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.278+0000 7f28a3480640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f28880052f0 0x7f289c123440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:05:33.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.278+0000 7f28a3480640 10 monclient(hunting): picked mon.noname-c con 0x7f28880052f0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:05:33.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.278+0000 7f28a3480640 10 monclient(hunting): start opening mon connection 2024-09-16T04:05:33.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.278+0000 7f28a3480640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f289c09eef0 con 0x7f28880039c0 2024-09-16T04:05:33.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.278+0000 7f28a3480640 10 monclient(hunting): start opening mon connection 2024-09-16T04:05:33.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.278+0000 7f28a3480640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f289c09ed70 con 0x7f2888004810 2024-09-16T04:05:33.291 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.278+0000 7f28a3480640 10 monclient(hunting): start opening mon connection 2024-09-16T04:05:33.291 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.278+0000 7f28a3480640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f289c135100 con 0x7f28880052f0 2024-09-16T04:05:33.291 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.278+0000 7f28a3480640 10 monclient(hunting): _renew_subs 2024-09-16T04:05:33.291 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.278+0000 7f28a3480640 10 monclient(hunting): authenticate will time out at 2380.248784s 2024-09-16T04:05:33.291 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.278+0000 7f28a1a1d640 1 -- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f28880052f0 msgr2=0x7f289c123440 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T04:05:33.291 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.278+0000 7f28a1a1d640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f28880052f0 0x7f289c123440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:05:33.291 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.278+0000 7f28a121c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f28880039c0 0x7f289c1ce700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:05:33.291 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.278+0000 7f28a121c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f28880039c0 0x7f289c1ce700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:56656/0 (socket says 172.21.15.42:56656) 2024-09-16T04:05:33.291 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.278+0000 7f28a121c640 1 -- 172.21.15.42:0/1867717355 learned_addr learned my addr 172.21.15.42:0/1867717355 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:05:33.291 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.278+0000 7f28a121c640 10 monclient(hunting): get_auth_request con 0x7f28880039c0 auth_method 0 2024-09-16T04:05:33.291 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.278+0000 7f28a121c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:05:33.291 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.278+0000 7f28a121c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:05:33.291 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.278+0000 7f28a121c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:05:33.292 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.278+0000 7f28a121c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:05:33.292 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.278+0000 7f28a121c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:05:33.292 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.278+0000 7f28a121c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:05:33.292 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.278+0000 7f28a0a1b640 1 --2- 172.21.15.42:0/1867717355 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2888004810 0x7f289c122f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:05:33.292 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.278+0000 7f28a0a1b640 10 monclient(hunting): get_auth_request con 0x7f2888004810 auth_method 0 2024-09-16T04:05:33.292 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.278+0000 7f28a0a1b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:05:33.292 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.278+0000 7f28a0a1b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:05:33.292 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.278+0000 7f28a0a1b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:05:33.292 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.278+0000 7f28a121c640 10 monclient(hunting): handle_auth_done global_id 984321 payload 824 2024-09-16T04:05:33.292 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.278+0000 7f28a121c640 1 -- 172.21.15.42:0/1867717355 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f28880052f0 msgr2=0x7f289c123440 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:05:33.292 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.278+0000 7f28a121c640 1 --2- 172.21.15.42:0/1867717355 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f28880052f0 0x7f289c123440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:33.292 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.278+0000 7f28a121c640 1 -- 172.21.15.42:0/1867717355 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2888004810 msgr2=0x7f289c122f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:05:33.292 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.278+0000 7f28a121c640 1 --2- 172.21.15.42:0/1867717355 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2888004810 0x7f289c122f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:33.293 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.278+0000 7f28a121c640 10 monclient: _finish_hunting 0 2024-09-16T04:05:33.293 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.278+0000 7f28a121c640 1 monclient: found mon.noname-a 2024-09-16T04:05:33.293 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.278+0000 7f28a121c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:05:33.293 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.278+0000 7f28a121c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:05:33.293 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.278+0000 7f28a121c640 1 -- 172.21.15.42:0/1867717355 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f289c127870 con 0x7f28880039c0 2024-09-16T04:05:33.293 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.278+0000 7f28a121c640 1 --2- 172.21.15.42:0/1867717355 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f28880039c0 0x7f289c1ce700 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f289001c700 tx=0x7f2890019670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:05:33.293 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.278+0000 7f288dffb640 10 client.?.objecter ms_handle_connect 0x7f28880039c0 2024-09-16T04:05:33.293 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.278+0000 7f288dffb640 10 client.?.objecter resend_mon_ops 2024-09-16T04:05:33.293 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.278+0000 7f288dffb640 1 -- 172.21.15.42:0/1867717355 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28900b7070 con 0x7f28880039c0 2024-09-16T04:05:33.293 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.278+0000 7f288dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:05:33.293 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.278+0000 7f288dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:05:33.293 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.278+0000 7f288dffb640 10 monclient: dump: 2024-09-16T04:05:33.293 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:05:33.293 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:05:33.294 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:33.294 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:33.294 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:05:33.294 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:05:33.294 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:05:33.294 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:05:33.294 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:05:33.294 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:05:33.294 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.278+0000 7f288dffb640 10 monclient: _finish_auth 0 2024-09-16T04:05:33.294 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.278+0000 7f288dffb640 10 monclient: _check_auth_tickets 2024-09-16T04:05:33.294 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.278+0000 7f288dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:05:33.294 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.278+0000 7f288dffb640 1 -- 172.21.15.42:0/1867717355 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f28900ae040 con 0x7f28880039c0 2024-09-16T04:05:33.295 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.278+0000 7f28a3480640 5 monclient: authenticate success, global_id 984321 2024-09-16T04:05:33.295 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.278+0000 7f28a3480640 10 monclient: _renew_subs 2024-09-16T04:05:33.295 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.278+0000 7f28a3480640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:05:33.295 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.278+0000 7f28a3480640 1 -- 172.21.15.42:0/1867717355 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f289c126bb0 con 0x7f28880039c0 2024-09-16T04:05:33.295 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.278+0000 7f288dffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:05:33.295 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.278+0000 7f288dffb640 1 -- 172.21.15.42:0/1867717355 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28900ad050 con 0x7f28880039c0 2024-09-16T04:05:33.295 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.278+0000 7f288dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:05:33.295 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.278+0000 7f288dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:05:33.295 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.278+0000 7f288dffb640 10 monclient: dump: 2024-09-16T04:05:33.295 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:05:33.295 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:05:33.295 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:33.295 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:33.295 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:05:33.296 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:05:33.296 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:05:33.296 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:05:33.296 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:05:33.296 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:05:33.296 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.278+0000 7f288f7fe640 4 set_mon_vals no callback set 2024-09-16T04:05:33.296 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.278+0000 7f28a3480640 10 client.984321.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:05:33.296 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.278+0000 7f28a3480640 10 monclient: _renew_subs 2024-09-16T04:05:33.296 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.278+0000 7f28a3480640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:05:33.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.278+0000 7f28a3480640 1 -- 172.21.15.42:0/1867717355 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f289c134550 con 0x7f28880039c0 2024-09-16T04:05:33.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.278+0000 7f28a3480640 1 librados: init done 2024-09-16T04:05:33.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.278+0000 7f28a3480640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:05:33.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.278+0000 7f28a3480640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:05:33.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.278+0000 7f28a3480640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:05:33.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.278+0000 7f28a3480640 1 -- 172.21.15.42:0/1867717355 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f289c09f8f0 con 0x7f28880039c0 2024-09-16T04:05:33.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.278+0000 7f288dffb640 1 -- 172.21.15.42:0/1867717355 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f28900ba070 con 0x7f28880039c0 2024-09-16T04:05:33.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.278+0000 7f288dffb640 1 --2- 172.21.15.42:0/1867717355 >> v2:172.21.15.154:6812/254845342 conn(0x7f28800380a0 0x7f288003a560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:05:33.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.278+0000 7f288dffb640 1 -- 172.21.15.42:0/1867717355 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(706..706 src has 451..706) ==== 5040+0+0 (secure 0 0 0) 0x7f28900e1d50 con 0x7f28880039c0 2024-09-16T04:05:33.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.278+0000 7f288dffb640 10 client.984321.objecter ms_dispatch 0x7f289c000ba0 osd_map(706..706 src has 451..706) 2024-09-16T04:05:33.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.278+0000 7f288dffb640 3 client.984321.objecter handle_osd_map got epochs [706,706] > 0 2024-09-16T04:05:33.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.278+0000 7f288dffb640 3 client.984321.objecter handle_osd_map decoding full epoch 706 2024-09-16T04:05:33.298 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.278+0000 7f288dffb640 20 client.984321.objecter dump_active .. 0 homeless 2024-09-16T04:05:33.298 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.278+0000 7f28a0a1b640 1 --2- 172.21.15.42:0/1867717355 >> v2:172.21.15.154:6812/254845342 conn(0x7f28800380a0 0x7f288003a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:05:33.298 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.278+0000 7f28a0a1b640 10 monclient: get_auth_request con 0x7f28800380a0 auth_method 0 2024-09-16T04:05:33.298 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.282+0000 7f28a0a1b640 1 --2- 172.21.15.42:0/1867717355 >> v2:172.21.15.154:6812/254845342 conn(0x7f28800380a0 0x7f288003a560 secure :-1 s=READY pgs=3851 cs=0 l=1 rev1=1 crypto rx=0x7f289c1247a0 tx=0x7f2898010040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:05:33.298 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.282+0000 7f288dffb640 10 client.984321.objecter ms_handle_connect 0x7f28800380a0 2024-09-16T04:05:33.298 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.286+0000 7f288dffb640 1 -- 172.21.15.42:0/1867717355 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f289c127870 con 0x7f28880039c0 2024-09-16T04:05:33.298 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.286+0000 7f288dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:05:33.298 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.286+0000 7f288dffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:05:33.521 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.514+0000 7f28a3480640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:05:33.521 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.514+0000 7f28a3480640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:05:33.521 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.514+0000 7f28a3480640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:05:33.521 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.514+0000 7f28a3480640 1 -- 172.21.15.42:0/1867717355 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f289c125480 con 0x7f28880039c0 2024-09-16T04:05:33.522 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.518+0000 7f288dffb640 1 -- 172.21.15.42:0/1867717355 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1555 (secure 0 0 0) 0x7f2890016550 con 0x7f28880039c0 2024-09-16T04:05:33.522 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.518+0000 7f288dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:05:33.522 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.518+0000 7f288dffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:05:33.522 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":220,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:05:33.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.518+0000 7f28a3480640 10 librados: watch_flush enter 2024-09-16T04:05:33.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.518+0000 7f28a3480640 10 librados: watch_flush exit 2024-09-16T04:05:33.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.518+0000 7f28a3480640 20 client.984321.objecter shutdown clearing up homeless session... 2024-09-16T04:05:33.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.518+0000 7f28a3480640 10 client.984321.objecter successfully canceled tick 2024-09-16T04:05:33.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.518+0000 7f28a3480640 1 -- 172.21.15.42:0/1867717355 >> v2:172.21.15.154:6812/254845342 conn(0x7f28800380a0 msgr2=0x7f288003a560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:05:33.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.518+0000 7f28a3480640 1 --2- 172.21.15.42:0/1867717355 >> v2:172.21.15.154:6812/254845342 conn(0x7f28800380a0 0x7f288003a560 secure :-1 s=READY pgs=3851 cs=0 l=1 rev1=1 crypto rx=0x7f289c1247a0 tx=0x7f2898010040 comp rx=0 tx=0).stop 2024-09-16T04:05:33.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.518+0000 7f28a3480640 10 monclient: shutdown 2024-09-16T04:05:33.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.518+0000 7f28a3480640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:05:33.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.518+0000 7f28a3480640 1 -- 172.21.15.42:0/1867717355 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f28880039c0 msgr2=0x7f289c1ce700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:05:33.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.518+0000 7f28a3480640 1 --2- 172.21.15.42:0/1867717355 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f28880039c0 0x7f289c1ce700 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f289001c700 tx=0x7f2890019670 comp rx=0 tx=0).stop 2024-09-16T04:05:33.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.518+0000 7f28a3480640 1 -- 172.21.15.42:0/1867717355 shutdown_connections 2024-09-16T04:05:33.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.518+0000 7f28a3480640 1 --2- 172.21.15.42:0/1867717355 >> v2:172.21.15.154:6812/254845342 conn(0x7f28800380a0 0x7f288003a560 unknown :-1 s=CLOSED pgs=3851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:33.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.518+0000 7f28a3480640 1 --2- 172.21.15.42:0/1867717355 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f28880052f0 0x7f289c123440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:33.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.518+0000 7f28a3480640 1 --2- 172.21.15.42:0/1867717355 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2888004810 0x7f289c122f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:33.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.518+0000 7f28a3480640 1 --2- 172.21.15.42:0/1867717355 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f28880039c0 0x7f289c1ce700 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:33.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.518+0000 7f28a3480640 1 -- 172.21.15.42:0/1867717355 >> 172.21.15.42:0/1867717355 conn(0x7f289c0a2420 msgr2=0x7f289c09d540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:05:33.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.518+0000 7f28a3480640 1 -- 172.21.15.42:0/1867717355 shutdown_connections 2024-09-16T04:05:33.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.518+0000 7f28a3480640 1 -- 172.21.15.42:0/1867717355 wait complete. 2024-09-16T04:05:33.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.518+0000 7f28a3480640 1 librados: shutdown 2024-09-16T04:05:33.542 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T04:05:33.542 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-16T04:05:33.542 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T04:05:33.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.662+0000 7f46af068640 10 monclient: get_monmap_and_config 2024-09-16T04:05:33.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.662+0000 7f46af068640 10 monclient: build_initial_monmap 2024-09-16T04:05:33.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.662+0000 7f46af068640 10 monclient: monmap: 2024-09-16T04:05:33.667 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:05:33.667 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:05:33.667 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:05:33.665481+0000 2024-09-16T04:05:33.667 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:05:33.665481+0000 2024-09-16T04:05:33.667 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:05:33.667 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:05:33.667 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:05:33.668 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:05:33.668 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:05:33.668 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:05:33.668 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:05:33.668 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:05:33.670 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.662+0000 7f46af068640 1 Processor -- start 2024-09-16T04:05:33.670 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.662+0000 7f46af068640 1 -- start start 2024-09-16T04:05:33.670 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.662+0000 7f46af068640 10 monclient: init 2024-09-16T04:05:33.670 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.662+0000 7f46af068640 10 monclient: _reopen_session rank -1 2024-09-16T04:05:33.670 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.662+0000 7f46af068640 10 monclient: _add_conns ranks=[0,2,3,4,1] 2024-09-16T04:05:33.670 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.662+0000 7f46af068640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f46a806db90 addr v1:172.21.15.42:6789/0 2024-09-16T04:05:33.670 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.662+0000 7f46af068640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f46a81340c0 0x7f46a813c190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:05:33.670 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.662+0000 7f46af068640 10 monclient(hunting): picked mon.noname-b con 0x7f46a81340c0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:05:33.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.662+0000 7f46af068640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f46a813c6d0 0x7f46a813eac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:05:33.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.662+0000 7f46af068640 10 monclient(hunting): picked mon.noname-a con 0x7f46a813c6d0 addr v2:172.21.15.42:3300/0 2024-09-16T04:05:33.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.662+0000 7f46af068640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f46a806ece0 con 0x7f46a806db90 2024-09-16T04:05:33.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.662+0000 7f46af068640 10 monclient(hunting): start opening mon connection 2024-09-16T04:05:33.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.662+0000 7f46af068640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f46a806df70 con 0x7f46a813c6d0 2024-09-16T04:05:33.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.662+0000 7f46af068640 10 monclient(hunting): start opening mon connection 2024-09-16T04:05:33.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.662+0000 7f46af068640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f46a809bf80 con 0x7f46a81340c0 2024-09-16T04:05:33.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.662+0000 7f46af068640 10 monclient(hunting): _renew_subs 2024-09-16T04:05:33.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.662+0000 7f46af068640 10 monclient(hunting): authenticate will time out at 2380.634964s 2024-09-16T04:05:33.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.662+0000 7f46a7fff640 1 -- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f46a81340c0 msgr2=0x7f46a813c190 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T04:05:33.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.662+0000 7f46a7fff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f46a81340c0 0x7f46a813c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:05:33.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.662+0000 7f46ad605640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f46a813c6d0 0x7f46a813eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:05:33.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.662+0000 7f46ace04640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f46a806db90 0x7f46a80a37f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:33502/0 (socket says 172.21.15.42:33502) 2024-09-16T04:05:33.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.662+0000 7f46ace04640 1 -- 172.21.15.42:0/1654822772 learned_addr learned my addr 172.21.15.42:0/1654822772 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T04:05:33.672 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.662+0000 7f46ad605640 10 monclient(hunting): get_auth_request con 0x7f46a813c6d0 auth_method 0 2024-09-16T04:05:33.672 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.662+0000 7f46ad605640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:05:33.672 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.662+0000 7f46ad605640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:05:33.672 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.662+0000 7f46ad605640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:05:33.672 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.662+0000 7f46ad605640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:05:33.672 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.662+0000 7f46ad605640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:05:33.672 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.662+0000 7f46ad605640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:05:33.672 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.662+0000 7f46ad605640 10 monclient(hunting): handle_auth_done global_id 984324 payload 274 2024-09-16T04:05:33.672 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.662+0000 7f46ad605640 1 -- 172.21.15.42:0/1654822772 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f46a81340c0 msgr2=0x7f46a813c190 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:05:33.672 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.662+0000 7f46ad605640 1 --2- 172.21.15.42:0/1654822772 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f46a81340c0 0x7f46a813c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:33.672 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.662+0000 7f46ad605640 1 -- 172.21.15.42:0/1654822772 >> v1:172.21.15.42:6789/0 conn(0x7f46a806db90 legacy=0x7f46a80a37f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:05:33.672 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.662+0000 7f46ad605640 10 monclient: _finish_hunting 0 2024-09-16T04:05:33.672 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.662+0000 7f46ad605640 1 monclient: found mon.noname-a 2024-09-16T04:05:33.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.662+0000 7f46a77fe640 1 -- 172.21.15.42:0/1654822772 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1003537433 0 0) 0x7f46a806ece0 con 0x7f46a806db90 2024-09-16T04:05:33.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.662+0000 7f46ad605640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:05:33.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.662+0000 7f46ad605640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:05:33.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.662+0000 7f46ad605640 1 -- 172.21.15.42:0/1654822772 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46a81344d0 con 0x7f46a813c6d0 2024-09-16T04:05:33.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.662+0000 7f46a77fe640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T04:05:33.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.666+0000 7f46ad605640 1 --2- 172.21.15.42:0/1654822772 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f46a813c6d0 0x7f46a813eac0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f46a0009d90 tx=0x7f46a001c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=682a9b8069e567bd server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:05:33.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.666+0000 7f46a77fe640 1 -- 172.21.15.42:0/1654822772 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46a00b2070 con 0x7f46a813c6d0 2024-09-16T04:05:33.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.666+0000 7f46a77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:05:33.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.666+0000 7f46a77fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:05:33.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.666+0000 7f46a77fe640 10 monclient: dump: 2024-09-16T04:05:33.681 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:05:33.681 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:05:33.681 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:33.682 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:33.682 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:05:33.682 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:05:33.682 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:05:33.682 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:05:33.682 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:05:33.682 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:05:33.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.666+0000 7f46a77fe640 10 monclient: _finish_auth 0 2024-09-16T04:05:33.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.666+0000 7f46a77fe640 10 monclient: _check_auth_tickets 2024-09-16T04:05:33.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.666+0000 7f46a77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:05:33.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.666+0000 7f46a77fe640 1 -- 172.21.15.42:0/1654822772 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f46a00a5910 con 0x7f46a813c6d0 2024-09-16T04:05:33.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.666+0000 7f46a77fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:05:33.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.666+0000 7f46a77fe640 1 -- 172.21.15.42:0/1654822772 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46a00a5ce0 con 0x7f46a813c6d0 2024-09-16T04:05:33.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.666+0000 7f46af068640 5 monclient: authenticate success, global_id 984324 2024-09-16T04:05:33.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.666+0000 7f46af068640 10 monclient: get_monmap_and_config success 2024-09-16T04:05:33.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.666+0000 7f46af068640 4 set_mon_vals no callback set 2024-09-16T04:05:33.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.666+0000 7f46a77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:05:33.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.666+0000 7f46a77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:05:33.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.666+0000 7f46a77fe640 10 monclient: dump: 2024-09-16T04:05:33.683 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:05:33.683 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:05:33.683 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:33.684 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:33.684 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:05:33.684 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:05:33.684 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:05:33.684 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:05:33.684 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:05:33.684 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:05:33.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.666+0000 7f46af068640 10 monclient: shutdown 2024-09-16T04:05:33.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.666+0000 7f46af068640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:05:33.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.666+0000 7f46af068640 1 -- 172.21.15.42:0/1654822772 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f46a813c6d0 msgr2=0x7f46a813eac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:05:33.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.666+0000 7f46af068640 1 --2- 172.21.15.42:0/1654822772 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f46a813c6d0 0x7f46a813eac0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f46a0009d90 tx=0x7f46a001c850 comp rx=0 tx=0).stop 2024-09-16T04:05:33.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.666+0000 7f46af068640 1 -- 172.21.15.42:0/1654822772 shutdown_connections 2024-09-16T04:05:33.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.666+0000 7f46af068640 1 --2- 172.21.15.42:0/1654822772 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f46a813c6d0 0x7f46a813eac0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:33.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.666+0000 7f46af068640 1 --2- 172.21.15.42:0/1654822772 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f46a81340c0 0x7f46a813c190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:33.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.666+0000 7f46af068640 1 -- 172.21.15.42:0/1654822772 >> 172.21.15.42:0/1654822772 conn(0x7f46a80a9fb0 msgr2=0x7f46a80aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:05:33.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.666+0000 7f46af068640 1 -- 172.21.15.42:0/1654822772 shutdown_connections 2024-09-16T04:05:33.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.666+0000 7f46af068640 1 -- 172.21.15.42:0/1654822772 wait complete. 2024-09-16T04:05:33.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.666+0000 7f46af068640 10 monclient: build_initial_monmap 2024-09-16T04:05:33.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.666+0000 7f46af068640 10 monclient: monmap: 2024-09-16T04:05:33.685 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:05:33.685 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:05:33.685 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:05:33.685 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:05:33.685 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:05:33.686 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:05:33.686 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:05:33.686 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:05:33.686 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:05:33.686 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:05:33.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.666+0000 7f46af068640 1 librados: starting msgr at 2024-09-16T04:05:33.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.666+0000 7f46af068640 1 librados: starting objecter 2024-09-16T04:05:33.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.670+0000 7f46af068640 1 Processor -- start 2024-09-16T04:05:33.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.670+0000 7f46af068640 1 -- start start 2024-09-16T04:05:33.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.670+0000 7f46af068640 1 librados: setting wanted keys 2024-09-16T04:05:33.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.670+0000 7f46af068640 1 librados: calling monclient init 2024-09-16T04:05:33.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.670+0000 7f46af068640 10 monclient: init 2024-09-16T04:05:33.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.670+0000 7f46af068640 10 monclient: _reopen_session rank -1 2024-09-16T04:05:33.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.670+0000 7f46af068640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:05:33.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.670+0000 7f46af068640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f46a806db90 0x7f46a81308b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:05:33.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.670+0000 7f46af068640 10 monclient(hunting): picked mon.noname-a con 0x7f46a806db90 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:05:33.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.670+0000 7f46af068640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f46a81340c0 0x7f46a8130df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:05:33.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.670+0000 7f46af068640 10 monclient(hunting): picked mon.noname-b con 0x7f46a81340c0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:05:33.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.670+0000 7f46af068640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f46a813c6d0 0x7f46a81395a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:05:33.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.670+0000 7f46af068640 10 monclient(hunting): picked mon.noname-c con 0x7f46a813c6d0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:05:33.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.670+0000 7f46af068640 10 monclient(hunting): start opening mon connection 2024-09-16T04:05:33.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.670+0000 7f46af068640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f46a8141820 con 0x7f46a806db90 2024-09-16T04:05:33.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.670+0000 7f46af068640 10 monclient(hunting): start opening mon connection 2024-09-16T04:05:33.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.670+0000 7f46af068640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f46a81419a0 con 0x7f46a81340c0 2024-09-16T04:05:33.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.670+0000 7f46af068640 10 monclient(hunting): start opening mon connection 2024-09-16T04:05:33.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.670+0000 7f46af068640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f46a8141b20 con 0x7f46a813c6d0 2024-09-16T04:05:33.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.670+0000 7f46af068640 10 monclient(hunting): _renew_subs 2024-09-16T04:05:33.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.670+0000 7f46af068640 10 monclient(hunting): authenticate will time out at 2380.640792s 2024-09-16T04:05:33.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.670+0000 7f46ad605640 1 -- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f46a813c6d0 msgr2=0x7f46a81395a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T04:05:33.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.670+0000 7f46ad605640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f46a813c6d0 0x7f46a81395a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:05:33.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.670+0000 7f46ace04640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f46a806db90 0x7f46a81308b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:05:33.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.670+0000 7f46ace04640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f46a806db90 0x7f46a81308b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:56674/0 (socket says 172.21.15.42:56674) 2024-09-16T04:05:33.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.670+0000 7f46ace04640 1 -- 172.21.15.42:0/599375740 learned_addr learned my addr 172.21.15.42:0/599375740 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:05:33.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.670+0000 7f46ace04640 10 monclient(hunting): get_auth_request con 0x7f46a806db90 auth_method 0 2024-09-16T04:05:33.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.670+0000 7f46ace04640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:05:33.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.670+0000 7f46ace04640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:05:33.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.670+0000 7f46ace04640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:05:33.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.670+0000 7f46ace04640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:05:33.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.670+0000 7f46ace04640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:05:33.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.670+0000 7f46ace04640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:05:33.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.670+0000 7f46a7fff640 1 --2- 172.21.15.42:0/599375740 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f46a81340c0 0x7f46a8130df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:05:33.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.670+0000 7f46a7fff640 10 monclient(hunting): get_auth_request con 0x7f46a81340c0 auth_method 0 2024-09-16T04:05:33.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.670+0000 7f46a7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:05:33.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.670+0000 7f46a7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:05:33.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.670+0000 7f46a7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:05:33.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.670+0000 7f46ace04640 10 monclient(hunting): handle_auth_done global_id 984330 payload 824 2024-09-16T04:05:33.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.670+0000 7f46ace04640 1 -- 172.21.15.42:0/599375740 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f46a813c6d0 msgr2=0x7f46a81395a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:05:33.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.670+0000 7f46ace04640 1 --2- 172.21.15.42:0/599375740 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f46a813c6d0 0x7f46a81395a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:33.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.670+0000 7f46ace04640 1 -- 172.21.15.42:0/599375740 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f46a81340c0 msgr2=0x7f46a8130df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:05:33.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.670+0000 7f46ace04640 1 --2- 172.21.15.42:0/599375740 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f46a81340c0 0x7f46a8130df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:33.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.670+0000 7f46ace04640 10 monclient: _finish_hunting 0 2024-09-16T04:05:33.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.670+0000 7f46ace04640 1 monclient: found mon.noname-a 2024-09-16T04:05:33.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.670+0000 7f46ace04640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:05:33.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.670+0000 7f46ace04640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:05:33.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.670+0000 7f46ace04640 1 -- 172.21.15.42:0/599375740 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46a809dc70 con 0x7f46a806db90 2024-09-16T04:05:33.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.670+0000 7f46ace04640 1 --2- 172.21.15.42:0/599375740 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f46a806db90 0x7f46a81308b0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f469c00c480 tx=0x7f469c00c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:05:33.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.670+0000 7f46a57fa640 10 client.?.objecter ms_handle_connect 0x7f46a806db90 2024-09-16T04:05:33.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.670+0000 7f46a57fa640 10 client.?.objecter resend_mon_ops 2024-09-16T04:05:33.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.670+0000 7f46a57fa640 1 -- 172.21.15.42:0/599375740 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f469c018070 con 0x7f46a806db90 2024-09-16T04:05:33.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.670+0000 7f46a57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:05:33.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.670+0000 7f46a57fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:05:33.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.670+0000 7f46a57fa640 10 monclient: dump: 2024-09-16T04:05:33.690 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:05:33.690 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:05:33.690 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:33.690 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:33.690 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:05:33.691 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:05:33.691 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:05:33.691 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:05:33.691 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:05:33.691 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:05:33.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.670+0000 7f46a57fa640 10 monclient: _finish_auth 0 2024-09-16T04:05:33.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.670+0000 7f46a57fa640 10 monclient: _check_auth_tickets 2024-09-16T04:05:33.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.670+0000 7f46a57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:05:33.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.670+0000 7f46a57fa640 1 -- 172.21.15.42:0/599375740 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f469c007500 con 0x7f46a806db90 2024-09-16T04:05:33.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.670+0000 7f46af068640 5 monclient: authenticate success, global_id 984330 2024-09-16T04:05:33.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.670+0000 7f46af068640 10 monclient: _renew_subs 2024-09-16T04:05:33.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.670+0000 7f46af068640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:05:33.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.670+0000 7f46af068640 1 -- 172.21.15.42:0/599375740 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46a809df50 con 0x7f46a806db90 2024-09-16T04:05:33.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.670+0000 7f46a57fa640 10 monclient: handle_config config(0 keys) 2024-09-16T04:05:33.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.670+0000 7f46a57fa640 1 -- 172.21.15.42:0/599375740 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f469c004a30 con 0x7f46a806db90 2024-09-16T04:05:33.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.670+0000 7f46a57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:05:33.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.670+0000 7f46a57fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:05:33.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.670+0000 7f46a57fa640 10 monclient: dump: 2024-09-16T04:05:33.692 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:05:33.692 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:05:33.692 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:33.692 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:33.692 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:05:33.692 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:05:33.692 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:05:33.692 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:05:33.692 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:05:33.693 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:05:33.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.670+0000 7f46a6ffd640 4 set_mon_vals no callback set 2024-09-16T04:05:33.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.670+0000 7f46af068640 10 client.984330.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:05:33.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.670+0000 7f46af068640 10 monclient: _renew_subs 2024-09-16T04:05:33.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.670+0000 7f46af068640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:05:33.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.670+0000 7f46af068640 1 -- 172.21.15.42:0/599375740 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46a809e230 con 0x7f46a806db90 2024-09-16T04:05:33.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.670+0000 7f46af068640 1 librados: init done 2024-09-16T04:05:33.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.670+0000 7f46a57fa640 1 -- 172.21.15.42:0/599375740 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f469c01b070 con 0x7f46a806db90 2024-09-16T04:05:33.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.670+0000 7f46af068640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:05:33.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.670+0000 7f46af068640 1 --2- 172.21.15.42:0/599375740 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f46a8139ae0 0x7f46a8139ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:05:33.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.670+0000 7f46af068640 10 monclient(hunting): start opening mon connection 2024-09-16T04:05:33.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.670+0000 7f46af068640 1 -- 172.21.15.42:0/599375740 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f46a80a4110 con 0x7f46a8139ae0 2024-09-16T04:05:33.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.670+0000 7f46a7fff640 1 --2- 172.21.15.42:0/599375740 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f46a8139ae0 0x7f46a8139ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:05:33.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.670+0000 7f46a7fff640 10 monclient: get_auth_request con 0x7f46a8139ae0 auth_method 0 2024-09-16T04:05:33.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.670+0000 7f46a7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:05:33.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.670+0000 7f46a7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:05:33.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.670+0000 7f46a7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:05:33.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.670+0000 7f46a57fa640 1 --2- 172.21.15.42:0/599375740 >> v2:172.21.15.154:6812/254845342 conn(0x7f468c0377c0 0x7f468c039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:05:33.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.670+0000 7f46a7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:05:33.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.670+0000 7f46a7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:05:33.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.670+0000 7f46a57fa640 1 -- 172.21.15.42:0/599375740 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(706..706 src has 451..706) ==== 5040+0+0 (secure 0 0 0) 0x7f469c048080 con 0x7f46a806db90 2024-09-16T04:05:33.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.670+0000 7f46a57fa640 10 client.984330.objecter ms_dispatch 0x7f46a8000ba0 osd_map(706..706 src has 451..706) 2024-09-16T04:05:33.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.670+0000 7f46a7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:05:33.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.670+0000 7f46a57fa640 3 client.984330.objecter handle_osd_map got epochs [706,706] > 0 2024-09-16T04:05:33.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.670+0000 7f46a57fa640 3 client.984330.objecter handle_osd_map decoding full epoch 706 2024-09-16T04:05:33.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.670+0000 7f46a57fa640 20 client.984330.objecter dump_active .. 0 homeless 2024-09-16T04:05:33.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.670+0000 7f46a7fff640 10 monclient(hunting): handle_auth_done global_id 984333 payload 824 2024-09-16T04:05:33.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.670+0000 7f46a7fff640 1 -- 172.21.15.42:0/599375740 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f46a809d990 con 0x7f46a8139ae0 2024-09-16T04:05:33.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.670+0000 7f46ad605640 1 --2- 172.21.15.42:0/599375740 >> v2:172.21.15.154:6812/254845342 conn(0x7f468c0377c0 0x7f468c039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:05:33.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.670+0000 7f46a7fff640 1 --2- 172.21.15.42:0/599375740 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f46a8139ae0 0x7f46a8139ec0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f46a813af30 tx=0x7f469400db90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:05:33.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.670+0000 7f46a57fa640 10 client.984330.objecter ms_handle_connect 0x7f46a8139ae0 2024-09-16T04:05:33.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.670+0000 7f46a57fa640 10 client.984330.objecter resend_mon_ops 2024-09-16T04:05:33.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.670+0000 7f46ad605640 10 monclient: get_auth_request con 0x7f468c0377c0 auth_method 0 2024-09-16T04:05:33.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.670+0000 7f46a57fa640 1 -- 172.21.15.42:0/599375740 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4694018070 con 0x7f46a8139ae0 2024-09-16T04:05:33.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.670+0000 7f46a57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:05:33.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.670+0000 7f46a57fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:05:33.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.670+0000 7f46a57fa640 10 monclient: dump: 2024-09-16T04:05:33.695 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:05:33.695 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:05:33.695 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:33.696 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:33.696 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:05:33.696 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:05:33.696 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:05:33.696 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:05:33.696 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:05:33.696 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:05:33.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.670+0000 7f46a57fa640 1 -- 172.21.15.42:0/599375740 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f4694005a50 con 0x7f46a8139ae0 2024-09-16T04:05:33.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.670+0000 7f46a57fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:05:33.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.670+0000 7f46a57fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:05:33.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.670+0000 7f46a57fa640 1 -- 172.21.15.42:0/599375740 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f46a8139ae0 msgr2=0x7f46a8139ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:05:33.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.670+0000 7f46a57fa640 1 --2- 172.21.15.42:0/599375740 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f46a8139ae0 0x7f46a8139ec0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f46a813af30 tx=0x7f469400db90 comp rx=0 tx=0).stop 2024-09-16T04:05:33.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.670+0000 7f46a57fa640 1 -- 172.21.15.42:0/599375740 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f46a8139ae0 msgr2=0x7f46a8139ec0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:05:33.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.670+0000 7f46a57fa640 1 --2- 172.21.15.42:0/599375740 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f46a8139ae0 0x7f46a8139ec0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:33.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.674+0000 7f46ad605640 1 --2- 172.21.15.42:0/599375740 >> v2:172.21.15.154:6812/254845342 conn(0x7f468c0377c0 0x7f468c039c80 secure :-1 s=READY pgs=3852 cs=0 l=1 rev1=1 crypto rx=0x7f46a00af000 tx=0x7f46a00a7070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:05:33.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.674+0000 7f46a57fa640 10 client.984330.objecter ms_handle_connect 0x7f468c0377c0 2024-09-16T04:05:33.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.678+0000 7f46af068640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:05:33.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.678+0000 7f46af068640 1 --2- 172.21.15.42:0/599375740 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f46a813a400 0x7f46a81349e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:05:33.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.678+0000 7f46af068640 10 monclient(hunting): start opening mon connection 2024-09-16T04:05:33.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.678+0000 7f46af068640 1 -- 172.21.15.42:0/599375740 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f46a80a37f0 con 0x7f46a813a400 2024-09-16T04:05:33.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.678+0000 7f46a7fff640 1 --2- 172.21.15.42:0/599375740 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f46a813a400 0x7f46a81349e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:05:33.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.678+0000 7f46a7fff640 10 monclient: get_auth_request con 0x7f46a813a400 auth_method 0 2024-09-16T04:05:33.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.678+0000 7f46a7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:05:33.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.678+0000 7f46a7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:05:33.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.678+0000 7f46a7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:05:33.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.678+0000 7f46a7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:05:33.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.678+0000 7f46a7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:05:33.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.678+0000 7f46a7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:05:33.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.678+0000 7f46a7fff640 10 monclient(hunting): handle_auth_done global_id 984336 payload 824 2024-09-16T04:05:33.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.678+0000 7f46a7fff640 1 -- 172.21.15.42:0/599375740 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f46a809caa0 con 0x7f46a813a400 2024-09-16T04:05:33.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.678+0000 7f46a57fa640 10 client.984330.objecter ms_handle_connect 0x7f46a813a400 2024-09-16T04:05:33.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.678+0000 7f46a57fa640 10 client.984330.objecter resend_mon_ops 2024-09-16T04:05:33.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.678+0000 7f46a7fff640 1 --2- 172.21.15.42:0/599375740 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f46a813a400 0x7f46a81349e0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f4694015000 tx=0x7f46940059e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:05:33.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.678+0000 7f46a57fa640 1 -- 172.21.15.42:0/599375740 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4694018040 con 0x7f46a813a400 2024-09-16T04:05:33.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.678+0000 7f46a57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:05:33.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.678+0000 7f46a57fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:05:33.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.678+0000 7f46a57fa640 10 monclient: dump: 2024-09-16T04:05:33.700 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:05:33.700 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:05:33.700 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:33.700 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:33.700 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:05:33.700 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:05:33.700 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:05:33.700 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:05:33.700 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:05:33.700 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:05:33.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.678+0000 7f46a57fa640 1 -- 172.21.15.42:0/599375740 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4703 (secure 0 0 0) 0x7f4694005110 con 0x7f46a813a400 2024-09-16T04:05:33.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.678+0000 7f46a57fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:05:33.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.678+0000 7f46a57fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:05:33.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.678+0000 7f46a57fa640 1 -- 172.21.15.42:0/599375740 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f46a813a400 msgr2=0x7f46a81349e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:05:33.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.678+0000 7f46a57fa640 1 --2- 172.21.15.42:0/599375740 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f46a813a400 0x7f46a81349e0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f4694015000 tx=0x7f46940059e0 comp rx=0 tx=0).stop 2024-09-16T04:05:33.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.678+0000 7f46a57fa640 1 -- 172.21.15.42:0/599375740 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f46a813a400 msgr2=0x7f46a81349e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:05:33.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.678+0000 7f46a57fa640 1 --2- 172.21.15.42:0/599375740 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f46a813a400 0x7f46a81349e0 secure :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f4694015000 tx=0x7f46940059e0 comp rx=0 tx=0).stop 2024-09-16T04:05:33.701 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T04:05:33.701 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:05:33.701 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:05:33.701 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "leader", 2024-09-16T04:05:33.702 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 220, 2024-09-16T04:05:33.702 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 93735, 2024-09-16T04:05:33.702 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T04:05:33.702 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T04:05:33.702 INFO:teuthology.orchestra.run.smithi042.stdout: 1 2024-09-16T04:05:33.702 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:05:33.702 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 0, 2024-09-16T04:05:33.702 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:05:33.702 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T04:05:33.702 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T04:05:33.702 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:05:33.702 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:05:33.702 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:05:33.703 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:05:33.703 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:05:33.703 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:05:33.703 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:05:33.703 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:05:33.703 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:05:33.703 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:05:33.703 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:05:33.703 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:05:33.703 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:05:33.703 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T04:05:33.703 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:05:33.703 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:05:33.703 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:05:33.704 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:05:33.704 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:05:33.704 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:05:33.704 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:05:33.704 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:05:33.704 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:05:33.704 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:05:33.704 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:05:33.704 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:05:33.704 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:05:33.704 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T04:05:33.704 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T04:05:33.704 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T04:05:33.704 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T04:05:33.705 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T04:05:33.705 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T04:05:33.705 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:05:33.705 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:05:33.705 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T04:05:33.705 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T04:05:33.705 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T04:05:33.705 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T04:05:33.705 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T04:05:33.705 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T04:05:33.705 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T04:05:33.705 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:05:33.705 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T04:05:33.705 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:05:33.706 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:05:33.706 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:05:33.706 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:05:33.706 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:05:33.706 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:05:33.706 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:05:33.706 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:05:33.706 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:05:33.706 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:05:33.706 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:05:33.706 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:05:33.706 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T04:05:33.706 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:05:33.706 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T04:05:33.707 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:05:33.707 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:05:33.707 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:05:33.707 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:05:33.707 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:05:33.707 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:05:33.707 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:05:33.707 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T04:05:33.707 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:05:33.707 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:05:33.707 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:05:33.707 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:05:33.707 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T04:05:33.707 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:05:33.707 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:05:33.708 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:05:33.708 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:05:33.708 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T04:05:33.708 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T04:05:33.708 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:05:33.708 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:05:33.708 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:05:33.708 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:05:33.708 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:05:33.708 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:05:33.708 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:05:33.708 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:05:33.709 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:05:33.709 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:05:33.709 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:05:33.709 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T04:05:33.709 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:05:33.709 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:05:33.709 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:05:33.709 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:05:33.709 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T04:05:33.709 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:05:33.709 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:05:33.710 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:05:33.710 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:05:33.710 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T04:05:33.710 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T04:05:33.710 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:05:33.710 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:05:33.710 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:05:33.710 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:05:33.710 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:05:33.710 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:05:33.710 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:05:33.710 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:05:33.710 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:05:33.710 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:05:33.711 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:05:33.711 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T04:05:33.711 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:05:33.711 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:05:33.711 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:05:33.711 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:05:33.711 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T04:05:33.711 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:05:33.711 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:05:33.711 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:05:33.711 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:05:33.711 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T04:05:33.711 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T04:05:33.711 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:05:33.712 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:05:33.712 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:05:33.712 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:05:33.712 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:05:33.712 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:05:33.712 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T04:05:33.712 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T04:05:33.712 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:05:33.712 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:05:33.712 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:05:33.712 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:05:33.712 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:05:33.712 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:05:33.713 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T04:05:33.713 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:05:33.713 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:05:33.713 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:05:33.713 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 3 2024-09-16T04:05:33.713 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:05:33.713 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:05:33.713 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T04:05:33.713 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:05:33.713 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:05:33.713 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:05:33.713 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 3 2024-09-16T04:05:33.713 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:05:33.713 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:05:33.714 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:05:33.714 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T04:05:33.714 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T04:05:33.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.678+0000 7f46af068640 10 librados: watch_flush enter 2024-09-16T04:05:33.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.678+0000 7f46af068640 10 librados: watch_flush exit 2024-09-16T04:05:33.716 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.678+0000 7f46af068640 20 client.984330.objecter shutdown clearing up homeless session... 2024-09-16T04:05:33.716 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.678+0000 7f46af068640 10 client.984330.objecter successfully canceled tick 2024-09-16T04:05:33.716 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.678+0000 7f46af068640 1 -- 172.21.15.42:0/599375740 >> v2:172.21.15.154:6812/254845342 conn(0x7f468c0377c0 msgr2=0x7f468c039c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:05:33.716 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.678+0000 7f46af068640 1 --2- 172.21.15.42:0/599375740 >> v2:172.21.15.154:6812/254845342 conn(0x7f468c0377c0 0x7f468c039c80 secure :-1 s=READY pgs=3852 cs=0 l=1 rev1=1 crypto rx=0x7f46a00af000 tx=0x7f46a00a7070 comp rx=0 tx=0).stop 2024-09-16T04:05:33.716 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.678+0000 7f46af068640 10 monclient: shutdown 2024-09-16T04:05:33.716 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.678+0000 7f46af068640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:05:33.716 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.678+0000 7f46af068640 1 -- 172.21.15.42:0/599375740 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f46a806db90 msgr2=0x7f46a81308b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:05:33.716 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.678+0000 7f46af068640 1 --2- 172.21.15.42:0/599375740 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f46a806db90 0x7f46a81308b0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f469c00c480 tx=0x7f469c00c940 comp rx=0 tx=0).stop 2024-09-16T04:05:33.716 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.678+0000 7f46ad605640 1 -- 172.21.15.42:0/599375740 reap_dead start 2024-09-16T04:05:33.716 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.678+0000 7f46ad605640 1 -- 172.21.15.42:0/599375740 reap_dead start 2024-09-16T04:05:33.716 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.678+0000 7f46af068640 1 -- 172.21.15.42:0/599375740 shutdown_connections 2024-09-16T04:05:33.716 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.678+0000 7f46af068640 1 -- 172.21.15.42:0/599375740 >> 172.21.15.42:0/599375740 conn(0x7f46a80a9fb0 msgr2=0x7f46a8132970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:05:33.717 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.678+0000 7f46af068640 1 -- 172.21.15.42:0/599375740 shutdown_connections 2024-09-16T04:05:33.717 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.678+0000 7f46af068640 1 -- 172.21.15.42:0/599375740 wait complete. 2024-09-16T04:05:33.717 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.678+0000 7f46af068640 1 librados: shutdown 2024-09-16T04:05:33.717 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T04:05:33.717 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-16T04:05:33.717 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T04:05:33.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.830+0000 7fe2dd27d640 10 monclient: get_monmap_and_config 2024-09-16T04:05:33.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.830+0000 7fe2dd27d640 10 monclient: build_initial_monmap 2024-09-16T04:05:33.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.830+0000 7fe2dd27d640 10 monclient: monmap: 2024-09-16T04:05:33.835 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:05:33.835 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:05:33.835 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:05:33.833414+0000 2024-09-16T04:05:33.835 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:05:33.833414+0000 2024-09-16T04:05:33.835 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:05:33.835 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:05:33.835 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:05:33.835 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:05:33.835 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:05:33.835 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:05:33.836 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:05:33.836 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:05:33.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.830+0000 7fe2dd27d640 1 Processor -- start 2024-09-16T04:05:33.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.830+0000 7fe2dd27d640 1 -- start start 2024-09-16T04:05:33.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.830+0000 7fe2dd27d640 10 monclient: init 2024-09-16T04:05:33.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.830+0000 7fe2dd27d640 10 monclient: _reopen_session rank -1 2024-09-16T04:05:33.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.830+0000 7fe2dd27d640 10 monclient: _add_conns ranks=[0,3,1,2,4] 2024-09-16T04:05:33.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.830+0000 7fe2dd27d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe2d806db90 addr v1:172.21.15.42:6789/0 2024-09-16T04:05:33.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.830+0000 7fe2dd27d640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe2d81340c0 0x7fe2d813c190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:05:33.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.830+0000 7fe2dd27d640 10 monclient(hunting): picked mon.noname-a con 0x7fe2d81340c0 addr v2:172.21.15.42:3300/0 2024-09-16T04:05:33.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.830+0000 7fe2dd27d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe2d813c6d0 addr v1:172.21.15.154:6789/0 2024-09-16T04:05:33.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.830+0000 7fe2dd27d640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe2d806ece0 con 0x7fe2d806db90 2024-09-16T04:05:33.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.830+0000 7fe2dd27d640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe2d809bf80 con 0x7fe2d813c6d0 2024-09-16T04:05:33.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.830+0000 7fe2dd27d640 10 monclient(hunting): start opening mon connection 2024-09-16T04:05:33.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.830+0000 7fe2dd27d640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7fe2d806df70 con 0x7fe2d81340c0 2024-09-16T04:05:33.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.830+0000 7fe2dd27d640 10 monclient(hunting): _renew_subs 2024-09-16T04:05:33.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.830+0000 7fe2dd27d640 10 monclient(hunting): authenticate will time out at 2380.802858s 2024-09-16T04:05:33.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.830+0000 7fe2d67fc640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe2d81340c0 0x7fe2d813c190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:05:33.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.830+0000 7fe2d6ffd640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fe2d806db90 0x7fe2d80a37f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:33518/0 (socket says 172.21.15.42:33518) 2024-09-16T04:05:33.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.830+0000 7fe2d6ffd640 1 -- 172.21.15.42:0/4103632177 learned_addr learned my addr 172.21.15.42:0/4103632177 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T04:05:33.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.830+0000 7fe2d67fc640 10 monclient(hunting): get_auth_request con 0x7fe2d81340c0 auth_method 0 2024-09-16T04:05:33.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.830+0000 7fe2d67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:05:33.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.830+0000 7fe2d67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:05:33.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.830+0000 7fe2d67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:05:33.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.830+0000 7fe2d67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:05:33.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.830+0000 7fe2d67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:05:33.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.830+0000 7fe2d67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:05:33.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.830+0000 7fe2d67fc640 10 monclient(hunting): handle_auth_done global_id 984339 payload 274 2024-09-16T04:05:33.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.830+0000 7fe2d67fc640 1 -- 172.21.15.42:0/4103632177 >> v1:172.21.15.154:6789/0 conn(0x7fe2d813c6d0 legacy=0x7fe2d813eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:05:33.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.830+0000 7fe2d67fc640 1 -- 172.21.15.42:0/4103632177 >> v1:172.21.15.42:6789/0 conn(0x7fe2d806db90 legacy=0x7fe2d80a37f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:05:33.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.830+0000 7fe2d67fc640 10 monclient: _finish_hunting 0 2024-09-16T04:05:33.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.830+0000 7fe2d67fc640 1 monclient: found mon.noname-a 2024-09-16T04:05:33.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.830+0000 7fe2d67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:05:33.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.830+0000 7fe2d67fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:05:33.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.830+0000 7fe2d67fc640 1 -- 172.21.15.42:0/4103632177 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2d81344d0 con 0x7fe2d81340c0 2024-09-16T04:05:33.846 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.834+0000 7fe2d67fc640 1 --2- 172.21.15.42:0/4103632177 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe2d81340c0 0x7fe2d813c190 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fe2cc0097c0 tx=0x7fe2cc01cba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=401b0dcedfe3f18b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:05:33.846 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.834+0000 7fe2d5ffb640 1 -- 172.21.15.42:0/4103632177 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2cc0b0070 con 0x7fe2d81340c0 2024-09-16T04:05:33.846 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.834+0000 7fe2d5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:05:33.846 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.834+0000 7fe2d5ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:05:33.847 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.834+0000 7fe2d5ffb640 10 monclient: dump: 2024-09-16T04:05:33.847 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:05:33.847 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:05:33.847 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:33.847 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:33.847 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:05:33.847 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:05:33.847 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:05:33.847 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:05:33.847 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:05:33.847 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:05:33.847 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.834+0000 7fe2d5ffb640 10 monclient: _finish_auth 0 2024-09-16T04:05:33.847 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.834+0000 7fe2d5ffb640 10 monclient: _check_auth_tickets 2024-09-16T04:05:33.848 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.834+0000 7fe2d5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:05:33.848 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.834+0000 7fe2d5ffb640 1 -- 172.21.15.42:0/4103632177 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe2cc0ab600 con 0x7fe2d81340c0 2024-09-16T04:05:33.848 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.834+0000 7fe2d5ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:05:33.848 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.834+0000 7fe2d5ffb640 1 -- 172.21.15.42:0/4103632177 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2cc0ab9d0 con 0x7fe2d81340c0 2024-09-16T04:05:33.848 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.834+0000 7fe2dd27d640 5 monclient: authenticate success, global_id 984339 2024-09-16T04:05:33.848 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.834+0000 7fe2dd27d640 10 monclient: get_monmap_and_config success 2024-09-16T04:05:33.848 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.834+0000 7fe2dd27d640 4 set_mon_vals no callback set 2024-09-16T04:05:33.848 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.834+0000 7fe2d5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:05:33.848 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.834+0000 7fe2d5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:05:33.848 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.834+0000 7fe2d5ffb640 10 monclient: dump: 2024-09-16T04:05:33.848 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:05:33.848 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:05:33.848 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:33.849 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:33.849 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:05:33.849 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:05:33.849 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:05:33.849 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:05:33.849 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:05:33.849 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:05:33.849 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.834+0000 7fe2dd27d640 10 monclient: shutdown 2024-09-16T04:05:33.849 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.834+0000 7fe2dd27d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:05:33.849 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.834+0000 7fe2dd27d640 1 -- 172.21.15.42:0/4103632177 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe2d81340c0 msgr2=0x7fe2d813c190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:05:33.849 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.834+0000 7fe2dd27d640 1 --2- 172.21.15.42:0/4103632177 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe2d81340c0 0x7fe2d813c190 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fe2cc0097c0 tx=0x7fe2cc01cba0 comp rx=0 tx=0).stop 2024-09-16T04:05:33.849 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.834+0000 7fe2dd27d640 1 -- 172.21.15.42:0/4103632177 shutdown_connections 2024-09-16T04:05:33.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.834+0000 7fe2dd27d640 1 --2- 172.21.15.42:0/4103632177 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe2d81340c0 0x7fe2d813c190 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:33.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.834+0000 7fe2dd27d640 1 -- 172.21.15.42:0/4103632177 >> 172.21.15.42:0/4103632177 conn(0x7fe2d80a9fb0 msgr2=0x7fe2d80aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:05:33.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.834+0000 7fe2dd27d640 1 -- 172.21.15.42:0/4103632177 shutdown_connections 2024-09-16T04:05:33.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.834+0000 7fe2dd27d640 1 -- 172.21.15.42:0/4103632177 wait complete. 2024-09-16T04:05:33.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.834+0000 7fe2dd27d640 10 monclient: build_initial_monmap 2024-09-16T04:05:33.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.834+0000 7fe2dd27d640 10 monclient: monmap: 2024-09-16T04:05:33.850 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:05:33.850 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:05:33.850 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:05:33.850 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:05:33.850 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:05:33.850 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:05:33.850 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:05:33.851 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:05:33.851 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:05:33.851 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:05:33.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.834+0000 7fe2dd27d640 1 librados: starting msgr at 2024-09-16T04:05:33.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.834+0000 7fe2dd27d640 1 librados: starting objecter 2024-09-16T04:05:33.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.834+0000 7fe2dd27d640 1 Processor -- start 2024-09-16T04:05:33.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.834+0000 7fe2dd27d640 1 -- start start 2024-09-16T04:05:33.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.834+0000 7fe2dd27d640 1 librados: setting wanted keys 2024-09-16T04:05:33.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.834+0000 7fe2dd27d640 1 librados: calling monclient init 2024-09-16T04:05:33.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.834+0000 7fe2dd27d640 10 monclient: init 2024-09-16T04:05:33.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.834+0000 7fe2dd27d640 10 monclient: _reopen_session rank -1 2024-09-16T04:05:33.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.834+0000 7fe2dd27d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:05:33.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.834+0000 7fe2dd27d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe2d806db90 0x7fe2d81cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:05:33.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.834+0000 7fe2dd27d640 10 monclient(hunting): picked mon.noname-b con 0x7fe2d806db90 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:05:33.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.834+0000 7fe2dd27d640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe2d81340c0 0x7fe2d813bc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:05:33.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.834+0000 7fe2dd27d640 10 monclient(hunting): picked mon.noname-a con 0x7fe2d81340c0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:05:33.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.834+0000 7fe2dd27d640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe2d813c6d0 0x7fe2d81cbdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:05:33.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.838+0000 7fe2dd27d640 10 monclient(hunting): picked mon.noname-c con 0x7fe2d813c6d0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:05:33.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.838+0000 7fe2dd27d640 10 monclient(hunting): start opening mon connection 2024-09-16T04:05:33.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.838+0000 7fe2dd27d640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fe2d8142b80 con 0x7fe2d81340c0 2024-09-16T04:05:33.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.838+0000 7fe2dd27d640 10 monclient(hunting): start opening mon connection 2024-09-16T04:05:33.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.838+0000 7fe2dd27d640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe2d8142a00 con 0x7fe2d806db90 2024-09-16T04:05:33.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.838+0000 7fe2dd27d640 10 monclient(hunting): start opening mon connection 2024-09-16T04:05:33.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.838+0000 7fe2dd27d640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fe2d8142d00 con 0x7fe2d813c6d0 2024-09-16T04:05:33.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.838+0000 7fe2dd27d640 10 monclient(hunting): _renew_subs 2024-09-16T04:05:33.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.838+0000 7fe2dd27d640 10 monclient(hunting): authenticate will time out at 2380.808039s 2024-09-16T04:05:33.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.838+0000 7fe2d77fe640 1 -- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe2d813c6d0 msgr2=0x7fe2d81cbdf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T04:05:33.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.838+0000 7fe2d77fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe2d813c6d0 0x7fe2d81cbdf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:05:33.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.838+0000 7fe2d67fc640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe2d81340c0 0x7fe2d813bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:05:33.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.838+0000 7fe2d67fc640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe2d81340c0 0x7fe2d813bc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:56710/0 (socket says 172.21.15.42:56710) 2024-09-16T04:05:33.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.838+0000 7fe2d67fc640 1 -- 172.21.15.42:0/4209781084 learned_addr learned my addr 172.21.15.42:0/4209781084 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:05:33.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.838+0000 7fe2d67fc640 10 monclient(hunting): get_auth_request con 0x7fe2d81340c0 auth_method 0 2024-09-16T04:05:33.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.838+0000 7fe2d67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:05:33.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.838+0000 7fe2d67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:05:33.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.838+0000 7fe2d67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:05:33.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.838+0000 7fe2d67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:05:33.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.838+0000 7fe2d67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:05:33.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.838+0000 7fe2d67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:05:33.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.838+0000 7fe2d6ffd640 1 --2- 172.21.15.42:0/4209781084 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe2d806db90 0x7fe2d81cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:05:33.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.838+0000 7fe2d67fc640 10 monclient(hunting): handle_auth_done global_id 984345 payload 824 2024-09-16T04:05:33.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.838+0000 7fe2d67fc640 1 -- 172.21.15.42:0/4209781084 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe2d813c6d0 msgr2=0x7fe2d81cbdf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:05:33.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.838+0000 7fe2d67fc640 1 --2- 172.21.15.42:0/4209781084 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe2d813c6d0 0x7fe2d81cbdf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:33.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.838+0000 7fe2d67fc640 1 -- 172.21.15.42:0/4209781084 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe2d806db90 msgr2=0x7fe2d81cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:05:33.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.838+0000 7fe2d67fc640 1 --2- 172.21.15.42:0/4209781084 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe2d806db90 0x7fe2d81cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:33.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.838+0000 7fe2d67fc640 10 monclient: _finish_hunting 0 2024-09-16T04:05:33.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.838+0000 7fe2d67fc640 1 monclient: found mon.noname-a 2024-09-16T04:05:33.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.838+0000 7fe2d67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:05:33.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.838+0000 7fe2d67fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:05:33.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.838+0000 7fe2d67fc640 1 -- 172.21.15.42:0/4209781084 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2d8136150 con 0x7fe2d81340c0 2024-09-16T04:05:33.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.838+0000 7fe2d6ffd640 10 monclient: get_auth_request con 0x7fe2d806db90 auth_method 0 2024-09-16T04:05:33.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.838+0000 7fe2d6ffd640 1 --2- 172.21.15.42:0/4209781084 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe2d806db90 0x7fe2d81cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T04:05:33.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.838+0000 7fe2d67fc640 1 --2- 172.21.15.42:0/4209781084 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe2d81340c0 0x7fe2d813bc80 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fe2cc018d50 tx=0x7fe2cc01a980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:05:33.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.838+0000 7fe2bf7fe640 10 client.?.objecter ms_handle_connect 0x7fe2d81340c0 2024-09-16T04:05:33.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.838+0000 7fe2bf7fe640 10 client.?.objecter resend_mon_ops 2024-09-16T04:05:33.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.838+0000 7fe2bf7fe640 1 -- 172.21.15.42:0/4209781084 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2cc0b0050 con 0x7fe2d81340c0 2024-09-16T04:05:33.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.838+0000 7fe2bf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:05:33.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.838+0000 7fe2bf7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:05:33.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.838+0000 7fe2bf7fe640 10 monclient: dump: 2024-09-16T04:05:33.855 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:05:33.855 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:05:33.855 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:33.855 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:33.855 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:05:33.855 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:05:33.855 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:05:33.855 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:05:33.855 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:05:33.855 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:05:33.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.838+0000 7fe2bf7fe640 10 monclient: _finish_auth 0 2024-09-16T04:05:33.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.838+0000 7fe2bf7fe640 10 monclient: _check_auth_tickets 2024-09-16T04:05:33.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.838+0000 7fe2bf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:05:33.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.838+0000 7fe2bf7fe640 1 -- 172.21.15.42:0/4209781084 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe2cc0bc070 con 0x7fe2d81340c0 2024-09-16T04:05:33.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.838+0000 7fe2dd27d640 5 monclient: authenticate success, global_id 984345 2024-09-16T04:05:33.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.838+0000 7fe2dd27d640 10 monclient: _renew_subs 2024-09-16T04:05:33.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.838+0000 7fe2dd27d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:05:33.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.838+0000 7fe2dd27d640 1 -- 172.21.15.42:0/4209781084 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2d8135180 con 0x7fe2d81340c0 2024-09-16T04:05:33.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.838+0000 7fe2bf7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:05:33.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.838+0000 7fe2bf7fe640 1 -- 172.21.15.42:0/4209781084 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2cc0b5a70 con 0x7fe2d81340c0 2024-09-16T04:05:33.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.838+0000 7fe2bf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:05:33.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.838+0000 7fe2bf7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:05:33.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.838+0000 7fe2bf7fe640 10 monclient: dump: 2024-09-16T04:05:33.856 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:05:33.857 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:05:33.857 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:33.857 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:33.857 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:05:33.857 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:05:33.857 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:05:33.857 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:05:33.857 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:05:33.857 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:05:33.857 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.838+0000 7fe2d57fa640 4 set_mon_vals no callback set 2024-09-16T04:05:33.857 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.838+0000 7fe2dd27d640 10 client.984345.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:05:33.857 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.838+0000 7fe2dd27d640 10 monclient: _renew_subs 2024-09-16T04:05:33.857 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.838+0000 7fe2dd27d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:05:33.858 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.838+0000 7fe2dd27d640 1 -- 172.21.15.42:0/4209781084 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2d806f6a0 con 0x7fe2d81340c0 2024-09-16T04:05:33.858 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.838+0000 7fe2dd27d640 1 librados: init done 2024-09-16T04:05:33.858 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.838+0000 7fe2dd27d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:05:33.858 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.838+0000 7fe2dd27d640 1 --2- 172.21.15.42:0/4209781084 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe2d813c1c0 0x7fe2d802c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:05:33.858 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.838+0000 7fe2dd27d640 10 monclient(hunting): start opening mon connection 2024-09-16T04:05:33.858 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.838+0000 7fe2dd27d640 1 -- 172.21.15.42:0/4209781084 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe2d8135e70 con 0x7fe2d813c1c0 2024-09-16T04:05:33.858 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.838+0000 7fe2bf7fe640 1 -- 172.21.15.42:0/4209781084 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fe2cc0ab480 con 0x7fe2d81340c0 2024-09-16T04:05:33.858 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.838+0000 7fe2bf7fe640 1 --2- 172.21.15.42:0/4209781084 >> v2:172.21.15.154:6812/254845342 conn(0x7fe2c40377c0 0x7fe2c4039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:05:33.858 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.838+0000 7fe2bf7fe640 1 -- 172.21.15.42:0/4209781084 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(706..706 src has 451..706) ==== 5040+0+0 (secure 0 0 0) 0x7fe2cc0e36c0 con 0x7fe2d81340c0 2024-09-16T04:05:33.858 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.838+0000 7fe2bf7fe640 10 client.984345.objecter ms_dispatch 0x7fe2d8000ba0 osd_map(706..706 src has 451..706) 2024-09-16T04:05:33.858 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.838+0000 7fe2bf7fe640 3 client.984345.objecter handle_osd_map got epochs [706,706] > 0 2024-09-16T04:05:33.858 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.838+0000 7fe2bf7fe640 3 client.984345.objecter handle_osd_map decoding full epoch 706 2024-09-16T04:05:33.858 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.838+0000 7fe2d6ffd640 1 --2- 172.21.15.42:0/4209781084 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe2d813c1c0 0x7fe2d802c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:05:33.858 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.838+0000 7fe2bf7fe640 20 client.984345.objecter dump_active .. 0 homeless 2024-09-16T04:05:33.859 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.838+0000 7fe2d6ffd640 10 monclient: get_auth_request con 0x7fe2d813c1c0 auth_method 0 2024-09-16T04:05:33.859 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.838+0000 7fe2d6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:05:33.859 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.838+0000 7fe2d6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:05:33.859 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.838+0000 7fe2d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:05:33.859 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.838+0000 7fe2d6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:05:33.859 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.838+0000 7fe2d6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:05:33.859 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.838+0000 7fe2d6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:05:33.859 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.838+0000 7fe2d77fe640 1 --2- 172.21.15.42:0/4209781084 >> v2:172.21.15.154:6812/254845342 conn(0x7fe2c40377c0 0x7fe2c4039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:05:33.859 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.838+0000 7fe2d77fe640 10 monclient: get_auth_request con 0x7fe2c40377c0 auth_method 0 2024-09-16T04:05:33.859 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.838+0000 7fe2d6ffd640 10 monclient(hunting): handle_auth_done global_id 904948 payload 824 2024-09-16T04:05:33.859 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.838+0000 7fe2d6ffd640 1 -- 172.21.15.42:0/4209781084 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe2d80a37f0 con 0x7fe2d813c1c0 2024-09-16T04:05:33.859 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.838+0000 7fe2bf7fe640 10 client.984345.objecter ms_handle_connect 0x7fe2c40377c0 2024-09-16T04:05:33.859 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.838+0000 7fe2d77fe640 1 --2- 172.21.15.42:0/4209781084 >> v2:172.21.15.154:6812/254845342 conn(0x7fe2c40377c0 0x7fe2c4039c80 secure :-1 s=READY pgs=3853 cs=0 l=1 rev1=1 crypto rx=0x7fe2d0003b00 tx=0x7fe2d0007d20 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:05:33.859 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.838+0000 7fe2bf7fe640 10 client.984345.objecter ms_handle_connect 0x7fe2d813c1c0 2024-09-16T04:05:33.860 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.838+0000 7fe2bf7fe640 10 client.984345.objecter resend_mon_ops 2024-09-16T04:05:33.860 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.838+0000 7fe2d6ffd640 1 --2- 172.21.15.42:0/4209781084 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe2d813c1c0 0x7fe2d802c650 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fe2c8000f70 tx=0x7fe2c800d880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:05:33.860 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.838+0000 7fe2bf7fe640 1 -- 172.21.15.42:0/4209781084 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2c8005cf0 con 0x7fe2d813c1c0 2024-09-16T04:05:33.860 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.838+0000 7fe2bf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:05:33.860 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.838+0000 7fe2bf7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:05:33.860 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.838+0000 7fe2bf7fe640 10 monclient: dump: 2024-09-16T04:05:33.860 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:05:33.860 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:05:33.860 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:33.860 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:33.860 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:05:33.860 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:05:33.860 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:05:33.861 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:05:33.861 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:05:33.861 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:05:33.861 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.842+0000 7fe2bf7fe640 1 -- 172.21.15.42:0/4209781084 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fe2c8005cf0 con 0x7fe2d813c1c0 2024-09-16T04:05:33.861 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.842+0000 7fe2bf7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:05:33.861 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.842+0000 7fe2bf7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:05:33.861 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.842+0000 7fe2bf7fe640 1 -- 172.21.15.42:0/4209781084 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe2d813c1c0 msgr2=0x7fe2d802c650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:05:33.861 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.842+0000 7fe2bf7fe640 1 --2- 172.21.15.42:0/4209781084 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe2d813c1c0 0x7fe2d802c650 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fe2c8000f70 tx=0x7fe2c800d880 comp rx=0 tx=0).stop 2024-09-16T04:05:33.861 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.842+0000 7fe2bf7fe640 1 -- 172.21.15.42:0/4209781084 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe2d813c1c0 msgr2=0x7fe2d802c650 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:05:33.861 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.842+0000 7fe2bf7fe640 1 --2- 172.21.15.42:0/4209781084 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe2d813c1c0 0x7fe2d802c650 secure :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fe2c8000f70 tx=0x7fe2c800d880 comp rx=0 tx=0).stop 2024-09-16T04:05:33.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.846+0000 7fe2dd27d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:05:33.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.846+0000 7fe2dd27d640 1 --2- 172.21.15.42:0/4209781084 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe2d802cb90 0x7fe2d802cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:05:33.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.846+0000 7fe2dd27d640 10 monclient(hunting): start opening mon connection 2024-09-16T04:05:33.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.846+0000 7fe2dd27d640 1 -- 172.21.15.42:0/4209781084 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe2d81cfd50 con 0x7fe2d802cb90 2024-09-16T04:05:33.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.846+0000 7fe2d6ffd640 1 --2- 172.21.15.42:0/4209781084 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe2d802cb90 0x7fe2d802cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:05:33.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.846+0000 7fe2d6ffd640 10 monclient: get_auth_request con 0x7fe2d802cb90 auth_method 0 2024-09-16T04:05:33.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.846+0000 7fe2d6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:05:33.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.846+0000 7fe2d6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:05:33.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.846+0000 7fe2d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:05:33.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.846+0000 7fe2d6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:05:33.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.846+0000 7fe2d6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:05:33.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.846+0000 7fe2d6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:05:33.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.846+0000 7fe2d6ffd640 10 monclient(hunting): handle_auth_done global_id 904951 payload 824 2024-09-16T04:05:33.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.846+0000 7fe2d6ffd640 1 -- 172.21.15.42:0/4209781084 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fe2d809de50 con 0x7fe2d802cb90 2024-09-16T04:05:33.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.846+0000 7fe2d6ffd640 1 --2- 172.21.15.42:0/4209781084 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe2d802cb90 0x7fe2d802cfa0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fe2c8000f70 tx=0x7fe2c8000ee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:05:33.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.846+0000 7fe2bf7fe640 10 client.984345.objecter ms_handle_connect 0x7fe2d802cb90 2024-09-16T04:05:33.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.846+0000 7fe2bf7fe640 10 client.984345.objecter resend_mon_ops 2024-09-16T04:05:33.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.846+0000 7fe2bf7fe640 1 -- 172.21.15.42:0/4209781084 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2c8005580 con 0x7fe2d802cb90 2024-09-16T04:05:33.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.846+0000 7fe2bf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:05:33.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.846+0000 7fe2bf7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:05:33.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.846+0000 7fe2bf7fe640 10 monclient: dump: 2024-09-16T04:05:33.864 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:05:33.864 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:05:33.865 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:33.865 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:33.865 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:05:33.865 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:05:33.865 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:05:33.865 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:05:33.865 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:05:33.865 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:05:33.865 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.850+0000 7fe2bf7fe640 1 -- 172.21.15.42:0/4209781084 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4869 (secure 0 0 0) 0x7fe2c8005580 con 0x7fe2d802cb90 2024-09-16T04:05:33.865 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.850+0000 7fe2bf7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:05:33.865 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.850+0000 7fe2bf7fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:05:33.865 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.850+0000 7fe2bf7fe640 1 -- 172.21.15.42:0/4209781084 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe2d802cb90 msgr2=0x7fe2d802cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:05:33.865 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.850+0000 7fe2bf7fe640 1 --2- 172.21.15.42:0/4209781084 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe2d802cb90 0x7fe2d802cfa0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fe2c8000f70 tx=0x7fe2c8000ee0 comp rx=0 tx=0).stop 2024-09-16T04:05:33.865 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.850+0000 7fe2bf7fe640 1 -- 172.21.15.42:0/4209781084 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe2d802cb90 msgr2=0x7fe2d802cfa0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:05:33.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.850+0000 7fe2bf7fe640 1 --2- 172.21.15.42:0/4209781084 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe2d802cb90 0x7fe2d802cfa0 secure :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fe2c8000f70 tx=0x7fe2c8000ee0 comp rx=0 tx=0).stop 2024-09-16T04:05:33.866 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T04:05:33.866 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:05:33.866 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:05:33.866 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T04:05:33.866 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 220, 2024-09-16T04:05:33.866 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 462028, 2024-09-16T04:05:33.866 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T04:05:33.866 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T04:05:33.866 INFO:teuthology.orchestra.run.smithi042.stdout: 1 2024-09-16T04:05:33.867 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:05:33.867 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 0, 2024-09-16T04:05:33.867 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:05:33.867 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T04:05:33.867 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T04:05:33.867 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:05:33.867 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:05:33.867 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:05:33.867 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:05:33.867 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:05:33.867 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:05:33.867 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:05:33.867 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:05:33.868 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:05:33.868 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:05:33.868 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:05:33.868 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:05:33.868 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:05:33.868 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T04:05:33.868 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:05:33.868 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:05:33.868 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:05:33.868 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:05:33.868 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:05:33.868 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:05:33.868 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:05:33.868 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:05:33.869 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:05:33.869 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:05:33.869 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:05:33.869 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:05:33.869 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:05:33.869 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T04:05:33.869 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T04:05:33.869 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T04:05:33.869 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T04:05:33.869 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T04:05:33.869 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T04:05:33.869 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:05:33.869 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:05:33.869 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T04:05:33.870 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T04:05:33.870 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T04:05:33.870 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T04:05:33.870 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T04:05:33.870 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T04:05:33.870 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T04:05:33.870 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:05:33.870 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T04:05:33.870 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:05:33.870 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:05:33.870 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:05:33.870 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:05:33.870 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:05:33.870 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:05:33.871 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:05:33.871 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:05:33.871 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:05:33.871 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:05:33.871 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:05:33.871 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:05:33.871 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T04:05:33.871 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:05:33.871 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T04:05:33.871 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:05:33.871 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:05:33.871 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:05:33.871 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:05:33.871 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:05:33.872 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:05:33.872 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:05:33.872 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T04:05:33.872 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:05:33.872 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:05:33.872 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:05:33.872 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:05:33.872 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T04:05:33.872 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:05:33.872 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:05:33.872 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:05:33.872 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:05:33.872 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T04:05:33.872 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T04:05:33.873 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:05:33.873 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:05:33.873 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:05:33.873 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:05:33.873 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:05:33.873 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:05:33.873 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:05:33.873 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:05:33.873 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:05:33.873 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:05:33.873 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:05:33.873 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T04:05:33.873 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:05:33.874 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:05:33.874 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:05:33.874 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:05:33.874 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T04:05:33.874 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:05:33.874 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:05:33.874 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:05:33.874 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:05:33.874 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T04:05:33.874 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T04:05:33.874 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:05:33.874 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:05:33.874 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:05:33.874 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:05:33.875 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:05:33.875 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:05:33.875 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:05:33.875 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:05:33.875 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:05:33.875 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:05:33.875 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:05:33.875 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T04:05:33.875 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:05:33.875 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:05:33.875 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:05:33.875 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:05:33.875 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T04:05:33.875 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:05:33.876 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:05:33.876 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:05:33.876 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:05:33.876 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T04:05:33.876 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T04:05:33.876 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:05:33.876 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:05:33.876 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:05:33.876 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:05:33.876 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:05:33.876 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:05:33.876 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T04:05:33.876 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T04:05:33.877 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:05:33.877 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:05:33.877 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:05:33.877 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:05:33.877 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:05:33.877 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:05:33.877 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T04:05:33.877 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:05:33.877 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:05:33.877 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:05:33.877 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 3 2024-09-16T04:05:33.877 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:05:33.877 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:05:33.878 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T04:05:33.878 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:05:33.878 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:05:33.878 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:05:33.878 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 5 2024-09-16T04:05:33.878 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:05:33.878 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:05:33.878 INFO:teuthology.orchestra.run.smithi042.stdout: "mgr": [ 2024-09-16T04:05:33.878 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:05:33.878 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:05:33.878 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:05:33.878 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:05:33.878 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:05:33.878 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:05:33.879 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:05:33.879 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T04:05:33.879 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T04:05:33.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.850+0000 7fe2dd27d640 10 librados: watch_flush enter 2024-09-16T04:05:33.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.850+0000 7fe2dd27d640 10 librados: watch_flush exit 2024-09-16T04:05:33.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.850+0000 7fe2dd27d640 20 client.984345.objecter shutdown clearing up homeless session... 2024-09-16T04:05:33.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.850+0000 7fe2dd27d640 10 client.984345.objecter successfully canceled tick 2024-09-16T04:05:33.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.850+0000 7fe2dd27d640 1 -- 172.21.15.42:0/4209781084 >> v2:172.21.15.154:6812/254845342 conn(0x7fe2c40377c0 msgr2=0x7fe2c4039c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:05:33.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.850+0000 7fe2dd27d640 1 --2- 172.21.15.42:0/4209781084 >> v2:172.21.15.154:6812/254845342 conn(0x7fe2c40377c0 0x7fe2c4039c80 secure :-1 s=READY pgs=3853 cs=0 l=1 rev1=1 crypto rx=0x7fe2d0003b00 tx=0x7fe2d0007d20 comp rx=0 tx=0).stop 2024-09-16T04:05:33.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.850+0000 7fe2dd27d640 10 monclient: shutdown 2024-09-16T04:05:33.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.850+0000 7fe2dd27d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:05:33.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.850+0000 7fe2dd27d640 1 -- 172.21.15.42:0/4209781084 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe2d81340c0 msgr2=0x7fe2d813bc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:05:33.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.850+0000 7fe2dd27d640 1 --2- 172.21.15.42:0/4209781084 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe2d81340c0 0x7fe2d813bc80 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fe2cc018d50 tx=0x7fe2cc01a980 comp rx=0 tx=0).stop 2024-09-16T04:05:33.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.850+0000 7fe2d77fe640 1 -- 172.21.15.42:0/4209781084 reap_dead start 2024-09-16T04:05:33.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.850+0000 7fe2d77fe640 1 -- 172.21.15.42:0/4209781084 reap_dead start 2024-09-16T04:05:33.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.850+0000 7fe2dd27d640 1 -- 172.21.15.42:0/4209781084 shutdown_connections 2024-09-16T04:05:33.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.850+0000 7fe2dd27d640 1 -- 172.21.15.42:0/4209781084 >> 172.21.15.42:0/4209781084 conn(0x7fe2d80a9fb0 msgr2=0x7fe2d80a9310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:05:33.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.850+0000 7fe2dd27d640 1 -- 172.21.15.42:0/4209781084 shutdown_connections 2024-09-16T04:05:33.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.850+0000 7fe2dd27d640 1 -- 172.21.15.42:0/4209781084 wait complete. 2024-09-16T04:05:33.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:33.850+0000 7fe2dd27d640 1 librados: shutdown 2024-09-16T04:05:33.882 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T04:05:33.882 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-16T04:05:53.884 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-16T04:05:53.884 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-09-16T04:05:53.884 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-16T04:05:53.884 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-16T04:05:53.887 INFO:tasks.ceph.mon.c:Started 2024-09-16T04:05:53.888 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T04:05:53.888 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:05:54.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.001+0000 7f8c5151d640 10 monclient: get_monmap_and_config 2024-09-16T04:05:54.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.001+0000 7f8c5151d640 10 monclient: build_initial_monmap 2024-09-16T04:05:54.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.001+0000 7f8c5151d640 10 monclient: monmap: 2024-09-16T04:05:54.008 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:05:54.008 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:05:54.009 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:05:54.006759+0000 2024-09-16T04:05:54.009 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:05:54.006759+0000 2024-09-16T04:05:54.009 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:05:54.009 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:05:54.009 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:05:54.009 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:05:54.009 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:05:54.009 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:05:54.009 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:05:54.009 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:05:54.023 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.001+0000 7f8c5151d640 1 Processor -- start 2024-09-16T04:05:54.023 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.001+0000 7f8c5151d640 1 -- start start 2024-09-16T04:05:54.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.001+0000 7f8c5151d640 10 monclient: init 2024-09-16T04:05:54.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.001+0000 7f8c5151d640 10 monclient: _reopen_session rank -1 2024-09-16T04:05:54.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.001+0000 7f8c5151d640 10 monclient: _add_conns ranks=[1,2,0,4,3] 2024-09-16T04:05:54.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.001+0000 7f8c5151d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8c4c0a39b0 addr v1:172.21.15.154:6789/0 2024-09-16T04:05:54.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.001+0000 7f8c5151d640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f8c4c0a3d90 0x7f8c4c13c380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:05:54.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.001+0000 7f8c5151d640 10 monclient(hunting): picked mon.noname-b con 0x7f8c4c0a3d90 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:05:54.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.001+0000 7f8c5151d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8c4c13c8c0 addr v1:172.21.15.42:6789/0 2024-09-16T04:05:54.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.001+0000 7f8c5151d640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8c4c134290 con 0x7f8c4c13c8c0 2024-09-16T04:05:54.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.001+0000 7f8c5151d640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8c4c09c140 con 0x7f8c4c0a39b0 2024-09-16T04:05:54.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.001+0000 7f8c5151d640 10 monclient(hunting): start opening mon connection 2024-09-16T04:05:54.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.001+0000 7f8c5151d640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f8c4c09c420 con 0x7f8c4c0a3d90 2024-09-16T04:05:54.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.001+0000 7f8c5151d640 10 monclient(hunting): _renew_subs 2024-09-16T04:05:54.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.001+0000 7f8c5151d640 10 monclient(hunting): authenticate will time out at 2400.975818s 2024-09-16T04:05:54.025 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.001+0000 7f8c4b7fe640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f8c4c13c8c0 0x7f8c4c13ecb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:54468/0 (socket says 172.21.15.42:54468) 2024-09-16T04:05:54.025 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.001+0000 7f8c4b7fe640 1 -- 172.21.15.42:0/1457015711 learned_addr learned my addr 172.21.15.42:0/1457015711 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T04:05:54.025 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.001+0000 7f8c4a7fc640 1 --2- 172.21.15.42:0/1457015711 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f8c4c0a3d90 0x7f8c4c13c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:05:54.025 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.005+0000 7f8c4a7fc640 10 monclient(hunting): get_auth_request con 0x7f8c4c0a3d90 auth_method 0 2024-09-16T04:05:54.025 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.005+0000 7f8c4a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:05:54.025 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.005+0000 7f8c4a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:05:54.025 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.005+0000 7f8c4a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:05:54.025 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.005+0000 7f8c4a7fc640 1 -- 172.21.15.42:0/1457015711 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f8c4c0a3d90 msgr2=0x7f8c4c13c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:05:54.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.005+0000 7f8c4a7fc640 1 -- 172.21.15.42:0/1457015711 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f8c4c0a3d90 msgr2=0x7f8c4c13c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:05:54.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.005+0000 7f8c4a7fc640 1 --2- 172.21.15.42:0/1457015711 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f8c4c0a3d90 0x7f8c4c13c380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:05:54.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.005+0000 7f8c4a7fc640 1 --2- 172.21.15.42:0/1457015711 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f8c4c0a3d90 0x7f8c4c13c380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:05:54.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.005+0000 7f8c49ffb640 1 -- 172.21.15.42:0/1457015711 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3818872783 0 0) 0x7f8c4c134290 con 0x7f8c4c13c8c0 2024-09-16T04:05:54.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.005+0000 7f8c49ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:05:54.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.005+0000 7f8c49ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:05:54.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.005+0000 7f8c49ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:05:54.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.005+0000 7f8c49ffb640 10 monclient(hunting): my global_id is 984348 2024-09-16T04:05:54.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.005+0000 7f8c49ffb640 1 -- 172.21.15.42:0/1457015711 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8c34003530 con 0x7f8c4c13c8c0 2024-09-16T04:05:54.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.005+0000 7f8c49ffb640 1 -- 172.21.15.42:0/1457015711 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1245715396 0 0) 0x7f8c34003530 con 0x7f8c4c13c8c0 2024-09-16T04:05:54.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.005+0000 7f8c49ffb640 1 -- 172.21.15.42:0/1457015711 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f8c4c0a3d90 msgr2=0x7f8c4c13c380 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:05:54.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.005+0000 7f8c49ffb640 1 --2- 172.21.15.42:0/1457015711 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f8c4c0a3d90 0x7f8c4c13c380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:54.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.005+0000 7f8c49ffb640 1 -- 172.21.15.42:0/1457015711 >> v1:172.21.15.154:6789/0 conn(0x7f8c4c0a39b0 legacy=0x7f8c4c133b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:05:54.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.005+0000 7f8c49ffb640 10 monclient: _finish_hunting 0 2024-09-16T04:05:54.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.005+0000 7f8c49ffb640 1 monclient: found mon.noname-a-legacy 2024-09-16T04:05:54.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.005+0000 7f8c49ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:05:54.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.005+0000 7f8c49ffb640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.42:6789/0 2024-09-16T04:05:54.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.005+0000 7f8c49ffb640 1 -- 172.21.15.42:0/1457015711 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c4c06dd60 con 0x7f8c4c13c8c0 2024-09-16T04:05:54.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.005+0000 7f8c49ffb640 10 monclient: _finish_auth 0 2024-09-16T04:05:54.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.005+0000 7f8c49ffb640 10 monclient: _check_auth_tickets 2024-09-16T04:05:54.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.005+0000 7f8c49ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:05:54.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.005+0000 7f8c5151d640 5 monclient: authenticate success, global_id 984348 2024-09-16T04:05:54.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.005+0000 7f8c5151d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:05:54.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.005+0000 7f8c49ffb640 1 -- 172.21.15.42:0/1457015711 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2177413734 0 0) 0x7f8c44001a00 con 0x7f8c4c13c8c0 2024-09-16T04:05:54.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.005+0000 7f8c49ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:05:54.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.005+0000 7f8c49ffb640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T04:05:54.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.005+0000 7f8c49ffb640 10 monclient: dump: 2024-09-16T04:05:54.028 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:05:54.028 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:05:54.028 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:54.029 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:54.029 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:05:54.029 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:05:54.029 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:05:54.029 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:05:54.029 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:05:54.029 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:05:54.029 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.005+0000 7f8c49ffb640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] but i'm connected to v1:172.21.15.42:6789/0, reconnecting 2024-09-16T04:05:54.029 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.005+0000 7f8c49ffb640 10 monclient: _reopen_session rank -1 2024-09-16T04:05:54.029 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.005+0000 7f8c49ffb640 1 -- 172.21.15.42:0/1457015711 >> v1:172.21.15.42:6789/0 conn(0x7f8c4c13c8c0 legacy=0x7f8c4c13ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:05:54.029 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.005+0000 7f8c49ffb640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:05:54.030 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.005+0000 7f8c49ffb640 1 --2- 172.21.15.42:0/1457015711 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f8c4c0a3d90 0x7f8c340039f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:05:54.030 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.005+0000 7f8c49ffb640 10 monclient(hunting): picked mon.c con 0x7f8c4c0a3d90 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:05:54.030 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.005+0000 7f8c49ffb640 1 --2- 172.21.15.42:0/1457015711 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8c34004020 0x7f8c34004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:05:54.030 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.005+0000 7f8c49ffb640 10 monclient(hunting): picked mon.a con 0x7f8c34004020 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:05:54.030 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.005+0000 7f8c49ffb640 1 --2- 172.21.15.42:0/1457015711 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8c34004da0 0x7f8c34005220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:05:54.030 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.005+0000 7f8c49ffb640 10 monclient(hunting): picked mon.b con 0x7f8c34004da0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:05:54.030 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.005+0000 7f8c49ffb640 10 monclient(hunting): start opening mon connection 2024-09-16T04:05:54.030 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.005+0000 7f8c49ffb640 1 -- 172.21.15.42:0/1457015711 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f8c4c09c420 con 0x7f8c34004020 2024-09-16T04:05:54.030 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.005+0000 7f8c49ffb640 10 monclient(hunting): start opening mon connection 2024-09-16T04:05:54.030 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.005+0000 7f8c49ffb640 1 -- 172.21.15.42:0/1457015711 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f8c3400bcb0 con 0x7f8c34004da0 2024-09-16T04:05:54.030 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.005+0000 7f8c49ffb640 10 monclient(hunting): start opening mon connection 2024-09-16T04:05:54.030 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.005+0000 7f8c49ffb640 1 -- 172.21.15.42:0/1457015711 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f8c3400beb0 con 0x7f8c4c0a3d90 2024-09-16T04:05:54.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.005+0000 7f8c49ffb640 10 monclient(hunting): _renew_subs 2024-09-16T04:05:54.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.005+0000 7f8c49ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:05:54.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.005+0000 7f8c5151d640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-16T04:05:54.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.005+0000 7f8c4affd640 1 --2- 172.21.15.42:0/1457015711 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f8c4c0a3d90 0x7f8c340039f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:05:54.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.005+0000 7f8c4a7fc640 1 --2- 172.21.15.42:0/1457015711 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8c34004020 0x7f8c34004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:05:54.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.005+0000 7f8c4a7fc640 10 monclient(hunting): get_auth_request con 0x7f8c34004020 auth_method 0 2024-09-16T04:05:54.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.005+0000 7f8c4a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:05:54.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.005+0000 7f8c4a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:05:54.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.005+0000 7f8c4a7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:05:54.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.005+0000 7f8c4affd640 10 monclient(hunting): get_auth_request con 0x7f8c4c0a3d90 auth_method 0 2024-09-16T04:05:54.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.005+0000 7f8c4affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:05:54.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.005+0000 7f8c4affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:05:54.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.005+0000 7f8c4affd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:05:54.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.005+0000 7f8c4a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:05:54.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.005+0000 7f8c4a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:05:54.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.005+0000 7f8c4a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:05:54.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.005+0000 7f8c4affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:05:54.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.005+0000 7f8c4affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:05:54.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.005+0000 7f8c4affd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:05:54.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.005+0000 7f8c4b7fe640 1 --2- 172.21.15.42:0/1457015711 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8c34004da0 0x7f8c34005220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:05:54.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.005+0000 7f8c4b7fe640 10 monclient(hunting): get_auth_request con 0x7f8c34004da0 auth_method 0 2024-09-16T04:05:54.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.005+0000 7f8c4b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:05:54.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.005+0000 7f8c4b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:05:54.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.005+0000 7f8c4b7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:05:54.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.005+0000 7f8c4a7fc640 10 monclient(hunting): handle_auth_done global_id 984348 payload 293 2024-09-16T04:05:54.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.005+0000 7f8c4a7fc640 1 -- 172.21.15.42:0/1457015711 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f8c4c0a3d90 msgr2=0x7f8c340039f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:05:54.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.005+0000 7f8c4a7fc640 1 --2- 172.21.15.42:0/1457015711 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f8c4c0a3d90 0x7f8c340039f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:54.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.005+0000 7f8c4a7fc640 1 -- 172.21.15.42:0/1457015711 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8c34004da0 msgr2=0x7f8c34005220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:05:54.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.005+0000 7f8c4a7fc640 1 --2- 172.21.15.42:0/1457015711 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8c34004da0 0x7f8c34005220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:54.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.005+0000 7f8c4a7fc640 10 monclient: _finish_hunting 0 2024-09-16T04:05:54.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.005+0000 7f8c4a7fc640 1 monclient: found mon.a 2024-09-16T04:05:54.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.005+0000 7f8c4a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:05:54.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.005+0000 7f8c4a7fc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:05:54.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.005+0000 7f8c4a7fc640 1 -- 172.21.15.42:0/1457015711 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c34003530 con 0x7f8c34004020 2024-09-16T04:05:54.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.005+0000 7f8c4a7fc640 10 monclient: _finish_auth 0 2024-09-16T04:05:54.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.005+0000 7f8c4a7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:05:54.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.005+0000 7f8c4a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:05:54.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.005+0000 7f8c4a7fc640 1 --2- 172.21.15.42:0/1457015711 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8c34004020 0x7f8c34004810 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f8c40000ee0 tx=0x7f8c4001c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d67415f164e8cfc server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:05:54.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.005+0000 7f8c49ffb640 1 -- 172.21.15.42:0/1457015711 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c400b0070 con 0x7f8c34004020 2024-09-16T04:05:54.034 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.005+0000 7f8c49ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:05:54.034 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.005+0000 7f8c49ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:05:54.034 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.005+0000 7f8c49ffb640 10 monclient: dump: 2024-09-16T04:05:54.034 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:05:54.034 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:05:54.034 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:54.034 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:54.034 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:05:54.034 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:05:54.034 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:05:54.034 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:05:54.034 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:05:54.034 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:05:54.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.005+0000 7f8c5151d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:05:54.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.005+0000 7f8c49ffb640 1 -- 172.21.15.42:0/1457015711 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8c400ab600 con 0x7f8c34004020 2024-09-16T04:05:54.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.005+0000 7f8c49ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:05:54.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.005+0000 7f8c5151d640 10 monclient: get_monmap_and_config success 2024-09-16T04:05:54.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.005+0000 7f8c5151d640 4 set_mon_vals no callback set 2024-09-16T04:05:54.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.005+0000 7f8c49ffb640 1 -- 172.21.15.42:0/1457015711 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c4000f040 con 0x7f8c34004020 2024-09-16T04:05:54.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.005+0000 7f8c49ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:05:54.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.005+0000 7f8c49ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:05:54.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.005+0000 7f8c49ffb640 10 monclient: dump: 2024-09-16T04:05:54.035 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:05:54.035 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:05:54.035 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:54.036 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:54.036 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:05:54.036 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:05:54.036 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:05:54.036 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:05:54.036 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:05:54.036 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:05:54.036 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.005+0000 7f8c5151d640 10 monclient: shutdown 2024-09-16T04:05:54.036 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.005+0000 7f8c5151d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:05:54.036 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.005+0000 7f8c5151d640 1 -- 172.21.15.42:0/1457015711 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8c34004020 msgr2=0x7f8c34004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:05:54.036 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.005+0000 7f8c5151d640 1 --2- 172.21.15.42:0/1457015711 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8c34004020 0x7f8c34004810 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f8c40000ee0 tx=0x7f8c4001c4d0 comp rx=0 tx=0).stop 2024-09-16T04:05:54.036 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.005+0000 7f8c4b7fe640 1 -- 172.21.15.42:0/1457015711 reap_dead start 2024-09-16T04:05:54.036 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.005+0000 7f8c5151d640 1 -- 172.21.15.42:0/1457015711 shutdown_connections 2024-09-16T04:05:54.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.005+0000 7f8c5151d640 1 -- 172.21.15.42:0/1457015711 >> 172.21.15.42:0/1457015711 conn(0x7f8c4c0a9fb0 msgr2=0x7f8c4c0aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:05:54.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.009+0000 7f8c5151d640 1 -- 172.21.15.42:0/1457015711 shutdown_connections 2024-09-16T04:05:54.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.009+0000 7f8c5151d640 1 -- 172.21.15.42:0/1457015711 wait complete. 2024-09-16T04:05:54.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.009+0000 7f8c5151d640 10 monclient: build_initial_monmap 2024-09-16T04:05:54.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.009+0000 7f8c5151d640 10 monclient: monmap: 2024-09-16T04:05:54.037 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:05:54.037 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:05:54.037 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:05:54.037 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:05:54.037 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:05:54.037 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:05:54.037 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:05:54.037 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:05:54.038 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:05:54.038 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:05:54.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.009+0000 7f8c5151d640 1 librados: starting msgr at 2024-09-16T04:05:54.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.009+0000 7f8c5151d640 1 librados: starting objecter 2024-09-16T04:05:54.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.009+0000 7f8c5151d640 1 Processor -- start 2024-09-16T04:05:54.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.009+0000 7f8c5151d640 1 -- start start 2024-09-16T04:05:54.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.009+0000 7f8c5151d640 1 librados: setting wanted keys 2024-09-16T04:05:54.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.009+0000 7f8c5151d640 1 librados: calling monclient init 2024-09-16T04:05:54.039 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.009+0000 7f8c5151d640 10 monclient: init 2024-09-16T04:05:54.039 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.009+0000 7f8c5151d640 10 monclient: _reopen_session rank -1 2024-09-16T04:05:54.039 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.009+0000 7f8c5151d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:05:54.039 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.009+0000 7f8c5151d640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8c34004020 0x7f8c4c0a3260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:05:54.039 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.009+0000 7f8c5151d640 10 monclient(hunting): picked mon.noname-a con 0x7f8c34004020 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:05:54.039 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.009+0000 7f8c5151d640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f8c4c09c940 0x7f8c4c09cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:05:54.039 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.009+0000 7f8c5151d640 10 monclient(hunting): picked mon.noname-c con 0x7f8c4c09c940 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:05:54.039 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.009+0000 7f8c5151d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8c4c09d300 0x7f8c4c1d6170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:05:54.039 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.009+0000 7f8c5151d640 10 monclient(hunting): picked mon.noname-b con 0x7f8c4c09d300 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:05:54.039 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.009+0000 7f8c5151d640 10 monclient(hunting): start opening mon connection 2024-09-16T04:05:54.039 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.009+0000 7f8c5151d640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f8c4c140f70 con 0x7f8c34004020 2024-09-16T04:05:54.039 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.009+0000 7f8c5151d640 10 monclient(hunting): start opening mon connection 2024-09-16T04:05:54.039 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.009+0000 7f8c5151d640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f8c4c140df0 con 0x7f8c4c09d300 2024-09-16T04:05:54.040 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.009+0000 7f8c5151d640 10 monclient(hunting): start opening mon connection 2024-09-16T04:05:54.040 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.009+0000 7f8c5151d640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f8c4c1410f0 con 0x7f8c4c09c940 2024-09-16T04:05:54.040 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.009+0000 7f8c5151d640 10 monclient(hunting): _renew_subs 2024-09-16T04:05:54.040 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.009+0000 7f8c5151d640 10 monclient(hunting): authenticate will time out at 2400.981490s 2024-09-16T04:05:54.040 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.009+0000 7f8c4a7fc640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f8c4c09c940 0x7f8c4c09cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:05:54.040 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.009+0000 7f8c4affd640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8c34004020 0x7f8c4c0a3260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:05:54.040 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.009+0000 7f8c4a7fc640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f8c4c09c940 0x7f8c4c09cdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:33326/0 (socket says 172.21.15.42:33326) 2024-09-16T04:05:54.040 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.009+0000 7f8c4a7fc640 1 -- 172.21.15.42:0/520055634 learned_addr learned my addr 172.21.15.42:0/520055634 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:05:54.040 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.009+0000 7f8c4a7fc640 10 monclient(hunting): get_auth_request con 0x7f8c4c09c940 auth_method 0 2024-09-16T04:05:54.040 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.009+0000 7f8c4a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:05:54.040 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.009+0000 7f8c4a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:05:54.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.009+0000 7f8c4a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:05:54.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.009+0000 7f8c4affd640 10 monclient(hunting): get_auth_request con 0x7f8c34004020 auth_method 0 2024-09-16T04:05:54.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.009+0000 7f8c4affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:05:54.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.009+0000 7f8c4affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:05:54.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.009+0000 7f8c4affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:05:54.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.009+0000 7f8c4a7fc640 1 -- 172.21.15.42:0/520055634 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f8c4c09c940 msgr2=0x7f8c4c09cdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-16T04:05:54.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.009+0000 7f8c4a7fc640 1 -- 172.21.15.42:0/520055634 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f8c4c09c940 msgr2=0x7f8c4c09cdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:05:54.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.009+0000 7f8c4a7fc640 1 --2- 172.21.15.42:0/520055634 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f8c4c09c940 0x7f8c4c09cdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:05:54.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.009+0000 7f8c4affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:05:54.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.009+0000 7f8c4affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:05:54.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.009+0000 7f8c4affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:05:54.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.009+0000 7f8c4a7fc640 1 --2- 172.21.15.42:0/520055634 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f8c4c09c940 0x7f8c4c09cdc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:05:54.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.009+0000 7f8c4b7fe640 1 --2- 172.21.15.42:0/520055634 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8c4c09d300 0x7f8c4c1d6170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:05:54.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.009+0000 7f8c4b7fe640 10 monclient(hunting): get_auth_request con 0x7f8c4c09d300 auth_method 0 2024-09-16T04:05:54.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.009+0000 7f8c4b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:05:54.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.009+0000 7f8c4b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:05:54.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.009+0000 7f8c4b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:05:54.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.009+0000 7f8c4affd640 10 monclient(hunting): handle_auth_done global_id 984351 payload 824 2024-09-16T04:05:54.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.009+0000 7f8c4affd640 1 -- 172.21.15.42:0/520055634 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f8c4c09c940 msgr2=0x7f8c4c09cdc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:05:54.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.009+0000 7f8c4affd640 1 --2- 172.21.15.42:0/520055634 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f8c4c09c940 0x7f8c4c09cdc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:54.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.009+0000 7f8c4affd640 1 -- 172.21.15.42:0/520055634 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8c4c09d300 msgr2=0x7f8c4c1d6170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:05:54.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.009+0000 7f8c4affd640 1 --2- 172.21.15.42:0/520055634 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8c4c09d300 0x7f8c4c1d6170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:54.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.009+0000 7f8c4affd640 10 monclient: _finish_hunting 0 2024-09-16T04:05:54.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.009+0000 7f8c4affd640 1 monclient: found mon.noname-a 2024-09-16T04:05:54.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.009+0000 7f8c4affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:05:54.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.009+0000 7f8c4affd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:05:54.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.009+0000 7f8c4affd640 1 -- 172.21.15.42:0/520055634 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c4c140070 con 0x7f8c34004020 2024-09-16T04:05:54.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.009+0000 7f8c4affd640 1 --2- 172.21.15.42:0/520055634 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8c34004020 0x7f8c4c0a3260 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f8c3c00f5f0 tx=0x7f8c3c00fab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:05:54.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.009+0000 7f8c337fe640 10 client.?.objecter ms_handle_connect 0x7f8c34004020 2024-09-16T04:05:54.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.009+0000 7f8c337fe640 10 client.?.objecter resend_mon_ops 2024-09-16T04:05:54.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.009+0000 7f8c337fe640 1 -- 172.21.15.42:0/520055634 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c3c0052e0 con 0x7f8c34004020 2024-09-16T04:05:54.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.009+0000 7f8c337fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:05:54.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.009+0000 7f8c337fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:05:54.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.009+0000 7f8c337fe640 10 monclient: dump: 2024-09-16T04:05:54.043 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:05:54.043 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:05:54.043 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:54.043 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:54.044 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:05:54.044 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:05:54.044 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:05:54.044 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:05:54.044 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:05:54.044 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:05:54.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.009+0000 7f8c337fe640 10 monclient: _finish_auth 0 2024-09-16T04:05:54.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.009+0000 7f8c337fe640 10 monclient: _check_auth_tickets 2024-09-16T04:05:54.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.009+0000 7f8c337fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:05:54.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.009+0000 7f8c337fe640 1 -- 172.21.15.42:0/520055634 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8c3c018070 con 0x7f8c34004020 2024-09-16T04:05:54.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.009+0000 7f8c5151d640 5 monclient: authenticate success, global_id 984351 2024-09-16T04:05:54.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.009+0000 7f8c5151d640 10 monclient: _renew_subs 2024-09-16T04:05:54.045 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.009+0000 7f8c5151d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:05:54.045 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.009+0000 7f8c5151d640 1 -- 172.21.15.42:0/520055634 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c4c06f680 con 0x7f8c34004020 2024-09-16T04:05:54.045 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.009+0000 7f8c337fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:05:54.045 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.009+0000 7f8c337fe640 1 -- 172.21.15.42:0/520055634 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c3c003820 con 0x7f8c34004020 2024-09-16T04:05:54.045 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.009+0000 7f8c337fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:05:54.045 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.009+0000 7f8c337fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:05:54.045 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.009+0000 7f8c337fe640 10 monclient: dump: 2024-09-16T04:05:54.045 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:05:54.045 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:05:54.045 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:54.045 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:54.045 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:05:54.045 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:05:54.046 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:05:54.046 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:05:54.046 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:05:54.046 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:05:54.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.009+0000 7f8c48ff9640 4 set_mon_vals no callback set 2024-09-16T04:05:54.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.009+0000 7f8c5151d640 10 client.984351.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:05:54.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.009+0000 7f8c5151d640 10 monclient: _renew_subs 2024-09-16T04:05:54.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.009+0000 7f8c5151d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:05:54.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.009+0000 7f8c5151d640 1 -- 172.21.15.42:0/520055634 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c4c09e800 con 0x7f8c34004020 2024-09-16T04:05:54.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.009+0000 7f8c5151d640 1 librados: init done 2024-09-16T04:05:54.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.009+0000 7f8c5151d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:05:54.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.009+0000 7f8c5151d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:05:54.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.009+0000 7f8c5151d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:05:54.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.009+0000 7f8c337fe640 1 -- 172.21.15.42:0/520055634 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f8c3c0039c0 con 0x7f8c34004020 2024-09-16T04:05:54.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.009+0000 7f8c5151d640 1 -- 172.21.15.42:0/520055634 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c4c09e360 con 0x7f8c34004020 2024-09-16T04:05:54.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.009+0000 7f8c337fe640 1 --2- 172.21.15.42:0/520055634 >> v2:172.21.15.154:6812/254845342 conn(0x7f8c38037840 0x7f8c38039d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:05:54.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.009+0000 7f8c337fe640 1 -- 172.21.15.42:0/520055634 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(706..706 src has 451..706) ==== 5040+0+0 (secure 0 0 0) 0x7f8c3c048a90 con 0x7f8c34004020 2024-09-16T04:05:54.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.009+0000 7f8c337fe640 10 client.984351.objecter ms_dispatch 0x7f8c4c000ba0 osd_map(706..706 src has 451..706) 2024-09-16T04:05:54.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.009+0000 7f8c337fe640 3 client.984351.objecter handle_osd_map got epochs [706,706] > 0 2024-09-16T04:05:54.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.009+0000 7f8c337fe640 3 client.984351.objecter handle_osd_map decoding full epoch 706 2024-09-16T04:05:54.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.009+0000 7f8c337fe640 20 client.984351.objecter dump_active .. 0 homeless 2024-09-16T04:05:54.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.009+0000 7f8c4a7fc640 1 --2- 172.21.15.42:0/520055634 >> v2:172.21.15.154:6812/254845342 conn(0x7f8c38037840 0x7f8c38039d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:05:54.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.009+0000 7f8c4a7fc640 10 monclient: get_auth_request con 0x7f8c38037840 auth_method 0 2024-09-16T04:05:54.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.013+0000 7f8c4a7fc640 1 --2- 172.21.15.42:0/520055634 >> v2:172.21.15.154:6812/254845342 conn(0x7f8c38037840 0x7f8c38039d00 secure :-1 s=READY pgs=3855 cs=0 l=1 rev1=1 crypto rx=0x7f8c40000ee0 tx=0x7f8c400046d0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:05:54.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.013+0000 7f8c337fe640 10 client.984351.objecter ms_handle_connect 0x7f8c38037840 2024-09-16T04:05:54.048 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.017+0000 7f8c337fe640 1 -- 172.21.15.42:0/520055634 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f8c4c06f680 con 0x7f8c34004020 2024-09-16T04:05:54.048 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.017+0000 7f8c337fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:05:54.048 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.017+0000 7f8c337fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:05:54.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.237+0000 7f8c5151d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:05:54.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.237+0000 7f8c5151d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:05:54.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.237+0000 7f8c5151d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:05:54.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.237+0000 7f8c5151d640 1 -- 172.21.15.42:0/520055634 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f8c4c13fcd0 con 0x7f8c34004020 2024-09-16T04:05:54.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.241+0000 7f8c337fe640 1 -- 172.21.15.42:0/520055634 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7f8c4c140070 con 0x7f8c34004020 2024-09-16T04:05:54.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.241+0000 7f8c337fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:05:54.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.241+0000 7f8c337fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:05:54.245 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":220,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":21,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:05:54.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.241+0000 7f8c5151d640 10 librados: watch_flush enter 2024-09-16T04:05:54.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.241+0000 7f8c5151d640 10 librados: watch_flush exit 2024-09-16T04:05:54.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.241+0000 7f8c5151d640 20 client.984351.objecter shutdown clearing up homeless session... 2024-09-16T04:05:54.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.241+0000 7f8c5151d640 10 client.984351.objecter successfully canceled tick 2024-09-16T04:05:54.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.241+0000 7f8c5151d640 1 -- 172.21.15.42:0/520055634 >> v2:172.21.15.154:6812/254845342 conn(0x7f8c38037840 msgr2=0x7f8c38039d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:05:54.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.241+0000 7f8c5151d640 1 --2- 172.21.15.42:0/520055634 >> v2:172.21.15.154:6812/254845342 conn(0x7f8c38037840 0x7f8c38039d00 secure :-1 s=READY pgs=3855 cs=0 l=1 rev1=1 crypto rx=0x7f8c40000ee0 tx=0x7f8c400046d0 comp rx=0 tx=0).stop 2024-09-16T04:05:54.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.241+0000 7f8c5151d640 10 monclient: shutdown 2024-09-16T04:05:54.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.241+0000 7f8c5151d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:05:54.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.241+0000 7f8c5151d640 1 -- 172.21.15.42:0/520055634 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8c34004020 msgr2=0x7f8c4c0a3260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:05:54.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.241+0000 7f8c5151d640 1 --2- 172.21.15.42:0/520055634 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8c34004020 0x7f8c4c0a3260 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f8c3c00f5f0 tx=0x7f8c3c00fab0 comp rx=0 tx=0).stop 2024-09-16T04:05:54.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.241+0000 7f8c5151d640 1 -- 172.21.15.42:0/520055634 shutdown_connections 2024-09-16T04:05:54.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.241+0000 7f8c5151d640 1 --2- 172.21.15.42:0/520055634 >> v2:172.21.15.154:6812/254845342 conn(0x7f8c38037840 0x7f8c38039d00 unknown :-1 s=CLOSED pgs=3855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:54.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.241+0000 7f8c5151d640 1 --2- 172.21.15.42:0/520055634 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8c4c09d300 0x7f8c4c1d6170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:54.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.241+0000 7f8c5151d640 1 --2- 172.21.15.42:0/520055634 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f8c4c09c940 0x7f8c4c09cdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:54.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.241+0000 7f8c5151d640 1 --2- 172.21.15.42:0/520055634 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8c34004020 0x7f8c4c0a3260 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:54.249 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.241+0000 7f8c5151d640 1 -- 172.21.15.42:0/520055634 >> 172.21.15.42:0/520055634 conn(0x7f8c4c0a9fb0 msgr2=0x7f8c4c0a87d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:05:54.249 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.241+0000 7f8c5151d640 1 -- 172.21.15.42:0/520055634 shutdown_connections 2024-09-16T04:05:54.249 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.241+0000 7f8c5151d640 1 -- 172.21.15.42:0/520055634 wait complete. 2024-09-16T04:05:54.249 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:54.241+0000 7f8c5151d640 1 librados: shutdown 2024-09-16T04:05:57.264 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:05:57.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.389+0000 7fbd2a5ee640 10 monclient: get_monmap_and_config 2024-09-16T04:05:57.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.389+0000 7fbd2a5ee640 10 monclient: build_initial_monmap 2024-09-16T04:05:57.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.389+0000 7fbd2a5ee640 10 monclient: monmap: 2024-09-16T04:05:57.395 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:05:57.395 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:05:57.396 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:05:57.393899+0000 2024-09-16T04:05:57.396 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:05:57.393899+0000 2024-09-16T04:05:57.396 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:05:57.396 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:05:57.396 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:05:57.396 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:05:57.396 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:05:57.396 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:05:57.396 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:05:57.396 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:05:57.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.389+0000 7fbd2a5ee640 1 Processor -- start 2024-09-16T04:05:57.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.389+0000 7fbd2a5ee640 1 -- start start 2024-09-16T04:05:57.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.389+0000 7fbd2a5ee640 10 monclient: init 2024-09-16T04:05:57.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.389+0000 7fbd2a5ee640 10 monclient: _reopen_session rank -1 2024-09-16T04:05:57.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.389+0000 7fbd2a5ee640 10 monclient: _add_conns ranks=[1,3,4,0,2] 2024-09-16T04:05:57.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.389+0000 7fbd2a5ee640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbd240a1330 addr v1:172.21.15.154:6789/0 2024-09-16T04:05:57.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.389+0000 7fbd2a5ee640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fbd240a1710 0x7fbd240a1b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:05:57.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.389+0000 7fbd2a5ee640 10 monclient(hunting): picked mon.noname-a con 0x7fbd240a1710 addr v2:172.21.15.42:3300/0 2024-09-16T04:05:57.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.389+0000 7fbd2a5ee640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fbd241393a0 0x7fbd2413b790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:05:57.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.389+0000 7fbd2a5ee640 10 monclient(hunting): picked mon.noname-c con 0x7fbd241393a0 addr v2:172.21.15.154:3300/0 2024-09-16T04:05:57.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.389+0000 7fbd2a5ee640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbd2409e4c0 con 0x7fbd240a1330 2024-09-16T04:05:57.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.389+0000 7fbd2a5ee640 10 monclient(hunting): start opening mon connection 2024-09-16T04:05:57.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.389+0000 7fbd2a5ee640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7fbd2409edf0 con 0x7fbd240a1710 2024-09-16T04:05:57.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.389+0000 7fbd2a5ee640 10 monclient(hunting): start opening mon connection 2024-09-16T04:05:57.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.389+0000 7fbd2a5ee640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fbd2409cd60 con 0x7fbd241393a0 2024-09-16T04:05:57.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.389+0000 7fbd2a5ee640 10 monclient(hunting): _renew_subs 2024-09-16T04:05:57.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.389+0000 7fbd2a5ee640 10 monclient(hunting): authenticate will time out at 2404.363328s 2024-09-16T04:05:57.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.389+0000 7fbd237fe640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fbd240a1710 0x7fbd240a1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:05:57.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.389+0000 7fbd237fe640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fbd240a1710 0x7fbd240a1b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:38660/0 (socket says 172.21.15.42:38660) 2024-09-16T04:05:57.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.389+0000 7fbd237fe640 1 -- 172.21.15.42:0/3660328545 learned_addr learned my addr 172.21.15.42:0/3660328545 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:05:57.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.389+0000 7fbd237fe640 10 monclient(hunting): get_auth_request con 0x7fbd240a1710 auth_method 0 2024-09-16T04:05:57.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.389+0000 7fbd237fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:05:57.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.389+0000 7fbd237fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:05:57.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.389+0000 7fbd237fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:05:57.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.389+0000 7fbd237fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:05:57.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.389+0000 7fbd237fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:05:57.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.389+0000 7fbd237fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:05:57.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.389+0000 7fbd28b8b640 1 --2- 172.21.15.42:0/3660328545 >> v2:172.21.15.154:3300/0 conn(0x7fbd241393a0 0x7fbd2413b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:05:57.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.389+0000 7fbd237fe640 10 monclient(hunting): handle_auth_done global_id 984354 payload 274 2024-09-16T04:05:57.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.389+0000 7fbd237fe640 1 -- 172.21.15.42:0/3660328545 >> v2:172.21.15.154:3300/0 conn(0x7fbd241393a0 msgr2=0x7fbd2413b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:05:57.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.389+0000 7fbd237fe640 1 --2- 172.21.15.42:0/3660328545 >> v2:172.21.15.154:3300/0 conn(0x7fbd241393a0 0x7fbd2413b790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:57.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.389+0000 7fbd237fe640 1 -- 172.21.15.42:0/3660328545 >> v1:172.21.15.154:6789/0 conn(0x7fbd240a1330 legacy=0x7fbd2409f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:05:57.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.389+0000 7fbd237fe640 10 monclient: _finish_hunting 0 2024-09-16T04:05:57.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.389+0000 7fbd237fe640 1 monclient: found mon.noname-a 2024-09-16T04:05:57.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.389+0000 7fbd237fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:05:57.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.389+0000 7fbd237fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:05:57.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.389+0000 7fbd237fe640 1 -- 172.21.15.42:0/3660328545 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd240a2080 con 0x7fbd240a1710 2024-09-16T04:05:57.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.389+0000 7fbd28b8b640 10 monclient: get_auth_request con 0x7fbd241393a0 auth_method 0 2024-09-16T04:05:57.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.389+0000 7fbd28b8b640 1 --2- 172.21.15.42:0/3660328545 >> v2:172.21.15.154:3300/0 conn(0x7fbd241393a0 0x7fbd2413b790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T04:05:57.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.393+0000 7fbd237fe640 1 --2- 172.21.15.42:0/3660328545 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbd240a1710 0x7fbd240a1b40 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fbd2409df50 tx=0x7fbd0c01c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b19e63f364f14245 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:05:57.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.393+0000 7fbd22ffd640 1 -- 172.21.15.42:0/3660328545 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd0c0b0070 con 0x7fbd240a1710 2024-09-16T04:05:57.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.393+0000 7fbd22ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:05:57.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.393+0000 7fbd22ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:05:57.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.393+0000 7fbd22ffd640 10 monclient: dump: 2024-09-16T04:05:57.408 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:05:57.408 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:05:57.408 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:57.408 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:57.408 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:05:57.408 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:05:57.408 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:05:57.408 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:05:57.408 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:05:57.408 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:05:57.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.393+0000 7fbd22ffd640 10 monclient: _finish_auth 0 2024-09-16T04:05:57.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.393+0000 7fbd22ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:05:57.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.393+0000 7fbd22ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:05:57.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.393+0000 7fbd22ffd640 1 -- 172.21.15.42:0/3660328545 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbd0c0ab600 con 0x7fbd240a1710 2024-09-16T04:05:57.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.393+0000 7fbd22ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:05:57.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.393+0000 7fbd22ffd640 1 -- 172.21.15.42:0/3660328545 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd0c0ab9d0 con 0x7fbd240a1710 2024-09-16T04:05:57.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.393+0000 7fbd2a5ee640 5 monclient: authenticate success, global_id 984354 2024-09-16T04:05:57.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.393+0000 7fbd2a5ee640 10 monclient: get_monmap_and_config success 2024-09-16T04:05:57.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.393+0000 7fbd2a5ee640 4 set_mon_vals no callback set 2024-09-16T04:05:57.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.393+0000 7fbd22ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:05:57.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.393+0000 7fbd22ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:05:57.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.393+0000 7fbd22ffd640 10 monclient: dump: 2024-09-16T04:05:57.410 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:05:57.410 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:05:57.411 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:57.411 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:57.411 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:05:57.411 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:05:57.411 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:05:57.411 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:05:57.411 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:05:57.412 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:05:57.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.393+0000 7fbd2a5ee640 10 monclient: shutdown 2024-09-16T04:05:57.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.393+0000 7fbd2a5ee640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:05:57.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.393+0000 7fbd2a5ee640 1 -- 172.21.15.42:0/3660328545 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbd240a1710 msgr2=0x7fbd240a1b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:05:57.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.393+0000 7fbd2a5ee640 1 --2- 172.21.15.42:0/3660328545 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbd240a1710 0x7fbd240a1b40 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fbd2409df50 tx=0x7fbd0c01c3a0 comp rx=0 tx=0).stop 2024-09-16T04:05:57.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.393+0000 7fbd2a5ee640 1 -- 172.21.15.42:0/3660328545 shutdown_connections 2024-09-16T04:05:57.413 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.393+0000 7fbd2a5ee640 1 --2- 172.21.15.42:0/3660328545 >> v2:172.21.15.154:3300/0 conn(0x7fbd241393a0 0x7fbd2413b790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:57.413 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.393+0000 7fbd2a5ee640 1 --2- 172.21.15.42:0/3660328545 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbd240a1710 0x7fbd240a1b40 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:57.413 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.393+0000 7fbd2a5ee640 1 -- 172.21.15.42:0/3660328545 >> 172.21.15.42:0/3660328545 conn(0x7fbd240a2420 msgr2=0x7fbd240a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:05:57.413 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.393+0000 7fbd2a5ee640 1 -- 172.21.15.42:0/3660328545 shutdown_connections 2024-09-16T04:05:57.413 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.393+0000 7fbd2a5ee640 1 -- 172.21.15.42:0/3660328545 wait complete. 2024-09-16T04:05:57.413 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.393+0000 7fbd2a5ee640 10 monclient: build_initial_monmap 2024-09-16T04:05:57.413 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.393+0000 7fbd2a5ee640 10 monclient: monmap: 2024-09-16T04:05:57.414 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:05:57.414 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:05:57.414 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:05:57.414 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:05:57.414 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:05:57.415 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:05:57.415 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:05:57.415 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:05:57.415 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:05:57.415 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:05:57.415 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.393+0000 7fbd2a5ee640 1 librados: starting msgr at 2024-09-16T04:05:57.416 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.393+0000 7fbd2a5ee640 1 librados: starting objecter 2024-09-16T04:05:57.416 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.393+0000 7fbd2a5ee640 1 Processor -- start 2024-09-16T04:05:57.416 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.393+0000 7fbd2a5ee640 1 -- start start 2024-09-16T04:05:57.416 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.393+0000 7fbd2a5ee640 1 librados: setting wanted keys 2024-09-16T04:05:57.416 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.393+0000 7fbd2a5ee640 1 librados: calling monclient init 2024-09-16T04:05:57.417 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.393+0000 7fbd2a5ee640 10 monclient: init 2024-09-16T04:05:57.417 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.397+0000 7fbd2a5ee640 10 monclient: _reopen_session rank -1 2024-09-16T04:05:57.417 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.397+0000 7fbd2a5ee640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:05:57.417 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.397+0000 7fbd2a5ee640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbd240a1330 0x7fbd241cbb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:05:57.417 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.397+0000 7fbd2a5ee640 10 monclient(hunting): picked mon.noname-a con 0x7fbd240a1330 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:05:57.417 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.397+0000 7fbd2a5ee640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbd240a1710 0x7fbd24134760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:05:57.418 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.397+0000 7fbd2a5ee640 10 monclient(hunting): picked mon.noname-c con 0x7fbd240a1710 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:05:57.418 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.397+0000 7fbd2a5ee640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbd241393a0 0x7fbd241cf030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:05:57.418 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.397+0000 7fbd2a5ee640 10 monclient(hunting): picked mon.noname-b con 0x7fbd241393a0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:05:57.418 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.397+0000 7fbd2a5ee640 10 monclient(hunting): start opening mon connection 2024-09-16T04:05:57.418 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.397+0000 7fbd2a5ee640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fbd2413e880 con 0x7fbd240a1330 2024-09-16T04:05:57.418 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.397+0000 7fbd2a5ee640 10 monclient(hunting): start opening mon connection 2024-09-16T04:05:57.419 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.397+0000 7fbd2a5ee640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fbd2413e700 con 0x7fbd241393a0 2024-09-16T04:05:57.419 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.397+0000 7fbd2a5ee640 10 monclient(hunting): start opening mon connection 2024-09-16T04:05:57.419 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.397+0000 7fbd2a5ee640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fbd2413ea00 con 0x7fbd240a1710 2024-09-16T04:05:57.419 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.397+0000 7fbd2a5ee640 10 monclient(hunting): _renew_subs 2024-09-16T04:05:57.419 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.397+0000 7fbd2a5ee640 10 monclient(hunting): authenticate will time out at 2404.368307s 2024-09-16T04:05:57.420 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.397+0000 7fbd23fff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbd240a1330 0x7fbd241cbb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:05:57.420 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.397+0000 7fbd237fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbd240a1710 0x7fbd24134760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:05:57.420 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.397+0000 7fbd23fff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbd240a1330 0x7fbd241cbb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:38670/0 (socket says 172.21.15.42:38670) 2024-09-16T04:05:57.420 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.397+0000 7fbd23fff640 1 -- 172.21.15.42:0/2781345496 learned_addr learned my addr 172.21.15.42:0/2781345496 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:05:57.420 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.397+0000 7fbd23fff640 10 monclient(hunting): get_auth_request con 0x7fbd240a1330 auth_method 0 2024-09-16T04:05:57.421 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.397+0000 7fbd23fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:05:57.421 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.397+0000 7fbd23fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:05:57.421 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.397+0000 7fbd23fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:05:57.421 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.397+0000 7fbd237fe640 10 monclient(hunting): get_auth_request con 0x7fbd240a1710 auth_method 0 2024-09-16T04:05:57.421 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.397+0000 7fbd237fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:05:57.421 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.397+0000 7fbd237fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:05:57.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.397+0000 7fbd237fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:05:57.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.397+0000 7fbd23fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:05:57.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.397+0000 7fbd23fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:05:57.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.397+0000 7fbd237fe640 1 -- 172.21.15.42:0/2781345496 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbd240a1710 msgr2=0x7fbd24134760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-16T04:05:57.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.397+0000 7fbd237fe640 1 -- 172.21.15.42:0/2781345496 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbd240a1710 msgr2=0x7fbd24134760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:05:57.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.397+0000 7fbd237fe640 1 --2- 172.21.15.42:0/2781345496 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbd240a1710 0x7fbd24134760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:05:57.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.397+0000 7fbd23fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:05:57.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.397+0000 7fbd237fe640 1 --2- 172.21.15.42:0/2781345496 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbd240a1710 0x7fbd24134760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:05:57.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.397+0000 7fbd23fff640 10 monclient(hunting): handle_auth_done global_id 984357 payload 824 2024-09-16T04:05:57.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.397+0000 7fbd23fff640 1 -- 172.21.15.42:0/2781345496 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbd240a1710 msgr2=0x7fbd24134760 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:05:57.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.397+0000 7fbd23fff640 1 --2- 172.21.15.42:0/2781345496 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbd240a1710 0x7fbd24134760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:57.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.397+0000 7fbd23fff640 1 -- 172.21.15.42:0/2781345496 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbd241393a0 msgr2=0x7fbd241cf030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:05:57.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.397+0000 7fbd23fff640 1 --2- 172.21.15.42:0/2781345496 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbd241393a0 0x7fbd241cf030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:57.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.397+0000 7fbd23fff640 10 monclient: _finish_hunting 0 2024-09-16T04:05:57.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.397+0000 7fbd23fff640 1 monclient: found mon.noname-a 2024-09-16T04:05:57.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.397+0000 7fbd23fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:05:57.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.397+0000 7fbd23fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:05:57.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.397+0000 7fbd23fff640 1 -- 172.21.15.42:0/2781345496 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd240a4370 con 0x7fbd240a1330 2024-09-16T04:05:57.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.397+0000 7fbd23fff640 1 --2- 172.21.15.42:0/2781345496 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbd240a1330 0x7fbd241cbb30 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fbd1c00c7a0 tx=0x7fbd1c00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:05:57.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.397+0000 7fbd20ff9640 10 client.?.objecter ms_handle_connect 0x7fbd240a1330 2024-09-16T04:05:57.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.397+0000 7fbd20ff9640 10 client.?.objecter resend_mon_ops 2024-09-16T04:05:57.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.397+0000 7fbd20ff9640 1 -- 172.21.15.42:0/2781345496 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd1c017070 con 0x7fbd240a1330 2024-09-16T04:05:57.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.397+0000 7fbd20ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:05:57.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.397+0000 7fbd20ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:05:57.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.397+0000 7fbd20ff9640 10 monclient: dump: 2024-09-16T04:05:57.426 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:05:57.426 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:05:57.426 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:57.427 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:57.427 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:05:57.427 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:05:57.427 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:05:57.427 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:05:57.427 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:05:57.428 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:05:57.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.397+0000 7fbd20ff9640 10 monclient: _finish_auth 0 2024-09-16T04:05:57.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.397+0000 7fbd20ff9640 10 monclient: _check_auth_tickets 2024-09-16T04:05:57.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.397+0000 7fbd20ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:05:57.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.397+0000 7fbd20ff9640 1 -- 172.21.15.42:0/2781345496 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbd1c003db0 con 0x7fbd240a1330 2024-09-16T04:05:57.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.397+0000 7fbd2a5ee640 5 monclient: authenticate success, global_id 984357 2024-09-16T04:05:57.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.397+0000 7fbd2a5ee640 10 monclient: _renew_subs 2024-09-16T04:05:57.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.397+0000 7fbd2a5ee640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:05:57.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.397+0000 7fbd2a5ee640 1 -- 172.21.15.42:0/2781345496 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd24139170 con 0x7fbd240a1330 2024-09-16T04:05:57.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.397+0000 7fbd20ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T04:05:57.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.397+0000 7fbd20ff9640 1 -- 172.21.15.42:0/2781345496 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd1c0052b0 con 0x7fbd240a1330 2024-09-16T04:05:57.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.397+0000 7fbd20ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:05:57.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.397+0000 7fbd20ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:05:57.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.397+0000 7fbd20ff9640 10 monclient: dump: 2024-09-16T04:05:57.430 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:05:57.430 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:05:57.431 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:57.431 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:05:57.431 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:05:57.431 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:05:57.431 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:05:57.431 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:05:57.432 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:05:57.432 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:05:57.432 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.397+0000 7fbd227fc640 4 set_mon_vals no callback set 2024-09-16T04:05:57.432 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.397+0000 7fbd2a5ee640 10 client.984357.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:05:57.432 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.397+0000 7fbd2a5ee640 10 monclient: _renew_subs 2024-09-16T04:05:57.432 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.397+0000 7fbd2a5ee640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:05:57.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.397+0000 7fbd2a5ee640 1 -- 172.21.15.42:0/2781345496 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd24121e40 con 0x7fbd240a1330 2024-09-16T04:05:57.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.397+0000 7fbd2a5ee640 1 librados: init done 2024-09-16T04:05:57.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.397+0000 7fbd2a5ee640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:05:57.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.397+0000 7fbd2a5ee640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:05:57.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.397+0000 7fbd20ff9640 1 -- 172.21.15.42:0/2781345496 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fbd1c01a070 con 0x7fbd240a1330 2024-09-16T04:05:57.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.397+0000 7fbd2a5ee640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:05:57.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.397+0000 7fbd2a5ee640 1 -- 172.21.15.42:0/2781345496 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbd240a1b40 con 0x7fbd240a1330 2024-09-16T04:05:57.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.397+0000 7fbd20ff9640 1 --2- 172.21.15.42:0/2781345496 >> v2:172.21.15.154:6812/254845342 conn(0x7fbd04037840 0x7fbd04039d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:05:57.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.397+0000 7fbd20ff9640 1 -- 172.21.15.42:0/2781345496 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(706..706 src has 451..706) ==== 5040+0+0 (secure 0 0 0) 0x7fbd1c048360 con 0x7fbd240a1330 2024-09-16T04:05:57.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.397+0000 7fbd20ff9640 10 client.984357.objecter ms_dispatch 0x7fbd24000ba0 osd_map(706..706 src has 451..706) 2024-09-16T04:05:57.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.397+0000 7fbd20ff9640 3 client.984357.objecter handle_osd_map got epochs [706,706] > 0 2024-09-16T04:05:57.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.397+0000 7fbd20ff9640 3 client.984357.objecter handle_osd_map decoding full epoch 706 2024-09-16T04:05:57.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.397+0000 7fbd20ff9640 20 client.984357.objecter dump_active .. 0 homeless 2024-09-16T04:05:57.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.397+0000 7fbd237fe640 1 --2- 172.21.15.42:0/2781345496 >> v2:172.21.15.154:6812/254845342 conn(0x7fbd04037840 0x7fbd04039d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:05:57.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.397+0000 7fbd237fe640 10 monclient: get_auth_request con 0x7fbd04037840 auth_method 0 2024-09-16T04:05:57.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.397+0000 7fbd20ff9640 10 client.984357.objecter ms_handle_connect 0x7fbd04037840 2024-09-16T04:05:57.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.397+0000 7fbd237fe640 1 --2- 172.21.15.42:0/2781345496 >> v2:172.21.15.154:6812/254845342 conn(0x7fbd04037840 0x7fbd04039d00 secure :-1 s=READY pgs=3856 cs=0 l=1 rev1=1 crypto rx=0x7fbd0c0ad000 tx=0x7fbd0c0a6000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:05:57.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.401+0000 7fbd20ff9640 1 -- 172.21.15.42:0/2781345496 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7fbd1c01a360 con 0x7fbd240a1330 2024-09-16T04:05:57.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.401+0000 7fbd20ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:05:57.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.401+0000 7fbd20ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:05:57.637 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.629+0000 7fbd2a5ee640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:05:57.637 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.629+0000 7fbd2a5ee640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:05:57.637 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.629+0000 7fbd2a5ee640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:05:57.637 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.629+0000 7fbd2a5ee640 1 -- 172.21.15.42:0/2781345496 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fbd240741a0 con 0x7fbd240a1330 2024-09-16T04:05:57.637 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.629+0000 7fbd20ff9640 1 -- 172.21.15.42:0/2781345496 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7fbd1c01bdd0 con 0x7fbd240a1330 2024-09-16T04:05:57.638 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.629+0000 7fbd20ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:05:57.638 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.629+0000 7fbd20ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:05:57.638 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":220,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":24,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:05:57.639 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.629+0000 7fbd2a5ee640 10 librados: watch_flush enter 2024-09-16T04:05:57.639 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.629+0000 7fbd2a5ee640 10 librados: watch_flush exit 2024-09-16T04:05:57.639 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.629+0000 7fbd2a5ee640 20 client.984357.objecter shutdown clearing up homeless session... 2024-09-16T04:05:57.639 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.629+0000 7fbd2a5ee640 10 client.984357.objecter successfully canceled tick 2024-09-16T04:05:57.639 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.629+0000 7fbd2a5ee640 1 -- 172.21.15.42:0/2781345496 >> v2:172.21.15.154:6812/254845342 conn(0x7fbd04037840 msgr2=0x7fbd04039d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:05:57.640 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.629+0000 7fbd2a5ee640 1 --2- 172.21.15.42:0/2781345496 >> v2:172.21.15.154:6812/254845342 conn(0x7fbd04037840 0x7fbd04039d00 secure :-1 s=READY pgs=3856 cs=0 l=1 rev1=1 crypto rx=0x7fbd0c0ad000 tx=0x7fbd0c0a6000 comp rx=0 tx=0).stop 2024-09-16T04:05:57.640 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.629+0000 7fbd2a5ee640 10 monclient: shutdown 2024-09-16T04:05:57.640 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.629+0000 7fbd2a5ee640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:05:57.640 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.633+0000 7fbd2a5ee640 1 -- 172.21.15.42:0/2781345496 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbd240a1330 msgr2=0x7fbd241cbb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:05:57.640 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.633+0000 7fbd2a5ee640 1 --2- 172.21.15.42:0/2781345496 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbd240a1330 0x7fbd241cbb30 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fbd1c00c7a0 tx=0x7fbd1c00cc60 comp rx=0 tx=0).stop 2024-09-16T04:05:57.640 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.633+0000 7fbd2a5ee640 1 -- 172.21.15.42:0/2781345496 shutdown_connections 2024-09-16T04:05:57.640 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.633+0000 7fbd2a5ee640 1 --2- 172.21.15.42:0/2781345496 >> v2:172.21.15.154:6812/254845342 conn(0x7fbd04037840 0x7fbd04039d00 unknown :-1 s=CLOSED pgs=3856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:57.640 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.633+0000 7fbd2a5ee640 1 --2- 172.21.15.42:0/2781345496 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbd241393a0 0x7fbd241cf030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:57.640 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.633+0000 7fbd2a5ee640 1 --2- 172.21.15.42:0/2781345496 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbd240a1710 0x7fbd24134760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:57.640 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.633+0000 7fbd2a5ee640 1 --2- 172.21.15.42:0/2781345496 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbd240a1330 0x7fbd241cbb30 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:05:57.640 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.633+0000 7fbd2a5ee640 1 -- 172.21.15.42:0/2781345496 >> 172.21.15.42:0/2781345496 conn(0x7fbd240a2420 msgr2=0x7fbd24137d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:05:57.640 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.633+0000 7fbd2a5ee640 1 -- 172.21.15.42:0/2781345496 shutdown_connections 2024-09-16T04:05:57.640 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.633+0000 7fbd2a5ee640 1 -- 172.21.15.42:0/2781345496 wait complete. 2024-09-16T04:05:57.641 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:05:57.633+0000 7fbd2a5ee640 1 librados: shutdown 2024-09-16T04:06:00.655 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:06:00.786 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.781+0000 7f2833ed9640 10 monclient: get_monmap_and_config 2024-09-16T04:06:00.786 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.781+0000 7f2833ed9640 10 monclient: build_initial_monmap 2024-09-16T04:06:00.786 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.781+0000 7f2833ed9640 10 monclient: monmap: 2024-09-16T04:06:00.786 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:06:00.786 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:06:00.786 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:06:00.784611+0000 2024-09-16T04:06:00.786 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:06:00.784611+0000 2024-09-16T04:06:00.786 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:06:00.787 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:06:00.787 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:06:00.787 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:06:00.787 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:06:00.787 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:06:00.787 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:06:00.787 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:06:00.789 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.781+0000 7f2833ed9640 1 Processor -- start 2024-09-16T04:06:00.789 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.781+0000 7f2833ed9640 1 -- start start 2024-09-16T04:06:00.789 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.781+0000 7f2833ed9640 10 monclient: init 2024-09-16T04:06:00.789 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.781+0000 7f2833ed9640 10 monclient: _reopen_session rank -1 2024-09-16T04:06:00.789 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.781+0000 7f2833ed9640 10 monclient: _add_conns ranks=[3,1,2,0,4] 2024-09-16T04:06:00.789 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.781+0000 7f2833ed9640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f282c0a1330 0x7f282c0a1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:06:00.789 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.781+0000 7f2833ed9640 10 monclient(hunting): picked mon.noname-a con 0x7f282c0a1330 addr v2:172.21.15.42:3300/0 2024-09-16T04:06:00.789 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.781+0000 7f2833ed9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f282c06dac0 addr v1:172.21.15.154:6789/0 2024-09-16T04:06:00.789 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.781+0000 7f2833ed9640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f282c0a1c50 0x7f282c135810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:06:00.789 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.781+0000 7f2833ed9640 10 monclient(hunting): picked mon.noname-b con 0x7f282c0a1c50 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:06:00.790 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.781+0000 7f2833ed9640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f282c09c090 con 0x7f282c06dac0 2024-09-16T04:06:00.790 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.781+0000 7f2833ed9640 10 monclient(hunting): start opening mon connection 2024-09-16T04:06:00.790 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.781+0000 7f2833ed9640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f282c09f610 con 0x7f282c0a1330 2024-09-16T04:06:00.790 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.781+0000 7f2833ed9640 10 monclient(hunting): start opening mon connection 2024-09-16T04:06:00.790 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.781+0000 7f2833ed9640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f282c09f8f0 con 0x7f282c0a1c50 2024-09-16T04:06:00.790 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.781+0000 7f2833ed9640 10 monclient(hunting): _renew_subs 2024-09-16T04:06:00.790 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.781+0000 7f2833ed9640 10 monclient(hunting): authenticate will time out at 2407.754077s 2024-09-16T04:06:00.790 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.781+0000 7f2832476640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f282c0a1c50 0x7f282c135810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:06:00.790 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.781+0000 7f2832476640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f282c0a1c50 0x7f282c135810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:58176/0 (socket says 172.21.15.42:58176) 2024-09-16T04:06:00.790 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.781+0000 7f2832476640 1 -- 172.21.15.42:0/4110182965 learned_addr learned my addr 172.21.15.42:0/4110182965 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:06:00.790 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.781+0000 7f2832476640 10 monclient(hunting): get_auth_request con 0x7f282c0a1c50 auth_method 0 2024-09-16T04:06:00.790 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.781+0000 7f2832476640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:06:00.790 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.781+0000 7f2832476640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:00.791 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.781+0000 7f2832476640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:00.791 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.781+0000 7f2831c75640 1 --2- 172.21.15.42:0/4110182965 >> v2:172.21.15.42:3300/0 conn(0x7f282c0a1330 0x7f282c0a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:06:00.791 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.781+0000 7f2831c75640 10 monclient(hunting): get_auth_request con 0x7f282c0a1330 auth_method 0 2024-09-16T04:06:00.791 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.781+0000 7f2831c75640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:06:00.791 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.781+0000 7f2831c75640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:00.791 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.781+0000 7f2831c75640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:00.791 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.781+0000 7f2832476640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:06:00.791 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.781+0000 7f2832476640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:06:00.791 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.781+0000 7f2832476640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:06:00.791 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.781+0000 7f2831c75640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:06:00.791 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.781+0000 7f2831c75640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:06:00.791 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.781+0000 7f2831c75640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:06:00.791 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.781+0000 7f2832476640 10 monclient(hunting): handle_auth_done global_id 1004099 payload 274 2024-09-16T04:06:00.791 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.781+0000 7f2832476640 1 -- 172.21.15.42:0/4110182965 >> v2:172.21.15.42:3300/0 conn(0x7f282c0a1330 msgr2=0x7f282c0a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:06:00.792 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.781+0000 7f2832476640 1 --2- 172.21.15.42:0/4110182965 >> v2:172.21.15.42:3300/0 conn(0x7f282c0a1330 0x7f282c0a1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:06:00.792 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.781+0000 7f2832476640 1 -- 172.21.15.42:0/4110182965 >> v1:172.21.15.154:6789/0 conn(0x7f282c06dac0 legacy=0x7f282c135100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:00.792 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.781+0000 7f2832476640 10 monclient: _finish_hunting 0 2024-09-16T04:06:00.792 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.781+0000 7f2832476640 1 monclient: found mon.noname-b 2024-09-16T04:06:00.792 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.781+0000 7f2832476640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:06:00.792 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.781+0000 7f2832476640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T04:06:00.792 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.781+0000 7f2832476640 1 -- 172.21.15.42:0/4110182965 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f282c09fbd0 con 0x7f282c0a1c50 2024-09-16T04:06:00.798 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.785+0000 7f2832476640 1 --2- 172.21.15.42:0/4110182965 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f282c0a1c50 0x7f282c135810 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f281c009870 tx=0x7f281c01c4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3af4868a2b88834f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:06:00.798 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.785+0000 7f2830c73640 1 -- 172.21.15.42:0/4110182965 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f281c0b1070 con 0x7f282c0a1c50 2024-09-16T04:06:00.798 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.785+0000 7f2830c73640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:00.798 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.785+0000 7f2830c73640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:06:00.798 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.785+0000 7f2830c73640 10 monclient: dump: 2024-09-16T04:06:00.799 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:06:00.799 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:06:00.799 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:06:00.799 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:06:00.799 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:00.799 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:06:00.799 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:06:00.799 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:06:00.799 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:06:00.799 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:06:00.799 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.785+0000 7f2830c73640 10 monclient: _finish_auth 0 2024-09-16T04:06:00.799 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.785+0000 7f2830c73640 10 monclient: _check_auth_tickets 2024-09-16T04:06:00.799 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.785+0000 7f2830c73640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:06:00.800 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.785+0000 7f2833ed9640 5 monclient: authenticate success, global_id 1004099 2024-09-16T04:06:00.800 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.785+0000 7f2833ed9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:06:00.800 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.785+0000 7f2830c73640 1 -- 172.21.15.42:0/4110182965 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f281c0ac620 con 0x7f282c0a1c50 2024-09-16T04:06:00.800 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.785+0000 7f2830c73640 10 monclient: handle_config config(0 keys) 2024-09-16T04:06:00.800 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.785+0000 7f2833ed9640 10 monclient: get_monmap_and_config success 2024-09-16T04:06:00.800 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.785+0000 7f2833ed9640 4 set_mon_vals no callback set 2024-09-16T04:06:00.800 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.785+0000 7f2830c73640 1 -- 172.21.15.42:0/4110182965 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f281c0ac9f0 con 0x7f282c0a1c50 2024-09-16T04:06:00.800 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.785+0000 7f2830c73640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:00.800 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.785+0000 7f2830c73640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:06:00.800 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.785+0000 7f2830c73640 10 monclient: dump: 2024-09-16T04:06:00.800 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:06:00.801 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:06:00.801 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:06:00.801 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:06:00.801 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:00.801 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:06:00.801 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:06:00.801 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:06:00.801 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:06:00.801 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:06:00.801 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.785+0000 7f2833ed9640 10 monclient: shutdown 2024-09-16T04:06:00.801 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.785+0000 7f2833ed9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:06:00.801 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.785+0000 7f2833ed9640 1 -- 172.21.15.42:0/4110182965 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f282c0a1c50 msgr2=0x7f282c135810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:00.801 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.785+0000 7f2833ed9640 1 --2- 172.21.15.42:0/4110182965 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f282c0a1c50 0x7f282c135810 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f281c009870 tx=0x7f281c01c4b0 comp rx=0 tx=0).stop 2024-09-16T04:06:00.802 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.785+0000 7f2833ed9640 1 -- 172.21.15.42:0/4110182965 shutdown_connections 2024-09-16T04:06:00.802 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.785+0000 7f2833ed9640 1 --2- 172.21.15.42:0/4110182965 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f282c0a1c50 0x7f282c135810 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:06:00.802 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.785+0000 7f2833ed9640 1 --2- 172.21.15.42:0/4110182965 >> v2:172.21.15.42:3300/0 conn(0x7f282c0a1330 0x7f282c0a1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:06:00.802 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.785+0000 7f2833ed9640 1 -- 172.21.15.42:0/4110182965 >> 172.21.15.42:0/4110182965 conn(0x7f282c0a2420 msgr2=0x7f282c0a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:06:00.802 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.785+0000 7f2833ed9640 1 -- 172.21.15.42:0/4110182965 shutdown_connections 2024-09-16T04:06:00.802 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.785+0000 7f2833ed9640 1 -- 172.21.15.42:0/4110182965 wait complete. 2024-09-16T04:06:00.802 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.785+0000 7f2833ed9640 10 monclient: build_initial_monmap 2024-09-16T04:06:00.802 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.785+0000 7f2833ed9640 10 monclient: monmap: 2024-09-16T04:06:00.802 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:06:00.802 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:06:00.802 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:06:00.802 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:06:00.802 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:06:00.802 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:06:00.803 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:06:00.803 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:06:00.803 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:06:00.803 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:06:00.803 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.785+0000 7f2833ed9640 1 librados: starting msgr at 2024-09-16T04:06:00.803 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.785+0000 7f2833ed9640 1 librados: starting objecter 2024-09-16T04:06:00.803 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.785+0000 7f2833ed9640 1 Processor -- start 2024-09-16T04:06:00.803 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.785+0000 7f2833ed9640 1 -- start start 2024-09-16T04:06:00.803 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.785+0000 7f2833ed9640 1 librados: setting wanted keys 2024-09-16T04:06:00.803 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.785+0000 7f2833ed9640 1 librados: calling monclient init 2024-09-16T04:06:00.803 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.785+0000 7f2833ed9640 10 monclient: init 2024-09-16T04:06:00.803 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.785+0000 7f2833ed9640 10 monclient: _reopen_session rank -1 2024-09-16T04:06:00.803 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.785+0000 7f2833ed9640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:06:00.803 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.785+0000 7f2833ed9640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f282c0a1330 0x7f282c13cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:06:00.804 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.785+0000 7f2833ed9640 10 monclient(hunting): picked mon.noname-b con 0x7f282c0a1330 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:06:00.804 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.785+0000 7f2833ed9640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f282c0a1c50 0x7f282c1350b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:06:00.804 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.785+0000 7f2833ed9640 10 monclient(hunting): picked mon.noname-a con 0x7f282c0a1c50 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:06:00.804 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.785+0000 7f2833ed9640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f282c1355f0 0x7f282c09cb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:06:00.804 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.785+0000 7f2833ed9640 10 monclient(hunting): picked mon.noname-c con 0x7f282c1355f0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:06:00.804 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.785+0000 7f2833ed9640 10 monclient(hunting): start opening mon connection 2024-09-16T04:06:00.804 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.785+0000 7f2833ed9640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f282c135d50 con 0x7f282c0a1c50 2024-09-16T04:06:00.804 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.785+0000 7f2833ed9640 10 monclient(hunting): start opening mon connection 2024-09-16T04:06:00.804 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.785+0000 7f2833ed9640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f282c09ef80 con 0x7f282c0a1330 2024-09-16T04:06:00.804 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.785+0000 7f2833ed9640 10 monclient(hunting): start opening mon connection 2024-09-16T04:06:00.804 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.785+0000 7f2833ed9640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f282c135ed0 con 0x7f282c1355f0 2024-09-16T04:06:00.804 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.785+0000 7f2833ed9640 10 monclient(hunting): _renew_subs 2024-09-16T04:06:00.804 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.785+0000 7f2833ed9640 10 monclient(hunting): authenticate will time out at 2407.759126s 2024-09-16T04:06:00.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.785+0000 7f2832476640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f282c1355f0 0x7f282c09cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:06:00.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.785+0000 7f2831474640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f282c0a1c50 0x7f282c1350b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:06:00.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.785+0000 7f2832476640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f282c1355f0 0x7f282c09cb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:58184/0 (socket says 172.21.15.42:58184) 2024-09-16T04:06:00.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.785+0000 7f2832476640 1 -- 172.21.15.42:0/4232448622 learned_addr learned my addr 172.21.15.42:0/4232448622 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:06:00.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.785+0000 7f2832476640 10 monclient(hunting): get_auth_request con 0x7f282c1355f0 auth_method 0 2024-09-16T04:06:00.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.785+0000 7f2832476640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:06:00.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.785+0000 7f2832476640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:00.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.785+0000 7f2832476640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:00.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.785+0000 7f2831474640 10 monclient(hunting): get_auth_request con 0x7f282c0a1c50 auth_method 0 2024-09-16T04:06:00.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.785+0000 7f2831474640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:06:00.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.785+0000 7f2831474640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:00.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.785+0000 7f2831474640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:00.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.785+0000 7f2832476640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:06:00.806 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.785+0000 7f2832476640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:06:00.806 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.785+0000 7f2832476640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:06:00.806 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.785+0000 7f2831474640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:06:00.806 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.785+0000 7f2831474640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:06:00.806 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.785+0000 7f2831474640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:06:00.806 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.785+0000 7f2831c75640 1 --2- 172.21.15.42:0/4232448622 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f282c0a1330 0x7f282c13cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:06:00.806 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.785+0000 7f2831c75640 10 monclient(hunting): get_auth_request con 0x7f282c0a1330 auth_method 0 2024-09-16T04:06:00.806 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.785+0000 7f2831c75640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:06:00.806 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.785+0000 7f2831c75640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:00.806 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.785+0000 7f2831c75640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:00.806 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.785+0000 7f2832476640 10 monclient(hunting): handle_auth_done global_id 1004102 payload 824 2024-09-16T04:06:00.806 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.785+0000 7f2832476640 1 -- 172.21.15.42:0/4232448622 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f282c0a1330 msgr2=0x7f282c13cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:00.806 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.785+0000 7f2832476640 1 --2- 172.21.15.42:0/4232448622 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f282c0a1330 0x7f282c13cc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:06:00.806 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.785+0000 7f2832476640 1 -- 172.21.15.42:0/4232448622 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f282c0a1c50 msgr2=0x7f282c1350b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:00.807 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.785+0000 7f2832476640 1 --2- 172.21.15.42:0/4232448622 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f282c0a1c50 0x7f282c1350b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:06:00.807 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.785+0000 7f2832476640 10 monclient: _finish_hunting 0 2024-09-16T04:06:00.807 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.785+0000 7f2832476640 1 monclient: found mon.noname-c 2024-09-16T04:06:00.807 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.785+0000 7f2832476640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:06:00.807 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.785+0000 7f2832476640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T04:06:00.807 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.785+0000 7f2832476640 1 -- 172.21.15.42:0/4232448622 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f282c071250 con 0x7f282c1355f0 2024-09-16T04:06:00.807 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.785+0000 7f2831474640 1 --2- 172.21.15.42:0/4232448622 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f282c0a1c50 0x7f282c1350b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:06:00.807 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.785+0000 7f2832476640 1 --2- 172.21.15.42:0/4232448622 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f282c1355f0 0x7f282c09cb60 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f281c0af040 tx=0x7f281c019250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:06:00.807 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.785+0000 7f28227fc640 10 client.?.objecter ms_handle_connect 0x7f282c1355f0 2024-09-16T04:06:00.807 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.785+0000 7f28227fc640 10 client.?.objecter resend_mon_ops 2024-09-16T04:06:00.807 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.789+0000 7f28227fc640 1 -- 172.21.15.42:0/4232448622 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f281c0bb020 con 0x7f282c1355f0 2024-09-16T04:06:00.807 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.789+0000 7f28227fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:00.807 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.789+0000 7f28227fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:06:00.808 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.789+0000 7f28227fc640 10 monclient: dump: 2024-09-16T04:06:00.808 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:06:00.808 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:06:00.808 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:06:00.808 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:06:00.808 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:00.808 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:06:00.808 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:06:00.808 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:06:00.808 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:06:00.808 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:06:00.808 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.789+0000 7f28227fc640 10 monclient: _finish_auth 0 2024-09-16T04:06:00.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.789+0000 7f28227fc640 10 monclient: _check_auth_tickets 2024-09-16T04:06:00.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.789+0000 7f28227fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:06:00.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.789+0000 7f28227fc640 1 -- 172.21.15.42:0/4232448622 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f281c0b7420 con 0x7f282c1355f0 2024-09-16T04:06:00.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.789+0000 7f2833ed9640 5 monclient: authenticate success, global_id 1004102 2024-09-16T04:06:00.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.789+0000 7f2833ed9640 10 monclient: _renew_subs 2024-09-16T04:06:00.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.789+0000 7f2833ed9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:06:00.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.789+0000 7f2833ed9640 1 -- 172.21.15.42:0/4232448622 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f282c06e6c0 con 0x7f282c1355f0 2024-09-16T04:06:00.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.789+0000 7f28227fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:06:00.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.789+0000 7f28227fc640 1 -- 172.21.15.42:0/4232448622 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f281c0b1040 con 0x7f282c1355f0 2024-09-16T04:06:00.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.789+0000 7f28227fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:00.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.789+0000 7f28227fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:06:00.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.789+0000 7f28227fc640 10 monclient: dump: 2024-09-16T04:06:00.809 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:06:00.810 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:06:00.810 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:06:00.810 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:06:00.810 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:00.810 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:06:00.810 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:06:00.810 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:06:00.810 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:06:00.810 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:06:00.810 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.789+0000 7f2823fff640 4 set_mon_vals no callback set 2024-09-16T04:06:00.810 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.789+0000 7f2833ed9640 10 client.1004102.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:06:00.810 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.789+0000 7f2833ed9640 10 monclient: _renew_subs 2024-09-16T04:06:00.810 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.789+0000 7f2833ed9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:06:00.810 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.789+0000 7f2833ed9640 1 -- 172.21.15.42:0/4232448622 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f282c120380 con 0x7f282c1355f0 2024-09-16T04:06:00.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.789+0000 7f2833ed9640 1 librados: init done 2024-09-16T04:06:00.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.789+0000 7f28227fc640 1 -- 172.21.15.42:0/4232448622 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f281c0be020 con 0x7f282c1355f0 2024-09-16T04:06:00.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.789+0000 7f2833ed9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:06:00.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.789+0000 7f2833ed9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:06:00.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.789+0000 7f2833ed9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:06:00.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.789+0000 7f2833ed9640 1 -- 172.21.15.42:0/4232448622 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f282c0a1710 con 0x7f282c1355f0 2024-09-16T04:06:00.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.789+0000 7f28227fc640 1 --2- 172.21.15.42:0/4232448622 >> v2:172.21.15.154:6812/254845342 conn(0x7f2810037810 0x7f2810039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:06:00.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.789+0000 7f28227fc640 1 -- 172.21.15.42:0/4232448622 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(706..706 src has 451..706) ==== 5040+0+0 (secure 0 0 0) 0x7f281c0e2f80 con 0x7f282c1355f0 2024-09-16T04:06:00.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.789+0000 7f28227fc640 10 client.1004102.objecter ms_dispatch 0x7f282c000ba0 osd_map(706..706 src has 451..706) 2024-09-16T04:06:00.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.789+0000 7f28227fc640 3 client.1004102.objecter handle_osd_map got epochs [706,706] > 0 2024-09-16T04:06:00.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.789+0000 7f28227fc640 3 client.1004102.objecter handle_osd_map decoding full epoch 706 2024-09-16T04:06:00.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.789+0000 7f28227fc640 20 client.1004102.objecter dump_active .. 0 homeless 2024-09-16T04:06:00.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.789+0000 7f2831c75640 1 --2- 172.21.15.42:0/4232448622 >> v2:172.21.15.154:6812/254845342 conn(0x7f2810037810 0x7f2810039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:06:00.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.789+0000 7f2831c75640 10 monclient: get_auth_request con 0x7f2810037810 auth_method 0 2024-09-16T04:06:00.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.789+0000 7f2831c75640 1 --2- 172.21.15.42:0/4232448622 >> v2:172.21.15.154:6812/254845342 conn(0x7f2810037810 0x7f2810039cd0 secure :-1 s=READY pgs=3857 cs=0 l=1 rev1=1 crypto rx=0x7f28240074a0 tx=0x7f282400b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:06:00.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.789+0000 7f28227fc640 10 client.1004102.objecter ms_handle_connect 0x7f2810037810 2024-09-16T04:06:00.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.793+0000 7f28227fc640 1 -- 172.21.15.42:0/4232448622 <== mon.2 v2:172.21.15.42:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f281c0c5030 con 0x7f282c1355f0 2024-09-16T04:06:00.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.793+0000 7f28227fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:06:00.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:00.793+0000 7f28227fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:06:01.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.021+0000 7f2833ed9640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:06:01.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.021+0000 7f2833ed9640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:06:01.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.021+0000 7f2833ed9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:06:01.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.021+0000 7f2833ed9640 1 -- 172.21.15.42:0/4232448622 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f282c069040 con 0x7f282c1355f0 2024-09-16T04:06:01.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.021+0000 7f28227fc640 1 -- 172.21.15.42:0/4232448622 <== mon.2 v2:172.21.15.42:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7f281c016130 con 0x7f282c1355f0 2024-09-16T04:06:01.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.021+0000 7f28227fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:06:01.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.021+0000 7f28227fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:06:01.028 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":222,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:06:01.030 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.025+0000 7f2833ed9640 10 librados: watch_flush enter 2024-09-16T04:06:01.030 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.025+0000 7f2833ed9640 10 librados: watch_flush exit 2024-09-16T04:06:01.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.025+0000 7f2833ed9640 20 client.1004102.objecter shutdown clearing up homeless session... 2024-09-16T04:06:01.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.025+0000 7f2833ed9640 10 client.1004102.objecter successfully canceled tick 2024-09-16T04:06:01.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.025+0000 7f2833ed9640 1 -- 172.21.15.42:0/4232448622 >> v2:172.21.15.154:6812/254845342 conn(0x7f2810037810 msgr2=0x7f2810039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:01.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.025+0000 7f2833ed9640 1 --2- 172.21.15.42:0/4232448622 >> v2:172.21.15.154:6812/254845342 conn(0x7f2810037810 0x7f2810039cd0 secure :-1 s=READY pgs=3857 cs=0 l=1 rev1=1 crypto rx=0x7f28240074a0 tx=0x7f282400b040 comp rx=0 tx=0).stop 2024-09-16T04:06:01.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.025+0000 7f2833ed9640 10 monclient: shutdown 2024-09-16T04:06:01.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.025+0000 7f2833ed9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:06:01.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.025+0000 7f2833ed9640 1 -- 172.21.15.42:0/4232448622 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f282c1355f0 msgr2=0x7f282c09cb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:01.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.025+0000 7f2833ed9640 1 --2- 172.21.15.42:0/4232448622 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f282c1355f0 0x7f282c09cb60 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f281c0af040 tx=0x7f281c019250 comp rx=0 tx=0).stop 2024-09-16T04:06:01.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.025+0000 7f2833ed9640 1 -- 172.21.15.42:0/4232448622 shutdown_connections 2024-09-16T04:06:01.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.025+0000 7f2833ed9640 1 --2- 172.21.15.42:0/4232448622 >> v2:172.21.15.154:6812/254845342 conn(0x7f2810037810 0x7f2810039cd0 unknown :-1 s=CLOSED pgs=3857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:06:01.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.025+0000 7f2833ed9640 1 --2- 172.21.15.42:0/4232448622 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f282c1355f0 0x7f282c09cb60 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:06:01.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.025+0000 7f2833ed9640 1 --2- 172.21.15.42:0/4232448622 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f282c0a1c50 0x7f282c1350b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:06:01.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.025+0000 7f2833ed9640 1 --2- 172.21.15.42:0/4232448622 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f282c0a1330 0x7f282c13cc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:06:01.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.025+0000 7f2833ed9640 1 -- 172.21.15.42:0/4232448622 >> 172.21.15.42:0/4232448622 conn(0x7f282c0a2420 msgr2=0x7f282c06f860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:06:01.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.025+0000 7f2833ed9640 1 -- 172.21.15.42:0/4232448622 shutdown_connections 2024-09-16T04:06:01.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.025+0000 7f2833ed9640 1 -- 172.21.15.42:0/4232448622 wait complete. 2024-09-16T04:06:01.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.025+0000 7f2833ed9640 1 librados: shutdown 2024-09-16T04:06:01.046 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T04:06:01.046 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T04:06:01.046 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T04:06:01.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.161+0000 7fa022ff2640 10 monclient: get_monmap_and_config 2024-09-16T04:06:01.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.161+0000 7fa022ff2640 10 monclient: build_initial_monmap 2024-09-16T04:06:01.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.161+0000 7fa022ff2640 10 monclient: monmap: 2024-09-16T04:06:01.169 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:06:01.169 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:06:01.170 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:06:01.167303+0000 2024-09-16T04:06:01.170 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:06:01.167303+0000 2024-09-16T04:06:01.170 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:06:01.170 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:06:01.170 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:06:01.170 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:06:01.170 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:06:01.170 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:06:01.170 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:06:01.170 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:06:01.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.161+0000 7fa022ff2640 1 Processor -- start 2024-09-16T04:06:01.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.161+0000 7fa022ff2640 1 -- start start 2024-09-16T04:06:01.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.161+0000 7fa022ff2640 10 monclient: init 2024-09-16T04:06:01.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.165+0000 7fa022ff2640 10 monclient: _reopen_session rank -1 2024-09-16T04:06:01.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.165+0000 7fa022ff2640 10 monclient: _add_conns ranks=[0,2,1,4,3] 2024-09-16T04:06:01.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.165+0000 7fa022ff2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa01c0a1330 addr v1:172.21.15.42:6789/0 2024-09-16T04:06:01.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.165+0000 7fa022ff2640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa01c0a1710 0x7fa01c0a1b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:06:01.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.165+0000 7fa022ff2640 10 monclient(hunting): picked mon.noname-b con 0x7fa01c0a1710 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:06:01.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.165+0000 7fa022ff2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa01c1393a0 addr v1:172.21.15.154:6789/0 2024-09-16T04:06:01.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.165+0000 7fa022ff2640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa01c09e4c0 con 0x7fa01c0a1330 2024-09-16T04:06:01.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.165+0000 7fa022ff2640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa01c0a2050 con 0x7fa01c1393a0 2024-09-16T04:06:01.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.165+0000 7fa022ff2640 10 monclient(hunting): start opening mon connection 2024-09-16T04:06:01.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.165+0000 7fa022ff2640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fa01c09edf0 con 0x7fa01c0a1710 2024-09-16T04:06:01.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.165+0000 7fa022ff2640 10 monclient(hunting): _renew_subs 2024-09-16T04:06:01.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.165+0000 7fa022ff2640 10 monclient(hunting): authenticate will time out at 2408.136564s 2024-09-16T04:06:01.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.165+0000 7fa01bfff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa01c0a1710 0x7fa01c0a1b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:06:01.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.165+0000 7fa020d8e640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fa01c0a1330 0x7fa01c09f180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:56032/0 (socket says 172.21.15.42:56032) 2024-09-16T04:06:01.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.165+0000 7fa020d8e640 1 -- 172.21.15.42:0/100513063 learned_addr learned my addr 172.21.15.42:0/100513063 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T04:06:01.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.165+0000 7fa01bfff640 10 monclient(hunting): get_auth_request con 0x7fa01c0a1710 auth_method 0 2024-09-16T04:06:01.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.165+0000 7fa01bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:06:01.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.165+0000 7fa01bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:01.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.165+0000 7fa01bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:01.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.165+0000 7fa01bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:06:01.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.165+0000 7fa01bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:06:01.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.165+0000 7fa01bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:06:01.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.165+0000 7fa01b7fe640 1 -- 172.21.15.42:0/100513063 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1450452265 0 0) 0x7fa01c09e4c0 con 0x7fa01c0a1330 2024-09-16T04:06:01.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.165+0000 7fa01b7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:06:01.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.165+0000 7fa01b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:01.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.165+0000 7fa01b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:01.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.165+0000 7fa01b7fe640 10 monclient(hunting): my global_id is 984366 2024-09-16T04:06:01.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.165+0000 7fa01b7fe640 1 -- 172.21.15.42:0/100513063 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa004003530 con 0x7fa01c0a1330 2024-09-16T04:06:01.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.165+0000 7fa01bfff640 10 monclient(hunting): handle_auth_done global_id 1004105 payload 274 2024-09-16T04:06:01.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.165+0000 7fa01bfff640 1 -- 172.21.15.42:0/100513063 >> v1:172.21.15.154:6789/0 conn(0x7fa01c1393a0 legacy=0x7fa01c13b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:01.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.165+0000 7fa01bfff640 1 -- 172.21.15.42:0/100513063 >> v1:172.21.15.42:6789/0 conn(0x7fa01c0a1330 legacy=0x7fa01c09f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:01.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.165+0000 7fa01bfff640 10 monclient: _finish_hunting 0 2024-09-16T04:06:01.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.165+0000 7fa01bfff640 1 monclient: found mon.noname-b 2024-09-16T04:06:01.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.165+0000 7fa01bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:06:01.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.165+0000 7fa01bfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T04:06:01.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.165+0000 7fa01bfff640 1 -- 172.21.15.42:0/100513063 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa01c0a3170 con 0x7fa01c0a1710 2024-09-16T04:06:01.190 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.169+0000 7fa01bfff640 1 --2- 172.21.15.42:0/100513063 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa01c0a1710 0x7fa01c0a1b10 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fa0100097f0 tx=0x7fa01001c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=791f66ed94c10da3 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:06:01.190 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.169+0000 7fa01b7fe640 1 -- 172.21.15.42:0/100513063 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0100b0070 con 0x7fa01c0a1710 2024-09-16T04:06:01.191 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.169+0000 7fa01b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:01.191 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.169+0000 7fa01b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:06:01.191 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.169+0000 7fa01b7fe640 10 monclient: dump: 2024-09-16T04:06:01.191 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:06:01.191 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:06:01.191 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:06:01.191 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:06:01.191 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:01.191 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:06:01.192 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:06:01.192 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:06:01.192 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:06:01.192 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:06:01.192 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.169+0000 7fa01b7fe640 10 monclient: _finish_auth 0 2024-09-16T04:06:01.192 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.169+0000 7fa01b7fe640 10 monclient: _check_auth_tickets 2024-09-16T04:06:01.192 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.169+0000 7fa01b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:06:01.192 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.169+0000 7fa022ff2640 5 monclient: authenticate success, global_id 1004105 2024-09-16T04:06:01.193 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.169+0000 7fa022ff2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:06:01.193 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.169+0000 7fa01b7fe640 1 -- 172.21.15.42:0/100513063 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa0100ab600 con 0x7fa01c0a1710 2024-09-16T04:06:01.193 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.169+0000 7fa01b7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:06:01.193 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.169+0000 7fa01b7fe640 1 -- 172.21.15.42:0/100513063 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0100ab9b0 con 0x7fa01c0a1710 2024-09-16T04:06:01.193 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.169+0000 7fa01b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:01.193 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.169+0000 7fa01b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:06:01.193 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.169+0000 7fa01b7fe640 10 monclient: dump: 2024-09-16T04:06:01.193 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:06:01.193 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:06:01.193 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:06:01.193 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:06:01.193 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:01.193 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:06:01.194 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:06:01.194 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:06:01.194 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:06:01.194 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:06:01.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.169+0000 7fa022ff2640 10 monclient: get_monmap_and_config success 2024-09-16T04:06:01.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.169+0000 7fa022ff2640 4 set_mon_vals no callback set 2024-09-16T04:06:01.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.169+0000 7fa022ff2640 10 monclient: shutdown 2024-09-16T04:06:01.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.169+0000 7fa022ff2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:06:01.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.169+0000 7fa022ff2640 1 -- 172.21.15.42:0/100513063 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa01c0a1710 msgr2=0x7fa01c0a1b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:01.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.169+0000 7fa022ff2640 1 --2- 172.21.15.42:0/100513063 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa01c0a1710 0x7fa01c0a1b10 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fa0100097f0 tx=0x7fa01001c3a0 comp rx=0 tx=0).stop 2024-09-16T04:06:01.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.169+0000 7fa022ff2640 1 -- 172.21.15.42:0/100513063 shutdown_connections 2024-09-16T04:06:01.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.169+0000 7fa022ff2640 1 --2- 172.21.15.42:0/100513063 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa01c0a1710 0x7fa01c0a1b10 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:06:01.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.169+0000 7fa022ff2640 1 -- 172.21.15.42:0/100513063 >> 172.21.15.42:0/100513063 conn(0x7fa01c0a2420 msgr2=0x7fa01c0a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:06:01.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.169+0000 7fa022ff2640 1 -- 172.21.15.42:0/100513063 shutdown_connections 2024-09-16T04:06:01.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.169+0000 7fa022ff2640 1 -- 172.21.15.42:0/100513063 wait complete. 2024-09-16T04:06:01.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.169+0000 7fa022ff2640 10 monclient: build_initial_monmap 2024-09-16T04:06:01.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.169+0000 7fa022ff2640 10 monclient: monmap: 2024-09-16T04:06:01.195 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:06:01.195 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:06:01.195 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:06:01.195 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:06:01.195 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:06:01.195 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:06:01.196 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:06:01.196 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:06:01.196 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:06:01.196 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:06:01.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.169+0000 7fa022ff2640 1 librados: starting msgr at 2024-09-16T04:06:01.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.169+0000 7fa022ff2640 1 librados: starting objecter 2024-09-16T04:06:01.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.169+0000 7fa022ff2640 1 Processor -- start 2024-09-16T04:06:01.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.169+0000 7fa022ff2640 1 -- start start 2024-09-16T04:06:01.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.169+0000 7fa022ff2640 1 librados: setting wanted keys 2024-09-16T04:06:01.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.169+0000 7fa022ff2640 1 librados: calling monclient init 2024-09-16T04:06:01.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.169+0000 7fa022ff2640 10 monclient: init 2024-09-16T04:06:01.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.169+0000 7fa022ff2640 10 monclient: _reopen_session rank -1 2024-09-16T04:06:01.197 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.169+0000 7fa022ff2640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:06:01.197 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.169+0000 7fa022ff2640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa01c0a1330 0x7fa01c133cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:06:01.197 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.169+0000 7fa022ff2640 10 monclient(hunting): picked mon.noname-c con 0x7fa01c0a1330 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:06:01.197 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.169+0000 7fa022ff2640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa01c0a1710 0x7fa01c134200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:06:01.197 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.169+0000 7fa022ff2640 10 monclient(hunting): picked mon.noname-b con 0x7fa01c0a1710 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:06:01.197 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.169+0000 7fa022ff2640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa01c1393a0 0x7fa01c11bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:06:01.197 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.169+0000 7fa022ff2640 10 monclient(hunting): picked mon.noname-a con 0x7fa01c1393a0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:06:01.197 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.169+0000 7fa022ff2640 10 monclient(hunting): start opening mon connection 2024-09-16T04:06:01.197 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.169+0000 7fa022ff2640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fa01c13f8e0 con 0x7fa01c1393a0 2024-09-16T04:06:01.198 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.169+0000 7fa022ff2640 10 monclient(hunting): start opening mon connection 2024-09-16T04:06:01.198 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.169+0000 7fa022ff2640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa01c09cd40 con 0x7fa01c0a1710 2024-09-16T04:06:01.198 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.169+0000 7fa022ff2640 10 monclient(hunting): start opening mon connection 2024-09-16T04:06:01.198 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.169+0000 7fa022ff2640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fa01c13fa60 con 0x7fa01c0a1330 2024-09-16T04:06:01.198 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.169+0000 7fa022ff2640 10 monclient(hunting): _renew_subs 2024-09-16T04:06:01.198 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.169+0000 7fa022ff2640 10 monclient(hunting): authenticate will time out at 2408.143192s 2024-09-16T04:06:01.198 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.169+0000 7fa020d8e640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa01c0a1330 0x7fa01c133cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:06:01.198 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.169+0000 7fa02158f640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa01c1393a0 0x7fa01c11bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:06:01.198 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.169+0000 7fa020d8e640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa01c0a1330 0x7fa01c133cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:58208/0 (socket says 172.21.15.42:58208) 2024-09-16T04:06:01.198 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.169+0000 7fa020d8e640 1 -- 172.21.15.42:0/3955050766 learned_addr learned my addr 172.21.15.42:0/3955050766 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:06:01.198 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.169+0000 7fa020d8e640 10 monclient(hunting): get_auth_request con 0x7fa01c0a1330 auth_method 0 2024-09-16T04:06:01.198 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.169+0000 7fa020d8e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:06:01.198 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.169+0000 7fa020d8e640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:01.199 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.169+0000 7fa020d8e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:01.199 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.169+0000 7fa02158f640 10 monclient(hunting): get_auth_request con 0x7fa01c1393a0 auth_method 0 2024-09-16T04:06:01.199 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.169+0000 7fa02158f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:06:01.199 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.169+0000 7fa02158f640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:01.199 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.169+0000 7fa02158f640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:01.199 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.169+0000 7fa02158f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:06:01.199 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.169+0000 7fa02158f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:06:01.199 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.169+0000 7fa02158f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:06:01.199 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.169+0000 7fa020d8e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:06:01.199 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.169+0000 7fa020d8e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:06:01.200 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.169+0000 7fa020d8e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:06:01.200 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.169+0000 7fa01bfff640 1 --2- 172.21.15.42:0/3955050766 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa01c0a1710 0x7fa01c134200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:06:01.200 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.169+0000 7fa01bfff640 10 monclient(hunting): get_auth_request con 0x7fa01c0a1710 auth_method 0 2024-09-16T04:06:01.200 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.169+0000 7fa01bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:06:01.200 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.169+0000 7fa01bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:01.200 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.169+0000 7fa01bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:01.200 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.169+0000 7fa020d8e640 10 monclient(hunting): handle_auth_done global_id 1004108 payload 824 2024-09-16T04:06:01.200 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.169+0000 7fa020d8e640 1 -- 172.21.15.42:0/3955050766 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa01c0a1710 msgr2=0x7fa01c134200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:01.200 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.169+0000 7fa020d8e640 1 --2- 172.21.15.42:0/3955050766 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa01c0a1710 0x7fa01c134200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:06:01.200 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.169+0000 7fa020d8e640 1 -- 172.21.15.42:0/3955050766 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa01c1393a0 msgr2=0x7fa01c11bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:01.201 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.169+0000 7fa020d8e640 1 --2- 172.21.15.42:0/3955050766 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa01c1393a0 0x7fa01c11bf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:06:01.201 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.169+0000 7fa020d8e640 10 monclient: _finish_hunting 0 2024-09-16T04:06:01.201 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.169+0000 7fa020d8e640 1 monclient: found mon.noname-c 2024-09-16T04:06:01.201 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.169+0000 7fa020d8e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:06:01.201 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.169+0000 7fa020d8e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T04:06:01.201 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.169+0000 7fa020d8e640 1 -- 172.21.15.42:0/3955050766 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa01c11ea00 con 0x7fa01c0a1330 2024-09-16T04:06:01.201 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.169+0000 7fa02158f640 1 --2- 172.21.15.42:0/3955050766 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa01c1393a0 0x7fa01c11bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:06:01.201 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.169+0000 7fa020d8e640 1 --2- 172.21.15.42:0/3955050766 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa01c0a1330 0x7fa01c133cc0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fa00c00c4d0 tx=0x7fa00c00c990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:06:01.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.169+0000 7fa0197fa640 10 client.?.objecter ms_handle_connect 0x7fa01c0a1330 2024-09-16T04:06:01.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.169+0000 7fa0197fa640 10 client.?.objecter resend_mon_ops 2024-09-16T04:06:01.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.173+0000 7fa0197fa640 1 -- 172.21.15.42:0/3955050766 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa00c017070 con 0x7fa01c0a1330 2024-09-16T04:06:01.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.173+0000 7fa0197fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:01.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.173+0000 7fa0197fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:06:01.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.173+0000 7fa0197fa640 10 monclient: dump: 2024-09-16T04:06:01.202 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:06:01.202 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:06:01.203 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:06:01.203 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:06:01.203 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:01.203 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:06:01.203 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:06:01.203 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:06:01.203 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:06:01.203 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:06:01.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.173+0000 7fa0197fa640 10 monclient: _finish_auth 0 2024-09-16T04:06:01.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.173+0000 7fa0197fa640 10 monclient: _check_auth_tickets 2024-09-16T04:06:01.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.173+0000 7fa0197fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:06:01.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.173+0000 7fa022ff2640 5 monclient: authenticate success, global_id 1004108 2024-09-16T04:06:01.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.173+0000 7fa0197fa640 1 -- 172.21.15.42:0/3955050766 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa00c005cf0 con 0x7fa01c0a1330 2024-09-16T04:06:01.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.173+0000 7fa022ff2640 10 monclient: _renew_subs 2024-09-16T04:06:01.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.173+0000 7fa022ff2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:06:01.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.173+0000 7fa022ff2640 1 -- 172.21.15.42:0/3955050766 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa01c0a4ea0 con 0x7fa01c0a1330 2024-09-16T04:06:01.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.173+0000 7fa0197fa640 10 monclient: handle_config config(0 keys) 2024-09-16T04:06:01.205 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.173+0000 7fa0197fa640 1 -- 172.21.15.42:0/3955050766 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa00c007270 con 0x7fa01c0a1330 2024-09-16T04:06:01.205 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.173+0000 7fa0197fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:01.205 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.173+0000 7fa0197fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:06:01.205 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.173+0000 7fa0197fa640 10 monclient: dump: 2024-09-16T04:06:01.205 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:06:01.205 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:06:01.205 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:06:01.205 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:06:01.205 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:01.205 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:06:01.205 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:06:01.206 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:06:01.206 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:06:01.206 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:06:01.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.173+0000 7fa022ff2640 10 client.1004108.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:06:01.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.173+0000 7fa022ff2640 10 monclient: _renew_subs 2024-09-16T04:06:01.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.173+0000 7fa01affd640 4 set_mon_vals no callback set 2024-09-16T04:06:01.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.173+0000 7fa022ff2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:06:01.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.173+0000 7fa022ff2640 1 -- 172.21.15.42:0/3955050766 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa01c073a30 con 0x7fa01c0a1330 2024-09-16T04:06:01.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.173+0000 7fa022ff2640 1 librados: init done 2024-09-16T04:06:01.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.173+0000 7fa020d8e640 1 -- 172.21.15.42:0/3955050766 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa01c0a1330 msgr2=0x7fa01c133cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-16T04:06:01.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.173+0000 7fa020d8e640 1 -- 172.21.15.42:0/3955050766 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa01c0a1330 msgr2=0x7fa01c133cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:06:01.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.173+0000 7fa020d8e640 1 --2- 172.21.15.42:0/3955050766 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa01c0a1330 0x7fa01c133cc0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fa00c00c4d0 tx=0x7fa00c00c990 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:06:01.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.173+0000 7fa020d8e640 1 --2- 172.21.15.42:0/3955050766 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa01c0a1330 0x7fa01c133cc0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fa00c00c4d0 tx=0x7fa00c00c990 comp rx=0 tx=0).stop 2024-09-16T04:06:01.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.173+0000 7fa0197fa640 10 monclient: ms_handle_reset current mon [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:06:01.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.173+0000 7fa0197fa640 10 monclient: _reopen_session rank -1 2024-09-16T04:06:01.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.173+0000 7fa0197fa640 1 -- 172.21.15.42:0/3955050766 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa01c0a1330 msgr2=0x7fa01c133cc0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:06:01.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.173+0000 7fa0197fa640 1 --2- 172.21.15.42:0/3955050766 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa01c0a1330 0x7fa01c133cc0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:06:01.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.173+0000 7fa0197fa640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:06:01.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.173+0000 7fa0197fa640 1 --2- 172.21.15.42:0/3955050766 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa01c0a1710 0x7fa008007210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:06:01.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.173+0000 7fa0197fa640 10 monclient(hunting): picked mon.b con 0x7fa01c0a1710 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:06:01.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.173+0000 7fa0197fa640 1 --2- 172.21.15.42:0/3955050766 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa008007be0 0x7fa00800a070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:06:01.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.173+0000 7fa0197fa640 10 monclient(hunting): picked mon.c con 0x7fa008007be0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:06:01.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.173+0000 7fa0197fa640 1 --2- 172.21.15.42:0/3955050766 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa01c1393a0 0x7fa00800cb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:06:01.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.173+0000 7fa0197fa640 10 monclient(hunting): picked mon.a con 0x7fa01c1393a0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:06:01.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.173+0000 7fa0197fa640 10 monclient(hunting): start opening mon connection 2024-09-16T04:06:01.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.173+0000 7fa0197fa640 1 -- 172.21.15.42:0/3955050766 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fa00800d560 con 0x7fa01c1393a0 2024-09-16T04:06:01.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.173+0000 7fa0197fa640 10 monclient(hunting): start opening mon connection 2024-09-16T04:06:01.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.173+0000 7fa0197fa640 1 -- 172.21.15.42:0/3955050766 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa00800d6e0 con 0x7fa01c0a1710 2024-09-16T04:06:01.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.173+0000 7fa0197fa640 10 monclient(hunting): start opening mon connection 2024-09-16T04:06:01.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.173+0000 7fa0197fa640 1 -- 172.21.15.42:0/3955050766 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fa00800d890 con 0x7fa008007be0 2024-09-16T04:06:01.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.173+0000 7fa0197fa640 10 monclient(hunting): _renew_subs 2024-09-16T04:06:01.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.173+0000 7fa022ff2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:06:01.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.173+0000 7fa022ff2640 1 --2- 172.21.15.42:0/3955050766 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa01c11e3a0 0x7fa01c133cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:06:01.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.173+0000 7fa022ff2640 10 monclient(hunting): start opening mon connection 2024-09-16T04:06:01.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.173+0000 7fa022ff2640 1 -- 172.21.15.42:0/3955050766 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fa01c0a1b10 con 0x7fa01c11e3a0 2024-09-16T04:06:01.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.173+0000 7fa02158f640 1 --2- 172.21.15.42:0/3955050766 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa01c1393a0 0x7fa00800cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:06:01.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.173+0000 7fa02158f640 10 monclient(hunting): get_auth_request con 0x7fa01c1393a0 auth_method 0 2024-09-16T04:06:01.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.173+0000 7fa02158f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:06:01.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.173+0000 7fa02158f640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:01.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.173+0000 7fa02158f640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:06:01.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.173+0000 7fa01bfff640 1 --2- 172.21.15.42:0/3955050766 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa008007be0 0x7fa00800a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:06:01.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.173+0000 7fa01bfff640 10 monclient(hunting): get_auth_request con 0x7fa008007be0 auth_method 0 2024-09-16T04:06:01.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.173+0000 7fa01bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:06:01.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.173+0000 7fa01bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:01.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.173+0000 7fa01bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:06:01.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.173+0000 7fa020d8e640 1 --2- 172.21.15.42:0/3955050766 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa01c11e3a0 0x7fa01c133cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:06:01.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.173+0000 7fa020d8e640 10 monclient(hunting): get_auth_request con 0x7fa01c11e3a0 auth_method 0 2024-09-16T04:06:01.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.173+0000 7fa020d8e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:06:01.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.173+0000 7fa020d8e640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:01.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.173+0000 7fa020d8e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:01.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.173+0000 7fa02158f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:06:01.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.173+0000 7fa02158f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:06:01.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.173+0000 7fa02158f640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:06:01.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.173+0000 7fa01bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:06:01.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.173+0000 7fa01bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:06:01.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.173+0000 7fa01bfff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:06:01.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.173+0000 7fa020d8e640 1 --2- 172.21.15.42:0/3955050766 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa01c0a1710 0x7fa008007210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:06:01.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.173+0000 7fa020d8e640 10 monclient(hunting): get_auth_request con 0x7fa01c0a1710 auth_method 0 2024-09-16T04:06:01.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.173+0000 7fa020d8e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:06:01.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.173+0000 7fa020d8e640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:01.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.173+0000 7fa020d8e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:06:01.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.173+0000 7fa02158f640 10 monclient(hunting): handle_auth_done global_id 1004108 payload 293 2024-09-16T04:06:01.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.173+0000 7fa02158f640 1 -- 172.21.15.42:0/3955050766 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa008007be0 msgr2=0x7fa00800a070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:01.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.173+0000 7fa02158f640 1 --2- 172.21.15.42:0/3955050766 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa008007be0 0x7fa00800a070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:06:01.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.173+0000 7fa02158f640 1 -- 172.21.15.42:0/3955050766 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa01c0a1710 msgr2=0x7fa008007210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:01.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.173+0000 7fa02158f640 1 --2- 172.21.15.42:0/3955050766 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa01c0a1710 0x7fa008007210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:06:01.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.173+0000 7fa02158f640 10 monclient: _finish_hunting 0 2024-09-16T04:06:01.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.173+0000 7fa02158f640 1 monclient: found mon.a 2024-09-16T04:06:01.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.173+0000 7fa02158f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:06:01.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.173+0000 7fa02158f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:06:01.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.173+0000 7fa02158f640 1 -- 172.21.15.42:0/3955050766 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+,osdmap=0}) -- 0x7fa00800dbc0 con 0x7fa01c1393a0 2024-09-16T04:06:01.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.173+0000 7fa02158f640 10 monclient: _finish_auth 0 2024-09-16T04:06:01.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.173+0000 7fa02158f640 10 monclient: _check_auth_tickets 2024-09-16T04:06:01.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.173+0000 7fa02158f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:06:01.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.173+0000 7fa020d8e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:06:01.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.173+0000 7fa020d8e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:06:01.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.173+0000 7fa020d8e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:06:01.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.173+0000 7fa02158f640 1 --2- 172.21.15.42:0/3955050766 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa01c1393a0 0x7fa00800cb50 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fa01c072380 tx=0x7fa00c007e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:06:01.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.173+0000 7fa0197fa640 10 client.1004108.objecter ms_handle_connect 0x7fa01c1393a0 2024-09-16T04:06:01.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.173+0000 7fa0197fa640 10 client.1004108.objecter resend_mon_ops 2024-09-16T04:06:01.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.173+0000 7fa020d8e640 10 monclient(hunting): handle_auth_done global_id 984372 payload 824 2024-09-16T04:06:01.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.173+0000 7fa020d8e640 1 -- 172.21.15.42:0/3955050766 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa01c11cff0 con 0x7fa01c11e3a0 2024-09-16T04:06:01.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.173+0000 7fa0197fa640 1 -- 172.21.15.42:0/3955050766 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa014001f40 con 0x7fa01c1393a0 2024-09-16T04:06:01.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.173+0000 7fa0197fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:01.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.173+0000 7fa0197fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:06:01.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.173+0000 7fa0197fa640 10 monclient: dump: 2024-09-16T04:06:01.214 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:06:01.214 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:06:01.214 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:06:01.214 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:06:01.214 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:01.214 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:06:01.214 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:06:01.214 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:06:01.214 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:06:01.214 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:06:01.214 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.173+0000 7fa0197fa640 1 -- 172.21.15.42:0/3955050766 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa014018070 con 0x7fa01c1393a0 2024-09-16T04:06:01.214 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.173+0000 7fa0197fa640 10 monclient: handle_config config(0 keys) 2024-09-16T04:06:01.214 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.173+0000 7fa01a7fc640 4 set_mon_vals no callback set 2024-09-16T04:06:01.214 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.173+0000 7fa0197fa640 10 client.1004108.objecter ms_handle_connect 0x7fa01c11e3a0 2024-09-16T04:06:01.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.173+0000 7fa0197fa640 10 client.1004108.objecter resend_mon_ops 2024-09-16T04:06:01.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.173+0000 7fa020d8e640 1 --2- 172.21.15.42:0/3955050766 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa01c11e3a0 0x7fa01c133cc0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fa00c007fd0 tx=0x7fa00c000ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:06:01.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.173+0000 7fa0197fa640 1 -- 172.21.15.42:0/3955050766 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fa014003980 con 0x7fa01c1393a0 2024-09-16T04:06:01.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.173+0000 7fa0197fa640 1 --2- 172.21.15.42:0/3955050766 >> v2:172.21.15.154:6812/254845342 conn(0x7fa01c0a1330 0x7fa008042060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:06:01.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.173+0000 7fa0197fa640 1 -- 172.21.15.42:0/3955050766 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa00c021060 con 0x7fa01c11e3a0 2024-09-16T04:06:01.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.173+0000 7fa0197fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:01.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.173+0000 7fa0197fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:06:01.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.173+0000 7fa0197fa640 10 monclient: dump: 2024-09-16T04:06:01.215 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:06:01.215 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:06:01.215 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:06:01.215 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:06:01.215 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:01.215 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:06:01.216 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:06:01.216 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:06:01.216 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:06:01.216 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:06:01.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.173+0000 7fa0197fa640 1 -- 172.21.15.42:0/3955050766 <== mon.0 v2:172.21.15.42:3300/0 4 ==== osd_map(706..706 src has 451..706) ==== 5040+0+0 (secure 0 0 0) 0x7fa014046830 con 0x7fa01c1393a0 2024-09-16T04:06:01.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.173+0000 7fa0197fa640 10 client.1004108.objecter ms_dispatch 0x7fa01c000ba0 osd_map(706..706 src has 451..706) 2024-09-16T04:06:01.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.173+0000 7fa0197fa640 3 client.1004108.objecter handle_osd_map got epochs [706,706] > 0 2024-09-16T04:06:01.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.173+0000 7fa0197fa640 3 client.1004108.objecter handle_osd_map decoding full epoch 706 2024-09-16T04:06:01.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.173+0000 7fa0197fa640 20 client.1004108.objecter dump_active .. 0 homeless 2024-09-16T04:06:01.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.173+0000 7fa0197fa640 1 -- 172.21.15.42:0/3955050766 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fa00c029680 con 0x7fa01c11e3a0 2024-09-16T04:06:01.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.173+0000 7fa0197fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:06:01.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.173+0000 7fa0197fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:06:01.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.173+0000 7fa0197fa640 1 -- 172.21.15.42:0/3955050766 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa01c11e3a0 msgr2=0x7fa01c133cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:01.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.173+0000 7fa0197fa640 1 --2- 172.21.15.42:0/3955050766 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa01c11e3a0 0x7fa01c133cc0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fa00c007fd0 tx=0x7fa00c000ee0 comp rx=0 tx=0).stop 2024-09-16T04:06:01.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.173+0000 7fa0197fa640 1 -- 172.21.15.42:0/3955050766 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa01c11e3a0 msgr2=0x7fa01c133cc0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:06:01.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.173+0000 7fa0197fa640 1 --2- 172.21.15.42:0/3955050766 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa01c11e3a0 0x7fa01c133cc0 secure :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fa00c007fd0 tx=0x7fa00c000ee0 comp rx=0 tx=0).stop 2024-09-16T04:06:01.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.173+0000 7fa01bfff640 1 --2- 172.21.15.42:0/3955050766 >> v2:172.21.15.154:6812/254845342 conn(0x7fa01c0a1330 0x7fa008042060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:06:01.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.173+0000 7fa01bfff640 10 monclient: get_auth_request con 0x7fa01c0a1330 auth_method 0 2024-09-16T04:06:01.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.173+0000 7fa01bfff640 1 --2- 172.21.15.42:0/3955050766 >> v2:172.21.15.154:6812/254845342 conn(0x7fa01c0a1330 0x7fa008042060 secure :-1 s=READY pgs=3858 cs=0 l=1 rev1=1 crypto rx=0x7fa0100ad000 tx=0x7fa0100a6000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:06:01.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.173+0000 7fa0197fa640 10 client.1004108.objecter ms_handle_connect 0x7fa01c0a1330 2024-09-16T04:06:01.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.181+0000 7fa022ff2640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:06:01.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.181+0000 7fa022ff2640 1 --2- 172.21.15.42:0/3955050766 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa01c134200 0x7fa01c11bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:06:01.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.181+0000 7fa022ff2640 10 monclient(hunting): start opening mon connection 2024-09-16T04:06:01.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.181+0000 7fa022ff2640 1 -- 172.21.15.42:0/3955050766 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fa01c11e780 con 0x7fa01c134200 2024-09-16T04:06:01.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.181+0000 7fa020d8e640 1 --2- 172.21.15.42:0/3955050766 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa01c134200 0x7fa01c11bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:06:01.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.181+0000 7fa020d8e640 10 monclient: get_auth_request con 0x7fa01c134200 auth_method 0 2024-09-16T04:06:01.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.181+0000 7fa020d8e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:06:01.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.181+0000 7fa020d8e640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:01.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.181+0000 7fa020d8e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:01.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.181+0000 7fa020d8e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:06:01.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.181+0000 7fa020d8e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:06:01.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.181+0000 7fa020d8e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:06:01.220 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.181+0000 7fa020d8e640 10 monclient(hunting): handle_auth_done global_id 984375 payload 824 2024-09-16T04:06:01.220 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.181+0000 7fa020d8e640 1 -- 172.21.15.42:0/3955050766 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fa01c069040 con 0x7fa01c134200 2024-09-16T04:06:01.220 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.181+0000 7fa0197fa640 10 client.1004108.objecter ms_handle_connect 0x7fa01c134200 2024-09-16T04:06:01.220 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.181+0000 7fa0197fa640 10 client.1004108.objecter resend_mon_ops 2024-09-16T04:06:01.220 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.181+0000 7fa020d8e640 1 --2- 172.21.15.42:0/3955050766 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa01c134200 0x7fa01c11bf70 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fa00c00c4d0 tx=0x7fa00c01a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:06:01.220 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.181+0000 7fa0197fa640 1 -- 172.21.15.42:0/3955050766 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa00c029ea0 con 0x7fa01c134200 2024-09-16T04:06:01.220 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.181+0000 7fa0197fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:01.220 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.181+0000 7fa0197fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:06:01.220 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.181+0000 7fa0197fa640 10 monclient: dump: 2024-09-16T04:06:01.220 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:06:01.220 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:06:01.221 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:06:01.221 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:06:01.221 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:01.221 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:06:01.221 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:06:01.221 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:06:01.221 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:06:01.222 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:06:01.222 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.181+0000 7fa0197fa640 1 -- 172.21.15.42:0/3955050766 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4715 (secure 0 0 0) 0x7fa00c01c440 con 0x7fa01c134200 2024-09-16T04:06:01.222 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.181+0000 7fa0197fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:06:01.222 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.181+0000 7fa0197fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:06:01.222 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.181+0000 7fa0197fa640 1 -- 172.21.15.42:0/3955050766 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa01c134200 msgr2=0x7fa01c11bf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:01.222 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.181+0000 7fa0197fa640 1 --2- 172.21.15.42:0/3955050766 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa01c134200 0x7fa01c11bf70 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fa00c00c4d0 tx=0x7fa00c01a000 comp rx=0 tx=0).stop 2024-09-16T04:06:01.222 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.181+0000 7fa0197fa640 1 -- 172.21.15.42:0/3955050766 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa01c134200 msgr2=0x7fa01c11bf70 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:06:01.223 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.181+0000 7fa0197fa640 1 --2- 172.21.15.42:0/3955050766 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa01c134200 0x7fa01c11bf70 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:06:01.223 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T04:06:01.223 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:06:01.223 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:06:01.224 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "leader", 2024-09-16T04:06:01.224 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 222, 2024-09-16T04:06:01.224 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 121238, 2024-09-16T04:06:01.224 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T04:06:01.224 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T04:06:01.224 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T04:06:01.224 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T04:06:01.224 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:06:01.224 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 2, 2024-09-16T04:06:01.224 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:06:01.224 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T04:06:01.225 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T04:06:01.225 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:06:01.225 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:06:01.225 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:06:01.225 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:06:01.225 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:06:01.225 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:06:01.225 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:06:01.225 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:06:01.226 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:06:01.226 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:06:01.226 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:06:01.226 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:06:01.226 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:06:01.226 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T04:06:01.226 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:06:01.226 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:06:01.226 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:06:01.226 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:06:01.226 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:06:01.226 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:06:01.227 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:06:01.227 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:06:01.227 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:06:01.227 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:06:01.227 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:06:01.227 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:06:01.227 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:06:01.227 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T04:06:01.227 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T04:06:01.227 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T04:06:01.227 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T04:06:01.227 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T04:06:01.227 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T04:06:01.227 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:06:01.228 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:06:01.228 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T04:06:01.228 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T04:06:01.228 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T04:06:01.228 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T04:06:01.228 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T04:06:01.228 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T04:06:01.228 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T04:06:01.228 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:06:01.228 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T04:06:01.228 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:06:01.228 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:06:01.229 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:06:01.229 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:06:01.229 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:06:01.229 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:06:01.229 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:06:01.229 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:06:01.229 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:06:01.229 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:06:01.229 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:06:01.229 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:06:01.229 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T04:06:01.229 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:06:01.230 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T04:06:01.230 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:06:01.230 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:06:01.230 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:06:01.230 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:06:01.230 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:06:01.230 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:06:01.230 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:06:01.230 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T04:06:01.230 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:06:01.230 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:06:01.230 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:06:01.230 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:06:01.230 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T04:06:01.230 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:06:01.231 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:06:01.231 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:06:01.231 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:06:01.231 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T04:06:01.231 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T04:06:01.231 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:06:01.231 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:06:01.231 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:06:01.231 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:06:01.231 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:06:01.231 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:06:01.231 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:06:01.231 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:06:01.231 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:06:01.231 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:06:01.232 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:06:01.232 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T04:06:01.232 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:06:01.232 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:06:01.232 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:06:01.232 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:06:01.232 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T04:06:01.232 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:06:01.232 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:06:01.232 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:06:01.232 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:06:01.233 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T04:06:01.233 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T04:06:01.233 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:06:01.233 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:06:01.233 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:06:01.233 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:06:01.233 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:06:01.233 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:06:01.233 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:06:01.233 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:06:01.234 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:06:01.234 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:06:01.234 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:06:01.234 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T04:06:01.234 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:06:01.234 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:06:01.234 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:06:01.234 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:06:01.234 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T04:06:01.234 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:06:01.234 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:06:01.235 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:06:01.235 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:06:01.235 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T04:06:01.235 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T04:06:01.235 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:06:01.235 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:06:01.235 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:06:01.235 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:06:01.235 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:06:01.235 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:06:01.235 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T04:06:01.235 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T04:06:01.235 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:06:01.236 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:06:01.236 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:06:01.236 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:06:01.236 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:06:01.236 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:06:01.236 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T04:06:01.236 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:06:01.236 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:06:01.236 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:06:01.236 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 3 2024-09-16T04:06:01.236 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:06:01.236 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:06:01.236 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T04:06:01.237 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:06:01.237 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:06:01.237 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:06:01.237 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 3 2024-09-16T04:06:01.237 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:06:01.237 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:06:01.237 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:06:01.237 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T04:06:01.237 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T04:06:01.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.181+0000 7fa022ff2640 10 librados: watch_flush enter 2024-09-16T04:06:01.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.181+0000 7fa022ff2640 10 librados: watch_flush exit 2024-09-16T04:06:01.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.181+0000 7fa022ff2640 20 client.1004108.objecter shutdown clearing up homeless session... 2024-09-16T04:06:01.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.181+0000 7fa022ff2640 10 client.1004108.objecter successfully canceled tick 2024-09-16T04:06:01.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.181+0000 7fa022ff2640 1 -- 172.21.15.42:0/3955050766 >> v2:172.21.15.154:6812/254845342 conn(0x7fa01c0a1330 msgr2=0x7fa008042060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:01.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.181+0000 7fa022ff2640 1 --2- 172.21.15.42:0/3955050766 >> v2:172.21.15.154:6812/254845342 conn(0x7fa01c0a1330 0x7fa008042060 secure :-1 s=READY pgs=3858 cs=0 l=1 rev1=1 crypto rx=0x7fa0100ad000 tx=0x7fa0100a6000 comp rx=0 tx=0).stop 2024-09-16T04:06:01.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.181+0000 7fa022ff2640 10 monclient: shutdown 2024-09-16T04:06:01.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.181+0000 7fa022ff2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:06:01.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.181+0000 7fa022ff2640 1 -- 172.21.15.42:0/3955050766 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa01c1393a0 msgr2=0x7fa00800cb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:01.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.181+0000 7fa022ff2640 1 --2- 172.21.15.42:0/3955050766 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa01c1393a0 0x7fa00800cb50 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fa01c072380 tx=0x7fa00c007e00 comp rx=0 tx=0).stop 2024-09-16T04:06:01.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.181+0000 7fa02158f640 1 -- 172.21.15.42:0/3955050766 reap_dead start 2024-09-16T04:06:01.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.181+0000 7fa02158f640 1 -- 172.21.15.42:0/3955050766 reap_dead start 2024-09-16T04:06:01.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.181+0000 7fa022ff2640 1 -- 172.21.15.42:0/3955050766 shutdown_connections 2024-09-16T04:06:01.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.181+0000 7fa022ff2640 1 -- 172.21.15.42:0/3955050766 >> 172.21.15.42:0/3955050766 conn(0x7fa01c0a2420 msgr2=0x7fa01c09ba50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:06:01.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.181+0000 7fa022ff2640 1 -- 172.21.15.42:0/3955050766 shutdown_connections 2024-09-16T04:06:01.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.181+0000 7fa022ff2640 1 -- 172.21.15.42:0/3955050766 wait complete. 2024-09-16T04:06:01.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.181+0000 7fa022ff2640 1 librados: shutdown 2024-09-16T04:06:01.241 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:06:01.241 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T04:06:01.241 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T04:06:01.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.349+0000 7fa57796d640 10 monclient: get_monmap_and_config 2024-09-16T04:06:01.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.349+0000 7fa57796d640 10 monclient: build_initial_monmap 2024-09-16T04:06:01.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.349+0000 7fa57796d640 10 monclient: monmap: 2024-09-16T04:06:01.357 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:06:01.357 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:06:01.358 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:06:01.355626+0000 2024-09-16T04:06:01.358 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:06:01.355626+0000 2024-09-16T04:06:01.358 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:06:01.358 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:06:01.358 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:06:01.358 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:06:01.358 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:06:01.358 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:06:01.358 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:06:01.358 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:06:01.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.353+0000 7fa57796d640 1 Processor -- start 2024-09-16T04:06:01.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.353+0000 7fa57796d640 1 -- start start 2024-09-16T04:06:01.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.353+0000 7fa57796d640 10 monclient: init 2024-09-16T04:06:01.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.353+0000 7fa57796d640 10 monclient: _reopen_session rank -1 2024-09-16T04:06:01.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.353+0000 7fa57796d640 10 monclient: _add_conns ranks=[4,1,0,3,2] 2024-09-16T04:06:01.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.353+0000 7fa57796d640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fa5700a1330 0x7fa5700a1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:06:01.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.353+0000 7fa57796d640 10 monclient(hunting): picked mon.noname-c con 0x7fa5700a1330 addr v2:172.21.15.154:3300/0 2024-09-16T04:06:01.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.353+0000 7fa57796d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa57006dac0 addr v1:172.21.15.154:6789/0 2024-09-16T04:06:01.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.353+0000 7fa57796d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa5700a1c50 addr v1:172.21.15.42:6789/0 2024-09-16T04:06:01.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.353+0000 7fa57796d640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa570135f20 con 0x7fa5700a1c50 2024-09-16T04:06:01.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.353+0000 7fa57796d640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa57009c090 con 0x7fa57006dac0 2024-09-16T04:06:01.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.353+0000 7fa57796d640 10 monclient(hunting): start opening mon connection 2024-09-16T04:06:01.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.353+0000 7fa57796d640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fa57009f610 con 0x7fa5700a1330 2024-09-16T04:06:01.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.353+0000 7fa57796d640 10 monclient(hunting): _renew_subs 2024-09-16T04:06:01.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.353+0000 7fa57796d640 10 monclient(hunting): authenticate will time out at 2408.325087s 2024-09-16T04:06:01.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.353+0000 7fa575f0a640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fa5700a1c50 0x7fa570135810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:56042/0 (socket says 172.21.15.42:56042) 2024-09-16T04:06:01.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.353+0000 7fa575f0a640 1 -- 172.21.15.42:0/473179314 learned_addr learned my addr 172.21.15.42:0/473179314 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T04:06:01.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.353+0000 7fa575709640 1 --2- 172.21.15.42:0/473179314 >> v2:172.21.15.154:3300/0 conn(0x7fa5700a1330 0x7fa5700a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:06:01.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.353+0000 7fa575709640 10 monclient(hunting): get_auth_request con 0x7fa5700a1330 auth_method 0 2024-09-16T04:06:01.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.353+0000 7fa575709640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:06:01.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.353+0000 7fa575709640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:01.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.353+0000 7fa575709640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:01.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.353+0000 7fa567fff640 1 -- 172.21.15.42:0/473179314 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1386655022 0 0) 0x7fa570135f20 con 0x7fa5700a1c50 2024-09-16T04:06:01.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.353+0000 7fa567fff640 10 monclient(hunting): _negotiate 2024-09-16T04:06:01.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.353+0000 7fa567fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:01.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.353+0000 7fa567fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:01.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.353+0000 7fa567fff640 10 monclient(hunting): my global_id is 984378 2024-09-16T04:06:01.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.353+0000 7fa567fff640 1 -- 172.21.15.42:0/473179314 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa55c003530 con 0x7fa5700a1c50 2024-09-16T04:06:01.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.353+0000 7fa575709640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:06:01.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.353+0000 7fa575709640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:06:01.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.353+0000 7fa575709640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:06:01.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.353+0000 7fa567fff640 1 -- 172.21.15.42:0/473179314 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1332707273 0 0) 0x7fa55c003530 con 0x7fa5700a1c50 2024-09-16T04:06:01.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.353+0000 7fa567fff640 1 -- 172.21.15.42:0/473179314 >> v2:172.21.15.154:3300/0 conn(0x7fa5700a1330 msgr2=0x7fa5700a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:06:01.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.353+0000 7fa567fff640 1 --2- 172.21.15.42:0/473179314 >> v2:172.21.15.154:3300/0 conn(0x7fa5700a1330 0x7fa5700a1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:06:01.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.353+0000 7fa567fff640 1 -- 172.21.15.42:0/473179314 >> v1:172.21.15.154:6789/0 conn(0x7fa57006dac0 legacy=0x7fa570135100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:01.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.353+0000 7fa567fff640 10 monclient: _finish_hunting 0 2024-09-16T04:06:01.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.353+0000 7fa567fff640 1 monclient: found mon.noname-a-legacy 2024-09-16T04:06:01.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.353+0000 7fa567fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:06:01.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.353+0000 7fa567fff640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.42:6789/0 2024-09-16T04:06:01.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.353+0000 7fa567fff640 1 -- 172.21.15.42:0/473179314 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa57009f8f0 con 0x7fa5700a1c50 2024-09-16T04:06:01.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.353+0000 7fa567fff640 10 monclient: _finish_auth 0 2024-09-16T04:06:01.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.353+0000 7fa567fff640 10 monclient: _check_auth_tickets 2024-09-16T04:06:01.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.353+0000 7fa567fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:06:01.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.353+0000 7fa567fff640 1 -- 172.21.15.42:0/473179314 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2177413734 0 0) 0x7fa568001680 con 0x7fa5700a1c50 2024-09-16T04:06:01.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.353+0000 7fa57796d640 5 monclient: authenticate success, global_id 984378 2024-09-16T04:06:01.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.353+0000 7fa57796d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:06:01.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.353+0000 7fa567fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:01.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.353+0000 7fa567fff640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T04:06:01.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.353+0000 7fa567fff640 10 monclient: dump: 2024-09-16T04:06:01.374 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:06:01.374 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:06:01.374 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:06:01.374 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:06:01.375 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:01.375 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:06:01.375 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:06:01.375 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:06:01.375 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:06:01.375 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:06:01.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.353+0000 7fa567fff640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] but i'm connected to v1:172.21.15.42:6789/0, reconnecting 2024-09-16T04:06:01.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.353+0000 7fa567fff640 10 monclient: _reopen_session rank -1 2024-09-16T04:06:01.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.353+0000 7fa567fff640 1 -- 172.21.15.42:0/473179314 >> v1:172.21.15.42:6789/0 conn(0x7fa5700a1c50 legacy=0x7fa570135810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:01.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.353+0000 7fa567fff640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:06:01.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.353+0000 7fa567fff640 1 --2- 172.21.15.42:0/473179314 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa55c0039c0 0x7fa55c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:06:01.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.353+0000 7fa567fff640 10 monclient(hunting): picked mon.a con 0x7fa55c0039c0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:06:01.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.353+0000 7fa567fff640 1 --2- 172.21.15.42:0/473179314 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa55c004810 0x7fa55c004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:06:01.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.353+0000 7fa567fff640 10 monclient(hunting): picked mon.c con 0x7fa55c004810 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:06:01.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.353+0000 7fa567fff640 1 --2- 172.21.15.42:0/473179314 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa55c0051d0 0x7fa55c00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:06:01.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.353+0000 7fa567fff640 10 monclient(hunting): picked mon.b con 0x7fa55c0051d0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:06:01.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.353+0000 7fa567fff640 10 monclient(hunting): start opening mon connection 2024-09-16T04:06:01.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.353+0000 7fa567fff640 1 -- 172.21.15.42:0/473179314 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fa57009f610 con 0x7fa55c0039c0 2024-09-16T04:06:01.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.353+0000 7fa567fff640 10 monclient(hunting): start opening mon connection 2024-09-16T04:06:01.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.353+0000 7fa567fff640 1 -- 172.21.15.42:0/473179314 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa55c00c170 con 0x7fa55c0051d0 2024-09-16T04:06:01.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.353+0000 7fa567fff640 10 monclient(hunting): start opening mon connection 2024-09-16T04:06:01.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.353+0000 7fa567fff640 1 -- 172.21.15.42:0/473179314 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fa55c00c370 con 0x7fa55c004810 2024-09-16T04:06:01.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.353+0000 7fa567fff640 10 monclient(hunting): _renew_subs 2024-09-16T04:06:01.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.353+0000 7fa567fff640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:06:01.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.353+0000 7fa57796d640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-16T04:06:01.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.353+0000 7fa575709640 1 --2- 172.21.15.42:0/473179314 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa55c0039c0 0x7fa55c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:06:01.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.353+0000 7fa575709640 10 monclient(hunting): get_auth_request con 0x7fa55c0039c0 auth_method 0 2024-09-16T04:06:01.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.353+0000 7fa575709640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:06:01.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.353+0000 7fa575709640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:01.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.353+0000 7fa575709640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:06:01.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.353+0000 7fa574f08640 1 --2- 172.21.15.42:0/473179314 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa55c004810 0x7fa55c004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:06:01.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.353+0000 7fa574f08640 10 monclient(hunting): get_auth_request con 0x7fa55c004810 auth_method 0 2024-09-16T04:06:01.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.353+0000 7fa574f08640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:06:01.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.353+0000 7fa574f08640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:01.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.353+0000 7fa574f08640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:06:01.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.353+0000 7fa575709640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:06:01.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.353+0000 7fa575709640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:06:01.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.353+0000 7fa575709640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:06:01.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.353+0000 7fa574f08640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:06:01.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.353+0000 7fa574f08640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:06:01.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.353+0000 7fa574f08640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:06:01.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.353+0000 7fa575f0a640 1 --2- 172.21.15.42:0/473179314 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa55c0051d0 0x7fa55c00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:06:01.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.353+0000 7fa575f0a640 10 monclient(hunting): get_auth_request con 0x7fa55c0051d0 auth_method 0 2024-09-16T04:06:01.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.353+0000 7fa575f0a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:06:01.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.353+0000 7fa575f0a640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:01.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.353+0000 7fa575f0a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:06:01.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.353+0000 7fa575709640 10 monclient(hunting): handle_auth_done global_id 984378 payload 293 2024-09-16T04:06:01.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.353+0000 7fa575709640 1 -- 172.21.15.42:0/473179314 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa55c004810 msgr2=0x7fa55c004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:06:01.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.353+0000 7fa575709640 1 --2- 172.21.15.42:0/473179314 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa55c004810 0x7fa55c004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:06:01.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.353+0000 7fa575709640 1 -- 172.21.15.42:0/473179314 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa55c0051d0 msgr2=0x7fa55c00b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:06:01.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.353+0000 7fa575709640 1 --2- 172.21.15.42:0/473179314 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa55c0051d0 0x7fa55c00b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:06:01.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.353+0000 7fa575709640 10 monclient: _finish_hunting 0 2024-09-16T04:06:01.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.353+0000 7fa575709640 1 monclient: found mon.a 2024-09-16T04:06:01.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.353+0000 7fa575709640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:06:01.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.353+0000 7fa575709640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:06:01.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.353+0000 7fa575f0a640 1 -- 172.21.15.42:0/473179314 reap_dead start 2024-09-16T04:06:01.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.353+0000 7fa575709640 1 -- 172.21.15.42:0/473179314 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa55c003530 con 0x7fa55c0039c0 2024-09-16T04:06:01.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.353+0000 7fa575709640 10 monclient: _finish_auth 0 2024-09-16T04:06:01.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.353+0000 7fa575709640 10 monclient: _check_auth_tickets 2024-09-16T04:06:01.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.353+0000 7fa575709640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:06:01.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.357+0000 7fa575709640 1 --2- 172.21.15.42:0/473179314 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa55c0039c0 0x7fa55c003de0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fa560009ae0 tx=0x7fa56001ca20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c38fac070cab466 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:06:01.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.357+0000 7fa567fff640 1 -- 172.21.15.42:0/473179314 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5600b2070 con 0x7fa55c0039c0 2024-09-16T04:06:01.380 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.357+0000 7fa567fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:01.380 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.357+0000 7fa567fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:06:01.380 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.357+0000 7fa567fff640 10 monclient: dump: 2024-09-16T04:06:01.380 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:06:01.380 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:06:01.380 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:06:01.380 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:06:01.380 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:01.380 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:06:01.380 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:06:01.380 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:06:01.380 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:06:01.381 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:06:01.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.357+0000 7fa567fff640 1 -- 172.21.15.42:0/473179314 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa5600a5570 con 0x7fa55c0039c0 2024-09-16T04:06:01.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.357+0000 7fa567fff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:06:01.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.357+0000 7fa57796d640 10 monclient: get_monmap_and_config success 2024-09-16T04:06:01.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.357+0000 7fa567fff640 1 -- 172.21.15.42:0/473179314 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5600096b0 con 0x7fa55c0039c0 2024-09-16T04:06:01.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.357+0000 7fa567fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:01.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.357+0000 7fa57796d640 4 set_mon_vals no callback set 2024-09-16T04:06:01.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.357+0000 7fa567fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:06:01.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.357+0000 7fa567fff640 10 monclient: dump: 2024-09-16T04:06:01.384 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:06:01.384 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:06:01.384 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:06:01.384 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:06:01.384 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:01.385 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:06:01.385 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:06:01.385 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:06:01.385 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:06:01.385 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:06:01.385 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.357+0000 7fa57796d640 10 monclient: shutdown 2024-09-16T04:06:01.385 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.357+0000 7fa57796d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:06:01.385 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.357+0000 7fa57796d640 1 -- 172.21.15.42:0/473179314 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa55c0039c0 msgr2=0x7fa55c003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:01.385 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.357+0000 7fa57796d640 1 --2- 172.21.15.42:0/473179314 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa55c0039c0 0x7fa55c003de0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fa560009ae0 tx=0x7fa56001ca20 comp rx=0 tx=0).stop 2024-09-16T04:06:01.385 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.357+0000 7fa57796d640 1 -- 172.21.15.42:0/473179314 shutdown_connections 2024-09-16T04:06:01.385 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.357+0000 7fa57796d640 1 --2- 172.21.15.42:0/473179314 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa55c0039c0 0x7fa55c003de0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:06:01.385 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.357+0000 7fa57796d640 1 -- 172.21.15.42:0/473179314 >> 172.21.15.42:0/473179314 conn(0x7fa5700a2420 msgr2=0x7fa5700a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:06:01.386 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.357+0000 7fa57796d640 1 -- 172.21.15.42:0/473179314 shutdown_connections 2024-09-16T04:06:01.386 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.357+0000 7fa57796d640 1 -- 172.21.15.42:0/473179314 wait complete. 2024-09-16T04:06:01.386 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.357+0000 7fa57796d640 10 monclient: build_initial_monmap 2024-09-16T04:06:01.386 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.357+0000 7fa57796d640 10 monclient: monmap: 2024-09-16T04:06:01.386 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:06:01.386 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:06:01.386 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:06:01.386 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:06:01.386 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:06:01.386 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:06:01.386 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:06:01.386 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:06:01.386 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:06:01.386 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:06:01.387 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.357+0000 7fa57796d640 1 librados: starting msgr at 2024-09-16T04:06:01.387 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.357+0000 7fa57796d640 1 librados: starting objecter 2024-09-16T04:06:01.387 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.357+0000 7fa57796d640 1 Processor -- start 2024-09-16T04:06:01.387 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.357+0000 7fa57796d640 1 -- start start 2024-09-16T04:06:01.387 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.357+0000 7fa57796d640 1 librados: setting wanted keys 2024-09-16T04:06:01.387 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.357+0000 7fa57796d640 1 librados: calling monclient init 2024-09-16T04:06:01.387 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.357+0000 7fa57796d640 10 monclient: init 2024-09-16T04:06:01.387 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.357+0000 7fa57796d640 10 monclient: _reopen_session rank -1 2024-09-16T04:06:01.387 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.357+0000 7fa57796d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:06:01.387 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.357+0000 7fa57796d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa55c0039c0 0x7fa5701cb750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:06:01.387 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.357+0000 7fa57796d640 10 monclient(hunting): picked mon.noname-b con 0x7fa55c0039c0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:06:01.387 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.357+0000 7fa57796d640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa5701cbc90 0x7fa570122af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:06:01.387 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.357+0000 7fa57796d640 10 monclient(hunting): picked mon.noname-c con 0x7fa5701cbc90 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:06:01.388 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.357+0000 7fa57796d640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa570123030 0x7fa5701234b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:06:01.388 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.357+0000 7fa57796d640 10 monclient(hunting): picked mon.noname-a con 0x7fa570123030 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:06:01.388 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.357+0000 7fa57796d640 10 monclient(hunting): start opening mon connection 2024-09-16T04:06:01.388 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.357+0000 7fa57796d640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fa570135910 con 0x7fa570123030 2024-09-16T04:06:01.388 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.357+0000 7fa57796d640 10 monclient(hunting): start opening mon connection 2024-09-16T04:06:01.388 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.357+0000 7fa57796d640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa57009ef80 con 0x7fa55c0039c0 2024-09-16T04:06:01.388 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.357+0000 7fa57796d640 10 monclient(hunting): start opening mon connection 2024-09-16T04:06:01.388 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.357+0000 7fa57796d640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fa570135a90 con 0x7fa5701cbc90 2024-09-16T04:06:01.388 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.357+0000 7fa57796d640 10 monclient(hunting): _renew_subs 2024-09-16T04:06:01.388 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.357+0000 7fa57796d640 10 monclient(hunting): authenticate will time out at 2408.331363s 2024-09-16T04:06:01.388 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.357+0000 7fa574f08640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa5701cbc90 0x7fa570122af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:06:01.388 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.357+0000 7fa575f0a640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa570123030 0x7fa5701234b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:06:01.388 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.357+0000 7fa574f08640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa5701cbc90 0x7fa570122af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:58234/0 (socket says 172.21.15.42:58234) 2024-09-16T04:06:01.389 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.357+0000 7fa574f08640 1 -- 172.21.15.42:0/2599629667 learned_addr learned my addr 172.21.15.42:0/2599629667 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:06:01.389 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.357+0000 7fa574f08640 10 monclient(hunting): get_auth_request con 0x7fa5701cbc90 auth_method 0 2024-09-16T04:06:01.389 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.357+0000 7fa574f08640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:06:01.389 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.357+0000 7fa574f08640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:01.389 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.357+0000 7fa574f08640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:01.389 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.357+0000 7fa575f0a640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa570123030 0x7fa5701234b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:38744/0 (socket says 172.21.15.42:38744) 2024-09-16T04:06:01.389 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.357+0000 7fa575f0a640 10 monclient(hunting): get_auth_request con 0x7fa570123030 auth_method 0 2024-09-16T04:06:01.389 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.357+0000 7fa575f0a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:06:01.389 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.357+0000 7fa575f0a640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:01.389 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.357+0000 7fa575f0a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:01.389 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.357+0000 7fa574f08640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:06:01.389 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.357+0000 7fa574f08640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:06:01.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.357+0000 7fa574f08640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:06:01.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.357+0000 7fa575f0a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:06:01.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.357+0000 7fa575f0a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:06:01.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.357+0000 7fa575f0a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:06:01.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.357+0000 7fa575709640 1 --2- 172.21.15.42:0/2599629667 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa55c0039c0 0x7fa5701cb750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:06:01.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.357+0000 7fa575709640 10 monclient(hunting): get_auth_request con 0x7fa55c0039c0 auth_method 0 2024-09-16T04:06:01.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.357+0000 7fa575709640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:06:01.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.357+0000 7fa575709640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:01.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.357+0000 7fa575709640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:01.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.357+0000 7fa574f08640 10 monclient(hunting): handle_auth_done global_id 1004111 payload 824 2024-09-16T04:06:01.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.357+0000 7fa574f08640 1 -- 172.21.15.42:0/2599629667 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa55c0039c0 msgr2=0x7fa5701cb750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:01.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.357+0000 7fa574f08640 1 --2- 172.21.15.42:0/2599629667 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa55c0039c0 0x7fa5701cb750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:06:01.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.357+0000 7fa574f08640 1 -- 172.21.15.42:0/2599629667 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa570123030 msgr2=0x7fa5701234b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:01.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.357+0000 7fa574f08640 1 --2- 172.21.15.42:0/2599629667 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa570123030 0x7fa5701234b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:06:01.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.357+0000 7fa574f08640 10 monclient: _finish_hunting 0 2024-09-16T04:06:01.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.357+0000 7fa574f08640 1 monclient: found mon.noname-c 2024-09-16T04:06:01.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.357+0000 7fa574f08640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:06:01.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.357+0000 7fa574f08640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T04:06:01.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.357+0000 7fa574f08640 1 -- 172.21.15.42:0/2599629667 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5701264c0 con 0x7fa5701cbc90 2024-09-16T04:06:01.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.357+0000 7fa575709640 1 --2- 172.21.15.42:0/2599629667 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa55c0039c0 0x7fa5701cb750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T04:06:01.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.357+0000 7fa575f0a640 1 --2- 172.21.15.42:0/2599629667 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa570123030 0x7fa5701234b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:06:01.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.361+0000 7fa574f08640 1 --2- 172.21.15.42:0/2599629667 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa5701cbc90 0x7fa570122af0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fa56c00f910 tx=0x7fa56c00fd10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:06:01.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.361+0000 7fa565ffb640 10 client.?.objecter ms_handle_connect 0x7fa5701cbc90 2024-09-16T04:06:01.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.361+0000 7fa565ffb640 10 client.?.objecter resend_mon_ops 2024-09-16T04:06:01.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.361+0000 7fa565ffb640 1 -- 172.21.15.42:0/2599629667 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa56c003b90 con 0x7fa5701cbc90 2024-09-16T04:06:01.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.361+0000 7fa565ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:01.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.361+0000 7fa565ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:06:01.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.361+0000 7fa565ffb640 10 monclient: dump: 2024-09-16T04:06:01.392 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:06:01.392 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:06:01.392 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:06:01.392 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:06:01.392 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:01.392 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:06:01.392 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:06:01.392 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:06:01.392 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:06:01.392 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:06:01.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.361+0000 7fa565ffb640 10 monclient: _finish_auth 0 2024-09-16T04:06:01.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.361+0000 7fa565ffb640 10 monclient: _check_auth_tickets 2024-09-16T04:06:01.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.361+0000 7fa565ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:06:01.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.361+0000 7fa565ffb640 1 -- 172.21.15.42:0/2599629667 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa56c018070 con 0x7fa5701cbc90 2024-09-16T04:06:01.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.361+0000 7fa565ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:06:01.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.361+0000 7fa565ffb640 1 -- 172.21.15.42:0/2599629667 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa56c0050d0 con 0x7fa5701cbc90 2024-09-16T04:06:01.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.361+0000 7fa57796d640 5 monclient: authenticate success, global_id 1004111 2024-09-16T04:06:01.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.361+0000 7fa57796d640 10 monclient: _renew_subs 2024-09-16T04:06:01.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.361+0000 7fa57796d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:06:01.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.361+0000 7fa57796d640 1 -- 172.21.15.42:0/2599629667 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa570124c30 con 0x7fa5701cbc90 2024-09-16T04:06:01.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.361+0000 7fa565ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:01.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.361+0000 7fa565ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:06:01.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.361+0000 7fa565ffb640 10 monclient: dump: 2024-09-16T04:06:01.394 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:06:01.394 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:06:01.394 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:06:01.394 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:06:01.394 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:01.394 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:06:01.394 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:06:01.394 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:06:01.394 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:06:01.394 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:06:01.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.361+0000 7fa5677fe640 4 set_mon_vals no callback set 2024-09-16T04:06:01.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.361+0000 7fa57796d640 10 client.1004111.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:06:01.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.361+0000 7fa57796d640 10 monclient: _renew_subs 2024-09-16T04:06:01.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.361+0000 7fa57796d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:06:01.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.361+0000 7fa57796d640 1 -- 172.21.15.42:0/2599629667 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa57006e520 con 0x7fa5701cbc90 2024-09-16T04:06:01.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.361+0000 7fa57796d640 1 librados: init done 2024-09-16T04:06:01.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.361+0000 7fa565ffb640 1 -- 172.21.15.42:0/2599629667 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fa56c005270 con 0x7fa5701cbc90 2024-09-16T04:06:01.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.361+0000 7fa57796d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:06:01.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.361+0000 7fa57796d640 1 --2- 172.21.15.42:0/2599629667 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa57002d7d0 0x7fa5701239f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:06:01.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.361+0000 7fa57796d640 10 monclient(hunting): start opening mon connection 2024-09-16T04:06:01.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.361+0000 7fa57796d640 1 -- 172.21.15.42:0/2599629667 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fa57002dbb0 con 0x7fa57002d7d0 2024-09-16T04:06:01.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.361+0000 7fa565ffb640 1 --2- 172.21.15.42:0/2599629667 >> v2:172.21.15.154:6812/254845342 conn(0x7fa554037810 0x7fa554039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:06:01.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.361+0000 7fa565ffb640 1 -- 172.21.15.42:0/2599629667 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(706..706 src has 451..706) ==== 5040+0+0 (secure 0 0 0) 0x7fa56c048c70 con 0x7fa5701cbc90 2024-09-16T04:06:01.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.361+0000 7fa565ffb640 10 client.1004111.objecter ms_dispatch 0x7fa570000ba0 osd_map(706..706 src has 451..706) 2024-09-16T04:06:01.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.361+0000 7fa565ffb640 3 client.1004111.objecter handle_osd_map got epochs [706,706] > 0 2024-09-16T04:06:01.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.361+0000 7fa575709640 1 --2- 172.21.15.42:0/2599629667 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa57002d7d0 0x7fa5701239f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:06:01.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.361+0000 7fa565ffb640 3 client.1004111.objecter handle_osd_map decoding full epoch 706 2024-09-16T04:06:01.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.361+0000 7fa575709640 10 monclient: get_auth_request con 0x7fa57002d7d0 auth_method 0 2024-09-16T04:06:01.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.361+0000 7fa575709640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:06:01.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.361+0000 7fa575709640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:01.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.361+0000 7fa575709640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:01.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.361+0000 7fa575709640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:06:01.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.361+0000 7fa575709640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:06:01.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.361+0000 7fa565ffb640 20 client.1004111.objecter dump_active .. 0 homeless 2024-09-16T04:06:01.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.361+0000 7fa575709640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:06:01.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.361+0000 7fa575709640 10 monclient(hunting): handle_auth_done global_id 1004114 payload 824 2024-09-16T04:06:01.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.361+0000 7fa575709640 1 -- 172.21.15.42:0/2599629667 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa570071b90 con 0x7fa57002d7d0 2024-09-16T04:06:01.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.361+0000 7fa575f0a640 1 --2- 172.21.15.42:0/2599629667 >> v2:172.21.15.154:6812/254845342 conn(0x7fa554037810 0x7fa554039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:06:01.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.361+0000 7fa575f0a640 10 monclient: get_auth_request con 0x7fa554037810 auth_method 0 2024-09-16T04:06:01.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.361+0000 7fa565ffb640 10 client.1004111.objecter ms_handle_connect 0x7fa57002d7d0 2024-09-16T04:06:01.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.361+0000 7fa565ffb640 10 client.1004111.objecter resend_mon_ops 2024-09-16T04:06:01.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.361+0000 7fa575709640 1 --2- 172.21.15.42:0/2599629667 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa57002d7d0 0x7fa5701239f0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fa560009ab0 tx=0x7fa5600197c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:06:01.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.361+0000 7fa565ffb640 1 -- 172.21.15.42:0/2599629667 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5600b2040 con 0x7fa57002d7d0 2024-09-16T04:06:01.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.361+0000 7fa565ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:01.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.361+0000 7fa565ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:06:01.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.361+0000 7fa565ffb640 10 monclient: dump: 2024-09-16T04:06:01.397 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:06:01.397 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:06:01.397 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:06:01.397 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:06:01.397 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:01.397 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:06:01.398 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:06:01.398 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:06:01.398 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:06:01.398 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:06:01.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.361+0000 7fa565ffb640 1 -- 172.21.15.42:0/2599629667 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fa5600096b0 con 0x7fa57002d7d0 2024-09-16T04:06:01.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.361+0000 7fa565ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:06:01.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.361+0000 7fa565ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:06:01.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.361+0000 7fa565ffb640 1 -- 172.21.15.42:0/2599629667 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa57002d7d0 msgr2=0x7fa5701239f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:01.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.361+0000 7fa565ffb640 1 --2- 172.21.15.42:0/2599629667 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa57002d7d0 0x7fa5701239f0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fa560009ab0 tx=0x7fa5600197c0 comp rx=0 tx=0).stop 2024-09-16T04:06:01.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.361+0000 7fa565ffb640 1 -- 172.21.15.42:0/2599629667 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa57002d7d0 msgr2=0x7fa5701239f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:06:01.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.361+0000 7fa565ffb640 1 --2- 172.21.15.42:0/2599629667 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa57002d7d0 0x7fa5701239f0 secure :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fa560009ab0 tx=0x7fa5600197c0 comp rx=0 tx=0).stop 2024-09-16T04:06:01.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.361+0000 7fa565ffb640 10 client.1004111.objecter ms_handle_connect 0x7fa554037810 2024-09-16T04:06:01.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.361+0000 7fa575f0a640 1 --2- 172.21.15.42:0/2599629667 >> v2:172.21.15.154:6812/254845342 conn(0x7fa554037810 0x7fa554039cd0 secure :-1 s=READY pgs=3859 cs=0 l=1 rev1=1 crypto rx=0x7fa5700a2340 tx=0x7fa568007040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:06:01.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.369+0000 7fa57796d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:06:01.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.369+0000 7fa57796d640 1 --2- 172.21.15.42:0/2599629667 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa570062ea0 0x7fa5700632b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:06:01.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.369+0000 7fa57796d640 10 monclient(hunting): start opening mon connection 2024-09-16T04:06:01.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.369+0000 7fa57796d640 1 -- 172.21.15.42:0/2599629667 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fa570135e20 con 0x7fa570062ea0 2024-09-16T04:06:01.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.369+0000 7fa575709640 1 --2- 172.21.15.42:0/2599629667 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa570062ea0 0x7fa5700632b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:06:01.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.369+0000 7fa575709640 10 monclient: get_auth_request con 0x7fa570062ea0 auth_method 0 2024-09-16T04:06:01.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.369+0000 7fa575709640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:06:01.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.369+0000 7fa575709640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:01.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.369+0000 7fa575709640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:01.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.369+0000 7fa575709640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:06:01.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.369+0000 7fa575709640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:06:01.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.369+0000 7fa575709640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:06:01.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.369+0000 7fa575709640 10 monclient(hunting): handle_auth_done global_id 1004117 payload 824 2024-09-16T04:06:01.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.369+0000 7fa575709640 1 -- 172.21.15.42:0/2599629667 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fa57011f690 con 0x7fa570062ea0 2024-09-16T04:06:01.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.369+0000 7fa565ffb640 10 client.1004111.objecter ms_handle_connect 0x7fa570062ea0 2024-09-16T04:06:01.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.369+0000 7fa565ffb640 10 client.1004111.objecter resend_mon_ops 2024-09-16T04:06:01.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.369+0000 7fa575709640 1 --2- 172.21.15.42:0/2599629667 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa570062ea0 0x7fa5700632b0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fa560009ae0 tx=0x7fa55c005840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:06:01.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.369+0000 7fa575709640 1 -- 172.21.15.42:0/2599629667 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa570062ea0 msgr2=0x7fa5700632b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-16T04:06:01.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.369+0000 7fa575709640 1 -- 172.21.15.42:0/2599629667 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa570062ea0 msgr2=0x7fa5700632b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:06:01.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.369+0000 7fa575709640 1 --2- 172.21.15.42:0/2599629667 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa570062ea0 0x7fa5700632b0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fa560009ae0 tx=0x7fa55c005840 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:06:01.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.369+0000 7fa575709640 1 --2- 172.21.15.42:0/2599629667 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa570062ea0 0x7fa5700632b0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fa560009ae0 tx=0x7fa55c005840 comp rx=0 tx=0).stop 2024-09-16T04:06:01.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.369+0000 7fa565ffb640 1 -- 172.21.15.42:0/2599629667 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa570062ea0 msgr2=0x7fa5700632b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:06:01.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.369+0000 7fa565ffb640 1 --2- 172.21.15.42:0/2599629667 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa570062ea0 0x7fa5700632b0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:06:01.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.369+0000 7fa565ffb640 1 --2- 172.21.15.42:0/2599629667 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa55403f520 0x7fa554041910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:06:01.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.369+0000 7fa565ffb640 1 -- 172.21.15.42:0/2599629667 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa570062ea0 msgr2=0x7fa5700632b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:06:01.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.369+0000 7fa565ffb640 1 --2- 172.21.15.42:0/2599629667 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa570062ea0 0x7fa5700632b0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:06:01.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.369+0000 7fa565ffb640 10 monclient(hunting): start opening mon connection 2024-09-16T04:06:01.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.369+0000 7fa565ffb640 1 -- 172.21.15.42:0/2599629667 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fa554041f20 con 0x7fa55403f520 2024-09-16T04:06:01.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.369+0000 7fa575709640 1 --2- 172.21.15.42:0/2599629667 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa55403f520 0x7fa554041910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:06:01.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.369+0000 7fa575709640 10 monclient: get_auth_request con 0x7fa55403f520 auth_method 0 2024-09-16T04:06:01.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.369+0000 7fa575709640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:06:01.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.369+0000 7fa575709640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:01.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.369+0000 7fa575709640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:01.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.369+0000 7fa575709640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:06:01.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.369+0000 7fa575709640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:06:01.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.369+0000 7fa575709640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:06:01.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.369+0000 7fa575709640 10 monclient(hunting): handle_auth_done global_id 1004120 payload 824 2024-09-16T04:06:01.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.369+0000 7fa575709640 1 -- 172.21.15.42:0/2599629667 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fa5600096b0 con 0x7fa55403f520 2024-09-16T04:06:01.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.369+0000 7fa575709640 1 --2- 172.21.15.42:0/2599629667 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa55403f520 0x7fa554041910 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fa560006f90 tx=0x7fa560008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:06:01.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.369+0000 7fa565ffb640 10 client.1004111.objecter ms_handle_connect 0x7fa55403f520 2024-09-16T04:06:01.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.369+0000 7fa565ffb640 10 client.1004111.objecter resend_mon_ops 2024-09-16T04:06:01.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.369+0000 7fa565ffb640 1 -- 172.21.15.42:0/2599629667 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5600c3070 con 0x7fa55403f520 2024-09-16T04:06:01.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.369+0000 7fa565ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:01.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.369+0000 7fa565ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:06:01.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.369+0000 7fa565ffb640 10 monclient: dump: 2024-09-16T04:06:01.405 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:06:01.405 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:06:01.405 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:06:01.405 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:06:01.405 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:01.406 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:06:01.406 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:06:01.406 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:06:01.406 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:06:01.406 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:06:01.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.369+0000 7fa565ffb640 1 -- 172.21.15.42:0/2599629667 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4544 (secure 0 0 0) 0x7fa5600bfe50 con 0x7fa55403f520 2024-09-16T04:06:01.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.369+0000 7fa565ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:06:01.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.369+0000 7fa565ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:06:01.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.369+0000 7fa565ffb640 1 -- 172.21.15.42:0/2599629667 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa55403f520 msgr2=0x7fa554041910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:01.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.369+0000 7fa565ffb640 1 --2- 172.21.15.42:0/2599629667 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa55403f520 0x7fa554041910 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fa560006f90 tx=0x7fa560008040 comp rx=0 tx=0).stop 2024-09-16T04:06:01.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.369+0000 7fa565ffb640 1 -- 172.21.15.42:0/2599629667 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa55403f520 msgr2=0x7fa554041910 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:06:01.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.369+0000 7fa565ffb640 1 --2- 172.21.15.42:0/2599629667 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa55403f520 0x7fa554041910 secure :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fa560006f90 tx=0x7fa560008040 comp rx=0 tx=0).stop 2024-09-16T04:06:01.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.369+0000 7fa575f0a640 1 -- 172.21.15.42:0/2599629667 reap_dead start 2024-09-16T04:06:01.407 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T04:06:01.407 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:06:01.407 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:06:01.407 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T04:06:01.407 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 222, 2024-09-16T04:06:01.407 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 7399, 2024-09-16T04:06:01.407 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T04:06:01.407 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T04:06:01.407 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T04:06:01.407 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T04:06:01.407 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:06:01.408 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 2, 2024-09-16T04:06:01.408 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:06:01.408 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T04:06:01.408 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T04:06:01.408 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:06:01.408 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:06:01.408 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:06:01.408 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:06:01.408 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:06:01.408 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:06:01.408 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:06:01.408 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:06:01.408 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:06:01.409 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:06:01.409 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:06:01.409 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:06:01.409 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:06:01.409 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T04:06:01.409 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:06:01.409 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:06:01.409 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:06:01.409 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:06:01.409 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:06:01.409 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:06:01.409 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:06:01.410 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:06:01.410 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:06:01.410 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:06:01.410 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:06:01.410 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:06:01.410 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:06:01.410 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T04:06:01.410 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T04:06:01.410 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T04:06:01.410 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T04:06:01.410 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T04:06:01.410 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T04:06:01.410 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:06:01.411 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:06:01.411 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T04:06:01.411 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T04:06:01.411 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T04:06:01.411 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T04:06:01.411 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T04:06:01.411 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T04:06:01.411 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T04:06:01.411 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:06:01.411 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T04:06:01.411 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:06:01.411 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:06:01.411 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:06:01.412 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:06:01.412 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:06:01.412 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:06:01.412 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:06:01.412 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:06:01.412 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:06:01.412 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:06:01.412 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:06:01.412 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:06:01.412 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T04:06:01.412 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:06:01.412 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T04:06:01.412 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:06:01.413 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:06:01.413 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:06:01.413 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:06:01.413 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:06:01.413 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:06:01.413 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:06:01.413 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T04:06:01.413 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:06:01.413 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:06:01.413 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:06:01.413 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:06:01.413 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T04:06:01.414 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:06:01.414 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:06:01.414 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:06:01.414 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:06:01.414 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T04:06:01.414 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T04:06:01.414 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:06:01.414 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:06:01.414 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:06:01.414 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:06:01.414 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:06:01.414 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:06:01.414 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:06:01.414 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:06:01.415 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:06:01.415 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:06:01.415 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:06:01.415 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T04:06:01.415 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:06:01.415 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:06:01.415 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:06:01.415 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:06:01.415 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T04:06:01.415 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:06:01.415 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:06:01.415 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:06:01.415 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:06:01.415 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T04:06:01.416 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T04:06:01.416 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:06:01.416 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:06:01.416 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:06:01.416 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:06:01.416 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:06:01.416 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:06:01.416 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:06:01.416 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:06:01.416 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:06:01.416 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:06:01.416 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:06:01.416 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T04:06:01.416 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:06:01.416 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:06:01.417 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:06:01.417 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:06:01.417 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T04:06:01.417 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:06:01.417 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:06:01.417 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:06:01.417 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:06:01.417 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T04:06:01.417 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T04:06:01.417 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:06:01.417 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:06:01.417 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:06:01.417 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:06:01.417 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:06:01.418 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:06:01.418 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T04:06:01.418 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T04:06:01.418 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:06:01.418 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:06:01.418 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:06:01.418 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:06:01.418 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:06:01.418 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:06:01.418 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T04:06:01.418 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:06:01.418 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:06:01.418 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:06:01.419 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 2 2024-09-16T04:06:01.419 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:06:01.419 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:06:01.419 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:06:01.419 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T04:06:01.419 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T04:06:01.420 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.369+0000 7fa57796d640 10 librados: watch_flush enter 2024-09-16T04:06:01.421 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.369+0000 7fa57796d640 10 librados: watch_flush exit 2024-09-16T04:06:01.421 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.369+0000 7fa57796d640 20 client.1004111.objecter shutdown clearing up homeless session... 2024-09-16T04:06:01.421 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.369+0000 7fa57796d640 10 client.1004111.objecter successfully canceled tick 2024-09-16T04:06:01.421 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.369+0000 7fa57796d640 1 -- 172.21.15.42:0/2599629667 >> v2:172.21.15.154:6812/254845342 conn(0x7fa554037810 msgr2=0x7fa554039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:01.421 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.369+0000 7fa57796d640 1 --2- 172.21.15.42:0/2599629667 >> v2:172.21.15.154:6812/254845342 conn(0x7fa554037810 0x7fa554039cd0 secure :-1 s=READY pgs=3859 cs=0 l=1 rev1=1 crypto rx=0x7fa5700a2340 tx=0x7fa568007040 comp rx=0 tx=0).stop 2024-09-16T04:06:01.421 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.369+0000 7fa57796d640 10 monclient: shutdown 2024-09-16T04:06:01.421 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.369+0000 7fa57796d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:06:01.421 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.369+0000 7fa57796d640 1 -- 172.21.15.42:0/2599629667 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa5701cbc90 msgr2=0x7fa570122af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:01.421 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.369+0000 7fa57796d640 1 --2- 172.21.15.42:0/2599629667 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa5701cbc90 0x7fa570122af0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fa56c00f910 tx=0x7fa56c00fd10 comp rx=0 tx=0).stop 2024-09-16T04:06:01.421 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.369+0000 7fa57796d640 1 -- 172.21.15.42:0/2599629667 shutdown_connections 2024-09-16T04:06:01.421 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.369+0000 7fa57796d640 1 --2- 172.21.15.42:0/2599629667 >> v2:172.21.15.154:6812/254845342 conn(0x7fa554037810 0x7fa554039cd0 unknown :-1 s=CLOSED pgs=3859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:06:01.421 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.369+0000 7fa57796d640 1 --2- 172.21.15.42:0/2599629667 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa5701cbc90 0x7fa570122af0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:06:01.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.369+0000 7fa57796d640 1 -- 172.21.15.42:0/2599629667 >> 172.21.15.42:0/2599629667 conn(0x7fa5700a2420 msgr2=0x7fa570070470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:06:01.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.369+0000 7fa57796d640 1 -- 172.21.15.42:0/2599629667 shutdown_connections 2024-09-16T04:06:01.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.369+0000 7fa57796d640 1 -- 172.21.15.42:0/2599629667 wait complete. 2024-09-16T04:06:01.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.369+0000 7fa57796d640 1 librados: shutdown 2024-09-16T04:06:01.422 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:06:01.422 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T04:06:01.422 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T04:06:01.544 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.537+0000 7f096fbdb640 10 monclient: get_monmap_and_config 2024-09-16T04:06:01.544 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.537+0000 7f096fbdb640 10 monclient: build_initial_monmap 2024-09-16T04:06:01.544 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.537+0000 7f096fbdb640 10 monclient: monmap: 2024-09-16T04:06:01.544 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:06:01.544 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:06:01.545 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:06:01.542804+0000 2024-09-16T04:06:01.545 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:06:01.542804+0000 2024-09-16T04:06:01.545 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:06:01.545 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:06:01.545 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:06:01.545 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:06:01.545 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:06:01.545 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:06:01.545 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:06:01.545 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:06:01.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.537+0000 7f096fbdb640 1 Processor -- start 2024-09-16T04:06:01.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.537+0000 7f096fbdb640 1 -- start start 2024-09-16T04:06:01.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.537+0000 7f096fbdb640 10 monclient: init 2024-09-16T04:06:01.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.541+0000 7f096fbdb640 10 monclient: _reopen_session rank -1 2024-09-16T04:06:01.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.541+0000 7f096fbdb640 10 monclient: _add_conns ranks=[0,1,3,4,2] 2024-09-16T04:06:01.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.541+0000 7f096fbdb640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f09680a39b0 addr v1:172.21.15.42:6789/0 2024-09-16T04:06:01.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.541+0000 7f096fbdb640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f09680a3d90 addr v1:172.21.15.154:6789/0 2024-09-16T04:06:01.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.541+0000 7f096fbdb640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f096813daa0 0x7f096813fe90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:06:01.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.541+0000 7f096fbdb640 10 monclient(hunting): picked mon.noname-a con 0x7f096813daa0 addr v2:172.21.15.42:3300/0 2024-09-16T04:06:01.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.541+0000 7f096fbdb640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0968134290 con 0x7f09680a39b0 2024-09-16T04:06:01.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.541+0000 7f096fbdb640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f096809c140 con 0x7f09680a3d90 2024-09-16T04:06:01.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.541+0000 7f096fbdb640 10 monclient(hunting): start opening mon connection 2024-09-16T04:06:01.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.541+0000 7f096fbdb640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f096809c420 con 0x7f096813daa0 2024-09-16T04:06:01.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.541+0000 7f096fbdb640 10 monclient(hunting): _renew_subs 2024-09-16T04:06:01.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.541+0000 7f096fbdb640 10 monclient(hunting): authenticate will time out at 2408.512578s 2024-09-16T04:06:01.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.541+0000 7f096e178640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f096813daa0 0x7f096813fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:06:01.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.541+0000 7f096e178640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f096813daa0 0x7f096813fe90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:38750/0 (socket says 172.21.15.42:38750) 2024-09-16T04:06:01.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.541+0000 7f096e178640 1 -- 172.21.15.42:0/1033795946 learned_addr learned my addr 172.21.15.42:0/1033795946 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:06:01.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.541+0000 7f096e178640 10 monclient(hunting): get_auth_request con 0x7f096813daa0 auth_method 0 2024-09-16T04:06:01.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.541+0000 7f096e178640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:06:01.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.541+0000 7f096e178640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:01.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.541+0000 7f096e178640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:01.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.541+0000 7f096d977640 1 --1- 172.21.15.42:0/1033795946 >> v1:172.21.15.42:6789/0 conn(0x7f09680a39b0 0x7f0968133b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:56056/0 (socket says 172.21.15.42:56056) 2024-09-16T04:06:01.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.541+0000 7f096e178640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:06:01.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.541+0000 7f096e178640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:06:01.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.541+0000 7f096e178640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:06:01.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.541+0000 7f096e178640 10 monclient(hunting): handle_auth_done global_id 984384 payload 274 2024-09-16T04:06:01.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.541+0000 7f096e178640 1 -- 172.21.15.42:0/1033795946 >> v1:172.21.15.154:6789/0 conn(0x7f09680a3d90 legacy=0x7f096813c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:06:01.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.541+0000 7f096e178640 1 -- 172.21.15.42:0/1033795946 >> v1:172.21.15.42:6789/0 conn(0x7f09680a39b0 legacy=0x7f0968133b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:01.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.541+0000 7f096e178640 10 monclient: _finish_hunting 0 2024-09-16T04:06:01.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.541+0000 7f096e178640 1 monclient: found mon.noname-a 2024-09-16T04:06:01.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.541+0000 7f096e178640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:06:01.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.541+0000 7f096e178640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:06:01.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.541+0000 7f096e178640 1 -- 172.21.15.42:0/1033795946 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f096806dd60 con 0x7f096813daa0 2024-09-16T04:06:01.568 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.541+0000 7f096e178640 1 --2- 172.21.15.42:0/1033795946 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f096813daa0 0x7f096813fe90 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f0960009870 tx=0x7f096001c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a58e3ceda44ddc29 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:06:01.568 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.541+0000 7f096c975640 1 -- 172.21.15.42:0/1033795946 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09600b0070 con 0x7f096813daa0 2024-09-16T04:06:01.569 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.541+0000 7f096c975640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:01.569 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.541+0000 7f096c975640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:06:01.569 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.541+0000 7f096c975640 10 monclient: dump: 2024-09-16T04:06:01.569 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:06:01.569 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:06:01.570 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:06:01.570 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:06:01.570 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:01.570 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:06:01.570 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:06:01.571 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:06:01.571 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:06:01.571 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:06:01.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.541+0000 7f096c975640 10 monclient: _finish_auth 0 2024-09-16T04:06:01.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.541+0000 7f096c975640 10 monclient: _check_auth_tickets 2024-09-16T04:06:01.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.541+0000 7f096c975640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:06:01.572 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.541+0000 7f096c975640 1 -- 172.21.15.42:0/1033795946 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f09600ab600 con 0x7f096813daa0 2024-09-16T04:06:01.572 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.541+0000 7f096fbdb640 5 monclient: authenticate success, global_id 984384 2024-09-16T04:06:01.572 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.541+0000 7f096fbdb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:06:01.572 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.541+0000 7f096c975640 10 monclient: handle_config config(0 keys) 2024-09-16T04:06:01.572 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.541+0000 7f096fbdb640 10 monclient: get_monmap_and_config success 2024-09-16T04:06:01.573 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.541+0000 7f096fbdb640 4 set_mon_vals no callback set 2024-09-16T04:06:01.573 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.541+0000 7f096c975640 1 -- 172.21.15.42:0/1033795946 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09600ab9d0 con 0x7f096813daa0 2024-09-16T04:06:01.573 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.541+0000 7f096c975640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:01.573 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.541+0000 7f096c975640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:06:01.573 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.541+0000 7f096c975640 10 monclient: dump: 2024-09-16T04:06:01.573 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:06:01.574 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:06:01.574 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:06:01.574 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:06:01.574 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:01.574 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:06:01.574 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:06:01.575 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:06:01.575 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:06:01.575 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:06:01.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.545+0000 7f096fbdb640 10 monclient: shutdown 2024-09-16T04:06:01.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.545+0000 7f096fbdb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:06:01.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.545+0000 7f096fbdb640 1 -- 172.21.15.42:0/1033795946 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f096813daa0 msgr2=0x7f096813fe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:01.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.545+0000 7f096fbdb640 1 --2- 172.21.15.42:0/1033795946 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f096813daa0 0x7f096813fe90 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f0960009870 tx=0x7f096001c3e0 comp rx=0 tx=0).stop 2024-09-16T04:06:01.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.545+0000 7f096fbdb640 1 -- 172.21.15.42:0/1033795946 shutdown_connections 2024-09-16T04:06:01.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.545+0000 7f096fbdb640 1 --2- 172.21.15.42:0/1033795946 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f096813daa0 0x7f096813fe90 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:06:01.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.545+0000 7f096fbdb640 1 -- 172.21.15.42:0/1033795946 >> 172.21.15.42:0/1033795946 conn(0x7f09680a9fb0 msgr2=0x7f09680aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:06:01.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.545+0000 7f096fbdb640 1 -- 172.21.15.42:0/1033795946 shutdown_connections 2024-09-16T04:06:01.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.545+0000 7f096fbdb640 1 -- 172.21.15.42:0/1033795946 wait complete. 2024-09-16T04:06:01.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.545+0000 7f096fbdb640 10 monclient: build_initial_monmap 2024-09-16T04:06:01.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.545+0000 7f096fbdb640 10 monclient: monmap: 2024-09-16T04:06:01.577 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:06:01.578 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:06:01.578 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:06:01.578 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:06:01.578 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:06:01.578 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:06:01.578 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:06:01.578 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:06:01.579 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:06:01.579 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:06:01.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.545+0000 7f096fbdb640 1 librados: starting msgr at 2024-09-16T04:06:01.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.545+0000 7f096fbdb640 1 librados: starting objecter 2024-09-16T04:06:01.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.545+0000 7f096fbdb640 1 Processor -- start 2024-09-16T04:06:01.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.545+0000 7f096fbdb640 1 -- start start 2024-09-16T04:06:01.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.545+0000 7f096fbdb640 1 librados: setting wanted keys 2024-09-16T04:06:01.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.545+0000 7f096fbdb640 1 librados: calling monclient init 2024-09-16T04:06:01.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.545+0000 7f096fbdb640 10 monclient: init 2024-09-16T04:06:01.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.545+0000 7f096fbdb640 10 monclient: _reopen_session rank -1 2024-09-16T04:06:01.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.545+0000 7f096fbdb640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:06:01.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.545+0000 7f096fbdb640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f09680a39b0 0x7f0968130660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:06:01.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.545+0000 7f096fbdb640 10 monclient(hunting): picked mon.noname-c con 0x7f09680a39b0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:06:01.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.545+0000 7f096fbdb640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f09680a3d90 0x7f0968130ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:06:01.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.545+0000 7f096fbdb640 10 monclient(hunting): picked mon.noname-b con 0x7f09680a3d90 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:06:01.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.545+0000 7f096fbdb640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f096813daa0 0x7f096813a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:06:01.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.545+0000 7f096fbdb640 10 monclient(hunting): picked mon.noname-a con 0x7f096813daa0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:06:01.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.545+0000 7f096fbdb640 10 monclient(hunting): start opening mon connection 2024-09-16T04:06:01.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.545+0000 7f096fbdb640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f0968142f80 con 0x7f096813daa0 2024-09-16T04:06:01.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.545+0000 7f096fbdb640 10 monclient(hunting): start opening mon connection 2024-09-16T04:06:01.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.545+0000 7f096fbdb640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f0968142e00 con 0x7f09680a3d90 2024-09-16T04:06:01.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.545+0000 7f096fbdb640 10 monclient(hunting): start opening mon connection 2024-09-16T04:06:01.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.545+0000 7f096fbdb640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f0968143100 con 0x7f09680a39b0 2024-09-16T04:06:01.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.545+0000 7f096fbdb640 10 monclient(hunting): _renew_subs 2024-09-16T04:06:01.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.545+0000 7f096fbdb640 10 monclient(hunting): authenticate will time out at 2408.518041s 2024-09-16T04:06:01.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.545+0000 7f096d977640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f09680a39b0 0x7f0968130660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:06:01.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.545+0000 7f096e178640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f096813daa0 0x7f096813a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:06:01.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.545+0000 7f096d977640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f09680a39b0 0x7f0968130660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:58262/0 (socket says 172.21.15.42:58262) 2024-09-16T04:06:01.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.545+0000 7f096d977640 1 -- 172.21.15.42:0/703552676 learned_addr learned my addr 172.21.15.42:0/703552676 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:06:01.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.545+0000 7f096d977640 10 monclient(hunting): get_auth_request con 0x7f09680a39b0 auth_method 0 2024-09-16T04:06:01.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.545+0000 7f096d977640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:06:01.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.545+0000 7f096d977640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:01.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.545+0000 7f096d977640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:01.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.545+0000 7f096e178640 10 monclient(hunting): get_auth_request con 0x7f096813daa0 auth_method 0 2024-09-16T04:06:01.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.545+0000 7f096e178640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:06:01.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.545+0000 7f096e178640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:01.582 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.545+0000 7f096e178640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:01.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.545+0000 7f096d977640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:06:01.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.545+0000 7f096d977640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:06:01.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.545+0000 7f096d977640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:06:01.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.545+0000 7f096d176640 1 --2- 172.21.15.42:0/703552676 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f09680a3d90 0x7f0968130ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:06:01.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.545+0000 7f096e178640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:06:01.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.545+0000 7f096e178640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:06:01.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.545+0000 7f096e178640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:06:01.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.545+0000 7f096d176640 10 monclient(hunting): get_auth_request con 0x7f09680a3d90 auth_method 0 2024-09-16T04:06:01.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.545+0000 7f096d176640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:06:01.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.545+0000 7f096d176640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:01.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.545+0000 7f096d176640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:01.583 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.545+0000 7f096d977640 10 monclient(hunting): handle_auth_done global_id 1004123 payload 824 2024-09-16T04:06:01.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.545+0000 7f096d977640 1 -- 172.21.15.42:0/703552676 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f09680a3d90 msgr2=0x7f0968130ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:01.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.545+0000 7f096d977640 1 --2- 172.21.15.42:0/703552676 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f09680a3d90 0x7f0968130ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:06:01.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.545+0000 7f096d977640 1 -- 172.21.15.42:0/703552676 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f096813daa0 msgr2=0x7f096813a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:01.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.545+0000 7f096d977640 1 --2- 172.21.15.42:0/703552676 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f096813daa0 0x7f096813a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:06:01.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.545+0000 7f096d977640 10 monclient: _finish_hunting 0 2024-09-16T04:06:01.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.545+0000 7f096d977640 1 monclient: found mon.noname-c 2024-09-16T04:06:01.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.545+0000 7f096d977640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:06:01.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.545+0000 7f096d977640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T04:06:01.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.545+0000 7f096d977640 1 -- 172.21.15.42:0/703552676 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f096809d940 con 0x7f09680a39b0 2024-09-16T04:06:01.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.545+0000 7f096e178640 1 --2- 172.21.15.42:0/703552676 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f096813daa0 0x7f096813a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:06:01.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.545+0000 7f096d176640 1 --2- 172.21.15.42:0/703552676 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f09680a3d90 0x7f0968130ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T04:06:01.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.545+0000 7f096d977640 1 --2- 172.21.15.42:0/703552676 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f09680a39b0 0x7f0968130660 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f095c00c370 tx=0x7f095c00c830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:06:01.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.545+0000 7f095a7fc640 10 client.?.objecter ms_handle_connect 0x7f09680a39b0 2024-09-16T04:06:01.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.545+0000 7f095a7fc640 10 client.?.objecter resend_mon_ops 2024-09-16T04:06:01.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.545+0000 7f095a7fc640 1 -- 172.21.15.42:0/703552676 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f095c0074e0 con 0x7f09680a39b0 2024-09-16T04:06:01.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.545+0000 7f095a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:01.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.545+0000 7f095a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:06:01.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.545+0000 7f095a7fc640 10 monclient: dump: 2024-09-16T04:06:01.585 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:06:01.585 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:06:01.585 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:06:01.585 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:06:01.586 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:01.586 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:06:01.586 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:06:01.586 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:06:01.586 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:06:01.586 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:06:01.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.545+0000 7f095a7fc640 10 monclient: _finish_auth 0 2024-09-16T04:06:01.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.545+0000 7f095a7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:06:01.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.545+0000 7f095a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:06:01.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.545+0000 7f095a7fc640 1 -- 172.21.15.42:0/703552676 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f095c007680 con 0x7f09680a39b0 2024-09-16T04:06:01.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.545+0000 7f095a7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:06:01.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.545+0000 7f095a7fc640 1 -- 172.21.15.42:0/703552676 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f095c004a30 con 0x7f09680a39b0 2024-09-16T04:06:01.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.545+0000 7f095a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:01.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.545+0000 7f095a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:06:01.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.545+0000 7f095a7fc640 10 monclient: dump: 2024-09-16T04:06:01.587 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:06:01.587 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:06:01.587 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:06:01.587 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:06:01.587 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:01.587 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:06:01.587 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:06:01.588 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:06:01.588 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:06:01.588 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:06:01.588 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.545+0000 7f095bfff640 4 set_mon_vals no callback set 2024-09-16T04:06:01.588 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.545+0000 7f096fbdb640 5 monclient: authenticate success, global_id 1004123 2024-09-16T04:06:01.588 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.545+0000 7f096fbdb640 10 monclient: _renew_subs 2024-09-16T04:06:01.588 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.545+0000 7f096fbdb640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:06:01.589 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.545+0000 7f096fbdb640 1 -- 172.21.15.42:0/703552676 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f096806f680 con 0x7f09680a39b0 2024-09-16T04:06:01.589 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.545+0000 7f096fbdb640 10 client.1004123.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:06:01.589 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.545+0000 7f096fbdb640 10 monclient: _renew_subs 2024-09-16T04:06:01.589 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.545+0000 7f096fbdb640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:06:01.589 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.545+0000 7f096fbdb640 1 -- 172.21.15.42:0/703552676 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f096813be80 con 0x7f09680a39b0 2024-09-16T04:06:01.589 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.545+0000 7f096fbdb640 1 librados: init done 2024-09-16T04:06:01.589 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.549+0000 7f096fbdb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:06:01.589 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.549+0000 7f096fbdb640 1 --2- 172.21.15.42:0/703552676 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f096813a770 0x7f096813ab50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:06:01.589 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.549+0000 7f096fbdb640 10 monclient(hunting): start opening mon connection 2024-09-16T04:06:01.589 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.549+0000 7f096fbdb640 1 -- 172.21.15.42:0/703552676 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f0968133b80 con 0x7f096813a770 2024-09-16T04:06:01.589 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.549+0000 7f095a7fc640 1 -- 172.21.15.42:0/703552676 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f095c01a020 con 0x7f09680a39b0 2024-09-16T04:06:01.590 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.549+0000 7f095a7fc640 1 --2- 172.21.15.42:0/703552676 >> v2:172.21.15.154:6812/254845342 conn(0x7f094c037840 0x7f094c039d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:06:01.590 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.549+0000 7f095a7fc640 1 -- 172.21.15.42:0/703552676 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(706..706 src has 451..706) ==== 5040+0+0 (secure 0 0 0) 0x7f095c048320 con 0x7f09680a39b0 2024-09-16T04:06:01.590 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.549+0000 7f095a7fc640 10 client.1004123.objecter ms_dispatch 0x7f0968000ba0 osd_map(706..706 src has 451..706) 2024-09-16T04:06:01.590 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.549+0000 7f095a7fc640 3 client.1004123.objecter handle_osd_map got epochs [706,706] > 0 2024-09-16T04:06:01.590 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.549+0000 7f095a7fc640 3 client.1004123.objecter handle_osd_map decoding full epoch 706 2024-09-16T04:06:01.590 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.549+0000 7f096d176640 1 --2- 172.21.15.42:0/703552676 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f096813a770 0x7f096813ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:06:01.590 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.549+0000 7f095a7fc640 20 client.1004123.objecter dump_active .. 0 homeless 2024-09-16T04:06:01.590 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.549+0000 7f096d176640 10 monclient: get_auth_request con 0x7f096813a770 auth_method 0 2024-09-16T04:06:01.590 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.549+0000 7f096d176640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:06:01.590 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.549+0000 7f096d176640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:01.590 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.549+0000 7f096d176640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:01.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.549+0000 7f096d176640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:06:01.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.549+0000 7f096d176640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:06:01.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.549+0000 7f096d176640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:06:01.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.549+0000 7f096e178640 1 --2- 172.21.15.42:0/703552676 >> v2:172.21.15.154:6812/254845342 conn(0x7f094c037840 0x7f094c039d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:06:01.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.549+0000 7f096e178640 10 monclient: get_auth_request con 0x7f094c037840 auth_method 0 2024-09-16T04:06:01.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.549+0000 7f096d176640 10 monclient(hunting): handle_auth_done global_id 904981 payload 824 2024-09-16T04:06:01.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.549+0000 7f096d176640 1 -- 172.21.15.42:0/703552676 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f09680a5750 con 0x7f096813a770 2024-09-16T04:06:01.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.549+0000 7f096d176640 1 --2- 172.21.15.42:0/703552676 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f096813a770 0x7f096813ab50 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f0968140070 tx=0x7f096400ef00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:06:01.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.549+0000 7f095a7fc640 10 client.1004123.objecter ms_handle_connect 0x7f096813a770 2024-09-16T04:06:01.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.549+0000 7f095a7fc640 10 client.1004123.objecter resend_mon_ops 2024-09-16T04:06:01.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.549+0000 7f095a7fc640 10 client.1004123.objecter ms_handle_connect 0x7f094c037840 2024-09-16T04:06:01.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.549+0000 7f096e178640 1 --2- 172.21.15.42:0/703552676 >> v2:172.21.15.154:6812/254845342 conn(0x7f094c037840 0x7f094c039d00 secure :-1 s=READY pgs=3860 cs=0 l=1 rev1=1 crypto rx=0x7f09600ad000 tx=0x7f0960008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:06:01.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.549+0000 7f095a7fc640 1 -- 172.21.15.42:0/703552676 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f096401a070 con 0x7f096813a770 2024-09-16T04:06:01.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.549+0000 7f095a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:01.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.549+0000 7f095a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:06:01.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.549+0000 7f095a7fc640 10 monclient: dump: 2024-09-16T04:06:01.593 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:06:01.593 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:06:01.593 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:06:01.593 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:06:01.593 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:01.593 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:06:01.593 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:06:01.593 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:06:01.593 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:06:01.594 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:06:01.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.549+0000 7f095a7fc640 1 -- 172.21.15.42:0/703552676 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f0964006b60 con 0x7f096813a770 2024-09-16T04:06:01.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.549+0000 7f095a7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:06:01.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.549+0000 7f095a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:06:01.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.549+0000 7f095a7fc640 1 -- 172.21.15.42:0/703552676 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f096813a770 msgr2=0x7f096813ab50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:01.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.549+0000 7f095a7fc640 1 --2- 172.21.15.42:0/703552676 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f096813a770 0x7f096813ab50 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f0968140070 tx=0x7f096400ef00 comp rx=0 tx=0).stop 2024-09-16T04:06:01.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.549+0000 7f095a7fc640 1 -- 172.21.15.42:0/703552676 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f096813a770 msgr2=0x7f096813ab50 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:06:01.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.549+0000 7f095a7fc640 1 --2- 172.21.15.42:0/703552676 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f096813a770 0x7f096813ab50 secure :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f0968140070 tx=0x7f096400ef00 comp rx=0 tx=0).stop 2024-09-16T04:06:01.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.557+0000 7f096fbdb640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:06:01.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.557+0000 7f096fbdb640 1 --2- 172.21.15.42:0/703552676 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f096813b090 0x7f0968136bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:06:01.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.557+0000 7f096fbdb640 10 monclient(hunting): start opening mon connection 2024-09-16T04:06:01.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.557+0000 7f096fbdb640 1 -- 172.21.15.42:0/703552676 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f096813ca70 con 0x7f096813b090 2024-09-16T04:06:01.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.557+0000 7f096d176640 1 --2- 172.21.15.42:0/703552676 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f096813b090 0x7f0968136bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:06:01.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.557+0000 7f096d176640 10 monclient: get_auth_request con 0x7f096813b090 auth_method 0 2024-09-16T04:06:01.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.557+0000 7f096d176640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:06:01.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.557+0000 7f096d176640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:01.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.557+0000 7f096d176640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:01.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.557+0000 7f096d176640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:06:01.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.557+0000 7f096d176640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:06:01.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.557+0000 7f096d176640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:06:01.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.557+0000 7f096d176640 10 monclient(hunting): handle_auth_done global_id 904984 payload 824 2024-09-16T04:06:01.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.557+0000 7f096d176640 1 -- 172.21.15.42:0/703552676 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f0968068e40 con 0x7f096813b090 2024-09-16T04:06:01.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.557+0000 7f096d176640 1 --2- 172.21.15.42:0/703552676 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f096813b090 0x7f0968136bc0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f0968140070 tx=0x7f0964003af0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:06:01.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.557+0000 7f095a7fc640 10 client.1004123.objecter ms_handle_connect 0x7f096813b090 2024-09-16T04:06:01.596 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.557+0000 7f095a7fc640 10 client.1004123.objecter resend_mon_ops 2024-09-16T04:06:01.596 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.557+0000 7f095a7fc640 1 -- 172.21.15.42:0/703552676 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f096401a040 con 0x7f096813b090 2024-09-16T04:06:01.596 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.557+0000 7f095a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:01.596 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.557+0000 7f095a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:06:01.596 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.557+0000 7f095a7fc640 10 monclient: dump: 2024-09-16T04:06:01.596 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:06:01.596 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:06:01.596 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:06:01.596 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:06:01.596 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:01.596 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:06:01.596 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:06:01.596 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:06:01.597 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:06:01.597 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:06:01.597 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.557+0000 7f095a7fc640 1 -- 172.21.15.42:0/703552676 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4880 (secure 0 0 0) 0x7f096400cc70 con 0x7f096813b090 2024-09-16T04:06:01.597 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.557+0000 7f095a7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:06:01.597 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.557+0000 7f095a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:06:01.597 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.557+0000 7f095a7fc640 1 -- 172.21.15.42:0/703552676 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f096813b090 msgr2=0x7f0968136bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:01.597 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.557+0000 7f095a7fc640 1 --2- 172.21.15.42:0/703552676 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f096813b090 0x7f0968136bc0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f0968140070 tx=0x7f0964003af0 comp rx=0 tx=0).stop 2024-09-16T04:06:01.597 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.557+0000 7f095a7fc640 1 -- 172.21.15.42:0/703552676 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f096813b090 msgr2=0x7f0968136bc0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:06:01.597 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.557+0000 7f095a7fc640 1 --2- 172.21.15.42:0/703552676 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f096813b090 0x7f0968136bc0 secure :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f0968140070 tx=0x7f0964003af0 comp rx=0 tx=0).stop 2024-09-16T04:06:01.598 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T04:06:01.598 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:06:01.598 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:06:01.598 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T04:06:01.598 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 222, 2024-09-16T04:06:01.598 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 489739, 2024-09-16T04:06:01.598 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T04:06:01.598 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T04:06:01.598 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T04:06:01.598 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T04:06:01.598 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:06:01.598 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 2, 2024-09-16T04:06:01.598 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:06:01.599 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T04:06:01.599 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T04:06:01.599 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:06:01.599 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:06:01.599 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:06:01.599 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:06:01.599 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:06:01.599 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:06:01.599 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:06:01.599 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:06:01.599 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:06:01.599 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:06:01.599 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:06:01.599 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:06:01.600 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:06:01.600 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T04:06:01.600 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:06:01.600 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:06:01.600 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:06:01.600 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:06:01.600 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:06:01.600 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:06:01.600 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:06:01.600 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:06:01.600 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:06:01.600 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:06:01.600 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:06:01.600 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:06:01.601 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:06:01.601 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T04:06:01.601 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T04:06:01.601 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T04:06:01.601 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T04:06:01.601 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T04:06:01.601 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T04:06:01.601 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:06:01.602 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:06:01.602 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T04:06:01.602 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T04:06:01.602 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T04:06:01.602 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T04:06:01.602 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T04:06:01.602 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T04:06:01.602 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T04:06:01.602 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:06:01.602 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T04:06:01.602 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:06:01.603 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:06:01.603 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:06:01.603 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:06:01.603 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:06:01.603 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:06:01.603 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:06:01.603 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:06:01.603 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:06:01.604 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:06:01.604 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:06:01.604 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:06:01.604 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T04:06:01.604 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:06:01.604 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T04:06:01.604 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:06:01.604 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:06:01.604 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:06:01.604 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:06:01.604 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:06:01.604 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:06:01.604 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:06:01.605 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T04:06:01.605 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:06:01.605 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:06:01.605 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:06:01.605 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:06:01.605 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T04:06:01.605 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:06:01.605 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:06:01.605 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:06:01.605 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:06:01.605 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T04:06:01.605 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T04:06:01.605 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:06:01.606 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:06:01.606 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:06:01.606 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:06:01.606 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:06:01.606 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:06:01.606 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:06:01.606 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:06:01.606 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:06:01.606 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:06:01.606 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:06:01.606 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T04:06:01.606 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:06:01.606 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:06:01.607 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:06:01.607 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:06:01.607 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T04:06:01.607 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:06:01.607 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:06:01.607 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:06:01.607 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:06:01.607 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T04:06:01.607 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T04:06:01.607 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:06:01.607 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:06:01.607 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:06:01.608 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:06:01.608 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:06:01.608 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:06:01.608 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:06:01.608 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:06:01.608 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:06:01.608 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:06:01.608 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:06:01.608 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T04:06:01.608 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:06:01.608 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:06:01.608 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:06:01.608 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:06:01.608 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T04:06:01.609 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:06:01.609 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:06:01.609 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:06:01.609 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:06:01.609 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T04:06:01.609 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T04:06:01.609 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:06:01.609 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:06:01.609 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:06:01.609 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:06:01.609 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:06:01.609 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:06:01.609 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T04:06:01.609 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T04:06:01.610 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:06:01.610 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:06:01.610 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:06:01.610 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:06:01.610 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:06:01.610 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:06:01.610 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T04:06:01.610 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:06:01.610 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:06:01.610 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:06:01.610 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 3 2024-09-16T04:06:01.610 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:06:01.610 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:06:01.611 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T04:06:01.611 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:06:01.611 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:06:01.611 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:06:01.611 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 5 2024-09-16T04:06:01.611 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:06:01.611 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:06:01.611 INFO:teuthology.orchestra.run.smithi042.stdout: "mgr": [ 2024-09-16T04:06:01.612 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:06:01.612 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:06:01.612 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:06:01.612 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:06:01.612 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:06:01.612 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:06:01.612 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:06:01.612 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T04:06:01.612 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T04:06:01.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.557+0000 7f096fbdb640 10 librados: watch_flush enter 2024-09-16T04:06:01.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.557+0000 7f096fbdb640 10 librados: watch_flush exit 2024-09-16T04:06:01.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.557+0000 7f096fbdb640 20 client.1004123.objecter shutdown clearing up homeless session... 2024-09-16T04:06:01.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.557+0000 7f096fbdb640 10 client.1004123.objecter successfully canceled tick 2024-09-16T04:06:01.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.557+0000 7f096fbdb640 1 -- 172.21.15.42:0/703552676 >> v2:172.21.15.154:6812/254845342 conn(0x7f094c037840 msgr2=0x7f094c039d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:01.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.557+0000 7f096fbdb640 1 --2- 172.21.15.42:0/703552676 >> v2:172.21.15.154:6812/254845342 conn(0x7f094c037840 0x7f094c039d00 secure :-1 s=READY pgs=3860 cs=0 l=1 rev1=1 crypto rx=0x7f09600ad000 tx=0x7f0960008040 comp rx=0 tx=0).stop 2024-09-16T04:06:01.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.557+0000 7f096fbdb640 10 monclient: shutdown 2024-09-16T04:06:01.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.557+0000 7f096fbdb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:06:01.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.557+0000 7f096fbdb640 1 -- 172.21.15.42:0/703552676 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f09680a39b0 msgr2=0x7f0968130660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:01.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.557+0000 7f096fbdb640 1 --2- 172.21.15.42:0/703552676 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f09680a39b0 0x7f0968130660 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f095c00c370 tx=0x7f095c00c830 comp rx=0 tx=0).stop 2024-09-16T04:06:01.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.557+0000 7f096e178640 1 -- 172.21.15.42:0/703552676 reap_dead start 2024-09-16T04:06:01.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.557+0000 7f096e178640 1 -- 172.21.15.42:0/703552676 reap_dead start 2024-09-16T04:06:01.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.557+0000 7f096fbdb640 1 -- 172.21.15.42:0/703552676 shutdown_connections 2024-09-16T04:06:01.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.557+0000 7f096fbdb640 1 -- 172.21.15.42:0/703552676 >> 172.21.15.42:0/703552676 conn(0x7f09680a9fb0 msgr2=0x7f0968132f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:06:01.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.557+0000 7f096fbdb640 1 -- 172.21.15.42:0/703552676 shutdown_connections 2024-09-16T04:06:01.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.557+0000 7f096fbdb640 1 -- 172.21.15.42:0/703552676 wait complete. 2024-09-16T04:06:01.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.557+0000 7f096fbdb640 1 librados: shutdown 2024-09-16T04:06:01.616 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:06:01.617 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T04:06:01.617 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T04:06:01.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.725+0000 7fe2211db640 10 monclient: get_monmap_and_config 2024-09-16T04:06:01.733 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.725+0000 7fe2211db640 10 monclient: build_initial_monmap 2024-09-16T04:06:01.733 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.725+0000 7fe2211db640 10 monclient: monmap: 2024-09-16T04:06:01.733 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:06:01.733 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:06:01.733 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:06:01.731092+0000 2024-09-16T04:06:01.733 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:06:01.731092+0000 2024-09-16T04:06:01.733 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:06:01.733 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:06:01.733 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:06:01.733 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:06:01.733 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:06:01.733 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:06:01.733 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:06:01.733 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:06:01.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.725+0000 7fe2211db640 1 Processor -- start 2024-09-16T04:06:01.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.725+0000 7fe2211db640 1 -- start start 2024-09-16T04:06:01.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.725+0000 7fe2211db640 10 monclient: init 2024-09-16T04:06:01.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.729+0000 7fe2211db640 10 monclient: _reopen_session rank -1 2024-09-16T04:06:01.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.729+0000 7fe2211db640 10 monclient: _add_conns ranks=[2,1,4,0,3] 2024-09-16T04:06:01.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.729+0000 7fe2211db640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe21c0a1330 0x7fe21c0a1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:06:01.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.729+0000 7fe2211db640 10 monclient(hunting): picked mon.noname-b con 0x7fe21c0a1330 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:06:01.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.729+0000 7fe2211db640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe21c06dac0 addr v1:172.21.15.154:6789/0 2024-09-16T04:06:01.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.729+0000 7fe2211db640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fe21c0a1c50 0x7fe21c135810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:06:01.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.729+0000 7fe2211db640 10 monclient(hunting): picked mon.noname-c con 0x7fe21c0a1c50 addr v2:172.21.15.154:3300/0 2024-09-16T04:06:01.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.729+0000 7fe2211db640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe21c09c090 con 0x7fe21c06dac0 2024-09-16T04:06:01.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.729+0000 7fe2211db640 10 monclient(hunting): start opening mon connection 2024-09-16T04:06:01.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.729+0000 7fe2211db640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fe21c09c890 con 0x7fe21c0a1c50 2024-09-16T04:06:01.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.729+0000 7fe2211db640 10 monclient(hunting): start opening mon connection 2024-09-16T04:06:01.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.729+0000 7fe2211db640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fe21c09f610 con 0x7fe21c0a1330 2024-09-16T04:06:01.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.729+0000 7fe2211db640 10 monclient(hunting): _renew_subs 2024-09-16T04:06:01.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.729+0000 7fe2211db640 10 monclient(hunting): authenticate will time out at 2408.700506s 2024-09-16T04:06:01.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.729+0000 7fe21ad9d640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe21c0a1330 0x7fe21c0a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:06:01.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.729+0000 7fe21ad9d640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe21c0a1330 0x7fe21c0a1710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:58270/0 (socket says 172.21.15.42:58270) 2024-09-16T04:06:01.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.729+0000 7fe21ad9d640 1 -- 172.21.15.42:0/2752333475 learned_addr learned my addr 172.21.15.42:0/2752333475 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:06:01.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.729+0000 7fe21ad9d640 10 monclient(hunting): get_auth_request con 0x7fe21c0a1330 auth_method 0 2024-09-16T04:06:01.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.729+0000 7fe21ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:06:01.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.729+0000 7fe21ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:01.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.729+0000 7fe21ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:01.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.729+0000 7fe21ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:06:01.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.729+0000 7fe21ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:06:01.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.729+0000 7fe21ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:06:01.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.729+0000 7fe21b59e640 1 --2- 172.21.15.42:0/2752333475 >> v2:172.21.15.154:3300/0 conn(0x7fe21c0a1c50 0x7fe21c135810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:06:01.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.729+0000 7fe21ad9d640 10 monclient(hunting): handle_auth_done global_id 1004126 payload 274 2024-09-16T04:06:01.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.729+0000 7fe21ad9d640 1 -- 172.21.15.42:0/2752333475 >> v2:172.21.15.154:3300/0 conn(0x7fe21c0a1c50 msgr2=0x7fe21c135810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:06:01.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.729+0000 7fe21ad9d640 1 --2- 172.21.15.42:0/2752333475 >> v2:172.21.15.154:3300/0 conn(0x7fe21c0a1c50 0x7fe21c135810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:06:01.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.729+0000 7fe21ad9d640 1 -- 172.21.15.42:0/2752333475 >> v1:172.21.15.154:6789/0 conn(0x7fe21c06dac0 legacy=0x7fe21c135100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:06:01.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.729+0000 7fe21ad9d640 10 monclient: _finish_hunting 0 2024-09-16T04:06:01.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.729+0000 7fe21ad9d640 1 monclient: found mon.noname-b 2024-09-16T04:06:01.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.729+0000 7fe21ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:06:01.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.729+0000 7fe21ad9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T04:06:01.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.729+0000 7fe21ad9d640 1 -- 172.21.15.42:0/2752333475 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe21c09f8f0 con 0x7fe21c0a1330 2024-09-16T04:06:01.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.729+0000 7fe21b59e640 10 monclient: get_auth_request con 0x7fe21c0a1c50 auth_method 0 2024-09-16T04:06:01.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.729+0000 7fe21b59e640 1 --2- 172.21.15.42:0/2752333475 >> v2:172.21.15.154:3300/0 conn(0x7fe21c0a1c50 0x7fe21c135810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T04:06:01.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.729+0000 7fe21ad9d640 1 --2- 172.21.15.42:0/2752333475 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe21c0a1330 0x7fe21c0a1710 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fe20c009870 tx=0x7fe20c01c440 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2876907cde3667f0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:06:01.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.729+0000 7fe219d9b640 1 -- 172.21.15.42:0/2752333475 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe20c0ab620 con 0x7fe21c0a1330 2024-09-16T04:06:01.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.729+0000 7fe219d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:01.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.729+0000 7fe219d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:06:01.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.729+0000 7fe219d9b640 10 monclient: dump: 2024-09-16T04:06:01.750 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:06:01.750 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:06:01.750 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:06:01.750 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:06:01.751 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:01.751 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:06:01.751 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:06:01.751 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:06:01.751 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:06:01.751 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:06:01.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.729+0000 7fe219d9b640 10 monclient: _finish_auth 0 2024-09-16T04:06:01.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.729+0000 7fe219d9b640 10 monclient: _check_auth_tickets 2024-09-16T04:06:01.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.729+0000 7fe219d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:06:01.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.729+0000 7fe219d9b640 1 -- 172.21.15.42:0/2752333475 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe20c0b0070 con 0x7fe21c0a1330 2024-09-16T04:06:01.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.729+0000 7fe219d9b640 10 monclient: handle_config config(0 keys) 2024-09-16T04:06:01.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.729+0000 7fe219d9b640 1 -- 172.21.15.42:0/2752333475 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe20c0ab9e0 con 0x7fe21c0a1330 2024-09-16T04:06:01.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.729+0000 7fe2211db640 5 monclient: authenticate success, global_id 1004126 2024-09-16T04:06:01.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.729+0000 7fe2211db640 10 monclient: get_monmap_and_config success 2024-09-16T04:06:01.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.729+0000 7fe2211db640 4 set_mon_vals no callback set 2024-09-16T04:06:01.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.729+0000 7fe219d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:01.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.729+0000 7fe219d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:06:01.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.729+0000 7fe219d9b640 10 monclient: dump: 2024-09-16T04:06:01.752 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:06:01.752 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:06:01.752 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:06:01.752 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:06:01.752 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:01.752 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:06:01.752 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:06:01.752 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:06:01.752 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:06:01.753 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:06:01.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.733+0000 7fe2211db640 10 monclient: shutdown 2024-09-16T04:06:01.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.733+0000 7fe2211db640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:06:01.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.733+0000 7fe2211db640 1 -- 172.21.15.42:0/2752333475 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe21c0a1330 msgr2=0x7fe21c0a1710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:01.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.733+0000 7fe2211db640 1 --2- 172.21.15.42:0/2752333475 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe21c0a1330 0x7fe21c0a1710 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fe20c009870 tx=0x7fe20c01c440 comp rx=0 tx=0).stop 2024-09-16T04:06:01.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.733+0000 7fe2211db640 1 -- 172.21.15.42:0/2752333475 shutdown_connections 2024-09-16T04:06:01.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.733+0000 7fe2211db640 1 --2- 172.21.15.42:0/2752333475 >> v2:172.21.15.154:3300/0 conn(0x7fe21c0a1c50 0x7fe21c135810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:06:01.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.733+0000 7fe2211db640 1 --2- 172.21.15.42:0/2752333475 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe21c0a1330 0x7fe21c0a1710 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:06:01.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.733+0000 7fe2211db640 1 -- 172.21.15.42:0/2752333475 >> 172.21.15.42:0/2752333475 conn(0x7fe21c0a2420 msgr2=0x7fe21c0a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:06:01.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.733+0000 7fe2211db640 1 -- 172.21.15.42:0/2752333475 shutdown_connections 2024-09-16T04:06:01.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.733+0000 7fe2211db640 1 -- 172.21.15.42:0/2752333475 wait complete. 2024-09-16T04:06:01.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.733+0000 7fe2211db640 10 monclient: build_initial_monmap 2024-09-16T04:06:01.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.733+0000 7fe2211db640 10 monclient: monmap: 2024-09-16T04:06:01.754 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:06:01.754 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:06:01.754 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:06:01.754 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:06:01.754 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:06:01.754 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:06:01.754 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:06:01.754 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:06:01.754 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:06:01.754 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:06:01.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.733+0000 7fe2211db640 1 librados: starting msgr at 2024-09-16T04:06:01.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.733+0000 7fe2211db640 1 librados: starting objecter 2024-09-16T04:06:01.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.733+0000 7fe2211db640 1 Processor -- start 2024-09-16T04:06:01.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.733+0000 7fe2211db640 1 -- start start 2024-09-16T04:06:01.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.733+0000 7fe2211db640 1 librados: setting wanted keys 2024-09-16T04:06:01.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.733+0000 7fe2211db640 1 librados: calling monclient init 2024-09-16T04:06:01.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.733+0000 7fe2211db640 10 monclient: init 2024-09-16T04:06:01.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.733+0000 7fe2211db640 10 monclient: _reopen_session rank -1 2024-09-16T04:06:01.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.733+0000 7fe2211db640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:06:01.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.733+0000 7fe2211db640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe21c0a1330 0x7fe21c1ccd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:06:01.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.733+0000 7fe2211db640 10 monclient(hunting): picked mon.noname-c con 0x7fe21c0a1330 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:06:01.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.733+0000 7fe2211db640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe21c0a1c50 0x7fe21c135180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:06:01.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.733+0000 7fe2211db640 10 monclient(hunting): picked mon.noname-b con 0x7fe21c0a1c50 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:06:01.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.733+0000 7fe2211db640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe21c1356c0 0x7fe21c11bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:06:01.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.733+0000 7fe2211db640 10 monclient(hunting): picked mon.noname-a con 0x7fe21c1356c0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:06:01.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.733+0000 7fe2211db640 10 monclient(hunting): start opening mon connection 2024-09-16T04:06:01.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.733+0000 7fe2211db640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fe21c09ef30 con 0x7fe21c1356c0 2024-09-16T04:06:01.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.733+0000 7fe2211db640 10 monclient(hunting): start opening mon connection 2024-09-16T04:06:01.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.733+0000 7fe2211db640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe21c09edb0 con 0x7fe21c0a1c50 2024-09-16T04:06:01.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.733+0000 7fe2211db640 10 monclient(hunting): start opening mon connection 2024-09-16T04:06:01.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.733+0000 7fe2211db640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fe21c135d50 con 0x7fe21c0a1330 2024-09-16T04:06:01.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.733+0000 7fe2211db640 10 monclient(hunting): _renew_subs 2024-09-16T04:06:01.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.733+0000 7fe2211db640 10 monclient(hunting): authenticate will time out at 2408.705963s 2024-09-16T04:06:01.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.733+0000 7fe21ad9d640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe21c0a1330 0x7fe21c1ccd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:06:01.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.733+0000 7fe21b59e640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe21c1356c0 0x7fe21c11bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:06:01.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.733+0000 7fe21ad9d640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe21c0a1330 0x7fe21c1ccd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:58278/0 (socket says 172.21.15.42:58278) 2024-09-16T04:06:01.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.733+0000 7fe21ad9d640 1 -- 172.21.15.42:0/1785569520 learned_addr learned my addr 172.21.15.42:0/1785569520 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:06:01.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.733+0000 7fe21ad9d640 10 monclient(hunting): get_auth_request con 0x7fe21c0a1330 auth_method 0 2024-09-16T04:06:01.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.733+0000 7fe21ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:06:01.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.733+0000 7fe21ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:01.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.733+0000 7fe21ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:01.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.733+0000 7fe21b59e640 10 monclient(hunting): get_auth_request con 0x7fe21c1356c0 auth_method 0 2024-09-16T04:06:01.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.733+0000 7fe21b59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:06:01.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.733+0000 7fe21b59e640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:01.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.733+0000 7fe21b59e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:01.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.733+0000 7fe21ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:06:01.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.733+0000 7fe21ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:06:01.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.733+0000 7fe21ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:06:01.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.733+0000 7fe21b59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:06:01.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.733+0000 7fe21b59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:06:01.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.733+0000 7fe21b59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:06:01.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.733+0000 7fe21ad9d640 10 monclient(hunting): handle_auth_done global_id 1004129 payload 824 2024-09-16T04:06:01.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.733+0000 7fe21ad9d640 1 -- 172.21.15.42:0/1785569520 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe21c0a1c50 msgr2=0x7fe21c135180 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-16T04:06:01.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.733+0000 7fe21ad9d640 1 --2- 172.21.15.42:0/1785569520 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe21c0a1c50 0x7fe21c135180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:06:01.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.733+0000 7fe21ad9d640 1 -- 172.21.15.42:0/1785569520 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe21c1356c0 msgr2=0x7fe21c11bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:01.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.733+0000 7fe21ad9d640 1 --2- 172.21.15.42:0/1785569520 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe21c1356c0 0x7fe21c11bf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:06:01.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.733+0000 7fe21ad9d640 10 monclient: _finish_hunting 0 2024-09-16T04:06:01.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.733+0000 7fe21ad9d640 1 monclient: found mon.noname-c 2024-09-16T04:06:01.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.733+0000 7fe21ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:06:01.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.733+0000 7fe21ad9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T04:06:01.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.733+0000 7fe21ad9d640 1 -- 172.21.15.42:0/1785569520 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe21c11f830 con 0x7fe21c0a1330 2024-09-16T04:06:01.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.733+0000 7fe21b59e640 1 --2- 172.21.15.42:0/1785569520 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe21c1356c0 0x7fe21c11bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:06:01.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.733+0000 7fe21ad9d640 1 --2- 172.21.15.42:0/1785569520 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe21c0a1330 0x7fe21c1ccd20 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fe20c01aaf0 tx=0x7fe20c01ac80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:06:01.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.733+0000 7fe2037fe640 10 client.?.objecter ms_handle_connect 0x7fe21c0a1330 2024-09-16T04:06:01.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.733+0000 7fe2037fe640 10 client.?.objecter resend_mon_ops 2024-09-16T04:06:01.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.733+0000 7fe2037fe640 1 -- 172.21.15.42:0/1785569520 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe20c0abd90 con 0x7fe21c0a1330 2024-09-16T04:06:01.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.733+0000 7fe2037fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:01.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.733+0000 7fe2037fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:06:01.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.733+0000 7fe2037fe640 10 monclient: dump: 2024-09-16T04:06:01.759 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:06:01.759 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:06:01.759 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:06:01.759 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:06:01.759 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:01.759 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:06:01.759 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:06:01.759 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:06:01.759 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:06:01.760 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:06:01.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.733+0000 7fe2037fe640 10 monclient: _finish_auth 0 2024-09-16T04:06:01.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.733+0000 7fe2037fe640 10 monclient: _check_auth_tickets 2024-09-16T04:06:01.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.733+0000 7fe2037fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:06:01.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.733+0000 7fe2037fe640 1 -- 172.21.15.42:0/1785569520 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe20c0be3f0 con 0x7fe21c0a1330 2024-09-16T04:06:01.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.733+0000 7fe2037fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:06:01.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.733+0000 7fe2037fe640 1 -- 172.21.15.42:0/1785569520 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe20c0be7e0 con 0x7fe21c0a1330 2024-09-16T04:06:01.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.733+0000 7fe2037fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:01.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.733+0000 7fe2037fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:06:01.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.733+0000 7fe2037fe640 10 monclient: dump: 2024-09-16T04:06:01.760 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:06:01.761 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:06:01.761 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:06:01.761 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:06:01.761 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:01.761 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:06:01.761 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:06:01.761 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:06:01.761 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:06:01.761 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:06:01.761 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.733+0000 7fe21959a640 4 set_mon_vals no callback set 2024-09-16T04:06:01.761 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.733+0000 7fe2211db640 5 monclient: authenticate success, global_id 1004129 2024-09-16T04:06:01.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.733+0000 7fe2211db640 10 monclient: _renew_subs 2024-09-16T04:06:01.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.733+0000 7fe2211db640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:06:01.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.733+0000 7fe2211db640 1 -- 172.21.15.42:0/1785569520 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe21c11fb10 con 0x7fe21c0a1330 2024-09-16T04:06:01.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.733+0000 7fe2211db640 10 client.1004129.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:06:01.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.733+0000 7fe2211db640 10 monclient: _renew_subs 2024-09-16T04:06:01.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.733+0000 7fe2211db640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:06:01.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.733+0000 7fe2211db640 1 -- 172.21.15.42:0/1785569520 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fe21c06e740 con 0x7fe21c0a1330 2024-09-16T04:06:01.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.733+0000 7fe2211db640 1 librados: init done 2024-09-16T04:06:01.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.737+0000 7fe2211db640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:06:01.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.737+0000 7fe2211db640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:06:01.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.737+0000 7fe2037fe640 1 -- 172.21.15.42:0/1785569520 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fe20c0c7440 con 0x7fe21c0a1330 2024-09-16T04:06:01.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.737+0000 7fe2211db640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:06:01.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.737+0000 7fe2211db640 1 -- 172.21.15.42:0/1785569520 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe21c11e5a0 con 0x7fe21c0a1330 2024-09-16T04:06:01.763 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.737+0000 7fe2037fe640 1 --2- 172.21.15.42:0/1785569520 >> v2:172.21.15.154:6812/254845342 conn(0x7fe208037810 0x7fe208039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:06:01.763 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.737+0000 7fe2037fe640 1 -- 172.21.15.42:0/1785569520 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(706..706 src has 451..706) ==== 5040+0+0 (secure 0 0 0) 0x7fe20c0e3230 con 0x7fe21c0a1330 2024-09-16T04:06:01.763 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.737+0000 7fe2037fe640 10 client.1004129.objecter ms_dispatch 0x7fe21c000ba0 osd_map(706..706 src has 451..706) 2024-09-16T04:06:01.763 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.737+0000 7fe2037fe640 3 client.1004129.objecter handle_osd_map got epochs [706,706] > 0 2024-09-16T04:06:01.763 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.737+0000 7fe2037fe640 3 client.1004129.objecter handle_osd_map decoding full epoch 706 2024-09-16T04:06:01.763 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.737+0000 7fe2037fe640 20 client.1004129.objecter dump_active .. 0 homeless 2024-09-16T04:06:01.763 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.737+0000 7fe21a59c640 1 --2- 172.21.15.42:0/1785569520 >> v2:172.21.15.154:6812/254845342 conn(0x7fe208037810 0x7fe208039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:06:01.763 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.737+0000 7fe21a59c640 10 monclient: get_auth_request con 0x7fe208037810 auth_method 0 2024-09-16T04:06:01.763 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.737+0000 7fe21a59c640 1 --2- 172.21.15.42:0/1785569520 >> v2:172.21.15.154:6812/254845342 conn(0x7fe208037810 0x7fe208039cd0 secure :-1 s=READY pgs=3861 cs=0 l=1 rev1=1 crypto rx=0x7fe21c11d540 tx=0x7fe2100079c0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:06:01.763 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.737+0000 7fe2037fe640 10 client.1004129.objecter ms_handle_connect 0x7fe208037810 2024-09-16T04:06:01.764 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.741+0000 7fe2037fe640 1 -- 172.21.15.42:0/1785569520 <== mon.2 v2:172.21.15.42:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7fe20c0160f0 con 0x7fe21c0a1330 2024-09-16T04:06:01.764 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.741+0000 7fe2037fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:06:01.764 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.741+0000 7fe2037fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:06:01.973 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.965+0000 7fe2211db640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T04:06:01.973 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.965+0000 7fe2211db640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T04:06:01.973 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.965+0000 7fe2211db640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:06:01.973 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.965+0000 7fe2211db640 1 -- 172.21.15.42:0/1785569520 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7fe21c0a1710 con 0x7fe21c0a1330 2024-09-16T04:06:01.978 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:01.973+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:01.978 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:01.973+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=27,config=2,health=11,kv=9,logm=51} crc {auth=3937957330,config=692619927,health=4271281240,kv=3915008329,logm=3614655599}) 2024-09-16T04:06:01.978 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:01.973+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=26,config=2,health=11,kv=9,logm=52} crc {auth=2655264590,config=692619927,health=2911090982,kv=3276479304,logm=4054740629}) 2024-09-16T04:06:01.978 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:01.973+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:01.978 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:01.973+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=27,config=2,health=11,kv=9,logm=51} crc {auth=3937957330,config=692619927,health=4271281240,kv=3915008329,logm=3614655599}) 2024-09-16T04:06:01.978 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:01.973+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=26,config=2,health=11,kv=10,logm=51} crc {auth=3932490496,config=692619927,health=2911090982,kv=3446188498,logm=1504406628}) 2024-09-16T04:06:01.979 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.973+0000 7fe2037fe640 1 -- 172.21.15.42:0/1785569520 <== mon.2 v2:172.21.15.42:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7fe20c0b43f0 con 0x7fe21c0a1330 2024-09-16T04:06:01.979 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.973+0000 7fe2037fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T04:06:01.979 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.973+0000 7fe2037fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:06:01.982 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:01.977+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:01.982 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:01.977+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2230402583}) 2024-09-16T04:06:01.982 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:01.977+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=852321405}) 2024-09-16T04:06:01.982 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:01.977+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:01.982 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:01.977+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2230402583}) 2024-09-16T04:06:01.982 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:01.977+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=273738619}) 2024-09-16T04:06:01.984 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.977+0000 7fe2211db640 10 librados: watch_flush enter 2024-09-16T04:06:01.984 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.977+0000 7fe2211db640 10 librados: watch_flush exit 2024-09-16T04:06:01.984 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.977+0000 7fe2211db640 20 client.1004129.objecter shutdown clearing up homeless session... 2024-09-16T04:06:01.984 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.977+0000 7fe2211db640 10 client.1004129.objecter successfully canceled tick 2024-09-16T04:06:01.984 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.977+0000 7fe2211db640 1 -- 172.21.15.42:0/1785569520 >> v2:172.21.15.154:6812/254845342 conn(0x7fe208037810 msgr2=0x7fe208039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:01.984 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.977+0000 7fe2211db640 1 --2- 172.21.15.42:0/1785569520 >> v2:172.21.15.154:6812/254845342 conn(0x7fe208037810 0x7fe208039cd0 secure :-1 s=READY pgs=3861 cs=0 l=1 rev1=1 crypto rx=0x7fe21c11d540 tx=0x7fe2100079c0 comp rx=0 tx=0).stop 2024-09-16T04:06:01.984 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.977+0000 7fe2211db640 10 monclient: shutdown 2024-09-16T04:06:01.984 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.977+0000 7fe2211db640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:06:01.984 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.977+0000 7fe2211db640 1 -- 172.21.15.42:0/1785569520 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe21c0a1330 msgr2=0x7fe21c1ccd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:01.984 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.977+0000 7fe2211db640 1 --2- 172.21.15.42:0/1785569520 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe21c0a1330 0x7fe21c1ccd20 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fe20c01aaf0 tx=0x7fe20c01ac80 comp rx=0 tx=0).stop 2024-09-16T04:06:01.984 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.977+0000 7fe2211db640 1 -- 172.21.15.42:0/1785569520 shutdown_connections 2024-09-16T04:06:01.984 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.977+0000 7fe2211db640 1 --2- 172.21.15.42:0/1785569520 >> v2:172.21.15.154:6812/254845342 conn(0x7fe208037810 0x7fe208039cd0 unknown :-1 s=CLOSED pgs=3861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:06:01.985 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.977+0000 7fe2211db640 1 --2- 172.21.15.42:0/1785569520 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe21c1356c0 0x7fe21c11bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:06:01.985 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.977+0000 7fe2211db640 1 --2- 172.21.15.42:0/1785569520 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe21c0a1c50 0x7fe21c135180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:06:01.985 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.977+0000 7fe2211db640 1 --2- 172.21.15.42:0/1785569520 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe21c0a1330 0x7fe21c1ccd20 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:06:01.985 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.977+0000 7fe2211db640 1 -- 172.21.15.42:0/1785569520 >> 172.21.15.42:0/1785569520 conn(0x7fe21c0a2420 msgr2=0x7fe21c071880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:06:01.985 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.977+0000 7fe2211db640 1 -- 172.21.15.42:0/1785569520 shutdown_connections 2024-09-16T04:06:01.985 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.977+0000 7fe2211db640 1 -- 172.21.15.42:0/1785569520 wait complete. 2024-09-16T04:06:01.985 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:01.977+0000 7fe2211db640 1 librados: shutdown 2024-09-16T04:06:01.986 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:01.981+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:01.987 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:01.981+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1061193084}) 2024-09-16T04:06:01.987 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:01.981+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3917192812}) 2024-09-16T04:06:01.987 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:01.981+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:01.987 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:01.981+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1061193084}) 2024-09-16T04:06:01.987 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:01.981+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2136932056}) 2024-09-16T04:06:01.992 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:01.985+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:01.992 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:01.985+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1549034038}) 2024-09-16T04:06:01.992 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:01.985+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=196669768}) 2024-09-16T04:06:01.992 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:01.985+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:01.992 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:01.985+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1549034038}) 2024-09-16T04:06:01.992 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:01.985+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2649108898}) 2024-09-16T04:06:01.996 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:01.989+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:01.996 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:01.989+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1004848474}) 2024-09-16T04:06:01.996 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:01.989+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3398954790}) 2024-09-16T04:06:01.996 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:01.989+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:01.996 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:01.989+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1004848474}) 2024-09-16T04:06:01.996 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:01.989+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2013085986}) 2024-09-16T04:06:02.001 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T04:06:02.003 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:01.997+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.003 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:01.997+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2313469064}) 2024-09-16T04:06:02.003 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:01.997+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1908083208}) 2024-09-16T04:06:02.004 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:01.997+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.004 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:01.997+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2313469064}) 2024-09-16T04:06:02.004 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:01.997+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=889073996}) 2024-09-16T04:06:02.005 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:01.997+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.005 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:01.997+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3427717592}) 2024-09-16T04:06:02.005 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:01.997+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=53160626}) 2024-09-16T04:06:02.005 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:01.997+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.005 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:01.997+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3427717592}) 2024-09-16T04:06:02.005 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:01.997+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3550574422}) 2024-09-16T04:06:02.007 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.001+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.007 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.001+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3769664828}) 2024-09-16T04:06:02.007 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.001+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3643805203}) 2024-09-16T04:06:02.007 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.001+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.007 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.001+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3769664828}) 2024-09-16T04:06:02.007 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.001+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1625056534}) 2024-09-16T04:06:02.009 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.001+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.009 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.001+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2239867180}) 2024-09-16T04:06:02.009 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.001+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3926479474}) 2024-09-16T04:06:02.009 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.001+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.009 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.001+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2239867180}) 2024-09-16T04:06:02.010 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.001+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=209288549}) 2024-09-16T04:06:02.011 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.005+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.011 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.005+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=301010687}) 2024-09-16T04:06:02.011 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.005+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3533069995}) 2024-09-16T04:06:02.011 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.005+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.011 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.005+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=301010687}) 2024-09-16T04:06:02.011 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.005+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1623345178}) 2024-09-16T04:06:02.013 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.005+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.013 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.005+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1843279083}) 2024-09-16T04:06:02.013 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.005+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3961297275}) 2024-09-16T04:06:02.013 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.005+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.013 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.005+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1843279083}) 2024-09-16T04:06:02.013 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.005+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3859327373}) 2024-09-16T04:06:02.014 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.009+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.014 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.009+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1351983832}) 2024-09-16T04:06:02.015 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.009+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3221290109}) 2024-09-16T04:06:02.015 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.009+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.015 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.009+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1351983832}) 2024-09-16T04:06:02.015 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.009+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2342752029}) 2024-09-16T04:06:02.016 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.009+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.016 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.009+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2049418458}) 2024-09-16T04:06:02.016 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.009+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3439915899}) 2024-09-16T04:06:02.016 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.009+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.017 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.009+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2049418458}) 2024-09-16T04:06:02.017 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.009+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4086204145}) 2024-09-16T04:06:02.018 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.013+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.018 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.013+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1034374555}) 2024-09-16T04:06:02.018 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.013+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3409763857}) 2024-09-16T04:06:02.018 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.013+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.018 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.013+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1034374555}) 2024-09-16T04:06:02.019 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.013+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1109205053}) 2024-09-16T04:06:02.020 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.013+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.020 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.013+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3562873578}) 2024-09-16T04:06:02.021 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.013+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1071607423}) 2024-09-16T04:06:02.021 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.013+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.021 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.013+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3562873578}) 2024-09-16T04:06:02.021 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.013+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1616464694}) 2024-09-16T04:06:02.022 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.017+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.022 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.017+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=717630220}) 2024-09-16T04:06:02.022 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.017+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1568786243}) 2024-09-16T04:06:02.023 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.017+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.023 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.017+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=717630220}) 2024-09-16T04:06:02.023 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.017+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1288313324}) 2024-09-16T04:06:02.024 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.017+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.024 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.017+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1637521731}) 2024-09-16T04:06:02.024 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.017+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=717602539}) 2024-09-16T04:06:02.025 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.017+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.025 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.017+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1637521731}) 2024-09-16T04:06:02.025 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.017+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1638273198}) 2024-09-16T04:06:02.026 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.021+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.026 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.021+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2014802378}) 2024-09-16T04:06:02.026 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.021+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3956295937}) 2024-09-16T04:06:02.026 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.021+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.026 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.021+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2014802378}) 2024-09-16T04:06:02.027 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.021+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=959428237}) 2024-09-16T04:06:02.028 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.021+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.028 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.021+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3499919092}) 2024-09-16T04:06:02.029 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.021+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1983673689}) 2024-09-16T04:06:02.029 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.021+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.029 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.021+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3499919092}) 2024-09-16T04:06:02.029 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.021+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2572397423}) 2024-09-16T04:06:02.030 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.025+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.030 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.025+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=163129160}) 2024-09-16T04:06:02.030 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.025+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3021839463}) 2024-09-16T04:06:02.030 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.025+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.031 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.025+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=163129160}) 2024-09-16T04:06:02.031 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.025+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3725109383}) 2024-09-16T04:06:02.032 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.025+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.032 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.025+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2643166639}) 2024-09-16T04:06:02.032 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.025+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2663647527}) 2024-09-16T04:06:02.032 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.025+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.033 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.025+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2643166639}) 2024-09-16T04:06:02.033 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.025+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=259741526}) 2024-09-16T04:06:02.034 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.029+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.034 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.029+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=205363520}) 2024-09-16T04:06:02.034 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.029+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4087733553}) 2024-09-16T04:06:02.034 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.029+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.034 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.029+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=205363520}) 2024-09-16T04:06:02.035 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.029+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2224299175}) 2024-09-16T04:06:02.036 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.029+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.036 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.029+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3271997380}) 2024-09-16T04:06:02.036 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.029+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3957377648}) 2024-09-16T04:06:02.036 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.029+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.036 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.029+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3271997380}) 2024-09-16T04:06:02.037 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.029+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=98695749}) 2024-09-16T04:06:02.038 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.033+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.038 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.033+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2553547171}) 2024-09-16T04:06:02.038 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.033+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=30005363}) 2024-09-16T04:06:02.038 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.033+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.038 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.033+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2553547171}) 2024-09-16T04:06:02.038 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.033+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=897955126}) 2024-09-16T04:06:02.040 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.033+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.040 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.033+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3027405030}) 2024-09-16T04:06:02.040 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.033+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1613056135}) 2024-09-16T04:06:02.040 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.033+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.040 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.033+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3027405030}) 2024-09-16T04:06:02.040 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.033+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=161135145}) 2024-09-16T04:06:02.042 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.037+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.042 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.037+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2184988223}) 2024-09-16T04:06:02.042 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.037+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3649688548}) 2024-09-16T04:06:02.042 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.037+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.042 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.037+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2184988223}) 2024-09-16T04:06:02.042 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.037+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=489993583}) 2024-09-16T04:06:02.044 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.037+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.044 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.037+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=366334518}) 2024-09-16T04:06:02.044 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.037+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3308111708}) 2024-09-16T04:06:02.044 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.037+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.044 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.037+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=366334518}) 2024-09-16T04:06:02.044 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.037+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3476896138}) 2024-09-16T04:06:02.046 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.041+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.046 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.041+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2881482526}) 2024-09-16T04:06:02.046 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.041+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2556558272}) 2024-09-16T04:06:02.046 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.041+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.046 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.041+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2881482526}) 2024-09-16T04:06:02.047 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.041+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2171474788}) 2024-09-16T04:06:02.048 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.041+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.048 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.041+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2666812165}) 2024-09-16T04:06:02.048 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.041+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2988580893}) 2024-09-16T04:06:02.048 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.041+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.048 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.041+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2666812165}) 2024-09-16T04:06:02.048 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.041+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2880632432}) 2024-09-16T04:06:02.050 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.045+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.050 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.045+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4193330076}) 2024-09-16T04:06:02.050 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.045+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3016095524}) 2024-09-16T04:06:02.050 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.045+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.050 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.045+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4193330076}) 2024-09-16T04:06:02.050 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.045+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=11311663}) 2024-09-16T04:06:02.051 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.045+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.052 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.045+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=882078217}) 2024-09-16T04:06:02.052 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.045+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2074805372}) 2024-09-16T04:06:02.052 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.045+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.052 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.045+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=882078217}) 2024-09-16T04:06:02.052 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.045+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2794191397}) 2024-09-16T04:06:02.053 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.049+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.053 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.049+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1449080857}) 2024-09-16T04:06:02.054 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.049+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=288234134}) 2024-09-16T04:06:02.054 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.049+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.054 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.049+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1449080857}) 2024-09-16T04:06:02.054 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.049+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=991360698}) 2024-09-16T04:06:02.055 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.049+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.055 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.049+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2085959085}) 2024-09-16T04:06:02.055 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.049+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3466602831}) 2024-09-16T04:06:02.055 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.049+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.055 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.049+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2085959085}) 2024-09-16T04:06:02.055 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.049+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=767587850}) 2024-09-16T04:06:02.057 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.049+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.057 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.049+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2403633905}) 2024-09-16T04:06:02.057 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.049+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3898297550}) 2024-09-16T04:06:02.057 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.049+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.057 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.049+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2403633905}) 2024-09-16T04:06:02.057 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.049+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3161882586}) 2024-09-16T04:06:02.059 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.053+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.059 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.053+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=409592371}) 2024-09-16T04:06:02.059 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.053+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2599600381}) 2024-09-16T04:06:02.059 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.053+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.059 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.053+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=409592371}) 2024-09-16T04:06:02.059 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.053+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1549688731}) 2024-09-16T04:06:02.061 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.057+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.061 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.057+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1685361586}) 2024-09-16T04:06:02.062 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.057+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=737592979}) 2024-09-16T04:06:02.062 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.057+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.062 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.057+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1685361586}) 2024-09-16T04:06:02.062 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.057+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4133358435}) 2024-09-16T04:06:02.063 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.057+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.063 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.057+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4135056332}) 2024-09-16T04:06:02.063 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.057+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4278530518}) 2024-09-16T04:06:02.063 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.057+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.063 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.057+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4135056332}) 2024-09-16T04:06:02.064 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.057+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=49751628}) 2024-09-16T04:06:02.065 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.057+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.065 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.057+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=957775796}) 2024-09-16T04:06:02.065 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.057+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=302997476}) 2024-09-16T04:06:02.065 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.057+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.065 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.057+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=957775796}) 2024-09-16T04:06:02.065 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.057+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1509134141}) 2024-09-16T04:06:02.067 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.061+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.067 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.061+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3932898704}) 2024-09-16T04:06:02.067 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.061+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2200512067}) 2024-09-16T04:06:02.067 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.061+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.067 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.061+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3932898704}) 2024-09-16T04:06:02.067 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.061+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2492108076}) 2024-09-16T04:06:02.069 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.061+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.069 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.061+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1292889709}) 2024-09-16T04:06:02.069 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.061+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2549512443}) 2024-09-16T04:06:02.069 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.061+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.069 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.061+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1292889709}) 2024-09-16T04:06:02.069 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.061+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1815653161}) 2024-09-16T04:06:02.071 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.065+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.071 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.065+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3657458462}) 2024-09-16T04:06:02.071 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.065+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=644431346}) 2024-09-16T04:06:02.071 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.065+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.071 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.065+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3657458462}) 2024-09-16T04:06:02.071 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.065+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3100654565}) 2024-09-16T04:06:02.072 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.065+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.072 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.065+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2966550963}) 2024-09-16T04:06:02.073 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.065+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=109288429}) 2024-09-16T04:06:02.073 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.065+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.073 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.065+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2966550963}) 2024-09-16T04:06:02.073 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.065+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4127113450}) 2024-09-16T04:06:02.074 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.069+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.074 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.069+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2945708666}) 2024-09-16T04:06:02.074 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.069+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4209587376}) 2024-09-16T04:06:02.074 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.069+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.075 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.069+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2945708666}) 2024-09-16T04:06:02.075 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.069+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3555097226}) 2024-09-16T04:06:02.076 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.069+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.076 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.069+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3796943551}) 2024-09-16T04:06:02.076 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.069+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2905800838}) 2024-09-16T04:06:02.076 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.069+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.076 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.069+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3796943551}) 2024-09-16T04:06:02.077 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.069+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1024394533}) 2024-09-16T04:06:02.078 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.073+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.078 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.073+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3616627189}) 2024-09-16T04:06:02.078 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.073+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3574463939}) 2024-09-16T04:06:02.078 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.073+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.078 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.073+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3616627189}) 2024-09-16T04:06:02.078 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.073+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2365489809}) 2024-09-16T04:06:02.080 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.073+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.080 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.073+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1092894779}) 2024-09-16T04:06:02.080 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.073+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1186822155}) 2024-09-16T04:06:02.080 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.073+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.080 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.073+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1092894779}) 2024-09-16T04:06:02.080 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.073+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3258791881}) 2024-09-16T04:06:02.082 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.077+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.082 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.077+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1786206414}) 2024-09-16T04:06:02.082 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.077+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2992239316}) 2024-09-16T04:06:02.082 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.077+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.082 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.077+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1786206414}) 2024-09-16T04:06:02.082 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.077+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=66693977}) 2024-09-16T04:06:02.084 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.077+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.084 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.077+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1455573686}) 2024-09-16T04:06:02.084 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.077+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3083891562}) 2024-09-16T04:06:02.084 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.077+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.084 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.077+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1455573686}) 2024-09-16T04:06:02.084 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.077+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1947958366}) 2024-09-16T04:06:02.086 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.081+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.086 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.081+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4210552740}) 2024-09-16T04:06:02.086 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.081+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2984551874}) 2024-09-16T04:06:02.086 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.081+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.086 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.081+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4210552740}) 2024-09-16T04:06:02.086 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.081+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4188614168}) 2024-09-16T04:06:02.088 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.081+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.088 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.081+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3973081446}) 2024-09-16T04:06:02.088 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.081+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=51592956}) 2024-09-16T04:06:02.088 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.081+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.088 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.081+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3973081446}) 2024-09-16T04:06:02.088 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.081+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1231405147}) 2024-09-16T04:06:02.089 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.085+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.089 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.085+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1863622073}) 2024-09-16T04:06:02.089 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.085+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3387409102}) 2024-09-16T04:06:02.090 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.085+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.090 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.085+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1863622073}) 2024-09-16T04:06:02.090 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.085+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=331455937}) 2024-09-16T04:06:02.091 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.085+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.091 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.085+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2615481896}) 2024-09-16T04:06:02.091 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.085+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2553849812}) 2024-09-16T04:06:02.092 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.085+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.092 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.085+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2615481896}) 2024-09-16T04:06:02.092 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.085+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1726826446}) 2024-09-16T04:06:02.093 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.085+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.093 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.085+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4146093477}) 2024-09-16T04:06:02.093 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.085+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3664207258}) 2024-09-16T04:06:02.093 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.085+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.093 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.085+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4146093477}) 2024-09-16T04:06:02.093 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.085+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2881782812}) 2024-09-16T04:06:02.095 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.089+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.095 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.089+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2024324556}) 2024-09-16T04:06:02.095 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.089+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4334668}) 2024-09-16T04:06:02.095 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.089+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.095 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.089+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2024324556}) 2024-09-16T04:06:02.095 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.089+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1475329986}) 2024-09-16T04:06:02.096 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.089+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.096 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.089+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2775855758}) 2024-09-16T04:06:02.097 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.089+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1280006140}) 2024-09-16T04:06:02.097 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.089+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.097 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.089+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2775855758}) 2024-09-16T04:06:02.097 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.089+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3505081971}) 2024-09-16T04:06:02.098 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.093+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.098 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.093+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1075879632}) 2024-09-16T04:06:02.098 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.093+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1237295956}) 2024-09-16T04:06:02.098 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.093+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.099 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.093+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1075879632}) 2024-09-16T04:06:02.099 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.093+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1090831311}) 2024-09-16T04:06:02.100 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.093+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.100 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.093+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3764299541}) 2024-09-16T04:06:02.100 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.093+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1477363819}) 2024-09-16T04:06:02.100 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.093+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.100 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.093+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3764299541}) 2024-09-16T04:06:02.100 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.093+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=446107542}) 2024-09-16T04:06:02.102 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.097+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.102 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.097+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=237804603}) 2024-09-16T04:06:02.102 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.097+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2650774268}) 2024-09-16T04:06:02.102 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.097+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.102 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.097+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=237804603}) 2024-09-16T04:06:02.102 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.097+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2791658999}) 2024-09-16T04:06:02.104 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.097+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.104 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.097+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=81887333}) 2024-09-16T04:06:02.104 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.097+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1669654740}) 2024-09-16T04:06:02.104 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.097+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.104 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.097+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=81887333}) 2024-09-16T04:06:02.104 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.097+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3460141492}) 2024-09-16T04:06:02.105 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.101+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.106 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.101+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1183887863}) 2024-09-16T04:06:02.106 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.101+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=872877203}) 2024-09-16T04:06:02.106 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.101+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.106 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.101+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1183887863}) 2024-09-16T04:06:02.106 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.101+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4195808392}) 2024-09-16T04:06:02.107 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.101+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.107 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.101+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3463717665}) 2024-09-16T04:06:02.107 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.101+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3664236903}) 2024-09-16T04:06:02.108 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.101+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.108 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.101+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3463717665}) 2024-09-16T04:06:02.108 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.101+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2934261375}) 2024-09-16T04:06:02.109 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.101+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.109 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.101+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=407725592}) 2024-09-16T04:06:02.109 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.101+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2346692675}) 2024-09-16T04:06:02.109 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.101+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.109 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.101+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=407725592}) 2024-09-16T04:06:02.109 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.101+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4197603336}) 2024-09-16T04:06:02.111 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.105+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.111 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.105+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=926641818}) 2024-09-16T04:06:02.111 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.105+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4177729295}) 2024-09-16T04:06:02.111 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.105+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.111 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.105+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=926641818}) 2024-09-16T04:06:02.111 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.105+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1520108145}) 2024-09-16T04:06:02.113 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.105+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.113 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.105+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2997414001}) 2024-09-16T04:06:02.113 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.105+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=462152779}) 2024-09-16T04:06:02.113 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.105+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.113 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.105+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2997414001}) 2024-09-16T04:06:02.113 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.105+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=756227889}) 2024-09-16T04:06:02.114 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.109+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.115 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.109+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4219345582}) 2024-09-16T04:06:02.115 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.109+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1323029627}) 2024-09-16T04:06:02.115 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.109+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.115 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.109+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4219345582}) 2024-09-16T04:06:02.115 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.109+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=824961366}) 2024-09-16T04:06:02.116 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.109+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.116 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.109+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3891650017}) 2024-09-16T04:06:02.116 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.109+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1114807692}) 2024-09-16T04:06:02.116 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.109+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.116 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.109+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3891650017}) 2024-09-16T04:06:02.117 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.109+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2268699023}) 2024-09-16T04:06:02.118 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.113+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.118 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.113+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4071285587}) 2024-09-16T04:06:02.118 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.113+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3635470561}) 2024-09-16T04:06:02.118 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.113+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.118 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.113+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4071285587}) 2024-09-16T04:06:02.118 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.113+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2204351962}) 2024-09-16T04:06:02.120 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.113+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.120 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.113+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2483695127}) 2024-09-16T04:06:02.120 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.113+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=649825421}) 2024-09-16T04:06:02.120 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.113+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.120 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.113+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2483695127}) 2024-09-16T04:06:02.120 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.113+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3332775911}) 2024-09-16T04:06:02.121 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.117+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.122 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.117+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2111523034}) 2024-09-16T04:06:02.122 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.117+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2710561524}) 2024-09-16T04:06:02.122 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.117+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.122 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.117+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2111523034}) 2024-09-16T04:06:02.122 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.117+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=512919273}) 2024-09-16T04:06:02.123 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.117+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.123 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.117+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=288842286}) 2024-09-16T04:06:02.123 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.117+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2526315144}) 2024-09-16T04:06:02.123 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.117+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.123 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.117+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=288842286}) 2024-09-16T04:06:02.123 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.117+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=639282867}) 2024-09-16T04:06:02.125 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.117+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.125 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.117+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1716498767}) 2024-09-16T04:06:02.125 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.117+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=447288729}) 2024-09-16T04:06:02.125 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.117+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.125 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.117+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1716498767}) 2024-09-16T04:06:02.125 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.117+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1415983392}) 2024-09-16T04:06:02.127 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.121+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.127 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.121+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2658614031}) 2024-09-16T04:06:02.127 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.121+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3327884793}) 2024-09-16T04:06:02.127 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.121+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.127 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.121+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2658614031}) 2024-09-16T04:06:02.127 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.121+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2576760576}) 2024-09-16T04:06:02.128 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.121+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.128 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.121+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3638193550}) 2024-09-16T04:06:02.129 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.121+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1387965279}) 2024-09-16T04:06:02.129 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.121+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.129 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.121+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3638193550}) 2024-09-16T04:06:02.129 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.121+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1229622405}) 2024-09-16T04:06:02.130 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.125+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.130 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.125+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2345569987}) 2024-09-16T04:06:02.130 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.125+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2984546278}) 2024-09-16T04:06:02.130 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.125+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.130 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.125+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2345569987}) 2024-09-16T04:06:02.130 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.125+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4173542970}) 2024-09-16T04:06:02.132 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.125+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.132 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.125+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3883166801}) 2024-09-16T04:06:02.132 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.125+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3334319907}) 2024-09-16T04:06:02.132 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.125+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.132 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.125+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3883166801}) 2024-09-16T04:06:02.132 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.125+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2456649610}) 2024-09-16T04:06:02.133 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.129+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.133 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.129+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4110884863}) 2024-09-16T04:06:02.134 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.129+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2268300771}) 2024-09-16T04:06:02.134 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.129+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.134 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.129+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4110884863}) 2024-09-16T04:06:02.134 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.129+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2183436069}) 2024-09-16T04:06:02.135 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.129+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.135 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.129+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3525695852}) 2024-09-16T04:06:02.135 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.129+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2293257015}) 2024-09-16T04:06:02.135 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.129+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.135 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.129+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3525695852}) 2024-09-16T04:06:02.135 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.129+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2851729675}) 2024-09-16T04:06:02.137 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.129+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.137 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.129+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1161168154}) 2024-09-16T04:06:02.137 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.129+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1374026240}) 2024-09-16T04:06:02.137 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.129+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.137 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.129+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1161168154}) 2024-09-16T04:06:02.137 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.129+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1530436338}) 2024-09-16T04:06:02.138 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.133+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.139 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.133+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2211058341}) 2024-09-16T04:06:02.139 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.133+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1113614075}) 2024-09-16T04:06:02.139 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.133+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.139 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.133+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2211058341}) 2024-09-16T04:06:02.139 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.133+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2402857441}) 2024-09-16T04:06:02.140 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.133+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.140 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.133+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3496353202}) 2024-09-16T04:06:02.140 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.133+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4226572780}) 2024-09-16T04:06:02.140 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.133+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.140 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.133+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3496353202}) 2024-09-16T04:06:02.141 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.133+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=823831021}) 2024-09-16T04:06:02.142 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.137+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.142 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.137+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4100079759}) 2024-09-16T04:06:02.142 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.137+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=952851509}) 2024-09-16T04:06:02.142 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.137+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.142 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.137+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4100079759}) 2024-09-16T04:06:02.142 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.137+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=228369886}) 2024-09-16T04:06:02.144 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.137+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.144 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.137+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3877613396}) 2024-09-16T04:06:02.144 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.137+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2953932808}) 2024-09-16T04:06:02.144 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.137+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.144 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.137+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3877613396}) 2024-09-16T04:06:02.144 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.137+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3568010570}) 2024-09-16T04:06:02.145 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.141+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.146 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.141+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1367244623}) 2024-09-16T04:06:02.146 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.141+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2635797492}) 2024-09-16T04:06:02.146 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.141+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.146 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.141+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1367244623}) 2024-09-16T04:06:02.146 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.141+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=703212575}) 2024-09-16T04:06:02.147 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.141+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.147 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.141+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4086003631}) 2024-09-16T04:06:02.147 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.141+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=255288185}) 2024-09-16T04:06:02.147 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.141+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.147 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.141+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4086003631}) 2024-09-16T04:06:02.147 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.141+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4039838154}) 2024-09-16T04:06:02.149 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.141+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.149 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.141+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1996545457}) 2024-09-16T04:06:02.149 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.141+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=823483021}) 2024-09-16T04:06:02.149 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.141+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.149 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.141+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1996545457}) 2024-09-16T04:06:02.149 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.141+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3543502682}) 2024-09-16T04:06:02.151 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.145+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.151 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.145+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=463986290}) 2024-09-16T04:06:02.151 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.145+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4088621486}) 2024-09-16T04:06:02.151 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.145+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.151 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.145+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=463986290}) 2024-09-16T04:06:02.151 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.145+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=719406407}) 2024-09-16T04:06:02.153 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.145+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.153 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.145+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2402899995}) 2024-09-16T04:06:02.153 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.145+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1700695202}) 2024-09-16T04:06:02.153 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.145+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.153 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.145+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2402899995}) 2024-09-16T04:06:02.153 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.145+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=453622796}) 2024-09-16T04:06:02.154 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.149+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.154 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.149+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=687361523}) 2024-09-16T04:06:02.155 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.149+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3754187760}) 2024-09-16T04:06:02.155 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.149+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.155 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.149+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=687361523}) 2024-09-16T04:06:02.155 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.149+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4214608780}) 2024-09-16T04:06:02.156 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.149+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.156 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.149+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3781812708}) 2024-09-16T04:06:02.157 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.149+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=662512542}) 2024-09-16T04:06:02.157 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.149+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.157 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.149+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3781812708}) 2024-09-16T04:06:02.157 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.149+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3974954021}) 2024-09-16T04:06:02.158 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.153+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.158 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.153+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2526165714}) 2024-09-16T04:06:02.158 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.153+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1899517018}) 2024-09-16T04:06:02.158 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.153+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.158 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.153+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2526165714}) 2024-09-16T04:06:02.158 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.153+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4056407395}) 2024-09-16T04:06:02.160 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.153+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.160 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.153+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2825426849}) 2024-09-16T04:06:02.160 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.153+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3288450890}) 2024-09-16T04:06:02.160 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.153+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.160 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.153+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2825426849}) 2024-09-16T04:06:02.160 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.153+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1251677298}) 2024-09-16T04:06:02.162 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.157+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.162 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.157+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2422306877}) 2024-09-16T04:06:02.162 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.157+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=900957799}) 2024-09-16T04:06:02.162 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.157+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.162 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.157+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2422306877}) 2024-09-16T04:06:02.162 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.157+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=379873785}) 2024-09-16T04:06:02.164 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.157+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.164 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.157+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1965707113}) 2024-09-16T04:06:02.164 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.157+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1641908940}) 2024-09-16T04:06:02.164 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.157+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.164 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.157+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1965707113}) 2024-09-16T04:06:02.164 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.157+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=386238433}) 2024-09-16T04:06:02.165 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.157+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.165 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.161+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=955507229}) 2024-09-16T04:06:02.165 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.161+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3084118675}) 2024-09-16T04:06:02.165 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.161+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.165 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.161+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=955507229}) 2024-09-16T04:06:02.166 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.161+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=794895243}) 2024-09-16T04:06:02.167 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.161+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.167 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.161+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=626413607}) 2024-09-16T04:06:02.167 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.161+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3035884937}) 2024-09-16T04:06:02.167 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.161+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.167 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.161+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=626413607}) 2024-09-16T04:06:02.167 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.161+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1083176353}) 2024-09-16T04:06:02.169 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.161+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.169 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.161+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3741526590}) 2024-09-16T04:06:02.169 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.161+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4154796274}) 2024-09-16T04:06:02.169 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.161+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.169 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.161+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3741526590}) 2024-09-16T04:06:02.169 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.161+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=367388587}) 2024-09-16T04:06:02.170 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.165+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.170 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.165+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=579713026}) 2024-09-16T04:06:02.171 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.165+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=51339104}) 2024-09-16T04:06:02.171 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.165+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.171 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.165+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=579713026}) 2024-09-16T04:06:02.171 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.165+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=247737768}) 2024-09-16T04:06:02.172 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.165+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.172 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.165+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4283425800}) 2024-09-16T04:06:02.172 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.165+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3151779514}) 2024-09-16T04:06:02.173 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.165+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.173 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.165+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4283425800}) 2024-09-16T04:06:02.173 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.165+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3661683895}) 2024-09-16T04:06:02.174 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.169+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.174 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.169+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1693283036}) 2024-09-16T04:06:02.174 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.169+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2324280970}) 2024-09-16T04:06:02.174 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.169+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.174 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.169+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1693283036}) 2024-09-16T04:06:02.175 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.169+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1211125771}) 2024-09-16T04:06:02.176 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.169+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.176 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.169+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3338652880}) 2024-09-16T04:06:02.176 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.169+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3399265364}) 2024-09-16T04:06:02.176 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.169+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.176 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.169+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3338652880}) 2024-09-16T04:06:02.176 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.169+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=719243730}) 2024-09-16T04:06:02.178 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.173+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.178 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.173+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1005116947}) 2024-09-16T04:06:02.178 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.173+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1288501935}) 2024-09-16T04:06:02.178 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.173+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.178 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.173+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1005116947}) 2024-09-16T04:06:02.178 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.173+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3290355002}) 2024-09-16T04:06:02.179 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.173+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.179 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.173+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2878739200}) 2024-09-16T04:06:02.180 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.173+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2405123201}) 2024-09-16T04:06:02.180 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.173+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.180 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.173+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2878739200}) 2024-09-16T04:06:02.180 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.173+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2313498063}) 2024-09-16T04:06:02.181 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.173+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.181 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.173+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2563697441}) 2024-09-16T04:06:02.181 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.173+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1583709006}) 2024-09-16T04:06:02.181 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.177+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.182 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.177+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2563697441}) 2024-09-16T04:06:02.182 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.177+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2070505154}) 2024-09-16T04:06:02.183 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.177+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.183 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.177+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2352429468}) 2024-09-16T04:06:02.183 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.177+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3809360518}) 2024-09-16T04:06:02.183 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.177+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.183 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.177+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2352429468}) 2024-09-16T04:06:02.183 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.177+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1105564545}) 2024-09-16T04:06:02.185 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.177+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.185 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.177+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2214263165}) 2024-09-16T04:06:02.185 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.177+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3154641262}) 2024-09-16T04:06:02.186 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.177+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.186 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.177+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2214263165}) 2024-09-16T04:06:02.186 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.177+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1721413820}) 2024-09-16T04:06:02.187 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.181+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.187 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.181+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3290124985}) 2024-09-16T04:06:02.187 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.181+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=947043842}) 2024-09-16T04:06:02.187 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.181+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.187 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.181+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3290124985}) 2024-09-16T04:06:02.188 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.181+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3961620080}) 2024-09-16T04:06:02.189 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.185+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.189 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.185+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=800258420}) 2024-09-16T04:06:02.189 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.185+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1377568319}) 2024-09-16T04:06:02.190 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.185+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.190 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.185+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=800258420}) 2024-09-16T04:06:02.190 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.185+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1964847763}) 2024-09-16T04:06:02.191 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.185+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.192 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.185+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3446365502}) 2024-09-16T04:06:02.192 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.185+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2047131951}) 2024-09-16T04:06:02.192 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.185+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.192 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.185+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3446365502}) 2024-09-16T04:06:02.192 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.185+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4022790207}) 2024-09-16T04:06:02.193 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.189+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.193 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.189+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=656859454}) 2024-09-16T04:06:02.194 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.189+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=805355928}) 2024-09-16T04:06:02.194 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.189+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.194 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.189+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=656859454}) 2024-09-16T04:06:02.194 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.189+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3614180254}) 2024-09-16T04:06:02.195 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.189+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.196 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.189+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3586131462}) 2024-09-16T04:06:02.196 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.189+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1994644534}) 2024-09-16T04:06:02.196 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.189+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.196 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.189+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3586131462}) 2024-09-16T04:06:02.196 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.189+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2396573012}) 2024-09-16T04:06:02.197 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.193+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.197 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.193+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=17750425}) 2024-09-16T04:06:02.198 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.193+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2358952161}) 2024-09-16T04:06:02.198 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.193+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.198 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.193+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=17750425}) 2024-09-16T04:06:02.198 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.193+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1457297700}) 2024-09-16T04:06:02.199 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.193+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.199 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.193+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1682973562}) 2024-09-16T04:06:02.200 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.193+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2703757462}) 2024-09-16T04:06:02.200 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.193+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.200 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.193+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1682973562}) 2024-09-16T04:06:02.200 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.193+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=889821394}) 2024-09-16T04:06:02.202 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.197+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.202 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.197+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4293299378}) 2024-09-16T04:06:02.202 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.197+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2652326487}) 2024-09-16T04:06:02.202 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.197+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.202 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.197+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4293299378}) 2024-09-16T04:06:02.202 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.197+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1651689532}) 2024-09-16T04:06:02.204 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.197+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.204 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.197+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2945572579}) 2024-09-16T04:06:02.204 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.197+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1935321352}) 2024-09-16T04:06:02.204 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.197+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.204 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.197+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2945572579}) 2024-09-16T04:06:02.204 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.197+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2480970078}) 2024-09-16T04:06:02.206 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.201+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.206 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.201+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3499233263}) 2024-09-16T04:06:02.206 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.201+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4245554662}) 2024-09-16T04:06:02.206 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.201+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.206 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.201+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3499233263}) 2024-09-16T04:06:02.206 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.201+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2894852357}) 2024-09-16T04:06:02.208 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.201+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.208 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.201+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3679179675}) 2024-09-16T04:06:02.208 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.201+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2088587036}) 2024-09-16T04:06:02.208 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.201+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.208 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.201+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3679179675}) 2024-09-16T04:06:02.208 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.201+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=760032527}) 2024-09-16T04:06:02.210 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.205+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.210 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.205+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=842051083}) 2024-09-16T04:06:02.210 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.205+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=824793501}) 2024-09-16T04:06:02.210 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.205+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.210 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.205+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=842051083}) 2024-09-16T04:06:02.210 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.205+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1997902950}) 2024-09-16T04:06:02.212 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.205+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.212 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.205+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3188026179}) 2024-09-16T04:06:02.213 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.205+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4170322970}) 2024-09-16T04:06:02.213 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.205+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.213 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.205+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3188026179}) 2024-09-16T04:06:02.213 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.205+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3554363898}) 2024-09-16T04:06:02.214 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.209+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.214 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.209+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3223679084}) 2024-09-16T04:06:02.215 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.209+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=306423372}) 2024-09-16T04:06:02.215 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.209+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.215 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.209+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3223679084}) 2024-09-16T04:06:02.215 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.209+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2463666555}) 2024-09-16T04:06:02.217 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.209+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.217 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.209+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3320431619}) 2024-09-16T04:06:02.217 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.209+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3678706066}) 2024-09-16T04:06:02.217 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.209+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.217 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.209+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3320431619}) 2024-09-16T04:06:02.217 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.209+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4291903313}) 2024-09-16T04:06:02.219 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.213+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.219 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.213+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=455970945}) 2024-09-16T04:06:02.219 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.213+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2147515203}) 2024-09-16T04:06:02.219 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.213+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.219 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.213+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=455970945}) 2024-09-16T04:06:02.219 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.213+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2906615969}) 2024-09-16T04:06:02.221 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.213+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.221 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.213+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=776382526}) 2024-09-16T04:06:02.221 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.213+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2882103649}) 2024-09-16T04:06:02.221 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.213+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.221 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.213+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=776382526}) 2024-09-16T04:06:02.221 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.213+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3316201833}) 2024-09-16T04:06:02.223 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.217+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.223 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.217+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3919564172}) 2024-09-16T04:06:02.223 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.217+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3667891872}) 2024-09-16T04:06:02.223 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.217+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.223 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.217+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3919564172}) 2024-09-16T04:06:02.223 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.217+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3502755420}) 2024-09-16T04:06:02.225 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.217+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.225 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.217+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=772807098}) 2024-09-16T04:06:02.225 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.217+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3362609124}) 2024-09-16T04:06:02.225 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.217+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.225 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.217+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=772807098}) 2024-09-16T04:06:02.225 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.217+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1089771757}) 2024-09-16T04:06:02.227 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.221+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.227 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.221+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1673864799}) 2024-09-16T04:06:02.227 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.221+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3483610278}) 2024-09-16T04:06:02.227 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.221+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.227 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.221+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1673864799}) 2024-09-16T04:06:02.227 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.221+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3545678296}) 2024-09-16T04:06:02.229 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.221+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.229 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.221+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3717209772}) 2024-09-16T04:06:02.229 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.221+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1094210689}) 2024-09-16T04:06:02.229 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.221+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.229 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.221+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3717209772}) 2024-09-16T04:06:02.229 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.221+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1228581187}) 2024-09-16T04:06:02.231 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.225+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.231 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.225+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=52851186}) 2024-09-16T04:06:02.231 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.225+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2361561760}) 2024-09-16T04:06:02.231 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.225+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.231 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.225+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=52851186}) 2024-09-16T04:06:02.231 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.225+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=770334347}) 2024-09-16T04:06:02.233 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.225+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.233 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.225+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4281379460}) 2024-09-16T04:06:02.233 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.225+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=291791432}) 2024-09-16T04:06:02.233 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.225+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.233 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.225+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4281379460}) 2024-09-16T04:06:02.233 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.225+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2379089816}) 2024-09-16T04:06:02.235 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.229+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.235 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.229+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2747535173}) 2024-09-16T04:06:02.235 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.229+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1635960458}) 2024-09-16T04:06:02.235 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.229+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.235 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.229+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2747535173}) 2024-09-16T04:06:02.235 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.229+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2190455381}) 2024-09-16T04:06:02.237 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.229+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.237 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.229+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=890536712}) 2024-09-16T04:06:02.237 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.229+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4136543961}) 2024-09-16T04:06:02.237 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.229+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.237 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.229+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=890536712}) 2024-09-16T04:06:02.237 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.229+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1477834977}) 2024-09-16T04:06:02.238 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.233+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.239 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.233+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1260001750}) 2024-09-16T04:06:02.239 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.233+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3918593735}) 2024-09-16T04:06:02.239 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.233+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.239 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.233+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1260001750}) 2024-09-16T04:06:02.239 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.233+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3442279744}) 2024-09-16T04:06:02.240 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.233+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.240 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.233+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4006531707}) 2024-09-16T04:06:02.241 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.233+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=169041496}) 2024-09-16T04:06:02.241 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.233+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.241 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.233+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4006531707}) 2024-09-16T04:06:02.241 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.233+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=738777185}) 2024-09-16T04:06:02.243 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.237+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.243 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.237+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=34,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=11,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=21} crc {logm=4234034028,mdsmap=210808725,mgr=460109776,mgr_command_descs=2107673600,mgr_metadata=1367272202,mgrstat=671458694,mon_config_key=1363415709,monmap=3937513981,osd_metadata=191738038,osd_pg_creating=783066558,osd_snap=1538147421}) 2024-09-16T04:06:02.243 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.237+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=31,mdsmap=3,mgr=6,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=11,monmap=3,osd_metadata=4,osd_pg_creating=1,osd_snap=27} crc {logm=1577706364,mdsmap=4088524775,mgr=4082366291,mgr_command_descs=2107673600,mgr_metadata=1367272202,mgrstat=246990011,mon_config_key=85057112,monmap=3937513981,osd_metadata=3428237747,osd_pg_creating=783066558,osd_snap=1533930918}) 2024-09-16T04:06:02.243 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.237+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.243 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.237+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=34,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=11,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=21} crc {logm=4234034028,mdsmap=210808725,mgr=460109776,mgr_command_descs=2107673600,mgr_metadata=1367272202,mgrstat=671458694,mon_config_key=1363415709,monmap=3937513981,osd_metadata=191738038,osd_pg_creating=783066558,osd_snap=1538147421}) 2024-09-16T04:06:02.243 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.237+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=36,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=18} crc {logm=1107205131,mdsmap=4088524775,mgr=460109776,mgr_command_descs=2107673600,mgr_metadata=1367272202,mgrstat=453054658,mon_config_key=85057112,monmap=3937513981,osd_metadata=4182959540,osd_pg_creating=783066558,osd_snap=4073325951}) 2024-09-16T04:06:02.245 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.241+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.246 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.241+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=18,osdmap=82} crc {osd_snap=850374676,osdmap=2808521977}) 2024-09-16T04:06:02.246 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.241+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=17,osdmap=83} crc {osd_snap=731200016,osdmap=2896991687}) 2024-09-16T04:06:02.246 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.241+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.246 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.241+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=18,osdmap=82} crc {osd_snap=850374676,osdmap=2808521977}) 2024-09-16T04:06:02.246 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.241+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=16,osdmap=84} crc {osd_snap=3198788175,osdmap=2323100488}) 2024-09-16T04:06:02.248 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.241+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.248 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.241+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1508959804}) 2024-09-16T04:06:02.248 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.241+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2103912198}) 2024-09-16T04:06:02.248 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.241+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.248 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.241+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1508959804}) 2024-09-16T04:06:02.248 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.241+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=109129696}) 2024-09-16T04:06:02.251 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.245+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.251 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.245+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3401631806}) 2024-09-16T04:06:02.251 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.245+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2948362982}) 2024-09-16T04:06:02.251 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.245+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.251 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.245+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3401631806}) 2024-09-16T04:06:02.251 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.245+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2007543836}) 2024-09-16T04:06:02.265 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.257+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.265 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.257+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3692347860}) 2024-09-16T04:06:02.265 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.257+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=96} crc {osdmap=3650405102}) 2024-09-16T04:06:02.265 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.257+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:06:02.265 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.257+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3692347860}) 2024-09-16T04:06:02.266 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:06:02.257+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=98} crc {osdmap=837456903}) 2024-09-16T04:06:03.002 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T04:06:03.002 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:06:03.143 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.137+0000 7ffab9143640 10 monclient: get_monmap_and_config 2024-09-16T04:06:03.143 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.137+0000 7ffab9143640 10 monclient: build_initial_monmap 2024-09-16T04:06:03.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.137+0000 7ffab9143640 10 monclient: monmap: 2024-09-16T04:06:03.144 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:06:03.144 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:06:03.144 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:06:03.142002+0000 2024-09-16T04:06:03.144 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:06:03.142002+0000 2024-09-16T04:06:03.144 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:06:03.144 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:06:03.144 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:06:03.144 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:06:03.144 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:06:03.144 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:06:03.144 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:06:03.144 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:06:03.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.137+0000 7ffab9143640 1 Processor -- start 2024-09-16T04:06:03.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.137+0000 7ffab9143640 1 -- start start 2024-09-16T04:06:03.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.137+0000 7ffab9143640 10 monclient: init 2024-09-16T04:06:03.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.137+0000 7ffab9143640 10 monclient: _reopen_session rank -1 2024-09-16T04:06:03.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.137+0000 7ffab9143640 10 monclient: _add_conns ranks=[3,4,0,1,2] 2024-09-16T04:06:03.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.137+0000 7ffab9143640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ffab40a2f10 0x7ffab40a1330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:06:03.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.137+0000 7ffab9143640 10 monclient(hunting): picked mon.noname-a con 0x7ffab40a2f10 addr v2:172.21.15.42:3300/0 2024-09-16T04:06:03.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.137+0000 7ffab9143640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7ffab409e070 0x7ffab40a1870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:06:03.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.137+0000 7ffab9143640 10 monclient(hunting): picked mon.noname-c con 0x7ffab409e070 addr v2:172.21.15.154:3300/0 2024-09-16T04:06:03.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.137+0000 7ffab9143640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ffab40a1db0 addr v1:172.21.15.42:6789/0 2024-09-16T04:06:03.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.137+0000 7ffab9143640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffab409dc50 con 0x7ffab40a1db0 2024-09-16T04:06:03.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.137+0000 7ffab9143640 10 monclient(hunting): start opening mon connection 2024-09-16T04:06:03.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.137+0000 7ffab9143640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7ffab40a21b0 con 0x7ffab40a2f10 2024-09-16T04:06:03.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.137+0000 7ffab9143640 10 monclient(hunting): start opening mon connection 2024-09-16T04:06:03.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.137+0000 7ffab9143640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7ffab40a32f0 con 0x7ffab409e070 2024-09-16T04:06:03.148 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.137+0000 7ffab9143640 10 monclient(hunting): _renew_subs 2024-09-16T04:06:03.148 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.137+0000 7ffab9143640 10 monclient(hunting): authenticate will time out at 2410.111413s 2024-09-16T04:06:03.148 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.137+0000 7ffab2d9d640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ffab40a2f10 0x7ffab40a1330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:06:03.148 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.137+0000 7ffab359e640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7ffab40a1db0 0x7ffab413b8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:56060/0 (socket says 172.21.15.42:56060) 2024-09-16T04:06:03.148 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.137+0000 7ffab359e640 1 -- 172.21.15.42:0/3420481463 learned_addr learned my addr 172.21.15.42:0/3420481463 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T04:06:03.148 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.137+0000 7ffab2d9d640 10 monclient(hunting): get_auth_request con 0x7ffab40a2f10 auth_method 0 2024-09-16T04:06:03.148 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.137+0000 7ffab2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:06:03.148 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.137+0000 7ffab2d9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:03.148 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.137+0000 7ffab2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:03.148 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.137+0000 7ffab2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:06:03.148 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.137+0000 7ffab2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:06:03.148 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.137+0000 7ffab2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:06:03.148 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.137+0000 7ffab259c640 1 --2- 172.21.15.42:0/3420481463 >> v2:172.21.15.154:3300/0 conn(0x7ffab409e070 0x7ffab40a1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:06:03.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.141+0000 7ffab259c640 10 monclient(hunting): get_auth_request con 0x7ffab409e070 auth_method 0 2024-09-16T04:06:03.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.141+0000 7ffab259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:06:03.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.141+0000 7ffab259c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:03.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.141+0000 7ffab259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:03.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.141+0000 7ffab2d9d640 10 monclient(hunting): handle_auth_done global_id 984396 payload 274 2024-09-16T04:06:03.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.141+0000 7ffab2d9d640 1 -- 172.21.15.42:0/3420481463 >> v2:172.21.15.154:3300/0 conn(0x7ffab409e070 msgr2=0x7ffab40a1870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:06:03.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.141+0000 7ffab2d9d640 1 --2- 172.21.15.42:0/3420481463 >> v2:172.21.15.154:3300/0 conn(0x7ffab409e070 0x7ffab40a1870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:06:03.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.141+0000 7ffab2d9d640 1 -- 172.21.15.42:0/3420481463 >> v1:172.21.15.42:6789/0 conn(0x7ffab40a1db0 legacy=0x7ffab413b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:03.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.141+0000 7ffab2d9d640 10 monclient: _finish_hunting 0 2024-09-16T04:06:03.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.141+0000 7ffab2d9d640 1 monclient: found mon.noname-a 2024-09-16T04:06:03.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.141+0000 7ffab2d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:06:03.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.141+0000 7ffab2d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:06:03.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.141+0000 7ffab2d9d640 1 -- 172.21.15.42:0/3420481463 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffab4073290 con 0x7ffab40a2f10 2024-09-16T04:06:03.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.141+0000 7ffab2d9d640 1 --2- 172.21.15.42:0/3420481463 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ffab40a2f10 0x7ffab40a1330 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7ffab40a00e0 tx=0x7ffaa401c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf4041d92a4826bd server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:06:03.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.141+0000 7ffab1d9b640 1 -- 172.21.15.42:0/3420481463 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaa40a66d0 con 0x7ffab40a2f10 2024-09-16T04:06:03.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.141+0000 7ffab1d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:03.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.141+0000 7ffab1d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:06:03.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.141+0000 7ffab1d9b640 10 monclient: dump: 2024-09-16T04:06:03.156 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:06:03.156 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:06:03.156 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:06:03.156 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:06:03.156 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:03.156 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:06:03.156 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:06:03.156 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:06:03.156 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:06:03.156 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:06:03.157 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.141+0000 7ffab1d9b640 10 monclient: _finish_auth 0 2024-09-16T04:06:03.157 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.141+0000 7ffab1d9b640 10 monclient: _check_auth_tickets 2024-09-16T04:06:03.157 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.141+0000 7ffab1d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:06:03.157 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.141+0000 7ffab9143640 5 monclient: authenticate success, global_id 984396 2024-09-16T04:06:03.157 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.141+0000 7ffab9143640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:06:03.157 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.141+0000 7ffab1d9b640 1 -- 172.21.15.42:0/3420481463 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffaa40a6870 con 0x7ffab40a2f10 2024-09-16T04:06:03.157 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.141+0000 7ffab1d9b640 10 monclient: handle_config config(0 keys) 2024-09-16T04:06:03.157 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.141+0000 7ffab9143640 10 monclient: get_monmap_and_config success 2024-09-16T04:06:03.157 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.141+0000 7ffab9143640 4 set_mon_vals no callback set 2024-09-16T04:06:03.157 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.141+0000 7ffab1d9b640 1 -- 172.21.15.42:0/3420481463 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaa40a6c40 con 0x7ffab40a2f10 2024-09-16T04:06:03.157 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.141+0000 7ffab1d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:03.157 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.141+0000 7ffab1d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:06:03.157 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.141+0000 7ffab1d9b640 10 monclient: dump: 2024-09-16T04:06:03.157 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:06:03.158 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:06:03.158 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:06:03.158 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:06:03.158 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:03.158 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:06:03.158 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:06:03.158 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:06:03.158 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:06:03.158 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:06:03.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.141+0000 7ffab9143640 10 monclient: shutdown 2024-09-16T04:06:03.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.141+0000 7ffab9143640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:06:03.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.141+0000 7ffab9143640 1 -- 172.21.15.42:0/3420481463 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ffab40a2f10 msgr2=0x7ffab40a1330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:03.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.141+0000 7ffab9143640 1 --2- 172.21.15.42:0/3420481463 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ffab40a2f10 0x7ffab40a1330 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7ffab40a00e0 tx=0x7ffaa401c7f0 comp rx=0 tx=0).stop 2024-09-16T04:06:03.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.141+0000 7ffab9143640 1 -- 172.21.15.42:0/3420481463 shutdown_connections 2024-09-16T04:06:03.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.141+0000 7ffab9143640 1 --2- 172.21.15.42:0/3420481463 >> v2:172.21.15.154:3300/0 conn(0x7ffab409e070 0x7ffab40a1870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:06:03.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.141+0000 7ffab9143640 1 --2- 172.21.15.42:0/3420481463 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ffab40a2f10 0x7ffab40a1330 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:06:03.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.141+0000 7ffab9143640 1 -- 172.21.15.42:0/3420481463 >> 172.21.15.42:0/3420481463 conn(0x7ffab40a2420 msgr2=0x7ffab40a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:06:03.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.141+0000 7ffab9143640 1 -- 172.21.15.42:0/3420481463 shutdown_connections 2024-09-16T04:06:03.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.141+0000 7ffab9143640 1 -- 172.21.15.42:0/3420481463 wait complete. 2024-09-16T04:06:03.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.141+0000 7ffab9143640 10 monclient: build_initial_monmap 2024-09-16T04:06:03.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.141+0000 7ffab9143640 10 monclient: monmap: 2024-09-16T04:06:03.159 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:06:03.159 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:06:03.159 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:06:03.159 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:06:03.159 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:06:03.159 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:06:03.160 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:06:03.160 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:06:03.160 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:06:03.160 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:06:03.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.141+0000 7ffab9143640 1 librados: starting msgr at 2024-09-16T04:06:03.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.141+0000 7ffab9143640 1 librados: starting objecter 2024-09-16T04:06:03.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.145+0000 7ffab9143640 1 Processor -- start 2024-09-16T04:06:03.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.145+0000 7ffab9143640 1 -- start start 2024-09-16T04:06:03.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.145+0000 7ffab9143640 1 librados: setting wanted keys 2024-09-16T04:06:03.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.145+0000 7ffab9143640 1 librados: calling monclient init 2024-09-16T04:06:03.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.145+0000 7ffab9143640 10 monclient: init 2024-09-16T04:06:03.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.145+0000 7ffab9143640 10 monclient: _reopen_session rank -1 2024-09-16T04:06:03.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.145+0000 7ffab9143640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:06:03.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.145+0000 7ffab9143640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ffab409e070 0x7ffab41d3270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:06:03.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.145+0000 7ffab9143640 10 monclient(hunting): picked mon.noname-c con 0x7ffab409e070 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:06:03.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.145+0000 7ffab9143640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffab40a1db0 0x7ffab41d39a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:06:03.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.145+0000 7ffab9143640 10 monclient(hunting): picked mon.noname-b con 0x7ffab40a1db0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:06:03.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.145+0000 7ffab9143640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ffab40a2f10 0x7ffab41409d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:06:03.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.145+0000 7ffab9143640 10 monclient(hunting): picked mon.noname-a con 0x7ffab40a2f10 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:06:03.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.145+0000 7ffab9143640 10 monclient(hunting): start opening mon connection 2024-09-16T04:06:03.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.145+0000 7ffab9143640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7ffab4145d70 con 0x7ffab40a2f10 2024-09-16T04:06:03.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.145+0000 7ffab9143640 10 monclient(hunting): start opening mon connection 2024-09-16T04:06:03.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.145+0000 7ffab9143640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ffab409d720 con 0x7ffab40a1db0 2024-09-16T04:06:03.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.145+0000 7ffab9143640 10 monclient(hunting): start opening mon connection 2024-09-16T04:06:03.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.145+0000 7ffab9143640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7ffab4145ef0 con 0x7ffab409e070 2024-09-16T04:06:03.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.145+0000 7ffab9143640 10 monclient(hunting): _renew_subs 2024-09-16T04:06:03.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.145+0000 7ffab9143640 10 monclient(hunting): authenticate will time out at 2410.116562s 2024-09-16T04:06:03.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.145+0000 7ffab359e640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ffab40a2f10 0x7ffab41409d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:06:03.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.145+0000 7ffab2d9d640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ffab409e070 0x7ffab41d3270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:06:03.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.145+0000 7ffab359e640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ffab40a2f10 0x7ffab41409d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:38774/0 (socket says 172.21.15.42:38774) 2024-09-16T04:06:03.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.145+0000 7ffab359e640 1 -- 172.21.15.42:0/1862751047 learned_addr learned my addr 172.21.15.42:0/1862751047 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:06:03.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.145+0000 7ffab359e640 10 monclient(hunting): get_auth_request con 0x7ffab40a2f10 auth_method 0 2024-09-16T04:06:03.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.145+0000 7ffab359e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:06:03.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.145+0000 7ffab359e640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:03.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.145+0000 7ffab359e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:03.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.145+0000 7ffab2d9d640 10 monclient(hunting): get_auth_request con 0x7ffab409e070 auth_method 0 2024-09-16T04:06:03.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.145+0000 7ffab2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:06:03.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.145+0000 7ffab2d9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:03.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.145+0000 7ffab2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:03.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.145+0000 7ffab359e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:06:03.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.145+0000 7ffab359e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:06:03.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.145+0000 7ffab359e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:06:03.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.145+0000 7ffab2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:06:03.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.145+0000 7ffab2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:06:03.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.145+0000 7ffab2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:06:03.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.145+0000 7ffab359e640 10 monclient(hunting): handle_auth_done global_id 984402 payload 824 2024-09-16T04:06:03.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.145+0000 7ffab359e640 1 -- 172.21.15.42:0/1862751047 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ffab409e070 msgr2=0x7ffab41d3270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:03.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.145+0000 7ffab359e640 1 --2- 172.21.15.42:0/1862751047 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ffab409e070 0x7ffab41d3270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:06:03.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.145+0000 7ffab259c640 1 --2- 172.21.15.42:0/1862751047 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffab40a1db0 0x7ffab41d39a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:06:03.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.145+0000 7ffab359e640 1 -- 172.21.15.42:0/1862751047 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffab40a1db0 msgr2=0x7ffab41d39a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:03.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.145+0000 7ffab359e640 1 --2- 172.21.15.42:0/1862751047 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffab40a1db0 0x7ffab41d39a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:06:03.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.145+0000 7ffab359e640 10 monclient: _finish_hunting 0 2024-09-16T04:06:03.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.145+0000 7ffab359e640 1 monclient: found mon.noname-a 2024-09-16T04:06:03.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.145+0000 7ffab359e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:06:03.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.145+0000 7ffab359e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:06:03.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.145+0000 7ffab359e640 1 -- 172.21.15.42:0/1862751047 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffab4125200 con 0x7ffab40a2f10 2024-09-16T04:06:03.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.145+0000 7ffab2d9d640 1 --2- 172.21.15.42:0/1862751047 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ffab409e070 0x7ffab41d3270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:06:03.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.145+0000 7ffab359e640 1 --2- 172.21.15.42:0/1862751047 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ffab40a2f10 0x7ffab41409d0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7ffaac00c390 tx=0x7ffaac00c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:06:03.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.145+0000 7ffaab7fe640 10 client.?.objecter ms_handle_connect 0x7ffab40a2f10 2024-09-16T04:06:03.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.145+0000 7ffaab7fe640 10 client.?.objecter resend_mon_ops 2024-09-16T04:06:03.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.145+0000 7ffaab7fe640 1 -- 172.21.15.42:0/1862751047 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaac017070 con 0x7ffab40a2f10 2024-09-16T04:06:03.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.145+0000 7ffaab7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:03.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.145+0000 7ffaab7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:06:03.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.145+0000 7ffaab7fe640 10 monclient: dump: 2024-09-16T04:06:03.164 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:06:03.164 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:06:03.165 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:06:03.165 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:06:03.165 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:03.165 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:06:03.165 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:06:03.165 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:06:03.165 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:06:03.165 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:06:03.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.145+0000 7ffaab7fe640 10 monclient: _finish_auth 0 2024-09-16T04:06:03.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.145+0000 7ffaab7fe640 10 monclient: _check_auth_tickets 2024-09-16T04:06:03.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.145+0000 7ffaab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:06:03.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.145+0000 7ffaab7fe640 1 -- 172.21.15.42:0/1862751047 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffaac005cf0 con 0x7ffab40a2f10 2024-09-16T04:06:03.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.145+0000 7ffaab7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:06:03.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.145+0000 7ffaab7fe640 1 -- 172.21.15.42:0/1862751047 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaac007310 con 0x7ffab40a2f10 2024-09-16T04:06:03.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.145+0000 7ffaab7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:03.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.145+0000 7ffaab7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:06:03.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.145+0000 7ffaab7fe640 10 monclient: dump: 2024-09-16T04:06:03.166 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:06:03.166 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:06:03.166 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:06:03.166 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:06:03.166 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:03.166 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:06:03.166 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:06:03.166 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:06:03.166 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:06:03.166 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:06:03.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.145+0000 7ffab9143640 5 monclient: authenticate success, global_id 984402 2024-09-16T04:06:03.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.145+0000 7ffab9143640 10 monclient: _renew_subs 2024-09-16T04:06:03.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.145+0000 7ffab0d99640 4 set_mon_vals no callback set 2024-09-16T04:06:03.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.145+0000 7ffab9143640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:06:03.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.145+0000 7ffab9143640 1 -- 172.21.15.42:0/1862751047 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffab4072310 con 0x7ffab40a2f10 2024-09-16T04:06:03.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.145+0000 7ffab9143640 10 client.984402.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:06:03.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.145+0000 7ffab9143640 10 monclient: _renew_subs 2024-09-16T04:06:03.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.145+0000 7ffab9143640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:06:03.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.145+0000 7ffab9143640 1 -- 172.21.15.42:0/1862751047 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffab4124f20 con 0x7ffab40a2f10 2024-09-16T04:06:03.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.145+0000 7ffab9143640 1 librados: init done 2024-09-16T04:06:03.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.145+0000 7ffab9143640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:06:03.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.145+0000 7ffab9143640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:06:03.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.145+0000 7ffab9143640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:06:03.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.145+0000 7ffab9143640 1 -- 172.21.15.42:0/1862751047 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffab413b8f0 con 0x7ffab40a2f10 2024-09-16T04:06:03.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.145+0000 7ffaab7fe640 1 -- 172.21.15.42:0/1862751047 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7ffaac01a070 con 0x7ffab40a2f10 2024-09-16T04:06:03.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.145+0000 7ffaab7fe640 1 --2- 172.21.15.42:0/1862751047 >> v2:172.21.15.154:6812/254845342 conn(0x7ffa94037860 0x7ffa94039d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:06:03.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.145+0000 7ffaab7fe640 1 -- 172.21.15.42:0/1862751047 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(706..706 src has 451..706) ==== 5040+0+0 (secure 0 0 0) 0x7ffaac047ff0 con 0x7ffab40a2f10 2024-09-16T04:06:03.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.145+0000 7ffaab7fe640 10 client.984402.objecter ms_dispatch 0x7ffab4000ba0 osd_map(706..706 src has 451..706) 2024-09-16T04:06:03.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.145+0000 7ffaab7fe640 3 client.984402.objecter handle_osd_map got epochs [706,706] > 0 2024-09-16T04:06:03.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.145+0000 7ffaab7fe640 3 client.984402.objecter handle_osd_map decoding full epoch 706 2024-09-16T04:06:03.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.145+0000 7ffaab7fe640 20 client.984402.objecter dump_active .. 0 homeless 2024-09-16T04:06:03.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.145+0000 7ffab2d9d640 1 --2- 172.21.15.42:0/1862751047 >> v2:172.21.15.154:6812/254845342 conn(0x7ffa94037860 0x7ffa94039d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:06:03.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.145+0000 7ffab2d9d640 10 monclient: get_auth_request con 0x7ffa94037860 auth_method 0 2024-09-16T04:06:03.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.149+0000 7ffab2d9d640 1 --2- 172.21.15.42:0/1862751047 >> v2:172.21.15.154:6812/254845342 conn(0x7ffa94037860 0x7ffa94039d20 secure :-1 s=READY pgs=3862 cs=0 l=1 rev1=1 crypto rx=0x7ffaa40af000 tx=0x7ffaa40a7000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:06:03.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.149+0000 7ffaab7fe640 10 client.984402.objecter ms_handle_connect 0x7ffa94037860 2024-09-16T04:06:03.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.149+0000 7ffaab7fe640 1 -- 172.21.15.42:0/1862751047 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7ffaac019e10 con 0x7ffab40a2f10 2024-09-16T04:06:03.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.149+0000 7ffaab7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:06:03.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.149+0000 7ffaab7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:06:03.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.385+0000 7ffab9143640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:06:03.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.385+0000 7ffab9143640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:06:03.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.385+0000 7ffab9143640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:06:03.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.385+0000 7ffab9143640 1 -- 172.21.15.42:0/1862751047 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7ffab4069040 con 0x7ffab40a2f10 2024-09-16T04:06:03.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.385+0000 7ffaab7fe640 1 -- 172.21.15.42:0/1862751047 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7ffaac01b8d0 con 0x7ffab40a2f10 2024-09-16T04:06:03.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.385+0000 7ffaab7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:06:03.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.385+0000 7ffaab7fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:06:03.391 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":222,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":4,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:06:03.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.385+0000 7ffab9143640 10 librados: watch_flush enter 2024-09-16T04:06:03.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.385+0000 7ffab9143640 10 librados: watch_flush exit 2024-09-16T04:06:03.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.385+0000 7ffab9143640 20 client.984402.objecter shutdown clearing up homeless session... 2024-09-16T04:06:03.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.385+0000 7ffab9143640 10 client.984402.objecter successfully canceled tick 2024-09-16T04:06:03.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.385+0000 7ffab9143640 1 -- 172.21.15.42:0/1862751047 >> v2:172.21.15.154:6812/254845342 conn(0x7ffa94037860 msgr2=0x7ffa94039d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:03.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.385+0000 7ffab9143640 1 --2- 172.21.15.42:0/1862751047 >> v2:172.21.15.154:6812/254845342 conn(0x7ffa94037860 0x7ffa94039d20 secure :-1 s=READY pgs=3862 cs=0 l=1 rev1=1 crypto rx=0x7ffaa40af000 tx=0x7ffaa40a7000 comp rx=0 tx=0).stop 2024-09-16T04:06:03.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.385+0000 7ffab9143640 10 monclient: shutdown 2024-09-16T04:06:03.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.385+0000 7ffab9143640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:06:03.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.385+0000 7ffab9143640 1 -- 172.21.15.42:0/1862751047 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ffab40a2f10 msgr2=0x7ffab41409d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:03.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.385+0000 7ffab9143640 1 --2- 172.21.15.42:0/1862751047 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ffab40a2f10 0x7ffab41409d0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7ffaac00c390 tx=0x7ffaac00c850 comp rx=0 tx=0).stop 2024-09-16T04:06:03.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.385+0000 7ffab9143640 1 -- 172.21.15.42:0/1862751047 shutdown_connections 2024-09-16T04:06:03.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.385+0000 7ffab9143640 1 --2- 172.21.15.42:0/1862751047 >> v2:172.21.15.154:6812/254845342 conn(0x7ffa94037860 0x7ffa94039d20 unknown :-1 s=CLOSED pgs=3862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:06:03.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.385+0000 7ffab9143640 1 --2- 172.21.15.42:0/1862751047 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ffab40a2f10 0x7ffab41409d0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:06:03.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.385+0000 7ffab9143640 1 --2- 172.21.15.42:0/1862751047 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ffab40a1db0 0x7ffab41d39a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:06:03.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.385+0000 7ffab9143640 1 --2- 172.21.15.42:0/1862751047 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ffab409e070 0x7ffab41d3270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:06:03.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.389+0000 7ffab9143640 1 -- 172.21.15.42:0/1862751047 >> 172.21.15.42:0/1862751047 conn(0x7ffab40a2420 msgr2=0x7ffab40a3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:06:03.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.389+0000 7ffab9143640 1 -- 172.21.15.42:0/1862751047 shutdown_connections 2024-09-16T04:06:03.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.389+0000 7ffab9143640 1 -- 172.21.15.42:0/1862751047 wait complete. 2024-09-16T04:06:03.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.389+0000 7ffab9143640 1 librados: shutdown 2024-09-16T04:06:03.410 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T04:06:03.410 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T04:06:03.410 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T04:06:03.411 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T04:06:03.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.533+0000 7fd18345e640 10 monclient: get_monmap_and_config 2024-09-16T04:06:03.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.533+0000 7fd18345e640 10 monclient: build_initial_monmap 2024-09-16T04:06:03.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.533+0000 7fd18345e640 10 monclient: monmap: 2024-09-16T04:06:03.540 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:06:03.540 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:06:03.540 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:06:03.538584+0000 2024-09-16T04:06:03.541 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:06:03.538584+0000 2024-09-16T04:06:03.541 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:06:03.541 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:06:03.541 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:06:03.541 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:06:03.541 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:06:03.541 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:06:03.541 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:06:03.541 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:06:03.544 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.533+0000 7fd18345e640 1 Processor -- start 2024-09-16T04:06:03.544 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.533+0000 7fd18345e640 1 -- start start 2024-09-16T04:06:03.544 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.533+0000 7fd18345e640 10 monclient: init 2024-09-16T04:06:03.544 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.537+0000 7fd18345e640 10 monclient: _reopen_session rank -1 2024-09-16T04:06:03.544 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.537+0000 7fd18345e640 10 monclient: _add_conns ranks=[4,2,3,0,1] 2024-09-16T04:06:03.544 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.537+0000 7fd18345e640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fd17c0a1330 0x7fd17c09f180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:06:03.544 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.537+0000 7fd18345e640 10 monclient(hunting): picked mon.noname-c con 0x7fd17c0a1330 addr v2:172.21.15.154:3300/0 2024-09-16T04:06:03.544 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.537+0000 7fd18345e640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd17c0a1710 0x7fd17c0a1b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:06:03.544 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.537+0000 7fd18345e640 10 monclient(hunting): picked mon.noname-b con 0x7fd17c0a1710 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:06:03.544 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.537+0000 7fd18345e640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd17c1340f0 0x7fd17c1345a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:06:03.545 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.537+0000 7fd18345e640 10 monclient(hunting): picked mon.noname-a con 0x7fd17c1340f0 addr v2:172.21.15.42:3300/0 2024-09-16T04:06:03.545 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.537+0000 7fd18345e640 10 monclient(hunting): start opening mon connection 2024-09-16T04:06:03.545 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.537+0000 7fd18345e640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7fd17c09edf0 con 0x7fd17c1340f0 2024-09-16T04:06:03.545 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.537+0000 7fd18345e640 10 monclient(hunting): start opening mon connection 2024-09-16T04:06:03.545 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.537+0000 7fd18345e640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fd17c09cd60 con 0x7fd17c0a1330 2024-09-16T04:06:03.545 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.537+0000 7fd18345e640 10 monclient(hunting): start opening mon connection 2024-09-16T04:06:03.545 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.537+0000 7fd18345e640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fd17c09e4c0 con 0x7fd17c0a1710 2024-09-16T04:06:03.545 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.537+0000 7fd18345e640 10 monclient(hunting): _renew_subs 2024-09-16T04:06:03.545 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.537+0000 7fd18345e640 10 monclient(hunting): authenticate will time out at 2410.508287s 2024-09-16T04:06:03.545 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.537+0000 7fd1819fb640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd17c1340f0 0x7fd17c1345a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:06:03.545 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.537+0000 7fd1809f9640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd17c0a1710 0x7fd17c0a1b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:06:03.545 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.537+0000 7fd1819fb640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd17c1340f0 0x7fd17c1345a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:38780/0 (socket says 172.21.15.42:38780) 2024-09-16T04:06:03.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.537+0000 7fd1819fb640 1 -- 172.21.15.42:0/2002957827 learned_addr learned my addr 172.21.15.42:0/2002957827 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:06:03.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.537+0000 7fd1809f9640 10 monclient(hunting): get_auth_request con 0x7fd17c0a1710 auth_method 0 2024-09-16T04:06:03.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.537+0000 7fd1809f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:06:03.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.537+0000 7fd1809f9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:03.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.537+0000 7fd1809f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:03.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.537+0000 7fd1819fb640 10 monclient(hunting): get_auth_request con 0x7fd17c1340f0 auth_method 0 2024-09-16T04:06:03.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.537+0000 7fd1819fb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:06:03.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.537+0000 7fd1819fb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:03.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.537+0000 7fd1819fb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:03.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.537+0000 7fd1819fb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:06:03.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.537+0000 7fd1819fb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:06:03.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.537+0000 7fd1819fb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:06:03.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.537+0000 7fd1809f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:06:03.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.537+0000 7fd1809f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:06:03.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.537+0000 7fd1809f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:06:03.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.537+0000 7fd1811fa640 1 --2- 172.21.15.42:0/2002957827 >> v2:172.21.15.154:3300/0 conn(0x7fd17c0a1330 0x7fd17c09f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:06:03.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.537+0000 7fd1811fa640 10 monclient(hunting): get_auth_request con 0x7fd17c0a1330 auth_method 0 2024-09-16T04:06:03.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.537+0000 7fd1811fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:06:03.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.537+0000 7fd1811fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:03.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.537+0000 7fd1811fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:03.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.537+0000 7fd1819fb640 10 monclient(hunting): handle_auth_done global_id 984405 payload 274 2024-09-16T04:06:03.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.537+0000 7fd1819fb640 1 -- 172.21.15.42:0/2002957827 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd17c0a1710 msgr2=0x7fd17c0a1b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:06:03.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.537+0000 7fd1819fb640 1 --2- 172.21.15.42:0/2002957827 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd17c0a1710 0x7fd17c0a1b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:06:03.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.537+0000 7fd1819fb640 1 -- 172.21.15.42:0/2002957827 >> v2:172.21.15.154:3300/0 conn(0x7fd17c0a1330 msgr2=0x7fd17c09f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:06:03.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.537+0000 7fd1819fb640 1 --2- 172.21.15.42:0/2002957827 >> v2:172.21.15.154:3300/0 conn(0x7fd17c0a1330 0x7fd17c09f180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:06:03.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.537+0000 7fd1819fb640 10 monclient: _finish_hunting 0 2024-09-16T04:06:03.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.537+0000 7fd1819fb640 1 monclient: found mon.noname-a 2024-09-16T04:06:03.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.537+0000 7fd1819fb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:06:03.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.537+0000 7fd1819fb640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:06:03.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.537+0000 7fd1819fb640 1 -- 172.21.15.42:0/2002957827 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd17c0a2050 con 0x7fd17c1340f0 2024-09-16T04:06:03.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.537+0000 7fd1819fb640 1 --2- 172.21.15.42:0/2002957827 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd17c1340f0 0x7fd17c1345a0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fd1780098a0 tx=0x7fd17801c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ba2c72308f53996 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:06:03.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.537+0000 7fd173fff640 1 -- 172.21.15.42:0/2002957827 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1780b0070 con 0x7fd17c1340f0 2024-09-16T04:06:03.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.537+0000 7fd173fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:03.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.537+0000 7fd173fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:06:03.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.537+0000 7fd173fff640 10 monclient: dump: 2024-09-16T04:06:03.555 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:06:03.555 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:06:03.555 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:06:03.555 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:06:03.555 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:03.555 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:06:03.555 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:06:03.556 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:06:03.556 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:06:03.556 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:06:03.556 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.537+0000 7fd173fff640 10 monclient: _finish_auth 0 2024-09-16T04:06:03.556 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.537+0000 7fd173fff640 10 monclient: _check_auth_tickets 2024-09-16T04:06:03.556 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.537+0000 7fd173fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:06:03.556 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.537+0000 7fd173fff640 1 -- 172.21.15.42:0/2002957827 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd1780ab600 con 0x7fd17c1340f0 2024-09-16T04:06:03.556 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.537+0000 7fd18345e640 5 monclient: authenticate success, global_id 984405 2024-09-16T04:06:03.556 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.537+0000 7fd18345e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:06:03.556 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.537+0000 7fd173fff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:06:03.557 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.537+0000 7fd18345e640 10 monclient: get_monmap_and_config success 2024-09-16T04:06:03.557 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.537+0000 7fd18345e640 4 set_mon_vals no callback set 2024-09-16T04:06:03.557 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.537+0000 7fd173fff640 1 -- 172.21.15.42:0/2002957827 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1780ab9d0 con 0x7fd17c1340f0 2024-09-16T04:06:03.557 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.537+0000 7fd173fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:03.557 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.537+0000 7fd173fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:06:03.557 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.537+0000 7fd173fff640 10 monclient: dump: 2024-09-16T04:06:03.557 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:06:03.557 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:06:03.558 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:06:03.558 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:06:03.558 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:03.558 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:06:03.558 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:06:03.558 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:06:03.558 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:06:03.558 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:06:03.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.537+0000 7fd18345e640 10 monclient: shutdown 2024-09-16T04:06:03.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.537+0000 7fd18345e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:06:03.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.537+0000 7fd18345e640 1 -- 172.21.15.42:0/2002957827 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd17c1340f0 msgr2=0x7fd17c1345a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:03.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.537+0000 7fd18345e640 1 --2- 172.21.15.42:0/2002957827 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd17c1340f0 0x7fd17c1345a0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fd1780098a0 tx=0x7fd17801c3c0 comp rx=0 tx=0).stop 2024-09-16T04:06:03.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.537+0000 7fd18345e640 1 -- 172.21.15.42:0/2002957827 shutdown_connections 2024-09-16T04:06:03.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.537+0000 7fd18345e640 1 --2- 172.21.15.42:0/2002957827 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd17c1340f0 0x7fd17c1345a0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:06:03.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.537+0000 7fd18345e640 1 --2- 172.21.15.42:0/2002957827 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd17c0a1710 0x7fd17c0a1b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:06:03.560 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.537+0000 7fd18345e640 1 --2- 172.21.15.42:0/2002957827 >> v2:172.21.15.154:3300/0 conn(0x7fd17c0a1330 0x7fd17c09f180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:06:03.560 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.537+0000 7fd18345e640 1 -- 172.21.15.42:0/2002957827 >> 172.21.15.42:0/2002957827 conn(0x7fd17c0a2420 msgr2=0x7fd17c0a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:06:03.560 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.537+0000 7fd18345e640 1 -- 172.21.15.42:0/2002957827 shutdown_connections 2024-09-16T04:06:03.560 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.537+0000 7fd18345e640 1 -- 172.21.15.42:0/2002957827 wait complete. 2024-09-16T04:06:03.560 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.541+0000 7fd18345e640 10 monclient: build_initial_monmap 2024-09-16T04:06:03.560 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.541+0000 7fd18345e640 10 monclient: monmap: 2024-09-16T04:06:03.560 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:06:03.560 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:06:03.560 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:06:03.560 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:06:03.561 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:06:03.561 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:06:03.561 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:06:03.561 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:06:03.561 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:06:03.561 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:06:03.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.541+0000 7fd18345e640 1 librados: starting msgr at 2024-09-16T04:06:03.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.541+0000 7fd18345e640 1 librados: starting objecter 2024-09-16T04:06:03.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.541+0000 7fd18345e640 1 Processor -- start 2024-09-16T04:06:03.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.541+0000 7fd18345e640 1 -- start start 2024-09-16T04:06:03.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.541+0000 7fd18345e640 1 librados: setting wanted keys 2024-09-16T04:06:03.562 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.541+0000 7fd18345e640 1 librados: calling monclient init 2024-09-16T04:06:03.562 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.541+0000 7fd18345e640 10 monclient: init 2024-09-16T04:06:03.562 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.541+0000 7fd18345e640 10 monclient: _reopen_session rank -1 2024-09-16T04:06:03.562 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.541+0000 7fd18345e640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:06:03.562 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.541+0000 7fd18345e640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd17c0a1330 0x7fd17c11bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:06:03.562 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.541+0000 7fd18345e640 10 monclient(hunting): picked mon.noname-c con 0x7fd17c0a1330 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:06:03.562 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.541+0000 7fd18345e640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd17c0a1710 0x7fd17c11c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:06:03.562 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.541+0000 7fd18345e640 10 monclient(hunting): picked mon.noname-a con 0x7fd17c0a1710 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:06:03.562 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.541+0000 7fd18345e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd17c1340f0 0x7fd17c11c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:06:03.562 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.541+0000 7fd18345e640 10 monclient(hunting): picked mon.noname-b con 0x7fd17c1340f0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:06:03.562 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.541+0000 7fd18345e640 10 monclient(hunting): start opening mon connection 2024-09-16T04:06:03.562 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.541+0000 7fd18345e640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fd17c09d1b0 con 0x7fd17c0a1710 2024-09-16T04:06:03.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.541+0000 7fd18345e640 10 monclient(hunting): start opening mon connection 2024-09-16T04:06:03.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.541+0000 7fd18345e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd17c09d030 con 0x7fd17c1340f0 2024-09-16T04:06:03.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.541+0000 7fd18345e640 10 monclient(hunting): start opening mon connection 2024-09-16T04:06:03.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.541+0000 7fd18345e640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fd17c09c260 con 0x7fd17c0a1330 2024-09-16T04:06:03.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.541+0000 7fd18345e640 10 monclient(hunting): _renew_subs 2024-09-16T04:06:03.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.541+0000 7fd18345e640 10 monclient(hunting): authenticate will time out at 2410.513494s 2024-09-16T04:06:03.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.541+0000 7fd1809f9640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd17c0a1710 0x7fd17c11c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:06:03.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.541+0000 7fd1809f9640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd17c0a1710 0x7fd17c11c4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:38782/0 (socket says 172.21.15.42:38782) 2024-09-16T04:06:03.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.541+0000 7fd1809f9640 1 -- 172.21.15.42:0/749399702 learned_addr learned my addr 172.21.15.42:0/749399702 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:06:03.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.541+0000 7fd1809f9640 10 monclient(hunting): get_auth_request con 0x7fd17c0a1710 auth_method 0 2024-09-16T04:06:03.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.541+0000 7fd1809f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:06:03.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.541+0000 7fd1809f9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:03.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.541+0000 7fd1809f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:03.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.541+0000 7fd1811fa640 1 --2- 172.21.15.42:0/749399702 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd17c0a1330 0x7fd17c11bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:06:03.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.541+0000 7fd1809f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:06:03.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.541+0000 7fd1809f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:06:03.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.541+0000 7fd1809f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:06:03.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.541+0000 7fd1811fa640 10 monclient(hunting): get_auth_request con 0x7fd17c0a1330 auth_method 0 2024-09-16T04:06:03.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.541+0000 7fd1811fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:06:03.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.541+0000 7fd1811fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:03.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.541+0000 7fd1811fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:03.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.541+0000 7fd1811fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:06:03.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.541+0000 7fd1811fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:06:03.565 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.541+0000 7fd1811fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:06:03.565 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.541+0000 7fd1809f9640 10 monclient(hunting): handle_auth_done global_id 984408 payload 824 2024-09-16T04:06:03.565 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.541+0000 7fd1819fb640 1 --2- 172.21.15.42:0/749399702 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd17c1340f0 0x7fd17c11c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:06:03.565 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.541+0000 7fd1809f9640 1 -- 172.21.15.42:0/749399702 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd17c0a1330 msgr2=0x7fd17c11bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:03.565 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.541+0000 7fd1809f9640 1 --2- 172.21.15.42:0/749399702 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd17c0a1330 0x7fd17c11bf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:06:03.565 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.541+0000 7fd1809f9640 1 -- 172.21.15.42:0/749399702 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd17c1340f0 msgr2=0x7fd17c11c9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:03.565 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.541+0000 7fd1809f9640 1 --2- 172.21.15.42:0/749399702 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd17c1340f0 0x7fd17c11c9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:06:03.565 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.541+0000 7fd1809f9640 10 monclient: _finish_hunting 0 2024-09-16T04:06:03.565 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.541+0000 7fd1809f9640 1 monclient: found mon.noname-a 2024-09-16T04:06:03.565 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.541+0000 7fd1809f9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:06:03.565 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.541+0000 7fd1809f9640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:06:03.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.541+0000 7fd1809f9640 1 -- 172.21.15.42:0/749399702 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd17c071de0 con 0x7fd17c0a1710 2024-09-16T04:06:03.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.541+0000 7fd1819fb640 10 monclient: get_auth_request con 0x7fd17c1340f0 auth_method 0 2024-09-16T04:06:03.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.541+0000 7fd1819fb640 1 --2- 172.21.15.42:0/749399702 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd17c1340f0 0x7fd17c11c9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T04:06:03.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.541+0000 7fd1809f9640 1 --2- 172.21.15.42:0/749399702 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd17c0a1710 0x7fd17c11c4b0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fd16c00e900 tx=0x7fd16c00edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:06:03.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.541+0000 7fd171ffb640 10 client.?.objecter ms_handle_connect 0x7fd17c0a1710 2024-09-16T04:06:03.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.541+0000 7fd171ffb640 10 client.?.objecter resend_mon_ops 2024-09-16T04:06:03.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.541+0000 7fd171ffb640 1 -- 172.21.15.42:0/749399702 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd16c017070 con 0x7fd17c0a1710 2024-09-16T04:06:03.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.541+0000 7fd171ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:03.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.541+0000 7fd171ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:06:03.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.541+0000 7fd171ffb640 10 monclient: dump: 2024-09-16T04:06:03.566 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:06:03.566 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:06:03.566 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:06:03.567 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:06:03.567 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:03.567 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:06:03.567 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:06:03.567 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:06:03.567 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:06:03.567 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:06:03.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.541+0000 7fd171ffb640 10 monclient: _finish_auth 0 2024-09-16T04:06:03.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.541+0000 7fd171ffb640 10 monclient: _check_auth_tickets 2024-09-16T04:06:03.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.541+0000 7fd171ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:06:03.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.541+0000 7fd171ffb640 1 -- 172.21.15.42:0/749399702 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd16c00ca00 con 0x7fd17c0a1710 2024-09-16T04:06:03.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.541+0000 7fd18345e640 5 monclient: authenticate success, global_id 984408 2024-09-16T04:06:03.568 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.541+0000 7fd18345e640 10 monclient: _renew_subs 2024-09-16T04:06:03.568 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.541+0000 7fd18345e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:06:03.568 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.541+0000 7fd18345e640 1 -- 172.21.15.42:0/749399702 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd17c0a0be0 con 0x7fd17c0a1710 2024-09-16T04:06:03.568 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.541+0000 7fd171ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:06:03.568 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.541+0000 7fd171ffb640 1 -- 172.21.15.42:0/749399702 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd16c00cdd0 con 0x7fd17c0a1710 2024-09-16T04:06:03.568 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.541+0000 7fd171ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:03.568 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.541+0000 7fd171ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:06:03.568 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.541+0000 7fd171ffb640 10 monclient: dump: 2024-09-16T04:06:03.568 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:06:03.568 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:06:03.569 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:06:03.569 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:06:03.569 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:03.569 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:06:03.569 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:06:03.569 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:06:03.569 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:06:03.569 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:06:03.569 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.541+0000 7fd18345e640 10 client.984408.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:06:03.569 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.541+0000 7fd1737fe640 4 set_mon_vals no callback set 2024-09-16T04:06:03.569 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.541+0000 7fd18345e640 10 monclient: _renew_subs 2024-09-16T04:06:03.569 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.541+0000 7fd18345e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:06:03.570 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.541+0000 7fd18345e640 1 -- 172.21.15.42:0/749399702 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd17c11e3e0 con 0x7fd17c0a1710 2024-09-16T04:06:03.570 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.541+0000 7fd18345e640 1 librados: init done 2024-09-16T04:06:03.570 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.541+0000 7fd18345e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:06:03.570 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.541+0000 7fd18345e640 1 --2- 172.21.15.42:0/749399702 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd17c132970 0x7fd17c132d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:06:03.570 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.541+0000 7fd18345e640 10 monclient(hunting): start opening mon connection 2024-09-16T04:06:03.570 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.541+0000 7fd18345e640 1 -- 172.21.15.42:0/749399702 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fd17c09c660 con 0x7fd17c132970 2024-09-16T04:06:03.570 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.541+0000 7fd1811fa640 1 --2- 172.21.15.42:0/749399702 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd17c132970 0x7fd17c132d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:06:03.570 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.541+0000 7fd171ffb640 1 -- 172.21.15.42:0/749399702 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fd16c01a070 con 0x7fd17c0a1710 2024-09-16T04:06:03.570 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.541+0000 7fd1811fa640 10 monclient: get_auth_request con 0x7fd17c132970 auth_method 0 2024-09-16T04:06:03.570 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.541+0000 7fd1811fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:06:03.570 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.541+0000 7fd1811fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:03.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.541+0000 7fd1811fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:03.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.541+0000 7fd1811fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:06:03.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.541+0000 7fd1811fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:06:03.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.541+0000 7fd1811fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:06:03.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.541+0000 7fd171ffb640 1 --2- 172.21.15.42:0/749399702 >> v2:172.21.15.154:6812/254845342 conn(0x7fd160037810 0x7fd160039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:06:03.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.541+0000 7fd171ffb640 1 -- 172.21.15.42:0/749399702 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(706..706 src has 451..706) ==== 5040+0+0 (secure 0 0 0) 0x7fd16c048c60 con 0x7fd17c0a1710 2024-09-16T04:06:03.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.541+0000 7fd171ffb640 10 client.984408.objecter ms_dispatch 0x7fd17c000ba0 osd_map(706..706 src has 451..706) 2024-09-16T04:06:03.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.541+0000 7fd171ffb640 3 client.984408.objecter handle_osd_map got epochs [706,706] > 0 2024-09-16T04:06:03.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.541+0000 7fd171ffb640 3 client.984408.objecter handle_osd_map decoding full epoch 706 2024-09-16T04:06:03.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.541+0000 7fd1811fa640 10 monclient(hunting): handle_auth_done global_id 984411 payload 824 2024-09-16T04:06:03.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.541+0000 7fd1811fa640 1 -- 172.21.15.42:0/749399702 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd17c11e720 con 0x7fd17c132970 2024-09-16T04:06:03.572 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.541+0000 7fd171ffb640 20 client.984408.objecter dump_active .. 0 homeless 2024-09-16T04:06:03.572 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.541+0000 7fd171ffb640 10 client.984408.objecter ms_handle_connect 0x7fd17c132970 2024-09-16T04:06:03.572 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.541+0000 7fd171ffb640 10 client.984408.objecter resend_mon_ops 2024-09-16T04:06:03.572 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.541+0000 7fd1811fa640 1 --2- 172.21.15.42:0/749399702 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd17c132970 0x7fd17c132d50 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fd174009fd0 tx=0x7fd174007700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:06:03.572 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.545+0000 7fd1819fb640 1 --2- 172.21.15.42:0/749399702 >> v2:172.21.15.154:6812/254845342 conn(0x7fd160037810 0x7fd160039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:06:03.572 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.545+0000 7fd171ffb640 1 -- 172.21.15.42:0/749399702 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd174015420 con 0x7fd17c132970 2024-09-16T04:06:03.572 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.545+0000 7fd1819fb640 10 monclient: get_auth_request con 0x7fd160037810 auth_method 0 2024-09-16T04:06:03.572 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.545+0000 7fd171ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:03.572 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.545+0000 7fd171ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:06:03.572 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.545+0000 7fd171ffb640 10 monclient: dump: 2024-09-16T04:06:03.572 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:06:03.573 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:06:03.573 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:06:03.573 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:06:03.573 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:03.573 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:06:03.573 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:06:03.573 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:06:03.573 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:06:03.573 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:06:03.573 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.545+0000 7fd171ffb640 1 -- 172.21.15.42:0/749399702 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fd17401a070 con 0x7fd17c132970 2024-09-16T04:06:03.573 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.545+0000 7fd171ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:06:03.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.545+0000 7fd171ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:06:03.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.545+0000 7fd171ffb640 1 -- 172.21.15.42:0/749399702 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd17c132970 msgr2=0x7fd17c132d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:03.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.545+0000 7fd171ffb640 1 --2- 172.21.15.42:0/749399702 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd17c132970 0x7fd17c132d50 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fd174009fd0 tx=0x7fd174007700 comp rx=0 tx=0).stop 2024-09-16T04:06:03.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.545+0000 7fd171ffb640 1 -- 172.21.15.42:0/749399702 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd17c132970 msgr2=0x7fd17c132d50 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:06:03.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.545+0000 7fd171ffb640 1 --2- 172.21.15.42:0/749399702 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd17c132970 0x7fd17c132d50 secure :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fd174009fd0 tx=0x7fd174007700 comp rx=0 tx=0).stop 2024-09-16T04:06:03.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.545+0000 7fd1819fb640 1 --2- 172.21.15.42:0/749399702 >> v2:172.21.15.154:6812/254845342 conn(0x7fd160037810 0x7fd160039cd0 secure :-1 s=READY pgs=3863 cs=0 l=1 rev1=1 crypto rx=0x7fd1780ad000 tx=0x7fd178008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:06:03.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.545+0000 7fd171ffb640 10 client.984408.objecter ms_handle_connect 0x7fd160037810 2024-09-16T04:06:03.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.549+0000 7fd18345e640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:06:03.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.549+0000 7fd18345e640 1 --2- 172.21.15.42:0/749399702 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd17c133290 0x7fd17c136390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:06:03.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.549+0000 7fd18345e640 10 monclient(hunting): start opening mon connection 2024-09-16T04:06:03.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.549+0000 7fd18345e640 1 -- 172.21.15.42:0/749399702 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fd17c1336a0 con 0x7fd17c133290 2024-09-16T04:06:03.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.549+0000 7fd1811fa640 1 --2- 172.21.15.42:0/749399702 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd17c133290 0x7fd17c136390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:06:03.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.549+0000 7fd1811fa640 10 monclient: get_auth_request con 0x7fd17c133290 auth_method 0 2024-09-16T04:06:03.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.549+0000 7fd1811fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:06:03.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.549+0000 7fd1811fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:03.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.549+0000 7fd1811fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:03.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.549+0000 7fd1811fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:06:03.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.549+0000 7fd1811fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:06:03.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.549+0000 7fd1811fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:06:03.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.549+0000 7fd1811fa640 10 monclient(hunting): handle_auth_done global_id 984414 payload 824 2024-09-16T04:06:03.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.549+0000 7fd1811fa640 1 -- 172.21.15.42:0/749399702 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fd17c13a100 con 0x7fd17c133290 2024-09-16T04:06:03.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.549+0000 7fd171ffb640 10 client.984408.objecter ms_handle_connect 0x7fd17c133290 2024-09-16T04:06:03.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.549+0000 7fd1811fa640 1 --2- 172.21.15.42:0/749399702 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd17c133290 0x7fd17c136390 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fd174009fd0 tx=0x7fd174004a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:06:03.578 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.549+0000 7fd171ffb640 10 client.984408.objecter resend_mon_ops 2024-09-16T04:06:03.578 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.549+0000 7fd171ffb640 1 -- 172.21.15.42:0/749399702 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd174015630 con 0x7fd17c133290 2024-09-16T04:06:03.578 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.549+0000 7fd171ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:03.578 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.549+0000 7fd171ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:06:03.578 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.549+0000 7fd171ffb640 10 monclient: dump: 2024-09-16T04:06:03.578 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:06:03.578 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:06:03.578 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:06:03.578 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:06:03.578 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:03.578 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:06:03.579 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:06:03.579 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:06:03.579 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:06:03.579 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:06:03.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.549+0000 7fd171ffb640 1 -- 172.21.15.42:0/749399702 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4715 (secure 0 0 0) 0x7fd174015630 con 0x7fd17c133290 2024-09-16T04:06:03.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.549+0000 7fd171ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:06:03.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.549+0000 7fd171ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:06:03.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.549+0000 7fd171ffb640 1 -- 172.21.15.42:0/749399702 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd17c133290 msgr2=0x7fd17c136390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:03.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.549+0000 7fd171ffb640 1 --2- 172.21.15.42:0/749399702 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd17c133290 0x7fd17c136390 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fd174009fd0 tx=0x7fd174004a50 comp rx=0 tx=0).stop 2024-09-16T04:06:03.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.549+0000 7fd171ffb640 1 -- 172.21.15.42:0/749399702 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd17c133290 msgr2=0x7fd17c136390 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:06:03.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.549+0000 7fd171ffb640 1 --2- 172.21.15.42:0/749399702 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd17c133290 0x7fd17c136390 secure :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fd174009fd0 tx=0x7fd174004a50 comp rx=0 tx=0).stop 2024-09-16T04:06:03.580 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T04:06:03.580 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:06:03.580 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:06:03.580 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "leader", 2024-09-16T04:06:03.580 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 222, 2024-09-16T04:06:03.580 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 123606, 2024-09-16T04:06:03.580 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T04:06:03.580 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T04:06:03.581 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T04:06:03.581 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T04:06:03.581 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:06:03.581 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 4, 2024-09-16T04:06:03.581 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:06:03.581 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T04:06:03.581 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T04:06:03.581 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:06:03.581 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:06:03.581 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:06:03.581 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:06:03.582 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:06:03.582 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:06:03.582 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:06:03.582 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:06:03.582 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:06:03.582 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:06:03.582 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:06:03.582 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:06:03.582 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:06:03.582 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T04:06:03.583 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:06:03.583 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:06:03.583 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:06:03.583 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:06:03.583 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:06:03.583 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:06:03.583 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:06:03.583 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:06:03.583 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:06:03.583 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:06:03.583 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:06:03.584 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:06:03.584 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:06:03.584 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T04:06:03.584 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T04:06:03.584 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T04:06:03.584 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T04:06:03.584 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T04:06:03.584 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T04:06:03.584 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:06:03.584 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:06:03.585 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T04:06:03.585 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T04:06:03.585 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T04:06:03.585 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T04:06:03.585 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T04:06:03.585 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T04:06:03.585 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T04:06:03.585 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:06:03.585 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T04:06:03.585 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:06:03.585 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:06:03.585 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:06:03.586 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:06:03.586 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:06:03.586 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:06:03.586 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:06:03.586 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:06:03.586 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:06:03.586 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:06:03.586 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:06:03.586 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:06:03.586 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T04:06:03.586 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:06:03.586 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T04:06:03.587 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:06:03.587 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:06:03.587 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:06:03.587 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:06:03.587 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:06:03.587 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:06:03.587 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:06:03.587 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T04:06:03.587 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:06:03.587 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:06:03.587 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:06:03.588 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:06:03.588 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T04:06:03.588 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:06:03.588 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:06:03.588 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:06:03.588 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:06:03.588 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T04:06:03.588 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T04:06:03.588 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:06:03.588 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:06:03.588 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:06:03.588 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:06:03.589 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:06:03.589 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:06:03.589 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:06:03.589 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:06:03.589 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:06:03.589 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:06:03.589 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:06:03.589 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T04:06:03.589 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:06:03.590 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:06:03.590 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:06:03.590 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:06:03.590 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T04:06:03.590 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:06:03.590 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:06:03.590 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:06:03.590 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:06:03.590 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T04:06:03.590 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T04:06:03.590 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:06:03.591 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:06:03.591 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:06:03.591 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:06:03.591 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:06:03.591 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:06:03.591 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:06:03.591 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:06:03.591 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:06:03.591 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:06:03.591 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:06:03.591 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T04:06:03.592 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:06:03.592 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:06:03.592 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:06:03.592 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:06:03.592 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T04:06:03.592 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:06:03.592 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:06:03.592 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:06:03.592 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:06:03.592 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T04:06:03.592 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T04:06:03.593 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:06:03.593 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:06:03.593 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:06:03.593 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:06:03.593 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:06:03.593 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:06:03.593 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T04:06:03.593 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T04:06:03.593 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:06:03.593 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:06:03.593 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:06:03.593 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:06:03.594 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:06:03.594 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:06:03.594 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T04:06:03.594 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:06:03.594 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:06:03.594 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:06:03.594 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 3 2024-09-16T04:06:03.594 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:06:03.594 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:06:03.594 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T04:06:03.594 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:06:03.594 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:06:03.595 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:06:03.595 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 3 2024-09-16T04:06:03.595 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:06:03.595 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:06:03.595 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:06:03.595 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T04:06:03.595 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T04:06:03.597 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.549+0000 7fd18345e640 10 librados: watch_flush enter 2024-09-16T04:06:03.597 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.549+0000 7fd18345e640 10 librados: watch_flush exit 2024-09-16T04:06:03.597 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.549+0000 7fd18345e640 20 client.984408.objecter shutdown clearing up homeless session... 2024-09-16T04:06:03.598 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.549+0000 7fd18345e640 10 client.984408.objecter successfully canceled tick 2024-09-16T04:06:03.598 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.549+0000 7fd18345e640 1 -- 172.21.15.42:0/749399702 >> v2:172.21.15.154:6812/254845342 conn(0x7fd160037810 msgr2=0x7fd160039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:03.598 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.549+0000 7fd18345e640 1 --2- 172.21.15.42:0/749399702 >> v2:172.21.15.154:6812/254845342 conn(0x7fd160037810 0x7fd160039cd0 secure :-1 s=READY pgs=3863 cs=0 l=1 rev1=1 crypto rx=0x7fd1780ad000 tx=0x7fd178008040 comp rx=0 tx=0).stop 2024-09-16T04:06:03.598 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.549+0000 7fd18345e640 10 monclient: shutdown 2024-09-16T04:06:03.598 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.549+0000 7fd18345e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:06:03.598 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.549+0000 7fd18345e640 1 -- 172.21.15.42:0/749399702 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd17c0a1710 msgr2=0x7fd17c11c4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:03.598 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.549+0000 7fd18345e640 1 --2- 172.21.15.42:0/749399702 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd17c0a1710 0x7fd17c11c4b0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fd16c00e900 tx=0x7fd16c00edc0 comp rx=0 tx=0).stop 2024-09-16T04:06:03.598 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.549+0000 7fd1819fb640 1 -- 172.21.15.42:0/749399702 reap_dead start 2024-09-16T04:06:03.599 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.549+0000 7fd1819fb640 1 -- 172.21.15.42:0/749399702 reap_dead start 2024-09-16T04:06:03.599 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.549+0000 7fd18345e640 1 -- 172.21.15.42:0/749399702 shutdown_connections 2024-09-16T04:06:03.599 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.549+0000 7fd18345e640 1 -- 172.21.15.42:0/749399702 >> 172.21.15.42:0/749399702 conn(0x7fd17c0a2420 msgr2=0x7fd17c070450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:06:03.599 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.549+0000 7fd18345e640 1 -- 172.21.15.42:0/749399702 shutdown_connections 2024-09-16T04:06:03.599 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.549+0000 7fd18345e640 1 -- 172.21.15.42:0/749399702 wait complete. 2024-09-16T04:06:03.599 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.553+0000 7fd18345e640 1 librados: shutdown 2024-09-16T04:06:03.600 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:06:03.600 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T04:06:03.600 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T04:06:03.725 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.717+0000 7f5113a08640 10 monclient: get_monmap_and_config 2024-09-16T04:06:03.725 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.717+0000 7f5113a08640 10 monclient: build_initial_monmap 2024-09-16T04:06:03.726 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.717+0000 7f5113a08640 10 monclient: monmap: 2024-09-16T04:06:03.726 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:06:03.726 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:06:03.726 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:06:03.724037+0000 2024-09-16T04:06:03.726 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:06:03.724037+0000 2024-09-16T04:06:03.726 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:06:03.726 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:06:03.726 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:06:03.726 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:06:03.726 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:06:03.727 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:06:03.727 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:06:03.727 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:06:03.729 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.721+0000 7f5113a08640 1 Processor -- start 2024-09-16T04:06:03.729 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.721+0000 7f5113a08640 1 -- start start 2024-09-16T04:06:03.729 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.721+0000 7f5113a08640 10 monclient: init 2024-09-16T04:06:03.729 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.721+0000 7f5113a08640 10 monclient: _reopen_session rank -1 2024-09-16T04:06:03.729 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.721+0000 7f5113a08640 10 monclient: _add_conns ranks=[2,1,3,4,0] 2024-09-16T04:06:03.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.721+0000 7f5113a08640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f510c0a39b0 0x7f510c133b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:06:03.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.721+0000 7f5113a08640 10 monclient(hunting): picked mon.noname-b con 0x7f510c0a39b0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:06:03.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.721+0000 7f5113a08640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f510c0a3d90 addr v1:172.21.15.154:6789/0 2024-09-16T04:06:03.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.721+0000 7f5113a08640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f510c137710 0x7f510c13f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:06:03.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.721+0000 7f5113a08640 10 monclient(hunting): picked mon.noname-a con 0x7f510c137710 addr v2:172.21.15.42:3300/0 2024-09-16T04:06:03.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.721+0000 7f5113a08640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f510c09c140 con 0x7f510c0a3d90 2024-09-16T04:06:03.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.721+0000 7f5113a08640 10 monclient(hunting): start opening mon connection 2024-09-16T04:06:03.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.721+0000 7f5113a08640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f510c09c420 con 0x7f510c137710 2024-09-16T04:06:03.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.721+0000 7f5113a08640 10 monclient(hunting): start opening mon connection 2024-09-16T04:06:03.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.721+0000 7f5113a08640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f510c06dd60 con 0x7f510c0a39b0 2024-09-16T04:06:03.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.721+0000 7f5113a08640 10 monclient(hunting): _renew_subs 2024-09-16T04:06:03.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.721+0000 7f5113a08640 10 monclient(hunting): authenticate will time out at 2410.693710s 2024-09-16T04:06:03.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.721+0000 7f51117a4640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f510c0a39b0 0x7f510c133b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:06:03.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.721+0000 7f5111fa5640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f510c137710 0x7f510c13f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:06:03.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.721+0000 7f51117a4640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f510c0a39b0 0x7f510c133b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:58312/0 (socket says 172.21.15.42:58312) 2024-09-16T04:06:03.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.721+0000 7f5111fa5640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f510c137710 0x7f510c13f3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:38798/0 (socket says 172.21.15.42:38798) 2024-09-16T04:06:03.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.721+0000 7f51117a4640 1 -- 172.21.15.42:0/4063785972 learned_addr learned my addr 172.21.15.42:0/4063785972 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:06:03.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.721+0000 7f51117a4640 10 monclient(hunting): get_auth_request con 0x7f510c0a39b0 auth_method 0 2024-09-16T04:06:03.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.721+0000 7f51117a4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:06:03.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.721+0000 7f51117a4640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:03.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.721+0000 7f51117a4640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:03.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.721+0000 7f5111fa5640 10 monclient(hunting): get_auth_request con 0x7f510c137710 auth_method 0 2024-09-16T04:06:03.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.721+0000 7f5111fa5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:06:03.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.721+0000 7f5111fa5640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:03.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.721+0000 7f5111fa5640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:03.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.721+0000 7f5111fa5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:06:03.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.721+0000 7f5111fa5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:06:03.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.721+0000 7f5111fa5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:06:03.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.721+0000 7f51117a4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:06:03.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.721+0000 7f51117a4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:06:03.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.721+0000 7f51117a4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:06:03.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.721+0000 7f51117a4640 10 monclient(hunting): handle_auth_done global_id 1004141 payload 274 2024-09-16T04:06:03.733 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.721+0000 7f51117a4640 1 -- 172.21.15.42:0/4063785972 >> v2:172.21.15.42:3300/0 conn(0x7f510c137710 msgr2=0x7f510c13f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:06:03.733 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.721+0000 7f51117a4640 1 --2- 172.21.15.42:0/4063785972 >> v2:172.21.15.42:3300/0 conn(0x7f510c137710 0x7f510c13f3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:06:03.733 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.721+0000 7f51117a4640 1 -- 172.21.15.42:0/4063785972 >> v1:172.21.15.154:6789/0 conn(0x7f510c0a3d90 legacy=0x7f510c137000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:06:03.733 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.721+0000 7f51117a4640 10 monclient: _finish_hunting 0 2024-09-16T04:06:03.733 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.721+0000 7f51117a4640 1 monclient: found mon.noname-b 2024-09-16T04:06:03.733 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.721+0000 7f51117a4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:06:03.733 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.721+0000 7f51117a4640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T04:06:03.733 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.721+0000 7f51117a4640 1 -- 172.21.15.42:0/4063785972 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f510c0a5e20 con 0x7f510c0a39b0 2024-09-16T04:06:03.733 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.721+0000 7f5111fa5640 1 --2- 172.21.15.42:0/4063785972 >> v2:172.21.15.42:3300/0 conn(0x7f510c137710 0x7f510c13f3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:06:03.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.725+0000 7f51117a4640 1 --2- 172.21.15.42:0/4063785972 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f510c0a39b0 0x7f510c133b80 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f5108009870 tx=0x7f510801c3e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ae0abde04e47841c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:06:03.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.725+0000 7f5103fff640 1 -- 172.21.15.42:0/4063785972 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51080b0070 con 0x7f510c0a39b0 2024-09-16T04:06:03.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.725+0000 7f5103fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:03.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.725+0000 7f5103fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:06:03.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.725+0000 7f5103fff640 10 monclient: dump: 2024-09-16T04:06:03.741 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:06:03.741 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:06:03.741 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:06:03.741 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:06:03.741 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:03.741 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:06:03.741 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:06:03.741 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:06:03.741 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:06:03.741 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:06:03.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.725+0000 7f5103fff640 10 monclient: _finish_auth 0 2024-09-16T04:06:03.742 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.725+0000 7f5103fff640 10 monclient: _check_auth_tickets 2024-09-16T04:06:03.742 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.725+0000 7f5103fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:06:03.742 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.725+0000 7f5113a08640 5 monclient: authenticate success, global_id 1004141 2024-09-16T04:06:03.742 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.725+0000 7f5113a08640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:06:03.742 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.725+0000 7f5103fff640 1 -- 172.21.15.42:0/4063785972 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f51080ab600 con 0x7f510c0a39b0 2024-09-16T04:06:03.742 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.725+0000 7f5103fff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:06:03.742 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.725+0000 7f5113a08640 10 monclient: get_monmap_and_config success 2024-09-16T04:06:03.742 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.725+0000 7f5113a08640 4 set_mon_vals no callback set 2024-09-16T04:06:03.742 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.725+0000 7f5103fff640 1 -- 172.21.15.42:0/4063785972 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51080ab9d0 con 0x7f510c0a39b0 2024-09-16T04:06:03.742 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.725+0000 7f5103fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:03.742 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.725+0000 7f5103fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:06:03.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.725+0000 7f5103fff640 10 monclient: dump: 2024-09-16T04:06:03.743 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:06:03.743 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:06:03.743 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:06:03.743 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:06:03.743 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:03.743 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:06:03.743 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:06:03.743 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:06:03.743 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:06:03.743 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:06:03.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.725+0000 7f5113a08640 10 monclient: shutdown 2024-09-16T04:06:03.744 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.725+0000 7f5113a08640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:06:03.744 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.725+0000 7f5113a08640 1 -- 172.21.15.42:0/4063785972 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f510c0a39b0 msgr2=0x7f510c133b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:03.744 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.725+0000 7f5113a08640 1 --2- 172.21.15.42:0/4063785972 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f510c0a39b0 0x7f510c133b80 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f5108009870 tx=0x7f510801c3e0 comp rx=0 tx=0).stop 2024-09-16T04:06:03.744 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.725+0000 7f5113a08640 1 -- 172.21.15.42:0/4063785972 shutdown_connections 2024-09-16T04:06:03.744 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.725+0000 7f5113a08640 1 --2- 172.21.15.42:0/4063785972 >> v2:172.21.15.42:3300/0 conn(0x7f510c137710 0x7f510c13f3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:06:03.744 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.725+0000 7f5113a08640 1 --2- 172.21.15.42:0/4063785972 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f510c0a39b0 0x7f510c133b80 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:06:03.744 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.725+0000 7f5113a08640 1 -- 172.21.15.42:0/4063785972 >> 172.21.15.42:0/4063785972 conn(0x7f510c0a9fb0 msgr2=0x7f510c0aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:06:03.744 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.725+0000 7f5113a08640 1 -- 172.21.15.42:0/4063785972 shutdown_connections 2024-09-16T04:06:03.744 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.725+0000 7f5113a08640 1 -- 172.21.15.42:0/4063785972 wait complete. 2024-09-16T04:06:03.744 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.725+0000 7f5113a08640 10 monclient: build_initial_monmap 2024-09-16T04:06:03.744 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.725+0000 7f5113a08640 10 monclient: monmap: 2024-09-16T04:06:03.744 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:06:03.745 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:06:03.745 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:06:03.745 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:06:03.745 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:06:03.745 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:06:03.745 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:06:03.745 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:06:03.745 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:06:03.745 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:06:03.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.725+0000 7f5113a08640 1 librados: starting msgr at 2024-09-16T04:06:03.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.725+0000 7f5113a08640 1 librados: starting objecter 2024-09-16T04:06:03.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.725+0000 7f5113a08640 1 Processor -- start 2024-09-16T04:06:03.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.725+0000 7f5113a08640 1 -- start start 2024-09-16T04:06:03.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.725+0000 7f5113a08640 1 librados: setting wanted keys 2024-09-16T04:06:03.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.725+0000 7f5113a08640 1 librados: calling monclient init 2024-09-16T04:06:03.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.725+0000 7f5113a08640 10 monclient: init 2024-09-16T04:06:03.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.725+0000 7f5113a08640 10 monclient: _reopen_session rank -1 2024-09-16T04:06:03.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.725+0000 7f5113a08640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:06:03.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.725+0000 7f5113a08640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f510c0a39b0 0x7f510c1cf760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:06:03.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.725+0000 7f5113a08640 10 monclient(hunting): picked mon.noname-c con 0x7f510c0a39b0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:06:03.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.725+0000 7f5113a08640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f510c0a3d90 0x7f510c1cfca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:06:03.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.725+0000 7f5113a08640 10 monclient(hunting): picked mon.noname-a con 0x7f510c0a3d90 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:06:03.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.725+0000 7f5113a08640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f510c137710 0x7f510c1d01e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:06:03.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.725+0000 7f5113a08640 10 monclient(hunting): picked mon.noname-b con 0x7f510c137710 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:06:03.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.725+0000 7f5113a08640 10 monclient(hunting): start opening mon connection 2024-09-16T04:06:03.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.729+0000 7f5113a08640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f510c142310 con 0x7f510c0a3d90 2024-09-16T04:06:03.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.729+0000 7f5113a08640 10 monclient(hunting): start opening mon connection 2024-09-16T04:06:03.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.729+0000 7f5113a08640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f510c142490 con 0x7f510c137710 2024-09-16T04:06:03.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.729+0000 7f5113a08640 10 monclient(hunting): start opening mon connection 2024-09-16T04:06:03.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.729+0000 7f5113a08640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f510c06f680 con 0x7f510c0a39b0 2024-09-16T04:06:03.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.729+0000 7f5113a08640 10 monclient(hunting): _renew_subs 2024-09-16T04:06:03.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.729+0000 7f5113a08640 10 monclient(hunting): authenticate will time out at 2410.700024s 2024-09-16T04:06:03.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.729+0000 7f51117a4640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f510c0a39b0 0x7f510c1cf760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:06:03.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.729+0000 7f51117a4640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f510c0a39b0 0x7f510c1cf760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:58322/0 (socket says 172.21.15.42:58322) 2024-09-16T04:06:03.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.729+0000 7f51117a4640 1 -- 172.21.15.42:0/3564551628 learned_addr learned my addr 172.21.15.42:0/3564551628 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:06:03.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.729+0000 7f51117a4640 10 monclient(hunting): get_auth_request con 0x7f510c0a39b0 auth_method 0 2024-09-16T04:06:03.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.729+0000 7f51117a4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:06:03.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.729+0000 7f51117a4640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:03.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.729+0000 7f51117a4640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:03.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.729+0000 7f5110fa3640 1 --2- 172.21.15.42:0/3564551628 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f510c0a3d90 0x7f510c1cfca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:06:03.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.729+0000 7f5110fa3640 10 monclient(hunting): get_auth_request con 0x7f510c0a3d90 auth_method 0 2024-09-16T04:06:03.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.729+0000 7f5110fa3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:06:03.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.729+0000 7f5110fa3640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:03.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.729+0000 7f5110fa3640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:03.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.729+0000 7f51117a4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:06:03.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.729+0000 7f51117a4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:06:03.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.729+0000 7f51117a4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:06:03.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.729+0000 7f5110fa3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:06:03.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.729+0000 7f5110fa3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:06:03.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.729+0000 7f5110fa3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:06:03.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.729+0000 7f5111fa5640 1 --2- 172.21.15.42:0/3564551628 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f510c137710 0x7f510c1d01e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:06:03.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.729+0000 7f51117a4640 10 monclient(hunting): handle_auth_done global_id 1004144 payload 824 2024-09-16T04:06:03.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.729+0000 7f51117a4640 1 -- 172.21.15.42:0/3564551628 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f510c137710 msgr2=0x7f510c1d01e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:03.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.729+0000 7f51117a4640 1 --2- 172.21.15.42:0/3564551628 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f510c137710 0x7f510c1d01e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:06:03.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.729+0000 7f51117a4640 1 -- 172.21.15.42:0/3564551628 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f510c0a3d90 msgr2=0x7f510c1cfca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:03.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.729+0000 7f51117a4640 1 --2- 172.21.15.42:0/3564551628 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f510c0a3d90 0x7f510c1cfca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:06:03.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.729+0000 7f51117a4640 10 monclient: _finish_hunting 0 2024-09-16T04:06:03.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.729+0000 7f51117a4640 1 monclient: found mon.noname-c 2024-09-16T04:06:03.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.729+0000 7f51117a4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:06:03.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.729+0000 7f51117a4640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T04:06:03.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.729+0000 7f51117a4640 1 -- 172.21.15.42:0/3564551628 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f510c13ca60 con 0x7f510c0a39b0 2024-09-16T04:06:03.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.729+0000 7f5111fa5640 10 monclient: get_auth_request con 0x7f510c137710 auth_method 0 2024-09-16T04:06:03.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.729+0000 7f5111fa5640 1 --2- 172.21.15.42:0/3564551628 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f510c137710 0x7f510c1d01e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T04:06:03.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.729+0000 7f51117a4640 1 --2- 172.21.15.42:0/3564551628 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f510c0a39b0 0x7f510c1cf760 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f51080ae040 tx=0x7f5108019180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:06:03.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.729+0000 7f5101ffb640 10 client.?.objecter ms_handle_connect 0x7f510c0a39b0 2024-09-16T04:06:03.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.729+0000 7f5101ffb640 10 client.?.objecter resend_mon_ops 2024-09-16T04:06:03.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.729+0000 7f5101ffb640 1 -- 172.21.15.42:0/3564551628 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51080b0040 con 0x7f510c0a39b0 2024-09-16T04:06:03.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.729+0000 7f5101ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:03.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.729+0000 7f5101ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:06:03.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.729+0000 7f5101ffb640 10 monclient: dump: 2024-09-16T04:06:03.750 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:06:03.751 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:06:03.751 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:06:03.751 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:06:03.751 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:03.751 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:06:03.751 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:06:03.751 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:06:03.751 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:06:03.751 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:06:03.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.729+0000 7f5101ffb640 10 monclient: _finish_auth 0 2024-09-16T04:06:03.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.729+0000 7f5101ffb640 10 monclient: _check_auth_tickets 2024-09-16T04:06:03.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.729+0000 7f5101ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:06:03.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.729+0000 7f5101ffb640 1 -- 172.21.15.42:0/3564551628 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f51080b9070 con 0x7f510c0a39b0 2024-09-16T04:06:03.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.729+0000 7f5113a08640 5 monclient: authenticate success, global_id 1004144 2024-09-16T04:06:03.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.729+0000 7f5113a08640 10 monclient: _renew_subs 2024-09-16T04:06:03.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.729+0000 7f5113a08640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:06:03.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.729+0000 7f5113a08640 1 -- 172.21.15.42:0/3564551628 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f510c13bb50 con 0x7f510c0a39b0 2024-09-16T04:06:03.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.729+0000 7f5101ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:06:03.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.729+0000 7f5101ffb640 1 -- 172.21.15.42:0/3564551628 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51080b4880 con 0x7f510c0a39b0 2024-09-16T04:06:03.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.729+0000 7f5101ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:03.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.729+0000 7f5101ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:06:03.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.729+0000 7f5101ffb640 10 monclient: dump: 2024-09-16T04:06:03.753 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:06:03.753 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:06:03.753 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:06:03.753 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:06:03.753 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:03.753 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:06:03.753 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:06:03.753 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:06:03.753 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:06:03.754 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:06:03.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.729+0000 7f5113a08640 10 client.1004144.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:06:03.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.729+0000 7f5113a08640 10 monclient: _renew_subs 2024-09-16T04:06:03.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.729+0000 7f5113a08640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:06:03.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.729+0000 7f5113a08640 1 -- 172.21.15.42:0/3564551628 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f510c13deb0 con 0x7f510c0a39b0 2024-09-16T04:06:03.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.729+0000 7f51037fe640 4 set_mon_vals no callback set 2024-09-16T04:06:03.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.729+0000 7f5113a08640 1 librados: init done 2024-09-16T04:06:03.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.729+0000 7f5113a08640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:06:03.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.729+0000 7f5113a08640 1 --2- 172.21.15.42:0/3564551628 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f510c12d2d0 0x7f510c12d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:06:03.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.729+0000 7f5113a08640 10 monclient(hunting): start opening mon connection 2024-09-16T04:06:03.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.729+0000 7f5113a08640 1 -- 172.21.15.42:0/3564551628 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f510c1342d0 con 0x7f510c12d2d0 2024-09-16T04:06:03.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.729+0000 7f5110fa3640 1 --2- 172.21.15.42:0/3564551628 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f510c12d2d0 0x7f510c12d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:06:03.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.729+0000 7f5110fa3640 10 monclient: get_auth_request con 0x7f510c12d2d0 auth_method 0 2024-09-16T04:06:03.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.729+0000 7f5110fa3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:06:03.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.729+0000 7f5110fa3640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:03.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.729+0000 7f5110fa3640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:03.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.729+0000 7f5110fa3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:06:03.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.729+0000 7f5110fa3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:06:03.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.729+0000 7f5101ffb640 1 -- 172.21.15.42:0/3564551628 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f51080b4a20 con 0x7f510c0a39b0 2024-09-16T04:06:03.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.729+0000 7f5110fa3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:06:03.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.729+0000 7f5110fa3640 10 monclient(hunting): handle_auth_done global_id 1004147 payload 824 2024-09-16T04:06:03.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.729+0000 7f5101ffb640 1 --2- 172.21.15.42:0/3564551628 >> v2:172.21.15.154:6812/254845342 conn(0x7f50f0037810 0x7f50f0039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:06:03.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.729+0000 7f5110fa3640 1 -- 172.21.15.42:0/3564551628 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f510c13d7a0 con 0x7f510c12d2d0 2024-09-16T04:06:03.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.729+0000 7f5101ffb640 1 -- 172.21.15.42:0/3564551628 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(706..706 src has 451..706) ==== 5040+0+0 (secure 0 0 0) 0x7f51080e2a90 con 0x7f510c0a39b0 2024-09-16T04:06:03.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.729+0000 7f5101ffb640 10 client.1004144.objecter ms_dispatch 0x7f510c000ba0 osd_map(706..706 src has 451..706) 2024-09-16T04:06:03.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.729+0000 7f5101ffb640 3 client.1004144.objecter handle_osd_map got epochs [706,706] > 0 2024-09-16T04:06:03.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.729+0000 7f5101ffb640 3 client.1004144.objecter handle_osd_map decoding full epoch 706 2024-09-16T04:06:03.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.729+0000 7f5101ffb640 20 client.1004144.objecter dump_active .. 0 homeless 2024-09-16T04:06:03.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.729+0000 7f5101ffb640 10 client.1004144.objecter ms_handle_connect 0x7f510c12d2d0 2024-09-16T04:06:03.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.729+0000 7f5101ffb640 10 client.1004144.objecter resend_mon_ops 2024-09-16T04:06:03.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.729+0000 7f5110fa3640 1 --2- 172.21.15.42:0/3564551628 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f510c12d2d0 0x7f510c12d6b0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f50fc007fd0 tx=0x7f50fc00fc00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:06:03.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.729+0000 7f5101ffb640 1 -- 172.21.15.42:0/3564551628 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50fc003970 con 0x7f510c12d2d0 2024-09-16T04:06:03.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.729+0000 7f5101ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:03.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.729+0000 7f5101ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:06:03.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.729+0000 7f5101ffb640 10 monclient: dump: 2024-09-16T04:06:03.757 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:06:03.757 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:06:03.757 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:06:03.757 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:06:03.757 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:03.757 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:06:03.757 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:06:03.758 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:06:03.758 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:06:03.758 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:06:03.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.729+0000 7f5111fa5640 1 --2- 172.21.15.42:0/3564551628 >> v2:172.21.15.154:6812/254845342 conn(0x7f50f0037810 0x7f50f0039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:06:03.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.729+0000 7f5111fa5640 10 monclient: get_auth_request con 0x7f50f0037810 auth_method 0 2024-09-16T04:06:03.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.729+0000 7f5101ffb640 1 -- 172.21.15.42:0/3564551628 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f50fc019070 con 0x7f510c12d2d0 2024-09-16T04:06:03.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.729+0000 7f5101ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:06:03.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.729+0000 7f5101ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:06:03.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.729+0000 7f5101ffb640 1 -- 172.21.15.42:0/3564551628 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f510c12d2d0 msgr2=0x7f510c12d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:03.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.729+0000 7f5101ffb640 1 --2- 172.21.15.42:0/3564551628 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f510c12d2d0 0x7f510c12d6b0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f50fc007fd0 tx=0x7f50fc00fc00 comp rx=0 tx=0).stop 2024-09-16T04:06:03.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.729+0000 7f5101ffb640 1 -- 172.21.15.42:0/3564551628 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f510c12d2d0 msgr2=0x7f510c12d6b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:06:03.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.729+0000 7f5101ffb640 1 --2- 172.21.15.42:0/3564551628 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f510c12d2d0 0x7f510c12d6b0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:06:03.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.729+0000 7f5101ffb640 10 client.1004144.objecter ms_handle_connect 0x7f50f0037810 2024-09-16T04:06:03.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.729+0000 7f5111fa5640 1 --2- 172.21.15.42:0/3564551628 >> v2:172.21.15.154:6812/254845342 conn(0x7f50f0037810 0x7f50f0039cd0 secure :-1 s=READY pgs=3864 cs=0 l=1 rev1=1 crypto rx=0x7f510c132dd0 tx=0x7f510400b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:06:03.761 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.737+0000 7f5113a08640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:06:03.761 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.737+0000 7f5113a08640 1 --2- 172.21.15.42:0/3564551628 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f510c12dbf0 0x7f510c12f500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:06:03.761 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.737+0000 7f5113a08640 10 monclient(hunting): start opening mon connection 2024-09-16T04:06:03.761 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.737+0000 7f5113a08640 1 -- 172.21.15.42:0/3564551628 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f510c06df10 con 0x7f510c12dbf0 2024-09-16T04:06:03.761 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.737+0000 7f5110fa3640 1 --2- 172.21.15.42:0/3564551628 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f510c12dbf0 0x7f510c12f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:06:03.761 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.737+0000 7f5110fa3640 10 monclient: get_auth_request con 0x7f510c12dbf0 auth_method 0 2024-09-16T04:06:03.761 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.737+0000 7f5110fa3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:06:03.761 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.737+0000 7f5110fa3640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:03.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.737+0000 7f5110fa3640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:03.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.737+0000 7f5110fa3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:06:03.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.737+0000 7f5110fa3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:06:03.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.737+0000 7f5110fa3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:06:03.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.737+0000 7f5110fa3640 10 monclient(hunting): handle_auth_done global_id 1004150 payload 824 2024-09-16T04:06:03.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.737+0000 7f5110fa3640 1 -- 172.21.15.42:0/3564551628 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f510c133b80 con 0x7f510c12dbf0 2024-09-16T04:06:03.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.737+0000 7f5110fa3640 1 --2- 172.21.15.42:0/3564551628 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f510c12dbf0 0x7f510c12f500 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f50fc007fd0 tx=0x7f50fc003770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:06:03.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.737+0000 7f5101ffb640 10 client.1004144.objecter ms_handle_connect 0x7f510c12dbf0 2024-09-16T04:06:03.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.737+0000 7f5101ffb640 10 client.1004144.objecter resend_mon_ops 2024-09-16T04:06:03.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.737+0000 7f5101ffb640 1 -- 172.21.15.42:0/3564551628 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50fc00dd40 con 0x7f510c12dbf0 2024-09-16T04:06:03.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.737+0000 7f5101ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:03.763 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.737+0000 7f5101ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:06:03.763 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.737+0000 7f5101ffb640 10 monclient: dump: 2024-09-16T04:06:03.763 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:06:03.763 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:06:03.763 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:06:03.763 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:06:03.763 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:03.763 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:06:03.763 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:06:03.763 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:06:03.764 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:06:03.764 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:06:03.764 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.737+0000 7f5101ffb640 1 -- 172.21.15.42:0/3564551628 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4544 (secure 0 0 0) 0x7f50fc00dd40 con 0x7f510c12dbf0 2024-09-16T04:06:03.764 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.737+0000 7f5101ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:06:03.764 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.737+0000 7f5101ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:06:03.764 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.737+0000 7f5101ffb640 1 -- 172.21.15.42:0/3564551628 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f510c12dbf0 msgr2=0x7f510c12f500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:03.764 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.737+0000 7f5101ffb640 1 --2- 172.21.15.42:0/3564551628 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f510c12dbf0 0x7f510c12f500 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f50fc007fd0 tx=0x7f50fc003770 comp rx=0 tx=0).stop 2024-09-16T04:06:03.764 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.737+0000 7f5101ffb640 1 -- 172.21.15.42:0/3564551628 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f510c12dbf0 msgr2=0x7f510c12f500 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:06:03.764 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.737+0000 7f5101ffb640 1 --2- 172.21.15.42:0/3564551628 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f510c12dbf0 0x7f510c12f500 secure :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f50fc007fd0 tx=0x7f50fc003770 comp rx=0 tx=0).stop 2024-09-16T04:06:03.765 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T04:06:03.765 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:06:03.765 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:06:03.765 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T04:06:03.765 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 222, 2024-09-16T04:06:03.765 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 9767, 2024-09-16T04:06:03.765 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T04:06:03.765 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T04:06:03.765 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T04:06:03.765 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T04:06:03.766 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:06:03.766 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 4, 2024-09-16T04:06:03.766 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:06:03.766 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T04:06:03.766 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T04:06:03.766 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:06:03.766 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:06:03.766 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:06:03.766 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:06:03.766 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:06:03.766 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:06:03.767 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:06:03.767 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:06:03.767 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:06:03.767 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:06:03.767 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:06:03.767 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:06:03.767 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:06:03.767 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T04:06:03.767 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:06:03.767 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:06:03.767 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:06:03.768 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:06:03.768 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:06:03.768 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:06:03.768 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:06:03.768 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:06:03.768 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:06:03.768 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:06:03.768 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:06:03.768 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:06:03.768 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:06:03.768 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T04:06:03.769 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T04:06:03.769 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T04:06:03.769 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T04:06:03.769 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T04:06:03.769 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T04:06:03.769 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:06:03.769 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:06:03.769 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T04:06:03.769 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T04:06:03.769 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T04:06:03.769 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T04:06:03.770 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T04:06:03.770 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T04:06:03.770 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T04:06:03.770 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:06:03.770 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T04:06:03.770 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:06:03.770 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:06:03.770 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:06:03.770 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:06:03.770 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:06:03.770 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:06:03.770 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:06:03.771 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:06:03.771 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:06:03.771 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:06:03.771 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:06:03.771 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:06:03.771 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T04:06:03.771 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:06:03.771 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T04:06:03.771 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:06:03.771 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:06:03.771 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:06:03.771 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:06:03.772 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:06:03.772 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:06:03.772 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:06:03.772 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T04:06:03.772 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:06:03.772 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:06:03.772 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:06:03.772 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:06:03.772 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T04:06:03.772 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:06:03.772 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:06:03.773 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:06:03.773 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:06:03.773 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T04:06:03.773 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T04:06:03.773 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:06:03.773 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:06:03.773 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:06:03.773 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:06:03.773 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:06:03.773 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:06:03.773 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:06:03.773 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:06:03.774 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:06:03.774 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:06:03.774 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:06:03.774 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T04:06:03.774 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:06:03.774 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:06:03.774 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:06:03.774 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:06:03.774 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T04:06:03.774 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:06:03.774 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:06:03.774 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:06:03.775 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:06:03.775 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T04:06:03.775 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T04:06:03.775 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:06:03.775 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:06:03.775 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:06:03.775 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:06:03.775 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:06:03.775 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:06:03.775 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:06:03.775 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:06:03.775 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:06:03.776 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:06:03.776 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:06:03.776 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T04:06:03.776 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:06:03.776 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:06:03.776 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:06:03.776 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:06:03.776 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T04:06:03.776 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:06:03.776 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:06:03.776 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:06:03.776 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:06:03.777 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T04:06:03.777 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T04:06:03.777 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:06:03.777 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:06:03.777 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:06:03.777 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:06:03.777 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:06:03.777 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:06:03.777 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T04:06:03.777 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T04:06:03.777 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:06:03.777 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:06:03.778 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:06:03.778 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:06:03.778 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:06:03.778 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:06:03.778 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T04:06:03.778 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:06:03.778 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:06:03.778 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:06:03.778 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 2 2024-09-16T04:06:03.778 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:06:03.778 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:06:03.778 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:06:03.779 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T04:06:03.779 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T04:06:03.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.737+0000 7f5113a08640 10 librados: watch_flush enter 2024-09-16T04:06:03.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.737+0000 7f5113a08640 10 librados: watch_flush exit 2024-09-16T04:06:03.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.737+0000 7f5113a08640 20 client.1004144.objecter shutdown clearing up homeless session... 2024-09-16T04:06:03.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.737+0000 7f5113a08640 10 client.1004144.objecter successfully canceled tick 2024-09-16T04:06:03.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.737+0000 7f5113a08640 1 -- 172.21.15.42:0/3564551628 >> v2:172.21.15.154:6812/254845342 conn(0x7f50f0037810 msgr2=0x7f50f0039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:03.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.737+0000 7f5113a08640 1 --2- 172.21.15.42:0/3564551628 >> v2:172.21.15.154:6812/254845342 conn(0x7f50f0037810 0x7f50f0039cd0 secure :-1 s=READY pgs=3864 cs=0 l=1 rev1=1 crypto rx=0x7f510c132dd0 tx=0x7f510400b040 comp rx=0 tx=0).stop 2024-09-16T04:06:03.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.737+0000 7f5113a08640 10 monclient: shutdown 2024-09-16T04:06:03.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.737+0000 7f5113a08640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:06:03.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.737+0000 7f5113a08640 1 -- 172.21.15.42:0/3564551628 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f510c0a39b0 msgr2=0x7f510c1cf760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:03.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.737+0000 7f5113a08640 1 --2- 172.21.15.42:0/3564551628 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f510c0a39b0 0x7f510c1cf760 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f51080ae040 tx=0x7f5108019180 comp rx=0 tx=0).stop 2024-09-16T04:06:03.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.737+0000 7f5111fa5640 1 -- 172.21.15.42:0/3564551628 reap_dead start 2024-09-16T04:06:03.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.737+0000 7f5113a08640 1 -- 172.21.15.42:0/3564551628 shutdown_connections 2024-09-16T04:06:03.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.737+0000 7f5113a08640 1 --2- 172.21.15.42:0/3564551628 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f510c0a39b0 0x7f510c1cf760 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:06:03.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.737+0000 7f5113a08640 1 -- 172.21.15.42:0/3564551628 >> 172.21.15.42:0/3564551628 conn(0x7f510c0a9fb0 msgr2=0x7f510c0a8a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:06:03.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.737+0000 7f5113a08640 1 -- 172.21.15.42:0/3564551628 shutdown_connections 2024-09-16T04:06:03.782 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.737+0000 7f5113a08640 1 -- 172.21.15.42:0/3564551628 wait complete. 2024-09-16T04:06:03.782 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.737+0000 7f5113a08640 1 librados: shutdown 2024-09-16T04:06:03.782 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:06:03.782 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T04:06:03.782 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T04:06:03.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.893+0000 7f31b602b640 10 monclient: get_monmap_and_config 2024-09-16T04:06:03.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.893+0000 7f31b602b640 10 monclient: build_initial_monmap 2024-09-16T04:06:03.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.893+0000 7f31b602b640 10 monclient: monmap: 2024-09-16T04:06:03.901 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:06:03.901 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:06:03.901 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:06:03.899822+0000 2024-09-16T04:06:03.901 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:06:03.899822+0000 2024-09-16T04:06:03.902 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:06:03.902 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:06:03.902 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:06:03.902 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:06:03.902 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:06:03.902 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:06:03.902 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:06:03.902 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:06:03.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.897+0000 7f31b602b640 1 Processor -- start 2024-09-16T04:06:03.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.897+0000 7f31b602b640 1 -- start start 2024-09-16T04:06:03.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.897+0000 7f31b602b640 10 monclient: init 2024-09-16T04:06:03.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.897+0000 7f31b602b640 10 monclient: _reopen_session rank -1 2024-09-16T04:06:03.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.897+0000 7f31b602b640 10 monclient: _add_conns ranks=[2,0,4,1,3] 2024-09-16T04:06:03.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.897+0000 7f31b602b640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f31b006db90 0x7f31b01340c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:06:03.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.897+0000 7f31b602b640 10 monclient(hunting): picked mon.noname-b con 0x7f31b006db90 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:06:03.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.897+0000 7f31b602b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f31b00a37f0 addr v1:172.21.15.42:6789/0 2024-09-16T04:06:03.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.897+0000 7f31b602b640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f31b00a3bd0 0x7f31b0137520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:06:03.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.897+0000 7f31b602b640 10 monclient(hunting): picked mon.noname-c con 0x7f31b00a3bd0 addr v2:172.21.15.154:3300/0 2024-09-16T04:06:03.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.897+0000 7f31b602b640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f31b006ece0 con 0x7f31b00a37f0 2024-09-16T04:06:03.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.897+0000 7f31b602b640 10 monclient(hunting): start opening mon connection 2024-09-16T04:06:03.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.897+0000 7f31b602b640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f31b006df70 con 0x7f31b00a3bd0 2024-09-16T04:06:03.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.897+0000 7f31b602b640 10 monclient(hunting): start opening mon connection 2024-09-16T04:06:03.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.897+0000 7f31b602b640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f31b009bf80 con 0x7f31b006db90 2024-09-16T04:06:03.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.897+0000 7f31b602b640 10 monclient(hunting): _renew_subs 2024-09-16T04:06:03.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.897+0000 7f31b602b640 10 monclient(hunting): authenticate will time out at 2410.869479s 2024-09-16T04:06:03.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.897+0000 7f31aeffd640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f31b00a37f0 0x7f31b0136e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:56070/0 (socket says 172.21.15.42:56070) 2024-09-16T04:06:03.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.897+0000 7f31aeffd640 1 -- 172.21.15.42:0/3161888583 learned_addr learned my addr 172.21.15.42:0/3161888583 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T04:06:03.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.897+0000 7f31af7fe640 1 --2- 172.21.15.42:0/3161888583 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f31b006db90 0x7f31b01340c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:06:03.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.897+0000 7f31af7fe640 10 monclient(hunting): get_auth_request con 0x7f31b006db90 auth_method 0 2024-09-16T04:06:03.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.897+0000 7f31af7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:06:03.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.897+0000 7f31af7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:03.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.897+0000 7f31af7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:03.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.897+0000 7f31af7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:06:03.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.897+0000 7f31af7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:06:03.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.897+0000 7f31af7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:06:03.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.897+0000 7f31ae7fc640 1 -- 172.21.15.42:0/3161888583 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 518991634 0 0) 0x7f31b006ece0 con 0x7f31b00a37f0 2024-09-16T04:06:03.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.897+0000 7f31ae7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:06:03.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.897+0000 7f31ae7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:03.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.897+0000 7f31ae7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:03.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.897+0000 7f31ae7fc640 10 monclient(hunting): my global_id is 984423 2024-09-16T04:06:03.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.897+0000 7f31ae7fc640 1 -- 172.21.15.42:0/3161888583 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3198003530 con 0x7f31b00a37f0 2024-09-16T04:06:03.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.897+0000 7f31af7fe640 10 monclient(hunting): handle_auth_done global_id 1004153 payload 274 2024-09-16T04:06:03.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.897+0000 7f31af7fe640 1 -- 172.21.15.42:0/3161888583 >> v2:172.21.15.154:3300/0 conn(0x7f31b00a3bd0 msgr2=0x7f31b0137520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:06:03.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.897+0000 7f31af7fe640 1 --2- 172.21.15.42:0/3161888583 >> v2:172.21.15.154:3300/0 conn(0x7f31b00a3bd0 0x7f31b0137520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:06:03.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.897+0000 7f31af7fe640 1 -- 172.21.15.42:0/3161888583 >> v1:172.21.15.42:6789/0 conn(0x7f31b00a37f0 legacy=0x7f31b0136e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:03.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.897+0000 7f31af7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:06:03.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.897+0000 7f31af7fe640 1 monclient: found mon.noname-b 2024-09-16T04:06:03.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.897+0000 7f31af7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:06:03.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.897+0000 7f31af7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T04:06:03.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.897+0000 7f31af7fe640 1 -- 172.21.15.42:0/3161888583 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31b0134780 con 0x7f31b006db90 2024-09-16T04:06:03.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.901+0000 7f31af7fe640 1 --2- 172.21.15.42:0/3161888583 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f31b006db90 0x7f31b01340c0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f31a00097c0 tx=0x7f31a001c360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ee8e58804c731d7b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:06:03.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.901+0000 7f31ae7fc640 1 -- 172.21.15.42:0/3161888583 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31a00b0070 con 0x7f31b006db90 2024-09-16T04:06:03.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.901+0000 7f31ae7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:03.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.901+0000 7f31ae7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:06:03.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.901+0000 7f31ae7fc640 10 monclient: dump: 2024-09-16T04:06:03.913 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:06:03.913 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:06:03.914 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:06:03.914 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:06:03.914 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:03.914 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:06:03.914 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:06:03.914 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:06:03.914 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:06:03.914 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:06:03.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.901+0000 7f31ae7fc640 10 monclient: _finish_auth 0 2024-09-16T04:06:03.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.901+0000 7f31ae7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:06:03.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.901+0000 7f31ae7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:06:03.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.901+0000 7f31ae7fc640 1 -- 172.21.15.42:0/3161888583 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f31a00ab600 con 0x7f31b006db90 2024-09-16T04:06:03.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.901+0000 7f31ae7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:06:03.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.901+0000 7f31b602b640 5 monclient: authenticate success, global_id 1004153 2024-09-16T04:06:03.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.901+0000 7f31b602b640 10 monclient: get_monmap_and_config success 2024-09-16T04:06:03.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.901+0000 7f31b602b640 4 set_mon_vals no callback set 2024-09-16T04:06:03.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.901+0000 7f31ae7fc640 1 -- 172.21.15.42:0/3161888583 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31a00ab9b0 con 0x7f31b006db90 2024-09-16T04:06:03.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.901+0000 7f31ae7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:03.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.901+0000 7f31ae7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:06:03.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.901+0000 7f31ae7fc640 10 monclient: dump: 2024-09-16T04:06:03.915 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:06:03.915 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:06:03.915 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:06:03.915 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:06:03.915 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:03.915 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:06:03.916 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:06:03.916 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:06:03.916 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:06:03.916 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:06:03.916 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.901+0000 7f31b602b640 10 monclient: shutdown 2024-09-16T04:06:03.916 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.901+0000 7f31b602b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:06:03.916 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.901+0000 7f31b602b640 1 -- 172.21.15.42:0/3161888583 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f31b006db90 msgr2=0x7f31b01340c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:03.916 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.901+0000 7f31b602b640 1 --2- 172.21.15.42:0/3161888583 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f31b006db90 0x7f31b01340c0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f31a00097c0 tx=0x7f31a001c360 comp rx=0 tx=0).stop 2024-09-16T04:06:03.916 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.901+0000 7f31b602b640 1 -- 172.21.15.42:0/3161888583 shutdown_connections 2024-09-16T04:06:03.916 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.901+0000 7f31b602b640 1 --2- 172.21.15.42:0/3161888583 >> v2:172.21.15.154:3300/0 conn(0x7f31b00a3bd0 0x7f31b0137520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:06:03.916 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.901+0000 7f31b602b640 1 --2- 172.21.15.42:0/3161888583 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f31b006db90 0x7f31b01340c0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:06:03.916 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.901+0000 7f31b602b640 1 -- 172.21.15.42:0/3161888583 >> 172.21.15.42:0/3161888583 conn(0x7f31b00a9fb0 msgr2=0x7f31b00aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:06:03.916 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.901+0000 7f31b602b640 1 -- 172.21.15.42:0/3161888583 shutdown_connections 2024-09-16T04:06:03.917 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.901+0000 7f31b602b640 1 -- 172.21.15.42:0/3161888583 wait complete. 2024-09-16T04:06:03.917 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.901+0000 7f31b602b640 10 monclient: build_initial_monmap 2024-09-16T04:06:03.917 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.901+0000 7f31b602b640 10 monclient: monmap: 2024-09-16T04:06:03.917 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:06:03.917 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:06:03.917 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:06:03.917 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:06:03.917 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:06:03.917 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:06:03.917 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:06:03.917 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:06:03.917 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:06:03.917 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:06:03.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.901+0000 7f31b602b640 1 librados: starting msgr at 2024-09-16T04:06:03.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.901+0000 7f31b602b640 1 librados: starting objecter 2024-09-16T04:06:03.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.901+0000 7f31b602b640 1 Processor -- start 2024-09-16T04:06:03.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.901+0000 7f31b602b640 1 -- start start 2024-09-16T04:06:03.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.901+0000 7f31b602b640 1 librados: setting wanted keys 2024-09-16T04:06:03.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.901+0000 7f31b602b640 1 librados: calling monclient init 2024-09-16T04:06:03.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.901+0000 7f31b602b640 10 monclient: init 2024-09-16T04:06:03.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.901+0000 7f31b602b640 10 monclient: _reopen_session rank -1 2024-09-16T04:06:03.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.901+0000 7f31b602b640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:06:03.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.901+0000 7f31b602b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f31b006db90 0x7f31b01309b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:06:03.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.901+0000 7f31b602b640 10 monclient(hunting): picked mon.noname-b con 0x7f31b006db90 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:06:03.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.901+0000 7f31b602b640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f31b00a37f0 0x7f31b00a13c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:06:03.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.901+0000 7f31b602b640 10 monclient(hunting): picked mon.noname-c con 0x7f31b00a37f0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:06:03.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.901+0000 7f31b602b640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f31b00a3bd0 0x7f31b00a1900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:06:03.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.901+0000 7f31b602b640 10 monclient(hunting): picked mon.noname-a con 0x7f31b00a3bd0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:06:03.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.901+0000 7f31b602b640 10 monclient(hunting): start opening mon connection 2024-09-16T04:06:03.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.901+0000 7f31b602b640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f31b01419d0 con 0x7f31b00a3bd0 2024-09-16T04:06:03.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.901+0000 7f31b602b640 10 monclient(hunting): start opening mon connection 2024-09-16T04:06:03.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.901+0000 7f31b602b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f31b00a4100 con 0x7f31b006db90 2024-09-16T04:06:03.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.901+0000 7f31b602b640 10 monclient(hunting): start opening mon connection 2024-09-16T04:06:03.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.901+0000 7f31b602b640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f31b0141b50 con 0x7f31b00a37f0 2024-09-16T04:06:03.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.901+0000 7f31b602b640 10 monclient(hunting): _renew_subs 2024-09-16T04:06:03.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.901+0000 7f31b602b640 10 monclient(hunting): authenticate will time out at 2410.875778s 2024-09-16T04:06:03.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.901+0000 7f31aeffd640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f31b00a37f0 0x7f31b00a13c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:06:03.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.901+0000 7f31affff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f31b00a3bd0 0x7f31b00a1900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:06:03.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.901+0000 7f31aeffd640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f31b00a37f0 0x7f31b00a13c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:58376/0 (socket says 172.21.15.42:58376) 2024-09-16T04:06:03.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.905+0000 7f31aeffd640 1 -- 172.21.15.42:0/982837735 learned_addr learned my addr 172.21.15.42:0/982837735 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:06:03.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.905+0000 7f31affff640 10 monclient(hunting): get_auth_request con 0x7f31b00a3bd0 auth_method 0 2024-09-16T04:06:03.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.905+0000 7f31affff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:06:03.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.905+0000 7f31affff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:03.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.905+0000 7f31affff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:03.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.905+0000 7f31aeffd640 10 monclient(hunting): get_auth_request con 0x7f31b00a37f0 auth_method 0 2024-09-16T04:06:03.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.905+0000 7f31aeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:06:03.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.905+0000 7f31aeffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:03.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.905+0000 7f31aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:03.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.905+0000 7f31affff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:06:03.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.905+0000 7f31affff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:06:03.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.905+0000 7f31affff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:06:03.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.905+0000 7f31aeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:06:03.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.905+0000 7f31aeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:06:03.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.905+0000 7f31aeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:06:03.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.905+0000 7f31af7fe640 1 --2- 172.21.15.42:0/982837735 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f31b006db90 0x7f31b01309b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:06:03.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.905+0000 7f31af7fe640 10 monclient(hunting): get_auth_request con 0x7f31b006db90 auth_method 0 2024-09-16T04:06:03.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.905+0000 7f31af7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:06:03.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.905+0000 7f31af7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:03.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.905+0000 7f31af7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:03.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.905+0000 7f31affff640 10 monclient(hunting): handle_auth_done global_id 984426 payload 824 2024-09-16T04:06:03.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.905+0000 7f31affff640 1 -- 172.21.15.42:0/982837735 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f31b00a37f0 msgr2=0x7f31b00a13c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:03.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.905+0000 7f31affff640 1 --2- 172.21.15.42:0/982837735 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f31b00a37f0 0x7f31b00a13c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:06:03.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.905+0000 7f31affff640 1 -- 172.21.15.42:0/982837735 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f31b006db90 msgr2=0x7f31b01309b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:03.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.905+0000 7f31affff640 1 --2- 172.21.15.42:0/982837735 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f31b006db90 0x7f31b01309b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:06:03.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.905+0000 7f31affff640 10 monclient: _finish_hunting 0 2024-09-16T04:06:03.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.905+0000 7f31affff640 1 monclient: found mon.noname-a 2024-09-16T04:06:03.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.905+0000 7f31affff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:06:03.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.905+0000 7f31affff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:06:03.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.905+0000 7f31affff640 1 -- 172.21.15.42:0/982837735 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31b0138170 con 0x7f31b00a3bd0 2024-09-16T04:06:03.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.905+0000 7f31affff640 1 --2- 172.21.15.42:0/982837735 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f31b00a3bd0 0x7f31b00a1900 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f31a800be20 tx=0x7f31a800bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:06:03.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.905+0000 7f3197fff640 10 client.?.objecter ms_handle_connect 0x7f31b00a3bd0 2024-09-16T04:06:03.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.905+0000 7f3197fff640 10 client.?.objecter resend_mon_ops 2024-09-16T04:06:03.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.905+0000 7f3197fff640 1 -- 172.21.15.42:0/982837735 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31a8019070 con 0x7f31b00a3bd0 2024-09-16T04:06:03.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.905+0000 7f3197fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:03.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.905+0000 7f3197fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:06:03.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.905+0000 7f3197fff640 10 monclient: dump: 2024-09-16T04:06:03.922 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:06:03.923 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:06:03.923 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:06:03.923 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:06:03.923 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:03.923 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:06:03.923 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:06:03.923 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:06:03.923 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:06:03.923 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:06:03.923 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.905+0000 7f3197fff640 10 monclient: _finish_auth 0 2024-09-16T04:06:03.923 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.905+0000 7f3197fff640 10 monclient: _check_auth_tickets 2024-09-16T04:06:03.923 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.905+0000 7f3197fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:06:03.923 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.905+0000 7f31b602b640 5 monclient: authenticate success, global_id 984426 2024-09-16T04:06:03.923 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.905+0000 7f3197fff640 1 -- 172.21.15.42:0/982837735 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f31a8014440 con 0x7f31b00a3bd0 2024-09-16T04:06:03.924 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.905+0000 7f3197fff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:06:03.924 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.905+0000 7f31b602b640 10 monclient: _renew_subs 2024-09-16T04:06:03.924 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.905+0000 7f3197fff640 1 -- 172.21.15.42:0/982837735 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31a8014830 con 0x7f31b00a3bd0 2024-09-16T04:06:03.924 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.905+0000 7f31b602b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:06:03.924 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.905+0000 7f31b602b640 1 -- 172.21.15.42:0/982837735 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31b0139620 con 0x7f31b00a3bd0 2024-09-16T04:06:03.924 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.905+0000 7f3197fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:03.924 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.905+0000 7f3197fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:06:03.924 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.905+0000 7f3197fff640 10 monclient: dump: 2024-09-16T04:06:03.924 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:06:03.924 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:06:03.924 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:06:03.924 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:06:03.925 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:03.925 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:06:03.925 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:06:03.925 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:06:03.925 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:06:03.925 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:06:03.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.905+0000 7f31ad7fa640 4 set_mon_vals no callback set 2024-09-16T04:06:03.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.905+0000 7f31b602b640 10 client.984426.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:06:03.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.905+0000 7f31b602b640 10 monclient: _renew_subs 2024-09-16T04:06:03.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.905+0000 7f31b602b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:06:03.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.905+0000 7f31b602b640 1 -- 172.21.15.42:0/982837735 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31b01330c0 con 0x7f31b00a3bd0 2024-09-16T04:06:03.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.905+0000 7f31b602b640 1 librados: init done 2024-09-16T04:06:03.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.905+0000 7f31b602b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:06:03.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.905+0000 7f31b602b640 1 --2- 172.21.15.42:0/982837735 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f31b0130ef0 0x7f31b00a1e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:06:03.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.905+0000 7f31b602b640 10 monclient(hunting): start opening mon connection 2024-09-16T04:06:03.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.905+0000 7f31b602b640 1 -- 172.21.15.42:0/982837735 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f31b01340c0 con 0x7f31b0130ef0 2024-09-16T04:06:03.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.905+0000 7f3197fff640 1 -- 172.21.15.42:0/982837735 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f31a801c070 con 0x7f31b00a3bd0 2024-09-16T04:06:03.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.905+0000 7f3197fff640 1 --2- 172.21.15.42:0/982837735 >> v2:172.21.15.154:6812/254845342 conn(0x7f319c037810 0x7f319c039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:06:03.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.905+0000 7f3197fff640 1 -- 172.21.15.42:0/982837735 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(706..706 src has 451..706) ==== 5040+0+0 (secure 0 0 0) 0x7f31a8048b60 con 0x7f31b00a3bd0 2024-09-16T04:06:03.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.905+0000 7f3197fff640 10 client.984426.objecter ms_dispatch 0x7f31b0000ba0 osd_map(706..706 src has 451..706) 2024-09-16T04:06:03.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.905+0000 7f3197fff640 3 client.984426.objecter handle_osd_map got epochs [706,706] > 0 2024-09-16T04:06:03.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.905+0000 7f3197fff640 3 client.984426.objecter handle_osd_map decoding full epoch 706 2024-09-16T04:06:03.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.905+0000 7f31af7fe640 1 --2- 172.21.15.42:0/982837735 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f31b0130ef0 0x7f31b00a1e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:06:03.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.905+0000 7f31af7fe640 10 monclient: get_auth_request con 0x7f31b0130ef0 auth_method 0 2024-09-16T04:06:03.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.905+0000 7f3197fff640 20 client.984426.objecter dump_active .. 0 homeless 2024-09-16T04:06:03.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.905+0000 7f31af7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:06:03.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.905+0000 7f31af7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:03.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.905+0000 7f31af7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:03.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.905+0000 7f31af7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:06:03.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.905+0000 7f31af7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:06:03.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.905+0000 7f31af7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:06:03.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.905+0000 7f31aeffd640 1 --2- 172.21.15.42:0/982837735 >> v2:172.21.15.154:6812/254845342 conn(0x7f319c037810 0x7f319c039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:06:03.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.905+0000 7f31aeffd640 10 monclient: get_auth_request con 0x7f319c037810 auth_method 0 2024-09-16T04:06:03.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.905+0000 7f31af7fe640 10 monclient(hunting): handle_auth_done global_id 904996 payload 824 2024-09-16T04:06:03.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.905+0000 7f31af7fe640 1 -- 172.21.15.42:0/982837735 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f31b0137ef0 con 0x7f31b0130ef0 2024-09-16T04:06:03.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.905+0000 7f31af7fe640 1 --2- 172.21.15.42:0/982837735 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f31b0130ef0 0x7f31b00a1e40 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f31b006e4f0 tx=0x7f31a001aba0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:06:03.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.905+0000 7f3197fff640 10 client.984426.objecter ms_handle_connect 0x7f31b0130ef0 2024-09-16T04:06:03.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.905+0000 7f3197fff640 10 client.984426.objecter resend_mon_ops 2024-09-16T04:06:03.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.905+0000 7f3197fff640 10 client.984426.objecter ms_handle_connect 0x7f319c037810 2024-09-16T04:06:03.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.905+0000 7f31aeffd640 1 --2- 172.21.15.42:0/982837735 >> v2:172.21.15.154:6812/254845342 conn(0x7f319c037810 0x7f319c039cd0 secure :-1 s=READY pgs=3865 cs=0 l=1 rev1=1 crypto rx=0x7f31b00a2c60 tx=0x7f31a4009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:06:03.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.905+0000 7f3197fff640 1 -- 172.21.15.42:0/982837735 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31a00bb070 con 0x7f31b0130ef0 2024-09-16T04:06:03.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.905+0000 7f3197fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:03.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.905+0000 7f3197fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:06:03.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.905+0000 7f3197fff640 10 monclient: dump: 2024-09-16T04:06:03.928 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:06:03.928 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:06:03.928 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:06:03.928 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:06:03.928 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:03.929 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:06:03.929 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:06:03.929 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:06:03.929 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:06:03.929 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:06:03.929 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.909+0000 7f3197fff640 1 -- 172.21.15.42:0/982837735 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f31a00b1040 con 0x7f31b0130ef0 2024-09-16T04:06:03.929 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.909+0000 7f3197fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:06:03.929 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.909+0000 7f3197fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:06:03.929 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.909+0000 7f3197fff640 1 -- 172.21.15.42:0/982837735 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f31b0130ef0 msgr2=0x7f31b00a1e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:03.929 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.909+0000 7f3197fff640 1 --2- 172.21.15.42:0/982837735 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f31b0130ef0 0x7f31b00a1e40 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f31b006e4f0 tx=0x7f31a001aba0 comp rx=0 tx=0).stop 2024-09-16T04:06:03.929 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.909+0000 7f3197fff640 1 -- 172.21.15.42:0/982837735 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f31b0130ef0 msgr2=0x7f31b00a1e40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:06:03.929 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.909+0000 7f3197fff640 1 --2- 172.21.15.42:0/982837735 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f31b0130ef0 0x7f31b00a1e40 secure :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f31b006e4f0 tx=0x7f31a001aba0 comp rx=0 tx=0).stop 2024-09-16T04:06:03.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.913+0000 7f31b602b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:06:03.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.913+0000 7f31b602b640 1 --2- 172.21.15.42:0/982837735 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f31b00a2380 0x7f31b009c710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:06:03.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.913+0000 7f31b602b640 10 monclient(hunting): start opening mon connection 2024-09-16T04:06:03.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.913+0000 7f31b602b640 1 -- 172.21.15.42:0/982837735 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f31b00a2790 con 0x7f31b00a2380 2024-09-16T04:06:03.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.913+0000 7f31af7fe640 1 --2- 172.21.15.42:0/982837735 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f31b00a2380 0x7f31b009c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:06:03.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.913+0000 7f31af7fe640 10 monclient: get_auth_request con 0x7f31b00a2380 auth_method 0 2024-09-16T04:06:03.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.913+0000 7f31af7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:06:03.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.913+0000 7f31af7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:03.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.913+0000 7f31af7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:03.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.913+0000 7f31af7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:06:03.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.913+0000 7f31af7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:06:03.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.913+0000 7f31af7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:06:03.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.913+0000 7f31af7fe640 10 monclient(hunting): handle_auth_done global_id 904999 payload 824 2024-09-16T04:06:03.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.913+0000 7f31af7fe640 1 -- 172.21.15.42:0/982837735 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f31b01393a0 con 0x7f31b00a2380 2024-09-16T04:06:03.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.913+0000 7f3197fff640 10 client.984426.objecter ms_handle_connect 0x7f31b00a2380 2024-09-16T04:06:03.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.913+0000 7f31af7fe640 1 --2- 172.21.15.42:0/982837735 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f31b00a2380 0x7f31b009c710 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f31a00097c0 tx=0x7f31a0019100 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:06:03.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.913+0000 7f3197fff640 10 client.984426.objecter resend_mon_ops 2024-09-16T04:06:03.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.913+0000 7f3197fff640 1 -- 172.21.15.42:0/982837735 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31a00bb050 con 0x7f31b00a2380 2024-09-16T04:06:03.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.913+0000 7f3197fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:03.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.913+0000 7f3197fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:06:03.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.913+0000 7f3197fff640 10 monclient: dump: 2024-09-16T04:06:03.932 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:06:03.932 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:06:03.932 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:06:03.932 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:06:03.932 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:03.933 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:06:03.933 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:06:03.933 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:06:03.933 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:06:03.933 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:06:03.933 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.917+0000 7f3197fff640 1 -- 172.21.15.42:0/982837735 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4880 (secure 0 0 0) 0x7f31a00bc040 con 0x7f31b00a2380 2024-09-16T04:06:03.933 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.917+0000 7f3197fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:06:03.933 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.917+0000 7f3197fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:06:03.933 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.917+0000 7f3197fff640 1 -- 172.21.15.42:0/982837735 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f31b00a2380 msgr2=0x7f31b009c710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:03.933 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.917+0000 7f3197fff640 1 --2- 172.21.15.42:0/982837735 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f31b00a2380 0x7f31b009c710 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f31a00097c0 tx=0x7f31a0019100 comp rx=0 tx=0).stop 2024-09-16T04:06:03.933 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.917+0000 7f3197fff640 1 -- 172.21.15.42:0/982837735 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f31b00a2380 msgr2=0x7f31b009c710 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:06:03.933 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.917+0000 7f3197fff640 1 --2- 172.21.15.42:0/982837735 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f31b00a2380 0x7f31b009c710 secure :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f31a00097c0 tx=0x7f31a0019100 comp rx=0 tx=0).stop 2024-09-16T04:06:03.934 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T04:06:03.934 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:06:03.934 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:06:03.934 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T04:06:03.934 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 222, 2024-09-16T04:06:03.934 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 492095, 2024-09-16T04:06:03.934 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T04:06:03.934 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T04:06:03.934 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T04:06:03.934 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T04:06:03.934 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:06:03.935 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 5, 2024-09-16T04:06:03.935 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:06:03.935 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T04:06:03.935 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T04:06:03.935 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:06:03.935 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:06:03.935 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:06:03.935 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:06:03.935 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:06:03.935 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:06:03.935 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:06:03.935 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:06:03.935 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:06:03.936 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:06:03.936 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:06:03.936 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:06:03.936 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:06:03.936 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T04:06:03.936 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:06:03.936 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:06:03.936 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:06:03.936 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:06:03.936 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:06:03.936 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:06:03.936 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:06:03.936 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:06:03.936 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:06:03.937 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:06:03.937 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:06:03.937 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:06:03.937 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:06:03.937 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T04:06:03.937 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T04:06:03.937 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T04:06:03.937 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T04:06:03.937 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T04:06:03.937 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T04:06:03.937 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:06:03.937 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:06:03.937 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T04:06:03.937 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T04:06:03.938 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T04:06:03.938 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T04:06:03.938 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T04:06:03.938 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T04:06:03.938 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T04:06:03.938 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:06:03.938 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T04:06:03.938 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:06:03.938 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:06:03.938 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:06:03.938 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:06:03.938 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:06:03.938 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:06:03.938 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:06:03.939 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:06:03.939 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:06:03.939 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:06:03.939 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:06:03.939 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:06:03.939 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T04:06:03.939 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:06:03.939 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T04:06:03.939 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:06:03.939 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:06:03.939 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:06:03.939 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:06:03.939 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:06:03.939 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:06:03.940 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:06:03.940 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T04:06:03.940 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:06:03.940 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:06:03.940 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:06:03.940 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:06:03.940 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T04:06:03.940 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:06:03.940 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:06:03.940 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:06:03.941 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:06:03.941 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T04:06:03.941 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T04:06:03.941 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:06:03.941 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:06:03.941 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:06:03.941 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:06:03.941 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:06:03.941 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:06:03.941 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:06:03.942 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:06:03.942 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:06:03.942 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:06:03.942 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:06:03.942 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T04:06:03.942 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:06:03.942 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:06:03.942 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:06:03.942 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:06:03.942 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T04:06:03.942 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:06:03.942 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:06:03.942 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:06:03.942 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:06:03.943 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T04:06:03.943 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T04:06:03.943 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:06:03.943 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:06:03.943 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:06:03.943 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:06:03.943 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:06:03.943 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:06:03.943 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:06:03.943 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:06:03.943 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:06:03.943 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:06:03.943 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:06:03.943 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T04:06:03.944 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:06:03.944 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:06:03.944 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:06:03.944 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:06:03.944 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T04:06:03.944 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:06:03.944 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:06:03.944 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:06:03.944 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:06:03.944 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T04:06:03.944 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T04:06:03.944 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:06:03.944 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:06:03.944 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:06:03.945 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:06:03.945 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:06:03.945 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:06:03.945 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T04:06:03.945 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T04:06:03.945 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:06:03.945 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:06:03.945 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:06:03.945 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:06:03.945 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:06:03.945 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:06:03.945 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T04:06:03.945 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:06:03.946 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:06:03.946 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:06:03.946 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 3 2024-09-16T04:06:03.946 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:06:03.946 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:06:03.946 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T04:06:03.946 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:06:03.946 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:06:03.946 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:06:03.946 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 5 2024-09-16T04:06:03.946 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:06:03.946 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:06:03.946 INFO:teuthology.orchestra.run.smithi042.stdout: "mgr": [ 2024-09-16T04:06:03.946 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:06:03.946 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:06:03.947 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:06:03.947 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:06:03.947 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:06:03.947 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:06:03.947 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:06:03.947 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T04:06:03.947 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T04:06:03.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.917+0000 7f31b602b640 10 librados: watch_flush enter 2024-09-16T04:06:03.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.917+0000 7f31b602b640 10 librados: watch_flush exit 2024-09-16T04:06:03.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.917+0000 7f31b602b640 20 client.984426.objecter shutdown clearing up homeless session... 2024-09-16T04:06:03.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.917+0000 7f31b602b640 10 client.984426.objecter successfully canceled tick 2024-09-16T04:06:03.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.917+0000 7f31b602b640 1 -- 172.21.15.42:0/982837735 >> v2:172.21.15.154:6812/254845342 conn(0x7f319c037810 msgr2=0x7f319c039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:03.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.917+0000 7f31b602b640 1 --2- 172.21.15.42:0/982837735 >> v2:172.21.15.154:6812/254845342 conn(0x7f319c037810 0x7f319c039cd0 secure :-1 s=READY pgs=3865 cs=0 l=1 rev1=1 crypto rx=0x7f31b00a2c60 tx=0x7f31a4009040 comp rx=0 tx=0).stop 2024-09-16T04:06:03.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.917+0000 7f31b602b640 10 monclient: shutdown 2024-09-16T04:06:03.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.917+0000 7f31b602b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:06:03.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.917+0000 7f31b602b640 1 -- 172.21.15.42:0/982837735 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f31b00a3bd0 msgr2=0x7f31b00a1900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:03.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.917+0000 7f31b602b640 1 --2- 172.21.15.42:0/982837735 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f31b00a3bd0 0x7f31b00a1900 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f31a800be20 tx=0x7f31a800bf10 comp rx=0 tx=0).stop 2024-09-16T04:06:03.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.917+0000 7f31affff640 1 -- 172.21.15.42:0/982837735 reap_dead start 2024-09-16T04:06:03.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.917+0000 7f31affff640 1 -- 172.21.15.42:0/982837735 reap_dead start 2024-09-16T04:06:03.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.917+0000 7f31b602b640 1 -- 172.21.15.42:0/982837735 shutdown_connections 2024-09-16T04:06:03.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.917+0000 7f31b602b640 1 -- 172.21.15.42:0/982837735 >> 172.21.15.42:0/982837735 conn(0x7f31b00a9fb0 msgr2=0x7f31b01369f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:06:03.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.917+0000 7f31b602b640 1 -- 172.21.15.42:0/982837735 shutdown_connections 2024-09-16T04:06:03.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.917+0000 7f31b602b640 1 -- 172.21.15.42:0/982837735 wait complete. 2024-09-16T04:06:03.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:03.917+0000 7f31b602b640 1 librados: shutdown 2024-09-16T04:06:03.950 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:06:03.950 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-09-16T04:06:03.950 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['a'] 2024-09-16T04:06:03.950 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-09-16T04:06:03.951 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-16T04:06:03.951 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-16T04:06:03.951 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T04:06:03.962 INFO:tasks.ceph.mon.c:Stopped 2024-09-16T04:06:03.963 INFO:tasks.mon_thrash:Sending STOP to mon a 2024-09-16T04:06:03.963 INFO:tasks.ceph.mon.a:Sent signal 19 2024-09-16T04:06:03.963 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-16T04:06:18.964 INFO:tasks.mon_thrash:Sending CONT to mon a 2024-09-16T04:06:18.965 INFO:tasks.ceph.mon.a:Sent signal 18 2024-09-16T04:06:18.965 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T04:06:18.966 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:06:19.098 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:19.093+0000 7f0ef02a2640 10 monclient: get_monmap_and_config 2024-09-16T04:06:19.099 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:19.093+0000 7f0ef02a2640 10 monclient: build_initial_monmap 2024-09-16T04:06:19.099 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:19.093+0000 7f0ef02a2640 10 monclient: monmap: 2024-09-16T04:06:19.099 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:06:19.099 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:06:19.099 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:06:19.097066+0000 2024-09-16T04:06:19.099 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:06:19.097066+0000 2024-09-16T04:06:19.099 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:06:19.099 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:06:19.099 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:06:19.099 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:06:19.099 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:06:19.099 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:06:19.099 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:06:19.099 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:06:19.101 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:19.093+0000 7f0ef02a2640 1 Processor -- start 2024-09-16T04:06:19.101 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:19.093+0000 7f0ef02a2640 1 -- start start 2024-09-16T04:06:19.101 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:19.093+0000 7f0ef02a2640 10 monclient: init 2024-09-16T04:06:19.101 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:19.093+0000 7f0ef02a2640 10 monclient: _reopen_session rank -1 2024-09-16T04:06:19.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:19.093+0000 7f0ef02a2640 10 monclient: _add_conns ranks=[3,2,0,4,1] 2024-09-16T04:06:19.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:19.093+0000 7f0ef02a2640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f0ee806db90 0x7f0ee81340c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:06:19.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:19.093+0000 7f0ef02a2640 10 monclient(hunting): picked mon.noname-a con 0x7f0ee806db90 addr v2:172.21.15.42:3300/0 2024-09-16T04:06:19.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:19.093+0000 7f0ef02a2640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0ee80a37f0 0x7f0ee80a3c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:06:19.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:19.093+0000 7f0ef02a2640 10 monclient(hunting): picked mon.noname-b con 0x7f0ee80a37f0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:06:19.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:19.093+0000 7f0ef02a2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0ee8136e10 addr v1:172.21.15.42:6789/0 2024-09-16T04:06:19.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:19.093+0000 7f0ef02a2640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ee806ece0 con 0x7f0ee8136e10 2024-09-16T04:06:19.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:19.093+0000 7f0ef02a2640 10 monclient(hunting): start opening mon connection 2024-09-16T04:06:19.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:19.093+0000 7f0ef02a2640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f0ee806df70 con 0x7f0ee806db90 2024-09-16T04:06:19.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:19.093+0000 7f0ef02a2640 10 monclient(hunting): start opening mon connection 2024-09-16T04:06:19.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:19.093+0000 7f0ef02a2640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f0ee809bf80 con 0x7f0ee80a37f0 2024-09-16T04:06:19.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:19.093+0000 7f0ef02a2640 10 monclient(hunting): _renew_subs 2024-09-16T04:06:19.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:19.093+0000 7f0ef02a2640 10 monclient(hunting): authenticate will time out at 2426.066859s 2024-09-16T04:06:19.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:19.093+0000 7f0eed83d640 1 -- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0ee80a37f0 msgr2=0x7f0ee80a3c00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T04:06:19.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:19.093+0000 7f0eed83d640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0ee80a37f0 0x7f0ee80a3c00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:06:19.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:19.093+0000 7f0eee03e640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f0ee806db90 0x7f0ee81340c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:06:19.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:19.093+0000 7f0eee83f640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f0ee8136e10 0x7f0ee8137240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:54722/0 (socket says 172.21.15.42:54722) 2024-09-16T04:06:19.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:19.093+0000 7f0eee83f640 1 -- 172.21.15.42:0/1215379741 learned_addr learned my addr 172.21.15.42:0/1215379741 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T04:06:19.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:19.093+0000 7f0eee03e640 10 monclient(hunting): get_auth_request con 0x7f0ee806db90 auth_method 0 2024-09-16T04:06:19.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:19.093+0000 7f0eee03e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:06:19.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:19.093+0000 7f0eee03e640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:19.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:19.093+0000 7f0eee03e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:19.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:19.093+0000 7f0eee03e640 1 -- 172.21.15.42:0/1215379741 >> v2:172.21.15.42:3300/0 conn(0x7f0ee806db90 msgr2=0x7f0ee81340c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:06:19.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:19.093+0000 7f0eee03e640 1 -- 172.21.15.42:0/1215379741 >> v2:172.21.15.42:3300/0 conn(0x7f0ee806db90 msgr2=0x7f0ee81340c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:06:19.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:19.093+0000 7f0eee03e640 1 --2- 172.21.15.42:0/1215379741 >> v2:172.21.15.42:3300/0 conn(0x7f0ee806db90 0x7f0ee81340c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:06:19.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:19.093+0000 7f0eee03e640 1 --2- 172.21.15.42:0/1215379741 >> v2:172.21.15.42:3300/0 conn(0x7f0ee806db90 0x7f0ee81340c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:06:19.301 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:19.297+0000 7f0eed83d640 1 -- 172.21.15.42:0/1215379741 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0ee80a37f0 msgr2=0x7f0ee80a3c00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T04:06:19.301 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:19.297+0000 7f0eed83d640 1 --2- 172.21.15.42:0/1215379741 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0ee80a37f0 0x7f0ee80a3c00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T04:06:19.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:19.297+0000 7f0eee03e640 1 --2- 172.21.15.42:0/1215379741 >> v2:172.21.15.42:3300/0 conn(0x7f0ee806db90 0x7f0ee81340c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:06:19.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:19.297+0000 7f0eee03e640 10 monclient(hunting): get_auth_request con 0x7f0ee806db90 auth_method 0 2024-09-16T04:06:19.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:19.297+0000 7f0eee03e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:06:19.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:19.297+0000 7f0eee03e640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:19.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:19.297+0000 7f0eee03e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:06:19.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:19.297+0000 7f0eee03e640 1 -- 172.21.15.42:0/1215379741 >> v2:172.21.15.42:3300/0 conn(0x7f0ee806db90 msgr2=0x7f0ee81340c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:06:19.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:19.297+0000 7f0eee03e640 1 -- 172.21.15.42:0/1215379741 >> v2:172.21.15.42:3300/0 conn(0x7f0ee806db90 msgr2=0x7f0ee81340c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:06:19.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:19.297+0000 7f0eee03e640 1 --2- 172.21.15.42:0/1215379741 >> v2:172.21.15.42:3300/0 conn(0x7f0ee806db90 0x7f0ee81340c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:06:19.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:19.297+0000 7f0eee03e640 1 --2- 172.21.15.42:0/1215379741 >> v2:172.21.15.42:3300/0 conn(0x7f0ee806db90 0x7f0ee81340c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T04:06:19.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:19.701+0000 7f0eed83d640 1 -- 172.21.15.42:0/1215379741 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0ee80a37f0 msgr2=0x7f0ee80a3c00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T04:06:19.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:19.701+0000 7f0eed83d640 1 --2- 172.21.15.42:0/1215379741 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0ee80a37f0 0x7f0ee80a3c00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T04:06:19.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:19.701+0000 7f0eee03e640 1 --2- 172.21.15.42:0/1215379741 >> v2:172.21.15.42:3300/0 conn(0x7f0ee806db90 0x7f0ee81340c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:06:19.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:19.701+0000 7f0eee03e640 10 monclient(hunting): get_auth_request con 0x7f0ee806db90 auth_method 0 2024-09-16T04:06:19.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:19.701+0000 7f0eee03e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:06:19.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:19.701+0000 7f0eee03e640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:19.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:19.701+0000 7f0eee03e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:06:19.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:19.701+0000 7f0eee03e640 1 -- 172.21.15.42:0/1215379741 >> v2:172.21.15.42:3300/0 conn(0x7f0ee806db90 msgr2=0x7f0ee81340c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:06:19.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:19.701+0000 7f0eee03e640 1 -- 172.21.15.42:0/1215379741 >> v2:172.21.15.42:3300/0 conn(0x7f0ee806db90 msgr2=0x7f0ee81340c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:06:19.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:19.701+0000 7f0eee03e640 1 --2- 172.21.15.42:0/1215379741 >> v2:172.21.15.42:3300/0 conn(0x7f0ee806db90 0x7f0ee81340c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:06:19.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:19.701+0000 7f0eee03e640 1 --2- 172.21.15.42:0/1215379741 >> v2:172.21.15.42:3300/0 conn(0x7f0ee806db90 0x7f0ee81340c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T04:06:20.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:20.505+0000 7f0eed83d640 1 -- 172.21.15.42:0/1215379741 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0ee80a37f0 msgr2=0x7f0ee80a3c00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T04:06:20.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:20.505+0000 7f0eed83d640 1 --2- 172.21.15.42:0/1215379741 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0ee80a37f0 0x7f0ee80a3c00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T04:06:20.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:20.505+0000 7f0eee03e640 1 --2- 172.21.15.42:0/1215379741 >> v2:172.21.15.42:3300/0 conn(0x7f0ee806db90 0x7f0ee81340c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:06:20.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:20.505+0000 7f0eee03e640 10 monclient(hunting): get_auth_request con 0x7f0ee806db90 auth_method 0 2024-09-16T04:06:20.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:20.505+0000 7f0eee03e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:06:20.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:20.505+0000 7f0eee03e640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:20.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:20.505+0000 7f0eee03e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:06:20.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:20.505+0000 7f0eee03e640 1 -- 172.21.15.42:0/1215379741 >> v2:172.21.15.42:3300/0 conn(0x7f0ee806db90 msgr2=0x7f0ee81340c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:06:20.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:20.505+0000 7f0eee03e640 1 -- 172.21.15.42:0/1215379741 >> v2:172.21.15.42:3300/0 conn(0x7f0ee806db90 msgr2=0x7f0ee81340c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:06:20.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:20.505+0000 7f0eee03e640 1 --2- 172.21.15.42:0/1215379741 >> v2:172.21.15.42:3300/0 conn(0x7f0ee806db90 0x7f0ee81340c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:06:20.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:20.505+0000 7f0eee03e640 1 --2- 172.21.15.42:0/1215379741 >> v2:172.21.15.42:3300/0 conn(0x7f0ee806db90 0x7f0ee81340c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T04:06:22.101 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:22.093+0000 7f0edffff640 10 monclient(hunting): tick 2024-09-16T04:06:22.101 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:22.093+0000 7f0edffff640 10 monclient(hunting): _check_auth_tickets 2024-09-16T04:06:22.101 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:22.093+0000 7f0edffff640 1 monclient(hunting): continuing hunt 2024-09-16T04:06:22.101 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:22.093+0000 7f0edffff640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T04:06:22.101 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:22.093+0000 7f0edffff640 1 -- 172.21.15.42:0/1215379741 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0ee80a37f0 msgr2=0x7f0ee80a3c00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:06:22.101 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:22.093+0000 7f0edffff640 1 --2- 172.21.15.42:0/1215379741 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0ee80a37f0 0x7f0ee80a3c00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:06:22.101 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:22.093+0000 7f0edffff640 1 -- 172.21.15.42:0/1215379741 >> v2:172.21.15.42:3300/0 conn(0x7f0ee806db90 msgr2=0x7f0ee81340c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:06:22.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:22.093+0000 7f0edffff640 1 --2- 172.21.15.42:0/1215379741 >> v2:172.21.15.42:3300/0 conn(0x7f0ee806db90 0x7f0ee81340c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:06:22.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:22.093+0000 7f0edffff640 1 -- 172.21.15.42:0/1215379741 >> v1:172.21.15.42:6789/0 conn(0x7f0ee8136e10 legacy=0x7f0ee8137240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:22.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:22.093+0000 7f0edffff640 10 monclient: _add_conns ranks=[1,4] 2024-09-16T04:06:22.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:22.093+0000 7f0edffff640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0ecc0036f0 addr v1:172.21.15.154:6789/0 2024-09-16T04:06:22.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:22.093+0000 7f0edffff640 1 --2- 172.21.15.42:0/1215379741 >> v2:172.21.15.154:3300/0 conn(0x7f0ecc007420 0x7f0ecc0098e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:06:22.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:22.093+0000 7f0edffff640 10 monclient(hunting): picked mon.noname-c con 0x7f0ecc007420 addr v2:172.21.15.154:3300/0 2024-09-16T04:06:22.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:22.093+0000 7f0edffff640 1 -- 172.21.15.42:0/1215379741 --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ee8134780 con 0x7f0ecc0036f0 2024-09-16T04:06:22.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:22.093+0000 7f0edffff640 10 monclient(hunting): start opening mon connection 2024-09-16T04:06:22.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:22.093+0000 7f0edffff640 1 -- 172.21.15.42:0/1215379741 --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f0ee806df70 con 0x7f0ecc007420 2024-09-16T04:06:22.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:22.093+0000 7f0edffff640 10 monclient(hunting): _renew_subs 2024-09-16T04:06:22.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:22.093+0000 7f0eed83d640 1 --2- 172.21.15.42:0/1215379741 >> v2:172.21.15.154:3300/0 conn(0x7f0ecc007420 0x7f0ecc0098e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:06:22.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:22.093+0000 7f0eed83d640 10 monclient(hunting): get_auth_request con 0x7f0ecc007420 auth_method 0 2024-09-16T04:06:22.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:22.093+0000 7f0eed83d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:06:22.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:22.093+0000 7f0eed83d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:22.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:22.093+0000 7f0eed83d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:22.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:22.093+0000 7f0eed83d640 1 -- 172.21.15.42:0/1215379741 >> v2:172.21.15.154:3300/0 conn(0x7f0ecc007420 msgr2=0x7f0ecc0098e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:06:22.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:22.093+0000 7f0eed83d640 1 -- 172.21.15.42:0/1215379741 >> v2:172.21.15.154:3300/0 conn(0x7f0ecc007420 msgr2=0x7f0ecc0098e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:06:22.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:22.093+0000 7f0eed83d640 1 --2- 172.21.15.42:0/1215379741 >> v2:172.21.15.154:3300/0 conn(0x7f0ecc007420 0x7f0ecc0098e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:06:22.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:22.093+0000 7f0eed83d640 1 --2- 172.21.15.42:0/1215379741 >> v2:172.21.15.154:3300/0 conn(0x7f0ecc007420 0x7f0ecc0098e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:06:22.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:22.297+0000 7f0eed83d640 1 --2- 172.21.15.42:0/1215379741 >> v2:172.21.15.154:3300/0 conn(0x7f0ecc007420 0x7f0ecc0098e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:06:22.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:22.297+0000 7f0eed83d640 10 monclient(hunting): get_auth_request con 0x7f0ecc007420 auth_method 0 2024-09-16T04:06:22.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:22.297+0000 7f0eed83d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:06:22.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:22.297+0000 7f0eed83d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:22.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:22.297+0000 7f0eed83d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:06:22.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:22.297+0000 7f0eed83d640 1 -- 172.21.15.42:0/1215379741 >> v2:172.21.15.154:3300/0 conn(0x7f0ecc007420 msgr2=0x7f0ecc0098e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:06:22.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:22.297+0000 7f0eed83d640 1 -- 172.21.15.42:0/1215379741 >> v2:172.21.15.154:3300/0 conn(0x7f0ecc007420 msgr2=0x7f0ecc0098e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:06:22.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:22.297+0000 7f0eed83d640 1 --2- 172.21.15.42:0/1215379741 >> v2:172.21.15.154:3300/0 conn(0x7f0ecc007420 0x7f0ecc0098e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:06:22.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:22.297+0000 7f0eed83d640 1 --2- 172.21.15.42:0/1215379741 >> v2:172.21.15.154:3300/0 conn(0x7f0ecc007420 0x7f0ecc0098e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T04:06:22.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:22.701+0000 7f0eed83d640 1 --2- 172.21.15.42:0/1215379741 >> v2:172.21.15.154:3300/0 conn(0x7f0ecc007420 0x7f0ecc0098e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:06:22.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:22.701+0000 7f0eed83d640 10 monclient(hunting): get_auth_request con 0x7f0ecc007420 auth_method 0 2024-09-16T04:06:22.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:22.701+0000 7f0eed83d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:06:22.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:22.701+0000 7f0eed83d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:22.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:22.701+0000 7f0eed83d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:06:22.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:22.701+0000 7f0eed83d640 1 -- 172.21.15.42:0/1215379741 >> v2:172.21.15.154:3300/0 conn(0x7f0ecc007420 msgr2=0x7f0ecc0098e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:06:22.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:22.701+0000 7f0eed83d640 1 -- 172.21.15.42:0/1215379741 >> v2:172.21.15.154:3300/0 conn(0x7f0ecc007420 msgr2=0x7f0ecc0098e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:06:22.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:22.701+0000 7f0eed83d640 1 --2- 172.21.15.42:0/1215379741 >> v2:172.21.15.154:3300/0 conn(0x7f0ecc007420 0x7f0ecc0098e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:06:22.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:22.701+0000 7f0eed83d640 1 --2- 172.21.15.42:0/1215379741 >> v2:172.21.15.154:3300/0 conn(0x7f0ecc007420 0x7f0ecc0098e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T04:06:23.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:23.505+0000 7f0eed83d640 1 --2- 172.21.15.42:0/1215379741 >> v2:172.21.15.154:3300/0 conn(0x7f0ecc007420 0x7f0ecc0098e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:06:23.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:23.505+0000 7f0eed83d640 10 monclient(hunting): get_auth_request con 0x7f0ecc007420 auth_method 0 2024-09-16T04:06:23.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:23.505+0000 7f0eed83d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:06:23.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:23.505+0000 7f0eed83d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:23.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:23.505+0000 7f0eed83d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:06:23.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:23.505+0000 7f0eed83d640 1 -- 172.21.15.42:0/1215379741 >> v2:172.21.15.154:3300/0 conn(0x7f0ecc007420 msgr2=0x7f0ecc0098e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:06:23.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:23.505+0000 7f0eed83d640 1 -- 172.21.15.42:0/1215379741 >> v2:172.21.15.154:3300/0 conn(0x7f0ecc007420 msgr2=0x7f0ecc0098e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:06:23.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:23.505+0000 7f0eed83d640 1 --2- 172.21.15.42:0/1215379741 >> v2:172.21.15.154:3300/0 conn(0x7f0ecc007420 0x7f0ecc0098e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:06:23.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:23.505+0000 7f0eed83d640 1 --2- 172.21.15.42:0/1215379741 >> v2:172.21.15.154:3300/0 conn(0x7f0ecc007420 0x7f0ecc0098e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T04:06:24.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.037+0000 7f0eed03c640 1 -- 172.21.15.42:0/1215379741 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 872888441 0 0) 0x7f0ee8134780 con 0x7f0ecc0036f0 2024-09-16T04:06:24.045 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.037+0000 7f0eed03c640 10 monclient(hunting): _negotiate 2024-09-16T04:06:24.045 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.037+0000 7f0eed03c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:24.045 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.037+0000 7f0eed03c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:24.045 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.037+0000 7f0eed03c640 10 monclient(hunting): my global_id is 905002 2024-09-16T04:06:24.045 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.037+0000 7f0eed03c640 1 -- 172.21.15.42:0/1215379741 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0ed4003530 con 0x7f0ecc0036f0 2024-09-16T04:06:24.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.045+0000 7f0eed03c640 1 -- 172.21.15.42:0/1215379741 <== mon.1 v1:172.21.15.154:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3535876133 0 0) 0x7f0ed4003530 con 0x7f0ecc0036f0 2024-09-16T04:06:24.053 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.045+0000 7f0eed03c640 1 -- 172.21.15.42:0/1215379741 >> v2:172.21.15.154:3300/0 conn(0x7f0ecc007420 msgr2=0x7f0ecc0098e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:06:24.053 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.045+0000 7f0eed03c640 1 --2- 172.21.15.42:0/1215379741 >> v2:172.21.15.154:3300/0 conn(0x7f0ecc007420 0x7f0ecc0098e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:06:24.053 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.045+0000 7f0eed03c640 10 monclient: _finish_hunting 0 2024-09-16T04:06:24.053 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.045+0000 7f0eed03c640 1 monclient: found mon.noname-c-legacy 2024-09-16T04:06:24.053 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.045+0000 7f0eed03c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:06:24.053 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.045+0000 7f0eed03c640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.154:6789/0 2024-09-16T04:06:24.053 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.045+0000 7f0eed03c640 1 -- 172.21.15.42:0/1215379741 --> v1:172.21.15.154:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ecc00b050 con 0x7f0ecc0036f0 2024-09-16T04:06:24.053 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.045+0000 7f0eed03c640 10 monclient: _finish_auth 0 2024-09-16T04:06:24.053 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.045+0000 7f0eed03c640 10 monclient: _check_auth_tickets 2024-09-16T04:06:24.053 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.045+0000 7f0eed03c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:06:24.053 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.045+0000 7f0eed03c640 1 -- 172.21.15.42:0/1215379741 <== mon.1 v1:172.21.15.154:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2177413734 0 0) 0x7f0ee4009a00 con 0x7f0ecc0036f0 2024-09-16T04:06:24.053 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.045+0000 7f0ef02a2640 5 monclient: authenticate success, global_id 905002 2024-09-16T04:06:24.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.045+0000 7f0ef02a2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:06:24.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.045+0000 7f0eed03c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:24.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.045+0000 7f0eed03c640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-16T04:06:24.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.045+0000 7f0eed03c640 10 monclient: dump: 2024-09-16T04:06:24.054 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:06:24.054 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:06:24.054 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:06:24.054 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:06:24.054 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:24.054 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:06:24.054 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:06:24.054 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:06:24.055 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:06:24.055 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:06:24.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.045+0000 7f0eed03c640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] but i'm connected to v1:172.21.15.154:6789/0, reconnecting 2024-09-16T04:06:24.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.045+0000 7f0eed03c640 10 monclient: _reopen_session rank -1 2024-09-16T04:06:24.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.045+0000 7f0eed03c640 1 -- 172.21.15.42:0/1215379741 >> v1:172.21.15.154:6789/0 conn(0x7f0ecc0036f0 legacy=0x7f0ecc005b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:24.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.045+0000 7f0eee83f640 1 -- 172.21.15.42:0/1215379741 reap_dead start 2024-09-16T04:06:24.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.045+0000 7f0eed03c640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:06:24.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.045+0000 7f0eed03c640 1 --2- 172.21.15.42:0/1215379741 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0ed40039c0 0x7f0ed4003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:06:24.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.045+0000 7f0eed03c640 10 monclient(hunting): picked mon.c con 0x7f0ed40039c0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:06:24.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.045+0000 7f0eed03c640 1 --2- 172.21.15.42:0/1215379741 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0ed4004810 0x7f0ed4004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:06:24.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.045+0000 7f0eed03c640 10 monclient(hunting): picked mon.a con 0x7f0ed4004810 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:06:24.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.045+0000 7f0eed03c640 1 --2- 172.21.15.42:0/1215379741 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0ed4005220 0x7f0ed400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:06:24.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.045+0000 7f0eed03c640 10 monclient(hunting): picked mon.b con 0x7f0ed4005220 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:06:24.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.045+0000 7f0eed03c640 10 monclient(hunting): start opening mon connection 2024-09-16T04:06:24.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.045+0000 7f0eed03c640 1 -- 172.21.15.42:0/1215379741 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f0ee806df70 con 0x7f0ed4004810 2024-09-16T04:06:24.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.045+0000 7f0eed03c640 10 monclient(hunting): start opening mon connection 2024-09-16T04:06:24.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.045+0000 7f0eed03c640 1 -- 172.21.15.42:0/1215379741 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f0ed400c150 con 0x7f0ed4005220 2024-09-16T04:06:24.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.045+0000 7f0eed03c640 10 monclient(hunting): start opening mon connection 2024-09-16T04:06:24.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.045+0000 7f0eed03c640 1 -- 172.21.15.42:0/1215379741 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f0ed400c350 con 0x7f0ed40039c0 2024-09-16T04:06:24.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.045+0000 7f0eed03c640 10 monclient(hunting): _renew_subs 2024-09-16T04:06:24.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.045+0000 7f0eed03c640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:06:24.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.045+0000 7f0ef02a2640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-16T04:06:24.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.045+0000 7f0eed83d640 1 --2- 172.21.15.42:0/1215379741 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0ed4004810 0x7f0ed4004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:06:24.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.045+0000 7f0eee03e640 1 -- 172.21.15.42:0/1215379741 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0ed40039c0 msgr2=0x7f0ed4003dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T04:06:24.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.045+0000 7f0eed83d640 10 monclient(hunting): get_auth_request con 0x7f0ed4004810 auth_method 0 2024-09-16T04:06:24.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.045+0000 7f0eed83d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:06:24.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.045+0000 7f0eed83d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:24.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.045+0000 7f0eed83d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:06:24.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.045+0000 7f0eee03e640 1 --2- 172.21.15.42:0/1215379741 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0ed40039c0 0x7f0ed4003dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:06:24.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.045+0000 7f0eed83d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:06:24.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.045+0000 7f0eed83d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:06:24.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.045+0000 7f0eed83d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:06:24.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.045+0000 7f0eee83f640 1 --2- 172.21.15.42:0/1215379741 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0ed4005220 0x7f0ed400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:06:24.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.045+0000 7f0eee83f640 10 monclient(hunting): get_auth_request con 0x7f0ed4005220 auth_method 0 2024-09-16T04:06:24.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.045+0000 7f0eee83f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:06:24.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.045+0000 7f0eee83f640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:24.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.045+0000 7f0eee83f640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:06:24.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.045+0000 7f0eed83d640 10 monclient(hunting): handle_auth_done global_id 905002 payload 293 2024-09-16T04:06:24.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.045+0000 7f0eed83d640 1 -- 172.21.15.42:0/1215379741 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0ed40039c0 msgr2=0x7f0ed4003dc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:06:24.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.045+0000 7f0eed83d640 1 --2- 172.21.15.42:0/1215379741 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0ed40039c0 0x7f0ed4003dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:06:24.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.045+0000 7f0eed83d640 1 -- 172.21.15.42:0/1215379741 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0ed4005220 msgr2=0x7f0ed400b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:06:24.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.045+0000 7f0eed83d640 1 --2- 172.21.15.42:0/1215379741 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0ed4005220 0x7f0ed400b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:06:24.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.045+0000 7f0eed83d640 10 monclient: _finish_hunting 0 2024-09-16T04:06:24.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.045+0000 7f0eed83d640 1 monclient: found mon.a 2024-09-16T04:06:24.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.045+0000 7f0eed83d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:06:24.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.045+0000 7f0eed83d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:06:24.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.045+0000 7f0eed83d640 1 -- 172.21.15.42:0/1215379741 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ed4003530 con 0x7f0ed4004810 2024-09-16T04:06:24.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.045+0000 7f0eed83d640 10 monclient: _finish_auth 0 2024-09-16T04:06:24.059 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.045+0000 7f0eed83d640 10 monclient: _check_auth_tickets 2024-09-16T04:06:24.059 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.045+0000 7f0eed83d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:06:24.060 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.049+0000 7f0eed83d640 1 --2- 172.21.15.42:0/1215379741 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0ed4004810 0x7f0ed4004c90 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f0ed8003990 tx=0x7f0ed801d450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3e146bf4655378f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:06:24.060 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.049+0000 7f0eed03c640 1 -- 172.21.15.42:0/1215379741 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ed80b3070 con 0x7f0ed4004810 2024-09-16T04:06:24.060 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.049+0000 7f0eed03c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:24.060 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.049+0000 7f0eed03c640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:06:24.061 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.049+0000 7f0eed03c640 10 monclient: dump: 2024-09-16T04:06:24.061 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:06:24.061 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:06:24.061 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:06:24.061 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:06:24.061 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:24.061 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:06:24.061 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:06:24.061 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:06:24.061 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:06:24.061 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:06:24.061 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.049+0000 7f0ef02a2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:06:24.061 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.049+0000 7f0eed03c640 1 -- 172.21.15.42:0/1215379741 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ed80ae410 con 0x7f0ed4004810 2024-09-16T04:06:24.061 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.049+0000 7f0eed03c640 10 monclient: handle_config config(0 keys) 2024-09-16T04:06:24.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.049+0000 7f0eed03c640 1 -- 172.21.15.42:0/1215379741 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ed8009a00 con 0x7f0ed4004810 2024-09-16T04:06:24.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.049+0000 7f0ef02a2640 10 monclient: get_monmap_and_config success 2024-09-16T04:06:24.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.049+0000 7f0ef02a2640 4 set_mon_vals no callback set 2024-09-16T04:06:24.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.049+0000 7f0eed03c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:24.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.049+0000 7f0eed03c640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:06:24.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.049+0000 7f0eed03c640 10 monclient: dump: 2024-09-16T04:06:24.062 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:06:24.062 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:06:24.062 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:06:24.062 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:06:24.062 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:24.062 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:06:24.062 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:06:24.062 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:06:24.063 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:06:24.063 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:06:24.063 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.053+0000 7f0ef02a2640 10 monclient: shutdown 2024-09-16T04:06:24.063 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.053+0000 7f0ef02a2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:06:24.063 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.053+0000 7f0ef02a2640 1 -- 172.21.15.42:0/1215379741 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0ed4004810 msgr2=0x7f0ed4004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:24.063 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.053+0000 7f0ef02a2640 1 --2- 172.21.15.42:0/1215379741 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0ed4004810 0x7f0ed4004c90 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f0ed8003990 tx=0x7f0ed801d450 comp rx=0 tx=0).stop 2024-09-16T04:06:24.063 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.053+0000 7f0ef02a2640 1 -- 172.21.15.42:0/1215379741 shutdown_connections 2024-09-16T04:06:24.063 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.053+0000 7f0ef02a2640 1 --2- 172.21.15.42:0/1215379741 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0ed4005220 0x7f0ed400b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:06:24.063 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.053+0000 7f0ef02a2640 1 --2- 172.21.15.42:0/1215379741 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0ed4004810 0x7f0ed4004c90 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:06:24.063 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.053+0000 7f0ef02a2640 1 --2- 172.21.15.42:0/1215379741 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0ed40039c0 0x7f0ed4003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:06:24.063 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.053+0000 7f0ef02a2640 1 -- 172.21.15.42:0/1215379741 >> 172.21.15.42:0/1215379741 conn(0x7f0ee80a9fb0 msgr2=0x7f0ee80aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:06:24.063 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.053+0000 7f0ef02a2640 1 -- 172.21.15.42:0/1215379741 shutdown_connections 2024-09-16T04:06:24.063 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.053+0000 7f0ef02a2640 1 -- 172.21.15.42:0/1215379741 wait complete. 2024-09-16T04:06:24.064 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.053+0000 7f0ef02a2640 10 monclient: build_initial_monmap 2024-09-16T04:06:24.064 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.053+0000 7f0ef02a2640 10 monclient: monmap: 2024-09-16T04:06:24.064 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:06:24.064 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:06:24.064 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:06:24.064 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:06:24.064 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:06:24.064 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:06:24.064 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:06:24.064 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:06:24.064 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:06:24.064 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:06:24.064 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.053+0000 7f0ef02a2640 1 librados: starting msgr at 2024-09-16T04:06:24.064 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.053+0000 7f0ef02a2640 1 librados: starting objecter 2024-09-16T04:06:24.065 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.053+0000 7f0ef02a2640 1 Processor -- start 2024-09-16T04:06:24.065 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.053+0000 7f0ef02a2640 1 -- start start 2024-09-16T04:06:24.065 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.053+0000 7f0ef02a2640 1 librados: setting wanted keys 2024-09-16T04:06:24.065 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.053+0000 7f0ef02a2640 1 librados: calling monclient init 2024-09-16T04:06:24.065 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.053+0000 7f0ef02a2640 10 monclient: init 2024-09-16T04:06:24.070 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.053+0000 7f0ef02a2640 10 monclient: _reopen_session rank -1 2024-09-16T04:06:24.070 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.053+0000 7f0ef02a2640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:06:24.071 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.053+0000 7f0ef02a2640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0ed40039c0 0x7f0ee81cf440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:06:24.071 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.053+0000 7f0ef02a2640 10 monclient(hunting): picked mon.noname-b con 0x7f0ed40039c0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:06:24.071 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.053+0000 7f0ef02a2640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0ed4004810 0x7f0ee81cf980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:06:24.071 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.053+0000 7f0ef02a2640 10 monclient(hunting): picked mon.noname-a con 0x7f0ed4004810 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:06:24.071 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.053+0000 7f0ef02a2640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0ed4005220 0x7f0ee81cfec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:06:24.071 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.053+0000 7f0ef02a2640 10 monclient(hunting): picked mon.noname-c con 0x7f0ed4005220 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:06:24.071 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.053+0000 7f0ef02a2640 10 monclient(hunting): start opening mon connection 2024-09-16T04:06:24.071 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.053+0000 7f0ef02a2640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f0ee8137bb0 con 0x7f0ed4004810 2024-09-16T04:06:24.071 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.053+0000 7f0ef02a2640 10 monclient(hunting): start opening mon connection 2024-09-16T04:06:24.071 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.053+0000 7f0ef02a2640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f0ee8137a30 con 0x7f0ed40039c0 2024-09-16T04:06:24.071 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.053+0000 7f0ef02a2640 10 monclient(hunting): start opening mon connection 2024-09-16T04:06:24.071 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.053+0000 7f0ef02a2640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f0ee813a180 con 0x7f0ed4005220 2024-09-16T04:06:24.071 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.053+0000 7f0ef02a2640 10 monclient(hunting): _renew_subs 2024-09-16T04:06:24.071 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.053+0000 7f0ef02a2640 10 monclient(hunting): authenticate will time out at 2431.027817s 2024-09-16T04:06:24.072 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.053+0000 7f0eee83f640 1 -- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0ed4005220 msgr2=0x7f0ee81cfec0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T04:06:24.072 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.053+0000 7f0eee83f640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0ed4005220 0x7f0ee81cfec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:06:24.072 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.053+0000 7f0eed83d640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0ed4004810 0x7f0ee81cf980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:06:24.072 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.057+0000 7f0eed83d640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0ed4004810 0x7f0ee81cf980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:47352/0 (socket says 172.21.15.42:47352) 2024-09-16T04:06:24.072 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.057+0000 7f0eed83d640 1 -- 172.21.15.42:0/1118420383 learned_addr learned my addr 172.21.15.42:0/1118420383 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:06:24.072 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.057+0000 7f0eed83d640 10 monclient(hunting): get_auth_request con 0x7f0ed4004810 auth_method 0 2024-09-16T04:06:24.072 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.057+0000 7f0eed83d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:06:24.072 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.057+0000 7f0eed83d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:24.072 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.057+0000 7f0eed83d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:24.072 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.057+0000 7f0eee03e640 1 --2- 172.21.15.42:0/1118420383 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0ed40039c0 0x7f0ee81cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:06:24.072 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.057+0000 7f0eee03e640 10 monclient(hunting): get_auth_request con 0x7f0ed40039c0 auth_method 0 2024-09-16T04:06:24.072 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.057+0000 7f0eee03e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:06:24.072 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.057+0000 7f0eee03e640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:24.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.057+0000 7f0eee03e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:24.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.057+0000 7f0eed83d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:06:24.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.057+0000 7f0eed83d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:06:24.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.057+0000 7f0eed83d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:06:24.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.057+0000 7f0eee03e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:06:24.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.057+0000 7f0eee03e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:06:24.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.057+0000 7f0eee03e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:06:24.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.057+0000 7f0eed83d640 10 monclient(hunting): handle_auth_done global_id 984429 payload 824 2024-09-16T04:06:24.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.057+0000 7f0eed83d640 1 -- 172.21.15.42:0/1118420383 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0ed4005220 msgr2=0x7f0ee81cfec0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:06:24.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.057+0000 7f0eed83d640 1 --2- 172.21.15.42:0/1118420383 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0ed4005220 0x7f0ee81cfec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:06:24.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.057+0000 7f0eed83d640 1 -- 172.21.15.42:0/1118420383 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0ed40039c0 msgr2=0x7f0ee81cf440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:24.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.057+0000 7f0eed83d640 1 --2- 172.21.15.42:0/1118420383 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0ed40039c0 0x7f0ee81cf440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:06:24.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.057+0000 7f0eed83d640 10 monclient: _finish_hunting 0 2024-09-16T04:06:24.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.057+0000 7f0eed83d640 1 monclient: found mon.noname-a 2024-09-16T04:06:24.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.057+0000 7f0eed83d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:06:24.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.057+0000 7f0eed83d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:06:24.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.057+0000 7f0eed83d640 1 -- 172.21.15.42:0/1118420383 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ee8122690 con 0x7f0ed4004810 2024-09-16T04:06:24.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.057+0000 7f0eed83d640 1 --2- 172.21.15.42:0/1118420383 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0ed4004810 0x7f0ee81cf980 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f0ed8000eb0 tx=0x7f0ed4005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:06:24.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.057+0000 7f0edeffd640 10 client.?.objecter ms_handle_connect 0x7f0ed4004810 2024-09-16T04:06:24.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.057+0000 7f0edeffd640 10 client.?.objecter resend_mon_ops 2024-09-16T04:06:24.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.057+0000 7f0edeffd640 1 -- 172.21.15.42:0/1118420383 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ed80bd070 con 0x7f0ed4004810 2024-09-16T04:06:24.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.057+0000 7f0edeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:24.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.057+0000 7f0edeffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:06:24.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.057+0000 7f0edeffd640 10 monclient: dump: 2024-09-16T04:06:24.074 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:06:24.074 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:06:24.074 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:06:24.074 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:06:24.075 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:24.075 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:06:24.075 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:06:24.075 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:06:24.075 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:06:24.075 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:06:24.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.057+0000 7f0edeffd640 10 monclient: _finish_auth 0 2024-09-16T04:06:24.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.057+0000 7f0edeffd640 10 monclient: _check_auth_tickets 2024-09-16T04:06:24.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.057+0000 7f0edeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:06:24.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.057+0000 7f0edeffd640 1 -- 172.21.15.42:0/1118420383 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ed8014000 con 0x7f0ed4004810 2024-09-16T04:06:24.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.057+0000 7f0ef02a2640 5 monclient: authenticate success, global_id 984429 2024-09-16T04:06:24.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.057+0000 7f0ef02a2640 10 monclient: _renew_subs 2024-09-16T04:06:24.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.057+0000 7f0ef02a2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:06:24.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.057+0000 7f0ef02a2640 1 -- 172.21.15.42:0/1118420383 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ee8122970 con 0x7f0ed4004810 2024-09-16T04:06:24.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.057+0000 7f0edeffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:06:24.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.057+0000 7f0edeffd640 1 -- 172.21.15.42:0/1118420383 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ed80b3040 con 0x7f0ed4004810 2024-09-16T04:06:24.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.057+0000 7f0edeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:24.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.057+0000 7f0edeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:06:24.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.057+0000 7f0edeffd640 10 monclient: dump: 2024-09-16T04:06:24.076 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:06:24.076 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:06:24.076 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:06:24.076 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:06:24.076 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:24.076 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:06:24.076 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:06:24.076 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:06:24.076 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:06:24.077 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:06:24.077 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.057+0000 7f0ef02a2640 10 client.984429.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:06:24.077 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.057+0000 7f0ef02a2640 10 monclient: _renew_subs 2024-09-16T04:06:24.077 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.057+0000 7f0ef02a2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:06:24.077 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.057+0000 7f0ef02a2640 1 -- 172.21.15.42:0/1118420383 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ee8130bb0 con 0x7f0ed4004810 2024-09-16T04:06:24.077 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.057+0000 7f0ef02a2640 1 librados: init done 2024-09-16T04:06:24.077 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.057+0000 7f0eec83b640 4 set_mon_vals no callback set 2024-09-16T04:06:24.077 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.057+0000 7f0ef02a2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:06:24.077 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.057+0000 7f0ef02a2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:06:24.077 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.057+0000 7f0ef02a2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:06:24.077 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.057+0000 7f0ef02a2640 1 -- 172.21.15.42:0/1118420383 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0ee813a420 con 0x7f0ed4004810 2024-09-16T04:06:24.077 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.057+0000 7f0edeffd640 1 -- 172.21.15.42:0/1118420383 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f0ed80b6070 con 0x7f0ed4004810 2024-09-16T04:06:24.077 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.057+0000 7f0edeffd640 1 --2- 172.21.15.42:0/1118420383 >> v2:172.21.15.154:6812/254845342 conn(0x7f0ecc038120 0x7f0ecc03a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:06:24.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.057+0000 7f0edeffd640 1 -- 172.21.15.42:0/1118420383 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(706..706 src has 451..706) ==== 5040+0+0 (secure 0 0 0) 0x7f0ed80e38a0 con 0x7f0ed4004810 2024-09-16T04:06:24.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.057+0000 7f0edeffd640 10 client.984429.objecter ms_dispatch 0x7f0ee8000ba0 osd_map(706..706 src has 451..706) 2024-09-16T04:06:24.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.057+0000 7f0edeffd640 3 client.984429.objecter handle_osd_map got epochs [706,706] > 0 2024-09-16T04:06:24.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.057+0000 7f0edeffd640 3 client.984429.objecter handle_osd_map decoding full epoch 706 2024-09-16T04:06:24.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.057+0000 7f0edeffd640 20 client.984429.objecter dump_active .. 0 homeless 2024-09-16T04:06:24.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.057+0000 7f0eee03e640 1 --2- 172.21.15.42:0/1118420383 >> v2:172.21.15.154:6812/254845342 conn(0x7f0ecc038120 0x7f0ecc03a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:06:24.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.057+0000 7f0eee03e640 10 monclient: get_auth_request con 0x7f0ecc038120 auth_method 0 2024-09-16T04:06:24.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.061+0000 7f0eee03e640 1 --2- 172.21.15.42:0/1118420383 >> v2:172.21.15.154:6812/254845342 conn(0x7f0ecc038120 0x7f0ecc03a5e0 secure :-1 s=READY pgs=3866 cs=0 l=1 rev1=1 crypto rx=0x7f0ee4007060 tx=0x7f0ee400b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:06:24.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.061+0000 7f0edeffd640 10 client.984429.objecter ms_handle_connect 0x7f0ecc038120 2024-09-16T04:06:24.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.065+0000 7f0edeffd640 1 -- 172.21.15.42:0/1118420383 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f0ed80a5e20 con 0x7f0ed4004810 2024-09-16T04:06:24.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.065+0000 7f0edeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:06:24.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.065+0000 7f0edeffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:06:24.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.305+0000 7f0ef02a2640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:06:24.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.305+0000 7f0ef02a2640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:06:24.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.305+0000 7f0ef02a2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:06:24.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.305+0000 7f0ef02a2640 1 -- 172.21.15.42:0/1118420383 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f0ee806f680 con 0x7f0ed4004810 2024-09-16T04:06:24.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.305+0000 7f0edeffd640 1 -- 172.21.15.42:0/1118420383 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1555 (secure 0 0 0) 0x7f0ed8017460 con 0x7f0ed4004810 2024-09-16T04:06:24.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.305+0000 7f0edeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:06:24.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.305+0000 7f0edeffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:06:24.312 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":226,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:06:24.314 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.309+0000 7f0ef02a2640 10 librados: watch_flush enter 2024-09-16T04:06:24.314 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.309+0000 7f0ef02a2640 10 librados: watch_flush exit 2024-09-16T04:06:24.314 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.309+0000 7f0ef02a2640 20 client.984429.objecter shutdown clearing up homeless session... 2024-09-16T04:06:24.314 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.309+0000 7f0ef02a2640 10 client.984429.objecter successfully canceled tick 2024-09-16T04:06:24.314 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.309+0000 7f0ef02a2640 1 -- 172.21.15.42:0/1118420383 >> v2:172.21.15.154:6812/254845342 conn(0x7f0ecc038120 msgr2=0x7f0ecc03a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:24.314 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.309+0000 7f0ef02a2640 1 --2- 172.21.15.42:0/1118420383 >> v2:172.21.15.154:6812/254845342 conn(0x7f0ecc038120 0x7f0ecc03a5e0 secure :-1 s=READY pgs=3866 cs=0 l=1 rev1=1 crypto rx=0x7f0ee4007060 tx=0x7f0ee400b040 comp rx=0 tx=0).stop 2024-09-16T04:06:24.314 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.309+0000 7f0ef02a2640 10 monclient: shutdown 2024-09-16T04:06:24.314 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.309+0000 7f0ef02a2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:06:24.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.309+0000 7f0ef02a2640 1 -- 172.21.15.42:0/1118420383 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0ed4004810 msgr2=0x7f0ee81cf980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:24.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.309+0000 7f0ef02a2640 1 --2- 172.21.15.42:0/1118420383 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0ed4004810 0x7f0ee81cf980 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f0ed8000eb0 tx=0x7f0ed4005840 comp rx=0 tx=0).stop 2024-09-16T04:06:24.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.309+0000 7f0ef02a2640 1 -- 172.21.15.42:0/1118420383 shutdown_connections 2024-09-16T04:06:24.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.309+0000 7f0ef02a2640 1 --2- 172.21.15.42:0/1118420383 >> v2:172.21.15.154:6812/254845342 conn(0x7f0ecc038120 0x7f0ecc03a5e0 unknown :-1 s=CLOSED pgs=3866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:06:24.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.309+0000 7f0ef02a2640 1 --2- 172.21.15.42:0/1118420383 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0ed4005220 0x7f0ee81cfec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:06:24.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.309+0000 7f0ef02a2640 1 --2- 172.21.15.42:0/1118420383 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0ed4004810 0x7f0ee81cf980 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:06:24.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.309+0000 7f0ef02a2640 1 --2- 172.21.15.42:0/1118420383 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0ed40039c0 0x7f0ee81cf440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:06:24.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.309+0000 7f0ef02a2640 1 -- 172.21.15.42:0/1118420383 >> 172.21.15.42:0/1118420383 conn(0x7f0ee80a9fb0 msgr2=0x7f0ee8131790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:06:24.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.309+0000 7f0ef02a2640 1 -- 172.21.15.42:0/1118420383 shutdown_connections 2024-09-16T04:06:24.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.309+0000 7f0ef02a2640 1 -- 172.21.15.42:0/1118420383 wait complete. 2024-09-16T04:06:24.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.309+0000 7f0ef02a2640 1 librados: shutdown 2024-09-16T04:06:24.331 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T04:06:24.332 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-16T04:06:24.332 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T04:06:24.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.453+0000 7f708c785640 10 monclient: get_monmap_and_config 2024-09-16T04:06:24.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.453+0000 7f708c785640 10 monclient: build_initial_monmap 2024-09-16T04:06:24.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.453+0000 7f708c785640 10 monclient: monmap: 2024-09-16T04:06:24.458 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:06:24.458 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:06:24.458 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:06:24.456748+0000 2024-09-16T04:06:24.459 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:06:24.456748+0000 2024-09-16T04:06:24.459 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:06:24.459 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:06:24.459 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:06:24.459 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:06:24.459 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:06:24.459 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:06:24.459 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:06:24.459 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:06:24.461 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.453+0000 7f708c785640 1 Processor -- start 2024-09-16T04:06:24.461 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.453+0000 7f708c785640 1 -- start start 2024-09-16T04:06:24.461 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.453+0000 7f708c785640 10 monclient: init 2024-09-16T04:06:24.461 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.453+0000 7f708c785640 10 monclient: _reopen_session rank -1 2024-09-16T04:06:24.461 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.453+0000 7f708c785640 10 monclient: _add_conns ranks=[3,4,1,2,0] 2024-09-16T04:06:24.462 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.453+0000 7f708c785640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f70840a1330 0x7f70840a1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:06:24.462 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.453+0000 7f708c785640 10 monclient(hunting): picked mon.noname-a con 0x7f70840a1330 addr v2:172.21.15.42:3300/0 2024-09-16T04:06:24.462 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.453+0000 7f708c785640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f708406dac0 0x7f70840a1c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:06:24.462 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.453+0000 7f708c785640 10 monclient(hunting): picked mon.noname-c con 0x7f708406dac0 addr v2:172.21.15.154:3300/0 2024-09-16T04:06:24.462 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.453+0000 7f708c785640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7084135100 addr v1:172.21.15.154:6789/0 2024-09-16T04:06:24.462 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.453+0000 7f708c785640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f708409c7d0 con 0x7f7084135100 2024-09-16T04:06:24.462 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.453+0000 7f708c785640 10 monclient(hunting): start opening mon connection 2024-09-16T04:06:24.462 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.453+0000 7f708c785640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f708409c090 con 0x7f70840a1330 2024-09-16T04:06:24.462 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.453+0000 7f708c785640 10 monclient(hunting): start opening mon connection 2024-09-16T04:06:24.462 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.453+0000 7f708c785640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f708409f610 con 0x7f708406dac0 2024-09-16T04:06:24.462 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.453+0000 7f708c785640 10 monclient(hunting): _renew_subs 2024-09-16T04:06:24.462 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.453+0000 7f708c785640 10 monclient(hunting): authenticate will time out at 2431.426210s 2024-09-16T04:06:24.462 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.453+0000 7f708a521640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f70840a1330 0x7f70840a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:06:24.462 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.453+0000 7f708a521640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f70840a1330 0x7f70840a1710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:47358/0 (socket says 172.21.15.42:47358) 2024-09-16T04:06:24.463 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.453+0000 7f708a521640 1 -- 172.21.15.42:0/4237241536 learned_addr learned my addr 172.21.15.42:0/4237241536 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:06:24.463 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.453+0000 7f708a521640 10 monclient(hunting): get_auth_request con 0x7f70840a1330 auth_method 0 2024-09-16T04:06:24.463 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.453+0000 7f708a521640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:06:24.463 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.453+0000 7f708a521640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:24.463 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.453+0000 7f708a521640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:24.463 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.453+0000 7f708a521640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:06:24.463 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.453+0000 7f708a521640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:06:24.463 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.453+0000 7f708a521640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:06:24.463 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.453+0000 7f7089d20640 1 --2- 172.21.15.42:0/4237241536 >> v2:172.21.15.154:3300/0 conn(0x7f708406dac0 0x7f70840a1c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:06:24.463 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.453+0000 7f7089d20640 10 monclient(hunting): get_auth_request con 0x7f708406dac0 auth_method 0 2024-09-16T04:06:24.463 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.453+0000 7f7089d20640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:06:24.463 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.453+0000 7f7089d20640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:24.463 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.453+0000 7f7089d20640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:24.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.453+0000 7f708a521640 10 monclient(hunting): handle_auth_done global_id 984432 payload 274 2024-09-16T04:06:24.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.453+0000 7f708a521640 1 -- 172.21.15.42:0/4237241536 >> v2:172.21.15.154:3300/0 conn(0x7f708406dac0 msgr2=0x7f70840a1c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:06:24.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.453+0000 7f708a521640 1 --2- 172.21.15.42:0/4237241536 >> v2:172.21.15.154:3300/0 conn(0x7f708406dac0 0x7f70840a1c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:06:24.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.453+0000 7f708a521640 1 -- 172.21.15.42:0/4237241536 >> v1:172.21.15.154:6789/0 conn(0x7f7084135100 legacy=0x7f7084135550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:06:24.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.453+0000 7f708a521640 10 monclient: _finish_hunting 0 2024-09-16T04:06:24.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.453+0000 7f708a521640 1 monclient: found mon.noname-a 2024-09-16T04:06:24.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.453+0000 7f708a521640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:06:24.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.453+0000 7f708a521640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:06:24.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.453+0000 7f708a521640 1 -- 172.21.15.42:0/4237241536 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f708409f8f0 con 0x7f70840a1330 2024-09-16T04:06:24.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.453+0000 7f7089d20640 1 --2- 172.21.15.42:0/4237241536 >> v2:172.21.15.154:3300/0 conn(0x7f708406dac0 0x7f70840a1c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T04:06:24.471 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.457+0000 7f708a521640 1 --2- 172.21.15.42:0/4237241536 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f70840a1330 0x7f70840a1710 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f708406da10 tx=0x7f707801c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d1ac5defcffa526 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:06:24.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.457+0000 7f708951f640 1 -- 172.21.15.42:0/4237241536 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70780aa650 con 0x7f70840a1330 2024-09-16T04:06:24.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.457+0000 7f708951f640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:24.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.457+0000 7f708951f640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:06:24.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.457+0000 7f708951f640 10 monclient: dump: 2024-09-16T04:06:24.472 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:06:24.473 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:06:24.473 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:06:24.473 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:06:24.473 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:24.473 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:06:24.473 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:06:24.474 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:06:24.474 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:06:24.474 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:06:24.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.457+0000 7f708951f640 10 monclient: _finish_auth 0 2024-09-16T04:06:24.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.457+0000 7f708951f640 10 monclient: _check_auth_tickets 2024-09-16T04:06:24.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.457+0000 7f708951f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:06:24.475 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.457+0000 7f708c785640 5 monclient: authenticate success, global_id 984432 2024-09-16T04:06:24.475 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.457+0000 7f708c785640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:06:24.475 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.457+0000 7f708951f640 1 -- 172.21.15.42:0/4237241536 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f70780af070 con 0x7f70840a1330 2024-09-16T04:06:24.475 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.457+0000 7f708951f640 10 monclient: handle_config config(0 keys) 2024-09-16T04:06:24.475 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.457+0000 7f708951f640 1 -- 172.21.15.42:0/4237241536 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70780aaa10 con 0x7f70840a1330 2024-09-16T04:06:24.476 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.457+0000 7f708951f640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:24.476 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.457+0000 7f708951f640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:06:24.476 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.457+0000 7f708951f640 10 monclient: dump: 2024-09-16T04:06:24.476 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:06:24.476 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:06:24.476 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:06:24.477 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:06:24.477 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:24.477 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:06:24.477 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:06:24.477 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:06:24.478 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:06:24.478 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:06:24.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.457+0000 7f708c785640 10 monclient: get_monmap_and_config success 2024-09-16T04:06:24.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.457+0000 7f708c785640 4 set_mon_vals no callback set 2024-09-16T04:06:24.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.457+0000 7f708c785640 10 monclient: shutdown 2024-09-16T04:06:24.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.457+0000 7f708c785640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:06:24.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.457+0000 7f708c785640 1 -- 172.21.15.42:0/4237241536 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f70840a1330 msgr2=0x7f70840a1710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:24.479 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.457+0000 7f708c785640 1 --2- 172.21.15.42:0/4237241536 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f70840a1330 0x7f70840a1710 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f708406da10 tx=0x7f707801c4d0 comp rx=0 tx=0).stop 2024-09-16T04:06:24.479 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.457+0000 7f708c785640 1 -- 172.21.15.42:0/4237241536 shutdown_connections 2024-09-16T04:06:24.479 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.457+0000 7f708c785640 1 --2- 172.21.15.42:0/4237241536 >> v2:172.21.15.154:3300/0 conn(0x7f708406dac0 0x7f70840a1c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:06:24.479 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.457+0000 7f708c785640 1 --2- 172.21.15.42:0/4237241536 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f70840a1330 0x7f70840a1710 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:06:24.479 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.457+0000 7f708c785640 1 -- 172.21.15.42:0/4237241536 >> 172.21.15.42:0/4237241536 conn(0x7f70840a2420 msgr2=0x7f70840a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:06:24.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.457+0000 7f708c785640 1 -- 172.21.15.42:0/4237241536 shutdown_connections 2024-09-16T04:06:24.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.457+0000 7f708c785640 1 -- 172.21.15.42:0/4237241536 wait complete. 2024-09-16T04:06:24.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.457+0000 7f708c785640 10 monclient: build_initial_monmap 2024-09-16T04:06:24.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.457+0000 7f708c785640 10 monclient: monmap: 2024-09-16T04:06:24.480 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:06:24.480 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:06:24.481 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:06:24.481 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:06:24.481 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:06:24.481 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:06:24.481 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:06:24.481 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:06:24.482 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:06:24.482 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:06:24.482 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.457+0000 7f708c785640 1 librados: starting msgr at 2024-09-16T04:06:24.482 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.457+0000 7f708c785640 1 librados: starting objecter 2024-09-16T04:06:24.482 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.457+0000 7f708c785640 1 Processor -- start 2024-09-16T04:06:24.482 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.457+0000 7f708c785640 1 -- start start 2024-09-16T04:06:24.483 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.457+0000 7f708c785640 1 librados: setting wanted keys 2024-09-16T04:06:24.483 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.457+0000 7f708c785640 1 librados: calling monclient init 2024-09-16T04:06:24.483 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.457+0000 7f708c785640 10 monclient: init 2024-09-16T04:06:24.483 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.457+0000 7f708c785640 10 monclient: _reopen_session rank -1 2024-09-16T04:06:24.483 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.457+0000 7f708c785640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:06:24.483 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.457+0000 7f708c785640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f70840a1330 0x7f708413c050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:06:24.484 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.457+0000 7f708c785640 10 monclient(hunting): picked mon.noname-a con 0x7f70840a1330 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:06:24.484 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.457+0000 7f708c785640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7084135100 0x7f7084134240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:06:24.484 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.457+0000 7f708c785640 10 monclient(hunting): picked mon.noname-b con 0x7f7084135100 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:06:24.484 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.457+0000 7f708c785640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7084134780 0x7f708411bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:06:24.484 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.457+0000 7f708c785640 10 monclient(hunting): picked mon.noname-c con 0x7f7084134780 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:06:24.484 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.457+0000 7f708c785640 10 monclient(hunting): start opening mon connection 2024-09-16T04:06:24.485 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.457+0000 7f708c785640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f708413fb80 con 0x7f70840a1330 2024-09-16T04:06:24.485 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.457+0000 7f708c785640 10 monclient(hunting): start opening mon connection 2024-09-16T04:06:24.485 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.457+0000 7f708c785640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f708409ef80 con 0x7f7084135100 2024-09-16T04:06:24.485 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.457+0000 7f708c785640 10 monclient(hunting): start opening mon connection 2024-09-16T04:06:24.485 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.457+0000 7f708c785640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f708413fd00 con 0x7f7084134780 2024-09-16T04:06:24.485 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.457+0000 7f708c785640 10 monclient(hunting): _renew_subs 2024-09-16T04:06:24.486 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.457+0000 7f708c785640 10 monclient(hunting): authenticate will time out at 2431.431965s 2024-09-16T04:06:24.486 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.461+0000 7f708ad22640 1 -- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7084134780 msgr2=0x7f708411bf70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T04:06:24.486 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.461+0000 7f708ad22640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7084134780 0x7f708411bf70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:06:24.486 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.461+0000 7f708a521640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f70840a1330 0x7f708413c050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:06:24.486 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.461+0000 7f708a521640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f70840a1330 0x7f708413c050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:47360/0 (socket says 172.21.15.42:47360) 2024-09-16T04:06:24.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.461+0000 7f708a521640 1 -- 172.21.15.42:0/2013850742 learned_addr learned my addr 172.21.15.42:0/2013850742 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:06:24.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.461+0000 7f708a521640 10 monclient(hunting): get_auth_request con 0x7f70840a1330 auth_method 0 2024-09-16T04:06:24.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.461+0000 7f708a521640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:06:24.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.461+0000 7f708a521640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:24.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.461+0000 7f708a521640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:24.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.461+0000 7f708a521640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:06:24.488 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.461+0000 7f708a521640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:06:24.488 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.461+0000 7f708a521640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:06:24.488 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.461+0000 7f7089d20640 1 --2- 172.21.15.42:0/2013850742 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7084135100 0x7f7084134240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:06:24.488 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.461+0000 7f7089d20640 10 monclient(hunting): get_auth_request con 0x7f7084135100 auth_method 0 2024-09-16T04:06:24.488 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.461+0000 7f7089d20640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:06:24.489 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.461+0000 7f7089d20640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:24.489 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.461+0000 7f7089d20640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:24.489 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.461+0000 7f708a521640 10 monclient(hunting): handle_auth_done global_id 984435 payload 824 2024-09-16T04:06:24.489 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.461+0000 7f708a521640 1 -- 172.21.15.42:0/2013850742 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7084134780 msgr2=0x7f708411bf70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:06:24.489 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.461+0000 7f708a521640 1 --2- 172.21.15.42:0/2013850742 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7084134780 0x7f708411bf70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:06:24.489 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.461+0000 7f708a521640 1 -- 172.21.15.42:0/2013850742 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7084135100 msgr2=0x7f7084134240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:24.490 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.461+0000 7f708a521640 1 --2- 172.21.15.42:0/2013850742 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7084135100 0x7f7084134240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:06:24.490 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.461+0000 7f708a521640 10 monclient: _finish_hunting 0 2024-09-16T04:06:24.490 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.461+0000 7f708a521640 1 monclient: found mon.noname-a 2024-09-16T04:06:24.490 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.461+0000 7f708a521640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:06:24.490 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.461+0000 7f708a521640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:06:24.490 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.461+0000 7f708a521640 1 -- 172.21.15.42:0/2013850742 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f708411ebd0 con 0x7f70840a1330 2024-09-16T04:06:24.490 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.461+0000 7f708a521640 1 --2- 172.21.15.42:0/2013850742 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f70840a1330 0x7f708413c050 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f70780ac040 tx=0x7f707801ad10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:06:24.491 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.461+0000 7f7076ffd640 10 client.?.objecter ms_handle_connect 0x7f70840a1330 2024-09-16T04:06:24.491 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.461+0000 7f7076ffd640 10 client.?.objecter resend_mon_ops 2024-09-16T04:06:24.491 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.461+0000 7f7076ffd640 1 -- 172.21.15.42:0/2013850742 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70780bd5f0 con 0x7f70840a1330 2024-09-16T04:06:24.491 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.461+0000 7f7076ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:24.491 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.461+0000 7f7076ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:06:24.491 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.461+0000 7f7076ffd640 10 monclient: dump: 2024-09-16T04:06:24.492 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:06:24.492 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:06:24.492 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:06:24.492 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:06:24.492 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:24.492 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:06:24.492 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:06:24.493 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:06:24.493 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:06:24.493 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:06:24.493 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.461+0000 7f7076ffd640 10 monclient: _finish_auth 0 2024-09-16T04:06:24.493 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.461+0000 7f7076ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:06:24.493 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.461+0000 7f7076ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:06:24.493 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.461+0000 7f708c785640 5 monclient: authenticate success, global_id 984435 2024-09-16T04:06:24.493 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.461+0000 7f7076ffd640 1 -- 172.21.15.42:0/2013850742 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f70780a6040 con 0x7f70840a1330 2024-09-16T04:06:24.493 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.461+0000 7f7076ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:06:24.493 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.461+0000 7f7076ffd640 1 -- 172.21.15.42:0/2013850742 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70780bd9f0 con 0x7f70840a1330 2024-09-16T04:06:24.493 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.461+0000 7f7076ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:24.493 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.461+0000 7f7076ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:06:24.494 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.461+0000 7f7077fff640 4 set_mon_vals no callback set 2024-09-16T04:06:24.494 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.461+0000 7f7076ffd640 10 monclient: dump: 2024-09-16T04:06:24.494 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:06:24.494 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:06:24.494 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:06:24.494 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:06:24.494 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:24.494 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:06:24.494 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:06:24.494 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:06:24.494 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:06:24.494 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:06:24.495 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.461+0000 7f708c785640 10 monclient: _renew_subs 2024-09-16T04:06:24.495 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.461+0000 7f708c785640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:06:24.495 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.461+0000 7f708c785640 1 -- 172.21.15.42:0/2013850742 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7084071b00 con 0x7f70840a1330 2024-09-16T04:06:24.495 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.461+0000 7f708c785640 10 client.984435.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:06:24.495 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.461+0000 7f708c785640 10 monclient: _renew_subs 2024-09-16T04:06:24.495 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.461+0000 7f708c785640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:06:24.495 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.461+0000 7f708c785640 1 -- 172.21.15.42:0/2013850742 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f708413cc90 con 0x7f70840a1330 2024-09-16T04:06:24.495 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.461+0000 7f708c785640 1 librados: init done 2024-09-16T04:06:24.495 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.461+0000 7f708c785640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:06:24.495 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.461+0000 7f708c785640 1 --2- 172.21.15.42:0/2013850742 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f708406dac0 0x7f708402c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:06:24.495 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.461+0000 7f708c785640 10 monclient(hunting): start opening mon connection 2024-09-16T04:06:24.495 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.461+0000 7f708c785640 1 -- 172.21.15.42:0/2013850742 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f708411e0b0 con 0x7f708406dac0 2024-09-16T04:06:24.495 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.461+0000 7f7089d20640 1 --2- 172.21.15.42:0/2013850742 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f708406dac0 0x7f708402c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:06:24.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.461+0000 7f7089d20640 10 monclient: get_auth_request con 0x7f708406dac0 auth_method 0 2024-09-16T04:06:24.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.461+0000 7f7089d20640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:06:24.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.461+0000 7f7089d20640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:24.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.461+0000 7f7089d20640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:24.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.461+0000 7f7089d20640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:06:24.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.461+0000 7f7089d20640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:06:24.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.461+0000 7f7076ffd640 1 -- 172.21.15.42:0/2013850742 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f70780c6440 con 0x7f70840a1330 2024-09-16T04:06:24.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.461+0000 7f7089d20640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:06:24.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.461+0000 7f7089d20640 10 monclient(hunting): handle_auth_done global_id 984438 payload 824 2024-09-16T04:06:24.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.461+0000 7f7089d20640 1 -- 172.21.15.42:0/2013850742 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7084135550 con 0x7f708406dac0 2024-09-16T04:06:24.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.461+0000 7f7076ffd640 1 --2- 172.21.15.42:0/2013850742 >> v2:172.21.15.154:6812/254845342 conn(0x7f70680377c0 0x7f7068039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:06:24.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.461+0000 7f7076ffd640 1 -- 172.21.15.42:0/2013850742 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(706..706 src has 451..706) ==== 5040+0+0 (secure 0 0 0) 0x7f70780e34b0 con 0x7f70840a1330 2024-09-16T04:06:24.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.461+0000 7f7076ffd640 10 client.984435.objecter ms_dispatch 0x7f7084000ba0 osd_map(706..706 src has 451..706) 2024-09-16T04:06:24.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.461+0000 7f7076ffd640 3 client.984435.objecter handle_osd_map got epochs [706,706] > 0 2024-09-16T04:06:24.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.461+0000 7f7076ffd640 3 client.984435.objecter handle_osd_map decoding full epoch 706 2024-09-16T04:06:24.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.461+0000 7f7076ffd640 20 client.984435.objecter dump_active .. 0 homeless 2024-09-16T04:06:24.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.461+0000 7f7076ffd640 10 client.984435.objecter ms_handle_connect 0x7f708406dac0 2024-09-16T04:06:24.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.461+0000 7f7076ffd640 10 client.984435.objecter resend_mon_ops 2024-09-16T04:06:24.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.461+0000 7f7089d20640 1 --2- 172.21.15.42:0/2013850742 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f708406dac0 0x7f708402c650 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f708413c890 tx=0x7f7080007700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:06:24.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.461+0000 7f7076ffd640 1 -- 172.21.15.42:0/2013850742 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7080014610 con 0x7f708406dac0 2024-09-16T04:06:24.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.461+0000 7f7076ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:24.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.461+0000 7f7076ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:06:24.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.461+0000 7f7076ffd640 10 monclient: dump: 2024-09-16T04:06:24.497 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:06:24.497 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:06:24.497 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:06:24.497 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:06:24.498 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:24.498 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:06:24.498 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:06:24.498 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:06:24.498 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:06:24.498 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:06:24.498 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.461+0000 7f708ad22640 1 --2- 172.21.15.42:0/2013850742 >> v2:172.21.15.154:6812/254845342 conn(0x7f70680377c0 0x7f7068039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:06:24.498 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.461+0000 7f708ad22640 10 monclient: get_auth_request con 0x7f70680377c0 auth_method 0 2024-09-16T04:06:24.498 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.461+0000 7f7076ffd640 1 -- 172.21.15.42:0/2013850742 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f7080019070 con 0x7f708406dac0 2024-09-16T04:06:24.498 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.461+0000 7f7076ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:06:24.498 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.461+0000 7f7076ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:06:24.499 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.461+0000 7f7076ffd640 1 -- 172.21.15.42:0/2013850742 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f708406dac0 msgr2=0x7f708402c650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:24.499 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.461+0000 7f7076ffd640 1 --2- 172.21.15.42:0/2013850742 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f708406dac0 0x7f708402c650 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f708413c890 tx=0x7f7080007700 comp rx=0 tx=0).stop 2024-09-16T04:06:24.499 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.461+0000 7f7076ffd640 1 -- 172.21.15.42:0/2013850742 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f708406dac0 msgr2=0x7f708402c650 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:06:24.499 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.461+0000 7f7076ffd640 1 --2- 172.21.15.42:0/2013850742 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f708406dac0 0x7f708402c650 secure :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f708413c890 tx=0x7f7080007700 comp rx=0 tx=0).stop 2024-09-16T04:06:24.499 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.461+0000 7f708ad22640 1 --2- 172.21.15.42:0/2013850742 >> v2:172.21.15.154:6812/254845342 conn(0x7f70680377c0 0x7f7068039c80 secure :-1 s=READY pgs=3867 cs=0 l=1 rev1=1 crypto rx=0x7f707c001040 tx=0x7f707c0079c0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:06:24.499 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.461+0000 7f7076ffd640 10 client.984435.objecter ms_handle_connect 0x7f70680377c0 2024-09-16T04:06:24.501 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.469+0000 7f708c785640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:06:24.501 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.469+0000 7f708c785640 1 --2- 172.21.15.42:0/2013850742 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f708402cb90 0x7f708402cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:06:24.501 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.469+0000 7f708c785640 10 monclient(hunting): start opening mon connection 2024-09-16T04:06:24.501 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.469+0000 7f708c785640 1 -- 172.21.15.42:0/2013850742 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f7084139150 con 0x7f708402cb90 2024-09-16T04:06:24.501 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.469+0000 7f7089d20640 1 --2- 172.21.15.42:0/2013850742 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f708402cb90 0x7f708402cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:06:24.501 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.469+0000 7f7089d20640 10 monclient: get_auth_request con 0x7f708402cb90 auth_method 0 2024-09-16T04:06:24.502 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.469+0000 7f7089d20640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:06:24.502 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.469+0000 7f7089d20640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:24.502 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.469+0000 7f7089d20640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:24.502 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.469+0000 7f7089d20640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:06:24.502 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.469+0000 7f7089d20640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:06:24.502 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.469+0000 7f7089d20640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:06:24.502 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.469+0000 7f7089d20640 10 monclient(hunting): handle_auth_done global_id 984441 payload 824 2024-09-16T04:06:24.502 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.469+0000 7f7089d20640 1 -- 172.21.15.42:0/2013850742 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f7084069040 con 0x7f708402cb90 2024-09-16T04:06:24.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.469+0000 7f7089d20640 1 --2- 172.21.15.42:0/2013850742 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f708402cb90 0x7f708402cfa0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f708413c890 tx=0x7f7080004a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:06:24.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.469+0000 7f7076ffd640 10 client.984435.objecter ms_handle_connect 0x7f708402cb90 2024-09-16T04:06:24.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.469+0000 7f7076ffd640 10 client.984435.objecter resend_mon_ops 2024-09-16T04:06:24.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.469+0000 7f7076ffd640 1 -- 172.21.15.42:0/2013850742 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7080014e30 con 0x7f708402cb90 2024-09-16T04:06:24.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.469+0000 7f7076ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:24.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.469+0000 7f7076ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:06:24.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.469+0000 7f7076ffd640 10 monclient: dump: 2024-09-16T04:06:24.503 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:06:24.503 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:06:24.503 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:06:24.503 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:06:24.503 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:24.503 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:06:24.503 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:06:24.504 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:06:24.504 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:06:24.504 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:06:24.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.469+0000 7f7076ffd640 1 -- 172.21.15.42:0/2013850742 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4704 (secure 0 0 0) 0x7f7080014e30 con 0x7f708402cb90 2024-09-16T04:06:24.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.469+0000 7f7076ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:06:24.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.469+0000 7f7076ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:06:24.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.469+0000 7f7076ffd640 1 -- 172.21.15.42:0/2013850742 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f708402cb90 msgr2=0x7f708402cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:24.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.469+0000 7f7076ffd640 1 --2- 172.21.15.42:0/2013850742 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f708402cb90 0x7f708402cfa0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f708413c890 tx=0x7f7080004a50 comp rx=0 tx=0).stop 2024-09-16T04:06:24.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.469+0000 7f7076ffd640 1 -- 172.21.15.42:0/2013850742 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f708402cb90 msgr2=0x7f708402cfa0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:06:24.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.469+0000 7f7076ffd640 1 --2- 172.21.15.42:0/2013850742 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f708402cb90 0x7f708402cfa0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:06:24.504 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T04:06:24.505 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:06:24.505 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:06:24.505 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "leader", 2024-09-16T04:06:24.505 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 226, 2024-09-16T04:06:24.505 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 144526, 2024-09-16T04:06:24.505 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T04:06:24.505 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T04:06:24.505 INFO:teuthology.orchestra.run.smithi042.stdout: 1 2024-09-16T04:06:24.505 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:06:24.506 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 0, 2024-09-16T04:06:24.506 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:06:24.506 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T04:06:24.506 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T04:06:24.506 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:06:24.506 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:06:24.506 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:06:24.506 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:06:24.506 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:06:24.506 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:06:24.506 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:06:24.506 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:06:24.507 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:06:24.507 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:06:24.507 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:06:24.507 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:06:24.507 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:06:24.507 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T04:06:24.507 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:06:24.507 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:06:24.507 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:06:24.507 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:06:24.507 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:06:24.507 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:06:24.507 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:06:24.507 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:06:24.508 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:06:24.508 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:06:24.508 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:06:24.508 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:06:24.508 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:06:24.508 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T04:06:24.508 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T04:06:24.508 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T04:06:24.508 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T04:06:24.508 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T04:06:24.508 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T04:06:24.508 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:06:24.508 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:06:24.509 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T04:06:24.509 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T04:06:24.509 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T04:06:24.509 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T04:06:24.509 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T04:06:24.509 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T04:06:24.509 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T04:06:24.509 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:06:24.510 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T04:06:24.510 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:06:24.510 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:06:24.510 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:06:24.510 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:06:24.510 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:06:24.510 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:06:24.510 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:06:24.511 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:06:24.511 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:06:24.511 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:06:24.511 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:06:24.511 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:06:24.511 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T04:06:24.511 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:06:24.511 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T04:06:24.511 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:06:24.512 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:06:24.512 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:06:24.512 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:06:24.512 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:06:24.512 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:06:24.512 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:06:24.512 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T04:06:24.512 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:06:24.513 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:06:24.513 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:06:24.513 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:06:24.513 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T04:06:24.513 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:06:24.513 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:06:24.513 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:06:24.514 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:06:24.514 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T04:06:24.514 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T04:06:24.514 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:06:24.514 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:06:24.514 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:06:24.514 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:06:24.514 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:06:24.515 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:06:24.515 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:06:24.515 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:06:24.515 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:06:24.515 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:06:24.515 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:06:24.515 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T04:06:24.515 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:06:24.516 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:06:24.516 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:06:24.516 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:06:24.516 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T04:06:24.516 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:06:24.516 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:06:24.516 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:06:24.516 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:06:24.517 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T04:06:24.517 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T04:06:24.517 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:06:24.517 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:06:24.517 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:06:24.517 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:06:24.518 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:06:24.518 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:06:24.518 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:06:24.518 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:06:24.518 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:06:24.518 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:06:24.519 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:06:24.519 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T04:06:24.519 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:06:24.519 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:06:24.519 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:06:24.519 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:06:24.519 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T04:06:24.519 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:06:24.519 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:06:24.519 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:06:24.520 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:06:24.520 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T04:06:24.520 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T04:06:24.520 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:06:24.520 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:06:24.520 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:06:24.520 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:06:24.520 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:06:24.520 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:06:24.520 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T04:06:24.520 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T04:06:24.521 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:06:24.521 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:06:24.521 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:06:24.521 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:06:24.521 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:06:24.521 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:06:24.521 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T04:06:24.521 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:06:24.521 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:06:24.521 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:06:24.521 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 3 2024-09-16T04:06:24.521 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:06:24.522 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:06:24.522 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T04:06:24.522 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:06:24.522 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:06:24.522 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:06:24.522 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 3 2024-09-16T04:06:24.522 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:06:24.522 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:06:24.522 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:06:24.522 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T04:06:24.522 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T04:06:24.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.469+0000 7f708c785640 10 librados: watch_flush enter 2024-09-16T04:06:24.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.469+0000 7f708c785640 10 librados: watch_flush exit 2024-09-16T04:06:24.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.469+0000 7f708c785640 20 client.984435.objecter shutdown clearing up homeless session... 2024-09-16T04:06:24.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.469+0000 7f708c785640 10 client.984435.objecter successfully canceled tick 2024-09-16T04:06:24.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.469+0000 7f708c785640 1 -- 172.21.15.42:0/2013850742 >> v2:172.21.15.154:6812/254845342 conn(0x7f70680377c0 msgr2=0x7f7068039c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:24.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.469+0000 7f708c785640 1 --2- 172.21.15.42:0/2013850742 >> v2:172.21.15.154:6812/254845342 conn(0x7f70680377c0 0x7f7068039c80 secure :-1 s=READY pgs=3867 cs=0 l=1 rev1=1 crypto rx=0x7f707c001040 tx=0x7f707c0079c0 comp rx=0 tx=0).stop 2024-09-16T04:06:24.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.469+0000 7f708c785640 10 monclient: shutdown 2024-09-16T04:06:24.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.469+0000 7f708c785640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:06:24.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.469+0000 7f708c785640 1 -- 172.21.15.42:0/2013850742 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f70840a1330 msgr2=0x7f708413c050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:24.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.469+0000 7f708c785640 1 --2- 172.21.15.42:0/2013850742 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f70840a1330 0x7f708413c050 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f70780ac040 tx=0x7f707801ad10 comp rx=0 tx=0).stop 2024-09-16T04:06:24.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.469+0000 7f708ad22640 1 -- 172.21.15.42:0/2013850742 reap_dead start 2024-09-16T04:06:24.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.469+0000 7f708ad22640 1 -- 172.21.15.42:0/2013850742 reap_dead start 2024-09-16T04:06:24.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.469+0000 7f708c785640 1 -- 172.21.15.42:0/2013850742 shutdown_connections 2024-09-16T04:06:24.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.469+0000 7f708c785640 1 -- 172.21.15.42:0/2013850742 >> 172.21.15.42:0/2013850742 conn(0x7f70840a2420 msgr2=0x7f7084136320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:06:24.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.469+0000 7f708c785640 1 -- 172.21.15.42:0/2013850742 shutdown_connections 2024-09-16T04:06:24.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.469+0000 7f708c785640 1 -- 172.21.15.42:0/2013850742 wait complete. 2024-09-16T04:06:24.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.469+0000 7f708c785640 1 librados: shutdown 2024-09-16T04:06:24.526 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T04:06:24.526 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-16T04:06:24.527 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T04:06:24.651 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.645+0000 7f7bd2f29640 10 monclient: get_monmap_and_config 2024-09-16T04:06:24.651 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.645+0000 7f7bd2f29640 10 monclient: build_initial_monmap 2024-09-16T04:06:24.651 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.645+0000 7f7bd2f29640 10 monclient: monmap: 2024-09-16T04:06:24.651 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:06:24.651 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:06:24.651 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:06:24.649228+0000 2024-09-16T04:06:24.652 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:06:24.649228+0000 2024-09-16T04:06:24.652 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:06:24.652 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:06:24.652 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:06:24.652 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:06:24.652 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:06:24.652 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:06:24.652 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:06:24.652 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:06:24.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.645+0000 7f7bd2f29640 1 Processor -- start 2024-09-16T04:06:24.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.645+0000 7f7bd2f29640 1 -- start start 2024-09-16T04:06:24.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.645+0000 7f7bd2f29640 10 monclient: init 2024-09-16T04:06:24.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.645+0000 7f7bd2f29640 10 monclient: _reopen_session rank -1 2024-09-16T04:06:24.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.645+0000 7f7bd2f29640 10 monclient: _add_conns ranks=[0,3,4,1,2] 2024-09-16T04:06:24.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.645+0000 7f7bd2f29640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7bcc0a1330 addr v1:172.21.15.42:6789/0 2024-09-16T04:06:24.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.645+0000 7f7bd2f29640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f7bcc06dac0 0x7f7bcc13a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:06:24.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.645+0000 7f7bd2f29640 10 monclient(hunting): picked mon.noname-a con 0x7f7bcc06dac0 addr v2:172.21.15.42:3300/0 2024-09-16T04:06:24.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.645+0000 7f7bd2f29640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f7bcc0a1e20 0x7f7bcc13c890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:06:24.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.645+0000 7f7bd2f29640 10 monclient(hunting): picked mon.noname-c con 0x7f7bcc0a1e20 addr v2:172.21.15.154:3300/0 2024-09-16T04:06:24.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.645+0000 7f7bd2f29640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7bcc09c090 con 0x7f7bcc0a1330 2024-09-16T04:06:24.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.645+0000 7f7bd2f29640 10 monclient(hunting): start opening mon connection 2024-09-16T04:06:24.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.645+0000 7f7bd2f29640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f7bcc09f610 con 0x7f7bcc06dac0 2024-09-16T04:06:24.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.645+0000 7f7bd2f29640 10 monclient(hunting): start opening mon connection 2024-09-16T04:06:24.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.645+0000 7f7bd2f29640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f7bcc09f8f0 con 0x7f7bcc0a1e20 2024-09-16T04:06:24.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.645+0000 7f7bd2f29640 10 monclient(hunting): _renew_subs 2024-09-16T04:06:24.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.645+0000 7f7bd2f29640 10 monclient(hunting): authenticate will time out at 2431.618815s 2024-09-16T04:06:24.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.645+0000 7f7bcbfff640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f7bcc06dac0 0x7f7bcc13a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:06:24.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.645+0000 7f7bd0cc5640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f7bcc0a1330 0x7f7bcc0a1710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:38006/0 (socket says 172.21.15.42:38006) 2024-09-16T04:06:24.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.645+0000 7f7bd0cc5640 1 -- 172.21.15.42:0/3725478130 learned_addr learned my addr 172.21.15.42:0/3725478130 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T04:06:24.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.645+0000 7f7bcbfff640 10 monclient(hunting): get_auth_request con 0x7f7bcc06dac0 auth_method 0 2024-09-16T04:06:24.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.645+0000 7f7bcbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:06:24.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.645+0000 7f7bcbfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:24.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.645+0000 7f7bcbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:24.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.645+0000 7f7bcbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:06:24.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.645+0000 7f7bcbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:06:24.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.645+0000 7f7bcbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:06:24.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.645+0000 7f7bd14c6640 1 --2- 172.21.15.42:0/3725478130 >> v2:172.21.15.154:3300/0 conn(0x7f7bcc0a1e20 0x7f7bcc13c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:06:24.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.645+0000 7f7bd14c6640 10 monclient(hunting): get_auth_request con 0x7f7bcc0a1e20 auth_method 0 2024-09-16T04:06:24.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.645+0000 7f7bd14c6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:06:24.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.645+0000 7f7bd14c6640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:24.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.645+0000 7f7bd14c6640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:24.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.645+0000 7f7bcbfff640 10 monclient(hunting): handle_auth_done global_id 984444 payload 274 2024-09-16T04:06:24.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.645+0000 7f7bcbfff640 1 -- 172.21.15.42:0/3725478130 >> v2:172.21.15.154:3300/0 conn(0x7f7bcc0a1e20 msgr2=0x7f7bcc13c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:06:24.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.645+0000 7f7bcbfff640 1 --2- 172.21.15.42:0/3725478130 >> v2:172.21.15.154:3300/0 conn(0x7f7bcc0a1e20 0x7f7bcc13c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:06:24.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.645+0000 7f7bcbfff640 1 -- 172.21.15.42:0/3725478130 >> v1:172.21.15.42:6789/0 conn(0x7f7bcc0a1330 legacy=0x7f7bcc0a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:24.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.645+0000 7f7bcbfff640 10 monclient: _finish_hunting 0 2024-09-16T04:06:24.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.645+0000 7f7bcbfff640 1 monclient: found mon.noname-a 2024-09-16T04:06:24.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.645+0000 7f7bcbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:06:24.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.645+0000 7f7bcbfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:06:24.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.645+0000 7f7bcbfff640 1 -- 172.21.15.42:0/3725478130 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7bcc09fbd0 con 0x7f7bcc06dac0 2024-09-16T04:06:24.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.645+0000 7f7bcb7fe640 1 -- 172.21.15.42:0/3725478130 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3653696827 0 0) 0x7f7bcc09c090 con 0x7f7bcc0a1330 2024-09-16T04:06:24.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.645+0000 7f7bcb7fe640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T04:06:24.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.649+0000 7f7bcbfff640 1 --2- 172.21.15.42:0/3725478130 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7bcc06dac0 0x7f7bcc13a340 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f7bcc09c800 tx=0x7f7bb401c3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a85df6ea4085d788 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:06:24.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.649+0000 7f7bcb7fe640 1 -- 172.21.15.42:0/3725478130 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bb40b0070 con 0x7f7bcc06dac0 2024-09-16T04:06:24.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.649+0000 7f7bcb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:24.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.649+0000 7f7bcb7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:06:24.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.649+0000 7f7bcb7fe640 10 monclient: dump: 2024-09-16T04:06:24.667 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:06:24.667 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:06:24.667 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:06:24.667 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:06:24.667 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:24.668 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:06:24.668 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:06:24.668 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:06:24.668 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:06:24.668 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:06:24.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.649+0000 7f7bcb7fe640 10 monclient: _finish_auth 0 2024-09-16T04:06:24.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.649+0000 7f7bcb7fe640 10 monclient: _check_auth_tickets 2024-09-16T04:06:24.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.649+0000 7f7bcb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:06:24.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.649+0000 7f7bcb7fe640 1 -- 172.21.15.42:0/3725478130 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7bb40ab600 con 0x7f7bcc06dac0 2024-09-16T04:06:24.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.649+0000 7f7bcb7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:06:24.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.649+0000 7f7bcb7fe640 1 -- 172.21.15.42:0/3725478130 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bb40a4e60 con 0x7f7bcc06dac0 2024-09-16T04:06:24.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.649+0000 7f7bd2f29640 5 monclient: authenticate success, global_id 984444 2024-09-16T04:06:24.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.649+0000 7f7bd2f29640 10 monclient: get_monmap_and_config success 2024-09-16T04:06:24.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.649+0000 7f7bd2f29640 4 set_mon_vals no callback set 2024-09-16T04:06:24.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.649+0000 7f7bcb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:24.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.649+0000 7f7bcb7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:06:24.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.649+0000 7f7bcb7fe640 10 monclient: dump: 2024-09-16T04:06:24.669 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:06:24.670 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:06:24.670 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:06:24.670 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:06:24.670 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:24.670 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:06:24.671 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:06:24.671 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:06:24.671 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:06:24.671 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:06:24.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.649+0000 7f7bd2f29640 10 monclient: shutdown 2024-09-16T04:06:24.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.649+0000 7f7bd2f29640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:06:24.672 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.649+0000 7f7bd2f29640 1 -- 172.21.15.42:0/3725478130 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7bcc06dac0 msgr2=0x7f7bcc13a340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:24.672 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.649+0000 7f7bd2f29640 1 --2- 172.21.15.42:0/3725478130 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7bcc06dac0 0x7f7bcc13a340 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f7bcc09c800 tx=0x7f7bb401c3d0 comp rx=0 tx=0).stop 2024-09-16T04:06:24.672 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.649+0000 7f7bd2f29640 1 -- 172.21.15.42:0/3725478130 shutdown_connections 2024-09-16T04:06:24.672 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.649+0000 7f7bd2f29640 1 --2- 172.21.15.42:0/3725478130 >> v2:172.21.15.154:3300/0 conn(0x7f7bcc0a1e20 0x7f7bcc13c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:06:24.672 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.649+0000 7f7bd2f29640 1 --2- 172.21.15.42:0/3725478130 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7bcc06dac0 0x7f7bcc13a340 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:06:24.672 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.649+0000 7f7bd2f29640 1 -- 172.21.15.42:0/3725478130 >> 172.21.15.42:0/3725478130 conn(0x7f7bcc0a2420 msgr2=0x7f7bcc0a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:06:24.672 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.649+0000 7f7bd2f29640 1 -- 172.21.15.42:0/3725478130 shutdown_connections 2024-09-16T04:06:24.672 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.649+0000 7f7bd2f29640 1 -- 172.21.15.42:0/3725478130 wait complete. 2024-09-16T04:06:24.672 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.649+0000 7f7bd2f29640 10 monclient: build_initial_monmap 2024-09-16T04:06:24.672 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.649+0000 7f7bd2f29640 10 monclient: monmap: 2024-09-16T04:06:24.672 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:06:24.672 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:06:24.672 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:06:24.673 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:06:24.673 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:06:24.673 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:06:24.673 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:06:24.673 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:06:24.673 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:06:24.673 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:06:24.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.649+0000 7f7bd2f29640 1 librados: starting msgr at 2024-09-16T04:06:24.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.649+0000 7f7bd2f29640 1 librados: starting objecter 2024-09-16T04:06:24.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.649+0000 7f7bd2f29640 1 Processor -- start 2024-09-16T04:06:24.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.649+0000 7f7bd2f29640 1 -- start start 2024-09-16T04:06:24.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.649+0000 7f7bd2f29640 1 librados: setting wanted keys 2024-09-16T04:06:24.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.649+0000 7f7bd2f29640 1 librados: calling monclient init 2024-09-16T04:06:24.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.649+0000 7f7bd2f29640 10 monclient: init 2024-09-16T04:06:24.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.653+0000 7f7bd2f29640 10 monclient: _reopen_session rank -1 2024-09-16T04:06:24.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.653+0000 7f7bd2f29640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:06:24.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.653+0000 7f7bd2f29640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7bcc0a1330 0x7f7bcc134bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:06:24.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.653+0000 7f7bd2f29640 10 monclient(hunting): picked mon.noname-b con 0x7f7bcc0a1330 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:06:24.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.653+0000 7f7bd2f29640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7bcc0a1e20 0x7f7bcc1350f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:06:24.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.653+0000 7f7bd2f29640 10 monclient(hunting): picked mon.noname-c con 0x7f7bcc0a1e20 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:06:24.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.653+0000 7f7bd2f29640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7bcc11bf70 0x7f7bcc11c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:06:24.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.653+0000 7f7bd2f29640 10 monclient(hunting): picked mon.noname-a con 0x7f7bcc11bf70 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:06:24.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.653+0000 7f7bd2f29640 10 monclient(hunting): start opening mon connection 2024-09-16T04:06:24.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.653+0000 7f7bd2f29640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f7bcc13f800 con 0x7f7bcc11bf70 2024-09-16T04:06:24.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.653+0000 7f7bd2f29640 10 monclient(hunting): start opening mon connection 2024-09-16T04:06:24.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.653+0000 7f7bd2f29640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f7bcc09ef80 con 0x7f7bcc0a1330 2024-09-16T04:06:24.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.653+0000 7f7bd2f29640 10 monclient(hunting): start opening mon connection 2024-09-16T04:06:24.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.653+0000 7f7bd2f29640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f7bcc13f980 con 0x7f7bcc0a1e20 2024-09-16T04:06:24.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.653+0000 7f7bd2f29640 10 monclient(hunting): _renew_subs 2024-09-16T04:06:24.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.653+0000 7f7bd2f29640 10 monclient(hunting): authenticate will time out at 2431.624236s 2024-09-16T04:06:24.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.653+0000 7f7bcbfff640 1 -- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7bcc0a1e20 msgr2=0x7f7bcc1350f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T04:06:24.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.653+0000 7f7bcbfff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7bcc0a1e20 0x7f7bcc1350f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:06:24.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.653+0000 7f7bd14c6640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7bcc11bf70 0x7f7bcc11c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:06:24.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.653+0000 7f7bd14c6640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7bcc11bf70 0x7f7bcc11c3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:47396/0 (socket says 172.21.15.42:47396) 2024-09-16T04:06:24.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.653+0000 7f7bd14c6640 1 -- 172.21.15.42:0/3660079456 learned_addr learned my addr 172.21.15.42:0/3660079456 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:06:24.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.653+0000 7f7bd14c6640 10 monclient(hunting): get_auth_request con 0x7f7bcc11bf70 auth_method 0 2024-09-16T04:06:24.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.653+0000 7f7bd14c6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:06:24.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.653+0000 7f7bd14c6640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:24.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.653+0000 7f7bd14c6640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:24.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.653+0000 7f7bd14c6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:06:24.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.653+0000 7f7bd14c6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:06:24.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.653+0000 7f7bd14c6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:06:24.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.653+0000 7f7bd0cc5640 1 --2- 172.21.15.42:0/3660079456 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7bcc0a1330 0x7f7bcc134bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:06:24.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.653+0000 7f7bd0cc5640 10 monclient(hunting): get_auth_request con 0x7f7bcc0a1330 auth_method 0 2024-09-16T04:06:24.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.653+0000 7f7bd0cc5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:06:24.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.653+0000 7f7bd0cc5640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:24.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.653+0000 7f7bd0cc5640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:24.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.653+0000 7f7bd14c6640 10 monclient(hunting): handle_auth_done global_id 984450 payload 824 2024-09-16T04:06:24.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.653+0000 7f7bd14c6640 1 -- 172.21.15.42:0/3660079456 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7bcc0a1e20 msgr2=0x7f7bcc1350f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:06:24.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.653+0000 7f7bd14c6640 1 --2- 172.21.15.42:0/3660079456 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7bcc0a1e20 0x7f7bcc1350f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:06:24.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.653+0000 7f7bd14c6640 1 -- 172.21.15.42:0/3660079456 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7bcc0a1330 msgr2=0x7f7bcc134bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:24.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.653+0000 7f7bd14c6640 1 --2- 172.21.15.42:0/3660079456 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7bcc0a1330 0x7f7bcc134bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:06:24.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.653+0000 7f7bd14c6640 10 monclient: _finish_hunting 0 2024-09-16T04:06:24.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.653+0000 7f7bd14c6640 1 monclient: found mon.noname-a 2024-09-16T04:06:24.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.653+0000 7f7bd14c6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:06:24.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.653+0000 7f7bd14c6640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:06:24.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.653+0000 7f7bd14c6640 1 -- 172.21.15.42:0/3660079456 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7bcc11fa70 con 0x7f7bcc11bf70 2024-09-16T04:06:24.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.653+0000 7f7bd14c6640 1 --2- 172.21.15.42:0/3660079456 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7bcc11bf70 0x7f7bcc11c3d0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f7bbc00d830 tx=0x7f7bbc00dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:06:24.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.653+0000 7f7bc97fa640 10 client.?.objecter ms_handle_connect 0x7f7bcc11bf70 2024-09-16T04:06:24.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.653+0000 7f7bc97fa640 10 client.?.objecter resend_mon_ops 2024-09-16T04:06:24.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.653+0000 7f7bc97fa640 1 -- 172.21.15.42:0/3660079456 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bbc00b660 con 0x7f7bcc11bf70 2024-09-16T04:06:24.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.653+0000 7f7bc97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:24.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.653+0000 7f7bc97fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:06:24.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.653+0000 7f7bc97fa640 10 monclient: dump: 2024-09-16T04:06:24.677 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:06:24.678 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:06:24.678 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:06:24.678 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:06:24.678 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:24.678 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:06:24.678 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:06:24.678 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:06:24.678 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:06:24.678 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:06:24.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.653+0000 7f7bc97fa640 10 monclient: _finish_auth 0 2024-09-16T04:06:24.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.653+0000 7f7bc97fa640 10 monclient: _check_auth_tickets 2024-09-16T04:06:24.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.653+0000 7f7bc97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:06:24.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.653+0000 7f7bd2f29640 5 monclient: authenticate success, global_id 984450 2024-09-16T04:06:24.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.653+0000 7f7bc97fa640 1 -- 172.21.15.42:0/3660079456 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7bbc019070 con 0x7f7bcc11bf70 2024-09-16T04:06:24.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.653+0000 7f7bc97fa640 10 monclient: handle_config config(0 keys) 2024-09-16T04:06:24.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.653+0000 7f7bd2f29640 10 monclient: _renew_subs 2024-09-16T04:06:24.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.653+0000 7f7bc97fa640 1 -- 172.21.15.42:0/3660079456 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bbc00ba40 con 0x7f7bcc11bf70 2024-09-16T04:06:24.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.653+0000 7f7bd2f29640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:06:24.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.653+0000 7f7bd2f29640 1 -- 172.21.15.42:0/3660079456 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7bcc073a30 con 0x7f7bcc11bf70 2024-09-16T04:06:24.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.653+0000 7f7bc97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:24.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.653+0000 7f7bc97fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:06:24.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.653+0000 7f7bcaffd640 4 set_mon_vals no callback set 2024-09-16T04:06:24.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.653+0000 7f7bc97fa640 10 monclient: dump: 2024-09-16T04:06:24.679 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:06:24.680 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:06:24.680 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:06:24.680 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:06:24.680 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:24.680 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:06:24.680 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:06:24.680 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:06:24.680 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:06:24.680 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:06:24.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.653+0000 7f7bd2f29640 10 client.984450.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:06:24.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.653+0000 7f7bd2f29640 10 monclient: _renew_subs 2024-09-16T04:06:24.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.653+0000 7f7bd2f29640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:06:24.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.653+0000 7f7bd2f29640 1 -- 172.21.15.42:0/3660079456 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7bcc11f4b0 con 0x7f7bcc11bf70 2024-09-16T04:06:24.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.653+0000 7f7bd2f29640 1 librados: init done 2024-09-16T04:06:24.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.653+0000 7f7bd2f29640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:06:24.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.653+0000 7f7bd2f29640 1 --2- 172.21.15.42:0/3660079456 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7bcc06dac0 0x7f7bcc11c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:06:24.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.653+0000 7f7bd2f29640 10 monclient(hunting): start opening mon connection 2024-09-16T04:06:24.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.653+0000 7f7bd2f29640 1 -- 172.21.15.42:0/3660079456 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f7bcc0a1710 con 0x7f7bcc06dac0 2024-09-16T04:06:24.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.653+0000 7f7bc97fa640 1 -- 172.21.15.42:0/3660079456 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f7bbc01e460 con 0x7f7bcc11bf70 2024-09-16T04:06:24.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.653+0000 7f7bd0cc5640 1 --2- 172.21.15.42:0/3660079456 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7bcc06dac0 0x7f7bcc11c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:06:24.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.653+0000 7f7bd0cc5640 10 monclient: get_auth_request con 0x7f7bcc06dac0 auth_method 0 2024-09-16T04:06:24.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.653+0000 7f7bd0cc5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:06:24.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.653+0000 7f7bd0cc5640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:24.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.653+0000 7f7bd0cc5640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:24.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.653+0000 7f7bc97fa640 1 --2- 172.21.15.42:0/3660079456 >> v2:172.21.15.154:6812/254845342 conn(0x7f7bac0377c0 0x7f7bac039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:06:24.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.653+0000 7f7bd0cc5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:06:24.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.653+0000 7f7bd0cc5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:06:24.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.653+0000 7f7bc97fa640 1 -- 172.21.15.42:0/3660079456 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(706..706 src has 451..706) ==== 5040+0+0 (secure 0 0 0) 0x7f7bbc0493a0 con 0x7f7bcc11bf70 2024-09-16T04:06:24.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.653+0000 7f7bc97fa640 10 client.984450.objecter ms_dispatch 0x7f7bcc000ba0 osd_map(706..706 src has 451..706) 2024-09-16T04:06:24.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.653+0000 7f7bc97fa640 3 client.984450.objecter handle_osd_map got epochs [706,706] > 0 2024-09-16T04:06:24.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.653+0000 7f7bc97fa640 3 client.984450.objecter handle_osd_map decoding full epoch 706 2024-09-16T04:06:24.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.653+0000 7f7bd0cc5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:06:24.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.653+0000 7f7bc97fa640 20 client.984450.objecter dump_active .. 0 homeless 2024-09-16T04:06:24.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.653+0000 7f7bcbfff640 1 --2- 172.21.15.42:0/3660079456 >> v2:172.21.15.154:6812/254845342 conn(0x7f7bac0377c0 0x7f7bac039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:06:24.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.653+0000 7f7bcbfff640 10 monclient: get_auth_request con 0x7f7bac0377c0 auth_method 0 2024-09-16T04:06:24.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.653+0000 7f7bd0cc5640 10 monclient(hunting): handle_auth_done global_id 905020 payload 824 2024-09-16T04:06:24.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.653+0000 7f7bd0cc5640 1 -- 172.21.15.42:0/3660079456 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7bcc11e060 con 0x7f7bcc06dac0 2024-09-16T04:06:24.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.653+0000 7f7bd0cc5640 1 --2- 172.21.15.42:0/3660079456 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7bcc06dac0 0x7f7bcc11c910 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f7bc4001570 tx=0x7f7bc4002070 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:06:24.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.653+0000 7f7bc97fa640 10 client.984450.objecter ms_handle_connect 0x7f7bcc06dac0 2024-09-16T04:06:24.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.653+0000 7f7bc97fa640 10 client.984450.objecter resend_mon_ops 2024-09-16T04:06:24.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.653+0000 7f7bc97fa640 10 client.984450.objecter ms_handle_connect 0x7f7bac0377c0 2024-09-16T04:06:24.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.653+0000 7f7bcbfff640 1 --2- 172.21.15.42:0/3660079456 >> v2:172.21.15.154:6812/254845342 conn(0x7f7bac0377c0 0x7f7bac039c80 secure :-1 s=READY pgs=3868 cs=0 l=1 rev1=1 crypto rx=0x7f7bb40ad000 tx=0x7f7bb4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:06:24.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.653+0000 7f7bc97fa640 1 -- 172.21.15.42:0/3660079456 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bc4004d80 con 0x7f7bcc06dac0 2024-09-16T04:06:24.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.653+0000 7f7bc97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:24.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.653+0000 7f7bc97fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:06:24.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.653+0000 7f7bc97fa640 10 monclient: dump: 2024-09-16T04:06:24.683 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:06:24.683 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:06:24.683 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:06:24.683 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:06:24.683 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:24.683 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:06:24.683 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:06:24.683 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:06:24.684 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:06:24.684 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:06:24.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.657+0000 7f7bc97fa640 1 -- 172.21.15.42:0/3660079456 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f7bc4004d80 con 0x7f7bcc06dac0 2024-09-16T04:06:24.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.657+0000 7f7bc97fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:06:24.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.657+0000 7f7bc97fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:06:24.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.657+0000 7f7bc97fa640 1 -- 172.21.15.42:0/3660079456 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7bcc06dac0 msgr2=0x7f7bcc11c910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:24.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.657+0000 7f7bc97fa640 1 --2- 172.21.15.42:0/3660079456 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7bcc06dac0 0x7f7bcc11c910 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f7bc4001570 tx=0x7f7bc4002070 comp rx=0 tx=0).stop 2024-09-16T04:06:24.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.657+0000 7f7bc97fa640 1 -- 172.21.15.42:0/3660079456 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7bcc06dac0 msgr2=0x7f7bcc11c910 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:06:24.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.657+0000 7f7bc97fa640 1 --2- 172.21.15.42:0/3660079456 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7bcc06dac0 0x7f7bcc11c910 secure :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f7bc4001570 tx=0x7f7bc4002070 comp rx=0 tx=0).stop 2024-09-16T04:06:24.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.661+0000 7f7bd2f29640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:06:24.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.661+0000 7f7bd2f29640 1 --2- 172.21.15.42:0/3660079456 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7bcc11f680 0x7f7bcc11ce50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:06:24.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.661+0000 7f7bd2f29640 10 monclient(hunting): start opening mon connection 2024-09-16T04:06:24.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.661+0000 7f7bd2f29640 1 -- 172.21.15.42:0/3660079456 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f7bcc11d390 con 0x7f7bcc11f680 2024-09-16T04:06:24.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.661+0000 7f7bd0cc5640 1 --2- 172.21.15.42:0/3660079456 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7bcc11f680 0x7f7bcc11ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:06:24.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.661+0000 7f7bd0cc5640 10 monclient: get_auth_request con 0x7f7bcc11f680 auth_method 0 2024-09-16T04:06:24.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.661+0000 7f7bd0cc5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:06:24.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.661+0000 7f7bd0cc5640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:24.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.661+0000 7f7bd0cc5640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:24.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.661+0000 7f7bd0cc5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:06:24.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.661+0000 7f7bd0cc5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:06:24.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.661+0000 7f7bd0cc5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:06:24.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.661+0000 7f7bd0cc5640 10 monclient(hunting): handle_auth_done global_id 905023 payload 824 2024-09-16T04:06:24.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.661+0000 7f7bd0cc5640 1 -- 172.21.15.42:0/3660079456 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f7bcc135630 con 0x7f7bcc11f680 2024-09-16T04:06:24.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.661+0000 7f7bd0cc5640 1 --2- 172.21.15.42:0/3660079456 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7bcc11f680 0x7f7bcc11ce50 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f7bc4015000 tx=0x7f7bc4001410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:06:24.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.661+0000 7f7bc97fa640 10 client.984450.objecter ms_handle_connect 0x7f7bcc11f680 2024-09-16T04:06:24.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.661+0000 7f7bc97fa640 10 client.984450.objecter resend_mon_ops 2024-09-16T04:06:24.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.665+0000 7f7bc97fa640 1 -- 172.21.15.42:0/3660079456 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bc401d110 con 0x7f7bcc11f680 2024-09-16T04:06:24.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.665+0000 7f7bc97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:06:24.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.665+0000 7f7bc97fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:06:24.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.665+0000 7f7bc97fa640 10 monclient: dump: 2024-09-16T04:06:24.688 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:06:24.688 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:06:24.688 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:06:24.688 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:06:24.688 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:06:24.688 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:06:24.688 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:06:24.688 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:06:24.688 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:06:24.688 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:06:24.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.665+0000 7f7bc97fa640 1 -- 172.21.15.42:0/3660079456 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4869 (secure 0 0 0) 0x7f7bc401d110 con 0x7f7bcc11f680 2024-09-16T04:06:24.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.665+0000 7f7bc97fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:06:24.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.665+0000 7f7bc97fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:06:24.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.665+0000 7f7bc97fa640 1 -- 172.21.15.42:0/3660079456 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7bcc11f680 msgr2=0x7f7bcc11ce50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:24.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.665+0000 7f7bc97fa640 1 --2- 172.21.15.42:0/3660079456 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7bcc11f680 0x7f7bcc11ce50 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f7bc4015000 tx=0x7f7bc4001410 comp rx=0 tx=0).stop 2024-09-16T04:06:24.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.665+0000 7f7bc97fa640 1 -- 172.21.15.42:0/3660079456 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7bcc11f680 msgr2=0x7f7bcc11ce50 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:06:24.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.665+0000 7f7bc97fa640 1 --2- 172.21.15.42:0/3660079456 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7bcc11f680 0x7f7bcc11ce50 secure :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f7bc4015000 tx=0x7f7bc4001410 comp rx=0 tx=0).stop 2024-09-16T04:06:24.689 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T04:06:24.689 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:06:24.690 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:06:24.690 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T04:06:24.690 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 226, 2024-09-16T04:06:24.690 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 512843, 2024-09-16T04:06:24.690 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T04:06:24.690 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T04:06:24.690 INFO:teuthology.orchestra.run.smithi042.stdout: 1 2024-09-16T04:06:24.690 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:06:24.690 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 0, 2024-09-16T04:06:24.690 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:06:24.690 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T04:06:24.690 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T04:06:24.690 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:06:24.690 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:06:24.691 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:06:24.691 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:06:24.691 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:06:24.691 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:06:24.691 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:06:24.691 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:06:24.691 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:06:24.691 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:06:24.691 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:06:24.691 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:06:24.691 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:06:24.691 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T04:06:24.691 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:06:24.692 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:06:24.692 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:06:24.692 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:06:24.692 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:06:24.692 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:06:24.692 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:06:24.692 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:06:24.692 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:06:24.692 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:06:24.692 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:06:24.692 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:06:24.692 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:06:24.692 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T04:06:24.693 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T04:06:24.693 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T04:06:24.693 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T04:06:24.693 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T04:06:24.693 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T04:06:24.693 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:06:24.693 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:06:24.694 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T04:06:24.694 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T04:06:24.694 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T04:06:24.694 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T04:06:24.694 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T04:06:24.694 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T04:06:24.694 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T04:06:24.694 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:06:24.694 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T04:06:24.694 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:06:24.694 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:06:24.694 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:06:24.695 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:06:24.695 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:06:24.695 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:06:24.695 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:06:24.695 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:06:24.695 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:06:24.695 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:06:24.696 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:06:24.696 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:06:24.696 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T04:06:24.696 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:06:24.696 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T04:06:24.697 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:06:24.697 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:06:24.697 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:06:24.697 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:06:24.697 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:06:24.697 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:06:24.697 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:06:24.698 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T04:06:24.698 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:06:24.698 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:06:24.698 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:06:24.698 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:06:24.698 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T04:06:24.698 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:06:24.698 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:06:24.698 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:06:24.698 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:06:24.698 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T04:06:24.698 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T04:06:24.698 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:06:24.698 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:06:24.699 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:06:24.699 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:06:24.699 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:06:24.699 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:06:24.699 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:06:24.699 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:06:24.699 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:06:24.699 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:06:24.699 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:06:24.699 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T04:06:24.699 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:06:24.699 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:06:24.699 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:06:24.699 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:06:24.700 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T04:06:24.700 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:06:24.700 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:06:24.700 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:06:24.700 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:06:24.700 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T04:06:24.700 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T04:06:24.700 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:06:24.700 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:06:24.700 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:06:24.700 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:06:24.700 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:06:24.700 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:06:24.701 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:06:24.701 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:06:24.701 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:06:24.701 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:06:24.701 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:06:24.701 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T04:06:24.701 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:06:24.701 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:06:24.701 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:06:24.701 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:06:24.701 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T04:06:24.702 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:06:24.702 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:06:24.702 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:06:24.702 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:06:24.702 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T04:06:24.702 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T04:06:24.702 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:06:24.702 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:06:24.702 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:06:24.702 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:06:24.702 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:06:24.702 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:06:24.703 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T04:06:24.703 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T04:06:24.703 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:06:24.703 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:06:24.703 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:06:24.703 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:06:24.703 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:06:24.703 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:06:24.703 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T04:06:24.703 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:06:24.703 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:06:24.703 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:06:24.704 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 3 2024-09-16T04:06:24.704 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:06:24.704 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:06:24.704 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T04:06:24.704 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:06:24.705 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:06:24.705 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:06:24.705 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 5 2024-09-16T04:06:24.705 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:06:24.705 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:06:24.705 INFO:teuthology.orchestra.run.smithi042.stdout: "mgr": [ 2024-09-16T04:06:24.705 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:06:24.706 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:06:24.706 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:06:24.706 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:06:24.706 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:06:24.706 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:06:24.706 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:06:24.706 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T04:06:24.706 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T04:06:24.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.665+0000 7f7bd2f29640 10 librados: watch_flush enter 2024-09-16T04:06:24.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.665+0000 7f7bd2f29640 10 librados: watch_flush exit 2024-09-16T04:06:24.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.665+0000 7f7bd2f29640 20 client.984450.objecter shutdown clearing up homeless session... 2024-09-16T04:06:24.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.665+0000 7f7bd2f29640 10 client.984450.objecter successfully canceled tick 2024-09-16T04:06:24.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.665+0000 7f7bd2f29640 1 -- 172.21.15.42:0/3660079456 >> v2:172.21.15.154:6812/254845342 conn(0x7f7bac0377c0 msgr2=0x7f7bac039c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:24.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.665+0000 7f7bd2f29640 1 --2- 172.21.15.42:0/3660079456 >> v2:172.21.15.154:6812/254845342 conn(0x7f7bac0377c0 0x7f7bac039c80 secure :-1 s=READY pgs=3868 cs=0 l=1 rev1=1 crypto rx=0x7f7bb40ad000 tx=0x7f7bb4008040 comp rx=0 tx=0).stop 2024-09-16T04:06:24.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.665+0000 7f7bd2f29640 10 monclient: shutdown 2024-09-16T04:06:24.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.665+0000 7f7bd2f29640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:06:24.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.665+0000 7f7bd2f29640 1 -- 172.21.15.42:0/3660079456 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7bcc11bf70 msgr2=0x7f7bcc11c3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:06:24.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.665+0000 7f7bd2f29640 1 --2- 172.21.15.42:0/3660079456 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7bcc11bf70 0x7f7bcc11c3d0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f7bbc00d830 tx=0x7f7bbc00dcf0 comp rx=0 tx=0).stop 2024-09-16T04:06:24.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.665+0000 7f7bd14c6640 1 -- 172.21.15.42:0/3660079456 reap_dead start 2024-09-16T04:06:24.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.665+0000 7f7bd14c6640 1 -- 172.21.15.42:0/3660079456 reap_dead start 2024-09-16T04:06:24.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.665+0000 7f7bd2f29640 1 -- 172.21.15.42:0/3660079456 shutdown_connections 2024-09-16T04:06:24.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.665+0000 7f7bd2f29640 1 -- 172.21.15.42:0/3660079456 >> 172.21.15.42:0/3660079456 conn(0x7f7bcc0a2420 msgr2=0x7f7bcc0a45b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:06:24.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.665+0000 7f7bd2f29640 1 -- 172.21.15.42:0/3660079456 shutdown_connections 2024-09-16T04:06:24.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.665+0000 7f7bd2f29640 1 -- 172.21.15.42:0/3660079456 wait complete. 2024-09-16T04:06:24.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:24.665+0000 7f7bd2f29640 1 librados: shutdown 2024-09-16T04:06:24.710 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T04:06:24.710 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-16T04:06:44.711 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-16T04:06:44.711 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-09-16T04:06:44.711 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-16T04:06:44.711 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-16T04:06:44.714 INFO:tasks.ceph.mon.c:Started 2024-09-16T04:06:44.714 INFO:tasks.mon_thrash:Sending STOP to mon a 2024-09-16T04:06:44.715 INFO:tasks.ceph.mon.a:Sent signal 19 2024-09-16T04:06:44.715 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-16T04:06:59.715 INFO:tasks.mon_thrash:Sending CONT to mon a 2024-09-16T04:06:59.716 INFO:tasks.ceph.mon.a:Sent signal 18 2024-09-16T04:06:59.716 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T04:06:59.716 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:06:59.847 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:59.841+0000 7f57868ff640 10 monclient: get_monmap_and_config 2024-09-16T04:06:59.847 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:59.841+0000 7f57868ff640 10 monclient: build_initial_monmap 2024-09-16T04:06:59.847 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:59.841+0000 7f57868ff640 10 monclient: monmap: 2024-09-16T04:06:59.847 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:06:59.848 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:06:59.848 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:06:59.845875+0000 2024-09-16T04:06:59.848 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:06:59.845875+0000 2024-09-16T04:06:59.848 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:06:59.848 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:06:59.848 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:06:59.848 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:06:59.848 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:06:59.848 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:06:59.848 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:06:59.848 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:06:59.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:59.841+0000 7f57868ff640 1 Processor -- start 2024-09-16T04:06:59.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:59.841+0000 7f57868ff640 1 -- start start 2024-09-16T04:06:59.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:59.841+0000 7f57868ff640 10 monclient: init 2024-09-16T04:06:59.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:59.841+0000 7f57868ff640 10 monclient: _reopen_session rank -1 2024-09-16T04:06:59.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:59.841+0000 7f57868ff640 10 monclient: _add_conns ranks=[3,4,1,0,2] 2024-09-16T04:06:59.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:59.841+0000 7f57868ff640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f57800a39b0 0x7f5780133b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:06:59.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:59.841+0000 7f57868ff640 10 monclient(hunting): picked mon.noname-a con 0x7f57800a39b0 addr v2:172.21.15.42:3300/0 2024-09-16T04:06:59.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:59.845+0000 7f57868ff640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f57800a3d90 0x7f5780137000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:06:59.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:59.845+0000 7f57868ff640 10 monclient(hunting): picked mon.noname-c con 0x7f57800a3d90 addr v2:172.21.15.154:3300/0 2024-09-16T04:06:59.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:59.845+0000 7f57868ff640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5780137540 addr v1:172.21.15.154:6789/0 2024-09-16T04:06:59.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:59.845+0000 7f57868ff640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f578009c140 con 0x7f5780137540 2024-09-16T04:06:59.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:59.845+0000 7f57868ff640 10 monclient(hunting): start opening mon connection 2024-09-16T04:06:59.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:59.845+0000 7f57868ff640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f578009c420 con 0x7f57800a39b0 2024-09-16T04:06:59.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:59.845+0000 7f57868ff640 10 monclient(hunting): start opening mon connection 2024-09-16T04:06:59.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:59.845+0000 7f57868ff640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f578006dd60 con 0x7f57800a3d90 2024-09-16T04:06:59.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:59.845+0000 7f57868ff640 10 monclient(hunting): _renew_subs 2024-09-16T04:06:59.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:59.845+0000 7f57868ff640 10 monclient(hunting): authenticate will time out at 2466.816075s 2024-09-16T04:06:59.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:59.845+0000 7f577ffff640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f57800a39b0 0x7f5780133b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:06:59.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:59.845+0000 7f577ffff640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f57800a39b0 0x7f5780133b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:33296/0 (socket says 172.21.15.42:33296) 2024-09-16T04:06:59.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:59.845+0000 7f577ffff640 1 -- 172.21.15.42:0/483821658 learned_addr learned my addr 172.21.15.42:0/483821658 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:06:59.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:59.845+0000 7f577ffff640 10 monclient(hunting): get_auth_request con 0x7f57800a39b0 auth_method 0 2024-09-16T04:06:59.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:59.845+0000 7f577ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:06:59.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:59.845+0000 7f577ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:59.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:59.845+0000 7f577ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:59.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:59.845+0000 7f577ffff640 1 -- 172.21.15.42:0/483821658 >> v2:172.21.15.42:3300/0 conn(0x7f57800a39b0 msgr2=0x7f5780133b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:06:59.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:59.845+0000 7f577ffff640 1 -- 172.21.15.42:0/483821658 >> v2:172.21.15.42:3300/0 conn(0x7f57800a39b0 msgr2=0x7f5780133b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:06:59.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:59.845+0000 7f577ffff640 1 --2- 172.21.15.42:0/483821658 >> v2:172.21.15.42:3300/0 conn(0x7f57800a39b0 0x7f5780133b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:06:59.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:59.845+0000 7f577ffff640 1 --2- 172.21.15.42:0/483821658 >> v2:172.21.15.42:3300/0 conn(0x7f57800a39b0 0x7f5780133b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:06:59.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:59.845+0000 7f57777fe640 1 --2- 172.21.15.42:0/483821658 >> v2:172.21.15.154:3300/0 conn(0x7f57800a3d90 0x7f5780137000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:06:59.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:59.845+0000 7f57777fe640 10 monclient(hunting): get_auth_request con 0x7f57800a3d90 auth_method 0 2024-09-16T04:06:59.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:59.845+0000 7f57777fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:06:59.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:59.845+0000 7f57777fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:06:59.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:59.845+0000 7f57777fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:06:59.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:59.845+0000 7f57777fe640 1 -- 172.21.15.42:0/483821658 >> v2:172.21.15.154:3300/0 conn(0x7f57800a3d90 msgr2=0x7f5780137000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:06:59.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:59.845+0000 7f57777fe640 1 -- 172.21.15.42:0/483821658 >> v2:172.21.15.154:3300/0 conn(0x7f57800a3d90 msgr2=0x7f5780137000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:06:59.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:59.845+0000 7f57777fe640 1 --2- 172.21.15.42:0/483821658 >> v2:172.21.15.154:3300/0 conn(0x7f57800a3d90 0x7f5780137000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:06:59.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:06:59.845+0000 7f57777fe640 1 --2- 172.21.15.42:0/483821658 >> v2:172.21.15.154:3300/0 conn(0x7f57800a3d90 0x7f5780137000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:07:00.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:00.049+0000 7f577ffff640 1 --2- 172.21.15.42:0/483821658 >> v2:172.21.15.42:3300/0 conn(0x7f57800a39b0 0x7f5780133b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:00.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:00.049+0000 7f577ffff640 10 monclient(hunting): get_auth_request con 0x7f57800a39b0 auth_method 0 2024-09-16T04:07:00.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:00.049+0000 7f577ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:00.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:00.049+0000 7f577ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:00.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:00.049+0000 7f577ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:07:00.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:00.049+0000 7f57777fe640 1 --2- 172.21.15.42:0/483821658 >> v2:172.21.15.154:3300/0 conn(0x7f57800a3d90 0x7f5780137000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:00.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:00.049+0000 7f577ffff640 1 -- 172.21.15.42:0/483821658 >> v2:172.21.15.42:3300/0 conn(0x7f57800a39b0 msgr2=0x7f5780133b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:07:00.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:00.049+0000 7f577ffff640 1 -- 172.21.15.42:0/483821658 >> v2:172.21.15.42:3300/0 conn(0x7f57800a39b0 msgr2=0x7f5780133b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:07:00.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:00.049+0000 7f577ffff640 1 --2- 172.21.15.42:0/483821658 >> v2:172.21.15.42:3300/0 conn(0x7f57800a39b0 0x7f5780133b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:07:00.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:00.049+0000 7f57777fe640 10 monclient(hunting): get_auth_request con 0x7f57800a3d90 auth_method 0 2024-09-16T04:07:00.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:00.049+0000 7f57777fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:00.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:00.049+0000 7f57777fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:00.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:00.049+0000 7f57777fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:07:00.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:00.049+0000 7f577ffff640 1 --2- 172.21.15.42:0/483821658 >> v2:172.21.15.42:3300/0 conn(0x7f57800a39b0 0x7f5780133b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T04:07:00.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:00.049+0000 7f57777fe640 1 -- 172.21.15.42:0/483821658 >> v2:172.21.15.154:3300/0 conn(0x7f57800a3d90 msgr2=0x7f5780137000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:07:00.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:00.049+0000 7f57777fe640 1 -- 172.21.15.42:0/483821658 >> v2:172.21.15.154:3300/0 conn(0x7f57800a3d90 msgr2=0x7f5780137000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:07:00.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:00.049+0000 7f57777fe640 1 --2- 172.21.15.42:0/483821658 >> v2:172.21.15.154:3300/0 conn(0x7f57800a3d90 0x7f5780137000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:07:00.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:00.049+0000 7f57777fe640 1 --2- 172.21.15.42:0/483821658 >> v2:172.21.15.154:3300/0 conn(0x7f57800a3d90 0x7f5780137000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T04:07:00.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:00.453+0000 7f577ffff640 1 --2- 172.21.15.42:0/483821658 >> v2:172.21.15.42:3300/0 conn(0x7f57800a39b0 0x7f5780133b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:00.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:00.453+0000 7f577ffff640 10 monclient(hunting): get_auth_request con 0x7f57800a39b0 auth_method 0 2024-09-16T04:07:00.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:00.453+0000 7f577ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:00.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:00.453+0000 7f577ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:00.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:00.453+0000 7f577ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:07:00.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:00.453+0000 7f57777fe640 1 --2- 172.21.15.42:0/483821658 >> v2:172.21.15.154:3300/0 conn(0x7f57800a3d90 0x7f5780137000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:00.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:00.453+0000 7f57777fe640 10 monclient(hunting): get_auth_request con 0x7f57800a3d90 auth_method 0 2024-09-16T04:07:00.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:00.453+0000 7f57777fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:00.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:00.453+0000 7f57777fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:00.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:00.453+0000 7f57777fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:07:00.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:00.453+0000 7f577ffff640 1 -- 172.21.15.42:0/483821658 >> v2:172.21.15.42:3300/0 conn(0x7f57800a39b0 msgr2=0x7f5780133b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:07:00.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:00.453+0000 7f577ffff640 1 -- 172.21.15.42:0/483821658 >> v2:172.21.15.42:3300/0 conn(0x7f57800a39b0 msgr2=0x7f5780133b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:07:00.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:00.453+0000 7f577ffff640 1 --2- 172.21.15.42:0/483821658 >> v2:172.21.15.42:3300/0 conn(0x7f57800a39b0 0x7f5780133b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:07:00.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:00.453+0000 7f577ffff640 1 --2- 172.21.15.42:0/483821658 >> v2:172.21.15.42:3300/0 conn(0x7f57800a39b0 0x7f5780133b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T04:07:00.460 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:00.453+0000 7f57777fe640 1 -- 172.21.15.42:0/483821658 >> v2:172.21.15.154:3300/0 conn(0x7f57800a3d90 msgr2=0x7f5780137000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:07:00.460 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:00.453+0000 7f57777fe640 1 -- 172.21.15.42:0/483821658 >> v2:172.21.15.154:3300/0 conn(0x7f57800a3d90 msgr2=0x7f5780137000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:07:00.460 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:00.453+0000 7f57777fe640 1 --2- 172.21.15.42:0/483821658 >> v2:172.21.15.154:3300/0 conn(0x7f57800a3d90 0x7f5780137000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:07:00.460 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:00.453+0000 7f57777fe640 1 --2- 172.21.15.42:0/483821658 >> v2:172.21.15.154:3300/0 conn(0x7f57800a3d90 0x7f5780137000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T04:07:01.260 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:01.257+0000 7f577ffff640 1 --2- 172.21.15.42:0/483821658 >> v2:172.21.15.42:3300/0 conn(0x7f57800a39b0 0x7f5780133b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:01.260 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:01.257+0000 7f577ffff640 10 monclient(hunting): get_auth_request con 0x7f57800a39b0 auth_method 0 2024-09-16T04:07:01.260 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:01.257+0000 7f577ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:01.261 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:01.257+0000 7f577ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:01.261 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:01.257+0000 7f577ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:07:01.261 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:01.257+0000 7f577ffff640 1 -- 172.21.15.42:0/483821658 >> v2:172.21.15.42:3300/0 conn(0x7f57800a39b0 msgr2=0x7f5780133b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:07:01.261 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:01.257+0000 7f577ffff640 1 -- 172.21.15.42:0/483821658 >> v2:172.21.15.42:3300/0 conn(0x7f57800a39b0 msgr2=0x7f5780133b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:07:01.261 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:01.257+0000 7f577ffff640 1 --2- 172.21.15.42:0/483821658 >> v2:172.21.15.42:3300/0 conn(0x7f57800a39b0 0x7f5780133b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:07:01.261 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:01.257+0000 7f57777fe640 1 --2- 172.21.15.42:0/483821658 >> v2:172.21.15.154:3300/0 conn(0x7f57800a3d90 0x7f5780137000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:01.261 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:01.257+0000 7f577ffff640 1 --2- 172.21.15.42:0/483821658 >> v2:172.21.15.42:3300/0 conn(0x7f57800a39b0 0x7f5780133b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T04:07:01.261 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:01.257+0000 7f57777fe640 10 monclient(hunting): get_auth_request con 0x7f57800a3d90 auth_method 0 2024-09-16T04:07:01.261 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:01.257+0000 7f57777fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:01.261 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:01.257+0000 7f57777fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:01.261 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:01.257+0000 7f57777fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:07:01.261 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:01.257+0000 7f57777fe640 1 -- 172.21.15.42:0/483821658 >> v2:172.21.15.154:3300/0 conn(0x7f57800a3d90 msgr2=0x7f5780137000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:07:01.262 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:01.257+0000 7f57777fe640 1 -- 172.21.15.42:0/483821658 >> v2:172.21.15.154:3300/0 conn(0x7f57800a3d90 msgr2=0x7f5780137000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:07:01.262 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:01.257+0000 7f57777fe640 1 --2- 172.21.15.42:0/483821658 >> v2:172.21.15.154:3300/0 conn(0x7f57800a3d90 0x7f5780137000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:07:01.262 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:01.257+0000 7f57777fe640 1 --2- 172.21.15.42:0/483821658 >> v2:172.21.15.154:3300/0 conn(0x7f57800a3d90 0x7f5780137000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T04:07:02.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:02.845+0000 7f577e7fc640 10 monclient(hunting): tick 2024-09-16T04:07:02.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:02.845+0000 7f577e7fc640 10 monclient(hunting): _check_auth_tickets 2024-09-16T04:07:02.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:02.845+0000 7f577e7fc640 1 monclient(hunting): continuing hunt 2024-09-16T04:07:02.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:02.845+0000 7f577e7fc640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T04:07:02.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:02.845+0000 7f577e7fc640 1 -- 172.21.15.42:0/483821658 >> v2:172.21.15.154:3300/0 conn(0x7f57800a3d90 msgr2=0x7f5780137000 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:07:02.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:02.845+0000 7f577e7fc640 1 --2- 172.21.15.42:0/483821658 >> v2:172.21.15.154:3300/0 conn(0x7f57800a3d90 0x7f5780137000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:02.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:02.845+0000 7f577e7fc640 1 -- 172.21.15.42:0/483821658 >> v2:172.21.15.42:3300/0 conn(0x7f57800a39b0 msgr2=0x7f5780133b80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:07:02.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:02.845+0000 7f577e7fc640 1 --2- 172.21.15.42:0/483821658 >> v2:172.21.15.42:3300/0 conn(0x7f57800a39b0 0x7f5780133b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:02.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:02.845+0000 7f577e7fc640 1 -- 172.21.15.42:0/483821658 >> v1:172.21.15.154:6789/0 conn(0x7f5780137540 legacy=0x7f578013e390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:02.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:02.845+0000 7f577e7fc640 10 monclient: _add_conns ranks=[2,0] 2024-09-16T04:07:02.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:02.845+0000 7f577e7fc640 1 --2- 172.21.15.42:0/483821658 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f57600036a0 0x7f5760005ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:02.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:02.845+0000 7f577e7fc640 10 monclient(hunting): picked mon.noname-b con 0x7f57600036a0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:07:02.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:02.845+0000 7f577e7fc640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5760006200 addr v1:172.21.15.42:6789/0 2024-09-16T04:07:02.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:02.845+0000 7f577e7fc640 1 -- 172.21.15.42:0/483821658 --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f57800a5e20 con 0x7f5760006200 2024-09-16T04:07:02.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:02.845+0000 7f577e7fc640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:02.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:02.845+0000 7f577e7fc640 1 -- 172.21.15.42:0/483821658 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f578009c420 con 0x7f57600036a0 2024-09-16T04:07:02.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:02.845+0000 7f577e7fc640 10 monclient(hunting): _renew_subs 2024-09-16T04:07:02.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:02.845+0000 7f577ffff640 1 --2- 172.21.15.42:0/483821658 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f57600036a0 0x7f5760005ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:02.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:02.845+0000 7f577ffff640 10 monclient(hunting): get_auth_request con 0x7f57600036a0 auth_method 0 2024-09-16T04:07:02.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:02.845+0000 7f577ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:02.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:02.845+0000 7f577ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:02.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:02.845+0000 7f577ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:02.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:02.845+0000 7f577ffff640 1 -- 172.21.15.42:0/483821658 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f57600036a0 msgr2=0x7f5760005ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:07:02.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:02.845+0000 7f577ffff640 1 -- 172.21.15.42:0/483821658 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f57600036a0 msgr2=0x7f5760005ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:07:02.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:02.845+0000 7f577ffff640 1 --2- 172.21.15.42:0/483821658 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f57600036a0 0x7f5760005ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:07:02.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:02.845+0000 7f577ffff640 1 --2- 172.21.15.42:0/483821658 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f57600036a0 0x7f5760005ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:07:03.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:03.049+0000 7f577ffff640 1 --2- 172.21.15.42:0/483821658 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f57600036a0 0x7f5760005ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:03.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:03.049+0000 7f577ffff640 10 monclient(hunting): get_auth_request con 0x7f57600036a0 auth_method 0 2024-09-16T04:07:03.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:03.049+0000 7f577ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:03.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:03.049+0000 7f577ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:03.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:03.049+0000 7f577ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:07:03.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:03.049+0000 7f577ffff640 1 -- 172.21.15.42:0/483821658 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f57600036a0 msgr2=0x7f5760005ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:07:03.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:03.049+0000 7f577ffff640 1 -- 172.21.15.42:0/483821658 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f57600036a0 msgr2=0x7f5760005ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:07:03.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:03.049+0000 7f577ffff640 1 --2- 172.21.15.42:0/483821658 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f57600036a0 0x7f5760005ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:07:03.055 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:03.049+0000 7f577ffff640 1 --2- 172.21.15.42:0/483821658 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f57600036a0 0x7f5760005ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T04:07:03.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:03.453+0000 7f577ffff640 1 --2- 172.21.15.42:0/483821658 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f57600036a0 0x7f5760005ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:03.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:03.453+0000 7f577ffff640 10 monclient(hunting): get_auth_request con 0x7f57600036a0 auth_method 0 2024-09-16T04:07:03.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:03.453+0000 7f577ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:03.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:03.453+0000 7f577ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:03.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:03.453+0000 7f577ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:07:03.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:03.453+0000 7f577ffff640 1 -- 172.21.15.42:0/483821658 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f57600036a0 msgr2=0x7f5760005ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:07:03.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:03.453+0000 7f577ffff640 1 -- 172.21.15.42:0/483821658 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f57600036a0 msgr2=0x7f5760005ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:07:03.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:03.453+0000 7f577ffff640 1 --2- 172.21.15.42:0/483821658 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f57600036a0 0x7f5760005ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:07:03.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:03.453+0000 7f577ffff640 1 --2- 172.21.15.42:0/483821658 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f57600036a0 0x7f5760005ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T04:07:04.260 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.257+0000 7f577ffff640 1 --2- 172.21.15.42:0/483821658 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f57600036a0 0x7f5760005ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:04.260 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.257+0000 7f577ffff640 10 monclient(hunting): get_auth_request con 0x7f57600036a0 auth_method 0 2024-09-16T04:07:04.260 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.257+0000 7f577ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:04.260 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.257+0000 7f577ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:04.260 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.257+0000 7f577ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:07:04.260 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.257+0000 7f577ffff640 1 -- 172.21.15.42:0/483821658 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f57600036a0 msgr2=0x7f5760005ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:07:04.260 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.257+0000 7f577ffff640 1 -- 172.21.15.42:0/483821658 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f57600036a0 msgr2=0x7f5760005ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:07:04.260 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.257+0000 7f577ffff640 1 --2- 172.21.15.42:0/483821658 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f57600036a0 0x7f5760005ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:07:04.260 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.257+0000 7f577ffff640 1 --2- 172.21.15.42:0/483821658 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f57600036a0 0x7f5760005ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T04:07:04.771 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.765+0000 7f577f7fe640 1 -- 172.21.15.42:0/483821658 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3629490309 0 0) 0x7f57800a5e20 con 0x7f5760006200 2024-09-16T04:07:04.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.765+0000 7f577f7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:07:04.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.765+0000 7f577f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:04.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.765+0000 7f577f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:04.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.765+0000 7f577f7fe640 10 monclient(hunting): my global_id is 984453 2024-09-16T04:07:04.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.765+0000 7f577f7fe640 1 -- 172.21.15.42:0/483821658 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f576c003530 con 0x7f5760006200 2024-09-16T04:07:04.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.817+0000 7f577f7fe640 1 -- 172.21.15.42:0/483821658 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 437614736 0 0) 0x7f576c003530 con 0x7f5760006200 2024-09-16T04:07:04.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.817+0000 7f577f7fe640 1 -- 172.21.15.42:0/483821658 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f57600036a0 msgr2=0x7f5760005ad0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:07:04.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.817+0000 7f577f7fe640 1 --2- 172.21.15.42:0/483821658 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f57600036a0 0x7f5760005ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:04.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.817+0000 7f577f7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:07:04.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.817+0000 7f577f7fe640 1 monclient: found mon.noname-a-legacy 2024-09-16T04:07:04.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.817+0000 7f577f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:07:04.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.817+0000 7f577f7fe640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.42:6789/0 2024-09-16T04:07:04.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.817+0000 7f577f7fe640 1 -- 172.21.15.42:0/483821658 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f576000b000 con 0x7f5760006200 2024-09-16T04:07:04.828 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.817+0000 7f577f7fe640 10 monclient: _finish_auth 0 2024-09-16T04:07:04.828 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.817+0000 7f577f7fe640 10 monclient: _check_auth_tickets 2024-09-16T04:07:04.828 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.817+0000 7f577f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:07:04.828 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.817+0000 7f577f7fe640 1 -- 172.21.15.42:0/483821658 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2177413734 0 0) 0x7f576800f040 con 0x7f5760006200 2024-09-16T04:07:04.828 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.817+0000 7f577f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:04.828 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.817+0000 7f577f7fe640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T04:07:04.828 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.817+0000 7f577f7fe640 10 monclient: dump: 2024-09-16T04:07:04.828 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:07:04.828 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:04.828 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:04.828 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:04.828 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:04.828 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:04.829 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:07:04.829 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:07:04.829 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:07:04.829 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:04.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.817+0000 7f577f7fe640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] but i'm connected to v1:172.21.15.42:6789/0, reconnecting 2024-09-16T04:07:04.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.817+0000 7f577f7fe640 10 monclient: _reopen_session rank -1 2024-09-16T04:07:04.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.817+0000 7f577f7fe640 1 -- 172.21.15.42:0/483821658 >> v1:172.21.15.42:6789/0 conn(0x7f5760006200 legacy=0x7f57600086c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:04.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.821+0000 7f5784e9c640 1 -- 172.21.15.42:0/483821658 reap_dead start 2024-09-16T04:07:04.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.821+0000 7f577f7fe640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:07:04.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.821+0000 7f577f7fe640 1 --2- 172.21.15.42:0/483821658 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f576c0039c0 0x7f576c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:04.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.821+0000 7f577f7fe640 10 monclient(hunting): picked mon.b con 0x7f576c0039c0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:07:04.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.821+0000 7f577f7fe640 1 --2- 172.21.15.42:0/483821658 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f576c004810 0x7f576c004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:04.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.821+0000 7f577f7fe640 10 monclient(hunting): picked mon.c con 0x7f576c004810 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:07:04.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.821+0000 7f577f7fe640 1 --2- 172.21.15.42:0/483821658 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f576c0052f0 0x7f576c00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:04.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.821+0000 7f577f7fe640 10 monclient(hunting): picked mon.a con 0x7f576c0052f0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:07:04.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.821+0000 7f577f7fe640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:04.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.821+0000 7f577f7fe640 1 -- 172.21.15.42:0/483821658 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f578009c420 con 0x7f576c0052f0 2024-09-16T04:07:04.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.821+0000 7f577f7fe640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:04.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.821+0000 7f577f7fe640 1 -- 172.21.15.42:0/483821658 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f576c00c240 con 0x7f576c0039c0 2024-09-16T04:07:04.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.821+0000 7f577f7fe640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:04.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.821+0000 7f577f7fe640 1 -- 172.21.15.42:0/483821658 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f576c00c440 con 0x7f576c004810 2024-09-16T04:07:04.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.821+0000 7f577f7fe640 10 monclient(hunting): _renew_subs 2024-09-16T04:07:04.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.821+0000 7f577f7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:07:04.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.821+0000 7f57777fe640 1 --2- 172.21.15.42:0/483821658 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f576c004810 0x7f576c004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:04.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.821+0000 7f5784e9c640 1 --2- 172.21.15.42:0/483821658 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f576c0052f0 0x7f576c00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:04.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.821+0000 7f57777fe640 10 monclient(hunting): get_auth_request con 0x7f576c004810 auth_method 0 2024-09-16T04:07:04.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.821+0000 7f57777fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:04.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.821+0000 7f57777fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:04.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.821+0000 7f57777fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:07:04.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.821+0000 7f5784e9c640 10 monclient(hunting): get_auth_request con 0x7f576c0052f0 auth_method 0 2024-09-16T04:07:04.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.821+0000 7f5784e9c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:04.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.821+0000 7f5784e9c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:04.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.821+0000 7f5784e9c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:07:04.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.821+0000 7f57777fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:07:04.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.821+0000 7f57777fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:07:04.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.821+0000 7f57777fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:07:04.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.821+0000 7f577ffff640 1 --2- 172.21.15.42:0/483821658 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f576c0039c0 0x7f576c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:04.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.821+0000 7f5784e9c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:07:04.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.821+0000 7f5784e9c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:07:04.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.821+0000 7f5784e9c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:07:04.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.821+0000 7f577ffff640 10 monclient(hunting): get_auth_request con 0x7f576c0039c0 auth_method 0 2024-09-16T04:07:04.832 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.821+0000 7f577ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:04.832 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.821+0000 7f577ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:04.832 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.821+0000 7f577ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:07:04.832 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.821+0000 7f57777fe640 10 monclient(hunting): handle_auth_done global_id 984453 payload 293 2024-09-16T04:07:04.832 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.821+0000 7f57777fe640 1 -- 172.21.15.42:0/483821658 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f576c0039c0 msgr2=0x7f576c003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:07:04.832 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.821+0000 7f57777fe640 1 --2- 172.21.15.42:0/483821658 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f576c0039c0 0x7f576c003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:04.832 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.821+0000 7f57777fe640 1 -- 172.21.15.42:0/483821658 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f576c0052f0 msgr2=0x7f576c00b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:07:04.832 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.821+0000 7f57777fe640 1 --2- 172.21.15.42:0/483821658 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f576c0052f0 0x7f576c00b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:04.832 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.821+0000 7f57777fe640 10 monclient: _finish_hunting 0 2024-09-16T04:07:04.832 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.821+0000 7f57777fe640 1 monclient: found mon.c 2024-09-16T04:07:04.832 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.821+0000 7f57777fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:07:04.832 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.821+0000 7f57777fe640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:07:04.832 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.821+0000 7f57777fe640 1 -- 172.21.15.42:0/483821658 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f576c003530 con 0x7f576c004810 2024-09-16T04:07:04.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.821+0000 7f57777fe640 10 monclient: _finish_auth 0 2024-09-16T04:07:04.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.821+0000 7f57777fe640 10 monclient: _check_auth_tickets 2024-09-16T04:07:04.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.821+0000 7f57777fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:07:04.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.821+0000 7f57868ff640 5 monclient: authenticate success, global_id 984453 2024-09-16T04:07:04.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.821+0000 7f57868ff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:07:04.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.825+0000 7f57777fe640 1 --2- 172.21.15.42:0/483821658 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f576c004810 0x7f576c004c90 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f5768000ee0 tx=0x7f576801ff60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2a79e79928c0ec18 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:07:04.847 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.841+0000 7f577f7fe640 1 -- 172.21.15.42:0/483821658 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57680af070 con 0x7f576c004810 2024-09-16T04:07:04.847 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.841+0000 7f577f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:04.847 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.841+0000 7f577f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:07:04.847 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.841+0000 7f577f7fe640 10 monclient: dump: 2024-09-16T04:07:04.847 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:07:04.847 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:04.847 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:04.847 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:04.848 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:04.848 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:04.848 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:07:04.848 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:07:04.848 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:07:04.848 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:04.848 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.841+0000 7f57868ff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:07:04.848 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.841+0000 7f577f7fe640 1 -- 172.21.15.42:0/483821658 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f57680a9b30 con 0x7f576c004810 2024-09-16T04:07:04.848 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.841+0000 7f577f7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:07:04.848 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.841+0000 7f57868ff640 10 monclient: get_monmap_and_config success 2024-09-16T04:07:04.848 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.841+0000 7f57868ff640 4 set_mon_vals no callback set 2024-09-16T04:07:04.848 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.841+0000 7f577f7fe640 1 -- 172.21.15.42:0/483821658 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f576800ec20 con 0x7f576c004810 2024-09-16T04:07:04.848 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.841+0000 7f577f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:04.848 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.841+0000 7f577f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:07:04.849 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.841+0000 7f577f7fe640 10 monclient: dump: 2024-09-16T04:07:04.849 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:07:04.849 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:04.849 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:04.849 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:04.849 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:04.849 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:04.849 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:07:04.849 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:07:04.849 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:07:04.850 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:04.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.841+0000 7f57868ff640 10 monclient: shutdown 2024-09-16T04:07:04.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.841+0000 7f57868ff640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:07:04.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.841+0000 7f57868ff640 1 -- 172.21.15.42:0/483821658 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f576c004810 msgr2=0x7f576c004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:04.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.841+0000 7f57868ff640 1 --2- 172.21.15.42:0/483821658 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f576c004810 0x7f576c004c90 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f5768000ee0 tx=0x7f576801ff60 comp rx=0 tx=0).stop 2024-09-16T04:07:04.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.841+0000 7f57868ff640 1 -- 172.21.15.42:0/483821658 shutdown_connections 2024-09-16T04:07:04.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.841+0000 7f57868ff640 1 --2- 172.21.15.42:0/483821658 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f576c0052f0 0x7f576c00b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:04.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.841+0000 7f57868ff640 1 --2- 172.21.15.42:0/483821658 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f576c004810 0x7f576c004c90 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:04.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.841+0000 7f57868ff640 1 --2- 172.21.15.42:0/483821658 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f576c0039c0 0x7f576c003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:04.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.841+0000 7f57868ff640 1 -- 172.21.15.42:0/483821658 >> 172.21.15.42:0/483821658 conn(0x7f57800a9fb0 msgr2=0x7f57800aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:07:04.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.841+0000 7f57868ff640 1 -- 172.21.15.42:0/483821658 shutdown_connections 2024-09-16T04:07:04.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.841+0000 7f57868ff640 1 -- 172.21.15.42:0/483821658 wait complete. 2024-09-16T04:07:04.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.845+0000 7f57868ff640 10 monclient: build_initial_monmap 2024-09-16T04:07:04.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.845+0000 7f57868ff640 10 monclient: monmap: 2024-09-16T04:07:04.856 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:07:04.856 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:07:04.857 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:07:04.857 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:07:04.857 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:07:04.857 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:04.857 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:07:04.857 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:07:04.857 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:07:04.857 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:04.857 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.845+0000 7f57868ff640 1 librados: starting msgr at 2024-09-16T04:07:04.857 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.845+0000 7f57868ff640 1 librados: starting objecter 2024-09-16T04:07:04.857 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.845+0000 7f57868ff640 1 Processor -- start 2024-09-16T04:07:04.857 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.845+0000 7f57868ff640 1 -- start start 2024-09-16T04:07:04.857 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.845+0000 7f57868ff640 1 librados: setting wanted keys 2024-09-16T04:07:04.858 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.845+0000 7f57868ff640 1 librados: calling monclient init 2024-09-16T04:07:04.858 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.845+0000 7f57868ff640 10 monclient: init 2024-09-16T04:07:04.858 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.845+0000 7f57868ff640 10 monclient: _reopen_session rank -1 2024-09-16T04:07:04.858 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.845+0000 7f57868ff640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:07:04.858 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.845+0000 7f57868ff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f576c0039c0 0x7f57801cc190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:04.858 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.845+0000 7f57868ff640 10 monclient(hunting): picked mon.noname-a con 0x7f576c0039c0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:07:04.858 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.845+0000 7f57868ff640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f576c004810 0x7f57801cc6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:04.858 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.845+0000 7f57868ff640 10 monclient(hunting): picked mon.noname-b con 0x7f576c004810 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:07:04.858 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.845+0000 7f57868ff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f576c0052f0 0x7f57800a13b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:04.858 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.845+0000 7f57868ff640 10 monclient(hunting): picked mon.noname-c con 0x7f576c0052f0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:07:04.858 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.845+0000 7f57868ff640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:04.858 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.845+0000 7f57868ff640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f5780137390 con 0x7f576c0039c0 2024-09-16T04:07:04.858 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.845+0000 7f57868ff640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:04.858 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.845+0000 7f57868ff640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f5780137210 con 0x7f576c004810 2024-09-16T04:07:04.859 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.845+0000 7f57868ff640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:04.859 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.845+0000 7f57868ff640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f5780130ba0 con 0x7f576c0052f0 2024-09-16T04:07:04.859 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.845+0000 7f57868ff640 10 monclient(hunting): _renew_subs 2024-09-16T04:07:04.859 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.845+0000 7f57868ff640 10 monclient(hunting): authenticate will time out at 2471.818034s 2024-09-16T04:07:04.859 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.845+0000 7f5784e9c640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f576c0052f0 0x7f57800a13b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:04.859 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.845+0000 7f5784e9c640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f576c0052f0 0x7f57800a13b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:56768/0 (socket says 172.21.15.42:56768) 2024-09-16T04:07:04.859 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.845+0000 7f5784e9c640 1 -- 172.21.15.42:0/2720463500 learned_addr learned my addr 172.21.15.42:0/2720463500 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:07:04.859 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.845+0000 7f5784e9c640 10 monclient(hunting): get_auth_request con 0x7f576c0052f0 auth_method 0 2024-09-16T04:07:04.859 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.845+0000 7f5784e9c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:04.859 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.845+0000 7f5784e9c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:04.859 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.845+0000 7f5784e9c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:04.859 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.845+0000 7f577ffff640 1 --2- 172.21.15.42:0/2720463500 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f576c0039c0 0x7f57801cc190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:04.859 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.845+0000 7f57777fe640 1 --2- 172.21.15.42:0/2720463500 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f576c004810 0x7f57801cc6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:04.860 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.845+0000 7f577ffff640 10 monclient(hunting): get_auth_request con 0x7f576c0039c0 auth_method 0 2024-09-16T04:07:04.860 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.845+0000 7f577ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:04.860 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.845+0000 7f577ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:04.860 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.845+0000 7f577ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:04.860 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.845+0000 7f57777fe640 10 monclient(hunting): get_auth_request con 0x7f576c004810 auth_method 0 2024-09-16T04:07:04.860 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.845+0000 7f57777fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:04.860 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.845+0000 7f5784e9c640 1 -- 172.21.15.42:0/2720463500 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f576c0052f0 msgr2=0x7f57800a13b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-16T04:07:04.860 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.845+0000 7f5784e9c640 1 -- 172.21.15.42:0/2720463500 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f576c0052f0 msgr2=0x7f57800a13b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:07:04.860 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.845+0000 7f5784e9c640 1 --2- 172.21.15.42:0/2720463500 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f576c0052f0 0x7f57800a13b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:07:04.860 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.845+0000 7f57777fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:04.860 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.845+0000 7f57777fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:04.860 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.845+0000 7f5784e9c640 1 --2- 172.21.15.42:0/2720463500 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f576c0052f0 0x7f57800a13b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:07:04.860 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.845+0000 7f57777fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:07:04.861 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.845+0000 7f57777fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:07:04.861 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.845+0000 7f57777fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:07:04.861 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.845+0000 7f577ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:07:04.861 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.845+0000 7f577ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:07:04.861 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.845+0000 7f577ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:07:04.861 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.845+0000 7f577ffff640 10 monclient(hunting): handle_auth_done global_id 984456 payload 824 2024-09-16T04:07:04.861 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.845+0000 7f577ffff640 1 -- 172.21.15.42:0/2720463500 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f576c0052f0 msgr2=0x7f57800a13b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:07:04.861 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.845+0000 7f577ffff640 1 --2- 172.21.15.42:0/2720463500 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f576c0052f0 0x7f57800a13b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:04.861 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.845+0000 7f577ffff640 1 -- 172.21.15.42:0/2720463500 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f576c004810 msgr2=0x7f57801cc6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:04.861 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.845+0000 7f577ffff640 1 --2- 172.21.15.42:0/2720463500 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f576c004810 0x7f57801cc6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:04.861 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.845+0000 7f577ffff640 10 monclient: _finish_hunting 0 2024-09-16T04:07:04.861 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.845+0000 7f577ffff640 1 monclient: found mon.noname-a 2024-09-16T04:07:04.862 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.845+0000 7f577ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:07:04.862 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.845+0000 7f577ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:07:04.862 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.845+0000 7f577ffff640 1 -- 172.21.15.42:0/2720463500 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57801218b0 con 0x7f576c0039c0 2024-09-16T04:07:04.862 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.845+0000 7f57777fe640 1 --2- 172.21.15.42:0/2720463500 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f576c004810 0x7f57801cc6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:07:04.862 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.845+0000 7f577ffff640 1 --2- 172.21.15.42:0/2720463500 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f576c0039c0 0x7f57801cc190 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f5770017840 tx=0x7f5770017d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:07:04.862 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.845+0000 7f577d7fa640 10 client.?.objecter ms_handle_connect 0x7f576c0039c0 2024-09-16T04:07:04.862 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.845+0000 7f577d7fa640 10 client.?.objecter resend_mon_ops 2024-09-16T04:07:04.862 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.845+0000 7f577d7fa640 1 -- 172.21.15.42:0/2720463500 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f577000f040 con 0x7f576c0039c0 2024-09-16T04:07:04.862 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.845+0000 7f577d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:04.862 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.845+0000 7f577d7fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:07:04.862 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.845+0000 7f577d7fa640 10 monclient: dump: 2024-09-16T04:07:04.863 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:07:04.863 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:04.863 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:04.863 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:04.863 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:04.863 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:04.863 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:07:04.863 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:07:04.863 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:07:04.863 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:04.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.845+0000 7f577d7fa640 10 monclient: _finish_auth 0 2024-09-16T04:07:04.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.845+0000 7f577d7fa640 10 monclient: _check_auth_tickets 2024-09-16T04:07:04.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.845+0000 7f577d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:07:04.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.845+0000 7f577d7fa640 1 -- 172.21.15.42:0/2720463500 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5770016b60 con 0x7f576c0039c0 2024-09-16T04:07:04.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.845+0000 7f577d7fa640 10 monclient: handle_config config(0 keys) 2024-09-16T04:07:04.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.845+0000 7f577d7fa640 1 -- 172.21.15.42:0/2720463500 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5770004020 con 0x7f576c0039c0 2024-09-16T04:07:04.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.845+0000 7f577d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:04.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.845+0000 7f577d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:07:04.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.845+0000 7f577d7fa640 10 monclient: dump: 2024-09-16T04:07:04.864 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:07:04.864 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:04.864 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:04.864 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:04.864 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:04.864 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:04.864 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:07:04.865 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:07:04.865 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:07:04.865 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:04.865 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.845+0000 7f57868ff640 5 monclient: authenticate success, global_id 984456 2024-09-16T04:07:04.865 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.845+0000 7f57868ff640 10 monclient: _renew_subs 2024-09-16T04:07:04.865 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.845+0000 7f577effd640 4 set_mon_vals no callback set 2024-09-16T04:07:04.865 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.845+0000 7f57868ff640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:07:04.865 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.845+0000 7f57868ff640 1 -- 172.21.15.42:0/2720463500 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5780121b90 con 0x7f576c0039c0 2024-09-16T04:07:04.865 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.849+0000 7f57868ff640 10 client.984456.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:07:04.865 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.849+0000 7f57868ff640 10 monclient: _renew_subs 2024-09-16T04:07:04.865 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.849+0000 7f57868ff640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:07:04.865 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.849+0000 7f57868ff640 1 -- 172.21.15.42:0/2720463500 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57800a95f0 con 0x7f576c0039c0 2024-09-16T04:07:04.865 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.849+0000 7f57868ff640 1 librados: init done 2024-09-16T04:07:04.865 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.849+0000 7f57868ff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:07:04.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.849+0000 7f57868ff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:07:04.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.849+0000 7f57868ff640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:07:04.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.849+0000 7f57868ff640 1 -- 172.21.15.42:0/2720463500 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5780137970 con 0x7f576c0039c0 2024-09-16T04:07:04.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.849+0000 7f577d7fa640 1 -- 172.21.15.42:0/2720463500 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f577002a020 con 0x7f576c0039c0 2024-09-16T04:07:04.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.849+0000 7f577d7fa640 1 --2- 172.21.15.42:0/2720463500 >> v2:172.21.15.154:6812/254845342 conn(0x7f5760038180 0x7f576003a640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:04.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.849+0000 7f577d7fa640 1 -- 172.21.15.42:0/2720463500 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(706..706 src has 451..706) ==== 5040+0+0 (secure 0 0 0) 0x7f577004a210 con 0x7f576c0039c0 2024-09-16T04:07:04.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.849+0000 7f577d7fa640 10 client.984456.objecter ms_dispatch 0x7f5780000ba0 osd_map(706..706 src has 451..706) 2024-09-16T04:07:04.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.849+0000 7f577d7fa640 3 client.984456.objecter handle_osd_map got epochs [706,706] > 0 2024-09-16T04:07:04.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.849+0000 7f577d7fa640 3 client.984456.objecter handle_osd_map decoding full epoch 706 2024-09-16T04:07:04.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.849+0000 7f577d7fa640 20 client.984456.objecter dump_active .. 0 homeless 2024-09-16T04:07:04.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.849+0000 7f57777fe640 1 --2- 172.21.15.42:0/2720463500 >> v2:172.21.15.154:6812/254845342 conn(0x7f5760038180 0x7f576003a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:04.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.849+0000 7f57777fe640 10 monclient: get_auth_request con 0x7f5760038180 auth_method 0 2024-09-16T04:07:04.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.849+0000 7f57777fe640 1 --2- 172.21.15.42:0/2720463500 >> v2:172.21.15.154:6812/254845342 conn(0x7f5760038180 0x7f576003a640 secure :-1 s=READY pgs=3870 cs=0 l=1 rev1=1 crypto rx=0x7f57680b4000 tx=0x7f5768004740 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:07:04.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.853+0000 7f577d7fa640 10 client.984456.objecter ms_handle_connect 0x7f5760038180 2024-09-16T04:07:04.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.853+0000 7f577d7fa640 1 -- 172.21.15.42:0/2720463500 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f5770020070 con 0x7f576c0039c0 2024-09-16T04:07:04.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.853+0000 7f577d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:07:04.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:04.853+0000 7f577d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:07:05.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.081+0000 7f57868ff640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:07:05.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.081+0000 7f57868ff640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:07:05.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.081+0000 7f57868ff640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:07:05.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.081+0000 7f57868ff640 1 -- 172.21.15.42:0/2720463500 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f5780120400 con 0x7f576c0039c0 2024-09-16T04:07:05.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.081+0000 7f577d7fa640 1 -- 172.21.15.42:0/2720463500 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7f577002b530 con 0x7f576c0039c0 2024-09-16T04:07:05.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.081+0000 7f577d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:07:05.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.081+0000 7f577d7fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:07:05.089 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":232,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:07:05.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.085+0000 7f57868ff640 10 librados: watch_flush enter 2024-09-16T04:07:05.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.085+0000 7f57868ff640 10 librados: watch_flush exit 2024-09-16T04:07:05.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.085+0000 7f57868ff640 20 client.984456.objecter shutdown clearing up homeless session... 2024-09-16T04:07:05.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.085+0000 7f57868ff640 10 client.984456.objecter successfully canceled tick 2024-09-16T04:07:05.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.085+0000 7f57868ff640 1 -- 172.21.15.42:0/2720463500 >> v2:172.21.15.154:6812/254845342 conn(0x7f5760038180 msgr2=0x7f576003a640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:05.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.085+0000 7f57868ff640 1 --2- 172.21.15.42:0/2720463500 >> v2:172.21.15.154:6812/254845342 conn(0x7f5760038180 0x7f576003a640 secure :-1 s=READY pgs=3870 cs=0 l=1 rev1=1 crypto rx=0x7f57680b4000 tx=0x7f5768004740 comp rx=0 tx=0).stop 2024-09-16T04:07:05.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.085+0000 7f57868ff640 10 monclient: shutdown 2024-09-16T04:07:05.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.085+0000 7f57868ff640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:07:05.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.085+0000 7f57868ff640 1 -- 172.21.15.42:0/2720463500 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f576c0039c0 msgr2=0x7f57801cc190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:05.093 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.085+0000 7f57868ff640 1 --2- 172.21.15.42:0/2720463500 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f576c0039c0 0x7f57801cc190 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f5770017840 tx=0x7f5770017d00 comp rx=0 tx=0).stop 2024-09-16T04:07:05.093 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.085+0000 7f57868ff640 1 -- 172.21.15.42:0/2720463500 shutdown_connections 2024-09-16T04:07:05.093 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.085+0000 7f57868ff640 1 --2- 172.21.15.42:0/2720463500 >> v2:172.21.15.154:6812/254845342 conn(0x7f5760038180 0x7f576003a640 unknown :-1 s=CLOSED pgs=3870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:05.093 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.085+0000 7f57868ff640 1 --2- 172.21.15.42:0/2720463500 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f576c0052f0 0x7f57800a13b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:05.093 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.085+0000 7f57868ff640 1 --2- 172.21.15.42:0/2720463500 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f576c004810 0x7f57801cc6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:05.093 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.085+0000 7f57868ff640 1 --2- 172.21.15.42:0/2720463500 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f576c0039c0 0x7f57801cc190 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:05.093 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.085+0000 7f57868ff640 1 -- 172.21.15.42:0/2720463500 >> 172.21.15.42:0/2720463500 conn(0x7f57800a9fb0 msgr2=0x7f57800a7a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:07:05.093 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.085+0000 7f57868ff640 1 -- 172.21.15.42:0/2720463500 shutdown_connections 2024-09-16T04:07:05.093 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.085+0000 7f57868ff640 1 -- 172.21.15.42:0/2720463500 wait complete. 2024-09-16T04:07:05.093 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.085+0000 7f57868ff640 1 librados: shutdown 2024-09-16T04:07:05.108 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T04:07:05.109 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T04:07:05.109 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T04:07:05.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.229+0000 7f349750f640 10 monclient: get_monmap_and_config 2024-09-16T04:07:05.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.229+0000 7f349750f640 10 monclient: build_initial_monmap 2024-09-16T04:07:05.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.229+0000 7f349750f640 10 monclient: monmap: 2024-09-16T04:07:05.235 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:07:05.235 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:05.235 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:07:05.233040+0000 2024-09-16T04:07:05.235 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:07:05.233040+0000 2024-09-16T04:07:05.235 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:07:05.235 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:05.235 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:07:05.235 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:07:05.235 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:07:05.235 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:07:05.235 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:07:05.235 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:05.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.229+0000 7f349750f640 1 Processor -- start 2024-09-16T04:07:05.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.229+0000 7f349750f640 1 -- start start 2024-09-16T04:07:05.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.229+0000 7f349750f640 10 monclient: init 2024-09-16T04:07:05.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.229+0000 7f349750f640 10 monclient: _reopen_session rank -1 2024-09-16T04:07:05.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.229+0000 7f349750f640 10 monclient: _add_conns ranks=[2,4,0,3,1] 2024-09-16T04:07:05.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.229+0000 7f349750f640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f34900a1330 0x7f34900a1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:05.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.229+0000 7f349750f640 10 monclient(hunting): picked mon.noname-b con 0x7f34900a1330 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:07:05.242 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.229+0000 7f349750f640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f349006dac0 0x7f34900a1c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:05.242 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.229+0000 7f349750f640 10 monclient(hunting): picked mon.noname-c con 0x7f349006dac0 addr v2:172.21.15.154:3300/0 2024-09-16T04:07:05.242 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.229+0000 7f349750f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3490135100 addr v1:172.21.15.42:6789/0 2024-09-16T04:07:05.242 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.229+0000 7f349750f640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f349009c7d0 con 0x7f3490135100 2024-09-16T04:07:05.242 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.229+0000 7f349750f640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:05.242 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.229+0000 7f349750f640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f349009c090 con 0x7f349006dac0 2024-09-16T04:07:05.242 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.229+0000 7f349750f640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:05.242 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.229+0000 7f349750f640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f349009f610 con 0x7f34900a1330 2024-09-16T04:07:05.242 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.229+0000 7f349750f640 10 monclient(hunting): _renew_subs 2024-09-16T04:07:05.242 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.229+0000 7f349750f640 10 monclient(hunting): authenticate will time out at 2472.202473s 2024-09-16T04:07:05.242 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.229+0000 7f3495aac640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f3490135100 0x7f3490135550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:42850/0 (socket says 172.21.15.42:42850) 2024-09-16T04:07:05.242 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.229+0000 7f34952ab640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f34900a1330 0x7f34900a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:05.242 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.229+0000 7f3495aac640 1 -- 172.21.15.42:0/1779980896 learned_addr learned my addr 172.21.15.42:0/1779980896 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T04:07:05.243 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.229+0000 7f34952ab640 10 monclient(hunting): get_auth_request con 0x7f34900a1330 auth_method 0 2024-09-16T04:07:05.243 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.229+0000 7f34952ab640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:05.243 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.229+0000 7f34952ab640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:05.243 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.229+0000 7f34952ab640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:05.243 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.229+0000 7f34952ab640 1 -- 172.21.15.42:0/1779980896 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f34900a1330 msgr2=0x7f34900a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:07:05.243 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.229+0000 7f34952ab640 1 -- 172.21.15.42:0/1779980896 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f34900a1330 msgr2=0x7f34900a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:07:05.243 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.229+0000 7f34952ab640 1 --2- 172.21.15.42:0/1779980896 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f34900a1330 0x7f34900a1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:07:05.243 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.229+0000 7f34952ab640 1 --2- 172.21.15.42:0/1779980896 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f34900a1330 0x7f34900a1710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:07:05.243 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.229+0000 7f3494aaa640 1 --2- 172.21.15.42:0/1779980896 >> v2:172.21.15.154:3300/0 conn(0x7f349006dac0 0x7f34900a1c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:05.243 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.229+0000 7f3487fff640 1 -- 172.21.15.42:0/1779980896 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2819351764 0 0) 0x7f349009c7d0 con 0x7f3490135100 2024-09-16T04:07:05.243 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.229+0000 7f3487fff640 10 monclient(hunting): _negotiate 2024-09-16T04:07:05.243 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.229+0000 7f3487fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:05.243 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.229+0000 7f3487fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:05.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.229+0000 7f3487fff640 10 monclient(hunting): my global_id is 984459 2024-09-16T04:07:05.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.229+0000 7f3487fff640 1 -- 172.21.15.42:0/1779980896 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f347c003530 con 0x7f3490135100 2024-09-16T04:07:05.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.229+0000 7f3494aaa640 10 monclient(hunting): get_auth_request con 0x7f349006dac0 auth_method 0 2024-09-16T04:07:05.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.229+0000 7f3494aaa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:05.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.229+0000 7f3494aaa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:05.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.229+0000 7f3494aaa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:05.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.229+0000 7f3494aaa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:07:05.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.229+0000 7f3494aaa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:07:05.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.229+0000 7f3494aaa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:07:05.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.229+0000 7f3487fff640 1 -- 172.21.15.42:0/1779980896 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1410593606 0 0) 0x7f347c003530 con 0x7f3490135100 2024-09-16T04:07:05.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.229+0000 7f3487fff640 1 -- 172.21.15.42:0/1779980896 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f34900a1330 msgr2=0x7f34900a1710 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:07:05.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.229+0000 7f3487fff640 1 --2- 172.21.15.42:0/1779980896 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f34900a1330 0x7f34900a1710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:05.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.229+0000 7f3487fff640 1 -- 172.21.15.42:0/1779980896 >> v2:172.21.15.154:3300/0 conn(0x7f349006dac0 msgr2=0x7f34900a1c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:07:05.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.229+0000 7f3487fff640 1 --2- 172.21.15.42:0/1779980896 >> v2:172.21.15.154:3300/0 conn(0x7f349006dac0 0x7f34900a1c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:05.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.229+0000 7f3487fff640 10 monclient: _finish_hunting 0 2024-09-16T04:07:05.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.229+0000 7f3487fff640 1 monclient: found mon.noname-a-legacy 2024-09-16T04:07:05.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.229+0000 7f3487fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:07:05.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.229+0000 7f3487fff640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.42:6789/0 2024-09-16T04:07:05.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.229+0000 7f3487fff640 1 -- 172.21.15.42:0/1779980896 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f349009f8f0 con 0x7f3490135100 2024-09-16T04:07:05.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.229+0000 7f3487fff640 10 monclient: _finish_auth 0 2024-09-16T04:07:05.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.229+0000 7f3487fff640 10 monclient: _check_auth_tickets 2024-09-16T04:07:05.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.229+0000 7f3487fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:07:05.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.229+0000 7f3487fff640 1 -- 172.21.15.42:0/1779980896 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2177413734 0 0) 0x7f3488001a00 con 0x7f3490135100 2024-09-16T04:07:05.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.229+0000 7f349750f640 5 monclient: authenticate success, global_id 984459 2024-09-16T04:07:05.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.229+0000 7f3487fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:05.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.229+0000 7f3487fff640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T04:07:05.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.229+0000 7f3487fff640 10 monclient: dump: 2024-09-16T04:07:05.246 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:07:05.246 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:05.246 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:05.246 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:05.246 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:05.246 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:05.246 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:07:05.246 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:07:05.246 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:07:05.246 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:05.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.229+0000 7f3487fff640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] but i'm connected to v1:172.21.15.42:6789/0, reconnecting 2024-09-16T04:07:05.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.229+0000 7f3487fff640 10 monclient: _reopen_session rank -1 2024-09-16T04:07:05.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.229+0000 7f3487fff640 1 -- 172.21.15.42:0/1779980896 >> v1:172.21.15.42:6789/0 conn(0x7f3490135100 legacy=0x7f3490135550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:05.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.229+0000 7f3487fff640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:07:05.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.229+0000 7f3487fff640 1 --2- 172.21.15.42:0/1779980896 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f34900a1330 0x7f347c0039c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:05.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.229+0000 7f3487fff640 10 monclient(hunting): picked mon.c con 0x7f34900a1330 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:07:05.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.229+0000 7f3487fff640 1 --2- 172.21.15.42:0/1779980896 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f347c003f40 0x7f347c004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:05.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.229+0000 7f3487fff640 10 monclient(hunting): picked mon.a con 0x7f347c003f40 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:07:05.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.229+0000 7f3487fff640 1 --2- 172.21.15.42:0/1779980896 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f347c004d50 0x7f347c0051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:05.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.229+0000 7f3487fff640 10 monclient(hunting): picked mon.b con 0x7f347c004d50 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:07:05.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.229+0000 7f3487fff640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:05.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.229+0000 7f3487fff640 1 -- 172.21.15.42:0/1779980896 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f349009c090 con 0x7f347c003f40 2024-09-16T04:07:05.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.229+0000 7f3487fff640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:05.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.229+0000 7f3487fff640 1 -- 172.21.15.42:0/1779980896 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f349009f610 con 0x7f347c004d50 2024-09-16T04:07:05.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.229+0000 7f3487fff640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:05.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.229+0000 7f3487fff640 1 -- 172.21.15.42:0/1779980896 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f347c00bad0 con 0x7f34900a1330 2024-09-16T04:07:05.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.229+0000 7f3487fff640 10 monclient(hunting): _renew_subs 2024-09-16T04:07:05.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.229+0000 7f3487fff640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:07:05.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.229+0000 7f3494aaa640 1 --2- 172.21.15.42:0/1779980896 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f347c003f40 0x7f347c004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:05.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.229+0000 7f349750f640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-16T04:07:05.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.229+0000 7f3494aaa640 10 monclient(hunting): get_auth_request con 0x7f347c003f40 auth_method 0 2024-09-16T04:07:05.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.229+0000 7f3494aaa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:05.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.229+0000 7f3494aaa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:05.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.229+0000 7f34952ab640 1 --2- 172.21.15.42:0/1779980896 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f34900a1330 0x7f347c0039c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:05.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.229+0000 7f3494aaa640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:07:05.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.229+0000 7f34952ab640 10 monclient(hunting): get_auth_request con 0x7f34900a1330 auth_method 0 2024-09-16T04:07:05.249 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.229+0000 7f34952ab640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:05.249 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.229+0000 7f34952ab640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:05.249 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.229+0000 7f34952ab640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:07:05.249 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.229+0000 7f3494aaa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:07:05.249 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.229+0000 7f3494aaa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:07:05.249 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.229+0000 7f3494aaa640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:07:05.249 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.229+0000 7f34952ab640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:07:05.249 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.229+0000 7f34952ab640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:07:05.249 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.229+0000 7f34952ab640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:07:05.249 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.229+0000 7f3495aac640 1 --2- 172.21.15.42:0/1779980896 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f347c004d50 0x7f347c0051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:05.249 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.229+0000 7f34952ab640 10 monclient(hunting): handle_auth_done global_id 984459 payload 293 2024-09-16T04:07:05.249 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.229+0000 7f34952ab640 1 -- 172.21.15.42:0/1779980896 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f347c004d50 msgr2=0x7f347c0051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:07:05.249 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.229+0000 7f34952ab640 1 --2- 172.21.15.42:0/1779980896 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f347c004d50 0x7f347c0051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:05.250 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.229+0000 7f34952ab640 1 -- 172.21.15.42:0/1779980896 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f347c003f40 msgr2=0x7f347c004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:07:05.250 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.229+0000 7f34952ab640 1 --2- 172.21.15.42:0/1779980896 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f347c003f40 0x7f347c004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:05.250 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.229+0000 7f34952ab640 10 monclient: _finish_hunting 0 2024-09-16T04:07:05.250 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.229+0000 7f34952ab640 1 monclient: found mon.c 2024-09-16T04:07:05.250 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.229+0000 7f34952ab640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:07:05.250 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.229+0000 7f34952ab640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:07:05.250 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.229+0000 7f34952ab640 1 -- 172.21.15.42:0/1779980896 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f347c003530 con 0x7f34900a1330 2024-09-16T04:07:05.250 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.229+0000 7f34952ab640 10 monclient: _finish_auth 0 2024-09-16T04:07:05.250 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.229+0000 7f34952ab640 10 monclient: _check_auth_tickets 2024-09-16T04:07:05.250 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.229+0000 7f34952ab640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:07:05.250 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.229+0000 7f3494aaa640 1 --2- 172.21.15.42:0/1779980896 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f347c003f40 0x7f347c004810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:07:05.250 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.229+0000 7f3495aac640 10 monclient: get_auth_request con 0x7f347c004d50 auth_method 0 2024-09-16T04:07:05.250 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.229+0000 7f3495aac640 1 --2- 172.21.15.42:0/1779980896 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f347c004d50 0x7f347c0051d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T04:07:05.251 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.233+0000 7f34952ab640 1 --2- 172.21.15.42:0/1779980896 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f34900a1330 0x7f347c0039c0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f348000c850 tx=0x7f348001c4e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6128b693eb33ed0a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:07:05.251 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.233+0000 7f3487fff640 1 -- 172.21.15.42:0/1779980896 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f348000f040 con 0x7f34900a1330 2024-09-16T04:07:05.251 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.233+0000 7f3487fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:05.251 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.233+0000 7f3487fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:07:05.251 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.233+0000 7f3487fff640 10 monclient: dump: 2024-09-16T04:07:05.251 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:07:05.251 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:05.251 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:05.251 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:05.251 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:05.251 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:05.251 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:07:05.251 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:07:05.252 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:07:05.252 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:05.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.233+0000 7f3487fff640 1 -- 172.21.15.42:0/1779980896 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f34800a95f0 con 0x7f34900a1330 2024-09-16T04:07:05.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.233+0000 7f349750f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:07:05.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.233+0000 7f3487fff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:07:05.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.233+0000 7f349750f640 10 monclient: get_monmap_and_config success 2024-09-16T04:07:05.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.233+0000 7f3487fff640 1 -- 172.21.15.42:0/1779980896 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34800a99e0 con 0x7f34900a1330 2024-09-16T04:07:05.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.233+0000 7f3487fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:05.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.233+0000 7f349750f640 4 set_mon_vals no callback set 2024-09-16T04:07:05.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.233+0000 7f3487fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:07:05.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.233+0000 7f3487fff640 10 monclient: dump: 2024-09-16T04:07:05.252 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:07:05.252 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:05.253 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:05.253 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:05.253 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:05.253 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:05.253 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:07:05.253 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:07:05.253 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:07:05.253 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:05.253 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.233+0000 7f349750f640 10 monclient: shutdown 2024-09-16T04:07:05.253 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.233+0000 7f349750f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:07:05.253 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.233+0000 7f349750f640 1 -- 172.21.15.42:0/1779980896 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f34900a1330 msgr2=0x7f347c0039c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:05.253 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.233+0000 7f349750f640 1 --2- 172.21.15.42:0/1779980896 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f34900a1330 0x7f347c0039c0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f348000c850 tx=0x7f348001c4e0 comp rx=0 tx=0).stop 2024-09-16T04:07:05.253 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.233+0000 7f3495aac640 1 -- 172.21.15.42:0/1779980896 reap_dead start 2024-09-16T04:07:05.254 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.233+0000 7f349750f640 1 -- 172.21.15.42:0/1779980896 shutdown_connections 2024-09-16T04:07:05.254 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.233+0000 7f349750f640 1 -- 172.21.15.42:0/1779980896 >> 172.21.15.42:0/1779980896 conn(0x7f34900a2420 msgr2=0x7f34900a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:07:05.254 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.233+0000 7f349750f640 1 -- 172.21.15.42:0/1779980896 shutdown_connections 2024-09-16T04:07:05.254 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.233+0000 7f349750f640 1 -- 172.21.15.42:0/1779980896 wait complete. 2024-09-16T04:07:05.254 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.233+0000 7f349750f640 10 monclient: build_initial_monmap 2024-09-16T04:07:05.254 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.233+0000 7f349750f640 10 monclient: monmap: 2024-09-16T04:07:05.254 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:07:05.254 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:07:05.254 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:07:05.254 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:07:05.254 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:07:05.254 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:05.254 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:07:05.255 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:07:05.255 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:07:05.255 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:05.255 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.233+0000 7f349750f640 1 librados: starting msgr at 2024-09-16T04:07:05.255 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.233+0000 7f349750f640 1 librados: starting objecter 2024-09-16T04:07:05.255 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.233+0000 7f349750f640 1 Processor -- start 2024-09-16T04:07:05.255 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.233+0000 7f349750f640 1 -- start start 2024-09-16T04:07:05.255 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.233+0000 7f349750f640 1 librados: setting wanted keys 2024-09-16T04:07:05.255 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.233+0000 7f349750f640 1 librados: calling monclient init 2024-09-16T04:07:05.255 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.233+0000 7f349750f640 10 monclient: init 2024-09-16T04:07:05.261 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.237+0000 7f349750f640 10 monclient: _reopen_session rank -1 2024-09-16T04:07:05.262 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.237+0000 7f349750f640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:07:05.262 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.237+0000 7f349750f640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f34900a1330 0x7f349009cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:05.262 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.237+0000 7f349750f640 10 monclient(hunting): picked mon.noname-a con 0x7f34900a1330 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:07:05.262 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.237+0000 7f349750f640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f349013bb80 0x7f349009d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:05.262 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.237+0000 7f349750f640 10 monclient(hunting): picked mon.noname-c con 0x7f349013bb80 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:07:05.262 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.237+0000 7f349750f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f349009d690 0x7f34901d5d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:05.262 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.237+0000 7f349750f640 10 monclient(hunting): picked mon.noname-b con 0x7f349009d690 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:07:05.262 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.237+0000 7f349750f640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:05.262 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.237+0000 7f349750f640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f349013db70 con 0x7f34900a1330 2024-09-16T04:07:05.262 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.237+0000 7f349750f640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:05.262 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.237+0000 7f349750f640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f349009ef80 con 0x7f349009d690 2024-09-16T04:07:05.262 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.237+0000 7f349750f640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:05.262 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.237+0000 7f349750f640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f349013dcf0 con 0x7f349013bb80 2024-09-16T04:07:05.263 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.237+0000 7f349750f640 10 monclient(hunting): _renew_subs 2024-09-16T04:07:05.263 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.237+0000 7f349750f640 10 monclient(hunting): authenticate will time out at 2472.208524s 2024-09-16T04:07:05.263 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.237+0000 7f34952ab640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f34900a1330 0x7f349009cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:05.263 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.237+0000 7f3494aaa640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f349013bb80 0x7f349009d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:05.263 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.237+0000 7f34952ab640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f34900a1330 0x7f349009cc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:51452/0 (socket says 172.21.15.42:51452) 2024-09-16T04:07:05.263 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.237+0000 7f34952ab640 1 -- 172.21.15.42:0/58729199 learned_addr learned my addr 172.21.15.42:0/58729199 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:07:05.263 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.237+0000 7f34952ab640 10 monclient(hunting): get_auth_request con 0x7f34900a1330 auth_method 0 2024-09-16T04:07:05.263 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.237+0000 7f34952ab640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:05.263 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.237+0000 7f34952ab640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:05.263 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.237+0000 7f34952ab640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:05.263 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.237+0000 7f3494aaa640 10 monclient(hunting): get_auth_request con 0x7f349013bb80 auth_method 0 2024-09-16T04:07:05.263 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.237+0000 7f3494aaa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:05.263 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.237+0000 7f3494aaa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:05.264 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.237+0000 7f3494aaa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:05.264 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.237+0000 7f34952ab640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:07:05.264 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.237+0000 7f34952ab640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:07:05.264 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.237+0000 7f34952ab640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:07:05.264 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.237+0000 7f3494aaa640 1 -- 172.21.15.42:0/58729199 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f349013bb80 msgr2=0x7f349009d150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-16T04:07:05.264 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.237+0000 7f3494aaa640 1 -- 172.21.15.42:0/58729199 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f349013bb80 msgr2=0x7f349009d150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:07:05.264 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.237+0000 7f3494aaa640 1 --2- 172.21.15.42:0/58729199 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f349013bb80 0x7f349009d150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:07:05.264 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.237+0000 7f3494aaa640 1 --2- 172.21.15.42:0/58729199 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f349013bb80 0x7f349009d150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:07:05.264 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.237+0000 7f34952ab640 10 monclient(hunting): handle_auth_done global_id 984462 payload 824 2024-09-16T04:07:05.264 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.237+0000 7f3495aac640 1 --2- 172.21.15.42:0/58729199 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f349009d690 0x7f34901d5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:05.264 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.237+0000 7f34952ab640 1 -- 172.21.15.42:0/58729199 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f349013bb80 msgr2=0x7f349009d150 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:07:05.264 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.237+0000 7f34952ab640 1 --2- 172.21.15.42:0/58729199 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f349013bb80 0x7f349009d150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:05.264 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.237+0000 7f34952ab640 1 -- 172.21.15.42:0/58729199 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f349009d690 msgr2=0x7f34901d5d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:05.265 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.237+0000 7f34952ab640 1 --2- 172.21.15.42:0/58729199 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f349009d690 0x7f34901d5d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:05.265 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.237+0000 7f34952ab640 10 monclient: _finish_hunting 0 2024-09-16T04:07:05.265 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.237+0000 7f34952ab640 1 monclient: found mon.noname-a 2024-09-16T04:07:05.265 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.237+0000 7f34952ab640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:07:05.265 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.237+0000 7f34952ab640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:07:05.265 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.237+0000 7f34952ab640 1 -- 172.21.15.42:0/58729199 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f349013ae40 con 0x7f34900a1330 2024-09-16T04:07:05.265 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.237+0000 7f34952ab640 1 --2- 172.21.15.42:0/58729199 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f34900a1330 0x7f349009cc10 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f34800ab040 tx=0x7f3480019280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:07:05.265 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.237+0000 7f3485ffb640 10 client.?.objecter ms_handle_connect 0x7f34900a1330 2024-09-16T04:07:05.265 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.237+0000 7f3485ffb640 10 client.?.objecter resend_mon_ops 2024-09-16T04:07:05.265 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.237+0000 7f3485ffb640 1 -- 172.21.15.42:0/58729199 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34800c0070 con 0x7f34900a1330 2024-09-16T04:07:05.265 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.237+0000 7f3485ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:05.265 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.237+0000 7f3485ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:07:05.265 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.237+0000 7f3485ffb640 10 monclient: dump: 2024-09-16T04:07:05.266 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:07:05.266 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:05.266 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:05.266 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:05.266 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:05.266 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:05.266 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:07:05.266 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:07:05.266 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:07:05.266 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:05.266 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.237+0000 7f3485ffb640 10 monclient: _finish_auth 0 2024-09-16T04:07:05.266 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.237+0000 7f3485ffb640 10 monclient: _check_auth_tickets 2024-09-16T04:07:05.266 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.237+0000 7f3485ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:07:05.266 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.237+0000 7f3485ffb640 1 -- 172.21.15.42:0/58729199 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f34800a9e40 con 0x7f34900a1330 2024-09-16T04:07:05.267 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.237+0000 7f349750f640 5 monclient: authenticate success, global_id 984462 2024-09-16T04:07:05.267 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.237+0000 7f349750f640 10 monclient: _renew_subs 2024-09-16T04:07:05.267 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.237+0000 7f349750f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:07:05.267 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.237+0000 7f349750f640 1 -- 172.21.15.42:0/58729199 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f349013b120 con 0x7f34900a1330 2024-09-16T04:07:05.267 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.237+0000 7f3485ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:07:05.267 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.237+0000 7f3485ffb640 1 -- 172.21.15.42:0/58729199 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34800bb680 con 0x7f34900a1330 2024-09-16T04:07:05.267 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.237+0000 7f3485ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:05.267 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.237+0000 7f3485ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:07:05.267 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.237+0000 7f3485ffb640 10 monclient: dump: 2024-09-16T04:07:05.267 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:07:05.268 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:05.268 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:05.268 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:05.268 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:05.268 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:05.268 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:07:05.268 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:07:05.268 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:07:05.268 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:05.268 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.237+0000 7f34877fe640 4 set_mon_vals no callback set 2024-09-16T04:07:05.268 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.237+0000 7f349750f640 10 client.984462.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:07:05.268 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.237+0000 7f349750f640 10 monclient: _renew_subs 2024-09-16T04:07:05.268 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.237+0000 7f349750f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:07:05.269 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.237+0000 7f349750f640 1 -- 172.21.15.42:0/58729199 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34900a1000 con 0x7f34900a1330 2024-09-16T04:07:05.269 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.237+0000 7f349750f640 1 librados: init done 2024-09-16T04:07:05.269 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.237+0000 7f349750f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:07:05.269 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.237+0000 7f349750f640 1 --2- 172.21.15.42:0/58729199 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f349006f1f0 0x7f349006f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:05.269 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.237+0000 7f349750f640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:05.269 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.237+0000 7f349750f640 1 -- 172.21.15.42:0/58729199 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f3490073a30 con 0x7f349006f1f0 2024-09-16T04:07:05.269 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.237+0000 7f3485ffb640 1 -- 172.21.15.42:0/58729199 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f34800c3070 con 0x7f34900a1330 2024-09-16T04:07:05.269 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.237+0000 7f3494aaa640 1 --2- 172.21.15.42:0/58729199 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f349006f1f0 0x7f349006f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:05.269 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.237+0000 7f3494aaa640 10 monclient: get_auth_request con 0x7f349006f1f0 auth_method 0 2024-09-16T04:07:05.269 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.237+0000 7f3494aaa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:05.269 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.237+0000 7f3494aaa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:05.269 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.237+0000 7f3494aaa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:05.269 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.237+0000 7f3494aaa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:07:05.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.237+0000 7f3494aaa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:07:05.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.237+0000 7f3485ffb640 1 --2- 172.21.15.42:0/58729199 >> v2:172.21.15.154:6812/254845342 conn(0x7f3474037840 0x7f3474039d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:05.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.237+0000 7f3494aaa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:07:05.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.237+0000 7f3485ffb640 1 -- 172.21.15.42:0/58729199 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(706..706 src has 451..706) ==== 5040+0+0 (secure 0 0 0) 0x7f34800e3250 con 0x7f34900a1330 2024-09-16T04:07:05.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.237+0000 7f3485ffb640 10 client.984462.objecter ms_dispatch 0x7f3490000ba0 osd_map(706..706 src has 451..706) 2024-09-16T04:07:05.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.237+0000 7f3485ffb640 3 client.984462.objecter handle_osd_map got epochs [706,706] > 0 2024-09-16T04:07:05.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.237+0000 7f3485ffb640 3 client.984462.objecter handle_osd_map decoding full epoch 706 2024-09-16T04:07:05.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.237+0000 7f3494aaa640 10 monclient(hunting): handle_auth_done global_id 984465 payload 824 2024-09-16T04:07:05.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.237+0000 7f3485ffb640 20 client.984462.objecter dump_active .. 0 homeless 2024-09-16T04:07:05.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.237+0000 7f3494aaa640 1 -- 172.21.15.42:0/58729199 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f34900a1c50 con 0x7f349006f1f0 2024-09-16T04:07:05.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.237+0000 7f3485ffb640 10 client.984462.objecter ms_handle_connect 0x7f349006f1f0 2024-09-16T04:07:05.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.237+0000 7f3485ffb640 10 client.984462.objecter resend_mon_ops 2024-09-16T04:07:05.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.237+0000 7f3494aaa640 1 --2- 172.21.15.42:0/58729199 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f349006f1f0 0x7f349006f5d0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f348c014a30 tx=0x7f348c014ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:07:05.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.237+0000 7f3495aac640 1 --2- 172.21.15.42:0/58729199 >> v2:172.21.15.154:6812/254845342 conn(0x7f3474037840 0x7f3474039d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:05.271 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.237+0000 7f3495aac640 10 monclient: get_auth_request con 0x7f3474037840 auth_method 0 2024-09-16T04:07:05.271 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.237+0000 7f3485ffb640 1 -- 172.21.15.42:0/58729199 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f348c012c60 con 0x7f349006f1f0 2024-09-16T04:07:05.271 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.237+0000 7f3485ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:05.271 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.237+0000 7f3485ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:07:05.271 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.237+0000 7f3485ffb640 10 monclient: dump: 2024-09-16T04:07:05.271 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:07:05.271 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:05.271 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:05.271 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:05.271 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:05.271 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:05.271 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:07:05.271 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:07:05.271 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:07:05.272 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:05.272 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.237+0000 7f3485ffb640 1 -- 172.21.15.42:0/58729199 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f348c012c60 con 0x7f349006f1f0 2024-09-16T04:07:05.272 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.237+0000 7f3485ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:07:05.272 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.237+0000 7f3485ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:07:05.272 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.237+0000 7f3485ffb640 1 -- 172.21.15.42:0/58729199 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f349006f1f0 msgr2=0x7f349006f5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:05.272 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.237+0000 7f3485ffb640 1 --2- 172.21.15.42:0/58729199 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f349006f1f0 0x7f349006f5d0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f348c014a30 tx=0x7f348c014ef0 comp rx=0 tx=0).stop 2024-09-16T04:07:05.272 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.237+0000 7f3485ffb640 1 -- 172.21.15.42:0/58729199 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f349006f1f0 msgr2=0x7f349006f5d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:07:05.272 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.237+0000 7f3485ffb640 1 --2- 172.21.15.42:0/58729199 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f349006f1f0 0x7f349006f5d0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:05.272 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.237+0000 7f3485ffb640 10 client.984462.objecter ms_handle_connect 0x7f3474037840 2024-09-16T04:07:05.272 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.237+0000 7f3495aac640 1 --2- 172.21.15.42:0/58729199 >> v2:172.21.15.154:6812/254845342 conn(0x7f3474037840 0x7f3474039d00 secure :-1 s=READY pgs=3871 cs=0 l=1 rev1=1 crypto rx=0x7f3490068f60 tx=0x7f3488009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:07:05.272 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.245+0000 7f349750f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:07:05.272 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.245+0000 7f349750f640 1 --2- 172.21.15.42:0/58729199 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f349006fb10 0x7f34901da270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:05.273 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.245+0000 7f349750f640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:05.273 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.245+0000 7f349750f640 1 -- 172.21.15.42:0/58729199 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f349013a2e0 con 0x7f349006fb10 2024-09-16T04:07:05.273 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.245+0000 7f3494aaa640 1 --2- 172.21.15.42:0/58729199 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f349006fb10 0x7f34901da270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:05.273 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.245+0000 7f3494aaa640 10 monclient: get_auth_request con 0x7f349006fb10 auth_method 0 2024-09-16T04:07:05.273 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.245+0000 7f3494aaa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:05.273 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.245+0000 7f3494aaa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:05.273 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.245+0000 7f3494aaa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:05.273 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.245+0000 7f3494aaa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:07:05.273 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.245+0000 7f3494aaa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:07:05.273 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.245+0000 7f3494aaa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:07:05.273 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.245+0000 7f3494aaa640 10 monclient(hunting): handle_auth_done global_id 984468 payload 824 2024-09-16T04:07:05.273 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.245+0000 7f3494aaa640 1 -- 172.21.15.42:0/58729199 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f3490069040 con 0x7f349006fb10 2024-09-16T04:07:05.273 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.245+0000 7f3485ffb640 10 client.984462.objecter ms_handle_connect 0x7f349006fb10 2024-09-16T04:07:05.274 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.245+0000 7f3494aaa640 1 --2- 172.21.15.42:0/58729199 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f349006fb10 0x7f34901da270 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f348c01f000 tx=0x7f348c004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:07:05.274 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.245+0000 7f3485ffb640 10 client.984462.objecter resend_mon_ops 2024-09-16T04:07:05.274 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.245+0000 7f3485ffb640 1 -- 172.21.15.42:0/58729199 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f348c02f440 con 0x7f349006fb10 2024-09-16T04:07:05.274 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.245+0000 7f3485ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:05.274 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.245+0000 7f3485ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:07:05.274 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.245+0000 7f3485ffb640 10 monclient: dump: 2024-09-16T04:07:05.274 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:07:05.274 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:05.274 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:05.274 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:05.274 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:05.274 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:05.274 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:07:05.274 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:07:05.275 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:07:05.275 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:05.275 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.245+0000 7f3485ffb640 1 -- 172.21.15.42:0/58729199 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4715 (secure 0 0 0) 0x7f348c02f440 con 0x7f349006fb10 2024-09-16T04:07:05.275 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.245+0000 7f3485ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:07:05.275 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.245+0000 7f3485ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:07:05.275 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.245+0000 7f3485ffb640 1 -- 172.21.15.42:0/58729199 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f349006fb10 msgr2=0x7f34901da270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:05.275 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.245+0000 7f3485ffb640 1 --2- 172.21.15.42:0/58729199 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f349006fb10 0x7f34901da270 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f348c01f000 tx=0x7f348c004740 comp rx=0 tx=0).stop 2024-09-16T04:07:05.275 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.245+0000 7f3485ffb640 1 -- 172.21.15.42:0/58729199 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f349006fb10 msgr2=0x7f34901da270 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:07:05.275 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.245+0000 7f3485ffb640 1 --2- 172.21.15.42:0/58729199 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f349006fb10 0x7f34901da270 secure :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f348c01f000 tx=0x7f348c004740 comp rx=0 tx=0).stop 2024-09-16T04:07:05.275 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T04:07:05.276 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:07:05.276 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:07:05.276 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "leader", 2024-09-16T04:07:05.276 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 232, 2024-09-16T04:07:05.276 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 185302, 2024-09-16T04:07:05.276 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T04:07:05.276 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T04:07:05.276 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T04:07:05.276 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T04:07:05.276 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:07:05.276 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 0, 2024-09-16T04:07:05.276 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:07:05.276 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T04:07:05.277 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T04:07:05.277 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:07:05.277 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:07:05.277 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:07:05.277 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:07:05.277 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:07:05.277 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:07:05.277 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:07:05.277 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:07:05.277 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:07:05.277 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:07:05.277 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:07:05.277 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:07:05.278 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:07:05.278 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T04:07:05.278 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:07:05.278 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:07:05.278 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:07:05.278 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:07:05.278 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:07:05.278 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:07:05.278 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:07:05.278 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:07:05.278 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:07:05.278 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:07:05.278 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:07:05.278 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:07:05.279 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:05.279 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T04:07:05.279 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T04:07:05.279 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T04:07:05.279 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T04:07:05.279 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T04:07:05.279 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T04:07:05.279 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:07:05.279 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:07:05.279 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T04:07:05.279 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T04:07:05.279 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T04:07:05.279 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T04:07:05.280 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T04:07:05.280 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T04:07:05.280 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T04:07:05.280 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:07:05.280 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T04:07:05.280 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:07:05.280 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:07:05.280 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:07:05.280 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:07:05.280 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:07:05.280 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:07:05.280 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:07:05.280 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:07:05.281 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:07:05.281 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:07:05.281 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:07:05.281 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:07:05.281 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T04:07:05.281 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:05.281 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T04:07:05.281 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:05.281 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:07:05.281 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:07:05.281 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:07:05.281 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:07:05.281 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:05.282 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:07:05.282 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T04:07:05.282 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:07:05.282 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:05.282 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:05.282 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:07:05.282 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T04:07:05.282 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:07:05.282 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:07:05.282 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:07:05.282 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:05.282 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T04:07:05.282 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T04:07:05.283 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:07:05.283 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:07:05.283 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:07:05.283 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:05.283 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:05.283 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:07:05.283 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:07:05.283 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:07:05.283 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:07:05.283 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:05.283 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:07:05.283 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T04:07:05.283 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:07:05.283 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:05.284 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:05.284 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:07:05.284 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T04:07:05.284 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:07:05.284 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:07:05.284 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:07:05.284 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:05.284 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T04:07:05.284 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T04:07:05.284 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:07:05.284 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:07:05.284 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:07:05.284 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:05.285 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:05.285 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:07:05.285 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:07:05.285 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:07:05.285 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:07:05.285 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:05.285 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:07:05.285 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T04:07:05.285 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:07:05.285 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:05.285 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:05.285 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:07:05.285 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T04:07:05.285 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:07:05.286 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:07:05.286 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:07:05.286 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:05.286 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T04:07:05.286 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T04:07:05.286 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:07:05.286 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:07:05.286 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:07:05.286 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:07:05.286 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:07:05.286 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:05.286 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T04:07:05.286 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T04:07:05.286 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:05.287 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:07:05.287 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:07:05.287 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:07:05.287 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:07:05.287 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:07:05.287 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T04:07:05.287 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:05.287 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:07:05.287 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:07:05.287 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 3 2024-09-16T04:07:05.287 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:07:05.287 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:07:05.287 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T04:07:05.288 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:05.288 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:07:05.288 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:07:05.288 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 3 2024-09-16T04:07:05.288 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:07:05.288 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:07:05.288 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:05.288 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T04:07:05.288 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T04:07:05.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.245+0000 7f349750f640 10 librados: watch_flush enter 2024-09-16T04:07:05.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.245+0000 7f349750f640 10 librados: watch_flush exit 2024-09-16T04:07:05.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.245+0000 7f349750f640 20 client.984462.objecter shutdown clearing up homeless session... 2024-09-16T04:07:05.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.245+0000 7f349750f640 10 client.984462.objecter successfully canceled tick 2024-09-16T04:07:05.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.245+0000 7f349750f640 1 -- 172.21.15.42:0/58729199 >> v2:172.21.15.154:6812/254845342 conn(0x7f3474037840 msgr2=0x7f3474039d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:05.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.245+0000 7f349750f640 1 --2- 172.21.15.42:0/58729199 >> v2:172.21.15.154:6812/254845342 conn(0x7f3474037840 0x7f3474039d00 secure :-1 s=READY pgs=3871 cs=0 l=1 rev1=1 crypto rx=0x7f3490068f60 tx=0x7f3488009040 comp rx=0 tx=0).stop 2024-09-16T04:07:05.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.245+0000 7f349750f640 10 monclient: shutdown 2024-09-16T04:07:05.291 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.245+0000 7f349750f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:07:05.291 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.245+0000 7f349750f640 1 -- 172.21.15.42:0/58729199 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f34900a1330 msgr2=0x7f349009cc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:05.291 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.245+0000 7f349750f640 1 --2- 172.21.15.42:0/58729199 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f34900a1330 0x7f349009cc10 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f34800ab040 tx=0x7f3480019280 comp rx=0 tx=0).stop 2024-09-16T04:07:05.291 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.245+0000 7f3495aac640 1 -- 172.21.15.42:0/58729199 reap_dead start 2024-09-16T04:07:05.291 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.245+0000 7f3495aac640 1 -- 172.21.15.42:0/58729199 reap_dead start 2024-09-16T04:07:05.291 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.245+0000 7f349750f640 1 -- 172.21.15.42:0/58729199 shutdown_connections 2024-09-16T04:07:05.291 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.245+0000 7f349750f640 1 -- 172.21.15.42:0/58729199 >> 172.21.15.42:0/58729199 conn(0x7f34900a2420 msgr2=0x7f3490134620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:07:05.291 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.245+0000 7f349750f640 1 -- 172.21.15.42:0/58729199 shutdown_connections 2024-09-16T04:07:05.291 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.245+0000 7f349750f640 1 -- 172.21.15.42:0/58729199 wait complete. 2024-09-16T04:07:05.291 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.245+0000 7f349750f640 1 librados: shutdown 2024-09-16T04:07:05.291 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:07:05.292 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T04:07:05.292 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T04:07:05.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.405+0000 7f35bedc6640 10 monclient: get_monmap_and_config 2024-09-16T04:07:05.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.405+0000 7f35bedc6640 10 monclient: build_initial_monmap 2024-09-16T04:07:05.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.405+0000 7f35bedc6640 10 monclient: monmap: 2024-09-16T04:07:05.410 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:07:05.410 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:05.410 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:07:05.408859+0000 2024-09-16T04:07:05.410 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:07:05.408859+0000 2024-09-16T04:07:05.411 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:07:05.411 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:05.411 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:07:05.411 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:07:05.411 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:07:05.411 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:07:05.411 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:07:05.411 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:05.413 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.405+0000 7f35bedc6640 1 Processor -- start 2024-09-16T04:07:05.413 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.405+0000 7f35bedc6640 1 -- start start 2024-09-16T04:07:05.413 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.405+0000 7f35bedc6640 10 monclient: init 2024-09-16T04:07:05.413 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.405+0000 7f35bedc6640 10 monclient: _reopen_session rank -1 2024-09-16T04:07:05.413 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.405+0000 7f35bedc6640 10 monclient: _add_conns ranks=[4,1,0,3,2] 2024-09-16T04:07:05.414 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.405+0000 7f35bedc6640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f35b80a1330 0x7f35b80a1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:05.414 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.405+0000 7f35bedc6640 10 monclient(hunting): picked mon.noname-c con 0x7f35b80a1330 addr v2:172.21.15.154:3300/0 2024-09-16T04:07:05.414 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.405+0000 7f35bedc6640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f35b806dac0 addr v1:172.21.15.154:6789/0 2024-09-16T04:07:05.414 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.405+0000 7f35bedc6640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f35b80a1c50 addr v1:172.21.15.42:6789/0 2024-09-16T04:07:05.414 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.405+0000 7f35bedc6640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f35b8135f20 con 0x7f35b80a1c50 2024-09-16T04:07:05.414 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.405+0000 7f35bedc6640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f35b809c090 con 0x7f35b806dac0 2024-09-16T04:07:05.414 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.405+0000 7f35bedc6640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:05.414 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.405+0000 7f35bedc6640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f35b809f610 con 0x7f35b80a1330 2024-09-16T04:07:05.414 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.405+0000 7f35bedc6640 10 monclient(hunting): _renew_subs 2024-09-16T04:07:05.414 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.405+0000 7f35bedc6640 10 monclient(hunting): authenticate will time out at 2472.378294s 2024-09-16T04:07:05.414 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.405+0000 7f35bd363640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f35b80a1c50 0x7f35b8135810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:42860/0 (socket says 172.21.15.42:42860) 2024-09-16T04:07:05.414 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.405+0000 7f35bd363640 1 -- 172.21.15.42:0/893413533 learned_addr learned my addr 172.21.15.42:0/893413533 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T04:07:05.414 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.405+0000 7f35bcb62640 1 --2- 172.21.15.42:0/893413533 >> v2:172.21.15.154:3300/0 conn(0x7f35b80a1330 0x7f35b80a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:05.414 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.405+0000 7f35bcb62640 10 monclient(hunting): get_auth_request con 0x7f35b80a1330 auth_method 0 2024-09-16T04:07:05.415 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.405+0000 7f35bcb62640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:05.415 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.405+0000 7f35bcb62640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:05.415 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.405+0000 7f35bcb62640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:05.415 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.405+0000 7f35bcb62640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:07:05.415 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.405+0000 7f35bcb62640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:07:05.415 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.405+0000 7f35b77fe640 1 -- 172.21.15.42:0/893413533 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1607062173 0 0) 0x7f35b8135f20 con 0x7f35b80a1c50 2024-09-16T04:07:05.415 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.405+0000 7f35bcb62640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:07:05.415 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.405+0000 7f35b77fe640 10 monclient(hunting): _negotiate 2024-09-16T04:07:05.415 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.405+0000 7f35b77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:05.415 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.405+0000 7f35b77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:05.415 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.405+0000 7f35b77fe640 10 monclient(hunting): my global_id is 984471 2024-09-16T04:07:05.415 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.405+0000 7f35b77fe640 1 -- 172.21.15.42:0/893413533 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f35a0003530 con 0x7f35b80a1c50 2024-09-16T04:07:05.415 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.405+0000 7f35bcb62640 10 monclient(hunting): handle_auth_done global_id 905032 payload 274 2024-09-16T04:07:05.415 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.405+0000 7f35bcb62640 1 -- 172.21.15.42:0/893413533 >> v1:172.21.15.154:6789/0 conn(0x7f35b806dac0 legacy=0x7f35b8135100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:05.416 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.405+0000 7f35bcb62640 1 -- 172.21.15.42:0/893413533 >> v1:172.21.15.42:6789/0 conn(0x7f35b80a1c50 legacy=0x7f35b8135810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:05.416 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.405+0000 7f35b77fe640 1 -- 172.21.15.42:0/893413533 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3537618899 0 0) 0x7f35a0003530 con 0x7f35b80a1c50 2024-09-16T04:07:05.416 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.405+0000 7f35bcb62640 10 monclient: _finish_hunting 0 2024-09-16T04:07:05.416 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.405+0000 7f35bcb62640 1 monclient: found mon.noname-c 2024-09-16T04:07:05.416 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.405+0000 7f35bcb62640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:07:05.416 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.405+0000 7f35bcb62640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.154:3300/0 2024-09-16T04:07:05.416 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.405+0000 7f35bcb62640 1 -- 172.21.15.42:0/893413533 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35b809f8f0 con 0x7f35b80a1330 2024-09-16T04:07:05.416 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.405+0000 7f35b77fe640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T04:07:05.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.409+0000 7f35bcb62640 1 --2- 172.21.15.42:0/893413533 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f35b80a1330 0x7f35b80a1710 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f35b0009d90 tx=0x7f35b001c7b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ca38f08d17910fad server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:07:05.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.409+0000 7f35b77fe640 1 -- 172.21.15.42:0/893413533 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35b00b2070 con 0x7f35b80a1330 2024-09-16T04:07:05.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.409+0000 7f35b77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:05.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.409+0000 7f35b77fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:07:05.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.409+0000 7f35b77fe640 10 monclient: dump: 2024-09-16T04:07:05.423 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:07:05.423 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:05.423 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:05.424 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:05.424 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:05.424 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:05.424 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:07:05.424 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:07:05.424 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:07:05.424 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:05.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.409+0000 7f35b77fe640 10 monclient: _finish_auth 0 2024-09-16T04:07:05.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.409+0000 7f35b77fe640 10 monclient: _check_auth_tickets 2024-09-16T04:07:05.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.409+0000 7f35b77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:07:05.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.409+0000 7f35bedc6640 5 monclient: authenticate success, global_id 905032 2024-09-16T04:07:05.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.409+0000 7f35bedc6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:07:05.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.409+0000 7f35b77fe640 1 -- 172.21.15.42:0/893413533 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f35b00a5580 con 0x7f35b80a1330 2024-09-16T04:07:05.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.409+0000 7f35b77fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:07:05.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.409+0000 7f35bedc6640 10 monclient: get_monmap_and_config success 2024-09-16T04:07:05.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.409+0000 7f35b77fe640 1 -- 172.21.15.42:0/893413533 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35a8001680 con 0x7f35b80a1330 2024-09-16T04:07:05.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.409+0000 7f35b77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:05.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.409+0000 7f35bedc6640 4 set_mon_vals no callback set 2024-09-16T04:07:05.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.409+0000 7f35b77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:07:05.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.409+0000 7f35b77fe640 10 monclient: dump: 2024-09-16T04:07:05.425 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:07:05.425 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:05.425 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:05.425 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:05.425 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:05.425 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:05.425 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:07:05.426 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:07:05.426 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:07:05.426 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:05.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.409+0000 7f35bedc6640 10 monclient: shutdown 2024-09-16T04:07:05.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.409+0000 7f35bedc6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:07:05.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.409+0000 7f35bedc6640 1 -- 172.21.15.42:0/893413533 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f35b80a1330 msgr2=0x7f35b80a1710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:05.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.409+0000 7f35bedc6640 1 --2- 172.21.15.42:0/893413533 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f35b80a1330 0x7f35b80a1710 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f35b0009d90 tx=0x7f35b001c7b0 comp rx=0 tx=0).stop 2024-09-16T04:07:05.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.409+0000 7f35bedc6640 1 -- 172.21.15.42:0/893413533 shutdown_connections 2024-09-16T04:07:05.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.409+0000 7f35bedc6640 1 --2- 172.21.15.42:0/893413533 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f35b80a1330 0x7f35b80a1710 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:05.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.409+0000 7f35bedc6640 1 -- 172.21.15.42:0/893413533 >> 172.21.15.42:0/893413533 conn(0x7f35b80a2420 msgr2=0x7f35b80a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:07:05.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.409+0000 7f35bedc6640 1 -- 172.21.15.42:0/893413533 shutdown_connections 2024-09-16T04:07:05.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.409+0000 7f35bedc6640 1 -- 172.21.15.42:0/893413533 wait complete. 2024-09-16T04:07:05.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.409+0000 7f35bedc6640 10 monclient: build_initial_monmap 2024-09-16T04:07:05.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.409+0000 7f35bedc6640 10 monclient: monmap: 2024-09-16T04:07:05.427 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:07:05.427 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:07:05.427 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:07:05.427 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:07:05.427 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:07:05.427 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:05.427 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:07:05.427 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:07:05.427 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:07:05.427 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:05.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.409+0000 7f35bedc6640 1 librados: starting msgr at 2024-09-16T04:07:05.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.409+0000 7f35bedc6640 1 librados: starting objecter 2024-09-16T04:07:05.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.409+0000 7f35bedc6640 1 Processor -- start 2024-09-16T04:07:05.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.409+0000 7f35bedc6640 1 -- start start 2024-09-16T04:07:05.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.409+0000 7f35bedc6640 1 librados: setting wanted keys 2024-09-16T04:07:05.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.409+0000 7f35bedc6640 1 librados: calling monclient init 2024-09-16T04:07:05.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.409+0000 7f35bedc6640 10 monclient: init 2024-09-16T04:07:05.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.409+0000 7f35bedc6640 10 monclient: _reopen_session rank -1 2024-09-16T04:07:05.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.409+0000 7f35bedc6640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:07:05.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.409+0000 7f35bedc6640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f35b80a1330 0x7f35b813cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:05.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.409+0000 7f35bedc6640 10 monclient(hunting): picked mon.noname-a con 0x7f35b80a1330 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:07:05.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.413+0000 7f35bedc6640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f35b80a1c50 0x7f35b81351b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:05.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.413+0000 7f35bedc6640 10 monclient(hunting): picked mon.noname-c con 0x7f35b80a1c50 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:07:05.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.413+0000 7f35bedc6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f35b81356f0 0x7f35b81cec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:05.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.413+0000 7f35bedc6640 10 monclient(hunting): picked mon.noname-b con 0x7f35b81356f0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:07:05.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.413+0000 7f35bedc6640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:05.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.413+0000 7f35bedc6640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f35b809eef0 con 0x7f35b80a1330 2024-09-16T04:07:05.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.413+0000 7f35bedc6640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:05.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.413+0000 7f35bedc6640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f35b809ed70 con 0x7f35b81356f0 2024-09-16T04:07:05.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.413+0000 7f35bedc6640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:05.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.413+0000 7f35bedc6640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f35b8140b90 con 0x7f35b80a1c50 2024-09-16T04:07:05.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.413+0000 7f35bedc6640 10 monclient(hunting): _renew_subs 2024-09-16T04:07:05.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.413+0000 7f35bedc6640 10 monclient(hunting): authenticate will time out at 2472.384053s 2024-09-16T04:07:05.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.413+0000 7f35bcb62640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f35b80a1330 0x7f35b813cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:05.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.413+0000 7f35b7fff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f35b80a1c50 0x7f35b81351b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:05.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.413+0000 7f35bcb62640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f35b80a1330 0x7f35b813cd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:51476/0 (socket says 172.21.15.42:51476) 2024-09-16T04:07:05.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.413+0000 7f35bcb62640 1 -- 172.21.15.42:0/3254261314 learned_addr learned my addr 172.21.15.42:0/3254261314 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:07:05.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.413+0000 7f35bcb62640 10 monclient(hunting): get_auth_request con 0x7f35b80a1330 auth_method 0 2024-09-16T04:07:05.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.413+0000 7f35bcb62640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:05.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.413+0000 7f35bcb62640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:05.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.413+0000 7f35bcb62640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:05.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.413+0000 7f35b7fff640 10 monclient(hunting): get_auth_request con 0x7f35b80a1c50 auth_method 0 2024-09-16T04:07:05.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.413+0000 7f35b7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:05.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.413+0000 7f35b7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:05.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.413+0000 7f35b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:05.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.413+0000 7f35bcb62640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:07:05.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.413+0000 7f35bcb62640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:07:05.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.413+0000 7f35bcb62640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:07:05.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.413+0000 7f35b7fff640 1 -- 172.21.15.42:0/3254261314 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f35b80a1c50 msgr2=0x7f35b81351b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-16T04:07:05.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.413+0000 7f35b7fff640 1 -- 172.21.15.42:0/3254261314 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f35b80a1c50 msgr2=0x7f35b81351b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:07:05.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.413+0000 7f35b7fff640 1 --2- 172.21.15.42:0/3254261314 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f35b80a1c50 0x7f35b81351b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:07:05.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.413+0000 7f35b7fff640 1 --2- 172.21.15.42:0/3254261314 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f35b80a1c50 0x7f35b81351b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:07:05.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.413+0000 7f35bcb62640 10 monclient(hunting): handle_auth_done global_id 984474 payload 824 2024-09-16T04:07:05.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.413+0000 7f35bd363640 1 --2- 172.21.15.42:0/3254261314 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f35b81356f0 0x7f35b81cec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:05.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.413+0000 7f35bcb62640 1 -- 172.21.15.42:0/3254261314 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f35b80a1c50 msgr2=0x7f35b81351b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:07:05.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.413+0000 7f35bcb62640 1 --2- 172.21.15.42:0/3254261314 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f35b80a1c50 0x7f35b81351b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:05.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.413+0000 7f35bcb62640 1 -- 172.21.15.42:0/3254261314 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f35b81356f0 msgr2=0x7f35b81cec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:05.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.413+0000 7f35bcb62640 1 --2- 172.21.15.42:0/3254261314 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f35b81356f0 0x7f35b81cec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:05.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.413+0000 7f35bcb62640 10 monclient: _finish_hunting 0 2024-09-16T04:07:05.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.413+0000 7f35bcb62640 1 monclient: found mon.noname-a 2024-09-16T04:07:05.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.413+0000 7f35bcb62640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:07:05.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.413+0000 7f35bcb62640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:07:05.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.413+0000 7f35bcb62640 1 -- 172.21.15.42:0/3254261314 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35b813e170 con 0x7f35b80a1330 2024-09-16T04:07:05.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.413+0000 7f35bd363640 10 monclient: get_auth_request con 0x7f35b81356f0 auth_method 0 2024-09-16T04:07:05.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.413+0000 7f35bd363640 1 --2- 172.21.15.42:0/3254261314 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f35b81356f0 0x7f35b81cec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T04:07:05.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.413+0000 7f35bcb62640 1 --2- 172.21.15.42:0/3254261314 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f35b80a1330 0x7f35b813cd90 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f35b001a3e0 tx=0x7f35b0019550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:07:05.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.413+0000 7f35b57fa640 10 client.?.objecter ms_handle_connect 0x7f35b80a1330 2024-09-16T04:07:05.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.413+0000 7f35b57fa640 10 client.?.objecter resend_mon_ops 2024-09-16T04:07:05.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.413+0000 7f35b57fa640 1 -- 172.21.15.42:0/3254261314 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35b00b2050 con 0x7f35b80a1330 2024-09-16T04:07:05.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.413+0000 7f35b57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:05.432 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.413+0000 7f35b57fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:07:05.432 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.413+0000 7f35b57fa640 10 monclient: dump: 2024-09-16T04:07:05.432 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:07:05.432 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:05.432 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:05.432 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:05.432 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:05.432 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:05.432 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:07:05.432 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:07:05.432 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:07:05.432 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:05.432 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.413+0000 7f35b57fa640 10 monclient: _finish_auth 0 2024-09-16T04:07:05.432 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.413+0000 7f35b57fa640 10 monclient: _check_auth_tickets 2024-09-16T04:07:05.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.413+0000 7f35b57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:07:05.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.413+0000 7f35bedc6640 5 monclient: authenticate success, global_id 984474 2024-09-16T04:07:05.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.413+0000 7f35b57fa640 1 -- 172.21.15.42:0/3254261314 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f35b00b3040 con 0x7f35b80a1330 2024-09-16T04:07:05.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.413+0000 7f35bedc6640 10 monclient: _renew_subs 2024-09-16T04:07:05.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.413+0000 7f35bedc6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:07:05.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.413+0000 7f35bedc6640 1 -- 172.21.15.42:0/3254261314 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35b8071330 con 0x7f35b80a1330 2024-09-16T04:07:05.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.413+0000 7f35b57fa640 10 monclient: handle_config config(0 keys) 2024-09-16T04:07:05.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.413+0000 7f35b57fa640 1 -- 172.21.15.42:0/3254261314 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35b00a6bf0 con 0x7f35b80a1330 2024-09-16T04:07:05.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.413+0000 7f35b57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:05.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.413+0000 7f35b57fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:07:05.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.413+0000 7f35bedc6640 10 client.984474.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:07:05.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.413+0000 7f35b57fa640 10 monclient: dump: 2024-09-16T04:07:05.433 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:07:05.433 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:05.434 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:05.434 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:05.434 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:05.434 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:05.434 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:07:05.434 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:07:05.434 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:07:05.434 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:05.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.413+0000 7f35bedc6640 10 monclient: _renew_subs 2024-09-16T04:07:05.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.413+0000 7f35bedc6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:07:05.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.413+0000 7f35bedc6640 1 -- 172.21.15.42:0/3254261314 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35b8120590 con 0x7f35b80a1330 2024-09-16T04:07:05.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.413+0000 7f35bedc6640 1 librados: init done 2024-09-16T04:07:05.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.413+0000 7f35b6ffd640 4 set_mon_vals no callback set 2024-09-16T04:07:05.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.413+0000 7f35bedc6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:07:05.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.413+0000 7f35bedc6640 1 --2- 172.21.15.42:0/3254261314 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f35b806dac0 0x7f35b802c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:05.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.413+0000 7f35bedc6640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:05.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.413+0000 7f35bedc6640 1 -- 172.21.15.42:0/3254261314 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f35b8073a30 con 0x7f35b806dac0 2024-09-16T04:07:05.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.413+0000 7f35b7fff640 1 --2- 172.21.15.42:0/3254261314 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f35b806dac0 0x7f35b802c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:05.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.413+0000 7f35b7fff640 10 monclient: get_auth_request con 0x7f35b806dac0 auth_method 0 2024-09-16T04:07:05.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.413+0000 7f35b7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:05.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.413+0000 7f35b7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:05.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.413+0000 7f35b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:05.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.413+0000 7f35b7fff640 1 -- 172.21.15.42:0/3254261314 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f35b806dac0 msgr2=0x7f35b802c650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-16T04:07:05.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.413+0000 7f35b7fff640 1 -- 172.21.15.42:0/3254261314 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f35b806dac0 msgr2=0x7f35b802c650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:07:05.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.413+0000 7f35b7fff640 1 --2- 172.21.15.42:0/3254261314 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f35b806dac0 0x7f35b802c650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:07:05.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.413+0000 7f35b7fff640 1 --2- 172.21.15.42:0/3254261314 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f35b806dac0 0x7f35b802c650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:07:05.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.413+0000 7f35b57fa640 1 -- 172.21.15.42:0/3254261314 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f35b00c0020 con 0x7f35b80a1330 2024-09-16T04:07:05.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.413+0000 7f35b57fa640 1 --2- 172.21.15.42:0/3254261314 >> v2:172.21.15.154:6812/254845342 conn(0x7f35a4037810 0x7f35a4039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:05.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.413+0000 7f35b57fa640 1 -- 172.21.15.42:0/3254261314 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(706..706 src has 451..706) ==== 5040+0+0 (secure 0 0 0) 0x7f35b00e3380 con 0x7f35b80a1330 2024-09-16T04:07:05.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.413+0000 7f35b57fa640 10 client.984474.objecter ms_dispatch 0x7f35b8000ba0 osd_map(706..706 src has 451..706) 2024-09-16T04:07:05.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.413+0000 7f35b57fa640 3 client.984474.objecter handle_osd_map got epochs [706,706] > 0 2024-09-16T04:07:05.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.413+0000 7f35b57fa640 3 client.984474.objecter handle_osd_map decoding full epoch 706 2024-09-16T04:07:05.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.413+0000 7f35b57fa640 20 client.984474.objecter dump_active .. 0 homeless 2024-09-16T04:07:05.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.413+0000 7f35bd363640 1 --2- 172.21.15.42:0/3254261314 >> v2:172.21.15.154:6812/254845342 conn(0x7f35a4037810 0x7f35a4039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:05.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.413+0000 7f35bd363640 10 monclient: get_auth_request con 0x7f35a4037810 auth_method 0 2024-09-16T04:07:05.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.413+0000 7f35bd363640 1 --2- 172.21.15.42:0/3254261314 >> v2:172.21.15.154:6812/254845342 conn(0x7f35a4037810 0x7f35a4039cd0 secure :-1 s=READY pgs=3872 cs=0 l=1 rev1=1 crypto rx=0x7f35a8000c10 tx=0x7f35a8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:07:05.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.413+0000 7f35b57fa640 10 client.984474.objecter ms_handle_connect 0x7f35a4037810 2024-09-16T04:07:05.620 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.617+0000 7f35b7fff640 1 --2- 172.21.15.42:0/3254261314 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f35b806dac0 0x7f35b802c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:05.621 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.617+0000 7f35b7fff640 10 monclient: get_auth_request con 0x7f35b806dac0 auth_method 0 2024-09-16T04:07:05.621 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.617+0000 7f35b7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:05.621 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.617+0000 7f35b7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:05.621 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.617+0000 7f35b7fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:07:05.621 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.617+0000 7f35b7fff640 1 -- 172.21.15.42:0/3254261314 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f35b806dac0 msgr2=0x7f35b802c650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-16T04:07:05.621 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.617+0000 7f35b7fff640 1 -- 172.21.15.42:0/3254261314 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f35b806dac0 msgr2=0x7f35b802c650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:07:05.621 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.617+0000 7f35b7fff640 1 --2- 172.21.15.42:0/3254261314 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f35b806dac0 0x7f35b802c650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:07:05.621 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:05.617+0000 7f35b7fff640 1 --2- 172.21.15.42:0/3254261314 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f35b806dac0 0x7f35b802c650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T04:07:06.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:06.021+0000 7f35b7fff640 1 --2- 172.21.15.42:0/3254261314 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f35b806dac0 0x7f35b802c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:06.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:06.021+0000 7f35b7fff640 10 monclient: get_auth_request con 0x7f35b806dac0 auth_method 0 2024-09-16T04:07:06.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:06.021+0000 7f35b7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:06.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:06.021+0000 7f35b7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:06.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:06.021+0000 7f35b7fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:07:06.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:06.021+0000 7f35b7fff640 1 -- 172.21.15.42:0/3254261314 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f35b806dac0 msgr2=0x7f35b802c650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-16T04:07:06.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:06.021+0000 7f35b7fff640 1 -- 172.21.15.42:0/3254261314 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f35b806dac0 msgr2=0x7f35b802c650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:07:06.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:06.021+0000 7f35b7fff640 1 --2- 172.21.15.42:0/3254261314 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f35b806dac0 0x7f35b802c650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:07:06.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:06.021+0000 7f35b7fff640 1 --2- 172.21.15.42:0/3254261314 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f35b806dac0 0x7f35b802c650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T04:07:06.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:06.825+0000 7f35b7fff640 1 --2- 172.21.15.42:0/3254261314 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f35b806dac0 0x7f35b802c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:06.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:06.825+0000 7f35b7fff640 10 monclient: get_auth_request con 0x7f35b806dac0 auth_method 0 2024-09-16T04:07:06.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:06.825+0000 7f35b7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:06.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:06.825+0000 7f35b7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:06.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:06.825+0000 7f35b7fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:07:06.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:06.825+0000 7f35b7fff640 1 -- 172.21.15.42:0/3254261314 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f35b806dac0 msgr2=0x7f35b802c650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-16T04:07:06.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:06.825+0000 7f35b7fff640 1 -- 172.21.15.42:0/3254261314 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f35b806dac0 msgr2=0x7f35b802c650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:07:06.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:06.825+0000 7f35b7fff640 1 --2- 172.21.15.42:0/3254261314 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f35b806dac0 0x7f35b802c650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:07:06.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:06.825+0000 7f35b7fff640 1 --2- 172.21.15.42:0/3254261314 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f35b806dac0 0x7f35b802c650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T04:07:08.418 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:08.413+0000 7f359ffff640 10 monclient: tick 2024-09-16T04:07:08.418 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:08.413+0000 7f359ffff640 10 monclient: _check_auth_tickets 2024-09-16T04:07:08.418 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:08.413+0000 7f359ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:07:08.418 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:08.413+0000 7f359ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:07:08.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:08.429+0000 7f35b7fff640 1 --2- 172.21.15.42:0/3254261314 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f35b806dac0 0x7f35b802c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:08.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:08.429+0000 7f35b7fff640 10 monclient: get_auth_request con 0x7f35b806dac0 auth_method 0 2024-09-16T04:07:08.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:08.429+0000 7f35b7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:08.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:08.429+0000 7f35b7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:08.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:08.429+0000 7f35b7fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:07:08.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:08.429+0000 7f35b7fff640 1 -- 172.21.15.42:0/3254261314 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f35b806dac0 msgr2=0x7f35b802c650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-16T04:07:08.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:08.429+0000 7f35b7fff640 1 -- 172.21.15.42:0/3254261314 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f35b806dac0 msgr2=0x7f35b802c650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:07:08.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:08.429+0000 7f35b7fff640 1 --2- 172.21.15.42:0/3254261314 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f35b806dac0 0x7f35b802c650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:07:08.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:08.429+0000 7f35b7fff640 1 --2- 172.21.15.42:0/3254261314 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f35b806dac0 0x7f35b802c650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-09-16T04:07:09.418 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:09.413+0000 7f359ffff640 10 monclient: tick 2024-09-16T04:07:09.418 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:09.413+0000 7f359ffff640 10 monclient: _check_auth_tickets 2024-09-16T04:07:09.419 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:09.413+0000 7f359ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:07:09.419 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:09.413+0000 7f359ffff640 5 monclient: _check_tell_commands timeout tell command 1 2024-09-16T04:07:09.419 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:09.413+0000 7f359ffff640 1 -- 172.21.15.42:0/3254261314 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f35b806dac0 msgr2=0x7f35b802c650 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:07:09.419 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:09.413+0000 7f359ffff640 1 --2- 172.21.15.42:0/3254261314 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f35b806dac0 0x7f35b802c650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:09.419 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:09.413+0000 7f359ffff640 1 --2- 172.21.15.42:0/3254261314 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3590003550 0x7f35900059e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:09.419 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:09.413+0000 7f359ffff640 1 -- 172.21.15.42:0/3254261314 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f35b806dac0 msgr2=0x7f35b802c650 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:07:09.419 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:09.413+0000 7f359ffff640 1 --2- 172.21.15.42:0/3254261314 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f35b806dac0 0x7f35b802c650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:09.419 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:09.413+0000 7f359ffff640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:09.419 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:09.413+0000 7f359ffff640 1 -- 172.21.15.42:0/3254261314 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f35b8073a30 con 0x7f3590003550 2024-09-16T04:07:09.419 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:09.413+0000 7f35b7fff640 1 --2- 172.21.15.42:0/3254261314 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3590003550 0x7f35900059e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:09.419 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:09.413+0000 7f35b7fff640 10 monclient: get_auth_request con 0x7f3590003550 auth_method 0 2024-09-16T04:07:09.419 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:09.413+0000 7f35b7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:09.419 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:09.413+0000 7f35b7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:09.419 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:09.413+0000 7f35b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:09.420 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:09.413+0000 7f35b7fff640 1 -- 172.21.15.42:0/3254261314 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3590003550 msgr2=0x7f35900059e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-16T04:07:09.420 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:09.413+0000 7f35b7fff640 1 -- 172.21.15.42:0/3254261314 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3590003550 msgr2=0x7f35900059e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:07:09.420 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:09.413+0000 7f35b7fff640 1 --2- 172.21.15.42:0/3254261314 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3590003550 0x7f35900059e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:07:09.420 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:09.413+0000 7f35b7fff640 1 --2- 172.21.15.42:0/3254261314 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3590003550 0x7f35900059e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:07:09.619 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:09.617+0000 7f35b7fff640 1 --2- 172.21.15.42:0/3254261314 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3590003550 0x7f35900059e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:09.620 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:09.617+0000 7f35b7fff640 10 monclient: get_auth_request con 0x7f3590003550 auth_method 0 2024-09-16T04:07:09.620 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:09.617+0000 7f35b7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:09.620 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:09.617+0000 7f35b7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:09.620 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:09.617+0000 7f35b7fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:07:09.620 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:09.617+0000 7f35b7fff640 1 -- 172.21.15.42:0/3254261314 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3590003550 msgr2=0x7f35900059e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-16T04:07:09.620 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:09.617+0000 7f35b7fff640 1 -- 172.21.15.42:0/3254261314 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3590003550 msgr2=0x7f35900059e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:07:09.620 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:09.617+0000 7f35b7fff640 1 --2- 172.21.15.42:0/3254261314 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3590003550 0x7f35900059e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:07:09.620 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:09.617+0000 7f35b7fff640 1 --2- 172.21.15.42:0/3254261314 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3590003550 0x7f35900059e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T04:07:10.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.021+0000 7f35b7fff640 1 --2- 172.21.15.42:0/3254261314 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3590003550 0x7f35900059e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:10.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.021+0000 7f35b7fff640 10 monclient: get_auth_request con 0x7f3590003550 auth_method 0 2024-09-16T04:07:10.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.021+0000 7f35b7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:10.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.021+0000 7f35b7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:10.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.021+0000 7f35b7fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:07:10.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.021+0000 7f35b7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:07:10.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.021+0000 7f35b7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:07:10.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.021+0000 7f35b7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:07:10.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.021+0000 7f35b7fff640 10 monclient(hunting): handle_auth_done global_id 1014101 payload 824 2024-09-16T04:07:10.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.021+0000 7f35b7fff640 1 -- 172.21.15.42:0/3254261314 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f35900060c0 con 0x7f3590003550 2024-09-16T04:07:10.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.021+0000 7f35b7fff640 1 --2- 172.21.15.42:0/3254261314 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3590003550 0x7f35900059e0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f35b8072ae0 tx=0x7f35ac017af0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:07:10.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.021+0000 7f35b57fa640 10 client.984474.objecter ms_handle_connect 0x7f3590003550 2024-09-16T04:07:10.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.021+0000 7f35b57fa640 10 client.984474.objecter resend_mon_ops 2024-09-16T04:07:10.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.021+0000 7f35b57fa640 1 -- 172.21.15.42:0/3254261314 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35ac007020 con 0x7f3590003550 2024-09-16T04:07:10.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.021+0000 7f35b57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:10.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.021+0000 7f35b57fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:07:10.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.021+0000 7f35b57fa640 10 monclient: dump: 2024-09-16T04:07:10.028 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:07:10.028 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:10.029 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:10.029 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:10.029 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:10.029 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:10.029 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:07:10.029 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:07:10.029 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:07:10.029 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:10.029 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.021+0000 7f35b57fa640 1 -- 172.21.15.42:0/3254261314 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f35ac007020 con 0x7f3590003550 2024-09-16T04:07:10.029 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.021+0000 7f35b57fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:07:10.029 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.021+0000 7f35b57fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:07:10.029 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.021+0000 7f35b57fa640 1 -- 172.21.15.42:0/3254261314 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3590003550 msgr2=0x7f35900059e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:10.029 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.021+0000 7f35b57fa640 1 --2- 172.21.15.42:0/3254261314 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3590003550 0x7f35900059e0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f35b8072ae0 tx=0x7f35ac017af0 comp rx=0 tx=0).stop 2024-09-16T04:07:10.030 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.021+0000 7f35b57fa640 1 -- 172.21.15.42:0/3254261314 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3590003550 msgr2=0x7f35900059e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:07:10.030 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.021+0000 7f35b57fa640 1 --2- 172.21.15.42:0/3254261314 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3590003550 0x7f35900059e0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:10.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.029+0000 7f35bedc6640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:07:10.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.029+0000 7f35bedc6640 1 --2- 172.21.15.42:0/3254261314 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f35b802cb90 0x7f35b802cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:10.036 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.029+0000 7f35bedc6640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:10.036 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.029+0000 7f35bedc6640 1 -- 172.21.15.42:0/3254261314 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f35b8139a40 con 0x7f35b802cb90 2024-09-16T04:07:10.036 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.029+0000 7f35b7fff640 1 --2- 172.21.15.42:0/3254261314 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f35b802cb90 0x7f35b802cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:10.036 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.029+0000 7f35b7fff640 10 monclient: get_auth_request con 0x7f35b802cb90 auth_method 0 2024-09-16T04:07:10.036 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.029+0000 7f35b7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:10.036 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.029+0000 7f35b7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:10.036 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.029+0000 7f35b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:10.036 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.029+0000 7f35b7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:07:10.036 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.029+0000 7f35b7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:07:10.036 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.029+0000 7f35b7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:07:10.036 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.029+0000 7f35b7fff640 10 monclient(hunting): handle_auth_done global_id 1014104 payload 824 2024-09-16T04:07:10.036 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.029+0000 7f35b7fff640 1 -- 172.21.15.42:0/3254261314 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f35b8069040 con 0x7f35b802cb90 2024-09-16T04:07:10.036 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.029+0000 7f35b7fff640 1 --2- 172.21.15.42:0/3254261314 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f35b802cb90 0x7f35b802cfa0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f35ac020000 tx=0x7f35ac000ee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:07:10.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.029+0000 7f35b57fa640 10 client.984474.objecter ms_handle_connect 0x7f35b802cb90 2024-09-16T04:07:10.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.029+0000 7f35b57fa640 10 client.984474.objecter resend_mon_ops 2024-09-16T04:07:10.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.029+0000 7f35b57fa640 1 -- 172.21.15.42:0/3254261314 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35ac015e80 con 0x7f35b802cb90 2024-09-16T04:07:10.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.029+0000 7f35b57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:10.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.029+0000 7f35b57fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:07:10.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.029+0000 7f35b57fa640 10 monclient: dump: 2024-09-16T04:07:10.037 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:07:10.037 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:10.037 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:10.037 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:10.037 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:10.037 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:10.037 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:07:10.037 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:07:10.038 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:07:10.038 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:10.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.029+0000 7f35b57fa640 1 -- 172.21.15.42:0/3254261314 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4545 (secure 0 0 0) 0x7f35ac015e80 con 0x7f35b802cb90 2024-09-16T04:07:10.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.029+0000 7f35b57fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:07:10.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.029+0000 7f35b57fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:07:10.039 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.029+0000 7f35b57fa640 1 -- 172.21.15.42:0/3254261314 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f35b802cb90 msgr2=0x7f35b802cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:10.039 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.029+0000 7f35b57fa640 1 --2- 172.21.15.42:0/3254261314 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f35b802cb90 0x7f35b802cfa0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f35ac020000 tx=0x7f35ac000ee0 comp rx=0 tx=0).stop 2024-09-16T04:07:10.039 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.029+0000 7f35b57fa640 1 -- 172.21.15.42:0/3254261314 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f35b802cb90 msgr2=0x7f35b802cfa0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:07:10.039 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.029+0000 7f35b57fa640 1 --2- 172.21.15.42:0/3254261314 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f35b802cb90 0x7f35b802cfa0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:10.039 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.029+0000 7f35bd363640 1 -- 172.21.15.42:0/3254261314 reap_dead start 2024-09-16T04:07:10.039 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T04:07:10.039 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:07:10.039 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:07:10.039 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T04:07:10.039 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 232, 2024-09-16T04:07:10.040 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 25231, 2024-09-16T04:07:10.040 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T04:07:10.040 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T04:07:10.040 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T04:07:10.040 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T04:07:10.040 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:07:10.040 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 5, 2024-09-16T04:07:10.040 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:07:10.040 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T04:07:10.040 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T04:07:10.040 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:07:10.040 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:07:10.041 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:07:10.041 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:07:10.041 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:07:10.041 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:07:10.041 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:07:10.041 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:07:10.041 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:07:10.041 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:07:10.041 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:07:10.041 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:07:10.045 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:07:10.045 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T04:07:10.046 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:07:10.046 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:07:10.046 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:07:10.046 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:07:10.046 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:07:10.046 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:07:10.046 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:07:10.046 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:07:10.046 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:07:10.046 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:07:10.046 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:07:10.046 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:07:10.046 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:10.047 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T04:07:10.047 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T04:07:10.047 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T04:07:10.047 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T04:07:10.047 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T04:07:10.047 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T04:07:10.047 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:07:10.047 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:07:10.047 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T04:07:10.047 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T04:07:10.047 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T04:07:10.047 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T04:07:10.047 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T04:07:10.047 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T04:07:10.048 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T04:07:10.048 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:07:10.048 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T04:07:10.048 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:07:10.048 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:07:10.048 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:07:10.048 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:07:10.048 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:07:10.048 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:07:10.048 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:07:10.048 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:07:10.048 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:07:10.049 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:07:10.049 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:07:10.049 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:07:10.049 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T04:07:10.049 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:10.049 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T04:07:10.049 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:10.049 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:07:10.049 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:07:10.049 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:07:10.049 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:07:10.049 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:10.049 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:07:10.049 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T04:07:10.050 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:07:10.050 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:10.050 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:10.050 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:07:10.050 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T04:07:10.050 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:07:10.050 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:07:10.050 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:07:10.050 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:10.050 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T04:07:10.050 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T04:07:10.050 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:07:10.050 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:07:10.050 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:07:10.051 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:10.051 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:10.051 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:07:10.051 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:07:10.051 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:07:10.051 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:07:10.051 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:10.051 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:07:10.051 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T04:07:10.051 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:07:10.051 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:10.052 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:10.052 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:07:10.052 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T04:07:10.052 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:07:10.052 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:07:10.052 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:07:10.052 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:10.052 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T04:07:10.052 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T04:07:10.052 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:07:10.052 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:07:10.052 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:07:10.052 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:10.052 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:10.053 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:07:10.053 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:07:10.053 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:07:10.053 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:07:10.053 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:10.053 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:07:10.053 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T04:07:10.053 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:07:10.053 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:10.053 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:10.053 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:07:10.053 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T04:07:10.053 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:07:10.053 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:07:10.054 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:07:10.054 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:10.054 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T04:07:10.054 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T04:07:10.054 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:07:10.054 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:07:10.054 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:07:10.054 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:07:10.054 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:07:10.054 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:10.054 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T04:07:10.054 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T04:07:10.054 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:10.054 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:07:10.055 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:07:10.055 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:07:10.055 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:07:10.055 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:07:10.055 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T04:07:10.055 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:10.055 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:07:10.055 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:07:10.055 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:07:10.055 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:07:10.055 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:07:10.055 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:10.055 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T04:07:10.055 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T04:07:10.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.029+0000 7f35bedc6640 10 librados: watch_flush enter 2024-09-16T04:07:10.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.029+0000 7f35bedc6640 10 librados: watch_flush exit 2024-09-16T04:07:10.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.029+0000 7f35bedc6640 20 client.984474.objecter shutdown clearing up homeless session... 2024-09-16T04:07:10.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.029+0000 7f35bedc6640 10 client.984474.objecter successfully canceled tick 2024-09-16T04:07:10.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.029+0000 7f35bedc6640 1 -- 172.21.15.42:0/3254261314 >> v2:172.21.15.154:6812/254845342 conn(0x7f35a4037810 msgr2=0x7f35a4039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:10.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.029+0000 7f35bedc6640 1 --2- 172.21.15.42:0/3254261314 >> v2:172.21.15.154:6812/254845342 conn(0x7f35a4037810 0x7f35a4039cd0 secure :-1 s=READY pgs=3872 cs=0 l=1 rev1=1 crypto rx=0x7f35a8000c10 tx=0x7f35a8009040 comp rx=0 tx=0).stop 2024-09-16T04:07:10.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.029+0000 7f35bedc6640 10 monclient: shutdown 2024-09-16T04:07:10.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.029+0000 7f35bedc6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:07:10.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.029+0000 7f35bedc6640 1 -- 172.21.15.42:0/3254261314 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f35b80a1330 msgr2=0x7f35b813cd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:10.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.029+0000 7f35bedc6640 1 --2- 172.21.15.42:0/3254261314 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f35b80a1330 0x7f35b813cd90 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f35b001a3e0 tx=0x7f35b0019550 comp rx=0 tx=0).stop 2024-09-16T04:07:10.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.029+0000 7f35bedc6640 1 -- 172.21.15.42:0/3254261314 shutdown_connections 2024-09-16T04:07:10.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.029+0000 7f35bedc6640 1 --2- 172.21.15.42:0/3254261314 >> v2:172.21.15.154:6812/254845342 conn(0x7f35a4037810 0x7f35a4039cd0 unknown :-1 s=CLOSED pgs=3872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:10.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.029+0000 7f35bedc6640 1 --2- 172.21.15.42:0/3254261314 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f35b80a1330 0x7f35b813cd90 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:10.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.029+0000 7f35bedc6640 1 -- 172.21.15.42:0/3254261314 >> 172.21.15.42:0/3254261314 conn(0x7f35b80a2420 msgr2=0x7f35b80718b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:07:10.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.029+0000 7f35bedc6640 1 -- 172.21.15.42:0/3254261314 shutdown_connections 2024-09-16T04:07:10.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.029+0000 7f35bedc6640 1 -- 172.21.15.42:0/3254261314 wait complete. 2024-09-16T04:07:10.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.033+0000 7f35bedc6640 1 librados: shutdown 2024-09-16T04:07:10.060 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:07:10.061 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T04:07:10.061 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T04:07:10.189 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.185+0000 7fe24f57d640 10 monclient: get_monmap_and_config 2024-09-16T04:07:10.190 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.185+0000 7fe24f57d640 10 monclient: build_initial_monmap 2024-09-16T04:07:10.190 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.185+0000 7fe24f57d640 10 monclient: monmap: 2024-09-16T04:07:10.190 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:07:10.190 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:10.190 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:07:10.188275+0000 2024-09-16T04:07:10.190 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:07:10.188275+0000 2024-09-16T04:07:10.190 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:07:10.190 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:10.190 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:07:10.190 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:07:10.190 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:07:10.190 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:07:10.190 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:07:10.190 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:10.193 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.185+0000 7fe24f57d640 1 Processor -- start 2024-09-16T04:07:10.193 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.185+0000 7fe24f57d640 1 -- start start 2024-09-16T04:07:10.193 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.185+0000 7fe24f57d640 10 monclient: init 2024-09-16T04:07:10.193 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.185+0000 7fe24f57d640 10 monclient: _reopen_session rank -1 2024-09-16T04:07:10.193 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.185+0000 7fe24f57d640 10 monclient: _add_conns ranks=[4,0,3,2,1] 2024-09-16T04:07:10.193 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.185+0000 7fe24f57d640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fe2480a39b0 0x7fe248133b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:10.193 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.185+0000 7fe24f57d640 10 monclient(hunting): picked mon.noname-c con 0x7fe2480a39b0 addr v2:172.21.15.154:3300/0 2024-09-16T04:07:10.193 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.185+0000 7fe24f57d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe2480a3d90 addr v1:172.21.15.42:6789/0 2024-09-16T04:07:10.193 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.185+0000 7fe24f57d640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe248137710 0x7fe24813f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:10.193 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.185+0000 7fe24f57d640 10 monclient(hunting): picked mon.noname-a con 0x7fe248137710 addr v2:172.21.15.42:3300/0 2024-09-16T04:07:10.193 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.185+0000 7fe24f57d640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe24809c140 con 0x7fe2480a3d90 2024-09-16T04:07:10.193 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.185+0000 7fe24f57d640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:10.193 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.185+0000 7fe24f57d640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7fe24809c420 con 0x7fe248137710 2024-09-16T04:07:10.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.185+0000 7fe24f57d640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:10.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.185+0000 7fe24f57d640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fe24806dd60 con 0x7fe2480a39b0 2024-09-16T04:07:10.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.185+0000 7fe24f57d640 10 monclient(hunting): _renew_subs 2024-09-16T04:07:10.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.185+0000 7fe24f57d640 10 monclient(hunting): authenticate will time out at 2477.157744s 2024-09-16T04:07:10.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.185+0000 7fe24db1a640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe248137710 0x7fe24813f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:10.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.185+0000 7fe24cb18640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fe2480a3d90 0x7fe248137000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:42866/0 (socket says 172.21.15.42:42866) 2024-09-16T04:07:10.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.185+0000 7fe24cb18640 1 -- 172.21.15.42:0/2323743495 learned_addr learned my addr 172.21.15.42:0/2323743495 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T04:07:10.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.185+0000 7fe24db1a640 10 monclient(hunting): get_auth_request con 0x7fe248137710 auth_method 0 2024-09-16T04:07:10.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.185+0000 7fe24db1a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:10.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.185+0000 7fe24db1a640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:10.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.185+0000 7fe24db1a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:10.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.185+0000 7fe24db1a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:07:10.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.185+0000 7fe24db1a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:07:10.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.185+0000 7fe24db1a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:07:10.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.185+0000 7fe24d319640 1 --2- 172.21.15.42:0/2323743495 >> v2:172.21.15.154:3300/0 conn(0x7fe2480a39b0 0x7fe248133b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:10.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.185+0000 7fe24db1a640 10 monclient(hunting): handle_auth_done global_id 984477 payload 274 2024-09-16T04:07:10.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.185+0000 7fe24db1a640 1 -- 172.21.15.42:0/2323743495 >> v2:172.21.15.154:3300/0 conn(0x7fe2480a39b0 msgr2=0x7fe248133b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:07:10.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.185+0000 7fe24db1a640 1 --2- 172.21.15.42:0/2323743495 >> v2:172.21.15.154:3300/0 conn(0x7fe2480a39b0 0x7fe248133b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:10.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.185+0000 7fe24db1a640 1 -- 172.21.15.42:0/2323743495 >> v1:172.21.15.42:6789/0 conn(0x7fe2480a3d90 legacy=0x7fe248137000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:10.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.185+0000 7fe24db1a640 10 monclient: _finish_hunting 0 2024-09-16T04:07:10.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.185+0000 7fe24db1a640 1 monclient: found mon.noname-a 2024-09-16T04:07:10.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.185+0000 7fe23ffff640 1 -- 172.21.15.42:0/2323743495 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 996616437 0 0) 0x7fe24809c140 con 0x7fe2480a3d90 2024-09-16T04:07:10.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.185+0000 7fe24db1a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:07:10.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.185+0000 7fe24db1a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:07:10.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.185+0000 7fe24db1a640 1 -- 172.21.15.42:0/2323743495 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2480a5e20 con 0x7fe248137710 2024-09-16T04:07:10.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.185+0000 7fe24d319640 10 monclient: get_auth_request con 0x7fe2480a39b0 auth_method 0 2024-09-16T04:07:10.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.185+0000 7fe24d319640 1 --2- 172.21.15.42:0/2323743495 >> v2:172.21.15.154:3300/0 conn(0x7fe2480a39b0 0x7fe248133b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T04:07:10.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.185+0000 7fe23ffff640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T04:07:10.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.189+0000 7fe24db1a640 1 --2- 172.21.15.42:0/2323743495 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe248137710 0x7fe24813f3a0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fe2480a7750 tx=0x7fe23801c340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a67012a175f0bf19 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:07:10.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.189+0000 7fe23ffff640 1 -- 172.21.15.42:0/2323743495 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2380b0070 con 0x7fe248137710 2024-09-16T04:07:10.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.189+0000 7fe23ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:10.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.189+0000 7fe23ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:07:10.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.189+0000 7fe23ffff640 10 monclient: dump: 2024-09-16T04:07:10.202 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:07:10.202 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:10.202 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:10.202 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:10.202 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:10.202 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:10.202 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:07:10.202 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:07:10.203 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:07:10.203 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:10.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.189+0000 7fe23ffff640 10 monclient: _finish_auth 0 2024-09-16T04:07:10.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.189+0000 7fe23ffff640 10 monclient: _check_auth_tickets 2024-09-16T04:07:10.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.189+0000 7fe23ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:07:10.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.189+0000 7fe23ffff640 1 -- 172.21.15.42:0/2323743495 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe2380ab600 con 0x7fe248137710 2024-09-16T04:07:10.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.189+0000 7fe23ffff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:07:10.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.189+0000 7fe23ffff640 1 -- 172.21.15.42:0/2323743495 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2380ab9d0 con 0x7fe248137710 2024-09-16T04:07:10.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.189+0000 7fe23ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:10.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.189+0000 7fe23ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:07:10.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.189+0000 7fe23ffff640 10 monclient: dump: 2024-09-16T04:07:10.203 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:07:10.203 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:10.203 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:10.203 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:10.204 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:10.204 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:10.204 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:07:10.204 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:07:10.204 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:07:10.204 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:10.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.189+0000 7fe24f57d640 5 monclient: authenticate success, global_id 984477 2024-09-16T04:07:10.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.189+0000 7fe24f57d640 10 monclient: get_monmap_and_config success 2024-09-16T04:07:10.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.189+0000 7fe24f57d640 4 set_mon_vals no callback set 2024-09-16T04:07:10.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.189+0000 7fe24f57d640 10 monclient: shutdown 2024-09-16T04:07:10.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.189+0000 7fe24f57d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:07:10.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.189+0000 7fe24f57d640 1 -- 172.21.15.42:0/2323743495 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe248137710 msgr2=0x7fe24813f3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:10.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.189+0000 7fe24f57d640 1 --2- 172.21.15.42:0/2323743495 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe248137710 0x7fe24813f3a0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fe2480a7750 tx=0x7fe23801c340 comp rx=0 tx=0).stop 2024-09-16T04:07:10.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.189+0000 7fe24f57d640 1 -- 172.21.15.42:0/2323743495 shutdown_connections 2024-09-16T04:07:10.205 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.189+0000 7fe24f57d640 1 --2- 172.21.15.42:0/2323743495 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe248137710 0x7fe24813f3a0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:10.205 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.189+0000 7fe24f57d640 1 --2- 172.21.15.42:0/2323743495 >> v2:172.21.15.154:3300/0 conn(0x7fe2480a39b0 0x7fe248133b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:10.205 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.189+0000 7fe24f57d640 1 -- 172.21.15.42:0/2323743495 >> 172.21.15.42:0/2323743495 conn(0x7fe2480a9fb0 msgr2=0x7fe2480aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:07:10.205 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.189+0000 7fe24f57d640 1 -- 172.21.15.42:0/2323743495 shutdown_connections 2024-09-16T04:07:10.205 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.189+0000 7fe24f57d640 1 -- 172.21.15.42:0/2323743495 wait complete. 2024-09-16T04:07:10.205 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.189+0000 7fe24f57d640 10 monclient: build_initial_monmap 2024-09-16T04:07:10.205 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.189+0000 7fe24f57d640 10 monclient: monmap: 2024-09-16T04:07:10.205 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:07:10.205 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:07:10.205 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:07:10.208 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:07:10.208 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:07:10.208 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:10.209 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:07:10.209 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:07:10.209 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:07:10.209 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:10.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.189+0000 7fe24f57d640 1 librados: starting msgr at 2024-09-16T04:07:10.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.189+0000 7fe24f57d640 1 librados: starting objecter 2024-09-16T04:07:10.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.189+0000 7fe24f57d640 1 Processor -- start 2024-09-16T04:07:10.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.189+0000 7fe24f57d640 1 -- start start 2024-09-16T04:07:10.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.189+0000 7fe24f57d640 1 librados: setting wanted keys 2024-09-16T04:07:10.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.189+0000 7fe24f57d640 1 librados: calling monclient init 2024-09-16T04:07:10.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.189+0000 7fe24f57d640 10 monclient: init 2024-09-16T04:07:10.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.189+0000 7fe24f57d640 10 monclient: _reopen_session rank -1 2024-09-16T04:07:10.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.189+0000 7fe24f57d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:07:10.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.189+0000 7fe24f57d640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe2480a39b0 0x7fe248130560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:10.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.189+0000 7fe24f57d640 10 monclient(hunting): picked mon.noname-a con 0x7fe2480a39b0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:07:10.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.189+0000 7fe24f57d640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe2480a3d90 0x7fe248130aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:10.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.189+0000 7fe24f57d640 10 monclient(hunting): picked mon.noname-c con 0x7fe2480a3d90 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:07:10.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.189+0000 7fe24f57d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe248137710 0x7fe24813d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:10.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.189+0000 7fe24f57d640 10 monclient(hunting): picked mon.noname-b con 0x7fe248137710 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:07:10.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.189+0000 7fe24f57d640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:10.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.189+0000 7fe24f57d640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fe248142310 con 0x7fe2480a39b0 2024-09-16T04:07:10.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.189+0000 7fe24f57d640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:10.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.189+0000 7fe24f57d640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe248142490 con 0x7fe248137710 2024-09-16T04:07:10.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.189+0000 7fe24f57d640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:10.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.189+0000 7fe24f57d640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fe248142610 con 0x7fe2480a3d90 2024-09-16T04:07:10.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.189+0000 7fe24f57d640 10 monclient(hunting): _renew_subs 2024-09-16T04:07:10.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.189+0000 7fe24f57d640 10 monclient(hunting): authenticate will time out at 2477.163533s 2024-09-16T04:07:10.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.189+0000 7fe24d319640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe2480a39b0 0x7fe248130560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:10.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.189+0000 7fe24cb18640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe2480a3d90 0x7fe248130aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:10.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.189+0000 7fe24d319640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe2480a39b0 0x7fe248130560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:51488/0 (socket says 172.21.15.42:51488) 2024-09-16T04:07:10.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.189+0000 7fe24d319640 1 -- 172.21.15.42:0/472692053 learned_addr learned my addr 172.21.15.42:0/472692053 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:07:10.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.189+0000 7fe24cb18640 10 monclient(hunting): get_auth_request con 0x7fe2480a3d90 auth_method 0 2024-09-16T04:07:10.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.189+0000 7fe24cb18640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:10.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.189+0000 7fe24cb18640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:10.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.189+0000 7fe24cb18640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:10.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.189+0000 7fe24d319640 10 monclient(hunting): get_auth_request con 0x7fe2480a39b0 auth_method 0 2024-09-16T04:07:10.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.189+0000 7fe24d319640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:10.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.189+0000 7fe24d319640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:10.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.189+0000 7fe24d319640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:10.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.189+0000 7fe24cb18640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:07:10.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.189+0000 7fe24cb18640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:07:10.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.189+0000 7fe24cb18640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:07:10.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.189+0000 7fe24d319640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:07:10.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.189+0000 7fe24d319640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:07:10.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.189+0000 7fe24d319640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:07:10.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.189+0000 7fe24cb18640 10 monclient(hunting): handle_auth_done global_id 1014107 payload 824 2024-09-16T04:07:10.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.193+0000 7fe24db1a640 1 --2- 172.21.15.42:0/472692053 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe248137710 0x7fe24813d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:10.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.193+0000 7fe24cb18640 1 -- 172.21.15.42:0/472692053 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe248137710 msgr2=0x7fe24813d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:10.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.193+0000 7fe24cb18640 1 --2- 172.21.15.42:0/472692053 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe248137710 0x7fe24813d2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:10.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.193+0000 7fe24cb18640 1 -- 172.21.15.42:0/472692053 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe2480a39b0 msgr2=0x7fe248130560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:10.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.193+0000 7fe24cb18640 1 --2- 172.21.15.42:0/472692053 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe2480a39b0 0x7fe248130560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:10.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.193+0000 7fe24cb18640 10 monclient: _finish_hunting 0 2024-09-16T04:07:10.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.193+0000 7fe24cb18640 1 monclient: found mon.noname-c 2024-09-16T04:07:10.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.193+0000 7fe24cb18640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:07:10.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.193+0000 7fe24cb18640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T04:07:10.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.193+0000 7fe24cb18640 1 -- 172.21.15.42:0/472692053 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe24809e170 con 0x7fe2480a3d90 2024-09-16T04:07:10.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.193+0000 7fe24db1a640 10 monclient: get_auth_request con 0x7fe248137710 auth_method 0 2024-09-16T04:07:10.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.193+0000 7fe24db1a640 1 --2- 172.21.15.42:0/472692053 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe248137710 0x7fe24813d2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T04:07:10.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.193+0000 7fe24d319640 1 --2- 172.21.15.42:0/472692053 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe2480a39b0 0x7fe248130560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:07:10.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.193+0000 7fe24cb18640 1 --2- 172.21.15.42:0/472692053 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe2480a3d90 0x7fe248130aa0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fe24400c810 tx=0x7fe24400ccd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:07:10.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.193+0000 7fe23dffb640 10 client.?.objecter ms_handle_connect 0x7fe2480a3d90 2024-09-16T04:07:10.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.193+0000 7fe23dffb640 10 client.?.objecter resend_mon_ops 2024-09-16T04:07:10.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.193+0000 7fe23dffb640 1 -- 172.21.15.42:0/472692053 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe244018070 con 0x7fe2480a3d90 2024-09-16T04:07:10.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.193+0000 7fe23dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:10.214 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.193+0000 7fe23dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:07:10.214 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.193+0000 7fe23dffb640 10 monclient: dump: 2024-09-16T04:07:10.214 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:07:10.214 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:10.214 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:10.214 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:10.214 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:10.214 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:10.214 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:07:10.214 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:07:10.214 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:07:10.214 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:10.214 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.193+0000 7fe23dffb640 10 monclient: _finish_auth 0 2024-09-16T04:07:10.214 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.193+0000 7fe23dffb640 10 monclient: _check_auth_tickets 2024-09-16T04:07:10.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.193+0000 7fe23dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:07:10.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.193+0000 7fe23dffb640 1 -- 172.21.15.42:0/472692053 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe24400a640 con 0x7fe2480a3d90 2024-09-16T04:07:10.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.193+0000 7fe24f57d640 5 monclient: authenticate success, global_id 1014107 2024-09-16T04:07:10.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.193+0000 7fe24f57d640 10 monclient: _renew_subs 2024-09-16T04:07:10.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.193+0000 7fe24f57d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:07:10.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.193+0000 7fe24f57d640 1 -- 172.21.15.42:0/472692053 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe24809e450 con 0x7fe2480a3d90 2024-09-16T04:07:10.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.193+0000 7fe23dffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:07:10.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.193+0000 7fe23dffb640 1 -- 172.21.15.42:0/472692053 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe24400aa30 con 0x7fe2480a3d90 2024-09-16T04:07:10.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.193+0000 7fe23dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:10.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.193+0000 7fe23dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:07:10.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.193+0000 7fe23dffb640 10 monclient: dump: 2024-09-16T04:07:10.215 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:07:10.215 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:10.215 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:10.216 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:10.216 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:10.216 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:10.216 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:07:10.216 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:07:10.216 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:07:10.216 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:10.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.193+0000 7fe23f7fe640 4 set_mon_vals no callback set 2024-09-16T04:07:10.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.193+0000 7fe24f57d640 10 client.1014107.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:07:10.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.193+0000 7fe24f57d640 10 monclient: _renew_subs 2024-09-16T04:07:10.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.193+0000 7fe24f57d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:07:10.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.193+0000 7fe24f57d640 1 -- 172.21.15.42:0/472692053 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fe24813f4a0 con 0x7fe2480a3d90 2024-09-16T04:07:10.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.193+0000 7fe24f57d640 1 librados: init done 2024-09-16T04:07:10.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.193+0000 7fe24f57d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:07:10.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.193+0000 7fe24f57d640 1 --2- 172.21.15.42:0/472692053 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe24813dff0 0x7fe24802c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:10.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.193+0000 7fe24f57d640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:10.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.193+0000 7fe24f57d640 1 -- 172.21.15.42:0/472692053 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe2481342d0 con 0x7fe24813dff0 2024-09-16T04:07:10.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.193+0000 7fe23dffb640 1 -- 172.21.15.42:0/472692053 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fe24401b070 con 0x7fe2480a3d90 2024-09-16T04:07:10.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.193+0000 7fe23dffb640 1 --2- 172.21.15.42:0/472692053 >> v2:172.21.15.154:6812/254845342 conn(0x7fe22c037810 0x7fe22c039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:10.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.193+0000 7fe24d319640 1 --2- 172.21.15.42:0/472692053 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe24813dff0 0x7fe24802c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:10.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.193+0000 7fe24d319640 10 monclient: get_auth_request con 0x7fe24813dff0 auth_method 0 2024-09-16T04:07:10.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.193+0000 7fe24d319640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:10.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.193+0000 7fe24d319640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:10.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.193+0000 7fe24d319640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:10.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.193+0000 7fe23dffb640 1 -- 172.21.15.42:0/472692053 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(706..706 src has 451..706) ==== 5040+0+0 (secure 0 0 0) 0x7fe244048200 con 0x7fe2480a3d90 2024-09-16T04:07:10.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.193+0000 7fe23dffb640 10 client.1014107.objecter ms_dispatch 0x7fe248000ba0 osd_map(706..706 src has 451..706) 2024-09-16T04:07:10.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.193+0000 7fe23dffb640 3 client.1014107.objecter handle_osd_map got epochs [706,706] > 0 2024-09-16T04:07:10.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.193+0000 7fe23dffb640 3 client.1014107.objecter handle_osd_map decoding full epoch 706 2024-09-16T04:07:10.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.193+0000 7fe23dffb640 20 client.1014107.objecter dump_active .. 0 homeless 2024-09-16T04:07:10.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.193+0000 7fe24d319640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:07:10.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.193+0000 7fe24d319640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:07:10.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.193+0000 7fe24d319640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:07:10.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.193+0000 7fe24db1a640 1 --2- 172.21.15.42:0/472692053 >> v2:172.21.15.154:6812/254845342 conn(0x7fe22c037810 0x7fe22c039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:10.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.193+0000 7fe24db1a640 10 monclient: get_auth_request con 0x7fe22c037810 auth_method 0 2024-09-16T04:07:10.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.193+0000 7fe24d319640 10 monclient(hunting): handle_auth_done global_id 905035 payload 824 2024-09-16T04:07:10.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.193+0000 7fe24d319640 1 -- 172.21.15.42:0/472692053 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe24806f680 con 0x7fe24813dff0 2024-09-16T04:07:10.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.193+0000 7fe23dffb640 10 client.1014107.objecter ms_handle_connect 0x7fe24813dff0 2024-09-16T04:07:10.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.193+0000 7fe23dffb640 10 client.1014107.objecter resend_mon_ops 2024-09-16T04:07:10.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.193+0000 7fe24d319640 1 --2- 172.21.15.42:0/472692053 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe24813dff0 0x7fe24802c650 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fe24000ae90 tx=0x7fe24000d590 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:07:10.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.193+0000 7fe23dffb640 10 client.1014107.objecter ms_handle_connect 0x7fe22c037810 2024-09-16T04:07:10.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.193+0000 7fe24db1a640 1 --2- 172.21.15.42:0/472692053 >> v2:172.21.15.154:6812/254845342 conn(0x7fe22c037810 0x7fe22c039cd0 secure :-1 s=READY pgs=3873 cs=0 l=1 rev1=1 crypto rx=0x7fe2380ad000 tx=0x7fe238008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:07:10.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.193+0000 7fe23dffb640 1 -- 172.21.15.42:0/472692053 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe240004220 con 0x7fe24813dff0 2024-09-16T04:07:10.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.193+0000 7fe23dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:10.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.193+0000 7fe23dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:07:10.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.193+0000 7fe23dffb640 10 monclient: dump: 2024-09-16T04:07:10.219 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:07:10.219 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:10.219 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:10.219 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:10.219 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:10.219 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:10.219 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:07:10.219 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:07:10.219 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:07:10.219 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:10.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.193+0000 7fe23dffb640 1 -- 172.21.15.42:0/472692053 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fe240004220 con 0x7fe24813dff0 2024-09-16T04:07:10.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.193+0000 7fe23dffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:07:10.220 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.193+0000 7fe23dffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:07:10.220 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.193+0000 7fe23dffb640 1 -- 172.21.15.42:0/472692053 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe24813dff0 msgr2=0x7fe24802c650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:10.220 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.193+0000 7fe23dffb640 1 --2- 172.21.15.42:0/472692053 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe24813dff0 0x7fe24802c650 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fe24000ae90 tx=0x7fe24000d590 comp rx=0 tx=0).stop 2024-09-16T04:07:10.220 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.193+0000 7fe23dffb640 1 -- 172.21.15.42:0/472692053 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe24813dff0 msgr2=0x7fe24802c650 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:07:10.220 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.193+0000 7fe23dffb640 1 --2- 172.21.15.42:0/472692053 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe24813dff0 0x7fe24802c650 secure :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fe24000ae90 tx=0x7fe24000d590 comp rx=0 tx=0).stop 2024-09-16T04:07:10.222 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.201+0000 7fe24f57d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:07:10.222 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.201+0000 7fe24f57d640 1 --2- 172.21.15.42:0/472692053 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe24813e3d0 0x7fe24802cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:10.222 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.201+0000 7fe24f57d640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:10.222 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.201+0000 7fe24f57d640 1 -- 172.21.15.42:0/472692053 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe2480a5750 con 0x7fe24813e3d0 2024-09-16T04:07:10.222 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.201+0000 7fe24d319640 1 --2- 172.21.15.42:0/472692053 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe24813e3d0 0x7fe24802cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:10.222 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.201+0000 7fe24d319640 10 monclient: get_auth_request con 0x7fe24813e3d0 auth_method 0 2024-09-16T04:07:10.222 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.201+0000 7fe24d319640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:10.222 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.201+0000 7fe24d319640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:10.222 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.201+0000 7fe24d319640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:10.222 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.201+0000 7fe24d319640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:07:10.222 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.201+0000 7fe24d319640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:07:10.222 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.201+0000 7fe24d319640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:07:10.222 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.201+0000 7fe24d319640 10 monclient(hunting): handle_auth_done global_id 905038 payload 824 2024-09-16T04:07:10.223 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.201+0000 7fe24d319640 1 -- 172.21.15.42:0/472692053 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fe248139840 con 0x7fe24813e3d0 2024-09-16T04:07:10.223 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.201+0000 7fe23dffb640 10 client.1014107.objecter ms_handle_connect 0x7fe24813e3d0 2024-09-16T04:07:10.223 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.201+0000 7fe24d319640 1 --2- 172.21.15.42:0/472692053 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe24813e3d0 0x7fe24802cb90 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fe24000ae90 tx=0x7fe2400046d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:07:10.223 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.201+0000 7fe23dffb640 10 client.1014107.objecter resend_mon_ops 2024-09-16T04:07:10.223 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.201+0000 7fe23dffb640 1 -- 172.21.15.42:0/472692053 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe240003c40 con 0x7fe24813e3d0 2024-09-16T04:07:10.223 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.201+0000 7fe23dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:10.223 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.201+0000 7fe23dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:07:10.223 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.201+0000 7fe23dffb640 10 monclient: dump: 2024-09-16T04:07:10.223 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:07:10.223 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:10.223 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:10.223 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:10.224 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:10.224 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:10.224 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:07:10.224 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:07:10.224 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:07:10.224 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:10.224 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.201+0000 7fe23dffb640 1 -- 172.21.15.42:0/472692053 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4880 (secure 0 0 0) 0x7fe240003c40 con 0x7fe24813e3d0 2024-09-16T04:07:10.224 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.205+0000 7fe23dffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:07:10.224 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.205+0000 7fe23dffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:07:10.224 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.205+0000 7fe23dffb640 1 -- 172.21.15.42:0/472692053 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe24813e3d0 msgr2=0x7fe24802cb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:10.224 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.205+0000 7fe23dffb640 1 --2- 172.21.15.42:0/472692053 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe24813e3d0 0x7fe24802cb90 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fe24000ae90 tx=0x7fe2400046d0 comp rx=0 tx=0).stop 2024-09-16T04:07:10.224 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.205+0000 7fe23dffb640 1 -- 172.21.15.42:0/472692053 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe24813e3d0 msgr2=0x7fe24802cb90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:07:10.224 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.205+0000 7fe23dffb640 1 --2- 172.21.15.42:0/472692053 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe24813e3d0 0x7fe24802cb90 secure :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fe24000ae90 tx=0x7fe2400046d0 comp rx=0 tx=0).stop 2024-09-16T04:07:10.225 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T04:07:10.225 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:07:10.225 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:07:10.225 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T04:07:10.225 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 232, 2024-09-16T04:07:10.225 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 558382, 2024-09-16T04:07:10.225 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T04:07:10.225 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T04:07:10.225 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T04:07:10.225 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T04:07:10.225 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:07:10.225 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 5, 2024-09-16T04:07:10.226 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:07:10.226 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T04:07:10.226 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T04:07:10.226 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:07:10.226 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:07:10.226 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:07:10.226 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:07:10.226 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:07:10.226 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:07:10.226 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:07:10.226 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:07:10.226 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:07:10.226 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:07:10.226 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:07:10.226 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:07:10.227 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:07:10.227 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T04:07:10.227 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:07:10.227 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:07:10.227 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:07:10.227 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:07:10.227 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:07:10.227 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:07:10.227 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:07:10.227 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:07:10.227 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:07:10.227 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:07:10.227 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:07:10.228 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:07:10.228 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:10.228 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T04:07:10.228 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T04:07:10.228 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T04:07:10.228 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T04:07:10.228 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T04:07:10.228 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T04:07:10.228 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:07:10.228 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:07:10.228 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T04:07:10.228 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T04:07:10.228 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T04:07:10.229 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T04:07:10.229 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T04:07:10.229 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T04:07:10.229 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T04:07:10.229 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:07:10.229 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T04:07:10.229 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:07:10.229 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:07:10.229 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:07:10.229 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:07:10.229 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:07:10.230 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:07:10.230 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:07:10.230 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:07:10.230 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:07:10.230 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:07:10.230 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:07:10.230 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:07:10.230 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T04:07:10.230 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:10.230 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T04:07:10.230 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:10.230 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:07:10.230 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:07:10.230 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:07:10.231 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:07:10.231 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:10.231 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:07:10.231 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T04:07:10.231 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:07:10.231 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:10.231 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:10.231 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:07:10.231 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T04:07:10.231 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:07:10.231 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:07:10.231 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:07:10.231 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:10.231 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T04:07:10.231 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T04:07:10.232 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:07:10.232 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:07:10.232 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:07:10.232 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:10.232 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:10.232 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:07:10.232 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:07:10.232 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:07:10.232 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:07:10.232 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:10.232 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:07:10.232 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T04:07:10.232 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:07:10.232 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:10.233 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:10.233 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:07:10.233 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T04:07:10.233 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:07:10.233 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:07:10.233 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:07:10.233 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:10.233 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T04:07:10.233 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T04:07:10.233 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:07:10.233 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:07:10.233 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:07:10.233 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:10.233 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:10.234 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:07:10.234 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:07:10.234 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:07:10.234 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:07:10.234 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:10.234 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:07:10.234 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T04:07:10.234 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:07:10.234 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:10.234 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:10.234 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:07:10.234 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T04:07:10.234 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:07:10.234 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:07:10.235 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:07:10.235 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:10.235 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T04:07:10.235 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T04:07:10.235 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:07:10.235 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:07:10.235 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:07:10.235 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:07:10.235 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:07:10.235 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:10.235 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T04:07:10.235 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T04:07:10.235 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:10.235 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:07:10.235 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:07:10.236 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:07:10.236 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:07:10.236 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:07:10.236 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T04:07:10.236 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:10.236 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:07:10.236 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:07:10.236 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 3 2024-09-16T04:07:10.236 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:07:10.236 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:07:10.236 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T04:07:10.236 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:10.236 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:07:10.236 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:07:10.237 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 5 2024-09-16T04:07:10.237 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:07:10.237 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:07:10.237 INFO:teuthology.orchestra.run.smithi042.stdout: "mgr": [ 2024-09-16T04:07:10.237 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:10.237 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:07:10.237 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:07:10.237 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:07:10.237 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:07:10.237 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:07:10.237 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:10.237 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T04:07:10.237 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T04:07:10.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.205+0000 7fe24f57d640 10 librados: watch_flush enter 2024-09-16T04:07:10.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.205+0000 7fe24f57d640 10 librados: watch_flush exit 2024-09-16T04:07:10.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.205+0000 7fe24f57d640 20 client.1014107.objecter shutdown clearing up homeless session... 2024-09-16T04:07:10.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.205+0000 7fe24f57d640 10 client.1014107.objecter successfully canceled tick 2024-09-16T04:07:10.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.205+0000 7fe24f57d640 1 -- 172.21.15.42:0/472692053 >> v2:172.21.15.154:6812/254845342 conn(0x7fe22c037810 msgr2=0x7fe22c039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:10.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.205+0000 7fe24f57d640 1 --2- 172.21.15.42:0/472692053 >> v2:172.21.15.154:6812/254845342 conn(0x7fe22c037810 0x7fe22c039cd0 secure :-1 s=READY pgs=3873 cs=0 l=1 rev1=1 crypto rx=0x7fe2380ad000 tx=0x7fe238008040 comp rx=0 tx=0).stop 2024-09-16T04:07:10.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.205+0000 7fe24f57d640 10 monclient: shutdown 2024-09-16T04:07:10.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.205+0000 7fe24f57d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:07:10.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.205+0000 7fe24f57d640 1 -- 172.21.15.42:0/472692053 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe2480a3d90 msgr2=0x7fe248130aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:10.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.205+0000 7fe24f57d640 1 --2- 172.21.15.42:0/472692053 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe2480a3d90 0x7fe248130aa0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fe24400c810 tx=0x7fe24400ccd0 comp rx=0 tx=0).stop 2024-09-16T04:07:10.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.205+0000 7fe24db1a640 1 -- 172.21.15.42:0/472692053 reap_dead start 2024-09-16T04:07:10.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.205+0000 7fe24f57d640 1 -- 172.21.15.42:0/472692053 shutdown_connections 2024-09-16T04:07:10.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.205+0000 7fe24f57d640 1 --2- 172.21.15.42:0/472692053 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe2480a3d90 0x7fe248130aa0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:10.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.205+0000 7fe24f57d640 1 -- 172.21.15.42:0/472692053 >> 172.21.15.42:0/472692053 conn(0x7fe2480a9fb0 msgr2=0x7fe248131fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:07:10.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.205+0000 7fe24f57d640 1 -- 172.21.15.42:0/472692053 shutdown_connections 2024-09-16T04:07:10.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.205+0000 7fe24f57d640 1 -- 172.21.15.42:0/472692053 wait complete. 2024-09-16T04:07:10.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.205+0000 7fe24f57d640 1 librados: shutdown 2024-09-16T04:07:10.240 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:07:10.240 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T04:07:10.241 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T04:07:10.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.349+0000 7f3375901640 10 monclient: get_monmap_and_config 2024-09-16T04:07:10.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.349+0000 7f3375901640 10 monclient: build_initial_monmap 2024-09-16T04:07:10.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.349+0000 7f3375901640 10 monclient: monmap: 2024-09-16T04:07:10.356 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:07:10.356 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:10.357 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:07:10.354985+0000 2024-09-16T04:07:10.357 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:07:10.354985+0000 2024-09-16T04:07:10.357 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:07:10.357 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:10.357 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:07:10.357 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:07:10.357 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:07:10.357 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:07:10.357 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:07:10.357 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:10.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.349+0000 7f3375901640 1 Processor -- start 2024-09-16T04:07:10.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.349+0000 7f3375901640 1 -- start start 2024-09-16T04:07:10.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.349+0000 7f3375901640 10 monclient: init 2024-09-16T04:07:10.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.353+0000 7f3375901640 10 monclient: _reopen_session rank -1 2024-09-16T04:07:10.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.353+0000 7f3375901640 10 monclient: _add_conns ranks=[2,0,3,4,1] 2024-09-16T04:07:10.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.353+0000 7f3375901640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f33700a1330 0x7f33700a1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:10.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.353+0000 7f3375901640 10 monclient(hunting): picked mon.noname-b con 0x7f33700a1330 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:07:10.360 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.353+0000 7f3375901640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f337006dac0 addr v1:172.21.15.42:6789/0 2024-09-16T04:07:10.360 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.353+0000 7f3375901640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f33700a1c50 0x7f3370135810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:10.360 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.353+0000 7f3375901640 10 monclient(hunting): picked mon.noname-a con 0x7f33700a1c50 addr v2:172.21.15.42:3300/0 2024-09-16T04:07:10.360 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.353+0000 7f3375901640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f337009c090 con 0x7f337006dac0 2024-09-16T04:07:10.360 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.353+0000 7f3375901640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:10.360 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.353+0000 7f3375901640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f337009c890 con 0x7f33700a1c50 2024-09-16T04:07:10.360 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.353+0000 7f3375901640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:10.360 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.353+0000 7f3375901640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f337009f610 con 0x7f33700a1330 2024-09-16T04:07:10.360 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.353+0000 7f3375901640 10 monclient(hunting): _renew_subs 2024-09-16T04:07:10.360 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.353+0000 7f3375901640 10 monclient(hunting): authenticate will time out at 2477.324455s 2024-09-16T04:07:10.360 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.353+0000 7f336effd640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f33700a1330 0x7f33700a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:10.360 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.353+0000 7f336f7fe640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f33700a1c50 0x7f3370135810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:10.360 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.353+0000 7f336e7fc640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f337006dac0 0x7f3370135100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:42882/0 (socket says 172.21.15.42:42882) 2024-09-16T04:07:10.361 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.353+0000 7f336e7fc640 1 -- 172.21.15.42:0/2675113635 learned_addr learned my addr 172.21.15.42:0/2675113635 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T04:07:10.361 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.353+0000 7f336f7fe640 10 monclient(hunting): get_auth_request con 0x7f33700a1c50 auth_method 0 2024-09-16T04:07:10.361 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.353+0000 7f336f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:10.361 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.353+0000 7f336f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:10.361 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.353+0000 7f336f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:10.361 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.353+0000 7f336effd640 10 monclient(hunting): get_auth_request con 0x7f33700a1330 auth_method 0 2024-09-16T04:07:10.361 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.353+0000 7f336effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:10.361 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.353+0000 7f336effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:10.361 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.353+0000 7f336effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:10.361 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.353+0000 7f336f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:07:10.361 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.353+0000 7f336f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:07:10.361 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.353+0000 7f336f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:07:10.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.353+0000 7f336effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:07:10.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.353+0000 7f336effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:07:10.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.353+0000 7f336effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:07:10.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.353+0000 7f336f7fe640 10 monclient(hunting): handle_auth_done global_id 984486 payload 274 2024-09-16T04:07:10.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.353+0000 7f336f7fe640 1 -- 172.21.15.42:0/2675113635 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f33700a1330 msgr2=0x7f33700a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:07:10.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.353+0000 7f336f7fe640 1 --2- 172.21.15.42:0/2675113635 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f33700a1330 0x7f33700a1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:10.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.353+0000 7f336f7fe640 1 -- 172.21.15.42:0/2675113635 >> v1:172.21.15.42:6789/0 conn(0x7f337006dac0 legacy=0x7f3370135100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:10.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.353+0000 7f336f7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:07:10.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.353+0000 7f336f7fe640 1 monclient: found mon.noname-a 2024-09-16T04:07:10.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.353+0000 7f336f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:07:10.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.353+0000 7f336f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:07:10.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.353+0000 7f336f7fe640 1 -- 172.21.15.42:0/2675113635 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f337009f8f0 con 0x7f33700a1c50 2024-09-16T04:07:10.368 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.353+0000 7f336f7fe640 1 --2- 172.21.15.42:0/2675113635 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f33700a1c50 0x7f3370135810 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f33680097c0 tx=0x7f336801c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=775459f7a86dac56 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:07:10.368 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.353+0000 7f336dffb640 1 -- 172.21.15.42:0/2675113635 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33680b0070 con 0x7f33700a1c50 2024-09-16T04:07:10.368 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.353+0000 7f336dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:10.369 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.353+0000 7f336dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:07:10.369 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.353+0000 7f336dffb640 10 monclient: dump: 2024-09-16T04:07:10.369 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:07:10.369 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:10.369 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:10.369 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:10.369 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:10.369 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:10.369 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:07:10.369 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:07:10.369 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:07:10.369 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:10.369 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.353+0000 7f336dffb640 10 monclient: _finish_auth 0 2024-09-16T04:07:10.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.353+0000 7f336dffb640 10 monclient: _check_auth_tickets 2024-09-16T04:07:10.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.353+0000 7f336dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:07:10.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.353+0000 7f3375901640 5 monclient: authenticate success, global_id 984486 2024-09-16T04:07:10.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.353+0000 7f3375901640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:07:10.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.353+0000 7f336dffb640 1 -- 172.21.15.42:0/2675113635 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3368004230 con 0x7f33700a1c50 2024-09-16T04:07:10.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.353+0000 7f336dffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:07:10.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.353+0000 7f336dffb640 1 -- 172.21.15.42:0/2675113635 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3368003620 con 0x7f33700a1c50 2024-09-16T04:07:10.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.353+0000 7f3375901640 10 monclient: get_monmap_and_config success 2024-09-16T04:07:10.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.353+0000 7f3375901640 4 set_mon_vals no callback set 2024-09-16T04:07:10.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.353+0000 7f336dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:10.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.353+0000 7f336dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:07:10.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.353+0000 7f336dffb640 10 monclient: dump: 2024-09-16T04:07:10.370 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:07:10.371 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:10.371 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:10.371 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:10.371 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:10.371 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:10.371 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:07:10.371 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:07:10.371 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:07:10.371 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:10.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.353+0000 7f3375901640 10 monclient: shutdown 2024-09-16T04:07:10.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.353+0000 7f3375901640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:07:10.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.353+0000 7f3375901640 1 -- 172.21.15.42:0/2675113635 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f33700a1c50 msgr2=0x7f3370135810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:10.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.353+0000 7f3375901640 1 --2- 172.21.15.42:0/2675113635 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f33700a1c50 0x7f3370135810 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f33680097c0 tx=0x7f336801c3c0 comp rx=0 tx=0).stop 2024-09-16T04:07:10.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.353+0000 7f3375901640 1 -- 172.21.15.42:0/2675113635 shutdown_connections 2024-09-16T04:07:10.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.353+0000 7f3375901640 1 --2- 172.21.15.42:0/2675113635 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f33700a1c50 0x7f3370135810 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:10.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.353+0000 7f3375901640 1 --2- 172.21.15.42:0/2675113635 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f33700a1330 0x7f33700a1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:10.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.353+0000 7f3375901640 1 -- 172.21.15.42:0/2675113635 >> 172.21.15.42:0/2675113635 conn(0x7f33700a2420 msgr2=0x7f33700a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:07:10.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.357+0000 7f3375901640 1 -- 172.21.15.42:0/2675113635 shutdown_connections 2024-09-16T04:07:10.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.357+0000 7f3375901640 1 -- 172.21.15.42:0/2675113635 wait complete. 2024-09-16T04:07:10.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.357+0000 7f3375901640 10 monclient: build_initial_monmap 2024-09-16T04:07:10.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.357+0000 7f3375901640 10 monclient: monmap: 2024-09-16T04:07:10.372 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:07:10.372 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:07:10.372 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:07:10.372 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:07:10.373 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:07:10.373 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:10.373 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:07:10.373 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:07:10.373 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:07:10.373 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:10.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.357+0000 7f3375901640 1 librados: starting msgr at 2024-09-16T04:07:10.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.357+0000 7f3375901640 1 librados: starting objecter 2024-09-16T04:07:10.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.357+0000 7f3375901640 1 Processor -- start 2024-09-16T04:07:10.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.357+0000 7f3375901640 1 -- start start 2024-09-16T04:07:10.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.357+0000 7f3375901640 1 librados: setting wanted keys 2024-09-16T04:07:10.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.357+0000 7f3375901640 1 librados: calling monclient init 2024-09-16T04:07:10.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.357+0000 7f3375901640 10 monclient: init 2024-09-16T04:07:10.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.357+0000 7f3375901640 10 monclient: _reopen_session rank -1 2024-09-16T04:07:10.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.357+0000 7f3375901640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:07:10.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.357+0000 7f3375901640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f33700a1330 0x7f33701cca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:10.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.357+0000 7f3375901640 10 monclient(hunting): picked mon.noname-a con 0x7f33700a1330 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:07:10.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.357+0000 7f3375901640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f33700a1c50 0x7f3370134ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:10.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.357+0000 7f3375901640 10 monclient(hunting): picked mon.noname-c con 0x7f33700a1c50 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:07:10.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.357+0000 7f3375901640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3370135430 0x7f337011bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:10.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.357+0000 7f3375901640 10 monclient(hunting): picked mon.noname-b con 0x7f3370135430 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:07:10.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.357+0000 7f3375901640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:10.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.357+0000 7f3375901640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f337009eef0 con 0x7f33700a1330 2024-09-16T04:07:10.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.357+0000 7f3375901640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:10.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.357+0000 7f3375901640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f337009ed70 con 0x7f3370135430 2024-09-16T04:07:10.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.357+0000 7f3375901640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:10.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.357+0000 7f3375901640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f3370135d50 con 0x7f33700a1c50 2024-09-16T04:07:10.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.357+0000 7f3375901640 10 monclient(hunting): _renew_subs 2024-09-16T04:07:10.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.357+0000 7f3375901640 10 monclient(hunting): authenticate will time out at 2477.329460s 2024-09-16T04:07:10.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.357+0000 7f336effd640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f33700a1330 0x7f33701cca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:10.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.357+0000 7f336e7fc640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f33700a1c50 0x7f3370134ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:10.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.357+0000 7f336effd640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f33700a1330 0x7f33701cca80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:51510/0 (socket says 172.21.15.42:51510) 2024-09-16T04:07:10.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.357+0000 7f336effd640 1 -- 172.21.15.42:0/3754706975 learned_addr learned my addr 172.21.15.42:0/3754706975 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:07:10.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.357+0000 7f336effd640 10 monclient(hunting): get_auth_request con 0x7f33700a1330 auth_method 0 2024-09-16T04:07:10.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.357+0000 7f336effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:10.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.357+0000 7f336effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:10.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.357+0000 7f336effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:10.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.357+0000 7f336e7fc640 10 monclient(hunting): get_auth_request con 0x7f33700a1c50 auth_method 0 2024-09-16T04:07:10.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.357+0000 7f336e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:10.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.357+0000 7f336e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:10.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.357+0000 7f336e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:10.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.357+0000 7f336effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:07:10.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.357+0000 7f336effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:07:10.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.357+0000 7f336effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:07:10.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.357+0000 7f336e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:07:10.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.357+0000 7f336e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:07:10.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.357+0000 7f336e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:07:10.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.357+0000 7f336effd640 10 monclient(hunting): handle_auth_done global_id 984492 payload 824 2024-09-16T04:07:10.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.357+0000 7f336effd640 1 -- 172.21.15.42:0/3754706975 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f33700a1c50 msgr2=0x7f3370134ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:10.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.357+0000 7f336effd640 1 --2- 172.21.15.42:0/3754706975 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f33700a1c50 0x7f3370134ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:10.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.357+0000 7f336effd640 1 -- 172.21.15.42:0/3754706975 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3370135430 msgr2=0x7f337011bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:10.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.357+0000 7f336effd640 1 --2- 172.21.15.42:0/3754706975 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3370135430 0x7f337011bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:10.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.357+0000 7f336effd640 10 monclient: _finish_hunting 0 2024-09-16T04:07:10.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.357+0000 7f336effd640 1 monclient: found mon.noname-a 2024-09-16T04:07:10.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.357+0000 7f336effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:07:10.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.357+0000 7f336effd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:07:10.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.357+0000 7f336effd640 1 -- 172.21.15.42:0/3754706975 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f337011e0f0 con 0x7f33700a1330 2024-09-16T04:07:10.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.357+0000 7f336effd640 1 --2- 172.21.15.42:0/3754706975 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f33700a1330 0x7f33701cca80 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f3360009db0 tx=0x7f336000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:07:10.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.357+0000 7f3367fff640 10 client.?.objecter ms_handle_connect 0x7f33700a1330 2024-09-16T04:07:10.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.357+0000 7f3367fff640 10 client.?.objecter resend_mon_ops 2024-09-16T04:07:10.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.357+0000 7f3367fff640 1 -- 172.21.15.42:0/3754706975 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3360018020 con 0x7f33700a1330 2024-09-16T04:07:10.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.357+0000 7f3367fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:10.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.357+0000 7f3367fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:07:10.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.357+0000 7f3367fff640 10 monclient: dump: 2024-09-16T04:07:10.377 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:07:10.377 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:10.378 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:10.378 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:10.378 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:10.378 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:10.378 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:07:10.378 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:07:10.378 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:07:10.378 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:10.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.357+0000 7f3367fff640 10 monclient: _finish_auth 0 2024-09-16T04:07:10.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.357+0000 7f3367fff640 10 monclient: _check_auth_tickets 2024-09-16T04:07:10.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.357+0000 7f3367fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:07:10.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.357+0000 7f3367fff640 1 -- 172.21.15.42:0/3754706975 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3360014480 con 0x7f33700a1330 2024-09-16T04:07:10.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.357+0000 7f3367fff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:07:10.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.357+0000 7f3367fff640 1 -- 172.21.15.42:0/3754706975 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3360014850 con 0x7f33700a1330 2024-09-16T04:07:10.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.357+0000 7f3375901640 5 monclient: authenticate success, global_id 984492 2024-09-16T04:07:10.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.357+0000 7f3375901640 10 monclient: _renew_subs 2024-09-16T04:07:10.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.357+0000 7f3375901640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:07:10.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.357+0000 7f3375901640 1 -- 172.21.15.42:0/3754706975 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f337011f880 con 0x7f33700a1330 2024-09-16T04:07:10.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.357+0000 7f336d7fa640 4 set_mon_vals no callback set 2024-09-16T04:07:10.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.357+0000 7f3367fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:10.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.357+0000 7f3367fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:07:10.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.357+0000 7f3367fff640 10 monclient: dump: 2024-09-16T04:07:10.379 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:07:10.379 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:10.379 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:10.379 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:10.379 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:10.379 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:10.380 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:07:10.380 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:07:10.380 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:07:10.380 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:10.380 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.357+0000 7f3375901640 10 client.984492.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:07:10.380 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.357+0000 7f3375901640 10 monclient: _renew_subs 2024-09-16T04:07:10.380 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.357+0000 7f3375901640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:07:10.380 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.357+0000 7f3375901640 1 -- 172.21.15.42:0/3754706975 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f337011e310 con 0x7f33700a1330 2024-09-16T04:07:10.380 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.357+0000 7f3375901640 1 librados: init done 2024-09-16T04:07:10.380 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.357+0000 7f3367fff640 1 -- 172.21.15.42:0/3754706975 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f336001b020 con 0x7f33700a1330 2024-09-16T04:07:10.380 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.357+0000 7f3375901640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:07:10.380 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.357+0000 7f3375901640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:07:10.380 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.357+0000 7f3375901640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:07:10.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.357+0000 7f3375901640 1 -- 172.21.15.42:0/3754706975 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f33700a1710 con 0x7f33700a1330 2024-09-16T04:07:10.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.357+0000 7f3367fff640 1 --2- 172.21.15.42:0/3754706975 >> v2:172.21.15.154:6812/254845342 conn(0x7f3350037810 0x7f3350039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:10.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.357+0000 7f3367fff640 1 -- 172.21.15.42:0/3754706975 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(706..706 src has 451..706) ==== 5040+0+0 (secure 0 0 0) 0x7f3360049690 con 0x7f33700a1330 2024-09-16T04:07:10.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.357+0000 7f3367fff640 10 client.984492.objecter ms_dispatch 0x7f3370000ba0 osd_map(706..706 src has 451..706) 2024-09-16T04:07:10.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.357+0000 7f3367fff640 3 client.984492.objecter handle_osd_map got epochs [706,706] > 0 2024-09-16T04:07:10.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.357+0000 7f3367fff640 3 client.984492.objecter handle_osd_map decoding full epoch 706 2024-09-16T04:07:10.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.357+0000 7f3367fff640 20 client.984492.objecter dump_active .. 0 homeless 2024-09-16T04:07:10.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.357+0000 7f336e7fc640 1 --2- 172.21.15.42:0/3754706975 >> v2:172.21.15.154:6812/254845342 conn(0x7f3350037810 0x7f3350039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:10.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.361+0000 7f336e7fc640 10 monclient: get_auth_request con 0x7f3350037810 auth_method 0 2024-09-16T04:07:10.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.361+0000 7f336e7fc640 1 --2- 172.21.15.42:0/3754706975 >> v2:172.21.15.154:6812/254845342 conn(0x7f3350037810 0x7f3350039cd0 secure :-1 s=READY pgs=3874 cs=0 l=1 rev1=1 crypto rx=0x7f337013cfd0 tx=0x7f3358007e90 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:07:10.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.361+0000 7f3367fff640 10 client.984492.objecter ms_handle_connect 0x7f3350037810 2024-09-16T04:07:10.382 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.365+0000 7f3367fff640 1 -- 172.21.15.42:0/3754706975 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f3360022070 con 0x7f33700a1330 2024-09-16T04:07:10.382 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.365+0000 7f3367fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:07:10.382 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.365+0000 7f3367fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:07:10.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.589+0000 7f3375901640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T04:07:10.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.589+0000 7f3375901640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T04:07:10.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.589+0000 7f3375901640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:07:10.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.589+0000 7f3375901640 1 -- 172.21.15.42:0/3754706975 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f337011de70 con 0x7f33700a1330 2024-09-16T04:07:10.596 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.589+0000 7f3367fff640 1 -- 172.21.15.42:0/3754706975 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f3360010040 con 0x7f33700a1330 2024-09-16T04:07:10.596 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.589+0000 7f3367fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T04:07:10.596 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.589+0000 7f3367fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:07:10.597 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.589+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.597 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.589+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=29,config=2,health=12,kv=8,logm=49} crc {auth=1116919813,config=692619927,health=1281989808,kv=352998907,logm=3673901496}) 2024-09-16T04:07:10.597 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.589+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=28,config=2,health=12,kv=8,logm=50} crc {auth=2898509361,config=692619927,health=1281989808,kv=2323084916,logm=2924471888}) 2024-09-16T04:07:10.597 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.589+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.597 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.589+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=29,config=2,health=12,kv=8,logm=49} crc {auth=1116919813,config=692619927,health=1281989808,kv=352998907,logm=3673901496}) 2024-09-16T04:07:10.597 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.589+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=29,config=2,health=11,kv=10,logm=48} crc {auth=1116919813,config=692619927,health=2720041106,kv=3446188498,logm=2056026682}) 2024-09-16T04:07:10.598 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.593+0000 7f3375901640 10 librados: watch_flush enter 2024-09-16T04:07:10.599 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.593+0000 7f3375901640 10 librados: watch_flush exit 2024-09-16T04:07:10.599 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.593+0000 7f3375901640 20 client.984492.objecter shutdown clearing up homeless session... 2024-09-16T04:07:10.599 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.593+0000 7f3375901640 10 client.984492.objecter successfully canceled tick 2024-09-16T04:07:10.599 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.593+0000 7f3375901640 1 -- 172.21.15.42:0/3754706975 >> v2:172.21.15.154:6812/254845342 conn(0x7f3350037810 msgr2=0x7f3350039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:10.599 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.593+0000 7f3375901640 1 --2- 172.21.15.42:0/3754706975 >> v2:172.21.15.154:6812/254845342 conn(0x7f3350037810 0x7f3350039cd0 secure :-1 s=READY pgs=3874 cs=0 l=1 rev1=1 crypto rx=0x7f337013cfd0 tx=0x7f3358007e90 comp rx=0 tx=0).stop 2024-09-16T04:07:10.599 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.593+0000 7f3375901640 10 monclient: shutdown 2024-09-16T04:07:10.599 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.593+0000 7f3375901640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:07:10.599 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.593+0000 7f3375901640 1 -- 172.21.15.42:0/3754706975 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f33700a1330 msgr2=0x7f33701cca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:10.599 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.593+0000 7f3375901640 1 --2- 172.21.15.42:0/3754706975 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f33700a1330 0x7f33701cca80 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f3360009db0 tx=0x7f336000c490 comp rx=0 tx=0).stop 2024-09-16T04:07:10.599 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.593+0000 7f3375901640 1 -- 172.21.15.42:0/3754706975 shutdown_connections 2024-09-16T04:07:10.599 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.593+0000 7f3375901640 1 --2- 172.21.15.42:0/3754706975 >> v2:172.21.15.154:6812/254845342 conn(0x7f3350037810 0x7f3350039cd0 unknown :-1 s=CLOSED pgs=3874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:10.599 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.593+0000 7f3375901640 1 --2- 172.21.15.42:0/3754706975 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3370135430 0x7f337011bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:10.599 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.593+0000 7f3375901640 1 --2- 172.21.15.42:0/3754706975 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f33700a1c50 0x7f3370134ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:10.600 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.593+0000 7f3375901640 1 --2- 172.21.15.42:0/3754706975 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f33700a1330 0x7f33701cca80 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:10.600 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.593+0000 7f3375901640 1 -- 172.21.15.42:0/3754706975 >> 172.21.15.42:0/3754706975 conn(0x7f33700a2420 msgr2=0x7f337006f900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:07:10.600 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.593+0000 7f3375901640 1 -- 172.21.15.42:0/3754706975 shutdown_connections 2024-09-16T04:07:10.600 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.593+0000 7f3375901640 1 -- 172.21.15.42:0/3754706975 wait complete. 2024-09-16T04:07:10.600 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:10.593+0000 7f3375901640 1 librados: shutdown 2024-09-16T04:07:10.600 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.593+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.601 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.593+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3081963874}) 2024-09-16T04:07:10.601 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.593+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=836961685}) 2024-09-16T04:07:10.601 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.593+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.601 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.593+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3081963874}) 2024-09-16T04:07:10.601 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.593+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=593783795}) 2024-09-16T04:07:10.602 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.597+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.602 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.597+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3945745574}) 2024-09-16T04:07:10.602 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.597+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2376389331}) 2024-09-16T04:07:10.602 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.597+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.603 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.597+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3945745574}) 2024-09-16T04:07:10.603 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.597+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2597828239}) 2024-09-16T04:07:10.605 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.597+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.605 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.597+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=341248823}) 2024-09-16T04:07:10.605 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.597+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1047318361}) 2024-09-16T04:07:10.605 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.597+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.605 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.597+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=341248823}) 2024-09-16T04:07:10.605 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.597+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4193186278}) 2024-09-16T04:07:10.608 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.601+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.608 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.601+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1290546300}) 2024-09-16T04:07:10.608 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.601+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4099317360}) 2024-09-16T04:07:10.608 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.601+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.608 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.601+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1290546300}) 2024-09-16T04:07:10.608 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.601+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=263627894}) 2024-09-16T04:07:10.610 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.605+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.610 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.605+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3892241660}) 2024-09-16T04:07:10.610 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.605+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1423000117}) 2024-09-16T04:07:10.610 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.605+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.610 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.605+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3892241660}) 2024-09-16T04:07:10.610 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.605+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=42329428}) 2024-09-16T04:07:10.612 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.605+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.612 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.605+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1086777756}) 2024-09-16T04:07:10.612 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.605+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1120644125}) 2024-09-16T04:07:10.612 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.605+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.612 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.605+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1086777756}) 2024-09-16T04:07:10.612 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.605+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3308560495}) 2024-09-16T04:07:10.613 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.609+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.613 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.609+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2206081932}) 2024-09-16T04:07:10.613 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.609+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2190253199}) 2024-09-16T04:07:10.613 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.609+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.614 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.609+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2206081932}) 2024-09-16T04:07:10.614 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.609+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1923308528}) 2024-09-16T04:07:10.615 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.609+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.615 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.609+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=281297590}) 2024-09-16T04:07:10.615 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.609+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1716158614}) 2024-09-16T04:07:10.615 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.609+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.615 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.609+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=281297590}) 2024-09-16T04:07:10.615 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.609+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2754484529}) 2024-09-16T04:07:10.616 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T04:07:10.617 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.609+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.617 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.609+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3738017075}) 2024-09-16T04:07:10.617 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.609+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1123380164}) 2024-09-16T04:07:10.617 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.609+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.617 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.609+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3738017075}) 2024-09-16T04:07:10.617 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.609+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=740237935}) 2024-09-16T04:07:10.618 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.613+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.618 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.613+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2415699880}) 2024-09-16T04:07:10.619 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.613+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1221392920}) 2024-09-16T04:07:10.619 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.613+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.619 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.613+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2415699880}) 2024-09-16T04:07:10.619 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.613+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=807494101}) 2024-09-16T04:07:10.620 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.613+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.620 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.613+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2501535159}) 2024-09-16T04:07:10.620 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.613+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1669477959}) 2024-09-16T04:07:10.620 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.613+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.620 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.613+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2501535159}) 2024-09-16T04:07:10.620 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.613+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1571468257}) 2024-09-16T04:07:10.622 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.617+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.622 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.617+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1355964974}) 2024-09-16T04:07:10.622 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.617+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2536370369}) 2024-09-16T04:07:10.622 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.617+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.622 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.617+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1355964974}) 2024-09-16T04:07:10.622 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.617+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4224478406}) 2024-09-16T04:07:10.625 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.617+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.625 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.617+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=458196013}) 2024-09-16T04:07:10.625 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.617+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3616393821}) 2024-09-16T04:07:10.625 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.617+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.625 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.617+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=458196013}) 2024-09-16T04:07:10.625 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.617+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4136741608}) 2024-09-16T04:07:10.626 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.621+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.626 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.621+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=914642460}) 2024-09-16T04:07:10.626 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.621+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=69615633}) 2024-09-16T04:07:10.626 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.621+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.626 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.621+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=914642460}) 2024-09-16T04:07:10.626 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.621+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3861682655}) 2024-09-16T04:07:10.628 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.621+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.628 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.621+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1188023086}) 2024-09-16T04:07:10.628 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.621+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=750074166}) 2024-09-16T04:07:10.628 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.621+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.628 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.621+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1188023086}) 2024-09-16T04:07:10.628 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.621+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3410598873}) 2024-09-16T04:07:10.629 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.625+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.629 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.625+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2976578344}) 2024-09-16T04:07:10.629 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.625+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=747012577}) 2024-09-16T04:07:10.630 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.625+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.630 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.625+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2976578344}) 2024-09-16T04:07:10.630 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.625+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=49328211}) 2024-09-16T04:07:10.631 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.625+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.631 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.625+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1716810387}) 2024-09-16T04:07:10.631 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.625+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2812832075}) 2024-09-16T04:07:10.631 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.625+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.631 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.625+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1716810387}) 2024-09-16T04:07:10.631 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.625+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2508010331}) 2024-09-16T04:07:10.633 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.625+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.633 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.625+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2289713740}) 2024-09-16T04:07:10.633 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.625+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=467366187}) 2024-09-16T04:07:10.633 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.625+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.633 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.625+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2289713740}) 2024-09-16T04:07:10.633 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.625+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4014910478}) 2024-09-16T04:07:10.635 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.629+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.635 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.629+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2426924636}) 2024-09-16T04:07:10.635 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.629+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1745959161}) 2024-09-16T04:07:10.635 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.629+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.635 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.629+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2426924636}) 2024-09-16T04:07:10.635 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.629+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3588111808}) 2024-09-16T04:07:10.636 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.629+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.636 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.629+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1270023751}) 2024-09-16T04:07:10.637 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.629+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=160405470}) 2024-09-16T04:07:10.637 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.629+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.637 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.629+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1270023751}) 2024-09-16T04:07:10.637 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.629+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3621045779}) 2024-09-16T04:07:10.638 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.633+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.638 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.633+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2995438445}) 2024-09-16T04:07:10.638 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.633+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1570949924}) 2024-09-16T04:07:10.638 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.633+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.638 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.633+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2995438445}) 2024-09-16T04:07:10.638 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.633+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3136822407}) 2024-09-16T04:07:10.640 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.633+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.640 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.633+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=983233308}) 2024-09-16T04:07:10.640 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.633+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1588440749}) 2024-09-16T04:07:10.640 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.633+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.640 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.633+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=983233308}) 2024-09-16T04:07:10.640 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.633+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3090461505}) 2024-09-16T04:07:10.642 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.637+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.642 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.637+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1822770060}) 2024-09-16T04:07:10.642 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.637+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2488005969}) 2024-09-16T04:07:10.642 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.637+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.642 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.637+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1822770060}) 2024-09-16T04:07:10.642 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.637+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2689153686}) 2024-09-16T04:07:10.643 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.637+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.643 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.637+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2973325944}) 2024-09-16T04:07:10.644 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.637+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1128675737}) 2024-09-16T04:07:10.644 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.637+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.644 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.637+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2973325944}) 2024-09-16T04:07:10.644 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.637+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2258702046}) 2024-09-16T04:07:10.645 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.641+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.645 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.641+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=94453465}) 2024-09-16T04:07:10.645 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.641+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=832460147}) 2024-09-16T04:07:10.646 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.641+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.646 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.641+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=94453465}) 2024-09-16T04:07:10.646 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.641+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2869670499}) 2024-09-16T04:07:10.647 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.641+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.648 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.641+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2880893074}) 2024-09-16T04:07:10.648 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.641+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2725552082}) 2024-09-16T04:07:10.648 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.641+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.648 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.641+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2880893074}) 2024-09-16T04:07:10.648 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.641+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3617856471}) 2024-09-16T04:07:10.649 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.641+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.649 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.641+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2409648394}) 2024-09-16T04:07:10.649 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.641+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=724256276}) 2024-09-16T04:07:10.649 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.641+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.649 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.641+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2409648394}) 2024-09-16T04:07:10.649 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.641+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3666136932}) 2024-09-16T04:07:10.650 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.645+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.651 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.645+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3334775543}) 2024-09-16T04:07:10.651 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.645+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2540682125}) 2024-09-16T04:07:10.651 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.645+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.651 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.645+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3334775543}) 2024-09-16T04:07:10.651 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.645+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1735849187}) 2024-09-16T04:07:10.652 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.645+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.652 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.645+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3674028781}) 2024-09-16T04:07:10.652 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.645+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=899872328}) 2024-09-16T04:07:10.652 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.645+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.652 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.645+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3674028781}) 2024-09-16T04:07:10.653 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.645+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1439610673}) 2024-09-16T04:07:10.654 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.649+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.654 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.649+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2107187275}) 2024-09-16T04:07:10.654 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.649+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2178367477}) 2024-09-16T04:07:10.654 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.649+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.654 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.649+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2107187275}) 2024-09-16T04:07:10.654 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.649+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=250655188}) 2024-09-16T04:07:10.656 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.649+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.656 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.649+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1040866468}) 2024-09-16T04:07:10.656 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.649+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1448101255}) 2024-09-16T04:07:10.656 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.649+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.656 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.649+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1040866468}) 2024-09-16T04:07:10.656 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.649+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3855355593}) 2024-09-16T04:07:10.657 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.653+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.657 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.653+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2200689345}) 2024-09-16T04:07:10.657 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.653+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=498972237}) 2024-09-16T04:07:10.658 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.653+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.658 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.653+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2200689345}) 2024-09-16T04:07:10.658 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.653+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3619968543}) 2024-09-16T04:07:10.659 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.653+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.659 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.653+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=192293679}) 2024-09-16T04:07:10.659 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.653+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3284206292}) 2024-09-16T04:07:10.659 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.653+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.659 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.653+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=192293679}) 2024-09-16T04:07:10.659 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.653+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=163685478}) 2024-09-16T04:07:10.661 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.653+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.661 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.653+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3814406805}) 2024-09-16T04:07:10.661 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.653+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2403833584}) 2024-09-16T04:07:10.661 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.653+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.661 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.653+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3814406805}) 2024-09-16T04:07:10.661 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.653+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3124265400}) 2024-09-16T04:07:10.663 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.657+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.663 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.657+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2169129254}) 2024-09-16T04:07:10.663 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.657+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=260400260}) 2024-09-16T04:07:10.663 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.657+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.663 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.657+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2169129254}) 2024-09-16T04:07:10.663 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.657+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2816299707}) 2024-09-16T04:07:10.664 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.657+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.664 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.657+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4024692771}) 2024-09-16T04:07:10.664 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.657+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=799842601}) 2024-09-16T04:07:10.665 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.657+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.665 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.657+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4024692771}) 2024-09-16T04:07:10.665 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.657+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1827777900}) 2024-09-16T04:07:10.666 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.661+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.666 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.661+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2132929567}) 2024-09-16T04:07:10.667 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.661+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1879962258}) 2024-09-16T04:07:10.667 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.661+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.667 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.661+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2132929567}) 2024-09-16T04:07:10.667 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.661+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3990808129}) 2024-09-16T04:07:10.668 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.661+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.668 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.661+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2122313536}) 2024-09-16T04:07:10.668 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.661+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1229760619}) 2024-09-16T04:07:10.668 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.661+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.668 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.661+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2122313536}) 2024-09-16T04:07:10.668 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.661+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3261511029}) 2024-09-16T04:07:10.670 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.665+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.670 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.665+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=614472298}) 2024-09-16T04:07:10.670 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.665+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2812537028}) 2024-09-16T04:07:10.670 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.665+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.670 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.665+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=614472298}) 2024-09-16T04:07:10.670 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.665+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1042726922}) 2024-09-16T04:07:10.672 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.665+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.672 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.665+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=748871821}) 2024-09-16T04:07:10.672 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.665+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4040148556}) 2024-09-16T04:07:10.672 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.665+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.672 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.665+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=748871821}) 2024-09-16T04:07:10.672 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.665+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1503339483}) 2024-09-16T04:07:10.673 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.669+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.673 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.669+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4186190578}) 2024-09-16T04:07:10.673 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.669+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=962127305}) 2024-09-16T04:07:10.674 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.669+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.674 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.669+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4186190578}) 2024-09-16T04:07:10.674 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.669+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3635032968}) 2024-09-16T04:07:10.675 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.669+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.675 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.669+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1541059792}) 2024-09-16T04:07:10.675 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.669+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1384345613}) 2024-09-16T04:07:10.675 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.669+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.675 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.669+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1541059792}) 2024-09-16T04:07:10.675 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.669+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4062623259}) 2024-09-16T04:07:10.677 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.669+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.677 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.669+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2799059811}) 2024-09-16T04:07:10.677 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.669+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=101185686}) 2024-09-16T04:07:10.677 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.669+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.677 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.669+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2799059811}) 2024-09-16T04:07:10.677 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.669+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=241147521}) 2024-09-16T04:07:10.678 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.673+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.678 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.673+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1565542277}) 2024-09-16T04:07:10.679 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.673+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=287146550}) 2024-09-16T04:07:10.679 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.673+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.679 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.673+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1565542277}) 2024-09-16T04:07:10.679 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.673+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2262798927}) 2024-09-16T04:07:10.680 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.673+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.680 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.673+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=963277777}) 2024-09-16T04:07:10.681 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.673+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2262733830}) 2024-09-16T04:07:10.681 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.673+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.681 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.673+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=963277777}) 2024-09-16T04:07:10.681 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.673+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3477653062}) 2024-09-16T04:07:10.682 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.677+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.682 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.677+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3746215641}) 2024-09-16T04:07:10.682 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.677+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2388403942}) 2024-09-16T04:07:10.682 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.677+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.683 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.677+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3746215641}) 2024-09-16T04:07:10.683 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.677+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3717535796}) 2024-09-16T04:07:10.684 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.677+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.684 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.677+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3833059993}) 2024-09-16T04:07:10.684 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.677+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4160358105}) 2024-09-16T04:07:10.684 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.677+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.684 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.677+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3833059993}) 2024-09-16T04:07:10.684 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.677+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=102096413}) 2024-09-16T04:07:10.686 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.681+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.686 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.681+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=685640233}) 2024-09-16T04:07:10.686 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.681+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=924919176}) 2024-09-16T04:07:10.686 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.681+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.686 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.681+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=685640233}) 2024-09-16T04:07:10.686 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.681+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=778092219}) 2024-09-16T04:07:10.687 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.681+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.688 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.681+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2165646786}) 2024-09-16T04:07:10.688 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.681+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3143389856}) 2024-09-16T04:07:10.688 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.681+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.688 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.681+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2165646786}) 2024-09-16T04:07:10.688 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.681+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3171947300}) 2024-09-16T04:07:10.689 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.685+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.689 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.685+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4270101969}) 2024-09-16T04:07:10.689 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.685+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4090098446}) 2024-09-16T04:07:10.689 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.685+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.690 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.685+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4270101969}) 2024-09-16T04:07:10.690 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.685+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1808018512}) 2024-09-16T04:07:10.691 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.685+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.691 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.685+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=880136404}) 2024-09-16T04:07:10.691 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.685+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2893182003}) 2024-09-16T04:07:10.691 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.685+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.691 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.685+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=880136404}) 2024-09-16T04:07:10.691 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.685+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=53657301}) 2024-09-16T04:07:10.693 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.685+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.693 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.685+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1450889490}) 2024-09-16T04:07:10.693 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.685+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1769644572}) 2024-09-16T04:07:10.693 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.685+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.693 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.685+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1450889490}) 2024-09-16T04:07:10.693 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.685+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3219220466}) 2024-09-16T04:07:10.695 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.689+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.695 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.689+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4074577237}) 2024-09-16T04:07:10.695 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.689+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=894478563}) 2024-09-16T04:07:10.695 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.689+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.696 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.689+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4074577237}) 2024-09-16T04:07:10.696 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.689+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3171003567}) 2024-09-16T04:07:10.697 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.689+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.697 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.689+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1044377531}) 2024-09-16T04:07:10.697 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.689+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3832439620}) 2024-09-16T04:07:10.697 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.689+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.697 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.689+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1044377531}) 2024-09-16T04:07:10.697 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.689+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2840484634}) 2024-09-16T04:07:10.698 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.693+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.698 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.693+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2355253157}) 2024-09-16T04:07:10.699 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.693+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3955794746}) 2024-09-16T04:07:10.699 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.693+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.699 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.693+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2355253157}) 2024-09-16T04:07:10.699 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.693+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=280179672}) 2024-09-16T04:07:10.700 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.693+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.700 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.693+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1858682908}) 2024-09-16T04:07:10.700 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.693+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1394131139}) 2024-09-16T04:07:10.700 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.693+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.700 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.693+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1858682908}) 2024-09-16T04:07:10.701 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.693+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1248417888}) 2024-09-16T04:07:10.702 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.697+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.702 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.697+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1906693085}) 2024-09-16T04:07:10.702 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.697+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1005176027}) 2024-09-16T04:07:10.702 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.697+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.703 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.697+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1906693085}) 2024-09-16T04:07:10.703 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.697+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3134351959}) 2024-09-16T04:07:10.704 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.697+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.704 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.697+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3571406937}) 2024-09-16T04:07:10.704 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.697+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3465463212}) 2024-09-16T04:07:10.704 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.697+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.704 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.697+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3571406937}) 2024-09-16T04:07:10.704 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.697+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1886556072}) 2024-09-16T04:07:10.705 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.701+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.705 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.701+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2730094709}) 2024-09-16T04:07:10.705 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.701+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=236571586}) 2024-09-16T04:07:10.705 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.701+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.706 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.701+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2730094709}) 2024-09-16T04:07:10.706 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.701+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3943923291}) 2024-09-16T04:07:10.707 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.701+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.707 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.701+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3382758600}) 2024-09-16T04:07:10.707 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.701+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1052782222}) 2024-09-16T04:07:10.707 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.701+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.707 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.701+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3382758600}) 2024-09-16T04:07:10.707 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.701+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3413308022}) 2024-09-16T04:07:10.709 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.701+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.709 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.701+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=754937706}) 2024-09-16T04:07:10.709 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.701+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2471280138}) 2024-09-16T04:07:10.709 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.701+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.709 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.701+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=754937706}) 2024-09-16T04:07:10.709 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.701+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=270965690}) 2024-09-16T04:07:10.710 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.705+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.711 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.705+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1949990888}) 2024-09-16T04:07:10.711 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.705+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3140232923}) 2024-09-16T04:07:10.711 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.705+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.711 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.705+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1949990888}) 2024-09-16T04:07:10.711 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.705+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3362832628}) 2024-09-16T04:07:10.712 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.705+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.713 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.705+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2487454598}) 2024-09-16T04:07:10.713 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.705+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3011814530}) 2024-09-16T04:07:10.713 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.705+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.713 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.705+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2487454598}) 2024-09-16T04:07:10.713 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.705+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2323476303}) 2024-09-16T04:07:10.714 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.709+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.714 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.709+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1482218099}) 2024-09-16T04:07:10.714 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.709+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2243453543}) 2024-09-16T04:07:10.714 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.709+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.714 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.709+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1482218099}) 2024-09-16T04:07:10.715 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.709+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4158798857}) 2024-09-16T04:07:10.716 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.709+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.716 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.709+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2258505048}) 2024-09-16T04:07:10.716 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.709+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=808335758}) 2024-09-16T04:07:10.717 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.709+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.717 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.709+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2258505048}) 2024-09-16T04:07:10.717 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.709+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2691587179}) 2024-09-16T04:07:10.718 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.713+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.718 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.713+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4144804041}) 2024-09-16T04:07:10.718 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.713+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2935388996}) 2024-09-16T04:07:10.718 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.713+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.718 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.713+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4144804041}) 2024-09-16T04:07:10.718 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.713+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3837625671}) 2024-09-16T04:07:10.720 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.713+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.720 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.713+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3910509330}) 2024-09-16T04:07:10.720 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.713+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2493520473}) 2024-09-16T04:07:10.720 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.713+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.720 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.713+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3910509330}) 2024-09-16T04:07:10.720 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.713+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3410537995}) 2024-09-16T04:07:10.722 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.717+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.722 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.717+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=360198071}) 2024-09-16T04:07:10.722 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.717+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4107206429}) 2024-09-16T04:07:10.722 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.717+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.722 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.717+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=360198071}) 2024-09-16T04:07:10.722 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.717+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3294900002}) 2024-09-16T04:07:10.724 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.717+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.724 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.717+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3035284551}) 2024-09-16T04:07:10.724 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.717+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3895736436}) 2024-09-16T04:07:10.724 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.717+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.724 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.717+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3035284551}) 2024-09-16T04:07:10.724 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.717+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=572694798}) 2024-09-16T04:07:10.725 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.721+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.725 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.721+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2864800665}) 2024-09-16T04:07:10.725 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.721+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=346713726}) 2024-09-16T04:07:10.725 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.721+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.726 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.721+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2864800665}) 2024-09-16T04:07:10.726 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.721+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1705559933}) 2024-09-16T04:07:10.727 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.721+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.727 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.721+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4089136054}) 2024-09-16T04:07:10.727 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.721+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2520837236}) 2024-09-16T04:07:10.727 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.721+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.727 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.721+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4089136054}) 2024-09-16T04:07:10.727 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.721+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2449704278}) 2024-09-16T04:07:10.728 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.721+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.729 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.721+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3100218831}) 2024-09-16T04:07:10.729 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.721+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=278673239}) 2024-09-16T04:07:10.729 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.721+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.729 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.721+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3100218831}) 2024-09-16T04:07:10.729 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.721+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=605906841}) 2024-09-16T04:07:10.730 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.725+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.730 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.725+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2958118116}) 2024-09-16T04:07:10.730 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.725+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3182554550}) 2024-09-16T04:07:10.730 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.725+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.730 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.725+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2958118116}) 2024-09-16T04:07:10.730 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.725+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2349503084}) 2024-09-16T04:07:10.732 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.725+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.732 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.725+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=320159086}) 2024-09-16T04:07:10.732 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.725+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1173067806}) 2024-09-16T04:07:10.732 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.725+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.732 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.725+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=320159086}) 2024-09-16T04:07:10.732 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.725+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1051991642}) 2024-09-16T04:07:10.734 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.729+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.734 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.729+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1309071209}) 2024-09-16T04:07:10.734 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.729+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2193618224}) 2024-09-16T04:07:10.734 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.729+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.734 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.729+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1309071209}) 2024-09-16T04:07:10.734 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.729+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1092899580}) 2024-09-16T04:07:10.736 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.729+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.736 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.729+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1112639624}) 2024-09-16T04:07:10.736 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.729+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=33988652}) 2024-09-16T04:07:10.736 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.729+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.736 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.729+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1112639624}) 2024-09-16T04:07:10.736 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.729+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2054033090}) 2024-09-16T04:07:10.737 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.733+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.737 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.733+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1910843253}) 2024-09-16T04:07:10.738 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.733+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1310672451}) 2024-09-16T04:07:10.738 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.733+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.738 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.733+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1910843253}) 2024-09-16T04:07:10.738 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.733+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4055969593}) 2024-09-16T04:07:10.739 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.733+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.739 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.733+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2486210065}) 2024-09-16T04:07:10.739 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.733+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=25137998}) 2024-09-16T04:07:10.739 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.733+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.739 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.733+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2486210065}) 2024-09-16T04:07:10.739 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.733+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=605384001}) 2024-09-16T04:07:10.741 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.733+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.741 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.733+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1996895387}) 2024-09-16T04:07:10.741 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.733+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1877632371}) 2024-09-16T04:07:10.741 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.733+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.741 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.733+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1996895387}) 2024-09-16T04:07:10.741 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.733+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1259963581}) 2024-09-16T04:07:10.742 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.737+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.742 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.737+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1204434542}) 2024-09-16T04:07:10.742 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.737+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3648736957}) 2024-09-16T04:07:10.743 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.737+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.743 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.737+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1204434542}) 2024-09-16T04:07:10.743 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.737+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=271521906}) 2024-09-16T04:07:10.744 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.737+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.744 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.737+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2323400070}) 2024-09-16T04:07:10.744 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.737+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2461177862}) 2024-09-16T04:07:10.744 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.737+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.744 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.737+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2323400070}) 2024-09-16T04:07:10.745 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.737+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=828853680}) 2024-09-16T04:07:10.746 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.741+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.746 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.741+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3537691654}) 2024-09-16T04:07:10.746 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.741+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2753874648}) 2024-09-16T04:07:10.746 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.741+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.746 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.741+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3537691654}) 2024-09-16T04:07:10.747 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.741+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=771718898}) 2024-09-16T04:07:10.748 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.741+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.749 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.741+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3204612929}) 2024-09-16T04:07:10.749 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.741+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=392826126}) 2024-09-16T04:07:10.749 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.741+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.749 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.741+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3204612929}) 2024-09-16T04:07:10.749 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.741+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4029026465}) 2024-09-16T04:07:10.750 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.745+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.750 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.745+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3038015290}) 2024-09-16T04:07:10.750 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.745+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1998384284}) 2024-09-16T04:07:10.750 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.745+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.750 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.745+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3038015290}) 2024-09-16T04:07:10.750 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.745+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3103814823}) 2024-09-16T04:07:10.752 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.745+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.752 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.745+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2637359590}) 2024-09-16T04:07:10.752 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.745+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=62854681}) 2024-09-16T04:07:10.752 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.745+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.752 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.745+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2637359590}) 2024-09-16T04:07:10.752 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.745+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2031960620}) 2024-09-16T04:07:10.754 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.749+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.754 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.749+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=482257023}) 2024-09-16T04:07:10.754 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.749+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2837673929}) 2024-09-16T04:07:10.754 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.749+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.754 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.749+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=482257023}) 2024-09-16T04:07:10.754 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.749+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=661437275}) 2024-09-16T04:07:10.756 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.749+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.756 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.749+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1734874194}) 2024-09-16T04:07:10.756 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.749+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2949568397}) 2024-09-16T04:07:10.756 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.749+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.756 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.749+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1734874194}) 2024-09-16T04:07:10.757 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.749+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=253656291}) 2024-09-16T04:07:10.758 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.753+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.758 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.753+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2843317676}) 2024-09-16T04:07:10.758 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.753+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1919292826}) 2024-09-16T04:07:10.758 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.753+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.758 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.753+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2843317676}) 2024-09-16T04:07:10.758 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.753+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=531353501}) 2024-09-16T04:07:10.759 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.753+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.759 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.753+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=841065254}) 2024-09-16T04:07:10.760 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.753+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2215791575}) 2024-09-16T04:07:10.760 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.753+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.760 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.753+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=841065254}) 2024-09-16T04:07:10.760 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.753+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3509986024}) 2024-09-16T04:07:10.761 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.753+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.761 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.757+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=49400023}) 2024-09-16T04:07:10.761 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.757+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=735905188}) 2024-09-16T04:07:10.761 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.757+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.761 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.757+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=49400023}) 2024-09-16T04:07:10.761 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.757+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=445436766}) 2024-09-16T04:07:10.763 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.757+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.763 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.757+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1851200497}) 2024-09-16T04:07:10.763 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.757+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=142333407}) 2024-09-16T04:07:10.763 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.757+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.763 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.757+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1851200497}) 2024-09-16T04:07:10.763 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.757+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2093326468}) 2024-09-16T04:07:10.764 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.757+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.765 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.757+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2287408715}) 2024-09-16T04:07:10.765 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.757+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3396448456}) 2024-09-16T04:07:10.765 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.757+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.765 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.757+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2287408715}) 2024-09-16T04:07:10.765 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.757+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=27097466}) 2024-09-16T04:07:10.767 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.761+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.767 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.761+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4290746154}) 2024-09-16T04:07:10.767 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.761+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3844748329}) 2024-09-16T04:07:10.767 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.761+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.767 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.761+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4290746154}) 2024-09-16T04:07:10.767 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.761+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1052911902}) 2024-09-16T04:07:10.769 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.761+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.769 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.761+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1927380505}) 2024-09-16T04:07:10.769 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.761+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1639951015}) 2024-09-16T04:07:10.769 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.761+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.769 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.761+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1927380505}) 2024-09-16T04:07:10.769 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.761+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1528553293}) 2024-09-16T04:07:10.771 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.765+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.771 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.765+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=938462381}) 2024-09-16T04:07:10.771 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.765+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1490245063}) 2024-09-16T04:07:10.771 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.765+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.771 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.765+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=938462381}) 2024-09-16T04:07:10.771 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.765+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=293791917}) 2024-09-16T04:07:10.772 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.765+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.772 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.765+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3934261411}) 2024-09-16T04:07:10.772 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.765+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2136674021}) 2024-09-16T04:07:10.773 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.765+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.773 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.765+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3934261411}) 2024-09-16T04:07:10.773 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.765+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1668798534}) 2024-09-16T04:07:10.774 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.769+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.774 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.769+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1406216391}) 2024-09-16T04:07:10.774 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.769+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3970056139}) 2024-09-16T04:07:10.774 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.769+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.774 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.769+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1406216391}) 2024-09-16T04:07:10.774 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.769+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3094368662}) 2024-09-16T04:07:10.776 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.769+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.776 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.769+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1471537747}) 2024-09-16T04:07:10.776 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.769+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2636344572}) 2024-09-16T04:07:10.776 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.769+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.776 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.769+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1471537747}) 2024-09-16T04:07:10.776 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.769+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4069363503}) 2024-09-16T04:07:10.778 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.773+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.778 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.773+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=846884575}) 2024-09-16T04:07:10.778 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.773+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4016714743}) 2024-09-16T04:07:10.778 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.773+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.778 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.773+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=846884575}) 2024-09-16T04:07:10.778 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.773+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3087478516}) 2024-09-16T04:07:10.779 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.773+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.779 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.773+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1406874420}) 2024-09-16T04:07:10.780 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.773+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1949989262}) 2024-09-16T04:07:10.780 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.773+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.780 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.773+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1406874420}) 2024-09-16T04:07:10.780 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.773+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4288691536}) 2024-09-16T04:07:10.782 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.777+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.782 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.777+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2967860377}) 2024-09-16T04:07:10.782 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.777+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1715049945}) 2024-09-16T04:07:10.782 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.777+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.782 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.777+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2967860377}) 2024-09-16T04:07:10.782 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.777+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1961156156}) 2024-09-16T04:07:10.783 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.777+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.783 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.777+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3723825405}) 2024-09-16T04:07:10.783 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.777+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2254375279}) 2024-09-16T04:07:10.784 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.777+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.784 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.777+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3723825405}) 2024-09-16T04:07:10.784 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.777+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1203630795}) 2024-09-16T04:07:10.785 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.777+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.785 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.777+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=579322975}) 2024-09-16T04:07:10.785 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.777+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3156938458}) 2024-09-16T04:07:10.785 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.777+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.785 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.777+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=579322975}) 2024-09-16T04:07:10.785 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.777+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1293731519}) 2024-09-16T04:07:10.787 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.781+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.787 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.781+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1726888834}) 2024-09-16T04:07:10.787 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.781+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=647454472}) 2024-09-16T04:07:10.787 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.781+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.787 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.781+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1726888834}) 2024-09-16T04:07:10.787 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.781+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4260219841}) 2024-09-16T04:07:10.789 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.781+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.789 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.781+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3912725245}) 2024-09-16T04:07:10.789 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.781+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3492899983}) 2024-09-16T04:07:10.789 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.781+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.789 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.781+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3912725245}) 2024-09-16T04:07:10.789 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.781+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3154080232}) 2024-09-16T04:07:10.790 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.785+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.790 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.785+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3851956181}) 2024-09-16T04:07:10.791 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.785+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2557009180}) 2024-09-16T04:07:10.791 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.785+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.791 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.785+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3851956181}) 2024-09-16T04:07:10.791 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.785+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1449202053}) 2024-09-16T04:07:10.792 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.785+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.792 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.785+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=288994453}) 2024-09-16T04:07:10.793 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.785+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=596550332}) 2024-09-16T04:07:10.793 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.785+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.793 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.785+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=288994453}) 2024-09-16T04:07:10.793 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.785+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=158149988}) 2024-09-16T04:07:10.794 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.789+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.794 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.789+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3384547721}) 2024-09-16T04:07:10.794 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.789+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1471088803}) 2024-09-16T04:07:10.794 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.789+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.795 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.789+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3384547721}) 2024-09-16T04:07:10.795 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.789+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3423328966}) 2024-09-16T04:07:10.796 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.789+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.796 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.789+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=862454917}) 2024-09-16T04:07:10.796 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.789+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1624254931}) 2024-09-16T04:07:10.796 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.789+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.796 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.789+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=862454917}) 2024-09-16T04:07:10.796 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.789+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3436052154}) 2024-09-16T04:07:10.798 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.793+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.798 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.793+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2578983461}) 2024-09-16T04:07:10.798 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.793+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3851362281}) 2024-09-16T04:07:10.798 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.793+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.798 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.793+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2578983461}) 2024-09-16T04:07:10.798 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.793+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4080870480}) 2024-09-16T04:07:10.800 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.793+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.800 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.793+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3808857753}) 2024-09-16T04:07:10.800 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.793+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=670892438}) 2024-09-16T04:07:10.800 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.793+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.800 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.793+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3808857753}) 2024-09-16T04:07:10.800 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.793+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3378767967}) 2024-09-16T04:07:10.802 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.797+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.802 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.797+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2108173215}) 2024-09-16T04:07:10.802 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.797+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1798952792}) 2024-09-16T04:07:10.802 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.797+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.802 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.797+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2108173215}) 2024-09-16T04:07:10.802 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.797+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1205415901}) 2024-09-16T04:07:10.804 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.797+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.804 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.797+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2537294434}) 2024-09-16T04:07:10.804 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.797+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3468223974}) 2024-09-16T04:07:10.804 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.797+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.804 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.797+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2537294434}) 2024-09-16T04:07:10.804 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.797+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=573536457}) 2024-09-16T04:07:10.806 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.801+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.806 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.801+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4121786265}) 2024-09-16T04:07:10.806 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.801+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=744837384}) 2024-09-16T04:07:10.806 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.801+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.806 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.801+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4121786265}) 2024-09-16T04:07:10.806 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.801+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1118414738}) 2024-09-16T04:07:10.808 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.801+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.808 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.801+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2678256718}) 2024-09-16T04:07:10.808 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.801+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2618274426}) 2024-09-16T04:07:10.808 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.801+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.808 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.801+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2678256718}) 2024-09-16T04:07:10.808 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.801+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3543767981}) 2024-09-16T04:07:10.809 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.805+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.809 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.805+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3852635173}) 2024-09-16T04:07:10.809 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.805+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2662006177}) 2024-09-16T04:07:10.810 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.805+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.810 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.805+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3852635173}) 2024-09-16T04:07:10.810 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.805+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3492138301}) 2024-09-16T04:07:10.811 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.805+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.811 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.805+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3593985619}) 2024-09-16T04:07:10.811 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.805+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2729321010}) 2024-09-16T04:07:10.811 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.805+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.812 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.805+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3593985619}) 2024-09-16T04:07:10.812 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.805+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2932806581}) 2024-09-16T04:07:10.813 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.805+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.813 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.805+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=667955208}) 2024-09-16T04:07:10.813 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.805+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1666302626}) 2024-09-16T04:07:10.813 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.805+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.813 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.805+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=667955208}) 2024-09-16T04:07:10.814 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.809+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3946941446}) 2024-09-16T04:07:10.815 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.809+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.815 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.809+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=604470216}) 2024-09-16T04:07:10.815 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.809+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3881861984}) 2024-09-16T04:07:10.816 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.809+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.816 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.809+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=604470216}) 2024-09-16T04:07:10.816 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.809+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1699174844}) 2024-09-16T04:07:10.817 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.809+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.817 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.809+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1848355175}) 2024-09-16T04:07:10.817 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.809+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=25213722}) 2024-09-16T04:07:10.817 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.809+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.817 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.809+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1848355175}) 2024-09-16T04:07:10.817 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.809+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1476162085}) 2024-09-16T04:07:10.819 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.813+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.819 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.813+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=975698911}) 2024-09-16T04:07:10.819 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.813+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1181343960}) 2024-09-16T04:07:10.819 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.813+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.819 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.813+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=975698911}) 2024-09-16T04:07:10.819 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.813+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=903776710}) 2024-09-16T04:07:10.820 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.813+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.821 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.813+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=339859371}) 2024-09-16T04:07:10.821 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.813+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2011930869}) 2024-09-16T04:07:10.821 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.813+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.821 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.813+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=339859371}) 2024-09-16T04:07:10.821 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.813+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1900337526}) 2024-09-16T04:07:10.822 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.817+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.823 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.817+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3124240767}) 2024-09-16T04:07:10.823 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.817+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=225043799}) 2024-09-16T04:07:10.823 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.817+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.823 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.817+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3124240767}) 2024-09-16T04:07:10.823 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.817+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=705580191}) 2024-09-16T04:07:10.824 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.817+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.824 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.817+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1565628678}) 2024-09-16T04:07:10.824 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.817+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=777352113}) 2024-09-16T04:07:10.824 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.817+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.824 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.817+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1565628678}) 2024-09-16T04:07:10.825 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.817+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1429564690}) 2024-09-16T04:07:10.826 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.821+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.826 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.821+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3552703399}) 2024-09-16T04:07:10.826 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.821+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2674828814}) 2024-09-16T04:07:10.826 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.821+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.826 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.821+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3552703399}) 2024-09-16T04:07:10.826 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.821+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=171640921}) 2024-09-16T04:07:10.828 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.821+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.828 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.821+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2623462412}) 2024-09-16T04:07:10.828 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.821+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1389444844}) 2024-09-16T04:07:10.828 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.821+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.828 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.821+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2623462412}) 2024-09-16T04:07:10.828 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.821+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1142021096}) 2024-09-16T04:07:10.830 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.825+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.830 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.825+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3393938362}) 2024-09-16T04:07:10.830 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.825+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1719280623}) 2024-09-16T04:07:10.830 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.825+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.830 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.825+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3393938362}) 2024-09-16T04:07:10.830 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.825+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=374037490}) 2024-09-16T04:07:10.831 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.825+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.832 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.825+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=455264977}) 2024-09-16T04:07:10.832 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.825+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4271296308}) 2024-09-16T04:07:10.832 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.825+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.832 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.825+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=455264977}) 2024-09-16T04:07:10.832 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.825+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3056093732}) 2024-09-16T04:07:10.833 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.829+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.833 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.829+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1733686949}) 2024-09-16T04:07:10.833 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.829+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2109532707}) 2024-09-16T04:07:10.834 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.829+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.834 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.829+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1733686949}) 2024-09-16T04:07:10.834 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.829+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1542035273}) 2024-09-16T04:07:10.835 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.829+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.835 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.829+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1684815408}) 2024-09-16T04:07:10.835 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.829+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=694786179}) 2024-09-16T04:07:10.835 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.829+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.835 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.829+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1684815408}) 2024-09-16T04:07:10.835 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.829+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4021686550}) 2024-09-16T04:07:10.837 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.829+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.837 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.829+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=160973770}) 2024-09-16T04:07:10.837 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.833+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2795297561}) 2024-09-16T04:07:10.837 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.833+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.838 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.833+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=160973770}) 2024-09-16T04:07:10.838 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.833+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1402634664}) 2024-09-16T04:07:10.844 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.837+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.845 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.837+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=56,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=10,monmap=2,osd_metadata=6,osd_snap=1} crc {logm=2283341669,mdsmap=4088524775,mgr=460109776,mgr_command_descs=2107673600,mgr_metadata=1367272202,mgrstat=4206078713,mon_config_key=2782868103,monmap=3140514009,osd_metadata=4182959540,osd_snap=3823785925}) 2024-09-16T04:07:10.845 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.837+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=51,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=9,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=6} crc {logm=612673038,mdsmap=4088524775,mgr=460109776,mgr_command_descs=2107673600,mgr_metadata=1367272202,mgrstat=1430898275,mon_config_key=737216639,monmap=3937513981,osd_metadata=4182959540,osd_pg_creating=783066558,osd_snap=2908773830}) 2024-09-16T04:07:10.845 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.837+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.845 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.837+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=56,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=10,monmap=2,osd_metadata=6,osd_snap=1} crc {logm=2283341669,mdsmap=4088524775,mgr=460109776,mgr_command_descs=2107673600,mgr_metadata=1367272202,mgrstat=4206078713,mon_config_key=2782868103,monmap=3140514009,osd_metadata=4182959540,osd_snap=3823785925}) 2024-09-16T04:07:10.845 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.837+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=54,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=11,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=2} crc {logm=2447768038,mdsmap=4088524775,mgr=460109776,mgr_command_descs=2107673600,mgr_metadata=1367272202,mgrstat=1145516249,mon_config_key=85057112,monmap=3937513981,osd_metadata=498355664,osd_pg_creating=783066558,osd_snap=497215832}) 2024-09-16T04:07:10.855 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.849+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.855 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.849+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=39,osdmap=61} crc {osd_snap=3395727368,osdmap=1294297793}) 2024-09-16T04:07:10.855 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.849+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=40,osdmap=60} crc {osd_snap=3457091699,osdmap=1215281681}) 2024-09-16T04:07:10.855 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.849+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.855 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.849+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=39,osdmap=61} crc {osd_snap=3395727368,osdmap=1294297793}) 2024-09-16T04:07:10.855 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.849+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=40,osdmap=60} crc {osd_snap=3735466666,osdmap=3499055108}) 2024-09-16T04:07:10.862 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.857+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.862 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.857+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=4174724584}) 2024-09-16T04:07:10.862 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.857+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=38079740}) 2024-09-16T04:07:10.862 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.857+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.862 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.857+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=4174724584}) 2024-09-16T04:07:10.862 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.857+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=11852693}) 2024-09-16T04:07:10.908 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.901+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.908 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.901+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2634562403}) 2024-09-16T04:07:10.908 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.901+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2981717433}) 2024-09-16T04:07:10.908 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.901+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.908 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.901+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2634562403}) 2024-09-16T04:07:10.908 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.901+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1436785919}) 2024-09-16T04:07:10.913 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.905+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.913 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.905+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3403594488}) 2024-09-16T04:07:10.913 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.905+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=136332540}) 2024-09-16T04:07:10.913 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.905+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.913 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.905+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3403594488}) 2024-09-16T04:07:10.913 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.905+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=343525807}) 2024-09-16T04:07:10.927 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.921+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.927 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.921+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=21} crc {osdmap=3838074264}) 2024-09-16T04:07:10.927 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.921+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=19} crc {osdmap=543976664}) 2024-09-16T04:07:10.927 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.921+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:10.927 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.921+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=21} crc {osdmap=3838074264}) 2024-09-16T04:07:10.928 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:10.921+0000 7f860662e640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=19} crc {osdmap=3844405817}) 2024-09-16T04:07:11.617 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T04:07:11.617 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:07:11.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.745+0000 7f16dfa37640 10 monclient: get_monmap_and_config 2024-09-16T04:07:11.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.745+0000 7f16dfa37640 10 monclient: build_initial_monmap 2024-09-16T04:07:11.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.745+0000 7f16dfa37640 10 monclient: monmap: 2024-09-16T04:07:11.751 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:07:11.751 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:11.751 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:07:11.749628+0000 2024-09-16T04:07:11.751 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:07:11.749628+0000 2024-09-16T04:07:11.751 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:07:11.751 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:11.752 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:07:11.752 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:07:11.752 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:07:11.752 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:07:11.752 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:07:11.752 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:11.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.745+0000 7f16dfa37640 1 Processor -- start 2024-09-16T04:07:11.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.745+0000 7f16dfa37640 1 -- start start 2024-09-16T04:07:11.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.745+0000 7f16dfa37640 10 monclient: init 2024-09-16T04:07:11.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.745+0000 7f16dfa37640 10 monclient: _reopen_session rank -1 2024-09-16T04:07:11.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.745+0000 7f16dfa37640 10 monclient: _add_conns ranks=[1,3,4,0,2] 2024-09-16T04:07:11.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.745+0000 7f16dfa37640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f16d80a2f10 addr v1:172.21.15.154:6789/0 2024-09-16T04:07:11.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.745+0000 7f16dfa37640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f16d809e070 0x7f16d80a1a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:11.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.745+0000 7f16dfa37640 10 monclient(hunting): picked mon.noname-a con 0x7f16d809e070 addr v2:172.21.15.42:3300/0 2024-09-16T04:07:11.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.745+0000 7f16dfa37640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f16d8140b30 0x7f16d8142f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:11.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.745+0000 7f16dfa37640 10 monclient(hunting): picked mon.noname-c con 0x7f16d8140b30 addr v2:172.21.15.154:3300/0 2024-09-16T04:07:11.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.745+0000 7f16dfa37640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f16d809dc50 con 0x7f16d80a2f10 2024-09-16T04:07:11.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.745+0000 7f16dfa37640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:11.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.745+0000 7f16dfa37640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f16d80a32f0 con 0x7f16d809e070 2024-09-16T04:07:11.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.745+0000 7f16dfa37640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:11.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.745+0000 7f16dfa37640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f16d8073290 con 0x7f16d8140b30 2024-09-16T04:07:11.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.745+0000 7f16dfa37640 10 monclient(hunting): _renew_subs 2024-09-16T04:07:11.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.745+0000 7f16dfa37640 10 monclient(hunting): authenticate will time out at 2478.719101s 2024-09-16T04:07:11.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.745+0000 7f16dcfd2640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f16d809e070 0x7f16d80a1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:11.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.745+0000 7f16dcfd2640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f16d809e070 0x7f16d80a1a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:51526/0 (socket says 172.21.15.42:51526) 2024-09-16T04:07:11.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.745+0000 7f16dcfd2640 1 -- 172.21.15.42:0/2109260687 learned_addr learned my addr 172.21.15.42:0/2109260687 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:07:11.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.745+0000 7f16dcfd2640 10 monclient(hunting): get_auth_request con 0x7f16d809e070 auth_method 0 2024-09-16T04:07:11.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.745+0000 7f16dcfd2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:11.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.745+0000 7f16dcfd2640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:11.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.745+0000 7f16dcfd2640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:11.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.745+0000 7f16dcfd2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:07:11.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.745+0000 7f16dcfd2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:07:11.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.745+0000 7f16dcfd2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:07:11.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.745+0000 7f16ddfd4640 1 --2- 172.21.15.42:0/2109260687 >> v2:172.21.15.154:3300/0 conn(0x7f16d8140b30 0x7f16d8142f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:11.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.745+0000 7f16dcfd2640 10 monclient(hunting): handle_auth_done global_id 984495 payload 274 2024-09-16T04:07:11.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.745+0000 7f16dcfd2640 1 -- 172.21.15.42:0/2109260687 >> v2:172.21.15.154:3300/0 conn(0x7f16d8140b30 msgr2=0x7f16d8142f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:07:11.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.745+0000 7f16dcfd2640 1 --2- 172.21.15.42:0/2109260687 >> v2:172.21.15.154:3300/0 conn(0x7f16d8140b30 0x7f16d8142f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:11.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.745+0000 7f16dcfd2640 1 -- 172.21.15.42:0/2109260687 >> v1:172.21.15.154:6789/0 conn(0x7f16d80a2f10 legacy=0x7f16d80a1330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:07:11.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.745+0000 7f16dcfd2640 10 monclient: _finish_hunting 0 2024-09-16T04:07:11.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.745+0000 7f16dcfd2640 1 monclient: found mon.noname-a 2024-09-16T04:07:11.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.745+0000 7f16dcfd2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:07:11.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.745+0000 7f16dcfd2640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:07:11.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.745+0000 7f16dcfd2640 1 -- 172.21.15.42:0/2109260687 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16d809e600 con 0x7f16d809e070 2024-09-16T04:07:11.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.745+0000 7f16ddfd4640 10 monclient: get_auth_request con 0x7f16d8140b30 auth_method 0 2024-09-16T04:07:11.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.745+0000 7f16ddfd4640 1 --2- 172.21.15.42:0/2109260687 >> v2:172.21.15.154:3300/0 conn(0x7f16d8140b30 0x7f16d8142f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T04:07:11.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.749+0000 7f16dcfd2640 1 --2- 172.21.15.42:0/2109260687 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f16d809e070 0x7f16d80a1a40 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f16d809d2b0 tx=0x7f16cc01c370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf4319d9bfa32a75 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:07:11.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.749+0000 7f16cbfff640 1 -- 172.21.15.42:0/2109260687 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16cc0b0070 con 0x7f16d809e070 2024-09-16T04:07:11.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.749+0000 7f16cbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:11.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.749+0000 7f16cbfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:07:11.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.749+0000 7f16cbfff640 10 monclient: dump: 2024-09-16T04:07:11.763 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:07:11.763 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:11.763 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:11.763 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:11.763 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:11.763 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:11.763 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:07:11.763 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:07:11.763 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:07:11.763 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:11.763 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.749+0000 7f16cbfff640 10 monclient: _finish_auth 0 2024-09-16T04:07:11.763 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.749+0000 7f16cbfff640 10 monclient: _check_auth_tickets 2024-09-16T04:07:11.764 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.749+0000 7f16cbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:07:11.764 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.749+0000 7f16cbfff640 1 -- 172.21.15.42:0/2109260687 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f16cc0ab600 con 0x7f16d809e070 2024-09-16T04:07:11.764 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.749+0000 7f16cbfff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:07:11.764 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.749+0000 7f16cbfff640 1 -- 172.21.15.42:0/2109260687 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16cc0ab9d0 con 0x7f16d809e070 2024-09-16T04:07:11.764 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.749+0000 7f16dfa37640 5 monclient: authenticate success, global_id 984495 2024-09-16T04:07:11.764 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.749+0000 7f16dfa37640 10 monclient: get_monmap_and_config success 2024-09-16T04:07:11.764 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.749+0000 7f16dfa37640 4 set_mon_vals no callback set 2024-09-16T04:07:11.764 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.749+0000 7f16cbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:11.764 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.749+0000 7f16cbfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:07:11.764 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.749+0000 7f16cbfff640 10 monclient: dump: 2024-09-16T04:07:11.764 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:07:11.765 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:11.765 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:11.765 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:11.765 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:11.765 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:11.765 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:07:11.765 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:07:11.765 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:07:11.765 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:11.765 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.749+0000 7f16dfa37640 10 monclient: shutdown 2024-09-16T04:07:11.765 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.749+0000 7f16dfa37640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:07:11.765 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.749+0000 7f16dfa37640 1 -- 172.21.15.42:0/2109260687 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f16d809e070 msgr2=0x7f16d80a1a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:11.766 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.749+0000 7f16dfa37640 1 --2- 172.21.15.42:0/2109260687 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f16d809e070 0x7f16d80a1a40 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f16d809d2b0 tx=0x7f16cc01c370 comp rx=0 tx=0).stop 2024-09-16T04:07:11.766 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.749+0000 7f16dfa37640 1 -- 172.21.15.42:0/2109260687 shutdown_connections 2024-09-16T04:07:11.766 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.749+0000 7f16dfa37640 1 --2- 172.21.15.42:0/2109260687 >> v2:172.21.15.154:3300/0 conn(0x7f16d8140b30 0x7f16d8142f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:11.766 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.749+0000 7f16dfa37640 1 --2- 172.21.15.42:0/2109260687 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f16d809e070 0x7f16d80a1a40 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:11.766 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.749+0000 7f16dfa37640 1 -- 172.21.15.42:0/2109260687 >> 172.21.15.42:0/2109260687 conn(0x7f16d80a2420 msgr2=0x7f16d80a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:07:11.766 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.749+0000 7f16dfa37640 1 -- 172.21.15.42:0/2109260687 shutdown_connections 2024-09-16T04:07:11.766 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.749+0000 7f16dfa37640 1 -- 172.21.15.42:0/2109260687 wait complete. 2024-09-16T04:07:11.766 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.753+0000 7f16dfa37640 10 monclient: build_initial_monmap 2024-09-16T04:07:11.766 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.753+0000 7f16dfa37640 10 monclient: monmap: 2024-09-16T04:07:11.766 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:07:11.766 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:07:11.766 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:07:11.766 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:07:11.767 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:07:11.767 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:11.767 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:07:11.767 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:07:11.767 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:07:11.767 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:11.767 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.753+0000 7f16dfa37640 1 librados: starting msgr at 2024-09-16T04:07:11.767 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.753+0000 7f16dfa37640 1 librados: starting objecter 2024-09-16T04:07:11.767 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.753+0000 7f16dfa37640 1 Processor -- start 2024-09-16T04:07:11.767 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.753+0000 7f16dfa37640 1 -- start start 2024-09-16T04:07:11.767 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.753+0000 7f16dfa37640 1 librados: setting wanted keys 2024-09-16T04:07:11.767 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.753+0000 7f16dfa37640 1 librados: calling monclient init 2024-09-16T04:07:11.767 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.753+0000 7f16dfa37640 10 monclient: init 2024-09-16T04:07:11.767 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.753+0000 7f16dfa37640 10 monclient: _reopen_session rank -1 2024-09-16T04:07:11.768 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.753+0000 7f16dfa37640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:07:11.768 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.753+0000 7f16dfa37640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f16d809e070 0x7f16d8138c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:11.768 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.753+0000 7f16dfa37640 10 monclient(hunting): picked mon.noname-c con 0x7f16d809e070 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:07:11.768 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.753+0000 7f16dfa37640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f16d80a2f10 0x7f16d8139150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:11.768 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.753+0000 7f16dfa37640 10 monclient(hunting): picked mon.noname-b con 0x7f16d80a2f10 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:07:11.768 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.753+0000 7f16dfa37640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f16d8140b30 0x7f16d81d3180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:11.768 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.753+0000 7f16dfa37640 10 monclient(hunting): picked mon.noname-a con 0x7f16d8140b30 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:07:11.768 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.753+0000 7f16dfa37640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:11.768 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.753+0000 7f16dfa37640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f16d8145e00 con 0x7f16d8140b30 2024-09-16T04:07:11.768 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.753+0000 7f16dfa37640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:11.768 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.753+0000 7f16dfa37640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f16d8145c80 con 0x7f16d80a2f10 2024-09-16T04:07:11.768 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.753+0000 7f16dfa37640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:11.768 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.753+0000 7f16dfa37640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f16d8145f80 con 0x7f16d809e070 2024-09-16T04:07:11.769 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.753+0000 7f16dfa37640 10 monclient(hunting): _renew_subs 2024-09-16T04:07:11.769 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.753+0000 7f16dfa37640 10 monclient(hunting): authenticate will time out at 2478.724952s 2024-09-16T04:07:11.769 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.753+0000 7f16dd7d3640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f16d809e070 0x7f16d8138c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:11.769 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.753+0000 7f16dd7d3640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f16d809e070 0x7f16d8138c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:56946/0 (socket says 172.21.15.42:56946) 2024-09-16T04:07:11.769 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.753+0000 7f16dd7d3640 1 -- 172.21.15.42:0/988530111 learned_addr learned my addr 172.21.15.42:0/988530111 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:07:11.769 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.753+0000 7f16dd7d3640 10 monclient(hunting): get_auth_request con 0x7f16d809e070 auth_method 0 2024-09-16T04:07:11.769 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.753+0000 7f16dd7d3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:11.769 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.753+0000 7f16dd7d3640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:11.769 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.753+0000 7f16dd7d3640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:11.769 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.753+0000 7f16ddfd4640 1 --2- 172.21.15.42:0/988530111 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f16d8140b30 0x7f16d81d3180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:11.769 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.753+0000 7f16ddfd4640 10 monclient(hunting): get_auth_request con 0x7f16d8140b30 auth_method 0 2024-09-16T04:07:11.769 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.753+0000 7f16ddfd4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:11.769 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.753+0000 7f16ddfd4640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:11.770 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.753+0000 7f16ddfd4640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:11.770 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.753+0000 7f16dd7d3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:07:11.770 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.753+0000 7f16dd7d3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:07:11.770 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.753+0000 7f16dd7d3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:07:11.770 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.753+0000 7f16ddfd4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:07:11.770 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.753+0000 7f16ddfd4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:07:11.770 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.753+0000 7f16ddfd4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:07:11.770 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.753+0000 7f16dcfd2640 1 --2- 172.21.15.42:0/988530111 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f16d80a2f10 0x7f16d8139150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:11.770 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.753+0000 7f16dcfd2640 10 monclient(hunting): get_auth_request con 0x7f16d80a2f10 auth_method 0 2024-09-16T04:07:11.770 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.753+0000 7f16dcfd2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:11.770 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.753+0000 7f16dcfd2640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:11.770 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.753+0000 7f16dcfd2640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:11.770 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.753+0000 7f16dd7d3640 10 monclient(hunting): handle_auth_done global_id 1014116 payload 824 2024-09-16T04:07:11.770 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.753+0000 7f16dd7d3640 1 -- 172.21.15.42:0/988530111 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f16d80a2f10 msgr2=0x7f16d8139150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:11.771 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.753+0000 7f16dd7d3640 1 --2- 172.21.15.42:0/988530111 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f16d80a2f10 0x7f16d8139150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:11.771 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.753+0000 7f16dd7d3640 1 -- 172.21.15.42:0/988530111 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f16d8140b30 msgr2=0x7f16d81d3180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:11.771 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.753+0000 7f16dd7d3640 1 --2- 172.21.15.42:0/988530111 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f16d8140b30 0x7f16d81d3180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:11.771 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.753+0000 7f16dd7d3640 10 monclient: _finish_hunting 0 2024-09-16T04:07:11.771 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.753+0000 7f16dd7d3640 1 monclient: found mon.noname-c 2024-09-16T04:07:11.771 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.753+0000 7f16dd7d3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:07:11.771 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.753+0000 7f16dd7d3640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T04:07:11.771 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.753+0000 7f16dd7d3640 1 -- 172.21.15.42:0/988530111 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16d80718b0 con 0x7f16d809e070 2024-09-16T04:07:11.771 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.753+0000 7f16ddfd4640 1 --2- 172.21.15.42:0/988530111 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f16d8140b30 0x7f16d81d3180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:07:11.771 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.753+0000 7f16dd7d3640 1 --2- 172.21.15.42:0/988530111 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f16d809e070 0x7f16d8138c10 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f16d400c7c0 tx=0x7f16d400cc80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:07:11.771 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.753+0000 7f16c9ffb640 10 client.?.objecter ms_handle_connect 0x7f16d809e070 2024-09-16T04:07:11.771 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.753+0000 7f16c9ffb640 10 client.?.objecter resend_mon_ops 2024-09-16T04:07:11.771 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.753+0000 7f16c9ffb640 1 -- 172.21.15.42:0/988530111 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16d4016020 con 0x7f16d809e070 2024-09-16T04:07:11.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.753+0000 7f16c9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:11.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.753+0000 7f16c9ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:07:11.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.753+0000 7f16c9ffb640 10 monclient: dump: 2024-09-16T04:07:11.772 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:07:11.772 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:11.772 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:11.772 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:11.772 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:11.772 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:11.772 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:07:11.772 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:07:11.772 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:07:11.772 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:11.773 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.753+0000 7f16c9ffb640 10 monclient: _finish_auth 0 2024-09-16T04:07:11.773 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.753+0000 7f16c9ffb640 10 monclient: _check_auth_tickets 2024-09-16T04:07:11.773 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.753+0000 7f16c9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:07:11.773 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.753+0000 7f16c9ffb640 1 -- 172.21.15.42:0/988530111 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f16d400aa50 con 0x7f16d809e070 2024-09-16T04:07:11.773 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.753+0000 7f16c9ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:07:11.773 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.753+0000 7f16c9ffb640 1 -- 172.21.15.42:0/988530111 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16d4012420 con 0x7f16d809e070 2024-09-16T04:07:11.773 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.753+0000 7f16dfa37640 5 monclient: authenticate success, global_id 1014116 2024-09-16T04:07:11.773 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.753+0000 7f16dfa37640 10 monclient: _renew_subs 2024-09-16T04:07:11.773 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.753+0000 7f16dfa37640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:07:11.773 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.753+0000 7f16dfa37640 1 -- 172.21.15.42:0/988530111 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16d80a4a80 con 0x7f16d809e070 2024-09-16T04:07:11.773 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.753+0000 7f16cb7fe640 4 set_mon_vals no callback set 2024-09-16T04:07:11.773 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.753+0000 7f16c9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:11.773 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.753+0000 7f16c9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:07:11.774 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.753+0000 7f16c9ffb640 10 monclient: dump: 2024-09-16T04:07:11.774 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:07:11.774 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:11.774 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:11.774 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:11.774 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:11.774 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:11.774 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:07:11.774 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:07:11.774 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:07:11.774 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:11.774 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.753+0000 7f16dfa37640 10 client.1014116.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:07:11.774 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.753+0000 7f16dfa37640 10 monclient: _renew_subs 2024-09-16T04:07:11.774 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.753+0000 7f16dfa37640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:07:11.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.753+0000 7f16dfa37640 1 -- 172.21.15.42:0/988530111 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f16d8073a30 con 0x7f16d809e070 2024-09-16T04:07:11.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.753+0000 7f16dfa37640 1 librados: init done 2024-09-16T04:07:11.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.753+0000 7f16dfa37640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:07:11.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.753+0000 7f16c9ffb640 1 -- 172.21.15.42:0/988530111 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f16d4019020 con 0x7f16d809e070 2024-09-16T04:07:11.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.753+0000 7f16dfa37640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:07:11.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.753+0000 7f16dfa37640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:07:11.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.753+0000 7f16dfa37640 1 -- 172.21.15.42:0/988530111 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f16d80a1330 con 0x7f16d809e070 2024-09-16T04:07:11.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.753+0000 7f16c9ffb640 1 --2- 172.21.15.42:0/988530111 >> v2:172.21.15.154:6812/254845342 conn(0x7f16bc037810 0x7f16bc039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:11.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.753+0000 7f16c9ffb640 1 -- 172.21.15.42:0/988530111 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(706..706 src has 451..706) ==== 5040+0+0 (secure 0 0 0) 0x7f16d4048c80 con 0x7f16d809e070 2024-09-16T04:07:11.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.753+0000 7f16c9ffb640 10 client.1014116.objecter ms_dispatch 0x7f16d8000ba0 osd_map(706..706 src has 451..706) 2024-09-16T04:07:11.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.753+0000 7f16c9ffb640 3 client.1014116.objecter handle_osd_map got epochs [706,706] > 0 2024-09-16T04:07:11.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.753+0000 7f16c9ffb640 3 client.1014116.objecter handle_osd_map decoding full epoch 706 2024-09-16T04:07:11.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.753+0000 7f16c9ffb640 20 client.1014116.objecter dump_active .. 0 homeless 2024-09-16T04:07:11.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.753+0000 7f16dcfd2640 1 --2- 172.21.15.42:0/988530111 >> v2:172.21.15.154:6812/254845342 conn(0x7f16bc037810 0x7f16bc039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:11.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.753+0000 7f16dcfd2640 10 monclient: get_auth_request con 0x7f16bc037810 auth_method 0 2024-09-16T04:07:11.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.757+0000 7f16dcfd2640 1 --2- 172.21.15.42:0/988530111 >> v2:172.21.15.154:6812/254845342 conn(0x7f16bc037810 0x7f16bc039cd0 secure :-1 s=READY pgs=3875 cs=0 l=1 rev1=1 crypto rx=0x7f16cc0ad000 tx=0x7f16cc008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:07:11.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.757+0000 7f16c9ffb640 10 client.1014116.objecter ms_handle_connect 0x7f16bc037810 2024-09-16T04:07:11.777 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.761+0000 7f16c9ffb640 1 -- 172.21.15.42:0/988530111 <== mon.2 v2:172.21.15.42:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f16d4020030 con 0x7f16d809e070 2024-09-16T04:07:11.777 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.761+0000 7f16c9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:07:11.777 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.761+0000 7f16c9ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:07:11.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.989+0000 7f16dfa37640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:07:11.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.989+0000 7f16dfa37640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:07:11.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.989+0000 7f16dfa37640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:07:11.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.989+0000 7f16dfa37640 1 -- 172.21.15.42:0/988530111 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f16d8139690 con 0x7f16d809e070 2024-09-16T04:07:11.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.989+0000 7f16c9ffb640 1 -- 172.21.15.42:0/988530111 <== mon.2 v2:172.21.15.42:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7f16d4012850 con 0x7f16d809e070 2024-09-16T04:07:11.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.989+0000 7f16c9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:07:11.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.989+0000 7f16c9ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:07:11.995 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":232,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:07:11.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.989+0000 7f16dfa37640 10 librados: watch_flush enter 2024-09-16T04:07:11.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.989+0000 7f16dfa37640 10 librados: watch_flush exit 2024-09-16T04:07:11.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.989+0000 7f16dfa37640 20 client.1014116.objecter shutdown clearing up homeless session... 2024-09-16T04:07:11.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.989+0000 7f16dfa37640 10 client.1014116.objecter successfully canceled tick 2024-09-16T04:07:11.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.989+0000 7f16dfa37640 1 -- 172.21.15.42:0/988530111 >> v2:172.21.15.154:6812/254845342 conn(0x7f16bc037810 msgr2=0x7f16bc039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:11.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.989+0000 7f16dfa37640 1 --2- 172.21.15.42:0/988530111 >> v2:172.21.15.154:6812/254845342 conn(0x7f16bc037810 0x7f16bc039cd0 secure :-1 s=READY pgs=3875 cs=0 l=1 rev1=1 crypto rx=0x7f16cc0ad000 tx=0x7f16cc008040 comp rx=0 tx=0).stop 2024-09-16T04:07:11.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.989+0000 7f16dfa37640 10 monclient: shutdown 2024-09-16T04:07:11.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.989+0000 7f16dfa37640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:07:11.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.989+0000 7f16dfa37640 1 -- 172.21.15.42:0/988530111 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f16d809e070 msgr2=0x7f16d8138c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:11.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.989+0000 7f16dfa37640 1 --2- 172.21.15.42:0/988530111 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f16d809e070 0x7f16d8138c10 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f16d400c7c0 tx=0x7f16d400cc80 comp rx=0 tx=0).stop 2024-09-16T04:07:11.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.989+0000 7f16dfa37640 1 -- 172.21.15.42:0/988530111 shutdown_connections 2024-09-16T04:07:11.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.989+0000 7f16dfa37640 1 --2- 172.21.15.42:0/988530111 >> v2:172.21.15.154:6812/254845342 conn(0x7f16bc037810 0x7f16bc039cd0 unknown :-1 s=CLOSED pgs=3875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:11.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.989+0000 7f16dfa37640 1 --2- 172.21.15.42:0/988530111 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f16d8140b30 0x7f16d81d3180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:11.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.989+0000 7f16dfa37640 1 --2- 172.21.15.42:0/988530111 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f16d80a2f10 0x7f16d8139150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:11.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.989+0000 7f16dfa37640 1 --2- 172.21.15.42:0/988530111 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f16d809e070 0x7f16d8138c10 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:11.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.989+0000 7f16dfa37640 1 -- 172.21.15.42:0/988530111 >> 172.21.15.42:0/988530111 conn(0x7f16d80a2420 msgr2=0x7f16d813baf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:07:11.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.989+0000 7f16dfa37640 1 -- 172.21.15.42:0/988530111 shutdown_connections 2024-09-16T04:07:11.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.989+0000 7f16dfa37640 1 -- 172.21.15.42:0/988530111 wait complete. 2024-09-16T04:07:11.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:11.989+0000 7f16dfa37640 1 librados: shutdown 2024-09-16T04:07:12.014 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T04:07:12.014 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T04:07:12.014 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T04:07:12.014 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T04:07:12.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.133+0000 7f3d3a81b640 10 monclient: get_monmap_and_config 2024-09-16T04:07:12.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.133+0000 7f3d3a81b640 10 monclient: build_initial_monmap 2024-09-16T04:07:12.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.133+0000 7f3d3a81b640 10 monclient: monmap: 2024-09-16T04:07:12.139 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:07:12.139 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:12.139 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:07:12.137293+0000 2024-09-16T04:07:12.139 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:07:12.137293+0000 2024-09-16T04:07:12.139 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:07:12.139 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:12.139 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:07:12.139 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:07:12.140 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:07:12.140 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:07:12.140 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:07:12.140 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:12.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.133+0000 7f3d3a81b640 1 Processor -- start 2024-09-16T04:07:12.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.133+0000 7f3d3a81b640 1 -- start start 2024-09-16T04:07:12.143 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.133+0000 7f3d3a81b640 10 monclient: init 2024-09-16T04:07:12.143 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.133+0000 7f3d3a81b640 10 monclient: _reopen_session rank -1 2024-09-16T04:07:12.143 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.133+0000 7f3d3a81b640 10 monclient: _add_conns ranks=[4,1,2,3,0] 2024-09-16T04:07:12.143 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.133+0000 7f3d3a81b640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f3d340a1330 0x7f3d340a1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:12.143 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.133+0000 7f3d3a81b640 10 monclient(hunting): picked mon.noname-c con 0x7f3d340a1330 addr v2:172.21.15.154:3300/0 2024-09-16T04:07:12.143 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.133+0000 7f3d3a81b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3d3406dac0 addr v1:172.21.15.154:6789/0 2024-09-16T04:07:12.143 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.133+0000 7f3d3a81b640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3d340a1c50 0x7f3d34135810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:12.143 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.133+0000 7f3d3a81b640 10 monclient(hunting): picked mon.noname-b con 0x7f3d340a1c50 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:07:12.143 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.133+0000 7f3d3a81b640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d3409c090 con 0x7f3d3406dac0 2024-09-16T04:07:12.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.133+0000 7f3d3a81b640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:12.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.133+0000 7f3d3a81b640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f3d3409f610 con 0x7f3d340a1330 2024-09-16T04:07:12.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.133+0000 7f3d3a81b640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:12.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.133+0000 7f3d3a81b640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f3d3409f8f0 con 0x7f3d340a1c50 2024-09-16T04:07:12.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.133+0000 7f3d3a81b640 10 monclient(hunting): _renew_subs 2024-09-16T04:07:12.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.133+0000 7f3d3a81b640 10 monclient(hunting): authenticate will time out at 2479.106809s 2024-09-16T04:07:12.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.133+0000 7f3d38db8640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3d340a1c50 0x7f3d34135810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:12.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.133+0000 7f3d38db8640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3d340a1c50 0x7f3d34135810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:56948/0 (socket says 172.21.15.42:56948) 2024-09-16T04:07:12.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.133+0000 7f3d38db8640 1 -- 172.21.15.42:0/2431191604 learned_addr learned my addr 172.21.15.42:0/2431191604 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:07:12.145 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.133+0000 7f3d38db8640 10 monclient(hunting): get_auth_request con 0x7f3d340a1c50 auth_method 0 2024-09-16T04:07:12.145 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.133+0000 7f3d38db8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:12.145 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.133+0000 7f3d38db8640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:12.145 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.133+0000 7f3d38db8640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:12.145 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.133+0000 7f3d33fff640 1 --2- 172.21.15.42:0/2431191604 >> v2:172.21.15.154:3300/0 conn(0x7f3d340a1330 0x7f3d340a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:12.145 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.133+0000 7f3d38db8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:07:12.145 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.133+0000 7f3d38db8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:07:12.145 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.133+0000 7f3d38db8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:07:12.145 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.133+0000 7f3d33fff640 10 monclient(hunting): get_auth_request con 0x7f3d340a1330 auth_method 0 2024-09-16T04:07:12.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.133+0000 7f3d33fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:12.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.133+0000 7f3d33fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:12.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.133+0000 7f3d33fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:12.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.133+0000 7f3d38db8640 10 monclient(hunting): handle_auth_done global_id 1014119 payload 274 2024-09-16T04:07:12.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.133+0000 7f3d38db8640 1 -- 172.21.15.42:0/2431191604 >> v2:172.21.15.154:3300/0 conn(0x7f3d340a1330 msgr2=0x7f3d340a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:07:12.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.133+0000 7f3d38db8640 1 --2- 172.21.15.42:0/2431191604 >> v2:172.21.15.154:3300/0 conn(0x7f3d340a1330 0x7f3d340a1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:12.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.133+0000 7f3d38db8640 1 -- 172.21.15.42:0/2431191604 >> v1:172.21.15.154:6789/0 conn(0x7f3d3406dac0 legacy=0x7f3d34135100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:07:12.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.133+0000 7f3d38db8640 10 monclient: _finish_hunting 0 2024-09-16T04:07:12.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.133+0000 7f3d38db8640 1 monclient: found mon.noname-b 2024-09-16T04:07:12.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.133+0000 7f3d38db8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:07:12.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.133+0000 7f3d38db8640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T04:07:12.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.133+0000 7f3d38db8640 1 -- 172.21.15.42:0/2431191604 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d3409fbd0 con 0x7f3d340a1c50 2024-09-16T04:07:12.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.137+0000 7f3d38db8640 1 --2- 172.21.15.42:0/2431191604 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3d340a1c50 0x7f3d34135810 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f3d2c009e50 tx=0x7f3d2c01c940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4f6e17ea4c11415c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:07:12.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.137+0000 7f3d32ffd640 1 -- 172.21.15.42:0/2431191604 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d2c0a56d0 con 0x7f3d340a1c50 2024-09-16T04:07:12.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.137+0000 7f3d32ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:12.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.137+0000 7f3d32ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:07:12.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.137+0000 7f3d32ffd640 10 monclient: dump: 2024-09-16T04:07:12.154 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:07:12.155 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:12.155 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:12.155 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:12.155 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:12.155 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:12.155 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:07:12.155 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:07:12.156 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:07:12.156 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:12.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.137+0000 7f3d32ffd640 10 monclient: _finish_auth 0 2024-09-16T04:07:12.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.137+0000 7f3d32ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:07:12.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.137+0000 7f3d32ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:07:12.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.137+0000 7f3d32ffd640 1 -- 172.21.15.42:0/2431191604 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d2c0a5870 con 0x7f3d340a1c50 2024-09-16T04:07:12.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.137+0000 7f3d32ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:07:12.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.137+0000 7f3d32ffd640 1 -- 172.21.15.42:0/2431191604 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d2c0a5c40 con 0x7f3d340a1c50 2024-09-16T04:07:12.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.137+0000 7f3d32ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:12.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.137+0000 7f3d32ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:07:12.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.137+0000 7f3d32ffd640 10 monclient: dump: 2024-09-16T04:07:12.157 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:07:12.157 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:12.157 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:12.157 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:12.157 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:12.157 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:12.157 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:07:12.157 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:07:12.157 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:07:12.157 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:12.157 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.137+0000 7f3d3a81b640 5 monclient: authenticate success, global_id 1014119 2024-09-16T04:07:12.157 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.137+0000 7f3d3a81b640 10 monclient: get_monmap_and_config success 2024-09-16T04:07:12.157 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.137+0000 7f3d3a81b640 4 set_mon_vals no callback set 2024-09-16T04:07:12.157 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.137+0000 7f3d3a81b640 10 monclient: shutdown 2024-09-16T04:07:12.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.137+0000 7f3d3a81b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:07:12.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.137+0000 7f3d3a81b640 1 -- 172.21.15.42:0/2431191604 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3d340a1c50 msgr2=0x7f3d34135810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:12.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.137+0000 7f3d3a81b640 1 --2- 172.21.15.42:0/2431191604 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3d340a1c50 0x7f3d34135810 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f3d2c009e50 tx=0x7f3d2c01c940 comp rx=0 tx=0).stop 2024-09-16T04:07:12.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.137+0000 7f3d3a81b640 1 -- 172.21.15.42:0/2431191604 shutdown_connections 2024-09-16T04:07:12.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.137+0000 7f3d3a81b640 1 --2- 172.21.15.42:0/2431191604 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3d340a1c50 0x7f3d34135810 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:12.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.137+0000 7f3d3a81b640 1 --2- 172.21.15.42:0/2431191604 >> v2:172.21.15.154:3300/0 conn(0x7f3d340a1330 0x7f3d340a1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:12.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.137+0000 7f3d3a81b640 1 -- 172.21.15.42:0/2431191604 >> 172.21.15.42:0/2431191604 conn(0x7f3d340a2420 msgr2=0x7f3d340a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:07:12.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.137+0000 7f3d3a81b640 1 -- 172.21.15.42:0/2431191604 shutdown_connections 2024-09-16T04:07:12.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.137+0000 7f3d3a81b640 1 -- 172.21.15.42:0/2431191604 wait complete. 2024-09-16T04:07:12.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.137+0000 7f3d3a81b640 10 monclient: build_initial_monmap 2024-09-16T04:07:12.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.137+0000 7f3d3a81b640 10 monclient: monmap: 2024-09-16T04:07:12.158 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:07:12.158 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:07:12.159 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:07:12.159 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:07:12.159 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:07:12.159 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:12.159 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:07:12.159 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:07:12.159 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:07:12.159 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:12.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.137+0000 7f3d3a81b640 1 librados: starting msgr at 2024-09-16T04:07:12.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.141+0000 7f3d3a81b640 1 librados: starting objecter 2024-09-16T04:07:12.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.141+0000 7f3d3a81b640 1 Processor -- start 2024-09-16T04:07:12.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.141+0000 7f3d3a81b640 1 -- start start 2024-09-16T04:07:12.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.141+0000 7f3d3a81b640 1 librados: setting wanted keys 2024-09-16T04:07:12.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.141+0000 7f3d3a81b640 1 librados: calling monclient init 2024-09-16T04:07:12.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.141+0000 7f3d3a81b640 10 monclient: init 2024-09-16T04:07:12.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.141+0000 7f3d3a81b640 10 monclient: _reopen_session rank -1 2024-09-16T04:07:12.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.141+0000 7f3d3a81b640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:07:12.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.141+0000 7f3d3a81b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d340a1330 0x7f3d34134b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:12.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.141+0000 7f3d3a81b640 10 monclient(hunting): picked mon.noname-b con 0x7f3d340a1330 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:07:12.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.141+0000 7f3d3a81b640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3d340a1c50 0x7f3d34135050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:12.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.141+0000 7f3d3a81b640 10 monclient(hunting): picked mon.noname-c con 0x7f3d340a1c50 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:07:12.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.141+0000 7f3d3a81b640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3d3411bf70 0x7f3d3411c400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:12.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.141+0000 7f3d3a81b640 10 monclient(hunting): picked mon.noname-a con 0x7f3d3411bf70 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:07:12.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.141+0000 7f3d3a81b640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:12.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.141+0000 7f3d3a81b640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f3d34135d50 con 0x7f3d3411bf70 2024-09-16T04:07:12.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.141+0000 7f3d3a81b640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:12.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.141+0000 7f3d3a81b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f3d3409ef80 con 0x7f3d340a1330 2024-09-16T04:07:12.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.141+0000 7f3d3a81b640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:12.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.141+0000 7f3d3a81b640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f3d34135ed0 con 0x7f3d340a1c50 2024-09-16T04:07:12.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.141+0000 7f3d3a81b640 10 monclient(hunting): _renew_subs 2024-09-16T04:07:12.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.141+0000 7f3d3a81b640 10 monclient(hunting): authenticate will time out at 2479.113007s 2024-09-16T04:07:12.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.141+0000 7f3d337fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3d340a1c50 0x7f3d34135050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:12.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.141+0000 7f3d38db8640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3d3411bf70 0x7f3d3411c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:12.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.141+0000 7f3d337fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3d340a1c50 0x7f3d34135050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:56962/0 (socket says 172.21.15.42:56962) 2024-09-16T04:07:12.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.141+0000 7f3d337fe640 1 -- 172.21.15.42:0/1984703963 learned_addr learned my addr 172.21.15.42:0/1984703963 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:07:12.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.141+0000 7f3d337fe640 10 monclient(hunting): get_auth_request con 0x7f3d340a1c50 auth_method 0 2024-09-16T04:07:12.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.141+0000 7f3d337fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:12.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.141+0000 7f3d337fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:12.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.141+0000 7f3d337fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:12.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.141+0000 7f3d38db8640 10 monclient(hunting): get_auth_request con 0x7f3d3411bf70 auth_method 0 2024-09-16T04:07:12.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.141+0000 7f3d38db8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:12.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.141+0000 7f3d38db8640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:12.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.141+0000 7f3d38db8640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:12.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.141+0000 7f3d337fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:07:12.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.141+0000 7f3d337fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:07:12.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.141+0000 7f3d337fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:07:12.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.141+0000 7f3d38db8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:07:12.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.141+0000 7f3d38db8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:07:12.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.141+0000 7f3d38db8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:07:12.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.141+0000 7f3d337fe640 10 monclient(hunting): handle_auth_done global_id 1014122 payload 824 2024-09-16T04:07:12.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.141+0000 7f3d337fe640 1 -- 172.21.15.42:0/1984703963 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d340a1330 msgr2=0x7f3d34134b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:12.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.141+0000 7f3d33fff640 1 --2- 172.21.15.42:0/1984703963 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d340a1330 0x7f3d34134b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:12.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.141+0000 7f3d337fe640 1 --2- 172.21.15.42:0/1984703963 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d340a1330 0x7f3d34134b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:12.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.141+0000 7f3d337fe640 1 -- 172.21.15.42:0/1984703963 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3d3411bf70 msgr2=0x7f3d3411c400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:12.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.141+0000 7f3d337fe640 1 --2- 172.21.15.42:0/1984703963 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3d3411bf70 0x7f3d3411c400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:12.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.141+0000 7f3d337fe640 10 monclient: _finish_hunting 0 2024-09-16T04:07:12.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.141+0000 7f3d337fe640 1 monclient: found mon.noname-c 2024-09-16T04:07:12.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.141+0000 7f3d337fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:07:12.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.141+0000 7f3d337fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T04:07:12.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.141+0000 7f3d337fe640 1 -- 172.21.15.42:0/1984703963 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d3411f590 con 0x7f3d340a1c50 2024-09-16T04:07:12.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.141+0000 7f3d33fff640 10 monclient: get_auth_request con 0x7f3d340a1330 auth_method 0 2024-09-16T04:07:12.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.141+0000 7f3d38db8640 1 --2- 172.21.15.42:0/1984703963 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3d3411bf70 0x7f3d3411c400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:07:12.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.141+0000 7f3d33fff640 1 --2- 172.21.15.42:0/1984703963 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d340a1330 0x7f3d34134b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T04:07:12.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.141+0000 7f3d337fe640 1 --2- 172.21.15.42:0/1984703963 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3d340a1c50 0x7f3d34135050 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f3d2800c480 tx=0x7f3d2800c940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:07:12.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.141+0000 7f3d30ff9640 10 client.?.objecter ms_handle_connect 0x7f3d340a1c50 2024-09-16T04:07:12.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.141+0000 7f3d30ff9640 10 client.?.objecter resend_mon_ops 2024-09-16T04:07:12.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.141+0000 7f3d30ff9640 1 -- 172.21.15.42:0/1984703963 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d28012830 con 0x7f3d340a1c50 2024-09-16T04:07:12.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.141+0000 7f3d30ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:12.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.141+0000 7f3d30ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:07:12.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.141+0000 7f3d30ff9640 10 monclient: dump: 2024-09-16T04:07:12.165 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:07:12.165 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:12.165 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:12.165 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:12.165 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:12.165 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:12.165 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:07:12.165 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:07:12.165 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:07:12.165 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:12.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.141+0000 7f3d30ff9640 10 monclient: _finish_auth 0 2024-09-16T04:07:12.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.141+0000 7f3d30ff9640 10 monclient: _check_auth_tickets 2024-09-16T04:07:12.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.141+0000 7f3d30ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:07:12.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.141+0000 7f3d30ff9640 1 -- 172.21.15.42:0/1984703963 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d28017070 con 0x7f3d340a1c50 2024-09-16T04:07:12.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.141+0000 7f3d30ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T04:07:12.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.141+0000 7f3d30ff9640 1 -- 172.21.15.42:0/1984703963 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d28012c30 con 0x7f3d340a1c50 2024-09-16T04:07:12.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.141+0000 7f3d30ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:12.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.141+0000 7f3d30ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:07:12.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.141+0000 7f3d30ff9640 10 monclient: dump: 2024-09-16T04:07:12.166 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:07:12.166 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:12.166 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:12.167 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:12.167 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:12.167 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:12.167 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:07:12.167 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:07:12.168 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:07:12.168 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:12.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.141+0000 7f3d3a81b640 5 monclient: authenticate success, global_id 1014122 2024-09-16T04:07:12.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.141+0000 7f3d3a81b640 10 monclient: _renew_subs 2024-09-16T04:07:12.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.141+0000 7f3d31ffb640 4 set_mon_vals no callback set 2024-09-16T04:07:12.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.141+0000 7f3d3a81b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:07:12.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.141+0000 7f3d3a81b640 1 -- 172.21.15.42:0/1984703963 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d34071230 con 0x7f3d340a1c50 2024-09-16T04:07:12.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.141+0000 7f3d3a81b640 10 client.1014122.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:07:12.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.141+0000 7f3d3a81b640 10 monclient: _renew_subs 2024-09-16T04:07:12.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.141+0000 7f3d3a81b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:07:12.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.141+0000 7f3d3a81b640 1 -- 172.21.15.42:0/1984703963 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d3411e300 con 0x7f3d340a1c50 2024-09-16T04:07:12.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.141+0000 7f3d3a81b640 1 librados: init done 2024-09-16T04:07:12.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.141+0000 7f3d3a81b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:07:12.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.141+0000 7f3d3a81b640 1 --2- 172.21.15.42:0/1984703963 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3d3406dac0 0x7f3d3411c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:12.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.141+0000 7f3d3a81b640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:12.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.141+0000 7f3d3a81b640 1 -- 172.21.15.42:0/1984703963 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f3d340a1710 con 0x7f3d3406dac0 2024-09-16T04:07:12.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.141+0000 7f3d30ff9640 1 -- 172.21.15.42:0/1984703963 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f3d2801c460 con 0x7f3d340a1c50 2024-09-16T04:07:12.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.141+0000 7f3d33fff640 1 --2- 172.21.15.42:0/1984703963 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3d3406dac0 0x7f3d3411c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:12.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.141+0000 7f3d33fff640 10 monclient: get_auth_request con 0x7f3d3406dac0 auth_method 0 2024-09-16T04:07:12.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.141+0000 7f3d33fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:12.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.141+0000 7f3d33fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:12.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.141+0000 7f3d33fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:12.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.141+0000 7f3d33fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:07:12.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.141+0000 7f3d33fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:07:12.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.141+0000 7f3d33fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:07:12.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.141+0000 7f3d30ff9640 1 --2- 172.21.15.42:0/1984703963 >> v2:172.21.15.154:6812/254845342 conn(0x7f3d20037810 0x7f3d20039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:12.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.141+0000 7f3d30ff9640 1 -- 172.21.15.42:0/1984703963 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(706..706 src has 451..706) ==== 5040+0+0 (secure 0 0 0) 0x7f3d28048a90 con 0x7f3d340a1c50 2024-09-16T04:07:12.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.141+0000 7f3d30ff9640 10 client.1014122.objecter ms_dispatch 0x7f3d34000ba0 osd_map(706..706 src has 451..706) 2024-09-16T04:07:12.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.141+0000 7f3d30ff9640 3 client.1014122.objecter handle_osd_map got epochs [706,706] > 0 2024-09-16T04:07:12.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.141+0000 7f3d30ff9640 3 client.1014122.objecter handle_osd_map decoding full epoch 706 2024-09-16T04:07:12.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.141+0000 7f3d33fff640 10 monclient(hunting): handle_auth_done global_id 984504 payload 824 2024-09-16T04:07:12.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.141+0000 7f3d33fff640 1 -- 172.21.15.42:0/1984703963 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3d3411de60 con 0x7f3d3406dac0 2024-09-16T04:07:12.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.141+0000 7f3d30ff9640 20 client.1014122.objecter dump_active .. 0 homeless 2024-09-16T04:07:12.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.141+0000 7f3d30ff9640 10 client.1014122.objecter ms_handle_connect 0x7f3d3406dac0 2024-09-16T04:07:12.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.141+0000 7f3d30ff9640 10 client.1014122.objecter resend_mon_ops 2024-09-16T04:07:12.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.141+0000 7f3d33fff640 1 --2- 172.21.15.42:0/1984703963 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3d3406dac0 0x7f3d3411c940 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f3d2400e740 tx=0x7f3d2400ec00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:07:12.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.141+0000 7f3d38db8640 1 --2- 172.21.15.42:0/1984703963 >> v2:172.21.15.154:6812/254845342 conn(0x7f3d20037810 0x7f3d20039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:12.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.141+0000 7f3d38db8640 10 monclient: get_auth_request con 0x7f3d20037810 auth_method 0 2024-09-16T04:07:12.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.141+0000 7f3d30ff9640 1 -- 172.21.15.42:0/1984703963 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d24017070 con 0x7f3d3406dac0 2024-09-16T04:07:12.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.141+0000 7f3d30ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:12.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.141+0000 7f3d30ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:07:12.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.141+0000 7f3d30ff9640 10 monclient: dump: 2024-09-16T04:07:12.173 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:07:12.173 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:12.173 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:12.173 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:12.173 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:12.173 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:12.173 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:07:12.173 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:07:12.174 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:07:12.174 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:12.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.145+0000 7f3d30ff9640 1 -- 172.21.15.42:0/1984703963 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f3d24004020 con 0x7f3d3406dac0 2024-09-16T04:07:12.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.145+0000 7f3d30ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:07:12.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.145+0000 7f3d30ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:07:12.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.145+0000 7f3d30ff9640 1 -- 172.21.15.42:0/1984703963 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3d3406dac0 msgr2=0x7f3d3411c940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:12.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.145+0000 7f3d30ff9640 1 --2- 172.21.15.42:0/1984703963 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3d3406dac0 0x7f3d3411c940 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f3d2400e740 tx=0x7f3d2400ec00 comp rx=0 tx=0).stop 2024-09-16T04:07:12.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.145+0000 7f3d30ff9640 1 -- 172.21.15.42:0/1984703963 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3d3406dac0 msgr2=0x7f3d3411c940 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:07:12.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.145+0000 7f3d30ff9640 1 --2- 172.21.15.42:0/1984703963 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3d3406dac0 0x7f3d3411c940 secure :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f3d2400e740 tx=0x7f3d2400ec00 comp rx=0 tx=0).stop 2024-09-16T04:07:12.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.145+0000 7f3d30ff9640 10 client.1014122.objecter ms_handle_connect 0x7f3d20037810 2024-09-16T04:07:12.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.145+0000 7f3d38db8640 1 --2- 172.21.15.42:0/1984703963 >> v2:172.21.15.154:6812/254845342 conn(0x7f3d20037810 0x7f3d20039cd0 secure :-1 s=READY pgs=3876 cs=0 l=1 rev1=1 crypto rx=0x7f3d2c0ae000 tx=0x7f3d2c0a7070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:07:12.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.149+0000 7f3d3a81b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:07:12.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.149+0000 7f3d3a81b640 1 --2- 172.21.15.42:0/1984703963 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3d34138330 0x7f3d34138740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:12.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.149+0000 7f3d3a81b640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:12.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.149+0000 7f3d3a81b640 1 -- 172.21.15.42:0/1984703963 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f3d3411f310 con 0x7f3d34138330 2024-09-16T04:07:12.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.149+0000 7f3d33fff640 1 --2- 172.21.15.42:0/1984703963 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3d34138330 0x7f3d34138740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:12.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.149+0000 7f3d33fff640 10 monclient: get_auth_request con 0x7f3d34138330 auth_method 0 2024-09-16T04:07:12.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.149+0000 7f3d33fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:12.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.149+0000 7f3d33fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:12.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.149+0000 7f3d33fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:12.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.149+0000 7f3d33fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:07:12.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.149+0000 7f3d33fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:07:12.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.149+0000 7f3d33fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:07:12.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.149+0000 7f3d33fff640 10 monclient(hunting): handle_auth_done global_id 984507 payload 824 2024-09-16T04:07:12.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.149+0000 7f3d33fff640 1 -- 172.21.15.42:0/1984703963 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f3d34069040 con 0x7f3d34138330 2024-09-16T04:07:12.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.149+0000 7f3d30ff9640 10 client.1014122.objecter ms_handle_connect 0x7f3d34138330 2024-09-16T04:07:12.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.149+0000 7f3d30ff9640 10 client.1014122.objecter resend_mon_ops 2024-09-16T04:07:12.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.149+0000 7f3d33fff640 1 --2- 172.21.15.42:0/1984703963 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3d34138330 0x7f3d34138740 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f3d24000c10 tx=0x7f3d240046d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:07:12.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.149+0000 7f3d30ff9640 1 -- 172.21.15.42:0/1984703963 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d2400f040 con 0x7f3d34138330 2024-09-16T04:07:12.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.149+0000 7f3d30ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:12.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.149+0000 7f3d30ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:07:12.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.149+0000 7f3d30ff9640 10 monclient: dump: 2024-09-16T04:07:12.178 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:07:12.178 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:12.178 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:12.178 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:12.178 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:12.178 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:12.178 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:07:12.179 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:07:12.179 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:07:12.179 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:12.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.149+0000 7f3d30ff9640 1 -- 172.21.15.42:0/1984703963 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4715 (secure 0 0 0) 0x7f3d24003950 con 0x7f3d34138330 2024-09-16T04:07:12.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.149+0000 7f3d30ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:07:12.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.149+0000 7f3d30ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:07:12.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.149+0000 7f3d30ff9640 1 -- 172.21.15.42:0/1984703963 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3d34138330 msgr2=0x7f3d34138740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:12.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.149+0000 7f3d30ff9640 1 --2- 172.21.15.42:0/1984703963 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3d34138330 0x7f3d34138740 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f3d24000c10 tx=0x7f3d240046d0 comp rx=0 tx=0).stop 2024-09-16T04:07:12.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.149+0000 7f3d30ff9640 1 -- 172.21.15.42:0/1984703963 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3d34138330 msgr2=0x7f3d34138740 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:07:12.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.149+0000 7f3d30ff9640 1 --2- 172.21.15.42:0/1984703963 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3d34138330 0x7f3d34138740 secure :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f3d24000c10 tx=0x7f3d240046d0 comp rx=0 tx=0).stop 2024-09-16T04:07:12.179 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T04:07:12.180 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:07:12.180 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:07:12.180 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "leader", 2024-09-16T04:07:12.180 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 232, 2024-09-16T04:07:12.180 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 192206, 2024-09-16T04:07:12.180 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T04:07:12.180 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T04:07:12.180 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T04:07:12.180 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T04:07:12.180 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:07:12.180 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 7, 2024-09-16T04:07:12.180 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:07:12.181 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T04:07:12.181 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T04:07:12.181 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:07:12.181 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:07:12.181 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:07:12.181 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:07:12.181 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:07:12.181 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:07:12.181 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:07:12.181 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:07:12.181 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:07:12.181 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:07:12.181 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:07:12.181 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:07:12.181 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:07:12.182 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T04:07:12.182 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:07:12.182 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:07:12.182 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:07:12.182 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:07:12.182 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:07:12.182 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:07:12.182 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:07:12.182 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:07:12.182 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:07:12.182 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:07:12.182 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:07:12.182 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:07:12.183 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:12.183 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T04:07:12.183 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T04:07:12.183 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T04:07:12.183 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T04:07:12.183 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T04:07:12.183 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T04:07:12.183 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:07:12.183 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:07:12.183 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T04:07:12.183 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T04:07:12.183 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T04:07:12.183 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T04:07:12.183 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T04:07:12.184 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T04:07:12.184 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T04:07:12.184 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:07:12.184 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T04:07:12.184 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:07:12.184 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:07:12.184 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:07:12.184 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:07:12.184 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:07:12.184 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:07:12.184 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:07:12.184 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:07:12.184 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:07:12.184 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:07:12.185 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:07:12.185 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:07:12.185 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T04:07:12.185 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:12.185 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T04:07:12.185 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:12.185 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:07:12.185 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:07:12.185 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:07:12.185 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:07:12.185 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:12.185 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:07:12.185 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T04:07:12.185 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:07:12.186 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:12.186 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:12.186 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:07:12.186 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T04:07:12.186 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:07:12.186 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:07:12.186 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:07:12.186 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:12.186 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T04:07:12.186 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T04:07:12.186 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:07:12.186 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:07:12.186 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:07:12.186 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:12.187 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:12.187 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:07:12.187 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:07:12.187 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:07:12.187 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:07:12.187 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:12.187 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:07:12.187 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T04:07:12.187 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:07:12.187 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:12.187 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:12.187 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:07:12.187 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T04:07:12.187 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:07:12.187 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:07:12.188 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:07:12.188 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:12.188 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T04:07:12.188 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T04:07:12.188 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:07:12.188 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:07:12.188 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:07:12.188 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:12.188 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:12.188 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:07:12.188 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:07:12.188 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:07:12.188 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:07:12.189 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:12.189 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:07:12.189 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T04:07:12.189 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:07:12.189 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:12.189 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:12.189 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:07:12.189 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T04:07:12.189 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:07:12.189 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:07:12.189 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:07:12.189 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:12.189 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T04:07:12.189 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T04:07:12.190 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:07:12.190 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:07:12.190 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:07:12.190 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:07:12.190 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:07:12.190 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:12.190 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T04:07:12.190 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T04:07:12.190 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:12.190 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:07:12.190 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:07:12.190 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:07:12.190 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:07:12.190 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:07:12.191 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T04:07:12.191 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:12.191 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:07:12.191 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:07:12.191 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 3 2024-09-16T04:07:12.191 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:07:12.191 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:07:12.191 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T04:07:12.191 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:12.191 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:07:12.191 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:07:12.191 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 2 2024-09-16T04:07:12.191 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:07:12.191 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:07:12.192 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:12.192 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T04:07:12.192 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T04:07:12.193 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.149+0000 7f3d3a81b640 10 librados: watch_flush enter 2024-09-16T04:07:12.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.149+0000 7f3d3a81b640 10 librados: watch_flush exit 2024-09-16T04:07:12.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.149+0000 7f3d3a81b640 20 client.1014122.objecter shutdown clearing up homeless session... 2024-09-16T04:07:12.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.149+0000 7f3d3a81b640 10 client.1014122.objecter successfully canceled tick 2024-09-16T04:07:12.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.149+0000 7f3d3a81b640 1 -- 172.21.15.42:0/1984703963 >> v2:172.21.15.154:6812/254845342 conn(0x7f3d20037810 msgr2=0x7f3d20039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:12.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.149+0000 7f3d3a81b640 1 --2- 172.21.15.42:0/1984703963 >> v2:172.21.15.154:6812/254845342 conn(0x7f3d20037810 0x7f3d20039cd0 secure :-1 s=READY pgs=3876 cs=0 l=1 rev1=1 crypto rx=0x7f3d2c0ae000 tx=0x7f3d2c0a7070 comp rx=0 tx=0).stop 2024-09-16T04:07:12.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.149+0000 7f3d3a81b640 10 monclient: shutdown 2024-09-16T04:07:12.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.149+0000 7f3d3a81b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:07:12.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.149+0000 7f3d3a81b640 1 -- 172.21.15.42:0/1984703963 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3d340a1c50 msgr2=0x7f3d34135050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:12.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.149+0000 7f3d38db8640 1 -- 172.21.15.42:0/1984703963 reap_dead start 2024-09-16T04:07:12.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.149+0000 7f3d3a81b640 1 --2- 172.21.15.42:0/1984703963 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3d340a1c50 0x7f3d34135050 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f3d2800c480 tx=0x7f3d2800c940 comp rx=0 tx=0).stop 2024-09-16T04:07:12.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.149+0000 7f3d3a81b640 1 -- 172.21.15.42:0/1984703963 shutdown_connections 2024-09-16T04:07:12.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.149+0000 7f3d3a81b640 1 --2- 172.21.15.42:0/1984703963 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3d340a1c50 0x7f3d34135050 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:12.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.149+0000 7f3d3a81b640 1 -- 172.21.15.42:0/1984703963 >> 172.21.15.42:0/1984703963 conn(0x7f3d340a2420 msgr2=0x7f3d340a0a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:07:12.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.153+0000 7f3d3a81b640 1 -- 172.21.15.42:0/1984703963 shutdown_connections 2024-09-16T04:07:12.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.153+0000 7f3d3a81b640 1 -- 172.21.15.42:0/1984703963 wait complete. 2024-09-16T04:07:12.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.153+0000 7f3d3a81b640 1 librados: shutdown 2024-09-16T04:07:12.195 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:07:12.195 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T04:07:12.195 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T04:07:12.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.313+0000 7f919821a640 10 monclient: get_monmap_and_config 2024-09-16T04:07:12.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.313+0000 7f919821a640 10 monclient: build_initial_monmap 2024-09-16T04:07:12.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.313+0000 7f919821a640 10 monclient: monmap: 2024-09-16T04:07:12.321 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:07:12.321 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:12.321 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:07:12.319775+0000 2024-09-16T04:07:12.322 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:07:12.319775+0000 2024-09-16T04:07:12.322 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:07:12.322 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:12.322 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:07:12.322 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:07:12.322 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:07:12.322 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:07:12.322 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:07:12.322 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:12.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.317+0000 7f919821a640 1 Processor -- start 2024-09-16T04:07:12.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.317+0000 7f919821a640 1 -- start start 2024-09-16T04:07:12.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.317+0000 7f919821a640 10 monclient: init 2024-09-16T04:07:12.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.317+0000 7f919821a640 10 monclient: _reopen_session rank -1 2024-09-16T04:07:12.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.317+0000 7f919821a640 10 monclient: _add_conns ranks=[2,1,4,3,0] 2024-09-16T04:07:12.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.317+0000 7f919821a640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f919009fb00 0x7f91900a1330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:12.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.317+0000 7f919821a640 10 monclient(hunting): picked mon.noname-b con 0x7f919009fb00 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:07:12.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.317+0000 7f919821a640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f919009ba50 addr v1:172.21.15.154:6789/0 2024-09-16T04:07:12.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.317+0000 7f919821a640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f919009c5a0 0x7f919013c240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:12.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.317+0000 7f919821a640 10 monclient(hunting): picked mon.noname-c con 0x7f919009c5a0 addr v2:172.21.15.154:3300/0 2024-09-16T04:07:12.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.317+0000 7f919821a640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f91900a1e20 con 0x7f919009ba50 2024-09-16T04:07:12.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.317+0000 7f919821a640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:12.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.317+0000 7f919821a640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f919009e250 con 0x7f919009c5a0 2024-09-16T04:07:12.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.317+0000 7f919821a640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:12.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.317+0000 7f919821a640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f919009ea30 con 0x7f919009fb00 2024-09-16T04:07:12.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.317+0000 7f919821a640 10 monclient(hunting): _renew_subs 2024-09-16T04:07:12.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.317+0000 7f919821a640 10 monclient(hunting): authenticate will time out at 2479.289489s 2024-09-16T04:07:12.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.317+0000 7f9195fb6640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f919009fb00 0x7f91900a1330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:12.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.317+0000 7f9195fb6640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f919009fb00 0x7f91900a1330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:56972/0 (socket says 172.21.15.42:56972) 2024-09-16T04:07:12.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.317+0000 7f9195fb6640 1 -- 172.21.15.42:0/3869486269 learned_addr learned my addr 172.21.15.42:0/3869486269 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:07:12.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.317+0000 7f9195fb6640 10 monclient(hunting): get_auth_request con 0x7f919009fb00 auth_method 0 2024-09-16T04:07:12.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.317+0000 7f9195fb6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:12.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.317+0000 7f9195fb6640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:12.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.317+0000 7f9195fb6640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:12.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.317+0000 7f9195fb6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:07:12.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.317+0000 7f9195fb6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:07:12.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.317+0000 7f91967b7640 1 --2- 172.21.15.42:0/3869486269 >> v2:172.21.15.154:3300/0 conn(0x7f919009c5a0 0x7f919013c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:12.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.317+0000 7f9195fb6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:07:12.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.317+0000 7f91967b7640 10 monclient(hunting): get_auth_request con 0x7f919009c5a0 auth_method 0 2024-09-16T04:07:12.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.317+0000 7f91967b7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:12.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.317+0000 7f91967b7640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:12.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.317+0000 7f91967b7640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:12.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.317+0000 7f9195fb6640 10 monclient(hunting): handle_auth_done global_id 1014125 payload 274 2024-09-16T04:07:12.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.317+0000 7f9195fb6640 1 -- 172.21.15.42:0/3869486269 >> v2:172.21.15.154:3300/0 conn(0x7f919009c5a0 msgr2=0x7f919013c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:07:12.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.317+0000 7f9195fb6640 1 --2- 172.21.15.42:0/3869486269 >> v2:172.21.15.154:3300/0 conn(0x7f919009c5a0 0x7f919013c240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:12.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.317+0000 7f9195fb6640 1 -- 172.21.15.42:0/3869486269 >> v1:172.21.15.154:6789/0 conn(0x7f919009ba50 legacy=0x7f919009be90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:07:12.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.317+0000 7f9195fb6640 10 monclient: _finish_hunting 0 2024-09-16T04:07:12.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.317+0000 7f9195fb6640 1 monclient: found mon.noname-b 2024-09-16T04:07:12.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.317+0000 7f9195fb6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:07:12.329 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.317+0000 7f9195fb6640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T04:07:12.329 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.317+0000 7f9195fb6640 1 -- 172.21.15.42:0/3869486269 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f919009fee0 con 0x7f919009fb00 2024-09-16T04:07:12.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.321+0000 7f9195fb6640 1 --2- 172.21.15.42:0/3869486269 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f919009fb00 0x7f91900a1330 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f918c009870 tx=0x7f918c01c3e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5e495314c6b72a62 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:07:12.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.321+0000 7f9194fb4640 1 -- 172.21.15.42:0/3869486269 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f918c0b1070 con 0x7f919009fb00 2024-09-16T04:07:12.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.321+0000 7f9194fb4640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:12.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.321+0000 7f9194fb4640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:07:12.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.321+0000 7f9194fb4640 10 monclient: dump: 2024-09-16T04:07:12.337 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:07:12.337 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:12.337 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:12.337 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:12.337 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:12.338 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:12.338 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:07:12.338 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:07:12.338 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:07:12.338 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:12.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.321+0000 7f9194fb4640 10 monclient: _finish_auth 0 2024-09-16T04:07:12.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.321+0000 7f9194fb4640 10 monclient: _check_auth_tickets 2024-09-16T04:07:12.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.321+0000 7f9194fb4640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:07:12.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.321+0000 7f9194fb4640 1 -- 172.21.15.42:0/3869486269 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f918c0ac600 con 0x7f919009fb00 2024-09-16T04:07:12.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.321+0000 7f919821a640 5 monclient: authenticate success, global_id 1014125 2024-09-16T04:07:12.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.321+0000 7f919821a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:07:12.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.321+0000 7f9194fb4640 10 monclient: handle_config config(0 keys) 2024-09-16T04:07:12.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.321+0000 7f919821a640 10 monclient: get_monmap_and_config success 2024-09-16T04:07:12.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.321+0000 7f919821a640 4 set_mon_vals no callback set 2024-09-16T04:07:12.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.321+0000 7f9194fb4640 1 -- 172.21.15.42:0/3869486269 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f918c0ac9d0 con 0x7f919009fb00 2024-09-16T04:07:12.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.321+0000 7f9194fb4640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:12.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.321+0000 7f9194fb4640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:07:12.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.321+0000 7f9194fb4640 10 monclient: dump: 2024-09-16T04:07:12.339 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:07:12.339 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:12.339 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:12.339 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:12.339 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:12.339 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:12.340 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:07:12.340 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:07:12.340 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:07:12.340 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:12.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.321+0000 7f919821a640 10 monclient: shutdown 2024-09-16T04:07:12.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.321+0000 7f919821a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:07:12.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.321+0000 7f919821a640 1 -- 172.21.15.42:0/3869486269 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f919009fb00 msgr2=0x7f91900a1330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:12.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.321+0000 7f919821a640 1 --2- 172.21.15.42:0/3869486269 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f919009fb00 0x7f91900a1330 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f918c009870 tx=0x7f918c01c3e0 comp rx=0 tx=0).stop 2024-09-16T04:07:12.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.321+0000 7f919821a640 1 -- 172.21.15.42:0/3869486269 shutdown_connections 2024-09-16T04:07:12.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.321+0000 7f919821a640 1 --2- 172.21.15.42:0/3869486269 >> v2:172.21.15.154:3300/0 conn(0x7f919009c5a0 0x7f919013c240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:12.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.321+0000 7f919821a640 1 --2- 172.21.15.42:0/3869486269 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f919009fb00 0x7f91900a1330 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:12.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.321+0000 7f919821a640 1 -- 172.21.15.42:0/3869486269 >> 172.21.15.42:0/3869486269 conn(0x7f91900a2420 msgr2=0x7f91900a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:07:12.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.321+0000 7f919821a640 1 -- 172.21.15.42:0/3869486269 shutdown_connections 2024-09-16T04:07:12.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.321+0000 7f919821a640 1 -- 172.21.15.42:0/3869486269 wait complete. 2024-09-16T04:07:12.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.321+0000 7f919821a640 10 monclient: build_initial_monmap 2024-09-16T04:07:12.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.321+0000 7f919821a640 10 monclient: monmap: 2024-09-16T04:07:12.342 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:07:12.342 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:07:12.342 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:07:12.342 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:07:12.342 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:07:12.342 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:12.342 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:07:12.342 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:07:12.342 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:07:12.342 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:12.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.325+0000 7f919821a640 1 librados: starting msgr at 2024-09-16T04:07:12.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.325+0000 7f919821a640 1 librados: starting objecter 2024-09-16T04:07:12.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.325+0000 7f919821a640 1 Processor -- start 2024-09-16T04:07:12.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.325+0000 7f919821a640 1 -- start start 2024-09-16T04:07:12.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.325+0000 7f919821a640 1 librados: setting wanted keys 2024-09-16T04:07:12.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.325+0000 7f919821a640 1 librados: calling monclient init 2024-09-16T04:07:12.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.325+0000 7f919821a640 10 monclient: init 2024-09-16T04:07:12.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.325+0000 7f919821a640 10 monclient: _reopen_session rank -1 2024-09-16T04:07:12.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.325+0000 7f919821a640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:07:12.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.325+0000 7f919821a640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f919009ba50 0x7f91901cf0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:12.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.325+0000 7f919821a640 10 monclient(hunting): picked mon.noname-a con 0x7f919009ba50 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:07:12.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.325+0000 7f919821a640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f919009c5a0 0x7f919013b800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:12.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.325+0000 7f919821a640 10 monclient(hunting): picked mon.noname-b con 0x7f919009c5a0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:07:12.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.325+0000 7f919821a640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f919013bd40 0x7f919006eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:12.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.325+0000 7f919821a640 10 monclient(hunting): picked mon.noname-c con 0x7f919013bd40 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:07:12.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.325+0000 7f919821a640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:12.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.325+0000 7f919821a640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f91900a1870 con 0x7f919009ba50 2024-09-16T04:07:12.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.325+0000 7f919821a640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:12.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.325+0000 7f919821a640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f919009d130 con 0x7f919009c5a0 2024-09-16T04:07:12.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.325+0000 7f919821a640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:12.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.325+0000 7f919821a640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f91900a19f0 con 0x7f919013bd40 2024-09-16T04:07:12.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.325+0000 7f919821a640 10 monclient(hunting): _renew_subs 2024-09-16T04:07:12.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.325+0000 7f919821a640 10 monclient(hunting): authenticate will time out at 2479.296931s 2024-09-16T04:07:12.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.325+0000 7f91967b7640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f919013bd40 0x7f919006eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:12.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.325+0000 7f9195fb6640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f919009ba50 0x7f91901cf0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:12.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.325+0000 7f91967b7640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f919013bd40 0x7f919006eac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:56984/0 (socket says 172.21.15.42:56984) 2024-09-16T04:07:12.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.325+0000 7f91967b7640 1 -- 172.21.15.42:0/3758495917 learned_addr learned my addr 172.21.15.42:0/3758495917 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:07:12.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.325+0000 7f91967b7640 10 monclient(hunting): get_auth_request con 0x7f919013bd40 auth_method 0 2024-09-16T04:07:12.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.325+0000 7f91967b7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:12.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.325+0000 7f91967b7640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:12.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.325+0000 7f91967b7640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:12.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.325+0000 7f9195fb6640 10 monclient(hunting): get_auth_request con 0x7f919009ba50 auth_method 0 2024-09-16T04:07:12.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.325+0000 7f9195fb6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:12.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.325+0000 7f9195fb6640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:12.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.325+0000 7f9195fb6640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:12.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.325+0000 7f9195fb6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:07:12.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.325+0000 7f9195fb6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:07:12.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.325+0000 7f9195fb6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:07:12.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.325+0000 7f91967b7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:07:12.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.325+0000 7f91967b7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:07:12.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.325+0000 7f91967b7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:07:12.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.325+0000 7f91957b5640 1 --2- 172.21.15.42:0/3758495917 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f919009c5a0 0x7f919013b800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:12.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.325+0000 7f91957b5640 10 monclient(hunting): get_auth_request con 0x7f919009c5a0 auth_method 0 2024-09-16T04:07:12.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.325+0000 7f91957b5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:12.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.325+0000 7f91957b5640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:12.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.325+0000 7f91957b5640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:12.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.325+0000 7f91957b5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:07:12.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.325+0000 7f91957b5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:07:12.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.325+0000 7f91957b5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:07:12.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.325+0000 7f91967b7640 10 monclient(hunting): handle_auth_done global_id 1014128 payload 824 2024-09-16T04:07:12.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.325+0000 7f91967b7640 1 -- 172.21.15.42:0/3758495917 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f919009c5a0 msgr2=0x7f919013b800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:12.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.325+0000 7f91967b7640 1 --2- 172.21.15.42:0/3758495917 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f919009c5a0 0x7f919013b800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:12.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.325+0000 7f91967b7640 1 -- 172.21.15.42:0/3758495917 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f919009ba50 msgr2=0x7f91901cf0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:12.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.325+0000 7f91967b7640 1 --2- 172.21.15.42:0/3758495917 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f919009ba50 0x7f91901cf0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:12.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.325+0000 7f91967b7640 10 monclient: _finish_hunting 0 2024-09-16T04:07:12.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.325+0000 7f91967b7640 1 monclient: found mon.noname-c 2024-09-16T04:07:12.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.325+0000 7f91967b7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:07:12.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.325+0000 7f91967b7640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T04:07:12.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.325+0000 7f91967b7640 1 -- 172.21.15.42:0/3758495917 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f919013c1f0 con 0x7f919013bd40 2024-09-16T04:07:12.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.325+0000 7f9195fb6640 1 --2- 172.21.15.42:0/3758495917 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f919009ba50 0x7f91901cf0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:07:12.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.325+0000 7f91967b7640 1 --2- 172.21.15.42:0/3758495917 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f919013bd40 0x7f919006eac0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f9188007e60 tx=0x7f918800e4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:07:12.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.325+0000 7f91867fc640 10 client.?.objecter ms_handle_connect 0x7f919013bd40 2024-09-16T04:07:12.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.325+0000 7f91867fc640 10 client.?.objecter resend_mon_ops 2024-09-16T04:07:12.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.325+0000 7f91867fc640 1 -- 172.21.15.42:0/3758495917 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9188014620 con 0x7f919013bd40 2024-09-16T04:07:12.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.325+0000 7f91867fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:12.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.325+0000 7f91867fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:07:12.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.325+0000 7f91867fc640 10 monclient: dump: 2024-09-16T04:07:12.349 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:07:12.349 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:12.349 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:12.349 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:12.349 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:12.349 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:12.349 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:07:12.349 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:07:12.349 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:07:12.349 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:12.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.325+0000 7f91867fc640 10 monclient: _finish_auth 0 2024-09-16T04:07:12.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.325+0000 7f91867fc640 10 monclient: _check_auth_tickets 2024-09-16T04:07:12.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.325+0000 7f91867fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:07:12.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.325+0000 7f91867fc640 1 -- 172.21.15.42:0/3758495917 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9188019070 con 0x7f919013bd40 2024-09-16T04:07:12.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.325+0000 7f91867fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:07:12.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.325+0000 7f91867fc640 1 -- 172.21.15.42:0/3758495917 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9188014a00 con 0x7f919013bd40 2024-09-16T04:07:12.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.325+0000 7f919821a640 5 monclient: authenticate success, global_id 1014128 2024-09-16T04:07:12.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.325+0000 7f919821a640 10 monclient: _renew_subs 2024-09-16T04:07:12.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.325+0000 7f919821a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:07:12.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.325+0000 7f919821a640 1 -- 172.21.15.42:0/3758495917 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9190073a30 con 0x7f919013bd40 2024-09-16T04:07:12.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.325+0000 7f9187fff640 4 set_mon_vals no callback set 2024-09-16T04:07:12.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.325+0000 7f91867fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:12.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.325+0000 7f91867fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:07:12.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.325+0000 7f91867fc640 10 monclient: dump: 2024-09-16T04:07:12.350 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:07:12.350 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:12.350 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:12.351 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:12.351 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:12.351 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:12.351 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:07:12.351 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:07:12.351 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:07:12.351 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:12.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.325+0000 7f919821a640 10 client.1014128.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:07:12.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.325+0000 7f919821a640 10 monclient: _renew_subs 2024-09-16T04:07:12.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.325+0000 7f919821a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:07:12.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.325+0000 7f919821a640 1 -- 172.21.15.42:0/3758495917 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f9190070760 con 0x7f919013bd40 2024-09-16T04:07:12.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.325+0000 7f919821a640 1 librados: init done 2024-09-16T04:07:12.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.325+0000 7f919821a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:07:12.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.325+0000 7f919821a640 1 --2- 172.21.15.42:0/3758495917 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f919002c650 0x7f919002ca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:12.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.325+0000 7f919821a640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:12.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.325+0000 7f919821a640 1 -- 172.21.15.42:0/3758495917 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f9190128b60 con 0x7f919002c650 2024-09-16T04:07:12.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.325+0000 7f91867fc640 1 -- 172.21.15.42:0/3758495917 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f9188014ba0 con 0x7f919013bd40 2024-09-16T04:07:12.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.325+0000 7f9195fb6640 1 --2- 172.21.15.42:0/3758495917 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f919002c650 0x7f919002ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:12.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.325+0000 7f9195fb6640 10 monclient: get_auth_request con 0x7f919002c650 auth_method 0 2024-09-16T04:07:12.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.325+0000 7f9195fb6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:12.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.325+0000 7f9195fb6640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:12.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.325+0000 7f9195fb6640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:12.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.325+0000 7f91867fc640 1 --2- 172.21.15.42:0/3758495917 >> v2:172.21.15.154:6812/254845342 conn(0x7f9174037810 0x7f9174039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:12.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.325+0000 7f9195fb6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:07:12.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.325+0000 7f9195fb6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:07:12.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.325+0000 7f9195fb6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:07:12.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.325+0000 7f91867fc640 1 -- 172.21.15.42:0/3758495917 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(706..706 src has 451..706) ==== 5040+0+0 (secure 0 0 0) 0x7f9188049440 con 0x7f919013bd40 2024-09-16T04:07:12.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.325+0000 7f91867fc640 10 client.1014128.objecter ms_dispatch 0x7f9190000ba0 osd_map(706..706 src has 451..706) 2024-09-16T04:07:12.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.325+0000 7f91867fc640 3 client.1014128.objecter handle_osd_map got epochs [706,706] > 0 2024-09-16T04:07:12.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.325+0000 7f91867fc640 3 client.1014128.objecter handle_osd_map decoding full epoch 706 2024-09-16T04:07:12.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.325+0000 7f91867fc640 20 client.1014128.objecter dump_active .. 0 homeless 2024-09-16T04:07:12.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.325+0000 7f9195fb6640 10 monclient(hunting): handle_auth_done global_id 1014131 payload 824 2024-09-16T04:07:12.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.325+0000 7f9195fb6640 1 -- 172.21.15.42:0/3758495917 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f919009d4a0 con 0x7f919002c650 2024-09-16T04:07:12.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.325+0000 7f91957b5640 1 --2- 172.21.15.42:0/3758495917 >> v2:172.21.15.154:6812/254845342 conn(0x7f9174037810 0x7f9174039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:12.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.325+0000 7f91957b5640 10 monclient: get_auth_request con 0x7f9174037810 auth_method 0 2024-09-16T04:07:12.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.325+0000 7f91867fc640 10 client.1014128.objecter ms_handle_connect 0x7f919002c650 2024-09-16T04:07:12.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.325+0000 7f9195fb6640 1 --2- 172.21.15.42:0/3758495917 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f919002c650 0x7f919002ca30 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f91900a69e0 tx=0x7f918c019180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:07:12.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.325+0000 7f91867fc640 10 client.1014128.objecter resend_mon_ops 2024-09-16T04:07:12.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.325+0000 7f91867fc640 1 -- 172.21.15.42:0/3758495917 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f918c0b1040 con 0x7f919002c650 2024-09-16T04:07:12.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.325+0000 7f91867fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:12.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.325+0000 7f91867fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:07:12.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.325+0000 7f91867fc640 10 monclient: dump: 2024-09-16T04:07:12.354 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:07:12.354 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:12.354 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:12.354 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:12.354 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:12.354 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:12.355 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:07:12.355 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:07:12.355 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:07:12.355 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:12.355 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.329+0000 7f91867fc640 1 -- 172.21.15.42:0/3758495917 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f918c0b6640 con 0x7f919002c650 2024-09-16T04:07:12.355 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.329+0000 7f91867fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:07:12.355 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.329+0000 7f91867fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:07:12.355 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.329+0000 7f91867fc640 1 -- 172.21.15.42:0/3758495917 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f919002c650 msgr2=0x7f919002ca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:12.355 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.329+0000 7f91867fc640 1 --2- 172.21.15.42:0/3758495917 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f919002c650 0x7f919002ca30 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f91900a69e0 tx=0x7f918c019180 comp rx=0 tx=0).stop 2024-09-16T04:07:12.355 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.329+0000 7f91867fc640 1 -- 172.21.15.42:0/3758495917 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f919002c650 msgr2=0x7f919002ca30 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:07:12.355 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.329+0000 7f91867fc640 1 --2- 172.21.15.42:0/3758495917 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f919002c650 0x7f919002ca30 secure :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f91900a69e0 tx=0x7f918c019180 comp rx=0 tx=0).stop 2024-09-16T04:07:12.355 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.329+0000 7f91867fc640 10 client.1014128.objecter ms_handle_connect 0x7f9174037810 2024-09-16T04:07:12.355 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.329+0000 7f91957b5640 1 --2- 172.21.15.42:0/3758495917 >> v2:172.21.15.154:6812/254845342 conn(0x7f9174037810 0x7f9174039cd0 secure :-1 s=READY pgs=3877 cs=0 l=1 rev1=1 crypto rx=0x7f9190128530 tx=0x7f9180007e40 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:07:12.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.333+0000 7f919821a640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:07:12.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.333+0000 7f919821a640 1 --2- 172.21.15.42:0/3758495917 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f919002cf70 0x7f919002d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:12.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.333+0000 7f919821a640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:12.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.333+0000 7f919821a640 1 -- 172.21.15.42:0/3758495917 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f919011f690 con 0x7f919002cf70 2024-09-16T04:07:12.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.333+0000 7f9195fb6640 1 --2- 172.21.15.42:0/3758495917 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f919002cf70 0x7f919002d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:12.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.333+0000 7f9195fb6640 10 monclient: get_auth_request con 0x7f919002cf70 auth_method 0 2024-09-16T04:07:12.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.333+0000 7f9195fb6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:12.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.333+0000 7f9195fb6640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:12.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.333+0000 7f9195fb6640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:12.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.333+0000 7f9195fb6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:07:12.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.333+0000 7f9195fb6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:07:12.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.333+0000 7f9195fb6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:07:12.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.333+0000 7f9195fb6640 10 monclient(hunting): handle_auth_done global_id 1014134 payload 824 2024-09-16T04:07:12.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.333+0000 7f9195fb6640 1 -- 172.21.15.42:0/3758495917 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f9190128f40 con 0x7f919002cf70 2024-09-16T04:07:12.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.333+0000 7f91867fc640 10 client.1014128.objecter ms_handle_connect 0x7f919002cf70 2024-09-16T04:07:12.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.333+0000 7f91867fc640 10 client.1014128.objecter resend_mon_ops 2024-09-16T04:07:12.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.333+0000 7f9195fb6640 1 --2- 172.21.15.42:0/3758495917 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f919002cf70 0x7f919002d380 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f918c01c420 tx=0x7f918c009240 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:07:12.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.333+0000 7f91867fc640 1 -- 172.21.15.42:0/3758495917 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f918c0b1070 con 0x7f919002cf70 2024-09-16T04:07:12.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.333+0000 7f91867fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:12.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.333+0000 7f91867fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:07:12.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.333+0000 7f91867fc640 10 monclient: dump: 2024-09-16T04:07:12.359 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:07:12.359 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:12.359 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:12.359 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:12.359 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:12.359 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:12.359 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:07:12.359 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:07:12.359 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:07:12.359 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:12.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.333+0000 7f91867fc640 1 -- 172.21.15.42:0/3758495917 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4545 (secure 0 0 0) 0x7f918c0c27f0 con 0x7f919002cf70 2024-09-16T04:07:12.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.333+0000 7f91867fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:07:12.360 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.333+0000 7f91867fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:07:12.360 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.333+0000 7f91867fc640 1 -- 172.21.15.42:0/3758495917 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f919002cf70 msgr2=0x7f919002d380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:12.360 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.333+0000 7f91867fc640 1 --2- 172.21.15.42:0/3758495917 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f919002cf70 0x7f919002d380 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f918c01c420 tx=0x7f918c009240 comp rx=0 tx=0).stop 2024-09-16T04:07:12.360 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.333+0000 7f91867fc640 1 -- 172.21.15.42:0/3758495917 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f919002cf70 msgr2=0x7f919002d380 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:07:12.360 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.333+0000 7f91867fc640 1 --2- 172.21.15.42:0/3758495917 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f919002cf70 0x7f919002d380 secure :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f918c01c420 tx=0x7f918c009240 comp rx=0 tx=0).stop 2024-09-16T04:07:12.360 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T04:07:12.360 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:07:12.360 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:07:12.360 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T04:07:12.361 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 232, 2024-09-16T04:07:12.361 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 27535, 2024-09-16T04:07:12.361 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T04:07:12.361 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T04:07:12.361 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T04:07:12.361 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T04:07:12.361 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:07:12.361 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 7, 2024-09-16T04:07:12.361 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:07:12.361 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T04:07:12.361 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T04:07:12.361 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:07:12.361 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:07:12.361 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:07:12.362 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:07:12.362 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:07:12.362 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:07:12.362 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:07:12.362 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:07:12.362 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:07:12.362 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:07:12.362 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:07:12.362 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:07:12.362 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:07:12.362 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T04:07:12.362 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:07:12.362 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:07:12.362 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:07:12.363 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:07:12.363 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:07:12.363 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:07:12.363 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:07:12.363 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:07:12.363 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:07:12.363 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:07:12.363 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:07:12.363 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:07:12.363 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:12.363 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T04:07:12.363 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T04:07:12.363 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T04:07:12.364 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T04:07:12.364 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T04:07:12.364 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T04:07:12.364 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:07:12.364 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:07:12.364 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T04:07:12.364 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T04:07:12.364 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T04:07:12.364 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T04:07:12.365 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T04:07:12.365 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T04:07:12.365 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T04:07:12.365 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:07:12.365 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T04:07:12.365 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:07:12.365 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:07:12.365 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:07:12.365 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:07:12.365 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:07:12.365 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:07:12.365 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:07:12.365 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:07:12.365 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:07:12.366 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:07:12.366 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:07:12.366 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:07:12.366 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T04:07:12.366 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:12.366 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T04:07:12.366 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:12.366 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:07:12.366 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:07:12.366 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:07:12.366 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:07:12.366 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:12.366 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:07:12.366 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T04:07:12.367 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:07:12.367 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:12.367 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:12.367 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:07:12.367 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T04:07:12.367 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:07:12.367 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:07:12.367 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:07:12.367 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:12.367 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T04:07:12.367 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T04:07:12.367 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:07:12.367 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:07:12.367 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:07:12.368 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:12.368 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:12.368 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:07:12.368 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:07:12.368 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:07:12.368 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:07:12.368 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:12.368 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:07:12.368 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T04:07:12.368 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:07:12.368 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:12.368 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:12.368 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:07:12.369 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T04:07:12.369 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:07:12.369 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:07:12.369 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:07:12.369 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:12.369 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T04:07:12.369 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T04:07:12.369 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:07:12.369 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:07:12.369 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:07:12.369 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:12.369 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:12.369 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:07:12.369 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:07:12.369 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:07:12.370 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:07:12.370 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:12.370 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:07:12.370 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T04:07:12.370 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:07:12.370 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:12.370 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:12.370 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:07:12.370 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T04:07:12.370 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:07:12.370 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:07:12.370 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:07:12.370 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:12.371 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T04:07:12.371 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T04:07:12.371 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:07:12.371 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:07:12.371 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:07:12.371 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:07:12.371 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:07:12.371 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:12.371 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T04:07:12.371 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T04:07:12.371 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:12.371 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:07:12.371 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:07:12.371 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:07:12.372 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:07:12.372 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:07:12.372 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T04:07:12.372 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:12.372 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:07:12.372 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:07:12.372 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 2 2024-09-16T04:07:12.372 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:07:12.372 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:07:12.372 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:12.372 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T04:07:12.372 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T04:07:12.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.333+0000 7f919821a640 10 librados: watch_flush enter 2024-09-16T04:07:12.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.333+0000 7f919821a640 10 librados: watch_flush exit 2024-09-16T04:07:12.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.333+0000 7f919821a640 20 client.1014128.objecter shutdown clearing up homeless session... 2024-09-16T04:07:12.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.333+0000 7f919821a640 10 client.1014128.objecter successfully canceled tick 2024-09-16T04:07:12.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.333+0000 7f919821a640 1 -- 172.21.15.42:0/3758495917 >> v2:172.21.15.154:6812/254845342 conn(0x7f9174037810 msgr2=0x7f9174039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:12.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.333+0000 7f919821a640 1 --2- 172.21.15.42:0/3758495917 >> v2:172.21.15.154:6812/254845342 conn(0x7f9174037810 0x7f9174039cd0 secure :-1 s=READY pgs=3877 cs=0 l=1 rev1=1 crypto rx=0x7f9190128530 tx=0x7f9180007e40 comp rx=0 tx=0).stop 2024-09-16T04:07:12.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.333+0000 7f919821a640 10 monclient: shutdown 2024-09-16T04:07:12.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.333+0000 7f919821a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:07:12.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.333+0000 7f919821a640 1 -- 172.21.15.42:0/3758495917 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f919013bd40 msgr2=0x7f919006eac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:12.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.333+0000 7f919821a640 1 --2- 172.21.15.42:0/3758495917 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f919013bd40 0x7f919006eac0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f9188007e60 tx=0x7f918800e4b0 comp rx=0 tx=0).stop 2024-09-16T04:07:12.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.333+0000 7f91967b7640 1 -- 172.21.15.42:0/3758495917 reap_dead start 2024-09-16T04:07:12.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.333+0000 7f91967b7640 1 -- 172.21.15.42:0/3758495917 reap_dead start 2024-09-16T04:07:12.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.337+0000 7f919821a640 1 -- 172.21.15.42:0/3758495917 shutdown_connections 2024-09-16T04:07:12.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.337+0000 7f919821a640 1 -- 172.21.15.42:0/3758495917 >> 172.21.15.42:0/3758495917 conn(0x7f91900a2420 msgr2=0x7f9190070ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:07:12.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.337+0000 7f919821a640 1 -- 172.21.15.42:0/3758495917 shutdown_connections 2024-09-16T04:07:12.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.337+0000 7f919821a640 1 -- 172.21.15.42:0/3758495917 wait complete. 2024-09-16T04:07:12.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.337+0000 7f919821a640 1 librados: shutdown 2024-09-16T04:07:12.376 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:07:12.376 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T04:07:12.376 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T04:07:12.491 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.485+0000 7f8379962640 10 monclient: get_monmap_and_config 2024-09-16T04:07:12.491 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.485+0000 7f8379962640 10 monclient: build_initial_monmap 2024-09-16T04:07:12.491 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.485+0000 7f8379962640 10 monclient: monmap: 2024-09-16T04:07:12.491 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:07:12.491 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:12.491 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:07:12.489551+0000 2024-09-16T04:07:12.491 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:07:12.489551+0000 2024-09-16T04:07:12.491 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:07:12.491 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:12.492 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:07:12.492 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:07:12.492 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:07:12.492 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:07:12.492 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:07:12.492 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:12.494 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.485+0000 7f8379962640 1 Processor -- start 2024-09-16T04:07:12.494 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.485+0000 7f8379962640 1 -- start start 2024-09-16T04:07:12.494 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.485+0000 7f8379962640 10 monclient: init 2024-09-16T04:07:12.494 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.485+0000 7f8379962640 10 monclient: _reopen_session rank -1 2024-09-16T04:07:12.494 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.485+0000 7f8379962640 10 monclient: _add_conns ranks=[2,4,1,3,0] 2024-09-16T04:07:12.494 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.485+0000 7f8379962640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f83740a39b0 0x7f8374133b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:12.494 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.485+0000 7f8379962640 10 monclient(hunting): picked mon.noname-b con 0x7f83740a39b0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:07:12.495 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.485+0000 7f8379962640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f83740a3d90 0x7f8374137000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:12.495 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.485+0000 7f8379962640 10 monclient(hunting): picked mon.noname-c con 0x7f83740a3d90 addr v2:172.21.15.154:3300/0 2024-09-16T04:07:12.495 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.485+0000 7f8379962640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8374137540 addr v1:172.21.15.154:6789/0 2024-09-16T04:07:12.495 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.485+0000 7f8379962640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f837409c140 con 0x7f8374137540 2024-09-16T04:07:12.495 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.485+0000 7f8379962640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:12.495 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.485+0000 7f8379962640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f837409c420 con 0x7f83740a3d90 2024-09-16T04:07:12.495 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.485+0000 7f8379962640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:12.495 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.485+0000 7f8379962640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f837406dd60 con 0x7f83740a39b0 2024-09-16T04:07:12.495 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.485+0000 7f8379962640 10 monclient(hunting): _renew_subs 2024-09-16T04:07:12.495 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.485+0000 7f8379962640 10 monclient(hunting): authenticate will time out at 2479.459712s 2024-09-16T04:07:12.495 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.485+0000 7f8372ffd640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f83740a39b0 0x7f8374133b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:12.495 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.489+0000 7f8372ffd640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f83740a39b0 0x7f8374133b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:57014/0 (socket says 172.21.15.42:57014) 2024-09-16T04:07:12.495 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.489+0000 7f8372ffd640 1 -- 172.21.15.42:0/1730015338 learned_addr learned my addr 172.21.15.42:0/1730015338 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:07:12.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.489+0000 7f8372ffd640 10 monclient(hunting): get_auth_request con 0x7f83740a39b0 auth_method 0 2024-09-16T04:07:12.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.489+0000 7f8372ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:12.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.489+0000 7f8372ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:12.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.489+0000 7f8372ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:12.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.489+0000 7f83727fc640 1 --2- 172.21.15.42:0/1730015338 >> v2:172.21.15.154:3300/0 conn(0x7f83740a3d90 0x7f8374137000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:12.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.489+0000 7f83727fc640 10 monclient(hunting): get_auth_request con 0x7f83740a3d90 auth_method 0 2024-09-16T04:07:12.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.489+0000 7f83727fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:12.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.489+0000 7f83727fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:12.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.489+0000 7f83727fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:12.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.489+0000 7f8372ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:07:12.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.489+0000 7f8372ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:07:12.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.489+0000 7f8372ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:07:12.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.489+0000 7f83727fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:07:12.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.489+0000 7f83727fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:07:12.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.489+0000 7f83727fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:07:12.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.489+0000 7f8372ffd640 10 monclient(hunting): handle_auth_done global_id 1014137 payload 274 2024-09-16T04:07:12.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.489+0000 7f8372ffd640 1 -- 172.21.15.42:0/1730015338 >> v2:172.21.15.154:3300/0 conn(0x7f83740a3d90 msgr2=0x7f8374137000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:07:12.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.489+0000 7f8372ffd640 1 --2- 172.21.15.42:0/1730015338 >> v2:172.21.15.154:3300/0 conn(0x7f83740a3d90 0x7f8374137000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:12.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.489+0000 7f8372ffd640 1 -- 172.21.15.42:0/1730015338 >> v1:172.21.15.154:6789/0 conn(0x7f8374137540 legacy=0x7f837413e390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:12.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.489+0000 7f8372ffd640 10 monclient: _finish_hunting 0 2024-09-16T04:07:12.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.489+0000 7f8372ffd640 1 monclient: found mon.noname-b 2024-09-16T04:07:12.498 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.489+0000 7f8372ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:07:12.498 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.489+0000 7f8372ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T04:07:12.498 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.489+0000 7f8372ffd640 1 -- 172.21.15.42:0/1730015338 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83740a5e20 con 0x7f83740a39b0 2024-09-16T04:07:12.505 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.489+0000 7f8372ffd640 1 --2- 172.21.15.42:0/1730015338 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f83740a39b0 0x7f8374133b80 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f8364009e50 tx=0x7f836401c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=95bdd7d16e52fc33 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:07:12.505 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.489+0000 7f8371ffb640 1 -- 172.21.15.42:0/1730015338 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83640a55b0 con 0x7f83740a39b0 2024-09-16T04:07:12.505 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.489+0000 7f8371ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:12.505 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.489+0000 7f8371ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:07:12.505 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.489+0000 7f8371ffb640 10 monclient: dump: 2024-09-16T04:07:12.505 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:07:12.505 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:12.505 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:12.505 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:12.505 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:12.506 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:12.506 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:07:12.506 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:07:12.506 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:07:12.506 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:12.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.489+0000 7f8371ffb640 10 monclient: _finish_auth 0 2024-09-16T04:07:12.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.489+0000 7f8371ffb640 10 monclient: _check_auth_tickets 2024-09-16T04:07:12.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.489+0000 7f8371ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:07:12.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.489+0000 7f8371ffb640 1 -- 172.21.15.42:0/1730015338 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f83640a5750 con 0x7f83740a39b0 2024-09-16T04:07:12.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.489+0000 7f8379962640 5 monclient: authenticate success, global_id 1014137 2024-09-16T04:07:12.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.489+0000 7f8371ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:07:12.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.489+0000 7f8371ffb640 1 -- 172.21.15.42:0/1730015338 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83640a5b20 con 0x7f83740a39b0 2024-09-16T04:07:12.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.489+0000 7f8379962640 10 monclient: get_monmap_and_config success 2024-09-16T04:07:12.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.489+0000 7f8379962640 4 set_mon_vals no callback set 2024-09-16T04:07:12.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.489+0000 7f8371ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:12.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.489+0000 7f8371ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:07:12.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.489+0000 7f8371ffb640 10 monclient: dump: 2024-09-16T04:07:12.507 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:07:12.507 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:12.507 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:12.507 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:12.507 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:12.507 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:12.507 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:07:12.507 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:07:12.507 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:07:12.507 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:12.508 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.489+0000 7f8379962640 10 monclient: shutdown 2024-09-16T04:07:12.508 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.489+0000 7f8379962640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:07:12.508 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.489+0000 7f8379962640 1 -- 172.21.15.42:0/1730015338 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f83740a39b0 msgr2=0x7f8374133b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:12.508 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.489+0000 7f8379962640 1 --2- 172.21.15.42:0/1730015338 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f83740a39b0 0x7f8374133b80 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f8364009e50 tx=0x7f836401c8f0 comp rx=0 tx=0).stop 2024-09-16T04:07:12.508 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.493+0000 7f8379962640 1 -- 172.21.15.42:0/1730015338 shutdown_connections 2024-09-16T04:07:12.508 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.493+0000 7f8379962640 1 --2- 172.21.15.42:0/1730015338 >> v2:172.21.15.154:3300/0 conn(0x7f83740a3d90 0x7f8374137000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:12.508 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.493+0000 7f8379962640 1 --2- 172.21.15.42:0/1730015338 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f83740a39b0 0x7f8374133b80 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:12.508 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.493+0000 7f8379962640 1 -- 172.21.15.42:0/1730015338 >> 172.21.15.42:0/1730015338 conn(0x7f83740a9fb0 msgr2=0x7f83740aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:07:12.508 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.493+0000 7f8379962640 1 -- 172.21.15.42:0/1730015338 shutdown_connections 2024-09-16T04:07:12.508 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.493+0000 7f8379962640 1 -- 172.21.15.42:0/1730015338 wait complete. 2024-09-16T04:07:12.508 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.493+0000 7f8379962640 10 monclient: build_initial_monmap 2024-09-16T04:07:12.508 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.493+0000 7f8379962640 10 monclient: monmap: 2024-09-16T04:07:12.508 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:07:12.509 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:07:12.509 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:07:12.509 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:07:12.509 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:07:12.509 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:12.509 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:07:12.509 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:07:12.509 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:07:12.509 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:12.509 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.493+0000 7f8379962640 1 librados: starting msgr at 2024-09-16T04:07:12.509 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.493+0000 7f8379962640 1 librados: starting objecter 2024-09-16T04:07:12.509 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.493+0000 7f8379962640 1 Processor -- start 2024-09-16T04:07:12.509 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.493+0000 7f8379962640 1 -- start start 2024-09-16T04:07:12.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.493+0000 7f8379962640 1 librados: setting wanted keys 2024-09-16T04:07:12.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.493+0000 7f8379962640 1 librados: calling monclient init 2024-09-16T04:07:12.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.493+0000 7f8379962640 10 monclient: init 2024-09-16T04:07:12.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.493+0000 7f8379962640 10 monclient: _reopen_session rank -1 2024-09-16T04:07:12.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.493+0000 7f8379962640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:07:12.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.493+0000 7f8379962640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f83740a39b0 0x7f83741308b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:12.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.493+0000 7f8379962640 10 monclient(hunting): picked mon.noname-c con 0x7f83740a39b0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:07:12.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.493+0000 7f8379962640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f83740a3d90 0x7f837413d100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:12.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.493+0000 7f8379962640 10 monclient(hunting): picked mon.noname-a con 0x7f83740a3d90 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:07:12.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.493+0000 7f8379962640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8374137540 0x7f837413d640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:12.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.493+0000 7f8379962640 10 monclient(hunting): picked mon.noname-b con 0x7f8374137540 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:07:12.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.493+0000 7f8379962640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:12.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.493+0000 7f8379962640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f83741424e0 con 0x7f83740a3d90 2024-09-16T04:07:12.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.493+0000 7f8379962640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:12.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.493+0000 7f8379962640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f8374142660 con 0x7f8374137540 2024-09-16T04:07:12.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.493+0000 7f8379962640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:12.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.493+0000 7f8379962640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f837409e7d0 con 0x7f83740a39b0 2024-09-16T04:07:12.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.493+0000 7f8379962640 10 monclient(hunting): _renew_subs 2024-09-16T04:07:12.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.493+0000 7f8379962640 10 monclient(hunting): authenticate will time out at 2479.465570s 2024-09-16T04:07:12.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.493+0000 7f8372ffd640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f83740a39b0 0x7f83741308b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:12.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.493+0000 7f8372ffd640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f83740a39b0 0x7f83741308b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:57026/0 (socket says 172.21.15.42:57026) 2024-09-16T04:07:12.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.493+0000 7f8372ffd640 1 -- 172.21.15.42:0/4250445256 learned_addr learned my addr 172.21.15.42:0/4250445256 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:07:12.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.493+0000 7f8372ffd640 10 monclient(hunting): get_auth_request con 0x7f83740a39b0 auth_method 0 2024-09-16T04:07:12.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.493+0000 7f8372ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:12.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.493+0000 7f8372ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:12.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.493+0000 7f8372ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:12.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.493+0000 7f83727fc640 1 --2- 172.21.15.42:0/4250445256 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f83740a3d90 0x7f837413d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:12.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.493+0000 7f83727fc640 10 monclient(hunting): get_auth_request con 0x7f83740a3d90 auth_method 0 2024-09-16T04:07:12.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.493+0000 7f83727fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:12.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.493+0000 7f83727fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:12.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.493+0000 7f83727fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:12.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.493+0000 7f8372ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:07:12.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.493+0000 7f8372ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:07:12.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.493+0000 7f8372ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:07:12.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.493+0000 7f83727fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:07:12.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.493+0000 7f83727fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:07:12.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.493+0000 7f83727fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:07:12.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.493+0000 7f8372ffd640 10 monclient(hunting): handle_auth_done global_id 1014140 payload 824 2024-09-16T04:07:12.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.493+0000 7f83737fe640 1 --2- 172.21.15.42:0/4250445256 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8374137540 0x7f837413d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:12.513 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.493+0000 7f8372ffd640 1 -- 172.21.15.42:0/4250445256 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8374137540 msgr2=0x7f837413d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:12.513 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.493+0000 7f8372ffd640 1 --2- 172.21.15.42:0/4250445256 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8374137540 0x7f837413d640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:12.513 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.493+0000 7f8372ffd640 1 -- 172.21.15.42:0/4250445256 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f83740a3d90 msgr2=0x7f837413d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:12.513 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.493+0000 7f8372ffd640 1 --2- 172.21.15.42:0/4250445256 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f83740a3d90 0x7f837413d100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:12.513 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.493+0000 7f8372ffd640 10 monclient: _finish_hunting 0 2024-09-16T04:07:12.513 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.493+0000 7f8372ffd640 1 monclient: found mon.noname-c 2024-09-16T04:07:12.513 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.493+0000 7f8372ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:07:12.513 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.493+0000 7f8372ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T04:07:12.513 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.493+0000 7f8372ffd640 1 -- 172.21.15.42:0/4250445256 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8374137070 con 0x7f83740a39b0 2024-09-16T04:07:12.513 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.493+0000 7f83737fe640 10 monclient: get_auth_request con 0x7f8374137540 auth_method 0 2024-09-16T04:07:12.514 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.493+0000 7f83737fe640 1 --2- 172.21.15.42:0/4250445256 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8374137540 0x7f837413d640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T04:07:12.514 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.493+0000 7f8372ffd640 1 --2- 172.21.15.42:0/4250445256 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f83740a39b0 0x7f83741308b0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f83640af040 tx=0x7f8364019690 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:07:12.514 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.493+0000 7f835bfff640 10 client.?.objecter ms_handle_connect 0x7f83740a39b0 2024-09-16T04:07:12.514 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.493+0000 7f835bfff640 10 client.?.objecter resend_mon_ops 2024-09-16T04:07:12.514 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.493+0000 7f835bfff640 1 -- 172.21.15.42:0/4250445256 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83640bc020 con 0x7f83740a39b0 2024-09-16T04:07:12.514 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.493+0000 7f835bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:12.514 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.493+0000 7f835bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:07:12.514 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.493+0000 7f835bfff640 10 monclient: dump: 2024-09-16T04:07:12.514 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:07:12.514 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:12.514 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:12.514 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:12.514 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:12.514 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:12.515 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:07:12.515 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:07:12.515 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:07:12.515 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:12.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.493+0000 7f835bfff640 10 monclient: _finish_auth 0 2024-09-16T04:07:12.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.493+0000 7f835bfff640 10 monclient: _check_auth_tickets 2024-09-16T04:07:12.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.493+0000 7f835bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:07:12.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.493+0000 7f835bfff640 1 -- 172.21.15.42:0/4250445256 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f83640b86b0 con 0x7f83740a39b0 2024-09-16T04:07:12.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.493+0000 7f835bfff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:07:12.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.493+0000 7f835bfff640 1 -- 172.21.15.42:0/4250445256 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8364009510 con 0x7f83740a39b0 2024-09-16T04:07:12.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.493+0000 7f8379962640 5 monclient: authenticate success, global_id 1014140 2024-09-16T04:07:12.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.493+0000 7f8379962640 10 monclient: _renew_subs 2024-09-16T04:07:12.516 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.493+0000 7f8379962640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:07:12.516 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.493+0000 7f8379962640 1 -- 172.21.15.42:0/4250445256 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f837406f680 con 0x7f83740a39b0 2024-09-16T04:07:12.516 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.493+0000 7f835bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:12.516 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.493+0000 7f835bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:07:12.516 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.493+0000 7f835bfff640 10 monclient: dump: 2024-09-16T04:07:12.516 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:07:12.516 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:12.516 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:12.516 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:12.516 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:12.516 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:12.516 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:07:12.516 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:07:12.516 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:07:12.517 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:12.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.493+0000 7f83717fa640 4 set_mon_vals no callback set 2024-09-16T04:07:12.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.493+0000 7f8379962640 10 client.1014140.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:07:12.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.493+0000 7f8379962640 10 monclient: _renew_subs 2024-09-16T04:07:12.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.493+0000 7f8379962640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:07:12.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.493+0000 7f8379962640 1 -- 172.21.15.42:0/4250445256 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f837409e4f0 con 0x7f83740a39b0 2024-09-16T04:07:12.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.493+0000 7f8379962640 1 librados: init done 2024-09-16T04:07:12.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.493+0000 7f8379962640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:07:12.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.493+0000 7f8379962640 1 --2- 172.21.15.42:0/4250445256 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8374130df0 0x7f837413db80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:12.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.493+0000 7f8379962640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:12.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.493+0000 7f8379962640 1 -- 172.21.15.42:0/4250445256 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f83741342d0 con 0x7f8374130df0 2024-09-16T04:07:12.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.493+0000 7f835bfff640 1 -- 172.21.15.42:0/4250445256 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f83640bf020 con 0x7f83740a39b0 2024-09-16T04:07:12.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.493+0000 7f835bfff640 1 --2- 172.21.15.42:0/4250445256 >> v2:172.21.15.154:6812/254845342 conn(0x7f8360037810 0x7f8360039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:12.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.493+0000 7f835bfff640 1 -- 172.21.15.42:0/4250445256 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(706..706 src has 451..706) ==== 5040+0+0 (secure 0 0 0) 0x7f83640e39b0 con 0x7f83740a39b0 2024-09-16T04:07:12.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.493+0000 7f83727fc640 1 --2- 172.21.15.42:0/4250445256 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8374130df0 0x7f837413db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:12.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.493+0000 7f835bfff640 10 client.1014140.objecter ms_dispatch 0x7f8374000ba0 osd_map(706..706 src has 451..706) 2024-09-16T04:07:12.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.493+0000 7f835bfff640 3 client.1014140.objecter handle_osd_map got epochs [706,706] > 0 2024-09-16T04:07:12.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.493+0000 7f835bfff640 3 client.1014140.objecter handle_osd_map decoding full epoch 706 2024-09-16T04:07:12.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.493+0000 7f83727fc640 10 monclient: get_auth_request con 0x7f8374130df0 auth_method 0 2024-09-16T04:07:12.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.493+0000 7f83727fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:12.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.493+0000 7f83727fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:12.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.493+0000 7f83727fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:12.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.493+0000 7f835bfff640 20 client.1014140.objecter dump_active .. 0 homeless 2024-09-16T04:07:12.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.493+0000 7f83727fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:07:12.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.493+0000 7f83727fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:07:12.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.493+0000 7f83727fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:07:12.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.497+0000 7f83737fe640 1 --2- 172.21.15.42:0/4250445256 >> v2:172.21.15.154:6812/254845342 conn(0x7f8360037810 0x7f8360039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:12.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.497+0000 7f83737fe640 10 monclient: get_auth_request con 0x7f8360037810 auth_method 0 2024-09-16T04:07:12.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.497+0000 7f83727fc640 10 monclient(hunting): handle_auth_done global_id 905056 payload 824 2024-09-16T04:07:12.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.497+0000 7f83727fc640 1 -- 172.21.15.42:0/4250445256 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f837409d040 con 0x7f8374130df0 2024-09-16T04:07:12.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.497+0000 7f835bfff640 10 client.1014140.objecter ms_handle_connect 0x7f8374130df0 2024-09-16T04:07:12.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.497+0000 7f835bfff640 10 client.1014140.objecter resend_mon_ops 2024-09-16T04:07:12.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.497+0000 7f83727fc640 1 --2- 172.21.15.42:0/4250445256 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8374130df0 0x7f837413db80 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f837413eb40 tx=0x7f8368011e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:07:12.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.497+0000 7f83737fe640 1 --2- 172.21.15.42:0/4250445256 >> v2:172.21.15.154:6812/254845342 conn(0x7f8360037810 0x7f8360039cd0 secure :-1 s=READY pgs=3878 cs=0 l=1 rev1=1 crypto rx=0x7f836c000f70 tx=0x7f836c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:07:12.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.497+0000 7f835bfff640 10 client.1014140.objecter ms_handle_connect 0x7f8360037810 2024-09-16T04:07:12.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.497+0000 7f835bfff640 1 -- 172.21.15.42:0/4250445256 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8368019070 con 0x7f8374130df0 2024-09-16T04:07:12.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.497+0000 7f835bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:12.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.497+0000 7f835bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:07:12.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.497+0000 7f835bfff640 10 monclient: dump: 2024-09-16T04:07:12.519 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:07:12.519 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:12.520 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:12.520 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:12.520 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:12.520 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:12.520 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:07:12.520 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:07:12.520 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:07:12.520 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:12.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.497+0000 7f835bfff640 1 -- 172.21.15.42:0/4250445256 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f8368004240 con 0x7f8374130df0 2024-09-16T04:07:12.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.497+0000 7f835bfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:07:12.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.497+0000 7f835bfff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:07:12.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.497+0000 7f835bfff640 1 -- 172.21.15.42:0/4250445256 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8374130df0 msgr2=0x7f837413db80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:12.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.497+0000 7f835bfff640 1 --2- 172.21.15.42:0/4250445256 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8374130df0 0x7f837413db80 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f837413eb40 tx=0x7f8368011e30 comp rx=0 tx=0).stop 2024-09-16T04:07:12.521 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.497+0000 7f835bfff640 1 -- 172.21.15.42:0/4250445256 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8374130df0 msgr2=0x7f837413db80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:07:12.521 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.497+0000 7f835bfff640 1 --2- 172.21.15.42:0/4250445256 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8374130df0 0x7f837413db80 secure :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f837413eb40 tx=0x7f8368011e30 comp rx=0 tx=0).stop 2024-09-16T04:07:12.522 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.501+0000 7f8379962640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:07:12.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.501+0000 7f8379962640 1 --2- 172.21.15.42:0/4250445256 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f837413e0c0 0x7f8374139140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:12.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.501+0000 7f8379962640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:12.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.501+0000 7f8379962640 1 -- 172.21.15.42:0/4250445256 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f837406df10 con 0x7f837413e0c0 2024-09-16T04:07:12.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.505+0000 7f83727fc640 1 --2- 172.21.15.42:0/4250445256 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f837413e0c0 0x7f8374139140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:12.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.505+0000 7f83727fc640 10 monclient: get_auth_request con 0x7f837413e0c0 auth_method 0 2024-09-16T04:07:12.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.505+0000 7f83727fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:12.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.505+0000 7f83727fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:12.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.505+0000 7f83727fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:12.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.505+0000 7f83727fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:07:12.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.505+0000 7f83727fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:07:12.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.505+0000 7f83727fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:07:12.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.505+0000 7f83727fc640 10 monclient(hunting): handle_auth_done global_id 905059 payload 824 2024-09-16T04:07:12.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.505+0000 7f83727fc640 1 -- 172.21.15.42:0/4250445256 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f837413e4d0 con 0x7f837413e0c0 2024-09-16T04:07:12.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.505+0000 7f835bfff640 10 client.1014140.objecter ms_handle_connect 0x7f837413e0c0 2024-09-16T04:07:12.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.505+0000 7f835bfff640 10 client.1014140.objecter resend_mon_ops 2024-09-16T04:07:12.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.505+0000 7f83727fc640 1 --2- 172.21.15.42:0/4250445256 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f837413e0c0 0x7f8374139140 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f836801b000 tx=0x7f83680046d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:07:12.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.505+0000 7f835bfff640 1 -- 172.21.15.42:0/4250445256 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8368019040 con 0x7f837413e0c0 2024-09-16T04:07:12.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.505+0000 7f835bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:12.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.505+0000 7f835bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:07:12.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.505+0000 7f835bfff640 10 monclient: dump: 2024-09-16T04:07:12.524 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:07:12.524 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:12.524 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:12.524 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:12.525 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:12.525 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:12.525 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:07:12.525 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:07:12.525 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:07:12.525 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:12.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.505+0000 7f835bfff640 1 -- 172.21.15.42:0/4250445256 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4880 (secure 0 0 0) 0x7f8368003c50 con 0x7f837413e0c0 2024-09-16T04:07:12.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.505+0000 7f835bfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:07:12.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.505+0000 7f835bfff640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:07:12.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.505+0000 7f835bfff640 1 -- 172.21.15.42:0/4250445256 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f837413e0c0 msgr2=0x7f8374139140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:12.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.505+0000 7f835bfff640 1 --2- 172.21.15.42:0/4250445256 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f837413e0c0 0x7f8374139140 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f836801b000 tx=0x7f83680046d0 comp rx=0 tx=0).stop 2024-09-16T04:07:12.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.505+0000 7f835bfff640 1 -- 172.21.15.42:0/4250445256 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f837413e0c0 msgr2=0x7f8374139140 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:07:12.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.505+0000 7f835bfff640 1 --2- 172.21.15.42:0/4250445256 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f837413e0c0 0x7f8374139140 secure :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f836801b000 tx=0x7f83680046d0 comp rx=0 tx=0).stop 2024-09-16T04:07:12.526 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T04:07:12.526 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:07:12.526 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:07:12.526 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T04:07:12.526 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 232, 2024-09-16T04:07:12.526 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 560682, 2024-09-16T04:07:12.526 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T04:07:12.526 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T04:07:12.526 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T04:07:12.526 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T04:07:12.527 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:07:12.527 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 7, 2024-09-16T04:07:12.527 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:07:12.527 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T04:07:12.527 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T04:07:12.527 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:07:12.527 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:07:12.527 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:07:12.527 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:07:12.527 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:07:12.527 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:07:12.527 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:07:12.528 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:07:12.528 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:07:12.528 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:07:12.528 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:07:12.528 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:07:12.528 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:07:12.528 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T04:07:12.528 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:07:12.528 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:07:12.528 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:07:12.529 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:07:12.529 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:07:12.529 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:07:12.529 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:07:12.529 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:07:12.529 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:07:12.529 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:07:12.529 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:07:12.529 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:07:12.529 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:12.529 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T04:07:12.530 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T04:07:12.530 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T04:07:12.530 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T04:07:12.530 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T04:07:12.530 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T04:07:12.530 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:07:12.530 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:07:12.530 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T04:07:12.530 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T04:07:12.530 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T04:07:12.530 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T04:07:12.530 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T04:07:12.531 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T04:07:12.531 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T04:07:12.531 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:07:12.531 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T04:07:12.531 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:07:12.531 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:07:12.531 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:07:12.531 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:07:12.531 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:07:12.531 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:07:12.531 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:07:12.531 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:07:12.531 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:07:12.532 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:07:12.532 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:07:12.532 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:07:12.532 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T04:07:12.532 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:12.532 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T04:07:12.532 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:12.532 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:07:12.532 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:07:12.532 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:07:12.532 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:07:12.532 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:12.532 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:07:12.533 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T04:07:12.533 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:07:12.533 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:12.533 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:12.533 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:07:12.533 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T04:07:12.533 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:07:12.533 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:07:12.533 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:07:12.533 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:12.533 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T04:07:12.533 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T04:07:12.533 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:07:12.534 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:07:12.534 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:07:12.534 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:12.534 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:12.534 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:07:12.534 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:07:12.534 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:07:12.534 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:07:12.534 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:12.534 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:07:12.534 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T04:07:12.534 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:07:12.534 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:12.534 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:12.535 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:07:12.535 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T04:07:12.535 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:07:12.535 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:07:12.535 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:07:12.535 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:12.535 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T04:07:12.535 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T04:07:12.535 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:07:12.535 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:07:12.535 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:07:12.535 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:12.535 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:12.535 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:07:12.536 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:07:12.536 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:07:12.536 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:07:12.536 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:12.536 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:07:12.536 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T04:07:12.536 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:07:12.536 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:12.536 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:12.536 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:07:12.536 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T04:07:12.536 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:07:12.537 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:07:12.537 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:07:12.537 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:12.537 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T04:07:12.537 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T04:07:12.537 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:07:12.537 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:07:12.537 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:07:12.537 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:07:12.537 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:07:12.537 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:12.537 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T04:07:12.538 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T04:07:12.538 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:12.538 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:07:12.538 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:07:12.538 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:07:12.538 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:07:12.538 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:07:12.538 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T04:07:12.538 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:12.538 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:07:12.539 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:07:12.539 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 3 2024-09-16T04:07:12.539 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:07:12.539 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:07:12.539 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T04:07:12.539 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:12.539 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:07:12.539 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:07:12.539 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 5 2024-09-16T04:07:12.539 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:07:12.539 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:07:12.539 INFO:teuthology.orchestra.run.smithi042.stdout: "mgr": [ 2024-09-16T04:07:12.540 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:12.540 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:07:12.540 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:07:12.540 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:07:12.540 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:07:12.540 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:07:12.540 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:12.540 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T04:07:12.540 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T04:07:12.542 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.505+0000 7f8379962640 10 librados: watch_flush enter 2024-09-16T04:07:12.542 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.505+0000 7f8379962640 10 librados: watch_flush exit 2024-09-16T04:07:12.542 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.505+0000 7f8379962640 20 client.1014140.objecter shutdown clearing up homeless session... 2024-09-16T04:07:12.542 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.505+0000 7f8379962640 10 client.1014140.objecter successfully canceled tick 2024-09-16T04:07:12.542 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.505+0000 7f8379962640 1 -- 172.21.15.42:0/4250445256 >> v2:172.21.15.154:6812/254845342 conn(0x7f8360037810 msgr2=0x7f8360039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:12.542 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.505+0000 7f8379962640 1 --2- 172.21.15.42:0/4250445256 >> v2:172.21.15.154:6812/254845342 conn(0x7f8360037810 0x7f8360039cd0 secure :-1 s=READY pgs=3878 cs=0 l=1 rev1=1 crypto rx=0x7f836c000f70 tx=0x7f836c009040 comp rx=0 tx=0).stop 2024-09-16T04:07:12.543 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.505+0000 7f8379962640 10 monclient: shutdown 2024-09-16T04:07:12.543 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.505+0000 7f8379962640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:07:12.543 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.505+0000 7f8379962640 1 -- 172.21.15.42:0/4250445256 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f83740a39b0 msgr2=0x7f83741308b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:12.543 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.505+0000 7f8379962640 1 --2- 172.21.15.42:0/4250445256 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f83740a39b0 0x7f83741308b0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f83640af040 tx=0x7f8364019690 comp rx=0 tx=0).stop 2024-09-16T04:07:12.543 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.505+0000 7f83737fe640 1 -- 172.21.15.42:0/4250445256 reap_dead start 2024-09-16T04:07:12.543 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.505+0000 7f83737fe640 1 -- 172.21.15.42:0/4250445256 reap_dead start 2024-09-16T04:07:12.543 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.505+0000 7f8379962640 1 -- 172.21.15.42:0/4250445256 shutdown_connections 2024-09-16T04:07:12.543 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.505+0000 7f8379962640 1 -- 172.21.15.42:0/4250445256 >> 172.21.15.42:0/4250445256 conn(0x7f83740a9fb0 msgr2=0x7f83740a8970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:07:12.543 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.505+0000 7f8379962640 1 -- 172.21.15.42:0/4250445256 shutdown_connections 2024-09-16T04:07:12.543 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.505+0000 7f8379962640 1 -- 172.21.15.42:0/4250445256 wait complete. 2024-09-16T04:07:12.543 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.505+0000 7f8379962640 1 librados: shutdown 2024-09-16T04:07:12.544 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:07:12.544 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-09-16T04:07:12.544 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-16T04:07:12.544 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-09-16T04:07:12.544 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-16T04:07:12.544 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-16T04:07:12.544 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T04:07:12.565 INFO:tasks.ceph.mon.a:Stopped 2024-09-16T04:07:12.565 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T04:07:12.565 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:07:12.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.725+0000 7fc727f67640 10 monclient: get_monmap_and_config 2024-09-16T04:07:12.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.725+0000 7fc727f67640 10 monclient: build_initial_monmap 2024-09-16T04:07:12.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.725+0000 7fc727f67640 10 monclient: monmap: 2024-09-16T04:07:12.731 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:07:12.731 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:12.731 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:07:12.729206+0000 2024-09-16T04:07:12.731 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:07:12.729206+0000 2024-09-16T04:07:12.731 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:07:12.731 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:12.731 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:07:12.731 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:07:12.731 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:07:12.731 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:07:12.731 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:07:12.731 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:12.733 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.725+0000 7fc727f67640 1 Processor -- start 2024-09-16T04:07:12.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.725+0000 7fc727f67640 1 -- start start 2024-09-16T04:07:12.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.725+0000 7fc727f67640 10 monclient: init 2024-09-16T04:07:12.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.725+0000 7fc727f67640 10 monclient: _reopen_session rank -1 2024-09-16T04:07:12.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.725+0000 7fc727f67640 10 monclient: _add_conns ranks=[3,1,2,4,0] 2024-09-16T04:07:12.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.725+0000 7fc727f67640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fc72006db90 0x7fc7201340c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:12.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.725+0000 7fc727f67640 10 monclient(hunting): picked mon.noname-a con 0x7fc72006db90 addr v2:172.21.15.42:3300/0 2024-09-16T04:07:12.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.725+0000 7fc727f67640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc7200a37f0 addr v1:172.21.15.154:6789/0 2024-09-16T04:07:12.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.725+0000 7fc727f67640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc7200a3c00 0x7fc720137520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:12.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.725+0000 7fc727f67640 10 monclient(hunting): picked mon.noname-b con 0x7fc7200a3c00 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:07:12.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.725+0000 7fc727f67640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc72006ece0 con 0x7fc7200a37f0 2024-09-16T04:07:12.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.725+0000 7fc727f67640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:12.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.725+0000 7fc727f67640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7fc72006df70 con 0x7fc72006db90 2024-09-16T04:07:12.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.725+0000 7fc727f67640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:12.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.725+0000 7fc727f67640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fc72009bf80 con 0x7fc7200a3c00 2024-09-16T04:07:12.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.725+0000 7fc727f67640 10 monclient(hunting): _renew_subs 2024-09-16T04:07:12.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.725+0000 7fc727f67640 10 monclient(hunting): authenticate will time out at 2479.698641s 2024-09-16T04:07:12.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.725+0000 7fc725d03640 1 -- >> v2:172.21.15.42:3300/0 conn(0x7fc72006db90 msgr2=0x7fc7201340c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3300/0 2024-09-16T04:07:12.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.725+0000 7fc725d03640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fc72006db90 0x7fc7201340c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:07:12.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.725+0000 7fc726504640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc7200a3c00 0x7fc720137520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:12.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.725+0000 7fc726504640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc7200a3c00 0x7fc720137520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:57070/0 (socket says 172.21.15.42:57070) 2024-09-16T04:07:12.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.725+0000 7fc726504640 1 -- 172.21.15.42:0/2108632737 learned_addr learned my addr 172.21.15.42:0/2108632737 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:07:12.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.725+0000 7fc726504640 10 monclient(hunting): get_auth_request con 0x7fc7200a3c00 auth_method 0 2024-09-16T04:07:12.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.725+0000 7fc726504640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:12.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.725+0000 7fc726504640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:12.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.725+0000 7fc726504640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:12.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.725+0000 7fc726504640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:07:12.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.725+0000 7fc726504640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:07:12.736 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.725+0000 7fc726504640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:07:12.736 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.725+0000 7fc726504640 10 monclient(hunting): handle_auth_done global_id 1014143 payload 274 2024-09-16T04:07:12.736 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.725+0000 7fc726504640 1 -- 172.21.15.42:0/2108632737 >> v2:172.21.15.42:3300/0 conn(0x7fc72006db90 msgr2=0x7fc7201340c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:07:12.736 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.725+0000 7fc726504640 1 --2- 172.21.15.42:0/2108632737 >> v2:172.21.15.42:3300/0 conn(0x7fc72006db90 0x7fc7201340c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:12.736 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.725+0000 7fc726504640 1 -- 172.21.15.42:0/2108632737 >> v1:172.21.15.154:6789/0 conn(0x7fc7200a37f0 legacy=0x7fc720136e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:07:12.736 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.725+0000 7fc726504640 10 monclient: _finish_hunting 0 2024-09-16T04:07:12.736 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.725+0000 7fc726504640 1 monclient: found mon.noname-b 2024-09-16T04:07:12.736 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.725+0000 7fc726504640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:07:12.736 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.725+0000 7fc726504640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T04:07:12.736 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.725+0000 7fc726504640 1 -- 172.21.15.42:0/2108632737 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc720134780 con 0x7fc7200a3c00 2024-09-16T04:07:12.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.729+0000 7fc726504640 1 --2- 172.21.15.42:0/2108632737 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc7200a3c00 0x7fc720137520 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fc718009e50 tx=0x7fc71801c8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=33b33a08288196a4 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:07:12.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.729+0000 7fc724d01640 1 -- 172.21.15.42:0/2108632737 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7180a5660 con 0x7fc7200a3c00 2024-09-16T04:07:12.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.729+0000 7fc724d01640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:12.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.729+0000 7fc724d01640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:07:12.742 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.729+0000 7fc724d01640 10 monclient: dump: 2024-09-16T04:07:12.742 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:07:12.742 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:12.742 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:12.742 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:12.742 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:12.742 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:12.742 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:07:12.742 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:07:12.742 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:07:12.742 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:12.742 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.729+0000 7fc724d01640 10 monclient: _finish_auth 0 2024-09-16T04:07:12.742 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.729+0000 7fc724d01640 10 monclient: _check_auth_tickets 2024-09-16T04:07:12.742 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.729+0000 7fc724d01640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:07:12.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.729+0000 7fc724d01640 1 -- 172.21.15.42:0/2108632737 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc7180a5800 con 0x7fc7200a3c00 2024-09-16T04:07:12.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.729+0000 7fc727f67640 5 monclient: authenticate success, global_id 1014143 2024-09-16T04:07:12.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.729+0000 7fc727f67640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:07:12.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.729+0000 7fc724d01640 10 monclient: handle_config config(0 keys) 2024-09-16T04:07:12.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.729+0000 7fc724d01640 1 -- 172.21.15.42:0/2108632737 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7180a5bd0 con 0x7fc7200a3c00 2024-09-16T04:07:12.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.729+0000 7fc727f67640 10 monclient: get_monmap_and_config success 2024-09-16T04:07:12.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.729+0000 7fc727f67640 4 set_mon_vals no callback set 2024-09-16T04:07:12.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.729+0000 7fc724d01640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:12.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.729+0000 7fc724d01640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:07:12.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.729+0000 7fc724d01640 10 monclient: dump: 2024-09-16T04:07:12.743 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:07:12.743 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:12.743 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:12.744 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:12.744 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:12.744 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:12.744 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:07:12.744 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:07:12.744 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:07:12.744 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:12.744 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.729+0000 7fc727f67640 10 monclient: shutdown 2024-09-16T04:07:12.744 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.729+0000 7fc727f67640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:07:12.744 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.729+0000 7fc727f67640 1 -- 172.21.15.42:0/2108632737 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc7200a3c00 msgr2=0x7fc720137520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:12.744 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.729+0000 7fc727f67640 1 --2- 172.21.15.42:0/2108632737 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc7200a3c00 0x7fc720137520 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fc718009e50 tx=0x7fc71801c8d0 comp rx=0 tx=0).stop 2024-09-16T04:07:12.744 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.729+0000 7fc727f67640 1 -- 172.21.15.42:0/2108632737 shutdown_connections 2024-09-16T04:07:12.744 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.729+0000 7fc727f67640 1 --2- 172.21.15.42:0/2108632737 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc7200a3c00 0x7fc720137520 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:12.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.729+0000 7fc727f67640 1 --2- 172.21.15.42:0/2108632737 >> v2:172.21.15.42:3300/0 conn(0x7fc72006db90 0x7fc7201340c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:12.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.729+0000 7fc727f67640 1 -- 172.21.15.42:0/2108632737 >> 172.21.15.42:0/2108632737 conn(0x7fc7200a9fb0 msgr2=0x7fc7200aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:07:12.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.729+0000 7fc727f67640 1 -- 172.21.15.42:0/2108632737 shutdown_connections 2024-09-16T04:07:12.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.729+0000 7fc727f67640 1 -- 172.21.15.42:0/2108632737 wait complete. 2024-09-16T04:07:12.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.729+0000 7fc727f67640 10 monclient: build_initial_monmap 2024-09-16T04:07:12.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.729+0000 7fc727f67640 10 monclient: monmap: 2024-09-16T04:07:12.745 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:07:12.745 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:07:12.745 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:07:12.745 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:07:12.745 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:07:12.745 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:12.745 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:07:12.746 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:07:12.746 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:07:12.746 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:12.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.729+0000 7fc727f67640 1 librados: starting msgr at 2024-09-16T04:07:12.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.729+0000 7fc727f67640 1 librados: starting objecter 2024-09-16T04:07:12.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.729+0000 7fc727f67640 1 Processor -- start 2024-09-16T04:07:12.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.729+0000 7fc727f67640 1 -- start start 2024-09-16T04:07:12.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.729+0000 7fc727f67640 1 librados: setting wanted keys 2024-09-16T04:07:12.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.729+0000 7fc727f67640 1 librados: calling monclient init 2024-09-16T04:07:12.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.729+0000 7fc727f67640 10 monclient: init 2024-09-16T04:07:12.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.729+0000 7fc727f67640 10 monclient: _reopen_session rank -1 2024-09-16T04:07:12.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.729+0000 7fc727f67640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:07:12.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.729+0000 7fc727f67640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc72006db90 0x7fc72013ce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:12.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.729+0000 7fc727f67640 10 monclient(hunting): picked mon.noname-c con 0x7fc72006db90 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:07:12.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.729+0000 7fc727f67640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc7200a37f0 0x7fc72013d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:12.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.729+0000 7fc727f67640 10 monclient(hunting): picked mon.noname-b con 0x7fc7200a37f0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:07:12.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.729+0000 7fc727f67640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc7200a3c00 0x7fc72013d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:12.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.729+0000 7fc727f67640 10 monclient(hunting): picked mon.noname-a con 0x7fc7200a3c00 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:07:12.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.729+0000 7fc727f67640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:12.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.729+0000 7fc727f67640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fc7201419d0 con 0x7fc7200a3c00 2024-09-16T04:07:12.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.729+0000 7fc727f67640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:12.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.729+0000 7fc727f67640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc720137c70 con 0x7fc7200a37f0 2024-09-16T04:07:12.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.729+0000 7fc727f67640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:12.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.729+0000 7fc727f67640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fc720141b50 con 0x7fc72006db90 2024-09-16T04:07:12.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.729+0000 7fc727f67640 10 monclient(hunting): _renew_subs 2024-09-16T04:07:12.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.729+0000 7fc727f67640 10 monclient(hunting): authenticate will time out at 2479.703724s 2024-09-16T04:07:12.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.729+0000 7fc726504640 1 -- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc7200a3c00 msgr2=0x7fc72013d8a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.42:3300/0 2024-09-16T04:07:12.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.729+0000 7fc726504640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc7200a3c00 0x7fc72013d8a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:07:12.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.729+0000 7fc725d03640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc72006db90 0x7fc72013ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:12.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.729+0000 7fc725d03640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc72006db90 0x7fc72013ce20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:57074/0 (socket says 172.21.15.42:57074) 2024-09-16T04:07:12.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.729+0000 7fc725d03640 1 -- 172.21.15.42:0/2262170553 learned_addr learned my addr 172.21.15.42:0/2262170553 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:07:12.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.729+0000 7fc725d03640 10 monclient(hunting): get_auth_request con 0x7fc72006db90 auth_method 0 2024-09-16T04:07:12.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.729+0000 7fc725d03640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:12.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.729+0000 7fc725d03640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:12.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.729+0000 7fc725d03640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:12.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.729+0000 7fc725d03640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:07:12.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.729+0000 7fc725d03640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:07:12.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.729+0000 7fc725d03640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:07:12.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.733+0000 7fc725d03640 10 monclient(hunting): handle_auth_done global_id 1014146 payload 824 2024-09-16T04:07:12.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.733+0000 7fc725d03640 1 -- 172.21.15.42:0/2262170553 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc7200a37f0 msgr2=0x7fc72013d360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:12.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.733+0000 7fc725d03640 1 --2- 172.21.15.42:0/2262170553 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc7200a37f0 0x7fc72013d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:12.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.733+0000 7fc725d03640 1 -- 172.21.15.42:0/2262170553 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc7200a3c00 msgr2=0x7fc72013d8a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:07:12.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.733+0000 7fc725d03640 1 --2- 172.21.15.42:0/2262170553 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc7200a3c00 0x7fc72013d8a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:12.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.733+0000 7fc725d03640 10 monclient: _finish_hunting 0 2024-09-16T04:07:12.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.733+0000 7fc725d03640 1 monclient: found mon.noname-c 2024-09-16T04:07:12.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.733+0000 7fc725d03640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:07:12.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.733+0000 7fc725d03640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T04:07:12.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.733+0000 7fc725d03640 1 -- 172.21.15.42:0/2262170553 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc72006f6a0 con 0x7fc72006db90 2024-09-16T04:07:12.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.733+0000 7fc725d03640 1 --2- 172.21.15.42:0/2262170553 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc72006db90 0x7fc72013ce20 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fc71000b550 tx=0x7fc71000ba10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:07:12.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.733+0000 7fc7167fc640 10 client.?.objecter ms_handle_connect 0x7fc72006db90 2024-09-16T04:07:12.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.733+0000 7fc7167fc640 10 client.?.objecter resend_mon_ops 2024-09-16T04:07:12.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.733+0000 7fc7167fc640 1 -- 172.21.15.42:0/2262170553 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc710003bd0 con 0x7fc72006db90 2024-09-16T04:07:12.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.733+0000 7fc7167fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:12.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.733+0000 7fc7167fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:07:12.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.733+0000 7fc7167fc640 10 monclient: dump: 2024-09-16T04:07:12.750 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:07:12.750 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:12.750 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:12.750 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:12.750 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:12.750 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:12.750 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:07:12.750 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:07:12.750 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:07:12.750 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:12.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.733+0000 7fc7167fc640 10 monclient: _finish_auth 0 2024-09-16T04:07:12.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.733+0000 7fc7167fc640 10 monclient: _check_auth_tickets 2024-09-16T04:07:12.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.733+0000 7fc7167fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:07:12.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.733+0000 7fc7167fc640 1 -- 172.21.15.42:0/2262170553 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc710015070 con 0x7fc72006db90 2024-09-16T04:07:12.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.733+0000 7fc7167fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:07:12.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.733+0000 7fc7167fc640 1 -- 172.21.15.42:0/2262170553 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7100050f0 con 0x7fc72006db90 2024-09-16T04:07:12.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.733+0000 7fc727f67640 5 monclient: authenticate success, global_id 1014146 2024-09-16T04:07:12.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.733+0000 7fc727f67640 10 monclient: _renew_subs 2024-09-16T04:07:12.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.733+0000 7fc727f67640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:07:12.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.733+0000 7fc727f67640 1 -- 172.21.15.42:0/2262170553 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc72013ea70 con 0x7fc72006db90 2024-09-16T04:07:12.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.733+0000 7fc7167fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:12.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.733+0000 7fc7167fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:07:12.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.733+0000 7fc717fff640 4 set_mon_vals no callback set 2024-09-16T04:07:12.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.733+0000 7fc7167fc640 10 monclient: dump: 2024-09-16T04:07:12.751 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:07:12.751 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:12.752 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:12.752 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:12.752 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:12.752 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:12.752 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:07:12.752 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:07:12.752 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:07:12.752 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:12.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.733+0000 7fc727f67640 10 client.1014146.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:07:12.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.733+0000 7fc727f67640 10 monclient: _renew_subs 2024-09-16T04:07:12.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.733+0000 7fc727f67640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:07:12.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.733+0000 7fc727f67640 1 -- 172.21.15.42:0/2262170553 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc72009dc10 con 0x7fc72006db90 2024-09-16T04:07:12.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.733+0000 7fc727f67640 1 librados: init done 2024-09-16T04:07:12.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.733+0000 7fc727f67640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:07:12.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.733+0000 7fc727f67640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:07:12.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.733+0000 7fc727f67640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:07:12.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.733+0000 7fc7167fc640 1 -- 172.21.15.42:0/2262170553 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fc710005290 con 0x7fc72006db90 2024-09-16T04:07:12.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.733+0000 7fc727f67640 1 -- 172.21.15.42:0/2262170553 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc7201340c0 con 0x7fc72006db90 2024-09-16T04:07:12.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.733+0000 7fc7167fc640 1 --2- 172.21.15.42:0/2262170553 >> v2:172.21.15.154:6812/254845342 conn(0x7fc7040377c0 0x7fc704039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:12.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.733+0000 7fc7167fc640 1 -- 172.21.15.42:0/2262170553 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(706..706 src has 451..706) ==== 5040+0+0 (secure 0 0 0) 0x7fc710048850 con 0x7fc72006db90 2024-09-16T04:07:12.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.733+0000 7fc7167fc640 10 client.1014146.objecter ms_dispatch 0x7fc720000ba0 osd_map(706..706 src has 451..706) 2024-09-16T04:07:12.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.733+0000 7fc7167fc640 3 client.1014146.objecter handle_osd_map got epochs [706,706] > 0 2024-09-16T04:07:12.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.733+0000 7fc7167fc640 3 client.1014146.objecter handle_osd_map decoding full epoch 706 2024-09-16T04:07:12.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.733+0000 7fc7167fc640 20 client.1014146.objecter dump_active .. 0 homeless 2024-09-16T04:07:12.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.733+0000 7fc725502640 1 --2- 172.21.15.42:0/2262170553 >> v2:172.21.15.154:6812/254845342 conn(0x7fc7040377c0 0x7fc704039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:12.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.733+0000 7fc725502640 10 monclient: get_auth_request con 0x7fc7040377c0 auth_method 0 2024-09-16T04:07:12.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.733+0000 7fc725502640 1 --2- 172.21.15.42:0/2262170553 >> v2:172.21.15.154:6812/254845342 conn(0x7fc7040377c0 0x7fc704039c80 secure :-1 s=READY pgs=3879 cs=0 l=1 rev1=1 crypto rx=0x7fc720133a70 tx=0x7fc71c0079c0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:07:12.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.733+0000 7fc7167fc640 10 client.1014146.objecter ms_handle_connect 0x7fc7040377c0 2024-09-16T04:07:12.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.737+0000 7fc7167fc640 1 -- 172.21.15.42:0/2262170553 <== mon.2 v2:172.21.15.42:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7fc71000dde0 con 0x7fc72006db90 2024-09-16T04:07:12.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.737+0000 7fc7167fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:07:12.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.737+0000 7fc7167fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:07:12.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.961+0000 7fc727f67640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:07:12.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.961+0000 7fc727f67640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:07:12.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.961+0000 7fc727f67640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:07:12.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.961+0000 7fc727f67640 1 -- 172.21.15.42:0/2262170553 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fc72009cd80 con 0x7fc72006db90 2024-09-16T04:07:12.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.961+0000 7fc7167fc640 1 -- 172.21.15.42:0/2262170553 <== mon.2 v2:172.21.15.42:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7fc7100199e0 con 0x7fc72006db90 2024-09-16T04:07:12.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.961+0000 7fc7167fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:07:12.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.961+0000 7fc7167fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:07:12.967 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":232,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:07:12.969 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.961+0000 7fc727f67640 10 librados: watch_flush enter 2024-09-16T04:07:12.969 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.961+0000 7fc727f67640 10 librados: watch_flush exit 2024-09-16T04:07:12.969 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.961+0000 7fc727f67640 20 client.1014146.objecter shutdown clearing up homeless session... 2024-09-16T04:07:12.969 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.961+0000 7fc727f67640 10 client.1014146.objecter successfully canceled tick 2024-09-16T04:07:12.969 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.961+0000 7fc727f67640 1 -- 172.21.15.42:0/2262170553 >> v2:172.21.15.154:6812/254845342 conn(0x7fc7040377c0 msgr2=0x7fc704039c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:12.969 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.961+0000 7fc727f67640 1 --2- 172.21.15.42:0/2262170553 >> v2:172.21.15.154:6812/254845342 conn(0x7fc7040377c0 0x7fc704039c80 secure :-1 s=READY pgs=3879 cs=0 l=1 rev1=1 crypto rx=0x7fc720133a70 tx=0x7fc71c0079c0 comp rx=0 tx=0).stop 2024-09-16T04:07:12.969 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.961+0000 7fc727f67640 10 monclient: shutdown 2024-09-16T04:07:12.969 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.961+0000 7fc727f67640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:07:12.969 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.961+0000 7fc727f67640 1 -- 172.21.15.42:0/2262170553 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc72006db90 msgr2=0x7fc72013ce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:12.969 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.961+0000 7fc727f67640 1 --2- 172.21.15.42:0/2262170553 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc72006db90 0x7fc72013ce20 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fc71000b550 tx=0x7fc71000ba10 comp rx=0 tx=0).stop 2024-09-16T04:07:12.969 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.961+0000 7fc727f67640 1 -- 172.21.15.42:0/2262170553 shutdown_connections 2024-09-16T04:07:12.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.961+0000 7fc727f67640 1 --2- 172.21.15.42:0/2262170553 >> v2:172.21.15.154:6812/254845342 conn(0x7fc7040377c0 0x7fc704039c80 unknown :-1 s=CLOSED pgs=3879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:12.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.961+0000 7fc727f67640 1 --2- 172.21.15.42:0/2262170553 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc7200a3c00 0x7fc72013d8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:12.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.961+0000 7fc727f67640 1 --2- 172.21.15.42:0/2262170553 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc7200a37f0 0x7fc72013d360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:12.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.961+0000 7fc727f67640 1 --2- 172.21.15.42:0/2262170553 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc72006db90 0x7fc72013ce20 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:12.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.961+0000 7fc727f67640 1 -- 172.21.15.42:0/2262170553 >> 172.21.15.42:0/2262170553 conn(0x7fc7200a9fb0 msgr2=0x7fc720136af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:07:12.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.961+0000 7fc727f67640 1 -- 172.21.15.42:0/2262170553 shutdown_connections 2024-09-16T04:07:12.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.961+0000 7fc727f67640 1 -- 172.21.15.42:0/2262170553 wait complete. 2024-09-16T04:07:12.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:12.961+0000 7fc727f67640 1 librados: shutdown 2024-09-16T04:07:15.984 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:07:16.115 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.109+0000 7fe0a9707640 10 monclient: get_monmap_and_config 2024-09-16T04:07:16.116 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.109+0000 7fe0a9707640 10 monclient: build_initial_monmap 2024-09-16T04:07:16.116 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.109+0000 7fe0a9707640 10 monclient: monmap: 2024-09-16T04:07:16.116 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:07:16.116 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:16.116 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:07:16.114196+0000 2024-09-16T04:07:16.116 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:07:16.114196+0000 2024-09-16T04:07:16.116 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:07:16.116 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:16.116 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:07:16.116 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:07:16.116 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:07:16.116 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:07:16.116 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:07:16.117 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:16.118 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.109+0000 7fe0a9707640 1 Processor -- start 2024-09-16T04:07:16.119 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.109+0000 7fe0a9707640 1 -- start start 2024-09-16T04:07:16.119 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.109+0000 7fe0a9707640 10 monclient: init 2024-09-16T04:07:16.119 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.109+0000 7fe0a9707640 10 monclient: _reopen_session rank -1 2024-09-16T04:07:16.119 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.109+0000 7fe0a9707640 10 monclient: _add_conns ranks=[1,2,4,3,0] 2024-09-16T04:07:16.119 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.109+0000 7fe0a9707640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe0a409f7b0 addr v1:172.21.15.154:6789/0 2024-09-16T04:07:16.119 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.109+0000 7fe0a9707640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe0a409fb90 0x7fe0a413c0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:16.119 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.109+0000 7fe0a9707640 10 monclient(hunting): picked mon.noname-b con 0x7fe0a409fb90 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:07:16.119 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.109+0000 7fe0a9707640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fe0a413c630 0x7fe0a413ea20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:16.119 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.109+0000 7fe0a9707640 10 monclient(hunting): picked mon.noname-c con 0x7fe0a413c630 addr v2:172.21.15.154:3300/0 2024-09-16T04:07:16.119 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.109+0000 7fe0a9707640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe0a4134080 con 0x7fe0a409f7b0 2024-09-16T04:07:16.119 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.109+0000 7fe0a9707640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:16.120 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.109+0000 7fe0a9707640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fe0a409bca0 con 0x7fe0a413c630 2024-09-16T04:07:16.120 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.109+0000 7fe0a9707640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:16.120 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.109+0000 7fe0a9707640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fe0a409bf80 con 0x7fe0a409fb90 2024-09-16T04:07:16.120 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.109+0000 7fe0a9707640 10 monclient(hunting): _renew_subs 2024-09-16T04:07:16.120 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.109+0000 7fe0a9707640 10 monclient(hunting): authenticate will time out at 2483.083678s 2024-09-16T04:07:16.120 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.109+0000 7fe0a27fc640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe0a409fb90 0x7fe0a413c0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:16.120 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.109+0000 7fe0a27fc640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe0a409fb90 0x7fe0a413c0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:53300/0 (socket says 172.21.15.42:53300) 2024-09-16T04:07:16.120 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.109+0000 7fe0a27fc640 1 -- 172.21.15.42:0/1099258910 learned_addr learned my addr 172.21.15.42:0/1099258910 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:07:16.120 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.109+0000 7fe0a27fc640 10 monclient(hunting): get_auth_request con 0x7fe0a409fb90 auth_method 0 2024-09-16T04:07:16.120 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.109+0000 7fe0a27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:16.120 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.109+0000 7fe0a27fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:16.120 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.109+0000 7fe0a27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:16.120 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.113+0000 7fe0a27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:07:16.120 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.113+0000 7fe0a27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:07:16.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.113+0000 7fe0a27fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:07:16.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.113+0000 7fe0a37fe640 1 --2- 172.21.15.42:0/1099258910 >> v2:172.21.15.154:3300/0 conn(0x7fe0a413c630 0x7fe0a413ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:16.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.113+0000 7fe0a37fe640 10 monclient(hunting): get_auth_request con 0x7fe0a413c630 auth_method 0 2024-09-16T04:07:16.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.113+0000 7fe0a37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:16.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.113+0000 7fe0a37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:16.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.113+0000 7fe0a37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:16.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.113+0000 7fe0a27fc640 10 monclient(hunting): handle_auth_done global_id 1014149 payload 274 2024-09-16T04:07:16.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.113+0000 7fe0a27fc640 1 -- 172.21.15.42:0/1099258910 >> v2:172.21.15.154:3300/0 conn(0x7fe0a413c630 msgr2=0x7fe0a413ea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:07:16.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.113+0000 7fe0a27fc640 1 --2- 172.21.15.42:0/1099258910 >> v2:172.21.15.154:3300/0 conn(0x7fe0a413c630 0x7fe0a413ea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:16.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.113+0000 7fe0a27fc640 1 -- 172.21.15.42:0/1099258910 >> v1:172.21.15.154:6789/0 conn(0x7fe0a409f7b0 legacy=0x7fe0a4133970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:07:16.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.113+0000 7fe0a27fc640 10 monclient: _finish_hunting 0 2024-09-16T04:07:16.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.113+0000 7fe0a27fc640 1 monclient: found mon.noname-b 2024-09-16T04:07:16.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.113+0000 7fe0a27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:07:16.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.113+0000 7fe0a27fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T04:07:16.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.113+0000 7fe0a27fc640 1 -- 172.21.15.42:0/1099258910 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0a409c260 con 0x7fe0a409fb90 2024-09-16T04:07:16.128 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.113+0000 7fe0a27fc640 1 --2- 172.21.15.42:0/1099258910 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe0a409fb90 0x7fe0a413c0f0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fe094009e50 tx=0x7fe09401c940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=210d0b4b3a8d49e1 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:07:16.128 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.113+0000 7fe0a1ffb640 1 -- 172.21.15.42:0/1099258910 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0940a56a0 con 0x7fe0a409fb90 2024-09-16T04:07:16.128 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.113+0000 7fe0a1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:16.128 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.113+0000 7fe0a1ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:07:16.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.113+0000 7fe0a1ffb640 10 monclient: dump: 2024-09-16T04:07:16.129 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:07:16.129 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:16.129 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:16.129 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:16.129 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:16.129 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:16.129 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:07:16.129 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:07:16.129 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:07:16.129 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:16.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.113+0000 7fe0a1ffb640 10 monclient: _finish_auth 0 2024-09-16T04:07:16.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.113+0000 7fe0a1ffb640 10 monclient: _check_auth_tickets 2024-09-16T04:07:16.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.113+0000 7fe0a1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:07:16.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.113+0000 7fe0a1ffb640 1 -- 172.21.15.42:0/1099258910 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe0940a5840 con 0x7fe0a409fb90 2024-09-16T04:07:16.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.113+0000 7fe0a9707640 5 monclient: authenticate success, global_id 1014149 2024-09-16T04:07:16.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.113+0000 7fe0a9707640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:07:16.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.113+0000 7fe0a1ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:07:16.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.113+0000 7fe0a1ffb640 1 -- 172.21.15.42:0/1099258910 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0940a5c10 con 0x7fe0a409fb90 2024-09-16T04:07:16.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.113+0000 7fe0a9707640 10 monclient: get_monmap_and_config success 2024-09-16T04:07:16.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.113+0000 7fe0a9707640 4 set_mon_vals no callback set 2024-09-16T04:07:16.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.113+0000 7fe0a1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:16.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.113+0000 7fe0a1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:07:16.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.113+0000 7fe0a1ffb640 10 monclient: dump: 2024-09-16T04:07:16.130 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:07:16.130 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:16.131 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:16.131 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:16.131 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:16.131 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:16.131 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:07:16.131 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:07:16.131 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:07:16.131 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:16.131 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.113+0000 7fe0a9707640 10 monclient: shutdown 2024-09-16T04:07:16.131 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.113+0000 7fe0a9707640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:07:16.131 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.113+0000 7fe0a9707640 1 -- 172.21.15.42:0/1099258910 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe0a409fb90 msgr2=0x7fe0a413c0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:16.131 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.113+0000 7fe0a9707640 1 --2- 172.21.15.42:0/1099258910 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe0a409fb90 0x7fe0a413c0f0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fe094009e50 tx=0x7fe09401c940 comp rx=0 tx=0).stop 2024-09-16T04:07:16.132 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.113+0000 7fe0a9707640 1 -- 172.21.15.42:0/1099258910 shutdown_connections 2024-09-16T04:07:16.132 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.113+0000 7fe0a9707640 1 --2- 172.21.15.42:0/1099258910 >> v2:172.21.15.154:3300/0 conn(0x7fe0a413c630 0x7fe0a413ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:16.132 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.113+0000 7fe0a9707640 1 --2- 172.21.15.42:0/1099258910 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe0a409fb90 0x7fe0a413c0f0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:16.132 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.113+0000 7fe0a9707640 1 -- 172.21.15.42:0/1099258910 >> 172.21.15.42:0/1099258910 conn(0x7fe0a40a2c70 msgr2=0x7fe0a40756e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:07:16.132 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.113+0000 7fe0a9707640 1 -- 172.21.15.42:0/1099258910 shutdown_connections 2024-09-16T04:07:16.132 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.113+0000 7fe0a9707640 1 -- 172.21.15.42:0/1099258910 wait complete. 2024-09-16T04:07:16.132 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.117+0000 7fe0a9707640 10 monclient: build_initial_monmap 2024-09-16T04:07:16.132 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.117+0000 7fe0a9707640 10 monclient: monmap: 2024-09-16T04:07:16.132 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:07:16.132 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:07:16.132 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:07:16.132 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:07:16.133 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:07:16.133 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:16.133 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:07:16.133 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:07:16.133 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:07:16.133 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:16.133 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.117+0000 7fe0a9707640 1 librados: starting msgr at 2024-09-16T04:07:16.133 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.117+0000 7fe0a9707640 1 librados: starting objecter 2024-09-16T04:07:16.133 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.117+0000 7fe0a9707640 1 Processor -- start 2024-09-16T04:07:16.133 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.117+0000 7fe0a9707640 1 -- start start 2024-09-16T04:07:16.133 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.117+0000 7fe0a9707640 1 librados: setting wanted keys 2024-09-16T04:07:16.133 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.117+0000 7fe0a9707640 1 librados: calling monclient init 2024-09-16T04:07:16.133 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.117+0000 7fe0a9707640 10 monclient: init 2024-09-16T04:07:16.133 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.117+0000 7fe0a9707640 10 monclient: _reopen_session rank -1 2024-09-16T04:07:16.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.117+0000 7fe0a9707640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:07:16.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.117+0000 7fe0a9707640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe0a409f7b0 0x7fe0a412d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:16.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.117+0000 7fe0a9707640 10 monclient(hunting): picked mon.noname-b con 0x7fe0a409f7b0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:07:16.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.117+0000 7fe0a9707640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe0a409fb90 0x7fe0a412dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:16.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.117+0000 7fe0a9707640 10 monclient(hunting): picked mon.noname-a con 0x7fe0a409fb90 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:07:16.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.117+0000 7fe0a9707640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe0a413c630 0x7fe0a41ceda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:16.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.117+0000 7fe0a9707640 10 monclient(hunting): picked mon.noname-c con 0x7fe0a413c630 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:07:16.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.117+0000 7fe0a9707640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:16.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.117+0000 7fe0a9707640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fe0a4141900 con 0x7fe0a409fb90 2024-09-16T04:07:16.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.117+0000 7fe0a9707640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:16.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.117+0000 7fe0a9707640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe0a4141780 con 0x7fe0a409f7b0 2024-09-16T04:07:16.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.117+0000 7fe0a9707640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:16.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.117+0000 7fe0a9707640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fe0a4141a80 con 0x7fe0a413c630 2024-09-16T04:07:16.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.117+0000 7fe0a9707640 10 monclient(hunting): _renew_subs 2024-09-16T04:07:16.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.117+0000 7fe0a9707640 10 monclient(hunting): authenticate will time out at 2483.088936s 2024-09-16T04:07:16.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.117+0000 7fe0a27fc640 1 -- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe0a409fb90 msgr2=0x7fe0a412dbd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.42:3300/0 2024-09-16T04:07:16.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.117+0000 7fe0a27fc640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe0a409fb90 0x7fe0a412dbd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:07:16.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.117+0000 7fe0a37fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe0a413c630 0x7fe0a41ceda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:16.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.117+0000 7fe0a37fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe0a413c630 0x7fe0a41ceda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:53308/0 (socket says 172.21.15.42:53308) 2024-09-16T04:07:16.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.117+0000 7fe0a37fe640 1 -- 172.21.15.42:0/189589361 learned_addr learned my addr 172.21.15.42:0/189589361 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:07:16.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.117+0000 7fe0a37fe640 10 monclient(hunting): get_auth_request con 0x7fe0a413c630 auth_method 0 2024-09-16T04:07:16.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.117+0000 7fe0a37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:16.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.117+0000 7fe0a37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:16.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.117+0000 7fe0a37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:16.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.117+0000 7fe0a37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:07:16.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.117+0000 7fe0a37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:07:16.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.117+0000 7fe0a37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:07:16.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.117+0000 7fe0a2ffd640 1 --2- 172.21.15.42:0/189589361 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe0a409f7b0 0x7fe0a412d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:16.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.117+0000 7fe0a2ffd640 10 monclient(hunting): get_auth_request con 0x7fe0a409f7b0 auth_method 0 2024-09-16T04:07:16.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.117+0000 7fe0a2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:16.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.117+0000 7fe0a2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:16.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.117+0000 7fe0a2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:16.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.117+0000 7fe0a37fe640 10 monclient(hunting): handle_auth_done global_id 1014152 payload 824 2024-09-16T04:07:16.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.117+0000 7fe0a37fe640 1 -- 172.21.15.42:0/189589361 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe0a409f7b0 msgr2=0x7fe0a412d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:16.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.117+0000 7fe0a37fe640 1 --2- 172.21.15.42:0/189589361 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe0a409f7b0 0x7fe0a412d690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:16.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.117+0000 7fe0a37fe640 1 -- 172.21.15.42:0/189589361 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe0a409fb90 msgr2=0x7fe0a412dbd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:07:16.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.117+0000 7fe0a37fe640 1 --2- 172.21.15.42:0/189589361 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe0a409fb90 0x7fe0a412dbd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:16.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.117+0000 7fe0a37fe640 10 monclient: _finish_hunting 0 2024-09-16T04:07:16.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.117+0000 7fe0a37fe640 1 monclient: found mon.noname-c 2024-09-16T04:07:16.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.117+0000 7fe0a37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:07:16.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.117+0000 7fe0a37fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T04:07:16.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.117+0000 7fe0a37fe640 1 -- 172.21.15.42:0/189589361 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0a412e110 con 0x7fe0a413c630 2024-09-16T04:07:16.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.117+0000 7fe0a37fe640 1 --2- 172.21.15.42:0/189589361 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe0a413c630 0x7fe0a41ceda0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fe09c00d520 tx=0x7fe09c00d9e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:07:16.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.117+0000 7fe09b7fe640 10 client.?.objecter ms_handle_connect 0x7fe0a413c630 2024-09-16T04:07:16.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.117+0000 7fe09b7fe640 10 client.?.objecter resend_mon_ops 2024-09-16T04:07:16.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.117+0000 7fe09b7fe640 1 -- 172.21.15.42:0/189589361 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe09c0125f0 con 0x7fe0a413c630 2024-09-16T04:07:16.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.117+0000 7fe09b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:16.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.117+0000 7fe09b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:07:16.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.117+0000 7fe09b7fe640 10 monclient: dump: 2024-09-16T04:07:16.138 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:07:16.138 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:16.138 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:16.138 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:16.138 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:16.138 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:16.138 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:07:16.138 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:07:16.138 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:07:16.139 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:16.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.117+0000 7fe09b7fe640 10 monclient: _finish_auth 0 2024-09-16T04:07:16.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.117+0000 7fe09b7fe640 10 monclient: _check_auth_tickets 2024-09-16T04:07:16.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.117+0000 7fe09b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:07:16.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.117+0000 7fe09b7fe640 1 -- 172.21.15.42:0/189589361 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe09c019070 con 0x7fe0a413c630 2024-09-16T04:07:16.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.117+0000 7fe09b7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:07:16.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.117+0000 7fe09b7fe640 1 -- 172.21.15.42:0/189589361 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe09c0129f0 con 0x7fe0a413c630 2024-09-16T04:07:16.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.117+0000 7fe09b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:16.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.117+0000 7fe09b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:07:16.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.117+0000 7fe09b7fe640 10 monclient: dump: 2024-09-16T04:07:16.139 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:07:16.139 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:16.139 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:16.139 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:16.140 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:16.140 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:16.140 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:07:16.140 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:07:16.140 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:07:16.140 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:16.140 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.117+0000 7fe0a9707640 5 monclient: authenticate success, global_id 1014152 2024-09-16T04:07:16.140 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.117+0000 7fe0a0ff9640 4 set_mon_vals no callback set 2024-09-16T04:07:16.140 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.117+0000 7fe0a9707640 10 monclient: _renew_subs 2024-09-16T04:07:16.140 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.117+0000 7fe0a9707640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:07:16.140 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.117+0000 7fe0a9707640 1 -- 172.21.15.42:0/189589361 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0a4073c60 con 0x7fe0a413c630 2024-09-16T04:07:16.140 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.117+0000 7fe0a9707640 10 client.1014152.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:07:16.140 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.117+0000 7fe0a9707640 10 monclient: _renew_subs 2024-09-16T04:07:16.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.117+0000 7fe0a9707640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:07:16.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.117+0000 7fe0a9707640 1 -- 172.21.15.42:0/189589361 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0a413bb90 con 0x7fe0a413c630 2024-09-16T04:07:16.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.117+0000 7fe0a9707640 1 librados: init done 2024-09-16T04:07:16.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.117+0000 7fe0a9707640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:07:16.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.117+0000 7fe0a9707640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:07:16.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.117+0000 7fe0a9707640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:07:16.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.117+0000 7fe0a9707640 1 -- 172.21.15.42:0/189589361 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe0a4133970 con 0x7fe0a413c630 2024-09-16T04:07:16.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.117+0000 7fe09b7fe640 1 -- 172.21.15.42:0/189589361 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fe09c012b90 con 0x7fe0a413c630 2024-09-16T04:07:16.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.117+0000 7fe09b7fe640 1 --2- 172.21.15.42:0/189589361 >> v2:172.21.15.154:6812/254845342 conn(0x7fe0840377f0 0x7fe084039cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:16.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.117+0000 7fe09b7fe640 1 -- 172.21.15.42:0/189589361 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(706..706 src has 451..706) ==== 5040+0+0 (secure 0 0 0) 0x7fe09c048ef0 con 0x7fe0a413c630 2024-09-16T04:07:16.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.117+0000 7fe09b7fe640 10 client.1014152.objecter ms_dispatch 0x7fe0a4000ba0 osd_map(706..706 src has 451..706) 2024-09-16T04:07:16.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.117+0000 7fe09b7fe640 3 client.1014152.objecter handle_osd_map got epochs [706,706] > 0 2024-09-16T04:07:16.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.117+0000 7fe09b7fe640 3 client.1014152.objecter handle_osd_map decoding full epoch 706 2024-09-16T04:07:16.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.117+0000 7fe09b7fe640 20 client.1014152.objecter dump_active .. 0 homeless 2024-09-16T04:07:16.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.117+0000 7fe0a2ffd640 1 --2- 172.21.15.42:0/189589361 >> v2:172.21.15.154:6812/254845342 conn(0x7fe0840377f0 0x7fe084039cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:16.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.117+0000 7fe0a2ffd640 10 monclient: get_auth_request con 0x7fe0840377f0 auth_method 0 2024-09-16T04:07:16.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.121+0000 7fe0a2ffd640 1 --2- 172.21.15.42:0/189589361 >> v2:172.21.15.154:6812/254845342 conn(0x7fe0840377f0 0x7fe084039cb0 secure :-1 s=READY pgs=3880 cs=0 l=1 rev1=1 crypto rx=0x7fe0a409f1d0 tx=0x7fe08c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:07:16.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.121+0000 7fe09b7fe640 10 client.1014152.objecter ms_handle_connect 0x7fe0840377f0 2024-09-16T04:07:16.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.121+0000 7fe09b7fe640 1 -- 172.21.15.42:0/189589361 <== mon.2 v2:172.21.15.42:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7fe09c00f300 con 0x7fe0a413c630 2024-09-16T04:07:16.143 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.121+0000 7fe09b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:07:16.143 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.121+0000 7fe09b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:07:16.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.345+0000 7fe0a9707640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:07:16.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.345+0000 7fe0a9707640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:07:16.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.345+0000 7fe0a9707640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:07:16.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.345+0000 7fe0a9707640 1 -- 172.21.15.42:0/189589361 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fe0a4067b70 con 0x7fe0a413c630 2024-09-16T04:07:16.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.349+0000 7fe09b7fe640 1 -- 172.21.15.42:0/189589361 <== mon.2 v2:172.21.15.42:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1562 (secure 0 0 0) 0x7fe09c01e840 con 0x7fe0a413c630 2024-09-16T04:07:16.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.349+0000 7fe09b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:07:16.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.349+0000 7fe09b7fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:07:16.354 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":232,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:07:16.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.349+0000 7fe0a9707640 10 librados: watch_flush enter 2024-09-16T04:07:16.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.349+0000 7fe0a9707640 10 librados: watch_flush exit 2024-09-16T04:07:16.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.349+0000 7fe0a9707640 20 client.1014152.objecter shutdown clearing up homeless session... 2024-09-16T04:07:16.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.349+0000 7fe0a9707640 10 client.1014152.objecter successfully canceled tick 2024-09-16T04:07:16.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.349+0000 7fe0a9707640 1 -- 172.21.15.42:0/189589361 >> v2:172.21.15.154:6812/254845342 conn(0x7fe0840377f0 msgr2=0x7fe084039cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:16.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.349+0000 7fe0a9707640 1 --2- 172.21.15.42:0/189589361 >> v2:172.21.15.154:6812/254845342 conn(0x7fe0840377f0 0x7fe084039cb0 secure :-1 s=READY pgs=3880 cs=0 l=1 rev1=1 crypto rx=0x7fe0a409f1d0 tx=0x7fe08c009040 comp rx=0 tx=0).stop 2024-09-16T04:07:16.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.349+0000 7fe0a9707640 10 monclient: shutdown 2024-09-16T04:07:16.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.349+0000 7fe0a9707640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:07:16.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.349+0000 7fe0a9707640 1 -- 172.21.15.42:0/189589361 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe0a413c630 msgr2=0x7fe0a41ceda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:16.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.349+0000 7fe0a9707640 1 --2- 172.21.15.42:0/189589361 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe0a413c630 0x7fe0a41ceda0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fe09c00d520 tx=0x7fe09c00d9e0 comp rx=0 tx=0).stop 2024-09-16T04:07:16.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.349+0000 7fe0a9707640 1 -- 172.21.15.42:0/189589361 shutdown_connections 2024-09-16T04:07:16.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.349+0000 7fe0a9707640 1 --2- 172.21.15.42:0/189589361 >> v2:172.21.15.154:6812/254845342 conn(0x7fe0840377f0 0x7fe084039cb0 unknown :-1 s=CLOSED pgs=3880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:16.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.349+0000 7fe0a9707640 1 --2- 172.21.15.42:0/189589361 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe0a413c630 0x7fe0a41ceda0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:16.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.349+0000 7fe0a9707640 1 --2- 172.21.15.42:0/189589361 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe0a409fb90 0x7fe0a412dbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:16.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.349+0000 7fe0a9707640 1 --2- 172.21.15.42:0/189589361 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe0a409f7b0 0x7fe0a412d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:16.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.349+0000 7fe0a9707640 1 -- 172.21.15.42:0/189589361 >> 172.21.15.42:0/189589361 conn(0x7fe0a40a2c70 msgr2=0x7fe0a40a0470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:07:16.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.349+0000 7fe0a9707640 1 -- 172.21.15.42:0/189589361 shutdown_connections 2024-09-16T04:07:16.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.349+0000 7fe0a9707640 1 -- 172.21.15.42:0/189589361 wait complete. 2024-09-16T04:07:16.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:16.349+0000 7fe0a9707640 1 librados: shutdown 2024-09-16T04:07:19.374 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:07:19.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.501+0000 7fc7c3972640 10 monclient: get_monmap_and_config 2024-09-16T04:07:19.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.501+0000 7fc7c3972640 10 monclient: build_initial_monmap 2024-09-16T04:07:19.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.501+0000 7fc7c3972640 10 monclient: monmap: 2024-09-16T04:07:19.507 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:07:19.507 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:19.507 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:07:19.505908+0000 2024-09-16T04:07:19.508 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:07:19.505908+0000 2024-09-16T04:07:19.508 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:07:19.508 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:19.508 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:07:19.508 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:07:19.508 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:07:19.508 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:07:19.508 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:07:19.508 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:19.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.501+0000 7fc7c3972640 1 Processor -- start 2024-09-16T04:07:19.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.501+0000 7fc7c3972640 1 -- start start 2024-09-16T04:07:19.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.501+0000 7fc7c3972640 10 monclient: init 2024-09-16T04:07:19.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.501+0000 7fc7c3972640 10 monclient: _reopen_session rank -1 2024-09-16T04:07:19.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.501+0000 7fc7c3972640 10 monclient: _add_conns ranks=[1,0,2,3,4] 2024-09-16T04:07:19.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.501+0000 7fc7c3972640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc7bc0a39b0 addr v1:172.21.15.154:6789/0 2024-09-16T04:07:19.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.501+0000 7fc7c3972640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc7bc0a3d90 addr v1:172.21.15.42:6789/0 2024-09-16T04:07:19.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.501+0000 7fc7c3972640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc7bc13daa0 0x7fc7bc13fe90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:19.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.501+0000 7fc7c3972640 10 monclient(hunting): picked mon.noname-b con 0x7fc7bc13daa0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:07:19.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.501+0000 7fc7c3972640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc7bc134290 con 0x7fc7bc0a3d90 2024-09-16T04:07:19.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.501+0000 7fc7c0f0d640 1 -- >> v1:172.21.15.42:6789/0 conn(0x7fc7bc0a3d90 legacy=0x7fc7bc13c380 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.42:6789/0 2024-09-16T04:07:19.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.501+0000 7fc7c3972640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc7bc09c140 con 0x7fc7bc0a39b0 2024-09-16T04:07:19.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.501+0000 7fc7c3972640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:19.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.501+0000 7fc7c3972640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fc7bc09c420 con 0x7fc7bc13daa0 2024-09-16T04:07:19.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.501+0000 7fc7c3972640 10 monclient(hunting): _renew_subs 2024-09-16T04:07:19.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.501+0000 7fc7c3972640 10 monclient(hunting): authenticate will time out at 2486.475649s 2024-09-16T04:07:19.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.501+0000 7fc7c1f0f640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc7bc13daa0 0x7fc7bc13fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:19.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.501+0000 7fc7c1f0f640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc7bc13daa0 0x7fc7bc13fe90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:53324/0 (socket says 172.21.15.42:53324) 2024-09-16T04:07:19.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.501+0000 7fc7c1f0f640 1 -- 172.21.15.42:0/2376585229 learned_addr learned my addr 172.21.15.42:0/2376585229 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:07:19.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.501+0000 7fc7c1f0f640 10 monclient(hunting): get_auth_request con 0x7fc7bc13daa0 auth_method 0 2024-09-16T04:07:19.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.501+0000 7fc7c1f0f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:19.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.501+0000 7fc7c1f0f640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:19.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.501+0000 7fc7c1f0f640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:19.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.505+0000 7fc7c1f0f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:07:19.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.505+0000 7fc7c1f0f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:07:19.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.505+0000 7fc7c1f0f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:07:19.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.505+0000 7fc7c1f0f640 10 monclient(hunting): handle_auth_done global_id 1014155 payload 274 2024-09-16T04:07:19.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.505+0000 7fc7c1f0f640 1 -- 172.21.15.42:0/2376585229 >> v1:172.21.15.154:6789/0 conn(0x7fc7bc0a39b0 legacy=0x7fc7bc133b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:19.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.505+0000 7fc7c1f0f640 1 -- 172.21.15.42:0/2376585229 >> v1:172.21.15.42:6789/0 conn(0x7fc7bc0a3d90 legacy=0x7fc7bc13c380 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:07:19.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.505+0000 7fc7c1f0f640 10 monclient: _finish_hunting 0 2024-09-16T04:07:19.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.505+0000 7fc7c1f0f640 1 monclient: found mon.noname-b 2024-09-16T04:07:19.513 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.505+0000 7fc7c1f0f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:07:19.513 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.505+0000 7fc7c1f0f640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T04:07:19.513 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.505+0000 7fc7c1f0f640 1 -- 172.21.15.42:0/2376585229 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7bc06dd60 con 0x7fc7bc13daa0 2024-09-16T04:07:19.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.505+0000 7fc7c1f0f640 1 --2- 172.21.15.42:0/2376585229 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc7bc13daa0 0x7fc7bc13fe90 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fc7b4009e80 tx=0x7fc7b401c8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a31a17a5ffa77c7a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:07:19.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.505+0000 7fc7b3fff640 1 -- 172.21.15.42:0/2376585229 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7b40a5590 con 0x7fc7bc13daa0 2024-09-16T04:07:19.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.505+0000 7fc7b3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:19.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.505+0000 7fc7b3fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:07:19.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.505+0000 7fc7b3fff640 10 monclient: dump: 2024-09-16T04:07:19.518 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:07:19.519 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:19.519 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:19.519 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:19.519 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:19.519 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:19.519 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:07:19.519 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:07:19.519 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:07:19.519 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:19.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.505+0000 7fc7b3fff640 10 monclient: _finish_auth 0 2024-09-16T04:07:19.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.505+0000 7fc7b3fff640 10 monclient: _check_auth_tickets 2024-09-16T04:07:19.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.505+0000 7fc7b3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:07:19.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.505+0000 7fc7b3fff640 1 -- 172.21.15.42:0/2376585229 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc7b40a5730 con 0x7fc7bc13daa0 2024-09-16T04:07:19.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.505+0000 7fc7c3972640 5 monclient: authenticate success, global_id 1014155 2024-09-16T04:07:19.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.505+0000 7fc7c3972640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:07:19.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.505+0000 7fc7b3fff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:07:19.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.505+0000 7fc7c3972640 10 monclient: get_monmap_and_config success 2024-09-16T04:07:19.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.505+0000 7fc7c3972640 4 set_mon_vals no callback set 2024-09-16T04:07:19.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.505+0000 7fc7b3fff640 1 -- 172.21.15.42:0/2376585229 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7b40a5b00 con 0x7fc7bc13daa0 2024-09-16T04:07:19.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.505+0000 7fc7b3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:19.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.505+0000 7fc7b3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:07:19.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.505+0000 7fc7b3fff640 10 monclient: dump: 2024-09-16T04:07:19.520 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:07:19.520 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:19.520 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:19.520 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:19.520 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:19.521 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:19.521 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:07:19.521 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:07:19.521 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:07:19.521 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:19.521 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.505+0000 7fc7c3972640 10 monclient: shutdown 2024-09-16T04:07:19.521 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.505+0000 7fc7c3972640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:07:19.521 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.505+0000 7fc7c3972640 1 -- 172.21.15.42:0/2376585229 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc7bc13daa0 msgr2=0x7fc7bc13fe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:19.521 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.505+0000 7fc7c3972640 1 --2- 172.21.15.42:0/2376585229 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc7bc13daa0 0x7fc7bc13fe90 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fc7b4009e80 tx=0x7fc7b401c8d0 comp rx=0 tx=0).stop 2024-09-16T04:07:19.521 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.505+0000 7fc7c3972640 1 -- 172.21.15.42:0/2376585229 shutdown_connections 2024-09-16T04:07:19.521 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.505+0000 7fc7c3972640 1 --2- 172.21.15.42:0/2376585229 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc7bc13daa0 0x7fc7bc13fe90 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:19.521 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.505+0000 7fc7c3972640 1 -- 172.21.15.42:0/2376585229 >> 172.21.15.42:0/2376585229 conn(0x7fc7bc0a9fb0 msgr2=0x7fc7bc0aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:07:19.521 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.505+0000 7fc7c3972640 1 -- 172.21.15.42:0/2376585229 shutdown_connections 2024-09-16T04:07:19.521 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.505+0000 7fc7c3972640 1 -- 172.21.15.42:0/2376585229 wait complete. 2024-09-16T04:07:19.522 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.505+0000 7fc7c3972640 10 monclient: build_initial_monmap 2024-09-16T04:07:19.522 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.505+0000 7fc7c3972640 10 monclient: monmap: 2024-09-16T04:07:19.522 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:07:19.522 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:07:19.522 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:07:19.522 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:07:19.522 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:07:19.522 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:19.522 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:07:19.522 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:07:19.522 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:07:19.522 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:19.522 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.509+0000 7fc7c3972640 1 librados: starting msgr at 2024-09-16T04:07:19.522 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.509+0000 7fc7c3972640 1 librados: starting objecter 2024-09-16T04:07:19.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.509+0000 7fc7c3972640 1 Processor -- start 2024-09-16T04:07:19.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.509+0000 7fc7c3972640 1 -- start start 2024-09-16T04:07:19.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.509+0000 7fc7c3972640 1 librados: setting wanted keys 2024-09-16T04:07:19.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.509+0000 7fc7c3972640 1 librados: calling monclient init 2024-09-16T04:07:19.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.509+0000 7fc7c3972640 10 monclient: init 2024-09-16T04:07:19.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.509+0000 7fc7c3972640 10 monclient: _reopen_session rank -1 2024-09-16T04:07:19.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.509+0000 7fc7c3972640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:07:19.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.509+0000 7fc7c3972640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc7bc0a39b0 0x7fc7bc1305f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:19.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.509+0000 7fc7c3972640 10 monclient(hunting): picked mon.noname-b con 0x7fc7bc0a39b0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:07:19.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.509+0000 7fc7c3972640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc7bc0a3d90 0x7fc7bc130b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:19.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.509+0000 7fc7c3972640 10 monclient(hunting): picked mon.noname-c con 0x7fc7bc0a3d90 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:07:19.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.509+0000 7fc7c3972640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc7bc13daa0 0x7fc7bc13acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:19.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.509+0000 7fc7c3972640 10 monclient(hunting): picked mon.noname-a con 0x7fc7bc13daa0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:07:19.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.509+0000 7fc7c3972640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:19.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.509+0000 7fc7c3972640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fc7bc142f80 con 0x7fc7bc13daa0 2024-09-16T04:07:19.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.509+0000 7fc7c3972640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:19.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.509+0000 7fc7c3972640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc7bc142e00 con 0x7fc7bc0a39b0 2024-09-16T04:07:19.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.509+0000 7fc7c3972640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:19.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.509+0000 7fc7c3972640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fc7bc143100 con 0x7fc7bc0a3d90 2024-09-16T04:07:19.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.509+0000 7fc7c3972640 10 monclient(hunting): _renew_subs 2024-09-16T04:07:19.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.509+0000 7fc7c3972640 10 monclient(hunting): authenticate will time out at 2486.480917s 2024-09-16T04:07:19.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.509+0000 7fc7c1f0f640 1 -- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc7bc13daa0 msgr2=0x7fc7bc13acb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.42:3300/0 2024-09-16T04:07:19.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.509+0000 7fc7c1f0f640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc7bc13daa0 0x7fc7bc13acb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:07:19.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.509+0000 7fc7c0f0d640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc7bc0a3d90 0x7fc7bc130b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:19.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.509+0000 7fc7c0f0d640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc7bc0a3d90 0x7fc7bc130b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:53332/0 (socket says 172.21.15.42:53332) 2024-09-16T04:07:19.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.509+0000 7fc7c0f0d640 1 -- 172.21.15.42:0/709949175 learned_addr learned my addr 172.21.15.42:0/709949175 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:07:19.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.509+0000 7fc7c0f0d640 10 monclient(hunting): get_auth_request con 0x7fc7bc0a3d90 auth_method 0 2024-09-16T04:07:19.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.509+0000 7fc7c0f0d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:19.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.509+0000 7fc7c0f0d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:19.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.509+0000 7fc7c0f0d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:19.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.509+0000 7fc7c0f0d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:07:19.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.509+0000 7fc7c0f0d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:07:19.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.509+0000 7fc7c0f0d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:07:19.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.509+0000 7fc7c170e640 1 --2- 172.21.15.42:0/709949175 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc7bc0a39b0 0x7fc7bc1305f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:19.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.509+0000 7fc7c170e640 10 monclient(hunting): get_auth_request con 0x7fc7bc0a39b0 auth_method 0 2024-09-16T04:07:19.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.509+0000 7fc7c170e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:19.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.509+0000 7fc7c170e640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:19.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.509+0000 7fc7c170e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:19.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.509+0000 7fc7c0f0d640 10 monclient(hunting): handle_auth_done global_id 1014158 payload 824 2024-09-16T04:07:19.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.509+0000 7fc7c0f0d640 1 -- 172.21.15.42:0/709949175 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc7bc0a39b0 msgr2=0x7fc7bc1305f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:19.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.509+0000 7fc7c0f0d640 1 --2- 172.21.15.42:0/709949175 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc7bc0a39b0 0x7fc7bc1305f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:19.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.509+0000 7fc7c0f0d640 1 -- 172.21.15.42:0/709949175 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc7bc13daa0 msgr2=0x7fc7bc13acb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:07:19.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.509+0000 7fc7c0f0d640 1 --2- 172.21.15.42:0/709949175 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc7bc13daa0 0x7fc7bc13acb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:19.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.509+0000 7fc7c0f0d640 10 monclient: _finish_hunting 0 2024-09-16T04:07:19.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.509+0000 7fc7c0f0d640 1 monclient: found mon.noname-c 2024-09-16T04:07:19.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.509+0000 7fc7c0f0d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:07:19.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.509+0000 7fc7c0f0d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T04:07:19.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.509+0000 7fc7c0f0d640 1 -- 172.21.15.42:0/709949175 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7bc09e350 con 0x7fc7bc0a3d90 2024-09-16T04:07:19.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.509+0000 7fc7c0f0d640 1 --2- 172.21.15.42:0/709949175 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc7bc0a3d90 0x7fc7bc130b30 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fc7ac00b530 tx=0x7fc7ac00b9f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:07:19.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.509+0000 7fc7b1ffb640 10 client.?.objecter ms_handle_connect 0x7fc7bc0a3d90 2024-09-16T04:07:19.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.509+0000 7fc7b1ffb640 10 client.?.objecter resend_mon_ops 2024-09-16T04:07:19.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.509+0000 7fc7b1ffb640 1 -- 172.21.15.42:0/709949175 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7ac013020 con 0x7fc7bc0a3d90 2024-09-16T04:07:19.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.509+0000 7fc7b1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:19.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.509+0000 7fc7b1ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:07:19.527 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.509+0000 7fc7b1ffb640 10 monclient: dump: 2024-09-16T04:07:19.527 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:07:19.527 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:19.527 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:19.527 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:19.527 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:19.527 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:19.527 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:07:19.527 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:07:19.527 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:07:19.527 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:19.527 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.509+0000 7fc7b1ffb640 10 monclient: _finish_auth 0 2024-09-16T04:07:19.527 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.509+0000 7fc7b1ffb640 10 monclient: _check_auth_tickets 2024-09-16T04:07:19.527 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.509+0000 7fc7b1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:07:19.528 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.509+0000 7fc7c3972640 5 monclient: authenticate success, global_id 1014158 2024-09-16T04:07:19.528 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.509+0000 7fc7c3972640 10 monclient: _renew_subs 2024-09-16T04:07:19.528 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.509+0000 7fc7b1ffb640 1 -- 172.21.15.42:0/709949175 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc7ac003df0 con 0x7fc7bc0a3d90 2024-09-16T04:07:19.528 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.509+0000 7fc7c3972640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:07:19.528 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.509+0000 7fc7c3972640 1 -- 172.21.15.42:0/709949175 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7bc09d6b0 con 0x7fc7bc0a3d90 2024-09-16T04:07:19.528 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.509+0000 7fc7b1ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:07:19.528 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.509+0000 7fc7b1ffb640 1 -- 172.21.15.42:0/709949175 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7ac0052b0 con 0x7fc7bc0a3d90 2024-09-16T04:07:19.528 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.509+0000 7fc7b1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:19.528 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.509+0000 7fc7b1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:07:19.528 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.509+0000 7fc7b1ffb640 10 monclient: dump: 2024-09-16T04:07:19.528 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:07:19.528 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:19.528 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:19.528 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:19.529 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:19.529 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:19.529 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:07:19.529 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:07:19.529 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:07:19.529 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:19.529 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.509+0000 7fc7b37fe640 4 set_mon_vals no callback set 2024-09-16T04:07:19.529 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.509+0000 7fc7c3972640 10 client.1014158.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:07:19.529 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.509+0000 7fc7c3972640 10 monclient: _renew_subs 2024-09-16T04:07:19.529 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.509+0000 7fc7c3972640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:07:19.529 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.509+0000 7fc7c3972640 1 -- 172.21.15.42:0/709949175 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7bc133230 con 0x7fc7bc0a3d90 2024-09-16T04:07:19.529 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.509+0000 7fc7c3972640 1 librados: init done 2024-09-16T04:07:19.529 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.509+0000 7fc7c3972640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:07:19.529 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.509+0000 7fc7c3972640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:07:19.530 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.509+0000 7fc7c3972640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:07:19.530 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.509+0000 7fc7c3972640 1 -- 172.21.15.42:0/709949175 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc7bc13fea0 con 0x7fc7bc0a3d90 2024-09-16T04:07:19.530 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.509+0000 7fc7b1ffb640 1 -- 172.21.15.42:0/709949175 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fc7ac016020 con 0x7fc7bc0a3d90 2024-09-16T04:07:19.530 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.509+0000 7fc7b1ffb640 1 --2- 172.21.15.42:0/709949175 >> v2:172.21.15.154:6812/254845342 conn(0x7fc7a00377f0 0x7fc7a0039cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:19.530 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.509+0000 7fc7b1ffb640 1 -- 172.21.15.42:0/709949175 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(706..706 src has 451..706) ==== 5040+0+0 (secure 0 0 0) 0x7fc7ac0488e0 con 0x7fc7bc0a3d90 2024-09-16T04:07:19.530 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.509+0000 7fc7b1ffb640 10 client.1014158.objecter ms_dispatch 0x7fc7bc000ba0 osd_map(706..706 src has 451..706) 2024-09-16T04:07:19.530 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.509+0000 7fc7b1ffb640 3 client.1014158.objecter handle_osd_map got epochs [706,706] > 0 2024-09-16T04:07:19.530 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.509+0000 7fc7b1ffb640 3 client.1014158.objecter handle_osd_map decoding full epoch 706 2024-09-16T04:07:19.530 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.509+0000 7fc7b1ffb640 20 client.1014158.objecter dump_active .. 0 homeless 2024-09-16T04:07:19.530 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.509+0000 7fc7c170e640 1 --2- 172.21.15.42:0/709949175 >> v2:172.21.15.154:6812/254845342 conn(0x7fc7a00377f0 0x7fc7a0039cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:19.531 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.509+0000 7fc7c170e640 10 monclient: get_auth_request con 0x7fc7a00377f0 auth_method 0 2024-09-16T04:07:19.531 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.513+0000 7fc7c170e640 1 --2- 172.21.15.42:0/709949175 >> v2:172.21.15.154:6812/254845342 conn(0x7fc7a00377f0 0x7fc7a0039cb0 secure :-1 s=READY pgs=3881 cs=0 l=1 rev1=1 crypto rx=0x7fc7b8000f70 tx=0x7fc7b8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:07:19.531 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.513+0000 7fc7b1ffb640 10 client.1014158.objecter ms_handle_connect 0x7fc7a00377f0 2024-09-16T04:07:19.531 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.513+0000 7fc7b1ffb640 1 -- 172.21.15.42:0/709949175 <== mon.2 v2:172.21.15.42:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7fc7ac01d020 con 0x7fc7bc0a3d90 2024-09-16T04:07:19.531 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.513+0000 7fc7b1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:07:19.531 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.513+0000 7fc7b1ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:07:19.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.741+0000 7fc7c3972640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:07:19.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.741+0000 7fc7c3972640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:07:19.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.741+0000 7fc7c3972640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:07:19.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.741+0000 7fc7c3972640 1 -- 172.21.15.42:0/709949175 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fc7bc133b80 con 0x7fc7bc0a3d90 2024-09-16T04:07:19.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.745+0000 7fc7b1ffb640 1 -- 172.21.15.42:0/709949175 <== mon.2 v2:172.21.15.42:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1562 (secure 0 0 0) 0x7fc7bc09e350 con 0x7fc7bc0a3d90 2024-09-16T04:07:19.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.745+0000 7fc7b1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:07:19.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.745+0000 7fc7b1ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:07:19.750 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":232,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":14,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:07:19.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.745+0000 7fc7c3972640 10 librados: watch_flush enter 2024-09-16T04:07:19.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.745+0000 7fc7c3972640 10 librados: watch_flush exit 2024-09-16T04:07:19.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.745+0000 7fc7c3972640 20 client.1014158.objecter shutdown clearing up homeless session... 2024-09-16T04:07:19.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.745+0000 7fc7c3972640 10 client.1014158.objecter successfully canceled tick 2024-09-16T04:07:19.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.745+0000 7fc7c3972640 1 -- 172.21.15.42:0/709949175 >> v2:172.21.15.154:6812/254845342 conn(0x7fc7a00377f0 msgr2=0x7fc7a0039cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:19.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.745+0000 7fc7c3972640 1 --2- 172.21.15.42:0/709949175 >> v2:172.21.15.154:6812/254845342 conn(0x7fc7a00377f0 0x7fc7a0039cb0 secure :-1 s=READY pgs=3881 cs=0 l=1 rev1=1 crypto rx=0x7fc7b8000f70 tx=0x7fc7b8009040 comp rx=0 tx=0).stop 2024-09-16T04:07:19.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.745+0000 7fc7c3972640 10 monclient: shutdown 2024-09-16T04:07:19.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.745+0000 7fc7c3972640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:07:19.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.745+0000 7fc7c3972640 1 -- 172.21.15.42:0/709949175 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc7bc0a3d90 msgr2=0x7fc7bc130b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:19.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.745+0000 7fc7c3972640 1 --2- 172.21.15.42:0/709949175 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc7bc0a3d90 0x7fc7bc130b30 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fc7ac00b530 tx=0x7fc7ac00b9f0 comp rx=0 tx=0).stop 2024-09-16T04:07:19.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.745+0000 7fc7c3972640 1 -- 172.21.15.42:0/709949175 shutdown_connections 2024-09-16T04:07:19.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.745+0000 7fc7c3972640 1 --2- 172.21.15.42:0/709949175 >> v2:172.21.15.154:6812/254845342 conn(0x7fc7a00377f0 0x7fc7a0039cb0 unknown :-1 s=CLOSED pgs=3881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:19.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.745+0000 7fc7c3972640 1 --2- 172.21.15.42:0/709949175 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc7bc13daa0 0x7fc7bc13acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:19.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.745+0000 7fc7c3972640 1 --2- 172.21.15.42:0/709949175 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc7bc0a3d90 0x7fc7bc130b30 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:19.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.745+0000 7fc7c3972640 1 --2- 172.21.15.42:0/709949175 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc7bc0a39b0 0x7fc7bc1305f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:19.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.745+0000 7fc7c3972640 1 -- 172.21.15.42:0/709949175 >> 172.21.15.42:0/709949175 conn(0x7fc7bc0a9fb0 msgr2=0x7fc7bc0a8e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:07:19.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.745+0000 7fc7c3972640 1 -- 172.21.15.42:0/709949175 shutdown_connections 2024-09-16T04:07:19.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.745+0000 7fc7c3972640 1 -- 172.21.15.42:0/709949175 wait complete. 2024-09-16T04:07:19.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:19.745+0000 7fc7c3972640 1 librados: shutdown 2024-09-16T04:07:22.770 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:07:22.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:22.897+0000 7f1e5e5b2640 10 monclient: get_monmap_and_config 2024-09-16T04:07:22.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:22.897+0000 7f1e5e5b2640 10 monclient: build_initial_monmap 2024-09-16T04:07:22.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:22.897+0000 7f1e5e5b2640 10 monclient: monmap: 2024-09-16T04:07:22.904 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:07:22.904 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:22.904 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:07:22.902131+0000 2024-09-16T04:07:22.904 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:07:22.902131+0000 2024-09-16T04:07:22.904 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:07:22.904 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:22.904 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:07:22.904 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:07:22.904 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:07:22.904 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:07:22.904 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:07:22.904 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:22.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:22.897+0000 7f1e5e5b2640 1 Processor -- start 2024-09-16T04:07:22.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:22.897+0000 7f1e5e5b2640 1 -- start start 2024-09-16T04:07:22.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:22.897+0000 7f1e5e5b2640 10 monclient: init 2024-09-16T04:07:22.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:22.897+0000 7f1e5e5b2640 10 monclient: _reopen_session rank -1 2024-09-16T04:07:22.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:22.897+0000 7f1e5e5b2640 10 monclient: _add_conns ranks=[3,1,2,4,0] 2024-09-16T04:07:22.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:22.897+0000 7f1e5e5b2640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f1e5806db90 0x7f1e581340c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:22.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:22.897+0000 7f1e5e5b2640 10 monclient(hunting): picked mon.noname-a con 0x7f1e5806db90 addr v2:172.21.15.42:3300/0 2024-09-16T04:07:22.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:22.897+0000 7f1e5e5b2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1e580a37f0 addr v1:172.21.15.154:6789/0 2024-09-16T04:07:22.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:22.897+0000 7f1e5e5b2640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1e580a3c00 0x7f1e58137520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:22.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:22.897+0000 7f1e5e5b2640 10 monclient(hunting): picked mon.noname-b con 0x7f1e580a3c00 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:07:22.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:22.897+0000 7f1e5e5b2640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1e5806ece0 con 0x7f1e580a37f0 2024-09-16T04:07:22.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:22.897+0000 7f1e5e5b2640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:22.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:22.897+0000 7f1e5e5b2640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f1e5806df70 con 0x7f1e5806db90 2024-09-16T04:07:22.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:22.897+0000 7f1e5e5b2640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:22.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:22.897+0000 7f1e5e5b2640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f1e5809bf80 con 0x7f1e580a3c00 2024-09-16T04:07:22.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:22.897+0000 7f1e5e5b2640 10 monclient(hunting): _renew_subs 2024-09-16T04:07:22.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:22.897+0000 7f1e5e5b2640 10 monclient(hunting): authenticate will time out at 2489.871698s 2024-09-16T04:07:22.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:22.897+0000 7f1e57fff640 1 -- >> v2:172.21.15.42:3300/0 conn(0x7f1e5806db90 msgr2=0x7f1e581340c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3300/0 2024-09-16T04:07:22.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:22.897+0000 7f1e57fff640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f1e5806db90 0x7f1e581340c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:07:22.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:22.897+0000 7f1e5cb4f640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1e580a3c00 0x7f1e58137520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:22.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:22.897+0000 7f1e5cb4f640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1e580a3c00 0x7f1e58137520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:53340/0 (socket says 172.21.15.42:53340) 2024-09-16T04:07:22.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:22.897+0000 7f1e5cb4f640 1 -- 172.21.15.42:0/1848860569 learned_addr learned my addr 172.21.15.42:0/1848860569 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:07:22.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:22.897+0000 7f1e5cb4f640 10 monclient(hunting): get_auth_request con 0x7f1e580a3c00 auth_method 0 2024-09-16T04:07:22.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:22.897+0000 7f1e5cb4f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:22.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:22.897+0000 7f1e5cb4f640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:22.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:22.897+0000 7f1e5cb4f640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:22.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:22.901+0000 7f1e5cb4f640 1 -- 172.21.15.42:0/1848860569 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1e580a3c00 msgr2=0x7f1e58137520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T04:07:22.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:22.901+0000 7f1e5cb4f640 1 -- 172.21.15.42:0/1848860569 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1e580a3c00 msgr2=0x7f1e58137520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:07:22.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:22.901+0000 7f1e5cb4f640 1 --2- 172.21.15.42:0/1848860569 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1e580a3c00 0x7f1e58137520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:07:22.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:22.901+0000 7f1e5cb4f640 1 --2- 172.21.15.42:0/1848860569 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1e580a3c00 0x7f1e58137520 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:07:23.105 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:23.101+0000 7f1e57fff640 1 -- 172.21.15.42:0/1848860569 >> v2:172.21.15.42:3300/0 conn(0x7f1e5806db90 msgr2=0x7f1e581340c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3300/0 2024-09-16T04:07:23.105 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:23.101+0000 7f1e57fff640 1 --2- 172.21.15.42:0/1848860569 >> v2:172.21.15.42:3300/0 conn(0x7f1e5806db90 0x7f1e581340c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T04:07:23.110 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:23.105+0000 7f1e5cb4f640 1 --2- 172.21.15.42:0/1848860569 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1e580a3c00 0x7f1e58137520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:23.110 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:23.105+0000 7f1e5cb4f640 10 monclient(hunting): get_auth_request con 0x7f1e580a3c00 auth_method 0 2024-09-16T04:07:23.110 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:23.105+0000 7f1e5cb4f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:23.110 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:23.105+0000 7f1e5cb4f640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:23.110 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:23.105+0000 7f1e5cb4f640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:07:23.110 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:23.105+0000 7f1e5cb4f640 1 -- 172.21.15.42:0/1848860569 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1e580a3c00 msgr2=0x7f1e58137520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:07:23.110 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:23.105+0000 7f1e5cb4f640 1 -- 172.21.15.42:0/1848860569 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1e580a3c00 msgr2=0x7f1e58137520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:07:23.111 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:23.105+0000 7f1e5cb4f640 1 --2- 172.21.15.42:0/1848860569 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1e580a3c00 0x7f1e58137520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:07:23.111 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:23.105+0000 7f1e5cb4f640 1 --2- 172.21.15.42:0/1848860569 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1e580a3c00 0x7f1e58137520 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T04:07:23.509 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:23.505+0000 7f1e57fff640 1 -- 172.21.15.42:0/1848860569 >> v2:172.21.15.42:3300/0 conn(0x7f1e5806db90 msgr2=0x7f1e581340c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3300/0 2024-09-16T04:07:23.509 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:23.505+0000 7f1e57fff640 1 --2- 172.21.15.42:0/1848860569 >> v2:172.21.15.42:3300/0 conn(0x7f1e5806db90 0x7f1e581340c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T04:07:23.514 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:23.509+0000 7f1e5cb4f640 1 --2- 172.21.15.42:0/1848860569 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1e580a3c00 0x7f1e58137520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:23.514 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:23.509+0000 7f1e5cb4f640 10 monclient(hunting): get_auth_request con 0x7f1e580a3c00 auth_method 0 2024-09-16T04:07:23.514 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:23.509+0000 7f1e5cb4f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:23.514 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:23.509+0000 7f1e5cb4f640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:23.514 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:23.509+0000 7f1e5cb4f640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:07:23.514 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:23.509+0000 7f1e5cb4f640 1 -- 172.21.15.42:0/1848860569 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1e580a3c00 msgr2=0x7f1e58137520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:07:23.514 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:23.509+0000 7f1e5cb4f640 1 -- 172.21.15.42:0/1848860569 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1e580a3c00 msgr2=0x7f1e58137520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:07:23.514 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:23.509+0000 7f1e5cb4f640 1 --2- 172.21.15.42:0/1848860569 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1e580a3c00 0x7f1e58137520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:07:23.514 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:23.509+0000 7f1e5cb4f640 1 --2- 172.21.15.42:0/1848860569 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1e580a3c00 0x7f1e58137520 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T04:07:24.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:24.309+0000 7f1e57fff640 1 -- 172.21.15.42:0/1848860569 >> v2:172.21.15.42:3300/0 conn(0x7f1e5806db90 msgr2=0x7f1e581340c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3300/0 2024-09-16T04:07:24.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:24.309+0000 7f1e57fff640 1 --2- 172.21.15.42:0/1848860569 >> v2:172.21.15.42:3300/0 conn(0x7f1e5806db90 0x7f1e581340c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T04:07:24.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:24.313+0000 7f1e5cb4f640 1 --2- 172.21.15.42:0/1848860569 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1e580a3c00 0x7f1e58137520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:24.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:24.313+0000 7f1e5cb4f640 10 monclient(hunting): get_auth_request con 0x7f1e580a3c00 auth_method 0 2024-09-16T04:07:24.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:24.313+0000 7f1e5cb4f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:24.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:24.313+0000 7f1e5cb4f640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:24.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:24.313+0000 7f1e5cb4f640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:07:24.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:24.313+0000 7f1e5cb4f640 1 -- 172.21.15.42:0/1848860569 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1e580a3c00 msgr2=0x7f1e58137520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:07:24.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:24.313+0000 7f1e5cb4f640 1 -- 172.21.15.42:0/1848860569 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1e580a3c00 msgr2=0x7f1e58137520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:07:24.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:24.313+0000 7f1e5cb4f640 1 --2- 172.21.15.42:0/1848860569 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1e580a3c00 0x7f1e58137520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:07:24.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:24.313+0000 7f1e5cb4f640 1 --2- 172.21.15.42:0/1848860569 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1e580a3c00 0x7f1e58137520 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T04:07:25.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:25.897+0000 7f1e55ffb640 10 monclient(hunting): tick 2024-09-16T04:07:25.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:25.897+0000 7f1e55ffb640 10 monclient(hunting): _check_auth_tickets 2024-09-16T04:07:25.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:25.897+0000 7f1e55ffb640 1 monclient(hunting): continuing hunt 2024-09-16T04:07:25.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:25.897+0000 7f1e55ffb640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T04:07:25.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:25.897+0000 7f1e55ffb640 1 -- 172.21.15.42:0/1848860569 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1e580a3c00 msgr2=0x7f1e58137520 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:07:25.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:25.897+0000 7f1e55ffb640 1 --2- 172.21.15.42:0/1848860569 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1e580a3c00 0x7f1e58137520 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:25.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:25.897+0000 7f1e55ffb640 1 -- 172.21.15.42:0/1848860569 >> v2:172.21.15.42:3300/0 conn(0x7f1e5806db90 msgr2=0x7f1e581340c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:07:25.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:25.897+0000 7f1e55ffb640 1 --2- 172.21.15.42:0/1848860569 >> v2:172.21.15.42:3300/0 conn(0x7f1e5806db90 0x7f1e581340c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:25.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:25.897+0000 7f1e55ffb640 1 -- 172.21.15.42:0/1848860569 >> v1:172.21.15.154:6789/0 conn(0x7f1e580a37f0 legacy=0x7f1e58136e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:25.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:25.897+0000 7f1e55ffb640 10 monclient: _add_conns ranks=[0,4] 2024-09-16T04:07:25.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:25.897+0000 7f1e55ffb640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1e380036d0 addr v1:172.21.15.42:6789/0 2024-09-16T04:07:25.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:25.897+0000 7f1e55ffb640 1 --2- 172.21.15.42:0/1848860569 >> v2:172.21.15.154:3300/0 conn(0x7f1e38007420 0x7f1e380098e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:25.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:25.901+0000 7f1e55ffb640 10 monclient(hunting): picked mon.noname-c con 0x7f1e38007420 addr v2:172.21.15.154:3300/0 2024-09-16T04:07:25.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:25.901+0000 7f1e55ffb640 1 -- 172.21.15.42:0/1848860569 --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1e58134780 con 0x7f1e380036d0 2024-09-16T04:07:25.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:25.901+0000 7f1e55ffb640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:25.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:25.901+0000 7f1e55ffb640 1 -- 172.21.15.42:0/1848860569 --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f1e5806df70 con 0x7f1e38007420 2024-09-16T04:07:25.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:25.901+0000 7f1e55ffb640 10 monclient(hunting): _renew_subs 2024-09-16T04:07:25.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:25.901+0000 7f1e57fff640 1 -- 172.21.15.42:0/1848860569 >> v1:172.21.15.42:6789/0 conn(0x7f1e380036d0 legacy=0x7f1e38005b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.42:6789/0 2024-09-16T04:07:25.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:25.901+0000 7f1e577fe640 1 --2- 172.21.15.42:0/1848860569 >> v2:172.21.15.154:3300/0 conn(0x7f1e38007420 0x7f1e380098e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:25.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:25.901+0000 7f1e577fe640 10 monclient(hunting): get_auth_request con 0x7f1e38007420 auth_method 0 2024-09-16T04:07:25.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:25.901+0000 7f1e577fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:25.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:25.901+0000 7f1e577fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:25.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:25.901+0000 7f1e577fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:25.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:25.901+0000 7f1e577fe640 1 -- 172.21.15.42:0/1848860569 >> v2:172.21.15.154:3300/0 conn(0x7f1e38007420 msgr2=0x7f1e380098e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:07:25.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:25.901+0000 7f1e577fe640 1 -- 172.21.15.42:0/1848860569 >> v2:172.21.15.154:3300/0 conn(0x7f1e38007420 msgr2=0x7f1e380098e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:07:25.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:25.901+0000 7f1e577fe640 1 --2- 172.21.15.42:0/1848860569 >> v2:172.21.15.154:3300/0 conn(0x7f1e38007420 0x7f1e380098e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:07:25.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:25.901+0000 7f1e577fe640 1 --2- 172.21.15.42:0/1848860569 >> v2:172.21.15.154:3300/0 conn(0x7f1e38007420 0x7f1e380098e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:07:26.109 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:26.105+0000 7f1e57fff640 1 -- 172.21.15.42:0/1848860569 >> v1:172.21.15.42:6789/0 conn(0x7f1e380036d0 legacy=0x7f1e38005b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.42:6789/0 2024-09-16T04:07:26.110 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:26.105+0000 7f1e577fe640 1 --2- 172.21.15.42:0/1848860569 >> v2:172.21.15.154:3300/0 conn(0x7f1e38007420 0x7f1e380098e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:26.110 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:26.105+0000 7f1e577fe640 10 monclient(hunting): get_auth_request con 0x7f1e38007420 auth_method 0 2024-09-16T04:07:26.110 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:26.105+0000 7f1e577fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:26.110 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:26.105+0000 7f1e577fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:26.110 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:26.105+0000 7f1e577fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:07:26.111 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:26.105+0000 7f1e577fe640 1 -- 172.21.15.42:0/1848860569 >> v2:172.21.15.154:3300/0 conn(0x7f1e38007420 msgr2=0x7f1e380098e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:07:26.111 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:26.105+0000 7f1e577fe640 1 -- 172.21.15.42:0/1848860569 >> v2:172.21.15.154:3300/0 conn(0x7f1e38007420 msgr2=0x7f1e380098e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:07:26.111 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:26.105+0000 7f1e577fe640 1 --2- 172.21.15.42:0/1848860569 >> v2:172.21.15.154:3300/0 conn(0x7f1e38007420 0x7f1e380098e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:07:26.111 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:26.105+0000 7f1e577fe640 1 --2- 172.21.15.42:0/1848860569 >> v2:172.21.15.154:3300/0 conn(0x7f1e38007420 0x7f1e380098e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T04:07:26.513 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:26.509+0000 7f1e57fff640 1 -- 172.21.15.42:0/1848860569 >> v1:172.21.15.42:6789/0 conn(0x7f1e380036d0 legacy=0x7f1e38005b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.42:6789/0 2024-09-16T04:07:26.514 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:26.509+0000 7f1e577fe640 1 --2- 172.21.15.42:0/1848860569 >> v2:172.21.15.154:3300/0 conn(0x7f1e38007420 0x7f1e380098e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:26.514 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:26.509+0000 7f1e577fe640 10 monclient(hunting): get_auth_request con 0x7f1e38007420 auth_method 0 2024-09-16T04:07:26.514 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:26.509+0000 7f1e577fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:26.514 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:26.509+0000 7f1e577fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:26.514 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:26.509+0000 7f1e577fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:07:26.514 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:26.509+0000 7f1e577fe640 1 -- 172.21.15.42:0/1848860569 >> v2:172.21.15.154:3300/0 conn(0x7f1e38007420 msgr2=0x7f1e380098e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:07:26.514 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:26.509+0000 7f1e577fe640 1 -- 172.21.15.42:0/1848860569 >> v2:172.21.15.154:3300/0 conn(0x7f1e38007420 msgr2=0x7f1e380098e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:07:26.514 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:26.509+0000 7f1e577fe640 1 --2- 172.21.15.42:0/1848860569 >> v2:172.21.15.154:3300/0 conn(0x7f1e38007420 0x7f1e380098e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:07:26.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:26.509+0000 7f1e577fe640 1 --2- 172.21.15.42:0/1848860569 >> v2:172.21.15.154:3300/0 conn(0x7f1e38007420 0x7f1e380098e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T04:07:27.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.313+0000 7f1e57fff640 1 -- 172.21.15.42:0/1848860569 >> v1:172.21.15.42:6789/0 conn(0x7f1e380036d0 legacy=0x7f1e38005b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.42:6789/0 2024-09-16T04:07:27.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.313+0000 7f1e577fe640 1 --2- 172.21.15.42:0/1848860569 >> v2:172.21.15.154:3300/0 conn(0x7f1e38007420 0x7f1e380098e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:27.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.313+0000 7f1e577fe640 10 monclient(hunting): get_auth_request con 0x7f1e38007420 auth_method 0 2024-09-16T04:07:27.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.313+0000 7f1e577fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:27.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.313+0000 7f1e577fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:27.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.313+0000 7f1e577fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:07:27.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.313+0000 7f1e577fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:07:27.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.313+0000 7f1e577fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:07:27.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.313+0000 7f1e577fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:07:27.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.313+0000 7f1e577fe640 10 monclient(hunting): handle_auth_done global_id 905071 payload 274 2024-09-16T04:07:27.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.313+0000 7f1e577fe640 1 -- 172.21.15.42:0/1848860569 >> v1:172.21.15.42:6789/0 conn(0x7f1e380036d0 legacy=0x7f1e38005b50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:07:27.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.313+0000 7f1e577fe640 10 monclient: _finish_hunting 0 2024-09-16T04:07:27.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.313+0000 7f1e577fe640 1 monclient: found mon.noname-c 2024-09-16T04:07:27.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.313+0000 7f1e577fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:07:27.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.313+0000 7f1e577fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.154:3300/0 2024-09-16T04:07:27.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.313+0000 7f1e577fe640 1 -- 172.21.15.42:0/1848860569 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e3800b050 con 0x7f1e38007420 2024-09-16T04:07:27.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.317+0000 7f1e577fe640 1 --2- 172.21.15.42:0/1848860569 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1e38007420 0x7f1e380098e0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f1e48001660 tx=0x7f1e4801cd20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4ed621b5dd273813 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:07:27.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.317+0000 7f1e56ffd640 1 -- 172.21.15.42:0/1848860569 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e48006b60 con 0x7f1e38007420 2024-09-16T04:07:27.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.317+0000 7f1e56ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:27.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.317+0000 7f1e56ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:07:27.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.317+0000 7f1e56ffd640 10 monclient: dump: 2024-09-16T04:07:27.325 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:07:27.325 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:27.325 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:27.325 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:27.325 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:27.325 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:27.325 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:07:27.325 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:07:27.325 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:07:27.326 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:27.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.317+0000 7f1e56ffd640 10 monclient: _finish_auth 0 2024-09-16T04:07:27.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.317+0000 7f1e56ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:07:27.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.317+0000 7f1e56ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:07:27.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.317+0000 7f1e56ffd640 1 -- 172.21.15.42:0/1848860569 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e48006d00 con 0x7f1e38007420 2024-09-16T04:07:27.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.317+0000 7f1e56ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:07:27.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.317+0000 7f1e56ffd640 1 -- 172.21.15.42:0/1848860569 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e48003b80 con 0x7f1e38007420 2024-09-16T04:07:27.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.317+0000 7f1e56ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:27.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.317+0000 7f1e56ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:07:27.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.317+0000 7f1e56ffd640 10 monclient: dump: 2024-09-16T04:07:27.326 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:07:27.326 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:27.326 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:27.326 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:27.327 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:27.327 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:27.327 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:07:27.327 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:07:27.327 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:07:27.327 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:27.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.317+0000 7f1e5e5b2640 5 monclient: authenticate success, global_id 905071 2024-09-16T04:07:27.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.317+0000 7f1e5e5b2640 10 monclient: get_monmap_and_config success 2024-09-16T04:07:27.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.317+0000 7f1e5e5b2640 4 set_mon_vals no callback set 2024-09-16T04:07:27.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.321+0000 7f1e5e5b2640 10 monclient: shutdown 2024-09-16T04:07:27.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.321+0000 7f1e5e5b2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:07:27.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.321+0000 7f1e5e5b2640 1 -- 172.21.15.42:0/1848860569 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1e38007420 msgr2=0x7f1e380098e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:27.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.321+0000 7f1e5e5b2640 1 --2- 172.21.15.42:0/1848860569 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1e38007420 0x7f1e380098e0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f1e48001660 tx=0x7f1e4801cd20 comp rx=0 tx=0).stop 2024-09-16T04:07:27.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.321+0000 7f1e5cb4f640 1 -- 172.21.15.42:0/1848860569 reap_dead start 2024-09-16T04:07:27.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.321+0000 7f1e5e5b2640 1 -- 172.21.15.42:0/1848860569 shutdown_connections 2024-09-16T04:07:27.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.321+0000 7f1e5e5b2640 1 --2- 172.21.15.42:0/1848860569 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1e38007420 0x7f1e380098e0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:27.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.321+0000 7f1e5e5b2640 1 -- 172.21.15.42:0/1848860569 >> 172.21.15.42:0/1848860569 conn(0x7f1e580a9fb0 msgr2=0x7f1e580aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:07:27.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.321+0000 7f1e5e5b2640 1 -- 172.21.15.42:0/1848860569 shutdown_connections 2024-09-16T04:07:27.329 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.321+0000 7f1e5e5b2640 1 -- 172.21.15.42:0/1848860569 wait complete. 2024-09-16T04:07:27.329 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.321+0000 7f1e5e5b2640 10 monclient: build_initial_monmap 2024-09-16T04:07:27.329 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.321+0000 7f1e5e5b2640 10 monclient: monmap: 2024-09-16T04:07:27.329 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:07:27.329 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:07:27.329 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:07:27.329 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:07:27.329 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:07:27.329 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:27.329 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:07:27.329 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:07:27.329 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:07:27.329 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:27.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.321+0000 7f1e5e5b2640 1 librados: starting msgr at 2024-09-16T04:07:27.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.321+0000 7f1e5e5b2640 1 librados: starting objecter 2024-09-16T04:07:27.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.321+0000 7f1e5e5b2640 1 Processor -- start 2024-09-16T04:07:27.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.321+0000 7f1e5e5b2640 1 -- start start 2024-09-16T04:07:27.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.321+0000 7f1e5e5b2640 1 librados: setting wanted keys 2024-09-16T04:07:27.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.321+0000 7f1e5e5b2640 1 librados: calling monclient init 2024-09-16T04:07:27.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.321+0000 7f1e5e5b2640 10 monclient: init 2024-09-16T04:07:27.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.325+0000 7f1e5e5b2640 10 monclient: _reopen_session rank -1 2024-09-16T04:07:27.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.325+0000 7f1e5e5b2640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:07:27.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.325+0000 7f1e5e5b2640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1e38007420 0x7f1e581cedc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:27.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.325+0000 7f1e5e5b2640 10 monclient(hunting): picked mon.noname-a con 0x7f1e38007420 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:07:27.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.325+0000 7f1e5e5b2640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1e5813e4f0 0x7f1e5813e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:27.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.325+0000 7f1e5e5b2640 10 monclient(hunting): picked mon.noname-b con 0x7f1e5813e4f0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:07:27.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.325+0000 7f1e57fff640 1 -- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1e38007420 msgr2=0x7f1e581cedc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.42:3300/0 2024-09-16T04:07:27.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.325+0000 7f1e57fff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1e38007420 0x7f1e581cedc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:07:27.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.325+0000 7f1e5e5b2640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1e580a18b0 0x7f1e580a1d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:27.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.325+0000 7f1e5e5b2640 10 monclient(hunting): picked mon.noname-c con 0x7f1e580a18b0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:07:27.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.325+0000 7f1e5e5b2640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:27.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.325+0000 7f1e5e5b2640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f1e580a40d0 con 0x7f1e38007420 2024-09-16T04:07:27.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.325+0000 7f1e5e5b2640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:27.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.325+0000 7f1e5e5b2640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f1e58137c70 con 0x7f1e5813e4f0 2024-09-16T04:07:27.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.325+0000 7f1e5e5b2640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:27.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.325+0000 7f1e5e5b2640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f1e581419d0 con 0x7f1e580a18b0 2024-09-16T04:07:27.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.325+0000 7f1e5e5b2640 10 monclient(hunting): _renew_subs 2024-09-16T04:07:27.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.325+0000 7f1e5e5b2640 10 monclient(hunting): authenticate will time out at 2494.297197s 2024-09-16T04:07:27.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.325+0000 7f1e5cb4f640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1e580a18b0 0x7f1e580a1d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:27.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.325+0000 7f1e5cb4f640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1e580a18b0 0x7f1e580a1d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:34726/0 (socket says 172.21.15.42:34726) 2024-09-16T04:07:27.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.325+0000 7f1e5cb4f640 1 -- 172.21.15.42:0/2908687044 learned_addr learned my addr 172.21.15.42:0/2908687044 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:07:27.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.325+0000 7f1e5cb4f640 10 monclient(hunting): get_auth_request con 0x7f1e580a18b0 auth_method 0 2024-09-16T04:07:27.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.325+0000 7f1e5cb4f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:27.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.325+0000 7f1e5cb4f640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:27.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.325+0000 7f1e5cb4f640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:27.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.325+0000 7f1e577fe640 1 --2- 172.21.15.42:0/2908687044 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1e5813e4f0 0x7f1e5813e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:27.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.325+0000 7f1e5cb4f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:07:27.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.325+0000 7f1e5cb4f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:07:27.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.325+0000 7f1e5cb4f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:07:27.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.325+0000 7f1e577fe640 10 monclient(hunting): get_auth_request con 0x7f1e5813e4f0 auth_method 0 2024-09-16T04:07:27.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.325+0000 7f1e577fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:27.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.325+0000 7f1e577fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:27.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.325+0000 7f1e577fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:27.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.325+0000 7f1e577fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:07:27.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.325+0000 7f1e577fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:07:27.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.325+0000 7f1e577fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:07:27.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.325+0000 7f1e5cb4f640 10 monclient(hunting): handle_auth_done global_id 1014161 payload 824 2024-09-16T04:07:27.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.325+0000 7f1e5cb4f640 1 -- 172.21.15.42:0/2908687044 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1e5813e4f0 msgr2=0x7f1e5813e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:27.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.325+0000 7f1e5cb4f640 1 --2- 172.21.15.42:0/2908687044 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1e5813e4f0 0x7f1e5813e970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:27.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.325+0000 7f1e5cb4f640 1 -- 172.21.15.42:0/2908687044 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1e38007420 msgr2=0x7f1e581cedc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:07:27.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.325+0000 7f1e5cb4f640 1 --2- 172.21.15.42:0/2908687044 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1e38007420 0x7f1e581cedc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:27.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.325+0000 7f1e5cb4f640 10 monclient: _finish_hunting 0 2024-09-16T04:07:27.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.325+0000 7f1e5cb4f640 1 monclient: found mon.noname-c 2024-09-16T04:07:27.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.325+0000 7f1e5cb4f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:07:27.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.325+0000 7f1e5cb4f640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T04:07:27.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.325+0000 7f1e5cb4f640 1 -- 172.21.15.42:0/2908687044 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e58138580 con 0x7f1e580a18b0 2024-09-16T04:07:27.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.325+0000 7f1e5cb4f640 1 --2- 172.21.15.42:0/2908687044 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1e580a18b0 0x7f1e580a1d30 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f1e50009100 tx=0x7f1e50014a30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:07:27.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.325+0000 7f1e54ff9640 10 client.?.objecter ms_handle_connect 0x7f1e580a18b0 2024-09-16T04:07:27.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.325+0000 7f1e54ff9640 10 client.?.objecter resend_mon_ops 2024-09-16T04:07:27.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.325+0000 7f1e54ff9640 1 -- 172.21.15.42:0/2908687044 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e5001d070 con 0x7f1e580a18b0 2024-09-16T04:07:27.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.325+0000 7f1e54ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:27.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.325+0000 7f1e54ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:07:27.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.325+0000 7f1e54ff9640 10 monclient: dump: 2024-09-16T04:07:27.337 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:07:27.337 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:27.337 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:27.337 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:27.337 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:27.337 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:27.337 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:07:27.337 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:07:27.337 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:07:27.337 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:27.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.325+0000 7f1e54ff9640 10 monclient: _finish_auth 0 2024-09-16T04:07:27.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.325+0000 7f1e54ff9640 10 monclient: _check_auth_tickets 2024-09-16T04:07:27.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.325+0000 7f1e54ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:07:27.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.325+0000 7f1e54ff9640 1 -- 172.21.15.42:0/2908687044 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e500168a0 con 0x7f1e580a18b0 2024-09-16T04:07:27.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.325+0000 7f1e54ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T04:07:27.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.325+0000 7f1e54ff9640 1 -- 172.21.15.42:0/2908687044 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e5000f040 con 0x7f1e580a18b0 2024-09-16T04:07:27.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.325+0000 7f1e5e5b2640 5 monclient: authenticate success, global_id 1014161 2024-09-16T04:07:27.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.325+0000 7f1e5e5b2640 10 monclient: _renew_subs 2024-09-16T04:07:27.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.325+0000 7f1e5e5b2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:07:27.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.325+0000 7f1e567fc640 4 set_mon_vals no callback set 2024-09-16T04:07:27.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.325+0000 7f1e5e5b2640 1 -- 172.21.15.42:0/2908687044 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e581cf300 con 0x7f1e580a18b0 2024-09-16T04:07:27.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.325+0000 7f1e54ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:27.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.325+0000 7f1e54ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:07:27.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.325+0000 7f1e54ff9640 10 monclient: dump: 2024-09-16T04:07:27.338 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:07:27.338 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:27.338 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:27.338 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:27.339 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:27.339 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:27.339 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:07:27.339 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:07:27.339 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:07:27.339 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:27.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.325+0000 7f1e5e5b2640 10 client.1014161.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:07:27.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.325+0000 7f1e5e5b2640 10 monclient: _renew_subs 2024-09-16T04:07:27.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.325+0000 7f1e5e5b2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:07:27.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.325+0000 7f1e5e5b2640 1 -- 172.21.15.42:0/2908687044 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e581396f0 con 0x7f1e580a18b0 2024-09-16T04:07:27.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.325+0000 7f1e5e5b2640 1 librados: init done 2024-09-16T04:07:27.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.325+0000 7f1e54ff9640 1 -- 172.21.15.42:0/2908687044 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f1e50020070 con 0x7f1e580a18b0 2024-09-16T04:07:27.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.325+0000 7f1e54ff9640 1 --2- 172.21.15.42:0/2908687044 >> v2:172.21.15.154:6812/254845342 conn(0x7f1e38038730 0x7f1e3803abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:27.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.325+0000 7f1e54ff9640 1 -- 172.21.15.42:0/2908687044 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(706..706 src has 451..706) ==== 5040+0+0 (secure 0 0 0) 0x7f1e500491a0 con 0x7f1e580a18b0 2024-09-16T04:07:27.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.325+0000 7f1e54ff9640 10 client.1014161.objecter ms_dispatch 0x7f1e58000ba0 osd_map(706..706 src has 451..706) 2024-09-16T04:07:27.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.325+0000 7f1e54ff9640 3 client.1014161.objecter handle_osd_map got epochs [706,706] > 0 2024-09-16T04:07:27.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.325+0000 7f1e54ff9640 3 client.1014161.objecter handle_osd_map decoding full epoch 706 2024-09-16T04:07:27.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.329+0000 7f1e5e5b2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:07:27.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.329+0000 7f1e5e5b2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:07:27.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.329+0000 7f1e5e5b2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:07:27.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.329+0000 7f1e5e5b2640 1 -- 172.21.15.42:0/2908687044 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1e581340c0 con 0x7f1e580a18b0 2024-09-16T04:07:27.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.329+0000 7f1e54ff9640 20 client.1014161.objecter dump_active .. 0 homeless 2024-09-16T04:07:27.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.329+0000 7f1e57fff640 1 --2- 172.21.15.42:0/2908687044 >> v2:172.21.15.154:6812/254845342 conn(0x7f1e38038730 0x7f1e3803abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:27.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.329+0000 7f1e57fff640 10 monclient: get_auth_request con 0x7f1e38038730 auth_method 0 2024-09-16T04:07:27.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.329+0000 7f1e57fff640 1 --2- 172.21.15.42:0/2908687044 >> v2:172.21.15.154:6812/254845342 conn(0x7f1e38038730 0x7f1e3803abf0 secure :-1 s=READY pgs=3882 cs=0 l=1 rev1=1 crypto rx=0x7f1e40003990 tx=0x7f1e4000a390 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:07:27.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.329+0000 7f1e54ff9640 10 client.1014161.objecter ms_handle_connect 0x7f1e38038730 2024-09-16T04:07:27.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.333+0000 7f1e54ff9640 1 -- 172.21.15.42:0/2908687044 <== mon.2 v2:172.21.15.42:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f1e50027020 con 0x7f1e580a18b0 2024-09-16T04:07:27.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.333+0000 7f1e54ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:07:27.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.333+0000 7f1e54ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:07:27.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.581+0000 7f1e5e5b2640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:07:27.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.581+0000 7f1e5e5b2640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:07:27.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.581+0000 7f1e5e5b2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:07:27.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.581+0000 7f1e5e5b2640 1 -- 172.21.15.42:0/2908687044 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f1e58133500 con 0x7f1e580a18b0 2024-09-16T04:07:27.588 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.581+0000 7f1e54ff9640 1 -- 172.21.15.42:0/2908687044 <== mon.2 v2:172.21.15.42:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1555 (secure 0 0 0) 0x7f1e50017470 con 0x7f1e580a18b0 2024-09-16T04:07:27.588 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.581+0000 7f1e54ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:07:27.588 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.581+0000 7f1e54ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:07:27.588 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":234,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:07:27.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.585+0000 7f1e5e5b2640 10 librados: watch_flush enter 2024-09-16T04:07:27.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.585+0000 7f1e5e5b2640 10 librados: watch_flush exit 2024-09-16T04:07:27.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.585+0000 7f1e5e5b2640 20 client.1014161.objecter shutdown clearing up homeless session... 2024-09-16T04:07:27.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.585+0000 7f1e5e5b2640 10 client.1014161.objecter successfully canceled tick 2024-09-16T04:07:27.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.585+0000 7f1e5e5b2640 1 -- 172.21.15.42:0/2908687044 >> v2:172.21.15.154:6812/254845342 conn(0x7f1e38038730 msgr2=0x7f1e3803abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:27.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.585+0000 7f1e5e5b2640 1 --2- 172.21.15.42:0/2908687044 >> v2:172.21.15.154:6812/254845342 conn(0x7f1e38038730 0x7f1e3803abf0 secure :-1 s=READY pgs=3882 cs=0 l=1 rev1=1 crypto rx=0x7f1e40003990 tx=0x7f1e4000a390 comp rx=0 tx=0).stop 2024-09-16T04:07:27.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.585+0000 7f1e5e5b2640 10 monclient: shutdown 2024-09-16T04:07:27.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.585+0000 7f1e5e5b2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:07:27.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.585+0000 7f1e5e5b2640 1 -- 172.21.15.42:0/2908687044 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1e580a18b0 msgr2=0x7f1e580a1d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:27.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.585+0000 7f1e5e5b2640 1 --2- 172.21.15.42:0/2908687044 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1e580a18b0 0x7f1e580a1d30 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f1e50009100 tx=0x7f1e50014a30 comp rx=0 tx=0).stop 2024-09-16T04:07:27.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.585+0000 7f1e5e5b2640 1 -- 172.21.15.42:0/2908687044 shutdown_connections 2024-09-16T04:07:27.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.585+0000 7f1e5e5b2640 1 --2- 172.21.15.42:0/2908687044 >> v2:172.21.15.154:6812/254845342 conn(0x7f1e38038730 0x7f1e3803abf0 unknown :-1 s=CLOSED pgs=3882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:27.592 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.585+0000 7f1e5e5b2640 1 --2- 172.21.15.42:0/2908687044 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1e580a18b0 0x7f1e580a1d30 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:27.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.585+0000 7f1e5e5b2640 1 --2- 172.21.15.42:0/2908687044 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1e5813e4f0 0x7f1e5813e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:27.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.585+0000 7f1e5e5b2640 1 --2- 172.21.15.42:0/2908687044 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1e38007420 0x7f1e581cedc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:27.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.585+0000 7f1e5e5b2640 1 -- 172.21.15.42:0/2908687044 >> 172.21.15.42:0/2908687044 conn(0x7f1e580a9fb0 msgr2=0x7f1e580a89e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:07:27.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.585+0000 7f1e5e5b2640 1 -- 172.21.15.42:0/2908687044 shutdown_connections 2024-09-16T04:07:27.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.585+0000 7f1e5e5b2640 1 -- 172.21.15.42:0/2908687044 wait complete. 2024-09-16T04:07:27.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.585+0000 7f1e5e5b2640 1 librados: shutdown 2024-09-16T04:07:27.608 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T04:07:27.608 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-16T04:07:27.608 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T04:07:27.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.725+0000 7f5dbda19640 10 monclient: get_monmap_and_config 2024-09-16T04:07:27.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.725+0000 7f5dbda19640 10 monclient: build_initial_monmap 2024-09-16T04:07:27.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.725+0000 7f5dbda19640 10 monclient: monmap: 2024-09-16T04:07:27.730 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:07:27.730 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:27.730 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:07:27.728789+0000 2024-09-16T04:07:27.731 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:07:27.728789+0000 2024-09-16T04:07:27.731 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:07:27.731 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:27.731 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:07:27.731 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:07:27.731 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:07:27.731 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:07:27.731 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:07:27.731 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:27.733 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.725+0000 7f5dbda19640 1 Processor -- start 2024-09-16T04:07:27.733 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.725+0000 7f5dbda19640 1 -- start start 2024-09-16T04:07:27.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.725+0000 7f5dbda19640 10 monclient: init 2024-09-16T04:07:27.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.725+0000 7f5dbda19640 10 monclient: _reopen_session rank -1 2024-09-16T04:07:27.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.725+0000 7f5dbda19640 10 monclient: _add_conns ranks=[4,1,3,0,2] 2024-09-16T04:07:27.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.725+0000 7f5dbda19640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f5db809fb00 0x7f5db809ba50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:27.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.725+0000 7f5dbda19640 10 monclient(hunting): picked mon.noname-c con 0x7f5db809fb00 addr v2:172.21.15.154:3300/0 2024-09-16T04:07:27.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.725+0000 7f5dbda19640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5db809bf90 addr v1:172.21.15.154:6789/0 2024-09-16T04:07:27.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.725+0000 7f5dbda19640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f5db809c400 0x7f5db813b840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:27.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.725+0000 7f5dbda19640 10 monclient(hunting): picked mon.noname-a con 0x7f5db809c400 addr v2:172.21.15.42:3300/0 2024-09-16T04:07:27.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.725+0000 7f5dbda19640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5db80a1e20 con 0x7f5db809bf90 2024-09-16T04:07:27.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.725+0000 7f5dbda19640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:27.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.725+0000 7f5dbda19640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f5db809e250 con 0x7f5db809c400 2024-09-16T04:07:27.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.725+0000 7f5dbda19640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:27.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.725+0000 7f5dbda19640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f5db809c850 con 0x7f5db809fb00 2024-09-16T04:07:27.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.725+0000 7f5dbda19640 10 monclient(hunting): _renew_subs 2024-09-16T04:07:27.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.725+0000 7f5dbda19640 10 monclient(hunting): authenticate will time out at 2494.698195s 2024-09-16T04:07:27.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.725+0000 7f5db77fe640 1 -- >> v2:172.21.15.42:3300/0 conn(0x7f5db809c400 msgr2=0x7f5db813b840 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3300/0 2024-09-16T04:07:27.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.725+0000 7f5db77fe640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f5db809c400 0x7f5db813b840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:07:27.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.725+0000 7f5db6ffd640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f5db809fb00 0x7f5db809ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:27.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.725+0000 7f5db6ffd640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f5db809fb00 0x7f5db809ba50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.42:60140/0 (socket says 172.21.15.42:60140) 2024-09-16T04:07:27.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.725+0000 7f5db6ffd640 1 -- 172.21.15.42:0/2450389674 learned_addr learned my addr 172.21.15.42:0/2450389674 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:07:27.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.725+0000 7f5db6ffd640 10 monclient(hunting): get_auth_request con 0x7f5db809fb00 auth_method 0 2024-09-16T04:07:27.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.725+0000 7f5db6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:27.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.725+0000 7f5db6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:27.736 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.725+0000 7f5db6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:27.736 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.725+0000 7f5db67fc640 1 --1- 172.21.15.42:0/2450389674 >> v1:172.21.15.154:6789/0 conn(0x7f5db809bf90 0x7f5db8138110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.42:41720/0 (socket says 172.21.15.42:41720) 2024-09-16T04:07:27.736 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.725+0000 7f5db6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:07:27.736 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.725+0000 7f5db6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:07:27.736 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.725+0000 7f5db6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:07:27.736 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.725+0000 7f5db6ffd640 10 monclient(hunting): handle_auth_done global_id 905077 payload 274 2024-09-16T04:07:27.736 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.725+0000 7f5db6ffd640 1 -- 172.21.15.42:0/2450389674 >> v2:172.21.15.42:3300/0 conn(0x7f5db809c400 msgr2=0x7f5db813b840 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:07:27.736 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.725+0000 7f5db6ffd640 1 --2- 172.21.15.42:0/2450389674 >> v2:172.21.15.42:3300/0 conn(0x7f5db809c400 0x7f5db813b840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:27.736 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.725+0000 7f5db6ffd640 1 -- 172.21.15.42:0/2450389674 >> v1:172.21.15.154:6789/0 conn(0x7f5db809bf90 legacy=0x7f5db8138110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:27.736 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.725+0000 7f5db6ffd640 10 monclient: _finish_hunting 0 2024-09-16T04:07:27.736 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.725+0000 7f5db6ffd640 1 monclient: found mon.noname-c 2024-09-16T04:07:27.737 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.725+0000 7f5db5ffb640 1 -- 172.21.15.42:0/2450389674 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2017363057 0 0) 0x7f5db80a1e20 con 0x7f5db809bf90 2024-09-16T04:07:27.737 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.725+0000 7f5db6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:07:27.737 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.725+0000 7f5db6ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.154:3300/0 2024-09-16T04:07:27.737 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.725+0000 7f5db6ffd640 1 -- 172.21.15.42:0/2450389674 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5db809ea30 con 0x7f5db809fb00 2024-09-16T04:07:27.737 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.725+0000 7f5db5ffb640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T04:07:27.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.729+0000 7f5db6ffd640 1 --2- 172.21.15.42:0/2450389674 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5db809fb00 0x7f5db809ba50 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f5db80a68e0 tx=0x7f5da801c400 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7b5f4dd06e5ff690 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:07:27.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.729+0000 7f5db5ffb640 1 -- 172.21.15.42:0/2450389674 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5da80b2070 con 0x7f5db809fb00 2024-09-16T04:07:27.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.729+0000 7f5db5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:27.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.729+0000 7f5db5ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:07:27.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.729+0000 7f5db5ffb640 10 monclient: dump: 2024-09-16T04:07:27.746 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:07:27.746 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:27.746 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:27.746 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:27.746 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:27.746 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:27.746 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:07:27.746 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:07:27.746 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:07:27.747 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:27.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.729+0000 7f5db5ffb640 10 monclient: _finish_auth 0 2024-09-16T04:07:27.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.729+0000 7f5db5ffb640 10 monclient: _check_auth_tickets 2024-09-16T04:07:27.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.729+0000 7f5db5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:07:27.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.729+0000 7f5dbda19640 5 monclient: authenticate success, global_id 905077 2024-09-16T04:07:27.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.729+0000 7f5dbda19640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:07:27.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.729+0000 7f5db5ffb640 1 -- 172.21.15.42:0/2450389674 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5da80ad680 con 0x7f5db809fb00 2024-09-16T04:07:27.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.729+0000 7f5db5ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:07:27.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.729+0000 7f5db5ffb640 1 -- 172.21.15.42:0/2450389674 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5da80ada50 con 0x7f5db809fb00 2024-09-16T04:07:27.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.729+0000 7f5dbda19640 10 monclient: get_monmap_and_config success 2024-09-16T04:07:27.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.729+0000 7f5dbda19640 4 set_mon_vals no callback set 2024-09-16T04:07:27.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.729+0000 7f5db5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:27.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.729+0000 7f5db5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:07:27.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.729+0000 7f5db5ffb640 10 monclient: dump: 2024-09-16T04:07:27.748 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:07:27.748 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:27.748 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:27.748 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:27.748 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:27.748 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:27.748 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:07:27.748 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:07:27.748 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:07:27.749 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:27.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.729+0000 7f5dbda19640 10 monclient: shutdown 2024-09-16T04:07:27.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.729+0000 7f5dbda19640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:07:27.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.729+0000 7f5dbda19640 1 -- 172.21.15.42:0/2450389674 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5db809fb00 msgr2=0x7f5db809ba50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:27.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.729+0000 7f5dbda19640 1 --2- 172.21.15.42:0/2450389674 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5db809fb00 0x7f5db809ba50 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f5db80a68e0 tx=0x7f5da801c400 comp rx=0 tx=0).stop 2024-09-16T04:07:27.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.729+0000 7f5dbda19640 1 -- 172.21.15.42:0/2450389674 shutdown_connections 2024-09-16T04:07:27.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.729+0000 7f5dbda19640 1 --2- 172.21.15.42:0/2450389674 >> v2:172.21.15.42:3300/0 conn(0x7f5db809c400 0x7f5db813b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:27.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.729+0000 7f5dbda19640 1 --2- 172.21.15.42:0/2450389674 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5db809fb00 0x7f5db809ba50 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:27.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.729+0000 7f5dbda19640 1 -- 172.21.15.42:0/2450389674 >> 172.21.15.42:0/2450389674 conn(0x7f5db80a2420 msgr2=0x7f5db80a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:07:27.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.729+0000 7f5dbda19640 1 -- 172.21.15.42:0/2450389674 shutdown_connections 2024-09-16T04:07:27.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.729+0000 7f5dbda19640 1 -- 172.21.15.42:0/2450389674 wait complete. 2024-09-16T04:07:27.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.729+0000 7f5dbda19640 10 monclient: build_initial_monmap 2024-09-16T04:07:27.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.729+0000 7f5dbda19640 10 monclient: monmap: 2024-09-16T04:07:27.750 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:07:27.750 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:07:27.750 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:07:27.750 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:07:27.750 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:07:27.750 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:27.750 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:07:27.750 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:07:27.750 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:07:27.750 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:27.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.729+0000 7f5dbda19640 1 librados: starting msgr at 2024-09-16T04:07:27.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.729+0000 7f5dbda19640 1 librados: starting objecter 2024-09-16T04:07:27.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.729+0000 7f5dbda19640 1 Processor -- start 2024-09-16T04:07:27.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.729+0000 7f5dbda19640 1 -- start start 2024-09-16T04:07:27.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.729+0000 7f5dbda19640 1 librados: setting wanted keys 2024-09-16T04:07:27.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.729+0000 7f5dbda19640 1 librados: calling monclient init 2024-09-16T04:07:27.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.729+0000 7f5dbda19640 10 monclient: init 2024-09-16T04:07:27.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.733+0000 7f5dbda19640 10 monclient: _reopen_session rank -1 2024-09-16T04:07:27.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.733+0000 7f5dbda19640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:07:27.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.733+0000 7f5dbda19640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5db809bf90 0x7f5db81cedd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:27.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.733+0000 7f5dbda19640 10 monclient(hunting): picked mon.noname-b con 0x7f5db809bf90 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:07:27.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.733+0000 7f5dbda19640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5db809c400 0x7f5db81353e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:27.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.733+0000 7f5dbda19640 10 monclient(hunting): picked mon.noname-a con 0x7f5db809c400 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:07:27.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.733+0000 7f5dbda19640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5db809fb00 0x7f5db8135920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:27.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.733+0000 7f5dbda19640 10 monclient(hunting): picked mon.noname-c con 0x7f5db809fb00 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:07:27.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.733+0000 7f5dbda19640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:27.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.733+0000 7f5dbda19640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f5db80a32f0 con 0x7f5db809c400 2024-09-16T04:07:27.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.733+0000 7f5dbda19640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:27.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.733+0000 7f5dbda19640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f5db80a3170 con 0x7f5db809bf90 2024-09-16T04:07:27.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.733+0000 7f5dbda19640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:27.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.733+0000 7f5dbda19640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f5db809d380 con 0x7f5db809fb00 2024-09-16T04:07:27.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.733+0000 7f5dbda19640 10 monclient(hunting): _renew_subs 2024-09-16T04:07:27.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.733+0000 7f5dbda19640 10 monclient(hunting): authenticate will time out at 2494.704283s 2024-09-16T04:07:27.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.733+0000 7f5db67fc640 1 -- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5db809c400 msgr2=0x7f5db81353e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.42:3300/0 2024-09-16T04:07:27.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.733+0000 7f5db67fc640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5db809c400 0x7f5db81353e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:07:27.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.733+0000 7f5db77fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5db809fb00 0x7f5db8135920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:27.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.733+0000 7f5db77fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5db809fb00 0x7f5db8135920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:34734/0 (socket says 172.21.15.42:34734) 2024-09-16T04:07:27.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.733+0000 7f5db77fe640 1 -- 172.21.15.42:0/1176129390 learned_addr learned my addr 172.21.15.42:0/1176129390 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:07:27.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.733+0000 7f5db77fe640 10 monclient(hunting): get_auth_request con 0x7f5db809fb00 auth_method 0 2024-09-16T04:07:27.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.733+0000 7f5db77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:27.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.733+0000 7f5db77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:27.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.733+0000 7f5db77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:27.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.733+0000 7f5db77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:07:27.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.733+0000 7f5db77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:07:27.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.733+0000 7f5db77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:07:27.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.733+0000 7f5db6ffd640 1 --2- 172.21.15.42:0/1176129390 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5db809bf90 0x7f5db81cedd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:27.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.733+0000 7f5db6ffd640 10 monclient(hunting): get_auth_request con 0x7f5db809bf90 auth_method 0 2024-09-16T04:07:27.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.733+0000 7f5db6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:27.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.733+0000 7f5db6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:27.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.733+0000 7f5db6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:27.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.733+0000 7f5db77fe640 10 monclient(hunting): handle_auth_done global_id 1014164 payload 824 2024-09-16T04:07:27.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.733+0000 7f5db77fe640 1 -- 172.21.15.42:0/1176129390 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5db809bf90 msgr2=0x7f5db81cedd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:27.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.733+0000 7f5db77fe640 1 --2- 172.21.15.42:0/1176129390 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5db809bf90 0x7f5db81cedd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:27.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.733+0000 7f5db77fe640 1 -- 172.21.15.42:0/1176129390 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5db809c400 msgr2=0x7f5db81353e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:07:27.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.733+0000 7f5db77fe640 1 --2- 172.21.15.42:0/1176129390 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5db809c400 0x7f5db81353e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:27.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.733+0000 7f5db77fe640 10 monclient: _finish_hunting 0 2024-09-16T04:07:27.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.733+0000 7f5db77fe640 1 monclient: found mon.noname-c 2024-09-16T04:07:27.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.733+0000 7f5db77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:07:27.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.733+0000 7f5db77fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T04:07:27.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.733+0000 7f5db77fe640 1 -- 172.21.15.42:0/1176129390 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5db8071cf0 con 0x7f5db809fb00 2024-09-16T04:07:27.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.733+0000 7f5db6ffd640 1 --2- 172.21.15.42:0/1176129390 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5db809bf90 0x7f5db81cedd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T04:07:27.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.733+0000 7f5db77fe640 1 --2- 172.21.15.42:0/1176129390 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5db809fb00 0x7f5db8135920 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f5db000be60 tx=0x7f5db000bf50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:07:27.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.733+0000 7f5d9ffff640 10 client.?.objecter ms_handle_connect 0x7f5db809fb00 2024-09-16T04:07:27.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.733+0000 7f5d9ffff640 10 client.?.objecter resend_mon_ops 2024-09-16T04:07:27.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.733+0000 7f5d9ffff640 1 -- 172.21.15.42:0/1176129390 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5db0016430 con 0x7f5db809fb00 2024-09-16T04:07:27.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.733+0000 7f5d9ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:27.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.733+0000 7f5d9ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:07:27.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.733+0000 7f5d9ffff640 10 monclient: dump: 2024-09-16T04:07:27.754 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:07:27.755 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:27.755 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:27.755 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:27.755 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:27.755 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:27.755 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:07:27.755 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:07:27.755 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:07:27.755 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:27.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.733+0000 7f5d9ffff640 10 monclient: _finish_auth 0 2024-09-16T04:07:27.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.733+0000 7f5d9ffff640 10 monclient: _check_auth_tickets 2024-09-16T04:07:27.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.733+0000 7f5d9ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:07:27.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.733+0000 7f5d9ffff640 1 -- 172.21.15.42:0/1176129390 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5db0010030 con 0x7f5db809fb00 2024-09-16T04:07:27.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.733+0000 7f5dbda19640 5 monclient: authenticate success, global_id 1014164 2024-09-16T04:07:27.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.733+0000 7f5dbda19640 10 monclient: _renew_subs 2024-09-16T04:07:27.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.733+0000 7f5dbda19640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:07:27.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.733+0000 7f5dbda19640 1 -- 172.21.15.42:0/1176129390 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5db8070150 con 0x7f5db809fb00 2024-09-16T04:07:27.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.733+0000 7f5d9ffff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:07:27.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.733+0000 7f5d9ffff640 1 -- 172.21.15.42:0/1176129390 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5db0016850 con 0x7f5db809fb00 2024-09-16T04:07:27.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.733+0000 7f5d9ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:27.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.733+0000 7f5d9ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:07:27.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.733+0000 7f5dbda19640 10 client.1014164.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:07:27.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.733+0000 7f5d9ffff640 10 monclient: dump: 2024-09-16T04:07:27.756 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:07:27.756 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:27.756 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:27.756 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:27.756 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:27.757 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:27.757 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:07:27.757 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:07:27.757 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:07:27.757 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:27.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.733+0000 7f5db4ff9640 4 set_mon_vals no callback set 2024-09-16T04:07:27.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.733+0000 7f5dbda19640 10 monclient: _renew_subs 2024-09-16T04:07:27.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.733+0000 7f5dbda19640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:07:27.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.733+0000 7f5dbda19640 1 -- 172.21.15.42:0/1176129390 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f5db81217d0 con 0x7f5db809fb00 2024-09-16T04:07:27.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.733+0000 7f5dbda19640 1 librados: init done 2024-09-16T04:07:27.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.733+0000 7f5dbda19640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:07:27.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.733+0000 7f5dbda19640 1 --2- 172.21.15.42:0/1176129390 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5db806d900 0x7f5db806dce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:27.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.733+0000 7f5dbda19640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:27.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.733+0000 7f5dbda19640 1 -- 172.21.15.42:0/1176129390 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f5db809ba50 con 0x7f5db806d900 2024-09-16T04:07:27.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.733+0000 7f5db6ffd640 1 --2- 172.21.15.42:0/1176129390 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5db806d900 0x7f5db806dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:27.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.733+0000 7f5d9ffff640 1 -- 172.21.15.42:0/1176129390 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f5db00153e0 con 0x7f5db809fb00 2024-09-16T04:07:27.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.733+0000 7f5db6ffd640 10 monclient: get_auth_request con 0x7f5db806d900 auth_method 0 2024-09-16T04:07:27.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.733+0000 7f5db6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:27.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.733+0000 7f5db6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:27.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.733+0000 7f5db6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:27.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.733+0000 7f5db6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:07:27.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.733+0000 7f5db6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:07:27.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.733+0000 7f5db6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:07:27.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.733+0000 7f5d9ffff640 1 --2- 172.21.15.42:0/1176129390 >> v2:172.21.15.154:6812/254845342 conn(0x7f5da40377c0 0x7f5da4039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:27.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.733+0000 7f5db6ffd640 10 monclient(hunting): handle_auth_done global_id 1014167 payload 824 2024-09-16T04:07:27.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.733+0000 7f5d9ffff640 1 -- 172.21.15.42:0/1176129390 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(706..706 src has 451..706) ==== 5040+0+0 (secure 0 0 0) 0x7f5db0049130 con 0x7f5db809fb00 2024-09-16T04:07:27.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.733+0000 7f5d9ffff640 10 client.1014164.objecter ms_dispatch 0x7f5db8000ba0 osd_map(706..706 src has 451..706) 2024-09-16T04:07:27.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.733+0000 7f5d9ffff640 3 client.1014164.objecter handle_osd_map got epochs [706,706] > 0 2024-09-16T04:07:27.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.733+0000 7f5d9ffff640 3 client.1014164.objecter handle_osd_map decoding full epoch 706 2024-09-16T04:07:27.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.733+0000 7f5db6ffd640 1 -- 172.21.15.42:0/1176129390 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5db80008e0 con 0x7f5db806d900 2024-09-16T04:07:27.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.733+0000 7f5d9ffff640 20 client.1014164.objecter dump_active .. 0 homeless 2024-09-16T04:07:27.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.733+0000 7f5d9ffff640 10 client.1014164.objecter ms_handle_connect 0x7f5db806d900 2024-09-16T04:07:27.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.733+0000 7f5d9ffff640 10 client.1014164.objecter resend_mon_ops 2024-09-16T04:07:27.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.733+0000 7f5db6ffd640 1 --2- 172.21.15.42:0/1176129390 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5db806d900 0x7f5db806dce0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f5db806c650 tx=0x7f5da80191a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:07:27.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.733+0000 7f5d9ffff640 1 -- 172.21.15.42:0/1176129390 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5da80bd070 con 0x7f5db806d900 2024-09-16T04:07:27.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.733+0000 7f5d9ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:27.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.733+0000 7f5d9ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:07:27.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.733+0000 7f5d9ffff640 10 monclient: dump: 2024-09-16T04:07:27.759 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:07:27.760 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:27.760 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:27.760 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:27.760 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:27.760 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:27.760 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:07:27.760 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:07:27.760 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:07:27.760 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:27.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.733+0000 7f5db67fc640 1 --2- 172.21.15.42:0/1176129390 >> v2:172.21.15.154:6812/254845342 conn(0x7f5da40377c0 0x7f5da4039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:27.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.733+0000 7f5db67fc640 10 monclient: get_auth_request con 0x7f5da40377c0 auth_method 0 2024-09-16T04:07:27.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.733+0000 7f5d9ffff640 1 -- 172.21.15.42:0/1176129390 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f5da80ad9a0 con 0x7f5db806d900 2024-09-16T04:07:27.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.733+0000 7f5d9ffff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:07:27.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.733+0000 7f5d9ffff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:07:27.761 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.733+0000 7f5d9ffff640 1 -- 172.21.15.42:0/1176129390 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5db806d900 msgr2=0x7f5db806dce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:27.761 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.733+0000 7f5d9ffff640 1 --2- 172.21.15.42:0/1176129390 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5db806d900 0x7f5db806dce0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f5db806c650 tx=0x7f5da80191a0 comp rx=0 tx=0).stop 2024-09-16T04:07:27.761 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.733+0000 7f5d9ffff640 1 -- 172.21.15.42:0/1176129390 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5db806d900 msgr2=0x7f5db806dce0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:07:27.761 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.733+0000 7f5d9ffff640 1 --2- 172.21.15.42:0/1176129390 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5db806d900 0x7f5db806dce0 secure :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f5db806c650 tx=0x7f5da80191a0 comp rx=0 tx=0).stop 2024-09-16T04:07:27.761 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.733+0000 7f5db67fc640 1 --2- 172.21.15.42:0/1176129390 >> v2:172.21.15.154:6812/254845342 conn(0x7f5da40377c0 0x7f5da4039c80 secure :-1 s=READY pgs=3883 cs=0 l=1 rev1=1 crypto rx=0x7f5db81213e0 tx=0x7f5dac007a50 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:07:27.761 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.733+0000 7f5d9ffff640 10 client.1014164.objecter ms_handle_connect 0x7f5da40377c0 2024-09-16T04:07:27.763 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.741+0000 7f5dbda19640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:07:27.763 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.741+0000 7f5dbda19640 1 --2- 172.21.15.42:0/1176129390 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5db806e220 0x7f5db813ab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:27.763 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.741+0000 7f5dbda19640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:27.763 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.741+0000 7f5dbda19640 1 -- 172.21.15.42:0/1176129390 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f5db806e630 con 0x7f5db806e220 2024-09-16T04:07:27.763 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.741+0000 7f5db6ffd640 1 --2- 172.21.15.42:0/1176129390 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5db806e220 0x7f5db813ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:27.763 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.741+0000 7f5db6ffd640 10 monclient: get_auth_request con 0x7f5db806e220 auth_method 0 2024-09-16T04:07:27.763 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.741+0000 7f5db6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:27.763 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.741+0000 7f5db6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:27.763 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.741+0000 7f5db6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:27.763 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.741+0000 7f5db6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:07:27.764 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.741+0000 7f5db6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:07:27.764 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.741+0000 7f5db6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:07:27.764 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.741+0000 7f5db6ffd640 10 monclient(hunting): handle_auth_done global_id 1014170 payload 824 2024-09-16T04:07:27.764 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.741+0000 7f5db6ffd640 1 -- 172.21.15.42:0/1176129390 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f5db8120600 con 0x7f5db806e220 2024-09-16T04:07:27.764 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.741+0000 7f5d9ffff640 10 client.1014164.objecter ms_handle_connect 0x7f5db806e220 2024-09-16T04:07:27.764 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.741+0000 7f5d9ffff640 10 client.1014164.objecter resend_mon_ops 2024-09-16T04:07:27.764 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.741+0000 7f5db6ffd640 1 --2- 172.21.15.42:0/1176129390 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5db806e220 0x7f5db813ab00 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f5da801a030 tx=0x7f5da8008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:07:27.764 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.741+0000 7f5d9ffff640 1 -- 172.21.15.42:0/1176129390 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5da80b2040 con 0x7f5db806e220 2024-09-16T04:07:27.764 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.741+0000 7f5d9ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:27.764 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.741+0000 7f5d9ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:07:27.764 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.741+0000 7f5d9ffff640 10 monclient: dump: 2024-09-16T04:07:27.764 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:07:27.765 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:27.765 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:27.765 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:27.765 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:27.765 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:27.765 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:07:27.765 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:07:27.765 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:07:27.765 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:27.765 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.741+0000 7f5d9ffff640 1 -- 172.21.15.42:0/1176129390 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4701 (secure 0 0 0) 0x7f5da8003c40 con 0x7f5db806e220 2024-09-16T04:07:27.765 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.741+0000 7f5d9ffff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:07:27.765 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.741+0000 7f5d9ffff640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:07:27.765 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.741+0000 7f5d9ffff640 1 -- 172.21.15.42:0/1176129390 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5db806e220 msgr2=0x7f5db813ab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:27.765 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.741+0000 7f5d9ffff640 1 --2- 172.21.15.42:0/1176129390 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5db806e220 0x7f5db813ab00 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f5da801a030 tx=0x7f5da8008040 comp rx=0 tx=0).stop 2024-09-16T04:07:27.766 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.741+0000 7f5d9ffff640 1 -- 172.21.15.42:0/1176129390 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5db806e220 msgr2=0x7f5db813ab00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:07:27.766 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.741+0000 7f5d9ffff640 1 --2- 172.21.15.42:0/1176129390 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5db806e220 0x7f5db813ab00 secure :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f5da801a030 tx=0x7f5da8008040 comp rx=0 tx=0).stop 2024-09-16T04:07:27.766 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T04:07:27.766 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:07:27.766 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:07:27.766 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T04:07:27.766 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 234, 2024-09-16T04:07:27.766 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 42943, 2024-09-16T04:07:27.767 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T04:07:27.767 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T04:07:27.767 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T04:07:27.767 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:07:27.767 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 0, 2024-09-16T04:07:27.767 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:07:27.767 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T04:07:27.767 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T04:07:27.767 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:07:27.767 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:07:27.767 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:07:27.767 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:07:27.767 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:07:27.768 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:07:27.768 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:07:27.768 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:07:27.768 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:07:27.768 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:07:27.768 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:07:27.768 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:07:27.768 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:07:27.768 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T04:07:27.768 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:07:27.768 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:07:27.768 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:07:27.768 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:07:27.768 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:07:27.769 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:07:27.769 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:07:27.769 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:07:27.769 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:07:27.769 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:07:27.769 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:07:27.769 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:07:27.769 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:27.769 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T04:07:27.769 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T04:07:27.769 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T04:07:27.769 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T04:07:27.769 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T04:07:27.770 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T04:07:27.770 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:07:27.770 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:07:27.770 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T04:07:27.770 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T04:07:27.770 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T04:07:27.770 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T04:07:27.770 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T04:07:27.770 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T04:07:27.770 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T04:07:27.770 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:07:27.770 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T04:07:27.770 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:07:27.771 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:07:27.771 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:07:27.771 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:07:27.771 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:07:27.771 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:07:27.771 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:07:27.771 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:07:27.771 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:07:27.771 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:07:27.771 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:07:27.771 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:07:27.771 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T04:07:27.771 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:27.771 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T04:07:27.772 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:27.772 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:07:27.772 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:07:27.772 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:07:27.772 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:07:27.772 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:27.772 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:07:27.772 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T04:07:27.772 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:07:27.772 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:27.772 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:27.772 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:07:27.773 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T04:07:27.773 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:07:27.773 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:07:27.773 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:07:27.773 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:27.773 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T04:07:27.773 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T04:07:27.773 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:07:27.773 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:07:27.773 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:07:27.773 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:27.773 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:27.773 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:07:27.774 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:07:27.774 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:07:27.774 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:07:27.774 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:27.774 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:07:27.774 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T04:07:27.774 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:07:27.774 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:27.774 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:27.774 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:07:27.774 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T04:07:27.774 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:07:27.774 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:07:27.775 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:07:27.775 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:27.775 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T04:07:27.775 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T04:07:27.775 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:07:27.775 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:07:27.775 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:07:27.775 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:27.775 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:27.775 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:07:27.775 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:07:27.775 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:07:27.775 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:07:27.775 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:27.776 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:07:27.776 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T04:07:27.776 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:07:27.776 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:27.776 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:27.776 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:07:27.776 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T04:07:27.776 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:07:27.776 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:07:27.776 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:07:27.776 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:27.776 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T04:07:27.776 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T04:07:27.776 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:07:27.777 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:07:27.777 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:07:27.777 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:07:27.777 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:07:27.777 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:27.777 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T04:07:27.777 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T04:07:27.777 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:27.777 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:07:27.777 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:07:27.778 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:07:27.778 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:07:27.778 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:07:27.778 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T04:07:27.778 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:27.778 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:07:27.778 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:07:27.778 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 3 2024-09-16T04:07:27.778 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:07:27.778 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:07:27.778 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T04:07:27.778 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:27.778 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:07:27.778 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:07:27.779 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 3 2024-09-16T04:07:27.779 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:07:27.779 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:07:27.779 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:27.779 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T04:07:27.779 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T04:07:27.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.741+0000 7f5dbda19640 10 librados: watch_flush enter 2024-09-16T04:07:27.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.741+0000 7f5dbda19640 10 librados: watch_flush exit 2024-09-16T04:07:27.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.741+0000 7f5dbda19640 20 client.1014164.objecter shutdown clearing up homeless session... 2024-09-16T04:07:27.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.741+0000 7f5dbda19640 10 client.1014164.objecter successfully canceled tick 2024-09-16T04:07:27.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.741+0000 7f5dbda19640 1 -- 172.21.15.42:0/1176129390 >> v2:172.21.15.154:6812/254845342 conn(0x7f5da40377c0 msgr2=0x7f5da4039c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:27.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.741+0000 7f5dbda19640 1 --2- 172.21.15.42:0/1176129390 >> v2:172.21.15.154:6812/254845342 conn(0x7f5da40377c0 0x7f5da4039c80 secure :-1 s=READY pgs=3883 cs=0 l=1 rev1=1 crypto rx=0x7f5db81213e0 tx=0x7f5dac007a50 comp rx=0 tx=0).stop 2024-09-16T04:07:27.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.741+0000 7f5dbda19640 10 monclient: shutdown 2024-09-16T04:07:27.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.741+0000 7f5dbda19640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:07:27.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.741+0000 7f5dbda19640 1 -- 172.21.15.42:0/1176129390 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5db809fb00 msgr2=0x7f5db8135920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:27.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.741+0000 7f5dbda19640 1 --2- 172.21.15.42:0/1176129390 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5db809fb00 0x7f5db8135920 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f5db000be60 tx=0x7f5db000bf50 comp rx=0 tx=0).stop 2024-09-16T04:07:27.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.741+0000 7f5db77fe640 1 -- 172.21.15.42:0/1176129390 reap_dead start 2024-09-16T04:07:27.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.741+0000 7f5db77fe640 1 -- 172.21.15.42:0/1176129390 reap_dead start 2024-09-16T04:07:27.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.741+0000 7f5dbda19640 1 -- 172.21.15.42:0/1176129390 shutdown_connections 2024-09-16T04:07:27.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.741+0000 7f5dbda19640 1 -- 172.21.15.42:0/1176129390 >> 172.21.15.42:0/1176129390 conn(0x7f5db80a2420 msgr2=0x7f5db8070580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:07:27.782 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.741+0000 7f5dbda19640 1 -- 172.21.15.42:0/1176129390 shutdown_connections 2024-09-16T04:07:27.782 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.741+0000 7f5dbda19640 1 -- 172.21.15.42:0/1176129390 wait complete. 2024-09-16T04:07:27.782 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.741+0000 7f5dbda19640 1 librados: shutdown 2024-09-16T04:07:27.782 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T04:07:27.782 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-16T04:07:27.782 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T04:07:27.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.893+0000 7fe31565e640 10 monclient: get_monmap_and_config 2024-09-16T04:07:27.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.893+0000 7fe31565e640 10 monclient: build_initial_monmap 2024-09-16T04:07:27.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.893+0000 7fe31565e640 10 monclient: monmap: 2024-09-16T04:07:27.898 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:07:27.898 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:27.898 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:07:27.896775+0000 2024-09-16T04:07:27.899 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:07:27.896775+0000 2024-09-16T04:07:27.899 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:07:27.899 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:27.899 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:07:27.899 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:07:27.899 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:07:27.899 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:07:27.899 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:07:27.899 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:27.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.893+0000 7fe31565e640 1 Processor -- start 2024-09-16T04:07:27.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.893+0000 7fe31565e640 1 -- start start 2024-09-16T04:07:27.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.893+0000 7fe31565e640 10 monclient: init 2024-09-16T04:07:27.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.893+0000 7fe31565e640 10 monclient: _reopen_session rank -1 2024-09-16T04:07:27.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.893+0000 7fe31565e640 10 monclient: _add_conns ranks=[3,2,0,4,1] 2024-09-16T04:07:27.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.893+0000 7fe31565e640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe3100a1330 0x7fe3100a1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:27.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.893+0000 7fe31565e640 10 monclient(hunting): picked mon.noname-a con 0x7fe3100a1330 addr v2:172.21.15.42:3300/0 2024-09-16T04:07:27.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.893+0000 7fe31565e640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe31009c920 0x7fe3100a1c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:27.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.893+0000 7fe31565e640 10 monclient(hunting): picked mon.noname-b con 0x7fe31009c920 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:07:27.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.893+0000 7fe31565e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe310072d40 addr v1:172.21.15.42:6789/0 2024-09-16T04:07:27.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.893+0000 7fe31565e640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe310072350 con 0x7fe310072d40 2024-09-16T04:07:27.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.893+0000 7fe31565e640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:27.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.893+0000 7fe31565e640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7fe310072b50 con 0x7fe3100a1330 2024-09-16T04:07:27.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.893+0000 7fe31565e640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:27.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.893+0000 7fe31565e640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fe3100a2190 con 0x7fe31009c920 2024-09-16T04:07:27.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.893+0000 7fe31565e640 10 monclient(hunting): _renew_subs 2024-09-16T04:07:27.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.893+0000 7fe31565e640 10 monclient(hunting): authenticate will time out at 2494.866009s 2024-09-16T04:07:27.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.893+0000 7fe30effd640 1 -- >> v2:172.21.15.42:3300/0 conn(0x7fe3100a1330 msgr2=0x7fe3100a1710 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3300/0 2024-09-16T04:07:27.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.893+0000 7fe30f7fe640 1 -- >> v1:172.21.15.42:6789/0 conn(0x7fe310072d40 legacy=0x7fe310137120 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.42:6789/0 2024-09-16T04:07:27.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.893+0000 7fe30effd640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe3100a1330 0x7fe3100a1710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:07:27.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.893+0000 7fe30e7fc640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe31009c920 0x7fe3100a1c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:27.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.893+0000 7fe30e7fc640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe31009c920 0x7fe3100a1c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:34772/0 (socket says 172.21.15.42:34772) 2024-09-16T04:07:27.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.893+0000 7fe30e7fc640 1 -- 172.21.15.42:0/2218287759 learned_addr learned my addr 172.21.15.42:0/2218287759 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:07:27.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.893+0000 7fe30e7fc640 10 monclient(hunting): get_auth_request con 0x7fe31009c920 auth_method 0 2024-09-16T04:07:27.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.893+0000 7fe30e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:27.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.893+0000 7fe30e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:27.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.893+0000 7fe30e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:27.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.893+0000 7fe30e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:07:27.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.893+0000 7fe30e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:07:27.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.893+0000 7fe30e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:07:27.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.893+0000 7fe30e7fc640 10 monclient(hunting): handle_auth_done global_id 1014173 payload 274 2024-09-16T04:07:27.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.893+0000 7fe30e7fc640 1 -- 172.21.15.42:0/2218287759 >> v2:172.21.15.42:3300/0 conn(0x7fe3100a1330 msgr2=0x7fe3100a1710 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:07:27.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.893+0000 7fe30e7fc640 1 --2- 172.21.15.42:0/2218287759 >> v2:172.21.15.42:3300/0 conn(0x7fe3100a1330 0x7fe3100a1710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:27.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.893+0000 7fe30e7fc640 1 -- 172.21.15.42:0/2218287759 >> v1:172.21.15.42:6789/0 conn(0x7fe310072d40 legacy=0x7fe310137120 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:07:27.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.893+0000 7fe30e7fc640 10 monclient: _finish_hunting 0 2024-09-16T04:07:27.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.893+0000 7fe30e7fc640 1 monclient: found mon.noname-b 2024-09-16T04:07:27.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.893+0000 7fe30e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:07:27.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.893+0000 7fe30e7fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T04:07:27.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.893+0000 7fe30e7fc640 1 -- 172.21.15.42:0/2218287759 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe31009bc60 con 0x7fe31009c920 2024-09-16T04:07:27.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.897+0000 7fe30e7fc640 1 --2- 172.21.15.42:0/2218287759 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe31009c920 0x7fe3100a1c50 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fe300009870 tx=0x7fe30001c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9d95f186c7f25336 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:07:27.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.897+0000 7fe30dffb640 1 -- 172.21.15.42:0/2218287759 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3000b2070 con 0x7fe31009c920 2024-09-16T04:07:27.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.897+0000 7fe30dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:27.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.897+0000 7fe30dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:07:27.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.897+0000 7fe30dffb640 10 monclient: dump: 2024-09-16T04:07:27.913 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:07:27.913 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:27.913 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:27.913 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:27.913 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:27.913 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:27.914 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:07:27.914 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:07:27.914 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:07:27.914 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:27.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.897+0000 7fe30dffb640 10 monclient: _finish_auth 0 2024-09-16T04:07:27.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.897+0000 7fe30dffb640 10 monclient: _check_auth_tickets 2024-09-16T04:07:27.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.897+0000 7fe30dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:07:27.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.897+0000 7fe30dffb640 1 -- 172.21.15.42:0/2218287759 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe3000ad640 con 0x7fe31009c920 2024-09-16T04:07:27.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.897+0000 7fe31565e640 5 monclient: authenticate success, global_id 1014173 2024-09-16T04:07:27.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.897+0000 7fe31565e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:07:27.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.897+0000 7fe30dffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:07:27.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.897+0000 7fe31565e640 10 monclient: get_monmap_and_config success 2024-09-16T04:07:27.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.897+0000 7fe30dffb640 1 -- 172.21.15.42:0/2218287759 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3000ada10 con 0x7fe31009c920 2024-09-16T04:07:27.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.897+0000 7fe30dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:27.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.897+0000 7fe31565e640 4 set_mon_vals no callback set 2024-09-16T04:07:27.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.897+0000 7fe30dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:07:27.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.897+0000 7fe30dffb640 10 monclient: dump: 2024-09-16T04:07:27.915 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:07:27.915 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:27.915 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:27.915 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:27.915 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:27.915 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:27.915 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:07:27.915 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:07:27.915 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:07:27.915 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:27.916 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.897+0000 7fe31565e640 10 monclient: shutdown 2024-09-16T04:07:27.916 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.897+0000 7fe31565e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:07:27.916 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.897+0000 7fe31565e640 1 -- 172.21.15.42:0/2218287759 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe31009c920 msgr2=0x7fe3100a1c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:27.916 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.897+0000 7fe31565e640 1 --2- 172.21.15.42:0/2218287759 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe31009c920 0x7fe3100a1c50 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fe300009870 tx=0x7fe30001c490 comp rx=0 tx=0).stop 2024-09-16T04:07:27.916 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.897+0000 7fe31565e640 1 -- 172.21.15.42:0/2218287759 shutdown_connections 2024-09-16T04:07:27.916 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.897+0000 7fe31565e640 1 --2- 172.21.15.42:0/2218287759 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe31009c920 0x7fe3100a1c50 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:27.916 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.897+0000 7fe31565e640 1 --2- 172.21.15.42:0/2218287759 >> v2:172.21.15.42:3300/0 conn(0x7fe3100a1330 0x7fe3100a1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:27.916 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.897+0000 7fe31565e640 1 -- 172.21.15.42:0/2218287759 >> 172.21.15.42:0/2218287759 conn(0x7fe3100a2420 msgr2=0x7fe3100a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:07:27.916 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.897+0000 7fe31565e640 1 -- 172.21.15.42:0/2218287759 shutdown_connections 2024-09-16T04:07:27.916 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.897+0000 7fe31565e640 1 -- 172.21.15.42:0/2218287759 wait complete. 2024-09-16T04:07:27.916 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.901+0000 7fe31565e640 10 monclient: build_initial_monmap 2024-09-16T04:07:27.916 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.901+0000 7fe31565e640 10 monclient: monmap: 2024-09-16T04:07:27.917 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:07:27.917 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:07:27.917 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:07:27.917 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:07:27.917 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:07:27.917 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:27.917 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:07:27.917 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:07:27.917 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:07:27.917 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:27.917 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.901+0000 7fe31565e640 1 librados: starting msgr at 2024-09-16T04:07:27.917 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.901+0000 7fe31565e640 1 librados: starting objecter 2024-09-16T04:07:27.917 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.901+0000 7fe31565e640 1 Processor -- start 2024-09-16T04:07:27.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.901+0000 7fe31565e640 1 -- start start 2024-09-16T04:07:27.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.901+0000 7fe31565e640 1 librados: setting wanted keys 2024-09-16T04:07:27.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.901+0000 7fe31565e640 1 librados: calling monclient init 2024-09-16T04:07:27.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.901+0000 7fe31565e640 10 monclient: init 2024-09-16T04:07:27.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.901+0000 7fe31565e640 10 monclient: _reopen_session rank -1 2024-09-16T04:07:27.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.901+0000 7fe31565e640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:07:27.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.901+0000 7fe31565e640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe310072d40 0x7fe3101cf0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:27.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.901+0000 7fe31565e640 10 monclient(hunting): picked mon.noname-a con 0x7fe310072d40 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:07:27.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.901+0000 7fe31565e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe31009c920 0x7fe310136370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:27.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.901+0000 7fe31565e640 10 monclient(hunting): picked mon.noname-b con 0x7fe31009c920 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:07:27.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.901+0000 7fe31565e640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe3100a1330 0x7fe31011cd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:27.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.901+0000 7fe31565e640 10 monclient(hunting): picked mon.noname-c con 0x7fe3100a1330 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:07:27.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.901+0000 7fe31565e640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:27.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.901+0000 7fe31565e640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fe310141b20 con 0x7fe310072d40 2024-09-16T04:07:27.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.901+0000 7fe31565e640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:27.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.901+0000 7fe31565e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe3101419a0 con 0x7fe31009c920 2024-09-16T04:07:27.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.901+0000 7fe31565e640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:27.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.901+0000 7fe31565e640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fe310141ca0 con 0x7fe3100a1330 2024-09-16T04:07:27.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.901+0000 7fe31565e640 10 monclient(hunting): _renew_subs 2024-09-16T04:07:27.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.901+0000 7fe31565e640 10 monclient(hunting): authenticate will time out at 2494.873366s 2024-09-16T04:07:27.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.901+0000 7fe30effd640 1 -- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe310072d40 msgr2=0x7fe3101cf0b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.42:3300/0 2024-09-16T04:07:27.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.901+0000 7fe30effd640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe310072d40 0x7fe3101cf0b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:07:27.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.901+0000 7fe30f7fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe3100a1330 0x7fe31011cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:27.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.901+0000 7fe30f7fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe3100a1330 0x7fe31011cd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:34784/0 (socket says 172.21.15.42:34784) 2024-09-16T04:07:27.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.901+0000 7fe30f7fe640 1 -- 172.21.15.42:0/4189866002 learned_addr learned my addr 172.21.15.42:0/4189866002 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:07:27.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.901+0000 7fe30f7fe640 10 monclient(hunting): get_auth_request con 0x7fe3100a1330 auth_method 0 2024-09-16T04:07:27.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.901+0000 7fe30f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:27.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.901+0000 7fe30f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:27.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.901+0000 7fe30f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:27.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.901+0000 7fe30e7fc640 1 --2- 172.21.15.42:0/4189866002 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe31009c920 0x7fe310136370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:27.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.901+0000 7fe30e7fc640 10 monclient(hunting): get_auth_request con 0x7fe31009c920 auth_method 0 2024-09-16T04:07:27.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.901+0000 7fe30e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:27.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.901+0000 7fe30e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:27.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.901+0000 7fe30e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:27.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.901+0000 7fe30f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:07:27.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.901+0000 7fe30f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:07:27.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.901+0000 7fe30f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:07:27.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.901+0000 7fe30e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:07:27.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.901+0000 7fe30e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:07:27.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.901+0000 7fe30e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:07:27.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.901+0000 7fe30f7fe640 10 monclient(hunting): handle_auth_done global_id 1014176 payload 824 2024-09-16T04:07:27.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.901+0000 7fe30f7fe640 1 -- 172.21.15.42:0/4189866002 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe31009c920 msgr2=0x7fe310136370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:27.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.901+0000 7fe30f7fe640 1 --2- 172.21.15.42:0/4189866002 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe31009c920 0x7fe310136370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:27.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.901+0000 7fe30f7fe640 1 -- 172.21.15.42:0/4189866002 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe310072d40 msgr2=0x7fe3101cf0b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:07:27.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.901+0000 7fe30f7fe640 1 --2- 172.21.15.42:0/4189866002 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe310072d40 0x7fe3101cf0b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:27.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.901+0000 7fe30f7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:07:27.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.901+0000 7fe30f7fe640 1 monclient: found mon.noname-c 2024-09-16T04:07:27.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.901+0000 7fe30f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:07:27.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.901+0000 7fe30f7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T04:07:27.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.901+0000 7fe30f7fe640 1 -- 172.21.15.42:0/4189866002 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe310120d00 con 0x7fe3100a1330 2024-09-16T04:07:27.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.901+0000 7fe30f7fe640 1 --2- 172.21.15.42:0/4189866002 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe3100a1330 0x7fe31011cd70 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fe30800be90 tx=0x7fe30800bf80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:07:27.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.901+0000 7fe2f77fe640 10 client.?.objecter ms_handle_connect 0x7fe3100a1330 2024-09-16T04:07:27.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.901+0000 7fe2f77fe640 10 client.?.objecter resend_mon_ops 2024-09-16T04:07:27.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.901+0000 7fe2f77fe640 1 -- 172.21.15.42:0/4189866002 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe308015470 con 0x7fe3100a1330 2024-09-16T04:07:27.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.901+0000 7fe2f77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:27.923 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.901+0000 7fe2f77fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:07:27.923 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.901+0000 7fe2f77fe640 10 monclient: dump: 2024-09-16T04:07:27.923 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:07:27.923 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:27.923 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:27.923 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:27.923 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:27.923 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:27.923 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:07:27.923 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:07:27.923 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:07:27.923 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:27.923 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.901+0000 7fe2f77fe640 10 monclient: _finish_auth 0 2024-09-16T04:07:27.923 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.901+0000 7fe2f77fe640 10 monclient: _check_auth_tickets 2024-09-16T04:07:27.924 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.901+0000 7fe2f77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:07:27.924 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.901+0000 7fe2f77fe640 1 -- 172.21.15.42:0/4189866002 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe308010070 con 0x7fe3100a1330 2024-09-16T04:07:27.924 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.901+0000 7fe2f77fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:07:27.924 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.901+0000 7fe2f77fe640 1 -- 172.21.15.42:0/4189866002 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe308015890 con 0x7fe3100a1330 2024-09-16T04:07:27.924 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.901+0000 7fe2f77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:27.924 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.901+0000 7fe2f77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:07:27.924 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.901+0000 7fe2f77fe640 10 monclient: dump: 2024-09-16T04:07:27.924 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:07:27.924 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:27.924 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:27.924 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:27.924 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:27.925 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:27.925 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:07:27.925 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:07:27.925 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:07:27.925 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:27.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.901+0000 7fe31565e640 5 monclient: authenticate success, global_id 1014176 2024-09-16T04:07:27.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.901+0000 7fe31565e640 10 monclient: _renew_subs 2024-09-16T04:07:27.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.901+0000 7fe31565e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:07:27.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.901+0000 7fe31565e640 1 -- 172.21.15.42:0/4189866002 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3100a5020 con 0x7fe3100a1330 2024-09-16T04:07:27.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.901+0000 7fe30d7fa640 4 set_mon_vals no callback set 2024-09-16T04:07:27.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.901+0000 7fe31565e640 10 client.1014176.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:07:27.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.901+0000 7fe31565e640 10 monclient: _renew_subs 2024-09-16T04:07:27.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.901+0000 7fe31565e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:07:27.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.901+0000 7fe31565e640 1 -- 172.21.15.42:0/4189866002 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fe310073a30 con 0x7fe3100a1330 2024-09-16T04:07:27.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.901+0000 7fe31565e640 1 librados: init done 2024-09-16T04:07:27.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.901+0000 7fe31565e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:07:27.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.901+0000 7fe31565e640 1 --2- 172.21.15.42:0/4189866002 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe3101368b0 0x7fe31011d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:27.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.901+0000 7fe31565e640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:27.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.901+0000 7fe31565e640 1 -- 172.21.15.42:0/4189866002 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe3101206a0 con 0x7fe3101368b0 2024-09-16T04:07:27.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.901+0000 7fe2f77fe640 1 -- 172.21.15.42:0/4189866002 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fe308005070 con 0x7fe3100a1330 2024-09-16T04:07:27.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.901+0000 7fe2f77fe640 1 --2- 172.21.15.42:0/4189866002 >> v2:172.21.15.154:6812/254845342 conn(0x7fe2fc0377f0 0x7fe2fc039cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:27.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.901+0000 7fe30effd640 1 --2- 172.21.15.42:0/4189866002 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe3101368b0 0x7fe31011d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:27.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.901+0000 7fe2f77fe640 1 -- 172.21.15.42:0/4189866002 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(706..706 src has 451..706) ==== 5040+0+0 (secure 0 0 0) 0x7fe3080491b0 con 0x7fe3100a1330 2024-09-16T04:07:27.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.901+0000 7fe30effd640 10 monclient: get_auth_request con 0x7fe3101368b0 auth_method 0 2024-09-16T04:07:27.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.901+0000 7fe2f77fe640 10 client.1014176.objecter ms_dispatch 0x7fe310000ba0 osd_map(706..706 src has 451..706) 2024-09-16T04:07:27.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.901+0000 7fe30effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:27.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.901+0000 7fe30effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:27.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.901+0000 7fe30effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:27.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.901+0000 7fe2f77fe640 3 client.1014176.objecter handle_osd_map got epochs [706,706] > 0 2024-09-16T04:07:27.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.901+0000 7fe2f77fe640 3 client.1014176.objecter handle_osd_map decoding full epoch 706 2024-09-16T04:07:27.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.905+0000 7fe2f77fe640 20 client.1014176.objecter dump_active .. 0 homeless 2024-09-16T04:07:27.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.905+0000 7fe30effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:07:27.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.905+0000 7fe30effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:07:27.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.905+0000 7fe30effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:07:27.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.905+0000 7fe30e7fc640 1 --2- 172.21.15.42:0/4189866002 >> v2:172.21.15.154:6812/254845342 conn(0x7fe2fc0377f0 0x7fe2fc039cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:27.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.905+0000 7fe30e7fc640 10 monclient: get_auth_request con 0x7fe2fc0377f0 auth_method 0 2024-09-16T04:07:27.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.905+0000 7fe30effd640 10 monclient(hunting): handle_auth_done global_id 905089 payload 824 2024-09-16T04:07:27.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.905+0000 7fe30effd640 1 -- 172.21.15.42:0/4189866002 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe310137120 con 0x7fe3101368b0 2024-09-16T04:07:27.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.905+0000 7fe2f77fe640 10 client.1014176.objecter ms_handle_connect 0x7fe3101368b0 2024-09-16T04:07:27.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.905+0000 7fe30effd640 1 --2- 172.21.15.42:0/4189866002 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe3101368b0 0x7fe31011d2b0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fe30400b450 tx=0x7fe30400b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:07:27.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.905+0000 7fe2f77fe640 10 client.1014176.objecter resend_mon_ops 2024-09-16T04:07:27.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.905+0000 7fe2f77fe640 10 client.1014176.objecter ms_handle_connect 0x7fe2fc0377f0 2024-09-16T04:07:27.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.905+0000 7fe30e7fc640 1 --2- 172.21.15.42:0/4189866002 >> v2:172.21.15.154:6812/254845342 conn(0x7fe2fc0377f0 0x7fe2fc039cb0 secure :-1 s=READY pgs=3884 cs=0 l=1 rev1=1 crypto rx=0x7fe3000af000 tx=0x7fe300008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:07:27.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.905+0000 7fe2f77fe640 1 -- 172.21.15.42:0/4189866002 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe304003dd0 con 0x7fe3101368b0 2024-09-16T04:07:27.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.905+0000 7fe2f77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:27.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.905+0000 7fe2f77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:07:27.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.905+0000 7fe2f77fe640 10 monclient: dump: 2024-09-16T04:07:27.928 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:07:27.928 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:27.928 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:27.929 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:27.929 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:27.929 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:27.929 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:07:27.929 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:07:27.929 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:07:27.929 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:27.929 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.905+0000 7fe2f77fe640 1 -- 172.21.15.42:0/4189866002 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fe304015070 con 0x7fe3101368b0 2024-09-16T04:07:27.929 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.905+0000 7fe2f77fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:07:27.929 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.905+0000 7fe2f77fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:07:27.929 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.905+0000 7fe2f77fe640 1 -- 172.21.15.42:0/4189866002 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe3101368b0 msgr2=0x7fe31011d2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:27.929 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.905+0000 7fe2f77fe640 1 --2- 172.21.15.42:0/4189866002 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe3101368b0 0x7fe31011d2b0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fe30400b450 tx=0x7fe30400b910 comp rx=0 tx=0).stop 2024-09-16T04:07:27.930 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.905+0000 7fe2f77fe640 1 -- 172.21.15.42:0/4189866002 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe3101368b0 msgr2=0x7fe31011d2b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:07:27.930 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.905+0000 7fe2f77fe640 1 --2- 172.21.15.42:0/4189866002 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe3101368b0 0x7fe31011d2b0 secure :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fe30400b450 tx=0x7fe30400b910 comp rx=0 tx=0).stop 2024-09-16T04:07:27.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.913+0000 7fe31565e640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:07:27.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.913+0000 7fe31565e640 1 --2- 172.21.15.42:0/4189866002 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe31011d7f0 0x7fe31013b570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:27.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.913+0000 7fe31565e640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:27.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.913+0000 7fe31565e640 1 -- 172.21.15.42:0/4189866002 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe31011dc00 con 0x7fe31011d7f0 2024-09-16T04:07:27.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.913+0000 7fe30effd640 1 --2- 172.21.15.42:0/4189866002 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe31011d7f0 0x7fe31013b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:27.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.913+0000 7fe30effd640 10 monclient: get_auth_request con 0x7fe31011d7f0 auth_method 0 2024-09-16T04:07:27.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.913+0000 7fe30effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:27.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.913+0000 7fe30effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:27.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.913+0000 7fe30effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:27.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.913+0000 7fe30effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:07:27.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.913+0000 7fe30effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:07:27.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.913+0000 7fe30effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:07:27.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.913+0000 7fe30effd640 10 monclient(hunting): handle_auth_done global_id 905092 payload 824 2024-09-16T04:07:27.933 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.913+0000 7fe30effd640 1 -- 172.21.15.42:0/4189866002 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fe310069040 con 0x7fe31011d7f0 2024-09-16T04:07:27.933 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.913+0000 7fe2f77fe640 10 client.1014176.objecter ms_handle_connect 0x7fe31011d7f0 2024-09-16T04:07:27.933 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.913+0000 7fe30effd640 1 --2- 172.21.15.42:0/4189866002 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe31011d7f0 0x7fe31013b570 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fe30400b450 tx=0x7fe3040059e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:07:27.933 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.913+0000 7fe2f77fe640 10 client.1014176.objecter resend_mon_ops 2024-09-16T04:07:27.933 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.913+0000 7fe2f77fe640 1 -- 172.21.15.42:0/4189866002 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe304005610 con 0x7fe31011d7f0 2024-09-16T04:07:27.933 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.913+0000 7fe2f77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:27.933 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.913+0000 7fe2f77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:07:27.933 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.913+0000 7fe2f77fe640 10 monclient: dump: 2024-09-16T04:07:27.933 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:07:27.933 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:27.933 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:27.933 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:27.933 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:27.934 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:27.934 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:07:27.934 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:07:27.934 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:07:27.934 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:27.934 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.913+0000 7fe2f77fe640 1 -- 172.21.15.42:0/4189866002 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4871 (secure 0 0 0) 0x7fe304005610 con 0x7fe31011d7f0 2024-09-16T04:07:27.934 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.913+0000 7fe2f77fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:07:27.934 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.913+0000 7fe2f77fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:07:27.934 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.913+0000 7fe2f77fe640 1 -- 172.21.15.42:0/4189866002 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe31011d7f0 msgr2=0x7fe31013b570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:27.934 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.913+0000 7fe2f77fe640 1 --2- 172.21.15.42:0/4189866002 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe31011d7f0 0x7fe31013b570 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fe30400b450 tx=0x7fe3040059e0 comp rx=0 tx=0).stop 2024-09-16T04:07:27.934 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.913+0000 7fe2f77fe640 1 -- 172.21.15.42:0/4189866002 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe31011d7f0 msgr2=0x7fe31013b570 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:07:27.934 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.913+0000 7fe2f77fe640 1 --2- 172.21.15.42:0/4189866002 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe31011d7f0 0x7fe31013b570 secure :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fe30400b450 tx=0x7fe3040059e0 comp rx=0 tx=0).stop 2024-09-16T04:07:27.935 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T04:07:27.935 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:07:27.935 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:07:27.935 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "leader", 2024-09-16T04:07:27.935 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 234, 2024-09-16T04:07:27.935 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 576093, 2024-09-16T04:07:27.935 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T04:07:27.935 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T04:07:27.935 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T04:07:27.935 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:07:27.935 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 1, 2024-09-16T04:07:27.936 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:07:27.936 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T04:07:27.936 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T04:07:27.936 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:07:27.936 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:07:27.936 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:07:27.936 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:07:27.936 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:07:27.936 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:07:27.936 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:07:27.936 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:07:27.936 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:07:27.937 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:07:27.937 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:07:27.937 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:07:27.937 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:07:27.937 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T04:07:27.937 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:07:27.937 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:07:27.937 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:07:27.937 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:07:27.937 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:07:27.937 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:07:27.937 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:07:27.937 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:07:27.938 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:07:27.938 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:07:27.938 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:07:27.938 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:07:27.938 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:27.938 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T04:07:27.938 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T04:07:27.938 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T04:07:27.938 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T04:07:27.938 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T04:07:27.938 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T04:07:27.938 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:07:27.938 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:07:27.938 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T04:07:27.939 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T04:07:27.939 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T04:07:27.939 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T04:07:27.939 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T04:07:27.939 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T04:07:27.939 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T04:07:27.939 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:07:27.939 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T04:07:27.939 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:07:27.939 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:07:27.939 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:07:27.939 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:07:27.939 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:07:27.939 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:07:27.940 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:07:27.940 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:07:27.940 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:07:27.940 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:07:27.940 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:07:27.940 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:07:27.940 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T04:07:27.940 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:27.940 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T04:07:27.940 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:27.940 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:07:27.940 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:07:27.940 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:07:27.941 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:07:27.941 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:27.941 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:07:27.941 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T04:07:27.941 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:07:27.941 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:27.941 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:27.941 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:07:27.941 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T04:07:27.941 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:07:27.941 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:07:27.941 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:07:27.941 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:27.941 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T04:07:27.942 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T04:07:27.942 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:07:27.942 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:07:27.942 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:07:27.942 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:27.942 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:27.942 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:07:27.942 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:07:27.942 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:07:27.942 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:07:27.942 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:27.942 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:07:27.942 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T04:07:27.942 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:07:27.942 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:27.943 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:27.943 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:07:27.943 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T04:07:27.943 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:07:27.943 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:07:27.943 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:07:27.943 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:27.943 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T04:07:27.943 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T04:07:27.943 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:07:27.944 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:07:27.944 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:07:27.944 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:27.944 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:27.944 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:07:27.944 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:07:27.944 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:07:27.944 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:07:27.944 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:27.944 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:07:27.944 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T04:07:27.944 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:07:27.944 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:27.945 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:27.945 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:07:27.945 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T04:07:27.945 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:07:27.945 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:07:27.945 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:07:27.945 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:27.945 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T04:07:27.945 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T04:07:27.945 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:07:27.945 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:07:27.945 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:07:27.945 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:07:27.946 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:07:27.946 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:27.946 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T04:07:27.946 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T04:07:27.946 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:27.946 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:07:27.946 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:07:27.946 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:07:27.946 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:07:27.946 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:07:27.946 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T04:07:27.946 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:27.946 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:07:27.946 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:07:27.947 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 3 2024-09-16T04:07:27.947 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:07:27.947 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:07:27.947 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T04:07:27.947 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:27.947 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:07:27.947 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:07:27.947 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 5 2024-09-16T04:07:27.947 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:07:27.947 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:07:27.947 INFO:teuthology.orchestra.run.smithi042.stdout: "mgr": [ 2024-09-16T04:07:27.947 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:27.947 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:07:27.947 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:07:27.947 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:07:27.948 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:07:27.948 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:07:27.948 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:27.948 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T04:07:27.948 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T04:07:27.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.913+0000 7fe31565e640 10 librados: watch_flush enter 2024-09-16T04:07:27.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.913+0000 7fe31565e640 10 librados: watch_flush exit 2024-09-16T04:07:27.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.913+0000 7fe31565e640 20 client.1014176.objecter shutdown clearing up homeless session... 2024-09-16T04:07:27.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.913+0000 7fe31565e640 10 client.1014176.objecter successfully canceled tick 2024-09-16T04:07:27.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.913+0000 7fe31565e640 1 -- 172.21.15.42:0/4189866002 >> v2:172.21.15.154:6812/254845342 conn(0x7fe2fc0377f0 msgr2=0x7fe2fc039cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:27.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.913+0000 7fe31565e640 1 --2- 172.21.15.42:0/4189866002 >> v2:172.21.15.154:6812/254845342 conn(0x7fe2fc0377f0 0x7fe2fc039cb0 secure :-1 s=READY pgs=3884 cs=0 l=1 rev1=1 crypto rx=0x7fe3000af000 tx=0x7fe300008040 comp rx=0 tx=0).stop 2024-09-16T04:07:27.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.913+0000 7fe31565e640 10 monclient: shutdown 2024-09-16T04:07:27.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.913+0000 7fe31565e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:07:27.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.913+0000 7fe31565e640 1 -- 172.21.15.42:0/4189866002 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe3100a1330 msgr2=0x7fe31011cd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:27.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.913+0000 7fe30f7fe640 1 -- 172.21.15.42:0/4189866002 reap_dead start 2024-09-16T04:07:27.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.913+0000 7fe31565e640 1 --2- 172.21.15.42:0/4189866002 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe3100a1330 0x7fe31011cd70 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fe30800be90 tx=0x7fe30800bf80 comp rx=0 tx=0).stop 2024-09-16T04:07:27.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.913+0000 7fe31565e640 1 -- 172.21.15.42:0/4189866002 shutdown_connections 2024-09-16T04:07:27.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.913+0000 7fe31565e640 1 --2- 172.21.15.42:0/4189866002 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe3100a1330 0x7fe31011cd70 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:27.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.913+0000 7fe31565e640 1 -- 172.21.15.42:0/4189866002 >> 172.21.15.42:0/4189866002 conn(0x7fe3100a2420 msgr2=0x7fe31009cd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:07:27.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.913+0000 7fe31565e640 1 -- 172.21.15.42:0/4189866002 shutdown_connections 2024-09-16T04:07:27.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.913+0000 7fe31565e640 1 -- 172.21.15.42:0/4189866002 wait complete. 2024-09-16T04:07:27.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:27.913+0000 7fe31565e640 1 librados: shutdown 2024-09-16T04:07:27.952 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T04:07:27.952 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-16T04:07:47.952 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-16T04:07:47.953 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-09-16T04:07:47.953 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-16T04:07:47.953 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-16T04:07:47.956 INFO:tasks.ceph.mon.a:Started 2024-09-16T04:07:47.956 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T04:07:47.957 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:07:48.093 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.085+0000 7fd669832640 10 monclient: get_monmap_and_config 2024-09-16T04:07:48.093 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.085+0000 7fd669832640 10 monclient: build_initial_monmap 2024-09-16T04:07:48.093 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.085+0000 7fd669832640 10 monclient: monmap: 2024-09-16T04:07:48.093 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:07:48.093 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:48.093 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:07:48.091514+0000 2024-09-16T04:07:48.093 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:07:48.091514+0000 2024-09-16T04:07:48.094 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:07:48.094 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:48.094 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:07:48.094 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:07:48.094 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:07:48.094 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:07:48.094 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:07:48.094 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:48.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.085+0000 7fd669832640 1 Processor -- start 2024-09-16T04:07:48.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.089+0000 7fd669832640 1 -- start start 2024-09-16T04:07:48.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.089+0000 7fd669832640 10 monclient: init 2024-09-16T04:07:48.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.089+0000 7fd669832640 10 monclient: _reopen_session rank -1 2024-09-16T04:07:48.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.089+0000 7fd669832640 10 monclient: _add_conns ranks=[0,1,4,2,3] 2024-09-16T04:07:48.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.089+0000 7fd669832640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd6640a3860 addr v1:172.21.15.42:6789/0 2024-09-16T04:07:48.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.089+0000 7fd669832640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd6640a3c40 addr v1:172.21.15.154:6789/0 2024-09-16T04:07:48.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.089+0000 7fd669832640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fd66413d8f0 0x7fd66413fce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:48.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.089+0000 7fd669832640 10 monclient(hunting): picked mon.noname-c con 0x7fd66413d8f0 addr v2:172.21.15.154:3300/0 2024-09-16T04:07:48.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.089+0000 7fd669832640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd664137730 con 0x7fd6640a3860 2024-09-16T04:07:48.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.089+0000 7fd669832640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd66409c010 con 0x7fd6640a3c40 2024-09-16T04:07:48.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.089+0000 7fd669832640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:48.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.089+0000 7fd669832640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fd6640a5e80 con 0x7fd66413d8f0 2024-09-16T04:07:48.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.089+0000 7fd669832640 10 monclient(hunting): _renew_subs 2024-09-16T04:07:48.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.089+0000 7fd669832640 10 monclient(hunting): authenticate will time out at 2515.060655s 2024-09-16T04:07:48.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.089+0000 7fd662ffd640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fd6640a3860 0x7fd664137020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:43646/0 (socket says 172.21.15.42:43646) 2024-09-16T04:07:48.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.089+0000 7fd662ffd640 1 -- 172.21.15.42:0/2207012643 learned_addr learned my addr 172.21.15.42:0/2207012643 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T04:07:48.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.089+0000 7fd6637fe640 1 --2- 172.21.15.42:0/2207012643 >> v2:172.21.15.154:3300/0 conn(0x7fd66413d8f0 0x7fd66413fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:48.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.089+0000 7fd6637fe640 10 monclient(hunting): get_auth_request con 0x7fd66413d8f0 auth_method 0 2024-09-16T04:07:48.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.089+0000 7fd6637fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:48.098 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.089+0000 7fd6637fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:48.098 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.089+0000 7fd6637fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:48.098 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.089+0000 7fd6637fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:07:48.098 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.089+0000 7fd6637fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:07:48.098 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.089+0000 7fd6637fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:07:48.098 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.089+0000 7fd6637fe640 10 monclient(hunting): handle_auth_done global_id 905095 payload 274 2024-09-16T04:07:48.098 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.089+0000 7fd6637fe640 1 -- 172.21.15.42:0/2207012643 >> v1:172.21.15.154:6789/0 conn(0x7fd6640a3c40 legacy=0x7fd66413c1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:48.098 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.089+0000 7fd6637fe640 1 -- 172.21.15.42:0/2207012643 >> v1:172.21.15.42:6789/0 conn(0x7fd6640a3860 legacy=0x7fd664137020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:48.098 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.089+0000 7fd6637fe640 10 monclient: _finish_hunting 0 2024-09-16T04:07:48.098 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.089+0000 7fd6637fe640 1 monclient: found mon.noname-c 2024-09-16T04:07:48.098 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.089+0000 7fd6637fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:07:48.098 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.089+0000 7fd6637fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.154:3300/0 2024-09-16T04:07:48.098 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.089+0000 7fd6637fe640 1 -- 172.21.15.42:0/2207012643 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd66409c2f0 con 0x7fd66413d8f0 2024-09-16T04:07:48.110 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.089+0000 7fd6637fe640 1 --2- 172.21.15.42:0/2207012643 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd66413d8f0 0x7fd66413fce0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fd6640a6000 tx=0x7fd65c01c810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cd6904f49de7ef5d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:07:48.110 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.093+0000 7fd661ffb640 1 -- 172.21.15.42:0/2207012643 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd65c0a66e0 con 0x7fd66413d8f0 2024-09-16T04:07:48.110 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.093+0000 7fd661ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:48.111 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.093+0000 7fd661ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:07:48.111 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.093+0000 7fd661ffb640 10 monclient: dump: 2024-09-16T04:07:48.111 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:07:48.111 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:48.111 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:48.111 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:48.111 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:48.111 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:48.111 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:07:48.112 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:07:48.112 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:07:48.112 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:48.112 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.093+0000 7fd661ffb640 10 monclient: _finish_auth 0 2024-09-16T04:07:48.112 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.093+0000 7fd661ffb640 10 monclient: _check_auth_tickets 2024-09-16T04:07:48.112 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.093+0000 7fd661ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:07:48.112 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.093+0000 7fd669832640 5 monclient: authenticate success, global_id 905095 2024-09-16T04:07:48.112 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.093+0000 7fd669832640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:07:48.112 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.093+0000 7fd661ffb640 1 -- 172.21.15.42:0/2207012643 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd65c0a6880 con 0x7fd66413d8f0 2024-09-16T04:07:48.113 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.093+0000 7fd661ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:07:48.113 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.093+0000 7fd669832640 10 monclient: get_monmap_and_config success 2024-09-16T04:07:48.114 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.093+0000 7fd669832640 4 set_mon_vals no callback set 2024-09-16T04:07:48.114 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.093+0000 7fd661ffb640 1 -- 172.21.15.42:0/2207012643 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd65c0a6c50 con 0x7fd66413d8f0 2024-09-16T04:07:48.114 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.093+0000 7fd661ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:48.114 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.093+0000 7fd661ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:07:48.115 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.093+0000 7fd661ffb640 10 monclient: dump: 2024-09-16T04:07:48.115 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:07:48.115 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:48.115 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:48.115 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:48.116 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:48.116 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:48.116 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:07:48.116 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:07:48.116 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:07:48.117 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:48.117 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.093+0000 7fd669832640 10 monclient: shutdown 2024-09-16T04:07:48.117 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.093+0000 7fd669832640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:07:48.117 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.093+0000 7fd669832640 1 -- 172.21.15.42:0/2207012643 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd66413d8f0 msgr2=0x7fd66413fce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:48.117 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.093+0000 7fd669832640 1 --2- 172.21.15.42:0/2207012643 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd66413d8f0 0x7fd66413fce0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fd6640a6000 tx=0x7fd65c01c810 comp rx=0 tx=0).stop 2024-09-16T04:07:48.117 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.093+0000 7fd669832640 1 -- 172.21.15.42:0/2207012643 shutdown_connections 2024-09-16T04:07:48.117 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.093+0000 7fd669832640 1 --2- 172.21.15.42:0/2207012643 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd66413d8f0 0x7fd66413fce0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:48.118 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.093+0000 7fd669832640 1 -- 172.21.15.42:0/2207012643 >> 172.21.15.42:0/2207012643 conn(0x7fd6640a9f40 msgr2=0x7fd6640aa350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:07:48.118 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.093+0000 7fd669832640 1 -- 172.21.15.42:0/2207012643 shutdown_connections 2024-09-16T04:07:48.118 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.093+0000 7fd669832640 1 -- 172.21.15.42:0/2207012643 wait complete. 2024-09-16T04:07:48.118 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.093+0000 7fd669832640 10 monclient: build_initial_monmap 2024-09-16T04:07:48.118 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.093+0000 7fd669832640 10 monclient: monmap: 2024-09-16T04:07:48.118 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:07:48.118 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:07:48.119 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:07:48.119 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:07:48.119 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:07:48.119 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:48.119 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:07:48.119 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:07:48.119 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:07:48.119 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:48.119 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.093+0000 7fd669832640 1 librados: starting msgr at 2024-09-16T04:07:48.120 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.093+0000 7fd669832640 1 librados: starting objecter 2024-09-16T04:07:48.120 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.093+0000 7fd669832640 1 Processor -- start 2024-09-16T04:07:48.120 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.093+0000 7fd669832640 1 -- start start 2024-09-16T04:07:48.120 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.093+0000 7fd669832640 1 librados: setting wanted keys 2024-09-16T04:07:48.120 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.093+0000 7fd669832640 1 librados: calling monclient init 2024-09-16T04:07:48.120 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.093+0000 7fd669832640 10 monclient: init 2024-09-16T04:07:48.120 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.093+0000 7fd669832640 10 monclient: _reopen_session rank -1 2024-09-16T04:07:48.120 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.093+0000 7fd669832640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:07:48.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.093+0000 7fd669832640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd6640a3860 0x7fd664130a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:48.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.093+0000 7fd669832640 10 monclient(hunting): picked mon.noname-c con 0x7fd6640a3860 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:07:48.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.093+0000 7fd669832640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd6640a3c40 0x7fd66413a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:48.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.093+0000 7fd669832640 10 monclient(hunting): picked mon.noname-a con 0x7fd6640a3c40 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:07:48.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.093+0000 7fd669832640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd66413d8f0 0x7fd66413a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:48.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.093+0000 7fd669832640 10 monclient(hunting): picked mon.noname-b con 0x7fd66413d8f0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:07:48.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.093+0000 7fd669832640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:48.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.093+0000 7fd669832640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fd664142dd0 con 0x7fd6640a3c40 2024-09-16T04:07:48.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.093+0000 7fd669832640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:48.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.093+0000 7fd669832640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd664142c50 con 0x7fd66413d8f0 2024-09-16T04:07:48.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.093+0000 7fd669832640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:48.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.093+0000 7fd669832640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fd664142f50 con 0x7fd6640a3860 2024-09-16T04:07:48.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.093+0000 7fd669832640 10 monclient(hunting): _renew_subs 2024-09-16T04:07:48.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.093+0000 7fd669832640 10 monclient(hunting): authenticate will time out at 2515.066144s 2024-09-16T04:07:48.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.093+0000 7fd6627fc640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd6640a3c40 0x7fd66413a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:48.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.093+0000 7fd662ffd640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd6640a3860 0x7fd664130a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:48.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.093+0000 7fd6627fc640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd6640a3c40 0x7fd66413a160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:45230/0 (socket says 172.21.15.42:45230) 2024-09-16T04:07:48.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.093+0000 7fd6627fc640 1 -- 172.21.15.42:0/4156907471 learned_addr learned my addr 172.21.15.42:0/4156907471 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:07:48.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.093+0000 7fd6627fc640 10 monclient(hunting): get_auth_request con 0x7fd6640a3c40 auth_method 0 2024-09-16T04:07:48.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.093+0000 7fd6627fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:48.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.093+0000 7fd6627fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:48.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.093+0000 7fd6627fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:48.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.093+0000 7fd662ffd640 10 monclient(hunting): get_auth_request con 0x7fd6640a3860 auth_method 0 2024-09-16T04:07:48.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.093+0000 7fd662ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:48.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.093+0000 7fd662ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:48.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.093+0000 7fd662ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:48.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.093+0000 7fd6627fc640 1 -- 172.21.15.42:0/4156907471 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd6640a3c40 msgr2=0x7fd66413a160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-16T04:07:48.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.093+0000 7fd6627fc640 1 -- 172.21.15.42:0/4156907471 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd6640a3c40 msgr2=0x7fd66413a160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:07:48.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.093+0000 7fd6627fc640 1 --2- 172.21.15.42:0/4156907471 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd6640a3c40 0x7fd66413a160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:07:48.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.093+0000 7fd6627fc640 1 --2- 172.21.15.42:0/4156907471 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd6640a3c40 0x7fd66413a160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:07:48.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.093+0000 7fd6637fe640 1 --2- 172.21.15.42:0/4156907471 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd66413d8f0 0x7fd66413a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:48.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.093+0000 7fd662ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:07:48.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.093+0000 7fd662ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:07:48.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.093+0000 7fd662ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:07:48.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.093+0000 7fd6637fe640 10 monclient(hunting): get_auth_request con 0x7fd66413d8f0 auth_method 0 2024-09-16T04:07:48.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.093+0000 7fd6637fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:48.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.093+0000 7fd6637fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:48.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.093+0000 7fd6637fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:48.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.093+0000 7fd6637fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:07:48.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.093+0000 7fd6637fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:07:48.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.093+0000 7fd6637fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:07:48.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.093+0000 7fd662ffd640 10 monclient(hunting): handle_auth_done global_id 1014179 payload 824 2024-09-16T04:07:48.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.093+0000 7fd662ffd640 1 -- 172.21.15.42:0/4156907471 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd66413d8f0 msgr2=0x7fd66413a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:48.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.093+0000 7fd662ffd640 1 --2- 172.21.15.42:0/4156907471 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd66413d8f0 0x7fd66413a6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:48.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.093+0000 7fd662ffd640 1 -- 172.21.15.42:0/4156907471 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd6640a3c40 msgr2=0x7fd66413a160 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:07:48.125 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.093+0000 7fd662ffd640 1 --2- 172.21.15.42:0/4156907471 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd6640a3c40 0x7fd66413a160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:48.125 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.093+0000 7fd662ffd640 10 monclient: _finish_hunting 0 2024-09-16T04:07:48.125 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.093+0000 7fd662ffd640 1 monclient: found mon.noname-c 2024-09-16T04:07:48.125 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.093+0000 7fd662ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:07:48.125 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.093+0000 7fd662ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T04:07:48.125 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.093+0000 7fd662ffd640 1 -- 172.21.15.42:0/4156907471 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd66409d840 con 0x7fd6640a3860 2024-09-16T04:07:48.125 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.093+0000 7fd662ffd640 1 --2- 172.21.15.42:0/4156907471 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd6640a3860 0x7fd664130a40 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fd65400c460 tx=0x7fd65400c920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:07:48.125 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.093+0000 7fd64bfff640 10 client.?.objecter ms_handle_connect 0x7fd6640a3860 2024-09-16T04:07:48.125 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.093+0000 7fd64bfff640 10 client.?.objecter resend_mon_ops 2024-09-16T04:07:48.125 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.093+0000 7fd64bfff640 1 -- 172.21.15.42:0/4156907471 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd654017070 con 0x7fd6640a3860 2024-09-16T04:07:48.125 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.093+0000 7fd64bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:48.125 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.093+0000 7fd64bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:07:48.125 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.093+0000 7fd64bfff640 10 monclient: dump: 2024-09-16T04:07:48.125 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:07:48.126 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:48.126 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:48.126 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:48.126 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:48.126 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:48.126 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:07:48.126 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:07:48.126 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:07:48.126 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:48.126 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.093+0000 7fd64bfff640 10 monclient: _finish_auth 0 2024-09-16T04:07:48.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.093+0000 7fd64bfff640 10 monclient: _check_auth_tickets 2024-09-16T04:07:48.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.093+0000 7fd64bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:07:48.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.093+0000 7fd64bfff640 1 -- 172.21.15.42:0/4156907471 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd654005cf0 con 0x7fd6640a3860 2024-09-16T04:07:48.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.093+0000 7fd669832640 5 monclient: authenticate success, global_id 1014179 2024-09-16T04:07:48.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.093+0000 7fd669832640 10 monclient: _renew_subs 2024-09-16T04:07:48.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.093+0000 7fd669832640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:07:48.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.093+0000 7fd669832640 1 -- 172.21.15.42:0/4156907471 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6640a8c50 con 0x7fd6640a3860 2024-09-16T04:07:48.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.093+0000 7fd64bfff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:07:48.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.093+0000 7fd64bfff640 1 -- 172.21.15.42:0/4156907471 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd654007270 con 0x7fd6640a3860 2024-09-16T04:07:48.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.093+0000 7fd64bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:48.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.093+0000 7fd64bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:07:48.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.093+0000 7fd64bfff640 10 monclient: dump: 2024-09-16T04:07:48.127 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:07:48.127 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:48.128 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:48.128 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:48.128 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:48.128 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:48.128 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:07:48.128 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:07:48.128 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:07:48.128 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:48.128 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.093+0000 7fd669832640 10 client.1014179.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:07:48.128 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.093+0000 7fd669832640 10 monclient: _renew_subs 2024-09-16T04:07:48.128 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.093+0000 7fd669832640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:07:48.128 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.093+0000 7fd669832640 1 -- 172.21.15.42:0/4156907471 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd66409d560 con 0x7fd6640a3860 2024-09-16T04:07:48.128 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.093+0000 7fd669832640 1 librados: init done 2024-09-16T04:07:48.128 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.093+0000 7fd660ff9640 4 set_mon_vals no callback set 2024-09-16T04:07:48.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.097+0000 7fd669832640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:07:48.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.097+0000 7fd669832640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:07:48.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.097+0000 7fd669832640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:07:48.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.097+0000 7fd64bfff640 1 -- 172.21.15.42:0/4156907471 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fd65401a070 con 0x7fd6640a3860 2024-09-16T04:07:48.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.097+0000 7fd669832640 1 -- 172.21.15.42:0/4156907471 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd664137020 con 0x7fd6640a3860 2024-09-16T04:07:48.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.097+0000 7fd64bfff640 1 --2- 172.21.15.42:0/4156907471 >> v2:172.21.15.154:6812/254845342 conn(0x7fd650037810 0x7fd650039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:48.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.097+0000 7fd64bfff640 1 -- 172.21.15.42:0/4156907471 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(706..706 src has 451..706) ==== 5040+0+0 (secure 0 0 0) 0x7fd654047e60 con 0x7fd6640a3860 2024-09-16T04:07:48.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.097+0000 7fd64bfff640 10 client.1014179.objecter ms_dispatch 0x7fd664000ba0 osd_map(706..706 src has 451..706) 2024-09-16T04:07:48.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.097+0000 7fd64bfff640 3 client.1014179.objecter handle_osd_map got epochs [706,706] > 0 2024-09-16T04:07:48.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.097+0000 7fd64bfff640 3 client.1014179.objecter handle_osd_map decoding full epoch 706 2024-09-16T04:07:48.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.097+0000 7fd64bfff640 20 client.1014179.objecter dump_active .. 0 homeless 2024-09-16T04:07:48.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.097+0000 7fd6627fc640 1 --2- 172.21.15.42:0/4156907471 >> v2:172.21.15.154:6812/254845342 conn(0x7fd650037810 0x7fd650039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:48.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.097+0000 7fd6627fc640 10 monclient: get_auth_request con 0x7fd650037810 auth_method 0 2024-09-16T04:07:48.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.097+0000 7fd64bfff640 10 client.1014179.objecter ms_handle_connect 0x7fd650037810 2024-09-16T04:07:48.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.097+0000 7fd6627fc640 1 --2- 172.21.15.42:0/4156907471 >> v2:172.21.15.154:6812/254845342 conn(0x7fd650037810 0x7fd650039cd0 secure :-1 s=READY pgs=3886 cs=0 l=1 rev1=1 crypto rx=0x7fd664131840 tx=0x7fd658009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:07:48.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.101+0000 7fd64bfff640 1 -- 172.21.15.42:0/4156907471 <== mon.2 v2:172.21.15.42:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7fd654007410 con 0x7fd6640a3860 2024-09-16T04:07:48.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.101+0000 7fd64bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:07:48.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.101+0000 7fd64bfff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:07:48.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.325+0000 7fd669832640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:07:48.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.325+0000 7fd669832640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:07:48.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.325+0000 7fd669832640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:07:48.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.325+0000 7fd669832640 1 -- 172.21.15.42:0/4156907471 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fd6640a5550 con 0x7fd6640a3860 2024-09-16T04:07:48.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.325+0000 7fd64bfff640 1 -- 172.21.15.42:0/4156907471 <== mon.2 v2:172.21.15.42:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7fd6540117a0 con 0x7fd6640a3860 2024-09-16T04:07:48.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.325+0000 7fd64bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:07:48.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.325+0000 7fd64bfff640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:07:48.337 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":234,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":21,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:07:48.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.329+0000 7fd669832640 10 librados: watch_flush enter 2024-09-16T04:07:48.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.329+0000 7fd669832640 10 librados: watch_flush exit 2024-09-16T04:07:48.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.329+0000 7fd669832640 20 client.1014179.objecter shutdown clearing up homeless session... 2024-09-16T04:07:48.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.329+0000 7fd669832640 10 client.1014179.objecter successfully canceled tick 2024-09-16T04:07:48.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.329+0000 7fd669832640 1 -- 172.21.15.42:0/4156907471 >> v2:172.21.15.154:6812/254845342 conn(0x7fd650037810 msgr2=0x7fd650039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:48.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.329+0000 7fd669832640 1 --2- 172.21.15.42:0/4156907471 >> v2:172.21.15.154:6812/254845342 conn(0x7fd650037810 0x7fd650039cd0 secure :-1 s=READY pgs=3886 cs=0 l=1 rev1=1 crypto rx=0x7fd664131840 tx=0x7fd658009040 comp rx=0 tx=0).stop 2024-09-16T04:07:48.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.329+0000 7fd669832640 10 monclient: shutdown 2024-09-16T04:07:48.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.329+0000 7fd669832640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:07:48.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.329+0000 7fd669832640 1 -- 172.21.15.42:0/4156907471 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd6640a3860 msgr2=0x7fd664130a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:48.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.329+0000 7fd669832640 1 --2- 172.21.15.42:0/4156907471 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd6640a3860 0x7fd664130a40 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fd65400c460 tx=0x7fd65400c920 comp rx=0 tx=0).stop 2024-09-16T04:07:48.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.329+0000 7fd669832640 1 -- 172.21.15.42:0/4156907471 shutdown_connections 2024-09-16T04:07:48.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.329+0000 7fd669832640 1 --2- 172.21.15.42:0/4156907471 >> v2:172.21.15.154:6812/254845342 conn(0x7fd650037810 0x7fd650039cd0 unknown :-1 s=CLOSED pgs=3886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:48.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.329+0000 7fd669832640 1 --2- 172.21.15.42:0/4156907471 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd66413d8f0 0x7fd66413a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:48.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.329+0000 7fd669832640 1 --2- 172.21.15.42:0/4156907471 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd6640a3c40 0x7fd66413a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:48.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.329+0000 7fd669832640 1 --2- 172.21.15.42:0/4156907471 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd6640a3860 0x7fd664130a40 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:48.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.329+0000 7fd669832640 1 -- 172.21.15.42:0/4156907471 >> 172.21.15.42:0/4156907471 conn(0x7fd6640a9f40 msgr2=0x7fd6640a7940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:07:48.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.329+0000 7fd669832640 1 -- 172.21.15.42:0/4156907471 shutdown_connections 2024-09-16T04:07:48.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.329+0000 7fd669832640 1 -- 172.21.15.42:0/4156907471 wait complete. 2024-09-16T04:07:48.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:48.329+0000 7fd669832640 1 librados: shutdown 2024-09-16T04:07:51.354 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:07:51.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.481+0000 7fd453f29640 10 monclient: get_monmap_and_config 2024-09-16T04:07:51.488 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.481+0000 7fd453f29640 10 monclient: build_initial_monmap 2024-09-16T04:07:51.488 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.481+0000 7fd453f29640 10 monclient: monmap: 2024-09-16T04:07:51.488 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:07:51.488 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:51.488 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:07:51.485938+0000 2024-09-16T04:07:51.488 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:07:51.485938+0000 2024-09-16T04:07:51.488 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:07:51.488 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:51.488 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:07:51.488 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:07:51.489 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:07:51.489 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:07:51.489 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:07:51.489 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:51.489 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.481+0000 7fd453f29640 1 Processor -- start 2024-09-16T04:07:51.489 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.481+0000 7fd453f29640 1 -- start start 2024-09-16T04:07:51.489 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.481+0000 7fd453f29640 10 monclient: init 2024-09-16T04:07:51.493 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.481+0000 7fd453f29640 10 monclient: _reopen_session rank -1 2024-09-16T04:07:51.493 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.481+0000 7fd453f29640 10 monclient: _add_conns ranks=[0,2,3,4,1] 2024-09-16T04:07:51.493 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.481+0000 7fd453f29640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd44c0a1330 addr v1:172.21.15.42:6789/0 2024-09-16T04:07:51.493 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.481+0000 7fd453f29640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd44c0a1710 0x7fd44c0a1b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:51.493 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.481+0000 7fd453f29640 10 monclient(hunting): picked mon.noname-b con 0x7fd44c0a1710 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:07:51.493 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.481+0000 7fd453f29640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd44c1393a0 0x7fd44c13b790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:51.493 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.481+0000 7fd453f29640 10 monclient(hunting): picked mon.noname-a con 0x7fd44c1393a0 addr v2:172.21.15.42:3300/0 2024-09-16T04:07:51.493 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.481+0000 7fd453f29640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd44c09e4c0 con 0x7fd44c0a1330 2024-09-16T04:07:51.493 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.481+0000 7fd453f29640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:51.494 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.481+0000 7fd453f29640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7fd44c09edf0 con 0x7fd44c1393a0 2024-09-16T04:07:51.494 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.481+0000 7fd453f29640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:51.494 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.481+0000 7fd453f29640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fd44c09cd60 con 0x7fd44c0a1710 2024-09-16T04:07:51.494 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.481+0000 7fd453f29640 10 monclient(hunting): _renew_subs 2024-09-16T04:07:51.494 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.481+0000 7fd453f29640 10 monclient(hunting): authenticate will time out at 2518.455371s 2024-09-16T04:07:51.494 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.481+0000 7fd4514c4640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd44c0a1710 0x7fd44c0a1b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:51.494 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.481+0000 7fd4524c6640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd44c1393a0 0x7fd44c13b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:51.494 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.481+0000 7fd451cc5640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fd44c0a1330 0x7fd44c09f180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:43656/0 (socket says 172.21.15.42:43656) 2024-09-16T04:07:51.494 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.481+0000 7fd451cc5640 1 -- 172.21.15.42:0/958496900 learned_addr learned my addr 172.21.15.42:0/958496900 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T04:07:51.494 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.481+0000 7fd4514c4640 10 monclient(hunting): get_auth_request con 0x7fd44c0a1710 auth_method 0 2024-09-16T04:07:51.494 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.481+0000 7fd4514c4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:51.494 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.481+0000 7fd4514c4640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:51.494 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.481+0000 7fd4514c4640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:51.495 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.481+0000 7fd4524c6640 10 monclient(hunting): get_auth_request con 0x7fd44c1393a0 auth_method 0 2024-09-16T04:07:51.495 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.481+0000 7fd4524c6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:51.495 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.481+0000 7fd4524c6640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:51.495 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.481+0000 7fd4524c6640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:51.495 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.481+0000 7fd4514c4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:07:51.495 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.481+0000 7fd4514c4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:07:51.495 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.481+0000 7fd4514c4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:07:51.495 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.481+0000 7fd4524c6640 1 -- 172.21.15.42:0/958496900 >> v2:172.21.15.42:3300/0 conn(0x7fd44c1393a0 msgr2=0x7fd44c13b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T04:07:51.495 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.481+0000 7fd4524c6640 1 -- 172.21.15.42:0/958496900 >> v2:172.21.15.42:3300/0 conn(0x7fd44c1393a0 msgr2=0x7fd44c13b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:07:51.495 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.481+0000 7fd4524c6640 1 --2- 172.21.15.42:0/958496900 >> v2:172.21.15.42:3300/0 conn(0x7fd44c1393a0 0x7fd44c13b790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:07:51.495 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.481+0000 7fd4524c6640 1 --2- 172.21.15.42:0/958496900 >> v2:172.21.15.42:3300/0 conn(0x7fd44c1393a0 0x7fd44c13b790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:07:51.495 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.481+0000 7fd4514c4640 10 monclient(hunting): handle_auth_done global_id 1014182 payload 274 2024-09-16T04:07:51.495 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.481+0000 7fd4514c4640 1 -- 172.21.15.42:0/958496900 >> v2:172.21.15.42:3300/0 conn(0x7fd44c1393a0 msgr2=0x7fd44c13b790 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:07:51.495 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.481+0000 7fd4514c4640 1 --2- 172.21.15.42:0/958496900 >> v2:172.21.15.42:3300/0 conn(0x7fd44c1393a0 0x7fd44c13b790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:51.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.481+0000 7fd4514c4640 1 -- 172.21.15.42:0/958496900 >> v1:172.21.15.42:6789/0 conn(0x7fd44c0a1330 legacy=0x7fd44c09f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:51.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.481+0000 7fd4514c4640 10 monclient: _finish_hunting 0 2024-09-16T04:07:51.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.481+0000 7fd4514c4640 1 monclient: found mon.noname-b 2024-09-16T04:07:51.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.481+0000 7fd4514c4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:07:51.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.481+0000 7fd4514c4640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T04:07:51.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.481+0000 7fd4514c4640 1 -- 172.21.15.42:0/958496900 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd44c0a2050 con 0x7fd44c0a1710 2024-09-16T04:07:51.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.485+0000 7fd4514c4640 1 --2- 172.21.15.42:0/958496900 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd44c0a1710 0x7fd44c0a1b10 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fd4400097c0 tx=0x7fd44001c360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=47aa4cdd5c909efd server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:07:51.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.485+0000 7fd450cc3640 1 -- 172.21.15.42:0/958496900 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4400b0070 con 0x7fd44c0a1710 2024-09-16T04:07:51.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.485+0000 7fd450cc3640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:51.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.485+0000 7fd450cc3640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:07:51.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.485+0000 7fd450cc3640 10 monclient: dump: 2024-09-16T04:07:51.496 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:07:51.496 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:51.496 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:51.497 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:51.497 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:51.497 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:51.497 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:07:51.497 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:07:51.497 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:07:51.497 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:51.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.485+0000 7fd450cc3640 10 monclient: _finish_auth 0 2024-09-16T04:07:51.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.485+0000 7fd450cc3640 10 monclient: _check_auth_tickets 2024-09-16T04:07:51.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.485+0000 7fd450cc3640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:07:51.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.485+0000 7fd453f29640 5 monclient: authenticate success, global_id 1014182 2024-09-16T04:07:51.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.485+0000 7fd450cc3640 1 -- 172.21.15.42:0/958496900 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd4400ab600 con 0x7fd44c0a1710 2024-09-16T04:07:51.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.485+0000 7fd450cc3640 10 monclient: handle_config config(0 keys) 2024-09-16T04:07:51.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.485+0000 7fd453f29640 10 monclient: get_monmap_and_config success 2024-09-16T04:07:51.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.485+0000 7fd453f29640 4 set_mon_vals no callback set 2024-09-16T04:07:51.498 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.485+0000 7fd450cc3640 1 -- 172.21.15.42:0/958496900 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4400ab9d0 con 0x7fd44c0a1710 2024-09-16T04:07:51.498 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.485+0000 7fd450cc3640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:51.498 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.485+0000 7fd450cc3640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:07:51.498 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.485+0000 7fd450cc3640 10 monclient: dump: 2024-09-16T04:07:51.498 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:07:51.498 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:51.498 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:51.498 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:51.498 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:51.498 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:51.498 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:07:51.498 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:07:51.498 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:07:51.499 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:51.499 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.485+0000 7fd453f29640 10 monclient: shutdown 2024-09-16T04:07:51.499 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.485+0000 7fd453f29640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:07:51.499 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.485+0000 7fd453f29640 1 -- 172.21.15.42:0/958496900 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd44c0a1710 msgr2=0x7fd44c0a1b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:51.499 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.485+0000 7fd453f29640 1 --2- 172.21.15.42:0/958496900 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd44c0a1710 0x7fd44c0a1b10 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fd4400097c0 tx=0x7fd44001c360 comp rx=0 tx=0).stop 2024-09-16T04:07:51.499 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.485+0000 7fd453f29640 1 -- 172.21.15.42:0/958496900 shutdown_connections 2024-09-16T04:07:51.499 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.485+0000 7fd453f29640 1 --2- 172.21.15.42:0/958496900 >> v2:172.21.15.42:3300/0 conn(0x7fd44c1393a0 0x7fd44c13b790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:51.499 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.485+0000 7fd453f29640 1 --2- 172.21.15.42:0/958496900 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd44c0a1710 0x7fd44c0a1b10 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:51.499 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.485+0000 7fd453f29640 1 -- 172.21.15.42:0/958496900 >> 172.21.15.42:0/958496900 conn(0x7fd44c0a2420 msgr2=0x7fd44c0a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:07:51.499 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.485+0000 7fd453f29640 1 -- 172.21.15.42:0/958496900 shutdown_connections 2024-09-16T04:07:51.499 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.485+0000 7fd453f29640 1 -- 172.21.15.42:0/958496900 wait complete. 2024-09-16T04:07:51.499 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.485+0000 7fd453f29640 10 monclient: build_initial_monmap 2024-09-16T04:07:51.499 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.485+0000 7fd453f29640 10 monclient: monmap: 2024-09-16T04:07:51.499 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:07:51.499 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:07:51.500 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:07:51.500 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:07:51.500 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:07:51.500 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:51.500 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:07:51.500 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:07:51.500 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:07:51.500 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:51.500 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.485+0000 7fd453f29640 1 librados: starting msgr at 2024-09-16T04:07:51.500 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.485+0000 7fd453f29640 1 librados: starting objecter 2024-09-16T04:07:51.500 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.485+0000 7fd453f29640 1 Processor -- start 2024-09-16T04:07:51.500 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.485+0000 7fd453f29640 1 -- start start 2024-09-16T04:07:51.500 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.485+0000 7fd453f29640 1 librados: setting wanted keys 2024-09-16T04:07:51.500 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.485+0000 7fd453f29640 1 librados: calling monclient init 2024-09-16T04:07:51.501 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.485+0000 7fd453f29640 10 monclient: init 2024-09-16T04:07:51.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.489+0000 7fd453f29640 10 monclient: _reopen_session rank -1 2024-09-16T04:07:51.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.489+0000 7fd453f29640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:07:51.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.489+0000 7fd453f29640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd44c0a1330 0x7fd44c1c9bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:51.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.489+0000 7fd453f29640 10 monclient(hunting): picked mon.noname-a con 0x7fd44c0a1330 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:07:51.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.489+0000 7fd453f29640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd44c0a1710 0x7fd44c071700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:51.505 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.489+0000 7fd453f29640 10 monclient(hunting): picked mon.noname-c con 0x7fd44c0a1710 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:07:51.505 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.489+0000 7fd453f29640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd44c1393a0 0x7fd44c134300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:51.505 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.489+0000 7fd453f29640 10 monclient(hunting): picked mon.noname-b con 0x7fd44c1393a0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:07:51.505 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.489+0000 7fd453f29640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:51.505 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.489+0000 7fd453f29640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fd44c13e880 con 0x7fd44c0a1330 2024-09-16T04:07:51.505 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.489+0000 7fd453f29640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:51.505 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.489+0000 7fd453f29640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd44c13e700 con 0x7fd44c1393a0 2024-09-16T04:07:51.505 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.489+0000 7fd453f29640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:51.505 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.489+0000 7fd453f29640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fd44c13ea00 con 0x7fd44c0a1710 2024-09-16T04:07:51.505 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.489+0000 7fd453f29640 10 monclient(hunting): _renew_subs 2024-09-16T04:07:51.505 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.489+0000 7fd453f29640 10 monclient(hunting): authenticate will time out at 2518.460392s 2024-09-16T04:07:51.505 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.489+0000 7fd451cc5640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd44c0a1330 0x7fd44c1c9bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:51.505 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.489+0000 7fd4514c4640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd44c0a1710 0x7fd44c071700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:51.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.489+0000 7fd451cc5640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd44c0a1330 0x7fd44c1c9bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:45248/0 (socket says 172.21.15.42:45248) 2024-09-16T04:07:51.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.489+0000 7fd451cc5640 1 -- 172.21.15.42:0/3560964453 learned_addr learned my addr 172.21.15.42:0/3560964453 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:07:51.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.489+0000 7fd451cc5640 10 monclient(hunting): get_auth_request con 0x7fd44c0a1330 auth_method 0 2024-09-16T04:07:51.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.489+0000 7fd451cc5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:51.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.489+0000 7fd451cc5640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:51.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.489+0000 7fd451cc5640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:51.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.489+0000 7fd4514c4640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd44c0a1710 0x7fd44c071700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:52036/0 (socket says 172.21.15.42:52036) 2024-09-16T04:07:51.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.489+0000 7fd4514c4640 10 monclient(hunting): get_auth_request con 0x7fd44c0a1710 auth_method 0 2024-09-16T04:07:51.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.489+0000 7fd4514c4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:51.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.489+0000 7fd4514c4640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:51.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.489+0000 7fd4514c4640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:51.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.489+0000 7fd451cc5640 1 -- 172.21.15.42:0/3560964453 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd44c0a1330 msgr2=0x7fd44c1c9bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-16T04:07:51.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.489+0000 7fd451cc5640 1 -- 172.21.15.42:0/3560964453 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd44c0a1330 msgr2=0x7fd44c1c9bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:07:51.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.489+0000 7fd451cc5640 1 --2- 172.21.15.42:0/3560964453 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd44c0a1330 0x7fd44c1c9bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:07:51.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.489+0000 7fd4514c4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:07:51.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.489+0000 7fd4514c4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:07:51.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.489+0000 7fd4514c4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:07:51.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.489+0000 7fd451cc5640 1 --2- 172.21.15.42:0/3560964453 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd44c0a1330 0x7fd44c1c9bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:07:51.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.489+0000 7fd4524c6640 1 --2- 172.21.15.42:0/3560964453 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd44c1393a0 0x7fd44c134300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:51.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.489+0000 7fd4514c4640 10 monclient(hunting): handle_auth_done global_id 1014185 payload 824 2024-09-16T04:07:51.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.489+0000 7fd4514c4640 1 -- 172.21.15.42:0/3560964453 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd44c1393a0 msgr2=0x7fd44c134300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:51.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.489+0000 7fd4514c4640 1 --2- 172.21.15.42:0/3560964453 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd44c1393a0 0x7fd44c134300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:51.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.489+0000 7fd4514c4640 1 -- 172.21.15.42:0/3560964453 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd44c0a1330 msgr2=0x7fd44c1c9bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:07:51.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.489+0000 7fd4514c4640 1 --2- 172.21.15.42:0/3560964453 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd44c0a1330 0x7fd44c1c9bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:51.508 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.489+0000 7fd4514c4640 10 monclient: _finish_hunting 0 2024-09-16T04:07:51.508 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.489+0000 7fd4514c4640 1 monclient: found mon.noname-c 2024-09-16T04:07:51.508 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.489+0000 7fd4514c4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:07:51.508 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.489+0000 7fd4514c4640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T04:07:51.508 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.489+0000 7fd4514c4640 1 -- 172.21.15.42:0/3560964453 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd44c072390 con 0x7fd44c0a1710 2024-09-16T04:07:51.508 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.489+0000 7fd4514c4640 1 --2- 172.21.15.42:0/3560964453 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd44c0a1710 0x7fd44c071700 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fd44c09cee0 tx=0x7fd440019100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:07:51.508 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.489+0000 7fd44a7fc640 10 client.?.objecter ms_handle_connect 0x7fd44c0a1710 2024-09-16T04:07:51.508 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.489+0000 7fd44a7fc640 10 client.?.objecter resend_mon_ops 2024-09-16T04:07:51.508 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.489+0000 7fd44a7fc640 1 -- 172.21.15.42:0/3560964453 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4400b0040 con 0x7fd44c0a1710 2024-09-16T04:07:51.508 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.489+0000 7fd44a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:51.508 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.489+0000 7fd44a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:07:51.508 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.489+0000 7fd44a7fc640 10 monclient: dump: 2024-09-16T04:07:51.509 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:07:51.509 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:51.509 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:51.509 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:51.509 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:51.509 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:51.509 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:07:51.509 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:07:51.509 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:07:51.510 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:51.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.489+0000 7fd44a7fc640 10 monclient: _finish_auth 0 2024-09-16T04:07:51.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.489+0000 7fd44a7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:07:51.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.489+0000 7fd44a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:07:51.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.489+0000 7fd44a7fc640 1 -- 172.21.15.42:0/3560964453 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd4400b9070 con 0x7fd44c0a1710 2024-09-16T04:07:51.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.489+0000 7fd453f29640 5 monclient: authenticate success, global_id 1014185 2024-09-16T04:07:51.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.489+0000 7fd453f29640 10 monclient: _renew_subs 2024-09-16T04:07:51.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.489+0000 7fd453f29640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:07:51.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.489+0000 7fd453f29640 1 -- 172.21.15.42:0/3560964453 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd44c139a40 con 0x7fd44c0a1710 2024-09-16T04:07:51.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.489+0000 7fd44a7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:07:51.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.489+0000 7fd44a7fc640 1 -- 172.21.15.42:0/3560964453 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4400b4850 con 0x7fd44c0a1710 2024-09-16T04:07:51.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.489+0000 7fd44a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:51.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.489+0000 7fd44a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:07:51.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.489+0000 7fd44a7fc640 10 monclient: dump: 2024-09-16T04:07:51.511 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:07:51.511 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:51.511 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:51.511 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:51.511 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:51.511 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:51.511 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:07:51.512 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:07:51.512 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:07:51.512 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:51.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.489+0000 7fd44bfff640 4 set_mon_vals no callback set 2024-09-16T04:07:51.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.489+0000 7fd453f29640 10 client.1014185.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:07:51.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.489+0000 7fd453f29640 10 monclient: _renew_subs 2024-09-16T04:07:51.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.489+0000 7fd453f29640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:07:51.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.489+0000 7fd453f29640 1 -- 172.21.15.42:0/3560964453 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd44c11ecb0 con 0x7fd44c0a1710 2024-09-16T04:07:51.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.489+0000 7fd453f29640 1 librados: init done 2024-09-16T04:07:51.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.489+0000 7fd453f29640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:07:51.513 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.489+0000 7fd453f29640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:07:51.513 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.489+0000 7fd453f29640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:07:51.513 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.489+0000 7fd453f29640 1 -- 172.21.15.42:0/3560964453 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd44c0a1b10 con 0x7fd44c0a1710 2024-09-16T04:07:51.513 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.489+0000 7fd44a7fc640 1 -- 172.21.15.42:0/3560964453 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fd4400ab4c0 con 0x7fd44c0a1710 2024-09-16T04:07:51.513 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.489+0000 7fd44a7fc640 1 --2- 172.21.15.42:0/3560964453 >> v2:172.21.15.154:6812/254845342 conn(0x7fd438037840 0x7fd438039d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:51.513 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.489+0000 7fd44a7fc640 1 -- 172.21.15.42:0/3560964453 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(706..706 src has 451..706) ==== 5040+0+0 (secure 0 0 0) 0x7fd4400e33c0 con 0x7fd44c0a1710 2024-09-16T04:07:51.513 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.489+0000 7fd44a7fc640 10 client.1014185.objecter ms_dispatch 0x7fd44c000ba0 osd_map(706..706 src has 451..706) 2024-09-16T04:07:51.513 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.489+0000 7fd44a7fc640 3 client.1014185.objecter handle_osd_map got epochs [706,706] > 0 2024-09-16T04:07:51.513 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.489+0000 7fd44a7fc640 3 client.1014185.objecter handle_osd_map decoding full epoch 706 2024-09-16T04:07:51.513 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.489+0000 7fd44a7fc640 20 client.1014185.objecter dump_active .. 0 homeless 2024-09-16T04:07:51.513 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.489+0000 7fd451cc5640 1 --2- 172.21.15.42:0/3560964453 >> v2:172.21.15.154:6812/254845342 conn(0x7fd438037840 0x7fd438039d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:51.513 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.489+0000 7fd451cc5640 10 monclient: get_auth_request con 0x7fd438037840 auth_method 0 2024-09-16T04:07:51.513 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.493+0000 7fd451cc5640 1 --2- 172.21.15.42:0/3560964453 >> v2:172.21.15.154:6812/254845342 conn(0x7fd438037840 0x7fd438039d00 secure :-1 s=READY pgs=3887 cs=0 l=1 rev1=1 crypto rx=0x7fd43c0018f0 tx=0x7fd43c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:07:51.514 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.493+0000 7fd44a7fc640 10 client.1014185.objecter ms_handle_connect 0x7fd438037840 2024-09-16T04:07:51.514 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.493+0000 7fd44a7fc640 1 -- 172.21.15.42:0/3560964453 <== mon.2 v2:172.21.15.42:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7fd440016070 con 0x7fd44c0a1710 2024-09-16T04:07:51.514 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.493+0000 7fd44a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:07:51.514 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.493+0000 7fd44a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:07:51.728 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.721+0000 7fd453f29640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:07:51.728 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.721+0000 7fd453f29640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:07:51.728 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.721+0000 7fd453f29640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:07:51.728 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.721+0000 7fd453f29640 1 -- 172.21.15.42:0/3560964453 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fd44c11e650 con 0x7fd44c0a1710 2024-09-16T04:07:51.728 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.721+0000 7fd44a7fc640 1 -- 172.21.15.42:0/3560964453 <== mon.2 v2:172.21.15.42:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7fd4400c79e0 con 0x7fd44c0a1710 2024-09-16T04:07:51.729 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.721+0000 7fd44a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:07:51.729 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.721+0000 7fd44a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:07:51.729 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":234,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":24,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:07:51.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.725+0000 7fd453f29640 10 librados: watch_flush enter 2024-09-16T04:07:51.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.725+0000 7fd453f29640 10 librados: watch_flush exit 2024-09-16T04:07:51.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.725+0000 7fd453f29640 20 client.1014185.objecter shutdown clearing up homeless session... 2024-09-16T04:07:51.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.725+0000 7fd453f29640 10 client.1014185.objecter successfully canceled tick 2024-09-16T04:07:51.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.725+0000 7fd453f29640 1 -- 172.21.15.42:0/3560964453 >> v2:172.21.15.154:6812/254845342 conn(0x7fd438037840 msgr2=0x7fd438039d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:51.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.725+0000 7fd453f29640 1 --2- 172.21.15.42:0/3560964453 >> v2:172.21.15.154:6812/254845342 conn(0x7fd438037840 0x7fd438039d00 secure :-1 s=READY pgs=3887 cs=0 l=1 rev1=1 crypto rx=0x7fd43c0018f0 tx=0x7fd43c009040 comp rx=0 tx=0).stop 2024-09-16T04:07:51.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.725+0000 7fd453f29640 10 monclient: shutdown 2024-09-16T04:07:51.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.725+0000 7fd453f29640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:07:51.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.725+0000 7fd453f29640 1 -- 172.21.15.42:0/3560964453 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd44c0a1710 msgr2=0x7fd44c071700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:51.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.725+0000 7fd453f29640 1 --2- 172.21.15.42:0/3560964453 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd44c0a1710 0x7fd44c071700 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fd44c09cee0 tx=0x7fd440019100 comp rx=0 tx=0).stop 2024-09-16T04:07:51.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.725+0000 7fd453f29640 1 -- 172.21.15.42:0/3560964453 shutdown_connections 2024-09-16T04:07:51.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.725+0000 7fd453f29640 1 --2- 172.21.15.42:0/3560964453 >> v2:172.21.15.154:6812/254845342 conn(0x7fd438037840 0x7fd438039d00 unknown :-1 s=CLOSED pgs=3887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:51.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.725+0000 7fd453f29640 1 --2- 172.21.15.42:0/3560964453 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd44c1393a0 0x7fd44c134300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:51.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.725+0000 7fd453f29640 1 --2- 172.21.15.42:0/3560964453 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd44c0a1710 0x7fd44c071700 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:51.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.725+0000 7fd453f29640 1 --2- 172.21.15.42:0/3560964453 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd44c0a1330 0x7fd44c1c9bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:51.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.725+0000 7fd453f29640 1 -- 172.21.15.42:0/3560964453 >> 172.21.15.42:0/3560964453 conn(0x7fd44c0a2420 msgr2=0x7fd44c0a3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:07:51.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.725+0000 7fd453f29640 1 -- 172.21.15.42:0/3560964453 shutdown_connections 2024-09-16T04:07:51.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.725+0000 7fd453f29640 1 -- 172.21.15.42:0/3560964453 wait complete. 2024-09-16T04:07:51.733 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:51.725+0000 7fd453f29640 1 librados: shutdown 2024-09-16T04:07:54.749 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:07:54.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.877+0000 7fb89000a640 10 monclient: get_monmap_and_config 2024-09-16T04:07:54.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.877+0000 7fb89000a640 10 monclient: build_initial_monmap 2024-09-16T04:07:54.883 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.877+0000 7fb89000a640 10 monclient: monmap: 2024-09-16T04:07:54.883 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:07:54.883 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:54.883 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:07:54.881287+0000 2024-09-16T04:07:54.883 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:07:54.881287+0000 2024-09-16T04:07:54.883 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:07:54.883 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:54.883 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:07:54.883 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:07:54.883 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:07:54.883 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:07:54.883 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:07:54.884 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:54.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.877+0000 7fb89000a640 1 Processor -- start 2024-09-16T04:07:54.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.877+0000 7fb89000a640 1 -- start start 2024-09-16T04:07:54.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.877+0000 7fb89000a640 10 monclient: init 2024-09-16T04:07:54.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.877+0000 7fb89000a640 10 monclient: _reopen_session rank -1 2024-09-16T04:07:54.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.877+0000 7fb89000a640 10 monclient: _add_conns ranks=[1,0,2,4,3] 2024-09-16T04:07:54.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.877+0000 7fb89000a640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb8880a1330 addr v1:172.21.15.154:6789/0 2024-09-16T04:07:54.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.877+0000 7fb89000a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb8880a1710 addr v1:172.21.15.42:6789/0 2024-09-16T04:07:54.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.877+0000 7fb89000a640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb88813a3b0 0x7fb88813c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:54.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.877+0000 7fb89000a640 10 monclient(hunting): picked mon.noname-b con 0x7fb88813a3b0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:07:54.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.877+0000 7fb89000a640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb88809e4c0 con 0x7fb8880a1710 2024-09-16T04:07:54.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.877+0000 7fb89000a640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb8880a3170 con 0x7fb8880a1330 2024-09-16T04:07:54.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.877+0000 7fb89000a640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:54.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.877+0000 7fb89000a640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fb88809edf0 con 0x7fb88813a3b0 2024-09-16T04:07:54.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.877+0000 7fb89000a640 10 monclient(hunting): _renew_subs 2024-09-16T04:07:54.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.877+0000 7fb89000a640 10 monclient(hunting): authenticate will time out at 2521.851535s 2024-09-16T04:07:54.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.877+0000 7fb88e5a7640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb88813a3b0 0x7fb88813c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:54.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.877+0000 7fb88e5a7640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb88813a3b0 0x7fb88813c7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:37144/0 (socket says 172.21.15.42:37144) 2024-09-16T04:07:54.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.877+0000 7fb88e5a7640 1 -- 172.21.15.42:0/64139543 learned_addr learned my addr 172.21.15.42:0/64139543 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:07:54.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.877+0000 7fb88e5a7640 10 monclient(hunting): get_auth_request con 0x7fb88813a3b0 auth_method 0 2024-09-16T04:07:54.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.877+0000 7fb88e5a7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:54.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.877+0000 7fb88e5a7640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:54.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.877+0000 7fb88e5a7640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:54.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.877+0000 7fb88d5a5640 1 --1- 172.21.15.42:0/64139543 >> v1:172.21.15.42:6789/0 conn(0x7fb8880a1710 0x7fb8880a1b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:44520/0 (socket says 172.21.15.42:44520) 2024-09-16T04:07:54.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.877+0000 7fb88e5a7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:07:54.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.877+0000 7fb88e5a7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:07:54.888 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.877+0000 7fb88e5a7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:07:54.888 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.881+0000 7fb88e5a7640 10 monclient(hunting): handle_auth_done global_id 1014188 payload 274 2024-09-16T04:07:54.888 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.881+0000 7fb88e5a7640 1 -- 172.21.15.42:0/64139543 >> v1:172.21.15.154:6789/0 conn(0x7fb8880a1330 legacy=0x7fb88809f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:07:54.888 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.881+0000 7fb88e5a7640 1 -- 172.21.15.42:0/64139543 >> v1:172.21.15.42:6789/0 conn(0x7fb8880a1710 legacy=0x7fb8880a1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:54.888 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.881+0000 7fb88cda4640 1 -- 172.21.15.42:0/64139543 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2010269613 0 0) 0x7fb88809e4c0 con 0x7fb8880a1710 2024-09-16T04:07:54.888 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.881+0000 7fb88e5a7640 10 monclient: _finish_hunting 0 2024-09-16T04:07:54.888 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.881+0000 7fb88e5a7640 1 monclient: found mon.noname-b 2024-09-16T04:07:54.888 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.881+0000 7fb88e5a7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:07:54.888 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.881+0000 7fb88e5a7640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T04:07:54.888 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.881+0000 7fb88e5a7640 1 -- 172.21.15.42:0/64139543 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb88809d030 con 0x7fb88813a3b0 2024-09-16T04:07:54.888 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.881+0000 7fb88cda4640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T04:07:54.895 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.881+0000 7fb88e5a7640 1 --2- 172.21.15.42:0/64139543 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb88813a3b0 0x7fb88813c7a0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fb880009e80 tx=0x7fb88001c910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ed8ad4e3b515ba0e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:07:54.895 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.881+0000 7fb88e5a7640 1 -- 172.21.15.42:0/64139543 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb88813a3b0 msgr2=0x7fb88813c7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 15 2024-09-16T04:07:54.895 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.881+0000 7fb88e5a7640 1 -- 172.21.15.42:0/64139543 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb88813a3b0 msgr2=0x7fb88813c7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:07:54.895 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.881+0000 7fb88e5a7640 1 --2- 172.21.15.42:0/64139543 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb88813a3b0 0x7fb88813c7a0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fb880009e80 tx=0x7fb88001c910 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:07:54.895 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.881+0000 7fb88e5a7640 1 --2- 172.21.15.42:0/64139543 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb88813a3b0 0x7fb88813c7a0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fb880009e80 tx=0x7fb88001c910 comp rx=0 tx=0).stop 2024-09-16T04:07:54.895 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.881+0000 7fb88cda4640 10 monclient: ms_handle_reset current mon [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:07:54.895 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.881+0000 7fb88cda4640 10 monclient: _reopen_session rank -1 2024-09-16T04:07:54.896 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.881+0000 7fb88cda4640 1 -- 172.21.15.42:0/64139543 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb88813a3b0 msgr2=0x7fb88813c7a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:07:54.896 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.881+0000 7fb88cda4640 1 --2- 172.21.15.42:0/64139543 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb88813a3b0 0x7fb88813c7a0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:54.896 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.881+0000 7fb88cda4640 10 monclient: _add_conns ranks=[4,3] 2024-09-16T04:07:54.896 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.881+0000 7fb88cda4640 1 --2- 172.21.15.42:0/64139543 >> v2:172.21.15.154:3300/0 conn(0x7fb8740056e0 0x7fb874007b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:54.896 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.881+0000 7fb88cda4640 10 monclient(hunting): picked mon.noname-c con 0x7fb8740056e0 addr v2:172.21.15.154:3300/0 2024-09-16T04:07:54.896 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.881+0000 7fb88cda4640 1 --2- 172.21.15.42:0/64139543 >> v2:172.21.15.42:3300/0 conn(0x7fb874008400 0x7fb87400a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:54.896 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.881+0000 7fb88cda4640 10 monclient(hunting): picked mon.noname-a con 0x7fb874008400 addr v2:172.21.15.42:3300/0 2024-09-16T04:07:54.896 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.881+0000 7fb88cda4640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:54.896 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.881+0000 7fb88cda4640 1 -- 172.21.15.42:0/64139543 --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7fb87400b190 con 0x7fb874008400 2024-09-16T04:07:54.896 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.881+0000 7fb88cda4640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:54.896 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.881+0000 7fb88cda4640 1 -- 172.21.15.42:0/64139543 --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fb87400b340 con 0x7fb8740056e0 2024-09-16T04:07:54.896 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.881+0000 7fb88cda4640 10 monclient(hunting): _renew_subs 2024-09-16T04:07:54.896 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.881+0000 7fb88cda4640 10 monclient(hunting): ms_handle_reset stray mon [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:07:54.897 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.881+0000 7fb88d5a5640 1 --2- 172.21.15.42:0/64139543 >> v2:172.21.15.42:3300/0 conn(0x7fb874008400 0x7fb87400a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:54.897 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.881+0000 7fb88d5a5640 10 monclient(hunting): get_auth_request con 0x7fb874008400 auth_method 0 2024-09-16T04:07:54.897 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.881+0000 7fb88d5a5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:54.897 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.881+0000 7fb88d5a5640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:54.897 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.881+0000 7fb88d5a5640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:07:54.897 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.881+0000 7fb88d5a5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:07:54.897 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.881+0000 7fb88d5a5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:07:54.897 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.881+0000 7fb88d5a5640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:07:54.897 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.881+0000 7fb88dda6640 1 --2- 172.21.15.42:0/64139543 >> v2:172.21.15.154:3300/0 conn(0x7fb8740056e0 0x7fb874007b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:54.897 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.881+0000 7fb88d5a5640 10 monclient(hunting): handle_auth_done global_id 1014188 payload 293 2024-09-16T04:07:54.897 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.881+0000 7fb88d5a5640 1 -- 172.21.15.42:0/64139543 >> v2:172.21.15.154:3300/0 conn(0x7fb8740056e0 msgr2=0x7fb874007b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:07:54.897 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.881+0000 7fb88d5a5640 1 --2- 172.21.15.42:0/64139543 >> v2:172.21.15.154:3300/0 conn(0x7fb8740056e0 0x7fb874007b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:54.897 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.881+0000 7fb88d5a5640 10 monclient: _finish_hunting 0 2024-09-16T04:07:54.897 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.881+0000 7fb88d5a5640 1 monclient: found mon.noname-a 2024-09-16T04:07:54.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.881+0000 7fb88d5a5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:07:54.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.881+0000 7fb88d5a5640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:07:54.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.881+0000 7fb88d5a5640 1 -- 172.21.15.42:0/64139543 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb88809e4c0 con 0x7fb874008400 2024-09-16T04:07:54.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.881+0000 7fb88dda6640 10 monclient: get_auth_request con 0x7fb8740056e0 auth_method 0 2024-09-16T04:07:54.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.881+0000 7fb88dda6640 1 --2- 172.21.15.42:0/64139543 >> v2:172.21.15.154:3300/0 conn(0x7fb8740056e0 0x7fb874007b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T04:07:54.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.881+0000 7fb88d5a5640 1 --2- 172.21.15.42:0/64139543 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb874008400 0x7fb87400a8c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fb874005690 tx=0x7fb880009810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6c178fd14d9dce8 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:07:54.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.881+0000 7fb88cda4640 1 -- 172.21.15.42:0/64139543 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb878001580 con 0x7fb874008400 2024-09-16T04:07:54.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.881+0000 7fb88cda4640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:54.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.881+0000 7fb88cda4640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:07:54.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.881+0000 7fb88cda4640 10 monclient: dump: 2024-09-16T04:07:54.898 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:07:54.898 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:54.898 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:54.898 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:54.899 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:54.899 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:54.899 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:07:54.899 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:07:54.899 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:07:54.899 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:54.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.881+0000 7fb88cda4640 10 monclient: _finish_auth 0 2024-09-16T04:07:54.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.881+0000 7fb88cda4640 10 monclient: _check_auth_tickets 2024-09-16T04:07:54.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.881+0000 7fb88cda4640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:07:54.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.881+0000 7fb88cda4640 1 -- 172.21.15.42:0/64139543 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb8780072f0 con 0x7fb874008400 2024-09-16T04:07:54.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.881+0000 7fb88cda4640 10 monclient: handle_config config(0 keys) 2024-09-16T04:07:54.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.881+0000 7fb89000a640 5 monclient: authenticate success, global_id 1014188 2024-09-16T04:07:54.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.881+0000 7fb88cda4640 1 -- 172.21.15.42:0/64139543 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8780047f0 con 0x7fb874008400 2024-09-16T04:07:54.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.881+0000 7fb88cda4640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:54.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.881+0000 7fb88cda4640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:07:54.900 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.881+0000 7fb88cda4640 10 monclient: dump: 2024-09-16T04:07:54.900 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:07:54.900 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:54.900 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:54.900 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:54.900 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:54.900 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:54.900 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:07:54.900 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:07:54.900 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:07:54.900 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:54.900 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.881+0000 7fb89000a640 10 monclient: get_monmap_and_config success 2024-09-16T04:07:54.900 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.881+0000 7fb89000a640 4 set_mon_vals no callback set 2024-09-16T04:07:54.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.881+0000 7fb89000a640 10 monclient: shutdown 2024-09-16T04:07:54.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.881+0000 7fb89000a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:07:54.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.881+0000 7fb89000a640 1 -- 172.21.15.42:0/64139543 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb874008400 msgr2=0x7fb87400a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:54.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.881+0000 7fb89000a640 1 --2- 172.21.15.42:0/64139543 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb874008400 0x7fb87400a8c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fb874005690 tx=0x7fb880009810 comp rx=0 tx=0).stop 2024-09-16T04:07:54.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.881+0000 7fb88e5a7640 1 -- 172.21.15.42:0/64139543 reap_dead start 2024-09-16T04:07:54.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.885+0000 7fb89000a640 1 -- 172.21.15.42:0/64139543 shutdown_connections 2024-09-16T04:07:54.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.885+0000 7fb89000a640 1 --2- 172.21.15.42:0/64139543 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb874008400 0x7fb87400a8c0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:54.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.885+0000 7fb89000a640 1 -- 172.21.15.42:0/64139543 >> 172.21.15.42:0/64139543 conn(0x7fb8880a2420 msgr2=0x7fb8880a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:07:54.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.885+0000 7fb89000a640 1 -- 172.21.15.42:0/64139543 shutdown_connections 2024-09-16T04:07:54.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.885+0000 7fb89000a640 1 -- 172.21.15.42:0/64139543 wait complete. 2024-09-16T04:07:54.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.885+0000 7fb89000a640 10 monclient: build_initial_monmap 2024-09-16T04:07:54.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.885+0000 7fb89000a640 10 monclient: monmap: 2024-09-16T04:07:54.901 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:07:54.901 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:07:54.902 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:07:54.902 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:07:54.902 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:07:54.902 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:54.902 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:07:54.902 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:07:54.902 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:07:54.902 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:54.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.885+0000 7fb89000a640 1 librados: starting msgr at 2024-09-16T04:07:54.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.885+0000 7fb89000a640 1 librados: starting objecter 2024-09-16T04:07:54.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.885+0000 7fb89000a640 1 Processor -- start 2024-09-16T04:07:54.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.885+0000 7fb89000a640 1 -- start start 2024-09-16T04:07:54.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.885+0000 7fb89000a640 1 librados: setting wanted keys 2024-09-16T04:07:54.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.885+0000 7fb89000a640 1 librados: calling monclient init 2024-09-16T04:07:54.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.885+0000 7fb89000a640 10 monclient: init 2024-09-16T04:07:54.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.885+0000 7fb89000a640 10 monclient: _reopen_session rank -1 2024-09-16T04:07:54.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.885+0000 7fb89000a640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:07:54.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.885+0000 7fb89000a640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb874008400 0x7fb888071d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:54.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.885+0000 7fb89000a640 10 monclient(hunting): picked mon.noname-b con 0x7fb874008400 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:07:54.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.885+0000 7fb89000a640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb888132210 0x7fb888132630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:54.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.885+0000 7fb89000a640 10 monclient(hunting): picked mon.noname-a con 0x7fb888132210 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:07:54.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.885+0000 7fb89000a640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb888132b70 0x7fb8881d5910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:54.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.885+0000 7fb89000a640 10 monclient(hunting): picked mon.noname-c con 0x7fb888132b70 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:07:54.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.885+0000 7fb89000a640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:54.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.885+0000 7fb89000a640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fb88809bea0 con 0x7fb888132210 2024-09-16T04:07:54.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.885+0000 7fb89000a640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:54.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.885+0000 7fb89000a640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fb88809bd20 con 0x7fb874008400 2024-09-16T04:07:54.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.885+0000 7fb89000a640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:54.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.885+0000 7fb89000a640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fb88809c020 con 0x7fb888132b70 2024-09-16T04:07:54.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.885+0000 7fb89000a640 10 monclient(hunting): _renew_subs 2024-09-16T04:07:54.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.885+0000 7fb89000a640 10 monclient(hunting): authenticate will time out at 2521.857699s 2024-09-16T04:07:54.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.885+0000 7fb88e5a7640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb888132b70 0x7fb8881d5910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:54.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.885+0000 7fb88d5a5640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb888132210 0x7fb888132630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:54.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.885+0000 7fb88e5a7640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb888132b70 0x7fb8881d5910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:37158/0 (socket says 172.21.15.42:37158) 2024-09-16T04:07:54.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.885+0000 7fb88d5a5640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb888132210 0x7fb888132630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:43398/0 (socket says 172.21.15.42:43398) 2024-09-16T04:07:54.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.885+0000 7fb88e5a7640 1 -- 172.21.15.42:0/1230567237 learned_addr learned my addr 172.21.15.42:0/1230567237 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:07:54.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.885+0000 7fb88e5a7640 10 monclient(hunting): get_auth_request con 0x7fb888132b70 auth_method 0 2024-09-16T04:07:54.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.885+0000 7fb88e5a7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:54.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.885+0000 7fb88e5a7640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:54.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.885+0000 7fb88e5a7640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:54.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.885+0000 7fb88d5a5640 10 monclient(hunting): get_auth_request con 0x7fb888132210 auth_method 0 2024-09-16T04:07:54.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.885+0000 7fb88d5a5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:54.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.885+0000 7fb88d5a5640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:54.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.885+0000 7fb88d5a5640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:54.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.885+0000 7fb88d5a5640 1 -- 172.21.15.42:0/1230567237 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb888132210 msgr2=0x7fb888132630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-16T04:07:54.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.885+0000 7fb88d5a5640 1 -- 172.21.15.42:0/1230567237 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb888132210 msgr2=0x7fb888132630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:07:54.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.885+0000 7fb88e5a7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:07:54.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.885+0000 7fb88e5a7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:07:54.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.885+0000 7fb88d5a5640 1 --2- 172.21.15.42:0/1230567237 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb888132210 0x7fb888132630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:07:54.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.885+0000 7fb88e5a7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:07:54.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.885+0000 7fb88d5a5640 1 --2- 172.21.15.42:0/1230567237 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb888132210 0x7fb888132630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:07:54.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.885+0000 7fb88dda6640 1 --2- 172.21.15.42:0/1230567237 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb874008400 0x7fb888071d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:54.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.885+0000 7fb88dda6640 10 monclient(hunting): get_auth_request con 0x7fb874008400 auth_method 0 2024-09-16T04:07:54.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.885+0000 7fb88dda6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:54.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.885+0000 7fb88dda6640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:54.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.885+0000 7fb88dda6640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:54.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.885+0000 7fb88e5a7640 10 monclient(hunting): handle_auth_done global_id 1014191 payload 824 2024-09-16T04:07:54.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.885+0000 7fb88e5a7640 1 -- 172.21.15.42:0/1230567237 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb874008400 msgr2=0x7fb888071d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:54.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.885+0000 7fb88e5a7640 1 --2- 172.21.15.42:0/1230567237 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb874008400 0x7fb888071d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:54.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.885+0000 7fb88e5a7640 1 -- 172.21.15.42:0/1230567237 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb888132210 msgr2=0x7fb888132630 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:07:54.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.885+0000 7fb88e5a7640 1 --2- 172.21.15.42:0/1230567237 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb888132210 0x7fb888132630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:54.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.885+0000 7fb88e5a7640 10 monclient: _finish_hunting 0 2024-09-16T04:07:54.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.885+0000 7fb88e5a7640 1 monclient: found mon.noname-c 2024-09-16T04:07:54.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.885+0000 7fb88e5a7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:07:54.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.885+0000 7fb88e5a7640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T04:07:54.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.885+0000 7fb88e5a7640 1 -- 172.21.15.42:0/1230567237 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb88811c110 con 0x7fb888132b70 2024-09-16T04:07:54.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.885+0000 7fb88e5a7640 1 --2- 172.21.15.42:0/1230567237 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb888132b70 0x7fb8881d5910 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fb88001a540 tx=0x7fb8800196b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:07:54.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.885+0000 7fb87e7fc640 10 client.?.objecter ms_handle_connect 0x7fb888132b70 2024-09-16T04:07:54.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.885+0000 7fb87e7fc640 10 client.?.objecter resend_mon_ops 2024-09-16T04:07:54.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.885+0000 7fb87e7fc640 1 -- 172.21.15.42:0/1230567237 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb880003e60 con 0x7fb888132b70 2024-09-16T04:07:54.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.885+0000 7fb87e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:54.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.885+0000 7fb87e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:07:54.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.885+0000 7fb87e7fc640 10 monclient: dump: 2024-09-16T04:07:54.907 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:07:54.907 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:54.907 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:54.907 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:54.907 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:54.907 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:54.907 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:07:54.907 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:07:54.907 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:07:54.907 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:54.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.885+0000 7fb87e7fc640 10 monclient: _finish_auth 0 2024-09-16T04:07:54.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.885+0000 7fb87e7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:07:54.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.885+0000 7fb87e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:07:54.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.885+0000 7fb87e7fc640 1 -- 172.21.15.42:0/1230567237 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb8800a5c20 con 0x7fb888132b70 2024-09-16T04:07:54.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.885+0000 7fb89000a640 5 monclient: authenticate success, global_id 1014191 2024-09-16T04:07:54.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.885+0000 7fb89000a640 10 monclient: _renew_subs 2024-09-16T04:07:54.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.885+0000 7fb89000a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:07:54.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.885+0000 7fb89000a640 1 -- 172.21.15.42:0/1230567237 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb888133e90 con 0x7fb888132b70 2024-09-16T04:07:54.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.885+0000 7fb87e7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:07:54.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.885+0000 7fb87e7fc640 1 -- 172.21.15.42:0/1230567237 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb880004040 con 0x7fb888132b70 2024-09-16T04:07:54.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.885+0000 7fb87e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:54.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.885+0000 7fb87e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:07:54.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.885+0000 7fb87e7fc640 10 monclient: dump: 2024-09-16T04:07:54.908 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:07:54.908 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:54.909 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:54.909 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:54.909 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:54.909 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:54.909 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:07:54.909 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:07:54.909 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:07:54.909 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:54.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.885+0000 7fb89000a640 10 client.1014191.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:07:54.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.885+0000 7fb89000a640 10 monclient: _renew_subs 2024-09-16T04:07:54.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.885+0000 7fb89000a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:07:54.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.885+0000 7fb89000a640 1 -- 172.21.15.42:0/1230567237 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8880a4ea0 con 0x7fb888132b70 2024-09-16T04:07:54.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.885+0000 7fb89000a640 1 librados: init done 2024-09-16T04:07:54.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.885+0000 7fb87ffff640 4 set_mon_vals no callback set 2024-09-16T04:07:54.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.885+0000 7fb89000a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:07:54.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.885+0000 7fb89000a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:07:54.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.885+0000 7fb89000a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:07:54.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.885+0000 7fb89000a640 1 -- 172.21.15.42:0/1230567237 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb8880a1b20 con 0x7fb888132b70 2024-09-16T04:07:54.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.885+0000 7fb87e7fc640 1 -- 172.21.15.42:0/1230567237 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fb8800be030 con 0x7fb888132b70 2024-09-16T04:07:54.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.885+0000 7fb87e7fc640 1 --2- 172.21.15.42:0/1230567237 >> v2:172.21.15.154:6812/254845342 conn(0x7fb86c037840 0x7fb86c039d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:54.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.885+0000 7fb87e7fc640 1 -- 172.21.15.42:0/1230567237 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(706..706 src has 451..706) ==== 5040+0+0 (secure 0 0 0) 0x7fb8800e36f0 con 0x7fb888132b70 2024-09-16T04:07:54.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.885+0000 7fb87e7fc640 10 client.1014191.objecter ms_dispatch 0x7fb888000ba0 osd_map(706..706 src has 451..706) 2024-09-16T04:07:54.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.885+0000 7fb87e7fc640 3 client.1014191.objecter handle_osd_map got epochs [706,706] > 0 2024-09-16T04:07:54.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.885+0000 7fb87e7fc640 3 client.1014191.objecter handle_osd_map decoding full epoch 706 2024-09-16T04:07:54.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.885+0000 7fb87e7fc640 20 client.1014191.objecter dump_active .. 0 homeless 2024-09-16T04:07:54.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.889+0000 7fb88dda6640 1 --2- 172.21.15.42:0/1230567237 >> v2:172.21.15.154:6812/254845342 conn(0x7fb86c037840 0x7fb86c039d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:54.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.889+0000 7fb88dda6640 10 monclient: get_auth_request con 0x7fb86c037840 auth_method 0 2024-09-16T04:07:54.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.889+0000 7fb88dda6640 1 --2- 172.21.15.42:0/1230567237 >> v2:172.21.15.154:6812/254845342 conn(0x7fb86c037840 0x7fb86c039d00 secure :-1 s=READY pgs=3888 cs=0 l=1 rev1=1 crypto rx=0x7fb884007fd0 tx=0x7fb884009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:07:54.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.889+0000 7fb87e7fc640 10 client.1014191.objecter ms_handle_connect 0x7fb86c037840 2024-09-16T04:07:54.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.893+0000 7fb87e7fc640 1 -- 172.21.15.42:0/1230567237 <== mon.2 v2:172.21.15.42:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7fb8800c5020 con 0x7fb888132b70 2024-09-16T04:07:54.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.893+0000 7fb87e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:07:54.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:54.893+0000 7fb87e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:07:55.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.117+0000 7fb89000a640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:07:55.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.117+0000 7fb89000a640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:07:55.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.117+0000 7fb89000a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:07:55.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.117+0000 7fb89000a640 1 -- 172.21.15.42:0/1230567237 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fb888132ff0 con 0x7fb888132b70 2024-09-16T04:07:55.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.117+0000 7fb87e7fc640 1 -- 172.21.15.42:0/1230567237 <== mon.2 v2:172.21.15.42:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7fb880016620 con 0x7fb888132b70 2024-09-16T04:07:55.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.117+0000 7fb87e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:07:55.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.117+0000 7fb87e7fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:07:55.124 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":238,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:07:55.126 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.117+0000 7fb89000a640 10 librados: watch_flush enter 2024-09-16T04:07:55.126 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.117+0000 7fb89000a640 10 librados: watch_flush exit 2024-09-16T04:07:55.126 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.117+0000 7fb89000a640 20 client.1014191.objecter shutdown clearing up homeless session... 2024-09-16T04:07:55.126 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.117+0000 7fb89000a640 10 client.1014191.objecter successfully canceled tick 2024-09-16T04:07:55.126 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.121+0000 7fb89000a640 1 -- 172.21.15.42:0/1230567237 >> v2:172.21.15.154:6812/254845342 conn(0x7fb86c037840 msgr2=0x7fb86c039d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:55.126 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.121+0000 7fb89000a640 1 --2- 172.21.15.42:0/1230567237 >> v2:172.21.15.154:6812/254845342 conn(0x7fb86c037840 0x7fb86c039d00 secure :-1 s=READY pgs=3888 cs=0 l=1 rev1=1 crypto rx=0x7fb884007fd0 tx=0x7fb884009040 comp rx=0 tx=0).stop 2024-09-16T04:07:55.126 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.121+0000 7fb89000a640 10 monclient: shutdown 2024-09-16T04:07:55.126 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.121+0000 7fb89000a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:07:55.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.121+0000 7fb89000a640 1 -- 172.21.15.42:0/1230567237 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb888132b70 msgr2=0x7fb8881d5910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:55.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.121+0000 7fb89000a640 1 --2- 172.21.15.42:0/1230567237 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb888132b70 0x7fb8881d5910 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fb88001a540 tx=0x7fb8800196b0 comp rx=0 tx=0).stop 2024-09-16T04:07:55.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.121+0000 7fb89000a640 1 -- 172.21.15.42:0/1230567237 shutdown_connections 2024-09-16T04:07:55.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.121+0000 7fb89000a640 1 --2- 172.21.15.42:0/1230567237 >> v2:172.21.15.154:6812/254845342 conn(0x7fb86c037840 0x7fb86c039d00 unknown :-1 s=CLOSED pgs=3888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:55.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.121+0000 7fb89000a640 1 --2- 172.21.15.42:0/1230567237 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb888132b70 0x7fb8881d5910 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:55.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.121+0000 7fb89000a640 1 --2- 172.21.15.42:0/1230567237 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb888132210 0x7fb888132630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:55.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.121+0000 7fb89000a640 1 --2- 172.21.15.42:0/1230567237 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb874008400 0x7fb888071d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:55.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.121+0000 7fb89000a640 1 -- 172.21.15.42:0/1230567237 >> 172.21.15.42:0/1230567237 conn(0x7fb8880a2420 msgr2=0x7fb8880a3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:07:55.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.121+0000 7fb89000a640 1 -- 172.21.15.42:0/1230567237 shutdown_connections 2024-09-16T04:07:55.128 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.121+0000 7fb89000a640 1 -- 172.21.15.42:0/1230567237 wait complete. 2024-09-16T04:07:55.128 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.121+0000 7fb89000a640 1 librados: shutdown 2024-09-16T04:07:55.143 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T04:07:55.143 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T04:07:55.143 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T04:07:55.267 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.261+0000 7fa8ffc90640 10 monclient: get_monmap_and_config 2024-09-16T04:07:55.267 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.261+0000 7fa8ffc90640 10 monclient: build_initial_monmap 2024-09-16T04:07:55.267 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.261+0000 7fa8ffc90640 10 monclient: monmap: 2024-09-16T04:07:55.267 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:07:55.267 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:55.267 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:07:55.265529+0000 2024-09-16T04:07:55.267 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:07:55.265529+0000 2024-09-16T04:07:55.267 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:07:55.267 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:55.267 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:07:55.268 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:07:55.268 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:07:55.268 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:07:55.268 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:07:55.268 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:55.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.261+0000 7fa8ffc90640 1 Processor -- start 2024-09-16T04:07:55.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.261+0000 7fa8ffc90640 1 -- start start 2024-09-16T04:07:55.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.261+0000 7fa8ffc90640 10 monclient: init 2024-09-16T04:07:55.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.261+0000 7fa8ffc90640 10 monclient: _reopen_session rank -1 2024-09-16T04:07:55.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.261+0000 7fa8ffc90640 10 monclient: _add_conns ranks=[0,3,1,2,4] 2024-09-16T04:07:55.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.261+0000 7fa8ffc90640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa8f80a1330 addr v1:172.21.15.42:6789/0 2024-09-16T04:07:55.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.261+0000 7fa8ffc90640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fa8f80a1710 0x7fa8f80a1b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:55.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.261+0000 7fa8ffc90640 10 monclient(hunting): picked mon.noname-a con 0x7fa8f80a1710 addr v2:172.21.15.42:3300/0 2024-09-16T04:07:55.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.261+0000 7fa8ffc90640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa8f81393a0 addr v1:172.21.15.154:6789/0 2024-09-16T04:07:55.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.261+0000 7fa8ffc90640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa8f809e4c0 con 0x7fa8f80a1330 2024-09-16T04:07:55.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.261+0000 7fa8ffc90640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa8f80a2080 con 0x7fa8f81393a0 2024-09-16T04:07:55.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.261+0000 7fa8ffc90640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:55.271 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.261+0000 7fa8ffc90640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7fa8f809edf0 con 0x7fa8f80a1710 2024-09-16T04:07:55.271 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.261+0000 7fa8ffc90640 10 monclient(hunting): _renew_subs 2024-09-16T04:07:55.271 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.261+0000 7fa8ffc90640 10 monclient(hunting): authenticate will time out at 2522.234953s 2024-09-16T04:07:55.271 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.261+0000 7fa8fd22b640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fa8f80a1710 0x7fa8f80a1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:55.271 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.261+0000 7fa8fd22b640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fa8f80a1710 0x7fa8f80a1b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:43412/0 (socket says 172.21.15.42:43412) 2024-09-16T04:07:55.271 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.261+0000 7fa8fd22b640 1 -- 172.21.15.42:0/1930914829 learned_addr learned my addr 172.21.15.42:0/1930914829 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:07:55.271 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.261+0000 7fa8fd22b640 10 monclient(hunting): get_auth_request con 0x7fa8f80a1710 auth_method 0 2024-09-16T04:07:55.271 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.261+0000 7fa8fd22b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:55.271 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.261+0000 7fa8fd22b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:55.271 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.261+0000 7fa8fd22b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:55.271 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.261+0000 7fa8fda2c640 1 --1- 172.21.15.42:0/1930914829 >> v1:172.21.15.42:6789/0 conn(0x7fa8f80a1330 0x7fa8f809f180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:44534/0 (socket says 172.21.15.42:44534) 2024-09-16T04:07:55.271 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.261+0000 7fa8fd22b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:07:55.271 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.261+0000 7fa8fd22b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:07:55.271 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.261+0000 7fa8fd22b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:07:55.272 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.261+0000 7fa8fd22b640 10 monclient(hunting): handle_auth_done global_id 1024104 payload 274 2024-09-16T04:07:55.272 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.261+0000 7fa8fd22b640 1 -- 172.21.15.42:0/1930914829 >> v1:172.21.15.154:6789/0 conn(0x7fa8f81393a0 legacy=0x7fa8f813b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:07:55.272 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.261+0000 7fa8fd22b640 1 -- 172.21.15.42:0/1930914829 >> v1:172.21.15.42:6789/0 conn(0x7fa8f80a1330 legacy=0x7fa8f809f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:55.272 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.261+0000 7fa8fd22b640 10 monclient: _finish_hunting 0 2024-09-16T04:07:55.272 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.261+0000 7fa8fd22b640 1 monclient: found mon.noname-a 2024-09-16T04:07:55.272 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.261+0000 7fa8fd22b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:07:55.272 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.261+0000 7fa8fd22b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:07:55.272 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.261+0000 7fa8fd22b640 1 -- 172.21.15.42:0/1930914829 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8f80a3170 con 0x7fa8f80a1710 2024-09-16T04:07:55.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.265+0000 7fa8fd22b640 1 --2- 172.21.15.42:0/1930914829 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa8f80a1710 0x7fa8f80a1b40 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fa8f0009e50 tx=0x7fa8f001c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8da149007926c6d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:07:55.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.265+0000 7fa8fca2a640 1 -- 172.21.15.42:0/1930914829 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8f00ac070 con 0x7fa8f80a1710 2024-09-16T04:07:55.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.265+0000 7fa8fca2a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:55.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.265+0000 7fa8fca2a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:07:55.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.265+0000 7fa8fca2a640 10 monclient: dump: 2024-09-16T04:07:55.284 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:07:55.284 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:55.284 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:55.284 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:55.284 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:55.284 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:55.284 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:07:55.284 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:07:55.284 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:07:55.284 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:55.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.265+0000 7fa8fca2a640 10 monclient: _finish_auth 0 2024-09-16T04:07:55.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.265+0000 7fa8fca2a640 10 monclient: _check_auth_tickets 2024-09-16T04:07:55.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.265+0000 7fa8fca2a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:07:55.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.265+0000 7fa8ffc90640 5 monclient: authenticate success, global_id 1024104 2024-09-16T04:07:55.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.265+0000 7fa8ffc90640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:07:55.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.265+0000 7fa8fca2a640 1 -- 172.21.15.42:0/1930914829 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa8f00a6480 con 0x7fa8f80a1710 2024-09-16T04:07:55.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.265+0000 7fa8fca2a640 10 monclient: handle_config config(0 keys) 2024-09-16T04:07:55.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.265+0000 7fa8ffc90640 10 monclient: get_monmap_and_config success 2024-09-16T04:07:55.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.265+0000 7fa8ffc90640 4 set_mon_vals no callback set 2024-09-16T04:07:55.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.265+0000 7fa8fca2a640 1 -- 172.21.15.42:0/1930914829 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8f00a6850 con 0x7fa8f80a1710 2024-09-16T04:07:55.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.265+0000 7fa8fca2a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:55.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.265+0000 7fa8fca2a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:07:55.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.265+0000 7fa8fca2a640 10 monclient: dump: 2024-09-16T04:07:55.285 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:07:55.285 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:55.286 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:55.286 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:55.286 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:55.286 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:55.286 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:07:55.286 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:07:55.286 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:07:55.286 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:55.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.265+0000 7fa8ffc90640 10 monclient: shutdown 2024-09-16T04:07:55.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.265+0000 7fa8ffc90640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:07:55.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.265+0000 7fa8ffc90640 1 -- 172.21.15.42:0/1930914829 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa8f80a1710 msgr2=0x7fa8f80a1b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:55.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.265+0000 7fa8ffc90640 1 --2- 172.21.15.42:0/1930914829 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa8f80a1710 0x7fa8f80a1b40 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fa8f0009e50 tx=0x7fa8f001c8b0 comp rx=0 tx=0).stop 2024-09-16T04:07:55.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.265+0000 7fa8ffc90640 1 -- 172.21.15.42:0/1930914829 shutdown_connections 2024-09-16T04:07:55.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.265+0000 7fa8ffc90640 1 --2- 172.21.15.42:0/1930914829 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa8f80a1710 0x7fa8f80a1b40 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:55.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.265+0000 7fa8ffc90640 1 -- 172.21.15.42:0/1930914829 >> 172.21.15.42:0/1930914829 conn(0x7fa8f80a2420 msgr2=0x7fa8f80a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:07:55.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.265+0000 7fa8ffc90640 1 -- 172.21.15.42:0/1930914829 shutdown_connections 2024-09-16T04:07:55.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.265+0000 7fa8ffc90640 1 -- 172.21.15.42:0/1930914829 wait complete. 2024-09-16T04:07:55.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.265+0000 7fa8ffc90640 10 monclient: build_initial_monmap 2024-09-16T04:07:55.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.265+0000 7fa8ffc90640 10 monclient: monmap: 2024-09-16T04:07:55.287 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:07:55.287 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:07:55.287 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:07:55.287 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:07:55.287 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:07:55.287 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:55.287 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:07:55.287 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:07:55.288 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:07:55.288 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:55.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.265+0000 7fa8ffc90640 1 librados: starting msgr at 2024-09-16T04:07:55.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.265+0000 7fa8ffc90640 1 librados: starting objecter 2024-09-16T04:07:55.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.265+0000 7fa8ffc90640 1 Processor -- start 2024-09-16T04:07:55.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.265+0000 7fa8ffc90640 1 -- start start 2024-09-16T04:07:55.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.265+0000 7fa8ffc90640 1 librados: setting wanted keys 2024-09-16T04:07:55.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.265+0000 7fa8ffc90640 1 librados: calling monclient init 2024-09-16T04:07:55.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.265+0000 7fa8ffc90640 10 monclient: init 2024-09-16T04:07:55.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.269+0000 7fa8ffc90640 10 monclient: _reopen_session rank -1 2024-09-16T04:07:55.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.269+0000 7fa8ffc90640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:07:55.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.269+0000 7fa8ffc90640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa8f80a1330 0x7fa8f81cee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:55.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.269+0000 7fa8ffc90640 10 monclient(hunting): picked mon.noname-b con 0x7fa8f80a1330 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:07:55.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.269+0000 7fa8ffc90640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa8f80a1710 0x7fa8f8134350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:55.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.269+0000 7fa8ffc90640 10 monclient(hunting): picked mon.noname-c con 0x7fa8f80a1710 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:07:55.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.269+0000 7fa8ffc90640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa8f81393a0 0x7fa8f8134890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:55.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.269+0000 7fa8ffc90640 10 monclient(hunting): picked mon.noname-a con 0x7fa8f81393a0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:07:55.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.269+0000 7fa8ffc90640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:55.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.269+0000 7fa8ffc90640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fa8f809c670 con 0x7fa8f81393a0 2024-09-16T04:07:55.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.269+0000 7fa8ffc90640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:55.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.269+0000 7fa8ffc90640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa8f809c4f0 con 0x7fa8f80a1330 2024-09-16T04:07:55.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.269+0000 7fa8ffc90640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:55.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.269+0000 7fa8ffc90640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fa8f809c7f0 con 0x7fa8f80a1710 2024-09-16T04:07:55.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.269+0000 7fa8ffc90640 10 monclient(hunting): _renew_subs 2024-09-16T04:07:55.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.269+0000 7fa8ffc90640 10 monclient(hunting): authenticate will time out at 2522.240229s 2024-09-16T04:07:55.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.269+0000 7fa8fd22b640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa8f80a1710 0x7fa8f8134350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:55.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.269+0000 7fa8fe22d640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa8f81393a0 0x7fa8f8134890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:55.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.269+0000 7fa8fd22b640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa8f80a1710 0x7fa8f8134350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:37168/0 (socket says 172.21.15.42:37168) 2024-09-16T04:07:55.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.269+0000 7fa8fd22b640 1 -- 172.21.15.42:0/4169681539 learned_addr learned my addr 172.21.15.42:0/4169681539 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:07:55.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.269+0000 7fa8fd22b640 10 monclient(hunting): get_auth_request con 0x7fa8f80a1710 auth_method 0 2024-09-16T04:07:55.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.269+0000 7fa8fd22b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:55.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.269+0000 7fa8fd22b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:55.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.269+0000 7fa8fd22b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:55.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.269+0000 7fa8fe22d640 10 monclient(hunting): get_auth_request con 0x7fa8f81393a0 auth_method 0 2024-09-16T04:07:55.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.269+0000 7fa8fe22d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:55.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.269+0000 7fa8fe22d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:55.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.269+0000 7fa8fe22d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:55.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.269+0000 7fa8fd22b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:07:55.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.269+0000 7fa8fd22b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:07:55.291 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.269+0000 7fa8fd22b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:07:55.291 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.269+0000 7fa8fe22d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:07:55.291 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.269+0000 7fa8fe22d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:07:55.291 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.269+0000 7fa8fe22d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:07:55.291 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.269+0000 7fa8fda2c640 1 --2- 172.21.15.42:0/4169681539 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa8f80a1330 0x7fa8f81cee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:55.291 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.269+0000 7fa8fda2c640 10 monclient(hunting): get_auth_request con 0x7fa8f80a1330 auth_method 0 2024-09-16T04:07:55.291 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.269+0000 7fa8fda2c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:55.291 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.269+0000 7fa8fda2c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:55.291 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.269+0000 7fa8fda2c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:55.291 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.269+0000 7fa8fd22b640 10 monclient(hunting): handle_auth_done global_id 1014194 payload 824 2024-09-16T04:07:55.291 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.269+0000 7fa8fd22b640 1 -- 172.21.15.42:0/4169681539 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa8f80a1330 msgr2=0x7fa8f81cee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:55.291 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.269+0000 7fa8fd22b640 1 --2- 172.21.15.42:0/4169681539 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa8f80a1330 0x7fa8f81cee60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:55.291 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.269+0000 7fa8fd22b640 1 -- 172.21.15.42:0/4169681539 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa8f81393a0 msgr2=0x7fa8f8134890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:55.292 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.269+0000 7fa8fd22b640 1 --2- 172.21.15.42:0/4169681539 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa8f81393a0 0x7fa8f8134890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:55.292 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.269+0000 7fa8fd22b640 10 monclient: _finish_hunting 0 2024-09-16T04:07:55.292 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.269+0000 7fa8fd22b640 1 monclient: found mon.noname-c 2024-09-16T04:07:55.292 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.269+0000 7fa8fd22b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:07:55.292 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.269+0000 7fa8fd22b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T04:07:55.292 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.269+0000 7fa8fd22b640 1 -- 172.21.15.42:0/4169681539 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8f8121a30 con 0x7fa8f80a1710 2024-09-16T04:07:55.292 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.269+0000 7fa8fe22d640 1 --2- 172.21.15.42:0/4169681539 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa8f81393a0 0x7fa8f8134890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:07:55.292 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.269+0000 7fa8fd22b640 1 --2- 172.21.15.42:0/4169681539 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa8f80a1710 0x7fa8f8134350 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fa8f001a4e0 tx=0x7fa8f0019650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:07:55.292 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.269+0000 7fa8ee7fc640 10 client.?.objecter ms_handle_connect 0x7fa8f80a1710 2024-09-16T04:07:55.292 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.269+0000 7fa8ee7fc640 10 client.?.objecter resend_mon_ops 2024-09-16T04:07:55.292 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.269+0000 7fa8ee7fc640 1 -- 172.21.15.42:0/4169681539 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8f00ac040 con 0x7fa8f80a1710 2024-09-16T04:07:55.292 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.269+0000 7fa8ee7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:55.292 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.269+0000 7fa8ee7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:07:55.292 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.269+0000 7fa8ee7fc640 10 monclient: dump: 2024-09-16T04:07:55.293 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:07:55.293 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:55.293 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:55.293 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:55.293 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:55.293 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:55.293 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:07:55.293 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:07:55.293 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:07:55.293 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:55.293 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.269+0000 7fa8ee7fc640 10 monclient: _finish_auth 0 2024-09-16T04:07:55.293 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.269+0000 7fa8ee7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:07:55.293 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.269+0000 7fa8ee7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:07:55.294 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.269+0000 7fa8ee7fc640 1 -- 172.21.15.42:0/4169681539 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa8f00a7660 con 0x7fa8f80a1710 2024-09-16T04:07:55.294 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.269+0000 7fa8ee7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:07:55.294 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.269+0000 7fa8ee7fc640 1 -- 172.21.15.42:0/4169681539 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8f00a7a50 con 0x7fa8f80a1710 2024-09-16T04:07:55.294 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.269+0000 7fa8ee7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:55.294 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.269+0000 7fa8ee7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:07:55.294 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.269+0000 7fa8ee7fc640 10 monclient: dump: 2024-09-16T04:07:55.294 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:07:55.294 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:55.294 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:55.294 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:55.294 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:55.294 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:55.294 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:07:55.294 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:07:55.294 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:07:55.295 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:55.295 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.269+0000 7fa8ffc90640 5 monclient: authenticate success, global_id 1014194 2024-09-16T04:07:55.295 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.269+0000 7fa8ffc90640 10 monclient: _renew_subs 2024-09-16T04:07:55.295 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.269+0000 7fa8ffc90640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:07:55.295 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.269+0000 7fa8effff640 4 set_mon_vals no callback set 2024-09-16T04:07:55.295 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.269+0000 7fa8ffc90640 1 -- 172.21.15.42:0/4169681539 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8f80a5460 con 0x7fa8f80a1710 2024-09-16T04:07:55.295 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.269+0000 7fa8ffc90640 10 client.1014194.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:07:55.295 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.269+0000 7fa8ffc90640 10 monclient: _renew_subs 2024-09-16T04:07:55.295 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.269+0000 7fa8ffc90640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:07:55.295 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.269+0000 7fa8ffc90640 1 -- 172.21.15.42:0/4169681539 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8f8073a30 con 0x7fa8f80a1710 2024-09-16T04:07:55.295 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.269+0000 7fa8ffc90640 1 librados: init done 2024-09-16T04:07:55.295 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.269+0000 7fa8ffc90640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:07:55.295 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.269+0000 7fa8ffc90640 1 --2- 172.21.15.42:0/4169681539 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa8f8121550 0x7fa8f8070b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:55.296 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.269+0000 7fa8ffc90640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:55.296 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.269+0000 7fa8ffc90640 1 -- 172.21.15.42:0/4169681539 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fa8f80a1b40 con 0x7fa8f8121550 2024-09-16T04:07:55.296 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.269+0000 7fa8ee7fc640 1 -- 172.21.15.42:0/4169681539 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fa8f00af070 con 0x7fa8f80a1710 2024-09-16T04:07:55.296 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.269+0000 7fa8fda2c640 1 --2- 172.21.15.42:0/4169681539 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa8f8121550 0x7fa8f8070b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:55.296 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.269+0000 7fa8fda2c640 10 monclient: get_auth_request con 0x7fa8f8121550 auth_method 0 2024-09-16T04:07:55.296 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.269+0000 7fa8fda2c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:55.296 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.269+0000 7fa8fda2c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:55.296 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.269+0000 7fa8fda2c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:55.296 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.269+0000 7fa8fda2c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:07:55.296 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.269+0000 7fa8fda2c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:07:55.296 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.269+0000 7fa8fda2c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:07:55.296 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.269+0000 7fa8ee7fc640 1 --2- 172.21.15.42:0/4169681539 >> v2:172.21.15.154:6812/254845342 conn(0x7fa8e4037810 0x7fa8e4039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:55.296 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.269+0000 7fa8fda2c640 10 monclient(hunting): handle_auth_done global_id 1024113 payload 824 2024-09-16T04:07:55.296 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.269+0000 7fa8ee7fc640 1 -- 172.21.15.42:0/4169681539 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(706..706 src has 451..706) ==== 5040+0+0 (secure 0 0 0) 0x7fa8f00e3bf0 con 0x7fa8f80a1710 2024-09-16T04:07:55.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.269+0000 7fa8ee7fc640 10 client.1014194.objecter ms_dispatch 0x7fa8f8000ba0 osd_map(706..706 src has 451..706) 2024-09-16T04:07:55.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.269+0000 7fa8ee7fc640 3 client.1014194.objecter handle_osd_map got epochs [706,706] > 0 2024-09-16T04:07:55.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.269+0000 7fa8ee7fc640 3 client.1014194.objecter handle_osd_map decoding full epoch 706 2024-09-16T04:07:55.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.269+0000 7fa8fda2c640 1 -- 172.21.15.42:0/4169681539 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa8f8120930 con 0x7fa8f8121550 2024-09-16T04:07:55.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.269+0000 7fa8ee7fc640 20 client.1014194.objecter dump_active .. 0 homeless 2024-09-16T04:07:55.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.269+0000 7fa8ee7fc640 10 client.1014194.objecter ms_handle_connect 0x7fa8f8121550 2024-09-16T04:07:55.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.269+0000 7fa8ee7fc640 10 client.1014194.objecter resend_mon_ops 2024-09-16T04:07:55.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.269+0000 7fa8fda2c640 1 --2- 172.21.15.42:0/4169681539 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa8f8121550 0x7fa8f8070b30 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fa8f400e400 tx=0x7fa8f400e8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:07:55.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.269+0000 7fa8ee7fc640 1 -- 172.21.15.42:0/4169681539 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8f40062e0 con 0x7fa8f8121550 2024-09-16T04:07:55.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.269+0000 7fa8ee7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:55.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.269+0000 7fa8ee7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:07:55.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.269+0000 7fa8fe22d640 1 --2- 172.21.15.42:0/4169681539 >> v2:172.21.15.154:6812/254845342 conn(0x7fa8e4037810 0x7fa8e4039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:55.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.269+0000 7fa8ee7fc640 10 monclient: dump: 2024-09-16T04:07:55.297 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:07:55.298 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:55.298 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:55.298 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:55.298 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:55.298 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:55.298 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:07:55.298 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:07:55.298 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:07:55.298 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:55.298 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.269+0000 7fa8fe22d640 10 monclient: get_auth_request con 0x7fa8e4037810 auth_method 0 2024-09-16T04:07:55.298 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.269+0000 7fa8ee7fc640 1 -- 172.21.15.42:0/4169681539 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fa8f4018070 con 0x7fa8f8121550 2024-09-16T04:07:55.298 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.269+0000 7fa8ee7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:07:55.298 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.269+0000 7fa8ee7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:07:55.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.269+0000 7fa8ee7fc640 1 -- 172.21.15.42:0/4169681539 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa8f8121550 msgr2=0x7fa8f8070b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:55.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.269+0000 7fa8ee7fc640 1 --2- 172.21.15.42:0/4169681539 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa8f8121550 0x7fa8f8070b30 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fa8f400e400 tx=0x7fa8f400e8c0 comp rx=0 tx=0).stop 2024-09-16T04:07:55.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.269+0000 7fa8ee7fc640 1 -- 172.21.15.42:0/4169681539 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa8f8121550 msgr2=0x7fa8f8070b30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:07:55.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.269+0000 7fa8ee7fc640 1 --2- 172.21.15.42:0/4169681539 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa8f8121550 0x7fa8f8070b30 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:55.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.269+0000 7fa8fe22d640 1 --2- 172.21.15.42:0/4169681539 >> v2:172.21.15.154:6812/254845342 conn(0x7fa8e4037810 0x7fa8e4039cd0 secure :-1 s=READY pgs=3889 cs=0 l=1 rev1=1 crypto rx=0x7fa8e8007fd0 tx=0x7fa8e8007dc0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:07:55.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.269+0000 7fa8ee7fc640 10 client.1014194.objecter ms_handle_connect 0x7fa8e4037810 2024-09-16T04:07:55.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.277+0000 7fa8ffc90640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:07:55.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.277+0000 7fa8ffc90640 1 --2- 172.21.15.42:0/4169681539 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa8f802c650 0x7fa8f802ca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:55.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.277+0000 7fa8ffc90640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:55.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.277+0000 7fa8ffc90640 1 -- 172.21.15.42:0/4169681539 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fa8f8138aa0 con 0x7fa8f802c650 2024-09-16T04:07:55.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.277+0000 7fa8fda2c640 1 --2- 172.21.15.42:0/4169681539 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa8f802c650 0x7fa8f802ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:55.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.277+0000 7fa8fda2c640 10 monclient: get_auth_request con 0x7fa8f802c650 auth_method 0 2024-09-16T04:07:55.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.277+0000 7fa8fda2c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:55.300 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.277+0000 7fa8fda2c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:55.300 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.277+0000 7fa8fda2c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:55.300 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.277+0000 7fa8fda2c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:07:55.300 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.277+0000 7fa8fda2c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:07:55.300 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.277+0000 7fa8fda2c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:07:55.300 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.277+0000 7fa8fda2c640 1 -- 172.21.15.42:0/4169681539 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa8f802c650 msgr2=0x7fa8f802ca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-16T04:07:55.300 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.277+0000 7fa8fda2c640 1 -- 172.21.15.42:0/4169681539 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa8f802c650 msgr2=0x7fa8f802ca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:07:55.300 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.277+0000 7fa8fda2c640 1 --2- 172.21.15.42:0/4169681539 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa8f802c650 0x7fa8f802ca60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:07:55.300 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.277+0000 7fa8fda2c640 1 --2- 172.21.15.42:0/4169681539 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa8f802c650 0x7fa8f802ca60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:07:55.486 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.481+0000 7fa8fda2c640 1 --2- 172.21.15.42:0/4169681539 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa8f802c650 0x7fa8f802ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:55.486 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.481+0000 7fa8fda2c640 10 monclient: get_auth_request con 0x7fa8f802c650 auth_method 0 2024-09-16T04:07:55.486 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.481+0000 7fa8fda2c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:55.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.481+0000 7fa8fda2c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:55.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.481+0000 7fa8fda2c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:07:55.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.481+0000 7fa8fda2c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:07:55.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.481+0000 7fa8fda2c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:07:55.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.481+0000 7fa8fda2c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:07:55.489 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.481+0000 7fa8fda2c640 10 monclient(hunting): handle_auth_done global_id 1024119 payload 824 2024-09-16T04:07:55.489 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.481+0000 7fa8fda2c640 1 -- 172.21.15.42:0/4169681539 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fa8f8069040 con 0x7fa8f802c650 2024-09-16T04:07:55.489 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.481+0000 7fa8fda2c640 1 --2- 172.21.15.42:0/4169681539 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa8f802c650 0x7fa8f802ca60 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fa8f400e400 tx=0x7fa8f4000bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:07:55.489 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.481+0000 7fa8ee7fc640 10 client.1014194.objecter ms_handle_connect 0x7fa8f802c650 2024-09-16T04:07:55.489 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.481+0000 7fa8ee7fc640 10 client.1014194.objecter resend_mon_ops 2024-09-16T04:07:55.490 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.481+0000 7fa8ee7fc640 1 -- 172.21.15.42:0/4169681539 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8f401d7c0 con 0x7fa8f802c650 2024-09-16T04:07:55.490 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.481+0000 7fa8ee7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:55.490 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.481+0000 7fa8ee7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:07:55.490 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.481+0000 7fa8ee7fc640 10 monclient: dump: 2024-09-16T04:07:55.490 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:07:55.490 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:55.490 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:55.490 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:55.490 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:55.490 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:55.490 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:07:55.490 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:07:55.490 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:07:55.490 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:55.491 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.481+0000 7fa8ee7fc640 1 -- 172.21.15.42:0/4169681539 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4546 (secure 0 0 0) 0x7fa8f401d7c0 con 0x7fa8f802c650 2024-09-16T04:07:55.491 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.481+0000 7fa8ee7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:07:55.491 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.481+0000 7fa8ee7fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:07:55.491 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.481+0000 7fa8ee7fc640 1 -- 172.21.15.42:0/4169681539 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa8f802c650 msgr2=0x7fa8f802ca60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:55.491 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.481+0000 7fa8ee7fc640 1 --2- 172.21.15.42:0/4169681539 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa8f802c650 0x7fa8f802ca60 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fa8f400e400 tx=0x7fa8f4000bd0 comp rx=0 tx=0).stop 2024-09-16T04:07:55.491 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.481+0000 7fa8ee7fc640 1 -- 172.21.15.42:0/4169681539 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa8f802c650 msgr2=0x7fa8f802ca60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:07:55.491 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.481+0000 7fa8ee7fc640 1 --2- 172.21.15.42:0/4169681539 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa8f802c650 0x7fa8f802ca60 secure :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fa8f400e400 tx=0x7fa8f4000bd0 comp rx=0 tx=0).stop 2024-09-16T04:07:55.491 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T04:07:55.491 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:07:55.491 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:07:55.492 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "leader", 2024-09-16T04:07:55.492 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 238, 2024-09-16T04:07:55.492 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 7451, 2024-09-16T04:07:55.492 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T04:07:55.492 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T04:07:55.492 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T04:07:55.492 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T04:07:55.492 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:07:55.492 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 1, 2024-09-16T04:07:55.492 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:07:55.492 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T04:07:55.492 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T04:07:55.492 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:07:55.492 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:07:55.493 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:07:55.493 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:07:55.493 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:07:55.493 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:07:55.493 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:07:55.493 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:07:55.493 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:07:55.493 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:07:55.493 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:07:55.493 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:07:55.494 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:07:55.494 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T04:07:55.494 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:07:55.494 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:07:55.494 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:07:55.494 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:07:55.494 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:07:55.494 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:07:55.494 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:07:55.494 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:07:55.494 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:07:55.494 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:07:55.494 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:07:55.495 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:07:55.495 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:55.495 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T04:07:55.495 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T04:07:55.495 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T04:07:55.495 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T04:07:55.495 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T04:07:55.495 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T04:07:55.495 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:07:55.495 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:07:55.496 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T04:07:55.496 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T04:07:55.496 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T04:07:55.496 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T04:07:55.496 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T04:07:55.496 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T04:07:55.496 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T04:07:55.496 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:07:55.496 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T04:07:55.496 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:07:55.496 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:07:55.496 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:07:55.496 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:07:55.497 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:07:55.497 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:07:55.497 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:07:55.497 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:07:55.497 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:07:55.497 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:07:55.497 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:07:55.497 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:07:55.497 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T04:07:55.497 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:55.497 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T04:07:55.497 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:55.497 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:07:55.497 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:07:55.498 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:07:55.498 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:07:55.498 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:55.498 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:07:55.498 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T04:07:55.498 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:07:55.498 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:55.498 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:55.498 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:07:55.498 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T04:07:55.498 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:07:55.498 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:07:55.498 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:07:55.498 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:55.499 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T04:07:55.499 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T04:07:55.499 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:07:55.499 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:07:55.499 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:07:55.499 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:55.499 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:55.499 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:07:55.499 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:07:55.499 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:07:55.499 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:07:55.499 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:55.499 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:07:55.499 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T04:07:55.499 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:07:55.500 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:55.500 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:55.500 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:07:55.500 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T04:07:55.500 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:07:55.500 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:07:55.500 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:07:55.500 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:55.500 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T04:07:55.500 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T04:07:55.501 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:07:55.501 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:07:55.501 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:07:55.501 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:55.501 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:55.501 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:07:55.501 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:07:55.501 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:07:55.501 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:07:55.501 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:55.502 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:07:55.502 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T04:07:55.502 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:07:55.502 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:55.502 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:55.502 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:07:55.502 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T04:07:55.502 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:07:55.502 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:07:55.502 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:07:55.502 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:55.502 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T04:07:55.502 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T04:07:55.502 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:07:55.502 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:07:55.503 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:07:55.503 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:07:55.503 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:07:55.503 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:55.503 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T04:07:55.503 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T04:07:55.503 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:55.503 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:07:55.503 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:07:55.503 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:07:55.503 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:07:55.503 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:07:55.503 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T04:07:55.503 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:55.504 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:07:55.504 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:07:55.504 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:07:55.504 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:07:55.504 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:07:55.504 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:55.504 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T04:07:55.504 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T04:07:55.505 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.481+0000 7fa8ffc90640 10 librados: watch_flush enter 2024-09-16T04:07:55.505 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.481+0000 7fa8ffc90640 10 librados: watch_flush exit 2024-09-16T04:07:55.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.481+0000 7fa8ffc90640 20 client.1014194.objecter shutdown clearing up homeless session... 2024-09-16T04:07:55.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.481+0000 7fa8ffc90640 10 client.1014194.objecter successfully canceled tick 2024-09-16T04:07:55.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.481+0000 7fa8ffc90640 1 -- 172.21.15.42:0/4169681539 >> v2:172.21.15.154:6812/254845342 conn(0x7fa8e4037810 msgr2=0x7fa8e4039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:55.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.481+0000 7fa8ffc90640 1 --2- 172.21.15.42:0/4169681539 >> v2:172.21.15.154:6812/254845342 conn(0x7fa8e4037810 0x7fa8e4039cd0 secure :-1 s=READY pgs=3889 cs=0 l=1 rev1=1 crypto rx=0x7fa8e8007fd0 tx=0x7fa8e8007dc0 comp rx=0 tx=0).stop 2024-09-16T04:07:55.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.481+0000 7fa8ffc90640 10 monclient: shutdown 2024-09-16T04:07:55.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.481+0000 7fa8ffc90640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:07:55.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.481+0000 7fa8ffc90640 1 -- 172.21.15.42:0/4169681539 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa8f80a1710 msgr2=0x7fa8f8134350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:55.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.481+0000 7fa8ffc90640 1 --2- 172.21.15.42:0/4169681539 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa8f80a1710 0x7fa8f8134350 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fa8f001a4e0 tx=0x7fa8f0019650 comp rx=0 tx=0).stop 2024-09-16T04:07:55.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.481+0000 7fa8fe22d640 1 -- 172.21.15.42:0/4169681539 reap_dead start 2024-09-16T04:07:55.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.481+0000 7fa8ffc90640 1 -- 172.21.15.42:0/4169681539 shutdown_connections 2024-09-16T04:07:55.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.481+0000 7fa8ffc90640 1 --2- 172.21.15.42:0/4169681539 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa8f80a1710 0x7fa8f8134350 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:55.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.481+0000 7fa8ffc90640 1 -- 172.21.15.42:0/4169681539 >> 172.21.15.42:0/4169681539 conn(0x7fa8f80a2420 msgr2=0x7fa8f813bb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:07:55.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.481+0000 7fa8ffc90640 1 -- 172.21.15.42:0/4169681539 shutdown_connections 2024-09-16T04:07:55.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.485+0000 7fa8ffc90640 1 -- 172.21.15.42:0/4169681539 wait complete. 2024-09-16T04:07:55.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.485+0000 7fa8ffc90640 1 librados: shutdown 2024-09-16T04:07:55.513 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:07:55.513 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T04:07:55.513 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T04:07:55.647 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.641+0000 7fbcf2cb9640 10 monclient: get_monmap_and_config 2024-09-16T04:07:55.647 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.641+0000 7fbcf2cb9640 10 monclient: build_initial_monmap 2024-09-16T04:07:55.647 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.641+0000 7fbcf2cb9640 10 monclient: monmap: 2024-09-16T04:07:55.647 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:07:55.647 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:55.647 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:07:55.645574+0000 2024-09-16T04:07:55.647 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:07:55.645574+0000 2024-09-16T04:07:55.647 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:07:55.647 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:55.647 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:07:55.648 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:07:55.648 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:07:55.648 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:07:55.648 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:07:55.648 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:55.650 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.641+0000 7fbcf2cb9640 1 Processor -- start 2024-09-16T04:07:55.650 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.641+0000 7fbcf2cb9640 1 -- start start 2024-09-16T04:07:55.650 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.641+0000 7fbcf2cb9640 10 monclient: init 2024-09-16T04:07:55.650 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.641+0000 7fbcf2cb9640 10 monclient: _reopen_session rank -1 2024-09-16T04:07:55.650 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.641+0000 7fbcf2cb9640 10 monclient: _add_conns ranks=[0,3,4,1,2] 2024-09-16T04:07:55.650 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.641+0000 7fbcf2cb9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbcec0a1330 addr v1:172.21.15.42:6789/0 2024-09-16T04:07:55.650 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.641+0000 7fbcf2cb9640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fbcec06dac0 0x7fbcec13a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:55.650 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.641+0000 7fbcf2cb9640 10 monclient(hunting): picked mon.noname-a con 0x7fbcec06dac0 addr v2:172.21.15.42:3300/0 2024-09-16T04:07:55.650 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.641+0000 7fbcf2cb9640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fbcec0a1e20 0x7fbcec13c890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:55.651 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.641+0000 7fbcf2cb9640 10 monclient(hunting): picked mon.noname-c con 0x7fbcec0a1e20 addr v2:172.21.15.154:3300/0 2024-09-16T04:07:55.651 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.641+0000 7fbcf2cb9640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbcec09c090 con 0x7fbcec0a1330 2024-09-16T04:07:55.651 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.641+0000 7fbcf2cb9640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:55.651 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.641+0000 7fbcf2cb9640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7fbcec09f610 con 0x7fbcec06dac0 2024-09-16T04:07:55.651 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.641+0000 7fbcf2cb9640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:55.651 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.641+0000 7fbcf2cb9640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fbcec09f8f0 con 0x7fbcec0a1e20 2024-09-16T04:07:55.651 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.641+0000 7fbcf2cb9640 10 monclient(hunting): _renew_subs 2024-09-16T04:07:55.651 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.641+0000 7fbcf2cb9640 10 monclient(hunting): authenticate will time out at 2522.615016s 2024-09-16T04:07:55.651 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.641+0000 7fbcf0a55640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fbcec0a1330 0x7fbcec0a1710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:44546/0 (socket says 172.21.15.42:44546) 2024-09-16T04:07:55.651 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.641+0000 7fbcf0a55640 1 -- 172.21.15.42:0/1286391981 learned_addr learned my addr 172.21.15.42:0/1286391981 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T04:07:55.651 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.641+0000 7fbcebfff640 1 --2- 172.21.15.42:0/1286391981 >> v2:172.21.15.42:3300/0 conn(0x7fbcec06dac0 0x7fbcec13a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:55.651 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.641+0000 7fbcebfff640 10 monclient(hunting): get_auth_request con 0x7fbcec06dac0 auth_method 0 2024-09-16T04:07:55.651 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.641+0000 7fbcebfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:55.652 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.641+0000 7fbcebfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:55.652 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.641+0000 7fbcebfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:55.652 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.641+0000 7fbcebfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:07:55.652 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.641+0000 7fbcebfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:07:55.652 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.641+0000 7fbcebfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:07:55.652 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.641+0000 7fbcf1256640 1 --2- 172.21.15.42:0/1286391981 >> v2:172.21.15.154:3300/0 conn(0x7fbcec0a1e20 0x7fbcec13c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:55.652 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.641+0000 7fbcf1256640 10 monclient(hunting): get_auth_request con 0x7fbcec0a1e20 auth_method 0 2024-09-16T04:07:55.652 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.641+0000 7fbcf1256640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:55.652 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.641+0000 7fbcf1256640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:55.652 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.641+0000 7fbcf1256640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:55.652 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.641+0000 7fbceb7fe640 1 -- 172.21.15.42:0/1286391981 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 5122617 0 0) 0x7fbcec09c090 con 0x7fbcec0a1330 2024-09-16T04:07:55.652 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.641+0000 7fbceb7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:07:55.652 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.641+0000 7fbceb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:55.652 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.641+0000 7fbceb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:55.653 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.641+0000 7fbceb7fe640 10 monclient(hunting): my global_id is 1024125 2024-09-16T04:07:55.653 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.641+0000 7fbceb7fe640 1 -- 172.21.15.42:0/1286391981 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbcd8003530 con 0x7fbcec0a1330 2024-09-16T04:07:55.653 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.641+0000 7fbcebfff640 10 monclient(hunting): handle_auth_done global_id 1024122 payload 274 2024-09-16T04:07:55.653 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.641+0000 7fbcebfff640 1 -- 172.21.15.42:0/1286391981 >> v2:172.21.15.154:3300/0 conn(0x7fbcec0a1e20 msgr2=0x7fbcec13c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:07:55.653 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.641+0000 7fbcebfff640 1 --2- 172.21.15.42:0/1286391981 >> v2:172.21.15.154:3300/0 conn(0x7fbcec0a1e20 0x7fbcec13c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:55.653 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.641+0000 7fbcebfff640 1 -- 172.21.15.42:0/1286391981 >> v1:172.21.15.42:6789/0 conn(0x7fbcec0a1330 legacy=0x7fbcec0a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:55.653 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.641+0000 7fbcebfff640 10 monclient: _finish_hunting 0 2024-09-16T04:07:55.653 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.641+0000 7fbcebfff640 1 monclient: found mon.noname-a 2024-09-16T04:07:55.653 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.641+0000 7fbcebfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:07:55.653 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.641+0000 7fbcebfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:07:55.653 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.641+0000 7fbcebfff640 1 -- 172.21.15.42:0/1286391981 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbcec09fbd0 con 0x7fbcec06dac0 2024-09-16T04:07:55.659 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.645+0000 7fbcebfff640 1 --2- 172.21.15.42:0/1286391981 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbcec06dac0 0x7fbcec13a340 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fbcec09c800 tx=0x7fbcdc01c3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65106c4e1e2ce875 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:07:55.659 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.645+0000 7fbceb7fe640 1 -- 172.21.15.42:0/1286391981 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcdc0b0070 con 0x7fbcec06dac0 2024-09-16T04:07:55.659 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.645+0000 7fbceb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:55.659 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.645+0000 7fbceb7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:07:55.660 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.645+0000 7fbceb7fe640 10 monclient: dump: 2024-09-16T04:07:55.660 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:07:55.660 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:55.660 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:55.660 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:55.660 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:55.660 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:55.660 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:07:55.660 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:07:55.660 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:07:55.660 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:55.660 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.645+0000 7fbceb7fe640 10 monclient: _finish_auth 0 2024-09-16T04:07:55.660 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.645+0000 7fbceb7fe640 10 monclient: _check_auth_tickets 2024-09-16T04:07:55.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.645+0000 7fbceb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:07:55.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.645+0000 7fbcf2cb9640 5 monclient: authenticate success, global_id 1024122 2024-09-16T04:07:55.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.645+0000 7fbcf2cb9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:07:55.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.645+0000 7fbceb7fe640 1 -- 172.21.15.42:0/1286391981 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbcdc0ab600 con 0x7fbcec06dac0 2024-09-16T04:07:55.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.645+0000 7fbceb7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:07:55.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.645+0000 7fbceb7fe640 1 -- 172.21.15.42:0/1286391981 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcdc0a4e60 con 0x7fbcec06dac0 2024-09-16T04:07:55.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.645+0000 7fbcf2cb9640 10 monclient: get_monmap_and_config success 2024-09-16T04:07:55.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.645+0000 7fbcf2cb9640 4 set_mon_vals no callback set 2024-09-16T04:07:55.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.645+0000 7fbceb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:55.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.645+0000 7fbceb7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:07:55.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.645+0000 7fbceb7fe640 10 monclient: dump: 2024-09-16T04:07:55.661 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:07:55.661 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:55.661 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:55.662 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:55.662 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:55.662 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:55.662 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:07:55.662 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:07:55.662 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:07:55.662 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:55.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.645+0000 7fbcf2cb9640 10 monclient: shutdown 2024-09-16T04:07:55.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.645+0000 7fbcf2cb9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:07:55.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.645+0000 7fbcf2cb9640 1 -- 172.21.15.42:0/1286391981 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbcec06dac0 msgr2=0x7fbcec13a340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:55.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.645+0000 7fbcf2cb9640 1 --2- 172.21.15.42:0/1286391981 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbcec06dac0 0x7fbcec13a340 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fbcec09c800 tx=0x7fbcdc01c3d0 comp rx=0 tx=0).stop 2024-09-16T04:07:55.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.645+0000 7fbcf2cb9640 1 -- 172.21.15.42:0/1286391981 shutdown_connections 2024-09-16T04:07:55.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.645+0000 7fbcf2cb9640 1 --2- 172.21.15.42:0/1286391981 >> v2:172.21.15.154:3300/0 conn(0x7fbcec0a1e20 0x7fbcec13c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:55.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.645+0000 7fbcf2cb9640 1 --2- 172.21.15.42:0/1286391981 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbcec06dac0 0x7fbcec13a340 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:55.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.645+0000 7fbcf2cb9640 1 -- 172.21.15.42:0/1286391981 >> 172.21.15.42:0/1286391981 conn(0x7fbcec0a2420 msgr2=0x7fbcec0a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:07:55.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.645+0000 7fbcf2cb9640 1 -- 172.21.15.42:0/1286391981 shutdown_connections 2024-09-16T04:07:55.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.645+0000 7fbcf2cb9640 1 -- 172.21.15.42:0/1286391981 wait complete. 2024-09-16T04:07:55.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.645+0000 7fbcf2cb9640 10 monclient: build_initial_monmap 2024-09-16T04:07:55.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.645+0000 7fbcf2cb9640 10 monclient: monmap: 2024-09-16T04:07:55.663 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:07:55.663 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:07:55.663 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:07:55.663 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:07:55.663 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:07:55.663 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:55.663 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:07:55.663 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:07:55.663 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:07:55.664 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:55.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.645+0000 7fbcf2cb9640 1 librados: starting msgr at 2024-09-16T04:07:55.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.645+0000 7fbcf2cb9640 1 librados: starting objecter 2024-09-16T04:07:55.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.645+0000 7fbcf2cb9640 1 Processor -- start 2024-09-16T04:07:55.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.649+0000 7fbcf2cb9640 1 -- start start 2024-09-16T04:07:55.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.649+0000 7fbcf2cb9640 1 librados: setting wanted keys 2024-09-16T04:07:55.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.649+0000 7fbcf2cb9640 1 librados: calling monclient init 2024-09-16T04:07:55.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.649+0000 7fbcf2cb9640 10 monclient: init 2024-09-16T04:07:55.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.649+0000 7fbcf2cb9640 10 monclient: _reopen_session rank -1 2024-09-16T04:07:55.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.649+0000 7fbcf2cb9640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:07:55.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.649+0000 7fbcf2cb9640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbcec0a1330 0x7fbcec120380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:55.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.649+0000 7fbcf2cb9640 10 monclient(hunting): picked mon.noname-b con 0x7fbcec0a1330 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:07:55.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.649+0000 7fbcf2cb9640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbcec0a1e20 0x7fbcec09d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:55.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.649+0000 7fbcf2cb9640 10 monclient(hunting): picked mon.noname-a con 0x7fbcec0a1e20 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:07:55.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.649+0000 7fbcf2cb9640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbcec1352b0 0x7fbcec135740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:55.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.649+0000 7fbcf2cb9640 10 monclient(hunting): picked mon.noname-c con 0x7fbcec1352b0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:07:55.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.649+0000 7fbcf2cb9640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:55.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.649+0000 7fbcf2cb9640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fbcec13f800 con 0x7fbcec0a1e20 2024-09-16T04:07:55.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.649+0000 7fbcf2cb9640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:55.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.649+0000 7fbcf2cb9640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fbcec09ef80 con 0x7fbcec0a1330 2024-09-16T04:07:55.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.649+0000 7fbcf2cb9640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:55.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.649+0000 7fbcf2cb9640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fbcec13f980 con 0x7fbcec1352b0 2024-09-16T04:07:55.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.649+0000 7fbcf2cb9640 10 monclient(hunting): _renew_subs 2024-09-16T04:07:55.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.649+0000 7fbcf2cb9640 10 monclient(hunting): authenticate will time out at 2522.620538s 2024-09-16T04:07:55.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.649+0000 7fbcebfff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbcec0a1e20 0x7fbcec09d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:55.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.649+0000 7fbcf1256640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbcec1352b0 0x7fbcec135740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:55.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.649+0000 7fbcf1256640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbcec1352b0 0x7fbcec135740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:37180/0 (socket says 172.21.15.42:37180) 2024-09-16T04:07:55.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.649+0000 7fbcebfff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbcec0a1e20 0x7fbcec09d020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:43470/0 (socket says 172.21.15.42:43470) 2024-09-16T04:07:55.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.649+0000 7fbcf1256640 1 -- 172.21.15.42:0/3257747177 learned_addr learned my addr 172.21.15.42:0/3257747177 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:07:55.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.649+0000 7fbcf1256640 10 monclient(hunting): get_auth_request con 0x7fbcec1352b0 auth_method 0 2024-09-16T04:07:55.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.649+0000 7fbcf1256640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:55.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.649+0000 7fbcf1256640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:55.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.649+0000 7fbcf1256640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:55.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.649+0000 7fbcebfff640 10 monclient(hunting): get_auth_request con 0x7fbcec0a1e20 auth_method 0 2024-09-16T04:07:55.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.649+0000 7fbcebfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:55.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.649+0000 7fbcebfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:55.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.649+0000 7fbcebfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:55.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.649+0000 7fbcebfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:07:55.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.649+0000 7fbcebfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:07:55.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.649+0000 7fbcebfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:07:55.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.649+0000 7fbcf1256640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:07:55.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.649+0000 7fbcf1256640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:07:55.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.649+0000 7fbcf1256640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:07:55.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.649+0000 7fbcf0a55640 1 --2- 172.21.15.42:0/3257747177 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbcec0a1330 0x7fbcec120380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:55.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.649+0000 7fbcf0a55640 10 monclient(hunting): get_auth_request con 0x7fbcec0a1330 auth_method 0 2024-09-16T04:07:55.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.649+0000 7fbcf0a55640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:55.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.649+0000 7fbcf0a55640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:55.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.649+0000 7fbcf0a55640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:55.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.649+0000 7fbcebfff640 10 monclient(hunting): handle_auth_done global_id 1024128 payload 824 2024-09-16T04:07:55.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.649+0000 7fbcebfff640 1 -- 172.21.15.42:0/3257747177 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbcec1352b0 msgr2=0x7fbcec135740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:55.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.649+0000 7fbcebfff640 1 --2- 172.21.15.42:0/3257747177 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbcec1352b0 0x7fbcec135740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:55.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.649+0000 7fbcebfff640 1 -- 172.21.15.42:0/3257747177 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbcec0a1330 msgr2=0x7fbcec120380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:55.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.649+0000 7fbcebfff640 1 --2- 172.21.15.42:0/3257747177 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbcec0a1330 0x7fbcec120380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:55.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.649+0000 7fbcebfff640 10 monclient: _finish_hunting 0 2024-09-16T04:07:55.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.649+0000 7fbcebfff640 1 monclient: found mon.noname-a 2024-09-16T04:07:55.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.649+0000 7fbcebfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:07:55.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.649+0000 7fbcebfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:07:55.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.649+0000 7fbcebfff640 1 -- 172.21.15.42:0/3257747177 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbcec073a30 con 0x7fbcec0a1e20 2024-09-16T04:07:55.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.649+0000 7fbcebfff640 1 --2- 172.21.15.42:0/3257747177 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbcec0a1e20 0x7fbcec09d020 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fbcdc0ab000 tx=0x7fbcdc019170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:07:55.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.649+0000 7fbce97fa640 10 client.?.objecter ms_handle_connect 0x7fbcec0a1e20 2024-09-16T04:07:55.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.649+0000 7fbce97fa640 10 client.?.objecter resend_mon_ops 2024-09-16T04:07:55.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.649+0000 7fbce97fa640 1 -- 172.21.15.42:0/3257747177 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcdc0b0040 con 0x7fbcec0a1e20 2024-09-16T04:07:55.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.649+0000 7fbce97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:55.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.649+0000 7fbce97fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:07:55.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.649+0000 7fbce97fa640 10 monclient: dump: 2024-09-16T04:07:55.668 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:07:55.668 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:55.668 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:55.668 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:55.669 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:55.669 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:55.669 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:07:55.669 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:07:55.669 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:07:55.669 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:55.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.649+0000 7fbce97fa640 10 monclient: _finish_auth 0 2024-09-16T04:07:55.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.649+0000 7fbce97fa640 10 monclient: _check_auth_tickets 2024-09-16T04:07:55.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.649+0000 7fbce97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:07:55.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.649+0000 7fbce97fa640 1 -- 172.21.15.42:0/3257747177 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbcdc0b9070 con 0x7fbcec0a1e20 2024-09-16T04:07:55.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.649+0000 7fbcf2cb9640 5 monclient: authenticate success, global_id 1024128 2024-09-16T04:07:55.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.649+0000 7fbcf2cb9640 10 monclient: _renew_subs 2024-09-16T04:07:55.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.649+0000 7fbcf2cb9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:07:55.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.649+0000 7fbcf2cb9640 1 -- 172.21.15.42:0/3257747177 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbcec06e1d0 con 0x7fbcec0a1e20 2024-09-16T04:07:55.670 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.649+0000 7fbce97fa640 10 monclient: handle_config config(0 keys) 2024-09-16T04:07:55.670 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.649+0000 7fbce97fa640 1 -- 172.21.15.42:0/3257747177 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcdc0ab9d0 con 0x7fbcec0a1e20 2024-09-16T04:07:55.670 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.649+0000 7fbce97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:55.670 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.649+0000 7fbce97fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:07:55.670 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.649+0000 7fbce97fa640 10 monclient: dump: 2024-09-16T04:07:55.670 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:07:55.670 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:55.670 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:55.670 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:55.670 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:55.670 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:55.670 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:07:55.670 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:07:55.670 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:07:55.671 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:55.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.649+0000 7fbceaffd640 4 set_mon_vals no callback set 2024-09-16T04:07:55.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.649+0000 7fbcf2cb9640 10 client.1024128.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:07:55.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.649+0000 7fbcf2cb9640 10 monclient: _renew_subs 2024-09-16T04:07:55.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.649+0000 7fbcf2cb9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:07:55.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.649+0000 7fbcf2cb9640 1 -- 172.21.15.42:0/3257747177 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbcec121c60 con 0x7fbcec0a1e20 2024-09-16T04:07:55.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.649+0000 7fbcf2cb9640 1 librados: init done 2024-09-16T04:07:55.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.649+0000 7fbcf2cb9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:07:55.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.649+0000 7fbcf2cb9640 1 --2- 172.21.15.42:0/3257747177 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbcec06dac0 0x7fbcec1cf460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:55.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.649+0000 7fbcf2cb9640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:55.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.649+0000 7fbcf2cb9640 1 -- 172.21.15.42:0/3257747177 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fbcec121700 con 0x7fbcec06dac0 2024-09-16T04:07:55.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.649+0000 7fbcf0a55640 1 --2- 172.21.15.42:0/3257747177 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbcec06dac0 0x7fbcec1cf460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:55.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.649+0000 7fbcf0a55640 10 monclient: get_auth_request con 0x7fbcec06dac0 auth_method 0 2024-09-16T04:07:55.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.649+0000 7fbcf0a55640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:55.672 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.649+0000 7fbcf0a55640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:55.672 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.649+0000 7fbcf0a55640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:55.672 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.649+0000 7fbce97fa640 1 -- 172.21.15.42:0/3257747177 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fbcdc0abb70 con 0x7fbcec0a1e20 2024-09-16T04:07:55.672 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.649+0000 7fbcf0a55640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:07:55.672 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.649+0000 7fbcf0a55640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:07:55.672 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.649+0000 7fbcf0a55640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:07:55.672 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.649+0000 7fbcf0a55640 10 monclient(hunting): handle_auth_done global_id 1014200 payload 824 2024-09-16T04:07:55.672 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.649+0000 7fbce97fa640 1 --2- 172.21.15.42:0/3257747177 >> v2:172.21.15.154:6812/254845342 conn(0x7fbccc037810 0x7fbccc039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:55.672 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.649+0000 7fbcf0a55640 1 -- 172.21.15.42:0/3257747177 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbcec0a1710 con 0x7fbcec06dac0 2024-09-16T04:07:55.672 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.649+0000 7fbce97fa640 1 -- 172.21.15.42:0/3257747177 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(706..706 src has 451..706) ==== 5040+0+0 (secure 0 0 0) 0x7fbcdc0e27c0 con 0x7fbcec0a1e20 2024-09-16T04:07:55.672 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.649+0000 7fbce97fa640 10 client.1024128.objecter ms_dispatch 0x7fbcec000ba0 osd_map(706..706 src has 451..706) 2024-09-16T04:07:55.672 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.649+0000 7fbce97fa640 3 client.1024128.objecter handle_osd_map got epochs [706,706] > 0 2024-09-16T04:07:55.672 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.649+0000 7fbce97fa640 3 client.1024128.objecter handle_osd_map decoding full epoch 706 2024-09-16T04:07:55.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.649+0000 7fbce97fa640 20 client.1024128.objecter dump_active .. 0 homeless 2024-09-16T04:07:55.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.649+0000 7fbcf0a55640 1 --2- 172.21.15.42:0/3257747177 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbcec06dac0 0x7fbcec1cf460 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fbcd4000f70 tx=0x7fbcd400e960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:07:55.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.649+0000 7fbce97fa640 10 client.1024128.objecter ms_handle_connect 0x7fbcec06dac0 2024-09-16T04:07:55.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.649+0000 7fbce97fa640 10 client.1024128.objecter resend_mon_ops 2024-09-16T04:07:55.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.649+0000 7fbce97fa640 1 -- 172.21.15.42:0/3257747177 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcd4005ad0 con 0x7fbcec06dac0 2024-09-16T04:07:55.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.649+0000 7fbce97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:55.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.649+0000 7fbcf1256640 1 --2- 172.21.15.42:0/3257747177 >> v2:172.21.15.154:6812/254845342 conn(0x7fbccc037810 0x7fbccc039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:55.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.649+0000 7fbce97fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:07:55.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.649+0000 7fbce97fa640 10 monclient: dump: 2024-09-16T04:07:55.673 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:07:55.673 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:55.673 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:55.673 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:55.673 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:55.674 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:55.674 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:07:55.674 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:07:55.674 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:07:55.674 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:55.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.649+0000 7fbcf1256640 10 monclient: get_auth_request con 0x7fbccc037810 auth_method 0 2024-09-16T04:07:55.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.649+0000 7fbce97fa640 1 -- 172.21.15.42:0/3257747177 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fbcd4018070 con 0x7fbcec06dac0 2024-09-16T04:07:55.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.649+0000 7fbce97fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:07:55.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.649+0000 7fbce97fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:07:55.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.649+0000 7fbce97fa640 1 -- 172.21.15.42:0/3257747177 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbcec06dac0 msgr2=0x7fbcec1cf460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:55.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.649+0000 7fbce97fa640 1 --2- 172.21.15.42:0/3257747177 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbcec06dac0 0x7fbcec1cf460 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fbcd4000f70 tx=0x7fbcd400e960 comp rx=0 tx=0).stop 2024-09-16T04:07:55.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.649+0000 7fbce97fa640 1 -- 172.21.15.42:0/3257747177 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbcec06dac0 msgr2=0x7fbcec1cf460 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:07:55.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.649+0000 7fbce97fa640 1 --2- 172.21.15.42:0/3257747177 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbcec06dac0 0x7fbcec1cf460 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:55.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.649+0000 7fbce97fa640 10 client.1024128.objecter ms_handle_connect 0x7fbccc037810 2024-09-16T04:07:55.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.649+0000 7fbcf1256640 1 --2- 172.21.15.42:0/3257747177 >> v2:172.21.15.154:6812/254845342 conn(0x7fbccc037810 0x7fbccc039cd0 secure :-1 s=READY pgs=3890 cs=0 l=1 rev1=1 crypto rx=0x7fbce4007520 tx=0x7fbce40098d0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:07:55.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.657+0000 7fbcf2cb9640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:07:55.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.657+0000 7fbcf2cb9640 1 --2- 172.21.15.42:0/3257747177 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbcec02c650 0x7fbcec02ca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:55.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.657+0000 7fbcf2cb9640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:55.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.657+0000 7fbcf2cb9640 1 -- 172.21.15.42:0/3257747177 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fbcec138a40 con 0x7fbcec02c650 2024-09-16T04:07:55.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.657+0000 7fbcf0a55640 1 --2- 172.21.15.42:0/3257747177 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbcec02c650 0x7fbcec02ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:55.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.657+0000 7fbcf0a55640 10 monclient: get_auth_request con 0x7fbcec02c650 auth_method 0 2024-09-16T04:07:55.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.657+0000 7fbcf0a55640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:55.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.657+0000 7fbcf0a55640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:55.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.657+0000 7fbcf0a55640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:55.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.657+0000 7fbcf0a55640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:07:55.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.657+0000 7fbcf0a55640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:07:55.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.657+0000 7fbcf0a55640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:07:55.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.657+0000 7fbcf0a55640 10 monclient(hunting): handle_auth_done global_id 1014203 payload 824 2024-09-16T04:07:55.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.657+0000 7fbcf0a55640 1 -- 172.21.15.42:0/3257747177 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fbcec069040 con 0x7fbcec02c650 2024-09-16T04:07:55.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.657+0000 7fbcf0a55640 1 --2- 172.21.15.42:0/3257747177 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbcec02c650 0x7fbcec02ca30 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fbcd4000f70 tx=0x7fbcd40010b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:07:55.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.657+0000 7fbce97fa640 10 client.1024128.objecter ms_handle_connect 0x7fbcec02c650 2024-09-16T04:07:55.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.657+0000 7fbce97fa640 10 client.1024128.objecter resend_mon_ops 2024-09-16T04:07:55.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.657+0000 7fbce97fa640 1 -- 172.21.15.42:0/3257747177 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcd40072e0 con 0x7fbcec02c650 2024-09-16T04:07:55.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.657+0000 7fbce97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:55.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.657+0000 7fbce97fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:07:55.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.657+0000 7fbce97fa640 10 monclient: dump: 2024-09-16T04:07:55.678 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:07:55.678 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:55.678 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:55.678 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:55.678 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:55.678 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:55.678 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:07:55.678 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:07:55.678 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:07:55.678 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:55.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.657+0000 7fbce97fa640 1 -- 172.21.15.42:0/3257747177 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4712 (secure 0 0 0) 0x7fbcd40072e0 con 0x7fbcec02c650 2024-09-16T04:07:55.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.657+0000 7fbce97fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:07:55.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.657+0000 7fbce97fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:07:55.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.657+0000 7fbce97fa640 1 -- 172.21.15.42:0/3257747177 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbcec02c650 msgr2=0x7fbcec02ca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:55.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.657+0000 7fbce97fa640 1 --2- 172.21.15.42:0/3257747177 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbcec02c650 0x7fbcec02ca30 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fbcd4000f70 tx=0x7fbcd40010b0 comp rx=0 tx=0).stop 2024-09-16T04:07:55.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.657+0000 7fbce97fa640 1 -- 172.21.15.42:0/3257747177 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbcec02c650 msgr2=0x7fbcec02ca30 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:07:55.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.657+0000 7fbce97fa640 1 --2- 172.21.15.42:0/3257747177 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbcec02c650 0x7fbcec02ca30 secure :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fbcd4000f70 tx=0x7fbcd40010b0 comp rx=0 tx=0).stop 2024-09-16T04:07:55.679 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T04:07:55.679 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:07:55.679 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:07:55.679 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T04:07:55.680 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 238, 2024-09-16T04:07:55.680 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 70859, 2024-09-16T04:07:55.680 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T04:07:55.680 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T04:07:55.680 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T04:07:55.680 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T04:07:55.680 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:07:55.680 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 1, 2024-09-16T04:07:55.680 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:07:55.680 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T04:07:55.680 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T04:07:55.680 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:07:55.680 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:07:55.681 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:07:55.681 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:07:55.681 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:07:55.681 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:07:55.681 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:07:55.681 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:07:55.681 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:07:55.681 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:07:55.681 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:07:55.681 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:07:55.681 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:07:55.681 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T04:07:55.681 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:07:55.681 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:07:55.682 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:07:55.682 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:07:55.682 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:07:55.682 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:07:55.682 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:07:55.682 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:07:55.682 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:07:55.682 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:07:55.682 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:07:55.682 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:07:55.682 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:55.682 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T04:07:55.682 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T04:07:55.683 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T04:07:55.683 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T04:07:55.683 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T04:07:55.683 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T04:07:55.683 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:07:55.683 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:07:55.683 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T04:07:55.683 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T04:07:55.683 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T04:07:55.683 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T04:07:55.683 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T04:07:55.683 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T04:07:55.683 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T04:07:55.683 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:07:55.683 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T04:07:55.684 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:07:55.684 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:07:55.684 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:07:55.684 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:07:55.684 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:07:55.684 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:07:55.684 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:07:55.684 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:07:55.684 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:07:55.684 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:07:55.684 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:07:55.684 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:07:55.684 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T04:07:55.684 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:55.684 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T04:07:55.685 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:55.685 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:07:55.685 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:07:55.685 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:07:55.689 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:07:55.689 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:55.689 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:07:55.689 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T04:07:55.689 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:07:55.690 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:55.690 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:55.690 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:07:55.690 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T04:07:55.690 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:07:55.690 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:07:55.690 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:07:55.690 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:55.690 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T04:07:55.690 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T04:07:55.690 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:07:55.690 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:07:55.690 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:07:55.690 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:55.691 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:55.691 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:07:55.691 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:07:55.691 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:07:55.691 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:07:55.691 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:55.691 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:07:55.691 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T04:07:55.691 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:07:55.691 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:55.691 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:55.691 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:07:55.691 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T04:07:55.691 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:07:55.692 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:07:55.692 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:07:55.692 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:55.692 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T04:07:55.692 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T04:07:55.692 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:07:55.692 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:07:55.692 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:07:55.692 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:55.692 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:55.692 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:07:55.692 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:07:55.692 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:07:55.693 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:07:55.693 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:55.693 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:07:55.693 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T04:07:55.693 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:07:55.693 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:55.693 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:55.693 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:07:55.693 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T04:07:55.693 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:07:55.693 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:07:55.693 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:07:55.693 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:55.694 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T04:07:55.694 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T04:07:55.694 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:07:55.694 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:07:55.694 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:07:55.694 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:07:55.694 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:07:55.694 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:55.694 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T04:07:55.694 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T04:07:55.694 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:55.694 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:07:55.694 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:07:55.694 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:07:55.695 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:07:55.695 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:07:55.695 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T04:07:55.695 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:55.695 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:07:55.695 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:07:55.695 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 3 2024-09-16T04:07:55.695 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:07:55.695 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:07:55.695 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T04:07:55.695 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:55.695 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:07:55.695 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:07:55.695 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 2 2024-09-16T04:07:55.696 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:07:55.696 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:07:55.696 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:55.696 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T04:07:55.696 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T04:07:55.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.657+0000 7fbcf2cb9640 10 librados: watch_flush enter 2024-09-16T04:07:55.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.657+0000 7fbcf2cb9640 10 librados: watch_flush exit 2024-09-16T04:07:55.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.657+0000 7fbcf2cb9640 20 client.1024128.objecter shutdown clearing up homeless session... 2024-09-16T04:07:55.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.657+0000 7fbcf2cb9640 10 client.1024128.objecter successfully canceled tick 2024-09-16T04:07:55.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.657+0000 7fbcf2cb9640 1 -- 172.21.15.42:0/3257747177 >> v2:172.21.15.154:6812/254845342 conn(0x7fbccc037810 msgr2=0x7fbccc039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:55.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.657+0000 7fbcf2cb9640 1 --2- 172.21.15.42:0/3257747177 >> v2:172.21.15.154:6812/254845342 conn(0x7fbccc037810 0x7fbccc039cd0 secure :-1 s=READY pgs=3890 cs=0 l=1 rev1=1 crypto rx=0x7fbce4007520 tx=0x7fbce40098d0 comp rx=0 tx=0).stop 2024-09-16T04:07:55.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.657+0000 7fbcf2cb9640 10 monclient: shutdown 2024-09-16T04:07:55.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.657+0000 7fbcf2cb9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:07:55.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.657+0000 7fbcf2cb9640 1 -- 172.21.15.42:0/3257747177 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbcec0a1e20 msgr2=0x7fbcec09d020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:55.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.657+0000 7fbcf2cb9640 1 --2- 172.21.15.42:0/3257747177 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbcec0a1e20 0x7fbcec09d020 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fbcdc0ab000 tx=0x7fbcdc019170 comp rx=0 tx=0).stop 2024-09-16T04:07:55.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.657+0000 7fbcf1256640 1 -- 172.21.15.42:0/3257747177 reap_dead start 2024-09-16T04:07:55.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.657+0000 7fbcf1256640 1 -- 172.21.15.42:0/3257747177 reap_dead start 2024-09-16T04:07:55.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.657+0000 7fbcf2cb9640 1 -- 172.21.15.42:0/3257747177 shutdown_connections 2024-09-16T04:07:55.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.657+0000 7fbcf2cb9640 1 -- 172.21.15.42:0/3257747177 >> 172.21.15.42:0/3257747177 conn(0x7fbcec0a2420 msgr2=0x7fbcec06e720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:07:55.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.657+0000 7fbcf2cb9640 1 -- 172.21.15.42:0/3257747177 shutdown_connections 2024-09-16T04:07:55.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.657+0000 7fbcf2cb9640 1 -- 172.21.15.42:0/3257747177 wait complete. 2024-09-16T04:07:55.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.657+0000 7fbcf2cb9640 1 librados: shutdown 2024-09-16T04:07:55.699 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:07:55.699 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T04:07:55.700 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T04:07:55.818 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.813+0000 7fc6f24f5640 10 monclient: get_monmap_and_config 2024-09-16T04:07:55.818 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.813+0000 7fc6f24f5640 10 monclient: build_initial_monmap 2024-09-16T04:07:55.818 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.813+0000 7fc6f24f5640 10 monclient: monmap: 2024-09-16T04:07:55.818 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:07:55.818 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:55.818 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:07:55.816460+0000 2024-09-16T04:07:55.818 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:07:55.816460+0000 2024-09-16T04:07:55.818 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:07:55.818 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:55.818 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:07:55.818 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:07:55.818 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:07:55.819 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:07:55.819 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:07:55.819 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:55.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.813+0000 7fc6f24f5640 1 Processor -- start 2024-09-16T04:07:55.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.813+0000 7fc6f24f5640 1 -- start start 2024-09-16T04:07:55.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.813+0000 7fc6f24f5640 10 monclient: init 2024-09-16T04:07:55.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.813+0000 7fc6f24f5640 10 monclient: _reopen_session rank -1 2024-09-16T04:07:55.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.813+0000 7fc6f24f5640 10 monclient: _add_conns ranks=[0,2,1,3,4] 2024-09-16T04:07:55.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.813+0000 7fc6f24f5640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc6ec0a1330 addr v1:172.21.15.42:6789/0 2024-09-16T04:07:55.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.813+0000 7fc6f24f5640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc6ec06dac0 0x7fc6ec13a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:55.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.813+0000 7fc6f24f5640 10 monclient(hunting): picked mon.noname-b con 0x7fc6ec06dac0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:07:55.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.813+0000 7fc6f24f5640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc6ec0a1e20 addr v1:172.21.15.154:6789/0 2024-09-16T04:07:55.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.813+0000 7fc6f24f5640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc6ec09c7d0 con 0x7fc6ec0a1330 2024-09-16T04:07:55.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.813+0000 7fc6f24f5640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc6ec09c090 con 0x7fc6ec0a1e20 2024-09-16T04:07:55.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.813+0000 7fc6f24f5640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:55.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.813+0000 7fc6f24f5640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fc6ec09f610 con 0x7fc6ec06dac0 2024-09-16T04:07:55.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.813+0000 7fc6f24f5640 10 monclient(hunting): _renew_subs 2024-09-16T04:07:55.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.813+0000 7fc6f24f5640 10 monclient(hunting): authenticate will time out at 2522.785911s 2024-09-16T04:07:55.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.813+0000 7fc6ebfff640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fc6ec0a1330 0x7fc6ec0a1710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:44560/0 (socket says 172.21.15.42:44560) 2024-09-16T04:07:55.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.813+0000 7fc6ebfff640 1 -- 172.21.15.42:0/3414898129 learned_addr learned my addr 172.21.15.42:0/3414898129 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T04:07:55.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.813+0000 7fc6eb7fe640 1 --2- 172.21.15.42:0/3414898129 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc6ec06dac0 0x7fc6ec13a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:55.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.813+0000 7fc6eb7fe640 10 monclient(hunting): get_auth_request con 0x7fc6ec06dac0 auth_method 0 2024-09-16T04:07:55.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.813+0000 7fc6eb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:55.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.813+0000 7fc6eb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:55.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.813+0000 7fc6eb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:55.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.813+0000 7fc6eb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:07:55.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.813+0000 7fc6eb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:07:55.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.813+0000 7fc6eb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:07:55.823 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.813+0000 7fc6eaffd640 1 -- 172.21.15.42:0/3414898129 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3910476318 0 0) 0x7fc6ec09c7d0 con 0x7fc6ec0a1330 2024-09-16T04:07:55.823 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.813+0000 7fc6eaffd640 10 monclient(hunting): _negotiate 2024-09-16T04:07:55.823 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.813+0000 7fc6eaffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:55.823 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.813+0000 7fc6eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:55.823 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.813+0000 7fc6eaffd640 10 monclient(hunting): my global_id is 1024131 2024-09-16T04:07:55.823 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.813+0000 7fc6eaffd640 1 -- 172.21.15.42:0/3414898129 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc6d8003530 con 0x7fc6ec0a1330 2024-09-16T04:07:55.823 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.813+0000 7fc6eb7fe640 10 monclient(hunting): handle_auth_done global_id 1014206 payload 274 2024-09-16T04:07:55.823 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.813+0000 7fc6eb7fe640 1 -- 172.21.15.42:0/3414898129 >> v1:172.21.15.154:6789/0 conn(0x7fc6ec0a1e20 legacy=0x7fc6ec13c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:55.823 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.813+0000 7fc6eb7fe640 1 -- 172.21.15.42:0/3414898129 >> v1:172.21.15.42:6789/0 conn(0x7fc6ec0a1330 legacy=0x7fc6ec0a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:55.823 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.813+0000 7fc6eb7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:07:55.823 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.813+0000 7fc6eb7fe640 1 monclient: found mon.noname-b 2024-09-16T04:07:55.823 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.813+0000 7fc6eb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:07:55.823 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.813+0000 7fc6eb7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T04:07:55.823 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.813+0000 7fc6eb7fe640 1 -- 172.21.15.42:0/3414898129 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6ec09f8f0 con 0x7fc6ec06dac0 2024-09-16T04:07:55.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.817+0000 7fc6eb7fe640 1 --2- 172.21.15.42:0/3414898129 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc6ec06dac0 0x7fc6ec13a340 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fc6dc0097f0 tx=0x7fc6dc01c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b96c9f0daeccef70 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:07:55.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.817+0000 7fc6eaffd640 1 -- 172.21.15.42:0/3414898129 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6dc0aa610 con 0x7fc6ec06dac0 2024-09-16T04:07:55.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.817+0000 7fc6eaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:55.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.817+0000 7fc6eaffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:07:55.832 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.817+0000 7fc6eaffd640 10 monclient: dump: 2024-09-16T04:07:55.832 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:07:55.832 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:55.832 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:55.832 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:55.832 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:55.832 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:55.832 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:07:55.832 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:07:55.832 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:07:55.832 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:55.832 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.817+0000 7fc6eaffd640 10 monclient: _finish_auth 0 2024-09-16T04:07:55.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.817+0000 7fc6eaffd640 10 monclient: _check_auth_tickets 2024-09-16T04:07:55.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.817+0000 7fc6eaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:07:55.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.817+0000 7fc6f24f5640 5 monclient: authenticate success, global_id 1014206 2024-09-16T04:07:55.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.817+0000 7fc6f24f5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:07:55.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.817+0000 7fc6eaffd640 1 -- 172.21.15.42:0/3414898129 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc6dc0af070 con 0x7fc6ec06dac0 2024-09-16T04:07:55.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.817+0000 7fc6eaffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:07:55.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.817+0000 7fc6eaffd640 1 -- 172.21.15.42:0/3414898129 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6dc0aa9b0 con 0x7fc6ec06dac0 2024-09-16T04:07:55.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.817+0000 7fc6eaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:55.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.817+0000 7fc6eaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:07:55.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.817+0000 7fc6eaffd640 10 monclient: dump: 2024-09-16T04:07:55.833 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:07:55.833 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:55.833 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:55.833 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:55.834 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:55.834 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:55.834 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:07:55.834 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:07:55.834 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:07:55.834 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:55.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.817+0000 7fc6f24f5640 10 monclient: get_monmap_and_config success 2024-09-16T04:07:55.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.817+0000 7fc6f24f5640 4 set_mon_vals no callback set 2024-09-16T04:07:55.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.817+0000 7fc6f24f5640 10 monclient: shutdown 2024-09-16T04:07:55.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.817+0000 7fc6f24f5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:07:55.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.817+0000 7fc6f24f5640 1 -- 172.21.15.42:0/3414898129 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc6ec06dac0 msgr2=0x7fc6ec13a340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:55.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.817+0000 7fc6f24f5640 1 --2- 172.21.15.42:0/3414898129 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc6ec06dac0 0x7fc6ec13a340 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fc6dc0097f0 tx=0x7fc6dc01c490 comp rx=0 tx=0).stop 2024-09-16T04:07:55.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.817+0000 7fc6f24f5640 1 -- 172.21.15.42:0/3414898129 shutdown_connections 2024-09-16T04:07:55.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.817+0000 7fc6f24f5640 1 --2- 172.21.15.42:0/3414898129 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc6ec06dac0 0x7fc6ec13a340 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:55.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.817+0000 7fc6f24f5640 1 -- 172.21.15.42:0/3414898129 >> 172.21.15.42:0/3414898129 conn(0x7fc6ec0a2420 msgr2=0x7fc6ec0a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:07:55.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.817+0000 7fc6f24f5640 1 -- 172.21.15.42:0/3414898129 shutdown_connections 2024-09-16T04:07:55.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.817+0000 7fc6f24f5640 1 -- 172.21.15.42:0/3414898129 wait complete. 2024-09-16T04:07:55.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.817+0000 7fc6f24f5640 10 monclient: build_initial_monmap 2024-09-16T04:07:55.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.817+0000 7fc6f24f5640 10 monclient: monmap: 2024-09-16T04:07:55.835 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:07:55.835 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:07:55.835 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:07:55.835 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:07:55.835 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:07:55.835 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:55.835 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:07:55.835 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:07:55.835 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:07:55.836 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:55.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.817+0000 7fc6f24f5640 1 librados: starting msgr at 2024-09-16T04:07:55.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.817+0000 7fc6f24f5640 1 librados: starting objecter 2024-09-16T04:07:55.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.817+0000 7fc6f24f5640 1 Processor -- start 2024-09-16T04:07:55.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.817+0000 7fc6f24f5640 1 -- start start 2024-09-16T04:07:55.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.817+0000 7fc6f24f5640 1 librados: setting wanted keys 2024-09-16T04:07:55.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.817+0000 7fc6f24f5640 1 librados: calling monclient init 2024-09-16T04:07:55.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.817+0000 7fc6f24f5640 10 monclient: init 2024-09-16T04:07:55.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.817+0000 7fc6f24f5640 10 monclient: _reopen_session rank -1 2024-09-16T04:07:55.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.817+0000 7fc6f24f5640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:07:55.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.817+0000 7fc6f24f5640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc6ec0a1330 0x7fc6ec135370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:55.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.817+0000 7fc6f24f5640 10 monclient(hunting): picked mon.noname-b con 0x7fc6ec0a1330 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:07:55.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.817+0000 7fc6f24f5640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc6ec0a1e20 0x7fc6ec11cb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:55.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.817+0000 7fc6f24f5640 10 monclient(hunting): picked mon.noname-a con 0x7fc6ec0a1e20 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:07:55.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.817+0000 7fc6f24f5640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc6ec11d060 0x7fc6ec11d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:55.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.817+0000 7fc6f24f5640 10 monclient(hunting): picked mon.noname-c con 0x7fc6ec11d060 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:07:55.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.817+0000 7fc6f24f5640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:55.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.817+0000 7fc6f24f5640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fc6ec140950 con 0x7fc6ec0a1e20 2024-09-16T04:07:55.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.817+0000 7fc6f24f5640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:55.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.817+0000 7fc6f24f5640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc6ec1407d0 con 0x7fc6ec0a1330 2024-09-16T04:07:55.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.817+0000 7fc6f24f5640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:55.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.817+0000 7fc6f24f5640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fc6ec140ad0 con 0x7fc6ec11d060 2024-09-16T04:07:55.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.817+0000 7fc6f24f5640 10 monclient(hunting): _renew_subs 2024-09-16T04:07:55.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.817+0000 7fc6f24f5640 10 monclient(hunting): authenticate will time out at 2522.791474s 2024-09-16T04:07:55.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.817+0000 7fc6eb7fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc6ec0a1e20 0x7fc6ec11cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:55.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.817+0000 7fc6f0a92640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc6ec11d060 0x7fc6ec11d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:55.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.817+0000 7fc6eb7fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc6ec0a1e20 0x7fc6ec11cb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:43484/0 (socket says 172.21.15.42:43484) 2024-09-16T04:07:55.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.817+0000 7fc6eb7fe640 1 -- 172.21.15.42:0/2011281623 learned_addr learned my addr 172.21.15.42:0/2011281623 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:07:55.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.817+0000 7fc6f0a92640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc6ec11d060 0x7fc6ec11d4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:37204/0 (socket says 172.21.15.42:37204) 2024-09-16T04:07:55.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.817+0000 7fc6eb7fe640 10 monclient(hunting): get_auth_request con 0x7fc6ec0a1e20 auth_method 0 2024-09-16T04:07:55.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.817+0000 7fc6eb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:55.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.817+0000 7fc6eb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:55.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.817+0000 7fc6eb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:55.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.817+0000 7fc6f0a92640 10 monclient(hunting): get_auth_request con 0x7fc6ec11d060 auth_method 0 2024-09-16T04:07:55.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.817+0000 7fc6f0a92640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:55.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.817+0000 7fc6f0a92640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:55.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.817+0000 7fc6f0a92640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:55.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.817+0000 7fc6eb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:07:55.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.817+0000 7fc6eb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:07:55.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.817+0000 7fc6eb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:07:55.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.817+0000 7fc6f0a92640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:07:55.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.817+0000 7fc6f0a92640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:07:55.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.817+0000 7fc6f0a92640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:07:55.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.817+0000 7fc6ebfff640 1 --2- 172.21.15.42:0/2011281623 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc6ec0a1330 0x7fc6ec135370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:55.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.817+0000 7fc6ebfff640 10 monclient(hunting): get_auth_request con 0x7fc6ec0a1330 auth_method 0 2024-09-16T04:07:55.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.817+0000 7fc6ebfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:55.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.817+0000 7fc6ebfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:55.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.817+0000 7fc6ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:55.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.821+0000 7fc6f0a92640 10 monclient(hunting): handle_auth_done global_id 1014209 payload 824 2024-09-16T04:07:55.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.821+0000 7fc6f0a92640 1 -- 172.21.15.42:0/2011281623 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc6ec0a1330 msgr2=0x7fc6ec135370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:55.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.821+0000 7fc6f0a92640 1 --2- 172.21.15.42:0/2011281623 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc6ec0a1330 0x7fc6ec135370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:55.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.821+0000 7fc6f0a92640 1 -- 172.21.15.42:0/2011281623 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc6ec0a1e20 msgr2=0x7fc6ec11cb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:55.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.821+0000 7fc6f0a92640 1 --2- 172.21.15.42:0/2011281623 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc6ec0a1e20 0x7fc6ec11cb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:55.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.821+0000 7fc6f0a92640 10 monclient: _finish_hunting 0 2024-09-16T04:07:55.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.821+0000 7fc6f0a92640 1 monclient: found mon.noname-c 2024-09-16T04:07:55.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.821+0000 7fc6f0a92640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:07:55.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.821+0000 7fc6f0a92640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T04:07:55.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.821+0000 7fc6f0a92640 1 -- 172.21.15.42:0/2011281623 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6ec1c6070 con 0x7fc6ec11d060 2024-09-16T04:07:55.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.821+0000 7fc6eb7fe640 1 --2- 172.21.15.42:0/2011281623 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc6ec0a1e20 0x7fc6ec11cb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:07:55.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.821+0000 7fc6ebfff640 1 --2- 172.21.15.42:0/2011281623 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc6ec0a1330 0x7fc6ec135370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T04:07:55.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.821+0000 7fc6f0a92640 1 --2- 172.21.15.42:0/2011281623 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc6ec11d060 0x7fc6ec11d4f0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fc6e400c700 tx=0x7fc6e400cbc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:07:55.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.821+0000 7fc6e8ff9640 10 client.?.objecter ms_handle_connect 0x7fc6ec11d060 2024-09-16T04:07:55.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.821+0000 7fc6e8ff9640 10 client.?.objecter resend_mon_ops 2024-09-16T04:07:55.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.821+0000 7fc6e8ff9640 1 -- 172.21.15.42:0/2011281623 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6e4003b80 con 0x7fc6ec11d060 2024-09-16T04:07:55.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.821+0000 7fc6e8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:55.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.821+0000 7fc6e8ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:07:55.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.821+0000 7fc6e8ff9640 10 monclient: dump: 2024-09-16T04:07:55.840 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:07:55.841 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:55.841 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:55.841 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:55.841 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:55.841 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:55.841 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:07:55.841 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:07:55.841 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:07:55.841 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:55.841 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.821+0000 7fc6e8ff9640 10 monclient: _finish_auth 0 2024-09-16T04:07:55.841 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.821+0000 7fc6e8ff9640 10 monclient: _check_auth_tickets 2024-09-16T04:07:55.841 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.821+0000 7fc6e8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:07:55.841 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.821+0000 7fc6e8ff9640 1 -- 172.21.15.42:0/2011281623 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc6e4003d20 con 0x7fc6ec11d060 2024-09-16T04:07:55.841 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.821+0000 7fc6e8ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T04:07:55.842 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.821+0000 7fc6e8ff9640 1 -- 172.21.15.42:0/2011281623 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6e400aa30 con 0x7fc6ec11d060 2024-09-16T04:07:55.842 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.821+0000 7fc6f24f5640 5 monclient: authenticate success, global_id 1014209 2024-09-16T04:07:55.842 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.821+0000 7fc6f24f5640 10 monclient: _renew_subs 2024-09-16T04:07:55.842 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.821+0000 7fc6f24f5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:07:55.842 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.821+0000 7fc6f24f5640 1 -- 172.21.15.42:0/2011281623 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6ec1207d0 con 0x7fc6ec11d060 2024-09-16T04:07:55.842 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.821+0000 7fc6e8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:55.842 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.821+0000 7fc6ea7fc640 4 set_mon_vals no callback set 2024-09-16T04:07:55.842 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.821+0000 7fc6e8ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:07:55.842 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.821+0000 7fc6e8ff9640 10 monclient: dump: 2024-09-16T04:07:55.842 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:07:55.842 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:55.842 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:55.842 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:55.843 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:55.843 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:55.843 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:07:55.843 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:07:55.843 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:07:55.843 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:55.843 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.821+0000 7fc6f24f5640 10 client.1014209.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:07:55.843 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.821+0000 7fc6f24f5640 10 monclient: _renew_subs 2024-09-16T04:07:55.843 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.821+0000 7fc6f24f5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:07:55.843 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.821+0000 7fc6f24f5640 1 -- 172.21.15.42:0/2011281623 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6ec09cb60 con 0x7fc6ec11d060 2024-09-16T04:07:55.843 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.821+0000 7fc6f24f5640 1 librados: init done 2024-09-16T04:07:55.843 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.821+0000 7fc6e8ff9640 1 -- 172.21.15.42:0/2011281623 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fc6e400abd0 con 0x7fc6ec11d060 2024-09-16T04:07:55.843 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.821+0000 7fc6f24f5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:07:55.843 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.821+0000 7fc6f24f5640 1 --2- 172.21.15.42:0/2011281623 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc6ec06dac0 0x7fc6ec11da30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:55.844 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.821+0000 7fc6f24f5640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:55.844 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.821+0000 7fc6f24f5640 1 -- 172.21.15.42:0/2011281623 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc6ec13d880 con 0x7fc6ec06dac0 2024-09-16T04:07:55.844 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.821+0000 7fc6e8ff9640 1 --2- 172.21.15.42:0/2011281623 >> v2:172.21.15.154:6812/254845342 conn(0x7fc6cc037810 0x7fc6cc039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:55.844 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.821+0000 7fc6e8ff9640 1 -- 172.21.15.42:0/2011281623 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(706..706 src has 451..706) ==== 5040+0+0 (secure 0 0 0) 0x7fc6e4048b60 con 0x7fc6ec11d060 2024-09-16T04:07:55.844 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.821+0000 7fc6e8ff9640 10 client.1014209.objecter ms_dispatch 0x7fc6ec000ba0 osd_map(706..706 src has 451..706) 2024-09-16T04:07:55.844 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.821+0000 7fc6e8ff9640 3 client.1014209.objecter handle_osd_map got epochs [706,706] > 0 2024-09-16T04:07:55.844 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.821+0000 7fc6e8ff9640 3 client.1014209.objecter handle_osd_map decoding full epoch 706 2024-09-16T04:07:55.844 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.821+0000 7fc6ebfff640 1 --2- 172.21.15.42:0/2011281623 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc6ec06dac0 0x7fc6ec11da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:55.844 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.821+0000 7fc6ebfff640 10 monclient: get_auth_request con 0x7fc6ec06dac0 auth_method 0 2024-09-16T04:07:55.844 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.821+0000 7fc6ebfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:55.844 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.821+0000 7fc6ebfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:55.844 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.821+0000 7fc6ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:55.845 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.821+0000 7fc6e8ff9640 20 client.1014209.objecter dump_active .. 0 homeless 2024-09-16T04:07:55.845 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.821+0000 7fc6ebfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:07:55.845 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.821+0000 7fc6ebfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:07:55.845 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.821+0000 7fc6ebfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:07:55.845 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.821+0000 7fc6eb7fe640 1 --2- 172.21.15.42:0/2011281623 >> v2:172.21.15.154:6812/254845342 conn(0x7fc6cc037810 0x7fc6cc039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:55.845 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.821+0000 7fc6eb7fe640 10 monclient: get_auth_request con 0x7fc6cc037810 auth_method 0 2024-09-16T04:07:55.845 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.821+0000 7fc6ebfff640 10 monclient(hunting): handle_auth_done global_id 905122 payload 824 2024-09-16T04:07:55.845 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.821+0000 7fc6ebfff640 1 -- 172.21.15.42:0/2011281623 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc6ec073a30 con 0x7fc6ec06dac0 2024-09-16T04:07:55.845 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.821+0000 7fc6e8ff9640 10 client.1014209.objecter ms_handle_connect 0x7fc6ec06dac0 2024-09-16T04:07:55.845 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.821+0000 7fc6e8ff9640 10 client.1014209.objecter resend_mon_ops 2024-09-16T04:07:55.845 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.821+0000 7fc6ebfff640 1 --2- 172.21.15.42:0/2011281623 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc6ec06dac0 0x7fc6ec11da30 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fc6d4000f70 tx=0x7fc6d400eb90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:07:55.845 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.821+0000 7fc6e8ff9640 1 -- 172.21.15.42:0/2011281623 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6d40058c0 con 0x7fc6ec06dac0 2024-09-16T04:07:55.845 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.821+0000 7fc6e8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:55.845 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.821+0000 7fc6e8ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:07:55.846 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.821+0000 7fc6e8ff9640 10 monclient: dump: 2024-09-16T04:07:55.846 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:07:55.846 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:55.846 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:55.846 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:55.846 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:55.846 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:55.846 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:07:55.846 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:07:55.846 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:07:55.846 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:55.846 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.821+0000 7fc6e8ff9640 10 client.1014209.objecter ms_handle_connect 0x7fc6cc037810 2024-09-16T04:07:55.846 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.821+0000 7fc6eb7fe640 1 --2- 172.21.15.42:0/2011281623 >> v2:172.21.15.154:6812/254845342 conn(0x7fc6cc037810 0x7fc6cc039cd0 secure :-1 s=READY pgs=3891 cs=0 l=1 rev1=1 crypto rx=0x7fc6dc0190a0 tx=0x7fc6dc0b5040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:07:55.846 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.821+0000 7fc6e8ff9640 1 -- 172.21.15.42:0/2011281623 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fc6d4018070 con 0x7fc6ec06dac0 2024-09-16T04:07:55.847 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.821+0000 7fc6e8ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:07:55.847 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.821+0000 7fc6e8ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:07:55.847 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.821+0000 7fc6e8ff9640 1 -- 172.21.15.42:0/2011281623 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc6ec06dac0 msgr2=0x7fc6ec11da30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:55.847 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.821+0000 7fc6e8ff9640 1 --2- 172.21.15.42:0/2011281623 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc6ec06dac0 0x7fc6ec11da30 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fc6d4000f70 tx=0x7fc6d400eb90 comp rx=0 tx=0).stop 2024-09-16T04:07:55.847 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.821+0000 7fc6e8ff9640 1 -- 172.21.15.42:0/2011281623 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc6ec06dac0 msgr2=0x7fc6ec11da30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:07:55.847 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.821+0000 7fc6e8ff9640 1 --2- 172.21.15.42:0/2011281623 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc6ec06dac0 0x7fc6ec11da30 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:55.849 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.829+0000 7fc6f24f5640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:07:55.849 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.829+0000 7fc6f24f5640 1 --2- 172.21.15.42:0/2011281623 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc6ec06eec0 0x7fc6ec06f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:55.849 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.829+0000 7fc6f24f5640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:55.849 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.829+0000 7fc6f24f5640 1 -- 172.21.15.42:0/2011281623 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc6ec0a1710 con 0x7fc6ec06eec0 2024-09-16T04:07:55.849 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.829+0000 7fc6ebfff640 1 --2- 172.21.15.42:0/2011281623 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc6ec06eec0 0x7fc6ec06f2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:55.849 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.829+0000 7fc6ebfff640 10 monclient: get_auth_request con 0x7fc6ec06eec0 auth_method 0 2024-09-16T04:07:55.849 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.829+0000 7fc6ebfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:55.849 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.829+0000 7fc6ebfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:55.849 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.829+0000 7fc6ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:55.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.829+0000 7fc6ebfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:07:55.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.829+0000 7fc6ebfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:07:55.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.829+0000 7fc6ebfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:07:55.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.829+0000 7fc6ebfff640 10 monclient(hunting): handle_auth_done global_id 905125 payload 824 2024-09-16T04:07:55.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.829+0000 7fc6ebfff640 1 -- 172.21.15.42:0/2011281623 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fc6ec11f0a0 con 0x7fc6ec06eec0 2024-09-16T04:07:55.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.829+0000 7fc6ebfff640 1 --2- 172.21.15.42:0/2011281623 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc6ec06eec0 0x7fc6ec06f2a0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fc6d4000f70 tx=0x7fc6d4001160 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:07:55.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.829+0000 7fc6e8ff9640 10 client.1014209.objecter ms_handle_connect 0x7fc6ec06eec0 2024-09-16T04:07:55.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.829+0000 7fc6e8ff9640 10 client.1014209.objecter resend_mon_ops 2024-09-16T04:07:55.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.829+0000 7fc6e8ff9640 1 -- 172.21.15.42:0/2011281623 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6d40070d0 con 0x7fc6ec06eec0 2024-09-16T04:07:55.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.829+0000 7fc6e8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:55.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.829+0000 7fc6e8ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:07:55.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.829+0000 7fc6e8ff9640 10 monclient: dump: 2024-09-16T04:07:55.850 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:07:55.850 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:55.851 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:55.851 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:55.851 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:55.851 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:55.851 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:07:55.851 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:07:55.851 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:07:55.851 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:55.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.829+0000 7fc6e8ff9640 1 -- 172.21.15.42:0/2011281623 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4880 (secure 0 0 0) 0x7fc6d40070d0 con 0x7fc6ec06eec0 2024-09-16T04:07:55.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.829+0000 7fc6e8ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:07:55.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.829+0000 7fc6e8ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:07:55.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.829+0000 7fc6e8ff9640 1 -- 172.21.15.42:0/2011281623 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc6ec06eec0 msgr2=0x7fc6ec06f2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:55.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.829+0000 7fc6e8ff9640 1 --2- 172.21.15.42:0/2011281623 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc6ec06eec0 0x7fc6ec06f2a0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fc6d4000f70 tx=0x7fc6d4001160 comp rx=0 tx=0).stop 2024-09-16T04:07:55.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.829+0000 7fc6e8ff9640 1 -- 172.21.15.42:0/2011281623 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc6ec06eec0 msgr2=0x7fc6ec06f2a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:07:55.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.829+0000 7fc6e8ff9640 1 --2- 172.21.15.42:0/2011281623 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc6ec06eec0 0x7fc6ec06f2a0 secure :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fc6d4000f70 tx=0x7fc6d4001160 comp rx=0 tx=0).stop 2024-09-16T04:07:55.852 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T04:07:55.852 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:07:55.852 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:07:55.852 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T04:07:55.852 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 238, 2024-09-16T04:07:55.852 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 604013, 2024-09-16T04:07:55.853 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T04:07:55.853 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T04:07:55.853 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T04:07:55.853 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T04:07:55.853 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:07:55.853 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 1, 2024-09-16T04:07:55.853 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:07:55.853 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T04:07:55.853 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T04:07:55.853 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:07:55.853 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:07:55.853 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:07:55.853 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:07:55.853 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:07:55.854 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:07:55.854 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:07:55.854 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:07:55.854 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:07:55.854 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:07:55.854 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:07:55.854 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:07:55.854 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:07:55.854 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T04:07:55.854 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:07:55.854 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:07:55.854 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:07:55.854 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:07:55.854 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:07:55.855 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:07:55.855 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:07:55.855 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:07:55.855 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:07:55.855 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:07:55.855 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:07:55.855 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:07:55.855 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:55.855 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T04:07:55.855 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T04:07:55.855 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T04:07:55.855 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T04:07:55.855 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T04:07:55.856 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T04:07:55.856 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:07:55.856 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:07:55.856 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T04:07:55.856 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T04:07:55.856 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T04:07:55.856 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T04:07:55.856 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T04:07:55.856 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T04:07:55.856 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T04:07:55.856 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:07:55.856 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T04:07:55.856 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:07:55.857 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:07:55.857 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:07:55.857 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:07:55.857 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:07:55.857 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:07:55.857 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:07:55.857 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:07:55.857 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:07:55.857 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:07:55.857 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:07:55.857 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:07:55.857 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T04:07:55.857 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:55.857 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T04:07:55.858 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:55.858 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:07:55.858 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:07:55.858 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:07:55.858 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:07:55.858 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:55.858 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:07:55.858 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T04:07:55.858 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:07:55.858 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:55.858 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:55.858 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:07:55.858 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T04:07:55.859 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:07:55.859 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:07:55.859 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:07:55.859 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:55.859 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T04:07:55.859 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T04:07:55.859 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:07:55.859 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:07:55.859 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:07:55.859 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:55.859 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:55.859 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:07:55.859 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:07:55.859 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:07:55.860 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:07:55.860 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:55.860 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:07:55.860 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T04:07:55.860 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:07:55.860 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:55.860 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:55.860 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:07:55.860 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T04:07:55.860 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:07:55.860 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:07:55.860 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:07:55.860 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:55.861 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T04:07:55.861 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T04:07:55.861 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:07:55.861 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:07:55.861 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:07:55.861 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:55.861 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:55.861 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:07:55.861 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:07:55.861 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:07:55.861 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:07:55.861 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:55.861 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:07:55.862 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T04:07:55.862 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:07:55.862 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:55.862 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:55.862 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:07:55.862 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T04:07:55.862 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:07:55.862 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:07:55.862 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:07:55.862 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:55.862 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T04:07:55.862 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T04:07:55.862 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:07:55.862 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:07:55.863 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:07:55.863 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:07:55.863 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:07:55.863 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:55.863 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T04:07:55.863 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T04:07:55.863 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:55.863 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:07:55.863 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:07:55.863 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:07:55.863 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:07:55.863 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:07:55.863 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T04:07:55.863 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:55.864 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:07:55.864 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:07:55.864 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 3 2024-09-16T04:07:55.864 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:07:55.864 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:07:55.864 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T04:07:55.864 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:55.864 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:07:55.864 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:07:55.864 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 5 2024-09-16T04:07:55.864 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:07:55.864 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:07:55.864 INFO:teuthology.orchestra.run.smithi042.stdout: "mgr": [ 2024-09-16T04:07:55.865 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:55.865 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:07:55.865 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:07:55.865 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:07:55.865 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:07:55.865 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:07:55.865 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:55.865 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T04:07:55.865 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T04:07:55.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.829+0000 7fc6f24f5640 10 librados: watch_flush enter 2024-09-16T04:07:55.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.829+0000 7fc6f24f5640 10 librados: watch_flush exit 2024-09-16T04:07:55.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.829+0000 7fc6f24f5640 20 client.1014209.objecter shutdown clearing up homeless session... 2024-09-16T04:07:55.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.829+0000 7fc6f24f5640 10 client.1014209.objecter successfully canceled tick 2024-09-16T04:07:55.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.833+0000 7fc6f24f5640 1 -- 172.21.15.42:0/2011281623 >> v2:172.21.15.154:6812/254845342 conn(0x7fc6cc037810 msgr2=0x7fc6cc039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:55.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.833+0000 7fc6f24f5640 1 --2- 172.21.15.42:0/2011281623 >> v2:172.21.15.154:6812/254845342 conn(0x7fc6cc037810 0x7fc6cc039cd0 secure :-1 s=READY pgs=3891 cs=0 l=1 rev1=1 crypto rx=0x7fc6dc0190a0 tx=0x7fc6dc0b5040 comp rx=0 tx=0).stop 2024-09-16T04:07:55.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.833+0000 7fc6f24f5640 10 monclient: shutdown 2024-09-16T04:07:55.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.833+0000 7fc6f24f5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:07:55.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.833+0000 7fc6f24f5640 1 -- 172.21.15.42:0/2011281623 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc6ec11d060 msgr2=0x7fc6ec11d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:55.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.833+0000 7fc6f24f5640 1 --2- 172.21.15.42:0/2011281623 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc6ec11d060 0x7fc6ec11d4f0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fc6e400c700 tx=0x7fc6e400cbc0 comp rx=0 tx=0).stop 2024-09-16T04:07:55.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.833+0000 7fc6f0a92640 1 -- 172.21.15.42:0/2011281623 reap_dead start 2024-09-16T04:07:55.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.833+0000 7fc6f0a92640 1 -- 172.21.15.42:0/2011281623 reap_dead start 2024-09-16T04:07:55.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.833+0000 7fc6f24f5640 1 -- 172.21.15.42:0/2011281623 shutdown_connections 2024-09-16T04:07:55.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.833+0000 7fc6f24f5640 1 -- 172.21.15.42:0/2011281623 >> 172.21.15.42:0/2011281623 conn(0x7fc6ec0a2420 msgr2=0x7fc6ec0a45b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:07:55.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.833+0000 7fc6f24f5640 1 -- 172.21.15.42:0/2011281623 shutdown_connections 2024-09-16T04:07:55.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.833+0000 7fc6f24f5640 1 -- 172.21.15.42:0/2011281623 wait complete. 2024-09-16T04:07:55.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.833+0000 7fc6f24f5640 1 librados: shutdown 2024-09-16T04:07:55.868 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:07:55.869 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T04:07:55.869 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T04:07:55.984 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.977+0000 7f9eed283640 10 monclient: get_monmap_and_config 2024-09-16T04:07:55.984 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.977+0000 7f9eed283640 10 monclient: build_initial_monmap 2024-09-16T04:07:55.984 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.977+0000 7f9eed283640 10 monclient: monmap: 2024-09-16T04:07:55.984 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:07:55.984 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:55.984 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:07:55.982748+0000 2024-09-16T04:07:55.984 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:07:55.982748+0000 2024-09-16T04:07:55.984 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:07:55.985 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:55.985 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:07:55.985 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:07:55.985 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:07:55.985 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:07:55.985 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:07:55.985 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:55.987 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.977+0000 7f9eed283640 1 Processor -- start 2024-09-16T04:07:55.987 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.977+0000 7f9eed283640 1 -- start start 2024-09-16T04:07:55.987 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.977+0000 7f9eed283640 10 monclient: init 2024-09-16T04:07:55.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.981+0000 7f9eed283640 10 monclient: _reopen_session rank -1 2024-09-16T04:07:55.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.981+0000 7f9eed283640 10 monclient: _add_conns ranks=[3,0,4,1,2] 2024-09-16T04:07:55.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.981+0000 7f9eed283640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f9ee806db90 0x7f9ee81340c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:55.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.981+0000 7f9eed283640 10 monclient(hunting): picked mon.noname-a con 0x7f9ee806db90 addr v2:172.21.15.42:3300/0 2024-09-16T04:07:55.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.981+0000 7f9eed283640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9ee80a37f0 addr v1:172.21.15.42:6789/0 2024-09-16T04:07:55.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.981+0000 7f9eed283640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f9ee80a3c00 0x7f9ee8137520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:55.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.981+0000 7f9eed283640 10 monclient(hunting): picked mon.noname-c con 0x7f9ee80a3c00 addr v2:172.21.15.154:3300/0 2024-09-16T04:07:55.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.981+0000 7f9eed283640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9ee806ece0 con 0x7f9ee80a37f0 2024-09-16T04:07:55.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.981+0000 7f9eed283640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:55.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.981+0000 7f9eed283640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f9ee806df70 con 0x7f9ee806db90 2024-09-16T04:07:55.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.981+0000 7f9eed283640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:55.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.981+0000 7f9eed283640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f9ee809bf80 con 0x7f9ee80a3c00 2024-09-16T04:07:55.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.981+0000 7f9eed283640 10 monclient(hunting): _renew_subs 2024-09-16T04:07:55.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.981+0000 7f9eed283640 10 monclient(hunting): authenticate will time out at 2522.952324s 2024-09-16T04:07:55.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.981+0000 7f9ee6ffd640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f9ee806db90 0x7f9ee81340c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:55.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.981+0000 7f9ee67fc640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f9ee80a37f0 0x7f9ee8136e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:44574/0 (socket says 172.21.15.42:44574) 2024-09-16T04:07:55.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.981+0000 7f9ee67fc640 1 -- 172.21.15.42:0/1932130191 learned_addr learned my addr 172.21.15.42:0/1932130191 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T04:07:55.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.981+0000 7f9ee6ffd640 10 monclient(hunting): get_auth_request con 0x7f9ee806db90 auth_method 0 2024-09-16T04:07:55.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.981+0000 7f9ee6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:55.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.981+0000 7f9ee6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:55.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.981+0000 7f9ee6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:55.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.981+0000 7f9ee6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:07:55.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.981+0000 7f9ee6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:07:55.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.981+0000 7f9ee6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:07:55.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.981+0000 7f9ee77fe640 1 --2- 172.21.15.42:0/1932130191 >> v2:172.21.15.154:3300/0 conn(0x7f9ee80a3c00 0x7f9ee8137520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:55.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.981+0000 7f9ee77fe640 10 monclient(hunting): get_auth_request con 0x7f9ee80a3c00 auth_method 0 2024-09-16T04:07:55.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.981+0000 7f9ee77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:55.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.981+0000 7f9ee77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:55.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.981+0000 7f9ee77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:55.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.981+0000 7f9ee6ffd640 10 monclient(hunting): handle_auth_done global_id 1024137 payload 274 2024-09-16T04:07:55.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.981+0000 7f9ee6ffd640 1 -- 172.21.15.42:0/1932130191 >> v2:172.21.15.154:3300/0 conn(0x7f9ee80a3c00 msgr2=0x7f9ee8137520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:07:55.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.981+0000 7f9ee6ffd640 1 --2- 172.21.15.42:0/1932130191 >> v2:172.21.15.154:3300/0 conn(0x7f9ee80a3c00 0x7f9ee8137520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:55.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.981+0000 7f9ee6ffd640 1 -- 172.21.15.42:0/1932130191 >> v1:172.21.15.42:6789/0 conn(0x7f9ee80a37f0 legacy=0x7f9ee8136e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:55.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.981+0000 7f9ee6ffd640 10 monclient: _finish_hunting 0 2024-09-16T04:07:55.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.981+0000 7f9ee6ffd640 1 monclient: found mon.noname-a 2024-09-16T04:07:55.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.981+0000 7f9ee5ffb640 1 -- 172.21.15.42:0/1932130191 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 793356081 0 0) 0x7f9ee806ece0 con 0x7f9ee80a37f0 2024-09-16T04:07:55.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.981+0000 7f9ee6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:07:55.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.981+0000 7f9ee6ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:07:55.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.981+0000 7f9ee6ffd640 1 -- 172.21.15.42:0/1932130191 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ee8134780 con 0x7f9ee806db90 2024-09-16T04:07:55.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.981+0000 7f9ee5ffb640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T04:07:55.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.981+0000 7f9ee6ffd640 1 --2- 172.21.15.42:0/1932130191 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9ee806db90 0x7f9ee81340c0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f9ee809bbd0 tx=0x7f9edc01c310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6962dcfa0dd34f5 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:07:55.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.981+0000 7f9ee5ffb640 1 -- 172.21.15.42:0/1932130191 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9edc0b0070 con 0x7f9ee806db90 2024-09-16T04:07:55.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.981+0000 7f9ee5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:55.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.981+0000 7f9ee5ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:07:55.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.981+0000 7f9ee5ffb640 10 monclient: dump: 2024-09-16T04:07:55.997 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:07:55.997 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:55.997 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:55.997 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:55.997 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:55.997 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:55.997 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:07:55.997 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:07:55.997 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:07:55.997 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:55.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.981+0000 7f9ee5ffb640 10 monclient: _finish_auth 0 2024-09-16T04:07:55.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.981+0000 7f9ee5ffb640 10 monclient: _check_auth_tickets 2024-09-16T04:07:55.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.981+0000 7f9ee5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:07:55.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.981+0000 7f9eed283640 5 monclient: authenticate success, global_id 1024137 2024-09-16T04:07:55.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.981+0000 7f9ee5ffb640 1 -- 172.21.15.42:0/1932130191 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9edc0ab600 con 0x7f9ee806db90 2024-09-16T04:07:55.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.981+0000 7f9eed283640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:07:55.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.981+0000 7f9ee5ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:07:55.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.981+0000 7f9eed283640 10 monclient: get_monmap_and_config success 2024-09-16T04:07:55.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.981+0000 7f9eed283640 4 set_mon_vals no callback set 2024-09-16T04:07:55.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.981+0000 7f9ee5ffb640 1 -- 172.21.15.42:0/1932130191 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9edc0ab9d0 con 0x7f9ee806db90 2024-09-16T04:07:55.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.981+0000 7f9ee5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:55.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.981+0000 7f9ee5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:07:55.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.981+0000 7f9ee5ffb640 10 monclient: dump: 2024-09-16T04:07:55.998 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:07:55.998 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:55.999 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:55.999 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:55.999 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:55.999 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:55.999 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:07:55.999 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:07:55.999 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:07:55.999 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:55.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.981+0000 7f9eed283640 10 monclient: shutdown 2024-09-16T04:07:55.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.981+0000 7f9eed283640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:07:55.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.981+0000 7f9eed283640 1 -- 172.21.15.42:0/1932130191 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9ee806db90 msgr2=0x7f9ee81340c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:55.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.981+0000 7f9eed283640 1 --2- 172.21.15.42:0/1932130191 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9ee806db90 0x7f9ee81340c0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f9ee809bbd0 tx=0x7f9edc01c310 comp rx=0 tx=0).stop 2024-09-16T04:07:55.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.985+0000 7f9eed283640 1 -- 172.21.15.42:0/1932130191 shutdown_connections 2024-09-16T04:07:56.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.985+0000 7f9eed283640 1 --2- 172.21.15.42:0/1932130191 >> v2:172.21.15.154:3300/0 conn(0x7f9ee80a3c00 0x7f9ee8137520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:56.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.985+0000 7f9eed283640 1 --2- 172.21.15.42:0/1932130191 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9ee806db90 0x7f9ee81340c0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:56.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.985+0000 7f9eed283640 1 -- 172.21.15.42:0/1932130191 >> 172.21.15.42:0/1932130191 conn(0x7f9ee80a9fb0 msgr2=0x7f9ee80aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:07:56.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.985+0000 7f9eed283640 1 -- 172.21.15.42:0/1932130191 shutdown_connections 2024-09-16T04:07:56.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.985+0000 7f9eed283640 1 -- 172.21.15.42:0/1932130191 wait complete. 2024-09-16T04:07:56.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.985+0000 7f9eed283640 10 monclient: build_initial_monmap 2024-09-16T04:07:56.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.985+0000 7f9eed283640 10 monclient: monmap: 2024-09-16T04:07:56.000 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:07:56.000 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:07:56.000 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:07:56.000 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:07:56.000 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:07:56.000 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:56.000 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:07:56.001 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:07:56.001 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:07:56.001 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:56.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.985+0000 7f9eed283640 1 librados: starting msgr at 2024-09-16T04:07:56.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.985+0000 7f9eed283640 1 librados: starting objecter 2024-09-16T04:07:56.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.985+0000 7f9eed283640 1 Processor -- start 2024-09-16T04:07:56.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.985+0000 7f9eed283640 1 -- start start 2024-09-16T04:07:56.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.985+0000 7f9eed283640 1 librados: setting wanted keys 2024-09-16T04:07:56.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.985+0000 7f9eed283640 1 librados: calling monclient init 2024-09-16T04:07:56.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.985+0000 7f9eed283640 10 monclient: init 2024-09-16T04:07:56.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.985+0000 7f9eed283640 10 monclient: _reopen_session rank -1 2024-09-16T04:07:56.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.985+0000 7f9eed283640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:07:56.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.985+0000 7f9eed283640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9ee806db90 0x7f9ee80a1690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:56.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.985+0000 7f9eed283640 10 monclient(hunting): picked mon.noname-a con 0x7f9ee806db90 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:07:56.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.985+0000 7f9eed283640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9ee80a37f0 0x7f9ee80a1bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:56.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.985+0000 7f9eed283640 10 monclient(hunting): picked mon.noname-b con 0x7f9ee80a37f0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:07:56.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.985+0000 7f9eed283640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9ee80a3c00 0x7f9ee80a2110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:56.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.985+0000 7f9eed283640 10 monclient(hunting): picked mon.noname-c con 0x7f9ee80a3c00 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:07:56.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.985+0000 7f9eed283640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:56.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.985+0000 7f9eed283640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f9ee80a40d0 con 0x7f9ee806db90 2024-09-16T04:07:56.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.985+0000 7f9eed283640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:56.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.985+0000 7f9eed283640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f9ee8137c70 con 0x7f9ee80a37f0 2024-09-16T04:07:56.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.985+0000 7f9eed283640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:56.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.985+0000 7f9eed283640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f9ee81419d0 con 0x7f9ee80a3c00 2024-09-16T04:07:56.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.985+0000 7f9eed283640 10 monclient(hunting): _renew_subs 2024-09-16T04:07:56.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.985+0000 7f9eed283640 10 monclient(hunting): authenticate will time out at 2522.957613s 2024-09-16T04:07:56.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.985+0000 7f9ee6ffd640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9ee806db90 0x7f9ee80a1690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:56.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.985+0000 7f9ee77fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9ee80a3c00 0x7f9ee80a2110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:56.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.985+0000 7f9ee6ffd640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9ee806db90 0x7f9ee80a1690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:43516/0 (socket says 172.21.15.42:43516) 2024-09-16T04:07:56.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.985+0000 7f9ee6ffd640 1 -- 172.21.15.42:0/341935786 learned_addr learned my addr 172.21.15.42:0/341935786 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:07:56.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.985+0000 7f9ee77fe640 10 monclient(hunting): get_auth_request con 0x7f9ee80a3c00 auth_method 0 2024-09-16T04:07:56.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.985+0000 7f9ee77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:56.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.985+0000 7f9ee77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:56.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.985+0000 7f9ee77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:56.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.985+0000 7f9ee6ffd640 10 monclient(hunting): get_auth_request con 0x7f9ee806db90 auth_method 0 2024-09-16T04:07:56.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.985+0000 7f9ee6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:56.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.985+0000 7f9ee6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:56.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.985+0000 7f9ee6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:56.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.985+0000 7f9ee77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:07:56.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.985+0000 7f9ee77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:07:56.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.985+0000 7f9ee77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:07:56.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.985+0000 7f9ee6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:07:56.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.985+0000 7f9ee6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:07:56.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.985+0000 7f9ee6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:07:56.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.985+0000 7f9ee67fc640 1 --2- 172.21.15.42:0/341935786 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9ee80a37f0 0x7f9ee80a1bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:56.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.985+0000 7f9ee77fe640 10 monclient(hunting): handle_auth_done global_id 1014212 payload 824 2024-09-16T04:07:56.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.985+0000 7f9ee77fe640 1 -- 172.21.15.42:0/341935786 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9ee80a37f0 msgr2=0x7f9ee80a1bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:56.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.985+0000 7f9ee77fe640 1 --2- 172.21.15.42:0/341935786 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9ee80a37f0 0x7f9ee80a1bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:56.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.985+0000 7f9ee77fe640 1 -- 172.21.15.42:0/341935786 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9ee806db90 msgr2=0x7f9ee80a1690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:56.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.985+0000 7f9ee77fe640 1 --2- 172.21.15.42:0/341935786 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9ee806db90 0x7f9ee80a1690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:56.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.985+0000 7f9ee77fe640 10 monclient: _finish_hunting 0 2024-09-16T04:07:56.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.985+0000 7f9ee77fe640 1 monclient: found mon.noname-c 2024-09-16T04:07:56.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.985+0000 7f9ee77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:07:56.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.985+0000 7f9ee77fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T04:07:56.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.985+0000 7f9ee77fe640 1 -- 172.21.15.42:0/341935786 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ee806f6a0 con 0x7f9ee80a3c00 2024-09-16T04:07:56.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.985+0000 7f9ee67fc640 10 monclient: get_auth_request con 0x7f9ee80a37f0 auth_method 0 2024-09-16T04:07:56.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.985+0000 7f9ee67fc640 1 --2- 172.21.15.42:0/341935786 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9ee80a37f0 0x7f9ee80a1bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T04:07:56.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.985+0000 7f9ee6ffd640 1 --2- 172.21.15.42:0/341935786 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9ee806db90 0x7f9ee80a1690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:07:56.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.985+0000 7f9ee77fe640 1 --2- 172.21.15.42:0/341935786 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9ee80a3c00 0x7f9ee80a2110 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f9ee0007d90 tx=0x7f9ee000e4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:07:56.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.985+0000 7f9ecf7fe640 10 client.?.objecter ms_handle_connect 0x7f9ee80a3c00 2024-09-16T04:07:56.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.985+0000 7f9ecf7fe640 10 client.?.objecter resend_mon_ops 2024-09-16T04:07:56.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.985+0000 7f9ecf7fe640 1 -- 172.21.15.42:0/341935786 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ee000edd0 con 0x7f9ee80a3c00 2024-09-16T04:07:56.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.985+0000 7f9ecf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:56.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.985+0000 7f9ecf7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:07:56.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.985+0000 7f9ecf7fe640 10 monclient: dump: 2024-09-16T04:07:56.006 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:07:56.006 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:56.006 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:56.006 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:56.006 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:56.006 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:56.006 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:07:56.006 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:07:56.006 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:07:56.006 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:56.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.985+0000 7f9ecf7fe640 10 monclient: _finish_auth 0 2024-09-16T04:07:56.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.985+0000 7f9ecf7fe640 10 monclient: _check_auth_tickets 2024-09-16T04:07:56.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.985+0000 7f9ecf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:07:56.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.985+0000 7f9ecf7fe640 1 -- 172.21.15.42:0/341935786 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9ee00039a0 con 0x7f9ee80a3c00 2024-09-16T04:07:56.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.985+0000 7f9eed283640 5 monclient: authenticate success, global_id 1014212 2024-09-16T04:07:56.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.985+0000 7f9eed283640 10 monclient: _renew_subs 2024-09-16T04:07:56.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.985+0000 7f9eed283640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:07:56.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.985+0000 7f9eed283640 1 -- 172.21.15.42:0/341935786 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ee80a32f0 con 0x7f9ee80a3c00 2024-09-16T04:07:56.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.985+0000 7f9ecf7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:07:56.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.985+0000 7f9ecf7fe640 1 -- 172.21.15.42:0/341935786 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ee0003d90 con 0x7f9ee80a3c00 2024-09-16T04:07:56.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.985+0000 7f9ecf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:56.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.985+0000 7f9ecf7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:07:56.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.985+0000 7f9ecf7fe640 10 monclient: dump: 2024-09-16T04:07:56.007 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:07:56.007 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:56.007 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:56.007 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:56.008 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:56.008 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:56.008 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:07:56.008 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:07:56.008 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:07:56.008 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:56.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.985+0000 7f9eed283640 10 client.1014212.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:07:56.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.985+0000 7f9eed283640 10 monclient: _renew_subs 2024-09-16T04:07:56.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.985+0000 7f9eed283640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:07:56.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.985+0000 7f9eed283640 1 -- 172.21.15.42:0/341935786 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ee8138510 con 0x7f9ee80a3c00 2024-09-16T04:07:56.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.985+0000 7f9eed283640 1 librados: init done 2024-09-16T04:07:56.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.985+0000 7f9ee57fa640 4 set_mon_vals no callback set 2024-09-16T04:07:56.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.985+0000 7f9ecf7fe640 1 -- 172.21.15.42:0/341935786 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f9ee001b020 con 0x7f9ee80a3c00 2024-09-16T04:07:56.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.985+0000 7f9ecf7fe640 1 --2- 172.21.15.42:0/341935786 >> v2:172.21.15.154:6812/254845342 conn(0x7f9ed4037810 0x7f9ed4039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:56.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.985+0000 7f9eed283640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:07:56.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.985+0000 7f9eed283640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:07:56.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.985+0000 7f9eed283640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:07:56.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.985+0000 7f9eed283640 1 -- 172.21.15.42:0/341935786 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ee81340c0 con 0x7f9ee80a3c00 2024-09-16T04:07:56.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.985+0000 7f9ecf7fe640 1 -- 172.21.15.42:0/341935786 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(706..706 src has 451..706) ==== 5040+0+0 (secure 0 0 0) 0x7f9ee0048e00 con 0x7f9ee80a3c00 2024-09-16T04:07:56.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.985+0000 7f9ecf7fe640 10 client.1014212.objecter ms_dispatch 0x7f9ee8000ba0 osd_map(706..706 src has 451..706) 2024-09-16T04:07:56.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.985+0000 7f9ecf7fe640 3 client.1014212.objecter handle_osd_map got epochs [706,706] > 0 2024-09-16T04:07:56.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.985+0000 7f9ecf7fe640 3 client.1014212.objecter handle_osd_map decoding full epoch 706 2024-09-16T04:07:56.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.989+0000 7f9ecf7fe640 20 client.1014212.objecter dump_active .. 0 homeless 2024-09-16T04:07:56.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.989+0000 7f9ee6ffd640 1 --2- 172.21.15.42:0/341935786 >> v2:172.21.15.154:6812/254845342 conn(0x7f9ed4037810 0x7f9ed4039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:56.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.989+0000 7f9ee6ffd640 10 monclient: get_auth_request con 0x7f9ed4037810 auth_method 0 2024-09-16T04:07:56.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.989+0000 7f9ee6ffd640 1 --2- 172.21.15.42:0/341935786 >> v2:172.21.15.154:6812/254845342 conn(0x7f9ed4037810 0x7f9ed4039cd0 secure :-1 s=READY pgs=3892 cs=0 l=1 rev1=1 crypto rx=0x7f9edc01c350 tx=0x7f9edc008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:07:56.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.989+0000 7f9ecf7fe640 10 client.1014212.objecter ms_handle_connect 0x7f9ed4037810 2024-09-16T04:07:56.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.993+0000 7f9ecf7fe640 1 -- 172.21.15.42:0/341935786 <== mon.2 v2:172.21.15.42:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f9ee0004110 con 0x7f9ee80a3c00 2024-09-16T04:07:56.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.993+0000 7f9ecf7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:07:56.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:55.993+0000 7f9ecf7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:07:56.223 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:56.217+0000 7f9eed283640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T04:07:56.224 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:56.217+0000 7f9eed283640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T04:07:56.224 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:56.217+0000 7f9eed283640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:07:56.224 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:56.217+0000 7f9eed283640 1 -- 172.21.15.42:0/341935786 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f9ee8074170 con 0x7f9ee80a3c00 2024-09-16T04:07:56.225 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:56.217+0000 7f9ecf7fe640 1 -- 172.21.15.42:0/341935786 <== mon.2 v2:172.21.15.42:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f9ee001cc90 con 0x7f9ee80a3c00 2024-09-16T04:07:56.225 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:56.217+0000 7f9ecf7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T04:07:56.225 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:56.217+0000 7f9ecf7fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:07:56.227 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.221+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.227 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.221+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=30,config=2,health=11,kv=9,logm=48} crc {auth=3265663900,config=692619927,health=2889297937,kv=3276479304,logm=1217337293}) 2024-09-16T04:07:56.227 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.221+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=27,config=2,health=11,kv=10,logm=50} crc {auth=2911015188,config=692619927,health=1277370358,kv=3446188498,logm=3927020217}) 2024-09-16T04:07:56.227 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.221+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.227 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.221+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=30,config=2,health=11,kv=9,logm=48} crc {auth=3265663900,config=692619927,health=2889297937,kv=3276479304,logm=1217337293}) 2024-09-16T04:07:56.227 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.221+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=29,config=2,health=12,kv=9,logm=48} crc {auth=3308930419,config=692619927,health=2018877389,kv=867007345,logm=699403686}) 2024-09-16T04:07:56.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:56.221+0000 7f9eed283640 10 librados: watch_flush enter 2024-09-16T04:07:56.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:56.221+0000 7f9eed283640 10 librados: watch_flush exit 2024-09-16T04:07:56.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:56.221+0000 7f9eed283640 20 client.1014212.objecter shutdown clearing up homeless session... 2024-09-16T04:07:56.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:56.221+0000 7f9eed283640 10 client.1014212.objecter successfully canceled tick 2024-09-16T04:07:56.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:56.221+0000 7f9eed283640 1 -- 172.21.15.42:0/341935786 >> v2:172.21.15.154:6812/254845342 conn(0x7f9ed4037810 msgr2=0x7f9ed4039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:56.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:56.221+0000 7f9eed283640 1 --2- 172.21.15.42:0/341935786 >> v2:172.21.15.154:6812/254845342 conn(0x7f9ed4037810 0x7f9ed4039cd0 secure :-1 s=READY pgs=3892 cs=0 l=1 rev1=1 crypto rx=0x7f9edc01c350 tx=0x7f9edc008040 comp rx=0 tx=0).stop 2024-09-16T04:07:56.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:56.221+0000 7f9eed283640 10 monclient: shutdown 2024-09-16T04:07:56.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:56.221+0000 7f9eed283640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:07:56.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:56.221+0000 7f9eed283640 1 -- 172.21.15.42:0/341935786 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9ee80a3c00 msgr2=0x7f9ee80a2110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:56.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:56.221+0000 7f9eed283640 1 --2- 172.21.15.42:0/341935786 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9ee80a3c00 0x7f9ee80a2110 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f9ee0007d90 tx=0x7f9ee000e4b0 comp rx=0 tx=0).stop 2024-09-16T04:07:56.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:56.221+0000 7f9eed283640 1 -- 172.21.15.42:0/341935786 shutdown_connections 2024-09-16T04:07:56.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:56.221+0000 7f9eed283640 1 --2- 172.21.15.42:0/341935786 >> v2:172.21.15.154:6812/254845342 conn(0x7f9ed4037810 0x7f9ed4039cd0 unknown :-1 s=CLOSED pgs=3892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:56.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:56.221+0000 7f9eed283640 1 --2- 172.21.15.42:0/341935786 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9ee80a3c00 0x7f9ee80a2110 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:56.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:56.221+0000 7f9eed283640 1 --2- 172.21.15.42:0/341935786 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9ee80a37f0 0x7f9ee80a1bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:56.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:56.221+0000 7f9eed283640 1 --2- 172.21.15.42:0/341935786 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9ee806db90 0x7f9ee80a1690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:56.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:56.221+0000 7f9eed283640 1 -- 172.21.15.42:0/341935786 >> 172.21.15.42:0/341935786 conn(0x7f9ee80a9fb0 msgr2=0x7f9ee81369f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:07:56.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:56.221+0000 7f9eed283640 1 -- 172.21.15.42:0/341935786 shutdown_connections 2024-09-16T04:07:56.230 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:56.221+0000 7f9eed283640 1 -- 172.21.15.42:0/341935786 wait complete. 2024-09-16T04:07:56.230 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:56.221+0000 7f9eed283640 1 librados: shutdown 2024-09-16T04:07:56.231 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.225+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.231 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.225+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=176112912}) 2024-09-16T04:07:56.231 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.225+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1634194647}) 2024-09-16T04:07:56.231 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.225+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.231 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.225+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=176112912}) 2024-09-16T04:07:56.231 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.225+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2004611289}) 2024-09-16T04:07:56.234 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.229+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.235 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.229+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3770804897}) 2024-09-16T04:07:56.235 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.229+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=761144596}) 2024-09-16T04:07:56.235 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.229+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.235 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.229+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3770804897}) 2024-09-16T04:07:56.235 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.229+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=31004890}) 2024-09-16T04:07:56.237 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.229+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.237 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.229+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=419217879}) 2024-09-16T04:07:56.237 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.229+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2836431740}) 2024-09-16T04:07:56.237 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.229+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.237 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.229+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=419217879}) 2024-09-16T04:07:56.237 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.229+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2969369652}) 2024-09-16T04:07:56.241 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.233+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.241 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.233+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3434155844}) 2024-09-16T04:07:56.241 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.233+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2324551881}) 2024-09-16T04:07:56.241 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.233+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.241 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.233+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3434155844}) 2024-09-16T04:07:56.241 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.237+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3126720417}) 2024-09-16T04:07:56.244 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.237+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.244 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.237+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1400611894}) 2024-09-16T04:07:56.244 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.237+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=722135172}) 2024-09-16T04:07:56.244 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.237+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.244 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.237+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1400611894}) 2024-09-16T04:07:56.244 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.237+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1282572160}) 2024-09-16T04:07:56.245 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T04:07:56.246 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.241+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.246 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.241+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1179766020}) 2024-09-16T04:07:56.246 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.241+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=166098484}) 2024-09-16T04:07:56.246 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.241+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.247 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.241+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1179766020}) 2024-09-16T04:07:56.247 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.241+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=623615945}) 2024-09-16T04:07:56.248 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.241+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.248 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.241+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=350961847}) 2024-09-16T04:07:56.249 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.241+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4052548305}) 2024-09-16T04:07:56.249 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.241+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.249 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.241+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=350961847}) 2024-09-16T04:07:56.249 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.241+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1110110835}) 2024-09-16T04:07:56.250 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.245+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.251 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.245+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2302477669}) 2024-09-16T04:07:56.251 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.245+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4203385438}) 2024-09-16T04:07:56.251 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.245+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.251 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.245+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2302477669}) 2024-09-16T04:07:56.251 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.245+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1514815418}) 2024-09-16T04:07:56.253 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.245+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.253 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.245+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2170365542}) 2024-09-16T04:07:56.253 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.245+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=614102122}) 2024-09-16T04:07:56.253 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.245+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.253 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.245+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2170365542}) 2024-09-16T04:07:56.253 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.245+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4096035844}) 2024-09-16T04:07:56.255 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.249+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.255 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.249+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1059890488}) 2024-09-16T04:07:56.255 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.249+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1879250255}) 2024-09-16T04:07:56.255 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.249+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.255 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.249+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1059890488}) 2024-09-16T04:07:56.255 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.249+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3762766698}) 2024-09-16T04:07:56.257 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.249+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.257 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.249+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=405315324}) 2024-09-16T04:07:56.257 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.249+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=588879847}) 2024-09-16T04:07:56.257 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.249+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.257 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.249+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=405315324}) 2024-09-16T04:07:56.257 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.249+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3329156700}) 2024-09-16T04:07:56.259 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.253+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.259 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.253+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3539490924}) 2024-09-16T04:07:56.259 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.253+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1707970641}) 2024-09-16T04:07:56.259 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.253+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.259 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.253+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3539490924}) 2024-09-16T04:07:56.259 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.253+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3600213712}) 2024-09-16T04:07:56.261 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.253+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.262 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.253+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4249844190}) 2024-09-16T04:07:56.262 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.253+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3402738385}) 2024-09-16T04:07:56.262 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.253+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.262 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.253+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4249844190}) 2024-09-16T04:07:56.262 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.253+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1248397840}) 2024-09-16T04:07:56.263 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.257+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.263 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.257+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3634730084}) 2024-09-16T04:07:56.263 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.257+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2607002949}) 2024-09-16T04:07:56.264 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.257+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.264 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.257+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3634730084}) 2024-09-16T04:07:56.264 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.257+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2441719367}) 2024-09-16T04:07:56.266 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.261+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.266 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.261+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3899064520}) 2024-09-16T04:07:56.266 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.261+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2930086007}) 2024-09-16T04:07:56.266 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.261+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.266 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.261+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3899064520}) 2024-09-16T04:07:56.266 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.261+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=163751071}) 2024-09-16T04:07:56.268 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.261+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.268 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.261+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=264316396}) 2024-09-16T04:07:56.268 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.261+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1727353096}) 2024-09-16T04:07:56.268 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.261+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.268 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.261+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=264316396}) 2024-09-16T04:07:56.268 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.261+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=728126879}) 2024-09-16T04:07:56.270 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.265+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.270 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.265+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3385884773}) 2024-09-16T04:07:56.270 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.265+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3506095277}) 2024-09-16T04:07:56.271 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.265+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.271 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.265+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3385884773}) 2024-09-16T04:07:56.271 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.265+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2773724419}) 2024-09-16T04:07:56.273 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.265+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.273 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.265+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1504556820}) 2024-09-16T04:07:56.273 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.265+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1852758847}) 2024-09-16T04:07:56.273 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.265+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.273 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.265+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1504556820}) 2024-09-16T04:07:56.273 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.265+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=975683584}) 2024-09-16T04:07:56.275 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.269+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.275 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.269+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=611661670}) 2024-09-16T04:07:56.275 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.269+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=277459698}) 2024-09-16T04:07:56.275 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.269+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.275 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.269+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=611661670}) 2024-09-16T04:07:56.275 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.269+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3164765802}) 2024-09-16T04:07:56.277 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.273+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.277 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.273+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1862410535}) 2024-09-16T04:07:56.278 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.273+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1280127732}) 2024-09-16T04:07:56.278 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.273+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.278 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.273+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1862410535}) 2024-09-16T04:07:56.278 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.273+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=737663076}) 2024-09-16T04:07:56.280 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.273+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.280 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.273+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2603159639}) 2024-09-16T04:07:56.280 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.273+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4011393317}) 2024-09-16T04:07:56.280 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.273+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.280 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.273+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2603159639}) 2024-09-16T04:07:56.280 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.273+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3251307138}) 2024-09-16T04:07:56.282 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.277+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.282 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.277+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1653358510}) 2024-09-16T04:07:56.282 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.277+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3876494672}) 2024-09-16T04:07:56.282 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.277+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.283 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.277+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1653358510}) 2024-09-16T04:07:56.283 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.277+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2344302774}) 2024-09-16T04:07:56.285 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.277+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.285 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.277+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2251056037}) 2024-09-16T04:07:56.285 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.277+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=570762357}) 2024-09-16T04:07:56.285 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.277+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.285 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.277+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2251056037}) 2024-09-16T04:07:56.285 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.277+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3598389136}) 2024-09-16T04:07:56.287 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.281+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.287 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.281+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1986892326}) 2024-09-16T04:07:56.287 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.281+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=55093991}) 2024-09-16T04:07:56.287 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.281+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.287 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.281+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1986892326}) 2024-09-16T04:07:56.287 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.281+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=270996709}) 2024-09-16T04:07:56.289 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.285+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.289 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.285+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=608519757}) 2024-09-16T04:07:56.290 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.285+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2073285461}) 2024-09-16T04:07:56.290 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.285+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.290 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.285+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=608519757}) 2024-09-16T04:07:56.290 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.285+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2813872720}) 2024-09-16T04:07:56.291 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.285+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.292 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.285+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3556514543}) 2024-09-16T04:07:56.292 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.285+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4280173410}) 2024-09-16T04:07:56.292 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.285+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.292 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.285+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3556514543}) 2024-09-16T04:07:56.292 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.285+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3540535268}) 2024-09-16T04:07:56.294 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.289+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.294 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.289+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=167951857}) 2024-09-16T04:07:56.294 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.289+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3157062040}) 2024-09-16T04:07:56.294 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.289+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.294 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.289+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=167951857}) 2024-09-16T04:07:56.294 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.289+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2120048779}) 2024-09-16T04:07:56.296 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.289+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.296 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.289+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=435672196}) 2024-09-16T04:07:56.296 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.289+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=767333973}) 2024-09-16T04:07:56.296 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.289+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.296 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.289+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=435672196}) 2024-09-16T04:07:56.296 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.289+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1357054022}) 2024-09-16T04:07:56.303 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.293+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.303 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.293+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4107890841}) 2024-09-16T04:07:56.303 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.293+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1341705181}) 2024-09-16T04:07:56.303 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.293+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.303 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.293+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4107890841}) 2024-09-16T04:07:56.303 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.293+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2355875266}) 2024-09-16T04:07:56.303 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.293+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.303 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.293+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=412664370}) 2024-09-16T04:07:56.304 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.293+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4191394433}) 2024-09-16T04:07:56.304 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.293+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.304 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.293+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=412664370}) 2024-09-16T04:07:56.304 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.293+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1943747765}) 2024-09-16T04:07:56.304 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.297+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.304 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.297+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2458277060}) 2024-09-16T04:07:56.304 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.297+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=692441777}) 2024-09-16T04:07:56.304 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.297+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.304 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.297+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2458277060}) 2024-09-16T04:07:56.304 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.297+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3143830921}) 2024-09-16T04:07:56.305 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.297+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.305 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.301+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3938094218}) 2024-09-16T04:07:56.306 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.301+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=796080966}) 2024-09-16T04:07:56.306 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.301+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.306 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.301+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3938094218}) 2024-09-16T04:07:56.306 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.301+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1256535100}) 2024-09-16T04:07:56.308 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.301+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.308 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.301+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1973933084}) 2024-09-16T04:07:56.308 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.301+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1275872191}) 2024-09-16T04:07:56.308 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.301+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.308 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.301+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1973933084}) 2024-09-16T04:07:56.308 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.301+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2644803146}) 2024-09-16T04:07:56.310 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.305+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.310 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.305+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3809458777}) 2024-09-16T04:07:56.310 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.305+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3842003625}) 2024-09-16T04:07:56.310 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.305+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.310 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.305+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3809458777}) 2024-09-16T04:07:56.311 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.305+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3269342708}) 2024-09-16T04:07:56.313 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.305+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.313 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.305+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3798844719}) 2024-09-16T04:07:56.313 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.305+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1731820829}) 2024-09-16T04:07:56.313 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.305+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.313 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.305+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3798844719}) 2024-09-16T04:07:56.313 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.305+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1762707853}) 2024-09-16T04:07:56.315 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.309+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.315 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.309+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3371550571}) 2024-09-16T04:07:56.315 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.309+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4201618622}) 2024-09-16T04:07:56.315 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.309+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.315 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.309+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3371550571}) 2024-09-16T04:07:56.315 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.309+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3922779973}) 2024-09-16T04:07:56.317 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.309+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.317 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.309+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4063036143}) 2024-09-16T04:07:56.317 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.309+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=935785902}) 2024-09-16T04:07:56.317 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.309+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.317 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.309+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4063036143}) 2024-09-16T04:07:56.317 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.313+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4157886475}) 2024-09-16T04:07:56.320 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.313+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.320 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.313+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2508425973}) 2024-09-16T04:07:56.320 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.313+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=875672096}) 2024-09-16T04:07:56.320 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.313+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.320 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.313+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2508425973}) 2024-09-16T04:07:56.320 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.313+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3063360598}) 2024-09-16T04:07:56.322 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.317+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.322 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.317+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3891714476}) 2024-09-16T04:07:56.322 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.317+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1296634656}) 2024-09-16T04:07:56.322 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.317+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.322 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.317+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3891714476}) 2024-09-16T04:07:56.322 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.317+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2173830708}) 2024-09-16T04:07:56.324 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.317+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.324 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.317+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=225525503}) 2024-09-16T04:07:56.324 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.317+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1001914391}) 2024-09-16T04:07:56.325 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.317+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.325 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.317+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=225525503}) 2024-09-16T04:07:56.325 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.317+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3348262933}) 2024-09-16T04:07:56.327 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.321+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.327 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.321+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=83839803}) 2024-09-16T04:07:56.327 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.321+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2750557457}) 2024-09-16T04:07:56.327 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.321+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.327 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.321+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=83839803}) 2024-09-16T04:07:56.327 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.321+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1819807323}) 2024-09-16T04:07:56.329 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.321+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.329 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.321+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3564732251}) 2024-09-16T04:07:56.329 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.321+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3264997401}) 2024-09-16T04:07:56.329 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.321+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.329 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.321+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3564732251}) 2024-09-16T04:07:56.329 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.321+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1558784650}) 2024-09-16T04:07:56.331 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.325+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.331 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.325+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1043068958}) 2024-09-16T04:07:56.332 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.325+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=938916530}) 2024-09-16T04:07:56.332 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.325+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.332 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.325+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1043068958}) 2024-09-16T04:07:56.332 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.325+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4145520507}) 2024-09-16T04:07:56.334 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.329+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.334 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.329+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2195368563}) 2024-09-16T04:07:56.334 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.329+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3092430244}) 2024-09-16T04:07:56.334 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.329+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.334 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.329+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2195368563}) 2024-09-16T04:07:56.334 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.329+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2285999264}) 2024-09-16T04:07:56.336 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.329+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.336 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.329+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1371473699}) 2024-09-16T04:07:56.336 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.329+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1064533810}) 2024-09-16T04:07:56.337 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.329+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.337 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.329+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1371473699}) 2024-09-16T04:07:56.337 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.329+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2505523732}) 2024-09-16T04:07:56.338 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.333+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.338 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.333+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2770573919}) 2024-09-16T04:07:56.338 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.333+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3927252846}) 2024-09-16T04:07:56.339 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.333+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.339 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.333+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2770573919}) 2024-09-16T04:07:56.339 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.333+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=292417304}) 2024-09-16T04:07:56.341 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.333+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.341 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.333+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2506439835}) 2024-09-16T04:07:56.341 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.333+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1821085221}) 2024-09-16T04:07:56.341 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.333+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.341 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.333+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2506439835}) 2024-09-16T04:07:56.341 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.333+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2589439361}) 2024-09-16T04:07:56.343 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.337+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.343 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.337+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2962635318}) 2024-09-16T04:07:56.343 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.337+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3557545862}) 2024-09-16T04:07:56.343 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.337+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.343 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.337+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2962635318}) 2024-09-16T04:07:56.343 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.337+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1658461112}) 2024-09-16T04:07:56.345 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.337+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.345 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.337+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1121894716}) 2024-09-16T04:07:56.345 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.337+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2456077607}) 2024-09-16T04:07:56.345 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.337+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.345 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.337+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1121894716}) 2024-09-16T04:07:56.345 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.337+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1398012849}) 2024-09-16T04:07:56.347 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.341+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.347 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.341+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1486238132}) 2024-09-16T04:07:56.347 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.341+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1803150701}) 2024-09-16T04:07:56.347 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.341+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.347 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.341+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1486238132}) 2024-09-16T04:07:56.347 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.341+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3687565830}) 2024-09-16T04:07:56.349 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.341+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.349 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.341+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=994242787}) 2024-09-16T04:07:56.349 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.341+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1042050189}) 2024-09-16T04:07:56.349 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.341+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.349 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.341+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=994242787}) 2024-09-16T04:07:56.349 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.341+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2261055985}) 2024-09-16T04:07:56.351 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.345+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.351 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.345+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3859127301}) 2024-09-16T04:07:56.351 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.345+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1993892379}) 2024-09-16T04:07:56.351 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.345+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.351 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.345+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3859127301}) 2024-09-16T04:07:56.351 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.345+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2979010893}) 2024-09-16T04:07:56.353 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.345+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.353 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.345+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1908982558}) 2024-09-16T04:07:56.353 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.345+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=578766284}) 2024-09-16T04:07:56.353 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.345+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.353 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.345+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1908982558}) 2024-09-16T04:07:56.353 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.345+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1986711238}) 2024-09-16T04:07:56.355 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.349+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.355 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.349+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2551828999}) 2024-09-16T04:07:56.355 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.349+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=294025925}) 2024-09-16T04:07:56.355 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.349+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.355 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.349+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2551828999}) 2024-09-16T04:07:56.355 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.349+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4019973055}) 2024-09-16T04:07:56.357 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.349+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.357 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.349+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=740370382}) 2024-09-16T04:07:56.357 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.349+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2777459336}) 2024-09-16T04:07:56.357 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.349+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.357 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.349+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=740370382}) 2024-09-16T04:07:56.357 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.349+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1660733402}) 2024-09-16T04:07:56.359 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.353+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.359 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.353+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3374281215}) 2024-09-16T04:07:56.359 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.353+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1694949826}) 2024-09-16T04:07:56.359 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.353+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.359 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.353+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3374281215}) 2024-09-16T04:07:56.359 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.353+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2031083685}) 2024-09-16T04:07:56.361 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.353+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.361 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.353+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1434759901}) 2024-09-16T04:07:56.361 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.353+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=646705172}) 2024-09-16T04:07:56.361 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.353+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.361 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.353+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1434759901}) 2024-09-16T04:07:56.361 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.353+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=165432295}) 2024-09-16T04:07:56.363 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.357+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.363 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.357+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1709057518}) 2024-09-16T04:07:56.363 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.357+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3975151994}) 2024-09-16T04:07:56.363 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.357+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.363 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.357+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1709057518}) 2024-09-16T04:07:56.363 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.357+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1411060655}) 2024-09-16T04:07:56.365 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.357+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.365 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.357+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1385542935}) 2024-09-16T04:07:56.365 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.357+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=101195476}) 2024-09-16T04:07:56.365 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.357+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.365 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.357+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1385542935}) 2024-09-16T04:07:56.365 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.357+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4153722424}) 2024-09-16T04:07:56.367 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.361+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.367 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.361+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1374341363}) 2024-09-16T04:07:56.367 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.361+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4164059645}) 2024-09-16T04:07:56.367 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.361+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.367 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.361+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1374341363}) 2024-09-16T04:07:56.367 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.361+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2831929342}) 2024-09-16T04:07:56.369 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.361+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.369 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.361+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=680105848}) 2024-09-16T04:07:56.369 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.361+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=58110666}) 2024-09-16T04:07:56.369 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.361+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.369 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.361+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=680105848}) 2024-09-16T04:07:56.370 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.361+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3718799021}) 2024-09-16T04:07:56.371 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.365+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.371 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.365+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3426036552}) 2024-09-16T04:07:56.371 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.365+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3530493167}) 2024-09-16T04:07:56.371 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.365+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.372 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.365+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3426036552}) 2024-09-16T04:07:56.372 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.365+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4024101977}) 2024-09-16T04:07:56.373 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.365+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.374 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.365+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1800622540}) 2024-09-16T04:07:56.374 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.365+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=201559049}) 2024-09-16T04:07:56.374 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.369+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.374 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.369+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1800622540}) 2024-09-16T04:07:56.374 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.369+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4260499962}) 2024-09-16T04:07:56.375 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.369+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.375 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.369+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3298821516}) 2024-09-16T04:07:56.375 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.369+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1637610708}) 2024-09-16T04:07:56.376 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.369+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.376 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.369+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3298821516}) 2024-09-16T04:07:56.376 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.369+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=877385035}) 2024-09-16T04:07:56.377 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.369+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.377 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.369+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3520068390}) 2024-09-16T04:07:56.377 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.373+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1812799947}) 2024-09-16T04:07:56.378 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.373+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.378 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.373+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3520068390}) 2024-09-16T04:07:56.378 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.373+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=569081691}) 2024-09-16T04:07:56.379 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.373+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.379 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.373+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=312842018}) 2024-09-16T04:07:56.379 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.373+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2709322584}) 2024-09-16T04:07:56.380 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.373+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.380 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.373+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=312842018}) 2024-09-16T04:07:56.380 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.373+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2992870144}) 2024-09-16T04:07:56.381 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.373+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.381 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.373+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3946823019}) 2024-09-16T04:07:56.382 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.377+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3701793148}) 2024-09-16T04:07:56.382 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.377+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.382 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.377+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3946823019}) 2024-09-16T04:07:56.382 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.377+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1922966836}) 2024-09-16T04:07:56.383 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.377+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.383 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.377+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=618621265}) 2024-09-16T04:07:56.384 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.377+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2052665130}) 2024-09-16T04:07:56.384 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.377+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.384 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.377+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=618621265}) 2024-09-16T04:07:56.384 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.377+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=854166667}) 2024-09-16T04:07:56.385 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.381+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.385 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.381+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=458857509}) 2024-09-16T04:07:56.385 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.381+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=741282482}) 2024-09-16T04:07:56.385 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.381+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.386 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.381+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=458857509}) 2024-09-16T04:07:56.386 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.381+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4022095297}) 2024-09-16T04:07:56.387 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.381+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.388 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.381+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3243166612}) 2024-09-16T04:07:56.388 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.381+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2567521412}) 2024-09-16T04:07:56.388 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.381+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.388 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.381+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3243166612}) 2024-09-16T04:07:56.388 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.381+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2831538976}) 2024-09-16T04:07:56.390 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.385+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.390 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.385+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3296568260}) 2024-09-16T04:07:56.390 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.385+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1798980144}) 2024-09-16T04:07:56.390 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.385+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.390 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.385+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3296568260}) 2024-09-16T04:07:56.390 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.385+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2286725962}) 2024-09-16T04:07:56.393 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.385+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.393 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.385+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2904285106}) 2024-09-16T04:07:56.393 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.385+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3408628134}) 2024-09-16T04:07:56.393 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.389+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.393 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.389+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2904285106}) 2024-09-16T04:07:56.394 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.389+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3941731856}) 2024-09-16T04:07:56.403 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.397+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.403 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.397+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1534941463}) 2024-09-16T04:07:56.404 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.397+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=931833395}) 2024-09-16T04:07:56.404 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.397+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.404 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.397+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1534941463}) 2024-09-16T04:07:56.404 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.397+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3346570911}) 2024-09-16T04:07:56.407 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.401+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.407 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.401+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1118281340}) 2024-09-16T04:07:56.407 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.401+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2106383174}) 2024-09-16T04:07:56.408 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.401+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.408 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.401+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1118281340}) 2024-09-16T04:07:56.408 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.401+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2093659981}) 2024-09-16T04:07:56.442 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.437+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.442 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.437+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4043399243}) 2024-09-16T04:07:56.442 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.437+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4289982814}) 2024-09-16T04:07:56.442 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.437+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.442 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.437+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4043399243}) 2024-09-16T04:07:56.442 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.437+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1060465720}) 2024-09-16T04:07:56.444 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.437+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.444 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.437+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2789724556}) 2024-09-16T04:07:56.444 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.437+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4205039915}) 2024-09-16T04:07:56.444 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.437+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.444 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.437+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2789724556}) 2024-09-16T04:07:56.444 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.437+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3781873185}) 2024-09-16T04:07:56.446 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.441+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.446 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.441+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2485928131}) 2024-09-16T04:07:56.446 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.441+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2200173648}) 2024-09-16T04:07:56.446 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.441+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.446 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.441+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2485928131}) 2024-09-16T04:07:56.446 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.441+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2343872032}) 2024-09-16T04:07:56.448 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.441+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.448 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.441+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3501638515}) 2024-09-16T04:07:56.448 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.441+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1091175443}) 2024-09-16T04:07:56.448 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.441+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.448 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.441+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3501638515}) 2024-09-16T04:07:56.448 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.441+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2765346126}) 2024-09-16T04:07:56.450 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.445+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.450 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.445+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2509244682}) 2024-09-16T04:07:56.450 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.445+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3898038420}) 2024-09-16T04:07:56.450 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.445+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.450 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.445+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2509244682}) 2024-09-16T04:07:56.450 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.445+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=460234291}) 2024-09-16T04:07:56.452 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.445+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.452 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.445+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3605968298}) 2024-09-16T04:07:56.452 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.445+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=461945041}) 2024-09-16T04:07:56.452 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.445+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.452 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.445+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3605968298}) 2024-09-16T04:07:56.452 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.445+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=78989710}) 2024-09-16T04:07:56.454 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.449+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.454 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.449+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3949817500}) 2024-09-16T04:07:56.454 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.449+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2742047350}) 2024-09-16T04:07:56.454 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.449+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.455 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.449+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3949817500}) 2024-09-16T04:07:56.455 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.449+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3626254736}) 2024-09-16T04:07:56.457 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.449+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.457 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.449+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3551007927}) 2024-09-16T04:07:56.457 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.449+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=805821028}) 2024-09-16T04:07:56.457 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.449+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.457 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.449+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3551007927}) 2024-09-16T04:07:56.457 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.449+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=201398998}) 2024-09-16T04:07:56.458 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.453+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.458 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.453+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2273206418}) 2024-09-16T04:07:56.458 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.453+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2503904698}) 2024-09-16T04:07:56.458 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.453+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.458 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.453+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2273206418}) 2024-09-16T04:07:56.458 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.453+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3515689846}) 2024-09-16T04:07:56.460 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.453+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.460 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.453+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1368749319}) 2024-09-16T04:07:56.460 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.453+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1938422259}) 2024-09-16T04:07:56.460 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.453+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.460 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.453+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1368749319}) 2024-09-16T04:07:56.460 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.453+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3939101776}) 2024-09-16T04:07:56.462 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.457+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.462 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.457+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1487049789}) 2024-09-16T04:07:56.463 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.457+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=908261960}) 2024-09-16T04:07:56.463 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.457+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.463 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.457+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1487049789}) 2024-09-16T04:07:56.463 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.457+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3605778465}) 2024-09-16T04:07:56.464 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.457+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.464 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.457+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=776552431}) 2024-09-16T04:07:56.465 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.457+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2480221035}) 2024-09-16T04:07:56.465 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.457+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.465 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.457+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=776552431}) 2024-09-16T04:07:56.465 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.457+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3326173292}) 2024-09-16T04:07:56.467 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.461+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.467 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.461+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4188535047}) 2024-09-16T04:07:56.467 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.461+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=433422208}) 2024-09-16T04:07:56.467 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.461+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.467 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.461+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4188535047}) 2024-09-16T04:07:56.467 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.461+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3769076239}) 2024-09-16T04:07:56.468 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.461+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.468 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.461+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2885258214}) 2024-09-16T04:07:56.468 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.461+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4090946607}) 2024-09-16T04:07:56.468 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.461+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.468 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.461+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2885258214}) 2024-09-16T04:07:56.468 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.461+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=375712311}) 2024-09-16T04:07:56.470 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.465+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.470 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.465+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1565725423}) 2024-09-16T04:07:56.470 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.465+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=212749840}) 2024-09-16T04:07:56.470 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.465+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.470 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.465+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1565725423}) 2024-09-16T04:07:56.470 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.465+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1880708970}) 2024-09-16T04:07:56.472 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.465+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.472 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.465+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4080720665}) 2024-09-16T04:07:56.472 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.465+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=289533473}) 2024-09-16T04:07:56.472 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.465+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.473 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.465+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4080720665}) 2024-09-16T04:07:56.473 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.465+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1509757256}) 2024-09-16T04:07:56.474 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.469+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.474 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.469+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1839010556}) 2024-09-16T04:07:56.474 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.469+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1628114206}) 2024-09-16T04:07:56.474 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.469+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.474 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.469+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1839010556}) 2024-09-16T04:07:56.474 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.469+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3597970233}) 2024-09-16T04:07:56.476 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.469+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.476 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.469+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3957067137}) 2024-09-16T04:07:56.476 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.469+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2382865588}) 2024-09-16T04:07:56.476 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.469+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.476 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.469+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3957067137}) 2024-09-16T04:07:56.476 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.469+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3337110633}) 2024-09-16T04:07:56.478 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.473+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.478 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.473+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3598585309}) 2024-09-16T04:07:56.478 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.473+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=987907590}) 2024-09-16T04:07:56.478 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.473+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.478 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.473+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3598585309}) 2024-09-16T04:07:56.478 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.473+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3850440409}) 2024-09-16T04:07:56.480 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.473+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.480 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.473+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=422124461}) 2024-09-16T04:07:56.480 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.473+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4193127101}) 2024-09-16T04:07:56.480 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.473+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.480 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.473+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=422124461}) 2024-09-16T04:07:56.480 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.473+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1327648151}) 2024-09-16T04:07:56.482 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.477+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.482 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.477+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3624773603}) 2024-09-16T04:07:56.482 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.477+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1190546477}) 2024-09-16T04:07:56.482 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.477+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.482 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.477+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3624773603}) 2024-09-16T04:07:56.482 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.477+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1264554739}) 2024-09-16T04:07:56.484 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.477+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.484 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.477+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2668359003}) 2024-09-16T04:07:56.484 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.477+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4099272773}) 2024-09-16T04:07:56.484 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.477+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.484 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.477+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2668359003}) 2024-09-16T04:07:56.484 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.477+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=140986968}) 2024-09-16T04:07:56.485 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.481+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.486 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.481+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=573248551}) 2024-09-16T04:07:56.486 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.481+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4194234370}) 2024-09-16T04:07:56.486 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.481+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.486 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.481+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=573248551}) 2024-09-16T04:07:56.486 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.481+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=964457091}) 2024-09-16T04:07:56.488 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.481+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.488 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.481+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=925683813}) 2024-09-16T04:07:56.488 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.481+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=383084610}) 2024-09-16T04:07:56.488 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.481+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.488 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.481+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=925683813}) 2024-09-16T04:07:56.488 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.481+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2466946488}) 2024-09-16T04:07:56.489 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.485+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.489 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.485+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1757323834}) 2024-09-16T04:07:56.490 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.485+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2159255065}) 2024-09-16T04:07:56.490 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.485+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.490 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.485+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1757323834}) 2024-09-16T04:07:56.490 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.485+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=614771285}) 2024-09-16T04:07:56.491 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.485+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.491 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.485+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=870593314}) 2024-09-16T04:07:56.491 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.485+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1603816376}) 2024-09-16T04:07:56.491 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.485+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.492 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.485+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=870593314}) 2024-09-16T04:07:56.492 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.485+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1408952101}) 2024-09-16T04:07:56.493 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.489+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.493 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.489+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3066253763}) 2024-09-16T04:07:56.493 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.489+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2069009658}) 2024-09-16T04:07:56.493 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.489+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.493 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.489+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3066253763}) 2024-09-16T04:07:56.494 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.489+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2232079690}) 2024-09-16T04:07:56.495 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.489+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.495 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.489+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2177920945}) 2024-09-16T04:07:56.495 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.489+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=558417837}) 2024-09-16T04:07:56.495 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.489+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.495 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.489+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2177920945}) 2024-09-16T04:07:56.495 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.489+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2663848570}) 2024-09-16T04:07:56.497 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.489+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.497 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.489+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3821225774}) 2024-09-16T04:07:56.497 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.489+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4074611345}) 2024-09-16T04:07:56.497 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.489+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.497 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.489+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3821225774}) 2024-09-16T04:07:56.497 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.489+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1175915989}) 2024-09-16T04:07:56.499 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.493+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.499 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.493+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2067906719}) 2024-09-16T04:07:56.499 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.493+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4070158066}) 2024-09-16T04:07:56.499 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.493+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.499 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.493+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2067906719}) 2024-09-16T04:07:56.499 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.493+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1893595741}) 2024-09-16T04:07:56.500 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.493+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.500 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.493+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2266882460}) 2024-09-16T04:07:56.501 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.493+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4158406944}) 2024-09-16T04:07:56.501 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.493+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.501 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.493+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2266882460}) 2024-09-16T04:07:56.501 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.493+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3791780228}) 2024-09-16T04:07:56.502 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.497+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.502 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.497+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2469959964}) 2024-09-16T04:07:56.502 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.497+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2306451745}) 2024-09-16T04:07:56.502 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.497+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.502 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.497+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2469959964}) 2024-09-16T04:07:56.502 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.497+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1533713080}) 2024-09-16T04:07:56.504 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.497+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.504 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.497+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1298513954}) 2024-09-16T04:07:56.504 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.497+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2808075479}) 2024-09-16T04:07:56.504 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.497+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.504 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.497+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1298513954}) 2024-09-16T04:07:56.505 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.497+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=791335903}) 2024-09-16T04:07:56.506 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.501+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.506 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.501+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3097340717}) 2024-09-16T04:07:56.506 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.501+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2099667190}) 2024-09-16T04:07:56.506 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.501+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.506 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.501+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3097340717}) 2024-09-16T04:07:56.506 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.501+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3132976696}) 2024-09-16T04:07:56.508 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.501+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.508 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.501+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1626512385}) 2024-09-16T04:07:56.508 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.501+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3916406368}) 2024-09-16T04:07:56.508 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.501+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.508 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.501+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1626512385}) 2024-09-16T04:07:56.508 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.501+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2280048419}) 2024-09-16T04:07:56.510 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.505+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.510 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.505+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3646173762}) 2024-09-16T04:07:56.510 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.505+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2278347414}) 2024-09-16T04:07:56.510 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.505+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.510 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.505+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3646173762}) 2024-09-16T04:07:56.510 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.505+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3618981258}) 2024-09-16T04:07:56.511 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.505+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.512 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.505+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4000987890}) 2024-09-16T04:07:56.512 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.505+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3140436057}) 2024-09-16T04:07:56.512 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.505+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.512 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.505+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4000987890}) 2024-09-16T04:07:56.512 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.505+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3836803319}) 2024-09-16T04:07:56.513 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.509+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.513 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.509+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=19478465}) 2024-09-16T04:07:56.513 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.509+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2727605548}) 2024-09-16T04:07:56.514 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.509+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.514 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.509+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=19478465}) 2024-09-16T04:07:56.514 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.509+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3608993339}) 2024-09-16T04:07:56.515 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.509+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.515 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.509+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3418097447}) 2024-09-16T04:07:56.515 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.509+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2336538042}) 2024-09-16T04:07:56.515 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.509+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.515 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.509+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3418097447}) 2024-09-16T04:07:56.515 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.509+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4156686919}) 2024-09-16T04:07:56.517 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.509+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.517 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.509+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3344095204}) 2024-09-16T04:07:56.517 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.509+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2664726568}) 2024-09-16T04:07:56.517 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.509+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.517 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.509+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3344095204}) 2024-09-16T04:07:56.517 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.509+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3984825924}) 2024-09-16T04:07:56.519 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.513+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.519 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.513+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1912727844}) 2024-09-16T04:07:56.519 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.513+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1501066419}) 2024-09-16T04:07:56.519 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.513+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.519 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.513+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1912727844}) 2024-09-16T04:07:56.519 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.513+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1475542726}) 2024-09-16T04:07:56.520 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.513+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.520 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.513+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=644616719}) 2024-09-16T04:07:56.520 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.513+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4259620622}) 2024-09-16T04:07:56.520 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.513+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.521 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.513+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=644616719}) 2024-09-16T04:07:56.521 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.513+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1399742940}) 2024-09-16T04:07:56.522 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.517+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.522 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.517+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1256403210}) 2024-09-16T04:07:56.522 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.517+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=542968033}) 2024-09-16T04:07:56.522 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.517+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.522 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.517+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1256403210}) 2024-09-16T04:07:56.522 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.517+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2783598286}) 2024-09-16T04:07:56.524 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.517+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.524 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.517+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1186200807}) 2024-09-16T04:07:56.524 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.517+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2401985186}) 2024-09-16T04:07:56.524 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.517+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.524 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.517+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1186200807}) 2024-09-16T04:07:56.524 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.517+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1638777231}) 2024-09-16T04:07:56.526 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.521+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.526 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.521+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=157126506}) 2024-09-16T04:07:56.526 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.521+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1280962072}) 2024-09-16T04:07:56.526 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.521+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.526 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.521+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=157126506}) 2024-09-16T04:07:56.526 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.521+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=840388828}) 2024-09-16T04:07:56.527 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.521+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.527 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.521+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=492368713}) 2024-09-16T04:07:56.528 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.521+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2799542791}) 2024-09-16T04:07:56.528 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.521+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.528 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.521+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=492368713}) 2024-09-16T04:07:56.528 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.521+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=126872372}) 2024-09-16T04:07:56.529 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.525+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.529 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.525+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2563014062}) 2024-09-16T04:07:56.530 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.525+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2071037210}) 2024-09-16T04:07:56.530 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.525+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.530 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.525+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2563014062}) 2024-09-16T04:07:56.530 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.525+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=213314858}) 2024-09-16T04:07:56.531 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.525+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.531 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.525+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3669277737}) 2024-09-16T04:07:56.531 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.525+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1968099025}) 2024-09-16T04:07:56.531 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.525+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.531 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.525+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3669277737}) 2024-09-16T04:07:56.532 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.525+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3526718301}) 2024-09-16T04:07:56.533 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.525+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.533 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.525+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3362156444}) 2024-09-16T04:07:56.533 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.525+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3258608916}) 2024-09-16T04:07:56.533 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.525+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.533 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.529+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3362156444}) 2024-09-16T04:07:56.533 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.529+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2721309512}) 2024-09-16T04:07:56.535 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.529+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.535 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.529+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1732263284}) 2024-09-16T04:07:56.535 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.529+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1079209100}) 2024-09-16T04:07:56.535 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.529+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.535 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.529+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1732263284}) 2024-09-16T04:07:56.535 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.529+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4200763339}) 2024-09-16T04:07:56.537 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.529+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.537 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.529+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1971551322}) 2024-09-16T04:07:56.537 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.529+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=166440161}) 2024-09-16T04:07:56.537 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.529+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.537 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.529+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1971551322}) 2024-09-16T04:07:56.537 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.529+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1871849611}) 2024-09-16T04:07:56.538 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.533+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.539 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.533+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1975053791}) 2024-09-16T04:07:56.539 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.533+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1551445706}) 2024-09-16T04:07:56.539 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.533+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.539 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.533+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1975053791}) 2024-09-16T04:07:56.539 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.533+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3949497240}) 2024-09-16T04:07:56.540 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.533+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.540 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.533+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=498225665}) 2024-09-16T04:07:56.540 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.533+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=612275206}) 2024-09-16T04:07:56.540 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.533+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.541 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.533+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=498225665}) 2024-09-16T04:07:56.541 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.533+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2489558192}) 2024-09-16T04:07:56.542 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.537+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.542 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.537+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3260563364}) 2024-09-16T04:07:56.542 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.537+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2071908140}) 2024-09-16T04:07:56.542 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.537+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.542 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.537+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3260563364}) 2024-09-16T04:07:56.542 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.537+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=584197242}) 2024-09-16T04:07:56.544 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.537+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.544 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.537+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4091990535}) 2024-09-16T04:07:56.544 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.537+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2286498177}) 2024-09-16T04:07:56.544 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.537+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.544 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.537+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4091990535}) 2024-09-16T04:07:56.545 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.537+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3946532395}) 2024-09-16T04:07:56.546 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.541+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.546 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.541+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1759048455}) 2024-09-16T04:07:56.546 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.541+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=132167149}) 2024-09-16T04:07:56.546 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.541+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.546 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.541+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1759048455}) 2024-09-16T04:07:56.546 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.541+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1100669406}) 2024-09-16T04:07:56.548 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.541+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.548 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.541+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=728156833}) 2024-09-16T04:07:56.548 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.541+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=719947494}) 2024-09-16T04:07:56.548 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.541+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.548 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.541+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=728156833}) 2024-09-16T04:07:56.548 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.541+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=841572958}) 2024-09-16T04:07:56.549 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.541+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.549 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.541+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3155604154}) 2024-09-16T04:07:56.549 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.541+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=315809991}) 2024-09-16T04:07:56.549 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.545+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.549 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.545+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3155604154}) 2024-09-16T04:07:56.550 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.545+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3052039500}) 2024-09-16T04:07:56.550 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.545+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.550 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.545+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2849576058}) 2024-09-16T04:07:56.550 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.545+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3926340131}) 2024-09-16T04:07:56.550 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.545+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.550 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.545+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2849576058}) 2024-09-16T04:07:56.551 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.545+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2183134227}) 2024-09-16T04:07:56.551 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.545+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.552 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.545+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=951996914}) 2024-09-16T04:07:56.552 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.545+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3150581414}) 2024-09-16T04:07:56.552 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.545+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.552 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.545+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=951996914}) 2024-09-16T04:07:56.552 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.545+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1469927477}) 2024-09-16T04:07:56.553 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.545+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.553 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.545+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2747225244}) 2024-09-16T04:07:56.553 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.545+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2976754632}) 2024-09-16T04:07:56.553 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.545+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.553 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.545+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2747225244}) 2024-09-16T04:07:56.553 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.545+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1770648222}) 2024-09-16T04:07:56.555 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.549+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.555 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.549+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=72,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=5} crc {logm=581701278,mdsmap=4088524775,mgr=460109776,mgr_command_descs=2107673600,mgr_metadata=1367272202,mgrstat=2427402747,mon_config_key=2891860318}) 2024-09-16T04:07:56.555 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.549+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=72,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=6} crc {logm=3340439757,mdsmap=4088524775,mgr=911650295,mgr_command_descs=2107673600,mgr_metadata=1367272202,mgrstat=2596357327,mon_config_key=470184650}) 2024-09-16T04:07:56.555 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.549+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.555 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.549+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=72,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=5} crc {logm=581701278,mdsmap=4088524775,mgr=460109776,mgr_command_descs=2107673600,mgr_metadata=1367272202,mgrstat=2427402747,mon_config_key=2891860318}) 2024-09-16T04:07:56.555 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.549+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=65,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=7,monmap=3,osd_metadata=2} crc {logm=2184406314,mdsmap=687171726,mgr=460109776,mgr_command_descs=2107673601,mgr_metadata=1367272202,mgrstat=2929524794,mon_config_key=637488965,monmap=3937513981,osd_metadata=4237927022}) 2024-09-16T04:07:56.557 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.549+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.557 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.553+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mon_config_key=6,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=43,osdmap=41} crc {mon_config_key=802391556,monmap=3937513981,osd_metadata=4182959540,osd_pg_creating=783066558,osd_snap=131582157,osdmap=853470104}) 2024-09-16T04:07:56.558 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.553+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mon_config_key=6,monmap=3,osd_metadata=6,osd_snap=42,osdmap=43} crc {mon_config_key=802391556,monmap=3937513981,osd_metadata=4182959540,osd_snap=638101952,osdmap=2540906474}) 2024-09-16T04:07:56.558 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.553+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.558 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.553+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mon_config_key=6,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=43,osdmap=41} crc {mon_config_key=802391556,monmap=3937513981,osd_metadata=4182959540,osd_pg_creating=783066558,osd_snap=131582157,osdmap=853470104}) 2024-09-16T04:07:56.558 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.553+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mon_config_key=6,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=43,osdmap=42} crc {mon_config_key=802391556,monmap=3937513981,osd_metadata=191738038,osd_pg_creating=783066558,osd_snap=823573645,osdmap=3185820214}) 2024-09-16T04:07:56.559 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.553+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.559 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.553+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3755121086}) 2024-09-16T04:07:56.560 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.553+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=492142691}) 2024-09-16T04:07:56.560 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.553+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.560 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.553+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3755121086}) 2024-09-16T04:07:56.560 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.553+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1665889929}) 2024-09-16T04:07:56.562 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.557+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.562 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.557+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3590048100}) 2024-09-16T04:07:56.562 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.557+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=624461177}) 2024-09-16T04:07:56.563 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.557+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.563 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.557+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3590048100}) 2024-09-16T04:07:56.563 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.557+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=52351064}) 2024-09-16T04:07:56.568 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.561+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.568 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.561+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1645929925}) 2024-09-16T04:07:56.568 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.561+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3704332744}) 2024-09-16T04:07:56.568 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.561+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.568 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.561+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1645929925}) 2024-09-16T04:07:56.568 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.561+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3727652509}) 2024-09-16T04:07:56.580 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.573+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.581 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.573+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=41} crc {osdmap=1651101678}) 2024-09-16T04:07:56.581 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.573+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=42} crc {osdmap=1672580502}) 2024-09-16T04:07:56.581 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.573+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:07:56.581 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.573+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=41} crc {osdmap=1651101678}) 2024-09-16T04:07:56.581 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:07:56.573+0000 7f45ae632640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=40} crc {osdmap=94693171}) 2024-09-16T04:07:57.246 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T04:07:57.247 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:07:57.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.373+0000 7f240f71d640 10 monclient: get_monmap_and_config 2024-09-16T04:07:57.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.373+0000 7f240f71d640 10 monclient: build_initial_monmap 2024-09-16T04:07:57.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.373+0000 7f240f71d640 10 monclient: monmap: 2024-09-16T04:07:57.379 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:07:57.379 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:57.379 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:07:57.377829+0000 2024-09-16T04:07:57.379 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:07:57.377829+0000 2024-09-16T04:07:57.380 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:07:57.380 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:57.380 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:07:57.380 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:07:57.380 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:07:57.380 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:07:57.380 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:07:57.380 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:57.382 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.373+0000 7f240f71d640 1 Processor -- start 2024-09-16T04:07:57.382 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.373+0000 7f240f71d640 1 -- start start 2024-09-16T04:07:57.382 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.373+0000 7f240f71d640 10 monclient: init 2024-09-16T04:07:57.382 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.373+0000 7f240f71d640 10 monclient: _reopen_session rank -1 2024-09-16T04:07:57.382 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.373+0000 7f240f71d640 10 monclient: _add_conns ranks=[1,2,3,0,4] 2024-09-16T04:07:57.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.373+0000 7f240f71d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f240806db90 addr v1:172.21.15.154:6789/0 2024-09-16T04:07:57.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.373+0000 7f240f71d640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f24081340c0 0x7f240813c190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:57.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.373+0000 7f240f71d640 10 monclient(hunting): picked mon.noname-b con 0x7f24081340c0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:07:57.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.373+0000 7f240f71d640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f240813c6d0 0x7f240813eac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:57.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.373+0000 7f240f71d640 10 monclient(hunting): picked mon.noname-a con 0x7f240813c6d0 addr v2:172.21.15.42:3300/0 2024-09-16T04:07:57.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.373+0000 7f240f71d640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f240806ece0 con 0x7f240806db90 2024-09-16T04:07:57.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.373+0000 7f240f71d640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:57.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.373+0000 7f240f71d640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f240806df70 con 0x7f240813c6d0 2024-09-16T04:07:57.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.373+0000 7f240f71d640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:57.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.373+0000 7f240f71d640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f240809bf80 con 0x7f24081340c0 2024-09-16T04:07:57.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.373+0000 7f240f71d640 10 monclient(hunting): _renew_subs 2024-09-16T04:07:57.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.373+0000 7f240f71d640 10 monclient(hunting): authenticate will time out at 2524.347472s 2024-09-16T04:07:57.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.373+0000 7f240ccb8640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f24081340c0 0x7f240813c190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:57.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.373+0000 7f240dcba640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f240813c6d0 0x7f240813eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:57.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.373+0000 7f240dcba640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f240813c6d0 0x7f240813eac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:43518/0 (socket says 172.21.15.42:43518) 2024-09-16T04:07:57.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.373+0000 7f240ccb8640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f24081340c0 0x7f240813c190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:37224/0 (socket says 172.21.15.42:37224) 2024-09-16T04:07:57.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.373+0000 7f240dcba640 1 -- 172.21.15.42:0/110541554 learned_addr learned my addr 172.21.15.42:0/110541554 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:07:57.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.373+0000 7f240dcba640 10 monclient(hunting): get_auth_request con 0x7f240813c6d0 auth_method 0 2024-09-16T04:07:57.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.373+0000 7f240dcba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:57.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.373+0000 7f240dcba640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:57.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.373+0000 7f240dcba640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:57.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.373+0000 7f240ccb8640 10 monclient(hunting): get_auth_request con 0x7f24081340c0 auth_method 0 2024-09-16T04:07:57.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.373+0000 7f240ccb8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:57.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.373+0000 7f240ccb8640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:57.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.373+0000 7f240ccb8640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:57.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.373+0000 7f240dcba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:07:57.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.373+0000 7f240dcba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:07:57.385 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.373+0000 7f240dcba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:07:57.385 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.373+0000 7f240ccb8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:07:57.385 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.373+0000 7f240ccb8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:07:57.385 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.373+0000 7f240ccb8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:07:57.385 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.377+0000 7f240dcba640 10 monclient(hunting): handle_auth_done global_id 1024146 payload 274 2024-09-16T04:07:57.385 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.377+0000 7f240dcba640 1 -- 172.21.15.42:0/110541554 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f24081340c0 msgr2=0x7f240813c190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:07:57.385 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.377+0000 7f240dcba640 1 --2- 172.21.15.42:0/110541554 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f24081340c0 0x7f240813c190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:57.385 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.377+0000 7f240dcba640 1 -- 172.21.15.42:0/110541554 >> v1:172.21.15.154:6789/0 conn(0x7f240806db90 legacy=0x7f24080a37f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:07:57.385 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.377+0000 7f240dcba640 10 monclient: _finish_hunting 0 2024-09-16T04:07:57.385 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.377+0000 7f240dcba640 1 monclient: found mon.noname-a 2024-09-16T04:07:57.385 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.377+0000 7f240dcba640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:07:57.385 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.377+0000 7f240dcba640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:07:57.385 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.377+0000 7f240dcba640 1 -- 172.21.15.42:0/110541554 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24081344d0 con 0x7f240813c6d0 2024-09-16T04:07:57.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.377+0000 7f240dcba640 1 --2- 172.21.15.42:0/110541554 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f240813c6d0 0x7f240813eac0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f2400009870 tx=0x7f240001c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a69e33999e252ef8 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:07:57.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.377+0000 7f23fffff640 1 -- 172.21.15.42:0/110541554 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24000b1070 con 0x7f240813c6d0 2024-09-16T04:07:57.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.377+0000 7f23fffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:57.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.377+0000 7f23fffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:07:57.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.377+0000 7f23fffff640 10 monclient: dump: 2024-09-16T04:07:57.392 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:07:57.392 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:57.392 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:57.392 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:57.392 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:57.392 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:57.392 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:07:57.392 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:07:57.392 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:07:57.392 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:57.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.377+0000 7f23fffff640 10 monclient: _finish_auth 0 2024-09-16T04:07:57.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.377+0000 7f23fffff640 10 monclient: _check_auth_tickets 2024-09-16T04:07:57.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.377+0000 7f23fffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:07:57.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.377+0000 7f23fffff640 1 -- 172.21.15.42:0/110541554 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f24000ac600 con 0x7f240813c6d0 2024-09-16T04:07:57.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.377+0000 7f240f71d640 5 monclient: authenticate success, global_id 1024146 2024-09-16T04:07:57.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.377+0000 7f240f71d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:07:57.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.377+0000 7f23fffff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:07:57.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.377+0000 7f23fffff640 1 -- 172.21.15.42:0/110541554 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24000ac9d0 con 0x7f240813c6d0 2024-09-16T04:07:57.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.377+0000 7f23fffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:57.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.377+0000 7f23fffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:07:57.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.377+0000 7f23fffff640 10 monclient: dump: 2024-09-16T04:07:57.393 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:07:57.393 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:57.393 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:57.393 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:57.393 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:57.394 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:57.394 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:07:57.394 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:07:57.394 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:07:57.394 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:57.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.377+0000 7f240f71d640 10 monclient: get_monmap_and_config success 2024-09-16T04:07:57.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.377+0000 7f240f71d640 4 set_mon_vals no callback set 2024-09-16T04:07:57.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.377+0000 7f240f71d640 10 monclient: shutdown 2024-09-16T04:07:57.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.377+0000 7f240f71d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:07:57.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.377+0000 7f240f71d640 1 -- 172.21.15.42:0/110541554 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f240813c6d0 msgr2=0x7f240813eac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:57.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.377+0000 7f240f71d640 1 --2- 172.21.15.42:0/110541554 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f240813c6d0 0x7f240813eac0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f2400009870 tx=0x7f240001c3e0 comp rx=0 tx=0).stop 2024-09-16T04:07:57.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.377+0000 7f240f71d640 1 -- 172.21.15.42:0/110541554 shutdown_connections 2024-09-16T04:07:57.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.377+0000 7f240f71d640 1 --2- 172.21.15.42:0/110541554 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f240813c6d0 0x7f240813eac0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:57.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.377+0000 7f240f71d640 1 --2- 172.21.15.42:0/110541554 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f24081340c0 0x7f240813c190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:57.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.377+0000 7f240f71d640 1 -- 172.21.15.42:0/110541554 >> 172.21.15.42:0/110541554 conn(0x7f24080a9fb0 msgr2=0x7f24080aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:07:57.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.377+0000 7f240f71d640 1 -- 172.21.15.42:0/110541554 shutdown_connections 2024-09-16T04:07:57.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.377+0000 7f240f71d640 1 -- 172.21.15.42:0/110541554 wait complete. 2024-09-16T04:07:57.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.377+0000 7f240f71d640 10 monclient: build_initial_monmap 2024-09-16T04:07:57.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.377+0000 7f240f71d640 10 monclient: monmap: 2024-09-16T04:07:57.395 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:07:57.395 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:07:57.395 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:07:57.395 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:07:57.395 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:07:57.395 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:57.395 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:07:57.395 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:07:57.395 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:07:57.396 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:57.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.381+0000 7f240f71d640 1 librados: starting msgr at 2024-09-16T04:07:57.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.381+0000 7f240f71d640 1 librados: starting objecter 2024-09-16T04:07:57.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.381+0000 7f240f71d640 1 Processor -- start 2024-09-16T04:07:57.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.381+0000 7f240f71d640 1 -- start start 2024-09-16T04:07:57.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.381+0000 7f240f71d640 1 librados: setting wanted keys 2024-09-16T04:07:57.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.381+0000 7f240f71d640 1 librados: calling monclient init 2024-09-16T04:07:57.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.381+0000 7f240f71d640 10 monclient: init 2024-09-16T04:07:57.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.381+0000 7f240f71d640 10 monclient: _reopen_session rank -1 2024-09-16T04:07:57.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.381+0000 7f240f71d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:07:57.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.381+0000 7f240f71d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f240806db90 0x7f240812d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:57.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.381+0000 7f240f71d640 10 monclient(hunting): picked mon.noname-b con 0x7f240806db90 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:07:57.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.381+0000 7f240f71d640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f24081340c0 0x7f240812dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:57.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.381+0000 7f240f71d640 10 monclient(hunting): picked mon.noname-c con 0x7f24081340c0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:07:57.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.381+0000 7f240f71d640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f240813c6d0 0x7f24081cf0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:57.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.381+0000 7f240f71d640 10 monclient(hunting): picked mon.noname-a con 0x7f240813c6d0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:07:57.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.381+0000 7f240f71d640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:57.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.381+0000 7f240f71d640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f2408141820 con 0x7f240813c6d0 2024-09-16T04:07:57.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.381+0000 7f240f71d640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:57.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.381+0000 7f240f71d640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f24081419a0 con 0x7f240806db90 2024-09-16T04:07:57.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.381+0000 7f240f71d640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:57.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.381+0000 7f240f71d640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f2408141b20 con 0x7f24081340c0 2024-09-16T04:07:57.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.381+0000 7f240f71d640 10 monclient(hunting): _renew_subs 2024-09-16T04:07:57.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.381+0000 7f240f71d640 10 monclient(hunting): authenticate will time out at 2524.353080s 2024-09-16T04:07:57.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.381+0000 7f240ccb8640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f24081340c0 0x7f240812dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:57.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.381+0000 7f240dcba640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f240813c6d0 0x7f24081cf0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:57.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.381+0000 7f240dcba640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f240813c6d0 0x7f24081cf0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:43522/0 (socket says 172.21.15.42:43522) 2024-09-16T04:07:57.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.381+0000 7f240dcba640 1 -- 172.21.15.42:0/2131329708 learned_addr learned my addr 172.21.15.42:0/2131329708 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:07:57.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.381+0000 7f240dcba640 10 monclient(hunting): get_auth_request con 0x7f240813c6d0 auth_method 0 2024-09-16T04:07:57.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.381+0000 7f240dcba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:57.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.381+0000 7f240dcba640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:57.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.381+0000 7f240dcba640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:57.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.381+0000 7f240ccb8640 10 monclient(hunting): get_auth_request con 0x7f24081340c0 auth_method 0 2024-09-16T04:07:57.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.381+0000 7f240ccb8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:57.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.381+0000 7f240ccb8640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:57.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.381+0000 7f240ccb8640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:57.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.381+0000 7f240dcba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:07:57.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.381+0000 7f240dcba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:07:57.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.381+0000 7f240dcba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:07:57.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.381+0000 7f240ccb8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:07:57.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.381+0000 7f240ccb8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:07:57.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.381+0000 7f240ccb8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:07:57.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.381+0000 7f240d4b9640 1 --2- 172.21.15.42:0/2131329708 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f240806db90 0x7f240812d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:57.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.381+0000 7f240dcba640 10 monclient(hunting): handle_auth_done global_id 1024149 payload 824 2024-09-16T04:07:57.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.381+0000 7f240dcba640 1 -- 172.21.15.42:0/2131329708 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f24081340c0 msgr2=0x7f240812dbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:57.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.381+0000 7f240dcba640 1 --2- 172.21.15.42:0/2131329708 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f24081340c0 0x7f240812dbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:57.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.381+0000 7f240dcba640 1 -- 172.21.15.42:0/2131329708 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f240806db90 msgr2=0x7f240812d6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:57.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.381+0000 7f240dcba640 1 --2- 172.21.15.42:0/2131329708 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f240806db90 0x7f240812d6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:57.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.381+0000 7f240dcba640 10 monclient: _finish_hunting 0 2024-09-16T04:07:57.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.381+0000 7f240dcba640 1 monclient: found mon.noname-a 2024-09-16T04:07:57.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.381+0000 7f240dcba640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:07:57.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.381+0000 7f240dcba640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:07:57.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.381+0000 7f240dcba640 1 -- 172.21.15.42:0/2131329708 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f240813af80 con 0x7f240813c6d0 2024-09-16T04:07:57.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.381+0000 7f240d4b9640 10 monclient: get_auth_request con 0x7f240806db90 auth_method 0 2024-09-16T04:07:57.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.381+0000 7f240d4b9640 1 --2- 172.21.15.42:0/2131329708 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f240806db90 0x7f240812d6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T04:07:57.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.381+0000 7f240dcba640 1 --2- 172.21.15.42:0/2131329708 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f240813c6d0 0x7f24081cf0e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f240001aa90 tx=0x7f240001ac20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:07:57.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.381+0000 7f23fdffb640 10 client.?.objecter ms_handle_connect 0x7f240813c6d0 2024-09-16T04:07:57.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.381+0000 7f23fdffb640 10 client.?.objecter resend_mon_ops 2024-09-16T04:07:57.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.381+0000 7f23fdffb640 1 -- 172.21.15.42:0/2131329708 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24000b1040 con 0x7f240813c6d0 2024-09-16T04:07:57.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.381+0000 7f23fdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:57.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.381+0000 7f23fdffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:07:57.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.381+0000 7f23fdffb640 10 monclient: dump: 2024-09-16T04:07:57.400 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:07:57.400 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:57.400 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:57.400 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:57.400 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:57.400 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:57.401 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:07:57.401 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:07:57.401 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:07:57.401 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:57.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.381+0000 7f23fdffb640 10 monclient: _finish_auth 0 2024-09-16T04:07:57.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.381+0000 7f23fdffb640 10 monclient: _check_auth_tickets 2024-09-16T04:07:57.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.381+0000 7f23fdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:07:57.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.381+0000 7f23fdffb640 1 -- 172.21.15.42:0/2131329708 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f24000b7450 con 0x7f240813c6d0 2024-09-16T04:07:57.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.381+0000 7f23fdffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:07:57.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.381+0000 7f23fdffb640 1 -- 172.21.15.42:0/2131329708 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24000b7840 con 0x7f240813c6d0 2024-09-16T04:07:57.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.381+0000 7f23fdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:57.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.381+0000 7f23fdffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:07:57.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.381+0000 7f23fdffb640 10 monclient: dump: 2024-09-16T04:07:57.401 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:07:57.402 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:57.402 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:57.402 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:57.402 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:57.402 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:57.402 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:07:57.402 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:07:57.402 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:07:57.402 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:57.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.381+0000 7f23ff7fe640 4 set_mon_vals no callback set 2024-09-16T04:07:57.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.381+0000 7f240f71d640 5 monclient: authenticate success, global_id 1024149 2024-09-16T04:07:57.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.381+0000 7f240f71d640 10 monclient: _renew_subs 2024-09-16T04:07:57.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.381+0000 7f240f71d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:07:57.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.381+0000 7f240f71d640 1 -- 172.21.15.42:0/2131329708 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f240806f6a0 con 0x7f240813c6d0 2024-09-16T04:07:57.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.381+0000 7f240f71d640 10 client.1024149.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:07:57.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.381+0000 7f240f71d640 10 monclient: _renew_subs 2024-09-16T04:07:57.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.381+0000 7f240f71d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:07:57.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.381+0000 7f240f71d640 1 -- 172.21.15.42:0/2131329708 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24081377c0 con 0x7f240813c6d0 2024-09-16T04:07:57.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.381+0000 7f240f71d640 1 librados: init done 2024-09-16T04:07:57.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.381+0000 7f240f71d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:07:57.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.381+0000 7f240f71d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:07:57.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.381+0000 7f240f71d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:07:57.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.381+0000 7f240f71d640 1 -- 172.21.15.42:0/2131329708 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f24080a37f0 con 0x7f240813c6d0 2024-09-16T04:07:57.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.381+0000 7f23fdffb640 1 -- 172.21.15.42:0/2131329708 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f24000be020 con 0x7f240813c6d0 2024-09-16T04:07:57.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.381+0000 7f23fdffb640 1 --2- 172.21.15.42:0/2131329708 >> v2:172.21.15.154:6812/254845342 conn(0x7f23ec037810 0x7f23ec039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:57.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.381+0000 7f23fdffb640 1 -- 172.21.15.42:0/2131329708 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(706..706 src has 451..706) ==== 5040+0+0 (secure 0 0 0) 0x7f24000e29c0 con 0x7f240813c6d0 2024-09-16T04:07:57.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.381+0000 7f23fdffb640 10 client.1024149.objecter ms_dispatch 0x7f2408000ba0 osd_map(706..706 src has 451..706) 2024-09-16T04:07:57.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.381+0000 7f23fdffb640 3 client.1024149.objecter handle_osd_map got epochs [706,706] > 0 2024-09-16T04:07:57.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.381+0000 7f23fdffb640 3 client.1024149.objecter handle_osd_map decoding full epoch 706 2024-09-16T04:07:57.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.381+0000 7f23fdffb640 20 client.1024149.objecter dump_active .. 0 homeless 2024-09-16T04:07:57.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.385+0000 7f240d4b9640 1 --2- 172.21.15.42:0/2131329708 >> v2:172.21.15.154:6812/254845342 conn(0x7f23ec037810 0x7f23ec039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:57.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.385+0000 7f240d4b9640 10 monclient: get_auth_request con 0x7f23ec037810 auth_method 0 2024-09-16T04:07:57.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.385+0000 7f240d4b9640 1 --2- 172.21.15.42:0/2131329708 >> v2:172.21.15.154:6812/254845342 conn(0x7f23ec037810 0x7f23ec039cd0 secure :-1 s=READY pgs=3893 cs=0 l=1 rev1=1 crypto rx=0x7f2404007e90 tx=0x7f24040073a0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:07:57.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.385+0000 7f23fdffb640 10 client.1024149.objecter ms_handle_connect 0x7f23ec037810 2024-09-16T04:07:57.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.389+0000 7f23fdffb640 1 -- 172.21.15.42:0/2131329708 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f24000160f0 con 0x7f240813c6d0 2024-09-16T04:07:57.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.389+0000 7f23fdffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:07:57.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.389+0000 7f23fdffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:07:57.619 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.613+0000 7f240f71d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:07:57.619 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.613+0000 7f240f71d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:07:57.619 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.613+0000 7f240f71d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:07:57.619 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.613+0000 7f240f71d640 1 -- 172.21.15.42:0/2131329708 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f2408068e40 con 0x7f240813c6d0 2024-09-16T04:07:57.620 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.613+0000 7f23fdffb640 1 -- 172.21.15.42:0/2131329708 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7f24000ab560 con 0x7f240813c6d0 2024-09-16T04:07:57.620 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.613+0000 7f23fdffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:07:57.620 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.613+0000 7f23fdffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:07:57.620 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":238,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:07:57.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.617+0000 7f240f71d640 10 librados: watch_flush enter 2024-09-16T04:07:57.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.617+0000 7f240f71d640 10 librados: watch_flush exit 2024-09-16T04:07:57.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.617+0000 7f240f71d640 20 client.1024149.objecter shutdown clearing up homeless session... 2024-09-16T04:07:57.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.617+0000 7f240f71d640 10 client.1024149.objecter successfully canceled tick 2024-09-16T04:07:57.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.617+0000 7f240f71d640 1 -- 172.21.15.42:0/2131329708 >> v2:172.21.15.154:6812/254845342 conn(0x7f23ec037810 msgr2=0x7f23ec039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:57.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.617+0000 7f240f71d640 1 --2- 172.21.15.42:0/2131329708 >> v2:172.21.15.154:6812/254845342 conn(0x7f23ec037810 0x7f23ec039cd0 secure :-1 s=READY pgs=3893 cs=0 l=1 rev1=1 crypto rx=0x7f2404007e90 tx=0x7f24040073a0 comp rx=0 tx=0).stop 2024-09-16T04:07:57.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.617+0000 7f240f71d640 10 monclient: shutdown 2024-09-16T04:07:57.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.617+0000 7f240f71d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:07:57.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.617+0000 7f240f71d640 1 -- 172.21.15.42:0/2131329708 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f240813c6d0 msgr2=0x7f24081cf0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:57.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.617+0000 7f240f71d640 1 --2- 172.21.15.42:0/2131329708 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f240813c6d0 0x7f24081cf0e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f240001aa90 tx=0x7f240001ac20 comp rx=0 tx=0).stop 2024-09-16T04:07:57.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.617+0000 7f240f71d640 1 -- 172.21.15.42:0/2131329708 shutdown_connections 2024-09-16T04:07:57.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.617+0000 7f240f71d640 1 --2- 172.21.15.42:0/2131329708 >> v2:172.21.15.154:6812/254845342 conn(0x7f23ec037810 0x7f23ec039cd0 unknown :-1 s=CLOSED pgs=3893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:57.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.617+0000 7f240f71d640 1 --2- 172.21.15.42:0/2131329708 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f240813c6d0 0x7f24081cf0e0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:57.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.617+0000 7f240f71d640 1 --2- 172.21.15.42:0/2131329708 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f24081340c0 0x7f240812dbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:57.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.617+0000 7f240f71d640 1 --2- 172.21.15.42:0/2131329708 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f240806db90 0x7f240812d6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:57.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.617+0000 7f240f71d640 1 -- 172.21.15.42:0/2131329708 >> 172.21.15.42:0/2131329708 conn(0x7f24080a9fb0 msgr2=0x7f24080a7a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:07:57.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.617+0000 7f240f71d640 1 -- 172.21.15.42:0/2131329708 shutdown_connections 2024-09-16T04:07:57.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.617+0000 7f240f71d640 1 -- 172.21.15.42:0/2131329708 wait complete. 2024-09-16T04:07:57.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.617+0000 7f240f71d640 1 librados: shutdown 2024-09-16T04:07:57.639 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T04:07:57.640 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T04:07:57.640 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T04:07:57.640 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T04:07:57.770 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.765+0000 7fb65e8ab640 10 monclient: get_monmap_and_config 2024-09-16T04:07:57.770 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.765+0000 7fb65e8ab640 10 monclient: build_initial_monmap 2024-09-16T04:07:57.770 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.765+0000 7fb65e8ab640 10 monclient: monmap: 2024-09-16T04:07:57.770 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:07:57.770 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:57.770 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:07:57.768472+0000 2024-09-16T04:07:57.770 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:07:57.768472+0000 2024-09-16T04:07:57.770 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:07:57.770 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:57.771 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:07:57.771 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:07:57.771 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:07:57.771 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:07:57.771 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:07:57.771 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:57.773 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.765+0000 7fb65e8ab640 1 Processor -- start 2024-09-16T04:07:57.774 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.765+0000 7fb65e8ab640 1 -- start start 2024-09-16T04:07:57.774 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.765+0000 7fb65e8ab640 10 monclient: init 2024-09-16T04:07:57.774 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.765+0000 7fb65e8ab640 10 monclient: _reopen_session rank -1 2024-09-16T04:07:57.774 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.765+0000 7fb65e8ab640 10 monclient: _add_conns ranks=[0,2,4,3,1] 2024-09-16T04:07:57.774 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.765+0000 7fb65e8ab640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb6580a1330 addr v1:172.21.15.42:6789/0 2024-09-16T04:07:57.774 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.765+0000 7fb65e8ab640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb65809c920 0x7fb65813c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:57.774 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.765+0000 7fb65e8ab640 10 monclient(hunting): picked mon.noname-b con 0x7fb65809c920 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:07:57.774 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.765+0000 7fb65e8ab640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fb658072d40 0x7fb65813eac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:57.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.765+0000 7fb65e8ab640 10 monclient(hunting): picked mon.noname-c con 0x7fb658072d40 addr v2:172.21.15.154:3300/0 2024-09-16T04:07:57.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.765+0000 7fb65e8ab640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb658072350 con 0x7fb6580a1330 2024-09-16T04:07:57.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.765+0000 7fb65e8ab640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:57.777 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.765+0000 7fb65e8ab640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fb658072b50 con 0x7fb658072d40 2024-09-16T04:07:57.777 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.765+0000 7fb65e8ab640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:57.777 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.765+0000 7fb65e8ab640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fb65809bc60 con 0x7fb65809c920 2024-09-16T04:07:57.777 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.765+0000 7fb65e8ab640 10 monclient(hunting): _renew_subs 2024-09-16T04:07:57.777 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.765+0000 7fb65e8ab640 10 monclient(hunting): authenticate will time out at 2524.738119s 2024-09-16T04:07:57.777 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.765+0000 7fb657fff640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fb6580a1330 0x7fb6580a1710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:44580/0 (socket says 172.21.15.42:44580) 2024-09-16T04:07:57.777 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.765+0000 7fb657fff640 1 -- 172.21.15.42:0/3826210220 learned_addr learned my addr 172.21.15.42:0/3826210220 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T04:07:57.777 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.765+0000 7fb64ffff640 1 --2- 172.21.15.42:0/3826210220 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb65809c920 0x7fb65813c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:57.778 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.765+0000 7fb64ffff640 10 monclient(hunting): get_auth_request con 0x7fb65809c920 auth_method 0 2024-09-16T04:07:57.778 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.765+0000 7fb64ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:57.778 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.765+0000 7fb64ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:57.778 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.765+0000 7fb64ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:57.778 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.765+0000 7fb64ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:07:57.779 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.765+0000 7fb64ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:07:57.779 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.765+0000 7fb64ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:07:57.779 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.765+0000 7fb65ce48640 1 --2- 172.21.15.42:0/3826210220 >> v2:172.21.15.154:3300/0 conn(0x7fb658072d40 0x7fb65813eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:57.779 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.765+0000 7fb6577fe640 1 -- 172.21.15.42:0/3826210220 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3808869969 0 0) 0x7fb658072350 con 0x7fb6580a1330 2024-09-16T04:07:57.779 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.765+0000 7fb6577fe640 10 monclient(hunting): _negotiate 2024-09-16T04:07:57.779 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.765+0000 7fb6577fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:57.779 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.765+0000 7fb6577fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:57.779 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.765+0000 7fb6577fe640 10 monclient(hunting): my global_id is 1024152 2024-09-16T04:07:57.779 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.765+0000 7fb6577fe640 1 -- 172.21.15.42:0/3826210220 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb644003530 con 0x7fb6580a1330 2024-09-16T04:07:57.779 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.765+0000 7fb65ce48640 10 monclient(hunting): get_auth_request con 0x7fb658072d40 auth_method 0 2024-09-16T04:07:57.779 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.765+0000 7fb65ce48640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:57.779 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.765+0000 7fb65ce48640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:57.779 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.765+0000 7fb65ce48640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:57.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.765+0000 7fb64ffff640 10 monclient(hunting): handle_auth_done global_id 1014221 payload 274 2024-09-16T04:07:57.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.765+0000 7fb64ffff640 1 -- 172.21.15.42:0/3826210220 >> v2:172.21.15.154:3300/0 conn(0x7fb658072d40 msgr2=0x7fb65813eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:07:57.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.765+0000 7fb64ffff640 1 --2- 172.21.15.42:0/3826210220 >> v2:172.21.15.154:3300/0 conn(0x7fb658072d40 0x7fb65813eac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:57.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.765+0000 7fb64ffff640 1 -- 172.21.15.42:0/3826210220 >> v1:172.21.15.42:6789/0 conn(0x7fb6580a1330 legacy=0x7fb6580a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:57.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.765+0000 7fb64ffff640 10 monclient: _finish_hunting 0 2024-09-16T04:07:57.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.765+0000 7fb64ffff640 1 monclient: found mon.noname-b 2024-09-16T04:07:57.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.765+0000 7fb64ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:07:57.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.765+0000 7fb64ffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T04:07:57.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.765+0000 7fb64ffff640 1 -- 172.21.15.42:0/3826210220 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb65809bf40 con 0x7fb65809c920 2024-09-16T04:07:57.789 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.769+0000 7fb64ffff640 1 --2- 172.21.15.42:0/3826210220 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb65809c920 0x7fb65813c570 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fb6400097c0 tx=0x7fb64001c360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9a5125032167e759 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:07:57.789 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.769+0000 7fb6577fe640 1 -- 172.21.15.42:0/3826210220 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6400b0070 con 0x7fb65809c920 2024-09-16T04:07:57.789 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.769+0000 7fb6577fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:57.789 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.769+0000 7fb6577fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:07:57.789 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.769+0000 7fb6577fe640 10 monclient: dump: 2024-09-16T04:07:57.789 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:07:57.790 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:57.790 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:57.790 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:57.790 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:57.790 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:57.790 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:07:57.790 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:07:57.790 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:07:57.791 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:57.791 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.769+0000 7fb6577fe640 10 monclient: _finish_auth 0 2024-09-16T04:07:57.791 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.769+0000 7fb6577fe640 10 monclient: _check_auth_tickets 2024-09-16T04:07:57.791 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.769+0000 7fb6577fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:07:57.791 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.769+0000 7fb6577fe640 1 -- 172.21.15.42:0/3826210220 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb6400ab600 con 0x7fb65809c920 2024-09-16T04:07:57.791 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.769+0000 7fb65e8ab640 5 monclient: authenticate success, global_id 1014221 2024-09-16T04:07:57.791 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.769+0000 7fb65e8ab640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:07:57.791 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.769+0000 7fb6577fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:07:57.791 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.769+0000 7fb65e8ab640 10 monclient: get_monmap_and_config success 2024-09-16T04:07:57.791 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.769+0000 7fb65e8ab640 4 set_mon_vals no callback set 2024-09-16T04:07:57.791 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.769+0000 7fb6577fe640 1 -- 172.21.15.42:0/3826210220 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6400ab9b0 con 0x7fb65809c920 2024-09-16T04:07:57.791 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.769+0000 7fb6577fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:57.791 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.769+0000 7fb6577fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:07:57.791 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.769+0000 7fb6577fe640 10 monclient: dump: 2024-09-16T04:07:57.792 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:07:57.792 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:57.792 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:57.792 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:57.792 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:57.792 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:57.792 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:07:57.792 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:07:57.792 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:07:57.792 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:57.792 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.769+0000 7fb65e8ab640 10 monclient: shutdown 2024-09-16T04:07:57.793 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.769+0000 7fb65e8ab640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:07:57.793 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.769+0000 7fb65e8ab640 1 -- 172.21.15.42:0/3826210220 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb65809c920 msgr2=0x7fb65813c570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:57.793 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.769+0000 7fb65e8ab640 1 --2- 172.21.15.42:0/3826210220 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb65809c920 0x7fb65813c570 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fb6400097c0 tx=0x7fb64001c360 comp rx=0 tx=0).stop 2024-09-16T04:07:57.793 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.769+0000 7fb65e8ab640 1 -- 172.21.15.42:0/3826210220 shutdown_connections 2024-09-16T04:07:57.793 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.769+0000 7fb65e8ab640 1 --2- 172.21.15.42:0/3826210220 >> v2:172.21.15.154:3300/0 conn(0x7fb658072d40 0x7fb65813eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:57.793 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.769+0000 7fb65e8ab640 1 --2- 172.21.15.42:0/3826210220 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb65809c920 0x7fb65813c570 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:57.793 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.769+0000 7fb65e8ab640 1 -- 172.21.15.42:0/3826210220 >> 172.21.15.42:0/3826210220 conn(0x7fb6580a2420 msgr2=0x7fb6580a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:07:57.793 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.769+0000 7fb65e8ab640 1 -- 172.21.15.42:0/3826210220 shutdown_connections 2024-09-16T04:07:57.793 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.769+0000 7fb65e8ab640 1 -- 172.21.15.42:0/3826210220 wait complete. 2024-09-16T04:07:57.793 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.769+0000 7fb65e8ab640 10 monclient: build_initial_monmap 2024-09-16T04:07:57.793 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.769+0000 7fb65e8ab640 10 monclient: monmap: 2024-09-16T04:07:57.793 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:07:57.794 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:07:57.794 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:07:57.794 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:07:57.794 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:07:57.794 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:57.794 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:07:57.794 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:07:57.794 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:07:57.794 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:57.794 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.769+0000 7fb65e8ab640 1 librados: starting msgr at 2024-09-16T04:07:57.794 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.769+0000 7fb65e8ab640 1 librados: starting objecter 2024-09-16T04:07:57.794 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.769+0000 7fb65e8ab640 1 Processor -- start 2024-09-16T04:07:57.794 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.769+0000 7fb65e8ab640 1 -- start start 2024-09-16T04:07:57.794 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.769+0000 7fb65e8ab640 1 librados: setting wanted keys 2024-09-16T04:07:57.795 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.769+0000 7fb65e8ab640 1 librados: calling monclient init 2024-09-16T04:07:57.795 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.769+0000 7fb65e8ab640 10 monclient: init 2024-09-16T04:07:57.795 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.769+0000 7fb65e8ab640 10 monclient: _reopen_session rank -1 2024-09-16T04:07:57.795 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.769+0000 7fb65e8ab640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:07:57.795 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.769+0000 7fb65e8ab640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb658072d40 0x7fb658133830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:57.795 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.769+0000 7fb65e8ab640 10 monclient(hunting): picked mon.noname-c con 0x7fb658072d40 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:07:57.795 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.769+0000 7fb65e8ab640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb65809c920 0x7fb658133d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:57.795 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.769+0000 7fb65e8ab640 10 monclient(hunting): picked mon.noname-a con 0x7fb65809c920 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:07:57.795 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.769+0000 7fb65e8ab640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb6580a1330 0x7fb6581342b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:57.795 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.769+0000 7fb65e8ab640 10 monclient(hunting): picked mon.noname-b con 0x7fb6580a1330 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:07:57.795 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.769+0000 7fb65e8ab640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:57.795 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.769+0000 7fb65e8ab640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fb658141c30 con 0x7fb65809c920 2024-09-16T04:07:57.795 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.769+0000 7fb65e8ab640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:57.795 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.769+0000 7fb65e8ab640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fb658141ab0 con 0x7fb6580a1330 2024-09-16T04:07:57.795 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.769+0000 7fb65e8ab640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:57.796 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.769+0000 7fb65e8ab640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fb658141db0 con 0x7fb658072d40 2024-09-16T04:07:57.796 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.769+0000 7fb65e8ab640 10 monclient(hunting): _renew_subs 2024-09-16T04:07:57.796 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.769+0000 7fb65e8ab640 10 monclient(hunting): authenticate will time out at 2524.743327s 2024-09-16T04:07:57.796 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.769+0000 7fb64ffff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb65809c920 0x7fb658133d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:57.796 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.769+0000 7fb657fff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb658072d40 0x7fb658133830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:57.796 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.769+0000 7fb657fff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb658072d40 0x7fb658133830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:37246/0 (socket says 172.21.15.42:37246) 2024-09-16T04:07:57.796 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.769+0000 7fb64ffff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb65809c920 0x7fb658133d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:43524/0 (socket says 172.21.15.42:43524) 2024-09-16T04:07:57.796 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.769+0000 7fb657fff640 1 -- 172.21.15.42:0/1674589955 learned_addr learned my addr 172.21.15.42:0/1674589955 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:07:57.796 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.769+0000 7fb657fff640 10 monclient(hunting): get_auth_request con 0x7fb658072d40 auth_method 0 2024-09-16T04:07:57.796 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.769+0000 7fb657fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:57.796 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.769+0000 7fb657fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:57.796 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.769+0000 7fb657fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:57.796 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.769+0000 7fb64ffff640 10 monclient(hunting): get_auth_request con 0x7fb65809c920 auth_method 0 2024-09-16T04:07:57.797 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.769+0000 7fb64ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:57.797 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.769+0000 7fb64ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:57.797 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.769+0000 7fb64ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:57.797 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.769+0000 7fb64ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:07:57.797 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.769+0000 7fb64ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:07:57.797 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.769+0000 7fb64ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:07:57.797 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.769+0000 7fb657fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:07:57.797 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.769+0000 7fb657fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:07:57.797 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.769+0000 7fb657fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:07:57.797 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.769+0000 7fb65ce48640 1 --2- 172.21.15.42:0/1674589955 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb6580a1330 0x7fb6581342b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:57.797 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.769+0000 7fb65ce48640 10 monclient(hunting): get_auth_request con 0x7fb6580a1330 auth_method 0 2024-09-16T04:07:57.797 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.769+0000 7fb65ce48640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:57.797 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.769+0000 7fb65ce48640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:57.798 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.769+0000 7fb65ce48640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:57.798 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.769+0000 7fb64ffff640 10 monclient(hunting): handle_auth_done global_id 1024155 payload 824 2024-09-16T04:07:57.798 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.769+0000 7fb64ffff640 1 -- 172.21.15.42:0/1674589955 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb658072d40 msgr2=0x7fb658133830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:57.798 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.769+0000 7fb64ffff640 1 --2- 172.21.15.42:0/1674589955 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb658072d40 0x7fb658133830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:57.798 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.769+0000 7fb64ffff640 1 -- 172.21.15.42:0/1674589955 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb6580a1330 msgr2=0x7fb6581342b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:57.798 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.769+0000 7fb64ffff640 1 --2- 172.21.15.42:0/1674589955 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb6580a1330 0x7fb6581342b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:57.798 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.769+0000 7fb64ffff640 10 monclient: _finish_hunting 0 2024-09-16T04:07:57.798 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.769+0000 7fb64ffff640 1 monclient: found mon.noname-a 2024-09-16T04:07:57.798 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.769+0000 7fb64ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:07:57.798 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.769+0000 7fb64ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:07:57.798 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.769+0000 7fb64ffff640 1 -- 172.21.15.42:0/1674589955 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb658073a30 con 0x7fb65809c920 2024-09-16T04:07:57.798 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.769+0000 7fb657fff640 1 --2- 172.21.15.42:0/1674589955 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb658072d40 0x7fb658133830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:07:57.799 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.773+0000 7fb64ffff640 1 --2- 172.21.15.42:0/1674589955 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb65809c920 0x7fb658133d70 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fb6400ab000 tx=0x7fb640019100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:07:57.799 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.773+0000 7fb6557fa640 10 client.?.objecter ms_handle_connect 0x7fb65809c920 2024-09-16T04:07:57.799 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.773+0000 7fb6557fa640 10 client.?.objecter resend_mon_ops 2024-09-16T04:07:57.799 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.773+0000 7fb6557fa640 1 -- 172.21.15.42:0/1674589955 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6400b8020 con 0x7fb65809c920 2024-09-16T04:07:57.799 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.773+0000 7fb6557fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:57.799 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.773+0000 7fb6557fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:07:57.799 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.773+0000 7fb6557fa640 10 monclient: dump: 2024-09-16T04:07:57.799 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:07:57.799 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:57.799 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:57.799 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:57.799 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:57.799 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:57.802 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:07:57.802 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:07:57.802 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:07:57.802 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:57.802 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.773+0000 7fb6557fa640 10 monclient: _finish_auth 0 2024-09-16T04:07:57.802 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.773+0000 7fb6557fa640 10 monclient: _check_auth_tickets 2024-09-16T04:07:57.802 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.773+0000 7fb6557fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:07:57.803 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.773+0000 7fb65e8ab640 5 monclient: authenticate success, global_id 1024155 2024-09-16T04:07:57.803 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.773+0000 7fb6557fa640 1 -- 172.21.15.42:0/1674589955 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb6400b4610 con 0x7fb65809c920 2024-09-16T04:07:57.803 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.773+0000 7fb6557fa640 10 monclient: handle_config config(0 keys) 2024-09-16T04:07:57.803 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.773+0000 7fb65e8ab640 10 monclient: _renew_subs 2024-09-16T04:07:57.803 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.773+0000 7fb6557fa640 1 -- 172.21.15.42:0/1674589955 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6400b0040 con 0x7fb65809c920 2024-09-16T04:07:57.803 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.773+0000 7fb65e8ab640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:07:57.803 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.773+0000 7fb65e8ab640 1 -- 172.21.15.42:0/1674589955 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb65813dd70 con 0x7fb65809c920 2024-09-16T04:07:57.803 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.773+0000 7fb656ffd640 4 set_mon_vals no callback set 2024-09-16T04:07:57.803 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.773+0000 7fb6557fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:57.803 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.773+0000 7fb6557fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:07:57.803 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.773+0000 7fb6557fa640 10 monclient: dump: 2024-09-16T04:07:57.803 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:07:57.804 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:57.804 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:57.804 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:57.804 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:57.804 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:57.804 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:07:57.804 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:07:57.804 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:07:57.804 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:57.804 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.773+0000 7fb65e8ab640 10 client.1024155.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:07:57.804 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.773+0000 7fb65e8ab640 10 monclient: _renew_subs 2024-09-16T04:07:57.804 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.773+0000 7fb65e8ab640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:07:57.804 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.773+0000 7fb65e8ab640 1 -- 172.21.15.42:0/1674589955 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb65811d2a0 con 0x7fb65809c920 2024-09-16T04:07:57.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.773+0000 7fb65e8ab640 1 librados: init done 2024-09-16T04:07:57.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.773+0000 7fb65e8ab640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:07:57.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.773+0000 7fb65e8ab640 1 --2- 172.21.15.42:0/1674589955 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb658062ea0 0x7fb658063280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:57.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.773+0000 7fb65e8ab640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:57.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.773+0000 7fb65e8ab640 1 -- 172.21.15.42:0/1674589955 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fb65813edd0 con 0x7fb658062ea0 2024-09-16T04:07:57.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.773+0000 7fb657fff640 1 --2- 172.21.15.42:0/1674589955 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb658062ea0 0x7fb658063280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:57.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.773+0000 7fb657fff640 10 monclient: get_auth_request con 0x7fb658062ea0 auth_method 0 2024-09-16T04:07:57.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.773+0000 7fb657fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:57.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.773+0000 7fb657fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:57.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.773+0000 7fb657fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:57.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.773+0000 7fb6557fa640 1 -- 172.21.15.42:0/1674589955 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fb6400bb020 con 0x7fb65809c920 2024-09-16T04:07:57.806 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.773+0000 7fb657fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:07:57.807 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.773+0000 7fb657fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:07:57.807 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.773+0000 7fb657fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:07:57.807 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.773+0000 7fb657fff640 10 monclient(hunting): handle_auth_done global_id 1024158 payload 824 2024-09-16T04:07:57.808 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.773+0000 7fb6557fa640 1 --2- 172.21.15.42:0/1674589955 >> v2:172.21.15.154:6812/254845342 conn(0x7fb638037810 0x7fb638039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:57.808 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.773+0000 7fb657fff640 1 -- 172.21.15.42:0/1674589955 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb6580a3be0 con 0x7fb658062ea0 2024-09-16T04:07:57.808 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.773+0000 7fb6557fa640 1 -- 172.21.15.42:0/1674589955 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(706..706 src has 451..706) ==== 5040+0+0 (secure 0 0 0) 0x7fb6400e2950 con 0x7fb65809c920 2024-09-16T04:07:57.808 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.773+0000 7fb6557fa640 10 client.1024155.objecter ms_dispatch 0x7fb658000ba0 osd_map(706..706 src has 451..706) 2024-09-16T04:07:57.808 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.773+0000 7fb6557fa640 3 client.1024155.objecter handle_osd_map got epochs [706,706] > 0 2024-09-16T04:07:57.808 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.773+0000 7fb6557fa640 3 client.1024155.objecter handle_osd_map decoding full epoch 706 2024-09-16T04:07:57.808 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.773+0000 7fb657fff640 1 --2- 172.21.15.42:0/1674589955 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb658062ea0 0x7fb658063280 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fb648005890 tx=0x7fb64800ff70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:07:57.808 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.773+0000 7fb6557fa640 20 client.1024155.objecter dump_active .. 0 homeless 2024-09-16T04:07:57.808 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.773+0000 7fb6557fa640 10 client.1024155.objecter ms_handle_connect 0x7fb658062ea0 2024-09-16T04:07:57.808 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.773+0000 7fb6557fa640 10 client.1024155.objecter resend_mon_ops 2024-09-16T04:07:57.808 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.773+0000 7fb6557fa640 1 -- 172.21.15.42:0/1674589955 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb64800dae0 con 0x7fb658062ea0 2024-09-16T04:07:57.808 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.773+0000 7fb6557fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:57.808 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.773+0000 7fb6557fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:07:57.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.773+0000 7fb6557fa640 10 monclient: dump: 2024-09-16T04:07:57.809 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:07:57.809 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:57.809 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:57.809 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:57.809 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:57.809 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:57.809 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:07:57.809 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:07:57.809 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:07:57.809 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:57.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.773+0000 7fb65ce48640 1 --2- 172.21.15.42:0/1674589955 >> v2:172.21.15.154:6812/254845342 conn(0x7fb638037810 0x7fb638039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:57.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.773+0000 7fb65ce48640 10 monclient: get_auth_request con 0x7fb638037810 auth_method 0 2024-09-16T04:07:57.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.773+0000 7fb6557fa640 1 -- 172.21.15.42:0/1674589955 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fb648018070 con 0x7fb658062ea0 2024-09-16T04:07:57.810 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.773+0000 7fb6557fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:07:57.810 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.773+0000 7fb6557fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:07:57.810 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.773+0000 7fb6557fa640 1 -- 172.21.15.42:0/1674589955 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb658062ea0 msgr2=0x7fb658063280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:57.810 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.773+0000 7fb6557fa640 1 --2- 172.21.15.42:0/1674589955 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb658062ea0 0x7fb658063280 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fb648005890 tx=0x7fb64800ff70 comp rx=0 tx=0).stop 2024-09-16T04:07:57.810 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.773+0000 7fb6557fa640 1 -- 172.21.15.42:0/1674589955 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb658062ea0 msgr2=0x7fb658063280 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:07:57.810 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.773+0000 7fb6557fa640 1 --2- 172.21.15.42:0/1674589955 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb658062ea0 0x7fb658063280 secure :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fb648005890 tx=0x7fb64800ff70 comp rx=0 tx=0).stop 2024-09-16T04:07:57.810 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.773+0000 7fb65ce48640 1 --2- 172.21.15.42:0/1674589955 >> v2:172.21.15.154:6812/254845342 conn(0x7fb638037810 0x7fb638039cd0 secure :-1 s=READY pgs=3894 cs=0 l=1 rev1=1 crypto rx=0x7fb650009f80 tx=0x7fb65000b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:07:57.810 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.773+0000 7fb6557fa640 10 client.1024155.objecter ms_handle_connect 0x7fb638037810 2024-09-16T04:07:57.810 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.781+0000 7fb65e8ab640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:07:57.810 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.781+0000 7fb65e8ab640 1 --2- 172.21.15.42:0/1674589955 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb65802c650 0x7fb65802ca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:57.810 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.781+0000 7fb65e8ab640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:57.810 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.781+0000 7fb65e8ab640 1 -- 172.21.15.42:0/1674589955 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fb6580a1710 con 0x7fb65802c650 2024-09-16T04:07:57.810 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.781+0000 7fb657fff640 1 --2- 172.21.15.42:0/1674589955 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb65802c650 0x7fb65802ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:57.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.781+0000 7fb657fff640 10 monclient: get_auth_request con 0x7fb65802c650 auth_method 0 2024-09-16T04:07:57.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.781+0000 7fb657fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:57.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.781+0000 7fb657fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:57.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.781+0000 7fb657fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:57.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.781+0000 7fb657fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:07:57.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.781+0000 7fb657fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:07:57.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.781+0000 7fb657fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:07:57.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.781+0000 7fb657fff640 10 monclient(hunting): handle_auth_done global_id 1024161 payload 824 2024-09-16T04:07:57.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.781+0000 7fb657fff640 1 -- 172.21.15.42:0/1674589955 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fb658139c60 con 0x7fb65802c650 2024-09-16T04:07:57.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.781+0000 7fb657fff640 1 --2- 172.21.15.42:0/1674589955 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb65802c650 0x7fb65802ca30 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fb648005890 tx=0x7fb648000ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:07:57.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.781+0000 7fb6557fa640 10 client.1024155.objecter ms_handle_connect 0x7fb65802c650 2024-09-16T04:07:57.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.781+0000 7fb6557fa640 10 client.1024155.objecter resend_mon_ops 2024-09-16T04:07:57.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.781+0000 7fb6557fa640 1 -- 172.21.15.42:0/1674589955 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6480072e0 con 0x7fb65802c650 2024-09-16T04:07:57.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.781+0000 7fb6557fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:57.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.781+0000 7fb6557fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:07:57.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.781+0000 7fb6557fa640 10 monclient: dump: 2024-09-16T04:07:57.812 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:07:57.812 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:57.812 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:57.812 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:57.812 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:57.812 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:57.812 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:07:57.812 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:07:57.812 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:07:57.812 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:57.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.781+0000 7fb6557fa640 1 -- 172.21.15.42:0/1674589955 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4546 (secure 0 0 0) 0x7fb6480072e0 con 0x7fb65802c650 2024-09-16T04:07:57.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.781+0000 7fb6557fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:07:57.813 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.781+0000 7fb6557fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:07:57.813 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.781+0000 7fb6557fa640 1 -- 172.21.15.42:0/1674589955 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb65802c650 msgr2=0x7fb65802ca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:57.813 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.781+0000 7fb6557fa640 1 --2- 172.21.15.42:0/1674589955 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb65802c650 0x7fb65802ca30 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fb648005890 tx=0x7fb648000ee0 comp rx=0 tx=0).stop 2024-09-16T04:07:57.813 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.781+0000 7fb6557fa640 1 -- 172.21.15.42:0/1674589955 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb65802c650 msgr2=0x7fb65802ca30 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:07:57.813 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.781+0000 7fb6557fa640 1 --2- 172.21.15.42:0/1674589955 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb65802c650 0x7fb65802ca30 secure :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fb648005890 tx=0x7fb648000ee0 comp rx=0 tx=0).stop 2024-09-16T04:07:57.813 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T04:07:57.813 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:07:57.813 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:07:57.813 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "leader", 2024-09-16T04:07:57.814 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 238, 2024-09-16T04:07:57.814 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 9751, 2024-09-16T04:07:57.814 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T04:07:57.814 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T04:07:57.814 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T04:07:57.814 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T04:07:57.814 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:07:57.814 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 3, 2024-09-16T04:07:57.814 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:07:57.814 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T04:07:57.814 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T04:07:57.814 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:07:57.814 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:07:57.815 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:07:57.815 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:07:57.815 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:07:57.815 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:07:57.815 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:07:57.815 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:07:57.815 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:07:57.815 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:07:57.815 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:07:57.815 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:07:57.815 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:07:57.815 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T04:07:57.815 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:07:57.816 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:07:57.816 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:07:57.816 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:07:57.816 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:07:57.816 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:07:57.816 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:07:57.816 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:07:57.816 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:07:57.816 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:07:57.816 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:07:57.816 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:07:57.816 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:57.816 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T04:07:57.817 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T04:07:57.817 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T04:07:57.817 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T04:07:57.817 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T04:07:57.817 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T04:07:57.817 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:07:57.817 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:07:57.817 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T04:07:57.817 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T04:07:57.817 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T04:07:57.817 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T04:07:57.817 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T04:07:57.817 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T04:07:57.817 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T04:07:57.818 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:07:57.818 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T04:07:57.818 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:07:57.818 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:07:57.818 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:07:57.818 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:07:57.818 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:07:57.818 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:07:57.818 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:07:57.818 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:07:57.818 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:07:57.818 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:07:57.818 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:07:57.818 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:07:57.819 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T04:07:57.819 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:57.819 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T04:07:57.819 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:57.819 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:07:57.819 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:07:57.819 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:07:57.819 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:07:57.819 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:57.819 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:07:57.820 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T04:07:57.820 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:07:57.820 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:57.820 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:57.820 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:07:57.820 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T04:07:57.820 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:07:57.820 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:07:57.820 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:07:57.820 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:57.820 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T04:07:57.821 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T04:07:57.821 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:07:57.821 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:07:57.821 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:07:57.821 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:57.821 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:57.821 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:07:57.821 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:07:57.821 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:07:57.821 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:07:57.821 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:57.821 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:07:57.821 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T04:07:57.822 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:07:57.822 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:57.822 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:57.822 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:07:57.822 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T04:07:57.822 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:07:57.822 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:07:57.822 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:07:57.822 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:57.822 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T04:07:57.822 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T04:07:57.822 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:07:57.822 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:07:57.822 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:07:57.823 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:57.823 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:57.823 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:07:57.823 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:07:57.823 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:07:57.823 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:07:57.823 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:57.823 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:07:57.823 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T04:07:57.823 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:07:57.823 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:57.823 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:57.823 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:07:57.823 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T04:07:57.824 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:07:57.824 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:07:57.824 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:07:57.824 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:57.824 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T04:07:57.824 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T04:07:57.824 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:07:57.824 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:07:57.824 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:07:57.824 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:07:57.824 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:07:57.824 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:57.824 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T04:07:57.824 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T04:07:57.825 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:57.825 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:07:57.825 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:07:57.825 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:07:57.825 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:07:57.825 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:07:57.825 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T04:07:57.825 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:57.825 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:07:57.825 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:07:57.825 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 2 2024-09-16T04:07:57.825 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:07:57.825 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:07:57.825 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:57.826 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T04:07:57.826 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T04:07:57.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.781+0000 7fb65e8ab640 10 librados: watch_flush enter 2024-09-16T04:07:57.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.781+0000 7fb65e8ab640 10 librados: watch_flush exit 2024-09-16T04:07:57.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.781+0000 7fb65e8ab640 20 client.1024155.objecter shutdown clearing up homeless session... 2024-09-16T04:07:57.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.781+0000 7fb65e8ab640 10 client.1024155.objecter successfully canceled tick 2024-09-16T04:07:57.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.781+0000 7fb65e8ab640 1 -- 172.21.15.42:0/1674589955 >> v2:172.21.15.154:6812/254845342 conn(0x7fb638037810 msgr2=0x7fb638039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:57.828 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.781+0000 7fb65e8ab640 1 --2- 172.21.15.42:0/1674589955 >> v2:172.21.15.154:6812/254845342 conn(0x7fb638037810 0x7fb638039cd0 secure :-1 s=READY pgs=3894 cs=0 l=1 rev1=1 crypto rx=0x7fb650009f80 tx=0x7fb65000b040 comp rx=0 tx=0).stop 2024-09-16T04:07:57.828 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.781+0000 7fb65e8ab640 10 monclient: shutdown 2024-09-16T04:07:57.828 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.781+0000 7fb65e8ab640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:07:57.828 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.781+0000 7fb65e8ab640 1 -- 172.21.15.42:0/1674589955 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb65809c920 msgr2=0x7fb658133d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:57.828 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.781+0000 7fb65ce48640 1 -- 172.21.15.42:0/1674589955 reap_dead start 2024-09-16T04:07:57.828 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.781+0000 7fb65e8ab640 1 --2- 172.21.15.42:0/1674589955 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb65809c920 0x7fb658133d70 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fb6400ab000 tx=0x7fb640019100 comp rx=0 tx=0).stop 2024-09-16T04:07:57.828 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.781+0000 7fb65e8ab640 1 -- 172.21.15.42:0/1674589955 shutdown_connections 2024-09-16T04:07:57.828 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.781+0000 7fb65e8ab640 1 --2- 172.21.15.42:0/1674589955 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb65809c920 0x7fb658133d70 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:57.828 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.781+0000 7fb65e8ab640 1 -- 172.21.15.42:0/1674589955 >> 172.21.15.42:0/1674589955 conn(0x7fb6580a2420 msgr2=0x7fb6580a4130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:07:57.828 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.781+0000 7fb65e8ab640 1 -- 172.21.15.42:0/1674589955 shutdown_connections 2024-09-16T04:07:57.828 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.781+0000 7fb65e8ab640 1 -- 172.21.15.42:0/1674589955 wait complete. 2024-09-16T04:07:57.828 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.781+0000 7fb65e8ab640 1 librados: shutdown 2024-09-16T04:07:57.829 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:07:57.829 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T04:07:57.829 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T04:07:57.946 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.941+0000 7fc6ec907640 10 monclient: get_monmap_and_config 2024-09-16T04:07:57.946 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.941+0000 7fc6ec907640 10 monclient: build_initial_monmap 2024-09-16T04:07:57.946 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.941+0000 7fc6ec907640 10 monclient: monmap: 2024-09-16T04:07:57.946 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:07:57.946 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:57.946 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:07:57.944980+0000 2024-09-16T04:07:57.946 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:07:57.944980+0000 2024-09-16T04:07:57.947 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:07:57.947 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:57.947 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:07:57.947 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:07:57.947 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:07:57.947 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:07:57.947 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:07:57.947 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:57.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.941+0000 7fc6ec907640 1 Processor -- start 2024-09-16T04:07:57.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.941+0000 7fc6ec907640 1 -- start start 2024-09-16T04:07:57.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.941+0000 7fc6ec907640 10 monclient: init 2024-09-16T04:07:57.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.941+0000 7fc6ec907640 10 monclient: _reopen_session rank -1 2024-09-16T04:07:57.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.941+0000 7fc6ec907640 10 monclient: _add_conns ranks=[1,2,4,0,3] 2024-09-16T04:07:57.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.941+0000 7fc6ec907640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc6e806db90 addr v1:172.21.15.154:6789/0 2024-09-16T04:07:57.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.941+0000 7fc6ec907640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc6e81340c0 0x7fc6e813c190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:57.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.941+0000 7fc6ec907640 10 monclient(hunting): picked mon.noname-b con 0x7fc6e81340c0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:07:57.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.941+0000 7fc6ec907640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fc6e813c6d0 0x7fc6e813eac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:57.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.941+0000 7fc6ec907640 10 monclient(hunting): picked mon.noname-c con 0x7fc6e813c6d0 addr v2:172.21.15.154:3300/0 2024-09-16T04:07:57.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.941+0000 7fc6ec907640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc6e806ece0 con 0x7fc6e806db90 2024-09-16T04:07:57.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.941+0000 7fc6ec907640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:57.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.941+0000 7fc6ec907640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fc6e806df70 con 0x7fc6e813c6d0 2024-09-16T04:07:57.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.941+0000 7fc6ec907640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:57.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.941+0000 7fc6ec907640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fc6e809bf80 con 0x7fc6e81340c0 2024-09-16T04:07:57.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.941+0000 7fc6ec907640 10 monclient(hunting): _renew_subs 2024-09-16T04:07:57.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.941+0000 7fc6ec907640 10 monclient(hunting): authenticate will time out at 2524.914897s 2024-09-16T04:07:57.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.941+0000 7fc6dffff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc6e81340c0 0x7fc6e813c190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:57.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.941+0000 7fc6dffff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc6e81340c0 0x7fc6e813c190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:37258/0 (socket says 172.21.15.42:37258) 2024-09-16T04:07:57.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.941+0000 7fc6dffff640 1 -- 172.21.15.42:0/779569346 learned_addr learned my addr 172.21.15.42:0/779569346 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:07:57.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.941+0000 7fc6dffff640 10 monclient(hunting): get_auth_request con 0x7fc6e81340c0 auth_method 0 2024-09-16T04:07:57.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.941+0000 7fc6dffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:57.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.941+0000 7fc6dffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:57.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.941+0000 7fc6dffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:57.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.941+0000 7fc6e6d9d640 1 --2- 172.21.15.42:0/779569346 >> v2:172.21.15.154:3300/0 conn(0x7fc6e813c6d0 0x7fc6e813eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:57.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.941+0000 7fc6e6d9d640 10 monclient(hunting): get_auth_request con 0x7fc6e813c6d0 auth_method 0 2024-09-16T04:07:57.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.941+0000 7fc6e6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:57.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.941+0000 7fc6e6d9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:57.952 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.941+0000 7fc6e6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:57.952 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.941+0000 7fc6dffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:07:57.952 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.941+0000 7fc6dffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:07:57.952 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.941+0000 7fc6dffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:07:57.952 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.941+0000 7fc6e6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:07:57.952 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.941+0000 7fc6e6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:07:57.952 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.941+0000 7fc6e6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:07:57.952 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.941+0000 7fc6dffff640 10 monclient(hunting): handle_auth_done global_id 1014227 payload 274 2024-09-16T04:07:57.952 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.941+0000 7fc6dffff640 1 -- 172.21.15.42:0/779569346 >> v2:172.21.15.154:3300/0 conn(0x7fc6e813c6d0 msgr2=0x7fc6e813eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:07:57.952 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.941+0000 7fc6dffff640 1 --2- 172.21.15.42:0/779569346 >> v2:172.21.15.154:3300/0 conn(0x7fc6e813c6d0 0x7fc6e813eac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:57.952 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.941+0000 7fc6dffff640 1 -- 172.21.15.42:0/779569346 >> v1:172.21.15.154:6789/0 conn(0x7fc6e806db90 legacy=0x7fc6e80a37f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:57.952 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.941+0000 7fc6dffff640 10 monclient: _finish_hunting 0 2024-09-16T04:07:57.952 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.941+0000 7fc6dffff640 1 monclient: found mon.noname-b 2024-09-16T04:07:57.953 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.941+0000 7fc6dffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:07:57.953 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.941+0000 7fc6dffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T04:07:57.953 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.941+0000 7fc6dffff640 1 -- 172.21.15.42:0/779569346 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6e81344d0 con 0x7fc6e81340c0 2024-09-16T04:07:57.959 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.945+0000 7fc6dffff640 1 --2- 172.21.15.42:0/779569346 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc6e81340c0 0x7fc6e813c190 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fc6d0009870 tx=0x7fc6d001c420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fcaa059e8a09909f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:07:57.959 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.945+0000 7fc6e5d9b640 1 -- 172.21.15.42:0/779569346 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6d00ab620 con 0x7fc6e81340c0 2024-09-16T04:07:57.960 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.945+0000 7fc6e5d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:57.960 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.945+0000 7fc6e5d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:07:57.960 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.945+0000 7fc6e5d9b640 10 monclient: dump: 2024-09-16T04:07:57.960 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:07:57.960 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:57.960 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:57.960 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:57.960 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:57.960 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:57.960 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:07:57.960 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:07:57.960 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:07:57.961 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:57.961 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.945+0000 7fc6e5d9b640 10 monclient: _finish_auth 0 2024-09-16T04:07:57.961 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.945+0000 7fc6e5d9b640 10 monclient: _check_auth_tickets 2024-09-16T04:07:57.961 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.945+0000 7fc6e5d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:07:57.961 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.945+0000 7fc6e5d9b640 1 -- 172.21.15.42:0/779569346 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc6d00b0070 con 0x7fc6e81340c0 2024-09-16T04:07:57.961 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.945+0000 7fc6e5d9b640 10 monclient: handle_config config(0 keys) 2024-09-16T04:07:57.961 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.945+0000 7fc6e5d9b640 1 -- 172.21.15.42:0/779569346 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6d00ab9e0 con 0x7fc6e81340c0 2024-09-16T04:07:57.961 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.945+0000 7fc6ec907640 5 monclient: authenticate success, global_id 1014227 2024-09-16T04:07:57.961 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.945+0000 7fc6ec907640 10 monclient: get_monmap_and_config success 2024-09-16T04:07:57.961 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.945+0000 7fc6ec907640 4 set_mon_vals no callback set 2024-09-16T04:07:57.961 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.945+0000 7fc6e5d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:57.961 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.945+0000 7fc6e5d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:07:57.961 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.945+0000 7fc6e5d9b640 10 monclient: dump: 2024-09-16T04:07:57.961 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:07:57.962 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:57.962 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:57.962 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:57.962 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:57.962 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:57.962 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:07:57.962 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:07:57.962 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:07:57.962 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:57.962 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.945+0000 7fc6ec907640 10 monclient: shutdown 2024-09-16T04:07:57.962 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.945+0000 7fc6ec907640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:07:57.962 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.945+0000 7fc6ec907640 1 -- 172.21.15.42:0/779569346 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc6e81340c0 msgr2=0x7fc6e813c190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:57.962 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.945+0000 7fc6ec907640 1 --2- 172.21.15.42:0/779569346 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc6e81340c0 0x7fc6e813c190 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fc6d0009870 tx=0x7fc6d001c420 comp rx=0 tx=0).stop 2024-09-16T04:07:57.963 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.945+0000 7fc6ec907640 1 -- 172.21.15.42:0/779569346 shutdown_connections 2024-09-16T04:07:57.963 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.945+0000 7fc6ec907640 1 --2- 172.21.15.42:0/779569346 >> v2:172.21.15.154:3300/0 conn(0x7fc6e813c6d0 0x7fc6e813eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:57.963 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.945+0000 7fc6ec907640 1 --2- 172.21.15.42:0/779569346 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc6e81340c0 0x7fc6e813c190 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:57.963 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.945+0000 7fc6ec907640 1 -- 172.21.15.42:0/779569346 >> 172.21.15.42:0/779569346 conn(0x7fc6e80a9fb0 msgr2=0x7fc6e80aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:07:57.963 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.945+0000 7fc6ec907640 1 -- 172.21.15.42:0/779569346 shutdown_connections 2024-09-16T04:07:57.963 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.945+0000 7fc6ec907640 1 -- 172.21.15.42:0/779569346 wait complete. 2024-09-16T04:07:57.963 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.945+0000 7fc6ec907640 10 monclient: build_initial_monmap 2024-09-16T04:07:57.963 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.945+0000 7fc6ec907640 10 monclient: monmap: 2024-09-16T04:07:57.963 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:07:57.963 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:07:57.963 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:07:57.963 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:07:57.963 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:07:57.963 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:57.964 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:07:57.964 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:07:57.964 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:07:57.964 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:57.964 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.949+0000 7fc6ec907640 1 librados: starting msgr at 2024-09-16T04:07:57.964 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.949+0000 7fc6ec907640 1 librados: starting objecter 2024-09-16T04:07:57.964 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.949+0000 7fc6ec907640 1 Processor -- start 2024-09-16T04:07:57.964 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.949+0000 7fc6ec907640 1 -- start start 2024-09-16T04:07:57.964 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.949+0000 7fc6ec907640 1 librados: setting wanted keys 2024-09-16T04:07:57.964 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.949+0000 7fc6ec907640 1 librados: calling monclient init 2024-09-16T04:07:57.964 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.949+0000 7fc6ec907640 10 monclient: init 2024-09-16T04:07:57.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.949+0000 7fc6ec907640 10 monclient: _reopen_session rank -1 2024-09-16T04:07:57.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.949+0000 7fc6ec907640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:07:57.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.949+0000 7fc6ec907640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc6e806db90 0x7fc6e81ceed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:57.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.949+0000 7fc6ec907640 10 monclient(hunting): picked mon.noname-c con 0x7fc6e806db90 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:07:57.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.949+0000 7fc6ec907640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc6e81340c0 0x7fc6e813af00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:57.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.949+0000 7fc6ec907640 10 monclient(hunting): picked mon.noname-a con 0x7fc6e81340c0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:07:57.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.949+0000 7fc6ec907640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc6e813c6d0 0x7fc6e813b440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:57.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.949+0000 7fc6ec907640 10 monclient(hunting): picked mon.noname-b con 0x7fc6e813c6d0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:07:57.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.949+0000 7fc6ec907640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:57.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.949+0000 7fc6ec907640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fc6e8141820 con 0x7fc6e81340c0 2024-09-16T04:07:57.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.949+0000 7fc6ec907640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:57.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.949+0000 7fc6ec907640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc6e81419a0 con 0x7fc6e813c6d0 2024-09-16T04:07:57.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.949+0000 7fc6ec907640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:57.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.949+0000 7fc6ec907640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fc6e8141b20 con 0x7fc6e806db90 2024-09-16T04:07:57.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.949+0000 7fc6ec907640 10 monclient(hunting): _renew_subs 2024-09-16T04:07:57.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.949+0000 7fc6ec907640 10 monclient(hunting): authenticate will time out at 2524.920896s 2024-09-16T04:07:57.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.949+0000 7fc6e659c640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc6e806db90 0x7fc6e81ceed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:57.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.949+0000 7fc6dffff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc6e81340c0 0x7fc6e813af00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:57.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.949+0000 7fc6e659c640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc6e806db90 0x7fc6e81ceed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:37268/0 (socket says 172.21.15.42:37268) 2024-09-16T04:07:57.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.949+0000 7fc6dffff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc6e81340c0 0x7fc6e813af00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:43542/0 (socket says 172.21.15.42:43542) 2024-09-16T04:07:57.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.949+0000 7fc6e659c640 1 -- 172.21.15.42:0/219249532 learned_addr learned my addr 172.21.15.42:0/219249532 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:07:57.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.949+0000 7fc6e659c640 10 monclient(hunting): get_auth_request con 0x7fc6e806db90 auth_method 0 2024-09-16T04:07:57.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.949+0000 7fc6e659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:57.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.949+0000 7fc6e659c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:57.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.949+0000 7fc6e659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:57.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.949+0000 7fc6dffff640 10 monclient(hunting): get_auth_request con 0x7fc6e81340c0 auth_method 0 2024-09-16T04:07:57.967 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.949+0000 7fc6dffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:57.967 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.949+0000 7fc6dffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:57.967 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.949+0000 7fc6dffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:57.967 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.949+0000 7fc6e659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:07:57.967 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.949+0000 7fc6e659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:07:57.967 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.949+0000 7fc6e659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:07:57.967 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.949+0000 7fc6dffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:07:57.967 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.949+0000 7fc6dffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:07:57.967 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.949+0000 7fc6dffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:07:57.967 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.949+0000 7fc6e659c640 10 monclient(hunting): handle_auth_done global_id 1014230 payload 824 2024-09-16T04:07:57.967 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.949+0000 7fc6e659c640 1 -- 172.21.15.42:0/219249532 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc6e813c6d0 msgr2=0x7fc6e813b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:57.967 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.949+0000 7fc6e659c640 1 --2- 172.21.15.42:0/219249532 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc6e813c6d0 0x7fc6e813b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:57.967 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.949+0000 7fc6e659c640 1 -- 172.21.15.42:0/219249532 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc6e81340c0 msgr2=0x7fc6e813af00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:57.967 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.949+0000 7fc6e659c640 1 --2- 172.21.15.42:0/219249532 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc6e81340c0 0x7fc6e813af00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:57.968 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.949+0000 7fc6e659c640 10 monclient: _finish_hunting 0 2024-09-16T04:07:57.968 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.949+0000 7fc6e659c640 1 monclient: found mon.noname-c 2024-09-16T04:07:57.968 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.949+0000 7fc6e659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:07:57.968 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.949+0000 7fc6e659c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T04:07:57.968 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.949+0000 7fc6e659c640 1 -- 172.21.15.42:0/219249532 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6e8132fc0 con 0x7fc6e806db90 2024-09-16T04:07:57.968 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.949+0000 7fc6e659c640 1 --2- 172.21.15.42:0/219249532 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc6e806db90 0x7fc6e81ceed0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fc6d800c3f0 tx=0x7fc6d800c8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:07:57.968 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.949+0000 7fc6deffd640 10 client.?.objecter ms_handle_connect 0x7fc6e806db90 2024-09-16T04:07:57.968 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.949+0000 7fc6deffd640 10 client.?.objecter resend_mon_ops 2024-09-16T04:07:57.968 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.949+0000 7fc6deffd640 1 -- 172.21.15.42:0/219249532 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6d8017070 con 0x7fc6e806db90 2024-09-16T04:07:57.968 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.949+0000 7fc6deffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:57.968 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.949+0000 7fc6deffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:07:57.968 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.949+0000 7fc6deffd640 10 monclient: dump: 2024-09-16T04:07:57.969 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:07:57.969 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:57.969 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:57.969 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:57.969 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:57.969 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:57.969 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:07:57.969 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:07:57.969 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:07:57.969 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:57.969 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.949+0000 7fc6deffd640 10 monclient: _finish_auth 0 2024-09-16T04:07:57.969 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.949+0000 7fc6deffd640 10 monclient: _check_auth_tickets 2024-09-16T04:07:57.969 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.949+0000 7fc6deffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:07:57.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.949+0000 7fc6deffd640 1 -- 172.21.15.42:0/219249532 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc6d80052d0 con 0x7fc6e806db90 2024-09-16T04:07:57.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.949+0000 7fc6ec907640 5 monclient: authenticate success, global_id 1014230 2024-09-16T04:07:57.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.949+0000 7fc6ec907640 10 monclient: _renew_subs 2024-09-16T04:07:57.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.949+0000 7fc6ec907640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:07:57.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.949+0000 7fc6ec907640 1 -- 172.21.15.42:0/219249532 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6e806f6a0 con 0x7fc6e806db90 2024-09-16T04:07:57.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.949+0000 7fc6deffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:07:57.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.949+0000 7fc6deffd640 1 -- 172.21.15.42:0/219249532 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6d8003840 con 0x7fc6e806db90 2024-09-16T04:07:57.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.949+0000 7fc6deffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:57.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.949+0000 7fc6deffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:07:57.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.949+0000 7fc6deffd640 10 monclient: dump: 2024-09-16T04:07:57.970 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:07:57.970 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:57.970 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:57.970 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:57.971 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:57.971 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:57.971 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:07:57.971 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:07:57.971 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:07:57.971 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:57.971 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.949+0000 7fc6ec907640 10 client.1014230.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:07:57.971 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.949+0000 7fc6ec907640 10 monclient: _renew_subs 2024-09-16T04:07:57.971 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.949+0000 7fc6ec907640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:07:57.971 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.949+0000 7fc6ec907640 1 -- 172.21.15.42:0/219249532 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6e80a34c0 con 0x7fc6e806db90 2024-09-16T04:07:57.971 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.949+0000 7fc6e559a640 4 set_mon_vals no callback set 2024-09-16T04:07:57.971 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.949+0000 7fc6ec907640 1 librados: init done 2024-09-16T04:07:57.971 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.949+0000 7fc6ec907640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:07:57.971 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.949+0000 7fc6ec907640 1 --2- 172.21.15.42:0/219249532 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc6e8136720 0x7fc6e80a1820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:57.971 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.949+0000 7fc6ec907640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:57.972 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.949+0000 7fc6ec907640 1 -- 172.21.15.42:0/219249532 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fc6e8136b00 con 0x7fc6e8136720 2024-09-16T04:07:57.972 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.949+0000 7fc6dffff640 1 --2- 172.21.15.42:0/219249532 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc6e8136720 0x7fc6e80a1820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:57.972 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.949+0000 7fc6dffff640 10 monclient: get_auth_request con 0x7fc6e8136720 auth_method 0 2024-09-16T04:07:57.972 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.949+0000 7fc6dffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:57.972 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.949+0000 7fc6dffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:57.972 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.949+0000 7fc6dffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:57.972 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.949+0000 7fc6dffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:07:57.972 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.949+0000 7fc6dffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:07:57.972 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.949+0000 7fc6dffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:07:57.972 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.949+0000 7fc6deffd640 1 -- 172.21.15.42:0/219249532 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fc6d801a070 con 0x7fc6e806db90 2024-09-16T04:07:57.972 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.949+0000 7fc6dffff640 10 monclient(hunting): handle_auth_done global_id 1014233 payload 824 2024-09-16T04:07:57.972 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.949+0000 7fc6dffff640 1 -- 172.21.15.42:0/219249532 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc6e80a37f0 con 0x7fc6e8136720 2024-09-16T04:07:57.972 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.949+0000 7fc6deffd640 1 --2- 172.21.15.42:0/219249532 >> v2:172.21.15.154:6812/254845342 conn(0x7fc6c8037810 0x7fc6c8039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:57.973 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.949+0000 7fc6deffd640 1 -- 172.21.15.42:0/219249532 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(706..706 src has 451..706) ==== 5040+0+0 (secure 0 0 0) 0x7fc6d8047f10 con 0x7fc6e806db90 2024-09-16T04:07:57.973 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.949+0000 7fc6deffd640 10 client.1014230.objecter ms_dispatch 0x7fc6e8000ba0 osd_map(706..706 src has 451..706) 2024-09-16T04:07:57.973 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.949+0000 7fc6deffd640 3 client.1014230.objecter handle_osd_map got epochs [706,706] > 0 2024-09-16T04:07:57.973 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.949+0000 7fc6deffd640 3 client.1014230.objecter handle_osd_map decoding full epoch 706 2024-09-16T04:07:57.973 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.949+0000 7fc6deffd640 20 client.1014230.objecter dump_active .. 0 homeless 2024-09-16T04:07:57.973 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.949+0000 7fc6deffd640 10 client.1014230.objecter ms_handle_connect 0x7fc6e8136720 2024-09-16T04:07:57.973 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.949+0000 7fc6deffd640 10 client.1014230.objecter resend_mon_ops 2024-09-16T04:07:57.973 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.949+0000 7fc6dffff640 1 --2- 172.21.15.42:0/219249532 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc6e8136720 0x7fc6e80a1820 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fc6e80a76f0 tx=0x7fc6d00191c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:07:57.973 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.949+0000 7fc6deffd640 1 -- 172.21.15.42:0/219249532 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6d00abd20 con 0x7fc6e8136720 2024-09-16T04:07:57.973 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.949+0000 7fc6deffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:57.973 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.949+0000 7fc6deffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:07:57.973 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.949+0000 7fc6deffd640 10 monclient: dump: 2024-09-16T04:07:57.973 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:07:57.974 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:57.974 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:57.974 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:57.974 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:57.974 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:57.974 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:07:57.974 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:07:57.974 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:07:57.974 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:57.974 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.949+0000 7fc6e6d9d640 1 --2- 172.21.15.42:0/219249532 >> v2:172.21.15.154:6812/254845342 conn(0x7fc6c8037810 0x7fc6c8039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:57.974 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.949+0000 7fc6e6d9d640 10 monclient: get_auth_request con 0x7fc6c8037810 auth_method 0 2024-09-16T04:07:57.974 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.949+0000 7fc6deffd640 1 -- 172.21.15.42:0/219249532 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fc6d00b0040 con 0x7fc6e8136720 2024-09-16T04:07:57.974 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.949+0000 7fc6deffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:07:57.974 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.949+0000 7fc6deffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:07:57.975 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.949+0000 7fc6deffd640 1 -- 172.21.15.42:0/219249532 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc6e8136720 msgr2=0x7fc6e80a1820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:57.975 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.949+0000 7fc6deffd640 1 --2- 172.21.15.42:0/219249532 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc6e8136720 0x7fc6e80a1820 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fc6e80a76f0 tx=0x7fc6d00191c0 comp rx=0 tx=0).stop 2024-09-16T04:07:57.975 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.949+0000 7fc6deffd640 1 -- 172.21.15.42:0/219249532 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc6e8136720 msgr2=0x7fc6e80a1820 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:07:57.975 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.949+0000 7fc6deffd640 1 --2- 172.21.15.42:0/219249532 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc6e8136720 0x7fc6e80a1820 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:57.975 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.953+0000 7fc6e6d9d640 1 --2- 172.21.15.42:0/219249532 >> v2:172.21.15.154:6812/254845342 conn(0x7fc6c8037810 0x7fc6c8039cd0 secure :-1 s=READY pgs=3895 cs=0 l=1 rev1=1 crypto rx=0x7fc6e8131430 tx=0x7fc6e000b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:07:57.975 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.953+0000 7fc6deffd640 10 client.1014230.objecter ms_handle_connect 0x7fc6c8037810 2024-09-16T04:07:57.976 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.957+0000 7fc6ec907640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:07:57.976 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.957+0000 7fc6ec907640 1 --2- 172.21.15.42:0/219249532 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc6e80a1d60 0x7fc6e80a2140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:57.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.957+0000 7fc6ec907640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:57.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.957+0000 7fc6ec907640 1 -- 172.21.15.42:0/219249532 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fc6e80a4110 con 0x7fc6e80a1d60 2024-09-16T04:07:57.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.957+0000 7fc6dffff640 1 --2- 172.21.15.42:0/219249532 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc6e80a1d60 0x7fc6e80a2140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:57.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.957+0000 7fc6dffff640 10 monclient: get_auth_request con 0x7fc6e80a1d60 auth_method 0 2024-09-16T04:07:57.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.957+0000 7fc6dffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:57.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.957+0000 7fc6dffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:57.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.957+0000 7fc6dffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:57.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.957+0000 7fc6dffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:07:57.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.957+0000 7fc6dffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:07:57.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.957+0000 7fc6dffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:07:57.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.957+0000 7fc6dffff640 10 monclient(hunting): handle_auth_done global_id 1014236 payload 824 2024-09-16T04:07:57.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.957+0000 7fc6dffff640 1 -- 172.21.15.42:0/219249532 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fc6e80a2680 con 0x7fc6e80a1d60 2024-09-16T04:07:57.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.957+0000 7fc6deffd640 10 client.1014230.objecter ms_handle_connect 0x7fc6e80a1d60 2024-09-16T04:07:57.978 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.957+0000 7fc6deffd640 10 client.1014230.objecter resend_mon_ops 2024-09-16T04:07:57.978 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.957+0000 7fc6dffff640 1 --2- 172.21.15.42:0/219249532 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc6e80a1d60 0x7fc6e80a2140 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fc6d001a050 tx=0x7fc6d0009240 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:07:57.978 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.957+0000 7fc6deffd640 1 -- 172.21.15.42:0/219249532 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6d0003ce0 con 0x7fc6e80a1d60 2024-09-16T04:07:57.978 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.957+0000 7fc6deffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:57.978 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.957+0000 7fc6deffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:07:57.978 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.957+0000 7fc6deffd640 10 monclient: dump: 2024-09-16T04:07:57.978 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:07:57.978 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:57.978 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:57.978 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:57.978 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:57.978 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:57.978 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:07:57.978 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:07:57.979 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:07:57.979 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:57.979 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.957+0000 7fc6deffd640 1 -- 172.21.15.42:0/219249532 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4712 (secure 0 0 0) 0x7fc6d0003ce0 con 0x7fc6e80a1d60 2024-09-16T04:07:57.979 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.957+0000 7fc6deffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:07:57.979 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.957+0000 7fc6deffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:07:57.979 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.957+0000 7fc6deffd640 1 -- 172.21.15.42:0/219249532 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc6e80a1d60 msgr2=0x7fc6e80a2140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:57.979 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.957+0000 7fc6deffd640 1 --2- 172.21.15.42:0/219249532 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc6e80a1d60 0x7fc6e80a2140 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fc6d001a050 tx=0x7fc6d0009240 comp rx=0 tx=0).stop 2024-09-16T04:07:57.979 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.957+0000 7fc6deffd640 1 -- 172.21.15.42:0/219249532 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc6e80a1d60 msgr2=0x7fc6e80a2140 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:07:57.979 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.957+0000 7fc6deffd640 1 --2- 172.21.15.42:0/219249532 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc6e80a1d60 0x7fc6e80a2140 secure :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fc6d001a050 tx=0x7fc6d0009240 comp rx=0 tx=0).stop 2024-09-16T04:07:57.979 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T04:07:57.980 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:07:57.980 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:07:57.980 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T04:07:57.980 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 238, 2024-09-16T04:07:57.980 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 73159, 2024-09-16T04:07:57.980 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T04:07:57.980 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T04:07:57.980 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T04:07:57.980 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T04:07:57.980 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:07:57.980 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 3, 2024-09-16T04:07:57.980 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:07:57.980 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T04:07:57.981 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T04:07:57.981 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:07:57.981 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:07:57.981 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:07:57.981 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:07:57.981 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:07:57.981 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:07:57.981 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:07:57.981 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:07:57.981 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:07:57.981 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:07:57.981 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:07:57.981 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:07:57.981 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:07:57.981 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T04:07:57.982 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:07:57.982 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:07:57.982 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:07:57.982 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:07:57.982 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:07:57.982 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:07:57.982 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:07:57.982 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:07:57.982 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:07:57.982 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:07:57.982 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:07:57.982 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:07:57.982 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:57.983 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T04:07:57.983 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T04:07:57.983 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T04:07:57.983 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T04:07:57.983 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T04:07:57.983 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T04:07:57.983 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:07:57.983 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:07:57.983 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T04:07:57.983 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T04:07:57.983 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T04:07:57.983 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T04:07:57.983 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T04:07:57.983 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T04:07:57.984 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T04:07:57.984 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:07:57.984 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T04:07:57.984 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:07:57.984 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:07:57.984 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:07:57.984 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:07:57.984 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:07:57.984 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:07:57.984 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:07:57.984 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:07:57.984 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:07:57.984 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:07:57.984 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:07:57.985 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:07:57.985 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T04:07:57.985 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:57.985 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T04:07:57.985 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:57.985 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:07:57.985 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:07:57.985 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:07:57.985 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:07:57.985 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:57.985 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:07:57.985 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T04:07:57.985 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:07:57.986 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:57.986 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:57.986 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:07:57.986 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T04:07:57.986 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:07:57.986 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:07:57.986 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:07:57.986 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:57.986 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T04:07:57.986 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T04:07:57.986 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:07:57.986 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:07:57.986 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:07:57.986 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:57.987 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:57.987 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:07:57.987 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:07:57.987 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:07:57.987 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:07:57.987 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:57.987 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:07:57.987 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T04:07:57.987 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:07:57.987 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:57.987 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:57.987 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:07:57.987 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T04:07:57.987 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:07:57.988 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:07:57.988 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:07:57.988 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:57.988 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T04:07:57.988 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T04:07:57.988 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:07:57.988 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:07:57.988 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:07:57.988 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:57.988 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:57.988 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:07:57.988 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:07:57.988 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:07:57.988 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:07:57.989 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:57.989 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:07:57.989 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T04:07:57.989 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:07:57.989 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:57.989 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:57.989 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:07:57.989 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T04:07:57.989 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:07:57.989 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:07:57.989 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:07:57.989 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:57.989 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T04:07:57.990 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T04:07:57.990 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:07:57.990 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:07:57.990 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:07:57.990 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:07:57.990 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:07:57.990 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:57.990 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T04:07:57.990 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T04:07:57.990 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:57.991 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:07:57.991 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:07:57.991 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:07:57.991 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:07:57.991 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:07:57.991 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T04:07:57.991 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:57.991 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:07:57.991 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:07:57.991 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 3 2024-09-16T04:07:57.991 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:07:57.991 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:07:57.991 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T04:07:57.991 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:57.992 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:07:57.992 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:07:57.992 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 3 2024-09-16T04:07:57.992 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:07:57.992 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:07:57.992 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:57.992 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T04:07:57.992 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T04:07:57.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.957+0000 7fc6ec907640 10 librados: watch_flush enter 2024-09-16T04:07:57.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.957+0000 7fc6ec907640 10 librados: watch_flush exit 2024-09-16T04:07:57.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.957+0000 7fc6ec907640 20 client.1014230.objecter shutdown clearing up homeless session... 2024-09-16T04:07:57.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.957+0000 7fc6ec907640 10 client.1014230.objecter successfully canceled tick 2024-09-16T04:07:57.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.957+0000 7fc6ec907640 1 -- 172.21.15.42:0/219249532 >> v2:172.21.15.154:6812/254845342 conn(0x7fc6c8037810 msgr2=0x7fc6c8039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:57.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.957+0000 7fc6ec907640 1 --2- 172.21.15.42:0/219249532 >> v2:172.21.15.154:6812/254845342 conn(0x7fc6c8037810 0x7fc6c8039cd0 secure :-1 s=READY pgs=3895 cs=0 l=1 rev1=1 crypto rx=0x7fc6e8131430 tx=0x7fc6e000b040 comp rx=0 tx=0).stop 2024-09-16T04:07:57.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.957+0000 7fc6ec907640 10 monclient: shutdown 2024-09-16T04:07:57.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.957+0000 7fc6ec907640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:07:57.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.957+0000 7fc6ec907640 1 -- 172.21.15.42:0/219249532 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc6e806db90 msgr2=0x7fc6e81ceed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:57.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.957+0000 7fc6e6d9d640 1 -- 172.21.15.42:0/219249532 reap_dead start 2024-09-16T04:07:57.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.957+0000 7fc6ec907640 1 --2- 172.21.15.42:0/219249532 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc6e806db90 0x7fc6e81ceed0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fc6d800c3f0 tx=0x7fc6d800c8b0 comp rx=0 tx=0).stop 2024-09-16T04:07:57.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.957+0000 7fc6ec907640 1 -- 172.21.15.42:0/219249532 shutdown_connections 2024-09-16T04:07:57.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.957+0000 7fc6ec907640 1 --2- 172.21.15.42:0/219249532 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc6e806db90 0x7fc6e81ceed0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:57.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.957+0000 7fc6ec907640 1 -- 172.21.15.42:0/219249532 >> 172.21.15.42:0/219249532 conn(0x7fc6e80a9fb0 msgr2=0x7fc6e80a79f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:07:57.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.957+0000 7fc6ec907640 1 -- 172.21.15.42:0/219249532 shutdown_connections 2024-09-16T04:07:57.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.957+0000 7fc6ec907640 1 -- 172.21.15.42:0/219249532 wait complete. 2024-09-16T04:07:57.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:57.957+0000 7fc6ec907640 1 librados: shutdown 2024-09-16T04:07:57.995 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:07:57.995 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T04:07:57.995 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T04:07:58.117 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.113+0000 7f691e0b4640 10 monclient: get_monmap_and_config 2024-09-16T04:07:58.117 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.113+0000 7f691e0b4640 10 monclient: build_initial_monmap 2024-09-16T04:07:58.118 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.113+0000 7f691e0b4640 10 monclient: monmap: 2024-09-16T04:07:58.118 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:07:58.118 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:58.118 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:07:58.116188+0000 2024-09-16T04:07:58.118 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:07:58.116188+0000 2024-09-16T04:07:58.118 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:07:58.118 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:58.118 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:07:58.118 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:07:58.118 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:07:58.118 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:07:58.118 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:07:58.119 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:58.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.113+0000 7f691e0b4640 1 Processor -- start 2024-09-16T04:07:58.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.113+0000 7f691e0b4640 1 -- start start 2024-09-16T04:07:58.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.113+0000 7f691e0b4640 10 monclient: init 2024-09-16T04:07:58.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.113+0000 7f691e0b4640 10 monclient: _reopen_session rank -1 2024-09-16T04:07:58.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.113+0000 7f691e0b4640 10 monclient: _add_conns ranks=[3,1,0,2,4] 2024-09-16T04:07:58.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.113+0000 7f691e0b4640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f69180a39b0 0x7f6918133b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:58.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.113+0000 7f691e0b4640 10 monclient(hunting): picked mon.noname-a con 0x7f69180a39b0 addr v2:172.21.15.42:3300/0 2024-09-16T04:07:58.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.113+0000 7f691e0b4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f69180a3d90 addr v1:172.21.15.154:6789/0 2024-09-16T04:07:58.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.113+0000 7f691e0b4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6918137710 addr v1:172.21.15.42:6789/0 2024-09-16T04:07:58.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.113+0000 7f691e0b4640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f691809c140 con 0x7f6918137710 2024-09-16T04:07:58.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.113+0000 7f691e0b4640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f691809c420 con 0x7f69180a3d90 2024-09-16T04:07:58.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.113+0000 7f691e0b4640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:58.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.113+0000 7f691e0b4640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f691806dd60 con 0x7f69180a39b0 2024-09-16T04:07:58.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.113+0000 7f691e0b4640 10 monclient(hunting): _renew_subs 2024-09-16T04:07:58.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.113+0000 7f691e0b4640 10 monclient(hunting): authenticate will time out at 2525.085496s 2024-09-16T04:07:58.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.113+0000 7f69177fe640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f69180a39b0 0x7f6918133b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:58.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.113+0000 7f6917fff640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f6918137710 0x7f691813f3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:44596/0 (socket says 172.21.15.42:44596) 2024-09-16T04:07:58.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.113+0000 7f6917fff640 1 -- 172.21.15.42:0/3100254899 learned_addr learned my addr 172.21.15.42:0/3100254899 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T04:07:58.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.113+0000 7f69177fe640 10 monclient(hunting): get_auth_request con 0x7f69180a39b0 auth_method 0 2024-09-16T04:07:58.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.113+0000 7f69177fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:58.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.113+0000 7f69177fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:58.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.113+0000 7f69177fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:58.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.113+0000 7f69177fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:07:58.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.113+0000 7f69177fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:07:58.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.113+0000 7f69177fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:07:58.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.113+0000 7f69177fe640 10 monclient(hunting): handle_auth_done global_id 1024167 payload 274 2024-09-16T04:07:58.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.113+0000 7f69177fe640 1 -- 172.21.15.42:0/3100254899 >> v1:172.21.15.154:6789/0 conn(0x7f69180a3d90 legacy=0x7f6918137000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:07:58.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.113+0000 7f69177fe640 1 -- 172.21.15.42:0/3100254899 >> v1:172.21.15.42:6789/0 conn(0x7f6918137710 legacy=0x7f691813f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:58.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.113+0000 7f69177fe640 10 monclient: _finish_hunting 0 2024-09-16T04:07:58.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.113+0000 7f69177fe640 1 monclient: found mon.noname-a 2024-09-16T04:07:58.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.113+0000 7f69167fc640 1 -- 172.21.15.42:0/3100254899 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4106378591 0 0) 0x7f691809c140 con 0x7f6918137710 2024-09-16T04:07:58.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.113+0000 7f69177fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:07:58.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.113+0000 7f69177fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:07:58.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.113+0000 7f69177fe640 1 -- 172.21.15.42:0/3100254899 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69180a5e20 con 0x7f69180a39b0 2024-09-16T04:07:58.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.113+0000 7f69167fc640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T04:07:58.131 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.117+0000 7f69177fe640 1 --2- 172.21.15.42:0/3100254899 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f69180a39b0 0x7f6918133b80 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f6908009d90 tx=0x7f690801c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a785b71cd95d3cc3 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:07:58.131 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.117+0000 7f69167fc640 1 -- 172.21.15.42:0/3100254899 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69080a66c0 con 0x7f69180a39b0 2024-09-16T04:07:58.131 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.117+0000 7f69167fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:58.131 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.117+0000 7f69167fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:07:58.131 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.117+0000 7f69167fc640 10 monclient: dump: 2024-09-16T04:07:58.132 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:07:58.132 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:58.132 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:58.132 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:58.132 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:58.132 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:58.132 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:07:58.132 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:07:58.132 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:07:58.133 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:58.133 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.117+0000 7f69167fc640 10 monclient: _finish_auth 0 2024-09-16T04:07:58.133 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.117+0000 7f69167fc640 10 monclient: _check_auth_tickets 2024-09-16T04:07:58.133 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.117+0000 7f69167fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:07:58.133 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.117+0000 7f691e0b4640 5 monclient: authenticate success, global_id 1024167 2024-09-16T04:07:58.133 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.117+0000 7f691e0b4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:07:58.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.117+0000 7f69167fc640 1 -- 172.21.15.42:0/3100254899 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f69080a6860 con 0x7f69180a39b0 2024-09-16T04:07:58.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.117+0000 7f69167fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:07:58.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.117+0000 7f69167fc640 1 -- 172.21.15.42:0/3100254899 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69080a6c30 con 0x7f69180a39b0 2024-09-16T04:07:58.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.117+0000 7f691e0b4640 10 monclient: get_monmap_and_config success 2024-09-16T04:07:58.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.117+0000 7f691e0b4640 4 set_mon_vals no callback set 2024-09-16T04:07:58.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.117+0000 7f69167fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:58.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.117+0000 7f69167fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:07:58.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.117+0000 7f69167fc640 10 monclient: dump: 2024-09-16T04:07:58.134 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:07:58.135 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:58.135 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:58.135 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:58.135 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:58.135 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:58.135 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:07:58.135 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:07:58.135 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:07:58.135 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:58.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.117+0000 7f691e0b4640 10 monclient: shutdown 2024-09-16T04:07:58.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.117+0000 7f691e0b4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:07:58.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.117+0000 7f691e0b4640 1 -- 172.21.15.42:0/3100254899 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f69180a39b0 msgr2=0x7f6918133b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:58.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.117+0000 7f691e0b4640 1 --2- 172.21.15.42:0/3100254899 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f69180a39b0 0x7f6918133b80 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f6908009d90 tx=0x7f690801c7f0 comp rx=0 tx=0).stop 2024-09-16T04:07:58.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.117+0000 7f691e0b4640 1 -- 172.21.15.42:0/3100254899 shutdown_connections 2024-09-16T04:07:58.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.117+0000 7f691e0b4640 1 --2- 172.21.15.42:0/3100254899 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f69180a39b0 0x7f6918133b80 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:58.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.117+0000 7f691e0b4640 1 -- 172.21.15.42:0/3100254899 >> 172.21.15.42:0/3100254899 conn(0x7f69180a9fb0 msgr2=0x7f69180aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:07:58.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.117+0000 7f691e0b4640 1 -- 172.21.15.42:0/3100254899 shutdown_connections 2024-09-16T04:07:58.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.117+0000 7f691e0b4640 1 -- 172.21.15.42:0/3100254899 wait complete. 2024-09-16T04:07:58.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.117+0000 7f691e0b4640 10 monclient: build_initial_monmap 2024-09-16T04:07:58.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.117+0000 7f691e0b4640 10 monclient: monmap: 2024-09-16T04:07:58.136 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:07:58.136 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:07:58.136 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:07:58.136 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:07:58.136 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:07:58.136 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:58.137 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:07:58.137 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:07:58.137 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:07:58.137 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:58.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.117+0000 7f691e0b4640 1 librados: starting msgr at 2024-09-16T04:07:58.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.117+0000 7f691e0b4640 1 librados: starting objecter 2024-09-16T04:07:58.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.117+0000 7f691e0b4640 1 Processor -- start 2024-09-16T04:07:58.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.117+0000 7f691e0b4640 1 -- start start 2024-09-16T04:07:58.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.117+0000 7f691e0b4640 1 librados: setting wanted keys 2024-09-16T04:07:58.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.117+0000 7f691e0b4640 1 librados: calling monclient init 2024-09-16T04:07:58.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.117+0000 7f691e0b4640 10 monclient: init 2024-09-16T04:07:58.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.117+0000 7f691e0b4640 10 monclient: _reopen_session rank -1 2024-09-16T04:07:58.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.117+0000 7f691e0b4640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:07:58.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.117+0000 7f691e0b4640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f69180a39b0 0x7f6918130660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:58.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.117+0000 7f691e0b4640 10 monclient(hunting): picked mon.noname-a con 0x7f69180a39b0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:07:58.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.117+0000 7f691e0b4640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f69180a3d90 0x7f6918130ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:58.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.117+0000 7f691e0b4640 10 monclient(hunting): picked mon.noname-b con 0x7f69180a3d90 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:07:58.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.117+0000 7f691e0b4640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f6918137710 0x7f69180a1660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:58.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.117+0000 7f691e0b4640 10 monclient(hunting): picked mon.noname-c con 0x7f6918137710 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:07:58.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.117+0000 7f691e0b4640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:58.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.117+0000 7f691e0b4640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f6918143900 con 0x7f69180a39b0 2024-09-16T04:07:58.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.117+0000 7f691e0b4640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:58.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.117+0000 7f691e0b4640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f691806dee0 con 0x7f69180a3d90 2024-09-16T04:07:58.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.117+0000 7f691e0b4640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:58.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.117+0000 7f691e0b4640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f6918143a80 con 0x7f6918137710 2024-09-16T04:07:58.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.117+0000 7f691e0b4640 10 monclient(hunting): _renew_subs 2024-09-16T04:07:58.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.117+0000 7f691e0b4640 10 monclient(hunting): authenticate will time out at 2525.091731s 2024-09-16T04:07:58.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.117+0000 7f69177fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f69180a39b0 0x7f6918130660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:58.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.117+0000 7f6917fff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f6918137710 0x7f69180a1660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:58.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.117+0000 7f69177fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f69180a39b0 0x7f6918130660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:43560/0 (socket says 172.21.15.42:43560) 2024-09-16T04:07:58.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.117+0000 7f69177fe640 1 -- 172.21.15.42:0/2328737812 learned_addr learned my addr 172.21.15.42:0/2328737812 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:07:58.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.117+0000 7f69177fe640 10 monclient(hunting): get_auth_request con 0x7f69180a39b0 auth_method 0 2024-09-16T04:07:58.140 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.117+0000 7f69177fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:58.140 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.117+0000 7f69177fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:58.140 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.117+0000 7f69177fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:58.140 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.121+0000 7f6917fff640 10 monclient(hunting): get_auth_request con 0x7f6918137710 auth_method 0 2024-09-16T04:07:58.140 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.121+0000 7f6917fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:58.140 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.121+0000 7f6917fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:58.140 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.121+0000 7f6917fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:58.140 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.121+0000 7f69177fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:07:58.140 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.121+0000 7f69177fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:07:58.140 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.121+0000 7f69177fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:07:58.140 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.121+0000 7f6916ffd640 1 --2- 172.21.15.42:0/2328737812 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f69180a3d90 0x7f6918130ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:58.140 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.121+0000 7f6916ffd640 10 monclient(hunting): get_auth_request con 0x7f69180a3d90 auth_method 0 2024-09-16T04:07:58.140 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.121+0000 7f6916ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:58.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.121+0000 7f6916ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:58.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.121+0000 7f6916ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:58.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.121+0000 7f6917fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:07:58.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.121+0000 7f6917fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:07:58.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.121+0000 7f6917fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:07:58.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.121+0000 7f69177fe640 10 monclient(hunting): handle_auth_done global_id 1024173 payload 824 2024-09-16T04:07:58.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.121+0000 7f69177fe640 1 -- 172.21.15.42:0/2328737812 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f6918137710 msgr2=0x7f69180a1660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:58.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.121+0000 7f69177fe640 1 --2- 172.21.15.42:0/2328737812 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f6918137710 0x7f69180a1660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:58.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.121+0000 7f69177fe640 1 -- 172.21.15.42:0/2328737812 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f69180a3d90 msgr2=0x7f6918130ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:58.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.121+0000 7f69177fe640 1 --2- 172.21.15.42:0/2328737812 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f69180a3d90 0x7f6918130ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:58.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.121+0000 7f69177fe640 10 monclient: _finish_hunting 0 2024-09-16T04:07:58.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.121+0000 7f69177fe640 1 monclient: found mon.noname-a 2024-09-16T04:07:58.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.121+0000 7f69177fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:07:58.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.121+0000 7f69177fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:07:58.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.121+0000 7f69177fe640 1 -- 172.21.15.42:0/2328737812 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6918133920 con 0x7f69180a39b0 2024-09-16T04:07:58.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.121+0000 7f6916ffd640 1 --2- 172.21.15.42:0/2328737812 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f69180a3d90 0x7f6918130ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T04:07:58.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.121+0000 7f69177fe640 1 --2- 172.21.15.42:0/2328737812 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f69180a39b0 0x7f6918130660 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f6908009d90 tx=0x7f6908019590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:07:58.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.121+0000 7f690ffff640 10 client.?.objecter ms_handle_connect 0x7f69180a39b0 2024-09-16T04:07:58.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.121+0000 7f690ffff640 10 client.?.objecter resend_mon_ops 2024-09-16T04:07:58.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.121+0000 7f690ffff640 1 -- 172.21.15.42:0/2328737812 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69080be070 con 0x7f69180a39b0 2024-09-16T04:07:58.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.121+0000 7f690ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:58.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.121+0000 7f690ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:07:58.142 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.121+0000 7f690ffff640 10 monclient: dump: 2024-09-16T04:07:58.142 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:07:58.142 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:58.142 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:58.142 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:58.143 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:58.143 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:58.143 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:07:58.143 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:07:58.143 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:07:58.143 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:58.143 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.121+0000 7f690ffff640 10 monclient: _finish_auth 0 2024-09-16T04:07:58.143 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.121+0000 7f690ffff640 10 monclient: _check_auth_tickets 2024-09-16T04:07:58.143 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.121+0000 7f690ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:07:58.143 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.121+0000 7f690ffff640 1 -- 172.21.15.42:0/2328737812 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f69080ae9f0 con 0x7f69180a39b0 2024-09-16T04:07:58.143 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.121+0000 7f690ffff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:07:58.143 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.121+0000 7f690ffff640 1 -- 172.21.15.42:0/2328737812 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69080aeda0 con 0x7f69180a39b0 2024-09-16T04:07:58.143 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.121+0000 7f691e0b4640 5 monclient: authenticate success, global_id 1024173 2024-09-16T04:07:58.143 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.121+0000 7f691e0b4640 10 monclient: _renew_subs 2024-09-16T04:07:58.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.121+0000 7f691e0b4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:07:58.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.121+0000 7f691e0b4640 1 -- 172.21.15.42:0/2328737812 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69181397c0 con 0x7f69180a39b0 2024-09-16T04:07:58.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.121+0000 7f690ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:58.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.121+0000 7f6915ffb640 4 set_mon_vals no callback set 2024-09-16T04:07:58.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.121+0000 7f690ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:07:58.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.121+0000 7f690ffff640 10 monclient: dump: 2024-09-16T04:07:58.144 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:07:58.144 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:58.145 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:58.145 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:58.145 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:58.145 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:58.145 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:07:58.145 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:07:58.145 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:07:58.145 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:58.145 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.121+0000 7f691e0b4640 10 client.1024173.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:07:58.145 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.121+0000 7f691e0b4640 10 monclient: _renew_subs 2024-09-16T04:07:58.145 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.121+0000 7f691e0b4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:07:58.145 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.121+0000 7f691e0b4640 1 -- 172.21.15.42:0/2328737812 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69181405d0 con 0x7f69180a39b0 2024-09-16T04:07:58.145 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.121+0000 7f691e0b4640 1 librados: init done 2024-09-16T04:07:58.145 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.121+0000 7f690ffff640 1 -- 172.21.15.42:0/2328737812 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f69080b8430 con 0x7f69180a39b0 2024-09-16T04:07:58.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.121+0000 7f690ffff640 1 --2- 172.21.15.42:0/2328737812 >> v2:172.21.15.154:6812/254845342 conn(0x7f68f8037840 0x7f68f8039d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:58.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.121+0000 7f690ffff640 1 -- 172.21.15.42:0/2328737812 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(706..706 src has 451..706) ==== 5040+0+0 (secure 0 0 0) 0x7f69080e42a0 con 0x7f69180a39b0 2024-09-16T04:07:58.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.121+0000 7f690ffff640 10 client.1024173.objecter ms_dispatch 0x7f6918000ba0 osd_map(706..706 src has 451..706) 2024-09-16T04:07:58.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.121+0000 7f690ffff640 3 client.1024173.objecter handle_osd_map got epochs [706,706] > 0 2024-09-16T04:07:58.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.121+0000 7f690ffff640 3 client.1024173.objecter handle_osd_map decoding full epoch 706 2024-09-16T04:07:58.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.121+0000 7f690ffff640 20 client.1024173.objecter dump_active .. 0 homeless 2024-09-16T04:07:58.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.121+0000 7f691e0b4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:07:58.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.121+0000 7f691e0b4640 1 --2- 172.21.15.42:0/2328737812 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6918139a00 0x7f69180a1ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:58.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.121+0000 7f691e0b4640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:58.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.121+0000 7f691e0b4640 1 -- 172.21.15.42:0/2328737812 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f691806f680 con 0x7f6918139a00 2024-09-16T04:07:58.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.121+0000 7f6916ffd640 1 --2- 172.21.15.42:0/2328737812 >> v2:172.21.15.154:6812/254845342 conn(0x7f68f8037840 0x7f68f8039d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:58.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.121+0000 7f6916ffd640 10 monclient: get_auth_request con 0x7f68f8037840 auth_method 0 2024-09-16T04:07:58.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.121+0000 7f6917fff640 1 --2- 172.21.15.42:0/2328737812 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6918139a00 0x7f69180a1ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:58.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.121+0000 7f6917fff640 10 monclient: get_auth_request con 0x7f6918139a00 auth_method 0 2024-09-16T04:07:58.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.121+0000 7f6917fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:58.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.121+0000 7f6917fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:58.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.121+0000 7f6917fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:58.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.121+0000 7f6917fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:07:58.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.121+0000 7f6917fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:07:58.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.121+0000 7f6917fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:07:58.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.121+0000 7f6917fff640 10 monclient(hunting): handle_auth_done global_id 905146 payload 824 2024-09-16T04:07:58.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.121+0000 7f6917fff640 1 -- 172.21.15.42:0/2328737812 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6918133b80 con 0x7f6918139a00 2024-09-16T04:07:58.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.125+0000 7f690ffff640 10 client.1024173.objecter ms_handle_connect 0x7f68f8037840 2024-09-16T04:07:58.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.125+0000 7f6916ffd640 1 --2- 172.21.15.42:0/2328737812 >> v2:172.21.15.154:6812/254845342 conn(0x7f68f8037840 0x7f68f8039d00 secure :-1 s=READY pgs=3896 cs=0 l=1 rev1=1 crypto rx=0x7f69180a2f00 tx=0x7f6900009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:07:58.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.125+0000 7f690ffff640 10 client.1024173.objecter ms_handle_connect 0x7f6918139a00 2024-09-16T04:07:58.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.125+0000 7f690ffff640 10 client.1024173.objecter resend_mon_ops 2024-09-16T04:07:58.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.125+0000 7f6917fff640 1 --2- 172.21.15.42:0/2328737812 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6918139a00 0x7f69180a1ba0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f6910000c10 tx=0x7f6910001980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:07:58.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.125+0000 7f690ffff640 1 -- 172.21.15.42:0/2328737812 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f691000f5e0 con 0x7f6918139a00 2024-09-16T04:07:58.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.125+0000 7f690ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:58.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.125+0000 7f690ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:07:58.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.125+0000 7f690ffff640 10 monclient: dump: 2024-09-16T04:07:58.152 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:07:58.152 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:58.152 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:58.152 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:58.152 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:58.152 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:58.152 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:07:58.152 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:07:58.152 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:07:58.152 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:58.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.125+0000 7f690ffff640 1 -- 172.21.15.42:0/2328737812 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f6910018070 con 0x7f6918139a00 2024-09-16T04:07:58.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.125+0000 7f690ffff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:07:58.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.125+0000 7f690ffff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:07:58.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.125+0000 7f690ffff640 1 -- 172.21.15.42:0/2328737812 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6918139a00 msgr2=0x7f69180a1ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:58.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.125+0000 7f690ffff640 1 --2- 172.21.15.42:0/2328737812 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6918139a00 0x7f69180a1ba0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f6910000c10 tx=0x7f6910001980 comp rx=0 tx=0).stop 2024-09-16T04:07:58.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.125+0000 7f690ffff640 1 -- 172.21.15.42:0/2328737812 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6918139a00 msgr2=0x7f69180a1ba0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:07:58.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.125+0000 7f690ffff640 1 --2- 172.21.15.42:0/2328737812 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6918139a00 0x7f69180a1ba0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:07:58.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.129+0000 7f691e0b4640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:07:58.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.129+0000 7f691e0b4640 1 --2- 172.21.15.42:0/2328737812 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f69180a20e0 0x7f69180a24f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:07:58.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.129+0000 7f691e0b4640 10 monclient(hunting): start opening mon connection 2024-09-16T04:07:58.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.129+0000 7f691e0b4640 1 -- 172.21.15.42:0/2328737812 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f69180a2a30 con 0x7f69180a20e0 2024-09-16T04:07:58.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.133+0000 7f6917fff640 1 --2- 172.21.15.42:0/2328737812 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f69180a20e0 0x7f69180a24f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:07:58.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.133+0000 7f6917fff640 10 monclient: get_auth_request con 0x7f69180a20e0 auth_method 0 2024-09-16T04:07:58.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.133+0000 7f6917fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:07:58.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.133+0000 7f6917fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:07:58.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.133+0000 7f6917fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:07:58.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.133+0000 7f6917fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:07:58.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.133+0000 7f6917fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:07:58.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.133+0000 7f6917fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:07:58.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.133+0000 7f6917fff640 10 monclient(hunting): handle_auth_done global_id 905149 payload 824 2024-09-16T04:07:58.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.133+0000 7f6917fff640 1 -- 172.21.15.42:0/2328737812 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f6918139e40 con 0x7f69180a20e0 2024-09-16T04:07:58.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.133+0000 7f6917fff640 1 --2- 172.21.15.42:0/2328737812 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f69180a20e0 0x7f69180a24f0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f6910017000 tx=0x7f6910000ee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:07:58.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.133+0000 7f690ffff640 10 client.1024173.objecter ms_handle_connect 0x7f69180a20e0 2024-09-16T04:07:58.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.133+0000 7f690ffff640 10 client.1024173.objecter resend_mon_ops 2024-09-16T04:07:58.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.133+0000 7f690ffff640 1 -- 172.21.15.42:0/2328737812 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6910027410 con 0x7f69180a20e0 2024-09-16T04:07:58.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.133+0000 7f690ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:07:58.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.133+0000 7f690ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:07:58.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.133+0000 7f690ffff640 10 monclient: dump: 2024-09-16T04:07:58.157 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:07:58.157 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:07:58.157 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:58.157 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:07:58.157 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:07:58.157 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:07:58.157 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:07:58.157 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:07:58.157 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:07:58.157 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:07:58.157 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.133+0000 7f690ffff640 1 -- 172.21.15.42:0/2328737812 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4880 (secure 0 0 0) 0x7f69100133f0 con 0x7f69180a20e0 2024-09-16T04:07:58.157 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.133+0000 7f690ffff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:07:58.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.133+0000 7f690ffff640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:07:58.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.133+0000 7f690ffff640 1 -- 172.21.15.42:0/2328737812 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f69180a20e0 msgr2=0x7f69180a24f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:58.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.133+0000 7f690ffff640 1 --2- 172.21.15.42:0/2328737812 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f69180a20e0 0x7f69180a24f0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f6910017000 tx=0x7f6910000ee0 comp rx=0 tx=0).stop 2024-09-16T04:07:58.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.133+0000 7f690ffff640 1 -- 172.21.15.42:0/2328737812 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f69180a20e0 msgr2=0x7f69180a24f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:07:58.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.133+0000 7f690ffff640 1 --2- 172.21.15.42:0/2328737812 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f69180a20e0 0x7f69180a24f0 secure :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f6910017000 tx=0x7f6910000ee0 comp rx=0 tx=0).stop 2024-09-16T04:07:58.158 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T04:07:58.158 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:07:58.158 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:07:58.158 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T04:07:58.159 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 238, 2024-09-16T04:07:58.159 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 606313, 2024-09-16T04:07:58.159 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T04:07:58.159 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T04:07:58.159 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T04:07:58.159 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T04:07:58.159 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:07:58.159 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 3, 2024-09-16T04:07:58.159 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:07:58.159 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T04:07:58.159 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T04:07:58.159 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:07:58.159 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:07:58.160 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:07:58.160 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:07:58.160 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:07:58.160 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:07:58.160 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:07:58.160 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:07:58.160 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:07:58.160 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:07:58.160 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:07:58.160 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:07:58.160 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:07:58.160 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T04:07:58.160 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:07:58.161 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:07:58.161 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:07:58.161 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:07:58.161 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:07:58.161 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:07:58.161 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:07:58.161 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:07:58.161 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:07:58.161 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:07:58.161 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:07:58.161 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:07:58.161 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:58.161 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T04:07:58.161 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T04:07:58.162 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T04:07:58.162 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T04:07:58.162 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T04:07:58.162 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T04:07:58.162 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:07:58.162 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:07:58.162 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T04:07:58.162 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T04:07:58.162 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T04:07:58.162 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T04:07:58.162 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T04:07:58.162 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T04:07:58.162 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T04:07:58.162 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:07:58.163 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T04:07:58.163 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:07:58.163 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:07:58.163 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:07:58.163 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:07:58.163 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:07:58.163 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:07:58.163 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:07:58.163 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:07:58.163 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:07:58.163 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:07:58.163 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:07:58.163 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:07:58.163 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T04:07:58.164 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:58.164 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T04:07:58.164 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:58.164 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:07:58.164 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:07:58.164 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:07:58.164 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:07:58.164 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:58.164 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:07:58.164 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T04:07:58.164 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:07:58.164 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:58.164 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:58.164 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:07:58.165 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T04:07:58.165 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:07:58.165 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:07:58.165 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:07:58.165 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:58.165 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T04:07:58.165 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T04:07:58.165 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:07:58.165 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:07:58.165 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:07:58.165 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:58.165 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:58.165 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:07:58.165 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:07:58.166 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:07:58.166 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:07:58.166 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:58.166 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:07:58.166 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T04:07:58.166 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:07:58.166 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:58.166 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:58.166 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:07:58.166 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T04:07:58.166 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:07:58.166 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:07:58.166 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:07:58.166 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:58.167 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T04:07:58.168 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T04:07:58.168 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:07:58.168 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:07:58.168 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:07:58.168 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:58.168 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:58.168 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:07:58.168 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:07:58.168 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:07:58.168 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:07:58.168 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:58.168 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:07:58.168 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T04:07:58.169 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:07:58.169 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:58.169 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:58.169 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:07:58.169 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T04:07:58.169 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:07:58.169 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:07:58.169 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:07:58.169 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:58.169 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T04:07:58.169 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T04:07:58.169 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:07:58.169 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:07:58.169 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:07:58.170 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:07:58.170 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:07:58.170 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:58.170 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T04:07:58.170 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T04:07:58.170 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:58.170 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:07:58.170 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:07:58.170 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:07:58.170 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:07:58.170 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:07:58.170 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T04:07:58.170 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:58.170 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:07:58.171 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:07:58.171 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 3 2024-09-16T04:07:58.171 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:07:58.171 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:07:58.171 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T04:07:58.171 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:58.171 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:07:58.171 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:07:58.171 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 5 2024-09-16T04:07:58.171 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:07:58.171 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:07:58.172 INFO:teuthology.orchestra.run.smithi042.stdout: "mgr": [ 2024-09-16T04:07:58.172 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:07:58.172 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:07:58.172 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:07:58.172 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:07:58.172 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:07:58.172 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:07:58.172 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:07:58.172 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T04:07:58.172 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T04:07:58.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.133+0000 7f691e0b4640 10 librados: watch_flush enter 2024-09-16T04:07:58.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.133+0000 7f691e0b4640 10 librados: watch_flush exit 2024-09-16T04:07:58.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.133+0000 7f691e0b4640 20 client.1024173.objecter shutdown clearing up homeless session... 2024-09-16T04:07:58.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.133+0000 7f691e0b4640 10 client.1024173.objecter successfully canceled tick 2024-09-16T04:07:58.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.133+0000 7f691e0b4640 1 -- 172.21.15.42:0/2328737812 >> v2:172.21.15.154:6812/254845342 conn(0x7f68f8037840 msgr2=0x7f68f8039d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:58.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.133+0000 7f691e0b4640 1 --2- 172.21.15.42:0/2328737812 >> v2:172.21.15.154:6812/254845342 conn(0x7f68f8037840 0x7f68f8039d00 secure :-1 s=READY pgs=3896 cs=0 l=1 rev1=1 crypto rx=0x7f69180a2f00 tx=0x7f6900009040 comp rx=0 tx=0).stop 2024-09-16T04:07:58.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.133+0000 7f691e0b4640 10 monclient: shutdown 2024-09-16T04:07:58.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.133+0000 7f691e0b4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:07:58.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.133+0000 7f691e0b4640 1 -- 172.21.15.42:0/2328737812 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f69180a39b0 msgr2=0x7f6918130660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:07:58.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.133+0000 7f691e0b4640 1 --2- 172.21.15.42:0/2328737812 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f69180a39b0 0x7f6918130660 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f6908009d90 tx=0x7f6908019590 comp rx=0 tx=0).stop 2024-09-16T04:07:58.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.133+0000 7f6917fff640 1 -- 172.21.15.42:0/2328737812 reap_dead start 2024-09-16T04:07:58.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.133+0000 7f6917fff640 1 -- 172.21.15.42:0/2328737812 reap_dead start 2024-09-16T04:07:58.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.133+0000 7f691e0b4640 1 -- 172.21.15.42:0/2328737812 shutdown_connections 2024-09-16T04:07:58.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.133+0000 7f691e0b4640 1 -- 172.21.15.42:0/2328737812 >> 172.21.15.42:0/2328737812 conn(0x7f69180a9fb0 msgr2=0x7f6918131cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:07:58.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.133+0000 7f691e0b4640 1 -- 172.21.15.42:0/2328737812 shutdown_connections 2024-09-16T04:07:58.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.133+0000 7f691e0b4640 1 -- 172.21.15.42:0/2328737812 wait complete. 2024-09-16T04:07:58.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:07:58.133+0000 7f691e0b4640 1 librados: shutdown 2024-09-16T04:07:58.176 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:07:58.176 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-09-16T04:07:58.176 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['b'] 2024-09-16T04:07:58.176 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-09-16T04:07:58.176 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-16T04:07:58.176 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-16T04:07:58.176 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T04:07:58.188 INFO:tasks.ceph.mon.a:Stopped 2024-09-16T04:07:58.189 INFO:tasks.mon_thrash:Sending STOP to mon b 2024-09-16T04:07:58.189 INFO:tasks.ceph.mon.b:Sent signal 19 2024-09-16T04:07:58.189 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-16T04:08:13.190 INFO:tasks.mon_thrash:Sending CONT to mon b 2024-09-16T04:08:13.191 INFO:tasks.ceph.mon.b:Sent signal 18 2024-09-16T04:08:13.191 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T04:08:13.191 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:08:13.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:13.316+0000 7f163cac3640 10 monclient: get_monmap_and_config 2024-09-16T04:08:13.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:13.316+0000 7f163cac3640 10 monclient: build_initial_monmap 2024-09-16T04:08:13.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:13.316+0000 7f163cac3640 10 monclient: monmap: 2024-09-16T04:08:13.324 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:08:13.325 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:08:13.325 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:08:13.323032+0000 2024-09-16T04:08:13.325 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:08:13.323032+0000 2024-09-16T04:08:13.325 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:13.325 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:08:13.325 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:08:13.325 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:08:13.325 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:08:13.325 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:08:13.325 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:08:13.325 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:08:13.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:13.316+0000 7f163cac3640 1 Processor -- start 2024-09-16T04:08:13.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:13.316+0000 7f163cac3640 1 -- start start 2024-09-16T04:08:13.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:13.316+0000 7f163cac3640 10 monclient: init 2024-09-16T04:08:13.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:13.320+0000 7f163cac3640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:13.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:13.320+0000 7f163cac3640 10 monclient: _add_conns ranks=[3,0,1,2,4] 2024-09-16T04:08:13.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:13.320+0000 7f163cac3640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f16380a1330 0x7f163809f180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:08:13.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:13.320+0000 7f163cac3640 10 monclient(hunting): picked mon.noname-a con 0x7f16380a1330 addr v2:172.21.15.42:3300/0 2024-09-16T04:08:13.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:13.320+0000 7f163cac3640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f16380a1710 addr v1:172.21.15.42:6789/0 2024-09-16T04:08:13.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:13.320+0000 7f163cac3640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f16381393a0 addr v1:172.21.15.154:6789/0 2024-09-16T04:08:13.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:13.320+0000 7f163cac3640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f163809e4c0 con 0x7f16380a1710 2024-09-16T04:08:13.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:13.320+0000 7f163cac3640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f163809f6c0 con 0x7f16381393a0 2024-09-16T04:08:13.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:13.320+0000 7f163cac3640 10 monclient(hunting): start opening mon connection 2024-09-16T04:08:13.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:13.320+0000 7f163cac3640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f163809edf0 con 0x7f16380a1330 2024-09-16T04:08:13.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:13.320+0000 7f163cac3640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:13.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:13.320+0000 7f163cac3640 10 monclient(hunting): authenticate will time out at 2540.292456s 2024-09-16T04:08:13.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:13.320+0000 7f1635d9b640 1 -- >> v1:172.21.15.42:6789/0 conn(0x7f16380a1710 legacy=0x7f16380a1b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.42:6789/0 2024-09-16T04:08:13.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:13.320+0000 7f163659c640 1 -- >> v2:172.21.15.42:3300/0 conn(0x7f16380a1330 msgr2=0x7f163809f180 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3300/0 2024-09-16T04:08:13.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:13.320+0000 7f163659c640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f16380a1330 0x7f163809f180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:08:13.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:13.320+0000 7f1636d9d640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f16381393a0 0x7f163813b790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.42:59060/0 (socket says 172.21.15.42:59060) 2024-09-16T04:08:13.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:13.320+0000 7f1636d9d640 1 -- 172.21.15.42:0/1295500634 learned_addr learned my addr 172.21.15.42:0/1295500634 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T04:08:13.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:13.524+0000 7f1635d9b640 1 -- 172.21.15.42:0/1295500634 >> v1:172.21.15.42:6789/0 conn(0x7f16380a1710 legacy=0x7f16380a1b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.42:6789/0 2024-09-16T04:08:13.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:13.524+0000 7f163659c640 1 -- 172.21.15.42:0/1295500634 >> v2:172.21.15.42:3300/0 conn(0x7f16380a1330 msgr2=0x7f163809f180 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3300/0 2024-09-16T04:08:13.527 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:13.524+0000 7f163659c640 1 --2- 172.21.15.42:0/1295500634 >> v2:172.21.15.42:3300/0 conn(0x7f16380a1330 0x7f163809f180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T04:08:13.933 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:13.928+0000 7f1635d9b640 1 -- 172.21.15.42:0/1295500634 >> v1:172.21.15.42:6789/0 conn(0x7f16380a1710 legacy=0x7f16380a1b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.42:6789/0 2024-09-16T04:08:13.933 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:13.928+0000 7f163659c640 1 -- 172.21.15.42:0/1295500634 >> v2:172.21.15.42:3300/0 conn(0x7f16380a1330 msgr2=0x7f163809f180 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3300/0 2024-09-16T04:08:13.933 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:13.928+0000 7f163659c640 1 --2- 172.21.15.42:0/1295500634 >> v2:172.21.15.42:3300/0 conn(0x7f16380a1330 0x7f163809f180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T04:08:14.737 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:14.732+0000 7f1635d9b640 1 -- 172.21.15.42:0/1295500634 >> v1:172.21.15.42:6789/0 conn(0x7f16380a1710 legacy=0x7f16380a1b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.42:6789/0 2024-09-16T04:08:14.738 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:14.732+0000 7f163659c640 1 -- 172.21.15.42:0/1295500634 >> v2:172.21.15.42:3300/0 conn(0x7f16380a1330 msgr2=0x7f163809f180 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3300/0 2024-09-16T04:08:14.738 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:14.732+0000 7f163659c640 1 --2- 172.21.15.42:0/1295500634 >> v2:172.21.15.42:3300/0 conn(0x7f16380a1330 0x7f163809f180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T04:08:16.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:16.320+0000 7f1627fff640 10 monclient(hunting): tick 2024-09-16T04:08:16.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:16.320+0000 7f1627fff640 10 monclient(hunting): _check_auth_tickets 2024-09-16T04:08:16.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:16.320+0000 7f1627fff640 1 monclient(hunting): continuing hunt 2024-09-16T04:08:16.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:16.320+0000 7f1627fff640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T04:08:16.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:16.320+0000 7f1627fff640 1 -- 172.21.15.42:0/1295500634 >> v2:172.21.15.42:3300/0 conn(0x7f16380a1330 msgr2=0x7f163809f180 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:08:16.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:16.320+0000 7f1627fff640 1 --2- 172.21.15.42:0/1295500634 >> v2:172.21.15.42:3300/0 conn(0x7f16380a1330 0x7f163809f180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:08:16.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:16.320+0000 7f1627fff640 1 -- 172.21.15.42:0/1295500634 >> v1:172.21.15.154:6789/0 conn(0x7f16381393a0 legacy=0x7f163813b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:16.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:16.320+0000 7f1627fff640 1 -- 172.21.15.42:0/1295500634 >> v1:172.21.15.42:6789/0 conn(0x7f16380a1710 legacy=0x7f16380a1b40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:08:16.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:16.320+0000 7f1627fff640 10 monclient: _add_conns ranks=[2,4] 2024-09-16T04:08:16.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:16.320+0000 7f1627fff640 1 --2- 172.21.15.42:0/1295500634 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f16180036d0 0x7f1618005b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:08:16.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:16.320+0000 7f1627fff640 10 monclient(hunting): picked mon.noname-b con 0x7f16180036d0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:08:16.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:16.320+0000 7f1627fff640 1 --2- 172.21.15.42:0/1295500634 >> v2:172.21.15.154:3300/0 conn(0x7f1618006250 0x7f1618008710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:08:16.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:16.320+0000 7f1627fff640 10 monclient(hunting): picked mon.noname-c con 0x7f1618006250 addr v2:172.21.15.154:3300/0 2024-09-16T04:08:16.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:16.320+0000 7f1627fff640 10 monclient(hunting): start opening mon connection 2024-09-16T04:08:16.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:16.320+0000 7f1627fff640 1 -- 172.21.15.42:0/1295500634 --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f163809edf0 con 0x7f1618006250 2024-09-16T04:08:16.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:16.320+0000 7f1627fff640 10 monclient(hunting): start opening mon connection 2024-09-16T04:08:16.329 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:16.320+0000 7f1627fff640 1 -- 172.21.15.42:0/1295500634 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f1618008dd0 con 0x7f16180036d0 2024-09-16T04:08:16.329 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:16.320+0000 7f1627fff640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:16.329 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:16.320+0000 7f163659c640 1 --2- 172.21.15.42:0/1295500634 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f16180036d0 0x7f1618005b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:08:16.329 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:16.320+0000 7f163659c640 10 monclient(hunting): get_auth_request con 0x7f16180036d0 auth_method 0 2024-09-16T04:08:16.329 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:16.320+0000 7f163659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:08:16.329 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:16.320+0000 7f163659c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:16.329 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:16.320+0000 7f163659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:16.329 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:16.320+0000 7f1635d9b640 1 --2- 172.21.15.42:0/1295500634 >> v2:172.21.15.154:3300/0 conn(0x7f1618006250 0x7f1618008710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:08:16.329 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:16.320+0000 7f1635d9b640 10 monclient(hunting): get_auth_request con 0x7f1618006250 auth_method 0 2024-09-16T04:08:16.329 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:16.320+0000 7f1635d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:08:16.329 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:16.320+0000 7f1635d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:16.329 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:16.320+0000 7f1635d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:16.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:16.320+0000 7f163659c640 1 -- 172.21.15.42:0/1295500634 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f16180036d0 msgr2=0x7f1618005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:08:16.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:16.320+0000 7f163659c640 1 -- 172.21.15.42:0/1295500634 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f16180036d0 msgr2=0x7f1618005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:08:16.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:16.320+0000 7f163659c640 1 --2- 172.21.15.42:0/1295500634 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f16180036d0 0x7f1618005b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:08:16.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:16.320+0000 7f163659c640 1 --2- 172.21.15.42:0/1295500634 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f16180036d0 0x7f1618005b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:08:16.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:16.320+0000 7f1635d9b640 1 -- 172.21.15.42:0/1295500634 >> v2:172.21.15.154:3300/0 conn(0x7f1618006250 msgr2=0x7f1618008710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:08:16.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:16.320+0000 7f1635d9b640 1 -- 172.21.15.42:0/1295500634 >> v2:172.21.15.154:3300/0 conn(0x7f1618006250 msgr2=0x7f1618008710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:08:16.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:16.320+0000 7f1635d9b640 1 --2- 172.21.15.42:0/1295500634 >> v2:172.21.15.154:3300/0 conn(0x7f1618006250 0x7f1618008710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:08:16.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:16.320+0000 7f1635d9b640 1 --2- 172.21.15.42:0/1295500634 >> v2:172.21.15.154:3300/0 conn(0x7f1618006250 0x7f1618008710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:08:16.531 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:16.524+0000 7f163659c640 1 --2- 172.21.15.42:0/1295500634 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f16180036d0 0x7f1618005b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:08:16.531 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:16.524+0000 7f163659c640 10 monclient(hunting): get_auth_request con 0x7f16180036d0 auth_method 0 2024-09-16T04:08:16.531 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:16.524+0000 7f163659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:08:16.531 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:16.524+0000 7f163659c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:16.531 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:16.524+0000 7f163659c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:08:16.531 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:16.524+0000 7f1635d9b640 1 --2- 172.21.15.42:0/1295500634 >> v2:172.21.15.154:3300/0 conn(0x7f1618006250 0x7f1618008710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:08:16.531 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:16.524+0000 7f1635d9b640 10 monclient(hunting): get_auth_request con 0x7f1618006250 auth_method 0 2024-09-16T04:08:16.531 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:16.524+0000 7f163659c640 1 -- 172.21.15.42:0/1295500634 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f16180036d0 msgr2=0x7f1618005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:08:16.531 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:16.524+0000 7f1635d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:08:16.532 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:16.524+0000 7f1635d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:16.532 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:16.524+0000 7f163659c640 1 -- 172.21.15.42:0/1295500634 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f16180036d0 msgr2=0x7f1618005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:08:16.532 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:16.524+0000 7f163659c640 1 --2- 172.21.15.42:0/1295500634 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f16180036d0 0x7f1618005b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:08:16.532 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:16.524+0000 7f1635d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:08:16.532 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:16.524+0000 7f163659c640 1 --2- 172.21.15.42:0/1295500634 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f16180036d0 0x7f1618005b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T04:08:16.532 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:16.524+0000 7f1635d9b640 1 -- 172.21.15.42:0/1295500634 >> v2:172.21.15.154:3300/0 conn(0x7f1618006250 msgr2=0x7f1618008710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:08:16.532 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:16.524+0000 7f1635d9b640 1 -- 172.21.15.42:0/1295500634 >> v2:172.21.15.154:3300/0 conn(0x7f1618006250 msgr2=0x7f1618008710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:08:16.532 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:16.524+0000 7f1635d9b640 1 --2- 172.21.15.42:0/1295500634 >> v2:172.21.15.154:3300/0 conn(0x7f1618006250 0x7f1618008710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:08:16.532 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:16.524+0000 7f1635d9b640 1 --2- 172.21.15.42:0/1295500634 >> v2:172.21.15.154:3300/0 conn(0x7f1618006250 0x7f1618008710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T04:08:16.934 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:16.928+0000 7f163659c640 1 --2- 172.21.15.42:0/1295500634 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f16180036d0 0x7f1618005b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:08:16.934 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:16.928+0000 7f163659c640 10 monclient(hunting): get_auth_request con 0x7f16180036d0 auth_method 0 2024-09-16T04:08:16.934 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:16.928+0000 7f163659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:08:16.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:16.928+0000 7f163659c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:16.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:16.928+0000 7f163659c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:08:16.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:16.928+0000 7f1635d9b640 1 --2- 172.21.15.42:0/1295500634 >> v2:172.21.15.154:3300/0 conn(0x7f1618006250 0x7f1618008710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:08:16.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:16.928+0000 7f1635d9b640 10 monclient(hunting): get_auth_request con 0x7f1618006250 auth_method 0 2024-09-16T04:08:16.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:16.928+0000 7f1635d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:08:16.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:16.928+0000 7f1635d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:16.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:16.928+0000 7f1635d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:08:16.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:16.928+0000 7f163659c640 1 -- 172.21.15.42:0/1295500634 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f16180036d0 msgr2=0x7f1618005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:08:16.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:16.928+0000 7f163659c640 1 -- 172.21.15.42:0/1295500634 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f16180036d0 msgr2=0x7f1618005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:08:16.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:16.928+0000 7f163659c640 1 --2- 172.21.15.42:0/1295500634 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f16180036d0 0x7f1618005b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:08:16.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:16.928+0000 7f163659c640 1 --2- 172.21.15.42:0/1295500634 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f16180036d0 0x7f1618005b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T04:08:16.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:16.928+0000 7f1635d9b640 1 -- 172.21.15.42:0/1295500634 >> v2:172.21.15.154:3300/0 conn(0x7f1618006250 msgr2=0x7f1618008710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:08:16.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:16.928+0000 7f1635d9b640 1 -- 172.21.15.42:0/1295500634 >> v2:172.21.15.154:3300/0 conn(0x7f1618006250 msgr2=0x7f1618008710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:08:16.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:16.928+0000 7f1635d9b640 1 --2- 172.21.15.42:0/1295500634 >> v2:172.21.15.154:3300/0 conn(0x7f1618006250 0x7f1618008710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:08:16.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:16.928+0000 7f1635d9b640 1 --2- 172.21.15.42:0/1295500634 >> v2:172.21.15.154:3300/0 conn(0x7f1618006250 0x7f1618008710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T04:08:17.738 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:17.732+0000 7f163659c640 1 --2- 172.21.15.42:0/1295500634 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f16180036d0 0x7f1618005b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:08:17.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:17.732+0000 7f163659c640 10 monclient(hunting): get_auth_request con 0x7f16180036d0 auth_method 0 2024-09-16T04:08:17.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:17.732+0000 7f163659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:08:17.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:17.732+0000 7f163659c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:17.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:17.732+0000 7f163659c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:08:17.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:17.732+0000 7f1635d9b640 1 --2- 172.21.15.42:0/1295500634 >> v2:172.21.15.154:3300/0 conn(0x7f1618006250 0x7f1618008710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:08:17.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:17.732+0000 7f1635d9b640 10 monclient(hunting): get_auth_request con 0x7f1618006250 auth_method 0 2024-09-16T04:08:17.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:17.732+0000 7f1635d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:08:17.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:17.732+0000 7f1635d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:17.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:17.732+0000 7f1635d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:08:17.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:17.732+0000 7f163659c640 1 -- 172.21.15.42:0/1295500634 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f16180036d0 msgr2=0x7f1618005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:08:17.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:17.732+0000 7f163659c640 1 -- 172.21.15.42:0/1295500634 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f16180036d0 msgr2=0x7f1618005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:08:17.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:17.732+0000 7f163659c640 1 --2- 172.21.15.42:0/1295500634 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f16180036d0 0x7f1618005b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:08:17.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:17.732+0000 7f163659c640 1 --2- 172.21.15.42:0/1295500634 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f16180036d0 0x7f1618005b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T04:08:17.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:17.732+0000 7f1635d9b640 1 -- 172.21.15.42:0/1295500634 >> v2:172.21.15.154:3300/0 conn(0x7f1618006250 msgr2=0x7f1618008710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:08:17.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:17.732+0000 7f1635d9b640 1 -- 172.21.15.42:0/1295500634 >> v2:172.21.15.154:3300/0 conn(0x7f1618006250 msgr2=0x7f1618008710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:08:17.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:17.732+0000 7f1635d9b640 1 --2- 172.21.15.42:0/1295500634 >> v2:172.21.15.154:3300/0 conn(0x7f1618006250 0x7f1618008710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:08:17.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:17.732+0000 7f1635d9b640 1 --2- 172.21.15.42:0/1295500634 >> v2:172.21.15.154:3300/0 conn(0x7f1618006250 0x7f1618008710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T04:08:19.329 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.320+0000 7f1627fff640 10 monclient(hunting): tick 2024-09-16T04:08:19.329 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.320+0000 7f1627fff640 10 monclient(hunting): _check_auth_tickets 2024-09-16T04:08:19.329 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.320+0000 7f1627fff640 1 monclient(hunting): continuing hunt 2024-09-16T04:08:19.329 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.320+0000 7f1627fff640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T04:08:19.329 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.320+0000 7f1627fff640 1 -- 172.21.15.42:0/1295500634 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f16180036d0 msgr2=0x7f1618005b20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:08:19.329 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.320+0000 7f1627fff640 1 --2- 172.21.15.42:0/1295500634 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f16180036d0 0x7f1618005b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:08:19.329 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.320+0000 7f1627fff640 1 -- 172.21.15.42:0/1295500634 >> v2:172.21.15.154:3300/0 conn(0x7f1618006250 msgr2=0x7f1618008710 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:08:19.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.320+0000 7f1627fff640 1 --2- 172.21.15.42:0/1295500634 >> v2:172.21.15.154:3300/0 conn(0x7f1618006250 0x7f1618008710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:08:19.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.320+0000 7f1627fff640 10 monclient: _add_conns ranks=[3,4,2,1,0] 2024-09-16T04:08:19.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.320+0000 7f1636d9d640 1 -- 172.21.15.42:0/1295500634 reap_dead start 2024-09-16T04:08:19.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.320+0000 7f1627fff640 1 --2- 172.21.15.42:0/1295500634 >> v2:172.21.15.42:3300/0 conn(0x7f16380a1330 0x7f161800b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:08:19.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.320+0000 7f1627fff640 10 monclient(hunting): picked mon.noname-a con 0x7f16380a1330 addr v2:172.21.15.42:3300/0 2024-09-16T04:08:19.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.320+0000 7f1627fff640 1 --2- 172.21.15.42:0/1295500634 >> v2:172.21.15.154:3300/0 conn(0x7f1618006250 0x7f1618008710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:08:19.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.320+0000 7f1627fff640 10 monclient(hunting): picked mon.noname-c con 0x7f1618006250 addr v2:172.21.15.154:3300/0 2024-09-16T04:08:19.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.320+0000 7f1627fff640 1 --2- 172.21.15.42:0/1295500634 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f16180036d0 0x7f161800b720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:08:19.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.320+0000 7f1627fff640 10 monclient(hunting): picked mon.noname-b con 0x7f16180036d0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:08:19.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.320+0000 7f1627fff640 10 monclient(hunting): start opening mon connection 2024-09-16T04:08:19.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.320+0000 7f1627fff640 1 -- 172.21.15.42:0/1295500634 --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f163809edf0 con 0x7f16380a1330 2024-09-16T04:08:19.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.320+0000 7f1627fff640 10 monclient(hunting): start opening mon connection 2024-09-16T04:08:19.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.320+0000 7f1627fff640 1 -- 172.21.15.42:0/1295500634 --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f1618008dd0 con 0x7f1618006250 2024-09-16T04:08:19.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.320+0000 7f1627fff640 10 monclient(hunting): start opening mon connection 2024-09-16T04:08:19.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.320+0000 7f1627fff640 1 -- 172.21.15.42:0/1295500634 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f1618005b60 con 0x7f16180036d0 2024-09-16T04:08:19.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.320+0000 7f1627fff640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:19.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.320+0000 7f163659c640 1 -- 172.21.15.42:0/1295500634 >> v2:172.21.15.42:3300/0 conn(0x7f16380a1330 msgr2=0x7f161800b1e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3300/0 2024-09-16T04:08:19.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.320+0000 7f163659c640 1 --2- 172.21.15.42:0/1295500634 >> v2:172.21.15.42:3300/0 conn(0x7f16380a1330 0x7f161800b1e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:08:19.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.320+0000 7f1636d9d640 1 --2- 172.21.15.42:0/1295500634 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f16180036d0 0x7f161800b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:08:19.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.320+0000 7f1636d9d640 10 monclient(hunting): get_auth_request con 0x7f16180036d0 auth_method 0 2024-09-16T04:08:19.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.320+0000 7f1636d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:08:19.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.320+0000 7f1636d9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:19.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.320+0000 7f1636d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:19.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.320+0000 7f1635d9b640 1 --2- 172.21.15.42:0/1295500634 >> v2:172.21.15.154:3300/0 conn(0x7f1618006250 0x7f1618008710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:08:19.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.320+0000 7f1635d9b640 10 monclient(hunting): get_auth_request con 0x7f1618006250 auth_method 0 2024-09-16T04:08:19.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.320+0000 7f1635d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:08:19.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.320+0000 7f1635d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:19.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.320+0000 7f1635d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:19.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.320+0000 7f1636d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:08:19.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.320+0000 7f1636d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:08:19.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.320+0000 7f1636d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:08:19.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.320+0000 7f1635d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:08:19.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.320+0000 7f1635d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:08:19.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.320+0000 7f1635d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:08:19.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.320+0000 7f1636d9d640 10 monclient(hunting): handle_auth_done global_id 1014242 payload 274 2024-09-16T04:08:19.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.320+0000 7f1636d9d640 1 -- 172.21.15.42:0/1295500634 >> v2:172.21.15.154:3300/0 conn(0x7f1618006250 msgr2=0x7f1618008710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:08:19.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.320+0000 7f1636d9d640 1 --2- 172.21.15.42:0/1295500634 >> v2:172.21.15.154:3300/0 conn(0x7f1618006250 0x7f1618008710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:08:19.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.320+0000 7f1636d9d640 1 -- 172.21.15.42:0/1295500634 >> v2:172.21.15.42:3300/0 conn(0x7f16380a1330 msgr2=0x7f161800b1e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:08:19.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.320+0000 7f1636d9d640 1 --2- 172.21.15.42:0/1295500634 >> v2:172.21.15.42:3300/0 conn(0x7f16380a1330 0x7f161800b1e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:08:19.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.320+0000 7f1636d9d640 10 monclient: _finish_hunting 0 2024-09-16T04:08:19.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.320+0000 7f1636d9d640 1 monclient: found mon.noname-b 2024-09-16T04:08:19.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.320+0000 7f1636d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:19.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.320+0000 7f1636d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T04:08:19.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.320+0000 7f1636d9d640 1 -- 172.21.15.42:0/1295500634 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16380a3170 con 0x7f16180036d0 2024-09-16T04:08:19.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.320+0000 7f1635d9b640 1 --2- 172.21.15.42:0/1295500634 >> v2:172.21.15.154:3300/0 conn(0x7f1618006250 0x7f1618008710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:08:19.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.328+0000 7f1636d9d640 1 --2- 172.21.15.42:0/1295500634 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f16180036d0 0x7f161800b720 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f16380a1af0 tx=0x7f163001c9b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=96d8b5297f5b00c5 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:08:19.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.328+0000 7f163559a640 1 -- 172.21.15.42:0/1295500634 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16300a5720 con 0x7f16180036d0 2024-09-16T04:08:19.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.328+0000 7f163559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:19.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.328+0000 7f163559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:08:19.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.328+0000 7f163559a640 10 monclient: dump: 2024-09-16T04:08:19.335 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:08:19.335 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:08:19.335 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:08:19.336 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:08:19.336 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:19.336 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:08:19.336 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:08:19.336 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:08:19.336 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:08:19.336 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:08:19.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.328+0000 7f163559a640 10 monclient: _finish_auth 0 2024-09-16T04:08:19.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.328+0000 7f163559a640 10 monclient: _check_auth_tickets 2024-09-16T04:08:19.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.328+0000 7f163559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:19.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.328+0000 7f163559a640 1 -- 172.21.15.42:0/1295500634 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f16300a9070 con 0x7f16180036d0 2024-09-16T04:08:19.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.328+0000 7f163559a640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:19.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.328+0000 7f163559a640 1 -- 172.21.15.42:0/1295500634 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16300a5ac0 con 0x7f16180036d0 2024-09-16T04:08:19.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.328+0000 7f163cac3640 5 monclient: authenticate success, global_id 1014242 2024-09-16T04:08:19.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.328+0000 7f163cac3640 10 monclient: get_monmap_and_config success 2024-09-16T04:08:19.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.328+0000 7f163cac3640 4 set_mon_vals no callback set 2024-09-16T04:08:19.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.328+0000 7f163559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:19.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.328+0000 7f163559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:08:19.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.328+0000 7f163559a640 10 monclient: dump: 2024-09-16T04:08:19.337 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:08:19.337 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:08:19.337 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:08:19.337 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:08:19.338 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:19.338 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:08:19.338 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:08:19.338 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:08:19.338 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:08:19.338 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:08:19.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.328+0000 7f163cac3640 10 monclient: shutdown 2024-09-16T04:08:19.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.328+0000 7f163cac3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:19.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.328+0000 7f163cac3640 1 -- 172.21.15.42:0/1295500634 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f16180036d0 msgr2=0x7f161800b720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:19.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.328+0000 7f163cac3640 1 --2- 172.21.15.42:0/1295500634 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f16180036d0 0x7f161800b720 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f16380a1af0 tx=0x7f163001c9b0 comp rx=0 tx=0).stop 2024-09-16T04:08:19.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.328+0000 7f163cac3640 1 -- 172.21.15.42:0/1295500634 shutdown_connections 2024-09-16T04:08:19.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.328+0000 7f163cac3640 1 --2- 172.21.15.42:0/1295500634 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f16180036d0 0x7f161800b720 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:08:19.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.328+0000 7f163cac3640 1 --2- 172.21.15.42:0/1295500634 >> v2:172.21.15.154:3300/0 conn(0x7f1618006250 0x7f1618008710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:08:19.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.328+0000 7f163cac3640 1 --2- 172.21.15.42:0/1295500634 >> v2:172.21.15.42:3300/0 conn(0x7f16380a1330 0x7f161800b1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:08:19.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.328+0000 7f163cac3640 1 -- 172.21.15.42:0/1295500634 >> 172.21.15.42:0/1295500634 conn(0x7f16380a2420 msgr2=0x7f16380a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:19.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.328+0000 7f163cac3640 1 -- 172.21.15.42:0/1295500634 shutdown_connections 2024-09-16T04:08:19.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.328+0000 7f163cac3640 1 -- 172.21.15.42:0/1295500634 wait complete. 2024-09-16T04:08:19.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.328+0000 7f163cac3640 10 monclient: build_initial_monmap 2024-09-16T04:08:19.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.328+0000 7f163cac3640 10 monclient: monmap: 2024-09-16T04:08:19.345 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:08:19.345 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:08:19.345 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:08:19.345 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:08:19.346 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:19.346 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:08:19.346 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:08:19.346 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:08:19.346 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:08:19.346 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:08:19.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.328+0000 7f163cac3640 1 librados: starting msgr at 2024-09-16T04:08:19.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.328+0000 7f163cac3640 1 librados: starting objecter 2024-09-16T04:08:19.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.328+0000 7f163cac3640 1 Processor -- start 2024-09-16T04:08:19.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.332+0000 7f163cac3640 1 -- start start 2024-09-16T04:08:19.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.332+0000 7f163cac3640 1 librados: setting wanted keys 2024-09-16T04:08:19.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.332+0000 7f163cac3640 1 librados: calling monclient init 2024-09-16T04:08:19.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.332+0000 7f163cac3640 10 monclient: init 2024-09-16T04:08:19.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.332+0000 7f163cac3640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:19.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.332+0000 7f163cac3640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:08:19.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.332+0000 7f163cac3640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f16180036d0 0x7f1638133160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:08:19.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.332+0000 7f163cac3640 10 monclient(hunting): picked mon.noname-a con 0x7f16180036d0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:08:19.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.332+0000 7f163cac3640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1618006250 0x7f16381336a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:08:19.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.332+0000 7f163cac3640 10 monclient(hunting): picked mon.noname-b con 0x7f1618006250 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:08:19.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.332+0000 7f163cac3640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f16380a1330 0x7f163811ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:08:19.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.332+0000 7f163cac3640 10 monclient(hunting): picked mon.noname-c con 0x7f16380a1330 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:08:19.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.332+0000 7f163cac3640 10 monclient(hunting): start opening mon connection 2024-09-16T04:08:19.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.332+0000 7f163cac3640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f1638134c90 con 0x7f16180036d0 2024-09-16T04:08:19.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.332+0000 7f163cac3640 10 monclient(hunting): start opening mon connection 2024-09-16T04:08:19.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.332+0000 7f163cac3640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f1638134b10 con 0x7f1618006250 2024-09-16T04:08:19.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.332+0000 7f163cac3640 10 monclient(hunting): start opening mon connection 2024-09-16T04:08:19.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.332+0000 7f163cac3640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f1638134e10 con 0x7f16380a1330 2024-09-16T04:08:19.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.332+0000 7f163cac3640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:19.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.332+0000 7f163659c640 1 -- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f16180036d0 msgr2=0x7f1638133160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.42:3300/0 2024-09-16T04:08:19.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.332+0000 7f163cac3640 10 monclient(hunting): authenticate will time out at 2546.305055s 2024-09-16T04:08:19.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.332+0000 7f163659c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f16180036d0 0x7f1638133160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:08:19.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.332+0000 7f1636d9d640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f16380a1330 0x7f163811ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:08:19.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.332+0000 7f1636d9d640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f16380a1330 0x7f163811ebb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:49924/0 (socket says 172.21.15.42:49924) 2024-09-16T04:08:19.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.332+0000 7f1636d9d640 1 -- 172.21.15.42:0/861971155 learned_addr learned my addr 172.21.15.42:0/861971155 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:08:19.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.332+0000 7f1636d9d640 10 monclient(hunting): get_auth_request con 0x7f16380a1330 auth_method 0 2024-09-16T04:08:19.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.332+0000 7f1636d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:08:19.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.332+0000 7f1636d9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:19.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.332+0000 7f1636d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:19.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.332+0000 7f1636d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:08:19.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.332+0000 7f1636d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:08:19.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.332+0000 7f1636d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:08:19.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.332+0000 7f1635d9b640 1 --2- 172.21.15.42:0/861971155 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1618006250 0x7f16381336a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:08:19.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.332+0000 7f1635d9b640 10 monclient(hunting): get_auth_request con 0x7f1618006250 auth_method 0 2024-09-16T04:08:19.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.332+0000 7f1635d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:08:19.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.332+0000 7f1635d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:19.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.332+0000 7f1635d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:19.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.332+0000 7f1636d9d640 10 monclient(hunting): handle_auth_done global_id 1014245 payload 824 2024-09-16T04:08:19.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.332+0000 7f1636d9d640 1 -- 172.21.15.42:0/861971155 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1618006250 msgr2=0x7f16381336a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:19.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.332+0000 7f1636d9d640 1 --2- 172.21.15.42:0/861971155 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1618006250 0x7f16381336a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:08:19.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.332+0000 7f1636d9d640 1 -- 172.21.15.42:0/861971155 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f16180036d0 msgr2=0x7f1638133160 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:08:19.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.332+0000 7f1636d9d640 1 --2- 172.21.15.42:0/861971155 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f16180036d0 0x7f1638133160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:08:19.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.332+0000 7f1636d9d640 10 monclient: _finish_hunting 0 2024-09-16T04:08:19.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.332+0000 7f1636d9d640 1 monclient: found mon.noname-c 2024-09-16T04:08:19.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.332+0000 7f1636d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:19.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.332+0000 7f1636d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T04:08:19.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.332+0000 7f1636d9d640 1 -- 172.21.15.42:0/861971155 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16381be950 con 0x7f16380a1330 2024-09-16T04:08:19.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.332+0000 7f1636d9d640 1 --2- 172.21.15.42:0/861971155 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f16380a1330 0x7f163811ebb0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f1630000f90 tx=0x7f163001b1f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:08:19.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.332+0000 7f1626ffd640 10 client.?.objecter ms_handle_connect 0x7f16380a1330 2024-09-16T04:08:19.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.332+0000 7f1626ffd640 10 client.?.objecter resend_mon_ops 2024-09-16T04:08:19.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.332+0000 7f1626ffd640 1 -- 172.21.15.42:0/861971155 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1630001900 con 0x7f16380a1330 2024-09-16T04:08:19.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.332+0000 7f1626ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:19.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.332+0000 7f1626ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:08:19.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.332+0000 7f1626ffd640 10 monclient: dump: 2024-09-16T04:08:19.351 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:08:19.351 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:08:19.351 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:08:19.352 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:08:19.352 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:19.352 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:08:19.352 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:08:19.352 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:08:19.352 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:08:19.352 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:08:19.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.332+0000 7f1626ffd640 10 monclient: _finish_auth 0 2024-09-16T04:08:19.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.332+0000 7f1626ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:08:19.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.332+0000 7f1626ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:19.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.332+0000 7f1626ffd640 1 -- 172.21.15.42:0/861971155 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1630007110 con 0x7f16380a1330 2024-09-16T04:08:19.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.332+0000 7f1626ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:19.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.332+0000 7f1626ffd640 1 -- 172.21.15.42:0/861971155 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16300074c0 con 0x7f16380a1330 2024-09-16T04:08:19.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.332+0000 7f163cac3640 5 monclient: authenticate success, global_id 1014245 2024-09-16T04:08:19.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.332+0000 7f1626ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:19.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.332+0000 7f1626ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:08:19.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.332+0000 7f1626ffd640 10 monclient: dump: 2024-09-16T04:08:19.353 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:08:19.353 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:08:19.353 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:08:19.353 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:08:19.353 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:19.353 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:08:19.353 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:08:19.353 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:08:19.354 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:08:19.354 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:08:19.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.332+0000 7f163cac3640 10 monclient: _renew_subs 2024-09-16T04:08:19.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.332+0000 7f163cac3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:08:19.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.332+0000 7f163cac3640 1 -- 172.21.15.42:0/861971155 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1638122580 con 0x7f16380a1330 2024-09-16T04:08:19.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.332+0000 7f1634d99640 4 set_mon_vals no callback set 2024-09-16T04:08:19.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.332+0000 7f163cac3640 10 client.1014245.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:08:19.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.332+0000 7f163cac3640 10 monclient: _renew_subs 2024-09-16T04:08:19.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.332+0000 7f163cac3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:08:19.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.332+0000 7f163cac3640 1 -- 172.21.15.42:0/861971155 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f1638122860 con 0x7f16380a1330 2024-09-16T04:08:19.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.332+0000 7f163cac3640 1 librados: init done 2024-09-16T04:08:19.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.332+0000 7f1626ffd640 1 -- 172.21.15.42:0/861971155 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f1630007660 con 0x7f16380a1330 2024-09-16T04:08:19.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.332+0000 7f163cac3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:19.355 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.332+0000 7f163cac3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:19.355 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.332+0000 7f163cac3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:08:19.355 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.332+0000 7f163cac3640 1 -- 172.21.15.42:0/861971155 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f163809ba50 con 0x7f16380a1330 2024-09-16T04:08:19.355 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.332+0000 7f1626ffd640 1 --2- 172.21.15.42:0/861971155 >> v2:172.21.15.154:6812/254845342 conn(0x7f1618038820 0x7f161803ace0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:08:19.355 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.332+0000 7f1626ffd640 1 -- 172.21.15.42:0/861971155 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(706..706 src has 451..706) ==== 5040+0+0 (secure 0 0 0) 0x7f16300e2c90 con 0x7f16380a1330 2024-09-16T04:08:19.355 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.332+0000 7f1626ffd640 10 client.1014245.objecter ms_dispatch 0x7f1638000ba0 osd_map(706..706 src has 451..706) 2024-09-16T04:08:19.355 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.332+0000 7f1626ffd640 3 client.1014245.objecter handle_osd_map got epochs [706,706] > 0 2024-09-16T04:08:19.355 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.332+0000 7f1626ffd640 3 client.1014245.objecter handle_osd_map decoding full epoch 706 2024-09-16T04:08:19.355 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.332+0000 7f1626ffd640 20 client.1014245.objecter dump_active .. 0 homeless 2024-09-16T04:08:19.355 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.336+0000 7f163659c640 1 --2- 172.21.15.42:0/861971155 >> v2:172.21.15.154:6812/254845342 conn(0x7f1618038820 0x7f161803ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:08:19.355 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.336+0000 7f163659c640 10 monclient: get_auth_request con 0x7f1618038820 auth_method 0 2024-09-16T04:08:19.355 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.336+0000 7f163659c640 1 --2- 172.21.15.42:0/861971155 >> v2:172.21.15.154:6812/254845342 conn(0x7f1618038820 0x7f161803ace0 secure :-1 s=READY pgs=3897 cs=0 l=1 rev1=1 crypto rx=0x7f1628003990 tx=0x7f1628003eb0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:08:19.355 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.336+0000 7f1626ffd640 10 client.1014245.objecter ms_handle_connect 0x7f1618038820 2024-09-16T04:08:19.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.340+0000 7f1626ffd640 1 -- 172.21.15.42:0/861971155 <== mon.2 v2:172.21.15.42:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f16381be950 con 0x7f16380a1330 2024-09-16T04:08:19.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.340+0000 7f1626ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:19.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.340+0000 7f1626ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:08:19.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.572+0000 7f163cac3640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:08:19.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.572+0000 7f163cac3640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:08:19.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.572+0000 7f163cac3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:08:19.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.572+0000 7f163cac3640 1 -- 172.21.15.42:0/861971155 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f1638121130 con 0x7f16380a1330 2024-09-16T04:08:19.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.572+0000 7f1626ffd640 1 -- 172.21.15.42:0/861971155 <== mon.2 v2:172.21.15.42:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1555 (secure 0 0 0) 0x7f1630016730 con 0x7f16380a1330 2024-09-16T04:08:19.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.572+0000 7f1626ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:08:19.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.572+0000 7f1626ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:08:19.581 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":240,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":1,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:08:19.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.576+0000 7f163cac3640 10 librados: watch_flush enter 2024-09-16T04:08:19.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.576+0000 7f163cac3640 10 librados: watch_flush exit 2024-09-16T04:08:19.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.576+0000 7f163cac3640 20 client.1014245.objecter shutdown clearing up homeless session... 2024-09-16T04:08:19.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.576+0000 7f163cac3640 10 client.1014245.objecter successfully canceled tick 2024-09-16T04:08:19.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.576+0000 7f163cac3640 1 -- 172.21.15.42:0/861971155 >> v2:172.21.15.154:6812/254845342 conn(0x7f1618038820 msgr2=0x7f161803ace0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:19.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.576+0000 7f163cac3640 1 --2- 172.21.15.42:0/861971155 >> v2:172.21.15.154:6812/254845342 conn(0x7f1618038820 0x7f161803ace0 secure :-1 s=READY pgs=3897 cs=0 l=1 rev1=1 crypto rx=0x7f1628003990 tx=0x7f1628003eb0 comp rx=0 tx=0).stop 2024-09-16T04:08:19.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.576+0000 7f163cac3640 10 monclient: shutdown 2024-09-16T04:08:19.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.576+0000 7f163cac3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:19.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.576+0000 7f163cac3640 1 -- 172.21.15.42:0/861971155 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f16380a1330 msgr2=0x7f163811ebb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:19.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.576+0000 7f163cac3640 1 --2- 172.21.15.42:0/861971155 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f16380a1330 0x7f163811ebb0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f1630000f90 tx=0x7f163001b1f0 comp rx=0 tx=0).stop 2024-09-16T04:08:19.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.576+0000 7f163cac3640 1 -- 172.21.15.42:0/861971155 shutdown_connections 2024-09-16T04:08:19.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.576+0000 7f163cac3640 1 --2- 172.21.15.42:0/861971155 >> v2:172.21.15.154:6812/254845342 conn(0x7f1618038820 0x7f161803ace0 unknown :-1 s=CLOSED pgs=3897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:08:19.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.576+0000 7f163cac3640 1 --2- 172.21.15.42:0/861971155 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f16380a1330 0x7f163811ebb0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:08:19.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.576+0000 7f163cac3640 1 --2- 172.21.15.42:0/861971155 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1618006250 0x7f16381336a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:08:19.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.576+0000 7f163cac3640 1 --2- 172.21.15.42:0/861971155 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f16180036d0 0x7f1638133160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:08:19.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.576+0000 7f163cac3640 1 -- 172.21.15.42:0/861971155 >> 172.21.15.42:0/861971155 conn(0x7f16380a2420 msgr2=0x7f16380a3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:19.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.576+0000 7f163cac3640 1 -- 172.21.15.42:0/861971155 shutdown_connections 2024-09-16T04:08:19.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.576+0000 7f163cac3640 1 -- 172.21.15.42:0/861971155 wait complete. 2024-09-16T04:08:19.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.576+0000 7f163cac3640 1 librados: shutdown 2024-09-16T04:08:19.600 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T04:08:19.600 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-16T04:08:19.601 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T04:08:19.729 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.720+0000 7f5f746eb640 10 monclient: get_monmap_and_config 2024-09-16T04:08:19.729 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.720+0000 7f5f746eb640 10 monclient: build_initial_monmap 2024-09-16T04:08:19.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.724+0000 7f5f746eb640 10 monclient: monmap: 2024-09-16T04:08:19.730 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:08:19.730 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:08:19.730 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:08:19.728117+0000 2024-09-16T04:08:19.730 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:08:19.728117+0000 2024-09-16T04:08:19.730 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:19.730 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:08:19.730 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:08:19.731 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:08:19.731 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:08:19.731 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:08:19.731 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:08:19.731 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:08:19.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.724+0000 7f5f746eb640 1 Processor -- start 2024-09-16T04:08:19.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.724+0000 7f5f746eb640 1 -- start start 2024-09-16T04:08:19.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.724+0000 7f5f746eb640 10 monclient: init 2024-09-16T04:08:19.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.724+0000 7f5f746eb640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:19.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.724+0000 7f5f746eb640 10 monclient: _add_conns ranks=[3,2,4,1,0] 2024-09-16T04:08:19.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.724+0000 7f5f746eb640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f5f6c06db90 0x7f5f6c1340c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:08:19.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.724+0000 7f5f746eb640 10 monclient(hunting): picked mon.noname-a con 0x7f5f6c06db90 addr v2:172.21.15.42:3300/0 2024-09-16T04:08:19.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.724+0000 7f5f746eb640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5f6c0a37f0 0x7f5f6c0a3c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:08:19.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.724+0000 7f5f746eb640 10 monclient(hunting): picked mon.noname-b con 0x7f5f6c0a37f0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:08:19.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.724+0000 7f5f746eb640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f5f6c136e10 0x7f5f6c137240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:08:19.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.724+0000 7f5f746eb640 10 monclient(hunting): picked mon.noname-c con 0x7f5f6c136e10 addr v2:172.21.15.154:3300/0 2024-09-16T04:08:19.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.724+0000 7f5f746eb640 10 monclient(hunting): start opening mon connection 2024-09-16T04:08:19.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.724+0000 7f5f746eb640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f5f6c06df70 con 0x7f5f6c06db90 2024-09-16T04:08:19.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.724+0000 7f5f746eb640 10 monclient(hunting): start opening mon connection 2024-09-16T04:08:19.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.724+0000 7f5f746eb640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f5f6c06ece0 con 0x7f5f6c136e10 2024-09-16T04:08:19.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.724+0000 7f5f746eb640 10 monclient(hunting): start opening mon connection 2024-09-16T04:08:19.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.724+0000 7f5f746eb640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f5f6c09bf80 con 0x7f5f6c0a37f0 2024-09-16T04:08:19.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.724+0000 7f5f746eb640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:19.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.724+0000 7f5f746eb640 10 monclient(hunting): authenticate will time out at 2546.697513s 2024-09-16T04:08:19.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.724+0000 7f5f72487640 1 -- >> v2:172.21.15.42:3300/0 conn(0x7f5f6c06db90 msgr2=0x7f5f6c1340c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3300/0 2024-09-16T04:08:19.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.724+0000 7f5f72487640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f5f6c06db90 0x7f5f6c1340c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:08:19.736 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.724+0000 7f5f71c86640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5f6c0a37f0 0x7f5f6c0a3c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:08:19.736 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.724+0000 7f5f71c86640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5f6c0a37f0 0x7f5f6c0a3c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:49928/0 (socket says 172.21.15.42:49928) 2024-09-16T04:08:19.736 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.724+0000 7f5f71c86640 1 -- 172.21.15.42:0/3518175283 learned_addr learned my addr 172.21.15.42:0/3518175283 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:08:19.736 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.724+0000 7f5f71c86640 10 monclient(hunting): get_auth_request con 0x7f5f6c0a37f0 auth_method 0 2024-09-16T04:08:19.736 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.724+0000 7f5f71c86640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:08:19.736 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.724+0000 7f5f71c86640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:19.736 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.724+0000 7f5f71c86640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:19.736 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.724+0000 7f5f71c86640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:08:19.736 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.724+0000 7f5f71c86640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:08:19.736 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.724+0000 7f5f71c86640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:08:19.737 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.724+0000 7f5f72c88640 1 --2- 172.21.15.42:0/3518175283 >> v2:172.21.15.154:3300/0 conn(0x7f5f6c136e10 0x7f5f6c137240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:08:19.737 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.724+0000 7f5f71c86640 10 monclient(hunting): handle_auth_done global_id 1014248 payload 274 2024-09-16T04:08:19.738 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.724+0000 7f5f71c86640 1 -- 172.21.15.42:0/3518175283 >> v2:172.21.15.154:3300/0 conn(0x7f5f6c136e10 msgr2=0x7f5f6c137240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:08:19.738 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.724+0000 7f5f71c86640 1 --2- 172.21.15.42:0/3518175283 >> v2:172.21.15.154:3300/0 conn(0x7f5f6c136e10 0x7f5f6c137240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:08:19.738 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.724+0000 7f5f71c86640 1 -- 172.21.15.42:0/3518175283 >> v2:172.21.15.42:3300/0 conn(0x7f5f6c06db90 msgr2=0x7f5f6c1340c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:08:19.738 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.724+0000 7f5f71c86640 1 --2- 172.21.15.42:0/3518175283 >> v2:172.21.15.42:3300/0 conn(0x7f5f6c06db90 0x7f5f6c1340c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:08:19.738 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.724+0000 7f5f71c86640 10 monclient: _finish_hunting 0 2024-09-16T04:08:19.738 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.724+0000 7f5f71c86640 1 monclient: found mon.noname-b 2024-09-16T04:08:19.738 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.724+0000 7f5f71c86640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:19.738 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.724+0000 7f5f71c86640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T04:08:19.738 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.724+0000 7f5f71c86640 1 -- 172.21.15.42:0/3518175283 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f6c134780 con 0x7f5f6c0a37f0 2024-09-16T04:08:19.738 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.724+0000 7f5f72c88640 10 monclient: get_auth_request con 0x7f5f6c136e10 auth_method 0 2024-09-16T04:08:19.738 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.724+0000 7f5f72c88640 1 --2- 172.21.15.42:0/3518175283 >> v2:172.21.15.154:3300/0 conn(0x7f5f6c136e10 0x7f5f6c137240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T04:08:19.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.728+0000 7f5f71c86640 1 --2- 172.21.15.42:0/3518175283 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5f6c0a37f0 0x7f5f6c0a3c00 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f5f6c131410 tx=0x7f5f5c01c400 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1b980dfbc91c5ccd server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:08:19.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.728+0000 7f5f71485640 1 -- 172.21.15.42:0/3518175283 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f5c0b0070 con 0x7f5f6c0a37f0 2024-09-16T04:08:19.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.728+0000 7f5f71485640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:19.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.728+0000 7f5f71485640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:08:19.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.728+0000 7f5f71485640 10 monclient: dump: 2024-09-16T04:08:19.745 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:08:19.745 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:08:19.746 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:08:19.746 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:08:19.746 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:19.746 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:08:19.746 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:08:19.746 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:08:19.746 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:08:19.746 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:08:19.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.728+0000 7f5f71485640 10 monclient: _finish_auth 0 2024-09-16T04:08:19.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.728+0000 7f5f71485640 10 monclient: _check_auth_tickets 2024-09-16T04:08:19.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.728+0000 7f5f71485640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:19.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.728+0000 7f5f71485640 1 -- 172.21.15.42:0/3518175283 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5f5c0a4e50 con 0x7f5f6c0a37f0 2024-09-16T04:08:19.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.728+0000 7f5f71485640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:19.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.728+0000 7f5f71485640 1 -- 172.21.15.42:0/3518175283 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f5c0ab820 con 0x7f5f6c0a37f0 2024-09-16T04:08:19.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.728+0000 7f5f746eb640 5 monclient: authenticate success, global_id 1014248 2024-09-16T04:08:19.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.728+0000 7f5f746eb640 10 monclient: get_monmap_and_config success 2024-09-16T04:08:19.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.728+0000 7f5f746eb640 4 set_mon_vals no callback set 2024-09-16T04:08:19.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.728+0000 7f5f71485640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:19.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.728+0000 7f5f71485640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:08:19.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.728+0000 7f5f71485640 10 monclient: dump: 2024-09-16T04:08:19.747 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:08:19.747 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:08:19.747 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:08:19.747 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:08:19.747 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:19.747 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:08:19.747 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:08:19.747 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:08:19.748 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:08:19.748 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:08:19.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.728+0000 7f5f746eb640 10 monclient: shutdown 2024-09-16T04:08:19.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.728+0000 7f5f746eb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:19.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.728+0000 7f5f746eb640 1 -- 172.21.15.42:0/3518175283 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5f6c0a37f0 msgr2=0x7f5f6c0a3c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:19.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.728+0000 7f5f746eb640 1 --2- 172.21.15.42:0/3518175283 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5f6c0a37f0 0x7f5f6c0a3c00 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f5f6c131410 tx=0x7f5f5c01c400 comp rx=0 tx=0).stop 2024-09-16T04:08:19.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.728+0000 7f5f746eb640 1 -- 172.21.15.42:0/3518175283 shutdown_connections 2024-09-16T04:08:19.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.728+0000 7f5f746eb640 1 --2- 172.21.15.42:0/3518175283 >> v2:172.21.15.154:3300/0 conn(0x7f5f6c136e10 0x7f5f6c137240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:08:19.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.728+0000 7f5f746eb640 1 --2- 172.21.15.42:0/3518175283 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5f6c0a37f0 0x7f5f6c0a3c00 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:08:19.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.728+0000 7f5f746eb640 1 --2- 172.21.15.42:0/3518175283 >> v2:172.21.15.42:3300/0 conn(0x7f5f6c06db90 0x7f5f6c1340c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:08:19.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.728+0000 7f5f746eb640 1 -- 172.21.15.42:0/3518175283 >> 172.21.15.42:0/3518175283 conn(0x7f5f6c0a9fb0 msgr2=0x7f5f6c0aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:19.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.728+0000 7f5f746eb640 1 -- 172.21.15.42:0/3518175283 shutdown_connections 2024-09-16T04:08:19.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.728+0000 7f5f746eb640 1 -- 172.21.15.42:0/3518175283 wait complete. 2024-09-16T04:08:19.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.728+0000 7f5f746eb640 10 monclient: build_initial_monmap 2024-09-16T04:08:19.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.728+0000 7f5f746eb640 10 monclient: monmap: 2024-09-16T04:08:19.749 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:08:19.749 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:08:19.749 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:08:19.749 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:08:19.749 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:19.749 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:08:19.749 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:08:19.749 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:08:19.749 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:08:19.749 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:08:19.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.728+0000 7f5f746eb640 1 librados: starting msgr at 2024-09-16T04:08:19.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.728+0000 7f5f746eb640 1 librados: starting objecter 2024-09-16T04:08:19.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.728+0000 7f5f746eb640 1 Processor -- start 2024-09-16T04:08:19.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.728+0000 7f5f746eb640 1 -- start start 2024-09-16T04:08:19.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.728+0000 7f5f746eb640 1 librados: setting wanted keys 2024-09-16T04:08:19.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.728+0000 7f5f746eb640 1 librados: calling monclient init 2024-09-16T04:08:19.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.728+0000 7f5f746eb640 10 monclient: init 2024-09-16T04:08:19.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.732+0000 7f5f746eb640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:19.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.732+0000 7f5f746eb640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:08:19.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.732+0000 7f5f746eb640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5f6c06db90 0x7f5f6c13d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:08:19.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.732+0000 7f5f746eb640 10 monclient(hunting): picked mon.noname-a con 0x7f5f6c06db90 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:08:19.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.732+0000 7f5f746eb640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5f6c0a37f0 0x7f5f6c0a2ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:08:19.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.732+0000 7f5f746eb640 10 monclient(hunting): picked mon.noname-c con 0x7f5f6c0a37f0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:08:19.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.732+0000 7f5f746eb640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5f6c136e10 0x7f5f6c0a3220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:08:19.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.732+0000 7f5f746eb640 10 monclient(hunting): picked mon.noname-b con 0x7f5f6c136e10 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:08:19.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.732+0000 7f5f746eb640 10 monclient(hunting): start opening mon connection 2024-09-16T04:08:19.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.732+0000 7f5f746eb640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f5f6c140bd0 con 0x7f5f6c06db90 2024-09-16T04:08:19.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.732+0000 7f5f746eb640 10 monclient(hunting): start opening mon connection 2024-09-16T04:08:19.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.732+0000 7f5f746eb640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f5f6c137c80 con 0x7f5f6c136e10 2024-09-16T04:08:19.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.732+0000 7f5f746eb640 10 monclient(hunting): start opening mon connection 2024-09-16T04:08:19.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.732+0000 7f5f746eb640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f5f6c140d50 con 0x7f5f6c0a37f0 2024-09-16T04:08:19.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.732+0000 7f5f746eb640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:19.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.732+0000 7f5f746eb640 10 monclient(hunting): authenticate will time out at 2546.704139s 2024-09-16T04:08:19.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.732+0000 7f5f72487640 1 -- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5f6c06db90 msgr2=0x7f5f6c13d260 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.42:3300/0 2024-09-16T04:08:19.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.732+0000 7f5f72487640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5f6c06db90 0x7f5f6c13d260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:08:19.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.732+0000 7f5f71c86640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5f6c0a37f0 0x7f5f6c0a2ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:08:19.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.732+0000 7f5f71c86640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5f6c0a37f0 0x7f5f6c0a2ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:49936/0 (socket says 172.21.15.42:49936) 2024-09-16T04:08:19.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.732+0000 7f5f71c86640 1 -- 172.21.15.42:0/4258462736 learned_addr learned my addr 172.21.15.42:0/4258462736 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:08:19.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.732+0000 7f5f71c86640 10 monclient(hunting): get_auth_request con 0x7f5f6c0a37f0 auth_method 0 2024-09-16T04:08:19.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.732+0000 7f5f71c86640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:08:19.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.732+0000 7f5f71c86640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:19.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.732+0000 7f5f71c86640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:19.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.732+0000 7f5f71c86640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:08:19.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.732+0000 7f5f71c86640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:08:19.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.732+0000 7f5f71c86640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:08:19.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.732+0000 7f5f71c86640 10 monclient(hunting): handle_auth_done global_id 1014251 payload 824 2024-09-16T04:08:19.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.732+0000 7f5f71c86640 1 -- 172.21.15.42:0/4258462736 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5f6c136e10 msgr2=0x7f5f6c0a3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:19.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.732+0000 7f5f71c86640 1 --2- 172.21.15.42:0/4258462736 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5f6c136e10 0x7f5f6c0a3220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:08:19.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.732+0000 7f5f71c86640 1 -- 172.21.15.42:0/4258462736 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5f6c06db90 msgr2=0x7f5f6c13d260 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:08:19.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.732+0000 7f5f71c86640 1 --2- 172.21.15.42:0/4258462736 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5f6c06db90 0x7f5f6c13d260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:08:19.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.732+0000 7f5f71c86640 10 monclient: _finish_hunting 0 2024-09-16T04:08:19.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.732+0000 7f5f71c86640 1 monclient: found mon.noname-c 2024-09-16T04:08:19.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.732+0000 7f5f71c86640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:19.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.732+0000 7f5f71c86640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T04:08:19.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.732+0000 7f5f71c86640 1 -- 172.21.15.42:0/4258462736 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f6c06f6a0 con 0x7f5f6c0a37f0 2024-09-16T04:08:19.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.732+0000 7f5f71c86640 1 --2- 172.21.15.42:0/4258462736 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5f6c0a37f0 0x7f5f6c0a2ce0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f5f5c01aab0 tx=0x7f5f5c0191a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:08:19.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.732+0000 7f5f62ffd640 10 client.?.objecter ms_handle_connect 0x7f5f6c0a37f0 2024-09-16T04:08:19.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.732+0000 7f5f62ffd640 10 client.?.objecter resend_mon_ops 2024-09-16T04:08:19.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.732+0000 7f5f62ffd640 1 -- 172.21.15.42:0/4258462736 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f5c0b0040 con 0x7f5f6c0a37f0 2024-09-16T04:08:19.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.732+0000 7f5f62ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:19.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.732+0000 7f5f62ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:08:19.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.732+0000 7f5f62ffd640 10 monclient: dump: 2024-09-16T04:08:19.753 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:08:19.753 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:08:19.753 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:08:19.753 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:08:19.753 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:19.754 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:08:19.754 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:08:19.754 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:08:19.754 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:08:19.754 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:08:19.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.732+0000 7f5f62ffd640 10 monclient: _finish_auth 0 2024-09-16T04:08:19.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.732+0000 7f5f62ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:08:19.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.732+0000 7f5f62ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:19.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.732+0000 7f5f62ffd640 1 -- 172.21.15.42:0/4258462736 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5f5c0b4620 con 0x7f5f6c0a37f0 2024-09-16T04:08:19.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.732+0000 7f5f746eb640 5 monclient: authenticate success, global_id 1014251 2024-09-16T04:08:19.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.732+0000 7f5f746eb640 10 monclient: _renew_subs 2024-09-16T04:08:19.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.732+0000 7f5f746eb640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:08:19.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.732+0000 7f5f746eb640 1 -- 172.21.15.42:0/4258462736 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f6c09f1d0 con 0x7f5f6c0a37f0 2024-09-16T04:08:19.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.732+0000 7f5f62ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:19.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.732+0000 7f5f62ffd640 1 -- 172.21.15.42:0/4258462736 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f5c0b49f0 con 0x7f5f6c0a37f0 2024-09-16T04:08:19.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.732+0000 7f5f62ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:19.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.732+0000 7f5f62ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:08:19.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.732+0000 7f5f62ffd640 10 monclient: dump: 2024-09-16T04:08:19.755 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:08:19.755 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:08:19.755 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:08:19.755 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:08:19.755 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:19.755 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:08:19.755 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:08:19.755 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:08:19.755 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:08:19.755 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:08:19.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.732+0000 7f5f63fff640 4 set_mon_vals no callback set 2024-09-16T04:08:19.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.732+0000 7f5f746eb640 10 client.1014251.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:08:19.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.732+0000 7f5f746eb640 10 monclient: _renew_subs 2024-09-16T04:08:19.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.732+0000 7f5f746eb640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:08:19.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.732+0000 7f5f746eb640 1 -- 172.21.15.42:0/4258462736 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f6c09d8b0 con 0x7f5f6c0a37f0 2024-09-16T04:08:19.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.732+0000 7f5f746eb640 1 librados: init done 2024-09-16T04:08:19.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.732+0000 7f5f746eb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:19.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.732+0000 7f5f746eb640 1 --2- 172.21.15.42:0/4258462736 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5f6c1cfa60 0x7f5f6c02c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:08:19.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.732+0000 7f5f746eb640 10 monclient(hunting): start opening mon connection 2024-09-16T04:08:19.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.732+0000 7f5f746eb640 1 -- 172.21.15.42:0/4258462736 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f5f6c133580 con 0x7f5f6c1cfa60 2024-09-16T04:08:19.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.732+0000 7f5f72487640 1 --2- 172.21.15.42:0/4258462736 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5f6c1cfa60 0x7f5f6c02c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:08:19.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.732+0000 7f5f72487640 10 monclient: get_auth_request con 0x7f5f6c1cfa60 auth_method 0 2024-09-16T04:08:19.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.732+0000 7f5f72487640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:08:19.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.732+0000 7f5f72487640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:19.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.732+0000 7f5f72487640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:19.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.732+0000 7f5f62ffd640 1 -- 172.21.15.42:0/4258462736 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f5f5c0b4b90 con 0x7f5f6c0a37f0 2024-09-16T04:08:19.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.732+0000 7f5f72487640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:08:19.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.732+0000 7f5f72487640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:08:19.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.732+0000 7f5f72487640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:08:19.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.732+0000 7f5f72487640 10 monclient(hunting): handle_auth_done global_id 1014254 payload 824 2024-09-16T04:08:19.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.732+0000 7f5f62ffd640 1 --2- 172.21.15.42:0/4258462736 >> v2:172.21.15.154:6812/254845342 conn(0x7f5f500377c0 0x7f5f50039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:08:19.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.732+0000 7f5f72487640 1 -- 172.21.15.42:0/4258462736 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5f6c09f890 con 0x7f5f6c1cfa60 2024-09-16T04:08:19.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.732+0000 7f5f62ffd640 1 -- 172.21.15.42:0/4258462736 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(706..706 src has 451..706) ==== 5040+0+0 (secure 0 0 0) 0x7f5f5c0e2ab0 con 0x7f5f6c0a37f0 2024-09-16T04:08:19.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.732+0000 7f5f62ffd640 10 client.1014251.objecter ms_dispatch 0x7f5f6c000ba0 osd_map(706..706 src has 451..706) 2024-09-16T04:08:19.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.732+0000 7f5f62ffd640 3 client.1014251.objecter handle_osd_map got epochs [706,706] > 0 2024-09-16T04:08:19.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.732+0000 7f5f62ffd640 3 client.1014251.objecter handle_osd_map decoding full epoch 706 2024-09-16T04:08:19.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.732+0000 7f5f62ffd640 20 client.1014251.objecter dump_active .. 0 homeless 2024-09-16T04:08:19.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.732+0000 7f5f62ffd640 10 client.1014251.objecter ms_handle_connect 0x7f5f6c1cfa60 2024-09-16T04:08:19.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.732+0000 7f5f62ffd640 10 client.1014251.objecter resend_mon_ops 2024-09-16T04:08:19.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.732+0000 7f5f72487640 1 --2- 172.21.15.42:0/4258462736 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5f6c1cfa60 0x7f5f6c02c650 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f5f6800bcc0 tx=0x7f5f68009f30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:08:19.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.732+0000 7f5f62ffd640 1 -- 172.21.15.42:0/4258462736 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f68014630 con 0x7f5f6c1cfa60 2024-09-16T04:08:19.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.732+0000 7f5f62ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:19.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.732+0000 7f5f62ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:08:19.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.732+0000 7f5f62ffd640 10 monclient: dump: 2024-09-16T04:08:19.758 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:08:19.758 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:08:19.758 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:08:19.758 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:08:19.758 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:19.758 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:08:19.758 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:08:19.758 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:08:19.758 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:08:19.759 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:08:19.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.732+0000 7f5f72c88640 1 --2- 172.21.15.42:0/4258462736 >> v2:172.21.15.154:6812/254845342 conn(0x7f5f500377c0 0x7f5f50039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:08:19.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.732+0000 7f5f72c88640 10 monclient: get_auth_request con 0x7f5f500377c0 auth_method 0 2024-09-16T04:08:19.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.732+0000 7f5f62ffd640 1 -- 172.21.15.42:0/4258462736 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f5f68014630 con 0x7f5f6c1cfa60 2024-09-16T04:08:19.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.732+0000 7f5f62ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:19.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.732+0000 7f5f62ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:08:19.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.732+0000 7f5f62ffd640 1 -- 172.21.15.42:0/4258462736 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5f6c1cfa60 msgr2=0x7f5f6c02c650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:19.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.732+0000 7f5f62ffd640 1 --2- 172.21.15.42:0/4258462736 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5f6c1cfa60 0x7f5f6c02c650 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f5f6800bcc0 tx=0x7f5f68009f30 comp rx=0 tx=0).stop 2024-09-16T04:08:19.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.732+0000 7f5f62ffd640 1 -- 172.21.15.42:0/4258462736 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5f6c1cfa60 msgr2=0x7f5f6c02c650 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:08:19.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.732+0000 7f5f62ffd640 1 --2- 172.21.15.42:0/4258462736 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5f6c1cfa60 0x7f5f6c02c650 secure :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f5f6800bcc0 tx=0x7f5f68009f30 comp rx=0 tx=0).stop 2024-09-16T04:08:19.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.732+0000 7f5f62ffd640 10 client.1014251.objecter ms_handle_connect 0x7f5f500377c0 2024-09-16T04:08:19.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.732+0000 7f5f72c88640 1 --2- 172.21.15.42:0/4258462736 >> v2:172.21.15.154:6812/254845342 conn(0x7f5f500377c0 0x7f5f50039c80 secure :-1 s=READY pgs=3898 cs=0 l=1 rev1=1 crypto rx=0x7f5f6c131760 tx=0x7f5f64008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:08:19.761 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.740+0000 7f5f746eb640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:08:19.761 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.740+0000 7f5f746eb640 1 --2- 172.21.15.42:0/4258462736 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5f6c1cfe40 0x7f5f6c02cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:08:19.761 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.740+0000 7f5f746eb640 10 monclient(hunting): start opening mon connection 2024-09-16T04:08:19.761 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.740+0000 7f5f746eb640 1 -- 172.21.15.42:0/4258462736 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f5f6c09e000 con 0x7f5f6c1cfe40 2024-09-16T04:08:19.761 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.740+0000 7f5f72487640 1 --2- 172.21.15.42:0/4258462736 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5f6c1cfe40 0x7f5f6c02cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:08:19.761 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.740+0000 7f5f72487640 10 monclient: get_auth_request con 0x7f5f6c1cfe40 auth_method 0 2024-09-16T04:08:19.761 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.740+0000 7f5f72487640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:08:19.761 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.740+0000 7f5f72487640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:19.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.740+0000 7f5f72487640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:19.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.740+0000 7f5f72487640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:08:19.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.740+0000 7f5f72487640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:08:19.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.740+0000 7f5f72487640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:08:19.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.740+0000 7f5f72487640 10 monclient(hunting): handle_auth_done global_id 1014257 payload 824 2024-09-16T04:08:19.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.740+0000 7f5f72487640 1 -- 172.21.15.42:0/4258462736 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f5f6c139630 con 0x7f5f6c1cfe40 2024-09-16T04:08:19.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.740+0000 7f5f72487640 1 --2- 172.21.15.42:0/4258462736 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5f6c1cfe40 0x7f5f6c02cb90 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f5f68014000 tx=0x7f5f68005d60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:08:19.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.740+0000 7f5f62ffd640 10 client.1014251.objecter ms_handle_connect 0x7f5f6c1cfe40 2024-09-16T04:08:19.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.740+0000 7f5f62ffd640 10 client.1014251.objecter resend_mon_ops 2024-09-16T04:08:19.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.740+0000 7f5f62ffd640 1 -- 172.21.15.42:0/4258462736 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f68019040 con 0x7f5f6c1cfe40 2024-09-16T04:08:19.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.740+0000 7f5f62ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:19.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.740+0000 7f5f62ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:08:19.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.740+0000 7f5f62ffd640 10 monclient: dump: 2024-09-16T04:08:19.762 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:08:19.763 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:08:19.763 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:08:19.763 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:08:19.763 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:19.763 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:08:19.763 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:08:19.763 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:08:19.763 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:08:19.763 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:08:19.763 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.740+0000 7f5f62ffd640 1 -- 172.21.15.42:0/4258462736 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4701 (secure 0 0 0) 0x7f5f680133d0 con 0x7f5f6c1cfe40 2024-09-16T04:08:19.763 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.740+0000 7f5f62ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:08:19.763 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.740+0000 7f5f62ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:08:19.763 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.740+0000 7f5f62ffd640 1 -- 172.21.15.42:0/4258462736 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5f6c1cfe40 msgr2=0x7f5f6c02cb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:19.763 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.740+0000 7f5f62ffd640 1 --2- 172.21.15.42:0/4258462736 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5f6c1cfe40 0x7f5f6c02cb90 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f5f68014000 tx=0x7f5f68005d60 comp rx=0 tx=0).stop 2024-09-16T04:08:19.763 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.740+0000 7f5f62ffd640 1 -- 172.21.15.42:0/4258462736 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5f6c1cfe40 msgr2=0x7f5f6c02cb90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:08:19.764 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.740+0000 7f5f62ffd640 1 --2- 172.21.15.42:0/4258462736 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5f6c1cfe40 0x7f5f6c02cb90 secure :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f5f68014000 tx=0x7f5f68005d60 comp rx=0 tx=0).stop 2024-09-16T04:08:19.764 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T04:08:19.764 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:08:19.764 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:08:19.764 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T04:08:19.764 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 240, 2024-09-16T04:08:19.764 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 94943, 2024-09-16T04:08:19.764 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T04:08:19.764 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T04:08:19.764 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T04:08:19.765 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:08:19.765 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 1, 2024-09-16T04:08:19.765 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:08:19.765 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T04:08:19.765 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T04:08:19.765 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:08:19.765 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:08:19.765 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:08:19.765 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:08:19.765 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:08:19.765 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:08:19.765 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:08:19.765 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:08:19.765 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:08:19.766 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:08:19.766 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:08:19.766 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:08:19.766 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:08:19.766 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T04:08:19.766 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:08:19.766 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:08:19.766 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:08:19.766 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:08:19.766 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:08:19.766 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:08:19.766 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:08:19.766 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:08:19.766 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:08:19.767 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:08:19.767 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:08:19.767 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:08:19.767 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:08:19.767 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T04:08:19.767 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T04:08:19.767 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T04:08:19.767 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T04:08:19.767 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T04:08:19.767 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T04:08:19.767 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:08:19.767 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:08:19.768 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T04:08:19.768 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T04:08:19.768 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T04:08:19.768 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T04:08:19.768 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T04:08:19.768 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T04:08:19.768 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T04:08:19.768 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:08:19.768 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T04:08:19.768 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:08:19.768 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:08:19.768 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:08:19.768 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:08:19.768 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:08:19.769 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:08:19.769 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:08:19.769 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:08:19.769 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:08:19.769 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:08:19.769 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:08:19.769 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:08:19.769 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T04:08:19.769 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:08:19.769 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T04:08:19.769 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:08:19.769 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:08:19.769 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:08:19.769 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:08:19.770 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:08:19.770 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:08:19.770 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:08:19.770 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T04:08:19.770 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:08:19.770 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:08:19.770 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:08:19.770 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:08:19.770 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T04:08:19.770 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:08:19.770 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:08:19.770 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:08:19.770 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:08:19.770 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T04:08:19.770 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T04:08:19.771 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:08:19.771 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:08:19.771 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:08:19.771 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:08:19.771 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:08:19.771 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:08:19.771 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:08:19.771 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:08:19.771 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:08:19.771 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:08:19.771 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:08:19.771 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T04:08:19.771 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:08:19.771 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:08:19.772 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:08:19.772 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:08:19.772 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T04:08:19.772 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:08:19.772 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:08:19.772 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:08:19.772 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:08:19.772 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T04:08:19.772 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T04:08:19.772 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:08:19.772 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:08:19.772 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:08:19.772 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:08:19.772 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:08:19.773 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:08:19.773 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:08:19.773 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:08:19.773 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:08:19.773 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:08:19.773 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:08:19.773 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T04:08:19.773 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:08:19.773 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:08:19.773 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:08:19.773 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:08:19.773 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T04:08:19.773 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:08:19.773 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:08:19.774 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:08:19.774 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:08:19.774 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T04:08:19.774 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T04:08:19.774 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:08:19.774 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:08:19.774 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:08:19.774 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:08:19.774 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:08:19.774 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:08:19.774 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T04:08:19.774 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T04:08:19.774 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:08:19.774 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:08:19.774 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:08:19.775 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:08:19.775 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:08:19.775 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:08:19.775 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T04:08:19.775 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:08:19.775 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:08:19.775 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:08:19.775 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 3 2024-09-16T04:08:19.775 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:08:19.775 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:08:19.775 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T04:08:19.775 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:08:19.775 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:08:19.775 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:08:19.775 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 3 2024-09-16T04:08:19.776 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:08:19.776 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:08:19.776 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:08:19.776 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T04:08:19.776 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T04:08:19.778 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.740+0000 7f5f746eb640 10 librados: watch_flush enter 2024-09-16T04:08:19.778 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.740+0000 7f5f746eb640 10 librados: watch_flush exit 2024-09-16T04:08:19.778 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.740+0000 7f5f746eb640 20 client.1014251.objecter shutdown clearing up homeless session... 2024-09-16T04:08:19.778 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.740+0000 7f5f746eb640 10 client.1014251.objecter successfully canceled tick 2024-09-16T04:08:19.778 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.740+0000 7f5f746eb640 1 -- 172.21.15.42:0/4258462736 >> v2:172.21.15.154:6812/254845342 conn(0x7f5f500377c0 msgr2=0x7f5f50039c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:19.778 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.740+0000 7f5f746eb640 1 --2- 172.21.15.42:0/4258462736 >> v2:172.21.15.154:6812/254845342 conn(0x7f5f500377c0 0x7f5f50039c80 secure :-1 s=READY pgs=3898 cs=0 l=1 rev1=1 crypto rx=0x7f5f6c131760 tx=0x7f5f64008040 comp rx=0 tx=0).stop 2024-09-16T04:08:19.778 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.740+0000 7f5f746eb640 10 monclient: shutdown 2024-09-16T04:08:19.778 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.740+0000 7f5f746eb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:19.778 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.740+0000 7f5f746eb640 1 -- 172.21.15.42:0/4258462736 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5f6c0a37f0 msgr2=0x7f5f6c0a2ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:19.778 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.740+0000 7f5f746eb640 1 --2- 172.21.15.42:0/4258462736 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5f6c0a37f0 0x7f5f6c0a2ce0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f5f5c01aab0 tx=0x7f5f5c0191a0 comp rx=0 tx=0).stop 2024-09-16T04:08:19.778 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.740+0000 7f5f72c88640 1 -- 172.21.15.42:0/4258462736 reap_dead start 2024-09-16T04:08:19.778 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.740+0000 7f5f72c88640 1 -- 172.21.15.42:0/4258462736 reap_dead start 2024-09-16T04:08:19.778 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.740+0000 7f5f746eb640 1 -- 172.21.15.42:0/4258462736 shutdown_connections 2024-09-16T04:08:19.779 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.740+0000 7f5f746eb640 1 -- 172.21.15.42:0/4258462736 >> 172.21.15.42:0/4258462736 conn(0x7f5f6c0a9fb0 msgr2=0x7f5f6c0a82a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:19.779 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.740+0000 7f5f746eb640 1 -- 172.21.15.42:0/4258462736 shutdown_connections 2024-09-16T04:08:19.779 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.740+0000 7f5f746eb640 1 -- 172.21.15.42:0/4258462736 wait complete. 2024-09-16T04:08:19.779 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.740+0000 7f5f746eb640 1 librados: shutdown 2024-09-16T04:08:19.779 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T04:08:19.779 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-16T04:08:19.779 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T04:08:19.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.900+0000 7f51e3f79640 10 monclient: get_monmap_and_config 2024-09-16T04:08:19.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.900+0000 7f51e3f79640 10 monclient: build_initial_monmap 2024-09-16T04:08:19.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.900+0000 7f51e3f79640 10 monclient: monmap: 2024-09-16T04:08:19.922 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:08:19.922 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:08:19.922 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:08:19.904552+0000 2024-09-16T04:08:19.922 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:08:19.904552+0000 2024-09-16T04:08:19.922 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:19.922 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:08:19.922 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:08:19.922 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:08:19.922 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:08:19.923 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:08:19.923 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:08:19.923 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:08:19.923 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.900+0000 7f51e3f79640 1 Processor -- start 2024-09-16T04:08:19.923 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.900+0000 7f51e3f79640 1 -- start start 2024-09-16T04:08:19.923 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.900+0000 7f51e3f79640 10 monclient: init 2024-09-16T04:08:19.923 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.900+0000 7f51e3f79640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:19.923 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.900+0000 7f51e3f79640 10 monclient: _add_conns ranks=[1,4,3,2,0] 2024-09-16T04:08:19.923 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.900+0000 7f51e3f79640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f51dc0a39b0 addr v1:172.21.15.154:6789/0 2024-09-16T04:08:19.923 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.900+0000 7f51e3f79640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f51dc0a3d90 0x7f51dc13c380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:08:19.923 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.900+0000 7f51e3f79640 10 monclient(hunting): picked mon.noname-c con 0x7f51dc0a3d90 addr v2:172.21.15.154:3300/0 2024-09-16T04:08:19.923 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.900+0000 7f51e3f79640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f51dc13c8c0 0x7f51dc13ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:08:19.923 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.900+0000 7f51e3f79640 10 monclient(hunting): picked mon.noname-a con 0x7f51dc13c8c0 addr v2:172.21.15.42:3300/0 2024-09-16T04:08:19.923 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.900+0000 7f51e3f79640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f51dc134290 con 0x7f51dc0a39b0 2024-09-16T04:08:19.924 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.900+0000 7f51e3f79640 10 monclient(hunting): start opening mon connection 2024-09-16T04:08:19.924 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.900+0000 7f51e3f79640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f51dc09c140 con 0x7f51dc13c8c0 2024-09-16T04:08:19.924 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.900+0000 7f51e3f79640 10 monclient(hunting): start opening mon connection 2024-09-16T04:08:19.924 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.900+0000 7f51e3f79640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f51dc09c420 con 0x7f51dc0a3d90 2024-09-16T04:08:19.924 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.900+0000 7f51e3f79640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:19.924 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.900+0000 7f51e3f79640 10 monclient(hunting): authenticate will time out at 2546.873989s 2024-09-16T04:08:19.924 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.900+0000 7f51e2516640 1 -- >> v2:172.21.15.42:3300/0 conn(0x7f51dc13c8c0 msgr2=0x7f51dc13ecb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3300/0 2024-09-16T04:08:19.924 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.900+0000 7f51e2516640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f51dc13c8c0 0x7f51dc13ecb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:08:19.924 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.900+0000 7f51e1514640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f51dc0a3d90 0x7f51dc13c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:08:19.924 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.900+0000 7f51e1514640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f51dc0a3d90 0x7f51dc13c380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.42:38680/0 (socket says 172.21.15.42:38680) 2024-09-16T04:08:19.924 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.900+0000 7f51e1514640 1 -- 172.21.15.42:0/1388524903 learned_addr learned my addr 172.21.15.42:0/1388524903 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:08:19.924 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.900+0000 7f51e1514640 10 monclient(hunting): get_auth_request con 0x7f51dc0a3d90 auth_method 0 2024-09-16T04:08:19.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.900+0000 7f51e1514640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:08:19.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.900+0000 7f51e1514640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:19.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.900+0000 7f51e1514640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:19.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.900+0000 7f51e1514640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:08:19.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.900+0000 7f51e1514640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:08:19.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.900+0000 7f51e1514640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:08:19.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.900+0000 7f51e1514640 10 monclient(hunting): handle_auth_done global_id 905158 payload 274 2024-09-16T04:08:19.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.900+0000 7f51e1514640 1 -- 172.21.15.42:0/1388524903 >> v2:172.21.15.42:3300/0 conn(0x7f51dc13c8c0 msgr2=0x7f51dc13ecb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:08:19.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.900+0000 7f51e1514640 1 --2- 172.21.15.42:0/1388524903 >> v2:172.21.15.42:3300/0 conn(0x7f51dc13c8c0 0x7f51dc13ecb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:08:19.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.900+0000 7f51e1514640 1 -- 172.21.15.42:0/1388524903 >> v1:172.21.15.154:6789/0 conn(0x7f51dc0a39b0 legacy=0x7f51dc133b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:19.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.900+0000 7f51e1514640 10 monclient: _finish_hunting 0 2024-09-16T04:08:19.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.900+0000 7f51e1514640 1 monclient: found mon.noname-c 2024-09-16T04:08:19.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.900+0000 7f51e1514640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:19.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.900+0000 7f51e1514640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.154:3300/0 2024-09-16T04:08:19.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.900+0000 7f51e1514640 1 -- 172.21.15.42:0/1388524903 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51dc06dd60 con 0x7f51dc0a3d90 2024-09-16T04:08:19.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.904+0000 7f51e1514640 1 --2- 172.21.15.42:0/1388524903 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f51dc0a3d90 0x7f51dc13c380 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f51dc0a56c0 tx=0x7f51cc01c400 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4cf120212125820f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:08:19.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.904+0000 7f51e0d13640 1 -- 172.21.15.42:0/1388524903 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51cc0b0070 con 0x7f51dc0a3d90 2024-09-16T04:08:19.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.904+0000 7f51e0d13640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:19.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.904+0000 7f51e0d13640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:08:19.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.904+0000 7f51e0d13640 10 monclient: dump: 2024-09-16T04:08:19.927 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:08:19.927 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:08:19.927 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:08:19.927 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:08:19.927 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:19.927 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:08:19.928 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:08:19.928 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:08:19.928 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:08:19.928 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:08:19.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.904+0000 7f51e0d13640 10 monclient: _finish_auth 0 2024-09-16T04:08:19.929 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.904+0000 7f51e0d13640 10 monclient: _check_auth_tickets 2024-09-16T04:08:19.929 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.904+0000 7f51e0d13640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:19.929 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.904+0000 7f51e0d13640 1 -- 172.21.15.42:0/1388524903 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f51cc0ab600 con 0x7f51dc0a3d90 2024-09-16T04:08:19.930 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.904+0000 7f51e3f79640 5 monclient: authenticate success, global_id 905158 2024-09-16T04:08:19.930 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.904+0000 7f51e3f79640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:08:19.930 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.904+0000 7f51e0d13640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:19.930 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.904+0000 7f51e3f79640 10 monclient: get_monmap_and_config success 2024-09-16T04:08:19.930 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.904+0000 7f51e3f79640 4 set_mon_vals no callback set 2024-09-16T04:08:19.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.904+0000 7f51e0d13640 1 -- 172.21.15.42:0/1388524903 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51cc0ab9d0 con 0x7f51dc0a3d90 2024-09-16T04:08:19.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.904+0000 7f51e0d13640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:19.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.904+0000 7f51e0d13640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:08:19.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.904+0000 7f51e0d13640 10 monclient: dump: 2024-09-16T04:08:19.931 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:08:19.931 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:08:19.931 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:08:19.931 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:08:19.931 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:19.931 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:08:19.931 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:08:19.931 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:08:19.932 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:08:19.932 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:08:19.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.904+0000 7f51e3f79640 10 monclient: shutdown 2024-09-16T04:08:19.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.904+0000 7f51e3f79640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:19.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.904+0000 7f51e3f79640 1 -- 172.21.15.42:0/1388524903 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f51dc0a3d90 msgr2=0x7f51dc13c380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:19.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.904+0000 7f51e3f79640 1 --2- 172.21.15.42:0/1388524903 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f51dc0a3d90 0x7f51dc13c380 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f51dc0a56c0 tx=0x7f51cc01c400 comp rx=0 tx=0).stop 2024-09-16T04:08:19.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.904+0000 7f51e3f79640 1 -- 172.21.15.42:0/1388524903 shutdown_connections 2024-09-16T04:08:19.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.904+0000 7f51e3f79640 1 --2- 172.21.15.42:0/1388524903 >> v2:172.21.15.42:3300/0 conn(0x7f51dc13c8c0 0x7f51dc13ecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:08:19.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.904+0000 7f51e3f79640 1 --2- 172.21.15.42:0/1388524903 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f51dc0a3d90 0x7f51dc13c380 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:08:19.933 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.904+0000 7f51e3f79640 1 -- 172.21.15.42:0/1388524903 >> 172.21.15.42:0/1388524903 conn(0x7f51dc0a9fb0 msgr2=0x7f51dc0aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:19.933 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.904+0000 7f51e3f79640 1 -- 172.21.15.42:0/1388524903 shutdown_connections 2024-09-16T04:08:19.933 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.904+0000 7f51e3f79640 1 -- 172.21.15.42:0/1388524903 wait complete. 2024-09-16T04:08:19.933 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.904+0000 7f51e3f79640 10 monclient: build_initial_monmap 2024-09-16T04:08:19.933 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.904+0000 7f51e3f79640 10 monclient: monmap: 2024-09-16T04:08:19.933 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:08:19.933 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:08:19.933 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:08:19.933 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:08:19.933 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:19.933 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:08:19.933 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:08:19.933 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:08:19.933 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:08:19.934 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:08:19.934 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.904+0000 7f51e3f79640 1 librados: starting msgr at 2024-09-16T04:08:19.934 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.904+0000 7f51e3f79640 1 librados: starting objecter 2024-09-16T04:08:19.934 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.904+0000 7f51e3f79640 1 Processor -- start 2024-09-16T04:08:19.934 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.904+0000 7f51e3f79640 1 -- start start 2024-09-16T04:08:19.934 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.904+0000 7f51e3f79640 1 librados: setting wanted keys 2024-09-16T04:08:19.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.904+0000 7f51e3f79640 1 librados: calling monclient init 2024-09-16T04:08:19.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.904+0000 7f51e3f79640 10 monclient: init 2024-09-16T04:08:19.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.904+0000 7f51e3f79640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:19.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.904+0000 7f51e3f79640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:08:19.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.904+0000 7f51e3f79640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f51dc0a39b0 0x7f51dc12d620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:08:19.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.904+0000 7f51e3f79640 10 monclient(hunting): picked mon.noname-a con 0x7f51dc0a39b0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:08:19.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.904+0000 7f51e3f79640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f51dc0a3d90 0x7f51dc12db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:08:19.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.904+0000 7f51e3f79640 10 monclient(hunting): picked mon.noname-b con 0x7f51dc0a3d90 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:08:19.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.904+0000 7f51e3f79640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f51dc13c8c0 0x7f51dc138130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:08:19.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.904+0000 7f51e3f79640 10 monclient(hunting): picked mon.noname-c con 0x7f51dc13c8c0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:08:19.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.904+0000 7f51e3f79640 10 monclient(hunting): start opening mon connection 2024-09-16T04:08:19.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.904+0000 7f51e3f79640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f51dc141da0 con 0x7f51dc0a39b0 2024-09-16T04:08:19.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.904+0000 7f51e3f79640 10 monclient(hunting): start opening mon connection 2024-09-16T04:08:19.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.904+0000 7f51e3f79640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f51dc141c20 con 0x7f51dc0a3d90 2024-09-16T04:08:19.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.904+0000 7f51e3f79640 10 monclient(hunting): start opening mon connection 2024-09-16T04:08:19.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.904+0000 7f51e3f79640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f51dc141f20 con 0x7f51dc13c8c0 2024-09-16T04:08:19.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.904+0000 7f51e3f79640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:19.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.904+0000 7f51e3f79640 10 monclient(hunting): authenticate will time out at 2546.879573s 2024-09-16T04:08:19.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.904+0000 7f51e1d15640 1 -- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f51dc0a39b0 msgr2=0x7f51dc12d620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.42:3300/0 2024-09-16T04:08:19.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.904+0000 7f51e1d15640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f51dc0a39b0 0x7f51dc12d620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:08:19.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.904+0000 7f51e2516640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f51dc13c8c0 0x7f51dc138130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:08:19.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.904+0000 7f51e2516640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f51dc13c8c0 0x7f51dc138130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:49966/0 (socket says 172.21.15.42:49966) 2024-09-16T04:08:19.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.904+0000 7f51e2516640 1 -- 172.21.15.42:0/2471944885 learned_addr learned my addr 172.21.15.42:0/2471944885 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:08:19.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.904+0000 7f51e2516640 10 monclient(hunting): get_auth_request con 0x7f51dc13c8c0 auth_method 0 2024-09-16T04:08:19.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.904+0000 7f51e2516640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:08:19.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.904+0000 7f51e2516640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:19.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.904+0000 7f51e2516640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:19.938 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.904+0000 7f51e2516640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:08:19.938 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.904+0000 7f51e2516640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:08:19.938 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.904+0000 7f51e2516640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:08:19.938 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.908+0000 7f51e1514640 1 --2- 172.21.15.42:0/2471944885 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f51dc0a3d90 0x7f51dc12db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:08:19.939 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.908+0000 7f51e1514640 10 monclient(hunting): get_auth_request con 0x7f51dc0a3d90 auth_method 0 2024-09-16T04:08:19.939 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.908+0000 7f51e1514640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:08:19.939 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.908+0000 7f51e1514640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:19.939 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.908+0000 7f51e1514640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:19.940 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.908+0000 7f51e2516640 10 monclient(hunting): handle_auth_done global_id 1014260 payload 824 2024-09-16T04:08:19.940 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.908+0000 7f51e2516640 1 -- 172.21.15.42:0/2471944885 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f51dc0a3d90 msgr2=0x7f51dc12db60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:19.941 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.908+0000 7f51e2516640 1 --2- 172.21.15.42:0/2471944885 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f51dc0a3d90 0x7f51dc12db60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:08:19.941 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.908+0000 7f51e2516640 1 -- 172.21.15.42:0/2471944885 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f51dc0a39b0 msgr2=0x7f51dc12d620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:08:19.941 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.908+0000 7f51e2516640 1 --2- 172.21.15.42:0/2471944885 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f51dc0a39b0 0x7f51dc12d620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:08:19.941 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.908+0000 7f51e2516640 10 monclient: _finish_hunting 0 2024-09-16T04:08:19.941 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.908+0000 7f51e2516640 1 monclient: found mon.noname-c 2024-09-16T04:08:19.942 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.908+0000 7f51e2516640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:19.942 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.908+0000 7f51e2516640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T04:08:19.942 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.908+0000 7f51e2516640 1 -- 172.21.15.42:0/2471944885 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51dc13acf0 con 0x7f51dc13c8c0 2024-09-16T04:08:19.942 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.908+0000 7f51e2516640 1 --2- 172.21.15.42:0/2471944885 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f51dc13c8c0 0x7f51dc138130 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f51d800b440 tx=0x7f51d800b900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:08:19.942 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.908+0000 7f51d27fc640 10 client.?.objecter ms_handle_connect 0x7f51dc13c8c0 2024-09-16T04:08:19.942 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.908+0000 7f51d27fc640 10 client.?.objecter resend_mon_ops 2024-09-16T04:08:19.942 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.908+0000 7f51d27fc640 1 -- 172.21.15.42:0/2471944885 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51d8017070 con 0x7f51dc13c8c0 2024-09-16T04:08:19.942 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.908+0000 7f51d27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:19.942 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.908+0000 7f51d27fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:08:19.942 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.908+0000 7f51d27fc640 10 monclient: dump: 2024-09-16T04:08:19.942 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:08:19.942 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:08:19.943 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:08:19.943 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:08:19.943 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:19.943 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:08:19.943 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:08:19.943 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:08:19.943 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:08:19.943 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:08:19.943 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.908+0000 7f51d27fc640 10 monclient: _finish_auth 0 2024-09-16T04:08:19.944 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.908+0000 7f51d27fc640 10 monclient: _check_auth_tickets 2024-09-16T04:08:19.944 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.908+0000 7f51d27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:19.944 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.908+0000 7f51d27fc640 1 -- 172.21.15.42:0/2471944885 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f51d8003df0 con 0x7f51dc13c8c0 2024-09-16T04:08:19.944 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.908+0000 7f51e3f79640 5 monclient: authenticate success, global_id 1014260 2024-09-16T04:08:19.944 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.908+0000 7f51e3f79640 10 monclient: _renew_subs 2024-09-16T04:08:19.944 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.908+0000 7f51e3f79640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:08:19.945 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.908+0000 7f51e3f79640 1 -- 172.21.15.42:0/2471944885 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51dc139180 con 0x7f51dc13c8c0 2024-09-16T04:08:19.945 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.908+0000 7f51d27fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:19.945 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.908+0000 7f51d27fc640 1 -- 172.21.15.42:0/2471944885 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51d8006310 con 0x7f51dc13c8c0 2024-09-16T04:08:19.945 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.908+0000 7f51d27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:19.946 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.908+0000 7f51d27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:08:19.946 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.908+0000 7f51d27fc640 10 monclient: dump: 2024-09-16T04:08:19.946 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:08:19.946 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:08:19.946 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:08:19.947 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:08:19.947 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:19.947 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:08:19.947 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:08:19.947 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:08:19.947 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:08:19.947 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:08:19.947 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.908+0000 7f51d37fe640 4 set_mon_vals no callback set 2024-09-16T04:08:19.947 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.908+0000 7f51e3f79640 10 client.1014260.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:08:19.947 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.908+0000 7f51e3f79640 10 monclient: _renew_subs 2024-09-16T04:08:19.947 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.908+0000 7f51e3f79640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:08:19.947 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.908+0000 7f51e3f79640 1 -- 172.21.15.42:0/2471944885 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f51dc13c3e0 con 0x7f51dc13c8c0 2024-09-16T04:08:19.947 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.908+0000 7f51e3f79640 1 librados: init done 2024-09-16T04:08:19.948 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.908+0000 7f51e3f79640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:19.948 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.908+0000 7f51e3f79640 1 --2- 172.21.15.42:0/2471944885 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f51dc138670 0x7f51dc138a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:08:19.948 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.908+0000 7f51e3f79640 10 monclient(hunting): start opening mon connection 2024-09-16T04:08:19.948 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.908+0000 7f51e3f79640 1 -- 172.21.15.42:0/2471944885 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f51dc138f90 con 0x7f51dc138670 2024-09-16T04:08:19.948 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.908+0000 7f51d27fc640 1 -- 172.21.15.42:0/2471944885 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f51d80064b0 con 0x7f51dc13c8c0 2024-09-16T04:08:19.948 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.908+0000 7f51d27fc640 1 --2- 172.21.15.42:0/2471944885 >> v2:172.21.15.154:6812/254845342 conn(0x7f51c00377c0 0x7f51c0039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:08:19.948 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.908+0000 7f51d27fc640 1 -- 172.21.15.42:0/2471944885 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(706..706 src has 451..706) ==== 5040+0+0 (secure 0 0 0) 0x7f51d8047e20 con 0x7f51dc13c8c0 2024-09-16T04:08:19.948 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.908+0000 7f51d27fc640 10 client.1014260.objecter ms_dispatch 0x7f51dc000ba0 osd_map(706..706 src has 451..706) 2024-09-16T04:08:19.948 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.908+0000 7f51d27fc640 3 client.1014260.objecter handle_osd_map got epochs [706,706] > 0 2024-09-16T04:08:19.948 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.908+0000 7f51d27fc640 3 client.1014260.objecter handle_osd_map decoding full epoch 706 2024-09-16T04:08:19.948 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.908+0000 7f51d27fc640 20 client.1014260.objecter dump_active .. 0 homeless 2024-09-16T04:08:19.948 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.908+0000 7f51e1d15640 1 --2- 172.21.15.42:0/2471944885 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f51dc138670 0x7f51dc138a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:08:19.948 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.908+0000 7f51e1d15640 10 monclient: get_auth_request con 0x7f51dc138670 auth_method 0 2024-09-16T04:08:19.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.908+0000 7f51e1d15640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:08:19.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.908+0000 7f51e1d15640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:19.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.908+0000 7f51e1d15640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:19.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.908+0000 7f51e1d15640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:08:19.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.908+0000 7f51e1d15640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:08:19.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.908+0000 7f51e1d15640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:08:19.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.908+0000 7f51e1514640 1 --2- 172.21.15.42:0/2471944885 >> v2:172.21.15.154:6812/254845342 conn(0x7f51c00377c0 0x7f51c0039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:08:19.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.908+0000 7f51e1514640 10 monclient: get_auth_request con 0x7f51c00377c0 auth_method 0 2024-09-16T04:08:19.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.908+0000 7f51e1d15640 10 monclient(hunting): handle_auth_done global_id 905164 payload 824 2024-09-16T04:08:19.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.908+0000 7f51e1d15640 1 -- 172.21.15.42:0/2471944885 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f51dc133b80 con 0x7f51dc138670 2024-09-16T04:08:19.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.908+0000 7f51d27fc640 10 client.1014260.objecter ms_handle_connect 0x7f51dc138670 2024-09-16T04:08:19.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.908+0000 7f51d27fc640 10 client.1014260.objecter resend_mon_ops 2024-09-16T04:08:19.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.908+0000 7f51e1d15640 1 --2- 172.21.15.42:0/2471944885 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f51dc138670 0x7f51dc138a50 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f51d40013c0 tx=0x7f51d400cb80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:08:19.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.908+0000 7f51d27fc640 10 client.1014260.objecter ms_handle_connect 0x7f51c00377c0 2024-09-16T04:08:19.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.908+0000 7f51e1514640 1 --2- 172.21.15.42:0/2471944885 >> v2:172.21.15.154:6812/254845342 conn(0x7f51c00377c0 0x7f51c0039c80 secure :-1 s=READY pgs=3899 cs=0 l=1 rev1=1 crypto rx=0x7f51cc0ad000 tx=0x7f51cc0a6000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:08:19.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.908+0000 7f51d27fc640 1 -- 172.21.15.42:0/2471944885 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51d4005d50 con 0x7f51dc138670 2024-09-16T04:08:19.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.908+0000 7f51d27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:19.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.908+0000 7f51d27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:08:19.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.908+0000 7f51d27fc640 10 monclient: dump: 2024-09-16T04:08:19.951 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:08:19.951 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:08:19.951 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:08:19.951 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:08:19.951 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:19.951 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:08:19.951 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:08:19.951 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:08:19.951 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:08:19.951 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:08:19.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.908+0000 7f51d27fc640 1 -- 172.21.15.42:0/2471944885 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f51d4005d50 con 0x7f51dc138670 2024-09-16T04:08:19.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.908+0000 7f51d27fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:19.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.908+0000 7f51d27fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:08:19.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.908+0000 7f51d27fc640 1 -- 172.21.15.42:0/2471944885 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f51dc138670 msgr2=0x7f51dc138a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:19.952 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.908+0000 7f51d27fc640 1 --2- 172.21.15.42:0/2471944885 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f51dc138670 0x7f51dc138a50 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f51d40013c0 tx=0x7f51d400cb80 comp rx=0 tx=0).stop 2024-09-16T04:08:19.952 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.912+0000 7f51d27fc640 1 -- 172.21.15.42:0/2471944885 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f51dc138670 msgr2=0x7f51dc138a50 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:08:19.952 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.912+0000 7f51d27fc640 1 --2- 172.21.15.42:0/2471944885 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f51dc138670 0x7f51dc138a50 secure :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f51d40013c0 tx=0x7f51d400cb80 comp rx=0 tx=0).stop 2024-09-16T04:08:19.954 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.916+0000 7f51e3f79640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:08:19.955 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.916+0000 7f51e3f79640 1 --2- 172.21.15.42:0/2471944885 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f51dc12f500 0x7f51dc12f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:08:19.955 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.916+0000 7f51e3f79640 10 monclient(hunting): start opening mon connection 2024-09-16T04:08:19.955 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.916+0000 7f51e3f79640 1 -- 172.21.15.42:0/2471944885 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f51dc0a5750 con 0x7f51dc12f500 2024-09-16T04:08:19.955 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.916+0000 7f51e1d15640 1 --2- 172.21.15.42:0/2471944885 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f51dc12f500 0x7f51dc12f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:08:19.955 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.916+0000 7f51e1d15640 10 monclient: get_auth_request con 0x7f51dc12f500 auth_method 0 2024-09-16T04:08:19.956 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.916+0000 7f51e1d15640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:08:19.956 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.916+0000 7f51e1d15640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:19.956 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.916+0000 7f51e1d15640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:19.956 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.916+0000 7f51e1d15640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:08:19.956 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.916+0000 7f51e1d15640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:08:19.957 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.916+0000 7f51e1d15640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:08:19.957 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.916+0000 7f51e1d15640 10 monclient(hunting): handle_auth_done global_id 905167 payload 824 2024-09-16T04:08:19.957 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.916+0000 7f51e1d15640 1 -- 172.21.15.42:0/2471944885 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f51dc12e0a0 con 0x7f51dc12f500 2024-09-16T04:08:19.957 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.916+0000 7f51e1d15640 1 --2- 172.21.15.42:0/2471944885 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f51dc12f500 0x7f51dc12f8e0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f51d4013000 tx=0x7f51d4001260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:08:19.957 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.916+0000 7f51d27fc640 10 client.1014260.objecter ms_handle_connect 0x7f51dc12f500 2024-09-16T04:08:19.957 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.916+0000 7f51d27fc640 10 client.1014260.objecter resend_mon_ops 2024-09-16T04:08:19.958 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.916+0000 7f51d27fc640 1 -- 172.21.15.42:0/2471944885 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51d400aa70 con 0x7f51dc12f500 2024-09-16T04:08:19.958 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.916+0000 7f51d27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:19.958 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.916+0000 7f51d27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:08:19.958 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.916+0000 7f51d27fc640 10 monclient: dump: 2024-09-16T04:08:19.958 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:08:19.959 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:08:19.959 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:08:19.959 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:08:19.959 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:19.959 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:08:19.959 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:08:19.960 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:08:19.960 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:08:19.960 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:08:19.961 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.920+0000 7f51d27fc640 1 -- 172.21.15.42:0/2471944885 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4871 (secure 0 0 0) 0x7f51d400aa70 con 0x7f51dc12f500 2024-09-16T04:08:19.961 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.920+0000 7f51d27fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:08:19.961 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.920+0000 7f51d27fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:08:19.961 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.920+0000 7f51d27fc640 1 -- 172.21.15.42:0/2471944885 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f51dc12f500 msgr2=0x7f51dc12f8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:19.961 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.920+0000 7f51d27fc640 1 --2- 172.21.15.42:0/2471944885 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f51dc12f500 0x7f51dc12f8e0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f51d4013000 tx=0x7f51d4001260 comp rx=0 tx=0).stop 2024-09-16T04:08:19.961 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.920+0000 7f51d27fc640 1 -- 172.21.15.42:0/2471944885 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f51dc12f500 msgr2=0x7f51dc12f8e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:08:19.962 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.920+0000 7f51d27fc640 1 --2- 172.21.15.42:0/2471944885 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f51dc12f500 0x7f51dc12f8e0 secure :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f51d4013000 tx=0x7f51d4001260 comp rx=0 tx=0).stop 2024-09-16T04:08:19.962 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T04:08:19.962 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:08:19.962 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:08:19.963 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "leader", 2024-09-16T04:08:19.963 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 240, 2024-09-16T04:08:19.963 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 628100, 2024-09-16T04:08:19.963 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T04:08:19.963 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T04:08:19.964 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T04:08:19.964 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:08:19.964 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 1, 2024-09-16T04:08:19.964 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:08:19.964 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T04:08:19.964 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T04:08:19.964 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:08:19.964 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:08:19.965 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:08:19.965 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:08:19.965 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:08:19.965 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:08:19.965 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:08:19.965 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:08:19.965 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:08:19.965 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:08:19.965 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:08:19.965 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:08:19.965 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:08:19.965 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T04:08:19.965 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:08:19.965 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:08:19.966 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:08:19.966 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:08:19.966 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:08:19.966 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:08:19.966 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:08:19.966 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:08:19.966 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:08:19.966 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:08:19.966 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:08:19.966 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:08:19.966 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:08:19.966 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T04:08:19.966 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T04:08:19.966 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T04:08:19.966 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T04:08:19.967 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T04:08:19.967 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T04:08:19.967 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:08:19.967 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:08:19.967 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T04:08:19.967 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T04:08:19.967 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T04:08:19.967 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T04:08:19.967 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T04:08:19.967 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T04:08:19.967 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T04:08:19.968 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:08:19.968 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T04:08:19.968 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:08:19.968 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:08:19.968 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:08:19.968 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:08:19.968 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:08:19.968 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:08:19.968 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:08:19.968 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:08:19.968 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:08:19.968 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:08:19.968 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:08:19.969 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:08:19.969 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T04:08:19.969 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:08:19.969 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T04:08:19.969 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:08:19.969 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:08:19.969 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:08:19.969 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:08:19.969 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:08:19.969 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:08:19.969 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:08:19.969 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T04:08:19.969 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:08:19.969 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:08:19.970 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:08:19.970 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:08:19.970 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T04:08:19.970 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:08:19.970 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:08:19.970 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:08:19.970 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:08:19.970 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T04:08:19.970 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T04:08:19.970 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:08:19.970 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:08:19.970 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:08:19.970 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:08:19.970 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:08:19.970 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:08:19.971 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:08:19.971 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:08:19.971 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:08:19.971 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:08:19.971 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:08:19.971 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T04:08:19.971 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:08:19.971 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:08:19.971 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:08:19.971 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:08:19.972 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T04:08:19.972 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:08:19.972 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:08:19.972 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:08:19.972 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:08:19.972 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T04:08:19.972 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T04:08:19.972 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:08:19.972 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:08:19.972 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:08:19.972 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:08:19.972 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:08:19.972 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:08:19.972 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:08:19.973 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:08:19.973 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:08:19.973 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:08:19.973 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:08:19.973 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T04:08:19.973 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:08:19.973 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:08:19.973 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:08:19.973 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:08:19.973 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T04:08:19.973 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:08:19.973 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:08:19.973 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:08:19.973 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:08:19.973 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T04:08:19.974 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T04:08:19.974 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:08:19.974 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:08:19.974 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:08:19.974 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:08:19.974 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:08:19.974 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:08:19.974 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T04:08:19.974 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T04:08:19.974 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:08:19.974 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:08:19.974 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:08:19.974 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:08:19.974 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:08:19.975 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:08:19.975 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T04:08:19.975 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:08:19.975 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:08:19.975 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:08:19.975 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 3 2024-09-16T04:08:19.975 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:08:19.975 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:08:19.975 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T04:08:19.975 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:08:19.975 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:08:19.975 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:08:19.975 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 5 2024-09-16T04:08:19.975 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:08:19.975 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:08:19.976 INFO:teuthology.orchestra.run.smithi042.stdout: "mgr": [ 2024-09-16T04:08:19.976 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:08:19.976 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:08:19.976 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:08:19.976 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:08:19.976 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:08:19.976 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:08:19.976 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:08:19.976 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T04:08:19.976 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T04:08:19.978 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.920+0000 7f51e3f79640 10 librados: watch_flush enter 2024-09-16T04:08:19.978 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.920+0000 7f51e3f79640 10 librados: watch_flush exit 2024-09-16T04:08:19.978 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.920+0000 7f51e3f79640 20 client.1014260.objecter shutdown clearing up homeless session... 2024-09-16T04:08:19.978 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.920+0000 7f51e3f79640 10 client.1014260.objecter successfully canceled tick 2024-09-16T04:08:19.978 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.920+0000 7f51e3f79640 1 -- 172.21.15.42:0/2471944885 >> v2:172.21.15.154:6812/254845342 conn(0x7f51c00377c0 msgr2=0x7f51c0039c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:19.978 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.920+0000 7f51e3f79640 1 --2- 172.21.15.42:0/2471944885 >> v2:172.21.15.154:6812/254845342 conn(0x7f51c00377c0 0x7f51c0039c80 secure :-1 s=READY pgs=3899 cs=0 l=1 rev1=1 crypto rx=0x7f51cc0ad000 tx=0x7f51cc0a6000 comp rx=0 tx=0).stop 2024-09-16T04:08:19.979 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.920+0000 7f51e3f79640 10 monclient: shutdown 2024-09-16T04:08:19.979 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.920+0000 7f51e3f79640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:19.979 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.920+0000 7f51e3f79640 1 -- 172.21.15.42:0/2471944885 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f51dc13c8c0 msgr2=0x7f51dc138130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:19.979 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.920+0000 7f51e3f79640 1 --2- 172.21.15.42:0/2471944885 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f51dc13c8c0 0x7f51dc138130 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f51d800b440 tx=0x7f51d800b900 comp rx=0 tx=0).stop 2024-09-16T04:08:19.979 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.920+0000 7f51e2516640 1 -- 172.21.15.42:0/2471944885 reap_dead start 2024-09-16T04:08:19.979 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.920+0000 7f51e3f79640 1 -- 172.21.15.42:0/2471944885 shutdown_connections 2024-09-16T04:08:19.979 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.920+0000 7f51e3f79640 1 --2- 172.21.15.42:0/2471944885 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f51dc13c8c0 0x7f51dc138130 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:08:19.979 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.920+0000 7f51e3f79640 1 -- 172.21.15.42:0/2471944885 >> 172.21.15.42:0/2471944885 conn(0x7f51dc0a9fb0 msgr2=0x7f51dc0a8ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:19.979 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.920+0000 7f51e3f79640 1 -- 172.21.15.42:0/2471944885 shutdown_connections 2024-09-16T04:08:19.979 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.920+0000 7f51e3f79640 1 -- 172.21.15.42:0/2471944885 wait complete. 2024-09-16T04:08:19.980 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:19.920+0000 7f51e3f79640 1 librados: shutdown 2024-09-16T04:08:19.980 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T04:08:19.980 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-16T04:08:39.981 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-16T04:08:39.982 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-09-16T04:08:39.982 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-16T04:08:39.982 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-16T04:08:39.985 INFO:tasks.ceph.mon.a:Started 2024-09-16T04:08:39.985 INFO:tasks.mon_thrash:Sending STOP to mon b 2024-09-16T04:08:39.986 INFO:tasks.ceph.mon.b:Sent signal 19 2024-09-16T04:08:39.986 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-16T04:08:54.987 INFO:tasks.mon_thrash:Sending CONT to mon b 2024-09-16T04:08:54.988 INFO:tasks.ceph.mon.b:Sent signal 18 2024-09-16T04:08:54.988 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T04:08:54.988 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:08:55.119 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.112+0000 7fc785724640 10 monclient: get_monmap_and_config 2024-09-16T04:08:55.119 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.112+0000 7fc785724640 10 monclient: build_initial_monmap 2024-09-16T04:08:55.119 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.112+0000 7fc785724640 10 monclient: monmap: 2024-09-16T04:08:55.119 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:08:55.119 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:08:55.119 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:08:55.117853+0000 2024-09-16T04:08:55.120 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:08:55.117853+0000 2024-09-16T04:08:55.120 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:55.120 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:08:55.120 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:08:55.120 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:08:55.120 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:08:55.120 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:08:55.120 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:08:55.120 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:08:55.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.112+0000 7fc785724640 1 Processor -- start 2024-09-16T04:08:55.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.112+0000 7fc785724640 1 -- start start 2024-09-16T04:08:55.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.112+0000 7fc785724640 10 monclient: init 2024-09-16T04:08:55.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.112+0000 7fc785724640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:55.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.112+0000 7fc785724640 10 monclient: _add_conns ranks=[0,4,1,2,3] 2024-09-16T04:08:55.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.112+0000 7fc785724640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc7800a1330 addr v1:172.21.15.42:6789/0 2024-09-16T04:08:55.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.112+0000 7fc785724640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fc78006dac0 0x7fc78013a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:08:55.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.112+0000 7fc785724640 10 monclient(hunting): picked mon.noname-c con 0x7fc78006dac0 addr v2:172.21.15.154:3300/0 2024-09-16T04:08:55.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.112+0000 7fc785724640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc7800a1e20 addr v1:172.21.15.154:6789/0 2024-09-16T04:08:55.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.112+0000 7fc785724640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc78009c090 con 0x7fc7800a1330 2024-09-16T04:08:55.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.112+0000 7fc785724640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc78009f610 con 0x7fc7800a1e20 2024-09-16T04:08:55.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.112+0000 7fc785724640 10 monclient(hunting): start opening mon connection 2024-09-16T04:08:55.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.112+0000 7fc785724640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fc78009f8f0 con 0x7fc78006dac0 2024-09-16T04:08:55.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.112+0000 7fc785724640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:55.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.112+0000 7fc785724640 10 monclient(hunting): authenticate will time out at 2582.087339s 2024-09-16T04:08:55.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.112+0000 7fc77effd640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fc7800a1330 0x7fc7800a1710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:34780/0 (socket says 172.21.15.42:34780) 2024-09-16T04:08:55.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.112+0000 7fc77effd640 1 -- 172.21.15.42:0/1460883585 learned_addr learned my addr 172.21.15.42:0/1460883585 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T04:08:55.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.112+0000 7fc77e7fc640 1 --2- 172.21.15.42:0/1460883585 >> v2:172.21.15.154:3300/0 conn(0x7fc78006dac0 0x7fc78013a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:08:55.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.112+0000 7fc77e7fc640 10 monclient(hunting): get_auth_request con 0x7fc78006dac0 auth_method 0 2024-09-16T04:08:55.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.112+0000 7fc77e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:08:55.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.112+0000 7fc77e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:55.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.112+0000 7fc77e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:55.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.112+0000 7fc77e7fc640 1 -- 172.21.15.42:0/1460883585 >> v2:172.21.15.154:3300/0 conn(0x7fc78006dac0 msgr2=0x7fc78013a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T04:08:55.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.112+0000 7fc77e7fc640 1 -- 172.21.15.42:0/1460883585 >> v2:172.21.15.154:3300/0 conn(0x7fc78006dac0 msgr2=0x7fc78013a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:08:55.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.112+0000 7fc77e7fc640 1 --2- 172.21.15.42:0/1460883585 >> v2:172.21.15.154:3300/0 conn(0x7fc78006dac0 0x7fc78013a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:08:55.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.116+0000 7fc77e7fc640 1 --2- 172.21.15.42:0/1460883585 >> v2:172.21.15.154:3300/0 conn(0x7fc78006dac0 0x7fc78013a340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:08:55.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.320+0000 7fc77e7fc640 1 --2- 172.21.15.42:0/1460883585 >> v2:172.21.15.154:3300/0 conn(0x7fc78006dac0 0x7fc78013a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:08:55.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.320+0000 7fc77e7fc640 10 monclient(hunting): get_auth_request con 0x7fc78006dac0 auth_method 0 2024-09-16T04:08:55.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.320+0000 7fc77e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:08:55.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.320+0000 7fc77e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:55.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.320+0000 7fc77e7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:08:55.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.320+0000 7fc77e7fc640 1 -- 172.21.15.42:0/1460883585 >> v2:172.21.15.154:3300/0 conn(0x7fc78006dac0 msgr2=0x7fc78013a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T04:08:55.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.320+0000 7fc77e7fc640 1 -- 172.21.15.42:0/1460883585 >> v2:172.21.15.154:3300/0 conn(0x7fc78006dac0 msgr2=0x7fc78013a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:08:55.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.320+0000 7fc77e7fc640 1 --2- 172.21.15.42:0/1460883585 >> v2:172.21.15.154:3300/0 conn(0x7fc78006dac0 0x7fc78013a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:08:55.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.320+0000 7fc77e7fc640 1 --2- 172.21.15.42:0/1460883585 >> v2:172.21.15.154:3300/0 conn(0x7fc78006dac0 0x7fc78013a340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T04:08:55.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.396+0000 7fc77dffb640 1 -- 172.21.15.42:0/1460883585 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2951973161 0 0) 0x7fc78009c090 con 0x7fc7800a1330 2024-09-16T04:08:55.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.396+0000 7fc77dffb640 10 monclient(hunting): _negotiate 2024-09-16T04:08:55.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.396+0000 7fc77dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:55.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.396+0000 7fc77dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:55.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.396+0000 7fc77dffb640 10 monclient(hunting): my global_id is 1034097 2024-09-16T04:08:55.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.396+0000 7fc77dffb640 1 -- 172.21.15.42:0/1460883585 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc76c003530 con 0x7fc7800a1330 2024-09-16T04:08:55.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.400+0000 7fc77dffb640 1 -- 172.21.15.42:0/1460883585 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2926586282 0 0) 0x7fc76c003530 con 0x7fc7800a1330 2024-09-16T04:08:55.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.400+0000 7fc77dffb640 1 -- 172.21.15.42:0/1460883585 >> v2:172.21.15.154:3300/0 conn(0x7fc78006dac0 msgr2=0x7fc78013a340 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:08:55.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.400+0000 7fc77dffb640 1 --2- 172.21.15.42:0/1460883585 >> v2:172.21.15.154:3300/0 conn(0x7fc78006dac0 0x7fc78013a340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:08:55.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.400+0000 7fc77dffb640 1 -- 172.21.15.42:0/1460883585 >> v1:172.21.15.154:6789/0 conn(0x7fc7800a1e20 legacy=0x7fc78013c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:55.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.400+0000 7fc77dffb640 10 monclient: _finish_hunting 0 2024-09-16T04:08:55.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.400+0000 7fc77dffb640 1 monclient: found mon.noname-a-legacy 2024-09-16T04:08:55.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.400+0000 7fc77dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:55.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.400+0000 7fc77dffb640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.42:6789/0 2024-09-16T04:08:55.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.400+0000 7fc77dffb640 1 -- 172.21.15.42:0/1460883585 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc78009fbd0 con 0x7fc7800a1330 2024-09-16T04:08:55.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.400+0000 7fc77dffb640 10 monclient: _finish_auth 0 2024-09-16T04:08:55.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.400+0000 7fc77dffb640 10 monclient: _check_auth_tickets 2024-09-16T04:08:55.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.400+0000 7fc77dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:55.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.400+0000 7fc77dffb640 1 -- 172.21.15.42:0/1460883585 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2177413734 0 0) 0x7fc768001680 con 0x7fc7800a1330 2024-09-16T04:08:55.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.400+0000 7fc785724640 5 monclient: authenticate success, global_id 1034097 2024-09-16T04:08:55.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.400+0000 7fc785724640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:08:55.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.400+0000 7fc77dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:55.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.400+0000 7fc77dffb640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T04:08:55.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.400+0000 7fc77dffb640 10 monclient: dump: 2024-09-16T04:08:55.410 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:08:55.410 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:08:55.410 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:08:55.410 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:08:55.410 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:55.410 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:08:55.410 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:08:55.410 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:08:55.410 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:08:55.411 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:08:55.411 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.400+0000 7fc77dffb640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] but i'm connected to v1:172.21.15.42:6789/0, reconnecting 2024-09-16T04:08:55.411 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.400+0000 7fc77dffb640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:55.411 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.400+0000 7fc77dffb640 1 -- 172.21.15.42:0/1460883585 >> v1:172.21.15.42:6789/0 conn(0x7fc7800a1330 legacy=0x7fc7800a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:55.411 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.400+0000 7fc77dffb640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:08:55.411 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.400+0000 7fc77dffb640 1 --2- 172.21.15.42:0/1460883585 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc76c003a10 0x7fc76c003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:08:55.411 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.400+0000 7fc77dffb640 10 monclient(hunting): picked mon.c con 0x7fc76c003a10 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:08:55.411 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.400+0000 7fc77dffb640 1 --2- 172.21.15.42:0/1460883585 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc76c004810 0x7fc76c004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:08:55.411 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.400+0000 7fc77dffb640 10 monclient(hunting): picked mon.a con 0x7fc76c004810 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:08:55.411 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.400+0000 7fc77dffb640 1 --2- 172.21.15.42:0/1460883585 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc76c0052f0 0x7fc76c00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:08:55.411 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.400+0000 7fc77dffb640 10 monclient(hunting): picked mon.b con 0x7fc76c0052f0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:08:55.411 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.400+0000 7fc77dffb640 10 monclient(hunting): start opening mon connection 2024-09-16T04:08:55.411 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.400+0000 7fc77dffb640 1 -- 172.21.15.42:0/1460883585 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fc78009f8f0 con 0x7fc76c004810 2024-09-16T04:08:55.411 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.400+0000 7fc77dffb640 10 monclient(hunting): start opening mon connection 2024-09-16T04:08:55.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.400+0000 7fc77dffb640 1 -- 172.21.15.42:0/1460883585 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc76c00c210 con 0x7fc76c0052f0 2024-09-16T04:08:55.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.400+0000 7fc77dffb640 10 monclient(hunting): start opening mon connection 2024-09-16T04:08:55.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.400+0000 7fc77dffb640 1 -- 172.21.15.42:0/1460883585 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fc76c00c410 con 0x7fc76c003a10 2024-09-16T04:08:55.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.400+0000 7fc77dffb640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:55.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.400+0000 7fc77dffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:08:55.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.400+0000 7fc785724640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-16T04:08:55.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.400+0000 7fc77effd640 1 --2- 172.21.15.42:0/1460883585 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc76c003a10 0x7fc76c003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:08:55.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.400+0000 7fc77effd640 10 monclient(hunting): get_auth_request con 0x7fc76c003a10 auth_method 0 2024-09-16T04:08:55.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.400+0000 7fc77effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:08:55.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.400+0000 7fc77effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:55.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.400+0000 7fc77effd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:08:55.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.400+0000 7fc77e7fc640 1 --2- 172.21.15.42:0/1460883585 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc76c004810 0x7fc76c004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:08:55.413 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.400+0000 7fc77f7fe640 1 --2- 172.21.15.42:0/1460883585 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc76c0052f0 0x7fc76c00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:08:55.413 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.400+0000 7fc77e7fc640 10 monclient(hunting): get_auth_request con 0x7fc76c004810 auth_method 0 2024-09-16T04:08:55.413 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.400+0000 7fc77e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:08:55.413 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.400+0000 7fc77e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:55.413 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.400+0000 7fc77e7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:08:55.413 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.400+0000 7fc77f7fe640 10 monclient(hunting): get_auth_request con 0x7fc76c0052f0 auth_method 0 2024-09-16T04:08:55.413 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.400+0000 7fc77f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:08:55.413 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.400+0000 7fc77f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:55.413 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.400+0000 7fc77f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:08:55.413 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.400+0000 7fc77effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:08:55.413 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.400+0000 7fc77effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:08:55.413 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.400+0000 7fc77effd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:08:55.413 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.400+0000 7fc77e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:08:55.413 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.400+0000 7fc77e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:08:55.414 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.400+0000 7fc77e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:08:55.414 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.400+0000 7fc77f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:08:55.414 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.400+0000 7fc77f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:08:55.414 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.400+0000 7fc77f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:08:55.414 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.400+0000 7fc77effd640 10 monclient(hunting): handle_auth_done global_id 1034097 payload 293 2024-09-16T04:08:55.414 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.400+0000 7fc77effd640 1 -- 172.21.15.42:0/1460883585 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc76c0052f0 msgr2=0x7fc76c00b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:08:55.414 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.400+0000 7fc77effd640 1 --2- 172.21.15.42:0/1460883585 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc76c0052f0 0x7fc76c00b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:08:55.414 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.400+0000 7fc77effd640 1 -- 172.21.15.42:0/1460883585 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc76c004810 msgr2=0x7fc76c004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:08:55.414 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.400+0000 7fc77effd640 1 --2- 172.21.15.42:0/1460883585 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc76c004810 0x7fc76c004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:08:55.414 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.400+0000 7fc77effd640 10 monclient: _finish_hunting 0 2024-09-16T04:08:55.414 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.400+0000 7fc77effd640 1 monclient: found mon.c 2024-09-16T04:08:55.414 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.400+0000 7fc77effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:55.415 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.400+0000 7fc77effd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:08:55.415 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.400+0000 7fc77effd640 1 -- 172.21.15.42:0/1460883585 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc76c003530 con 0x7fc76c003a10 2024-09-16T04:08:55.415 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.400+0000 7fc77effd640 10 monclient: _finish_auth 0 2024-09-16T04:08:55.415 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.400+0000 7fc77effd640 10 monclient: _check_auth_tickets 2024-09-16T04:08:55.415 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.400+0000 7fc77effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:55.415 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.400+0000 7fc77f7fe640 1 -- 172.21.15.42:0/1460883585 reap_dead start 2024-09-16T04:08:55.420 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.404+0000 7fc77effd640 1 --2- 172.21.15.42:0/1460883585 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc76c003a10 0x7fc76c003e30 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fc76800c110 tx=0x7fc76801cb60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d5685c9a1791814 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:08:55.420 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.404+0000 7fc77dffb640 1 -- 172.21.15.42:0/1460883585 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc768007c50 con 0x7fc76c003a10 2024-09-16T04:08:55.420 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.404+0000 7fc77dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:55.420 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.404+0000 7fc77dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:08:55.421 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.404+0000 7fc77dffb640 10 monclient: dump: 2024-09-16T04:08:55.421 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:08:55.421 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:08:55.421 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:08:55.421 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:08:55.421 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:55.421 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:08:55.421 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:08:55.421 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:08:55.421 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:08:55.421 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:08:55.421 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.404+0000 7fc77dffb640 1 -- 172.21.15.42:0/1460883585 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc7680a9070 con 0x7fc76c003a10 2024-09-16T04:08:55.421 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.404+0000 7fc785724640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:08:55.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.404+0000 7fc77dffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:55.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.404+0000 7fc785724640 10 monclient: get_monmap_and_config success 2024-09-16T04:08:55.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.404+0000 7fc785724640 4 set_mon_vals no callback set 2024-09-16T04:08:55.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.404+0000 7fc77dffb640 1 -- 172.21.15.42:0/1460883585 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc768005d80 con 0x7fc76c003a10 2024-09-16T04:08:55.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.404+0000 7fc77dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:55.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.404+0000 7fc77dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:08:55.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.404+0000 7fc77dffb640 10 monclient: dump: 2024-09-16T04:08:55.422 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:08:55.422 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:08:55.422 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:08:55.422 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:08:55.422 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:55.423 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:08:55.423 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:08:55.423 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:08:55.423 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:08:55.423 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:08:55.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.408+0000 7fc785724640 10 monclient: shutdown 2024-09-16T04:08:55.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.408+0000 7fc785724640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:55.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.408+0000 7fc785724640 1 -- 172.21.15.42:0/1460883585 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc76c003a10 msgr2=0x7fc76c003e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:55.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.408+0000 7fc785724640 1 --2- 172.21.15.42:0/1460883585 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc76c003a10 0x7fc76c003e30 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fc76800c110 tx=0x7fc76801cb60 comp rx=0 tx=0).stop 2024-09-16T04:08:55.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.408+0000 7fc785724640 1 -- 172.21.15.42:0/1460883585 shutdown_connections 2024-09-16T04:08:55.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.408+0000 7fc785724640 1 --2- 172.21.15.42:0/1460883585 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc76c003a10 0x7fc76c003e30 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:08:55.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.408+0000 7fc785724640 1 -- 172.21.15.42:0/1460883585 >> 172.21.15.42:0/1460883585 conn(0x7fc7800a2420 msgr2=0x7fc7800a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:55.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.408+0000 7fc785724640 1 -- 172.21.15.42:0/1460883585 shutdown_connections 2024-09-16T04:08:55.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.408+0000 7fc785724640 1 -- 172.21.15.42:0/1460883585 wait complete. 2024-09-16T04:08:55.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.408+0000 7fc785724640 10 monclient: build_initial_monmap 2024-09-16T04:08:55.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.408+0000 7fc785724640 10 monclient: monmap: 2024-09-16T04:08:55.424 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:08:55.424 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:08:55.424 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:08:55.424 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:08:55.424 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:55.424 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:08:55.424 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:08:55.424 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:08:55.424 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:08:55.424 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:08:55.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.408+0000 7fc785724640 1 librados: starting msgr at 2024-09-16T04:08:55.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.408+0000 7fc785724640 1 librados: starting objecter 2024-09-16T04:08:55.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.408+0000 7fc785724640 1 Processor -- start 2024-09-16T04:08:55.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.408+0000 7fc785724640 1 -- start start 2024-09-16T04:08:55.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.408+0000 7fc785724640 1 librados: setting wanted keys 2024-09-16T04:08:55.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.408+0000 7fc785724640 1 librados: calling monclient init 2024-09-16T04:08:55.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.408+0000 7fc785724640 10 monclient: init 2024-09-16T04:08:55.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.408+0000 7fc785724640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:55.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.408+0000 7fc785724640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:08:55.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.408+0000 7fc785724640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc76c003a10 0x7fc7801cf710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:08:55.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.408+0000 7fc785724640 10 monclient(hunting): picked mon.noname-b con 0x7fc76c003a10 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:08:55.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.408+0000 7fc785724640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc7801cfc50 0x7fc780123f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:08:55.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.408+0000 7fc785724640 10 monclient(hunting): picked mon.noname-a con 0x7fc7801cfc50 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:08:55.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.408+0000 7fc785724640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc7801244a0 0x7fc780124920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:08:55.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.408+0000 7fc785724640 10 monclient(hunting): picked mon.noname-c con 0x7fc7801244a0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:08:55.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.408+0000 7fc785724640 10 monclient(hunting): start opening mon connection 2024-09-16T04:08:55.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.408+0000 7fc785724640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fc780135910 con 0x7fc7801cfc50 2024-09-16T04:08:55.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.408+0000 7fc785724640 10 monclient(hunting): start opening mon connection 2024-09-16T04:08:55.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.408+0000 7fc785724640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc78009ef80 con 0x7fc76c003a10 2024-09-16T04:08:55.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.408+0000 7fc785724640 10 monclient(hunting): start opening mon connection 2024-09-16T04:08:55.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.408+0000 7fc785724640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fc780135a90 con 0x7fc7801244a0 2024-09-16T04:08:55.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.408+0000 7fc785724640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:55.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.408+0000 7fc785724640 10 monclient(hunting): authenticate will time out at 2582.383001s 2024-09-16T04:08:55.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.408+0000 7fc77f7fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc7801244a0 0x7fc780124920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:08:55.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.408+0000 7fc77f7fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc7801244a0 0x7fc780124920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:40238/0 (socket says 172.21.15.42:40238) 2024-09-16T04:08:55.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.408+0000 7fc77e7fc640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc7801cfc50 0x7fc780123f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:08:55.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.408+0000 7fc77f7fe640 1 -- 172.21.15.42:0/1383243527 learned_addr learned my addr 172.21.15.42:0/1383243527 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:08:55.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.408+0000 7fc77f7fe640 10 monclient(hunting): get_auth_request con 0x7fc7801244a0 auth_method 0 2024-09-16T04:08:55.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.408+0000 7fc77f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:08:55.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.408+0000 7fc77f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:55.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.408+0000 7fc77f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:55.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.408+0000 7fc77e7fc640 10 monclient(hunting): get_auth_request con 0x7fc7801cfc50 auth_method 0 2024-09-16T04:08:55.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.408+0000 7fc77e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:08:55.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.408+0000 7fc77e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:55.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.408+0000 7fc77e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:55.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.408+0000 7fc77f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:08:55.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.408+0000 7fc77f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:08:55.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.408+0000 7fc77effd640 1 --2- 172.21.15.42:0/1383243527 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc76c003a10 0x7fc7801cf710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:08:55.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.408+0000 7fc77f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:08:55.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.408+0000 7fc77effd640 10 monclient(hunting): get_auth_request con 0x7fc76c003a10 auth_method 0 2024-09-16T04:08:55.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.408+0000 7fc77effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:08:55.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.408+0000 7fc77effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:55.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.408+0000 7fc77effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:55.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.408+0000 7fc77e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:08:55.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.408+0000 7fc77e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:08:55.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.408+0000 7fc77e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:08:55.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.408+0000 7fc77effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:08:55.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.408+0000 7fc77effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:08:55.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.408+0000 7fc77effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:08:55.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.408+0000 7fc77f7fe640 10 monclient(hunting): handle_auth_done global_id 1014263 payload 824 2024-09-16T04:08:55.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.408+0000 7fc77f7fe640 1 -- 172.21.15.42:0/1383243527 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc76c003a10 msgr2=0x7fc7801cf710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:55.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.408+0000 7fc77f7fe640 1 --2- 172.21.15.42:0/1383243527 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc76c003a10 0x7fc7801cf710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:08:55.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.408+0000 7fc77f7fe640 1 -- 172.21.15.42:0/1383243527 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc7801cfc50 msgr2=0x7fc780123f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:55.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.408+0000 7fc77f7fe640 1 --2- 172.21.15.42:0/1383243527 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc7801cfc50 0x7fc780123f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:08:55.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.408+0000 7fc77f7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:08:55.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.408+0000 7fc77f7fe640 1 monclient: found mon.noname-c 2024-09-16T04:08:55.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.408+0000 7fc77f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:55.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.408+0000 7fc77f7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T04:08:55.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.408+0000 7fc77f7fe640 1 -- 172.21.15.42:0/1383243527 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc780071210 con 0x7fc7801244a0 2024-09-16T04:08:55.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.408+0000 7fc77e7fc640 1 --2- 172.21.15.42:0/1383243527 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc7801cfc50 0x7fc780123f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:08:55.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.412+0000 7fc77f7fe640 1 --2- 172.21.15.42:0/1383243527 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc7801244a0 0x7fc780124920 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fc780068f60 tx=0x7fc77800f900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:08:55.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.412+0000 7fc7777fe640 10 client.?.objecter ms_handle_connect 0x7fc7801244a0 2024-09-16T04:08:55.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.412+0000 7fc7777fe640 10 client.?.objecter resend_mon_ops 2024-09-16T04:08:55.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.412+0000 7fc7777fe640 1 -- 172.21.15.42:0/1383243527 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7780052e0 con 0x7fc7801244a0 2024-09-16T04:08:55.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.412+0000 7fc7777fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:55.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.412+0000 7fc7777fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:08:55.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.412+0000 7fc7777fe640 10 monclient: dump: 2024-09-16T04:08:55.430 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:08:55.430 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:08:55.430 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:08:55.430 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:08:55.430 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:55.430 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:08:55.430 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:08:55.430 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:08:55.430 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:08:55.430 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:08:55.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.412+0000 7fc7777fe640 10 monclient: _finish_auth 0 2024-09-16T04:08:55.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.412+0000 7fc7777fe640 10 monclient: _check_auth_tickets 2024-09-16T04:08:55.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.412+0000 7fc7777fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:55.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.412+0000 7fc7777fe640 1 -- 172.21.15.42:0/1383243527 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc778018070 con 0x7fc7801244a0 2024-09-16T04:08:55.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.412+0000 7fc785724640 5 monclient: authenticate success, global_id 1014263 2024-09-16T04:08:55.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.412+0000 7fc785724640 10 monclient: _renew_subs 2024-09-16T04:08:55.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.412+0000 7fc785724640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:08:55.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.412+0000 7fc785724640 1 -- 172.21.15.42:0/1383243527 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc780126720 con 0x7fc7801244a0 2024-09-16T04:08:55.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.412+0000 7fc7777fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:55.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.412+0000 7fc7777fe640 1 -- 172.21.15.42:0/1383243527 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc778003820 con 0x7fc7801244a0 2024-09-16T04:08:55.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.412+0000 7fc7777fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:55.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.412+0000 7fc7777fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:08:55.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.412+0000 7fc7777fe640 10 monclient: dump: 2024-09-16T04:08:55.431 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:08:55.431 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:08:55.432 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:08:55.432 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:08:55.432 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:55.432 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:08:55.432 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:08:55.432 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:08:55.432 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:08:55.432 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:08:55.432 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.412+0000 7fc77d7fa640 4 set_mon_vals no callback set 2024-09-16T04:08:55.432 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.412+0000 7fc785724640 10 client.1014263.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:08:55.432 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.412+0000 7fc785724640 10 monclient: _renew_subs 2024-09-16T04:08:55.432 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.412+0000 7fc785724640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:08:55.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.412+0000 7fc785724640 1 -- 172.21.15.42:0/1383243527 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc78006e660 con 0x7fc7801244a0 2024-09-16T04:08:55.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.412+0000 7fc785724640 1 librados: init done 2024-09-16T04:08:55.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.412+0000 7fc7777fe640 1 -- 172.21.15.42:0/1383243527 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fc7780039c0 con 0x7fc7801244a0 2024-09-16T04:08:55.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.412+0000 7fc785724640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:55.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.412+0000 7fc785724640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:55.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.412+0000 7fc785724640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:08:55.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.412+0000 7fc785724640 1 -- 172.21.15.42:0/1383243527 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc780135e20 con 0x7fc7801244a0 2024-09-16T04:08:55.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.412+0000 7fc7777fe640 1 --2- 172.21.15.42:0/1383243527 >> v2:172.21.15.154:6812/254845342 conn(0x7fc760037810 0x7fc760039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:08:55.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.412+0000 7fc7777fe640 1 -- 172.21.15.42:0/1383243527 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(706..706 src has 451..706) ==== 5040+0+0 (secure 0 0 0) 0x7fc7780486e0 con 0x7fc7801244a0 2024-09-16T04:08:55.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.412+0000 7fc7777fe640 10 client.1014263.objecter ms_dispatch 0x7fc780000ba0 osd_map(706..706 src has 451..706) 2024-09-16T04:08:55.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.412+0000 7fc7777fe640 3 client.1014263.objecter handle_osd_map got epochs [706,706] > 0 2024-09-16T04:08:55.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.412+0000 7fc7777fe640 3 client.1014263.objecter handle_osd_map decoding full epoch 706 2024-09-16T04:08:55.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.412+0000 7fc7777fe640 20 client.1014263.objecter dump_active .. 0 homeless 2024-09-16T04:08:55.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.412+0000 7fc77effd640 1 --2- 172.21.15.42:0/1383243527 >> v2:172.21.15.154:6812/254845342 conn(0x7fc760037810 0x7fc760039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:08:55.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.412+0000 7fc77effd640 10 monclient: get_auth_request con 0x7fc760037810 auth_method 0 2024-09-16T04:08:55.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.412+0000 7fc7777fe640 10 client.1014263.objecter ms_handle_connect 0x7fc760037810 2024-09-16T04:08:55.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.412+0000 7fc77effd640 1 --2- 172.21.15.42:0/1383243527 >> v2:172.21.15.154:6812/254845342 conn(0x7fc760037810 0x7fc760039cd0 secure :-1 s=READY pgs=3901 cs=0 l=1 rev1=1 crypto rx=0x7fc7680ab000 tx=0x7fc7680b1040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:08:55.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.420+0000 7fc7777fe640 1 -- 172.21.15.42:0/1383243527 <== mon.2 v2:172.21.15.42:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7fc780071210 con 0x7fc7801244a0 2024-09-16T04:08:55.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.420+0000 7fc7777fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:55.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.420+0000 7fc7777fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:08:55.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.656+0000 7fc785724640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:08:55.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.656+0000 7fc785724640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:08:55.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.656+0000 7fc785724640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:08:55.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.656+0000 7fc785724640 1 -- 172.21.15.42:0/1383243527 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fc7801264a0 con 0x7fc7801244a0 2024-09-16T04:08:55.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.656+0000 7fc7777fe640 1 -- 172.21.15.42:0/1383243527 <== mon.2 v2:172.21.15.42:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7fc778003cb0 con 0x7fc7801244a0 2024-09-16T04:08:55.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.656+0000 7fc7777fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:08:55.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.656+0000 7fc7777fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:08:55.664 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":248,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:08:55.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.660+0000 7fc785724640 10 librados: watch_flush enter 2024-09-16T04:08:55.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.660+0000 7fc785724640 10 librados: watch_flush exit 2024-09-16T04:08:55.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.660+0000 7fc785724640 20 client.1014263.objecter shutdown clearing up homeless session... 2024-09-16T04:08:55.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.660+0000 7fc785724640 10 client.1014263.objecter successfully canceled tick 2024-09-16T04:08:55.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.660+0000 7fc785724640 1 -- 172.21.15.42:0/1383243527 >> v2:172.21.15.154:6812/254845342 conn(0x7fc760037810 msgr2=0x7fc760039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:55.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.660+0000 7fc785724640 1 --2- 172.21.15.42:0/1383243527 >> v2:172.21.15.154:6812/254845342 conn(0x7fc760037810 0x7fc760039cd0 secure :-1 s=READY pgs=3901 cs=0 l=1 rev1=1 crypto rx=0x7fc7680ab000 tx=0x7fc7680b1040 comp rx=0 tx=0).stop 2024-09-16T04:08:55.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.660+0000 7fc785724640 10 monclient: shutdown 2024-09-16T04:08:55.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.660+0000 7fc785724640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:55.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.660+0000 7fc785724640 1 -- 172.21.15.42:0/1383243527 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc7801244a0 msgr2=0x7fc780124920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:55.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.660+0000 7fc785724640 1 --2- 172.21.15.42:0/1383243527 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc7801244a0 0x7fc780124920 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fc780068f60 tx=0x7fc77800f900 comp rx=0 tx=0).stop 2024-09-16T04:08:55.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.660+0000 7fc785724640 1 -- 172.21.15.42:0/1383243527 shutdown_connections 2024-09-16T04:08:55.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.660+0000 7fc785724640 1 --2- 172.21.15.42:0/1383243527 >> v2:172.21.15.154:6812/254845342 conn(0x7fc760037810 0x7fc760039cd0 unknown :-1 s=CLOSED pgs=3901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:08:55.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.660+0000 7fc785724640 1 --2- 172.21.15.42:0/1383243527 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc7801244a0 0x7fc780124920 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:08:55.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.660+0000 7fc785724640 1 --2- 172.21.15.42:0/1383243527 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc7801cfc50 0x7fc780123f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:08:55.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.660+0000 7fc785724640 1 --2- 172.21.15.42:0/1383243527 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc76c003a10 0x7fc7801cf710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:08:55.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.660+0000 7fc785724640 1 -- 172.21.15.42:0/1383243527 >> 172.21.15.42:0/1383243527 conn(0x7fc7800a2420 msgr2=0x7fc780071740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:55.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.660+0000 7fc785724640 1 -- 172.21.15.42:0/1383243527 shutdown_connections 2024-09-16T04:08:55.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.660+0000 7fc785724640 1 -- 172.21.15.42:0/1383243527 wait complete. 2024-09-16T04:08:55.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.660+0000 7fc785724640 1 librados: shutdown 2024-09-16T04:08:55.683 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T04:08:55.683 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T04:08:55.684 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T04:08:55.802 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.796+0000 7faa47989640 10 monclient: get_monmap_and_config 2024-09-16T04:08:55.802 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.796+0000 7faa47989640 10 monclient: build_initial_monmap 2024-09-16T04:08:55.802 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.796+0000 7faa47989640 10 monclient: monmap: 2024-09-16T04:08:55.802 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:08:55.803 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:08:55.803 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:08:55.800939+0000 2024-09-16T04:08:55.803 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:08:55.800939+0000 2024-09-16T04:08:55.803 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:55.803 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:08:55.803 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:08:55.803 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:08:55.803 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:08:55.803 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:08:55.803 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:08:55.803 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:08:55.808 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.796+0000 7faa47989640 1 Processor -- start 2024-09-16T04:08:55.808 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.796+0000 7faa47989640 1 -- start start 2024-09-16T04:08:55.808 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.796+0000 7faa47989640 10 monclient: init 2024-09-16T04:08:55.808 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.796+0000 7faa47989640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:55.808 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.796+0000 7faa47989640 10 monclient: _add_conns ranks=[3,2,1,0,4] 2024-09-16T04:08:55.808 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.796+0000 7faa47989640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7faa4006db90 0x7faa401340c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:08:55.808 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.796+0000 7faa47989640 10 monclient(hunting): picked mon.noname-a con 0x7faa4006db90 addr v2:172.21.15.42:3300/0 2024-09-16T04:08:55.808 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.796+0000 7faa47989640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7faa400a37f0 0x7faa400a3c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:08:55.808 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.796+0000 7faa47989640 10 monclient(hunting): picked mon.noname-b con 0x7faa400a37f0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:08:55.808 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.796+0000 7faa47989640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7faa40136e10 addr v1:172.21.15.154:6789/0 2024-09-16T04:08:55.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.796+0000 7faa47989640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa4006ece0 con 0x7faa40136e10 2024-09-16T04:08:55.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.796+0000 7faa47989640 10 monclient(hunting): start opening mon connection 2024-09-16T04:08:55.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.796+0000 7faa47989640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7faa4006df70 con 0x7faa4006db90 2024-09-16T04:08:55.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.796+0000 7faa47989640 10 monclient(hunting): start opening mon connection 2024-09-16T04:08:55.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.796+0000 7faa47989640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7faa4009bf80 con 0x7faa400a37f0 2024-09-16T04:08:55.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.796+0000 7faa47989640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:55.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.796+0000 7faa47989640 10 monclient(hunting): authenticate will time out at 2582.770437s 2024-09-16T04:08:55.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.796+0000 7faa44f24640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7faa400a37f0 0x7faa400a3c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:08:55.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.796+0000 7faa45725640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7faa4006db90 0x7faa401340c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:08:55.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.796+0000 7faa44f24640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7faa400a37f0 0x7faa400a3c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:40244/0 (socket says 172.21.15.42:40244) 2024-09-16T04:08:55.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.796+0000 7faa44f24640 1 -- 172.21.15.42:0/4161298148 learned_addr learned my addr 172.21.15.42:0/4161298148 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:08:55.810 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.796+0000 7faa44f24640 10 monclient(hunting): get_auth_request con 0x7faa400a37f0 auth_method 0 2024-09-16T04:08:55.810 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.796+0000 7faa44f24640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:08:55.810 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.796+0000 7faa44f24640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:55.810 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.796+0000 7faa44f24640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:55.810 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.796+0000 7faa45725640 10 monclient(hunting): get_auth_request con 0x7faa4006db90 auth_method 0 2024-09-16T04:08:55.810 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.796+0000 7faa45725640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:08:55.810 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.796+0000 7faa45725640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:55.810 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.796+0000 7faa45725640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:55.810 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.796+0000 7faa44f24640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:08:55.810 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.796+0000 7faa44f24640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:08:55.810 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.796+0000 7faa44f24640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:08:55.810 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.796+0000 7faa45725640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:08:55.810 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.796+0000 7faa45725640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:08:55.810 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.796+0000 7faa45725640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:08:55.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.796+0000 7faa44f24640 10 monclient(hunting): handle_auth_done global_id 1014266 payload 274 2024-09-16T04:08:55.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.796+0000 7faa44f24640 1 -- 172.21.15.42:0/4161298148 >> v2:172.21.15.42:3300/0 conn(0x7faa4006db90 msgr2=0x7faa401340c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:08:55.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.796+0000 7faa44f24640 1 --2- 172.21.15.42:0/4161298148 >> v2:172.21.15.42:3300/0 conn(0x7faa4006db90 0x7faa401340c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:08:55.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.796+0000 7faa44f24640 1 -- 172.21.15.42:0/4161298148 >> v1:172.21.15.154:6789/0 conn(0x7faa40136e10 legacy=0x7faa40137240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:55.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.796+0000 7faa44f24640 10 monclient: _finish_hunting 0 2024-09-16T04:08:55.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.796+0000 7faa44f24640 1 monclient: found mon.noname-b 2024-09-16T04:08:55.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.796+0000 7faa44f24640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:55.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.796+0000 7faa44f24640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T04:08:55.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.796+0000 7faa44f24640 1 -- 172.21.15.42:0/4161298148 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa40134780 con 0x7faa400a37f0 2024-09-16T04:08:55.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.796+0000 7faa45725640 1 --2- 172.21.15.42:0/4161298148 >> v2:172.21.15.42:3300/0 conn(0x7faa4006db90 0x7faa401340c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:08:55.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.800+0000 7faa44f24640 1 --2- 172.21.15.42:0/4161298148 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7faa400a37f0 0x7faa400a3c00 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7faa30009e50 tx=0x7faa3001c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9ebf7537dc55f891 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:08:55.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.800+0000 7faa3ffff640 1 -- 172.21.15.42:0/4161298148 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa300a55b0 con 0x7faa400a37f0 2024-09-16T04:08:55.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.800+0000 7faa3ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:55.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.800+0000 7faa3ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:08:55.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.800+0000 7faa3ffff640 10 monclient: dump: 2024-09-16T04:08:55.812 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:08:55.812 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:08:55.812 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:08:55.812 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:08:55.812 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:55.812 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:08:55.812 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:08:55.812 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:08:55.812 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:08:55.812 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:08:55.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.800+0000 7faa3ffff640 10 monclient: _finish_auth 0 2024-09-16T04:08:55.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.800+0000 7faa3ffff640 10 monclient: _check_auth_tickets 2024-09-16T04:08:55.813 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.800+0000 7faa3ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:55.813 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.800+0000 7faa3ffff640 1 -- 172.21.15.42:0/4161298148 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa300a5750 con 0x7faa400a37f0 2024-09-16T04:08:55.813 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.800+0000 7faa3ffff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:55.813 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.800+0000 7faa3ffff640 1 -- 172.21.15.42:0/4161298148 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa300a5b20 con 0x7faa400a37f0 2024-09-16T04:08:55.813 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.800+0000 7faa47989640 5 monclient: authenticate success, global_id 1014266 2024-09-16T04:08:55.813 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.800+0000 7faa47989640 10 monclient: get_monmap_and_config success 2024-09-16T04:08:55.813 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.800+0000 7faa47989640 4 set_mon_vals no callback set 2024-09-16T04:08:55.813 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.800+0000 7faa3ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:55.813 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.800+0000 7faa3ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:08:55.813 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.800+0000 7faa3ffff640 10 monclient: dump: 2024-09-16T04:08:55.813 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:08:55.813 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:08:55.813 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:08:55.814 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:08:55.814 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:55.814 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:08:55.814 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:08:55.814 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:08:55.814 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:08:55.814 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:08:55.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.800+0000 7faa47989640 10 monclient: shutdown 2024-09-16T04:08:55.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.800+0000 7faa47989640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:55.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.800+0000 7faa47989640 1 -- 172.21.15.42:0/4161298148 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7faa400a37f0 msgr2=0x7faa400a3c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:55.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.800+0000 7faa47989640 1 --2- 172.21.15.42:0/4161298148 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7faa400a37f0 0x7faa400a3c00 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7faa30009e50 tx=0x7faa3001c8f0 comp rx=0 tx=0).stop 2024-09-16T04:08:55.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.800+0000 7faa47989640 1 -- 172.21.15.42:0/4161298148 shutdown_connections 2024-09-16T04:08:55.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.800+0000 7faa47989640 1 --2- 172.21.15.42:0/4161298148 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7faa400a37f0 0x7faa400a3c00 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:08:55.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.800+0000 7faa47989640 1 --2- 172.21.15.42:0/4161298148 >> v2:172.21.15.42:3300/0 conn(0x7faa4006db90 0x7faa401340c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:08:55.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.800+0000 7faa47989640 1 -- 172.21.15.42:0/4161298148 >> 172.21.15.42:0/4161298148 conn(0x7faa400a9fb0 msgr2=0x7faa400aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:55.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.800+0000 7faa47989640 1 -- 172.21.15.42:0/4161298148 shutdown_connections 2024-09-16T04:08:55.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.800+0000 7faa47989640 1 -- 172.21.15.42:0/4161298148 wait complete. 2024-09-16T04:08:55.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.800+0000 7faa47989640 10 monclient: build_initial_monmap 2024-09-16T04:08:55.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.800+0000 7faa47989640 10 monclient: monmap: 2024-09-16T04:08:55.815 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:08:55.815 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:08:55.815 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:08:55.815 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:08:55.815 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:55.815 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:08:55.815 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:08:55.815 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:08:55.815 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:08:55.816 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:08:55.823 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.800+0000 7faa47989640 1 librados: starting msgr at 2024-09-16T04:08:55.823 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.800+0000 7faa47989640 1 librados: starting objecter 2024-09-16T04:08:55.823 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.800+0000 7faa47989640 1 Processor -- start 2024-09-16T04:08:55.823 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.800+0000 7faa47989640 1 -- start start 2024-09-16T04:08:55.823 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.800+0000 7faa47989640 1 librados: setting wanted keys 2024-09-16T04:08:55.823 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.800+0000 7faa47989640 1 librados: calling monclient init 2024-09-16T04:08:55.823 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.800+0000 7faa47989640 10 monclient: init 2024-09-16T04:08:55.823 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.800+0000 7faa47989640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:55.824 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.800+0000 7faa47989640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:08:55.824 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.800+0000 7faa47989640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7faa4006db90 0x7faa401cf550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:08:55.824 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.800+0000 7faa47989640 10 monclient(hunting): picked mon.noname-c con 0x7faa4006db90 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:08:55.824 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.800+0000 7faa47989640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faa400a37f0 0x7faa401cfa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:08:55.824 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.800+0000 7faa47989640 10 monclient(hunting): picked mon.noname-a con 0x7faa400a37f0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:08:55.824 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.800+0000 7faa47989640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faa40136e10 0x7faa400a1760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:08:55.824 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.800+0000 7faa47989640 10 monclient(hunting): picked mon.noname-b con 0x7faa40136e10 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:08:55.824 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.800+0000 7faa47989640 10 monclient(hunting): start opening mon connection 2024-09-16T04:08:55.824 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.800+0000 7faa47989640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7faa40141e40 con 0x7faa400a37f0 2024-09-16T04:08:55.824 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.800+0000 7faa47989640 10 monclient(hunting): start opening mon connection 2024-09-16T04:08:55.824 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.800+0000 7faa47989640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7faa40141cc0 con 0x7faa40136e10 2024-09-16T04:08:55.824 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.800+0000 7faa47989640 10 monclient(hunting): start opening mon connection 2024-09-16T04:08:55.825 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.800+0000 7faa47989640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7faa40141fc0 con 0x7faa4006db90 2024-09-16T04:08:55.825 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.800+0000 7faa47989640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:55.825 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.800+0000 7faa47989640 10 monclient(hunting): authenticate will time out at 2582.775667s 2024-09-16T04:08:55.825 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.800+0000 7faa45725640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7faa4006db90 0x7faa401cf550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:08:55.825 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.800+0000 7faa45725640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7faa4006db90 0x7faa401cf550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:40260/0 (socket says 172.21.15.42:40260) 2024-09-16T04:08:55.825 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.800+0000 7faa45725640 1 -- 172.21.15.42:0/1245321064 learned_addr learned my addr 172.21.15.42:0/1245321064 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:08:55.825 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.800+0000 7faa45725640 10 monclient(hunting): get_auth_request con 0x7faa4006db90 auth_method 0 2024-09-16T04:08:55.825 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.800+0000 7faa44f24640 1 --2- 172.21.15.42:0/1245321064 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faa400a37f0 0x7faa401cfa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:08:55.825 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.800+0000 7faa45725640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:08:55.825 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.800+0000 7faa45725640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:55.825 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.800+0000 7faa45725640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:55.825 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.800+0000 7faa44f24640 10 monclient(hunting): get_auth_request con 0x7faa400a37f0 auth_method 0 2024-09-16T04:08:55.825 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.800+0000 7faa44f24640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:08:55.826 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.800+0000 7faa44f24640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:55.826 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.800+0000 7faa44f24640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:55.826 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.800+0000 7faa45725640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:08:55.826 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.800+0000 7faa45725640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:08:55.826 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.800+0000 7faa45725640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:08:55.826 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.800+0000 7faa44f24640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:08:55.826 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.800+0000 7faa44f24640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:08:55.826 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.800+0000 7faa44f24640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:08:55.826 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.804+0000 7faa45f26640 1 --2- 172.21.15.42:0/1245321064 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faa40136e10 0x7faa400a1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:08:55.826 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.804+0000 7faa45f26640 10 monclient(hunting): get_auth_request con 0x7faa40136e10 auth_method 0 2024-09-16T04:08:55.826 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.804+0000 7faa45f26640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:08:55.826 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.804+0000 7faa45f26640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:55.826 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.804+0000 7faa45f26640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:55.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.804+0000 7faa45725640 10 monclient(hunting): handle_auth_done global_id 1014269 payload 824 2024-09-16T04:08:55.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.804+0000 7faa45725640 1 -- 172.21.15.42:0/1245321064 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faa40136e10 msgr2=0x7faa400a1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:55.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.804+0000 7faa45725640 1 --2- 172.21.15.42:0/1245321064 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faa40136e10 0x7faa400a1760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:08:55.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.804+0000 7faa45725640 1 -- 172.21.15.42:0/1245321064 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faa400a37f0 msgr2=0x7faa401cfa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:55.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.804+0000 7faa45725640 1 --2- 172.21.15.42:0/1245321064 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faa400a37f0 0x7faa401cfa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:08:55.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.804+0000 7faa45725640 10 monclient: _finish_hunting 0 2024-09-16T04:08:55.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.804+0000 7faa45725640 1 monclient: found mon.noname-c 2024-09-16T04:08:55.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.804+0000 7faa45725640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:55.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.804+0000 7faa45725640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T04:08:55.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.804+0000 7faa45725640 1 -- 172.21.15.42:0/1245321064 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa40133680 con 0x7faa4006db90 2024-09-16T04:08:55.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.804+0000 7faa44f24640 1 --2- 172.21.15.42:0/1245321064 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faa400a37f0 0x7faa401cfa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:08:55.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.804+0000 7faa45725640 1 --2- 172.21.15.42:0/1245321064 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7faa4006db90 0x7faa401cf550 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7faa2800e9b0 tx=0x7faa2800ee70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:08:55.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.804+0000 7faa3dffb640 10 client.?.objecter ms_handle_connect 0x7faa4006db90 2024-09-16T04:08:55.828 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.804+0000 7faa3dffb640 10 client.?.objecter resend_mon_ops 2024-09-16T04:08:55.828 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.804+0000 7faa3dffb640 1 -- 172.21.15.42:0/1245321064 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa28017070 con 0x7faa4006db90 2024-09-16T04:08:55.828 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.804+0000 7faa3dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:55.828 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.804+0000 7faa3dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:08:55.828 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.804+0000 7faa3dffb640 10 monclient: dump: 2024-09-16T04:08:55.828 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:08:55.828 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:08:55.828 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:08:55.828 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:08:55.828 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:55.828 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:08:55.828 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:08:55.828 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:08:55.829 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:08:55.829 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:08:55.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.804+0000 7faa3dffb640 10 monclient: _finish_auth 0 2024-09-16T04:08:55.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.804+0000 7faa3dffb640 10 monclient: _check_auth_tickets 2024-09-16T04:08:55.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.804+0000 7faa3dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:55.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.804+0000 7faa47989640 5 monclient: authenticate success, global_id 1014269 2024-09-16T04:08:55.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.804+0000 7faa3dffb640 1 -- 172.21.15.42:0/1245321064 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa2800ca00 con 0x7faa4006db90 2024-09-16T04:08:55.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.804+0000 7faa3dffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:55.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.804+0000 7faa3dffb640 1 -- 172.21.15.42:0/1245321064 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa2800cdd0 con 0x7faa4006db90 2024-09-16T04:08:55.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.804+0000 7faa3dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:55.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.804+0000 7faa3dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:08:55.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.804+0000 7faa3dffb640 10 monclient: dump: 2024-09-16T04:08:55.829 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:08:55.829 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:08:55.830 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:08:55.830 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:08:55.830 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:55.830 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:08:55.830 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:08:55.830 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:08:55.830 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:08:55.830 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:08:55.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.804+0000 7faa3effd640 4 set_mon_vals no callback set 2024-09-16T04:08:55.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.804+0000 7faa47989640 10 monclient: _renew_subs 2024-09-16T04:08:55.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.804+0000 7faa47989640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:08:55.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.804+0000 7faa47989640 1 -- 172.21.15.42:0/1245321064 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa400a5420 con 0x7faa4006db90 2024-09-16T04:08:55.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.804+0000 7faa47989640 10 client.1014269.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:08:55.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.804+0000 7faa47989640 10 monclient: _renew_subs 2024-09-16T04:08:55.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.804+0000 7faa47989640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:08:55.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.804+0000 7faa47989640 1 -- 172.21.15.42:0/1245321064 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7faa40139ab0 con 0x7faa4006db90 2024-09-16T04:08:55.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.804+0000 7faa47989640 1 librados: init done 2024-09-16T04:08:55.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.804+0000 7faa47989640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:55.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.804+0000 7faa47989640 1 --2- 172.21.15.42:0/1245321064 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faa401cffd0 0x7faa4002c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:08:55.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.804+0000 7faa47989640 10 monclient(hunting): start opening mon connection 2024-09-16T04:08:55.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.804+0000 7faa47989640 1 -- 172.21.15.42:0/1245321064 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7faa4013ea90 con 0x7faa401cffd0 2024-09-16T04:08:55.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.804+0000 7faa44f24640 1 --2- 172.21.15.42:0/1245321064 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faa401cffd0 0x7faa4002c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:08:55.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.804+0000 7faa44f24640 10 monclient: get_auth_request con 0x7faa401cffd0 auth_method 0 2024-09-16T04:08:55.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.804+0000 7faa44f24640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:08:55.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.804+0000 7faa44f24640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:55.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.804+0000 7faa44f24640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:55.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.804+0000 7faa44f24640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:08:55.832 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.804+0000 7faa44f24640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:08:55.832 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.804+0000 7faa3dffb640 1 -- 172.21.15.42:0/1245321064 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7faa2801a070 con 0x7faa4006db90 2024-09-16T04:08:55.832 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.804+0000 7faa44f24640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:08:55.832 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.804+0000 7faa44f24640 10 monclient(hunting): handle_auth_done global_id 1034109 payload 824 2024-09-16T04:08:55.832 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.804+0000 7faa44f24640 1 -- 172.21.15.42:0/1245321064 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7faa40139680 con 0x7faa401cffd0 2024-09-16T04:08:55.832 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.804+0000 7faa3dffb640 1 --2- 172.21.15.42:0/1245321064 >> v2:172.21.15.154:6812/254845342 conn(0x7faa20037810 0x7faa20039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:08:55.832 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.804+0000 7faa3dffb640 1 -- 172.21.15.42:0/1245321064 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(706..706 src has 451..706) ==== 5040+0+0 (secure 0 0 0) 0x7faa28048c90 con 0x7faa4006db90 2024-09-16T04:08:55.832 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.804+0000 7faa3dffb640 10 client.1014269.objecter ms_dispatch 0x7faa40000ba0 osd_map(706..706 src has 451..706) 2024-09-16T04:08:55.832 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.804+0000 7faa3dffb640 3 client.1014269.objecter handle_osd_map got epochs [706,706] > 0 2024-09-16T04:08:55.832 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.804+0000 7faa3dffb640 3 client.1014269.objecter handle_osd_map decoding full epoch 706 2024-09-16T04:08:55.832 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.804+0000 7faa3dffb640 20 client.1014269.objecter dump_active .. 0 homeless 2024-09-16T04:08:55.832 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.804+0000 7faa3dffb640 10 client.1014269.objecter ms_handle_connect 0x7faa401cffd0 2024-09-16T04:08:55.832 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.804+0000 7faa3dffb640 10 client.1014269.objecter resend_mon_ops 2024-09-16T04:08:55.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.804+0000 7faa44f24640 1 --2- 172.21.15.42:0/1245321064 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faa401cffd0 0x7faa4002c650 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7faa40131410 tx=0x7faa30019690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:08:55.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.804+0000 7faa3dffb640 1 -- 172.21.15.42:0/1245321064 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa30009510 con 0x7faa401cffd0 2024-09-16T04:08:55.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.804+0000 7faa3dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:55.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.804+0000 7faa3dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:08:55.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.804+0000 7faa3dffb640 10 monclient: dump: 2024-09-16T04:08:55.833 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:08:55.833 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:08:55.833 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:08:55.833 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:08:55.833 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:55.833 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:08:55.833 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:08:55.833 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:08:55.834 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:08:55.834 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:08:55.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.804+0000 7faa45f26640 1 --2- 172.21.15.42:0/1245321064 >> v2:172.21.15.154:6812/254845342 conn(0x7faa20037810 0x7faa20039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:08:55.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.804+0000 7faa45f26640 10 monclient: get_auth_request con 0x7faa20037810 auth_method 0 2024-09-16T04:08:55.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.804+0000 7faa3dffb640 1 -- 172.21.15.42:0/1245321064 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7faa300a55b0 con 0x7faa401cffd0 2024-09-16T04:08:55.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.804+0000 7faa3dffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:55.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.804+0000 7faa3dffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:08:55.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.804+0000 7faa3dffb640 1 -- 172.21.15.42:0/1245321064 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faa401cffd0 msgr2=0x7faa4002c650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:55.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.804+0000 7faa3dffb640 1 --2- 172.21.15.42:0/1245321064 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faa401cffd0 0x7faa4002c650 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7faa40131410 tx=0x7faa30019690 comp rx=0 tx=0).stop 2024-09-16T04:08:55.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.804+0000 7faa3dffb640 1 -- 172.21.15.42:0/1245321064 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faa401cffd0 msgr2=0x7faa4002c650 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:08:55.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.804+0000 7faa3dffb640 1 --2- 172.21.15.42:0/1245321064 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faa401cffd0 0x7faa4002c650 secure :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7faa40131410 tx=0x7faa30019690 comp rx=0 tx=0).stop 2024-09-16T04:08:55.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.804+0000 7faa3dffb640 10 client.1014269.objecter ms_handle_connect 0x7faa20037810 2024-09-16T04:08:55.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.804+0000 7faa45f26640 1 --2- 172.21.15.42:0/1245321064 >> v2:172.21.15.154:6812/254845342 conn(0x7faa20037810 0x7faa20039cd0 secure :-1 s=READY pgs=3902 cs=0 l=1 rev1=1 crypto rx=0x7faa40138b80 tx=0x7faa38009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:08:55.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.812+0000 7faa47989640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:08:55.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.812+0000 7faa47989640 1 --2- 172.21.15.42:0/1245321064 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faa4002cb90 0x7faa4002cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:08:55.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.812+0000 7faa47989640 10 monclient(hunting): start opening mon connection 2024-09-16T04:08:55.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.812+0000 7faa47989640 1 -- 172.21.15.42:0/1245321064 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7faa4009de20 con 0x7faa4002cb90 2024-09-16T04:08:55.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.812+0000 7faa44f24640 1 --2- 172.21.15.42:0/1245321064 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faa4002cb90 0x7faa4002cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:08:55.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.812+0000 7faa44f24640 10 monclient: get_auth_request con 0x7faa4002cb90 auth_method 0 2024-09-16T04:08:55.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.812+0000 7faa44f24640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:08:55.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.812+0000 7faa44f24640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:55.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.812+0000 7faa44f24640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:55.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.812+0000 7faa44f24640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:08:55.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.812+0000 7faa44f24640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:08:55.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.812+0000 7faa44f24640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:08:55.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.812+0000 7faa44f24640 10 monclient(hunting): handle_auth_done global_id 1034112 payload 824 2024-09-16T04:08:55.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.812+0000 7faa44f24640 1 -- 172.21.15.42:0/1245321064 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7faa40068e40 con 0x7faa4002cb90 2024-09-16T04:08:55.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.812+0000 7faa44f24640 1 --2- 172.21.15.42:0/1245321064 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faa4002cb90 0x7faa4002cfa0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7faa30009e50 tx=0x7faa300a7000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:08:55.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.812+0000 7faa3dffb640 10 client.1014269.objecter ms_handle_connect 0x7faa4002cb90 2024-09-16T04:08:55.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.812+0000 7faa3dffb640 10 client.1014269.objecter resend_mon_ops 2024-09-16T04:08:55.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.812+0000 7faa3dffb640 1 -- 172.21.15.42:0/1245321064 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa30009af0 con 0x7faa4002cb90 2024-09-16T04:08:55.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.812+0000 7faa3dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:55.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.812+0000 7faa3dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:08:55.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.812+0000 7faa3dffb640 10 monclient: dump: 2024-09-16T04:08:55.836 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:08:55.836 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:08:55.836 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:08:55.836 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:08:55.836 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:55.836 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:08:55.837 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:08:55.837 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:08:55.837 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:08:55.837 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:08:55.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.812+0000 7faa3dffb640 1 -- 172.21.15.42:0/1245321064 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4547 (secure 0 0 0) 0x7faa30009af0 con 0x7faa4002cb90 2024-09-16T04:08:55.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.812+0000 7faa3dffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:08:55.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.812+0000 7faa3dffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:08:55.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.812+0000 7faa3dffb640 1 -- 172.21.15.42:0/1245321064 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faa4002cb90 msgr2=0x7faa4002cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:55.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.812+0000 7faa3dffb640 1 --2- 172.21.15.42:0/1245321064 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faa4002cb90 0x7faa4002cfa0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7faa30009e50 tx=0x7faa300a7000 comp rx=0 tx=0).stop 2024-09-16T04:08:55.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.812+0000 7faa3dffb640 1 -- 172.21.15.42:0/1245321064 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faa4002cb90 msgr2=0x7faa4002cfa0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:08:55.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.812+0000 7faa3dffb640 1 --2- 172.21.15.42:0/1245321064 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faa4002cb90 0x7faa4002cfa0 secure :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7faa30009e50 tx=0x7faa300a7000 comp rx=0 tx=0).stop 2024-09-16T04:08:55.838 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T04:08:55.838 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:08:55.838 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:08:55.838 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "leader", 2024-09-16T04:08:55.838 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 248, 2024-09-16T04:08:55.838 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 15747, 2024-09-16T04:08:55.838 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T04:08:55.838 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T04:08:55.838 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T04:08:55.838 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T04:08:55.838 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:08:55.838 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 0, 2024-09-16T04:08:55.838 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:08:55.839 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T04:08:55.839 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T04:08:55.839 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:08:55.839 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:08:55.839 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:08:55.839 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:08:55.839 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:08:55.839 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:08:55.839 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:08:55.839 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:08:55.839 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:08:55.839 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:08:55.839 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:08:55.840 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:08:55.840 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:08:55.840 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T04:08:55.840 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:08:55.840 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:08:55.840 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:08:55.840 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:08:55.840 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:08:55.840 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:08:55.840 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:08:55.840 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:08:55.840 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:08:55.840 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:08:55.840 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:08:55.841 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:08:55.841 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:08:55.841 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T04:08:55.841 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T04:08:55.841 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T04:08:55.841 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T04:08:55.841 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T04:08:55.841 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T04:08:55.841 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:08:55.841 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:08:55.841 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T04:08:55.842 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T04:08:55.842 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T04:08:55.842 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T04:08:55.842 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T04:08:55.842 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T04:08:55.842 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T04:08:55.842 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:08:55.842 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T04:08:55.842 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:08:55.842 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:08:55.842 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:08:55.842 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:08:55.842 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:08:55.842 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:08:55.843 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:08:55.843 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:08:55.843 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:08:55.843 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:08:55.843 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:08:55.843 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:08:55.843 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T04:08:55.843 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:08:55.843 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T04:08:55.843 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:08:55.843 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:08:55.844 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:08:55.844 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:08:55.844 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:08:55.844 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:08:55.844 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:08:55.844 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T04:08:55.844 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:08:55.844 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:08:55.844 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:08:55.844 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:08:55.844 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T04:08:55.844 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:08:55.844 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:08:55.845 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:08:55.845 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:08:55.845 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T04:08:55.845 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T04:08:55.845 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:08:55.845 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:08:55.845 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:08:55.845 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:08:55.845 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:08:55.845 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:08:55.845 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:08:55.845 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:08:55.845 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:08:55.846 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:08:55.846 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:08:55.846 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T04:08:55.846 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:08:55.846 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:08:55.846 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:08:55.846 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:08:55.846 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T04:08:55.846 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:08:55.846 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:08:55.846 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:08:55.846 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:08:55.846 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T04:08:55.846 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T04:08:55.847 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:08:55.847 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:08:55.847 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:08:55.847 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:08:55.847 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:08:55.847 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:08:55.847 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:08:55.847 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:08:55.847 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:08:55.847 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:08:55.847 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:08:55.847 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T04:08:55.847 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:08:55.847 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:08:55.848 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:08:55.848 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:08:55.848 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T04:08:55.848 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:08:55.848 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:08:55.848 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:08:55.848 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:08:55.848 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T04:08:55.848 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T04:08:55.848 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:08:55.848 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:08:55.848 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:08:55.849 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:08:55.849 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:08:55.849 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:08:55.849 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T04:08:55.849 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T04:08:55.849 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:08:55.849 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:08:55.849 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:08:55.849 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:08:55.849 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:08:55.849 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:08:55.849 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T04:08:55.849 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:08:55.850 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:08:55.850 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:08:55.850 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:08:55.850 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:08:55.850 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:08:55.850 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:08:55.850 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T04:08:55.850 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T04:08:55.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.812+0000 7faa47989640 10 librados: watch_flush enter 2024-09-16T04:08:55.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.812+0000 7faa47989640 10 librados: watch_flush exit 2024-09-16T04:08:55.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.812+0000 7faa47989640 20 client.1014269.objecter shutdown clearing up homeless session... 2024-09-16T04:08:55.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.812+0000 7faa47989640 10 client.1014269.objecter successfully canceled tick 2024-09-16T04:08:55.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.812+0000 7faa47989640 1 -- 172.21.15.42:0/1245321064 >> v2:172.21.15.154:6812/254845342 conn(0x7faa20037810 msgr2=0x7faa20039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:55.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.812+0000 7faa47989640 1 --2- 172.21.15.42:0/1245321064 >> v2:172.21.15.154:6812/254845342 conn(0x7faa20037810 0x7faa20039cd0 secure :-1 s=READY pgs=3902 cs=0 l=1 rev1=1 crypto rx=0x7faa40138b80 tx=0x7faa38009040 comp rx=0 tx=0).stop 2024-09-16T04:08:55.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.812+0000 7faa47989640 10 monclient: shutdown 2024-09-16T04:08:55.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.812+0000 7faa47989640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:55.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.812+0000 7faa47989640 1 -- 172.21.15.42:0/1245321064 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7faa4006db90 msgr2=0x7faa401cf550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:55.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.812+0000 7faa45f26640 1 -- 172.21.15.42:0/1245321064 reap_dead start 2024-09-16T04:08:55.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.812+0000 7faa47989640 1 --2- 172.21.15.42:0/1245321064 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7faa4006db90 0x7faa401cf550 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7faa2800e9b0 tx=0x7faa2800ee70 comp rx=0 tx=0).stop 2024-09-16T04:08:55.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.812+0000 7faa47989640 1 -- 172.21.15.42:0/1245321064 shutdown_connections 2024-09-16T04:08:55.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.812+0000 7faa47989640 1 --2- 172.21.15.42:0/1245321064 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7faa4006db90 0x7faa401cf550 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:08:55.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.812+0000 7faa47989640 1 -- 172.21.15.42:0/1245321064 >> 172.21.15.42:0/1245321064 conn(0x7faa400a9fb0 msgr2=0x7faa401327f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:55.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.812+0000 7faa47989640 1 -- 172.21.15.42:0/1245321064 shutdown_connections 2024-09-16T04:08:55.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.812+0000 7faa47989640 1 -- 172.21.15.42:0/1245321064 wait complete. 2024-09-16T04:08:55.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.812+0000 7faa47989640 1 librados: shutdown 2024-09-16T04:08:55.853 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:08:55.853 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T04:08:55.854 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T04:08:55.978 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.972+0000 7fbd85938640 10 monclient: get_monmap_and_config 2024-09-16T04:08:55.978 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.972+0000 7fbd85938640 10 monclient: build_initial_monmap 2024-09-16T04:08:55.978 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.972+0000 7fbd85938640 10 monclient: monmap: 2024-09-16T04:08:55.978 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:08:55.978 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:08:55.978 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:08:55.976813+0000 2024-09-16T04:08:55.978 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:08:55.976813+0000 2024-09-16T04:08:55.978 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:55.979 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:08:55.979 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:08:55.979 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:08:55.979 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:08:55.979 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:08:55.979 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:08:55.979 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:08:55.981 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.972+0000 7fbd85938640 1 Processor -- start 2024-09-16T04:08:55.981 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.972+0000 7fbd85938640 1 -- start start 2024-09-16T04:08:55.981 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.972+0000 7fbd85938640 10 monclient: init 2024-09-16T04:08:55.981 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.972+0000 7fbd85938640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:55.981 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.972+0000 7fbd85938640 10 monclient: _add_conns ranks=[2,1,3,0,4] 2024-09-16T04:08:55.981 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.972+0000 7fbd85938640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbd800a1330 0x7fbd8009f180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:08:55.981 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.972+0000 7fbd85938640 10 monclient(hunting): picked mon.noname-b con 0x7fbd800a1330 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:08:55.981 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.972+0000 7fbd85938640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbd800a1710 addr v1:172.21.15.154:6789/0 2024-09-16T04:08:55.982 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.972+0000 7fbd85938640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fbd801393a0 0x7fbd8013b790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:08:55.982 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.972+0000 7fbd85938640 10 monclient(hunting): picked mon.noname-a con 0x7fbd801393a0 addr v2:172.21.15.42:3300/0 2024-09-16T04:08:55.982 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.972+0000 7fbd85938640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbd8009e4c0 con 0x7fbd800a1710 2024-09-16T04:08:55.982 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.972+0000 7fbd85938640 10 monclient(hunting): start opening mon connection 2024-09-16T04:08:55.982 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.972+0000 7fbd85938640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7fbd8009edf0 con 0x7fbd801393a0 2024-09-16T04:08:55.982 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.972+0000 7fbd85938640 10 monclient(hunting): start opening mon connection 2024-09-16T04:08:55.982 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.972+0000 7fbd85938640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fbd8009cd60 con 0x7fbd800a1330 2024-09-16T04:08:55.982 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.972+0000 7fbd85938640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:55.982 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.972+0000 7fbd85938640 10 monclient(hunting): authenticate will time out at 2582.946258s 2024-09-16T04:08:55.982 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.972+0000 7fbd7effd640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbd800a1330 0x7fbd8009f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:08:55.982 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.972+0000 7fbd7f7fe640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fbd801393a0 0x7fbd8013b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:08:55.982 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.972+0000 7fbd7effd640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbd800a1330 0x7fbd8009f180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:40274/0 (socket says 172.21.15.42:40274) 2024-09-16T04:08:55.982 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.972+0000 7fbd7effd640 1 -- 172.21.15.42:0/1695499004 learned_addr learned my addr 172.21.15.42:0/1695499004 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:08:55.983 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.972+0000 7fbd7effd640 10 monclient(hunting): get_auth_request con 0x7fbd800a1330 auth_method 0 2024-09-16T04:08:55.983 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.972+0000 7fbd7effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:08:55.983 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.972+0000 7fbd7effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:55.983 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.972+0000 7fbd7effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:55.983 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.972+0000 7fbd7f7fe640 10 monclient(hunting): get_auth_request con 0x7fbd801393a0 auth_method 0 2024-09-16T04:08:55.983 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.972+0000 7fbd7f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:08:55.983 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.972+0000 7fbd7f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:55.983 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.972+0000 7fbd7f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:55.983 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.972+0000 7fbd7effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:08:55.983 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.972+0000 7fbd7effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:08:55.983 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.972+0000 7fbd7effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:08:55.983 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.972+0000 7fbd7f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:08:55.983 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.972+0000 7fbd7f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:08:55.984 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.972+0000 7fbd7f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:08:55.984 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.972+0000 7fbd7effd640 10 monclient(hunting): handle_auth_done global_id 1014272 payload 274 2024-09-16T04:08:55.984 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.972+0000 7fbd7effd640 1 -- 172.21.15.42:0/1695499004 >> v2:172.21.15.42:3300/0 conn(0x7fbd801393a0 msgr2=0x7fbd8013b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:08:55.984 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.972+0000 7fbd7effd640 1 --2- 172.21.15.42:0/1695499004 >> v2:172.21.15.42:3300/0 conn(0x7fbd801393a0 0x7fbd8013b790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:08:55.984 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.972+0000 7fbd7effd640 1 -- 172.21.15.42:0/1695499004 >> v1:172.21.15.154:6789/0 conn(0x7fbd800a1710 legacy=0x7fbd800a1b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:08:55.984 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.972+0000 7fbd7effd640 10 monclient: _finish_hunting 0 2024-09-16T04:08:55.984 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.972+0000 7fbd7effd640 1 monclient: found mon.noname-b 2024-09-16T04:08:55.984 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.972+0000 7fbd7effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:55.984 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.972+0000 7fbd7effd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T04:08:55.984 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.972+0000 7fbd7effd640 1 -- 172.21.15.42:0/1695499004 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd8009f6c0 con 0x7fbd800a1330 2024-09-16T04:08:55.984 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.972+0000 7fbd7f7fe640 1 --2- 172.21.15.42:0/1695499004 >> v2:172.21.15.42:3300/0 conn(0x7fbd801393a0 0x7fbd8013b790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:08:55.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.976+0000 7fbd7effd640 1 --2- 172.21.15.42:0/1695499004 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbd800a1330 0x7fbd8009f180 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fbd78009e50 tx=0x7fbd7801c8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d7713db2b951dc39 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:08:55.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.976+0000 7fbd7dffb640 1 -- 172.21.15.42:0/1695499004 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd780a5640 con 0x7fbd800a1330 2024-09-16T04:08:55.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.976+0000 7fbd7dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:55.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.976+0000 7fbd7dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:08:55.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.976+0000 7fbd7dffb640 10 monclient: dump: 2024-09-16T04:08:55.991 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:08:55.992 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:08:55.992 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:08:55.992 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:08:55.992 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:55.992 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:08:55.992 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:08:55.992 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:08:55.992 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:08:55.992 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:08:55.992 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.976+0000 7fbd7dffb640 10 monclient: _finish_auth 0 2024-09-16T04:08:55.992 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.976+0000 7fbd7dffb640 10 monclient: _check_auth_tickets 2024-09-16T04:08:55.992 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.976+0000 7fbd7dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:55.992 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.976+0000 7fbd7dffb640 1 -- 172.21.15.42:0/1695499004 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbd780a57e0 con 0x7fbd800a1330 2024-09-16T04:08:55.992 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.976+0000 7fbd7dffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:55.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.976+0000 7fbd7dffb640 1 -- 172.21.15.42:0/1695499004 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd780a5bb0 con 0x7fbd800a1330 2024-09-16T04:08:55.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.976+0000 7fbd85938640 5 monclient: authenticate success, global_id 1014272 2024-09-16T04:08:55.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.976+0000 7fbd85938640 10 monclient: get_monmap_and_config success 2024-09-16T04:08:55.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.976+0000 7fbd85938640 4 set_mon_vals no callback set 2024-09-16T04:08:55.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.976+0000 7fbd7dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:55.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.976+0000 7fbd7dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:08:55.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.976+0000 7fbd7dffb640 10 monclient: dump: 2024-09-16T04:08:55.993 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:08:55.993 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:08:55.993 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:08:55.993 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:08:55.993 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:55.993 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:08:55.993 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:08:55.994 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:08:55.994 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:08:55.994 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:08:55.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.976+0000 7fbd85938640 10 monclient: shutdown 2024-09-16T04:08:55.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.976+0000 7fbd85938640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:55.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.976+0000 7fbd85938640 1 -- 172.21.15.42:0/1695499004 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbd800a1330 msgr2=0x7fbd8009f180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:55.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.976+0000 7fbd85938640 1 --2- 172.21.15.42:0/1695499004 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbd800a1330 0x7fbd8009f180 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fbd78009e50 tx=0x7fbd7801c8b0 comp rx=0 tx=0).stop 2024-09-16T04:08:55.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.976+0000 7fbd85938640 1 -- 172.21.15.42:0/1695499004 shutdown_connections 2024-09-16T04:08:55.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.976+0000 7fbd85938640 1 --2- 172.21.15.42:0/1695499004 >> v2:172.21.15.42:3300/0 conn(0x7fbd801393a0 0x7fbd8013b790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:08:55.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.976+0000 7fbd85938640 1 --2- 172.21.15.42:0/1695499004 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbd800a1330 0x7fbd8009f180 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:08:55.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.976+0000 7fbd85938640 1 -- 172.21.15.42:0/1695499004 >> 172.21.15.42:0/1695499004 conn(0x7fbd800a2420 msgr2=0x7fbd800a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:55.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.976+0000 7fbd85938640 1 -- 172.21.15.42:0/1695499004 shutdown_connections 2024-09-16T04:08:55.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.976+0000 7fbd85938640 1 -- 172.21.15.42:0/1695499004 wait complete. 2024-09-16T04:08:55.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.976+0000 7fbd85938640 10 monclient: build_initial_monmap 2024-09-16T04:08:55.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.976+0000 7fbd85938640 10 monclient: monmap: 2024-09-16T04:08:55.995 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:08:55.995 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:08:55.995 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:08:55.995 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:08:55.995 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:55.995 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:08:55.995 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:08:55.995 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:08:55.995 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:08:55.995 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:08:55.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.976+0000 7fbd85938640 1 librados: starting msgr at 2024-09-16T04:08:55.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.976+0000 7fbd85938640 1 librados: starting objecter 2024-09-16T04:08:55.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.976+0000 7fbd85938640 1 Processor -- start 2024-09-16T04:08:55.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.976+0000 7fbd85938640 1 -- start start 2024-09-16T04:08:55.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.976+0000 7fbd85938640 1 librados: setting wanted keys 2024-09-16T04:08:55.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.976+0000 7fbd85938640 1 librados: calling monclient init 2024-09-16T04:08:55.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.976+0000 7fbd85938640 10 monclient: init 2024-09-16T04:08:55.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.976+0000 7fbd85938640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:55.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.976+0000 7fbd85938640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:08:55.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.976+0000 7fbd85938640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbd800a1330 0x7fbd801cf210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:08:55.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.976+0000 7fbd85938640 10 monclient(hunting): picked mon.noname-b con 0x7fbd800a1330 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:08:55.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.976+0000 7fbd85938640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbd800a1710 0x7fbd801344a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:08:55.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.976+0000 7fbd85938640 10 monclient(hunting): picked mon.noname-c con 0x7fbd800a1710 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:08:55.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.976+0000 7fbd85938640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbd801393a0 0x7fbd801349e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:08:55.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.976+0000 7fbd85938640 10 monclient(hunting): picked mon.noname-a con 0x7fbd801393a0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:08:55.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.976+0000 7fbd85938640 10 monclient(hunting): start opening mon connection 2024-09-16T04:08:55.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.976+0000 7fbd85938640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fbd8013e880 con 0x7fbd801393a0 2024-09-16T04:08:55.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.976+0000 7fbd85938640 10 monclient(hunting): start opening mon connection 2024-09-16T04:08:55.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.976+0000 7fbd85938640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fbd8013e700 con 0x7fbd800a1330 2024-09-16T04:08:55.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.976+0000 7fbd85938640 10 monclient(hunting): start opening mon connection 2024-09-16T04:08:55.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.976+0000 7fbd85938640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fbd8013ea00 con 0x7fbd800a1710 2024-09-16T04:08:55.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.976+0000 7fbd85938640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:55.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.976+0000 7fbd85938640 10 monclient(hunting): authenticate will time out at 2582.951317s 2024-09-16T04:08:55.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.976+0000 7fbd7e7fc640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbd800a1710 0x7fbd801344a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:08:55.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.976+0000 7fbd7f7fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbd801393a0 0x7fbd801349e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:08:55.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.976+0000 7fbd7e7fc640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbd800a1710 0x7fbd801344a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:40280/0 (socket says 172.21.15.42:40280) 2024-09-16T04:08:55.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.976+0000 7fbd7e7fc640 1 -- 172.21.15.42:0/3624023164 learned_addr learned my addr 172.21.15.42:0/3624023164 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:08:55.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.976+0000 7fbd7e7fc640 10 monclient(hunting): get_auth_request con 0x7fbd800a1710 auth_method 0 2024-09-16T04:08:55.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.976+0000 7fbd7e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:08:55.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.976+0000 7fbd7e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:55.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.976+0000 7fbd7e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:55.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.976+0000 7fbd7f7fe640 10 monclient(hunting): get_auth_request con 0x7fbd801393a0 auth_method 0 2024-09-16T04:08:55.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.976+0000 7fbd7f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:08:55.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.976+0000 7fbd7f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:55.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.976+0000 7fbd7f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:55.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.976+0000 7fbd7e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:08:55.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.976+0000 7fbd7e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:08:55.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.976+0000 7fbd7e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:08:55.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.976+0000 7fbd7f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:08:55.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.976+0000 7fbd7f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:08:55.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.976+0000 7fbd7f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:08:55.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.976+0000 7fbd7effd640 1 --2- 172.21.15.42:0/3624023164 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbd800a1330 0x7fbd801cf210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:08:55.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.976+0000 7fbd7effd640 10 monclient(hunting): get_auth_request con 0x7fbd800a1330 auth_method 0 2024-09-16T04:08:55.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.976+0000 7fbd7effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:08:55.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.976+0000 7fbd7effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:55.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.976+0000 7fbd7effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:55.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.976+0000 7fbd7e7fc640 10 monclient(hunting): handle_auth_done global_id 1014275 payload 824 2024-09-16T04:08:55.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.976+0000 7fbd7e7fc640 1 -- 172.21.15.42:0/3624023164 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbd800a1330 msgr2=0x7fbd801cf210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:55.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.976+0000 7fbd7e7fc640 1 --2- 172.21.15.42:0/3624023164 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbd800a1330 0x7fbd801cf210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:08:55.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.976+0000 7fbd7e7fc640 1 -- 172.21.15.42:0/3624023164 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbd801393a0 msgr2=0x7fbd801349e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:55.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.976+0000 7fbd7e7fc640 1 --2- 172.21.15.42:0/3624023164 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbd801393a0 0x7fbd801349e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:08:55.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.976+0000 7fbd7e7fc640 10 monclient: _finish_hunting 0 2024-09-16T04:08:55.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.976+0000 7fbd7e7fc640 1 monclient: found mon.noname-c 2024-09-16T04:08:55.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.976+0000 7fbd7e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:55.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.976+0000 7fbd7e7fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T04:08:56.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.976+0000 7fbd7e7fc640 1 -- 172.21.15.42:0/3624023164 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd801218a0 con 0x7fbd800a1710 2024-09-16T04:08:56.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.976+0000 7fbd7f7fe640 1 --2- 172.21.15.42:0/3624023164 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbd801393a0 0x7fbd801349e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:08:56.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.980+0000 7fbd7e7fc640 1 --2- 172.21.15.42:0/3624023164 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbd800a1710 0x7fbd801344a0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fbd7400c410 tx=0x7fbd7400c8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:08:56.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.980+0000 7fbd67fff640 10 client.?.objecter ms_handle_connect 0x7fbd800a1710 2024-09-16T04:08:56.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.980+0000 7fbd67fff640 10 client.?.objecter resend_mon_ops 2024-09-16T04:08:56.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.980+0000 7fbd67fff640 1 -- 172.21.15.42:0/3624023164 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd74011790 con 0x7fbd800a1710 2024-09-16T04:08:56.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.980+0000 7fbd67fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:56.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.980+0000 7fbd67fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:08:56.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.980+0000 7fbd67fff640 10 monclient: dump: 2024-09-16T04:08:56.000 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:08:56.000 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:08:56.000 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:08:56.001 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:08:56.001 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:56.001 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:08:56.001 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:08:56.001 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:08:56.001 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:08:56.001 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:08:56.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.980+0000 7fbd67fff640 10 monclient: _finish_auth 0 2024-09-16T04:08:56.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.980+0000 7fbd67fff640 10 monclient: _check_auth_tickets 2024-09-16T04:08:56.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.980+0000 7fbd67fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:56.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.980+0000 7fbd67fff640 1 -- 172.21.15.42:0/3624023164 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbd74017070 con 0x7fbd800a1710 2024-09-16T04:08:56.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.980+0000 7fbd67fff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:56.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.980+0000 7fbd67fff640 1 -- 172.21.15.42:0/3624023164 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd74011bb0 con 0x7fbd800a1710 2024-09-16T04:08:56.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.980+0000 7fbd67fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:56.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.980+0000 7fbd67fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:08:56.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.980+0000 7fbd67fff640 10 monclient: dump: 2024-09-16T04:08:56.002 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:08:56.002 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:08:56.002 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:08:56.002 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:08:56.002 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:56.002 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:08:56.002 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:08:56.002 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:08:56.002 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:08:56.003 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:08:56.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.980+0000 7fbd7d7fa640 4 set_mon_vals no callback set 2024-09-16T04:08:56.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.980+0000 7fbd85938640 5 monclient: authenticate success, global_id 1014275 2024-09-16T04:08:56.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.980+0000 7fbd85938640 10 monclient: _renew_subs 2024-09-16T04:08:56.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.980+0000 7fbd85938640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:08:56.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.980+0000 7fbd85938640 1 -- 172.21.15.42:0/3624023164 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd800720b0 con 0x7fbd800a1710 2024-09-16T04:08:56.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.980+0000 7fbd85938640 10 client.1014275.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:08:56.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.980+0000 7fbd85938640 10 monclient: _renew_subs 2024-09-16T04:08:56.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.980+0000 7fbd85938640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:08:56.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.980+0000 7fbd85938640 1 -- 172.21.15.42:0/3624023164 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd80073a30 con 0x7fbd800a1710 2024-09-16T04:08:56.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.980+0000 7fbd85938640 1 librados: init done 2024-09-16T04:08:56.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.980+0000 7fbd85938640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:56.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.980+0000 7fbd85938640 1 --2- 172.21.15.42:0/3624023164 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbd80121ae0 0x7fbd8011ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:08:56.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.980+0000 7fbd85938640 10 monclient(hunting): start opening mon connection 2024-09-16T04:08:56.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.980+0000 7fbd85938640 1 -- 172.21.15.42:0/3624023164 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fbd8009f180 con 0x7fbd80121ae0 2024-09-16T04:08:56.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.980+0000 7fbd7effd640 1 --2- 172.21.15.42:0/3624023164 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbd80121ae0 0x7fbd8011ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:08:56.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.980+0000 7fbd67fff640 1 -- 172.21.15.42:0/3624023164 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fbd7401c460 con 0x7fbd800a1710 2024-09-16T04:08:56.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.980+0000 7fbd7effd640 10 monclient: get_auth_request con 0x7fbd80121ae0 auth_method 0 2024-09-16T04:08:56.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.980+0000 7fbd7effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:08:56.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.980+0000 7fbd7effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:56.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.980+0000 7fbd7effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:56.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.980+0000 7fbd7effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:08:56.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.980+0000 7fbd7effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:08:56.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.980+0000 7fbd7effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:08:56.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.980+0000 7fbd7effd640 10 monclient(hunting): handle_auth_done global_id 1014278 payload 824 2024-09-16T04:08:56.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.980+0000 7fbd67fff640 1 --2- 172.21.15.42:0/3624023164 >> v2:172.21.15.154:6812/254845342 conn(0x7fbd6c037870 0x7fbd6c039d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:08:56.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.980+0000 7fbd7effd640 1 -- 172.21.15.42:0/3624023164 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbd80120450 con 0x7fbd80121ae0 2024-09-16T04:08:56.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.980+0000 7fbd67fff640 1 -- 172.21.15.42:0/3624023164 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(706..706 src has 451..706) ==== 5040+0+0 (secure 0 0 0) 0x7fbd740487e0 con 0x7fbd800a1710 2024-09-16T04:08:56.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.980+0000 7fbd67fff640 10 client.1014275.objecter ms_dispatch 0x7fbd80000ba0 osd_map(706..706 src has 451..706) 2024-09-16T04:08:56.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.980+0000 7fbd67fff640 3 client.1014275.objecter handle_osd_map got epochs [706,706] > 0 2024-09-16T04:08:56.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.980+0000 7fbd67fff640 3 client.1014275.objecter handle_osd_map decoding full epoch 706 2024-09-16T04:08:56.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.980+0000 7fbd7effd640 1 --2- 172.21.15.42:0/3624023164 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbd80121ae0 0x7fbd8011ded0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fbd8006c630 tx=0x7fbd78019650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:08:56.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.980+0000 7fbd67fff640 20 client.1014275.objecter dump_active .. 0 homeless 2024-09-16T04:08:56.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.980+0000 7fbd67fff640 10 client.1014275.objecter ms_handle_connect 0x7fbd80121ae0 2024-09-16T04:08:56.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.980+0000 7fbd67fff640 10 client.1014275.objecter resend_mon_ops 2024-09-16T04:08:56.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.980+0000 7fbd67fff640 1 -- 172.21.15.42:0/3624023164 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd78009510 con 0x7fbd80121ae0 2024-09-16T04:08:56.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.980+0000 7fbd67fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:56.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.980+0000 7fbd67fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:08:56.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.980+0000 7fbd67fff640 10 monclient: dump: 2024-09-16T04:08:56.006 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:08:56.006 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:08:56.006 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:08:56.006 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:08:56.006 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:56.006 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:08:56.006 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:08:56.006 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:08:56.006 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:08:56.006 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:08:56.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.980+0000 7fbd7f7fe640 1 --2- 172.21.15.42:0/3624023164 >> v2:172.21.15.154:6812/254845342 conn(0x7fbd6c037870 0x7fbd6c039d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:08:56.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.980+0000 7fbd7f7fe640 10 monclient: get_auth_request con 0x7fbd6c037870 auth_method 0 2024-09-16T04:08:56.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.980+0000 7fbd67fff640 1 -- 172.21.15.42:0/3624023164 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fbd780ad3f0 con 0x7fbd80121ae0 2024-09-16T04:08:56.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.980+0000 7fbd67fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:56.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.980+0000 7fbd67fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:08:56.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.980+0000 7fbd67fff640 1 -- 172.21.15.42:0/3624023164 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbd80121ae0 msgr2=0x7fbd8011ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:56.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.980+0000 7fbd67fff640 1 --2- 172.21.15.42:0/3624023164 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbd80121ae0 0x7fbd8011ded0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fbd8006c630 tx=0x7fbd78019650 comp rx=0 tx=0).stop 2024-09-16T04:08:56.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.980+0000 7fbd67fff640 1 -- 172.21.15.42:0/3624023164 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbd80121ae0 msgr2=0x7fbd8011ded0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:08:56.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.980+0000 7fbd67fff640 1 --2- 172.21.15.42:0/3624023164 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbd80121ae0 0x7fbd8011ded0 secure :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fbd8006c630 tx=0x7fbd78019650 comp rx=0 tx=0).stop 2024-09-16T04:08:56.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.980+0000 7fbd67fff640 10 client.1014275.objecter ms_handle_connect 0x7fbd6c037870 2024-09-16T04:08:56.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.980+0000 7fbd7f7fe640 1 --2- 172.21.15.42:0/3624023164 >> v2:172.21.15.154:6812/254845342 conn(0x7fbd6c037870 0x7fbd6c039d30 secure :-1 s=READY pgs=3903 cs=0 l=1 rev1=1 crypto rx=0x7fbd70009580 tx=0x7fbd70009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:08:56.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.988+0000 7fbd85938640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:08:56.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.988+0000 7fbd85938640 1 --2- 172.21.15.42:0/3624023164 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbd8011e410 0x7fbd8011e820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:08:56.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.988+0000 7fbd85938640 10 monclient(hunting): start opening mon connection 2024-09-16T04:08:56.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.988+0000 7fbd85938640 1 -- 172.21.15.42:0/3624023164 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fbd8011ed60 con 0x7fbd8011e410 2024-09-16T04:08:56.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.988+0000 7fbd7effd640 1 --2- 172.21.15.42:0/3624023164 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbd8011e410 0x7fbd8011e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:08:56.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.988+0000 7fbd7effd640 10 monclient: get_auth_request con 0x7fbd8011e410 auth_method 0 2024-09-16T04:08:56.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.988+0000 7fbd7effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:08:56.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.988+0000 7fbd7effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:56.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.988+0000 7fbd7effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:56.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.988+0000 7fbd7effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:08:56.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.988+0000 7fbd7effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:08:56.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.988+0000 7fbd7effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:08:56.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.988+0000 7fbd7effd640 10 monclient(hunting): handle_auth_done global_id 1014281 payload 824 2024-09-16T04:08:56.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.988+0000 7fbd7effd640 1 -- 172.21.15.42:0/3624023164 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fbd80121ec0 con 0x7fbd8011e410 2024-09-16T04:08:56.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.988+0000 7fbd67fff640 10 client.1014275.objecter ms_handle_connect 0x7fbd8011e410 2024-09-16T04:08:56.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.988+0000 7fbd67fff640 10 client.1014275.objecter resend_mon_ops 2024-09-16T04:08:56.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.988+0000 7fbd7effd640 1 --2- 172.21.15.42:0/3624023164 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbd8011e410 0x7fbd8011e820 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fbd7801af60 tx=0x7fbd780a7000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:08:56.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.988+0000 7fbd67fff640 1 -- 172.21.15.42:0/3624023164 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd780c1b60 con 0x7fbd8011e410 2024-09-16T04:08:56.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.988+0000 7fbd67fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:56.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.988+0000 7fbd67fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:08:56.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.988+0000 7fbd67fff640 10 monclient: dump: 2024-09-16T04:08:56.010 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:08:56.010 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:08:56.011 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:08:56.011 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:08:56.011 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:56.011 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:08:56.011 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:08:56.011 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:08:56.011 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:08:56.011 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:08:56.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.988+0000 7fbd67fff640 1 -- 172.21.15.42:0/3624023164 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4713 (secure 0 0 0) 0x7fbd780c5070 con 0x7fbd8011e410 2024-09-16T04:08:56.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.988+0000 7fbd67fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:08:56.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.988+0000 7fbd67fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:08:56.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.988+0000 7fbd67fff640 1 -- 172.21.15.42:0/3624023164 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbd8011e410 msgr2=0x7fbd8011e820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:56.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.988+0000 7fbd67fff640 1 --2- 172.21.15.42:0/3624023164 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbd8011e410 0x7fbd8011e820 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fbd7801af60 tx=0x7fbd780a7000 comp rx=0 tx=0).stop 2024-09-16T04:08:56.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.988+0000 7fbd67fff640 1 -- 172.21.15.42:0/3624023164 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbd8011e410 msgr2=0x7fbd8011e820 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:08:56.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.988+0000 7fbd67fff640 1 --2- 172.21.15.42:0/3624023164 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbd8011e410 0x7fbd8011e820 secure :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fbd7801af60 tx=0x7fbd780a7000 comp rx=0 tx=0).stop 2024-09-16T04:08:56.012 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T04:08:56.012 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:08:56.012 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:08:56.012 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T04:08:56.012 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 248, 2024-09-16T04:08:56.012 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 131190, 2024-09-16T04:08:56.013 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T04:08:56.013 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T04:08:56.013 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T04:08:56.013 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T04:08:56.013 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:08:56.013 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 0, 2024-09-16T04:08:56.013 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:08:56.013 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T04:08:56.013 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T04:08:56.013 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:08:56.013 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:08:56.013 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:08:56.013 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:08:56.014 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:08:56.014 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:08:56.014 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:08:56.014 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:08:56.014 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:08:56.014 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:08:56.014 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:08:56.014 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:08:56.014 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:08:56.014 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T04:08:56.014 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:08:56.014 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:08:56.014 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:08:56.014 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:08:56.015 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:08:56.015 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:08:56.015 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:08:56.015 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:08:56.015 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:08:56.015 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:08:56.015 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:08:56.015 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:08:56.015 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:08:56.015 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T04:08:56.015 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T04:08:56.015 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T04:08:56.015 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T04:08:56.016 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T04:08:56.016 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T04:08:56.016 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:08:56.016 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:08:56.016 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T04:08:56.016 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T04:08:56.016 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T04:08:56.016 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T04:08:56.016 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T04:08:56.016 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T04:08:56.016 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T04:08:56.016 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:08:56.016 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T04:08:56.017 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:08:56.017 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:08:56.017 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:08:56.017 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:08:56.017 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:08:56.017 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:08:56.017 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:08:56.017 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:08:56.017 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:08:56.017 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:08:56.017 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:08:56.017 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:08:56.017 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T04:08:56.018 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:08:56.018 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T04:08:56.018 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:08:56.018 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:08:56.018 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:08:56.018 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:08:56.018 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:08:56.018 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:08:56.018 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:08:56.018 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T04:08:56.018 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:08:56.018 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:08:56.018 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:08:56.019 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:08:56.019 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T04:08:56.019 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:08:56.019 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:08:56.019 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:08:56.019 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:08:56.019 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T04:08:56.019 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T04:08:56.019 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:08:56.019 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:08:56.019 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:08:56.019 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:08:56.019 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:08:56.019 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:08:56.020 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:08:56.020 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:08:56.020 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:08:56.020 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:08:56.020 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:08:56.020 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T04:08:56.020 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:08:56.020 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:08:56.020 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:08:56.020 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:08:56.020 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T04:08:56.020 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:08:56.020 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:08:56.021 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:08:56.021 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:08:56.021 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T04:08:56.021 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T04:08:56.021 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:08:56.021 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:08:56.021 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:08:56.021 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:08:56.021 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:08:56.021 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:08:56.021 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:08:56.021 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:08:56.021 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:08:56.021 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:08:56.022 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:08:56.022 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T04:08:56.022 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:08:56.022 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:08:56.022 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:08:56.022 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:08:56.022 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T04:08:56.022 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:08:56.022 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:08:56.022 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:08:56.022 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:08:56.022 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T04:08:56.022 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T04:08:56.022 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:08:56.023 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:08:56.023 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:08:56.023 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:08:56.023 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:08:56.023 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:08:56.023 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T04:08:56.023 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T04:08:56.023 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:08:56.023 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:08:56.023 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:08:56.023 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:08:56.023 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:08:56.023 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:08:56.023 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T04:08:56.024 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:08:56.024 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:08:56.024 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:08:56.024 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 3 2024-09-16T04:08:56.024 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:08:56.024 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:08:56.024 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T04:08:56.024 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:08:56.024 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:08:56.024 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:08:56.024 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 3 2024-09-16T04:08:56.024 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:08:56.024 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:08:56.024 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:08:56.025 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T04:08:56.025 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T04:08:56.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.988+0000 7fbd85938640 10 librados: watch_flush enter 2024-09-16T04:08:56.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.988+0000 7fbd85938640 10 librados: watch_flush exit 2024-09-16T04:08:56.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.988+0000 7fbd85938640 20 client.1014275.objecter shutdown clearing up homeless session... 2024-09-16T04:08:56.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.988+0000 7fbd85938640 10 client.1014275.objecter successfully canceled tick 2024-09-16T04:08:56.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.988+0000 7fbd85938640 1 -- 172.21.15.42:0/3624023164 >> v2:172.21.15.154:6812/254845342 conn(0x7fbd6c037870 msgr2=0x7fbd6c039d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:56.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.988+0000 7fbd85938640 1 --2- 172.21.15.42:0/3624023164 >> v2:172.21.15.154:6812/254845342 conn(0x7fbd6c037870 0x7fbd6c039d30 secure :-1 s=READY pgs=3903 cs=0 l=1 rev1=1 crypto rx=0x7fbd70009580 tx=0x7fbd70009190 comp rx=0 tx=0).stop 2024-09-16T04:08:56.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.988+0000 7fbd85938640 10 monclient: shutdown 2024-09-16T04:08:56.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.988+0000 7fbd85938640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:56.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.988+0000 7fbd85938640 1 -- 172.21.15.42:0/3624023164 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbd800a1710 msgr2=0x7fbd801344a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:56.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.988+0000 7fbd85938640 1 --2- 172.21.15.42:0/3624023164 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbd800a1710 0x7fbd801344a0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fbd7400c410 tx=0x7fbd7400c8d0 comp rx=0 tx=0).stop 2024-09-16T04:08:56.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.988+0000 7fbd7f7fe640 1 -- 172.21.15.42:0/3624023164 reap_dead start 2024-09-16T04:08:56.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.988+0000 7fbd7f7fe640 1 -- 172.21.15.42:0/3624023164 reap_dead start 2024-09-16T04:08:56.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.988+0000 7fbd85938640 1 -- 172.21.15.42:0/3624023164 shutdown_connections 2024-09-16T04:08:56.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.988+0000 7fbd85938640 1 -- 172.21.15.42:0/3624023164 >> 172.21.15.42:0/3624023164 conn(0x7fbd800a2420 msgr2=0x7fbd8009ba50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:56.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.988+0000 7fbd85938640 1 -- 172.21.15.42:0/3624023164 shutdown_connections 2024-09-16T04:08:56.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.988+0000 7fbd85938640 1 -- 172.21.15.42:0/3624023164 wait complete. 2024-09-16T04:08:56.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:55.988+0000 7fbd85938640 1 librados: shutdown 2024-09-16T04:08:56.028 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:08:56.028 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T04:08:56.028 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T04:08:56.145 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.140+0000 7f850049e640 10 monclient: get_monmap_and_config 2024-09-16T04:08:56.145 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.140+0000 7f850049e640 10 monclient: build_initial_monmap 2024-09-16T04:08:56.145 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.140+0000 7f850049e640 10 monclient: monmap: 2024-09-16T04:08:56.145 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:08:56.145 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:08:56.146 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:08:56.144200+0000 2024-09-16T04:08:56.146 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:08:56.144200+0000 2024-09-16T04:08:56.146 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:56.146 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:08:56.146 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:08:56.146 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:08:56.146 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:08:56.146 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:08:56.146 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:08:56.146 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:08:56.148 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.140+0000 7f850049e640 1 Processor -- start 2024-09-16T04:08:56.148 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.140+0000 7f850049e640 1 -- start start 2024-09-16T04:08:56.148 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.140+0000 7f850049e640 10 monclient: init 2024-09-16T04:08:56.148 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.140+0000 7f850049e640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:56.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.140+0000 7f850049e640 10 monclient: _add_conns ranks=[1,0,2,4,3] 2024-09-16T04:08:56.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.140+0000 7f850049e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f84f80a39b0 addr v1:172.21.15.154:6789/0 2024-09-16T04:08:56.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.140+0000 7f850049e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f84f80a3d90 addr v1:172.21.15.42:6789/0 2024-09-16T04:08:56.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.140+0000 7f850049e640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f84f813daa0 0x7f84f813fe90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:08:56.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.140+0000 7f850049e640 10 monclient(hunting): picked mon.noname-b con 0x7f84f813daa0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:08:56.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.140+0000 7f850049e640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f84f8134290 con 0x7f84f80a3d90 2024-09-16T04:08:56.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.140+0000 7f850049e640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f84f809c140 con 0x7f84f80a39b0 2024-09-16T04:08:56.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.140+0000 7f850049e640 10 monclient(hunting): start opening mon connection 2024-09-16T04:08:56.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.140+0000 7f850049e640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f84f809c420 con 0x7f84f813daa0 2024-09-16T04:08:56.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.140+0000 7f850049e640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:56.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.140+0000 7f850049e640 10 monclient(hunting): authenticate will time out at 2583.113810s 2024-09-16T04:08:56.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.140+0000 7f84fea3b640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f84f813daa0 0x7f84f813fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:08:56.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.140+0000 7f84fda39640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f84f80a3d90 0x7f84f813c380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:34792/0 (socket says 172.21.15.42:34792) 2024-09-16T04:08:56.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.140+0000 7f84fda39640 1 -- 172.21.15.42:0/98679958 learned_addr learned my addr 172.21.15.42:0/98679958 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T04:08:56.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.140+0000 7f84fea3b640 10 monclient(hunting): get_auth_request con 0x7f84f813daa0 auth_method 0 2024-09-16T04:08:56.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.140+0000 7f84fea3b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:08:56.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.140+0000 7f84fea3b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:56.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.140+0000 7f84fea3b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:56.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.140+0000 7f84fea3b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:08:56.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.140+0000 7f84fea3b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:08:56.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.140+0000 7f84fea3b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:08:56.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.140+0000 7f84fea3b640 10 monclient(hunting): handle_auth_done global_id 1014284 payload 274 2024-09-16T04:08:56.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.140+0000 7f84fd238640 1 -- 172.21.15.42:0/98679958 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 449745625 0 0) 0x7f84f8134290 con 0x7f84f80a3d90 2024-09-16T04:08:56.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.140+0000 7f84fea3b640 1 -- 172.21.15.42:0/98679958 >> v1:172.21.15.154:6789/0 conn(0x7f84f80a39b0 legacy=0x7f84f8133b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:08:56.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.140+0000 7f84fea3b640 1 -- 172.21.15.42:0/98679958 >> v1:172.21.15.42:6789/0 conn(0x7f84f80a3d90 legacy=0x7f84f813c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:56.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.140+0000 7f84fea3b640 10 monclient: _finish_hunting 0 2024-09-16T04:08:56.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.140+0000 7f84fea3b640 1 monclient: found mon.noname-b 2024-09-16T04:08:56.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.140+0000 7f84fea3b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:56.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.140+0000 7f84fea3b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T04:08:56.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.140+0000 7f84fea3b640 1 -- 172.21.15.42:0/98679958 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84f806dd60 con 0x7f84f813daa0 2024-09-16T04:08:56.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.140+0000 7f84fd238640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T04:08:56.157 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.144+0000 7f84fea3b640 1 --2- 172.21.15.42:0/98679958 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f84f813daa0 0x7f84f813fe90 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f84f0009dc0 tx=0x7f84f001c850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fb4025f473faea08 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:08:56.157 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.144+0000 7f84fd238640 1 -- 172.21.15.42:0/98679958 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84f00b2070 con 0x7f84f813daa0 2024-09-16T04:08:56.157 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.144+0000 7f84fd238640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:56.157 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.144+0000 7f84fd238640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:08:56.157 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.144+0000 7f84fd238640 10 monclient: dump: 2024-09-16T04:08:56.157 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:08:56.157 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:08:56.157 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:08:56.157 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:08:56.157 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:56.158 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:08:56.158 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:08:56.158 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:08:56.158 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:08:56.158 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:08:56.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.144+0000 7f84fd238640 10 monclient: _finish_auth 0 2024-09-16T04:08:56.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.144+0000 7f84fd238640 10 monclient: _check_auth_tickets 2024-09-16T04:08:56.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.144+0000 7f84fd238640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:56.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.144+0000 7f84fd238640 1 -- 172.21.15.42:0/98679958 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f84f00a5710 con 0x7f84f813daa0 2024-09-16T04:08:56.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.144+0000 7f850049e640 5 monclient: authenticate success, global_id 1014284 2024-09-16T04:08:56.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.144+0000 7f850049e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:08:56.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.144+0000 7f84fd238640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:56.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.144+0000 7f850049e640 10 monclient: get_monmap_and_config success 2024-09-16T04:08:56.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.144+0000 7f84fd238640 1 -- 172.21.15.42:0/98679958 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84f00a5ae0 con 0x7f84f813daa0 2024-09-16T04:08:56.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.144+0000 7f850049e640 4 set_mon_vals no callback set 2024-09-16T04:08:56.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.144+0000 7f84fd238640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:56.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.144+0000 7f84fd238640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:08:56.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.144+0000 7f84fd238640 10 monclient: dump: 2024-09-16T04:08:56.159 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:08:56.159 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:08:56.159 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:08:56.159 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:08:56.160 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:56.160 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:08:56.160 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:08:56.160 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:08:56.160 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:08:56.160 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:08:56.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.144+0000 7f850049e640 10 monclient: shutdown 2024-09-16T04:08:56.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.144+0000 7f850049e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:56.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.144+0000 7f850049e640 1 -- 172.21.15.42:0/98679958 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f84f813daa0 msgr2=0x7f84f813fe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:56.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.144+0000 7f850049e640 1 --2- 172.21.15.42:0/98679958 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f84f813daa0 0x7f84f813fe90 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f84f0009dc0 tx=0x7f84f001c850 comp rx=0 tx=0).stop 2024-09-16T04:08:56.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.144+0000 7f850049e640 1 -- 172.21.15.42:0/98679958 shutdown_connections 2024-09-16T04:08:56.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.144+0000 7f850049e640 1 --2- 172.21.15.42:0/98679958 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f84f813daa0 0x7f84f813fe90 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:08:56.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.144+0000 7f850049e640 1 -- 172.21.15.42:0/98679958 >> 172.21.15.42:0/98679958 conn(0x7f84f80a9fb0 msgr2=0x7f84f80aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:56.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.144+0000 7f850049e640 1 -- 172.21.15.42:0/98679958 shutdown_connections 2024-09-16T04:08:56.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.144+0000 7f850049e640 1 -- 172.21.15.42:0/98679958 wait complete. 2024-09-16T04:08:56.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.144+0000 7f850049e640 10 monclient: build_initial_monmap 2024-09-16T04:08:56.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.144+0000 7f850049e640 10 monclient: monmap: 2024-09-16T04:08:56.161 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:08:56.161 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:08:56.161 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:08:56.161 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:08:56.161 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:56.161 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:08:56.161 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:08:56.161 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:08:56.161 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:08:56.161 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:08:56.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.144+0000 7f850049e640 1 librados: starting msgr at 2024-09-16T04:08:56.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.144+0000 7f850049e640 1 librados: starting objecter 2024-09-16T04:08:56.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.144+0000 7f850049e640 1 Processor -- start 2024-09-16T04:08:56.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.144+0000 7f850049e640 1 -- start start 2024-09-16T04:08:56.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.144+0000 7f850049e640 1 librados: setting wanted keys 2024-09-16T04:08:56.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.144+0000 7f850049e640 1 librados: calling monclient init 2024-09-16T04:08:56.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.144+0000 7f850049e640 10 monclient: init 2024-09-16T04:08:56.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.144+0000 7f850049e640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:56.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.144+0000 7f850049e640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:08:56.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.144+0000 7f850049e640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f84f80a39b0 0x7f84f8130300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:08:56.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.144+0000 7f850049e640 10 monclient(hunting): picked mon.noname-a con 0x7f84f80a39b0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:08:56.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.144+0000 7f850049e640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f84f80a3d90 0x7f84f8130840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:08:56.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.144+0000 7f850049e640 10 monclient(hunting): picked mon.noname-c con 0x7f84f80a3d90 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:08:56.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.144+0000 7f850049e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f84f813daa0 0x7f84f8130d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:08:56.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.144+0000 7f850049e640 10 monclient(hunting): picked mon.noname-b con 0x7f84f813daa0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:08:56.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.144+0000 7f850049e640 10 monclient(hunting): start opening mon connection 2024-09-16T04:08:56.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.144+0000 7f850049e640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f84f8142f80 con 0x7f84f80a39b0 2024-09-16T04:08:56.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.144+0000 7f850049e640 10 monclient(hunting): start opening mon connection 2024-09-16T04:08:56.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.144+0000 7f850049e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f84f8142e00 con 0x7f84f813daa0 2024-09-16T04:08:56.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.144+0000 7f850049e640 10 monclient(hunting): start opening mon connection 2024-09-16T04:08:56.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.144+0000 7f850049e640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f84f8143100 con 0x7f84f80a3d90 2024-09-16T04:08:56.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.144+0000 7f850049e640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:56.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.144+0000 7f850049e640 10 monclient(hunting): authenticate will time out at 2583.119227s 2024-09-16T04:08:56.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.144+0000 7f84fe23a640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f84f80a39b0 0x7f84f8130300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:08:56.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.144+0000 7f84fda39640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f84f80a3d90 0x7f84f8130840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:08:56.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.144+0000 7f84fe23a640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f84f80a39b0 0x7f84f8130300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:50020/0 (socket says 172.21.15.42:50020) 2024-09-16T04:08:56.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.144+0000 7f84fda39640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f84f80a3d90 0x7f84f8130840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:40322/0 (socket says 172.21.15.42:40322) 2024-09-16T04:08:56.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.144+0000 7f84fe23a640 1 -- 172.21.15.42:0/2034894742 learned_addr learned my addr 172.21.15.42:0/2034894742 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:08:56.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.144+0000 7f84fe23a640 10 monclient(hunting): get_auth_request con 0x7f84f80a39b0 auth_method 0 2024-09-16T04:08:56.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.144+0000 7f84fe23a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:08:56.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.144+0000 7f84fe23a640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:56.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.144+0000 7f84fe23a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:56.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.144+0000 7f84fda39640 10 monclient(hunting): get_auth_request con 0x7f84f80a3d90 auth_method 0 2024-09-16T04:08:56.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.144+0000 7f84fda39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:08:56.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.144+0000 7f84fda39640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:56.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.144+0000 7f84fda39640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:56.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.144+0000 7f84fe23a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:08:56.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.144+0000 7f84fe23a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:08:56.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.144+0000 7f84fe23a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:08:56.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.144+0000 7f84fda39640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:08:56.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.144+0000 7f84fda39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:08:56.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.144+0000 7f84fda39640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:08:56.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.144+0000 7f84fe23a640 10 monclient(hunting): handle_auth_done global_id 1034124 payload 824 2024-09-16T04:08:56.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.144+0000 7f84fe23a640 1 -- 172.21.15.42:0/2034894742 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f84f80a3d90 msgr2=0x7f84f8130840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:56.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.144+0000 7f84fe23a640 1 --2- 172.21.15.42:0/2034894742 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f84f80a3d90 0x7f84f8130840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:08:56.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.144+0000 7f84fe23a640 1 -- 172.21.15.42:0/2034894742 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f84f813daa0 msgr2=0x7f84f8130d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:56.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.144+0000 7f84fe23a640 1 --2- 172.21.15.42:0/2034894742 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f84f813daa0 0x7f84f8130d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:08:56.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.144+0000 7f84fe23a640 10 monclient: _finish_hunting 0 2024-09-16T04:08:56.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.144+0000 7f84fe23a640 1 monclient: found mon.noname-a 2024-09-16T04:08:56.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.144+0000 7f84fe23a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:56.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.144+0000 7f84fe23a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:08:56.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.144+0000 7f84fe23a640 1 -- 172.21.15.42:0/2034894742 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84f809d1b0 con 0x7f84f80a39b0 2024-09-16T04:08:56.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.144+0000 7f84fe23a640 1 --2- 172.21.15.42:0/2034894742 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f84f80a39b0 0x7f84f8130300 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f84e800c500 tx=0x7f84e800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:08:56.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.144+0000 7f84eeffd640 10 client.?.objecter ms_handle_connect 0x7f84f80a39b0 2024-09-16T04:08:56.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.144+0000 7f84eeffd640 10 client.?.objecter resend_mon_ops 2024-09-16T04:08:56.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.148+0000 7f84eeffd640 1 -- 172.21.15.42:0/2034894742 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84e8016020 con 0x7f84f80a39b0 2024-09-16T04:08:56.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.148+0000 7f84eeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:56.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.148+0000 7f84eeffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:08:56.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.148+0000 7f84eeffd640 10 monclient: dump: 2024-09-16T04:08:56.166 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:08:56.166 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:08:56.166 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:08:56.166 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:08:56.166 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:56.166 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:08:56.166 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:08:56.166 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:08:56.166 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:08:56.167 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:08:56.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.148+0000 7f84eeffd640 10 monclient: _finish_auth 0 2024-09-16T04:08:56.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.148+0000 7f84eeffd640 10 monclient: _check_auth_tickets 2024-09-16T04:08:56.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.148+0000 7f84eeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:56.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.148+0000 7f84eeffd640 1 -- 172.21.15.42:0/2034894742 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f84e8005220 con 0x7f84f80a39b0 2024-09-16T04:08:56.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.148+0000 7f84eeffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:56.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.148+0000 7f84eeffd640 1 -- 172.21.15.42:0/2034894742 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84e8007030 con 0x7f84f80a39b0 2024-09-16T04:08:56.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.148+0000 7f850049e640 5 monclient: authenticate success, global_id 1034124 2024-09-16T04:08:56.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.148+0000 7f850049e640 10 monclient: _renew_subs 2024-09-16T04:08:56.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.148+0000 7f850049e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:08:56.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.148+0000 7f850049e640 1 -- 172.21.15.42:0/2034894742 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84f813c8c0 con 0x7f84f80a39b0 2024-09-16T04:08:56.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.148+0000 7f84effff640 4 set_mon_vals no callback set 2024-09-16T04:08:56.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.148+0000 7f84eeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:56.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.148+0000 7f84eeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:08:56.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.148+0000 7f84eeffd640 10 monclient: dump: 2024-09-16T04:08:56.168 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:08:56.168 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:08:56.168 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:08:56.168 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:08:56.168 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:56.168 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:08:56.168 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:08:56.168 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:08:56.168 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:08:56.168 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:08:56.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.148+0000 7f850049e640 10 client.1034124.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:08:56.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.148+0000 7f850049e640 10 monclient: _renew_subs 2024-09-16T04:08:56.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.148+0000 7f850049e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:08:56.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.148+0000 7f850049e640 1 -- 172.21.15.42:0/2034894742 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84f809ced0 con 0x7f84f80a39b0 2024-09-16T04:08:56.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.148+0000 7f850049e640 1 librados: init done 2024-09-16T04:08:56.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.148+0000 7f84eeffd640 1 -- 172.21.15.42:0/2034894742 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f84e80071f0 con 0x7f84f80a39b0 2024-09-16T04:08:56.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.148+0000 7f850049e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:56.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.148+0000 7f850049e640 1 --2- 172.21.15.42:0/2034894742 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f84f8134bb0 0x7f84f8134f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:08:56.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.148+0000 7f850049e640 10 monclient(hunting): start opening mon connection 2024-09-16T04:08:56.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.148+0000 7f850049e640 1 -- 172.21.15.42:0/2034894742 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f84f8133b80 con 0x7f84f8134bb0 2024-09-16T04:08:56.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.148+0000 7f84eeffd640 1 --2- 172.21.15.42:0/2034894742 >> v2:172.21.15.154:6812/254845342 conn(0x7f84dc037810 0x7f84dc039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:08:56.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.148+0000 7f84eeffd640 1 -- 172.21.15.42:0/2034894742 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(706..706 src has 451..706) ==== 5040+0+0 (secure 0 0 0) 0x7f84e80480d0 con 0x7f84f80a39b0 2024-09-16T04:08:56.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.148+0000 7f84eeffd640 10 client.1034124.objecter ms_dispatch 0x7f84f8000ba0 osd_map(706..706 src has 451..706) 2024-09-16T04:08:56.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.148+0000 7f84eeffd640 3 client.1034124.objecter handle_osd_map got epochs [706,706] > 0 2024-09-16T04:08:56.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.148+0000 7f84eeffd640 3 client.1034124.objecter handle_osd_map decoding full epoch 706 2024-09-16T04:08:56.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.148+0000 7f84eeffd640 20 client.1034124.objecter dump_active .. 0 homeless 2024-09-16T04:08:56.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.148+0000 7f84fda39640 1 --2- 172.21.15.42:0/2034894742 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f84f8134bb0 0x7f84f8134f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:08:56.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.148+0000 7f84fea3b640 1 --2- 172.21.15.42:0/2034894742 >> v2:172.21.15.154:6812/254845342 conn(0x7f84dc037810 0x7f84dc039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:08:56.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.148+0000 7f84fda39640 10 monclient: get_auth_request con 0x7f84f8134bb0 auth_method 0 2024-09-16T04:08:56.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.148+0000 7f84fda39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:08:56.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.148+0000 7f84fda39640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:56.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.148+0000 7f84fda39640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:56.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.148+0000 7f84fea3b640 10 monclient: get_auth_request con 0x7f84dc037810 auth_method 0 2024-09-16T04:08:56.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.148+0000 7f84fda39640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:08:56.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.148+0000 7f84fda39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:08:56.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.148+0000 7f84fda39640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:08:56.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.148+0000 7f84fda39640 10 monclient(hunting): handle_auth_done global_id 905179 payload 824 2024-09-16T04:08:56.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.148+0000 7f84fda39640 1 -- 172.21.15.42:0/2034894742 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f84f80a5750 con 0x7f84f8134bb0 2024-09-16T04:08:56.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.148+0000 7f84eeffd640 10 client.1034124.objecter ms_handle_connect 0x7f84dc037810 2024-09-16T04:08:56.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.148+0000 7f84fea3b640 1 --2- 172.21.15.42:0/2034894742 >> v2:172.21.15.154:6812/254845342 conn(0x7f84dc037810 0x7f84dc039cd0 secure :-1 s=READY pgs=3904 cs=0 l=1 rev1=1 crypto rx=0x7f84f00af000 tx=0x7f84f0004ac0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:08:56.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.148+0000 7f84eeffd640 10 client.1034124.objecter ms_handle_connect 0x7f84f8134bb0 2024-09-16T04:08:56.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.148+0000 7f84eeffd640 10 client.1034124.objecter resend_mon_ops 2024-09-16T04:08:56.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.148+0000 7f84fda39640 1 --2- 172.21.15.42:0/2034894742 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f84f8134bb0 0x7f84f8134f90 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f84f4000f70 tx=0x7f84f4001980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:08:56.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.148+0000 7f84eeffd640 1 -- 172.21.15.42:0/2034894742 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84f400f5e0 con 0x7f84f8134bb0 2024-09-16T04:08:56.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.148+0000 7f84eeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:56.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.148+0000 7f84eeffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:08:56.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.148+0000 7f84eeffd640 10 monclient: dump: 2024-09-16T04:08:56.171 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:08:56.171 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:08:56.171 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:08:56.171 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:08:56.172 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:56.172 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:08:56.172 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:08:56.172 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:08:56.172 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:08:56.172 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:08:56.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.148+0000 7f84eeffd640 1 -- 172.21.15.42:0/2034894742 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f84f4018070 con 0x7f84f8134bb0 2024-09-16T04:08:56.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.148+0000 7f84eeffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:56.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.148+0000 7f84eeffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:08:56.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.148+0000 7f84eeffd640 1 -- 172.21.15.42:0/2034894742 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f84f8134bb0 msgr2=0x7f84f8134f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:56.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.148+0000 7f84eeffd640 1 --2- 172.21.15.42:0/2034894742 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f84f8134bb0 0x7f84f8134f90 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f84f4000f70 tx=0x7f84f4001980 comp rx=0 tx=0).stop 2024-09-16T04:08:56.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.148+0000 7f84eeffd640 1 -- 172.21.15.42:0/2034894742 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f84f8134bb0 msgr2=0x7f84f8134f90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:08:56.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.148+0000 7f84eeffd640 1 --2- 172.21.15.42:0/2034894742 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f84f8134bb0 0x7f84f8134f90 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:08:56.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.156+0000 7f850049e640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:08:56.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.156+0000 7f850049e640 1 --2- 172.21.15.42:0/2034894742 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f84f81354d0 0x7f84f813a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:08:56.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.156+0000 7f850049e640 10 monclient(hunting): start opening mon connection 2024-09-16T04:08:56.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.156+0000 7f850049e640 1 -- 172.21.15.42:0/2034894742 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f84f813c600 con 0x7f84f81354d0 2024-09-16T04:08:56.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.156+0000 7f84fda39640 1 --2- 172.21.15.42:0/2034894742 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f84f81354d0 0x7f84f813a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:08:56.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.156+0000 7f84fda39640 10 monclient: get_auth_request con 0x7f84f81354d0 auth_method 0 2024-09-16T04:08:56.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.156+0000 7f84fda39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:08:56.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.156+0000 7f84fda39640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:56.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.156+0000 7f84fda39640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:56.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.156+0000 7f84fda39640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:08:56.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.156+0000 7f84fda39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:08:56.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.156+0000 7f84fda39640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:08:56.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.156+0000 7f84fda39640 10 monclient(hunting): handle_auth_done global_id 905182 payload 824 2024-09-16T04:08:56.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.156+0000 7f84fda39640 1 -- 172.21.15.42:0/2034894742 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f84f8068e40 con 0x7f84f81354d0 2024-09-16T04:08:56.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.156+0000 7f84fda39640 1 --2- 172.21.15.42:0/2034894742 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f84f81354d0 0x7f84f813a100 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f84f4017000 tx=0x7f84f4000ee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:08:56.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.156+0000 7f84eeffd640 10 client.1034124.objecter ms_handle_connect 0x7f84f81354d0 2024-09-16T04:08:56.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.156+0000 7f84eeffd640 10 client.1034124.objecter resend_mon_ops 2024-09-16T04:08:56.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.156+0000 7f84eeffd640 1 -- 172.21.15.42:0/2034894742 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84f400fea0 con 0x7f84f81354d0 2024-09-16T04:08:56.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.156+0000 7f84eeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:56.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.156+0000 7f84eeffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:08:56.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.156+0000 7f84eeffd640 10 monclient: dump: 2024-09-16T04:08:56.176 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:08:56.176 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:08:56.176 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:08:56.176 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:08:56.176 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:56.176 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:08:56.176 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:08:56.176 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:08:56.176 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:08:56.176 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:08:56.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.156+0000 7f84eeffd640 1 -- 172.21.15.42:0/2034894742 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4880 (secure 0 0 0) 0x7f84f401e020 con 0x7f84f81354d0 2024-09-16T04:08:56.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.156+0000 7f84eeffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:08:56.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.156+0000 7f84eeffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:08:56.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.156+0000 7f84eeffd640 1 -- 172.21.15.42:0/2034894742 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f84f81354d0 msgr2=0x7f84f813a100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:56.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.156+0000 7f84eeffd640 1 --2- 172.21.15.42:0/2034894742 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f84f81354d0 0x7f84f813a100 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f84f4017000 tx=0x7f84f4000ee0 comp rx=0 tx=0).stop 2024-09-16T04:08:56.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.156+0000 7f84eeffd640 1 -- 172.21.15.42:0/2034894742 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f84f81354d0 msgr2=0x7f84f813a100 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:08:56.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.156+0000 7f84eeffd640 1 --2- 172.21.15.42:0/2034894742 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f84f81354d0 0x7f84f813a100 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:08:56.177 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T04:08:56.177 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:08:56.177 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:08:56.177 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T04:08:56.177 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 248, 2024-09-16T04:08:56.177 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 664339, 2024-09-16T04:08:56.178 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T04:08:56.178 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T04:08:56.178 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T04:08:56.178 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T04:08:56.178 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:08:56.178 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 0, 2024-09-16T04:08:56.178 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:08:56.178 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T04:08:56.178 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T04:08:56.178 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:08:56.178 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:08:56.178 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:08:56.178 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:08:56.179 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:08:56.179 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:08:56.179 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:08:56.179 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:08:56.179 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:08:56.179 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:08:56.179 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:08:56.179 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:08:56.179 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:08:56.179 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T04:08:56.179 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:08:56.179 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:08:56.179 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:08:56.180 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:08:56.180 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:08:56.180 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:08:56.180 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:08:56.180 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:08:56.180 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:08:56.180 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:08:56.180 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:08:56.180 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:08:56.180 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:08:56.180 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T04:08:56.180 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T04:08:56.180 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T04:08:56.181 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T04:08:56.181 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T04:08:56.181 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T04:08:56.181 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:08:56.181 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:08:56.181 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T04:08:56.181 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T04:08:56.181 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T04:08:56.181 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T04:08:56.181 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T04:08:56.181 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T04:08:56.181 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T04:08:56.182 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:08:56.182 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T04:08:56.182 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:08:56.182 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:08:56.182 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:08:56.182 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:08:56.182 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:08:56.182 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:08:56.182 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:08:56.182 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:08:56.182 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:08:56.182 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:08:56.182 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:08:56.182 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:08:56.183 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T04:08:56.183 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:08:56.183 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T04:08:56.183 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:08:56.183 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:08:56.183 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:08:56.183 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:08:56.183 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:08:56.183 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:08:56.183 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:08:56.183 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T04:08:56.183 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:08:56.184 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:08:56.184 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:08:56.184 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:08:56.184 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T04:08:56.184 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:08:56.184 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:08:56.184 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:08:56.184 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:08:56.184 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T04:08:56.184 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T04:08:56.184 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:08:56.184 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:08:56.184 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:08:56.185 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:08:56.185 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:08:56.185 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:08:56.185 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:08:56.185 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:08:56.185 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:08:56.185 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:08:56.185 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:08:56.185 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T04:08:56.185 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:08:56.185 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:08:56.185 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:08:56.185 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:08:56.186 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T04:08:56.186 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:08:56.186 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:08:56.186 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:08:56.186 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:08:56.186 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T04:08:56.186 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T04:08:56.186 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:08:56.186 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:08:56.186 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:08:56.186 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:08:56.186 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:08:56.186 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:08:56.186 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:08:56.187 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:08:56.187 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:08:56.187 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:08:56.187 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:08:56.187 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T04:08:56.187 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:08:56.187 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:08:56.187 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:08:56.187 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:08:56.187 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T04:08:56.187 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:08:56.187 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:08:56.187 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:08:56.187 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:08:56.188 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T04:08:56.188 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T04:08:56.188 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:08:56.188 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:08:56.188 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:08:56.188 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:08:56.188 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:08:56.188 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:08:56.188 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T04:08:56.188 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T04:08:56.188 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:08:56.188 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:08:56.188 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:08:56.188 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:08:56.189 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:08:56.189 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:08:56.189 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T04:08:56.189 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:08:56.189 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:08:56.189 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:08:56.189 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 3 2024-09-16T04:08:56.189 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:08:56.189 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:08:56.189 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T04:08:56.189 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:08:56.189 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:08:56.190 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:08:56.190 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 5 2024-09-16T04:08:56.190 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:08:56.190 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:08:56.190 INFO:teuthology.orchestra.run.smithi042.stdout: "mgr": [ 2024-09-16T04:08:56.190 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:08:56.190 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:08:56.190 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:08:56.190 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:08:56.190 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:08:56.190 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:08:56.190 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:08:56.191 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T04:08:56.191 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T04:08:56.192 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.160+0000 7f850049e640 10 librados: watch_flush enter 2024-09-16T04:08:56.192 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.160+0000 7f850049e640 10 librados: watch_flush exit 2024-09-16T04:08:56.193 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.160+0000 7f850049e640 20 client.1034124.objecter shutdown clearing up homeless session... 2024-09-16T04:08:56.193 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.160+0000 7f850049e640 10 client.1034124.objecter successfully canceled tick 2024-09-16T04:08:56.193 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.160+0000 7f850049e640 1 -- 172.21.15.42:0/2034894742 >> v2:172.21.15.154:6812/254845342 conn(0x7f84dc037810 msgr2=0x7f84dc039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:56.193 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.160+0000 7f850049e640 1 --2- 172.21.15.42:0/2034894742 >> v2:172.21.15.154:6812/254845342 conn(0x7f84dc037810 0x7f84dc039cd0 secure :-1 s=READY pgs=3904 cs=0 l=1 rev1=1 crypto rx=0x7f84f00af000 tx=0x7f84f0004ac0 comp rx=0 tx=0).stop 2024-09-16T04:08:56.193 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.160+0000 7f850049e640 10 monclient: shutdown 2024-09-16T04:08:56.193 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.160+0000 7f850049e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:56.193 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.160+0000 7f850049e640 1 -- 172.21.15.42:0/2034894742 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f84f80a39b0 msgr2=0x7f84f8130300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:56.193 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.160+0000 7f850049e640 1 --2- 172.21.15.42:0/2034894742 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f84f80a39b0 0x7f84f8130300 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f84e800c500 tx=0x7f84e800c9c0 comp rx=0 tx=0).stop 2024-09-16T04:08:56.193 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.160+0000 7f84fea3b640 1 -- 172.21.15.42:0/2034894742 reap_dead start 2024-09-16T04:08:56.193 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.160+0000 7f84fea3b640 1 -- 172.21.15.42:0/2034894742 reap_dead start 2024-09-16T04:08:56.193 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.160+0000 7f850049e640 1 -- 172.21.15.42:0/2034894742 shutdown_connections 2024-09-16T04:08:56.193 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.160+0000 7f850049e640 1 -- 172.21.15.42:0/2034894742 >> 172.21.15.42:0/2034894742 conn(0x7f84f80a9fb0 msgr2=0x7f84f81331e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:56.193 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.160+0000 7f850049e640 1 -- 172.21.15.42:0/2034894742 shutdown_connections 2024-09-16T04:08:56.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.160+0000 7f850049e640 1 -- 172.21.15.42:0/2034894742 wait complete. 2024-09-16T04:08:56.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.160+0000 7f850049e640 1 librados: shutdown 2024-09-16T04:08:56.194 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:08:56.194 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T04:08:56.194 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T04:08:56.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.304+0000 7f73ec6be640 10 monclient: get_monmap_and_config 2024-09-16T04:08:56.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.304+0000 7f73ec6be640 10 monclient: build_initial_monmap 2024-09-16T04:08:56.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.304+0000 7f73ec6be640 10 monclient: monmap: 2024-09-16T04:08:56.311 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:08:56.311 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:08:56.311 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:08:56.309282+0000 2024-09-16T04:08:56.311 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:08:56.309282+0000 2024-09-16T04:08:56.311 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:56.311 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:08:56.311 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:08:56.311 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:08:56.311 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:08:56.311 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:08:56.311 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:08:56.311 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:08:56.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.304+0000 7f73ec6be640 1 Processor -- start 2024-09-16T04:08:56.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.304+0000 7f73ec6be640 1 -- start start 2024-09-16T04:08:56.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.304+0000 7f73ec6be640 10 monclient: init 2024-09-16T04:08:56.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.304+0000 7f73ec6be640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:56.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.304+0000 7f73ec6be640 10 monclient: _add_conns ranks=[0,1,4,3,2] 2024-09-16T04:08:56.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.304+0000 7f73ec6be640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f73e40a39b0 addr v1:172.21.15.42:6789/0 2024-09-16T04:08:56.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.304+0000 7f73ec6be640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f73e40a3d90 addr v1:172.21.15.154:6789/0 2024-09-16T04:08:56.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.304+0000 7f73ec6be640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f73e413daa0 0x7f73e413fe90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:08:56.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.304+0000 7f73ec6be640 10 monclient(hunting): picked mon.noname-c con 0x7f73e413daa0 addr v2:172.21.15.154:3300/0 2024-09-16T04:08:56.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.304+0000 7f73ec6be640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f73e4134290 con 0x7f73e40a39b0 2024-09-16T04:08:56.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.304+0000 7f73ec6be640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f73e409c140 con 0x7f73e40a3d90 2024-09-16T04:08:56.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.304+0000 7f73ec6be640 10 monclient(hunting): start opening mon connection 2024-09-16T04:08:56.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.304+0000 7f73ec6be640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f73e409c420 con 0x7f73e413daa0 2024-09-16T04:08:56.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.304+0000 7f73ec6be640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:56.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.304+0000 7f73ec6be640 10 monclient(hunting): authenticate will time out at 2583.279144s 2024-09-16T04:08:56.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.304+0000 7f73ea45a640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f73e40a39b0 0x7f73e4133b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:34808/0 (socket says 172.21.15.42:34808) 2024-09-16T04:08:56.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.304+0000 7f73ea45a640 1 -- 172.21.15.42:0/32285077 learned_addr learned my addr 172.21.15.42:0/32285077 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T04:08:56.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.304+0000 7f73e9458640 1 -- 172.21.15.42:0/32285077 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1450872451 0 0) 0x7f73e4134290 con 0x7f73e40a39b0 2024-09-16T04:08:56.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.304+0000 7f73e9458640 10 monclient(hunting): _negotiate 2024-09-16T04:08:56.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.304+0000 7f73e9458640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:56.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.304+0000 7f73e9458640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:56.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.304+0000 7f73eac5b640 1 --2- 172.21.15.42:0/32285077 >> v2:172.21.15.154:3300/0 conn(0x7f73e413daa0 0x7f73e413fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:08:56.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.304+0000 7f73e9458640 10 monclient(hunting): my global_id is 1034127 2024-09-16T04:08:56.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.304+0000 7f73e9458640 1 -- 172.21.15.42:0/32285077 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f73d0003530 con 0x7f73e40a39b0 2024-09-16T04:08:56.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.304+0000 7f73eac5b640 10 monclient(hunting): get_auth_request con 0x7f73e413daa0 auth_method 0 2024-09-16T04:08:56.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.304+0000 7f73eac5b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:08:56.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.304+0000 7f73eac5b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:56.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.304+0000 7f73eac5b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:56.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.304+0000 7f73eac5b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:08:56.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.304+0000 7f73eac5b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:08:56.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.304+0000 7f73eac5b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:08:56.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.304+0000 7f73e9458640 1 -- 172.21.15.42:0/32285077 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1703181299 0 0) 0x7f73d0003530 con 0x7f73e40a39b0 2024-09-16T04:08:56.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.304+0000 7f73e9458640 1 -- 172.21.15.42:0/32285077 >> v2:172.21.15.154:3300/0 conn(0x7f73e413daa0 msgr2=0x7f73e413fe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:08:56.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.304+0000 7f73e9458640 1 --2- 172.21.15.42:0/32285077 >> v2:172.21.15.154:3300/0 conn(0x7f73e413daa0 0x7f73e413fe90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:08:56.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.304+0000 7f73e9458640 1 -- 172.21.15.42:0/32285077 >> v1:172.21.15.154:6789/0 conn(0x7f73e40a3d90 legacy=0x7f73e413c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:08:56.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.304+0000 7f73e9458640 10 monclient: _finish_hunting 0 2024-09-16T04:08:56.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.304+0000 7f73e9458640 1 monclient: found mon.noname-a-legacy 2024-09-16T04:08:56.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.304+0000 7f73e9458640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:56.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.304+0000 7f73e9458640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.42:6789/0 2024-09-16T04:08:56.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.304+0000 7f73e9458640 1 -- 172.21.15.42:0/32285077 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73e406dd60 con 0x7f73e40a39b0 2024-09-16T04:08:56.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.304+0000 7f73e9458640 10 monclient: _finish_auth 0 2024-09-16T04:08:56.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.304+0000 7f73e9458640 10 monclient: _check_auth_tickets 2024-09-16T04:08:56.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.304+0000 7f73e9458640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:56.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.304+0000 7f73e9458640 1 -- 172.21.15.42:0/32285077 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2177413734 0 0) 0x7f73e0001680 con 0x7f73e40a39b0 2024-09-16T04:08:56.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.304+0000 7f73e9458640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:56.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.304+0000 7f73e9458640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T04:08:56.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.304+0000 7f73e9458640 10 monclient: dump: 2024-09-16T04:08:56.320 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:08:56.320 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:08:56.320 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:08:56.320 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:08:56.320 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:56.320 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:08:56.320 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:08:56.320 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:08:56.320 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:08:56.320 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:08:56.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.304+0000 7f73e9458640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] but i'm connected to v1:172.21.15.42:6789/0, reconnecting 2024-09-16T04:08:56.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.304+0000 7f73e9458640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:56.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.304+0000 7f73e9458640 1 -- 172.21.15.42:0/32285077 >> v1:172.21.15.42:6789/0 conn(0x7f73e40a39b0 legacy=0x7f73e4133b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:56.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.304+0000 7f73e9458640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:08:56.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.304+0000 7f73e9458640 1 --2- 172.21.15.42:0/32285077 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f73d0003a10 0x7f73d0003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:08:56.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.308+0000 7f73e9458640 10 monclient(hunting): picked mon.c con 0x7f73d0003a10 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:08:56.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.308+0000 7f73e9458640 1 --2- 172.21.15.42:0/32285077 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f73d0004810 0x7f73d0004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:08:56.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.308+0000 7f73e9458640 10 monclient(hunting): picked mon.a con 0x7f73d0004810 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:08:56.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.308+0000 7f73e9458640 1 --2- 172.21.15.42:0/32285077 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f73d00052f0 0x7f73d000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:08:56.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.308+0000 7f73e9458640 10 monclient(hunting): picked mon.b con 0x7f73d00052f0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:08:56.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.308+0000 7f73e9458640 10 monclient(hunting): start opening mon connection 2024-09-16T04:08:56.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.308+0000 7f73e9458640 1 -- 172.21.15.42:0/32285077 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f73e409c420 con 0x7f73d0004810 2024-09-16T04:08:56.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.308+0000 7f73e9458640 10 monclient(hunting): start opening mon connection 2024-09-16T04:08:56.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.308+0000 7f73e9458640 1 -- 172.21.15.42:0/32285077 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f73d000c1f0 con 0x7f73d00052f0 2024-09-16T04:08:56.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.308+0000 7f73e9458640 10 monclient(hunting): start opening mon connection 2024-09-16T04:08:56.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.308+0000 7f73e9458640 1 -- 172.21.15.42:0/32285077 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f73d000c3f0 con 0x7f73d0003a10 2024-09-16T04:08:56.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.308+0000 7f73e9458640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:56.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.308+0000 7f73e9458640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:08:56.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.308+0000 7f73e9c59640 1 --2- 172.21.15.42:0/32285077 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f73d0004810 0x7f73d0004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:08:56.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.308+0000 7f73e9c59640 10 monclient(hunting): get_auth_request con 0x7f73d0004810 auth_method 0 2024-09-16T04:08:56.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.308+0000 7f73e9c59640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:08:56.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.308+0000 7f73e9c59640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:56.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.308+0000 7f73e9c59640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:08:56.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.308+0000 7f73ea45a640 1 --2- 172.21.15.42:0/32285077 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f73d0003a10 0x7f73d0003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:08:56.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.308+0000 7f73e9c59640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:08:56.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.308+0000 7f73e9c59640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:08:56.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.308+0000 7f73e9c59640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:08:56.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.308+0000 7f73ea45a640 10 monclient(hunting): get_auth_request con 0x7f73d0003a10 auth_method 0 2024-09-16T04:08:56.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.308+0000 7f73ea45a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:08:56.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.308+0000 7f73ea45a640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:56.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.308+0000 7f73ea45a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:08:56.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.308+0000 7f73eac5b640 1 --2- 172.21.15.42:0/32285077 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f73d00052f0 0x7f73d000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:08:56.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.308+0000 7f73eac5b640 10 monclient(hunting): get_auth_request con 0x7f73d00052f0 auth_method 0 2024-09-16T04:08:56.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.308+0000 7f73eac5b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:08:56.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.308+0000 7f73eac5b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:56.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.308+0000 7f73eac5b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:08:56.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.308+0000 7f73e9c59640 10 monclient(hunting): handle_auth_done global_id 1034127 payload 293 2024-09-16T04:08:56.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.308+0000 7f73e9c59640 1 -- 172.21.15.42:0/32285077 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f73d0003a10 msgr2=0x7f73d0003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:08:56.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.308+0000 7f73e9c59640 1 --2- 172.21.15.42:0/32285077 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f73d0003a10 0x7f73d0003e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:08:56.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.308+0000 7f73e9c59640 1 -- 172.21.15.42:0/32285077 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f73d00052f0 msgr2=0x7f73d000b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:08:56.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.308+0000 7f73e9c59640 1 --2- 172.21.15.42:0/32285077 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f73d00052f0 0x7f73d000b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:08:56.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.308+0000 7f73e9c59640 10 monclient: _finish_hunting 0 2024-09-16T04:08:56.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.308+0000 7f73e9c59640 1 monclient: found mon.a 2024-09-16T04:08:56.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.308+0000 7f73e9c59640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:56.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.308+0000 7f73e9c59640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:08:56.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.308+0000 7f73e9c59640 1 -- 172.21.15.42:0/32285077 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73d0003530 con 0x7f73d0004810 2024-09-16T04:08:56.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.308+0000 7f73e9c59640 10 monclient: _finish_auth 0 2024-09-16T04:08:56.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.308+0000 7f73e9c59640 10 monclient: _check_auth_tickets 2024-09-16T04:08:56.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.308+0000 7f73e9c59640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:56.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.308+0000 7f73eac5b640 1 -- 172.21.15.42:0/32285077 reap_dead start 2024-09-16T04:08:56.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.308+0000 7f73ec6be640 5 monclient: authenticate success, global_id 1034127 2024-09-16T04:08:56.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.308+0000 7f73ec6be640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:08:56.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.308+0000 7f73ea45a640 1 --2- 172.21.15.42:0/32285077 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f73d0003a10 0x7f73d0003e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T04:08:56.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.308+0000 7f73e9c59640 1 --2- 172.21.15.42:0/32285077 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f73d0004810 0x7f73d0004c90 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f73d4001780 tx=0x7f73d401c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d379866050ef12a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:08:56.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.308+0000 7f73e9458640 1 -- 172.21.15.42:0/32285077 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73d40013a0 con 0x7f73d0004810 2024-09-16T04:08:56.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.308+0000 7f73e9458640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:56.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.308+0000 7f73e9458640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:08:56.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.308+0000 7f73e9458640 10 monclient: dump: 2024-09-16T04:08:56.326 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:08:56.326 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:08:56.326 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:08:56.326 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:08:56.326 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:56.326 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:08:56.326 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:08:56.327 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:08:56.327 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:08:56.327 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:08:56.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.308+0000 7f73e9458640 1 -- 172.21.15.42:0/32285077 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f73d40052f0 con 0x7f73d0004810 2024-09-16T04:08:56.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.308+0000 7f73ec6be640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:08:56.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.308+0000 7f73e9458640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:56.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.308+0000 7f73ec6be640 10 monclient: get_monmap_and_config success 2024-09-16T04:08:56.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.308+0000 7f73ec6be640 4 set_mon_vals no callback set 2024-09-16T04:08:56.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.308+0000 7f73e9458640 1 -- 172.21.15.42:0/32285077 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73d4003840 con 0x7f73d0004810 2024-09-16T04:08:56.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.308+0000 7f73e9458640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:56.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.308+0000 7f73e9458640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:08:56.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.308+0000 7f73e9458640 10 monclient: dump: 2024-09-16T04:08:56.327 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:08:56.328 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:08:56.328 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:08:56.328 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:08:56.328 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:56.328 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:08:56.328 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:08:56.328 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:08:56.329 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:08:56.329 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:08:56.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.308+0000 7f73ec6be640 10 monclient: shutdown 2024-09-16T04:08:56.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.308+0000 7f73ec6be640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:56.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.308+0000 7f73ec6be640 1 -- 172.21.15.42:0/32285077 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f73d0004810 msgr2=0x7f73d0004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:56.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.308+0000 7f73ec6be640 1 --2- 172.21.15.42:0/32285077 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f73d0004810 0x7f73d0004c90 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f73d4001780 tx=0x7f73d401c3e0 comp rx=0 tx=0).stop 2024-09-16T04:08:56.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.308+0000 7f73ec6be640 1 -- 172.21.15.42:0/32285077 shutdown_connections 2024-09-16T04:08:56.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.308+0000 7f73ec6be640 1 --2- 172.21.15.42:0/32285077 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f73d0004810 0x7f73d0004c90 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:08:56.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.308+0000 7f73ec6be640 1 -- 172.21.15.42:0/32285077 >> 172.21.15.42:0/32285077 conn(0x7f73e40a9fb0 msgr2=0x7f73e40aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:56.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.308+0000 7f73ec6be640 1 -- 172.21.15.42:0/32285077 shutdown_connections 2024-09-16T04:08:56.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.308+0000 7f73ec6be640 1 -- 172.21.15.42:0/32285077 wait complete. 2024-09-16T04:08:56.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.312+0000 7f73ec6be640 10 monclient: build_initial_monmap 2024-09-16T04:08:56.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.312+0000 7f73ec6be640 10 monclient: monmap: 2024-09-16T04:08:56.335 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:08:56.335 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:08:56.335 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:08:56.335 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:08:56.335 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:56.336 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:08:56.336 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:08:56.336 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:08:56.336 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:08:56.336 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:08:56.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.312+0000 7f73ec6be640 1 librados: starting msgr at 2024-09-16T04:08:56.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.312+0000 7f73ec6be640 1 librados: starting objecter 2024-09-16T04:08:56.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.312+0000 7f73ec6be640 1 Processor -- start 2024-09-16T04:08:56.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.312+0000 7f73ec6be640 1 -- start start 2024-09-16T04:08:56.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.312+0000 7f73ec6be640 1 librados: setting wanted keys 2024-09-16T04:08:56.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.312+0000 7f73ec6be640 1 librados: calling monclient init 2024-09-16T04:08:56.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.312+0000 7f73ec6be640 10 monclient: init 2024-09-16T04:08:56.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.312+0000 7f73ec6be640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:56.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.312+0000 7f73ec6be640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:08:56.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.312+0000 7f73ec6be640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f73d0004810 0x7f73e409f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:08:56.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.312+0000 7f73ec6be640 10 monclient(hunting): picked mon.noname-a con 0x7f73d0004810 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:08:56.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.312+0000 7f73ec6be640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f73e409f700 0x7f73e409fb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:08:56.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.312+0000 7f73ec6be640 10 monclient(hunting): picked mon.noname-c con 0x7f73e409f700 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:08:56.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.312+0000 7f73ec6be640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f73e40a00a0 0x7f73e41d5c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:08:56.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.312+0000 7f73ec6be640 10 monclient(hunting): picked mon.noname-b con 0x7f73e40a00a0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:08:56.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.312+0000 7f73ec6be640 10 monclient(hunting): start opening mon connection 2024-09-16T04:08:56.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.312+0000 7f73ec6be640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f73e4137ba0 con 0x7f73d0004810 2024-09-16T04:08:56.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.312+0000 7f73ec6be640 10 monclient(hunting): start opening mon connection 2024-09-16T04:08:56.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.312+0000 7f73ec6be640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f73e4137a20 con 0x7f73e40a00a0 2024-09-16T04:08:56.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.312+0000 7f73ec6be640 10 monclient(hunting): start opening mon connection 2024-09-16T04:08:56.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.312+0000 7f73ec6be640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f73e4137d20 con 0x7f73e409f700 2024-09-16T04:08:56.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.312+0000 7f73ec6be640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:56.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.312+0000 7f73ec6be640 10 monclient(hunting): authenticate will time out at 2583.285009s 2024-09-16T04:08:56.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.312+0000 7f73e9c59640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f73e409f700 0x7f73e409fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:08:56.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.312+0000 7f73ea45a640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f73d0004810 0x7f73e409f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:08:56.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.312+0000 7f73e9c59640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f73e409f700 0x7f73e409fb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:40342/0 (socket says 172.21.15.42:40342) 2024-09-16T04:08:56.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.312+0000 7f73e9c59640 1 -- 172.21.15.42:0/466851550 learned_addr learned my addr 172.21.15.42:0/466851550 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:08:56.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.312+0000 7f73e9c59640 10 monclient(hunting): get_auth_request con 0x7f73e409f700 auth_method 0 2024-09-16T04:08:56.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.312+0000 7f73e9c59640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:08:56.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.312+0000 7f73e9c59640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:56.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.312+0000 7f73e9c59640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:56.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.312+0000 7f73ea45a640 10 monclient(hunting): get_auth_request con 0x7f73d0004810 auth_method 0 2024-09-16T04:08:56.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.312+0000 7f73ea45a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:08:56.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.312+0000 7f73ea45a640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:56.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.312+0000 7f73ea45a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:56.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.312+0000 7f73e9c59640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:08:56.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.312+0000 7f73e9c59640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:08:56.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.312+0000 7f73e9c59640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:08:56.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.312+0000 7f73ea45a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:08:56.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.312+0000 7f73ea45a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:08:56.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.312+0000 7f73ea45a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:08:56.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.312+0000 7f73e9c59640 10 monclient(hunting): handle_auth_done global_id 1014290 payload 824 2024-09-16T04:08:56.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.312+0000 7f73eac5b640 1 --2- 172.21.15.42:0/466851550 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f73e40a00a0 0x7f73e41d5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:08:56.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.312+0000 7f73e9c59640 1 -- 172.21.15.42:0/466851550 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f73e40a00a0 msgr2=0x7f73e41d5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:56.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.312+0000 7f73e9c59640 1 --2- 172.21.15.42:0/466851550 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f73e40a00a0 0x7f73e41d5c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:08:56.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.312+0000 7f73e9c59640 1 -- 172.21.15.42:0/466851550 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f73d0004810 msgr2=0x7f73e409f1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:56.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.312+0000 7f73e9c59640 1 --2- 172.21.15.42:0/466851550 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f73d0004810 0x7f73e409f1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:08:56.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.312+0000 7f73e9c59640 10 monclient: _finish_hunting 0 2024-09-16T04:08:56.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.312+0000 7f73e9c59640 1 monclient: found mon.noname-c 2024-09-16T04:08:56.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.312+0000 7f73e9c59640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:56.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.312+0000 7f73e9c59640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T04:08:56.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.312+0000 7f73e9c59640 1 -- 172.21.15.42:0/466851550 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73e413d4a0 con 0x7f73e409f700 2024-09-16T04:08:56.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.312+0000 7f73ea45a640 1 --2- 172.21.15.42:0/466851550 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f73d0004810 0x7f73e409f1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:08:56.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.312+0000 7f73e9c59640 1 --2- 172.21.15.42:0/466851550 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f73e409f700 0x7f73e409fb60 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f73d401c210 tx=0x7f73d0005840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:08:56.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.312+0000 7f73daffd640 10 client.?.objecter ms_handle_connect 0x7f73e409f700 2024-09-16T04:08:56.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.312+0000 7f73daffd640 10 client.?.objecter resend_mon_ops 2024-09-16T04:08:56.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.312+0000 7f73daffd640 1 -- 172.21.15.42:0/466851550 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73d40b8020 con 0x7f73e409f700 2024-09-16T04:08:56.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.312+0000 7f73daffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:56.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.312+0000 7f73daffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:08:56.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.312+0000 7f73daffd640 10 monclient: dump: 2024-09-16T04:08:56.341 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:08:56.341 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:08:56.341 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:08:56.341 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:08:56.344 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:56.344 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:08:56.344 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:08:56.344 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:08:56.344 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:08:56.344 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:08:56.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.312+0000 7f73daffd640 10 monclient: _finish_auth 0 2024-09-16T04:08:56.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.312+0000 7f73daffd640 10 monclient: _check_auth_tickets 2024-09-16T04:08:56.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.312+0000 7f73daffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:56.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.312+0000 7f73daffd640 1 -- 172.21.15.42:0/466851550 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f73d40054a0 con 0x7f73e409f700 2024-09-16T04:08:56.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.312+0000 7f73ec6be640 5 monclient: authenticate success, global_id 1014290 2024-09-16T04:08:56.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.312+0000 7f73ec6be640 10 monclient: _renew_subs 2024-09-16T04:08:56.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.312+0000 7f73ec6be640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:08:56.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.312+0000 7f73ec6be640 1 -- 172.21.15.42:0/466851550 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73e40a2650 con 0x7f73e409f700 2024-09-16T04:08:56.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.312+0000 7f73daffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:56.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.312+0000 7f73daffd640 1 -- 172.21.15.42:0/466851550 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73d4007a40 con 0x7f73e409f700 2024-09-16T04:08:56.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.312+0000 7f73daffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:56.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.312+0000 7f73daffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:08:56.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.312+0000 7f73daffd640 10 monclient: dump: 2024-09-16T04:08:56.346 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:08:56.346 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:08:56.346 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:08:56.346 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:08:56.346 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:56.346 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:08:56.346 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:08:56.346 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:08:56.346 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:08:56.346 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:08:56.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.312+0000 7f73e8c57640 4 set_mon_vals no callback set 2024-09-16T04:08:56.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.312+0000 7f73ec6be640 10 client.1014290.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:08:56.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.312+0000 7f73ec6be640 10 monclient: _renew_subs 2024-09-16T04:08:56.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.312+0000 7f73ec6be640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:08:56.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.312+0000 7f73ec6be640 1 -- 172.21.15.42:0/466851550 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f73e40a93d0 con 0x7f73e409f700 2024-09-16T04:08:56.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.312+0000 7f73ec6be640 1 librados: init done 2024-09-16T04:08:56.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.312+0000 7f73ec6be640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:56.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.312+0000 7f73ec6be640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:56.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.312+0000 7f73daffd640 1 -- 172.21.15.42:0/466851550 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f73d40bb020 con 0x7f73e409f700 2024-09-16T04:08:56.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.312+0000 7f73ec6be640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:08:56.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.312+0000 7f73ec6be640 1 -- 172.21.15.42:0/466851550 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f73e406f680 con 0x7f73e409f700 2024-09-16T04:08:56.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.312+0000 7f73daffd640 1 --2- 172.21.15.42:0/466851550 >> v2:172.21.15.154:6812/254845342 conn(0x7f73c8037810 0x7f73c8039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:08:56.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.312+0000 7f73daffd640 1 -- 172.21.15.42:0/466851550 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(706..706 src has 451..706) ==== 5040+0+0 (secure 0 0 0) 0x7f73d40e2a60 con 0x7f73e409f700 2024-09-16T04:08:56.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.312+0000 7f73daffd640 10 client.1014290.objecter ms_dispatch 0x7f73e4000ba0 osd_map(706..706 src has 451..706) 2024-09-16T04:08:56.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.312+0000 7f73daffd640 3 client.1014290.objecter handle_osd_map got epochs [706,706] > 0 2024-09-16T04:08:56.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.312+0000 7f73daffd640 3 client.1014290.objecter handle_osd_map decoding full epoch 706 2024-09-16T04:08:56.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.312+0000 7f73daffd640 20 client.1014290.objecter dump_active .. 0 homeless 2024-09-16T04:08:56.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.316+0000 7f73ea45a640 1 --2- 172.21.15.42:0/466851550 >> v2:172.21.15.154:6812/254845342 conn(0x7f73c8037810 0x7f73c8039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:08:56.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.316+0000 7f73ea45a640 10 monclient: get_auth_request con 0x7f73c8037810 auth_method 0 2024-09-16T04:08:56.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.316+0000 7f73ea45a640 1 --2- 172.21.15.42:0/466851550 >> v2:172.21.15.154:6812/254845342 conn(0x7f73c8037810 0x7f73c8039cd0 secure :-1 s=READY pgs=3905 cs=0 l=1 rev1=1 crypto rx=0x7f73e0001ec0 tx=0x7f73e0007040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:08:56.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.316+0000 7f73daffd640 10 client.1014290.objecter ms_handle_connect 0x7f73c8037810 2024-09-16T04:08:56.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.316+0000 7f73daffd640 1 -- 172.21.15.42:0/466851550 <== mon.2 v2:172.21.15.42:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f73d40c2030 con 0x7f73e409f700 2024-09-16T04:08:56.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.316+0000 7f73daffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:56.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.316+0000 7f73daffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:08:56.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.540+0000 7f73ec6be640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T04:08:56.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.540+0000 7f73ec6be640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T04:08:56.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.540+0000 7f73ec6be640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:08:56.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.540+0000 7f73ec6be640 1 -- 172.21.15.42:0/466851550 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f73e4134230 con 0x7f73e409f700 2024-09-16T04:08:56.550 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.544+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.550 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.544+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=29,config=2,health=12,kv=10,logm=47} crc {auth=1418691219,config=692619927,health=1488147316,kv=3446188498,logm=1528812129}) 2024-09-16T04:08:56.551 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.544+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=29,config=2,health=12,kv=10,logm=47} crc {auth=2038790731,config=692619927,health=1836163894,kv=1270464448,logm=2892042517}) 2024-09-16T04:08:56.551 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.544+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.551 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.544+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=29,config=2,health=12,kv=10,logm=47} crc {auth=1418691219,config=692619927,health=1488147316,kv=3446188498,logm=1528812129}) 2024-09-16T04:08:56.551 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.544+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=30,config=2,health=11,kv=10,logm=47} crc {auth=536252765,config=692619927,health=3273057861,kv=3446188498,logm=2220570347}) 2024-09-16T04:08:56.551 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.544+0000 7f73daffd640 1 -- 172.21.15.42:0/466851550 <== mon.2 v2:172.21.15.42:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f73e413d4a0 con 0x7f73e409f700 2024-09-16T04:08:56.551 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.544+0000 7f73daffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T04:08:56.551 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.544+0000 7f73daffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:08:56.553 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.544+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.553 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.544+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3040458601}) 2024-09-16T04:08:56.553 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.544+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4115927605}) 2024-09-16T04:08:56.553 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.544+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.553 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.544+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3040458601}) 2024-09-16T04:08:56.553 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.544+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1136985714}) 2024-09-16T04:08:56.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.548+0000 7f73ec6be640 10 librados: watch_flush enter 2024-09-16T04:08:56.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.548+0000 7f73ec6be640 10 librados: watch_flush exit 2024-09-16T04:08:56.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.548+0000 7f73ec6be640 20 client.1014290.objecter shutdown clearing up homeless session... 2024-09-16T04:08:56.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.548+0000 7f73ec6be640 10 client.1014290.objecter successfully canceled tick 2024-09-16T04:08:56.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.548+0000 7f73ec6be640 1 -- 172.21.15.42:0/466851550 >> v2:172.21.15.154:6812/254845342 conn(0x7f73c8037810 msgr2=0x7f73c8039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:56.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.548+0000 7f73ec6be640 1 --2- 172.21.15.42:0/466851550 >> v2:172.21.15.154:6812/254845342 conn(0x7f73c8037810 0x7f73c8039cd0 secure :-1 s=READY pgs=3905 cs=0 l=1 rev1=1 crypto rx=0x7f73e0001ec0 tx=0x7f73e0007040 comp rx=0 tx=0).stop 2024-09-16T04:08:56.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.548+0000 7f73ec6be640 10 monclient: shutdown 2024-09-16T04:08:56.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.548+0000 7f73ec6be640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:56.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.548+0000 7f73ec6be640 1 -- 172.21.15.42:0/466851550 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f73e409f700 msgr2=0x7f73e409fb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:56.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.548+0000 7f73ec6be640 1 --2- 172.21.15.42:0/466851550 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f73e409f700 0x7f73e409fb60 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f73d401c210 tx=0x7f73d0005840 comp rx=0 tx=0).stop 2024-09-16T04:08:56.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.548+0000 7f73ec6be640 1 -- 172.21.15.42:0/466851550 shutdown_connections 2024-09-16T04:08:56.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.548+0000 7f73ec6be640 1 --2- 172.21.15.42:0/466851550 >> v2:172.21.15.154:6812/254845342 conn(0x7f73c8037810 0x7f73c8039cd0 unknown :-1 s=CLOSED pgs=3905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:08:56.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.548+0000 7f73ec6be640 1 --2- 172.21.15.42:0/466851550 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f73e40a00a0 0x7f73e41d5c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:08:56.556 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.548+0000 7f73ec6be640 1 --2- 172.21.15.42:0/466851550 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f73e409f700 0x7f73e409fb60 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:08:56.556 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.548+0000 7f73ec6be640 1 --2- 172.21.15.42:0/466851550 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f73d0004810 0x7f73e409f1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:08:56.556 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.548+0000 7f73ec6be640 1 -- 172.21.15.42:0/466851550 >> 172.21.15.42:0/466851550 conn(0x7f73e40a9fb0 msgr2=0x7f73e41366d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:56.556 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.548+0000 7f73ec6be640 1 -- 172.21.15.42:0/466851550 shutdown_connections 2024-09-16T04:08:56.556 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.548+0000 7f73ec6be640 1 -- 172.21.15.42:0/466851550 wait complete. 2024-09-16T04:08:56.556 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:56.548+0000 7f73ec6be640 1 librados: shutdown 2024-09-16T04:08:56.557 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.548+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.557 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.548+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2400454693}) 2024-09-16T04:08:56.557 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.548+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2971948903}) 2024-09-16T04:08:56.557 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.548+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.557 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.548+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2400454693}) 2024-09-16T04:08:56.557 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.548+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4172847225}) 2024-09-16T04:08:56.559 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.552+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.560 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.552+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=56964874}) 2024-09-16T04:08:56.560 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.552+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1873131157}) 2024-09-16T04:08:56.560 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.552+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.560 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.552+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=56964874}) 2024-09-16T04:08:56.560 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.552+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=867133822}) 2024-09-16T04:08:56.563 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.556+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.563 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.556+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2287312720}) 2024-09-16T04:08:56.563 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.556+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2904527237}) 2024-09-16T04:08:56.564 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.556+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.564 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.556+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2287312720}) 2024-09-16T04:08:56.564 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.556+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3084735237}) 2024-09-16T04:08:56.566 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.560+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.566 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.560+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=358621379}) 2024-09-16T04:08:56.566 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.560+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2862049825}) 2024-09-16T04:08:56.566 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.560+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.566 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.560+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=358621379}) 2024-09-16T04:08:56.566 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.560+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4203238405}) 2024-09-16T04:08:56.567 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.560+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.568 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.560+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3790210034}) 2024-09-16T04:08:56.568 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.560+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4030316572}) 2024-09-16T04:08:56.568 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.560+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.568 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.560+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3790210034}) 2024-09-16T04:08:56.568 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.560+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=256755617}) 2024-09-16T04:08:56.569 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.564+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.569 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.564+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3721822414}) 2024-09-16T04:08:56.569 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.564+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3927209502}) 2024-09-16T04:08:56.569 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.564+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.569 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.564+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3721822414}) 2024-09-16T04:08:56.570 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.564+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1909381646}) 2024-09-16T04:08:56.571 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.564+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.571 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.564+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2394810478}) 2024-09-16T04:08:56.571 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.564+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3979520204}) 2024-09-16T04:08:56.571 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.564+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.571 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.564+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2394810478}) 2024-09-16T04:08:56.571 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.564+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1140334244}) 2024-09-16T04:08:56.573 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.564+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.573 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.564+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3023709165}) 2024-09-16T04:08:56.573 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.564+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2244346704}) 2024-09-16T04:08:56.573 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.564+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.573 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.564+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3023709165}) 2024-09-16T04:08:56.573 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.564+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2453783490}) 2024-09-16T04:08:56.574 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T04:08:56.575 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.568+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.575 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.568+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2411640635}) 2024-09-16T04:08:56.575 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.568+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3439770927}) 2024-09-16T04:08:56.575 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.568+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.575 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.568+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2411640635}) 2024-09-16T04:08:56.575 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.568+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1493340543}) 2024-09-16T04:08:56.576 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.568+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.576 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.568+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3199314533}) 2024-09-16T04:08:56.576 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.568+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2382135372}) 2024-09-16T04:08:56.577 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.568+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.577 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.568+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3199314533}) 2024-09-16T04:08:56.577 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.568+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2693385315}) 2024-09-16T04:08:56.578 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.572+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.578 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.572+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3442795613}) 2024-09-16T04:08:56.578 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.572+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3679198790}) 2024-09-16T04:08:56.578 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.572+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.578 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.572+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3442795613}) 2024-09-16T04:08:56.578 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.572+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=515438177}) 2024-09-16T04:08:56.580 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.572+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.580 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.572+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4264044803}) 2024-09-16T04:08:56.580 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.572+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3914750171}) 2024-09-16T04:08:56.580 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.572+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.580 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.572+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4264044803}) 2024-09-16T04:08:56.581 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.572+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3709054076}) 2024-09-16T04:08:56.582 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.576+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.582 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.576+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1174324709}) 2024-09-16T04:08:56.582 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.576+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3249291507}) 2024-09-16T04:08:56.582 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.576+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.582 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.576+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1174324709}) 2024-09-16T04:08:56.582 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.576+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1965053248}) 2024-09-16T04:08:56.583 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.576+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.584 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.576+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4136292135}) 2024-09-16T04:08:56.584 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.576+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2014220031}) 2024-09-16T04:08:56.584 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.576+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.584 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.576+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4136292135}) 2024-09-16T04:08:56.584 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.576+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3865728052}) 2024-09-16T04:08:56.585 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.580+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.586 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.580+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3933039488}) 2024-09-16T04:08:56.586 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.580+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2460014921}) 2024-09-16T04:08:56.586 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.580+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.586 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.580+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3933039488}) 2024-09-16T04:08:56.586 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.580+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3777230978}) 2024-09-16T04:08:56.587 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.580+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.587 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.580+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3257425230}) 2024-09-16T04:08:56.587 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.580+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=594474825}) 2024-09-16T04:08:56.587 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.580+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.587 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.580+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3257425230}) 2024-09-16T04:08:56.587 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.580+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3623781003}) 2024-09-16T04:08:56.589 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.580+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.589 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.580+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1551532847}) 2024-09-16T04:08:56.589 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.580+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1933595621}) 2024-09-16T04:08:56.589 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.580+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.589 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.580+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1551532847}) 2024-09-16T04:08:56.589 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.580+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2078169147}) 2024-09-16T04:08:56.590 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.584+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.591 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.584+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3630012570}) 2024-09-16T04:08:56.591 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.584+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3201061449}) 2024-09-16T04:08:56.591 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.584+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.591 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.584+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3630012570}) 2024-09-16T04:08:56.591 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.584+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1164972531}) 2024-09-16T04:08:56.593 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.584+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.593 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.584+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2273325357}) 2024-09-16T04:08:56.593 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.584+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3301089760}) 2024-09-16T04:08:56.593 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.584+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.593 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.584+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2273325357}) 2024-09-16T04:08:56.593 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.584+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1171336547}) 2024-09-16T04:08:56.594 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.588+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.594 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.588+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3375052891}) 2024-09-16T04:08:56.595 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.588+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=348935496}) 2024-09-16T04:08:56.595 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.588+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.595 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.588+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3375052891}) 2024-09-16T04:08:56.595 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.588+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3787450168}) 2024-09-16T04:08:56.596 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.588+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.596 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.588+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2439605087}) 2024-09-16T04:08:56.596 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.588+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1840209}) 2024-09-16T04:08:56.596 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.588+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.596 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.588+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2439605087}) 2024-09-16T04:08:56.596 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.588+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3278926703}) 2024-09-16T04:08:56.598 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.592+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.598 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.592+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2483479839}) 2024-09-16T04:08:56.598 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.592+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=381344567}) 2024-09-16T04:08:56.598 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.592+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.598 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.592+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2483479839}) 2024-09-16T04:08:56.598 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.592+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3620909634}) 2024-09-16T04:08:56.599 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.592+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.599 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.592+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2243246433}) 2024-09-16T04:08:56.600 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.592+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=302916243}) 2024-09-16T04:08:56.600 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.592+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.600 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.592+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2243246433}) 2024-09-16T04:08:56.600 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.592+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2908470540}) 2024-09-16T04:08:56.601 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.596+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.601 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.596+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2034376869}) 2024-09-16T04:08:56.601 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.596+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=547827489}) 2024-09-16T04:08:56.601 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.596+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.601 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.596+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2034376869}) 2024-09-16T04:08:56.601 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.596+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=647958791}) 2024-09-16T04:08:56.603 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.596+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.603 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.596+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=265166233}) 2024-09-16T04:08:56.603 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.596+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1325103863}) 2024-09-16T04:08:56.603 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.596+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.603 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.596+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=265166233}) 2024-09-16T04:08:56.603 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.596+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2368198780}) 2024-09-16T04:08:56.605 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.596+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.605 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.596+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3188796198}) 2024-09-16T04:08:56.605 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.596+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=136929895}) 2024-09-16T04:08:56.605 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.596+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.605 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.596+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3188796198}) 2024-09-16T04:08:56.605 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.596+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1261907617}) 2024-09-16T04:08:56.606 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.600+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.606 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.600+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3560864826}) 2024-09-16T04:08:56.606 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.600+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1307856396}) 2024-09-16T04:08:56.606 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.600+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.607 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.600+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3560864826}) 2024-09-16T04:08:56.607 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.600+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3099341276}) 2024-09-16T04:08:56.608 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.600+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.608 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.600+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2767582910}) 2024-09-16T04:08:56.608 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.600+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=59311950}) 2024-09-16T04:08:56.608 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.600+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.608 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.600+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2767582910}) 2024-09-16T04:08:56.608 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.600+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3497664583}) 2024-09-16T04:08:56.610 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.604+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.610 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.604+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2881065917}) 2024-09-16T04:08:56.610 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.604+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3273617676}) 2024-09-16T04:08:56.610 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.604+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.610 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.604+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2881065917}) 2024-09-16T04:08:56.610 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.604+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2613199739}) 2024-09-16T04:08:56.611 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.604+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.612 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.604+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=279826255}) 2024-09-16T04:08:56.612 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.604+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=920823110}) 2024-09-16T04:08:56.612 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.604+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.612 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.604+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=279826255}) 2024-09-16T04:08:56.612 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.604+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=256595382}) 2024-09-16T04:08:56.613 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.608+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.613 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.608+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2389221745}) 2024-09-16T04:08:56.613 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.608+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1808627671}) 2024-09-16T04:08:56.613 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.608+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.613 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.608+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2389221745}) 2024-09-16T04:08:56.613 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.608+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3169852923}) 2024-09-16T04:08:56.615 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.608+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.615 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.608+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3983524326}) 2024-09-16T04:08:56.615 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.608+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3012852922}) 2024-09-16T04:08:56.615 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.608+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.615 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.608+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3983524326}) 2024-09-16T04:08:56.615 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.608+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=814570669}) 2024-09-16T04:08:56.617 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.608+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.617 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.608+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1630801720}) 2024-09-16T04:08:56.617 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.608+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=453044618}) 2024-09-16T04:08:56.617 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.608+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.617 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.608+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1630801720}) 2024-09-16T04:08:56.617 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.608+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=81354815}) 2024-09-16T04:08:56.619 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.612+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.619 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.612+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2930312751}) 2024-09-16T04:08:56.619 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.612+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2689716416}) 2024-09-16T04:08:56.619 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.612+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.619 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.612+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2930312751}) 2024-09-16T04:08:56.619 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.612+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=563638469}) 2024-09-16T04:08:56.620 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.612+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.621 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.612+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1792500011}) 2024-09-16T04:08:56.621 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.612+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4078546723}) 2024-09-16T04:08:56.621 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.612+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.621 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.612+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1792500011}) 2024-09-16T04:08:56.621 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.612+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2929274673}) 2024-09-16T04:08:56.622 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.616+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.622 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.616+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4213236243}) 2024-09-16T04:08:56.622 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.616+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2623694596}) 2024-09-16T04:08:56.623 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.616+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.623 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.616+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4213236243}) 2024-09-16T04:08:56.623 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.616+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=421267703}) 2024-09-16T04:08:56.624 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.616+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.624 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.616+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=718746025}) 2024-09-16T04:08:56.624 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.616+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3436402232}) 2024-09-16T04:08:56.624 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.616+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.624 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.616+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=718746025}) 2024-09-16T04:08:56.624 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.616+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1266060182}) 2024-09-16T04:08:56.626 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.620+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.626 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.620+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3757143338}) 2024-09-16T04:08:56.626 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.620+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3826314157}) 2024-09-16T04:08:56.626 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.620+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.626 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.620+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3757143338}) 2024-09-16T04:08:56.626 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.620+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1401877929}) 2024-09-16T04:08:56.628 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.620+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.628 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.620+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1717089312}) 2024-09-16T04:08:56.628 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.620+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1135023202}) 2024-09-16T04:08:56.628 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.620+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.628 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.620+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1717089312}) 2024-09-16T04:08:56.628 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.620+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=955497554}) 2024-09-16T04:08:56.629 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.620+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.629 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.620+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3935609114}) 2024-09-16T04:08:56.629 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.624+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1273825901}) 2024-09-16T04:08:56.629 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.624+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.629 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.624+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3935609114}) 2024-09-16T04:08:56.629 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.624+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1055818829}) 2024-09-16T04:08:56.631 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.624+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.631 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.624+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3128256322}) 2024-09-16T04:08:56.631 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.624+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1099056859}) 2024-09-16T04:08:56.631 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.624+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.631 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.624+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3128256322}) 2024-09-16T04:08:56.631 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.624+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1344265089}) 2024-09-16T04:08:56.632 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.624+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.633 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.624+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3740563514}) 2024-09-16T04:08:56.633 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.624+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3128460464}) 2024-09-16T04:08:56.633 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.624+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.633 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.624+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3740563514}) 2024-09-16T04:08:56.633 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.624+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3299783384}) 2024-09-16T04:08:56.634 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.628+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.634 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.628+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=45493565}) 2024-09-16T04:08:56.634 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.628+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=73143693}) 2024-09-16T04:08:56.634 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.628+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.634 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.628+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=45493565}) 2024-09-16T04:08:56.634 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.628+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=739899911}) 2024-09-16T04:08:56.636 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.628+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.636 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.628+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1566987000}) 2024-09-16T04:08:56.636 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.628+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1372669936}) 2024-09-16T04:08:56.636 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.628+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.636 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.628+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1566987000}) 2024-09-16T04:08:56.636 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.628+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4248713901}) 2024-09-16T04:08:56.638 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.632+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.638 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.632+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2533821815}) 2024-09-16T04:08:56.638 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.632+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=301076257}) 2024-09-16T04:08:56.638 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.632+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.638 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.632+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2533821815}) 2024-09-16T04:08:56.638 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.632+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2945030880}) 2024-09-16T04:08:56.639 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.632+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.640 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.632+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2619333262}) 2024-09-16T04:08:56.640 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.632+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=964000296}) 2024-09-16T04:08:56.640 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.632+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.640 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.632+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2619333262}) 2024-09-16T04:08:56.640 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.632+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3419486487}) 2024-09-16T04:08:56.641 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.636+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.641 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.636+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1608551036}) 2024-09-16T04:08:56.641 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.636+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2073912544}) 2024-09-16T04:08:56.642 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.636+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.642 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.636+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1608551036}) 2024-09-16T04:08:56.642 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.636+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2479912926}) 2024-09-16T04:08:56.643 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.636+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.643 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.636+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3111910011}) 2024-09-16T04:08:56.643 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.636+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3350254231}) 2024-09-16T04:08:56.643 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.636+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.644 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.636+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3111910011}) 2024-09-16T04:08:56.644 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.636+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=718909998}) 2024-09-16T04:08:56.645 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.636+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.645 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.636+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=567238570}) 2024-09-16T04:08:56.645 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.640+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1146201348}) 2024-09-16T04:08:56.645 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.640+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.645 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.640+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=567238570}) 2024-09-16T04:08:56.646 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.640+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=62359569}) 2024-09-16T04:08:56.647 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.640+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.647 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.640+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=980427454}) 2024-09-16T04:08:56.647 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.640+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1664885291}) 2024-09-16T04:08:56.647 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.640+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.647 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.640+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=980427454}) 2024-09-16T04:08:56.647 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.640+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1428838872}) 2024-09-16T04:08:56.649 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.640+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.649 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.640+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2790575767}) 2024-09-16T04:08:56.649 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.640+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2908058366}) 2024-09-16T04:08:56.649 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.640+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.649 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.640+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2790575767}) 2024-09-16T04:08:56.649 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.640+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1864657336}) 2024-09-16T04:08:56.651 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.644+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.651 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.644+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3320811406}) 2024-09-16T04:08:56.651 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.644+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3305797967}) 2024-09-16T04:08:56.651 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.644+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.651 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.644+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3320811406}) 2024-09-16T04:08:56.651 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.644+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1644117370}) 2024-09-16T04:08:56.653 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.644+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.653 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.644+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4056715551}) 2024-09-16T04:08:56.653 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.644+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3474346508}) 2024-09-16T04:08:56.653 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.644+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.653 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.644+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4056715551}) 2024-09-16T04:08:56.653 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.644+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1919331410}) 2024-09-16T04:08:56.654 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.648+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.655 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.648+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2217812859}) 2024-09-16T04:08:56.655 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.648+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3021527301}) 2024-09-16T04:08:56.655 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.648+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.655 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.648+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2217812859}) 2024-09-16T04:08:56.655 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.648+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2611673785}) 2024-09-16T04:08:56.656 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.648+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.656 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.648+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3118911060}) 2024-09-16T04:08:56.656 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.648+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2326697239}) 2024-09-16T04:08:56.657 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.648+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.657 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.648+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3118911060}) 2024-09-16T04:08:56.657 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.648+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4147277387}) 2024-09-16T04:08:56.658 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.652+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.658 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.652+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1281241144}) 2024-09-16T04:08:56.658 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.652+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4090983944}) 2024-09-16T04:08:56.658 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.652+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.658 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.652+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1281241144}) 2024-09-16T04:08:56.658 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.652+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2013027556}) 2024-09-16T04:08:56.660 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.652+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.660 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.652+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=546892616}) 2024-09-16T04:08:56.660 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.652+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=920066723}) 2024-09-16T04:08:56.660 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.652+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.660 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.652+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=546892616}) 2024-09-16T04:08:56.660 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.652+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3267435713}) 2024-09-16T04:08:56.661 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.656+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.661 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.656+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4142464762}) 2024-09-16T04:08:56.661 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.656+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3765783889}) 2024-09-16T04:08:56.662 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.656+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.662 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.656+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4142464762}) 2024-09-16T04:08:56.662 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.656+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1876953412}) 2024-09-16T04:08:56.663 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.656+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.663 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.656+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2290265340}) 2024-09-16T04:08:56.663 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.656+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4113264865}) 2024-09-16T04:08:56.663 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.656+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.664 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.656+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2290265340}) 2024-09-16T04:08:56.664 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.656+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4255453842}) 2024-09-16T04:08:56.665 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.656+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.665 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.656+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4251842684}) 2024-09-16T04:08:56.665 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.656+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2870909964}) 2024-09-16T04:08:56.665 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.656+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.665 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.656+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4251842684}) 2024-09-16T04:08:56.665 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.656+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=538956238}) 2024-09-16T04:08:56.667 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.660+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.667 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.660+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=212684788}) 2024-09-16T04:08:56.667 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.660+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2308835990}) 2024-09-16T04:08:56.667 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.660+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.667 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.660+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=212684788}) 2024-09-16T04:08:56.667 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.660+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3492691303}) 2024-09-16T04:08:56.669 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.660+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.669 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.660+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=383324892}) 2024-09-16T04:08:56.669 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.660+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=131881680}) 2024-09-16T04:08:56.669 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.660+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.669 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.660+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=383324892}) 2024-09-16T04:08:56.669 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.660+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3023633094}) 2024-09-16T04:08:56.670 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.664+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.671 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.664+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1033573757}) 2024-09-16T04:08:56.671 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.664+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=862586451}) 2024-09-16T04:08:56.671 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.664+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.671 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.664+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1033573757}) 2024-09-16T04:08:56.671 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.664+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2173496805}) 2024-09-16T04:08:56.672 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.664+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.672 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.664+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3137619558}) 2024-09-16T04:08:56.672 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.664+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1580940326}) 2024-09-16T04:08:56.672 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.664+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.673 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.664+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3137619558}) 2024-09-16T04:08:56.673 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.664+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1783245854}) 2024-09-16T04:08:56.674 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.668+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.674 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.668+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2487397646}) 2024-09-16T04:08:56.674 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.668+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=377121776}) 2024-09-16T04:08:56.674 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.668+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.674 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.668+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2487397646}) 2024-09-16T04:08:56.674 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.668+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=943794530}) 2024-09-16T04:08:56.676 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.668+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.676 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.668+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1161793298}) 2024-09-16T04:08:56.676 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.668+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1593796703}) 2024-09-16T04:08:56.676 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.668+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.676 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.668+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1161793298}) 2024-09-16T04:08:56.676 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.668+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1002523458}) 2024-09-16T04:08:56.677 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.672+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.677 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.672+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4006286852}) 2024-09-16T04:08:56.678 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.672+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3299459715}) 2024-09-16T04:08:56.678 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.672+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.678 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.672+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4006286852}) 2024-09-16T04:08:56.678 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.672+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1694701140}) 2024-09-16T04:08:56.679 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.672+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.679 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.672+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=28350192}) 2024-09-16T04:08:56.680 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.672+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=567284577}) 2024-09-16T04:08:56.680 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.672+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.680 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.672+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=28350192}) 2024-09-16T04:08:56.680 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.672+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3529478093}) 2024-09-16T04:08:56.681 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.676+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.681 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.676+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=886848456}) 2024-09-16T04:08:56.681 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.676+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1460645601}) 2024-09-16T04:08:56.681 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.676+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.681 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.676+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=886848456}) 2024-09-16T04:08:56.681 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.676+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3677647623}) 2024-09-16T04:08:56.683 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.676+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.683 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.676+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3964424088}) 2024-09-16T04:08:56.683 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.676+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3425905222}) 2024-09-16T04:08:56.683 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.676+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.683 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.676+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3964424088}) 2024-09-16T04:08:56.683 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.676+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3549291523}) 2024-09-16T04:08:56.685 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.676+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.685 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.676+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2289190238}) 2024-09-16T04:08:56.685 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.676+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=318454218}) 2024-09-16T04:08:56.685 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.676+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.685 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.676+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2289190238}) 2024-09-16T04:08:56.685 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.676+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1546041534}) 2024-09-16T04:08:56.686 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.680+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.687 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.680+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=957463910}) 2024-09-16T04:08:56.687 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.680+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1734137918}) 2024-09-16T04:08:56.687 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.680+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.687 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.680+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=957463910}) 2024-09-16T04:08:56.687 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.680+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=239719019}) 2024-09-16T04:08:56.688 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.680+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.688 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.680+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=153710105}) 2024-09-16T04:08:56.689 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.680+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2902713189}) 2024-09-16T04:08:56.689 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.680+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.689 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.680+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=153710105}) 2024-09-16T04:08:56.689 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.680+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3314297444}) 2024-09-16T04:08:56.690 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.684+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.690 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.684+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3353825704}) 2024-09-16T04:08:56.690 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.684+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3959522985}) 2024-09-16T04:08:56.691 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.684+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.691 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.684+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3353825704}) 2024-09-16T04:08:56.691 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.684+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4288854254}) 2024-09-16T04:08:56.692 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.684+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.692 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.684+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1616513481}) 2024-09-16T04:08:56.692 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.684+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1914190}) 2024-09-16T04:08:56.692 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.684+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.692 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.684+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1616513481}) 2024-09-16T04:08:56.692 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.684+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=201762452}) 2024-09-16T04:08:56.694 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.688+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.694 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.688+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3874389405}) 2024-09-16T04:08:56.694 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.688+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1620951689}) 2024-09-16T04:08:56.694 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.688+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.694 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.688+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3874389405}) 2024-09-16T04:08:56.694 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.688+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3627698081}) 2024-09-16T04:08:56.696 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.688+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.696 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.688+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=296245152}) 2024-09-16T04:08:56.696 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.688+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=832914455}) 2024-09-16T04:08:56.696 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.688+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.696 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.688+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=296245152}) 2024-09-16T04:08:56.696 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.688+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2281981459}) 2024-09-16T04:08:56.698 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.692+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.698 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.692+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=411188149}) 2024-09-16T04:08:56.698 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.692+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=220150988}) 2024-09-16T04:08:56.698 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.692+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.698 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.692+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=411188149}) 2024-09-16T04:08:56.698 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.692+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4088715089}) 2024-09-16T04:08:56.699 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.692+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.700 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.692+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=832146710}) 2024-09-16T04:08:56.700 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.692+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=605740116}) 2024-09-16T04:08:56.700 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.692+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.700 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.692+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=832146710}) 2024-09-16T04:08:56.700 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.692+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4098894134}) 2024-09-16T04:08:56.701 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.696+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.701 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.696+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=615284368}) 2024-09-16T04:08:56.701 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.696+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3854477828}) 2024-09-16T04:08:56.702 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.696+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.702 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.696+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=615284368}) 2024-09-16T04:08:56.702 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.696+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=734434485}) 2024-09-16T04:08:56.703 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.696+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.703 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.696+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3583483550}) 2024-09-16T04:08:56.703 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.696+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1027035449}) 2024-09-16T04:08:56.704 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.696+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.704 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.696+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3583483550}) 2024-09-16T04:08:56.704 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.696+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3546819750}) 2024-09-16T04:08:56.705 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.696+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.705 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.696+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2621787565}) 2024-09-16T04:08:56.705 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.696+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=642976224}) 2024-09-16T04:08:56.705 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.700+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.705 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.700+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2621787565}) 2024-09-16T04:08:56.705 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.700+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3537062888}) 2024-09-16T04:08:56.707 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.700+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.707 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.700+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4060097738}) 2024-09-16T04:08:56.707 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.700+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1350104941}) 2024-09-16T04:08:56.707 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.700+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.707 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.700+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4060097738}) 2024-09-16T04:08:56.707 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.700+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=371934747}) 2024-09-16T04:08:56.709 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.700+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.709 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.700+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=714294342}) 2024-09-16T04:08:56.709 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.700+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=267558300}) 2024-09-16T04:08:56.709 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.700+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.709 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.700+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=714294342}) 2024-09-16T04:08:56.709 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.700+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3755146641}) 2024-09-16T04:08:56.710 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.704+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.711 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.704+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2702632127}) 2024-09-16T04:08:56.711 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.704+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2390816663}) 2024-09-16T04:08:56.711 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.704+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.711 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.704+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2702632127}) 2024-09-16T04:08:56.711 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.704+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3798620882}) 2024-09-16T04:08:56.712 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.704+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.712 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.704+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2908022771}) 2024-09-16T04:08:56.712 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.704+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1398145240}) 2024-09-16T04:08:56.712 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.704+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.712 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.704+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2908022771}) 2024-09-16T04:08:56.713 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.704+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=192888481}) 2024-09-16T04:08:56.714 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.708+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.714 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.708+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3091431173}) 2024-09-16T04:08:56.714 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.708+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2391015996}) 2024-09-16T04:08:56.714 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.708+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.714 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.708+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3091431173}) 2024-09-16T04:08:56.714 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.708+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2061854384}) 2024-09-16T04:08:56.716 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.708+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.716 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.708+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4292943055}) 2024-09-16T04:08:56.716 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.708+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=769825773}) 2024-09-16T04:08:56.716 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.708+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.716 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.708+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4292943055}) 2024-09-16T04:08:56.716 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.708+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=636676662}) 2024-09-16T04:08:56.719 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.712+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.719 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.712+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=188441875}) 2024-09-16T04:08:56.719 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.712+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3833474578}) 2024-09-16T04:08:56.719 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.712+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.719 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.712+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=188441875}) 2024-09-16T04:08:56.719 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.712+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=664620686}) 2024-09-16T04:08:56.720 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.712+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.720 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.712+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4193986621}) 2024-09-16T04:08:56.720 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.712+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2998526665}) 2024-09-16T04:08:56.720 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.712+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.720 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.712+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4193986621}) 2024-09-16T04:08:56.720 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.712+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1717799802}) 2024-09-16T04:08:56.722 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.716+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.722 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.716+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3855741842}) 2024-09-16T04:08:56.722 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.716+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2532030579}) 2024-09-16T04:08:56.723 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.716+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.723 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.716+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3855741842}) 2024-09-16T04:08:56.723 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.716+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1481204595}) 2024-09-16T04:08:56.724 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.716+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.724 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.716+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2608362452}) 2024-09-16T04:08:56.724 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.716+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=550149170}) 2024-09-16T04:08:56.724 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.716+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.724 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.716+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2608362452}) 2024-09-16T04:08:56.724 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.716+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3817556008}) 2024-09-16T04:08:56.725 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.716+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.725 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.716+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3491920278}) 2024-09-16T04:08:56.725 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.720+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3986674140}) 2024-09-16T04:08:56.725 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.720+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.725 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.720+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3491920278}) 2024-09-16T04:08:56.726 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.720+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2604406043}) 2024-09-16T04:08:56.727 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.720+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.727 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.720+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4177191597}) 2024-09-16T04:08:56.727 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.720+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2700444065}) 2024-09-16T04:08:56.727 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.720+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.727 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.720+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4177191597}) 2024-09-16T04:08:56.727 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.720+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=987372913}) 2024-09-16T04:08:56.729 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.720+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.729 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.720+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=85979359}) 2024-09-16T04:08:56.729 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.720+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=23600362}) 2024-09-16T04:08:56.729 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.720+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.729 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.720+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=85979359}) 2024-09-16T04:08:56.729 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.720+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3143775722}) 2024-09-16T04:08:56.730 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.724+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.731 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.724+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1601155436}) 2024-09-16T04:08:56.731 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.724+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2526914558}) 2024-09-16T04:08:56.731 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.724+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.731 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.724+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1601155436}) 2024-09-16T04:08:56.731 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.724+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1612238259}) 2024-09-16T04:08:56.732 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.724+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.732 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.724+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=132641722}) 2024-09-16T04:08:56.733 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.724+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=954032758}) 2024-09-16T04:08:56.733 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.724+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.733 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.724+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=132641722}) 2024-09-16T04:08:56.733 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.724+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=122194986}) 2024-09-16T04:08:56.734 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.728+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.734 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.728+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=287668324}) 2024-09-16T04:08:56.734 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.728+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=981063972}) 2024-09-16T04:08:56.735 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.728+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.735 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.728+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=287668324}) 2024-09-16T04:08:56.735 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.728+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4039265159}) 2024-09-16T04:08:56.736 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.728+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.736 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.728+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4048171363}) 2024-09-16T04:08:56.736 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.728+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3528867595}) 2024-09-16T04:08:56.737 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.728+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.737 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.728+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4048171363}) 2024-09-16T04:08:56.737 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.728+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2568277026}) 2024-09-16T04:08:56.738 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.732+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.738 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.732+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1012400133}) 2024-09-16T04:08:56.738 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.732+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1235613039}) 2024-09-16T04:08:56.738 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.732+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.739 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.732+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1012400133}) 2024-09-16T04:08:56.739 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.732+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3093770561}) 2024-09-16T04:08:56.740 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.732+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.740 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.732+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1274011366}) 2024-09-16T04:08:56.740 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.732+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3937602577}) 2024-09-16T04:08:56.740 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.732+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.740 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.732+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1274011366}) 2024-09-16T04:08:56.741 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.732+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1762594263}) 2024-09-16T04:08:56.742 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.736+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.742 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.736+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1738560648}) 2024-09-16T04:08:56.742 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.736+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=359087861}) 2024-09-16T04:08:56.742 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.736+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.742 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.736+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1738560648}) 2024-09-16T04:08:56.742 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.736+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1168839718}) 2024-09-16T04:08:56.744 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.736+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.744 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.736+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=219825393}) 2024-09-16T04:08:56.744 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.736+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=119297570}) 2024-09-16T04:08:56.744 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.736+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.744 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.736+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=219825393}) 2024-09-16T04:08:56.744 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.736+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3132050812}) 2024-09-16T04:08:56.746 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.740+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.746 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.740+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3419498465}) 2024-09-16T04:08:56.746 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.740+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1736222605}) 2024-09-16T04:08:56.746 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.740+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.746 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.740+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3419498465}) 2024-09-16T04:08:56.746 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.740+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1133989796}) 2024-09-16T04:08:56.747 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.740+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.747 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.740+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2577345707}) 2024-09-16T04:08:56.747 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.740+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1277252683}) 2024-09-16T04:08:56.747 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.740+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.748 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.740+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2577345707}) 2024-09-16T04:08:56.748 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.740+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2804715757}) 2024-09-16T04:08:56.749 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.740+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.749 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.740+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=97335000}) 2024-09-16T04:08:56.750 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.740+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3350452302}) 2024-09-16T04:08:56.750 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.740+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.750 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.740+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=97335000}) 2024-09-16T04:08:56.750 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.740+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1010493173}) 2024-09-16T04:08:56.753 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.744+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.753 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.744+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3478006107}) 2024-09-16T04:08:56.753 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.744+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3171162869}) 2024-09-16T04:08:56.753 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.744+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.753 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.744+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3478006107}) 2024-09-16T04:08:56.754 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.744+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3286480166}) 2024-09-16T04:08:56.754 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.744+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.754 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.744+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3046253599}) 2024-09-16T04:08:56.754 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.744+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=61257779}) 2024-09-16T04:08:56.754 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.744+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.754 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.744+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3046253599}) 2024-09-16T04:08:56.754 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.744+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3818981192}) 2024-09-16T04:08:56.755 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.748+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.755 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.748+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3030378618}) 2024-09-16T04:08:56.755 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.748+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1400751242}) 2024-09-16T04:08:56.755 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.748+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.755 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.748+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3030378618}) 2024-09-16T04:08:56.755 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.748+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=727689484}) 2024-09-16T04:08:56.756 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.748+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.756 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.748+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2623374938}) 2024-09-16T04:08:56.756 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.748+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2883827361}) 2024-09-16T04:08:56.756 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.748+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.756 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.748+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2623374938}) 2024-09-16T04:08:56.756 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.748+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=766575487}) 2024-09-16T04:08:56.757 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.752+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.758 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.752+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3917019345}) 2024-09-16T04:08:56.758 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.752+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1292134242}) 2024-09-16T04:08:56.758 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.752+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.758 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.752+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3917019345}) 2024-09-16T04:08:56.758 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.752+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4274909197}) 2024-09-16T04:08:56.759 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.752+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.759 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.752+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3226240381}) 2024-09-16T04:08:56.759 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.752+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3489324350}) 2024-09-16T04:08:56.760 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.752+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.760 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.752+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3226240381}) 2024-09-16T04:08:56.760 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.752+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2308437861}) 2024-09-16T04:08:56.761 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.752+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.761 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.752+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3998074761}) 2024-09-16T04:08:56.761 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.752+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1669601860}) 2024-09-16T04:08:56.761 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.756+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.761 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.756+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3998074761}) 2024-09-16T04:08:56.761 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.756+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3168656923}) 2024-09-16T04:08:56.763 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.756+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.764 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.756+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2757545475}) 2024-09-16T04:08:56.764 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.756+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4118106122}) 2024-09-16T04:08:56.764 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.756+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.764 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.756+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2757545475}) 2024-09-16T04:08:56.764 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.756+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2253909525}) 2024-09-16T04:08:56.765 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.756+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.765 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.756+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1050105435}) 2024-09-16T04:08:56.765 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.756+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3863159048}) 2024-09-16T04:08:56.765 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.756+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.765 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.756+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1050105435}) 2024-09-16T04:08:56.765 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.756+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2717924615}) 2024-09-16T04:08:56.766 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.760+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.767 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.760+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=211826416}) 2024-09-16T04:08:56.767 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.760+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2859053025}) 2024-09-16T04:08:56.767 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.760+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.767 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.760+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=211826416}) 2024-09-16T04:08:56.767 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.760+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2488101085}) 2024-09-16T04:08:56.768 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.760+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.768 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.760+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2940421061}) 2024-09-16T04:08:56.768 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.760+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1016842571}) 2024-09-16T04:08:56.768 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.760+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.769 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.760+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2940421061}) 2024-09-16T04:08:56.769 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.760+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3230096456}) 2024-09-16T04:08:56.770 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.764+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.770 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.764+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=664315773}) 2024-09-16T04:08:56.770 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.764+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4066317993}) 2024-09-16T04:08:56.770 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.764+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.770 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.764+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=664315773}) 2024-09-16T04:08:56.770 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.764+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3297409386}) 2024-09-16T04:08:56.772 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.764+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.772 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.764+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=75469176}) 2024-09-16T04:08:56.772 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.764+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2729405113}) 2024-09-16T04:08:56.772 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.764+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.772 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.764+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=75469176}) 2024-09-16T04:08:56.772 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.764+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3834965204}) 2024-09-16T04:08:56.774 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.768+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.774 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.768+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3134109215}) 2024-09-16T04:08:56.774 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.768+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=563423226}) 2024-09-16T04:08:56.774 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.768+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.774 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.768+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3134109215}) 2024-09-16T04:08:56.774 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.768+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4184582687}) 2024-09-16T04:08:56.776 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.768+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.776 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.768+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1988078333}) 2024-09-16T04:08:56.776 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.768+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4225352803}) 2024-09-16T04:08:56.776 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.768+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.776 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.768+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1988078333}) 2024-09-16T04:08:56.776 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.768+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1648270631}) 2024-09-16T04:08:56.778 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.772+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.778 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.772+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1427703494}) 2024-09-16T04:08:56.778 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.772+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2227039611}) 2024-09-16T04:08:56.778 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.772+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.778 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.772+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1427703494}) 2024-09-16T04:08:56.778 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.772+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1816579266}) 2024-09-16T04:08:56.780 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.772+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.780 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.772+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=149503530}) 2024-09-16T04:08:56.780 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.772+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2917315425}) 2024-09-16T04:08:56.780 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.772+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.780 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.772+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=149503530}) 2024-09-16T04:08:56.780 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.772+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2072293625}) 2024-09-16T04:08:56.781 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.776+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.782 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.776+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=228991682}) 2024-09-16T04:08:56.782 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.776+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=880749186}) 2024-09-16T04:08:56.782 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.776+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.782 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.776+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=228991682}) 2024-09-16T04:08:56.782 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.776+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2205256023}) 2024-09-16T04:08:56.783 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.776+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.783 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.776+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1199202418}) 2024-09-16T04:08:56.784 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.776+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=775774620}) 2024-09-16T04:08:56.784 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.776+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.784 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.776+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1199202418}) 2024-09-16T04:08:56.784 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.776+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2023686304}) 2024-09-16T04:08:56.785 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.780+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.785 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.780+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2809810674}) 2024-09-16T04:08:56.785 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.780+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2411757001}) 2024-09-16T04:08:56.785 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.780+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.785 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.780+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2809810674}) 2024-09-16T04:08:56.786 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.780+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2295673274}) 2024-09-16T04:08:56.787 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.780+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.787 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.780+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1932281322}) 2024-09-16T04:08:56.787 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.780+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=447060900}) 2024-09-16T04:08:56.787 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.780+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.788 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.780+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1932281322}) 2024-09-16T04:08:56.788 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.780+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=254966240}) 2024-09-16T04:08:56.789 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.780+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.789 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.780+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3486929849}) 2024-09-16T04:08:56.789 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.780+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=691520926}) 2024-09-16T04:08:56.789 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.784+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.789 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.784+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3486929849}) 2024-09-16T04:08:56.789 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.784+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2885925688}) 2024-09-16T04:08:56.791 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.784+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.791 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.784+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3373241676}) 2024-09-16T04:08:56.791 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.784+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=786870487}) 2024-09-16T04:08:56.791 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.784+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.791 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.784+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3373241676}) 2024-09-16T04:08:56.791 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.784+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3723920625}) 2024-09-16T04:08:56.793 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.784+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.793 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.784+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1041239475}) 2024-09-16T04:08:56.793 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.784+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2949000782}) 2024-09-16T04:08:56.793 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.784+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.793 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.784+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1041239475}) 2024-09-16T04:08:56.793 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.784+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1025864054}) 2024-09-16T04:08:56.795 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.788+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.795 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.788+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=67,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=7} crc {logm=473330256,mdsmap=4088524775,mgr=808849511,mgr_command_descs=2107673600,mgr_metadata=1367272202,mgrstat=2781514683,mon_config_key=3867845802}) 2024-09-16T04:08:56.796 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.788+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=67,mdsmap=3,mgr=6,mgr_command_descs=1,mgrstat=12,mon_config_key=11} crc {logm=212072176,mdsmap=4088524775,mgr=4251399302,mgr_command_descs=2107673600,mgrstat=3613656018,mon_config_key=85057112}) 2024-09-16T04:08:56.796 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.788+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.796 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.788+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=67,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=7} crc {logm=473330256,mdsmap=4088524775,mgr=808849511,mgr_command_descs=2107673600,mgr_metadata=1367272202,mgrstat=2781514683,mon_config_key=3867845802}) 2024-09-16T04:08:56.796 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.788+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=64,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=10} crc {logm=610405802,mdsmap=4088524775,mgr=460109776,mgr_command_descs=2107673600,mgr_metadata=1367272202,mgrstat=2781514683,mon_config_key=1822420877}) 2024-09-16T04:08:56.797 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.792+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.798 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.792+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mon_config_key=4,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=40,osdmap=46} crc {mon_config_key=2402045709,monmap=3937513981,osd_metadata=4182959540,osd_pg_creating=783066558,osd_snap=4041384033,osdmap=2975355754}) 2024-09-16T04:08:56.798 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.792+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mon_config_key=4,monmap=3,osd_metadata=6,osd_snap=41,osdmap=46} crc {mon_config_key=2402045709,monmap=3937513981,osd_metadata=1273868022,osd_snap=451814303,osdmap=1567815849}) 2024-09-16T04:08:56.798 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.792+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.798 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.792+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mon_config_key=4,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=40,osdmap=46} crc {mon_config_key=2402045709,monmap=3937513981,osd_metadata=4182959540,osd_pg_creating=783066558,osd_snap=4041384033,osdmap=2975355754}) 2024-09-16T04:08:56.798 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.792+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mon_config_key=4,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=42,osdmap=45} crc {mon_config_key=571888367,monmap=3937513981,osd_metadata=498355664,osd_pg_creating=783066558,osd_snap=2323932723,osdmap=197663949}) 2024-09-16T04:08:56.800 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.792+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.800 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.792+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1363265412}) 2024-09-16T04:08:56.800 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.792+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=4052065056}) 2024-09-16T04:08:56.800 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.792+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.801 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.792+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1363265412}) 2024-09-16T04:08:56.801 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.792+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2637096320}) 2024-09-16T04:08:56.802 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.796+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.802 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.796+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1313800657}) 2024-09-16T04:08:56.802 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.796+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1432624082}) 2024-09-16T04:08:56.802 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.796+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.802 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.796+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1313800657}) 2024-09-16T04:08:56.802 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.796+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3016494905}) 2024-09-16T04:08:56.806 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.800+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.806 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.800+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=591108803}) 2024-09-16T04:08:56.806 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.800+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1292093632}) 2024-09-16T04:08:56.806 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.800+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.806 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.800+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=591108803}) 2024-09-16T04:08:56.806 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.800+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2670901791}) 2024-09-16T04:08:56.821 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.816+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.821 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.816+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=37} crc {osdmap=3914888043}) 2024-09-16T04:08:56.822 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.816+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=38} crc {osdmap=2459963851}) 2024-09-16T04:08:56.822 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.816+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:08:56.822 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.816+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=37} crc {osdmap=3914888043}) 2024-09-16T04:08:56.822 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:08:56.816+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=38} crc {osdmap=1829981110}) 2024-09-16T04:08:57.575 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T04:08:57.575 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:08:57.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.700+0000 7fedc58cd640 10 monclient: get_monmap_and_config 2024-09-16T04:08:57.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.700+0000 7fedc58cd640 10 monclient: build_initial_monmap 2024-09-16T04:08:57.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.700+0000 7fedc58cd640 10 monclient: monmap: 2024-09-16T04:08:57.709 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:08:57.709 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:08:57.709 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:08:57.707544+0000 2024-09-16T04:08:57.709 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:08:57.707544+0000 2024-09-16T04:08:57.709 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:57.709 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:08:57.709 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:08:57.709 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:08:57.710 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:08:57.710 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:08:57.710 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:08:57.710 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:08:57.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.704+0000 7fedc58cd640 1 Processor -- start 2024-09-16T04:08:57.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.704+0000 7fedc58cd640 1 -- start start 2024-09-16T04:08:57.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.704+0000 7fedc58cd640 10 monclient: init 2024-09-16T04:08:57.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.704+0000 7fedc58cd640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:57.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.704+0000 7fedc58cd640 10 monclient: _add_conns ranks=[0,4,3,1,2] 2024-09-16T04:08:57.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.704+0000 7fedc58cd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fedc00a1330 addr v1:172.21.15.42:6789/0 2024-09-16T04:08:57.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.704+0000 7fedc58cd640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fedc006dac0 0x7fedc013a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:08:57.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.704+0000 7fedc58cd640 10 monclient(hunting): picked mon.noname-c con 0x7fedc006dac0 addr v2:172.21.15.154:3300/0 2024-09-16T04:08:57.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.704+0000 7fedc58cd640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fedc00a1e20 0x7fedc013c890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:08:57.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.704+0000 7fedc58cd640 10 monclient(hunting): picked mon.noname-a con 0x7fedc00a1e20 addr v2:172.21.15.42:3300/0 2024-09-16T04:08:57.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.704+0000 7fedc58cd640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fedc009c090 con 0x7fedc00a1330 2024-09-16T04:08:57.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.704+0000 7fedc58cd640 10 monclient(hunting): start opening mon connection 2024-09-16T04:08:57.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.704+0000 7fedc58cd640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7fedc009f610 con 0x7fedc00a1e20 2024-09-16T04:08:57.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.704+0000 7fedc58cd640 10 monclient(hunting): start opening mon connection 2024-09-16T04:08:57.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.704+0000 7fedc58cd640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fedc009f8f0 con 0x7fedc006dac0 2024-09-16T04:08:57.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.704+0000 7fedc58cd640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:57.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.704+0000 7fedc58cd640 10 monclient(hunting): authenticate will time out at 2584.676952s 2024-09-16T04:08:57.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.704+0000 7fedbeffd640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fedc00a1330 0x7fedc00a1710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:34816/0 (socket says 172.21.15.42:34816) 2024-09-16T04:08:57.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.704+0000 7fedbeffd640 1 -- 172.21.15.42:0/1678072817 learned_addr learned my addr 172.21.15.42:0/1678072817 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T04:08:57.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.704+0000 7fedbf7fe640 1 --2- 172.21.15.42:0/1678072817 >> v2:172.21.15.42:3300/0 conn(0x7fedc00a1e20 0x7fedc013c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:08:57.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.704+0000 7fedbf7fe640 10 monclient(hunting): get_auth_request con 0x7fedc00a1e20 auth_method 0 2024-09-16T04:08:57.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.704+0000 7fedbf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:08:57.714 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.704+0000 7fedbf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:57.714 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.704+0000 7fedbf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:57.714 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.704+0000 7fedb7fff640 1 --2- 172.21.15.42:0/1678072817 >> v2:172.21.15.154:3300/0 conn(0x7fedc006dac0 0x7fedc013a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:08:57.714 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.704+0000 7fedb7fff640 10 monclient(hunting): get_auth_request con 0x7fedc006dac0 auth_method 0 2024-09-16T04:08:57.714 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.704+0000 7fedb7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:08:57.714 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.704+0000 7fedb7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:57.714 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.704+0000 7fedb7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:57.714 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.704+0000 7fedbf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:08:57.714 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.704+0000 7fedbf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:08:57.714 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.704+0000 7fedbf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:08:57.714 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.704+0000 7fedbe7fc640 1 -- 172.21.15.42:0/1678072817 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2483274221 0 0) 0x7fedc009c090 con 0x7fedc00a1330 2024-09-16T04:08:57.714 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.704+0000 7fedbe7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:08:57.714 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.704+0000 7fedbe7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:57.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.704+0000 7fedbe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:57.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.704+0000 7fedbe7fc640 10 monclient(hunting): my global_id is 1034133 2024-09-16T04:08:57.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.704+0000 7fedbe7fc640 1 -- 172.21.15.42:0/1678072817 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fedac003530 con 0x7fedc00a1330 2024-09-16T04:08:57.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.704+0000 7fedb7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:08:57.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.704+0000 7fedb7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:08:57.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.704+0000 7fedb7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:08:57.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.704+0000 7fedbf7fe640 10 monclient(hunting): handle_auth_done global_id 1034136 payload 274 2024-09-16T04:08:57.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.704+0000 7fedbf7fe640 1 -- 172.21.15.42:0/1678072817 >> v2:172.21.15.154:3300/0 conn(0x7fedc006dac0 msgr2=0x7fedc013a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:08:57.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.704+0000 7fedbf7fe640 1 --2- 172.21.15.42:0/1678072817 >> v2:172.21.15.154:3300/0 conn(0x7fedc006dac0 0x7fedc013a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:08:57.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.704+0000 7fedbf7fe640 1 -- 172.21.15.42:0/1678072817 >> v1:172.21.15.42:6789/0 conn(0x7fedc00a1330 legacy=0x7fedc00a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:57.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.704+0000 7fedbf7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:08:57.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.704+0000 7fedbf7fe640 1 monclient: found mon.noname-a 2024-09-16T04:08:57.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.704+0000 7fedbf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:57.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.704+0000 7fedbf7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:08:57.716 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.704+0000 7fedbf7fe640 1 -- 172.21.15.42:0/1678072817 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fedc009fbd0 con 0x7fedc00a1e20 2024-09-16T04:08:57.725 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.704+0000 7fedbf7fe640 1 --2- 172.21.15.42:0/1678072817 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fedc00a1e20 0x7fedc013c890 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fedc006da10 tx=0x7fedb801c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a78a381928abbb45 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:08:57.725 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.708+0000 7fedbe7fc640 1 -- 172.21.15.42:0/1678072817 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedb80b2070 con 0x7fedc00a1e20 2024-09-16T04:08:57.725 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.708+0000 7fedbe7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:57.726 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.708+0000 7fedbe7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:08:57.726 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.708+0000 7fedbe7fc640 10 monclient: dump: 2024-09-16T04:08:57.726 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:08:57.726 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:08:57.726 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:08:57.726 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:08:57.726 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:57.726 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:08:57.726 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:08:57.726 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:08:57.726 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:08:57.726 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:08:57.726 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.708+0000 7fedbe7fc640 10 monclient: _finish_auth 0 2024-09-16T04:08:57.727 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.708+0000 7fedbe7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:08:57.727 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.708+0000 7fedbe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:57.727 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.708+0000 7fedbe7fc640 1 -- 172.21.15.42:0/1678072817 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fedb80a57d0 con 0x7fedc00a1e20 2024-09-16T04:08:57.727 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.708+0000 7fedbe7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:57.727 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.708+0000 7fedbe7fc640 1 -- 172.21.15.42:0/1678072817 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedb80a5ba0 con 0x7fedc00a1e20 2024-09-16T04:08:57.727 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.708+0000 7fedc58cd640 5 monclient: authenticate success, global_id 1034136 2024-09-16T04:08:57.727 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.708+0000 7fedc58cd640 10 monclient: get_monmap_and_config success 2024-09-16T04:08:57.727 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.708+0000 7fedc58cd640 4 set_mon_vals no callback set 2024-09-16T04:08:57.727 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.708+0000 7fedbe7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:57.727 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.708+0000 7fedbe7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:08:57.727 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.708+0000 7fedbe7fc640 10 monclient: dump: 2024-09-16T04:08:57.727 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:08:57.727 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:08:57.728 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:08:57.728 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:08:57.728 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:57.728 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:08:57.728 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:08:57.728 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:08:57.728 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:08:57.728 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:08:57.728 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.708+0000 7fedc58cd640 10 monclient: shutdown 2024-09-16T04:08:57.728 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.708+0000 7fedc58cd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:57.729 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.708+0000 7fedc58cd640 1 -- 172.21.15.42:0/1678072817 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fedc00a1e20 msgr2=0x7fedc013c890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:57.729 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.708+0000 7fedc58cd640 1 --2- 172.21.15.42:0/1678072817 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fedc00a1e20 0x7fedc013c890 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fedc006da10 tx=0x7fedb801c910 comp rx=0 tx=0).stop 2024-09-16T04:08:57.729 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.708+0000 7fedc58cd640 1 -- 172.21.15.42:0/1678072817 shutdown_connections 2024-09-16T04:08:57.729 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.708+0000 7fedc58cd640 1 --2- 172.21.15.42:0/1678072817 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fedc00a1e20 0x7fedc013c890 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:08:57.729 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.708+0000 7fedc58cd640 1 --2- 172.21.15.42:0/1678072817 >> v2:172.21.15.154:3300/0 conn(0x7fedc006dac0 0x7fedc013a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:08:57.729 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.708+0000 7fedc58cd640 1 -- 172.21.15.42:0/1678072817 >> 172.21.15.42:0/1678072817 conn(0x7fedc00a2420 msgr2=0x7fedc00a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:57.729 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.708+0000 7fedc58cd640 1 -- 172.21.15.42:0/1678072817 shutdown_connections 2024-09-16T04:08:57.729 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.708+0000 7fedc58cd640 1 -- 172.21.15.42:0/1678072817 wait complete. 2024-09-16T04:08:57.729 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.708+0000 7fedc58cd640 10 monclient: build_initial_monmap 2024-09-16T04:08:57.729 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.708+0000 7fedc58cd640 10 monclient: monmap: 2024-09-16T04:08:57.729 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:08:57.730 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:08:57.730 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:08:57.730 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:08:57.730 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:57.730 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:08:57.730 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:08:57.730 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:08:57.730 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:08:57.730 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:08:57.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.708+0000 7fedc58cd640 1 librados: starting msgr at 2024-09-16T04:08:57.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.708+0000 7fedc58cd640 1 librados: starting objecter 2024-09-16T04:08:57.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.708+0000 7fedc58cd640 1 Processor -- start 2024-09-16T04:08:57.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.708+0000 7fedc58cd640 1 -- start start 2024-09-16T04:08:57.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.708+0000 7fedc58cd640 1 librados: setting wanted keys 2024-09-16T04:08:57.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.708+0000 7fedc58cd640 1 librados: calling monclient init 2024-09-16T04:08:57.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.708+0000 7fedc58cd640 10 monclient: init 2024-09-16T04:08:57.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.708+0000 7fedc58cd640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:57.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.708+0000 7fedc58cd640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:08:57.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.708+0000 7fedc58cd640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fedc00a1330 0x7fedc01ceca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:08:57.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.708+0000 7fedc58cd640 10 monclient(hunting): picked mon.noname-b con 0x7fedc00a1330 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:08:57.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.708+0000 7fedc58cd640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fedc00a1e20 0x7fedc01352f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:08:57.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.708+0000 7fedc58cd640 10 monclient(hunting): picked mon.noname-c con 0x7fedc00a1e20 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:08:57.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.708+0000 7fedc58cd640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fedc0135830 0x7fedc009cb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:08:57.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.708+0000 7fedc58cd640 10 monclient(hunting): picked mon.noname-a con 0x7fedc0135830 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:08:57.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.708+0000 7fedc58cd640 10 monclient(hunting): start opening mon connection 2024-09-16T04:08:57.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.708+0000 7fedc58cd640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fedc013f800 con 0x7fedc0135830 2024-09-16T04:08:57.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.708+0000 7fedc58cd640 10 monclient(hunting): start opening mon connection 2024-09-16T04:08:57.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.708+0000 7fedc58cd640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fedc009ef80 con 0x7fedc00a1330 2024-09-16T04:08:57.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.708+0000 7fedc58cd640 10 monclient(hunting): start opening mon connection 2024-09-16T04:08:57.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.708+0000 7fedc58cd640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fedc013f980 con 0x7fedc00a1e20 2024-09-16T04:08:57.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.708+0000 7fedc58cd640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:57.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.708+0000 7fedc58cd640 10 monclient(hunting): authenticate will time out at 2584.682285s 2024-09-16T04:08:57.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.708+0000 7fedbf7fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fedc0135830 0x7fedc009cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:08:57.733 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.708+0000 7fedbf7fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fedc0135830 0x7fedc009cb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:50056/0 (socket says 172.21.15.42:50056) 2024-09-16T04:08:57.733 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.708+0000 7fedbf7fe640 1 -- 172.21.15.42:0/2243693895 learned_addr learned my addr 172.21.15.42:0/2243693895 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:08:57.733 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.708+0000 7fedbf7fe640 10 monclient(hunting): get_auth_request con 0x7fedc0135830 auth_method 0 2024-09-16T04:08:57.733 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.708+0000 7fedbf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:08:57.733 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.708+0000 7fedbf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:57.733 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.708+0000 7fedbf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:57.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.708+0000 7fedb7fff640 1 --2- 172.21.15.42:0/2243693895 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fedc00a1e20 0x7fedc01352f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:08:57.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.708+0000 7fedb7fff640 10 monclient(hunting): get_auth_request con 0x7fedc00a1e20 auth_method 0 2024-09-16T04:08:57.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.708+0000 7fedb7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:08:57.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.708+0000 7fedb7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:57.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.708+0000 7fedb7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:57.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.708+0000 7fedbf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:08:57.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.708+0000 7fedbf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:08:57.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.708+0000 7fedbf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:08:57.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.708+0000 7fedb7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:08:57.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.708+0000 7fedbeffd640 1 --2- 172.21.15.42:0/2243693895 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fedc00a1330 0x7fedc01ceca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:08:57.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.708+0000 7fedb7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:08:57.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.708+0000 7fedb7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:08:57.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.708+0000 7fedbeffd640 10 monclient(hunting): get_auth_request con 0x7fedc00a1330 auth_method 0 2024-09-16T04:08:57.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.708+0000 7fedbeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:08:57.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.708+0000 7fedbeffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:57.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.708+0000 7fedbeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:57.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.708+0000 7fedbf7fe640 10 monclient(hunting): handle_auth_done global_id 1034139 payload 824 2024-09-16T04:08:57.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.708+0000 7fedbf7fe640 1 -- 172.21.15.42:0/2243693895 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fedc00a1e20 msgr2=0x7fedc01352f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:57.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.708+0000 7fedbf7fe640 1 --2- 172.21.15.42:0/2243693895 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fedc00a1e20 0x7fedc01352f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:08:57.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.708+0000 7fedbf7fe640 1 -- 172.21.15.42:0/2243693895 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fedc00a1330 msgr2=0x7fedc01ceca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:57.736 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.708+0000 7fedbf7fe640 1 --2- 172.21.15.42:0/2243693895 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fedc00a1330 0x7fedc01ceca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:08:57.736 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.708+0000 7fedbf7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:08:57.736 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.708+0000 7fedbf7fe640 1 monclient: found mon.noname-a 2024-09-16T04:08:57.736 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.708+0000 7fedbf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:57.736 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.708+0000 7fedbf7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:08:57.736 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.708+0000 7fedbf7fe640 1 -- 172.21.15.42:0/2243693895 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fedc01219c0 con 0x7fedc0135830 2024-09-16T04:08:57.736 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.708+0000 7fedbeffd640 1 --2- 172.21.15.42:0/2243693895 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fedc00a1330 0x7fedc01ceca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T04:08:57.736 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.708+0000 7fedb7fff640 1 --2- 172.21.15.42:0/2243693895 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fedc00a1e20 0x7fedc01352f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:08:57.736 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.708+0000 7fedbf7fe640 1 --2- 172.21.15.42:0/2243693895 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fedc0135830 0x7fedc009cb60 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fedb8019520 tx=0x7fedb80196b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:08:57.736 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.708+0000 7fedbcff9640 10 client.?.objecter ms_handle_connect 0x7fedc0135830 2024-09-16T04:08:57.736 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.708+0000 7fedbcff9640 10 client.?.objecter resend_mon_ops 2024-09-16T04:08:57.736 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.708+0000 7fedbcff9640 1 -- 172.21.15.42:0/2243693895 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedb80b2050 con 0x7fedc0135830 2024-09-16T04:08:57.736 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.708+0000 7fedbcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:57.737 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.708+0000 7fedbcff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:08:57.737 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.708+0000 7fedbcff9640 10 monclient: dump: 2024-09-16T04:08:57.737 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:08:57.737 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:08:57.737 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:08:57.737 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:08:57.737 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:57.737 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:08:57.737 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:08:57.737 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:08:57.738 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:08:57.738 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:08:57.738 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.708+0000 7fedbcff9640 10 monclient: _finish_auth 0 2024-09-16T04:08:57.738 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.708+0000 7fedbcff9640 10 monclient: _check_auth_tickets 2024-09-16T04:08:57.738 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.708+0000 7fedbcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:57.738 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.708+0000 7fedbcff9640 1 -- 172.21.15.42:0/2243693895 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fedb80be070 con 0x7fedc0135830 2024-09-16T04:08:57.738 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.708+0000 7fedc58cd640 5 monclient: authenticate success, global_id 1034139 2024-09-16T04:08:57.738 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.708+0000 7fedc58cd640 10 monclient: _renew_subs 2024-09-16T04:08:57.738 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.708+0000 7fedc58cd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:08:57.738 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.708+0000 7fedc58cd640 1 -- 172.21.15.42:0/2243693895 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fedc006e260 con 0x7fedc0135830 2024-09-16T04:08:57.738 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.708+0000 7fedbcff9640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:57.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.708+0000 7fedbcff9640 1 -- 172.21.15.42:0/2243693895 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedb8009450 con 0x7fedc0135830 2024-09-16T04:08:57.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.708+0000 7fedbcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:57.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.708+0000 7fedbcff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:08:57.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.708+0000 7fedbcff9640 10 monclient: dump: 2024-09-16T04:08:57.739 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:08:57.739 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:08:57.739 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:08:57.739 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:08:57.739 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:57.739 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:08:57.739 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:08:57.739 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:08:57.739 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:08:57.740 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:08:57.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.708+0000 7fedbdffb640 4 set_mon_vals no callback set 2024-09-16T04:08:57.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.708+0000 7fedc58cd640 10 client.1034139.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:08:57.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.708+0000 7fedc58cd640 10 monclient: _renew_subs 2024-09-16T04:08:57.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.708+0000 7fedc58cd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:08:57.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.708+0000 7fedc58cd640 1 -- 172.21.15.42:0/2243693895 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fedc01216e0 con 0x7fedc0135830 2024-09-16T04:08:57.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.708+0000 7fedc58cd640 1 librados: init done 2024-09-16T04:08:57.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.712+0000 7fedbcff9640 1 -- 172.21.15.42:0/2243693895 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fedb80ad890 con 0x7fedc0135830 2024-09-16T04:08:57.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.712+0000 7fedbcff9640 1 --2- 172.21.15.42:0/2243693895 >> v2:172.21.15.154:6812/254845342 conn(0x7feda0037810 0x7feda0039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:08:57.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.712+0000 7fedc58cd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:57.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.712+0000 7fedc58cd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:57.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.712+0000 7fedc58cd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:08:57.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.712+0000 7fedbcff9640 1 -- 172.21.15.42:0/2243693895 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(706..706 src has 451..706) ==== 5040+0+0 (secure 0 0 0) 0x7fedb80e3cb0 con 0x7fedc0135830 2024-09-16T04:08:57.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.712+0000 7fedc58cd640 1 -- 172.21.15.42:0/2243693895 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fedc00a1710 con 0x7fedc0135830 2024-09-16T04:08:57.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.712+0000 7fedbcff9640 10 client.1034139.objecter ms_dispatch 0x7fedc0000ba0 osd_map(706..706 src has 451..706) 2024-09-16T04:08:57.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.712+0000 7fedbcff9640 3 client.1034139.objecter handle_osd_map got epochs [706,706] > 0 2024-09-16T04:08:57.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.712+0000 7fedbcff9640 3 client.1034139.objecter handle_osd_map decoding full epoch 706 2024-09-16T04:08:57.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.712+0000 7fedbcff9640 20 client.1034139.objecter dump_active .. 0 homeless 2024-09-16T04:08:57.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.712+0000 7fedbeffd640 1 --2- 172.21.15.42:0/2243693895 >> v2:172.21.15.154:6812/254845342 conn(0x7feda0037810 0x7feda0039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:08:57.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.712+0000 7fedbeffd640 10 monclient: get_auth_request con 0x7feda0037810 auth_method 0 2024-09-16T04:08:57.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.712+0000 7fedbcff9640 10 client.1034139.objecter ms_handle_connect 0x7feda0037810 2024-09-16T04:08:57.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.712+0000 7fedbeffd640 1 --2- 172.21.15.42:0/2243693895 >> v2:172.21.15.154:6812/254845342 conn(0x7feda0037810 0x7feda0039cd0 secure :-1 s=READY pgs=3906 cs=0 l=1 rev1=1 crypto rx=0x7fedb0005890 tx=0x7fedb0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:08:57.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.716+0000 7fedbcff9640 1 -- 172.21.15.42:0/2243693895 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7fedb8016500 con 0x7fedc0135830 2024-09-16T04:08:57.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.716+0000 7fedbcff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:57.742 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.716+0000 7fedbcff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:08:57.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.944+0000 7fedc58cd640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:08:57.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.944+0000 7fedc58cd640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:08:57.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.944+0000 7fedc58cd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:08:57.952 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.944+0000 7fedc58cd640 1 -- 172.21.15.42:0/2243693895 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fedc009e790 con 0x7fedc0135830 2024-09-16T04:08:57.952 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.944+0000 7fedbcff9640 1 -- 172.21.15.42:0/2243693895 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7fedb80adb80 con 0x7fedc0135830 2024-09-16T04:08:57.952 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.944+0000 7fedbcff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:08:57.952 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.944+0000 7fedbcff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:08:57.952 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":248,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:08:57.955 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.948+0000 7fedc58cd640 10 librados: watch_flush enter 2024-09-16T04:08:57.955 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.948+0000 7fedc58cd640 10 librados: watch_flush exit 2024-09-16T04:08:57.955 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.948+0000 7fedc58cd640 20 client.1034139.objecter shutdown clearing up homeless session... 2024-09-16T04:08:57.955 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.948+0000 7fedc58cd640 10 client.1034139.objecter successfully canceled tick 2024-09-16T04:08:57.955 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.948+0000 7fedc58cd640 1 -- 172.21.15.42:0/2243693895 >> v2:172.21.15.154:6812/254845342 conn(0x7feda0037810 msgr2=0x7feda0039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:57.955 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.948+0000 7fedc58cd640 1 --2- 172.21.15.42:0/2243693895 >> v2:172.21.15.154:6812/254845342 conn(0x7feda0037810 0x7feda0039cd0 secure :-1 s=READY pgs=3906 cs=0 l=1 rev1=1 crypto rx=0x7fedb0005890 tx=0x7fedb0009040 comp rx=0 tx=0).stop 2024-09-16T04:08:57.955 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.948+0000 7fedc58cd640 10 monclient: shutdown 2024-09-16T04:08:57.955 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.948+0000 7fedc58cd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:57.955 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.948+0000 7fedc58cd640 1 -- 172.21.15.42:0/2243693895 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fedc0135830 msgr2=0x7fedc009cb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:57.955 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.948+0000 7fedc58cd640 1 --2- 172.21.15.42:0/2243693895 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fedc0135830 0x7fedc009cb60 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fedb8019520 tx=0x7fedb80196b0 comp rx=0 tx=0).stop 2024-09-16T04:08:57.955 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.948+0000 7fedc58cd640 1 -- 172.21.15.42:0/2243693895 shutdown_connections 2024-09-16T04:08:57.955 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.948+0000 7fedc58cd640 1 --2- 172.21.15.42:0/2243693895 >> v2:172.21.15.154:6812/254845342 conn(0x7feda0037810 0x7feda0039cd0 unknown :-1 s=CLOSED pgs=3906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:08:57.956 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.948+0000 7fedc58cd640 1 --2- 172.21.15.42:0/2243693895 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fedc0135830 0x7fedc009cb60 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:08:57.956 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.948+0000 7fedc58cd640 1 --2- 172.21.15.42:0/2243693895 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fedc00a1e20 0x7fedc01352f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:08:57.956 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.948+0000 7fedc58cd640 1 --2- 172.21.15.42:0/2243693895 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fedc00a1330 0x7fedc01ceca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:08:57.956 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.948+0000 7fedc58cd640 1 -- 172.21.15.42:0/2243693895 >> 172.21.15.42:0/2243693895 conn(0x7fedc00a2420 msgr2=0x7fedc00a4dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:57.956 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.948+0000 7fedc58cd640 1 -- 172.21.15.42:0/2243693895 shutdown_connections 2024-09-16T04:08:57.956 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.948+0000 7fedc58cd640 1 -- 172.21.15.42:0/2243693895 wait complete. 2024-09-16T04:08:57.956 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:57.948+0000 7fedc58cd640 1 librados: shutdown 2024-09-16T04:08:57.971 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T04:08:57.971 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T04:08:57.971 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T04:08:57.972 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T04:08:58.099 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.092+0000 7f06759df640 10 monclient: get_monmap_and_config 2024-09-16T04:08:58.099 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.092+0000 7f06759df640 10 monclient: build_initial_monmap 2024-09-16T04:08:58.099 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.092+0000 7f06759df640 10 monclient: monmap: 2024-09-16T04:08:58.099 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:08:58.099 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:08:58.099 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:08:58.097665+0000 2024-09-16T04:08:58.099 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:08:58.097665+0000 2024-09-16T04:08:58.099 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:58.099 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:08:58.099 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:08:58.100 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:08:58.100 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:08:58.100 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:08:58.100 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:08:58.100 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:08:58.100 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.092+0000 7f06759df640 1 Processor -- start 2024-09-16T04:08:58.100 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.092+0000 7f06759df640 1 -- start start 2024-09-16T04:08:58.100 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.092+0000 7f06759df640 10 monclient: init 2024-09-16T04:08:58.105 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.092+0000 7f06759df640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:58.105 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.092+0000 7f06759df640 10 monclient: _add_conns ranks=[4,0,1,2,3] 2024-09-16T04:08:58.105 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.092+0000 7f06759df640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f06700a39b0 0x7f0670133b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:08:58.105 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.092+0000 7f06759df640 10 monclient(hunting): picked mon.noname-c con 0x7f06700a39b0 addr v2:172.21.15.154:3300/0 2024-09-16T04:08:58.105 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.092+0000 7f06759df640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f06700a3d90 addr v1:172.21.15.42:6789/0 2024-09-16T04:08:58.105 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.092+0000 7f06759df640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0670137710 addr v1:172.21.15.154:6789/0 2024-09-16T04:08:58.105 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.092+0000 7f06759df640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f067009c140 con 0x7f06700a3d90 2024-09-16T04:08:58.105 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.092+0000 7f06759df640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f067009c420 con 0x7f0670137710 2024-09-16T04:08:58.105 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.092+0000 7f06759df640 10 monclient(hunting): start opening mon connection 2024-09-16T04:08:58.105 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.092+0000 7f06759df640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f067006dd60 con 0x7f06700a39b0 2024-09-16T04:08:58.105 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.092+0000 7f06759df640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:58.105 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.092+0000 7f06759df640 10 monclient(hunting): authenticate will time out at 2585.067874s 2024-09-16T04:08:58.105 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.096+0000 7f06667fc640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f06700a3d90 0x7f0670137000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:34828/0 (socket says 172.21.15.42:34828) 2024-09-16T04:08:58.106 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.096+0000 7f06667fc640 1 -- 172.21.15.42:0/1623015102 learned_addr learned my addr 172.21.15.42:0/1623015102 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T04:08:58.106 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.096+0000 7f066effd640 1 --2- 172.21.15.42:0/1623015102 >> v2:172.21.15.154:3300/0 conn(0x7f06700a39b0 0x7f0670133b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:08:58.106 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.096+0000 7f066effd640 10 monclient(hunting): get_auth_request con 0x7f06700a39b0 auth_method 0 2024-09-16T04:08:58.106 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.096+0000 7f066e7fc640 1 -- 172.21.15.42:0/1623015102 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2483310017 0 0) 0x7f067009c140 con 0x7f06700a3d90 2024-09-16T04:08:58.106 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.096+0000 7f066effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:08:58.106 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.096+0000 7f066effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:58.106 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.096+0000 7f066effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:58.106 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.096+0000 7f066e7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:08:58.106 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.096+0000 7f066e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:58.106 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.096+0000 7f066e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:58.106 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.096+0000 7f066e7fc640 10 monclient(hunting): my global_id is 1034142 2024-09-16T04:08:58.106 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.096+0000 7f066e7fc640 1 -- 172.21.15.42:0/1623015102 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f065c003530 con 0x7f06700a3d90 2024-09-16T04:08:58.106 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.096+0000 7f066effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:08:58.107 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.096+0000 7f066effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:08:58.107 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.096+0000 7f066effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:08:58.107 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.096+0000 7f066e7fc640 1 -- 172.21.15.42:0/1623015102 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 764297094 0 0) 0x7f065c003530 con 0x7f06700a3d90 2024-09-16T04:08:58.107 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.096+0000 7f066e7fc640 1 -- 172.21.15.42:0/1623015102 >> v2:172.21.15.154:3300/0 conn(0x7f06700a39b0 msgr2=0x7f0670133b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:08:58.107 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.096+0000 7f066e7fc640 1 --2- 172.21.15.42:0/1623015102 >> v2:172.21.15.154:3300/0 conn(0x7f06700a39b0 0x7f0670133b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:08:58.107 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.096+0000 7f066e7fc640 1 -- 172.21.15.42:0/1623015102 >> v1:172.21.15.154:6789/0 conn(0x7f0670137710 legacy=0x7f067013f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:58.107 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.096+0000 7f066e7fc640 10 monclient: _finish_hunting 0 2024-09-16T04:08:58.107 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.096+0000 7f066e7fc640 1 monclient: found mon.noname-a-legacy 2024-09-16T04:08:58.107 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.096+0000 7f066e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:58.107 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.096+0000 7f066e7fc640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.42:6789/0 2024-09-16T04:08:58.107 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.096+0000 7f066e7fc640 1 -- 172.21.15.42:0/1623015102 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06700a5e20 con 0x7f06700a3d90 2024-09-16T04:08:58.107 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.096+0000 7f066e7fc640 10 monclient: _finish_auth 0 2024-09-16T04:08:58.107 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.096+0000 7f066e7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:08:58.107 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.096+0000 7f066e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:58.108 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.096+0000 7f066e7fc640 1 -- 172.21.15.42:0/1623015102 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2177413734 0 0) 0x7f0658001680 con 0x7f06700a3d90 2024-09-16T04:08:58.108 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.096+0000 7f066e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:58.108 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.096+0000 7f066e7fc640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T04:08:58.108 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.096+0000 7f066e7fc640 10 monclient: dump: 2024-09-16T04:08:58.108 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:08:58.108 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:08:58.108 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:08:58.108 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:08:58.108 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:58.108 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:08:58.108 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:08:58.108 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:08:58.108 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:08:58.109 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:08:58.109 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.096+0000 7f066e7fc640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] but i'm connected to v1:172.21.15.42:6789/0, reconnecting 2024-09-16T04:08:58.109 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.096+0000 7f066e7fc640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:58.109 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.096+0000 7f066e7fc640 1 -- 172.21.15.42:0/1623015102 >> v1:172.21.15.42:6789/0 conn(0x7f06700a3d90 legacy=0x7f0670137000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:58.109 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.096+0000 7f066e7fc640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:08:58.109 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.096+0000 7f066e7fc640 1 --2- 172.21.15.42:0/1623015102 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f065c0039c0 0x7f065c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:08:58.109 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.096+0000 7f066e7fc640 10 monclient(hunting): picked mon.c con 0x7f065c0039c0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:08:58.109 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.096+0000 7f066e7fc640 1 --2- 172.21.15.42:0/1623015102 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f065c004810 0x7f065c004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:08:58.109 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.096+0000 7f066e7fc640 10 monclient(hunting): picked mon.a con 0x7f065c004810 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:08:58.109 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.096+0000 7f066e7fc640 1 --2- 172.21.15.42:0/1623015102 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f065c0052f0 0x7f065c00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:08:58.109 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.096+0000 7f066e7fc640 10 monclient(hunting): picked mon.b con 0x7f065c0052f0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:08:58.109 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.096+0000 7f066e7fc640 10 monclient(hunting): start opening mon connection 2024-09-16T04:08:58.109 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.096+0000 7f066e7fc640 1 -- 172.21.15.42:0/1623015102 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f067006dd60 con 0x7f065c004810 2024-09-16T04:08:58.109 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.096+0000 7f066e7fc640 10 monclient(hunting): start opening mon connection 2024-09-16T04:08:58.110 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.096+0000 7f066e7fc640 1 -- 172.21.15.42:0/1623015102 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f065c00c1c0 con 0x7f065c0052f0 2024-09-16T04:08:58.110 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.096+0000 7f066e7fc640 10 monclient(hunting): start opening mon connection 2024-09-16T04:08:58.110 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.096+0000 7f066e7fc640 1 -- 172.21.15.42:0/1623015102 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f065c00c3c0 con 0x7f065c0039c0 2024-09-16T04:08:58.110 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.096+0000 7f066e7fc640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:58.110 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.096+0000 7f066e7fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:08:58.110 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.096+0000 7f06667fc640 1 --2- 172.21.15.42:0/1623015102 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f065c004810 0x7f065c004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:08:58.110 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.096+0000 7f06667fc640 10 monclient(hunting): get_auth_request con 0x7f065c004810 auth_method 0 2024-09-16T04:08:58.110 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.096+0000 7f066effd640 1 --2- 172.21.15.42:0/1623015102 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f065c0039c0 0x7f065c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:08:58.110 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.096+0000 7f06667fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:08:58.110 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.096+0000 7f06667fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:58.110 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.096+0000 7f06667fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:08:58.110 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.096+0000 7f066effd640 10 monclient(hunting): get_auth_request con 0x7f065c0039c0 auth_method 0 2024-09-16T04:08:58.110 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.096+0000 7f066effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:08:58.111 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.096+0000 7f066effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:58.111 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.096+0000 7f066effd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:08:58.111 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.096+0000 7f06667fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:08:58.111 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.096+0000 7f06667fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:08:58.111 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.096+0000 7f06667fc640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:08:58.111 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.096+0000 7f066effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:08:58.111 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.096+0000 7f066effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:08:58.111 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.096+0000 7f066effd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:08:58.111 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.096+0000 7f066f7fe640 1 --2- 172.21.15.42:0/1623015102 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f065c0052f0 0x7f065c00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:08:58.111 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.096+0000 7f066f7fe640 10 monclient(hunting): get_auth_request con 0x7f065c0052f0 auth_method 0 2024-09-16T04:08:58.111 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.096+0000 7f066f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:08:58.111 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.096+0000 7f066f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:58.111 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.096+0000 7f066f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:08:58.111 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.096+0000 7f06667fc640 10 monclient(hunting): handle_auth_done global_id 1034142 payload 293 2024-09-16T04:08:58.112 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.096+0000 7f06667fc640 1 -- 172.21.15.42:0/1623015102 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f065c0039c0 msgr2=0x7f065c003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:08:58.112 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.096+0000 7f06667fc640 1 --2- 172.21.15.42:0/1623015102 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f065c0039c0 0x7f065c003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:08:58.112 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.096+0000 7f06667fc640 1 -- 172.21.15.42:0/1623015102 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f065c0052f0 msgr2=0x7f065c00b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:08:58.112 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.096+0000 7f06667fc640 1 --2- 172.21.15.42:0/1623015102 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f065c0052f0 0x7f065c00b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:08:58.112 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.096+0000 7f06667fc640 10 monclient: _finish_hunting 0 2024-09-16T04:08:58.112 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.096+0000 7f06667fc640 1 monclient: found mon.a 2024-09-16T04:08:58.112 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.096+0000 7f06667fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:58.112 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.096+0000 7f06667fc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:08:58.112 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.096+0000 7f066f7fe640 1 -- 172.21.15.42:0/1623015102 reap_dead start 2024-09-16T04:08:58.112 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.096+0000 7f06667fc640 1 -- 172.21.15.42:0/1623015102 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f065c003530 con 0x7f065c004810 2024-09-16T04:08:58.112 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.096+0000 7f06667fc640 10 monclient: _finish_auth 0 2024-09-16T04:08:58.112 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.096+0000 7f06667fc640 10 monclient: _check_auth_tickets 2024-09-16T04:08:58.112 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.096+0000 7f06667fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:58.113 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.096+0000 7f06759df640 5 monclient: authenticate success, global_id 1034142 2024-09-16T04:08:58.113 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.096+0000 7f06759df640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:08:58.113 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.096+0000 7f06667fc640 1 --2- 172.21.15.42:0/1623015102 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f065c004810 0x7f065c004c90 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f0658001ec0 tx=0x7f065801cb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6884d1aaadb08e3d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:08:58.113 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.096+0000 7f066e7fc640 1 -- 172.21.15.42:0/1623015102 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0658007d40 con 0x7f065c004810 2024-09-16T04:08:58.113 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.096+0000 7f066e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:58.113 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.096+0000 7f066e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:08:58.113 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.096+0000 7f066e7fc640 10 monclient: dump: 2024-09-16T04:08:58.113 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:08:58.113 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:08:58.113 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:08:58.113 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:08:58.113 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:58.113 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:08:58.114 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:08:58.114 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:08:58.114 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:08:58.114 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:08:58.114 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.096+0000 7f06759df640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:08:58.114 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.096+0000 7f066e7fc640 1 -- 172.21.15.42:0/1623015102 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f06580a9070 con 0x7f065c004810 2024-09-16T04:08:58.114 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.096+0000 7f066e7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:58.114 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.096+0000 7f06759df640 10 monclient: get_monmap_and_config success 2024-09-16T04:08:58.114 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.096+0000 7f066e7fc640 1 -- 172.21.15.42:0/1623015102 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0658005d80 con 0x7f065c004810 2024-09-16T04:08:58.114 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.096+0000 7f066e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:58.114 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.096+0000 7f06759df640 4 set_mon_vals no callback set 2024-09-16T04:08:58.114 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.096+0000 7f066e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:08:58.114 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.096+0000 7f066e7fc640 10 monclient: dump: 2024-09-16T04:08:58.114 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:08:58.115 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:08:58.115 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:08:58.115 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:08:58.115 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:58.115 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:08:58.115 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:08:58.115 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:08:58.115 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:08:58.115 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:08:58.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.100+0000 7f06759df640 10 monclient: shutdown 2024-09-16T04:08:58.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.100+0000 7f06759df640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:58.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.100+0000 7f06759df640 1 -- 172.21.15.42:0/1623015102 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f065c004810 msgr2=0x7f065c004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:58.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.100+0000 7f06759df640 1 --2- 172.21.15.42:0/1623015102 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f065c004810 0x7f065c004c90 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f0658001ec0 tx=0x7f065801cb80 comp rx=0 tx=0).stop 2024-09-16T04:08:58.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.100+0000 7f06759df640 1 -- 172.21.15.42:0/1623015102 shutdown_connections 2024-09-16T04:08:58.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.100+0000 7f06759df640 1 --2- 172.21.15.42:0/1623015102 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f065c004810 0x7f065c004c90 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:08:58.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.100+0000 7f06759df640 1 -- 172.21.15.42:0/1623015102 >> 172.21.15.42:0/1623015102 conn(0x7f06700a9fb0 msgr2=0x7f06700aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:58.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.100+0000 7f06759df640 1 -- 172.21.15.42:0/1623015102 shutdown_connections 2024-09-16T04:08:58.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.100+0000 7f06759df640 1 -- 172.21.15.42:0/1623015102 wait complete. 2024-09-16T04:08:58.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.100+0000 7f06759df640 10 monclient: build_initial_monmap 2024-09-16T04:08:58.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.100+0000 7f06759df640 10 monclient: monmap: 2024-09-16T04:08:58.123 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:08:58.123 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:08:58.123 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:08:58.123 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:08:58.123 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:58.123 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:08:58.123 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:08:58.123 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:08:58.123 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:08:58.124 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:08:58.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.100+0000 7f06759df640 1 librados: starting msgr at 2024-09-16T04:08:58.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.100+0000 7f06759df640 1 librados: starting objecter 2024-09-16T04:08:58.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.100+0000 7f06759df640 1 Processor -- start 2024-09-16T04:08:58.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.100+0000 7f06759df640 1 -- start start 2024-09-16T04:08:58.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.100+0000 7f06759df640 1 librados: setting wanted keys 2024-09-16T04:08:58.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.100+0000 7f06759df640 1 librados: calling monclient init 2024-09-16T04:08:58.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.100+0000 7f06759df640 10 monclient: init 2024-09-16T04:08:58.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.100+0000 7f06759df640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:58.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.100+0000 7f06759df640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:08:58.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.100+0000 7f06759df640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f065c004810 0x7f06701cfe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:08:58.124 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.100+0000 7f06759df640 10 monclient(hunting): picked mon.noname-c con 0x7f065c004810 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:08:58.125 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.100+0000 7f06759df640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f06701d03d0 0x7f0670120200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:08:58.125 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.100+0000 7f06759df640 10 monclient(hunting): picked mon.noname-b con 0x7f06701d03d0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:08:58.125 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.100+0000 7f06759df640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0670120740 0x7f0670120bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:08:58.125 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.100+0000 7f06759df640 10 monclient(hunting): picked mon.noname-a con 0x7f0670120740 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:08:58.125 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.100+0000 7f06759df640 10 monclient(hunting): start opening mon connection 2024-09-16T04:08:58.125 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.100+0000 7f06759df640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f067013a700 con 0x7f0670120740 2024-09-16T04:08:58.125 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.100+0000 7f06759df640 10 monclient(hunting): start opening mon connection 2024-09-16T04:08:58.125 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.100+0000 7f06759df640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f067013a580 con 0x7f06701d03d0 2024-09-16T04:08:58.125 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.100+0000 7f06759df640 10 monclient(hunting): start opening mon connection 2024-09-16T04:08:58.125 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.100+0000 7f06759df640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f067013a880 con 0x7f065c004810 2024-09-16T04:08:58.125 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.100+0000 7f06759df640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:58.125 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.100+0000 7f06759df640 10 monclient(hunting): authenticate will time out at 2585.074247s 2024-09-16T04:08:58.125 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.100+0000 7f066effd640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f065c004810 0x7f06701cfe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:08:58.126 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.100+0000 7f066f7fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0670120740 0x7f0670120bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:08:58.126 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.100+0000 7f066effd640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f065c004810 0x7f06701cfe90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:40376/0 (socket says 172.21.15.42:40376) 2024-09-16T04:08:58.126 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.100+0000 7f066effd640 1 -- 172.21.15.42:0/3832995165 learned_addr learned my addr 172.21.15.42:0/3832995165 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:08:58.126 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.100+0000 7f066effd640 10 monclient(hunting): get_auth_request con 0x7f065c004810 auth_method 0 2024-09-16T04:08:58.126 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.100+0000 7f066effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:08:58.126 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.100+0000 7f066effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:58.126 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.100+0000 7f066effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:58.126 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.100+0000 7f066f7fe640 10 monclient(hunting): get_auth_request con 0x7f0670120740 auth_method 0 2024-09-16T04:08:58.126 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.100+0000 7f066f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:08:58.126 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.100+0000 7f066f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:58.126 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.100+0000 7f066f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:58.126 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.100+0000 7f066effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:08:58.126 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.100+0000 7f066effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:08:58.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.100+0000 7f066effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:08:58.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.100+0000 7f066f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:08:58.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.100+0000 7f066f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:08:58.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.100+0000 7f066f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:08:58.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.100+0000 7f06667fc640 1 --2- 172.21.15.42:0/3832995165 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f06701d03d0 0x7f0670120200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:08:58.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.100+0000 7f066effd640 10 monclient(hunting): handle_auth_done global_id 1014296 payload 824 2024-09-16T04:08:58.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.100+0000 7f066effd640 1 -- 172.21.15.42:0/3832995165 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f06701d03d0 msgr2=0x7f0670120200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:58.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.100+0000 7f066effd640 1 --2- 172.21.15.42:0/3832995165 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f06701d03d0 0x7f0670120200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:08:58.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.100+0000 7f066effd640 1 -- 172.21.15.42:0/3832995165 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0670120740 msgr2=0x7f0670120bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:58.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.100+0000 7f066effd640 1 --2- 172.21.15.42:0/3832995165 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0670120740 0x7f0670120bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:08:58.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.100+0000 7f066effd640 10 monclient: _finish_hunting 0 2024-09-16T04:08:58.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.100+0000 7f066effd640 1 monclient: found mon.noname-c 2024-09-16T04:08:58.128 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.100+0000 7f066effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:58.128 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.100+0000 7f066effd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T04:08:58.128 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.100+0000 7f066effd640 1 -- 172.21.15.42:0/3832995165 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06701229e0 con 0x7f065c004810 2024-09-16T04:08:58.128 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.100+0000 7f06667fc640 10 monclient: get_auth_request con 0x7f06701d03d0 auth_method 0 2024-09-16T04:08:58.128 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.100+0000 7f06667fc640 1 --2- 172.21.15.42:0/3832995165 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f06701d03d0 0x7f0670120200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T04:08:58.128 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.100+0000 7f066effd640 1 --2- 172.21.15.42:0/3832995165 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f065c004810 0x7f06701cfe90 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f0660011930 tx=0x7f0660011df0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:08:58.128 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.100+0000 7f066cff9640 10 client.?.objecter ms_handle_connect 0x7f065c004810 2024-09-16T04:08:58.128 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.100+0000 7f066cff9640 10 client.?.objecter resend_mon_ops 2024-09-16T04:08:58.128 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.100+0000 7f066cff9640 1 -- 172.21.15.42:0/3832995165 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0660019070 con 0x7f065c004810 2024-09-16T04:08:58.128 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.100+0000 7f066cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:58.128 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.100+0000 7f066cff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:08:58.128 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.100+0000 7f066cff9640 10 monclient: dump: 2024-09-16T04:08:58.128 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:08:58.129 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:08:58.129 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:08:58.129 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:08:58.129 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:58.129 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:08:58.129 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:08:58.129 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:08:58.129 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:08:58.129 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:08:58.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.100+0000 7f066cff9640 10 monclient: _finish_auth 0 2024-09-16T04:08:58.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.100+0000 7f066cff9640 10 monclient: _check_auth_tickets 2024-09-16T04:08:58.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.100+0000 7f066cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:58.129 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.100+0000 7f066cff9640 1 -- 172.21.15.42:0/3832995165 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0660004240 con 0x7f065c004810 2024-09-16T04:08:58.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.100+0000 7f066cff9640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:58.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.100+0000 7f066cff9640 1 -- 172.21.15.42:0/3832995165 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06600095b0 con 0x7f065c004810 2024-09-16T04:08:58.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.100+0000 7f066cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:58.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.100+0000 7f066cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:08:58.130 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.100+0000 7f066cff9640 10 monclient: dump: 2024-09-16T04:08:58.130 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:08:58.130 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:08:58.130 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:08:58.130 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:08:58.130 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:58.130 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:08:58.130 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:08:58.130 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:08:58.131 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:08:58.131 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:08:58.131 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.100+0000 7f06759df640 5 monclient: authenticate success, global_id 1014296 2024-09-16T04:08:58.131 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.100+0000 7f06759df640 10 monclient: _renew_subs 2024-09-16T04:08:58.131 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.100+0000 7f066d7fa640 4 set_mon_vals no callback set 2024-09-16T04:08:58.131 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.100+0000 7f06759df640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:08:58.131 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.100+0000 7f06759df640 1 -- 172.21.15.42:0/3832995165 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f067006f680 con 0x7f065c004810 2024-09-16T04:08:58.131 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.100+0000 7f06759df640 10 client.1014296.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:08:58.131 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.100+0000 7f06759df640 10 monclient: _renew_subs 2024-09-16T04:08:58.131 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.100+0000 7f06759df640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:08:58.131 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.100+0000 7f06759df640 1 -- 172.21.15.42:0/3832995165 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f0670122cc0 con 0x7f065c004810 2024-09-16T04:08:58.131 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.100+0000 7f06759df640 1 librados: init done 2024-09-16T04:08:58.131 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.100+0000 7f06759df640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:58.131 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.100+0000 7f06759df640 1 --2- 172.21.15.42:0/3832995165 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f067009e8e0 0x7f067009ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:08:58.132 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.100+0000 7f06759df640 10 monclient(hunting): start opening mon connection 2024-09-16T04:08:58.132 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.100+0000 7f06759df640 1 -- 172.21.15.42:0/3832995165 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f0670130c80 con 0x7f067009e8e0 2024-09-16T04:08:58.132 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.104+0000 7f06667fc640 1 --2- 172.21.15.42:0/3832995165 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f067009e8e0 0x7f067009ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:08:58.132 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.104+0000 7f06667fc640 10 monclient: get_auth_request con 0x7f067009e8e0 auth_method 0 2024-09-16T04:08:58.132 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.104+0000 7f06667fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:08:58.132 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.104+0000 7f06667fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:58.132 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.104+0000 7f06667fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:58.132 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.104+0000 7f06667fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:08:58.132 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.104+0000 7f06667fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:08:58.132 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.104+0000 7f06667fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:08:58.132 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.104+0000 7f066cff9640 1 -- 172.21.15.42:0/3832995165 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f066001c070 con 0x7f065c004810 2024-09-16T04:08:58.132 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.104+0000 7f06667fc640 10 monclient(hunting): handle_auth_done global_id 1034148 payload 824 2024-09-16T04:08:58.132 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.104+0000 7f06667fc640 1 -- 172.21.15.42:0/3832995165 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f067013ab20 con 0x7f067009e8e0 2024-09-16T04:08:58.133 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.104+0000 7f066cff9640 1 --2- 172.21.15.42:0/3832995165 >> v2:172.21.15.154:6812/254845342 conn(0x7f0650037810 0x7f0650039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:08:58.133 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.104+0000 7f066cff9640 1 -- 172.21.15.42:0/3832995165 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(706..706 src has 451..706) ==== 5040+0+0 (secure 0 0 0) 0x7f0660049140 con 0x7f065c004810 2024-09-16T04:08:58.133 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.104+0000 7f066cff9640 10 client.1014296.objecter ms_dispatch 0x7f0670000ba0 osd_map(706..706 src has 451..706) 2024-09-16T04:08:58.133 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.104+0000 7f066cff9640 3 client.1014296.objecter handle_osd_map got epochs [706,706] > 0 2024-09-16T04:08:58.133 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.104+0000 7f066cff9640 3 client.1014296.objecter handle_osd_map decoding full epoch 706 2024-09-16T04:08:58.133 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.104+0000 7f066cff9640 20 client.1014296.objecter dump_active .. 0 homeless 2024-09-16T04:08:58.133 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.104+0000 7f066cff9640 10 client.1014296.objecter ms_handle_connect 0x7f067009e8e0 2024-09-16T04:08:58.133 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.104+0000 7f066cff9640 10 client.1014296.objecter resend_mon_ops 2024-09-16T04:08:58.133 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.104+0000 7f06667fc640 1 --2- 172.21.15.42:0/3832995165 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f067009e8e0 0x7f067009ecc0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f065c0057f0 tx=0x7f065c005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:08:58.133 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.104+0000 7f066cff9640 1 -- 172.21.15.42:0/3832995165 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0658007e00 con 0x7f067009e8e0 2024-09-16T04:08:58.133 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.104+0000 7f066cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:58.133 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.104+0000 7f066cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:08:58.133 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.104+0000 7f066cff9640 10 monclient: dump: 2024-09-16T04:08:58.134 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:08:58.134 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:08:58.134 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:08:58.134 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:08:58.134 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:58.134 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:08:58.134 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:08:58.134 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:08:58.134 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:08:58.134 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:08:58.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.104+0000 7f066f7fe640 1 --2- 172.21.15.42:0/3832995165 >> v2:172.21.15.154:6812/254845342 conn(0x7f0650037810 0x7f0650039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:08:58.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.104+0000 7f066f7fe640 10 monclient: get_auth_request con 0x7f0650037810 auth_method 0 2024-09-16T04:08:58.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.104+0000 7f066cff9640 1 -- 172.21.15.42:0/3832995165 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f06580a9040 con 0x7f067009e8e0 2024-09-16T04:08:58.134 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.104+0000 7f066cff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:58.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.104+0000 7f066cff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:08:58.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.104+0000 7f066cff9640 1 -- 172.21.15.42:0/3832995165 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f067009e8e0 msgr2=0x7f067009ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:58.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.104+0000 7f066cff9640 1 --2- 172.21.15.42:0/3832995165 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f067009e8e0 0x7f067009ecc0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f065c0057f0 tx=0x7f065c005840 comp rx=0 tx=0).stop 2024-09-16T04:08:58.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.104+0000 7f066cff9640 1 -- 172.21.15.42:0/3832995165 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f067009e8e0 msgr2=0x7f067009ecc0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:08:58.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.104+0000 7f066cff9640 1 --2- 172.21.15.42:0/3832995165 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f067009e8e0 0x7f067009ecc0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:08:58.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.104+0000 7f066f7fe640 1 --2- 172.21.15.42:0/3832995165 >> v2:172.21.15.154:6812/254845342 conn(0x7f0650037810 0x7f0650039cd0 secure :-1 s=READY pgs=3907 cs=0 l=1 rev1=1 crypto rx=0x7f06700a4170 tx=0x7f0668009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:08:58.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.104+0000 7f066cff9640 10 client.1014296.objecter ms_handle_connect 0x7f0650037810 2024-09-16T04:08:58.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.108+0000 7f06759df640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:08:58.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.108+0000 7f06759df640 1 --2- 172.21.15.42:0/3832995165 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f067009f200 0x7f06701dadb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:08:58.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.108+0000 7f06759df640 10 monclient(hunting): start opening mon connection 2024-09-16T04:08:58.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.108+0000 7f06759df640 1 -- 172.21.15.42:0/3832995165 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f0670133780 con 0x7f067009f200 2024-09-16T04:08:58.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.108+0000 7f06667fc640 1 --2- 172.21.15.42:0/3832995165 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f067009f200 0x7f06701dadb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:08:58.135 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.108+0000 7f06667fc640 10 monclient: get_auth_request con 0x7f067009f200 auth_method 0 2024-09-16T04:08:58.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.108+0000 7f06667fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:08:58.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.108+0000 7f06667fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:58.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.108+0000 7f06667fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:58.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.108+0000 7f06667fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:08:58.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.108+0000 7f06667fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:08:58.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.108+0000 7f06667fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:08:58.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.108+0000 7f06667fc640 10 monclient(hunting): handle_auth_done global_id 1034151 payload 824 2024-09-16T04:08:58.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.108+0000 7f06667fc640 1 -- 172.21.15.42:0/3832995165 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f0670068e40 con 0x7f067009f200 2024-09-16T04:08:58.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.112+0000 7f06667fc640 1 --2- 172.21.15.42:0/3832995165 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f067009f200 0x7f06701dadb0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f0658001980 tx=0x7f06580a7000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:08:58.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.112+0000 7f066cff9640 10 client.1014296.objecter ms_handle_connect 0x7f067009f200 2024-09-16T04:08:58.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.112+0000 7f066cff9640 10 client.1014296.objecter resend_mon_ops 2024-09-16T04:08:58.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.112+0000 7f066cff9640 1 -- 172.21.15.42:0/3832995165 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06580a5a70 con 0x7f067009f200 2024-09-16T04:08:58.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.112+0000 7f066cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:58.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.112+0000 7f066cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:08:58.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.112+0000 7f066cff9640 10 monclient: dump: 2024-09-16T04:08:58.137 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:08:58.137 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:08:58.137 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:08:58.137 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:08:58.137 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:58.137 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:08:58.137 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:08:58.137 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:08:58.137 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:08:58.137 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:08:58.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.112+0000 7f066cff9640 1 -- 172.21.15.42:0/3832995165 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4547 (secure 0 0 0) 0x7f06580c1070 con 0x7f067009f200 2024-09-16T04:08:58.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.112+0000 7f066cff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:08:58.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.112+0000 7f066cff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:08:58.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.112+0000 7f066cff9640 1 -- 172.21.15.42:0/3832995165 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f067009f200 msgr2=0x7f06701dadb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:58.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.112+0000 7f066cff9640 1 --2- 172.21.15.42:0/3832995165 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f067009f200 0x7f06701dadb0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f0658001980 tx=0x7f06580a7000 comp rx=0 tx=0).stop 2024-09-16T04:08:58.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.112+0000 7f066cff9640 1 -- 172.21.15.42:0/3832995165 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f067009f200 msgr2=0x7f06701dadb0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:08:58.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.112+0000 7f066cff9640 1 --2- 172.21.15.42:0/3832995165 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f067009f200 0x7f06701dadb0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:08:58.138 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T04:08:58.138 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:08:58.138 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:08:58.138 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "leader", 2024-09-16T04:08:58.138 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 248, 2024-09-16T04:08:58.139 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 18047, 2024-09-16T04:08:58.139 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T04:08:58.139 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T04:08:58.139 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T04:08:58.139 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T04:08:58.139 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:08:58.139 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 2, 2024-09-16T04:08:58.139 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:08:58.139 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T04:08:58.139 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T04:08:58.139 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:08:58.139 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:08:58.139 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:08:58.140 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:08:58.140 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:08:58.140 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:08:58.140 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:08:58.140 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:08:58.140 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:08:58.140 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:08:58.140 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:08:58.140 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:08:58.140 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:08:58.140 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T04:08:58.140 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:08:58.140 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:08:58.140 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:08:58.141 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:08:58.141 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:08:58.141 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:08:58.141 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:08:58.141 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:08:58.141 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:08:58.141 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:08:58.141 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:08:58.141 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:08:58.141 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:08:58.141 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T04:08:58.141 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T04:08:58.141 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T04:08:58.142 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T04:08:58.142 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T04:08:58.142 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T04:08:58.142 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:08:58.142 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:08:58.142 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T04:08:58.142 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T04:08:58.142 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T04:08:58.142 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T04:08:58.142 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T04:08:58.142 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T04:08:58.142 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T04:08:58.142 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:08:58.142 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T04:08:58.142 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:08:58.143 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:08:58.143 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:08:58.143 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:08:58.143 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:08:58.143 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:08:58.143 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:08:58.143 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:08:58.143 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:08:58.143 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:08:58.143 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:08:58.143 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:08:58.143 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T04:08:58.143 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:08:58.143 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T04:08:58.143 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:08:58.143 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:08:58.144 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:08:58.144 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:08:58.144 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:08:58.144 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:08:58.144 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:08:58.144 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T04:08:58.144 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:08:58.144 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:08:58.144 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:08:58.144 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:08:58.144 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T04:08:58.144 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:08:58.144 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:08:58.144 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:08:58.145 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:08:58.145 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T04:08:58.145 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T04:08:58.145 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:08:58.145 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:08:58.145 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:08:58.145 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:08:58.145 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:08:58.145 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:08:58.145 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:08:58.145 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:08:58.145 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:08:58.146 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:08:58.146 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:08:58.146 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T04:08:58.146 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:08:58.146 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:08:58.146 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:08:58.146 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:08:58.146 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T04:08:58.146 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:08:58.146 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:08:58.146 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:08:58.146 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:08:58.146 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T04:08:58.146 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T04:08:58.147 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:08:58.147 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:08:58.147 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:08:58.147 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:08:58.147 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:08:58.147 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:08:58.147 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:08:58.147 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:08:58.147 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:08:58.147 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:08:58.147 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:08:58.147 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T04:08:58.147 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:08:58.148 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:08:58.148 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:08:58.148 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:08:58.148 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T04:08:58.148 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:08:58.148 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:08:58.148 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:08:58.148 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:08:58.148 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T04:08:58.148 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T04:08:58.148 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:08:58.148 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:08:58.148 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:08:58.149 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:08:58.149 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:08:58.149 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:08:58.149 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T04:08:58.149 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T04:08:58.149 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:08:58.149 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:08:58.149 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:08:58.149 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:08:58.149 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:08:58.149 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:08:58.149 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T04:08:58.149 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:08:58.150 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:08:58.150 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:08:58.150 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:08:58.150 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:08:58.150 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:08:58.150 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:08:58.150 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T04:08:58.150 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T04:08:58.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.112+0000 7f06759df640 10 librados: watch_flush enter 2024-09-16T04:08:58.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.112+0000 7f06759df640 10 librados: watch_flush exit 2024-09-16T04:08:58.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.112+0000 7f06759df640 20 client.1014296.objecter shutdown clearing up homeless session... 2024-09-16T04:08:58.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.112+0000 7f06759df640 10 client.1014296.objecter successfully canceled tick 2024-09-16T04:08:58.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.112+0000 7f06759df640 1 -- 172.21.15.42:0/3832995165 >> v2:172.21.15.154:6812/254845342 conn(0x7f0650037810 msgr2=0x7f0650039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:58.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.112+0000 7f06759df640 1 --2- 172.21.15.42:0/3832995165 >> v2:172.21.15.154:6812/254845342 conn(0x7f0650037810 0x7f0650039cd0 secure :-1 s=READY pgs=3907 cs=0 l=1 rev1=1 crypto rx=0x7f06700a4170 tx=0x7f0668009040 comp rx=0 tx=0).stop 2024-09-16T04:08:58.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.112+0000 7f06759df640 10 monclient: shutdown 2024-09-16T04:08:58.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.112+0000 7f06759df640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:58.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.112+0000 7f06759df640 1 -- 172.21.15.42:0/3832995165 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f065c004810 msgr2=0x7f06701cfe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:58.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.112+0000 7f06759df640 1 --2- 172.21.15.42:0/3832995165 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f065c004810 0x7f06701cfe90 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f0660011930 tx=0x7f0660011df0 comp rx=0 tx=0).stop 2024-09-16T04:08:58.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.112+0000 7f066f7fe640 1 -- 172.21.15.42:0/3832995165 reap_dead start 2024-09-16T04:08:58.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.112+0000 7f066f7fe640 1 -- 172.21.15.42:0/3832995165 reap_dead start 2024-09-16T04:08:58.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.112+0000 7f06759df640 1 -- 172.21.15.42:0/3832995165 shutdown_connections 2024-09-16T04:08:58.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.112+0000 7f06759df640 1 -- 172.21.15.42:0/3832995165 >> 172.21.15.42:0/3832995165 conn(0x7f06700a9fb0 msgr2=0x7f06700a7a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:58.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.112+0000 7f06759df640 1 -- 172.21.15.42:0/3832995165 shutdown_connections 2024-09-16T04:08:58.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.112+0000 7f06759df640 1 -- 172.21.15.42:0/3832995165 wait complete. 2024-09-16T04:08:58.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.112+0000 7f06759df640 1 librados: shutdown 2024-09-16T04:08:58.153 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:08:58.153 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T04:08:58.154 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T04:08:58.271 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.264+0000 7f8f100af640 10 monclient: get_monmap_and_config 2024-09-16T04:08:58.271 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.264+0000 7f8f100af640 10 monclient: build_initial_monmap 2024-09-16T04:08:58.271 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.264+0000 7f8f100af640 10 monclient: monmap: 2024-09-16T04:08:58.271 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:08:58.271 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:08:58.271 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:08:58.269491+0000 2024-09-16T04:08:58.271 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:08:58.269491+0000 2024-09-16T04:08:58.271 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:58.271 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:08:58.271 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:08:58.272 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:08:58.272 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:08:58.272 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:08:58.272 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:08:58.272 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:08:58.274 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.264+0000 7f8f100af640 1 Processor -- start 2024-09-16T04:08:58.274 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.264+0000 7f8f100af640 1 -- start start 2024-09-16T04:08:58.274 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.264+0000 7f8f100af640 10 monclient: init 2024-09-16T04:08:58.274 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.264+0000 7f8f100af640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:58.274 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.264+0000 7f8f100af640 10 monclient: _add_conns ranks=[4,0,2,1,3] 2024-09-16T04:08:58.274 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.264+0000 7f8f100af640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f8f080a4720 0x7f8f081374e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:08:58.275 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.264+0000 7f8f100af640 10 monclient(hunting): picked mon.noname-c con 0x7f8f080a4720 addr v2:172.21.15.154:3300/0 2024-09-16T04:08:58.275 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.264+0000 7f8f100af640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8f08137a20 addr v1:172.21.15.42:6789/0 2024-09-16T04:08:58.275 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.264+0000 7f8f100af640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f8f08137e30 0x7f8f080a5210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:08:58.275 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.264+0000 7f8f100af640 10 monclient(hunting): picked mon.noname-b con 0x7f8f08137e30 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:08:58.275 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.264+0000 7f8f100af640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8f080a1470 con 0x7f8f08137a20 2024-09-16T04:08:58.275 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.264+0000 7f8f100af640 10 monclient(hunting): start opening mon connection 2024-09-16T04:08:58.275 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.264+0000 7f8f100af640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f8f0806e8f0 con 0x7f8f080a4720 2024-09-16T04:08:58.275 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.264+0000 7f8f100af640 10 monclient(hunting): start opening mon connection 2024-09-16T04:08:58.275 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.264+0000 7f8f100af640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f8f080a5f60 con 0x7f8f08137e30 2024-09-16T04:08:58.275 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.264+0000 7f8f100af640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:58.275 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.264+0000 7f8f100af640 10 monclient(hunting): authenticate will time out at 2585.238942s 2024-09-16T04:08:58.275 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.264+0000 7f8f0e64c640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f8f08137e30 0x7f8f080a5210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:08:58.275 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.264+0000 7f8f0d64a640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f8f08137a20 0x7f8f080a4b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:34832/0 (socket says 172.21.15.42:34832) 2024-09-16T04:08:58.275 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.264+0000 7f8f0e64c640 10 monclient(hunting): get_auth_request con 0x7f8f08137e30 auth_method 0 2024-09-16T04:08:58.275 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.264+0000 7f8f0d64a640 1 -- 172.21.15.42:0/92118628 learned_addr learned my addr 172.21.15.42:0/92118628 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T04:08:58.276 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.264+0000 7f8f0e64c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:08:58.276 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.264+0000 7f8f0e64c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:58.276 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.264+0000 7f8f0e64c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:58.276 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.264+0000 7f8f0e64c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:08:58.276 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.264+0000 7f8f0e64c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:08:58.276 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.264+0000 7f8f0e64c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:08:58.276 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.264+0000 7f8f0de4b640 1 --2- 172.21.15.42:0/92118628 >> v2:172.21.15.154:3300/0 conn(0x7f8f080a4720 0x7f8f081374e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:08:58.276 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.264+0000 7f8f0de4b640 10 monclient(hunting): get_auth_request con 0x7f8f080a4720 auth_method 0 2024-09-16T04:08:58.276 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.264+0000 7f8f0de4b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:08:58.276 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.264+0000 7f8f0de4b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:58.276 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.264+0000 7f8f0de4b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:58.276 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.264+0000 7f8f0e64c640 10 monclient(hunting): handle_auth_done global_id 1014299 payload 274 2024-09-16T04:08:58.276 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.264+0000 7f8f0e64c640 1 -- 172.21.15.42:0/92118628 >> v2:172.21.15.154:3300/0 conn(0x7f8f080a4720 msgr2=0x7f8f081374e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:08:58.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.264+0000 7f8f0ce49640 1 -- 172.21.15.42:0/92118628 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2755917327 0 0) 0x7f8f080a1470 con 0x7f8f08137a20 2024-09-16T04:08:58.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.264+0000 7f8f0e64c640 1 --2- 172.21.15.42:0/92118628 >> v2:172.21.15.154:3300/0 conn(0x7f8f080a4720 0x7f8f081374e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:08:58.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.264+0000 7f8f0e64c640 1 -- 172.21.15.42:0/92118628 >> v1:172.21.15.42:6789/0 conn(0x7f8f08137a20 legacy=0x7f8f080a4b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:58.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.264+0000 7f8f0e64c640 10 monclient: _finish_hunting 0 2024-09-16T04:08:58.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.264+0000 7f8f0e64c640 1 monclient: found mon.noname-b 2024-09-16T04:08:58.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.264+0000 7f8f0e64c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:58.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.264+0000 7f8f0e64c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T04:08:58.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.264+0000 7f8f0e64c640 1 -- 172.21.15.42:0/92118628 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f080a1750 con 0x7f8f08137e30 2024-09-16T04:08:58.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.264+0000 7f8f0ce49640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T04:08:58.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.268+0000 7f8f0e64c640 1 --2- 172.21.15.42:0/92118628 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f8f08137e30 0x7f8f080a5210 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f8f040097c0 tx=0x7f8f0401c360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ed0494585b3ff057 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:08:58.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.268+0000 7f8f0ce49640 1 -- 172.21.15.42:0/92118628 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f040b0070 con 0x7f8f08137e30 2024-09-16T04:08:58.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.268+0000 7f8f0ce49640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:58.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.268+0000 7f8f0ce49640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:08:58.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.268+0000 7f8f0ce49640 10 monclient: dump: 2024-09-16T04:08:58.283 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:08:58.283 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:08:58.283 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:08:58.283 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:08:58.283 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:58.284 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:08:58.284 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:08:58.284 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:08:58.284 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:08:58.284 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:08:58.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.268+0000 7f8f0ce49640 10 monclient: _finish_auth 0 2024-09-16T04:08:58.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.268+0000 7f8f0ce49640 10 monclient: _check_auth_tickets 2024-09-16T04:08:58.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.268+0000 7f8f0ce49640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:58.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.268+0000 7f8f0ce49640 1 -- 172.21.15.42:0/92118628 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8f040ab600 con 0x7f8f08137e30 2024-09-16T04:08:58.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.268+0000 7f8f100af640 5 monclient: authenticate success, global_id 1014299 2024-09-16T04:08:58.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.268+0000 7f8f100af640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:08:58.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.268+0000 7f8f0ce49640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:58.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.268+0000 7f8f100af640 10 monclient: get_monmap_and_config success 2024-09-16T04:08:58.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.268+0000 7f8f0ce49640 1 -- 172.21.15.42:0/92118628 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f040ab9d0 con 0x7f8f08137e30 2024-09-16T04:08:58.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.268+0000 7f8f100af640 4 set_mon_vals no callback set 2024-09-16T04:08:58.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.268+0000 7f8f0ce49640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:58.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.268+0000 7f8f0ce49640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:08:58.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.268+0000 7f8f0ce49640 10 monclient: dump: 2024-09-16T04:08:58.285 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:08:58.285 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:08:58.285 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:08:58.285 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:08:58.285 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:58.285 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:08:58.285 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:08:58.285 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:08:58.285 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:08:58.286 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:08:58.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.268+0000 7f8f100af640 10 monclient: shutdown 2024-09-16T04:08:58.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.268+0000 7f8f100af640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:58.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.268+0000 7f8f100af640 1 -- 172.21.15.42:0/92118628 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f8f08137e30 msgr2=0x7f8f080a5210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:58.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.268+0000 7f8f100af640 1 --2- 172.21.15.42:0/92118628 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f8f08137e30 0x7f8f080a5210 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f8f040097c0 tx=0x7f8f0401c360 comp rx=0 tx=0).stop 2024-09-16T04:08:58.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.268+0000 7f8f100af640 1 -- 172.21.15.42:0/92118628 shutdown_connections 2024-09-16T04:08:58.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.268+0000 7f8f100af640 1 --2- 172.21.15.42:0/92118628 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f8f08137e30 0x7f8f080a5210 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:08:58.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.268+0000 7f8f100af640 1 --2- 172.21.15.42:0/92118628 >> v2:172.21.15.154:3300/0 conn(0x7f8f080a4720 0x7f8f081374e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:08:58.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.268+0000 7f8f100af640 1 -- 172.21.15.42:0/92118628 >> 172.21.15.42:0/92118628 conn(0x7f8f080a6820 msgr2=0x7f8f080a6c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:58.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.268+0000 7f8f100af640 1 -- 172.21.15.42:0/92118628 shutdown_connections 2024-09-16T04:08:58.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.268+0000 7f8f100af640 1 -- 172.21.15.42:0/92118628 wait complete. 2024-09-16T04:08:58.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.268+0000 7f8f100af640 10 monclient: build_initial_monmap 2024-09-16T04:08:58.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.268+0000 7f8f100af640 10 monclient: monmap: 2024-09-16T04:08:58.287 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:08:58.287 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:08:58.287 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:08:58.287 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:08:58.287 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:58.287 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:08:58.287 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:08:58.287 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:08:58.287 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:08:58.287 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:08:58.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.268+0000 7f8f100af640 1 librados: starting msgr at 2024-09-16T04:08:58.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.268+0000 7f8f100af640 1 librados: starting objecter 2024-09-16T04:08:58.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.268+0000 7f8f100af640 1 Processor -- start 2024-09-16T04:08:58.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.268+0000 7f8f100af640 1 -- start start 2024-09-16T04:08:58.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.268+0000 7f8f100af640 1 librados: setting wanted keys 2024-09-16T04:08:58.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.268+0000 7f8f100af640 1 librados: calling monclient init 2024-09-16T04:08:58.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.268+0000 7f8f100af640 10 monclient: init 2024-09-16T04:08:58.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.268+0000 7f8f100af640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:58.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.268+0000 7f8f100af640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:08:58.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.268+0000 7f8f100af640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f8f080a4720 0x7f8f081cf370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:08:58.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.268+0000 7f8f100af640 10 monclient(hunting): picked mon.noname-c con 0x7f8f080a4720 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:08:58.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.268+0000 7f8f100af640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8f08137a20 0x7f8f08136660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:08:58.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.272+0000 7f8f100af640 10 monclient(hunting): picked mon.noname-a con 0x7f8f08137a20 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:08:58.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.272+0000 7f8f100af640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8f08137e30 0x7f8f0813c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:08:58.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.272+0000 7f8f100af640 10 monclient(hunting): picked mon.noname-b con 0x7f8f08137e30 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:08:58.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.272+0000 7f8f100af640 10 monclient(hunting): start opening mon connection 2024-09-16T04:08:58.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.272+0000 7f8f100af640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f8f081420e0 con 0x7f8f08137a20 2024-09-16T04:08:58.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.272+0000 7f8f100af640 10 monclient(hunting): start opening mon connection 2024-09-16T04:08:58.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.272+0000 7f8f100af640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f8f08141f60 con 0x7f8f08137e30 2024-09-16T04:08:58.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.272+0000 7f8f100af640 10 monclient(hunting): start opening mon connection 2024-09-16T04:08:58.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.272+0000 7f8f100af640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f8f08142260 con 0x7f8f080a4720 2024-09-16T04:08:58.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.272+0000 7f8f100af640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:58.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.272+0000 7f8f100af640 10 monclient(hunting): authenticate will time out at 2585.244049s 2024-09-16T04:08:58.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.272+0000 7f8f0de4b640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f8f080a4720 0x7f8f081cf370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:08:58.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.272+0000 7f8f0de4b640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f8f080a4720 0x7f8f081cf370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:40402/0 (socket says 172.21.15.42:40402) 2024-09-16T04:08:58.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.272+0000 7f8f0de4b640 1 -- 172.21.15.42:0/1792073153 learned_addr learned my addr 172.21.15.42:0/1792073153 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:08:58.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.272+0000 7f8f0de4b640 10 monclient(hunting): get_auth_request con 0x7f8f080a4720 auth_method 0 2024-09-16T04:08:58.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.272+0000 7f8f0de4b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:08:58.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.272+0000 7f8f0de4b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:58.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.272+0000 7f8f0de4b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:58.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.272+0000 7f8f0d64a640 1 --2- 172.21.15.42:0/1792073153 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8f08137a20 0x7f8f08136660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:08:58.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.272+0000 7f8f0d64a640 10 monclient(hunting): get_auth_request con 0x7f8f08137a20 auth_method 0 2024-09-16T04:08:58.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.272+0000 7f8f0d64a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:08:58.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.272+0000 7f8f0d64a640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:58.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.272+0000 7f8f0d64a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:58.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.272+0000 7f8f0de4b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:08:58.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.272+0000 7f8f0de4b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:08:58.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.272+0000 7f8f0de4b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:08:58.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.272+0000 7f8f0d64a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:08:58.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.272+0000 7f8f0d64a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:08:58.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.272+0000 7f8f0d64a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:08:58.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.272+0000 7f8f0e64c640 1 --2- 172.21.15.42:0/1792073153 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8f08137e30 0x7f8f0813c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:08:58.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.272+0000 7f8f0e64c640 10 monclient(hunting): get_auth_request con 0x7f8f08137e30 auth_method 0 2024-09-16T04:08:58.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.272+0000 7f8f0e64c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:08:58.291 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.272+0000 7f8f0e64c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:58.291 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.272+0000 7f8f0e64c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:58.291 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.272+0000 7f8f0de4b640 10 monclient(hunting): handle_auth_done global_id 1014302 payload 824 2024-09-16T04:08:58.291 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.272+0000 7f8f0de4b640 1 -- 172.21.15.42:0/1792073153 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8f08137e30 msgr2=0x7f8f0813c3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:58.291 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.272+0000 7f8f0de4b640 1 --2- 172.21.15.42:0/1792073153 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8f08137e30 0x7f8f0813c3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:08:58.291 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.272+0000 7f8f0de4b640 1 -- 172.21.15.42:0/1792073153 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8f08137a20 msgr2=0x7f8f08136660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:58.291 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.272+0000 7f8f0de4b640 1 --2- 172.21.15.42:0/1792073153 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8f08137a20 0x7f8f08136660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:08:58.291 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.272+0000 7f8f0de4b640 10 monclient: _finish_hunting 0 2024-09-16T04:08:58.291 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.272+0000 7f8f0de4b640 1 monclient: found mon.noname-c 2024-09-16T04:08:58.291 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.272+0000 7f8f0de4b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:58.291 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.272+0000 7f8f0de4b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T04:08:58.291 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.272+0000 7f8f0de4b640 1 -- 172.21.15.42:0/1792073153 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f08120a30 con 0x7f8f080a4720 2024-09-16T04:08:58.291 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.272+0000 7f8f0d64a640 1 --2- 172.21.15.42:0/1792073153 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8f08137a20 0x7f8f08136660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:08:58.291 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.272+0000 7f8f0de4b640 1 --2- 172.21.15.42:0/1792073153 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f8f080a4720 0x7f8f081cf370 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f8f00007f50 tx=0x7f8f0000e570 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:08:58.292 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.272+0000 7f8efe7fc640 10 client.?.objecter ms_handle_connect 0x7f8f080a4720 2024-09-16T04:08:58.292 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.272+0000 7f8efe7fc640 10 client.?.objecter resend_mon_ops 2024-09-16T04:08:58.292 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.272+0000 7f8efe7fc640 1 -- 172.21.15.42:0/1792073153 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f00014620 con 0x7f8f080a4720 2024-09-16T04:08:58.292 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.272+0000 7f8efe7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:58.292 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.272+0000 7f8efe7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:08:58.292 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.272+0000 7f8efe7fc640 10 monclient: dump: 2024-09-16T04:08:58.292 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:08:58.292 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:08:58.292 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:08:58.292 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:08:58.292 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:58.292 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:08:58.292 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:08:58.293 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:08:58.293 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:08:58.293 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:08:58.293 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.272+0000 7f8efe7fc640 10 monclient: _finish_auth 0 2024-09-16T04:08:58.293 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.272+0000 7f8efe7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:08:58.293 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.272+0000 7f8efe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:58.293 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.272+0000 7f8efe7fc640 1 -- 172.21.15.42:0/1792073153 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8f00019070 con 0x7f8f080a4720 2024-09-16T04:08:58.293 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.272+0000 7f8f100af640 5 monclient: authenticate success, global_id 1014302 2024-09-16T04:08:58.293 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.272+0000 7f8f100af640 10 monclient: _renew_subs 2024-09-16T04:08:58.293 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.272+0000 7f8f100af640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:08:58.293 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.272+0000 7f8f100af640 1 -- 172.21.15.42:0/1792073153 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f08120d10 con 0x7f8f080a4720 2024-09-16T04:08:58.293 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.272+0000 7f8efe7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:58.293 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.272+0000 7f8efe7fc640 1 -- 172.21.15.42:0/1792073153 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f00014a00 con 0x7f8f080a4720 2024-09-16T04:08:58.293 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.272+0000 7f8efe7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:58.294 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.272+0000 7f8efe7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:08:58.294 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.272+0000 7f8efe7fc640 10 monclient: dump: 2024-09-16T04:08:58.294 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:08:58.294 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:08:58.294 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:08:58.294 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:08:58.294 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:58.294 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:08:58.294 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:08:58.294 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:08:58.294 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:08:58.294 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:08:58.294 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.272+0000 7f8efffff640 4 set_mon_vals no callback set 2024-09-16T04:08:58.294 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.272+0000 7f8f100af640 10 client.1014302.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:08:58.295 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.272+0000 7f8f100af640 10 monclient: _renew_subs 2024-09-16T04:08:58.295 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.272+0000 7f8f100af640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:08:58.295 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.272+0000 7f8f100af640 1 -- 172.21.15.42:0/1792073153 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f080a8690 con 0x7f8f080a4720 2024-09-16T04:08:58.295 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.272+0000 7f8f100af640 1 librados: init done 2024-09-16T04:08:58.295 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.272+0000 7f8efe7fc640 1 -- 172.21.15.42:0/1792073153 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f8f00014ba0 con 0x7f8f080a4720 2024-09-16T04:08:58.295 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.272+0000 7f8f100af640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:58.295 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.272+0000 7f8f100af640 1 --2- 172.21.15.42:0/1792073153 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f8f0813c910 0x7f8f0813ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:08:58.295 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.272+0000 7f8f100af640 10 monclient(hunting): start opening mon connection 2024-09-16T04:08:58.295 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.272+0000 7f8f100af640 1 -- 172.21.15.42:0/1792073153 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f8f0813d230 con 0x7f8f0813c910 2024-09-16T04:08:58.295 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.272+0000 7f8efe7fc640 1 --2- 172.21.15.42:0/1792073153 >> v2:172.21.15.154:6812/254845342 conn(0x7f8eec037810 0x7f8eec039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:08:58.295 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.272+0000 7f8f0d64a640 1 --2- 172.21.15.42:0/1792073153 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f8f0813c910 0x7f8f0813ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:08:58.295 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.272+0000 7f8efe7fc640 1 -- 172.21.15.42:0/1792073153 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(706..706 src has 451..706) ==== 5040+0+0 (secure 0 0 0) 0x7f8f000494d0 con 0x7f8f080a4720 2024-09-16T04:08:58.295 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.272+0000 7f8f0d64a640 10 monclient: get_auth_request con 0x7f8f0813c910 auth_method 0 2024-09-16T04:08:58.295 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.272+0000 7f8efe7fc640 10 client.1014302.objecter ms_dispatch 0x7f8f08000ba0 osd_map(706..706 src has 451..706) 2024-09-16T04:08:58.296 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.272+0000 7f8f0d64a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:08:58.296 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.272+0000 7f8f0d64a640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:58.296 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.272+0000 7f8f0d64a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:58.296 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.272+0000 7f8efe7fc640 3 client.1014302.objecter handle_osd_map got epochs [706,706] > 0 2024-09-16T04:08:58.296 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.272+0000 7f8efe7fc640 3 client.1014302.objecter handle_osd_map decoding full epoch 706 2024-09-16T04:08:58.296 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.272+0000 7f8f0d64a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:08:58.296 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.272+0000 7f8f0d64a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:08:58.296 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.272+0000 7f8f0d64a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:08:58.296 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.272+0000 7f8efe7fc640 20 client.1014302.objecter dump_active .. 0 homeless 2024-09-16T04:08:58.296 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.272+0000 7f8f0d64a640 10 monclient(hunting): handle_auth_done global_id 1014305 payload 824 2024-09-16T04:08:58.296 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.272+0000 7f8f0d64a640 1 -- 172.21.15.42:0/1792073153 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8f081374e0 con 0x7f8f0813c910 2024-09-16T04:08:58.296 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.272+0000 7f8f0e64c640 1 --2- 172.21.15.42:0/1792073153 >> v2:172.21.15.154:6812/254845342 conn(0x7f8eec037810 0x7f8eec039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:08:58.296 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.272+0000 7f8efe7fc640 10 client.1014302.objecter ms_handle_connect 0x7f8f0813c910 2024-09-16T04:08:58.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.272+0000 7f8efe7fc640 10 client.1014302.objecter resend_mon_ops 2024-09-16T04:08:58.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.272+0000 7f8f0d64a640 1 --2- 172.21.15.42:0/1792073153 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f8f0813c910 0x7f8f0813ccf0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f8f0813dc70 tx=0x7f8ef8001cb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:08:58.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.272+0000 7f8f0e64c640 10 monclient: get_auth_request con 0x7f8eec037810 auth_method 0 2024-09-16T04:08:58.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.272+0000 7f8efe7fc640 1 -- 172.21.15.42:0/1792073153 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ef800fac0 con 0x7f8f0813c910 2024-09-16T04:08:58.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.272+0000 7f8efe7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:58.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.272+0000 7f8efe7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:08:58.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.272+0000 7f8efe7fc640 10 monclient: dump: 2024-09-16T04:08:58.297 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:08:58.297 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:08:58.297 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:08:58.297 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:08:58.297 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:58.297 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:08:58.297 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:08:58.298 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:08:58.298 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:08:58.298 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:08:58.298 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.272+0000 7f8efe7fc640 1 -- 172.21.15.42:0/1792073153 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f8ef8018070 con 0x7f8f0813c910 2024-09-16T04:08:58.298 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.272+0000 7f8efe7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:58.298 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.272+0000 7f8efe7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:08:58.298 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.272+0000 7f8efe7fc640 1 -- 172.21.15.42:0/1792073153 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f8f0813c910 msgr2=0x7f8f0813ccf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:58.298 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.272+0000 7f8efe7fc640 1 --2- 172.21.15.42:0/1792073153 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f8f0813c910 0x7f8f0813ccf0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f8f0813dc70 tx=0x7f8ef8001cb0 comp rx=0 tx=0).stop 2024-09-16T04:08:58.298 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.272+0000 7f8efe7fc640 1 -- 172.21.15.42:0/1792073153 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f8f0813c910 msgr2=0x7f8f0813ccf0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:08:58.298 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.272+0000 7f8efe7fc640 1 --2- 172.21.15.42:0/1792073153 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f8f0813c910 0x7f8f0813ccf0 secure :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f8f0813dc70 tx=0x7f8ef8001cb0 comp rx=0 tx=0).stop 2024-09-16T04:08:58.298 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.272+0000 7f8efe7fc640 10 client.1014302.objecter ms_handle_connect 0x7f8eec037810 2024-09-16T04:08:58.298 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.272+0000 7f8f0e64c640 1 --2- 172.21.15.42:0/1792073153 >> v2:172.21.15.154:6812/254845342 conn(0x7f8eec037810 0x7f8eec039cd0 secure :-1 s=READY pgs=3908 cs=0 l=1 rev1=1 crypto rx=0x7f8f040ad000 tx=0x7f8f04008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:08:58.300 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.280+0000 7f8f100af640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:08:58.300 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.280+0000 7f8f100af640 1 --2- 172.21.15.42:0/1792073153 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f8f08138700 0x7f8f08138b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:08:58.300 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.280+0000 7f8f100af640 10 monclient(hunting): start opening mon connection 2024-09-16T04:08:58.300 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.280+0000 7f8f100af640 1 -- 172.21.15.42:0/1792073153 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f8f0806f820 con 0x7f8f08138700 2024-09-16T04:08:58.301 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.280+0000 7f8f0d64a640 1 --2- 172.21.15.42:0/1792073153 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f8f08138700 0x7f8f08138b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:08:58.301 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.280+0000 7f8f0d64a640 10 monclient: get_auth_request con 0x7f8f08138700 auth_method 0 2024-09-16T04:08:58.301 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.280+0000 7f8f0d64a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:08:58.301 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.280+0000 7f8f0d64a640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:58.301 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.280+0000 7f8f0d64a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:58.301 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.280+0000 7f8f0d64a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:08:58.301 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.280+0000 7f8f0d64a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:08:58.301 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.280+0000 7f8f0d64a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:08:58.301 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.280+0000 7f8f0d64a640 10 monclient(hunting): handle_auth_done global_id 1014308 payload 824 2024-09-16T04:08:58.301 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.280+0000 7f8f0d64a640 1 -- 172.21.15.42:0/1792073153 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f8f0813e950 con 0x7f8f08138700 2024-09-16T04:08:58.301 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.280+0000 7f8efe7fc640 10 client.1014302.objecter ms_handle_connect 0x7f8f08138700 2024-09-16T04:08:58.301 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.280+0000 7f8f0d64a640 1 --2- 172.21.15.42:0/1792073153 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f8f08138700 0x7f8f08138b10 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f8ef8014000 tx=0x7f8ef8001090 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:08:58.301 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.280+0000 7f8efe7fc640 10 client.1014302.objecter resend_mon_ops 2024-09-16T04:08:58.301 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.280+0000 7f8efe7fc640 1 -- 172.21.15.42:0/1792073153 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ef80136c0 con 0x7f8f08138700 2024-09-16T04:08:58.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.280+0000 7f8efe7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:58.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.280+0000 7f8efe7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:08:58.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.280+0000 7f8efe7fc640 10 monclient: dump: 2024-09-16T04:08:58.302 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:08:58.302 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:08:58.302 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:08:58.302 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:08:58.302 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:58.302 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:08:58.302 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:08:58.302 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:08:58.302 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:08:58.302 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:08:58.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.280+0000 7f8efe7fc640 1 -- 172.21.15.42:0/1792073153 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4713 (secure 0 0 0) 0x7f8ef80136c0 con 0x7f8f08138700 2024-09-16T04:08:58.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.280+0000 7f8efe7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:08:58.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.280+0000 7f8efe7fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:08:58.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.280+0000 7f8efe7fc640 1 -- 172.21.15.42:0/1792073153 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f8f08138700 msgr2=0x7f8f08138b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:58.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.280+0000 7f8efe7fc640 1 --2- 172.21.15.42:0/1792073153 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f8f08138700 0x7f8f08138b10 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f8ef8014000 tx=0x7f8ef8001090 comp rx=0 tx=0).stop 2024-09-16T04:08:58.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.280+0000 7f8efe7fc640 1 -- 172.21.15.42:0/1792073153 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f8f08138700 msgr2=0x7f8f08138b10 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:08:58.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.280+0000 7f8efe7fc640 1 --2- 172.21.15.42:0/1792073153 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f8f08138700 0x7f8f08138b10 secure :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f8ef8014000 tx=0x7f8ef8001090 comp rx=0 tx=0).stop 2024-09-16T04:08:58.303 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T04:08:58.303 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:08:58.303 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:08:58.303 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T04:08:58.304 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 248, 2024-09-16T04:08:58.304 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 133482, 2024-09-16T04:08:58.304 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T04:08:58.304 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T04:08:58.304 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T04:08:58.304 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T04:08:58.304 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:08:58.304 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 2, 2024-09-16T04:08:58.304 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:08:58.304 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T04:08:58.304 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T04:08:58.304 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:08:58.304 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:08:58.305 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:08:58.305 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:08:58.305 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:08:58.305 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:08:58.305 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:08:58.305 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:08:58.305 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:08:58.305 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:08:58.305 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:08:58.305 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:08:58.305 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:08:58.305 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T04:08:58.305 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:08:58.305 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:08:58.306 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:08:58.306 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:08:58.306 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:08:58.306 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:08:58.306 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:08:58.306 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:08:58.306 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:08:58.306 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:08:58.306 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:08:58.306 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:08:58.306 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:08:58.306 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T04:08:58.306 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T04:08:58.307 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T04:08:58.307 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T04:08:58.307 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T04:08:58.307 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T04:08:58.307 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:08:58.307 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:08:58.307 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T04:08:58.307 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T04:08:58.307 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T04:08:58.307 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T04:08:58.307 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T04:08:58.307 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T04:08:58.307 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T04:08:58.307 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:08:58.308 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T04:08:58.308 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:08:58.308 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:08:58.308 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:08:58.308 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:08:58.308 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:08:58.308 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:08:58.308 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:08:58.308 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:08:58.308 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:08:58.308 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:08:58.308 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:08:58.308 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:08:58.309 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T04:08:58.309 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:08:58.309 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T04:08:58.309 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:08:58.309 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:08:58.309 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:08:58.309 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:08:58.309 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:08:58.309 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:08:58.309 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:08:58.309 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T04:08:58.309 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:08:58.309 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:08:58.309 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:08:58.310 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:08:58.310 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T04:08:58.310 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:08:58.310 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:08:58.310 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:08:58.310 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:08:58.310 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T04:08:58.310 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T04:08:58.310 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:08:58.310 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:08:58.310 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:08:58.310 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:08:58.310 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:08:58.310 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:08:58.311 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:08:58.311 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:08:58.311 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:08:58.311 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:08:58.311 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:08:58.311 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T04:08:58.311 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:08:58.311 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:08:58.311 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:08:58.311 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:08:58.311 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T04:08:58.311 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:08:58.311 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:08:58.312 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:08:58.312 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:08:58.312 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T04:08:58.312 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T04:08:58.312 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:08:58.312 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:08:58.312 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:08:58.312 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:08:58.312 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:08:58.312 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:08:58.312 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:08:58.312 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:08:58.312 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:08:58.312 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:08:58.313 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:08:58.313 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T04:08:58.313 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:08:58.313 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:08:58.313 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:08:58.313 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:08:58.313 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T04:08:58.313 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:08:58.313 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:08:58.313 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:08:58.313 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:08:58.313 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T04:08:58.313 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T04:08:58.313 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:08:58.314 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:08:58.314 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:08:58.314 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:08:58.314 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:08:58.314 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:08:58.314 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T04:08:58.314 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T04:08:58.314 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:08:58.314 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:08:58.314 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:08:58.314 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:08:58.314 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:08:58.314 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:08:58.315 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T04:08:58.315 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:08:58.315 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:08:58.315 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:08:58.315 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 3 2024-09-16T04:08:58.315 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:08:58.315 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:08:58.315 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T04:08:58.315 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:08:58.315 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:08:58.315 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:08:58.315 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 3 2024-09-16T04:08:58.315 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:08:58.315 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:08:58.316 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:08:58.316 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T04:08:58.316 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T04:08:58.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.280+0000 7f8f100af640 10 librados: watch_flush enter 2024-09-16T04:08:58.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.280+0000 7f8f100af640 10 librados: watch_flush exit 2024-09-16T04:08:58.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.280+0000 7f8f100af640 20 client.1014302.objecter shutdown clearing up homeless session... 2024-09-16T04:08:58.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.280+0000 7f8f100af640 10 client.1014302.objecter successfully canceled tick 2024-09-16T04:08:58.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.280+0000 7f8f100af640 1 -- 172.21.15.42:0/1792073153 >> v2:172.21.15.154:6812/254845342 conn(0x7f8eec037810 msgr2=0x7f8eec039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:58.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.280+0000 7f8f100af640 1 --2- 172.21.15.42:0/1792073153 >> v2:172.21.15.154:6812/254845342 conn(0x7f8eec037810 0x7f8eec039cd0 secure :-1 s=READY pgs=3908 cs=0 l=1 rev1=1 crypto rx=0x7f8f040ad000 tx=0x7f8f04008040 comp rx=0 tx=0).stop 2024-09-16T04:08:58.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.280+0000 7f8f100af640 10 monclient: shutdown 2024-09-16T04:08:58.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.280+0000 7f8f100af640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:58.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.280+0000 7f8f100af640 1 -- 172.21.15.42:0/1792073153 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f8f080a4720 msgr2=0x7f8f081cf370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:58.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.280+0000 7f8f100af640 1 --2- 172.21.15.42:0/1792073153 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f8f080a4720 0x7f8f081cf370 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f8f00007f50 tx=0x7f8f0000e570 comp rx=0 tx=0).stop 2024-09-16T04:08:58.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.280+0000 7f8f0e64c640 1 -- 172.21.15.42:0/1792073153 reap_dead start 2024-09-16T04:08:58.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.280+0000 7f8f100af640 1 -- 172.21.15.42:0/1792073153 shutdown_connections 2024-09-16T04:08:58.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.280+0000 7f8f100af640 1 --2- 172.21.15.42:0/1792073153 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f8f080a4720 0x7f8f081cf370 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:08:58.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.280+0000 7f8f100af640 1 -- 172.21.15.42:0/1792073153 >> 172.21.15.42:0/1792073153 conn(0x7f8f080a6820 msgr2=0x7f8f080a0530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:58.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.280+0000 7f8f100af640 1 -- 172.21.15.42:0/1792073153 shutdown_connections 2024-09-16T04:08:58.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.280+0000 7f8f100af640 1 -- 172.21.15.42:0/1792073153 wait complete. 2024-09-16T04:08:58.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.280+0000 7f8f100af640 1 librados: shutdown 2024-09-16T04:08:58.319 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:08:58.319 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T04:08:58.319 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T04:08:58.438 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.432+0000 7f6b4343c640 10 monclient: get_monmap_and_config 2024-09-16T04:08:58.438 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.432+0000 7f6b4343c640 10 monclient: build_initial_monmap 2024-09-16T04:08:58.438 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.432+0000 7f6b4343c640 10 monclient: monmap: 2024-09-16T04:08:58.438 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:08:58.438 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:08:58.438 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:08:58.436408+0000 2024-09-16T04:08:58.438 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:08:58.436408+0000 2024-09-16T04:08:58.438 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:58.438 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:08:58.438 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:08:58.438 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:08:58.439 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:08:58.439 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:08:58.439 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:08:58.439 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:08:58.441 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.432+0000 7f6b4343c640 1 Processor -- start 2024-09-16T04:08:58.441 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.432+0000 7f6b4343c640 1 -- start start 2024-09-16T04:08:58.441 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.432+0000 7f6b4343c640 10 monclient: init 2024-09-16T04:08:58.441 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.432+0000 7f6b4343c640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:58.441 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.432+0000 7f6b4343c640 10 monclient: _add_conns ranks=[2,4,3,1,0] 2024-09-16T04:08:58.441 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.432+0000 7f6b4343c640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f6b3c06dc10 0x7f6b3c133970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:08:58.441 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.432+0000 7f6b4343c640 10 monclient(hunting): picked mon.noname-b con 0x7f6b3c06dc10 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:08:58.442 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.432+0000 7f6b4343c640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f6b3c0a38e0 0x7f6b3c0a3d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:08:58.442 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.432+0000 7f6b4343c640 10 monclient(hunting): picked mon.noname-c con 0x7f6b3c0a38e0 addr v2:172.21.15.154:3300/0 2024-09-16T04:08:58.442 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.432+0000 7f6b4343c640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f6b3c0a45b0 0x7f6b3c0a4a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:08:58.442 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.432+0000 7f6b4343c640 10 monclient(hunting): picked mon.noname-a con 0x7f6b3c0a45b0 addr v2:172.21.15.42:3300/0 2024-09-16T04:08:58.442 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.432+0000 7f6b4343c640 10 monclient(hunting): start opening mon connection 2024-09-16T04:08:58.442 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.432+0000 7f6b4343c640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f6b3c09be00 con 0x7f6b3c0a45b0 2024-09-16T04:08:58.442 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.432+0000 7f6b4343c640 10 monclient(hunting): start opening mon connection 2024-09-16T04:08:58.442 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.432+0000 7f6b4343c640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f6b3c09c350 con 0x7f6b3c0a38e0 2024-09-16T04:08:58.442 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.432+0000 7f6b4343c640 10 monclient(hunting): start opening mon connection 2024-09-16T04:08:58.442 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.432+0000 7f6b4343c640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f6b3c134870 con 0x7f6b3c06dc10 2024-09-16T04:08:58.442 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.432+0000 7f6b4343c640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:58.442 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.432+0000 7f6b4343c640 10 monclient(hunting): authenticate will time out at 2585.405832s 2024-09-16T04:08:58.442 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.432+0000 7f6b419d9640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f6b3c0a45b0 0x7f6b3c0a4a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:08:58.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.432+0000 7f6b411d8640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f6b3c06dc10 0x7f6b3c133970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:08:58.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.432+0000 7f6b419d9640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f6b3c0a45b0 0x7f6b3c0a4a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:50108/0 (socket says 172.21.15.42:50108) 2024-09-16T04:08:58.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.432+0000 7f6b419d9640 1 -- 172.21.15.42:0/605967658 learned_addr learned my addr 172.21.15.42:0/605967658 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:08:58.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.432+0000 7f6b419d9640 10 monclient(hunting): get_auth_request con 0x7f6b3c0a45b0 auth_method 0 2024-09-16T04:08:58.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.432+0000 7f6b419d9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:08:58.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.432+0000 7f6b419d9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:58.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.432+0000 7f6b419d9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:58.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.432+0000 7f6b411d8640 10 monclient(hunting): get_auth_request con 0x7f6b3c06dc10 auth_method 0 2024-09-16T04:08:58.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.432+0000 7f6b411d8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:08:58.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.432+0000 7f6b411d8640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:58.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.432+0000 7f6b411d8640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:58.443 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.432+0000 7f6b419d9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:08:58.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.432+0000 7f6b419d9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:08:58.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.432+0000 7f6b419d9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:08:58.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.432+0000 7f6b411d8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:08:58.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.432+0000 7f6b411d8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:08:58.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.432+0000 7f6b411d8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:08:58.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.432+0000 7f6b419d9640 10 monclient(hunting): handle_auth_done global_id 1034160 payload 274 2024-09-16T04:08:58.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.432+0000 7f6b419d9640 1 -- 172.21.15.42:0/605967658 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f6b3c06dc10 msgr2=0x7f6b3c133970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:08:58.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.432+0000 7f6b419d9640 1 --2- 172.21.15.42:0/605967658 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f6b3c06dc10 0x7f6b3c133970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:08:58.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.432+0000 7f6b419d9640 1 -- 172.21.15.42:0/605967658 >> v2:172.21.15.154:3300/0 conn(0x7f6b3c0a38e0 msgr2=0x7f6b3c0a3d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:08:58.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.432+0000 7f6b419d9640 1 --2- 172.21.15.42:0/605967658 >> v2:172.21.15.154:3300/0 conn(0x7f6b3c0a38e0 0x7f6b3c0a3d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:08:58.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.432+0000 7f6b419d9640 10 monclient: _finish_hunting 0 2024-09-16T04:08:58.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.432+0000 7f6b419d9640 1 monclient: found mon.noname-a 2024-09-16T04:08:58.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.432+0000 7f6b419d9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:58.445 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.432+0000 7f6b419d9640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:08:58.445 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.432+0000 7f6b419d9640 1 -- 172.21.15.42:0/605967658 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b3c09c070 con 0x7f6b3c0a45b0 2024-09-16T04:08:58.445 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.432+0000 7f6b411d8640 1 --2- 172.21.15.42:0/605967658 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f6b3c06dc10 0x7f6b3c133970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:08:58.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.436+0000 7f6b419d9640 1 --2- 172.21.15.42:0/605967658 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6b3c0a45b0 0x7f6b3c0a4a40 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f6b3c06e520 tx=0x7f6b3401c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c2a3d62419f8484 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:08:58.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.436+0000 7f6b33fff640 1 -- 172.21.15.42:0/605967658 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b340a55c0 con 0x7f6b3c0a45b0 2024-09-16T04:08:58.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.436+0000 7f6b33fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:58.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.436+0000 7f6b33fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:08:58.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.436+0000 7f6b33fff640 10 monclient: dump: 2024-09-16T04:08:58.452 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:08:58.452 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:08:58.452 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:08:58.452 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:08:58.452 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:58.452 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:08:58.453 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:08:58.453 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:08:58.453 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:08:58.453 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:08:58.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.436+0000 7f6b33fff640 10 monclient: _finish_auth 0 2024-09-16T04:08:58.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.436+0000 7f6b33fff640 10 monclient: _check_auth_tickets 2024-09-16T04:08:58.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.436+0000 7f6b33fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:58.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.436+0000 7f6b33fff640 1 -- 172.21.15.42:0/605967658 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6b340a5760 con 0x7f6b3c0a45b0 2024-09-16T04:08:58.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.436+0000 7f6b4343c640 5 monclient: authenticate success, global_id 1034160 2024-09-16T04:08:58.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.436+0000 7f6b4343c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:08:58.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.436+0000 7f6b33fff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:58.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.436+0000 7f6b33fff640 1 -- 172.21.15.42:0/605967658 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b340a5b30 con 0x7f6b3c0a45b0 2024-09-16T04:08:58.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.436+0000 7f6b33fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:58.454 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.436+0000 7f6b33fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:08:58.454 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.436+0000 7f6b33fff640 10 monclient: dump: 2024-09-16T04:08:58.454 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:08:58.454 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:08:58.454 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:08:58.454 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:08:58.454 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:58.454 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:08:58.454 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:08:58.454 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:08:58.454 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:08:58.454 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:08:58.454 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.436+0000 7f6b4343c640 10 monclient: get_monmap_and_config success 2024-09-16T04:08:58.454 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.436+0000 7f6b4343c640 4 set_mon_vals no callback set 2024-09-16T04:08:58.454 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.436+0000 7f6b4343c640 10 monclient: shutdown 2024-09-16T04:08:58.455 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.436+0000 7f6b4343c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:58.455 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.436+0000 7f6b4343c640 1 -- 172.21.15.42:0/605967658 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6b3c0a45b0 msgr2=0x7f6b3c0a4a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:58.455 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.436+0000 7f6b4343c640 1 --2- 172.21.15.42:0/605967658 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6b3c0a45b0 0x7f6b3c0a4a40 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f6b3c06e520 tx=0x7f6b3401c940 comp rx=0 tx=0).stop 2024-09-16T04:08:58.455 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.436+0000 7f6b4343c640 1 -- 172.21.15.42:0/605967658 shutdown_connections 2024-09-16T04:08:58.455 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.436+0000 7f6b4343c640 1 --2- 172.21.15.42:0/605967658 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6b3c0a45b0 0x7f6b3c0a4a40 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:08:58.455 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.436+0000 7f6b4343c640 1 --2- 172.21.15.42:0/605967658 >> v2:172.21.15.154:3300/0 conn(0x7f6b3c0a38e0 0x7f6b3c0a3d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:08:58.455 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.436+0000 7f6b4343c640 1 --2- 172.21.15.42:0/605967658 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f6b3c06dc10 0x7f6b3c133970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:08:58.455 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.436+0000 7f6b4343c640 1 -- 172.21.15.42:0/605967658 >> 172.21.15.42:0/605967658 conn(0x7f6b3c0a9fb0 msgr2=0x7f6b3c0aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:58.455 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.436+0000 7f6b4343c640 1 -- 172.21.15.42:0/605967658 shutdown_connections 2024-09-16T04:08:58.455 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.436+0000 7f6b4343c640 1 -- 172.21.15.42:0/605967658 wait complete. 2024-09-16T04:08:58.455 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.440+0000 7f6b4343c640 10 monclient: build_initial_monmap 2024-09-16T04:08:58.455 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.440+0000 7f6b4343c640 10 monclient: monmap: 2024-09-16T04:08:58.456 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:08:58.456 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:08:58.456 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:08:58.456 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:08:58.456 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:08:58.456 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:08:58.456 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:08:58.456 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:08:58.456 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:08:58.456 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:08:58.456 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.440+0000 7f6b4343c640 1 librados: starting msgr at 2024-09-16T04:08:58.456 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.440+0000 7f6b4343c640 1 librados: starting objecter 2024-09-16T04:08:58.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.440+0000 7f6b4343c640 1 Processor -- start 2024-09-16T04:08:58.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.440+0000 7f6b4343c640 1 -- start start 2024-09-16T04:08:58.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.440+0000 7f6b4343c640 1 librados: setting wanted keys 2024-09-16T04:08:58.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.440+0000 7f6b4343c640 1 librados: calling monclient init 2024-09-16T04:08:58.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.440+0000 7f6b4343c640 10 monclient: init 2024-09-16T04:08:58.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.440+0000 7f6b4343c640 10 monclient: _reopen_session rank -1 2024-09-16T04:08:58.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.440+0000 7f6b4343c640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:08:58.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.440+0000 7f6b4343c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6b3c06dc10 0x7f6b3c1cf4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:08:58.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.440+0000 7f6b4343c640 10 monclient(hunting): picked mon.noname-a con 0x7f6b3c06dc10 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:08:58.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.440+0000 7f6b4343c640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f6b3c0a38e0 0x7f6b3c1cfa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:08:58.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.440+0000 7f6b4343c640 10 monclient(hunting): picked mon.noname-c con 0x7f6b3c0a38e0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:08:58.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.440+0000 7f6b4343c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6b3c0a45b0 0x7f6b3c1cff50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:08:58.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.440+0000 7f6b4343c640 10 monclient(hunting): picked mon.noname-b con 0x7f6b3c0a45b0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:08:58.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.440+0000 7f6b4343c640 10 monclient(hunting): start opening mon connection 2024-09-16T04:08:58.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.440+0000 7f6b4343c640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f6b3c140ba0 con 0x7f6b3c06dc10 2024-09-16T04:08:58.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.440+0000 7f6b4343c640 10 monclient(hunting): start opening mon connection 2024-09-16T04:08:58.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.440+0000 7f6b4343c640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f6b3c140a20 con 0x7f6b3c0a45b0 2024-09-16T04:08:58.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.440+0000 7f6b4343c640 10 monclient(hunting): start opening mon connection 2024-09-16T04:08:58.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.440+0000 7f6b4343c640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f6b3c140d20 con 0x7f6b3c0a38e0 2024-09-16T04:08:58.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.440+0000 7f6b4343c640 10 monclient(hunting): _renew_subs 2024-09-16T04:08:58.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.440+0000 7f6b4343c640 10 monclient(hunting): authenticate will time out at 2585.413315s 2024-09-16T04:08:58.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.440+0000 7f6b409d7640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f6b3c0a38e0 0x7f6b3c1cfa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:08:58.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.440+0000 7f6b411d8640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6b3c06dc10 0x7f6b3c1cf4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:08:58.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.440+0000 7f6b411d8640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6b3c06dc10 0x7f6b3c1cf4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:50112/0 (socket says 172.21.15.42:50112) 2024-09-16T04:08:58.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.440+0000 7f6b409d7640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f6b3c0a38e0 0x7f6b3c1cfa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:40430/0 (socket says 172.21.15.42:40430) 2024-09-16T04:08:58.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.440+0000 7f6b411d8640 1 -- 172.21.15.42:0/1899202743 learned_addr learned my addr 172.21.15.42:0/1899202743 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:08:58.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.440+0000 7f6b411d8640 10 monclient(hunting): get_auth_request con 0x7f6b3c06dc10 auth_method 0 2024-09-16T04:08:58.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.440+0000 7f6b411d8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:08:58.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.440+0000 7f6b411d8640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:58.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.440+0000 7f6b411d8640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:58.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.440+0000 7f6b409d7640 10 monclient(hunting): get_auth_request con 0x7f6b3c0a38e0 auth_method 0 2024-09-16T04:08:58.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.440+0000 7f6b409d7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:08:58.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.440+0000 7f6b409d7640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:58.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.440+0000 7f6b409d7640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:58.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.440+0000 7f6b409d7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:08:58.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.440+0000 7f6b409d7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:08:58.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.440+0000 7f6b409d7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:08:58.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.440+0000 7f6b411d8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:08:58.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.440+0000 7f6b411d8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:08:58.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.440+0000 7f6b411d8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:08:58.460 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.440+0000 7f6b419d9640 1 --2- 172.21.15.42:0/1899202743 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6b3c0a45b0 0x7f6b3c1cff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:08:58.460 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.440+0000 7f6b419d9640 10 monclient(hunting): get_auth_request con 0x7f6b3c0a45b0 auth_method 0 2024-09-16T04:08:58.460 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.440+0000 7f6b419d9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:08:58.460 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.440+0000 7f6b419d9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:58.460 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.440+0000 7f6b419d9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:58.460 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.440+0000 7f6b409d7640 10 monclient(hunting): handle_auth_done global_id 1014314 payload 824 2024-09-16T04:08:58.460 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.440+0000 7f6b409d7640 1 -- 172.21.15.42:0/1899202743 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6b3c0a45b0 msgr2=0x7f6b3c1cff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:58.460 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.440+0000 7f6b409d7640 1 --2- 172.21.15.42:0/1899202743 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6b3c0a45b0 0x7f6b3c1cff50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:08:58.460 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.440+0000 7f6b409d7640 1 -- 172.21.15.42:0/1899202743 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6b3c06dc10 msgr2=0x7f6b3c1cf4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:58.460 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.440+0000 7f6b409d7640 1 --2- 172.21.15.42:0/1899202743 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6b3c06dc10 0x7f6b3c1cf4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:08:58.460 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.440+0000 7f6b409d7640 10 monclient: _finish_hunting 0 2024-09-16T04:08:58.460 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.440+0000 7f6b409d7640 1 monclient: found mon.noname-c 2024-09-16T04:08:58.460 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.440+0000 7f6b409d7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:08:58.461 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.440+0000 7f6b409d7640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T04:08:58.461 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.440+0000 7f6b409d7640 1 -- 172.21.15.42:0/1899202743 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b3c131bd0 con 0x7f6b3c0a38e0 2024-09-16T04:08:58.461 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.440+0000 7f6b409d7640 1 --2- 172.21.15.42:0/1899202743 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f6b3c0a38e0 0x7f6b3c1cfa10 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f6b2c00b3c0 tx=0x7f6b2c00b880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:08:58.461 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.440+0000 7f6b31ffb640 10 client.?.objecter ms_handle_connect 0x7f6b3c0a38e0 2024-09-16T04:08:58.461 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.440+0000 7f6b31ffb640 10 client.?.objecter resend_mon_ops 2024-09-16T04:08:58.461 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.440+0000 7f6b31ffb640 1 -- 172.21.15.42:0/1899202743 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b2c011830 con 0x7f6b3c0a38e0 2024-09-16T04:08:58.461 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.440+0000 7f6b31ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:58.461 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.440+0000 7f6b31ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:08:58.461 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.440+0000 7f6b31ffb640 10 monclient: dump: 2024-09-16T04:08:58.461 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:08:58.461 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:08:58.461 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:08:58.461 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:08:58.461 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:58.461 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:08:58.462 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:08:58.462 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:08:58.462 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:08:58.462 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:08:58.462 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.440+0000 7f6b31ffb640 10 monclient: _finish_auth 0 2024-09-16T04:08:58.462 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.440+0000 7f6b31ffb640 10 monclient: _check_auth_tickets 2024-09-16T04:08:58.462 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.440+0000 7f6b31ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:08:58.462 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.440+0000 7f6b31ffb640 1 -- 172.21.15.42:0/1899202743 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6b2c016070 con 0x7f6b3c0a38e0 2024-09-16T04:08:58.462 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.440+0000 7f6b4343c640 5 monclient: authenticate success, global_id 1014314 2024-09-16T04:08:58.462 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.440+0000 7f6b31ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:08:58.462 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.440+0000 7f6b31ffb640 1 -- 172.21.15.42:0/1899202743 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b2c011c30 con 0x7f6b3c0a38e0 2024-09-16T04:08:58.462 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.440+0000 7f6b4343c640 10 monclient: _renew_subs 2024-09-16T04:08:58.462 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.440+0000 7f6b4343c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:08:58.462 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.440+0000 7f6b4343c640 1 -- 172.21.15.42:0/1899202743 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b3c06f680 con 0x7f6b3c0a38e0 2024-09-16T04:08:58.463 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.440+0000 7f6b31ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:58.463 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.440+0000 7f6b31ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:08:58.463 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.440+0000 7f6b31ffb640 10 monclient: dump: 2024-09-16T04:08:58.463 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:08:58.463 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:08:58.463 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:08:58.463 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:08:58.463 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:58.463 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:08:58.463 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:08:58.463 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:08:58.463 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:08:58.463 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:08:58.463 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.440+0000 7f6b337fe640 4 set_mon_vals no callback set 2024-09-16T04:08:58.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.440+0000 7f6b4343c640 10 client.1014314.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:08:58.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.440+0000 7f6b4343c640 10 monclient: _renew_subs 2024-09-16T04:08:58.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.440+0000 7f6b4343c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:08:58.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.440+0000 7f6b4343c640 1 -- 172.21.15.42:0/1899202743 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b3c0a0a60 con 0x7f6b3c0a38e0 2024-09-16T04:08:58.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.440+0000 7f6b4343c640 1 librados: init done 2024-09-16T04:08:58.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.440+0000 7f6b4343c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:58.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.440+0000 7f6b4343c640 1 --2- 172.21.15.42:0/1899202743 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6b3c1cbfd0 0x7f6b3c1cc3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:08:58.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.440+0000 7f6b4343c640 10 monclient(hunting): start opening mon connection 2024-09-16T04:08:58.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.440+0000 7f6b4343c640 1 -- 172.21.15.42:0/1899202743 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f6b3c0a01c0 con 0x7f6b3c1cbfd0 2024-09-16T04:08:58.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.440+0000 7f6b31ffb640 1 -- 172.21.15.42:0/1899202743 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f6b2c01b460 con 0x7f6b3c0a38e0 2024-09-16T04:08:58.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.440+0000 7f6b411d8640 1 --2- 172.21.15.42:0/1899202743 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6b3c1cbfd0 0x7f6b3c1cc3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:08:58.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.440+0000 7f6b31ffb640 1 --2- 172.21.15.42:0/1899202743 >> v2:172.21.15.154:6812/254845342 conn(0x7f6b20037840 0x7f6b20039d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:08:58.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.440+0000 7f6b411d8640 10 monclient: get_auth_request con 0x7f6b3c1cbfd0 auth_method 0 2024-09-16T04:08:58.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.440+0000 7f6b411d8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:08:58.465 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.440+0000 7f6b411d8640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:58.465 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.440+0000 7f6b411d8640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:58.465 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.440+0000 7f6b31ffb640 1 -- 172.21.15.42:0/1899202743 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(706..706 src has 451..706) ==== 5040+0+0 (secure 0 0 0) 0x7f6b2c0487f0 con 0x7f6b3c0a38e0 2024-09-16T04:08:58.465 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.440+0000 7f6b31ffb640 10 client.1014314.objecter ms_dispatch 0x7f6b3c000ba0 osd_map(706..706 src has 451..706) 2024-09-16T04:08:58.465 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.440+0000 7f6b31ffb640 3 client.1014314.objecter handle_osd_map got epochs [706,706] > 0 2024-09-16T04:08:58.465 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.440+0000 7f6b31ffb640 3 client.1014314.objecter handle_osd_map decoding full epoch 706 2024-09-16T04:08:58.465 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.440+0000 7f6b411d8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:08:58.465 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.440+0000 7f6b411d8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:08:58.465 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.440+0000 7f6b411d8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:08:58.465 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.440+0000 7f6b31ffb640 20 client.1014314.objecter dump_active .. 0 homeless 2024-09-16T04:08:58.465 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.440+0000 7f6b419d9640 1 --2- 172.21.15.42:0/1899202743 >> v2:172.21.15.154:6812/254845342 conn(0x7f6b20037840 0x7f6b20039d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:08:58.465 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.440+0000 7f6b419d9640 10 monclient: get_auth_request con 0x7f6b20037840 auth_method 0 2024-09-16T04:08:58.465 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.444+0000 7f6b411d8640 10 monclient(hunting): handle_auth_done global_id 905209 payload 824 2024-09-16T04:08:58.466 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.444+0000 7f6b411d8640 1 -- 172.21.15.42:0/1899202743 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6b3c0a3d00 con 0x7f6b3c1cbfd0 2024-09-16T04:08:58.466 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.444+0000 7f6b411d8640 1 --2- 172.21.15.42:0/1899202743 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6b3c1cbfd0 0x7f6b3c1cc3b0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f6b38011c00 tx=0x7f6b3800f610 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:08:58.466 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.444+0000 7f6b31ffb640 10 client.1014314.objecter ms_handle_connect 0x7f6b3c1cbfd0 2024-09-16T04:08:58.466 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.444+0000 7f6b31ffb640 10 client.1014314.objecter resend_mon_ops 2024-09-16T04:08:58.466 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.444+0000 7f6b31ffb640 10 client.1014314.objecter ms_handle_connect 0x7f6b20037840 2024-09-16T04:08:58.466 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.444+0000 7f6b419d9640 1 --2- 172.21.15.42:0/1899202743 >> v2:172.21.15.154:6812/254845342 conn(0x7f6b20037840 0x7f6b20039d00 secure :-1 s=READY pgs=3909 cs=0 l=1 rev1=1 crypto rx=0x7f6b340ae000 tx=0x7f6b340a7070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:08:58.466 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.444+0000 7f6b31ffb640 1 -- 172.21.15.42:0/1899202743 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b38011cf0 con 0x7f6b3c1cbfd0 2024-09-16T04:08:58.466 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.444+0000 7f6b31ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:58.466 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.444+0000 7f6b31ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:08:58.466 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.444+0000 7f6b31ffb640 10 monclient: dump: 2024-09-16T04:08:58.466 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:08:58.466 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:08:58.466 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:08:58.466 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:08:58.467 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:58.467 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:08:58.467 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:08:58.467 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:08:58.467 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:08:58.467 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:08:58.467 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.444+0000 7f6b31ffb640 1 -- 172.21.15.42:0/1899202743 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f6b38011cf0 con 0x7f6b3c1cbfd0 2024-09-16T04:08:58.467 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.444+0000 7f6b31ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:08:58.467 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.444+0000 7f6b31ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:08:58.467 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.444+0000 7f6b31ffb640 1 -- 172.21.15.42:0/1899202743 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6b3c1cbfd0 msgr2=0x7f6b3c1cc3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:58.467 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.444+0000 7f6b31ffb640 1 --2- 172.21.15.42:0/1899202743 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6b3c1cbfd0 0x7f6b3c1cc3b0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f6b38011c00 tx=0x7f6b3800f610 comp rx=0 tx=0).stop 2024-09-16T04:08:58.467 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.444+0000 7f6b31ffb640 1 -- 172.21.15.42:0/1899202743 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6b3c1cbfd0 msgr2=0x7f6b3c1cc3b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:08:58.467 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.444+0000 7f6b31ffb640 1 --2- 172.21.15.42:0/1899202743 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6b3c1cbfd0 0x7f6b3c1cc3b0 secure :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f6b38011c00 tx=0x7f6b3800f610 comp rx=0 tx=0).stop 2024-09-16T04:08:58.469 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.448+0000 7f6b4343c640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:08:58.469 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.448+0000 7f6b4343c640 1 --2- 172.21.15.42:0/1899202743 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6b3c1cc8f0 0x7f6b3c12d010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:08:58.469 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.448+0000 7f6b4343c640 10 monclient(hunting): start opening mon connection 2024-09-16T04:08:58.469 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.448+0000 7f6b4343c640 1 -- 172.21.15.42:0/1899202743 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f6b3c0a0560 con 0x7f6b3c1cc8f0 2024-09-16T04:08:58.469 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.448+0000 7f6b411d8640 1 --2- 172.21.15.42:0/1899202743 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6b3c1cc8f0 0x7f6b3c12d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:08:58.469 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.452+0000 7f6b411d8640 10 monclient: get_auth_request con 0x7f6b3c1cc8f0 auth_method 0 2024-09-16T04:08:58.469 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.452+0000 7f6b411d8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:08:58.469 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.452+0000 7f6b411d8640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:08:58.469 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.452+0000 7f6b411d8640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:08:58.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.452+0000 7f6b411d8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:08:58.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.452+0000 7f6b411d8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:08:58.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.452+0000 7f6b411d8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:08:58.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.452+0000 7f6b411d8640 10 monclient(hunting): handle_auth_done global_id 905212 payload 824 2024-09-16T04:08:58.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.452+0000 7f6b411d8640 1 -- 172.21.15.42:0/1899202743 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f6b3c068e40 con 0x7f6b3c1cc8f0 2024-09-16T04:08:58.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.452+0000 7f6b411d8640 1 --2- 172.21.15.42:0/1899202743 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6b3c1cc8f0 0x7f6b3c12d010 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f6b38011c00 tx=0x7f6b380046d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:08:58.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.452+0000 7f6b31ffb640 10 client.1014314.objecter ms_handle_connect 0x7f6b3c1cc8f0 2024-09-16T04:08:58.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.452+0000 7f6b31ffb640 10 client.1014314.objecter resend_mon_ops 2024-09-16T04:08:58.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.452+0000 7f6b31ffb640 1 -- 172.21.15.42:0/1899202743 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b38004030 con 0x7f6b3c1cc8f0 2024-09-16T04:08:58.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.452+0000 7f6b31ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:08:58.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.452+0000 7f6b31ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:08:58.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.452+0000 7f6b31ffb640 10 monclient: dump: 2024-09-16T04:08:58.470 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:08:58.470 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:08:58.471 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:08:58.471 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:08:58.471 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:08:58.471 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:08:58.471 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:08:58.471 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:08:58.471 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:08:58.471 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:08:58.471 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.452+0000 7f6b31ffb640 1 -- 172.21.15.42:0/1899202743 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4880 (secure 0 0 0) 0x7f6b38004030 con 0x7f6b3c1cc8f0 2024-09-16T04:08:58.471 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.452+0000 7f6b31ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:08:58.471 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.452+0000 7f6b31ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:08:58.471 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.452+0000 7f6b31ffb640 1 -- 172.21.15.42:0/1899202743 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6b3c1cc8f0 msgr2=0x7f6b3c12d010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:58.471 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.452+0000 7f6b31ffb640 1 --2- 172.21.15.42:0/1899202743 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6b3c1cc8f0 0x7f6b3c12d010 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f6b38011c00 tx=0x7f6b380046d0 comp rx=0 tx=0).stop 2024-09-16T04:08:58.471 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.452+0000 7f6b31ffb640 1 -- 172.21.15.42:0/1899202743 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6b3c1cc8f0 msgr2=0x7f6b3c12d010 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:08:58.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.452+0000 7f6b31ffb640 1 --2- 172.21.15.42:0/1899202743 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6b3c1cc8f0 0x7f6b3c12d010 secure :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f6b38011c00 tx=0x7f6b380046d0 comp rx=0 tx=0).stop 2024-09-16T04:08:58.472 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T04:08:58.472 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:08:58.472 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:08:58.472 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T04:08:58.472 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 248, 2024-09-16T04:08:58.472 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 666631, 2024-09-16T04:08:58.472 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T04:08:58.472 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T04:08:58.472 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T04:08:58.473 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T04:08:58.473 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:08:58.473 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 3, 2024-09-16T04:08:58.473 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:08:58.473 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T04:08:58.473 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T04:08:58.473 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:08:58.473 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:08:58.473 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:08:58.473 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:08:58.473 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:08:58.473 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:08:58.474 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:08:58.474 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:08:58.474 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:08:58.474 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:08:58.474 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:08:58.474 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:08:58.474 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:08:58.474 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T04:08:58.474 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:08:58.474 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:08:58.474 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:08:58.474 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:08:58.474 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:08:58.474 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:08:58.475 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:08:58.475 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:08:58.475 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:08:58.475 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:08:58.475 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:08:58.475 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:08:58.475 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:08:58.475 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T04:08:58.475 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T04:08:58.475 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T04:08:58.475 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T04:08:58.475 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T04:08:58.475 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T04:08:58.475 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:08:58.476 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:08:58.476 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T04:08:58.476 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T04:08:58.476 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T04:08:58.476 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T04:08:58.476 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T04:08:58.476 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T04:08:58.476 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T04:08:58.476 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:08:58.476 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T04:08:58.476 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:08:58.476 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:08:58.476 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:08:58.477 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:08:58.477 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:08:58.477 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:08:58.477 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:08:58.477 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:08:58.477 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:08:58.477 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:08:58.477 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:08:58.477 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:08:58.477 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T04:08:58.477 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:08:58.477 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T04:08:58.477 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:08:58.477 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:08:58.478 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:08:58.478 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:08:58.478 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:08:58.478 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:08:58.478 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:08:58.478 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T04:08:58.478 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:08:58.478 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:08:58.478 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:08:58.478 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:08:58.478 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T04:08:58.478 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:08:58.478 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:08:58.478 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:08:58.479 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:08:58.479 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T04:08:58.479 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T04:08:58.479 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:08:58.479 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:08:58.479 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:08:58.479 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:08:58.479 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:08:58.479 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:08:58.479 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:08:58.479 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:08:58.479 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:08:58.479 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:08:58.479 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:08:58.480 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T04:08:58.480 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:08:58.480 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:08:58.480 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:08:58.480 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:08:58.480 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T04:08:58.480 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:08:58.480 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:08:58.480 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:08:58.480 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:08:58.480 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T04:08:58.480 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T04:08:58.480 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:08:58.481 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:08:58.481 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:08:58.481 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:08:58.481 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:08:58.481 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:08:58.481 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:08:58.481 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:08:58.481 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:08:58.481 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:08:58.481 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:08:58.481 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T04:08:58.481 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:08:58.481 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:08:58.481 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:08:58.482 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:08:58.482 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T04:08:58.482 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:08:58.482 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:08:58.482 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:08:58.482 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:08:58.482 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T04:08:58.482 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T04:08:58.482 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:08:58.482 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:08:58.482 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:08:58.482 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:08:58.482 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:08:58.482 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:08:58.483 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T04:08:58.483 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T04:08:58.483 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:08:58.483 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:08:58.483 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:08:58.483 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:08:58.483 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:08:58.483 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:08:58.483 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T04:08:58.483 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:08:58.483 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:08:58.483 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:08:58.483 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 3 2024-09-16T04:08:58.483 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:08:58.484 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:08:58.484 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T04:08:58.484 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:08:58.484 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:08:58.484 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:08:58.484 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 5 2024-09-16T04:08:58.484 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:08:58.484 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:08:58.484 INFO:teuthology.orchestra.run.smithi042.stdout: "mgr": [ 2024-09-16T04:08:58.484 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:08:58.484 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:08:58.484 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:08:58.484 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:08:58.484 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:08:58.485 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:08:58.485 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:08:58.485 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T04:08:58.485 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T04:08:58.486 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.452+0000 7f6b4343c640 10 librados: watch_flush enter 2024-09-16T04:08:58.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.452+0000 7f6b4343c640 10 librados: watch_flush exit 2024-09-16T04:08:58.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.452+0000 7f6b4343c640 20 client.1014314.objecter shutdown clearing up homeless session... 2024-09-16T04:08:58.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.452+0000 7f6b4343c640 10 client.1014314.objecter successfully canceled tick 2024-09-16T04:08:58.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.452+0000 7f6b4343c640 1 -- 172.21.15.42:0/1899202743 >> v2:172.21.15.154:6812/254845342 conn(0x7f6b20037840 msgr2=0x7f6b20039d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:58.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.452+0000 7f6b4343c640 1 --2- 172.21.15.42:0/1899202743 >> v2:172.21.15.154:6812/254845342 conn(0x7f6b20037840 0x7f6b20039d00 secure :-1 s=READY pgs=3909 cs=0 l=1 rev1=1 crypto rx=0x7f6b340ae000 tx=0x7f6b340a7070 comp rx=0 tx=0).stop 2024-09-16T04:08:58.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.452+0000 7f6b4343c640 10 monclient: shutdown 2024-09-16T04:08:58.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.452+0000 7f6b4343c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:08:58.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.452+0000 7f6b4343c640 1 -- 172.21.15.42:0/1899202743 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f6b3c0a38e0 msgr2=0x7f6b3c1cfa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:08:58.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.452+0000 7f6b4343c640 1 --2- 172.21.15.42:0/1899202743 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f6b3c0a38e0 0x7f6b3c1cfa10 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f6b2c00b3c0 tx=0x7f6b2c00b880 comp rx=0 tx=0).stop 2024-09-16T04:08:58.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.452+0000 7f6b419d9640 1 -- 172.21.15.42:0/1899202743 reap_dead start 2024-09-16T04:08:58.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.452+0000 7f6b419d9640 1 -- 172.21.15.42:0/1899202743 reap_dead start 2024-09-16T04:08:58.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.452+0000 7f6b4343c640 1 -- 172.21.15.42:0/1899202743 shutdown_connections 2024-09-16T04:08:58.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.452+0000 7f6b4343c640 1 -- 172.21.15.42:0/1899202743 >> 172.21.15.42:0/1899202743 conn(0x7f6b3c0a9fb0 msgr2=0x7f6b3c0a8840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:08:58.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.452+0000 7f6b4343c640 1 -- 172.21.15.42:0/1899202743 shutdown_connections 2024-09-16T04:08:58.488 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.452+0000 7f6b4343c640 1 -- 172.21.15.42:0/1899202743 wait complete. 2024-09-16T04:08:58.488 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:08:58.452+0000 7f6b4343c640 1 librados: shutdown 2024-09-16T04:08:58.488 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:08:58.488 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-09-16T04:08:58.488 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['c'] 2024-09-16T04:08:58.488 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-09-16T04:08:58.488 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-16T04:08:58.489 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-16T04:08:58.489 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T04:08:58.537 INFO:tasks.ceph.mon.b:Stopped 2024-09-16T04:08:58.537 INFO:tasks.mon_thrash:Sending STOP to mon c 2024-09-16T04:08:58.537 INFO:tasks.ceph.mon.c:Sent signal 19 2024-09-16T04:08:58.537 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-16T04:09:13.538 INFO:tasks.mon_thrash:Sending CONT to mon c 2024-09-16T04:09:13.539 INFO:tasks.ceph.mon.c:Sent signal 18 2024-09-16T04:09:13.539 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T04:09:13.540 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:09:13.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:13.668+0000 7f44008cb640 10 monclient: get_monmap_and_config 2024-09-16T04:09:13.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:13.668+0000 7f44008cb640 10 monclient: build_initial_monmap 2024-09-16T04:09:13.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:13.668+0000 7f44008cb640 10 monclient: monmap: 2024-09-16T04:09:13.675 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:09:13.675 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:09:13.675 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:09:13.673527+0000 2024-09-16T04:09:13.675 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:09:13.673527+0000 2024-09-16T04:09:13.675 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:13.675 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:09:13.675 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:09:13.676 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:09:13.676 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:09:13.676 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:09:13.676 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:09:13.676 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:09:13.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:13.668+0000 7f44008cb640 1 Processor -- start 2024-09-16T04:09:13.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:13.668+0000 7f44008cb640 1 -- start start 2024-09-16T04:09:13.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:13.668+0000 7f44008cb640 10 monclient: init 2024-09-16T04:09:13.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:13.668+0000 7f44008cb640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:13.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:13.668+0000 7f44008cb640 10 monclient: _add_conns ranks=[1,3,2,0,4] 2024-09-16T04:09:13.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:13.668+0000 7f44008cb640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f43fc0a1330 addr v1:172.21.15.154:6789/0 2024-09-16T04:09:13.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:13.668+0000 7f44008cb640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f43fc0a1710 0x7f43fc0a1b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:09:13.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:13.668+0000 7f44008cb640 10 monclient(hunting): picked mon.noname-a con 0x7f43fc0a1710 addr v2:172.21.15.42:3300/0 2024-09-16T04:09:13.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:13.668+0000 7f44008cb640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f43fc1393a0 0x7f43fc13b790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:09:13.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:13.668+0000 7f44008cb640 10 monclient(hunting): picked mon.noname-b con 0x7f43fc1393a0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:09:13.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:13.668+0000 7f44008cb640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f43fc09e4c0 con 0x7f43fc0a1330 2024-09-16T04:09:13.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:13.668+0000 7f44008cb640 10 monclient(hunting): start opening mon connection 2024-09-16T04:09:13.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:13.668+0000 7f44008cb640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f43fc09edf0 con 0x7f43fc0a1710 2024-09-16T04:09:13.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:13.668+0000 7f44008cb640 10 monclient(hunting): start opening mon connection 2024-09-16T04:09:13.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:13.668+0000 7f44008cb640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f43fc09cd60 con 0x7f43fc1393a0 2024-09-16T04:09:13.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:13.668+0000 7f44008cb640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:13.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:13.668+0000 7f44008cb640 10 monclient(hunting): authenticate will time out at 2600.643420s 2024-09-16T04:09:13.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:13.668+0000 7f43fa59c640 1 -- >> v1:172.21.15.154:6789/0 conn(0x7f43fc0a1330 legacy=0x7f43fc09f180 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.154:6789/0 2024-09-16T04:09:13.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:13.668+0000 7f43fad9d640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f43fc1393a0 0x7f43fc13b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:09:13.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:13.668+0000 7f43f9d9b640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f43fc0a1710 0x7f43fc0a1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:09:13.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:13.668+0000 7f43fad9d640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f43fc1393a0 0x7f43fc13b790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:58386/0 (socket says 172.21.15.42:58386) 2024-09-16T04:09:13.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:13.668+0000 7f43fad9d640 1 -- 172.21.15.42:0/2322582594 learned_addr learned my addr 172.21.15.42:0/2322582594 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:09:13.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:13.668+0000 7f43fad9d640 10 monclient(hunting): get_auth_request con 0x7f43fc1393a0 auth_method 0 2024-09-16T04:09:13.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:13.668+0000 7f43fad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:09:13.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:13.668+0000 7f43fad9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:13.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:13.668+0000 7f43fad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:13.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:13.668+0000 7f43f9d9b640 10 monclient(hunting): get_auth_request con 0x7f43fc0a1710 auth_method 0 2024-09-16T04:09:13.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:13.668+0000 7f43f9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:09:13.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:13.668+0000 7f43f9d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:13.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:13.668+0000 7f43f9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:13.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:13.668+0000 7f43fad9d640 1 -- 172.21.15.42:0/2322582594 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f43fc1393a0 msgr2=0x7f43fc13b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T04:09:13.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:13.668+0000 7f43fad9d640 1 -- 172.21.15.42:0/2322582594 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f43fc1393a0 msgr2=0x7f43fc13b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:09:13.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:13.668+0000 7f43fad9d640 1 --2- 172.21.15.42:0/2322582594 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f43fc1393a0 0x7f43fc13b790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:09:13.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:13.668+0000 7f43fad9d640 1 --2- 172.21.15.42:0/2322582594 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f43fc1393a0 0x7f43fc13b790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:09:13.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:13.668+0000 7f43f9d9b640 1 -- 172.21.15.42:0/2322582594 >> v2:172.21.15.42:3300/0 conn(0x7f43fc0a1710 msgr2=0x7f43fc0a1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:09:13.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:13.668+0000 7f43f9d9b640 1 -- 172.21.15.42:0/2322582594 >> v2:172.21.15.42:3300/0 conn(0x7f43fc0a1710 msgr2=0x7f43fc0a1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:09:13.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:13.668+0000 7f43f9d9b640 1 --2- 172.21.15.42:0/2322582594 >> v2:172.21.15.42:3300/0 conn(0x7f43fc0a1710 0x7f43fc0a1b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:09:13.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:13.668+0000 7f43f9d9b640 1 --2- 172.21.15.42:0/2322582594 >> v2:172.21.15.42:3300/0 conn(0x7f43fc0a1710 0x7f43fc0a1b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:09:13.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:13.872+0000 7f43fa59c640 1 -- 172.21.15.42:0/2322582594 >> v1:172.21.15.154:6789/0 conn(0x7f43fc0a1330 legacy=0x7f43fc09f180 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.154:6789/0 2024-09-16T04:09:13.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:13.872+0000 7f43fad9d640 1 --2- 172.21.15.42:0/2322582594 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f43fc1393a0 0x7f43fc13b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:09:13.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:13.872+0000 7f43f9d9b640 1 --2- 172.21.15.42:0/2322582594 >> v2:172.21.15.42:3300/0 conn(0x7f43fc0a1710 0x7f43fc0a1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:09:13.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:13.872+0000 7f43f9d9b640 10 monclient(hunting): get_auth_request con 0x7f43fc0a1710 auth_method 0 2024-09-16T04:09:13.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:13.872+0000 7f43f9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:09:13.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:13.872+0000 7f43f9d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:13.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:13.872+0000 7f43f9d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:09:13.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:13.872+0000 7f43fad9d640 10 monclient(hunting): get_auth_request con 0x7f43fc1393a0 auth_method 0 2024-09-16T04:09:13.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:13.872+0000 7f43fad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:09:13.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:13.872+0000 7f43fad9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:13.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:13.872+0000 7f43fad9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:09:13.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:13.872+0000 7f43f9d9b640 1 -- 172.21.15.42:0/2322582594 >> v2:172.21.15.42:3300/0 conn(0x7f43fc0a1710 msgr2=0x7f43fc0a1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:09:13.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:13.872+0000 7f43f9d9b640 1 -- 172.21.15.42:0/2322582594 >> v2:172.21.15.42:3300/0 conn(0x7f43fc0a1710 msgr2=0x7f43fc0a1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:09:13.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:13.872+0000 7f43fad9d640 1 -- 172.21.15.42:0/2322582594 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f43fc1393a0 msgr2=0x7f43fc13b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T04:09:13.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:13.872+0000 7f43f9d9b640 1 --2- 172.21.15.42:0/2322582594 >> v2:172.21.15.42:3300/0 conn(0x7f43fc0a1710 0x7f43fc0a1b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:09:13.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:13.872+0000 7f43fad9d640 1 -- 172.21.15.42:0/2322582594 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f43fc1393a0 msgr2=0x7f43fc13b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:09:13.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:13.872+0000 7f43fad9d640 1 --2- 172.21.15.42:0/2322582594 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f43fc1393a0 0x7f43fc13b790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:09:13.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:13.872+0000 7f43f9d9b640 1 --2- 172.21.15.42:0/2322582594 >> v2:172.21.15.42:3300/0 conn(0x7f43fc0a1710 0x7f43fc0a1b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T04:09:13.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:13.872+0000 7f43fad9d640 1 --2- 172.21.15.42:0/2322582594 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f43fc1393a0 0x7f43fc13b790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T04:09:14.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:14.276+0000 7f43f9d9b640 1 --2- 172.21.15.42:0/2322582594 >> v2:172.21.15.42:3300/0 conn(0x7f43fc0a1710 0x7f43fc0a1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:09:14.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:14.276+0000 7f43fa59c640 1 -- 172.21.15.42:0/2322582594 >> v1:172.21.15.154:6789/0 conn(0x7f43fc0a1330 legacy=0x7f43fc09f180 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.154:6789/0 2024-09-16T04:09:14.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:14.276+0000 7f43fad9d640 1 --2- 172.21.15.42:0/2322582594 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f43fc1393a0 0x7f43fc13b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:09:14.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:14.276+0000 7f43fad9d640 10 monclient(hunting): get_auth_request con 0x7f43fc1393a0 auth_method 0 2024-09-16T04:09:14.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:14.276+0000 7f43fad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:09:14.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:14.276+0000 7f43fad9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:14.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:14.276+0000 7f43fad9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:09:14.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:14.276+0000 7f43f9d9b640 10 monclient(hunting): get_auth_request con 0x7f43fc0a1710 auth_method 0 2024-09-16T04:09:14.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:14.276+0000 7f43f9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:09:14.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:14.276+0000 7f43f9d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:14.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:14.276+0000 7f43f9d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:09:14.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:14.276+0000 7f43f9d9b640 1 -- 172.21.15.42:0/2322582594 >> v2:172.21.15.42:3300/0 conn(0x7f43fc0a1710 msgr2=0x7f43fc0a1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T04:09:14.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:14.276+0000 7f43f9d9b640 1 -- 172.21.15.42:0/2322582594 >> v2:172.21.15.42:3300/0 conn(0x7f43fc0a1710 msgr2=0x7f43fc0a1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:09:14.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:14.276+0000 7f43f9d9b640 1 --2- 172.21.15.42:0/2322582594 >> v2:172.21.15.42:3300/0 conn(0x7f43fc0a1710 0x7f43fc0a1b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:09:14.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:14.276+0000 7f43fad9d640 1 -- 172.21.15.42:0/2322582594 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f43fc1393a0 msgr2=0x7f43fc13b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:09:14.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:14.276+0000 7f43fad9d640 1 -- 172.21.15.42:0/2322582594 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f43fc1393a0 msgr2=0x7f43fc13b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:09:14.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:14.276+0000 7f43fad9d640 1 --2- 172.21.15.42:0/2322582594 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f43fc1393a0 0x7f43fc13b790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:09:14.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:14.276+0000 7f43f9d9b640 1 --2- 172.21.15.42:0/2322582594 >> v2:172.21.15.42:3300/0 conn(0x7f43fc0a1710 0x7f43fc0a1b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T04:09:14.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:14.276+0000 7f43fad9d640 1 --2- 172.21.15.42:0/2322582594 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f43fc1393a0 0x7f43fc13b790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T04:09:15.083 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:15.080+0000 7f43fa59c640 1 -- 172.21.15.42:0/2322582594 >> v1:172.21.15.154:6789/0 conn(0x7f43fc0a1330 legacy=0x7f43fc09f180 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.154:6789/0 2024-09-16T04:09:15.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:15.080+0000 7f43fad9d640 1 --2- 172.21.15.42:0/2322582594 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f43fc1393a0 0x7f43fc13b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:09:15.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:15.080+0000 7f43fad9d640 10 monclient(hunting): get_auth_request con 0x7f43fc1393a0 auth_method 0 2024-09-16T04:09:15.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:15.080+0000 7f43fad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:09:15.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:15.080+0000 7f43fad9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:15.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:15.080+0000 7f43fad9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:09:15.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:15.080+0000 7f43f9d9b640 1 --2- 172.21.15.42:0/2322582594 >> v2:172.21.15.42:3300/0 conn(0x7f43fc0a1710 0x7f43fc0a1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:09:15.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:15.080+0000 7f43fad9d640 1 -- 172.21.15.42:0/2322582594 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f43fc1393a0 msgr2=0x7f43fc13b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:09:15.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:15.080+0000 7f43fad9d640 1 -- 172.21.15.42:0/2322582594 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f43fc1393a0 msgr2=0x7f43fc13b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:09:15.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:15.080+0000 7f43fad9d640 1 --2- 172.21.15.42:0/2322582594 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f43fc1393a0 0x7f43fc13b790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:09:15.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:15.080+0000 7f43f9d9b640 10 monclient(hunting): get_auth_request con 0x7f43fc0a1710 auth_method 0 2024-09-16T04:09:15.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:15.080+0000 7f43f9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:09:15.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:15.080+0000 7f43f9d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:15.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:15.080+0000 7f43f9d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:09:15.085 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:15.080+0000 7f43fad9d640 1 --2- 172.21.15.42:0/2322582594 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f43fc1393a0 0x7f43fc13b790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T04:09:15.085 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:15.080+0000 7f43f9d9b640 1 -- 172.21.15.42:0/2322582594 >> v2:172.21.15.42:3300/0 conn(0x7f43fc0a1710 msgr2=0x7f43fc0a1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:09:15.085 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:15.080+0000 7f43f9d9b640 1 -- 172.21.15.42:0/2322582594 >> v2:172.21.15.42:3300/0 conn(0x7f43fc0a1710 msgr2=0x7f43fc0a1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:09:15.085 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:15.080+0000 7f43f9d9b640 1 --2- 172.21.15.42:0/2322582594 >> v2:172.21.15.42:3300/0 conn(0x7f43fc0a1710 0x7f43fc0a1b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:09:15.085 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:15.080+0000 7f43f9d9b640 1 --2- 172.21.15.42:0/2322582594 >> v2:172.21.15.42:3300/0 conn(0x7f43fc0a1710 0x7f43fc0a1b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T04:09:16.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:16.668+0000 7f43f3fff640 10 monclient(hunting): tick 2024-09-16T04:09:16.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:16.668+0000 7f43f3fff640 10 monclient(hunting): _check_auth_tickets 2024-09-16T04:09:16.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:16.668+0000 7f43f3fff640 1 monclient(hunting): continuing hunt 2024-09-16T04:09:16.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:16.668+0000 7f43f3fff640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T04:09:16.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:16.668+0000 7f43f3fff640 1 -- 172.21.15.42:0/2322582594 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f43fc1393a0 msgr2=0x7f43fc13b790 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:09:16.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:16.668+0000 7f43f3fff640 1 --2- 172.21.15.42:0/2322582594 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f43fc1393a0 0x7f43fc13b790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:09:16.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:16.668+0000 7f43f3fff640 1 -- 172.21.15.42:0/2322582594 >> v2:172.21.15.42:3300/0 conn(0x7f43fc0a1710 msgr2=0x7f43fc0a1b40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:09:16.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:16.668+0000 7f43f3fff640 1 --2- 172.21.15.42:0/2322582594 >> v2:172.21.15.42:3300/0 conn(0x7f43fc0a1710 0x7f43fc0a1b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:09:16.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:16.668+0000 7f43f3fff640 1 -- 172.21.15.42:0/2322582594 >> v1:172.21.15.154:6789/0 conn(0x7f43fc0a1330 legacy=0x7f43fc09f180 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:09:16.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:16.668+0000 7f43f3fff640 10 monclient: _add_conns ranks=[4,0] 2024-09-16T04:09:16.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:16.668+0000 7f43f3fff640 1 --2- 172.21.15.42:0/2322582594 >> v2:172.21.15.154:3300/0 conn(0x7f43dc0036b0 0x7f43dc005b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:09:16.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:16.668+0000 7f43f3fff640 10 monclient(hunting): picked mon.noname-c con 0x7f43dc0036b0 addr v2:172.21.15.154:3300/0 2024-09-16T04:09:16.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:16.668+0000 7f43f3fff640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f43dc006200 addr v1:172.21.15.42:6789/0 2024-09-16T04:09:16.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:16.668+0000 7f43f3fff640 1 -- 172.21.15.42:0/2322582594 --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f43fc0a2080 con 0x7f43dc006200 2024-09-16T04:09:16.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:16.668+0000 7f43f3fff640 10 monclient(hunting): start opening mon connection 2024-09-16T04:09:16.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:16.668+0000 7f43f3fff640 1 -- 172.21.15.42:0/2322582594 --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f43fc09edf0 con 0x7f43dc0036b0 2024-09-16T04:09:16.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:16.668+0000 7f43f3fff640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:16.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:16.672+0000 7f43fa59c640 1 -- 172.21.15.42:0/2322582594 >> v2:172.21.15.154:3300/0 conn(0x7f43dc0036b0 msgr2=0x7f43dc005b10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T04:09:16.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:16.672+0000 7f43fa59c640 1 --2- 172.21.15.42:0/2322582594 >> v2:172.21.15.154:3300/0 conn(0x7f43dc0036b0 0x7f43dc005b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:09:16.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:16.876+0000 7f43fa59c640 1 -- 172.21.15.42:0/2322582594 >> v2:172.21.15.154:3300/0 conn(0x7f43dc0036b0 msgr2=0x7f43dc005b10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T04:09:16.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:16.876+0000 7f43fa59c640 1 --2- 172.21.15.42:0/2322582594 >> v2:172.21.15.154:3300/0 conn(0x7f43dc0036b0 0x7f43dc005b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T04:09:17.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:17.280+0000 7f43fa59c640 1 -- 172.21.15.42:0/2322582594 >> v2:172.21.15.154:3300/0 conn(0x7f43dc0036b0 msgr2=0x7f43dc005b10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T04:09:17.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:17.280+0000 7f43fa59c640 1 --2- 172.21.15.42:0/2322582594 >> v2:172.21.15.154:3300/0 conn(0x7f43dc0036b0 0x7f43dc005b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T04:09:18.087 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.084+0000 7f43fa59c640 1 -- 172.21.15.42:0/2322582594 >> v2:172.21.15.154:3300/0 conn(0x7f43dc0036b0 msgr2=0x7f43dc005b10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T04:09:18.087 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.084+0000 7f43fa59c640 1 --2- 172.21.15.42:0/2322582594 >> v2:172.21.15.154:3300/0 conn(0x7f43dc0036b0 0x7f43dc005b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T04:09:18.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.600+0000 7f43f959a640 1 -- 172.21.15.42:0/2322582594 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2600662862 0 0) 0x7f43fc0a2080 con 0x7f43dc006200 2024-09-16T04:09:18.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.600+0000 7f43f959a640 10 monclient(hunting): _negotiate 2024-09-16T04:09:18.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.600+0000 7f43f959a640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:18.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.600+0000 7f43f959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:18.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.600+0000 7f43f959a640 10 monclient(hunting): my global_id is 1034166 2024-09-16T04:09:18.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.600+0000 7f43f959a640 1 -- 172.21.15.42:0/2322582594 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f43e8003530 con 0x7f43dc006200 2024-09-16T04:09:18.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.604+0000 7f43f959a640 1 -- 172.21.15.42:0/2322582594 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1639701304 0 0) 0x7f43e8003530 con 0x7f43dc006200 2024-09-16T04:09:18.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.604+0000 7f43f959a640 1 -- 172.21.15.42:0/2322582594 >> v2:172.21.15.154:3300/0 conn(0x7f43dc0036b0 msgr2=0x7f43dc005b10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:09:18.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.604+0000 7f43f959a640 1 --2- 172.21.15.42:0/2322582594 >> v2:172.21.15.154:3300/0 conn(0x7f43dc0036b0 0x7f43dc005b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:09:18.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.604+0000 7f43f959a640 10 monclient: _finish_hunting 0 2024-09-16T04:09:18.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.604+0000 7f43f959a640 1 monclient: found mon.noname-a-legacy 2024-09-16T04:09:18.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.604+0000 7f43f959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:18.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.604+0000 7f43f959a640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.42:6789/0 2024-09-16T04:09:18.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.604+0000 7f43f959a640 1 -- 172.21.15.42:0/2322582594 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43fc09e4c0 con 0x7f43dc006200 2024-09-16T04:09:18.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.604+0000 7f43f959a640 10 monclient: _finish_auth 0 2024-09-16T04:09:18.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.604+0000 7f43f959a640 10 monclient: _check_auth_tickets 2024-09-16T04:09:18.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.604+0000 7f43f959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:18.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.604+0000 7f43f959a640 1 -- 172.21.15.42:0/2322582594 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2177413734 0 0) 0x7f43ec00f040 con 0x7f43dc006200 2024-09-16T04:09:18.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.604+0000 7f43f959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:18.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.604+0000 7f43f959a640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T04:09:18.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.604+0000 7f43f959a640 10 monclient: dump: 2024-09-16T04:09:18.614 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:09:18.614 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:09:18.614 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:09:18.614 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:09:18.615 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:18.615 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:09:18.615 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:09:18.615 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:09:18.615 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:09:18.615 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:09:18.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.604+0000 7f43f959a640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] but i'm connected to v1:172.21.15.42:6789/0, reconnecting 2024-09-16T04:09:18.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.604+0000 7f43f959a640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:18.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.604+0000 7f43f959a640 1 -- 172.21.15.42:0/2322582594 >> v1:172.21.15.42:6789/0 conn(0x7f43dc006200 legacy=0x7f43dc0086c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:18.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.604+0000 7f43fad9d640 1 -- 172.21.15.42:0/2322582594 reap_dead start 2024-09-16T04:09:18.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.604+0000 7f43f959a640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:09:18.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.604+0000 7f43f959a640 1 --2- 172.21.15.42:0/2322582594 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f43e80039c0 0x7f43e8003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:09:18.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.604+0000 7f43f959a640 10 monclient(hunting): picked mon.b con 0x7f43e80039c0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:09:18.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.604+0000 7f43f959a640 1 --2- 172.21.15.42:0/2322582594 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f43e8004810 0x7f43e8004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:09:18.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.604+0000 7f43f959a640 10 monclient(hunting): picked mon.a con 0x7f43e8004810 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:09:18.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.604+0000 7f43f959a640 1 --2- 172.21.15.42:0/2322582594 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f43e80052f0 0x7f43e800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:09:18.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.604+0000 7f43f959a640 10 monclient(hunting): picked mon.c con 0x7f43e80052f0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:09:18.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.604+0000 7f43f959a640 10 monclient(hunting): start opening mon connection 2024-09-16T04:09:18.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.604+0000 7f43f959a640 1 -- 172.21.15.42:0/2322582594 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f43fc09edf0 con 0x7f43e8004810 2024-09-16T04:09:18.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.604+0000 7f43f959a640 10 monclient(hunting): start opening mon connection 2024-09-16T04:09:18.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.604+0000 7f43f959a640 1 -- 172.21.15.42:0/2322582594 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f43e800c240 con 0x7f43e80039c0 2024-09-16T04:09:18.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.604+0000 7f43f959a640 10 monclient(hunting): start opening mon connection 2024-09-16T04:09:18.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.604+0000 7f43f959a640 1 -- 172.21.15.42:0/2322582594 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f43e800c440 con 0x7f43e80052f0 2024-09-16T04:09:18.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.604+0000 7f43f959a640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:18.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.604+0000 7f43f959a640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:09:18.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.604+0000 7f43fa59c640 1 -- 172.21.15.42:0/2322582594 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f43e80039c0 msgr2=0x7f43e8003dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T04:09:18.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.604+0000 7f43fad9d640 1 --2- 172.21.15.42:0/2322582594 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f43e80052f0 0x7f43e800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:09:18.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.604+0000 7f43f9d9b640 1 --2- 172.21.15.42:0/2322582594 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f43e8004810 0x7f43e8004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:09:18.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.604+0000 7f43fa59c640 1 --2- 172.21.15.42:0/2322582594 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f43e80039c0 0x7f43e8003dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:09:18.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.604+0000 7f43fad9d640 10 monclient(hunting): get_auth_request con 0x7f43e80052f0 auth_method 0 2024-09-16T04:09:18.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.604+0000 7f43fad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:09:18.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.604+0000 7f43fad9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:18.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.604+0000 7f43fad9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:09:18.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.604+0000 7f43f9d9b640 10 monclient(hunting): get_auth_request con 0x7f43e8004810 auth_method 0 2024-09-16T04:09:18.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.604+0000 7f43f9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:09:18.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.604+0000 7f43f9d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:18.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.604+0000 7f43f9d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:09:18.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.604+0000 7f43fad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:09:18.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.604+0000 7f43fad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:09:18.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.604+0000 7f43fad9d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:09:18.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.604+0000 7f43f9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:09:18.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.604+0000 7f43f9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:09:18.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.604+0000 7f43f9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:09:18.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.604+0000 7f43fad9d640 10 monclient(hunting): handle_auth_done global_id 1034166 payload 293 2024-09-16T04:09:18.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.604+0000 7f43fad9d640 1 -- 172.21.15.42:0/2322582594 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f43e80039c0 msgr2=0x7f43e8003dc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:09:18.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.604+0000 7f43fad9d640 1 --2- 172.21.15.42:0/2322582594 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f43e80039c0 0x7f43e8003dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:09:18.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.604+0000 7f43fad9d640 1 -- 172.21.15.42:0/2322582594 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f43e8004810 msgr2=0x7f43e8004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:09:18.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.604+0000 7f43fad9d640 1 --2- 172.21.15.42:0/2322582594 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f43e8004810 0x7f43e8004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:09:18.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.604+0000 7f43fad9d640 10 monclient: _finish_hunting 0 2024-09-16T04:09:18.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.604+0000 7f43fad9d640 1 monclient: found mon.c 2024-09-16T04:09:18.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.604+0000 7f43fad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:18.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.604+0000 7f43fad9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:09:18.619 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.604+0000 7f43fad9d640 1 -- 172.21.15.42:0/2322582594 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43e8003530 con 0x7f43e80052f0 2024-09-16T04:09:18.619 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.604+0000 7f43fad9d640 10 monclient: _finish_auth 0 2024-09-16T04:09:18.619 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.604+0000 7f43fad9d640 10 monclient: _check_auth_tickets 2024-09-16T04:09:18.619 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.604+0000 7f43fad9d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:18.619 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.604+0000 7f44008cb640 5 monclient: authenticate success, global_id 1034166 2024-09-16T04:09:18.619 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.604+0000 7f44008cb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:09:18.620 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.612+0000 7f43fad9d640 1 --2- 172.21.15.42:0/2322582594 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f43e80052f0 0x7f43e800b9d0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f43fc09d3b0 tx=0x7f43e401f2a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e1bc7b8f33b99e05 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:09:18.620 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.612+0000 7f43f959a640 1 -- 172.21.15.42:0/2322582594 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43e40b5070 con 0x7f43e80052f0 2024-09-16T04:09:18.621 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.612+0000 7f43f959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:18.621 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.612+0000 7f43f959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:09:18.621 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.612+0000 7f43f959a640 10 monclient: dump: 2024-09-16T04:09:18.621 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:09:18.621 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:09:18.621 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:09:18.621 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:09:18.621 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:18.621 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:09:18.621 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:09:18.621 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:09:18.621 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:09:18.622 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:09:18.622 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.612+0000 7f44008cb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:09:18.622 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.612+0000 7f43f959a640 1 -- 172.21.15.42:0/2322582594 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f43e40b0410 con 0x7f43e80052f0 2024-09-16T04:09:18.622 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.612+0000 7f43f959a640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:18.622 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.612+0000 7f44008cb640 10 monclient: get_monmap_and_config success 2024-09-16T04:09:18.622 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.612+0000 7f44008cb640 4 set_mon_vals no callback set 2024-09-16T04:09:18.622 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.612+0000 7f43f959a640 1 -- 172.21.15.42:0/2322582594 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43e400b430 con 0x7f43e80052f0 2024-09-16T04:09:18.622 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.612+0000 7f43f959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:18.622 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.612+0000 7f43f959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:09:18.622 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.612+0000 7f43f959a640 10 monclient: dump: 2024-09-16T04:09:18.622 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:09:18.622 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:09:18.622 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:09:18.623 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:09:18.623 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:18.623 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:09:18.623 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:09:18.623 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:09:18.623 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:09:18.623 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:09:18.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.612+0000 7f44008cb640 10 monclient: shutdown 2024-09-16T04:09:18.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.612+0000 7f44008cb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:18.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.612+0000 7f44008cb640 1 -- 172.21.15.42:0/2322582594 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f43e80052f0 msgr2=0x7f43e800b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:18.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.612+0000 7f44008cb640 1 --2- 172.21.15.42:0/2322582594 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f43e80052f0 0x7f43e800b9d0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f43fc09d3b0 tx=0x7f43e401f2a0 comp rx=0 tx=0).stop 2024-09-16T04:09:18.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.612+0000 7f44008cb640 1 -- 172.21.15.42:0/2322582594 shutdown_connections 2024-09-16T04:09:18.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.612+0000 7f44008cb640 1 --2- 172.21.15.42:0/2322582594 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f43e80052f0 0x7f43e800b9d0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:09:18.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.612+0000 7f44008cb640 1 --2- 172.21.15.42:0/2322582594 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f43e8004810 0x7f43e8004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:09:18.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.612+0000 7f44008cb640 1 --2- 172.21.15.42:0/2322582594 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f43e80039c0 0x7f43e8003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:09:18.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.612+0000 7f44008cb640 1 -- 172.21.15.42:0/2322582594 >> 172.21.15.42:0/2322582594 conn(0x7f43fc0a2420 msgr2=0x7f43fc0a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:18.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.612+0000 7f44008cb640 1 -- 172.21.15.42:0/2322582594 shutdown_connections 2024-09-16T04:09:18.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.612+0000 7f44008cb640 1 -- 172.21.15.42:0/2322582594 wait complete. 2024-09-16T04:09:18.628 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.616+0000 7f44008cb640 10 monclient: build_initial_monmap 2024-09-16T04:09:18.628 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.616+0000 7f44008cb640 10 monclient: monmap: 2024-09-16T04:09:18.629 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:09:18.629 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:09:18.629 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:09:18.629 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:09:18.629 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:18.629 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:09:18.629 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:09:18.629 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:09:18.629 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:09:18.629 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:09:18.629 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.616+0000 7f44008cb640 1 librados: starting msgr at 2024-09-16T04:09:18.629 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.616+0000 7f44008cb640 1 librados: starting objecter 2024-09-16T04:09:18.629 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.616+0000 7f44008cb640 1 Processor -- start 2024-09-16T04:09:18.629 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.616+0000 7f44008cb640 1 -- start start 2024-09-16T04:09:18.630 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.616+0000 7f44008cb640 1 librados: setting wanted keys 2024-09-16T04:09:18.630 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.616+0000 7f44008cb640 1 librados: calling monclient init 2024-09-16T04:09:18.630 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.616+0000 7f44008cb640 10 monclient: init 2024-09-16T04:09:18.630 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.616+0000 7f44008cb640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:18.630 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.616+0000 7f44008cb640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:09:18.630 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.616+0000 7f44008cb640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f43e80039c0 0x7f43fc1ce980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:09:18.630 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.616+0000 7f44008cb640 10 monclient(hunting): picked mon.noname-a con 0x7f43e80039c0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:09:18.630 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.616+0000 7f44008cb640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f43e8004810 0x7f43fc133570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:09:18.630 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.616+0000 7f44008cb640 10 monclient(hunting): picked mon.noname-c con 0x7f43e8004810 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:09:18.630 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.616+0000 7f44008cb640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f43e80052f0 0x7f43fc133ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:09:18.631 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.616+0000 7f44008cb640 10 monclient(hunting): picked mon.noname-b con 0x7f43e80052f0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:09:18.631 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.616+0000 7f44008cb640 10 monclient(hunting): start opening mon connection 2024-09-16T04:09:18.631 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.616+0000 7f44008cb640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f43fc1340f0 con 0x7f43e80039c0 2024-09-16T04:09:18.631 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.616+0000 7f44008cb640 10 monclient(hunting): start opening mon connection 2024-09-16T04:09:18.631 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.616+0000 7f44008cb640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f43fc0a1f00 con 0x7f43e80052f0 2024-09-16T04:09:18.631 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.616+0000 7f44008cb640 10 monclient(hunting): start opening mon connection 2024-09-16T04:09:18.631 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.616+0000 7f44008cb640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f43fc134270 con 0x7f43e8004810 2024-09-16T04:09:18.631 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.616+0000 7f44008cb640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:18.631 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.616+0000 7f44008cb640 10 monclient(hunting): authenticate will time out at 2605.591180s 2024-09-16T04:09:18.631 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.616+0000 7f43fa59c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f43e80039c0 0x7f43fc1ce980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:09:18.631 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.616+0000 7f43f9d9b640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f43e8004810 0x7f43fc133570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:09:18.632 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.616+0000 7f43f9d9b640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f43e8004810 0x7f43fc133570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:41344/0 (socket says 172.21.15.42:41344) 2024-09-16T04:09:18.632 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.616+0000 7f43f9d9b640 1 -- 172.21.15.42:0/1616486968 learned_addr learned my addr 172.21.15.42:0/1616486968 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:09:18.632 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.616+0000 7f43f9d9b640 10 monclient(hunting): get_auth_request con 0x7f43e8004810 auth_method 0 2024-09-16T04:09:18.632 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.616+0000 7f43f9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:09:18.632 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.616+0000 7f43f9d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:18.632 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.616+0000 7f43f9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:18.632 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.616+0000 7f43fa59c640 10 monclient(hunting): get_auth_request con 0x7f43e80039c0 auth_method 0 2024-09-16T04:09:18.632 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.616+0000 7f43fa59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:09:18.632 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.616+0000 7f43fa59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:18.632 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.616+0000 7f43fad9d640 1 -- 172.21.15.42:0/1616486968 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f43e80052f0 msgr2=0x7f43fc133ab0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T04:09:18.633 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.616+0000 7f43fa59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:18.633 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.616+0000 7f43fad9d640 1 --2- 172.21.15.42:0/1616486968 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f43e80052f0 0x7f43fc133ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:09:18.633 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.616+0000 7f43f9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:09:18.633 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.616+0000 7f43f9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:09:18.633 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.616+0000 7f43f9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:09:18.633 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.616+0000 7f43fa59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:09:18.633 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.616+0000 7f43fa59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:09:18.633 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.616+0000 7f43fa59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:09:18.634 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.616+0000 7f43fa59c640 10 monclient(hunting): handle_auth_done global_id 1034169 payload 824 2024-09-16T04:09:18.634 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.616+0000 7f43fa59c640 1 -- 172.21.15.42:0/1616486968 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f43e8004810 msgr2=0x7f43fc133570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:18.634 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.616+0000 7f43fa59c640 1 --2- 172.21.15.42:0/1616486968 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f43e8004810 0x7f43fc133570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:09:18.634 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.616+0000 7f43fa59c640 1 -- 172.21.15.42:0/1616486968 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f43e80052f0 msgr2=0x7f43fc133ab0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:09:18.634 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.616+0000 7f43fa59c640 1 --2- 172.21.15.42:0/1616486968 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f43e80052f0 0x7f43fc133ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:09:18.634 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.616+0000 7f43fa59c640 10 monclient: _finish_hunting 0 2024-09-16T04:09:18.634 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.616+0000 7f43fa59c640 1 monclient: found mon.noname-a 2024-09-16T04:09:18.634 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.616+0000 7f43fa59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:18.634 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.616+0000 7f43fa59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:09:18.634 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.616+0000 7f43fa59c640 1 -- 172.21.15.42:0/1616486968 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43fc1259c0 con 0x7f43e80039c0 2024-09-16T04:09:18.634 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.620+0000 7f43f9d9b640 1 --2- 172.21.15.42:0/1616486968 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f43e8004810 0x7f43fc133570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:09:18.634 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.620+0000 7f43fa59c640 1 --2- 172.21.15.42:0/1616486968 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f43e80039c0 0x7f43fc1ce980 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f43f4003be0 tx=0x7f43f400bc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:09:18.634 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.620+0000 7f43f2ffd640 10 client.?.objecter ms_handle_connect 0x7f43e80039c0 2024-09-16T04:09:18.635 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.620+0000 7f43f2ffd640 10 client.?.objecter resend_mon_ops 2024-09-16T04:09:18.635 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.620+0000 7f43f2ffd640 1 -- 172.21.15.42:0/1616486968 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43f400cad0 con 0x7f43e80039c0 2024-09-16T04:09:18.635 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.620+0000 7f43f2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:18.635 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.620+0000 7f43f2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:09:18.635 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.620+0000 7f43f2ffd640 10 monclient: dump: 2024-09-16T04:09:18.635 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:09:18.635 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:09:18.635 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:09:18.635 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:09:18.635 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:18.635 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:09:18.635 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:09:18.635 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:09:18.635 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:09:18.635 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:09:18.636 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.620+0000 7f43f2ffd640 10 monclient: _finish_auth 0 2024-09-16T04:09:18.636 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.620+0000 7f43f2ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:09:18.636 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.620+0000 7f43f2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:18.636 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.620+0000 7f43f2ffd640 1 -- 172.21.15.42:0/1616486968 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f43f4010030 con 0x7f43e80039c0 2024-09-16T04:09:18.636 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.620+0000 7f43f2ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:18.636 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.620+0000 7f43f2ffd640 1 -- 172.21.15.42:0/1616486968 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43f4016430 con 0x7f43e80039c0 2024-09-16T04:09:18.636 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.620+0000 7f43f2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:18.636 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.620+0000 7f43f2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:09:18.636 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.620+0000 7f43f2ffd640 10 monclient: dump: 2024-09-16T04:09:18.637 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:09:18.637 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:09:18.637 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:09:18.637 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:09:18.637 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:18.637 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:09:18.637 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:09:18.637 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:09:18.637 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:09:18.637 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:09:18.638 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.620+0000 7f44008cb640 5 monclient: authenticate success, global_id 1034169 2024-09-16T04:09:18.638 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.620+0000 7f44008cb640 10 monclient: _renew_subs 2024-09-16T04:09:18.638 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.620+0000 7f43f8d99640 4 set_mon_vals no callback set 2024-09-16T04:09:18.638 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.620+0000 7f44008cb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:09:18.638 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.620+0000 7f44008cb640 1 -- 172.21.15.42:0/1616486968 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43fc124130 con 0x7f43e80039c0 2024-09-16T04:09:18.638 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.620+0000 7f44008cb640 10 client.1034169.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:09:18.638 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.620+0000 7f44008cb640 10 monclient: _renew_subs 2024-09-16T04:09:18.638 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.620+0000 7f44008cb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:09:18.638 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.620+0000 7f44008cb640 1 -- 172.21.15.42:0/1616486968 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43fc0a5610 con 0x7f43e80039c0 2024-09-16T04:09:18.638 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.620+0000 7f44008cb640 1 librados: init done 2024-09-16T04:09:18.638 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.620+0000 7f43f2ffd640 1 -- 172.21.15.42:0/1616486968 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f43f40153e0 con 0x7f43e80039c0 2024-09-16T04:09:18.638 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.620+0000 7f44008cb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:18.638 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.620+0000 7f44008cb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:18.638 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.620+0000 7f44008cb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:09:18.639 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.620+0000 7f44008cb640 1 -- 172.21.15.42:0/1616486968 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f43fc134600 con 0x7f43e80039c0 2024-09-16T04:09:18.639 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.620+0000 7f43f2ffd640 1 --2- 172.21.15.42:0/1616486968 >> v2:172.21.15.154:6812/254845342 conn(0x7f43dc037e10 0x7f43dc03a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:09:18.639 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.620+0000 7f43f2ffd640 1 -- 172.21.15.42:0/1616486968 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(706..706 src has 451..706) ==== 5040+0+0 (secure 0 0 0) 0x7f43f4048950 con 0x7f43e80039c0 2024-09-16T04:09:18.639 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.620+0000 7f43f2ffd640 10 client.1034169.objecter ms_dispatch 0x7f43fc000ba0 osd_map(706..706 src has 451..706) 2024-09-16T04:09:18.639 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.620+0000 7f43f2ffd640 3 client.1034169.objecter handle_osd_map got epochs [706,706] > 0 2024-09-16T04:09:18.639 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.620+0000 7f43f2ffd640 3 client.1034169.objecter handle_osd_map decoding full epoch 706 2024-09-16T04:09:18.639 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.620+0000 7f43f2ffd640 20 client.1034169.objecter dump_active .. 0 homeless 2024-09-16T04:09:18.639 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.620+0000 7f43f9d9b640 1 --2- 172.21.15.42:0/1616486968 >> v2:172.21.15.154:6812/254845342 conn(0x7f43dc037e10 0x7f43dc03a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:09:18.639 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.620+0000 7f43f9d9b640 10 monclient: get_auth_request con 0x7f43dc037e10 auth_method 0 2024-09-16T04:09:18.639 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.620+0000 7f43f9d9b640 1 --2- 172.21.15.42:0/1616486968 >> v2:172.21.15.154:6812/254845342 conn(0x7f43dc037e10 0x7f43dc03a2d0 secure :-1 s=READY pgs=3911 cs=0 l=1 rev1=1 crypto rx=0x7f43ec00a010 tx=0x7f43ec010040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:09:18.639 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.620+0000 7f43f2ffd640 10 client.1034169.objecter ms_handle_connect 0x7f43dc037e10 2024-09-16T04:09:18.640 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.624+0000 7f43f2ffd640 1 -- 172.21.15.42:0/1616486968 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f43f40156d0 con 0x7f43e80039c0 2024-09-16T04:09:18.640 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.624+0000 7f43f2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:18.640 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.624+0000 7f43f2ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:09:18.860 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.852+0000 7f44008cb640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:09:18.861 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.852+0000 7f44008cb640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:09:18.861 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.852+0000 7f44008cb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:09:18.861 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.852+0000 7f44008cb640 1 -- 172.21.15.42:0/1616486968 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f43fc1ceec0 con 0x7f43e80039c0 2024-09-16T04:09:18.861 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.852+0000 7f43f2ffd640 1 -- 172.21.15.42:0/1616486968 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1555 (secure 0 0 0) 0x7f43f4015910 con 0x7f43e80039c0 2024-09-16T04:09:18.861 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.852+0000 7f43f2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:09:18.861 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.852+0000 7f43f2ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:09:18.862 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":250,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:09:18.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.856+0000 7f44008cb640 10 librados: watch_flush enter 2024-09-16T04:09:18.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.856+0000 7f44008cb640 10 librados: watch_flush exit 2024-09-16T04:09:18.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.856+0000 7f44008cb640 20 client.1034169.objecter shutdown clearing up homeless session... 2024-09-16T04:09:18.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.856+0000 7f44008cb640 10 client.1034169.objecter successfully canceled tick 2024-09-16T04:09:18.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.856+0000 7f44008cb640 1 -- 172.21.15.42:0/1616486968 >> v2:172.21.15.154:6812/254845342 conn(0x7f43dc037e10 msgr2=0x7f43dc03a2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:18.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.856+0000 7f44008cb640 1 --2- 172.21.15.42:0/1616486968 >> v2:172.21.15.154:6812/254845342 conn(0x7f43dc037e10 0x7f43dc03a2d0 secure :-1 s=READY pgs=3911 cs=0 l=1 rev1=1 crypto rx=0x7f43ec00a010 tx=0x7f43ec010040 comp rx=0 tx=0).stop 2024-09-16T04:09:18.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.856+0000 7f44008cb640 10 monclient: shutdown 2024-09-16T04:09:18.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.856+0000 7f44008cb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:18.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.856+0000 7f44008cb640 1 -- 172.21.15.42:0/1616486968 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f43e80039c0 msgr2=0x7f43fc1ce980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:18.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.856+0000 7f44008cb640 1 --2- 172.21.15.42:0/1616486968 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f43e80039c0 0x7f43fc1ce980 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f43f4003be0 tx=0x7f43f400bc70 comp rx=0 tx=0).stop 2024-09-16T04:09:18.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.856+0000 7f44008cb640 1 -- 172.21.15.42:0/1616486968 shutdown_connections 2024-09-16T04:09:18.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.856+0000 7f44008cb640 1 --2- 172.21.15.42:0/1616486968 >> v2:172.21.15.154:6812/254845342 conn(0x7f43dc037e10 0x7f43dc03a2d0 unknown :-1 s=CLOSED pgs=3911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:09:18.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.856+0000 7f44008cb640 1 --2- 172.21.15.42:0/1616486968 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f43e80052f0 0x7f43fc133ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:09:18.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.856+0000 7f44008cb640 1 --2- 172.21.15.42:0/1616486968 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f43e8004810 0x7f43fc133570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:09:18.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.856+0000 7f44008cb640 1 --2- 172.21.15.42:0/1616486968 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f43e80039c0 0x7f43fc1ce980 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:09:18.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.856+0000 7f44008cb640 1 -- 172.21.15.42:0/1616486968 >> 172.21.15.42:0/1616486968 conn(0x7f43fc0a2420 msgr2=0x7f43fc0a41b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:18.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.856+0000 7f44008cb640 1 -- 172.21.15.42:0/1616486968 shutdown_connections 2024-09-16T04:09:18.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.856+0000 7f44008cb640 1 -- 172.21.15.42:0/1616486968 wait complete. 2024-09-16T04:09:18.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:18.856+0000 7f44008cb640 1 librados: shutdown 2024-09-16T04:09:18.880 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T04:09:18.880 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-16T04:09:18.881 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T04:09:19.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.000+0000 7f6485608640 10 monclient: get_monmap_and_config 2024-09-16T04:09:19.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.000+0000 7f6485608640 10 monclient: build_initial_monmap 2024-09-16T04:09:19.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.000+0000 7f6485608640 10 monclient: monmap: 2024-09-16T04:09:19.010 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:09:19.010 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:09:19.010 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:09:19.008056+0000 2024-09-16T04:09:19.010 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:09:19.008056+0000 2024-09-16T04:09:19.010 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:19.010 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:09:19.010 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:09:19.010 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:09:19.010 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:09:19.010 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:09:19.010 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:09:19.010 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:09:19.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.004+0000 7f6485608640 1 Processor -- start 2024-09-16T04:09:19.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.004+0000 7f6485608640 1 -- start start 2024-09-16T04:09:19.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.004+0000 7f6485608640 10 monclient: init 2024-09-16T04:09:19.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.004+0000 7f6485608640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:19.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.004+0000 7f6485608640 10 monclient: _add_conns ranks=[3,2,4,1,0] 2024-09-16T04:09:19.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.004+0000 7f6485608640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f64800a1330 0x7f64800a1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:09:19.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.004+0000 7f6485608640 10 monclient(hunting): picked mon.noname-a con 0x7f64800a1330 addr v2:172.21.15.42:3300/0 2024-09-16T04:09:19.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.004+0000 7f6485608640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f648006dac0 0x7f64800a1c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:09:19.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.004+0000 7f6485608640 10 monclient(hunting): picked mon.noname-b con 0x7f648006dac0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:09:19.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.004+0000 7f6485608640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f6480135100 0x7f6480135550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:09:19.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.004+0000 7f6485608640 10 monclient(hunting): picked mon.noname-c con 0x7f6480135100 addr v2:172.21.15.154:3300/0 2024-09-16T04:09:19.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.004+0000 7f6485608640 10 monclient(hunting): start opening mon connection 2024-09-16T04:09:19.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.004+0000 7f6485608640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f648009c7d0 con 0x7f64800a1330 2024-09-16T04:09:19.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.004+0000 7f6485608640 10 monclient(hunting): start opening mon connection 2024-09-16T04:09:19.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.004+0000 7f6485608640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f648009c090 con 0x7f6480135100 2024-09-16T04:09:19.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.004+0000 7f6485608640 10 monclient(hunting): start opening mon connection 2024-09-16T04:09:19.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.004+0000 7f6485608640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f648009f610 con 0x7f648006dac0 2024-09-16T04:09:19.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.004+0000 7f6485608640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:19.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.004+0000 7f6485608640 10 monclient(hunting): authenticate will time out at 2605.977625s 2024-09-16T04:09:19.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.004+0000 7f647effd640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f64800a1330 0x7f64800a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:09:19.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.004+0000 7f647e7fc640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f648006dac0 0x7f64800a1c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:09:19.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.004+0000 7f647e7fc640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f648006dac0 0x7f64800a1c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:41360/0 (socket says 172.21.15.42:41360) 2024-09-16T04:09:19.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.004+0000 7f647e7fc640 1 -- 172.21.15.42:0/1610712216 learned_addr learned my addr 172.21.15.42:0/1610712216 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:09:19.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.004+0000 7f647effd640 10 monclient(hunting): get_auth_request con 0x7f64800a1330 auth_method 0 2024-09-16T04:09:19.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.004+0000 7f647effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:09:19.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.004+0000 7f647effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:19.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.004+0000 7f647effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:19.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.004+0000 7f647e7fc640 10 monclient(hunting): get_auth_request con 0x7f648006dac0 auth_method 0 2024-09-16T04:09:19.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.004+0000 7f647e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:09:19.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.004+0000 7f647e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:19.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.004+0000 7f647e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:19.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.004+0000 7f647effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:09:19.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.004+0000 7f647effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:09:19.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.004+0000 7f647effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:09:19.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.004+0000 7f647e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:09:19.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.004+0000 7f647e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:09:19.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.004+0000 7f647e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:09:19.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.004+0000 7f647f7fe640 1 -- 172.21.15.42:0/1610712216 >> v2:172.21.15.154:3300/0 conn(0x7f6480135100 msgr2=0x7f6480135550 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T04:09:19.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.004+0000 7f647effd640 10 monclient(hunting): handle_auth_done global_id 1034172 payload 274 2024-09-16T04:09:19.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.004+0000 7f647f7fe640 1 --2- 172.21.15.42:0/1610712216 >> v2:172.21.15.154:3300/0 conn(0x7f6480135100 0x7f6480135550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:09:19.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.004+0000 7f647effd640 1 -- 172.21.15.42:0/1610712216 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f648006dac0 msgr2=0x7f64800a1c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:09:19.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.004+0000 7f647effd640 1 --2- 172.21.15.42:0/1610712216 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f648006dac0 0x7f64800a1c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:09:19.016 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.004+0000 7f647effd640 1 -- 172.21.15.42:0/1610712216 >> v2:172.21.15.154:3300/0 conn(0x7f6480135100 msgr2=0x7f6480135550 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:09:19.016 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.004+0000 7f647effd640 1 --2- 172.21.15.42:0/1610712216 >> v2:172.21.15.154:3300/0 conn(0x7f6480135100 0x7f6480135550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:09:19.016 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.004+0000 7f647effd640 10 monclient: _finish_hunting 0 2024-09-16T04:09:19.016 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.004+0000 7f647effd640 1 monclient: found mon.noname-a 2024-09-16T04:09:19.016 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.004+0000 7f647effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:19.016 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.004+0000 7f647effd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:09:19.016 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.004+0000 7f647effd640 1 -- 172.21.15.42:0/1610712216 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f648009f8f0 con 0x7f64800a1330 2024-09-16T04:09:19.016 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.004+0000 7f647e7fc640 1 --2- 172.21.15.42:0/1610712216 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f648006dac0 0x7f64800a1c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:09:19.022 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.008+0000 7f647effd640 1 --2- 172.21.15.42:0/1610712216 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f64800a1330 0x7f64800a1710 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f648009bb00 tx=0x7f647001c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=baf7f7db6f85c4ec server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:09:19.022 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.008+0000 7f647dffb640 1 -- 172.21.15.42:0/1610712216 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64700b0070 con 0x7f64800a1330 2024-09-16T04:09:19.023 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.008+0000 7f647dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:19.023 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.008+0000 7f647dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:09:19.023 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.008+0000 7f647dffb640 10 monclient: dump: 2024-09-16T04:09:19.023 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:09:19.023 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:09:19.023 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:09:19.023 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:09:19.023 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:19.023 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:09:19.023 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:09:19.023 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:09:19.023 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:09:19.023 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:09:19.023 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.008+0000 7f647dffb640 10 monclient: _finish_auth 0 2024-09-16T04:09:19.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.008+0000 7f647dffb640 10 monclient: _check_auth_tickets 2024-09-16T04:09:19.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.008+0000 7f647dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:19.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.008+0000 7f647dffb640 1 -- 172.21.15.42:0/1610712216 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f64700ab600 con 0x7f64800a1330 2024-09-16T04:09:19.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.008+0000 7f647dffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:19.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.008+0000 7f647dffb640 1 -- 172.21.15.42:0/1610712216 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64700ab9d0 con 0x7f64800a1330 2024-09-16T04:09:19.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.008+0000 7f6485608640 5 monclient: authenticate success, global_id 1034172 2024-09-16T04:09:19.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.008+0000 7f6485608640 10 monclient: get_monmap_and_config success 2024-09-16T04:09:19.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.008+0000 7f6485608640 4 set_mon_vals no callback set 2024-09-16T04:09:19.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.008+0000 7f647dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:19.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.008+0000 7f647dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:09:19.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.008+0000 7f647dffb640 10 monclient: dump: 2024-09-16T04:09:19.024 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:09:19.024 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:09:19.025 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:09:19.025 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:09:19.025 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:19.025 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:09:19.025 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:09:19.025 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:09:19.025 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:09:19.025 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:09:19.025 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.008+0000 7f6485608640 10 monclient: shutdown 2024-09-16T04:09:19.025 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.008+0000 7f6485608640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:19.025 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.008+0000 7f6485608640 1 -- 172.21.15.42:0/1610712216 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f64800a1330 msgr2=0x7f64800a1710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:19.025 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.008+0000 7f6485608640 1 --2- 172.21.15.42:0/1610712216 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f64800a1330 0x7f64800a1710 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f648009bb00 tx=0x7f647001c480 comp rx=0 tx=0).stop 2024-09-16T04:09:19.025 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.008+0000 7f6485608640 1 -- 172.21.15.42:0/1610712216 shutdown_connections 2024-09-16T04:09:19.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.008+0000 7f6485608640 1 --2- 172.21.15.42:0/1610712216 >> v2:172.21.15.154:3300/0 conn(0x7f6480135100 0x7f6480135550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:09:19.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.008+0000 7f6485608640 1 --2- 172.21.15.42:0/1610712216 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f648006dac0 0x7f64800a1c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:09:19.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.008+0000 7f6485608640 1 --2- 172.21.15.42:0/1610712216 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f64800a1330 0x7f64800a1710 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:09:19.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.008+0000 7f6485608640 1 -- 172.21.15.42:0/1610712216 >> 172.21.15.42:0/1610712216 conn(0x7f64800a2420 msgr2=0x7f64800a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:19.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.008+0000 7f6485608640 1 -- 172.21.15.42:0/1610712216 shutdown_connections 2024-09-16T04:09:19.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.008+0000 7f6485608640 1 -- 172.21.15.42:0/1610712216 wait complete. 2024-09-16T04:09:19.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.008+0000 7f6485608640 10 monclient: build_initial_monmap 2024-09-16T04:09:19.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.008+0000 7f6485608640 10 monclient: monmap: 2024-09-16T04:09:19.026 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:09:19.026 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:09:19.026 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:09:19.026 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:09:19.026 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:19.026 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:09:19.027 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:09:19.027 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:09:19.027 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:09:19.027 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:09:19.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.012+0000 7f6485608640 1 librados: starting msgr at 2024-09-16T04:09:19.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.012+0000 7f6485608640 1 librados: starting objecter 2024-09-16T04:09:19.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.012+0000 7f6485608640 1 Processor -- start 2024-09-16T04:09:19.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.012+0000 7f6485608640 1 -- start start 2024-09-16T04:09:19.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.012+0000 7f6485608640 1 librados: setting wanted keys 2024-09-16T04:09:19.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.012+0000 7f6485608640 1 librados: calling monclient init 2024-09-16T04:09:19.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.012+0000 7f6485608640 10 monclient: init 2024-09-16T04:09:19.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.012+0000 7f6485608640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:19.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.012+0000 7f6485608640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:09:19.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.012+0000 7f6485608640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f64800a1330 0x7f6480139bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:09:19.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.012+0000 7f6485608640 10 monclient(hunting): picked mon.noname-c con 0x7f64800a1330 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:09:19.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.012+0000 7f6485608640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6480135100 0x7f648013a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:09:19.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.012+0000 7f6485608640 10 monclient(hunting): picked mon.noname-a con 0x7f6480135100 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:09:19.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.012+0000 7f6485608640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f648013a650 0x7f64801d5c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:09:19.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.012+0000 7f6485608640 10 monclient(hunting): picked mon.noname-b con 0x7f648013a650 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:09:19.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.012+0000 7f6485608640 10 monclient(hunting): start opening mon connection 2024-09-16T04:09:19.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.012+0000 7f6485608640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f648009eef0 con 0x7f6480135100 2024-09-16T04:09:19.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.012+0000 7f6485608640 10 monclient(hunting): start opening mon connection 2024-09-16T04:09:19.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.012+0000 7f6485608640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f648009ed70 con 0x7f648013a650 2024-09-16T04:09:19.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.012+0000 7f6485608640 10 monclient(hunting): start opening mon connection 2024-09-16T04:09:19.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.012+0000 7f6485608640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f648013eb70 con 0x7f64800a1330 2024-09-16T04:09:19.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.012+0000 7f6485608640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:19.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.012+0000 7f6485608640 10 monclient(hunting): authenticate will time out at 2605.984937s 2024-09-16T04:09:19.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.012+0000 7f647effd640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f64800a1330 0x7f6480139bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:09:19.029 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.012+0000 7f647effd640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f64800a1330 0x7f6480139bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:41366/0 (socket says 172.21.15.42:41366) 2024-09-16T04:09:19.029 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.012+0000 7f647e7fc640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6480135100 0x7f648013a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:09:19.029 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.012+0000 7f647effd640 1 -- 172.21.15.42:0/908268144 learned_addr learned my addr 172.21.15.42:0/908268144 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:09:19.029 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.012+0000 7f647effd640 10 monclient(hunting): get_auth_request con 0x7f64800a1330 auth_method 0 2024-09-16T04:09:19.029 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.012+0000 7f647effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:09:19.029 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.012+0000 7f647effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:19.029 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.012+0000 7f647effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:19.029 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.012+0000 7f647e7fc640 10 monclient(hunting): get_auth_request con 0x7f6480135100 auth_method 0 2024-09-16T04:09:19.029 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.012+0000 7f647f7fe640 1 -- 172.21.15.42:0/908268144 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f648013a650 msgr2=0x7f64801d5c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T04:09:19.030 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.012+0000 7f647e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:09:19.030 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.012+0000 7f647e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:19.030 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.012+0000 7f647e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:19.030 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.012+0000 7f647f7fe640 1 --2- 172.21.15.42:0/908268144 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f648013a650 0x7f64801d5c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:09:19.030 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.012+0000 7f647effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:09:19.030 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.012+0000 7f647effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:09:19.030 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.012+0000 7f647effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:09:19.030 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.012+0000 7f647e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:09:19.030 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.012+0000 7f647e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:09:19.030 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.012+0000 7f647e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:09:19.030 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.012+0000 7f647effd640 10 monclient(hunting): handle_auth_done global_id 1014323 payload 824 2024-09-16T04:09:19.030 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.012+0000 7f647effd640 1 -- 172.21.15.42:0/908268144 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f648013a650 msgr2=0x7f64801d5c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:09:19.030 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.012+0000 7f647effd640 1 --2- 172.21.15.42:0/908268144 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f648013a650 0x7f64801d5c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:09:19.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.012+0000 7f647effd640 1 -- 172.21.15.42:0/908268144 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6480135100 msgr2=0x7f648013a110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:19.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.012+0000 7f647effd640 1 --2- 172.21.15.42:0/908268144 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6480135100 0x7f648013a110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:09:19.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.012+0000 7f647effd640 10 monclient: _finish_hunting 0 2024-09-16T04:09:19.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.012+0000 7f647effd640 1 monclient: found mon.noname-c 2024-09-16T04:09:19.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.012+0000 7f647effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:19.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.012+0000 7f647effd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T04:09:19.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.012+0000 7f647effd640 1 -- 172.21.15.42:0/908268144 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f648009e920 con 0x7f64800a1330 2024-09-16T04:09:19.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.012+0000 7f647effd640 1 --2- 172.21.15.42:0/908268144 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f64800a1330 0x7f6480139bd0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f647001ab30 tx=0x7f64800a22d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:09:19.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.012+0000 7f64677fe640 10 client.?.objecter ms_handle_connect 0x7f64800a1330 2024-09-16T04:09:19.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.012+0000 7f64677fe640 10 client.?.objecter resend_mon_ops 2024-09-16T04:09:19.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.012+0000 7f64677fe640 1 -- 172.21.15.42:0/908268144 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64700b0040 con 0x7f64800a1330 2024-09-16T04:09:19.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.012+0000 7f64677fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:19.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.012+0000 7f64677fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:09:19.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.012+0000 7f64677fe640 10 monclient: dump: 2024-09-16T04:09:19.032 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:09:19.032 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:09:19.032 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:09:19.032 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:09:19.032 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:19.032 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:09:19.032 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:09:19.032 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:09:19.032 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:09:19.032 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:09:19.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.012+0000 7f64677fe640 10 monclient: _finish_auth 0 2024-09-16T04:09:19.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.012+0000 7f64677fe640 10 monclient: _check_auth_tickets 2024-09-16T04:09:19.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.012+0000 7f64677fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:19.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.012+0000 7f64677fe640 1 -- 172.21.15.42:0/908268144 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f64700b9070 con 0x7f64800a1330 2024-09-16T04:09:19.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.012+0000 7f6485608640 5 monclient: authenticate success, global_id 1014323 2024-09-16T04:09:19.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.012+0000 7f6485608640 10 monclient: _renew_subs 2024-09-16T04:09:19.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.012+0000 7f6485608640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:09:19.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.012+0000 7f6485608640 1 -- 172.21.15.42:0/908268144 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6480120520 con 0x7f64800a1330 2024-09-16T04:09:19.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.012+0000 7f64677fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:19.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.012+0000 7f64677fe640 1 -- 172.21.15.42:0/908268144 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64700b4650 con 0x7f64800a1330 2024-09-16T04:09:19.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.012+0000 7f64677fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:19.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.012+0000 7f64677fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:09:19.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.012+0000 7f64677fe640 10 monclient: dump: 2024-09-16T04:09:19.033 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:09:19.034 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:09:19.034 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:09:19.034 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:09:19.034 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:19.034 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:09:19.034 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:09:19.034 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:09:19.034 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:09:19.034 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:09:19.034 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.012+0000 7f6485608640 10 client.1014323.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:09:19.034 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.012+0000 7f6485608640 10 monclient: _renew_subs 2024-09-16T04:09:19.034 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.012+0000 7f6485608640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:09:19.034 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.012+0000 7f6485608640 1 -- 172.21.15.42:0/908268144 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f64800a5360 con 0x7f64800a1330 2024-09-16T04:09:19.034 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.012+0000 7f647cff9640 4 set_mon_vals no callback set 2024-09-16T04:09:19.034 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.012+0000 7f6485608640 1 librados: init done 2024-09-16T04:09:19.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.012+0000 7f6485608640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:19.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.012+0000 7f6485608640 1 --2- 172.21.15.42:0/908268144 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f648006dac0 0x7f6480136320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:09:19.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.012+0000 7f6485608640 10 monclient(hunting): start opening mon connection 2024-09-16T04:09:19.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.012+0000 7f6485608640 1 -- 172.21.15.42:0/908268144 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f648009ccd0 con 0x7f648006dac0 2024-09-16T04:09:19.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.012+0000 7f647e7fc640 1 --2- 172.21.15.42:0/908268144 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f648006dac0 0x7f6480136320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:09:19.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.012+0000 7f64677fe640 1 -- 172.21.15.42:0/908268144 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f64700b47f0 con 0x7f64800a1330 2024-09-16T04:09:19.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.012+0000 7f647e7fc640 10 monclient: get_auth_request con 0x7f648006dac0 auth_method 0 2024-09-16T04:09:19.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.012+0000 7f647e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:09:19.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.012+0000 7f647e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:19.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.012+0000 7f647e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:19.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.012+0000 7f647e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:09:19.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.012+0000 7f647e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:09:19.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.012+0000 7f647e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:09:19.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.012+0000 7f64677fe640 1 --2- 172.21.15.42:0/908268144 >> v2:172.21.15.154:6812/254845342 conn(0x7f646c0377c0 0x7f646c039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:09:19.036 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.012+0000 7f64677fe640 1 -- 172.21.15.42:0/908268144 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(706..706 src has 451..706) ==== 5040+0+0 (secure 0 0 0) 0x7f64700e2810 con 0x7f64800a1330 2024-09-16T04:09:19.036 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.012+0000 7f64677fe640 10 client.1014323.objecter ms_dispatch 0x7f6480000ba0 osd_map(706..706 src has 451..706) 2024-09-16T04:09:19.036 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.012+0000 7f64677fe640 3 client.1014323.objecter handle_osd_map got epochs [706,706] > 0 2024-09-16T04:09:19.036 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.012+0000 7f64677fe640 3 client.1014323.objecter handle_osd_map decoding full epoch 706 2024-09-16T04:09:19.036 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.012+0000 7f647e7fc640 10 monclient(hunting): handle_auth_done global_id 1034178 payload 824 2024-09-16T04:09:19.036 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.012+0000 7f647e7fc640 1 -- 172.21.15.42:0/908268144 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6480135550 con 0x7f648006dac0 2024-09-16T04:09:19.036 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.012+0000 7f64677fe640 20 client.1014323.objecter dump_active .. 0 homeless 2024-09-16T04:09:19.036 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.012+0000 7f64677fe640 10 client.1014323.objecter ms_handle_connect 0x7f648006dac0 2024-09-16T04:09:19.036 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.012+0000 7f64677fe640 10 client.1014323.objecter resend_mon_ops 2024-09-16T04:09:19.036 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.012+0000 7f647e7fc640 1 --2- 172.21.15.42:0/908268144 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f648006dac0 0x7f6480136320 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f648013b300 tx=0x7f6474011d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:09:19.036 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.012+0000 7f64677fe640 1 -- 172.21.15.42:0/908268144 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f647400fa70 con 0x7f648006dac0 2024-09-16T04:09:19.036 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.012+0000 7f64677fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:19.036 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.012+0000 7f64677fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:09:19.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.012+0000 7f64677fe640 10 monclient: dump: 2024-09-16T04:09:19.037 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:09:19.037 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:09:19.037 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:09:19.037 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:09:19.037 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:19.037 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:09:19.037 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:09:19.037 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:09:19.037 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:09:19.037 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:09:19.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.012+0000 7f647f7fe640 1 --2- 172.21.15.42:0/908268144 >> v2:172.21.15.154:6812/254845342 conn(0x7f646c0377c0 0x7f646c039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:09:19.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.012+0000 7f647f7fe640 10 monclient: get_auth_request con 0x7f646c0377c0 auth_method 0 2024-09-16T04:09:19.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.012+0000 7f64677fe640 1 -- 172.21.15.42:0/908268144 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f647400fa70 con 0x7f648006dac0 2024-09-16T04:09:19.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.012+0000 7f64677fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:19.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.012+0000 7f64677fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:09:19.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.012+0000 7f64677fe640 1 -- 172.21.15.42:0/908268144 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f648006dac0 msgr2=0x7f6480136320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:19.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.012+0000 7f64677fe640 1 --2- 172.21.15.42:0/908268144 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f648006dac0 0x7f6480136320 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f648013b300 tx=0x7f6474011d80 comp rx=0 tx=0).stop 2024-09-16T04:09:19.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.012+0000 7f64677fe640 1 -- 172.21.15.42:0/908268144 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f648006dac0 msgr2=0x7f6480136320 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:09:19.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.012+0000 7f64677fe640 1 --2- 172.21.15.42:0/908268144 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f648006dac0 0x7f6480136320 secure :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f648013b300 tx=0x7f6474011d80 comp rx=0 tx=0).stop 2024-09-16T04:09:19.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.016+0000 7f64677fe640 10 client.1014323.objecter ms_handle_connect 0x7f646c0377c0 2024-09-16T04:09:19.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.016+0000 7f647f7fe640 1 --2- 172.21.15.42:0/908268144 >> v2:172.21.15.154:6812/254845342 conn(0x7f646c0377c0 0x7f646c039c80 secure :-1 s=READY pgs=3912 cs=0 l=1 rev1=1 crypto rx=0x7f6478003cc0 tx=0x7f6478009270 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:09:19.039 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.020+0000 7f6485608640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:09:19.040 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.020+0000 7f6485608640 1 --2- 172.21.15.42:0/908268144 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f648013aa80 0x7f6480136860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:09:19.040 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.020+0000 7f6485608640 10 monclient(hunting): start opening mon connection 2024-09-16T04:09:19.040 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.020+0000 7f6485608640 1 -- 172.21.15.42:0/908268144 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f648009db10 con 0x7f648013aa80 2024-09-16T04:09:19.040 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.020+0000 7f647e7fc640 1 --2- 172.21.15.42:0/908268144 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f648013aa80 0x7f6480136860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:09:19.040 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.020+0000 7f647e7fc640 10 monclient: get_auth_request con 0x7f648013aa80 auth_method 0 2024-09-16T04:09:19.040 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.020+0000 7f647e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:09:19.040 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.020+0000 7f647e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:19.040 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.020+0000 7f647e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:19.040 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.020+0000 7f647e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:09:19.040 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.020+0000 7f647e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:09:19.040 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.020+0000 7f647e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:09:19.040 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.020+0000 7f647e7fc640 10 monclient(hunting): handle_auth_done global_id 1034181 payload 824 2024-09-16T04:09:19.040 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.020+0000 7f647e7fc640 1 -- 172.21.15.42:0/908268144 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f648006ea90 con 0x7f648013aa80 2024-09-16T04:09:19.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.020+0000 7f64677fe640 10 client.1014323.objecter ms_handle_connect 0x7f648013aa80 2024-09-16T04:09:19.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.020+0000 7f647e7fc640 1 --2- 172.21.15.42:0/908268144 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f648013aa80 0x7f6480136860 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f647401c000 tx=0x7f64740046d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:09:19.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.020+0000 7f64677fe640 10 client.1014323.objecter resend_mon_ops 2024-09-16T04:09:19.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.020+0000 7f64677fe640 1 -- 172.21.15.42:0/908268144 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6474014640 con 0x7f648013aa80 2024-09-16T04:09:19.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.020+0000 7f64677fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:19.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.020+0000 7f64677fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:09:19.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.020+0000 7f64677fe640 10 monclient: dump: 2024-09-16T04:09:19.041 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:09:19.041 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:09:19.041 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:09:19.041 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:09:19.041 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:19.041 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:09:19.042 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:09:19.042 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:09:19.042 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:09:19.042 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:09:19.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.020+0000 7f64677fe640 1 -- 172.21.15.42:0/908268144 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4870 (secure 0 0 0) 0x7f647402b470 con 0x7f648013aa80 2024-09-16T04:09:19.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.020+0000 7f64677fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:09:19.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.020+0000 7f64677fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:09:19.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.020+0000 7f64677fe640 1 -- 172.21.15.42:0/908268144 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f648013aa80 msgr2=0x7f6480136860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:19.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.020+0000 7f64677fe640 1 --2- 172.21.15.42:0/908268144 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f648013aa80 0x7f6480136860 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f647401c000 tx=0x7f64740046d0 comp rx=0 tx=0).stop 2024-09-16T04:09:19.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.020+0000 7f64677fe640 1 -- 172.21.15.42:0/908268144 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f648013aa80 msgr2=0x7f6480136860 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:09:19.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.020+0000 7f64677fe640 1 --2- 172.21.15.42:0/908268144 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f648013aa80 0x7f6480136860 secure :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f647401c000 tx=0x7f64740046d0 comp rx=0 tx=0).stop 2024-09-16T04:09:19.043 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T04:09:19.043 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:09:19.043 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:09:19.043 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "leader", 2024-09-16T04:09:19.043 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 250, 2024-09-16T04:09:19.043 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 38955, 2024-09-16T04:09:19.043 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T04:09:19.043 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T04:09:19.043 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T04:09:19.043 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:09:19.043 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 0, 2024-09-16T04:09:19.044 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:09:19.044 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T04:09:19.044 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T04:09:19.044 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:09:19.044 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:09:19.044 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:09:19.044 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:09:19.044 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:09:19.044 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:09:19.044 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:09:19.044 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:09:19.044 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:09:19.044 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:09:19.045 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:09:19.045 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:09:19.045 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:09:19.045 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T04:09:19.045 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:09:19.045 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:09:19.045 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:09:19.045 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:09:19.045 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:09:19.045 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:09:19.045 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:09:19.045 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:09:19.045 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:09:19.046 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:09:19.046 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:09:19.046 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:09:19.046 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:09:19.046 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T04:09:19.046 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T04:09:19.046 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T04:09:19.046 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T04:09:19.046 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T04:09:19.046 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T04:09:19.046 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:09:19.046 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:09:19.046 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T04:09:19.046 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T04:09:19.047 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T04:09:19.047 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T04:09:19.047 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T04:09:19.047 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T04:09:19.047 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T04:09:19.047 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:09:19.047 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T04:09:19.047 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:09:19.047 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:09:19.047 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:09:19.047 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:09:19.047 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:09:19.047 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:09:19.047 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:09:19.048 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:09:19.048 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:09:19.048 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:09:19.048 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:09:19.048 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:09:19.048 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T04:09:19.048 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:09:19.048 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T04:09:19.048 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:09:19.048 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:09:19.048 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:09:19.048 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:09:19.048 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:09:19.048 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:09:19.049 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:09:19.049 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T04:09:19.049 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:09:19.049 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:09:19.049 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:09:19.049 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:09:19.049 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T04:09:19.049 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:09:19.049 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:09:19.049 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:09:19.049 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:09:19.049 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T04:09:19.049 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T04:09:19.049 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:09:19.049 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:09:19.050 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:09:19.050 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:09:19.050 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:09:19.050 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:09:19.050 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:09:19.050 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:09:19.050 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:09:19.050 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:09:19.050 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:09:19.050 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T04:09:19.050 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:09:19.050 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:09:19.050 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:09:19.050 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:09:19.051 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T04:09:19.051 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:09:19.051 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:09:19.051 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:09:19.051 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:09:19.051 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T04:09:19.051 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T04:09:19.051 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:09:19.051 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:09:19.051 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:09:19.051 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:09:19.051 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:09:19.051 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:09:19.051 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:09:19.051 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:09:19.051 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:09:19.052 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:09:19.052 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:09:19.052 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T04:09:19.052 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:09:19.052 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:09:19.052 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:09:19.052 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:09:19.052 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T04:09:19.052 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:09:19.052 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:09:19.052 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:09:19.052 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:09:19.052 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T04:09:19.052 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T04:09:19.052 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:09:19.053 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:09:19.053 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:09:19.053 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:09:19.053 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:09:19.053 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:09:19.053 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T04:09:19.053 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T04:09:19.053 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:09:19.053 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:09:19.053 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:09:19.053 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:09:19.053 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:09:19.053 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:09:19.053 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T04:09:19.054 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:09:19.054 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:09:19.054 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:09:19.054 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 2 2024-09-16T04:09:19.054 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:09:19.054 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:09:19.054 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T04:09:19.054 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:09:19.054 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:09:19.054 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:09:19.054 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 4 2024-09-16T04:09:19.054 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:09:19.054 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:09:19.054 INFO:teuthology.orchestra.run.smithi042.stdout: "mgr": [ 2024-09-16T04:09:19.054 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:09:19.055 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:09:19.055 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:09:19.055 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:09:19.055 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:09:19.055 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:09:19.055 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:09:19.055 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T04:09:19.055 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T04:09:19.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.020+0000 7f6485608640 10 librados: watch_flush enter 2024-09-16T04:09:19.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.020+0000 7f6485608640 10 librados: watch_flush exit 2024-09-16T04:09:19.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.020+0000 7f6485608640 20 client.1014323.objecter shutdown clearing up homeless session... 2024-09-16T04:09:19.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.020+0000 7f6485608640 10 client.1014323.objecter successfully canceled tick 2024-09-16T04:09:19.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.020+0000 7f6485608640 1 -- 172.21.15.42:0/908268144 >> v2:172.21.15.154:6812/254845342 conn(0x7f646c0377c0 msgr2=0x7f646c039c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:19.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.020+0000 7f6485608640 1 --2- 172.21.15.42:0/908268144 >> v2:172.21.15.154:6812/254845342 conn(0x7f646c0377c0 0x7f646c039c80 secure :-1 s=READY pgs=3912 cs=0 l=1 rev1=1 crypto rx=0x7f6478003cc0 tx=0x7f6478009270 comp rx=0 tx=0).stop 2024-09-16T04:09:19.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.020+0000 7f6485608640 10 monclient: shutdown 2024-09-16T04:09:19.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.020+0000 7f6485608640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:19.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.020+0000 7f6485608640 1 -- 172.21.15.42:0/908268144 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f64800a1330 msgr2=0x7f6480139bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:19.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.020+0000 7f6485608640 1 --2- 172.21.15.42:0/908268144 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f64800a1330 0x7f6480139bd0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f647001ab30 tx=0x7f64800a22d0 comp rx=0 tx=0).stop 2024-09-16T04:09:19.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.020+0000 7f647f7fe640 1 -- 172.21.15.42:0/908268144 reap_dead start 2024-09-16T04:09:19.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.020+0000 7f647f7fe640 1 -- 172.21.15.42:0/908268144 reap_dead start 2024-09-16T04:09:19.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.020+0000 7f6485608640 1 -- 172.21.15.42:0/908268144 shutdown_connections 2024-09-16T04:09:19.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.020+0000 7f6485608640 1 -- 172.21.15.42:0/908268144 >> 172.21.15.42:0/908268144 conn(0x7f64800a2420 msgr2=0x7f64800a3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:19.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.020+0000 7f6485608640 1 -- 172.21.15.42:0/908268144 shutdown_connections 2024-09-16T04:09:19.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.020+0000 7f6485608640 1 -- 172.21.15.42:0/908268144 wait complete. 2024-09-16T04:09:19.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.020+0000 7f6485608640 1 librados: shutdown 2024-09-16T04:09:19.058 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T04:09:19.058 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-16T04:09:19.058 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T04:09:19.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.172+0000 7f068589d640 10 monclient: get_monmap_and_config 2024-09-16T04:09:19.180 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.172+0000 7f068589d640 10 monclient: build_initial_monmap 2024-09-16T04:09:19.180 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.172+0000 7f068589d640 10 monclient: monmap: 2024-09-16T04:09:19.180 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:09:19.180 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:09:19.180 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:09:19.178575+0000 2024-09-16T04:09:19.180 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:09:19.178575+0000 2024-09-16T04:09:19.180 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:19.180 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:09:19.180 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:09:19.180 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:09:19.180 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:09:19.180 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:09:19.180 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:09:19.181 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:09:19.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.172+0000 7f068589d640 1 Processor -- start 2024-09-16T04:09:19.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.172+0000 7f068589d640 1 -- start start 2024-09-16T04:09:19.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.172+0000 7f068589d640 10 monclient: init 2024-09-16T04:09:19.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.172+0000 7f068589d640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:19.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.172+0000 7f068589d640 10 monclient: _add_conns ranks=[4,2,3,1,0] 2024-09-16T04:09:19.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.176+0000 7f068589d640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f06800a39b0 0x7f0680133b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:09:19.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.176+0000 7f068589d640 10 monclient(hunting): picked mon.noname-c con 0x7f06800a39b0 addr v2:172.21.15.154:3300/0 2024-09-16T04:09:19.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.176+0000 7f068589d640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f06800a3d90 0x7f0680137000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:09:19.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.176+0000 7f068589d640 10 monclient(hunting): picked mon.noname-b con 0x7f06800a3d90 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:09:19.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.176+0000 7f068589d640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f0680137540 0x7f0680137990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:09:19.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.176+0000 7f068589d640 10 monclient(hunting): picked mon.noname-a con 0x7f0680137540 addr v2:172.21.15.42:3300/0 2024-09-16T04:09:19.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.176+0000 7f068589d640 10 monclient(hunting): start opening mon connection 2024-09-16T04:09:19.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.176+0000 7f068589d640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f068009c140 con 0x7f0680137540 2024-09-16T04:09:19.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.176+0000 7f068589d640 10 monclient(hunting): start opening mon connection 2024-09-16T04:09:19.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.176+0000 7f068589d640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f068009c420 con 0x7f06800a39b0 2024-09-16T04:09:19.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.176+0000 7f068589d640 10 monclient(hunting): start opening mon connection 2024-09-16T04:09:19.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.176+0000 7f068589d640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f068006dd60 con 0x7f06800a3d90 2024-09-16T04:09:19.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.176+0000 7f068589d640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:19.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.176+0000 7f068589d640 10 monclient(hunting): authenticate will time out at 2606.148130s 2024-09-16T04:09:19.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.176+0000 7f067effd640 1 -- >> v2:172.21.15.154:3300/0 conn(0x7f06800a39b0 msgr2=0x7f0680133b80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T04:09:19.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.176+0000 7f067effd640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f06800a39b0 0x7f0680133b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:09:19.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.176+0000 7f067f7fe640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f0680137540 0x7f0680137990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:09:19.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.176+0000 7f067e7fc640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f06800a3d90 0x7f0680137000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:09:19.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.176+0000 7f067f7fe640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f0680137540 0x7f0680137990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:40396/0 (socket says 172.21.15.42:40396) 2024-09-16T04:09:19.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.176+0000 7f067e7fc640 10 monclient(hunting): get_auth_request con 0x7f06800a3d90 auth_method 0 2024-09-16T04:09:19.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.176+0000 7f067f7fe640 1 -- 172.21.15.42:0/3590161012 learned_addr learned my addr 172.21.15.42:0/3590161012 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:09:19.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.176+0000 7f067e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:09:19.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.176+0000 7f067e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:19.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.176+0000 7f067e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:19.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.176+0000 7f067f7fe640 10 monclient(hunting): get_auth_request con 0x7f0680137540 auth_method 0 2024-09-16T04:09:19.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.176+0000 7f067f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:09:19.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.176+0000 7f067f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:19.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.176+0000 7f067f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:19.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.176+0000 7f067e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:09:19.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.176+0000 7f067e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:09:19.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.176+0000 7f067e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:09:19.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.176+0000 7f067f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:09:19.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.176+0000 7f067f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:09:19.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.176+0000 7f067f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:09:19.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.176+0000 7f067e7fc640 10 monclient(hunting): handle_auth_done global_id 1014326 payload 274 2024-09-16T04:09:19.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.176+0000 7f067e7fc640 1 -- 172.21.15.42:0/3590161012 >> v2:172.21.15.154:3300/0 conn(0x7f06800a39b0 msgr2=0x7f0680133b80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:09:19.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.176+0000 7f067e7fc640 1 --2- 172.21.15.42:0/3590161012 >> v2:172.21.15.154:3300/0 conn(0x7f06800a39b0 0x7f0680133b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:09:19.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.176+0000 7f067e7fc640 1 -- 172.21.15.42:0/3590161012 >> v2:172.21.15.42:3300/0 conn(0x7f0680137540 msgr2=0x7f0680137990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:09:19.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.176+0000 7f067e7fc640 1 --2- 172.21.15.42:0/3590161012 >> v2:172.21.15.42:3300/0 conn(0x7f0680137540 0x7f0680137990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:09:19.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.176+0000 7f067e7fc640 10 monclient: _finish_hunting 0 2024-09-16T04:09:19.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.176+0000 7f067e7fc640 1 monclient: found mon.noname-b 2024-09-16T04:09:19.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.176+0000 7f067e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:19.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.176+0000 7f067e7fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T04:09:19.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.176+0000 7f067e7fc640 1 -- 172.21.15.42:0/3590161012 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06800a5e20 con 0x7f06800a3d90 2024-09-16T04:09:19.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.176+0000 7f067f7fe640 1 --2- 172.21.15.42:0/3590161012 >> v2:172.21.15.42:3300/0 conn(0x7f0680137540 0x7f0680137990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:09:19.192 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.176+0000 7f067e7fc640 1 --2- 172.21.15.42:0/3590161012 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f06800a3d90 0x7f0680137000 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f06800a7750 tx=0x7f067401c360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=858a61dcea16b568 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:09:19.192 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.176+0000 7f067dffb640 1 -- 172.21.15.42:0/3590161012 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06740b0070 con 0x7f06800a3d90 2024-09-16T04:09:19.192 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.176+0000 7f067dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:19.192 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.176+0000 7f067dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:09:19.192 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.176+0000 7f067dffb640 10 monclient: dump: 2024-09-16T04:09:19.193 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:09:19.193 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:09:19.193 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:09:19.193 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:09:19.193 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:19.193 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:09:19.193 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:09:19.193 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:09:19.193 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:09:19.193 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:09:19.193 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.176+0000 7f067dffb640 10 monclient: _finish_auth 0 2024-09-16T04:09:19.193 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.176+0000 7f067dffb640 10 monclient: _check_auth_tickets 2024-09-16T04:09:19.193 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.176+0000 7f067dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:19.193 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.176+0000 7f067dffb640 1 -- 172.21.15.42:0/3590161012 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f06740ab600 con 0x7f06800a3d90 2024-09-16T04:09:19.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.176+0000 7f068589d640 5 monclient: authenticate success, global_id 1014326 2024-09-16T04:09:19.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.176+0000 7f068589d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:09:19.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.176+0000 7f067dffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:19.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.176+0000 7f068589d640 10 monclient: get_monmap_and_config success 2024-09-16T04:09:19.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.176+0000 7f067dffb640 1 -- 172.21.15.42:0/3590161012 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06740ab9d0 con 0x7f06800a3d90 2024-09-16T04:09:19.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.176+0000 7f067dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:19.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.176+0000 7f068589d640 4 set_mon_vals no callback set 2024-09-16T04:09:19.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.176+0000 7f067dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:09:19.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.176+0000 7f067dffb640 10 monclient: dump: 2024-09-16T04:09:19.194 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:09:19.194 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:09:19.194 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:09:19.194 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:09:19.194 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:19.195 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:09:19.195 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:09:19.195 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:09:19.195 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:09:19.195 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:09:19.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.176+0000 7f068589d640 10 monclient: shutdown 2024-09-16T04:09:19.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.176+0000 7f068589d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:19.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.176+0000 7f068589d640 1 -- 172.21.15.42:0/3590161012 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f06800a3d90 msgr2=0x7f0680137000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:19.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.176+0000 7f068589d640 1 --2- 172.21.15.42:0/3590161012 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f06800a3d90 0x7f0680137000 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f06800a7750 tx=0x7f067401c360 comp rx=0 tx=0).stop 2024-09-16T04:09:19.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.176+0000 7f068589d640 1 -- 172.21.15.42:0/3590161012 shutdown_connections 2024-09-16T04:09:19.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.176+0000 7f068589d640 1 --2- 172.21.15.42:0/3590161012 >> v2:172.21.15.42:3300/0 conn(0x7f0680137540 0x7f0680137990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:09:19.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.176+0000 7f068589d640 1 --2- 172.21.15.42:0/3590161012 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f06800a3d90 0x7f0680137000 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:09:19.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.176+0000 7f068589d640 1 --2- 172.21.15.42:0/3590161012 >> v2:172.21.15.154:3300/0 conn(0x7f06800a39b0 0x7f0680133b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:09:19.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.176+0000 7f068589d640 1 -- 172.21.15.42:0/3590161012 >> 172.21.15.42:0/3590161012 conn(0x7f06800a9fb0 msgr2=0x7f06800aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:19.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.176+0000 7f068589d640 1 -- 172.21.15.42:0/3590161012 shutdown_connections 2024-09-16T04:09:19.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.176+0000 7f068589d640 1 -- 172.21.15.42:0/3590161012 wait complete. 2024-09-16T04:09:19.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.180+0000 7f068589d640 10 monclient: build_initial_monmap 2024-09-16T04:09:19.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.180+0000 7f068589d640 10 monclient: monmap: 2024-09-16T04:09:19.196 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:09:19.196 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:09:19.196 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:09:19.196 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:09:19.196 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:19.196 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:09:19.196 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:09:19.196 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:09:19.196 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:09:19.196 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:09:19.197 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.180+0000 7f068589d640 1 librados: starting msgr at 2024-09-16T04:09:19.197 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.180+0000 7f068589d640 1 librados: starting objecter 2024-09-16T04:09:19.197 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.180+0000 7f068589d640 1 Processor -- start 2024-09-16T04:09:19.197 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.180+0000 7f068589d640 1 -- start start 2024-09-16T04:09:19.197 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.180+0000 7f068589d640 1 librados: setting wanted keys 2024-09-16T04:09:19.197 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.180+0000 7f068589d640 1 librados: calling monclient init 2024-09-16T04:09:19.197 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.180+0000 7f068589d640 10 monclient: init 2024-09-16T04:09:19.197 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.180+0000 7f068589d640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:19.197 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.180+0000 7f068589d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:09:19.197 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.180+0000 7f068589d640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f06800a39b0 0x7f06801cf200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:09:19.197 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.180+0000 7f068589d640 10 monclient(hunting): picked mon.noname-c con 0x7f06800a39b0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:09:19.197 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.180+0000 7f068589d640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f06800a3d90 0x7f06800a2d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:09:19.197 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.180+0000 7f068589d640 10 monclient(hunting): picked mon.noname-a con 0x7f06800a3d90 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:09:19.197 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.180+0000 7f068589d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0680137540 0x7f06800a3260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:09:19.198 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.180+0000 7f068589d640 10 monclient(hunting): picked mon.noname-b con 0x7f0680137540 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:09:19.198 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.180+0000 7f068589d640 10 monclient(hunting): start opening mon connection 2024-09-16T04:09:19.198 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.180+0000 7f068589d640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f0680140dc0 con 0x7f06800a3d90 2024-09-16T04:09:19.198 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.180+0000 7f068589d640 10 monclient(hunting): start opening mon connection 2024-09-16T04:09:19.198 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.180+0000 7f068589d640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f0680140f40 con 0x7f0680137540 2024-09-16T04:09:19.198 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.180+0000 7f068589d640 10 monclient(hunting): start opening mon connection 2024-09-16T04:09:19.198 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.180+0000 7f068589d640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f06800a4e00 con 0x7f06800a39b0 2024-09-16T04:09:19.198 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.180+0000 7f068589d640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:19.198 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.180+0000 7f068589d640 10 monclient(hunting): authenticate will time out at 2606.153241s 2024-09-16T04:09:19.198 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.180+0000 7f067e7fc640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f06800a3d90 0x7f06800a2d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:09:19.198 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.180+0000 7f067effd640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f06800a39b0 0x7f06801cf200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:09:19.198 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.180+0000 7f067e7fc640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f06800a3d90 0x7f06800a2d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:40404/0 (socket says 172.21.15.42:40404) 2024-09-16T04:09:19.198 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.180+0000 7f067effd640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f06800a39b0 0x7f06801cf200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:41378/0 (socket says 172.21.15.42:41378) 2024-09-16T04:09:19.198 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.180+0000 7f067e7fc640 1 -- 172.21.15.42:0/4064430655 learned_addr learned my addr 172.21.15.42:0/4064430655 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:09:19.199 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.180+0000 7f067e7fc640 10 monclient(hunting): get_auth_request con 0x7f06800a3d90 auth_method 0 2024-09-16T04:09:19.199 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.180+0000 7f067e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:09:19.199 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.180+0000 7f067e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:19.199 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.180+0000 7f067e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:19.199 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.180+0000 7f067effd640 10 monclient(hunting): get_auth_request con 0x7f06800a39b0 auth_method 0 2024-09-16T04:09:19.199 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.180+0000 7f067f7fe640 1 -- 172.21.15.42:0/4064430655 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0680137540 msgr2=0x7f06800a3260 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T04:09:19.199 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.180+0000 7f067effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:09:19.199 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.180+0000 7f067effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:19.199 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.180+0000 7f067effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:19.199 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.180+0000 7f067f7fe640 1 --2- 172.21.15.42:0/4064430655 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0680137540 0x7f06800a3260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:09:19.199 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.180+0000 7f067e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:09:19.199 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.180+0000 7f067e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:09:19.199 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.180+0000 7f067e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:09:19.199 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.180+0000 7f067effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:09:19.199 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.180+0000 7f067effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:09:19.200 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.180+0000 7f067effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:09:19.200 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.180+0000 7f067e7fc640 10 monclient(hunting): handle_auth_done global_id 1034187 payload 824 2024-09-16T04:09:19.200 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.180+0000 7f067e7fc640 1 -- 172.21.15.42:0/4064430655 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f06800a39b0 msgr2=0x7f06801cf200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:19.200 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.180+0000 7f067e7fc640 1 --2- 172.21.15.42:0/4064430655 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f06800a39b0 0x7f06801cf200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:09:19.200 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.180+0000 7f067e7fc640 1 -- 172.21.15.42:0/4064430655 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0680137540 msgr2=0x7f06800a3260 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:09:19.200 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.180+0000 7f067e7fc640 1 --2- 172.21.15.42:0/4064430655 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0680137540 0x7f06800a3260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:09:19.200 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.180+0000 7f067e7fc640 10 monclient: _finish_hunting 0 2024-09-16T04:09:19.200 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.180+0000 7f067e7fc640 1 monclient: found mon.noname-a 2024-09-16T04:09:19.200 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.180+0000 7f067e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:19.200 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.180+0000 7f067e7fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:09:19.200 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.180+0000 7f067e7fc640 1 -- 172.21.15.42:0/4064430655 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f068009d7b0 con 0x7f06800a3d90 2024-09-16T04:09:19.200 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.180+0000 7f067effd640 1 --2- 172.21.15.42:0/4064430655 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f06800a39b0 0x7f06801cf200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:09:19.200 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.180+0000 7f067e7fc640 1 --2- 172.21.15.42:0/4064430655 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f06800a3d90 0x7f06800a2d20 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f06740ae040 tx=0x7f067401aba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:09:19.201 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.180+0000 7f0667fff640 10 client.?.objecter ms_handle_connect 0x7f06800a3d90 2024-09-16T04:09:19.201 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.180+0000 7f0667fff640 10 client.?.objecter resend_mon_ops 2024-09-16T04:09:19.201 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.180+0000 7f0667fff640 1 -- 172.21.15.42:0/4064430655 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06740b0050 con 0x7f06800a3d90 2024-09-16T04:09:19.201 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.180+0000 7f0667fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:19.201 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.180+0000 7f0667fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:09:19.201 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.180+0000 7f0667fff640 10 monclient: dump: 2024-09-16T04:09:19.201 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:09:19.201 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:09:19.201 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:09:19.201 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:09:19.201 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:19.201 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:09:19.201 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:09:19.201 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:09:19.201 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:09:19.202 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:09:19.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.180+0000 7f0667fff640 10 monclient: _finish_auth 0 2024-09-16T04:09:19.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.180+0000 7f0667fff640 10 monclient: _check_auth_tickets 2024-09-16T04:09:19.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.180+0000 7f0667fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:19.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.180+0000 7f068589d640 5 monclient: authenticate success, global_id 1034187 2024-09-16T04:09:19.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.180+0000 7f0667fff640 1 -- 172.21.15.42:0/4064430655 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f06740bc070 con 0x7f06800a3d90 2024-09-16T04:09:19.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.180+0000 7f068589d640 10 monclient: _renew_subs 2024-09-16T04:09:19.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.180+0000 7f068589d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:09:19.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.180+0000 7f068589d640 1 -- 172.21.15.42:0/4064430655 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f068013cfc0 con 0x7f06800a3d90 2024-09-16T04:09:19.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.180+0000 7f0667fff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:19.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.180+0000 7f0667fff640 1 -- 172.21.15.42:0/4064430655 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06740b5a10 con 0x7f06800a3d90 2024-09-16T04:09:19.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.180+0000 7f0667fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:19.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.180+0000 7f0667fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:09:19.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.180+0000 7f068589d640 10 client.1034187.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:09:19.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.180+0000 7f0667fff640 10 monclient: dump: 2024-09-16T04:09:19.203 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:09:19.203 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:09:19.203 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:09:19.203 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:09:19.203 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:19.203 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:09:19.203 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:09:19.203 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:09:19.203 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:09:19.203 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:09:19.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.180+0000 7f068589d640 10 monclient: _renew_subs 2024-09-16T04:09:19.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.180+0000 7f067d7fa640 4 set_mon_vals no callback set 2024-09-16T04:09:19.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.180+0000 7f068589d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:09:19.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.180+0000 7f068589d640 1 -- 172.21.15.42:0/4064430655 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f068006f680 con 0x7f06800a3d90 2024-09-16T04:09:19.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.180+0000 7f068589d640 1 librados: init done 2024-09-16T04:09:19.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.180+0000 7f068589d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:19.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.180+0000 7f0667fff640 1 -- 172.21.15.42:0/4064430655 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f06740b5bb0 con 0x7f06800a3d90 2024-09-16T04:09:19.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.180+0000 7f068589d640 1 --2- 172.21.15.42:0/4064430655 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f068009cf10 0x7f068013b470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:09:19.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.180+0000 7f068589d640 10 monclient(hunting): start opening mon connection 2024-09-16T04:09:19.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.180+0000 7f068589d640 1 -- 172.21.15.42:0/4064430655 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f06801342d0 con 0x7f068009cf10 2024-09-16T04:09:19.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.180+0000 7f067effd640 1 --2- 172.21.15.42:0/4064430655 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f068009cf10 0x7f068013b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:09:19.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.180+0000 7f067effd640 10 monclient: get_auth_request con 0x7f068009cf10 auth_method 0 2024-09-16T04:09:19.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.180+0000 7f067effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:09:19.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.180+0000 7f067effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:19.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.180+0000 7f067effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:19.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.180+0000 7f0667fff640 1 --2- 172.21.15.42:0/4064430655 >> v2:172.21.15.154:6812/254845342 conn(0x7f066c0377f0 0x7f066c039cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:09:19.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.180+0000 7f067effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:09:19.205 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.180+0000 7f067effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:09:19.205 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.180+0000 7f067effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:09:19.205 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.180+0000 7f0667fff640 1 -- 172.21.15.42:0/4064430655 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(706..706 src has 451..706) ==== 5040+0+0 (secure 0 0 0) 0x7f06740e35f0 con 0x7f06800a3d90 2024-09-16T04:09:19.205 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.180+0000 7f0667fff640 10 client.1034187.objecter ms_dispatch 0x7f0680000ba0 osd_map(706..706 src has 451..706) 2024-09-16T04:09:19.205 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.180+0000 7f0667fff640 3 client.1034187.objecter handle_osd_map got epochs [706,706] > 0 2024-09-16T04:09:19.205 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.180+0000 7f0667fff640 3 client.1034187.objecter handle_osd_map decoding full epoch 706 2024-09-16T04:09:19.205 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.180+0000 7f0667fff640 20 client.1034187.objecter dump_active .. 0 homeless 2024-09-16T04:09:19.205 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.180+0000 7f067effd640 10 monclient(hunting): handle_auth_done global_id 1014332 payload 824 2024-09-16T04:09:19.205 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.180+0000 7f067effd640 1 -- 172.21.15.42:0/4064430655 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f06801cf740 con 0x7f068009cf10 2024-09-16T04:09:19.205 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.180+0000 7f0667fff640 10 client.1034187.objecter ms_handle_connect 0x7f068009cf10 2024-09-16T04:09:19.205 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.180+0000 7f0667fff640 10 client.1034187.objecter resend_mon_ops 2024-09-16T04:09:19.205 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.180+0000 7f067effd640 1 --2- 172.21.15.42:0/4064430655 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f068009cf10 0x7f068013b470 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f06700053c0 tx=0x7f0670005880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:09:19.205 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.180+0000 7f067f7fe640 1 --2- 172.21.15.42:0/4064430655 >> v2:172.21.15.154:6812/254845342 conn(0x7f066c0377f0 0x7f066c039cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:09:19.205 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.180+0000 7f067f7fe640 10 monclient: get_auth_request con 0x7f066c0377f0 auth_method 0 2024-09-16T04:09:19.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.180+0000 7f0667fff640 1 -- 172.21.15.42:0/4064430655 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0670016070 con 0x7f068009cf10 2024-09-16T04:09:19.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.180+0000 7f0667fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:19.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.180+0000 7f0667fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:09:19.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.180+0000 7f0667fff640 10 monclient: dump: 2024-09-16T04:09:19.206 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:09:19.206 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:09:19.206 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:09:19.206 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:09:19.206 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:19.206 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:09:19.206 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:09:19.206 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:09:19.206 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:09:19.206 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:09:19.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.184+0000 7f0667fff640 1 -- 172.21.15.42:0/4064430655 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f067000ea40 con 0x7f068009cf10 2024-09-16T04:09:19.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.184+0000 7f0667fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:19.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.184+0000 7f0667fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:09:19.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.184+0000 7f0667fff640 1 -- 172.21.15.42:0/4064430655 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f068009cf10 msgr2=0x7f068013b470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:19.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.184+0000 7f0667fff640 1 --2- 172.21.15.42:0/4064430655 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f068009cf10 0x7f068013b470 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f06700053c0 tx=0x7f0670005880 comp rx=0 tx=0).stop 2024-09-16T04:09:19.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.184+0000 7f0667fff640 1 -- 172.21.15.42:0/4064430655 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f068009cf10 msgr2=0x7f068013b470 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:09:19.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.184+0000 7f0667fff640 1 --2- 172.21.15.42:0/4064430655 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f068009cf10 0x7f068013b470 secure :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f06700053c0 tx=0x7f0670005880 comp rx=0 tx=0).stop 2024-09-16T04:09:19.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.184+0000 7f067f7fe640 1 --2- 172.21.15.42:0/4064430655 >> v2:172.21.15.154:6812/254845342 conn(0x7f066c0377f0 0x7f066c039cb0 secure :-1 s=READY pgs=3913 cs=0 l=1 rev1=1 crypto rx=0x7f0680133830 tx=0x7f067800b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:09:19.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.184+0000 7f0667fff640 10 client.1034187.objecter ms_handle_connect 0x7f066c0377f0 2024-09-16T04:09:19.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.188+0000 7f068589d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:09:19.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.188+0000 7f068589d640 1 --2- 172.21.15.42:0/4064430655 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f068013b9b0 0x7f068013bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:09:19.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.188+0000 7f068589d640 10 monclient(hunting): start opening mon connection 2024-09-16T04:09:19.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.188+0000 7f068589d640 1 -- 172.21.15.42:0/4064430655 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f068013c300 con 0x7f068013b9b0 2024-09-16T04:09:19.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.188+0000 7f067effd640 1 --2- 172.21.15.42:0/4064430655 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f068013b9b0 0x7f068013bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:09:19.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.188+0000 7f067effd640 10 monclient: get_auth_request con 0x7f068013b9b0 auth_method 0 2024-09-16T04:09:19.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.188+0000 7f067effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:09:19.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.188+0000 7f067effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:19.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.188+0000 7f067effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:19.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.188+0000 7f067effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:09:19.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.188+0000 7f067effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:09:19.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.188+0000 7f067effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:09:19.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.188+0000 7f067effd640 10 monclient(hunting): handle_auth_done global_id 1014335 payload 824 2024-09-16T04:09:19.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.188+0000 7f067effd640 1 -- 172.21.15.42:0/4064430655 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f0680137000 con 0x7f068013b9b0 2024-09-16T04:09:19.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.188+0000 7f0667fff640 10 client.1034187.objecter ms_handle_connect 0x7f068013b9b0 2024-09-16T04:09:19.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.188+0000 7f0667fff640 10 client.1034187.objecter resend_mon_ops 2024-09-16T04:09:19.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.188+0000 7f067effd640 1 --2- 172.21.15.42:0/4064430655 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f068013b9b0 0x7f068013bdc0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f06700053c0 tx=0x7f06700059e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:09:19.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.188+0000 7f0667fff640 1 -- 172.21.15.42:0/4064430655 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0670016040 con 0x7f068013b9b0 2024-09-16T04:09:19.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.188+0000 7f0667fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:19.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.188+0000 7f0667fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:09:19.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.188+0000 7f0667fff640 10 monclient: dump: 2024-09-16T04:09:19.211 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:09:19.211 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:09:19.211 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:09:19.211 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:09:19.211 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:19.211 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:09:19.211 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:09:19.211 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:09:19.211 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:09:19.211 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:09:19.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.188+0000 7f0667fff640 1 -- 172.21.15.42:0/4064430655 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4702 (secure 0 0 0) 0x7f0670005a50 con 0x7f068013b9b0 2024-09-16T04:09:19.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.188+0000 7f0667fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:09:19.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.188+0000 7f0667fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:09:19.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.188+0000 7f0667fff640 1 -- 172.21.15.42:0/4064430655 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f068013b9b0 msgr2=0x7f068013bdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:19.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.188+0000 7f0667fff640 1 --2- 172.21.15.42:0/4064430655 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f068013b9b0 0x7f068013bdc0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f06700053c0 tx=0x7f06700059e0 comp rx=0 tx=0).stop 2024-09-16T04:09:19.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.188+0000 7f0667fff640 1 -- 172.21.15.42:0/4064430655 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f068013b9b0 msgr2=0x7f068013bdc0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:09:19.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.188+0000 7f0667fff640 1 --2- 172.21.15.42:0/4064430655 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f068013b9b0 0x7f068013bdc0 secure :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f06700053c0 tx=0x7f06700059e0 comp rx=0 tx=0).stop 2024-09-16T04:09:19.212 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T04:09:19.212 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:09:19.212 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:09:19.212 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T04:09:19.212 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 250, 2024-09-16T04:09:19.213 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 154390, 2024-09-16T04:09:19.213 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T04:09:19.213 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T04:09:19.213 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T04:09:19.213 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:09:19.213 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 0, 2024-09-16T04:09:19.213 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:09:19.213 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T04:09:19.213 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T04:09:19.213 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:09:19.213 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:09:19.213 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:09:19.213 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:09:19.213 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:09:19.214 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:09:19.214 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:09:19.214 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:09:19.214 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:09:19.214 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:09:19.214 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:09:19.214 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:09:19.214 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:09:19.214 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T04:09:19.214 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:09:19.214 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:09:19.214 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:09:19.214 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:09:19.214 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:09:19.214 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:09:19.215 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:09:19.215 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:09:19.215 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:09:19.215 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:09:19.215 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:09:19.215 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:09:19.215 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:09:19.215 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T04:09:19.215 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T04:09:19.215 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T04:09:19.215 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T04:09:19.215 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T04:09:19.215 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T04:09:19.216 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:09:19.216 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:09:19.216 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T04:09:19.216 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T04:09:19.216 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T04:09:19.216 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T04:09:19.216 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T04:09:19.216 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T04:09:19.216 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T04:09:19.216 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:09:19.216 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T04:09:19.216 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:09:19.216 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:09:19.216 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:09:19.217 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:09:19.217 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:09:19.217 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:09:19.217 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:09:19.217 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:09:19.217 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:09:19.217 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:09:19.217 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:09:19.217 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:09:19.217 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T04:09:19.217 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:09:19.217 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T04:09:19.217 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:09:19.217 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:09:19.218 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:09:19.218 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:09:19.218 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:09:19.218 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:09:19.218 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:09:19.218 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T04:09:19.218 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:09:19.218 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:09:19.218 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:09:19.218 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:09:19.218 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T04:09:19.218 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:09:19.218 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:09:19.219 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:09:19.219 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:09:19.219 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T04:09:19.219 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T04:09:19.219 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:09:19.219 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:09:19.219 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:09:19.219 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:09:19.219 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:09:19.219 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:09:19.219 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:09:19.219 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:09:19.219 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:09:19.220 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:09:19.220 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:09:19.220 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T04:09:19.220 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:09:19.220 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:09:19.220 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:09:19.220 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:09:19.220 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T04:09:19.220 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:09:19.220 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:09:19.220 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:09:19.220 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:09:19.220 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T04:09:19.220 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T04:09:19.221 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:09:19.221 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:09:19.221 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:09:19.221 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:09:19.221 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:09:19.221 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:09:19.221 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:09:19.221 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:09:19.221 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:09:19.221 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:09:19.221 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:09:19.221 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T04:09:19.221 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:09:19.221 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:09:19.222 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:09:19.222 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:09:19.222 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T04:09:19.222 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:09:19.222 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:09:19.222 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:09:19.222 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:09:19.222 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T04:09:19.222 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T04:09:19.222 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:09:19.223 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:09:19.223 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:09:19.223 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:09:19.223 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:09:19.223 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:09:19.223 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T04:09:19.223 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T04:09:19.223 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:09:19.223 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:09:19.223 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:09:19.223 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:09:19.223 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:09:19.223 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:09:19.223 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T04:09:19.224 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:09:19.224 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:09:19.224 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:09:19.224 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 4 2024-09-16T04:09:19.224 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:09:19.224 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:09:19.224 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T04:09:19.224 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:09:19.224 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:09:19.224 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:09:19.224 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 3 2024-09-16T04:09:19.224 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:09:19.224 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:09:19.224 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:09:19.225 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T04:09:19.225 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T04:09:19.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.192+0000 7f068589d640 10 librados: watch_flush enter 2024-09-16T04:09:19.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.192+0000 7f068589d640 10 librados: watch_flush exit 2024-09-16T04:09:19.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.192+0000 7f068589d640 20 client.1034187.objecter shutdown clearing up homeless session... 2024-09-16T04:09:19.227 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.192+0000 7f068589d640 10 client.1034187.objecter successfully canceled tick 2024-09-16T04:09:19.227 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.192+0000 7f068589d640 1 -- 172.21.15.42:0/4064430655 >> v2:172.21.15.154:6812/254845342 conn(0x7f066c0377f0 msgr2=0x7f066c039cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:19.227 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.192+0000 7f068589d640 1 --2- 172.21.15.42:0/4064430655 >> v2:172.21.15.154:6812/254845342 conn(0x7f066c0377f0 0x7f066c039cb0 secure :-1 s=READY pgs=3913 cs=0 l=1 rev1=1 crypto rx=0x7f0680133830 tx=0x7f067800b040 comp rx=0 tx=0).stop 2024-09-16T04:09:19.227 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.192+0000 7f068589d640 10 monclient: shutdown 2024-09-16T04:09:19.227 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.192+0000 7f068589d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:19.227 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.192+0000 7f068589d640 1 -- 172.21.15.42:0/4064430655 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f06800a3d90 msgr2=0x7f06800a2d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:19.227 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.192+0000 7f068589d640 1 --2- 172.21.15.42:0/4064430655 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f06800a3d90 0x7f06800a2d20 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f06740ae040 tx=0x7f067401aba0 comp rx=0 tx=0).stop 2024-09-16T04:09:19.227 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.192+0000 7f067f7fe640 1 -- 172.21.15.42:0/4064430655 reap_dead start 2024-09-16T04:09:19.227 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.192+0000 7f067f7fe640 1 -- 172.21.15.42:0/4064430655 reap_dead start 2024-09-16T04:09:19.227 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.192+0000 7f068589d640 1 -- 172.21.15.42:0/4064430655 shutdown_connections 2024-09-16T04:09:19.227 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.192+0000 7f068589d640 1 -- 172.21.15.42:0/4064430655 >> 172.21.15.42:0/4064430655 conn(0x7f06800a9fb0 msgr2=0x7f06800a8000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:19.227 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.192+0000 7f068589d640 1 -- 172.21.15.42:0/4064430655 shutdown_connections 2024-09-16T04:09:19.227 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.192+0000 7f068589d640 1 -- 172.21.15.42:0/4064430655 wait complete. 2024-09-16T04:09:19.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:19.192+0000 7f068589d640 1 librados: shutdown 2024-09-16T04:09:19.228 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T04:09:19.228 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-16T04:09:39.228 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-16T04:09:39.229 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-09-16T04:09:39.229 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-16T04:09:39.229 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-16T04:09:39.233 INFO:tasks.ceph.mon.b:Started 2024-09-16T04:09:39.233 INFO:tasks.mon_thrash:Sending STOP to mon c 2024-09-16T04:09:39.233 INFO:tasks.ceph.mon.c:Sent signal 19 2024-09-16T04:09:39.233 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-16T04:09:54.234 INFO:tasks.mon_thrash:Sending CONT to mon c 2024-09-16T04:09:54.235 INFO:tasks.ceph.mon.c:Sent signal 18 2024-09-16T04:09:54.235 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T04:09:54.235 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:09:54.367 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:54.364+0000 7f1955708640 10 monclient: get_monmap_and_config 2024-09-16T04:09:54.367 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:54.364+0000 7f1955708640 10 monclient: build_initial_monmap 2024-09-16T04:09:54.367 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:54.364+0000 7f1955708640 10 monclient: monmap: 2024-09-16T04:09:54.367 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:09:54.367 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:09:54.367 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:09:54.365693+0000 2024-09-16T04:09:54.367 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:09:54.365693+0000 2024-09-16T04:09:54.367 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:54.367 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:09:54.368 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:09:54.368 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:09:54.368 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:09:54.368 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:09:54.368 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:09:54.368 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:09:54.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:54.364+0000 7f1955708640 1 Processor -- start 2024-09-16T04:09:54.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:54.364+0000 7f1955708640 1 -- start start 2024-09-16T04:09:54.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:54.364+0000 7f1955708640 10 monclient: init 2024-09-16T04:09:54.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:54.364+0000 7f1955708640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:54.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:54.364+0000 7f1955708640 10 monclient: _add_conns ranks=[3,4,2,0,1] 2024-09-16T04:09:54.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:54.364+0000 7f1955708640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f195006db90 0x7f19501340c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:09:54.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:54.364+0000 7f1955708640 10 monclient(hunting): picked mon.noname-a con 0x7f195006db90 addr v2:172.21.15.42:3300/0 2024-09-16T04:09:54.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:54.364+0000 7f1955708640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f19500a37f0 0x7f19500a3c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:09:54.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:54.364+0000 7f1955708640 10 monclient(hunting): picked mon.noname-c con 0x7f19500a37f0 addr v2:172.21.15.154:3300/0 2024-09-16T04:09:54.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:54.364+0000 7f1955708640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1950136e10 0x7f1950137220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:09:54.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:54.364+0000 7f1955708640 10 monclient(hunting): picked mon.noname-b con 0x7f1950136e10 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:09:54.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:54.364+0000 7f1955708640 10 monclient(hunting): start opening mon connection 2024-09-16T04:09:54.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:54.364+0000 7f1955708640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f195006df70 con 0x7f195006db90 2024-09-16T04:09:54.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:54.364+0000 7f1955708640 10 monclient(hunting): start opening mon connection 2024-09-16T04:09:54.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:54.364+0000 7f1955708640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f195006ece0 con 0x7f19500a37f0 2024-09-16T04:09:54.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:54.364+0000 7f1955708640 10 monclient(hunting): start opening mon connection 2024-09-16T04:09:54.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:54.364+0000 7f1955708640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f195009bf80 con 0x7f1950136e10 2024-09-16T04:09:54.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:54.364+0000 7f1955708640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:54.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:54.364+0000 7f1955708640 10 monclient(hunting): authenticate will time out at 2641.335190s 2024-09-16T04:09:54.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:54.364+0000 7f194effd640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f195006db90 0x7f19501340c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:09:54.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:54.364+0000 7f194f7fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1950136e10 0x7f1950137220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:09:54.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:54.364+0000 7f194effd640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f195006db90 0x7f19501340c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:59908/0 (socket says 172.21.15.42:59908) 2024-09-16T04:09:54.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:54.364+0000 7f194effd640 1 -- 172.21.15.42:0/1931961705 learned_addr learned my addr 172.21.15.42:0/1931961705 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:09:54.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:54.364+0000 7f194f7fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1950136e10 0x7f1950137220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:51076/0 (socket says 172.21.15.42:51076) 2024-09-16T04:09:54.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:54.364+0000 7f194effd640 10 monclient(hunting): get_auth_request con 0x7f195006db90 auth_method 0 2024-09-16T04:09:54.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:54.364+0000 7f194effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:09:54.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:54.364+0000 7f194effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:54.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:54.364+0000 7f194effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:54.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:54.364+0000 7f194f7fe640 10 monclient(hunting): get_auth_request con 0x7f1950136e10 auth_method 0 2024-09-16T04:09:54.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:54.364+0000 7f194f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:09:54.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:54.364+0000 7f194f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:54.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:54.364+0000 7f194f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:54.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:54.364+0000 7f194f7fe640 1 -- 172.21.15.42:0/1931961705 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1950136e10 msgr2=0x7f1950137220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T04:09:54.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:54.364+0000 7f194effd640 1 -- 172.21.15.42:0/1931961705 >> v2:172.21.15.42:3300/0 conn(0x7f195006db90 msgr2=0x7f19501340c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:09:54.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:54.364+0000 7f194effd640 1 -- 172.21.15.42:0/1931961705 >> v2:172.21.15.42:3300/0 conn(0x7f195006db90 msgr2=0x7f19501340c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:09:54.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:54.364+0000 7f194f7fe640 1 -- 172.21.15.42:0/1931961705 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1950136e10 msgr2=0x7f1950137220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:09:54.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:54.364+0000 7f194effd640 1 --2- 172.21.15.42:0/1931961705 >> v2:172.21.15.42:3300/0 conn(0x7f195006db90 0x7f19501340c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:09:54.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:54.364+0000 7f194f7fe640 1 --2- 172.21.15.42:0/1931961705 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1950136e10 0x7f1950137220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:09:54.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:54.364+0000 7f194effd640 1 --2- 172.21.15.42:0/1931961705 >> v2:172.21.15.42:3300/0 conn(0x7f195006db90 0x7f19501340c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:09:54.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:54.364+0000 7f194f7fe640 1 --2- 172.21.15.42:0/1931961705 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1950136e10 0x7f1950137220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:09:54.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:54.364+0000 7f194e7fc640 1 --2- 172.21.15.42:0/1931961705 >> v2:172.21.15.154:3300/0 conn(0x7f19500a37f0 0x7f19500a3c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:09:54.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:54.364+0000 7f194e7fc640 10 monclient(hunting): get_auth_request con 0x7f19500a37f0 auth_method 0 2024-09-16T04:09:54.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:54.364+0000 7f194e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:09:54.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:54.364+0000 7f194e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:54.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:54.364+0000 7f194e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:54.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:54.364+0000 7f194e7fc640 1 -- 172.21.15.42:0/1931961705 >> v2:172.21.15.154:3300/0 conn(0x7f19500a37f0 msgr2=0x7f19500a3c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:09:54.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:54.364+0000 7f194e7fc640 1 -- 172.21.15.42:0/1931961705 >> v2:172.21.15.154:3300/0 conn(0x7f19500a37f0 msgr2=0x7f19500a3c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:09:54.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:54.364+0000 7f194e7fc640 1 --2- 172.21.15.42:0/1931961705 >> v2:172.21.15.154:3300/0 conn(0x7f19500a37f0 0x7f19500a3c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:09:54.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:54.368+0000 7f194e7fc640 1 --2- 172.21.15.42:0/1931961705 >> v2:172.21.15.154:3300/0 conn(0x7f19500a37f0 0x7f19500a3c00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:09:54.570 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:54.568+0000 7f194f7fe640 1 --2- 172.21.15.42:0/1931961705 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1950136e10 0x7f1950137220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:09:54.570 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:54.568+0000 7f194effd640 1 --2- 172.21.15.42:0/1931961705 >> v2:172.21.15.42:3300/0 conn(0x7f195006db90 0x7f19501340c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:09:54.570 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:54.568+0000 7f194f7fe640 10 monclient(hunting): get_auth_request con 0x7f1950136e10 auth_method 0 2024-09-16T04:09:54.570 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:54.568+0000 7f194f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:09:54.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:54.568+0000 7f194f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:54.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:54.568+0000 7f194f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:09:54.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:54.568+0000 7f194effd640 10 monclient(hunting): get_auth_request con 0x7f195006db90 auth_method 0 2024-09-16T04:09:54.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:54.568+0000 7f194effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:09:54.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:54.568+0000 7f194effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:54.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:54.568+0000 7f194effd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:09:54.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:54.568+0000 7f194effd640 1 -- 172.21.15.42:0/1931961705 >> v2:172.21.15.42:3300/0 conn(0x7f195006db90 msgr2=0x7f19501340c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:09:54.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:54.568+0000 7f194effd640 1 -- 172.21.15.42:0/1931961705 >> v2:172.21.15.42:3300/0 conn(0x7f195006db90 msgr2=0x7f19501340c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:09:54.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:54.568+0000 7f194effd640 1 --2- 172.21.15.42:0/1931961705 >> v2:172.21.15.42:3300/0 conn(0x7f195006db90 0x7f19501340c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:09:54.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:54.568+0000 7f194f7fe640 1 -- 172.21.15.42:0/1931961705 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1950136e10 msgr2=0x7f1950137220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:09:54.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:54.568+0000 7f194f7fe640 1 -- 172.21.15.42:0/1931961705 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1950136e10 msgr2=0x7f1950137220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:09:54.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:54.568+0000 7f194f7fe640 1 --2- 172.21.15.42:0/1931961705 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1950136e10 0x7f1950137220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:09:54.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:54.568+0000 7f194f7fe640 1 --2- 172.21.15.42:0/1931961705 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1950136e10 0x7f1950137220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T04:09:54.572 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:54.568+0000 7f194effd640 1 --2- 172.21.15.42:0/1931961705 >> v2:172.21.15.42:3300/0 conn(0x7f195006db90 0x7f19501340c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T04:09:54.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:54.572+0000 7f194e7fc640 1 --2- 172.21.15.42:0/1931961705 >> v2:172.21.15.154:3300/0 conn(0x7f19500a37f0 0x7f19500a3c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:09:54.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:54.572+0000 7f194e7fc640 10 monclient(hunting): get_auth_request con 0x7f19500a37f0 auth_method 0 2024-09-16T04:09:54.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:54.572+0000 7f194e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:09:54.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:54.572+0000 7f194e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:54.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:54.572+0000 7f194e7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:09:54.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:54.572+0000 7f194e7fc640 1 -- 172.21.15.42:0/1931961705 >> v2:172.21.15.154:3300/0 conn(0x7f19500a37f0 msgr2=0x7f19500a3c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:09:54.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:54.572+0000 7f194e7fc640 1 -- 172.21.15.42:0/1931961705 >> v2:172.21.15.154:3300/0 conn(0x7f19500a37f0 msgr2=0x7f19500a3c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:09:54.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:54.572+0000 7f194e7fc640 1 --2- 172.21.15.42:0/1931961705 >> v2:172.21.15.154:3300/0 conn(0x7f19500a37f0 0x7f19500a3c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:09:54.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:54.572+0000 7f194e7fc640 1 --2- 172.21.15.42:0/1931961705 >> v2:172.21.15.154:3300/0 conn(0x7f19500a37f0 0x7f19500a3c00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T04:09:54.974 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:54.972+0000 7f194f7fe640 1 --2- 172.21.15.42:0/1931961705 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1950136e10 0x7f1950137220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:09:54.975 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:54.972+0000 7f194effd640 1 --2- 172.21.15.42:0/1931961705 >> v2:172.21.15.42:3300/0 conn(0x7f195006db90 0x7f19501340c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:09:54.975 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:54.972+0000 7f194f7fe640 10 monclient(hunting): get_auth_request con 0x7f1950136e10 auth_method 0 2024-09-16T04:09:54.975 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:54.972+0000 7f194f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:09:54.975 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:54.972+0000 7f194f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:54.975 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:54.972+0000 7f194f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:09:54.975 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:54.972+0000 7f194effd640 10 monclient(hunting): get_auth_request con 0x7f195006db90 auth_method 0 2024-09-16T04:09:54.975 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:54.972+0000 7f194effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:09:54.975 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:54.972+0000 7f194effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:54.975 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:54.972+0000 7f194effd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:09:54.975 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:54.972+0000 7f194f7fe640 1 -- 172.21.15.42:0/1931961705 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1950136e10 msgr2=0x7f1950137220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:09:54.975 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:54.972+0000 7f194f7fe640 1 -- 172.21.15.42:0/1931961705 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1950136e10 msgr2=0x7f1950137220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:09:54.975 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:54.972+0000 7f194f7fe640 1 --2- 172.21.15.42:0/1931961705 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1950136e10 0x7f1950137220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:09:54.976 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:54.972+0000 7f194effd640 1 -- 172.21.15.42:0/1931961705 >> v2:172.21.15.42:3300/0 conn(0x7f195006db90 msgr2=0x7f19501340c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:09:54.976 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:54.972+0000 7f194effd640 1 -- 172.21.15.42:0/1931961705 >> v2:172.21.15.42:3300/0 conn(0x7f195006db90 msgr2=0x7f19501340c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:09:54.976 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:54.972+0000 7f194effd640 1 --2- 172.21.15.42:0/1931961705 >> v2:172.21.15.42:3300/0 conn(0x7f195006db90 0x7f19501340c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:09:54.976 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:54.972+0000 7f194f7fe640 1 --2- 172.21.15.42:0/1931961705 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1950136e10 0x7f1950137220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T04:09:54.976 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:54.972+0000 7f194effd640 1 --2- 172.21.15.42:0/1931961705 >> v2:172.21.15.42:3300/0 conn(0x7f195006db90 0x7f19501340c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T04:09:54.978 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:54.976+0000 7f194e7fc640 1 --2- 172.21.15.42:0/1931961705 >> v2:172.21.15.154:3300/0 conn(0x7f19500a37f0 0x7f19500a3c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:09:54.978 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:54.976+0000 7f194e7fc640 10 monclient(hunting): get_auth_request con 0x7f19500a37f0 auth_method 0 2024-09-16T04:09:54.978 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:54.976+0000 7f194e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:09:54.978 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:54.976+0000 7f194e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:54.978 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:54.976+0000 7f194e7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:09:54.978 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:54.976+0000 7f194e7fc640 1 -- 172.21.15.42:0/1931961705 >> v2:172.21.15.154:3300/0 conn(0x7f19500a37f0 msgr2=0x7f19500a3c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:09:54.978 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:54.976+0000 7f194e7fc640 1 -- 172.21.15.42:0/1931961705 >> v2:172.21.15.154:3300/0 conn(0x7f19500a37f0 msgr2=0x7f19500a3c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:09:54.978 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:54.976+0000 7f194e7fc640 1 --2- 172.21.15.42:0/1931961705 >> v2:172.21.15.154:3300/0 conn(0x7f19500a37f0 0x7f19500a3c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:09:54.978 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:54.976+0000 7f194e7fc640 1 --2- 172.21.15.42:0/1931961705 >> v2:172.21.15.154:3300/0 conn(0x7f19500a37f0 0x7f19500a3c00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T04:09:55.778 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:55.776+0000 7f194f7fe640 1 --2- 172.21.15.42:0/1931961705 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1950136e10 0x7f1950137220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:09:55.779 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:55.776+0000 7f194effd640 1 --2- 172.21.15.42:0/1931961705 >> v2:172.21.15.42:3300/0 conn(0x7f195006db90 0x7f19501340c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:09:55.779 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:55.776+0000 7f194f7fe640 10 monclient(hunting): get_auth_request con 0x7f1950136e10 auth_method 0 2024-09-16T04:09:55.779 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:55.776+0000 7f194f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:09:55.779 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:55.776+0000 7f194f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:55.779 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:55.776+0000 7f194f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:09:55.779 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:55.776+0000 7f194effd640 10 monclient(hunting): get_auth_request con 0x7f195006db90 auth_method 0 2024-09-16T04:09:55.779 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:55.776+0000 7f194effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:09:55.779 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:55.776+0000 7f194effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:55.779 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:55.776+0000 7f194effd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:09:55.779 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:55.776+0000 7f194f7fe640 1 -- 172.21.15.42:0/1931961705 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1950136e10 msgr2=0x7f1950137220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:09:55.779 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:55.776+0000 7f194f7fe640 1 -- 172.21.15.42:0/1931961705 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1950136e10 msgr2=0x7f1950137220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:09:55.779 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:55.776+0000 7f194f7fe640 1 --2- 172.21.15.42:0/1931961705 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1950136e10 0x7f1950137220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:09:55.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:55.776+0000 7f194effd640 1 -- 172.21.15.42:0/1931961705 >> v2:172.21.15.42:3300/0 conn(0x7f195006db90 msgr2=0x7f19501340c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:09:55.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:55.776+0000 7f194effd640 1 -- 172.21.15.42:0/1931961705 >> v2:172.21.15.42:3300/0 conn(0x7f195006db90 msgr2=0x7f19501340c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:09:55.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:55.776+0000 7f194effd640 1 --2- 172.21.15.42:0/1931961705 >> v2:172.21.15.42:3300/0 conn(0x7f195006db90 0x7f19501340c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:09:55.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:55.776+0000 7f194effd640 1 --2- 172.21.15.42:0/1931961705 >> v2:172.21.15.42:3300/0 conn(0x7f195006db90 0x7f19501340c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T04:09:55.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:55.776+0000 7f194f7fe640 1 --2- 172.21.15.42:0/1931961705 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1950136e10 0x7f1950137220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T04:09:55.782 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:55.780+0000 7f194e7fc640 1 --2- 172.21.15.42:0/1931961705 >> v2:172.21.15.154:3300/0 conn(0x7f19500a37f0 0x7f19500a3c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:09:55.782 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:55.780+0000 7f194e7fc640 10 monclient(hunting): get_auth_request con 0x7f19500a37f0 auth_method 0 2024-09-16T04:09:55.782 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:55.780+0000 7f194e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:09:55.782 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:55.780+0000 7f194e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:55.782 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:55.780+0000 7f194e7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:09:55.782 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:55.780+0000 7f194e7fc640 1 -- 172.21.15.42:0/1931961705 >> v2:172.21.15.154:3300/0 conn(0x7f19500a37f0 msgr2=0x7f19500a3c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:09:55.782 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:55.780+0000 7f194e7fc640 1 -- 172.21.15.42:0/1931961705 >> v2:172.21.15.154:3300/0 conn(0x7f19500a37f0 msgr2=0x7f19500a3c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:09:55.782 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:55.780+0000 7f194e7fc640 1 --2- 172.21.15.42:0/1931961705 >> v2:172.21.15.154:3300/0 conn(0x7f19500a37f0 0x7f19500a3c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:09:55.782 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:55.780+0000 7f194e7fc640 1 --2- 172.21.15.42:0/1931961705 >> v2:172.21.15.154:3300/0 conn(0x7f19500a37f0 0x7f19500a3c00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T04:09:57.369 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:57.364+0000 7f194cff9640 10 monclient(hunting): tick 2024-09-16T04:09:57.369 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:57.364+0000 7f194cff9640 10 monclient(hunting): _check_auth_tickets 2024-09-16T04:09:57.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:57.364+0000 7f194cff9640 1 monclient(hunting): continuing hunt 2024-09-16T04:09:57.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:57.364+0000 7f194cff9640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T04:09:57.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:57.364+0000 7f194cff9640 1 -- 172.21.15.42:0/1931961705 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1950136e10 msgr2=0x7f1950137220 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:09:57.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:57.364+0000 7f194cff9640 1 --2- 172.21.15.42:0/1931961705 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1950136e10 0x7f1950137220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:09:57.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:57.364+0000 7f194cff9640 1 -- 172.21.15.42:0/1931961705 >> v2:172.21.15.154:3300/0 conn(0x7f19500a37f0 msgr2=0x7f19500a3c00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:09:57.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:57.364+0000 7f194cff9640 1 --2- 172.21.15.42:0/1931961705 >> v2:172.21.15.154:3300/0 conn(0x7f19500a37f0 0x7f19500a3c00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:09:57.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:57.364+0000 7f194cff9640 1 -- 172.21.15.42:0/1931961705 >> v2:172.21.15.42:3300/0 conn(0x7f195006db90 msgr2=0x7f19501340c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:09:57.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:57.364+0000 7f194cff9640 1 --2- 172.21.15.42:0/1931961705 >> v2:172.21.15.42:3300/0 conn(0x7f195006db90 0x7f19501340c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:09:57.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:57.364+0000 7f194cff9640 10 monclient: _add_conns ranks=[0,1] 2024-09-16T04:09:57.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:57.364+0000 7f194cff9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f19380036d0 addr v1:172.21.15.42:6789/0 2024-09-16T04:09:57.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:57.364+0000 7f194cff9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1938007400 addr v1:172.21.15.154:6789/0 2024-09-16T04:09:57.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:57.364+0000 7f194cff9640 1 -- 172.21.15.42:0/1931961705 --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1950134780 con 0x7f19380036d0 2024-09-16T04:09:57.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:57.364+0000 7f194cff9640 1 -- 172.21.15.42:0/1931961705 --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f193800c140 con 0x7f1938007400 2024-09-16T04:09:57.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:57.364+0000 7f194cff9640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:59.148 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.144+0000 7f194dffb640 1 -- 172.21.15.42:0/1931961705 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 113458390 0 0) 0x7f1950134780 con 0x7f19380036d0 2024-09-16T04:09:59.148 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.144+0000 7f194dffb640 10 monclient(hunting): _negotiate 2024-09-16T04:09:59.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.144+0000 7f194dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:59.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.144+0000 7f194dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:59.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.144+0000 7f194dffb640 10 monclient(hunting): my global_id is 1034190 2024-09-16T04:09:59.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.144+0000 7f194dffb640 1 -- 172.21.15.42:0/1931961705 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f193c003530 con 0x7f19380036d0 2024-09-16T04:09:59.157 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.156+0000 7f194dffb640 1 -- 172.21.15.42:0/1931961705 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3677203715 0 0) 0x7f193c003530 con 0x7f19380036d0 2024-09-16T04:09:59.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.156+0000 7f194dffb640 1 -- 172.21.15.42:0/1931961705 >> v1:172.21.15.154:6789/0 conn(0x7f1938007400 legacy=0x7f19380098a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:59.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.156+0000 7f194dffb640 10 monclient: _finish_hunting 0 2024-09-16T04:09:59.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.156+0000 7f194dffb640 1 monclient: found mon.noname-a-legacy 2024-09-16T04:09:59.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.156+0000 7f194dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:59.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.156+0000 7f194dffb640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.42:6789/0 2024-09-16T04:09:59.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.156+0000 7f194dffb640 1 -- 172.21.15.42:0/1931961705 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f193800d440 con 0x7f19380036d0 2024-09-16T04:09:59.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.156+0000 7f194dffb640 10 monclient: _finish_auth 0 2024-09-16T04:09:59.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.156+0000 7f194dffb640 10 monclient: _check_auth_tickets 2024-09-16T04:09:59.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.156+0000 7f194dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:59.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.156+0000 7f194dffb640 1 -- 172.21.15.42:0/1931961705 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2177413734 0 0) 0x7f194400f040 con 0x7f19380036d0 2024-09-16T04:09:59.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.156+0000 7f194dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:59.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.156+0000 7f194dffb640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T04:09:59.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.156+0000 7f194dffb640 10 monclient: dump: 2024-09-16T04:09:59.159 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:09:59.159 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:09:59.159 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:09:59.159 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:09:59.159 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:59.159 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:09:59.159 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:09:59.159 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:09:59.159 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:09:59.159 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:09:59.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.156+0000 7f194dffb640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] but i'm connected to v1:172.21.15.42:6789/0, reconnecting 2024-09-16T04:09:59.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.156+0000 7f194dffb640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:59.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.156+0000 7f194dffb640 1 -- 172.21.15.42:0/1931961705 >> v1:172.21.15.42:6789/0 conn(0x7f19380036d0 legacy=0x7f1938005b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:59.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.156+0000 7f194dffb640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:09:59.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.156+0000 7f194dffb640 1 --2- 172.21.15.42:0/1931961705 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f193c003a90 0x7f193c003e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:09:59.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.156+0000 7f194f7fe640 1 -- 172.21.15.42:0/1931961705 reap_dead start 2024-09-16T04:09:59.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.156+0000 7f194dffb640 10 monclient(hunting): picked mon.a con 0x7f193c003a90 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:09:59.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.156+0000 7f194dffb640 1 --2- 172.21.15.42:0/1931961705 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f193c004810 0x7f193c004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:09:59.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.156+0000 7f194dffb640 10 monclient(hunting): picked mon.c con 0x7f193c004810 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:09:59.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.156+0000 7f194dffb640 1 --2- 172.21.15.42:0/1931961705 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f193c0099c0 0x7f193c00bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:09:59.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.156+0000 7f194dffb640 10 monclient(hunting): picked mon.b con 0x7f193c0099c0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:09:59.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.156+0000 7f194dffb640 10 monclient(hunting): start opening mon connection 2024-09-16T04:09:59.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.156+0000 7f194dffb640 1 -- 172.21.15.42:0/1931961705 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f193c00c3a0 con 0x7f193c003a90 2024-09-16T04:09:59.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.156+0000 7f194dffb640 10 monclient(hunting): start opening mon connection 2024-09-16T04:09:59.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.156+0000 7f194dffb640 1 -- 172.21.15.42:0/1931961705 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f193c00c550 con 0x7f193c0099c0 2024-09-16T04:09:59.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.156+0000 7f194dffb640 10 monclient(hunting): start opening mon connection 2024-09-16T04:09:59.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.156+0000 7f194dffb640 1 -- 172.21.15.42:0/1931961705 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f193c00c750 con 0x7f193c004810 2024-09-16T04:09:59.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.156+0000 7f194dffb640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:59.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.156+0000 7f194dffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:09:59.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.156+0000 7f194effd640 1 --2- 172.21.15.42:0/1931961705 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f193c003a90 0x7f193c003e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:09:59.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.156+0000 7f194e7fc640 1 --2- 172.21.15.42:0/1931961705 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f193c004810 0x7f193c004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:09:59.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.156+0000 7f194effd640 10 monclient(hunting): get_auth_request con 0x7f193c003a90 auth_method 0 2024-09-16T04:09:59.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.156+0000 7f194effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:09:59.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.156+0000 7f194effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:59.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.156+0000 7f194effd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:09:59.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.156+0000 7f194e7fc640 10 monclient(hunting): get_auth_request con 0x7f193c004810 auth_method 0 2024-09-16T04:09:59.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.156+0000 7f194e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:09:59.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.156+0000 7f194e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:59.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.156+0000 7f194e7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:09:59.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.156+0000 7f194f7fe640 1 --2- 172.21.15.42:0/1931961705 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f193c0099c0 0x7f193c00bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:09:59.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.156+0000 7f194effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:09:59.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.156+0000 7f194effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:09:59.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.156+0000 7f194effd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:09:59.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.156+0000 7f194f7fe640 10 monclient(hunting): get_auth_request con 0x7f193c0099c0 auth_method 0 2024-09-16T04:09:59.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.156+0000 7f194f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:09:59.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.156+0000 7f194f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:59.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.156+0000 7f194f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:09:59.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.156+0000 7f194e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:09:59.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.156+0000 7f194e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:09:59.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.156+0000 7f194e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:09:59.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.156+0000 7f194effd640 10 monclient(hunting): handle_auth_done global_id 1034190 payload 293 2024-09-16T04:09:59.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.156+0000 7f194effd640 1 -- 172.21.15.42:0/1931961705 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f193c004810 msgr2=0x7f193c004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:09:59.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.156+0000 7f194effd640 1 --2- 172.21.15.42:0/1931961705 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f193c004810 0x7f193c004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:09:59.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.156+0000 7f194effd640 1 -- 172.21.15.42:0/1931961705 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f193c0099c0 msgr2=0x7f193c00bdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:09:59.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.156+0000 7f194effd640 1 --2- 172.21.15.42:0/1931961705 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f193c0099c0 0x7f193c00bdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:09:59.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.156+0000 7f194effd640 10 monclient: _finish_hunting 0 2024-09-16T04:09:59.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.156+0000 7f194effd640 1 monclient: found mon.a 2024-09-16T04:09:59.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.156+0000 7f194effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:59.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.156+0000 7f194effd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:09:59.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.156+0000 7f194effd640 1 -- 172.21.15.42:0/1931961705 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f193c003530 con 0x7f193c003a90 2024-09-16T04:09:59.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.156+0000 7f194effd640 10 monclient: _finish_auth 0 2024-09-16T04:09:59.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.156+0000 7f194effd640 10 monclient: _check_auth_tickets 2024-09-16T04:09:59.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.156+0000 7f194effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:59.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.156+0000 7f1955708640 5 monclient: authenticate success, global_id 1034190 2024-09-16T04:09:59.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.156+0000 7f1955708640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:09:59.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.156+0000 7f194f7fe640 1 --2- 172.21.15.42:0/1931961705 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f193c0099c0 0x7f193c00bdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T04:09:59.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.160+0000 7f194effd640 1 --2- 172.21.15.42:0/1931961705 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f193c003a90 0x7f193c003e90 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f194400cfc0 tx=0x7f194401d9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe32f0d50bfab519 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:09:59.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.160+0000 7f194dffb640 1 -- 172.21.15.42:0/1931961705 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19440af020 con 0x7f193c003a90 2024-09-16T04:09:59.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.160+0000 7f194dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:59.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.160+0000 7f194dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:09:59.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.160+0000 7f194dffb640 10 monclient: dump: 2024-09-16T04:09:59.171 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:09:59.171 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:09:59.172 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:09:59.172 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:09:59.172 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:59.172 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:09:59.172 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:09:59.172 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:09:59.172 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:09:59.173 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:09:59.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.160+0000 7f194dffb640 1 -- 172.21.15.42:0/1931961705 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f19440a7670 con 0x7f193c003a90 2024-09-16T04:09:59.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.160+0000 7f1955708640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:09:59.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.160+0000 7f194dffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:59.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.160+0000 7f1955708640 10 monclient: get_monmap_and_config success 2024-09-16T04:09:59.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.160+0000 7f1955708640 4 set_mon_vals no callback set 2024-09-16T04:09:59.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.160+0000 7f194dffb640 1 -- 172.21.15.42:0/1931961705 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f194400cc10 con 0x7f193c003a90 2024-09-16T04:09:59.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.160+0000 7f194dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:59.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.160+0000 7f194dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:09:59.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.160+0000 7f194dffb640 10 monclient: dump: 2024-09-16T04:09:59.173 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:09:59.173 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:09:59.173 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:09:59.174 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:09:59.174 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:59.174 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:09:59.174 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:09:59.174 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:09:59.174 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:09:59.174 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:09:59.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.160+0000 7f1955708640 10 monclient: shutdown 2024-09-16T04:09:59.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.160+0000 7f1955708640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:59.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.160+0000 7f1955708640 1 -- 172.21.15.42:0/1931961705 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f193c003a90 msgr2=0x7f193c003e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:59.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.160+0000 7f1955708640 1 --2- 172.21.15.42:0/1931961705 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f193c003a90 0x7f193c003e90 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f194400cfc0 tx=0x7f194401d9b0 comp rx=0 tx=0).stop 2024-09-16T04:09:59.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.160+0000 7f1955708640 1 -- 172.21.15.42:0/1931961705 shutdown_connections 2024-09-16T04:09:59.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.160+0000 7f1955708640 1 --2- 172.21.15.42:0/1931961705 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f193c0099c0 0x7f193c00bdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:09:59.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.160+0000 7f1955708640 1 --2- 172.21.15.42:0/1931961705 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f193c004810 0x7f193c004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:09:59.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.160+0000 7f1955708640 1 --2- 172.21.15.42:0/1931961705 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f193c003a90 0x7f193c003e90 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:09:59.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.160+0000 7f1955708640 1 -- 172.21.15.42:0/1931961705 >> 172.21.15.42:0/1931961705 conn(0x7f19500a9fb0 msgr2=0x7f19500aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:59.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.160+0000 7f1955708640 1 -- 172.21.15.42:0/1931961705 shutdown_connections 2024-09-16T04:09:59.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.160+0000 7f1955708640 1 -- 172.21.15.42:0/1931961705 wait complete. 2024-09-16T04:09:59.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.164+0000 7f1955708640 10 monclient: build_initial_monmap 2024-09-16T04:09:59.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.164+0000 7f1955708640 10 monclient: monmap: 2024-09-16T04:09:59.175 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:09:59.175 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:09:59.175 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:09:59.175 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:09:59.175 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:59.175 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:09:59.176 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:09:59.176 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:09:59.176 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:09:59.176 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:09:59.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.164+0000 7f1955708640 1 librados: starting msgr at 2024-09-16T04:09:59.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.164+0000 7f1955708640 1 librados: starting objecter 2024-09-16T04:09:59.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.164+0000 7f1955708640 1 Processor -- start 2024-09-16T04:09:59.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.164+0000 7f1955708640 1 -- start start 2024-09-16T04:09:59.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.164+0000 7f1955708640 1 librados: setting wanted keys 2024-09-16T04:09:59.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.164+0000 7f1955708640 1 librados: calling monclient init 2024-09-16T04:09:59.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.164+0000 7f1955708640 10 monclient: init 2024-09-16T04:09:59.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.164+0000 7f1955708640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:59.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.164+0000 7f1955708640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:09:59.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.164+0000 7f1955708640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f193c003a90 0x7f19500a20b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:09:59.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.164+0000 7f1955708640 10 monclient(hunting): picked mon.noname-c con 0x7f193c003a90 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:09:59.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.164+0000 7f1955708640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f193c004810 0x7f19500a25f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:09:59.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.164+0000 7f1955708640 10 monclient(hunting): picked mon.noname-b con 0x7f193c004810 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:09:59.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.164+0000 7f1955708640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f193c0099c0 0x7f19500a2b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:09:59.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.164+0000 7f1955708640 10 monclient(hunting): picked mon.noname-a con 0x7f193c0099c0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:09:59.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.164+0000 7f1955708640 10 monclient(hunting): start opening mon connection 2024-09-16T04:09:59.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.164+0000 7f1955708640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f1950137bb0 con 0x7f193c0099c0 2024-09-16T04:09:59.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.164+0000 7f1955708640 10 monclient(hunting): start opening mon connection 2024-09-16T04:09:59.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.164+0000 7f1955708640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f1950137a30 con 0x7f193c004810 2024-09-16T04:09:59.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.164+0000 7f1955708640 10 monclient(hunting): start opening mon connection 2024-09-16T04:09:59.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.164+0000 7f1955708640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f195013a180 con 0x7f193c003a90 2024-09-16T04:09:59.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.164+0000 7f1955708640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:59.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.164+0000 7f1955708640 10 monclient(hunting): authenticate will time out at 2646.133727s 2024-09-16T04:09:59.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.164+0000 7f194effd640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f193c003a90 0x7f19500a20b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:09:59.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.164+0000 7f194f7fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f193c0099c0 0x7f19500a2b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:09:59.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.164+0000 7f194effd640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f193c003a90 0x7f19500a20b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:51096/0 (socket says 172.21.15.42:51096) 2024-09-16T04:09:59.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.164+0000 7f194effd640 1 -- 172.21.15.42:0/630114140 learned_addr learned my addr 172.21.15.42:0/630114140 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:09:59.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.164+0000 7f194effd640 10 monclient(hunting): get_auth_request con 0x7f193c003a90 auth_method 0 2024-09-16T04:09:59.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.164+0000 7f194effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:09:59.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.164+0000 7f194effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:59.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.164+0000 7f194effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:59.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.164+0000 7f194f7fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f193c0099c0 0x7f19500a2b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:59962/0 (socket says 172.21.15.42:59962) 2024-09-16T04:09:59.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.164+0000 7f194f7fe640 10 monclient(hunting): get_auth_request con 0x7f193c0099c0 auth_method 0 2024-09-16T04:09:59.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.164+0000 7f194f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:09:59.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.164+0000 7f194f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:59.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.164+0000 7f194f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:59.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.164+0000 7f194e7fc640 1 --2- 172.21.15.42:0/630114140 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f193c004810 0x7f19500a25f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:09:59.180 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.164+0000 7f194e7fc640 10 monclient(hunting): get_auth_request con 0x7f193c004810 auth_method 0 2024-09-16T04:09:59.180 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.164+0000 7f194e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:09:59.180 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.164+0000 7f194e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:59.180 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.164+0000 7f194e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:59.180 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.164+0000 7f194effd640 1 -- 172.21.15.42:0/630114140 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f193c003a90 msgr2=0x7f19500a20b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-16T04:09:59.180 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.164+0000 7f194f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:09:59.180 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.164+0000 7f194f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:09:59.180 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.164+0000 7f194effd640 1 -- 172.21.15.42:0/630114140 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f193c003a90 msgr2=0x7f19500a20b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:09:59.180 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.164+0000 7f194effd640 1 --2- 172.21.15.42:0/630114140 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f193c003a90 0x7f19500a20b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:09:59.180 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.164+0000 7f194f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:09:59.180 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.164+0000 7f194effd640 1 --2- 172.21.15.42:0/630114140 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f193c003a90 0x7f19500a20b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:09:59.180 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.164+0000 7f194e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:09:59.180 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.164+0000 7f194e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:09:59.181 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.164+0000 7f194e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:09:59.181 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.164+0000 7f194f7fe640 10 monclient(hunting): handle_auth_done global_id 1034193 payload 824 2024-09-16T04:09:59.181 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.164+0000 7f194f7fe640 1 -- 172.21.15.42:0/630114140 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f193c003a90 msgr2=0x7f19500a20b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:09:59.181 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.164+0000 7f194f7fe640 1 --2- 172.21.15.42:0/630114140 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f193c003a90 0x7f19500a20b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:09:59.181 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.164+0000 7f194f7fe640 1 -- 172.21.15.42:0/630114140 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f193c004810 msgr2=0x7f19500a25f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:59.181 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.164+0000 7f194f7fe640 1 --2- 172.21.15.42:0/630114140 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f193c004810 0x7f19500a25f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:09:59.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.164+0000 7f194f7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:09:59.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.164+0000 7f194f7fe640 1 monclient: found mon.noname-a 2024-09-16T04:09:59.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.164+0000 7f194f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:59.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.164+0000 7f194f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:09:59.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.164+0000 7f194f7fe640 1 -- 172.21.15.42:0/630114140 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1950122490 con 0x7f193c0099c0 2024-09-16T04:09:59.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.164+0000 7f194f7fe640 1 --2- 172.21.15.42:0/630114140 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f193c0099c0 0x7f19500a2b30 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f1950069180 tx=0x7f1948004780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:09:59.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.164+0000 7f19377fe640 10 client.?.objecter ms_handle_connect 0x7f193c0099c0 2024-09-16T04:09:59.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.164+0000 7f19377fe640 10 client.?.objecter resend_mon_ops 2024-09-16T04:09:59.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.164+0000 7f19377fe640 1 -- 172.21.15.42:0/630114140 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f194801c030 con 0x7f193c0099c0 2024-09-16T04:09:59.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.164+0000 7f19377fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:59.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.164+0000 7f19377fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:09:59.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.164+0000 7f19377fe640 10 monclient: dump: 2024-09-16T04:09:59.183 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:09:59.183 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:09:59.183 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:09:59.183 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:09:59.183 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:59.183 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:09:59.183 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:09:59.183 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:09:59.183 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:09:59.183 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:09:59.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.164+0000 7f19377fe640 10 monclient: _finish_auth 0 2024-09-16T04:09:59.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.164+0000 7f19377fe640 10 monclient: _check_auth_tickets 2024-09-16T04:09:59.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.164+0000 7f19377fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:59.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.164+0000 7f19377fe640 1 -- 172.21.15.42:0/630114140 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1948015a30 con 0x7f193c0099c0 2024-09-16T04:09:59.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.164+0000 7f19377fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:59.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.164+0000 7f1955708640 5 monclient: authenticate success, global_id 1034193 2024-09-16T04:09:59.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.164+0000 7f19377fe640 1 -- 172.21.15.42:0/630114140 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1948015c00 con 0x7f193c0099c0 2024-09-16T04:09:59.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.164+0000 7f19377fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:59.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.164+0000 7f19377fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:09:59.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.164+0000 7f19377fe640 10 monclient: dump: 2024-09-16T04:09:59.184 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:09:59.184 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:09:59.185 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:09:59.185 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:09:59.185 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:59.185 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:09:59.185 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:09:59.185 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:09:59.185 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:09:59.185 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:09:59.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.164+0000 7f194d7fa640 4 set_mon_vals no callback set 2024-09-16T04:09:59.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.164+0000 7f1955708640 10 monclient: _renew_subs 2024-09-16T04:09:59.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.164+0000 7f1955708640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:09:59.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.164+0000 7f1955708640 1 -- 172.21.15.42:0/630114140 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19501208d0 con 0x7f193c0099c0 2024-09-16T04:09:59.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.164+0000 7f1955708640 10 client.1034193.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:09:59.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.164+0000 7f1955708640 10 monclient: _renew_subs 2024-09-16T04:09:59.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.164+0000 7f1955708640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:09:59.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.164+0000 7f1955708640 1 -- 172.21.15.42:0/630114140 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19500a8c70 con 0x7f193c0099c0 2024-09-16T04:09:59.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.164+0000 7f1955708640 1 librados: init done 2024-09-16T04:09:59.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.168+0000 7f19377fe640 1 -- 172.21.15.42:0/630114140 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f194801f030 con 0x7f193c0099c0 2024-09-16T04:09:59.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.168+0000 7f1955708640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:59.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.168+0000 7f1955708640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:59.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.168+0000 7f1955708640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:09:59.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.168+0000 7f1955708640 1 -- 172.21.15.42:0/630114140 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f195013a420 con 0x7f193c0099c0 2024-09-16T04:09:59.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.168+0000 7f19377fe640 1 --2- 172.21.15.42:0/630114140 >> v2:172.21.15.154:6812/254845342 conn(0x7f1938038330 0x7f193803a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:09:59.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.168+0000 7f19377fe640 1 -- 172.21.15.42:0/630114140 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(706..706 src has 451..706) ==== 5040+0+0 (secure 0 0 0) 0x7f1948049570 con 0x7f193c0099c0 2024-09-16T04:09:59.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.168+0000 7f19377fe640 10 client.1034193.objecter ms_dispatch 0x7f1950000ba0 osd_map(706..706 src has 451..706) 2024-09-16T04:09:59.187 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.168+0000 7f19377fe640 3 client.1034193.objecter handle_osd_map got epochs [706,706] > 0 2024-09-16T04:09:59.187 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.168+0000 7f19377fe640 3 client.1034193.objecter handle_osd_map decoding full epoch 706 2024-09-16T04:09:59.187 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.168+0000 7f19377fe640 20 client.1034193.objecter dump_active .. 0 homeless 2024-09-16T04:09:59.187 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.168+0000 7f194effd640 1 --2- 172.21.15.42:0/630114140 >> v2:172.21.15.154:6812/254845342 conn(0x7f1938038330 0x7f193803a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:09:59.187 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.168+0000 7f194effd640 10 monclient: get_auth_request con 0x7f1938038330 auth_method 0 2024-09-16T04:09:59.187 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.168+0000 7f194effd640 1 --2- 172.21.15.42:0/630114140 >> v2:172.21.15.154:6812/254845342 conn(0x7f1938038330 0x7f193803a7f0 secure :-1 s=READY pgs=3916 cs=0 l=1 rev1=1 crypto rx=0x7f1944000c10 tx=0x7f1944004740 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:09:59.187 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.168+0000 7f19377fe640 10 client.1034193.objecter ms_handle_connect 0x7f1938038330 2024-09-16T04:09:59.188 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.172+0000 7f19377fe640 1 -- 172.21.15.42:0/630114140 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f1948026070 con 0x7f193c0099c0 2024-09-16T04:09:59.188 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.172+0000 7f19377fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:59.188 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.172+0000 7f19377fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:09:59.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.400+0000 7f1955708640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:09:59.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.400+0000 7f1955708640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:09:59.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.400+0000 7f1955708640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:09:59.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.400+0000 7f1955708640 1 -- 172.21.15.42:0/630114140 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f1950120d60 con 0x7f193c0099c0 2024-09-16T04:09:59.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.404+0000 7f19377fe640 1 -- 172.21.15.42:0/630114140 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7f1948015da0 con 0x7f193c0099c0 2024-09-16T04:09:59.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.404+0000 7f19377fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:09:59.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.404+0000 7f19377fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:09:59.406 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":256,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:09:59.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.404+0000 7f1955708640 10 librados: watch_flush enter 2024-09-16T04:09:59.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.404+0000 7f1955708640 10 librados: watch_flush exit 2024-09-16T04:09:59.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.404+0000 7f1955708640 20 client.1034193.objecter shutdown clearing up homeless session... 2024-09-16T04:09:59.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.404+0000 7f1955708640 10 client.1034193.objecter successfully canceled tick 2024-09-16T04:09:59.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.404+0000 7f1955708640 1 -- 172.21.15.42:0/630114140 >> v2:172.21.15.154:6812/254845342 conn(0x7f1938038330 msgr2=0x7f193803a7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:59.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.404+0000 7f1955708640 1 --2- 172.21.15.42:0/630114140 >> v2:172.21.15.154:6812/254845342 conn(0x7f1938038330 0x7f193803a7f0 secure :-1 s=READY pgs=3916 cs=0 l=1 rev1=1 crypto rx=0x7f1944000c10 tx=0x7f1944004740 comp rx=0 tx=0).stop 2024-09-16T04:09:59.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.404+0000 7f1955708640 10 monclient: shutdown 2024-09-16T04:09:59.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.404+0000 7f1955708640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:59.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.404+0000 7f1955708640 1 -- 172.21.15.42:0/630114140 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f193c0099c0 msgr2=0x7f19500a2b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:59.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.404+0000 7f1955708640 1 --2- 172.21.15.42:0/630114140 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f193c0099c0 0x7f19500a2b30 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f1950069180 tx=0x7f1948004780 comp rx=0 tx=0).stop 2024-09-16T04:09:59.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.404+0000 7f1955708640 1 -- 172.21.15.42:0/630114140 shutdown_connections 2024-09-16T04:09:59.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.404+0000 7f1955708640 1 --2- 172.21.15.42:0/630114140 >> v2:172.21.15.154:6812/254845342 conn(0x7f1938038330 0x7f193803a7f0 unknown :-1 s=CLOSED pgs=3916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:09:59.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.404+0000 7f1955708640 1 --2- 172.21.15.42:0/630114140 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f193c0099c0 0x7f19500a2b30 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:09:59.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.404+0000 7f1955708640 1 --2- 172.21.15.42:0/630114140 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f193c004810 0x7f19500a25f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:09:59.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.404+0000 7f1955708640 1 --2- 172.21.15.42:0/630114140 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f193c003a90 0x7f19500a20b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:09:59.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.404+0000 7f1955708640 1 -- 172.21.15.42:0/630114140 >> 172.21.15.42:0/630114140 conn(0x7f19500a9fb0 msgr2=0x7f1950136460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:59.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.404+0000 7f1955708640 1 -- 172.21.15.42:0/630114140 shutdown_connections 2024-09-16T04:09:59.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.404+0000 7f1955708640 1 -- 172.21.15.42:0/630114140 wait complete. 2024-09-16T04:09:59.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.404+0000 7f1955708640 1 librados: shutdown 2024-09-16T04:09:59.424 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T04:09:59.425 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T04:09:59.425 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T04:09:59.544 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.540+0000 7f07e35e0640 10 monclient: get_monmap_and_config 2024-09-16T04:09:59.544 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.540+0000 7f07e35e0640 10 monclient: build_initial_monmap 2024-09-16T04:09:59.544 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.540+0000 7f07e35e0640 10 monclient: monmap: 2024-09-16T04:09:59.544 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:09:59.544 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:09:59.544 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:09:59.542692+0000 2024-09-16T04:09:59.544 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:09:59.542692+0000 2024-09-16T04:09:59.544 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:59.545 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:09:59.545 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:09:59.545 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:09:59.545 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:09:59.545 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:09:59.545 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:09:59.545 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:09:59.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.540+0000 7f07e35e0640 1 Processor -- start 2024-09-16T04:09:59.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.540+0000 7f07e35e0640 1 -- start start 2024-09-16T04:09:59.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.540+0000 7f07e35e0640 10 monclient: init 2024-09-16T04:09:59.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.540+0000 7f07e35e0640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:59.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.540+0000 7f07e35e0640 10 monclient: _add_conns ranks=[2,0,4,3,1] 2024-09-16T04:09:59.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.544+0000 7f07e35e0640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f07dc0a1330 0x7f07dc0a1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:09:59.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.544+0000 7f07e35e0640 10 monclient(hunting): picked mon.noname-b con 0x7f07dc0a1330 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:09:59.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.544+0000 7f07e35e0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f07dc06dac0 addr v1:172.21.15.42:6789/0 2024-09-16T04:09:59.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.544+0000 7f07e35e0640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f07dc0a1c50 0x7f07dc135810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:09:59.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.544+0000 7f07e35e0640 10 monclient(hunting): picked mon.noname-c con 0x7f07dc0a1c50 addr v2:172.21.15.154:3300/0 2024-09-16T04:09:59.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.544+0000 7f07e35e0640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f07dc09c090 con 0x7f07dc06dac0 2024-09-16T04:09:59.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.544+0000 7f07e35e0640 10 monclient(hunting): start opening mon connection 2024-09-16T04:09:59.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.544+0000 7f07e35e0640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f07dc09c890 con 0x7f07dc0a1c50 2024-09-16T04:09:59.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.544+0000 7f07e35e0640 10 monclient(hunting): start opening mon connection 2024-09-16T04:09:59.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.544+0000 7f07e35e0640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f07dc09f610 con 0x7f07dc0a1330 2024-09-16T04:09:59.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.544+0000 7f07e35e0640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:59.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.544+0000 7f07e35e0640 10 monclient(hunting): authenticate will time out at 2646.512104s 2024-09-16T04:09:59.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.544+0000 7f07e137c640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f07dc0a1330 0x7f07dc0a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:09:59.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.544+0000 7f07e0b7b640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f07dc06dac0 0x7f07dc135100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:37312/0 (socket says 172.21.15.42:37312) 2024-09-16T04:09:59.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.544+0000 7f07e0b7b640 1 -- 172.21.15.42:0/4254984277 learned_addr learned my addr 172.21.15.42:0/4254984277 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T04:09:59.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.544+0000 7f07e137c640 10 monclient(hunting): get_auth_request con 0x7f07dc0a1330 auth_method 0 2024-09-16T04:09:59.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.544+0000 7f07e137c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:09:59.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.544+0000 7f07e137c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:59.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.544+0000 7f07e137c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:59.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.544+0000 7f07e137c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:09:59.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.544+0000 7f07e137c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:09:59.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.544+0000 7f07e137c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:09:59.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.544+0000 7f07e1b7d640 1 --2- 172.21.15.42:0/4254984277 >> v2:172.21.15.154:3300/0 conn(0x7f07dc0a1c50 0x7f07dc135810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:09:59.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.544+0000 7f07e1b7d640 10 monclient(hunting): get_auth_request con 0x7f07dc0a1c50 auth_method 0 2024-09-16T04:09:59.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.544+0000 7f07e1b7d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:09:59.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.544+0000 7f07e1b7d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:59.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.544+0000 7f07e1b7d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:59.552 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.544+0000 7f07e137c640 10 monclient(hunting): handle_auth_done global_id 1014338 payload 274 2024-09-16T04:09:59.552 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.544+0000 7f07d3fff640 1 -- 172.21.15.42:0/4254984277 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2765487941 0 0) 0x7f07dc09c090 con 0x7f07dc06dac0 2024-09-16T04:09:59.552 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.544+0000 7f07e137c640 1 -- 172.21.15.42:0/4254984277 >> v2:172.21.15.154:3300/0 conn(0x7f07dc0a1c50 msgr2=0x7f07dc135810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:09:59.552 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.544+0000 7f07e137c640 1 --2- 172.21.15.42:0/4254984277 >> v2:172.21.15.154:3300/0 conn(0x7f07dc0a1c50 0x7f07dc135810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:09:59.552 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.544+0000 7f07e137c640 1 -- 172.21.15.42:0/4254984277 >> v1:172.21.15.42:6789/0 conn(0x7f07dc06dac0 legacy=0x7f07dc135100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:59.552 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.544+0000 7f07e137c640 10 monclient: _finish_hunting 0 2024-09-16T04:09:59.552 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.544+0000 7f07e137c640 1 monclient: found mon.noname-b 2024-09-16T04:09:59.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.544+0000 7f07e137c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:59.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.544+0000 7f07e137c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T04:09:59.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.544+0000 7f07e137c640 1 -- 172.21.15.42:0/4254984277 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07dc09f8f0 con 0x7f07dc0a1330 2024-09-16T04:09:59.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.544+0000 7f07d3fff640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T04:09:59.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.544+0000 7f07e137c640 1 --2- 172.21.15.42:0/4254984277 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f07dc0a1330 0x7f07dc0a1710 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f07cc0097c0 tx=0x7f07cc01c3c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3da682ce0711fc15 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:09:59.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.544+0000 7f07d3fff640 1 -- 172.21.15.42:0/4254984277 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07cc0b1070 con 0x7f07dc0a1330 2024-09-16T04:09:59.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.544+0000 7f07d3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:59.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.544+0000 7f07d3fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:09:59.560 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.544+0000 7f07d3fff640 10 monclient: dump: 2024-09-16T04:09:59.560 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:09:59.560 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:09:59.560 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:09:59.560 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:09:59.560 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:59.560 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:09:59.560 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:09:59.560 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:09:59.560 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:09:59.560 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:09:59.560 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.544+0000 7f07d3fff640 10 monclient: _finish_auth 0 2024-09-16T04:09:59.560 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.544+0000 7f07d3fff640 10 monclient: _check_auth_tickets 2024-09-16T04:09:59.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.544+0000 7f07d3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:59.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.544+0000 7f07d3fff640 1 -- 172.21.15.42:0/4254984277 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f07cc0a8050 con 0x7f07dc0a1330 2024-09-16T04:09:59.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.544+0000 7f07e35e0640 5 monclient: authenticate success, global_id 1014338 2024-09-16T04:09:59.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.544+0000 7f07e35e0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:09:59.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.544+0000 7f07d3fff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:59.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.544+0000 7f07d3fff640 1 -- 172.21.15.42:0/4254984277 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07cc0ac860 con 0x7f07dc0a1330 2024-09-16T04:09:59.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.544+0000 7f07d3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:59.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.544+0000 7f07d3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:09:59.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.544+0000 7f07d3fff640 10 monclient: dump: 2024-09-16T04:09:59.561 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:09:59.561 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:09:59.561 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:09:59.561 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:09:59.561 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:59.562 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:09:59.562 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:09:59.562 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:09:59.562 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:09:59.562 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:09:59.562 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.544+0000 7f07e35e0640 10 monclient: get_monmap_and_config success 2024-09-16T04:09:59.562 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.544+0000 7f07e35e0640 4 set_mon_vals no callback set 2024-09-16T04:09:59.562 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.544+0000 7f07e35e0640 10 monclient: shutdown 2024-09-16T04:09:59.562 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.544+0000 7f07e35e0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:59.562 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.544+0000 7f07e35e0640 1 -- 172.21.15.42:0/4254984277 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f07dc0a1330 msgr2=0x7f07dc0a1710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:59.562 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.544+0000 7f07e35e0640 1 --2- 172.21.15.42:0/4254984277 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f07dc0a1330 0x7f07dc0a1710 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f07cc0097c0 tx=0x7f07cc01c3c0 comp rx=0 tx=0).stop 2024-09-16T04:09:59.562 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.544+0000 7f07e35e0640 1 -- 172.21.15.42:0/4254984277 shutdown_connections 2024-09-16T04:09:59.562 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.544+0000 7f07e35e0640 1 --2- 172.21.15.42:0/4254984277 >> v2:172.21.15.154:3300/0 conn(0x7f07dc0a1c50 0x7f07dc135810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:09:59.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.544+0000 7f07e35e0640 1 --2- 172.21.15.42:0/4254984277 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f07dc0a1330 0x7f07dc0a1710 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:09:59.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.544+0000 7f07e35e0640 1 -- 172.21.15.42:0/4254984277 >> 172.21.15.42:0/4254984277 conn(0x7f07dc0a2420 msgr2=0x7f07dc0a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:59.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.544+0000 7f07e35e0640 1 -- 172.21.15.42:0/4254984277 shutdown_connections 2024-09-16T04:09:59.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.544+0000 7f07e35e0640 1 -- 172.21.15.42:0/4254984277 wait complete. 2024-09-16T04:09:59.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.548+0000 7f07e35e0640 10 monclient: build_initial_monmap 2024-09-16T04:09:59.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.548+0000 7f07e35e0640 10 monclient: monmap: 2024-09-16T04:09:59.563 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:09:59.563 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:09:59.563 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:09:59.563 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:09:59.563 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:59.564 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:09:59.564 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:09:59.564 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:09:59.564 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:09:59.564 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:09:59.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.548+0000 7f07e35e0640 1 librados: starting msgr at 2024-09-16T04:09:59.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.548+0000 7f07e35e0640 1 librados: starting objecter 2024-09-16T04:09:59.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.548+0000 7f07e35e0640 1 Processor -- start 2024-09-16T04:09:59.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.548+0000 7f07e35e0640 1 -- start start 2024-09-16T04:09:59.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.548+0000 7f07e35e0640 1 librados: setting wanted keys 2024-09-16T04:09:59.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.548+0000 7f07e35e0640 1 librados: calling monclient init 2024-09-16T04:09:59.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.548+0000 7f07e35e0640 10 monclient: init 2024-09-16T04:09:59.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.548+0000 7f07e35e0640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:59.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.548+0000 7f07e35e0640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:09:59.565 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.548+0000 7f07e35e0640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f07dc0a1330 0x7f07dc1348a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:09:59.565 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.548+0000 7f07e35e0640 10 monclient(hunting): picked mon.noname-a con 0x7f07dc0a1330 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:09:59.565 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.548+0000 7f07e35e0640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f07dc0a1c50 0x7f07dc134de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:09:59.565 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.548+0000 7f07e35e0640 10 monclient(hunting): picked mon.noname-c con 0x7f07dc0a1c50 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:09:59.565 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.548+0000 7f07e35e0640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f07dc11bf70 0x7f07dc11c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:09:59.565 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.548+0000 7f07e35e0640 10 monclient(hunting): picked mon.noname-b con 0x7f07dc11bf70 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:09:59.565 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.548+0000 7f07e35e0640 10 monclient(hunting): start opening mon connection 2024-09-16T04:09:59.565 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.548+0000 7f07e35e0640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f07dc09ef30 con 0x7f07dc0a1330 2024-09-16T04:09:59.565 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.548+0000 7f07e35e0640 10 monclient(hunting): start opening mon connection 2024-09-16T04:09:59.565 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.548+0000 7f07e35e0640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f07dc09edb0 con 0x7f07dc11bf70 2024-09-16T04:09:59.565 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.548+0000 7f07e35e0640 10 monclient(hunting): start opening mon connection 2024-09-16T04:09:59.565 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.548+0000 7f07e35e0640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f07dc135d50 con 0x7f07dc0a1c50 2024-09-16T04:09:59.565 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.548+0000 7f07e35e0640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:59.565 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.548+0000 7f07e35e0640 10 monclient(hunting): authenticate will time out at 2646.517258s 2024-09-16T04:09:59.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.548+0000 7f07e0b7b640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f07dc0a1c50 0x7f07dc134de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:09:59.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.548+0000 7f07e137c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f07dc0a1330 0x7f07dc1348a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:09:59.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.548+0000 7f07e137c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f07dc0a1330 0x7f07dc1348a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:59972/0 (socket says 172.21.15.42:59972) 2024-09-16T04:09:59.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.548+0000 7f07e137c640 1 -- 172.21.15.42:0/3296310576 learned_addr learned my addr 172.21.15.42:0/3296310576 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:09:59.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.548+0000 7f07e137c640 10 monclient(hunting): get_auth_request con 0x7f07dc0a1330 auth_method 0 2024-09-16T04:09:59.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.548+0000 7f07e137c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:09:59.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.548+0000 7f07e137c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:59.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.548+0000 7f07e137c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:59.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.548+0000 7f07e0b7b640 10 monclient(hunting): get_auth_request con 0x7f07dc0a1c50 auth_method 0 2024-09-16T04:09:59.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.548+0000 7f07e0b7b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:09:59.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.548+0000 7f07e0b7b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:59.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.548+0000 7f07e0b7b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:59.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.548+0000 7f07e137c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:09:59.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.548+0000 7f07e137c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:09:59.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.548+0000 7f07e137c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:09:59.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.548+0000 7f07e0b7b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:09:59.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.548+0000 7f07e0b7b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:09:59.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.548+0000 7f07e0b7b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:09:59.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.548+0000 7f07e1b7d640 1 --2- 172.21.15.42:0/3296310576 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f07dc11bf70 0x7f07dc11c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:09:59.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.548+0000 7f07e1b7d640 10 monclient(hunting): get_auth_request con 0x7f07dc11bf70 auth_method 0 2024-09-16T04:09:59.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.548+0000 7f07e1b7d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:09:59.568 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.548+0000 7f07e1b7d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:59.568 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.548+0000 7f07e1b7d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:59.568 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.548+0000 7f07e137c640 10 monclient(hunting): handle_auth_done global_id 1034199 payload 824 2024-09-16T04:09:59.568 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.548+0000 7f07e137c640 1 -- 172.21.15.42:0/3296310576 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f07dc0a1c50 msgr2=0x7f07dc134de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:59.568 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.548+0000 7f07e137c640 1 --2- 172.21.15.42:0/3296310576 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f07dc0a1c50 0x7f07dc134de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:09:59.568 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.548+0000 7f07e137c640 1 -- 172.21.15.42:0/3296310576 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f07dc11bf70 msgr2=0x7f07dc11c3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:59.568 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.548+0000 7f07e137c640 1 --2- 172.21.15.42:0/3296310576 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f07dc11bf70 0x7f07dc11c3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:09:59.568 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.548+0000 7f07e137c640 10 monclient: _finish_hunting 0 2024-09-16T04:09:59.568 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.548+0000 7f07e137c640 1 monclient: found mon.noname-a 2024-09-16T04:09:59.568 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.548+0000 7f07e137c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:59.568 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.548+0000 7f07e137c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:09:59.568 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.548+0000 7f07e137c640 1 -- 172.21.15.42:0/3296310576 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07dc11f600 con 0x7f07dc0a1330 2024-09-16T04:09:59.568 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.548+0000 7f07e0b7b640 1 --2- 172.21.15.42:0/3296310576 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f07dc0a1c50 0x7f07dc134de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:09:59.569 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.548+0000 7f07e137c640 1 --2- 172.21.15.42:0/3296310576 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f07dc0a1330 0x7f07dc1348a0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f07cc019ff0 tx=0x7f07cc019160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:09:59.569 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.548+0000 7f07d1ffb640 10 client.?.objecter ms_handle_connect 0x7f07dc0a1330 2024-09-16T04:09:59.569 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.548+0000 7f07d1ffb640 10 client.?.objecter resend_mon_ops 2024-09-16T04:09:59.569 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.548+0000 7f07d1ffb640 1 -- 172.21.15.42:0/3296310576 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07cc0b9070 con 0x7f07dc0a1330 2024-09-16T04:09:59.569 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.548+0000 7f07d1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:59.569 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.548+0000 7f07d1ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:09:59.569 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.548+0000 7f07d1ffb640 10 monclient: dump: 2024-09-16T04:09:59.569 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:09:59.569 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:09:59.570 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:09:59.570 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:09:59.570 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:59.570 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:09:59.570 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:09:59.570 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:09:59.570 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:09:59.570 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:09:59.570 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.548+0000 7f07d1ffb640 10 monclient: _finish_auth 0 2024-09-16T04:09:59.570 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.548+0000 7f07d1ffb640 10 monclient: _check_auth_tickets 2024-09-16T04:09:59.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.548+0000 7f07d1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:59.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.548+0000 7f07d1ffb640 1 -- 172.21.15.42:0/3296310576 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f07cc0b5410 con 0x7f07dc0a1330 2024-09-16T04:09:59.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.548+0000 7f07d1ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:59.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.548+0000 7f07d1ffb640 1 -- 172.21.15.42:0/3296310576 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07cc0b1040 con 0x7f07dc0a1330 2024-09-16T04:09:59.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.548+0000 7f07d1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:59.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.548+0000 7f07d1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:09:59.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.548+0000 7f07d1ffb640 10 monclient: dump: 2024-09-16T04:09:59.571 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:09:59.571 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:09:59.571 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:09:59.571 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:09:59.571 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:59.572 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:09:59.572 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:09:59.572 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:09:59.572 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:09:59.572 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:09:59.572 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.548+0000 7f07d37fe640 4 set_mon_vals no callback set 2024-09-16T04:09:59.572 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.548+0000 7f07e35e0640 5 monclient: authenticate success, global_id 1034199 2024-09-16T04:09:59.572 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.548+0000 7f07e35e0640 10 monclient: _renew_subs 2024-09-16T04:09:59.572 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.548+0000 7f07e35e0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:09:59.572 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.548+0000 7f07e35e0640 1 -- 172.21.15.42:0/3296310576 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07dc06e4b0 con 0x7f07dc0a1330 2024-09-16T04:09:59.572 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.548+0000 7f07e35e0640 10 client.1034199.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:09:59.572 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.548+0000 7f07e35e0640 10 monclient: _renew_subs 2024-09-16T04:09:59.572 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.548+0000 7f07e35e0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:09:59.573 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.548+0000 7f07e35e0640 1 -- 172.21.15.42:0/3296310576 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07dc11f040 con 0x7f07dc0a1330 2024-09-16T04:09:59.573 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.548+0000 7f07e35e0640 1 librados: init done 2024-09-16T04:09:59.573 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.548+0000 7f07d1ffb640 1 -- 172.21.15.42:0/3296310576 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f07cc0bc070 con 0x7f07dc0a1330 2024-09-16T04:09:59.573 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.548+0000 7f07e35e0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:59.573 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.548+0000 7f07e35e0640 1 --2- 172.21.15.42:0/3296310576 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f07dc06dac0 0x7f07dc02d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:09:59.573 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.548+0000 7f07e35e0640 10 monclient(hunting): start opening mon connection 2024-09-16T04:09:59.573 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.548+0000 7f07e35e0640 1 -- 172.21.15.42:0/3296310576 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f07dc11e090 con 0x7f07dc06dac0 2024-09-16T04:09:59.573 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.548+0000 7f07e0b7b640 1 --2- 172.21.15.42:0/3296310576 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f07dc06dac0 0x7f07dc02d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:09:59.573 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.548+0000 7f07e0b7b640 10 monclient: get_auth_request con 0x7f07dc06dac0 auth_method 0 2024-09-16T04:09:59.573 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.548+0000 7f07e0b7b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:09:59.573 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.548+0000 7f07e0b7b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:59.573 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.548+0000 7f07e0b7b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:59.573 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.548+0000 7f07d1ffb640 1 --2- 172.21.15.42:0/3296310576 >> v2:172.21.15.154:6812/254845342 conn(0x7f07c0037810 0x7f07c0039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:09:59.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.548+0000 7f07d1ffb640 1 -- 172.21.15.42:0/3296310576 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(706..706 src has 451..706) ==== 5040+0+0 (secure 0 0 0) 0x7f07cc0e2c10 con 0x7f07dc0a1330 2024-09-16T04:09:59.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.548+0000 7f07d1ffb640 10 client.1034199.objecter ms_dispatch 0x7f07dc000ba0 osd_map(706..706 src has 451..706) 2024-09-16T04:09:59.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.548+0000 7f07e0b7b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:09:59.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.548+0000 7f07d1ffb640 3 client.1034199.objecter handle_osd_map got epochs [706,706] > 0 2024-09-16T04:09:59.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.548+0000 7f07d1ffb640 3 client.1034199.objecter handle_osd_map decoding full epoch 706 2024-09-16T04:09:59.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.548+0000 7f07e0b7b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:09:59.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.548+0000 7f07e0b7b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:09:59.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.548+0000 7f07d1ffb640 20 client.1034199.objecter dump_active .. 0 homeless 2024-09-16T04:09:59.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.548+0000 7f07e0b7b640 10 monclient(hunting): handle_auth_done global_id 1034202 payload 824 2024-09-16T04:09:59.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.548+0000 7f07e0b7b640 1 -- 172.21.15.42:0/3296310576 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f07dc11ee40 con 0x7f07dc06dac0 2024-09-16T04:09:59.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.552+0000 7f07e0b7b640 1 --2- 172.21.15.42:0/3296310576 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f07dc06dac0 0x7f07dc02d7d0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f07dc135d20 tx=0x7f07d8001cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:09:59.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.552+0000 7f07d1ffb640 10 client.1034199.objecter ms_handle_connect 0x7f07dc06dac0 2024-09-16T04:09:59.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.552+0000 7f07d1ffb640 10 client.1034199.objecter resend_mon_ops 2024-09-16T04:09:59.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.552+0000 7f07e1b7d640 1 --2- 172.21.15.42:0/3296310576 >> v2:172.21.15.154:6812/254845342 conn(0x7f07c0037810 0x7f07c0039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:09:59.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.552+0000 7f07e1b7d640 10 monclient: get_auth_request con 0x7f07c0037810 auth_method 0 2024-09-16T04:09:59.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.552+0000 7f07d1ffb640 1 -- 172.21.15.42:0/3296310576 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07d800fac0 con 0x7f07dc06dac0 2024-09-16T04:09:59.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.552+0000 7f07d1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:59.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.552+0000 7f07d1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:09:59.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.552+0000 7f07d1ffb640 10 monclient: dump: 2024-09-16T04:09:59.575 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:09:59.575 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:09:59.575 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:09:59.575 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:09:59.575 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:59.575 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:09:59.575 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:09:59.576 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:09:59.576 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:09:59.576 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:09:59.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.552+0000 7f07d1ffb640 1 -- 172.21.15.42:0/3296310576 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f07d8018070 con 0x7f07dc06dac0 2024-09-16T04:09:59.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.552+0000 7f07d1ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:59.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.552+0000 7f07d1ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:09:59.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.552+0000 7f07d1ffb640 1 -- 172.21.15.42:0/3296310576 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f07dc06dac0 msgr2=0x7f07dc02d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:59.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.552+0000 7f07d1ffb640 1 --2- 172.21.15.42:0/3296310576 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f07dc06dac0 0x7f07dc02d7d0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f07dc135d20 tx=0x7f07d8001cb0 comp rx=0 tx=0).stop 2024-09-16T04:09:59.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.552+0000 7f07d1ffb640 1 -- 172.21.15.42:0/3296310576 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f07dc06dac0 msgr2=0x7f07dc02d7d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:09:59.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.552+0000 7f07d1ffb640 1 --2- 172.21.15.42:0/3296310576 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f07dc06dac0 0x7f07dc02d7d0 secure :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f07dc135d20 tx=0x7f07d8001cb0 comp rx=0 tx=0).stop 2024-09-16T04:09:59.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.552+0000 7f07d1ffb640 10 client.1034199.objecter ms_handle_connect 0x7f07c0037810 2024-09-16T04:09:59.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.552+0000 7f07e1b7d640 1 --2- 172.21.15.42:0/3296310576 >> v2:172.21.15.154:6812/254845342 conn(0x7f07c0037810 0x7f07c0039cd0 secure :-1 s=READY pgs=3917 cs=0 l=1 rev1=1 crypto rx=0x7f07d4009f80 tx=0x7f07d400b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:09:59.578 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.556+0000 7f07e35e0640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:09:59.578 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.556+0000 7f07e35e0640 1 --2- 172.21.15.42:0/3296310576 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f07dc11c910 0x7f07dc062ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:09:59.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.556+0000 7f07e35e0640 10 monclient(hunting): start opening mon connection 2024-09-16T04:09:59.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.556+0000 7f07e35e0640 1 -- 172.21.15.42:0/3296310576 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f07dc139a40 con 0x7f07dc11c910 2024-09-16T04:09:59.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.556+0000 7f07e0b7b640 1 --2- 172.21.15.42:0/3296310576 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f07dc11c910 0x7f07dc062ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:09:59.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.556+0000 7f07e0b7b640 10 monclient: get_auth_request con 0x7f07dc11c910 auth_method 0 2024-09-16T04:09:59.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.556+0000 7f07e0b7b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:09:59.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.556+0000 7f07e0b7b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:59.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.556+0000 7f07e0b7b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:59.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.556+0000 7f07e0b7b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:09:59.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.556+0000 7f07e0b7b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:09:59.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.556+0000 7f07e0b7b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:09:59.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.556+0000 7f07e0b7b640 10 monclient(hunting): handle_auth_done global_id 1034205 payload 824 2024-09-16T04:09:59.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.556+0000 7f07e0b7b640 1 -- 172.21.15.42:0/3296310576 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f07dc11dbf0 con 0x7f07dc11c910 2024-09-16T04:09:59.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.556+0000 7f07d1ffb640 10 client.1034199.objecter ms_handle_connect 0x7f07dc11c910 2024-09-16T04:09:59.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.556+0000 7f07d1ffb640 10 client.1034199.objecter resend_mon_ops 2024-09-16T04:09:59.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.556+0000 7f07e0b7b640 1 --2- 172.21.15.42:0/3296310576 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f07dc11c910 0x7f07dc062ea0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f07d8014000 tx=0x7f07d8001090 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:09:59.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.556+0000 7f07d1ffb640 1 -- 172.21.15.42:0/3296310576 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07d80136c0 con 0x7f07dc11c910 2024-09-16T04:09:59.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.556+0000 7f07d1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:59.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.556+0000 7f07d1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:09:59.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.556+0000 7f07d1ffb640 10 monclient: dump: 2024-09-16T04:09:59.580 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:09:59.580 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:09:59.580 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:09:59.580 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:09:59.580 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:59.580 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:09:59.581 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:09:59.581 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:09:59.581 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:09:59.581 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:09:59.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.556+0000 7f07d1ffb640 1 -- 172.21.15.42:0/3296310576 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4881 (secure 0 0 0) 0x7f07d80136c0 con 0x7f07dc11c910 2024-09-16T04:09:59.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.556+0000 7f07d1ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:09:59.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.556+0000 7f07d1ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:09:59.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.556+0000 7f07d1ffb640 1 -- 172.21.15.42:0/3296310576 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f07dc11c910 msgr2=0x7f07dc062ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:59.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.556+0000 7f07d1ffb640 1 --2- 172.21.15.42:0/3296310576 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f07dc11c910 0x7f07dc062ea0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f07d8014000 tx=0x7f07d8001090 comp rx=0 tx=0).stop 2024-09-16T04:09:59.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.556+0000 7f07d1ffb640 1 -- 172.21.15.42:0/3296310576 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f07dc11c910 msgr2=0x7f07dc062ea0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:09:59.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.556+0000 7f07d1ffb640 1 --2- 172.21.15.42:0/3296310576 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f07dc11c910 0x7f07dc062ea0 secure :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f07d8014000 tx=0x7f07d8001090 comp rx=0 tx=0).stop 2024-09-16T04:09:59.582 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T04:09:59.582 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:09:59.582 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:09:59.582 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "leader", 2024-09-16T04:09:59.582 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 256, 2024-09-16T04:09:59.582 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 79491, 2024-09-16T04:09:59.582 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T04:09:59.582 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T04:09:59.582 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T04:09:59.582 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T04:09:59.582 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:09:59.582 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 0, 2024-09-16T04:09:59.583 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:09:59.583 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T04:09:59.583 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T04:09:59.583 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:09:59.583 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:09:59.583 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:09:59.583 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:09:59.583 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:09:59.583 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:09:59.583 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:09:59.583 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:09:59.583 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:09:59.583 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:09:59.584 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:09:59.584 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:09:59.584 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:09:59.584 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T04:09:59.584 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:09:59.584 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:09:59.584 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:09:59.584 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:09:59.584 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:09:59.584 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:09:59.584 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:09:59.584 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:09:59.585 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:09:59.585 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:09:59.585 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:09:59.585 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:09:59.585 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:09:59.585 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T04:09:59.585 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T04:09:59.585 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T04:09:59.585 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T04:09:59.585 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T04:09:59.585 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T04:09:59.585 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:09:59.585 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:09:59.586 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T04:09:59.586 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T04:09:59.586 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T04:09:59.586 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T04:09:59.586 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T04:09:59.586 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T04:09:59.586 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T04:09:59.586 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:09:59.586 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T04:09:59.586 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:09:59.586 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:09:59.586 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:09:59.586 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:09:59.586 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:09:59.587 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:09:59.587 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:09:59.587 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:09:59.587 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:09:59.587 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:09:59.587 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:09:59.587 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:09:59.587 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T04:09:59.587 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:09:59.587 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T04:09:59.587 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:09:59.587 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:09:59.587 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:09:59.587 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:09:59.588 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:09:59.588 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:09:59.588 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:09:59.588 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T04:09:59.588 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:09:59.588 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:09:59.588 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:09:59.588 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:09:59.588 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T04:09:59.588 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:09:59.588 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:09:59.588 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:09:59.588 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:09:59.589 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T04:09:59.589 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T04:09:59.589 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:09:59.589 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:09:59.589 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:09:59.589 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:09:59.589 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:09:59.589 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:09:59.589 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:09:59.589 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:09:59.589 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:09:59.589 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:09:59.590 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:09:59.590 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T04:09:59.590 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:09:59.590 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:09:59.590 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:09:59.590 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:09:59.590 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T04:09:59.590 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:09:59.590 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:09:59.590 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:09:59.590 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:09:59.590 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T04:09:59.590 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T04:09:59.590 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:09:59.591 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:09:59.591 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:09:59.591 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:09:59.591 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:09:59.591 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:09:59.591 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:09:59.591 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:09:59.591 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:09:59.591 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:09:59.591 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:09:59.591 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T04:09:59.591 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:09:59.592 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:09:59.592 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:09:59.592 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:09:59.592 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T04:09:59.592 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:09:59.592 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:09:59.592 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:09:59.592 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:09:59.592 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T04:09:59.592 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T04:09:59.593 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:09:59.593 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:09:59.593 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:09:59.593 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:09:59.593 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:09:59.593 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:09:59.593 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T04:09:59.593 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T04:09:59.593 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:09:59.593 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:09:59.593 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:09:59.593 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:09:59.594 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:09:59.594 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:09:59.594 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T04:09:59.594 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:09:59.594 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:09:59.594 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:09:59.594 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 2 2024-09-16T04:09:59.594 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:09:59.594 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:09:59.595 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T04:09:59.595 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:09:59.595 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:09:59.595 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:09:59.595 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 5 2024-09-16T04:09:59.595 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:09:59.595 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:09:59.595 INFO:teuthology.orchestra.run.smithi042.stdout: "mgr": [ 2024-09-16T04:09:59.595 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:09:59.595 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:09:59.595 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:09:59.595 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:09:59.595 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:09:59.595 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:09:59.596 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:09:59.596 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T04:09:59.596 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T04:09:59.597 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.560+0000 7f07e35e0640 10 librados: watch_flush enter 2024-09-16T04:09:59.597 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.560+0000 7f07e35e0640 10 librados: watch_flush exit 2024-09-16T04:09:59.598 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.560+0000 7f07e35e0640 20 client.1034199.objecter shutdown clearing up homeless session... 2024-09-16T04:09:59.598 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.560+0000 7f07e35e0640 10 client.1034199.objecter successfully canceled tick 2024-09-16T04:09:59.598 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.560+0000 7f07e35e0640 1 -- 172.21.15.42:0/3296310576 >> v2:172.21.15.154:6812/254845342 conn(0x7f07c0037810 msgr2=0x7f07c0039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:59.598 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.560+0000 7f07e35e0640 1 --2- 172.21.15.42:0/3296310576 >> v2:172.21.15.154:6812/254845342 conn(0x7f07c0037810 0x7f07c0039cd0 secure :-1 s=READY pgs=3917 cs=0 l=1 rev1=1 crypto rx=0x7f07d4009f80 tx=0x7f07d400b040 comp rx=0 tx=0).stop 2024-09-16T04:09:59.598 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.560+0000 7f07e35e0640 10 monclient: shutdown 2024-09-16T04:09:59.598 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.560+0000 7f07e35e0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:59.598 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.560+0000 7f07e35e0640 1 -- 172.21.15.42:0/3296310576 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f07dc0a1330 msgr2=0x7f07dc1348a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:59.598 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.560+0000 7f07e35e0640 1 --2- 172.21.15.42:0/3296310576 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f07dc0a1330 0x7f07dc1348a0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f07cc019ff0 tx=0x7f07cc019160 comp rx=0 tx=0).stop 2024-09-16T04:09:59.598 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.560+0000 7f07e1b7d640 1 -- 172.21.15.42:0/3296310576 reap_dead start 2024-09-16T04:09:59.599 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.560+0000 7f07e1b7d640 1 -- 172.21.15.42:0/3296310576 reap_dead start 2024-09-16T04:09:59.599 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.560+0000 7f07e35e0640 1 -- 172.21.15.42:0/3296310576 shutdown_connections 2024-09-16T04:09:59.599 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.560+0000 7f07e35e0640 1 -- 172.21.15.42:0/3296310576 >> 172.21.15.42:0/3296310576 conn(0x7f07dc0a2420 msgr2=0x7f07dc0708b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:59.599 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.560+0000 7f07e35e0640 1 -- 172.21.15.42:0/3296310576 shutdown_connections 2024-09-16T04:09:59.599 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.560+0000 7f07e35e0640 1 -- 172.21.15.42:0/3296310576 wait complete. 2024-09-16T04:09:59.599 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.560+0000 7f07e35e0640 1 librados: shutdown 2024-09-16T04:09:59.599 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:09:59.599 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T04:09:59.600 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T04:09:59.717 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.716+0000 7fd1b0e2e640 10 monclient: get_monmap_and_config 2024-09-16T04:09:59.717 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.716+0000 7fd1b0e2e640 10 monclient: build_initial_monmap 2024-09-16T04:09:59.717 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.716+0000 7fd1b0e2e640 10 monclient: monmap: 2024-09-16T04:09:59.717 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:09:59.718 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:09:59.718 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:09:59.716250+0000 2024-09-16T04:09:59.718 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:09:59.716250+0000 2024-09-16T04:09:59.718 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:59.718 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:09:59.718 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:09:59.718 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:09:59.718 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:09:59.718 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:09:59.718 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:09:59.718 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:09:59.720 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.716+0000 7fd1b0e2e640 1 Processor -- start 2024-09-16T04:09:59.720 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.716+0000 7fd1b0e2e640 1 -- start start 2024-09-16T04:09:59.721 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.716+0000 7fd1b0e2e640 10 monclient: init 2024-09-16T04:09:59.721 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.716+0000 7fd1b0e2e640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:59.721 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.716+0000 7fd1b0e2e640 10 monclient: _add_conns ranks=[4,1,3,2,0] 2024-09-16T04:09:59.721 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.716+0000 7fd1b0e2e640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fd1ac073080 0x7fd1ac073460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:09:59.721 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.716+0000 7fd1b0e2e640 10 monclient(hunting): picked mon.noname-c con 0x7fd1ac073080 addr v2:172.21.15.154:3300/0 2024-09-16T04:09:59.721 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.716+0000 7fd1b0e2e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd1ac0739a0 addr v1:172.21.15.154:6789/0 2024-09-16T04:09:59.721 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.716+0000 7fd1b0e2e640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd1ac09f670 0x7fd1ac0a1070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:09:59.721 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.716+0000 7fd1b0e2e640 10 monclient(hunting): picked mon.noname-a con 0x7fd1ac09f670 addr v2:172.21.15.42:3300/0 2024-09-16T04:09:59.721 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.716+0000 7fd1b0e2e640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd1ac0a1cd0 con 0x7fd1ac0739a0 2024-09-16T04:09:59.721 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.716+0000 7fd1b0e2e640 10 monclient(hunting): start opening mon connection 2024-09-16T04:09:59.721 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.716+0000 7fd1b0e2e640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7fd1ac1326d0 con 0x7fd1ac09f670 2024-09-16T04:09:59.721 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.716+0000 7fd1b0e2e640 10 monclient(hunting): start opening mon connection 2024-09-16T04:09:59.721 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.716+0000 7fd1b0e2e640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fd1ac0a1a10 con 0x7fd1ac073080 2024-09-16T04:09:59.721 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.716+0000 7fd1b0e2e640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:59.722 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.716+0000 7fd1b0e2e640 10 monclient(hunting): authenticate will time out at 2646.685839s 2024-09-16T04:09:59.722 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.716+0000 7fd1aad9d640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd1ac09f670 0x7fd1ac0a1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:09:59.722 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.716+0000 7fd1aad9d640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd1ac09f670 0x7fd1ac0a1070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:60012/0 (socket says 172.21.15.42:60012) 2024-09-16T04:09:59.722 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.716+0000 7fd1aad9d640 1 -- 172.21.15.42:0/3440199118 learned_addr learned my addr 172.21.15.42:0/3440199118 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:09:59.722 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.716+0000 7fd1aad9d640 10 monclient(hunting): get_auth_request con 0x7fd1ac09f670 auth_method 0 2024-09-16T04:09:59.722 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.716+0000 7fd1aad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:09:59.722 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.716+0000 7fd1aad9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:59.722 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.716+0000 7fd1aad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:59.722 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.716+0000 7fd1aad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:09:59.722 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.716+0000 7fd1aad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:09:59.722 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.716+0000 7fd1aad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:09:59.722 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.716+0000 7fd1aa59c640 1 --2- 172.21.15.42:0/3440199118 >> v2:172.21.15.154:3300/0 conn(0x7fd1ac073080 0x7fd1ac073460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:09:59.722 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.716+0000 7fd1aa59c640 10 monclient(hunting): get_auth_request con 0x7fd1ac073080 auth_method 0 2024-09-16T04:09:59.722 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.716+0000 7fd1aa59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:09:59.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.716+0000 7fd1aa59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:59.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.716+0000 7fd1aa59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:59.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.716+0000 7fd1aad9d640 10 monclient(hunting): handle_auth_done global_id 1034208 payload 274 2024-09-16T04:09:59.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.716+0000 7fd1aad9d640 1 -- 172.21.15.42:0/3440199118 >> v2:172.21.15.154:3300/0 conn(0x7fd1ac073080 msgr2=0x7fd1ac073460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:09:59.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.716+0000 7fd1aad9d640 1 --2- 172.21.15.42:0/3440199118 >> v2:172.21.15.154:3300/0 conn(0x7fd1ac073080 0x7fd1ac073460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:09:59.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.716+0000 7fd1aad9d640 1 -- 172.21.15.42:0/3440199118 >> v1:172.21.15.154:6789/0 conn(0x7fd1ac0739a0 legacy=0x7fd1ac0a0960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:09:59.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.716+0000 7fd1aad9d640 10 monclient: _finish_hunting 0 2024-09-16T04:09:59.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.716+0000 7fd1aad9d640 1 monclient: found mon.noname-a 2024-09-16T04:09:59.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.716+0000 7fd1aad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:59.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.716+0000 7fd1aad9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:09:59.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.716+0000 7fd1aad9d640 1 -- 172.21.15.42:0/3440199118 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1ac09ccc0 con 0x7fd1ac09f670 2024-09-16T04:09:59.729 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.720+0000 7fd1aad9d640 1 --2- 172.21.15.42:0/3440199118 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd1ac09f670 0x7fd1ac0a1070 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fd1ac0a6fb0 tx=0x7fd1a401c430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9edf06700cefeae server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:09:59.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.720+0000 7fd1a959a640 1 -- 172.21.15.42:0/3440199118 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1a40b0070 con 0x7fd1ac09f670 2024-09-16T04:09:59.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.720+0000 7fd1a959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:59.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.720+0000 7fd1a959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:09:59.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.720+0000 7fd1a959a640 10 monclient: dump: 2024-09-16T04:09:59.730 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:09:59.730 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:09:59.730 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:09:59.730 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:09:59.730 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:59.730 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:09:59.730 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:09:59.730 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:09:59.730 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:09:59.731 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:09:59.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.720+0000 7fd1a959a640 10 monclient: _finish_auth 0 2024-09-16T04:09:59.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.720+0000 7fd1a959a640 10 monclient: _check_auth_tickets 2024-09-16T04:09:59.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.720+0000 7fd1a959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:59.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.720+0000 7fd1a959a640 1 -- 172.21.15.42:0/3440199118 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd1a40ab600 con 0x7fd1ac09f670 2024-09-16T04:09:59.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.720+0000 7fd1b0e2e640 5 monclient: authenticate success, global_id 1034208 2024-09-16T04:09:59.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.720+0000 7fd1b0e2e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:09:59.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.720+0000 7fd1a959a640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:59.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.720+0000 7fd1b0e2e640 10 monclient: get_monmap_and_config success 2024-09-16T04:09:59.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.720+0000 7fd1a959a640 1 -- 172.21.15.42:0/3440199118 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1a40ab9d0 con 0x7fd1ac09f670 2024-09-16T04:09:59.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.720+0000 7fd1b0e2e640 4 set_mon_vals no callback set 2024-09-16T04:09:59.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.720+0000 7fd1a959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:59.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.720+0000 7fd1a959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:09:59.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.720+0000 7fd1a959a640 10 monclient: dump: 2024-09-16T04:09:59.732 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:09:59.732 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:09:59.732 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:09:59.732 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:09:59.732 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:59.732 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:09:59.732 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:09:59.732 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:09:59.732 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:09:59.732 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:09:59.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.720+0000 7fd1b0e2e640 10 monclient: shutdown 2024-09-16T04:09:59.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.720+0000 7fd1b0e2e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:59.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.720+0000 7fd1b0e2e640 1 -- 172.21.15.42:0/3440199118 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd1ac09f670 msgr2=0x7fd1ac0a1070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:59.732 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.720+0000 7fd1b0e2e640 1 --2- 172.21.15.42:0/3440199118 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd1ac09f670 0x7fd1ac0a1070 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fd1ac0a6fb0 tx=0x7fd1a401c430 comp rx=0 tx=0).stop 2024-09-16T04:09:59.733 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.720+0000 7fd1b0e2e640 1 -- 172.21.15.42:0/3440199118 shutdown_connections 2024-09-16T04:09:59.733 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.720+0000 7fd1b0e2e640 1 --2- 172.21.15.42:0/3440199118 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd1ac09f670 0x7fd1ac0a1070 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:09:59.733 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.720+0000 7fd1b0e2e640 1 --2- 172.21.15.42:0/3440199118 >> v2:172.21.15.154:3300/0 conn(0x7fd1ac073080 0x7fd1ac073460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:09:59.733 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.720+0000 7fd1b0e2e640 1 -- 172.21.15.42:0/3440199118 >> 172.21.15.42:0/3440199118 conn(0x7fd1ac06bc40 msgr2=0x7fd1ac06c050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:59.733 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.720+0000 7fd1b0e2e640 1 -- 172.21.15.42:0/3440199118 shutdown_connections 2024-09-16T04:09:59.733 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.720+0000 7fd1b0e2e640 1 -- 172.21.15.42:0/3440199118 wait complete. 2024-09-16T04:09:59.733 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.720+0000 7fd1b0e2e640 10 monclient: build_initial_monmap 2024-09-16T04:09:59.733 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.720+0000 7fd1b0e2e640 10 monclient: monmap: 2024-09-16T04:09:59.733 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:09:59.733 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:09:59.733 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:09:59.733 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:09:59.733 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:59.733 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:09:59.734 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:09:59.734 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:09:59.734 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:09:59.734 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:09:59.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.720+0000 7fd1b0e2e640 1 librados: starting msgr at 2024-09-16T04:09:59.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.720+0000 7fd1b0e2e640 1 librados: starting objecter 2024-09-16T04:09:59.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.720+0000 7fd1b0e2e640 1 Processor -- start 2024-09-16T04:09:59.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.720+0000 7fd1b0e2e640 1 -- start start 2024-09-16T04:09:59.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.720+0000 7fd1b0e2e640 1 librados: setting wanted keys 2024-09-16T04:09:59.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.720+0000 7fd1b0e2e640 1 librados: calling monclient init 2024-09-16T04:09:59.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.720+0000 7fd1b0e2e640 10 monclient: init 2024-09-16T04:09:59.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.720+0000 7fd1b0e2e640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:59.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.720+0000 7fd1b0e2e640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:09:59.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.720+0000 7fd1b0e2e640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd1ac073080 0x7fd1ac1cef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:09:59.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.720+0000 7fd1b0e2e640 10 monclient(hunting): picked mon.noname-c con 0x7fd1ac073080 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:09:59.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.720+0000 7fd1b0e2e640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd1ac0739a0 0x7fd1ac1cf480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:09:59.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.720+0000 7fd1b0e2e640 10 monclient(hunting): picked mon.noname-a con 0x7fd1ac0739a0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:09:59.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.720+0000 7fd1b0e2e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd1ac09f670 0x7fd1ac1cf9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:09:59.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.720+0000 7fd1b0e2e640 10 monclient(hunting): picked mon.noname-b con 0x7fd1ac09f670 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:09:59.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.720+0000 7fd1b0e2e640 10 monclient(hunting): start opening mon connection 2024-09-16T04:09:59.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.720+0000 7fd1b0e2e640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fd1ac0a17c0 con 0x7fd1ac0739a0 2024-09-16T04:09:59.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.720+0000 7fd1b0e2e640 10 monclient(hunting): start opening mon connection 2024-09-16T04:09:59.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.720+0000 7fd1b0e2e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd1ac131e60 con 0x7fd1ac09f670 2024-09-16T04:09:59.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.720+0000 7fd1b0e2e640 10 monclient(hunting): start opening mon connection 2024-09-16T04:09:59.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.720+0000 7fd1b0e2e640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fd1ac13fa80 con 0x7fd1ac073080 2024-09-16T04:09:59.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.720+0000 7fd1b0e2e640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:59.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.720+0000 7fd1b0e2e640 10 monclient(hunting): authenticate will time out at 2646.690795s 2024-09-16T04:09:59.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.720+0000 7fd1a9d9b640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd1ac0739a0 0x7fd1ac1cf480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:09:59.735 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.720+0000 7fd1aa59c640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd1ac073080 0x7fd1ac1cef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:09:59.736 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.720+0000 7fd1a9d9b640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd1ac0739a0 0x7fd1ac1cf480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:60028/0 (socket says 172.21.15.42:60028) 2024-09-16T04:09:59.736 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.720+0000 7fd1a9d9b640 1 -- 172.21.15.42:0/3516245281 learned_addr learned my addr 172.21.15.42:0/3516245281 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:09:59.736 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.720+0000 7fd1a9d9b640 10 monclient(hunting): get_auth_request con 0x7fd1ac0739a0 auth_method 0 2024-09-16T04:09:59.736 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.720+0000 7fd1a9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:09:59.736 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.720+0000 7fd1a9d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:59.736 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.720+0000 7fd1a9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:59.736 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.720+0000 7fd1aa59c640 10 monclient(hunting): get_auth_request con 0x7fd1ac073080 auth_method 0 2024-09-16T04:09:59.736 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.720+0000 7fd1aa59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:09:59.736 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.720+0000 7fd1aa59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:59.736 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.720+0000 7fd1aa59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:59.736 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.720+0000 7fd1aa59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:09:59.736 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.720+0000 7fd1aa59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:09:59.736 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.720+0000 7fd1aa59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:09:59.736 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.720+0000 7fd1a9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:09:59.737 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.720+0000 7fd1a9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:09:59.737 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.720+0000 7fd1a9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:09:59.737 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.720+0000 7fd1aad9d640 1 --2- 172.21.15.42:0/3516245281 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd1ac09f670 0x7fd1ac1cf9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:09:59.737 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.720+0000 7fd1aad9d640 10 monclient(hunting): get_auth_request con 0x7fd1ac09f670 auth_method 0 2024-09-16T04:09:59.737 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.720+0000 7fd1aad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:09:59.737 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.720+0000 7fd1aad9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:59.737 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.720+0000 7fd1aad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:59.737 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.720+0000 7fd1aa59c640 10 monclient(hunting): handle_auth_done global_id 1014344 payload 824 2024-09-16T04:09:59.737 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.720+0000 7fd1aa59c640 1 -- 172.21.15.42:0/3516245281 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd1ac09f670 msgr2=0x7fd1ac1cf9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:59.737 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.720+0000 7fd1aa59c640 1 --2- 172.21.15.42:0/3516245281 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd1ac09f670 0x7fd1ac1cf9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:09:59.737 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.720+0000 7fd1aa59c640 1 -- 172.21.15.42:0/3516245281 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd1ac0739a0 msgr2=0x7fd1ac1cf480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:59.737 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.720+0000 7fd1aa59c640 1 --2- 172.21.15.42:0/3516245281 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd1ac0739a0 0x7fd1ac1cf480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:09:59.737 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.720+0000 7fd1aa59c640 10 monclient: _finish_hunting 0 2024-09-16T04:09:59.737 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.720+0000 7fd1aa59c640 1 monclient: found mon.noname-c 2024-09-16T04:09:59.738 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.720+0000 7fd1aa59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:59.738 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.720+0000 7fd1aa59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T04:09:59.738 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.720+0000 7fd1aa59c640 1 -- 172.21.15.42:0/3516245281 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1ac13abb0 con 0x7fd1ac073080 2024-09-16T04:09:59.738 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.720+0000 7fd1aa59c640 1 --2- 172.21.15.42:0/3516245281 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd1ac073080 0x7fd1ac1cef40 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fd19c007e60 tx=0x7fd19c00e4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:09:59.738 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.720+0000 7fd1a2ffd640 10 client.?.objecter ms_handle_connect 0x7fd1ac073080 2024-09-16T04:09:59.738 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.720+0000 7fd1a2ffd640 10 client.?.objecter resend_mon_ops 2024-09-16T04:09:59.738 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.720+0000 7fd1a2ffd640 1 -- 172.21.15.42:0/3516245281 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd19c014620 con 0x7fd1ac073080 2024-09-16T04:09:59.738 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.720+0000 7fd1a2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:59.738 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.720+0000 7fd1a2ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:09:59.738 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.720+0000 7fd1a2ffd640 10 monclient: dump: 2024-09-16T04:09:59.738 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:09:59.738 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:09:59.738 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:09:59.738 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:09:59.738 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:59.739 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:09:59.739 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:09:59.739 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:09:59.739 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:09:59.739 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:09:59.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.720+0000 7fd1a2ffd640 10 monclient: _finish_auth 0 2024-09-16T04:09:59.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.720+0000 7fd1a2ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:09:59.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.720+0000 7fd1a2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:59.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.720+0000 7fd1a2ffd640 1 -- 172.21.15.42:0/3516245281 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd19c019070 con 0x7fd1ac073080 2024-09-16T04:09:59.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.720+0000 7fd1b0e2e640 5 monclient: authenticate success, global_id 1014344 2024-09-16T04:09:59.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.720+0000 7fd1b0e2e640 10 monclient: _renew_subs 2024-09-16T04:09:59.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.720+0000 7fd1b0e2e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:09:59.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.720+0000 7fd1b0e2e640 1 -- 172.21.15.42:0/3516245281 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1ac13a2a0 con 0x7fd1ac073080 2024-09-16T04:09:59.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.720+0000 7fd1a2ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:59.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.720+0000 7fd1a2ffd640 1 -- 172.21.15.42:0/3516245281 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd19c0149e0 con 0x7fd1ac073080 2024-09-16T04:09:59.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.720+0000 7fd1a2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:59.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.720+0000 7fd1a2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:09:59.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.720+0000 7fd1a2ffd640 10 monclient: dump: 2024-09-16T04:09:59.740 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:09:59.740 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:09:59.740 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:09:59.740 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:09:59.740 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:59.740 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:09:59.740 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:09:59.740 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:09:59.740 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:09:59.740 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:09:59.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.720+0000 7fd1b0e2e640 10 client.1014344.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:09:59.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.720+0000 7fd1b0e2e640 10 monclient: _renew_subs 2024-09-16T04:09:59.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.720+0000 7fd1b0e2e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:09:59.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.720+0000 7fd1b0e2e640 1 -- 172.21.15.42:0/3516245281 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1ac072b90 con 0x7fd1ac073080 2024-09-16T04:09:59.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.720+0000 7fd1a8d99640 4 set_mon_vals no callback set 2024-09-16T04:09:59.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.720+0000 7fd1b0e2e640 1 librados: init done 2024-09-16T04:09:59.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.724+0000 7fd1b0e2e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:59.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.724+0000 7fd1b0e2e640 1 --2- 172.21.15.42:0/3516245281 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd1ac12d6c0 0x7fd1ac12daa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:09:59.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.724+0000 7fd1b0e2e640 10 monclient(hunting): start opening mon connection 2024-09-16T04:09:59.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.724+0000 7fd1b0e2e640 1 -- 172.21.15.42:0/3516245281 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fd1ac13adf0 con 0x7fd1ac12d6c0 2024-09-16T04:09:59.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.724+0000 7fd1a9d9b640 1 --2- 172.21.15.42:0/3516245281 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd1ac12d6c0 0x7fd1ac12daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:09:59.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.724+0000 7fd1a9d9b640 10 monclient: get_auth_request con 0x7fd1ac12d6c0 auth_method 0 2024-09-16T04:09:59.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.724+0000 7fd1a9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:09:59.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.724+0000 7fd1a9d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:59.742 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.724+0000 7fd1a9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:59.742 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.724+0000 7fd1a9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:09:59.742 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.724+0000 7fd1a9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:09:59.742 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.724+0000 7fd1a9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:09:59.742 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.724+0000 7fd1a2ffd640 1 -- 172.21.15.42:0/3516245281 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fd19c014b80 con 0x7fd1ac073080 2024-09-16T04:09:59.742 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.724+0000 7fd1a9d9b640 10 monclient(hunting): handle_auth_done global_id 1014347 payload 824 2024-09-16T04:09:59.742 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.724+0000 7fd1a9d9b640 1 -- 172.21.15.42:0/3516245281 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd1ac073460 con 0x7fd1ac12d6c0 2024-09-16T04:09:59.742 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.724+0000 7fd1a2ffd640 1 --2- 172.21.15.42:0/3516245281 >> v2:172.21.15.154:6812/254845342 conn(0x7fd18c037810 0x7fd18c039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:09:59.742 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.724+0000 7fd1a2ffd640 1 -- 172.21.15.42:0/3516245281 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(706..706 src has 451..706) ==== 5040+0+0 (secure 0 0 0) 0x7fd19c049470 con 0x7fd1ac073080 2024-09-16T04:09:59.742 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.724+0000 7fd1a2ffd640 10 client.1014344.objecter ms_dispatch 0x7fd1ac000ba0 osd_map(706..706 src has 451..706) 2024-09-16T04:09:59.742 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.724+0000 7fd1a2ffd640 3 client.1014344.objecter handle_osd_map got epochs [706,706] > 0 2024-09-16T04:09:59.742 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.724+0000 7fd1a2ffd640 3 client.1014344.objecter handle_osd_map decoding full epoch 706 2024-09-16T04:09:59.742 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.724+0000 7fd1a9d9b640 1 --2- 172.21.15.42:0/3516245281 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd1ac12d6c0 0x7fd1ac12daa0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fd1ac072880 tx=0x7fd19400fc00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:09:59.742 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.724+0000 7fd1a2ffd640 20 client.1014344.objecter dump_active .. 0 homeless 2024-09-16T04:09:59.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.724+0000 7fd1a2ffd640 10 client.1014344.objecter ms_handle_connect 0x7fd1ac12d6c0 2024-09-16T04:09:59.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.724+0000 7fd1a2ffd640 10 client.1014344.objecter resend_mon_ops 2024-09-16T04:09:59.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.724+0000 7fd1a2ffd640 1 -- 172.21.15.42:0/3516245281 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd194018030 con 0x7fd1ac12d6c0 2024-09-16T04:09:59.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.724+0000 7fd1a2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:59.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.724+0000 7fd1a2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:09:59.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.724+0000 7fd1a2ffd640 10 monclient: dump: 2024-09-16T04:09:59.743 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:09:59.743 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:09:59.743 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:09:59.743 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:09:59.743 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:59.743 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:09:59.743 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:09:59.743 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:09:59.743 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:09:59.744 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:09:59.744 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.724+0000 7fd1aad9d640 1 --2- 172.21.15.42:0/3516245281 >> v2:172.21.15.154:6812/254845342 conn(0x7fd18c037810 0x7fd18c039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:09:59.744 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.724+0000 7fd1aad9d640 10 monclient: get_auth_request con 0x7fd18c037810 auth_method 0 2024-09-16T04:09:59.744 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.724+0000 7fd1a2ffd640 1 -- 172.21.15.42:0/3516245281 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fd194003a30 con 0x7fd1ac12d6c0 2024-09-16T04:09:59.744 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.724+0000 7fd1a2ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:59.744 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.724+0000 7fd1a2ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:09:59.744 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.724+0000 7fd1a2ffd640 1 -- 172.21.15.42:0/3516245281 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd1ac12d6c0 msgr2=0x7fd1ac12daa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:59.744 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.724+0000 7fd1a2ffd640 1 --2- 172.21.15.42:0/3516245281 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd1ac12d6c0 0x7fd1ac12daa0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fd1ac072880 tx=0x7fd19400fc00 comp rx=0 tx=0).stop 2024-09-16T04:09:59.744 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.724+0000 7fd1a2ffd640 1 -- 172.21.15.42:0/3516245281 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd1ac12d6c0 msgr2=0x7fd1ac12daa0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:09:59.744 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.724+0000 7fd1a2ffd640 1 --2- 172.21.15.42:0/3516245281 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd1ac12d6c0 0x7fd1ac12daa0 secure :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fd1ac072880 tx=0x7fd19400fc00 comp rx=0 tx=0).stop 2024-09-16T04:09:59.744 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.724+0000 7fd1aad9d640 1 --2- 172.21.15.42:0/3516245281 >> v2:172.21.15.154:6812/254845342 conn(0x7fd18c037810 0x7fd18c039cd0 secure :-1 s=READY pgs=3918 cs=0 l=1 rev1=1 crypto rx=0x7fd1ac09fec0 tx=0x7fd1a4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:09:59.744 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.724+0000 7fd1a2ffd640 10 client.1014344.objecter ms_handle_connect 0x7fd18c037810 2024-09-16T04:09:59.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.728+0000 7fd1b0e2e640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:09:59.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.728+0000 7fd1b0e2e640 1 --2- 172.21.15.42:0/3516245281 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd1ac12dfe0 0x7fd1ac06e010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:09:59.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.728+0000 7fd1b0e2e640 10 monclient(hunting): start opening mon connection 2024-09-16T04:09:59.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.728+0000 7fd1b0e2e640 1 -- 172.21.15.42:0/3516245281 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fd1ac09e670 con 0x7fd1ac12dfe0 2024-09-16T04:09:59.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.732+0000 7fd1a9d9b640 1 --2- 172.21.15.42:0/3516245281 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd1ac12dfe0 0x7fd1ac06e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:09:59.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.732+0000 7fd1a9d9b640 10 monclient: get_auth_request con 0x7fd1ac12dfe0 auth_method 0 2024-09-16T04:09:59.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.732+0000 7fd1a9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:09:59.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.732+0000 7fd1a9d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:59.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.732+0000 7fd1a9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:59.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.732+0000 7fd1a9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:09:59.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.732+0000 7fd1a9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:09:59.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.732+0000 7fd1a9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:09:59.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.732+0000 7fd1a9d9b640 10 monclient(hunting): handle_auth_done global_id 1014350 payload 824 2024-09-16T04:09:59.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.732+0000 7fd1a9d9b640 1 -- 172.21.15.42:0/3516245281 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fd1ac06b3e0 con 0x7fd1ac12dfe0 2024-09-16T04:09:59.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.732+0000 7fd1a2ffd640 10 client.1014344.objecter ms_handle_connect 0x7fd1ac12dfe0 2024-09-16T04:09:59.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.732+0000 7fd1a2ffd640 10 client.1014344.objecter resend_mon_ops 2024-09-16T04:09:59.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.732+0000 7fd1a9d9b640 1 --2- 172.21.15.42:0/3516245281 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd1ac12dfe0 0x7fd1ac06e010 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fd194018000 tx=0x7fd194007f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:09:59.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.732+0000 7fd1a2ffd640 1 -- 172.21.15.42:0/3516245281 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd19400dac0 con 0x7fd1ac12dfe0 2024-09-16T04:09:59.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.732+0000 7fd1a2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:59.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.732+0000 7fd1a2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:09:59.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.732+0000 7fd1a2ffd640 10 monclient: dump: 2024-09-16T04:09:59.748 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:09:59.748 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:09:59.748 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:09:59.748 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:09:59.748 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:59.748 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:09:59.748 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:09:59.748 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:09:59.748 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:09:59.748 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:09:59.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.732+0000 7fd1a2ffd640 1 -- 172.21.15.42:0/3516245281 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4713 (secure 0 0 0) 0x7fd19400dac0 con 0x7fd1ac12dfe0 2024-09-16T04:09:59.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.732+0000 7fd1a2ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:09:59.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.732+0000 7fd1a2ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:09:59.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.732+0000 7fd1a2ffd640 1 -- 172.21.15.42:0/3516245281 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd1ac12dfe0 msgr2=0x7fd1ac06e010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:59.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.732+0000 7fd1a2ffd640 1 --2- 172.21.15.42:0/3516245281 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd1ac12dfe0 0x7fd1ac06e010 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fd194018000 tx=0x7fd194007f90 comp rx=0 tx=0).stop 2024-09-16T04:09:59.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.732+0000 7fd1a2ffd640 1 -- 172.21.15.42:0/3516245281 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd1ac12dfe0 msgr2=0x7fd1ac06e010 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:09:59.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.732+0000 7fd1a2ffd640 1 --2- 172.21.15.42:0/3516245281 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd1ac12dfe0 0x7fd1ac06e010 secure :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fd194018000 tx=0x7fd194007f90 comp rx=0 tx=0).stop 2024-09-16T04:09:59.749 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T04:09:59.749 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:09:59.749 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:09:59.749 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T04:09:59.750 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 256, 2024-09-16T04:09:59.750 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 194934, 2024-09-16T04:09:59.750 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T04:09:59.750 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T04:09:59.750 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T04:09:59.750 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T04:09:59.750 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:09:59.750 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 0, 2024-09-16T04:09:59.750 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:09:59.750 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T04:09:59.750 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T04:09:59.750 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:09:59.750 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:09:59.750 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:09:59.751 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:09:59.751 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:09:59.751 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:09:59.751 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:09:59.751 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:09:59.751 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:09:59.751 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:09:59.751 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:09:59.751 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:09:59.751 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:09:59.751 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T04:09:59.751 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:09:59.751 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:09:59.751 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:09:59.752 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:09:59.752 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:09:59.752 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:09:59.752 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:09:59.752 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:09:59.752 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:09:59.752 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:09:59.752 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:09:59.752 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:09:59.752 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:09:59.752 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T04:09:59.752 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T04:09:59.752 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T04:09:59.752 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T04:09:59.753 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T04:09:59.753 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T04:09:59.753 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:09:59.753 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:09:59.753 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T04:09:59.753 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T04:09:59.753 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T04:09:59.753 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T04:09:59.753 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T04:09:59.753 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T04:09:59.753 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T04:09:59.753 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:09:59.753 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T04:09:59.754 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:09:59.754 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:09:59.754 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:09:59.754 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:09:59.754 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:09:59.754 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:09:59.754 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:09:59.754 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:09:59.754 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:09:59.754 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:09:59.754 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:09:59.754 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:09:59.754 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T04:09:59.754 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:09:59.754 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T04:09:59.755 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:09:59.755 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:09:59.755 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:09:59.755 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:09:59.755 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:09:59.755 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:09:59.755 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:09:59.755 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T04:09:59.755 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:09:59.755 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:09:59.755 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:09:59.755 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:09:59.755 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T04:09:59.755 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:09:59.756 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:09:59.756 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:09:59.756 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:09:59.756 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T04:09:59.756 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T04:09:59.756 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:09:59.756 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:09:59.756 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:09:59.756 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:09:59.756 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:09:59.756 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:09:59.756 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:09:59.756 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:09:59.756 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:09:59.757 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:09:59.757 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:09:59.757 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T04:09:59.757 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:09:59.757 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:09:59.757 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:09:59.757 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:09:59.757 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T04:09:59.757 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:09:59.757 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:09:59.757 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:09:59.757 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:09:59.757 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T04:09:59.757 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T04:09:59.758 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:09:59.758 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:09:59.758 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:09:59.758 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:09:59.758 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:09:59.758 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:09:59.758 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:09:59.758 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:09:59.758 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:09:59.758 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:09:59.758 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:09:59.758 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T04:09:59.758 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:09:59.758 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:09:59.758 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:09:59.759 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:09:59.759 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T04:09:59.759 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:09:59.759 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:09:59.759 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:09:59.759 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:09:59.759 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T04:09:59.759 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T04:09:59.759 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:09:59.759 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:09:59.759 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:09:59.759 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:09:59.759 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:09:59.759 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:09:59.760 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T04:09:59.760 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T04:09:59.760 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:09:59.760 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:09:59.760 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:09:59.760 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:09:59.760 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:09:59.760 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:09:59.760 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T04:09:59.760 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:09:59.760 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:09:59.760 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:09:59.760 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 4 2024-09-16T04:09:59.760 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:09:59.761 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:09:59.761 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T04:09:59.761 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:09:59.761 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:09:59.761 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:09:59.761 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 4 2024-09-16T04:09:59.761 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:09:59.761 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:09:59.761 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:09:59.761 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T04:09:59.761 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T04:09:59.763 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.732+0000 7fd1b0e2e640 10 librados: watch_flush enter 2024-09-16T04:09:59.763 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.732+0000 7fd1b0e2e640 10 librados: watch_flush exit 2024-09-16T04:09:59.763 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.732+0000 7fd1b0e2e640 20 client.1014344.objecter shutdown clearing up homeless session... 2024-09-16T04:09:59.763 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.732+0000 7fd1b0e2e640 10 client.1014344.objecter successfully canceled tick 2024-09-16T04:09:59.763 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.732+0000 7fd1b0e2e640 1 -- 172.21.15.42:0/3516245281 >> v2:172.21.15.154:6812/254845342 conn(0x7fd18c037810 msgr2=0x7fd18c039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:59.763 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.732+0000 7fd1b0e2e640 1 --2- 172.21.15.42:0/3516245281 >> v2:172.21.15.154:6812/254845342 conn(0x7fd18c037810 0x7fd18c039cd0 secure :-1 s=READY pgs=3918 cs=0 l=1 rev1=1 crypto rx=0x7fd1ac09fec0 tx=0x7fd1a4008040 comp rx=0 tx=0).stop 2024-09-16T04:09:59.763 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.732+0000 7fd1b0e2e640 10 monclient: shutdown 2024-09-16T04:09:59.763 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.732+0000 7fd1b0e2e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:59.763 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.732+0000 7fd1b0e2e640 1 -- 172.21.15.42:0/3516245281 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd1ac073080 msgr2=0x7fd1ac1cef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:59.763 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.732+0000 7fd1b0e2e640 1 --2- 172.21.15.42:0/3516245281 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd1ac073080 0x7fd1ac1cef40 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fd19c007e60 tx=0x7fd19c00e4b0 comp rx=0 tx=0).stop 2024-09-16T04:09:59.764 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.732+0000 7fd1aad9d640 1 -- 172.21.15.42:0/3516245281 reap_dead start 2024-09-16T04:09:59.764 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.732+0000 7fd1aad9d640 1 -- 172.21.15.42:0/3516245281 reap_dead start 2024-09-16T04:09:59.764 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.732+0000 7fd1b0e2e640 1 -- 172.21.15.42:0/3516245281 shutdown_connections 2024-09-16T04:09:59.764 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.732+0000 7fd1b0e2e640 1 -- 172.21.15.42:0/3516245281 >> 172.21.15.42:0/3516245281 conn(0x7fd1ac06bc40 msgr2=0x7fd1ac0a4760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:59.764 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.732+0000 7fd1b0e2e640 1 -- 172.21.15.42:0/3516245281 shutdown_connections 2024-09-16T04:09:59.764 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.732+0000 7fd1b0e2e640 1 -- 172.21.15.42:0/3516245281 wait complete. 2024-09-16T04:09:59.764 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.732+0000 7fd1b0e2e640 1 librados: shutdown 2024-09-16T04:09:59.764 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:09:59.764 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T04:09:59.764 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T04:09:59.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.876+0000 7f8f6e4bb640 10 monclient: get_monmap_and_config 2024-09-16T04:09:59.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.876+0000 7f8f6e4bb640 10 monclient: build_initial_monmap 2024-09-16T04:09:59.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.876+0000 7f8f6e4bb640 10 monclient: monmap: 2024-09-16T04:09:59.878 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:09:59.878 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:09:59.878 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:09:59.876318+0000 2024-09-16T04:09:59.878 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:09:59.876318+0000 2024-09-16T04:09:59.878 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:59.878 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:09:59.878 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:09:59.878 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:09:59.878 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:09:59.878 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:09:59.878 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:09:59.878 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:09:59.883 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.876+0000 7f8f6e4bb640 1 Processor -- start 2024-09-16T04:09:59.883 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.876+0000 7f8f6e4bb640 1 -- start start 2024-09-16T04:09:59.883 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.876+0000 7f8f6e4bb640 10 monclient: init 2024-09-16T04:09:59.883 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.876+0000 7f8f6e4bb640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:59.883 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.876+0000 7f8f6e4bb640 10 monclient: _add_conns ranks=[1,4,0,2,3] 2024-09-16T04:09:59.883 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.876+0000 7f8f6e4bb640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8f680a2340 addr v1:172.21.15.154:6789/0 2024-09-16T04:09:59.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.876+0000 7f8f6e4bb640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f8f68070a70 0x7f8f6813a550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:09:59.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.876+0000 7f8f6e4bb640 10 monclient(hunting): picked mon.noname-c con 0x7f8f68070a70 addr v2:172.21.15.154:3300/0 2024-09-16T04:09:59.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.876+0000 7f8f6e4bb640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8f680a2e30 addr v1:172.21.15.42:6789/0 2024-09-16T04:09:59.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.876+0000 7f8f6e4bb640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8f68071a10 con 0x7f8f680a2e30 2024-09-16T04:09:59.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.876+0000 7f8f6e4bb640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8f68071ec0 con 0x7f8f680a2340 2024-09-16T04:09:59.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.876+0000 7f8f6e4bb640 10 monclient(hunting): start opening mon connection 2024-09-16T04:09:59.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.876+0000 7f8f6e4bb640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f8f6806c9a0 con 0x7f8f68070a70 2024-09-16T04:09:59.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.876+0000 7f8f6e4bb640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:59.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.876+0000 7f8f6e4bb640 10 monclient(hunting): authenticate will time out at 2646.845936s 2024-09-16T04:09:59.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.876+0000 7f8f6ca58640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f8f680a2e30 0x7f8f6813caa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:37314/0 (socket says 172.21.15.42:37314) 2024-09-16T04:09:59.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.876+0000 7f8f6ca58640 1 -- 172.21.15.42:0/2129262573 learned_addr learned my addr 172.21.15.42:0/2129262573 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T04:09:59.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.876+0000 7f8f677fe640 1 --2- 172.21.15.42:0/2129262573 >> v2:172.21.15.154:3300/0 conn(0x7f8f68070a70 0x7f8f6813a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:09:59.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.876+0000 7f8f677fe640 10 monclient(hunting): get_auth_request con 0x7f8f68070a70 auth_method 0 2024-09-16T04:09:59.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.876+0000 7f8f677fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:09:59.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.876+0000 7f8f677fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:59.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.876+0000 7f8f677fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:59.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.876+0000 7f8f66ffd640 1 -- 172.21.15.42:0/2129262573 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1528383482 0 0) 0x7f8f68071a10 con 0x7f8f680a2e30 2024-09-16T04:09:59.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.876+0000 7f8f66ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:09:59.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.876+0000 7f8f66ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:59.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.876+0000 7f8f66ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:59.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.876+0000 7f8f66ffd640 10 monclient(hunting): my global_id is 1034214 2024-09-16T04:09:59.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.876+0000 7f8f66ffd640 1 -- 172.21.15.42:0/2129262573 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8f50003530 con 0x7f8f680a2e30 2024-09-16T04:09:59.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.876+0000 7f8f677fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:09:59.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.876+0000 7f8f677fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:09:59.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.876+0000 7f8f677fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:09:59.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.876+0000 7f8f66ffd640 1 -- 172.21.15.42:0/2129262573 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1126479707 0 0) 0x7f8f50003530 con 0x7f8f680a2e30 2024-09-16T04:09:59.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.876+0000 7f8f66ffd640 1 -- 172.21.15.42:0/2129262573 >> v2:172.21.15.154:3300/0 conn(0x7f8f68070a70 msgr2=0x7f8f6813a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:09:59.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.876+0000 7f8f66ffd640 1 --2- 172.21.15.42:0/2129262573 >> v2:172.21.15.154:3300/0 conn(0x7f8f68070a70 0x7f8f6813a550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:09:59.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.876+0000 7f8f66ffd640 1 -- 172.21.15.42:0/2129262573 >> v1:172.21.15.154:6789/0 conn(0x7f8f680a2340 legacy=0x7f8f680a2720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:59.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.876+0000 7f8f66ffd640 10 monclient: _finish_hunting 0 2024-09-16T04:09:59.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.876+0000 7f8f66ffd640 1 monclient: found mon.noname-a-legacy 2024-09-16T04:09:59.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.876+0000 7f8f66ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:59.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.876+0000 7f8f66ffd640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.42:6789/0 2024-09-16T04:09:59.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.876+0000 7f8f66ffd640 1 -- 172.21.15.42:0/2129262573 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f6806bd60 con 0x7f8f680a2e30 2024-09-16T04:09:59.888 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.876+0000 7f8f66ffd640 10 monclient: _finish_auth 0 2024-09-16T04:09:59.888 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.876+0000 7f8f66ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:09:59.888 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.876+0000 7f8f66ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:59.888 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.876+0000 7f8f6e4bb640 5 monclient: authenticate success, global_id 1034214 2024-09-16T04:09:59.888 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.876+0000 7f8f66ffd640 1 -- 172.21.15.42:0/2129262573 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2177413734 0 0) 0x7f8f60001a00 con 0x7f8f680a2e30 2024-09-16T04:09:59.888 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.876+0000 7f8f6e4bb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:09:59.888 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.876+0000 7f8f66ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:59.888 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.876+0000 7f8f66ffd640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T04:09:59.888 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.876+0000 7f8f66ffd640 10 monclient: dump: 2024-09-16T04:09:59.888 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:09:59.888 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:09:59.889 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:09:59.889 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:09:59.889 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:59.889 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:09:59.889 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:09:59.889 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:09:59.889 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:09:59.889 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:09:59.889 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.876+0000 7f8f66ffd640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] but i'm connected to v1:172.21.15.42:6789/0, reconnecting 2024-09-16T04:09:59.889 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.876+0000 7f8f66ffd640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:59.889 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.876+0000 7f8f66ffd640 1 -- 172.21.15.42:0/2129262573 >> v1:172.21.15.42:6789/0 conn(0x7f8f680a2e30 legacy=0x7f8f6813caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:59.889 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.876+0000 7f8f66ffd640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:09:59.889 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.876+0000 7f8f66ffd640 1 --2- 172.21.15.42:0/2129262573 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8f500039c0 0x7f8f50003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:09:59.890 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.876+0000 7f8f66ffd640 10 monclient(hunting): picked mon.b con 0x7f8f500039c0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:09:59.890 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.876+0000 7f8f66ffd640 1 --2- 172.21.15.42:0/2129262573 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8f50004810 0x7f8f50004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:09:59.890 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.876+0000 7f8f66ffd640 10 monclient(hunting): picked mon.a con 0x7f8f50004810 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:09:59.890 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.876+0000 7f8f66ffd640 1 --2- 172.21.15.42:0/2129262573 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f8f500052f0 0x7f8f5000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:09:59.890 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.876+0000 7f8f66ffd640 10 monclient(hunting): picked mon.c con 0x7f8f500052f0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:09:59.890 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.876+0000 7f8f66ffd640 10 monclient(hunting): start opening mon connection 2024-09-16T04:09:59.890 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.876+0000 7f8f66ffd640 1 -- 172.21.15.42:0/2129262573 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f8f6806c9a0 con 0x7f8f50004810 2024-09-16T04:09:59.890 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.876+0000 7f8f66ffd640 10 monclient(hunting): start opening mon connection 2024-09-16T04:09:59.890 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.876+0000 7f8f66ffd640 1 -- 172.21.15.42:0/2129262573 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f8f5000c1c0 con 0x7f8f500039c0 2024-09-16T04:09:59.890 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.876+0000 7f8f66ffd640 10 monclient(hunting): start opening mon connection 2024-09-16T04:09:59.891 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.876+0000 7f8f66ffd640 1 -- 172.21.15.42:0/2129262573 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f8f5000c3c0 con 0x7f8f500052f0 2024-09-16T04:09:59.891 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.876+0000 7f8f66ffd640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:59.891 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.876+0000 7f8f66ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:09:59.891 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.876+0000 7f8f6e4bb640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-16T04:09:59.891 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.876+0000 7f8f677fe640 1 --2- 172.21.15.42:0/2129262573 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8f50004810 0x7f8f50004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:09:59.891 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.876+0000 7f8f677fe640 10 monclient(hunting): get_auth_request con 0x7f8f50004810 auth_method 0 2024-09-16T04:09:59.891 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.876+0000 7f8f677fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:09:59.891 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.876+0000 7f8f677fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:59.891 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.876+0000 7f8f677fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:09:59.891 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.876+0000 7f8f6ca58640 1 --2- 172.21.15.42:0/2129262573 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f8f500052f0 0x7f8f5000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:09:59.891 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.876+0000 7f8f677fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:09:59.891 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.876+0000 7f8f677fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:09:59.891 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.876+0000 7f8f677fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:09:59.891 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.876+0000 7f8f6ca58640 10 monclient(hunting): get_auth_request con 0x7f8f500052f0 auth_method 0 2024-09-16T04:09:59.892 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.876+0000 7f8f6ca58640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:09:59.892 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.876+0000 7f8f6ca58640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:59.892 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.876+0000 7f8f6ca58640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:09:59.892 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.876+0000 7f8f6ca58640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:09:59.892 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.876+0000 7f8f6ca58640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:09:59.892 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.876+0000 7f8f6ca58640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:09:59.892 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.876+0000 7f8f67fff640 1 --2- 172.21.15.42:0/2129262573 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8f500039c0 0x7f8f50003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:09:59.892 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.876+0000 7f8f677fe640 10 monclient(hunting): handle_auth_done global_id 1034214 payload 293 2024-09-16T04:09:59.892 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.876+0000 7f8f677fe640 1 -- 172.21.15.42:0/2129262573 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f8f500052f0 msgr2=0x7f8f5000b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:09:59.893 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.876+0000 7f8f677fe640 1 --2- 172.21.15.42:0/2129262573 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f8f500052f0 0x7f8f5000b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:09:59.893 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.876+0000 7f8f677fe640 1 -- 172.21.15.42:0/2129262573 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8f500039c0 msgr2=0x7f8f50003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:09:59.893 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.876+0000 7f8f677fe640 1 --2- 172.21.15.42:0/2129262573 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8f500039c0 0x7f8f50003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:09:59.893 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.876+0000 7f8f677fe640 10 monclient: _finish_hunting 0 2024-09-16T04:09:59.893 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.876+0000 7f8f677fe640 1 monclient: found mon.a 2024-09-16T04:09:59.893 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.876+0000 7f8f677fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:59.893 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.876+0000 7f8f677fe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:09:59.894 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.876+0000 7f8f677fe640 1 -- 172.21.15.42:0/2129262573 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f50003530 con 0x7f8f50004810 2024-09-16T04:09:59.894 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.876+0000 7f8f677fe640 10 monclient: _finish_auth 0 2024-09-16T04:09:59.894 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.876+0000 7f8f677fe640 10 monclient: _check_auth_tickets 2024-09-16T04:09:59.894 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.876+0000 7f8f677fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:59.894 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.876+0000 7f8f6ca58640 1 -- 172.21.15.42:0/2129262573 reap_dead start 2024-09-16T04:09:59.894 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.876+0000 7f8f67fff640 10 monclient: get_auth_request con 0x7f8f500039c0 auth_method 0 2024-09-16T04:09:59.894 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.876+0000 7f8f67fff640 1 --2- 172.21.15.42:0/2129262573 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8f500039c0 0x7f8f50003e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T04:09:59.894 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.880+0000 7f8f677fe640 1 --2- 172.21.15.42:0/2129262573 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8f50004810 0x7f8f50004c90 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f8f5c009ac0 tx=0x7f8f5c01ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f712150e2c045143 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:09:59.894 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.880+0000 7f8f66ffd640 1 -- 172.21.15.42:0/2129262573 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f5c009690 con 0x7f8f50004810 2024-09-16T04:09:59.894 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.880+0000 7f8f66ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:59.894 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.880+0000 7f8f66ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:09:59.894 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.880+0000 7f8f66ffd640 10 monclient: dump: 2024-09-16T04:09:59.895 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:09:59.895 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:09:59.895 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:09:59.895 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:09:59.896 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:59.896 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:09:59.896 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:09:59.896 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:09:59.896 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:09:59.896 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:09:59.896 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.880+0000 7f8f6e4bb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:09:59.896 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.880+0000 7f8f66ffd640 1 -- 172.21.15.42:0/2129262573 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8f5c0a6500 con 0x7f8f50004810 2024-09-16T04:09:59.896 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.880+0000 7f8f66ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:59.896 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.880+0000 7f8f6e4bb640 10 monclient: get_monmap_and_config success 2024-09-16T04:09:59.896 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.880+0000 7f8f6e4bb640 4 set_mon_vals no callback set 2024-09-16T04:09:59.896 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.880+0000 7f8f66ffd640 1 -- 172.21.15.42:0/2129262573 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f5c0a68d0 con 0x7f8f50004810 2024-09-16T04:09:59.896 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.880+0000 7f8f66ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:59.897 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.880+0000 7f8f66ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:09:59.897 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.880+0000 7f8f66ffd640 10 monclient: dump: 2024-09-16T04:09:59.897 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:09:59.897 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:09:59.897 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:09:59.897 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:09:59.897 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:59.897 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:09:59.897 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:09:59.897 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:09:59.897 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:09:59.897 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:09:59.897 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.880+0000 7f8f6e4bb640 10 monclient: shutdown 2024-09-16T04:09:59.897 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.880+0000 7f8f6e4bb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:09:59.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.880+0000 7f8f6e4bb640 1 -- 172.21.15.42:0/2129262573 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8f50004810 msgr2=0x7f8f50004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:59.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.880+0000 7f8f6e4bb640 1 --2- 172.21.15.42:0/2129262573 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8f50004810 0x7f8f50004c90 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f8f5c009ac0 tx=0x7f8f5c01ca30 comp rx=0 tx=0).stop 2024-09-16T04:09:59.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.880+0000 7f8f6e4bb640 1 -- 172.21.15.42:0/2129262573 shutdown_connections 2024-09-16T04:09:59.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.880+0000 7f8f6e4bb640 1 --2- 172.21.15.42:0/2129262573 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8f50004810 0x7f8f50004c90 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:09:59.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.880+0000 7f8f6e4bb640 1 -- 172.21.15.42:0/2129262573 >> 172.21.15.42:0/2129262573 conn(0x7f8f680a3430 msgr2=0x7f8f680a3810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:09:59.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.880+0000 7f8f6e4bb640 1 -- 172.21.15.42:0/2129262573 shutdown_connections 2024-09-16T04:09:59.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.880+0000 7f8f6e4bb640 1 -- 172.21.15.42:0/2129262573 wait complete. 2024-09-16T04:09:59.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.880+0000 7f8f6e4bb640 10 monclient: build_initial_monmap 2024-09-16T04:09:59.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.880+0000 7f8f6e4bb640 10 monclient: monmap: 2024-09-16T04:09:59.905 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:09:59.905 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:09:59.905 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:09:59.906 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:09:59.906 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:09:59.906 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:09:59.906 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:09:59.906 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:09:59.906 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:09:59.906 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:09:59.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.880+0000 7f8f6e4bb640 1 librados: starting msgr at 2024-09-16T04:09:59.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.880+0000 7f8f6e4bb640 1 librados: starting objecter 2024-09-16T04:09:59.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.880+0000 7f8f6e4bb640 1 Processor -- start 2024-09-16T04:09:59.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.880+0000 7f8f6e4bb640 1 -- start start 2024-09-16T04:09:59.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.880+0000 7f8f6e4bb640 1 librados: setting wanted keys 2024-09-16T04:09:59.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.880+0000 7f8f6e4bb640 1 librados: calling monclient init 2024-09-16T04:09:59.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.880+0000 7f8f6e4bb640 10 monclient: init 2024-09-16T04:09:59.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.880+0000 7f8f6e4bb640 10 monclient: _reopen_session rank -1 2024-09-16T04:09:59.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.880+0000 7f8f6e4bb640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:09:59.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.880+0000 7f8f6e4bb640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8f50004810 0x7f8f681cd5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:09:59.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.880+0000 7f8f6e4bb640 10 monclient(hunting): picked mon.noname-b con 0x7f8f50004810 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:09:59.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.880+0000 7f8f6e4bb640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8f681cdaf0 0x7f8f681226a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:09:59.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.880+0000 7f8f6e4bb640 10 monclient(hunting): picked mon.noname-a con 0x7f8f681cdaf0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:09:59.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.880+0000 7f8f6e4bb640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f8f681cdf30 0x7f8f68122be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:09:59.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.880+0000 7f8f6e4bb640 10 monclient(hunting): picked mon.noname-c con 0x7f8f681cdf30 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:09:59.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.880+0000 7f8f6e4bb640 10 monclient(hunting): start opening mon connection 2024-09-16T04:09:59.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.880+0000 7f8f6e4bb640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f8f680a4300 con 0x7f8f681cdaf0 2024-09-16T04:09:59.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.880+0000 7f8f6e4bb640 10 monclient(hunting): start opening mon connection 2024-09-16T04:09:59.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.880+0000 7f8f6e4bb640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f8f680a4180 con 0x7f8f50004810 2024-09-16T04:09:59.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.880+0000 7f8f6e4bb640 10 monclient(hunting): start opening mon connection 2024-09-16T04:09:59.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.880+0000 7f8f6e4bb640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f8f68135910 con 0x7f8f681cdf30 2024-09-16T04:09:59.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.880+0000 7f8f6e4bb640 10 monclient(hunting): _renew_subs 2024-09-16T04:09:59.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.880+0000 7f8f6e4bb640 10 monclient(hunting): authenticate will time out at 2646.851838s 2024-09-16T04:09:59.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.880+0000 7f8f677fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8f681cdaf0 0x7f8f681226a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:09:59.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.880+0000 7f8f6ca58640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f8f681cdf30 0x7f8f68122be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:09:59.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.880+0000 7f8f677fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8f681cdaf0 0x7f8f681226a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:60054/0 (socket says 172.21.15.42:60054) 2024-09-16T04:09:59.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.880+0000 7f8f677fe640 1 -- 172.21.15.42:0/2668623158 learned_addr learned my addr 172.21.15.42:0/2668623158 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:09:59.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.884+0000 7f8f677fe640 10 monclient(hunting): get_auth_request con 0x7f8f681cdaf0 auth_method 0 2024-09-16T04:09:59.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.884+0000 7f8f677fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:09:59.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.884+0000 7f8f677fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:59.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.884+0000 7f8f677fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:59.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.884+0000 7f8f6ca58640 10 monclient(hunting): get_auth_request con 0x7f8f681cdf30 auth_method 0 2024-09-16T04:09:59.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.884+0000 7f8f6ca58640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:09:59.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.884+0000 7f8f6ca58640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:59.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.884+0000 7f8f6ca58640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:59.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.884+0000 7f8f677fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:09:59.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.884+0000 7f8f677fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:09:59.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.884+0000 7f8f677fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:09:59.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.884+0000 7f8f6ca58640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:09:59.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.884+0000 7f8f6ca58640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:09:59.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.884+0000 7f8f6ca58640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:09:59.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.884+0000 7f8f67fff640 1 --2- 172.21.15.42:0/2668623158 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8f50004810 0x7f8f681cd5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:09:59.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.884+0000 7f8f67fff640 10 monclient(hunting): get_auth_request con 0x7f8f50004810 auth_method 0 2024-09-16T04:09:59.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.884+0000 7f8f67fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:09:59.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.884+0000 7f8f67fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:59.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.884+0000 7f8f67fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:59.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.884+0000 7f8f677fe640 10 monclient(hunting): handle_auth_done global_id 1034217 payload 824 2024-09-16T04:09:59.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.884+0000 7f8f677fe640 1 -- 172.21.15.42:0/2668623158 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f8f681cdf30 msgr2=0x7f8f68122be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:59.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.884+0000 7f8f677fe640 1 --2- 172.21.15.42:0/2668623158 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f8f681cdf30 0x7f8f68122be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:09:59.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.884+0000 7f8f677fe640 1 -- 172.21.15.42:0/2668623158 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8f50004810 msgr2=0x7f8f681cd5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:59.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.884+0000 7f8f677fe640 1 --2- 172.21.15.42:0/2668623158 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8f50004810 0x7f8f681cd5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:09:59.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.884+0000 7f8f677fe640 10 monclient: _finish_hunting 0 2024-09-16T04:09:59.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.884+0000 7f8f677fe640 1 monclient: found mon.noname-a 2024-09-16T04:09:59.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.884+0000 7f8f677fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:09:59.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.884+0000 7f8f677fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:09:59.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.884+0000 7f8f677fe640 1 -- 172.21.15.42:0/2668623158 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f68126070 con 0x7f8f681cdaf0 2024-09-16T04:09:59.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.884+0000 7f8f6ca58640 1 --2- 172.21.15.42:0/2668623158 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f8f681cdf30 0x7f8f68122be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:09:59.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.884+0000 7f8f677fe640 1 --2- 172.21.15.42:0/2668623158 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8f681cdaf0 0x7f8f681226a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f8f5c0b0040 tx=0x7f8f50005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:09:59.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.884+0000 7f8f64ff9640 10 client.?.objecter ms_handle_connect 0x7f8f681cdaf0 2024-09-16T04:09:59.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.884+0000 7f8f64ff9640 10 client.?.objecter resend_mon_ops 2024-09-16T04:09:59.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.884+0000 7f8f64ff9640 1 -- 172.21.15.42:0/2668623158 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f5c0a68b0 con 0x7f8f681cdaf0 2024-09-16T04:09:59.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.884+0000 7f8f64ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:59.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.884+0000 7f8f64ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:09:59.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.884+0000 7f8f64ff9640 10 monclient: dump: 2024-09-16T04:09:59.911 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:09:59.911 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:09:59.911 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:09:59.911 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:09:59.911 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:59.911 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:09:59.911 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:09:59.911 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:09:59.911 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:09:59.911 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:09:59.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.884+0000 7f8f64ff9640 10 monclient: _finish_auth 0 2024-09-16T04:09:59.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.884+0000 7f8f64ff9640 10 monclient: _check_auth_tickets 2024-09-16T04:09:59.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.884+0000 7f8f64ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:09:59.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.884+0000 7f8f6e4bb640 5 monclient: authenticate success, global_id 1034217 2024-09-16T04:09:59.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.884+0000 7f8f64ff9640 1 -- 172.21.15.42:0/2668623158 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8f5c0a6a50 con 0x7f8f681cdaf0 2024-09-16T04:09:59.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.884+0000 7f8f64ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T04:09:59.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.884+0000 7f8f64ff9640 1 -- 172.21.15.42:0/2668623158 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f5c0a6e20 con 0x7f8f681cdaf0 2024-09-16T04:09:59.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.884+0000 7f8f64ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:59.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.884+0000 7f8f64ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:09:59.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.884+0000 7f8f64ff9640 10 monclient: dump: 2024-09-16T04:09:59.912 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:09:59.912 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:09:59.912 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:09:59.912 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:09:59.913 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:59.913 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:09:59.913 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:09:59.913 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:09:59.913 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:09:59.913 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:09:59.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.884+0000 7f8f65ffb640 4 set_mon_vals no callback set 2024-09-16T04:09:59.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.884+0000 7f8f6e4bb640 10 monclient: _renew_subs 2024-09-16T04:09:59.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.884+0000 7f8f6e4bb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:09:59.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.884+0000 7f8f6e4bb640 1 -- 172.21.15.42:0/2668623158 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f681247a0 con 0x7f8f681cdaf0 2024-09-16T04:09:59.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.884+0000 7f8f6e4bb640 10 client.1034217.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:09:59.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.884+0000 7f8f6e4bb640 10 monclient: _renew_subs 2024-09-16T04:09:59.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.884+0000 7f8f6e4bb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:09:59.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.884+0000 7f8f6e4bb640 1 -- 172.21.15.42:0/2668623158 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f68134ef0 con 0x7f8f681cdaf0 2024-09-16T04:09:59.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.884+0000 7f8f6e4bb640 1 librados: init done 2024-09-16T04:09:59.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.884+0000 7f8f6e4bb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:59.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.884+0000 7f8f6e4bb640 1 --2- 172.21.15.42:0/2668623158 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8f68123120 0x7f8f68123500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:09:59.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.884+0000 7f8f6e4bb640 10 monclient(hunting): start opening mon connection 2024-09-16T04:09:59.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.884+0000 7f8f6e4bb640 1 -- 172.21.15.42:0/2668623158 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f8f6809fc70 con 0x7f8f68123120 2024-09-16T04:09:59.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.884+0000 7f8f64ff9640 1 -- 172.21.15.42:0/2668623158 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f8f5c006040 con 0x7f8f681cdaf0 2024-09-16T04:09:59.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.884+0000 7f8f64ff9640 1 --2- 172.21.15.42:0/2668623158 >> v2:172.21.15.154:6812/254845342 conn(0x7f8f54037810 0x7f8f54039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:09:59.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.884+0000 7f8f67fff640 1 --2- 172.21.15.42:0/2668623158 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8f68123120 0x7f8f68123500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:09:59.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.884+0000 7f8f64ff9640 1 -- 172.21.15.42:0/2668623158 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(706..706 src has 451..706) ==== 5040+0+0 (secure 0 0 0) 0x7f8f5c0e3920 con 0x7f8f681cdaf0 2024-09-16T04:09:59.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.884+0000 7f8f64ff9640 10 client.1034217.objecter ms_dispatch 0x7f8f68000ba0 osd_map(706..706 src has 451..706) 2024-09-16T04:09:59.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.884+0000 7f8f64ff9640 3 client.1034217.objecter handle_osd_map got epochs [706,706] > 0 2024-09-16T04:09:59.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.884+0000 7f8f64ff9640 3 client.1034217.objecter handle_osd_map decoding full epoch 706 2024-09-16T04:09:59.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.884+0000 7f8f67fff640 10 monclient: get_auth_request con 0x7f8f68123120 auth_method 0 2024-09-16T04:09:59.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.884+0000 7f8f67fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:09:59.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.884+0000 7f8f67fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:59.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.884+0000 7f8f67fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:59.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.884+0000 7f8f64ff9640 20 client.1034217.objecter dump_active .. 0 homeless 2024-09-16T04:09:59.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.884+0000 7f8f67fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:09:59.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.884+0000 7f8f67fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:09:59.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.884+0000 7f8f67fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:09:59.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.884+0000 7f8f6ca58640 1 --2- 172.21.15.42:0/2668623158 >> v2:172.21.15.154:6812/254845342 conn(0x7f8f54037810 0x7f8f54039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:09:59.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.884+0000 7f8f6ca58640 10 monclient: get_auth_request con 0x7f8f54037810 auth_method 0 2024-09-16T04:09:59.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.884+0000 7f8f67fff640 10 monclient(hunting): handle_auth_done global_id 1044121 payload 824 2024-09-16T04:09:59.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.884+0000 7f8f67fff640 1 -- 172.21.15.42:0/2668623158 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8f6809c0d0 con 0x7f8f68123120 2024-09-16T04:09:59.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.884+0000 7f8f64ff9640 10 client.1034217.objecter ms_handle_connect 0x7f8f68123120 2024-09-16T04:09:59.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.884+0000 7f8f64ff9640 10 client.1034217.objecter resend_mon_ops 2024-09-16T04:09:59.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.884+0000 7f8f67fff640 1 --2- 172.21.15.42:0/2668623158 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8f68123120 0x7f8f68123500 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f8f6806a7c0 tx=0x7f8f5800f8e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:09:59.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.884+0000 7f8f64ff9640 10 client.1034217.objecter ms_handle_connect 0x7f8f54037810 2024-09-16T04:09:59.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.884+0000 7f8f6ca58640 1 --2- 172.21.15.42:0/2668623158 >> v2:172.21.15.154:6812/254845342 conn(0x7f8f54037810 0x7f8f54039cd0 secure :-1 s=READY pgs=3919 cs=0 l=1 rev1=1 crypto rx=0x7f8f68068270 tx=0x7f8f60006040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:09:59.916 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.884+0000 7f8f64ff9640 1 -- 172.21.15.42:0/2668623158 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f58018070 con 0x7f8f68123120 2024-09-16T04:09:59.916 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.884+0000 7f8f64ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:09:59.916 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.884+0000 7f8f64ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:09:59.916 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.884+0000 7f8f64ff9640 10 monclient: dump: 2024-09-16T04:09:59.916 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:09:59.916 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:09:59.916 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:09:59.916 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:09:59.916 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:09:59.916 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:09:59.916 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:09:59.916 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:09:59.916 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:09:59.916 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:09:59.917 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.888+0000 7f8f64ff9640 1 -- 172.21.15.42:0/2668623158 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f8f580053a0 con 0x7f8f68123120 2024-09-16T04:09:59.917 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.888+0000 7f8f64ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:09:59.917 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.888+0000 7f8f64ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:09:59.917 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.888+0000 7f8f64ff9640 1 -- 172.21.15.42:0/2668623158 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8f68123120 msgr2=0x7f8f68123500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:09:59.917 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.888+0000 7f8f64ff9640 1 --2- 172.21.15.42:0/2668623158 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8f68123120 0x7f8f68123500 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f8f6806a7c0 tx=0x7f8f5800f8e0 comp rx=0 tx=0).stop 2024-09-16T04:09:59.917 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.888+0000 7f8f64ff9640 1 -- 172.21.15.42:0/2668623158 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8f68123120 msgr2=0x7f8f68123500 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:09:59.917 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.888+0000 7f8f64ff9640 1 --2- 172.21.15.42:0/2668623158 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8f68123120 0x7f8f68123500 secure :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f8f6806a7c0 tx=0x7f8f5800f8e0 comp rx=0 tx=0).stop 2024-09-16T04:09:59.917 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.892+0000 7f8f6e4bb640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:09:59.917 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.892+0000 7f8f6e4bb640 1 --2- 172.21.15.42:0/2668623158 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8f6809d3c0 0x7f8f6809d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:09:59.917 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.892+0000 7f8f6e4bb640 10 monclient(hunting): start opening mon connection 2024-09-16T04:09:59.917 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.892+0000 7f8f6e4bb640 1 -- 172.21.15.42:0/2668623158 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f8f68123a40 con 0x7f8f6809d3c0 2024-09-16T04:09:59.917 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.892+0000 7f8f67fff640 1 --2- 172.21.15.42:0/2668623158 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8f6809d3c0 0x7f8f6809d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:09:59.917 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.892+0000 7f8f67fff640 10 monclient: get_auth_request con 0x7f8f6809d3c0 auth_method 0 2024-09-16T04:09:59.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.892+0000 7f8f67fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:09:59.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.892+0000 7f8f67fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:09:59.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.892+0000 7f8f67fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:09:59.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.892+0000 7f8f67fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:09:59.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.892+0000 7f8f67fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:09:59.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.892+0000 7f8f67fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:09:59.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.892+0000 7f8f67fff640 10 monclient(hunting): handle_auth_done global_id 1044124 payload 824 2024-09-16T04:09:59.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.892+0000 7f8f67fff640 1 -- 172.21.15.42:0/2668623158 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f8f68126590 con 0x7f8f6809d3c0 2024-09-16T04:09:59.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.896+0000 7f8f67fff640 1 -- 172.21.15.42:0/2668623158 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8f6809d3c0 msgr2=0x7f8f6809d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-16T04:09:59.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.896+0000 7f8f67fff640 1 -- 172.21.15.42:0/2668623158 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8f6809d3c0 msgr2=0x7f8f6809d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:09:59.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.896+0000 7f8f67fff640 1 --2- 172.21.15.42:0/2668623158 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8f6809d3c0 0x7f8f6809d7d0 secure :-1 s=COMPRESSION_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0x7f8f58015000 tx=0x7f8f58001130 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:09:59.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:09:59.896+0000 7f8f67fff640 1 --2- 172.21.15.42:0/2668623158 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8f6809d3c0 0x7f8f6809d7d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:10:00.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.100+0000 7f8f67fff640 1 --2- 172.21.15.42:0/2668623158 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8f6809d3c0 0x7f8f6809d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:10:00.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.100+0000 7f8f67fff640 10 monclient: get_auth_request con 0x7f8f6809d3c0 auth_method 0 2024-09-16T04:10:00.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.100+0000 7f8f67fff640 10 monclient: get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:10:00.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.100+0000 7f8f67fff640 10 monclient: _init_auth method 2 2024-09-16T04:10:00.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.100+0000 7f8f67fff640 10 monclient: _init_auth already have auth, reseting 2024-09-16T04:10:00.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.100+0000 7f8f67fff640 10 monclient: handle_auth_reply_more payload 9 2024-09-16T04:10:00.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.100+0000 7f8f67fff640 10 monclient: handle_auth_reply_more payload_len 9 2024-09-16T04:10:00.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.100+0000 7f8f67fff640 10 monclient: handle_auth_reply_more responding with 132 bytes 2024-09-16T04:10:00.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.100+0000 7f8f67fff640 10 monclient: handle_auth_done global_id 1044124 payload 293 2024-09-16T04:10:00.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.100+0000 7f8f67fff640 1 --2- 172.21.15.42:0/2668623158 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8f6809d3c0 0x7f8f6809d7d0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f8f58009010 tx=0x7f8f5800b000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:10:00.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.100+0000 7f8f64ff9640 10 client.1034217.objecter ms_handle_connect 0x7f8f6809d3c0 2024-09-16T04:10:00.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.100+0000 7f8f64ff9640 10 client.1034217.objecter resend_mon_ops 2024-09-16T04:10:00.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.100+0000 7f8f64ff9640 1 -- 172.21.15.42:0/2668623158 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f5802b020 con 0x7f8f6809d3c0 2024-09-16T04:10:00.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.100+0000 7f8f64ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:00.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.100+0000 7f8f64ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:10:00.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.100+0000 7f8f64ff9640 10 monclient: dump: 2024-09-16T04:10:00.104 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:10:00.104 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:10:00.104 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:00.105 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:00.105 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:00.105 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:10:00.105 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:10:00.105 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:10:00.105 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:10:00.105 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:10:00.105 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.100+0000 7f8f64ff9640 1 -- 172.21.15.42:0/2668623158 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4545 (secure 0 0 0) 0x7f8f58019040 con 0x7f8f6809d3c0 2024-09-16T04:10:00.105 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.100+0000 7f8f64ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:10:00.105 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.100+0000 7f8f64ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:10:00.105 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.100+0000 7f8f64ff9640 1 -- 172.21.15.42:0/2668623158 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8f6809d3c0 msgr2=0x7f8f6809d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:00.105 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.100+0000 7f8f64ff9640 1 --2- 172.21.15.42:0/2668623158 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8f6809d3c0 0x7f8f6809d7d0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f8f58009010 tx=0x7f8f5800b000 comp rx=0 tx=0).stop 2024-09-16T04:10:00.105 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.100+0000 7f8f64ff9640 1 -- 172.21.15.42:0/2668623158 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8f6809d3c0 msgr2=0x7f8f6809d7d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:10:00.105 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.100+0000 7f8f64ff9640 1 --2- 172.21.15.42:0/2668623158 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8f6809d3c0 0x7f8f6809d7d0 secure :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f8f58009010 tx=0x7f8f5800b000 comp rx=0 tx=0).stop 2024-09-16T04:10:00.106 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T04:10:00.106 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:10:00.106 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:10:00.106 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T04:10:00.106 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 256, 2024-09-16T04:10:00.106 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 20791, 2024-09-16T04:10:00.106 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T04:10:00.106 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T04:10:00.106 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T04:10:00.106 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T04:10:00.107 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:10:00.107 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 0, 2024-09-16T04:10:00.107 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:10:00.107 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T04:10:00.107 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T04:10:00.107 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:10:00.107 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:10:00.107 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:10:00.107 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:10:00.107 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:10:00.107 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:10:00.107 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:10:00.107 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:10:00.107 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:10:00.108 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:10:00.108 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:10:00.108 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:10:00.108 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:10:00.108 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T04:10:00.108 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:10:00.108 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:10:00.108 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:10:00.108 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:10:00.108 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:10:00.108 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:10:00.108 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:10:00.108 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:10:00.109 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:10:00.109 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:10:00.109 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:10:00.109 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:10:00.109 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:10:00.109 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T04:10:00.109 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T04:10:00.109 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T04:10:00.109 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T04:10:00.109 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T04:10:00.109 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T04:10:00.109 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:10:00.109 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:10:00.109 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T04:10:00.110 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T04:10:00.110 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T04:10:00.110 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T04:10:00.110 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T04:10:00.110 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T04:10:00.110 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T04:10:00.110 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:10:00.110 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T04:10:00.110 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:10:00.110 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:10:00.110 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:10:00.110 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:10:00.110 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:10:00.110 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:10:00.111 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:10:00.111 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:10:00.111 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:10:00.111 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:10:00.111 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:10:00.111 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:10:00.111 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T04:10:00.111 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:10:00.111 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T04:10:00.111 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:10:00.111 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:10:00.111 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:10:00.111 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:10:00.111 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:10:00.112 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:10:00.112 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:10:00.112 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T04:10:00.112 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:10:00.112 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:10:00.112 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:10:00.112 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:10:00.112 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T04:10:00.112 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:10:00.112 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:10:00.112 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:10:00.112 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:10:00.112 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T04:10:00.113 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T04:10:00.113 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:10:00.113 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:10:00.113 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:10:00.113 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:10:00.113 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:10:00.114 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:10:00.114 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:10:00.114 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:10:00.114 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:10:00.114 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:10:00.114 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:10:00.114 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T04:10:00.114 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:10:00.114 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:10:00.114 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:10:00.114 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:10:00.115 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T04:10:00.115 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:10:00.115 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:10:00.115 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:10:00.115 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:10:00.115 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T04:10:00.115 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T04:10:00.115 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:10:00.115 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:10:00.115 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:10:00.116 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:10:00.116 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:10:00.116 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:10:00.116 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:10:00.116 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:10:00.116 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:10:00.116 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:10:00.116 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:10:00.116 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T04:10:00.116 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:10:00.116 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:10:00.116 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:10:00.116 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:10:00.116 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T04:10:00.117 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:10:00.117 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:10:00.117 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:10:00.117 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:10:00.117 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T04:10:00.117 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T04:10:00.117 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:10:00.117 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:10:00.117 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:10:00.117 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:10:00.117 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:10:00.117 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:10:00.117 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T04:10:00.118 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T04:10:00.118 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:10:00.118 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:10:00.118 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:10:00.118 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:10:00.118 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:10:00.118 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:10:00.118 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T04:10:00.118 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:10:00.118 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:10:00.118 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:10:00.118 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:10:00.118 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:10:00.118 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:10:00.118 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:10:00.119 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T04:10:00.119 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T04:10:00.120 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.100+0000 7f8f6e4bb640 10 librados: watch_flush enter 2024-09-16T04:10:00.120 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.100+0000 7f8f6e4bb640 10 librados: watch_flush exit 2024-09-16T04:10:00.120 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.100+0000 7f8f6e4bb640 20 client.1034217.objecter shutdown clearing up homeless session... 2024-09-16T04:10:00.120 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.100+0000 7f8f6e4bb640 10 client.1034217.objecter successfully canceled tick 2024-09-16T04:10:00.120 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.100+0000 7f8f6e4bb640 1 -- 172.21.15.42:0/2668623158 >> v2:172.21.15.154:6812/254845342 conn(0x7f8f54037810 msgr2=0x7f8f54039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:00.120 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.100+0000 7f8f6e4bb640 1 --2- 172.21.15.42:0/2668623158 >> v2:172.21.15.154:6812/254845342 conn(0x7f8f54037810 0x7f8f54039cd0 secure :-1 s=READY pgs=3919 cs=0 l=1 rev1=1 crypto rx=0x7f8f68068270 tx=0x7f8f60006040 comp rx=0 tx=0).stop 2024-09-16T04:10:00.120 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.100+0000 7f8f6e4bb640 10 monclient: shutdown 2024-09-16T04:10:00.120 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.100+0000 7f8f6e4bb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:10:00.120 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.100+0000 7f8f6e4bb640 1 -- 172.21.15.42:0/2668623158 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8f681cdaf0 msgr2=0x7f8f681226a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:00.120 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.100+0000 7f8f6e4bb640 1 --2- 172.21.15.42:0/2668623158 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8f681cdaf0 0x7f8f681226a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f8f5c0b0040 tx=0x7f8f50005840 comp rx=0 tx=0).stop 2024-09-16T04:10:00.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.100+0000 7f8f6ca58640 1 -- 172.21.15.42:0/2668623158 reap_dead start 2024-09-16T04:10:00.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.100+0000 7f8f6e4bb640 1 -- 172.21.15.42:0/2668623158 shutdown_connections 2024-09-16T04:10:00.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.100+0000 7f8f6e4bb640 1 --2- 172.21.15.42:0/2668623158 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8f681cdaf0 0x7f8f681226a0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:10:00.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.100+0000 7f8f6e4bb640 1 -- 172.21.15.42:0/2668623158 >> 172.21.15.42:0/2668623158 conn(0x7f8f680a3430 msgr2=0x7f8f6809f380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:10:00.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.100+0000 7f8f6e4bb640 1 -- 172.21.15.42:0/2668623158 shutdown_connections 2024-09-16T04:10:00.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.104+0000 7f8f6e4bb640 1 -- 172.21.15.42:0/2668623158 wait complete. 2024-09-16T04:10:00.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.104+0000 7f8f6e4bb640 1 librados: shutdown 2024-09-16T04:10:00.128 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:10:00.129 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T04:10:00.129 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T04:10:00.264 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.260+0000 7f543d9b1640 10 monclient: get_monmap_and_config 2024-09-16T04:10:00.264 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.260+0000 7f543d9b1640 10 monclient: build_initial_monmap 2024-09-16T04:10:00.264 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.260+0000 7f543d9b1640 10 monclient: monmap: 2024-09-16T04:10:00.264 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:10:00.264 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:10:00.264 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:10:00.262939+0000 2024-09-16T04:10:00.264 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:10:00.262939+0000 2024-09-16T04:10:00.265 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:10:00.265 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:10:00.265 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:10:00.265 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:10:00.265 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:10:00.265 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:10:00.265 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:10:00.265 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:10:00.267 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.260+0000 7f543d9b1640 1 Processor -- start 2024-09-16T04:10:00.267 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.260+0000 7f543d9b1640 1 -- start start 2024-09-16T04:10:00.267 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.260+0000 7f543d9b1640 10 monclient: init 2024-09-16T04:10:00.267 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.264+0000 7f543d9b1640 10 monclient: _reopen_session rank -1 2024-09-16T04:10:00.268 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.264+0000 7f543d9b1640 10 monclient: _add_conns ranks=[4,3,2,1,0] 2024-09-16T04:10:00.268 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.264+0000 7f543d9b1640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f54380a39b0 0x7f5438133b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:10:00.268 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.264+0000 7f543d9b1640 10 monclient(hunting): picked mon.noname-c con 0x7f54380a39b0 addr v2:172.21.15.154:3300/0 2024-09-16T04:10:00.268 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.264+0000 7f543d9b1640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f54380a3d90 0x7f5438137000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:10:00.268 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.264+0000 7f543d9b1640 10 monclient(hunting): picked mon.noname-a con 0x7f54380a3d90 addr v2:172.21.15.42:3300/0 2024-09-16T04:10:00.268 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.264+0000 7f543d9b1640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5438137540 0x7f5438137990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:10:00.268 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.264+0000 7f543d9b1640 10 monclient(hunting): picked mon.noname-b con 0x7f5438137540 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:10:00.268 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.264+0000 7f543d9b1640 10 monclient(hunting): start opening mon connection 2024-09-16T04:10:00.268 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.264+0000 7f543d9b1640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f543809c140 con 0x7f54380a3d90 2024-09-16T04:10:00.268 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.264+0000 7f543d9b1640 10 monclient(hunting): start opening mon connection 2024-09-16T04:10:00.268 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.264+0000 7f543d9b1640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f543809c420 con 0x7f54380a39b0 2024-09-16T04:10:00.268 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.264+0000 7f543d9b1640 10 monclient(hunting): start opening mon connection 2024-09-16T04:10:00.268 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.264+0000 7f543d9b1640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f543806dd60 con 0x7f5438137540 2024-09-16T04:10:00.268 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.264+0000 7f543d9b1640 10 monclient(hunting): _renew_subs 2024-09-16T04:10:00.269 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.264+0000 7f543d9b1640 10 monclient(hunting): authenticate will time out at 2647.232379s 2024-09-16T04:10:00.269 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.264+0000 7f54377fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5438137540 0x7f5438137990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:10:00.269 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.264+0000 7f54367fc640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f54380a3d90 0x7f5438137000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:10:00.269 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.264+0000 7f54377fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5438137540 0x7f5438137990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:51172/0 (socket says 172.21.15.42:51172) 2024-09-16T04:10:00.269 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.264+0000 7f54377fe640 1 -- 172.21.15.42:0/2969455917 learned_addr learned my addr 172.21.15.42:0/2969455917 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:10:00.269 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.264+0000 7f54367fc640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f54380a3d90 0x7f5438137000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:60058/0 (socket says 172.21.15.42:60058) 2024-09-16T04:10:00.269 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.264+0000 7f54377fe640 10 monclient(hunting): get_auth_request con 0x7f5438137540 auth_method 0 2024-09-16T04:10:00.269 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.264+0000 7f54377fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:10:00.269 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.264+0000 7f54377fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:00.269 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.264+0000 7f54377fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:00.269 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.264+0000 7f54367fc640 10 monclient(hunting): get_auth_request con 0x7f54380a3d90 auth_method 0 2024-09-16T04:10:00.269 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.264+0000 7f54367fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:10:00.269 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.264+0000 7f54367fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:00.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.264+0000 7f54367fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:00.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.264+0000 7f54377fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:10:00.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.264+0000 7f54377fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:10:00.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.264+0000 7f54377fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:10:00.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.264+0000 7f5436ffd640 1 --2- 172.21.15.42:0/2969455917 >> v2:172.21.15.154:3300/0 conn(0x7f54380a39b0 0x7f5438133b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:10:00.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.264+0000 7f54367fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:10:00.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.264+0000 7f54367fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:10:00.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.264+0000 7f54367fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:10:00.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.264+0000 7f5436ffd640 10 monclient(hunting): get_auth_request con 0x7f54380a39b0 auth_method 0 2024-09-16T04:10:00.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.264+0000 7f5436ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:10:00.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.264+0000 7f5436ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:00.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.264+0000 7f5436ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:00.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.264+0000 7f54377fe640 10 monclient(hunting): handle_auth_done global_id 1014356 payload 274 2024-09-16T04:10:00.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.264+0000 7f54377fe640 1 -- 172.21.15.42:0/2969455917 >> v2:172.21.15.154:3300/0 conn(0x7f54380a39b0 msgr2=0x7f5438133b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:10:00.271 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.264+0000 7f54377fe640 1 --2- 172.21.15.42:0/2969455917 >> v2:172.21.15.154:3300/0 conn(0x7f54380a39b0 0x7f5438133b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:10:00.271 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.264+0000 7f54377fe640 1 -- 172.21.15.42:0/2969455917 >> v2:172.21.15.42:3300/0 conn(0x7f54380a3d90 msgr2=0x7f5438137000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:10:00.271 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.264+0000 7f54377fe640 1 --2- 172.21.15.42:0/2969455917 >> v2:172.21.15.42:3300/0 conn(0x7f54380a3d90 0x7f5438137000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:10:00.271 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.264+0000 7f54377fe640 10 monclient: _finish_hunting 0 2024-09-16T04:10:00.271 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.264+0000 7f54377fe640 1 monclient: found mon.noname-b 2024-09-16T04:10:00.271 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.264+0000 7f54377fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:10:00.271 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.264+0000 7f54377fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T04:10:00.271 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.264+0000 7f54377fe640 1 -- 172.21.15.42:0/2969455917 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54380a5e20 con 0x7f5438137540 2024-09-16T04:10:00.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.264+0000 7f54377fe640 1 --2- 172.21.15.42:0/2969455917 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5438137540 0x7f5438137990 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f54380a7750 tx=0x7f543001c8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7065e904a3b46456 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:10:00.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.264+0000 7f5435ffb640 1 -- 172.21.15.42:0/2969455917 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54300a5530 con 0x7f5438137540 2024-09-16T04:10:00.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.268+0000 7f5435ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:00.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.268+0000 7f5435ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:10:00.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.268+0000 7f5435ffb640 10 monclient: dump: 2024-09-16T04:10:00.277 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:10:00.277 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:10:00.277 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:00.277 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:00.277 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:00.277 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:10:00.278 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:10:00.278 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:10:00.278 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:10:00.278 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:10:00.278 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.268+0000 7f5435ffb640 10 monclient: _finish_auth 0 2024-09-16T04:10:00.278 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.268+0000 7f5435ffb640 10 monclient: _check_auth_tickets 2024-09-16T04:10:00.278 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.268+0000 7f5435ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:00.278 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.268+0000 7f543d9b1640 5 monclient: authenticate success, global_id 1014356 2024-09-16T04:10:00.278 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.268+0000 7f5435ffb640 1 -- 172.21.15.42:0/2969455917 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f54300a56d0 con 0x7f5438137540 2024-09-16T04:10:00.278 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.268+0000 7f5435ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:10:00.278 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.268+0000 7f543d9b1640 10 monclient: get_monmap_and_config success 2024-09-16T04:10:00.278 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.268+0000 7f543d9b1640 4 set_mon_vals no callback set 2024-09-16T04:10:00.278 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.268+0000 7f5435ffb640 1 -- 172.21.15.42:0/2969455917 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54300a5aa0 con 0x7f5438137540 2024-09-16T04:10:00.278 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.268+0000 7f5435ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:00.279 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.268+0000 7f5435ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:10:00.279 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.268+0000 7f5435ffb640 10 monclient: dump: 2024-09-16T04:10:00.279 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:10:00.279 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:10:00.279 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:00.279 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:00.279 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:00.279 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:10:00.279 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:10:00.279 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:10:00.279 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:10:00.279 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:10:00.279 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.268+0000 7f543d9b1640 10 monclient: shutdown 2024-09-16T04:10:00.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.268+0000 7f543d9b1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:10:00.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.268+0000 7f543d9b1640 1 -- 172.21.15.42:0/2969455917 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5438137540 msgr2=0x7f5438137990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:00.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.268+0000 7f543d9b1640 1 --2- 172.21.15.42:0/2969455917 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5438137540 0x7f5438137990 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f54380a7750 tx=0x7f543001c8b0 comp rx=0 tx=0).stop 2024-09-16T04:10:00.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.268+0000 7f543d9b1640 1 -- 172.21.15.42:0/2969455917 shutdown_connections 2024-09-16T04:10:00.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.268+0000 7f543d9b1640 1 --2- 172.21.15.42:0/2969455917 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5438137540 0x7f5438137990 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:10:00.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.268+0000 7f543d9b1640 1 --2- 172.21.15.42:0/2969455917 >> v2:172.21.15.42:3300/0 conn(0x7f54380a3d90 0x7f5438137000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:10:00.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.268+0000 7f543d9b1640 1 --2- 172.21.15.42:0/2969455917 >> v2:172.21.15.154:3300/0 conn(0x7f54380a39b0 0x7f5438133b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:10:00.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.268+0000 7f543d9b1640 1 -- 172.21.15.42:0/2969455917 >> 172.21.15.42:0/2969455917 conn(0x7f54380a9fb0 msgr2=0x7f54380aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:10:00.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.268+0000 7f543d9b1640 1 -- 172.21.15.42:0/2969455917 shutdown_connections 2024-09-16T04:10:00.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.268+0000 7f543d9b1640 1 -- 172.21.15.42:0/2969455917 wait complete. 2024-09-16T04:10:00.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.268+0000 7f543d9b1640 10 monclient: build_initial_monmap 2024-09-16T04:10:00.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.268+0000 7f543d9b1640 10 monclient: monmap: 2024-09-16T04:10:00.280 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:10:00.280 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:10:00.281 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:10:00.281 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:10:00.281 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:10:00.281 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:10:00.281 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:10:00.281 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:10:00.281 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:10:00.281 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:10:00.281 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.268+0000 7f543d9b1640 1 librados: starting msgr at 2024-09-16T04:10:00.281 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.268+0000 7f543d9b1640 1 librados: starting objecter 2024-09-16T04:10:00.281 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.268+0000 7f543d9b1640 1 Processor -- start 2024-09-16T04:10:00.281 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.268+0000 7f543d9b1640 1 -- start start 2024-09-16T04:10:00.281 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.268+0000 7f543d9b1640 1 librados: setting wanted keys 2024-09-16T04:10:00.281 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.268+0000 7f543d9b1640 1 librados: calling monclient init 2024-09-16T04:10:00.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.268+0000 7f543d9b1640 10 monclient: init 2024-09-16T04:10:00.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.268+0000 7f543d9b1640 10 monclient: _reopen_session rank -1 2024-09-16T04:10:00.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.268+0000 7f543d9b1640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:10:00.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.268+0000 7f543d9b1640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f54380a39b0 0x7f54381cf0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:10:00.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.268+0000 7f543d9b1640 10 monclient(hunting): picked mon.noname-c con 0x7f54380a39b0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:10:00.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.268+0000 7f543d9b1640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f54380a3d90 0x7f54380a2ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:10:00.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.268+0000 7f543d9b1640 10 monclient(hunting): picked mon.noname-a con 0x7f54380a3d90 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:10:00.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.268+0000 7f543d9b1640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5438137540 0x7f543813b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:10:00.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.268+0000 7f543d9b1640 10 monclient(hunting): picked mon.noname-b con 0x7f5438137540 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:10:00.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.268+0000 7f543d9b1640 10 monclient(hunting): start opening mon connection 2024-09-16T04:10:00.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.268+0000 7f543d9b1640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f5438140dc0 con 0x7f54380a3d90 2024-09-16T04:10:00.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.268+0000 7f543d9b1640 10 monclient(hunting): start opening mon connection 2024-09-16T04:10:00.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.268+0000 7f543d9b1640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f5438140f40 con 0x7f5438137540 2024-09-16T04:10:00.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.268+0000 7f543d9b1640 10 monclient(hunting): start opening mon connection 2024-09-16T04:10:00.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.268+0000 7f543d9b1640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f54381cffe0 con 0x7f54380a39b0 2024-09-16T04:10:00.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.268+0000 7f543d9b1640 10 monclient(hunting): _renew_subs 2024-09-16T04:10:00.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.268+0000 7f543d9b1640 10 monclient(hunting): authenticate will time out at 2647.238259s 2024-09-16T04:10:00.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.268+0000 7f54367fc640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f54380a3d90 0x7f54380a2ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:10:00.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.268+0000 7f5436ffd640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f54380a39b0 0x7f54381cf0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:10:00.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.268+0000 7f54367fc640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f54380a3d90 0x7f54380a2ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:60074/0 (socket says 172.21.15.42:60074) 2024-09-16T04:10:00.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.268+0000 7f54367fc640 1 -- 172.21.15.42:0/3429690945 learned_addr learned my addr 172.21.15.42:0/3429690945 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:10:00.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.268+0000 7f54367fc640 10 monclient(hunting): get_auth_request con 0x7f54380a3d90 auth_method 0 2024-09-16T04:10:00.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.268+0000 7f54367fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:10:00.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.268+0000 7f54367fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:00.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.268+0000 7f54367fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:00.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.268+0000 7f5436ffd640 10 monclient(hunting): get_auth_request con 0x7f54380a39b0 auth_method 0 2024-09-16T04:10:00.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.268+0000 7f5436ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:10:00.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.268+0000 7f5436ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:00.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.268+0000 7f5436ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:00.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.268+0000 7f54367fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:10:00.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.268+0000 7f54367fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:10:00.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.268+0000 7f54367fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:10:00.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.268+0000 7f5436ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:10:00.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.268+0000 7f5436ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:10:00.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.268+0000 7f5436ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:10:00.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.268+0000 7f54377fe640 1 --2- 172.21.15.42:0/3429690945 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5438137540 0x7f543813b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:10:00.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.268+0000 7f54377fe640 10 monclient(hunting): get_auth_request con 0x7f5438137540 auth_method 0 2024-09-16T04:10:00.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.268+0000 7f54377fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:10:00.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.268+0000 7f54377fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:00.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.268+0000 7f54377fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:00.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.268+0000 7f54367fc640 10 monclient(hunting): handle_auth_done global_id 1034223 payload 824 2024-09-16T04:10:00.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.268+0000 7f54367fc640 1 -- 172.21.15.42:0/3429690945 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f54380a39b0 msgr2=0x7f54381cf0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:00.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.268+0000 7f54367fc640 1 --2- 172.21.15.42:0/3429690945 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f54380a39b0 0x7f54381cf0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:10:00.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.268+0000 7f54367fc640 1 -- 172.21.15.42:0/3429690945 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5438137540 msgr2=0x7f543813b220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:00.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.268+0000 7f54367fc640 1 --2- 172.21.15.42:0/3429690945 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5438137540 0x7f543813b220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:10:00.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.268+0000 7f54367fc640 10 monclient: _finish_hunting 0 2024-09-16T04:10:00.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.268+0000 7f54367fc640 1 monclient: found mon.noname-a 2024-09-16T04:10:00.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.268+0000 7f54367fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:10:00.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.268+0000 7f54367fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:10:00.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.268+0000 7f54367fc640 1 -- 172.21.15.42:0/3429690945 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f543809d490 con 0x7f54380a3d90 2024-09-16T04:10:00.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.268+0000 7f54367fc640 1 --2- 172.21.15.42:0/3429690945 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f54380a3d90 0x7f54380a2ce0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f542c00e930 tx=0x7f542c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:10:00.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.268+0000 7f541ffff640 10 client.?.objecter ms_handle_connect 0x7f54380a3d90 2024-09-16T04:10:00.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.268+0000 7f541ffff640 10 client.?.objecter resend_mon_ops 2024-09-16T04:10:00.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.268+0000 7f541ffff640 1 -- 172.21.15.42:0/3429690945 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f542c017070 con 0x7f54380a3d90 2024-09-16T04:10:00.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.268+0000 7f541ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:00.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.268+0000 7f541ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:10:00.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.268+0000 7f541ffff640 10 monclient: dump: 2024-09-16T04:10:00.288 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:10:00.288 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:10:00.288 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:00.288 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:00.288 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:00.289 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:10:00.289 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:10:00.289 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:10:00.289 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:10:00.289 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:10:00.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.268+0000 7f541ffff640 10 monclient: _finish_auth 0 2024-09-16T04:10:00.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.268+0000 7f541ffff640 10 monclient: _check_auth_tickets 2024-09-16T04:10:00.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.268+0000 7f541ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:00.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.268+0000 7f541ffff640 1 -- 172.21.15.42:0/3429690945 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f542c00c9a0 con 0x7f54380a3d90 2024-09-16T04:10:00.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.268+0000 7f541ffff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:10:00.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.268+0000 7f541ffff640 1 -- 172.21.15.42:0/3429690945 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f542c00cd50 con 0x7f54380a3d90 2024-09-16T04:10:00.291 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.268+0000 7f541ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:00.291 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.268+0000 7f541ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:10:00.291 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.268+0000 7f541ffff640 10 monclient: dump: 2024-09-16T04:10:00.291 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:10:00.291 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:10:00.291 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:00.292 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:00.292 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:00.292 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:10:00.292 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:10:00.292 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:10:00.293 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:10:00.293 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:10:00.293 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.268+0000 7f54357fa640 4 set_mon_vals no callback set 2024-09-16T04:10:00.293 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.268+0000 7f543d9b1640 5 monclient: authenticate success, global_id 1034223 2024-09-16T04:10:00.293 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.268+0000 7f543d9b1640 10 monclient: _renew_subs 2024-09-16T04:10:00.293 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.268+0000 7f543d9b1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:10:00.293 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.268+0000 7f543d9b1640 1 -- 172.21.15.42:0/3429690945 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54381cf610 con 0x7f54380a3d90 2024-09-16T04:10:00.294 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.268+0000 7f543d9b1640 10 client.1034223.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:10:00.294 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.268+0000 7f543d9b1640 10 monclient: _renew_subs 2024-09-16T04:10:00.294 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.268+0000 7f543d9b1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:10:00.294 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.268+0000 7f543d9b1640 1 -- 172.21.15.42:0/3429690945 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f543809d1b0 con 0x7f54380a3d90 2024-09-16T04:10:00.294 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.268+0000 7f543d9b1640 1 librados: init done 2024-09-16T04:10:00.295 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.272+0000 7f543d9b1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:00.295 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.272+0000 7f543d9b1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:00.295 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.272+0000 7f543d9b1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:10:00.295 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.272+0000 7f541ffff640 1 -- 172.21.15.42:0/3429690945 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f542c01a070 con 0x7f54380a3d90 2024-09-16T04:10:00.295 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.272+0000 7f543d9b1640 1 -- 172.21.15.42:0/3429690945 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5438137000 con 0x7f54380a3d90 2024-09-16T04:10:00.295 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.272+0000 7f541ffff640 1 --2- 172.21.15.42:0/3429690945 >> v2:172.21.15.154:6812/254845342 conn(0x7f5424037810 0x7f5424039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:10:00.295 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.272+0000 7f541ffff640 1 -- 172.21.15.42:0/3429690945 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(706..706 src has 451..706) ==== 5040+0+0 (secure 0 0 0) 0x7f542c048e00 con 0x7f54380a3d90 2024-09-16T04:10:00.296 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.272+0000 7f541ffff640 10 client.1034223.objecter ms_dispatch 0x7f5438000ba0 osd_map(706..706 src has 451..706) 2024-09-16T04:10:00.296 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.272+0000 7f541ffff640 3 client.1034223.objecter handle_osd_map got epochs [706,706] > 0 2024-09-16T04:10:00.296 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.272+0000 7f541ffff640 3 client.1034223.objecter handle_osd_map decoding full epoch 706 2024-09-16T04:10:00.296 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.272+0000 7f541ffff640 20 client.1034223.objecter dump_active .. 0 homeless 2024-09-16T04:10:00.296 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.272+0000 7f5436ffd640 1 --2- 172.21.15.42:0/3429690945 >> v2:172.21.15.154:6812/254845342 conn(0x7f5424037810 0x7f5424039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:10:00.296 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.272+0000 7f5436ffd640 10 monclient: get_auth_request con 0x7f5424037810 auth_method 0 2024-09-16T04:10:00.296 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.272+0000 7f5436ffd640 1 --2- 172.21.15.42:0/3429690945 >> v2:172.21.15.154:6812/254845342 conn(0x7f5424037810 0x7f5424039cd0 secure :-1 s=READY pgs=3920 cs=0 l=1 rev1=1 crypto rx=0x7f5428007a60 tx=0x7f5428008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:10:00.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.272+0000 7f541ffff640 10 client.1034223.objecter ms_handle_connect 0x7f5424037810 2024-09-16T04:10:00.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.276+0000 7f541ffff640 1 -- 172.21.15.42:0/3429690945 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f542c011d50 con 0x7f54380a3d90 2024-09-16T04:10:00.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.276+0000 7f541ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:00.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.276+0000 7f541ffff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:10:00.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.504+0000 7f543d9b1640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T04:10:00.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.504+0000 7f543d9b1640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T04:10:00.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.504+0000 7f543d9b1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:10:00.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.504+0000 7f543d9b1640 1 -- 172.21.15.42:0/3429690945 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f54380a5750 con 0x7f54380a3d90 2024-09-16T04:10:00.508 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.504+0000 7f541ffff640 1 -- 172.21.15.42:0/3429690945 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f542c013470 con 0x7f54380a3d90 2024-09-16T04:10:00.509 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.504+0000 7f541ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T04:10:00.509 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.504+0000 7f541ffff640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:10:00.510 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.504+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.510 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.504+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=27,config=2,health=12,kv=10,logm=49} crc {auth=2329992260,config=692619928,health=4078287400,kv=3446188498,logm=118358068}) 2024-09-16T04:10:00.511 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.504+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=27,config=2,health=12,kv=9,logm=50} crc {auth=2329992260,config=692619927,health=4078287401,kv=542242125,logm=497405285}) 2024-09-16T04:10:00.511 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.504+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.511 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.504+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=27,config=2,health=12,kv=10,logm=49} crc {auth=2329992260,config=692619928,health=4078287400,kv=3446188498,logm=118358068}) 2024-09-16T04:10:00.511 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.504+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=26,config=2,health=12,kv=8,logm=52} crc {auth=1617855257,config=692619927,health=4078287400,kv=3915008329,logm=3644998417}) 2024-09-16T04:10:00.514 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.508+0000 7f543d9b1640 10 librados: watch_flush enter 2024-09-16T04:10:00.514 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.508+0000 7f543d9b1640 10 librados: watch_flush exit 2024-09-16T04:10:00.514 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.508+0000 7f543d9b1640 20 client.1034223.objecter shutdown clearing up homeless session... 2024-09-16T04:10:00.514 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.508+0000 7f543d9b1640 10 client.1034223.objecter successfully canceled tick 2024-09-16T04:10:00.514 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.508+0000 7f543d9b1640 1 -- 172.21.15.42:0/3429690945 >> v2:172.21.15.154:6812/254845342 conn(0x7f5424037810 msgr2=0x7f5424039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:00.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.508+0000 7f543d9b1640 1 --2- 172.21.15.42:0/3429690945 >> v2:172.21.15.154:6812/254845342 conn(0x7f5424037810 0x7f5424039cd0 secure :-1 s=READY pgs=3920 cs=0 l=1 rev1=1 crypto rx=0x7f5428007a60 tx=0x7f5428008040 comp rx=0 tx=0).stop 2024-09-16T04:10:00.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.508+0000 7f543d9b1640 10 monclient: shutdown 2024-09-16T04:10:00.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.508+0000 7f543d9b1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:10:00.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.508+0000 7f543d9b1640 1 -- 172.21.15.42:0/3429690945 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f54380a3d90 msgr2=0x7f54380a2ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:00.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.508+0000 7f543d9b1640 1 --2- 172.21.15.42:0/3429690945 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f54380a3d90 0x7f54380a2ce0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f542c00e930 tx=0x7f542c00edf0 comp rx=0 tx=0).stop 2024-09-16T04:10:00.516 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.508+0000 7f543d9b1640 1 -- 172.21.15.42:0/3429690945 shutdown_connections 2024-09-16T04:10:00.516 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.508+0000 7f543d9b1640 1 --2- 172.21.15.42:0/3429690945 >> v2:172.21.15.154:6812/254845342 conn(0x7f5424037810 0x7f5424039cd0 unknown :-1 s=CLOSED pgs=3920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:10:00.516 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.508+0000 7f543d9b1640 1 --2- 172.21.15.42:0/3429690945 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5438137540 0x7f543813b220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:10:00.516 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.508+0000 7f543d9b1640 1 --2- 172.21.15.42:0/3429690945 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f54380a3d90 0x7f54380a2ce0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:10:00.516 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.508+0000 7f543d9b1640 1 --2- 172.21.15.42:0/3429690945 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f54380a39b0 0x7f54381cf0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:10:00.516 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.508+0000 7f543d9b1640 1 -- 172.21.15.42:0/3429690945 >> 172.21.15.42:0/3429690945 conn(0x7f54380a9fb0 msgr2=0x7f54380a7fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:10:00.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.508+0000 7f543d9b1640 1 -- 172.21.15.42:0/3429690945 shutdown_connections 2024-09-16T04:10:00.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.508+0000 7f543d9b1640 1 -- 172.21.15.42:0/3429690945 wait complete. 2024-09-16T04:10:00.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:00.508+0000 7f543d9b1640 1 librados: shutdown 2024-09-16T04:10:00.519 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.508+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.520 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.508+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2475488227}) 2024-09-16T04:10:00.520 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.508+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=139959090}) 2024-09-16T04:10:00.520 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.508+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.520 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.508+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2475488227}) 2024-09-16T04:10:00.520 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.508+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=748231698}) 2024-09-16T04:10:00.521 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.512+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.521 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.512+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=742511639}) 2024-09-16T04:10:00.521 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.512+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4089847330}) 2024-09-16T04:10:00.521 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.512+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.521 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.512+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=742511639}) 2024-09-16T04:10:00.521 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.512+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3239345313}) 2024-09-16T04:10:00.521 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.516+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.521 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.516+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3288794223}) 2024-09-16T04:10:00.522 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.516+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3288025767}) 2024-09-16T04:10:00.522 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.516+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.522 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.516+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3288794223}) 2024-09-16T04:10:00.522 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.516+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3905152628}) 2024-09-16T04:10:00.524 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.516+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.524 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.516+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1589464930}) 2024-09-16T04:10:00.525 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.516+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=364211728}) 2024-09-16T04:10:00.525 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.516+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.525 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.516+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1589464930}) 2024-09-16T04:10:00.525 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.516+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1059332407}) 2024-09-16T04:10:00.525 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.520+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.525 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.520+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3314277118}) 2024-09-16T04:10:00.526 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.520+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3531520849}) 2024-09-16T04:10:00.526 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.520+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.526 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.520+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3314277118}) 2024-09-16T04:10:00.526 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.520+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=429872513}) 2024-09-16T04:10:00.529 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.520+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.530 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.520+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2447460727}) 2024-09-16T04:10:00.530 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.520+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1031797383}) 2024-09-16T04:10:00.530 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.520+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.530 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.520+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2447460727}) 2024-09-16T04:10:00.530 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.520+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3394645784}) 2024-09-16T04:10:00.531 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.524+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.531 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.524+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1912193133}) 2024-09-16T04:10:00.531 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.524+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=553931851}) 2024-09-16T04:10:00.531 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.524+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.531 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.524+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1912193133}) 2024-09-16T04:10:00.531 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.524+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3181523574}) 2024-09-16T04:10:00.532 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.524+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.532 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.524+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4088266824}) 2024-09-16T04:10:00.532 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.524+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2042792329}) 2024-09-16T04:10:00.532 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.524+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.532 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.524+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4088266824}) 2024-09-16T04:10:00.533 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.524+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3785885722}) 2024-09-16T04:10:00.534 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T04:10:00.539 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.528+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.539 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.528+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=957627005}) 2024-09-16T04:10:00.539 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.528+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2812734861}) 2024-09-16T04:10:00.539 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.528+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.539 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.528+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=957627005}) 2024-09-16T04:10:00.539 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.528+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1670546172}) 2024-09-16T04:10:00.540 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.528+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.540 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.528+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=442584733}) 2024-09-16T04:10:00.540 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.528+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=960846581}) 2024-09-16T04:10:00.540 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.528+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.540 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.528+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=442584733}) 2024-09-16T04:10:00.540 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.528+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2246586372}) 2024-09-16T04:10:00.541 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.532+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.541 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.532+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=303281165}) 2024-09-16T04:10:00.541 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.532+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=814461104}) 2024-09-16T04:10:00.541 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.532+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.541 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.532+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=303281165}) 2024-09-16T04:10:00.542 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.532+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4210230969}) 2024-09-16T04:10:00.542 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.532+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.542 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.532+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=62578928}) 2024-09-16T04:10:00.542 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.532+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=625714954}) 2024-09-16T04:10:00.542 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.532+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.542 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.532+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=62578928}) 2024-09-16T04:10:00.543 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.532+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2188590597}) 2024-09-16T04:10:00.543 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.536+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.543 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.536+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2468827903}) 2024-09-16T04:10:00.543 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.536+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1021315356}) 2024-09-16T04:10:00.543 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.536+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.544 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.536+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2468827903}) 2024-09-16T04:10:00.544 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.536+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=663388188}) 2024-09-16T04:10:00.549 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.536+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.550 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.536+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1524242470}) 2024-09-16T04:10:00.550 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.536+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4155490558}) 2024-09-16T04:10:00.550 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.536+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.550 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.536+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1524242470}) 2024-09-16T04:10:00.550 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.536+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2444381332}) 2024-09-16T04:10:00.551 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.540+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.551 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.540+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2346714833}) 2024-09-16T04:10:00.551 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.540+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=794520965}) 2024-09-16T04:10:00.551 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.540+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.552 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.540+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2346714833}) 2024-09-16T04:10:00.552 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.540+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4041349034}) 2024-09-16T04:10:00.552 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.540+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.552 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.540+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1108721876}) 2024-09-16T04:10:00.552 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.540+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3723714988}) 2024-09-16T04:10:00.552 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.540+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.553 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.540+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1108721876}) 2024-09-16T04:10:00.553 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.540+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1657907584}) 2024-09-16T04:10:00.553 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.544+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.553 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.544+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3667110946}) 2024-09-16T04:10:00.553 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.544+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3694061431}) 2024-09-16T04:10:00.554 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.544+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.554 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.544+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3667110946}) 2024-09-16T04:10:00.554 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.544+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2482282568}) 2024-09-16T04:10:00.554 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.544+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.554 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.544+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4271398340}) 2024-09-16T04:10:00.554 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.544+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=988976753}) 2024-09-16T04:10:00.555 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.544+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.555 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.544+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4271398340}) 2024-09-16T04:10:00.555 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.544+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=818726456}) 2024-09-16T04:10:00.562 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.548+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.562 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.548+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4010115967}) 2024-09-16T04:10:00.562 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.548+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2304796561}) 2024-09-16T04:10:00.562 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.548+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.563 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.548+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4010115967}) 2024-09-16T04:10:00.563 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.548+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=725685873}) 2024-09-16T04:10:00.563 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.548+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.563 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.548+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1784440489}) 2024-09-16T04:10:00.563 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.548+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3348936833}) 2024-09-16T04:10:00.563 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.548+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.564 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.548+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1784440489}) 2024-09-16T04:10:00.564 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.548+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=830651472}) 2024-09-16T04:10:00.564 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.552+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.564 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.552+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1013415552}) 2024-09-16T04:10:00.564 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.552+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3407469308}) 2024-09-16T04:10:00.565 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.552+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.565 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.552+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1013415552}) 2024-09-16T04:10:00.565 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.552+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2236900680}) 2024-09-16T04:10:00.565 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.552+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.565 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.552+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1846433821}) 2024-09-16T04:10:00.565 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.552+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2986400333}) 2024-09-16T04:10:00.566 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.552+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.566 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.552+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1846433821}) 2024-09-16T04:10:00.566 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.552+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3433003384}) 2024-09-16T04:10:00.566 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.556+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.566 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.556+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2330104184}) 2024-09-16T04:10:00.566 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.556+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1274007830}) 2024-09-16T04:10:00.567 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.556+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.567 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.556+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2330104184}) 2024-09-16T04:10:00.567 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.556+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3908231405}) 2024-09-16T04:10:00.567 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.556+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.567 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.556+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3346303054}) 2024-09-16T04:10:00.567 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.560+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=321175482}) 2024-09-16T04:10:00.568 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.560+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.568 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.560+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3346303054}) 2024-09-16T04:10:00.568 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.560+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4054928370}) 2024-09-16T04:10:00.575 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.560+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.575 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.560+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3444082604}) 2024-09-16T04:10:00.576 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.560+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=247096272}) 2024-09-16T04:10:00.576 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.560+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.576 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.560+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3444082604}) 2024-09-16T04:10:00.576 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.560+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4223416741}) 2024-09-16T04:10:00.576 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.564+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.577 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.564+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=895954046}) 2024-09-16T04:10:00.577 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.564+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3989492917}) 2024-09-16T04:10:00.577 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.564+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.577 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.564+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=895954046}) 2024-09-16T04:10:00.577 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.564+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2791823840}) 2024-09-16T04:10:00.577 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.564+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.578 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.564+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2476803797}) 2024-09-16T04:10:00.578 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.564+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2403672967}) 2024-09-16T04:10:00.578 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.564+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.578 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.564+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2476803797}) 2024-09-16T04:10:00.578 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.564+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1259728945}) 2024-09-16T04:10:00.579 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.568+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.579 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.568+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1309876156}) 2024-09-16T04:10:00.579 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.568+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=845442340}) 2024-09-16T04:10:00.579 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.568+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.579 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.568+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1309876156}) 2024-09-16T04:10:00.579 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.568+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1447703607}) 2024-09-16T04:10:00.580 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.568+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.580 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.568+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=94114772}) 2024-09-16T04:10:00.580 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.568+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1545776794}) 2024-09-16T04:10:00.580 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.568+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.580 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.568+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=94114772}) 2024-09-16T04:10:00.580 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.568+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4070450795}) 2024-09-16T04:10:00.581 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.568+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.581 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.572+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1951473554}) 2024-09-16T04:10:00.581 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.572+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2306379679}) 2024-09-16T04:10:00.581 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.572+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.581 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.572+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1951473554}) 2024-09-16T04:10:00.582 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.572+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2040987567}) 2024-09-16T04:10:00.582 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.572+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.582 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.572+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1171860042}) 2024-09-16T04:10:00.582 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.572+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1766882757}) 2024-09-16T04:10:00.582 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.572+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.582 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.572+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1171860042}) 2024-09-16T04:10:00.583 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.572+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3951067239}) 2024-09-16T04:10:00.588 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.576+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.588 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.576+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3677982870}) 2024-09-16T04:10:00.588 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.576+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3845790861}) 2024-09-16T04:10:00.588 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.576+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.589 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.576+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3677982870}) 2024-09-16T04:10:00.589 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.576+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=208592316}) 2024-09-16T04:10:00.589 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.576+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.589 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.576+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=332729475}) 2024-09-16T04:10:00.589 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.576+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=349213788}) 2024-09-16T04:10:00.590 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.576+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.590 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.576+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=332729475}) 2024-09-16T04:10:00.590 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.576+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2364361596}) 2024-09-16T04:10:00.590 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.580+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.590 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.580+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1562561162}) 2024-09-16T04:10:00.590 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.580+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2274259058}) 2024-09-16T04:10:00.591 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.580+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.591 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.580+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1562561162}) 2024-09-16T04:10:00.591 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.580+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1059454595}) 2024-09-16T04:10:00.591 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.580+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.591 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.580+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2625779892}) 2024-09-16T04:10:00.591 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.580+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=36073133}) 2024-09-16T04:10:00.592 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.580+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.592 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.580+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2625779892}) 2024-09-16T04:10:00.592 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.580+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3563159295}) 2024-09-16T04:10:00.592 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.580+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.592 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.580+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2653518815}) 2024-09-16T04:10:00.592 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.580+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1214847615}) 2024-09-16T04:10:00.593 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.584+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.593 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.584+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2653518815}) 2024-09-16T04:10:00.593 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.584+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3767587170}) 2024-09-16T04:10:00.593 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.584+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.593 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.584+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=364937094}) 2024-09-16T04:10:00.593 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.584+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2043382178}) 2024-09-16T04:10:00.594 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.584+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.594 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.584+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=364937094}) 2024-09-16T04:10:00.594 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.584+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=645679659}) 2024-09-16T04:10:00.599 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.584+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.599 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.588+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2739931740}) 2024-09-16T04:10:00.599 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.588+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2617145828}) 2024-09-16T04:10:00.599 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.588+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.599 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.588+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2739931740}) 2024-09-16T04:10:00.599 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.588+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3695433566}) 2024-09-16T04:10:00.599 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.588+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.600 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.588+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1055513525}) 2024-09-16T04:10:00.600 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.588+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3441318374}) 2024-09-16T04:10:00.600 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.588+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.600 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.588+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1055513525}) 2024-09-16T04:10:00.600 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.588+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2358523677}) 2024-09-16T04:10:00.600 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.588+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.601 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.588+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1341893221}) 2024-09-16T04:10:00.601 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.588+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3173033008}) 2024-09-16T04:10:00.601 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.588+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.601 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.588+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1341893221}) 2024-09-16T04:10:00.601 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.588+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=6919541}) 2024-09-16T04:10:00.601 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.592+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.601 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.592+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2210633961}) 2024-09-16T04:10:00.602 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.592+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2802541485}) 2024-09-16T04:10:00.602 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.592+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.602 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.592+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2210633961}) 2024-09-16T04:10:00.602 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.592+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=518166539}) 2024-09-16T04:10:00.602 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.592+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.602 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.596+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4069706405}) 2024-09-16T04:10:00.602 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.596+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2181443483}) 2024-09-16T04:10:00.602 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.596+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.602 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.596+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4069706405}) 2024-09-16T04:10:00.603 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.596+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2577796598}) 2024-09-16T04:10:00.604 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.596+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.604 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.596+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2868546322}) 2024-09-16T04:10:00.604 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.596+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2874569109}) 2024-09-16T04:10:00.604 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.596+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.604 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.596+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2868546322}) 2024-09-16T04:10:00.605 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.596+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=686529597}) 2024-09-16T04:10:00.605 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.600+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.605 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.600+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4035849395}) 2024-09-16T04:10:00.605 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.600+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1005151017}) 2024-09-16T04:10:00.605 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.600+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.605 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.600+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4035849395}) 2024-09-16T04:10:00.605 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.600+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2798551078}) 2024-09-16T04:10:00.605 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.600+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.605 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.600+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=907585364}) 2024-09-16T04:10:00.605 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.600+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3555785231}) 2024-09-16T04:10:00.605 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.600+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.605 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.600+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=907585364}) 2024-09-16T04:10:00.605 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.600+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3797797381}) 2024-09-16T04:10:00.606 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.604+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.606 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.604+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1027749689}) 2024-09-16T04:10:00.606 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.604+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3315622499}) 2024-09-16T04:10:00.606 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.604+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.606 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.604+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1027749689}) 2024-09-16T04:10:00.606 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.604+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4212566033}) 2024-09-16T04:10:00.607 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.604+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.607 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.604+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1109212331}) 2024-09-16T04:10:00.607 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.604+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3398057748}) 2024-09-16T04:10:00.607 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.604+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.608 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.604+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1109212331}) 2024-09-16T04:10:00.608 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.604+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2339122213}) 2024-09-16T04:10:00.609 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.608+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.609 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.608+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=305315666}) 2024-09-16T04:10:00.609 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.608+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3531923619}) 2024-09-16T04:10:00.609 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.608+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.609 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.608+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=305315666}) 2024-09-16T04:10:00.610 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.608+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=121362696}) 2024-09-16T04:10:00.611 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.608+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.611 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.608+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3444119857}) 2024-09-16T04:10:00.611 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.608+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4201277726}) 2024-09-16T04:10:00.611 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.608+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.612 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.608+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3444119857}) 2024-09-16T04:10:00.612 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.608+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3232993184}) 2024-09-16T04:10:00.613 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.612+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.613 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.612+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1577344666}) 2024-09-16T04:10:00.613 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.612+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2215376489}) 2024-09-16T04:10:00.613 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.612+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.613 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.612+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1577344666}) 2024-09-16T04:10:00.613 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.612+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1174599311}) 2024-09-16T04:10:00.615 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.612+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.616 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.612+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=947613850}) 2024-09-16T04:10:00.616 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.612+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1418777652}) 2024-09-16T04:10:00.616 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.612+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.616 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.612+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=947613850}) 2024-09-16T04:10:00.616 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.612+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2991603531}) 2024-09-16T04:10:00.617 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.616+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.618 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.616+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2789261705}) 2024-09-16T04:10:00.618 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.616+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=531311802}) 2024-09-16T04:10:00.618 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.616+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.618 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.616+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2789261705}) 2024-09-16T04:10:00.618 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.616+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=479995409}) 2024-09-16T04:10:00.620 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.616+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.620 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.616+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4064618364}) 2024-09-16T04:10:00.620 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.616+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1522838055}) 2024-09-16T04:10:00.620 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.616+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.620 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.616+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4064618364}) 2024-09-16T04:10:00.621 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.616+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4205055376}) 2024-09-16T04:10:00.625 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.620+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.625 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.620+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1666280565}) 2024-09-16T04:10:00.625 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.620+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=680942856}) 2024-09-16T04:10:00.626 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.620+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.626 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.620+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1666280565}) 2024-09-16T04:10:00.626 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.620+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2461299455}) 2024-09-16T04:10:00.626 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.620+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.626 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.620+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1177431754}) 2024-09-16T04:10:00.626 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.620+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=497790081}) 2024-09-16T04:10:00.627 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.620+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.627 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.620+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1177431754}) 2024-09-16T04:10:00.627 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.620+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3023692562}) 2024-09-16T04:10:00.629 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.624+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.629 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.624+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2335311326}) 2024-09-16T04:10:00.629 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.624+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1293144126}) 2024-09-16T04:10:00.629 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.624+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.630 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.624+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2335311326}) 2024-09-16T04:10:00.630 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.624+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1988326775}) 2024-09-16T04:10:00.630 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.624+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.630 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.624+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2825791478}) 2024-09-16T04:10:00.630 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.624+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1371366342}) 2024-09-16T04:10:00.630 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.624+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.631 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.624+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2825791478}) 2024-09-16T04:10:00.631 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.624+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1098353417}) 2024-09-16T04:10:00.632 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.628+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.632 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.628+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3979956411}) 2024-09-16T04:10:00.632 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.628+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1645292504}) 2024-09-16T04:10:00.632 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.628+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.633 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.628+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3979956411}) 2024-09-16T04:10:00.633 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.628+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4190123037}) 2024-09-16T04:10:00.634 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.628+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.634 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.628+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=885355834}) 2024-09-16T04:10:00.634 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.628+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3772030436}) 2024-09-16T04:10:00.634 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.628+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.634 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.628+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=885355834}) 2024-09-16T04:10:00.635 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.628+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=658087558}) 2024-09-16T04:10:00.636 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.632+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.636 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.632+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=458933422}) 2024-09-16T04:10:00.636 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.632+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3142688418}) 2024-09-16T04:10:00.636 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.632+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.636 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.632+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=458933422}) 2024-09-16T04:10:00.637 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.632+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1734867433}) 2024-09-16T04:10:00.638 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.632+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.638 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.632+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3582680131}) 2024-09-16T04:10:00.638 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.632+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=188276468}) 2024-09-16T04:10:00.638 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.632+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.638 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.632+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3582680131}) 2024-09-16T04:10:00.638 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.632+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4152030546}) 2024-09-16T04:10:00.639 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.636+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.640 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.636+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2978218833}) 2024-09-16T04:10:00.640 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.636+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2367854319}) 2024-09-16T04:10:00.640 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.636+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.640 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.636+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2978218833}) 2024-09-16T04:10:00.640 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.636+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4240457629}) 2024-09-16T04:10:00.642 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.640+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.643 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.640+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1409758859}) 2024-09-16T04:10:00.643 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.640+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=811030248}) 2024-09-16T04:10:00.643 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.640+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.643 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.640+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1409758859}) 2024-09-16T04:10:00.643 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.640+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3362257722}) 2024-09-16T04:10:00.645 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.640+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.645 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.640+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=20660748}) 2024-09-16T04:10:00.645 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.640+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1690378451}) 2024-09-16T04:10:00.645 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.640+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.645 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.640+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=20660748}) 2024-09-16T04:10:00.645 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.640+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4015063102}) 2024-09-16T04:10:00.647 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.644+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.648 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.644+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1999618051}) 2024-09-16T04:10:00.648 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.644+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3008039847}) 2024-09-16T04:10:00.648 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.644+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.648 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.644+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1999618051}) 2024-09-16T04:10:00.649 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.644+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=994889261}) 2024-09-16T04:10:00.649 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.648+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.649 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.648+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3106365535}) 2024-09-16T04:10:00.649 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.648+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4009418017}) 2024-09-16T04:10:00.649 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.648+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.649 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.648+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3106365535}) 2024-09-16T04:10:00.650 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.648+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1376149951}) 2024-09-16T04:10:00.651 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.648+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.651 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.648+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4063198885}) 2024-09-16T04:10:00.651 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.648+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2083858920}) 2024-09-16T04:10:00.652 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.648+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.652 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.648+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4063198885}) 2024-09-16T04:10:00.652 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.648+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1328943435}) 2024-09-16T04:10:00.653 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.652+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.654 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.652+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3085051479}) 2024-09-16T04:10:00.654 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.652+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3481566954}) 2024-09-16T04:10:00.654 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.652+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.654 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.652+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3085051479}) 2024-09-16T04:10:00.654 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.652+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2803723156}) 2024-09-16T04:10:00.656 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.652+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.656 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.652+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1037292434}) 2024-09-16T04:10:00.656 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.652+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3509461499}) 2024-09-16T04:10:00.656 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.652+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.656 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.652+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1037292434}) 2024-09-16T04:10:00.656 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.652+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2275516875}) 2024-09-16T04:10:00.658 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.656+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.658 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.656+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1732370148}) 2024-09-16T04:10:00.658 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.656+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2950257052}) 2024-09-16T04:10:00.659 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.656+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.659 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.656+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1732370148}) 2024-09-16T04:10:00.659 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.656+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3658961890}) 2024-09-16T04:10:00.661 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.656+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.661 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.656+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1264766365}) 2024-09-16T04:10:00.661 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.656+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2901644875}) 2024-09-16T04:10:00.661 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.656+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.661 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.656+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1264766365}) 2024-09-16T04:10:00.661 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.656+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2844049301}) 2024-09-16T04:10:00.663 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.660+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.663 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.660+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3866367168}) 2024-09-16T04:10:00.663 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.660+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2682196991}) 2024-09-16T04:10:00.663 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.660+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.663 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.660+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3866367168}) 2024-09-16T04:10:00.663 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.660+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2375213790}) 2024-09-16T04:10:00.665 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.660+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.665 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.660+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2323387364}) 2024-09-16T04:10:00.665 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.660+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=637893688}) 2024-09-16T04:10:00.665 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.660+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.665 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.660+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2323387364}) 2024-09-16T04:10:00.665 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.660+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3001355206}) 2024-09-16T04:10:00.667 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.664+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.667 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.664+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=505135566}) 2024-09-16T04:10:00.667 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.664+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1251777581}) 2024-09-16T04:10:00.668 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.664+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.668 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.664+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=505135566}) 2024-09-16T04:10:00.668 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.664+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=507914976}) 2024-09-16T04:10:00.670 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.668+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.670 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.668+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=272719281}) 2024-09-16T04:10:00.670 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.668+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3860351235}) 2024-09-16T04:10:00.670 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.668+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.670 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.668+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=272719281}) 2024-09-16T04:10:00.670 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.668+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2952510627}) 2024-09-16T04:10:00.672 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.668+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.672 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.668+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=339882826}) 2024-09-16T04:10:00.672 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.668+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=961831304}) 2024-09-16T04:10:00.672 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.668+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.672 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.668+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=339882826}) 2024-09-16T04:10:00.672 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.668+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1985033205}) 2024-09-16T04:10:00.674 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.672+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.674 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.672+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3832642384}) 2024-09-16T04:10:00.674 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.672+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2316631171}) 2024-09-16T04:10:00.674 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.672+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.674 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.672+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3832642384}) 2024-09-16T04:10:00.675 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.672+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=794180313}) 2024-09-16T04:10:00.676 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.672+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.676 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.672+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3264766060}) 2024-09-16T04:10:00.676 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.672+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1457371802}) 2024-09-16T04:10:00.677 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.672+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.677 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.672+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3264766060}) 2024-09-16T04:10:00.677 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.672+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=36735949}) 2024-09-16T04:10:00.679 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.676+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.679 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.676+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3373430628}) 2024-09-16T04:10:00.679 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.676+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4190853117}) 2024-09-16T04:10:00.679 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.676+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.679 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.676+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3373430628}) 2024-09-16T04:10:00.679 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.676+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2163723407}) 2024-09-16T04:10:00.681 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.676+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.681 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.676+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2080304960}) 2024-09-16T04:10:00.681 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.676+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=595701021}) 2024-09-16T04:10:00.681 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.676+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.681 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.676+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2080304960}) 2024-09-16T04:10:00.681 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.676+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2035806248}) 2024-09-16T04:10:00.683 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.680+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.683 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.680+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4031032350}) 2024-09-16T04:10:00.683 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.680+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2063067811}) 2024-09-16T04:10:00.683 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.680+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.683 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.680+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4031032350}) 2024-09-16T04:10:00.683 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.680+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=706601196}) 2024-09-16T04:10:00.685 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.684+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.685 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.684+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1196957166}) 2024-09-16T04:10:00.685 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.684+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=191151273}) 2024-09-16T04:10:00.685 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.684+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.686 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.684+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1196957166}) 2024-09-16T04:10:00.686 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.684+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4136055512}) 2024-09-16T04:10:00.687 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.684+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.687 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.684+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1704174697}) 2024-09-16T04:10:00.687 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.684+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=442961467}) 2024-09-16T04:10:00.688 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.684+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.688 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.684+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1704174697}) 2024-09-16T04:10:00.688 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.684+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1733787975}) 2024-09-16T04:10:00.689 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.688+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.690 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.688+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2986468379}) 2024-09-16T04:10:00.690 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.688+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2353813414}) 2024-09-16T04:10:00.690 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.688+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.690 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.688+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2986468379}) 2024-09-16T04:10:00.690 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.688+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=377979247}) 2024-09-16T04:10:00.692 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.688+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.692 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.688+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1626661401}) 2024-09-16T04:10:00.692 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.688+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4215236207}) 2024-09-16T04:10:00.692 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.688+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.692 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.688+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1626661401}) 2024-09-16T04:10:00.692 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.688+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2547937916}) 2024-09-16T04:10:00.694 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.692+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.694 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.692+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4222901250}) 2024-09-16T04:10:00.694 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.692+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1201086937}) 2024-09-16T04:10:00.694 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.692+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.694 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.692+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4222901250}) 2024-09-16T04:10:00.694 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.692+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1554441842}) 2024-09-16T04:10:00.697 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.692+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.697 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.692+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=419393141}) 2024-09-16T04:10:00.697 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.692+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2632849188}) 2024-09-16T04:10:00.697 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.692+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.697 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.692+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=419393141}) 2024-09-16T04:10:00.697 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.692+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=857887793}) 2024-09-16T04:10:00.698 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.696+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.699 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.696+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1463040731}) 2024-09-16T04:10:00.699 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.696+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1454302044}) 2024-09-16T04:10:00.699 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.696+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.699 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.696+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1463040731}) 2024-09-16T04:10:00.699 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.696+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3816871599}) 2024-09-16T04:10:00.701 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.696+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.701 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.696+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2070925801}) 2024-09-16T04:10:00.701 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.696+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4140098977}) 2024-09-16T04:10:00.701 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.696+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.701 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.696+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2070925801}) 2024-09-16T04:10:00.701 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.696+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=635579981}) 2024-09-16T04:10:00.705 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.700+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.705 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.700+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3953192068}) 2024-09-16T04:10:00.705 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.700+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1611217580}) 2024-09-16T04:10:00.705 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.700+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.705 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.700+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3953192068}) 2024-09-16T04:10:00.705 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.700+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1009667222}) 2024-09-16T04:10:00.707 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.704+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.707 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.704+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3177705930}) 2024-09-16T04:10:00.707 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.704+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1121876590}) 2024-09-16T04:10:00.707 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.704+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.707 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.704+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3177705930}) 2024-09-16T04:10:00.707 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.704+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3606744377}) 2024-09-16T04:10:00.709 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.704+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.709 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.704+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2987307965}) 2024-09-16T04:10:00.709 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.704+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2250604605}) 2024-09-16T04:10:00.709 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.704+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.709 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.704+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2987307965}) 2024-09-16T04:10:00.709 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.704+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2994843907}) 2024-09-16T04:10:00.711 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.708+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.711 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.708+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1185689256}) 2024-09-16T04:10:00.711 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.708+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3101100944}) 2024-09-16T04:10:00.711 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.708+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.711 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.708+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1185689256}) 2024-09-16T04:10:00.711 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.708+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2482935567}) 2024-09-16T04:10:00.713 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.712+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.713 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.712+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1646711323}) 2024-09-16T04:10:00.714 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.712+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2298555009}) 2024-09-16T04:10:00.714 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.712+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.714 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.712+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1646711323}) 2024-09-16T04:10:00.714 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.712+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2042877059}) 2024-09-16T04:10:00.715 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.712+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.716 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.712+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=136827562}) 2024-09-16T04:10:00.716 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.712+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=501760535}) 2024-09-16T04:10:00.716 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.712+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.716 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.712+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=136827562}) 2024-09-16T04:10:00.716 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.712+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1618697375}) 2024-09-16T04:10:00.718 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.716+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.718 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.716+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=153165003}) 2024-09-16T04:10:00.718 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.716+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=147069560}) 2024-09-16T04:10:00.718 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.716+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.718 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.716+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=153165003}) 2024-09-16T04:10:00.718 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.716+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=94212927}) 2024-09-16T04:10:00.720 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.716+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.720 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.716+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2369549605}) 2024-09-16T04:10:00.720 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.716+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=908796297}) 2024-09-16T04:10:00.720 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.716+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.720 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.716+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2369549605}) 2024-09-16T04:10:00.720 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.716+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3469624558}) 2024-09-16T04:10:00.722 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.720+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.722 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.720+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2821229892}) 2024-09-16T04:10:00.722 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.720+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=799900172}) 2024-09-16T04:10:00.722 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.720+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.722 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.720+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2821229892}) 2024-09-16T04:10:00.723 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.720+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3131340576}) 2024-09-16T04:10:00.724 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.720+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.725 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.720+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1674438938}) 2024-09-16T04:10:00.725 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.720+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1037397929}) 2024-09-16T04:10:00.725 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.720+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.725 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.720+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1674438938}) 2024-09-16T04:10:00.725 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.720+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2930727854}) 2024-09-16T04:10:00.727 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.724+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.727 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.724+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=778058715}) 2024-09-16T04:10:00.727 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.724+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2792665567}) 2024-09-16T04:10:00.727 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.724+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.727 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.724+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=778058715}) 2024-09-16T04:10:00.727 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.724+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=591107358}) 2024-09-16T04:10:00.729 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.724+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.729 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.724+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2005774327}) 2024-09-16T04:10:00.729 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.724+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=710258414}) 2024-09-16T04:10:00.729 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.724+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.729 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.724+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2005774327}) 2024-09-16T04:10:00.729 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.724+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3590497762}) 2024-09-16T04:10:00.731 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.728+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.731 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.728+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1054403870}) 2024-09-16T04:10:00.731 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.728+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2455880440}) 2024-09-16T04:10:00.731 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.728+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.732 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.728+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1054403870}) 2024-09-16T04:10:00.732 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.728+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3562959101}) 2024-09-16T04:10:00.733 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.732+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.734 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.732+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1301997127}) 2024-09-16T04:10:00.734 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.732+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2109498393}) 2024-09-16T04:10:00.734 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.732+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.734 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.732+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1301997127}) 2024-09-16T04:10:00.734 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.732+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1295162610}) 2024-09-16T04:10:00.736 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.732+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.736 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.732+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2436079409}) 2024-09-16T04:10:00.736 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.732+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2269493290}) 2024-09-16T04:10:00.736 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.732+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.736 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.732+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2436079409}) 2024-09-16T04:10:00.736 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.732+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1949462887}) 2024-09-16T04:10:00.738 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.736+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.738 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.736+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3658569498}) 2024-09-16T04:10:00.738 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.736+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=234034604}) 2024-09-16T04:10:00.738 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.736+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.738 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.736+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3658569498}) 2024-09-16T04:10:00.739 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.736+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2358331429}) 2024-09-16T04:10:00.740 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.736+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.741 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.736+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1769239953}) 2024-09-16T04:10:00.741 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.736+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3052155577}) 2024-09-16T04:10:00.741 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.736+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.741 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.736+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1769239953}) 2024-09-16T04:10:00.741 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.736+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2543645792}) 2024-09-16T04:10:00.742 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.740+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.743 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.740+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3483883731}) 2024-09-16T04:10:00.743 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.740+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3839995048}) 2024-09-16T04:10:00.743 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.740+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.743 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.740+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3483883731}) 2024-09-16T04:10:00.743 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.740+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2568755023}) 2024-09-16T04:10:00.745 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.740+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.745 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.740+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2437264786}) 2024-09-16T04:10:00.745 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.740+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1038069386}) 2024-09-16T04:10:00.745 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.740+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.745 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.740+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2437264786}) 2024-09-16T04:10:00.745 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.740+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3469443238}) 2024-09-16T04:10:00.747 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.744+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.747 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.744+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=831843798}) 2024-09-16T04:10:00.747 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.744+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2699057344}) 2024-09-16T04:10:00.747 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.744+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.748 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.744+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=831843798}) 2024-09-16T04:10:00.748 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.744+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4115622896}) 2024-09-16T04:10:00.751 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.748+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.751 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.748+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1785690944}) 2024-09-16T04:10:00.751 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.748+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2860044361}) 2024-09-16T04:10:00.751 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.748+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.751 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.748+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1785690944}) 2024-09-16T04:10:00.751 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.748+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3830222543}) 2024-09-16T04:10:00.753 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.752+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.753 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.752+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3484992297}) 2024-09-16T04:10:00.753 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.752+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1952991563}) 2024-09-16T04:10:00.754 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.752+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.754 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.752+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3484992297}) 2024-09-16T04:10:00.754 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.752+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1204429835}) 2024-09-16T04:10:00.756 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.752+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.756 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.752+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3457421338}) 2024-09-16T04:10:00.756 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.752+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=831018343}) 2024-09-16T04:10:00.756 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.752+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.756 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.752+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3457421338}) 2024-09-16T04:10:00.756 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.752+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1328152402}) 2024-09-16T04:10:00.758 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.756+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.758 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.756+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2069287223}) 2024-09-16T04:10:00.758 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.756+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=618079635}) 2024-09-16T04:10:00.758 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.756+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.758 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.756+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2069287223}) 2024-09-16T04:10:00.758 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.756+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3827321701}) 2024-09-16T04:10:00.760 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.756+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.760 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.756+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1771466556}) 2024-09-16T04:10:00.760 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.756+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=799452802}) 2024-09-16T04:10:00.760 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.756+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.760 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.756+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1771466556}) 2024-09-16T04:10:00.760 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.756+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3013281237}) 2024-09-16T04:10:00.763 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.760+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.763 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.760+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=70122158}) 2024-09-16T04:10:00.763 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.760+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3878146507}) 2024-09-16T04:10:00.763 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.760+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.764 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.760+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=70122158}) 2024-09-16T04:10:00.764 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.760+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1584568334}) 2024-09-16T04:10:00.764 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.760+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.765 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.760+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=198239374}) 2024-09-16T04:10:00.765 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.760+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2941256897}) 2024-09-16T04:10:00.765 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.760+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.765 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.760+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=198239374}) 2024-09-16T04:10:00.765 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.760+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=228857505}) 2024-09-16T04:10:00.767 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.764+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.767 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.764+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2973412680}) 2024-09-16T04:10:00.767 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.764+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1128233785}) 2024-09-16T04:10:00.767 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.764+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.767 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.764+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2973412680}) 2024-09-16T04:10:00.767 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.764+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4172542008}) 2024-09-16T04:10:00.769 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.764+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.769 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.764+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1170984411}) 2024-09-16T04:10:00.769 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.764+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2926753754}) 2024-09-16T04:10:00.769 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.764+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.769 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.764+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1170984411}) 2024-09-16T04:10:00.769 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.764+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2455373265}) 2024-09-16T04:10:00.771 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.768+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.771 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.768+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1357555525}) 2024-09-16T04:10:00.771 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.768+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=958882870}) 2024-09-16T04:10:00.771 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.768+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.771 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.768+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1357555525}) 2024-09-16T04:10:00.771 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.768+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3702729134}) 2024-09-16T04:10:00.773 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.768+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.773 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.768+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3367893618}) 2024-09-16T04:10:00.773 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.768+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2318687272}) 2024-09-16T04:10:00.773 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.768+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.773 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.768+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3367893618}) 2024-09-16T04:10:00.773 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.768+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1875889433}) 2024-09-16T04:10:00.775 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.772+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.775 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.772+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1975686672}) 2024-09-16T04:10:00.775 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.772+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2551636546}) 2024-09-16T04:10:00.775 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.772+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.775 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.772+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1975686672}) 2024-09-16T04:10:00.775 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.772+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=522321305}) 2024-09-16T04:10:00.777 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.776+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.777 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.776+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2899607785}) 2024-09-16T04:10:00.777 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.776+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1133417879}) 2024-09-16T04:10:00.777 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.776+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.778 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.776+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2899607785}) 2024-09-16T04:10:00.778 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.776+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=917472550}) 2024-09-16T04:10:00.780 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.776+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.781 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.776+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1450707257}) 2024-09-16T04:10:00.781 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.776+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3952117492}) 2024-09-16T04:10:00.781 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.776+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.781 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.776+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1450707257}) 2024-09-16T04:10:00.781 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.776+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3086883040}) 2024-09-16T04:10:00.782 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.780+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.782 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.780+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2512028937}) 2024-09-16T04:10:00.782 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.780+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4094667949}) 2024-09-16T04:10:00.782 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.780+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.782 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.780+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2512028937}) 2024-09-16T04:10:00.782 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.780+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4277997037}) 2024-09-16T04:10:00.784 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.780+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.784 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.780+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4124636478}) 2024-09-16T04:10:00.784 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.780+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2190078158}) 2024-09-16T04:10:00.784 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.780+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.784 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.780+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4124636478}) 2024-09-16T04:10:00.785 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.780+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=913302326}) 2024-09-16T04:10:00.786 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.784+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.786 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.784+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2607045951}) 2024-09-16T04:10:00.787 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.784+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=64594673}) 2024-09-16T04:10:00.787 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.784+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.787 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.784+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2607045951}) 2024-09-16T04:10:00.787 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.784+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1484193256}) 2024-09-16T04:10:00.789 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.784+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.789 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.784+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=486483083}) 2024-09-16T04:10:00.789 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.784+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1029162308}) 2024-09-16T04:10:00.789 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.784+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.789 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.784+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=486483083}) 2024-09-16T04:10:00.789 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.784+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1308808820}) 2024-09-16T04:10:00.791 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.788+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.791 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.788+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1253116287}) 2024-09-16T04:10:00.791 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.788+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3671565170}) 2024-09-16T04:10:00.791 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.788+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.791 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.788+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1253116287}) 2024-09-16T04:10:00.792 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.788+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2063351209}) 2024-09-16T04:10:00.793 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.792+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.794 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.792+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=361362005}) 2024-09-16T04:10:00.794 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.792+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2412269597}) 2024-09-16T04:10:00.794 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.792+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.794 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.792+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=361362005}) 2024-09-16T04:10:00.794 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.792+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3275218053}) 2024-09-16T04:10:00.796 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.792+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.796 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.792+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1175131122}) 2024-09-16T04:10:00.796 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.792+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3495733512}) 2024-09-16T04:10:00.796 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.792+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.796 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.792+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1175131122}) 2024-09-16T04:10:00.796 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.792+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2236964262}) 2024-09-16T04:10:00.798 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.796+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.798 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.796+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3582758063}) 2024-09-16T04:10:00.798 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.796+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2177325599}) 2024-09-16T04:10:00.799 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.796+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.799 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.796+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3582758063}) 2024-09-16T04:10:00.799 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.796+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4271133982}) 2024-09-16T04:10:00.801 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.796+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.801 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.796+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2533347201}) 2024-09-16T04:10:00.801 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.796+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3581159760}) 2024-09-16T04:10:00.801 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.796+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.801 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.796+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2533347201}) 2024-09-16T04:10:00.801 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.796+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=858003764}) 2024-09-16T04:10:00.804 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.800+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.804 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.800+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=12,mdsmap=3,mgr=6,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=42,osdmap=6} crc {logm=3797634720,mdsmap=4088524775,mgr=4251399302,mgr_command_descs=2107673600,mgr_metadata=1367272202,mgrstat=1298514717,mon_config_key=85057112,monmap=3937513981,osd_metadata=143105525,osd_pg_creating=783066558,osd_snap=952392750,osdmap=809960269}) 2024-09-16T04:10:00.804 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.800+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=13,mdsmap=3,mgr=6,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=9,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=42,osdmap=7} crc {logm=519332441,mdsmap=4088524775,mgr=566604210,mgr_command_descs=2107673600,mgr_metadata=1367272202,mgrstat=1298514717,mon_config_key=3119090094,monmap=3937513981,osd_metadata=4182959540,osd_pg_creating=783066558,osd_snap=2287121524,osdmap=1230316791}) 2024-09-16T04:10:00.804 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.800+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.804 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.800+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=12,mdsmap=3,mgr=6,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=42,osdmap=6} crc {logm=3797634720,mdsmap=4088524775,mgr=4251399302,mgr_command_descs=2107673600,mgr_metadata=1367272202,mgrstat=1298514717,mon_config_key=85057112,monmap=3937513981,osd_metadata=143105525,osd_pg_creating=783066558,osd_snap=952392750,osdmap=809960269}) 2024-09-16T04:10:00.804 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.800+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=12,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=41,osdmap=5} crc {logm=1125819340,mdsmap=4088524775,mgr=460109776,mgr_command_descs=2107673600,mgr_metadata=1367272202,mgrstat=1298514717,mon_config_key=85057112,monmap=3937513981,osd_metadata=4182959540,osd_pg_creating=783066558,osd_snap=4147112049,osdmap=1259260822}) 2024-09-16T04:10:00.806 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.804+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.806 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.804+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=134806556}) 2024-09-16T04:10:00.806 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.804+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2068482073}) 2024-09-16T04:10:00.806 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.804+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.806 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.804+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=134806556}) 2024-09-16T04:10:00.807 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.804+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1501896086}) 2024-09-16T04:10:00.809 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.804+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.809 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.804+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2252015474}) 2024-09-16T04:10:00.809 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.804+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3071363830}) 2024-09-16T04:10:00.809 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.804+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.809 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.804+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2252015474}) 2024-09-16T04:10:00.809 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.804+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=787137301}) 2024-09-16T04:10:00.813 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.808+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.813 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.808+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=4181958902}) 2024-09-16T04:10:00.813 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.808+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3288490253}) 2024-09-16T04:10:00.813 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.808+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.813 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.808+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=4181958902}) 2024-09-16T04:10:00.813 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.812+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3865788344}) 2024-09-16T04:10:00.831 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.828+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.831 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.828+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=82} crc {osdmap=2789769966}) 2024-09-16T04:10:00.831 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.828+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=76} crc {osdmap=3171540780}) 2024-09-16T04:10:00.831 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.828+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:00.831 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.828+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=82} crc {osdmap=2789769966}) 2024-09-16T04:10:00.831 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:00.828+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=82} crc {osdmap=3974654172}) 2024-09-16T04:10:01.535 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T04:10:01.536 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:10:01.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.668+0000 7f6fadec0640 10 monclient: get_monmap_and_config 2024-09-16T04:10:01.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.668+0000 7f6fadec0640 10 monclient: build_initial_monmap 2024-09-16T04:10:01.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.668+0000 7f6fadec0640 10 monclient: monmap: 2024-09-16T04:10:01.671 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:10:01.671 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:10:01.671 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:10:01.669570+0000 2024-09-16T04:10:01.671 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:10:01.669570+0000 2024-09-16T04:10:01.671 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:10:01.671 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:10:01.671 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:10:01.671 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:10:01.672 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:10:01.672 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:10:01.672 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:10:01.672 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:10:01.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.668+0000 7f6fadec0640 1 Processor -- start 2024-09-16T04:10:01.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.668+0000 7f6fadec0640 1 -- start start 2024-09-16T04:10:01.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.668+0000 7f6fadec0640 10 monclient: init 2024-09-16T04:10:01.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.668+0000 7f6fadec0640 10 monclient: _reopen_session rank -1 2024-09-16T04:10:01.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.668+0000 7f6fadec0640 10 monclient: _add_conns ranks=[1,3,0,2,4] 2024-09-16T04:10:01.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.668+0000 7f6fadec0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6fa80a1330 addr v1:172.21.15.154:6789/0 2024-09-16T04:10:01.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.668+0000 7f6fadec0640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f6fa806dac0 0x7f6fa813a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:10:01.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.668+0000 7f6fadec0640 10 monclient(hunting): picked mon.noname-a con 0x7f6fa806dac0 addr v2:172.21.15.42:3300/0 2024-09-16T04:10:01.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.668+0000 7f6fadec0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6fa80a1e20 addr v1:172.21.15.42:6789/0 2024-09-16T04:10:01.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.668+0000 7f6fadec0640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6fa809c090 con 0x7f6fa80a1e20 2024-09-16T04:10:01.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.668+0000 7f6fadec0640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6fa809f610 con 0x7f6fa80a1330 2024-09-16T04:10:01.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.668+0000 7f6fadec0640 10 monclient(hunting): start opening mon connection 2024-09-16T04:10:01.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.668+0000 7f6fadec0640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f6fa809f8f0 con 0x7f6fa806dac0 2024-09-16T04:10:01.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.668+0000 7f6fadec0640 10 monclient(hunting): _renew_subs 2024-09-16T04:10:01.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.668+0000 7f6fadec0640 10 monclient(hunting): authenticate will time out at 2648.639108s 2024-09-16T04:10:01.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.668+0000 7f6fa6ffd640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f6fa806dac0 0x7f6fa813a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:10:01.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.668+0000 7f6fa7fff640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f6fa80a1e20 0x7f6fa813c890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:37324/0 (socket says 172.21.15.42:37324) 2024-09-16T04:10:01.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.668+0000 7f6fa7fff640 1 -- 172.21.15.42:0/2410755329 learned_addr learned my addr 172.21.15.42:0/2410755329 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T04:10:01.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.668+0000 7f6fa6ffd640 1 --2- 172.21.15.42:0/2410755329 >> v2:172.21.15.42:3300/0 conn(0x7f6fa806dac0 0x7f6fa813a340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:60100/0 (socket says 172.21.15.42:60100) 2024-09-16T04:10:01.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.668+0000 7f6fa6ffd640 10 monclient(hunting): get_auth_request con 0x7f6fa806dac0 auth_method 0 2024-09-16T04:10:01.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.668+0000 7f6fa6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:10:01.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.668+0000 7f6fa6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:01.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.668+0000 7f6fa6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:01.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.668+0000 7f6fa6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:10:01.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.668+0000 7f6fa6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:10:01.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.668+0000 7f6fa6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:10:01.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.668+0000 7f6fa6ffd640 10 monclient(hunting): handle_auth_done global_id 1034226 payload 274 2024-09-16T04:10:01.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.668+0000 7f6fa6ffd640 1 -- 172.21.15.42:0/2410755329 >> v1:172.21.15.154:6789/0 conn(0x7f6fa80a1330 legacy=0x7f6fa80a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:10:01.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.668+0000 7f6fa6ffd640 1 -- 172.21.15.42:0/2410755329 >> v1:172.21.15.42:6789/0 conn(0x7f6fa80a1e20 legacy=0x7f6fa813c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:01.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.668+0000 7f6fa6ffd640 10 monclient: _finish_hunting 0 2024-09-16T04:10:01.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.668+0000 7f6fa6ffd640 1 monclient: found mon.noname-a 2024-09-16T04:10:01.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.668+0000 7f6fa6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:10:01.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.668+0000 7f6fa67fc640 1 -- 172.21.15.42:0/2410755329 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2983851157 0 0) 0x7f6fa809c090 con 0x7f6fa80a1e20 2024-09-16T04:10:01.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.668+0000 7f6fa6ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:10:01.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.668+0000 7f6fa6ffd640 1 -- 172.21.15.42:0/2410755329 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fa809fbd0 con 0x7f6fa806dac0 2024-09-16T04:10:01.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.668+0000 7f6fa67fc640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T04:10:01.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.672+0000 7f6fa6ffd640 1 --2- 172.21.15.42:0/2410755329 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6fa806dac0 0x7f6fa813a340 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f6f900097c0 tx=0x7f6f9001c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65994ddbd1298468 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:10:01.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.672+0000 7f6fa67fc640 1 -- 172.21.15.42:0/2410755329 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f900b0070 con 0x7f6fa806dac0 2024-09-16T04:10:01.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.672+0000 7f6fa67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:01.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.672+0000 7f6fa67fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:10:01.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.672+0000 7f6fa67fc640 10 monclient: dump: 2024-09-16T04:10:01.682 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:10:01.682 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:10:01.682 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:01.682 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:01.682 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:01.682 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:10:01.682 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:10:01.683 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:10:01.683 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:10:01.683 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:10:01.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.672+0000 7f6fa67fc640 10 monclient: _finish_auth 0 2024-09-16T04:10:01.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.672+0000 7f6fa67fc640 10 monclient: _check_auth_tickets 2024-09-16T04:10:01.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.672+0000 7f6fa67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:01.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.672+0000 7f6fa67fc640 1 -- 172.21.15.42:0/2410755329 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f900ab600 con 0x7f6fa806dac0 2024-09-16T04:10:01.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.672+0000 7f6fa67fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:10:01.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.672+0000 7f6fa67fc640 1 -- 172.21.15.42:0/2410755329 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f900ab9d0 con 0x7f6fa806dac0 2024-09-16T04:10:01.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.672+0000 7f6fadec0640 5 monclient: authenticate success, global_id 1034226 2024-09-16T04:10:01.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.672+0000 7f6fadec0640 10 monclient: get_monmap_and_config success 2024-09-16T04:10:01.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.672+0000 7f6fadec0640 4 set_mon_vals no callback set 2024-09-16T04:10:01.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.672+0000 7f6fa67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:01.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.672+0000 7f6fa67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:10:01.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.672+0000 7f6fa67fc640 10 monclient: dump: 2024-09-16T04:10:01.684 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:10:01.684 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:10:01.684 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:01.684 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:01.684 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:01.684 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:10:01.684 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:10:01.684 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:10:01.684 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:10:01.684 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:10:01.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.672+0000 7f6fadec0640 10 monclient: shutdown 2024-09-16T04:10:01.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.672+0000 7f6fadec0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:10:01.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.672+0000 7f6fadec0640 1 -- 172.21.15.42:0/2410755329 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6fa806dac0 msgr2=0x7f6fa813a340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:01.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.672+0000 7f6fadec0640 1 --2- 172.21.15.42:0/2410755329 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6fa806dac0 0x7f6fa813a340 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f6f900097c0 tx=0x7f6f9001c440 comp rx=0 tx=0).stop 2024-09-16T04:10:01.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.672+0000 7f6fadec0640 1 -- 172.21.15.42:0/2410755329 shutdown_connections 2024-09-16T04:10:01.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.672+0000 7f6fadec0640 1 --2- 172.21.15.42:0/2410755329 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6fa806dac0 0x7f6fa813a340 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:10:01.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.672+0000 7f6fadec0640 1 -- 172.21.15.42:0/2410755329 >> 172.21.15.42:0/2410755329 conn(0x7f6fa80a2420 msgr2=0x7f6fa80a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:10:01.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.672+0000 7f6fadec0640 1 -- 172.21.15.42:0/2410755329 shutdown_connections 2024-09-16T04:10:01.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.672+0000 7f6fadec0640 1 -- 172.21.15.42:0/2410755329 wait complete. 2024-09-16T04:10:01.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.672+0000 7f6fadec0640 10 monclient: build_initial_monmap 2024-09-16T04:10:01.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.672+0000 7f6fadec0640 10 monclient: monmap: 2024-09-16T04:10:01.685 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:10:01.685 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:10:01.685 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:10:01.685 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:10:01.685 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:10:01.686 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:10:01.686 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:10:01.686 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:10:01.686 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:10:01.686 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:10:01.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.672+0000 7f6fadec0640 1 librados: starting msgr at 2024-09-16T04:10:01.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.672+0000 7f6fadec0640 1 librados: starting objecter 2024-09-16T04:10:01.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.672+0000 7f6fadec0640 1 Processor -- start 2024-09-16T04:10:01.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.672+0000 7f6fadec0640 1 -- start start 2024-09-16T04:10:01.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.672+0000 7f6fadec0640 1 librados: setting wanted keys 2024-09-16T04:10:01.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.672+0000 7f6fadec0640 1 librados: calling monclient init 2024-09-16T04:10:01.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.672+0000 7f6fadec0640 10 monclient: init 2024-09-16T04:10:01.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.676+0000 7f6fadec0640 10 monclient: _reopen_session rank -1 2024-09-16T04:10:01.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.676+0000 7f6fadec0640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:10:01.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.676+0000 7f6fadec0640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f6fa80a1330 0x7f6fa813bee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:10:01.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.676+0000 7f6fadec0640 10 monclient(hunting): picked mon.noname-c con 0x7f6fa80a1330 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:10:01.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.676+0000 7f6fadec0640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6fa80a1e20 0x7f6fa81cd2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:10:01.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.676+0000 7f6fadec0640 10 monclient(hunting): picked mon.noname-a con 0x7f6fa80a1e20 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:10:01.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.676+0000 7f6fadec0640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6fa81cd810 0x7f6fa811bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:10:01.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.676+0000 7f6fadec0640 10 monclient(hunting): picked mon.noname-b con 0x7f6fa81cd810 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:10:01.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.676+0000 7f6fadec0640 10 monclient(hunting): start opening mon connection 2024-09-16T04:10:01.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.676+0000 7f6fadec0640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f6fa81409e0 con 0x7f6fa80a1e20 2024-09-16T04:10:01.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.676+0000 7f6fadec0640 10 monclient(hunting): start opening mon connection 2024-09-16T04:10:01.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.676+0000 7f6fadec0640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f6fa809efb0 con 0x7f6fa81cd810 2024-09-16T04:10:01.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.676+0000 7f6fadec0640 10 monclient(hunting): start opening mon connection 2024-09-16T04:10:01.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.676+0000 7f6fadec0640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f6fa8140b60 con 0x7f6fa80a1330 2024-09-16T04:10:01.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.676+0000 7f6fadec0640 10 monclient(hunting): _renew_subs 2024-09-16T04:10:01.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.676+0000 7f6fadec0640 10 monclient(hunting): authenticate will time out at 2648.644667s 2024-09-16T04:10:01.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.676+0000 7f6fa77fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f6fa80a1330 0x7f6fa813bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:10:01.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.676+0000 7f6fa77fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f6fa80a1330 0x7f6fa813bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:51186/0 (socket says 172.21.15.42:51186) 2024-09-16T04:10:01.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.676+0000 7f6fa77fe640 1 -- 172.21.15.42:0/3893819660 learned_addr learned my addr 172.21.15.42:0/3893819660 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:10:01.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.676+0000 7f6fa77fe640 10 monclient(hunting): get_auth_request con 0x7f6fa80a1330 auth_method 0 2024-09-16T04:10:01.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.676+0000 7f6fa77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:10:01.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.676+0000 7f6fa77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:01.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.676+0000 7f6fa77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:01.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.676+0000 7f6fa6ffd640 1 --2- 172.21.15.42:0/3893819660 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6fa80a1e20 0x7f6fa81cd2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:10:01.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.676+0000 7f6fa77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:10:01.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.676+0000 7f6fa77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:10:01.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.676+0000 7f6fa77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:10:01.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.676+0000 7f6fa6ffd640 10 monclient(hunting): get_auth_request con 0x7f6fa80a1e20 auth_method 0 2024-09-16T04:10:01.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.676+0000 7f6fa6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:10:01.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.676+0000 7f6fa6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:01.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.676+0000 7f6fa6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:01.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.676+0000 7f6fa6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:10:01.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.676+0000 7f6fa6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:10:01.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.676+0000 7f6fa6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:10:01.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.676+0000 7f6fa77fe640 10 monclient(hunting): handle_auth_done global_id 1014362 payload 824 2024-09-16T04:10:01.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.676+0000 7f6fa7fff640 1 --2- 172.21.15.42:0/3893819660 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6fa81cd810 0x7f6fa811bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:10:01.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.676+0000 7f6fa77fe640 1 -- 172.21.15.42:0/3893819660 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6fa81cd810 msgr2=0x7f6fa811bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:01.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.676+0000 7f6fa77fe640 1 --2- 172.21.15.42:0/3893819660 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6fa81cd810 0x7f6fa811bf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:10:01.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.676+0000 7f6fa77fe640 1 -- 172.21.15.42:0/3893819660 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6fa80a1e20 msgr2=0x7f6fa81cd2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:01.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.676+0000 7f6fa77fe640 1 --2- 172.21.15.42:0/3893819660 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6fa80a1e20 0x7f6fa81cd2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:10:01.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.676+0000 7f6fa77fe640 10 monclient: _finish_hunting 0 2024-09-16T04:10:01.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.676+0000 7f6fa77fe640 1 monclient: found mon.noname-c 2024-09-16T04:10:01.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.676+0000 7f6fa77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:10:01.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.676+0000 7f6fa77fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T04:10:01.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.676+0000 7f6fa77fe640 1 -- 172.21.15.42:0/3893819660 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fa81c5c80 con 0x7f6fa80a1330 2024-09-16T04:10:01.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.676+0000 7f6fa7fff640 10 monclient: get_auth_request con 0x7f6fa81cd810 auth_method 0 2024-09-16T04:10:01.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.676+0000 7f6fa7fff640 1 --2- 172.21.15.42:0/3893819660 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6fa81cd810 0x7f6fa811bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T04:10:01.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.676+0000 7f6fa77fe640 1 --2- 172.21.15.42:0/3893819660 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f6fa80a1330 0x7f6fa813bee0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f6fa000c7f0 tx=0x7f6fa000ccb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:10:01.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.676+0000 7f6f9ffff640 10 client.?.objecter ms_handle_connect 0x7f6fa80a1330 2024-09-16T04:10:01.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.676+0000 7f6f9ffff640 10 client.?.objecter resend_mon_ops 2024-09-16T04:10:01.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.676+0000 7f6f9ffff640 1 -- 172.21.15.42:0/3893819660 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fa0016020 con 0x7f6fa80a1330 2024-09-16T04:10:01.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.676+0000 7f6f9ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:01.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.676+0000 7f6f9ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:10:01.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.676+0000 7f6f9ffff640 10 monclient: dump: 2024-09-16T04:10:01.690 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:10:01.690 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:10:01.691 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:01.691 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:01.691 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:01.691 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:10:01.691 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:10:01.691 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:10:01.691 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:10:01.691 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:10:01.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.676+0000 7f6f9ffff640 10 monclient: _finish_auth 0 2024-09-16T04:10:01.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.676+0000 7f6f9ffff640 10 monclient: _check_auth_tickets 2024-09-16T04:10:01.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.676+0000 7f6f9ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:01.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.676+0000 7f6f9ffff640 1 -- 172.21.15.42:0/3893819660 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6fa000aa50 con 0x7f6fa80a1330 2024-09-16T04:10:01.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.676+0000 7f6fadec0640 5 monclient: authenticate success, global_id 1014362 2024-09-16T04:10:01.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.676+0000 7f6fadec0640 10 monclient: _renew_subs 2024-09-16T04:10:01.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.676+0000 7f6fadec0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:10:01.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.676+0000 7f6fadec0640 1 -- 172.21.15.42:0/3893819660 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6fa811e070 con 0x7f6fa80a1330 2024-09-16T04:10:01.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.676+0000 7f6f9ffff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:10:01.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.676+0000 7f6f9ffff640 1 -- 172.21.15.42:0/3893819660 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fa000ae40 con 0x7f6fa80a1330 2024-09-16T04:10:01.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.676+0000 7f6f9ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:01.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.676+0000 7f6f9ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:10:01.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.676+0000 7f6f9ffff640 10 monclient: dump: 2024-09-16T04:10:01.693 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:10:01.693 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:10:01.693 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:01.693 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:01.693 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:01.693 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:10:01.693 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:10:01.693 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:10:01.693 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:10:01.693 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:10:01.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.676+0000 7f6fa5ffb640 4 set_mon_vals no callback set 2024-09-16T04:10:01.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.676+0000 7f6fadec0640 10 client.1014362.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:10:01.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.676+0000 7f6fadec0640 10 monclient: _renew_subs 2024-09-16T04:10:01.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.676+0000 7f6fadec0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:10:01.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.676+0000 7f6fadec0640 1 -- 172.21.15.42:0/3893819660 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f6fa813d2a0 con 0x7f6fa80a1330 2024-09-16T04:10:01.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.676+0000 7f6fadec0640 1 librados: init done 2024-09-16T04:10:01.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.676+0000 7f6fadec0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:01.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.676+0000 7f6fadec0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:01.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.676+0000 7f6fadec0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:10:01.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.676+0000 7f6f9ffff640 1 -- 172.21.15.42:0/3893819660 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f6fa0019020 con 0x7f6fa80a1330 2024-09-16T04:10:01.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.676+0000 7f6fadec0640 1 -- 172.21.15.42:0/3893819660 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6fa80a1710 con 0x7f6fa80a1330 2024-09-16T04:10:01.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.676+0000 7f6f9ffff640 1 --2- 172.21.15.42:0/3893819660 >> v2:172.21.15.154:6812/254845342 conn(0x7f6f88037810 0x7f6f88039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:10:01.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.676+0000 7f6f9ffff640 1 -- 172.21.15.42:0/3893819660 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(706..706 src has 451..706) ==== 5040+0+0 (secure 0 0 0) 0x7f6fa0048be0 con 0x7f6fa80a1330 2024-09-16T04:10:01.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.676+0000 7f6f9ffff640 10 client.1014362.objecter ms_dispatch 0x7f6fa8000ba0 osd_map(706..706 src has 451..706) 2024-09-16T04:10:01.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.676+0000 7f6f9ffff640 3 client.1014362.objecter handle_osd_map got epochs [706,706] > 0 2024-09-16T04:10:01.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.676+0000 7f6f9ffff640 3 client.1014362.objecter handle_osd_map decoding full epoch 706 2024-09-16T04:10:01.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.676+0000 7f6f9ffff640 20 client.1014362.objecter dump_active .. 0 homeless 2024-09-16T04:10:01.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.676+0000 7f6fa6ffd640 1 --2- 172.21.15.42:0/3893819660 >> v2:172.21.15.154:6812/254845342 conn(0x7f6f88037810 0x7f6f88039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:10:01.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.676+0000 7f6fa6ffd640 10 monclient: get_auth_request con 0x7f6f88037810 auth_method 0 2024-09-16T04:10:01.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.680+0000 7f6f9ffff640 10 client.1014362.objecter ms_handle_connect 0x7f6f88037810 2024-09-16T04:10:01.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.680+0000 7f6fa6ffd640 1 --2- 172.21.15.42:0/3893819660 >> v2:172.21.15.154:6812/254845342 conn(0x7f6f88037810 0x7f6f88039cd0 secure :-1 s=READY pgs=3921 cs=0 l=1 rev1=1 crypto rx=0x7f6f900ad000 tx=0x7f6f900a6000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:10:01.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.680+0000 7f6f9ffff640 1 -- 172.21.15.42:0/3893819660 <== mon.2 v2:172.21.15.42:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f6fa0020030 con 0x7f6fa80a1330 2024-09-16T04:10:01.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.684+0000 7f6f9ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:01.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.684+0000 7f6f9ffff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:10:01.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.908+0000 7f6fadec0640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:10:01.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.908+0000 7f6fadec0640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:10:01.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.908+0000 7f6fadec0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:10:01.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.908+0000 7f6fadec0640 1 -- 172.21.15.42:0/3893819660 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f6fa811e870 con 0x7f6fa80a1330 2024-09-16T04:10:01.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.912+0000 7f6f9ffff640 1 -- 172.21.15.42:0/3893819660 <== mon.2 v2:172.21.15.42:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7f6fa0012dc0 con 0x7f6fa80a1330 2024-09-16T04:10:01.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.912+0000 7f6f9ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:10:01.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.912+0000 7f6f9ffff640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:10:01.914 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":256,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:10:01.916 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.912+0000 7f6fadec0640 10 librados: watch_flush enter 2024-09-16T04:10:01.916 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.912+0000 7f6fadec0640 10 librados: watch_flush exit 2024-09-16T04:10:01.917 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.912+0000 7f6fadec0640 20 client.1014362.objecter shutdown clearing up homeless session... 2024-09-16T04:10:01.917 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.912+0000 7f6fadec0640 10 client.1014362.objecter successfully canceled tick 2024-09-16T04:10:01.917 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.912+0000 7f6fadec0640 1 -- 172.21.15.42:0/3893819660 >> v2:172.21.15.154:6812/254845342 conn(0x7f6f88037810 msgr2=0x7f6f88039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:01.917 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.912+0000 7f6fadec0640 1 --2- 172.21.15.42:0/3893819660 >> v2:172.21.15.154:6812/254845342 conn(0x7f6f88037810 0x7f6f88039cd0 secure :-1 s=READY pgs=3921 cs=0 l=1 rev1=1 crypto rx=0x7f6f900ad000 tx=0x7f6f900a6000 comp rx=0 tx=0).stop 2024-09-16T04:10:01.917 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.912+0000 7f6fadec0640 10 monclient: shutdown 2024-09-16T04:10:01.917 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.912+0000 7f6fadec0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:10:01.917 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.912+0000 7f6fadec0640 1 -- 172.21.15.42:0/3893819660 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f6fa80a1330 msgr2=0x7f6fa813bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:01.917 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.912+0000 7f6fadec0640 1 --2- 172.21.15.42:0/3893819660 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f6fa80a1330 0x7f6fa813bee0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f6fa000c7f0 tx=0x7f6fa000ccb0 comp rx=0 tx=0).stop 2024-09-16T04:10:01.917 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.912+0000 7f6fadec0640 1 -- 172.21.15.42:0/3893819660 shutdown_connections 2024-09-16T04:10:01.917 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.912+0000 7f6fadec0640 1 --2- 172.21.15.42:0/3893819660 >> v2:172.21.15.154:6812/254845342 conn(0x7f6f88037810 0x7f6f88039cd0 unknown :-1 s=CLOSED pgs=3921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:10:01.917 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.912+0000 7f6fadec0640 1 --2- 172.21.15.42:0/3893819660 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6fa81cd810 0x7f6fa811bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:10:01.917 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.912+0000 7f6fadec0640 1 --2- 172.21.15.42:0/3893819660 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6fa80a1e20 0x7f6fa81cd2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:10:01.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.912+0000 7f6fadec0640 1 --2- 172.21.15.42:0/3893819660 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f6fa80a1330 0x7f6fa813bee0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:10:01.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.912+0000 7f6fadec0640 1 -- 172.21.15.42:0/3893819660 >> 172.21.15.42:0/3893819660 conn(0x7f6fa80a2420 msgr2=0x7f6fa80a4dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:10:01.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.912+0000 7f6fadec0640 1 -- 172.21.15.42:0/3893819660 shutdown_connections 2024-09-16T04:10:01.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.912+0000 7f6fadec0640 1 -- 172.21.15.42:0/3893819660 wait complete. 2024-09-16T04:10:01.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:01.912+0000 7f6fadec0640 1 librados: shutdown 2024-09-16T04:10:01.933 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T04:10:01.933 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T04:10:01.933 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T04:10:01.934 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T04:10:02.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.052+0000 7f4ad32ba640 10 monclient: get_monmap_and_config 2024-09-16T04:10:02.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.052+0000 7f4ad32ba640 10 monclient: build_initial_monmap 2024-09-16T04:10:02.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.052+0000 7f4ad32ba640 10 monclient: monmap: 2024-09-16T04:10:02.054 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:10:02.054 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:10:02.054 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:10:02.052730+0000 2024-09-16T04:10:02.054 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:10:02.052730+0000 2024-09-16T04:10:02.054 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:10:02.054 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:10:02.054 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:10:02.055 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:10:02.055 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:10:02.055 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:10:02.055 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:10:02.055 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:10:02.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.052+0000 7f4ad32ba640 1 Processor -- start 2024-09-16T04:10:02.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.052+0000 7f4ad32ba640 1 -- start start 2024-09-16T04:10:02.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.052+0000 7f4ad32ba640 10 monclient: init 2024-09-16T04:10:02.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.052+0000 7f4ad32ba640 10 monclient: _reopen_session rank -1 2024-09-16T04:10:02.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.052+0000 7f4ad32ba640 10 monclient: _add_conns ranks=[3,1,2,0,4] 2024-09-16T04:10:02.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.052+0000 7f4ad32ba640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f4acc0a1330 0x7f4acc09f180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:10:02.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.052+0000 7f4ad32ba640 10 monclient(hunting): picked mon.noname-a con 0x7f4acc0a1330 addr v2:172.21.15.42:3300/0 2024-09-16T04:10:02.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.052+0000 7f4ad32ba640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4acc0a1710 addr v1:172.21.15.154:6789/0 2024-09-16T04:10:02.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.052+0000 7f4ad32ba640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4acc1393a0 0x7f4acc13b790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:10:02.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.052+0000 7f4ad32ba640 10 monclient(hunting): picked mon.noname-b con 0x7f4acc1393a0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:10:02.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.052+0000 7f4ad32ba640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4acc09e4c0 con 0x7f4acc0a1710 2024-09-16T04:10:02.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.052+0000 7f4ad32ba640 10 monclient(hunting): start opening mon connection 2024-09-16T04:10:02.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.052+0000 7f4ad32ba640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f4acc09edf0 con 0x7f4acc0a1330 2024-09-16T04:10:02.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.052+0000 7f4ad32ba640 10 monclient(hunting): start opening mon connection 2024-09-16T04:10:02.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.052+0000 7f4ad32ba640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f4acc09cd60 con 0x7f4acc1393a0 2024-09-16T04:10:02.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.052+0000 7f4ad32ba640 10 monclient(hunting): _renew_subs 2024-09-16T04:10:02.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.052+0000 7f4ad32ba640 10 monclient(hunting): authenticate will time out at 2649.022161s 2024-09-16T04:10:02.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.052+0000 7f4ad1857640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4acc1393a0 0x7f4acc13b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:10:02.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.052+0000 7f4ad1056640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f4acc0a1330 0x7f4acc09f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:10:02.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.052+0000 7f4ad1857640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4acc1393a0 0x7f4acc13b790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:51198/0 (socket says 172.21.15.42:51198) 2024-09-16T04:10:02.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.052+0000 7f4ad1857640 1 -- 172.21.15.42:0/2928945613 learned_addr learned my addr 172.21.15.42:0/2928945613 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:10:02.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.052+0000 7f4ad1857640 10 monclient(hunting): get_auth_request con 0x7f4acc1393a0 auth_method 0 2024-09-16T04:10:02.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.052+0000 7f4ad1857640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:10:02.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.052+0000 7f4ad1857640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:02.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.052+0000 7f4ad1857640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:02.059 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.052+0000 7f4ad1056640 10 monclient(hunting): get_auth_request con 0x7f4acc0a1330 auth_method 0 2024-09-16T04:10:02.059 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.052+0000 7f4ad1056640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:10:02.059 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.052+0000 7f4ad1056640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:02.059 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.052+0000 7f4ad1056640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:02.059 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.052+0000 7f4ad1857640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:10:02.059 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.052+0000 7f4ad1857640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:10:02.059 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.052+0000 7f4ad1857640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:10:02.059 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.052+0000 7f4ad1056640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:10:02.059 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.052+0000 7f4ad1056640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:10:02.059 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.052+0000 7f4ad1056640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:10:02.059 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.052+0000 7f4ad1056640 10 monclient(hunting): handle_auth_done global_id 1034235 payload 274 2024-09-16T04:10:02.059 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.052+0000 7f4ad1056640 1 -- 172.21.15.42:0/2928945613 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4acc1393a0 msgr2=0x7f4acc13b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:10:02.059 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.052+0000 7f4ad1056640 1 --2- 172.21.15.42:0/2928945613 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4acc1393a0 0x7f4acc13b790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:10:02.059 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.052+0000 7f4ad1056640 1 -- 172.21.15.42:0/2928945613 >> v1:172.21.15.154:6789/0 conn(0x7f4acc0a1710 legacy=0x7f4acc0a1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:02.060 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.052+0000 7f4ad1056640 10 monclient: _finish_hunting 0 2024-09-16T04:10:02.060 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.052+0000 7f4ad1056640 1 monclient: found mon.noname-a 2024-09-16T04:10:02.060 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.052+0000 7f4ad1056640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:10:02.060 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.052+0000 7f4ad1056640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:10:02.060 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.052+0000 7f4ad1056640 1 -- 172.21.15.42:0/2928945613 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4acc09f6c0 con 0x7f4acc0a1330 2024-09-16T04:10:02.060 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.052+0000 7f4ad1857640 1 --2- 172.21.15.42:0/2928945613 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4acc1393a0 0x7f4acc13b790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:10:02.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.056+0000 7f4ad1056640 1 --2- 172.21.15.42:0/2928945613 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4acc0a1330 0x7f4acc09f180 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f4ac80097c0 tx=0x7f4ac801c360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15053222bc28be55 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:10:02.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.056+0000 7f4ac3fff640 1 -- 172.21.15.42:0/2928945613 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ac80b0070 con 0x7f4acc0a1330 2024-09-16T04:10:02.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.056+0000 7f4ac3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:02.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.056+0000 7f4ac3fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:10:02.067 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.056+0000 7f4ac3fff640 10 monclient: dump: 2024-09-16T04:10:02.067 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:10:02.067 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:10:02.067 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:02.067 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:02.067 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:02.067 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:10:02.067 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:10:02.067 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:10:02.067 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:10:02.067 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:10:02.067 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.056+0000 7f4ac3fff640 10 monclient: _finish_auth 0 2024-09-16T04:10:02.067 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.056+0000 7f4ac3fff640 10 monclient: _check_auth_tickets 2024-09-16T04:10:02.067 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.056+0000 7f4ac3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:02.068 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.056+0000 7f4ac3fff640 1 -- 172.21.15.42:0/2928945613 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ac80ab600 con 0x7f4acc0a1330 2024-09-16T04:10:02.068 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.056+0000 7f4ac3fff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:10:02.068 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.056+0000 7f4ac3fff640 1 -- 172.21.15.42:0/2928945613 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ac80ab9d0 con 0x7f4acc0a1330 2024-09-16T04:10:02.068 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.056+0000 7f4ad32ba640 5 monclient: authenticate success, global_id 1034235 2024-09-16T04:10:02.068 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.056+0000 7f4ad32ba640 10 monclient: get_monmap_and_config success 2024-09-16T04:10:02.068 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.056+0000 7f4ad32ba640 4 set_mon_vals no callback set 2024-09-16T04:10:02.068 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.056+0000 7f4ac3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:02.068 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.056+0000 7f4ac3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:10:02.068 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.056+0000 7f4ac3fff640 10 monclient: dump: 2024-09-16T04:10:02.068 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:10:02.068 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:10:02.068 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:02.068 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:02.069 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:02.069 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:10:02.069 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:10:02.069 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:10:02.069 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:10:02.069 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:10:02.069 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.056+0000 7f4ad32ba640 10 monclient: shutdown 2024-09-16T04:10:02.069 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.056+0000 7f4ad32ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:10:02.069 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.056+0000 7f4ad32ba640 1 -- 172.21.15.42:0/2928945613 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4acc0a1330 msgr2=0x7f4acc09f180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:02.069 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.056+0000 7f4ad32ba640 1 --2- 172.21.15.42:0/2928945613 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4acc0a1330 0x7f4acc09f180 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f4ac80097c0 tx=0x7f4ac801c360 comp rx=0 tx=0).stop 2024-09-16T04:10:02.069 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.056+0000 7f4ad32ba640 1 -- 172.21.15.42:0/2928945613 shutdown_connections 2024-09-16T04:10:02.069 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.056+0000 7f4ad32ba640 1 --2- 172.21.15.42:0/2928945613 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4acc1393a0 0x7f4acc13b790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:10:02.069 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.056+0000 7f4ad32ba640 1 --2- 172.21.15.42:0/2928945613 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4acc0a1330 0x7f4acc09f180 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:10:02.069 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.056+0000 7f4ad32ba640 1 -- 172.21.15.42:0/2928945613 >> 172.21.15.42:0/2928945613 conn(0x7f4acc0a2420 msgr2=0x7f4acc0a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:10:02.070 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.056+0000 7f4ad32ba640 1 -- 172.21.15.42:0/2928945613 shutdown_connections 2024-09-16T04:10:02.070 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.056+0000 7f4ad32ba640 1 -- 172.21.15.42:0/2928945613 wait complete. 2024-09-16T04:10:02.070 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.056+0000 7f4ad32ba640 10 monclient: build_initial_monmap 2024-09-16T04:10:02.070 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.056+0000 7f4ad32ba640 10 monclient: monmap: 2024-09-16T04:10:02.070 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:10:02.070 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:10:02.070 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:10:02.070 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:10:02.070 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:10:02.070 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:10:02.070 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:10:02.070 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:10:02.070 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:10:02.070 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:10:02.071 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.056+0000 7f4ad32ba640 1 librados: starting msgr at 2024-09-16T04:10:02.071 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.056+0000 7f4ad32ba640 1 librados: starting objecter 2024-09-16T04:10:02.071 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.056+0000 7f4ad32ba640 1 Processor -- start 2024-09-16T04:10:02.071 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.056+0000 7f4ad32ba640 1 -- start start 2024-09-16T04:10:02.071 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.056+0000 7f4ad32ba640 1 librados: setting wanted keys 2024-09-16T04:10:02.071 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.056+0000 7f4ad32ba640 1 librados: calling monclient init 2024-09-16T04:10:02.071 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.056+0000 7f4ad32ba640 10 monclient: init 2024-09-16T04:10:02.071 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.056+0000 7f4ad32ba640 10 monclient: _reopen_session rank -1 2024-09-16T04:10:02.071 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.056+0000 7f4ad32ba640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:10:02.071 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.056+0000 7f4ad32ba640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4acc0a1330 0x7f4acc071770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:10:02.071 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.056+0000 7f4ad32ba640 10 monclient(hunting): picked mon.noname-b con 0x7f4acc0a1330 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:10:02.071 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.056+0000 7f4ad32ba640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4acc0a1710 0x7f4acc134660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:10:02.071 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.056+0000 7f4ad32ba640 10 monclient(hunting): picked mon.noname-a con 0x7f4acc0a1710 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:10:02.071 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.056+0000 7f4ad32ba640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4acc1393a0 0x7f4acc134ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:10:02.072 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.056+0000 7f4ad32ba640 10 monclient(hunting): picked mon.noname-c con 0x7f4acc1393a0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:10:02.072 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.056+0000 7f4ad32ba640 10 monclient(hunting): start opening mon connection 2024-09-16T04:10:02.072 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.056+0000 7f4ad32ba640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f4acc09c670 con 0x7f4acc0a1710 2024-09-16T04:10:02.072 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.056+0000 7f4ad32ba640 10 monclient(hunting): start opening mon connection 2024-09-16T04:10:02.072 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.056+0000 7f4ad32ba640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f4acc09c4f0 con 0x7f4acc0a1330 2024-09-16T04:10:02.072 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.056+0000 7f4ad32ba640 10 monclient(hunting): start opening mon connection 2024-09-16T04:10:02.072 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.056+0000 7f4ad32ba640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f4acc09c7f0 con 0x7f4acc1393a0 2024-09-16T04:10:02.072 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.056+0000 7f4ad32ba640 10 monclient(hunting): _renew_subs 2024-09-16T04:10:02.072 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.056+0000 7f4ad32ba640 10 monclient(hunting): authenticate will time out at 2649.027967s 2024-09-16T04:10:02.072 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.060+0000 7f4ad0855640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4acc0a1710 0x7f4acc134660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:10:02.072 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.060+0000 7f4ad1857640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4acc1393a0 0x7f4acc134ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:10:02.072 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.060+0000 7f4ad1857640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4acc1393a0 0x7f4acc134ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:51210/0 (socket says 172.21.15.42:51210) 2024-09-16T04:10:02.072 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.060+0000 7f4ad1857640 1 -- 172.21.15.42:0/2868561710 learned_addr learned my addr 172.21.15.42:0/2868561710 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:10:02.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.060+0000 7f4ad1857640 10 monclient(hunting): get_auth_request con 0x7f4acc1393a0 auth_method 0 2024-09-16T04:10:02.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.060+0000 7f4ad1857640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:10:02.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.060+0000 7f4ad1857640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:02.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.060+0000 7f4ad1857640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:02.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.060+0000 7f4ad0855640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4acc0a1710 0x7f4acc134660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:60130/0 (socket says 172.21.15.42:60130) 2024-09-16T04:10:02.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.060+0000 7f4ad0855640 10 monclient(hunting): get_auth_request con 0x7f4acc0a1710 auth_method 0 2024-09-16T04:10:02.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.060+0000 7f4ad0855640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:10:02.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.060+0000 7f4ad0855640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:02.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.060+0000 7f4ad0855640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:02.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.060+0000 7f4ad1857640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:10:02.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.060+0000 7f4ad1857640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:10:02.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.060+0000 7f4ad1857640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:10:02.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.060+0000 7f4ad0855640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:10:02.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.060+0000 7f4ad0855640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:10:02.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.060+0000 7f4ad0855640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:10:02.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.060+0000 7f4ad1056640 1 --2- 172.21.15.42:0/2868561710 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4acc0a1330 0x7f4acc071770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:10:02.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.060+0000 7f4ad1056640 10 monclient(hunting): get_auth_request con 0x7f4acc0a1330 auth_method 0 2024-09-16T04:10:02.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.060+0000 7f4ad1056640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:10:02.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.060+0000 7f4ad1056640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:02.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.060+0000 7f4ad1056640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:02.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.060+0000 7f4ad1857640 10 monclient(hunting): handle_auth_done global_id 1014368 payload 824 2024-09-16T04:10:02.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.060+0000 7f4ad1857640 1 -- 172.21.15.42:0/2868561710 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4acc0a1330 msgr2=0x7f4acc071770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:02.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.060+0000 7f4ad1857640 1 --2- 172.21.15.42:0/2868561710 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4acc0a1330 0x7f4acc071770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:10:02.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.060+0000 7f4ad1857640 1 -- 172.21.15.42:0/2868561710 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4acc0a1710 msgr2=0x7f4acc134660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:02.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.060+0000 7f4ad1857640 1 --2- 172.21.15.42:0/2868561710 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4acc0a1710 0x7f4acc134660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:10:02.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.060+0000 7f4ad1857640 10 monclient: _finish_hunting 0 2024-09-16T04:10:02.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.060+0000 7f4ad1857640 1 monclient: found mon.noname-c 2024-09-16T04:10:02.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.060+0000 7f4ad1857640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:10:02.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.060+0000 7f4ad1857640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T04:10:02.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.060+0000 7f4ad1857640 1 -- 172.21.15.42:0/2868561710 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4acc122d70 con 0x7f4acc1393a0 2024-09-16T04:10:02.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.060+0000 7f4ad1857640 1 --2- 172.21.15.42:0/2868561710 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4acc1393a0 0x7f4acc134ba0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f4ac400e370 tx=0x7f4ac400e830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:10:02.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.060+0000 7f4ac1ffb640 10 client.?.objecter ms_handle_connect 0x7f4acc1393a0 2024-09-16T04:10:02.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.060+0000 7f4ac1ffb640 10 client.?.objecter resend_mon_ops 2024-09-16T04:10:02.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.060+0000 7f4ac1ffb640 1 -- 172.21.15.42:0/2868561710 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ac400b2d0 con 0x7f4acc1393a0 2024-09-16T04:10:02.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.060+0000 7f4ac1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:02.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.060+0000 7f4ac1ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:10:02.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.060+0000 7f4ac1ffb640 10 monclient: dump: 2024-09-16T04:10:02.075 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:10:02.075 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:10:02.075 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:02.075 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:02.075 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:02.076 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:10:02.076 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:10:02.076 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:10:02.076 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:10:02.076 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:10:02.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.060+0000 7f4ac1ffb640 10 monclient: _finish_auth 0 2024-09-16T04:10:02.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.060+0000 7f4ac1ffb640 10 monclient: _check_auth_tickets 2024-09-16T04:10:02.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.060+0000 7f4ac1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:02.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.060+0000 7f4ac1ffb640 1 -- 172.21.15.42:0/2868561710 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ac400b470 con 0x7f4acc1393a0 2024-09-16T04:10:02.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.060+0000 7f4ad32ba640 5 monclient: authenticate success, global_id 1014368 2024-09-16T04:10:02.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.060+0000 7f4ad32ba640 10 monclient: _renew_subs 2024-09-16T04:10:02.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.060+0000 7f4ad32ba640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:10:02.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.060+0000 7f4ad32ba640 1 -- 172.21.15.42:0/2868561710 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4acc123050 con 0x7f4acc1393a0 2024-09-16T04:10:02.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.060+0000 7f4ac1ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:10:02.077 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.060+0000 7f4ac1ffb640 1 -- 172.21.15.42:0/2868561710 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ac4003bc0 con 0x7f4acc1393a0 2024-09-16T04:10:02.077 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.060+0000 7f4ac1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:02.077 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.060+0000 7f4ac1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:10:02.077 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.060+0000 7f4ac1ffb640 10 monclient: dump: 2024-09-16T04:10:02.077 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:10:02.077 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:10:02.077 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:02.077 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:02.077 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:02.077 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:10:02.077 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:10:02.077 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:10:02.077 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:10:02.077 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:10:02.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.060+0000 7f4ac37fe640 4 set_mon_vals no callback set 2024-09-16T04:10:02.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.060+0000 7f4ad32ba640 10 client.1014368.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:10:02.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.060+0000 7f4ad32ba640 10 monclient: _renew_subs 2024-09-16T04:10:02.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.060+0000 7f4ad32ba640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:10:02.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.060+0000 7f4ad32ba640 1 -- 172.21.15.42:0/2868561710 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f4acc13b530 con 0x7f4acc1393a0 2024-09-16T04:10:02.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.060+0000 7f4ad32ba640 1 librados: init done 2024-09-16T04:10:02.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.060+0000 7f4ad32ba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:02.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.060+0000 7f4ad32ba640 1 --2- 172.21.15.42:0/2868561710 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4acc120380 0x7f4acc120760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:10:02.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.060+0000 7f4ad32ba640 10 monclient(hunting): start opening mon connection 2024-09-16T04:10:02.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.060+0000 7f4ad32ba640 1 -- 172.21.15.42:0/2868561710 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f4acc1cf440 con 0x7f4acc120380 2024-09-16T04:10:02.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.060+0000 7f4ad1056640 1 --2- 172.21.15.42:0/2868561710 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4acc120380 0x7f4acc120760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:10:02.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.060+0000 7f4ad1056640 10 monclient: get_auth_request con 0x7f4acc120380 auth_method 0 2024-09-16T04:10:02.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.060+0000 7f4ad1056640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:10:02.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.060+0000 7f4ad1056640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:02.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.060+0000 7f4ad1056640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:02.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.060+0000 7f4ac1ffb640 1 -- 172.21.15.42:0/2868561710 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f4ac4009600 con 0x7f4acc1393a0 2024-09-16T04:10:02.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.060+0000 7f4ad1056640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:10:02.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.060+0000 7f4ad1056640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:10:02.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.060+0000 7f4ad1056640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:10:02.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.060+0000 7f4ad1056640 10 monclient(hunting): handle_auth_done global_id 1034241 payload 824 2024-09-16T04:10:02.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.060+0000 7f4ad1056640 1 -- 172.21.15.42:0/2868561710 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4acc09f180 con 0x7f4acc120380 2024-09-16T04:10:02.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.060+0000 7f4ac1ffb640 1 --2- 172.21.15.42:0/2868561710 >> v2:172.21.15.154:6812/254845342 conn(0x7f4ab0037810 0x7f4ab0039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:10:02.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.060+0000 7f4ac1ffb640 1 -- 172.21.15.42:0/2868561710 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(706..706 src has 451..706) ==== 5040+0+0 (secure 0 0 0) 0x7f4ac4049460 con 0x7f4acc1393a0 2024-09-16T04:10:02.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.060+0000 7f4ac1ffb640 10 client.1014368.objecter ms_dispatch 0x7f4acc000ba0 osd_map(706..706 src has 451..706) 2024-09-16T04:10:02.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.060+0000 7f4ac1ffb640 3 client.1014368.objecter handle_osd_map got epochs [706,706] > 0 2024-09-16T04:10:02.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.060+0000 7f4ac1ffb640 3 client.1014368.objecter handle_osd_map decoding full epoch 706 2024-09-16T04:10:02.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.060+0000 7f4ac1ffb640 20 client.1014368.objecter dump_active .. 0 homeless 2024-09-16T04:10:02.080 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.060+0000 7f4ac1ffb640 10 client.1014368.objecter ms_handle_connect 0x7f4acc120380 2024-09-16T04:10:02.080 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.060+0000 7f4ac1ffb640 10 client.1014368.objecter resend_mon_ops 2024-09-16T04:10:02.080 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.060+0000 7f4ad1056640 1 --2- 172.21.15.42:0/2868561710 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4acc120380 0x7f4acc120760 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f4acc073490 tx=0x7f4ac801aba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:10:02.080 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.060+0000 7f4ac1ffb640 1 -- 172.21.15.42:0/2868561710 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ac80bb070 con 0x7f4acc120380 2024-09-16T04:10:02.080 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.060+0000 7f4ac1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:02.080 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.060+0000 7f4ac1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:10:02.080 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.060+0000 7f4ac1ffb640 10 monclient: dump: 2024-09-16T04:10:02.080 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:10:02.080 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:10:02.080 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:02.080 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:02.080 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:02.080 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:10:02.080 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:10:02.081 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:10:02.081 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:10:02.081 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:10:02.081 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.060+0000 7f4ad0855640 1 --2- 172.21.15.42:0/2868561710 >> v2:172.21.15.154:6812/254845342 conn(0x7f4ab0037810 0x7f4ab0039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:10:02.081 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.060+0000 7f4ad0855640 10 monclient: get_auth_request con 0x7f4ab0037810 auth_method 0 2024-09-16T04:10:02.081 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.060+0000 7f4ac1ffb640 1 -- 172.21.15.42:0/2868561710 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f4ac80b1040 con 0x7f4acc120380 2024-09-16T04:10:02.081 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.060+0000 7f4ac1ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:02.081 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.060+0000 7f4ac1ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:10:02.081 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.060+0000 7f4ac1ffb640 1 -- 172.21.15.42:0/2868561710 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4acc120380 msgr2=0x7f4acc120760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:02.081 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.060+0000 7f4ac1ffb640 1 --2- 172.21.15.42:0/2868561710 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4acc120380 0x7f4acc120760 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f4acc073490 tx=0x7f4ac801aba0 comp rx=0 tx=0).stop 2024-09-16T04:10:02.081 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.060+0000 7f4ac1ffb640 1 -- 172.21.15.42:0/2868561710 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4acc120380 msgr2=0x7f4acc120760 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:10:02.081 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.060+0000 7f4ac1ffb640 1 --2- 172.21.15.42:0/2868561710 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4acc120380 0x7f4acc120760 secure :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f4acc073490 tx=0x7f4ac801aba0 comp rx=0 tx=0).stop 2024-09-16T04:10:02.081 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.060+0000 7f4ad0855640 1 --2- 172.21.15.42:0/2868561710 >> v2:172.21.15.154:6812/254845342 conn(0x7f4ab0037810 0x7f4ab0039cd0 secure :-1 s=READY pgs=3922 cs=0 l=1 rev1=1 crypto rx=0x7f4acc071f20 tx=0x7f4abc009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:10:02.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.060+0000 7f4ac1ffb640 10 client.1014368.objecter ms_handle_connect 0x7f4ab0037810 2024-09-16T04:10:02.083 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.068+0000 7f4ad32ba640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:10:02.083 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.068+0000 7f4ad32ba640 1 --2- 172.21.15.42:0/2868561710 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4acc120ca0 0x7f4acc06d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:10:02.083 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.068+0000 7f4ad32ba640 10 monclient(hunting): start opening mon connection 2024-09-16T04:10:02.083 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.068+0000 7f4ad32ba640 1 -- 172.21.15.42:0/2868561710 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f4acc069040 con 0x7f4acc120ca0 2024-09-16T04:10:02.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.068+0000 7f4ad1056640 1 --2- 172.21.15.42:0/2868561710 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4acc120ca0 0x7f4acc06d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:10:02.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.068+0000 7f4ad1056640 10 monclient: get_auth_request con 0x7f4acc120ca0 auth_method 0 2024-09-16T04:10:02.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.068+0000 7f4ad1056640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:10:02.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.068+0000 7f4ad1056640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:02.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.068+0000 7f4ad1056640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:02.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.068+0000 7f4ad1056640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:10:02.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.068+0000 7f4ad1056640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:10:02.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.068+0000 7f4ad1056640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:10:02.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.068+0000 7f4ad1056640 10 monclient(hunting): handle_auth_done global_id 1034244 payload 824 2024-09-16T04:10:02.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.068+0000 7f4ad1056640 1 -- 172.21.15.42:0/2868561710 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f4acc1210b0 con 0x7f4acc120ca0 2024-09-16T04:10:02.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.068+0000 7f4ad1056640 1 --2- 172.21.15.42:0/2868561710 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4acc120ca0 0x7f4acc06d900 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f4ac8000c10 tx=0x7f4ac8019100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:10:02.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.068+0000 7f4ac1ffb640 10 client.1014368.objecter ms_handle_connect 0x7f4acc120ca0 2024-09-16T04:10:02.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.068+0000 7f4ac1ffb640 10 client.1014368.objecter resend_mon_ops 2024-09-16T04:10:02.085 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.068+0000 7f4ac1ffb640 1 -- 172.21.15.42:0/2868561710 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ac80bb050 con 0x7f4acc120ca0 2024-09-16T04:10:02.085 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.068+0000 7f4ac1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:02.085 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.068+0000 7f4ac1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:10:02.085 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.068+0000 7f4ac1ffb640 10 monclient: dump: 2024-09-16T04:10:02.085 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:10:02.085 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:10:02.085 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:02.085 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:02.085 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:02.085 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:10:02.085 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:10:02.085 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:10:02.085 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:10:02.085 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:10:02.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.068+0000 7f4ac1ffb640 1 -- 172.21.15.42:0/2868561710 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4881 (secure 0 0 0) 0x7f4ac80bc040 con 0x7f4acc120ca0 2024-09-16T04:10:02.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.068+0000 7f4ac1ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:10:02.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.068+0000 7f4ac1ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:10:02.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.068+0000 7f4ac1ffb640 1 -- 172.21.15.42:0/2868561710 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4acc120ca0 msgr2=0x7f4acc06d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:02.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.068+0000 7f4ac1ffb640 1 --2- 172.21.15.42:0/2868561710 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4acc120ca0 0x7f4acc06d900 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f4ac8000c10 tx=0x7f4ac8019100 comp rx=0 tx=0).stop 2024-09-16T04:10:02.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.068+0000 7f4ac1ffb640 1 -- 172.21.15.42:0/2868561710 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4acc120ca0 msgr2=0x7f4acc06d900 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:10:02.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.068+0000 7f4ac1ffb640 1 --2- 172.21.15.42:0/2868561710 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4acc120ca0 0x7f4acc06d900 secure :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f4ac8000c10 tx=0x7f4ac8019100 comp rx=0 tx=0).stop 2024-09-16T04:10:02.086 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T04:10:02.086 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:10:02.086 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:10:02.087 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "leader", 2024-09-16T04:10:02.087 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 256, 2024-09-16T04:10:02.087 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 82003, 2024-09-16T04:10:02.087 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T04:10:02.087 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T04:10:02.087 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T04:10:02.087 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T04:10:02.087 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:10:02.087 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 2, 2024-09-16T04:10:02.087 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:10:02.087 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T04:10:02.087 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T04:10:02.087 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:10:02.087 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:10:02.088 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:10:02.088 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:10:02.088 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:10:02.088 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:10:02.088 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:10:02.088 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:10:02.088 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:10:02.088 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:10:02.088 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:10:02.088 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:10:02.088 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:10:02.088 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T04:10:02.088 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:10:02.088 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:10:02.089 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:10:02.089 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:10:02.089 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:10:02.089 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:10:02.089 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:10:02.089 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:10:02.089 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:10:02.089 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:10:02.089 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:10:02.089 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:10:02.089 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:10:02.089 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T04:10:02.089 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T04:10:02.090 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T04:10:02.090 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T04:10:02.090 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T04:10:02.090 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T04:10:02.090 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:10:02.090 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:10:02.090 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T04:10:02.090 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T04:10:02.090 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T04:10:02.090 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T04:10:02.090 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T04:10:02.090 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T04:10:02.090 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T04:10:02.091 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:10:02.091 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T04:10:02.091 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:10:02.091 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:10:02.091 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:10:02.091 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:10:02.091 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:10:02.091 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:10:02.091 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:10:02.091 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:10:02.091 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:10:02.091 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:10:02.091 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:10:02.091 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:10:02.092 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T04:10:02.092 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:10:02.092 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T04:10:02.092 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:10:02.092 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:10:02.092 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:10:02.092 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:10:02.092 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:10:02.092 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:10:02.092 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:10:02.092 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T04:10:02.092 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:10:02.092 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:10:02.092 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:10:02.093 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:10:02.093 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T04:10:02.093 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:10:02.093 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:10:02.093 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:10:02.093 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:10:02.093 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T04:10:02.093 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T04:10:02.093 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:10:02.093 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:10:02.093 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:10:02.093 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:10:02.093 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:10:02.093 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:10:02.093 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:10:02.094 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:10:02.094 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:10:02.094 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:10:02.094 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:10:02.094 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T04:10:02.094 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:10:02.094 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:10:02.094 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:10:02.094 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:10:02.094 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T04:10:02.094 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:10:02.094 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:10:02.094 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:10:02.094 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:10:02.095 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T04:10:02.095 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T04:10:02.095 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:10:02.095 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:10:02.095 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:10:02.095 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:10:02.095 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:10:02.095 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:10:02.095 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:10:02.095 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:10:02.095 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:10:02.095 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:10:02.095 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:10:02.095 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T04:10:02.096 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:10:02.096 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:10:02.096 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:10:02.096 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:10:02.096 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T04:10:02.096 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:10:02.096 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:10:02.096 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:10:02.096 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:10:02.096 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T04:10:02.096 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T04:10:02.096 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:10:02.096 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:10:02.096 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:10:02.097 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:10:02.097 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:10:02.097 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:10:02.097 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T04:10:02.097 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T04:10:02.097 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:10:02.097 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:10:02.097 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:10:02.097 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:10:02.097 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:10:02.097 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:10:02.097 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T04:10:02.097 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:10:02.098 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:10:02.098 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:10:02.098 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 2 2024-09-16T04:10:02.098 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:10:02.098 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:10:02.098 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T04:10:02.098 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:10:02.098 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:10:02.098 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:10:02.098 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 4 2024-09-16T04:10:02.098 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:10:02.098 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:10:02.098 INFO:teuthology.orchestra.run.smithi042.stdout: "mgr": [ 2024-09-16T04:10:02.098 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:10:02.099 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:10:02.099 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:10:02.099 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:10:02.099 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:10:02.099 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:10:02.099 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:10:02.099 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T04:10:02.099 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T04:10:02.101 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.068+0000 7f4ad32ba640 10 librados: watch_flush enter 2024-09-16T04:10:02.101 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.068+0000 7f4ad32ba640 10 librados: watch_flush exit 2024-09-16T04:10:02.101 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.068+0000 7f4ad32ba640 20 client.1014368.objecter shutdown clearing up homeless session... 2024-09-16T04:10:02.101 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.068+0000 7f4ad32ba640 10 client.1014368.objecter successfully canceled tick 2024-09-16T04:10:02.101 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.068+0000 7f4ad32ba640 1 -- 172.21.15.42:0/2868561710 >> v2:172.21.15.154:6812/254845342 conn(0x7f4ab0037810 msgr2=0x7f4ab0039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:02.101 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.068+0000 7f4ad32ba640 1 --2- 172.21.15.42:0/2868561710 >> v2:172.21.15.154:6812/254845342 conn(0x7f4ab0037810 0x7f4ab0039cd0 secure :-1 s=READY pgs=3922 cs=0 l=1 rev1=1 crypto rx=0x7f4acc071f20 tx=0x7f4abc009040 comp rx=0 tx=0).stop 2024-09-16T04:10:02.101 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.068+0000 7f4ad32ba640 10 monclient: shutdown 2024-09-16T04:10:02.101 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.068+0000 7f4ad32ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:10:02.101 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.068+0000 7f4ad32ba640 1 -- 172.21.15.42:0/2868561710 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4acc1393a0 msgr2=0x7f4acc134ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:02.101 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.068+0000 7f4ad1857640 1 -- 172.21.15.42:0/2868561710 reap_dead start 2024-09-16T04:10:02.101 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.068+0000 7f4ad32ba640 1 --2- 172.21.15.42:0/2868561710 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4acc1393a0 0x7f4acc134ba0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f4ac400e370 tx=0x7f4ac400e830 comp rx=0 tx=0).stop 2024-09-16T04:10:02.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.068+0000 7f4ad32ba640 1 -- 172.21.15.42:0/2868561710 shutdown_connections 2024-09-16T04:10:02.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.068+0000 7f4ad32ba640 1 --2- 172.21.15.42:0/2868561710 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4acc1393a0 0x7f4acc134ba0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:10:02.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.068+0000 7f4ad32ba640 1 -- 172.21.15.42:0/2868561710 >> 172.21.15.42:0/2868561710 conn(0x7f4acc0a2420 msgr2=0x7f4acc0a46b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:10:02.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.068+0000 7f4ad32ba640 1 -- 172.21.15.42:0/2868561710 shutdown_connections 2024-09-16T04:10:02.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.068+0000 7f4ad32ba640 1 -- 172.21.15.42:0/2868561710 wait complete. 2024-09-16T04:10:02.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.068+0000 7f4ad32ba640 1 librados: shutdown 2024-09-16T04:10:02.102 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:10:02.102 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T04:10:02.102 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T04:10:02.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.216+0000 7f84ef422640 10 monclient: get_monmap_and_config 2024-09-16T04:10:02.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.216+0000 7f84ef422640 10 monclient: build_initial_monmap 2024-09-16T04:10:02.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.216+0000 7f84ef422640 10 monclient: monmap: 2024-09-16T04:10:02.219 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:10:02.219 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:10:02.219 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:10:02.217067+0000 2024-09-16T04:10:02.219 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:10:02.217067+0000 2024-09-16T04:10:02.219 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:10:02.219 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:10:02.219 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:10:02.219 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:10:02.219 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:10:02.219 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:10:02.219 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:10:02.219 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:10:02.222 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.216+0000 7f84ef422640 1 Processor -- start 2024-09-16T04:10:02.222 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.216+0000 7f84ef422640 1 -- start start 2024-09-16T04:10:02.222 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.216+0000 7f84ef422640 10 monclient: init 2024-09-16T04:10:02.222 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.216+0000 7f84ef422640 10 monclient: _reopen_session rank -1 2024-09-16T04:10:02.222 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.216+0000 7f84ef422640 10 monclient: _add_conns ranks=[4,2,1,3,0] 2024-09-16T04:10:02.222 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.216+0000 7f84ef422640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f84e80a39b0 0x7f84e8133b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:10:02.222 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.216+0000 7f84ef422640 10 monclient(hunting): picked mon.noname-c con 0x7f84e80a39b0 addr v2:172.21.15.154:3300/0 2024-09-16T04:10:02.222 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.216+0000 7f84ef422640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f84e80a3d90 0x7f84e8137000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:10:02.223 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.216+0000 7f84ef422640 10 monclient(hunting): picked mon.noname-b con 0x7f84e80a3d90 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:10:02.223 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.216+0000 7f84ef422640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f84e8137540 addr v1:172.21.15.154:6789/0 2024-09-16T04:10:02.223 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.216+0000 7f84ef422640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f84e809c140 con 0x7f84e8137540 2024-09-16T04:10:02.223 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.216+0000 7f84ef422640 10 monclient(hunting): start opening mon connection 2024-09-16T04:10:02.223 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.216+0000 7f84ef422640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f84e809c420 con 0x7f84e80a39b0 2024-09-16T04:10:02.223 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.216+0000 7f84ef422640 10 monclient(hunting): start opening mon connection 2024-09-16T04:10:02.223 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.216+0000 7f84ef422640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f84e806dd60 con 0x7f84e80a3d90 2024-09-16T04:10:02.223 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.216+0000 7f84ef422640 10 monclient(hunting): _renew_subs 2024-09-16T04:10:02.223 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.216+0000 7f84ef422640 10 monclient(hunting): authenticate will time out at 2649.187181s 2024-09-16T04:10:02.223 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.216+0000 7f84ec9bd640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f84e80a3d90 0x7f84e8137000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:10:02.223 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.216+0000 7f84ec9bd640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f84e80a3d90 0x7f84e8137000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:51218/0 (socket says 172.21.15.42:51218) 2024-09-16T04:10:02.223 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.216+0000 7f84ec9bd640 1 -- 172.21.15.42:0/3610927193 learned_addr learned my addr 172.21.15.42:0/3610927193 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:10:02.223 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.216+0000 7f84ec9bd640 10 monclient(hunting): get_auth_request con 0x7f84e80a3d90 auth_method 0 2024-09-16T04:10:02.224 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.216+0000 7f84ec9bd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:10:02.224 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.216+0000 7f84ec9bd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:02.224 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.216+0000 7f84ec9bd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:02.224 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.216+0000 7f84ec9bd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:10:02.224 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.216+0000 7f84ec9bd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:10:02.224 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.216+0000 7f84ed1be640 1 --2- 172.21.15.42:0/3610927193 >> v2:172.21.15.154:3300/0 conn(0x7f84e80a39b0 0x7f84e8133b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:10:02.224 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.216+0000 7f84ec9bd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:10:02.224 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.216+0000 7f84ed1be640 10 monclient(hunting): get_auth_request con 0x7f84e80a39b0 auth_method 0 2024-09-16T04:10:02.224 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.216+0000 7f84ed1be640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:10:02.224 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.216+0000 7f84ed1be640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:02.224 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.216+0000 7f84ed1be640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:02.224 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.216+0000 7f84ec9bd640 10 monclient(hunting): handle_auth_done global_id 1014371 payload 274 2024-09-16T04:10:02.224 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.216+0000 7f84ec9bd640 1 -- 172.21.15.42:0/3610927193 >> v2:172.21.15.154:3300/0 conn(0x7f84e80a39b0 msgr2=0x7f84e8133b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:10:02.224 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.216+0000 7f84ec9bd640 1 --2- 172.21.15.42:0/3610927193 >> v2:172.21.15.154:3300/0 conn(0x7f84e80a39b0 0x7f84e8133b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:10:02.225 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.216+0000 7f84ec9bd640 1 -- 172.21.15.42:0/3610927193 >> v1:172.21.15.154:6789/0 conn(0x7f84e8137540 legacy=0x7f84e813e390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:02.225 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.216+0000 7f84ec9bd640 10 monclient: _finish_hunting 0 2024-09-16T04:10:02.225 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.216+0000 7f84ec9bd640 1 monclient: found mon.noname-b 2024-09-16T04:10:02.225 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.216+0000 7f84ec9bd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:10:02.225 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.216+0000 7f84ec9bd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T04:10:02.225 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.216+0000 7f84ec9bd640 1 -- 172.21.15.42:0/3610927193 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84e80a5e20 con 0x7f84e80a3d90 2024-09-16T04:10:02.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.220+0000 7f84ec9bd640 1 --2- 172.21.15.42:0/3610927193 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f84e80a3d90 0x7f84e8137000 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f84d8009870 tx=0x7f84d801c420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5da12f2d0bbd524b server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:10:02.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.220+0000 7f84dffff640 1 -- 172.21.15.42:0/3610927193 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84d80b0070 con 0x7f84e80a3d90 2024-09-16T04:10:02.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.220+0000 7f84dffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:02.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.220+0000 7f84dffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:10:02.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.220+0000 7f84dffff640 10 monclient: dump: 2024-09-16T04:10:02.231 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:10:02.231 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:10:02.231 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:02.231 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:02.232 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:02.232 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:10:02.232 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:10:02.232 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:10:02.232 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:10:02.232 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:10:02.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.220+0000 7f84dffff640 10 monclient: _finish_auth 0 2024-09-16T04:10:02.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.220+0000 7f84dffff640 10 monclient: _check_auth_tickets 2024-09-16T04:10:02.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.220+0000 7f84dffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:02.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.220+0000 7f84dffff640 1 -- 172.21.15.42:0/3610927193 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f84d80ab600 con 0x7f84e80a3d90 2024-09-16T04:10:02.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.220+0000 7f84dffff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:10:02.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.220+0000 7f84dffff640 1 -- 172.21.15.42:0/3610927193 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84d80ab9d0 con 0x7f84e80a3d90 2024-09-16T04:10:02.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.220+0000 7f84ef422640 5 monclient: authenticate success, global_id 1014371 2024-09-16T04:10:02.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.220+0000 7f84ef422640 10 monclient: get_monmap_and_config success 2024-09-16T04:10:02.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.220+0000 7f84ef422640 4 set_mon_vals no callback set 2024-09-16T04:10:02.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.220+0000 7f84dffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:02.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.220+0000 7f84dffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:10:02.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.220+0000 7f84dffff640 10 monclient: dump: 2024-09-16T04:10:02.234 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:10:02.235 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:10:02.235 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:02.235 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:02.235 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:02.235 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:10:02.235 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:10:02.235 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:10:02.235 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:10:02.235 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:10:02.235 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.220+0000 7f84ef422640 10 monclient: shutdown 2024-09-16T04:10:02.235 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.220+0000 7f84ef422640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:10:02.235 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.220+0000 7f84ef422640 1 -- 172.21.15.42:0/3610927193 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f84e80a3d90 msgr2=0x7f84e8137000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:02.235 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.220+0000 7f84ef422640 1 --2- 172.21.15.42:0/3610927193 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f84e80a3d90 0x7f84e8137000 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f84d8009870 tx=0x7f84d801c420 comp rx=0 tx=0).stop 2024-09-16T04:10:02.235 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.220+0000 7f84ef422640 1 -- 172.21.15.42:0/3610927193 shutdown_connections 2024-09-16T04:10:02.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.220+0000 7f84ef422640 1 --2- 172.21.15.42:0/3610927193 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f84e80a3d90 0x7f84e8137000 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:10:02.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.220+0000 7f84ef422640 1 --2- 172.21.15.42:0/3610927193 >> v2:172.21.15.154:3300/0 conn(0x7f84e80a39b0 0x7f84e8133b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:10:02.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.220+0000 7f84ef422640 1 -- 172.21.15.42:0/3610927193 >> 172.21.15.42:0/3610927193 conn(0x7f84e80a9fb0 msgr2=0x7f84e80aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:10:02.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.220+0000 7f84ef422640 1 -- 172.21.15.42:0/3610927193 shutdown_connections 2024-09-16T04:10:02.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.220+0000 7f84ef422640 1 -- 172.21.15.42:0/3610927193 wait complete. 2024-09-16T04:10:02.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.220+0000 7f84ef422640 10 monclient: build_initial_monmap 2024-09-16T04:10:02.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.220+0000 7f84ef422640 10 monclient: monmap: 2024-09-16T04:10:02.236 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:10:02.236 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:10:02.236 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:10:02.236 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:10:02.236 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:10:02.236 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:10:02.237 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:10:02.237 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:10:02.237 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:10:02.237 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:10:02.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.220+0000 7f84ef422640 1 librados: starting msgr at 2024-09-16T04:10:02.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.220+0000 7f84ef422640 1 librados: starting objecter 2024-09-16T04:10:02.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.220+0000 7f84ef422640 1 Processor -- start 2024-09-16T04:10:02.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.220+0000 7f84ef422640 1 -- start start 2024-09-16T04:10:02.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.220+0000 7f84ef422640 1 librados: setting wanted keys 2024-09-16T04:10:02.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.220+0000 7f84ef422640 1 librados: calling monclient init 2024-09-16T04:10:02.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.220+0000 7f84ef422640 10 monclient: init 2024-09-16T04:10:02.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.224+0000 7f84ef422640 10 monclient: _reopen_session rank -1 2024-09-16T04:10:02.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.224+0000 7f84ef422640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:10:02.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.224+0000 7f84ef422640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f84e80a39b0 0x7f84e81cfb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:10:02.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.224+0000 7f84ef422640 10 monclient(hunting): picked mon.noname-b con 0x7f84e80a39b0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:10:02.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.224+0000 7f84ef422640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f84e80a3d90 0x7f84e80a2f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:10:02.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.224+0000 7f84ef422640 10 monclient(hunting): picked mon.noname-c con 0x7f84e80a3d90 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:10:02.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.224+0000 7f84ef422640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f84e8137540 0x7f84e80a3470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:10:02.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.224+0000 7f84ef422640 10 monclient(hunting): picked mon.noname-a con 0x7f84e8137540 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:10:02.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.224+0000 7f84ef422640 10 monclient(hunting): start opening mon connection 2024-09-16T04:10:02.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.224+0000 7f84ef422640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f84e81424e0 con 0x7f84e8137540 2024-09-16T04:10:02.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.224+0000 7f84ef422640 10 monclient(hunting): start opening mon connection 2024-09-16T04:10:02.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.224+0000 7f84ef422640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f84e8142660 con 0x7f84e80a39b0 2024-09-16T04:10:02.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.224+0000 7f84ef422640 10 monclient(hunting): start opening mon connection 2024-09-16T04:10:02.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.224+0000 7f84ef422640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f84e81427e0 con 0x7f84e80a3d90 2024-09-16T04:10:02.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.224+0000 7f84ef422640 10 monclient(hunting): _renew_subs 2024-09-16T04:10:02.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.224+0000 7f84ef422640 10 monclient(hunting): authenticate will time out at 2649.192114s 2024-09-16T04:10:02.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.224+0000 7f84ec9bd640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f84e80a3d90 0x7f84e80a2f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:10:02.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.224+0000 7f84ed9bf640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f84e8137540 0x7f84e80a3470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:10:02.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.224+0000 7f84ec9bd640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f84e80a3d90 0x7f84e80a2f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:51224/0 (socket says 172.21.15.42:51224) 2024-09-16T04:10:02.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.224+0000 7f84ec9bd640 1 -- 172.21.15.42:0/2195389854 learned_addr learned my addr 172.21.15.42:0/2195389854 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:10:02.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.224+0000 7f84ec9bd640 10 monclient(hunting): get_auth_request con 0x7f84e80a3d90 auth_method 0 2024-09-16T04:10:02.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.224+0000 7f84ec9bd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:10:02.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.224+0000 7f84ec9bd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:02.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.224+0000 7f84ec9bd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:02.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.224+0000 7f84ed9bf640 10 monclient(hunting): get_auth_request con 0x7f84e8137540 auth_method 0 2024-09-16T04:10:02.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.224+0000 7f84ed9bf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:10:02.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.224+0000 7f84ed9bf640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:02.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.224+0000 7f84ed9bf640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:02.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.224+0000 7f84ec9bd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:10:02.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.224+0000 7f84ec9bd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:10:02.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.224+0000 7f84ec9bd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:10:02.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.224+0000 7f84ed9bf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:10:02.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.224+0000 7f84ed9bf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:10:02.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.224+0000 7f84ed9bf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:10:02.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.224+0000 7f84ed1be640 1 --2- 172.21.15.42:0/2195389854 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f84e80a39b0 0x7f84e81cfb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:10:02.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.224+0000 7f84ed1be640 10 monclient(hunting): get_auth_request con 0x7f84e80a39b0 auth_method 0 2024-09-16T04:10:02.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.224+0000 7f84ed1be640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:10:02.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.224+0000 7f84ed1be640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:02.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.224+0000 7f84ed1be640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:02.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.224+0000 7f84ec9bd640 10 monclient(hunting): handle_auth_done global_id 1014374 payload 824 2024-09-16T04:10:02.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.224+0000 7f84ec9bd640 1 -- 172.21.15.42:0/2195389854 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f84e80a39b0 msgr2=0x7f84e81cfb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:02.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.224+0000 7f84ec9bd640 1 --2- 172.21.15.42:0/2195389854 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f84e80a39b0 0x7f84e81cfb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:10:02.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.224+0000 7f84ec9bd640 1 -- 172.21.15.42:0/2195389854 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f84e8137540 msgr2=0x7f84e80a3470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:02.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.224+0000 7f84ec9bd640 1 --2- 172.21.15.42:0/2195389854 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f84e8137540 0x7f84e80a3470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:10:02.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.224+0000 7f84ec9bd640 10 monclient: _finish_hunting 0 2024-09-16T04:10:02.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.224+0000 7f84ec9bd640 1 monclient: found mon.noname-c 2024-09-16T04:10:02.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.224+0000 7f84ec9bd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:10:02.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.224+0000 7f84ec9bd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T04:10:02.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.224+0000 7f84ec9bd640 1 -- 172.21.15.42:0/2195389854 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84e809e9d0 con 0x7f84e80a3d90 2024-09-16T04:10:02.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.224+0000 7f84ec9bd640 1 --2- 172.21.15.42:0/2195389854 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f84e80a3d90 0x7f84e80a2f30 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f84d801aad0 tx=0x7f84d80191c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:10:02.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.224+0000 7f84ddffb640 10 client.?.objecter ms_handle_connect 0x7f84e80a3d90 2024-09-16T04:10:02.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.224+0000 7f84ddffb640 10 client.?.objecter resend_mon_ops 2024-09-16T04:10:02.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.224+0000 7f84ddffb640 1 -- 172.21.15.42:0/2195389854 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84d80b0040 con 0x7f84e80a3d90 2024-09-16T04:10:02.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.224+0000 7f84ddffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:02.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.224+0000 7f84ddffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:10:02.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.224+0000 7f84ddffb640 10 monclient: dump: 2024-09-16T04:10:02.241 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:10:02.241 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:10:02.241 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:02.242 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:02.242 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:02.242 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:10:02.242 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:10:02.242 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:10:02.242 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:10:02.242 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:10:02.242 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.224+0000 7f84ddffb640 10 monclient: _finish_auth 0 2024-09-16T04:10:02.242 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.224+0000 7f84ddffb640 10 monclient: _check_auth_tickets 2024-09-16T04:10:02.242 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.224+0000 7f84ddffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:02.242 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.224+0000 7f84ef422640 5 monclient: authenticate success, global_id 1014374 2024-09-16T04:10:02.242 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.224+0000 7f84ef422640 10 monclient: _renew_subs 2024-09-16T04:10:02.242 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.224+0000 7f84ddffb640 1 -- 172.21.15.42:0/2195389854 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f84d80b4420 con 0x7f84e80a3d90 2024-09-16T04:10:02.242 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.224+0000 7f84ef422640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:10:02.243 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.224+0000 7f84ef422640 1 -- 172.21.15.42:0/2195389854 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84e81d0090 con 0x7f84e80a3d90 2024-09-16T04:10:02.243 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.224+0000 7f84ddffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:10:02.243 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.224+0000 7f84ddffb640 1 -- 172.21.15.42:0/2195389854 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84d80b4810 con 0x7f84e80a3d90 2024-09-16T04:10:02.243 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.224+0000 7f84ddffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:02.243 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.224+0000 7f84ddffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:10:02.243 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.224+0000 7f84ddffb640 10 monclient: dump: 2024-09-16T04:10:02.243 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:10:02.243 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:10:02.243 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:02.243 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:02.243 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:02.243 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:10:02.243 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:10:02.243 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:10:02.243 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:10:02.244 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:10:02.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.224+0000 7f84df7fe640 4 set_mon_vals no callback set 2024-09-16T04:10:02.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.224+0000 7f84ef422640 10 client.1014374.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:10:02.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.224+0000 7f84ef422640 10 monclient: _renew_subs 2024-09-16T04:10:02.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.224+0000 7f84ef422640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:10:02.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.224+0000 7f84ef422640 1 -- 172.21.15.42:0/2195389854 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f84e8133260 con 0x7f84e80a3d90 2024-09-16T04:10:02.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.224+0000 7f84ef422640 1 librados: init done 2024-09-16T04:10:02.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.224+0000 7f84ef422640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:02.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.224+0000 7f84ef422640 1 --2- 172.21.15.42:0/2195389854 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f84e813d2c0 0x7f84e813d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:10:02.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.224+0000 7f84ef422640 10 monclient(hunting): start opening mon connection 2024-09-16T04:10:02.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.224+0000 7f84ef422640 1 -- 172.21.15.42:0/2195389854 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f84e81342d0 con 0x7f84e813d2c0 2024-09-16T04:10:02.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.224+0000 7f84ddffb640 1 -- 172.21.15.42:0/2195389854 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f84d80b49b0 con 0x7f84e80a3d90 2024-09-16T04:10:02.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.224+0000 7f84ed1be640 1 --2- 172.21.15.42:0/2195389854 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f84e813d2c0 0x7f84e813d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:10:02.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.224+0000 7f84ed1be640 10 monclient: get_auth_request con 0x7f84e813d2c0 auth_method 0 2024-09-16T04:10:02.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.224+0000 7f84ed1be640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:10:02.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.224+0000 7f84ed1be640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:02.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.224+0000 7f84ed1be640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:02.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.224+0000 7f84ed1be640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:10:02.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.224+0000 7f84ed1be640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:10:02.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.224+0000 7f84ddffb640 1 --2- 172.21.15.42:0/2195389854 >> v2:172.21.15.154:6812/254845342 conn(0x7f84cc037810 0x7f84cc039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:10:02.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.224+0000 7f84ed1be640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:10:02.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.224+0000 7f84ddffb640 1 -- 172.21.15.42:0/2195389854 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(706..706 src has 451..706) ==== 5040+0+0 (secure 0 0 0) 0x7f84d80e2830 con 0x7f84e80a3d90 2024-09-16T04:10:02.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.224+0000 7f84ddffb640 10 client.1014374.objecter ms_dispatch 0x7f84e8000ba0 osd_map(706..706 src has 451..706) 2024-09-16T04:10:02.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.224+0000 7f84ddffb640 3 client.1014374.objecter handle_osd_map got epochs [706,706] > 0 2024-09-16T04:10:02.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.224+0000 7f84ddffb640 3 client.1014374.objecter handle_osd_map decoding full epoch 706 2024-09-16T04:10:02.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.224+0000 7f84ed1be640 10 monclient(hunting): handle_auth_done global_id 1014377 payload 824 2024-09-16T04:10:02.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.224+0000 7f84ddffb640 20 client.1014374.objecter dump_active .. 0 homeless 2024-09-16T04:10:02.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.224+0000 7f84ed1be640 1 -- 172.21.15.42:0/2195389854 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f84e8137090 con 0x7f84e813d2c0 2024-09-16T04:10:02.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.224+0000 7f84ddffb640 10 client.1014374.objecter ms_handle_connect 0x7f84e813d2c0 2024-09-16T04:10:02.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.224+0000 7f84ddffb640 10 client.1014374.objecter resend_mon_ops 2024-09-16T04:10:02.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.224+0000 7f84ed1be640 1 --2- 172.21.15.42:0/2195389854 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f84e813d2c0 0x7f84e813d6a0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f84e40045c0 tx=0x7f84e4007700 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:10:02.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.224+0000 7f84ed9bf640 1 --2- 172.21.15.42:0/2195389854 >> v2:172.21.15.154:6812/254845342 conn(0x7f84cc037810 0x7f84cc039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:10:02.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.224+0000 7f84ed9bf640 10 monclient: get_auth_request con 0x7f84cc037810 auth_method 0 2024-09-16T04:10:02.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.224+0000 7f84ddffb640 1 -- 172.21.15.42:0/2195389854 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84e4014610 con 0x7f84e813d2c0 2024-09-16T04:10:02.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.224+0000 7f84ddffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:02.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.224+0000 7f84ddffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:10:02.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.224+0000 7f84ddffb640 10 monclient: dump: 2024-09-16T04:10:02.246 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:10:02.246 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:10:02.246 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:02.247 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:02.247 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:02.247 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:10:02.247 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:10:02.247 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:10:02.247 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:10:02.247 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:10:02.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.224+0000 7f84ddffb640 1 -- 172.21.15.42:0/2195389854 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f84e4019070 con 0x7f84e813d2c0 2024-09-16T04:10:02.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.224+0000 7f84ddffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:02.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.224+0000 7f84ddffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:10:02.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.224+0000 7f84ddffb640 1 -- 172.21.15.42:0/2195389854 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f84e813d2c0 msgr2=0x7f84e813d6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:02.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.224+0000 7f84ddffb640 1 --2- 172.21.15.42:0/2195389854 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f84e813d2c0 0x7f84e813d6a0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f84e40045c0 tx=0x7f84e4007700 comp rx=0 tx=0).stop 2024-09-16T04:10:02.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.224+0000 7f84ddffb640 1 -- 172.21.15.42:0/2195389854 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f84e813d2c0 msgr2=0x7f84e813d6a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:10:02.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.224+0000 7f84ddffb640 1 --2- 172.21.15.42:0/2195389854 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f84e813d2c0 0x7f84e813d6a0 secure :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f84e40045c0 tx=0x7f84e4007700 comp rx=0 tx=0).stop 2024-09-16T04:10:02.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.224+0000 7f84ed9bf640 1 --2- 172.21.15.42:0/2195389854 >> v2:172.21.15.154:6812/254845342 conn(0x7f84cc037810 0x7f84cc039cd0 secure :-1 s=READY pgs=3923 cs=0 l=1 rev1=1 crypto rx=0x7f84e0007fd0 tx=0x7f84e0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:10:02.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.224+0000 7f84ddffb640 10 client.1014374.objecter ms_handle_connect 0x7f84cc037810 2024-09-16T04:10:02.249 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.232+0000 7f84ef422640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:10:02.250 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.232+0000 7f84ef422640 1 --2- 172.21.15.42:0/2195389854 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f84e813dbe0 0x7f84e8139140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:10:02.250 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.232+0000 7f84ef422640 10 monclient(hunting): start opening mon connection 2024-09-16T04:10:02.250 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.232+0000 7f84ef422640 1 -- 172.21.15.42:0/2195389854 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f84e8133b80 con 0x7f84e813dbe0 2024-09-16T04:10:02.250 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.232+0000 7f84ed1be640 1 --2- 172.21.15.42:0/2195389854 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f84e813dbe0 0x7f84e8139140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:10:02.250 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.232+0000 7f84ed1be640 10 monclient: get_auth_request con 0x7f84e813dbe0 auth_method 0 2024-09-16T04:10:02.250 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.232+0000 7f84ed1be640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:10:02.250 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.232+0000 7f84ed1be640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:02.250 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.232+0000 7f84ed1be640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:02.250 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.232+0000 7f84ed1be640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:10:02.250 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.232+0000 7f84ed1be640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:10:02.250 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.232+0000 7f84ed1be640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:10:02.250 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.232+0000 7f84ed1be640 10 monclient(hunting): handle_auth_done global_id 1014380 payload 824 2024-09-16T04:10:02.250 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.232+0000 7f84ed1be640 1 -- 172.21.15.42:0/2195389854 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f84e809d3d0 con 0x7f84e813dbe0 2024-09-16T04:10:02.250 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.232+0000 7f84ed1be640 1 --2- 172.21.15.42:0/2195389854 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f84e813dbe0 0x7f84e8139140 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f84e40045c0 tx=0x7f84e4004a50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:10:02.251 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.232+0000 7f84ddffb640 10 client.1014374.objecter ms_handle_connect 0x7f84e813dbe0 2024-09-16T04:10:02.251 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.232+0000 7f84ddffb640 10 client.1014374.objecter resend_mon_ops 2024-09-16T04:10:02.251 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.232+0000 7f84ddffb640 1 -- 172.21.15.42:0/2195389854 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84e4014810 con 0x7f84e813dbe0 2024-09-16T04:10:02.251 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.232+0000 7f84ddffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:02.251 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.232+0000 7f84ddffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:10:02.251 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.232+0000 7f84ddffb640 10 monclient: dump: 2024-09-16T04:10:02.251 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:10:02.251 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:10:02.251 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:02.251 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:02.251 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:02.251 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:10:02.251 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:10:02.252 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:10:02.252 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:10:02.252 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:10:02.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.232+0000 7f84ddffb640 1 -- 172.21.15.42:0/2195389854 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4713 (secure 0 0 0) 0x7f84e4014810 con 0x7f84e813dbe0 2024-09-16T04:10:02.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.232+0000 7f84ddffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:10:02.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.232+0000 7f84ddffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:10:02.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.232+0000 7f84ddffb640 1 -- 172.21.15.42:0/2195389854 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f84e813dbe0 msgr2=0x7f84e8139140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:02.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.232+0000 7f84ddffb640 1 --2- 172.21.15.42:0/2195389854 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f84e813dbe0 0x7f84e8139140 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f84e40045c0 tx=0x7f84e4004a50 comp rx=0 tx=0).stop 2024-09-16T04:10:02.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.232+0000 7f84ddffb640 1 -- 172.21.15.42:0/2195389854 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f84e813dbe0 msgr2=0x7f84e8139140 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:10:02.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.232+0000 7f84ddffb640 1 --2- 172.21.15.42:0/2195389854 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f84e813dbe0 0x7f84e8139140 secure :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f84e40045c0 tx=0x7f84e4004a50 comp rx=0 tx=0).stop 2024-09-16T04:10:02.252 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T04:10:02.253 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:10:02.253 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:10:02.253 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T04:10:02.253 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 256, 2024-09-16T04:10:02.253 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 197434, 2024-09-16T04:10:02.256 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T04:10:02.256 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T04:10:02.256 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T04:10:02.256 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T04:10:02.256 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:10:02.256 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 2, 2024-09-16T04:10:02.256 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:10:02.256 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T04:10:02.256 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T04:10:02.256 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:10:02.256 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:10:02.256 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:10:02.257 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:10:02.257 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:10:02.257 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:10:02.257 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:10:02.257 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:10:02.257 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:10:02.257 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:10:02.257 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:10:02.257 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:10:02.257 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:10:02.257 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T04:10:02.257 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:10:02.257 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:10:02.257 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:10:02.258 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:10:02.258 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:10:02.258 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:10:02.258 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:10:02.258 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:10:02.258 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:10:02.258 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:10:02.258 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:10:02.258 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:10:02.258 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:10:02.258 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T04:10:02.258 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T04:10:02.258 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T04:10:02.259 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T04:10:02.259 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T04:10:02.259 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T04:10:02.259 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:10:02.259 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:10:02.259 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T04:10:02.259 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T04:10:02.259 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T04:10:02.259 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T04:10:02.259 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T04:10:02.259 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T04:10:02.259 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T04:10:02.259 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:10:02.259 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T04:10:02.260 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:10:02.260 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:10:02.260 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:10:02.260 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:10:02.260 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:10:02.260 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:10:02.260 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:10:02.260 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:10:02.260 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:10:02.260 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:10:02.260 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:10:02.260 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:10:02.260 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T04:10:02.261 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:10:02.261 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T04:10:02.261 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:10:02.261 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:10:02.261 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:10:02.261 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:10:02.261 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:10:02.261 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:10:02.261 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:10:02.261 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T04:10:02.261 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:10:02.261 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:10:02.262 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:10:02.262 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:10:02.262 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T04:10:02.262 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:10:02.262 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:10:02.262 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:10:02.262 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:10:02.262 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T04:10:02.262 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T04:10:02.262 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:10:02.262 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:10:02.262 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:10:02.262 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:10:02.263 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:10:02.263 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:10:02.263 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:10:02.263 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:10:02.263 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:10:02.263 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:10:02.263 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:10:02.263 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T04:10:02.263 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:10:02.263 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:10:02.263 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:10:02.263 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:10:02.263 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T04:10:02.263 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:10:02.264 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:10:02.264 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:10:02.264 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:10:02.264 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T04:10:02.264 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T04:10:02.264 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:10:02.264 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:10:02.264 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:10:02.264 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:10:02.264 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:10:02.264 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:10:02.264 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:10:02.264 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:10:02.264 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:10:02.264 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:10:02.265 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:10:02.265 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T04:10:02.265 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:10:02.265 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:10:02.265 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:10:02.265 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:10:02.265 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T04:10:02.265 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:10:02.265 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:10:02.265 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:10:02.265 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:10:02.265 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T04:10:02.265 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T04:10:02.266 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:10:02.266 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:10:02.266 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:10:02.266 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:10:02.266 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:10:02.266 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:10:02.266 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T04:10:02.266 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T04:10:02.266 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:10:02.266 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:10:02.266 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:10:02.266 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:10:02.266 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:10:02.267 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:10:02.267 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T04:10:02.267 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:10:02.267 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:10:02.267 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:10:02.267 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 4 2024-09-16T04:10:02.267 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:10:02.267 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:10:02.267 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T04:10:02.267 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:10:02.267 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:10:02.267 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:10:02.267 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 4 2024-09-16T04:10:02.268 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:10:02.268 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:10:02.268 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:10:02.268 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T04:10:02.268 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T04:10:02.269 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.232+0000 7f84ef422640 10 librados: watch_flush enter 2024-09-16T04:10:02.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.232+0000 7f84ef422640 10 librados: watch_flush exit 2024-09-16T04:10:02.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.232+0000 7f84ef422640 20 client.1014374.objecter shutdown clearing up homeless session... 2024-09-16T04:10:02.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.232+0000 7f84ef422640 10 client.1014374.objecter successfully canceled tick 2024-09-16T04:10:02.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.232+0000 7f84ef422640 1 -- 172.21.15.42:0/2195389854 >> v2:172.21.15.154:6812/254845342 conn(0x7f84cc037810 msgr2=0x7f84cc039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:02.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.232+0000 7f84ef422640 1 --2- 172.21.15.42:0/2195389854 >> v2:172.21.15.154:6812/254845342 conn(0x7f84cc037810 0x7f84cc039cd0 secure :-1 s=READY pgs=3923 cs=0 l=1 rev1=1 crypto rx=0x7f84e0007fd0 tx=0x7f84e0009040 comp rx=0 tx=0).stop 2024-09-16T04:10:02.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.232+0000 7f84ef422640 10 monclient: shutdown 2024-09-16T04:10:02.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.232+0000 7f84ef422640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:10:02.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.232+0000 7f84ef422640 1 -- 172.21.15.42:0/2195389854 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f84e80a3d90 msgr2=0x7f84e80a2f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:02.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.232+0000 7f84ef422640 1 --2- 172.21.15.42:0/2195389854 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f84e80a3d90 0x7f84e80a2f30 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f84d801aad0 tx=0x7f84d80191c0 comp rx=0 tx=0).stop 2024-09-16T04:10:02.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.232+0000 7f84ed9bf640 1 -- 172.21.15.42:0/2195389854 reap_dead start 2024-09-16T04:10:02.270 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.232+0000 7f84ef422640 1 -- 172.21.15.42:0/2195389854 shutdown_connections 2024-09-16T04:10:02.271 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.232+0000 7f84ef422640 1 --2- 172.21.15.42:0/2195389854 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f84e80a3d90 0x7f84e80a2f30 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:10:02.271 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.232+0000 7f84ef422640 1 -- 172.21.15.42:0/2195389854 >> 172.21.15.42:0/2195389854 conn(0x7f84e80a9fb0 msgr2=0x7f84e80a8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:10:02.271 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.232+0000 7f84ef422640 1 -- 172.21.15.42:0/2195389854 shutdown_connections 2024-09-16T04:10:02.271 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.232+0000 7f84ef422640 1 -- 172.21.15.42:0/2195389854 wait complete. 2024-09-16T04:10:02.271 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.232+0000 7f84ef422640 1 librados: shutdown 2024-09-16T04:10:02.271 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:10:02.271 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T04:10:02.271 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T04:10:02.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.392+0000 7f0c9ac48640 10 monclient: get_monmap_and_config 2024-09-16T04:10:02.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.392+0000 7f0c9ac48640 10 monclient: build_initial_monmap 2024-09-16T04:10:02.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.392+0000 7f0c9ac48640 10 monclient: monmap: 2024-09-16T04:10:02.395 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:10:02.395 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:10:02.395 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:10:02.393140+0000 2024-09-16T04:10:02.395 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:10:02.393140+0000 2024-09-16T04:10:02.395 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:10:02.395 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:10:02.395 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:10:02.395 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:10:02.395 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:10:02.395 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:10:02.395 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:10:02.395 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:10:02.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.392+0000 7f0c9ac48640 1 Processor -- start 2024-09-16T04:10:02.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.392+0000 7f0c9ac48640 1 -- start start 2024-09-16T04:10:02.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.392+0000 7f0c9ac48640 10 monclient: init 2024-09-16T04:10:02.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.392+0000 7f0c9ac48640 10 monclient: _reopen_session rank -1 2024-09-16T04:10:02.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.392+0000 7f0c9ac48640 10 monclient: _add_conns ranks=[0,1,2,3,4] 2024-09-16T04:10:02.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.392+0000 7f0c9ac48640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0c940a1330 addr v1:172.21.15.42:6789/0 2024-09-16T04:10:02.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.392+0000 7f0c9ac48640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0c9406dac0 addr v1:172.21.15.154:6789/0 2024-09-16T04:10:02.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.392+0000 7f0c9ac48640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0c940a1e20 0x7f0c9413da70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:10:02.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.392+0000 7f0c9ac48640 10 monclient(hunting): picked mon.noname-b con 0x7f0c940a1e20 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:10:02.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.392+0000 7f0c9ac48640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c9409c090 con 0x7f0c940a1330 2024-09-16T04:10:02.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.392+0000 7f0c9ac48640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c9409f610 con 0x7f0c9406dac0 2024-09-16T04:10:02.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.392+0000 7f0c9ac48640 10 monclient(hunting): start opening mon connection 2024-09-16T04:10:02.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.392+0000 7f0c9ac48640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f0c9409c890 con 0x7f0c940a1e20 2024-09-16T04:10:02.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.392+0000 7f0c9ac48640 10 monclient(hunting): _renew_subs 2024-09-16T04:10:02.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.392+0000 7f0c9ac48640 10 monclient(hunting): authenticate will time out at 2649.362773s 2024-09-16T04:10:02.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.392+0000 7f0c989e4640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f0c940a1330 0x7f0c940a1710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:37332/0 (socket says 172.21.15.42:37332) 2024-09-16T04:10:02.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.392+0000 7f0c989e4640 1 -- 172.21.15.42:0/555758639 learned_addr learned my addr 172.21.15.42:0/555758639 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T04:10:02.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.392+0000 7f0c991e5640 1 --2- 172.21.15.42:0/555758639 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0c940a1e20 0x7f0c9413da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:10:02.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.392+0000 7f0c991e5640 10 monclient(hunting): get_auth_request con 0x7f0c940a1e20 auth_method 0 2024-09-16T04:10:02.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.392+0000 7f0c991e5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:10:02.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.392+0000 7f0c991e5640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:02.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.392+0000 7f0c991e5640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:02.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.392+0000 7f0c991e5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:10:02.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.392+0000 7f0c991e5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:10:02.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.392+0000 7f0c991e5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:10:02.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.392+0000 7f0c937fe640 1 -- 172.21.15.42:0/555758639 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 999252993 0 0) 0x7f0c9409c090 con 0x7f0c940a1330 2024-09-16T04:10:02.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.392+0000 7f0c937fe640 10 monclient(hunting): _negotiate 2024-09-16T04:10:02.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.392+0000 7f0c937fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:02.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.392+0000 7f0c937fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:02.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.392+0000 7f0c937fe640 10 monclient(hunting): my global_id is 1034250 2024-09-16T04:10:02.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.392+0000 7f0c937fe640 1 -- 172.21.15.42:0/555758639 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0c80003530 con 0x7f0c940a1330 2024-09-16T04:10:02.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.392+0000 7f0c991e5640 10 monclient(hunting): handle_auth_done global_id 1014383 payload 274 2024-09-16T04:10:02.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.392+0000 7f0c991e5640 1 -- 172.21.15.42:0/555758639 >> v1:172.21.15.154:6789/0 conn(0x7f0c9406dac0 legacy=0x7f0c9413a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:02.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.392+0000 7f0c991e5640 1 -- 172.21.15.42:0/555758639 >> v1:172.21.15.42:6789/0 conn(0x7f0c940a1330 legacy=0x7f0c940a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:02.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.392+0000 7f0c991e5640 10 monclient: _finish_hunting 0 2024-09-16T04:10:02.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.392+0000 7f0c991e5640 1 monclient: found mon.noname-b 2024-09-16T04:10:02.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.392+0000 7f0c991e5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:10:02.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.392+0000 7f0c991e5640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T04:10:02.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.392+0000 7f0c991e5640 1 -- 172.21.15.42:0/555758639 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c9409f8f0 con 0x7f0c940a1e20 2024-09-16T04:10:02.414 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.396+0000 7f0c991e5640 1 --2- 172.21.15.42:0/555758639 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0c940a1e20 0x7f0c9413da70 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f0c840097f0 tx=0x7f0c8401c400 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5b5392552a0b717e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:10:02.414 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.396+0000 7f0c937fe640 1 -- 172.21.15.42:0/555758639 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c840ab620 con 0x7f0c940a1e20 2024-09-16T04:10:02.415 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.396+0000 7f0c937fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:02.415 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.396+0000 7f0c937fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:10:02.415 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.396+0000 7f0c937fe640 10 monclient: dump: 2024-09-16T04:10:02.415 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:10:02.415 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:10:02.415 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:02.415 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:02.415 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:02.415 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:10:02.415 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:10:02.415 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:10:02.415 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:10:02.415 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:10:02.415 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.396+0000 7f0c937fe640 10 monclient: _finish_auth 0 2024-09-16T04:10:02.416 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.396+0000 7f0c937fe640 10 monclient: _check_auth_tickets 2024-09-16T04:10:02.416 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.396+0000 7f0c937fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:02.416 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.396+0000 7f0c937fe640 1 -- 172.21.15.42:0/555758639 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c840b0070 con 0x7f0c940a1e20 2024-09-16T04:10:02.416 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.396+0000 7f0c9ac48640 5 monclient: authenticate success, global_id 1014383 2024-09-16T04:10:02.416 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.396+0000 7f0c9ac48640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:10:02.416 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.396+0000 7f0c937fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:10:02.416 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.396+0000 7f0c9ac48640 10 monclient: get_monmap_and_config success 2024-09-16T04:10:02.416 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.396+0000 7f0c937fe640 1 -- 172.21.15.42:0/555758639 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c840ab9e0 con 0x7f0c940a1e20 2024-09-16T04:10:02.416 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.396+0000 7f0c9ac48640 4 set_mon_vals no callback set 2024-09-16T04:10:02.416 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.396+0000 7f0c937fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:02.416 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.396+0000 7f0c937fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:10:02.416 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.396+0000 7f0c937fe640 10 monclient: dump: 2024-09-16T04:10:02.417 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:10:02.417 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:10:02.417 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:02.417 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:02.417 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:02.417 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:10:02.417 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:10:02.417 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:10:02.417 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:10:02.417 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:10:02.417 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.396+0000 7f0c9ac48640 10 monclient: shutdown 2024-09-16T04:10:02.417 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.396+0000 7f0c9ac48640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:10:02.417 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.396+0000 7f0c9ac48640 1 -- 172.21.15.42:0/555758639 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0c940a1e20 msgr2=0x7f0c9413da70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:02.417 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.396+0000 7f0c9ac48640 1 --2- 172.21.15.42:0/555758639 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0c940a1e20 0x7f0c9413da70 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f0c840097f0 tx=0x7f0c8401c400 comp rx=0 tx=0).stop 2024-09-16T04:10:02.418 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.396+0000 7f0c9ac48640 1 -- 172.21.15.42:0/555758639 shutdown_connections 2024-09-16T04:10:02.418 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.396+0000 7f0c9ac48640 1 --2- 172.21.15.42:0/555758639 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0c940a1e20 0x7f0c9413da70 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:10:02.418 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.396+0000 7f0c9ac48640 1 -- 172.21.15.42:0/555758639 >> 172.21.15.42:0/555758639 conn(0x7f0c940a2420 msgr2=0x7f0c940a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:10:02.418 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.396+0000 7f0c9ac48640 1 -- 172.21.15.42:0/555758639 shutdown_connections 2024-09-16T04:10:02.418 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.396+0000 7f0c9ac48640 1 -- 172.21.15.42:0/555758639 wait complete. 2024-09-16T04:10:02.418 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.396+0000 7f0c9ac48640 10 monclient: build_initial_monmap 2024-09-16T04:10:02.418 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.396+0000 7f0c9ac48640 10 monclient: monmap: 2024-09-16T04:10:02.418 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:10:02.418 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:10:02.418 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:10:02.418 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:10:02.418 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:10:02.418 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:10:02.419 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:10:02.419 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:10:02.419 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:10:02.419 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:10:02.419 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.396+0000 7f0c9ac48640 1 librados: starting msgr at 2024-09-16T04:10:02.419 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.396+0000 7f0c9ac48640 1 librados: starting objecter 2024-09-16T04:10:02.419 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.396+0000 7f0c9ac48640 1 Processor -- start 2024-09-16T04:10:02.419 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.396+0000 7f0c9ac48640 1 -- start start 2024-09-16T04:10:02.419 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.396+0000 7f0c9ac48640 1 librados: setting wanted keys 2024-09-16T04:10:02.419 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.396+0000 7f0c9ac48640 1 librados: calling monclient init 2024-09-16T04:10:02.419 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.396+0000 7f0c9ac48640 10 monclient: init 2024-09-16T04:10:02.419 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.396+0000 7f0c9ac48640 10 monclient: _reopen_session rank -1 2024-09-16T04:10:02.419 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.396+0000 7f0c9ac48640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:10:02.419 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.396+0000 7f0c9ac48640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0c940a1330 0x7f0c9413bd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:10:02.420 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.396+0000 7f0c9ac48640 10 monclient(hunting): picked mon.noname-a con 0x7f0c940a1330 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:10:02.420 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.396+0000 7f0c9ac48640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0c940a1e20 0x7f0c94135140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:10:02.420 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.396+0000 7f0c9ac48640 10 monclient(hunting): picked mon.noname-c con 0x7f0c940a1e20 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:10:02.420 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.396+0000 7f0c9ac48640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0c94135680 0x7f0c9409cb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:10:02.420 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.396+0000 7f0c9ac48640 10 monclient(hunting): picked mon.noname-b con 0x7f0c94135680 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:10:02.420 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.396+0000 7f0c9ac48640 10 monclient(hunting): start opening mon connection 2024-09-16T04:10:02.420 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.396+0000 7f0c9ac48640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f0c94140950 con 0x7f0c940a1330 2024-09-16T04:10:02.420 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.400+0000 7f0c9ac48640 10 monclient(hunting): start opening mon connection 2024-09-16T04:10:02.420 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.400+0000 7f0c9ac48640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f0c941407d0 con 0x7f0c94135680 2024-09-16T04:10:02.420 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.400+0000 7f0c9ac48640 10 monclient(hunting): start opening mon connection 2024-09-16T04:10:02.420 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.400+0000 7f0c9ac48640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f0c94140ad0 con 0x7f0c940a1e20 2024-09-16T04:10:02.420 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.400+0000 7f0c9ac48640 10 monclient(hunting): _renew_subs 2024-09-16T04:10:02.421 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.400+0000 7f0c9ac48640 10 monclient(hunting): authenticate will time out at 2649.368023s 2024-09-16T04:10:02.421 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.400+0000 7f0c989e4640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0c940a1330 0x7f0c9413bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:10:02.421 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.400+0000 7f0c93fff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0c940a1e20 0x7f0c94135140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:10:02.421 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.400+0000 7f0c989e4640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0c940a1330 0x7f0c9413bd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:60162/0 (socket says 172.21.15.42:60162) 2024-09-16T04:10:02.421 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.400+0000 7f0c989e4640 1 -- 172.21.15.42:0/1808187598 learned_addr learned my addr 172.21.15.42:0/1808187598 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:10:02.421 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.400+0000 7f0c989e4640 10 monclient(hunting): get_auth_request con 0x7f0c940a1330 auth_method 0 2024-09-16T04:10:02.421 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.400+0000 7f0c989e4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:10:02.421 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.400+0000 7f0c989e4640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:02.421 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.400+0000 7f0c989e4640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:02.421 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.400+0000 7f0c93fff640 10 monclient(hunting): get_auth_request con 0x7f0c940a1e20 auth_method 0 2024-09-16T04:10:02.421 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.400+0000 7f0c93fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:10:02.421 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.400+0000 7f0c93fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:02.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.400+0000 7f0c93fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:02.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.400+0000 7f0c989e4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:10:02.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.400+0000 7f0c989e4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:10:02.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.400+0000 7f0c989e4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:10:02.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.400+0000 7f0c93fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:10:02.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.400+0000 7f0c93fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:10:02.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.400+0000 7f0c93fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:10:02.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.400+0000 7f0c989e4640 10 monclient(hunting): handle_auth_done global_id 1034253 payload 824 2024-09-16T04:10:02.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.400+0000 7f0c989e4640 1 -- 172.21.15.42:0/1808187598 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0c940a1e20 msgr2=0x7f0c94135140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:02.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.400+0000 7f0c991e5640 1 --2- 172.21.15.42:0/1808187598 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0c94135680 0x7f0c9409cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:10:02.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.400+0000 7f0c989e4640 1 --2- 172.21.15.42:0/1808187598 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0c940a1e20 0x7f0c94135140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:10:02.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.400+0000 7f0c989e4640 1 -- 172.21.15.42:0/1808187598 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0c94135680 msgr2=0x7f0c9409cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:02.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.400+0000 7f0c989e4640 1 --2- 172.21.15.42:0/1808187598 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0c94135680 0x7f0c9409cb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:10:02.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.400+0000 7f0c989e4640 10 monclient: _finish_hunting 0 2024-09-16T04:10:02.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.400+0000 7f0c989e4640 1 monclient: found mon.noname-a 2024-09-16T04:10:02.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.400+0000 7f0c989e4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:10:02.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.400+0000 7f0c989e4640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:10:02.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.400+0000 7f0c989e4640 1 -- 172.21.15.42:0/1808187598 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c9413ce40 con 0x7f0c940a1330 2024-09-16T04:10:02.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.400+0000 7f0c93fff640 1 --2- 172.21.15.42:0/1808187598 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0c940a1e20 0x7f0c94135140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:10:02.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.400+0000 7f0c991e5640 10 monclient: get_auth_request con 0x7f0c94135680 auth_method 0 2024-09-16T04:10:02.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.400+0000 7f0c991e5640 1 --2- 172.21.15.42:0/1808187598 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0c94135680 0x7f0c9409cb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T04:10:02.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.400+0000 7f0c989e4640 1 --2- 172.21.15.42:0/1808187598 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0c940a1330 0x7f0c9413bd50 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f0c8c00c4d0 tx=0x7f0c8c00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:10:02.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.400+0000 7f0c917fa640 10 client.?.objecter ms_handle_connect 0x7f0c940a1330 2024-09-16T04:10:02.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.400+0000 7f0c917fa640 10 client.?.objecter resend_mon_ops 2024-09-16T04:10:02.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.400+0000 7f0c917fa640 1 -- 172.21.15.42:0/1808187598 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c8c017070 con 0x7f0c940a1330 2024-09-16T04:10:02.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.400+0000 7f0c917fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:02.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.400+0000 7f0c917fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:10:02.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.400+0000 7f0c917fa640 10 monclient: dump: 2024-09-16T04:10:02.425 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:10:02.425 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:10:02.425 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:02.425 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:02.425 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:02.425 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:10:02.425 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:10:02.425 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:10:02.425 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:10:02.425 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:10:02.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.400+0000 7f0c917fa640 10 monclient: _finish_auth 0 2024-09-16T04:10:02.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.400+0000 7f0c917fa640 10 monclient: _check_auth_tickets 2024-09-16T04:10:02.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.400+0000 7f0c917fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:02.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.400+0000 7f0c9ac48640 5 monclient: authenticate success, global_id 1034253 2024-09-16T04:10:02.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.400+0000 7f0c9ac48640 10 monclient: _renew_subs 2024-09-16T04:10:02.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.400+0000 7f0c9ac48640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:10:02.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.400+0000 7f0c9ac48640 1 -- 172.21.15.42:0/1808187598 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c9406e140 con 0x7f0c940a1330 2024-09-16T04:10:02.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.400+0000 7f0c917fa640 1 -- 172.21.15.42:0/1808187598 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c8c005cf0 con 0x7f0c940a1330 2024-09-16T04:10:02.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.400+0000 7f0c917fa640 10 monclient: handle_config config(0 keys) 2024-09-16T04:10:02.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.400+0000 7f0c917fa640 1 -- 172.21.15.42:0/1808187598 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c8c007270 con 0x7f0c940a1330 2024-09-16T04:10:02.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.400+0000 7f0c917fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:02.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.400+0000 7f0c917fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:10:02.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.400+0000 7f0c917fa640 10 monclient: dump: 2024-09-16T04:10:02.426 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:10:02.427 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:10:02.427 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:02.427 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:02.427 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:02.427 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:10:02.427 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:10:02.427 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:10:02.427 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:10:02.427 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:10:02.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.400+0000 7f0c927fc640 4 set_mon_vals no callback set 2024-09-16T04:10:02.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.400+0000 7f0c9ac48640 10 client.1034253.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:10:02.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.400+0000 7f0c9ac48640 10 monclient: _renew_subs 2024-09-16T04:10:02.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.400+0000 7f0c9ac48640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:10:02.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.400+0000 7f0c9ac48640 1 -- 172.21.15.42:0/1808187598 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c94120800 con 0x7f0c940a1330 2024-09-16T04:10:02.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.400+0000 7f0c9ac48640 1 librados: init done 2024-09-16T04:10:02.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.400+0000 7f0c9ac48640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:02.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.400+0000 7f0c9ac48640 1 --2- 172.21.15.42:0/1808187598 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0c9406dac0 0x7f0c9402c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:10:02.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.400+0000 7f0c9ac48640 10 monclient(hunting): start opening mon connection 2024-09-16T04:10:02.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.400+0000 7f0c9ac48640 1 -- 172.21.15.42:0/1808187598 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f0c94120480 con 0x7f0c9406dac0 2024-09-16T04:10:02.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.400+0000 7f0c917fa640 1 -- 172.21.15.42:0/1808187598 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f0c8c01a070 con 0x7f0c940a1330 2024-09-16T04:10:02.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.400+0000 7f0c917fa640 1 --2- 172.21.15.42:0/1808187598 >> v2:172.21.15.154:6812/254845342 conn(0x7f0c74037810 0x7f0c74039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:10:02.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.400+0000 7f0c917fa640 1 -- 172.21.15.42:0/1808187598 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(706..706 src has 451..706) ==== 5040+0+0 (secure 0 0 0) 0x7f0c8c048070 con 0x7f0c940a1330 2024-09-16T04:10:02.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.400+0000 7f0c917fa640 10 client.1034253.objecter ms_dispatch 0x7f0c94000ba0 osd_map(706..706 src has 451..706) 2024-09-16T04:10:02.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.400+0000 7f0c917fa640 3 client.1034253.objecter handle_osd_map got epochs [706,706] > 0 2024-09-16T04:10:02.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.400+0000 7f0c917fa640 3 client.1034253.objecter handle_osd_map decoding full epoch 706 2024-09-16T04:10:02.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.400+0000 7f0c93fff640 1 --2- 172.21.15.42:0/1808187598 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0c9406dac0 0x7f0c9402c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:10:02.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.400+0000 7f0c93fff640 10 monclient: get_auth_request con 0x7f0c9406dac0 auth_method 0 2024-09-16T04:10:02.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.400+0000 7f0c93fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:10:02.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.400+0000 7f0c93fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:02.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.400+0000 7f0c93fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:02.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.400+0000 7f0c917fa640 20 client.1034253.objecter dump_active .. 0 homeless 2024-09-16T04:10:02.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.400+0000 7f0c93fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:10:02.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.400+0000 7f0c93fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:10:02.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.400+0000 7f0c93fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:10:02.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.400+0000 7f0c991e5640 1 --2- 172.21.15.42:0/1808187598 >> v2:172.21.15.154:6812/254845342 conn(0x7f0c74037810 0x7f0c74039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:10:02.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.400+0000 7f0c991e5640 10 monclient: get_auth_request con 0x7f0c74037810 auth_method 0 2024-09-16T04:10:02.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.400+0000 7f0c93fff640 10 monclient(hunting): handle_auth_done global_id 1044145 payload 824 2024-09-16T04:10:02.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.400+0000 7f0c93fff640 1 -- 172.21.15.42:0/1808187598 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0c94073a30 con 0x7f0c9406dac0 2024-09-16T04:10:02.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.400+0000 7f0c917fa640 10 client.1034253.objecter ms_handle_connect 0x7f0c9406dac0 2024-09-16T04:10:02.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.400+0000 7f0c93fff640 1 --2- 172.21.15.42:0/1808187598 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0c9406dac0 0x7f0c9402c650 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f0c9413c5a0 tx=0x7f0c7c00d520 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:10:02.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.400+0000 7f0c917fa640 10 client.1034253.objecter resend_mon_ops 2024-09-16T04:10:02.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.400+0000 7f0c917fa640 10 client.1034253.objecter ms_handle_connect 0x7f0c74037810 2024-09-16T04:10:02.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.400+0000 7f0c991e5640 1 --2- 172.21.15.42:0/1808187598 >> v2:172.21.15.154:6812/254845342 conn(0x7f0c74037810 0x7f0c74039cd0 secure :-1 s=READY pgs=3924 cs=0 l=1 rev1=1 crypto rx=0x7f0c840af000 tx=0x7f0c840a7000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:10:02.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.400+0000 7f0c917fa640 1 -- 172.21.15.42:0/1808187598 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c7c00de30 con 0x7f0c9406dac0 2024-09-16T04:10:02.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.400+0000 7f0c917fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:02.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.400+0000 7f0c917fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:10:02.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.400+0000 7f0c917fa640 10 monclient: dump: 2024-09-16T04:10:02.430 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:10:02.430 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:10:02.430 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:02.430 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:02.430 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:02.430 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:10:02.431 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:10:02.431 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:10:02.431 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:10:02.431 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:10:02.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.404+0000 7f0c917fa640 1 -- 172.21.15.42:0/1808187598 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f0c7c00de30 con 0x7f0c9406dac0 2024-09-16T04:10:02.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.404+0000 7f0c917fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:02.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.404+0000 7f0c917fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:10:02.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.404+0000 7f0c917fa640 1 -- 172.21.15.42:0/1808187598 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0c9406dac0 msgr2=0x7f0c9402c650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:02.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.404+0000 7f0c917fa640 1 --2- 172.21.15.42:0/1808187598 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0c9406dac0 0x7f0c9402c650 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f0c9413c5a0 tx=0x7f0c7c00d520 comp rx=0 tx=0).stop 2024-09-16T04:10:02.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.404+0000 7f0c917fa640 1 -- 172.21.15.42:0/1808187598 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0c9406dac0 msgr2=0x7f0c9402c650 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:10:02.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.404+0000 7f0c917fa640 1 --2- 172.21.15.42:0/1808187598 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0c9406dac0 0x7f0c9402c650 secure :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f0c9413c5a0 tx=0x7f0c7c00d520 comp rx=0 tx=0).stop 2024-09-16T04:10:02.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.408+0000 7f0c9ac48640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:10:02.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.408+0000 7f0c9ac48640 1 --2- 172.21.15.42:0/1808187598 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0c9402cb90 0x7f0c9402cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:10:02.432 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.408+0000 7f0c9ac48640 10 monclient(hunting): start opening mon connection 2024-09-16T04:10:02.432 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.408+0000 7f0c9ac48640 1 -- 172.21.15.42:0/1808187598 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f0c94135f40 con 0x7f0c9402cb90 2024-09-16T04:10:02.432 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.408+0000 7f0c93fff640 1 --2- 172.21.15.42:0/1808187598 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0c9402cb90 0x7f0c9402cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:10:02.432 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.408+0000 7f0c93fff640 10 monclient: get_auth_request con 0x7f0c9402cb90 auth_method 0 2024-09-16T04:10:02.432 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.408+0000 7f0c93fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:10:02.432 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.408+0000 7f0c93fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:02.432 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.408+0000 7f0c93fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:02.432 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.408+0000 7f0c93fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:10:02.432 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.408+0000 7f0c93fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:10:02.432 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.408+0000 7f0c93fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:10:02.432 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.408+0000 7f0c93fff640 10 monclient(hunting): handle_auth_done global_id 1044148 payload 824 2024-09-16T04:10:02.432 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.408+0000 7f0c93fff640 1 -- 172.21.15.42:0/1808187598 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f0c940a1710 con 0x7f0c9402cb90 2024-09-16T04:10:02.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.412+0000 7f0c917fa640 10 client.1034253.objecter ms_handle_connect 0x7f0c9402cb90 2024-09-16T04:10:02.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.412+0000 7f0c93fff640 1 --2- 172.21.15.42:0/1808187598 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0c9402cb90 0x7f0c9402cfa0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f0c7c002240 tx=0x7f0c7c0014b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:10:02.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.412+0000 7f0c917fa640 10 client.1034253.objecter resend_mon_ops 2024-09-16T04:10:02.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.412+0000 7f0c917fa640 1 -- 172.21.15.42:0/1808187598 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c7c01e3e0 con 0x7f0c9402cb90 2024-09-16T04:10:02.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.412+0000 7f0c917fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:02.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.412+0000 7f0c917fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:10:02.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.412+0000 7f0c917fa640 10 monclient: dump: 2024-09-16T04:10:02.433 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:10:02.433 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:10:02.433 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:02.433 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:02.433 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:02.433 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:10:02.434 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:10:02.434 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:10:02.434 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:10:02.434 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:10:02.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.412+0000 7f0c917fa640 1 -- 172.21.15.42:0/1808187598 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4545 (secure 0 0 0) 0x7f0c7c01e3e0 con 0x7f0c9402cb90 2024-09-16T04:10:02.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.412+0000 7f0c917fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:10:02.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.412+0000 7f0c917fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:10:02.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.412+0000 7f0c917fa640 1 -- 172.21.15.42:0/1808187598 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0c9402cb90 msgr2=0x7f0c9402cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:02.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.412+0000 7f0c917fa640 1 --2- 172.21.15.42:0/1808187598 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0c9402cb90 0x7f0c9402cfa0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f0c7c002240 tx=0x7f0c7c0014b0 comp rx=0 tx=0).stop 2024-09-16T04:10:02.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.412+0000 7f0c917fa640 1 -- 172.21.15.42:0/1808187598 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0c9402cb90 msgr2=0x7f0c9402cfa0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:10:02.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.412+0000 7f0c917fa640 1 --2- 172.21.15.42:0/1808187598 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0c9402cb90 0x7f0c9402cfa0 secure :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f0c7c002240 tx=0x7f0c7c0014b0 comp rx=0 tx=0).stop 2024-09-16T04:10:02.435 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T04:10:02.435 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:10:02.435 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:10:02.435 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T04:10:02.435 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 256, 2024-09-16T04:10:02.435 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 23099, 2024-09-16T04:10:02.435 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T04:10:02.435 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T04:10:02.435 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T04:10:02.435 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T04:10:02.435 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:10:02.436 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 3, 2024-09-16T04:10:02.436 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:10:02.436 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T04:10:02.436 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T04:10:02.436 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:10:02.436 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:10:02.436 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:10:02.436 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:10:02.436 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:10:02.436 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:10:02.436 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:10:02.436 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:10:02.436 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:10:02.436 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:10:02.437 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:10:02.437 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:10:02.437 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:10:02.437 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T04:10:02.437 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:10:02.437 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:10:02.437 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:10:02.437 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:10:02.437 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:10:02.437 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:10:02.437 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:10:02.437 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:10:02.437 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:10:02.437 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:10:02.438 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:10:02.438 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:10:02.438 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:10:02.438 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T04:10:02.438 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T04:10:02.438 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T04:10:02.438 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T04:10:02.438 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T04:10:02.438 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T04:10:02.438 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:10:02.438 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:10:02.438 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T04:10:02.438 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T04:10:02.439 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T04:10:02.439 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T04:10:02.439 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T04:10:02.439 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T04:10:02.439 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T04:10:02.439 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:10:02.439 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T04:10:02.439 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:10:02.439 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:10:02.439 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:10:02.439 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:10:02.439 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:10:02.439 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:10:02.439 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:10:02.440 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:10:02.440 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:10:02.440 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:10:02.440 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:10:02.440 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:10:02.440 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T04:10:02.440 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:10:02.440 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T04:10:02.440 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:10:02.440 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:10:02.440 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:10:02.440 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:10:02.440 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:10:02.440 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:10:02.441 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:10:02.441 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T04:10:02.441 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:10:02.441 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:10:02.441 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:10:02.441 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:10:02.441 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T04:10:02.441 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:10:02.441 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:10:02.441 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:10:02.441 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:10:02.441 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T04:10:02.441 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T04:10:02.442 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:10:02.442 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:10:02.442 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:10:02.442 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:10:02.442 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:10:02.442 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:10:02.442 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:10:02.442 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:10:02.442 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:10:02.442 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:10:02.442 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:10:02.442 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T04:10:02.442 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:10:02.442 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:10:02.443 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:10:02.443 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:10:02.443 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T04:10:02.443 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:10:02.443 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:10:02.443 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:10:02.443 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:10:02.443 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T04:10:02.443 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T04:10:02.443 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:10:02.443 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:10:02.443 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:10:02.443 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:10:02.443 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:10:02.444 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:10:02.444 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:10:02.444 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:10:02.444 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:10:02.444 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:10:02.444 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:10:02.444 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T04:10:02.444 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:10:02.444 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:10:02.444 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:10:02.444 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:10:02.444 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T04:10:02.444 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:10:02.444 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:10:02.445 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:10:02.445 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:10:02.445 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T04:10:02.445 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T04:10:02.445 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:10:02.445 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:10:02.445 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:10:02.445 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:10:02.445 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:10:02.445 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:10:02.445 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T04:10:02.445 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T04:10:02.445 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:10:02.445 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:10:02.446 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:10:02.446 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:10:02.446 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:10:02.446 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:10:02.446 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T04:10:02.446 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:10:02.446 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:10:02.446 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:10:02.446 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:10:02.446 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:10:02.446 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:10:02.446 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:10:02.446 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T04:10:02.446 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T04:10:02.449 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.412+0000 7f0c9ac48640 10 librados: watch_flush enter 2024-09-16T04:10:02.449 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.412+0000 7f0c9ac48640 10 librados: watch_flush exit 2024-09-16T04:10:02.449 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.412+0000 7f0c9ac48640 20 client.1034253.objecter shutdown clearing up homeless session... 2024-09-16T04:10:02.449 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.412+0000 7f0c9ac48640 10 client.1034253.objecter successfully canceled tick 2024-09-16T04:10:02.449 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.412+0000 7f0c9ac48640 1 -- 172.21.15.42:0/1808187598 >> v2:172.21.15.154:6812/254845342 conn(0x7f0c74037810 msgr2=0x7f0c74039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:02.449 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.412+0000 7f0c9ac48640 1 --2- 172.21.15.42:0/1808187598 >> v2:172.21.15.154:6812/254845342 conn(0x7f0c74037810 0x7f0c74039cd0 secure :-1 s=READY pgs=3924 cs=0 l=1 rev1=1 crypto rx=0x7f0c840af000 tx=0x7f0c840a7000 comp rx=0 tx=0).stop 2024-09-16T04:10:02.449 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.412+0000 7f0c9ac48640 10 monclient: shutdown 2024-09-16T04:10:02.449 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.412+0000 7f0c9ac48640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:10:02.450 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.412+0000 7f0c9ac48640 1 -- 172.21.15.42:0/1808187598 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0c940a1330 msgr2=0x7f0c9413bd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:02.450 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.412+0000 7f0c9ac48640 1 --2- 172.21.15.42:0/1808187598 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0c940a1330 0x7f0c9413bd50 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f0c8c00c4d0 tx=0x7f0c8c00c990 comp rx=0 tx=0).stop 2024-09-16T04:10:02.450 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.412+0000 7f0c991e5640 1 -- 172.21.15.42:0/1808187598 reap_dead start 2024-09-16T04:10:02.450 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.412+0000 7f0c991e5640 1 -- 172.21.15.42:0/1808187598 reap_dead start 2024-09-16T04:10:02.450 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.412+0000 7f0c9ac48640 1 -- 172.21.15.42:0/1808187598 shutdown_connections 2024-09-16T04:10:02.450 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.412+0000 7f0c9ac48640 1 -- 172.21.15.42:0/1808187598 >> 172.21.15.42:0/1808187598 conn(0x7f0c940a2420 msgr2=0x7f0c9413d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:10:02.450 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.412+0000 7f0c9ac48640 1 -- 172.21.15.42:0/1808187598 shutdown_connections 2024-09-16T04:10:02.450 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.412+0000 7f0c9ac48640 1 -- 172.21.15.42:0/1808187598 wait complete. 2024-09-16T04:10:02.450 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.412+0000 7f0c9ac48640 1 librados: shutdown 2024-09-16T04:10:02.450 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:10:02.451 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-09-16T04:10:02.451 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-16T04:10:02.451 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-09-16T04:10:02.451 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-16T04:10:02.451 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-16T04:10:02.452 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T04:10:02.475 INFO:tasks.ceph.mon.b:Stopped 2024-09-16T04:10:02.475 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T04:10:02.475 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:10:02.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.588+0000 7f52b89cb640 10 monclient: get_monmap_and_config 2024-09-16T04:10:02.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.588+0000 7f52b89cb640 10 monclient: build_initial_monmap 2024-09-16T04:10:02.591 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.588+0000 7f52b89cb640 10 monclient: monmap: 2024-09-16T04:10:02.591 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:10:02.591 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:10:02.591 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:10:02.589966+0000 2024-09-16T04:10:02.592 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:10:02.589966+0000 2024-09-16T04:10:02.592 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:10:02.592 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:10:02.592 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:10:02.592 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:10:02.592 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:10:02.592 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:10:02.592 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:10:02.592 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:10:02.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.588+0000 7f52b89cb640 1 Processor -- start 2024-09-16T04:10:02.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.588+0000 7f52b89cb640 1 -- start start 2024-09-16T04:10:02.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.588+0000 7f52b89cb640 10 monclient: init 2024-09-16T04:10:02.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.588+0000 7f52b89cb640 10 monclient: _reopen_session rank -1 2024-09-16T04:10:02.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.588+0000 7f52b89cb640 10 monclient: _add_conns ranks=[1,2,4,3,0] 2024-09-16T04:10:02.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.588+0000 7f52b89cb640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f52b406db90 addr v1:172.21.15.154:6789/0 2024-09-16T04:10:02.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.588+0000 7f52b89cb640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f52b41340c0 0x7f52b413c190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:10:02.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.588+0000 7f52b89cb640 10 monclient(hunting): picked mon.noname-b con 0x7f52b41340c0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:10:02.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.588+0000 7f52b89cb640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f52b413c6d0 0x7f52b413eac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:10:02.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.588+0000 7f52b89cb640 10 monclient(hunting): picked mon.noname-c con 0x7f52b413c6d0 addr v2:172.21.15.154:3300/0 2024-09-16T04:10:02.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.588+0000 7f52b89cb640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f52b406ece0 con 0x7f52b406db90 2024-09-16T04:10:02.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.588+0000 7f52b89cb640 10 monclient(hunting): start opening mon connection 2024-09-16T04:10:02.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.588+0000 7f52b89cb640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f52b406df70 con 0x7f52b413c6d0 2024-09-16T04:10:02.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.588+0000 7f52b89cb640 10 monclient(hunting): start opening mon connection 2024-09-16T04:10:02.595 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.588+0000 7f52b89cb640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f52b409bf80 con 0x7f52b41340c0 2024-09-16T04:10:02.596 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.588+0000 7f52b89cb640 10 monclient(hunting): _renew_subs 2024-09-16T04:10:02.596 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.588+0000 7f52b89cb640 10 monclient(hunting): authenticate will time out at 2649.559508s 2024-09-16T04:10:02.596 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.588+0000 7f52b259c640 1 -- >> v1:172.21.15.154:6789/0 conn(0x7f52b406db90 legacy=0x7f52b40a37f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.154:6789/0 2024-09-16T04:10:02.596 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.588+0000 7f52b2d9d640 1 -- >> v2:172.21.15.154:3300/0 conn(0x7f52b413c6d0 msgr2=0x7f52b413eac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T04:10:02.596 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.588+0000 7f52b1d9b640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f52b41340c0 0x7f52b413c190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:10:02.596 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.588+0000 7f52b2d9d640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f52b413c6d0 0x7f52b413eac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:10:02.596 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.588+0000 7f52b1d9b640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f52b41340c0 0x7f52b413c190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:51284/0 (socket says 172.21.15.42:51284) 2024-09-16T04:10:02.596 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.588+0000 7f52b1d9b640 1 -- 172.21.15.42:0/2990550694 learned_addr learned my addr 172.21.15.42:0/2990550694 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:10:02.596 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.588+0000 7f52b1d9b640 10 monclient(hunting): get_auth_request con 0x7f52b41340c0 auth_method 0 2024-09-16T04:10:02.596 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.588+0000 7f52b1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:10:02.596 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.588+0000 7f52b1d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:02.597 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.588+0000 7f52b1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:02.597 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.588+0000 7f52b1d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:10:02.597 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.588+0000 7f52b1d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:10:02.597 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.588+0000 7f52b1d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:10:02.597 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.592+0000 7f52b1d9b640 10 monclient(hunting): handle_auth_done global_id 1014389 payload 274 2024-09-16T04:10:02.597 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.592+0000 7f52b1d9b640 1 -- 172.21.15.42:0/2990550694 >> v2:172.21.15.154:3300/0 conn(0x7f52b413c6d0 msgr2=0x7f52b413eac0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:10:02.597 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.592+0000 7f52b1d9b640 1 --2- 172.21.15.42:0/2990550694 >> v2:172.21.15.154:3300/0 conn(0x7f52b413c6d0 0x7f52b413eac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:10:02.597 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.592+0000 7f52b1d9b640 1 -- 172.21.15.42:0/2990550694 >> v1:172.21.15.154:6789/0 conn(0x7f52b406db90 legacy=0x7f52b40a37f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:10:02.597 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.592+0000 7f52b1d9b640 10 monclient: _finish_hunting 0 2024-09-16T04:10:02.597 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.592+0000 7f52b1d9b640 1 monclient: found mon.noname-b 2024-09-16T04:10:02.597 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.592+0000 7f52b1d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:10:02.597 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.592+0000 7f52b1d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T04:10:02.598 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.592+0000 7f52b1d9b640 1 -- 172.21.15.42:0/2990550694 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52b41344d0 con 0x7f52b41340c0 2024-09-16T04:10:02.603 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.592+0000 7f52b1d9b640 1 --2- 172.21.15.42:0/2990550694 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f52b41340c0 0x7f52b413c190 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f529c009870 tx=0x7f529c01c400 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7d6f96635c382c2c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:10:02.603 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.592+0000 7f52b159a640 1 -- 172.21.15.42:0/2990550694 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f529c0b1070 con 0x7f52b41340c0 2024-09-16T04:10:02.604 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.592+0000 7f52b159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:02.604 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.592+0000 7f52b159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:10:02.604 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.592+0000 7f52b159a640 10 monclient: dump: 2024-09-16T04:10:02.604 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:10:02.604 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:10:02.604 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:02.604 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:02.604 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:02.604 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:10:02.604 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:10:02.604 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:10:02.604 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:10:02.605 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:10:02.605 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.592+0000 7f52b159a640 10 monclient: _finish_auth 0 2024-09-16T04:10:02.605 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.592+0000 7f52b159a640 10 monclient: _check_auth_tickets 2024-09-16T04:10:02.605 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.592+0000 7f52b159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:02.605 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.592+0000 7f52b159a640 1 -- 172.21.15.42:0/2990550694 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f529c0ac620 con 0x7f52b41340c0 2024-09-16T04:10:02.605 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.592+0000 7f52b159a640 10 monclient: handle_config config(0 keys) 2024-09-16T04:10:02.605 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.592+0000 7f52b159a640 1 -- 172.21.15.42:0/2990550694 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f529c0ac9f0 con 0x7f52b41340c0 2024-09-16T04:10:02.605 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.592+0000 7f52b89cb640 5 monclient: authenticate success, global_id 1014389 2024-09-16T04:10:02.605 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.592+0000 7f52b89cb640 10 monclient: get_monmap_and_config success 2024-09-16T04:10:02.605 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.592+0000 7f52b89cb640 4 set_mon_vals no callback set 2024-09-16T04:10:02.605 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.592+0000 7f52b159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:02.605 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.592+0000 7f52b159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:10:02.605 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.592+0000 7f52b159a640 10 monclient: dump: 2024-09-16T04:10:02.606 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:10:02.606 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:10:02.606 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:02.606 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:02.606 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:02.606 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:10:02.606 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:10:02.606 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:10:02.606 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:10:02.606 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:10:02.606 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.592+0000 7f52b89cb640 10 monclient: shutdown 2024-09-16T04:10:02.606 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.592+0000 7f52b89cb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:10:02.606 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.592+0000 7f52b89cb640 1 -- 172.21.15.42:0/2990550694 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f52b41340c0 msgr2=0x7f52b413c190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:02.606 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.592+0000 7f52b89cb640 1 --2- 172.21.15.42:0/2990550694 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f52b41340c0 0x7f52b413c190 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f529c009870 tx=0x7f529c01c400 comp rx=0 tx=0).stop 2024-09-16T04:10:02.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.592+0000 7f52b89cb640 1 -- 172.21.15.42:0/2990550694 shutdown_connections 2024-09-16T04:10:02.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.592+0000 7f52b89cb640 1 --2- 172.21.15.42:0/2990550694 >> v2:172.21.15.154:3300/0 conn(0x7f52b413c6d0 0x7f52b413eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:10:02.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.592+0000 7f52b89cb640 1 --2- 172.21.15.42:0/2990550694 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f52b41340c0 0x7f52b413c190 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:10:02.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.592+0000 7f52b89cb640 1 -- 172.21.15.42:0/2990550694 >> 172.21.15.42:0/2990550694 conn(0x7f52b40a9fb0 msgr2=0x7f52b40aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:10:02.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.592+0000 7f52b89cb640 1 -- 172.21.15.42:0/2990550694 shutdown_connections 2024-09-16T04:10:02.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.592+0000 7f52b89cb640 1 -- 172.21.15.42:0/2990550694 wait complete. 2024-09-16T04:10:02.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.596+0000 7f52b89cb640 10 monclient: build_initial_monmap 2024-09-16T04:10:02.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.596+0000 7f52b89cb640 10 monclient: monmap: 2024-09-16T04:10:02.607 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:10:02.607 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:10:02.607 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:10:02.607 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:10:02.607 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:10:02.608 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:10:02.608 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:10:02.608 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:10:02.608 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:10:02.608 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:10:02.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.596+0000 7f52b89cb640 1 librados: starting msgr at 2024-09-16T04:10:02.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.596+0000 7f52b89cb640 1 librados: starting objecter 2024-09-16T04:10:02.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.596+0000 7f52b89cb640 1 Processor -- start 2024-09-16T04:10:02.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.596+0000 7f52b89cb640 1 -- start start 2024-09-16T04:10:02.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.596+0000 7f52b89cb640 1 librados: setting wanted keys 2024-09-16T04:10:02.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.596+0000 7f52b89cb640 1 librados: calling monclient init 2024-09-16T04:10:02.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.596+0000 7f52b89cb640 10 monclient: init 2024-09-16T04:10:02.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.596+0000 7f52b89cb640 10 monclient: _reopen_session rank -1 2024-09-16T04:10:02.609 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.596+0000 7f52b89cb640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:10:02.609 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.596+0000 7f52b89cb640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f52b406db90 0x7f52b413b500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:10:02.609 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.596+0000 7f52b89cb640 10 monclient(hunting): picked mon.noname-b con 0x7f52b406db90 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:10:02.609 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.596+0000 7f52b89cb640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f52b41340c0 0x7f52b41cf320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:10:02.609 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.596+0000 7f52b89cb640 10 monclient(hunting): picked mon.noname-c con 0x7f52b41340c0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:10:02.609 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.596+0000 7f52b89cb640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f52b413c6d0 0x7f52b41cf860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:10:02.609 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.596+0000 7f52b89cb640 10 monclient(hunting): picked mon.noname-a con 0x7f52b413c6d0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:10:02.609 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.596+0000 7f52b89cb640 10 monclient(hunting): start opening mon connection 2024-09-16T04:10:02.609 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.596+0000 7f52b89cb640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f52b4141820 con 0x7f52b413c6d0 2024-09-16T04:10:02.609 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.596+0000 7f52b89cb640 10 monclient(hunting): start opening mon connection 2024-09-16T04:10:02.609 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.596+0000 7f52b89cb640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f52b41419a0 con 0x7f52b406db90 2024-09-16T04:10:02.609 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.596+0000 7f52b89cb640 10 monclient(hunting): start opening mon connection 2024-09-16T04:10:02.609 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.596+0000 7f52b89cb640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f52b4141b20 con 0x7f52b41340c0 2024-09-16T04:10:02.609 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.596+0000 7f52b89cb640 10 monclient(hunting): _renew_subs 2024-09-16T04:10:02.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.596+0000 7f52b89cb640 10 monclient(hunting): authenticate will time out at 2649.565013s 2024-09-16T04:10:02.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.596+0000 7f52b259c640 1 -- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f52b406db90 msgr2=0x7f52b413b500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T04:10:02.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.596+0000 7f52b1d9b640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f52b41340c0 0x7f52b41cf320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:10:02.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.596+0000 7f52b2d9d640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f52b413c6d0 0x7f52b41cf860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:10:02.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.596+0000 7f52b259c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f52b406db90 0x7f52b413b500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:10:02.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.596+0000 7f52b1d9b640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f52b41340c0 0x7f52b41cf320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:51296/0 (socket says 172.21.15.42:51296) 2024-09-16T04:10:02.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.596+0000 7f52b2d9d640 10 monclient(hunting): get_auth_request con 0x7f52b413c6d0 auth_method 0 2024-09-16T04:10:02.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.596+0000 7f52b1d9b640 1 -- 172.21.15.42:0/452774176 learned_addr learned my addr 172.21.15.42:0/452774176 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:10:02.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.596+0000 7f52b2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:10:02.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.596+0000 7f52b2d9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:02.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.596+0000 7f52b2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:02.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.596+0000 7f52b1d9b640 10 monclient(hunting): get_auth_request con 0x7f52b41340c0 auth_method 0 2024-09-16T04:10:02.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.596+0000 7f52b1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:10:02.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.596+0000 7f52b1d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:02.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.596+0000 7f52b1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:02.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.596+0000 7f52b2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:10:02.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.596+0000 7f52b2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:10:02.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.596+0000 7f52b2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:10:02.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.596+0000 7f52b1d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:10:02.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.596+0000 7f52b1d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:10:02.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.596+0000 7f52b1d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:10:02.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.596+0000 7f52b1d9b640 10 monclient(hunting): handle_auth_done global_id 1014392 payload 824 2024-09-16T04:10:02.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.596+0000 7f52b1d9b640 1 -- 172.21.15.42:0/452774176 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f52b406db90 msgr2=0x7f52b413b500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:10:02.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.596+0000 7f52b1d9b640 1 --2- 172.21.15.42:0/452774176 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f52b406db90 0x7f52b413b500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:10:02.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.596+0000 7f52b1d9b640 1 -- 172.21.15.42:0/452774176 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f52b413c6d0 msgr2=0x7f52b41cf860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:02.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.596+0000 7f52b1d9b640 1 --2- 172.21.15.42:0/452774176 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f52b413c6d0 0x7f52b41cf860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:10:02.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.596+0000 7f52b1d9b640 10 monclient: _finish_hunting 0 2024-09-16T04:10:02.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.596+0000 7f52b1d9b640 1 monclient: found mon.noname-c 2024-09-16T04:10:02.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.596+0000 7f52b1d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:10:02.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.596+0000 7f52b1d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T04:10:02.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.596+0000 7f52b1d9b640 1 -- 172.21.15.42:0/452774176 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52b413a840 con 0x7f52b41340c0 2024-09-16T04:10:02.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.596+0000 7f52b2d9d640 1 --2- 172.21.15.42:0/452774176 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f52b413c6d0 0x7f52b41cf860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:10:02.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.596+0000 7f52b1d9b640 1 --2- 172.21.15.42:0/452774176 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f52b41340c0 0x7f52b41cf320 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f529c01aab0 tx=0x7f529c01ac40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:10:02.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.596+0000 7f52aaffd640 10 client.?.objecter ms_handle_connect 0x7f52b41340c0 2024-09-16T04:10:02.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.596+0000 7f52aaffd640 10 client.?.objecter resend_mon_ops 2024-09-16T04:10:02.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.596+0000 7f52aaffd640 1 -- 172.21.15.42:0/452774176 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f529c0bb020 con 0x7f52b41340c0 2024-09-16T04:10:02.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.596+0000 7f52aaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:02.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.596+0000 7f52aaffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:10:02.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.596+0000 7f52aaffd640 10 monclient: dump: 2024-09-16T04:10:02.613 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:10:02.613 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:10:02.613 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:02.613 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:02.613 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:02.613 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:10:02.613 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:10:02.613 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:10:02.613 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:10:02.613 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:10:02.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.596+0000 7f52aaffd640 10 monclient: _finish_auth 0 2024-09-16T04:10:02.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.596+0000 7f52aaffd640 10 monclient: _check_auth_tickets 2024-09-16T04:10:02.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.596+0000 7f52aaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:02.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.596+0000 7f52aaffd640 1 -- 172.21.15.42:0/452774176 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f529c0b7620 con 0x7f52b41340c0 2024-09-16T04:10:02.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.596+0000 7f52b89cb640 5 monclient: authenticate success, global_id 1014392 2024-09-16T04:10:02.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.596+0000 7f52b89cb640 10 monclient: _renew_subs 2024-09-16T04:10:02.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.596+0000 7f52b89cb640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:10:02.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.596+0000 7f52b89cb640 1 -- 172.21.15.42:0/452774176 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52b413ec90 con 0x7f52b41340c0 2024-09-16T04:10:02.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.596+0000 7f52aaffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:10:02.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.596+0000 7f52aaffd640 1 -- 172.21.15.42:0/452774176 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f529c0b1040 con 0x7f52b41340c0 2024-09-16T04:10:02.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.596+0000 7f52aaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:02.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.596+0000 7f52aaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:10:02.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.596+0000 7f52aaffd640 10 monclient: dump: 2024-09-16T04:10:02.614 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:10:02.614 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:10:02.615 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:02.615 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:02.615 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:02.615 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:10:02.615 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:10:02.615 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:10:02.615 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:10:02.615 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:10:02.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.596+0000 7f52b0d99640 4 set_mon_vals no callback set 2024-09-16T04:10:02.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.596+0000 7f52b89cb640 10 client.1014392.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:10:02.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.596+0000 7f52b89cb640 10 monclient: _renew_subs 2024-09-16T04:10:02.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.596+0000 7f52b89cb640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:10:02.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.596+0000 7f52b89cb640 1 -- 172.21.15.42:0/452774176 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f52b4139c20 con 0x7f52b41340c0 2024-09-16T04:10:02.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.596+0000 7f52b89cb640 1 librados: init done 2024-09-16T04:10:02.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.596+0000 7f52aaffd640 1 -- 172.21.15.42:0/452774176 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f529c0be020 con 0x7f52b41340c0 2024-09-16T04:10:02.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.596+0000 7f52b89cb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:02.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.596+0000 7f52b89cb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:02.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.596+0000 7f52b89cb640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:10:02.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.596+0000 7f52b89cb640 1 -- 172.21.15.42:0/452774176 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f52b40a37f0 con 0x7f52b41340c0 2024-09-16T04:10:02.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.596+0000 7f52aaffd640 1 --2- 172.21.15.42:0/452774176 >> v2:172.21.15.154:6812/254845342 conn(0x7f52940377f0 0x7f5294039cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:10:02.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.596+0000 7f52aaffd640 1 -- 172.21.15.42:0/452774176 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(706..706 src has 451..706) ==== 5040+0+0 (secure 0 0 0) 0x7f529c0e30c0 con 0x7f52b41340c0 2024-09-16T04:10:02.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.596+0000 7f52aaffd640 10 client.1014392.objecter ms_dispatch 0x7f52b4000ba0 osd_map(706..706 src has 451..706) 2024-09-16T04:10:02.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.596+0000 7f52aaffd640 3 client.1014392.objecter handle_osd_map got epochs [706,706] > 0 2024-09-16T04:10:02.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.596+0000 7f52aaffd640 3 client.1014392.objecter handle_osd_map decoding full epoch 706 2024-09-16T04:10:02.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.596+0000 7f52aaffd640 20 client.1014392.objecter dump_active .. 0 homeless 2024-09-16T04:10:02.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.596+0000 7f52b259c640 1 --2- 172.21.15.42:0/452774176 >> v2:172.21.15.154:6812/254845342 conn(0x7f52940377f0 0x7f5294039cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:10:02.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.596+0000 7f52b259c640 10 monclient: get_auth_request con 0x7f52940377f0 auth_method 0 2024-09-16T04:10:02.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.600+0000 7f52aaffd640 10 client.1014392.objecter ms_handle_connect 0x7f52940377f0 2024-09-16T04:10:02.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.600+0000 7f52b259c640 1 --2- 172.21.15.42:0/452774176 >> v2:172.21.15.154:6812/254845342 conn(0x7f52940377f0 0x7f5294039cb0 secure :-1 s=READY pgs=3925 cs=0 l=1 rev1=1 crypto rx=0x7f52ac003cc0 tx=0x7f52ac009270 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:10:02.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.600+0000 7f52aaffd640 1 -- 172.21.15.42:0/452774176 <== mon.2 v2:172.21.15.42:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f529c0c5030 con 0x7f52b41340c0 2024-09-16T04:10:02.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.600+0000 7f52aaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:02.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.600+0000 7f52aaffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:10:02.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.824+0000 7f52b89cb640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:10:02.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.824+0000 7f52b89cb640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:10:02.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.824+0000 7f52b89cb640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:10:02.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.824+0000 7f52b89cb640 1 -- 172.21.15.42:0/452774176 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f52b406f680 con 0x7f52b41340c0 2024-09-16T04:10:02.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.828+0000 7f52aaffd640 1 -- 172.21.15.42:0/452774176 <== mon.2 v2:172.21.15.42:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7f52b413ec90 con 0x7f52b41340c0 2024-09-16T04:10:02.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.828+0000 7f52aaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:10:02.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.828+0000 7f52aaffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:10:02.830 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":256,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:10:02.832 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.828+0000 7f52b89cb640 10 librados: watch_flush enter 2024-09-16T04:10:02.832 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.828+0000 7f52b89cb640 10 librados: watch_flush exit 2024-09-16T04:10:02.832 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.828+0000 7f52b89cb640 20 client.1014392.objecter shutdown clearing up homeless session... 2024-09-16T04:10:02.832 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.828+0000 7f52b89cb640 10 client.1014392.objecter successfully canceled tick 2024-09-16T04:10:02.832 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.828+0000 7f52b89cb640 1 -- 172.21.15.42:0/452774176 >> v2:172.21.15.154:6812/254845342 conn(0x7f52940377f0 msgr2=0x7f5294039cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:02.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.828+0000 7f52b89cb640 1 --2- 172.21.15.42:0/452774176 >> v2:172.21.15.154:6812/254845342 conn(0x7f52940377f0 0x7f5294039cb0 secure :-1 s=READY pgs=3925 cs=0 l=1 rev1=1 crypto rx=0x7f52ac003cc0 tx=0x7f52ac009270 comp rx=0 tx=0).stop 2024-09-16T04:10:02.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.828+0000 7f52b89cb640 10 monclient: shutdown 2024-09-16T04:10:02.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.828+0000 7f52b89cb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:10:02.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.828+0000 7f52b89cb640 1 -- 172.21.15.42:0/452774176 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f52b41340c0 msgr2=0x7f52b41cf320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:02.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.828+0000 7f52b89cb640 1 --2- 172.21.15.42:0/452774176 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f52b41340c0 0x7f52b41cf320 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f529c01aab0 tx=0x7f529c01ac40 comp rx=0 tx=0).stop 2024-09-16T04:10:02.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.828+0000 7f52b89cb640 1 -- 172.21.15.42:0/452774176 shutdown_connections 2024-09-16T04:10:02.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.828+0000 7f52b89cb640 1 --2- 172.21.15.42:0/452774176 >> v2:172.21.15.154:6812/254845342 conn(0x7f52940377f0 0x7f5294039cb0 unknown :-1 s=CLOSED pgs=3925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:10:02.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.828+0000 7f52b89cb640 1 --2- 172.21.15.42:0/452774176 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f52b413c6d0 0x7f52b41cf860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:10:02.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.828+0000 7f52b89cb640 1 --2- 172.21.15.42:0/452774176 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f52b41340c0 0x7f52b41cf320 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:10:02.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.828+0000 7f52b89cb640 1 --2- 172.21.15.42:0/452774176 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f52b406db90 0x7f52b413b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:10:02.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.828+0000 7f52b89cb640 1 -- 172.21.15.42:0/452774176 >> 172.21.15.42:0/452774176 conn(0x7f52b40a9fb0 msgr2=0x7f52b40a79f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:10:02.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.828+0000 7f52b89cb640 1 -- 172.21.15.42:0/452774176 shutdown_connections 2024-09-16T04:10:02.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.828+0000 7f52b89cb640 1 -- 172.21.15.42:0/452774176 wait complete. 2024-09-16T04:10:02.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:02.828+0000 7f52b89cb640 1 librados: shutdown 2024-09-16T04:10:05.849 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:10:05.983 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.979+0000 7fc970f62640 10 monclient: get_monmap_and_config 2024-09-16T04:10:05.983 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.979+0000 7fc970f62640 10 monclient: build_initial_monmap 2024-09-16T04:10:05.983 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.979+0000 7fc970f62640 10 monclient: monmap: 2024-09-16T04:10:05.983 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:10:05.983 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:10:05.983 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:10:05.981737+0000 2024-09-16T04:10:05.984 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:10:05.981737+0000 2024-09-16T04:10:05.984 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:10:05.984 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:10:05.984 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:10:05.984 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:10:05.984 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:10:05.984 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:10:05.984 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:10:05.984 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:10:05.986 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.979+0000 7fc970f62640 1 Processor -- start 2024-09-16T04:10:05.986 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.979+0000 7fc970f62640 1 -- start start 2024-09-16T04:10:05.987 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.979+0000 7fc970f62640 10 monclient: init 2024-09-16T04:10:05.987 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.979+0000 7fc970f62640 10 monclient: _reopen_session rank -1 2024-09-16T04:10:05.987 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.979+0000 7fc970f62640 10 monclient: _add_conns ranks=[0,1,2,4,3] 2024-09-16T04:10:05.987 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.979+0000 7fc970f62640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc96c0a39b0 addr v1:172.21.15.42:6789/0 2024-09-16T04:10:05.987 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.979+0000 7fc970f62640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc96c0a3d90 addr v1:172.21.15.154:6789/0 2024-09-16T04:10:05.987 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.979+0000 7fc970f62640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc96c13daa0 0x7fc96c13fe90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:10:05.987 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.979+0000 7fc970f62640 10 monclient(hunting): picked mon.noname-b con 0x7fc96c13daa0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:10:05.987 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.979+0000 7fc970f62640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc96c134290 con 0x7fc96c0a39b0 2024-09-16T04:10:05.987 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.979+0000 7fc970f62640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc96c09c140 con 0x7fc96c0a3d90 2024-09-16T04:10:05.987 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.979+0000 7fc970f62640 10 monclient(hunting): start opening mon connection 2024-09-16T04:10:05.987 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.979+0000 7fc970f62640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fc96c09c420 con 0x7fc96c13daa0 2024-09-16T04:10:05.987 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.979+0000 7fc970f62640 10 monclient(hunting): _renew_subs 2024-09-16T04:10:05.987 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.979+0000 7fc970f62640 10 monclient(hunting): authenticate will time out at 2652.951477s 2024-09-16T04:10:05.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.979+0000 7fc969d9b640 1 -- >> v1:172.21.15.154:6789/0 conn(0x7fc96c0a3d90 legacy=0x7fc96c13c380 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.154:6789/0 2024-09-16T04:10:05.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.979+0000 7fc96a59c640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fc96c0a39b0 0x7fc96c133b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:56558/0 (socket says 172.21.15.42:56558) 2024-09-16T04:10:05.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.979+0000 7fc96a59c640 1 -- 172.21.15.42:0/3223062503 learned_addr learned my addr 172.21.15.42:0/3223062503 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T04:10:05.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.979+0000 7fc96ad9d640 1 --2- 172.21.15.42:0/3223062503 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc96c13daa0 0x7fc96c13fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:10:05.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.979+0000 7fc96ad9d640 10 monclient(hunting): get_auth_request con 0x7fc96c13daa0 auth_method 0 2024-09-16T04:10:05.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.979+0000 7fc96ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:10:05.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.979+0000 7fc96ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:05.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.979+0000 7fc96ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:05.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.979+0000 7fc96ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:10:05.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.979+0000 7fc96ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:10:05.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.979+0000 7fc96ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:10:05.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.979+0000 7fc96ad9d640 10 monclient(hunting): handle_auth_done global_id 1014395 payload 274 2024-09-16T04:10:05.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.979+0000 7fc96959a640 1 -- 172.21.15.42:0/3223062503 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1582820256 0 0) 0x7fc96c134290 con 0x7fc96c0a39b0 2024-09-16T04:10:05.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.983+0000 7fc96ad9d640 1 -- 172.21.15.42:0/3223062503 >> v1:172.21.15.154:6789/0 conn(0x7fc96c0a3d90 legacy=0x7fc96c13c380 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:10:05.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.983+0000 7fc96ad9d640 1 -- 172.21.15.42:0/3223062503 >> v1:172.21.15.42:6789/0 conn(0x7fc96c0a39b0 legacy=0x7fc96c133b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:05.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.983+0000 7fc96ad9d640 10 monclient: _finish_hunting 0 2024-09-16T04:10:05.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.983+0000 7fc96ad9d640 1 monclient: found mon.noname-b 2024-09-16T04:10:05.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.983+0000 7fc96ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:10:05.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.983+0000 7fc96ad9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T04:10:05.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.983+0000 7fc96ad9d640 1 -- 172.21.15.42:0/3223062503 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc96c06dd60 con 0x7fc96c13daa0 2024-09-16T04:10:05.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.983+0000 7fc96959a640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T04:10:05.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.983+0000 7fc96ad9d640 1 --2- 172.21.15.42:0/3223062503 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc96c13daa0 0x7fc96c13fe90 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fc9640097f0 tx=0x7fc96401c360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e3d93de2077a86e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:10:05.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.983+0000 7fc96959a640 1 -- 172.21.15.42:0/3223062503 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9640b0070 con 0x7fc96c13daa0 2024-09-16T04:10:05.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.983+0000 7fc96959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:05.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.983+0000 7fc96959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:10:05.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.983+0000 7fc96959a640 10 monclient: dump: 2024-09-16T04:10:05.995 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:10:05.995 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:10:05.995 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:05.995 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:05.996 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:05.996 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:10:05.996 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:10:05.996 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:10:05.996 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:10:05.996 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:10:05.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.983+0000 7fc96959a640 10 monclient: _finish_auth 0 2024-09-16T04:10:05.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.983+0000 7fc96959a640 10 monclient: _check_auth_tickets 2024-09-16T04:10:05.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.983+0000 7fc96959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:05.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.983+0000 7fc970f62640 5 monclient: authenticate success, global_id 1014395 2024-09-16T04:10:05.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.983+0000 7fc970f62640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:10:05.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.983+0000 7fc96959a640 1 -- 172.21.15.42:0/3223062503 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc9640ab600 con 0x7fc96c13daa0 2024-09-16T04:10:05.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.983+0000 7fc96959a640 10 monclient: handle_config config(0 keys) 2024-09-16T04:10:05.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.983+0000 7fc970f62640 10 monclient: get_monmap_and_config success 2024-09-16T04:10:05.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.983+0000 7fc970f62640 4 set_mon_vals no callback set 2024-09-16T04:10:05.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.983+0000 7fc96959a640 1 -- 172.21.15.42:0/3223062503 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9640ab9d0 con 0x7fc96c13daa0 2024-09-16T04:10:05.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.983+0000 7fc96959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:05.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.983+0000 7fc96959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:10:05.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.983+0000 7fc96959a640 10 monclient: dump: 2024-09-16T04:10:05.997 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:10:05.997 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:10:05.997 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:05.997 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:05.997 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:05.997 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:10:05.997 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:10:05.998 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:10:05.998 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:10:05.998 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:10:05.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.983+0000 7fc970f62640 10 monclient: shutdown 2024-09-16T04:10:05.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.983+0000 7fc970f62640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:10:05.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.983+0000 7fc970f62640 1 -- 172.21.15.42:0/3223062503 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc96c13daa0 msgr2=0x7fc96c13fe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:05.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.983+0000 7fc970f62640 1 --2- 172.21.15.42:0/3223062503 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc96c13daa0 0x7fc96c13fe90 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fc9640097f0 tx=0x7fc96401c360 comp rx=0 tx=0).stop 2024-09-16T04:10:05.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.983+0000 7fc970f62640 1 -- 172.21.15.42:0/3223062503 shutdown_connections 2024-09-16T04:10:05.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.983+0000 7fc970f62640 1 --2- 172.21.15.42:0/3223062503 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc96c13daa0 0x7fc96c13fe90 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:10:05.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.983+0000 7fc970f62640 1 -- 172.21.15.42:0/3223062503 >> 172.21.15.42:0/3223062503 conn(0x7fc96c0a9fb0 msgr2=0x7fc96c0aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:10:05.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.983+0000 7fc970f62640 1 -- 172.21.15.42:0/3223062503 shutdown_connections 2024-09-16T04:10:05.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.983+0000 7fc970f62640 1 -- 172.21.15.42:0/3223062503 wait complete. 2024-09-16T04:10:05.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.983+0000 7fc970f62640 10 monclient: build_initial_monmap 2024-09-16T04:10:05.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.983+0000 7fc970f62640 10 monclient: monmap: 2024-09-16T04:10:05.999 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:10:05.999 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:10:05.999 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:10:05.999 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:10:05.999 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:10:05.999 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:10:05.999 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:10:05.999 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:10:05.999 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:10:05.999 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:10:05.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.983+0000 7fc970f62640 1 librados: starting msgr at 2024-09-16T04:10:06.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.983+0000 7fc970f62640 1 librados: starting objecter 2024-09-16T04:10:06.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.983+0000 7fc970f62640 1 Processor -- start 2024-09-16T04:10:06.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.983+0000 7fc970f62640 1 -- start start 2024-09-16T04:10:06.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.983+0000 7fc970f62640 1 librados: setting wanted keys 2024-09-16T04:10:06.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.983+0000 7fc970f62640 1 librados: calling monclient init 2024-09-16T04:10:06.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.983+0000 7fc970f62640 10 monclient: init 2024-09-16T04:10:06.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.987+0000 7fc970f62640 10 monclient: _reopen_session rank -1 2024-09-16T04:10:06.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.987+0000 7fc970f62640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:10:06.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.987+0000 7fc970f62640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc96c0a39b0 0x7fc96c1cf120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:10:06.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.987+0000 7fc970f62640 10 monclient(hunting): picked mon.noname-b con 0x7fc96c0a39b0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:10:06.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.987+0000 7fc970f62640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc96c0a3d90 0x7fc96c1cf660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:10:06.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.987+0000 7fc970f62640 10 monclient(hunting): picked mon.noname-c con 0x7fc96c0a3d90 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:10:06.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.987+0000 7fc970f62640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc96c13daa0 0x7fc96c1cfba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:10:06.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.987+0000 7fc970f62640 10 monclient(hunting): picked mon.noname-a con 0x7fc96c13daa0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:10:06.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.987+0000 7fc970f62640 10 monclient(hunting): start opening mon connection 2024-09-16T04:10:06.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.987+0000 7fc970f62640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fc96c142f80 con 0x7fc96c13daa0 2024-09-16T04:10:06.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.987+0000 7fc970f62640 10 monclient(hunting): start opening mon connection 2024-09-16T04:10:06.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.987+0000 7fc970f62640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fc96c142e00 con 0x7fc96c0a39b0 2024-09-16T04:10:06.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.987+0000 7fc970f62640 10 monclient(hunting): start opening mon connection 2024-09-16T04:10:06.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.987+0000 7fc970f62640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fc96c143100 con 0x7fc96c0a3d90 2024-09-16T04:10:06.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.987+0000 7fc970f62640 10 monclient(hunting): _renew_subs 2024-09-16T04:10:06.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.987+0000 7fc970f62640 10 monclient(hunting): authenticate will time out at 2652.956450s 2024-09-16T04:10:06.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.987+0000 7fc969d9b640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc96c0a3d90 0x7fc96c1cf660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:10:06.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.987+0000 7fc96ad9d640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc96c13daa0 0x7fc96c1cfba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:10:06.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.987+0000 7fc969d9b640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc96c0a3d90 0x7fc96c1cf660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:48202/0 (socket says 172.21.15.42:48202) 2024-09-16T04:10:06.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.987+0000 7fc969d9b640 1 -- 172.21.15.42:0/2730139023 learned_addr learned my addr 172.21.15.42:0/2730139023 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:10:06.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.987+0000 7fc969d9b640 10 monclient(hunting): get_auth_request con 0x7fc96c0a3d90 auth_method 0 2024-09-16T04:10:06.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.987+0000 7fc969d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:10:06.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.987+0000 7fc969d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:06.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.987+0000 7fc969d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:06.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.987+0000 7fc96a59c640 1 -- 172.21.15.42:0/2730139023 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc96c0a39b0 msgr2=0x7fc96c1cf120 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T04:10:06.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.987+0000 7fc96a59c640 1 --2- 172.21.15.42:0/2730139023 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc96c0a39b0 0x7fc96c1cf120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:10:06.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.987+0000 7fc969d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:10:06.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.987+0000 7fc969d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:10:06.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.987+0000 7fc969d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:10:06.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.987+0000 7fc96ad9d640 10 monclient(hunting): get_auth_request con 0x7fc96c13daa0 auth_method 0 2024-09-16T04:10:06.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.987+0000 7fc96ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:10:06.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.987+0000 7fc96ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:06.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.987+0000 7fc96ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:06.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.987+0000 7fc96ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:10:06.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.987+0000 7fc96ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:10:06.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.987+0000 7fc96ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:10:06.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.987+0000 7fc969d9b640 10 monclient(hunting): handle_auth_done global_id 1014398 payload 824 2024-09-16T04:10:06.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.987+0000 7fc969d9b640 1 -- 172.21.15.42:0/2730139023 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc96c0a39b0 msgr2=0x7fc96c1cf120 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:10:06.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.987+0000 7fc969d9b640 1 --2- 172.21.15.42:0/2730139023 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc96c0a39b0 0x7fc96c1cf120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:10:06.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.987+0000 7fc969d9b640 1 -- 172.21.15.42:0/2730139023 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc96c13daa0 msgr2=0x7fc96c1cfba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:06.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.987+0000 7fc969d9b640 1 --2- 172.21.15.42:0/2730139023 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc96c13daa0 0x7fc96c1cfba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:10:06.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.987+0000 7fc969d9b640 10 monclient: _finish_hunting 0 2024-09-16T04:10:06.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.987+0000 7fc969d9b640 1 monclient: found mon.noname-c 2024-09-16T04:10:06.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.987+0000 7fc969d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:10:06.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.987+0000 7fc969d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T04:10:06.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.987+0000 7fc969d9b640 1 -- 172.21.15.42:0/2730139023 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc96c09d0e0 con 0x7fc96c0a3d90 2024-09-16T04:10:06.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.987+0000 7fc969d9b640 1 --2- 172.21.15.42:0/2730139023 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc96c0a3d90 0x7fc96c1cf660 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fc96000b530 tx=0x7fc96000b9f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:10:06.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.987+0000 7fc95affd640 10 client.?.objecter ms_handle_connect 0x7fc96c0a3d90 2024-09-16T04:10:06.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.987+0000 7fc95affd640 10 client.?.objecter resend_mon_ops 2024-09-16T04:10:06.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.987+0000 7fc95affd640 1 -- 172.21.15.42:0/2730139023 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc960013020 con 0x7fc96c0a3d90 2024-09-16T04:10:06.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.987+0000 7fc95affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:06.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.987+0000 7fc95affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:10:06.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.987+0000 7fc95affd640 10 monclient: dump: 2024-09-16T04:10:06.007 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:10:06.007 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:10:06.007 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:06.007 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:06.007 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:06.007 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:10:06.007 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:10:06.007 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:10:06.007 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:10:06.007 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:10:06.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.987+0000 7fc95affd640 10 monclient: _finish_auth 0 2024-09-16T04:10:06.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.987+0000 7fc95affd640 10 monclient: _check_auth_tickets 2024-09-16T04:10:06.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.987+0000 7fc95affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:06.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.987+0000 7fc95affd640 1 -- 172.21.15.42:0/2730139023 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc960003df0 con 0x7fc96c0a3d90 2024-09-16T04:10:06.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.987+0000 7fc95affd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:10:06.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.987+0000 7fc95affd640 1 -- 172.21.15.42:0/2730139023 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9600052b0 con 0x7fc96c0a3d90 2024-09-16T04:10:06.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.987+0000 7fc970f62640 5 monclient: authenticate success, global_id 1014398 2024-09-16T04:10:06.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.987+0000 7fc970f62640 10 monclient: _renew_subs 2024-09-16T04:10:06.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.987+0000 7fc970f62640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:10:06.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.987+0000 7fc970f62640 1 -- 172.21.15.42:0/2730139023 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc96c09ce00 con 0x7fc96c0a3d90 2024-09-16T04:10:06.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.987+0000 7fc95affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:06.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.987+0000 7fc95affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:10:06.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.987+0000 7fc95affd640 10 monclient: dump: 2024-09-16T04:10:06.008 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:10:06.009 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:10:06.009 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:06.009 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:06.009 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:06.009 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:10:06.009 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:10:06.009 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:10:06.009 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:10:06.009 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:10:06.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.987+0000 7fc95bfff640 4 set_mon_vals no callback set 2024-09-16T04:10:06.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.987+0000 7fc970f62640 10 client.1014398.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:10:06.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.987+0000 7fc970f62640 10 monclient: _renew_subs 2024-09-16T04:10:06.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.987+0000 7fc970f62640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:10:06.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.987+0000 7fc970f62640 1 -- 172.21.15.42:0/2730139023 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc96c139e50 con 0x7fc96c0a3d90 2024-09-16T04:10:06.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.987+0000 7fc970f62640 1 librados: init done 2024-09-16T04:10:06.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.987+0000 7fc970f62640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:06.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.987+0000 7fc970f62640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:06.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.987+0000 7fc970f62640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:10:06.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.987+0000 7fc970f62640 1 -- 172.21.15.42:0/2730139023 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc96c133b80 con 0x7fc96c0a3d90 2024-09-16T04:10:06.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.987+0000 7fc95affd640 1 -- 172.21.15.42:0/2730139023 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fc960016020 con 0x7fc96c0a3d90 2024-09-16T04:10:06.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.987+0000 7fc95affd640 1 --2- 172.21.15.42:0/2730139023 >> v2:172.21.15.154:6812/254845342 conn(0x7fc94c0377c0 0x7fc94c039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:10:06.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.987+0000 7fc95affd640 1 -- 172.21.15.42:0/2730139023 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(706..706 src has 451..706) ==== 5040+0+0 (secure 0 0 0) 0x7fc960048860 con 0x7fc96c0a3d90 2024-09-16T04:10:06.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.987+0000 7fc95affd640 10 client.1014398.objecter ms_dispatch 0x7fc96c000ba0 osd_map(706..706 src has 451..706) 2024-09-16T04:10:06.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.987+0000 7fc95affd640 3 client.1014398.objecter handle_osd_map got epochs [706,706] > 0 2024-09-16T04:10:06.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.987+0000 7fc95affd640 3 client.1014398.objecter handle_osd_map decoding full epoch 706 2024-09-16T04:10:06.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.987+0000 7fc95affd640 20 client.1014398.objecter dump_active .. 0 homeless 2024-09-16T04:10:06.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.987+0000 7fc96a59c640 1 --2- 172.21.15.42:0/2730139023 >> v2:172.21.15.154:6812/254845342 conn(0x7fc94c0377c0 0x7fc94c039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:10:06.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.987+0000 7fc96a59c640 10 monclient: get_auth_request con 0x7fc94c0377c0 auth_method 0 2024-09-16T04:10:06.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.987+0000 7fc95affd640 10 client.1014398.objecter ms_handle_connect 0x7fc94c0377c0 2024-09-16T04:10:06.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.987+0000 7fc96a59c640 1 --2- 172.21.15.42:0/2730139023 >> v2:172.21.15.154:6812/254845342 conn(0x7fc94c0377c0 0x7fc94c039c80 secure :-1 s=READY pgs=3926 cs=0 l=1 rev1=1 crypto rx=0x7fc95c004ba0 tx=0x7fc95c001830 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:10:06.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.991+0000 7fc95affd640 1 -- 172.21.15.42:0/2730139023 <== mon.2 v2:172.21.15.42:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7fc96001d020 con 0x7fc96c0a3d90 2024-09-16T04:10:06.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.991+0000 7fc95affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:06.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:05.991+0000 7fc95affd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:10:06.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:06.211+0000 7fc970f62640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:10:06.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:06.211+0000 7fc970f62640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:10:06.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:06.211+0000 7fc970f62640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:10:06.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:06.211+0000 7fc970f62640 1 -- 172.21.15.42:0/2730139023 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fc96c1400a0 con 0x7fc96c0a3d90 2024-09-16T04:10:06.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:06.211+0000 7fc95affd640 1 -- 172.21.15.42:0/2730139023 <== mon.2 v2:172.21.15.42:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7fc9600056e0 con 0x7fc96c0a3d90 2024-09-16T04:10:06.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:06.211+0000 7fc95affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:10:06.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:06.211+0000 7fc95affd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:10:06.217 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":256,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:10:06.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:06.215+0000 7fc970f62640 10 librados: watch_flush enter 2024-09-16T04:10:06.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:06.215+0000 7fc970f62640 10 librados: watch_flush exit 2024-09-16T04:10:06.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:06.215+0000 7fc970f62640 20 client.1014398.objecter shutdown clearing up homeless session... 2024-09-16T04:10:06.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:06.215+0000 7fc970f62640 10 client.1014398.objecter successfully canceled tick 2024-09-16T04:10:06.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:06.215+0000 7fc970f62640 1 -- 172.21.15.42:0/2730139023 >> v2:172.21.15.154:6812/254845342 conn(0x7fc94c0377c0 msgr2=0x7fc94c039c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:06.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:06.215+0000 7fc970f62640 1 --2- 172.21.15.42:0/2730139023 >> v2:172.21.15.154:6812/254845342 conn(0x7fc94c0377c0 0x7fc94c039c80 secure :-1 s=READY pgs=3926 cs=0 l=1 rev1=1 crypto rx=0x7fc95c004ba0 tx=0x7fc95c001830 comp rx=0 tx=0).stop 2024-09-16T04:10:06.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:06.215+0000 7fc970f62640 10 monclient: shutdown 2024-09-16T04:10:06.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:06.215+0000 7fc970f62640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:10:06.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:06.215+0000 7fc970f62640 1 -- 172.21.15.42:0/2730139023 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc96c0a3d90 msgr2=0x7fc96c1cf660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:06.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:06.215+0000 7fc970f62640 1 --2- 172.21.15.42:0/2730139023 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc96c0a3d90 0x7fc96c1cf660 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fc96000b530 tx=0x7fc96000b9f0 comp rx=0 tx=0).stop 2024-09-16T04:10:06.220 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:06.215+0000 7fc970f62640 1 -- 172.21.15.42:0/2730139023 shutdown_connections 2024-09-16T04:10:06.220 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:06.215+0000 7fc970f62640 1 --2- 172.21.15.42:0/2730139023 >> v2:172.21.15.154:6812/254845342 conn(0x7fc94c0377c0 0x7fc94c039c80 unknown :-1 s=CLOSED pgs=3926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:10:06.220 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:06.215+0000 7fc970f62640 1 --2- 172.21.15.42:0/2730139023 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fc96c13daa0 0x7fc96c1cfba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:10:06.220 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:06.215+0000 7fc970f62640 1 --2- 172.21.15.42:0/2730139023 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fc96c0a3d90 0x7fc96c1cf660 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:10:06.220 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:06.215+0000 7fc970f62640 1 --2- 172.21.15.42:0/2730139023 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fc96c0a39b0 0x7fc96c1cf120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:10:06.220 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:06.215+0000 7fc970f62640 1 -- 172.21.15.42:0/2730139023 >> 172.21.15.42:0/2730139023 conn(0x7fc96c0a9fb0 msgr2=0x7fc96c0a8840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:10:06.220 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:06.215+0000 7fc970f62640 1 -- 172.21.15.42:0/2730139023 shutdown_connections 2024-09-16T04:10:06.220 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:06.215+0000 7fc970f62640 1 -- 172.21.15.42:0/2730139023 wait complete. 2024-09-16T04:10:06.220 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:06.215+0000 7fc970f62640 1 librados: shutdown 2024-09-16T04:10:09.235 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:10:09.367 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.363+0000 7fea37405640 10 monclient: get_monmap_and_config 2024-09-16T04:10:09.367 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.363+0000 7fea37405640 10 monclient: build_initial_monmap 2024-09-16T04:10:09.367 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.363+0000 7fea37405640 10 monclient: monmap: 2024-09-16T04:10:09.367 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:10:09.367 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:10:09.367 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:10:09.365801+0000 2024-09-16T04:10:09.367 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:10:09.365801+0000 2024-09-16T04:10:09.367 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:10:09.367 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:10:09.368 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:10:09.368 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:10:09.368 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:10:09.368 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:10:09.368 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:10:09.368 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:10:09.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.363+0000 7fea37405640 1 Processor -- start 2024-09-16T04:10:09.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.363+0000 7fea37405640 1 -- start start 2024-09-16T04:10:09.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.363+0000 7fea37405640 10 monclient: init 2024-09-16T04:10:09.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.363+0000 7fea37405640 10 monclient: _reopen_session rank -1 2024-09-16T04:10:09.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.363+0000 7fea37405640 10 monclient: _add_conns ranks=[2,0,1,4,3] 2024-09-16T04:10:09.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.363+0000 7fea37405640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fea300a1330 0x7fea300a1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:10:09.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.363+0000 7fea37405640 10 monclient(hunting): picked mon.noname-b con 0x7fea300a1330 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:10:09.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.363+0000 7fea37405640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fea3006dac0 addr v1:172.21.15.42:6789/0 2024-09-16T04:10:09.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.363+0000 7fea37405640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fea300a1c50 addr v1:172.21.15.154:6789/0 2024-09-16T04:10:09.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.363+0000 7fea37405640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fea3009c7d0 con 0x7fea3006dac0 2024-09-16T04:10:09.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.363+0000 7fea37405640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fea3009c090 con 0x7fea300a1c50 2024-09-16T04:10:09.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.363+0000 7fea37405640 10 monclient(hunting): start opening mon connection 2024-09-16T04:10:09.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.363+0000 7fea37405640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fea3009f610 con 0x7fea300a1330 2024-09-16T04:10:09.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.363+0000 7fea37405640 10 monclient(hunting): _renew_subs 2024-09-16T04:10:09.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.363+0000 7fea37405640 10 monclient(hunting): authenticate will time out at 2656.335265s 2024-09-16T04:10:09.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.363+0000 7fea359a2640 1 -- >> v1:172.21.15.154:6789/0 conn(0x7fea300a1c50 legacy=0x7fea30135810 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.154:6789/0 2024-09-16T04:10:09.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.363+0000 7fea349a0640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fea3006dac0 0x7fea30135100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:56572/0 (socket says 172.21.15.42:56572) 2024-09-16T04:10:09.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.363+0000 7fea349a0640 1 -- 172.21.15.42:0/1556603845 learned_addr learned my addr 172.21.15.42:0/1556603845 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T04:10:09.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.363+0000 7fea351a1640 1 --2- 172.21.15.42:0/1556603845 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fea300a1330 0x7fea300a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:10:09.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.363+0000 7fea351a1640 10 monclient(hunting): get_auth_request con 0x7fea300a1330 auth_method 0 2024-09-16T04:10:09.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.363+0000 7fea351a1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:10:09.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.363+0000 7fea351a1640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:09.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.363+0000 7fea351a1640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:09.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.363+0000 7fea351a1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:10:09.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.363+0000 7fea351a1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:10:09.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.363+0000 7fea351a1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:10:09.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.363+0000 7fea351a1640 10 monclient(hunting): handle_auth_done global_id 1014401 payload 274 2024-09-16T04:10:09.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.363+0000 7fea351a1640 1 -- 172.21.15.42:0/1556603845 >> v1:172.21.15.154:6789/0 conn(0x7fea300a1c50 legacy=0x7fea30135810 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:10:09.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.363+0000 7fea351a1640 1 -- 172.21.15.42:0/1556603845 >> v1:172.21.15.42:6789/0 conn(0x7fea3006dac0 legacy=0x7fea30135100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:09.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.363+0000 7fea351a1640 10 monclient: _finish_hunting 0 2024-09-16T04:10:09.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.363+0000 7fea351a1640 1 monclient: found mon.noname-b 2024-09-16T04:10:09.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.363+0000 7fea351a1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:10:09.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.363+0000 7fea351a1640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T04:10:09.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.363+0000 7fea351a1640 1 -- 172.21.15.42:0/1556603845 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea3009f8f0 con 0x7fea300a1330 2024-09-16T04:10:09.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.367+0000 7fea351a1640 1 --2- 172.21.15.42:0/1556603845 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fea300a1330 0x7fea300a1710 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fea2c0097f0 tx=0x7fea2c01c3f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c3401a29bdee2543 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:10:09.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.367+0000 7fea27fff640 1 -- 172.21.15.42:0/1556603845 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea2c0b0070 con 0x7fea300a1330 2024-09-16T04:10:09.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.367+0000 7fea27fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:09.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.367+0000 7fea27fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:10:09.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.367+0000 7fea27fff640 10 monclient: dump: 2024-09-16T04:10:09.378 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:10:09.378 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:10:09.378 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:09.378 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:09.378 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:09.378 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:10:09.378 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:10:09.379 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:10:09.379 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:10:09.379 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:10:09.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.367+0000 7fea27fff640 10 monclient: _finish_auth 0 2024-09-16T04:10:09.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.367+0000 7fea27fff640 10 monclient: _check_auth_tickets 2024-09-16T04:10:09.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.367+0000 7fea27fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:09.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.367+0000 7fea27fff640 1 -- 172.21.15.42:0/1556603845 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea2c0ab600 con 0x7fea300a1330 2024-09-16T04:10:09.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.367+0000 7fea27fff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:10:09.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.367+0000 7fea27fff640 1 -- 172.21.15.42:0/1556603845 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea2c0ab9d0 con 0x7fea300a1330 2024-09-16T04:10:09.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.367+0000 7fea37405640 5 monclient: authenticate success, global_id 1014401 2024-09-16T04:10:09.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.367+0000 7fea37405640 10 monclient: get_monmap_and_config success 2024-09-16T04:10:09.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.367+0000 7fea37405640 4 set_mon_vals no callback set 2024-09-16T04:10:09.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.367+0000 7fea27fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:09.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.367+0000 7fea27fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:10:09.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.367+0000 7fea27fff640 10 monclient: dump: 2024-09-16T04:10:09.380 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:10:09.380 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:10:09.380 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:09.380 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:09.380 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:09.380 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:10:09.380 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:10:09.380 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:10:09.380 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:10:09.380 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:10:09.380 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.367+0000 7fea37405640 10 monclient: shutdown 2024-09-16T04:10:09.380 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.367+0000 7fea37405640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:10:09.380 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.367+0000 7fea37405640 1 -- 172.21.15.42:0/1556603845 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fea300a1330 msgr2=0x7fea300a1710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:09.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.367+0000 7fea37405640 1 --2- 172.21.15.42:0/1556603845 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fea300a1330 0x7fea300a1710 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fea2c0097f0 tx=0x7fea2c01c3f0 comp rx=0 tx=0).stop 2024-09-16T04:10:09.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.367+0000 7fea37405640 1 -- 172.21.15.42:0/1556603845 shutdown_connections 2024-09-16T04:10:09.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.367+0000 7fea37405640 1 --2- 172.21.15.42:0/1556603845 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fea300a1330 0x7fea300a1710 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:10:09.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.367+0000 7fea37405640 1 -- 172.21.15.42:0/1556603845 >> 172.21.15.42:0/1556603845 conn(0x7fea300a2420 msgr2=0x7fea300a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:10:09.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.367+0000 7fea37405640 1 -- 172.21.15.42:0/1556603845 shutdown_connections 2024-09-16T04:10:09.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.367+0000 7fea37405640 1 -- 172.21.15.42:0/1556603845 wait complete. 2024-09-16T04:10:09.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.367+0000 7fea37405640 10 monclient: build_initial_monmap 2024-09-16T04:10:09.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.367+0000 7fea37405640 10 monclient: monmap: 2024-09-16T04:10:09.381 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:10:09.381 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:10:09.381 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:10:09.381 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:10:09.382 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:10:09.382 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:10:09.382 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:10:09.382 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:10:09.382 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:10:09.383 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:10:09.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.371+0000 7fea37405640 1 librados: starting msgr at 2024-09-16T04:10:09.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.371+0000 7fea37405640 1 librados: starting objecter 2024-09-16T04:10:09.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.371+0000 7fea37405640 1 Processor -- start 2024-09-16T04:10:09.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.371+0000 7fea37405640 1 -- start start 2024-09-16T04:10:09.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.371+0000 7fea37405640 1 librados: setting wanted keys 2024-09-16T04:10:09.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.371+0000 7fea37405640 1 librados: calling monclient init 2024-09-16T04:10:09.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.371+0000 7fea37405640 10 monclient: init 2024-09-16T04:10:09.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.371+0000 7fea37405640 10 monclient: _reopen_session rank -1 2024-09-16T04:10:09.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.371+0000 7fea37405640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:10:09.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.371+0000 7fea37405640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fea300a1330 0x7fea3013cf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:10:09.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.371+0000 7fea37405640 10 monclient(hunting): picked mon.noname-c con 0x7fea300a1330 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:10:09.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.371+0000 7fea37405640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fea300a1c50 0x7fea301350d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:10:09.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.371+0000 7fea37405640 10 monclient(hunting): picked mon.noname-b con 0x7fea300a1c50 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:10:09.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.371+0000 7fea37405640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fea30135610 0x7fea301ccdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:10:09.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.371+0000 7fea37405640 10 monclient(hunting): picked mon.noname-a con 0x7fea30135610 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:10:09.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.371+0000 7fea37405640 10 monclient(hunting): start opening mon connection 2024-09-16T04:10:09.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.371+0000 7fea37405640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fea3009ef30 con 0x7fea30135610 2024-09-16T04:10:09.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.371+0000 7fea37405640 10 monclient(hunting): start opening mon connection 2024-09-16T04:10:09.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.371+0000 7fea37405640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fea3009edb0 con 0x7fea300a1c50 2024-09-16T04:10:09.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.371+0000 7fea37405640 10 monclient(hunting): start opening mon connection 2024-09-16T04:10:09.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.371+0000 7fea37405640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fea30140b90 con 0x7fea300a1330 2024-09-16T04:10:09.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.371+0000 7fea37405640 10 monclient(hunting): _renew_subs 2024-09-16T04:10:09.385 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.371+0000 7fea37405640 10 monclient(hunting): authenticate will time out at 2656.341053s 2024-09-16T04:10:09.385 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.371+0000 7fea351a1640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fea300a1330 0x7fea3013cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:10:09.385 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.371+0000 7fea351a1640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fea300a1330 0x7fea3013cf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:48222/0 (socket says 172.21.15.42:48222) 2024-09-16T04:10:09.385 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.371+0000 7fea351a1640 1 -- 172.21.15.42:0/1212118359 learned_addr learned my addr 172.21.15.42:0/1212118359 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:10:09.385 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.371+0000 7fea351a1640 10 monclient(hunting): get_auth_request con 0x7fea300a1330 auth_method 0 2024-09-16T04:10:09.385 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.371+0000 7fea351a1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:10:09.385 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.371+0000 7fea351a1640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:09.385 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.371+0000 7fea351a1640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:09.385 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.371+0000 7fea359a2640 1 --2- 172.21.15.42:0/1212118359 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fea30135610 0x7fea301ccdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:10:09.386 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.371+0000 7fea349a0640 1 -- 172.21.15.42:0/1212118359 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fea300a1c50 msgr2=0x7fea301350d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T04:10:09.386 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.371+0000 7fea359a2640 10 monclient(hunting): get_auth_request con 0x7fea30135610 auth_method 0 2024-09-16T04:10:09.386 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.371+0000 7fea359a2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:10:09.386 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.371+0000 7fea359a2640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:09.386 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.371+0000 7fea359a2640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:09.386 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.371+0000 7fea349a0640 1 --2- 172.21.15.42:0/1212118359 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fea300a1c50 0x7fea301350d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:10:09.387 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.371+0000 7fea351a1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:10:09.387 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.371+0000 7fea351a1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:10:09.387 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.371+0000 7fea351a1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:10:09.387 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.371+0000 7fea359a2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:10:09.387 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.371+0000 7fea359a2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:10:09.387 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.371+0000 7fea359a2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:10:09.387 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.371+0000 7fea351a1640 1 -- 172.21.15.42:0/1212118359 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fea300a1330 msgr2=0x7fea3013cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-16T04:10:09.387 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.371+0000 7fea351a1640 1 -- 172.21.15.42:0/1212118359 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fea300a1330 msgr2=0x7fea3013cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:10:09.387 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.371+0000 7fea351a1640 1 --2- 172.21.15.42:0/1212118359 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fea300a1330 0x7fea3013cf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:10:09.387 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.371+0000 7fea351a1640 1 --2- 172.21.15.42:0/1212118359 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fea300a1330 0x7fea3013cf80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:10:09.387 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.371+0000 7fea359a2640 10 monclient(hunting): handle_auth_done global_id 1034265 payload 824 2024-09-16T04:10:09.387 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.371+0000 7fea359a2640 1 -- 172.21.15.42:0/1212118359 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fea300a1330 msgr2=0x7fea3013cf80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:10:09.387 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.371+0000 7fea359a2640 1 --2- 172.21.15.42:0/1212118359 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fea300a1330 0x7fea3013cf80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:10:09.388 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.371+0000 7fea359a2640 1 -- 172.21.15.42:0/1212118359 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fea300a1c50 msgr2=0x7fea301350d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:10:09.388 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.371+0000 7fea359a2640 1 --2- 172.21.15.42:0/1212118359 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fea300a1c50 0x7fea301350d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:10:09.388 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.371+0000 7fea359a2640 10 monclient: _finish_hunting 0 2024-09-16T04:10:09.388 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.371+0000 7fea359a2640 1 monclient: found mon.noname-a 2024-09-16T04:10:09.388 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.371+0000 7fea359a2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:10:09.388 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.371+0000 7fea359a2640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:10:09.388 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.371+0000 7fea359a2640 1 -- 172.21.15.42:0/1212118359 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea3011e790 con 0x7fea30135610 2024-09-16T04:10:09.388 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.371+0000 7fea359a2640 1 --2- 172.21.15.42:0/1212118359 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fea30135610 0x7fea301ccdc0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fea2800ba60 tx=0x7fea2800bf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:10:09.388 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.371+0000 7fea25ffb640 10 client.?.objecter ms_handle_connect 0x7fea30135610 2024-09-16T04:10:09.388 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.371+0000 7fea25ffb640 10 client.?.objecter resend_mon_ops 2024-09-16T04:10:09.388 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.371+0000 7fea25ffb640 1 -- 172.21.15.42:0/1212118359 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea2800cbe0 con 0x7fea30135610 2024-09-16T04:10:09.388 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.371+0000 7fea25ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:09.388 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.371+0000 7fea25ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:10:09.389 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.371+0000 7fea25ffb640 10 monclient: dump: 2024-09-16T04:10:09.389 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:10:09.389 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:10:09.389 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:09.389 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:09.389 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:09.390 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:10:09.390 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:10:09.390 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:10:09.390 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:10:09.390 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:10:09.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.371+0000 7fea25ffb640 10 monclient: _finish_auth 0 2024-09-16T04:10:09.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.371+0000 7fea25ffb640 10 monclient: _check_auth_tickets 2024-09-16T04:10:09.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.371+0000 7fea25ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:09.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.371+0000 7fea25ffb640 1 -- 172.21.15.42:0/1212118359 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea28010070 con 0x7fea30135610 2024-09-16T04:10:09.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.371+0000 7fea37405640 5 monclient: authenticate success, global_id 1034265 2024-09-16T04:10:09.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.371+0000 7fea37405640 10 monclient: _renew_subs 2024-09-16T04:10:09.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.371+0000 7fea37405640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:10:09.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.371+0000 7fea37405640 1 -- 172.21.15.42:0/1212118359 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea3011ea70 con 0x7fea30135610 2024-09-16T04:10:09.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.371+0000 7fea25ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:10:09.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.371+0000 7fea25ffb640 1 -- 172.21.15.42:0/1212118359 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea28015470 con 0x7fea30135610 2024-09-16T04:10:09.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.371+0000 7fea25ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:09.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.371+0000 7fea25ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:10:09.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.371+0000 7fea25ffb640 10 monclient: dump: 2024-09-16T04:10:09.391 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:10:09.391 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:10:09.391 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:09.391 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:09.391 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:09.391 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:10:09.391 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:10:09.391 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:10:09.392 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:10:09.392 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:10:09.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.371+0000 7fea277fe640 4 set_mon_vals no callback set 2024-09-16T04:10:09.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.371+0000 7fea37405640 10 client.1034265.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:10:09.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.371+0000 7fea37405640 10 monclient: _renew_subs 2024-09-16T04:10:09.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.371+0000 7fea37405640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:10:09.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.371+0000 7fea37405640 1 -- 172.21.15.42:0/1212118359 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea3013db40 con 0x7fea30135610 2024-09-16T04:10:09.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.371+0000 7fea37405640 1 librados: init done 2024-09-16T04:10:09.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.371+0000 7fea37405640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:09.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.371+0000 7fea37405640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:09.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.371+0000 7fea37405640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:10:09.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.371+0000 7fea25ffb640 1 -- 172.21.15.42:0/1212118359 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fea28015610 con 0x7fea30135610 2024-09-16T04:10:09.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.371+0000 7fea37405640 1 -- 172.21.15.42:0/1212118359 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fea300a1710 con 0x7fea30135610 2024-09-16T04:10:09.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.371+0000 7fea25ffb640 1 --2- 172.21.15.42:0/1212118359 >> v2:172.21.15.154:6812/254845342 conn(0x7fea140377c0 0x7fea14039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:10:09.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.371+0000 7fea25ffb640 1 -- 172.21.15.42:0/1212118359 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(706..706 src has 451..706) ==== 5040+0+0 (secure 0 0 0) 0x7fea28048d50 con 0x7fea30135610 2024-09-16T04:10:09.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.371+0000 7fea25ffb640 10 client.1034265.objecter ms_dispatch 0x7fea30000ba0 osd_map(706..706 src has 451..706) 2024-09-16T04:10:09.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.371+0000 7fea25ffb640 3 client.1034265.objecter handle_osd_map got epochs [706,706] > 0 2024-09-16T04:10:09.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.371+0000 7fea25ffb640 3 client.1034265.objecter handle_osd_map decoding full epoch 706 2024-09-16T04:10:09.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.371+0000 7fea25ffb640 20 client.1034265.objecter dump_active .. 0 homeless 2024-09-16T04:10:09.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.371+0000 7fea351a1640 1 --2- 172.21.15.42:0/1212118359 >> v2:172.21.15.154:6812/254845342 conn(0x7fea140377c0 0x7fea14039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:10:09.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.371+0000 7fea351a1640 10 monclient: get_auth_request con 0x7fea140377c0 auth_method 0 2024-09-16T04:10:09.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.375+0000 7fea351a1640 1 --2- 172.21.15.42:0/1212118359 >> v2:172.21.15.154:6812/254845342 conn(0x7fea140377c0 0x7fea14039c80 secure :-1 s=READY pgs=3927 cs=0 l=1 rev1=1 crypto rx=0x7fea2c01aaa0 tx=0x7fea2c0a6000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:10:09.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.375+0000 7fea25ffb640 10 client.1034265.objecter ms_handle_connect 0x7fea140377c0 2024-09-16T04:10:09.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.379+0000 7fea25ffb640 1 -- 172.21.15.42:0/1212118359 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7fea280149d0 con 0x7fea30135610 2024-09-16T04:10:09.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.379+0000 7fea25ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:09.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.379+0000 7fea25ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:10:09.602 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.599+0000 7fea37405640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:10:09.602 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.599+0000 7fea37405640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:10:09.602 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.599+0000 7fea37405640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:10:09.602 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.599+0000 7fea37405640 1 -- 172.21.15.42:0/1212118359 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fea3011c450 con 0x7fea30135610 2024-09-16T04:10:09.602 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.599+0000 7fea25ffb640 1 -- 172.21.15.42:0/1212118359 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1562 (secure 0 0 0) 0x7fea28015ce0 con 0x7fea30135610 2024-09-16T04:10:09.602 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.599+0000 7fea25ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:10:09.603 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.599+0000 7fea25ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:10:09.603 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":256,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:10:09.605 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.599+0000 7fea37405640 10 librados: watch_flush enter 2024-09-16T04:10:09.605 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.599+0000 7fea37405640 10 librados: watch_flush exit 2024-09-16T04:10:09.605 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.599+0000 7fea37405640 20 client.1034265.objecter shutdown clearing up homeless session... 2024-09-16T04:10:09.605 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.599+0000 7fea37405640 10 client.1034265.objecter successfully canceled tick 2024-09-16T04:10:09.605 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.599+0000 7fea37405640 1 -- 172.21.15.42:0/1212118359 >> v2:172.21.15.154:6812/254845342 conn(0x7fea140377c0 msgr2=0x7fea14039c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:09.605 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.599+0000 7fea37405640 1 --2- 172.21.15.42:0/1212118359 >> v2:172.21.15.154:6812/254845342 conn(0x7fea140377c0 0x7fea14039c80 secure :-1 s=READY pgs=3927 cs=0 l=1 rev1=1 crypto rx=0x7fea2c01aaa0 tx=0x7fea2c0a6000 comp rx=0 tx=0).stop 2024-09-16T04:10:09.605 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.599+0000 7fea37405640 10 monclient: shutdown 2024-09-16T04:10:09.605 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.599+0000 7fea37405640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:10:09.605 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.599+0000 7fea37405640 1 -- 172.21.15.42:0/1212118359 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fea30135610 msgr2=0x7fea301ccdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:09.605 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.599+0000 7fea37405640 1 --2- 172.21.15.42:0/1212118359 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fea30135610 0x7fea301ccdc0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fea2800ba60 tx=0x7fea2800bf20 comp rx=0 tx=0).stop 2024-09-16T04:10:09.606 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.599+0000 7fea37405640 1 -- 172.21.15.42:0/1212118359 shutdown_connections 2024-09-16T04:10:09.606 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.599+0000 7fea37405640 1 --2- 172.21.15.42:0/1212118359 >> v2:172.21.15.154:6812/254845342 conn(0x7fea140377c0 0x7fea14039c80 unknown :-1 s=CLOSED pgs=3927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:10:09.606 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.599+0000 7fea37405640 1 --2- 172.21.15.42:0/1212118359 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fea30135610 0x7fea301ccdc0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:10:09.606 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.599+0000 7fea37405640 1 --2- 172.21.15.42:0/1212118359 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fea300a1c50 0x7fea301350d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:10:09.606 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.599+0000 7fea37405640 1 --2- 172.21.15.42:0/1212118359 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fea300a1330 0x7fea3013cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:10:09.606 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.599+0000 7fea37405640 1 -- 172.21.15.42:0/1212118359 >> 172.21.15.42:0/1212118359 conn(0x7fea300a2420 msgr2=0x7fea300a4ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:10:09.606 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.599+0000 7fea37405640 1 -- 172.21.15.42:0/1212118359 shutdown_connections 2024-09-16T04:10:09.606 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.599+0000 7fea37405640 1 -- 172.21.15.42:0/1212118359 wait complete. 2024-09-16T04:10:09.606 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:09.599+0000 7fea37405640 1 librados: shutdown 2024-09-16T04:10:12.621 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:10:12.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:12.751+0000 7ff4ccf9f640 10 monclient: get_monmap_and_config 2024-09-16T04:10:12.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:12.751+0000 7ff4ccf9f640 10 monclient: build_initial_monmap 2024-09-16T04:10:12.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:12.751+0000 7ff4ccf9f640 10 monclient: monmap: 2024-09-16T04:10:12.755 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:10:12.755 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:10:12.755 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:10:12.753691+0000 2024-09-16T04:10:12.755 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:10:12.753691+0000 2024-09-16T04:10:12.755 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:10:12.755 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:10:12.756 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:10:12.756 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:10:12.756 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:10:12.756 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:10:12.756 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:10:12.756 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:10:12.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:12.751+0000 7ff4ccf9f640 1 Processor -- start 2024-09-16T04:10:12.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:12.751+0000 7ff4ccf9f640 1 -- start start 2024-09-16T04:10:12.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:12.751+0000 7ff4ccf9f640 10 monclient: init 2024-09-16T04:10:12.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:12.751+0000 7ff4ccf9f640 10 monclient: _reopen_session rank -1 2024-09-16T04:10:12.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:12.751+0000 7ff4ccf9f640 10 monclient: _add_conns ranks=[4,3,0,1,2] 2024-09-16T04:10:12.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:12.751+0000 7ff4ccf9f640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7ff4c80a39b0 0x7ff4c8133b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:10:12.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:12.751+0000 7ff4ccf9f640 10 monclient(hunting): picked mon.noname-c con 0x7ff4c80a39b0 addr v2:172.21.15.154:3300/0 2024-09-16T04:10:12.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:12.751+0000 7ff4ccf9f640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff4c80a3d90 0x7ff4c8137000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:10:12.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:12.751+0000 7ff4ccf9f640 10 monclient(hunting): picked mon.noname-a con 0x7ff4c80a3d90 addr v2:172.21.15.42:3300/0 2024-09-16T04:10:12.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:12.751+0000 7ff4ccf9f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff4c8137540 addr v1:172.21.15.42:6789/0 2024-09-16T04:10:12.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:12.751+0000 7ff4ccf9f640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff4c809c140 con 0x7ff4c8137540 2024-09-16T04:10:12.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:12.751+0000 7ff4ccf9f640 10 monclient(hunting): start opening mon connection 2024-09-16T04:10:12.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:12.751+0000 7ff4ccf9f640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7ff4c809c420 con 0x7ff4c80a3d90 2024-09-16T04:10:12.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:12.751+0000 7ff4ccf9f640 10 monclient(hunting): start opening mon connection 2024-09-16T04:10:12.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:12.751+0000 7ff4ccf9f640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7ff4c806dd60 con 0x7ff4c80a39b0 2024-09-16T04:10:12.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:12.751+0000 7ff4ccf9f640 10 monclient(hunting): _renew_subs 2024-09-16T04:10:12.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:12.751+0000 7ff4ccf9f640 10 monclient(hunting): authenticate will time out at 2659.723268s 2024-09-16T04:10:12.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:12.751+0000 7ff4c5d9b640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff4c80a3d90 0x7ff4c8137000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:10:12.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:12.751+0000 7ff4c6d9d640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7ff4c8137540 0x7ff4c813e390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:56574/0 (socket says 172.21.15.42:56574) 2024-09-16T04:10:12.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:12.751+0000 7ff4c6d9d640 1 -- 172.21.15.42:0/4063708353 learned_addr learned my addr 172.21.15.42:0/4063708353 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T04:10:12.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:12.751+0000 7ff4c659c640 1 -- 172.21.15.42:0/4063708353 >> v2:172.21.15.154:3300/0 conn(0x7ff4c80a39b0 msgr2=0x7ff4c8133b80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T04:10:12.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:12.751+0000 7ff4c5d9b640 10 monclient(hunting): get_auth_request con 0x7ff4c80a3d90 auth_method 0 2024-09-16T04:10:12.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:12.751+0000 7ff4c5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:10:12.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:12.751+0000 7ff4c659c640 1 --2- 172.21.15.42:0/4063708353 >> v2:172.21.15.154:3300/0 conn(0x7ff4c80a39b0 0x7ff4c8133b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:10:12.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:12.751+0000 7ff4c5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:12.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:12.751+0000 7ff4c5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:12.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:12.751+0000 7ff4c5d9b640 1 -- 172.21.15.42:0/4063708353 >> v2:172.21.15.42:3300/0 conn(0x7ff4c80a3d90 msgr2=0x7ff4c8137000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:10:12.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:12.751+0000 7ff4c5d9b640 1 -- 172.21.15.42:0/4063708353 >> v2:172.21.15.42:3300/0 conn(0x7ff4c80a3d90 msgr2=0x7ff4c8137000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:10:12.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:12.751+0000 7ff4c5d9b640 1 --2- 172.21.15.42:0/4063708353 >> v2:172.21.15.42:3300/0 conn(0x7ff4c80a3d90 0x7ff4c8137000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:10:12.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:12.751+0000 7ff4c5d9b640 1 --2- 172.21.15.42:0/4063708353 >> v2:172.21.15.42:3300/0 conn(0x7ff4c80a3d90 0x7ff4c8137000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:10:12.958 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:12.955+0000 7ff4c659c640 1 -- 172.21.15.42:0/4063708353 >> v2:172.21.15.154:3300/0 conn(0x7ff4c80a39b0 msgr2=0x7ff4c8133b80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T04:10:12.958 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:12.955+0000 7ff4c659c640 1 --2- 172.21.15.42:0/4063708353 >> v2:172.21.15.154:3300/0 conn(0x7ff4c80a39b0 0x7ff4c8133b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T04:10:12.958 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:12.955+0000 7ff4c5d9b640 1 --2- 172.21.15.42:0/4063708353 >> v2:172.21.15.42:3300/0 conn(0x7ff4c80a3d90 0x7ff4c8137000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:10:12.958 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:12.955+0000 7ff4c5d9b640 10 monclient(hunting): get_auth_request con 0x7ff4c80a3d90 auth_method 0 2024-09-16T04:10:12.958 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:12.955+0000 7ff4c5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:10:12.958 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:12.955+0000 7ff4c5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:12.958 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:12.955+0000 7ff4c5d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:10:12.959 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:12.955+0000 7ff4c5d9b640 1 -- 172.21.15.42:0/4063708353 >> v2:172.21.15.42:3300/0 conn(0x7ff4c80a3d90 msgr2=0x7ff4c8137000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:10:12.959 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:12.955+0000 7ff4c5d9b640 1 -- 172.21.15.42:0/4063708353 >> v2:172.21.15.42:3300/0 conn(0x7ff4c80a3d90 msgr2=0x7ff4c8137000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:10:12.959 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:12.955+0000 7ff4c5d9b640 1 --2- 172.21.15.42:0/4063708353 >> v2:172.21.15.42:3300/0 conn(0x7ff4c80a3d90 0x7ff4c8137000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:10:12.959 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:12.955+0000 7ff4c5d9b640 1 --2- 172.21.15.42:0/4063708353 >> v2:172.21.15.42:3300/0 conn(0x7ff4c80a3d90 0x7ff4c8137000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T04:10:13.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:13.359+0000 7ff4c659c640 1 -- 172.21.15.42:0/4063708353 >> v2:172.21.15.154:3300/0 conn(0x7ff4c80a39b0 msgr2=0x7ff4c8133b80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T04:10:13.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:13.359+0000 7ff4c5d9b640 1 --2- 172.21.15.42:0/4063708353 >> v2:172.21.15.42:3300/0 conn(0x7ff4c80a3d90 0x7ff4c8137000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:10:13.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:13.359+0000 7ff4c659c640 1 --2- 172.21.15.42:0/4063708353 >> v2:172.21.15.154:3300/0 conn(0x7ff4c80a39b0 0x7ff4c8133b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T04:10:13.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:13.359+0000 7ff4c5d9b640 10 monclient(hunting): get_auth_request con 0x7ff4c80a3d90 auth_method 0 2024-09-16T04:10:13.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:13.359+0000 7ff4c5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:10:13.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:13.359+0000 7ff4c5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:13.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:13.359+0000 7ff4c5d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:10:13.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:13.359+0000 7ff4c5d9b640 1 -- 172.21.15.42:0/4063708353 >> v2:172.21.15.42:3300/0 conn(0x7ff4c80a3d90 msgr2=0x7ff4c8137000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:10:13.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:13.359+0000 7ff4c5d9b640 1 -- 172.21.15.42:0/4063708353 >> v2:172.21.15.42:3300/0 conn(0x7ff4c80a3d90 msgr2=0x7ff4c8137000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:10:13.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:13.359+0000 7ff4c5d9b640 1 --2- 172.21.15.42:0/4063708353 >> v2:172.21.15.42:3300/0 conn(0x7ff4c80a3d90 0x7ff4c8137000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:10:13.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:13.359+0000 7ff4c5d9b640 1 --2- 172.21.15.42:0/4063708353 >> v2:172.21.15.42:3300/0 conn(0x7ff4c80a3d90 0x7ff4c8137000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T04:10:14.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:14.163+0000 7ff4c659c640 1 -- 172.21.15.42:0/4063708353 >> v2:172.21.15.154:3300/0 conn(0x7ff4c80a39b0 msgr2=0x7ff4c8133b80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T04:10:14.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:14.163+0000 7ff4c659c640 1 --2- 172.21.15.42:0/4063708353 >> v2:172.21.15.154:3300/0 conn(0x7ff4c80a39b0 0x7ff4c8133b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T04:10:14.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:14.163+0000 7ff4c5d9b640 1 --2- 172.21.15.42:0/4063708353 >> v2:172.21.15.42:3300/0 conn(0x7ff4c80a3d90 0x7ff4c8137000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:10:14.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:14.163+0000 7ff4c5d9b640 10 monclient(hunting): get_auth_request con 0x7ff4c80a3d90 auth_method 0 2024-09-16T04:10:14.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:14.163+0000 7ff4c5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:10:14.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:14.163+0000 7ff4c5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:14.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:14.163+0000 7ff4c5d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:10:14.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:14.163+0000 7ff4c5d9b640 1 -- 172.21.15.42:0/4063708353 >> v2:172.21.15.42:3300/0 conn(0x7ff4c80a3d90 msgr2=0x7ff4c8137000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:10:14.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:14.163+0000 7ff4c5d9b640 1 -- 172.21.15.42:0/4063708353 >> v2:172.21.15.42:3300/0 conn(0x7ff4c80a3d90 msgr2=0x7ff4c8137000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:10:14.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:14.163+0000 7ff4c5d9b640 1 --2- 172.21.15.42:0/4063708353 >> v2:172.21.15.42:3300/0 conn(0x7ff4c80a3d90 0x7ff4c8137000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:10:14.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:14.163+0000 7ff4c5d9b640 1 --2- 172.21.15.42:0/4063708353 >> v2:172.21.15.42:3300/0 conn(0x7ff4c80a3d90 0x7ff4c8137000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T04:10:15.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:15.751+0000 7ff4bffff640 10 monclient(hunting): tick 2024-09-16T04:10:15.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:15.751+0000 7ff4bffff640 10 monclient(hunting): _check_auth_tickets 2024-09-16T04:10:15.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:15.751+0000 7ff4bffff640 1 monclient(hunting): continuing hunt 2024-09-16T04:10:15.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:15.751+0000 7ff4bffff640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T04:10:15.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:15.751+0000 7ff4bffff640 1 -- 172.21.15.42:0/4063708353 >> v2:172.21.15.154:3300/0 conn(0x7ff4c80a39b0 msgr2=0x7ff4c8133b80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:10:15.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:15.751+0000 7ff4bffff640 1 --2- 172.21.15.42:0/4063708353 >> v2:172.21.15.154:3300/0 conn(0x7ff4c80a39b0 0x7ff4c8133b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:10:15.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:15.751+0000 7ff4bffff640 1 -- 172.21.15.42:0/4063708353 >> v2:172.21.15.42:3300/0 conn(0x7ff4c80a3d90 msgr2=0x7ff4c8137000 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:10:15.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:15.751+0000 7ff4bffff640 1 --2- 172.21.15.42:0/4063708353 >> v2:172.21.15.42:3300/0 conn(0x7ff4c80a3d90 0x7ff4c8137000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:10:15.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:15.751+0000 7ff4bffff640 1 -- 172.21.15.42:0/4063708353 >> v1:172.21.15.42:6789/0 conn(0x7ff4c8137540 legacy=0x7ff4c813e390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:15.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:15.751+0000 7ff4bffff640 10 monclient: _add_conns ranks=[1,2] 2024-09-16T04:10:15.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:15.751+0000 7ff4bffff640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff4a80036c0 addr v1:172.21.15.154:6789/0 2024-09-16T04:10:15.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:15.751+0000 7ff4bffff640 1 --2- 172.21.15.42:0/4063708353 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff4a8007390 0x7ff4a8009850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:10:15.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:15.751+0000 7ff4bffff640 10 monclient(hunting): picked mon.noname-b con 0x7ff4a8007390 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:10:15.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:15.751+0000 7ff4bffff640 1 -- 172.21.15.42:0/4063708353 --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff4c80a5e20 con 0x7ff4a80036c0 2024-09-16T04:10:15.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:15.751+0000 7ff4bffff640 10 monclient(hunting): start opening mon connection 2024-09-16T04:10:15.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:15.751+0000 7ff4bffff640 1 -- 172.21.15.42:0/4063708353 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7ff4c809c420 con 0x7ff4a8007390 2024-09-16T04:10:15.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:15.751+0000 7ff4bffff640 10 monclient(hunting): _renew_subs 2024-09-16T04:10:15.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:15.751+0000 7ff4c659c640 1 -- 172.21.15.42:0/4063708353 >> v1:172.21.15.154:6789/0 conn(0x7ff4a80036c0 legacy=0x7ff4a8005b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.154:6789/0 2024-09-16T04:10:15.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:15.751+0000 7ff4c5d9b640 1 --2- 172.21.15.42:0/4063708353 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff4a8007390 0x7ff4a8009850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:10:15.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:15.755+0000 7ff4c5d9b640 10 monclient(hunting): get_auth_request con 0x7ff4a8007390 auth_method 0 2024-09-16T04:10:15.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:15.755+0000 7ff4c5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:10:15.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:15.755+0000 7ff4c5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:15.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:15.755+0000 7ff4c5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:15.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:15.755+0000 7ff4c5d9b640 1 -- 172.21.15.42:0/4063708353 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff4a8007390 msgr2=0x7ff4a8009850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:10:15.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:15.755+0000 7ff4c5d9b640 1 -- 172.21.15.42:0/4063708353 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff4a8007390 msgr2=0x7ff4a8009850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:10:15.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:15.755+0000 7ff4c5d9b640 1 --2- 172.21.15.42:0/4063708353 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff4a8007390 0x7ff4a8009850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:10:15.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:15.755+0000 7ff4c5d9b640 1 --2- 172.21.15.42:0/4063708353 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff4a8007390 0x7ff4a8009850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:10:15.958 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:15.955+0000 7ff4c659c640 1 -- 172.21.15.42:0/4063708353 >> v1:172.21.15.154:6789/0 conn(0x7ff4a80036c0 legacy=0x7ff4a8005b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.154:6789/0 2024-09-16T04:10:15.962 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:15.959+0000 7ff4c5d9b640 1 --2- 172.21.15.42:0/4063708353 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff4a8007390 0x7ff4a8009850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:10:15.962 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:15.959+0000 7ff4c5d9b640 10 monclient(hunting): get_auth_request con 0x7ff4a8007390 auth_method 0 2024-09-16T04:10:15.962 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:15.959+0000 7ff4c5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:10:15.962 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:15.959+0000 7ff4c5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:15.962 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:15.959+0000 7ff4c5d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:10:15.962 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:15.959+0000 7ff4c5d9b640 1 -- 172.21.15.42:0/4063708353 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff4a8007390 msgr2=0x7ff4a8009850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:10:15.962 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:15.959+0000 7ff4c5d9b640 1 -- 172.21.15.42:0/4063708353 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff4a8007390 msgr2=0x7ff4a8009850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:10:15.962 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:15.959+0000 7ff4c5d9b640 1 --2- 172.21.15.42:0/4063708353 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff4a8007390 0x7ff4a8009850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:10:15.962 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:15.959+0000 7ff4c5d9b640 1 --2- 172.21.15.42:0/4063708353 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff4a8007390 0x7ff4a8009850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T04:10:16.361 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:16.359+0000 7ff4c659c640 1 -- 172.21.15.42:0/4063708353 >> v1:172.21.15.154:6789/0 conn(0x7ff4a80036c0 legacy=0x7ff4a8005b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.154:6789/0 2024-09-16T04:10:16.366 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:16.363+0000 7ff4c5d9b640 1 --2- 172.21.15.42:0/4063708353 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff4a8007390 0x7ff4a8009850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:10:16.366 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:16.363+0000 7ff4c5d9b640 10 monclient(hunting): get_auth_request con 0x7ff4a8007390 auth_method 0 2024-09-16T04:10:16.366 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:16.363+0000 7ff4c5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:10:16.366 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:16.363+0000 7ff4c5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:16.366 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:16.363+0000 7ff4c5d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:10:16.366 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:16.363+0000 7ff4c5d9b640 1 -- 172.21.15.42:0/4063708353 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff4a8007390 msgr2=0x7ff4a8009850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:10:16.366 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:16.363+0000 7ff4c5d9b640 1 -- 172.21.15.42:0/4063708353 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff4a8007390 msgr2=0x7ff4a8009850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:10:16.366 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:16.363+0000 7ff4c5d9b640 1 --2- 172.21.15.42:0/4063708353 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff4a8007390 0x7ff4a8009850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:10:16.366 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:16.363+0000 7ff4c5d9b640 1 --2- 172.21.15.42:0/4063708353 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff4a8007390 0x7ff4a8009850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T04:10:17.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:17.163+0000 7ff4c659c640 1 -- 172.21.15.42:0/4063708353 >> v1:172.21.15.154:6789/0 conn(0x7ff4a80036c0 legacy=0x7ff4a8005b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.154:6789/0 2024-09-16T04:10:17.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:17.167+0000 7ff4c5d9b640 1 --2- 172.21.15.42:0/4063708353 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff4a8007390 0x7ff4a8009850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:10:17.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:17.167+0000 7ff4c5d9b640 10 monclient(hunting): get_auth_request con 0x7ff4a8007390 auth_method 0 2024-09-16T04:10:17.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:17.167+0000 7ff4c5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:10:17.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:17.167+0000 7ff4c5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:17.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:17.167+0000 7ff4c5d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:10:17.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:17.167+0000 7ff4c5d9b640 1 -- 172.21.15.42:0/4063708353 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff4a8007390 msgr2=0x7ff4a8009850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:10:17.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:17.167+0000 7ff4c5d9b640 1 -- 172.21.15.42:0/4063708353 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff4a8007390 msgr2=0x7ff4a8009850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:10:17.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:17.167+0000 7ff4c5d9b640 1 --2- 172.21.15.42:0/4063708353 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff4a8007390 0x7ff4a8009850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:10:17.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:17.167+0000 7ff4c5d9b640 1 --2- 172.21.15.42:0/4063708353 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff4a8007390 0x7ff4a8009850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T04:10:18.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.751+0000 7ff4bffff640 10 monclient(hunting): tick 2024-09-16T04:10:18.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.751+0000 7ff4bffff640 10 monclient(hunting): _check_auth_tickets 2024-09-16T04:10:18.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.751+0000 7ff4bffff640 1 monclient(hunting): continuing hunt 2024-09-16T04:10:18.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.751+0000 7ff4bffff640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T04:10:18.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.751+0000 7ff4bffff640 1 -- 172.21.15.42:0/4063708353 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff4a8007390 msgr2=0x7ff4a8009850 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:10:18.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.751+0000 7ff4bffff640 1 --2- 172.21.15.42:0/4063708353 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff4a8007390 0x7ff4a8009850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:10:18.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.751+0000 7ff4bffff640 1 -- 172.21.15.42:0/4063708353 >> v1:172.21.15.154:6789/0 conn(0x7ff4a80036c0 legacy=0x7ff4a8005b20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:10:18.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.751+0000 7ff4c6d9d640 1 -- 172.21.15.42:0/4063708353 reap_dead start 2024-09-16T04:10:18.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.751+0000 7ff4bffff640 10 monclient: _add_conns ranks=[0,4,3,1,2] 2024-09-16T04:10:18.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.751+0000 7ff4bffff640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff4a8009f80 addr v1:172.21.15.42:6789/0 2024-09-16T04:10:18.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.751+0000 7ff4bffff640 1 --2- 172.21.15.42:0/4063708353 >> v2:172.21.15.154:3300/0 conn(0x7ff4a800ab10 0x7ff4a8009850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:10:18.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.755+0000 7ff4bffff640 10 monclient(hunting): picked mon.noname-c con 0x7ff4a800ab10 addr v2:172.21.15.154:3300/0 2024-09-16T04:10:18.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.755+0000 7ff4bffff640 1 --2- 172.21.15.42:0/4063708353 >> v2:172.21.15.42:3300/0 conn(0x7ff4a8005c60 0x7ff4a800d440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:10:18.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.755+0000 7ff4bffff640 10 monclient(hunting): picked mon.noname-a con 0x7ff4a8005c60 addr v2:172.21.15.42:3300/0 2024-09-16T04:10:18.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.755+0000 7ff4bffff640 1 -- 172.21.15.42:0/4063708353 --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff4a800b020 con 0x7ff4a8009f80 2024-09-16T04:10:18.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.755+0000 7ff4bffff640 10 monclient(hunting): start opening mon connection 2024-09-16T04:10:18.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.755+0000 7ff4bffff640 1 -- 172.21.15.42:0/4063708353 --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7ff4c809c420 con 0x7ff4a8005c60 2024-09-16T04:10:18.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.755+0000 7ff4bffff640 10 monclient(hunting): start opening mon connection 2024-09-16T04:10:18.761 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.755+0000 7ff4bffff640 1 -- 172.21.15.42:0/4063708353 --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7ff4c806dd60 con 0x7ff4a800ab10 2024-09-16T04:10:18.761 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.755+0000 7ff4bffff640 10 monclient(hunting): _renew_subs 2024-09-16T04:10:18.761 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.755+0000 7ff4c6d9d640 1 --2- 172.21.15.42:0/4063708353 >> v2:172.21.15.42:3300/0 conn(0x7ff4a8005c60 0x7ff4a800d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:10:18.761 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.755+0000 7ff4c5d9b640 1 -- 172.21.15.42:0/4063708353 >> v2:172.21.15.154:3300/0 conn(0x7ff4a800ab10 msgr2=0x7ff4a8009850 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T04:10:18.761 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.755+0000 7ff4c6d9d640 10 monclient(hunting): get_auth_request con 0x7ff4a8005c60 auth_method 0 2024-09-16T04:10:18.761 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.755+0000 7ff4c6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:10:18.761 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.755+0000 7ff4c6d9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:18.761 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.755+0000 7ff4c6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:18.761 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.755+0000 7ff4c5d9b640 1 --2- 172.21.15.42:0/4063708353 >> v2:172.21.15.154:3300/0 conn(0x7ff4a800ab10 0x7ff4a8009850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:10:18.761 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.755+0000 7ff4c6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:10:18.761 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.755+0000 7ff4c6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:10:18.761 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.755+0000 7ff4c6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:10:18.761 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.755+0000 7ff4c559a640 1 -- 172.21.15.42:0/4063708353 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3980430067 0 0) 0x7ff4a800b020 con 0x7ff4a8009f80 2024-09-16T04:10:18.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.755+0000 7ff4c559a640 10 monclient(hunting): _negotiate 2024-09-16T04:10:18.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.755+0000 7ff4c559a640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:18.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.755+0000 7ff4c559a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:18.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.755+0000 7ff4c559a640 10 monclient(hunting): my global_id is 1034271 2024-09-16T04:10:18.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.755+0000 7ff4c559a640 1 -- 172.21.15.42:0/4063708353 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff4b4003530 con 0x7ff4a8009f80 2024-09-16T04:10:18.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.755+0000 7ff4c6d9d640 10 monclient(hunting): handle_auth_done global_id 1034268 payload 274 2024-09-16T04:10:18.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.755+0000 7ff4c6d9d640 1 -- 172.21.15.42:0/4063708353 >> v2:172.21.15.154:3300/0 conn(0x7ff4a800ab10 msgr2=0x7ff4a8009850 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:10:18.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.755+0000 7ff4c6d9d640 1 --2- 172.21.15.42:0/4063708353 >> v2:172.21.15.154:3300/0 conn(0x7ff4a800ab10 0x7ff4a8009850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:10:18.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.755+0000 7ff4c6d9d640 1 -- 172.21.15.42:0/4063708353 >> v1:172.21.15.42:6789/0 conn(0x7ff4a8009f80 legacy=0x7ff4a800a360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:18.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.755+0000 7ff4c6d9d640 10 monclient: _finish_hunting 0 2024-09-16T04:10:18.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.755+0000 7ff4c6d9d640 1 monclient: found mon.noname-a 2024-09-16T04:10:18.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.755+0000 7ff4c6d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:10:18.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.755+0000 7ff4c6d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:10:18.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.755+0000 7ff4c6d9d640 1 -- 172.21.15.42:0/4063708353 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4c80a5e20 con 0x7ff4a8005c60 2024-09-16T04:10:18.764 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.759+0000 7ff4c6d9d640 1 --2- 172.21.15.42:0/4063708353 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff4a8005c60 0x7ff4a800d440 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7ff4c80a7a40 tx=0x7ff4c001c190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3eb6ef23bd415009 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:10:18.772 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.759+0000 7ff4c559a640 1 -- 172.21.15.42:0/4063708353 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4c00a4bc0 con 0x7ff4a8005c60 2024-09-16T04:10:18.773 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.759+0000 7ff4c559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:18.773 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.759+0000 7ff4c559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:10:18.773 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.759+0000 7ff4c559a640 10 monclient: dump: 2024-09-16T04:10:18.773 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:10:18.773 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:10:18.773 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:18.773 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:18.773 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:18.773 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:10:18.773 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:10:18.773 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:10:18.773 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:10:18.774 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:10:18.774 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.759+0000 7ff4c559a640 10 monclient: _finish_auth 0 2024-09-16T04:10:18.774 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.759+0000 7ff4c559a640 10 monclient: _check_auth_tickets 2024-09-16T04:10:18.774 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.759+0000 7ff4c559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:18.774 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.759+0000 7ff4ccf9f640 5 monclient: authenticate success, global_id 1034268 2024-09-16T04:10:18.774 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.759+0000 7ff4c559a640 1 -- 172.21.15.42:0/4063708353 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4c00a8020 con 0x7ff4a8005c60 2024-09-16T04:10:18.774 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.759+0000 7ff4c559a640 10 monclient: handle_config config(0 keys) 2024-09-16T04:10:18.774 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.759+0000 7ff4ccf9f640 10 monclient: get_monmap_and_config success 2024-09-16T04:10:18.774 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.759+0000 7ff4ccf9f640 4 set_mon_vals no callback set 2024-09-16T04:10:18.774 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.759+0000 7ff4c559a640 1 -- 172.21.15.42:0/4063708353 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4c00048b0 con 0x7ff4a8005c60 2024-09-16T04:10:18.774 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.759+0000 7ff4c559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:18.774 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.759+0000 7ff4c559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:10:18.774 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.759+0000 7ff4c559a640 10 monclient: dump: 2024-09-16T04:10:18.774 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:10:18.775 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:10:18.775 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:18.775 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:18.775 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:18.775 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:10:18.775 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:10:18.775 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:10:18.775 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:10:18.775 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:10:18.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.763+0000 7ff4ccf9f640 10 monclient: shutdown 2024-09-16T04:10:18.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.763+0000 7ff4ccf9f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:10:18.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.763+0000 7ff4ccf9f640 1 -- 172.21.15.42:0/4063708353 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff4a8005c60 msgr2=0x7ff4a800d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:18.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.763+0000 7ff4ccf9f640 1 --2- 172.21.15.42:0/4063708353 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff4a8005c60 0x7ff4a800d440 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7ff4c80a7a40 tx=0x7ff4c001c190 comp rx=0 tx=0).stop 2024-09-16T04:10:18.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.763+0000 7ff4ccf9f640 1 -- 172.21.15.42:0/4063708353 shutdown_connections 2024-09-16T04:10:18.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.763+0000 7ff4ccf9f640 1 --2- 172.21.15.42:0/4063708353 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff4a8005c60 0x7ff4a800d440 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:10:18.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.763+0000 7ff4ccf9f640 1 --2- 172.21.15.42:0/4063708353 >> v2:172.21.15.154:3300/0 conn(0x7ff4a800ab10 0x7ff4a8009850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:10:18.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.763+0000 7ff4ccf9f640 1 -- 172.21.15.42:0/4063708353 >> 172.21.15.42:0/4063708353 conn(0x7ff4c80a9fb0 msgr2=0x7ff4c80aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:10:18.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.763+0000 7ff4ccf9f640 1 -- 172.21.15.42:0/4063708353 shutdown_connections 2024-09-16T04:10:18.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.763+0000 7ff4ccf9f640 1 -- 172.21.15.42:0/4063708353 wait complete. 2024-09-16T04:10:18.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.763+0000 7ff4ccf9f640 10 monclient: build_initial_monmap 2024-09-16T04:10:18.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.763+0000 7ff4ccf9f640 10 monclient: monmap: 2024-09-16T04:10:18.776 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:10:18.776 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:10:18.776 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:10:18.776 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:10:18.776 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:10:18.777 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:10:18.777 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:10:18.777 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:10:18.777 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:10:18.777 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:10:18.777 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.763+0000 7ff4ccf9f640 1 librados: starting msgr at 2024-09-16T04:10:18.777 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.763+0000 7ff4ccf9f640 1 librados: starting objecter 2024-09-16T04:10:18.777 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.763+0000 7ff4ccf9f640 1 Processor -- start 2024-09-16T04:10:18.777 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.763+0000 7ff4ccf9f640 1 -- start start 2024-09-16T04:10:18.777 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.763+0000 7ff4ccf9f640 1 librados: setting wanted keys 2024-09-16T04:10:18.777 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.763+0000 7ff4ccf9f640 1 librados: calling monclient init 2024-09-16T04:10:18.777 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.763+0000 7ff4ccf9f640 10 monclient: init 2024-09-16T04:10:18.777 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.763+0000 7ff4ccf9f640 10 monclient: _reopen_session rank -1 2024-09-16T04:10:18.778 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.763+0000 7ff4ccf9f640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:10:18.778 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.763+0000 7ff4ccf9f640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff4a8005c60 0x7ff4c81cfa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:10:18.778 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.763+0000 7ff4ccf9f640 10 monclient(hunting): picked mon.noname-a con 0x7ff4a8005c60 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:10:18.778 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.763+0000 7ff4ccf9f640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff4a8009f80 0x7ff4c81cffd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:10:18.778 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.763+0000 7ff4ccf9f640 10 monclient(hunting): picked mon.noname-c con 0x7ff4a8009f80 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:10:18.778 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.763+0000 7ff4ccf9f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff4a800ab10 0x7ff4c81d0510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:10:18.778 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.763+0000 7ff4ccf9f640 10 monclient(hunting): picked mon.noname-b con 0x7ff4a800ab10 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:10:18.778 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.763+0000 7ff4ccf9f640 10 monclient(hunting): start opening mon connection 2024-09-16T04:10:18.778 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.763+0000 7ff4ccf9f640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7ff4c813a700 con 0x7ff4a8005c60 2024-09-16T04:10:18.778 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.763+0000 7ff4ccf9f640 10 monclient(hunting): start opening mon connection 2024-09-16T04:10:18.778 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.763+0000 7ff4ccf9f640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff4c813a580 con 0x7ff4a800ab10 2024-09-16T04:10:18.778 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.763+0000 7ff4ccf9f640 10 monclient(hunting): start opening mon connection 2024-09-16T04:10:18.778 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.763+0000 7ff4ccf9f640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7ff4c813a880 con 0x7ff4a8009f80 2024-09-16T04:10:18.778 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.763+0000 7ff4ccf9f640 10 monclient(hunting): _renew_subs 2024-09-16T04:10:18.779 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.763+0000 7ff4ccf9f640 10 monclient(hunting): authenticate will time out at 2665.735920s 2024-09-16T04:10:18.779 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.767+0000 7ff4c5d9b640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff4a8009f80 0x7ff4c81cffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:10:18.779 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.767+0000 7ff4c6d9d640 1 -- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff4a800ab10 msgr2=0x7ff4c81d0510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T04:10:18.779 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.767+0000 7ff4c6d9d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff4a800ab10 0x7ff4c81d0510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:10:18.779 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.767+0000 7ff4c5d9b640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff4a8009f80 0x7ff4c81cffd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:40382/0 (socket says 172.21.15.42:40382) 2024-09-16T04:10:18.779 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.767+0000 7ff4c659c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff4a8005c60 0x7ff4c81cfa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:10:18.779 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.767+0000 7ff4c5d9b640 1 -- 172.21.15.42:0/613166006 learned_addr learned my addr 172.21.15.42:0/613166006 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:10:18.779 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.767+0000 7ff4c5d9b640 10 monclient(hunting): get_auth_request con 0x7ff4a8009f80 auth_method 0 2024-09-16T04:10:18.779 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.767+0000 7ff4c5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:10:18.779 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.767+0000 7ff4c5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:18.779 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.767+0000 7ff4c5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:18.779 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.767+0000 7ff4c659c640 10 monclient(hunting): get_auth_request con 0x7ff4a8005c60 auth_method 0 2024-09-16T04:10:18.779 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.767+0000 7ff4c659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:10:18.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.767+0000 7ff4c659c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:18.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.767+0000 7ff4c659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:18.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.767+0000 7ff4c5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:10:18.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.767+0000 7ff4c5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:10:18.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.767+0000 7ff4c5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:10:18.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.767+0000 7ff4c659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:10:18.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.767+0000 7ff4c659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:10:18.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.767+0000 7ff4c659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:10:18.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.767+0000 7ff4c5d9b640 10 monclient(hunting): handle_auth_done global_id 1014407 payload 824 2024-09-16T04:10:18.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.767+0000 7ff4c5d9b640 1 -- 172.21.15.42:0/613166006 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff4a800ab10 msgr2=0x7ff4c81d0510 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:10:18.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.767+0000 7ff4c5d9b640 1 --2- 172.21.15.42:0/613166006 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff4a800ab10 0x7ff4c81d0510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:10:18.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.767+0000 7ff4c5d9b640 1 -- 172.21.15.42:0/613166006 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff4a8005c60 msgr2=0x7ff4c81cfa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:18.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.767+0000 7ff4c5d9b640 1 --2- 172.21.15.42:0/613166006 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff4a8005c60 0x7ff4c81cfa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:10:18.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.767+0000 7ff4c5d9b640 10 monclient: _finish_hunting 0 2024-09-16T04:10:18.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.767+0000 7ff4c5d9b640 1 monclient: found mon.noname-c 2024-09-16T04:10:18.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.767+0000 7ff4c5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:10:18.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.767+0000 7ff4c5d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T04:10:18.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.767+0000 7ff4c5d9b640 1 -- 172.21.15.42:0/613166006 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4c8122e00 con 0x7ff4a8009f80 2024-09-16T04:10:18.782 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.767+0000 7ff4c5d9b640 1 --2- 172.21.15.42:0/613166006 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff4a8009f80 0x7ff4c81cffd0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7ff4b80046b0 tx=0x7ff4b8017780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:10:18.782 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.767+0000 7ff4beffd640 10 client.?.objecter ms_handle_connect 0x7ff4a8009f80 2024-09-16T04:10:18.782 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.767+0000 7ff4beffd640 10 client.?.objecter resend_mon_ops 2024-09-16T04:10:18.782 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.767+0000 7ff4beffd640 1 -- 172.21.15.42:0/613166006 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4b80047a0 con 0x7ff4a8009f80 2024-09-16T04:10:18.782 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.767+0000 7ff4beffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:18.782 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.767+0000 7ff4beffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:10:18.782 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.767+0000 7ff4beffd640 10 monclient: dump: 2024-09-16T04:10:18.782 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:10:18.782 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:10:18.782 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:18.782 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:18.782 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:18.782 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:10:18.782 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:10:18.783 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:10:18.783 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:10:18.783 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:10:18.783 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.767+0000 7ff4beffd640 10 monclient: _finish_auth 0 2024-09-16T04:10:18.783 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.767+0000 7ff4beffd640 10 monclient: _check_auth_tickets 2024-09-16T04:10:18.783 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.767+0000 7ff4beffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:18.783 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.767+0000 7ff4beffd640 1 -- 172.21.15.42:0/613166006 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4b8003d80 con 0x7ff4a8009f80 2024-09-16T04:10:18.783 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.767+0000 7ff4ccf9f640 5 monclient: authenticate success, global_id 1014407 2024-09-16T04:10:18.783 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.767+0000 7ff4ccf9f640 10 monclient: _renew_subs 2024-09-16T04:10:18.783 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.767+0000 7ff4ccf9f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:10:18.783 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.767+0000 7ff4ccf9f640 1 -- 172.21.15.42:0/613166006 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4c8133230 con 0x7ff4a8009f80 2024-09-16T04:10:18.783 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.767+0000 7ff4beffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:10:18.783 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.767+0000 7ff4beffd640 1 -- 172.21.15.42:0/613166006 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4b8021670 con 0x7ff4a8009f80 2024-09-16T04:10:18.783 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.767+0000 7ff4beffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:18.784 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.767+0000 7ff4beffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:10:18.784 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.767+0000 7ff4beffd640 10 monclient: dump: 2024-09-16T04:10:18.784 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:10:18.784 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:10:18.784 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:18.784 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:18.784 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:18.784 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:10:18.784 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:10:18.784 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:10:18.784 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:10:18.784 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:10:18.784 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.767+0000 7ff4c4d99640 4 set_mon_vals no callback set 2024-09-16T04:10:18.785 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.767+0000 7ff4ccf9f640 10 client.1014407.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:10:18.785 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.767+0000 7ff4ccf9f640 10 monclient: _renew_subs 2024-09-16T04:10:18.785 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.767+0000 7ff4ccf9f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:10:18.785 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.767+0000 7ff4ccf9f640 1 -- 172.21.15.42:0/613166006 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4c8130d60 con 0x7ff4a8009f80 2024-09-16T04:10:18.785 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.767+0000 7ff4ccf9f640 1 librados: init done 2024-09-16T04:10:18.785 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.767+0000 7ff4ccf9f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:18.785 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.767+0000 7ff4ccf9f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:18.785 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.767+0000 7ff4ccf9f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:10:18.785 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.767+0000 7ff4beffd640 1 -- 172.21.15.42:0/613166006 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7ff4b8015040 con 0x7ff4a8009f80 2024-09-16T04:10:18.785 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.767+0000 7ff4ccf9f640 1 -- 172.21.15.42:0/613166006 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff4c813aa00 con 0x7ff4a8009f80 2024-09-16T04:10:18.785 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.767+0000 7ff4beffd640 1 --2- 172.21.15.42:0/613166006 >> v2:172.21.15.154:6812/254845342 conn(0x7ff4a8039230 0x7ff4a803b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:10:18.785 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.767+0000 7ff4beffd640 1 -- 172.21.15.42:0/613166006 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(706..706 src has 451..706) ==== 5040+0+0 (secure 0 0 0) 0x7ff4b804a250 con 0x7ff4a8009f80 2024-09-16T04:10:18.785 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.767+0000 7ff4beffd640 10 client.1014407.objecter ms_dispatch 0x7ff4c8000ba0 osd_map(706..706 src has 451..706) 2024-09-16T04:10:18.785 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.767+0000 7ff4beffd640 3 client.1014407.objecter handle_osd_map got epochs [706,706] > 0 2024-09-16T04:10:18.786 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.767+0000 7ff4beffd640 3 client.1014407.objecter handle_osd_map decoding full epoch 706 2024-09-16T04:10:18.786 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.767+0000 7ff4beffd640 20 client.1014407.objecter dump_active .. 0 homeless 2024-09-16T04:10:18.786 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.767+0000 7ff4c659c640 1 --2- 172.21.15.42:0/613166006 >> v2:172.21.15.154:6812/254845342 conn(0x7ff4a8039230 0x7ff4a803b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:10:18.786 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.767+0000 7ff4c659c640 10 monclient: get_auth_request con 0x7ff4a8039230 auth_method 0 2024-09-16T04:10:18.786 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.767+0000 7ff4c659c640 1 --2- 172.21.15.42:0/613166006 >> v2:172.21.15.154:6812/254845342 conn(0x7ff4a8039230 0x7ff4a803b6f0 secure :-1 s=READY pgs=3928 cs=0 l=1 rev1=1 crypto rx=0x7ff4b0003ad0 tx=0x7ff4b0009300 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:10:18.786 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.767+0000 7ff4beffd640 10 client.1014407.objecter ms_handle_connect 0x7ff4a8039230 2024-09-16T04:10:18.787 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.771+0000 7ff4beffd640 1 -- 172.21.15.42:0/613166006 <== mon.2 v2:172.21.15.42:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7ff4b802ad90 con 0x7ff4a8009f80 2024-09-16T04:10:18.787 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.771+0000 7ff4beffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:18.787 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:18.771+0000 7ff4beffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:10:19.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.007+0000 7ff4ccf9f640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:10:19.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.007+0000 7ff4ccf9f640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:10:19.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.007+0000 7ff4ccf9f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:10:19.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.007+0000 7ff4ccf9f640 1 -- 172.21.15.42:0/613166006 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7ff4c80a5750 con 0x7ff4a8009f80 2024-09-16T04:10:19.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.007+0000 7ff4beffd640 1 -- 172.21.15.42:0/613166006 <== mon.2 v2:172.21.15.42:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1555 (secure 0 0 0) 0x7ff4b8016570 con 0x7ff4a8009f80 2024-09-16T04:10:19.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.007+0000 7ff4beffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:10:19.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.007+0000 7ff4beffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:10:19.011 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":260,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:10:19.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.007+0000 7ff4ccf9f640 10 librados: watch_flush enter 2024-09-16T04:10:19.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.007+0000 7ff4ccf9f640 10 librados: watch_flush exit 2024-09-16T04:10:19.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.007+0000 7ff4ccf9f640 20 client.1014407.objecter shutdown clearing up homeless session... 2024-09-16T04:10:19.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.007+0000 7ff4ccf9f640 10 client.1014407.objecter successfully canceled tick 2024-09-16T04:10:19.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.007+0000 7ff4ccf9f640 1 -- 172.21.15.42:0/613166006 >> v2:172.21.15.154:6812/254845342 conn(0x7ff4a8039230 msgr2=0x7ff4a803b6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:19.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.007+0000 7ff4ccf9f640 1 --2- 172.21.15.42:0/613166006 >> v2:172.21.15.154:6812/254845342 conn(0x7ff4a8039230 0x7ff4a803b6f0 secure :-1 s=READY pgs=3928 cs=0 l=1 rev1=1 crypto rx=0x7ff4b0003ad0 tx=0x7ff4b0009300 comp rx=0 tx=0).stop 2024-09-16T04:10:19.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.007+0000 7ff4ccf9f640 10 monclient: shutdown 2024-09-16T04:10:19.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.007+0000 7ff4ccf9f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:10:19.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.007+0000 7ff4ccf9f640 1 -- 172.21.15.42:0/613166006 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff4a8009f80 msgr2=0x7ff4c81cffd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:19.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.007+0000 7ff4ccf9f640 1 --2- 172.21.15.42:0/613166006 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff4a8009f80 0x7ff4c81cffd0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7ff4b80046b0 tx=0x7ff4b8017780 comp rx=0 tx=0).stop 2024-09-16T04:10:19.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.007+0000 7ff4ccf9f640 1 -- 172.21.15.42:0/613166006 shutdown_connections 2024-09-16T04:10:19.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.007+0000 7ff4ccf9f640 1 --2- 172.21.15.42:0/613166006 >> v2:172.21.15.154:6812/254845342 conn(0x7ff4a8039230 0x7ff4a803b6f0 unknown :-1 s=CLOSED pgs=3928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:10:19.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.007+0000 7ff4ccf9f640 1 --2- 172.21.15.42:0/613166006 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff4a800ab10 0x7ff4c81d0510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:10:19.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.007+0000 7ff4ccf9f640 1 --2- 172.21.15.42:0/613166006 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff4a8009f80 0x7ff4c81cffd0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:10:19.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.007+0000 7ff4ccf9f640 1 --2- 172.21.15.42:0/613166006 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff4a8005c60 0x7ff4c81cfa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:10:19.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.007+0000 7ff4ccf9f640 1 -- 172.21.15.42:0/613166006 >> 172.21.15.42:0/613166006 conn(0x7ff4c80a9fb0 msgr2=0x7ff4c8131980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:10:19.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.007+0000 7ff4ccf9f640 1 -- 172.21.15.42:0/613166006 shutdown_connections 2024-09-16T04:10:19.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.007+0000 7ff4ccf9f640 1 -- 172.21.15.42:0/613166006 wait complete. 2024-09-16T04:10:19.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.007+0000 7ff4ccf9f640 1 librados: shutdown 2024-09-16T04:10:19.029 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T04:10:19.030 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-16T04:10:19.030 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T04:10:19.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.147+0000 7ff1dc5fa640 10 monclient: get_monmap_and_config 2024-09-16T04:10:19.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.147+0000 7ff1dc5fa640 10 monclient: build_initial_monmap 2024-09-16T04:10:19.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.147+0000 7ff1dc5fa640 10 monclient: monmap: 2024-09-16T04:10:19.150 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:10:19.150 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:10:19.150 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:10:19.148333+0000 2024-09-16T04:10:19.150 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:10:19.148333+0000 2024-09-16T04:10:19.150 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:10:19.150 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:10:19.150 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:10:19.150 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:10:19.150 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:10:19.151 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:10:19.151 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:10:19.151 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:10:19.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.147+0000 7ff1dc5fa640 1 Processor -- start 2024-09-16T04:10:19.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.147+0000 7ff1dc5fa640 1 -- start start 2024-09-16T04:10:19.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.147+0000 7ff1dc5fa640 10 monclient: init 2024-09-16T04:10:19.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.147+0000 7ff1dc5fa640 10 monclient: _reopen_session rank -1 2024-09-16T04:10:19.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.147+0000 7ff1dc5fa640 10 monclient: _add_conns ranks=[0,4,2,3,1] 2024-09-16T04:10:19.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.147+0000 7ff1dc5fa640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff1d40a4720 addr v1:172.21.15.42:6789/0 2024-09-16T04:10:19.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.147+0000 7ff1dc5fa640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7ff1d4137bf0 0x7ff1d413c720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:10:19.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.147+0000 7ff1dc5fa640 10 monclient(hunting): picked mon.noname-c con 0x7ff1d4137bf0 addr v2:172.21.15.154:3300/0 2024-09-16T04:10:19.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.147+0000 7ff1dc5fa640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff1d4138000 0x7ff1d413ec70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:10:19.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.147+0000 7ff1dc5fa640 10 monclient(hunting): picked mon.noname-b con 0x7ff1d4138000 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:10:19.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.147+0000 7ff1dc5fa640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff1d40a1470 con 0x7ff1d40a4720 2024-09-16T04:10:19.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.147+0000 7ff1dc5fa640 10 monclient(hunting): start opening mon connection 2024-09-16T04:10:19.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.147+0000 7ff1dc5fa640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7ff1d406e8f0 con 0x7ff1d4137bf0 2024-09-16T04:10:19.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.147+0000 7ff1dc5fa640 10 monclient(hunting): start opening mon connection 2024-09-16T04:10:19.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.147+0000 7ff1dc5fa640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7ff1d40a5f60 con 0x7ff1d4138000 2024-09-16T04:10:19.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.147+0000 7ff1dc5fa640 10 monclient(hunting): _renew_subs 2024-09-16T04:10:19.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.147+0000 7ff1dc5fa640 10 monclient(hunting): authenticate will time out at 2666.117843s 2024-09-16T04:10:19.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.147+0000 7ff1da396640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7ff1d40a4720 0x7ff1d41374e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:55716/0 (socket says 172.21.15.42:55716) 2024-09-16T04:10:19.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.147+0000 7ff1da396640 1 -- 172.21.15.42:0/691416413 learned_addr learned my addr 172.21.15.42:0/691416413 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T04:10:19.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.147+0000 7ff1dab97640 1 --2- 172.21.15.42:0/691416413 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff1d4138000 0x7ff1d413ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:10:19.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.147+0000 7ff1dab97640 10 monclient(hunting): get_auth_request con 0x7ff1d4138000 auth_method 0 2024-09-16T04:10:19.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.147+0000 7ff1dab97640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:10:19.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.147+0000 7ff1dab97640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:19.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.147+0000 7ff1dab97640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:19.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.147+0000 7ff1dab97640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:10:19.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.147+0000 7ff1dab97640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:10:19.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.147+0000 7ff1dab97640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:10:19.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.147+0000 7ff1d9394640 1 -- 172.21.15.42:0/691416413 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3962510738 0 0) 0x7ff1d40a1470 con 0x7ff1d40a4720 2024-09-16T04:10:19.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.147+0000 7ff1d9394640 10 monclient(hunting): _negotiate 2024-09-16T04:10:19.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.147+0000 7ff1d9394640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:19.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.147+0000 7ff1d9394640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:19.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.147+0000 7ff1d9394640 10 monclient(hunting): my global_id is 1034277 2024-09-16T04:10:19.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.147+0000 7ff1d9394640 1 -- 172.21.15.42:0/691416413 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff1bc003530 con 0x7ff1d40a4720 2024-09-16T04:10:19.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.147+0000 7ff1dab97640 10 monclient(hunting): handle_auth_done global_id 1014410 payload 274 2024-09-16T04:10:19.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.147+0000 7ff1dab97640 1 -- 172.21.15.42:0/691416413 >> v2:172.21.15.154:3300/0 conn(0x7ff1d4137bf0 msgr2=0x7ff1d413c720 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-16T04:10:19.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.147+0000 7ff1dab97640 1 --2- 172.21.15.42:0/691416413 >> v2:172.21.15.154:3300/0 conn(0x7ff1d4137bf0 0x7ff1d413c720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:10:19.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.147+0000 7ff1dab97640 1 -- 172.21.15.42:0/691416413 >> v1:172.21.15.42:6789/0 conn(0x7ff1d40a4720 legacy=0x7ff1d41374e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:19.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.147+0000 7ff1dab97640 10 monclient: _finish_hunting 0 2024-09-16T04:10:19.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.147+0000 7ff1dab97640 1 monclient: found mon.noname-b 2024-09-16T04:10:19.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.147+0000 7ff1dab97640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:10:19.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.147+0000 7ff1dab97640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T04:10:19.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.147+0000 7ff1dab97640 1 -- 172.21.15.42:0/691416413 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1d40a1750 con 0x7ff1d4138000 2024-09-16T04:10:19.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.151+0000 7ff1dab97640 1 --2- 172.21.15.42:0/691416413 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff1d4138000 0x7ff1d413ec70 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7ff1c40097c0 tx=0x7ff1c401c380 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c8a249160a52c0a6 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:10:19.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.151+0000 7ff1d9394640 1 -- 172.21.15.42:0/691416413 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1c40b0070 con 0x7ff1d4138000 2024-09-16T04:10:19.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.151+0000 7ff1d9394640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:19.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.151+0000 7ff1d9394640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:10:19.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.151+0000 7ff1d9394640 10 monclient: dump: 2024-09-16T04:10:19.163 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:10:19.163 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:10:19.163 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:19.163 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:19.163 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:19.163 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:10:19.163 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:10:19.164 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:10:19.164 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:10:19.164 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:10:19.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.151+0000 7ff1d9394640 10 monclient: _finish_auth 0 2024-09-16T04:10:19.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.151+0000 7ff1d9394640 10 monclient: _check_auth_tickets 2024-09-16T04:10:19.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.151+0000 7ff1d9394640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:19.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.151+0000 7ff1d9394640 1 -- 172.21.15.42:0/691416413 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff1c40ab600 con 0x7ff1d4138000 2024-09-16T04:10:19.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.151+0000 7ff1d9394640 10 monclient: handle_config config(0 keys) 2024-09-16T04:10:19.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.151+0000 7ff1d9394640 1 -- 172.21.15.42:0/691416413 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1c40ab9d0 con 0x7ff1d4138000 2024-09-16T04:10:19.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.151+0000 7ff1dc5fa640 5 monclient: authenticate success, global_id 1014410 2024-09-16T04:10:19.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.151+0000 7ff1dc5fa640 10 monclient: get_monmap_and_config success 2024-09-16T04:10:19.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.151+0000 7ff1dc5fa640 4 set_mon_vals no callback set 2024-09-16T04:10:19.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.151+0000 7ff1d9394640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:19.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.151+0000 7ff1d9394640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:10:19.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.151+0000 7ff1d9394640 10 monclient: dump: 2024-09-16T04:10:19.165 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:10:19.165 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:10:19.165 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:19.165 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:19.165 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:19.166 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:10:19.166 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:10:19.166 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:10:19.167 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:10:19.167 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:10:19.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.151+0000 7ff1dc5fa640 10 monclient: shutdown 2024-09-16T04:10:19.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.151+0000 7ff1dc5fa640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:10:19.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.151+0000 7ff1dc5fa640 1 -- 172.21.15.42:0/691416413 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff1d4138000 msgr2=0x7ff1d413ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:19.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.151+0000 7ff1dc5fa640 1 --2- 172.21.15.42:0/691416413 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff1d4138000 0x7ff1d413ec70 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7ff1c40097c0 tx=0x7ff1c401c380 comp rx=0 tx=0).stop 2024-09-16T04:10:19.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.151+0000 7ff1dc5fa640 1 -- 172.21.15.42:0/691416413 shutdown_connections 2024-09-16T04:10:19.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.151+0000 7ff1dc5fa640 1 --2- 172.21.15.42:0/691416413 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff1d4138000 0x7ff1d413ec70 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:10:19.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.151+0000 7ff1dc5fa640 1 --2- 172.21.15.42:0/691416413 >> v2:172.21.15.154:3300/0 conn(0x7ff1d4137bf0 0x7ff1d413c720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:10:19.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.151+0000 7ff1dc5fa640 1 -- 172.21.15.42:0/691416413 >> 172.21.15.42:0/691416413 conn(0x7ff1d40a6820 msgr2=0x7ff1d40a6c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:10:19.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.151+0000 7ff1dc5fa640 1 -- 172.21.15.42:0/691416413 shutdown_connections 2024-09-16T04:10:19.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.151+0000 7ff1dc5fa640 1 -- 172.21.15.42:0/691416413 wait complete. 2024-09-16T04:10:19.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.151+0000 7ff1dc5fa640 10 monclient: build_initial_monmap 2024-09-16T04:10:19.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.151+0000 7ff1dc5fa640 10 monclient: monmap: 2024-09-16T04:10:19.168 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:10:19.168 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:10:19.168 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:10:19.168 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:10:19.168 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:10:19.168 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:10:19.168 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:10:19.169 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:10:19.169 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:10:19.169 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:10:19.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.151+0000 7ff1dc5fa640 1 librados: starting msgr at 2024-09-16T04:10:19.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.151+0000 7ff1dc5fa640 1 librados: starting objecter 2024-09-16T04:10:19.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.151+0000 7ff1dc5fa640 1 Processor -- start 2024-09-16T04:10:19.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.151+0000 7ff1dc5fa640 1 -- start start 2024-09-16T04:10:19.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.151+0000 7ff1dc5fa640 1 librados: setting wanted keys 2024-09-16T04:10:19.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.151+0000 7ff1dc5fa640 1 librados: calling monclient init 2024-09-16T04:10:19.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.151+0000 7ff1dc5fa640 10 monclient: init 2024-09-16T04:10:19.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.155+0000 7ff1dc5fa640 10 monclient: _reopen_session rank -1 2024-09-16T04:10:19.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.155+0000 7ff1dc5fa640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:10:19.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.155+0000 7ff1dc5fa640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff1d40a4720 0x7ff1d4136870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:10:19.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.155+0000 7ff1dc5fa640 10 monclient(hunting): picked mon.noname-a con 0x7ff1d40a4720 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:10:19.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.155+0000 7ff1dc5fa640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff1d4137bf0 0x7ff1d41cf6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:10:19.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.155+0000 7ff1dc5fa640 10 monclient(hunting): picked mon.noname-b con 0x7ff1d4137bf0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:10:19.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.155+0000 7ff1dc5fa640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff1d4138000 0x7ff1d413b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:10:19.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.155+0000 7ff1dc5fa640 10 monclient(hunting): picked mon.noname-c con 0x7ff1d4138000 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:10:19.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.155+0000 7ff1dc5fa640 10 monclient(hunting): start opening mon connection 2024-09-16T04:10:19.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.155+0000 7ff1dc5fa640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7ff1d4141d60 con 0x7ff1d40a4720 2024-09-16T04:10:19.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.155+0000 7ff1dc5fa640 10 monclient(hunting): start opening mon connection 2024-09-16T04:10:19.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.155+0000 7ff1dc5fa640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff1d4141be0 con 0x7ff1d4137bf0 2024-09-16T04:10:19.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.155+0000 7ff1dc5fa640 10 monclient(hunting): start opening mon connection 2024-09-16T04:10:19.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.155+0000 7ff1dc5fa640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7ff1d4141ee0 con 0x7ff1d4138000 2024-09-16T04:10:19.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.155+0000 7ff1dc5fa640 10 monclient(hunting): _renew_subs 2024-09-16T04:10:19.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.155+0000 7ff1dc5fa640 10 monclient(hunting): authenticate will time out at 2666.124257s 2024-09-16T04:10:19.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.155+0000 7ff1da396640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff1d40a4720 0x7ff1d4136870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:10:19.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.155+0000 7ff1da396640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff1d40a4720 0x7ff1d4136870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:48686/0 (socket says 172.21.15.42:48686) 2024-09-16T04:10:19.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.155+0000 7ff1da396640 1 -- 172.21.15.42:0/2634021647 learned_addr learned my addr 172.21.15.42:0/2634021647 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:10:19.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.155+0000 7ff1da396640 10 monclient(hunting): get_auth_request con 0x7ff1d40a4720 auth_method 0 2024-09-16T04:10:19.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.155+0000 7ff1da396640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:10:19.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.155+0000 7ff1da396640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:19.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.155+0000 7ff1da396640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:19.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.155+0000 7ff1dab97640 1 --2- 172.21.15.42:0/2634021647 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff1d4138000 0x7ff1d413b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:10:19.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.155+0000 7ff1dab97640 10 monclient(hunting): get_auth_request con 0x7ff1d4138000 auth_method 0 2024-09-16T04:10:19.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.155+0000 7ff1dab97640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:10:19.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.155+0000 7ff1dab97640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:19.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.155+0000 7ff1dab97640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:19.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.155+0000 7ff1d9b95640 1 -- 172.21.15.42:0/2634021647 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff1d4137bf0 msgr2=0x7ff1d41cf6b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T04:10:19.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.155+0000 7ff1da396640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:10:19.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.155+0000 7ff1da396640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:10:19.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.155+0000 7ff1d9b95640 1 --2- 172.21.15.42:0/2634021647 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff1d4137bf0 0x7ff1d41cf6b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:10:19.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.155+0000 7ff1da396640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:10:19.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.155+0000 7ff1dab97640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:10:19.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.155+0000 7ff1dab97640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:10:19.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.155+0000 7ff1dab97640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:10:19.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.155+0000 7ff1da396640 10 monclient(hunting): handle_auth_done global_id 1034280 payload 824 2024-09-16T04:10:19.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.155+0000 7ff1da396640 1 -- 172.21.15.42:0/2634021647 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff1d4138000 msgr2=0x7ff1d413b2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:19.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.155+0000 7ff1da396640 1 --2- 172.21.15.42:0/2634021647 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff1d4138000 0x7ff1d413b2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:10:19.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.155+0000 7ff1da396640 1 -- 172.21.15.42:0/2634021647 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff1d4137bf0 msgr2=0x7ff1d41cf6b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:10:19.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.155+0000 7ff1da396640 1 --2- 172.21.15.42:0/2634021647 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff1d4137bf0 0x7ff1d41cf6b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:10:19.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.155+0000 7ff1da396640 10 monclient: _finish_hunting 0 2024-09-16T04:10:19.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.155+0000 7ff1da396640 1 monclient: found mon.noname-a 2024-09-16T04:10:19.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.155+0000 7ff1da396640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:10:19.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.155+0000 7ff1da396640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:10:19.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.155+0000 7ff1da396640 1 -- 172.21.15.42:0/2634021647 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1d41205c0 con 0x7ff1d40a4720 2024-09-16T04:10:19.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.155+0000 7ff1da396640 1 --2- 172.21.15.42:0/2634021647 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff1d40a4720 0x7ff1d4136870 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7ff1cc00caa0 tx=0x7ff1cc00cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:10:19.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.155+0000 7ff1d2ffd640 10 client.?.objecter ms_handle_connect 0x7ff1d40a4720 2024-09-16T04:10:19.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.155+0000 7ff1d2ffd640 10 client.?.objecter resend_mon_ops 2024-09-16T04:10:19.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.155+0000 7ff1d2ffd640 1 -- 172.21.15.42:0/2634021647 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1cc017070 con 0x7ff1d40a4720 2024-09-16T04:10:19.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.155+0000 7ff1d2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:19.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.155+0000 7ff1d2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:10:19.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.155+0000 7ff1d2ffd640 10 monclient: dump: 2024-09-16T04:10:19.174 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:10:19.174 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:10:19.174 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:19.174 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:19.174 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:19.174 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:10:19.174 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:10:19.174 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:10:19.174 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:10:19.174 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:10:19.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.155+0000 7ff1d2ffd640 10 monclient: _finish_auth 0 2024-09-16T04:10:19.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.155+0000 7ff1d2ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:10:19.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.155+0000 7ff1d2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:19.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.155+0000 7ff1dc5fa640 5 monclient: authenticate success, global_id 1034280 2024-09-16T04:10:19.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.155+0000 7ff1d2ffd640 1 -- 172.21.15.42:0/2634021647 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff1cc00aba0 con 0x7ff1d40a4720 2024-09-16T04:10:19.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.155+0000 7ff1dc5fa640 10 monclient: _renew_subs 2024-09-16T04:10:19.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.155+0000 7ff1dc5fa640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:10:19.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.155+0000 7ff1dc5fa640 1 -- 172.21.15.42:0/2634021647 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1d4120b80 con 0x7ff1d40a4720 2024-09-16T04:10:19.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.155+0000 7ff1d2ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:10:19.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.155+0000 7ff1d2ffd640 1 -- 172.21.15.42:0/2634021647 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1cc012400 con 0x7ff1d40a4720 2024-09-16T04:10:19.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.155+0000 7ff1d2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:19.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.155+0000 7ff1dc5fa640 10 client.1034280.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:10:19.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.155+0000 7ff1d2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:10:19.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.155+0000 7ff1d2ffd640 10 monclient: dump: 2024-09-16T04:10:19.175 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:10:19.175 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:10:19.175 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:19.175 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:19.176 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:19.176 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:10:19.176 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:10:19.176 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:10:19.176 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:10:19.176 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:10:19.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.155+0000 7ff1d8b93640 4 set_mon_vals no callback set 2024-09-16T04:10:19.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.155+0000 7ff1dc5fa640 10 monclient: _renew_subs 2024-09-16T04:10:19.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.155+0000 7ff1dc5fa640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:10:19.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.155+0000 7ff1dc5fa640 1 -- 172.21.15.42:0/2634021647 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1d413cf50 con 0x7ff1d40a4720 2024-09-16T04:10:19.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.155+0000 7ff1dc5fa640 1 librados: init done 2024-09-16T04:10:19.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.155+0000 7ff1dc5fa640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:19.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.155+0000 7ff1dc5fa640 1 --2- 172.21.15.42:0/2634021647 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff1d413d630 0x7ff1d413b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:10:19.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.155+0000 7ff1dc5fa640 10 monclient(hunting): start opening mon connection 2024-09-16T04:10:19.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.155+0000 7ff1dc5fa640 1 -- 172.21.15.42:0/2634021647 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7ff1d41209f0 con 0x7ff1d413d630 2024-09-16T04:10:19.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.155+0000 7ff1d9b95640 1 --2- 172.21.15.42:0/2634021647 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff1d413d630 0x7ff1d413b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:10:19.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.155+0000 7ff1d9b95640 10 monclient: get_auth_request con 0x7ff1d413d630 auth_method 0 2024-09-16T04:10:19.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.155+0000 7ff1d9b95640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:10:19.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.155+0000 7ff1d9b95640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:19.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.155+0000 7ff1d9b95640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:19.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.155+0000 7ff1d2ffd640 1 -- 172.21.15.42:0/2634021647 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7ff1cc01a070 con 0x7ff1d40a4720 2024-09-16T04:10:19.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.155+0000 7ff1d9b95640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:10:19.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.155+0000 7ff1d9b95640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:10:19.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.155+0000 7ff1d9b95640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:10:19.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.155+0000 7ff1d9b95640 10 monclient(hunting): handle_auth_done global_id 1034283 payload 824 2024-09-16T04:10:19.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.155+0000 7ff1d2ffd640 1 --2- 172.21.15.42:0/2634021647 >> v2:172.21.15.154:6812/254845342 conn(0x7ff1c00377c0 0x7ff1c0039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:10:19.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.155+0000 7ff1d9b95640 1 -- 172.21.15.42:0/2634021647 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff1d41374e0 con 0x7ff1d413d630 2024-09-16T04:10:19.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.155+0000 7ff1d2ffd640 1 -- 172.21.15.42:0/2634021647 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(706..706 src has 451..706) ==== 5040+0+0 (secure 0 0 0) 0x7ff1cc048960 con 0x7ff1d40a4720 2024-09-16T04:10:19.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.155+0000 7ff1d2ffd640 10 client.1034280.objecter ms_dispatch 0x7ff1d4000ba0 osd_map(706..706 src has 451..706) 2024-09-16T04:10:19.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.155+0000 7ff1d2ffd640 3 client.1034280.objecter handle_osd_map got epochs [706,706] > 0 2024-09-16T04:10:19.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.155+0000 7ff1d2ffd640 3 client.1034280.objecter handle_osd_map decoding full epoch 706 2024-09-16T04:10:19.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.155+0000 7ff1d2ffd640 20 client.1034280.objecter dump_active .. 0 homeless 2024-09-16T04:10:19.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.155+0000 7ff1d9b95640 1 --2- 172.21.15.42:0/2634021647 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff1d413d630 0x7ff1d413b7f0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7ff1d413ca30 tx=0x7ff1c800b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:10:19.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.155+0000 7ff1d2ffd640 10 client.1034280.objecter ms_handle_connect 0x7ff1d413d630 2024-09-16T04:10:19.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.155+0000 7ff1d2ffd640 10 client.1034280.objecter resend_mon_ops 2024-09-16T04:10:19.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.155+0000 7ff1d2ffd640 1 -- 172.21.15.42:0/2634021647 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1c8016070 con 0x7ff1d413d630 2024-09-16T04:10:19.180 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.155+0000 7ff1d2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:19.180 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.155+0000 7ff1d2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:10:19.180 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.155+0000 7ff1d2ffd640 10 monclient: dump: 2024-09-16T04:10:19.180 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:10:19.180 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:10:19.180 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:19.180 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:19.180 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:19.180 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:10:19.180 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:10:19.180 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:10:19.180 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:10:19.180 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:10:19.181 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.155+0000 7ff1dab97640 1 --2- 172.21.15.42:0/2634021647 >> v2:172.21.15.154:6812/254845342 conn(0x7ff1c00377c0 0x7ff1c0039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:10:19.181 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.155+0000 7ff1dab97640 10 monclient: get_auth_request con 0x7ff1c00377c0 auth_method 0 2024-09-16T04:10:19.181 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.155+0000 7ff1d2ffd640 1 -- 172.21.15.42:0/2634021647 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7ff1c8003d10 con 0x7ff1d413d630 2024-09-16T04:10:19.181 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.155+0000 7ff1d2ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:19.181 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.155+0000 7ff1d2ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:10:19.181 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.155+0000 7ff1d2ffd640 1 -- 172.21.15.42:0/2634021647 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff1d413d630 msgr2=0x7ff1d413b7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:19.181 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.155+0000 7ff1d2ffd640 1 --2- 172.21.15.42:0/2634021647 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff1d413d630 0x7ff1d413b7f0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7ff1d413ca30 tx=0x7ff1c800b8e0 comp rx=0 tx=0).stop 2024-09-16T04:10:19.181 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.155+0000 7ff1d2ffd640 1 -- 172.21.15.42:0/2634021647 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff1d413d630 msgr2=0x7ff1d413b7f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:10:19.181 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.155+0000 7ff1d2ffd640 1 --2- 172.21.15.42:0/2634021647 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff1d413d630 0x7ff1d413b7f0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:10:19.181 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.159+0000 7ff1dab97640 1 --2- 172.21.15.42:0/2634021647 >> v2:172.21.15.154:6812/254845342 conn(0x7ff1c00377c0 0x7ff1c0039c80 secure :-1 s=READY pgs=3929 cs=0 l=1 rev1=1 crypto rx=0x7ff1c40ad000 tx=0x7ff1c4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:10:19.181 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.159+0000 7ff1d2ffd640 10 client.1034280.objecter ms_handle_connect 0x7ff1c00377c0 2024-09-16T04:10:19.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.163+0000 7ff1dc5fa640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:10:19.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.163+0000 7ff1dc5fa640 1 --2- 172.21.15.42:0/2634021647 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff1d41cfbf0 0x7ff1d413bd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:10:19.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.163+0000 7ff1dc5fa640 10 monclient(hunting): start opening mon connection 2024-09-16T04:10:19.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.163+0000 7ff1dc5fa640 1 -- 172.21.15.42:0/2634021647 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7ff1d41cf1e0 con 0x7ff1d41cfbf0 2024-09-16T04:10:19.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.163+0000 7ff1d9b95640 1 --2- 172.21.15.42:0/2634021647 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff1d41cfbf0 0x7ff1d413bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:10:19.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.163+0000 7ff1d9b95640 10 monclient: get_auth_request con 0x7ff1d41cfbf0 auth_method 0 2024-09-16T04:10:19.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.163+0000 7ff1d9b95640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:10:19.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.163+0000 7ff1d9b95640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:19.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.163+0000 7ff1d9b95640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:19.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.163+0000 7ff1d9b95640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:10:19.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.163+0000 7ff1d9b95640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:10:19.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.163+0000 7ff1d9b95640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:10:19.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.163+0000 7ff1d9b95640 10 monclient(hunting): handle_auth_done global_id 1034286 payload 824 2024-09-16T04:10:19.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.163+0000 7ff1d9b95640 1 -- 172.21.15.42:0/2634021647 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ff1d41da2d0 con 0x7ff1d41cfbf0 2024-09-16T04:10:19.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.163+0000 7ff1d2ffd640 10 client.1034280.objecter ms_handle_connect 0x7ff1d41cfbf0 2024-09-16T04:10:19.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.163+0000 7ff1d9b95640 1 --2- 172.21.15.42:0/2634021647 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff1d41cfbf0 0x7ff1d413bd30 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7ff1d406b300 tx=0x7ff1c8005960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:10:19.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.163+0000 7ff1d2ffd640 10 client.1034280.objecter resend_mon_ops 2024-09-16T04:10:19.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.163+0000 7ff1d2ffd640 1 -- 172.21.15.42:0/2634021647 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1c8016040 con 0x7ff1d41cfbf0 2024-09-16T04:10:19.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.163+0000 7ff1d2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:19.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.163+0000 7ff1d2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:10:19.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.163+0000 7ff1d2ffd640 10 monclient: dump: 2024-09-16T04:10:19.185 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:10:19.185 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:10:19.185 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:19.185 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:19.185 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:19.185 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:10:19.185 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:10:19.185 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:10:19.185 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:10:19.185 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:10:19.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.163+0000 7ff1d2ffd640 1 -- 172.21.15.42:0/2634021647 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4870 (secure 0 0 0) 0x7ff1c8011400 con 0x7ff1d41cfbf0 2024-09-16T04:10:19.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.163+0000 7ff1d2ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:10:19.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.163+0000 7ff1d2ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:10:19.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.163+0000 7ff1d2ffd640 1 -- 172.21.15.42:0/2634021647 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff1d41cfbf0 msgr2=0x7ff1d413bd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:19.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.163+0000 7ff1d2ffd640 1 --2- 172.21.15.42:0/2634021647 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff1d41cfbf0 0x7ff1d413bd30 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7ff1d406b300 tx=0x7ff1c8005960 comp rx=0 tx=0).stop 2024-09-16T04:10:19.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.163+0000 7ff1d2ffd640 1 -- 172.21.15.42:0/2634021647 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff1d41cfbf0 msgr2=0x7ff1d413bd30 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:10:19.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.163+0000 7ff1d2ffd640 1 --2- 172.21.15.42:0/2634021647 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff1d41cfbf0 0x7ff1d413bd30 secure :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7ff1d406b300 tx=0x7ff1c8005960 comp rx=0 tx=0).stop 2024-09-16T04:10:19.186 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T04:10:19.186 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:10:19.186 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:10:19.187 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "leader", 2024-09-16T04:10:19.187 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 260, 2024-09-16T04:10:19.187 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 99099, 2024-09-16T04:10:19.187 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T04:10:19.187 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T04:10:19.187 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T04:10:19.187 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:10:19.187 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 1, 2024-09-16T04:10:19.187 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:10:19.187 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T04:10:19.187 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T04:10:19.187 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:10:19.187 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:10:19.187 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:10:19.188 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:10:19.188 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:10:19.188 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:10:19.188 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:10:19.188 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:10:19.188 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:10:19.188 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:10:19.188 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:10:19.188 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:10:19.188 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:10:19.188 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T04:10:19.188 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:10:19.188 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:10:19.189 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:10:19.189 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:10:19.189 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:10:19.189 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:10:19.189 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:10:19.189 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:10:19.189 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:10:19.189 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:10:19.189 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:10:19.189 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:10:19.189 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:10:19.189 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T04:10:19.190 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T04:10:19.190 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T04:10:19.190 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T04:10:19.190 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T04:10:19.190 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T04:10:19.190 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:10:19.190 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:10:19.190 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T04:10:19.190 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T04:10:19.190 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T04:10:19.190 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T04:10:19.190 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T04:10:19.190 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T04:10:19.190 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T04:10:19.191 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:10:19.191 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T04:10:19.191 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:10:19.191 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:10:19.191 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:10:19.191 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:10:19.191 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:10:19.191 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:10:19.191 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:10:19.191 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:10:19.191 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:10:19.191 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:10:19.191 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:10:19.191 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:10:19.192 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T04:10:19.192 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:10:19.192 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T04:10:19.192 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:10:19.192 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:10:19.192 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:10:19.192 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:10:19.192 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:10:19.192 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:10:19.193 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:10:19.193 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T04:10:19.193 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:10:19.193 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:10:19.193 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:10:19.193 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:10:19.193 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T04:10:19.193 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:10:19.193 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:10:19.193 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:10:19.194 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:10:19.194 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T04:10:19.194 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T04:10:19.194 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:10:19.194 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:10:19.194 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:10:19.194 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:10:19.194 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:10:19.194 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:10:19.194 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:10:19.194 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:10:19.195 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:10:19.195 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:10:19.195 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:10:19.195 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T04:10:19.195 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:10:19.195 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:10:19.195 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:10:19.195 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:10:19.195 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T04:10:19.195 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:10:19.195 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:10:19.195 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:10:19.195 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:10:19.195 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T04:10:19.196 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T04:10:19.196 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:10:19.196 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:10:19.196 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:10:19.196 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:10:19.196 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:10:19.196 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:10:19.196 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:10:19.196 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:10:19.196 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:10:19.196 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:10:19.196 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:10:19.196 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T04:10:19.197 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:10:19.197 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:10:19.197 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:10:19.197 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:10:19.197 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T04:10:19.197 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:10:19.197 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:10:19.197 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:10:19.197 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:10:19.197 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T04:10:19.197 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T04:10:19.197 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:10:19.197 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:10:19.197 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:10:19.198 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:10:19.198 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:10:19.198 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:10:19.198 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T04:10:19.198 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T04:10:19.198 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:10:19.198 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:10:19.198 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:10:19.198 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:10:19.198 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:10:19.198 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:10:19.198 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T04:10:19.198 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:10:19.198 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:10:19.198 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:10:19.199 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 2 2024-09-16T04:10:19.199 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:10:19.199 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:10:19.199 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T04:10:19.199 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:10:19.199 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:10:19.199 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:10:19.199 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 5 2024-09-16T04:10:19.199 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:10:19.199 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:10:19.199 INFO:teuthology.orchestra.run.smithi042.stdout: "mgr": [ 2024-09-16T04:10:19.199 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:10:19.199 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:10:19.199 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:10:19.200 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:10:19.200 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:10:19.200 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:10:19.200 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:10:19.200 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T04:10:19.200 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T04:10:19.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.163+0000 7ff1dc5fa640 10 librados: watch_flush enter 2024-09-16T04:10:19.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.163+0000 7ff1dc5fa640 10 librados: watch_flush exit 2024-09-16T04:10:19.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.163+0000 7ff1dc5fa640 20 client.1034280.objecter shutdown clearing up homeless session... 2024-09-16T04:10:19.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.163+0000 7ff1dc5fa640 10 client.1034280.objecter successfully canceled tick 2024-09-16T04:10:19.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.163+0000 7ff1dc5fa640 1 -- 172.21.15.42:0/2634021647 >> v2:172.21.15.154:6812/254845342 conn(0x7ff1c00377c0 msgr2=0x7ff1c0039c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:19.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.163+0000 7ff1dc5fa640 1 --2- 172.21.15.42:0/2634021647 >> v2:172.21.15.154:6812/254845342 conn(0x7ff1c00377c0 0x7ff1c0039c80 secure :-1 s=READY pgs=3929 cs=0 l=1 rev1=1 crypto rx=0x7ff1c40ad000 tx=0x7ff1c4008040 comp rx=0 tx=0).stop 2024-09-16T04:10:19.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.163+0000 7ff1dc5fa640 10 monclient: shutdown 2024-09-16T04:10:19.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.163+0000 7ff1dc5fa640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:10:19.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.163+0000 7ff1dc5fa640 1 -- 172.21.15.42:0/2634021647 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff1d40a4720 msgr2=0x7ff1d4136870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:19.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.163+0000 7ff1dc5fa640 1 --2- 172.21.15.42:0/2634021647 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff1d40a4720 0x7ff1d4136870 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7ff1cc00caa0 tx=0x7ff1cc00cf60 comp rx=0 tx=0).stop 2024-09-16T04:10:19.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.163+0000 7ff1dab97640 1 -- 172.21.15.42:0/2634021647 reap_dead start 2024-09-16T04:10:19.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.163+0000 7ff1dab97640 1 -- 172.21.15.42:0/2634021647 reap_dead start 2024-09-16T04:10:19.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.163+0000 7ff1dc5fa640 1 -- 172.21.15.42:0/2634021647 shutdown_connections 2024-09-16T04:10:19.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.163+0000 7ff1dc5fa640 1 -- 172.21.15.42:0/2634021647 >> 172.21.15.42:0/2634021647 conn(0x7ff1d40a6820 msgr2=0x7ff1d406d5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:10:19.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.163+0000 7ff1dc5fa640 1 -- 172.21.15.42:0/2634021647 shutdown_connections 2024-09-16T04:10:19.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.163+0000 7ff1dc5fa640 1 -- 172.21.15.42:0/2634021647 wait complete. 2024-09-16T04:10:19.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.163+0000 7ff1dc5fa640 1 librados: shutdown 2024-09-16T04:10:19.204 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T04:10:19.204 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-16T04:10:19.204 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T04:10:19.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.319+0000 7f5d65529640 10 monclient: get_monmap_and_config 2024-09-16T04:10:19.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.319+0000 7f5d65529640 10 monclient: build_initial_monmap 2024-09-16T04:10:19.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.319+0000 7f5d65529640 10 monclient: monmap: 2024-09-16T04:10:19.322 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:10:19.322 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:10:19.322 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:10:19.320731+0000 2024-09-16T04:10:19.322 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:10:19.320731+0000 2024-09-16T04:10:19.322 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:10:19.322 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:10:19.323 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:10:19.323 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:10:19.323 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:10:19.323 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:10:19.323 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:10:19.323 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:10:19.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.319+0000 7f5d65529640 1 Processor -- start 2024-09-16T04:10:19.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.319+0000 7f5d65529640 1 -- start start 2024-09-16T04:10:19.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.319+0000 7f5d65529640 10 monclient: init 2024-09-16T04:10:19.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.319+0000 7f5d65529640 10 monclient: _reopen_session rank -1 2024-09-16T04:10:19.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.319+0000 7f5d65529640 10 monclient: _add_conns ranks=[3,1,0,4,2] 2024-09-16T04:10:19.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.319+0000 7f5d65529640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f5d600a1330 0x7f5d600a1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:10:19.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.319+0000 7f5d65529640 10 monclient(hunting): picked mon.noname-a con 0x7f5d600a1330 addr v2:172.21.15.42:3300/0 2024-09-16T04:10:19.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.319+0000 7f5d65529640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5d6006dac0 addr v1:172.21.15.154:6789/0 2024-09-16T04:10:19.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.319+0000 7f5d65529640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5d600a1c50 addr v1:172.21.15.42:6789/0 2024-09-16T04:10:19.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.319+0000 7f5d65529640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5d60135f20 con 0x7f5d600a1c50 2024-09-16T04:10:19.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.319+0000 7f5d65529640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5d6009c090 con 0x7f5d6006dac0 2024-09-16T04:10:19.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.319+0000 7f5d65529640 10 monclient(hunting): start opening mon connection 2024-09-16T04:10:19.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.319+0000 7f5d65529640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f5d6009f610 con 0x7f5d600a1330 2024-09-16T04:10:19.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.319+0000 7f5d65529640 10 monclient(hunting): _renew_subs 2024-09-16T04:10:19.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.319+0000 7f5d65529640 10 monclient(hunting): authenticate will time out at 2666.290164s 2024-09-16T04:10:19.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.319+0000 7f5d5f7fe640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f5d600a1c50 0x7f5d60135810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:55724/0 (socket says 172.21.15.42:55724) 2024-09-16T04:10:19.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.319+0000 7f5d5f7fe640 1 -- 172.21.15.42:0/972312721 learned_addr learned my addr 172.21.15.42:0/972312721 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T04:10:19.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.319+0000 7f5d5effd640 1 --2- 172.21.15.42:0/972312721 >> v2:172.21.15.42:3300/0 conn(0x7f5d600a1330 0x7f5d600a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:10:19.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.319+0000 7f5d567fc640 1 -- 172.21.15.42:0/972312721 >> v1:172.21.15.154:6789/0 conn(0x7f5d6006dac0 legacy=0x7f5d60135100 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.154:6789/0 2024-09-16T04:10:19.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.319+0000 7f5d5effd640 10 monclient(hunting): get_auth_request con 0x7f5d600a1330 auth_method 0 2024-09-16T04:10:19.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.319+0000 7f5d5effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:10:19.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.319+0000 7f5d5effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:19.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.319+0000 7f5d5effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:19.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.319+0000 7f5d5effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:10:19.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.319+0000 7f5d5effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:10:19.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.319+0000 7f5d5effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:10:19.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.319+0000 7f5d5e7fc640 1 -- 172.21.15.42:0/972312721 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1358674886 0 0) 0x7f5d60135f20 con 0x7f5d600a1c50 2024-09-16T04:10:19.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.319+0000 7f5d5e7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:10:19.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.319+0000 7f5d5e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:19.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.319+0000 7f5d5e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:19.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.319+0000 7f5d5e7fc640 10 monclient(hunting): my global_id is 1034292 2024-09-16T04:10:19.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.319+0000 7f5d5e7fc640 1 -- 172.21.15.42:0/972312721 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5d40003530 con 0x7f5d600a1c50 2024-09-16T04:10:19.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.319+0000 7f5d5effd640 10 monclient(hunting): handle_auth_done global_id 1034289 payload 274 2024-09-16T04:10:19.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.319+0000 7f5d5effd640 1 -- 172.21.15.42:0/972312721 >> v1:172.21.15.154:6789/0 conn(0x7f5d6006dac0 legacy=0x7f5d60135100 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:10:19.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.319+0000 7f5d5effd640 1 -- 172.21.15.42:0/972312721 >> v1:172.21.15.42:6789/0 conn(0x7f5d600a1c50 legacy=0x7f5d60135810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:19.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.319+0000 7f5d5effd640 10 monclient: _finish_hunting 0 2024-09-16T04:10:19.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.319+0000 7f5d5effd640 1 monclient: found mon.noname-a 2024-09-16T04:10:19.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.319+0000 7f5d5effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:10:19.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.319+0000 7f5d5effd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:10:19.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.319+0000 7f5d5effd640 1 -- 172.21.15.42:0/972312721 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d6009f8f0 con 0x7f5d600a1330 2024-09-16T04:10:19.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.323+0000 7f5d5effd640 1 --2- 172.21.15.42:0/972312721 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5d600a1330 0x7f5d600a1710 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f5d500097c0 tx=0x7f5d5001c380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8127789a0821fa8a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:10:19.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.323+0000 7f5d5e7fc640 1 -- 172.21.15.42:0/972312721 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d500b1070 con 0x7f5d600a1330 2024-09-16T04:10:19.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.323+0000 7f5d5e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:19.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.323+0000 7f5d5e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:10:19.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.323+0000 7f5d5e7fc640 10 monclient: dump: 2024-09-16T04:10:19.338 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:10:19.338 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:10:19.338 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:19.338 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:19.338 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:19.338 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:10:19.338 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:10:19.339 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:10:19.339 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:10:19.339 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:10:19.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.323+0000 7f5d5e7fc640 10 monclient: _finish_auth 0 2024-09-16T04:10:19.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.323+0000 7f5d5e7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:10:19.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.323+0000 7f5d5e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:19.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.323+0000 7f5d65529640 5 monclient: authenticate success, global_id 1034289 2024-09-16T04:10:19.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.323+0000 7f5d65529640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:10:19.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.323+0000 7f5d5e7fc640 1 -- 172.21.15.42:0/972312721 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5d500ac600 con 0x7f5d600a1330 2024-09-16T04:10:19.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.323+0000 7f5d5e7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:10:19.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.323+0000 7f5d65529640 10 monclient: get_monmap_and_config success 2024-09-16T04:10:19.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.323+0000 7f5d65529640 4 set_mon_vals no callback set 2024-09-16T04:10:19.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.323+0000 7f5d5e7fc640 1 -- 172.21.15.42:0/972312721 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d500ac9b0 con 0x7f5d600a1330 2024-09-16T04:10:19.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.323+0000 7f5d5e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:19.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.323+0000 7f5d5e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:10:19.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.323+0000 7f5d5e7fc640 10 monclient: dump: 2024-09-16T04:10:19.340 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:10:19.340 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:10:19.340 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:19.340 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:19.340 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:19.340 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:10:19.340 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:10:19.340 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:10:19.341 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:10:19.341 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:10:19.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.323+0000 7f5d65529640 10 monclient: shutdown 2024-09-16T04:10:19.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.323+0000 7f5d65529640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:10:19.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.323+0000 7f5d65529640 1 -- 172.21.15.42:0/972312721 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5d600a1330 msgr2=0x7f5d600a1710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:19.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.323+0000 7f5d65529640 1 --2- 172.21.15.42:0/972312721 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5d600a1330 0x7f5d600a1710 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f5d500097c0 tx=0x7f5d5001c380 comp rx=0 tx=0).stop 2024-09-16T04:10:19.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.323+0000 7f5d65529640 1 -- 172.21.15.42:0/972312721 shutdown_connections 2024-09-16T04:10:19.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.323+0000 7f5d65529640 1 --2- 172.21.15.42:0/972312721 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5d600a1330 0x7f5d600a1710 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:10:19.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.323+0000 7f5d65529640 1 -- 172.21.15.42:0/972312721 >> 172.21.15.42:0/972312721 conn(0x7f5d600a2420 msgr2=0x7f5d600a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:10:19.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.323+0000 7f5d65529640 1 -- 172.21.15.42:0/972312721 shutdown_connections 2024-09-16T04:10:19.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.323+0000 7f5d65529640 1 -- 172.21.15.42:0/972312721 wait complete. 2024-09-16T04:10:19.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.323+0000 7f5d65529640 10 monclient: build_initial_monmap 2024-09-16T04:10:19.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.323+0000 7f5d65529640 10 monclient: monmap: 2024-09-16T04:10:19.342 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:10:19.342 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:10:19.342 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:10:19.342 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:10:19.342 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:10:19.342 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:10:19.342 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:10:19.343 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:10:19.343 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:10:19.343 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:10:19.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.323+0000 7f5d65529640 1 librados: starting msgr at 2024-09-16T04:10:19.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.323+0000 7f5d65529640 1 librados: starting objecter 2024-09-16T04:10:19.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.323+0000 7f5d65529640 1 Processor -- start 2024-09-16T04:10:19.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.323+0000 7f5d65529640 1 -- start start 2024-09-16T04:10:19.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.323+0000 7f5d65529640 1 librados: setting wanted keys 2024-09-16T04:10:19.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.323+0000 7f5d65529640 1 librados: calling monclient init 2024-09-16T04:10:19.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.323+0000 7f5d65529640 10 monclient: init 2024-09-16T04:10:19.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.323+0000 7f5d65529640 10 monclient: _reopen_session rank -1 2024-09-16T04:10:19.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.323+0000 7f5d65529640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:10:19.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.323+0000 7f5d65529640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5d600a1330 0x7f5d60135040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:10:19.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.323+0000 7f5d65529640 10 monclient(hunting): picked mon.noname-a con 0x7f5d600a1330 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:10:19.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.323+0000 7f5d65529640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5d600a1c50 0x7f5d6013cc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:10:19.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.323+0000 7f5d65529640 10 monclient(hunting): picked mon.noname-c con 0x7f5d600a1c50 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:10:19.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.323+0000 7f5d65529640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5d60135580 0x7f5d6013d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:10:19.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.323+0000 7f5d65529640 10 monclient(hunting): picked mon.noname-b con 0x7f5d60135580 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:10:19.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.323+0000 7f5d65529640 10 monclient(hunting): start opening mon connection 2024-09-16T04:10:19.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.323+0000 7f5d65529640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f5d6009eef0 con 0x7f5d600a1330 2024-09-16T04:10:19.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.323+0000 7f5d65529640 10 monclient(hunting): start opening mon connection 2024-09-16T04:10:19.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.323+0000 7f5d65529640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f5d6009ed70 con 0x7f5d60135580 2024-09-16T04:10:19.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.323+0000 7f5d65529640 10 monclient(hunting): start opening mon connection 2024-09-16T04:10:19.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.323+0000 7f5d65529640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f5d60140b90 con 0x7f5d600a1c50 2024-09-16T04:10:19.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.323+0000 7f5d65529640 10 monclient(hunting): _renew_subs 2024-09-16T04:10:19.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.323+0000 7f5d65529640 10 monclient(hunting): authenticate will time out at 2666.295789s 2024-09-16T04:10:19.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.323+0000 7f5d5effd640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5d600a1330 0x7f5d60135040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:10:19.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.323+0000 7f5d5f7fe640 1 -- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5d60135580 msgr2=0x7f5d6013d190 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T04:10:19.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.323+0000 7f5d5f7fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5d60135580 0x7f5d6013d190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:10:19.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.327+0000 7f5d5effd640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5d600a1330 0x7f5d60135040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:48724/0 (socket says 172.21.15.42:48724) 2024-09-16T04:10:19.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.327+0000 7f5d5effd640 1 -- 172.21.15.42:0/2774339535 learned_addr learned my addr 172.21.15.42:0/2774339535 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:10:19.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.327+0000 7f5d5effd640 10 monclient(hunting): get_auth_request con 0x7f5d600a1330 auth_method 0 2024-09-16T04:10:19.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.327+0000 7f5d5effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:10:19.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.327+0000 7f5d5effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:19.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.327+0000 7f5d5effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:19.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.327+0000 7f5d567fc640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5d600a1c50 0x7f5d6013cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:10:19.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.327+0000 7f5d567fc640 10 monclient(hunting): get_auth_request con 0x7f5d600a1c50 auth_method 0 2024-09-16T04:10:19.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.327+0000 7f5d567fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:10:19.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.327+0000 7f5d567fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:19.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.327+0000 7f5d567fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:19.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.327+0000 7f5d5effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:10:19.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.327+0000 7f5d5effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:10:19.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.327+0000 7f5d5effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:10:19.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.327+0000 7f5d567fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:10:19.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.327+0000 7f5d567fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:10:19.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.327+0000 7f5d567fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:10:19.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.327+0000 7f5d5effd640 10 monclient(hunting): handle_auth_done global_id 1034295 payload 824 2024-09-16T04:10:19.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.327+0000 7f5d5effd640 1 -- 172.21.15.42:0/2774339535 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5d600a1c50 msgr2=0x7f5d6013cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:19.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.327+0000 7f5d5effd640 1 --2- 172.21.15.42:0/2774339535 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5d600a1c50 0x7f5d6013cc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:10:19.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.327+0000 7f5d5effd640 1 -- 172.21.15.42:0/2774339535 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5d60135580 msgr2=0x7f5d6013d190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:10:19.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.327+0000 7f5d5effd640 1 --2- 172.21.15.42:0/2774339535 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5d60135580 0x7f5d6013d190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:10:19.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.327+0000 7f5d5effd640 10 monclient: _finish_hunting 0 2024-09-16T04:10:19.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.327+0000 7f5d5effd640 1 monclient: found mon.noname-a 2024-09-16T04:10:19.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.327+0000 7f5d5effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:10:19.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.327+0000 7f5d5effd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:10:19.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.327+0000 7f5d5effd640 1 -- 172.21.15.42:0/2774339535 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d601c6000 con 0x7f5d600a1330 2024-09-16T04:10:19.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.327+0000 7f5d5effd640 1 --2- 172.21.15.42:0/2774339535 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5d600a1330 0x7f5d60135040 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f5d50000c10 tx=0x7f5d5001abc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:10:19.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.327+0000 7f5d57fff640 10 client.?.objecter ms_handle_connect 0x7f5d600a1330 2024-09-16T04:10:19.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.327+0000 7f5d57fff640 10 client.?.objecter resend_mon_ops 2024-09-16T04:10:19.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.327+0000 7f5d57fff640 1 -- 172.21.15.42:0/2774339535 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d500b1050 con 0x7f5d600a1330 2024-09-16T04:10:19.347 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.327+0000 7f5d57fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:19.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.327+0000 7f5d57fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:10:19.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.327+0000 7f5d57fff640 10 monclient: dump: 2024-09-16T04:10:19.348 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:10:19.348 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:10:19.348 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:19.348 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:19.348 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:19.348 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:10:19.348 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:10:19.348 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:10:19.348 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:10:19.349 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:10:19.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.327+0000 7f5d57fff640 10 monclient: _finish_auth 0 2024-09-16T04:10:19.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.327+0000 7f5d57fff640 10 monclient: _check_auth_tickets 2024-09-16T04:10:19.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.327+0000 7f5d57fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:19.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.327+0000 7f5d57fff640 1 -- 172.21.15.42:0/2774339535 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5d500b2040 con 0x7f5d600a1330 2024-09-16T04:10:19.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.327+0000 7f5d57fff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:10:19.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.327+0000 7f5d57fff640 1 -- 172.21.15.42:0/2774339535 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d500ac9c0 con 0x7f5d600a1330 2024-09-16T04:10:19.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.327+0000 7f5d65529640 5 monclient: authenticate success, global_id 1034295 2024-09-16T04:10:19.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.327+0000 7f5d65529640 10 monclient: _renew_subs 2024-09-16T04:10:19.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.327+0000 7f5d65529640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:10:19.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.327+0000 7f5d65529640 1 -- 172.21.15.42:0/2774339535 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d60120700 con 0x7f5d600a1330 2024-09-16T04:10:19.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.327+0000 7f5d57fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:19.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.327+0000 7f5d5dffb640 4 set_mon_vals no callback set 2024-09-16T04:10:19.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.327+0000 7f5d57fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:10:19.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.327+0000 7f5d57fff640 10 monclient: dump: 2024-09-16T04:10:19.350 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:10:19.350 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:10:19.350 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:19.350 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:19.350 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:19.350 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:10:19.350 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:10:19.350 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:10:19.350 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:10:19.350 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:10:19.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.327+0000 7f5d65529640 10 client.1034295.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:10:19.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.327+0000 7f5d65529640 10 monclient: _renew_subs 2024-09-16T04:10:19.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.327+0000 7f5d65529640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:10:19.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.327+0000 7f5d65529640 1 -- 172.21.15.42:0/2774339535 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d60073a30 con 0x7f5d600a1330 2024-09-16T04:10:19.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.327+0000 7f5d65529640 1 librados: init done 2024-09-16T04:10:19.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.327+0000 7f5d65529640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:19.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.327+0000 7f5d65529640 1 --2- 172.21.15.42:0/2774339535 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5d6006da40 0x7f5d6002c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:10:19.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.327+0000 7f5d65529640 10 monclient(hunting): start opening mon connection 2024-09-16T04:10:19.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.327+0000 7f5d65529640 1 -- 172.21.15.42:0/2774339535 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f5d600711e0 con 0x7f5d6006da40 2024-09-16T04:10:19.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.327+0000 7f5d57fff640 1 -- 172.21.15.42:0/2774339535 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f5d500acb60 con 0x7f5d600a1330 2024-09-16T04:10:19.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.327+0000 7f5d567fc640 1 --2- 172.21.15.42:0/2774339535 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5d6006da40 0x7f5d6002c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:10:19.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.327+0000 7f5d567fc640 10 monclient: get_auth_request con 0x7f5d6006da40 auth_method 0 2024-09-16T04:10:19.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.327+0000 7f5d567fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:10:19.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.327+0000 7f5d567fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:19.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.327+0000 7f5d567fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:19.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.327+0000 7f5d567fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:10:19.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.327+0000 7f5d567fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:10:19.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.327+0000 7f5d567fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:10:19.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.327+0000 7f5d57fff640 1 --2- 172.21.15.42:0/2774339535 >> v2:172.21.15.154:6812/254845342 conn(0x7f5d440377f0 0x7f5d44039cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:10:19.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.327+0000 7f5d57fff640 1 -- 172.21.15.42:0/2774339535 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(706..706 src has 451..706) ==== 5040+0+0 (secure 0 0 0) 0x7f5d500e29d0 con 0x7f5d600a1330 2024-09-16T04:10:19.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.327+0000 7f5d57fff640 10 client.1034295.objecter ms_dispatch 0x7f5d60000ba0 osd_map(706..706 src has 451..706) 2024-09-16T04:10:19.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.327+0000 7f5d57fff640 3 client.1034295.objecter handle_osd_map got epochs [706,706] > 0 2024-09-16T04:10:19.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.327+0000 7f5d57fff640 3 client.1034295.objecter handle_osd_map decoding full epoch 706 2024-09-16T04:10:19.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.327+0000 7f5d57fff640 20 client.1034295.objecter dump_active .. 0 homeless 2024-09-16T04:10:19.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.327+0000 7f5d567fc640 10 monclient(hunting): handle_auth_done global_id 1014419 payload 824 2024-09-16T04:10:19.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.327+0000 7f5d567fc640 1 -- 172.21.15.42:0/2774339535 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5d600a1710 con 0x7f5d6006da40 2024-09-16T04:10:19.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.327+0000 7f5d57fff640 10 client.1034295.objecter ms_handle_connect 0x7f5d6006da40 2024-09-16T04:10:19.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.327+0000 7f5d57fff640 10 client.1034295.objecter resend_mon_ops 2024-09-16T04:10:19.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.327+0000 7f5d567fc640 1 --2- 172.21.15.42:0/2774339535 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5d6006da40 0x7f5d6002c650 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f5d60135f90 tx=0x7f5d48009a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:10:19.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.327+0000 7f5d5f7fe640 1 --2- 172.21.15.42:0/2774339535 >> v2:172.21.15.154:6812/254845342 conn(0x7f5d440377f0 0x7f5d44039cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:10:19.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.327+0000 7f5d5f7fe640 10 monclient: get_auth_request con 0x7f5d440377f0 auth_method 0 2024-09-16T04:10:19.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.327+0000 7f5d57fff640 1 -- 172.21.15.42:0/2774339535 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d48016070 con 0x7f5d6006da40 2024-09-16T04:10:19.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.327+0000 7f5d57fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:19.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.327+0000 7f5d57fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:10:19.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.327+0000 7f5d57fff640 10 monclient: dump: 2024-09-16T04:10:19.353 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:10:19.353 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:10:19.353 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:19.353 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:19.354 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:19.354 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:10:19.354 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:10:19.354 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:10:19.354 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:10:19.354 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:10:19.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.327+0000 7f5d57fff640 1 -- 172.21.15.42:0/2774339535 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f5d48012600 con 0x7f5d6006da40 2024-09-16T04:10:19.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.327+0000 7f5d57fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:19.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.327+0000 7f5d57fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:10:19.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.327+0000 7f5d57fff640 1 -- 172.21.15.42:0/2774339535 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5d6006da40 msgr2=0x7f5d6002c650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:19.355 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.327+0000 7f5d57fff640 1 --2- 172.21.15.42:0/2774339535 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5d6006da40 0x7f5d6002c650 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f5d60135f90 tx=0x7f5d48009a40 comp rx=0 tx=0).stop 2024-09-16T04:10:19.355 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.327+0000 7f5d57fff640 1 -- 172.21.15.42:0/2774339535 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5d6006da40 msgr2=0x7f5d6002c650 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:10:19.355 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.327+0000 7f5d57fff640 1 --2- 172.21.15.42:0/2774339535 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5d6006da40 0x7f5d6002c650 secure :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f5d60135f90 tx=0x7f5d48009a40 comp rx=0 tx=0).stop 2024-09-16T04:10:19.355 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.327+0000 7f5d57fff640 10 client.1034295.objecter ms_handle_connect 0x7f5d440377f0 2024-09-16T04:10:19.355 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.327+0000 7f5d5f7fe640 1 --2- 172.21.15.42:0/2774339535 >> v2:172.21.15.154:6812/254845342 conn(0x7f5d440377f0 0x7f5d44039cb0 secure :-1 s=READY pgs=3930 cs=0 l=1 rev1=1 crypto rx=0x7f5d58004ba0 tx=0x7f5d58001720 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:10:19.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.335+0000 7f5d65529640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:10:19.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.335+0000 7f5d65529640 1 --2- 172.21.15.42:0/2774339535 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5d6002cb90 0x7f5d6002cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:10:19.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.335+0000 7f5d65529640 10 monclient(hunting): start opening mon connection 2024-09-16T04:10:19.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.335+0000 7f5d65529640 1 -- 172.21.15.42:0/2774339535 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f5d60139a40 con 0x7f5d6002cb90 2024-09-16T04:10:19.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.335+0000 7f5d567fc640 1 --2- 172.21.15.42:0/2774339535 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5d6002cb90 0x7f5d6002cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:10:19.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.335+0000 7f5d567fc640 10 monclient: get_auth_request con 0x7f5d6002cb90 auth_method 0 2024-09-16T04:10:19.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.335+0000 7f5d567fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:10:19.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.335+0000 7f5d567fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:19.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.335+0000 7f5d567fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:19.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.335+0000 7f5d567fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:10:19.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.335+0000 7f5d567fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:10:19.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.335+0000 7f5d567fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:10:19.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.335+0000 7f5d567fc640 10 monclient(hunting): handle_auth_done global_id 1014422 payload 824 2024-09-16T04:10:19.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.335+0000 7f5d567fc640 1 -- 172.21.15.42:0/2774339535 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f5d6009d9a0 con 0x7f5d6002cb90 2024-09-16T04:10:19.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.335+0000 7f5d567fc640 1 --2- 172.21.15.42:0/2774339535 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5d6002cb90 0x7f5d6002cfa0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f5d48018000 tx=0x7f5d480059e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:10:19.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.335+0000 7f5d57fff640 10 client.1034295.objecter ms_handle_connect 0x7f5d6002cb90 2024-09-16T04:10:19.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.335+0000 7f5d57fff640 10 client.1034295.objecter resend_mon_ops 2024-09-16T04:10:19.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.335+0000 7f5d57fff640 1 -- 172.21.15.42:0/2774339535 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d48016040 con 0x7f5d6002cb90 2024-09-16T04:10:19.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.335+0000 7f5d57fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:19.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.335+0000 7f5d57fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:10:19.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.335+0000 7f5d57fff640 10 monclient: dump: 2024-09-16T04:10:19.358 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:10:19.358 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:10:19.358 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:19.358 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:19.358 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:19.358 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:10:19.358 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:10:19.358 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:10:19.358 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:10:19.359 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:10:19.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.335+0000 7f5d57fff640 1 -- 172.21.15.42:0/2774339535 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4702 (secure 0 0 0) 0x7f5d4801b450 con 0x7f5d6002cb90 2024-09-16T04:10:19.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.335+0000 7f5d57fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:10:19.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.335+0000 7f5d57fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:10:19.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.335+0000 7f5d57fff640 1 -- 172.21.15.42:0/2774339535 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5d6002cb90 msgr2=0x7f5d6002cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:19.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.335+0000 7f5d57fff640 1 --2- 172.21.15.42:0/2774339535 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5d6002cb90 0x7f5d6002cfa0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f5d48018000 tx=0x7f5d480059e0 comp rx=0 tx=0).stop 2024-09-16T04:10:19.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.335+0000 7f5d57fff640 1 -- 172.21.15.42:0/2774339535 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5d6002cb90 msgr2=0x7f5d6002cfa0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:10:19.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.335+0000 7f5d57fff640 1 --2- 172.21.15.42:0/2774339535 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5d6002cb90 0x7f5d6002cfa0 secure :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f5d48018000 tx=0x7f5d480059e0 comp rx=0 tx=0).stop 2024-09-16T04:10:19.359 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T04:10:19.359 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:10:19.360 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:10:19.360 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T04:10:19.360 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 260, 2024-09-16T04:10:19.360 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 214538, 2024-09-16T04:10:19.360 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T04:10:19.360 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T04:10:19.360 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T04:10:19.360 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:10:19.360 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 2, 2024-09-16T04:10:19.360 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:10:19.360 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T04:10:19.360 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T04:10:19.360 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:10:19.361 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:10:19.361 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:10:19.361 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:10:19.361 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:10:19.361 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:10:19.361 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:10:19.361 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:10:19.361 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:10:19.361 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:10:19.361 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:10:19.361 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:10:19.361 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:10:19.361 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T04:10:19.361 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:10:19.362 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:10:19.362 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:10:19.362 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:10:19.362 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:10:19.362 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:10:19.362 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:10:19.362 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:10:19.363 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:10:19.363 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:10:19.363 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:10:19.363 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:10:19.363 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:10:19.363 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T04:10:19.363 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T04:10:19.363 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T04:10:19.363 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T04:10:19.363 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T04:10:19.363 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T04:10:19.363 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:10:19.363 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:10:19.363 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T04:10:19.364 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T04:10:19.364 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T04:10:19.364 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T04:10:19.364 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T04:10:19.364 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T04:10:19.364 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T04:10:19.364 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:10:19.364 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T04:10:19.364 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:10:19.364 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:10:19.365 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:10:19.365 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:10:19.365 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:10:19.365 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:10:19.365 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:10:19.365 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:10:19.365 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:10:19.365 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:10:19.365 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:10:19.365 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:10:19.365 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T04:10:19.366 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:10:19.366 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T04:10:19.366 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:10:19.366 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:10:19.366 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:10:19.366 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:10:19.366 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:10:19.366 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:10:19.366 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:10:19.367 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T04:10:19.367 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:10:19.367 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:10:19.367 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:10:19.367 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:10:19.367 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T04:10:19.367 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:10:19.367 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:10:19.367 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:10:19.367 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:10:19.367 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T04:10:19.368 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T04:10:19.368 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:10:19.368 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:10:19.368 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:10:19.368 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:10:19.368 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:10:19.368 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:10:19.368 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:10:19.368 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:10:19.368 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:10:19.369 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:10:19.369 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:10:19.369 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T04:10:19.369 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:10:19.369 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:10:19.369 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:10:19.369 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:10:19.369 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T04:10:19.369 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:10:19.369 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:10:19.369 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:10:19.369 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:10:19.369 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T04:10:19.370 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T04:10:19.370 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:10:19.370 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:10:19.370 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:10:19.370 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:10:19.370 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:10:19.370 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:10:19.370 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:10:19.370 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:10:19.370 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:10:19.371 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:10:19.371 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:10:19.371 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T04:10:19.371 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:10:19.371 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:10:19.371 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:10:19.371 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:10:19.371 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T04:10:19.371 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:10:19.371 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:10:19.371 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:10:19.371 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:10:19.371 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T04:10:19.371 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T04:10:19.371 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:10:19.372 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:10:19.372 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:10:19.372 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:10:19.372 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:10:19.372 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:10:19.372 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T04:10:19.373 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T04:10:19.373 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:10:19.373 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:10:19.373 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:10:19.373 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:10:19.373 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:10:19.373 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:10:19.373 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T04:10:19.373 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:10:19.373 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:10:19.373 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:10:19.373 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 4 2024-09-16T04:10:19.373 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:10:19.373 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:10:19.373 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T04:10:19.374 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:10:19.374 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:10:19.374 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:10:19.374 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 3 2024-09-16T04:10:19.374 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:10:19.374 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:10:19.374 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:10:19.374 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T04:10:19.374 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T04:10:19.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.335+0000 7f5d65529640 10 librados: watch_flush enter 2024-09-16T04:10:19.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.335+0000 7f5d65529640 10 librados: watch_flush exit 2024-09-16T04:10:19.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.335+0000 7f5d65529640 20 client.1034295.objecter shutdown clearing up homeless session... 2024-09-16T04:10:19.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.335+0000 7f5d65529640 10 client.1034295.objecter successfully canceled tick 2024-09-16T04:10:19.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.335+0000 7f5d65529640 1 -- 172.21.15.42:0/2774339535 >> v2:172.21.15.154:6812/254845342 conn(0x7f5d440377f0 msgr2=0x7f5d44039cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:19.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.335+0000 7f5d65529640 1 --2- 172.21.15.42:0/2774339535 >> v2:172.21.15.154:6812/254845342 conn(0x7f5d440377f0 0x7f5d44039cb0 secure :-1 s=READY pgs=3930 cs=0 l=1 rev1=1 crypto rx=0x7f5d58004ba0 tx=0x7f5d58001720 comp rx=0 tx=0).stop 2024-09-16T04:10:19.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.335+0000 7f5d65529640 10 monclient: shutdown 2024-09-16T04:10:19.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.335+0000 7f5d65529640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:10:19.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.335+0000 7f5d65529640 1 -- 172.21.15.42:0/2774339535 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5d600a1330 msgr2=0x7f5d60135040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:19.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.335+0000 7f5d65529640 1 --2- 172.21.15.42:0/2774339535 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5d600a1330 0x7f5d60135040 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f5d50000c10 tx=0x7f5d5001abc0 comp rx=0 tx=0).stop 2024-09-16T04:10:19.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.335+0000 7f5d5f7fe640 1 -- 172.21.15.42:0/2774339535 reap_dead start 2024-09-16T04:10:19.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.335+0000 7f5d5f7fe640 1 -- 172.21.15.42:0/2774339535 reap_dead start 2024-09-16T04:10:19.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.335+0000 7f5d65529640 1 -- 172.21.15.42:0/2774339535 shutdown_connections 2024-09-16T04:10:19.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.335+0000 7f5d65529640 1 -- 172.21.15.42:0/2774339535 >> 172.21.15.42:0/2774339535 conn(0x7f5d600a2420 msgr2=0x7f5d60071710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:10:19.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.335+0000 7f5d65529640 1 -- 172.21.15.42:0/2774339535 shutdown_connections 2024-09-16T04:10:19.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.335+0000 7f5d65529640 1 -- 172.21.15.42:0/2774339535 wait complete. 2024-09-16T04:10:19.379 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:19.335+0000 7f5d65529640 1 librados: shutdown 2024-09-16T04:10:19.379 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T04:10:19.379 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-16T04:10:39.380 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-16T04:10:39.380 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-09-16T04:10:39.380 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-16T04:10:39.381 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-16T04:10:39.384 INFO:tasks.ceph.mon.b:Started 2024-09-16T04:10:39.384 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T04:10:39.384 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:10:39.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.511+0000 7ff793a99640 10 monclient: get_monmap_and_config 2024-09-16T04:10:39.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.511+0000 7ff793a99640 10 monclient: build_initial_monmap 2024-09-16T04:10:39.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.511+0000 7ff793a99640 10 monclient: monmap: 2024-09-16T04:10:39.516 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:10:39.516 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:10:39.516 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:10:39.513832+0000 2024-09-16T04:10:39.516 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:10:39.513832+0000 2024-09-16T04:10:39.516 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:10:39.516 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:10:39.516 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:10:39.516 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:10:39.516 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:10:39.517 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:10:39.517 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:10:39.517 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:10:39.521 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.511+0000 7ff793a99640 1 Processor -- start 2024-09-16T04:10:39.521 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.511+0000 7ff793a99640 1 -- start start 2024-09-16T04:10:39.521 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.511+0000 7ff793a99640 10 monclient: init 2024-09-16T04:10:39.521 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.511+0000 7ff793a99640 10 monclient: _reopen_session rank -1 2024-09-16T04:10:39.521 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.511+0000 7ff793a99640 10 monclient: _add_conns ranks=[2,1,3,4,0] 2024-09-16T04:10:39.521 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.511+0000 7ff793a99640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff78c09fe60 0x7ff78c0a0240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:10:39.521 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.511+0000 7ff793a99640 10 monclient(hunting): picked mon.noname-b con 0x7ff78c09fe60 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:10:39.521 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.511+0000 7ff793a99640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff78c0a2000 addr v1:172.21.15.154:6789/0 2024-09-16T04:10:39.521 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.511+0000 7ff793a99640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff78c06e8f0 0x7ff78c0a0780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:10:39.521 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.511+0000 7ff793a99640 10 monclient(hunting): picked mon.noname-a con 0x7ff78c06e8f0 addr v2:172.21.15.42:3300/0 2024-09-16T04:10:39.521 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.511+0000 7ff793a99640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff78c0a2770 con 0x7ff78c0a2000 2024-09-16T04:10:39.522 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.515+0000 7ff793a99640 10 monclient(hunting): start opening mon connection 2024-09-16T04:10:39.522 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.515+0000 7ff793a99640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7ff78c0a1e50 con 0x7ff78c06e8f0 2024-09-16T04:10:39.522 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.515+0000 7ff793a99640 10 monclient(hunting): start opening mon connection 2024-09-16T04:10:39.522 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.515+0000 7ff793a99640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7ff78c0a0cc0 con 0x7ff78c09fe60 2024-09-16T04:10:39.522 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.515+0000 7ff793a99640 10 monclient(hunting): _renew_subs 2024-09-16T04:10:39.522 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.515+0000 7ff793a99640 10 monclient(hunting): authenticate will time out at 2686.484032s 2024-09-16T04:10:39.522 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.515+0000 7ff792036640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff78c06e8f0 0x7ff78c0a0780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:10:39.522 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.515+0000 7ff791835640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff78c09fe60 0x7ff78c0a0240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:10:39.522 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.515+0000 7ff792036640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff78c06e8f0 0x7ff78c0a0780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:38626/0 (socket says 172.21.15.42:38626) 2024-09-16T04:10:39.522 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.515+0000 7ff792036640 1 -- 172.21.15.42:0/1650900211 learned_addr learned my addr 172.21.15.42:0/1650900211 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:10:39.522 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.515+0000 7ff792036640 10 monclient(hunting): get_auth_request con 0x7ff78c06e8f0 auth_method 0 2024-09-16T04:10:39.522 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.515+0000 7ff792036640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:10:39.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.515+0000 7ff792036640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:39.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.515+0000 7ff792036640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:39.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.515+0000 7ff791835640 10 monclient(hunting): get_auth_request con 0x7ff78c09fe60 auth_method 0 2024-09-16T04:10:39.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.515+0000 7ff791835640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:10:39.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.515+0000 7ff791835640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:39.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.515+0000 7ff791835640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:39.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.515+0000 7ff791835640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:10:39.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.515+0000 7ff791835640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:10:39.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.515+0000 7ff791835640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:10:39.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.515+0000 7ff792036640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:10:39.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.515+0000 7ff792036640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:10:39.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.515+0000 7ff792036640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:10:39.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.515+0000 7ff792036640 10 monclient(hunting): handle_auth_done global_id 1034298 payload 274 2024-09-16T04:10:39.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.515+0000 7ff792036640 1 -- 172.21.15.42:0/1650900211 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff78c09fe60 msgr2=0x7ff78c0a0240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:10:39.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.515+0000 7ff792036640 1 --2- 172.21.15.42:0/1650900211 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff78c09fe60 0x7ff78c0a0240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:10:39.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.515+0000 7ff792036640 1 -- 172.21.15.42:0/1650900211 >> v1:172.21.15.154:6789/0 conn(0x7ff78c0a2000 legacy=0x7ff78c137120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:39.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.515+0000 7ff792036640 10 monclient: _finish_hunting 0 2024-09-16T04:10:39.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.515+0000 7ff792036640 1 monclient: found mon.noname-a 2024-09-16T04:10:39.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.515+0000 7ff792036640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:10:39.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.515+0000 7ff792036640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:10:39.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.515+0000 7ff792036640 1 -- 172.21.15.42:0/1650900211 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff78c072580 con 0x7ff78c06e8f0 2024-09-16T04:10:39.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.515+0000 7ff791835640 1 --2- 172.21.15.42:0/1650900211 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff78c09fe60 0x7ff78c0a0240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:10:39.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.515+0000 7ff792036640 1 --2- 172.21.15.42:0/1650900211 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff78c06e8f0 0x7ff78c0a0780 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7ff78c0a0e40 tx=0x7ff78401c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=880ef31a7abb2c44 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:10:39.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.515+0000 7ff790833640 1 -- 172.21.15.42:0/1650900211 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7840b0070 con 0x7ff78c06e8f0 2024-09-16T04:10:39.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.515+0000 7ff790833640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:39.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.515+0000 7ff790833640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:10:39.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.515+0000 7ff790833640 10 monclient: dump: 2024-09-16T04:10:39.525 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:10:39.525 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:10:39.525 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:39.525 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:39.525 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:39.525 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:10:39.525 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:10:39.525 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:10:39.526 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:10:39.526 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:10:39.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.515+0000 7ff790833640 10 monclient: _finish_auth 0 2024-09-16T04:10:39.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.515+0000 7ff790833640 10 monclient: _check_auth_tickets 2024-09-16T04:10:39.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.515+0000 7ff790833640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:39.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.515+0000 7ff793a99640 5 monclient: authenticate success, global_id 1034298 2024-09-16T04:10:39.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.515+0000 7ff793a99640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:10:39.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.515+0000 7ff790833640 1 -- 172.21.15.42:0/1650900211 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff7840ab600 con 0x7ff78c06e8f0 2024-09-16T04:10:39.527 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.515+0000 7ff790833640 10 monclient: handle_config config(0 keys) 2024-09-16T04:10:39.527 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.515+0000 7ff793a99640 10 monclient: get_monmap_and_config success 2024-09-16T04:10:39.527 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.515+0000 7ff790833640 1 -- 172.21.15.42:0/1650900211 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7840ab9d0 con 0x7ff78c06e8f0 2024-09-16T04:10:39.527 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.515+0000 7ff790833640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:39.527 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.515+0000 7ff793a99640 4 set_mon_vals no callback set 2024-09-16T04:10:39.527 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.515+0000 7ff790833640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:10:39.527 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.515+0000 7ff790833640 10 monclient: dump: 2024-09-16T04:10:39.527 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:10:39.527 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:10:39.527 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:39.527 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:39.528 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:39.528 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:10:39.528 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:10:39.528 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:10:39.528 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:10:39.528 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:10:39.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.519+0000 7ff793a99640 10 monclient: shutdown 2024-09-16T04:10:39.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.519+0000 7ff793a99640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:10:39.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.519+0000 7ff793a99640 1 -- 172.21.15.42:0/1650900211 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff78c06e8f0 msgr2=0x7ff78c0a0780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:39.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.519+0000 7ff793a99640 1 --2- 172.21.15.42:0/1650900211 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff78c06e8f0 0x7ff78c0a0780 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7ff78c0a0e40 tx=0x7ff78401c400 comp rx=0 tx=0).stop 2024-09-16T04:10:39.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.519+0000 7ff793a99640 1 -- 172.21.15.42:0/1650900211 shutdown_connections 2024-09-16T04:10:39.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.519+0000 7ff793a99640 1 --2- 172.21.15.42:0/1650900211 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff78c06e8f0 0x7ff78c0a0780 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:10:39.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.519+0000 7ff793a99640 1 --2- 172.21.15.42:0/1650900211 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff78c09fe60 0x7ff78c0a0240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:10:39.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.519+0000 7ff793a99640 1 -- 172.21.15.42:0/1650900211 >> 172.21.15.42:0/1650900211 conn(0x7ff78c075ae0 msgr2=0x7ff78c075ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:10:39.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.519+0000 7ff793a99640 1 -- 172.21.15.42:0/1650900211 shutdown_connections 2024-09-16T04:10:39.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.519+0000 7ff793a99640 1 -- 172.21.15.42:0/1650900211 wait complete. 2024-09-16T04:10:39.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.519+0000 7ff793a99640 10 monclient: build_initial_monmap 2024-09-16T04:10:39.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.519+0000 7ff793a99640 10 monclient: monmap: 2024-09-16T04:10:39.536 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:10:39.536 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:10:39.536 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:10:39.536 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:10:39.536 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:10:39.536 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:10:39.536 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:10:39.536 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:10:39.536 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:10:39.536 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:10:39.536 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.519+0000 7ff793a99640 1 librados: starting msgr at 2024-09-16T04:10:39.536 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.519+0000 7ff793a99640 1 librados: starting objecter 2024-09-16T04:10:39.537 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.519+0000 7ff793a99640 1 Processor -- start 2024-09-16T04:10:39.537 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.519+0000 7ff793a99640 1 -- start start 2024-09-16T04:10:39.537 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.519+0000 7ff793a99640 1 librados: setting wanted keys 2024-09-16T04:10:39.537 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.519+0000 7ff793a99640 1 librados: calling monclient init 2024-09-16T04:10:39.537 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.519+0000 7ff793a99640 10 monclient: init 2024-09-16T04:10:39.537 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.519+0000 7ff793a99640 10 monclient: _reopen_session rank -1 2024-09-16T04:10:39.537 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.519+0000 7ff793a99640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:10:39.537 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.519+0000 7ff793a99640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff78c06e8f0 0x7ff78c1cf2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:10:39.538 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.519+0000 7ff793a99640 10 monclient(hunting): picked mon.noname-c con 0x7ff78c06e8f0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:10:39.538 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.519+0000 7ff793a99640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff78c09fe60 0x7ff78c1366d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:10:39.538 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.519+0000 7ff793a99640 10 monclient(hunting): picked mon.noname-a con 0x7ff78c09fe60 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:10:39.538 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.519+0000 7ff793a99640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff78c0a2000 0x7ff78c11d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:10:39.538 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.519+0000 7ff793a99640 10 monclient(hunting): picked mon.noname-b con 0x7ff78c0a2000 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:10:39.538 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.519+0000 7ff793a99640 10 monclient(hunting): start opening mon connection 2024-09-16T04:10:39.538 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.519+0000 7ff793a99640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7ff78c141a40 con 0x7ff78c09fe60 2024-09-16T04:10:39.538 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.519+0000 7ff793a99640 10 monclient(hunting): start opening mon connection 2024-09-16T04:10:39.538 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.519+0000 7ff793a99640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff78c1418c0 con 0x7ff78c0a2000 2024-09-16T04:10:39.538 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.519+0000 7ff793a99640 10 monclient(hunting): start opening mon connection 2024-09-16T04:10:39.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.519+0000 7ff793a99640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7ff78c141bc0 con 0x7ff78c06e8f0 2024-09-16T04:10:39.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.519+0000 7ff793a99640 10 monclient(hunting): _renew_subs 2024-09-16T04:10:39.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.519+0000 7ff793a99640 10 monclient(hunting): authenticate will time out at 2686.490172s 2024-09-16T04:10:39.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.519+0000 7ff791835640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff78c06e8f0 0x7ff78c1cf2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:10:39.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.519+0000 7ff791034640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff78c09fe60 0x7ff78c1366d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:10:39.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.519+0000 7ff791835640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff78c06e8f0 0x7ff78c1cf2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:48596/0 (socket says 172.21.15.42:48596) 2024-09-16T04:10:39.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.519+0000 7ff791034640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff78c09fe60 0x7ff78c1366d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:38640/0 (socket says 172.21.15.42:38640) 2024-09-16T04:10:39.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.519+0000 7ff791835640 1 -- 172.21.15.42:0/571292570 learned_addr learned my addr 172.21.15.42:0/571292570 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:10:39.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.519+0000 7ff791034640 10 monclient(hunting): get_auth_request con 0x7ff78c09fe60 auth_method 0 2024-09-16T04:10:39.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.519+0000 7ff791034640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:10:39.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.519+0000 7ff791034640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:39.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.519+0000 7ff791034640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:39.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.519+0000 7ff791835640 10 monclient(hunting): get_auth_request con 0x7ff78c06e8f0 auth_method 0 2024-09-16T04:10:39.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.519+0000 7ff791835640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:10:39.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.519+0000 7ff791835640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:39.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.519+0000 7ff791835640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:39.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.519+0000 7ff791034640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:10:39.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.519+0000 7ff791034640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:10:39.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.519+0000 7ff791034640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:10:39.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.519+0000 7ff791835640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:10:39.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.519+0000 7ff791835640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:10:39.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.519+0000 7ff791835640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:10:39.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.519+0000 7ff792036640 1 --2- 172.21.15.42:0/571292570 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff78c0a2000 0x7ff78c11d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:10:39.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.519+0000 7ff792036640 10 monclient(hunting): get_auth_request con 0x7ff78c0a2000 auth_method 0 2024-09-16T04:10:39.541 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.519+0000 7ff792036640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:10:39.541 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.519+0000 7ff792036640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:39.541 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.519+0000 7ff792036640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:39.541 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.519+0000 7ff792036640 1 -- 172.21.15.42:0/571292570 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff78c0a2000 msgr2=0x7ff78c11d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-16T04:10:39.541 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.519+0000 7ff792036640 1 -- 172.21.15.42:0/571292570 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff78c0a2000 msgr2=0x7ff78c11d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:10:39.541 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.519+0000 7ff792036640 1 --2- 172.21.15.42:0/571292570 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff78c0a2000 0x7ff78c11d210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:10:39.541 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.519+0000 7ff792036640 1 --2- 172.21.15.42:0/571292570 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff78c0a2000 0x7ff78c11d210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:10:39.542 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.519+0000 7ff791034640 10 monclient(hunting): handle_auth_done global_id 1034301 payload 824 2024-09-16T04:10:39.542 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.519+0000 7ff791034640 1 -- 172.21.15.42:0/571292570 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff78c06e8f0 msgr2=0x7ff78c1cf2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:39.542 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.519+0000 7ff791034640 1 --2- 172.21.15.42:0/571292570 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff78c06e8f0 0x7ff78c1cf2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:10:39.542 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.519+0000 7ff791034640 1 -- 172.21.15.42:0/571292570 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff78c0a2000 msgr2=0x7ff78c11d210 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:10:39.542 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.519+0000 7ff791034640 1 --2- 172.21.15.42:0/571292570 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff78c0a2000 0x7ff78c11d210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:10:39.542 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.519+0000 7ff791034640 10 monclient: _finish_hunting 0 2024-09-16T04:10:39.542 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.519+0000 7ff791034640 1 monclient: found mon.noname-a 2024-09-16T04:10:39.542 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.519+0000 7ff791034640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:10:39.543 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.519+0000 7ff791034640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:10:39.543 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.519+0000 7ff791034640 1 -- 172.21.15.42:0/571292570 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff78c0a5820 con 0x7ff78c09fe60 2024-09-16T04:10:39.543 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.519+0000 7ff791034640 1 --2- 172.21.15.42:0/571292570 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff78c09fe60 0x7ff78c1366d0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7ff77c00c3c0 tx=0x7ff77c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:10:39.543 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.519+0000 7ff7827fc640 10 client.?.objecter ms_handle_connect 0x7ff78c09fe60 2024-09-16T04:10:39.543 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.519+0000 7ff7827fc640 10 client.?.objecter resend_mon_ops 2024-09-16T04:10:39.543 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.519+0000 7ff7827fc640 1 -- 172.21.15.42:0/571292570 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff77c0052d0 con 0x7ff78c09fe60 2024-09-16T04:10:39.543 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.519+0000 7ff7827fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:39.543 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.519+0000 7ff7827fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:10:39.543 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.519+0000 7ff7827fc640 10 monclient: dump: 2024-09-16T04:10:39.544 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:10:39.544 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:10:39.544 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:39.544 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:39.544 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:39.544 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:10:39.544 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:10:39.545 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:10:39.545 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:10:39.545 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:10:39.545 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.519+0000 7ff7827fc640 10 monclient: _finish_auth 0 2024-09-16T04:10:39.545 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.519+0000 7ff7827fc640 10 monclient: _check_auth_tickets 2024-09-16T04:10:39.545 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.519+0000 7ff7827fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:39.545 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.519+0000 7ff7827fc640 1 -- 172.21.15.42:0/571292570 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff77c005470 con 0x7ff78c09fe60 2024-09-16T04:10:39.545 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.519+0000 7ff793a99640 5 monclient: authenticate success, global_id 1034301 2024-09-16T04:10:39.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.519+0000 7ff793a99640 10 monclient: _renew_subs 2024-09-16T04:10:39.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.519+0000 7ff793a99640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:10:39.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.519+0000 7ff793a99640 1 -- 172.21.15.42:0/571292570 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff78c121060 con 0x7ff78c09fe60 2024-09-16T04:10:39.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.519+0000 7ff7827fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:10:39.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.519+0000 7ff7827fc640 1 -- 172.21.15.42:0/571292570 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff77c007260 con 0x7ff78c09fe60 2024-09-16T04:10:39.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.519+0000 7ff7827fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:39.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.519+0000 7ff7827fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:10:39.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.519+0000 7ff7827fc640 10 monclient: dump: 2024-09-16T04:10:39.546 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:10:39.547 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:10:39.547 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:39.547 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:39.547 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:39.547 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:10:39.547 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:10:39.547 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:10:39.548 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:10:39.548 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:10:39.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.519+0000 7ff783fff640 4 set_mon_vals no callback set 2024-09-16T04:10:39.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.519+0000 7ff793a99640 10 client.1034301.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:10:39.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.519+0000 7ff793a99640 10 monclient: _renew_subs 2024-09-16T04:10:39.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.519+0000 7ff793a99640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:10:39.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.519+0000 7ff793a99640 1 -- 172.21.15.42:0/571292570 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff78c0a61a0 con 0x7ff78c09fe60 2024-09-16T04:10:39.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.519+0000 7ff793a99640 1 librados: init done 2024-09-16T04:10:39.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.523+0000 7ff7827fc640 1 -- 172.21.15.42:0/571292570 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7ff77c007400 con 0x7ff78c09fe60 2024-09-16T04:10:39.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.523+0000 7ff793a99640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:39.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.523+0000 7ff793a99640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:39.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.523+0000 7ff793a99640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:10:39.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.523+0000 7ff793a99640 1 -- 172.21.15.42:0/571292570 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff78c11ed70 con 0x7ff78c09fe60 2024-09-16T04:10:39.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.523+0000 7ff7827fc640 1 --2- 172.21.15.42:0/571292570 >> v2:172.21.15.154:6812/254845342 conn(0x7ff770037840 0x7ff770039d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:10:39.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.523+0000 7ff7827fc640 1 -- 172.21.15.42:0/571292570 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(706..706 src has 451..706) ==== 5040+0+0 (secure 0 0 0) 0x7ff77c0480a0 con 0x7ff78c09fe60 2024-09-16T04:10:39.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.523+0000 7ff7827fc640 10 client.1034301.objecter ms_dispatch 0x7ff78c000ba0 osd_map(706..706 src has 451..706) 2024-09-16T04:10:39.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.523+0000 7ff7827fc640 3 client.1034301.objecter handle_osd_map got epochs [706,706] > 0 2024-09-16T04:10:39.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.523+0000 7ff7827fc640 3 client.1034301.objecter handle_osd_map decoding full epoch 706 2024-09-16T04:10:39.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.523+0000 7ff7827fc640 20 client.1034301.objecter dump_active .. 0 homeless 2024-09-16T04:10:39.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.523+0000 7ff791835640 1 --2- 172.21.15.42:0/571292570 >> v2:172.21.15.154:6812/254845342 conn(0x7ff770037840 0x7ff770039d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:10:39.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.523+0000 7ff791835640 10 monclient: get_auth_request con 0x7ff770037840 auth_method 0 2024-09-16T04:10:39.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.523+0000 7ff791835640 1 --2- 172.21.15.42:0/571292570 >> v2:172.21.15.154:6812/254845342 conn(0x7ff770037840 0x7ff770039d00 secure :-1 s=READY pgs=3932 cs=0 l=1 rev1=1 crypto rx=0x7ff788009bd0 tx=0x7ff78800b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:10:39.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.523+0000 7ff7827fc640 10 client.1034301.objecter ms_handle_connect 0x7ff770037840 2024-09-16T04:10:39.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.527+0000 7ff7827fc640 1 -- 172.21.15.42:0/571292570 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7ff78c0a2770 con 0x7ff78c09fe60 2024-09-16T04:10:39.551 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.527+0000 7ff7827fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:39.551 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.527+0000 7ff7827fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:10:39.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.751+0000 7ff793a99640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:10:39.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.751+0000 7ff793a99640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:10:39.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.751+0000 7ff793a99640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:10:39.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.751+0000 7ff793a99640 1 -- 172.21.15.42:0/571292570 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7ff78c11faf0 con 0x7ff78c09fe60 2024-09-16T04:10:39.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.751+0000 7ff7827fc640 1 -- 172.21.15.42:0/571292570 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7ff77c0067e0 con 0x7ff78c09fe60 2024-09-16T04:10:39.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.751+0000 7ff7827fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:10:39.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.751+0000 7ff7827fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:10:39.756 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":260,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":22,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:10:39.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.751+0000 7ff793a99640 10 librados: watch_flush enter 2024-09-16T04:10:39.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.751+0000 7ff793a99640 10 librados: watch_flush exit 2024-09-16T04:10:39.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.751+0000 7ff793a99640 20 client.1034301.objecter shutdown clearing up homeless session... 2024-09-16T04:10:39.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.751+0000 7ff793a99640 10 client.1034301.objecter successfully canceled tick 2024-09-16T04:10:39.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.751+0000 7ff793a99640 1 -- 172.21.15.42:0/571292570 >> v2:172.21.15.154:6812/254845342 conn(0x7ff770037840 msgr2=0x7ff770039d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:39.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.751+0000 7ff793a99640 1 --2- 172.21.15.42:0/571292570 >> v2:172.21.15.154:6812/254845342 conn(0x7ff770037840 0x7ff770039d00 secure :-1 s=READY pgs=3932 cs=0 l=1 rev1=1 crypto rx=0x7ff788009bd0 tx=0x7ff78800b040 comp rx=0 tx=0).stop 2024-09-16T04:10:39.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.751+0000 7ff793a99640 10 monclient: shutdown 2024-09-16T04:10:39.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.751+0000 7ff793a99640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:10:39.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.751+0000 7ff793a99640 1 -- 172.21.15.42:0/571292570 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff78c09fe60 msgr2=0x7ff78c1366d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:39.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.751+0000 7ff793a99640 1 --2- 172.21.15.42:0/571292570 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff78c09fe60 0x7ff78c1366d0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7ff77c00c3c0 tx=0x7ff77c00c880 comp rx=0 tx=0).stop 2024-09-16T04:10:39.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.755+0000 7ff793a99640 1 -- 172.21.15.42:0/571292570 shutdown_connections 2024-09-16T04:10:39.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.755+0000 7ff793a99640 1 --2- 172.21.15.42:0/571292570 >> v2:172.21.15.154:6812/254845342 conn(0x7ff770037840 0x7ff770039d00 unknown :-1 s=CLOSED pgs=3932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:10:39.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.755+0000 7ff793a99640 1 --2- 172.21.15.42:0/571292570 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff78c0a2000 0x7ff78c11d210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:10:39.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.755+0000 7ff793a99640 1 --2- 172.21.15.42:0/571292570 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff78c09fe60 0x7ff78c1366d0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:10:39.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.755+0000 7ff793a99640 1 --2- 172.21.15.42:0/571292570 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff78c06e8f0 0x7ff78c1cf2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:10:39.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.755+0000 7ff793a99640 1 -- 172.21.15.42:0/571292570 >> 172.21.15.42:0/571292570 conn(0x7ff78c075ae0 msgr2=0x7ff78c070770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:10:39.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.755+0000 7ff793a99640 1 -- 172.21.15.42:0/571292570 shutdown_connections 2024-09-16T04:10:39.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.755+0000 7ff793a99640 1 -- 172.21.15.42:0/571292570 wait complete. 2024-09-16T04:10:39.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:39.755+0000 7ff793a99640 1 librados: shutdown 2024-09-16T04:10:42.775 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:10:42.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.903+0000 7fba72db8640 10 monclient: get_monmap_and_config 2024-09-16T04:10:42.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.903+0000 7fba72db8640 10 monclient: build_initial_monmap 2024-09-16T04:10:42.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.903+0000 7fba72db8640 10 monclient: monmap: 2024-09-16T04:10:42.907 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:10:42.907 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:10:42.907 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:10:42.905784+0000 2024-09-16T04:10:42.907 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:10:42.905784+0000 2024-09-16T04:10:42.908 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:10:42.908 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:10:42.908 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:10:42.908 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:10:42.908 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:10:42.908 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:10:42.908 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:10:42.908 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:10:42.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.903+0000 7fba72db8640 1 Processor -- start 2024-09-16T04:10:42.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.903+0000 7fba72db8640 1 -- start start 2024-09-16T04:10:42.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.903+0000 7fba72db8640 10 monclient: init 2024-09-16T04:10:42.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.903+0000 7fba72db8640 10 monclient: _reopen_session rank -1 2024-09-16T04:10:42.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.903+0000 7fba72db8640 10 monclient: _add_conns ranks=[1,4,0,2,3] 2024-09-16T04:10:42.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.903+0000 7fba72db8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fba6c0a1330 addr v1:172.21.15.154:6789/0 2024-09-16T04:10:42.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.903+0000 7fba72db8640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fba6c06dac0 0x7fba6c13a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:10:42.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.903+0000 7fba72db8640 10 monclient(hunting): picked mon.noname-c con 0x7fba6c06dac0 addr v2:172.21.15.154:3300/0 2024-09-16T04:10:42.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.903+0000 7fba72db8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fba6c0a1e20 addr v1:172.21.15.42:6789/0 2024-09-16T04:10:42.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.903+0000 7fba72db8640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fba6c09c090 con 0x7fba6c0a1e20 2024-09-16T04:10:42.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.903+0000 7fba72db8640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fba6c09f610 con 0x7fba6c0a1330 2024-09-16T04:10:42.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.903+0000 7fba72db8640 10 monclient(hunting): start opening mon connection 2024-09-16T04:10:42.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.903+0000 7fba72db8640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fba6c09f8f0 con 0x7fba6c06dac0 2024-09-16T04:10:42.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.903+0000 7fba72db8640 10 monclient(hunting): _renew_subs 2024-09-16T04:10:42.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.903+0000 7fba72db8640 10 monclient(hunting): authenticate will time out at 2689.875242s 2024-09-16T04:10:42.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.903+0000 7fba71355640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fba6c0a1e20 0x7fba6c13c890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:40282/0 (socket says 172.21.15.42:40282) 2024-09-16T04:10:42.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.903+0000 7fba71355640 1 -- 172.21.15.42:0/3568368582 learned_addr learned my addr 172.21.15.42:0/3568368582 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T04:10:42.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.903+0000 7fba6bfff640 1 --2- 172.21.15.42:0/3568368582 >> v2:172.21.15.154:3300/0 conn(0x7fba6c06dac0 0x7fba6c13a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:10:42.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.903+0000 7fba6b7fe640 1 -- 172.21.15.42:0/3568368582 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 817892379 0 0) 0x7fba6c09c090 con 0x7fba6c0a1e20 2024-09-16T04:10:42.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.903+0000 7fba6b7fe640 10 monclient(hunting): _negotiate 2024-09-16T04:10:42.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.903+0000 7fba6b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:42.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.903+0000 7fba6b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:42.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.903+0000 7fba6b7fe640 10 monclient(hunting): my global_id is 1034304 2024-09-16T04:10:42.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.903+0000 7fba6b7fe640 1 -- 172.21.15.42:0/3568368582 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fba58003530 con 0x7fba6c0a1e20 2024-09-16T04:10:42.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.903+0000 7fba6bfff640 10 monclient(hunting): get_auth_request con 0x7fba6c06dac0 auth_method 0 2024-09-16T04:10:42.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.903+0000 7fba6bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:10:42.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.903+0000 7fba6bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:42.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.903+0000 7fba6bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:42.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.907+0000 7fba6bfff640 1 -- 172.21.15.42:0/3568368582 >> v2:172.21.15.154:3300/0 conn(0x7fba6c06dac0 msgr2=0x7fba6c13a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:10:42.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.907+0000 7fba6bfff640 1 -- 172.21.15.42:0/3568368582 >> v2:172.21.15.154:3300/0 conn(0x7fba6c06dac0 msgr2=0x7fba6c13a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:10:42.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.907+0000 7fba6bfff640 1 --2- 172.21.15.42:0/3568368582 >> v2:172.21.15.154:3300/0 conn(0x7fba6c06dac0 0x7fba6c13a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:10:42.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.907+0000 7fba6bfff640 1 --2- 172.21.15.42:0/3568368582 >> v2:172.21.15.154:3300/0 conn(0x7fba6c06dac0 0x7fba6c13a340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:10:42.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.907+0000 7fba6b7fe640 1 -- 172.21.15.42:0/3568368582 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2454630615 0 0) 0x7fba58003530 con 0x7fba6c0a1e20 2024-09-16T04:10:42.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.907+0000 7fba6b7fe640 1 -- 172.21.15.42:0/3568368582 >> v2:172.21.15.154:3300/0 conn(0x7fba6c06dac0 msgr2=0x7fba6c13a340 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:10:42.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.907+0000 7fba6b7fe640 1 --2- 172.21.15.42:0/3568368582 >> v2:172.21.15.154:3300/0 conn(0x7fba6c06dac0 0x7fba6c13a340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:10:42.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.907+0000 7fba6b7fe640 1 -- 172.21.15.42:0/3568368582 >> v1:172.21.15.154:6789/0 conn(0x7fba6c0a1330 legacy=0x7fba6c0a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:42.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.907+0000 7fba6b7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:10:42.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.907+0000 7fba6b7fe640 1 monclient: found mon.noname-a-legacy 2024-09-16T04:10:42.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.907+0000 7fba6b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:10:42.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.907+0000 7fba6b7fe640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.42:6789/0 2024-09-16T04:10:42.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.907+0000 7fba6b7fe640 1 -- 172.21.15.42:0/3568368582 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba6c09fbd0 con 0x7fba6c0a1e20 2024-09-16T04:10:42.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.907+0000 7fba6b7fe640 10 monclient: _finish_auth 0 2024-09-16T04:10:42.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.907+0000 7fba6b7fe640 10 monclient: _check_auth_tickets 2024-09-16T04:10:42.916 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.907+0000 7fba6b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:42.916 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.907+0000 7fba6b7fe640 1 -- 172.21.15.42:0/3568368582 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2177413734 0 0) 0x7fba64001a00 con 0x7fba6c0a1e20 2024-09-16T04:10:42.916 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.907+0000 7fba6b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:42.916 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.907+0000 7fba6b7fe640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T04:10:42.916 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.907+0000 7fba6b7fe640 10 monclient: dump: 2024-09-16T04:10:42.916 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:10:42.916 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:10:42.916 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:42.916 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:42.916 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:42.916 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:10:42.916 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:10:42.916 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:10:42.917 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:10:42.917 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:10:42.917 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.907+0000 7fba6b7fe640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] but i'm connected to v1:172.21.15.42:6789/0, reconnecting 2024-09-16T04:10:42.917 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.907+0000 7fba6b7fe640 10 monclient: _reopen_session rank -1 2024-09-16T04:10:42.917 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.907+0000 7fba6b7fe640 1 -- 172.21.15.42:0/3568368582 >> v1:172.21.15.42:6789/0 conn(0x7fba6c0a1e20 legacy=0x7fba6c13c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:42.917 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.907+0000 7fba6b7fe640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:10:42.917 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.907+0000 7fba6b7fe640 1 --2- 172.21.15.42:0/3568368582 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fba580039c0 0x7fba58003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:10:42.917 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.907+0000 7fba6b7fe640 10 monclient(hunting): picked mon.c con 0x7fba580039c0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:10:42.917 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.907+0000 7fba6b7fe640 1 --2- 172.21.15.42:0/3568368582 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fba58004810 0x7fba58004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:10:42.917 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.907+0000 7fba6b7fe640 10 monclient(hunting): picked mon.b con 0x7fba58004810 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:10:42.917 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.907+0000 7fba6b7fe640 1 --2- 172.21.15.42:0/3568368582 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fba580052f0 0x7fba5800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:10:42.917 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.907+0000 7fba6b7fe640 10 monclient(hunting): picked mon.a con 0x7fba580052f0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:10:42.917 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.907+0000 7fba6b7fe640 10 monclient(hunting): start opening mon connection 2024-09-16T04:10:42.917 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.907+0000 7fba6b7fe640 1 -- 172.21.15.42:0/3568368582 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fba6c09f8f0 con 0x7fba580052f0 2024-09-16T04:10:42.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.907+0000 7fba6b7fe640 10 monclient(hunting): start opening mon connection 2024-09-16T04:10:42.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.907+0000 7fba6b7fe640 1 -- 172.21.15.42:0/3568368582 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fba5800c1c0 con 0x7fba58004810 2024-09-16T04:10:42.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.907+0000 7fba6b7fe640 10 monclient(hunting): start opening mon connection 2024-09-16T04:10:42.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.907+0000 7fba6b7fe640 1 -- 172.21.15.42:0/3568368582 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fba5800c3c0 con 0x7fba580039c0 2024-09-16T04:10:42.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.907+0000 7fba6b7fe640 10 monclient(hunting): _renew_subs 2024-09-16T04:10:42.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.907+0000 7fba6b7fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:10:42.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.907+0000 7fba70b54640 1 --2- 172.21.15.42:0/3568368582 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fba580039c0 0x7fba58003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:10:42.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.907+0000 7fba70b54640 10 monclient(hunting): get_auth_request con 0x7fba580039c0 auth_method 0 2024-09-16T04:10:42.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.907+0000 7fba70b54640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:10:42.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.907+0000 7fba70b54640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:42.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.907+0000 7fba70b54640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:10:42.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.907+0000 7fba71355640 1 --2- 172.21.15.42:0/3568368582 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fba580052f0 0x7fba5800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:10:42.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.907+0000 7fba71355640 10 monclient(hunting): get_auth_request con 0x7fba580052f0 auth_method 0 2024-09-16T04:10:42.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.907+0000 7fba71355640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:10:42.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.907+0000 7fba71355640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:42.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.907+0000 7fba71355640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:10:42.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.907+0000 7fba70b54640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:10:42.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.907+0000 7fba70b54640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:10:42.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.907+0000 7fba70b54640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:10:42.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.907+0000 7fba6bfff640 1 --2- 172.21.15.42:0/3568368582 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fba58004810 0x7fba58004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:10:42.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.907+0000 7fba6bfff640 10 monclient(hunting): get_auth_request con 0x7fba58004810 auth_method 0 2024-09-16T04:10:42.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.907+0000 7fba6bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:10:42.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.907+0000 7fba6bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:42.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.907+0000 7fba6bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:10:42.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.907+0000 7fba71355640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:10:42.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.907+0000 7fba71355640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:10:42.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.907+0000 7fba71355640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:10:42.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.907+0000 7fba6bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:10:42.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.907+0000 7fba6bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:10:42.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.907+0000 7fba6bfff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:10:42.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.907+0000 7fba70b54640 10 monclient(hunting): handle_auth_done global_id 1034304 payload 293 2024-09-16T04:10:42.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.907+0000 7fba70b54640 1 -- 172.21.15.42:0/3568368582 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fba58004810 msgr2=0x7fba58004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:10:42.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.907+0000 7fba70b54640 1 --2- 172.21.15.42:0/3568368582 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fba58004810 0x7fba58004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:10:42.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.907+0000 7fba70b54640 1 -- 172.21.15.42:0/3568368582 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fba580052f0 msgr2=0x7fba5800b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:10:42.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.907+0000 7fba70b54640 1 --2- 172.21.15.42:0/3568368582 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fba580052f0 0x7fba5800b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:10:42.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.907+0000 7fba70b54640 10 monclient: _finish_hunting 0 2024-09-16T04:10:42.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.907+0000 7fba70b54640 1 monclient: found mon.c 2024-09-16T04:10:42.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.907+0000 7fba70b54640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:10:42.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.907+0000 7fba70b54640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:10:42.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.907+0000 7fba70b54640 1 -- 172.21.15.42:0/3568368582 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba58003530 con 0x7fba580039c0 2024-09-16T04:10:42.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.907+0000 7fba70b54640 10 monclient: _finish_auth 0 2024-09-16T04:10:42.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.907+0000 7fba70b54640 10 monclient: _check_auth_tickets 2024-09-16T04:10:42.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.907+0000 7fba70b54640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:42.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.907+0000 7fba72db8640 5 monclient: authenticate success, global_id 1034304 2024-09-16T04:10:42.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.907+0000 7fba72db8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:10:42.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.907+0000 7fba71355640 1 --2- 172.21.15.42:0/3568368582 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fba580052f0 0x7fba5800b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:10:42.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.907+0000 7fba71355640 1 -- 172.21.15.42:0/3568368582 reap_dead start 2024-09-16T04:10:42.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.907+0000 7fba70b54640 1 --2- 172.21.15.42:0/3568368582 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fba580039c0 0x7fba58003de0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fba54001750 tx=0x7fba5401c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7af50a886f3aab53 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:10:42.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.911+0000 7fba6b7fe640 1 -- 172.21.15.42:0/3568368582 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba540013a0 con 0x7fba580039c0 2024-09-16T04:10:42.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.911+0000 7fba6b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:42.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.911+0000 7fba6b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:10:42.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.911+0000 7fba6b7fe640 10 monclient: dump: 2024-09-16T04:10:42.927 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:10:42.927 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:10:42.927 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:42.927 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:42.927 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:42.927 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:10:42.927 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:10:42.927 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:10:42.928 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:10:42.928 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:10:42.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.911+0000 7fba72db8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:10:42.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.911+0000 7fba6b7fe640 1 -- 172.21.15.42:0/3568368582 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fba540052f0 con 0x7fba580039c0 2024-09-16T04:10:42.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.911+0000 7fba6b7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:10:42.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.911+0000 7fba72db8640 10 monclient: get_monmap_and_config success 2024-09-16T04:10:42.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.911+0000 7fba72db8640 4 set_mon_vals no callback set 2024-09-16T04:10:42.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.911+0000 7fba6b7fe640 1 -- 172.21.15.42:0/3568368582 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba54003840 con 0x7fba580039c0 2024-09-16T04:10:42.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.911+0000 7fba6b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:42.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.911+0000 7fba6b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:10:42.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.911+0000 7fba6b7fe640 10 monclient: dump: 2024-09-16T04:10:42.928 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:10:42.928 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:10:42.929 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:42.929 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:42.929 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:42.929 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:10:42.929 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:10:42.929 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:10:42.929 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:10:42.929 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:10:42.929 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.911+0000 7fba72db8640 10 monclient: shutdown 2024-09-16T04:10:42.929 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.911+0000 7fba72db8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:10:42.929 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.911+0000 7fba72db8640 1 -- 172.21.15.42:0/3568368582 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fba580039c0 msgr2=0x7fba58003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:42.929 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.911+0000 7fba72db8640 1 --2- 172.21.15.42:0/3568368582 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fba580039c0 0x7fba58003de0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fba54001750 tx=0x7fba5401c3a0 comp rx=0 tx=0).stop 2024-09-16T04:10:42.929 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.911+0000 7fba72db8640 1 -- 172.21.15.42:0/3568368582 shutdown_connections 2024-09-16T04:10:42.930 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.911+0000 7fba72db8640 1 --2- 172.21.15.42:0/3568368582 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fba580039c0 0x7fba58003de0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:10:42.930 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.911+0000 7fba72db8640 1 -- 172.21.15.42:0/3568368582 >> 172.21.15.42:0/3568368582 conn(0x7fba6c0a2420 msgr2=0x7fba6c0a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:10:42.930 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.911+0000 7fba72db8640 1 -- 172.21.15.42:0/3568368582 shutdown_connections 2024-09-16T04:10:42.930 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.911+0000 7fba72db8640 1 -- 172.21.15.42:0/3568368582 wait complete. 2024-09-16T04:10:42.930 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.911+0000 7fba72db8640 10 monclient: build_initial_monmap 2024-09-16T04:10:42.930 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.911+0000 7fba72db8640 10 monclient: monmap: 2024-09-16T04:10:42.930 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:10:42.930 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:10:42.930 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:10:42.930 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:10:42.930 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:10:42.930 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:10:42.930 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:10:42.930 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:10:42.931 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:10:42.931 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:10:42.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.911+0000 7fba72db8640 1 librados: starting msgr at 2024-09-16T04:10:42.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.911+0000 7fba72db8640 1 librados: starting objecter 2024-09-16T04:10:42.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.911+0000 7fba72db8640 1 Processor -- start 2024-09-16T04:10:42.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.911+0000 7fba72db8640 1 -- start start 2024-09-16T04:10:42.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.911+0000 7fba72db8640 1 librados: setting wanted keys 2024-09-16T04:10:42.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.911+0000 7fba72db8640 1 librados: calling monclient init 2024-09-16T04:10:42.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.911+0000 7fba72db8640 10 monclient: init 2024-09-16T04:10:42.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.911+0000 7fba72db8640 10 monclient: _reopen_session rank -1 2024-09-16T04:10:42.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.911+0000 7fba72db8640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:10:42.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.911+0000 7fba72db8640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fba580039c0 0x7fba6c1cc3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:10:42.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.911+0000 7fba72db8640 10 monclient(hunting): picked mon.noname-b con 0x7fba580039c0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:10:42.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.911+0000 7fba72db8640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fba6c135490 0x7fba6c1358d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:10:42.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.911+0000 7fba72db8640 10 monclient(hunting): picked mon.noname-c con 0x7fba6c135490 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:10:42.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.911+0000 7fba72db8640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fba6c09cb90 0x7fba6c09d010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:10:42.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.911+0000 7fba72db8640 10 monclient(hunting): picked mon.noname-a con 0x7fba6c09cb90 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:10:42.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.911+0000 7fba72db8640 10 monclient(hunting): start opening mon connection 2024-09-16T04:10:42.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.911+0000 7fba72db8640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fba6c09eef0 con 0x7fba6c09cb90 2024-09-16T04:10:42.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.911+0000 7fba72db8640 10 monclient(hunting): start opening mon connection 2024-09-16T04:10:42.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.911+0000 7fba72db8640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fba6c09ed70 con 0x7fba580039c0 2024-09-16T04:10:42.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.911+0000 7fba72db8640 10 monclient(hunting): start opening mon connection 2024-09-16T04:10:42.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.911+0000 7fba72db8640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fba6c0a1710 con 0x7fba6c135490 2024-09-16T04:10:42.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.911+0000 7fba72db8640 10 monclient(hunting): _renew_subs 2024-09-16T04:10:42.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.911+0000 7fba72db8640 10 monclient(hunting): authenticate will time out at 2689.883520s 2024-09-16T04:10:42.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.911+0000 7fba6bfff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fba6c135490 0x7fba6c1358d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:10:42.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.911+0000 7fba6bfff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fba6c135490 0x7fba6c1358d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:48610/0 (socket says 172.21.15.42:48610) 2024-09-16T04:10:42.933 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.911+0000 7fba71355640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fba6c09cb90 0x7fba6c09d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:10:42.933 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.911+0000 7fba6bfff640 1 -- 172.21.15.42:0/193966224 learned_addr learned my addr 172.21.15.42:0/193966224 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:10:42.933 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.911+0000 7fba6bfff640 10 monclient(hunting): get_auth_request con 0x7fba6c135490 auth_method 0 2024-09-16T04:10:42.933 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.911+0000 7fba6bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:10:42.933 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.911+0000 7fba6bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:42.933 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.911+0000 7fba6bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:42.933 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.911+0000 7fba71355640 10 monclient(hunting): get_auth_request con 0x7fba6c09cb90 auth_method 0 2024-09-16T04:10:42.933 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.911+0000 7fba71355640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:10:42.933 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.911+0000 7fba71355640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:42.933 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.911+0000 7fba71355640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:42.933 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.911+0000 7fba6bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:10:42.933 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.911+0000 7fba6bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:10:42.933 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.911+0000 7fba6bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:10:42.933 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.911+0000 7fba70b54640 1 --2- 172.21.15.42:0/193966224 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fba580039c0 0x7fba6c1cc3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:10:42.934 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.911+0000 7fba71355640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:10:42.934 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.911+0000 7fba71355640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:10:42.934 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.911+0000 7fba71355640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:10:42.934 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.911+0000 7fba70b54640 10 monclient(hunting): get_auth_request con 0x7fba580039c0 auth_method 0 2024-09-16T04:10:42.934 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.911+0000 7fba70b54640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:10:42.934 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.911+0000 7fba70b54640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:42.934 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.911+0000 7fba70b54640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:42.934 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.915+0000 7fba70b54640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:10:42.934 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.915+0000 7fba70b54640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:10:42.934 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.915+0000 7fba70b54640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:10:42.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.915+0000 7fba6bfff640 10 monclient(hunting): handle_auth_done global_id 1014431 payload 824 2024-09-16T04:10:42.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.915+0000 7fba6bfff640 1 -- 172.21.15.42:0/193966224 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fba580039c0 msgr2=0x7fba6c1cc3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:42.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.915+0000 7fba6bfff640 1 --2- 172.21.15.42:0/193966224 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fba580039c0 0x7fba6c1cc3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:10:42.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.915+0000 7fba6bfff640 1 -- 172.21.15.42:0/193966224 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fba6c09cb90 msgr2=0x7fba6c09d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:42.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.915+0000 7fba6bfff640 1 --2- 172.21.15.42:0/193966224 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fba6c09cb90 0x7fba6c09d010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:10:42.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.915+0000 7fba6bfff640 10 monclient: _finish_hunting 0 2024-09-16T04:10:42.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.915+0000 7fba6bfff640 1 monclient: found mon.noname-c 2024-09-16T04:10:42.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.915+0000 7fba6bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:10:42.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.915+0000 7fba6bfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T04:10:42.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.915+0000 7fba6bfff640 1 -- 172.21.15.42:0/193966224 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba6c071250 con 0x7fba6c135490 2024-09-16T04:10:42.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.915+0000 7fba71355640 1 --2- 172.21.15.42:0/193966224 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fba6c09cb90 0x7fba6c09d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:10:42.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.915+0000 7fba6bfff640 1 --2- 172.21.15.42:0/193966224 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fba6c135490 0x7fba6c1358d0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fba5c011660 tx=0x7fba5c011a60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:10:42.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.915+0000 7fba697fa640 10 client.?.objecter ms_handle_connect 0x7fba6c135490 2024-09-16T04:10:42.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.915+0000 7fba697fa640 10 client.?.objecter resend_mon_ops 2024-09-16T04:10:42.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.915+0000 7fba697fa640 1 -- 172.21.15.42:0/193966224 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba5c019070 con 0x7fba6c135490 2024-09-16T04:10:42.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.915+0000 7fba697fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:42.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.915+0000 7fba697fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:10:42.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.915+0000 7fba697fa640 10 monclient: dump: 2024-09-16T04:10:42.936 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:10:42.936 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:10:42.936 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:42.936 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:42.936 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:42.936 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:10:42.936 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:10:42.936 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:10:42.936 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:10:42.937 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:10:42.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.915+0000 7fba697fa640 10 monclient: _finish_auth 0 2024-09-16T04:10:42.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.915+0000 7fba697fa640 10 monclient: _check_auth_tickets 2024-09-16T04:10:42.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.915+0000 7fba697fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:42.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.915+0000 7fba697fa640 1 -- 172.21.15.42:0/193966224 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fba5c004220 con 0x7fba6c135490 2024-09-16T04:10:42.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.915+0000 7fba72db8640 5 monclient: authenticate success, global_id 1014431 2024-09-16T04:10:42.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.915+0000 7fba72db8640 10 monclient: _renew_subs 2024-09-16T04:10:42.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.915+0000 7fba72db8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:10:42.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.915+0000 7fba72db8640 1 -- 172.21.15.42:0/193966224 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba6c06e6e0 con 0x7fba6c135490 2024-09-16T04:10:42.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.915+0000 7fba697fa640 10 monclient: handle_config config(0 keys) 2024-09-16T04:10:42.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.915+0000 7fba697fa640 1 -- 172.21.15.42:0/193966224 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba5c00f040 con 0x7fba6c135490 2024-09-16T04:10:42.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.915+0000 7fba697fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:42.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.915+0000 7fba697fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:10:42.938 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.915+0000 7fba72db8640 10 client.1014431.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:10:42.938 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.915+0000 7fba697fa640 10 monclient: dump: 2024-09-16T04:10:42.938 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:10:42.938 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:10:42.938 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:42.938 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:42.938 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:42.938 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:10:42.938 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:10:42.938 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:10:42.938 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:10:42.938 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:10:42.938 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.915+0000 7fba72db8640 10 monclient: _renew_subs 2024-09-16T04:10:42.938 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.915+0000 7fba72db8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:10:42.938 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.915+0000 7fba72db8640 1 -- 172.21.15.42:0/193966224 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fba6c1288f0 con 0x7fba6c135490 2024-09-16T04:10:42.939 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.915+0000 7fba72db8640 1 librados: init done 2024-09-16T04:10:42.939 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.915+0000 7fba6affd640 4 set_mon_vals no callback set 2024-09-16T04:10:42.939 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.915+0000 7fba697fa640 1 -- 172.21.15.42:0/193966224 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fba5c015750 con 0x7fba6c135490 2024-09-16T04:10:42.939 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.915+0000 7fba72db8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:42.939 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.915+0000 7fba72db8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:42.939 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.915+0000 7fba72db8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:10:42.939 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.915+0000 7fba697fa640 1 --2- 172.21.15.42:0/193966224 >> v2:172.21.15.154:6812/254845342 conn(0x7fba4c037810 0x7fba4c039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:10:42.939 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.915+0000 7fba72db8640 1 -- 172.21.15.42:0/193966224 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fba6c0a1aa0 con 0x7fba6c135490 2024-09-16T04:10:42.939 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.915+0000 7fba697fa640 1 -- 172.21.15.42:0/193966224 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(706..706 src has 451..706) ==== 5040+0+0 (secure 0 0 0) 0x7fba5c048cc0 con 0x7fba6c135490 2024-09-16T04:10:42.939 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.915+0000 7fba697fa640 10 client.1014431.objecter ms_dispatch 0x7fba6c000ba0 osd_map(706..706 src has 451..706) 2024-09-16T04:10:42.939 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.915+0000 7fba697fa640 3 client.1014431.objecter handle_osd_map got epochs [706,706] > 0 2024-09-16T04:10:42.939 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.915+0000 7fba697fa640 3 client.1014431.objecter handle_osd_map decoding full epoch 706 2024-09-16T04:10:42.939 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.915+0000 7fba697fa640 20 client.1014431.objecter dump_active .. 0 homeless 2024-09-16T04:10:42.939 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.915+0000 7fba70b54640 1 --2- 172.21.15.42:0/193966224 >> v2:172.21.15.154:6812/254845342 conn(0x7fba4c037810 0x7fba4c039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:10:42.940 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.915+0000 7fba70b54640 10 monclient: get_auth_request con 0x7fba4c037810 auth_method 0 2024-09-16T04:10:42.940 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.915+0000 7fba70b54640 1 --2- 172.21.15.42:0/193966224 >> v2:172.21.15.154:6812/254845342 conn(0x7fba4c037810 0x7fba4c039cd0 secure :-1 s=READY pgs=3933 cs=0 l=1 rev1=1 crypto rx=0x7fba58004790 tx=0x7fba540011a0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:10:42.940 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.915+0000 7fba697fa640 10 client.1014431.objecter ms_handle_connect 0x7fba4c037810 2024-09-16T04:10:42.940 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.919+0000 7fba697fa640 1 -- 172.21.15.42:0/193966224 <== mon.2 v2:172.21.15.42:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7fba5c0147e0 con 0x7fba6c135490 2024-09-16T04:10:42.940 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.919+0000 7fba697fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:42.940 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:42.919+0000 7fba697fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:10:43.148 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.143+0000 7fba72db8640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:10:43.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.143+0000 7fba72db8640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:10:43.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.143+0000 7fba72db8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:10:43.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.143+0000 7fba72db8640 1 -- 172.21.15.42:0/193966224 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fba6c1274a0 con 0x7fba6c135490 2024-09-16T04:10:43.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.143+0000 7fba697fa640 1 -- 172.21.15.42:0/193966224 <== mon.2 v2:172.21.15.42:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7fba5c014a40 con 0x7fba6c135490 2024-09-16T04:10:43.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.147+0000 7fba697fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:10:43.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.147+0000 7fba697fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:10:43.150 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":262,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:10:43.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.147+0000 7fba72db8640 10 librados: watch_flush enter 2024-09-16T04:10:43.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.147+0000 7fba72db8640 10 librados: watch_flush exit 2024-09-16T04:10:43.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.147+0000 7fba72db8640 20 client.1014431.objecter shutdown clearing up homeless session... 2024-09-16T04:10:43.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.147+0000 7fba72db8640 10 client.1014431.objecter successfully canceled tick 2024-09-16T04:10:43.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.147+0000 7fba72db8640 1 -- 172.21.15.42:0/193966224 >> v2:172.21.15.154:6812/254845342 conn(0x7fba4c037810 msgr2=0x7fba4c039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:43.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.147+0000 7fba72db8640 1 --2- 172.21.15.42:0/193966224 >> v2:172.21.15.154:6812/254845342 conn(0x7fba4c037810 0x7fba4c039cd0 secure :-1 s=READY pgs=3933 cs=0 l=1 rev1=1 crypto rx=0x7fba58004790 tx=0x7fba540011a0 comp rx=0 tx=0).stop 2024-09-16T04:10:43.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.147+0000 7fba72db8640 10 monclient: shutdown 2024-09-16T04:10:43.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.147+0000 7fba72db8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:10:43.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.147+0000 7fba72db8640 1 -- 172.21.15.42:0/193966224 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fba6c135490 msgr2=0x7fba6c1358d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:43.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.147+0000 7fba72db8640 1 --2- 172.21.15.42:0/193966224 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fba6c135490 0x7fba6c1358d0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fba5c011660 tx=0x7fba5c011a60 comp rx=0 tx=0).stop 2024-09-16T04:10:43.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.147+0000 7fba72db8640 1 -- 172.21.15.42:0/193966224 shutdown_connections 2024-09-16T04:10:43.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.147+0000 7fba72db8640 1 --2- 172.21.15.42:0/193966224 >> v2:172.21.15.154:6812/254845342 conn(0x7fba4c037810 0x7fba4c039cd0 unknown :-1 s=CLOSED pgs=3933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:10:43.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.147+0000 7fba72db8640 1 --2- 172.21.15.42:0/193966224 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fba6c09cb90 0x7fba6c09d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:10:43.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.147+0000 7fba72db8640 1 --2- 172.21.15.42:0/193966224 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fba6c135490 0x7fba6c1358d0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:10:43.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.147+0000 7fba72db8640 1 --2- 172.21.15.42:0/193966224 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fba580039c0 0x7fba6c1cc3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:10:43.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.147+0000 7fba72db8640 1 -- 172.21.15.42:0/193966224 >> 172.21.15.42:0/193966224 conn(0x7fba6c0a2420 msgr2=0x7fba6c06f7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:10:43.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.147+0000 7fba72db8640 1 -- 172.21.15.42:0/193966224 shutdown_connections 2024-09-16T04:10:43.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.147+0000 7fba72db8640 1 -- 172.21.15.42:0/193966224 wait complete. 2024-09-16T04:10:43.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.147+0000 7fba72db8640 1 librados: shutdown 2024-09-16T04:10:43.168 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T04:10:43.168 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T04:10:43.168 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T04:10:43.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.283+0000 7f495894a640 10 monclient: get_monmap_and_config 2024-09-16T04:10:43.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.283+0000 7f495894a640 10 monclient: build_initial_monmap 2024-09-16T04:10:43.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.283+0000 7f495894a640 10 monclient: monmap: 2024-09-16T04:10:43.286 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:10:43.286 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:10:43.286 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:10:43.284503+0000 2024-09-16T04:10:43.286 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:10:43.284503+0000 2024-09-16T04:10:43.286 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:10:43.286 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:10:43.286 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:10:43.286 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:10:43.287 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:10:43.287 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:10:43.287 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:10:43.287 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:10:43.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.283+0000 7f495894a640 1 Processor -- start 2024-09-16T04:10:43.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.283+0000 7f495894a640 1 -- start start 2024-09-16T04:10:43.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.283+0000 7f495894a640 10 monclient: init 2024-09-16T04:10:43.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.283+0000 7f495894a640 10 monclient: _reopen_session rank -1 2024-09-16T04:10:43.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.283+0000 7f495894a640 10 monclient: _add_conns ranks=[4,3,2,1,0] 2024-09-16T04:10:43.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.283+0000 7f495894a640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f49540a1330 0x7f49540a1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:10:43.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.283+0000 7f495894a640 10 monclient(hunting): picked mon.noname-c con 0x7f49540a1330 addr v2:172.21.15.154:3300/0 2024-09-16T04:10:43.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.283+0000 7f495894a640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f495406dac0 0x7f49540a1c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:10:43.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.283+0000 7f495894a640 10 monclient(hunting): picked mon.noname-a con 0x7f495406dac0 addr v2:172.21.15.42:3300/0 2024-09-16T04:10:43.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.283+0000 7f495894a640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4954135100 0x7f4954135550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:10:43.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.283+0000 7f495894a640 10 monclient(hunting): picked mon.noname-b con 0x7f4954135100 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:10:43.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.283+0000 7f495894a640 10 monclient(hunting): start opening mon connection 2024-09-16T04:10:43.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.283+0000 7f495894a640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f495409c7d0 con 0x7f495406dac0 2024-09-16T04:10:43.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.283+0000 7f495894a640 10 monclient(hunting): start opening mon connection 2024-09-16T04:10:43.291 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.283+0000 7f495894a640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f495409c090 con 0x7f49540a1330 2024-09-16T04:10:43.291 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.283+0000 7f495894a640 10 monclient(hunting): start opening mon connection 2024-09-16T04:10:43.291 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.283+0000 7f495894a640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f495409f610 con 0x7f4954135100 2024-09-16T04:10:43.291 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.283+0000 7f495894a640 10 monclient(hunting): _renew_subs 2024-09-16T04:10:43.291 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.283+0000 7f495894a640 10 monclient(hunting): authenticate will time out at 2690.254098s 2024-09-16T04:10:43.291 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.283+0000 7f4952d9d640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4954135100 0x7f4954135550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:10:43.291 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.283+0000 7f4951d9b640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f495406dac0 0x7f49540a1c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:10:43.291 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.283+0000 7f4952d9d640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4954135100 0x7f4954135550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:48622/0 (socket says 172.21.15.42:48622) 2024-09-16T04:10:43.291 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.283+0000 7f4952d9d640 1 -- 172.21.15.42:0/2466439213 learned_addr learned my addr 172.21.15.42:0/2466439213 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:10:43.291 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.283+0000 7f4952d9d640 10 monclient(hunting): get_auth_request con 0x7f4954135100 auth_method 0 2024-09-16T04:10:43.291 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.283+0000 7f4952d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:10:43.291 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.283+0000 7f4952d9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:43.291 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.283+0000 7f4952d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:43.292 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.283+0000 7f4951d9b640 10 monclient(hunting): get_auth_request con 0x7f495406dac0 auth_method 0 2024-09-16T04:10:43.292 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.283+0000 7f4951d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:10:43.292 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.283+0000 7f4951d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:43.292 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.283+0000 7f4951d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:43.292 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.283+0000 7f4952d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:10:43.292 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.283+0000 7f4952d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:10:43.292 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.283+0000 7f4952d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:10:43.292 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.283+0000 7f4951d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:10:43.292 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.283+0000 7f4951d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:10:43.292 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.283+0000 7f495259c640 1 --2- 172.21.15.42:0/2466439213 >> v2:172.21.15.154:3300/0 conn(0x7f49540a1330 0x7f49540a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:10:43.293 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.283+0000 7f4951d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:10:43.293 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.283+0000 7f495259c640 10 monclient(hunting): get_auth_request con 0x7f49540a1330 auth_method 0 2024-09-16T04:10:43.293 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.283+0000 7f495259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:10:43.293 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.283+0000 7f495259c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:43.293 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.283+0000 7f495259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:43.293 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.283+0000 7f4952d9d640 10 monclient(hunting): handle_auth_done global_id 1014434 payload 274 2024-09-16T04:10:43.293 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.283+0000 7f4952d9d640 1 -- 172.21.15.42:0/2466439213 >> v2:172.21.15.154:3300/0 conn(0x7f49540a1330 msgr2=0x7f49540a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:10:43.293 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.283+0000 7f4952d9d640 1 --2- 172.21.15.42:0/2466439213 >> v2:172.21.15.154:3300/0 conn(0x7f49540a1330 0x7f49540a1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:10:43.293 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.283+0000 7f4952d9d640 1 -- 172.21.15.42:0/2466439213 >> v2:172.21.15.42:3300/0 conn(0x7f495406dac0 msgr2=0x7f49540a1c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:10:43.293 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.283+0000 7f4952d9d640 1 --2- 172.21.15.42:0/2466439213 >> v2:172.21.15.42:3300/0 conn(0x7f495406dac0 0x7f49540a1c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:10:43.294 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.283+0000 7f4952d9d640 10 monclient: _finish_hunting 0 2024-09-16T04:10:43.294 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.283+0000 7f4952d9d640 1 monclient: found mon.noname-b 2024-09-16T04:10:43.294 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.283+0000 7f4952d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:10:43.294 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.283+0000 7f4952d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T04:10:43.294 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.283+0000 7f4952d9d640 1 -- 172.21.15.42:0/2466439213 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f495409f8f0 con 0x7f4954135100 2024-09-16T04:10:43.294 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.283+0000 7f4951d9b640 1 --2- 172.21.15.42:0/2466439213 >> v2:172.21.15.42:3300/0 conn(0x7f495406dac0 0x7f49540a1c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:10:43.300 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.287+0000 7f4952d9d640 1 --2- 172.21.15.42:0/2466439213 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4954135100 0x7f4954135550 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f495406add0 tx=0x7f494c01c440 comp rx=0 tx=0).ready entity=mon.2 client_cookie=764b4c0f39b1ac91 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:10:43.300 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.287+0000 7f495159a640 1 -- 172.21.15.42:0/2466439213 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f494c0b1070 con 0x7f4954135100 2024-09-16T04:10:43.300 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.287+0000 7f495159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:43.300 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.287+0000 7f495159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:10:43.300 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.287+0000 7f495159a640 10 monclient: dump: 2024-09-16T04:10:43.300 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:10:43.300 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:10:43.300 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:43.300 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:43.300 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:43.300 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:10:43.301 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:10:43.301 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:10:43.301 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:10:43.301 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:10:43.301 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.287+0000 7f495159a640 10 monclient: _finish_auth 0 2024-09-16T04:10:43.301 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.287+0000 7f495159a640 10 monclient: _check_auth_tickets 2024-09-16T04:10:43.301 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.287+0000 7f495159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:43.301 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.287+0000 7f495159a640 1 -- 172.21.15.42:0/2466439213 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f494c0ac620 con 0x7f4954135100 2024-09-16T04:10:43.301 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.287+0000 7f495894a640 5 monclient: authenticate success, global_id 1014434 2024-09-16T04:10:43.301 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.287+0000 7f495894a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:10:43.301 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.287+0000 7f495159a640 10 monclient: handle_config config(0 keys) 2024-09-16T04:10:43.301 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.287+0000 7f495894a640 10 monclient: get_monmap_and_config success 2024-09-16T04:10:43.301 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.287+0000 7f495894a640 4 set_mon_vals no callback set 2024-09-16T04:10:43.301 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.287+0000 7f495159a640 1 -- 172.21.15.42:0/2466439213 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f494c0ac9f0 con 0x7f4954135100 2024-09-16T04:10:43.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.287+0000 7f495159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:43.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.287+0000 7f495159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:10:43.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.287+0000 7f495159a640 10 monclient: dump: 2024-09-16T04:10:43.302 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:10:43.302 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:10:43.302 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:43.302 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:43.302 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:43.302 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:10:43.302 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:10:43.302 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:10:43.302 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:10:43.302 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:10:43.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.287+0000 7f495894a640 10 monclient: shutdown 2024-09-16T04:10:43.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.287+0000 7f495894a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:10:43.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.287+0000 7f495894a640 1 -- 172.21.15.42:0/2466439213 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4954135100 msgr2=0x7f4954135550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:43.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.287+0000 7f495894a640 1 --2- 172.21.15.42:0/2466439213 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4954135100 0x7f4954135550 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f495406add0 tx=0x7f494c01c440 comp rx=0 tx=0).stop 2024-09-16T04:10:43.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.287+0000 7f495894a640 1 -- 172.21.15.42:0/2466439213 shutdown_connections 2024-09-16T04:10:43.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.287+0000 7f495894a640 1 --2- 172.21.15.42:0/2466439213 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4954135100 0x7f4954135550 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:10:43.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.287+0000 7f495894a640 1 --2- 172.21.15.42:0/2466439213 >> v2:172.21.15.42:3300/0 conn(0x7f495406dac0 0x7f49540a1c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:10:43.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.287+0000 7f495894a640 1 --2- 172.21.15.42:0/2466439213 >> v2:172.21.15.154:3300/0 conn(0x7f49540a1330 0x7f49540a1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:10:43.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.287+0000 7f495894a640 1 -- 172.21.15.42:0/2466439213 >> 172.21.15.42:0/2466439213 conn(0x7f49540a2420 msgr2=0x7f49540a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:10:43.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.287+0000 7f495894a640 1 -- 172.21.15.42:0/2466439213 shutdown_connections 2024-09-16T04:10:43.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.287+0000 7f495894a640 1 -- 172.21.15.42:0/2466439213 wait complete. 2024-09-16T04:10:43.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.287+0000 7f495894a640 10 monclient: build_initial_monmap 2024-09-16T04:10:43.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.287+0000 7f495894a640 10 monclient: monmap: 2024-09-16T04:10:43.303 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:10:43.304 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:10:43.304 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:10:43.304 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:10:43.304 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:10:43.304 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:10:43.304 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:10:43.304 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:10:43.304 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:10:43.304 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:10:43.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.287+0000 7f495894a640 1 librados: starting msgr at 2024-09-16T04:10:43.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.287+0000 7f495894a640 1 librados: starting objecter 2024-09-16T04:10:43.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.287+0000 7f495894a640 1 Processor -- start 2024-09-16T04:10:43.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.287+0000 7f495894a640 1 -- start start 2024-09-16T04:10:43.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.287+0000 7f495894a640 1 librados: setting wanted keys 2024-09-16T04:10:43.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.287+0000 7f495894a640 1 librados: calling monclient init 2024-09-16T04:10:43.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.287+0000 7f495894a640 10 monclient: init 2024-09-16T04:10:43.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.287+0000 7f495894a640 10 monclient: _reopen_session rank -1 2024-09-16T04:10:43.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.287+0000 7f495894a640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:10:43.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.287+0000 7f495894a640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f49540a1330 0x7f49541cf0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:10:43.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.287+0000 7f495894a640 10 monclient(hunting): picked mon.noname-b con 0x7f49540a1330 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:10:43.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.287+0000 7f495894a640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4954135100 0x7f49541346f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:10:43.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.287+0000 7f495894a640 10 monclient(hunting): picked mon.noname-c con 0x7f4954135100 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:10:43.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.287+0000 7f495894a640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4954134c30 0x7f495409cb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:10:43.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.287+0000 7f495894a640 10 monclient(hunting): picked mon.noname-a con 0x7f4954134c30 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:10:43.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.287+0000 7f495894a640 10 monclient(hunting): start opening mon connection 2024-09-16T04:10:43.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.287+0000 7f495894a640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f495409eef0 con 0x7f4954134c30 2024-09-16T04:10:43.306 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.287+0000 7f495894a640 10 monclient(hunting): start opening mon connection 2024-09-16T04:10:43.306 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.287+0000 7f495894a640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f495409ed70 con 0x7f49540a1330 2024-09-16T04:10:43.306 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.287+0000 7f495894a640 10 monclient(hunting): start opening mon connection 2024-09-16T04:10:43.306 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.287+0000 7f495894a640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f495413eb70 con 0x7f4954135100 2024-09-16T04:10:43.306 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.287+0000 7f495894a640 10 monclient(hunting): _renew_subs 2024-09-16T04:10:43.306 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.287+0000 7f495894a640 10 monclient(hunting): authenticate will time out at 2690.259159s 2024-09-16T04:10:43.306 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.287+0000 7f4951d9b640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4954135100 0x7f49541346f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:10:43.306 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.287+0000 7f4952d9d640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4954134c30 0x7f495409cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:10:43.306 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.287+0000 7f4951d9b640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4954135100 0x7f49541346f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:48626/0 (socket says 172.21.15.42:48626) 2024-09-16T04:10:43.306 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.287+0000 7f4951d9b640 1 -- 172.21.15.42:0/1459638735 learned_addr learned my addr 172.21.15.42:0/1459638735 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:10:43.306 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.287+0000 7f4951d9b640 10 monclient(hunting): get_auth_request con 0x7f4954135100 auth_method 0 2024-09-16T04:10:43.306 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.287+0000 7f4951d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:10:43.306 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.287+0000 7f4951d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:43.307 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.287+0000 7f4951d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:43.307 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.287+0000 7f4952d9d640 10 monclient(hunting): get_auth_request con 0x7f4954134c30 auth_method 0 2024-09-16T04:10:43.307 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.287+0000 7f4952d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:10:43.307 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.287+0000 7f4952d9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:43.307 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.287+0000 7f4952d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:43.307 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.287+0000 7f4951d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:10:43.307 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.287+0000 7f4951d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:10:43.307 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.287+0000 7f4951d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:10:43.307 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.287+0000 7f4952d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:10:43.307 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.287+0000 7f4952d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:10:43.307 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.287+0000 7f4952d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:10:43.307 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.287+0000 7f4951d9b640 10 monclient(hunting): handle_auth_done global_id 1014437 payload 824 2024-09-16T04:10:43.307 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.287+0000 7f495259c640 1 --2- 172.21.15.42:0/1459638735 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f49540a1330 0x7f49541cf0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:10:43.307 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.287+0000 7f4951d9b640 1 -- 172.21.15.42:0/1459638735 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f49540a1330 msgr2=0x7f49541cf0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:43.308 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.287+0000 7f4951d9b640 1 --2- 172.21.15.42:0/1459638735 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f49540a1330 0x7f49541cf0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:10:43.308 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.287+0000 7f4951d9b640 1 -- 172.21.15.42:0/1459638735 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4954134c30 msgr2=0x7f495409cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:43.308 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.287+0000 7f4951d9b640 1 --2- 172.21.15.42:0/1459638735 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4954134c30 0x7f495409cb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:10:43.308 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.287+0000 7f4951d9b640 10 monclient: _finish_hunting 0 2024-09-16T04:10:43.308 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.287+0000 7f4951d9b640 1 monclient: found mon.noname-c 2024-09-16T04:10:43.308 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.287+0000 7f4951d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:10:43.308 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.287+0000 7f4951d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T04:10:43.308 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.287+0000 7f4951d9b640 1 -- 172.21.15.42:0/1459638735 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49541d0000 con 0x7f4954135100 2024-09-16T04:10:43.308 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.287+0000 7f4952d9d640 1 --2- 172.21.15.42:0/1459638735 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4954134c30 0x7f495409cb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:10:43.308 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.287+0000 7f4951d9b640 1 --2- 172.21.15.42:0/1459638735 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4954135100 0x7f49541346f0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f494400efc0 tx=0x7f494400c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:10:43.308 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.287+0000 7f494affd640 10 client.?.objecter ms_handle_connect 0x7f4954135100 2024-09-16T04:10:43.308 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.287+0000 7f494affd640 10 client.?.objecter resend_mon_ops 2024-09-16T04:10:43.308 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.287+0000 7f494affd640 1 -- 172.21.15.42:0/1459638735 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4944018020 con 0x7f4954135100 2024-09-16T04:10:43.309 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.291+0000 7f494affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:43.309 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.291+0000 7f494affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:10:43.309 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.291+0000 7f494affd640 10 monclient: dump: 2024-09-16T04:10:43.309 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:10:43.309 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:10:43.309 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:43.309 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:43.309 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:43.309 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:10:43.309 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:10:43.309 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:10:43.309 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:10:43.309 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:10:43.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.291+0000 7f494affd640 10 monclient: _finish_auth 0 2024-09-16T04:10:43.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.291+0000 7f494affd640 10 monclient: _check_auth_tickets 2024-09-16T04:10:43.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.291+0000 7f494affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:43.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.291+0000 7f494affd640 1 -- 172.21.15.42:0/1459638735 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4944014680 con 0x7f4954135100 2024-09-16T04:10:43.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.291+0000 7f494affd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:10:43.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.291+0000 7f494affd640 1 -- 172.21.15.42:0/1459638735 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4944014a50 con 0x7f4954135100 2024-09-16T04:10:43.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.291+0000 7f494affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:43.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.291+0000 7f494affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:10:43.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.291+0000 7f494affd640 10 monclient: dump: 2024-09-16T04:10:43.310 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:10:43.310 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:10:43.310 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:43.310 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:43.311 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:43.311 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:10:43.311 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:10:43.311 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:10:43.311 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:10:43.311 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:10:43.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.291+0000 7f495894a640 5 monclient: authenticate success, global_id 1014437 2024-09-16T04:10:43.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.291+0000 7f495894a640 10 monclient: _renew_subs 2024-09-16T04:10:43.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.291+0000 7f4950d99640 4 set_mon_vals no callback set 2024-09-16T04:10:43.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.291+0000 7f495894a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:10:43.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.291+0000 7f495894a640 1 -- 172.21.15.42:0/1459638735 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4954073a30 con 0x7f4954135100 2024-09-16T04:10:43.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.291+0000 7f495894a640 10 client.1014437.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:10:43.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.291+0000 7f495894a640 10 monclient: _renew_subs 2024-09-16T04:10:43.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.291+0000 7f495894a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:10:43.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.291+0000 7f495894a640 1 -- 172.21.15.42:0/1459638735 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f4954121dd0 con 0x7f4954135100 2024-09-16T04:10:43.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.291+0000 7f495894a640 1 librados: init done 2024-09-16T04:10:43.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.291+0000 7f495894a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:43.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.291+0000 7f495894a640 1 --2- 172.21.15.42:0/1459638735 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f495406dac0 0x7f495409d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:10:43.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.291+0000 7f494affd640 1 -- 172.21.15.42:0/1459638735 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f494401b020 con 0x7f4954135100 2024-09-16T04:10:43.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.291+0000 7f495894a640 10 monclient(hunting): start opening mon connection 2024-09-16T04:10:43.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.291+0000 7f495894a640 1 -- 172.21.15.42:0/1459638735 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f4954135550 con 0x7f495406dac0 2024-09-16T04:10:43.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.291+0000 7f495259c640 1 --2- 172.21.15.42:0/1459638735 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f495406dac0 0x7f495409d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:10:43.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.291+0000 7f494affd640 1 --2- 172.21.15.42:0/1459638735 >> v2:172.21.15.154:6812/254845342 conn(0x7f4934037810 0x7f4934039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:10:43.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.291+0000 7f495259c640 10 monclient: get_auth_request con 0x7f495406dac0 auth_method 0 2024-09-16T04:10:43.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.291+0000 7f495259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:10:43.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.291+0000 7f495259c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:43.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.291+0000 7f495259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:43.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.291+0000 7f494affd640 1 -- 172.21.15.42:0/1459638735 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(706..706 src has 451..706) ==== 5040+0+0 (secure 0 0 0) 0x7f4944049b60 con 0x7f4954135100 2024-09-16T04:10:43.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.291+0000 7f494affd640 10 client.1014437.objecter ms_dispatch 0x7f4954000ba0 osd_map(706..706 src has 451..706) 2024-09-16T04:10:43.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.291+0000 7f494affd640 3 client.1014437.objecter handle_osd_map got epochs [706,706] > 0 2024-09-16T04:10:43.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.291+0000 7f494affd640 3 client.1014437.objecter handle_osd_map decoding full epoch 706 2024-09-16T04:10:43.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.291+0000 7f495259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:10:43.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.291+0000 7f495259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:10:43.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.291+0000 7f495259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:10:43.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.291+0000 7f494affd640 20 client.1014437.objecter dump_active .. 0 homeless 2024-09-16T04:10:43.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.291+0000 7f495259c640 10 monclient(hunting): handle_auth_done global_id 1034316 payload 824 2024-09-16T04:10:43.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.291+0000 7f495259c640 1 -- 172.21.15.42:0/1459638735 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f49540a4c60 con 0x7f495406dac0 2024-09-16T04:10:43.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.291+0000 7f4952d9d640 1 --2- 172.21.15.42:0/1459638735 >> v2:172.21.15.154:6812/254845342 conn(0x7f4934037810 0x7f4934039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:10:43.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.291+0000 7f4952d9d640 10 monclient: get_auth_request con 0x7f4934037810 auth_method 0 2024-09-16T04:10:43.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.291+0000 7f495259c640 1 --2- 172.21.15.42:0/1459638735 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f495406dac0 0x7f495409d0a0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f493c00e510 tx=0x7f493c00e9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:10:43.314 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.291+0000 7f494affd640 10 client.1014437.objecter ms_handle_connect 0x7f495406dac0 2024-09-16T04:10:43.314 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.291+0000 7f494affd640 10 client.1014437.objecter resend_mon_ops 2024-09-16T04:10:43.314 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.291+0000 7f494affd640 1 -- 172.21.15.42:0/1459638735 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f493c017070 con 0x7f495406dac0 2024-09-16T04:10:43.314 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.291+0000 7f494affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:43.314 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.291+0000 7f494affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:10:43.314 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.291+0000 7f494affd640 10 monclient: dump: 2024-09-16T04:10:43.314 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:10:43.314 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:10:43.314 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:43.314 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:43.314 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:43.314 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:10:43.314 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:10:43.314 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:10:43.315 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:10:43.315 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:10:43.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.291+0000 7f494affd640 1 -- 172.21.15.42:0/1459638735 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f493c013610 con 0x7f495406dac0 2024-09-16T04:10:43.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.291+0000 7f494affd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:43.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.291+0000 7f494affd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:10:43.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.291+0000 7f494affd640 1 -- 172.21.15.42:0/1459638735 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f495406dac0 msgr2=0x7f495409d0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:43.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.291+0000 7f494affd640 1 --2- 172.21.15.42:0/1459638735 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f495406dac0 0x7f495409d0a0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f493c00e510 tx=0x7f493c00e9d0 comp rx=0 tx=0).stop 2024-09-16T04:10:43.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.291+0000 7f494affd640 1 -- 172.21.15.42:0/1459638735 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f495406dac0 msgr2=0x7f495409d0a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:10:43.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.291+0000 7f494affd640 1 --2- 172.21.15.42:0/1459638735 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f495406dac0 0x7f495409d0a0 secure :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f493c00e510 tx=0x7f493c00e9d0 comp rx=0 tx=0).stop 2024-09-16T04:10:43.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.291+0000 7f4952d9d640 1 --2- 172.21.15.42:0/1459638735 >> v2:172.21.15.154:6812/254845342 conn(0x7f4934037810 0x7f4934039cd0 secure :-1 s=READY pgs=3934 cs=0 l=1 rev1=1 crypto rx=0x7f494c0ae000 tx=0x7f494c0a7000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:10:43.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.291+0000 7f494affd640 10 client.1014437.objecter ms_handle_connect 0x7f4934037810 2024-09-16T04:10:43.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.299+0000 7f495894a640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:10:43.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.299+0000 7f495894a640 1 --2- 172.21.15.42:0/1459638735 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f495409d5e0 0x7f4954138330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:10:43.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.299+0000 7f495894a640 10 monclient(hunting): start opening mon connection 2024-09-16T04:10:43.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.299+0000 7f495894a640 1 -- 172.21.15.42:0/1459638735 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f495409d9f0 con 0x7f495409d5e0 2024-09-16T04:10:43.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.299+0000 7f495259c640 1 --2- 172.21.15.42:0/1459638735 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f495409d5e0 0x7f4954138330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:10:43.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.299+0000 7f495259c640 10 monclient: get_auth_request con 0x7f495409d5e0 auth_method 0 2024-09-16T04:10:43.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.299+0000 7f495259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:10:43.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.299+0000 7f495259c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:43.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.299+0000 7f495259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:43.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.299+0000 7f495259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:10:43.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.299+0000 7f495259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:10:43.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.299+0000 7f495259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:10:43.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.299+0000 7f495259c640 10 monclient(hunting): handle_auth_done global_id 1034319 payload 824 2024-09-16T04:10:43.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.299+0000 7f495259c640 1 -- 172.21.15.42:0/1459638735 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f4954069040 con 0x7f495409d5e0 2024-09-16T04:10:43.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.299+0000 7f495259c640 1 --2- 172.21.15.42:0/1459638735 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f495409d5e0 0x7f4954138330 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f493c008010 tx=0x7f493c004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:10:43.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.299+0000 7f494affd640 10 client.1014437.objecter ms_handle_connect 0x7f495409d5e0 2024-09-16T04:10:43.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.299+0000 7f494affd640 10 client.1014437.objecter resend_mon_ops 2024-09-16T04:10:43.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.299+0000 7f494affd640 1 -- 172.21.15.42:0/1459638735 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f493c00f040 con 0x7f495409d5e0 2024-09-16T04:10:43.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.299+0000 7f494affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:43.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.299+0000 7f494affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:10:43.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.299+0000 7f494affd640 10 monclient: dump: 2024-09-16T04:10:43.319 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:10:43.319 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:10:43.319 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:43.319 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:43.319 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:43.319 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:10:43.319 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:10:43.319 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:10:43.319 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:10:43.319 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:10:43.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.299+0000 7f494affd640 1 -- 172.21.15.42:0/1459638735 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4882 (secure 0 0 0) 0x7f493c026390 con 0x7f495409d5e0 2024-09-16T04:10:43.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.299+0000 7f494affd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:10:43.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.299+0000 7f494affd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:10:43.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.299+0000 7f494affd640 1 -- 172.21.15.42:0/1459638735 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f495409d5e0 msgr2=0x7f4954138330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:43.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.299+0000 7f494affd640 1 --2- 172.21.15.42:0/1459638735 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f495409d5e0 0x7f4954138330 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f493c008010 tx=0x7f493c004740 comp rx=0 tx=0).stop 2024-09-16T04:10:43.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.299+0000 7f494affd640 1 -- 172.21.15.42:0/1459638735 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f495409d5e0 msgr2=0x7f4954138330 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:10:43.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.299+0000 7f494affd640 1 --2- 172.21.15.42:0/1459638735 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f495409d5e0 0x7f4954138330 secure :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f493c008010 tx=0x7f493c004740 comp rx=0 tx=0).stop 2024-09-16T04:10:43.320 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T04:10:43.320 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:10:43.320 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:10:43.320 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "leader", 2024-09-16T04:10:43.321 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 262, 2024-09-16T04:10:43.321 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 123234, 2024-09-16T04:10:43.321 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T04:10:43.321 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T04:10:43.321 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T04:10:43.321 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T04:10:43.321 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:10:43.321 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 0, 2024-09-16T04:10:43.321 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:10:43.321 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T04:10:43.321 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T04:10:43.321 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:10:43.321 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:10:43.322 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:10:43.322 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:10:43.322 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:10:43.322 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:10:43.322 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:10:43.322 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:10:43.322 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:10:43.322 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:10:43.322 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:10:43.322 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:10:43.322 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:10:43.322 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T04:10:43.322 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:10:43.323 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:10:43.323 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:10:43.323 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:10:43.323 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:10:43.323 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:10:43.323 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:10:43.323 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:10:43.323 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:10:43.323 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:10:43.323 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:10:43.323 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:10:43.323 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:10:43.323 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T04:10:43.323 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T04:10:43.324 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T04:10:43.324 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T04:10:43.324 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T04:10:43.324 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T04:10:43.324 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:10:43.324 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:10:43.324 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T04:10:43.324 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T04:10:43.324 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T04:10:43.324 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T04:10:43.324 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T04:10:43.324 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T04:10:43.324 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T04:10:43.324 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:10:43.325 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T04:10:43.325 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:10:43.325 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:10:43.325 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:10:43.325 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:10:43.325 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:10:43.325 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:10:43.325 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:10:43.325 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:10:43.325 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:10:43.325 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:10:43.325 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:10:43.325 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:10:43.325 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T04:10:43.326 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:10:43.326 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T04:10:43.326 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:10:43.326 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:10:43.326 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:10:43.326 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:10:43.326 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:10:43.326 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:10:43.326 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:10:43.326 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T04:10:43.326 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:10:43.326 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:10:43.326 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:10:43.326 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:10:43.327 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T04:10:43.327 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:10:43.327 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:10:43.327 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:10:43.327 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:10:43.327 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T04:10:43.327 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T04:10:43.327 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:10:43.327 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:10:43.327 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:10:43.327 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:10:43.327 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:10:43.327 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:10:43.327 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:10:43.328 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:10:43.328 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:10:43.328 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:10:43.328 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:10:43.328 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T04:10:43.328 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:10:43.328 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:10:43.328 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:10:43.328 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:10:43.328 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T04:10:43.328 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:10:43.328 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:10:43.328 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:10:43.329 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:10:43.329 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T04:10:43.329 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T04:10:43.329 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:10:43.329 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:10:43.329 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:10:43.329 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:10:43.329 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:10:43.329 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:10:43.329 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:10:43.329 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:10:43.329 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:10:43.329 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:10:43.329 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:10:43.330 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T04:10:43.330 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:10:43.330 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:10:43.330 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:10:43.330 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:10:43.330 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T04:10:43.330 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:10:43.330 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:10:43.330 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:10:43.330 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:10:43.330 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T04:10:43.330 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T04:10:43.330 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:10:43.330 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:10:43.331 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:10:43.331 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:10:43.331 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:10:43.331 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:10:43.331 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T04:10:43.331 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T04:10:43.331 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:10:43.331 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:10:43.331 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:10:43.331 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:10:43.331 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:10:43.331 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:10:43.331 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T04:10:43.331 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:10:43.332 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:10:43.332 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:10:43.332 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 2 2024-09-16T04:10:43.332 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:10:43.332 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:10:43.332 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T04:10:43.332 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:10:43.332 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:10:43.332 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:10:43.332 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 4 2024-09-16T04:10:43.332 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:10:43.332 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:10:43.332 INFO:teuthology.orchestra.run.smithi042.stdout: "mgr": [ 2024-09-16T04:10:43.332 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:10:43.333 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:10:43.333 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:10:43.333 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:10:43.333 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:10:43.333 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:10:43.333 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:10:43.333 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T04:10:43.333 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T04:10:43.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.299+0000 7f495894a640 10 librados: watch_flush enter 2024-09-16T04:10:43.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.299+0000 7f495894a640 10 librados: watch_flush exit 2024-09-16T04:10:43.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.299+0000 7f495894a640 20 client.1014437.objecter shutdown clearing up homeless session... 2024-09-16T04:10:43.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.299+0000 7f495894a640 10 client.1014437.objecter successfully canceled tick 2024-09-16T04:10:43.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.299+0000 7f495894a640 1 -- 172.21.15.42:0/1459638735 >> v2:172.21.15.154:6812/254845342 conn(0x7f4934037810 msgr2=0x7f4934039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:43.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.299+0000 7f495894a640 1 --2- 172.21.15.42:0/1459638735 >> v2:172.21.15.154:6812/254845342 conn(0x7f4934037810 0x7f4934039cd0 secure :-1 s=READY pgs=3934 cs=0 l=1 rev1=1 crypto rx=0x7f494c0ae000 tx=0x7f494c0a7000 comp rx=0 tx=0).stop 2024-09-16T04:10:43.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.299+0000 7f495894a640 10 monclient: shutdown 2024-09-16T04:10:43.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.299+0000 7f495894a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:10:43.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.299+0000 7f495894a640 1 -- 172.21.15.42:0/1459638735 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4954135100 msgr2=0x7f49541346f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:43.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.299+0000 7f495894a640 1 --2- 172.21.15.42:0/1459638735 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4954135100 0x7f49541346f0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f494400efc0 tx=0x7f494400c490 comp rx=0 tx=0).stop 2024-09-16T04:10:43.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.299+0000 7f4952d9d640 1 -- 172.21.15.42:0/1459638735 reap_dead start 2024-09-16T04:10:43.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.299+0000 7f4952d9d640 1 -- 172.21.15.42:0/1459638735 reap_dead start 2024-09-16T04:10:43.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.299+0000 7f495894a640 1 -- 172.21.15.42:0/1459638735 shutdown_connections 2024-09-16T04:10:43.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.299+0000 7f495894a640 1 -- 172.21.15.42:0/1459638735 >> 172.21.15.42:0/1459638735 conn(0x7f49540a2420 msgr2=0x7f49540a3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:10:43.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.299+0000 7f495894a640 1 -- 172.21.15.42:0/1459638735 shutdown_connections 2024-09-16T04:10:43.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.299+0000 7f495894a640 1 -- 172.21.15.42:0/1459638735 wait complete. 2024-09-16T04:10:43.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.299+0000 7f495894a640 1 librados: shutdown 2024-09-16T04:10:43.336 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:10:43.336 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T04:10:43.337 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T04:10:43.455 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.451+0000 7fbd3f27b640 10 monclient: get_monmap_and_config 2024-09-16T04:10:43.455 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.451+0000 7fbd3f27b640 10 monclient: build_initial_monmap 2024-09-16T04:10:43.456 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.451+0000 7fbd3f27b640 10 monclient: monmap: 2024-09-16T04:10:43.456 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:10:43.456 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:10:43.456 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:10:43.454201+0000 2024-09-16T04:10:43.456 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:10:43.454201+0000 2024-09-16T04:10:43.456 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:10:43.456 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:10:43.456 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:10:43.456 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:10:43.456 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:10:43.457 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:10:43.457 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:10:43.457 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:10:43.467 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.451+0000 7fbd3f27b640 1 Processor -- start 2024-09-16T04:10:43.467 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.451+0000 7fbd3f27b640 1 -- start start 2024-09-16T04:10:43.467 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.451+0000 7fbd3f27b640 10 monclient: init 2024-09-16T04:10:43.467 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.451+0000 7fbd3f27b640 10 monclient: _reopen_session rank -1 2024-09-16T04:10:43.467 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.451+0000 7fbd3f27b640 10 monclient: _add_conns ranks=[2,1,3,4,0] 2024-09-16T04:10:43.467 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.451+0000 7fbd3f27b640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbd38130b20 0x7fbd38130f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:10:43.467 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.451+0000 7fbd3f27b640 10 monclient(hunting): picked mon.noname-b con 0x7fbd38130b20 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:10:43.467 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.451+0000 7fbd3f27b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbd380a2250 addr v1:172.21.15.154:6789/0 2024-09-16T04:10:43.467 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.451+0000 7fbd3f27b640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fbd38073420 0x7fbd3813f240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:10:43.467 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.451+0000 7fbd3f27b640 10 monclient(hunting): picked mon.noname-a con 0x7fbd38073420 addr v2:172.21.15.42:3300/0 2024-09-16T04:10:43.468 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.451+0000 7fbd3f27b640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbd380a1040 con 0x7fbd380a2250 2024-09-16T04:10:43.468 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.451+0000 7fbd3f27b640 10 monclient(hunting): start opening mon connection 2024-09-16T04:10:43.468 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.451+0000 7fbd3f27b640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7fbd380a12b0 con 0x7fbd38073420 2024-09-16T04:10:43.468 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.451+0000 7fbd3f27b640 10 monclient(hunting): start opening mon connection 2024-09-16T04:10:43.468 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.451+0000 7fbd3f27b640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fbd380a1590 con 0x7fbd38130b20 2024-09-16T04:10:43.468 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.451+0000 7fbd3f27b640 10 monclient(hunting): _renew_subs 2024-09-16T04:10:43.468 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.451+0000 7fbd3f27b640 10 monclient(hunting): authenticate will time out at 2690.423524s 2024-09-16T04:10:43.468 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.451+0000 7fbd3d818640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fbd38073420 0x7fbd3813f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:10:43.468 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.451+0000 7fbd3d017640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbd38130b20 0x7fbd38130f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:10:43.468 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.451+0000 7fbd3d017640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbd38130b20 0x7fbd38130f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:48642/0 (socket says 172.21.15.42:48642) 2024-09-16T04:10:43.468 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.451+0000 7fbd3d818640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fbd38073420 0x7fbd3813f240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:38704/0 (socket says 172.21.15.42:38704) 2024-09-16T04:10:43.468 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.451+0000 7fbd3d017640 1 -- 172.21.15.42:0/44241831 learned_addr learned my addr 172.21.15.42:0/44241831 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:10:43.468 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.451+0000 7fbd3d017640 10 monclient(hunting): get_auth_request con 0x7fbd38130b20 auth_method 0 2024-09-16T04:10:43.469 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.451+0000 7fbd3d017640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:10:43.469 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.451+0000 7fbd3d017640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:43.469 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.451+0000 7fbd3d017640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:43.469 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.451+0000 7fbd3d818640 10 monclient(hunting): get_auth_request con 0x7fbd38073420 auth_method 0 2024-09-16T04:10:43.469 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.451+0000 7fbd3d818640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:10:43.469 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.451+0000 7fbd3d818640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:43.469 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.451+0000 7fbd3d818640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:43.469 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.451+0000 7fbd3d818640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:10:43.469 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.451+0000 7fbd3d818640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:10:43.469 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.451+0000 7fbd3d818640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:10:43.469 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.451+0000 7fbd3d017640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:10:43.469 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.451+0000 7fbd3d017640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:10:43.469 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.451+0000 7fbd3d017640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:10:43.469 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.455+0000 7fbd3d818640 10 monclient(hunting): handle_auth_done global_id 1034322 payload 274 2024-09-16T04:10:43.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.455+0000 7fbd3d818640 1 -- 172.21.15.42:0/44241831 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbd38130b20 msgr2=0x7fbd38130f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:10:43.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.455+0000 7fbd3d818640 1 --2- 172.21.15.42:0/44241831 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbd38130b20 0x7fbd38130f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:10:43.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.455+0000 7fbd3d818640 1 -- 172.21.15.42:0/44241831 >> v1:172.21.15.154:6789/0 conn(0x7fbd380a2250 legacy=0x7fbd38072d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:10:43.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.455+0000 7fbd3d818640 10 monclient: _finish_hunting 0 2024-09-16T04:10:43.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.455+0000 7fbd3d818640 1 monclient: found mon.noname-a 2024-09-16T04:10:43.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.455+0000 7fbd3d818640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:10:43.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.455+0000 7fbd3d818640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:10:43.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.455+0000 7fbd3d818640 1 -- 172.21.15.42:0/44241831 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd380a1870 con 0x7fbd38073420 2024-09-16T04:10:43.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.455+0000 7fbd3d017640 1 --2- 172.21.15.42:0/44241831 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbd38130b20 0x7fbd38130f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:10:43.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.455+0000 7fbd3d818640 1 --2- 172.21.15.42:0/44241831 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbd38073420 0x7fbd3813f240 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fbd300097c0 tx=0x7fbd3001c320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27bbd00f97c2bf09 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:10:43.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.455+0000 7fbd2ffff640 1 -- 172.21.15.42:0/44241831 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd300b0070 con 0x7fbd38073420 2024-09-16T04:10:43.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.455+0000 7fbd2ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:43.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.455+0000 7fbd2ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:10:43.471 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.455+0000 7fbd2ffff640 10 monclient: dump: 2024-09-16T04:10:43.471 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:10:43.471 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:10:43.471 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:43.471 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:43.471 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:43.471 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:10:43.471 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:10:43.471 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:10:43.471 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:10:43.471 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:10:43.471 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.455+0000 7fbd2ffff640 10 monclient: _finish_auth 0 2024-09-16T04:10:43.471 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.455+0000 7fbd2ffff640 10 monclient: _check_auth_tickets 2024-09-16T04:10:43.471 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.455+0000 7fbd2ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:43.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.455+0000 7fbd2ffff640 1 -- 172.21.15.42:0/44241831 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbd300ab600 con 0x7fbd38073420 2024-09-16T04:10:43.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.455+0000 7fbd3f27b640 5 monclient: authenticate success, global_id 1034322 2024-09-16T04:10:43.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.455+0000 7fbd3f27b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:10:43.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.455+0000 7fbd2ffff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:10:43.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.455+0000 7fbd3f27b640 10 monclient: get_monmap_and_config success 2024-09-16T04:10:43.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.455+0000 7fbd2ffff640 1 -- 172.21.15.42:0/44241831 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd300ab9d0 con 0x7fbd38073420 2024-09-16T04:10:43.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.455+0000 7fbd3f27b640 4 set_mon_vals no callback set 2024-09-16T04:10:43.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.455+0000 7fbd2ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:43.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.455+0000 7fbd2ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:10:43.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.455+0000 7fbd2ffff640 10 monclient: dump: 2024-09-16T04:10:43.472 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:10:43.472 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:10:43.472 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:43.473 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:43.473 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:43.473 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:10:43.473 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:10:43.473 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:10:43.473 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:10:43.473 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:10:43.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.455+0000 7fbd3f27b640 10 monclient: shutdown 2024-09-16T04:10:43.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.455+0000 7fbd3f27b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:10:43.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.455+0000 7fbd3f27b640 1 -- 172.21.15.42:0/44241831 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbd38073420 msgr2=0x7fbd3813f240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:43.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.455+0000 7fbd3f27b640 1 --2- 172.21.15.42:0/44241831 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbd38073420 0x7fbd3813f240 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fbd300097c0 tx=0x7fbd3001c320 comp rx=0 tx=0).stop 2024-09-16T04:10:43.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.455+0000 7fbd3f27b640 1 -- 172.21.15.42:0/44241831 shutdown_connections 2024-09-16T04:10:43.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.455+0000 7fbd3f27b640 1 --2- 172.21.15.42:0/44241831 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbd38073420 0x7fbd3813f240 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:10:43.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.455+0000 7fbd3f27b640 1 --2- 172.21.15.42:0/44241831 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbd38130b20 0x7fbd38130f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:10:43.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.455+0000 7fbd3f27b640 1 -- 172.21.15.42:0/44241831 >> 172.21.15.42:0/44241831 conn(0x7fbd380a6210 msgr2=0x7fbd380a6620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:10:43.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.455+0000 7fbd3f27b640 1 -- 172.21.15.42:0/44241831 shutdown_connections 2024-09-16T04:10:43.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.455+0000 7fbd3f27b640 1 -- 172.21.15.42:0/44241831 wait complete. 2024-09-16T04:10:43.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.455+0000 7fbd3f27b640 10 monclient: build_initial_monmap 2024-09-16T04:10:43.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.455+0000 7fbd3f27b640 10 monclient: monmap: 2024-09-16T04:10:43.476 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:10:43.476 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:10:43.477 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:10:43.477 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:10:43.477 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:10:43.477 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:10:43.477 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:10:43.477 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:10:43.477 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:10:43.477 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:10:43.477 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.455+0000 7fbd3f27b640 1 librados: starting msgr at 2024-09-16T04:10:43.477 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.455+0000 7fbd3f27b640 1 librados: starting objecter 2024-09-16T04:10:43.477 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.455+0000 7fbd3f27b640 1 Processor -- start 2024-09-16T04:10:43.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.459+0000 7fbd3f27b640 1 -- start start 2024-09-16T04:10:43.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.459+0000 7fbd3f27b640 1 librados: setting wanted keys 2024-09-16T04:10:43.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.459+0000 7fbd3f27b640 1 librados: calling monclient init 2024-09-16T04:10:43.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.459+0000 7fbd3f27b640 10 monclient: init 2024-09-16T04:10:43.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.459+0000 7fbd3f27b640 10 monclient: _reopen_session rank -1 2024-09-16T04:10:43.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.459+0000 7fbd3f27b640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:10:43.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.459+0000 7fbd3f27b640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbd38073420 0x7fbd3809f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:10:43.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.459+0000 7fbd3f27b640 10 monclient(hunting): picked mon.noname-c con 0x7fbd38073420 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:10:43.479 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.459+0000 7fbd3f27b640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbd380a2250 0x7fbd38137740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:10:43.479 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.459+0000 7fbd3f27b640 10 monclient(hunting): picked mon.noname-a con 0x7fbd380a2250 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:10:43.479 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.459+0000 7fbd3f27b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbd38130b20 0x7fbd38137c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:10:43.479 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.459+0000 7fbd3f27b640 10 monclient(hunting): picked mon.noname-b con 0x7fbd38130b20 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:10:43.479 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.459+0000 7fbd3f27b640 10 monclient(hunting): start opening mon connection 2024-09-16T04:10:43.479 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.459+0000 7fbd3f27b640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fbd381315c0 con 0x7fbd380a2250 2024-09-16T04:10:43.479 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.459+0000 7fbd3f27b640 10 monclient(hunting): start opening mon connection 2024-09-16T04:10:43.479 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.459+0000 7fbd3f27b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fbd38131440 con 0x7fbd38130b20 2024-09-16T04:10:43.479 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.459+0000 7fbd3f27b640 10 monclient(hunting): start opening mon connection 2024-09-16T04:10:43.479 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.459+0000 7fbd3f27b640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fbd38141fa0 con 0x7fbd38073420 2024-09-16T04:10:43.479 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.459+0000 7fbd3f27b640 10 monclient(hunting): _renew_subs 2024-09-16T04:10:43.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.459+0000 7fbd3f27b640 10 monclient(hunting): authenticate will time out at 2690.428531s 2024-09-16T04:10:43.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.459+0000 7fbd3c816640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbd380a2250 0x7fbd38137740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:10:43.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.459+0000 7fbd3d017640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbd38073420 0x7fbd3809f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:10:43.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.459+0000 7fbd3c816640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbd380a2250 0x7fbd38137740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:38716/0 (socket says 172.21.15.42:38716) 2024-09-16T04:10:43.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.459+0000 7fbd3c816640 1 -- 172.21.15.42:0/3030528906 learned_addr learned my addr 172.21.15.42:0/3030528906 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:10:43.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.459+0000 7fbd3c816640 10 monclient(hunting): get_auth_request con 0x7fbd380a2250 auth_method 0 2024-09-16T04:10:43.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.459+0000 7fbd3c816640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:10:43.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.459+0000 7fbd3c816640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:43.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.459+0000 7fbd3c816640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:43.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.459+0000 7fbd3d017640 10 monclient(hunting): get_auth_request con 0x7fbd38073420 auth_method 0 2024-09-16T04:10:43.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.459+0000 7fbd3d017640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:10:43.481 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.459+0000 7fbd3d017640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:43.481 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.459+0000 7fbd3d017640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:43.481 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.459+0000 7fbd3c816640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:10:43.481 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.459+0000 7fbd3c816640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:10:43.481 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.459+0000 7fbd3c816640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:10:43.481 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.459+0000 7fbd3d017640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:10:43.481 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.459+0000 7fbd3d017640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:10:43.481 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.459+0000 7fbd3d017640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:10:43.481 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.459+0000 7fbd3c816640 10 monclient(hunting): handle_auth_done global_id 1034325 payload 824 2024-09-16T04:10:43.481 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.459+0000 7fbd3c816640 1 -- 172.21.15.42:0/3030528906 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbd38073420 msgr2=0x7fbd3809f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:43.482 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.459+0000 7fbd3c816640 1 --2- 172.21.15.42:0/3030528906 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbd38073420 0x7fbd3809f780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:10:43.482 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.459+0000 7fbd3c816640 1 -- 172.21.15.42:0/3030528906 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbd38130b20 msgr2=0x7fbd38137c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:43.482 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.459+0000 7fbd3c816640 1 --2- 172.21.15.42:0/3030528906 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbd38130b20 0x7fbd38137c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:10:43.482 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.459+0000 7fbd3c816640 10 monclient: _finish_hunting 0 2024-09-16T04:10:43.482 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.459+0000 7fbd3c816640 1 monclient: found mon.noname-a 2024-09-16T04:10:43.482 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.459+0000 7fbd3c816640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:10:43.482 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.459+0000 7fbd3c816640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:10:43.482 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.459+0000 7fbd3c816640 1 -- 172.21.15.42:0/3030528906 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd3809d230 con 0x7fbd380a2250 2024-09-16T04:10:43.483 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.459+0000 7fbd3c816640 1 --2- 172.21.15.42:0/3030528906 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbd380a2250 0x7fbd38137740 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fbd3400c7c0 tx=0x7fbd3400cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:10:43.483 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.459+0000 7fbd2dffb640 10 client.?.objecter ms_handle_connect 0x7fbd380a2250 2024-09-16T04:10:43.483 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.459+0000 7fbd2dffb640 10 client.?.objecter resend_mon_ops 2024-09-16T04:10:43.483 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.459+0000 7fbd2dffb640 1 -- 172.21.15.42:0/3030528906 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd34016020 con 0x7fbd380a2250 2024-09-16T04:10:43.483 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.459+0000 7fbd2dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:43.483 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.459+0000 7fbd2dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:10:43.483 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.459+0000 7fbd2dffb640 10 monclient: dump: 2024-09-16T04:10:43.483 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:10:43.483 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:10:43.483 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:43.483 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:43.484 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:43.484 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:10:43.484 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:10:43.485 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:10:43.485 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:10:43.485 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:10:43.485 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.459+0000 7fbd2dffb640 10 monclient: _finish_auth 0 2024-09-16T04:10:43.485 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.459+0000 7fbd2dffb640 10 monclient: _check_auth_tickets 2024-09-16T04:10:43.485 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.459+0000 7fbd2dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:43.485 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.459+0000 7fbd3f27b640 5 monclient: authenticate success, global_id 1034325 2024-09-16T04:10:43.485 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.459+0000 7fbd3f27b640 10 monclient: _renew_subs 2024-09-16T04:10:43.485 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.459+0000 7fbd2dffb640 1 -- 172.21.15.42:0/3030528906 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbd3400aa50 con 0x7fbd380a2250 2024-09-16T04:10:43.485 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.459+0000 7fbd3f27b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:10:43.485 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.459+0000 7fbd3f27b640 1 -- 172.21.15.42:0/3030528906 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd381381c0 con 0x7fbd380a2250 2024-09-16T04:10:43.485 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.459+0000 7fbd2dffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:10:43.486 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.459+0000 7fbd2dffb640 1 -- 172.21.15.42:0/3030528906 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd34012420 con 0x7fbd380a2250 2024-09-16T04:10:43.486 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.459+0000 7fbd2dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:43.486 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.459+0000 7fbd2dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:10:43.486 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.459+0000 7fbd2dffb640 10 monclient: dump: 2024-09-16T04:10:43.486 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:10:43.486 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:10:43.486 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:43.486 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:43.486 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:43.486 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:10:43.486 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:10:43.487 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:10:43.487 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:10:43.487 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:10:43.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.459+0000 7fbd3f27b640 10 client.1034325.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:10:43.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.459+0000 7fbd2effd640 4 set_mon_vals no callback set 2024-09-16T04:10:43.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.459+0000 7fbd3f27b640 10 monclient: _renew_subs 2024-09-16T04:10:43.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.459+0000 7fbd3f27b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:10:43.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.459+0000 7fbd3f27b640 1 -- 172.21.15.42:0/3030528906 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd380a4330 con 0x7fbd380a2250 2024-09-16T04:10:43.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.459+0000 7fbd3f27b640 1 librados: init done 2024-09-16T04:10:43.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.459+0000 7fbd3f27b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:43.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.459+0000 7fbd3f27b640 1 --2- 172.21.15.42:0/3030528906 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbd381cf970 0x7fbd3802c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:10:43.488 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.459+0000 7fbd3f27b640 10 monclient(hunting): start opening mon connection 2024-09-16T04:10:43.488 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.459+0000 7fbd3f27b640 1 -- 172.21.15.42:0/3030528906 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fbd38139f70 con 0x7fbd381cf970 2024-09-16T04:10:43.488 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.459+0000 7fbd3d017640 1 --2- 172.21.15.42:0/3030528906 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbd381cf970 0x7fbd3802c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:10:43.488 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.459+0000 7fbd3d017640 10 monclient: get_auth_request con 0x7fbd381cf970 auth_method 0 2024-09-16T04:10:43.488 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.459+0000 7fbd3d017640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:10:43.488 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.459+0000 7fbd3d017640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:43.488 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.459+0000 7fbd3d017640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:43.488 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.459+0000 7fbd2dffb640 1 -- 172.21.15.42:0/3030528906 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fbd34019020 con 0x7fbd380a2250 2024-09-16T04:10:43.488 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.459+0000 7fbd3d017640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:10:43.488 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.459+0000 7fbd3d017640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:10:43.489 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.459+0000 7fbd3d017640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:10:43.489 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.459+0000 7fbd2dffb640 1 --2- 172.21.15.42:0/3030528906 >> v2:172.21.15.154:6812/254845342 conn(0x7fbd1c037810 0x7fbd1c039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:10:43.489 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.459+0000 7fbd3d017640 10 monclient(hunting): handle_auth_done global_id 1014446 payload 824 2024-09-16T04:10:43.489 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.459+0000 7fbd3d017640 1 -- 172.21.15.42:0/3030528906 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbd38130f00 con 0x7fbd381cf970 2024-09-16T04:10:43.489 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.459+0000 7fbd2dffb640 1 -- 172.21.15.42:0/3030528906 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(706..706 src has 451..706) ==== 5040+0+0 (secure 0 0 0) 0x7fbd34048cd0 con 0x7fbd380a2250 2024-09-16T04:10:43.489 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.459+0000 7fbd2dffb640 10 client.1034325.objecter ms_dispatch 0x7fbd38000ba0 osd_map(706..706 src has 451..706) 2024-09-16T04:10:43.490 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.459+0000 7fbd2dffb640 3 client.1034325.objecter handle_osd_map got epochs [706,706] > 0 2024-09-16T04:10:43.490 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.459+0000 7fbd2dffb640 3 client.1034325.objecter handle_osd_map decoding full epoch 706 2024-09-16T04:10:43.490 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.459+0000 7fbd2dffb640 20 client.1034325.objecter dump_active .. 0 homeless 2024-09-16T04:10:43.490 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.459+0000 7fbd2dffb640 10 client.1034325.objecter ms_handle_connect 0x7fbd381cf970 2024-09-16T04:10:43.490 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.459+0000 7fbd2dffb640 10 client.1034325.objecter resend_mon_ops 2024-09-16T04:10:43.490 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.459+0000 7fbd3d017640 1 --2- 172.21.15.42:0/3030528906 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbd381cf970 0x7fbd3802c650 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fbd280118f0 tx=0x7fbd28011db0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:10:43.490 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.459+0000 7fbd3d818640 1 --2- 172.21.15.42:0/3030528906 >> v2:172.21.15.154:6812/254845342 conn(0x7fbd1c037810 0x7fbd1c039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:10:43.490 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.459+0000 7fbd3d818640 10 monclient: get_auth_request con 0x7fbd1c037810 auth_method 0 2024-09-16T04:10:43.490 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.459+0000 7fbd2dffb640 1 -- 172.21.15.42:0/3030528906 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd2800faa0 con 0x7fbd381cf970 2024-09-16T04:10:43.490 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.459+0000 7fbd2dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:43.490 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.459+0000 7fbd2dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:10:43.491 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.459+0000 7fbd2dffb640 10 monclient: dump: 2024-09-16T04:10:43.491 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:10:43.491 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:10:43.491 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:43.491 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:43.491 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:43.491 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:10:43.491 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:10:43.491 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:10:43.491 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:10:43.491 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:10:43.492 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.459+0000 7fbd2dffb640 1 -- 172.21.15.42:0/3030528906 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fbd2800faa0 con 0x7fbd381cf970 2024-09-16T04:10:43.492 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.459+0000 7fbd2dffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:43.492 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.459+0000 7fbd2dffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:10:43.492 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.459+0000 7fbd2dffb640 1 -- 172.21.15.42:0/3030528906 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbd381cf970 msgr2=0x7fbd3802c650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:43.492 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.459+0000 7fbd2dffb640 1 --2- 172.21.15.42:0/3030528906 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbd381cf970 0x7fbd3802c650 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fbd280118f0 tx=0x7fbd28011db0 comp rx=0 tx=0).stop 2024-09-16T04:10:43.492 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.459+0000 7fbd2dffb640 1 -- 172.21.15.42:0/3030528906 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbd381cf970 msgr2=0x7fbd3802c650 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:10:43.492 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.459+0000 7fbd2dffb640 1 --2- 172.21.15.42:0/3030528906 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbd381cf970 0x7fbd3802c650 secure :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fbd280118f0 tx=0x7fbd28011db0 comp rx=0 tx=0).stop 2024-09-16T04:10:43.492 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.459+0000 7fbd2dffb640 10 client.1034325.objecter ms_handle_connect 0x7fbd1c037810 2024-09-16T04:10:43.493 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.459+0000 7fbd3d818640 1 --2- 172.21.15.42:0/3030528906 >> v2:172.21.15.154:6812/254845342 conn(0x7fbd1c037810 0x7fbd1c039cd0 secure :-1 s=READY pgs=3935 cs=0 l=1 rev1=1 crypto rx=0x7fbd300ad000 tx=0x7fbd30004740 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:10:43.495 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.467+0000 7fbd3f27b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:10:43.495 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.467+0000 7fbd3f27b640 1 --2- 172.21.15.42:0/3030528906 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbd381cfd50 0x7fbd3802cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:10:43.495 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.467+0000 7fbd3f27b640 10 monclient(hunting): start opening mon connection 2024-09-16T04:10:43.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.467+0000 7fbd3f27b640 1 -- 172.21.15.42:0/3030528906 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fbd3812f0c0 con 0x7fbd381cfd50 2024-09-16T04:10:43.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.467+0000 7fbd3d017640 1 --2- 172.21.15.42:0/3030528906 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbd381cfd50 0x7fbd3802cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:10:43.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.467+0000 7fbd3d017640 10 monclient: get_auth_request con 0x7fbd381cfd50 auth_method 0 2024-09-16T04:10:43.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.467+0000 7fbd3d017640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:10:43.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.467+0000 7fbd3d017640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:43.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.467+0000 7fbd3d017640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:43.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.467+0000 7fbd3d017640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:10:43.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.467+0000 7fbd3d017640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:10:43.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.467+0000 7fbd3d017640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:10:43.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.467+0000 7fbd3d017640 10 monclient(hunting): handle_auth_done global_id 1014449 payload 824 2024-09-16T04:10:43.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.467+0000 7fbd3d017640 1 -- 172.21.15.42:0/3030528906 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fbd3806b490 con 0x7fbd381cfd50 2024-09-16T04:10:43.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.467+0000 7fbd3d017640 1 --2- 172.21.15.42:0/3030528906 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbd381cfd50 0x7fbd3802cb90 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fbd2801c000 tx=0x7fbd280046d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:10:43.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.467+0000 7fbd2dffb640 10 client.1034325.objecter ms_handle_connect 0x7fbd381cfd50 2024-09-16T04:10:43.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.467+0000 7fbd2dffb640 10 client.1034325.objecter resend_mon_ops 2024-09-16T04:10:43.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.467+0000 7fbd2dffb640 1 -- 172.21.15.42:0/3030528906 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd28014640 con 0x7fbd381cfd50 2024-09-16T04:10:43.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.467+0000 7fbd2dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:43.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.467+0000 7fbd2dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:10:43.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.467+0000 7fbd2dffb640 10 monclient: dump: 2024-09-16T04:10:43.497 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:10:43.497 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:10:43.497 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:43.497 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:43.497 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:43.498 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:10:43.498 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:10:43.498 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:10:43.498 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:10:43.498 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:10:43.498 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.467+0000 7fbd2dffb640 1 -- 172.21.15.42:0/3030528906 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4713 (secure 0 0 0) 0x7fbd2802b470 con 0x7fbd381cfd50 2024-09-16T04:10:43.498 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.467+0000 7fbd2dffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:10:43.498 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.467+0000 7fbd2dffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:10:43.498 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.467+0000 7fbd2dffb640 1 -- 172.21.15.42:0/3030528906 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbd381cfd50 msgr2=0x7fbd3802cb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:43.498 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.467+0000 7fbd2dffb640 1 --2- 172.21.15.42:0/3030528906 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbd381cfd50 0x7fbd3802cb90 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fbd2801c000 tx=0x7fbd280046d0 comp rx=0 tx=0).stop 2024-09-16T04:10:43.498 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.467+0000 7fbd2dffb640 1 -- 172.21.15.42:0/3030528906 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbd381cfd50 msgr2=0x7fbd3802cb90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:10:43.498 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.467+0000 7fbd2dffb640 1 --2- 172.21.15.42:0/3030528906 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbd381cfd50 0x7fbd3802cb90 secure :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fbd2801c000 tx=0x7fbd280046d0 comp rx=0 tx=0).stop 2024-09-16T04:10:43.499 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T04:10:43.499 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:10:43.499 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:10:43.499 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T04:10:43.499 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 262, 2024-09-16T04:10:43.499 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 238670, 2024-09-16T04:10:43.499 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T04:10:43.499 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T04:10:43.500 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T04:10:43.500 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T04:10:43.500 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:10:43.500 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 0, 2024-09-16T04:10:43.500 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:10:43.500 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T04:10:43.500 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T04:10:43.500 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:10:43.500 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:10:43.500 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:10:43.501 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:10:43.501 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:10:43.501 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:10:43.501 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:10:43.501 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:10:43.501 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:10:43.501 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:10:43.501 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:10:43.501 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:10:43.501 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:10:43.501 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T04:10:43.502 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:10:43.502 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:10:43.502 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:10:43.502 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:10:43.502 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:10:43.502 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:10:43.502 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:10:43.502 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:10:43.502 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:10:43.502 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:10:43.502 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:10:43.502 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:10:43.503 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:10:43.503 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T04:10:43.503 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T04:10:43.503 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T04:10:43.503 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T04:10:43.503 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T04:10:43.503 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T04:10:43.503 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:10:43.503 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:10:43.503 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T04:10:43.503 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T04:10:43.504 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T04:10:43.504 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T04:10:43.504 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T04:10:43.504 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T04:10:43.504 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T04:10:43.504 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:10:43.504 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T04:10:43.504 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:10:43.504 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:10:43.504 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:10:43.504 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:10:43.504 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:10:43.505 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:10:43.505 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:10:43.505 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:10:43.505 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:10:43.505 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:10:43.505 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:10:43.505 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:10:43.505 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T04:10:43.505 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:10:43.506 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T04:10:43.506 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:10:43.506 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:10:43.506 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:10:43.506 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:10:43.506 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:10:43.506 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:10:43.506 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:10:43.506 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T04:10:43.506 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:10:43.506 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:10:43.506 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:10:43.507 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:10:43.507 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T04:10:43.507 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:10:43.507 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:10:43.507 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:10:43.507 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:10:43.507 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T04:10:43.507 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T04:10:43.507 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:10:43.507 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:10:43.507 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:10:43.508 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:10:43.508 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:10:43.508 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:10:43.508 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:10:43.508 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:10:43.508 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:10:43.508 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:10:43.508 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:10:43.508 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T04:10:43.508 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:10:43.508 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:10:43.509 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:10:43.509 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:10:43.509 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T04:10:43.509 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:10:43.509 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:10:43.509 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:10:43.509 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:10:43.509 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T04:10:43.509 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T04:10:43.509 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:10:43.509 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:10:43.509 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:10:43.510 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:10:43.510 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:10:43.510 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:10:43.510 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:10:43.510 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:10:43.510 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:10:43.510 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:10:43.510 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:10:43.510 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T04:10:43.510 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:10:43.510 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:10:43.511 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:10:43.511 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:10:43.511 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T04:10:43.511 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:10:43.511 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:10:43.511 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:10:43.511 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:10:43.511 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T04:10:43.511 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T04:10:43.511 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:10:43.512 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:10:43.512 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:10:43.512 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:10:43.512 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:10:43.512 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:10:43.512 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T04:10:43.512 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T04:10:43.512 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:10:43.512 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:10:43.512 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:10:43.512 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:10:43.513 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:10:43.513 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:10:43.513 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T04:10:43.513 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:10:43.513 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:10:43.513 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:10:43.513 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 4 2024-09-16T04:10:43.513 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:10:43.513 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:10:43.513 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T04:10:43.513 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:10:43.514 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:10:43.514 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:10:43.514 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 3 2024-09-16T04:10:43.514 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:10:43.514 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:10:43.514 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:10:43.514 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T04:10:43.514 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T04:10:43.516 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.467+0000 7fbd3f27b640 10 librados: watch_flush enter 2024-09-16T04:10:43.516 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.467+0000 7fbd3f27b640 10 librados: watch_flush exit 2024-09-16T04:10:43.516 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.467+0000 7fbd3f27b640 20 client.1034325.objecter shutdown clearing up homeless session... 2024-09-16T04:10:43.516 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.467+0000 7fbd3f27b640 10 client.1034325.objecter successfully canceled tick 2024-09-16T04:10:43.516 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.467+0000 7fbd3f27b640 1 -- 172.21.15.42:0/3030528906 >> v2:172.21.15.154:6812/254845342 conn(0x7fbd1c037810 msgr2=0x7fbd1c039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:43.516 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.467+0000 7fbd3f27b640 1 --2- 172.21.15.42:0/3030528906 >> v2:172.21.15.154:6812/254845342 conn(0x7fbd1c037810 0x7fbd1c039cd0 secure :-1 s=READY pgs=3935 cs=0 l=1 rev1=1 crypto rx=0x7fbd300ad000 tx=0x7fbd30004740 comp rx=0 tx=0).stop 2024-09-16T04:10:43.516 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.467+0000 7fbd3f27b640 10 monclient: shutdown 2024-09-16T04:10:43.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.467+0000 7fbd3f27b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:10:43.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.467+0000 7fbd3f27b640 1 -- 172.21.15.42:0/3030528906 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbd380a2250 msgr2=0x7fbd38137740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:43.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.467+0000 7fbd3f27b640 1 --2- 172.21.15.42:0/3030528906 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbd380a2250 0x7fbd38137740 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fbd3400c7c0 tx=0x7fbd3400cc80 comp rx=0 tx=0).stop 2024-09-16T04:10:43.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.467+0000 7fbd3d818640 1 -- 172.21.15.42:0/3030528906 reap_dead start 2024-09-16T04:10:43.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.467+0000 7fbd3d818640 1 -- 172.21.15.42:0/3030528906 reap_dead start 2024-09-16T04:10:43.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.467+0000 7fbd3f27b640 1 -- 172.21.15.42:0/3030528906 shutdown_connections 2024-09-16T04:10:43.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.467+0000 7fbd3f27b640 1 -- 172.21.15.42:0/3030528906 >> 172.21.15.42:0/3030528906 conn(0x7fbd380a6210 msgr2=0x7fbd38133aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:10:43.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.467+0000 7fbd3f27b640 1 -- 172.21.15.42:0/3030528906 shutdown_connections 2024-09-16T04:10:43.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.467+0000 7fbd3f27b640 1 -- 172.21.15.42:0/3030528906 wait complete. 2024-09-16T04:10:43.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.467+0000 7fbd3f27b640 1 librados: shutdown 2024-09-16T04:10:43.518 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:10:43.518 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T04:10:43.518 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T04:10:43.632 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.627+0000 7f003dfa3640 10 monclient: get_monmap_and_config 2024-09-16T04:10:43.632 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.627+0000 7f003dfa3640 10 monclient: build_initial_monmap 2024-09-16T04:10:43.633 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.627+0000 7f003dfa3640 10 monclient: monmap: 2024-09-16T04:10:43.633 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:10:43.633 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:10:43.633 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:10:43.631089+0000 2024-09-16T04:10:43.633 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:10:43.631089+0000 2024-09-16T04:10:43.633 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:10:43.633 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:10:43.633 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:10:43.633 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:10:43.633 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:10:43.633 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:10:43.634 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:10:43.634 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:10:43.636 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.627+0000 7f003dfa3640 1 Processor -- start 2024-09-16T04:10:43.636 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.627+0000 7f003dfa3640 1 -- start start 2024-09-16T04:10:43.636 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.627+0000 7f003dfa3640 10 monclient: init 2024-09-16T04:10:43.636 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.631+0000 7f003dfa3640 10 monclient: _reopen_session rank -1 2024-09-16T04:10:43.636 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.631+0000 7f003dfa3640 10 monclient: _add_conns ranks=[2,1,4,0,3] 2024-09-16T04:10:43.636 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.631+0000 7f003dfa3640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f00380a1330 0x7f003809f180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:10:43.636 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.631+0000 7f003dfa3640 10 monclient(hunting): picked mon.noname-b con 0x7f00380a1330 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:10:43.636 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.631+0000 7f003dfa3640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f00380a1710 addr v1:172.21.15.154:6789/0 2024-09-16T04:10:43.637 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.631+0000 7f003dfa3640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f00381393a0 0x7f003813b790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:10:43.637 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.631+0000 7f003dfa3640 10 monclient(hunting): picked mon.noname-c con 0x7f00381393a0 addr v2:172.21.15.154:3300/0 2024-09-16T04:10:43.637 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.631+0000 7f003dfa3640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f003809e4c0 con 0x7f00380a1710 2024-09-16T04:10:43.637 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.631+0000 7f003dfa3640 10 monclient(hunting): start opening mon connection 2024-09-16T04:10:43.637 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.631+0000 7f003dfa3640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f003809edf0 con 0x7f00381393a0 2024-09-16T04:10:43.637 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.631+0000 7f003dfa3640 10 monclient(hunting): start opening mon connection 2024-09-16T04:10:43.637 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.631+0000 7f003dfa3640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f003809cd60 con 0x7f00380a1330 2024-09-16T04:10:43.637 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.631+0000 7f003dfa3640 10 monclient(hunting): _renew_subs 2024-09-16T04:10:43.637 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.631+0000 7f003dfa3640 10 monclient(hunting): authenticate will time out at 2690.600683s 2024-09-16T04:10:43.637 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.631+0000 7f00377fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f00380a1330 0x7f003809f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:10:43.637 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.631+0000 7f00377fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f00380a1330 0x7f003809f180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:48672/0 (socket says 172.21.15.42:48672) 2024-09-16T04:10:43.638 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.631+0000 7f00377fe640 1 -- 172.21.15.42:0/125581004 learned_addr learned my addr 172.21.15.42:0/125581004 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:10:43.638 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.631+0000 7f00377fe640 10 monclient(hunting): get_auth_request con 0x7f00380a1330 auth_method 0 2024-09-16T04:10:43.638 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.631+0000 7f00377fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:10:43.638 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.631+0000 7f00377fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:43.638 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.631+0000 7f00377fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:43.638 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.631+0000 7f00377fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:10:43.638 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.631+0000 7f00377fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:10:43.638 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.631+0000 7f00377fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:10:43.638 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.631+0000 7f0037fff640 1 --2- 172.21.15.42:0/125581004 >> v2:172.21.15.154:3300/0 conn(0x7f00381393a0 0x7f003813b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:10:43.638 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.631+0000 7f0037fff640 10 monclient(hunting): get_auth_request con 0x7f00381393a0 auth_method 0 2024-09-16T04:10:43.638 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.631+0000 7f0037fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:10:43.639 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.631+0000 7f0037fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:43.639 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.631+0000 7f0037fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:43.639 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.631+0000 7f00377fe640 10 monclient(hunting): handle_auth_done global_id 1014452 payload 274 2024-09-16T04:10:43.639 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.631+0000 7f00377fe640 1 -- 172.21.15.42:0/125581004 >> v2:172.21.15.154:3300/0 conn(0x7f00381393a0 msgr2=0x7f003813b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:10:43.639 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.631+0000 7f00377fe640 1 --2- 172.21.15.42:0/125581004 >> v2:172.21.15.154:3300/0 conn(0x7f00381393a0 0x7f003813b790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:10:43.639 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.631+0000 7f00377fe640 1 -- 172.21.15.42:0/125581004 >> v1:172.21.15.154:6789/0 conn(0x7f00380a1710 legacy=0x7f00380a1b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:10:43.639 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.631+0000 7f00377fe640 10 monclient: _finish_hunting 0 2024-09-16T04:10:43.639 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.631+0000 7f00377fe640 1 monclient: found mon.noname-b 2024-09-16T04:10:43.639 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.631+0000 7f00377fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:10:43.639 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.631+0000 7f00377fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T04:10:43.640 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.631+0000 7f00377fe640 1 -- 172.21.15.42:0/125581004 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f003809f6c0 con 0x7f00380a1330 2024-09-16T04:10:43.647 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.631+0000 7f00377fe640 1 --2- 172.21.15.42:0/125581004 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f00380a1330 0x7f003809f180 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f0020009e50 tx=0x7f002001c8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=27a7debc5d814251 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:10:43.647 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.631+0000 7f00367fc640 1 -- 172.21.15.42:0/125581004 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00200a5640 con 0x7f00380a1330 2024-09-16T04:10:43.647 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.631+0000 7f00367fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:43.647 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.631+0000 7f00367fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:10:43.647 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.631+0000 7f00367fc640 10 monclient: dump: 2024-09-16T04:10:43.647 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:10:43.647 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:10:43.648 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:43.648 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:43.648 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:43.648 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:10:43.648 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:10:43.648 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:10:43.648 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:10:43.648 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:10:43.648 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.631+0000 7f00367fc640 10 monclient: _finish_auth 0 2024-09-16T04:10:43.648 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.631+0000 7f00367fc640 10 monclient: _check_auth_tickets 2024-09-16T04:10:43.648 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.631+0000 7f00367fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:43.648 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.631+0000 7f003dfa3640 5 monclient: authenticate success, global_id 1014452 2024-09-16T04:10:43.649 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.631+0000 7f00367fc640 1 -- 172.21.15.42:0/125581004 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f00200a57e0 con 0x7f00380a1330 2024-09-16T04:10:43.649 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.631+0000 7f003dfa3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:10:43.649 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.631+0000 7f00367fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:10:43.649 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.631+0000 7f003dfa3640 10 monclient: get_monmap_and_config success 2024-09-16T04:10:43.649 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.631+0000 7f003dfa3640 4 set_mon_vals no callback set 2024-09-16T04:10:43.649 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.631+0000 7f00367fc640 1 -- 172.21.15.42:0/125581004 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00200a5bb0 con 0x7f00380a1330 2024-09-16T04:10:43.649 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.631+0000 7f00367fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:43.649 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.631+0000 7f00367fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:10:43.649 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.631+0000 7f00367fc640 10 monclient: dump: 2024-09-16T04:10:43.649 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:10:43.650 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:10:43.650 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:43.650 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:43.650 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:43.650 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:10:43.650 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:10:43.650 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:10:43.650 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:10:43.650 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:10:43.650 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.635+0000 7f003dfa3640 10 monclient: shutdown 2024-09-16T04:10:43.650 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.635+0000 7f003dfa3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:10:43.650 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.635+0000 7f003dfa3640 1 -- 172.21.15.42:0/125581004 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f00380a1330 msgr2=0x7f003809f180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:43.650 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.635+0000 7f003dfa3640 1 --2- 172.21.15.42:0/125581004 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f00380a1330 0x7f003809f180 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f0020009e50 tx=0x7f002001c8b0 comp rx=0 tx=0).stop 2024-09-16T04:10:43.651 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.635+0000 7f003dfa3640 1 -- 172.21.15.42:0/125581004 shutdown_connections 2024-09-16T04:10:43.651 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.635+0000 7f003dfa3640 1 --2- 172.21.15.42:0/125581004 >> v2:172.21.15.154:3300/0 conn(0x7f00381393a0 0x7f003813b790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:10:43.651 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.635+0000 7f003dfa3640 1 --2- 172.21.15.42:0/125581004 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f00380a1330 0x7f003809f180 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:10:43.651 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.635+0000 7f003dfa3640 1 -- 172.21.15.42:0/125581004 >> 172.21.15.42:0/125581004 conn(0x7f00380a2420 msgr2=0x7f00380a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:10:43.651 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.635+0000 7f003dfa3640 1 -- 172.21.15.42:0/125581004 shutdown_connections 2024-09-16T04:10:43.651 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.635+0000 7f003dfa3640 1 -- 172.21.15.42:0/125581004 wait complete. 2024-09-16T04:10:43.651 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.635+0000 7f003dfa3640 10 monclient: build_initial_monmap 2024-09-16T04:10:43.651 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.635+0000 7f003dfa3640 10 monclient: monmap: 2024-09-16T04:10:43.651 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:10:43.651 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:10:43.651 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:10:43.651 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:10:43.652 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:10:43.652 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:10:43.652 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:10:43.652 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:10:43.652 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:10:43.652 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:10:43.652 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.635+0000 7f003dfa3640 1 librados: starting msgr at 2024-09-16T04:10:43.652 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.635+0000 7f003dfa3640 1 librados: starting objecter 2024-09-16T04:10:43.652 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.635+0000 7f003dfa3640 1 Processor -- start 2024-09-16T04:10:43.652 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.635+0000 7f003dfa3640 1 -- start start 2024-09-16T04:10:43.653 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.635+0000 7f003dfa3640 1 librados: setting wanted keys 2024-09-16T04:10:43.653 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.635+0000 7f003dfa3640 1 librados: calling monclient init 2024-09-16T04:10:43.653 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.635+0000 7f003dfa3640 10 monclient: init 2024-09-16T04:10:43.653 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.635+0000 7f003dfa3640 10 monclient: _reopen_session rank -1 2024-09-16T04:10:43.653 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.635+0000 7f003dfa3640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:10:43.653 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.635+0000 7f003dfa3640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f00380a1330 0x7f00381cf210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:10:43.653 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.635+0000 7f003dfa3640 10 monclient(hunting): picked mon.noname-a con 0x7f00380a1330 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:10:43.653 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.635+0000 7f003dfa3640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f00380a1710 0x7f00381344a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:10:43.653 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.635+0000 7f003dfa3640 10 monclient(hunting): picked mon.noname-b con 0x7f00380a1710 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:10:43.653 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.635+0000 7f003dfa3640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f00381393a0 0x7f00381349e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:10:43.653 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.635+0000 7f003dfa3640 10 monclient(hunting): picked mon.noname-c con 0x7f00381393a0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:10:43.653 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.635+0000 7f003dfa3640 10 monclient(hunting): start opening mon connection 2024-09-16T04:10:43.653 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.635+0000 7f003dfa3640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f003813e880 con 0x7f00380a1330 2024-09-16T04:10:43.654 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.635+0000 7f003dfa3640 10 monclient(hunting): start opening mon connection 2024-09-16T04:10:43.654 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.635+0000 7f003dfa3640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f003813e700 con 0x7f00380a1710 2024-09-16T04:10:43.654 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.635+0000 7f003dfa3640 10 monclient(hunting): start opening mon connection 2024-09-16T04:10:43.654 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.635+0000 7f003dfa3640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f003813ea00 con 0x7f00381393a0 2024-09-16T04:10:43.654 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.635+0000 7f003dfa3640 10 monclient(hunting): _renew_subs 2024-09-16T04:10:43.654 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.635+0000 7f003dfa3640 10 monclient(hunting): authenticate will time out at 2690.605817s 2024-09-16T04:10:43.654 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.635+0000 7f00377fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f00380a1330 0x7f00381cf210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:10:43.654 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.635+0000 7f0037fff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f00381393a0 0x7f00381349e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:10:43.654 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.635+0000 7f00377fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f00380a1330 0x7f00381cf210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:38732/0 (socket says 172.21.15.42:38732) 2024-09-16T04:10:43.654 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.635+0000 7f00377fe640 1 -- 172.21.15.42:0/929737019 learned_addr learned my addr 172.21.15.42:0/929737019 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:10:43.654 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.635+0000 7f00377fe640 10 monclient(hunting): get_auth_request con 0x7f00380a1330 auth_method 0 2024-09-16T04:10:43.654 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.635+0000 7f00377fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:10:43.654 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.635+0000 7f00377fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:43.655 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.635+0000 7f00377fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:43.655 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.635+0000 7f0037fff640 10 monclient(hunting): get_auth_request con 0x7f00381393a0 auth_method 0 2024-09-16T04:10:43.655 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.635+0000 7f0037fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:10:43.655 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.635+0000 7f0037fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:43.655 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.635+0000 7f0037fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:43.655 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.635+0000 7f0036ffd640 1 --2- 172.21.15.42:0/929737019 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f00380a1710 0x7f00381344a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:10:43.655 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.635+0000 7f0036ffd640 10 monclient(hunting): get_auth_request con 0x7f00380a1710 auth_method 0 2024-09-16T04:10:43.655 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.635+0000 7f0036ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:10:43.655 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.635+0000 7f0036ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:43.655 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.635+0000 7f0036ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:43.655 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.635+0000 7f00377fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:10:43.655 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.635+0000 7f00377fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:10:43.655 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.635+0000 7f00377fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:10:43.656 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.635+0000 7f0037fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:10:43.656 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.635+0000 7f0037fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:10:43.656 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.635+0000 7f0037fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:10:43.656 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.635+0000 7f0036ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:10:43.656 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.635+0000 7f0036ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:10:43.656 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.635+0000 7f0036ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:10:43.656 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.635+0000 7f00377fe640 10 monclient(hunting): handle_auth_done global_id 1034328 payload 824 2024-09-16T04:10:43.656 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.635+0000 7f00377fe640 1 -- 172.21.15.42:0/929737019 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f00381393a0 msgr2=0x7f00381349e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:43.656 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.635+0000 7f00377fe640 1 --2- 172.21.15.42:0/929737019 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f00381393a0 0x7f00381349e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:10:43.656 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.635+0000 7f00377fe640 1 -- 172.21.15.42:0/929737019 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f00380a1710 msgr2=0x7f00381344a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:43.656 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.635+0000 7f00377fe640 1 --2- 172.21.15.42:0/929737019 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f00380a1710 0x7f00381344a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:10:43.656 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.635+0000 7f00377fe640 10 monclient: _finish_hunting 0 2024-09-16T04:10:43.657 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.635+0000 7f00377fe640 1 monclient: found mon.noname-a 2024-09-16T04:10:43.657 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.635+0000 7f00377fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:10:43.657 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.635+0000 7f00377fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:10:43.657 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.635+0000 7f00377fe640 1 -- 172.21.15.42:0/929737019 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00381218a0 con 0x7f00380a1330 2024-09-16T04:10:43.657 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.635+0000 7f00377fe640 1 --2- 172.21.15.42:0/929737019 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f00380a1330 0x7f00381cf210 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f002001af60 tx=0x7f0020019650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:10:43.657 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.635+0000 7f002ffff640 10 client.?.objecter ms_handle_connect 0x7f00380a1330 2024-09-16T04:10:43.657 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.635+0000 7f002ffff640 10 client.?.objecter resend_mon_ops 2024-09-16T04:10:43.657 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.635+0000 7f002ffff640 1 -- 172.21.15.42:0/929737019 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00200bc020 con 0x7f00380a1330 2024-09-16T04:10:43.657 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.635+0000 7f002ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:43.657 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.635+0000 7f002ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:10:43.657 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.635+0000 7f002ffff640 10 monclient: dump: 2024-09-16T04:10:43.657 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:10:43.657 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:10:43.657 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:43.658 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:43.658 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:43.658 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:10:43.658 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:10:43.658 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:10:43.658 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:10:43.658 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:10:43.658 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.635+0000 7f002ffff640 10 monclient: _finish_auth 0 2024-09-16T04:10:43.658 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.635+0000 7f002ffff640 10 monclient: _check_auth_tickets 2024-09-16T04:10:43.658 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.635+0000 7f002ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:43.659 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.635+0000 7f003dfa3640 5 monclient: authenticate success, global_id 1034328 2024-09-16T04:10:43.659 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.635+0000 7f002ffff640 1 -- 172.21.15.42:0/929737019 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f00200b8670 con 0x7f00380a1330 2024-09-16T04:10:43.659 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.635+0000 7f002ffff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:10:43.659 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.635+0000 7f002ffff640 1 -- 172.21.15.42:0/929737019 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00200b8a40 con 0x7f00380a1330 2024-09-16T04:10:43.659 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.635+0000 7f002ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:43.659 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.635+0000 7f002ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:10:43.659 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.635+0000 7f00357fa640 4 set_mon_vals no callback set 2024-09-16T04:10:43.659 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.635+0000 7f002ffff640 10 monclient: dump: 2024-09-16T04:10:43.659 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:10:43.659 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:10:43.659 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:43.659 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:43.659 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:43.659 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:10:43.660 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:10:43.660 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:10:43.660 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:10:43.660 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:10:43.660 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.635+0000 7f003dfa3640 10 monclient: _renew_subs 2024-09-16T04:10:43.660 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.635+0000 7f003dfa3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:10:43.660 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.635+0000 7f003dfa3640 1 -- 172.21.15.42:0/929737019 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00380720b0 con 0x7f00380a1330 2024-09-16T04:10:43.660 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.635+0000 7f003dfa3640 10 client.1034328.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:10:43.660 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.635+0000 7f003dfa3640 10 monclient: _renew_subs 2024-09-16T04:10:43.660 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.635+0000 7f003dfa3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:10:43.660 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.635+0000 7f003dfa3640 1 -- 172.21.15.42:0/929737019 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0038073a30 con 0x7f00380a1330 2024-09-16T04:10:43.660 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.635+0000 7f003dfa3640 1 librados: init done 2024-09-16T04:10:43.660 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.635+0000 7f003dfa3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:43.660 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.635+0000 7f003dfa3640 1 --2- 172.21.15.42:0/929737019 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0038121ae0 0x7f003811ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:10:43.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.635+0000 7f003dfa3640 10 monclient(hunting): start opening mon connection 2024-09-16T04:10:43.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.635+0000 7f003dfa3640 1 -- 172.21.15.42:0/929737019 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f003809f180 con 0x7f0038121ae0 2024-09-16T04:10:43.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.635+0000 7f002ffff640 1 -- 172.21.15.42:0/929737019 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f00200bf020 con 0x7f00380a1330 2024-09-16T04:10:43.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.635+0000 7f0036ffd640 1 --2- 172.21.15.42:0/929737019 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0038121ae0 0x7f003811ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:10:43.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.639+0000 7f002ffff640 1 --2- 172.21.15.42:0/929737019 >> v2:172.21.15.154:6812/254845342 conn(0x7f0018037870 0x7f0018039d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:10:43.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.639+0000 7f0036ffd640 10 monclient: get_auth_request con 0x7f0038121ae0 auth_method 0 2024-09-16T04:10:43.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.639+0000 7f0036ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:10:43.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.639+0000 7f0036ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:43.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.639+0000 7f0036ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:43.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.639+0000 7f002ffff640 1 -- 172.21.15.42:0/929737019 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(706..706 src has 451..706) ==== 5040+0+0 (secure 0 0 0) 0x7f00200e3900 con 0x7f00380a1330 2024-09-16T04:10:43.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.639+0000 7f002ffff640 10 client.1034328.objecter ms_dispatch 0x7f0038000ba0 osd_map(706..706 src has 451..706) 2024-09-16T04:10:43.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.639+0000 7f002ffff640 3 client.1034328.objecter handle_osd_map got epochs [706,706] > 0 2024-09-16T04:10:43.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.639+0000 7f002ffff640 3 client.1034328.objecter handle_osd_map decoding full epoch 706 2024-09-16T04:10:43.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.639+0000 7f002ffff640 20 client.1034328.objecter dump_active .. 0 homeless 2024-09-16T04:10:43.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.639+0000 7f0036ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:10:43.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.639+0000 7f0036ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:10:43.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.639+0000 7f0036ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:10:43.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.639+0000 7f0037fff640 1 --2- 172.21.15.42:0/929737019 >> v2:172.21.15.154:6812/254845342 conn(0x7f0018037870 0x7f0018039d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:10:43.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.639+0000 7f0037fff640 10 monclient: get_auth_request con 0x7f0018037870 auth_method 0 2024-09-16T04:10:43.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.639+0000 7f0036ffd640 10 monclient(hunting): handle_auth_done global_id 1064110 payload 824 2024-09-16T04:10:43.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.639+0000 7f0036ffd640 1 -- 172.21.15.42:0/929737019 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0038120450 con 0x7f0038121ae0 2024-09-16T04:10:43.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.639+0000 7f0037fff640 1 --2- 172.21.15.42:0/929737019 >> v2:172.21.15.154:6812/254845342 conn(0x7f0018037870 0x7f0018039d30 secure :-1 s=READY pgs=3936 cs=0 l=1 rev1=1 crypto rx=0x7f00381214f0 tx=0x7f0030008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:10:43.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.639+0000 7f002ffff640 10 client.1034328.objecter ms_handle_connect 0x7f0018037870 2024-09-16T04:10:43.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.639+0000 7f002ffff640 10 client.1034328.objecter ms_handle_connect 0x7f0038121ae0 2024-09-16T04:10:43.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.639+0000 7f002ffff640 10 client.1034328.objecter resend_mon_ops 2024-09-16T04:10:43.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.639+0000 7f0036ffd640 1 --2- 172.21.15.42:0/929737019 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0038121ae0 0x7f003811ded0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f0028007fd0 tx=0x7f002800fc00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:10:43.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.639+0000 7f002ffff640 1 -- 172.21.15.42:0/929737019 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0028003970 con 0x7f0038121ae0 2024-09-16T04:10:43.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.639+0000 7f002ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:43.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.639+0000 7f002ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:10:43.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.639+0000 7f002ffff640 10 monclient: dump: 2024-09-16T04:10:43.663 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:10:43.663 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:10:43.663 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:43.663 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:43.663 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:43.663 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:10:43.664 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:10:43.664 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:10:43.664 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:10:43.664 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:10:43.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.639+0000 7f002ffff640 1 -- 172.21.15.42:0/929737019 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f0028019070 con 0x7f0038121ae0 2024-09-16T04:10:43.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.639+0000 7f002ffff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:43.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.639+0000 7f002ffff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:10:43.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.639+0000 7f002ffff640 1 -- 172.21.15.42:0/929737019 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0038121ae0 msgr2=0x7f003811ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:43.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.639+0000 7f002ffff640 1 --2- 172.21.15.42:0/929737019 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0038121ae0 0x7f003811ded0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f0028007fd0 tx=0x7f002800fc00 comp rx=0 tx=0).stop 2024-09-16T04:10:43.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.639+0000 7f002ffff640 1 -- 172.21.15.42:0/929737019 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0038121ae0 msgr2=0x7f003811ded0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:10:43.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.639+0000 7f002ffff640 1 --2- 172.21.15.42:0/929737019 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0038121ae0 0x7f003811ded0 secure :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f0028007fd0 tx=0x7f002800fc00 comp rx=0 tx=0).stop 2024-09-16T04:10:43.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.647+0000 7f003dfa3640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:10:43.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.647+0000 7f003dfa3640 1 --2- 172.21.15.42:0/929737019 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f003811e410 0x7f003811e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:10:43.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.647+0000 7f003dfa3640 10 monclient(hunting): start opening mon connection 2024-09-16T04:10:43.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.647+0000 7f003dfa3640 1 -- 172.21.15.42:0/929737019 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f003811ed30 con 0x7f003811e410 2024-09-16T04:10:43.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.647+0000 7f0036ffd640 1 --2- 172.21.15.42:0/929737019 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f003811e410 0x7f003811e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:10:43.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.647+0000 7f0036ffd640 10 monclient: get_auth_request con 0x7f003811e410 auth_method 0 2024-09-16T04:10:43.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.647+0000 7f0036ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:10:43.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.647+0000 7f0036ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:43.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.647+0000 7f0036ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:43.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.647+0000 7f0036ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:10:43.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.647+0000 7f0036ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:10:43.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.647+0000 7f0036ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:10:43.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.647+0000 7f0036ffd640 10 monclient(hunting): handle_auth_done global_id 1064113 payload 824 2024-09-16T04:10:43.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.647+0000 7f0036ffd640 1 -- 172.21.15.42:0/929737019 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f0038121ec0 con 0x7f003811e410 2024-09-16T04:10:43.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.647+0000 7f0036ffd640 1 --2- 172.21.15.42:0/929737019 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f003811e410 0x7f003811e7f0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f0028016040 tx=0x7f0028003770 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:10:43.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.647+0000 7f002ffff640 10 client.1034328.objecter ms_handle_connect 0x7f003811e410 2024-09-16T04:10:43.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.647+0000 7f002ffff640 10 client.1034328.objecter resend_mon_ops 2024-09-16T04:10:43.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.647+0000 7f002ffff640 1 -- 172.21.15.42:0/929737019 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f002800db80 con 0x7f003811e410 2024-09-16T04:10:43.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.647+0000 7f002ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:43.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.647+0000 7f002ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:10:43.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.647+0000 7f002ffff640 10 monclient: dump: 2024-09-16T04:10:43.667 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:10:43.667 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:10:43.668 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:43.668 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:43.668 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:43.668 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:10:43.668 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:10:43.668 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:10:43.668 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:10:43.668 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:10:43.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.647+0000 7f002ffff640 1 -- 172.21.15.42:0/929737019 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4544 (secure 0 0 0) 0x7f002800db80 con 0x7f003811e410 2024-09-16T04:10:43.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.647+0000 7f002ffff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:10:43.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.647+0000 7f002ffff640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:10:43.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.647+0000 7f002ffff640 1 -- 172.21.15.42:0/929737019 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f003811e410 msgr2=0x7f003811e7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:43.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.647+0000 7f002ffff640 1 --2- 172.21.15.42:0/929737019 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f003811e410 0x7f003811e7f0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f0028016040 tx=0x7f0028003770 comp rx=0 tx=0).stop 2024-09-16T04:10:43.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.647+0000 7f002ffff640 1 -- 172.21.15.42:0/929737019 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f003811e410 msgr2=0x7f003811e7f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:10:43.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.647+0000 7f002ffff640 1 --2- 172.21.15.42:0/929737019 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f003811e410 0x7f003811e7f0 secure :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f0028016040 tx=0x7f0028003770 comp rx=0 tx=0).stop 2024-09-16T04:10:43.669 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T04:10:43.669 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:10:43.669 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:10:43.669 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T04:10:43.669 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 262, 2024-09-16T04:10:43.669 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 4179, 2024-09-16T04:10:43.669 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T04:10:43.669 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T04:10:43.670 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T04:10:43.670 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T04:10:43.670 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:10:43.670 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 1, 2024-09-16T04:10:43.670 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:10:43.670 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T04:10:43.670 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T04:10:43.670 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:10:43.670 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:10:43.670 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:10:43.670 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:10:43.670 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:10:43.670 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:10:43.670 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:10:43.671 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:10:43.671 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:10:43.671 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:10:43.671 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:10:43.671 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:10:43.671 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:10:43.671 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T04:10:43.671 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:10:43.671 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:10:43.671 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:10:43.671 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:10:43.671 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:10:43.671 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:10:43.671 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:10:43.672 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:10:43.672 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:10:43.672 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:10:43.672 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:10:43.672 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:10:43.672 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:10:43.672 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T04:10:43.672 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T04:10:43.672 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T04:10:43.672 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T04:10:43.672 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T04:10:43.672 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T04:10:43.672 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:10:43.673 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:10:43.673 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T04:10:43.673 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T04:10:43.673 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T04:10:43.673 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T04:10:43.673 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T04:10:43.673 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T04:10:43.673 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T04:10:43.673 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:10:43.673 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T04:10:43.673 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:10:43.673 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:10:43.673 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:10:43.673 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:10:43.674 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:10:43.674 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:10:43.674 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:10:43.674 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:10:43.674 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:10:43.674 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:10:43.674 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:10:43.674 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:10:43.674 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T04:10:43.674 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:10:43.674 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T04:10:43.674 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:10:43.674 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:10:43.674 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:10:43.675 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:10:43.675 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:10:43.675 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:10:43.675 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:10:43.675 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T04:10:43.675 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:10:43.675 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:10:43.675 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:10:43.675 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:10:43.675 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T04:10:43.675 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:10:43.675 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:10:43.675 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:10:43.675 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:10:43.676 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T04:10:43.676 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T04:10:43.676 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:10:43.676 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:10:43.676 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:10:43.676 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:10:43.676 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:10:43.676 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:10:43.676 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:10:43.676 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:10:43.676 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:10:43.676 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:10:43.676 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:10:43.676 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T04:10:43.677 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:10:43.677 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:10:43.677 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:10:43.677 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:10:43.677 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T04:10:43.677 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:10:43.677 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:10:43.677 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:10:43.677 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:10:43.677 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T04:10:43.677 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T04:10:43.677 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:10:43.677 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:10:43.677 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:10:43.677 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:10:43.678 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:10:43.678 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:10:43.678 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:10:43.678 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:10:43.678 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:10:43.678 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:10:43.678 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:10:43.678 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T04:10:43.678 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:10:43.678 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:10:43.678 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:10:43.678 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:10:43.678 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T04:10:43.679 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:10:43.679 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:10:43.679 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:10:43.679 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:10:43.679 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T04:10:43.679 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T04:10:43.679 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:10:43.679 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:10:43.679 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:10:43.679 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:10:43.679 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:10:43.679 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:10:43.679 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T04:10:43.679 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T04:10:43.679 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:10:43.680 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:10:43.680 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:10:43.680 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:10:43.680 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:10:43.680 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:10:43.680 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T04:10:43.680 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:10:43.680 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:10:43.680 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:10:43.680 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:10:43.680 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:10:43.680 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:10:43.681 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:10:43.681 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T04:10:43.681 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T04:10:43.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.647+0000 7f003dfa3640 10 librados: watch_flush enter 2024-09-16T04:10:43.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.647+0000 7f003dfa3640 10 librados: watch_flush exit 2024-09-16T04:10:43.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.647+0000 7f003dfa3640 20 client.1034328.objecter shutdown clearing up homeless session... 2024-09-16T04:10:43.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.647+0000 7f003dfa3640 10 client.1034328.objecter successfully canceled tick 2024-09-16T04:10:43.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.647+0000 7f003dfa3640 1 -- 172.21.15.42:0/929737019 >> v2:172.21.15.154:6812/254845342 conn(0x7f0018037870 msgr2=0x7f0018039d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:43.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.647+0000 7f003dfa3640 1 --2- 172.21.15.42:0/929737019 >> v2:172.21.15.154:6812/254845342 conn(0x7f0018037870 0x7f0018039d30 secure :-1 s=READY pgs=3936 cs=0 l=1 rev1=1 crypto rx=0x7f00381214f0 tx=0x7f0030008040 comp rx=0 tx=0).stop 2024-09-16T04:10:43.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.647+0000 7f003dfa3640 10 monclient: shutdown 2024-09-16T04:10:43.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.647+0000 7f003dfa3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:10:43.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.647+0000 7f003dfa3640 1 -- 172.21.15.42:0/929737019 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f00380a1330 msgr2=0x7f00381cf210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:43.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.647+0000 7f003dfa3640 1 --2- 172.21.15.42:0/929737019 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f00380a1330 0x7f00381cf210 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f002001af60 tx=0x7f0020019650 comp rx=0 tx=0).stop 2024-09-16T04:10:43.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.647+0000 7f0037fff640 1 -- 172.21.15.42:0/929737019 reap_dead start 2024-09-16T04:10:43.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.647+0000 7f0037fff640 1 -- 172.21.15.42:0/929737019 reap_dead start 2024-09-16T04:10:43.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.647+0000 7f003dfa3640 1 -- 172.21.15.42:0/929737019 shutdown_connections 2024-09-16T04:10:43.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.647+0000 7f003dfa3640 1 -- 172.21.15.42:0/929737019 >> 172.21.15.42:0/929737019 conn(0x7f00380a2420 msgr2=0x7f003809ba50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:10:43.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.647+0000 7f003dfa3640 1 -- 172.21.15.42:0/929737019 shutdown_connections 2024-09-16T04:10:43.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.647+0000 7f003dfa3640 1 -- 172.21.15.42:0/929737019 wait complete. 2024-09-16T04:10:43.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.647+0000 7f003dfa3640 1 librados: shutdown 2024-09-16T04:10:43.684 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:10:43.684 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T04:10:43.685 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T04:10:43.800 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.795+0000 7fa8e402f640 10 monclient: get_monmap_and_config 2024-09-16T04:10:43.800 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.795+0000 7fa8e402f640 10 monclient: build_initial_monmap 2024-09-16T04:10:43.800 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.795+0000 7fa8e402f640 10 monclient: monmap: 2024-09-16T04:10:43.800 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:10:43.801 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:10:43.801 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:10:43.799055+0000 2024-09-16T04:10:43.801 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:10:43.799055+0000 2024-09-16T04:10:43.801 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:10:43.801 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:10:43.801 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:10:43.801 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:10:43.801 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:10:43.801 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:10:43.801 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:10:43.801 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:10:43.803 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.795+0000 7fa8e402f640 1 Processor -- start 2024-09-16T04:10:43.804 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.795+0000 7fa8e402f640 1 -- start start 2024-09-16T04:10:43.804 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.795+0000 7fa8e402f640 10 monclient: init 2024-09-16T04:10:43.804 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.799+0000 7fa8e402f640 10 monclient: _reopen_session rank -1 2024-09-16T04:10:43.804 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.799+0000 7fa8e402f640 10 monclient: _add_conns ranks=[3,1,0,4,2] 2024-09-16T04:10:43.804 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.799+0000 7fa8e402f640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fa8dc0a3860 0x7fa8dc0a3c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:10:43.804 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.799+0000 7fa8e402f640 10 monclient(hunting): picked mon.noname-a con 0x7fa8dc0a3860 addr v2:172.21.15.42:3300/0 2024-09-16T04:10:43.804 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.799+0000 7fa8e402f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa8dc137020 addr v1:172.21.15.154:6789/0 2024-09-16T04:10:43.804 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.799+0000 7fa8e402f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa8dc137460 addr v1:172.21.15.42:6789/0 2024-09-16T04:10:43.804 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.799+0000 7fa8e402f640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa8dc0a5360 con 0x7fa8dc137460 2024-09-16T04:10:43.804 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.799+0000 7fa8e402f640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa8dc09c010 con 0x7fa8dc137020 2024-09-16T04:10:43.804 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.799+0000 7fa8e402f640 10 monclient(hunting): start opening mon connection 2024-09-16T04:10:43.804 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.799+0000 7fa8e402f640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7fa8dc0a5e80 con 0x7fa8dc0a3860 2024-09-16T04:10:43.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.799+0000 7fa8e402f640 10 monclient(hunting): _renew_subs 2024-09-16T04:10:43.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.799+0000 7fa8e402f640 10 monclient(hunting): authenticate will time out at 2690.768545s 2024-09-16T04:10:43.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.799+0000 7fa8e1dcb640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fa8dc0a3860 0x7fa8dc0a3c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:10:43.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.799+0000 7fa8e1dcb640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fa8dc0a3860 0x7fa8dc0a3c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:38746/0 (socket says 172.21.15.42:38746) 2024-09-16T04:10:43.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.799+0000 7fa8e1dcb640 1 -- 172.21.15.42:0/4181421679 learned_addr learned my addr 172.21.15.42:0/4181421679 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:10:43.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.799+0000 7fa8e1dcb640 10 monclient(hunting): get_auth_request con 0x7fa8dc0a3860 auth_method 0 2024-09-16T04:10:43.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.799+0000 7fa8e1dcb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:10:43.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.799+0000 7fa8e1dcb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:43.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.799+0000 7fa8e1dcb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:43.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.799+0000 7fa8e1dcb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:10:43.806 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.799+0000 7fa8e1dcb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:10:43.806 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.799+0000 7fa8e1dcb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:10:43.806 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.799+0000 7fa8e1dcb640 10 monclient(hunting): handle_auth_done global_id 1034331 payload 274 2024-09-16T04:10:43.806 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.799+0000 7fa8e1dcb640 1 -- 172.21.15.42:0/4181421679 >> v1:172.21.15.154:6789/0 conn(0x7fa8dc137020 legacy=0x7fa8dc0a4540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:43.806 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.799+0000 7fa8e1dcb640 1 -- 172.21.15.42:0/4181421679 >> v1:172.21.15.42:6789/0 conn(0x7fa8dc137460 legacy=0x7fa8dc0a4c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:43.806 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.799+0000 7fa8e1dcb640 10 monclient: _finish_hunting 0 2024-09-16T04:10:43.806 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.799+0000 7fa8e1dcb640 1 monclient: found mon.noname-a 2024-09-16T04:10:43.806 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.799+0000 7fa8e1dcb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:10:43.806 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.799+0000 7fa8e1dcb640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:10:43.806 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.799+0000 7fa8e1dcb640 1 -- 172.21.15.42:0/4181421679 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8dc09c2f0 con 0x7fa8dc0a3860 2024-09-16T04:10:43.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.799+0000 7fa8e1dcb640 1 --2- 172.21.15.42:0/4181421679 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa8dc0a3860 0x7fa8dc0a3c40 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fa8cc009e50 tx=0x7fa8cc01c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26ee62fdbd910102 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:10:43.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.799+0000 7fa8e0dc9640 1 -- 172.21.15.42:0/4181421679 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8cc0a56c0 con 0x7fa8dc0a3860 2024-09-16T04:10:43.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.799+0000 7fa8e0dc9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:43.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.799+0000 7fa8e0dc9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:10:43.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.799+0000 7fa8e0dc9640 10 monclient: dump: 2024-09-16T04:10:43.813 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:10:43.813 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:10:43.813 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:43.813 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:43.813 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:43.813 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:10:43.813 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:10:43.813 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:10:43.813 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:10:43.813 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:10:43.813 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.799+0000 7fa8e0dc9640 10 monclient: _finish_auth 0 2024-09-16T04:10:43.813 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.799+0000 7fa8e0dc9640 10 monclient: _check_auth_tickets 2024-09-16T04:10:43.813 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.799+0000 7fa8e0dc9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:43.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.799+0000 7fa8e0dc9640 1 -- 172.21.15.42:0/4181421679 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa8cc0a5860 con 0x7fa8dc0a3860 2024-09-16T04:10:43.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.799+0000 7fa8e402f640 5 monclient: authenticate success, global_id 1034331 2024-09-16T04:10:43.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.799+0000 7fa8e402f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:10:43.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.799+0000 7fa8e0dc9640 10 monclient: handle_config config(0 keys) 2024-09-16T04:10:43.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.799+0000 7fa8e402f640 10 monclient: get_monmap_and_config success 2024-09-16T04:10:43.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.799+0000 7fa8e0dc9640 1 -- 172.21.15.42:0/4181421679 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8cc0a5c30 con 0x7fa8dc0a3860 2024-09-16T04:10:43.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.799+0000 7fa8e0dc9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:43.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.799+0000 7fa8e402f640 4 set_mon_vals no callback set 2024-09-16T04:10:43.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.799+0000 7fa8e0dc9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:10:43.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.799+0000 7fa8e0dc9640 10 monclient: dump: 2024-09-16T04:10:43.814 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:10:43.815 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:10:43.815 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:43.815 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:43.815 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:43.815 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:10:43.815 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:10:43.815 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:10:43.815 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:10:43.815 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:10:43.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.803+0000 7fa8e402f640 10 monclient: shutdown 2024-09-16T04:10:43.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.803+0000 7fa8e402f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:10:43.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.803+0000 7fa8e402f640 1 -- 172.21.15.42:0/4181421679 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa8dc0a3860 msgr2=0x7fa8dc0a3c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:43.817 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.803+0000 7fa8e402f640 1 --2- 172.21.15.42:0/4181421679 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa8dc0a3860 0x7fa8dc0a3c40 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fa8cc009e50 tx=0x7fa8cc01c8f0 comp rx=0 tx=0).stop 2024-09-16T04:10:43.817 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.803+0000 7fa8e402f640 1 -- 172.21.15.42:0/4181421679 shutdown_connections 2024-09-16T04:10:43.817 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.803+0000 7fa8e402f640 1 --2- 172.21.15.42:0/4181421679 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa8dc0a3860 0x7fa8dc0a3c40 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:10:43.818 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.803+0000 7fa8e402f640 1 -- 172.21.15.42:0/4181421679 >> 172.21.15.42:0/4181421679 conn(0x7fa8dc0a9f40 msgr2=0x7fa8dc0aa350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:10:43.818 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.803+0000 7fa8e402f640 1 -- 172.21.15.42:0/4181421679 shutdown_connections 2024-09-16T04:10:43.818 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.803+0000 7fa8e402f640 1 -- 172.21.15.42:0/4181421679 wait complete. 2024-09-16T04:10:43.818 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.803+0000 7fa8e402f640 10 monclient: build_initial_monmap 2024-09-16T04:10:43.818 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.803+0000 7fa8e402f640 10 monclient: monmap: 2024-09-16T04:10:43.818 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:10:43.819 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:10:43.819 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:10:43.819 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:10:43.819 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:10:43.819 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:10:43.819 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:10:43.820 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:10:43.820 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:10:43.820 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:10:43.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.803+0000 7fa8e402f640 1 librados: starting msgr at 2024-09-16T04:10:43.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.803+0000 7fa8e402f640 1 librados: starting objecter 2024-09-16T04:10:43.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.803+0000 7fa8e402f640 1 Processor -- start 2024-09-16T04:10:43.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.803+0000 7fa8e402f640 1 -- start start 2024-09-16T04:10:43.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.803+0000 7fa8e402f640 1 librados: setting wanted keys 2024-09-16T04:10:43.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.803+0000 7fa8e402f640 1 librados: calling monclient init 2024-09-16T04:10:43.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.803+0000 7fa8e402f640 10 monclient: init 2024-09-16T04:10:43.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.803+0000 7fa8e402f640 10 monclient: _reopen_session rank -1 2024-09-16T04:10:43.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.803+0000 7fa8e402f640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:10:43.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.803+0000 7fa8e402f640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa8dc0a3860 0x7fa8dc1cf330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:10:43.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.803+0000 7fa8e402f640 10 monclient(hunting): picked mon.noname-c con 0x7fa8dc0a3860 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:10:43.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.803+0000 7fa8e402f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa8dc137020 0x7fa8dc1cf870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:10:43.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.803+0000 7fa8e402f640 10 monclient(hunting): picked mon.noname-b con 0x7fa8dc137020 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:10:43.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.803+0000 7fa8e402f640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa8dc137460 0x7fa8dc0a16f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:10:43.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.803+0000 7fa8e402f640 10 monclient(hunting): picked mon.noname-a con 0x7fa8dc137460 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:10:43.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.803+0000 7fa8e402f640 10 monclient(hunting): start opening mon connection 2024-09-16T04:10:43.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.803+0000 7fa8e402f640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fa8dc142db0 con 0x7fa8dc137460 2024-09-16T04:10:43.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.803+0000 7fa8e402f640 10 monclient(hunting): start opening mon connection 2024-09-16T04:10:43.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.803+0000 7fa8e402f640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa8dc142c30 con 0x7fa8dc137020 2024-09-16T04:10:43.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.803+0000 7fa8e402f640 10 monclient(hunting): start opening mon connection 2024-09-16T04:10:43.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.803+0000 7fa8e402f640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fa8dc142f30 con 0x7fa8dc0a3860 2024-09-16T04:10:43.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.803+0000 7fa8e402f640 10 monclient(hunting): _renew_subs 2024-09-16T04:10:43.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.803+0000 7fa8e402f640 10 monclient(hunting): authenticate will time out at 2690.774385s 2024-09-16T04:10:43.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.803+0000 7fa8e1dcb640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa8dc0a3860 0x7fa8dc1cf330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:10:43.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.803+0000 7fa8e25cc640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa8dc137460 0x7fa8dc0a16f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:10:43.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.803+0000 7fa8e25cc640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa8dc137460 0x7fa8dc0a16f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:38756/0 (socket says 172.21.15.42:38756) 2024-09-16T04:10:43.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.803+0000 7fa8e25cc640 1 -- 172.21.15.42:0/1417364925 learned_addr learned my addr 172.21.15.42:0/1417364925 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:10:43.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.803+0000 7fa8e25cc640 10 monclient(hunting): get_auth_request con 0x7fa8dc137460 auth_method 0 2024-09-16T04:10:43.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.803+0000 7fa8e25cc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:10:43.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.803+0000 7fa8e25cc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:43.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.803+0000 7fa8e25cc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:43.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.803+0000 7fa8e1dcb640 10 monclient(hunting): get_auth_request con 0x7fa8dc0a3860 auth_method 0 2024-09-16T04:10:43.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.803+0000 7fa8e1dcb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:10:43.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.803+0000 7fa8e1dcb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:43.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.803+0000 7fa8e1dcb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:43.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.803+0000 7fa8e25cc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:10:43.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.803+0000 7fa8e25cc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:10:43.823 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.803+0000 7fa8e25cc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:10:43.823 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.803+0000 7fa8e1dcb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:10:43.823 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.803+0000 7fa8e1dcb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:10:43.823 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.803+0000 7fa8e1dcb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:10:43.823 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.803+0000 7fa8e15ca640 1 --2- 172.21.15.42:0/1417364925 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa8dc137020 0x7fa8dc1cf870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:10:43.823 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.803+0000 7fa8e15ca640 10 monclient(hunting): get_auth_request con 0x7fa8dc137020 auth_method 0 2024-09-16T04:10:43.823 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.803+0000 7fa8e15ca640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:10:43.823 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.803+0000 7fa8e15ca640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:43.823 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.803+0000 7fa8e15ca640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:43.823 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.803+0000 7fa8e1dcb640 10 monclient(hunting): handle_auth_done global_id 1014458 payload 824 2024-09-16T04:10:43.823 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.803+0000 7fa8e1dcb640 1 -- 172.21.15.42:0/1417364925 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa8dc137020 msgr2=0x7fa8dc1cf870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:43.824 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.803+0000 7fa8e1dcb640 1 --2- 172.21.15.42:0/1417364925 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa8dc137020 0x7fa8dc1cf870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:10:43.824 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.803+0000 7fa8e1dcb640 1 -- 172.21.15.42:0/1417364925 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa8dc137460 msgr2=0x7fa8dc0a16f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:43.824 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.803+0000 7fa8e1dcb640 1 --2- 172.21.15.42:0/1417364925 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa8dc137460 0x7fa8dc0a16f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:10:43.824 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.803+0000 7fa8e1dcb640 10 monclient: _finish_hunting 0 2024-09-16T04:10:43.824 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.803+0000 7fa8e1dcb640 1 monclient: found mon.noname-c 2024-09-16T04:10:43.824 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.803+0000 7fa8e1dcb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:10:43.824 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.803+0000 7fa8e1dcb640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T04:10:43.824 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.803+0000 7fa8e1dcb640 1 -- 172.21.15.42:0/1417364925 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8dc139db0 con 0x7fa8dc0a3860 2024-09-16T04:10:43.825 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.803+0000 7fa8e25cc640 1 --2- 172.21.15.42:0/1417364925 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa8dc137460 0x7fa8dc0a16f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:10:43.825 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.803+0000 7fa8e1dcb640 1 --2- 172.21.15.42:0/1417364925 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa8dc0a3860 0x7fa8dc1cf330 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fa8dc06edd0 tx=0x7fa8cc019690 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:10:43.825 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.803+0000 7fa8da7fc640 10 client.?.objecter ms_handle_connect 0x7fa8dc0a3860 2024-09-16T04:10:43.825 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.803+0000 7fa8da7fc640 10 client.?.objecter resend_mon_ops 2024-09-16T04:10:43.825 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.803+0000 7fa8da7fc640 1 -- 172.21.15.42:0/1417364925 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8cc0bc030 con 0x7fa8dc0a3860 2024-09-16T04:10:43.825 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.803+0000 7fa8da7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:43.825 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.803+0000 7fa8da7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:10:43.825 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.803+0000 7fa8da7fc640 10 monclient: dump: 2024-09-16T04:10:43.825 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:10:43.825 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:10:43.825 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:43.825 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:43.825 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:43.826 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:10:43.826 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:10:43.826 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:10:43.826 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:10:43.826 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:10:43.826 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.803+0000 7fa8da7fc640 10 monclient: _finish_auth 0 2024-09-16T04:10:43.826 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.803+0000 7fa8da7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:10:43.826 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.803+0000 7fa8da7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:43.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.803+0000 7fa8da7fc640 1 -- 172.21.15.42:0/1417364925 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa8cc0b8470 con 0x7fa8dc0a3860 2024-09-16T04:10:43.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.803+0000 7fa8e402f640 5 monclient: authenticate success, global_id 1014458 2024-09-16T04:10:43.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.803+0000 7fa8e402f640 10 monclient: _renew_subs 2024-09-16T04:10:43.828 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.803+0000 7fa8e402f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:10:43.828 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.803+0000 7fa8e402f640 1 -- 172.21.15.42:0/1417364925 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8dc0a9700 con 0x7fa8dc0a3860 2024-09-16T04:10:43.828 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.803+0000 7fa8da7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:10:43.828 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.803+0000 7fa8da7fc640 1 -- 172.21.15.42:0/1417364925 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8cc0b8860 con 0x7fa8dc0a3860 2024-09-16T04:10:43.828 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.803+0000 7fa8da7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:43.828 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.803+0000 7fa8da7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:10:43.828 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.803+0000 7fa8da7fc640 10 monclient: dump: 2024-09-16T04:10:43.828 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:10:43.828 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:10:43.828 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:43.828 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:43.828 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:43.829 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:10:43.829 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:10:43.829 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:10:43.829 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:10:43.829 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:10:43.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.803+0000 7fa8e402f640 10 client.1014458.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:10:43.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.803+0000 7fa8e402f640 10 monclient: _renew_subs 2024-09-16T04:10:43.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.803+0000 7fa8e402f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:10:43.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.803+0000 7fa8e402f640 1 -- 172.21.15.42:0/1417364925 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8dc139ad0 con 0x7fa8dc0a3860 2024-09-16T04:10:43.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.803+0000 7fa8e402f640 1 librados: init done 2024-09-16T04:10:43.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.803+0000 7fa8dbfff640 4 set_mon_vals no callback set 2024-09-16T04:10:43.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.807+0000 7fa8da7fc640 1 -- 172.21.15.42:0/1417364925 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fa8cc0bf070 con 0x7fa8dc0a3860 2024-09-16T04:10:43.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.807+0000 7fa8e402f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:43.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.807+0000 7fa8e402f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:43.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.807+0000 7fa8e402f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:10:43.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.807+0000 7fa8e402f640 1 -- 172.21.15.42:0/1417364925 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa8dc0a3c40 con 0x7fa8dc0a3860 2024-09-16T04:10:43.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.807+0000 7fa8da7fc640 1 --2- 172.21.15.42:0/1417364925 >> v2:172.21.15.154:6812/254845342 conn(0x7fa8c8037810 0x7fa8c8039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:10:43.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.807+0000 7fa8da7fc640 1 -- 172.21.15.42:0/1417364925 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(706..706 src has 451..706) ==== 5040+0+0 (secure 0 0 0) 0x7fa8cc0e3910 con 0x7fa8dc0a3860 2024-09-16T04:10:43.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.807+0000 7fa8da7fc640 10 client.1014458.objecter ms_dispatch 0x7fa8dc000ba0 osd_map(706..706 src has 451..706) 2024-09-16T04:10:43.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.807+0000 7fa8da7fc640 3 client.1014458.objecter handle_osd_map got epochs [706,706] > 0 2024-09-16T04:10:43.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.807+0000 7fa8da7fc640 3 client.1014458.objecter handle_osd_map decoding full epoch 706 2024-09-16T04:10:43.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.807+0000 7fa8da7fc640 20 client.1014458.objecter dump_active .. 0 homeless 2024-09-16T04:10:43.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.807+0000 7fa8e15ca640 1 --2- 172.21.15.42:0/1417364925 >> v2:172.21.15.154:6812/254845342 conn(0x7fa8c8037810 0x7fa8c8039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:10:43.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.807+0000 7fa8e15ca640 10 monclient: get_auth_request con 0x7fa8c8037810 auth_method 0 2024-09-16T04:10:43.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.807+0000 7fa8da7fc640 10 client.1014458.objecter ms_handle_connect 0x7fa8c8037810 2024-09-16T04:10:43.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.807+0000 7fa8e15ca640 1 --2- 172.21.15.42:0/1417364925 >> v2:172.21.15.154:6812/254845342 conn(0x7fa8c8037810 0x7fa8c8039cd0 secure :-1 s=READY pgs=3937 cs=0 l=1 rev1=1 crypto rx=0x7fa8dc0a2e20 tx=0x7fa8d0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:10:43.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.811+0000 7fa8da7fc640 1 -- 172.21.15.42:0/1417364925 <== mon.2 v2:172.21.15.42:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7fa8cc0c6070 con 0x7fa8dc0a3860 2024-09-16T04:10:43.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.811+0000 7fa8da7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:43.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:43.811+0000 7fa8da7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:10:44.040 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:44.035+0000 7fa8e402f640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T04:10:44.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:44.035+0000 7fa8e402f640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T04:10:44.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:44.035+0000 7fa8e402f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:10:44.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:44.035+0000 7fa8e402f640 1 -- 172.21.15.42:0/1417364925 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7fa8dc0a5530 con 0x7fa8dc0a3860 2024-09-16T04:10:44.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:44.039+0000 7fa8da7fc640 1 -- 172.21.15.42:0/1417364925 <== mon.2 v2:172.21.15.42:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7fa8cc016600 con 0x7fa8dc0a3860 2024-09-16T04:10:44.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:44.039+0000 7fa8da7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T04:10:44.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:44.039+0000 7fa8da7fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:10:44.044 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.039+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.044 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.039+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=28,config=2,health=12,kv=10,logm=48} crc {auth=346485904,config=692619927,health=2824782951,kv=3446188498,logm=3051044703}) 2024-09-16T04:10:44.044 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.039+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=29,config=2,health=12,kv=10,logm=47} crc {auth=3181292678,config=692619927,health=4253921481,kv=3446188498,logm=2496638038}) 2024-09-16T04:10:44.044 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.039+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.044 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.039+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=28,config=2,health=12,kv=10,logm=48} crc {auth=346485904,config=692619927,health=2824782951,kv=3446188498,logm=3051044703}) 2024-09-16T04:10:44.044 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.039+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=28,config=2,health=12,kv=10,logm=48} crc {auth=28495842,config=692619927,health=2824782951,kv=3446188498,logm=1714706018}) 2024-09-16T04:10:44.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:44.043+0000 7fa8e402f640 10 librados: watch_flush enter 2024-09-16T04:10:44.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:44.043+0000 7fa8e402f640 10 librados: watch_flush exit 2024-09-16T04:10:44.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:44.043+0000 7fa8e402f640 20 client.1014458.objecter shutdown clearing up homeless session... 2024-09-16T04:10:44.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:44.043+0000 7fa8e402f640 10 client.1014458.objecter successfully canceled tick 2024-09-16T04:10:44.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:44.043+0000 7fa8e402f640 1 -- 172.21.15.42:0/1417364925 >> v2:172.21.15.154:6812/254845342 conn(0x7fa8c8037810 msgr2=0x7fa8c8039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:44.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:44.043+0000 7fa8e402f640 1 --2- 172.21.15.42:0/1417364925 >> v2:172.21.15.154:6812/254845342 conn(0x7fa8c8037810 0x7fa8c8039cd0 secure :-1 s=READY pgs=3937 cs=0 l=1 rev1=1 crypto rx=0x7fa8dc0a2e20 tx=0x7fa8d0009040 comp rx=0 tx=0).stop 2024-09-16T04:10:44.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:44.043+0000 7fa8e402f640 10 monclient: shutdown 2024-09-16T04:10:44.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:44.043+0000 7fa8e402f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:10:44.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:44.043+0000 7fa8e402f640 1 -- 172.21.15.42:0/1417364925 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa8dc0a3860 msgr2=0x7fa8dc1cf330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:44.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:44.043+0000 7fa8e402f640 1 --2- 172.21.15.42:0/1417364925 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa8dc0a3860 0x7fa8dc1cf330 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fa8dc06edd0 tx=0x7fa8cc019690 comp rx=0 tx=0).stop 2024-09-16T04:10:44.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:44.043+0000 7fa8e402f640 1 -- 172.21.15.42:0/1417364925 shutdown_connections 2024-09-16T04:10:44.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:44.043+0000 7fa8e402f640 1 --2- 172.21.15.42:0/1417364925 >> v2:172.21.15.154:6812/254845342 conn(0x7fa8c8037810 0x7fa8c8039cd0 unknown :-1 s=CLOSED pgs=3937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:10:44.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:44.043+0000 7fa8e402f640 1 --2- 172.21.15.42:0/1417364925 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa8dc137460 0x7fa8dc0a16f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:10:44.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:44.043+0000 7fa8e402f640 1 --2- 172.21.15.42:0/1417364925 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa8dc137020 0x7fa8dc1cf870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:10:44.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:44.043+0000 7fa8e402f640 1 --2- 172.21.15.42:0/1417364925 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa8dc0a3860 0x7fa8dc1cf330 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:10:44.048 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:44.043+0000 7fa8e402f640 1 -- 172.21.15.42:0/1417364925 >> 172.21.15.42:0/1417364925 conn(0x7fa8dc0a9f40 msgr2=0x7fa8dc133b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:10:44.048 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:44.043+0000 7fa8e402f640 1 -- 172.21.15.42:0/1417364925 shutdown_connections 2024-09-16T04:10:44.048 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:44.043+0000 7fa8e402f640 1 -- 172.21.15.42:0/1417364925 wait complete. 2024-09-16T04:10:44.048 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.043+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.048 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.043+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=418850934}) 2024-09-16T04:10:44.048 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:44.043+0000 7fa8e402f640 1 librados: shutdown 2024-09-16T04:10:44.049 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.043+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2392380730}) 2024-09-16T04:10:44.050 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.043+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.050 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.043+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=418850934}) 2024-09-16T04:10:44.050 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.043+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3136425418}) 2024-09-16T04:10:44.050 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.047+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.050 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.047+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3604987222}) 2024-09-16T04:10:44.050 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.047+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4223051195}) 2024-09-16T04:10:44.050 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.047+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.050 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.047+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3604987222}) 2024-09-16T04:10:44.051 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.047+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=309968270}) 2024-09-16T04:10:44.052 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.047+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.052 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.047+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1878305756}) 2024-09-16T04:10:44.052 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.047+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4249514972}) 2024-09-16T04:10:44.052 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.047+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.052 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.047+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1878305756}) 2024-09-16T04:10:44.052 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.047+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=549379911}) 2024-09-16T04:10:44.055 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.051+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.055 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.051+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3528361390}) 2024-09-16T04:10:44.055 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.051+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4010047756}) 2024-09-16T04:10:44.055 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.051+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.055 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.051+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3528361390}) 2024-09-16T04:10:44.055 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.051+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4092502902}) 2024-09-16T04:10:44.058 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.055+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.059 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.055+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1731935929}) 2024-09-16T04:10:44.059 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.055+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2619577664}) 2024-09-16T04:10:44.059 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.055+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.059 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.055+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1731935929}) 2024-09-16T04:10:44.059 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.055+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2632766295}) 2024-09-16T04:10:44.060 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.055+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.060 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.055+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1446049394}) 2024-09-16T04:10:44.060 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.055+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1836721791}) 2024-09-16T04:10:44.061 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.055+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.061 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.055+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1446049394}) 2024-09-16T04:10:44.061 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.055+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2380208120}) 2024-09-16T04:10:44.062 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.059+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.062 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.059+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1034549574}) 2024-09-16T04:10:44.062 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.059+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1730667977}) 2024-09-16T04:10:44.063 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.059+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.063 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.059+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1034549574}) 2024-09-16T04:10:44.063 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.059+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1079868128}) 2024-09-16T04:10:44.064 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.059+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.064 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.059+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4204491056}) 2024-09-16T04:10:44.064 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.059+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=36872719}) 2024-09-16T04:10:44.064 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.059+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.064 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.059+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4204491056}) 2024-09-16T04:10:44.064 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.059+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2024799702}) 2024-09-16T04:10:44.066 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T04:10:44.067 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.063+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.067 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.063+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2558777767}) 2024-09-16T04:10:44.067 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.063+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1379192604}) 2024-09-16T04:10:44.067 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.063+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.067 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.063+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2558777767}) 2024-09-16T04:10:44.067 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.063+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3009650661}) 2024-09-16T04:10:44.068 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.063+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.068 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.063+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3859099909}) 2024-09-16T04:10:44.068 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.063+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3684133965}) 2024-09-16T04:10:44.068 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.063+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.068 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.063+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3859099909}) 2024-09-16T04:10:44.068 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.063+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1583242326}) 2024-09-16T04:10:44.069 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.067+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.070 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.067+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3038461052}) 2024-09-16T04:10:44.070 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.067+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3984691549}) 2024-09-16T04:10:44.070 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.067+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.070 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.067+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3038461052}) 2024-09-16T04:10:44.070 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.067+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1717205918}) 2024-09-16T04:10:44.071 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.067+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.071 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.067+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3789127371}) 2024-09-16T04:10:44.071 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.067+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4100825166}) 2024-09-16T04:10:44.071 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.067+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.072 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.067+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3789127371}) 2024-09-16T04:10:44.072 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.067+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3522548657}) 2024-09-16T04:10:44.073 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.071+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.073 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.071+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3333054686}) 2024-09-16T04:10:44.073 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.071+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3663151689}) 2024-09-16T04:10:44.073 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.071+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.073 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.071+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3333054686}) 2024-09-16T04:10:44.074 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.071+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4048129465}) 2024-09-16T04:10:44.075 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.071+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.075 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.071+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1335152913}) 2024-09-16T04:10:44.075 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.071+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1943021785}) 2024-09-16T04:10:44.075 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.071+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.075 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.071+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1335152913}) 2024-09-16T04:10:44.075 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.071+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1239625801}) 2024-09-16T04:10:44.076 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.071+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.077 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.071+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2528375468}) 2024-09-16T04:10:44.077 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.071+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3451113331}) 2024-09-16T04:10:44.077 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.071+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.077 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.071+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2528375468}) 2024-09-16T04:10:44.077 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.071+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1571096719}) 2024-09-16T04:10:44.078 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.075+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.079 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.075+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2318279522}) 2024-09-16T04:10:44.079 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.075+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3983689615}) 2024-09-16T04:10:44.079 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.075+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.079 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.075+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2318279522}) 2024-09-16T04:10:44.079 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.075+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3662261492}) 2024-09-16T04:10:44.080 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.075+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.080 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.075+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=723231405}) 2024-09-16T04:10:44.080 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.075+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=74128232}) 2024-09-16T04:10:44.080 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.075+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.080 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.075+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=723231405}) 2024-09-16T04:10:44.081 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.075+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=57749598}) 2024-09-16T04:10:44.082 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.079+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.082 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.079+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3625202752}) 2024-09-16T04:10:44.082 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.079+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2033665886}) 2024-09-16T04:10:44.082 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.079+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.082 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.079+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3625202752}) 2024-09-16T04:10:44.082 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.079+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2845962853}) 2024-09-16T04:10:44.084 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.079+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.084 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.079+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4146160504}) 2024-09-16T04:10:44.084 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.079+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3235909147}) 2024-09-16T04:10:44.084 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.079+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.084 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.079+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4146160504}) 2024-09-16T04:10:44.084 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.079+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2575696355}) 2024-09-16T04:10:44.085 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.083+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.086 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.083+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3458291282}) 2024-09-16T04:10:44.086 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.083+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3891995562}) 2024-09-16T04:10:44.086 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.083+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.086 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.083+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3458291282}) 2024-09-16T04:10:44.086 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.083+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3914803832}) 2024-09-16T04:10:44.087 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.083+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.088 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.083+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3486065073}) 2024-09-16T04:10:44.088 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.083+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2155122303}) 2024-09-16T04:10:44.088 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.083+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.088 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.083+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3486065073}) 2024-09-16T04:10:44.088 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.083+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=652460743}) 2024-09-16T04:10:44.089 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.087+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.089 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.087+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=393393734}) 2024-09-16T04:10:44.089 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.087+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2483448059}) 2024-09-16T04:10:44.090 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.087+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.090 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.087+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=393393734}) 2024-09-16T04:10:44.090 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.087+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2363002142}) 2024-09-16T04:10:44.092 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.087+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.092 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.087+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3156915613}) 2024-09-16T04:10:44.092 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.087+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=559317803}) 2024-09-16T04:10:44.092 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.087+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.092 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.087+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3156915613}) 2024-09-16T04:10:44.092 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.087+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=358402911}) 2024-09-16T04:10:44.093 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.091+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.093 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.091+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4081284338}) 2024-09-16T04:10:44.093 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.091+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1249415034}) 2024-09-16T04:10:44.094 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.091+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.094 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.091+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4081284338}) 2024-09-16T04:10:44.094 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.091+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1290738218}) 2024-09-16T04:10:44.095 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.091+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.095 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.091+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2809050669}) 2024-09-16T04:10:44.095 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.091+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4260053198}) 2024-09-16T04:10:44.095 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.091+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.096 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.091+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2809050669}) 2024-09-16T04:10:44.096 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.091+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=56982260}) 2024-09-16T04:10:44.097 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.095+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.097 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.095+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2378277556}) 2024-09-16T04:10:44.097 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.095+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=309107767}) 2024-09-16T04:10:44.097 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.095+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.098 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.095+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2378277556}) 2024-09-16T04:10:44.098 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.095+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=253527112}) 2024-09-16T04:10:44.099 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.095+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.099 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.095+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=531962704}) 2024-09-16T04:10:44.099 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.095+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1102668772}) 2024-09-16T04:10:44.099 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.095+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.099 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.095+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=531962704}) 2024-09-16T04:10:44.099 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.095+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1614205514}) 2024-09-16T04:10:44.101 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.095+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.101 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.095+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=714227740}) 2024-09-16T04:10:44.101 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.095+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=505041376}) 2024-09-16T04:10:44.101 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.095+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.101 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.095+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=714227740}) 2024-09-16T04:10:44.101 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.095+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=372492723}) 2024-09-16T04:10:44.103 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.099+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.103 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.099+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=884263310}) 2024-09-16T04:10:44.103 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.099+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=47772574}) 2024-09-16T04:10:44.103 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.099+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.103 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.099+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=884263310}) 2024-09-16T04:10:44.103 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.099+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3725019516}) 2024-09-16T04:10:44.105 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.099+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.105 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.099+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2481481918}) 2024-09-16T04:10:44.105 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.099+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1202816506}) 2024-09-16T04:10:44.105 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.099+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.105 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.099+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2481481918}) 2024-09-16T04:10:44.105 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.099+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2505339936}) 2024-09-16T04:10:44.106 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.103+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.106 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.103+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1405280542}) 2024-09-16T04:10:44.106 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.103+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2069523344}) 2024-09-16T04:10:44.107 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.103+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.107 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.103+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1405280542}) 2024-09-16T04:10:44.107 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.103+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3725914466}) 2024-09-16T04:10:44.108 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.103+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.108 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.103+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2729861622}) 2024-09-16T04:10:44.108 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.103+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1154231240}) 2024-09-16T04:10:44.108 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.103+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.108 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.103+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2729861622}) 2024-09-16T04:10:44.108 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.103+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2002305638}) 2024-09-16T04:10:44.110 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.107+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.110 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.107+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1065809681}) 2024-09-16T04:10:44.110 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.107+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=332515238}) 2024-09-16T04:10:44.110 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.107+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.110 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.107+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1065809681}) 2024-09-16T04:10:44.110 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.107+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1806697467}) 2024-09-16T04:10:44.112 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.107+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.112 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.107+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=816188411}) 2024-09-16T04:10:44.112 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.107+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3588198874}) 2024-09-16T04:10:44.113 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.107+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.113 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.107+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=816188411}) 2024-09-16T04:10:44.113 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.107+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3647584856}) 2024-09-16T04:10:44.114 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.111+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.114 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.111+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3502631790}) 2024-09-16T04:10:44.114 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.111+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=738673010}) 2024-09-16T04:10:44.114 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.111+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.114 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.111+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3502631790}) 2024-09-16T04:10:44.114 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.111+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=478732564}) 2024-09-16T04:10:44.116 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.111+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.116 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.111+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3076800864}) 2024-09-16T04:10:44.116 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.111+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2605287616}) 2024-09-16T04:10:44.117 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.111+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.117 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.111+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3076800864}) 2024-09-16T04:10:44.117 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.111+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2376462301}) 2024-09-16T04:10:44.118 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.115+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.118 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.115+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3338315685}) 2024-09-16T04:10:44.118 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.115+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=373542265}) 2024-09-16T04:10:44.118 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.115+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.119 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.115+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3338315685}) 2024-09-16T04:10:44.119 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.115+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3038552527}) 2024-09-16T04:10:44.120 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.115+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.120 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.115+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1258466294}) 2024-09-16T04:10:44.121 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.115+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=423181223}) 2024-09-16T04:10:44.121 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.115+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.121 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.115+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1258466294}) 2024-09-16T04:10:44.121 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.115+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3946275480}) 2024-09-16T04:10:44.122 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.119+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.123 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.119+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3991816908}) 2024-09-16T04:10:44.123 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.119+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=903230656}) 2024-09-16T04:10:44.123 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.119+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.123 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.119+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3991816908}) 2024-09-16T04:10:44.123 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.119+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=519008358}) 2024-09-16T04:10:44.124 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.119+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.124 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.119+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=132534152}) 2024-09-16T04:10:44.125 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.119+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3593623017}) 2024-09-16T04:10:44.125 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.119+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.125 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.119+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=132534152}) 2024-09-16T04:10:44.125 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.119+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1292157843}) 2024-09-16T04:10:44.126 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.123+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.126 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.123+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1925798336}) 2024-09-16T04:10:44.126 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.123+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1365403805}) 2024-09-16T04:10:44.126 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.123+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.126 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.123+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1925798336}) 2024-09-16T04:10:44.127 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.123+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4087493685}) 2024-09-16T04:10:44.128 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.123+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.128 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.123+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2334871232}) 2024-09-16T04:10:44.128 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.123+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2615423646}) 2024-09-16T04:10:44.128 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.123+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.128 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.123+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2334871232}) 2024-09-16T04:10:44.129 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.123+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3124109144}) 2024-09-16T04:10:44.130 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.127+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.130 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.127+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3105142281}) 2024-09-16T04:10:44.130 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.127+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2717725812}) 2024-09-16T04:10:44.130 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.127+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.130 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.127+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3105142281}) 2024-09-16T04:10:44.131 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.127+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1331124714}) 2024-09-16T04:10:44.132 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.127+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.132 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.127+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3670789137}) 2024-09-16T04:10:44.132 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.127+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2497572522}) 2024-09-16T04:10:44.133 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.127+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.133 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.127+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3670789137}) 2024-09-16T04:10:44.133 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.127+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3244118367}) 2024-09-16T04:10:44.134 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.131+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.134 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.131+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=307745087}) 2024-09-16T04:10:44.134 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.131+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2663124040}) 2024-09-16T04:10:44.134 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.131+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.134 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.131+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=307745087}) 2024-09-16T04:10:44.134 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.131+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=693897340}) 2024-09-16T04:10:44.136 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.131+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.136 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.131+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1930676674}) 2024-09-16T04:10:44.136 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.131+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=887625945}) 2024-09-16T04:10:44.136 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.131+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.137 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.131+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1930676674}) 2024-09-16T04:10:44.137 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.131+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3536434075}) 2024-09-16T04:10:44.138 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.135+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.138 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.135+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=952742801}) 2024-09-16T04:10:44.138 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.135+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2401628472}) 2024-09-16T04:10:44.138 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.135+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.138 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.135+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=952742801}) 2024-09-16T04:10:44.138 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.135+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1175224715}) 2024-09-16T04:10:44.140 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.135+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.140 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.135+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2344429131}) 2024-09-16T04:10:44.140 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.135+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2591628290}) 2024-09-16T04:10:44.140 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.135+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.140 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.135+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2344429131}) 2024-09-16T04:10:44.140 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.135+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1204607978}) 2024-09-16T04:10:44.142 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.139+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.142 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.139+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1400918809}) 2024-09-16T04:10:44.142 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.139+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2932499734}) 2024-09-16T04:10:44.142 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.139+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.142 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.139+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1400918809}) 2024-09-16T04:10:44.143 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.139+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1660774085}) 2024-09-16T04:10:44.144 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.139+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.144 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.139+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2504804887}) 2024-09-16T04:10:44.144 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.139+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3413770328}) 2024-09-16T04:10:44.144 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.139+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.144 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.139+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2504804887}) 2024-09-16T04:10:44.145 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.139+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2702653670}) 2024-09-16T04:10:44.146 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.143+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.146 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.143+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4217668569}) 2024-09-16T04:10:44.146 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.143+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3337417229}) 2024-09-16T04:10:44.146 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.143+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.147 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.143+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4217668569}) 2024-09-16T04:10:44.147 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.143+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2855517316}) 2024-09-16T04:10:44.149 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.143+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.149 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.143+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3695912581}) 2024-09-16T04:10:44.149 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.143+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=900304198}) 2024-09-16T04:10:44.149 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.143+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.149 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.143+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3695912581}) 2024-09-16T04:10:44.149 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.143+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3699200638}) 2024-09-16T04:10:44.151 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.147+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.151 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.147+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1625503449}) 2024-09-16T04:10:44.151 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.147+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=339510949}) 2024-09-16T04:10:44.151 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.147+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.151 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.147+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1625503449}) 2024-09-16T04:10:44.152 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.147+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3176041759}) 2024-09-16T04:10:44.154 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.151+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.154 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.151+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3203729184}) 2024-09-16T04:10:44.154 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.151+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=325224131}) 2024-09-16T04:10:44.154 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.151+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.154 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.151+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3203729184}) 2024-09-16T04:10:44.154 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.151+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=307568769}) 2024-09-16T04:10:44.155 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.151+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.155 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.151+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3498025305}) 2024-09-16T04:10:44.155 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.151+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=150414279}) 2024-09-16T04:10:44.155 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.151+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.155 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.151+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3498025305}) 2024-09-16T04:10:44.155 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.151+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4018150268}) 2024-09-16T04:10:44.157 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.155+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.157 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.155+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3461142798}) 2024-09-16T04:10:44.157 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.155+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1671115426}) 2024-09-16T04:10:44.157 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.155+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.157 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.155+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3461142798}) 2024-09-16T04:10:44.158 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.155+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3933549008}) 2024-09-16T04:10:44.159 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.155+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.159 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.155+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=478587347}) 2024-09-16T04:10:44.159 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.155+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3326241128}) 2024-09-16T04:10:44.159 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.155+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.159 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.155+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=478587347}) 2024-09-16T04:10:44.159 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.155+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3714749402}) 2024-09-16T04:10:44.161 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.159+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.161 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.159+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=201021257}) 2024-09-16T04:10:44.161 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.159+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1938202918}) 2024-09-16T04:10:44.161 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.159+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.161 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.159+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=201021257}) 2024-09-16T04:10:44.161 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.159+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2902202657}) 2024-09-16T04:10:44.163 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.159+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.163 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.159+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=832807739}) 2024-09-16T04:10:44.163 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.159+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1846940996}) 2024-09-16T04:10:44.163 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.159+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.163 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.159+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=832807739}) 2024-09-16T04:10:44.163 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.159+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1636924356}) 2024-09-16T04:10:44.165 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.159+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.165 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.159+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=528465276}) 2024-09-16T04:10:44.165 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.159+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2543220692}) 2024-09-16T04:10:44.165 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.159+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.165 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.159+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=528465276}) 2024-09-16T04:10:44.165 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.159+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3828583443}) 2024-09-16T04:10:44.167 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.163+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.167 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.163+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=737334843}) 2024-09-16T04:10:44.167 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.163+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3169251926}) 2024-09-16T04:10:44.167 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.163+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.167 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.163+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=737334843}) 2024-09-16T04:10:44.167 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.163+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3468393066}) 2024-09-16T04:10:44.169 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.163+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.169 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.163+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2667522222}) 2024-09-16T04:10:44.169 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.163+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1283543380}) 2024-09-16T04:10:44.169 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.163+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.169 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.163+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2667522222}) 2024-09-16T04:10:44.169 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.163+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1811112117}) 2024-09-16T04:10:44.171 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.167+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.171 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.167+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3185727674}) 2024-09-16T04:10:44.171 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.167+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=328578840}) 2024-09-16T04:10:44.171 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.167+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.171 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.167+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3185727674}) 2024-09-16T04:10:44.171 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.167+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2426125718}) 2024-09-16T04:10:44.173 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.171+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.173 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.171+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=111972353}) 2024-09-16T04:10:44.173 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.171+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4057326401}) 2024-09-16T04:10:44.173 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.171+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.173 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.171+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=111972353}) 2024-09-16T04:10:44.173 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.171+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3979176253}) 2024-09-16T04:10:44.175 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.171+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.175 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.171+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2991654361}) 2024-09-16T04:10:44.175 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.171+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2503686739}) 2024-09-16T04:10:44.175 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.171+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.175 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.171+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2991654361}) 2024-09-16T04:10:44.175 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.171+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1398301107}) 2024-09-16T04:10:44.177 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.171+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.177 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.171+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=479374788}) 2024-09-16T04:10:44.177 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.171+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1081789685}) 2024-09-16T04:10:44.177 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.171+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.177 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.171+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=479374788}) 2024-09-16T04:10:44.177 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.171+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3367238125}) 2024-09-16T04:10:44.179 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.175+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.179 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.175+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1237495903}) 2024-09-16T04:10:44.179 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.175+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2087885296}) 2024-09-16T04:10:44.179 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.175+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.179 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.175+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1237495903}) 2024-09-16T04:10:44.179 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.175+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3984373117}) 2024-09-16T04:10:44.181 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.179+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.181 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.179+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=979919794}) 2024-09-16T04:10:44.181 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.179+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2825152925}) 2024-09-16T04:10:44.181 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.179+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.181 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.179+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=979919794}) 2024-09-16T04:10:44.182 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.179+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=965130020}) 2024-09-16T04:10:44.183 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.179+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.184 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.179+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=608832028}) 2024-09-16T04:10:44.184 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.179+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2098161044}) 2024-09-16T04:10:44.184 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.179+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.184 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.179+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=608832028}) 2024-09-16T04:10:44.184 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.179+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=921567749}) 2024-09-16T04:10:44.185 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.183+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.185 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.183+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2339383837}) 2024-09-16T04:10:44.185 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.183+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2209068377}) 2024-09-16T04:10:44.185 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.183+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.185 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.183+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2339383837}) 2024-09-16T04:10:44.186 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.183+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=35451295}) 2024-09-16T04:10:44.187 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.183+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.187 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.183+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1748292966}) 2024-09-16T04:10:44.187 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.183+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3097174639}) 2024-09-16T04:10:44.187 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.183+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.188 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.183+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1748292966}) 2024-09-16T04:10:44.188 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.183+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3825506011}) 2024-09-16T04:10:44.189 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.187+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.189 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.187+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1266475424}) 2024-09-16T04:10:44.189 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.187+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2969387833}) 2024-09-16T04:10:44.189 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.187+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.190 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.187+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1266475424}) 2024-09-16T04:10:44.190 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.187+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1505923499}) 2024-09-16T04:10:44.191 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.187+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.192 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.187+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3715084243}) 2024-09-16T04:10:44.192 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.187+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=957943224}) 2024-09-16T04:10:44.192 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.187+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.192 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.187+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3715084243}) 2024-09-16T04:10:44.192 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.187+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2692508224}) 2024-09-16T04:10:44.193 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.191+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.194 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.191+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1430031402}) 2024-09-16T04:10:44.194 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.191+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2386233874}) 2024-09-16T04:10:44.194 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.191+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.194 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.191+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1430031402}) 2024-09-16T04:10:44.194 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.191+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4201342191}) 2024-09-16T04:10:44.195 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.191+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.195 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.191+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3559099104}) 2024-09-16T04:10:44.195 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.191+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4071200480}) 2024-09-16T04:10:44.195 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.191+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.196 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.191+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3559099104}) 2024-09-16T04:10:44.196 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.191+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=583409907}) 2024-09-16T04:10:44.197 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.195+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.197 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.195+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1308211197}) 2024-09-16T04:10:44.197 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.195+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1468320417}) 2024-09-16T04:10:44.197 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.195+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.198 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.195+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1308211197}) 2024-09-16T04:10:44.198 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.195+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3685148693}) 2024-09-16T04:10:44.199 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.195+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.199 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.195+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2303704383}) 2024-09-16T04:10:44.199 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.195+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3667333161}) 2024-09-16T04:10:44.199 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.195+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.200 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.195+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2303704383}) 2024-09-16T04:10:44.200 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.195+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1468885161}) 2024-09-16T04:10:44.201 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.199+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.201 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.199+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2258051880}) 2024-09-16T04:10:44.201 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.199+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4218308429}) 2024-09-16T04:10:44.202 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.199+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.202 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.199+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2258051880}) 2024-09-16T04:10:44.202 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.199+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3653184660}) 2024-09-16T04:10:44.203 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.199+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.203 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.199+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1082987587}) 2024-09-16T04:10:44.204 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.199+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2958414246}) 2024-09-16T04:10:44.204 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.199+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.204 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.199+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1082987587}) 2024-09-16T04:10:44.204 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.199+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=435586542}) 2024-09-16T04:10:44.205 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.203+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.205 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.203+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4029785475}) 2024-09-16T04:10:44.205 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.203+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3087549277}) 2024-09-16T04:10:44.206 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.203+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.206 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.203+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4029785475}) 2024-09-16T04:10:44.206 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.203+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1598675231}) 2024-09-16T04:10:44.207 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.203+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.207 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.203+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3261705771}) 2024-09-16T04:10:44.207 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.203+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2582031526}) 2024-09-16T04:10:44.207 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.203+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.208 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.203+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3261705771}) 2024-09-16T04:10:44.208 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.203+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2216964176}) 2024-09-16T04:10:44.209 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.207+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.209 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.207+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2962609638}) 2024-09-16T04:10:44.209 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.207+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=793017925}) 2024-09-16T04:10:44.210 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.207+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.210 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.207+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2962609638}) 2024-09-16T04:10:44.210 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.207+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1654907277}) 2024-09-16T04:10:44.211 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.207+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.211 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.207+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3338643072}) 2024-09-16T04:10:44.211 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.207+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=86218801}) 2024-09-16T04:10:44.211 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.207+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.211 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.207+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3338643072}) 2024-09-16T04:10:44.211 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.207+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2732970279}) 2024-09-16T04:10:44.213 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.211+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.213 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.211+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3473844820}) 2024-09-16T04:10:44.213 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.211+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1931054809}) 2024-09-16T04:10:44.213 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.211+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.214 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.211+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3473844820}) 2024-09-16T04:10:44.214 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.211+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2956513205}) 2024-09-16T04:10:44.215 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.211+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.216 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.211+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3157409114}) 2024-09-16T04:10:44.216 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.211+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=58618569}) 2024-09-16T04:10:44.216 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.211+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.216 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.211+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3157409114}) 2024-09-16T04:10:44.216 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.211+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4215363060}) 2024-09-16T04:10:44.218 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.215+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.218 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.215+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2176754741}) 2024-09-16T04:10:44.218 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.215+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1924962090}) 2024-09-16T04:10:44.218 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.215+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.218 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.215+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2176754741}) 2024-09-16T04:10:44.218 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.215+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3186567423}) 2024-09-16T04:10:44.225 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.223+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.226 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.223+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2398528011}) 2024-09-16T04:10:44.226 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.223+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2278101965}) 2024-09-16T04:10:44.226 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.223+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.226 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.223+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2398528011}) 2024-09-16T04:10:44.226 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.223+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1931372159}) 2024-09-16T04:10:44.231 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.227+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.231 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.227+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1255044857}) 2024-09-16T04:10:44.231 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.227+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4006409460}) 2024-09-16T04:10:44.231 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.227+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.231 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.227+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1255044857}) 2024-09-16T04:10:44.231 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.227+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1382731733}) 2024-09-16T04:10:44.266 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.263+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.266 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.263+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=254719984}) 2024-09-16T04:10:44.266 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.263+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4056539027}) 2024-09-16T04:10:44.266 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.263+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.266 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.263+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=254719984}) 2024-09-16T04:10:44.266 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.263+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=842766645}) 2024-09-16T04:10:44.271 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.267+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.271 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.267+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2262223543}) 2024-09-16T04:10:44.271 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.267+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3038660901}) 2024-09-16T04:10:44.271 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.267+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.271 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.267+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2262223543}) 2024-09-16T04:10:44.271 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.267+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3641075466}) 2024-09-16T04:10:44.273 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.267+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.273 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.267+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4266209196}) 2024-09-16T04:10:44.273 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.267+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3748864223}) 2024-09-16T04:10:44.274 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.267+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.274 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.271+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4266209196}) 2024-09-16T04:10:44.274 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.271+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1639214824}) 2024-09-16T04:10:44.275 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.271+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.275 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.271+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1013261666}) 2024-09-16T04:10:44.275 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.271+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1469002796}) 2024-09-16T04:10:44.275 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.271+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.275 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.271+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1013261666}) 2024-09-16T04:10:44.275 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.271+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3860449076}) 2024-09-16T04:10:44.277 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.271+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.277 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.271+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1570165693}) 2024-09-16T04:10:44.277 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.275+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1876428176}) 2024-09-16T04:10:44.277 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.275+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.277 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.275+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1570165693}) 2024-09-16T04:10:44.277 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.275+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3919342620}) 2024-09-16T04:10:44.279 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.275+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.279 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.275+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4217393367}) 2024-09-16T04:10:44.279 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.275+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=969667363}) 2024-09-16T04:10:44.279 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.275+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.279 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.275+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4217393367}) 2024-09-16T04:10:44.280 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.275+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2752144390}) 2024-09-16T04:10:44.281 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.279+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.281 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.279+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1928384119}) 2024-09-16T04:10:44.281 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.279+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3153332169}) 2024-09-16T04:10:44.281 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.279+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.281 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.279+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1928384119}) 2024-09-16T04:10:44.282 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.279+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4192725821}) 2024-09-16T04:10:44.283 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.279+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.283 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.279+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4039348321}) 2024-09-16T04:10:44.283 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.279+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2941168492}) 2024-09-16T04:10:44.283 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.279+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.283 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.279+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4039348321}) 2024-09-16T04:10:44.284 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.279+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2610092092}) 2024-09-16T04:10:44.286 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.283+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.286 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.283+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3260795832}) 2024-09-16T04:10:44.286 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.283+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2359724319}) 2024-09-16T04:10:44.287 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.283+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.287 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.283+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3260795832}) 2024-09-16T04:10:44.287 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.283+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3944523531}) 2024-09-16T04:10:44.288 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.283+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.288 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.283+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=820872956}) 2024-09-16T04:10:44.288 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.283+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1309139725}) 2024-09-16T04:10:44.288 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.283+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.288 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.283+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=820872956}) 2024-09-16T04:10:44.288 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.283+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4288742053}) 2024-09-16T04:10:44.289 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.287+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.290 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.287+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3799169735}) 2024-09-16T04:10:44.290 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.287+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2943992655}) 2024-09-16T04:10:44.290 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.287+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.290 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.287+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3799169735}) 2024-09-16T04:10:44.290 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.287+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3224595470}) 2024-09-16T04:10:44.291 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.287+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.291 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.287+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1105259129}) 2024-09-16T04:10:44.291 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.287+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=670088164}) 2024-09-16T04:10:44.291 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.287+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.291 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.287+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1105259129}) 2024-09-16T04:10:44.292 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.287+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=985406557}) 2024-09-16T04:10:44.293 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.291+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.294 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.291+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1075665260}) 2024-09-16T04:10:44.294 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.291+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=29431677}) 2024-09-16T04:10:44.294 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.291+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.294 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.291+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1075665260}) 2024-09-16T04:10:44.294 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.291+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1221871957}) 2024-09-16T04:10:44.296 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.291+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.296 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.291+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1834260342}) 2024-09-16T04:10:44.296 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.291+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1023734254}) 2024-09-16T04:10:44.296 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.291+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.296 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.291+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1834260342}) 2024-09-16T04:10:44.296 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.291+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=82682623}) 2024-09-16T04:10:44.297 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.295+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.297 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.295+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3503002231}) 2024-09-16T04:10:44.298 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.295+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1619963076}) 2024-09-16T04:10:44.298 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.295+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.298 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.295+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3503002231}) 2024-09-16T04:10:44.298 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.295+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=87482381}) 2024-09-16T04:10:44.299 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.295+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.299 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.295+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2034007872}) 2024-09-16T04:10:44.299 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.295+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2008744781}) 2024-09-16T04:10:44.300 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.295+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.300 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.295+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2034007872}) 2024-09-16T04:10:44.300 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.295+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1604357808}) 2024-09-16T04:10:44.301 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.299+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.302 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.299+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1161460183}) 2024-09-16T04:10:44.302 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.299+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1634986035}) 2024-09-16T04:10:44.302 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.299+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.302 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.299+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1161460183}) 2024-09-16T04:10:44.302 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.299+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=267748775}) 2024-09-16T04:10:44.304 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.299+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.304 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.299+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3809195029}) 2024-09-16T04:10:44.304 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.299+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=663556360}) 2024-09-16T04:10:44.304 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.299+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.304 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.299+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3809195029}) 2024-09-16T04:10:44.304 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.299+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1338609113}) 2024-09-16T04:10:44.306 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.303+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.306 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.303+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3973331520}) 2024-09-16T04:10:44.307 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.303+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1420648355}) 2024-09-16T04:10:44.307 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.303+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.308 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.303+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3973331520}) 2024-09-16T04:10:44.308 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.303+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=586642667}) 2024-09-16T04:10:44.308 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.303+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.308 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.303+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3708358443}) 2024-09-16T04:10:44.308 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.303+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1819847812}) 2024-09-16T04:10:44.309 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.303+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.309 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.303+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3708358443}) 2024-09-16T04:10:44.309 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.303+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4243896839}) 2024-09-16T04:10:44.310 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.307+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.310 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.307+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=184678515}) 2024-09-16T04:10:44.310 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.307+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2484009497}) 2024-09-16T04:10:44.311 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.307+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.311 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.307+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=184678515}) 2024-09-16T04:10:44.311 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.307+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4224836141}) 2024-09-16T04:10:44.312 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.307+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.312 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.307+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4069863804}) 2024-09-16T04:10:44.312 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.307+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2153668774}) 2024-09-16T04:10:44.312 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.307+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.313 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.307+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4069863804}) 2024-09-16T04:10:44.313 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.307+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=961745596}) 2024-09-16T04:10:44.314 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.311+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.314 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.311+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3203393153}) 2024-09-16T04:10:44.315 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.311+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1837199974}) 2024-09-16T04:10:44.315 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.311+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.315 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.311+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3203393153}) 2024-09-16T04:10:44.315 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.311+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3639310095}) 2024-09-16T04:10:44.317 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.311+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.317 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.311+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2385750369}) 2024-09-16T04:10:44.317 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.311+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1979715308}) 2024-09-16T04:10:44.317 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.311+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.317 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.311+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2385750369}) 2024-09-16T04:10:44.317 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.311+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3289014274}) 2024-09-16T04:10:44.319 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.315+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.319 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.315+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2668832133}) 2024-09-16T04:10:44.319 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.315+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3432942883}) 2024-09-16T04:10:44.319 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.315+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.319 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.315+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2668832133}) 2024-09-16T04:10:44.319 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.315+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3253249262}) 2024-09-16T04:10:44.321 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.315+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.321 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.315+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3405448369}) 2024-09-16T04:10:44.321 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.315+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=984225895}) 2024-09-16T04:10:44.321 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.315+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.321 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.315+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3405448369}) 2024-09-16T04:10:44.321 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.315+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3683609241}) 2024-09-16T04:10:44.323 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.319+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.323 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.319+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=971194918}) 2024-09-16T04:10:44.323 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.319+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3499944551}) 2024-09-16T04:10:44.323 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.319+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.323 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.319+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=971194918}) 2024-09-16T04:10:44.323 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.319+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3853217325}) 2024-09-16T04:10:44.325 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.319+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.325 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.319+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3542678700}) 2024-09-16T04:10:44.325 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.319+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1778100368}) 2024-09-16T04:10:44.325 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.319+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.325 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.319+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3542678700}) 2024-09-16T04:10:44.325 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.319+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=313207676}) 2024-09-16T04:10:44.327 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.323+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.327 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.323+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=451626316}) 2024-09-16T04:10:44.327 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.323+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=683899174}) 2024-09-16T04:10:44.327 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.323+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.327 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.323+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=451626316}) 2024-09-16T04:10:44.327 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.323+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2020028182}) 2024-09-16T04:10:44.330 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.323+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.330 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.323+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2516540484}) 2024-09-16T04:10:44.330 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.323+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1685274763}) 2024-09-16T04:10:44.330 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.323+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.330 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.327+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2516540484}) 2024-09-16T04:10:44.330 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.327+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=332670992}) 2024-09-16T04:10:44.331 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.327+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.331 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.327+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2019273671}) 2024-09-16T04:10:44.331 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.327+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2637699265}) 2024-09-16T04:10:44.331 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.327+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.332 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.327+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2019273671}) 2024-09-16T04:10:44.332 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.327+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3210314186}) 2024-09-16T04:10:44.333 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.331+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.333 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.331+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4209799355}) 2024-09-16T04:10:44.333 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.331+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3438873810}) 2024-09-16T04:10:44.333 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.331+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.334 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.331+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4209799355}) 2024-09-16T04:10:44.334 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.331+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=941970055}) 2024-09-16T04:10:44.335 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.331+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.335 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.331+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1745363635}) 2024-09-16T04:10:44.336 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.331+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1356021855}) 2024-09-16T04:10:44.336 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.331+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.336 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.331+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1745363635}) 2024-09-16T04:10:44.336 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.331+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1336247099}) 2024-09-16T04:10:44.337 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.335+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.337 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.335+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3383036324}) 2024-09-16T04:10:44.338 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.335+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=111303447}) 2024-09-16T04:10:44.338 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.335+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.338 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.335+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3383036324}) 2024-09-16T04:10:44.338 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.335+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1563830053}) 2024-09-16T04:10:44.339 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.335+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.339 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.335+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2425266103}) 2024-09-16T04:10:44.340 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.335+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1502359270}) 2024-09-16T04:10:44.340 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.335+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.340 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.335+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2425266103}) 2024-09-16T04:10:44.340 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.335+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=586066943}) 2024-09-16T04:10:44.342 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.339+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.342 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.339+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3024417937}) 2024-09-16T04:10:44.342 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.339+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=395647588}) 2024-09-16T04:10:44.342 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.339+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.342 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.339+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3024417937}) 2024-09-16T04:10:44.342 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.339+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2244513623}) 2024-09-16T04:10:44.344 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.339+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.344 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.339+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1923124681}) 2024-09-16T04:10:44.344 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.339+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2578191535}) 2024-09-16T04:10:44.344 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.339+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.345 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.339+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1923124681}) 2024-09-16T04:10:44.345 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.339+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3248686939}) 2024-09-16T04:10:44.346 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.343+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.346 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.343+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=307549480}) 2024-09-16T04:10:44.346 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.343+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3269746930}) 2024-09-16T04:10:44.346 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.343+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.346 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.343+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=307549480}) 2024-09-16T04:10:44.347 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.343+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2348323979}) 2024-09-16T04:10:44.348 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.343+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.348 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.343+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3110706390}) 2024-09-16T04:10:44.348 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.343+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3216397857}) 2024-09-16T04:10:44.349 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.343+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.349 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.343+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3110706390}) 2024-09-16T04:10:44.349 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.343+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2220898359}) 2024-09-16T04:10:44.350 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.347+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.350 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.347+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2443556323}) 2024-09-16T04:10:44.350 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.347+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1064041267}) 2024-09-16T04:10:44.350 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.347+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.350 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.347+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2443556323}) 2024-09-16T04:10:44.351 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.347+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2807346628}) 2024-09-16T04:10:44.352 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.347+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.352 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.347+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2244973655}) 2024-09-16T04:10:44.352 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.347+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1072119585}) 2024-09-16T04:10:44.352 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.347+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.352 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.347+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2244973655}) 2024-09-16T04:10:44.352 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.347+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=777608926}) 2024-09-16T04:10:44.354 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.351+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.354 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.351+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2941831875}) 2024-09-16T04:10:44.355 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.351+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2436013745}) 2024-09-16T04:10:44.355 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.351+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.355 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.351+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2941831875}) 2024-09-16T04:10:44.355 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.351+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1816857563}) 2024-09-16T04:10:44.356 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.351+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.356 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.351+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3465299306}) 2024-09-16T04:10:44.356 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.351+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3029942186}) 2024-09-16T04:10:44.357 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.351+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.357 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.351+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3465299306}) 2024-09-16T04:10:44.357 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.351+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3060313132}) 2024-09-16T04:10:44.358 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.355+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.358 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.355+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2715545728}) 2024-09-16T04:10:44.358 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.355+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=457041658}) 2024-09-16T04:10:44.358 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.355+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.358 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.355+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2715545728}) 2024-09-16T04:10:44.359 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.355+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3604906082}) 2024-09-16T04:10:44.360 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.355+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.360 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.355+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1249688431}) 2024-09-16T04:10:44.360 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.355+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2297330576}) 2024-09-16T04:10:44.360 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.355+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.360 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.355+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1249688431}) 2024-09-16T04:10:44.360 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.355+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4057993417}) 2024-09-16T04:10:44.361 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.359+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.361 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.359+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=231700695}) 2024-09-16T04:10:44.361 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.359+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3590673166}) 2024-09-16T04:10:44.362 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.359+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.362 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.359+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=231700695}) 2024-09-16T04:10:44.362 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.359+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2942308394}) 2024-09-16T04:10:44.363 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.359+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.363 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.359+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2028695774}) 2024-09-16T04:10:44.363 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.359+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3394988542}) 2024-09-16T04:10:44.363 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.359+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.363 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.359+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2028695774}) 2024-09-16T04:10:44.363 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.359+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2065370541}) 2024-09-16T04:10:44.365 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.359+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.365 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.359+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2649923471}) 2024-09-16T04:10:44.365 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.359+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3381173233}) 2024-09-16T04:10:44.365 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.359+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.365 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.359+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2649923471}) 2024-09-16T04:10:44.365 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.359+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1886563450}) 2024-09-16T04:10:44.367 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.363+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.367 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.363+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2086390777}) 2024-09-16T04:10:44.367 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.363+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2947088656}) 2024-09-16T04:10:44.368 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.363+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.368 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.363+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2086390777}) 2024-09-16T04:10:44.368 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.363+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2433270788}) 2024-09-16T04:10:44.368 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.363+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.368 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.363+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=52,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=9,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=5} crc {logm=898644184,mdsmap=4088524775,mgr=911650295,mgr_command_descs=2107673600,mgr_metadata=1367272202,mgrstat=2809615176,mon_config_key=2042193153,monmap=838153876,osd_metadata=4182959540,osd_pg_creating=783066558,osd_snap=3292740240}) 2024-09-16T04:10:44.369 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.363+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=50,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=10,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=5} crc {logm=2643621653,mdsmap=4088524775,mgr=911650295,mgr_command_descs=2107673600,mgr_metadata=1367272202,mgrstat=2217028545,mon_config_key=3448559827,monmap=3937513981,osd_metadata=4182959540,osd_pg_creating=783066558,osd_snap=3292740240}) 2024-09-16T04:10:44.369 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.363+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.369 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.363+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=52,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=9,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=5} crc {logm=898644184,mdsmap=4088524775,mgr=911650295,mgr_command_descs=2107673600,mgr_metadata=1367272202,mgrstat=2809615176,mon_config_key=2042193153,monmap=838153876,osd_metadata=4182959540,osd_pg_creating=783066558,osd_snap=3292740240}) 2024-09-16T04:10:44.369 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.363+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=53,mdsmap=2,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=9,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=5} crc {logm=1567436574,mdsmap=3140514009,mgr=1669371261,mgr_command_descs=2107673600,mgr_metadata=1367272202,mgrstat=2809615176,mon_config_key=1536518401,monmap=3937513981,osd_metadata=4182959540,osd_pg_creating=783066558,osd_snap=2373634642}) 2024-09-16T04:10:44.371 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.367+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.371 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.367+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=37,osdmap=63} crc {osd_snap=3863209448,osdmap=4257447884}) 2024-09-16T04:10:44.371 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.367+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=36,osdmap=64} crc {osd_snap=858138823,osdmap=3399979444}) 2024-09-16T04:10:44.371 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.367+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.371 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.367+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=37,osdmap=63} crc {osd_snap=3863209448,osdmap=4257447884}) 2024-09-16T04:10:44.371 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.367+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=38,osdmap=62} crc {osd_snap=1629907035,osdmap=3488923748}) 2024-09-16T04:10:44.373 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.371+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.373 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.371+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3579403577}) 2024-09-16T04:10:44.373 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.371+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=868631541}) 2024-09-16T04:10:44.373 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.371+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.373 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.371+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3579403577}) 2024-09-16T04:10:44.373 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.371+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2848676438}) 2024-09-16T04:10:44.376 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.371+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.376 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.371+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=450429384}) 2024-09-16T04:10:44.376 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.371+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3621325782}) 2024-09-16T04:10:44.376 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.371+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.376 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.371+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=450429384}) 2024-09-16T04:10:44.376 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.371+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1758646211}) 2024-09-16T04:10:44.381 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.375+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.381 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.375+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=60104018}) 2024-09-16T04:10:44.381 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.375+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1459786576}) 2024-09-16T04:10:44.381 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.375+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.381 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.375+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=60104018}) 2024-09-16T04:10:44.381 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.375+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=820880604}) 2024-09-16T04:10:44.397 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.391+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.397 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.391+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=23} crc {osdmap=3157152194}) 2024-09-16T04:10:44.397 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.391+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=23} crc {osdmap=1985997132}) 2024-09-16T04:10:44.397 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.391+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:10:44.397 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.391+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=23} crc {osdmap=3157152194}) 2024-09-16T04:10:44.397 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:10:44.391+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=21} crc {osdmap=2403019611}) 2024-09-16T04:10:45.067 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T04:10:45.067 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:10:45.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.199+0000 7fbfd6f20640 10 monclient: get_monmap_and_config 2024-09-16T04:10:45.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.199+0000 7fbfd6f20640 10 monclient: build_initial_monmap 2024-09-16T04:10:45.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.199+0000 7fbfd6f20640 10 monclient: monmap: 2024-09-16T04:10:45.203 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:10:45.203 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:10:45.203 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:10:45.201566+0000 2024-09-16T04:10:45.203 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:10:45.201566+0000 2024-09-16T04:10:45.203 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:10:45.203 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:10:45.203 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:10:45.203 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:10:45.203 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:10:45.204 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:10:45.204 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:10:45.204 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:10:45.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.199+0000 7fbfd6f20640 1 Processor -- start 2024-09-16T04:10:45.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.199+0000 7fbfd6f20640 1 -- start start 2024-09-16T04:10:45.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.199+0000 7fbfd6f20640 10 monclient: init 2024-09-16T04:10:45.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.199+0000 7fbfd6f20640 10 monclient: _reopen_session rank -1 2024-09-16T04:10:45.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.199+0000 7fbfd6f20640 10 monclient: _add_conns ranks=[4,3,2,0,1] 2024-09-16T04:10:45.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.199+0000 7fbfd6f20640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fbfd00a39b0 0x7fbfd0133b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:10:45.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.199+0000 7fbfd6f20640 10 monclient(hunting): picked mon.noname-c con 0x7fbfd00a39b0 addr v2:172.21.15.154:3300/0 2024-09-16T04:10:45.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.199+0000 7fbfd6f20640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fbfd00a3d90 0x7fbfd0137000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:10:45.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.199+0000 7fbfd6f20640 10 monclient(hunting): picked mon.noname-a con 0x7fbfd00a3d90 addr v2:172.21.15.42:3300/0 2024-09-16T04:10:45.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.199+0000 7fbfd6f20640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbfd0137540 0x7fbfd0137990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:10:45.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.199+0000 7fbfd6f20640 10 monclient(hunting): picked mon.noname-b con 0x7fbfd0137540 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:10:45.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.199+0000 7fbfd6f20640 10 monclient(hunting): start opening mon connection 2024-09-16T04:10:45.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.199+0000 7fbfd6f20640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7fbfd009c140 con 0x7fbfd00a3d90 2024-09-16T04:10:45.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.199+0000 7fbfd6f20640 10 monclient(hunting): start opening mon connection 2024-09-16T04:10:45.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.199+0000 7fbfd6f20640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fbfd009c420 con 0x7fbfd00a39b0 2024-09-16T04:10:45.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.199+0000 7fbfd6f20640 10 monclient(hunting): start opening mon connection 2024-09-16T04:10:45.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.199+0000 7fbfd6f20640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fbfd006dd60 con 0x7fbfd0137540 2024-09-16T04:10:45.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.199+0000 7fbfd6f20640 10 monclient(hunting): _renew_subs 2024-09-16T04:10:45.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.199+0000 7fbfd6f20640 10 monclient(hunting): authenticate will time out at 2692.171019s 2024-09-16T04:10:45.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.199+0000 7fbfd54bd640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbfd0137540 0x7fbfd0137990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:10:45.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.199+0000 7fbfcffff640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fbfd00a3d90 0x7fbfd0137000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:10:45.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.199+0000 7fbfd54bd640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbfd0137540 0x7fbfd0137990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:43232/0 (socket says 172.21.15.42:43232) 2024-09-16T04:10:45.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.199+0000 7fbfd54bd640 1 -- 172.21.15.42:0/3561460025 learned_addr learned my addr 172.21.15.42:0/3561460025 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:10:45.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.199+0000 7fbfd54bd640 10 monclient(hunting): get_auth_request con 0x7fbfd0137540 auth_method 0 2024-09-16T04:10:45.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.199+0000 7fbfd54bd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:10:45.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.199+0000 7fbfd54bd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:45.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.199+0000 7fbfd54bd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:45.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.199+0000 7fbfcffff640 10 monclient(hunting): get_auth_request con 0x7fbfd00a3d90 auth_method 0 2024-09-16T04:10:45.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.199+0000 7fbfcffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:10:45.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.199+0000 7fbfcffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:45.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.199+0000 7fbfcffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:45.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.199+0000 7fbfd54bd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:10:45.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.199+0000 7fbfd54bd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:10:45.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.199+0000 7fbfd54bd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:10:45.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.199+0000 7fbfcffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:10:45.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.199+0000 7fbfcffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:10:45.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.199+0000 7fbfcffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:10:45.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.199+0000 7fbfd4cbc640 1 --2- 172.21.15.42:0/3561460025 >> v2:172.21.15.154:3300/0 conn(0x7fbfd00a39b0 0x7fbfd0133b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:10:45.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.199+0000 7fbfd54bd640 10 monclient(hunting): handle_auth_done global_id 1014461 payload 274 2024-09-16T04:10:45.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.199+0000 7fbfd54bd640 1 -- 172.21.15.42:0/3561460025 >> v2:172.21.15.154:3300/0 conn(0x7fbfd00a39b0 msgr2=0x7fbfd0133b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:10:45.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.199+0000 7fbfd54bd640 1 --2- 172.21.15.42:0/3561460025 >> v2:172.21.15.154:3300/0 conn(0x7fbfd00a39b0 0x7fbfd0133b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:10:45.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.199+0000 7fbfd54bd640 1 -- 172.21.15.42:0/3561460025 >> v2:172.21.15.42:3300/0 conn(0x7fbfd00a3d90 msgr2=0x7fbfd0137000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:10:45.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.199+0000 7fbfd54bd640 1 --2- 172.21.15.42:0/3561460025 >> v2:172.21.15.42:3300/0 conn(0x7fbfd00a3d90 0x7fbfd0137000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:10:45.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.199+0000 7fbfd54bd640 10 monclient: _finish_hunting 0 2024-09-16T04:10:45.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.199+0000 7fbfd54bd640 1 monclient: found mon.noname-b 2024-09-16T04:10:45.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.199+0000 7fbfd54bd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:10:45.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.199+0000 7fbfd54bd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T04:10:45.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.199+0000 7fbfd54bd640 1 -- 172.21.15.42:0/3561460025 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfd00a5e20 con 0x7fbfd0137540 2024-09-16T04:10:45.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.199+0000 7fbfcffff640 1 --2- 172.21.15.42:0/3561460025 >> v2:172.21.15.42:3300/0 conn(0x7fbfd00a3d90 0x7fbfd0137000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:10:45.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.199+0000 7fbfd4cbc640 10 monclient: get_auth_request con 0x7fbfd00a39b0 auth_method 0 2024-09-16T04:10:45.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.199+0000 7fbfd4cbc640 1 --2- 172.21.15.42:0/3561460025 >> v2:172.21.15.154:3300/0 conn(0x7fbfd00a39b0 0x7fbfd0133b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T04:10:45.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.203+0000 7fbfd54bd640 1 --2- 172.21.15.42:0/3561460025 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbfd0137540 0x7fbfd0137990 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fbfd00a7750 tx=0x7fbfc801c8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7d80b7cca544d13d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:10:45.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.203+0000 7fbfcf7fe640 1 -- 172.21.15.42:0/3561460025 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfc80a5530 con 0x7fbfd0137540 2024-09-16T04:10:45.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.203+0000 7fbfcf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:45.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.203+0000 7fbfcf7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:10:45.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.203+0000 7fbfcf7fe640 10 monclient: dump: 2024-09-16T04:10:45.216 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:10:45.216 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:10:45.216 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:45.217 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:45.217 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:45.217 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:10:45.217 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:10:45.217 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:10:45.217 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:10:45.217 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:10:45.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.203+0000 7fbfcf7fe640 10 monclient: _finish_auth 0 2024-09-16T04:10:45.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.203+0000 7fbfcf7fe640 10 monclient: _check_auth_tickets 2024-09-16T04:10:45.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.203+0000 7fbfcf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:45.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.203+0000 7fbfcf7fe640 1 -- 172.21.15.42:0/3561460025 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbfc80a56d0 con 0x7fbfd0137540 2024-09-16T04:10:45.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.203+0000 7fbfd6f20640 5 monclient: authenticate success, global_id 1014461 2024-09-16T04:10:45.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.203+0000 7fbfd6f20640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:10:45.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.203+0000 7fbfcf7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:10:45.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.203+0000 7fbfd6f20640 10 monclient: get_monmap_and_config success 2024-09-16T04:10:45.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.203+0000 7fbfcf7fe640 1 -- 172.21.15.42:0/3561460025 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfc80a5aa0 con 0x7fbfd0137540 2024-09-16T04:10:45.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.203+0000 7fbfd6f20640 4 set_mon_vals no callback set 2024-09-16T04:10:45.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.203+0000 7fbfcf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:45.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.203+0000 7fbfcf7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:10:45.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.203+0000 7fbfcf7fe640 10 monclient: dump: 2024-09-16T04:10:45.218 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:10:45.218 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:10:45.218 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:45.218 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:45.218 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:45.218 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:10:45.218 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:10:45.218 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:10:45.219 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:10:45.219 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:10:45.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.203+0000 7fbfd6f20640 10 monclient: shutdown 2024-09-16T04:10:45.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.203+0000 7fbfd6f20640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:10:45.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.203+0000 7fbfd6f20640 1 -- 172.21.15.42:0/3561460025 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbfd0137540 msgr2=0x7fbfd0137990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:45.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.203+0000 7fbfd6f20640 1 --2- 172.21.15.42:0/3561460025 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbfd0137540 0x7fbfd0137990 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fbfd00a7750 tx=0x7fbfc801c8b0 comp rx=0 tx=0).stop 2024-09-16T04:10:45.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.203+0000 7fbfd6f20640 1 -- 172.21.15.42:0/3561460025 shutdown_connections 2024-09-16T04:10:45.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.203+0000 7fbfd6f20640 1 --2- 172.21.15.42:0/3561460025 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbfd0137540 0x7fbfd0137990 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:10:45.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.203+0000 7fbfd6f20640 1 --2- 172.21.15.42:0/3561460025 >> v2:172.21.15.42:3300/0 conn(0x7fbfd00a3d90 0x7fbfd0137000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:10:45.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.203+0000 7fbfd6f20640 1 --2- 172.21.15.42:0/3561460025 >> v2:172.21.15.154:3300/0 conn(0x7fbfd00a39b0 0x7fbfd0133b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:10:45.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.203+0000 7fbfd6f20640 1 -- 172.21.15.42:0/3561460025 >> 172.21.15.42:0/3561460025 conn(0x7fbfd00a9fb0 msgr2=0x7fbfd00aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:10:45.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.203+0000 7fbfd6f20640 1 -- 172.21.15.42:0/3561460025 shutdown_connections 2024-09-16T04:10:45.220 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.203+0000 7fbfd6f20640 1 -- 172.21.15.42:0/3561460025 wait complete. 2024-09-16T04:10:45.220 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.203+0000 7fbfd6f20640 10 monclient: build_initial_monmap 2024-09-16T04:10:45.220 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.203+0000 7fbfd6f20640 10 monclient: monmap: 2024-09-16T04:10:45.220 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:10:45.220 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:10:45.220 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:10:45.220 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:10:45.220 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:10:45.220 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:10:45.220 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:10:45.220 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:10:45.220 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:10:45.220 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:10:45.221 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.203+0000 7fbfd6f20640 1 librados: starting msgr at 2024-09-16T04:10:45.221 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.203+0000 7fbfd6f20640 1 librados: starting objecter 2024-09-16T04:10:45.221 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.203+0000 7fbfd6f20640 1 Processor -- start 2024-09-16T04:10:45.221 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.203+0000 7fbfd6f20640 1 -- start start 2024-09-16T04:10:45.221 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.203+0000 7fbfd6f20640 1 librados: setting wanted keys 2024-09-16T04:10:45.221 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.203+0000 7fbfd6f20640 1 librados: calling monclient init 2024-09-16T04:10:45.221 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.203+0000 7fbfd6f20640 10 monclient: init 2024-09-16T04:10:45.221 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.207+0000 7fbfd6f20640 10 monclient: _reopen_session rank -1 2024-09-16T04:10:45.221 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.207+0000 7fbfd6f20640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:10:45.221 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.207+0000 7fbfd6f20640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbfd00a39b0 0x7fbfd013b080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:10:45.221 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.207+0000 7fbfd6f20640 10 monclient(hunting): picked mon.noname-c con 0x7fbfd00a39b0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:10:45.221 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.207+0000 7fbfd6f20640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbfd00a3d90 0x7fbfd013b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:10:45.221 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.207+0000 7fbfd6f20640 10 monclient(hunting): picked mon.noname-a con 0x7fbfd00a3d90 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:10:45.221 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.207+0000 7fbfd6f20640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbfd0137540 0x7fbfd013bb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:10:45.222 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.207+0000 7fbfd6f20640 10 monclient(hunting): picked mon.noname-b con 0x7fbfd0137540 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:10:45.222 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.207+0000 7fbfd6f20640 10 monclient(hunting): start opening mon connection 2024-09-16T04:10:45.222 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.207+0000 7fbfd6f20640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fbfd0140dc0 con 0x7fbfd00a3d90 2024-09-16T04:10:45.222 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.207+0000 7fbfd6f20640 10 monclient(hunting): start opening mon connection 2024-09-16T04:10:45.222 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.207+0000 7fbfd6f20640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fbfd0140f40 con 0x7fbfd0137540 2024-09-16T04:10:45.222 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.207+0000 7fbfd6f20640 10 monclient(hunting): start opening mon connection 2024-09-16T04:10:45.222 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.207+0000 7fbfd6f20640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fbfd009d5a0 con 0x7fbfd00a39b0 2024-09-16T04:10:45.222 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.207+0000 7fbfd6f20640 10 monclient(hunting): _renew_subs 2024-09-16T04:10:45.222 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.207+0000 7fbfd6f20640 10 monclient(hunting): authenticate will time out at 2692.176243s 2024-09-16T04:10:45.222 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.207+0000 7fbfcffff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbfd00a3d90 0x7fbfd013b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:10:45.222 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.207+0000 7fbfd4cbc640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbfd00a39b0 0x7fbfd013b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:10:45.222 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.207+0000 7fbfcffff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbfd00a3d90 0x7fbfd013b5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:52778/0 (socket says 172.21.15.42:52778) 2024-09-16T04:10:45.222 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.207+0000 7fbfd4cbc640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbfd00a39b0 0x7fbfd013b080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:43234/0 (socket says 172.21.15.42:43234) 2024-09-16T04:10:45.222 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.207+0000 7fbfd4cbc640 1 -- 172.21.15.42:0/3023743159 learned_addr learned my addr 172.21.15.42:0/3023743159 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:10:45.223 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.207+0000 7fbfd4cbc640 10 monclient(hunting): get_auth_request con 0x7fbfd00a39b0 auth_method 0 2024-09-16T04:10:45.223 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.207+0000 7fbfd4cbc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:10:45.223 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.207+0000 7fbfd4cbc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:45.223 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.207+0000 7fbfd4cbc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:45.223 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.207+0000 7fbfcffff640 10 monclient(hunting): get_auth_request con 0x7fbfd00a3d90 auth_method 0 2024-09-16T04:10:45.223 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.207+0000 7fbfcffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:10:45.223 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.207+0000 7fbfcffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:45.223 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.207+0000 7fbfcffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:45.223 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.207+0000 7fbfd4cbc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:10:45.223 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.207+0000 7fbfd4cbc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:10:45.223 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.207+0000 7fbfd4cbc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:10:45.223 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.207+0000 7fbfcffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:10:45.223 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.207+0000 7fbfcffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:10:45.224 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.207+0000 7fbfcffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:10:45.224 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.207+0000 7fbfd4cbc640 10 monclient(hunting): handle_auth_done global_id 1014464 payload 824 2024-09-16T04:10:45.224 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.207+0000 7fbfd54bd640 1 --2- 172.21.15.42:0/3023743159 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbfd0137540 0x7fbfd013bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:10:45.224 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.207+0000 7fbfd4cbc640 1 -- 172.21.15.42:0/3023743159 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbfd0137540 msgr2=0x7fbfd013bb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:45.224 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.207+0000 7fbfd4cbc640 1 --2- 172.21.15.42:0/3023743159 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbfd0137540 0x7fbfd013bb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:10:45.224 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.207+0000 7fbfd4cbc640 1 -- 172.21.15.42:0/3023743159 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbfd00a3d90 msgr2=0x7fbfd013b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:45.224 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.207+0000 7fbfd4cbc640 1 --2- 172.21.15.42:0/3023743159 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbfd00a3d90 0x7fbfd013b5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:10:45.224 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.207+0000 7fbfd4cbc640 10 monclient: _finish_hunting 0 2024-09-16T04:10:45.224 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.207+0000 7fbfd4cbc640 1 monclient: found mon.noname-c 2024-09-16T04:10:45.224 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.207+0000 7fbfd4cbc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:10:45.224 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.207+0000 7fbfd4cbc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T04:10:45.224 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.207+0000 7fbfd4cbc640 1 -- 172.21.15.42:0/3023743159 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfd0132bb0 con 0x7fbfd00a39b0 2024-09-16T04:10:45.224 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.207+0000 7fbfcffff640 1 --2- 172.21.15.42:0/3023743159 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbfd00a3d90 0x7fbfd013b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:10:45.224 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.207+0000 7fbfd4cbc640 1 --2- 172.21.15.42:0/3023743159 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbfd00a39b0 0x7fbfd013b080 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fbfc000c7f0 tx=0x7fbfc000ccb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:10:45.225 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.207+0000 7fbfcd7fa640 10 client.?.objecter ms_handle_connect 0x7fbfd00a39b0 2024-09-16T04:10:45.225 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.207+0000 7fbfcd7fa640 10 client.?.objecter resend_mon_ops 2024-09-16T04:10:45.225 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.207+0000 7fbfcd7fa640 1 -- 172.21.15.42:0/3023743159 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfc0007a30 con 0x7fbfd00a39b0 2024-09-16T04:10:45.225 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.207+0000 7fbfcd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:45.225 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.207+0000 7fbfcd7fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:10:45.225 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.207+0000 7fbfcd7fa640 10 monclient: dump: 2024-09-16T04:10:45.225 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:10:45.225 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:10:45.225 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:45.225 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:45.225 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:45.225 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:10:45.225 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:10:45.225 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:10:45.226 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:10:45.226 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:10:45.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.207+0000 7fbfcd7fa640 10 monclient: _finish_auth 0 2024-09-16T04:10:45.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.207+0000 7fbfcd7fa640 10 monclient: _check_auth_tickets 2024-09-16T04:10:45.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.207+0000 7fbfcd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:45.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.207+0000 7fbfcd7fa640 1 -- 172.21.15.42:0/3023743159 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbfc0007bd0 con 0x7fbfd00a39b0 2024-09-16T04:10:45.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.207+0000 7fbfd6f20640 5 monclient: authenticate success, global_id 1014464 2024-09-16T04:10:45.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.207+0000 7fbfd6f20640 10 monclient: _renew_subs 2024-09-16T04:10:45.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.207+0000 7fbfd6f20640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:10:45.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.207+0000 7fbfd6f20640 1 -- 172.21.15.42:0/3023743159 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbfd00a4fb0 con 0x7fbfd00a39b0 2024-09-16T04:10:45.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.207+0000 7fbfcd7fa640 10 monclient: handle_config config(0 keys) 2024-09-16T04:10:45.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.207+0000 7fbfcd7fa640 1 -- 172.21.15.42:0/3023743159 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfc00113f0 con 0x7fbfd00a39b0 2024-09-16T04:10:45.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.207+0000 7fbfcd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:45.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.207+0000 7fbfcd7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:10:45.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.207+0000 7fbfcd7fa640 10 monclient: dump: 2024-09-16T04:10:45.227 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:10:45.227 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:10:45.227 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:45.227 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:45.227 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:45.227 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:10:45.227 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:10:45.227 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:10:45.227 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:10:45.227 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:10:45.227 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.207+0000 7fbfce7fc640 4 set_mon_vals no callback set 2024-09-16T04:10:45.227 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.207+0000 7fbfd6f20640 10 client.1014464.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:10:45.227 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.207+0000 7fbfd6f20640 10 monclient: _renew_subs 2024-09-16T04:10:45.227 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.207+0000 7fbfd6f20640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:10:45.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.207+0000 7fbfd6f20640 1 -- 172.21.15.42:0/3023743159 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fbfd009cd00 con 0x7fbfd00a39b0 2024-09-16T04:10:45.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.207+0000 7fbfd6f20640 1 librados: init done 2024-09-16T04:10:45.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.207+0000 7fbfd6f20640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:45.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.207+0000 7fbfd6f20640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:45.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.207+0000 7fbfd6f20640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:10:45.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.207+0000 7fbfd6f20640 1 -- 172.21.15.42:0/3023743159 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbfd0137000 con 0x7fbfd00a39b0 2024-09-16T04:10:45.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.207+0000 7fbfcd7fa640 1 -- 172.21.15.42:0/3023743159 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fbfc0011590 con 0x7fbfd00a39b0 2024-09-16T04:10:45.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.207+0000 7fbfcd7fa640 1 --2- 172.21.15.42:0/3023743159 >> v2:172.21.15.154:6812/254845342 conn(0x7fbfb0037810 0x7fbfb0039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:10:45.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.207+0000 7fbfcd7fa640 1 -- 172.21.15.42:0/3023743159 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(706..706 src has 451..706) ==== 5040+0+0 (secure 0 0 0) 0x7fbfc0048e50 con 0x7fbfd00a39b0 2024-09-16T04:10:45.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.207+0000 7fbfcd7fa640 10 client.1014464.objecter ms_dispatch 0x7fbfd0000ba0 osd_map(706..706 src has 451..706) 2024-09-16T04:10:45.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.207+0000 7fbfcd7fa640 3 client.1014464.objecter handle_osd_map got epochs [706,706] > 0 2024-09-16T04:10:45.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.207+0000 7fbfcd7fa640 3 client.1014464.objecter handle_osd_map decoding full epoch 706 2024-09-16T04:10:45.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.207+0000 7fbfcd7fa640 20 client.1014464.objecter dump_active .. 0 homeless 2024-09-16T04:10:45.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.207+0000 7fbfcffff640 1 --2- 172.21.15.42:0/3023743159 >> v2:172.21.15.154:6812/254845342 conn(0x7fbfb0037810 0x7fbfb0039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:10:45.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.207+0000 7fbfcffff640 10 monclient: get_auth_request con 0x7fbfb0037810 auth_method 0 2024-09-16T04:10:45.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.207+0000 7fbfcd7fa640 10 client.1014464.objecter ms_handle_connect 0x7fbfb0037810 2024-09-16T04:10:45.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.207+0000 7fbfcffff640 1 --2- 172.21.15.42:0/3023743159 >> v2:172.21.15.154:6812/254845342 conn(0x7fbfb0037810 0x7fbfb0039cd0 secure :-1 s=READY pgs=3938 cs=0 l=1 rev1=1 crypto rx=0x7fbfb8009580 tx=0x7fbfb8009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:10:45.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.211+0000 7fbfcd7fa640 1 -- 172.21.15.42:0/3023743159 <== mon.2 v2:172.21.15.42:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7fbfc0015020 con 0x7fbfd00a39b0 2024-09-16T04:10:45.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.211+0000 7fbfcd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:45.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.211+0000 7fbfcd7fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:10:45.450 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.447+0000 7fbfd6f20640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:10:45.450 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.447+0000 7fbfd6f20640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:10:45.450 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.447+0000 7fbfd6f20640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:10:45.450 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.447+0000 7fbfd6f20640 1 -- 172.21.15.42:0/3023743159 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fbfd00a5750 con 0x7fbfd00a39b0 2024-09-16T04:10:45.450 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.447+0000 7fbfcd7fa640 1 -- 172.21.15.42:0/3023743159 <== mon.2 v2:172.21.15.42:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7fbfc0020700 con 0x7fbfd00a39b0 2024-09-16T04:10:45.450 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.447+0000 7fbfcd7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:10:45.450 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.447+0000 7fbfcd7fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:10:45.451 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":262,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:10:45.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.447+0000 7fbfd6f20640 10 librados: watch_flush enter 2024-09-16T04:10:45.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.447+0000 7fbfd6f20640 10 librados: watch_flush exit 2024-09-16T04:10:45.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.447+0000 7fbfd6f20640 20 client.1014464.objecter shutdown clearing up homeless session... 2024-09-16T04:10:45.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.447+0000 7fbfd6f20640 10 client.1014464.objecter successfully canceled tick 2024-09-16T04:10:45.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.447+0000 7fbfd6f20640 1 -- 172.21.15.42:0/3023743159 >> v2:172.21.15.154:6812/254845342 conn(0x7fbfb0037810 msgr2=0x7fbfb0039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:45.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.447+0000 7fbfd6f20640 1 --2- 172.21.15.42:0/3023743159 >> v2:172.21.15.154:6812/254845342 conn(0x7fbfb0037810 0x7fbfb0039cd0 secure :-1 s=READY pgs=3938 cs=0 l=1 rev1=1 crypto rx=0x7fbfb8009580 tx=0x7fbfb8009190 comp rx=0 tx=0).stop 2024-09-16T04:10:45.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.447+0000 7fbfd6f20640 10 monclient: shutdown 2024-09-16T04:10:45.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.447+0000 7fbfd6f20640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:10:45.454 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.447+0000 7fbfd6f20640 1 -- 172.21.15.42:0/3023743159 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbfd00a39b0 msgr2=0x7fbfd013b080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:45.454 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.447+0000 7fbfd6f20640 1 --2- 172.21.15.42:0/3023743159 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbfd00a39b0 0x7fbfd013b080 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fbfc000c7f0 tx=0x7fbfc000ccb0 comp rx=0 tx=0).stop 2024-09-16T04:10:45.454 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.447+0000 7fbfd6f20640 1 -- 172.21.15.42:0/3023743159 shutdown_connections 2024-09-16T04:10:45.454 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.447+0000 7fbfd6f20640 1 --2- 172.21.15.42:0/3023743159 >> v2:172.21.15.154:6812/254845342 conn(0x7fbfb0037810 0x7fbfb0039cd0 unknown :-1 s=CLOSED pgs=3938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:10:45.454 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.447+0000 7fbfd6f20640 1 --2- 172.21.15.42:0/3023743159 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbfd0137540 0x7fbfd013bb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:10:45.454 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.447+0000 7fbfd6f20640 1 --2- 172.21.15.42:0/3023743159 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbfd00a3d90 0x7fbfd013b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:10:45.454 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.447+0000 7fbfd6f20640 1 --2- 172.21.15.42:0/3023743159 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbfd00a39b0 0x7fbfd013b080 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:10:45.454 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.447+0000 7fbfd6f20640 1 -- 172.21.15.42:0/3023743159 >> 172.21.15.42:0/3023743159 conn(0x7fbfd00a9fb0 msgr2=0x7fbfd00a8000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:10:45.454 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.447+0000 7fbfd6f20640 1 -- 172.21.15.42:0/3023743159 shutdown_connections 2024-09-16T04:10:45.454 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.447+0000 7fbfd6f20640 1 -- 172.21.15.42:0/3023743159 wait complete. 2024-09-16T04:10:45.454 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.447+0000 7fbfd6f20640 1 librados: shutdown 2024-09-16T04:10:45.470 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T04:10:45.470 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T04:10:45.470 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T04:10:45.470 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T04:10:45.602 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.599+0000 7f40ac3fe640 10 monclient: get_monmap_and_config 2024-09-16T04:10:45.602 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.599+0000 7f40ac3fe640 10 monclient: build_initial_monmap 2024-09-16T04:10:45.602 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.599+0000 7f40ac3fe640 10 monclient: monmap: 2024-09-16T04:10:45.602 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:10:45.602 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:10:45.603 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:10:45.600615+0000 2024-09-16T04:10:45.603 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:10:45.600615+0000 2024-09-16T04:10:45.603 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:10:45.603 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:10:45.603 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:10:45.603 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:10:45.603 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:10:45.603 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:10:45.604 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:10:45.604 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:10:45.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.599+0000 7f40ac3fe640 1 Processor -- start 2024-09-16T04:10:45.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.599+0000 7f40ac3fe640 1 -- start start 2024-09-16T04:10:45.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.599+0000 7f40ac3fe640 10 monclient: init 2024-09-16T04:10:45.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.599+0000 7f40ac3fe640 10 monclient: _reopen_session rank -1 2024-09-16T04:10:45.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.599+0000 7f40ac3fe640 10 monclient: _add_conns ranks=[3,2,1,4,0] 2024-09-16T04:10:45.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.599+0000 7f40ac3fe640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f40a40a39b0 0x7f40a4133b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:10:45.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.599+0000 7f40ac3fe640 10 monclient(hunting): picked mon.noname-a con 0x7f40a40a39b0 addr v2:172.21.15.42:3300/0 2024-09-16T04:10:45.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.599+0000 7f40ac3fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f40a40a3d90 0x7f40a4137000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:10:45.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.599+0000 7f40ac3fe640 10 monclient(hunting): picked mon.noname-b con 0x7f40a40a3d90 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:10:45.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.599+0000 7f40ac3fe640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f40a4137540 addr v1:172.21.15.154:6789/0 2024-09-16T04:10:45.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.599+0000 7f40ac3fe640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40a409c140 con 0x7f40a4137540 2024-09-16T04:10:45.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.599+0000 7f40ac3fe640 10 monclient(hunting): start opening mon connection 2024-09-16T04:10:45.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.599+0000 7f40ac3fe640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f40a409c420 con 0x7f40a40a39b0 2024-09-16T04:10:45.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.599+0000 7f40ac3fe640 10 monclient(hunting): start opening mon connection 2024-09-16T04:10:45.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.599+0000 7f40ac3fe640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f40a406dd60 con 0x7f40a40a3d90 2024-09-16T04:10:45.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.599+0000 7f40ac3fe640 10 monclient(hunting): _renew_subs 2024-09-16T04:10:45.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.599+0000 7f40ac3fe640 10 monclient(hunting): authenticate will time out at 2692.570714s 2024-09-16T04:10:45.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.599+0000 7f40a9999640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f40a40a3d90 0x7f40a4137000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:10:45.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.599+0000 7f40aa19a640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f40a40a39b0 0x7f40a4133b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:10:45.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.599+0000 7f40aa19a640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f40a40a39b0 0x7f40a4133b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:52792/0 (socket says 172.21.15.42:52792) 2024-09-16T04:10:45.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.599+0000 7f40a9999640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f40a40a3d90 0x7f40a4137000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:43244/0 (socket says 172.21.15.42:43244) 2024-09-16T04:10:45.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.599+0000 7f40aa19a640 1 -- 172.21.15.42:0/4227208075 learned_addr learned my addr 172.21.15.42:0/4227208075 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:10:45.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.599+0000 7f40aa19a640 10 monclient(hunting): get_auth_request con 0x7f40a40a39b0 auth_method 0 2024-09-16T04:10:45.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.599+0000 7f40aa19a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:10:45.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.599+0000 7f40aa19a640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:45.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.599+0000 7f40aa19a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:45.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.599+0000 7f40a9999640 10 monclient(hunting): get_auth_request con 0x7f40a40a3d90 auth_method 0 2024-09-16T04:10:45.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.599+0000 7f40a9999640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:10:45.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.599+0000 7f40a9999640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:45.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.599+0000 7f40a9999640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:45.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.599+0000 7f40a9999640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:10:45.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.599+0000 7f40a9999640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:10:45.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.599+0000 7f40a9999640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:10:45.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.599+0000 7f40aa19a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:10:45.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.599+0000 7f40aa19a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:10:45.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.599+0000 7f40aa19a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:10:45.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.599+0000 7f40a9999640 10 monclient(hunting): handle_auth_done global_id 1014467 payload 274 2024-09-16T04:10:45.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.599+0000 7f40a9999640 1 -- 172.21.15.42:0/4227208075 >> v2:172.21.15.42:3300/0 conn(0x7f40a40a39b0 msgr2=0x7f40a4133b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:10:45.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.599+0000 7f40a9999640 1 --2- 172.21.15.42:0/4227208075 >> v2:172.21.15.42:3300/0 conn(0x7f40a40a39b0 0x7f40a4133b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:10:45.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.599+0000 7f40a9999640 1 -- 172.21.15.42:0/4227208075 >> v1:172.21.15.154:6789/0 conn(0x7f40a4137540 legacy=0x7f40a413e390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:10:45.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.599+0000 7f40a9999640 10 monclient: _finish_hunting 0 2024-09-16T04:10:45.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.599+0000 7f40a9999640 1 monclient: found mon.noname-b 2024-09-16T04:10:45.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.599+0000 7f40a9999640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:10:45.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.599+0000 7f40a9999640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T04:10:45.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.599+0000 7f40a9999640 1 -- 172.21.15.42:0/4227208075 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40a40a5e20 con 0x7f40a40a3d90 2024-09-16T04:10:45.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.599+0000 7f40aa19a640 1 --2- 172.21.15.42:0/4227208075 >> v2:172.21.15.42:3300/0 conn(0x7f40a40a39b0 0x7f40a4133b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:10:45.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.603+0000 7f40a9999640 1 --2- 172.21.15.42:0/4227208075 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f40a40a3d90 0x7f40a4137000 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f40940097c0 tx=0x7f409401c320 comp rx=0 tx=0).ready entity=mon.2 client_cookie=79b92e4cf23a8e5c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:10:45.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.603+0000 7f40a9198640 1 -- 172.21.15.42:0/4227208075 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40940b0070 con 0x7f40a40a3d90 2024-09-16T04:10:45.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.603+0000 7f40a9198640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:45.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.603+0000 7f40a9198640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:10:45.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.603+0000 7f40a9198640 10 monclient: dump: 2024-09-16T04:10:45.618 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:10:45.619 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:10:45.619 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:45.619 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:45.619 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:45.619 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:10:45.619 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:10:45.619 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:10:45.619 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:10:45.619 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:10:45.619 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.603+0000 7f40a9198640 10 monclient: _finish_auth 0 2024-09-16T04:10:45.619 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.603+0000 7f40a9198640 10 monclient: _check_auth_tickets 2024-09-16T04:10:45.620 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.603+0000 7f40a9198640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:45.620 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.603+0000 7f40a9198640 1 -- 172.21.15.42:0/4227208075 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40940ab600 con 0x7f40a40a3d90 2024-09-16T04:10:45.620 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.603+0000 7f40a9198640 10 monclient: handle_config config(0 keys) 2024-09-16T04:10:45.620 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.603+0000 7f40a9198640 1 -- 172.21.15.42:0/4227208075 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40940ab9d0 con 0x7f40a40a3d90 2024-09-16T04:10:45.620 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.603+0000 7f40ac3fe640 5 monclient: authenticate success, global_id 1014467 2024-09-16T04:10:45.620 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.603+0000 7f40ac3fe640 10 monclient: get_monmap_and_config success 2024-09-16T04:10:45.621 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.603+0000 7f40ac3fe640 4 set_mon_vals no callback set 2024-09-16T04:10:45.621 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.603+0000 7f40a9198640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:45.621 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.603+0000 7f40a9198640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:10:45.621 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.603+0000 7f40a9198640 10 monclient: dump: 2024-09-16T04:10:45.621 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:10:45.621 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:10:45.621 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:45.621 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:45.621 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:45.621 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:10:45.621 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:10:45.621 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:10:45.621 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:10:45.622 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:10:45.622 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.603+0000 7f40ac3fe640 10 monclient: shutdown 2024-09-16T04:10:45.622 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.603+0000 7f40ac3fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:10:45.622 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.603+0000 7f40ac3fe640 1 -- 172.21.15.42:0/4227208075 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f40a40a3d90 msgr2=0x7f40a4137000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:45.622 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.603+0000 7f40ac3fe640 1 --2- 172.21.15.42:0/4227208075 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f40a40a3d90 0x7f40a4137000 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f40940097c0 tx=0x7f409401c320 comp rx=0 tx=0).stop 2024-09-16T04:10:45.622 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.603+0000 7f40ac3fe640 1 -- 172.21.15.42:0/4227208075 shutdown_connections 2024-09-16T04:10:45.622 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.603+0000 7f40ac3fe640 1 --2- 172.21.15.42:0/4227208075 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f40a40a3d90 0x7f40a4137000 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:10:45.622 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.603+0000 7f40ac3fe640 1 --2- 172.21.15.42:0/4227208075 >> v2:172.21.15.42:3300/0 conn(0x7f40a40a39b0 0x7f40a4133b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:10:45.622 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.603+0000 7f40ac3fe640 1 -- 172.21.15.42:0/4227208075 >> 172.21.15.42:0/4227208075 conn(0x7f40a40a9fb0 msgr2=0x7f40a40aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:10:45.622 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.603+0000 7f40ac3fe640 1 -- 172.21.15.42:0/4227208075 shutdown_connections 2024-09-16T04:10:45.622 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.603+0000 7f40ac3fe640 1 -- 172.21.15.42:0/4227208075 wait complete. 2024-09-16T04:10:45.622 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.603+0000 7f40ac3fe640 10 monclient: build_initial_monmap 2024-09-16T04:10:45.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.603+0000 7f40ac3fe640 10 monclient: monmap: 2024-09-16T04:10:45.623 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:10:45.623 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:10:45.623 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:10:45.623 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:10:45.623 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:10:45.623 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:10:45.623 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:10:45.623 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:10:45.623 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:10:45.623 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:10:45.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.603+0000 7f40ac3fe640 1 librados: starting msgr at 2024-09-16T04:10:45.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.603+0000 7f40ac3fe640 1 librados: starting objecter 2024-09-16T04:10:45.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.603+0000 7f40ac3fe640 1 Processor -- start 2024-09-16T04:10:45.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.603+0000 7f40ac3fe640 1 -- start start 2024-09-16T04:10:45.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.603+0000 7f40ac3fe640 1 librados: setting wanted keys 2024-09-16T04:10:45.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.603+0000 7f40ac3fe640 1 librados: calling monclient init 2024-09-16T04:10:45.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.603+0000 7f40ac3fe640 10 monclient: init 2024-09-16T04:10:45.625 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.603+0000 7f40ac3fe640 10 monclient: _reopen_session rank -1 2024-09-16T04:10:45.625 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.603+0000 7f40ac3fe640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:10:45.625 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.603+0000 7f40ac3fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f40a40a39b0 0x7f40a40a19d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:10:45.625 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.603+0000 7f40ac3fe640 10 monclient(hunting): picked mon.noname-a con 0x7f40a40a39b0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:10:45.625 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.603+0000 7f40ac3fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f40a40a3d90 0x7f40a40a1f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:10:45.625 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.603+0000 7f40ac3fe640 10 monclient(hunting): picked mon.noname-c con 0x7f40a40a3d90 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:10:45.625 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.603+0000 7f40ac3fe640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f40a4137540 0x7f40a40a2450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:10:45.626 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.603+0000 7f40ac3fe640 10 monclient(hunting): picked mon.noname-b con 0x7f40a4137540 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:10:45.626 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.603+0000 7f40ac3fe640 10 monclient(hunting): start opening mon connection 2024-09-16T04:10:45.626 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.603+0000 7f40ac3fe640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f40a41424e0 con 0x7f40a40a39b0 2024-09-16T04:10:45.626 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.603+0000 7f40ac3fe640 10 monclient(hunting): start opening mon connection 2024-09-16T04:10:45.626 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.603+0000 7f40ac3fe640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f40a4142660 con 0x7f40a4137540 2024-09-16T04:10:45.627 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.603+0000 7f40ac3fe640 10 monclient(hunting): start opening mon connection 2024-09-16T04:10:45.627 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.603+0000 7f40ac3fe640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f40a41427e0 con 0x7f40a40a3d90 2024-09-16T04:10:45.627 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.603+0000 7f40ac3fe640 10 monclient(hunting): _renew_subs 2024-09-16T04:10:45.627 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.603+0000 7f40ac3fe640 10 monclient(hunting): authenticate will time out at 2692.575899s 2024-09-16T04:10:45.627 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.607+0000 7f40a9999640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f40a40a3d90 0x7f40a40a1f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:10:45.627 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.607+0000 7f40aa19a640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f40a40a39b0 0x7f40a40a19d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:10:45.628 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.607+0000 7f40a9999640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f40a40a3d90 0x7f40a40a1f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:43260/0 (socket says 172.21.15.42:43260) 2024-09-16T04:10:45.628 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.607+0000 7f40aa19a640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f40a40a39b0 0x7f40a40a19d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:52806/0 (socket says 172.21.15.42:52806) 2024-09-16T04:10:45.628 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.607+0000 7f40a9999640 1 -- 172.21.15.42:0/1483818829 learned_addr learned my addr 172.21.15.42:0/1483818829 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:10:45.628 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.607+0000 7f40a9999640 10 monclient(hunting): get_auth_request con 0x7f40a40a3d90 auth_method 0 2024-09-16T04:10:45.628 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.607+0000 7f40a9999640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:10:45.628 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.607+0000 7f40a9999640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:45.629 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.607+0000 7f40a9999640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:45.629 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.607+0000 7f40aa19a640 10 monclient(hunting): get_auth_request con 0x7f40a40a39b0 auth_method 0 2024-09-16T04:10:45.629 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.607+0000 7f40aa19a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:10:45.629 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.607+0000 7f40aa19a640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:45.629 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.607+0000 7f40aa19a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:45.630 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.607+0000 7f40a9999640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:10:45.630 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.607+0000 7f40a9999640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:10:45.631 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.607+0000 7f40a9999640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:10:45.631 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.607+0000 7f40aa19a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:10:45.631 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.607+0000 7f40aa19a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:10:45.631 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.607+0000 7f40aa19a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:10:45.632 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.607+0000 7f40a9999640 10 monclient(hunting): handle_auth_done global_id 1014470 payload 824 2024-09-16T04:10:45.632 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.607+0000 7f40a9999640 1 -- 172.21.15.42:0/1483818829 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f40a4137540 msgr2=0x7f40a40a2450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:45.632 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.607+0000 7f40aa99b640 1 --2- 172.21.15.42:0/1483818829 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f40a4137540 0x7f40a40a2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:10:45.632 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.607+0000 7f40a9999640 1 --2- 172.21.15.42:0/1483818829 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f40a4137540 0x7f40a40a2450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:10:45.632 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.607+0000 7f40a9999640 1 -- 172.21.15.42:0/1483818829 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f40a40a39b0 msgr2=0x7f40a40a19d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:45.632 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.607+0000 7f40a9999640 1 --2- 172.21.15.42:0/1483818829 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f40a40a39b0 0x7f40a40a19d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:10:45.633 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.607+0000 7f40a9999640 10 monclient: _finish_hunting 0 2024-09-16T04:10:45.633 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.607+0000 7f40a9999640 1 monclient: found mon.noname-c 2024-09-16T04:10:45.633 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.607+0000 7f40a9999640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:10:45.633 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.607+0000 7f40a9999640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T04:10:45.633 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.607+0000 7f40a9999640 1 -- 172.21.15.42:0/1483818829 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40a4139e30 con 0x7f40a40a3d90 2024-09-16T04:10:45.634 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.607+0000 7f40aa99b640 10 monclient: get_auth_request con 0x7f40a4137540 auth_method 0 2024-09-16T04:10:45.634 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.607+0000 7f40aa19a640 1 --2- 172.21.15.42:0/1483818829 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f40a40a39b0 0x7f40a40a19d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:10:45.634 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.607+0000 7f40aa99b640 1 --2- 172.21.15.42:0/1483818829 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f40a4137540 0x7f40a40a2450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T04:10:45.634 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.607+0000 7f40a9999640 1 --2- 172.21.15.42:0/1483818829 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f40a40a3d90 0x7f40a40a1f10 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f409401a9d0 tx=0x7f40940190c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:10:45.634 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.607+0000 7f409affd640 10 client.?.objecter ms_handle_connect 0x7f40a40a3d90 2024-09-16T04:10:45.634 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.607+0000 7f409affd640 10 client.?.objecter resend_mon_ops 2024-09-16T04:10:45.635 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.607+0000 7f409affd640 1 -- 172.21.15.42:0/1483818829 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40940b8020 con 0x7f40a40a3d90 2024-09-16T04:10:45.635 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.607+0000 7f409affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:45.635 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.607+0000 7f409affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:10:45.635 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.607+0000 7f409affd640 10 monclient: dump: 2024-09-16T04:10:45.635 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:10:45.636 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:10:45.636 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:45.636 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:45.636 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:45.636 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:10:45.636 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:10:45.637 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:10:45.637 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:10:45.637 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:10:45.637 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.607+0000 7f409affd640 10 monclient: _finish_auth 0 2024-09-16T04:10:45.637 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.607+0000 7f409affd640 10 monclient: _check_auth_tickets 2024-09-16T04:10:45.638 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.607+0000 7f409affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:45.638 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.607+0000 7f409affd640 1 -- 172.21.15.42:0/1483818829 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40940b4410 con 0x7f40a40a3d90 2024-09-16T04:10:45.638 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.607+0000 7f409affd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:10:45.638 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.607+0000 7f409affd640 1 -- 172.21.15.42:0/1483818829 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40940b0040 con 0x7f40a40a3d90 2024-09-16T04:10:45.638 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.607+0000 7f409affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:45.639 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.607+0000 7f409affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:10:45.639 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.607+0000 7f409affd640 10 monclient: dump: 2024-09-16T04:10:45.639 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:10:45.639 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:10:45.640 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:45.640 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:45.640 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:45.640 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:10:45.640 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:10:45.640 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:10:45.640 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:10:45.640 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:10:45.641 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.607+0000 7f40a8997640 4 set_mon_vals no callback set 2024-09-16T04:10:45.641 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.607+0000 7f40ac3fe640 5 monclient: authenticate success, global_id 1014470 2024-09-16T04:10:45.641 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.607+0000 7f40ac3fe640 10 monclient: _renew_subs 2024-09-16T04:10:45.641 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.607+0000 7f40ac3fe640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:10:45.641 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.607+0000 7f40ac3fe640 1 -- 172.21.15.42:0/1483818829 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40a41332c0 con 0x7f40a40a3d90 2024-09-16T04:10:45.641 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.607+0000 7f40ac3fe640 10 client.1014470.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:10:45.641 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.607+0000 7f40ac3fe640 10 monclient: _renew_subs 2024-09-16T04:10:45.641 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.607+0000 7f40ac3fe640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:10:45.641 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.607+0000 7f40ac3fe640 1 -- 172.21.15.42:0/1483818829 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f40a406f680 con 0x7f40a40a3d90 2024-09-16T04:10:45.641 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.607+0000 7f40ac3fe640 1 librados: init done 2024-09-16T04:10:45.641 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.607+0000 7f40ac3fe640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:45.641 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.607+0000 7f40ac3fe640 1 --2- 172.21.15.42:0/1483818829 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f40a4139870 0x7f40a41d0140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:10:45.641 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.607+0000 7f40ac3fe640 10 monclient(hunting): start opening mon connection 2024-09-16T04:10:45.642 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.607+0000 7f40ac3fe640 1 -- 172.21.15.42:0/1483818829 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f40a41342d0 con 0x7f40a4139870 2024-09-16T04:10:45.642 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.607+0000 7f409affd640 1 -- 172.21.15.42:0/1483818829 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f40940bb020 con 0x7f40a40a3d90 2024-09-16T04:10:45.642 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.607+0000 7f40aa19a640 1 --2- 172.21.15.42:0/1483818829 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f40a4139870 0x7f40a41d0140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:10:45.642 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.607+0000 7f40aa19a640 10 monclient: get_auth_request con 0x7f40a4139870 auth_method 0 2024-09-16T04:10:45.642 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.607+0000 7f40aa19a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:10:45.642 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.607+0000 7f40aa19a640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:45.642 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.607+0000 7f40aa19a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:45.642 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.607+0000 7f40aa19a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:10:45.642 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.607+0000 7f40aa19a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:10:45.642 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.607+0000 7f40aa19a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:10:45.642 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.607+0000 7f409affd640 1 --2- 172.21.15.42:0/1483818829 >> v2:172.21.15.154:6812/254845342 conn(0x7f4088037810 0x7f4088039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:10:45.642 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.607+0000 7f40aa19a640 10 monclient(hunting): handle_auth_done global_id 1034352 payload 824 2024-09-16T04:10:45.643 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.607+0000 7f40aa19a640 1 -- 172.21.15.42:0/1483818829 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f40a4139c50 con 0x7f40a4139870 2024-09-16T04:10:45.643 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.607+0000 7f409affd640 1 -- 172.21.15.42:0/1483818829 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(706..706 src has 451..706) ==== 5040+0+0 (secure 0 0 0) 0x7f40940e27d0 con 0x7f40a40a3d90 2024-09-16T04:10:45.643 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.607+0000 7f409affd640 10 client.1014470.objecter ms_dispatch 0x7f40a4000ba0 osd_map(706..706 src has 451..706) 2024-09-16T04:10:45.643 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.607+0000 7f409affd640 3 client.1014470.objecter handle_osd_map got epochs [706,706] > 0 2024-09-16T04:10:45.643 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.607+0000 7f409affd640 3 client.1014470.objecter handle_osd_map decoding full epoch 706 2024-09-16T04:10:45.643 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.607+0000 7f409affd640 20 client.1014470.objecter dump_active .. 0 homeless 2024-09-16T04:10:45.643 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.607+0000 7f409affd640 10 client.1014470.objecter ms_handle_connect 0x7f40a4139870 2024-09-16T04:10:45.643 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.607+0000 7f409affd640 10 client.1014470.objecter resend_mon_ops 2024-09-16T04:10:45.643 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.607+0000 7f40aa19a640 1 --2- 172.21.15.42:0/1483818829 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f40a4139870 0x7f40a41d0140 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f40a0006cb0 tx=0x7f40a000f610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:10:45.643 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.607+0000 7f40aa99b640 1 --2- 172.21.15.42:0/1483818829 >> v2:172.21.15.154:6812/254845342 conn(0x7f4088037810 0x7f4088039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:10:45.643 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.607+0000 7f40aa99b640 10 monclient: get_auth_request con 0x7f4088037810 auth_method 0 2024-09-16T04:10:45.643 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.607+0000 7f409affd640 1 -- 172.21.15.42:0/1483818829 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40a0014440 con 0x7f40a4139870 2024-09-16T04:10:45.643 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.607+0000 7f409affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:45.644 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.607+0000 7f409affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:10:45.644 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.607+0000 7f409affd640 10 monclient: dump: 2024-09-16T04:10:45.644 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:10:45.644 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:10:45.644 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:45.644 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:45.644 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:45.644 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:10:45.644 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:10:45.644 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:10:45.644 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:10:45.644 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:10:45.644 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.607+0000 7f409affd640 1 -- 172.21.15.42:0/1483818829 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f40a0014440 con 0x7f40a4139870 2024-09-16T04:10:45.645 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.607+0000 7f409affd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:45.645 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.607+0000 7f409affd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:10:45.645 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.607+0000 7f409affd640 1 -- 172.21.15.42:0/1483818829 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f40a4139870 msgr2=0x7f40a41d0140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:45.645 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.607+0000 7f409affd640 1 --2- 172.21.15.42:0/1483818829 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f40a4139870 0x7f40a41d0140 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f40a0006cb0 tx=0x7f40a000f610 comp rx=0 tx=0).stop 2024-09-16T04:10:45.645 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.607+0000 7f409affd640 1 -- 172.21.15.42:0/1483818829 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f40a4139870 msgr2=0x7f40a41d0140 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:10:45.645 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.607+0000 7f409affd640 1 --2- 172.21.15.42:0/1483818829 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f40a4139870 0x7f40a41d0140 secure :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f40a0006cb0 tx=0x7f40a000f610 comp rx=0 tx=0).stop 2024-09-16T04:10:45.645 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.607+0000 7f409affd640 10 client.1014470.objecter ms_handle_connect 0x7f4088037810 2024-09-16T04:10:45.645 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.607+0000 7f40aa99b640 1 --2- 172.21.15.42:0/1483818829 >> v2:172.21.15.154:6812/254845342 conn(0x7f4088037810 0x7f4088039cd0 secure :-1 s=READY pgs=3939 cs=0 l=1 rev1=1 crypto rx=0x7f409c007fd0 tx=0x7f409c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:10:45.647 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.615+0000 7f40ac3fe640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:10:45.647 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.615+0000 7f40ac3fe640 1 --2- 172.21.15.42:0/1483818829 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f40a41d0680 0x7f40a402c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:10:45.647 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.615+0000 7f40ac3fe640 10 monclient(hunting): start opening mon connection 2024-09-16T04:10:45.647 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.615+0000 7f40ac3fe640 1 -- 172.21.15.42:0/1483818829 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f40a4133b80 con 0x7f40a41d0680 2024-09-16T04:10:45.647 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.615+0000 7f40aa19a640 1 --2- 172.21.15.42:0/1483818829 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f40a41d0680 0x7f40a402c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:10:45.647 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.615+0000 7f40aa19a640 10 monclient: get_auth_request con 0x7f40a41d0680 auth_method 0 2024-09-16T04:10:45.647 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.615+0000 7f40aa19a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:10:45.648 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.615+0000 7f40aa19a640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:45.648 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.615+0000 7f40aa19a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:45.648 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.615+0000 7f40aa19a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:10:45.648 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.615+0000 7f40aa19a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:10:45.648 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.615+0000 7f40aa19a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:10:45.648 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.615+0000 7f40aa19a640 10 monclient(hunting): handle_auth_done global_id 1034355 payload 824 2024-09-16T04:10:45.648 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.615+0000 7f40aa19a640 1 -- 172.21.15.42:0/1483818829 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f40a40a5750 con 0x7f40a41d0680 2024-09-16T04:10:45.648 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.615+0000 7f40aa19a640 1 --2- 172.21.15.42:0/1483818829 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f40a41d0680 0x7f40a402c650 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f40a0006cb0 tx=0x7f40a00046d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:10:45.648 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.615+0000 7f409affd640 10 client.1014470.objecter ms_handle_connect 0x7f40a41d0680 2024-09-16T04:10:45.648 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.615+0000 7f409affd640 10 client.1014470.objecter resend_mon_ops 2024-09-16T04:10:45.648 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.615+0000 7f409affd640 1 -- 172.21.15.42:0/1483818829 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40a0022980 con 0x7f40a41d0680 2024-09-16T04:10:45.648 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.615+0000 7f409affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:45.649 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.615+0000 7f409affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:10:45.649 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.615+0000 7f409affd640 10 monclient: dump: 2024-09-16T04:10:45.649 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:10:45.649 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:10:45.649 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:45.649 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:45.649 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:45.649 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:10:45.649 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:10:45.649 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:10:45.649 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:10:45.650 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:10:45.650 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.615+0000 7f409affd640 1 -- 172.21.15.42:0/1483818829 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4882 (secure 0 0 0) 0x7f40a0022980 con 0x7f40a41d0680 2024-09-16T04:10:45.650 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.615+0000 7f409affd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:10:45.650 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.615+0000 7f409affd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:10:45.650 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.615+0000 7f409affd640 1 -- 172.21.15.42:0/1483818829 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f40a41d0680 msgr2=0x7f40a402c650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:45.650 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.615+0000 7f409affd640 1 --2- 172.21.15.42:0/1483818829 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f40a41d0680 0x7f40a402c650 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f40a0006cb0 tx=0x7f40a00046d0 comp rx=0 tx=0).stop 2024-09-16T04:10:45.650 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.615+0000 7f409affd640 1 -- 172.21.15.42:0/1483818829 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f40a41d0680 msgr2=0x7f40a402c650 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:10:45.650 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.615+0000 7f409affd640 1 --2- 172.21.15.42:0/1483818829 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f40a41d0680 0x7f40a402c650 secure :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f40a0006cb0 tx=0x7f40a00046d0 comp rx=0 tx=0).stop 2024-09-16T04:10:45.650 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T04:10:45.650 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:10:45.651 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:10:45.651 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "leader", 2024-09-16T04:10:45.651 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 262, 2024-09-16T04:10:45.651 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 125550, 2024-09-16T04:10:45.651 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T04:10:45.651 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T04:10:45.651 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T04:10:45.651 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T04:10:45.651 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:10:45.651 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 3, 2024-09-16T04:10:45.652 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:10:45.652 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T04:10:45.652 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T04:10:45.652 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:10:45.652 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:10:45.652 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:10:45.652 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:10:45.652 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:10:45.652 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:10:45.652 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:10:45.652 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:10:45.652 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:10:45.652 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:10:45.653 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:10:45.653 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:10:45.653 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:10:45.653 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T04:10:45.653 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:10:45.653 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:10:45.653 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:10:45.653 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:10:45.653 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:10:45.653 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:10:45.653 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:10:45.653 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:10:45.653 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:10:45.654 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:10:45.654 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:10:45.654 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:10:45.654 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:10:45.654 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T04:10:45.654 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T04:10:45.654 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T04:10:45.654 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T04:10:45.654 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T04:10:45.654 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T04:10:45.654 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:10:45.655 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:10:45.655 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T04:10:45.655 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T04:10:45.655 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T04:10:45.655 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T04:10:45.655 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T04:10:45.655 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T04:10:45.655 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T04:10:45.655 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:10:45.655 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T04:10:45.655 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:10:45.655 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:10:45.655 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:10:45.656 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:10:45.656 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:10:45.656 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:10:45.656 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:10:45.656 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:10:45.656 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:10:45.656 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:10:45.656 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:10:45.656 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:10:45.656 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T04:10:45.656 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:10:45.656 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T04:10:45.656 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:10:45.656 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:10:45.657 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:10:45.657 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:10:45.657 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:10:45.657 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:10:45.657 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:10:45.657 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T04:10:45.657 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:10:45.657 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:10:45.657 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:10:45.657 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:10:45.657 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T04:10:45.657 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:10:45.657 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:10:45.657 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:10:45.658 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:10:45.658 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T04:10:45.658 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T04:10:45.658 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:10:45.658 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:10:45.658 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:10:45.658 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:10:45.658 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:10:45.658 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:10:45.658 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:10:45.658 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:10:45.658 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:10:45.658 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:10:45.658 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:10:45.659 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T04:10:45.659 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:10:45.659 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:10:45.659 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:10:45.659 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:10:45.659 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T04:10:45.659 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:10:45.659 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:10:45.659 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:10:45.659 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:10:45.659 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T04:10:45.659 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T04:10:45.659 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:10:45.659 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:10:45.660 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:10:45.660 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:10:45.660 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:10:45.660 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:10:45.660 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:10:45.660 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:10:45.660 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:10:45.660 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:10:45.660 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:10:45.660 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T04:10:45.660 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:10:45.660 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:10:45.661 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:10:45.661 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:10:45.661 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T04:10:45.661 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:10:45.661 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:10:45.661 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:10:45.661 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:10:45.661 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T04:10:45.661 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T04:10:45.661 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:10:45.661 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:10:45.661 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:10:45.661 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:10:45.661 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:10:45.662 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:10:45.662 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T04:10:45.662 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T04:10:45.662 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:10:45.662 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:10:45.662 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:10:45.662 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:10:45.662 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:10:45.662 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:10:45.662 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T04:10:45.662 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:10:45.662 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:10:45.663 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:10:45.663 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 2 2024-09-16T04:10:45.663 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:10:45.663 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:10:45.663 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T04:10:45.663 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:10:45.663 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:10:45.663 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:10:45.663 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 4 2024-09-16T04:10:45.663 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:10:45.663 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:10:45.663 INFO:teuthology.orchestra.run.smithi042.stdout: "mgr": [ 2024-09-16T04:10:45.663 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:10:45.664 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:10:45.664 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:10:45.664 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:10:45.664 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:10:45.664 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:10:45.664 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:10:45.664 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T04:10:45.664 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T04:10:45.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.615+0000 7f40ac3fe640 10 librados: watch_flush enter 2024-09-16T04:10:45.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.615+0000 7f40ac3fe640 10 librados: watch_flush exit 2024-09-16T04:10:45.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.615+0000 7f40ac3fe640 20 client.1014470.objecter shutdown clearing up homeless session... 2024-09-16T04:10:45.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.615+0000 7f40ac3fe640 10 client.1014470.objecter successfully canceled tick 2024-09-16T04:10:45.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.615+0000 7f40ac3fe640 1 -- 172.21.15.42:0/1483818829 >> v2:172.21.15.154:6812/254845342 conn(0x7f4088037810 msgr2=0x7f4088039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:45.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.615+0000 7f40ac3fe640 1 --2- 172.21.15.42:0/1483818829 >> v2:172.21.15.154:6812/254845342 conn(0x7f4088037810 0x7f4088039cd0 secure :-1 s=READY pgs=3939 cs=0 l=1 rev1=1 crypto rx=0x7f409c007fd0 tx=0x7f409c009040 comp rx=0 tx=0).stop 2024-09-16T04:10:45.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.615+0000 7f40ac3fe640 10 monclient: shutdown 2024-09-16T04:10:45.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.615+0000 7f40ac3fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:10:45.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.615+0000 7f40ac3fe640 1 -- 172.21.15.42:0/1483818829 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f40a40a3d90 msgr2=0x7f40a40a1f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:45.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.615+0000 7f40ac3fe640 1 --2- 172.21.15.42:0/1483818829 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f40a40a3d90 0x7f40a40a1f10 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f409401a9d0 tx=0x7f40940190c0 comp rx=0 tx=0).stop 2024-09-16T04:10:45.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.615+0000 7f40aa99b640 1 -- 172.21.15.42:0/1483818829 reap_dead start 2024-09-16T04:10:45.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.615+0000 7f40aa99b640 1 -- 172.21.15.42:0/1483818829 reap_dead start 2024-09-16T04:10:45.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.615+0000 7f40ac3fe640 1 -- 172.21.15.42:0/1483818829 shutdown_connections 2024-09-16T04:10:45.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.615+0000 7f40ac3fe640 1 -- 172.21.15.42:0/1483818829 >> 172.21.15.42:0/1483818829 conn(0x7f40a40a9fb0 msgr2=0x7f40a41321a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:10:45.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.615+0000 7f40ac3fe640 1 -- 172.21.15.42:0/1483818829 shutdown_connections 2024-09-16T04:10:45.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.615+0000 7f40ac3fe640 1 -- 172.21.15.42:0/1483818829 wait complete. 2024-09-16T04:10:45.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.615+0000 7f40ac3fe640 1 librados: shutdown 2024-09-16T04:10:45.668 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:10:45.668 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T04:10:45.668 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T04:10:45.789 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.787+0000 7f80735b8640 10 monclient: get_monmap_and_config 2024-09-16T04:10:45.790 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.787+0000 7f80735b8640 10 monclient: build_initial_monmap 2024-09-16T04:10:45.790 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.787+0000 7f80735b8640 10 monclient: monmap: 2024-09-16T04:10:45.790 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:10:45.790 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:10:45.790 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:10:45.788236+0000 2024-09-16T04:10:45.790 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:10:45.788236+0000 2024-09-16T04:10:45.791 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:10:45.791 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:10:45.791 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:10:45.791 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:10:45.791 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:10:45.791 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:10:45.791 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:10:45.791 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:10:45.794 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.787+0000 7f80735b8640 1 Processor -- start 2024-09-16T04:10:45.794 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.787+0000 7f80735b8640 1 -- start start 2024-09-16T04:10:45.794 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.787+0000 7f80735b8640 10 monclient: init 2024-09-16T04:10:45.794 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.787+0000 7f80735b8640 10 monclient: _reopen_session rank -1 2024-09-16T04:10:45.794 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.787+0000 7f80735b8640 10 monclient: _add_conns ranks=[3,0,1,4,2] 2024-09-16T04:10:45.794 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.787+0000 7f80735b8640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f806c0a39b0 0x7f806c133b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:10:45.795 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.787+0000 7f80735b8640 10 monclient(hunting): picked mon.noname-a con 0x7f806c0a39b0 addr v2:172.21.15.42:3300/0 2024-09-16T04:10:45.795 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.787+0000 7f80735b8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f806c0a3d90 addr v1:172.21.15.42:6789/0 2024-09-16T04:10:45.795 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.787+0000 7f80735b8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f806c137710 addr v1:172.21.15.154:6789/0 2024-09-16T04:10:45.795 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.787+0000 7f80735b8640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f806c09c140 con 0x7f806c0a3d90 2024-09-16T04:10:45.795 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.787+0000 7f80735b8640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f806c09c420 con 0x7f806c137710 2024-09-16T04:10:45.795 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.787+0000 7f80735b8640 10 monclient(hunting): start opening mon connection 2024-09-16T04:10:45.795 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.787+0000 7f80735b8640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f806c06dd60 con 0x7f806c0a39b0 2024-09-16T04:10:45.795 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.787+0000 7f80735b8640 10 monclient(hunting): _renew_subs 2024-09-16T04:10:45.796 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.787+0000 7f80735b8640 10 monclient(hunting): authenticate will time out at 2692.757717s 2024-09-16T04:10:45.796 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.787+0000 7f8071354640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f806c0a39b0 0x7f806c133b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:10:45.796 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.787+0000 7f8071354640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f806c0a39b0 0x7f806c133b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:52834/0 (socket says 172.21.15.42:52834) 2024-09-16T04:10:45.796 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.787+0000 7f8071354640 1 -- 172.21.15.42:0/3690718478 learned_addr learned my addr 172.21.15.42:0/3690718478 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:10:45.796 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.787+0000 7f8071354640 10 monclient(hunting): get_auth_request con 0x7f806c0a39b0 auth_method 0 2024-09-16T04:10:45.797 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.787+0000 7f8071354640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:10:45.797 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.787+0000 7f8071354640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:45.797 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.787+0000 7f8071354640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:45.797 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.787+0000 7f8071354640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:10:45.797 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.787+0000 7f8071354640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:10:45.797 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.787+0000 7f8071354640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:10:45.797 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.787+0000 7f8071354640 10 monclient(hunting): handle_auth_done global_id 1034358 payload 274 2024-09-16T04:10:45.797 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.787+0000 7f8071354640 1 -- 172.21.15.42:0/3690718478 >> v1:172.21.15.154:6789/0 conn(0x7f806c137710 legacy=0x7f806c13f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:45.797 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.787+0000 7f8071354640 1 -- 172.21.15.42:0/3690718478 >> v1:172.21.15.42:6789/0 conn(0x7f806c0a3d90 legacy=0x7f806c137000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:45.798 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.787+0000 7f8071354640 10 monclient: _finish_hunting 0 2024-09-16T04:10:45.798 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.787+0000 7f8071354640 1 monclient: found mon.noname-a 2024-09-16T04:10:45.798 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.787+0000 7f8071354640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:10:45.798 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.787+0000 7f8071354640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:10:45.798 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.787+0000 7f8071354640 1 -- 172.21.15.42:0/3690718478 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f806c0a5e20 con 0x7f806c0a39b0 2024-09-16T04:10:45.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.791+0000 7f8071354640 1 --2- 172.21.15.42:0/3690718478 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f806c0a39b0 0x7f806c133b80 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f8064009e50 tx=0x7f806401c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2c1847ff2c135ac server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:10:45.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.791+0000 7f806bfff640 1 -- 172.21.15.42:0/3690718478 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80640a55b0 con 0x7f806c0a39b0 2024-09-16T04:10:45.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.791+0000 7f806bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:45.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.791+0000 7f806bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:10:45.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.791+0000 7f806bfff640 10 monclient: dump: 2024-09-16T04:10:45.805 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:10:45.805 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:10:45.805 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:45.805 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:45.805 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:45.805 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:10:45.806 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:10:45.806 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:10:45.806 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:10:45.806 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:10:45.806 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.791+0000 7f806bfff640 10 monclient: _finish_auth 0 2024-09-16T04:10:45.806 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.791+0000 7f806bfff640 10 monclient: _check_auth_tickets 2024-09-16T04:10:45.806 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.791+0000 7f806bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:45.807 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.791+0000 7f80735b8640 5 monclient: authenticate success, global_id 1034358 2024-09-16T04:10:45.807 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.791+0000 7f80735b8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:10:45.807 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.791+0000 7f806bfff640 1 -- 172.21.15.42:0/3690718478 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f80640a5750 con 0x7f806c0a39b0 2024-09-16T04:10:45.807 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.791+0000 7f806bfff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:10:45.807 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.791+0000 7f80735b8640 10 monclient: get_monmap_and_config success 2024-09-16T04:10:45.807 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.791+0000 7f80735b8640 4 set_mon_vals no callback set 2024-09-16T04:10:45.807 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.791+0000 7f806bfff640 1 -- 172.21.15.42:0/3690718478 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80640a5b20 con 0x7f806c0a39b0 2024-09-16T04:10:45.807 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.791+0000 7f806bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:45.807 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.791+0000 7f806bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:10:45.807 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.791+0000 7f806bfff640 10 monclient: dump: 2024-09-16T04:10:45.807 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:10:45.808 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:10:45.808 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:45.808 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:45.808 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:45.808 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:10:45.808 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:10:45.808 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:10:45.808 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:10:45.808 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:10:45.808 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.791+0000 7f80735b8640 10 monclient: shutdown 2024-09-16T04:10:45.808 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.791+0000 7f80735b8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:10:45.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.791+0000 7f80735b8640 1 -- 172.21.15.42:0/3690718478 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f806c0a39b0 msgr2=0x7f806c133b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:45.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.791+0000 7f80735b8640 1 --2- 172.21.15.42:0/3690718478 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f806c0a39b0 0x7f806c133b80 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f8064009e50 tx=0x7f806401c8f0 comp rx=0 tx=0).stop 2024-09-16T04:10:45.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.791+0000 7f80735b8640 1 -- 172.21.15.42:0/3690718478 shutdown_connections 2024-09-16T04:10:45.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.791+0000 7f80735b8640 1 --2- 172.21.15.42:0/3690718478 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f806c0a39b0 0x7f806c133b80 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:10:45.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.791+0000 7f80735b8640 1 -- 172.21.15.42:0/3690718478 >> 172.21.15.42:0/3690718478 conn(0x7f806c0a9fb0 msgr2=0x7f806c0aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:10:45.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.791+0000 7f80735b8640 1 -- 172.21.15.42:0/3690718478 shutdown_connections 2024-09-16T04:10:45.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.791+0000 7f80735b8640 1 -- 172.21.15.42:0/3690718478 wait complete. 2024-09-16T04:10:45.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.791+0000 7f80735b8640 10 monclient: build_initial_monmap 2024-09-16T04:10:45.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.791+0000 7f80735b8640 10 monclient: monmap: 2024-09-16T04:10:45.810 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:10:45.810 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:10:45.810 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:10:45.810 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:10:45.810 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:10:45.810 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:10:45.810 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:10:45.811 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:10:45.811 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:10:45.811 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:10:45.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.791+0000 7f80735b8640 1 librados: starting msgr at 2024-09-16T04:10:45.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.791+0000 7f80735b8640 1 librados: starting objecter 2024-09-16T04:10:45.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.791+0000 7f80735b8640 1 Processor -- start 2024-09-16T04:10:45.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.791+0000 7f80735b8640 1 -- start start 2024-09-16T04:10:45.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.791+0000 7f80735b8640 1 librados: setting wanted keys 2024-09-16T04:10:45.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.791+0000 7f80735b8640 1 librados: calling monclient init 2024-09-16T04:10:45.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.791+0000 7f80735b8640 10 monclient: init 2024-09-16T04:10:45.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.795+0000 7f80735b8640 10 monclient: _reopen_session rank -1 2024-09-16T04:10:45.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.795+0000 7f80735b8640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:10:45.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.795+0000 7f80735b8640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f806c0a39b0 0x7f806c130a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:10:45.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.795+0000 7f80735b8640 10 monclient(hunting): picked mon.noname-b con 0x7f806c0a39b0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:10:45.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.795+0000 7f80735b8640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f806c0a3d90 0x7f806c130f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:10:45.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.795+0000 7f80735b8640 10 monclient(hunting): picked mon.noname-c con 0x7f806c0a3d90 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:10:45.813 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.795+0000 7f80735b8640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f806c137710 0x7f806c0a1720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:10:45.813 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.795+0000 7f80735b8640 10 monclient(hunting): picked mon.noname-a con 0x7f806c137710 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:10:45.813 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.795+0000 7f80735b8640 10 monclient(hunting): start opening mon connection 2024-09-16T04:10:45.813 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.795+0000 7f80735b8640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f806c143900 con 0x7f806c137710 2024-09-16T04:10:45.813 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.795+0000 7f80735b8640 10 monclient(hunting): start opening mon connection 2024-09-16T04:10:45.813 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.795+0000 7f80735b8640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f806c06dee0 con 0x7f806c0a39b0 2024-09-16T04:10:45.813 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.795+0000 7f80735b8640 10 monclient(hunting): start opening mon connection 2024-09-16T04:10:45.813 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.795+0000 7f80735b8640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f806c143a80 con 0x7f806c0a3d90 2024-09-16T04:10:45.813 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.795+0000 7f80735b8640 10 monclient(hunting): _renew_subs 2024-09-16T04:10:45.813 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.795+0000 7f80735b8640 10 monclient(hunting): authenticate will time out at 2692.764072s 2024-09-16T04:10:45.813 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.795+0000 7f8071b55640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f806c137710 0x7f806c0a1720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:10:45.813 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.795+0000 7f8070b53640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f806c0a3d90 0x7f806c130f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:10:45.813 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.795+0000 7f8071b55640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f806c137710 0x7f806c0a1720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:52836/0 (socket says 172.21.15.42:52836) 2024-09-16T04:10:45.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.795+0000 7f8070b53640 10 monclient(hunting): get_auth_request con 0x7f806c0a3d90 auth_method 0 2024-09-16T04:10:45.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.795+0000 7f8071b55640 1 -- 172.21.15.42:0/261731324 learned_addr learned my addr 172.21.15.42:0/261731324 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:10:45.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.795+0000 7f8070b53640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:10:45.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.795+0000 7f8070b53640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:45.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.795+0000 7f8070b53640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:45.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.795+0000 7f8071b55640 10 monclient(hunting): get_auth_request con 0x7f806c137710 auth_method 0 2024-09-16T04:10:45.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.795+0000 7f8071b55640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:10:45.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.795+0000 7f8071b55640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:45.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.795+0000 7f8071b55640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:45.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.795+0000 7f8070b53640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:10:45.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.795+0000 7f8070b53640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:10:45.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.795+0000 7f8070b53640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:10:45.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.795+0000 7f8071b55640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:10:45.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.795+0000 7f8071b55640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:10:45.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.795+0000 7f8071b55640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:10:45.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.795+0000 7f8071354640 1 --2- 172.21.15.42:0/261731324 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f806c0a39b0 0x7f806c130a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:10:45.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.795+0000 7f8071354640 10 monclient(hunting): get_auth_request con 0x7f806c0a39b0 auth_method 0 2024-09-16T04:10:45.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.795+0000 7f8071354640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:10:45.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.795+0000 7f8071354640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:45.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.795+0000 7f8071354640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:45.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.795+0000 7f8070b53640 10 monclient(hunting): handle_auth_done global_id 1014473 payload 824 2024-09-16T04:10:45.816 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.795+0000 7f8070b53640 1 -- 172.21.15.42:0/261731324 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f806c0a39b0 msgr2=0x7f806c130a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:45.816 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.795+0000 7f8070b53640 1 --2- 172.21.15.42:0/261731324 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f806c0a39b0 0x7f806c130a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:10:45.816 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.795+0000 7f8070b53640 1 -- 172.21.15.42:0/261731324 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f806c137710 msgr2=0x7f806c0a1720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:45.816 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.795+0000 7f8070b53640 1 --2- 172.21.15.42:0/261731324 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f806c137710 0x7f806c0a1720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:10:45.816 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.795+0000 7f8070b53640 10 monclient: _finish_hunting 0 2024-09-16T04:10:45.816 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.795+0000 7f8070b53640 1 monclient: found mon.noname-c 2024-09-16T04:10:45.816 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.795+0000 7f8070b53640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:10:45.816 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.795+0000 7f8070b53640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T04:10:45.816 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.795+0000 7f8070b53640 1 -- 172.21.15.42:0/261731324 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f806c139830 con 0x7f806c0a3d90 2024-09-16T04:10:45.817 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.795+0000 7f8071b55640 1 --2- 172.21.15.42:0/261731324 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f806c137710 0x7f806c0a1720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:10:45.817 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.795+0000 7f8070b53640 1 --2- 172.21.15.42:0/261731324 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f806c0a3d90 0x7f806c130f60 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f8060000f70 tx=0x7f806000c890 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:10:45.817 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.795+0000 7f8069ffb640 10 client.?.objecter ms_handle_connect 0x7f806c0a3d90 2024-09-16T04:10:45.817 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.795+0000 7f8069ffb640 10 client.?.objecter resend_mon_ops 2024-09-16T04:10:45.817 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.795+0000 7f8069ffb640 1 -- 172.21.15.42:0/261731324 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8060017070 con 0x7f806c0a3d90 2024-09-16T04:10:45.817 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.795+0000 7f8069ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:45.817 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.795+0000 7f8069ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:10:45.817 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.795+0000 7f8069ffb640 10 monclient: dump: 2024-09-16T04:10:45.817 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:10:45.817 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:10:45.818 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:45.818 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:45.818 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:45.818 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:10:45.818 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:10:45.818 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:10:45.818 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:10:45.818 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:10:45.818 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.795+0000 7f8069ffb640 10 monclient: _finish_auth 0 2024-09-16T04:10:45.818 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.795+0000 7f8069ffb640 10 monclient: _check_auth_tickets 2024-09-16T04:10:45.818 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.795+0000 7f8069ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:45.818 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.795+0000 7f80735b8640 5 monclient: authenticate success, global_id 1014473 2024-09-16T04:10:45.818 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.795+0000 7f8069ffb640 1 -- 172.21.15.42:0/261731324 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f80600052d0 con 0x7f806c0a3d90 2024-09-16T04:10:45.818 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.795+0000 7f80735b8640 10 monclient: _renew_subs 2024-09-16T04:10:45.819 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.795+0000 7f80735b8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:10:45.819 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.795+0000 7f80735b8640 1 -- 172.21.15.42:0/261731324 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f806c06f680 con 0x7f806c0a3d90 2024-09-16T04:10:45.819 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.795+0000 7f8069ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:10:45.819 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.795+0000 7f8069ffb640 1 -- 172.21.15.42:0/261731324 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80600054c0 con 0x7f806c0a3d90 2024-09-16T04:10:45.819 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.795+0000 7f8069ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:45.819 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.795+0000 7f8069ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:10:45.819 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.795+0000 7f8069ffb640 10 monclient: dump: 2024-09-16T04:10:45.819 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:10:45.819 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:10:45.819 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:45.819 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:45.819 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:45.819 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:10:45.820 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:10:45.820 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:10:45.820 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:10:45.820 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:10:45.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.795+0000 7f806b7fe640 4 set_mon_vals no callback set 2024-09-16T04:10:45.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.795+0000 7f80735b8640 10 client.1014473.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:10:45.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.795+0000 7f80735b8640 10 monclient: _renew_subs 2024-09-16T04:10:45.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.795+0000 7f80735b8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:10:45.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.795+0000 7f80735b8640 1 -- 172.21.15.42:0/261731324 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f806c0a3370 con 0x7f806c0a3d90 2024-09-16T04:10:45.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.795+0000 7f80735b8640 1 librados: init done 2024-09-16T04:10:45.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.795+0000 7f8069ffb640 1 -- 172.21.15.42:0/261731324 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f806001a070 con 0x7f806c0a3d90 2024-09-16T04:10:45.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.795+0000 7f80735b8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:45.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.795+0000 7f80735b8640 1 --2- 172.21.15.42:0/261731324 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f806c0a2420 0x7f806c02c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:10:45.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.795+0000 7f80735b8640 10 monclient(hunting): start opening mon connection 2024-09-16T04:10:45.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.795+0000 7f80735b8640 1 -- 172.21.15.42:0/261731324 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f806c133b80 con 0x7f806c0a2420 2024-09-16T04:10:45.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.795+0000 7f8071354640 1 --2- 172.21.15.42:0/261731324 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f806c0a2420 0x7f806c02c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:10:45.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.795+0000 7f8071354640 10 monclient: get_auth_request con 0x7f806c0a2420 auth_method 0 2024-09-16T04:10:45.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.795+0000 7f8071354640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:10:45.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.795+0000 7f8071354640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:45.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.795+0000 7f8071354640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:45.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.795+0000 7f8069ffb640 1 --2- 172.21.15.42:0/261731324 >> v2:172.21.15.154:6812/254845342 conn(0x7f8058037810 0x7f8058039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:10:45.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.795+0000 7f8071354640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:10:45.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.795+0000 7f8071354640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:10:45.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.795+0000 7f8071354640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:10:45.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.795+0000 7f8069ffb640 1 -- 172.21.15.42:0/261731324 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(706..706 src has 451..706) ==== 5040+0+0 (secure 0 0 0) 0x7f8060047fc0 con 0x7f806c0a3d90 2024-09-16T04:10:45.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.795+0000 7f8069ffb640 10 client.1014473.objecter ms_dispatch 0x7f806c000ba0 osd_map(706..706 src has 451..706) 2024-09-16T04:10:45.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.795+0000 7f8069ffb640 3 client.1014473.objecter handle_osd_map got epochs [706,706] > 0 2024-09-16T04:10:45.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.795+0000 7f8069ffb640 3 client.1014473.objecter handle_osd_map decoding full epoch 706 2024-09-16T04:10:45.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.795+0000 7f8071354640 10 monclient(hunting): handle_auth_done global_id 1014476 payload 824 2024-09-16T04:10:45.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.795+0000 7f8069ffb640 20 client.1014473.objecter dump_active .. 0 homeless 2024-09-16T04:10:45.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.795+0000 7f8071354640 1 -- 172.21.15.42:0/261731324 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f806c0a5750 con 0x7f806c0a2420 2024-09-16T04:10:45.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.795+0000 7f8071b55640 1 --2- 172.21.15.42:0/261731324 >> v2:172.21.15.154:6812/254845342 conn(0x7f8058037810 0x7f8058039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:10:45.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.795+0000 7f8069ffb640 10 client.1014473.objecter ms_handle_connect 0x7f806c0a2420 2024-09-16T04:10:45.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.795+0000 7f8069ffb640 10 client.1014473.objecter resend_mon_ops 2024-09-16T04:10:45.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.795+0000 7f8071354640 1 --2- 172.21.15.42:0/261731324 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f806c0a2420 0x7f806c02c650 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f806c0a56c0 tx=0x7f8064019690 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:10:45.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.795+0000 7f8071b55640 10 monclient: get_auth_request con 0x7f8058037810 auth_method 0 2024-09-16T04:10:45.823 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.795+0000 7f8069ffb640 1 -- 172.21.15.42:0/261731324 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80640b6620 con 0x7f806c0a2420 2024-09-16T04:10:45.823 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.795+0000 7f8069ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:45.823 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.795+0000 7f8069ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:10:45.823 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.795+0000 7f8069ffb640 10 monclient: dump: 2024-09-16T04:10:45.823 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:10:45.823 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:10:45.823 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:45.823 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:45.823 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:45.823 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:10:45.823 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:10:45.823 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:10:45.823 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:10:45.824 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:10:45.824 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.795+0000 7f8069ffb640 1 -- 172.21.15.42:0/261731324 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f80640b6620 con 0x7f806c0a2420 2024-09-16T04:10:45.824 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.795+0000 7f8069ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:45.824 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.795+0000 7f8069ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:10:45.824 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.795+0000 7f8069ffb640 1 -- 172.21.15.42:0/261731324 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f806c0a2420 msgr2=0x7f806c02c650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:45.824 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.795+0000 7f8069ffb640 1 --2- 172.21.15.42:0/261731324 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f806c0a2420 0x7f806c02c650 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f806c0a56c0 tx=0x7f8064019690 comp rx=0 tx=0).stop 2024-09-16T04:10:45.824 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.795+0000 7f8069ffb640 1 -- 172.21.15.42:0/261731324 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f806c0a2420 msgr2=0x7f806c02c650 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:10:45.825 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.795+0000 7f8069ffb640 1 --2- 172.21.15.42:0/261731324 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f806c0a2420 0x7f806c02c650 secure :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f806c0a56c0 tx=0x7f8064019690 comp rx=0 tx=0).stop 2024-09-16T04:10:45.825 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.795+0000 7f8071b55640 1 --2- 172.21.15.42:0/261731324 >> v2:172.21.15.154:6812/254845342 conn(0x7f8058037810 0x7f8058039cd0 secure :-1 s=READY pgs=3940 cs=0 l=1 rev1=1 crypto rx=0x7f805c001c20 tx=0x7f805c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:10:45.825 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.795+0000 7f8069ffb640 10 client.1014473.objecter ms_handle_connect 0x7f8058037810 2024-09-16T04:10:45.826 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.803+0000 7f80735b8640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:10:45.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.803+0000 7f80735b8640 1 --2- 172.21.15.42:0/261731324 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f806c0a2800 0x7f806c02cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:10:45.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.803+0000 7f80735b8640 10 monclient(hunting): start opening mon connection 2024-09-16T04:10:45.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.803+0000 7f80735b8640 1 -- 172.21.15.42:0/261731324 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f806c09dfe0 con 0x7f806c0a2800 2024-09-16T04:10:45.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.803+0000 7f8071354640 1 --2- 172.21.15.42:0/261731324 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f806c0a2800 0x7f806c02cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:10:45.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.803+0000 7f8071354640 10 monclient: get_auth_request con 0x7f806c0a2800 auth_method 0 2024-09-16T04:10:45.828 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.803+0000 7f8071354640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:10:45.828 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.803+0000 7f8071354640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:45.828 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.803+0000 7f8071354640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:45.828 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.803+0000 7f8071354640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:10:45.828 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.803+0000 7f8071354640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:10:45.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.803+0000 7f8071354640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:10:45.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.803+0000 7f8071354640 10 monclient(hunting): handle_auth_done global_id 1014479 payload 824 2024-09-16T04:10:45.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.803+0000 7f8071354640 1 -- 172.21.15.42:0/261731324 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f806c068e40 con 0x7f806c0a2800 2024-09-16T04:10:45.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.803+0000 7f8069ffb640 10 client.1014473.objecter ms_handle_connect 0x7f806c0a2800 2024-09-16T04:10:45.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.803+0000 7f8069ffb640 10 client.1014473.objecter resend_mon_ops 2024-09-16T04:10:45.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.803+0000 7f8071354640 1 --2- 172.21.15.42:0/261731324 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f806c0a2800 0x7f806c02cb90 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f8064009e50 tx=0x7f80640a7000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:10:45.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.803+0000 7f8069ffb640 1 -- 172.21.15.42:0/261731324 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80640c6070 con 0x7f806c0a2800 2024-09-16T04:10:45.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.803+0000 7f8069ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:45.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.803+0000 7f8069ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:10:45.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.803+0000 7f8069ffb640 10 monclient: dump: 2024-09-16T04:10:45.830 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:10:45.830 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:10:45.830 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:45.830 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:45.830 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:45.830 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:10:45.830 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:10:45.830 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:10:45.830 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:10:45.830 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:10:45.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.803+0000 7f8069ffb640 1 -- 172.21.15.42:0/261731324 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4713 (secure 0 0 0) 0x7f80640c29f0 con 0x7f806c0a2800 2024-09-16T04:10:45.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.803+0000 7f8069ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:10:45.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.803+0000 7f8069ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:10:45.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.803+0000 7f8069ffb640 1 -- 172.21.15.42:0/261731324 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f806c0a2800 msgr2=0x7f806c02cb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:45.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.803+0000 7f8069ffb640 1 --2- 172.21.15.42:0/261731324 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f806c0a2800 0x7f806c02cb90 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f8064009e50 tx=0x7f80640a7000 comp rx=0 tx=0).stop 2024-09-16T04:10:45.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.803+0000 7f8069ffb640 1 -- 172.21.15.42:0/261731324 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f806c0a2800 msgr2=0x7f806c02cb90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:10:45.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.803+0000 7f8069ffb640 1 --2- 172.21.15.42:0/261731324 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f806c0a2800 0x7f806c02cb90 secure :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f8064009e50 tx=0x7f80640a7000 comp rx=0 tx=0).stop 2024-09-16T04:10:45.831 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T04:10:45.831 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:10:45.832 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:10:45.832 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T04:10:45.832 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 262, 2024-09-16T04:10:45.832 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 241006, 2024-09-16T04:10:45.832 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T04:10:45.832 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T04:10:45.832 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T04:10:45.832 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T04:10:45.832 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:10:45.832 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 3, 2024-09-16T04:10:45.832 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:10:45.832 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T04:10:45.833 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T04:10:45.833 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:10:45.833 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:10:45.833 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:10:45.833 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:10:45.833 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:10:45.833 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:10:45.833 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:10:45.834 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:10:45.834 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:10:45.834 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:10:45.834 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:10:45.834 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:10:45.834 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:10:45.834 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T04:10:45.834 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:10:45.834 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:10:45.834 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:10:45.834 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:10:45.835 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:10:45.835 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:10:45.835 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:10:45.835 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:10:45.835 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:10:45.835 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:10:45.835 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:10:45.835 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:10:45.835 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:10:45.836 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T04:10:45.836 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T04:10:45.836 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T04:10:45.836 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T04:10:45.836 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T04:10:45.836 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T04:10:45.836 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:10:45.836 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:10:45.836 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T04:10:45.836 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T04:10:45.836 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T04:10:45.837 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T04:10:45.837 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T04:10:45.837 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T04:10:45.837 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T04:10:45.837 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:10:45.837 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T04:10:45.838 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:10:45.838 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:10:45.838 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:10:45.838 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:10:45.838 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:10:45.838 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:10:45.838 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:10:45.839 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:10:45.839 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:10:45.839 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:10:45.839 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:10:45.839 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:10:45.839 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T04:10:45.839 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:10:45.839 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T04:10:45.839 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:10:45.839 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:10:45.839 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:10:45.839 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:10:45.840 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:10:45.840 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:10:45.840 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:10:45.840 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T04:10:45.840 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:10:45.840 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:10:45.840 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:10:45.840 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:10:45.840 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T04:10:45.840 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:10:45.840 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:10:45.840 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:10:45.841 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:10:45.841 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T04:10:45.841 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T04:10:45.841 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:10:45.841 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:10:45.841 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:10:45.841 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:10:45.841 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:10:45.841 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:10:45.842 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:10:45.842 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:10:45.842 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:10:45.842 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:10:45.842 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:10:45.842 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T04:10:45.842 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:10:45.842 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:10:45.842 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:10:45.843 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:10:45.843 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T04:10:45.843 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:10:45.843 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:10:45.843 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:10:45.843 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:10:45.843 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T04:10:45.843 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T04:10:45.843 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:10:45.843 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:10:45.843 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:10:45.843 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:10:45.844 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:10:45.844 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:10:45.844 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:10:45.844 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:10:45.844 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:10:45.844 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:10:45.844 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:10:45.844 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T04:10:45.844 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:10:45.844 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:10:45.845 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:10:45.845 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:10:45.845 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T04:10:45.845 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:10:45.845 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:10:45.845 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:10:45.845 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:10:45.845 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T04:10:45.846 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T04:10:45.846 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:10:45.846 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:10:45.846 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:10:45.846 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:10:45.846 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:10:45.846 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:10:45.846 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T04:10:45.846 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T04:10:45.847 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:10:45.847 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:10:45.847 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:10:45.847 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:10:45.847 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:10:45.847 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:10:45.847 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T04:10:45.847 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:10:45.847 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:10:45.847 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:10:45.847 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 4 2024-09-16T04:10:45.847 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:10:45.848 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:10:45.848 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T04:10:45.848 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:10:45.848 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:10:45.848 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:10:45.849 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 4 2024-09-16T04:10:45.849 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:10:45.849 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:10:45.849 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:10:45.849 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T04:10:45.849 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T04:10:45.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.803+0000 7f80735b8640 10 librados: watch_flush enter 2024-09-16T04:10:45.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.803+0000 7f80735b8640 10 librados: watch_flush exit 2024-09-16T04:10:45.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.803+0000 7f80735b8640 20 client.1014473.objecter shutdown clearing up homeless session... 2024-09-16T04:10:45.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.803+0000 7f80735b8640 10 client.1014473.objecter successfully canceled tick 2024-09-16T04:10:45.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.803+0000 7f80735b8640 1 -- 172.21.15.42:0/261731324 >> v2:172.21.15.154:6812/254845342 conn(0x7f8058037810 msgr2=0x7f8058039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:45.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.803+0000 7f80735b8640 1 --2- 172.21.15.42:0/261731324 >> v2:172.21.15.154:6812/254845342 conn(0x7f8058037810 0x7f8058039cd0 secure :-1 s=READY pgs=3940 cs=0 l=1 rev1=1 crypto rx=0x7f805c001c20 tx=0x7f805c009040 comp rx=0 tx=0).stop 2024-09-16T04:10:45.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.803+0000 7f80735b8640 10 monclient: shutdown 2024-09-16T04:10:45.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.803+0000 7f80735b8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:10:45.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.803+0000 7f80735b8640 1 -- 172.21.15.42:0/261731324 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f806c0a3d90 msgr2=0x7f806c130f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:45.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.803+0000 7f80735b8640 1 --2- 172.21.15.42:0/261731324 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f806c0a3d90 0x7f806c130f60 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f8060000f70 tx=0x7f806000c890 comp rx=0 tx=0).stop 2024-09-16T04:10:45.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.803+0000 7f8071b55640 1 -- 172.21.15.42:0/261731324 reap_dead start 2024-09-16T04:10:45.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.803+0000 7f8071b55640 1 -- 172.21.15.42:0/261731324 reap_dead start 2024-09-16T04:10:45.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.803+0000 7f80735b8640 1 -- 172.21.15.42:0/261731324 shutdown_connections 2024-09-16T04:10:45.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.803+0000 7f80735b8640 1 -- 172.21.15.42:0/261731324 >> 172.21.15.42:0/261731324 conn(0x7f806c0a9fb0 msgr2=0x7f806c136d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:10:45.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.803+0000 7f80735b8640 1 -- 172.21.15.42:0/261731324 shutdown_connections 2024-09-16T04:10:45.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.803+0000 7f80735b8640 1 -- 172.21.15.42:0/261731324 wait complete. 2024-09-16T04:10:45.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.803+0000 7f80735b8640 1 librados: shutdown 2024-09-16T04:10:45.852 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:10:45.852 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T04:10:45.853 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T04:10:45.978 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.975+0000 7fbff814a640 10 monclient: get_monmap_and_config 2024-09-16T04:10:45.978 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.975+0000 7fbff814a640 10 monclient: build_initial_monmap 2024-09-16T04:10:45.978 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.975+0000 7fbff814a640 10 monclient: monmap: 2024-09-16T04:10:45.978 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:10:45.978 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:10:45.979 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:10:45.977075+0000 2024-09-16T04:10:45.979 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:10:45.977075+0000 2024-09-16T04:10:45.979 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:10:45.979 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:10:45.979 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:10:45.979 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:10:45.979 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:10:45.979 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:10:45.979 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:10:45.979 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:10:45.981 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.975+0000 7fbff814a640 1 Processor -- start 2024-09-16T04:10:45.981 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.975+0000 7fbff814a640 1 -- start start 2024-09-16T04:10:45.982 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.975+0000 7fbff814a640 10 monclient: init 2024-09-16T04:10:45.982 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.975+0000 7fbff814a640 10 monclient: _reopen_session rank -1 2024-09-16T04:10:45.982 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.975+0000 7fbff814a640 10 monclient: _add_conns ranks=[3,2,4,1,0] 2024-09-16T04:10:45.982 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.975+0000 7fbff814a640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fbff00a2f10 0x7fbff00a1330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:10:45.982 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.975+0000 7fbff814a640 10 monclient(hunting): picked mon.noname-a con 0x7fbff00a2f10 addr v2:172.21.15.42:3300/0 2024-09-16T04:10:45.982 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.975+0000 7fbff814a640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbff009e070 0x7fbff00a1870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:10:45.982 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.975+0000 7fbff814a640 10 monclient(hunting): picked mon.noname-b con 0x7fbff009e070 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:10:45.982 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.975+0000 7fbff814a640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fbff00a1db0 0x7fbff013b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:10:45.983 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.975+0000 7fbff814a640 10 monclient(hunting): picked mon.noname-c con 0x7fbff00a1db0 addr v2:172.21.15.154:3300/0 2024-09-16T04:10:45.983 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.975+0000 7fbff814a640 10 monclient(hunting): start opening mon connection 2024-09-16T04:10:45.983 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.975+0000 7fbff814a640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7fbff00a32f0 con 0x7fbff00a2f10 2024-09-16T04:10:45.983 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.975+0000 7fbff814a640 10 monclient(hunting): start opening mon connection 2024-09-16T04:10:45.983 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.975+0000 7fbff814a640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fbff009dc50 con 0x7fbff00a1db0 2024-09-16T04:10:45.983 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.975+0000 7fbff814a640 10 monclient(hunting): start opening mon connection 2024-09-16T04:10:45.983 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.975+0000 7fbff814a640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fbff0073290 con 0x7fbff009e070 2024-09-16T04:10:45.983 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.975+0000 7fbff814a640 10 monclient(hunting): _renew_subs 2024-09-16T04:10:45.983 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.975+0000 7fbff814a640 10 monclient(hunting): authenticate will time out at 2692.946497s 2024-09-16T04:10:45.983 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.975+0000 7fbff56e5640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbff009e070 0x7fbff00a1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:10:45.984 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.975+0000 7fbff5ee6640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fbff00a2f10 0x7fbff00a1330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:10:45.984 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.975+0000 7fbff56e5640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbff009e070 0x7fbff00a1870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:43306/0 (socket says 172.21.15.42:43306) 2024-09-16T04:10:45.984 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.975+0000 7fbff56e5640 1 -- 172.21.15.42:0/4006334011 learned_addr learned my addr 172.21.15.42:0/4006334011 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:10:45.984 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.975+0000 7fbff56e5640 10 monclient(hunting): get_auth_request con 0x7fbff009e070 auth_method 0 2024-09-16T04:10:45.984 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.975+0000 7fbff56e5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:10:45.984 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.975+0000 7fbff56e5640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:45.984 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.975+0000 7fbff56e5640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:45.984 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.975+0000 7fbff5ee6640 10 monclient(hunting): get_auth_request con 0x7fbff00a2f10 auth_method 0 2024-09-16T04:10:45.984 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.975+0000 7fbff5ee6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:10:45.984 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.975+0000 7fbff5ee6640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:45.984 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.975+0000 7fbff5ee6640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:45.985 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.975+0000 7fbff56e5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:10:45.985 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.975+0000 7fbff56e5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:10:45.985 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.975+0000 7fbff56e5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:10:45.985 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.975+0000 7fbff5ee6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:10:45.985 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.975+0000 7fbff5ee6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:10:45.985 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.975+0000 7fbff5ee6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:10:45.985 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.975+0000 7fbff66e7640 1 --2- 172.21.15.42:0/4006334011 >> v2:172.21.15.154:3300/0 conn(0x7fbff00a1db0 0x7fbff013b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:10:45.986 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.975+0000 7fbff66e7640 10 monclient(hunting): get_auth_request con 0x7fbff00a1db0 auth_method 0 2024-09-16T04:10:45.986 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.975+0000 7fbff66e7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:10:45.986 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.975+0000 7fbff66e7640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:45.986 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.975+0000 7fbff66e7640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:45.986 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.975+0000 7fbff56e5640 10 monclient(hunting): handle_auth_done global_id 1014482 payload 274 2024-09-16T04:10:45.986 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.975+0000 7fbff56e5640 1 -- 172.21.15.42:0/4006334011 >> v2:172.21.15.154:3300/0 conn(0x7fbff00a1db0 msgr2=0x7fbff013b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:10:45.986 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.975+0000 7fbff56e5640 1 --2- 172.21.15.42:0/4006334011 >> v2:172.21.15.154:3300/0 conn(0x7fbff00a1db0 0x7fbff013b8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:10:45.986 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.975+0000 7fbff56e5640 1 -- 172.21.15.42:0/4006334011 >> v2:172.21.15.42:3300/0 conn(0x7fbff00a2f10 msgr2=0x7fbff00a1330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:10:45.987 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.975+0000 7fbff56e5640 1 --2- 172.21.15.42:0/4006334011 >> v2:172.21.15.42:3300/0 conn(0x7fbff00a2f10 0x7fbff00a1330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:10:45.987 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.975+0000 7fbff56e5640 10 monclient: _finish_hunting 0 2024-09-16T04:10:45.987 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.975+0000 7fbff56e5640 1 monclient: found mon.noname-b 2024-09-16T04:10:45.987 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.975+0000 7fbff56e5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:10:45.987 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.975+0000 7fbff56e5640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T04:10:45.987 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.975+0000 7fbff56e5640 1 -- 172.21.15.42:0/4006334011 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbff009e600 con 0x7fbff009e070 2024-09-16T04:10:45.987 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.975+0000 7fbff5ee6640 1 --2- 172.21.15.42:0/4006334011 >> v2:172.21.15.42:3300/0 conn(0x7fbff00a2f10 0x7fbff00a1330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:10:45.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.979+0000 7fbff56e5640 1 --2- 172.21.15.42:0/4006334011 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbff009e070 0x7fbff00a1870 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fbff0068f60 tx=0x7fbfe001c120 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c31021b723edbc79 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:10:45.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.979+0000 7fbff4ee4640 1 -- 172.21.15.42:0/4006334011 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfe00b0070 con 0x7fbff009e070 2024-09-16T04:10:45.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.979+0000 7fbff4ee4640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:45.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.979+0000 7fbff4ee4640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:10:45.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.979+0000 7fbff4ee4640 10 monclient: dump: 2024-09-16T04:10:45.996 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:10:45.996 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:10:45.996 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:45.996 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:45.996 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:45.996 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:10:45.996 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:10:45.996 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:10:45.997 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:10:45.997 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:10:45.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.979+0000 7fbff4ee4640 10 monclient: _finish_auth 0 2024-09-16T04:10:45.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.979+0000 7fbff4ee4640 10 monclient: _check_auth_tickets 2024-09-16T04:10:45.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.979+0000 7fbff4ee4640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:45.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.979+0000 7fbff814a640 5 monclient: authenticate success, global_id 1014482 2024-09-16T04:10:45.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.979+0000 7fbff814a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:10:45.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.979+0000 7fbff4ee4640 1 -- 172.21.15.42:0/4006334011 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbfe00ab600 con 0x7fbff009e070 2024-09-16T04:10:45.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.979+0000 7fbff4ee4640 10 monclient: handle_config config(0 keys) 2024-09-16T04:10:45.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.979+0000 7fbff4ee4640 1 -- 172.21.15.42:0/4006334011 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfe00ab9d0 con 0x7fbff009e070 2024-09-16T04:10:45.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.979+0000 7fbff814a640 10 monclient: get_monmap_and_config success 2024-09-16T04:10:45.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.979+0000 7fbff814a640 4 set_mon_vals no callback set 2024-09-16T04:10:45.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.979+0000 7fbff4ee4640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:45.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.979+0000 7fbff4ee4640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:10:45.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.979+0000 7fbff4ee4640 10 monclient: dump: 2024-09-16T04:10:45.999 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:10:45.999 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:10:45.999 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:46.000 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:46.000 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:46.000 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:10:46.000 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:10:46.000 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:10:46.000 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:10:46.001 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:10:46.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.979+0000 7fbff814a640 10 monclient: shutdown 2024-09-16T04:10:46.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.979+0000 7fbff814a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:10:46.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.979+0000 7fbff814a640 1 -- 172.21.15.42:0/4006334011 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbff009e070 msgr2=0x7fbff00a1870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:46.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.979+0000 7fbff814a640 1 --2- 172.21.15.42:0/4006334011 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbff009e070 0x7fbff00a1870 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fbff0068f60 tx=0x7fbfe001c120 comp rx=0 tx=0).stop 2024-09-16T04:10:46.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.979+0000 7fbff814a640 1 -- 172.21.15.42:0/4006334011 shutdown_connections 2024-09-16T04:10:46.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.979+0000 7fbff814a640 1 --2- 172.21.15.42:0/4006334011 >> v2:172.21.15.154:3300/0 conn(0x7fbff00a1db0 0x7fbff013b8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:10:46.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.979+0000 7fbff814a640 1 --2- 172.21.15.42:0/4006334011 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbff009e070 0x7fbff00a1870 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:10:46.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.979+0000 7fbff814a640 1 --2- 172.21.15.42:0/4006334011 >> v2:172.21.15.42:3300/0 conn(0x7fbff00a2f10 0x7fbff00a1330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:10:46.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.979+0000 7fbff814a640 1 -- 172.21.15.42:0/4006334011 >> 172.21.15.42:0/4006334011 conn(0x7fbff00a2420 msgr2=0x7fbff00a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:10:46.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.979+0000 7fbff814a640 1 -- 172.21.15.42:0/4006334011 shutdown_connections 2024-09-16T04:10:46.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.979+0000 7fbff814a640 1 -- 172.21.15.42:0/4006334011 wait complete. 2024-09-16T04:10:46.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.979+0000 7fbff814a640 10 monclient: build_initial_monmap 2024-09-16T04:10:46.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.979+0000 7fbff814a640 10 monclient: monmap: 2024-09-16T04:10:46.003 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:10:46.003 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:10:46.003 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:10:46.003 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:10:46.004 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:10:46.004 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:10:46.004 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:10:46.004 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:10:46.004 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:10:46.004 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:10:46.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.979+0000 7fbff814a640 1 librados: starting msgr at 2024-09-16T04:10:46.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.979+0000 7fbff814a640 1 librados: starting objecter 2024-09-16T04:10:46.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.979+0000 7fbff814a640 1 Processor -- start 2024-09-16T04:10:46.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.979+0000 7fbff814a640 1 -- start start 2024-09-16T04:10:46.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.979+0000 7fbff814a640 1 librados: setting wanted keys 2024-09-16T04:10:46.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.979+0000 7fbff814a640 1 librados: calling monclient init 2024-09-16T04:10:46.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.979+0000 7fbff814a640 10 monclient: init 2024-09-16T04:10:46.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.983+0000 7fbff814a640 10 monclient: _reopen_session rank -1 2024-09-16T04:10:46.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.983+0000 7fbff814a640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:10:46.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.983+0000 7fbff814a640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbff009e070 0x7fbff01d32c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:10:46.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.983+0000 7fbff814a640 10 monclient(hunting): picked mon.noname-a con 0x7fbff009e070 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:10:46.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.983+0000 7fbff814a640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbff00a1db0 0x7fbff0138c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:10:46.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.983+0000 7fbff814a640 10 monclient(hunting): picked mon.noname-b con 0x7fbff00a1db0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:10:46.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.983+0000 7fbff814a640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbff00a2f10 0x7fbff013dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:10:46.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.983+0000 7fbff814a640 10 monclient(hunting): picked mon.noname-c con 0x7fbff00a2f10 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:10:46.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.983+0000 7fbff814a640 10 monclient(hunting): start opening mon connection 2024-09-16T04:10:46.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.983+0000 7fbff814a640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fbff0144cd0 con 0x7fbff009e070 2024-09-16T04:10:46.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.983+0000 7fbff814a640 10 monclient(hunting): start opening mon connection 2024-09-16T04:10:46.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.983+0000 7fbff814a640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fbff0144b50 con 0x7fbff00a1db0 2024-09-16T04:10:46.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.983+0000 7fbff814a640 10 monclient(hunting): start opening mon connection 2024-09-16T04:10:46.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.983+0000 7fbff814a640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fbff0144e50 con 0x7fbff00a2f10 2024-09-16T04:10:46.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.983+0000 7fbff814a640 10 monclient(hunting): _renew_subs 2024-09-16T04:10:46.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.983+0000 7fbff814a640 10 monclient(hunting): authenticate will time out at 2692.952219s 2024-09-16T04:10:46.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.983+0000 7fbff5ee6640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbff009e070 0x7fbff01d32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:10:46.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.983+0000 7fbff5ee6640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbff009e070 0x7fbff01d32c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:52856/0 (socket says 172.21.15.42:52856) 2024-09-16T04:10:46.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.983+0000 7fbff5ee6640 1 -- 172.21.15.42:0/4108861086 learned_addr learned my addr 172.21.15.42:0/4108861086 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:10:46.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.983+0000 7fbff5ee6640 10 monclient(hunting): get_auth_request con 0x7fbff009e070 auth_method 0 2024-09-16T04:10:46.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.983+0000 7fbff5ee6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:10:46.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.983+0000 7fbff5ee6640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:46.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.983+0000 7fbff5ee6640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:46.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.983+0000 7fbff66e7640 1 --2- 172.21.15.42:0/4108861086 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbff00a2f10 0x7fbff013dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:10:46.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.983+0000 7fbff66e7640 10 monclient(hunting): get_auth_request con 0x7fbff00a2f10 auth_method 0 2024-09-16T04:10:46.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.983+0000 7fbff66e7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:10:46.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.983+0000 7fbff66e7640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:46.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.983+0000 7fbff66e7640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:46.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.983+0000 7fbff5ee6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:10:46.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.983+0000 7fbff5ee6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:10:46.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.983+0000 7fbff5ee6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:10:46.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.983+0000 7fbff66e7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:10:46.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.983+0000 7fbff66e7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:10:46.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.983+0000 7fbff66e7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:10:46.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.983+0000 7fbff5ee6640 10 monclient(hunting): handle_auth_done global_id 1034370 payload 824 2024-09-16T04:10:46.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.983+0000 7fbff56e5640 1 --2- 172.21.15.42:0/4108861086 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbff00a1db0 0x7fbff0138c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:10:46.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.983+0000 7fbff5ee6640 1 -- 172.21.15.42:0/4108861086 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbff00a2f10 msgr2=0x7fbff013dd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:46.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.983+0000 7fbff5ee6640 1 --2- 172.21.15.42:0/4108861086 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbff00a2f10 0x7fbff013dd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:10:46.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.983+0000 7fbff5ee6640 1 -- 172.21.15.42:0/4108861086 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbff00a1db0 msgr2=0x7fbff0138c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:46.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.983+0000 7fbff5ee6640 1 --2- 172.21.15.42:0/4108861086 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbff00a1db0 0x7fbff0138c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:10:46.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.983+0000 7fbff5ee6640 10 monclient: _finish_hunting 0 2024-09-16T04:10:46.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.983+0000 7fbff5ee6640 1 monclient: found mon.noname-a 2024-09-16T04:10:46.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.983+0000 7fbff5ee6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:10:46.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.983+0000 7fbff5ee6640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:10:46.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.983+0000 7fbff5ee6640 1 -- 172.21.15.42:0/4108861086 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbff0140430 con 0x7fbff009e070 2024-09-16T04:10:46.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.983+0000 7fbff56e5640 10 monclient: get_auth_request con 0x7fbff00a1db0 auth_method 0 2024-09-16T04:10:46.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.983+0000 7fbff56e5640 1 --2- 172.21.15.42:0/4108861086 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbff00a1db0 0x7fbff0138c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T04:10:46.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.983+0000 7fbff5ee6640 1 --2- 172.21.15.42:0/4108861086 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbff009e070 0x7fbff01d32c0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fbfec00ebe0 tx=0x7fbfec00c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:10:46.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.983+0000 7fbfe67fc640 10 client.?.objecter ms_handle_connect 0x7fbff009e070 2024-09-16T04:10:46.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.983+0000 7fbfe67fc640 10 client.?.objecter resend_mon_ops 2024-09-16T04:10:46.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.983+0000 7fbfe67fc640 1 -- 172.21.15.42:0/4108861086 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfec017070 con 0x7fbff009e070 2024-09-16T04:10:46.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.983+0000 7fbfe67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:46.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.983+0000 7fbfe67fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:10:46.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.983+0000 7fbfe67fc640 10 monclient: dump: 2024-09-16T04:10:46.015 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:10:46.015 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:10:46.015 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:46.015 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:46.015 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:46.016 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:10:46.016 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:10:46.016 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:10:46.016 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:10:46.016 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:10:46.016 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.983+0000 7fbfe67fc640 10 monclient: _finish_auth 0 2024-09-16T04:10:46.016 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.983+0000 7fbfe67fc640 10 monclient: _check_auth_tickets 2024-09-16T04:10:46.016 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.983+0000 7fbfe67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:10:46.016 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.983+0000 7fbfe67fc640 1 -- 172.21.15.42:0/4108861086 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbfec00cc30 con 0x7fbff009e070 2024-09-16T04:10:46.016 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.983+0000 7fbff814a640 5 monclient: authenticate success, global_id 1034370 2024-09-16T04:10:46.016 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.983+0000 7fbff814a640 10 monclient: _renew_subs 2024-09-16T04:10:46.016 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.983+0000 7fbff814a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:10:46.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.983+0000 7fbff814a640 1 -- 172.21.15.42:0/4108861086 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbff01418e0 con 0x7fbff009e070 2024-09-16T04:10:46.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.983+0000 7fbfe67fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:10:46.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.983+0000 7fbfe67fc640 1 -- 172.21.15.42:0/4108861086 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfec013470 con 0x7fbff009e070 2024-09-16T04:10:46.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.983+0000 7fbfe67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:46.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.983+0000 7fbfe67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:10:46.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.983+0000 7fbfe67fc640 10 monclient: dump: 2024-09-16T04:10:46.017 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:10:46.017 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:10:46.017 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:46.017 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:46.017 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:46.017 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:10:46.017 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:10:46.018 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:10:46.018 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:10:46.018 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:10:46.018 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.983+0000 7fbff814a640 10 client.1034370.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:10:46.018 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.983+0000 7fbff814a640 10 monclient: _renew_subs 2024-09-16T04:10:46.018 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.983+0000 7fbff814a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:10:46.018 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.983+0000 7fbfe7fff640 4 set_mon_vals no callback set 2024-09-16T04:10:46.018 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.983+0000 7fbff814a640 1 -- 172.21.15.42:0/4108861086 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbff00a18b0 con 0x7fbff009e070 2024-09-16T04:10:46.018 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.983+0000 7fbff814a640 1 librados: init done 2024-09-16T04:10:46.018 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.983+0000 7fbff814a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:46.018 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.983+0000 7fbff814a640 1 --2- 172.21.15.42:0/4108861086 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbff0139170 0x7fbff013e240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:10:46.018 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.983+0000 7fbff814a640 10 monclient(hunting): start opening mon connection 2024-09-16T04:10:46.018 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.983+0000 7fbff814a640 1 -- 172.21.15.42:0/4108861086 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fbff013b8f0 con 0x7fbff0139170 2024-09-16T04:10:46.018 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.983+0000 7fbfe67fc640 1 -- 172.21.15.42:0/4108861086 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fbfec01a070 con 0x7fbff009e070 2024-09-16T04:10:46.019 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.983+0000 7fbfe67fc640 1 --2- 172.21.15.42:0/4108861086 >> v2:172.21.15.154:6812/254845342 conn(0x7fbfd4037810 0x7fbfd4039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:10:46.019 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.983+0000 7fbff56e5640 1 --2- 172.21.15.42:0/4108861086 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbff0139170 0x7fbff013e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:10:46.019 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.983+0000 7fbfe67fc640 1 -- 172.21.15.42:0/4108861086 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(706..706 src has 451..706) ==== 5040+0+0 (secure 0 0 0) 0x7fbfec048b50 con 0x7fbff009e070 2024-09-16T04:10:46.019 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.983+0000 7fbfe67fc640 10 client.1034370.objecter ms_dispatch 0x7fbff0000ba0 osd_map(706..706 src has 451..706) 2024-09-16T04:10:46.019 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.983+0000 7fbfe67fc640 3 client.1034370.objecter handle_osd_map got epochs [706,706] > 0 2024-09-16T04:10:46.019 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.983+0000 7fbfe67fc640 3 client.1034370.objecter handle_osd_map decoding full epoch 706 2024-09-16T04:10:46.019 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.983+0000 7fbff56e5640 10 monclient: get_auth_request con 0x7fbff0139170 auth_method 0 2024-09-16T04:10:46.019 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.983+0000 7fbff56e5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:10:46.019 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.983+0000 7fbff56e5640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:46.019 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.983+0000 7fbff56e5640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:46.019 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.983+0000 7fbfe67fc640 20 client.1034370.objecter dump_active .. 0 homeless 2024-09-16T04:10:46.019 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.983+0000 7fbff56e5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:10:46.019 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.983+0000 7fbff56e5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:10:46.020 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.983+0000 7fbff56e5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:10:46.020 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.983+0000 7fbff66e7640 1 --2- 172.21.15.42:0/4108861086 >> v2:172.21.15.154:6812/254845342 conn(0x7fbfd4037810 0x7fbfd4039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:10:46.020 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.983+0000 7fbff66e7640 10 monclient: get_auth_request con 0x7fbfd4037810 auth_method 0 2024-09-16T04:10:46.020 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.983+0000 7fbff56e5640 10 monclient(hunting): handle_auth_done global_id 1064131 payload 824 2024-09-16T04:10:46.020 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.983+0000 7fbff56e5640 1 -- 172.21.15.42:0/4108861086 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbff01401b0 con 0x7fbff0139170 2024-09-16T04:10:46.020 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.983+0000 7fbff56e5640 1 --2- 172.21.15.42:0/4108861086 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbff0139170 0x7fbff013e240 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fbff006add0 tx=0x7fbfe001a960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:10:46.020 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.983+0000 7fbfe67fc640 10 client.1034370.objecter ms_handle_connect 0x7fbff0139170 2024-09-16T04:10:46.020 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.983+0000 7fbfe67fc640 10 client.1034370.objecter resend_mon_ops 2024-09-16T04:10:46.020 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.983+0000 7fbfe67fc640 10 client.1034370.objecter ms_handle_connect 0x7fbfd4037810 2024-09-16T04:10:46.020 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.983+0000 7fbff66e7640 1 --2- 172.21.15.42:0/4108861086 >> v2:172.21.15.154:6812/254845342 conn(0x7fbfd4037810 0x7fbfd4039cd0 secure :-1 s=READY pgs=3941 cs=0 l=1 rev1=1 crypto rx=0x7fbff013f5c0 tx=0x7fbfe8009820 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:10:46.020 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.983+0000 7fbfe67fc640 1 -- 172.21.15.42:0/4108861086 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfe00ba030 con 0x7fbff0139170 2024-09-16T04:10:46.020 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.983+0000 7fbfe67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:46.020 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.983+0000 7fbfe67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:10:46.021 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.983+0000 7fbfe67fc640 10 monclient: dump: 2024-09-16T04:10:46.021 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:10:46.021 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:10:46.021 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:46.021 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:46.021 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:46.021 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:10:46.021 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:10:46.021 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:10:46.021 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:10:46.021 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:10:46.021 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.987+0000 7fbfe67fc640 1 -- 172.21.15.42:0/4108861086 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fbfe0009490 con 0x7fbff0139170 2024-09-16T04:10:46.021 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.987+0000 7fbfe67fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:10:46.022 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.987+0000 7fbfe67fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:10:46.022 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.987+0000 7fbfe67fc640 1 -- 172.21.15.42:0/4108861086 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbff0139170 msgr2=0x7fbff013e240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:46.022 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.987+0000 7fbfe67fc640 1 --2- 172.21.15.42:0/4108861086 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbff0139170 0x7fbff013e240 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fbff006add0 tx=0x7fbfe001a960 comp rx=0 tx=0).stop 2024-09-16T04:10:46.022 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.987+0000 7fbfe67fc640 1 -- 172.21.15.42:0/4108861086 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbff0139170 msgr2=0x7fbff013e240 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:10:46.022 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.987+0000 7fbfe67fc640 1 --2- 172.21.15.42:0/4108861086 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbff0139170 0x7fbff013e240 secure :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fbff006add0 tx=0x7fbfe001a960 comp rx=0 tx=0).stop 2024-09-16T04:10:46.023 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.991+0000 7fbff814a640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:10:46.023 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.991+0000 7fbff814a640 1 --2- 172.21.15.42:0/4108861086 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbff013e780 0x7fbff006f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:10:46.023 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.991+0000 7fbff814a640 10 monclient(hunting): start opening mon connection 2024-09-16T04:10:46.023 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.991+0000 7fbff814a640 1 -- 172.21.15.42:0/4108861086 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fbff013eb80 con 0x7fbff013e780 2024-09-16T04:10:46.023 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.991+0000 7fbff56e5640 1 --2- 172.21.15.42:0/4108861086 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbff013e780 0x7fbff006f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:10:46.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.991+0000 7fbff56e5640 10 monclient: get_auth_request con 0x7fbff013e780 auth_method 0 2024-09-16T04:10:46.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.991+0000 7fbff56e5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:10:46.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.991+0000 7fbff56e5640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:10:46.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.991+0000 7fbff56e5640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:10:46.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.991+0000 7fbff56e5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:10:46.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.991+0000 7fbff56e5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:10:46.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.991+0000 7fbff56e5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:10:46.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.991+0000 7fbff56e5640 10 monclient(hunting): handle_auth_done global_id 1064134 payload 824 2024-09-16T04:10:46.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.991+0000 7fbff56e5640 1 -- 172.21.15.42:0/4108861086 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fbff0141660 con 0x7fbff013e780 2024-09-16T04:10:46.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.991+0000 7fbff56e5640 1 --2- 172.21.15.42:0/4108861086 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbff013e780 0x7fbff006f9a0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fbfe001a7d0 tx=0x7fbfe0018ec0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:10:46.025 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.991+0000 7fbfe67fc640 10 client.1034370.objecter ms_handle_connect 0x7fbff013e780 2024-09-16T04:10:46.025 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.991+0000 7fbfe67fc640 10 client.1034370.objecter resend_mon_ops 2024-09-16T04:10:46.025 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.995+0000 7fbfe67fc640 1 -- 172.21.15.42:0/4108861086 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfe00ba040 con 0x7fbff013e780 2024-09-16T04:10:46.025 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.995+0000 7fbfe67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:10:46.025 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.995+0000 7fbfe67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:10:46.025 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.995+0000 7fbfe67fc640 10 monclient: dump: 2024-09-16T04:10:46.025 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:10:46.025 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:10:46.025 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:46.025 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:10:46.025 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:10:46.025 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:10:46.026 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:10:46.026 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:10:46.026 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:10:46.026 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:10:46.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.995+0000 7fbfe67fc640 1 -- 172.21.15.42:0/4108861086 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4544 (secure 0 0 0) 0x7fbfe00044d0 con 0x7fbff013e780 2024-09-16T04:10:46.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.995+0000 7fbfe67fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:10:46.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.995+0000 7fbfe67fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:10:46.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.995+0000 7fbfe67fc640 1 -- 172.21.15.42:0/4108861086 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbff013e780 msgr2=0x7fbff006f9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:46.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.995+0000 7fbfe67fc640 1 --2- 172.21.15.42:0/4108861086 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbff013e780 0x7fbff006f9a0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fbfe001a7d0 tx=0x7fbfe0018ec0 comp rx=0 tx=0).stop 2024-09-16T04:10:46.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.995+0000 7fbfe67fc640 1 -- 172.21.15.42:0/4108861086 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbff013e780 msgr2=0x7fbff006f9a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:10:46.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.995+0000 7fbfe67fc640 1 --2- 172.21.15.42:0/4108861086 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbff013e780 0x7fbff006f9a0 secure :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fbfe001a7d0 tx=0x7fbfe0018ec0 comp rx=0 tx=0).stop 2024-09-16T04:10:46.027 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T04:10:46.027 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:10:46.027 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:10:46.027 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T04:10:46.027 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 262, 2024-09-16T04:10:46.027 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 6527, 2024-09-16T04:10:46.027 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T04:10:46.027 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T04:10:46.027 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T04:10:46.027 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T04:10:46.027 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:10:46.028 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 3, 2024-09-16T04:10:46.028 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:10:46.028 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T04:10:46.028 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T04:10:46.028 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:10:46.028 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:10:46.028 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:10:46.028 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:10:46.028 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:10:46.028 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:10:46.028 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:10:46.028 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:10:46.028 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:10:46.029 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:10:46.029 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:10:46.029 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:10:46.029 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:10:46.029 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T04:10:46.029 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:10:46.029 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:10:46.029 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:10:46.030 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:10:46.030 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:10:46.030 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:10:46.030 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:10:46.030 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:10:46.030 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:10:46.030 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:10:46.030 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:10:46.030 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:10:46.030 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:10:46.030 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T04:10:46.030 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T04:10:46.030 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T04:10:46.031 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T04:10:46.031 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T04:10:46.031 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T04:10:46.031 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:10:46.031 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:10:46.031 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T04:10:46.031 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T04:10:46.031 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T04:10:46.031 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T04:10:46.031 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T04:10:46.031 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T04:10:46.032 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T04:10:46.032 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:10:46.032 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T04:10:46.032 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:10:46.032 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:10:46.032 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:10:46.032 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:10:46.032 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:10:46.032 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:10:46.032 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:10:46.032 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:10:46.032 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:10:46.032 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:10:46.033 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:10:46.033 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:10:46.033 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T04:10:46.033 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:10:46.033 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T04:10:46.033 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:10:46.033 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:10:46.033 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:10:46.033 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:10:46.033 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:10:46.034 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:10:46.034 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:10:46.034 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T04:10:46.034 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:10:46.034 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:10:46.034 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:10:46.034 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:10:46.034 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T04:10:46.034 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:10:46.034 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:10:46.035 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:10:46.035 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:10:46.035 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T04:10:46.035 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T04:10:46.035 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:10:46.035 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:10:46.035 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:10:46.036 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:10:46.036 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:10:46.036 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:10:46.036 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:10:46.036 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:10:46.036 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:10:46.036 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:10:46.036 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:10:46.036 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T04:10:46.036 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:10:46.036 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:10:46.036 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:10:46.036 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:10:46.037 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T04:10:46.037 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:10:46.037 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:10:46.037 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:10:46.037 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:10:46.037 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T04:10:46.037 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T04:10:46.037 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:10:46.037 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:10:46.037 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:10:46.037 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:10:46.037 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:10:46.037 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:10:46.038 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:10:46.038 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:10:46.038 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:10:46.038 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:10:46.038 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:10:46.038 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T04:10:46.038 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:10:46.038 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:10:46.038 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:10:46.039 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:10:46.039 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T04:10:46.039 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:10:46.039 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:10:46.039 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:10:46.039 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:10:46.039 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T04:10:46.039 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T04:10:46.039 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:10:46.039 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:10:46.039 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:10:46.039 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:10:46.039 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:10:46.040 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:10:46.040 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T04:10:46.040 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T04:10:46.040 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:10:46.040 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:10:46.040 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:10:46.040 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:10:46.040 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:10:46.040 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:10:46.040 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T04:10:46.040 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:10:46.040 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:10:46.040 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:10:46.040 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:10:46.041 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:10:46.041 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:10:46.041 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:10:46.041 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T04:10:46.041 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T04:10:46.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.995+0000 7fbff814a640 10 librados: watch_flush enter 2024-09-16T04:10:46.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.995+0000 7fbff814a640 10 librados: watch_flush exit 2024-09-16T04:10:46.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.995+0000 7fbff814a640 20 client.1034370.objecter shutdown clearing up homeless session... 2024-09-16T04:10:46.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.995+0000 7fbff814a640 10 client.1034370.objecter successfully canceled tick 2024-09-16T04:10:46.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.995+0000 7fbff814a640 1 -- 172.21.15.42:0/4108861086 >> v2:172.21.15.154:6812/254845342 conn(0x7fbfd4037810 msgr2=0x7fbfd4039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:46.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.995+0000 7fbff814a640 1 --2- 172.21.15.42:0/4108861086 >> v2:172.21.15.154:6812/254845342 conn(0x7fbfd4037810 0x7fbfd4039cd0 secure :-1 s=READY pgs=3941 cs=0 l=1 rev1=1 crypto rx=0x7fbff013f5c0 tx=0x7fbfe8009820 comp rx=0 tx=0).stop 2024-09-16T04:10:46.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.995+0000 7fbff814a640 10 monclient: shutdown 2024-09-16T04:10:46.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.995+0000 7fbff814a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:10:46.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.995+0000 7fbff814a640 1 -- 172.21.15.42:0/4108861086 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbff009e070 msgr2=0x7fbff01d32c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:10:46.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.995+0000 7fbff814a640 1 --2- 172.21.15.42:0/4108861086 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbff009e070 0x7fbff01d32c0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fbfec00ebe0 tx=0x7fbfec00c520 comp rx=0 tx=0).stop 2024-09-16T04:10:46.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.995+0000 7fbff66e7640 1 -- 172.21.15.42:0/4108861086 reap_dead start 2024-09-16T04:10:46.045 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.995+0000 7fbff66e7640 1 -- 172.21.15.42:0/4108861086 reap_dead start 2024-09-16T04:10:46.045 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.995+0000 7fbff814a640 1 -- 172.21.15.42:0/4108861086 shutdown_connections 2024-09-16T04:10:46.045 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.995+0000 7fbff814a640 1 -- 172.21.15.42:0/4108861086 >> 172.21.15.42:0/4108861086 conn(0x7fbff00a2420 msgr2=0x7fbff00a3800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:10:46.045 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.995+0000 7fbff814a640 1 -- 172.21.15.42:0/4108861086 shutdown_connections 2024-09-16T04:10:46.045 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.995+0000 7fbff814a640 1 -- 172.21.15.42:0/4108861086 wait complete. 2024-09-16T04:10:46.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:10:45.995+0000 7fbff814a640 1 librados: shutdown 2024-09-16T04:10:46.046 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:10:46.046 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-09-16T04:10:46.046 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['c'] 2024-09-16T04:10:46.046 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-09-16T04:10:46.046 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-16T04:10:46.047 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-16T04:10:46.047 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T04:10:46.067 INFO:tasks.ceph.mon.b:Stopped 2024-09-16T04:10:46.067 INFO:tasks.mon_thrash:Sending STOP to mon c 2024-09-16T04:10:46.067 INFO:tasks.ceph.mon.c:Sent signal 19 2024-09-16T04:10:46.067 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-16T04:11:01.068 INFO:tasks.mon_thrash:Sending CONT to mon c 2024-09-16T04:11:01.069 INFO:tasks.ceph.mon.c:Sent signal 18 2024-09-16T04:11:01.069 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T04:11:01.069 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:11:01.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:01.199+0000 7f79127df640 10 monclient: get_monmap_and_config 2024-09-16T04:11:01.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:01.199+0000 7f79127df640 10 monclient: build_initial_monmap 2024-09-16T04:11:01.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:01.199+0000 7f79127df640 10 monclient: monmap: 2024-09-16T04:11:01.203 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:11:01.203 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:11:01.203 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:11:01.201344+0000 2024-09-16T04:11:01.203 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:11:01.201344+0000 2024-09-16T04:11:01.203 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:11:01.203 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:11:01.203 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:11:01.203 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:11:01.203 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:11:01.203 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:11:01.204 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:11:01.204 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:11:01.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:01.199+0000 7f79127df640 1 Processor -- start 2024-09-16T04:11:01.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:01.199+0000 7f79127df640 1 -- start start 2024-09-16T04:11:01.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:01.199+0000 7f79127df640 10 monclient: init 2024-09-16T04:11:01.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:01.199+0000 7f79127df640 10 monclient: _reopen_session rank -1 2024-09-16T04:11:01.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:01.199+0000 7f79127df640 10 monclient: _add_conns ranks=[1,2,3,0,4] 2024-09-16T04:11:01.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:01.199+0000 7f79127df640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f790c0a1330 addr v1:172.21.15.154:6789/0 2024-09-16T04:11:01.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:01.199+0000 7f79127df640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f790c06dac0 0x7f790c13a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:11:01.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:01.199+0000 7f79127df640 10 monclient(hunting): picked mon.noname-b con 0x7f790c06dac0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:11:01.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:01.199+0000 7f79127df640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f790c0a1e20 0x7f790c13c890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:11:01.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:01.199+0000 7f79127df640 10 monclient(hunting): picked mon.noname-a con 0x7f790c0a1e20 addr v2:172.21.15.42:3300/0 2024-09-16T04:11:01.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:01.199+0000 7f79127df640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f790c09c7d0 con 0x7f790c0a1330 2024-09-16T04:11:01.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:01.199+0000 7f79127df640 10 monclient(hunting): start opening mon connection 2024-09-16T04:11:01.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:01.199+0000 7f79127df640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f790c09c090 con 0x7f790c0a1e20 2024-09-16T04:11:01.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:01.199+0000 7f79127df640 10 monclient(hunting): start opening mon connection 2024-09-16T04:11:01.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:01.199+0000 7f79127df640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f790c09f610 con 0x7f790c06dac0 2024-09-16T04:11:01.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:01.199+0000 7f79127df640 10 monclient(hunting): _renew_subs 2024-09-16T04:11:01.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:01.199+0000 7f79127df640 10 monclient(hunting): authenticate will time out at 2708.170961s 2024-09-16T04:11:01.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:01.199+0000 7f790b7fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f790c06dac0 0x7f790c13a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:11:01.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:01.199+0000 7f790bfff640 1 -- >> v1:172.21.15.154:6789/0 conn(0x7f790c0a1330 legacy=0x7f790c0a1710 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.154:6789/0 2024-09-16T04:11:01.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:01.199+0000 7f7910d7c640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f790c0a1e20 0x7f790c13c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:11:01.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:01.199+0000 7f790b7fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f790c06dac0 0x7f790c13a340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:37892/0 (socket says 172.21.15.42:37892) 2024-09-16T04:11:01.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:01.199+0000 7f790b7fe640 1 -- 172.21.15.42:0/1023987902 learned_addr learned my addr 172.21.15.42:0/1023987902 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:11:01.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:01.199+0000 7f790b7fe640 10 monclient(hunting): get_auth_request con 0x7f790c06dac0 auth_method 0 2024-09-16T04:11:01.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:01.199+0000 7f790b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:11:01.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:01.199+0000 7f790b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:01.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:01.199+0000 7f790b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:01.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:01.199+0000 7f7910d7c640 10 monclient(hunting): get_auth_request con 0x7f790c0a1e20 auth_method 0 2024-09-16T04:11:01.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:01.199+0000 7f7910d7c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:11:01.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:01.199+0000 7f7910d7c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:01.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:01.199+0000 7f7910d7c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:01.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:01.199+0000 7f790b7fe640 1 -- 172.21.15.42:0/1023987902 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f790c06dac0 msgr2=0x7f790c13a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:11:01.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:01.199+0000 7f790b7fe640 1 -- 172.21.15.42:0/1023987902 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f790c06dac0 msgr2=0x7f790c13a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:11:01.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:01.199+0000 7f790b7fe640 1 --2- 172.21.15.42:0/1023987902 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f790c06dac0 0x7f790c13a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:11:01.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:01.199+0000 7f7910d7c640 1 -- 172.21.15.42:0/1023987902 >> v2:172.21.15.42:3300/0 conn(0x7f790c0a1e20 msgr2=0x7f790c13c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T04:11:01.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:01.199+0000 7f7910d7c640 1 -- 172.21.15.42:0/1023987902 >> v2:172.21.15.42:3300/0 conn(0x7f790c0a1e20 msgr2=0x7f790c13c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:11:01.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:01.199+0000 7f7910d7c640 1 --2- 172.21.15.42:0/1023987902 >> v2:172.21.15.42:3300/0 conn(0x7f790c0a1e20 0x7f790c13c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:11:01.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:01.199+0000 7f790b7fe640 1 --2- 172.21.15.42:0/1023987902 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f790c06dac0 0x7f790c13a340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:11:01.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:01.199+0000 7f7910d7c640 1 --2- 172.21.15.42:0/1023987902 >> v2:172.21.15.42:3300/0 conn(0x7f790c0a1e20 0x7f790c13c890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:11:01.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:01.403+0000 7f790bfff640 1 -- 172.21.15.42:0/1023987902 >> v1:172.21.15.154:6789/0 conn(0x7f790c0a1330 legacy=0x7f790c0a1710 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.154:6789/0 2024-09-16T04:11:01.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:01.403+0000 7f790b7fe640 1 --2- 172.21.15.42:0/1023987902 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f790c06dac0 0x7f790c13a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:11:01.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:01.403+0000 7f790b7fe640 10 monclient(hunting): get_auth_request con 0x7f790c06dac0 auth_method 0 2024-09-16T04:11:01.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:01.403+0000 7f790b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:11:01.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:01.403+0000 7f790b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:01.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:01.403+0000 7f790b7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:11:01.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:01.403+0000 7f7910d7c640 1 --2- 172.21.15.42:0/1023987902 >> v2:172.21.15.42:3300/0 conn(0x7f790c0a1e20 0x7f790c13c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:11:01.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:01.403+0000 7f7910d7c640 10 monclient(hunting): get_auth_request con 0x7f790c0a1e20 auth_method 0 2024-09-16T04:11:01.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:01.403+0000 7f7910d7c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:11:01.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:01.403+0000 7f7910d7c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:01.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:01.403+0000 7f7910d7c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:11:01.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:01.403+0000 7f790b7fe640 1 -- 172.21.15.42:0/1023987902 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f790c06dac0 msgr2=0x7f790c13a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:11:01.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:01.403+0000 7f790b7fe640 1 -- 172.21.15.42:0/1023987902 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f790c06dac0 msgr2=0x7f790c13a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:11:01.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:01.403+0000 7f790b7fe640 1 --2- 172.21.15.42:0/1023987902 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f790c06dac0 0x7f790c13a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:11:01.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:01.403+0000 7f790b7fe640 1 --2- 172.21.15.42:0/1023987902 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f790c06dac0 0x7f790c13a340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T04:11:01.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:01.403+0000 7f7910d7c640 1 -- 172.21.15.42:0/1023987902 >> v2:172.21.15.42:3300/0 conn(0x7f790c0a1e20 msgr2=0x7f790c13c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T04:11:01.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:01.403+0000 7f7910d7c640 1 -- 172.21.15.42:0/1023987902 >> v2:172.21.15.42:3300/0 conn(0x7f790c0a1e20 msgr2=0x7f790c13c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:11:01.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:01.403+0000 7f7910d7c640 1 --2- 172.21.15.42:0/1023987902 >> v2:172.21.15.42:3300/0 conn(0x7f790c0a1e20 0x7f790c13c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:11:01.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:01.403+0000 7f7910d7c640 1 --2- 172.21.15.42:0/1023987902 >> v2:172.21.15.42:3300/0 conn(0x7f790c0a1e20 0x7f790c13c890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T04:11:01.810 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:01.807+0000 7f790bfff640 1 -- 172.21.15.42:0/1023987902 >> v1:172.21.15.154:6789/0 conn(0x7f790c0a1330 legacy=0x7f790c0a1710 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.154:6789/0 2024-09-16T04:11:01.810 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:01.807+0000 7f790b7fe640 1 --2- 172.21.15.42:0/1023987902 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f790c06dac0 0x7f790c13a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:11:01.810 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:01.807+0000 7f7910d7c640 1 --2- 172.21.15.42:0/1023987902 >> v2:172.21.15.42:3300/0 conn(0x7f790c0a1e20 0x7f790c13c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:11:01.810 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:01.807+0000 7f790b7fe640 10 monclient(hunting): get_auth_request con 0x7f790c06dac0 auth_method 0 2024-09-16T04:11:01.810 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:01.807+0000 7f790b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:11:01.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:01.807+0000 7f790b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:01.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:01.807+0000 7f790b7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:11:01.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:01.807+0000 7f7910d7c640 10 monclient(hunting): get_auth_request con 0x7f790c0a1e20 auth_method 0 2024-09-16T04:11:01.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:01.807+0000 7f7910d7c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:11:01.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:01.807+0000 7f7910d7c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:01.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:01.807+0000 7f7910d7c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:11:01.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:01.807+0000 7f790b7fe640 1 -- 172.21.15.42:0/1023987902 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f790c06dac0 msgr2=0x7f790c13a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:11:01.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:01.807+0000 7f790b7fe640 1 -- 172.21.15.42:0/1023987902 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f790c06dac0 msgr2=0x7f790c13a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:11:01.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:01.807+0000 7f7910d7c640 1 -- 172.21.15.42:0/1023987902 >> v2:172.21.15.42:3300/0 conn(0x7f790c0a1e20 msgr2=0x7f790c13c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T04:11:01.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:01.807+0000 7f790b7fe640 1 --2- 172.21.15.42:0/1023987902 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f790c06dac0 0x7f790c13a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:11:01.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:01.807+0000 7f7910d7c640 1 -- 172.21.15.42:0/1023987902 >> v2:172.21.15.42:3300/0 conn(0x7f790c0a1e20 msgr2=0x7f790c13c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:11:01.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:01.807+0000 7f7910d7c640 1 --2- 172.21.15.42:0/1023987902 >> v2:172.21.15.42:3300/0 conn(0x7f790c0a1e20 0x7f790c13c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:11:01.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:01.807+0000 7f790b7fe640 1 --2- 172.21.15.42:0/1023987902 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f790c06dac0 0x7f790c13a340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T04:11:01.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:01.807+0000 7f7910d7c640 1 --2- 172.21.15.42:0/1023987902 >> v2:172.21.15.42:3300/0 conn(0x7f790c0a1e20 0x7f790c13c890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T04:11:02.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:02.611+0000 7f790b7fe640 1 --2- 172.21.15.42:0/1023987902 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f790c06dac0 0x7f790c13a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:11:02.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:02.611+0000 7f790bfff640 1 -- 172.21.15.42:0/1023987902 >> v1:172.21.15.154:6789/0 conn(0x7f790c0a1330 legacy=0x7f790c0a1710 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.154:6789/0 2024-09-16T04:11:02.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:02.611+0000 7f790b7fe640 10 monclient(hunting): get_auth_request con 0x7f790c06dac0 auth_method 0 2024-09-16T04:11:02.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:02.611+0000 7f790b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:11:02.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:02.611+0000 7f790b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:02.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:02.611+0000 7f790b7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:11:02.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:02.611+0000 7f7910d7c640 1 --2- 172.21.15.42:0/1023987902 >> v2:172.21.15.42:3300/0 conn(0x7f790c0a1e20 0x7f790c13c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:11:02.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:02.611+0000 7f7910d7c640 10 monclient(hunting): get_auth_request con 0x7f790c0a1e20 auth_method 0 2024-09-16T04:11:02.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:02.611+0000 7f7910d7c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:11:02.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:02.611+0000 7f7910d7c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:02.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:02.611+0000 7f7910d7c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:11:02.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:02.611+0000 7f790b7fe640 1 -- 172.21.15.42:0/1023987902 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f790c06dac0 msgr2=0x7f790c13a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T04:11:02.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:02.611+0000 7f790b7fe640 1 -- 172.21.15.42:0/1023987902 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f790c06dac0 msgr2=0x7f790c13a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:11:02.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:02.611+0000 7f790b7fe640 1 --2- 172.21.15.42:0/1023987902 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f790c06dac0 0x7f790c13a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:11:02.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:02.611+0000 7f790b7fe640 1 --2- 172.21.15.42:0/1023987902 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f790c06dac0 0x7f790c13a340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T04:11:02.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:02.611+0000 7f7910d7c640 1 -- 172.21.15.42:0/1023987902 >> v2:172.21.15.42:3300/0 conn(0x7f790c0a1e20 msgr2=0x7f790c13c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:11:02.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:02.611+0000 7f7910d7c640 1 -- 172.21.15.42:0/1023987902 >> v2:172.21.15.42:3300/0 conn(0x7f790c0a1e20 msgr2=0x7f790c13c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:11:02.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:02.611+0000 7f7910d7c640 1 --2- 172.21.15.42:0/1023987902 >> v2:172.21.15.42:3300/0 conn(0x7f790c0a1e20 0x7f790c13c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:11:02.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:02.611+0000 7f7910d7c640 1 --2- 172.21.15.42:0/1023987902 >> v2:172.21.15.42:3300/0 conn(0x7f790c0a1e20 0x7f790c13c890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T04:11:04.205 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:04.199+0000 7f7909ffb640 10 monclient(hunting): tick 2024-09-16T04:11:04.205 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:04.199+0000 7f7909ffb640 10 monclient(hunting): _check_auth_tickets 2024-09-16T04:11:04.205 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:04.199+0000 7f7909ffb640 1 monclient(hunting): continuing hunt 2024-09-16T04:11:04.205 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:04.199+0000 7f7909ffb640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T04:11:04.205 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:04.199+0000 7f7909ffb640 1 -- 172.21.15.42:0/1023987902 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f790c06dac0 msgr2=0x7f790c13a340 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:11:04.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:04.199+0000 7f7909ffb640 1 --2- 172.21.15.42:0/1023987902 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f790c06dac0 0x7f790c13a340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:11:04.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:04.199+0000 7f7909ffb640 1 -- 172.21.15.42:0/1023987902 >> v2:172.21.15.42:3300/0 conn(0x7f790c0a1e20 msgr2=0x7f790c13c890 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:11:04.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:04.199+0000 7f7909ffb640 1 --2- 172.21.15.42:0/1023987902 >> v2:172.21.15.42:3300/0 conn(0x7f790c0a1e20 0x7f790c13c890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:11:04.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:04.199+0000 7f7909ffb640 1 -- 172.21.15.42:0/1023987902 >> v1:172.21.15.154:6789/0 conn(0x7f790c0a1330 legacy=0x7f790c0a1710 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:11:04.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:04.199+0000 7f7909ffb640 10 monclient: _add_conns ranks=[0,4] 2024-09-16T04:11:04.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:04.199+0000 7f7909ffb640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f78f80036b0 addr v1:172.21.15.42:6789/0 2024-09-16T04:11:04.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:04.199+0000 7f7909ffb640 1 --2- 172.21.15.42:0/1023987902 >> v2:172.21.15.154:3300/0 conn(0x7f78f8007340 0x7f78f8009800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:11:04.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:04.199+0000 7f7909ffb640 10 monclient(hunting): picked mon.noname-c con 0x7f78f8007340 addr v2:172.21.15.154:3300/0 2024-09-16T04:11:04.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:04.199+0000 7f7909ffb640 1 -- 172.21.15.42:0/1023987902 --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f790c09f8f0 con 0x7f78f80036b0 2024-09-16T04:11:04.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:04.199+0000 7f7909ffb640 10 monclient(hunting): start opening mon connection 2024-09-16T04:11:04.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:04.199+0000 7f7909ffb640 1 -- 172.21.15.42:0/1023987902 --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f790c09c090 con 0x7f78f8007340 2024-09-16T04:11:04.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:04.199+0000 7f7909ffb640 10 monclient(hunting): _renew_subs 2024-09-16T04:11:04.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:04.199+0000 7f790b7fe640 1 -- 172.21.15.42:0/1023987902 >> v2:172.21.15.154:3300/0 conn(0x7f78f8007340 msgr2=0x7f78f8009800 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T04:11:04.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:04.199+0000 7f790b7fe640 1 --2- 172.21.15.42:0/1023987902 >> v2:172.21.15.154:3300/0 conn(0x7f78f8007340 0x7f78f8009800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:11:04.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:04.403+0000 7f790b7fe640 1 -- 172.21.15.42:0/1023987902 >> v2:172.21.15.154:3300/0 conn(0x7f78f8007340 msgr2=0x7f78f8009800 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T04:11:04.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:04.403+0000 7f790b7fe640 1 --2- 172.21.15.42:0/1023987902 >> v2:172.21.15.154:3300/0 conn(0x7f78f8007340 0x7f78f8009800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T04:11:04.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:04.807+0000 7f790b7fe640 1 -- 172.21.15.42:0/1023987902 >> v2:172.21.15.154:3300/0 conn(0x7f78f8007340 msgr2=0x7f78f8009800 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T04:11:04.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:04.807+0000 7f790b7fe640 1 --2- 172.21.15.42:0/1023987902 >> v2:172.21.15.154:3300/0 conn(0x7f78f8007340 0x7f78f8009800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T04:11:05.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:05.611+0000 7f790b7fe640 1 -- 172.21.15.42:0/1023987902 >> v2:172.21.15.154:3300/0 conn(0x7f78f8007340 msgr2=0x7f78f8009800 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T04:11:05.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:05.611+0000 7f790b7fe640 1 --2- 172.21.15.42:0/1023987902 >> v2:172.21.15.154:3300/0 conn(0x7f78f8007340 0x7f78f8009800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T04:11:06.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.131+0000 7f790affd640 1 -- 172.21.15.42:0/1023987902 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3298615196 0 0) 0x7f790c09f8f0 con 0x7f78f80036b0 2024-09-16T04:11:06.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.131+0000 7f790affd640 10 monclient(hunting): _negotiate 2024-09-16T04:11:06.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.131+0000 7f790affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:06.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.131+0000 7f790affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:06.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.131+0000 7f790affd640 10 monclient(hunting): my global_id is 1034373 2024-09-16T04:11:06.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.131+0000 7f790affd640 1 -- 172.21.15.42:0/1023987902 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f78f4003530 con 0x7f78f80036b0 2024-09-16T04:11:06.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.139+0000 7f790affd640 1 -- 172.21.15.42:0/1023987902 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 763040893 0 0) 0x7f78f4003530 con 0x7f78f80036b0 2024-09-16T04:11:06.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.139+0000 7f790affd640 1 -- 172.21.15.42:0/1023987902 >> v2:172.21.15.154:3300/0 conn(0x7f78f8007340 msgr2=0x7f78f8009800 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:11:06.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.139+0000 7f790affd640 1 --2- 172.21.15.42:0/1023987902 >> v2:172.21.15.154:3300/0 conn(0x7f78f8007340 0x7f78f8009800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:11:06.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.139+0000 7f790affd640 10 monclient: _finish_hunting 0 2024-09-16T04:11:06.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.139+0000 7f790affd640 1 monclient: found mon.noname-a-legacy 2024-09-16T04:11:06.145 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.139+0000 7f790affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:11:06.145 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.139+0000 7f790affd640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.42:6789/0 2024-09-16T04:11:06.145 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.139+0000 7f790affd640 1 -- 172.21.15.42:0/1023987902 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f790c09c7d0 con 0x7f78f80036b0 2024-09-16T04:11:06.145 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.139+0000 7f790affd640 10 monclient: _finish_auth 0 2024-09-16T04:11:06.145 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.139+0000 7f790affd640 10 monclient: _check_auth_tickets 2024-09-16T04:11:06.145 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.139+0000 7f790affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:11:06.145 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.139+0000 7f790affd640 1 -- 172.21.15.42:0/1023987902 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2177413734 0 0) 0x7f78fc0045d0 con 0x7f78f80036b0 2024-09-16T04:11:06.145 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.139+0000 7f79127df640 5 monclient: authenticate success, global_id 1034373 2024-09-16T04:11:06.145 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.139+0000 7f79127df640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:11:06.145 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.139+0000 7f790affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:06.145 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.139+0000 7f790affd640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T04:11:06.145 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.139+0000 7f790affd640 10 monclient: dump: 2024-09-16T04:11:06.146 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:11:06.146 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:11:06.146 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:11:06.146 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:11:06.146 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:06.146 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:11:06.146 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:11:06.146 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:11:06.146 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:11:06.146 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:11:06.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.139+0000 7f790affd640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] but i'm connected to v1:172.21.15.42:6789/0, reconnecting 2024-09-16T04:11:06.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.139+0000 7f790affd640 10 monclient: _reopen_session rank -1 2024-09-16T04:11:06.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.139+0000 7f790affd640 1 -- 172.21.15.42:0/1023987902 >> v1:172.21.15.42:6789/0 conn(0x7f78f80036b0 legacy=0x7f78f8005b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:06.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.139+0000 7f790affd640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:11:06.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.139+0000 7f7910d7c640 1 -- 172.21.15.42:0/1023987902 reap_dead start 2024-09-16T04:11:06.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.139+0000 7f790affd640 1 --2- 172.21.15.42:0/1023987902 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f78f40039c0 0x7f78f4003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:11:06.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.139+0000 7f790affd640 10 monclient(hunting): picked mon.a con 0x7f78f40039c0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:11:06.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.139+0000 7f790affd640 1 --2- 172.21.15.42:0/1023987902 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f78f4004810 0x7f78f4004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:11:06.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.139+0000 7f790affd640 10 monclient(hunting): picked mon.c con 0x7f78f4004810 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:11:06.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.139+0000 7f790affd640 1 --2- 172.21.15.42:0/1023987902 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f78f40052f0 0x7f78f400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:11:06.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.139+0000 7f790affd640 10 monclient(hunting): picked mon.b con 0x7f78f40052f0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:11:06.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.139+0000 7f790affd640 10 monclient(hunting): start opening mon connection 2024-09-16T04:11:06.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.139+0000 7f790affd640 1 -- 172.21.15.42:0/1023987902 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f790c09c090 con 0x7f78f40039c0 2024-09-16T04:11:06.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.139+0000 7f790affd640 10 monclient(hunting): start opening mon connection 2024-09-16T04:11:06.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.139+0000 7f790affd640 1 -- 172.21.15.42:0/1023987902 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f78f400c240 con 0x7f78f40052f0 2024-09-16T04:11:06.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.139+0000 7f790affd640 10 monclient(hunting): start opening mon connection 2024-09-16T04:11:06.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.139+0000 7f790affd640 1 -- 172.21.15.42:0/1023987902 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f78f400c440 con 0x7f78f4004810 2024-09-16T04:11:06.148 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.139+0000 7f790affd640 10 monclient(hunting): _renew_subs 2024-09-16T04:11:06.148 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.139+0000 7f790affd640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:11:06.148 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.139+0000 7f79127df640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-16T04:11:06.148 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.139+0000 7f790b7fe640 1 --2- 172.21.15.42:0/1023987902 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f78f4004810 0x7f78f4004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:11:06.148 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.139+0000 7f7910d7c640 1 -- 172.21.15.42:0/1023987902 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f78f40052f0 msgr2=0x7f78f400b9d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T04:11:06.148 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.139+0000 7f790bfff640 1 --2- 172.21.15.42:0/1023987902 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f78f40039c0 0x7f78f4003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:11:06.148 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.139+0000 7f7910d7c640 1 --2- 172.21.15.42:0/1023987902 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f78f40052f0 0x7f78f400b9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:11:06.148 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.139+0000 7f790bfff640 10 monclient(hunting): get_auth_request con 0x7f78f40039c0 auth_method 0 2024-09-16T04:11:06.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.139+0000 7f790bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:11:06.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.139+0000 7f790bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:06.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.139+0000 7f790bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:11:06.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.139+0000 7f790b7fe640 10 monclient(hunting): get_auth_request con 0x7f78f4004810 auth_method 0 2024-09-16T04:11:06.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.139+0000 7f790b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:11:06.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.139+0000 7f790b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:06.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.139+0000 7f790b7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:11:06.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.139+0000 7f790bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:11:06.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.139+0000 7f790bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:11:06.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.139+0000 7f790bfff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:11:06.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.139+0000 7f790b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:11:06.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.139+0000 7f790b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:11:06.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.139+0000 7f790b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:11:06.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.139+0000 7f790bfff640 10 monclient(hunting): handle_auth_done global_id 1034373 payload 293 2024-09-16T04:11:06.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.139+0000 7f790bfff640 1 -- 172.21.15.42:0/1023987902 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f78f4004810 msgr2=0x7f78f4004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:11:06.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.139+0000 7f790bfff640 1 --2- 172.21.15.42:0/1023987902 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f78f4004810 0x7f78f4004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:11:06.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.139+0000 7f790bfff640 1 -- 172.21.15.42:0/1023987902 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f78f40052f0 msgr2=0x7f78f400b9d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:11:06.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.139+0000 7f790bfff640 1 --2- 172.21.15.42:0/1023987902 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f78f40052f0 0x7f78f400b9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:11:06.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.139+0000 7f790bfff640 10 monclient: _finish_hunting 0 2024-09-16T04:11:06.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.139+0000 7f790bfff640 1 monclient: found mon.a 2024-09-16T04:11:06.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.139+0000 7f790bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:11:06.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.139+0000 7f790bfff640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:11:06.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.139+0000 7f790bfff640 1 -- 172.21.15.42:0/1023987902 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78f4003530 con 0x7f78f40039c0 2024-09-16T04:11:06.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.139+0000 7f790bfff640 10 monclient: _finish_auth 0 2024-09-16T04:11:06.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.139+0000 7f790bfff640 10 monclient: _check_auth_tickets 2024-09-16T04:11:06.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.139+0000 7f790bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:11:06.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.143+0000 7f790bfff640 1 --2- 172.21.15.42:0/1023987902 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f78f40039c0 0x7f78f4003dc0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f78fc0061a0 tx=0x7f78fc01d6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ff531cf155eab73 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:11:06.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.143+0000 7f790affd640 1 -- 172.21.15.42:0/1023987902 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78fc00d320 con 0x7f78f40039c0 2024-09-16T04:11:06.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.143+0000 7f790affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:06.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.143+0000 7f790affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:11:06.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.143+0000 7f790affd640 10 monclient: dump: 2024-09-16T04:11:06.153 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:11:06.153 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:11:06.153 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:11:06.153 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:11:06.154 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:06.154 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:11:06.154 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:11:06.154 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:11:06.154 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:11:06.154 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:11:06.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.143+0000 7f79127df640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:11:06.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.143+0000 7f790affd640 1 -- 172.21.15.42:0/1023987902 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f78fc0af070 con 0x7f78f40039c0 2024-09-16T04:11:06.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.143+0000 7f790affd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:11:06.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.143+0000 7f79127df640 10 monclient: get_monmap_and_config success 2024-09-16T04:11:06.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.143+0000 7f79127df640 4 set_mon_vals no callback set 2024-09-16T04:11:06.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.143+0000 7f790affd640 1 -- 172.21.15.42:0/1023987902 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78fc0aa680 con 0x7f78f40039c0 2024-09-16T04:11:06.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.143+0000 7f790affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:06.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.143+0000 7f790affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:11:06.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.143+0000 7f790affd640 10 monclient: dump: 2024-09-16T04:11:06.155 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:11:06.155 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:11:06.155 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:11:06.155 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:11:06.155 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:06.156 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:11:06.156 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:11:06.156 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:11:06.156 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:11:06.156 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:11:06.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.147+0000 7f79127df640 10 monclient: shutdown 2024-09-16T04:11:06.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.147+0000 7f79127df640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:11:06.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.147+0000 7f79127df640 1 -- 172.21.15.42:0/1023987902 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f78f40039c0 msgr2=0x7f78f4003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:06.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.147+0000 7f79127df640 1 --2- 172.21.15.42:0/1023987902 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f78f40039c0 0x7f78f4003dc0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f78fc0061a0 tx=0x7f78fc01d6f0 comp rx=0 tx=0).stop 2024-09-16T04:11:06.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.147+0000 7f79127df640 1 -- 172.21.15.42:0/1023987902 shutdown_connections 2024-09-16T04:11:06.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.147+0000 7f79127df640 1 --2- 172.21.15.42:0/1023987902 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f78f40052f0 0x7f78f400b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:11:06.157 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.147+0000 7f79127df640 1 --2- 172.21.15.42:0/1023987902 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f78f4004810 0x7f78f4004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:11:06.157 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.147+0000 7f79127df640 1 --2- 172.21.15.42:0/1023987902 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f78f40039c0 0x7f78f4003dc0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:11:06.157 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.147+0000 7f79127df640 1 -- 172.21.15.42:0/1023987902 >> 172.21.15.42:0/1023987902 conn(0x7f790c0a2420 msgr2=0x7f790c0a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:11:06.157 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.147+0000 7f79127df640 1 -- 172.21.15.42:0/1023987902 shutdown_connections 2024-09-16T04:11:06.157 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.147+0000 7f79127df640 1 -- 172.21.15.42:0/1023987902 wait complete. 2024-09-16T04:11:06.157 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.147+0000 7f79127df640 10 monclient: build_initial_monmap 2024-09-16T04:11:06.157 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.147+0000 7f79127df640 10 monclient: monmap: 2024-09-16T04:11:06.157 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:11:06.157 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:11:06.157 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:11:06.157 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:11:06.158 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:11:06.158 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:11:06.158 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:11:06.158 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:11:06.158 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:11:06.158 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:11:06.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.147+0000 7f79127df640 1 librados: starting msgr at 2024-09-16T04:11:06.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.147+0000 7f79127df640 1 librados: starting objecter 2024-09-16T04:11:06.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.151+0000 7f79127df640 1 Processor -- start 2024-09-16T04:11:06.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.151+0000 7f79127df640 1 -- start start 2024-09-16T04:11:06.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.151+0000 7f79127df640 1 librados: setting wanted keys 2024-09-16T04:11:06.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.151+0000 7f79127df640 1 librados: calling monclient init 2024-09-16T04:11:06.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.151+0000 7f79127df640 10 monclient: init 2024-09-16T04:11:06.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.151+0000 7f79127df640 10 monclient: _reopen_session rank -1 2024-09-16T04:11:06.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.151+0000 7f79127df640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:11:06.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.151+0000 7f79127df640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f78f40039c0 0x7f790c1ccf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:11:06.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.151+0000 7f79127df640 10 monclient(hunting): picked mon.noname-c con 0x7f78f40039c0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:11:06.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.151+0000 7f79127df640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f78f4004810 0x7f790c1cd4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:11:06.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.151+0000 7f79127df640 10 monclient(hunting): picked mon.noname-a con 0x7f78f4004810 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:11:06.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.151+0000 7f79127df640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f78f40052f0 0x7f790c11f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:11:06.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.151+0000 7f79127df640 10 monclient(hunting): picked mon.noname-b con 0x7f78f40052f0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:11:06.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.151+0000 7f79127df640 10 monclient(hunting): start opening mon connection 2024-09-16T04:11:06.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.151+0000 7f79127df640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f790c09eef0 con 0x7f78f4004810 2024-09-16T04:11:06.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.151+0000 7f79127df640 10 monclient(hunting): start opening mon connection 2024-09-16T04:11:06.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.151+0000 7f79127df640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f790c09ed70 con 0x7f78f40052f0 2024-09-16T04:11:06.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.151+0000 7f79127df640 10 monclient(hunting): start opening mon connection 2024-09-16T04:11:06.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.151+0000 7f79127df640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f790c135910 con 0x7f78f40039c0 2024-09-16T04:11:06.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.151+0000 7f79127df640 10 monclient(hunting): _renew_subs 2024-09-16T04:11:06.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.151+0000 7f79127df640 10 monclient(hunting): authenticate will time out at 2713.121484s 2024-09-16T04:11:06.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.151+0000 7f790b7fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f78f4004810 0x7f790c1cd4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:11:06.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.151+0000 7f790b7fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f78f4004810 0x7f790c1cd4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:41038/0 (socket says 172.21.15.42:41038) 2024-09-16T04:11:06.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.151+0000 7f7910d7c640 1 -- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f78f40052f0 msgr2=0x7f790c11f6c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T04:11:06.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.151+0000 7f790b7fe640 1 -- 172.21.15.42:0/2749737446 learned_addr learned my addr 172.21.15.42:0/2749737446 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:11:06.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.151+0000 7f790bfff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f78f40039c0 0x7f790c1ccf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:11:06.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.151+0000 7f790b7fe640 10 monclient(hunting): get_auth_request con 0x7f78f4004810 auth_method 0 2024-09-16T04:11:06.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.151+0000 7f790b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:11:06.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.151+0000 7f790b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:06.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.151+0000 7f790b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:06.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.151+0000 7f7910d7c640 1 --2- 172.21.15.42:0/2749737446 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f78f40052f0 0x7f790c11f6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:11:06.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.151+0000 7f790bfff640 10 monclient(hunting): get_auth_request con 0x7f78f40039c0 auth_method 0 2024-09-16T04:11:06.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.151+0000 7f790bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:11:06.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.151+0000 7f790bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:06.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.151+0000 7f790bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:06.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.151+0000 7f790b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:11:06.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.151+0000 7f790b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:11:06.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.151+0000 7f790b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:11:06.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.151+0000 7f790bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:11:06.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.151+0000 7f790bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:11:06.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.151+0000 7f790bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:11:06.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.151+0000 7f790b7fe640 10 monclient(hunting): handle_auth_done global_id 1034376 payload 824 2024-09-16T04:11:06.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.151+0000 7f790b7fe640 1 -- 172.21.15.42:0/2749737446 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f78f40039c0 msgr2=0x7f790c1ccf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:06.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.151+0000 7f790b7fe640 1 --2- 172.21.15.42:0/2749737446 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f78f40039c0 0x7f790c1ccf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:11:06.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.151+0000 7f790b7fe640 1 -- 172.21.15.42:0/2749737446 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f78f40052f0 msgr2=0x7f790c11f6c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:11:06.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.151+0000 7f790b7fe640 1 --2- 172.21.15.42:0/2749737446 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f78f40052f0 0x7f790c11f6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:11:06.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.151+0000 7f790b7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:11:06.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.151+0000 7f790b7fe640 1 monclient: found mon.noname-a 2024-09-16T04:11:06.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.151+0000 7f790b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:11:06.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.151+0000 7f790b7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:11:06.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.151+0000 7f790b7fe640 1 -- 172.21.15.42:0/2749737446 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f790c121c40 con 0x7f78f4004810 2024-09-16T04:11:06.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.151+0000 7f790b7fe640 1 --2- 172.21.15.42:0/2749737446 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f78f4004810 0x7f790c1cd4b0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f7900007fd0 tx=0x7f7900016e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:11:06.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.151+0000 7f7908ff9640 10 client.?.objecter ms_handle_connect 0x7f78f4004810 2024-09-16T04:11:06.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.151+0000 7f7908ff9640 10 client.?.objecter resend_mon_ops 2024-09-16T04:11:06.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.151+0000 7f7908ff9640 1 -- 172.21.15.42:0/2749737446 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f790000f040 con 0x7f78f4004810 2024-09-16T04:11:06.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.151+0000 7f7908ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:06.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.151+0000 7f7908ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:11:06.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.151+0000 7f7908ff9640 10 monclient: dump: 2024-09-16T04:11:06.167 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:11:06.167 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:11:06.167 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:11:06.167 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:11:06.167 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:06.167 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:11:06.167 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:11:06.168 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:11:06.168 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:11:06.168 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:11:06.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.151+0000 7f7908ff9640 10 monclient: _finish_auth 0 2024-09-16T04:11:06.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.151+0000 7f7908ff9640 10 monclient: _check_auth_tickets 2024-09-16T04:11:06.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.151+0000 7f7908ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:11:06.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.151+0000 7f7908ff9640 1 -- 172.21.15.42:0/2749737446 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7900000e50 con 0x7f78f4004810 2024-09-16T04:11:06.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.151+0000 7f79127df640 5 monclient: authenticate success, global_id 1034376 2024-09-16T04:11:06.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.151+0000 7f79127df640 10 monclient: _renew_subs 2024-09-16T04:11:06.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.151+0000 7f79127df640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:11:06.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.151+0000 7f79127df640 1 -- 172.21.15.42:0/2749737446 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f790c121ec0 con 0x7f78f4004810 2024-09-16T04:11:06.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.151+0000 7f7908ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T04:11:06.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.151+0000 7f7908ff9640 1 -- 172.21.15.42:0/2749737446 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7900018450 con 0x7f78f4004810 2024-09-16T04:11:06.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.151+0000 7f7908ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:06.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.151+0000 7f7908ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:11:06.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.151+0000 7f7908ff9640 10 monclient: dump: 2024-09-16T04:11:06.169 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:11:06.169 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:11:06.169 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:11:06.169 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:11:06.169 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:06.169 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:11:06.169 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:11:06.169 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:11:06.170 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:11:06.170 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:11:06.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.151+0000 7f79127df640 10 client.1034376.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:11:06.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.151+0000 7f7909ffb640 4 set_mon_vals no callback set 2024-09-16T04:11:06.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.151+0000 7f79127df640 10 monclient: _renew_subs 2024-09-16T04:11:06.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.151+0000 7f79127df640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:11:06.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.151+0000 7f79127df640 1 -- 172.21.15.42:0/2749737446 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f790c06e400 con 0x7f78f4004810 2024-09-16T04:11:06.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.151+0000 7f79127df640 1 librados: init done 2024-09-16T04:11:06.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.151+0000 7f7908ff9640 1 -- 172.21.15.42:0/2749737446 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f7900020020 con 0x7f78f4004810 2024-09-16T04:11:06.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.151+0000 7f79127df640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:06.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.151+0000 7f79127df640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:06.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.151+0000 7f79127df640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:11:06.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.151+0000 7f79127df640 1 -- 172.21.15.42:0/2749737446 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f790c09f610 con 0x7f78f4004810 2024-09-16T04:11:06.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.155+0000 7f7908ff9640 1 --2- 172.21.15.42:0/2749737446 >> v2:172.21.15.154:6812/254845342 conn(0x7f78f8037e10 0x7f78f803a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:11:06.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.155+0000 7f7908ff9640 1 -- 172.21.15.42:0/2749737446 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(706..706 src has 451..706) ==== 5040+0+0 (secure 0 0 0) 0x7f7900049840 con 0x7f78f4004810 2024-09-16T04:11:06.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.155+0000 7f7908ff9640 10 client.1034376.objecter ms_dispatch 0x7f790c000ba0 osd_map(706..706 src has 451..706) 2024-09-16T04:11:06.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.155+0000 7f7908ff9640 3 client.1034376.objecter handle_osd_map got epochs [706,706] > 0 2024-09-16T04:11:06.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.155+0000 7f7908ff9640 3 client.1034376.objecter handle_osd_map decoding full epoch 706 2024-09-16T04:11:06.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.155+0000 7f7908ff9640 20 client.1034376.objecter dump_active .. 0 homeless 2024-09-16T04:11:06.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.155+0000 7f790bfff640 1 --2- 172.21.15.42:0/2749737446 >> v2:172.21.15.154:6812/254845342 conn(0x7f78f8037e10 0x7f78f803a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:11:06.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.155+0000 7f790bfff640 10 monclient: get_auth_request con 0x7f78f8037e10 auth_method 0 2024-09-16T04:11:06.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.155+0000 7f790bfff640 1 --2- 172.21.15.42:0/2749737446 >> v2:172.21.15.154:6812/254845342 conn(0x7f78f8037e10 0x7f78f803a2d0 secure :-1 s=READY pgs=3942 cs=0 l=1 rev1=1 crypto rx=0x7f78fc0ae000 tx=0x7f78fc0a8000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:11:06.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.155+0000 7f7908ff9640 10 client.1034376.objecter ms_handle_connect 0x7f78f8037e10 2024-09-16T04:11:06.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.159+0000 7f7908ff9640 1 -- 172.21.15.42:0/2749737446 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f79000177a0 con 0x7f78f4004810 2024-09-16T04:11:06.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.159+0000 7f7908ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:06.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.159+0000 7f7908ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:11:06.388 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.383+0000 7f79127df640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:11:06.388 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.383+0000 7f79127df640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:11:06.388 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.383+0000 7f79127df640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:11:06.388 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.383+0000 7f79127df640 1 -- 172.21.15.42:0/2749737446 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f790c069040 con 0x7f78f4004810 2024-09-16T04:11:06.388 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.383+0000 7f7908ff9640 1 -- 172.21.15.42:0/2749737446 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1555 (secure 0 0 0) 0x7f790002ac90 con 0x7f78f4004810 2024-09-16T04:11:06.388 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.383+0000 7f7908ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:11:06.388 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.383+0000 7f7908ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:11:06.389 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":264,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:11:06.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.387+0000 7f79127df640 10 librados: watch_flush enter 2024-09-16T04:11:06.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.387+0000 7f79127df640 10 librados: watch_flush exit 2024-09-16T04:11:06.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.387+0000 7f79127df640 20 client.1034376.objecter shutdown clearing up homeless session... 2024-09-16T04:11:06.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.387+0000 7f79127df640 10 client.1034376.objecter successfully canceled tick 2024-09-16T04:11:06.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.387+0000 7f79127df640 1 -- 172.21.15.42:0/2749737446 >> v2:172.21.15.154:6812/254845342 conn(0x7f78f8037e10 msgr2=0x7f78f803a2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:06.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.387+0000 7f79127df640 1 --2- 172.21.15.42:0/2749737446 >> v2:172.21.15.154:6812/254845342 conn(0x7f78f8037e10 0x7f78f803a2d0 secure :-1 s=READY pgs=3942 cs=0 l=1 rev1=1 crypto rx=0x7f78fc0ae000 tx=0x7f78fc0a8000 comp rx=0 tx=0).stop 2024-09-16T04:11:06.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.387+0000 7f79127df640 10 monclient: shutdown 2024-09-16T04:11:06.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.387+0000 7f79127df640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:11:06.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.387+0000 7f79127df640 1 -- 172.21.15.42:0/2749737446 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f78f4004810 msgr2=0x7f790c1cd4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:06.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.387+0000 7f79127df640 1 --2- 172.21.15.42:0/2749737446 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f78f4004810 0x7f790c1cd4b0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f7900007fd0 tx=0x7f7900016e70 comp rx=0 tx=0).stop 2024-09-16T04:11:06.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.387+0000 7f79127df640 1 -- 172.21.15.42:0/2749737446 shutdown_connections 2024-09-16T04:11:06.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.387+0000 7f79127df640 1 --2- 172.21.15.42:0/2749737446 >> v2:172.21.15.154:6812/254845342 conn(0x7f78f8037e10 0x7f78f803a2d0 unknown :-1 s=CLOSED pgs=3942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:11:06.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.387+0000 7f79127df640 1 --2- 172.21.15.42:0/2749737446 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f78f40052f0 0x7f790c11f6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:11:06.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.387+0000 7f79127df640 1 --2- 172.21.15.42:0/2749737446 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f78f4004810 0x7f790c1cd4b0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:11:06.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.387+0000 7f79127df640 1 --2- 172.21.15.42:0/2749737446 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f78f40039c0 0x7f790c1ccf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:11:06.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.387+0000 7f79127df640 1 -- 172.21.15.42:0/2749737446 >> 172.21.15.42:0/2749737446 conn(0x7f790c0a2420 msgr2=0x7f790c0a45b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:11:06.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.387+0000 7f79127df640 1 -- 172.21.15.42:0/2749737446 shutdown_connections 2024-09-16T04:11:06.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.387+0000 7f79127df640 1 -- 172.21.15.42:0/2749737446 wait complete. 2024-09-16T04:11:06.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.387+0000 7f79127df640 1 librados: shutdown 2024-09-16T04:11:06.406 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T04:11:06.406 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-16T04:11:06.406 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T04:11:06.521 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.515+0000 7f60454a6640 10 monclient: get_monmap_and_config 2024-09-16T04:11:06.522 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.515+0000 7f60454a6640 10 monclient: build_initial_monmap 2024-09-16T04:11:06.522 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.515+0000 7f60454a6640 10 monclient: monmap: 2024-09-16T04:11:06.522 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:11:06.522 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:11:06.522 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:11:06.519836+0000 2024-09-16T04:11:06.522 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:11:06.519836+0000 2024-09-16T04:11:06.522 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:11:06.523 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:11:06.523 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:11:06.523 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:11:06.523 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:11:06.523 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:11:06.523 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:11:06.523 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:11:06.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.519+0000 7f60454a6640 1 Processor -- start 2024-09-16T04:11:06.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.519+0000 7f60454a6640 1 -- start start 2024-09-16T04:11:06.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.519+0000 7f60454a6640 10 monclient: init 2024-09-16T04:11:06.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.519+0000 7f60454a6640 10 monclient: _reopen_session rank -1 2024-09-16T04:11:06.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.519+0000 7f60454a6640 10 monclient: _add_conns ranks=[0,2,4,3,1] 2024-09-16T04:11:06.536 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.519+0000 7f60454a6640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f60400a39b0 addr v1:172.21.15.42:6789/0 2024-09-16T04:11:06.536 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.519+0000 7f60454a6640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f60400a3d90 0x7f604013c380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:11:06.536 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.519+0000 7f60454a6640 10 monclient(hunting): picked mon.noname-b con 0x7f60400a3d90 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:11:06.536 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.519+0000 7f60454a6640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f604013c8c0 0x7f604013ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:11:06.536 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.519+0000 7f60454a6640 10 monclient(hunting): picked mon.noname-c con 0x7f604013c8c0 addr v2:172.21.15.154:3300/0 2024-09-16T04:11:06.536 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.519+0000 7f60454a6640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6040134290 con 0x7f60400a39b0 2024-09-16T04:11:06.537 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.519+0000 7f60454a6640 10 monclient(hunting): start opening mon connection 2024-09-16T04:11:06.537 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.519+0000 7f60454a6640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f604009c140 con 0x7f604013c8c0 2024-09-16T04:11:06.537 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.519+0000 7f60454a6640 10 monclient(hunting): start opening mon connection 2024-09-16T04:11:06.537 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.519+0000 7f60454a6640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f604009c420 con 0x7f60400a3d90 2024-09-16T04:11:06.537 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.519+0000 7f60454a6640 10 monclient(hunting): _renew_subs 2024-09-16T04:11:06.537 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.519+0000 7f60454a6640 10 monclient(hunting): authenticate will time out at 2713.489412s 2024-09-16T04:11:06.537 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.519+0000 7f603effd640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f60400a39b0 0x7f6040133b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:33828/0 (socket says 172.21.15.42:33828) 2024-09-16T04:11:06.537 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.519+0000 7f603effd640 1 -- 172.21.15.42:0/498298959 learned_addr learned my addr 172.21.15.42:0/498298959 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T04:11:06.537 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.519+0000 7f603e7fc640 1 --2- 172.21.15.42:0/498298959 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f60400a3d90 0x7f604013c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:11:06.537 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.519+0000 7f603e7fc640 10 monclient(hunting): get_auth_request con 0x7f60400a3d90 auth_method 0 2024-09-16T04:11:06.538 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.519+0000 7f603e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:11:06.538 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.519+0000 7f603e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:06.538 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.519+0000 7f603e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:06.538 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.519+0000 7f603f7fe640 1 -- 172.21.15.42:0/498298959 >> v2:172.21.15.154:3300/0 conn(0x7f604013c8c0 msgr2=0x7f604013ecb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T04:11:06.538 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.519+0000 7f603f7fe640 1 --2- 172.21.15.42:0/498298959 >> v2:172.21.15.154:3300/0 conn(0x7f604013c8c0 0x7f604013ecb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:11:06.538 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.519+0000 7f603e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:11:06.538 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.519+0000 7f603e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:11:06.538 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.519+0000 7f603e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:11:06.538 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.519+0000 7f603dffb640 1 -- 172.21.15.42:0/498298959 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1894195440 0 0) 0x7f6040134290 con 0x7f60400a39b0 2024-09-16T04:11:06.538 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.519+0000 7f603dffb640 10 monclient(hunting): _negotiate 2024-09-16T04:11:06.538 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.519+0000 7f603dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:06.538 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.519+0000 7f603dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:06.538 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.519+0000 7f603dffb640 10 monclient(hunting): my global_id is 1034379 2024-09-16T04:11:06.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.519+0000 7f603dffb640 1 -- 172.21.15.42:0/498298959 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6028003530 con 0x7f60400a39b0 2024-09-16T04:11:06.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.519+0000 7f603e7fc640 10 monclient(hunting): handle_auth_done global_id 1014491 payload 274 2024-09-16T04:11:06.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.519+0000 7f603e7fc640 1 -- 172.21.15.42:0/498298959 >> v2:172.21.15.154:3300/0 conn(0x7f604013c8c0 msgr2=0x7f604013ecb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:11:06.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.519+0000 7f603e7fc640 1 --2- 172.21.15.42:0/498298959 >> v2:172.21.15.154:3300/0 conn(0x7f604013c8c0 0x7f604013ecb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:11:06.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.519+0000 7f603e7fc640 1 -- 172.21.15.42:0/498298959 >> v1:172.21.15.42:6789/0 conn(0x7f60400a39b0 legacy=0x7f6040133b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:06.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.519+0000 7f603e7fc640 10 monclient: _finish_hunting 0 2024-09-16T04:11:06.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.519+0000 7f603e7fc640 1 monclient: found mon.noname-b 2024-09-16T04:11:06.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.519+0000 7f603e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:11:06.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.519+0000 7f603e7fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T04:11:06.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.519+0000 7f603e7fc640 1 -- 172.21.15.42:0/498298959 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f604006dd60 con 0x7f60400a3d90 2024-09-16T04:11:06.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.523+0000 7f603e7fc640 1 --2- 172.21.15.42:0/498298959 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f60400a3d90 0x7f604013c380 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f60340097c0 tx=0x7f603401c380 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b5d35e923881158c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:11:06.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.523+0000 7f603dffb640 1 -- 172.21.15.42:0/498298959 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60340b0070 con 0x7f60400a3d90 2024-09-16T04:11:06.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.523+0000 7f603dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:06.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.523+0000 7f603dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:11:06.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.523+0000 7f603dffb640 10 monclient: dump: 2024-09-16T04:11:06.540 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:11:06.540 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:11:06.540 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:11:06.540 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:11:06.540 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:06.540 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:11:06.540 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:11:06.540 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:11:06.540 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:11:06.540 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:11:06.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.523+0000 7f603dffb640 10 monclient: _finish_auth 0 2024-09-16T04:11:06.541 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.523+0000 7f603dffb640 10 monclient: _check_auth_tickets 2024-09-16T04:11:06.541 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.523+0000 7f603dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:11:06.541 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.523+0000 7f60454a6640 5 monclient: authenticate success, global_id 1014491 2024-09-16T04:11:06.541 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.523+0000 7f60454a6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:11:06.541 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.523+0000 7f603dffb640 1 -- 172.21.15.42:0/498298959 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f60340ab600 con 0x7f60400a3d90 2024-09-16T04:11:06.541 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.523+0000 7f603dffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:11:06.541 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.523+0000 7f60454a6640 10 monclient: get_monmap_and_config success 2024-09-16T04:11:06.541 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.523+0000 7f60454a6640 4 set_mon_vals no callback set 2024-09-16T04:11:06.541 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.523+0000 7f603dffb640 1 -- 172.21.15.42:0/498298959 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60340ab9b0 con 0x7f60400a3d90 2024-09-16T04:11:06.541 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.523+0000 7f603dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:06.541 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.523+0000 7f603dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:11:06.541 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.523+0000 7f603dffb640 10 monclient: dump: 2024-09-16T04:11:06.541 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:11:06.542 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:11:06.542 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:11:06.542 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:11:06.542 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:06.542 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:11:06.542 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:11:06.542 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:11:06.542 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:11:06.542 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:11:06.542 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.523+0000 7f60454a6640 10 monclient: shutdown 2024-09-16T04:11:06.542 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.523+0000 7f60454a6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:11:06.542 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.523+0000 7f60454a6640 1 -- 172.21.15.42:0/498298959 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f60400a3d90 msgr2=0x7f604013c380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:06.543 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.523+0000 7f60454a6640 1 --2- 172.21.15.42:0/498298959 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f60400a3d90 0x7f604013c380 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f60340097c0 tx=0x7f603401c380 comp rx=0 tx=0).stop 2024-09-16T04:11:06.543 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.523+0000 7f60454a6640 1 -- 172.21.15.42:0/498298959 shutdown_connections 2024-09-16T04:11:06.543 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.523+0000 7f60454a6640 1 --2- 172.21.15.42:0/498298959 >> v2:172.21.15.154:3300/0 conn(0x7f604013c8c0 0x7f604013ecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:11:06.543 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.523+0000 7f60454a6640 1 --2- 172.21.15.42:0/498298959 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f60400a3d90 0x7f604013c380 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:11:06.543 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.523+0000 7f60454a6640 1 -- 172.21.15.42:0/498298959 >> 172.21.15.42:0/498298959 conn(0x7f60400a9fb0 msgr2=0x7f60400aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:11:06.543 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.523+0000 7f60454a6640 1 -- 172.21.15.42:0/498298959 shutdown_connections 2024-09-16T04:11:06.543 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.523+0000 7f60454a6640 1 -- 172.21.15.42:0/498298959 wait complete. 2024-09-16T04:11:06.543 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.523+0000 7f60454a6640 10 monclient: build_initial_monmap 2024-09-16T04:11:06.543 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.523+0000 7f60454a6640 10 monclient: monmap: 2024-09-16T04:11:06.543 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:11:06.543 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:11:06.543 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:11:06.543 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:11:06.544 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:11:06.544 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:11:06.544 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:11:06.544 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:11:06.544 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:11:06.544 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:11:06.544 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.523+0000 7f60454a6640 1 librados: starting msgr at 2024-09-16T04:11:06.544 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.523+0000 7f60454a6640 1 librados: starting objecter 2024-09-16T04:11:06.544 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.523+0000 7f60454a6640 1 Processor -- start 2024-09-16T04:11:06.544 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.523+0000 7f60454a6640 1 -- start start 2024-09-16T04:11:06.544 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.523+0000 7f60454a6640 1 librados: setting wanted keys 2024-09-16T04:11:06.544 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.523+0000 7f60454a6640 1 librados: calling monclient init 2024-09-16T04:11:06.544 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.523+0000 7f60454a6640 10 monclient: init 2024-09-16T04:11:06.545 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.523+0000 7f60454a6640 10 monclient: _reopen_session rank -1 2024-09-16T04:11:06.545 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.523+0000 7f60454a6640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:11:06.545 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.523+0000 7f60454a6640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f60400a39b0 0x7f60401309b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:11:06.545 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.523+0000 7f60454a6640 10 monclient(hunting): picked mon.noname-a con 0x7f60400a39b0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:11:06.545 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.523+0000 7f60454a6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f60400a3d90 0x7f604013a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:11:06.545 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.523+0000 7f60454a6640 10 monclient(hunting): picked mon.noname-b con 0x7f60400a3d90 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:11:06.545 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.523+0000 7f60454a6640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f604013c8c0 0x7f604013a630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:11:06.545 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.523+0000 7f60454a6640 10 monclient(hunting): picked mon.noname-c con 0x7f604013c8c0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:11:06.545 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.523+0000 7f60454a6640 10 monclient(hunting): start opening mon connection 2024-09-16T04:11:06.545 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.523+0000 7f60454a6640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f6040141da0 con 0x7f60400a39b0 2024-09-16T04:11:06.545 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.523+0000 7f60454a6640 10 monclient(hunting): start opening mon connection 2024-09-16T04:11:06.545 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.523+0000 7f60454a6640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f6040141c20 con 0x7f60400a3d90 2024-09-16T04:11:06.545 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.523+0000 7f60454a6640 10 monclient(hunting): start opening mon connection 2024-09-16T04:11:06.545 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.523+0000 7f60454a6640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f6040141f20 con 0x7f604013c8c0 2024-09-16T04:11:06.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.523+0000 7f60454a6640 10 monclient(hunting): _renew_subs 2024-09-16T04:11:06.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.523+0000 7f60454a6640 10 monclient(hunting): authenticate will time out at 2713.494359s 2024-09-16T04:11:06.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.523+0000 7f603effd640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f60400a39b0 0x7f60401309b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:11:06.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.523+0000 7f603f7fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f604013c8c0 0x7f604013a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:11:06.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.523+0000 7f603effd640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f60400a39b0 0x7f60401309b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:41048/0 (socket says 172.21.15.42:41048) 2024-09-16T04:11:06.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.523+0000 7f603f7fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f604013c8c0 0x7f604013a630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:50074/0 (socket says 172.21.15.42:50074) 2024-09-16T04:11:06.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.523+0000 7f603effd640 1 -- 172.21.15.42:0/1903035896 learned_addr learned my addr 172.21.15.42:0/1903035896 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:11:06.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.523+0000 7f603effd640 10 monclient(hunting): get_auth_request con 0x7f60400a39b0 auth_method 0 2024-09-16T04:11:06.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.523+0000 7f603effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:11:06.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.523+0000 7f603effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:06.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.523+0000 7f603effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:06.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.523+0000 7f603f7fe640 10 monclient(hunting): get_auth_request con 0x7f604013c8c0 auth_method 0 2024-09-16T04:11:06.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.523+0000 7f603f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:11:06.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.523+0000 7f603f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:06.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.523+0000 7f603f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:06.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.523+0000 7f603e7fc640 1 -- 172.21.15.42:0/1903035896 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f60400a3d90 msgr2=0x7f604013a0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T04:11:06.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.523+0000 7f603e7fc640 1 --2- 172.21.15.42:0/1903035896 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f60400a3d90 0x7f604013a0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:11:06.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.523+0000 7f603effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:11:06.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.523+0000 7f603effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:11:06.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.523+0000 7f603effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:11:06.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.523+0000 7f603f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:11:06.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.523+0000 7f603f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:11:06.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.523+0000 7f603f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:11:06.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.523+0000 7f603effd640 10 monclient(hunting): handle_auth_done global_id 1034382 payload 824 2024-09-16T04:11:06.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.523+0000 7f603effd640 1 -- 172.21.15.42:0/1903035896 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f604013c8c0 msgr2=0x7f604013a630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:06.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.523+0000 7f603effd640 1 --2- 172.21.15.42:0/1903035896 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f604013c8c0 0x7f604013a630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:11:06.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.523+0000 7f603effd640 1 -- 172.21.15.42:0/1903035896 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f60400a3d90 msgr2=0x7f604013a0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:11:06.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.523+0000 7f603effd640 1 --2- 172.21.15.42:0/1903035896 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f60400a3d90 0x7f604013a0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:11:06.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.523+0000 7f603effd640 10 monclient: _finish_hunting 0 2024-09-16T04:11:06.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.523+0000 7f603effd640 1 monclient: found mon.noname-a 2024-09-16T04:11:06.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.523+0000 7f603effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:11:06.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.523+0000 7f603effd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:11:06.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.523+0000 7f603effd640 1 -- 172.21.15.42:0/1903035896 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f604009e790 con 0x7f60400a39b0 2024-09-16T04:11:06.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.523+0000 7f603effd640 1 --2- 172.21.15.42:0/1903035896 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f60400a39b0 0x7f60401309b0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f603000cbc0 tx=0x7f603000a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:11:06.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.523+0000 7f60277fe640 10 client.?.objecter ms_handle_connect 0x7f60400a39b0 2024-09-16T04:11:06.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.523+0000 7f60277fe640 10 client.?.objecter resend_mon_ops 2024-09-16T04:11:06.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.523+0000 7f60277fe640 1 -- 172.21.15.42:0/1903035896 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6030017070 con 0x7f60400a39b0 2024-09-16T04:11:06.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.523+0000 7f60277fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:06.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.523+0000 7f60277fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:11:06.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.523+0000 7f60277fe640 10 monclient: dump: 2024-09-16T04:11:06.549 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:11:06.549 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:11:06.549 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:11:06.549 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:11:06.549 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:06.549 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:11:06.549 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:11:06.549 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:11:06.550 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:11:06.550 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:11:06.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.523+0000 7f60277fe640 10 monclient: _finish_auth 0 2024-09-16T04:11:06.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.523+0000 7f60277fe640 10 monclient: _check_auth_tickets 2024-09-16T04:11:06.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.523+0000 7f60277fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:11:06.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.523+0000 7f60454a6640 5 monclient: authenticate success, global_id 1034382 2024-09-16T04:11:06.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.523+0000 7f60277fe640 1 -- 172.21.15.42:0/1903035896 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f603000ad40 con 0x7f60400a39b0 2024-09-16T04:11:06.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.523+0000 7f60454a6640 10 monclient: _renew_subs 2024-09-16T04:11:06.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.523+0000 7f60454a6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:11:06.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.523+0000 7f60454a6640 1 -- 172.21.15.42:0/1903035896 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f604006f680 con 0x7f60400a39b0 2024-09-16T04:11:06.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.523+0000 7f60277fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:11:06.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.523+0000 7f60277fe640 1 -- 172.21.15.42:0/1903035896 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6030012600 con 0x7f60400a39b0 2024-09-16T04:11:06.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.523+0000 7f60277fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:06.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.523+0000 7f60277fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:11:06.551 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.523+0000 7f60277fe640 10 monclient: dump: 2024-09-16T04:11:06.551 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:11:06.551 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:11:06.551 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:11:06.551 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:11:06.551 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:06.551 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:11:06.551 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:11:06.551 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:11:06.551 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:11:06.551 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:11:06.551 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.523+0000 7f60454a6640 10 client.1034382.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:11:06.551 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.523+0000 7f603d7fa640 4 set_mon_vals no callback set 2024-09-16T04:11:06.552 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.523+0000 7f60454a6640 10 monclient: _renew_subs 2024-09-16T04:11:06.552 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.523+0000 7f60454a6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:11:06.552 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.523+0000 7f60454a6640 1 -- 172.21.15.42:0/1903035896 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f604009e4b0 con 0x7f60400a39b0 2024-09-16T04:11:06.552 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.523+0000 7f60454a6640 1 librados: init done 2024-09-16T04:11:06.552 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.523+0000 7f60454a6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:06.552 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.523+0000 7f60454a6640 1 --2- 172.21.15.42:0/1903035896 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f604009f170 0x7f604013ab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:11:06.552 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.523+0000 7f60454a6640 10 monclient(hunting): start opening mon connection 2024-09-16T04:11:06.552 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.523+0000 7f60454a6640 1 -- 172.21.15.42:0/1903035896 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f604009e060 con 0x7f604009f170 2024-09-16T04:11:06.552 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.527+0000 7f603e7fc640 1 --2- 172.21.15.42:0/1903035896 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f604009f170 0x7f604013ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:11:06.552 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.527+0000 7f603e7fc640 10 monclient: get_auth_request con 0x7f604009f170 auth_method 0 2024-09-16T04:11:06.552 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.527+0000 7f603e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:11:06.552 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.527+0000 7f603e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:06.552 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.527+0000 7f603e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:06.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.527+0000 7f603e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:11:06.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.527+0000 7f603e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:11:06.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.527+0000 7f60277fe640 1 -- 172.21.15.42:0/1903035896 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f603001a070 con 0x7f60400a39b0 2024-09-16T04:11:06.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.527+0000 7f603e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:11:06.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.527+0000 7f603e7fc640 10 monclient(hunting): handle_auth_done global_id 1034385 payload 824 2024-09-16T04:11:06.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.527+0000 7f603e7fc640 1 -- 172.21.15.42:0/1903035896 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6040133b80 con 0x7f604009f170 2024-09-16T04:11:06.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.527+0000 7f60277fe640 1 --2- 172.21.15.42:0/1903035896 >> v2:172.21.15.154:6812/254845342 conn(0x7f602c0377c0 0x7f602c039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:11:06.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.527+0000 7f60277fe640 1 -- 172.21.15.42:0/1903035896 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(706..706 src has 451..706) ==== 5040+0+0 (secure 0 0 0) 0x7f6030048a20 con 0x7f60400a39b0 2024-09-16T04:11:06.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.527+0000 7f60277fe640 10 client.1034382.objecter ms_dispatch 0x7f6040000ba0 osd_map(706..706 src has 451..706) 2024-09-16T04:11:06.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.527+0000 7f60277fe640 3 client.1034382.objecter handle_osd_map got epochs [706,706] > 0 2024-09-16T04:11:06.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.527+0000 7f60277fe640 3 client.1034382.objecter handle_osd_map decoding full epoch 706 2024-09-16T04:11:06.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.527+0000 7f603e7fc640 1 --2- 172.21.15.42:0/1903035896 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f604009f170 0x7f604013ab70 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f60400a7750 tx=0x7f603401abc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:11:06.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.527+0000 7f60277fe640 20 client.1034382.objecter dump_active .. 0 homeless 2024-09-16T04:11:06.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.527+0000 7f60277fe640 10 client.1034382.objecter ms_handle_connect 0x7f604009f170 2024-09-16T04:11:06.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.527+0000 7f60277fe640 10 client.1034382.objecter resend_mon_ops 2024-09-16T04:11:06.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.527+0000 7f60277fe640 1 -- 172.21.15.42:0/1903035896 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60340b9030 con 0x7f604009f170 2024-09-16T04:11:06.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.527+0000 7f60277fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:06.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.527+0000 7f60277fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:11:06.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.527+0000 7f60277fe640 10 monclient: dump: 2024-09-16T04:11:06.554 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:11:06.554 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:11:06.554 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:11:06.554 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:11:06.554 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:06.554 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:11:06.554 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:11:06.554 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:11:06.555 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:11:06.555 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:11:06.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.527+0000 7f603f7fe640 1 --2- 172.21.15.42:0/1903035896 >> v2:172.21.15.154:6812/254845342 conn(0x7f602c0377c0 0x7f602c039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:11:06.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.527+0000 7f603f7fe640 10 monclient: get_auth_request con 0x7f602c0377c0 auth_method 0 2024-09-16T04:11:06.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.527+0000 7f60277fe640 1 -- 172.21.15.42:0/1903035896 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f60340b1040 con 0x7f604009f170 2024-09-16T04:11:06.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.527+0000 7f60277fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:06.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.527+0000 7f60277fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:11:06.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.527+0000 7f60277fe640 1 -- 172.21.15.42:0/1903035896 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f604009f170 msgr2=0x7f604013ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:06.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.527+0000 7f60277fe640 1 --2- 172.21.15.42:0/1903035896 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f604009f170 0x7f604013ab70 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f60400a7750 tx=0x7f603401abc0 comp rx=0 tx=0).stop 2024-09-16T04:11:06.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.527+0000 7f60277fe640 1 -- 172.21.15.42:0/1903035896 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f604009f170 msgr2=0x7f604013ab70 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:11:06.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.527+0000 7f60277fe640 1 --2- 172.21.15.42:0/1903035896 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f604009f170 0x7f604013ab70 secure :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f60400a7750 tx=0x7f603401abc0 comp rx=0 tx=0).stop 2024-09-16T04:11:06.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.527+0000 7f60277fe640 10 client.1034382.objecter ms_handle_connect 0x7f602c0377c0 2024-09-16T04:11:06.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.527+0000 7f603f7fe640 1 --2- 172.21.15.42:0/1903035896 >> v2:172.21.15.154:6812/254845342 conn(0x7f602c0377c0 0x7f602c039c80 secure :-1 s=READY pgs=3943 cs=0 l=1 rev1=1 crypto rx=0x7f6038006fd0 tx=0x7f6038006f60 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:11:06.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.531+0000 7f60454a6640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:11:06.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.531+0000 7f60454a6640 1 --2- 172.21.15.42:0/1903035896 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6040130ef0 0x7f604013b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:11:06.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.531+0000 7f60454a6640 10 monclient(hunting): start opening mon connection 2024-09-16T04:11:06.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.531+0000 7f60454a6640 1 -- 172.21.15.42:0/1903035896 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f60400a5750 con 0x7f6040130ef0 2024-09-16T04:11:06.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.531+0000 7f603e7fc640 1 --2- 172.21.15.42:0/1903035896 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6040130ef0 0x7f604013b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:11:06.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.531+0000 7f603e7fc640 10 monclient: get_auth_request con 0x7f6040130ef0 auth_method 0 2024-09-16T04:11:06.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.531+0000 7f603e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:11:06.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.531+0000 7f603e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:06.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.531+0000 7f603e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:06.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.531+0000 7f603e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:11:06.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.531+0000 7f603e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:11:06.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.531+0000 7f603e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:11:06.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.531+0000 7f603e7fc640 10 monclient(hunting): handle_auth_done global_id 1034388 payload 824 2024-09-16T04:11:06.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.531+0000 7f603e7fc640 1 -- 172.21.15.42:0/1903035896 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f6040068e40 con 0x7f6040130ef0 2024-09-16T04:11:06.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.531+0000 7f60277fe640 10 client.1034382.objecter ms_handle_connect 0x7f6040130ef0 2024-09-16T04:11:06.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.531+0000 7f60277fe640 10 client.1034382.objecter resend_mon_ops 2024-09-16T04:11:06.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.531+0000 7f603e7fc640 1 --2- 172.21.15.42:0/1903035896 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6040130ef0 0x7f604013b0b0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f6034000c10 tx=0x7f6034019120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:11:06.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.531+0000 7f60277fe640 1 -- 172.21.15.42:0/1903035896 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60340b9040 con 0x7f6040130ef0 2024-09-16T04:11:06.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.531+0000 7f60277fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:06.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.531+0000 7f60277fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:11:06.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.531+0000 7f60277fe640 10 monclient: dump: 2024-09-16T04:11:06.559 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:11:06.559 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:11:06.559 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:11:06.559 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:11:06.560 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:06.560 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:11:06.560 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:11:06.560 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:11:06.560 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:11:06.560 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:11:06.560 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.531+0000 7f60277fe640 1 -- 172.21.15.42:0/1903035896 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4871 (secure 0 0 0) 0x7f60340be460 con 0x7f6040130ef0 2024-09-16T04:11:06.560 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.531+0000 7f60277fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:11:06.560 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.531+0000 7f60277fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:11:06.560 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.531+0000 7f60277fe640 1 -- 172.21.15.42:0/1903035896 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6040130ef0 msgr2=0x7f604013b0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:06.560 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.531+0000 7f60277fe640 1 --2- 172.21.15.42:0/1903035896 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6040130ef0 0x7f604013b0b0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f6034000c10 tx=0x7f6034019120 comp rx=0 tx=0).stop 2024-09-16T04:11:06.560 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.531+0000 7f60277fe640 1 -- 172.21.15.42:0/1903035896 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6040130ef0 msgr2=0x7f604013b0b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:11:06.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.531+0000 7f60277fe640 1 --2- 172.21.15.42:0/1903035896 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6040130ef0 0x7f604013b0b0 secure :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f6034000c10 tx=0x7f6034019120 comp rx=0 tx=0).stop 2024-09-16T04:11:06.561 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T04:11:06.561 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:11:06.561 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:11:06.561 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "leader", 2024-09-16T04:11:06.561 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 264, 2024-09-16T04:11:06.561 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 146466, 2024-09-16T04:11:06.561 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T04:11:06.561 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T04:11:06.561 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T04:11:06.562 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:11:06.562 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 0, 2024-09-16T04:11:06.562 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:11:06.562 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T04:11:06.562 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T04:11:06.562 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:11:06.562 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:11:06.562 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:11:06.562 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:11:06.562 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:11:06.562 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:11:06.562 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:11:06.562 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:11:06.563 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:11:06.563 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:11:06.563 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:11:06.563 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:11:06.563 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:11:06.563 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T04:11:06.563 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:11:06.563 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:11:06.563 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:11:06.563 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:11:06.563 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:11:06.563 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:11:06.563 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:11:06.563 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:11:06.564 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:11:06.564 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:11:06.564 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:11:06.564 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:11:06.564 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:11:06.564 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T04:11:06.564 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T04:11:06.564 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T04:11:06.564 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T04:11:06.564 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T04:11:06.564 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T04:11:06.564 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:11:06.564 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:11:06.565 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T04:11:06.565 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T04:11:06.565 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T04:11:06.565 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T04:11:06.565 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T04:11:06.565 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T04:11:06.565 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T04:11:06.565 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:11:06.565 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T04:11:06.565 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:11:06.565 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:11:06.565 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:11:06.565 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:11:06.566 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:11:06.566 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:11:06.566 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:11:06.566 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:11:06.566 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:11:06.566 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:11:06.566 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:11:06.566 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:11:06.566 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T04:11:06.566 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:11:06.566 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T04:11:06.566 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:11:06.566 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:11:06.566 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:11:06.567 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:11:06.567 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:11:06.567 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:11:06.567 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:11:06.567 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T04:11:06.567 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:11:06.567 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:11:06.567 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:11:06.567 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:11:06.567 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T04:11:06.567 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:11:06.567 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:11:06.567 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:11:06.568 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:11:06.568 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T04:11:06.568 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T04:11:06.568 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:11:06.568 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:11:06.568 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:11:06.568 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:11:06.568 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:11:06.568 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:11:06.568 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:11:06.568 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:11:06.568 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:11:06.568 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:11:06.569 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:11:06.569 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T04:11:06.569 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:11:06.569 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:11:06.569 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:11:06.569 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:11:06.569 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T04:11:06.569 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:11:06.569 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:11:06.569 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:11:06.569 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:11:06.569 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T04:11:06.570 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T04:11:06.570 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:11:06.570 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:11:06.570 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:11:06.570 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:11:06.570 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:11:06.570 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:11:06.570 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:11:06.570 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:11:06.570 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:11:06.570 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:11:06.570 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:11:06.570 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T04:11:06.571 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:11:06.571 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:11:06.571 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:11:06.571 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:11:06.571 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T04:11:06.571 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:11:06.571 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:11:06.571 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:11:06.571 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:11:06.571 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T04:11:06.571 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T04:11:06.571 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:11:06.571 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:11:06.571 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:11:06.572 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:11:06.572 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:11:06.572 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:11:06.572 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T04:11:06.572 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T04:11:06.572 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:11:06.572 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:11:06.572 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:11:06.572 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:11:06.572 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:11:06.572 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:11:06.572 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T04:11:06.572 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:11:06.573 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:11:06.573 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:11:06.573 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 2 2024-09-16T04:11:06.573 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:11:06.573 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:11:06.573 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T04:11:06.573 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:11:06.573 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:11:06.573 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:11:06.573 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 5 2024-09-16T04:11:06.573 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:11:06.573 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:11:06.573 INFO:teuthology.orchestra.run.smithi042.stdout: "mgr": [ 2024-09-16T04:11:06.573 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:11:06.574 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:11:06.574 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:11:06.574 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:11:06.574 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:11:06.574 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:11:06.574 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:11:06.574 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T04:11:06.574 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T04:11:06.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.535+0000 7f60454a6640 10 librados: watch_flush enter 2024-09-16T04:11:06.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.535+0000 7f60454a6640 10 librados: watch_flush exit 2024-09-16T04:11:06.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.535+0000 7f60454a6640 20 client.1034382.objecter shutdown clearing up homeless session... 2024-09-16T04:11:06.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.535+0000 7f60454a6640 10 client.1034382.objecter successfully canceled tick 2024-09-16T04:11:06.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.535+0000 7f60454a6640 1 -- 172.21.15.42:0/1903035896 >> v2:172.21.15.154:6812/254845342 conn(0x7f602c0377c0 msgr2=0x7f602c039c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:06.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.535+0000 7f60454a6640 1 --2- 172.21.15.42:0/1903035896 >> v2:172.21.15.154:6812/254845342 conn(0x7f602c0377c0 0x7f602c039c80 secure :-1 s=READY pgs=3943 cs=0 l=1 rev1=1 crypto rx=0x7f6038006fd0 tx=0x7f6038006f60 comp rx=0 tx=0).stop 2024-09-16T04:11:06.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.535+0000 7f60454a6640 10 monclient: shutdown 2024-09-16T04:11:06.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.535+0000 7f60454a6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:11:06.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.535+0000 7f60454a6640 1 -- 172.21.15.42:0/1903035896 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f60400a39b0 msgr2=0x7f60401309b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:06.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.535+0000 7f60454a6640 1 --2- 172.21.15.42:0/1903035896 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f60400a39b0 0x7f60401309b0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f603000cbc0 tx=0x7f603000a430 comp rx=0 tx=0).stop 2024-09-16T04:11:06.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.535+0000 7f603f7fe640 1 -- 172.21.15.42:0/1903035896 reap_dead start 2024-09-16T04:11:06.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.535+0000 7f60454a6640 1 -- 172.21.15.42:0/1903035896 shutdown_connections 2024-09-16T04:11:06.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.535+0000 7f60454a6640 1 --2- 172.21.15.42:0/1903035896 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f60400a39b0 0x7f60401309b0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:11:06.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.535+0000 7f60454a6640 1 -- 172.21.15.42:0/1903035896 >> 172.21.15.42:0/1903035896 conn(0x7f60400a9fb0 msgr2=0x7f60400a9490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:11:06.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.535+0000 7f60454a6640 1 -- 172.21.15.42:0/1903035896 shutdown_connections 2024-09-16T04:11:06.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.535+0000 7f60454a6640 1 -- 172.21.15.42:0/1903035896 wait complete. 2024-09-16T04:11:06.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.535+0000 7f60454a6640 1 librados: shutdown 2024-09-16T04:11:06.577 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T04:11:06.577 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-16T04:11:06.578 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T04:11:06.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.691+0000 7f28c2985640 10 monclient: get_monmap_and_config 2024-09-16T04:11:06.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.691+0000 7f28c2985640 10 monclient: build_initial_monmap 2024-09-16T04:11:06.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.691+0000 7f28c2985640 10 monclient: monmap: 2024-09-16T04:11:06.695 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:11:06.695 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:11:06.695 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:11:06.693766+0000 2024-09-16T04:11:06.695 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:11:06.693766+0000 2024-09-16T04:11:06.696 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:11:06.696 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:11:06.696 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:11:06.696 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:11:06.696 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:11:06.696 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:11:06.696 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:11:06.696 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:11:06.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.691+0000 7f28c2985640 1 Processor -- start 2024-09-16T04:11:06.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.691+0000 7f28c2985640 1 -- start start 2024-09-16T04:11:06.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.691+0000 7f28c2985640 10 monclient: init 2024-09-16T04:11:06.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.691+0000 7f28c2985640 10 monclient: _reopen_session rank -1 2024-09-16T04:11:06.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.691+0000 7f28c2985640 10 monclient: _add_conns ranks=[1,3,0,2,4] 2024-09-16T04:11:06.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.691+0000 7f28c2985640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f28bc0a39b0 addr v1:172.21.15.154:6789/0 2024-09-16T04:11:06.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.691+0000 7f28c2985640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f28bc0a3d90 0x7f28bc13c380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:11:06.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.691+0000 7f28c2985640 10 monclient(hunting): picked mon.noname-a con 0x7f28bc0a3d90 addr v2:172.21.15.42:3300/0 2024-09-16T04:11:06.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.691+0000 7f28c2985640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f28bc13c8c0 addr v1:172.21.15.42:6789/0 2024-09-16T04:11:06.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.691+0000 7f28c2985640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f28bc134290 con 0x7f28bc13c8c0 2024-09-16T04:11:06.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.691+0000 7f28c2985640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f28bc09c140 con 0x7f28bc0a39b0 2024-09-16T04:11:06.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.691+0000 7f28c2985640 10 monclient(hunting): start opening mon connection 2024-09-16T04:11:06.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.691+0000 7f28c2985640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f28bc09c420 con 0x7f28bc0a3d90 2024-09-16T04:11:06.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.691+0000 7f28c2985640 10 monclient(hunting): _renew_subs 2024-09-16T04:11:06.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.691+0000 7f28c2985640 10 monclient(hunting): authenticate will time out at 2713.663972s 2024-09-16T04:11:06.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.695+0000 7f28b3fff640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f28bc0a3d90 0x7f28bc13c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:11:06.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.695+0000 7f28bbfff640 1 -- >> v1:172.21.15.154:6789/0 conn(0x7f28bc0a39b0 legacy=0x7f28bc133b80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.154:6789/0 2024-09-16T04:11:06.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.695+0000 7f28b3fff640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f28bc0a3d90 0x7f28bc13c380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:41072/0 (socket says 172.21.15.42:41072) 2024-09-16T04:11:06.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.695+0000 7f28b3fff640 1 -- 172.21.15.42:0/2918593548 learned_addr learned my addr 172.21.15.42:0/2918593548 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:11:06.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.695+0000 7f28b3fff640 10 monclient(hunting): get_auth_request con 0x7f28bc0a3d90 auth_method 0 2024-09-16T04:11:06.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.695+0000 7f28b3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:11:06.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.695+0000 7f28b3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:06.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.695+0000 7f28b3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:06.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.695+0000 7f28c0f22640 1 --1- 172.21.15.42:0/2918593548 >> v1:172.21.15.42:6789/0 conn(0x7f28bc13c8c0 0x7f28bc13ecb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:33840/0 (socket says 172.21.15.42:33840) 2024-09-16T04:11:06.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.695+0000 7f28b3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:11:06.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.695+0000 7f28b3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:11:06.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.695+0000 7f28b3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:11:06.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.695+0000 7f28b3fff640 10 monclient(hunting): handle_auth_done global_id 1034391 payload 274 2024-09-16T04:11:06.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.695+0000 7f28b3fff640 1 -- 172.21.15.42:0/2918593548 >> v1:172.21.15.154:6789/0 conn(0x7f28bc0a39b0 legacy=0x7f28bc133b80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:11:06.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.695+0000 7f28b3fff640 1 -- 172.21.15.42:0/2918593548 >> v1:172.21.15.42:6789/0 conn(0x7f28bc13c8c0 legacy=0x7f28bc13ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:06.702 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.695+0000 7f28b3fff640 10 monclient: _finish_hunting 0 2024-09-16T04:11:06.702 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.695+0000 7f28b3fff640 1 monclient: found mon.noname-a 2024-09-16T04:11:06.702 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.695+0000 7f28b3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:11:06.702 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.695+0000 7f28b3fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:11:06.702 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.695+0000 7f28b3fff640 1 -- 172.21.15.42:0/2918593548 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28bc06dd60 con 0x7f28bc0a3d90 2024-09-16T04:11:06.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.695+0000 7f28b3fff640 1 --2- 172.21.15.42:0/2918593548 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f28bc0a3d90 0x7f28bc13c380 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f28a4009870 tx=0x7f28a401c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31f49b069c0a1ba server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:11:06.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.695+0000 7f28bb7fe640 1 -- 172.21.15.42:0/2918593548 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28a40b0070 con 0x7f28bc0a3d90 2024-09-16T04:11:06.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.695+0000 7f28bb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:06.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.695+0000 7f28bb7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:11:06.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.695+0000 7f28bb7fe640 10 monclient: dump: 2024-09-16T04:11:06.708 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:11:06.708 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:11:06.709 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:11:06.709 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:11:06.709 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:06.709 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:11:06.709 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:11:06.709 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:11:06.709 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:11:06.709 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:11:06.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.695+0000 7f28bb7fe640 10 monclient: _finish_auth 0 2024-09-16T04:11:06.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.695+0000 7f28bb7fe640 10 monclient: _check_auth_tickets 2024-09-16T04:11:06.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.695+0000 7f28bb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:11:06.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.695+0000 7f28bb7fe640 1 -- 172.21.15.42:0/2918593548 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f28a40ab600 con 0x7f28bc0a3d90 2024-09-16T04:11:06.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.695+0000 7f28bb7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:11:06.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.695+0000 7f28bb7fe640 1 -- 172.21.15.42:0/2918593548 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28a40ab9d0 con 0x7f28bc0a3d90 2024-09-16T04:11:06.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.695+0000 7f28c2985640 5 monclient: authenticate success, global_id 1034391 2024-09-16T04:11:06.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.695+0000 7f28c2985640 10 monclient: get_monmap_and_config success 2024-09-16T04:11:06.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.695+0000 7f28c2985640 4 set_mon_vals no callback set 2024-09-16T04:11:06.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.695+0000 7f28bb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:06.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.695+0000 7f28bb7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:11:06.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.695+0000 7f28bb7fe640 10 monclient: dump: 2024-09-16T04:11:06.710 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:11:06.710 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:11:06.710 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:11:06.710 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:11:06.710 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:06.710 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:11:06.710 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:11:06.711 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:11:06.711 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:11:06.711 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:11:06.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.695+0000 7f28c2985640 10 monclient: shutdown 2024-09-16T04:11:06.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.695+0000 7f28c2985640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:11:06.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.695+0000 7f28c2985640 1 -- 172.21.15.42:0/2918593548 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f28bc0a3d90 msgr2=0x7f28bc13c380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:06.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.695+0000 7f28c2985640 1 --2- 172.21.15.42:0/2918593548 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f28bc0a3d90 0x7f28bc13c380 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f28a4009870 tx=0x7f28a401c3e0 comp rx=0 tx=0).stop 2024-09-16T04:11:06.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.695+0000 7f28c2985640 1 -- 172.21.15.42:0/2918593548 shutdown_connections 2024-09-16T04:11:06.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.695+0000 7f28c2985640 1 --2- 172.21.15.42:0/2918593548 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f28bc0a3d90 0x7f28bc13c380 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:11:06.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.695+0000 7f28c2985640 1 -- 172.21.15.42:0/2918593548 >> 172.21.15.42:0/2918593548 conn(0x7f28bc0a9fb0 msgr2=0x7f28bc0aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:11:06.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.695+0000 7f28c2985640 1 -- 172.21.15.42:0/2918593548 shutdown_connections 2024-09-16T04:11:06.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.695+0000 7f28c2985640 1 -- 172.21.15.42:0/2918593548 wait complete. 2024-09-16T04:11:06.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.695+0000 7f28c2985640 10 monclient: build_initial_monmap 2024-09-16T04:11:06.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.695+0000 7f28c2985640 10 monclient: monmap: 2024-09-16T04:11:06.712 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:11:06.712 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:11:06.712 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:11:06.712 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:11:06.712 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:11:06.712 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:11:06.712 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:11:06.712 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:11:06.712 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:11:06.712 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:11:06.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.699+0000 7f28c2985640 1 librados: starting msgr at 2024-09-16T04:11:06.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.699+0000 7f28c2985640 1 librados: starting objecter 2024-09-16T04:11:06.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.699+0000 7f28c2985640 1 Processor -- start 2024-09-16T04:11:06.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.699+0000 7f28c2985640 1 -- start start 2024-09-16T04:11:06.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.699+0000 7f28c2985640 1 librados: setting wanted keys 2024-09-16T04:11:06.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.699+0000 7f28c2985640 1 librados: calling monclient init 2024-09-16T04:11:06.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.699+0000 7f28c2985640 10 monclient: init 2024-09-16T04:11:06.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.699+0000 7f28c2985640 10 monclient: _reopen_session rank -1 2024-09-16T04:11:06.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.699+0000 7f28c2985640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:11:06.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.699+0000 7f28c2985640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f28bc0a39b0 0x7f28bc12b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:11:06.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.699+0000 7f28c2985640 10 monclient(hunting): picked mon.noname-c con 0x7f28bc0a39b0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:11:06.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.699+0000 7f28c2985640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f28bc0a3d90 0x7f28bc12ba50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:11:06.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.699+0000 7f28c2985640 10 monclient(hunting): picked mon.noname-b con 0x7f28bc0a3d90 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:11:06.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.699+0000 7f28c2985640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f28bc13c8c0 0x7f28bc138130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:11:06.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.699+0000 7f28c2985640 10 monclient(hunting): picked mon.noname-a con 0x7f28bc13c8c0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:11:06.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.699+0000 7f28c2985640 10 monclient(hunting): start opening mon connection 2024-09-16T04:11:06.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.699+0000 7f28c2985640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f28bc142f80 con 0x7f28bc13c8c0 2024-09-16T04:11:06.714 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.699+0000 7f28c2985640 10 monclient(hunting): start opening mon connection 2024-09-16T04:11:06.714 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.699+0000 7f28c2985640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f28bc142e00 con 0x7f28bc0a3d90 2024-09-16T04:11:06.714 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.699+0000 7f28c2985640 10 monclient(hunting): start opening mon connection 2024-09-16T04:11:06.714 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.699+0000 7f28c2985640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f28bc143100 con 0x7f28bc0a39b0 2024-09-16T04:11:06.714 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.699+0000 7f28c2985640 10 monclient(hunting): _renew_subs 2024-09-16T04:11:06.714 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.699+0000 7f28c2985640 10 monclient(hunting): authenticate will time out at 2713.668750s 2024-09-16T04:11:06.714 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.699+0000 7f28bbfff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f28bc0a39b0 0x7f28bc12b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:11:06.714 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.699+0000 7f28c0f22640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f28bc13c8c0 0x7f28bc138130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:11:06.714 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.699+0000 7f28bbfff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f28bc0a39b0 0x7f28bc12b510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:50090/0 (socket says 172.21.15.42:50090) 2024-09-16T04:11:06.714 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.699+0000 7f28c0f22640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f28bc13c8c0 0x7f28bc138130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:41078/0 (socket says 172.21.15.42:41078) 2024-09-16T04:11:06.714 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.699+0000 7f28bbfff640 1 -- 172.21.15.42:0/1892419784 learned_addr learned my addr 172.21.15.42:0/1892419784 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:11:06.714 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.699+0000 7f28bbfff640 10 monclient(hunting): get_auth_request con 0x7f28bc0a39b0 auth_method 0 2024-09-16T04:11:06.714 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.699+0000 7f28bbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:11:06.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.699+0000 7f28bbfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:06.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.699+0000 7f28bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:06.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.699+0000 7f28c0f22640 10 monclient(hunting): get_auth_request con 0x7f28bc13c8c0 auth_method 0 2024-09-16T04:11:06.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.699+0000 7f28c0f22640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:11:06.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.699+0000 7f28c0f22640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:06.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.699+0000 7f28c0f22640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:06.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.699+0000 7f28b3fff640 1 -- 172.21.15.42:0/1892419784 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f28bc0a3d90 msgr2=0x7f28bc12ba50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.154:3300/0 2024-09-16T04:11:06.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.699+0000 7f28b3fff640 1 --2- 172.21.15.42:0/1892419784 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f28bc0a3d90 0x7f28bc12ba50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:11:06.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.699+0000 7f28bbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:11:06.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.699+0000 7f28bbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:11:06.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.699+0000 7f28bbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:11:06.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.699+0000 7f28c0f22640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:11:06.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.699+0000 7f28c0f22640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:11:06.716 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.699+0000 7f28c0f22640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:11:06.716 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.699+0000 7f28c0f22640 10 monclient(hunting): handle_auth_done global_id 1034397 payload 824 2024-09-16T04:11:06.716 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.699+0000 7f28c0f22640 1 -- 172.21.15.42:0/1892419784 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f28bc0a39b0 msgr2=0x7f28bc12b510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:06.716 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.699+0000 7f28c0f22640 1 --2- 172.21.15.42:0/1892419784 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f28bc0a39b0 0x7f28bc12b510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:11:06.716 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.699+0000 7f28c0f22640 1 -- 172.21.15.42:0/1892419784 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f28bc0a3d90 msgr2=0x7f28bc12ba50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:11:06.716 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.699+0000 7f28c0f22640 1 --2- 172.21.15.42:0/1892419784 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f28bc0a3d90 0x7f28bc12ba50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:11:06.716 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.699+0000 7f28c0f22640 10 monclient: _finish_hunting 0 2024-09-16T04:11:06.716 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.699+0000 7f28c0f22640 1 monclient: found mon.noname-a 2024-09-16T04:11:06.716 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.699+0000 7f28c0f22640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:11:06.716 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.699+0000 7f28c0f22640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:11:06.716 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.699+0000 7f28c0f22640 1 -- 172.21.15.42:0/1892419784 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28bc13b980 con 0x7f28bc13c8c0 2024-09-16T04:11:06.716 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.699+0000 7f28c0f22640 1 --2- 172.21.15.42:0/1892419784 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f28bc13c8c0 0x7f28bc138130 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f28b400c730 tx=0x7f28b400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:11:06.716 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.699+0000 7f28b97fa640 10 client.?.objecter ms_handle_connect 0x7f28bc13c8c0 2024-09-16T04:11:06.716 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.699+0000 7f28b97fa640 10 client.?.objecter resend_mon_ops 2024-09-16T04:11:06.717 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.699+0000 7f28b97fa640 1 -- 172.21.15.42:0/1892419784 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28b4017070 con 0x7f28bc13c8c0 2024-09-16T04:11:06.717 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.699+0000 7f28b97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:06.717 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.699+0000 7f28b97fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:11:06.717 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.699+0000 7f28b97fa640 10 monclient: dump: 2024-09-16T04:11:06.717 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:11:06.717 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:11:06.717 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:11:06.717 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:11:06.717 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:06.717 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:11:06.717 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:11:06.717 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:11:06.717 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:11:06.718 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:11:06.718 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.699+0000 7f28b97fa640 10 monclient: _finish_auth 0 2024-09-16T04:11:06.718 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.699+0000 7f28b97fa640 10 monclient: _check_auth_tickets 2024-09-16T04:11:06.718 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.699+0000 7f28b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:11:06.718 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.699+0000 7f28b97fa640 1 -- 172.21.15.42:0/1892419784 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f28b400ce40 con 0x7f28bc13c8c0 2024-09-16T04:11:06.718 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.699+0000 7f28c2985640 5 monclient: authenticate success, global_id 1034397 2024-09-16T04:11:06.718 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.699+0000 7f28c2985640 10 monclient: _renew_subs 2024-09-16T04:11:06.718 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.699+0000 7f28c2985640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:11:06.718 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.699+0000 7f28c2985640 1 -- 172.21.15.42:0/1892419784 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28bc13fdf0 con 0x7f28bc13c8c0 2024-09-16T04:11:06.718 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.699+0000 7f28b97fa640 10 monclient: handle_config config(0 keys) 2024-09-16T04:11:06.718 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.699+0000 7f28b97fa640 1 -- 172.21.15.42:0/1892419784 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28b400a840 con 0x7f28bc13c8c0 2024-09-16T04:11:06.718 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.699+0000 7f28b97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:06.718 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.699+0000 7f28b97fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:11:06.718 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.699+0000 7f28b97fa640 10 monclient: dump: 2024-09-16T04:11:06.719 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:11:06.719 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:11:06.719 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:11:06.719 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:11:06.719 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:06.719 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:11:06.719 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:11:06.719 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:11:06.719 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:11:06.719 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:11:06.719 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.699+0000 7f28baffd640 4 set_mon_vals no callback set 2024-09-16T04:11:06.719 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.699+0000 7f28c2985640 10 client.1034397.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:11:06.719 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.699+0000 7f28c2985640 10 monclient: _renew_subs 2024-09-16T04:11:06.720 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.699+0000 7f28c2985640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:11:06.720 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.699+0000 7f28c2985640 1 -- 172.21.15.42:0/1892419784 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28bc13b6a0 con 0x7f28bc13c8c0 2024-09-16T04:11:06.720 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.699+0000 7f28c2985640 1 librados: init done 2024-09-16T04:11:06.720 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.699+0000 7f28c2985640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:06.720 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.699+0000 7f28c2985640 1 --2- 172.21.15.42:0/1892419784 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f28bc138670 0x7f28bc138a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:11:06.720 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.699+0000 7f28c2985640 10 monclient(hunting): start opening mon connection 2024-09-16T04:11:06.720 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.699+0000 7f28c2985640 1 -- 172.21.15.42:0/1892419784 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f28bc13a590 con 0x7f28bc138670 2024-09-16T04:11:06.720 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.699+0000 7f28bbfff640 1 --2- 172.21.15.42:0/1892419784 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f28bc138670 0x7f28bc138a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:11:06.720 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.699+0000 7f28bbfff640 10 monclient: get_auth_request con 0x7f28bc138670 auth_method 0 2024-09-16T04:11:06.720 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.699+0000 7f28bbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:11:06.720 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.699+0000 7f28bbfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:06.720 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.699+0000 7f28bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:06.720 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.699+0000 7f28bbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:11:06.721 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.699+0000 7f28bbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:11:06.721 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.699+0000 7f28bbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:11:06.721 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.699+0000 7f28b97fa640 1 -- 172.21.15.42:0/1892419784 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f28b401a070 con 0x7f28bc13c8c0 2024-09-16T04:11:06.721 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.699+0000 7f28bbfff640 10 monclient(hunting): handle_auth_done global_id 1014500 payload 824 2024-09-16T04:11:06.721 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.699+0000 7f28bbfff640 1 -- 172.21.15.42:0/1892419784 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f28bc133b80 con 0x7f28bc138670 2024-09-16T04:11:06.721 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.699+0000 7f28b97fa640 1 --2- 172.21.15.42:0/1892419784 >> v2:172.21.15.154:6812/254845342 conn(0x7f289c0377c0 0x7f289c039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:11:06.721 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.699+0000 7f28b97fa640 1 -- 172.21.15.42:0/1892419784 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(706..706 src has 451..706) ==== 5040+0+0 (secure 0 0 0) 0x7f28b4048230 con 0x7f28bc13c8c0 2024-09-16T04:11:06.721 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.699+0000 7f28b97fa640 10 client.1034397.objecter ms_dispatch 0x7f28bc000ba0 osd_map(706..706 src has 451..706) 2024-09-16T04:11:06.721 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.699+0000 7f28b97fa640 3 client.1034397.objecter handle_osd_map got epochs [706,706] > 0 2024-09-16T04:11:06.721 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.699+0000 7f28b97fa640 3 client.1034397.objecter handle_osd_map decoding full epoch 706 2024-09-16T04:11:06.721 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.699+0000 7f28bbfff640 1 --2- 172.21.15.42:0/1892419784 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f28bc138670 0x7f28bc138a50 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f28ac009710 tx=0x7f28ac009b10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:11:06.721 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.699+0000 7f28b97fa640 20 client.1034397.objecter dump_active .. 0 homeless 2024-09-16T04:11:06.721 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.699+0000 7f28b97fa640 10 client.1034397.objecter ms_handle_connect 0x7f28bc138670 2024-09-16T04:11:06.722 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.699+0000 7f28b97fa640 10 client.1034397.objecter resend_mon_ops 2024-09-16T04:11:06.722 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.699+0000 7f28b97fa640 1 -- 172.21.15.42:0/1892419784 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28ac016070 con 0x7f28bc138670 2024-09-16T04:11:06.722 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.699+0000 7f28b97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:06.722 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.699+0000 7f28b97fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:11:06.722 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.699+0000 7f28b97fa640 10 monclient: dump: 2024-09-16T04:11:06.722 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:11:06.722 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:11:06.722 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:11:06.722 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:11:06.722 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:06.722 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:11:06.722 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:11:06.722 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:11:06.722 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:11:06.723 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:11:06.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.699+0000 7f28b3fff640 1 --2- 172.21.15.42:0/1892419784 >> v2:172.21.15.154:6812/254845342 conn(0x7f289c0377c0 0x7f289c039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:11:06.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.699+0000 7f28b3fff640 10 monclient: get_auth_request con 0x7f289c0377c0 auth_method 0 2024-09-16T04:11:06.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.699+0000 7f28b97fa640 1 -- 172.21.15.42:0/1892419784 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f28ac00e7a0 con 0x7f28bc138670 2024-09-16T04:11:06.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.699+0000 7f28b97fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:06.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.699+0000 7f28b97fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:11:06.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.699+0000 7f28b97fa640 1 -- 172.21.15.42:0/1892419784 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f28bc138670 msgr2=0x7f28bc138a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:06.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.699+0000 7f28b97fa640 1 --2- 172.21.15.42:0/1892419784 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f28bc138670 0x7f28bc138a50 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f28ac009710 tx=0x7f28ac009b10 comp rx=0 tx=0).stop 2024-09-16T04:11:06.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.699+0000 7f28b97fa640 1 -- 172.21.15.42:0/1892419784 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f28bc138670 msgr2=0x7f28bc138a50 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:11:06.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.699+0000 7f28b97fa640 1 --2- 172.21.15.42:0/1892419784 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f28bc138670 0x7f28bc138a50 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:11:06.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.699+0000 7f28b97fa640 10 client.1034397.objecter ms_handle_connect 0x7f289c0377c0 2024-09-16T04:11:06.723 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.699+0000 7f28b3fff640 1 --2- 172.21.15.42:0/1892419784 >> v2:172.21.15.154:6812/254845342 conn(0x7f289c0377c0 0x7f289c039c80 secure :-1 s=READY pgs=3944 cs=0 l=1 rev1=1 crypto rx=0x7f28a40ad000 tx=0x7f28a4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:11:06.725 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.707+0000 7f28c2985640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:11:06.725 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.707+0000 7f28c2985640 1 --2- 172.21.15.42:0/1892419784 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f28bc138f90 0x7f28bc134bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:11:06.725 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.707+0000 7f28c2985640 10 monclient(hunting): start opening mon connection 2024-09-16T04:11:06.725 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.707+0000 7f28c2985640 1 -- 172.21.15.42:0/1892419784 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f28bc0a5750 con 0x7f28bc138f90 2024-09-16T04:11:06.725 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.707+0000 7f28bbfff640 1 --2- 172.21.15.42:0/1892419784 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f28bc138f90 0x7f28bc134bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:11:06.725 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.707+0000 7f28bbfff640 10 monclient: get_auth_request con 0x7f28bc138f90 auth_method 0 2024-09-16T04:11:06.725 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.707+0000 7f28bbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:11:06.725 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.707+0000 7f28bbfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:06.726 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.707+0000 7f28bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:06.726 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.707+0000 7f28bbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:11:06.726 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.707+0000 7f28bbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:11:06.726 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.707+0000 7f28bbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:11:06.726 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.707+0000 7f28bbfff640 10 monclient(hunting): handle_auth_done global_id 1014503 payload 824 2024-09-16T04:11:06.726 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.707+0000 7f28bbfff640 1 -- 172.21.15.42:0/1892419784 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f28bc13b090 con 0x7f28bc138f90 2024-09-16T04:11:06.726 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.707+0000 7f28bbfff640 1 --2- 172.21.15.42:0/1892419784 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f28bc138f90 0x7f28bc134bb0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f28ac018000 tx=0x7f28ac0059e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:11:06.726 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.707+0000 7f28b97fa640 10 client.1034397.objecter ms_handle_connect 0x7f28bc138f90 2024-09-16T04:11:06.726 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.707+0000 7f28b97fa640 10 client.1034397.objecter resend_mon_ops 2024-09-16T04:11:06.726 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.707+0000 7f28b97fa640 1 -- 172.21.15.42:0/1892419784 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28ac016040 con 0x7f28bc138f90 2024-09-16T04:11:06.726 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.707+0000 7f28b97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:06.726 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.707+0000 7f28b97fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:11:06.726 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.707+0000 7f28b97fa640 10 monclient: dump: 2024-09-16T04:11:06.727 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:11:06.727 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:11:06.727 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:11:06.727 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:11:06.727 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:06.727 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:11:06.727 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:11:06.727 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:11:06.727 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:11:06.727 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:11:06.727 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.707+0000 7f28b97fa640 1 -- 172.21.15.42:0/1892419784 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4702 (secure 0 0 0) 0x7f28ac01d0d0 con 0x7f28bc138f90 2024-09-16T04:11:06.727 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.707+0000 7f28b97fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:11:06.727 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.707+0000 7f28b97fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:11:06.727 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.707+0000 7f28b97fa640 1 -- 172.21.15.42:0/1892419784 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f28bc138f90 msgr2=0x7f28bc134bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:06.728 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.707+0000 7f28b97fa640 1 --2- 172.21.15.42:0/1892419784 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f28bc138f90 0x7f28bc134bb0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f28ac018000 tx=0x7f28ac0059e0 comp rx=0 tx=0).stop 2024-09-16T04:11:06.728 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.707+0000 7f28b97fa640 1 -- 172.21.15.42:0/1892419784 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f28bc138f90 msgr2=0x7f28bc134bb0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:11:06.728 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.707+0000 7f28b97fa640 1 --2- 172.21.15.42:0/1892419784 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f28bc138f90 0x7f28bc134bb0 secure :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f28ac018000 tx=0x7f28ac0059e0 comp rx=0 tx=0).stop 2024-09-16T04:11:06.728 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T04:11:06.728 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:11:06.728 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:11:06.728 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T04:11:06.728 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 264, 2024-09-16T04:11:06.728 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 261909, 2024-09-16T04:11:06.729 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T04:11:06.729 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T04:11:06.729 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T04:11:06.729 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:11:06.729 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 0, 2024-09-16T04:11:06.729 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:11:06.729 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T04:11:06.729 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T04:11:06.729 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:11:06.729 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:11:06.729 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:11:06.729 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:11:06.729 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:11:06.729 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:11:06.730 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:11:06.730 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:11:06.730 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:11:06.730 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:11:06.730 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:11:06.730 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:11:06.730 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:11:06.730 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T04:11:06.730 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:11:06.730 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:11:06.730 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:11:06.730 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:11:06.730 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:11:06.731 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:11:06.731 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:11:06.731 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:11:06.731 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:11:06.731 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:11:06.731 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:11:06.731 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:11:06.731 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:11:06.731 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T04:11:06.731 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T04:11:06.731 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T04:11:06.731 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T04:11:06.731 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T04:11:06.732 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T04:11:06.732 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:11:06.732 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:11:06.732 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T04:11:06.732 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T04:11:06.732 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T04:11:06.732 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T04:11:06.732 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T04:11:06.732 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T04:11:06.732 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T04:11:06.732 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:11:06.732 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T04:11:06.732 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:11:06.732 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:11:06.733 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:11:06.733 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:11:06.733 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:11:06.733 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:11:06.733 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:11:06.733 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:11:06.733 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:11:06.733 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:11:06.733 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:11:06.733 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:11:06.733 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T04:11:06.733 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:11:06.733 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T04:11:06.733 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:11:06.734 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:11:06.734 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:11:06.734 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:11:06.734 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:11:06.734 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:11:06.734 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:11:06.734 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T04:11:06.734 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:11:06.734 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:11:06.734 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:11:06.734 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:11:06.734 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T04:11:06.734 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:11:06.735 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:11:06.735 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:11:06.735 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:11:06.735 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T04:11:06.735 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T04:11:06.735 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:11:06.735 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:11:06.735 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:11:06.735 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:11:06.735 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:11:06.735 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:11:06.735 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:11:06.735 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:11:06.735 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:11:06.736 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:11:06.736 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:11:06.736 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T04:11:06.736 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:11:06.736 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:11:06.736 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:11:06.736 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:11:06.736 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T04:11:06.736 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:11:06.736 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:11:06.736 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:11:06.736 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:11:06.736 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T04:11:06.737 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T04:11:06.737 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:11:06.737 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:11:06.737 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:11:06.737 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:11:06.737 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:11:06.737 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:11:06.737 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:11:06.737 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:11:06.737 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:11:06.737 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:11:06.737 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:11:06.737 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T04:11:06.737 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:11:06.738 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:11:06.738 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:11:06.738 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:11:06.738 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T04:11:06.738 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:11:06.738 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:11:06.738 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:11:06.738 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:11:06.738 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T04:11:06.738 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T04:11:06.738 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:11:06.738 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:11:06.738 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:11:06.739 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:11:06.739 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:11:06.739 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:11:06.739 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T04:11:06.739 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T04:11:06.739 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:11:06.739 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:11:06.739 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:11:06.739 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:11:06.739 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:11:06.739 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:11:06.739 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T04:11:06.739 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:11:06.740 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:11:06.740 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:11:06.740 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 4 2024-09-16T04:11:06.740 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:11:06.740 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:11:06.740 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T04:11:06.740 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:11:06.740 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:11:06.740 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:11:06.740 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 3 2024-09-16T04:11:06.740 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:11:06.740 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:11:06.740 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:11:06.740 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T04:11:06.741 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T04:11:06.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.707+0000 7f28c2985640 10 librados: watch_flush enter 2024-09-16T04:11:06.744 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.707+0000 7f28c2985640 10 librados: watch_flush exit 2024-09-16T04:11:06.744 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.707+0000 7f28c2985640 20 client.1034397.objecter shutdown clearing up homeless session... 2024-09-16T04:11:06.744 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.707+0000 7f28c2985640 10 client.1034397.objecter successfully canceled tick 2024-09-16T04:11:06.744 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.707+0000 7f28c2985640 1 -- 172.21.15.42:0/1892419784 >> v2:172.21.15.154:6812/254845342 conn(0x7f289c0377c0 msgr2=0x7f289c039c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:06.744 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.707+0000 7f28c2985640 1 --2- 172.21.15.42:0/1892419784 >> v2:172.21.15.154:6812/254845342 conn(0x7f289c0377c0 0x7f289c039c80 secure :-1 s=READY pgs=3944 cs=0 l=1 rev1=1 crypto rx=0x7f28a40ad000 tx=0x7f28a4008040 comp rx=0 tx=0).stop 2024-09-16T04:11:06.744 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.707+0000 7f28c2985640 10 monclient: shutdown 2024-09-16T04:11:06.744 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.707+0000 7f28c2985640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:11:06.744 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.707+0000 7f28c2985640 1 -- 172.21.15.42:0/1892419784 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f28bc13c8c0 msgr2=0x7f28bc138130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:06.744 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.707+0000 7f28c2985640 1 --2- 172.21.15.42:0/1892419784 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f28bc13c8c0 0x7f28bc138130 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f28b400c730 tx=0x7f28b400cbf0 comp rx=0 tx=0).stop 2024-09-16T04:11:06.744 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.707+0000 7f28c0f22640 1 -- 172.21.15.42:0/1892419784 reap_dead start 2024-09-16T04:11:06.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.707+0000 7f28c0f22640 1 -- 172.21.15.42:0/1892419784 reap_dead start 2024-09-16T04:11:06.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.707+0000 7f28c2985640 1 -- 172.21.15.42:0/1892419784 shutdown_connections 2024-09-16T04:11:06.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.707+0000 7f28c2985640 1 -- 172.21.15.42:0/1892419784 >> 172.21.15.42:0/1892419784 conn(0x7f28bc0a9fb0 msgr2=0x7f28bc12c330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:11:06.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.707+0000 7f28c2985640 1 -- 172.21.15.42:0/1892419784 shutdown_connections 2024-09-16T04:11:06.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.707+0000 7f28c2985640 1 -- 172.21.15.42:0/1892419784 wait complete. 2024-09-16T04:11:06.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:06.707+0000 7f28c2985640 1 librados: shutdown 2024-09-16T04:11:06.745 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T04:11:06.746 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-16T04:11:26.746 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-16T04:11:26.747 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-09-16T04:11:26.747 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-16T04:11:26.747 DEBUG:teuthology.orchestra.run.smithi154:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-16T04:11:26.751 INFO:tasks.ceph.mon.b:Started 2024-09-16T04:11:26.751 INFO:tasks.mon_thrash:Sending STOP to mon c 2024-09-16T04:11:26.751 INFO:tasks.ceph.mon.c:Sent signal 19 2024-09-16T04:11:26.751 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-16T04:11:41.752 INFO:tasks.mon_thrash:Sending CONT to mon c 2024-09-16T04:11:41.753 INFO:tasks.ceph.mon.c:Sent signal 18 2024-09-16T04:11:41.753 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T04:11:41.753 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:11:41.883 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.879+0000 7fb0b5017640 10 monclient: get_monmap_and_config 2024-09-16T04:11:41.883 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.879+0000 7fb0b5017640 10 monclient: build_initial_monmap 2024-09-16T04:11:41.883 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.879+0000 7fb0b5017640 10 monclient: monmap: 2024-09-16T04:11:41.883 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:11:41.883 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:11:41.883 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:11:41.881878+0000 2024-09-16T04:11:41.884 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:11:41.881878+0000 2024-09-16T04:11:41.884 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:11:41.884 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:11:41.884 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:11:41.884 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:11:41.884 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:11:41.884 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:11:41.884 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:11:41.884 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:11:41.888 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.879+0000 7fb0b5017640 1 Processor -- start 2024-09-16T04:11:41.888 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.879+0000 7fb0b5017640 1 -- start start 2024-09-16T04:11:41.888 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.879+0000 7fb0b5017640 10 monclient: init 2024-09-16T04:11:41.888 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.879+0000 7fb0b5017640 10 monclient: _reopen_session rank -1 2024-09-16T04:11:41.888 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.879+0000 7fb0b5017640 10 monclient: _add_conns ranks=[4,1,3,0,2] 2024-09-16T04:11:41.888 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.879+0000 7fb0b5017640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fb0b006db90 0x7fb0b01340c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:11:41.889 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.879+0000 7fb0b5017640 10 monclient(hunting): picked mon.noname-c con 0x7fb0b006db90 addr v2:172.21.15.154:3300/0 2024-09-16T04:11:41.889 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.879+0000 7fb0b5017640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb0b00a37f0 addr v1:172.21.15.154:6789/0 2024-09-16T04:11:41.892 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.879+0000 7fb0b5017640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb0b00a3c00 0x7fb0b0137520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:11:41.892 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.879+0000 7fb0b5017640 10 monclient(hunting): picked mon.noname-a con 0x7fb0b00a3c00 addr v2:172.21.15.42:3300/0 2024-09-16T04:11:41.892 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.879+0000 7fb0b5017640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb0b006ece0 con 0x7fb0b00a37f0 2024-09-16T04:11:41.892 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.879+0000 7fb0b5017640 10 monclient(hunting): start opening mon connection 2024-09-16T04:11:41.892 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.879+0000 7fb0b5017640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7fb0b006df70 con 0x7fb0b00a3c00 2024-09-16T04:11:41.892 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.879+0000 7fb0b5017640 10 monclient(hunting): start opening mon connection 2024-09-16T04:11:41.892 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.879+0000 7fb0b5017640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fb0b009bf80 con 0x7fb0b006db90 2024-09-16T04:11:41.892 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.879+0000 7fb0b5017640 10 monclient(hunting): _renew_subs 2024-09-16T04:11:41.892 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.879+0000 7fb0b5017640 10 monclient(hunting): authenticate will time out at 2748.851303s 2024-09-16T04:11:41.892 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.879+0000 7fb0af59e640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb0b00a3c00 0x7fb0b0137520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:11:41.893 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.879+0000 7fb0af59e640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb0b00a3c00 0x7fb0b0137520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:57582/0 (socket says 172.21.15.42:57582) 2024-09-16T04:11:41.893 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.879+0000 7fb0af59e640 1 -- 172.21.15.42:0/884994301 learned_addr learned my addr 172.21.15.42:0/884994301 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:11:41.893 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.879+0000 7fb0af59e640 10 monclient(hunting): get_auth_request con 0x7fb0b00a3c00 auth_method 0 2024-09-16T04:11:41.893 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.879+0000 7fb0af59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:11:41.893 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.879+0000 7fb0af59e640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:41.893 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.879+0000 7fb0af59e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:41.893 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.879+0000 7fb0af59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:11:41.893 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.879+0000 7fb0af59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:11:41.893 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.879+0000 7fb0af59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:11:41.893 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.879+0000 7fb0aed9d640 1 --2- 172.21.15.42:0/884994301 >> v2:172.21.15.154:3300/0 conn(0x7fb0b006db90 0x7fb0b01340c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:11:41.893 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.879+0000 7fb0aed9d640 10 monclient(hunting): get_auth_request con 0x7fb0b006db90 auth_method 0 2024-09-16T04:11:41.893 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.879+0000 7fb0aed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:11:41.893 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.879+0000 7fb0aed9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:41.893 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.879+0000 7fb0aed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:41.894 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.879+0000 7fb0af59e640 10 monclient(hunting): handle_auth_done global_id 1034400 payload 274 2024-09-16T04:11:41.894 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.879+0000 7fb0af59e640 1 -- 172.21.15.42:0/884994301 >> v2:172.21.15.154:3300/0 conn(0x7fb0b006db90 msgr2=0x7fb0b01340c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:11:41.894 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.879+0000 7fb0af59e640 1 --2- 172.21.15.42:0/884994301 >> v2:172.21.15.154:3300/0 conn(0x7fb0b006db90 0x7fb0b01340c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:11:41.894 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.879+0000 7fb0af59e640 1 -- 172.21.15.42:0/884994301 >> v1:172.21.15.154:6789/0 conn(0x7fb0b00a37f0 legacy=0x7fb0b0136e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:11:41.894 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.879+0000 7fb0af59e640 10 monclient: _finish_hunting 0 2024-09-16T04:11:41.894 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.879+0000 7fb0af59e640 1 monclient: found mon.noname-a 2024-09-16T04:11:41.894 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.879+0000 7fb0af59e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:11:41.894 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.879+0000 7fb0af59e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:11:41.894 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.879+0000 7fb0af59e640 1 -- 172.21.15.42:0/884994301 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0b0134780 con 0x7fb0b00a3c00 2024-09-16T04:11:41.894 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.883+0000 7fb0af59e640 1 --2- 172.21.15.42:0/884994301 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb0b00a3c00 0x7fb0b0137520 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fb0b00a76f0 tx=0x7fb0a801d2a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1884b83a00a34991 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:11:41.894 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.883+0000 7fb0add9b640 1 -- 172.21.15.42:0/884994301 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0a80a5720 con 0x7fb0b00a3c00 2024-09-16T04:11:41.894 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.883+0000 7fb0add9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:41.894 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.883+0000 7fb0add9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:11:41.894 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.883+0000 7fb0add9b640 10 monclient: dump: 2024-09-16T04:11:41.895 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:11:41.895 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:11:41.895 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:11:41.895 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:11:41.895 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:41.895 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:11:41.895 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:11:41.895 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:11:41.895 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:11:41.895 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:11:41.895 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.883+0000 7fb0add9b640 10 monclient: _finish_auth 0 2024-09-16T04:11:41.895 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.883+0000 7fb0add9b640 10 monclient: _check_auth_tickets 2024-09-16T04:11:41.895 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.883+0000 7fb0add9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:11:41.896 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.883+0000 7fb0b5017640 5 monclient: authenticate success, global_id 1034400 2024-09-16T04:11:41.896 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.883+0000 7fb0add9b640 1 -- 172.21.15.42:0/884994301 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb0a80a58c0 con 0x7fb0b00a3c00 2024-09-16T04:11:41.896 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.883+0000 7fb0add9b640 10 monclient: handle_config config(0 keys) 2024-09-16T04:11:41.896 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.883+0000 7fb0b5017640 10 monclient: get_monmap_and_config success 2024-09-16T04:11:41.896 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.883+0000 7fb0add9b640 1 -- 172.21.15.42:0/884994301 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0a80a5c90 con 0x7fb0b00a3c00 2024-09-16T04:11:41.896 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.883+0000 7fb0b5017640 4 set_mon_vals no callback set 2024-09-16T04:11:41.896 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.883+0000 7fb0add9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:41.896 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.883+0000 7fb0add9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:11:41.896 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.883+0000 7fb0add9b640 10 monclient: dump: 2024-09-16T04:11:41.896 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:11:41.896 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:11:41.896 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:11:41.896 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:11:41.896 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:41.897 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:11:41.897 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:11:41.897 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:11:41.897 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:11:41.897 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:11:41.897 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.883+0000 7fb0b5017640 10 monclient: shutdown 2024-09-16T04:11:41.897 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.883+0000 7fb0b5017640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:11:41.897 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.883+0000 7fb0b5017640 1 -- 172.21.15.42:0/884994301 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb0b00a3c00 msgr2=0x7fb0b0137520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:41.897 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.883+0000 7fb0b5017640 1 --2- 172.21.15.42:0/884994301 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb0b00a3c00 0x7fb0b0137520 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fb0b00a76f0 tx=0x7fb0a801d2a0 comp rx=0 tx=0).stop 2024-09-16T04:11:41.897 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.883+0000 7fb0b5017640 1 -- 172.21.15.42:0/884994301 shutdown_connections 2024-09-16T04:11:41.897 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.883+0000 7fb0b5017640 1 --2- 172.21.15.42:0/884994301 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb0b00a3c00 0x7fb0b0137520 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:11:41.897 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.883+0000 7fb0b5017640 1 --2- 172.21.15.42:0/884994301 >> v2:172.21.15.154:3300/0 conn(0x7fb0b006db90 0x7fb0b01340c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:11:41.897 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.883+0000 7fb0b5017640 1 -- 172.21.15.42:0/884994301 >> 172.21.15.42:0/884994301 conn(0x7fb0b00a9fb0 msgr2=0x7fb0b00aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:11:41.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.883+0000 7fb0b5017640 1 -- 172.21.15.42:0/884994301 shutdown_connections 2024-09-16T04:11:41.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.883+0000 7fb0b5017640 1 -- 172.21.15.42:0/884994301 wait complete. 2024-09-16T04:11:41.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.883+0000 7fb0b5017640 10 monclient: build_initial_monmap 2024-09-16T04:11:41.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.883+0000 7fb0b5017640 10 monclient: monmap: 2024-09-16T04:11:41.898 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:11:41.898 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:11:41.898 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:11:41.898 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:11:41.898 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:11:41.898 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:11:41.898 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:11:41.898 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:11:41.898 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:11:41.898 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:11:41.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.883+0000 7fb0b5017640 1 librados: starting msgr at 2024-09-16T04:11:41.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.883+0000 7fb0b5017640 1 librados: starting objecter 2024-09-16T04:11:41.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.883+0000 7fb0b5017640 1 Processor -- start 2024-09-16T04:11:41.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.883+0000 7fb0b5017640 1 -- start start 2024-09-16T04:11:41.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.883+0000 7fb0b5017640 1 librados: setting wanted keys 2024-09-16T04:11:41.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.883+0000 7fb0b5017640 1 librados: calling monclient init 2024-09-16T04:11:41.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.883+0000 7fb0b5017640 10 monclient: init 2024-09-16T04:11:41.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.887+0000 7fb0b5017640 10 monclient: _reopen_session rank -1 2024-09-16T04:11:41.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.887+0000 7fb0b5017640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:11:41.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.887+0000 7fb0b5017640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb0b006db90 0x7fb0b01cecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:11:41.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.887+0000 7fb0b5017640 10 monclient(hunting): picked mon.noname-b con 0x7fb0b006db90 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:11:41.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.887+0000 7fb0b5017640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb0b00a37f0 0x7fb0b013e360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:11:41.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.887+0000 7fb0b5017640 10 monclient(hunting): picked mon.noname-c con 0x7fb0b00a37f0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:11:41.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.887+0000 7fb0b5017640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb0b00a3c00 0x7fb0b013e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:11:41.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.887+0000 7fb0b5017640 10 monclient(hunting): picked mon.noname-a con 0x7fb0b00a3c00 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:11:41.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.887+0000 7fb0b5017640 10 monclient(hunting): start opening mon connection 2024-09-16T04:11:41.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.887+0000 7fb0b5017640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fb0b00a40d0 con 0x7fb0b00a3c00 2024-09-16T04:11:41.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.887+0000 7fb0b5017640 10 monclient(hunting): start opening mon connection 2024-09-16T04:11:41.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.887+0000 7fb0b5017640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fb0b0137c70 con 0x7fb0b006db90 2024-09-16T04:11:41.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.887+0000 7fb0b5017640 10 monclient(hunting): start opening mon connection 2024-09-16T04:11:41.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.887+0000 7fb0b5017640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fb0b01419d0 con 0x7fb0b00a37f0 2024-09-16T04:11:41.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.887+0000 7fb0b5017640 10 monclient(hunting): _renew_subs 2024-09-16T04:11:41.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.887+0000 7fb0b5017640 10 monclient(hunting): authenticate will time out at 2748.856355s 2024-09-16T04:11:41.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.887+0000 7fb0ae59c640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb0b00a37f0 0x7fb0b013e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:11:41.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.887+0000 7fb0af59e640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb0b00a3c00 0x7fb0b013e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:11:41.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.887+0000 7fb0ae59c640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb0b00a37f0 0x7fb0b013e360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:37592/0 (socket says 172.21.15.42:37592) 2024-09-16T04:11:41.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.887+0000 7fb0ae59c640 1 -- 172.21.15.42:0/889576396 learned_addr learned my addr 172.21.15.42:0/889576396 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:11:41.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.887+0000 7fb0ae59c640 10 monclient(hunting): get_auth_request con 0x7fb0b00a37f0 auth_method 0 2024-09-16T04:11:41.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.887+0000 7fb0ae59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:11:41.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.887+0000 7fb0ae59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:41.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.887+0000 7fb0ae59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:41.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.887+0000 7fb0af59e640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb0b00a3c00 0x7fb0b013e8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:57586/0 (socket says 172.21.15.42:57586) 2024-09-16T04:11:41.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.887+0000 7fb0af59e640 10 monclient(hunting): get_auth_request con 0x7fb0b00a3c00 auth_method 0 2024-09-16T04:11:41.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.887+0000 7fb0af59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:11:41.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.887+0000 7fb0af59e640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:41.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.887+0000 7fb0af59e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:41.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.887+0000 7fb0af59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:11:41.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.887+0000 7fb0af59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:11:41.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.887+0000 7fb0af59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:11:41.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.887+0000 7fb0ae59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:11:41.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.887+0000 7fb0ae59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:11:41.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.887+0000 7fb0ae59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:11:41.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.887+0000 7fb0aed9d640 1 --2- 172.21.15.42:0/889576396 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb0b006db90 0x7fb0b01cecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:11:41.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.887+0000 7fb0aed9d640 10 monclient(hunting): get_auth_request con 0x7fb0b006db90 auth_method 0 2024-09-16T04:11:41.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.887+0000 7fb0aed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:11:41.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.887+0000 7fb0aed9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:41.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.887+0000 7fb0aed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:41.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.887+0000 7fb0af59e640 10 monclient(hunting): handle_auth_done global_id 1034403 payload 824 2024-09-16T04:11:41.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.887+0000 7fb0af59e640 1 -- 172.21.15.42:0/889576396 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb0b00a37f0 msgr2=0x7fb0b013e360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:41.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.887+0000 7fb0af59e640 1 --2- 172.21.15.42:0/889576396 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb0b00a37f0 0x7fb0b013e360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:11:41.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.887+0000 7fb0af59e640 1 -- 172.21.15.42:0/889576396 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb0b006db90 msgr2=0x7fb0b01cecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:41.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.887+0000 7fb0af59e640 1 --2- 172.21.15.42:0/889576396 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb0b006db90 0x7fb0b01cecd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:11:41.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.887+0000 7fb0af59e640 10 monclient: _finish_hunting 0 2024-09-16T04:11:41.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.887+0000 7fb0af59e640 1 monclient: found mon.noname-a 2024-09-16T04:11:41.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.887+0000 7fb0af59e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:11:41.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.887+0000 7fb0af59e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:11:41.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.887+0000 7fb0af59e640 1 -- 172.21.15.42:0/889576396 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0b0139b20 con 0x7fb0b00a3c00 2024-09-16T04:11:41.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.887+0000 7fb0ae59c640 1 --2- 172.21.15.42:0/889576396 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb0b00a37f0 0x7fb0b013e360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:11:41.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.887+0000 7fb0af59e640 1 --2- 172.21.15.42:0/889576396 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb0b00a3c00 0x7fb0b013e8a0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fb0a80ab000 tx=0x7fb0a80195e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:11:41.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.887+0000 7fb0a77fe640 10 client.?.objecter ms_handle_connect 0x7fb0b00a3c00 2024-09-16T04:11:41.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.887+0000 7fb0a77fe640 10 client.?.objecter resend_mon_ops 2024-09-16T04:11:41.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.887+0000 7fb0a77fe640 1 -- 172.21.15.42:0/889576396 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0a80bc030 con 0x7fb0b00a3c00 2024-09-16T04:11:41.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.887+0000 7fb0a77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:41.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.887+0000 7fb0a77fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:11:41.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.887+0000 7fb0a77fe640 10 monclient: dump: 2024-09-16T04:11:41.908 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:11:41.908 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:11:41.908 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:11:41.908 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:11:41.908 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:41.908 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:11:41.908 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:11:41.908 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:11:41.909 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:11:41.909 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:11:41.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.887+0000 7fb0a77fe640 10 monclient: _finish_auth 0 2024-09-16T04:11:41.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.887+0000 7fb0a77fe640 10 monclient: _check_auth_tickets 2024-09-16T04:11:41.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.887+0000 7fb0a77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:11:41.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.887+0000 7fb0b5017640 5 monclient: authenticate success, global_id 1034403 2024-09-16T04:11:41.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.887+0000 7fb0a77fe640 1 -- 172.21.15.42:0/889576396 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb0a80b8470 con 0x7fb0b00a3c00 2024-09-16T04:11:41.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.887+0000 7fb0b5017640 10 monclient: _renew_subs 2024-09-16T04:11:41.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.887+0000 7fb0b5017640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:11:41.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.887+0000 7fb0b5017640 1 -- 172.21.15.42:0/889576396 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0b01cf210 con 0x7fb0b00a3c00 2024-09-16T04:11:41.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.887+0000 7fb0a77fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:11:41.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.887+0000 7fb0a77fe640 1 -- 172.21.15.42:0/889576396 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0a80b8880 con 0x7fb0b00a3c00 2024-09-16T04:11:41.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.887+0000 7fb0a77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:41.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.887+0000 7fb0a77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:11:41.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.887+0000 7fb0a77fe640 10 monclient: dump: 2024-09-16T04:11:41.910 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:11:41.910 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:11:41.910 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:11:41.910 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:11:41.910 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:41.910 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:11:41.910 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:11:41.910 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:11:41.910 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:11:41.910 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:11:41.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.887+0000 7fb0b5017640 10 client.1034403.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:11:41.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.887+0000 7fb0b5017640 10 monclient: _renew_subs 2024-09-16T04:11:41.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.887+0000 7fb0acd99640 4 set_mon_vals no callback set 2024-09-16T04:11:41.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.887+0000 7fb0b5017640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:11:41.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.887+0000 7fb0b5017640 1 -- 172.21.15.42:0/889576396 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0b01332c0 con 0x7fb0b00a3c00 2024-09-16T04:11:41.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.887+0000 7fb0b5017640 1 librados: init done 2024-09-16T04:11:41.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.887+0000 7fb0b5017640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:41.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.887+0000 7fb0b5017640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:41.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.887+0000 7fb0b5017640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:11:41.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.887+0000 7fb0b5017640 1 -- 172.21.15.42:0/889576396 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb0b01340c0 con 0x7fb0b00a3c00 2024-09-16T04:11:41.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.887+0000 7fb0a77fe640 1 -- 172.21.15.42:0/889576396 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fb0a80bf070 con 0x7fb0b00a3c00 2024-09-16T04:11:41.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.887+0000 7fb0a77fe640 1 --2- 172.21.15.42:0/889576396 >> v2:172.21.15.154:6812/254845342 conn(0x7fb090037810 0x7fb090039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:11:41.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.887+0000 7fb0a77fe640 1 -- 172.21.15.42:0/889576396 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(706..706 src has 451..706) ==== 5040+0+0 (secure 0 0 0) 0x7fb0a80e38c0 con 0x7fb0b00a3c00 2024-09-16T04:11:41.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.887+0000 7fb0a77fe640 10 client.1034403.objecter ms_dispatch 0x7fb0b0000ba0 osd_map(706..706 src has 451..706) 2024-09-16T04:11:41.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.887+0000 7fb0a77fe640 3 client.1034403.objecter handle_osd_map got epochs [706,706] > 0 2024-09-16T04:11:41.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.887+0000 7fb0a77fe640 3 client.1034403.objecter handle_osd_map decoding full epoch 706 2024-09-16T04:11:41.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.887+0000 7fb0a77fe640 20 client.1034403.objecter dump_active .. 0 homeless 2024-09-16T04:11:41.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.887+0000 7fb0aed9d640 1 --2- 172.21.15.42:0/889576396 >> v2:172.21.15.154:6812/254845342 conn(0x7fb090037810 0x7fb090039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:11:41.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.887+0000 7fb0aed9d640 10 monclient: get_auth_request con 0x7fb090037810 auth_method 0 2024-09-16T04:11:41.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.887+0000 7fb0aed9d640 1 --2- 172.21.15.42:0/889576396 >> v2:172.21.15.154:6812/254845342 conn(0x7fb090037810 0x7fb090039cd0 secure :-1 s=READY pgs=3946 cs=0 l=1 rev1=1 crypto rx=0x7fb09800b930 tx=0x7fb098008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:11:41.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.887+0000 7fb0a77fe640 10 client.1034403.objecter ms_handle_connect 0x7fb090037810 2024-09-16T04:11:41.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.891+0000 7fb0a77fe640 1 -- 172.21.15.42:0/889576396 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7fb0a80c6070 con 0x7fb0b00a3c00 2024-09-16T04:11:41.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.891+0000 7fb0a77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:41.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:41.891+0000 7fb0a77fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:11:42.119 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.115+0000 7fb0b5017640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:11:42.119 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.115+0000 7fb0b5017640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:11:42.119 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.115+0000 7fb0b5017640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:11:42.120 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.115+0000 7fb0b5017640 1 -- 172.21.15.42:0/889576396 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fb0b01383f0 con 0x7fb0b00a3c00 2024-09-16T04:11:42.120 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.115+0000 7fb0a77fe640 1 -- 172.21.15.42:0/889576396 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7fb0a8016580 con 0x7fb0b00a3c00 2024-09-16T04:11:42.120 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.115+0000 7fb0a77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:11:42.120 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.115+0000 7fb0a77fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:11:42.120 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":268,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:11:42.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.119+0000 7fb0b5017640 10 librados: watch_flush enter 2024-09-16T04:11:42.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.119+0000 7fb0b5017640 10 librados: watch_flush exit 2024-09-16T04:11:42.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.119+0000 7fb0b5017640 20 client.1034403.objecter shutdown clearing up homeless session... 2024-09-16T04:11:42.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.119+0000 7fb0b5017640 10 client.1034403.objecter successfully canceled tick 2024-09-16T04:11:42.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.119+0000 7fb0b5017640 1 -- 172.21.15.42:0/889576396 >> v2:172.21.15.154:6812/254845342 conn(0x7fb090037810 msgr2=0x7fb090039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:42.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.119+0000 7fb0b5017640 1 --2- 172.21.15.42:0/889576396 >> v2:172.21.15.154:6812/254845342 conn(0x7fb090037810 0x7fb090039cd0 secure :-1 s=READY pgs=3946 cs=0 l=1 rev1=1 crypto rx=0x7fb09800b930 tx=0x7fb098008040 comp rx=0 tx=0).stop 2024-09-16T04:11:42.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.119+0000 7fb0b5017640 10 monclient: shutdown 2024-09-16T04:11:42.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.119+0000 7fb0b5017640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:11:42.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.119+0000 7fb0b5017640 1 -- 172.21.15.42:0/889576396 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb0b00a3c00 msgr2=0x7fb0b013e8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:42.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.119+0000 7fb0b5017640 1 --2- 172.21.15.42:0/889576396 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb0b00a3c00 0x7fb0b013e8a0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fb0a80ab000 tx=0x7fb0a80195e0 comp rx=0 tx=0).stop 2024-09-16T04:11:42.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.119+0000 7fb0b5017640 1 -- 172.21.15.42:0/889576396 shutdown_connections 2024-09-16T04:11:42.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.119+0000 7fb0b5017640 1 --2- 172.21.15.42:0/889576396 >> v2:172.21.15.154:6812/254845342 conn(0x7fb090037810 0x7fb090039cd0 unknown :-1 s=CLOSED pgs=3946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:11:42.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.119+0000 7fb0b5017640 1 --2- 172.21.15.42:0/889576396 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb0b00a3c00 0x7fb0b013e8a0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:11:42.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.119+0000 7fb0b5017640 1 --2- 172.21.15.42:0/889576396 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb0b00a37f0 0x7fb0b013e360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:11:42.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.119+0000 7fb0b5017640 1 --2- 172.21.15.42:0/889576396 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb0b006db90 0x7fb0b01cecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:11:42.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.119+0000 7fb0b5017640 1 -- 172.21.15.42:0/889576396 >> 172.21.15.42:0/889576396 conn(0x7fb0b00a9fb0 msgr2=0x7fb0b0136a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:11:42.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.119+0000 7fb0b5017640 1 -- 172.21.15.42:0/889576396 shutdown_connections 2024-09-16T04:11:42.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.119+0000 7fb0b5017640 1 -- 172.21.15.42:0/889576396 wait complete. 2024-09-16T04:11:42.123 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.119+0000 7fb0b5017640 1 librados: shutdown 2024-09-16T04:11:42.139 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T04:11:42.139 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T04:11:42.139 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T04:11:42.262 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.259+0000 7fb5b57a6640 10 monclient: get_monmap_and_config 2024-09-16T04:11:42.262 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.259+0000 7fb5b57a6640 10 monclient: build_initial_monmap 2024-09-16T04:11:42.263 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.259+0000 7fb5b57a6640 10 monclient: monmap: 2024-09-16T04:11:42.263 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:11:42.263 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:11:42.263 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:11:42.261382+0000 2024-09-16T04:11:42.263 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:11:42.261382+0000 2024-09-16T04:11:42.263 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:11:42.263 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:11:42.263 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:11:42.263 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:11:42.263 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:11:42.263 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:11:42.263 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:11:42.263 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:11:42.275 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.259+0000 7fb5b57a6640 1 Processor -- start 2024-09-16T04:11:42.276 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.259+0000 7fb5b57a6640 1 -- start start 2024-09-16T04:11:42.276 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.259+0000 7fb5b57a6640 10 monclient: init 2024-09-16T04:11:42.276 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.259+0000 7fb5b57a6640 10 monclient: _reopen_session rank -1 2024-09-16T04:11:42.276 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.259+0000 7fb5b57a6640 10 monclient: _add_conns ranks=[0,4,1,3,2] 2024-09-16T04:11:42.276 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.259+0000 7fb5b57a6640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb5b00a1330 addr v1:172.21.15.42:6789/0 2024-09-16T04:11:42.276 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.259+0000 7fb5b57a6640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fb5b006dac0 0x7fb5b013a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:11:42.276 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.259+0000 7fb5b57a6640 10 monclient(hunting): picked mon.noname-c con 0x7fb5b006dac0 addr v2:172.21.15.154:3300/0 2024-09-16T04:11:42.276 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.259+0000 7fb5b57a6640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb5b00a1e20 addr v1:172.21.15.154:6789/0 2024-09-16T04:11:42.276 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.259+0000 7fb5b57a6640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb5b009c090 con 0x7fb5b00a1330 2024-09-16T04:11:42.276 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.259+0000 7fb5b57a6640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb5b009f610 con 0x7fb5b00a1e20 2024-09-16T04:11:42.276 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.259+0000 7fb5b57a6640 10 monclient(hunting): start opening mon connection 2024-09-16T04:11:42.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.259+0000 7fb5b57a6640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fb5b009f8f0 con 0x7fb5b006dac0 2024-09-16T04:11:42.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.259+0000 7fb5b57a6640 10 monclient(hunting): _renew_subs 2024-09-16T04:11:42.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.259+0000 7fb5b57a6640 10 monclient(hunting): authenticate will time out at 2749.230954s 2024-09-16T04:11:42.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.259+0000 7fb5aeffd640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fb5b00a1330 0x7fb5b00a1710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:50822/0 (socket says 172.21.15.42:50822) 2024-09-16T04:11:42.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.259+0000 7fb5aeffd640 1 -- 172.21.15.42:0/2556195800 learned_addr learned my addr 172.21.15.42:0/2556195800 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T04:11:42.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.259+0000 7fb5ae7fc640 1 --2- 172.21.15.42:0/2556195800 >> v2:172.21.15.154:3300/0 conn(0x7fb5b006dac0 0x7fb5b013a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:11:42.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.259+0000 7fb5ae7fc640 10 monclient(hunting): get_auth_request con 0x7fb5b006dac0 auth_method 0 2024-09-16T04:11:42.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.259+0000 7fb5ae7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:11:42.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.259+0000 7fb5ae7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:42.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.259+0000 7fb5ae7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:42.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.259+0000 7fb5adffb640 1 -- 172.21.15.42:0/2556195800 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2432773085 0 0) 0x7fb5b009c090 con 0x7fb5b00a1330 2024-09-16T04:11:42.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.259+0000 7fb5adffb640 10 monclient(hunting): _negotiate 2024-09-16T04:11:42.278 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.259+0000 7fb5adffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:42.278 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.259+0000 7fb5adffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:42.278 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.259+0000 7fb5adffb640 10 monclient(hunting): my global_id is 1034406 2024-09-16T04:11:42.278 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.259+0000 7fb5adffb640 1 -- 172.21.15.42:0/2556195800 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb598003530 con 0x7fb5b00a1330 2024-09-16T04:11:42.278 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.259+0000 7fb5ae7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:11:42.278 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.259+0000 7fb5ae7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:11:42.278 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.259+0000 7fb5ae7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:11:42.278 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.259+0000 7fb5adffb640 1 -- 172.21.15.42:0/2556195800 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1565337466 0 0) 0x7fb598003530 con 0x7fb5b00a1330 2024-09-16T04:11:42.278 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.259+0000 7fb5adffb640 1 -- 172.21.15.42:0/2556195800 >> v2:172.21.15.154:3300/0 conn(0x7fb5b006dac0 msgr2=0x7fb5b013a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:11:42.278 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.259+0000 7fb5adffb640 1 --2- 172.21.15.42:0/2556195800 >> v2:172.21.15.154:3300/0 conn(0x7fb5b006dac0 0x7fb5b013a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:11:42.278 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.259+0000 7fb5adffb640 1 -- 172.21.15.42:0/2556195800 >> v1:172.21.15.154:6789/0 conn(0x7fb5b00a1e20 legacy=0x7fb5b013c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:42.279 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.259+0000 7fb5adffb640 10 monclient: _finish_hunting 0 2024-09-16T04:11:42.279 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.259+0000 7fb5adffb640 1 monclient: found mon.noname-a-legacy 2024-09-16T04:11:42.279 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.259+0000 7fb5adffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:11:42.279 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.259+0000 7fb5adffb640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.42:6789/0 2024-09-16T04:11:42.279 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.259+0000 7fb5adffb640 1 -- 172.21.15.42:0/2556195800 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5b009fbd0 con 0x7fb5b00a1330 2024-09-16T04:11:42.279 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.259+0000 7fb5adffb640 10 monclient: _finish_auth 0 2024-09-16T04:11:42.279 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.259+0000 7fb5adffb640 10 monclient: _check_auth_tickets 2024-09-16T04:11:42.279 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.259+0000 7fb5adffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:11:42.279 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.259+0000 7fb5adffb640 1 -- 172.21.15.42:0/2556195800 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2177413734 0 0) 0x7fb5a4001680 con 0x7fb5b00a1330 2024-09-16T04:11:42.279 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.259+0000 7fb5b57a6640 5 monclient: authenticate success, global_id 1034406 2024-09-16T04:11:42.279 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.259+0000 7fb5b57a6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:11:42.279 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.259+0000 7fb5adffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:42.279 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.259+0000 7fb5adffb640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T04:11:42.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.259+0000 7fb5adffb640 10 monclient: dump: 2024-09-16T04:11:42.280 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:11:42.280 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:11:42.280 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:11:42.280 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:11:42.280 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:42.280 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:11:42.280 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:11:42.281 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:11:42.281 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:11:42.281 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:11:42.281 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.259+0000 7fb5adffb640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] but i'm connected to v1:172.21.15.42:6789/0, reconnecting 2024-09-16T04:11:42.281 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.259+0000 7fb5adffb640 10 monclient: _reopen_session rank -1 2024-09-16T04:11:42.281 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.259+0000 7fb5adffb640 1 -- 172.21.15.42:0/2556195800 >> v1:172.21.15.42:6789/0 conn(0x7fb5b00a1330 legacy=0x7fb5b00a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:42.281 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.259+0000 7fb5adffb640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:11:42.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.259+0000 7fb5adffb640 1 --2- 172.21.15.42:0/2556195800 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb598003a10 0x7fb598003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:11:42.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.259+0000 7fb5adffb640 10 monclient(hunting): picked mon.b con 0x7fb598003a10 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:11:42.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.259+0000 7fb5adffb640 1 --2- 172.21.15.42:0/2556195800 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb598004810 0x7fb598004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:11:42.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.259+0000 7fb5adffb640 10 monclient(hunting): picked mon.c con 0x7fb598004810 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:11:42.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.259+0000 7fb5adffb640 1 --2- 172.21.15.42:0/2556195800 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb598005220 0x7fb59800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:11:42.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.259+0000 7fb5adffb640 10 monclient(hunting): picked mon.a con 0x7fb598005220 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:11:42.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.259+0000 7fb5adffb640 10 monclient(hunting): start opening mon connection 2024-09-16T04:11:42.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.259+0000 7fb5adffb640 1 -- 172.21.15.42:0/2556195800 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fb5b009f8f0 con 0x7fb598005220 2024-09-16T04:11:42.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.259+0000 7fb5adffb640 10 monclient(hunting): start opening mon connection 2024-09-16T04:11:42.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.259+0000 7fb5adffb640 1 -- 172.21.15.42:0/2556195800 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fb59800c170 con 0x7fb598003a10 2024-09-16T04:11:42.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.259+0000 7fb5adffb640 10 monclient(hunting): start opening mon connection 2024-09-16T04:11:42.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.259+0000 7fb5adffb640 1 -- 172.21.15.42:0/2556195800 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fb59800c370 con 0x7fb598004810 2024-09-16T04:11:42.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.259+0000 7fb5adffb640 10 monclient(hunting): _renew_subs 2024-09-16T04:11:42.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.259+0000 7fb5adffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:11:42.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.263+0000 7fb5b57a6640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-16T04:11:42.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.263+0000 7fb5ae7fc640 1 --2- 172.21.15.42:0/2556195800 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb598004810 0x7fb598004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:11:42.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.263+0000 7fb5ae7fc640 10 monclient(hunting): get_auth_request con 0x7fb598004810 auth_method 0 2024-09-16T04:11:42.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.263+0000 7fb5ae7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:11:42.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.263+0000 7fb5ae7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:42.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.263+0000 7fb5ae7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:11:42.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.263+0000 7fb5af7fe640 1 --2- 172.21.15.42:0/2556195800 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb598005220 0x7fb59800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:11:42.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.263+0000 7fb5af7fe640 10 monclient(hunting): get_auth_request con 0x7fb598005220 auth_method 0 2024-09-16T04:11:42.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.263+0000 7fb5af7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:11:42.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.263+0000 7fb5af7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:42.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.263+0000 7fb5af7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:11:42.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.263+0000 7fb5ae7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:11:42.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.263+0000 7fb5ae7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:11:42.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.263+0000 7fb5aeffd640 1 --2- 172.21.15.42:0/2556195800 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb598003a10 0x7fb598003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:11:42.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.263+0000 7fb5ae7fc640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:11:42.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.263+0000 7fb5af7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:11:42.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.263+0000 7fb5af7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:11:42.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.263+0000 7fb5af7fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:11:42.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.263+0000 7fb5aeffd640 10 monclient(hunting): get_auth_request con 0x7fb598003a10 auth_method 0 2024-09-16T04:11:42.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.263+0000 7fb5aeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:11:42.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.263+0000 7fb5aeffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:42.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.263+0000 7fb5aeffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:11:42.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.263+0000 7fb5ae7fc640 10 monclient(hunting): handle_auth_done global_id 1034406 payload 293 2024-09-16T04:11:42.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.263+0000 7fb5ae7fc640 1 -- 172.21.15.42:0/2556195800 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb598003a10 msgr2=0x7fb598003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:11:42.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.263+0000 7fb5ae7fc640 1 --2- 172.21.15.42:0/2556195800 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb598003a10 0x7fb598003e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:11:42.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.263+0000 7fb5ae7fc640 1 -- 172.21.15.42:0/2556195800 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb598005220 msgr2=0x7fb59800b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:11:42.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.263+0000 7fb5ae7fc640 1 --2- 172.21.15.42:0/2556195800 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb598005220 0x7fb59800b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:11:42.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.263+0000 7fb5ae7fc640 10 monclient: _finish_hunting 0 2024-09-16T04:11:42.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.263+0000 7fb5ae7fc640 1 monclient: found mon.c 2024-09-16T04:11:42.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.263+0000 7fb5ae7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:11:42.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.263+0000 7fb5ae7fc640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:11:42.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.263+0000 7fb5ae7fc640 1 -- 172.21.15.42:0/2556195800 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb598003530 con 0x7fb598004810 2024-09-16T04:11:42.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.263+0000 7fb5ae7fc640 10 monclient: _finish_auth 0 2024-09-16T04:11:42.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.263+0000 7fb5ae7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:11:42.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.263+0000 7fb5ae7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:11:42.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.263+0000 7fb5aeffd640 1 --2- 172.21.15.42:0/2556195800 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb598003a10 0x7fb598003e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T04:11:42.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.263+0000 7fb5af7fe640 1 --2- 172.21.15.42:0/2556195800 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb598005220 0x7fb59800b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:11:42.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.263+0000 7fb5af7fe640 1 -- 172.21.15.42:0/2556195800 reap_dead start 2024-09-16T04:11:42.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.263+0000 7fb5ae7fc640 1 --2- 172.21.15.42:0/2556195800 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb598004810 0x7fb598004c90 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fb5a0009f90 tx=0x7fb5a001cc20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7055b5e8d4de8bd3 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:11:42.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.263+0000 7fb5adffb640 1 -- 172.21.15.42:0/2556195800 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5a00b2070 con 0x7fb598004810 2024-09-16T04:11:42.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.263+0000 7fb5adffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:42.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.263+0000 7fb5adffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:11:42.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.263+0000 7fb5adffb640 10 monclient: dump: 2024-09-16T04:11:42.286 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:11:42.286 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:11:42.286 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:11:42.286 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:11:42.286 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:42.286 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:11:42.286 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:11:42.286 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:11:42.287 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:11:42.287 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:11:42.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.263+0000 7fb5b57a6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:11:42.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.263+0000 7fb5adffb640 1 -- 172.21.15.42:0/2556195800 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb5a00a5b20 con 0x7fb598004810 2024-09-16T04:11:42.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.263+0000 7fb5adffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:11:42.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.263+0000 7fb5b57a6640 10 monclient: get_monmap_and_config success 2024-09-16T04:11:42.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.263+0000 7fb5b57a6640 4 set_mon_vals no callback set 2024-09-16T04:11:42.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.263+0000 7fb5adffb640 1 -- 172.21.15.42:0/2556195800 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5a0009c10 con 0x7fb598004810 2024-09-16T04:11:42.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.263+0000 7fb5adffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:42.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.263+0000 7fb5adffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:11:42.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.263+0000 7fb5adffb640 10 monclient: dump: 2024-09-16T04:11:42.287 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:11:42.287 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:11:42.288 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:11:42.288 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:11:42.288 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:42.288 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:11:42.288 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:11:42.288 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:11:42.289 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:11:42.289 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:11:42.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.263+0000 7fb5b57a6640 10 monclient: shutdown 2024-09-16T04:11:42.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.263+0000 7fb5b57a6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:11:42.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.263+0000 7fb5b57a6640 1 -- 172.21.15.42:0/2556195800 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb598004810 msgr2=0x7fb598004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:42.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.263+0000 7fb5b57a6640 1 --2- 172.21.15.42:0/2556195800 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb598004810 0x7fb598004c90 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fb5a0009f90 tx=0x7fb5a001cc20 comp rx=0 tx=0).stop 2024-09-16T04:11:42.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.263+0000 7fb5b57a6640 1 -- 172.21.15.42:0/2556195800 shutdown_connections 2024-09-16T04:11:42.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.263+0000 7fb5b57a6640 1 --2- 172.21.15.42:0/2556195800 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb598004810 0x7fb598004c90 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:11:42.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.263+0000 7fb5b57a6640 1 -- 172.21.15.42:0/2556195800 >> 172.21.15.42:0/2556195800 conn(0x7fb5b00a2420 msgr2=0x7fb5b00a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:11:42.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.263+0000 7fb5b57a6640 1 -- 172.21.15.42:0/2556195800 shutdown_connections 2024-09-16T04:11:42.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.263+0000 7fb5b57a6640 1 -- 172.21.15.42:0/2556195800 wait complete. 2024-09-16T04:11:42.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.263+0000 7fb5b57a6640 10 monclient: build_initial_monmap 2024-09-16T04:11:42.291 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.263+0000 7fb5b57a6640 10 monclient: monmap: 2024-09-16T04:11:42.291 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:11:42.291 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:11:42.291 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:11:42.291 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:11:42.291 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:11:42.292 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:11:42.292 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:11:42.292 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:11:42.292 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:11:42.292 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:11:42.292 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.267+0000 7fb5b57a6640 1 librados: starting msgr at 2024-09-16T04:11:42.292 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.267+0000 7fb5b57a6640 1 librados: starting objecter 2024-09-16T04:11:42.292 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.267+0000 7fb5b57a6640 1 Processor -- start 2024-09-16T04:11:42.293 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.267+0000 7fb5b57a6640 1 -- start start 2024-09-16T04:11:42.293 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.267+0000 7fb5b57a6640 1 librados: setting wanted keys 2024-09-16T04:11:42.293 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.267+0000 7fb5b57a6640 1 librados: calling monclient init 2024-09-16T04:11:42.293 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.267+0000 7fb5b57a6640 10 monclient: init 2024-09-16T04:11:42.293 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.267+0000 7fb5b57a6640 10 monclient: _reopen_session rank -1 2024-09-16T04:11:42.293 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.267+0000 7fb5b57a6640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:11:42.293 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.267+0000 7fb5b57a6640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb598004810 0x7fb5b01ca380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:11:42.294 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.267+0000 7fb5b57a6640 10 monclient(hunting): picked mon.noname-a con 0x7fb598004810 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:11:42.294 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.267+0000 7fb5b57a6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb5b01ca8c0 0x7fb5b0122980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:11:42.294 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.267+0000 7fb5b57a6640 10 monclient(hunting): picked mon.noname-b con 0x7fb5b01ca8c0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:11:42.294 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.267+0000 7fb5b57a6640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb5b01cad20 0x7fb5b0122ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:11:42.295 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.267+0000 7fb5b57a6640 10 monclient(hunting): picked mon.noname-c con 0x7fb5b01cad20 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:11:42.295 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.267+0000 7fb5b57a6640 10 monclient(hunting): start opening mon connection 2024-09-16T04:11:42.295 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.267+0000 7fb5b57a6640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fb5b009eef0 con 0x7fb598004810 2024-09-16T04:11:42.295 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.267+0000 7fb5b57a6640 10 monclient(hunting): start opening mon connection 2024-09-16T04:11:42.295 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.267+0000 7fb5b57a6640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fb5b009ed70 con 0x7fb5b01ca8c0 2024-09-16T04:11:42.295 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.267+0000 7fb5b57a6640 10 monclient(hunting): start opening mon connection 2024-09-16T04:11:42.296 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.267+0000 7fb5b57a6640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fb5b00a1710 con 0x7fb5b01cad20 2024-09-16T04:11:42.296 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.267+0000 7fb5b57a6640 10 monclient(hunting): _renew_subs 2024-09-16T04:11:42.296 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.267+0000 7fb5b57a6640 10 monclient(hunting): authenticate will time out at 2749.236922s 2024-09-16T04:11:42.296 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.267+0000 7fb5aeffd640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb598004810 0x7fb5b01ca380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:11:42.296 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.267+0000 7fb5af7fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb5b01cad20 0x7fb5b0122ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:11:42.296 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.267+0000 7fb5aeffd640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb598004810 0x7fb5b01ca380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:57596/0 (socket says 172.21.15.42:57596) 2024-09-16T04:11:42.296 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.267+0000 7fb5aeffd640 1 -- 172.21.15.42:0/1591139730 learned_addr learned my addr 172.21.15.42:0/1591139730 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:11:42.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.267+0000 7fb5aeffd640 10 monclient(hunting): get_auth_request con 0x7fb598004810 auth_method 0 2024-09-16T04:11:42.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.267+0000 7fb5aeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:11:42.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.267+0000 7fb5aeffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:42.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.267+0000 7fb5aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:42.300 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.267+0000 7fb5af7fe640 10 monclient(hunting): get_auth_request con 0x7fb5b01cad20 auth_method 0 2024-09-16T04:11:42.300 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.267+0000 7fb5af7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:11:42.300 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.267+0000 7fb5af7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:42.300 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.267+0000 7fb5af7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:42.301 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.267+0000 7fb5aeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:11:42.301 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.267+0000 7fb5aeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:11:42.301 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.267+0000 7fb5aeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:11:42.301 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.267+0000 7fb5af7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:11:42.301 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.267+0000 7fb5af7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:11:42.301 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.267+0000 7fb5af7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:11:42.301 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.267+0000 7fb5ae7fc640 1 --2- 172.21.15.42:0/1591139730 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb5b01ca8c0 0x7fb5b0122980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:11:42.301 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.267+0000 7fb5ae7fc640 10 monclient(hunting): get_auth_request con 0x7fb5b01ca8c0 auth_method 0 2024-09-16T04:11:42.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.267+0000 7fb5ae7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:11:42.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.267+0000 7fb5ae7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:42.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.267+0000 7fb5ae7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:42.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.267+0000 7fb5aeffd640 10 monclient(hunting): handle_auth_done global_id 1034409 payload 824 2024-09-16T04:11:42.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.267+0000 7fb5aeffd640 1 -- 172.21.15.42:0/1591139730 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb5b01cad20 msgr2=0x7fb5b0122ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:42.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.267+0000 7fb5aeffd640 1 --2- 172.21.15.42:0/1591139730 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb5b01cad20 0x7fb5b0122ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:11:42.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.267+0000 7fb5aeffd640 1 -- 172.21.15.42:0/1591139730 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb5b01ca8c0 msgr2=0x7fb5b0122980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:42.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.267+0000 7fb5aeffd640 1 --2- 172.21.15.42:0/1591139730 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb5b01ca8c0 0x7fb5b0122980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:11:42.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.267+0000 7fb5aeffd640 10 monclient: _finish_hunting 0 2024-09-16T04:11:42.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.267+0000 7fb5aeffd640 1 monclient: found mon.noname-a 2024-09-16T04:11:42.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.267+0000 7fb5aeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:11:42.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.267+0000 7fb5aeffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:11:42.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.267+0000 7fb5aeffd640 1 -- 172.21.15.42:0/1591139730 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5b0125180 con 0x7fb598004810 2024-09-16T04:11:42.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.267+0000 7fb5af7fe640 1 --2- 172.21.15.42:0/1591139730 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb5b01cad20 0x7fb5b0122ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:11:42.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.267+0000 7fb5aeffd640 1 --2- 172.21.15.42:0/1591139730 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb598004810 0x7fb5b01ca380 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fb5a4000fc0 tx=0x7fb5a4011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:11:42.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.267+0000 7fb5977fe640 10 client.?.objecter ms_handle_connect 0x7fb598004810 2024-09-16T04:11:42.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.267+0000 7fb5977fe640 10 client.?.objecter resend_mon_ops 2024-09-16T04:11:42.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.267+0000 7fb5977fe640 1 -- 172.21.15.42:0/1591139730 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5a4010040 con 0x7fb598004810 2024-09-16T04:11:42.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.267+0000 7fb5977fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:42.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.267+0000 7fb5977fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:11:42.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.267+0000 7fb5977fe640 10 monclient: dump: 2024-09-16T04:11:42.305 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:11:42.305 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:11:42.305 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:11:42.306 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:11:42.306 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:42.306 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:11:42.306 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:11:42.306 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:11:42.306 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:11:42.307 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:11:42.307 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.267+0000 7fb5977fe640 10 monclient: _finish_auth 0 2024-09-16T04:11:42.307 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.267+0000 7fb5977fe640 10 monclient: _check_auth_tickets 2024-09-16T04:11:42.307 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.267+0000 7fb5977fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:11:42.307 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.267+0000 7fb5977fe640 1 -- 172.21.15.42:0/1591139730 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb5a401a070 con 0x7fb598004810 2024-09-16T04:11:42.307 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.267+0000 7fb5b57a6640 5 monclient: authenticate success, global_id 1034409 2024-09-16T04:11:42.308 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.267+0000 7fb5b57a6640 10 monclient: _renew_subs 2024-09-16T04:11:42.308 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.267+0000 7fb5b57a6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:11:42.308 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.267+0000 7fb5b57a6640 1 -- 172.21.15.42:0/1591139730 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5b0126350 con 0x7fb598004810 2024-09-16T04:11:42.308 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.267+0000 7fb5977fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:11:42.308 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.267+0000 7fb5977fe640 1 -- 172.21.15.42:0/1591139730 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5a400fb20 con 0x7fb598004810 2024-09-16T04:11:42.309 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.267+0000 7fb5977fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:42.309 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.267+0000 7fb5977fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:11:42.309 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.267+0000 7fb5977fe640 10 monclient: dump: 2024-09-16T04:11:42.309 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:11:42.309 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:11:42.309 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:11:42.310 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:11:42.310 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:42.310 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:11:42.310 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:11:42.310 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:11:42.310 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:11:42.311 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:11:42.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.267+0000 7fb5b57a6640 10 client.1034409.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:11:42.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.267+0000 7fb5b57a6640 10 monclient: _renew_subs 2024-09-16T04:11:42.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.267+0000 7fb5ad7fa640 4 set_mon_vals no callback set 2024-09-16T04:11:42.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.267+0000 7fb5b57a6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:11:42.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.267+0000 7fb5b57a6640 1 -- 172.21.15.42:0/1591139730 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5b0126630 con 0x7fb598004810 2024-09-16T04:11:42.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.267+0000 7fb5b57a6640 1 librados: init done 2024-09-16T04:11:42.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.267+0000 7fb5b57a6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:42.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.267+0000 7fb5b57a6640 1 --2- 172.21.15.42:0/1591139730 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb5b002c650 0x7fb5b002ca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:11:42.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.267+0000 7fb5b57a6640 10 monclient(hunting): start opening mon connection 2024-09-16T04:11:42.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.267+0000 7fb5b57a6640 1 -- 172.21.15.42:0/1591139730 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fb5b00712f0 con 0x7fb5b002c650 2024-09-16T04:11:42.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.267+0000 7fb5ae7fc640 1 --2- 172.21.15.42:0/1591139730 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb5b002c650 0x7fb5b002ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:11:42.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.267+0000 7fb5ae7fc640 10 monclient: get_auth_request con 0x7fb5b002c650 auth_method 0 2024-09-16T04:11:42.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.267+0000 7fb5ae7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:11:42.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.267+0000 7fb5ae7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:42.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.267+0000 7fb5ae7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:42.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.267+0000 7fb5ae7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:11:42.314 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.267+0000 7fb5ae7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:11:42.314 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.267+0000 7fb5ae7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:11:42.314 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.267+0000 7fb5977fe640 1 -- 172.21.15.42:0/1591139730 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fb5a400fcc0 con 0x7fb598004810 2024-09-16T04:11:42.314 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.267+0000 7fb5ae7fc640 10 monclient(hunting): handle_auth_done global_id 1034412 payload 824 2024-09-16T04:11:42.314 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.267+0000 7fb5ae7fc640 1 -- 172.21.15.42:0/1591139730 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb5b00a1aa0 con 0x7fb5b002c650 2024-09-16T04:11:42.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.267+0000 7fb5977fe640 1 --2- 172.21.15.42:0/1591139730 >> v2:172.21.15.154:6812/254845342 conn(0x7fb59c037810 0x7fb59c039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:11:42.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.267+0000 7fb5977fe640 1 -- 172.21.15.42:0/1591139730 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(706..706 src has 451..706) ==== 5040+0+0 (secure 0 0 0) 0x7fb5a404bcd0 con 0x7fb598004810 2024-09-16T04:11:42.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.267+0000 7fb5977fe640 10 client.1034409.objecter ms_dispatch 0x7fb5b0000ba0 osd_map(706..706 src has 451..706) 2024-09-16T04:11:42.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.267+0000 7fb5977fe640 3 client.1034409.objecter handle_osd_map got epochs [706,706] > 0 2024-09-16T04:11:42.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.267+0000 7fb5977fe640 3 client.1034409.objecter handle_osd_map decoding full epoch 706 2024-09-16T04:11:42.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.267+0000 7fb5977fe640 20 client.1034409.objecter dump_active .. 0 homeless 2024-09-16T04:11:42.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.267+0000 7fb5977fe640 10 client.1034409.objecter ms_handle_connect 0x7fb5b002c650 2024-09-16T04:11:42.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.267+0000 7fb5977fe640 10 client.1034409.objecter resend_mon_ops 2024-09-16T04:11:42.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.267+0000 7fb5ae7fc640 1 --2- 172.21.15.42:0/1591139730 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb5b002c650 0x7fb5b002ca30 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fb598004790 tx=0x7fb598005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:11:42.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.267+0000 7fb5977fe640 1 -- 172.21.15.42:0/1591139730 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5a00bd070 con 0x7fb5b002c650 2024-09-16T04:11:42.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.267+0000 7fb5977fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:42.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.267+0000 7fb5977fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:11:42.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.267+0000 7fb5977fe640 10 monclient: dump: 2024-09-16T04:11:42.317 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:11:42.317 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:11:42.317 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:11:42.318 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:11:42.318 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:42.318 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:11:42.318 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:11:42.318 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:11:42.318 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:11:42.319 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:11:42.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.267+0000 7fb5af7fe640 1 --2- 172.21.15.42:0/1591139730 >> v2:172.21.15.154:6812/254845342 conn(0x7fb59c037810 0x7fb59c039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:11:42.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.267+0000 7fb5af7fe640 10 monclient: get_auth_request con 0x7fb59c037810 auth_method 0 2024-09-16T04:11:42.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.267+0000 7fb5977fe640 1 -- 172.21.15.42:0/1591139730 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fb5a0003ba0 con 0x7fb5b002c650 2024-09-16T04:11:42.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.267+0000 7fb5977fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:42.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.267+0000 7fb5977fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:11:42.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.267+0000 7fb5977fe640 1 -- 172.21.15.42:0/1591139730 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb5b002c650 msgr2=0x7fb5b002ca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:42.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.267+0000 7fb5977fe640 1 --2- 172.21.15.42:0/1591139730 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb5b002c650 0x7fb5b002ca30 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fb598004790 tx=0x7fb598005840 comp rx=0 tx=0).stop 2024-09-16T04:11:42.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.267+0000 7fb5977fe640 1 -- 172.21.15.42:0/1591139730 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb5b002c650 msgr2=0x7fb5b002ca30 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:11:42.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.267+0000 7fb5977fe640 1 --2- 172.21.15.42:0/1591139730 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb5b002c650 0x7fb5b002ca30 secure :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fb598004790 tx=0x7fb598005840 comp rx=0 tx=0).stop 2024-09-16T04:11:42.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.271+0000 7fb5af7fe640 1 --2- 172.21.15.42:0/1591139730 >> v2:172.21.15.154:6812/254845342 conn(0x7fb59c037810 0x7fb59c039cd0 secure :-1 s=READY pgs=3947 cs=0 l=1 rev1=1 crypto rx=0x7fb5b006da10 tx=0x7fb5a8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:11:42.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.271+0000 7fb5977fe640 10 client.1034409.objecter ms_handle_connect 0x7fb59c037810 2024-09-16T04:11:42.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.275+0000 7fb5b57a6640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:11:42.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.275+0000 7fb5b57a6640 1 --2- 172.21.15.42:0/1591139730 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb5b002cf70 0x7fb5b002d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:11:42.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.275+0000 7fb5b57a6640 10 monclient(hunting): start opening mon connection 2024-09-16T04:11:42.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.275+0000 7fb5b57a6640 1 -- 172.21.15.42:0/1591139730 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fb5b0126870 con 0x7fb5b002cf70 2024-09-16T04:11:42.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.275+0000 7fb5ae7fc640 1 --2- 172.21.15.42:0/1591139730 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb5b002cf70 0x7fb5b002d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:11:42.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.275+0000 7fb5ae7fc640 10 monclient: get_auth_request con 0x7fb5b002cf70 auth_method 0 2024-09-16T04:11:42.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.275+0000 7fb5ae7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:11:42.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.275+0000 7fb5ae7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:42.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.275+0000 7fb5ae7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:42.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.275+0000 7fb5ae7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:11:42.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.275+0000 7fb5ae7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:11:42.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.275+0000 7fb5ae7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:11:42.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.275+0000 7fb5ae7fc640 10 monclient(hunting): handle_auth_done global_id 1034415 payload 824 2024-09-16T04:11:42.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.275+0000 7fb5ae7fc640 1 -- 172.21.15.42:0/1591139730 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fb5b011f690 con 0x7fb5b002cf70 2024-09-16T04:11:42.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.275+0000 7fb5ae7fc640 1 --2- 172.21.15.42:0/1591139730 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb5b002cf70 0x7fb5b002d3a0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fb5a001cc60 tx=0x7fb5a0009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:11:42.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.275+0000 7fb5977fe640 10 client.1034409.objecter ms_handle_connect 0x7fb5b002cf70 2024-09-16T04:11:42.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.275+0000 7fb5977fe640 10 client.1034409.objecter resend_mon_ops 2024-09-16T04:11:42.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.275+0000 7fb5977fe640 1 -- 172.21.15.42:0/1591139730 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5a00bd030 con 0x7fb5b002cf70 2024-09-16T04:11:42.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.275+0000 7fb5977fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:42.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.275+0000 7fb5977fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:11:42.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.275+0000 7fb5977fe640 10 monclient: dump: 2024-09-16T04:11:42.327 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:11:42.328 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:11:42.328 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:11:42.328 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:11:42.328 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:42.328 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:11:42.329 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:11:42.329 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:11:42.329 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:11:42.329 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:11:42.329 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.275+0000 7fb5977fe640 1 -- 172.21.15.42:0/1591139730 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4882 (secure 0 0 0) 0x7fb5a00c6070 con 0x7fb5b002cf70 2024-09-16T04:11:42.329 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.275+0000 7fb5977fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:11:42.329 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.275+0000 7fb5977fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:11:42.329 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.275+0000 7fb5977fe640 1 -- 172.21.15.42:0/1591139730 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb5b002cf70 msgr2=0x7fb5b002d3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:42.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.275+0000 7fb5977fe640 1 --2- 172.21.15.42:0/1591139730 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb5b002cf70 0x7fb5b002d3a0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fb5a001cc60 tx=0x7fb5a0009750 comp rx=0 tx=0).stop 2024-09-16T04:11:42.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.275+0000 7fb5977fe640 1 -- 172.21.15.42:0/1591139730 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb5b002cf70 msgr2=0x7fb5b002d3a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:11:42.330 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.275+0000 7fb5977fe640 1 --2- 172.21.15.42:0/1591139730 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb5b002cf70 0x7fb5b002d3a0 secure :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fb5a001cc60 tx=0x7fb5a0009750 comp rx=0 tx=0).stop 2024-09-16T04:11:42.330 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T04:11:42.331 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:11:42.331 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:11:42.331 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "leader", 2024-09-16T04:11:42.331 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 268, 2024-09-16T04:11:42.331 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 182210, 2024-09-16T04:11:42.331 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T04:11:42.331 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T04:11:42.331 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T04:11:42.331 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T04:11:42.332 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:11:42.332 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 0, 2024-09-16T04:11:42.332 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:11:42.332 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T04:11:42.332 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T04:11:42.332 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:11:42.332 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:11:42.332 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:11:42.332 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:11:42.332 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:11:42.332 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:11:42.332 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:11:42.332 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:11:42.333 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:11:42.333 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:11:42.333 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:11:42.333 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:11:42.333 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:11:42.333 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T04:11:42.333 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:11:42.333 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:11:42.333 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:11:42.333 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:11:42.333 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:11:42.333 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:11:42.334 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:11:42.334 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:11:42.334 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:11:42.334 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:11:42.334 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:11:42.334 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:11:42.334 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:11:42.334 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T04:11:42.334 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T04:11:42.334 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T04:11:42.334 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T04:11:42.335 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T04:11:42.335 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T04:11:42.335 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:11:42.335 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:11:42.335 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T04:11:42.335 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T04:11:42.335 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T04:11:42.335 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T04:11:42.335 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T04:11:42.335 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T04:11:42.335 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T04:11:42.335 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:11:42.335 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T04:11:42.336 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:11:42.336 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:11:42.336 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:11:42.336 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:11:42.336 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:11:42.336 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:11:42.337 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:11:42.337 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:11:42.337 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:11:42.337 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:11:42.337 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:11:42.338 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:11:42.338 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T04:11:42.338 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:11:42.338 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T04:11:42.338 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:11:42.338 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:11:42.338 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:11:42.338 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:11:42.338 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:11:42.338 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:11:42.338 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:11:42.339 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T04:11:42.339 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:11:42.339 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:11:42.339 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:11:42.339 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:11:42.339 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T04:11:42.339 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:11:42.339 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:11:42.339 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:11:42.339 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:11:42.340 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T04:11:42.340 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T04:11:42.340 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:11:42.340 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:11:42.340 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:11:42.340 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:11:42.340 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:11:42.340 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:11:42.340 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:11:42.340 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:11:42.341 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:11:42.341 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:11:42.341 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:11:42.341 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T04:11:42.341 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:11:42.341 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:11:42.341 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:11:42.341 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:11:42.341 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T04:11:42.341 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:11:42.341 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:11:42.341 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:11:42.341 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:11:42.342 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T04:11:42.342 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T04:11:42.342 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:11:42.342 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:11:42.342 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:11:42.342 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:11:42.342 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:11:42.342 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:11:42.342 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:11:42.342 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:11:42.342 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:11:42.342 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:11:42.342 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:11:42.343 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T04:11:42.343 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:11:42.343 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:11:42.343 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:11:42.343 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:11:42.343 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T04:11:42.343 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:11:42.343 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:11:42.343 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:11:42.343 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:11:42.343 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T04:11:42.343 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T04:11:42.343 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:11:42.344 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:11:42.344 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:11:42.344 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:11:42.344 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:11:42.344 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:11:42.344 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T04:11:42.344 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T04:11:42.344 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:11:42.344 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:11:42.344 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:11:42.344 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:11:42.344 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:11:42.344 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:11:42.345 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T04:11:42.345 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:11:42.345 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:11:42.345 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:11:42.345 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 2 2024-09-16T04:11:42.345 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:11:42.345 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:11:42.345 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T04:11:42.345 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:11:42.345 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:11:42.345 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:11:42.345 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 5 2024-09-16T04:11:42.345 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:11:42.346 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:11:42.346 INFO:teuthology.orchestra.run.smithi042.stdout: "mgr": [ 2024-09-16T04:11:42.346 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:11:42.346 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:11:42.346 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:11:42.346 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:11:42.346 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:11:42.346 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:11:42.346 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:11:42.346 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T04:11:42.346 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T04:11:42.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.275+0000 7fb5b57a6640 10 librados: watch_flush enter 2024-09-16T04:11:42.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.275+0000 7fb5b57a6640 10 librados: watch_flush exit 2024-09-16T04:11:42.348 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.275+0000 7fb5b57a6640 20 client.1034409.objecter shutdown clearing up homeless session... 2024-09-16T04:11:42.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.275+0000 7fb5b57a6640 10 client.1034409.objecter successfully canceled tick 2024-09-16T04:11:42.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.275+0000 7fb5b57a6640 1 -- 172.21.15.42:0/1591139730 >> v2:172.21.15.154:6812/254845342 conn(0x7fb59c037810 msgr2=0x7fb59c039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:42.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.275+0000 7fb5b57a6640 1 --2- 172.21.15.42:0/1591139730 >> v2:172.21.15.154:6812/254845342 conn(0x7fb59c037810 0x7fb59c039cd0 secure :-1 s=READY pgs=3947 cs=0 l=1 rev1=1 crypto rx=0x7fb5b006da10 tx=0x7fb5a8009040 comp rx=0 tx=0).stop 2024-09-16T04:11:42.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.275+0000 7fb5b57a6640 10 monclient: shutdown 2024-09-16T04:11:42.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.275+0000 7fb5b57a6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:11:42.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.275+0000 7fb5b57a6640 1 -- 172.21.15.42:0/1591139730 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb598004810 msgr2=0x7fb5b01ca380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:42.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.275+0000 7fb5b57a6640 1 --2- 172.21.15.42:0/1591139730 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb598004810 0x7fb5b01ca380 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fb5a4000fc0 tx=0x7fb5a4011b60 comp rx=0 tx=0).stop 2024-09-16T04:11:42.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.275+0000 7fb5af7fe640 1 -- 172.21.15.42:0/1591139730 reap_dead start 2024-09-16T04:11:42.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.275+0000 7fb5b57a6640 1 -- 172.21.15.42:0/1591139730 shutdown_connections 2024-09-16T04:11:42.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.275+0000 7fb5b57a6640 1 --2- 172.21.15.42:0/1591139730 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb598004810 0x7fb5b01ca380 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:11:42.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.275+0000 7fb5b57a6640 1 -- 172.21.15.42:0/1591139730 >> 172.21.15.42:0/1591139730 conn(0x7fb5b00a2420 msgr2=0x7fb5b0071840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:11:42.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.275+0000 7fb5b57a6640 1 -- 172.21.15.42:0/1591139730 shutdown_connections 2024-09-16T04:11:42.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.275+0000 7fb5b57a6640 1 -- 172.21.15.42:0/1591139730 wait complete. 2024-09-16T04:11:42.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.275+0000 7fb5b57a6640 1 librados: shutdown 2024-09-16T04:11:42.350 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:11:42.350 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T04:11:42.350 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T04:11:42.468 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.463+0000 7fe36c7c7640 10 monclient: get_monmap_and_config 2024-09-16T04:11:42.468 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.463+0000 7fe36c7c7640 10 monclient: build_initial_monmap 2024-09-16T04:11:42.468 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.463+0000 7fe36c7c7640 10 monclient: monmap: 2024-09-16T04:11:42.468 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:11:42.468 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:11:42.468 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:11:42.466923+0000 2024-09-16T04:11:42.468 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:11:42.466923+0000 2024-09-16T04:11:42.469 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:11:42.469 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:11:42.469 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:11:42.469 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:11:42.469 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:11:42.469 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:11:42.469 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:11:42.469 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:11:42.471 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.463+0000 7fe36c7c7640 1 Processor -- start 2024-09-16T04:11:42.471 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.463+0000 7fe36c7c7640 1 -- start start 2024-09-16T04:11:42.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.463+0000 7fe36c7c7640 10 monclient: init 2024-09-16T04:11:42.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.467+0000 7fe36c7c7640 10 monclient: _reopen_session rank -1 2024-09-16T04:11:42.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.467+0000 7fe36c7c7640 10 monclient: _add_conns ranks=[4,2,1,3,0] 2024-09-16T04:11:42.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.467+0000 7fe36c7c7640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fe3640a1330 0x7fe3640a1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:11:42.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.467+0000 7fe36c7c7640 10 monclient(hunting): picked mon.noname-c con 0x7fe3640a1330 addr v2:172.21.15.154:3300/0 2024-09-16T04:11:42.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.467+0000 7fe36c7c7640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe36406dac0 0x7fe3640a1c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:11:42.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.467+0000 7fe36c7c7640 10 monclient(hunting): picked mon.noname-b con 0x7fe36406dac0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:11:42.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.467+0000 7fe36c7c7640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe364135100 addr v1:172.21.15.154:6789/0 2024-09-16T04:11:42.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.467+0000 7fe36c7c7640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe36409c7d0 con 0x7fe364135100 2024-09-16T04:11:42.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.467+0000 7fe36c7c7640 10 monclient(hunting): start opening mon connection 2024-09-16T04:11:42.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.467+0000 7fe36c7c7640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fe36409c090 con 0x7fe3640a1330 2024-09-16T04:11:42.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.467+0000 7fe36c7c7640 10 monclient(hunting): start opening mon connection 2024-09-16T04:11:42.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.467+0000 7fe36c7c7640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fe36409f610 con 0x7fe36406dac0 2024-09-16T04:11:42.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.467+0000 7fe36c7c7640 10 monclient(hunting): _renew_subs 2024-09-16T04:11:42.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.467+0000 7fe36c7c7640 10 monclient(hunting): authenticate will time out at 2749.436348s 2024-09-16T04:11:42.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.467+0000 7fe369d62640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe36406dac0 0x7fe3640a1c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:11:42.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.467+0000 7fe369d62640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe36406dac0 0x7fe3640a1c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:37626/0 (socket says 172.21.15.42:37626) 2024-09-16T04:11:42.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.467+0000 7fe369d62640 1 -- 172.21.15.42:0/2994229856 learned_addr learned my addr 172.21.15.42:0/2994229856 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:11:42.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.467+0000 7fe369d62640 10 monclient(hunting): get_auth_request con 0x7fe36406dac0 auth_method 0 2024-09-16T04:11:42.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.467+0000 7fe369d62640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:11:42.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.467+0000 7fe369d62640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:42.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.467+0000 7fe369d62640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:42.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.467+0000 7fe369d62640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:11:42.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.467+0000 7fe369d62640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:11:42.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.467+0000 7fe369d62640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:11:42.475 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.467+0000 7fe36a563640 1 --2- 172.21.15.42:0/2994229856 >> v2:172.21.15.154:3300/0 conn(0x7fe3640a1330 0x7fe3640a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:11:42.475 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.467+0000 7fe36a563640 10 monclient(hunting): get_auth_request con 0x7fe3640a1330 auth_method 0 2024-09-16T04:11:42.475 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.467+0000 7fe36a563640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:11:42.475 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.467+0000 7fe36a563640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:42.475 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.467+0000 7fe36a563640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:42.475 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.467+0000 7fe369d62640 10 monclient(hunting): handle_auth_done global_id 1014512 payload 274 2024-09-16T04:11:42.475 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.467+0000 7fe369d62640 1 -- 172.21.15.42:0/2994229856 >> v2:172.21.15.154:3300/0 conn(0x7fe3640a1330 msgr2=0x7fe3640a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:11:42.475 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.467+0000 7fe369d62640 1 --2- 172.21.15.42:0/2994229856 >> v2:172.21.15.154:3300/0 conn(0x7fe3640a1330 0x7fe3640a1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:11:42.475 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.467+0000 7fe369d62640 1 -- 172.21.15.42:0/2994229856 >> v1:172.21.15.154:6789/0 conn(0x7fe364135100 legacy=0x7fe364135550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:11:42.476 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.467+0000 7fe369d62640 10 monclient: _finish_hunting 0 2024-09-16T04:11:42.476 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.467+0000 7fe369d62640 1 monclient: found mon.noname-b 2024-09-16T04:11:42.476 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.467+0000 7fe369d62640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:11:42.476 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.467+0000 7fe369d62640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T04:11:42.476 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.467+0000 7fe369d62640 1 -- 172.21.15.42:0/2994229856 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe36409f8f0 con 0x7fe36406dac0 2024-09-16T04:11:42.485 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.467+0000 7fe369d62640 1 --2- 172.21.15.42:0/2994229856 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe36406dac0 0x7fe3640a1c50 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fe354009e50 tx=0x7fe35401c9d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b8337ee318d3a0c0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:11:42.485 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.467+0000 7fe369561640 1 -- 172.21.15.42:0/2994229856 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3540a5760 con 0x7fe36406dac0 2024-09-16T04:11:42.485 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.467+0000 7fe369561640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:42.485 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.467+0000 7fe369561640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:11:42.485 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.467+0000 7fe369561640 10 monclient: dump: 2024-09-16T04:11:42.485 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:11:42.485 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:11:42.486 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:11:42.486 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:11:42.486 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:42.486 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:11:42.486 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:11:42.486 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:11:42.486 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:11:42.486 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:11:42.486 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.467+0000 7fe369561640 10 monclient: _finish_auth 0 2024-09-16T04:11:42.486 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.467+0000 7fe369561640 10 monclient: _check_auth_tickets 2024-09-16T04:11:42.486 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.467+0000 7fe369561640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:11:42.486 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.467+0000 7fe369561640 1 -- 172.21.15.42:0/2994229856 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe3540a5900 con 0x7fe36406dac0 2024-09-16T04:11:42.486 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.467+0000 7fe369561640 10 monclient: handle_config config(0 keys) 2024-09-16T04:11:42.486 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.467+0000 7fe369561640 1 -- 172.21.15.42:0/2994229856 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3540a5cd0 con 0x7fe36406dac0 2024-09-16T04:11:42.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.467+0000 7fe36c7c7640 5 monclient: authenticate success, global_id 1014512 2024-09-16T04:11:42.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.467+0000 7fe36c7c7640 10 monclient: get_monmap_and_config success 2024-09-16T04:11:42.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.467+0000 7fe36c7c7640 4 set_mon_vals no callback set 2024-09-16T04:11:42.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.467+0000 7fe369561640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:42.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.467+0000 7fe369561640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:11:42.487 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.467+0000 7fe369561640 10 monclient: dump: 2024-09-16T04:11:42.487 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:11:42.487 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:11:42.487 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:11:42.487 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:11:42.487 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:42.487 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:11:42.487 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:11:42.487 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:11:42.488 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:11:42.488 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:11:42.488 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.467+0000 7fe36c7c7640 10 monclient: shutdown 2024-09-16T04:11:42.488 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.467+0000 7fe36c7c7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:11:42.488 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.467+0000 7fe36c7c7640 1 -- 172.21.15.42:0/2994229856 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe36406dac0 msgr2=0x7fe3640a1c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:42.488 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.467+0000 7fe36c7c7640 1 --2- 172.21.15.42:0/2994229856 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe36406dac0 0x7fe3640a1c50 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fe354009e50 tx=0x7fe35401c9d0 comp rx=0 tx=0).stop 2024-09-16T04:11:42.488 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.467+0000 7fe36c7c7640 1 -- 172.21.15.42:0/2994229856 shutdown_connections 2024-09-16T04:11:42.488 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.467+0000 7fe36c7c7640 1 --2- 172.21.15.42:0/2994229856 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe36406dac0 0x7fe3640a1c50 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:11:42.488 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.467+0000 7fe36c7c7640 1 --2- 172.21.15.42:0/2994229856 >> v2:172.21.15.154:3300/0 conn(0x7fe3640a1330 0x7fe3640a1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:11:42.488 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.467+0000 7fe36c7c7640 1 -- 172.21.15.42:0/2994229856 >> 172.21.15.42:0/2994229856 conn(0x7fe3640a2420 msgr2=0x7fe3640a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:11:42.488 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.467+0000 7fe36c7c7640 1 -- 172.21.15.42:0/2994229856 shutdown_connections 2024-09-16T04:11:42.488 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.467+0000 7fe36c7c7640 1 -- 172.21.15.42:0/2994229856 wait complete. 2024-09-16T04:11:42.488 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.471+0000 7fe36c7c7640 10 monclient: build_initial_monmap 2024-09-16T04:11:42.489 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.471+0000 7fe36c7c7640 10 monclient: monmap: 2024-09-16T04:11:42.489 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:11:42.489 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:11:42.489 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:11:42.489 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:11:42.489 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:11:42.489 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:11:42.489 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:11:42.489 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:11:42.489 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:11:42.489 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:11:42.489 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.471+0000 7fe36c7c7640 1 librados: starting msgr at 2024-09-16T04:11:42.489 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.471+0000 7fe36c7c7640 1 librados: starting objecter 2024-09-16T04:11:42.490 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.471+0000 7fe36c7c7640 1 Processor -- start 2024-09-16T04:11:42.490 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.471+0000 7fe36c7c7640 1 -- start start 2024-09-16T04:11:42.490 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.471+0000 7fe36c7c7640 1 librados: setting wanted keys 2024-09-16T04:11:42.490 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.471+0000 7fe36c7c7640 1 librados: calling monclient init 2024-09-16T04:11:42.490 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.471+0000 7fe36c7c7640 10 monclient: init 2024-09-16T04:11:42.490 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.471+0000 7fe36c7c7640 10 monclient: _reopen_session rank -1 2024-09-16T04:11:42.490 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.471+0000 7fe36c7c7640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:11:42.490 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.471+0000 7fe36c7c7640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe3640a1330 0x7fe36413b270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:11:42.490 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.471+0000 7fe36c7c7640 10 monclient(hunting): picked mon.noname-c con 0x7fe3640a1330 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:11:42.490 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.471+0000 7fe36c7c7640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe364135100 0x7fe364134350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:11:42.490 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.471+0000 7fe36c7c7640 10 monclient(hunting): picked mon.noname-b con 0x7fe364135100 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:11:42.490 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.471+0000 7fe36c7c7640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe364134890 0x7fe3641cd090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:11:42.490 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.471+0000 7fe36c7c7640 10 monclient(hunting): picked mon.noname-a con 0x7fe364134890 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:11:42.490 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.471+0000 7fe36c7c7640 10 monclient(hunting): start opening mon connection 2024-09-16T04:11:42.491 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.471+0000 7fe36c7c7640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fe36413fd00 con 0x7fe364134890 2024-09-16T04:11:42.491 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.471+0000 7fe36c7c7640 10 monclient(hunting): start opening mon connection 2024-09-16T04:11:42.491 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.471+0000 7fe36c7c7640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe36413fb80 con 0x7fe364135100 2024-09-16T04:11:42.491 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.471+0000 7fe36c7c7640 10 monclient(hunting): start opening mon connection 2024-09-16T04:11:42.491 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.471+0000 7fe36c7c7640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fe36413fe80 con 0x7fe3640a1330 2024-09-16T04:11:42.491 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.471+0000 7fe36c7c7640 10 monclient(hunting): _renew_subs 2024-09-16T04:11:42.491 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.471+0000 7fe36c7c7640 10 monclient(hunting): authenticate will time out at 2749.441337s 2024-09-16T04:11:42.491 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.471+0000 7fe36a563640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe3640a1330 0x7fe36413b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:11:42.491 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.471+0000 7fe36a563640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe3640a1330 0x7fe36413b270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:37632/0 (socket says 172.21.15.42:37632) 2024-09-16T04:11:42.491 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.471+0000 7fe36a563640 1 -- 172.21.15.42:0/3869877162 learned_addr learned my addr 172.21.15.42:0/3869877162 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:11:42.491 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.471+0000 7fe36a563640 10 monclient(hunting): get_auth_request con 0x7fe3640a1330 auth_method 0 2024-09-16T04:11:42.491 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.471+0000 7fe36a563640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:11:42.491 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.471+0000 7fe36a563640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:42.492 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.471+0000 7fe36a563640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:42.492 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.471+0000 7fe36a563640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:11:42.492 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.471+0000 7fe36a563640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:11:42.492 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.471+0000 7fe36a563640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:11:42.492 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.471+0000 7fe36ad64640 1 --2- 172.21.15.42:0/3869877162 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe364134890 0x7fe3641cd090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:11:42.492 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.471+0000 7fe369d62640 1 --2- 172.21.15.42:0/3869877162 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe364135100 0x7fe364134350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:11:42.492 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.471+0000 7fe369d62640 10 monclient(hunting): get_auth_request con 0x7fe364135100 auth_method 0 2024-09-16T04:11:42.492 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.471+0000 7fe369d62640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:11:42.492 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.471+0000 7fe369d62640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:42.492 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.471+0000 7fe369d62640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:42.492 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.471+0000 7fe36ad64640 10 monclient(hunting): get_auth_request con 0x7fe364134890 auth_method 0 2024-09-16T04:11:42.492 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.471+0000 7fe36ad64640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:11:42.493 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.471+0000 7fe36ad64640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:42.493 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.471+0000 7fe36ad64640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:42.493 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.471+0000 7fe36a563640 10 monclient(hunting): handle_auth_done global_id 1014515 payload 824 2024-09-16T04:11:42.493 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.471+0000 7fe36a563640 1 -- 172.21.15.42:0/3869877162 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe364135100 msgr2=0x7fe364134350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:42.493 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.471+0000 7fe36a563640 1 --2- 172.21.15.42:0/3869877162 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe364135100 0x7fe364134350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:11:42.493 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.471+0000 7fe36a563640 1 -- 172.21.15.42:0/3869877162 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe364134890 msgr2=0x7fe3641cd090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:42.493 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.471+0000 7fe36a563640 1 --2- 172.21.15.42:0/3869877162 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe364134890 0x7fe3641cd090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:11:42.493 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.471+0000 7fe36a563640 10 monclient: _finish_hunting 0 2024-09-16T04:11:42.493 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.471+0000 7fe36a563640 1 monclient: found mon.noname-c 2024-09-16T04:11:42.493 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.471+0000 7fe36a563640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:11:42.493 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.471+0000 7fe36a563640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T04:11:42.493 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.471+0000 7fe36a563640 1 -- 172.21.15.42:0/3869877162 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe36411ece0 con 0x7fe3640a1330 2024-09-16T04:11:42.493 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.471+0000 7fe36a563640 1 --2- 172.21.15.42:0/3869877162 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe3640a1330 0x7fe36413b270 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fe35c00d830 tx=0x7fe35c00dcf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:11:42.493 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.471+0000 7fe35affd640 10 client.?.objecter ms_handle_connect 0x7fe3640a1330 2024-09-16T04:11:42.494 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.471+0000 7fe35affd640 10 client.?.objecter resend_mon_ops 2024-09-16T04:11:42.494 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.471+0000 7fe35affd640 1 -- 172.21.15.42:0/3869877162 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe35c00b6c0 con 0x7fe3640a1330 2024-09-16T04:11:42.494 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.471+0000 7fe35affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:42.494 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.471+0000 7fe35affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:11:42.494 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.471+0000 7fe35affd640 10 monclient: dump: 2024-09-16T04:11:42.494 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:11:42.494 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:11:42.494 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:11:42.494 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:11:42.494 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:42.494 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:11:42.494 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:11:42.494 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:11:42.494 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:11:42.494 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:11:42.495 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.471+0000 7fe35affd640 10 monclient: _finish_auth 0 2024-09-16T04:11:42.495 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.471+0000 7fe35affd640 10 monclient: _check_auth_tickets 2024-09-16T04:11:42.495 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.471+0000 7fe35affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:11:42.495 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.471+0000 7fe35affd640 1 -- 172.21.15.42:0/3869877162 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe35c019070 con 0x7fe3640a1330 2024-09-16T04:11:42.495 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.471+0000 7fe36c7c7640 5 monclient: authenticate success, global_id 1014515 2024-09-16T04:11:42.495 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.471+0000 7fe36c7c7640 10 monclient: _renew_subs 2024-09-16T04:11:42.495 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.471+0000 7fe36c7c7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:11:42.495 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.471+0000 7fe36c7c7640 1 -- 172.21.15.42:0/3869877162 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3640a50e0 con 0x7fe3640a1330 2024-09-16T04:11:42.495 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.471+0000 7fe35affd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:11:42.495 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.471+0000 7fe35affd640 1 -- 172.21.15.42:0/3869877162 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe35c00ba60 con 0x7fe3640a1330 2024-09-16T04:11:42.495 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.471+0000 7fe35affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:42.495 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.471+0000 7fe35affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:11:42.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.471+0000 7fe35affd640 10 monclient: dump: 2024-09-16T04:11:42.496 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:11:42.496 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:11:42.496 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:11:42.496 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:11:42.496 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:42.496 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:11:42.496 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:11:42.496 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:11:42.496 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:11:42.496 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:11:42.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.471+0000 7fe35bfff640 4 set_mon_vals no callback set 2024-09-16T04:11:42.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.471+0000 7fe36c7c7640 10 client.1014515.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:11:42.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.471+0000 7fe36c7c7640 10 monclient: _renew_subs 2024-09-16T04:11:42.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.471+0000 7fe36c7c7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:11:42.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.471+0000 7fe36c7c7640 1 -- 172.21.15.42:0/3869877162 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fe364071b50 con 0x7fe3640a1330 2024-09-16T04:11:42.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.471+0000 7fe36c7c7640 1 librados: init done 2024-09-16T04:11:42.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.471+0000 7fe36c7c7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:42.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.471+0000 7fe36c7c7640 1 --2- 172.21.15.42:0/3869877162 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe36406dac0 0x7fe3641cd5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:11:42.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.471+0000 7fe36c7c7640 10 monclient(hunting): start opening mon connection 2024-09-16T04:11:42.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.471+0000 7fe36c7c7640 1 -- 172.21.15.42:0/3869877162 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fe36411e5b0 con 0x7fe36406dac0 2024-09-16T04:11:42.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.471+0000 7fe369d62640 1 --2- 172.21.15.42:0/3869877162 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe36406dac0 0x7fe3641cd5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:11:42.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.471+0000 7fe35affd640 1 -- 172.21.15.42:0/3869877162 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fe35c01e460 con 0x7fe3640a1330 2024-09-16T04:11:42.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.471+0000 7fe369d62640 10 monclient: get_auth_request con 0x7fe36406dac0 auth_method 0 2024-09-16T04:11:42.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.471+0000 7fe369d62640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:11:42.498 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.471+0000 7fe369d62640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:42.498 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.471+0000 7fe369d62640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:42.498 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.471+0000 7fe369d62640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:11:42.498 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.471+0000 7fe369d62640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:11:42.498 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.471+0000 7fe369d62640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:11:42.498 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.471+0000 7fe35affd640 1 --2- 172.21.15.42:0/3869877162 >> v2:172.21.15.154:6812/254845342 conn(0x7fe350037810 0x7fe350039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:11:42.498 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.471+0000 7fe369d62640 10 monclient(hunting): handle_auth_done global_id 1014518 payload 824 2024-09-16T04:11:42.498 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.471+0000 7fe35affd640 1 -- 172.21.15.42:0/3869877162 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(706..706 src has 451..706) ==== 5040+0+0 (secure 0 0 0) 0x7fe35c049340 con 0x7fe3640a1330 2024-09-16T04:11:42.498 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.471+0000 7fe35affd640 10 client.1014515.objecter ms_dispatch 0x7fe364000ba0 osd_map(706..706 src has 451..706) 2024-09-16T04:11:42.498 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.471+0000 7fe35affd640 3 client.1014515.objecter handle_osd_map got epochs [706,706] > 0 2024-09-16T04:11:42.498 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.471+0000 7fe35affd640 3 client.1014515.objecter handle_osd_map decoding full epoch 706 2024-09-16T04:11:42.498 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.471+0000 7fe369d62640 1 -- 172.21.15.42:0/3869877162 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe364135550 con 0x7fe36406dac0 2024-09-16T04:11:42.498 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.471+0000 7fe35affd640 20 client.1014515.objecter dump_active .. 0 homeless 2024-09-16T04:11:42.498 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.471+0000 7fe35affd640 10 client.1014515.objecter ms_handle_connect 0x7fe36406dac0 2024-09-16T04:11:42.499 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.471+0000 7fe35affd640 10 client.1014515.objecter resend_mon_ops 2024-09-16T04:11:42.499 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.471+0000 7fe369d62640 1 --2- 172.21.15.42:0/3869877162 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe36406dac0 0x7fe3641cd5d0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fe36409bb00 tx=0x7fe3640a22d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:11:42.499 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.471+0000 7fe36ad64640 1 --2- 172.21.15.42:0/3869877162 >> v2:172.21.15.154:6812/254845342 conn(0x7fe350037810 0x7fe350039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:11:42.499 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.471+0000 7fe36ad64640 10 monclient: get_auth_request con 0x7fe350037810 auth_method 0 2024-09-16T04:11:42.499 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.471+0000 7fe35affd640 1 -- 172.21.15.42:0/3869877162 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3540bd070 con 0x7fe36406dac0 2024-09-16T04:11:42.499 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.471+0000 7fe35affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:42.499 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.471+0000 7fe35affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:11:42.499 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.471+0000 7fe35affd640 10 monclient: dump: 2024-09-16T04:11:42.499 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:11:42.499 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:11:42.499 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:11:42.499 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:11:42.499 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:42.499 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:11:42.500 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:11:42.500 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:11:42.500 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:11:42.500 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:11:42.500 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.475+0000 7fe35affd640 1 -- 172.21.15.42:0/3869877162 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fe3540b6620 con 0x7fe36406dac0 2024-09-16T04:11:42.500 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.475+0000 7fe35affd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:42.500 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.475+0000 7fe35affd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:11:42.500 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.475+0000 7fe35affd640 1 -- 172.21.15.42:0/3869877162 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe36406dac0 msgr2=0x7fe3641cd5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:42.500 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.475+0000 7fe35affd640 1 --2- 172.21.15.42:0/3869877162 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe36406dac0 0x7fe3641cd5d0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fe36409bb00 tx=0x7fe3640a22d0 comp rx=0 tx=0).stop 2024-09-16T04:11:42.500 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.475+0000 7fe35affd640 1 -- 172.21.15.42:0/3869877162 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe36406dac0 msgr2=0x7fe3641cd5d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:11:42.500 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.475+0000 7fe35affd640 1 --2- 172.21.15.42:0/3869877162 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe36406dac0 0x7fe3641cd5d0 secure :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fe36409bb00 tx=0x7fe3640a22d0 comp rx=0 tx=0).stop 2024-09-16T04:11:42.500 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.475+0000 7fe35affd640 10 client.1014515.objecter ms_handle_connect 0x7fe350037810 2024-09-16T04:11:42.500 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.475+0000 7fe36ad64640 1 --2- 172.21.15.42:0/3869877162 >> v2:172.21.15.154:6812/254845342 conn(0x7fe350037810 0x7fe350039cd0 secure :-1 s=READY pgs=3948 cs=0 l=1 rev1=1 crypto rx=0x7fe360004db0 tx=0x7fe360009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:11:42.501 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.479+0000 7fe36c7c7640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:11:42.501 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.479+0000 7fe36c7c7640 1 --2- 172.21.15.42:0/3869877162 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe3641cdb10 0x7fe364138330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:11:42.501 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.479+0000 7fe36c7c7640 10 monclient(hunting): start opening mon connection 2024-09-16T04:11:42.501 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.479+0000 7fe36c7c7640 1 -- 172.21.15.42:0/3869877162 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fe3641cdf20 con 0x7fe3641cdb10 2024-09-16T04:11:42.501 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.479+0000 7fe369d62640 1 --2- 172.21.15.42:0/3869877162 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe3641cdb10 0x7fe364138330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:11:42.501 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.479+0000 7fe369d62640 10 monclient: get_auth_request con 0x7fe3641cdb10 auth_method 0 2024-09-16T04:11:42.501 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.479+0000 7fe369d62640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:11:42.501 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.479+0000 7fe369d62640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:42.501 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.479+0000 7fe369d62640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:42.501 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.479+0000 7fe369d62640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:11:42.501 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.479+0000 7fe369d62640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:11:42.501 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.479+0000 7fe369d62640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:11:42.501 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.479+0000 7fe369d62640 10 monclient(hunting): handle_auth_done global_id 1014521 payload 824 2024-09-16T04:11:42.502 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.479+0000 7fe369d62640 1 -- 172.21.15.42:0/3869877162 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fe364069040 con 0x7fe3641cdb10 2024-09-16T04:11:42.502 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.479+0000 7fe35affd640 10 client.1014515.objecter ms_handle_connect 0x7fe3641cdb10 2024-09-16T04:11:42.502 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.479+0000 7fe35affd640 10 client.1014515.objecter resend_mon_ops 2024-09-16T04:11:42.502 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.479+0000 7fe369d62640 1 --2- 172.21.15.42:0/3869877162 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe3641cdb10 0x7fe364138330 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fe354000f90 tx=0x7fe3540a7070 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:11:42.502 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.479+0000 7fe35affd640 1 -- 172.21.15.42:0/3869877162 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3540c3a50 con 0x7fe3641cdb10 2024-09-16T04:11:42.502 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.479+0000 7fe35affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:42.502 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.479+0000 7fe35affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:11:42.502 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.479+0000 7fe35affd640 10 monclient: dump: 2024-09-16T04:11:42.502 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:11:42.502 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:11:42.502 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:11:42.502 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:11:42.502 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:42.502 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:11:42.503 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:11:42.503 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:11:42.503 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:11:42.503 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:11:42.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.479+0000 7fe35affd640 1 -- 172.21.15.42:0/3869877162 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4713 (secure 0 0 0) 0x7fe3540c7070 con 0x7fe3641cdb10 2024-09-16T04:11:42.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.479+0000 7fe35affd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:11:42.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.479+0000 7fe35affd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:11:42.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.479+0000 7fe35affd640 1 -- 172.21.15.42:0/3869877162 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe3641cdb10 msgr2=0x7fe364138330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:42.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.479+0000 7fe35affd640 1 --2- 172.21.15.42:0/3869877162 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe3641cdb10 0x7fe364138330 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fe354000f90 tx=0x7fe3540a7070 comp rx=0 tx=0).stop 2024-09-16T04:11:42.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.479+0000 7fe35affd640 1 -- 172.21.15.42:0/3869877162 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe3641cdb10 msgr2=0x7fe364138330 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:11:42.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.479+0000 7fe35affd640 1 --2- 172.21.15.42:0/3869877162 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe3641cdb10 0x7fe364138330 secure :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fe354000f90 tx=0x7fe3540a7070 comp rx=0 tx=0).stop 2024-09-16T04:11:42.504 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T04:11:42.504 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:11:42.504 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:11:42.504 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T04:11:42.504 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 268, 2024-09-16T04:11:42.504 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 297681, 2024-09-16T04:11:42.504 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T04:11:42.504 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T04:11:42.504 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T04:11:42.504 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T04:11:42.504 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:11:42.505 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 0, 2024-09-16T04:11:42.505 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:11:42.505 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T04:11:42.505 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T04:11:42.505 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:11:42.505 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:11:42.505 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:11:42.505 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:11:42.505 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:11:42.505 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:11:42.505 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:11:42.505 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:11:42.505 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:11:42.506 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:11:42.506 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:11:42.506 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:11:42.506 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:11:42.506 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T04:11:42.506 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:11:42.506 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:11:42.506 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:11:42.506 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:11:42.506 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:11:42.506 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:11:42.506 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:11:42.506 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:11:42.507 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:11:42.507 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:11:42.507 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:11:42.507 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:11:42.507 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:11:42.507 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T04:11:42.507 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T04:11:42.507 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T04:11:42.507 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T04:11:42.507 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T04:11:42.507 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T04:11:42.507 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:11:42.507 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:11:42.507 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T04:11:42.508 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T04:11:42.508 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T04:11:42.508 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T04:11:42.508 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T04:11:42.508 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T04:11:42.508 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T04:11:42.508 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:11:42.508 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T04:11:42.508 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:11:42.508 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:11:42.508 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:11:42.508 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:11:42.508 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:11:42.509 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:11:42.509 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:11:42.509 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:11:42.509 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:11:42.509 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:11:42.509 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:11:42.509 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:11:42.509 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T04:11:42.509 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:11:42.509 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T04:11:42.509 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:11:42.509 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:11:42.509 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:11:42.509 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:11:42.510 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:11:42.510 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:11:42.510 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:11:42.510 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T04:11:42.510 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:11:42.510 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:11:42.510 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:11:42.510 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:11:42.510 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T04:11:42.510 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:11:42.510 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:11:42.510 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:11:42.510 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:11:42.510 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T04:11:42.510 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T04:11:42.511 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:11:42.511 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:11:42.511 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:11:42.511 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:11:42.511 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:11:42.511 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:11:42.511 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:11:42.511 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:11:42.511 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:11:42.511 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:11:42.511 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:11:42.511 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T04:11:42.511 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:11:42.511 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:11:42.512 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:11:42.512 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:11:42.512 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T04:11:42.512 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:11:42.512 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:11:42.512 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:11:42.512 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:11:42.512 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T04:11:42.512 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T04:11:42.512 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:11:42.512 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:11:42.513 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:11:42.513 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:11:42.513 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:11:42.513 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:11:42.513 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:11:42.513 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:11:42.513 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:11:42.513 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:11:42.513 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:11:42.513 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T04:11:42.513 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:11:42.513 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:11:42.513 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:11:42.513 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:11:42.514 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T04:11:42.514 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:11:42.514 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:11:42.514 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:11:42.514 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:11:42.514 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T04:11:42.514 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T04:11:42.514 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:11:42.514 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:11:42.514 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:11:42.514 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:11:42.514 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:11:42.514 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:11:42.514 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T04:11:42.515 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T04:11:42.515 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:11:42.515 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:11:42.515 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:11:42.515 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:11:42.515 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:11:42.515 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:11:42.515 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T04:11:42.515 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:11:42.515 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:11:42.515 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:11:42.515 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 4 2024-09-16T04:11:42.516 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:11:42.516 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:11:42.516 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T04:11:42.516 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:11:42.516 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:11:42.516 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:11:42.516 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 4 2024-09-16T04:11:42.516 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:11:42.516 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:11:42.516 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:11:42.516 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T04:11:42.516 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T04:11:42.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.479+0000 7fe36c7c7640 10 librados: watch_flush enter 2024-09-16T04:11:42.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.479+0000 7fe36c7c7640 10 librados: watch_flush exit 2024-09-16T04:11:42.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.479+0000 7fe36c7c7640 20 client.1014515.objecter shutdown clearing up homeless session... 2024-09-16T04:11:42.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.479+0000 7fe36c7c7640 10 client.1014515.objecter successfully canceled tick 2024-09-16T04:11:42.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.479+0000 7fe36c7c7640 1 -- 172.21.15.42:0/3869877162 >> v2:172.21.15.154:6812/254845342 conn(0x7fe350037810 msgr2=0x7fe350039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:42.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.479+0000 7fe36c7c7640 1 --2- 172.21.15.42:0/3869877162 >> v2:172.21.15.154:6812/254845342 conn(0x7fe350037810 0x7fe350039cd0 secure :-1 s=READY pgs=3948 cs=0 l=1 rev1=1 crypto rx=0x7fe360004db0 tx=0x7fe360009040 comp rx=0 tx=0).stop 2024-09-16T04:11:42.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.479+0000 7fe36c7c7640 10 monclient: shutdown 2024-09-16T04:11:42.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.479+0000 7fe36c7c7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:11:42.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.479+0000 7fe36c7c7640 1 -- 172.21.15.42:0/3869877162 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe3640a1330 msgr2=0x7fe36413b270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:42.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.479+0000 7fe36c7c7640 1 --2- 172.21.15.42:0/3869877162 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe3640a1330 0x7fe36413b270 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fe35c00d830 tx=0x7fe35c00dcf0 comp rx=0 tx=0).stop 2024-09-16T04:11:42.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.479+0000 7fe36ad64640 1 -- 172.21.15.42:0/3869877162 reap_dead start 2024-09-16T04:11:42.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.479+0000 7fe36c7c7640 1 -- 172.21.15.42:0/3869877162 shutdown_connections 2024-09-16T04:11:42.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.479+0000 7fe36c7c7640 1 --2- 172.21.15.42:0/3869877162 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe3640a1330 0x7fe36413b270 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:11:42.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.479+0000 7fe36c7c7640 1 -- 172.21.15.42:0/3869877162 >> 172.21.15.42:0/3869877162 conn(0x7fe3640a2420 msgr2=0x7fe36413c590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:11:42.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.479+0000 7fe36c7c7640 1 -- 172.21.15.42:0/3869877162 shutdown_connections 2024-09-16T04:11:42.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.483+0000 7fe36c7c7640 1 -- 172.21.15.42:0/3869877162 wait complete. 2024-09-16T04:11:42.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.483+0000 7fe36c7c7640 1 librados: shutdown 2024-09-16T04:11:42.520 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:11:42.520 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T04:11:42.520 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T04:11:42.636 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.631+0000 7fd3b53a8640 10 monclient: get_monmap_and_config 2024-09-16T04:11:42.636 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.631+0000 7fd3b53a8640 10 monclient: build_initial_monmap 2024-09-16T04:11:42.636 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.631+0000 7fd3b53a8640 10 monclient: monmap: 2024-09-16T04:11:42.636 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:11:42.636 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:11:42.636 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:11:42.635167+0000 2024-09-16T04:11:42.637 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:11:42.635167+0000 2024-09-16T04:11:42.637 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:11:42.637 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:11:42.637 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:11:42.637 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:11:42.637 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:11:42.637 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:11:42.637 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:11:42.637 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:11:42.639 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.635+0000 7fd3b53a8640 1 Processor -- start 2024-09-16T04:11:42.640 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.635+0000 7fd3b53a8640 1 -- start start 2024-09-16T04:11:42.640 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.635+0000 7fd3b53a8640 10 monclient: init 2024-09-16T04:11:42.640 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.635+0000 7fd3b53a8640 10 monclient: _reopen_session rank -1 2024-09-16T04:11:42.640 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.635+0000 7fd3b53a8640 10 monclient: _add_conns ranks=[3,1,0,4,2] 2024-09-16T04:11:42.640 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.635+0000 7fd3b53a8640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd3b00a39b0 0x7fd3b0133b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:11:42.640 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.635+0000 7fd3b53a8640 10 monclient(hunting): picked mon.noname-a con 0x7fd3b00a39b0 addr v2:172.21.15.42:3300/0 2024-09-16T04:11:42.640 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.635+0000 7fd3b53a8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd3b00a3d90 addr v1:172.21.15.154:6789/0 2024-09-16T04:11:42.640 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.635+0000 7fd3b53a8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd3b0137710 addr v1:172.21.15.42:6789/0 2024-09-16T04:11:42.640 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.635+0000 7fd3b53a8640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd3b009c140 con 0x7fd3b0137710 2024-09-16T04:11:42.640 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.635+0000 7fd3b53a8640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd3b009c420 con 0x7fd3b00a3d90 2024-09-16T04:11:42.640 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.635+0000 7fd3b53a8640 10 monclient(hunting): start opening mon connection 2024-09-16T04:11:42.641 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.635+0000 7fd3b53a8640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7fd3b006dd60 con 0x7fd3b00a39b0 2024-09-16T04:11:42.641 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.635+0000 7fd3b53a8640 10 monclient(hunting): _renew_subs 2024-09-16T04:11:42.641 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.635+0000 7fd3b53a8640 10 monclient(hunting): authenticate will time out at 2749.604919s 2024-09-16T04:11:42.641 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.635+0000 7fd3aeffd640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd3b00a39b0 0x7fd3b0133b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:11:42.641 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.635+0000 7fd3aeffd640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd3b00a39b0 0x7fd3b0133b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:57628/0 (socket says 172.21.15.42:57628) 2024-09-16T04:11:42.641 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.635+0000 7fd3aeffd640 1 -- 172.21.15.42:0/2643181161 learned_addr learned my addr 172.21.15.42:0/2643181161 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:11:42.641 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.635+0000 7fd3aeffd640 10 monclient(hunting): get_auth_request con 0x7fd3b00a39b0 auth_method 0 2024-09-16T04:11:42.641 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.635+0000 7fd3aeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:11:42.641 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.635+0000 7fd3aeffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:42.641 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.635+0000 7fd3aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:42.641 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.635+0000 7fd3aeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:11:42.641 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.635+0000 7fd3aeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:11:42.642 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.635+0000 7fd3aeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:11:42.642 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.635+0000 7fd3aeffd640 10 monclient(hunting): handle_auth_done global_id 1034421 payload 274 2024-09-16T04:11:42.642 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.635+0000 7fd3aeffd640 1 -- 172.21.15.42:0/2643181161 >> v1:172.21.15.154:6789/0 conn(0x7fd3b00a3d90 legacy=0x7fd3b0137000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:11:42.642 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.635+0000 7fd3aeffd640 1 -- 172.21.15.42:0/2643181161 >> v1:172.21.15.42:6789/0 conn(0x7fd3b0137710 legacy=0x7fd3b013f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:42.642 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.635+0000 7fd3aeffd640 10 monclient: _finish_hunting 0 2024-09-16T04:11:42.642 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.635+0000 7fd3aeffd640 1 monclient: found mon.noname-a 2024-09-16T04:11:42.642 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.635+0000 7fd3aeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:11:42.642 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.635+0000 7fd3aeffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:11:42.642 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.635+0000 7fd3aeffd640 1 -- 172.21.15.42:0/2643181161 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3b00a5e20 con 0x7fd3b00a39b0 2024-09-16T04:11:42.648 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.635+0000 7fd3aeffd640 1 --2- 172.21.15.42:0/2643181161 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd3b00a39b0 0x7fd3b0133b80 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fd3a0009870 tx=0x7fd3a001c420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c22d0c9b2e87960 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:11:42.648 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.635+0000 7fd3adffb640 1 -- 172.21.15.42:0/2643181161 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3a00b0070 con 0x7fd3b00a39b0 2024-09-16T04:11:42.649 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.635+0000 7fd3adffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:42.649 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.635+0000 7fd3adffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:11:42.649 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.635+0000 7fd3adffb640 10 monclient: dump: 2024-09-16T04:11:42.649 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:11:42.649 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:11:42.649 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:11:42.649 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:11:42.649 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:42.649 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:11:42.649 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:11:42.649 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:11:42.649 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:11:42.649 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:11:42.650 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.635+0000 7fd3adffb640 10 monclient: _finish_auth 0 2024-09-16T04:11:42.650 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.635+0000 7fd3adffb640 10 monclient: _check_auth_tickets 2024-09-16T04:11:42.650 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.635+0000 7fd3adffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:11:42.650 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.635+0000 7fd3b53a8640 5 monclient: authenticate success, global_id 1034421 2024-09-16T04:11:42.650 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.635+0000 7fd3adffb640 1 -- 172.21.15.42:0/2643181161 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd3a00ab600 con 0x7fd3b00a39b0 2024-09-16T04:11:42.650 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.635+0000 7fd3b53a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:11:42.650 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.635+0000 7fd3adffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:11:42.650 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.635+0000 7fd3b53a8640 10 monclient: get_monmap_and_config success 2024-09-16T04:11:42.650 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.635+0000 7fd3b53a8640 4 set_mon_vals no callback set 2024-09-16T04:11:42.650 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.635+0000 7fd3adffb640 1 -- 172.21.15.42:0/2643181161 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3a00ab9d0 con 0x7fd3b00a39b0 2024-09-16T04:11:42.650 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.635+0000 7fd3adffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:42.650 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.635+0000 7fd3adffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:11:42.650 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.635+0000 7fd3adffb640 10 monclient: dump: 2024-09-16T04:11:42.650 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:11:42.651 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:11:42.651 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:11:42.651 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:11:42.651 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:42.651 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:11:42.651 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:11:42.651 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:11:42.651 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:11:42.651 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:11:42.651 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.639+0000 7fd3b53a8640 10 monclient: shutdown 2024-09-16T04:11:42.651 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.639+0000 7fd3b53a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:11:42.651 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.639+0000 7fd3b53a8640 1 -- 172.21.15.42:0/2643181161 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd3b00a39b0 msgr2=0x7fd3b0133b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:42.651 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.639+0000 7fd3b53a8640 1 --2- 172.21.15.42:0/2643181161 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd3b00a39b0 0x7fd3b0133b80 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fd3a0009870 tx=0x7fd3a001c420 comp rx=0 tx=0).stop 2024-09-16T04:11:42.652 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.639+0000 7fd3b53a8640 1 -- 172.21.15.42:0/2643181161 shutdown_connections 2024-09-16T04:11:42.652 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.639+0000 7fd3b53a8640 1 --2- 172.21.15.42:0/2643181161 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd3b00a39b0 0x7fd3b0133b80 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:11:42.652 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.639+0000 7fd3b53a8640 1 -- 172.21.15.42:0/2643181161 >> 172.21.15.42:0/2643181161 conn(0x7fd3b00a9fb0 msgr2=0x7fd3b00aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:11:42.652 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.639+0000 7fd3b53a8640 1 -- 172.21.15.42:0/2643181161 shutdown_connections 2024-09-16T04:11:42.652 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.639+0000 7fd3b53a8640 1 -- 172.21.15.42:0/2643181161 wait complete. 2024-09-16T04:11:42.652 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.639+0000 7fd3b53a8640 10 monclient: build_initial_monmap 2024-09-16T04:11:42.652 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.639+0000 7fd3b53a8640 10 monclient: monmap: 2024-09-16T04:11:42.652 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:11:42.652 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:11:42.652 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:11:42.652 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:11:42.652 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:11:42.652 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:11:42.652 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:11:42.653 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:11:42.653 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:11:42.653 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:11:42.653 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.639+0000 7fd3b53a8640 1 librados: starting msgr at 2024-09-16T04:11:42.653 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.639+0000 7fd3b53a8640 1 librados: starting objecter 2024-09-16T04:11:42.653 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.639+0000 7fd3b53a8640 1 Processor -- start 2024-09-16T04:11:42.653 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.639+0000 7fd3b53a8640 1 -- start start 2024-09-16T04:11:42.653 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.639+0000 7fd3b53a8640 1 librados: setting wanted keys 2024-09-16T04:11:42.653 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.639+0000 7fd3b53a8640 1 librados: calling monclient init 2024-09-16T04:11:42.653 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.639+0000 7fd3b53a8640 10 monclient: init 2024-09-16T04:11:42.653 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.639+0000 7fd3b53a8640 10 monclient: _reopen_session rank -1 2024-09-16T04:11:42.653 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.639+0000 7fd3b53a8640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:11:42.653 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.639+0000 7fd3b53a8640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd3b00a39b0 0x7fd3b0130660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:11:42.653 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.639+0000 7fd3b53a8640 10 monclient(hunting): picked mon.noname-c con 0x7fd3b00a39b0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:11:42.654 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.639+0000 7fd3b53a8640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd3b00a3d90 0x7fd3b0130ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:11:42.654 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.639+0000 7fd3b53a8640 10 monclient(hunting): picked mon.noname-a con 0x7fd3b00a3d90 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:11:42.654 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.639+0000 7fd3b53a8640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd3b0137710 0x7fd3b00a19a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:11:42.654 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.639+0000 7fd3b53a8640 10 monclient(hunting): picked mon.noname-b con 0x7fd3b0137710 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:11:42.654 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.639+0000 7fd3b53a8640 10 monclient(hunting): start opening mon connection 2024-09-16T04:11:42.654 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.639+0000 7fd3b53a8640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fd3b0143900 con 0x7fd3b00a3d90 2024-09-16T04:11:42.654 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.639+0000 7fd3b53a8640 10 monclient(hunting): start opening mon connection 2024-09-16T04:11:42.654 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.639+0000 7fd3b53a8640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd3b006dee0 con 0x7fd3b0137710 2024-09-16T04:11:42.654 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.639+0000 7fd3b53a8640 10 monclient(hunting): start opening mon connection 2024-09-16T04:11:42.654 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.639+0000 7fd3b53a8640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fd3b0143a80 con 0x7fd3b00a39b0 2024-09-16T04:11:42.654 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.639+0000 7fd3b53a8640 10 monclient(hunting): _renew_subs 2024-09-16T04:11:42.654 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.639+0000 7fd3b53a8640 10 monclient(hunting): authenticate will time out at 2749.609951s 2024-09-16T04:11:42.654 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.639+0000 7fd3aeffd640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd3b00a39b0 0x7fd3b0130660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:11:42.654 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.639+0000 7fd3ae7fc640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd3b00a3d90 0x7fd3b0130ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:11:42.655 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.639+0000 7fd3aeffd640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd3b00a39b0 0x7fd3b0130660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:37660/0 (socket says 172.21.15.42:37660) 2024-09-16T04:11:42.655 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.639+0000 7fd3aeffd640 1 -- 172.21.15.42:0/2717469761 learned_addr learned my addr 172.21.15.42:0/2717469761 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:11:42.655 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.639+0000 7fd3aeffd640 10 monclient(hunting): get_auth_request con 0x7fd3b00a39b0 auth_method 0 2024-09-16T04:11:42.655 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.639+0000 7fd3aeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:11:42.655 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.639+0000 7fd3aeffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:42.655 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.639+0000 7fd3aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:42.655 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.639+0000 7fd3ae7fc640 10 monclient(hunting): get_auth_request con 0x7fd3b00a3d90 auth_method 0 2024-09-16T04:11:42.655 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.639+0000 7fd3ae7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:11:42.655 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.639+0000 7fd3ae7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:42.655 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.639+0000 7fd3ae7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:42.655 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.639+0000 7fd3aeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:11:42.655 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.639+0000 7fd3aeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:11:42.655 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.639+0000 7fd3aeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:11:42.655 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.639+0000 7fd3ae7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:11:42.656 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.639+0000 7fd3ae7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:11:42.656 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.639+0000 7fd3ae7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:11:42.656 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.639+0000 7fd3af7fe640 1 --2- 172.21.15.42:0/2717469761 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd3b0137710 0x7fd3b00a19a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:11:42.656 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.639+0000 7fd3af7fe640 10 monclient(hunting): get_auth_request con 0x7fd3b0137710 auth_method 0 2024-09-16T04:11:42.656 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.639+0000 7fd3af7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:11:42.656 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.639+0000 7fd3af7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:42.656 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.639+0000 7fd3af7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:42.656 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.639+0000 7fd3aeffd640 10 monclient(hunting): handle_auth_done global_id 1014524 payload 824 2024-09-16T04:11:42.656 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.639+0000 7fd3aeffd640 1 -- 172.21.15.42:0/2717469761 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd3b0137710 msgr2=0x7fd3b00a19a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:42.656 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.639+0000 7fd3aeffd640 1 --2- 172.21.15.42:0/2717469761 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd3b0137710 0x7fd3b00a19a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:11:42.656 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.639+0000 7fd3aeffd640 1 -- 172.21.15.42:0/2717469761 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd3b00a3d90 msgr2=0x7fd3b0130ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:42.656 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.639+0000 7fd3aeffd640 1 --2- 172.21.15.42:0/2717469761 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd3b00a3d90 0x7fd3b0130ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:11:42.656 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.639+0000 7fd3aeffd640 10 monclient: _finish_hunting 0 2024-09-16T04:11:42.657 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.639+0000 7fd3aeffd640 1 monclient: found mon.noname-c 2024-09-16T04:11:42.657 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.639+0000 7fd3aeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:11:42.657 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.639+0000 7fd3aeffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T04:11:42.657 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.639+0000 7fd3aeffd640 1 -- 172.21.15.42:0/2717469761 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3b0139b10 con 0x7fd3b00a39b0 2024-09-16T04:11:42.657 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.639+0000 7fd3ae7fc640 1 --2- 172.21.15.42:0/2717469761 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd3b00a3d90 0x7fd3b0130ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:11:42.657 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.639+0000 7fd3aeffd640 1 --2- 172.21.15.42:0/2717469761 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd3b00a39b0 0x7fd3b0130660 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fd3a001a050 tx=0x7fd3a00191c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:11:42.657 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.639+0000 7fd3977fe640 10 client.?.objecter ms_handle_connect 0x7fd3b00a39b0 2024-09-16T04:11:42.657 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.639+0000 7fd3977fe640 10 client.?.objecter resend_mon_ops 2024-09-16T04:11:42.657 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.639+0000 7fd3977fe640 1 -- 172.21.15.42:0/2717469761 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3a00b0040 con 0x7fd3b00a39b0 2024-09-16T04:11:42.657 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.639+0000 7fd3977fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:42.657 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.639+0000 7fd3977fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:11:42.657 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.639+0000 7fd3977fe640 10 monclient: dump: 2024-09-16T04:11:42.657 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:11:42.657 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:11:42.658 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:11:42.658 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:11:42.658 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:42.658 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:11:42.658 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:11:42.658 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:11:42.658 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:11:42.658 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:11:42.658 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.639+0000 7fd3977fe640 10 monclient: _finish_auth 0 2024-09-16T04:11:42.658 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.639+0000 7fd3977fe640 10 monclient: _check_auth_tickets 2024-09-16T04:11:42.658 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.639+0000 7fd3977fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:11:42.658 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.639+0000 7fd3977fe640 1 -- 172.21.15.42:0/2717469761 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd3a00b9070 con 0x7fd3b00a39b0 2024-09-16T04:11:42.658 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.639+0000 7fd3b53a8640 5 monclient: authenticate success, global_id 1014524 2024-09-16T04:11:42.658 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.639+0000 7fd3b53a8640 10 monclient: _renew_subs 2024-09-16T04:11:42.659 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.639+0000 7fd3b53a8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:11:42.659 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.639+0000 7fd3b53a8640 1 -- 172.21.15.42:0/2717469761 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3b0139df0 con 0x7fd3b00a39b0 2024-09-16T04:11:42.659 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.639+0000 7fd3977fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:11:42.659 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.639+0000 7fd3977fe640 1 -- 172.21.15.42:0/2717469761 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3a00b48a0 con 0x7fd3b00a39b0 2024-09-16T04:11:42.659 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.639+0000 7fd3977fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:42.659 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.639+0000 7fd3977fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:11:42.659 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.639+0000 7fd3977fe640 10 monclient: dump: 2024-09-16T04:11:42.659 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:11:42.659 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:11:42.659 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:11:42.659 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:11:42.659 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:42.659 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:11:42.659 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:11:42.660 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:11:42.660 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:11:42.660 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:11:42.660 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.639+0000 7fd3b53a8640 10 client.1014524.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:11:42.660 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.639+0000 7fd3b53a8640 10 monclient: _renew_subs 2024-09-16T04:11:42.660 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.639+0000 7fd3ad7fa640 4 set_mon_vals no callback set 2024-09-16T04:11:42.660 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.639+0000 7fd3b53a8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:11:42.660 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.639+0000 7fd3b53a8640 1 -- 172.21.15.42:0/2717469761 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3b0139830 con 0x7fd3b00a39b0 2024-09-16T04:11:42.660 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.639+0000 7fd3b53a8640 1 librados: init done 2024-09-16T04:11:42.660 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.639+0000 7fd3b53a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:42.660 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.639+0000 7fd3b53a8640 1 --2- 172.21.15.42:0/2717469761 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd3b00a1ee0 0x7fd3b00a22c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:11:42.660 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.639+0000 7fd3b53a8640 10 monclient(hunting): start opening mon connection 2024-09-16T04:11:42.660 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.639+0000 7fd3b53a8640 1 -- 172.21.15.42:0/2717469761 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd3b013aad0 con 0x7fd3b00a1ee0 2024-09-16T04:11:42.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.639+0000 7fd3977fe640 1 -- 172.21.15.42:0/2717469761 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fd3a00b4a40 con 0x7fd3b00a39b0 2024-09-16T04:11:42.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.639+0000 7fd3977fe640 1 --2- 172.21.15.42:0/2717469761 >> v2:172.21.15.154:6812/254845342 conn(0x7fd39c037810 0x7fd39c039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:11:42.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.639+0000 7fd3977fe640 1 -- 172.21.15.42:0/2717469761 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(706..706 src has 451..706) ==== 5040+0+0 (secure 0 0 0) 0x7fd3a00e2bf0 con 0x7fd3b00a39b0 2024-09-16T04:11:42.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.639+0000 7fd3977fe640 10 client.1014524.objecter ms_dispatch 0x7fd3b0000ba0 osd_map(706..706 src has 451..706) 2024-09-16T04:11:42.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.639+0000 7fd3977fe640 3 client.1014524.objecter handle_osd_map got epochs [706,706] > 0 2024-09-16T04:11:42.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.639+0000 7fd3977fe640 3 client.1014524.objecter handle_osd_map decoding full epoch 706 2024-09-16T04:11:42.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.643+0000 7fd3977fe640 20 client.1014524.objecter dump_active .. 0 homeless 2024-09-16T04:11:42.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.643+0000 7fd3ae7fc640 1 --2- 172.21.15.42:0/2717469761 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd3b00a1ee0 0x7fd3b00a22c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:11:42.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.643+0000 7fd3ae7fc640 10 monclient: get_auth_request con 0x7fd3b00a1ee0 auth_method 0 2024-09-16T04:11:42.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.643+0000 7fd3ae7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:11:42.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.643+0000 7fd3ae7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:42.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.643+0000 7fd3ae7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:42.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.643+0000 7fd3ae7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:11:42.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.643+0000 7fd3ae7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:11:42.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.643+0000 7fd3af7fe640 1 --2- 172.21.15.42:0/2717469761 >> v2:172.21.15.154:6812/254845342 conn(0x7fd39c037810 0x7fd39c039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:11:42.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.643+0000 7fd3ae7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:11:42.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.643+0000 7fd3af7fe640 10 monclient: get_auth_request con 0x7fd39c037810 auth_method 0 2024-09-16T04:11:42.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.643+0000 7fd3ae7fc640 10 monclient(hunting): handle_auth_done global_id 1074121 payload 824 2024-09-16T04:11:42.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.643+0000 7fd3ae7fc640 1 -- 172.21.15.42:0/2717469761 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd3b0133b80 con 0x7fd3b00a1ee0 2024-09-16T04:11:42.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.643+0000 7fd3ae7fc640 1 -- 172.21.15.42:0/2717469761 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd3b00a1ee0 msgr2=0x7fd3b00a22c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-16T04:11:42.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.643+0000 7fd3ae7fc640 1 -- 172.21.15.42:0/2717469761 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd3b00a1ee0 msgr2=0x7fd3b00a22c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:11:42.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.643+0000 7fd3ae7fc640 1 --2- 172.21.15.42:0/2717469761 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd3b00a1ee0 0x7fd3b00a22c0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0x7fd3b00a3240 tx=0x7fd3a400ff00 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:11:42.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.643+0000 7fd3ae7fc640 1 --2- 172.21.15.42:0/2717469761 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd3b00a1ee0 0x7fd3b00a22c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:11:42.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.643+0000 7fd3977fe640 10 client.1014524.objecter ms_handle_connect 0x7fd39c037810 2024-09-16T04:11:42.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.643+0000 7fd3af7fe640 1 --2- 172.21.15.42:0/2717469761 >> v2:172.21.15.154:6812/254845342 conn(0x7fd39c037810 0x7fd39c039cd0 secure :-1 s=READY pgs=3949 cs=0 l=1 rev1=1 crypto rx=0x7fd3b0140140 tx=0x7fd3a8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:11:42.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.847+0000 7fd3ae7fc640 1 --2- 172.21.15.42:0/2717469761 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd3b00a1ee0 0x7fd3b00a22c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:11:42.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.847+0000 7fd3ae7fc640 10 monclient: get_auth_request con 0x7fd3b00a1ee0 auth_method 0 2024-09-16T04:11:42.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.847+0000 7fd3ae7fc640 10 monclient: get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:11:42.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.847+0000 7fd3ae7fc640 10 monclient: _init_auth method 2 2024-09-16T04:11:42.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.847+0000 7fd3ae7fc640 10 monclient: _init_auth already have auth, reseting 2024-09-16T04:11:42.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.847+0000 7fd3ae7fc640 10 monclient: handle_auth_reply_more payload 9 2024-09-16T04:11:42.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.847+0000 7fd3ae7fc640 10 monclient: handle_auth_reply_more payload_len 9 2024-09-16T04:11:42.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.847+0000 7fd3ae7fc640 10 monclient: handle_auth_reply_more responding with 132 bytes 2024-09-16T04:11:42.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.847+0000 7fd3ae7fc640 10 monclient: handle_auth_done global_id 1074121 payload 293 2024-09-16T04:11:42.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.847+0000 7fd3ae7fc640 1 --2- 172.21.15.42:0/2717469761 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd3b00a1ee0 0x7fd3b00a22c0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fd3a4001860 tx=0x7fd3a4009040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:11:42.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.847+0000 7fd3977fe640 10 client.1014524.objecter ms_handle_connect 0x7fd3b00a1ee0 2024-09-16T04:11:42.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.847+0000 7fd3977fe640 10 client.1014524.objecter resend_mon_ops 2024-09-16T04:11:42.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.847+0000 7fd3977fe640 1 -- 172.21.15.42:0/2717469761 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3a4005190 con 0x7fd3b00a1ee0 2024-09-16T04:11:42.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.847+0000 7fd3977fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:42.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.847+0000 7fd3977fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:11:42.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.847+0000 7fd3977fe640 10 monclient: dump: 2024-09-16T04:11:42.852 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:11:42.852 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:11:42.852 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:11:42.852 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:11:42.852 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:42.852 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:11:42.852 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:11:42.852 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:11:42.852 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:11:42.853 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:11:42.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.847+0000 7fd3977fe640 1 -- 172.21.15.42:0/2717469761 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fd3a4007020 con 0x7fd3b00a1ee0 2024-09-16T04:11:42.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.847+0000 7fd3977fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:42.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.847+0000 7fd3977fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:11:42.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.847+0000 7fd3977fe640 1 -- 172.21.15.42:0/2717469761 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd3b00a1ee0 msgr2=0x7fd3b00a22c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:42.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.847+0000 7fd3977fe640 1 --2- 172.21.15.42:0/2717469761 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd3b00a1ee0 0x7fd3b00a22c0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fd3a4001860 tx=0x7fd3a4009040 comp rx=0 tx=0).stop 2024-09-16T04:11:42.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.847+0000 7fd3977fe640 1 -- 172.21.15.42:0/2717469761 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd3b00a1ee0 msgr2=0x7fd3b00a22c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:11:42.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.847+0000 7fd3977fe640 1 --2- 172.21.15.42:0/2717469761 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd3b00a1ee0 0x7fd3b00a22c0 secure :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fd3a4001860 tx=0x7fd3a4009040 comp rx=0 tx=0).stop 2024-09-16T04:11:42.860 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.855+0000 7fd3b53a8640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:11:42.860 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.855+0000 7fd3b53a8640 1 --2- 172.21.15.42:0/2717469761 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd3b00a2800 0x7fd3b009d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:11:42.860 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.855+0000 7fd3b53a8640 10 monclient(hunting): start opening mon connection 2024-09-16T04:11:42.860 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.855+0000 7fd3b53a8640 1 -- 172.21.15.42:0/2717469761 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd3b00a5750 con 0x7fd3b00a2800 2024-09-16T04:11:42.860 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.855+0000 7fd3ae7fc640 1 --2- 172.21.15.42:0/2717469761 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd3b00a2800 0x7fd3b009d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:11:42.860 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.855+0000 7fd3ae7fc640 10 monclient: get_auth_request con 0x7fd3b00a2800 auth_method 0 2024-09-16T04:11:42.860 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.855+0000 7fd3ae7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:11:42.860 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.855+0000 7fd3ae7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:42.860 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.855+0000 7fd3ae7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:42.861 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.855+0000 7fd3ae7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:11:42.861 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.855+0000 7fd3ae7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:11:42.861 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.855+0000 7fd3ae7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:11:42.862 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.855+0000 7fd3ae7fc640 10 monclient(hunting): handle_auth_done global_id 1074124 payload 824 2024-09-16T04:11:42.862 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.855+0000 7fd3ae7fc640 1 -- 172.21.15.42:0/2717469761 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fd3b01386c0 con 0x7fd3b00a2800 2024-09-16T04:11:42.862 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.855+0000 7fd3ae7fc640 1 --2- 172.21.15.42:0/2717469761 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd3b00a2800 0x7fd3b009d8e0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fd3b006b6f0 tx=0x7fd3a4007f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:11:42.862 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.859+0000 7fd3977fe640 10 client.1014524.objecter ms_handle_connect 0x7fd3b00a2800 2024-09-16T04:11:42.862 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.859+0000 7fd3977fe640 10 client.1014524.objecter resend_mon_ops 2024-09-16T04:11:42.862 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.859+0000 7fd3977fe640 1 -- 172.21.15.42:0/2717469761 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3a4024040 con 0x7fd3b00a2800 2024-09-16T04:11:42.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.859+0000 7fd3977fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:42.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.859+0000 7fd3977fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:11:42.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.859+0000 7fd3977fe640 10 monclient: dump: 2024-09-16T04:11:42.863 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:11:42.863 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:11:42.863 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:11:42.863 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:11:42.863 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:42.863 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:11:42.863 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:11:42.863 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:11:42.863 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:11:42.863 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:11:42.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.859+0000 7fd3977fe640 1 -- 172.21.15.42:0/2717469761 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4545 (secure 0 0 0) 0x7fd3a4006e20 con 0x7fd3b00a2800 2024-09-16T04:11:42.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.859+0000 7fd3977fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:11:42.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.859+0000 7fd3977fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:11:42.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.859+0000 7fd3977fe640 1 -- 172.21.15.42:0/2717469761 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd3b00a2800 msgr2=0x7fd3b009d8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:42.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.859+0000 7fd3977fe640 1 --2- 172.21.15.42:0/2717469761 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd3b00a2800 0x7fd3b009d8e0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fd3b006b6f0 tx=0x7fd3a4007f90 comp rx=0 tx=0).stop 2024-09-16T04:11:42.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.859+0000 7fd3977fe640 1 -- 172.21.15.42:0/2717469761 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd3b00a2800 msgr2=0x7fd3b009d8e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:11:42.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.859+0000 7fd3977fe640 1 --2- 172.21.15.42:0/2717469761 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd3b00a2800 0x7fd3b009d8e0 secure :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fd3b006b6f0 tx=0x7fd3a4007f90 comp rx=0 tx=0).stop 2024-09-16T04:11:42.864 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T04:11:42.864 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:11:42.864 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:11:42.865 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T04:11:42.865 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 268, 2024-09-16T04:11:42.865 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 16031, 2024-09-16T04:11:42.865 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T04:11:42.865 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T04:11:42.865 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T04:11:42.865 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T04:11:42.865 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:11:42.865 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 1, 2024-09-16T04:11:42.865 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:11:42.865 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T04:11:42.865 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T04:11:42.865 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:11:42.865 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:11:42.866 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:11:42.866 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:11:42.866 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:11:42.866 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:11:42.866 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:11:42.866 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:11:42.866 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:11:42.866 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:11:42.866 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:11:42.866 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:11:42.866 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:11:42.866 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T04:11:42.866 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:11:42.867 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:11:42.867 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:11:42.867 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:11:42.867 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:11:42.867 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:11:42.867 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:11:42.867 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:11:42.867 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:11:42.867 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:11:42.867 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:11:42.867 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:11:42.868 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:11:42.868 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T04:11:42.868 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T04:11:42.868 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T04:11:42.868 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T04:11:42.868 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T04:11:42.869 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T04:11:42.869 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:11:42.869 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:11:42.869 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T04:11:42.869 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T04:11:42.869 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T04:11:42.869 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T04:11:42.869 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T04:11:42.869 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T04:11:42.869 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T04:11:42.869 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:11:42.869 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T04:11:42.870 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:11:42.870 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:11:42.870 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:11:42.870 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:11:42.870 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:11:42.870 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:11:42.870 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:11:42.870 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:11:42.870 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:11:42.870 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:11:42.871 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:11:42.871 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:11:42.871 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T04:11:42.871 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:11:42.871 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T04:11:42.871 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:11:42.871 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:11:42.871 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:11:42.871 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:11:42.871 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:11:42.871 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:11:42.871 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:11:42.871 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T04:11:42.871 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:11:42.872 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:11:42.872 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:11:42.872 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:11:42.872 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T04:11:42.872 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:11:42.872 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:11:42.872 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:11:42.872 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:11:42.872 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T04:11:42.872 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T04:11:42.872 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:11:42.872 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:11:42.872 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:11:42.873 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:11:42.873 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:11:42.873 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:11:42.873 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:11:42.873 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:11:42.873 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:11:42.873 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:11:42.873 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:11:42.873 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T04:11:42.873 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:11:42.873 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:11:42.873 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:11:42.874 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:11:42.874 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T04:11:42.874 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:11:42.874 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:11:42.874 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:11:42.874 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:11:42.874 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T04:11:42.874 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T04:11:42.874 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:11:42.874 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:11:42.874 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:11:42.874 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:11:42.874 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:11:42.875 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:11:42.875 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:11:42.875 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:11:42.875 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:11:42.875 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:11:42.875 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:11:42.875 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T04:11:42.875 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:11:42.875 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:11:42.875 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:11:42.875 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:11:42.875 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T04:11:42.875 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:11:42.876 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:11:42.876 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:11:42.876 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:11:42.876 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T04:11:42.876 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T04:11:42.876 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:11:42.876 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:11:42.876 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:11:42.876 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:11:42.876 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:11:42.876 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:11:42.876 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T04:11:42.876 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T04:11:42.876 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:11:42.877 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:11:42.877 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:11:42.877 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:11:42.877 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:11:42.877 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:11:42.877 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T04:11:42.877 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:11:42.877 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:11:42.877 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:11:42.877 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:11:42.877 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:11:42.877 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:11:42.877 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:11:42.878 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T04:11:42.878 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T04:11:42.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.859+0000 7fd3b53a8640 10 librados: watch_flush enter 2024-09-16T04:11:42.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.859+0000 7fd3b53a8640 10 librados: watch_flush exit 2024-09-16T04:11:42.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.859+0000 7fd3b53a8640 20 client.1014524.objecter shutdown clearing up homeless session... 2024-09-16T04:11:42.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.859+0000 7fd3b53a8640 10 client.1014524.objecter successfully canceled tick 2024-09-16T04:11:42.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.859+0000 7fd3b53a8640 1 -- 172.21.15.42:0/2717469761 >> v2:172.21.15.154:6812/254845342 conn(0x7fd39c037810 msgr2=0x7fd39c039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:42.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.859+0000 7fd3b53a8640 1 --2- 172.21.15.42:0/2717469761 >> v2:172.21.15.154:6812/254845342 conn(0x7fd39c037810 0x7fd39c039cd0 secure :-1 s=READY pgs=3949 cs=0 l=1 rev1=1 crypto rx=0x7fd3b0140140 tx=0x7fd3a8009040 comp rx=0 tx=0).stop 2024-09-16T04:11:42.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.859+0000 7fd3b53a8640 10 monclient: shutdown 2024-09-16T04:11:42.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.859+0000 7fd3b53a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:11:42.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.859+0000 7fd3b53a8640 1 -- 172.21.15.42:0/2717469761 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd3b00a39b0 msgr2=0x7fd3b0130660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:42.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.859+0000 7fd3b53a8640 1 --2- 172.21.15.42:0/2717469761 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd3b00a39b0 0x7fd3b0130660 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fd3a001a050 tx=0x7fd3a00191c0 comp rx=0 tx=0).stop 2024-09-16T04:11:42.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.859+0000 7fd3af7fe640 1 -- 172.21.15.42:0/2717469761 reap_dead start 2024-09-16T04:11:42.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.859+0000 7fd3af7fe640 1 -- 172.21.15.42:0/2717469761 reap_dead start 2024-09-16T04:11:42.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.859+0000 7fd3b53a8640 1 -- 172.21.15.42:0/2717469761 shutdown_connections 2024-09-16T04:11:42.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.859+0000 7fd3b53a8640 1 -- 172.21.15.42:0/2717469761 >> 172.21.15.42:0/2717469761 conn(0x7fd3b00a9fb0 msgr2=0x7fd3b0131980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:11:42.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.859+0000 7fd3b53a8640 1 -- 172.21.15.42:0/2717469761 shutdown_connections 2024-09-16T04:11:42.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.859+0000 7fd3b53a8640 1 -- 172.21.15.42:0/2717469761 wait complete. 2024-09-16T04:11:42.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:42.859+0000 7fd3b53a8640 1 librados: shutdown 2024-09-16T04:11:42.882 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:11:42.882 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T04:11:42.882 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T04:11:43.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.007+0000 7f5c87612640 10 monclient: get_monmap_and_config 2024-09-16T04:11:43.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.007+0000 7f5c87612640 10 monclient: build_initial_monmap 2024-09-16T04:11:43.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.007+0000 7f5c87612640 10 monclient: monmap: 2024-09-16T04:11:43.012 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:11:43.012 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:11:43.013 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:11:43.009329+0000 2024-09-16T04:11:43.013 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:11:43.009329+0000 2024-09-16T04:11:43.013 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:11:43.013 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:11:43.013 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:11:43.013 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:11:43.014 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:11:43.014 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:11:43.014 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:11:43.014 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:11:43.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.007+0000 7f5c87612640 1 Processor -- start 2024-09-16T04:11:43.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.007+0000 7f5c87612640 1 -- start start 2024-09-16T04:11:43.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.007+0000 7f5c87612640 10 monclient: init 2024-09-16T04:11:43.023 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.007+0000 7f5c87612640 10 monclient: _reopen_session rank -1 2024-09-16T04:11:43.023 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.007+0000 7f5c87612640 10 monclient: _add_conns ranks=[2,0,3,4,1] 2024-09-16T04:11:43.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.007+0000 7f5c87612640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5c800a1330 0x7f5c8009f180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:11:43.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.007+0000 7f5c87612640 10 monclient(hunting): picked mon.noname-b con 0x7f5c800a1330 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:11:43.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.007+0000 7f5c87612640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5c800a1710 addr v1:172.21.15.42:6789/0 2024-09-16T04:11:43.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.007+0000 7f5c87612640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f5c801393a0 0x7f5c8013b790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:11:43.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.007+0000 7f5c87612640 10 monclient(hunting): picked mon.noname-a con 0x7f5c801393a0 addr v2:172.21.15.42:3300/0 2024-09-16T04:11:43.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.007+0000 7f5c87612640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c8009e4c0 con 0x7f5c800a1710 2024-09-16T04:11:43.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.007+0000 7f5c87612640 10 monclient(hunting): start opening mon connection 2024-09-16T04:11:43.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.007+0000 7f5c87612640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f5c8009edf0 con 0x7f5c801393a0 2024-09-16T04:11:43.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.007+0000 7f5c87612640 10 monclient(hunting): start opening mon connection 2024-09-16T04:11:43.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.007+0000 7f5c87612640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f5c8009cd60 con 0x7f5c800a1330 2024-09-16T04:11:43.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.007+0000 7f5c87612640 10 monclient(hunting): _renew_subs 2024-09-16T04:11:43.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.007+0000 7f5c87612640 10 monclient(hunting): authenticate will time out at 2749.979324s 2024-09-16T04:11:43.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.007+0000 7f5c853ae640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5c800a1330 0x7f5c8009f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:11:43.025 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.007+0000 7f5c853ae640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5c800a1330 0x7f5c8009f180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:37662/0 (socket says 172.21.15.42:37662) 2024-09-16T04:11:43.025 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.007+0000 7f5c853ae640 1 -- 172.21.15.42:0/169230514 learned_addr learned my addr 172.21.15.42:0/169230514 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:11:43.025 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.007+0000 7f5c853ae640 10 monclient(hunting): get_auth_request con 0x7f5c800a1330 auth_method 0 2024-09-16T04:11:43.025 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.007+0000 7f5c853ae640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:11:43.025 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.007+0000 7f5c853ae640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:43.025 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.007+0000 7f5c853ae640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:43.025 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.007+0000 7f5c85baf640 1 --2- 172.21.15.42:0/169230514 >> v2:172.21.15.42:3300/0 conn(0x7f5c801393a0 0x7f5c8013b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:11:43.025 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.007+0000 7f5c853ae640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:11:43.025 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.007+0000 7f5c853ae640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:11:43.025 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.007+0000 7f5c853ae640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:11:43.025 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.007+0000 7f5c85baf640 10 monclient(hunting): get_auth_request con 0x7f5c801393a0 auth_method 0 2024-09-16T04:11:43.025 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.007+0000 7f5c85baf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:11:43.025 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.007+0000 7f5c85baf640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:43.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.007+0000 7f5c85baf640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:43.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.007+0000 7f5c853ae640 10 monclient(hunting): handle_auth_done global_id 1014527 payload 274 2024-09-16T04:11:43.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.007+0000 7f5c853ae640 1 -- 172.21.15.42:0/169230514 >> v2:172.21.15.42:3300/0 conn(0x7f5c801393a0 msgr2=0x7f5c8013b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:11:43.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.007+0000 7f5c853ae640 1 --2- 172.21.15.42:0/169230514 >> v2:172.21.15.42:3300/0 conn(0x7f5c801393a0 0x7f5c8013b790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:11:43.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.007+0000 7f5c853ae640 1 -- 172.21.15.42:0/169230514 >> v1:172.21.15.42:6789/0 conn(0x7f5c800a1710 legacy=0x7f5c800a1b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:43.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.007+0000 7f5c853ae640 10 monclient: _finish_hunting 0 2024-09-16T04:11:43.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.007+0000 7f5c853ae640 1 monclient: found mon.noname-b 2024-09-16T04:11:43.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.007+0000 7f5c853ae640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:11:43.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.007+0000 7f5c853ae640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T04:11:43.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.007+0000 7f5c853ae640 1 -- 172.21.15.42:0/169230514 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c8009f6c0 con 0x7f5c800a1330 2024-09-16T04:11:43.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.007+0000 7f5c85baf640 1 --2- 172.21.15.42:0/169230514 >> v2:172.21.15.42:3300/0 conn(0x7f5c801393a0 0x7f5c8013b790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T04:11:43.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.011+0000 7f5c853ae640 1 --2- 172.21.15.42:0/169230514 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5c800a1330 0x7f5c8009f180 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f5c7c009e50 tx=0x7f5c7c01c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ebcacac5ee65b33c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:11:43.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.011+0000 7f5c77fff640 1 -- 172.21.15.42:0/169230514 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c7c0a56c0 con 0x7f5c800a1330 2024-09-16T04:11:43.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.011+0000 7f5c77fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:43.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.011+0000 7f5c77fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:11:43.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.011+0000 7f5c77fff640 10 monclient: dump: 2024-09-16T04:11:43.027 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:11:43.027 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:11:43.027 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:11:43.027 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:11:43.027 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:43.027 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:11:43.027 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:11:43.027 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:11:43.027 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:11:43.027 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:11:43.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.011+0000 7f5c77fff640 10 monclient: _finish_auth 0 2024-09-16T04:11:43.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.011+0000 7f5c77fff640 10 monclient: _check_auth_tickets 2024-09-16T04:11:43.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.011+0000 7f5c77fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:11:43.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.011+0000 7f5c77fff640 1 -- 172.21.15.42:0/169230514 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5c7c0a5860 con 0x7f5c800a1330 2024-09-16T04:11:43.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.011+0000 7f5c87612640 5 monclient: authenticate success, global_id 1014527 2024-09-16T04:11:43.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.011+0000 7f5c87612640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:11:43.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.011+0000 7f5c77fff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:11:43.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.011+0000 7f5c87612640 10 monclient: get_monmap_and_config success 2024-09-16T04:11:43.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.011+0000 7f5c77fff640 1 -- 172.21.15.42:0/169230514 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c7c0a5c30 con 0x7f5c800a1330 2024-09-16T04:11:43.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.011+0000 7f5c87612640 4 set_mon_vals no callback set 2024-09-16T04:11:43.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.011+0000 7f5c77fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:43.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.011+0000 7f5c77fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:11:43.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.011+0000 7f5c77fff640 10 monclient: dump: 2024-09-16T04:11:43.028 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:11:43.028 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:11:43.029 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:11:43.029 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:11:43.029 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:43.029 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:11:43.029 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:11:43.029 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:11:43.029 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:11:43.029 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:11:43.029 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.011+0000 7f5c87612640 10 monclient: shutdown 2024-09-16T04:11:43.029 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.011+0000 7f5c87612640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:11:43.029 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.011+0000 7f5c87612640 1 -- 172.21.15.42:0/169230514 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5c800a1330 msgr2=0x7f5c8009f180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:43.029 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.011+0000 7f5c87612640 1 --2- 172.21.15.42:0/169230514 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5c800a1330 0x7f5c8009f180 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f5c7c009e50 tx=0x7f5c7c01c8f0 comp rx=0 tx=0).stop 2024-09-16T04:11:43.029 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.011+0000 7f5c87612640 1 -- 172.21.15.42:0/169230514 shutdown_connections 2024-09-16T04:11:43.029 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.011+0000 7f5c87612640 1 --2- 172.21.15.42:0/169230514 >> v2:172.21.15.42:3300/0 conn(0x7f5c801393a0 0x7f5c8013b790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:11:43.030 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.011+0000 7f5c87612640 1 --2- 172.21.15.42:0/169230514 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5c800a1330 0x7f5c8009f180 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:11:43.030 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.011+0000 7f5c87612640 1 -- 172.21.15.42:0/169230514 >> 172.21.15.42:0/169230514 conn(0x7f5c800a2420 msgr2=0x7f5c800a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:11:43.030 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.011+0000 7f5c87612640 1 -- 172.21.15.42:0/169230514 shutdown_connections 2024-09-16T04:11:43.030 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.011+0000 7f5c87612640 1 -- 172.21.15.42:0/169230514 wait complete. 2024-09-16T04:11:43.030 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.015+0000 7f5c87612640 10 monclient: build_initial_monmap 2024-09-16T04:11:43.030 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.015+0000 7f5c87612640 10 monclient: monmap: 2024-09-16T04:11:43.030 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:11:43.030 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:11:43.030 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:11:43.030 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:11:43.030 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:11:43.030 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:11:43.030 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:11:43.030 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:11:43.031 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:11:43.031 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:11:43.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.015+0000 7f5c87612640 1 librados: starting msgr at 2024-09-16T04:11:43.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.015+0000 7f5c87612640 1 librados: starting objecter 2024-09-16T04:11:43.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.015+0000 7f5c87612640 1 Processor -- start 2024-09-16T04:11:43.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.015+0000 7f5c87612640 1 -- start start 2024-09-16T04:11:43.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.015+0000 7f5c87612640 1 librados: setting wanted keys 2024-09-16T04:11:43.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.015+0000 7f5c87612640 1 librados: calling monclient init 2024-09-16T04:11:43.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.015+0000 7f5c87612640 10 monclient: init 2024-09-16T04:11:43.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.015+0000 7f5c87612640 10 monclient: _reopen_session rank -1 2024-09-16T04:11:43.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.015+0000 7f5c87612640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:11:43.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.015+0000 7f5c87612640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5c800a1330 0x7f5c801cccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:11:43.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.015+0000 7f5c87612640 10 monclient(hunting): picked mon.noname-c con 0x7f5c800a1330 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:11:43.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.015+0000 7f5c87612640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5c800a1710 0x7f5c801cd400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:11:43.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.015+0000 7f5c87612640 10 monclient(hunting): picked mon.noname-b con 0x7f5c800a1710 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:11:43.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.015+0000 7f5c87612640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5c801393a0 0x7f5c80070630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:11:43.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.015+0000 7f5c87612640 10 monclient(hunting): picked mon.noname-a con 0x7f5c801393a0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:11:43.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.015+0000 7f5c87612640 10 monclient(hunting): start opening mon connection 2024-09-16T04:11:43.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.015+0000 7f5c87612640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f5c8013c870 con 0x7f5c801393a0 2024-09-16T04:11:43.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.015+0000 7f5c87612640 10 monclient(hunting): start opening mon connection 2024-09-16T04:11:43.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.015+0000 7f5c87612640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f5c8013c6f0 con 0x7f5c800a1710 2024-09-16T04:11:43.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.015+0000 7f5c87612640 10 monclient(hunting): start opening mon connection 2024-09-16T04:11:43.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.015+0000 7f5c87612640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f5c8013c9f0 con 0x7f5c800a1330 2024-09-16T04:11:43.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.015+0000 7f5c87612640 10 monclient(hunting): _renew_subs 2024-09-16T04:11:43.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.015+0000 7f5c87612640 10 monclient(hunting): authenticate will time out at 2749.985107s 2024-09-16T04:11:43.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.015+0000 7f5c85baf640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5c801393a0 0x7f5c80070630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:11:43.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.015+0000 7f5c853ae640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5c800a1330 0x7f5c801cccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:11:43.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.015+0000 7f5c853ae640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5c800a1330 0x7f5c801cccf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:37664/0 (socket says 172.21.15.42:37664) 2024-09-16T04:11:43.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.015+0000 7f5c853ae640 1 -- 172.21.15.42:0/1644511287 learned_addr learned my addr 172.21.15.42:0/1644511287 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:11:43.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.015+0000 7f5c853ae640 10 monclient(hunting): get_auth_request con 0x7f5c800a1330 auth_method 0 2024-09-16T04:11:43.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.015+0000 7f5c85baf640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5c801393a0 0x7f5c80070630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:57658/0 (socket says 172.21.15.42:57658) 2024-09-16T04:11:43.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.015+0000 7f5c853ae640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:11:43.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.015+0000 7f5c853ae640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:43.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.015+0000 7f5c853ae640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:43.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.015+0000 7f5c85baf640 10 monclient(hunting): get_auth_request con 0x7f5c801393a0 auth_method 0 2024-09-16T04:11:43.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.015+0000 7f5c85baf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:11:43.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.015+0000 7f5c85baf640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:43.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.015+0000 7f5c85baf640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:43.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.015+0000 7f5c853ae640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:11:43.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.015+0000 7f5c853ae640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:11:43.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.015+0000 7f5c853ae640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:11:43.033 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.015+0000 7f5c85baf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:11:43.034 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.015+0000 7f5c85baf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:11:43.034 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.015+0000 7f5c85baf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:11:43.034 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.015+0000 7f5c84bad640 1 --2- 172.21.15.42:0/1644511287 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5c800a1710 0x7f5c801cd400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:11:43.034 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.015+0000 7f5c84bad640 10 monclient(hunting): get_auth_request con 0x7f5c800a1710 auth_method 0 2024-09-16T04:11:43.034 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.015+0000 7f5c84bad640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:11:43.034 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.015+0000 7f5c84bad640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:43.034 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.015+0000 7f5c84bad640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:43.034 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.015+0000 7f5c853ae640 10 monclient(hunting): handle_auth_done global_id 1014530 payload 824 2024-09-16T04:11:43.034 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.015+0000 7f5c853ae640 1 -- 172.21.15.42:0/1644511287 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5c800a1710 msgr2=0x7f5c801cd400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:43.034 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.015+0000 7f5c853ae640 1 --2- 172.21.15.42:0/1644511287 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5c800a1710 0x7f5c801cd400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:11:43.034 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.015+0000 7f5c853ae640 1 -- 172.21.15.42:0/1644511287 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5c801393a0 msgr2=0x7f5c80070630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:43.034 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.015+0000 7f5c853ae640 1 --2- 172.21.15.42:0/1644511287 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5c801393a0 0x7f5c80070630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:11:43.034 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.015+0000 7f5c853ae640 10 monclient: _finish_hunting 0 2024-09-16T04:11:43.034 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.015+0000 7f5c853ae640 1 monclient: found mon.noname-c 2024-09-16T04:11:43.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.015+0000 7f5c853ae640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:11:43.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.015+0000 7f5c853ae640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T04:11:43.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.015+0000 7f5c853ae640 1 -- 172.21.15.42:0/1644511287 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c80073a30 con 0x7f5c800a1330 2024-09-16T04:11:43.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.015+0000 7f5c853ae640 1 --2- 172.21.15.42:0/1644511287 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5c800a1330 0x7f5c801cccf0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f5c7c01a520 tx=0x7f5c7c019690 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:11:43.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.015+0000 7f5c75ffb640 10 client.?.objecter ms_handle_connect 0x7f5c800a1330 2024-09-16T04:11:43.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.015+0000 7f5c75ffb640 10 client.?.objecter resend_mon_ops 2024-09-16T04:11:43.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.015+0000 7f5c75ffb640 1 -- 172.21.15.42:0/1644511287 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c7c0bc020 con 0x7f5c800a1330 2024-09-16T04:11:43.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.015+0000 7f5c75ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:43.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.015+0000 7f5c75ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:11:43.035 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.015+0000 7f5c75ffb640 10 monclient: dump: 2024-09-16T04:11:43.035 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:11:43.035 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:11:43.035 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:11:43.035 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:11:43.036 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:43.036 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:11:43.036 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:11:43.036 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:11:43.036 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:11:43.036 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:11:43.036 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.015+0000 7f5c75ffb640 10 monclient: _finish_auth 0 2024-09-16T04:11:43.036 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.015+0000 7f5c75ffb640 10 monclient: _check_auth_tickets 2024-09-16T04:11:43.036 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.015+0000 7f5c75ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:11:43.036 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.015+0000 7f5c87612640 5 monclient: authenticate success, global_id 1014530 2024-09-16T04:11:43.036 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.015+0000 7f5c75ffb640 1 -- 172.21.15.42:0/1644511287 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5c7c0b8670 con 0x7f5c800a1330 2024-09-16T04:11:43.036 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.015+0000 7f5c75ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:11:43.036 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.015+0000 7f5c87612640 10 monclient: _renew_subs 2024-09-16T04:11:43.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.015+0000 7f5c75ffb640 1 -- 172.21.15.42:0/1644511287 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c7c0b8a40 con 0x7f5c800a1330 2024-09-16T04:11:43.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.015+0000 7f5c87612640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:11:43.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.015+0000 7f5c87612640 1 -- 172.21.15.42:0/1644511287 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c8011cca0 con 0x7f5c800a1330 2024-09-16T04:11:43.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.015+0000 7f5c75ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:43.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.015+0000 7f5c777fe640 4 set_mon_vals no callback set 2024-09-16T04:11:43.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.015+0000 7f5c75ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:11:43.037 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.015+0000 7f5c75ffb640 10 monclient: dump: 2024-09-16T04:11:43.037 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:11:43.037 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:11:43.037 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:11:43.037 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:11:43.037 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:43.037 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:11:43.037 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:11:43.038 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:11:43.038 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:11:43.038 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:11:43.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.015+0000 7f5c87612640 10 client.1014530.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:11:43.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.015+0000 7f5c87612640 10 monclient: _renew_subs 2024-09-16T04:11:43.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.015+0000 7f5c87612640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:11:43.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.015+0000 7f5c87612640 1 -- 172.21.15.42:0/1644511287 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c80072770 con 0x7f5c800a1330 2024-09-16T04:11:43.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.015+0000 7f5c87612640 1 librados: init done 2024-09-16T04:11:43.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.015+0000 7f5c87612640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:43.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.015+0000 7f5c87612640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:43.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.015+0000 7f5c87612640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:11:43.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.015+0000 7f5c75ffb640 1 -- 172.21.15.42:0/1644511287 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f5c7c0bf020 con 0x7f5c800a1330 2024-09-16T04:11:43.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.015+0000 7f5c87612640 1 -- 172.21.15.42:0/1644511287 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5c8009f180 con 0x7f5c800a1330 2024-09-16T04:11:43.038 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.015+0000 7f5c75ffb640 1 --2- 172.21.15.42:0/1644511287 >> v2:172.21.15.154:6812/254845342 conn(0x7f5c64037810 0x7f5c64039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:11:43.039 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.015+0000 7f5c75ffb640 1 -- 172.21.15.42:0/1644511287 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(706..706 src has 451..706) ==== 5040+0+0 (secure 0 0 0) 0x7f5c7c0e3c30 con 0x7f5c800a1330 2024-09-16T04:11:43.039 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.015+0000 7f5c75ffb640 10 client.1014530.objecter ms_dispatch 0x7f5c80000ba0 osd_map(706..706 src has 451..706) 2024-09-16T04:11:43.039 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.015+0000 7f5c75ffb640 3 client.1014530.objecter handle_osd_map got epochs [706,706] > 0 2024-09-16T04:11:43.039 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.015+0000 7f5c75ffb640 3 client.1014530.objecter handle_osd_map decoding full epoch 706 2024-09-16T04:11:43.039 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.015+0000 7f5c75ffb640 20 client.1014530.objecter dump_active .. 0 homeless 2024-09-16T04:11:43.039 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.015+0000 7f5c84bad640 1 --2- 172.21.15.42:0/1644511287 >> v2:172.21.15.154:6812/254845342 conn(0x7f5c64037810 0x7f5c64039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:11:43.039 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.015+0000 7f5c84bad640 10 monclient: get_auth_request con 0x7f5c64037810 auth_method 0 2024-09-16T04:11:43.039 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.019+0000 7f5c75ffb640 10 client.1014530.objecter ms_handle_connect 0x7f5c64037810 2024-09-16T04:11:43.039 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.019+0000 7f5c84bad640 1 --2- 172.21.15.42:0/1644511287 >> v2:172.21.15.154:6812/254845342 conn(0x7f5c64037810 0x7f5c64039cd0 secure :-1 s=READY pgs=3951 cs=0 l=1 rev1=1 crypto rx=0x7f5c80071ed0 tx=0x7f5c70009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:11:43.040 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.019+0000 7f5c75ffb640 1 -- 172.21.15.42:0/1644511287 <== mon.2 v2:172.21.15.42:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f5c7c0c6030 con 0x7f5c800a1330 2024-09-16T04:11:43.040 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.019+0000 7f5c75ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:43.040 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.019+0000 7f5c75ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:11:43.251 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.247+0000 7f5c87612640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T04:11:43.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.247+0000 7f5c87612640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T04:11:43.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.247+0000 7f5c87612640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:11:43.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.247+0000 7f5c87612640 1 -- 172.21.15.42:0/1644511287 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f5c80069040 con 0x7f5c800a1330 2024-09-16T04:11:43.255 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.251+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.255 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.251+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=25,config=2,health=11,kv=9,logm=53} crc {auth=1442961466,config=692619927,health=626613557,kv=2818795783,logm=363973135}) 2024-09-16T04:11:43.255 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.251+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=23,config=2,health=12,kv=10,logm=53} crc {auth=3314034531,config=692619927,health=186677657,kv=3446188498,logm=1861414035}) 2024-09-16T04:11:43.255 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.251+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.255 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.251+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=25,config=2,health=11,kv=9,logm=53} crc {auth=1442961466,config=692619927,health=626613557,kv=2818795783,logm=363973135}) 2024-09-16T04:11:43.256 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.251+0000 7f5c75ffb640 1 -- 172.21.15.42:0/1644511287 <== mon.2 v2:172.21.15.42:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f5c7c016600 con 0x7f5c800a1330 2024-09-16T04:11:43.256 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.251+0000 7f5c75ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T04:11:43.256 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.251+0000 7f5c75ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:11:43.256 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.251+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=26,config=2,health=12,kv=9,logm=51} crc {auth=1721588007,config=692619927,health=186677657,kv=2094645990,logm=4016348207}) 2024-09-16T04:11:43.257 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.255+0000 7f5c87612640 10 librados: watch_flush enter 2024-09-16T04:11:43.258 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.255+0000 7f5c87612640 10 librados: watch_flush exit 2024-09-16T04:11:43.258 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.255+0000 7f5c87612640 20 client.1014530.objecter shutdown clearing up homeless session... 2024-09-16T04:11:43.258 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.255+0000 7f5c87612640 10 client.1014530.objecter successfully canceled tick 2024-09-16T04:11:43.258 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.255+0000 7f5c87612640 1 -- 172.21.15.42:0/1644511287 >> v2:172.21.15.154:6812/254845342 conn(0x7f5c64037810 msgr2=0x7f5c64039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:43.258 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.255+0000 7f5c87612640 1 --2- 172.21.15.42:0/1644511287 >> v2:172.21.15.154:6812/254845342 conn(0x7f5c64037810 0x7f5c64039cd0 secure :-1 s=READY pgs=3951 cs=0 l=1 rev1=1 crypto rx=0x7f5c80071ed0 tx=0x7f5c70009040 comp rx=0 tx=0).stop 2024-09-16T04:11:43.258 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.255+0000 7f5c87612640 10 monclient: shutdown 2024-09-16T04:11:43.258 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.255+0000 7f5c87612640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:11:43.258 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.255+0000 7f5c87612640 1 -- 172.21.15.42:0/1644511287 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5c800a1330 msgr2=0x7f5c801cccf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:43.258 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.255+0000 7f5c87612640 1 --2- 172.21.15.42:0/1644511287 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5c800a1330 0x7f5c801cccf0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f5c7c01a520 tx=0x7f5c7c019690 comp rx=0 tx=0).stop 2024-09-16T04:11:43.259 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.255+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.259 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.255+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3237638086}) 2024-09-16T04:11:43.259 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.255+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3586861304}) 2024-09-16T04:11:43.259 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.255+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.259 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.255+0000 7f5c87612640 1 -- 172.21.15.42:0/1644511287 shutdown_connections 2024-09-16T04:11:43.260 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.255+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3237638086}) 2024-09-16T04:11:43.260 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.255+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3952972582}) 2024-09-16T04:11:43.261 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.255+0000 7f5c87612640 1 --2- 172.21.15.42:0/1644511287 >> v2:172.21.15.154:6812/254845342 conn(0x7f5c64037810 0x7f5c64039cd0 unknown :-1 s=CLOSED pgs=3951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:11:43.261 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.255+0000 7f5c87612640 1 --2- 172.21.15.42:0/1644511287 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5c801393a0 0x7f5c80070630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:11:43.261 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.255+0000 7f5c87612640 1 --2- 172.21.15.42:0/1644511287 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5c800a1710 0x7f5c801cd400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:11:43.261 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.255+0000 7f5c87612640 1 --2- 172.21.15.42:0/1644511287 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5c800a1330 0x7f5c801cccf0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:11:43.261 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.255+0000 7f5c87612640 1 -- 172.21.15.42:0/1644511287 >> 172.21.15.42:0/1644511287 conn(0x7f5c800a2420 msgr2=0x7f5c800a4e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:11:43.261 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.255+0000 7f5c87612640 1 -- 172.21.15.42:0/1644511287 shutdown_connections 2024-09-16T04:11:43.261 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.255+0000 7f5c87612640 1 -- 172.21.15.42:0/1644511287 wait complete. 2024-09-16T04:11:43.261 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:43.255+0000 7f5c87612640 1 librados: shutdown 2024-09-16T04:11:43.262 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.259+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.262 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.259+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3747680265}) 2024-09-16T04:11:43.262 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.259+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4224679299}) 2024-09-16T04:11:43.262 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.259+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.262 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.259+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3747680265}) 2024-09-16T04:11:43.262 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.259+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2534595243}) 2024-09-16T04:11:43.265 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.259+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.265 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.259+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1635398009}) 2024-09-16T04:11:43.265 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.259+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3598304162}) 2024-09-16T04:11:43.265 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.263+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.265 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.263+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1635398009}) 2024-09-16T04:11:43.265 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.263+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2046836400}) 2024-09-16T04:11:43.269 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.267+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.269 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.267+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=931852462}) 2024-09-16T04:11:43.269 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.267+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3194869314}) 2024-09-16T04:11:43.269 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.267+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.270 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.267+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=931852462}) 2024-09-16T04:11:43.270 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.267+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2509160894}) 2024-09-16T04:11:43.272 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.267+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.272 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.267+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=976455584}) 2024-09-16T04:11:43.273 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.267+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=449303570}) 2024-09-16T04:11:43.273 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.267+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.273 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.267+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=976455584}) 2024-09-16T04:11:43.273 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.267+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2029288578}) 2024-09-16T04:11:43.275 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.271+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.275 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.271+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2943632706}) 2024-09-16T04:11:43.275 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.271+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2271687807}) 2024-09-16T04:11:43.275 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.271+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.275 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.271+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2943632706}) 2024-09-16T04:11:43.275 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.271+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4117702727}) 2024-09-16T04:11:43.277 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T04:11:43.278 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.271+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.278 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.275+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4247131290}) 2024-09-16T04:11:43.278 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.275+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2094119525}) 2024-09-16T04:11:43.278 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.275+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.278 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.275+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4247131290}) 2024-09-16T04:11:43.278 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.275+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2072236255}) 2024-09-16T04:11:43.280 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.275+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.280 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.275+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3845948129}) 2024-09-16T04:11:43.280 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.275+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=433880837}) 2024-09-16T04:11:43.281 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.275+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.281 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.275+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3845948129}) 2024-09-16T04:11:43.281 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.275+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2920567207}) 2024-09-16T04:11:43.282 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.279+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.282 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.279+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3249199336}) 2024-09-16T04:11:43.283 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.279+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1336696575}) 2024-09-16T04:11:43.283 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.279+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.283 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.279+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3249199336}) 2024-09-16T04:11:43.283 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.279+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3108435279}) 2024-09-16T04:11:43.285 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.279+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.285 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.279+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=230758949}) 2024-09-16T04:11:43.285 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.279+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1858508630}) 2024-09-16T04:11:43.285 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.279+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.285 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.279+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=230758949}) 2024-09-16T04:11:43.285 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.279+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2260346554}) 2024-09-16T04:11:43.286 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.283+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.286 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.283+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3573619109}) 2024-09-16T04:11:43.286 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.283+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2028291304}) 2024-09-16T04:11:43.287 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.283+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.287 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.283+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3573619109}) 2024-09-16T04:11:43.287 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.283+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3501818543}) 2024-09-16T04:11:43.289 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.283+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.289 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.283+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1879735350}) 2024-09-16T04:11:43.289 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.283+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2337286221}) 2024-09-16T04:11:43.289 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.283+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.289 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.283+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1879735350}) 2024-09-16T04:11:43.289 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.283+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2959771678}) 2024-09-16T04:11:43.291 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.287+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.291 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.287+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2269185340}) 2024-09-16T04:11:43.291 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.287+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=503302614}) 2024-09-16T04:11:43.291 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.287+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.291 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.287+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2269185340}) 2024-09-16T04:11:43.291 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.287+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=369410112}) 2024-09-16T04:11:43.293 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.291+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.294 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.291+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2119175323}) 2024-09-16T04:11:43.294 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.291+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=20981054}) 2024-09-16T04:11:43.294 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.291+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.294 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.291+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2119175323}) 2024-09-16T04:11:43.294 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.291+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3876692131}) 2024-09-16T04:11:43.296 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.291+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.296 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.291+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=308591366}) 2024-09-16T04:11:43.296 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.291+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1025583603}) 2024-09-16T04:11:43.296 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.291+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.296 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.291+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=308591366}) 2024-09-16T04:11:43.296 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.291+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2882397115}) 2024-09-16T04:11:43.298 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.295+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.298 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.295+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=697840420}) 2024-09-16T04:11:43.298 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.295+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1801615555}) 2024-09-16T04:11:43.299 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.295+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.299 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.295+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=697840420}) 2024-09-16T04:11:43.299 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.295+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2515036495}) 2024-09-16T04:11:43.301 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.295+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.301 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.295+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3680640297}) 2024-09-16T04:11:43.301 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.295+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1678171142}) 2024-09-16T04:11:43.301 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.295+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.301 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.295+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3680640297}) 2024-09-16T04:11:43.301 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.295+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2270971440}) 2024-09-16T04:11:43.303 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.299+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.303 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.299+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1898206395}) 2024-09-16T04:11:43.303 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.299+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=188031253}) 2024-09-16T04:11:43.303 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.299+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.303 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.299+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1898206395}) 2024-09-16T04:11:43.303 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.299+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=229309799}) 2024-09-16T04:11:43.305 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.299+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.305 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.299+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=477717387}) 2024-09-16T04:11:43.305 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.303+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2548620918}) 2024-09-16T04:11:43.305 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.303+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.306 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.303+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=477717387}) 2024-09-16T04:11:43.306 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.303+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2857737334}) 2024-09-16T04:11:43.307 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.303+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.307 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.303+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3103670492}) 2024-09-16T04:11:43.307 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.303+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=728221691}) 2024-09-16T04:11:43.307 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.303+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.308 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.303+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3103670492}) 2024-09-16T04:11:43.308 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.303+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=997200106}) 2024-09-16T04:11:43.310 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.307+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.310 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.307+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2061956067}) 2024-09-16T04:11:43.310 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.307+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1338339568}) 2024-09-16T04:11:43.310 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.307+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.310 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.307+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2061956067}) 2024-09-16T04:11:43.310 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.307+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2501441996}) 2024-09-16T04:11:43.312 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.307+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.312 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.307+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2455955565}) 2024-09-16T04:11:43.312 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.307+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3746926824}) 2024-09-16T04:11:43.312 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.307+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.312 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.307+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2455955565}) 2024-09-16T04:11:43.312 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.307+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=172507687}) 2024-09-16T04:11:43.314 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.311+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.314 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.311+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2436356412}) 2024-09-16T04:11:43.314 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.311+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=37757451}) 2024-09-16T04:11:43.315 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.311+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.315 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.311+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2436356412}) 2024-09-16T04:11:43.315 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.311+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2556510777}) 2024-09-16T04:11:43.316 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.311+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.317 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.311+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2302594341}) 2024-09-16T04:11:43.317 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.311+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=451153116}) 2024-09-16T04:11:43.317 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.311+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.317 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.311+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2302594341}) 2024-09-16T04:11:43.317 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.311+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=150177275}) 2024-09-16T04:11:43.321 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.315+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.322 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.315+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1365053619}) 2024-09-16T04:11:43.322 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.315+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2496708644}) 2024-09-16T04:11:43.322 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.315+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.322 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.315+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1365053619}) 2024-09-16T04:11:43.322 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.315+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3660403113}) 2024-09-16T04:11:43.322 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.319+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.322 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.319+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3124463516}) 2024-09-16T04:11:43.323 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.319+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2951444466}) 2024-09-16T04:11:43.323 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.319+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.323 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.319+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3124463516}) 2024-09-16T04:11:43.323 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.319+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1940328361}) 2024-09-16T04:11:43.324 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.319+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.324 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.319+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1602031385}) 2024-09-16T04:11:43.324 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.319+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2909424901}) 2024-09-16T04:11:43.324 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.319+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.324 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.319+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1602031385}) 2024-09-16T04:11:43.324 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.319+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1512680535}) 2024-09-16T04:11:43.326 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.323+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.326 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.323+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3403452707}) 2024-09-16T04:11:43.326 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.323+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3879913811}) 2024-09-16T04:11:43.326 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.323+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.326 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.323+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3403452707}) 2024-09-16T04:11:43.326 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.323+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3101115474}) 2024-09-16T04:11:43.328 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.323+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.328 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.323+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2922895261}) 2024-09-16T04:11:43.328 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.323+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2728632460}) 2024-09-16T04:11:43.329 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.323+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.329 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.323+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2922895261}) 2024-09-16T04:11:43.329 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.323+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1047112313}) 2024-09-16T04:11:43.331 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.327+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.331 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.327+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=729001253}) 2024-09-16T04:11:43.331 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.327+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=757472839}) 2024-09-16T04:11:43.331 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.327+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.331 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.327+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=729001253}) 2024-09-16T04:11:43.331 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.327+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3286023063}) 2024-09-16T04:11:43.333 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.327+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.333 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.327+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3526192404}) 2024-09-16T04:11:43.333 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.327+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3921205953}) 2024-09-16T04:11:43.333 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.327+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.333 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.327+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3526192404}) 2024-09-16T04:11:43.333 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.327+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1112411435}) 2024-09-16T04:11:43.335 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.331+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.335 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.331+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=77263277}) 2024-09-16T04:11:43.335 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.331+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2617635919}) 2024-09-16T04:11:43.335 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.331+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.335 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.331+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=77263277}) 2024-09-16T04:11:43.335 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.331+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4258786391}) 2024-09-16T04:11:43.337 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.335+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.337 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.335+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4097100585}) 2024-09-16T04:11:43.337 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.335+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3010796633}) 2024-09-16T04:11:43.338 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.335+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.338 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.335+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4097100585}) 2024-09-16T04:11:43.338 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.335+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3835539533}) 2024-09-16T04:11:43.340 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.335+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.340 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.335+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4232463758}) 2024-09-16T04:11:43.340 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.335+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3425139670}) 2024-09-16T04:11:43.340 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.335+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.340 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.335+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4232463758}) 2024-09-16T04:11:43.340 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.335+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=741521627}) 2024-09-16T04:11:43.342 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.339+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.342 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.339+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3233060147}) 2024-09-16T04:11:43.342 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.339+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2684006817}) 2024-09-16T04:11:43.342 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.339+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.342 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.339+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3233060147}) 2024-09-16T04:11:43.342 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.339+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1830501987}) 2024-09-16T04:11:43.344 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.339+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.344 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.339+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=616590996}) 2024-09-16T04:11:43.344 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.339+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1787652100}) 2024-09-16T04:11:43.344 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.339+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.344 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.339+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=616590996}) 2024-09-16T04:11:43.345 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.339+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=647465476}) 2024-09-16T04:11:43.346 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.343+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.347 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.343+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4019950774}) 2024-09-16T04:11:43.347 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.343+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3639294776}) 2024-09-16T04:11:43.347 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.343+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.347 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.343+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4019950774}) 2024-09-16T04:11:43.347 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.343+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3198921348}) 2024-09-16T04:11:43.349 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.343+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.349 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.343+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2015547346}) 2024-09-16T04:11:43.349 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.343+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=325423075}) 2024-09-16T04:11:43.350 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.343+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.350 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.343+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2015547346}) 2024-09-16T04:11:43.350 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.343+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2199540996}) 2024-09-16T04:11:43.351 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.347+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.351 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.347+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=521301698}) 2024-09-16T04:11:43.351 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.347+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1538282656}) 2024-09-16T04:11:43.351 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.347+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.352 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.347+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=521301698}) 2024-09-16T04:11:43.352 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.347+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=684217009}) 2024-09-16T04:11:43.353 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.347+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.353 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.351+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=9471346}) 2024-09-16T04:11:43.353 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.351+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2889552000}) 2024-09-16T04:11:43.353 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.351+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.353 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.351+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=9471346}) 2024-09-16T04:11:43.353 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.351+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1936629455}) 2024-09-16T04:11:43.355 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.351+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.355 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.351+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2826344661}) 2024-09-16T04:11:43.355 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.351+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2309315628}) 2024-09-16T04:11:43.355 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.351+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.356 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.351+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2826344661}) 2024-09-16T04:11:43.356 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.351+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2735356465}) 2024-09-16T04:11:43.357 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.355+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.358 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.355+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1312530212}) 2024-09-16T04:11:43.358 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.355+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1907185769}) 2024-09-16T04:11:43.358 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.355+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.358 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.355+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1312530212}) 2024-09-16T04:11:43.358 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.355+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1736822938}) 2024-09-16T04:11:43.360 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.355+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.360 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.355+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2516754555}) 2024-09-16T04:11:43.360 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.355+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1112189722}) 2024-09-16T04:11:43.360 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.355+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.360 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.355+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2516754555}) 2024-09-16T04:11:43.360 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.355+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=759277351}) 2024-09-16T04:11:43.362 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.359+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.362 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.359+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3902565732}) 2024-09-16T04:11:43.362 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.359+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2338393671}) 2024-09-16T04:11:43.362 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.359+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.362 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.359+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3902565732}) 2024-09-16T04:11:43.362 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.359+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2681836100}) 2024-09-16T04:11:43.364 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.359+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.364 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.359+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1843986693}) 2024-09-16T04:11:43.364 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.359+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2545756985}) 2024-09-16T04:11:43.364 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.359+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.365 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.359+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1843986693}) 2024-09-16T04:11:43.365 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.359+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4168436886}) 2024-09-16T04:11:43.366 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.363+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.366 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.363+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=323213837}) 2024-09-16T04:11:43.366 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.363+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1028112956}) 2024-09-16T04:11:43.366 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.363+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.366 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.363+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=323213837}) 2024-09-16T04:11:43.366 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.363+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1154419386}) 2024-09-16T04:11:43.368 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.363+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.368 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.363+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2694957250}) 2024-09-16T04:11:43.368 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.363+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=581214839}) 2024-09-16T04:11:43.368 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.363+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.368 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.363+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2694957250}) 2024-09-16T04:11:43.368 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.363+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2215074800}) 2024-09-16T04:11:43.370 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.367+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.370 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.367+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1679173073}) 2024-09-16T04:11:43.370 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.367+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=502208512}) 2024-09-16T04:11:43.370 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.367+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.370 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.367+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1679173073}) 2024-09-16T04:11:43.370 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.367+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4080081445}) 2024-09-16T04:11:43.372 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.367+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.372 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.367+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3299955836}) 2024-09-16T04:11:43.372 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.367+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2716596994}) 2024-09-16T04:11:43.372 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.367+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.372 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.367+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3299955836}) 2024-09-16T04:11:43.373 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.367+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1373569133}) 2024-09-16T04:11:43.374 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.371+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.374 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.371+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2378247582}) 2024-09-16T04:11:43.374 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.371+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3775700170}) 2024-09-16T04:11:43.374 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.371+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.374 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.371+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2378247582}) 2024-09-16T04:11:43.375 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.371+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4140676250}) 2024-09-16T04:11:43.376 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.371+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.376 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.371+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4068179515}) 2024-09-16T04:11:43.376 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.371+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1821559709}) 2024-09-16T04:11:43.377 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.371+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.377 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.371+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4068179515}) 2024-09-16T04:11:43.377 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.371+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1860638820}) 2024-09-16T04:11:43.378 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.375+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.378 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.375+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2992019319}) 2024-09-16T04:11:43.378 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.375+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1419076146}) 2024-09-16T04:11:43.379 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.375+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.379 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.375+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2992019319}) 2024-09-16T04:11:43.379 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.375+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=646253271}) 2024-09-16T04:11:43.380 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.375+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.380 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.375+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4013512630}) 2024-09-16T04:11:43.381 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.375+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=844147887}) 2024-09-16T04:11:43.381 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.375+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.381 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.375+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4013512630}) 2024-09-16T04:11:43.381 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.375+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=212773399}) 2024-09-16T04:11:43.382 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.379+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.383 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.379+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2488123373}) 2024-09-16T04:11:43.383 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.379+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=690120051}) 2024-09-16T04:11:43.383 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.379+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.383 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.379+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2488123373}) 2024-09-16T04:11:43.383 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.379+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=966422421}) 2024-09-16T04:11:43.384 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.379+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.384 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.379+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4094181941}) 2024-09-16T04:11:43.384 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.379+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1437803357}) 2024-09-16T04:11:43.385 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.379+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.385 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.379+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4094181941}) 2024-09-16T04:11:43.385 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.379+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2594397783}) 2024-09-16T04:11:43.386 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.383+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.387 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.383+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3545908011}) 2024-09-16T04:11:43.387 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.383+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1945129588}) 2024-09-16T04:11:43.387 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.383+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.387 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.383+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3545908011}) 2024-09-16T04:11:43.387 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.383+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3948277089}) 2024-09-16T04:11:43.389 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.383+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.389 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.383+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=943632307}) 2024-09-16T04:11:43.389 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.383+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1691554395}) 2024-09-16T04:11:43.389 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.383+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.389 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.383+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=943632307}) 2024-09-16T04:11:43.389 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.383+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1362421639}) 2024-09-16T04:11:43.391 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.387+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.391 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.387+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2709178221}) 2024-09-16T04:11:43.391 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.387+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3646215039}) 2024-09-16T04:11:43.391 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.387+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.391 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.387+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2709178221}) 2024-09-16T04:11:43.391 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.387+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2590930824}) 2024-09-16T04:11:43.393 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.387+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.393 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.387+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=439019148}) 2024-09-16T04:11:43.393 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.387+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2098347386}) 2024-09-16T04:11:43.393 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.387+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.393 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.391+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=439019148}) 2024-09-16T04:11:43.393 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.391+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3617217027}) 2024-09-16T04:11:43.395 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.391+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.395 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.391+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1585991176}) 2024-09-16T04:11:43.395 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.391+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=535485780}) 2024-09-16T04:11:43.395 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.391+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.395 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.391+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1585991176}) 2024-09-16T04:11:43.395 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.391+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1839285921}) 2024-09-16T04:11:43.397 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.395+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.397 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.395+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4102221770}) 2024-09-16T04:11:43.397 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.395+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2907118869}) 2024-09-16T04:11:43.397 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.395+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.398 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.395+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4102221770}) 2024-09-16T04:11:43.398 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.395+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=557756196}) 2024-09-16T04:11:43.399 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.395+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.399 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.395+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2347275850}) 2024-09-16T04:11:43.400 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.395+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1742980384}) 2024-09-16T04:11:43.400 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.395+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.400 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.395+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2347275850}) 2024-09-16T04:11:43.400 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.395+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1125865448}) 2024-09-16T04:11:43.402 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.399+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.402 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.399+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2171433323}) 2024-09-16T04:11:43.402 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.399+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2383865207}) 2024-09-16T04:11:43.402 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.399+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.402 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.399+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2171433323}) 2024-09-16T04:11:43.402 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.399+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=63589715}) 2024-09-16T04:11:43.404 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.399+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.404 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.399+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4219711424}) 2024-09-16T04:11:43.404 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.399+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=797098368}) 2024-09-16T04:11:43.404 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.399+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.404 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.399+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4219711424}) 2024-09-16T04:11:43.404 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.399+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=951570589}) 2024-09-16T04:11:43.406 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.403+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.406 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.403+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=608446559}) 2024-09-16T04:11:43.407 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.403+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1077271384}) 2024-09-16T04:11:43.407 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.403+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.407 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.403+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=608446559}) 2024-09-16T04:11:43.407 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.403+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3456692846}) 2024-09-16T04:11:43.408 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.403+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.408 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.403+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1066342191}) 2024-09-16T04:11:43.408 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.403+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2276351811}) 2024-09-16T04:11:43.409 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.403+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.409 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.403+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1066342191}) 2024-09-16T04:11:43.409 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.403+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1123871950}) 2024-09-16T04:11:43.410 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.407+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.410 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.407+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3146460724}) 2024-09-16T04:11:43.411 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.407+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=304590637}) 2024-09-16T04:11:43.411 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.407+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.411 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.407+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3146460724}) 2024-09-16T04:11:43.411 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.407+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3827805173}) 2024-09-16T04:11:43.412 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.407+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.413 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.407+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2341532448}) 2024-09-16T04:11:43.413 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.407+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3802731916}) 2024-09-16T04:11:43.413 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.407+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.413 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.407+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2341532448}) 2024-09-16T04:11:43.413 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.407+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3685461831}) 2024-09-16T04:11:43.415 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.411+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.415 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.411+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3785365523}) 2024-09-16T04:11:43.415 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.411+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=353420146}) 2024-09-16T04:11:43.415 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.411+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.415 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.411+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3785365523}) 2024-09-16T04:11:43.416 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.411+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3598604540}) 2024-09-16T04:11:43.417 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.411+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.417 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.411+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=163886336}) 2024-09-16T04:11:43.417 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.411+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=704886418}) 2024-09-16T04:11:43.417 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.411+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.417 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.411+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=163886336}) 2024-09-16T04:11:43.417 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.411+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=142326022}) 2024-09-16T04:11:43.419 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.415+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.419 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.415+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2798244164}) 2024-09-16T04:11:43.419 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.415+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2275393302}) 2024-09-16T04:11:43.419 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.415+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.419 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.415+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2798244164}) 2024-09-16T04:11:43.419 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.415+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4032686071}) 2024-09-16T04:11:43.421 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.415+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.421 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.415+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2240050458}) 2024-09-16T04:11:43.421 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.415+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=177293000}) 2024-09-16T04:11:43.421 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.415+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.421 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.415+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2240050458}) 2024-09-16T04:11:43.422 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.415+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=801849779}) 2024-09-16T04:11:43.423 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.419+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.423 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.419+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1495578668}) 2024-09-16T04:11:43.423 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.419+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1744200260}) 2024-09-16T04:11:43.423 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.419+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.423 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.419+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1495578668}) 2024-09-16T04:11:43.423 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.419+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3520017854}) 2024-09-16T04:11:43.425 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.419+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.425 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.419+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=287392371}) 2024-09-16T04:11:43.425 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.419+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4162951799}) 2024-09-16T04:11:43.425 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.419+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.425 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.419+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=287392371}) 2024-09-16T04:11:43.425 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.419+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3217273782}) 2024-09-16T04:11:43.427 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.423+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.427 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.423+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2293329530}) 2024-09-16T04:11:43.427 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.423+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1181155041}) 2024-09-16T04:11:43.427 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.423+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.427 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.423+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2293329530}) 2024-09-16T04:11:43.427 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.423+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1599878997}) 2024-09-16T04:11:43.429 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.423+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.429 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.423+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=627077581}) 2024-09-16T04:11:43.429 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.423+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2628594653}) 2024-09-16T04:11:43.429 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.427+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.429 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.427+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=627077581}) 2024-09-16T04:11:43.429 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.427+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3555580663}) 2024-09-16T04:11:43.431 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.427+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.431 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.427+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1020089350}) 2024-09-16T04:11:43.431 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.427+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3153262505}) 2024-09-16T04:11:43.431 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.427+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.431 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.427+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1020089350}) 2024-09-16T04:11:43.431 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.427+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2665805686}) 2024-09-16T04:11:43.433 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.431+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.433 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.431+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=941227601}) 2024-09-16T04:11:43.433 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.431+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=281201370}) 2024-09-16T04:11:43.433 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.431+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.434 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.431+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=941227601}) 2024-09-16T04:11:43.434 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.431+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1067525957}) 2024-09-16T04:11:43.435 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.431+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.435 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.431+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2568140397}) 2024-09-16T04:11:43.435 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.431+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4043893433}) 2024-09-16T04:11:43.435 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.431+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.436 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.431+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2568140397}) 2024-09-16T04:11:43.436 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.431+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2919514149}) 2024-09-16T04:11:43.437 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.435+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.437 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.435+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=268083373}) 2024-09-16T04:11:43.437 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.435+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=639799334}) 2024-09-16T04:11:43.437 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.435+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.438 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.435+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=268083373}) 2024-09-16T04:11:43.438 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.435+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3184379159}) 2024-09-16T04:11:43.439 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.435+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.439 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.435+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3742866985}) 2024-09-16T04:11:43.439 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.435+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4030976070}) 2024-09-16T04:11:43.440 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.435+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.440 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.435+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3742866985}) 2024-09-16T04:11:43.440 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.435+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3129748881}) 2024-09-16T04:11:43.441 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.439+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.441 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.439+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1479199739}) 2024-09-16T04:11:43.442 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.439+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2767390770}) 2024-09-16T04:11:43.442 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.439+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.442 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.439+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1479199739}) 2024-09-16T04:11:43.442 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.439+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=414732785}) 2024-09-16T04:11:43.443 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.439+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.444 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.439+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1900483479}) 2024-09-16T04:11:43.444 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.439+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2943601624}) 2024-09-16T04:11:43.444 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.439+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.444 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.439+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1900483479}) 2024-09-16T04:11:43.444 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.439+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=329443434}) 2024-09-16T04:11:43.446 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.443+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.446 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.443+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3047833526}) 2024-09-16T04:11:43.446 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.443+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3147237650}) 2024-09-16T04:11:43.446 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.443+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.446 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.443+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3047833526}) 2024-09-16T04:11:43.446 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.443+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3104522023}) 2024-09-16T04:11:43.448 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.443+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.448 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.443+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1823398730}) 2024-09-16T04:11:43.448 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.443+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2214845792}) 2024-09-16T04:11:43.448 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.443+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.448 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.443+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1823398730}) 2024-09-16T04:11:43.448 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.443+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2547692643}) 2024-09-16T04:11:43.450 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.447+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.450 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.447+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4039696823}) 2024-09-16T04:11:43.450 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.447+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4077519784}) 2024-09-16T04:11:43.450 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.447+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.450 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.447+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4039696823}) 2024-09-16T04:11:43.451 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.447+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=677294218}) 2024-09-16T04:11:43.452 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.447+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.452 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.447+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3200813013}) 2024-09-16T04:11:43.452 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.447+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1683629178}) 2024-09-16T04:11:43.452 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.447+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.452 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.447+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3200813013}) 2024-09-16T04:11:43.452 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.447+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2092787284}) 2024-09-16T04:11:43.453 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.451+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.454 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.451+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1999959890}) 2024-09-16T04:11:43.454 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.451+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3615927730}) 2024-09-16T04:11:43.454 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.451+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.454 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.451+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1999959890}) 2024-09-16T04:11:43.454 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.451+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=776263320}) 2024-09-16T04:11:43.455 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.451+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.456 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.451+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=45060534}) 2024-09-16T04:11:43.456 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.451+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3262052000}) 2024-09-16T04:11:43.456 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.451+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.456 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.451+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=45060534}) 2024-09-16T04:11:43.456 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.451+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=803444321}) 2024-09-16T04:11:43.457 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.455+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.457 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.455+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2526692630}) 2024-09-16T04:11:43.458 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.455+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4272847418}) 2024-09-16T04:11:43.458 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.455+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.458 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.455+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2526692630}) 2024-09-16T04:11:43.458 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.455+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2098880253}) 2024-09-16T04:11:43.459 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.455+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.459 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.455+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3908564397}) 2024-09-16T04:11:43.459 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.455+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=836120985}) 2024-09-16T04:11:43.459 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.455+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.460 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.455+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3908564397}) 2024-09-16T04:11:43.460 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.455+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2594225633}) 2024-09-16T04:11:43.461 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.459+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.461 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.459+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3403153647}) 2024-09-16T04:11:43.461 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.459+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=215409975}) 2024-09-16T04:11:43.461 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.459+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.462 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.459+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3403153647}) 2024-09-16T04:11:43.462 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.459+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1789042108}) 2024-09-16T04:11:43.463 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.459+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.463 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.459+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1197303754}) 2024-09-16T04:11:43.463 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.459+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1303834607}) 2024-09-16T04:11:43.463 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.459+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.463 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.459+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1197303754}) 2024-09-16T04:11:43.463 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.459+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4156688871}) 2024-09-16T04:11:43.465 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.463+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.465 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.463+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1568836643}) 2024-09-16T04:11:43.465 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.463+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=807622964}) 2024-09-16T04:11:43.465 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.463+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.465 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.463+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1568836643}) 2024-09-16T04:11:43.465 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.463+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2288734350}) 2024-09-16T04:11:43.467 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.463+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.467 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.463+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=964353493}) 2024-09-16T04:11:43.467 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.463+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=19320062}) 2024-09-16T04:11:43.467 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.463+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.468 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.463+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=964353493}) 2024-09-16T04:11:43.468 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.463+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=578147858}) 2024-09-16T04:11:43.469 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.467+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.469 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.467+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2400237213}) 2024-09-16T04:11:43.469 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.467+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1678434701}) 2024-09-16T04:11:43.469 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.467+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.469 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.467+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2400237213}) 2024-09-16T04:11:43.469 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.467+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3247899234}) 2024-09-16T04:11:43.471 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.467+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.471 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.467+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2172068637}) 2024-09-16T04:11:43.471 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.467+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=654383524}) 2024-09-16T04:11:43.471 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.467+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.471 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.467+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2172068637}) 2024-09-16T04:11:43.471 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.467+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2644008917}) 2024-09-16T04:11:43.473 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.467+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.473 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.467+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=175191435}) 2024-09-16T04:11:43.473 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.467+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3654444603}) 2024-09-16T04:11:43.473 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.467+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.473 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.467+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=175191435}) 2024-09-16T04:11:43.473 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.467+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=656015491}) 2024-09-16T04:11:43.474 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.471+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.475 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.471+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=159967006}) 2024-09-16T04:11:43.475 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.471+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=746229165}) 2024-09-16T04:11:43.475 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.471+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.475 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.471+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=159967006}) 2024-09-16T04:11:43.475 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.471+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3688696448}) 2024-09-16T04:11:43.476 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.471+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.477 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.471+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3155556726}) 2024-09-16T04:11:43.477 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.471+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=185100200}) 2024-09-16T04:11:43.477 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.471+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.477 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.471+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3155556726}) 2024-09-16T04:11:43.477 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.471+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3331788552}) 2024-09-16T04:11:43.478 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.475+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.478 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.475+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1146821993}) 2024-09-16T04:11:43.478 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.475+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3901413018}) 2024-09-16T04:11:43.479 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.475+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.479 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.475+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1146821993}) 2024-09-16T04:11:43.479 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.475+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3734510564}) 2024-09-16T04:11:43.480 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.475+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.480 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.475+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2443528499}) 2024-09-16T04:11:43.480 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.475+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3717175544}) 2024-09-16T04:11:43.481 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.475+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.481 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.475+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2443528499}) 2024-09-16T04:11:43.481 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.475+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=102397723}) 2024-09-16T04:11:43.483 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.479+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.483 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.479+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3699576954}) 2024-09-16T04:11:43.483 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.479+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4104661402}) 2024-09-16T04:11:43.483 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.479+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.483 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.479+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3699576954}) 2024-09-16T04:11:43.483 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.479+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=498750646}) 2024-09-16T04:11:43.484 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.479+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.484 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.479+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2058055144}) 2024-09-16T04:11:43.484 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.479+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3285290187}) 2024-09-16T04:11:43.485 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.479+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.485 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.479+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2058055144}) 2024-09-16T04:11:43.485 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.479+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4086377120}) 2024-09-16T04:11:43.486 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.483+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.486 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.483+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2781335303}) 2024-09-16T04:11:43.486 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.483+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3652101249}) 2024-09-16T04:11:43.486 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.483+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.486 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.483+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2781335303}) 2024-09-16T04:11:43.486 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.483+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3327421021}) 2024-09-16T04:11:43.488 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.483+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.488 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.483+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2887734911}) 2024-09-16T04:11:43.488 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.483+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2075834817}) 2024-09-16T04:11:43.488 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.483+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.488 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.483+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2887734911}) 2024-09-16T04:11:43.488 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.483+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1644802018}) 2024-09-16T04:11:43.490 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.487+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.490 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.487+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=865474399}) 2024-09-16T04:11:43.490 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.487+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2499353288}) 2024-09-16T04:11:43.490 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.487+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.490 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.487+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=865474399}) 2024-09-16T04:11:43.490 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.487+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=993182538}) 2024-09-16T04:11:43.492 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.487+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.492 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.487+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3444754894}) 2024-09-16T04:11:43.492 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.487+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1920513480}) 2024-09-16T04:11:43.492 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.487+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.492 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.487+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3444754894}) 2024-09-16T04:11:43.492 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.487+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4247507964}) 2024-09-16T04:11:43.494 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.491+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.494 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.491+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2611513899}) 2024-09-16T04:11:43.494 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.491+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4078373728}) 2024-09-16T04:11:43.494 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.491+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.494 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.491+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2611513899}) 2024-09-16T04:11:43.494 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.491+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4207388577}) 2024-09-16T04:11:43.495 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.491+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.495 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.491+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3369944443}) 2024-09-16T04:11:43.496 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.491+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1849258651}) 2024-09-16T04:11:43.496 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.491+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.496 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.491+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3369944443}) 2024-09-16T04:11:43.496 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.491+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1994527058}) 2024-09-16T04:11:43.499 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.495+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.499 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.495+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3798388834}) 2024-09-16T04:11:43.499 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.495+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=763887309}) 2024-09-16T04:11:43.499 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.495+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.499 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.495+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3798388834}) 2024-09-16T04:11:43.500 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.495+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1380855325}) 2024-09-16T04:11:43.501 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.499+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.501 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.499+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=998849993}) 2024-09-16T04:11:43.501 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.499+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2584887247}) 2024-09-16T04:11:43.501 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.499+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.501 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.499+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=998849993}) 2024-09-16T04:11:43.502 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.499+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1597482944}) 2024-09-16T04:11:43.503 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.499+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.503 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.499+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1336135007}) 2024-09-16T04:11:43.503 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.499+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=465484576}) 2024-09-16T04:11:43.503 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.499+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.503 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.499+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1336135007}) 2024-09-16T04:11:43.503 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.499+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1452205800}) 2024-09-16T04:11:43.505 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.499+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.505 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.499+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=381513175}) 2024-09-16T04:11:43.505 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.503+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=103727601}) 2024-09-16T04:11:43.505 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.503+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.505 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.503+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=381513175}) 2024-09-16T04:11:43.505 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.503+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1837821688}) 2024-09-16T04:11:43.507 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.503+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.507 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.503+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2094616807}) 2024-09-16T04:11:43.507 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.503+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2899573284}) 2024-09-16T04:11:43.507 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.503+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.507 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.503+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2094616807}) 2024-09-16T04:11:43.508 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.503+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3199545566}) 2024-09-16T04:11:43.509 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.503+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.509 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.507+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2719488787}) 2024-09-16T04:11:43.509 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.507+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=937812577}) 2024-09-16T04:11:43.509 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.507+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.509 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.507+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2719488787}) 2024-09-16T04:11:43.509 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.507+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3545629655}) 2024-09-16T04:11:43.511 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.507+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.511 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.507+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2692347054}) 2024-09-16T04:11:43.511 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.507+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1550602465}) 2024-09-16T04:11:43.511 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.507+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.511 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.507+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2692347054}) 2024-09-16T04:11:43.511 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.507+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1608926636}) 2024-09-16T04:11:43.513 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.507+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.513 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.511+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2722143325}) 2024-09-16T04:11:43.513 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.511+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2961571047}) 2024-09-16T04:11:43.513 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.511+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.513 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.511+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2722143325}) 2024-09-16T04:11:43.514 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.511+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1555315943}) 2024-09-16T04:11:43.515 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.511+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.515 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.511+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=141039465}) 2024-09-16T04:11:43.515 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.511+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2314066695}) 2024-09-16T04:11:43.515 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.511+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.515 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.511+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=141039465}) 2024-09-16T04:11:43.515 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.511+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3137750104}) 2024-09-16T04:11:43.517 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.515+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.517 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.515+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=62484081}) 2024-09-16T04:11:43.517 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.515+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2516195643}) 2024-09-16T04:11:43.517 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.515+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.517 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.515+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=62484081}) 2024-09-16T04:11:43.517 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.515+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4240503011}) 2024-09-16T04:11:43.519 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.515+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.519 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.515+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1615057071}) 2024-09-16T04:11:43.519 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.515+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3080916013}) 2024-09-16T04:11:43.519 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.515+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.519 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.515+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1615057071}) 2024-09-16T04:11:43.519 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.515+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2249638018}) 2024-09-16T04:11:43.521 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.515+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.521 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.519+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4091412896}) 2024-09-16T04:11:43.521 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.519+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1603729255}) 2024-09-16T04:11:43.521 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.519+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.521 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.519+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4091412896}) 2024-09-16T04:11:43.521 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.519+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4280916567}) 2024-09-16T04:11:43.523 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.519+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.523 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.519+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=653248842}) 2024-09-16T04:11:43.524 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.519+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1271345811}) 2024-09-16T04:11:43.524 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.519+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.524 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.519+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=653248842}) 2024-09-16T04:11:43.524 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.519+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=824875930}) 2024-09-16T04:11:43.529 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.523+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.529 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.523+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3693270441}) 2024-09-16T04:11:43.529 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.523+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3260489780}) 2024-09-16T04:11:43.529 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.523+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.529 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.523+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3693270441}) 2024-09-16T04:11:43.529 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.523+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2929853365}) 2024-09-16T04:11:43.529 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.523+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.530 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.523+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4171741573}) 2024-09-16T04:11:43.530 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.523+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=689544633}) 2024-09-16T04:11:43.530 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.523+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.530 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.523+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4171741573}) 2024-09-16T04:11:43.530 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.523+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4139381243}) 2024-09-16T04:11:43.531 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.527+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.531 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.527+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2590340114}) 2024-09-16T04:11:43.531 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.527+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1789014675}) 2024-09-16T04:11:43.531 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.527+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.531 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.527+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2590340114}) 2024-09-16T04:11:43.531 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.527+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1631468806}) 2024-09-16T04:11:43.532 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.527+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.532 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.527+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1022069244}) 2024-09-16T04:11:43.532 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.527+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4048781733}) 2024-09-16T04:11:43.532 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.527+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.532 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.527+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1022069244}) 2024-09-16T04:11:43.532 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.527+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2861035004}) 2024-09-16T04:11:43.533 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.531+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.533 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.531+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3236225866}) 2024-09-16T04:11:43.533 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.531+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1377187659}) 2024-09-16T04:11:43.533 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.531+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.533 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.531+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3236225866}) 2024-09-16T04:11:43.533 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.531+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=17583517}) 2024-09-16T04:11:43.535 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.531+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.535 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.531+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=210045240}) 2024-09-16T04:11:43.535 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.531+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3784802893}) 2024-09-16T04:11:43.535 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.531+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.535 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.531+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=210045240}) 2024-09-16T04:11:43.535 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.531+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3564186124}) 2024-09-16T04:11:43.537 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.531+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.537 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.531+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4177162953}) 2024-09-16T04:11:43.537 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.531+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1087588084}) 2024-09-16T04:11:43.537 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.531+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.537 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.535+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4177162953}) 2024-09-16T04:11:43.537 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.535+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=960755418}) 2024-09-16T04:11:43.539 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.535+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.539 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.535+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4227741532}) 2024-09-16T04:11:43.539 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.535+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1306978760}) 2024-09-16T04:11:43.539 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.535+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.539 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.535+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4227741532}) 2024-09-16T04:11:43.539 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.535+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2442574352}) 2024-09-16T04:11:43.541 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.539+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.541 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.539+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2716250570}) 2024-09-16T04:11:43.541 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.539+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=779000707}) 2024-09-16T04:11:43.541 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.539+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.541 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.539+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2716250570}) 2024-09-16T04:11:43.542 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.539+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1328238776}) 2024-09-16T04:11:43.544 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.539+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.544 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.539+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=74,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=1} crc {logm=2295776807,mdsmap=4088524775,mgr=390803354,mgr_command_descs=2107673600,mgr_metadata=1367272202,mgrstat=200907595,mon_config_key=4067097899}) 2024-09-16T04:11:43.544 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.539+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=76,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=12} crc {logm=1706216550,mdsmap=4088524775,mgr=2815798751,mgr_command_descs=2107673600,mgr_metadata=1367272202,mgrstat=3472593074}) 2024-09-16T04:11:43.544 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.539+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.544 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.539+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=74,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=1} crc {logm=2295776807,mdsmap=4088524775,mgr=390803354,mgr_command_descs=2107673600,mgr_metadata=1367272202,mgrstat=200907595,mon_config_key=4067097899}) 2024-09-16T04:11:43.544 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.539+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=76,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=12} crc {logm=2886993017,mdsmap=4088524775,mgr=1669371261,mgr_command_descs=2107673600,mgr_metadata=1367272202,mgrstat=2225833753}) 2024-09-16T04:11:43.546 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.543+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.546 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.543+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mon_config_key=10,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=41,osdmap=39} crc {mon_config_key=1439144890,monmap=3937513981,osd_metadata=4182959540,osd_pg_creating=783066558,osd_snap=3231807714,osdmap=3516837052}) 2024-09-16T04:11:43.546 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.543+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mon_config_key=10,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=40,osdmap=40} crc {mon_config_key=1439144890,monmap=3937513981,osd_metadata=4182959540,osd_pg_creating=783066558,osd_snap=2636090764,osdmap=2890761867}) 2024-09-16T04:11:43.546 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.543+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.546 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.543+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mon_config_key=10,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=41,osdmap=39} crc {mon_config_key=1439144890,monmap=3937513981,osd_metadata=4182959540,osd_pg_creating=783066558,osd_snap=3231807714,osdmap=3516837052}) 2024-09-16T04:11:43.546 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.543+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mon_config_key=10,monmap=2,osd_metadata=5,osd_pg_creating=1,osd_snap=42,osdmap=40} crc {mon_config_key=1439144890,monmap=268092033,osd_metadata=951161650,osd_pg_creating=783066558,osd_snap=3489321063,osdmap=3287844725}) 2024-09-16T04:11:43.548 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.543+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.548 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.543+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2905246426}) 2024-09-16T04:11:43.548 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.543+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2990989847}) 2024-09-16T04:11:43.549 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.543+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.549 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.543+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2905246426}) 2024-09-16T04:11:43.549 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.543+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=338138538}) 2024-09-16T04:11:43.551 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.547+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.551 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.547+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=474010423}) 2024-09-16T04:11:43.551 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.547+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1768377125}) 2024-09-16T04:11:43.551 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.547+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.551 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.547+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=474010423}) 2024-09-16T04:11:43.551 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.547+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3339551168}) 2024-09-16T04:11:43.555 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.551+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.555 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.551+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2837011394}) 2024-09-16T04:11:43.556 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.551+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2211061739}) 2024-09-16T04:11:43.556 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.551+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.556 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.551+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2837011394}) 2024-09-16T04:11:43.556 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.551+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=590781806}) 2024-09-16T04:11:43.568 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.563+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.568 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.563+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=40} crc {osdmap=2253443477}) 2024-09-16T04:11:43.568 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.563+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=40} crc {osdmap=2850937348}) 2024-09-16T04:11:43.568 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.563+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:11:43.568 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.563+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=40} crc {osdmap=2253443477}) 2024-09-16T04:11:43.568 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:11:43.563+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=38} crc {osdmap=4214546471}) 2024-09-16T04:11:44.278 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T04:11:44.279 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:11:44.411 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.407+0000 7feb513b2640 10 monclient: get_monmap_and_config 2024-09-16T04:11:44.411 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.407+0000 7feb513b2640 10 monclient: build_initial_monmap 2024-09-16T04:11:44.411 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.407+0000 7feb513b2640 10 monclient: monmap: 2024-09-16T04:11:44.411 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:11:44.411 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:11:44.411 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:11:44.409686+0000 2024-09-16T04:11:44.411 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:11:44.409686+0000 2024-09-16T04:11:44.411 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:11:44.411 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:11:44.411 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:11:44.412 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:11:44.412 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:11:44.412 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:11:44.412 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:11:44.412 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:11:44.414 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.407+0000 7feb513b2640 1 Processor -- start 2024-09-16T04:11:44.414 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.407+0000 7feb513b2640 1 -- start start 2024-09-16T04:11:44.414 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.407+0000 7feb513b2640 10 monclient: init 2024-09-16T04:11:44.414 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.407+0000 7feb513b2640 10 monclient: _reopen_session rank -1 2024-09-16T04:11:44.414 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.407+0000 7feb513b2640 10 monclient: _add_conns ranks=[2,4,0,1,3] 2024-09-16T04:11:44.414 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.407+0000 7feb513b2640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7feb4c0a39b0 0x7feb4c133b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:11:44.414 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.407+0000 7feb513b2640 10 monclient(hunting): picked mon.noname-b con 0x7feb4c0a39b0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:11:44.414 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.407+0000 7feb513b2640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7feb4c0a3d90 0x7feb4c137000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:11:44.414 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.407+0000 7feb513b2640 10 monclient(hunting): picked mon.noname-c con 0x7feb4c0a3d90 addr v2:172.21.15.154:3300/0 2024-09-16T04:11:44.415 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.407+0000 7feb513b2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7feb4c137540 addr v1:172.21.15.42:6789/0 2024-09-16T04:11:44.415 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.407+0000 7feb513b2640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feb4c09c140 con 0x7feb4c137540 2024-09-16T04:11:44.415 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.407+0000 7feb513b2640 10 monclient(hunting): start opening mon connection 2024-09-16T04:11:44.415 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.407+0000 7feb513b2640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7feb4c09c420 con 0x7feb4c0a3d90 2024-09-16T04:11:44.415 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.407+0000 7feb513b2640 10 monclient(hunting): start opening mon connection 2024-09-16T04:11:44.415 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.407+0000 7feb513b2640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7feb4c06dd60 con 0x7feb4c0a39b0 2024-09-16T04:11:44.415 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.407+0000 7feb513b2640 10 monclient(hunting): _renew_subs 2024-09-16T04:11:44.415 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.407+0000 7feb513b2640 10 monclient(hunting): authenticate will time out at 2751.379461s 2024-09-16T04:11:44.416 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.407+0000 7feb4affd640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7feb4c0a39b0 0x7feb4c133b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:11:44.416 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.407+0000 7feb4b7fe640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7feb4c137540 0x7feb4c13e390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:49790/0 (socket says 172.21.15.42:49790) 2024-09-16T04:11:44.416 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.407+0000 7feb4b7fe640 1 -- 172.21.15.42:0/4251186493 learned_addr learned my addr 172.21.15.42:0/4251186493 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T04:11:44.416 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.407+0000 7feb4affd640 10 monclient(hunting): get_auth_request con 0x7feb4c0a39b0 auth_method 0 2024-09-16T04:11:44.416 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.407+0000 7feb4affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:11:44.416 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.407+0000 7feb4affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:44.416 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.407+0000 7feb4affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:44.416 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.407+0000 7feb4affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:11:44.416 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.407+0000 7feb4affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:11:44.416 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.407+0000 7feb4affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:11:44.416 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.407+0000 7feb4a7fc640 1 --2- 172.21.15.42:0/4251186493 >> v2:172.21.15.154:3300/0 conn(0x7feb4c0a3d90 0x7feb4c137000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:11:44.416 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.407+0000 7feb4a7fc640 10 monclient(hunting): get_auth_request con 0x7feb4c0a3d90 auth_method 0 2024-09-16T04:11:44.416 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.407+0000 7feb4a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:11:44.417 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.407+0000 7feb4a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:44.417 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.407+0000 7feb4a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:44.417 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.411+0000 7feb4affd640 10 monclient(hunting): handle_auth_done global_id 1014533 payload 274 2024-09-16T04:11:44.417 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.411+0000 7feb4affd640 1 -- 172.21.15.42:0/4251186493 >> v2:172.21.15.154:3300/0 conn(0x7feb4c0a3d90 msgr2=0x7feb4c137000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:11:44.417 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.411+0000 7feb4affd640 1 --2- 172.21.15.42:0/4251186493 >> v2:172.21.15.154:3300/0 conn(0x7feb4c0a3d90 0x7feb4c137000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:11:44.417 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.411+0000 7feb4affd640 1 -- 172.21.15.42:0/4251186493 >> v1:172.21.15.42:6789/0 conn(0x7feb4c137540 legacy=0x7feb4c13e390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:44.417 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.411+0000 7feb4affd640 10 monclient: _finish_hunting 0 2024-09-16T04:11:44.417 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.411+0000 7feb4affd640 1 monclient: found mon.noname-b 2024-09-16T04:11:44.417 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.411+0000 7feb4affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:11:44.417 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.411+0000 7feb4affd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T04:11:44.417 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.411+0000 7feb49ffb640 1 -- 172.21.15.42:0/4251186493 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 101816720 0 0) 0x7feb4c09c140 con 0x7feb4c137540 2024-09-16T04:11:44.417 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.411+0000 7feb4affd640 1 -- 172.21.15.42:0/4251186493 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb4c0a5e20 con 0x7feb4c0a39b0 2024-09-16T04:11:44.417 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.411+0000 7feb49ffb640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T04:11:44.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.411+0000 7feb4affd640 1 --2- 172.21.15.42:0/4251186493 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7feb4c0a39b0 0x7feb4c133b80 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7feb400097c0 tx=0x7feb4001c360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=591faecaf99bc9b2 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:11:44.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.411+0000 7feb49ffb640 1 -- 172.21.15.42:0/4251186493 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb400b0070 con 0x7feb4c0a39b0 2024-09-16T04:11:44.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.411+0000 7feb49ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:44.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.411+0000 7feb49ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:11:44.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.411+0000 7feb49ffb640 10 monclient: dump: 2024-09-16T04:11:44.424 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:11:44.424 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:11:44.424 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:11:44.424 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:11:44.424 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:44.424 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:11:44.424 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:11:44.424 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:11:44.424 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:11:44.424 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:11:44.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.411+0000 7feb49ffb640 10 monclient: _finish_auth 0 2024-09-16T04:11:44.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.411+0000 7feb49ffb640 10 monclient: _check_auth_tickets 2024-09-16T04:11:44.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.411+0000 7feb49ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:11:44.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.411+0000 7feb49ffb640 1 -- 172.21.15.42:0/4251186493 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feb400ab600 con 0x7feb4c0a39b0 2024-09-16T04:11:44.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.411+0000 7feb49ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:11:44.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.411+0000 7feb49ffb640 1 -- 172.21.15.42:0/4251186493 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb400ab9d0 con 0x7feb4c0a39b0 2024-09-16T04:11:44.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.411+0000 7feb513b2640 5 monclient: authenticate success, global_id 1014533 2024-09-16T04:11:44.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.411+0000 7feb513b2640 10 monclient: get_monmap_and_config success 2024-09-16T04:11:44.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.411+0000 7feb513b2640 4 set_mon_vals no callback set 2024-09-16T04:11:44.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.411+0000 7feb49ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:44.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.411+0000 7feb49ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:11:44.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.411+0000 7feb49ffb640 10 monclient: dump: 2024-09-16T04:11:44.425 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:11:44.425 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:11:44.426 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:11:44.426 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:11:44.426 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:44.426 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:11:44.426 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:11:44.426 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:11:44.426 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:11:44.426 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:11:44.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.411+0000 7feb513b2640 10 monclient: shutdown 2024-09-16T04:11:44.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.411+0000 7feb513b2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:11:44.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.411+0000 7feb513b2640 1 -- 172.21.15.42:0/4251186493 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7feb4c0a39b0 msgr2=0x7feb4c133b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:44.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.411+0000 7feb513b2640 1 --2- 172.21.15.42:0/4251186493 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7feb4c0a39b0 0x7feb4c133b80 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7feb400097c0 tx=0x7feb4001c360 comp rx=0 tx=0).stop 2024-09-16T04:11:44.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.411+0000 7feb513b2640 1 -- 172.21.15.42:0/4251186493 shutdown_connections 2024-09-16T04:11:44.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.411+0000 7feb513b2640 1 --2- 172.21.15.42:0/4251186493 >> v2:172.21.15.154:3300/0 conn(0x7feb4c0a3d90 0x7feb4c137000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:11:44.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.411+0000 7feb513b2640 1 --2- 172.21.15.42:0/4251186493 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7feb4c0a39b0 0x7feb4c133b80 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:11:44.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.411+0000 7feb513b2640 1 -- 172.21.15.42:0/4251186493 >> 172.21.15.42:0/4251186493 conn(0x7feb4c0a9fb0 msgr2=0x7feb4c0aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:11:44.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.411+0000 7feb513b2640 1 -- 172.21.15.42:0/4251186493 shutdown_connections 2024-09-16T04:11:44.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.411+0000 7feb513b2640 1 -- 172.21.15.42:0/4251186493 wait complete. 2024-09-16T04:11:44.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.411+0000 7feb513b2640 10 monclient: build_initial_monmap 2024-09-16T04:11:44.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.411+0000 7feb513b2640 10 monclient: monmap: 2024-09-16T04:11:44.427 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:11:44.427 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:11:44.427 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:11:44.428 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:11:44.428 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:11:44.428 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:11:44.428 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:11:44.428 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:11:44.428 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:11:44.428 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:11:44.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.411+0000 7feb513b2640 1 librados: starting msgr at 2024-09-16T04:11:44.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.411+0000 7feb513b2640 1 librados: starting objecter 2024-09-16T04:11:44.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.411+0000 7feb513b2640 1 Processor -- start 2024-09-16T04:11:44.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.415+0000 7feb513b2640 1 -- start start 2024-09-16T04:11:44.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.415+0000 7feb513b2640 1 librados: setting wanted keys 2024-09-16T04:11:44.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.415+0000 7feb513b2640 1 librados: calling monclient init 2024-09-16T04:11:44.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.415+0000 7feb513b2640 10 monclient: init 2024-09-16T04:11:44.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.415+0000 7feb513b2640 10 monclient: _reopen_session rank -1 2024-09-16T04:11:44.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.415+0000 7feb513b2640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:11:44.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.415+0000 7feb513b2640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feb4c0a39b0 0x7feb4c12d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:11:44.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.415+0000 7feb513b2640 10 monclient(hunting): picked mon.noname-a con 0x7feb4c0a39b0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:11:44.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.415+0000 7feb513b2640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7feb4c0a3d90 0x7feb4c12d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:11:44.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.415+0000 7feb513b2640 10 monclient(hunting): picked mon.noname-c con 0x7feb4c0a3d90 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:11:44.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.415+0000 7feb513b2640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feb4c137540 0x7feb4c12dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:11:44.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.415+0000 7feb513b2640 10 monclient(hunting): picked mon.noname-b con 0x7feb4c137540 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:11:44.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.415+0000 7feb513b2640 10 monclient(hunting): start opening mon connection 2024-09-16T04:11:44.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.415+0000 7feb513b2640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7feb4c1424e0 con 0x7feb4c0a39b0 2024-09-16T04:11:44.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.415+0000 7feb513b2640 10 monclient(hunting): start opening mon connection 2024-09-16T04:11:44.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.415+0000 7feb513b2640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7feb4c142660 con 0x7feb4c137540 2024-09-16T04:11:44.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.415+0000 7feb513b2640 10 monclient(hunting): start opening mon connection 2024-09-16T04:11:44.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.415+0000 7feb513b2640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7feb4c13d7f0 con 0x7feb4c0a3d90 2024-09-16T04:11:44.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.415+0000 7feb513b2640 10 monclient(hunting): _renew_subs 2024-09-16T04:11:44.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.415+0000 7feb513b2640 10 monclient(hunting): authenticate will time out at 2751.384513s 2024-09-16T04:11:44.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.415+0000 7feb4affd640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feb4c0a39b0 0x7feb4c12d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:11:44.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.415+0000 7feb4a7fc640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7feb4c0a3d90 0x7feb4c12d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:11:44.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.415+0000 7feb4affd640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feb4c0a39b0 0x7feb4c12d2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:35422/0 (socket says 172.21.15.42:35422) 2024-09-16T04:11:44.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.415+0000 7feb4affd640 1 -- 172.21.15.42:0/3194644793 learned_addr learned my addr 172.21.15.42:0/3194644793 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:11:44.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.415+0000 7feb4affd640 10 monclient(hunting): get_auth_request con 0x7feb4c0a39b0 auth_method 0 2024-09-16T04:11:44.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.415+0000 7feb4a7fc640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7feb4c0a3d90 0x7feb4c12d810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:51644/0 (socket says 172.21.15.42:51644) 2024-09-16T04:11:44.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.415+0000 7feb4affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:11:44.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.415+0000 7feb4affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:44.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.415+0000 7feb4affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:44.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.415+0000 7feb4a7fc640 10 monclient(hunting): get_auth_request con 0x7feb4c0a3d90 auth_method 0 2024-09-16T04:11:44.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.415+0000 7feb4a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:11:44.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.415+0000 7feb4a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:44.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.415+0000 7feb4a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:44.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.415+0000 7feb4affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:11:44.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.415+0000 7feb4affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:11:44.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.415+0000 7feb4affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:11:44.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.415+0000 7feb4a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:11:44.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.415+0000 7feb4a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:11:44.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.415+0000 7feb4a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:11:44.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.415+0000 7feb4b7fe640 1 --2- 172.21.15.42:0/3194644793 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feb4c137540 0x7feb4c12dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:11:44.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.415+0000 7feb4b7fe640 10 monclient(hunting): get_auth_request con 0x7feb4c137540 auth_method 0 2024-09-16T04:11:44.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.415+0000 7feb4b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:11:44.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.415+0000 7feb4b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:44.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.415+0000 7feb4b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:44.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.415+0000 7feb4affd640 10 monclient(hunting): handle_auth_done global_id 1034442 payload 824 2024-09-16T04:11:44.432 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.415+0000 7feb4affd640 1 -- 172.21.15.42:0/3194644793 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7feb4c0a3d90 msgr2=0x7feb4c12d810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:44.432 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.415+0000 7feb4affd640 1 --2- 172.21.15.42:0/3194644793 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7feb4c0a3d90 0x7feb4c12d810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:11:44.432 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.415+0000 7feb4affd640 1 -- 172.21.15.42:0/3194644793 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feb4c137540 msgr2=0x7feb4c12dd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:44.432 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.415+0000 7feb4affd640 1 --2- 172.21.15.42:0/3194644793 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feb4c137540 0x7feb4c12dd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:11:44.432 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.415+0000 7feb4affd640 10 monclient: _finish_hunting 0 2024-09-16T04:11:44.432 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.415+0000 7feb4affd640 1 monclient: found mon.noname-a 2024-09-16T04:11:44.432 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.415+0000 7feb4affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:11:44.432 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.415+0000 7feb4affd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:11:44.432 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.415+0000 7feb4affd640 1 -- 172.21.15.42:0/3194644793 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb4c13cf90 con 0x7feb4c0a39b0 2024-09-16T04:11:44.432 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.415+0000 7feb4a7fc640 1 --2- 172.21.15.42:0/3194644793 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7feb4c0a3d90 0x7feb4c12d810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:11:44.432 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.415+0000 7feb4affd640 1 --2- 172.21.15.42:0/3194644793 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feb4c0a39b0 0x7feb4c12d2d0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7feb400ae040 tx=0x7feb40019100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:11:44.432 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.415+0000 7feb337fe640 10 client.?.objecter ms_handle_connect 0x7feb4c0a39b0 2024-09-16T04:11:44.432 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.415+0000 7feb337fe640 10 client.?.objecter resend_mon_ops 2024-09-16T04:11:44.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.415+0000 7feb337fe640 1 -- 172.21.15.42:0/3194644793 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb400b8030 con 0x7feb4c0a39b0 2024-09-16T04:11:44.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.415+0000 7feb337fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:44.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.415+0000 7feb337fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:11:44.433 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.415+0000 7feb337fe640 10 monclient: dump: 2024-09-16T04:11:44.433 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:11:44.433 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:11:44.433 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:11:44.433 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:11:44.433 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:44.433 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:11:44.433 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:11:44.433 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:11:44.433 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:11:44.433 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:11:44.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.415+0000 7feb337fe640 10 monclient: _finish_auth 0 2024-09-16T04:11:44.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.415+0000 7feb337fe640 10 monclient: _check_auth_tickets 2024-09-16T04:11:44.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.415+0000 7feb337fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:11:44.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.415+0000 7feb513b2640 5 monclient: authenticate success, global_id 1034442 2024-09-16T04:11:44.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.415+0000 7feb337fe640 1 -- 172.21.15.42:0/3194644793 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feb400ab400 con 0x7feb4c0a39b0 2024-09-16T04:11:44.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.415+0000 7feb337fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:11:44.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.415+0000 7feb337fe640 1 -- 172.21.15.42:0/3194644793 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb400b0040 con 0x7feb4c0a39b0 2024-09-16T04:11:44.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.415+0000 7feb337fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:44.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.415+0000 7feb337fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:11:44.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.415+0000 7feb337fe640 10 monclient: dump: 2024-09-16T04:11:44.434 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:11:44.434 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:11:44.434 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:11:44.434 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:11:44.435 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:44.435 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:11:44.435 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:11:44.435 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:11:44.435 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:11:44.435 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:11:44.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.415+0000 7feb497fa640 4 set_mon_vals no callback set 2024-09-16T04:11:44.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.415+0000 7feb513b2640 10 monclient: _renew_subs 2024-09-16T04:11:44.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.415+0000 7feb513b2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:11:44.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.415+0000 7feb513b2640 1 -- 172.21.15.42:0/3194644793 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb4c13dad0 con 0x7feb4c0a39b0 2024-09-16T04:11:44.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.415+0000 7feb513b2640 10 client.1034442.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:11:44.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.415+0000 7feb513b2640 10 monclient: _renew_subs 2024-09-16T04:11:44.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.415+0000 7feb513b2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:11:44.435 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.415+0000 7feb513b2640 1 -- 172.21.15.42:0/3194644793 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb4c1372b0 con 0x7feb4c0a39b0 2024-09-16T04:11:44.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.415+0000 7feb513b2640 1 librados: init done 2024-09-16T04:11:44.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.415+0000 7feb513b2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:44.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.415+0000 7feb513b2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:44.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.415+0000 7feb513b2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:11:44.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.415+0000 7feb513b2640 1 -- 172.21.15.42:0/3194644793 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feb4c133b80 con 0x7feb4c0a39b0 2024-09-16T04:11:44.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.415+0000 7feb337fe640 1 -- 172.21.15.42:0/3194644793 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7feb400bb070 con 0x7feb4c0a39b0 2024-09-16T04:11:44.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.415+0000 7feb337fe640 1 --2- 172.21.15.42:0/3194644793 >> v2:172.21.15.154:6812/254845342 conn(0x7feb38037810 0x7feb38039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:11:44.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.415+0000 7feb337fe640 1 -- 172.21.15.42:0/3194644793 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(706..706 src has 451..706) ==== 5040+0+0 (secure 0 0 0) 0x7feb400e2600 con 0x7feb4c0a39b0 2024-09-16T04:11:44.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.415+0000 7feb337fe640 10 client.1034442.objecter ms_dispatch 0x7feb4c000ba0 osd_map(706..706 src has 451..706) 2024-09-16T04:11:44.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.415+0000 7feb337fe640 3 client.1034442.objecter handle_osd_map got epochs [706,706] > 0 2024-09-16T04:11:44.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.415+0000 7feb337fe640 3 client.1034442.objecter handle_osd_map decoding full epoch 706 2024-09-16T04:11:44.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.415+0000 7feb337fe640 20 client.1034442.objecter dump_active .. 0 homeless 2024-09-16T04:11:44.436 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.415+0000 7feb4a7fc640 1 --2- 172.21.15.42:0/3194644793 >> v2:172.21.15.154:6812/254845342 conn(0x7feb38037810 0x7feb38039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:11:44.437 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.415+0000 7feb4a7fc640 10 monclient: get_auth_request con 0x7feb38037810 auth_method 0 2024-09-16T04:11:44.437 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.419+0000 7feb4a7fc640 1 --2- 172.21.15.42:0/3194644793 >> v2:172.21.15.154:6812/254845342 conn(0x7feb38037810 0x7feb38039cd0 secure :-1 s=READY pgs=3952 cs=0 l=1 rev1=1 crypto rx=0x7feb4c13f040 tx=0x7feb3c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:11:44.437 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.419+0000 7feb337fe640 10 client.1034442.objecter ms_handle_connect 0x7feb38037810 2024-09-16T04:11:44.437 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.423+0000 7feb337fe640 1 -- 172.21.15.42:0/3194644793 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7feb400c2070 con 0x7feb4c0a39b0 2024-09-16T04:11:44.437 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.423+0000 7feb337fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:44.437 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.423+0000 7feb337fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:11:44.652 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.647+0000 7feb513b2640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:11:44.652 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.647+0000 7feb513b2640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:11:44.653 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.647+0000 7feb513b2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:11:44.653 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.647+0000 7feb513b2640 1 -- 172.21.15.42:0/3194644793 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7feb4c0a5750 con 0x7feb4c0a39b0 2024-09-16T04:11:44.653 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.651+0000 7feb337fe640 1 -- 172.21.15.42:0/3194644793 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7feb40016070 con 0x7feb4c0a39b0 2024-09-16T04:11:44.653 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.651+0000 7feb337fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:11:44.653 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.651+0000 7feb337fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:11:44.653 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":268,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:11:44.655 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.651+0000 7feb513b2640 10 librados: watch_flush enter 2024-09-16T04:11:44.656 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.651+0000 7feb513b2640 10 librados: watch_flush exit 2024-09-16T04:11:44.656 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.651+0000 7feb513b2640 20 client.1034442.objecter shutdown clearing up homeless session... 2024-09-16T04:11:44.656 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.651+0000 7feb513b2640 10 client.1034442.objecter successfully canceled tick 2024-09-16T04:11:44.656 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.651+0000 7feb513b2640 1 -- 172.21.15.42:0/3194644793 >> v2:172.21.15.154:6812/254845342 conn(0x7feb38037810 msgr2=0x7feb38039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:44.656 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.651+0000 7feb513b2640 1 --2- 172.21.15.42:0/3194644793 >> v2:172.21.15.154:6812/254845342 conn(0x7feb38037810 0x7feb38039cd0 secure :-1 s=READY pgs=3952 cs=0 l=1 rev1=1 crypto rx=0x7feb4c13f040 tx=0x7feb3c008040 comp rx=0 tx=0).stop 2024-09-16T04:11:44.656 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.651+0000 7feb513b2640 10 monclient: shutdown 2024-09-16T04:11:44.656 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.651+0000 7feb513b2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:11:44.656 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.651+0000 7feb513b2640 1 -- 172.21.15.42:0/3194644793 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feb4c0a39b0 msgr2=0x7feb4c12d2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:44.656 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.651+0000 7feb513b2640 1 --2- 172.21.15.42:0/3194644793 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feb4c0a39b0 0x7feb4c12d2d0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7feb400ae040 tx=0x7feb40019100 comp rx=0 tx=0).stop 2024-09-16T04:11:44.656 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.651+0000 7feb513b2640 1 -- 172.21.15.42:0/3194644793 shutdown_connections 2024-09-16T04:11:44.656 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.651+0000 7feb513b2640 1 --2- 172.21.15.42:0/3194644793 >> v2:172.21.15.154:6812/254845342 conn(0x7feb38037810 0x7feb38039cd0 unknown :-1 s=CLOSED pgs=3952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:11:44.656 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.651+0000 7feb513b2640 1 --2- 172.21.15.42:0/3194644793 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feb4c137540 0x7feb4c12dd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:11:44.656 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.651+0000 7feb513b2640 1 --2- 172.21.15.42:0/3194644793 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7feb4c0a3d90 0x7feb4c12d810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:11:44.657 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.651+0000 7feb513b2640 1 --2- 172.21.15.42:0/3194644793 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feb4c0a39b0 0x7feb4c12d2d0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:11:44.657 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.651+0000 7feb513b2640 1 -- 172.21.15.42:0/3194644793 >> 172.21.15.42:0/3194644793 conn(0x7feb4c0a9fb0 msgr2=0x7feb4c0a87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:11:44.657 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.651+0000 7feb513b2640 1 -- 172.21.15.42:0/3194644793 shutdown_connections 2024-09-16T04:11:44.657 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.651+0000 7feb513b2640 1 -- 172.21.15.42:0/3194644793 wait complete. 2024-09-16T04:11:44.657 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.651+0000 7feb513b2640 1 librados: shutdown 2024-09-16T04:11:44.672 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T04:11:44.672 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T04:11:44.672 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T04:11:44.672 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T04:11:44.798 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.795+0000 7f88a44f5640 10 monclient: get_monmap_and_config 2024-09-16T04:11:44.798 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.795+0000 7f88a44f5640 10 monclient: build_initial_monmap 2024-09-16T04:11:44.798 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.795+0000 7f88a44f5640 10 monclient: monmap: 2024-09-16T04:11:44.798 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:11:44.799 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:11:44.799 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:11:44.797214+0000 2024-09-16T04:11:44.799 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:11:44.797214+0000 2024-09-16T04:11:44.799 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:11:44.799 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:11:44.799 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:11:44.800 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:11:44.800 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:11:44.801 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:11:44.802 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:11:44.803 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:11:44.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.795+0000 7f88a44f5640 1 Processor -- start 2024-09-16T04:11:44.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.795+0000 7f88a44f5640 1 -- start start 2024-09-16T04:11:44.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.795+0000 7f88a44f5640 10 monclient: init 2024-09-16T04:11:44.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.795+0000 7f88a44f5640 10 monclient: _reopen_session rank -1 2024-09-16T04:11:44.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.795+0000 7f88a44f5640 10 monclient: _add_conns ranks=[0,4,2,3,1] 2024-09-16T04:11:44.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.795+0000 7f88a44f5640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f889c0a1330 addr v1:172.21.15.42:6789/0 2024-09-16T04:11:44.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.795+0000 7f88a44f5640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f889c06dac0 0x7f889c13a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:11:44.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.795+0000 7f88a44f5640 10 monclient(hunting): picked mon.noname-c con 0x7f889c06dac0 addr v2:172.21.15.154:3300/0 2024-09-16T04:11:44.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.795+0000 7f88a44f5640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f889c0a1e20 0x7f889c13c890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:11:44.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.795+0000 7f88a44f5640 10 monclient(hunting): picked mon.noname-b con 0x7f889c0a1e20 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:11:44.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.795+0000 7f88a44f5640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f889c09c090 con 0x7f889c0a1330 2024-09-16T04:11:44.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.795+0000 7f88a44f5640 10 monclient(hunting): start opening mon connection 2024-09-16T04:11:44.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.795+0000 7f88a44f5640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f889c09c890 con 0x7f889c06dac0 2024-09-16T04:11:44.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.795+0000 7f88a44f5640 10 monclient(hunting): start opening mon connection 2024-09-16T04:11:44.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.795+0000 7f88a44f5640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f889c09f610 con 0x7f889c0a1e20 2024-09-16T04:11:44.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.795+0000 7f88a44f5640 10 monclient(hunting): _renew_subs 2024-09-16T04:11:44.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.795+0000 7f88a44f5640 10 monclient(hunting): authenticate will time out at 2751.766634s 2024-09-16T04:11:44.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.795+0000 7f88a2291640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f889c0a1330 0x7f889c0a1710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:49796/0 (socket says 172.21.15.42:49796) 2024-09-16T04:11:44.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.795+0000 7f88a2a92640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f889c0a1e20 0x7f889c13c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:11:44.813 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.795+0000 7f88a2291640 1 -- 172.21.15.42:0/3980779844 learned_addr learned my addr 172.21.15.42:0/3980779844 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T04:11:44.813 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.795+0000 7f88a2a92640 10 monclient(hunting): get_auth_request con 0x7f889c0a1e20 auth_method 0 2024-09-16T04:11:44.813 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.795+0000 7f88a2a92640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:11:44.813 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.795+0000 7f88a2a92640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:44.813 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.795+0000 7f88a2a92640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:44.813 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.795+0000 7f88a2a92640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:11:44.813 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.795+0000 7f88a2a92640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:11:44.813 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.795+0000 7f88a2a92640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:11:44.813 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.795+0000 7f88a1a90640 1 --2- 172.21.15.42:0/3980779844 >> v2:172.21.15.154:3300/0 conn(0x7f889c06dac0 0x7f889c13a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:11:44.813 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.795+0000 7f88a2a92640 10 monclient(hunting): handle_auth_done global_id 1014539 payload 274 2024-09-16T04:11:44.813 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.795+0000 7f88a2a92640 1 -- 172.21.15.42:0/3980779844 >> v2:172.21.15.154:3300/0 conn(0x7f889c06dac0 msgr2=0x7f889c13a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:11:44.813 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.795+0000 7f88a2a92640 1 --2- 172.21.15.42:0/3980779844 >> v2:172.21.15.154:3300/0 conn(0x7f889c06dac0 0x7f889c13a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:11:44.813 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.795+0000 7f88a2a92640 1 -- 172.21.15.42:0/3980779844 >> v1:172.21.15.42:6789/0 conn(0x7f889c0a1330 legacy=0x7f889c0a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:44.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.795+0000 7f88a2a92640 10 monclient: _finish_hunting 0 2024-09-16T04:11:44.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.795+0000 7f88a2a92640 1 monclient: found mon.noname-b 2024-09-16T04:11:44.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.795+0000 7f88a2a92640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:11:44.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.795+0000 7f88a128f640 1 -- 172.21.15.42:0/3980779844 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1432123216 0 0) 0x7f889c09c090 con 0x7f889c0a1330 2024-09-16T04:11:44.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.795+0000 7f88a2a92640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T04:11:44.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.795+0000 7f88a2a92640 1 -- 172.21.15.42:0/3980779844 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f889c09f8f0 con 0x7f889c0a1e20 2024-09-16T04:11:44.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.795+0000 7f88a1a90640 10 monclient: get_auth_request con 0x7f889c06dac0 auth_method 0 2024-09-16T04:11:44.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.795+0000 7f88a1a90640 1 --2- 172.21.15.42:0/3980779844 >> v2:172.21.15.154:3300/0 conn(0x7f889c06dac0 0x7f889c13a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T04:11:44.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.795+0000 7f88a128f640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T04:11:44.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.799+0000 7f88a2a92640 1 --2- 172.21.15.42:0/3980779844 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f889c0a1e20 0x7f889c13c890 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f88940097c0 tx=0x7f889401c3c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6763ad4d64807333 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:11:44.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.799+0000 7f88a128f640 1 -- 172.21.15.42:0/3980779844 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88940b1070 con 0x7f889c0a1e20 2024-09-16T04:11:44.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.799+0000 7f88a128f640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:44.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.799+0000 7f88a128f640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:11:44.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.799+0000 7f88a128f640 10 monclient: dump: 2024-09-16T04:11:44.815 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:11:44.815 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:11:44.815 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:11:44.815 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:11:44.815 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:44.815 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:11:44.815 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:11:44.815 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:11:44.815 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:11:44.815 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:11:44.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.799+0000 7f88a128f640 10 monclient: _finish_auth 0 2024-09-16T04:11:44.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.799+0000 7f88a128f640 10 monclient: _check_auth_tickets 2024-09-16T04:11:44.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.799+0000 7f88a128f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:11:44.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.799+0000 7f88a128f640 1 -- 172.21.15.42:0/3980779844 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f88940a8050 con 0x7f889c0a1e20 2024-09-16T04:11:44.816 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.799+0000 7f88a128f640 10 monclient: handle_config config(0 keys) 2024-09-16T04:11:44.816 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.799+0000 7f88a44f5640 5 monclient: authenticate success, global_id 1014539 2024-09-16T04:11:44.816 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.799+0000 7f88a44f5640 10 monclient: get_monmap_and_config success 2024-09-16T04:11:44.816 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.799+0000 7f88a44f5640 4 set_mon_vals no callback set 2024-09-16T04:11:44.816 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.799+0000 7f88a128f640 1 -- 172.21.15.42:0/3980779844 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88940ac860 con 0x7f889c0a1e20 2024-09-16T04:11:44.816 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.799+0000 7f88a128f640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:44.816 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.799+0000 7f88a128f640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:11:44.816 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.799+0000 7f88a128f640 10 monclient: dump: 2024-09-16T04:11:44.816 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:11:44.816 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:11:44.816 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:11:44.816 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:11:44.816 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:44.817 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:11:44.817 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:11:44.817 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:11:44.817 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:11:44.817 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:11:44.817 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.799+0000 7f88a44f5640 10 monclient: shutdown 2024-09-16T04:11:44.817 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.799+0000 7f88a44f5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:11:44.817 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.799+0000 7f88a44f5640 1 -- 172.21.15.42:0/3980779844 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f889c0a1e20 msgr2=0x7f889c13c890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:44.817 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.799+0000 7f88a44f5640 1 --2- 172.21.15.42:0/3980779844 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f889c0a1e20 0x7f889c13c890 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f88940097c0 tx=0x7f889401c3c0 comp rx=0 tx=0).stop 2024-09-16T04:11:44.817 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.799+0000 7f88a44f5640 1 -- 172.21.15.42:0/3980779844 shutdown_connections 2024-09-16T04:11:44.817 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.799+0000 7f88a44f5640 1 --2- 172.21.15.42:0/3980779844 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f889c0a1e20 0x7f889c13c890 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:11:44.817 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.799+0000 7f88a44f5640 1 --2- 172.21.15.42:0/3980779844 >> v2:172.21.15.154:3300/0 conn(0x7f889c06dac0 0x7f889c13a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:11:44.817 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.799+0000 7f88a44f5640 1 -- 172.21.15.42:0/3980779844 >> 172.21.15.42:0/3980779844 conn(0x7f889c0a2420 msgr2=0x7f889c0a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:11:44.818 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.799+0000 7f88a44f5640 1 -- 172.21.15.42:0/3980779844 shutdown_connections 2024-09-16T04:11:44.818 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.799+0000 7f88a44f5640 1 -- 172.21.15.42:0/3980779844 wait complete. 2024-09-16T04:11:44.818 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.799+0000 7f88a44f5640 10 monclient: build_initial_monmap 2024-09-16T04:11:44.818 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.799+0000 7f88a44f5640 10 monclient: monmap: 2024-09-16T04:11:44.818 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:11:44.818 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:11:44.818 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:11:44.818 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:11:44.818 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:11:44.818 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:11:44.818 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:11:44.818 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:11:44.818 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:11:44.818 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:11:44.819 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.799+0000 7f88a44f5640 1 librados: starting msgr at 2024-09-16T04:11:44.819 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.799+0000 7f88a44f5640 1 librados: starting objecter 2024-09-16T04:11:44.819 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.799+0000 7f88a44f5640 1 Processor -- start 2024-09-16T04:11:44.819 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.799+0000 7f88a44f5640 1 -- start start 2024-09-16T04:11:44.819 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.799+0000 7f88a44f5640 1 librados: setting wanted keys 2024-09-16T04:11:44.819 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.799+0000 7f88a44f5640 1 librados: calling monclient init 2024-09-16T04:11:44.819 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.799+0000 7f88a44f5640 10 monclient: init 2024-09-16T04:11:44.819 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.799+0000 7f88a44f5640 10 monclient: _reopen_session rank -1 2024-09-16T04:11:44.819 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.799+0000 7f88a44f5640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:11:44.819 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.799+0000 7f88a44f5640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f889c0a1330 0x7f889c1cebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:11:44.819 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.799+0000 7f88a44f5640 10 monclient(hunting): picked mon.noname-b con 0x7f889c0a1330 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:11:44.819 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.799+0000 7f88a44f5640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f889c0a1e20 0x7f889c135430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:11:44.819 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.799+0000 7f88a44f5640 10 monclient(hunting): picked mon.noname-c con 0x7f889c0a1e20 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:11:44.819 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.799+0000 7f88a44f5640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f889c135970 0x7f889c09cb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:11:44.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.799+0000 7f88a44f5640 10 monclient(hunting): picked mon.noname-a con 0x7f889c135970 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:11:44.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.799+0000 7f88a44f5640 10 monclient(hunting): start opening mon connection 2024-09-16T04:11:44.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.799+0000 7f88a44f5640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f889c13f5f0 con 0x7f889c135970 2024-09-16T04:11:44.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.799+0000 7f88a44f5640 10 monclient(hunting): start opening mon connection 2024-09-16T04:11:44.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.799+0000 7f88a44f5640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f889c09efb0 con 0x7f889c0a1330 2024-09-16T04:11:44.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.799+0000 7f88a44f5640 10 monclient(hunting): start opening mon connection 2024-09-16T04:11:44.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.799+0000 7f88a44f5640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f889c13f770 con 0x7f889c0a1e20 2024-09-16T04:11:44.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.799+0000 7f88a44f5640 10 monclient(hunting): _renew_subs 2024-09-16T04:11:44.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.799+0000 7f88a44f5640 10 monclient(hunting): authenticate will time out at 2751.771915s 2024-09-16T04:11:44.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.803+0000 7f88a2a92640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f889c135970 0x7f889c09cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:11:44.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.803+0000 7f88a1a90640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f889c0a1e20 0x7f889c135430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:11:44.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.803+0000 7f88a2a92640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f889c135970 0x7f889c09cb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:35434/0 (socket says 172.21.15.42:35434) 2024-09-16T04:11:44.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.803+0000 7f88a2a92640 1 -- 172.21.15.42:0/2794854556 learned_addr learned my addr 172.21.15.42:0/2794854556 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:11:44.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.803+0000 7f88a2a92640 10 monclient(hunting): get_auth_request con 0x7f889c135970 auth_method 0 2024-09-16T04:11:44.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.803+0000 7f88a2a92640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:11:44.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.803+0000 7f88a2a92640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:44.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.803+0000 7f88a2a92640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:44.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.803+0000 7f88a1a90640 10 monclient(hunting): get_auth_request con 0x7f889c0a1e20 auth_method 0 2024-09-16T04:11:44.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.803+0000 7f88a1a90640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:11:44.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.803+0000 7f88a1a90640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:44.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.803+0000 7f88a1a90640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:44.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.803+0000 7f88a2a92640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:11:44.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.803+0000 7f88a2a92640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:11:44.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.803+0000 7f88a2a92640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:11:44.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.803+0000 7f88a1a90640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:11:44.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.803+0000 7f88a1a90640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:11:44.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.803+0000 7f88a1a90640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:11:44.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.803+0000 7f88a2291640 1 --2- 172.21.15.42:0/2794854556 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f889c0a1330 0x7f889c1cebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:11:44.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.803+0000 7f88a2291640 10 monclient(hunting): get_auth_request con 0x7f889c0a1330 auth_method 0 2024-09-16T04:11:44.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.803+0000 7f88a2291640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:11:44.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.803+0000 7f88a2291640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:44.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.803+0000 7f88a2291640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:44.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.803+0000 7f88a2a92640 10 monclient(hunting): handle_auth_done global_id 1034448 payload 824 2024-09-16T04:11:44.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.803+0000 7f88a2a92640 1 -- 172.21.15.42:0/2794854556 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f889c0a1e20 msgr2=0x7f889c135430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:44.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.803+0000 7f88a2a92640 1 --2- 172.21.15.42:0/2794854556 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f889c0a1e20 0x7f889c135430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:11:44.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.803+0000 7f88a2a92640 1 -- 172.21.15.42:0/2794854556 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f889c0a1330 msgr2=0x7f889c1cebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:44.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.803+0000 7f88a2a92640 1 --2- 172.21.15.42:0/2794854556 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f889c0a1330 0x7f889c1cebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:11:44.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.803+0000 7f88a2a92640 10 monclient: _finish_hunting 0 2024-09-16T04:11:44.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.803+0000 7f88a2a92640 1 monclient: found mon.noname-a 2024-09-16T04:11:44.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.803+0000 7f88a2a92640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:11:44.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.803+0000 7f88a2a92640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:11:44.823 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.803+0000 7f88a2a92640 1 -- 172.21.15.42:0/2794854556 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f889c121de0 con 0x7f889c135970 2024-09-16T04:11:44.823 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.803+0000 7f88a1a90640 1 --2- 172.21.15.42:0/2794854556 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f889c0a1e20 0x7f889c135430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:11:44.823 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.803+0000 7f88a2a92640 1 --2- 172.21.15.42:0/2794854556 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f889c135970 0x7f889c09cb60 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f88940097c0 tx=0x7f8894019160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:11:44.823 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.803+0000 7f889affd640 10 client.?.objecter ms_handle_connect 0x7f889c135970 2024-09-16T04:11:44.823 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.803+0000 7f889affd640 10 client.?.objecter resend_mon_ops 2024-09-16T04:11:44.823 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.803+0000 7f889affd640 1 -- 172.21.15.42:0/2794854556 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88940b1040 con 0x7f889c135970 2024-09-16T04:11:44.823 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.803+0000 7f889affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:44.823 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.803+0000 7f889affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:11:44.823 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.803+0000 7f889affd640 10 monclient: dump: 2024-09-16T04:11:44.823 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:11:44.823 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:11:44.823 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:11:44.823 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:11:44.823 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:44.824 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:11:44.824 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:11:44.824 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:11:44.824 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:11:44.824 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:11:44.824 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.803+0000 7f889affd640 10 monclient: _finish_auth 0 2024-09-16T04:11:44.824 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.803+0000 7f889affd640 10 monclient: _check_auth_tickets 2024-09-16T04:11:44.824 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.803+0000 7f889affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:11:44.824 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.803+0000 7f88a44f5640 5 monclient: authenticate success, global_id 1034448 2024-09-16T04:11:44.824 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.803+0000 7f889affd640 1 -- 172.21.15.42:0/2794854556 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8894004030 con 0x7f889c135970 2024-09-16T04:11:44.824 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.803+0000 7f889affd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:11:44.824 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.803+0000 7f889affd640 1 -- 172.21.15.42:0/2794854556 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8894004400 con 0x7f889c135970 2024-09-16T04:11:44.824 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.803+0000 7f889affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:44.825 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.803+0000 7f889affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:11:44.825 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.803+0000 7f88a0a8e640 4 set_mon_vals no callback set 2024-09-16T04:11:44.825 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.803+0000 7f889affd640 10 monclient: dump: 2024-09-16T04:11:44.825 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:11:44.825 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:11:44.825 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:11:44.825 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:11:44.825 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:44.825 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:11:44.825 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:11:44.825 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:11:44.825 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:11:44.825 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:11:44.825 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.803+0000 7f88a44f5640 10 monclient: _renew_subs 2024-09-16T04:11:44.826 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.803+0000 7f88a44f5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:11:44.826 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.803+0000 7f88a44f5640 1 -- 172.21.15.42:0/2794854556 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f889c06e3a0 con 0x7f889c135970 2024-09-16T04:11:44.826 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.803+0000 7f88a44f5640 10 client.1034448.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:11:44.826 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.803+0000 7f88a44f5640 10 monclient: _renew_subs 2024-09-16T04:11:44.826 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.803+0000 7f88a44f5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:11:44.826 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.803+0000 7f88a44f5640 1 -- 172.21.15.42:0/2794854556 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f889c073a30 con 0x7f889c135970 2024-09-16T04:11:44.826 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.803+0000 7f88a44f5640 1 librados: init done 2024-09-16T04:11:44.826 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.803+0000 7f88a44f5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:44.826 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.803+0000 7f88a44f5640 1 --2- 172.21.15.42:0/2794854556 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f889c06dac0 0x7f889c02c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:11:44.826 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.803+0000 7f88a44f5640 10 monclient(hunting): start opening mon connection 2024-09-16T04:11:44.826 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.803+0000 7f88a44f5640 1 -- 172.21.15.42:0/2794854556 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f889c0a1710 con 0x7f889c06dac0 2024-09-16T04:11:44.826 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.803+0000 7f88a2291640 1 --2- 172.21.15.42:0/2794854556 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f889c06dac0 0x7f889c02c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:11:44.826 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.803+0000 7f88a2291640 10 monclient: get_auth_request con 0x7f889c06dac0 auth_method 0 2024-09-16T04:11:44.826 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.803+0000 7f88a2291640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:11:44.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.803+0000 7f88a2291640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:44.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.803+0000 7f88a2291640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:44.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.803+0000 7f889affd640 1 -- 172.21.15.42:0/2794854556 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f88940b4070 con 0x7f889c135970 2024-09-16T04:11:44.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.803+0000 7f88a2291640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:11:44.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.803+0000 7f88a2291640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:11:44.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.803+0000 7f88a2291640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:11:44.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.803+0000 7f889affd640 1 --2- 172.21.15.42:0/2794854556 >> v2:172.21.15.154:6812/254845342 conn(0x7f887c037840 0x7f887c039d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:11:44.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.803+0000 7f88a2291640 10 monclient(hunting): handle_auth_done global_id 1034451 payload 824 2024-09-16T04:11:44.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.803+0000 7f88a2291640 1 -- 172.21.15.42:0/2794854556 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f889c121540 con 0x7f889c06dac0 2024-09-16T04:11:44.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.803+0000 7f889affd640 1 -- 172.21.15.42:0/2794854556 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(706..706 src has 451..706) ==== 5040+0+0 (secure 0 0 0) 0x7f88940e2f00 con 0x7f889c135970 2024-09-16T04:11:44.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.803+0000 7f889affd640 10 client.1034448.objecter ms_dispatch 0x7f889c000ba0 osd_map(706..706 src has 451..706) 2024-09-16T04:11:44.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.803+0000 7f889affd640 3 client.1034448.objecter handle_osd_map got epochs [706,706] > 0 2024-09-16T04:11:44.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.803+0000 7f889affd640 3 client.1034448.objecter handle_osd_map decoding full epoch 706 2024-09-16T04:11:44.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.803+0000 7f88a2291640 1 --2- 172.21.15.42:0/2794854556 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f889c06dac0 0x7f889c02c650 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f888c0011f0 tx=0x7f888c001980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:11:44.828 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.803+0000 7f889affd640 20 client.1034448.objecter dump_active .. 0 homeless 2024-09-16T04:11:44.828 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.803+0000 7f889affd640 10 client.1034448.objecter ms_handle_connect 0x7f889c06dac0 2024-09-16T04:11:44.828 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.803+0000 7f889affd640 10 client.1034448.objecter resend_mon_ops 2024-09-16T04:11:44.828 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.803+0000 7f889affd640 1 -- 172.21.15.42:0/2794854556 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f888c012590 con 0x7f889c06dac0 2024-09-16T04:11:44.828 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.803+0000 7f889affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:44.828 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.803+0000 7f889affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:11:44.828 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.803+0000 7f889affd640 10 monclient: dump: 2024-09-16T04:11:44.828 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:11:44.828 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:11:44.828 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:11:44.828 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:11:44.828 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:44.828 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:11:44.828 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:11:44.829 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:11:44.829 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:11:44.829 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:11:44.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.803+0000 7f88a1a90640 1 --2- 172.21.15.42:0/2794854556 >> v2:172.21.15.154:6812/254845342 conn(0x7f887c037840 0x7f887c039d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:11:44.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.803+0000 7f88a1a90640 10 monclient: get_auth_request con 0x7f887c037840 auth_method 0 2024-09-16T04:11:44.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.803+0000 7f889affd640 1 -- 172.21.15.42:0/2794854556 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f888c018070 con 0x7f889c06dac0 2024-09-16T04:11:44.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.803+0000 7f889affd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:44.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.803+0000 7f889affd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:11:44.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.803+0000 7f889affd640 1 -- 172.21.15.42:0/2794854556 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f889c06dac0 msgr2=0x7f889c02c650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:44.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.803+0000 7f889affd640 1 --2- 172.21.15.42:0/2794854556 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f889c06dac0 0x7f889c02c650 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f888c0011f0 tx=0x7f888c001980 comp rx=0 tx=0).stop 2024-09-16T04:11:44.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.803+0000 7f889affd640 1 -- 172.21.15.42:0/2794854556 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f889c06dac0 msgr2=0x7f889c02c650 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:11:44.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.803+0000 7f889affd640 1 --2- 172.21.15.42:0/2794854556 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f889c06dac0 0x7f889c02c650 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:11:44.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.803+0000 7f88a1a90640 1 --2- 172.21.15.42:0/2794854556 >> v2:172.21.15.154:6812/254845342 conn(0x7f887c037840 0x7f887c039d00 secure :-1 s=READY pgs=3953 cs=0 l=1 rev1=1 crypto rx=0x7f889c120c70 tx=0x7f8884007810 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:11:44.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.803+0000 7f889affd640 10 client.1034448.objecter ms_handle_connect 0x7f887c037840 2024-09-16T04:11:44.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.811+0000 7f88a44f5640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:11:44.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.811+0000 7f88a44f5640 1 --2- 172.21.15.42:0/2794854556 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f889c02cb90 0x7f889c02cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:11:44.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.811+0000 7f88a44f5640 10 monclient(hunting): start opening mon connection 2024-09-16T04:11:44.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.811+0000 7f88a44f5640 1 -- 172.21.15.42:0/2794854556 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f889c138a40 con 0x7f889c02cb90 2024-09-16T04:11:44.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.811+0000 7f88a2291640 1 --2- 172.21.15.42:0/2794854556 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f889c02cb90 0x7f889c02cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:11:44.832 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.811+0000 7f88a2291640 10 monclient: get_auth_request con 0x7f889c02cb90 auth_method 0 2024-09-16T04:11:44.832 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.811+0000 7f88a2291640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:11:44.832 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.811+0000 7f88a2291640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:44.832 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.811+0000 7f88a2291640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:44.832 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.811+0000 7f88a2291640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:11:44.832 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.811+0000 7f88a2291640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:11:44.832 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.811+0000 7f88a2291640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:11:44.832 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.811+0000 7f88a2291640 10 monclient(hunting): handle_auth_done global_id 1034454 payload 824 2024-09-16T04:11:44.832 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.811+0000 7f88a2291640 1 -- 172.21.15.42:0/2794854556 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f889c06f060 con 0x7f889c02cb90 2024-09-16T04:11:44.832 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.811+0000 7f88a2291640 1 --2- 172.21.15.42:0/2794854556 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f889c02cb90 0x7f889c02cfa0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f888c0011f0 tx=0x7f888c001090 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:11:44.832 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.811+0000 7f889affd640 10 client.1034448.objecter ms_handle_connect 0x7f889c02cb90 2024-09-16T04:11:44.832 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.811+0000 7f889affd640 10 client.1034448.objecter resend_mon_ops 2024-09-16T04:11:44.832 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.811+0000 7f889affd640 1 -- 172.21.15.42:0/2794854556 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f888c012e90 con 0x7f889c02cb90 2024-09-16T04:11:44.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.811+0000 7f889affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:44.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.811+0000 7f889affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:11:44.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.811+0000 7f889affd640 10 monclient: dump: 2024-09-16T04:11:44.833 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:11:44.833 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:11:44.833 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:11:44.833 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:11:44.833 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:44.833 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:11:44.833 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:11:44.833 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:11:44.833 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:11:44.833 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:11:44.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.811+0000 7f889affd640 1 -- 172.21.15.42:0/2794854556 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4882 (secure 0 0 0) 0x7f888c012e90 con 0x7f889c02cb90 2024-09-16T04:11:44.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.811+0000 7f889affd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:11:44.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.811+0000 7f889affd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:11:44.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.811+0000 7f889affd640 1 -- 172.21.15.42:0/2794854556 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f889c02cb90 msgr2=0x7f889c02cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:44.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.811+0000 7f889affd640 1 --2- 172.21.15.42:0/2794854556 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f889c02cb90 0x7f889c02cfa0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f888c0011f0 tx=0x7f888c001090 comp rx=0 tx=0).stop 2024-09-16T04:11:44.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.811+0000 7f889affd640 1 -- 172.21.15.42:0/2794854556 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f889c02cb90 msgr2=0x7f889c02cfa0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:11:44.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.811+0000 7f889affd640 1 --2- 172.21.15.42:0/2794854556 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f889c02cb90 0x7f889c02cfa0 secure :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f888c0011f0 tx=0x7f888c001090 comp rx=0 tx=0).stop 2024-09-16T04:11:44.834 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T04:11:44.834 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:11:44.834 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:11:44.835 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "leader", 2024-09-16T04:11:44.835 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 268, 2024-09-16T04:11:44.835 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 184746, 2024-09-16T04:11:44.835 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T04:11:44.835 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T04:11:44.835 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T04:11:44.835 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T04:11:44.835 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:11:44.835 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 3, 2024-09-16T04:11:44.835 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:11:44.835 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T04:11:44.835 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T04:11:44.835 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:11:44.836 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:11:44.836 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:11:44.836 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:11:44.836 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:11:44.836 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:11:44.836 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:11:44.836 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:11:44.836 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:11:44.836 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:11:44.836 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:11:44.836 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:11:44.836 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:11:44.836 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T04:11:44.836 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:11:44.837 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:11:44.837 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:11:44.837 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:11:44.837 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:11:44.837 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:11:44.837 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:11:44.837 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:11:44.837 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:11:44.837 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:11:44.837 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:11:44.837 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:11:44.837 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:11:44.837 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T04:11:44.838 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T04:11:44.838 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T04:11:44.838 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T04:11:44.838 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T04:11:44.838 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T04:11:44.838 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:11:44.838 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:11:44.838 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T04:11:44.838 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T04:11:44.838 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T04:11:44.838 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T04:11:44.838 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T04:11:44.838 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T04:11:44.839 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T04:11:44.839 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:11:44.839 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T04:11:44.839 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:11:44.839 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:11:44.839 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:11:44.839 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:11:44.839 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:11:44.839 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:11:44.839 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:11:44.839 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:11:44.839 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:11:44.839 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:11:44.839 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:11:44.840 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:11:44.840 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T04:11:44.840 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:11:44.840 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T04:11:44.840 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:11:44.840 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:11:44.840 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:11:44.840 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:11:44.840 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:11:44.840 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:11:44.840 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:11:44.840 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T04:11:44.840 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:11:44.841 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:11:44.841 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:11:44.841 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:11:44.841 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T04:11:44.841 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:11:44.841 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:11:44.841 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:11:44.841 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:11:44.841 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T04:11:44.841 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T04:11:44.841 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:11:44.841 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:11:44.841 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:11:44.841 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:11:44.842 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:11:44.842 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:11:44.842 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:11:44.842 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:11:44.842 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:11:44.842 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:11:44.842 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:11:44.842 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T04:11:44.842 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:11:44.842 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:11:44.842 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:11:44.842 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:11:44.842 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T04:11:44.842 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:11:44.843 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:11:44.843 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:11:44.843 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:11:44.843 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T04:11:44.843 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T04:11:44.843 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:11:44.843 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:11:44.843 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:11:44.843 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:11:44.843 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:11:44.843 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:11:44.843 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:11:44.843 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:11:44.844 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:11:44.844 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:11:44.844 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:11:44.844 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T04:11:44.844 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:11:44.844 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:11:44.844 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:11:44.844 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:11:44.844 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T04:11:44.844 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:11:44.844 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:11:44.844 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:11:44.844 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:11:44.844 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T04:11:44.845 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T04:11:44.845 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:11:44.845 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:11:44.845 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:11:44.845 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:11:44.845 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:11:44.845 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:11:44.845 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T04:11:44.845 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T04:11:44.845 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:11:44.845 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:11:44.846 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:11:44.846 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:11:44.846 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:11:44.846 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:11:44.846 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T04:11:44.846 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:11:44.846 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:11:44.846 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:11:44.846 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 2 2024-09-16T04:11:44.846 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:11:44.846 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:11:44.846 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T04:11:44.846 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:11:44.847 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:11:44.847 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:11:44.847 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 5 2024-09-16T04:11:44.847 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:11:44.847 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:11:44.847 INFO:teuthology.orchestra.run.smithi042.stdout: "mgr": [ 2024-09-16T04:11:44.847 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:11:44.847 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:11:44.847 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:11:44.847 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:11:44.847 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:11:44.847 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:11:44.847 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:11:44.847 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T04:11:44.847 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T04:11:44.849 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.811+0000 7f88a44f5640 10 librados: watch_flush enter 2024-09-16T04:11:44.849 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.811+0000 7f88a44f5640 10 librados: watch_flush exit 2024-09-16T04:11:44.849 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.811+0000 7f88a44f5640 20 client.1034448.objecter shutdown clearing up homeless session... 2024-09-16T04:11:44.849 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.811+0000 7f88a44f5640 10 client.1034448.objecter successfully canceled tick 2024-09-16T04:11:44.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.811+0000 7f88a44f5640 1 -- 172.21.15.42:0/2794854556 >> v2:172.21.15.154:6812/254845342 conn(0x7f887c037840 msgr2=0x7f887c039d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:44.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.811+0000 7f88a44f5640 1 --2- 172.21.15.42:0/2794854556 >> v2:172.21.15.154:6812/254845342 conn(0x7f887c037840 0x7f887c039d00 secure :-1 s=READY pgs=3953 cs=0 l=1 rev1=1 crypto rx=0x7f889c120c70 tx=0x7f8884007810 comp rx=0 tx=0).stop 2024-09-16T04:11:44.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.811+0000 7f88a44f5640 10 monclient: shutdown 2024-09-16T04:11:44.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.811+0000 7f88a44f5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:11:44.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.811+0000 7f88a44f5640 1 -- 172.21.15.42:0/2794854556 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f889c135970 msgr2=0x7f889c09cb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:44.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.811+0000 7f88a44f5640 1 --2- 172.21.15.42:0/2794854556 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f889c135970 0x7f889c09cb60 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f88940097c0 tx=0x7f8894019160 comp rx=0 tx=0).stop 2024-09-16T04:11:44.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.811+0000 7f88a2a92640 1 -- 172.21.15.42:0/2794854556 reap_dead start 2024-09-16T04:11:44.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.811+0000 7f88a2a92640 1 -- 172.21.15.42:0/2794854556 reap_dead start 2024-09-16T04:11:44.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.811+0000 7f88a44f5640 1 -- 172.21.15.42:0/2794854556 shutdown_connections 2024-09-16T04:11:44.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.811+0000 7f88a44f5640 1 -- 172.21.15.42:0/2794854556 >> 172.21.15.42:0/2794854556 conn(0x7f889c0a2420 msgr2=0x7f889c0a4ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:11:44.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.811+0000 7f88a44f5640 1 -- 172.21.15.42:0/2794854556 shutdown_connections 2024-09-16T04:11:44.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.811+0000 7f88a44f5640 1 -- 172.21.15.42:0/2794854556 wait complete. 2024-09-16T04:11:44.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.811+0000 7f88a44f5640 1 librados: shutdown 2024-09-16T04:11:44.851 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:11:44.851 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T04:11:44.851 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T04:11:44.968 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.963+0000 7f5c602fb640 10 monclient: get_monmap_and_config 2024-09-16T04:11:44.968 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.963+0000 7f5c602fb640 10 monclient: build_initial_monmap 2024-09-16T04:11:44.968 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.963+0000 7f5c602fb640 10 monclient: monmap: 2024-09-16T04:11:44.968 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:11:44.968 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:11:44.968 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:11:44.966760+0000 2024-09-16T04:11:44.968 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:11:44.966760+0000 2024-09-16T04:11:44.968 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:11:44.969 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:11:44.969 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:11:44.969 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:11:44.969 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:11:44.969 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:11:44.969 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:11:44.969 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:11:44.979 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.963+0000 7f5c602fb640 1 Processor -- start 2024-09-16T04:11:44.980 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.963+0000 7f5c602fb640 1 -- start start 2024-09-16T04:11:44.980 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.963+0000 7f5c602fb640 10 monclient: init 2024-09-16T04:11:44.980 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.967+0000 7f5c602fb640 10 monclient: _reopen_session rank -1 2024-09-16T04:11:44.981 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.967+0000 7f5c602fb640 10 monclient: _add_conns ranks=[3,2,4,0,1] 2024-09-16T04:11:44.981 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.967+0000 7f5c602fb640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f5c580a1330 0x7f5c580a1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:11:44.981 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.967+0000 7f5c602fb640 10 monclient(hunting): picked mon.noname-a con 0x7f5c580a1330 addr v2:172.21.15.42:3300/0 2024-09-16T04:11:44.981 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.967+0000 7f5c602fb640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5c5806dac0 0x7f5c580a1c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:11:44.981 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.967+0000 7f5c602fb640 10 monclient(hunting): picked mon.noname-b con 0x7f5c5806dac0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:11:44.981 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.967+0000 7f5c602fb640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f5c58135100 0x7f5c58135550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:11:44.981 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.967+0000 7f5c602fb640 10 monclient(hunting): picked mon.noname-c con 0x7f5c58135100 addr v2:172.21.15.154:3300/0 2024-09-16T04:11:44.981 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.967+0000 7f5c602fb640 10 monclient(hunting): start opening mon connection 2024-09-16T04:11:44.981 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.967+0000 7f5c602fb640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f5c5809c7d0 con 0x7f5c580a1330 2024-09-16T04:11:44.981 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.967+0000 7f5c602fb640 10 monclient(hunting): start opening mon connection 2024-09-16T04:11:44.981 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.967+0000 7f5c602fb640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f5c5809c090 con 0x7f5c58135100 2024-09-16T04:11:44.982 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.967+0000 7f5c602fb640 10 monclient(hunting): start opening mon connection 2024-09-16T04:11:44.982 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.967+0000 7f5c602fb640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f5c5809f610 con 0x7f5c5806dac0 2024-09-16T04:11:44.982 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.967+0000 7f5c602fb640 10 monclient(hunting): _renew_subs 2024-09-16T04:11:44.982 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.967+0000 7f5c602fb640 10 monclient(hunting): authenticate will time out at 2751.936188s 2024-09-16T04:11:44.982 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.967+0000 7f5c5e097640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f5c580a1330 0x7f5c580a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:11:44.982 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.967+0000 7f5c5d896640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5c5806dac0 0x7f5c580a1c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:11:44.982 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.967+0000 7f5c5e097640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f5c580a1330 0x7f5c580a1710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:35462/0 (socket says 172.21.15.42:35462) 2024-09-16T04:11:44.982 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.967+0000 7f5c5e097640 1 -- 172.21.15.42:0/3121364058 learned_addr learned my addr 172.21.15.42:0/3121364058 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:11:44.982 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.967+0000 7f5c5e097640 10 monclient(hunting): get_auth_request con 0x7f5c580a1330 auth_method 0 2024-09-16T04:11:44.982 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.967+0000 7f5c5e097640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:11:44.982 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.967+0000 7f5c5e097640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:44.982 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.967+0000 7f5c5e097640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:44.983 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.967+0000 7f5c5d896640 10 monclient(hunting): get_auth_request con 0x7f5c5806dac0 auth_method 0 2024-09-16T04:11:44.983 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.967+0000 7f5c5d896640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:11:44.983 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.967+0000 7f5c5d896640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:44.983 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.967+0000 7f5c5d896640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:44.983 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.967+0000 7f5c5e097640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:11:44.983 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.967+0000 7f5c5e097640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:11:44.983 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.967+0000 7f5c5e097640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:11:44.983 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.967+0000 7f5c5d896640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:11:44.983 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.967+0000 7f5c5d896640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:11:44.983 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.967+0000 7f5c5d896640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:11:44.983 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.967+0000 7f5c5e097640 10 monclient(hunting): handle_auth_done global_id 1034457 payload 274 2024-09-16T04:11:44.983 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.967+0000 7f5c5e097640 1 -- 172.21.15.42:0/3121364058 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5c5806dac0 msgr2=0x7f5c580a1c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:11:44.983 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.967+0000 7f5c5e097640 1 --2- 172.21.15.42:0/3121364058 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5c5806dac0 0x7f5c580a1c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:11:44.983 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.967+0000 7f5c5e898640 1 --2- 172.21.15.42:0/3121364058 >> v2:172.21.15.154:3300/0 conn(0x7f5c58135100 0x7f5c58135550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:11:44.984 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.967+0000 7f5c5e097640 1 -- 172.21.15.42:0/3121364058 >> v2:172.21.15.154:3300/0 conn(0x7f5c58135100 msgr2=0x7f5c58135550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:11:44.984 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.967+0000 7f5c5e097640 1 --2- 172.21.15.42:0/3121364058 >> v2:172.21.15.154:3300/0 conn(0x7f5c58135100 0x7f5c58135550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:11:44.984 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.967+0000 7f5c5e097640 10 monclient: _finish_hunting 0 2024-09-16T04:11:44.984 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.967+0000 7f5c5e097640 1 monclient: found mon.noname-a 2024-09-16T04:11:44.984 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.967+0000 7f5c5e097640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:11:44.984 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.967+0000 7f5c5e097640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:11:44.984 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.967+0000 7f5c5e097640 1 -- 172.21.15.42:0/3121364058 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c5809f8f0 con 0x7f5c580a1330 2024-09-16T04:11:44.984 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.967+0000 7f5c5e898640 10 monclient: get_auth_request con 0x7f5c58135100 auth_method 0 2024-09-16T04:11:44.984 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.967+0000 7f5c5e898640 1 --2- 172.21.15.42:0/3121364058 >> v2:172.21.15.154:3300/0 conn(0x7f5c58135100 0x7f5c58135550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T04:11:44.984 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.967+0000 7f5c5e097640 1 --2- 172.21.15.42:0/3121364058 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5c580a1330 0x7f5c580a1710 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f5c5809bb00 tx=0x7f5c4801c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ee8805fe8e7a3b3 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:11:44.984 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.967+0000 7f5c5d095640 1 -- 172.21.15.42:0/3121364058 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c480b0070 con 0x7f5c580a1330 2024-09-16T04:11:44.984 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.967+0000 7f5c5d095640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:44.984 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.967+0000 7f5c5d095640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:11:44.985 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.967+0000 7f5c5d095640 10 monclient: dump: 2024-09-16T04:11:44.985 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:11:44.985 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:11:44.985 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:11:44.985 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:11:44.985 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:44.985 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:11:44.985 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:11:44.985 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:11:44.985 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:11:44.986 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:11:44.986 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.967+0000 7f5c5d095640 10 monclient: _finish_auth 0 2024-09-16T04:11:44.986 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.967+0000 7f5c5d095640 10 monclient: _check_auth_tickets 2024-09-16T04:11:44.986 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.967+0000 7f5c5d095640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:11:44.986 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.967+0000 7f5c5d095640 1 -- 172.21.15.42:0/3121364058 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5c480ab600 con 0x7f5c580a1330 2024-09-16T04:11:44.986 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.967+0000 7f5c5d095640 10 monclient: handle_config config(0 keys) 2024-09-16T04:11:44.986 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.967+0000 7f5c5d095640 1 -- 172.21.15.42:0/3121364058 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c480ab9d0 con 0x7f5c580a1330 2024-09-16T04:11:44.986 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.967+0000 7f5c602fb640 5 monclient: authenticate success, global_id 1034457 2024-09-16T04:11:44.986 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.967+0000 7f5c602fb640 10 monclient: get_monmap_and_config success 2024-09-16T04:11:44.986 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.967+0000 7f5c602fb640 4 set_mon_vals no callback set 2024-09-16T04:11:44.986 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.967+0000 7f5c5d095640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:44.987 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.967+0000 7f5c5d095640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:11:44.987 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.967+0000 7f5c5d095640 10 monclient: dump: 2024-09-16T04:11:44.987 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:11:44.987 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:11:44.987 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:11:44.987 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:11:44.987 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:44.987 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:11:44.987 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:11:44.987 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:11:44.987 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:11:44.987 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:11:44.987 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.967+0000 7f5c602fb640 10 monclient: shutdown 2024-09-16T04:11:44.987 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.967+0000 7f5c602fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:11:44.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.967+0000 7f5c602fb640 1 -- 172.21.15.42:0/3121364058 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5c580a1330 msgr2=0x7f5c580a1710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:44.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.967+0000 7f5c602fb640 1 --2- 172.21.15.42:0/3121364058 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5c580a1330 0x7f5c580a1710 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f5c5809bb00 tx=0x7f5c4801c500 comp rx=0 tx=0).stop 2024-09-16T04:11:44.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.967+0000 7f5c602fb640 1 -- 172.21.15.42:0/3121364058 shutdown_connections 2024-09-16T04:11:44.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.967+0000 7f5c602fb640 1 --2- 172.21.15.42:0/3121364058 >> v2:172.21.15.154:3300/0 conn(0x7f5c58135100 0x7f5c58135550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:11:44.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.967+0000 7f5c602fb640 1 --2- 172.21.15.42:0/3121364058 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5c5806dac0 0x7f5c580a1c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:11:44.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.967+0000 7f5c602fb640 1 --2- 172.21.15.42:0/3121364058 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5c580a1330 0x7f5c580a1710 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:11:44.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.967+0000 7f5c602fb640 1 -- 172.21.15.42:0/3121364058 >> 172.21.15.42:0/3121364058 conn(0x7f5c580a2420 msgr2=0x7f5c580a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:11:44.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.967+0000 7f5c602fb640 1 -- 172.21.15.42:0/3121364058 shutdown_connections 2024-09-16T04:11:44.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.967+0000 7f5c602fb640 1 -- 172.21.15.42:0/3121364058 wait complete. 2024-09-16T04:11:44.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.971+0000 7f5c602fb640 10 monclient: build_initial_monmap 2024-09-16T04:11:44.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.971+0000 7f5c602fb640 10 monclient: monmap: 2024-09-16T04:11:44.989 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:11:44.989 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:11:44.989 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:11:44.989 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:11:44.989 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:11:44.989 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:11:44.989 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:11:44.989 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:11:44.989 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:11:44.989 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:11:44.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.971+0000 7f5c602fb640 1 librados: starting msgr at 2024-09-16T04:11:44.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.971+0000 7f5c602fb640 1 librados: starting objecter 2024-09-16T04:11:44.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.971+0000 7f5c602fb640 1 Processor -- start 2024-09-16T04:11:44.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.971+0000 7f5c602fb640 1 -- start start 2024-09-16T04:11:44.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.971+0000 7f5c602fb640 1 librados: setting wanted keys 2024-09-16T04:11:44.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.971+0000 7f5c602fb640 1 librados: calling monclient init 2024-09-16T04:11:44.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.971+0000 7f5c602fb640 10 monclient: init 2024-09-16T04:11:44.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.971+0000 7f5c602fb640 10 monclient: _reopen_session rank -1 2024-09-16T04:11:44.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.971+0000 7f5c602fb640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:11:44.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.971+0000 7f5c602fb640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5c580a1330 0x7f5c581ccf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:11:44.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.971+0000 7f5c602fb640 10 monclient(hunting): picked mon.noname-b con 0x7f5c580a1330 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:11:44.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.971+0000 7f5c602fb640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5c58135100 0x7f5c58134520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:11:44.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.971+0000 7f5c602fb640 10 monclient(hunting): picked mon.noname-c con 0x7f5c58135100 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:11:44.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.971+0000 7f5c602fb640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5c58134a60 0x7f5c5811bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:11:44.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.971+0000 7f5c602fb640 10 monclient(hunting): picked mon.noname-a con 0x7f5c58134a60 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:11:44.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.971+0000 7f5c602fb640 10 monclient(hunting): start opening mon connection 2024-09-16T04:11:44.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.971+0000 7f5c602fb640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f5c5809eef0 con 0x7f5c58134a60 2024-09-16T04:11:44.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.971+0000 7f5c602fb640 10 monclient(hunting): start opening mon connection 2024-09-16T04:11:44.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.971+0000 7f5c602fb640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f5c5809ed70 con 0x7f5c580a1330 2024-09-16T04:11:44.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.971+0000 7f5c602fb640 10 monclient(hunting): start opening mon connection 2024-09-16T04:11:44.992 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.971+0000 7f5c602fb640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f5c5813eb70 con 0x7f5c58135100 2024-09-16T04:11:44.992 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.971+0000 7f5c602fb640 10 monclient(hunting): _renew_subs 2024-09-16T04:11:44.992 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.971+0000 7f5c602fb640 10 monclient(hunting): authenticate will time out at 2751.941473s 2024-09-16T04:11:44.992 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.971+0000 7f5c5e898640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5c58134a60 0x7f5c5811bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:11:44.992 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.971+0000 7f5c5d896640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5c58135100 0x7f5c58134520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:11:44.992 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.971+0000 7f5c5e898640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5c58134a60 0x7f5c5811bf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:35478/0 (socket says 172.21.15.42:35478) 2024-09-16T04:11:44.992 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.971+0000 7f5c5e898640 1 -- 172.21.15.42:0/1429842977 learned_addr learned my addr 172.21.15.42:0/1429842977 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:11:44.992 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.971+0000 7f5c5e898640 10 monclient(hunting): get_auth_request con 0x7f5c58134a60 auth_method 0 2024-09-16T04:11:44.992 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.971+0000 7f5c5e898640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:11:44.992 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.971+0000 7f5c5e898640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:44.992 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.971+0000 7f5c5e898640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:44.992 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.971+0000 7f5c5d896640 10 monclient(hunting): get_auth_request con 0x7f5c58135100 auth_method 0 2024-09-16T04:11:44.992 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.971+0000 7f5c5d896640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:11:44.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.971+0000 7f5c5d896640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:44.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.971+0000 7f5c5d896640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:44.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.971+0000 7f5c5e898640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:11:44.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.971+0000 7f5c5e898640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:11:44.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.971+0000 7f5c5e898640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:11:44.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.971+0000 7f5c5d896640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:11:44.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.971+0000 7f5c5d896640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:11:44.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.971+0000 7f5c5d896640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:11:44.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.971+0000 7f5c5e097640 1 --2- 172.21.15.42:0/1429842977 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5c580a1330 0x7f5c581ccf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:11:44.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.971+0000 7f5c5e097640 10 monclient(hunting): get_auth_request con 0x7f5c580a1330 auth_method 0 2024-09-16T04:11:44.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.971+0000 7f5c5e097640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:11:44.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.971+0000 7f5c5e097640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:44.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.971+0000 7f5c5e097640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:44.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.971+0000 7f5c5d896640 10 monclient(hunting): handle_auth_done global_id 1014548 payload 824 2024-09-16T04:11:44.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.971+0000 7f5c5d896640 1 -- 172.21.15.42:0/1429842977 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5c580a1330 msgr2=0x7f5c581ccf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:44.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.971+0000 7f5c5d896640 1 --2- 172.21.15.42:0/1429842977 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5c580a1330 0x7f5c581ccf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:11:44.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.971+0000 7f5c5d896640 1 -- 172.21.15.42:0/1429842977 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5c58134a60 msgr2=0x7f5c5811bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:44.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.971+0000 7f5c5d896640 1 --2- 172.21.15.42:0/1429842977 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5c58134a60 0x7f5c5811bf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:11:44.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.971+0000 7f5c5d896640 10 monclient: _finish_hunting 0 2024-09-16T04:11:44.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.971+0000 7f5c5d896640 1 monclient: found mon.noname-c 2024-09-16T04:11:44.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.971+0000 7f5c5d896640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:11:44.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.971+0000 7f5c5d896640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T04:11:44.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.971+0000 7f5c5d896640 1 -- 172.21.15.42:0/1429842977 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c580a53b0 con 0x7f5c58135100 2024-09-16T04:11:44.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.971+0000 7f5c5e898640 1 --2- 172.21.15.42:0/1429842977 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5c58134a60 0x7f5c5811bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:11:44.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.971+0000 7f5c5d896640 1 --2- 172.21.15.42:0/1429842977 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5c58135100 0x7f5c58134520 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f5c5400e870 tx=0x7f5c5400ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:11:44.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.971+0000 7f5c4effd640 10 client.?.objecter ms_handle_connect 0x7f5c58135100 2024-09-16T04:11:44.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.971+0000 7f5c4effd640 10 client.?.objecter resend_mon_ops 2024-09-16T04:11:44.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.971+0000 7f5c4effd640 1 -- 172.21.15.42:0/1429842977 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c54017070 con 0x7f5c58135100 2024-09-16T04:11:44.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.971+0000 7f5c4effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:44.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.971+0000 7f5c4effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:11:44.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.971+0000 7f5c4effd640 10 monclient: dump: 2024-09-16T04:11:44.995 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:11:44.996 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:11:44.996 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:11:44.996 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:11:44.996 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:44.996 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:11:44.996 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:11:44.997 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:11:44.997 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:11:44.997 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:11:44.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.971+0000 7f5c4effd640 10 monclient: _finish_auth 0 2024-09-16T04:11:44.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.971+0000 7f5c4effd640 10 monclient: _check_auth_tickets 2024-09-16T04:11:44.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.971+0000 7f5c4effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:11:44.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.971+0000 7f5c4effd640 1 -- 172.21.15.42:0/1429842977 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5c5400c940 con 0x7f5c58135100 2024-09-16T04:11:44.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.971+0000 7f5c4effd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:11:44.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.971+0000 7f5c4effd640 1 -- 172.21.15.42:0/1429842977 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c5400ccf0 con 0x7f5c58135100 2024-09-16T04:11:44.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.971+0000 7f5c602fb640 5 monclient: authenticate success, global_id 1014548 2024-09-16T04:11:44.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.971+0000 7f5c602fb640 10 monclient: _renew_subs 2024-09-16T04:11:44.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.971+0000 7f5c602fb640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:11:44.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.971+0000 7f5c602fb640 1 -- 172.21.15.42:0/1429842977 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c5811f900 con 0x7f5c58135100 2024-09-16T04:11:44.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.971+0000 7f5c4effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:44.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.971+0000 7f5c4effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:11:44.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.971+0000 7f5c4effd640 10 monclient: dump: 2024-09-16T04:11:44.998 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:11:44.998 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:11:44.998 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:11:44.998 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:11:44.998 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:44.998 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:11:44.998 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:11:44.999 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:11:44.999 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:11:44.999 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:11:44.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.971+0000 7f5c4ffff640 4 set_mon_vals no callback set 2024-09-16T04:11:44.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.971+0000 7f5c602fb640 10 client.1014548.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:11:44.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.971+0000 7f5c602fb640 10 monclient: _renew_subs 2024-09-16T04:11:44.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.971+0000 7f5c602fb640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:11:44.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.971+0000 7f5c602fb640 1 -- 172.21.15.42:0/1429842977 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c5811fec0 con 0x7f5c58135100 2024-09-16T04:11:44.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.971+0000 7f5c602fb640 1 librados: init done 2024-09-16T04:11:44.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.971+0000 7f5c602fb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:44.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.971+0000 7f5c602fb640 1 --2- 172.21.15.42:0/1429842977 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5c5806dac0 0x7f5c5811c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:11:44.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.971+0000 7f5c602fb640 10 monclient(hunting): start opening mon connection 2024-09-16T04:11:44.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.971+0000 7f5c602fb640 1 -- 172.21.15.42:0/1429842977 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f5c58135550 con 0x7f5c5806dac0 2024-09-16T04:11:44.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.971+0000 7f5c5e097640 1 --2- 172.21.15.42:0/1429842977 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5c5806dac0 0x7f5c5811c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:11:45.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.971+0000 7f5c5e097640 10 monclient: get_auth_request con 0x7f5c5806dac0 auth_method 0 2024-09-16T04:11:45.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.971+0000 7f5c5e097640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:11:45.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.971+0000 7f5c5e097640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:45.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.971+0000 7f5c5e097640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:45.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.971+0000 7f5c4effd640 1 -- 172.21.15.42:0/1429842977 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f5c5401a070 con 0x7f5c58135100 2024-09-16T04:11:45.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.971+0000 7f5c5e097640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:11:45.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.971+0000 7f5c5e097640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:11:45.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.971+0000 7f5c5e097640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:11:45.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.971+0000 7f5c4effd640 1 --2- 172.21.15.42:0/1429842977 >> v2:172.21.15.154:6812/254845342 conn(0x7f5c3c037810 0x7f5c3c039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:11:45.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.971+0000 7f5c5e097640 10 monclient(hunting): handle_auth_done global_id 1014551 payload 824 2024-09-16T04:11:45.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.971+0000 7f5c5e097640 1 -- 172.21.15.42:0/1429842977 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5c5811e4b0 con 0x7f5c5806dac0 2024-09-16T04:11:45.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.971+0000 7f5c4effd640 1 -- 172.21.15.42:0/1429842977 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(706..706 src has 451..706) ==== 5040+0+0 (secure 0 0 0) 0x7f5c54048c00 con 0x7f5c58135100 2024-09-16T04:11:45.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.971+0000 7f5c4effd640 10 client.1014548.objecter ms_dispatch 0x7f5c58000ba0 osd_map(706..706 src has 451..706) 2024-09-16T04:11:45.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.971+0000 7f5c4effd640 3 client.1014548.objecter handle_osd_map got epochs [706,706] > 0 2024-09-16T04:11:45.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.971+0000 7f5c4effd640 3 client.1014548.objecter handle_osd_map decoding full epoch 706 2024-09-16T04:11:45.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.971+0000 7f5c4effd640 20 client.1014548.objecter dump_active .. 0 homeless 2024-09-16T04:11:45.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.971+0000 7f5c4effd640 10 client.1014548.objecter ms_handle_connect 0x7f5c5806dac0 2024-09-16T04:11:45.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.971+0000 7f5c4effd640 10 client.1014548.objecter resend_mon_ops 2024-09-16T04:11:45.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.971+0000 7f5c5e097640 1 --2- 172.21.15.42:0/1429842977 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5c5806dac0 0x7f5c5811c4b0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f5c5806add0 tx=0x7f5c480192a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:11:45.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.975+0000 7f5c5e898640 1 --2- 172.21.15.42:0/1429842977 >> v2:172.21.15.154:6812/254845342 conn(0x7f5c3c037810 0x7f5c3c039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:11:45.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.975+0000 7f5c5e898640 10 monclient: get_auth_request con 0x7f5c3c037810 auth_method 0 2024-09-16T04:11:45.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.975+0000 7f5c4effd640 1 -- 172.21.15.42:0/1429842977 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c480ba030 con 0x7f5c5806dac0 2024-09-16T04:11:45.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.975+0000 7f5c4effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:45.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.975+0000 7f5c4effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:11:45.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.975+0000 7f5c4effd640 10 monclient: dump: 2024-09-16T04:11:45.002 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:11:45.002 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:11:45.002 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:11:45.002 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:11:45.003 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:45.003 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:11:45.003 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:11:45.003 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:11:45.003 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:11:45.003 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:11:45.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.975+0000 7f5c4effd640 1 -- 172.21.15.42:0/1429842977 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f5c480aba70 con 0x7f5c5806dac0 2024-09-16T04:11:45.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.975+0000 7f5c4effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:45.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.975+0000 7f5c4effd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:11:45.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.975+0000 7f5c4effd640 1 -- 172.21.15.42:0/1429842977 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5c5806dac0 msgr2=0x7f5c5811c4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:45.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.975+0000 7f5c4effd640 1 --2- 172.21.15.42:0/1429842977 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5c5806dac0 0x7f5c5811c4b0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f5c5806add0 tx=0x7f5c480192a0 comp rx=0 tx=0).stop 2024-09-16T04:11:45.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.975+0000 7f5c4effd640 1 -- 172.21.15.42:0/1429842977 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5c5806dac0 msgr2=0x7f5c5811c4b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:11:45.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.975+0000 7f5c4effd640 1 --2- 172.21.15.42:0/1429842977 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5c5806dac0 0x7f5c5811c4b0 secure :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f5c5806add0 tx=0x7f5c480192a0 comp rx=0 tx=0).stop 2024-09-16T04:11:45.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.975+0000 7f5c5e898640 1 --2- 172.21.15.42:0/1429842977 >> v2:172.21.15.154:6812/254845342 conn(0x7f5c3c037810 0x7f5c3c039cd0 secure :-1 s=READY pgs=3954 cs=0 l=1 rev1=1 crypto rx=0x7f5c50007fd0 tx=0x7f5c50007840 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:11:45.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.975+0000 7f5c4effd640 10 client.1014548.objecter ms_handle_connect 0x7f5c3c037810 2024-09-16T04:11:45.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.979+0000 7f5c602fb640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:11:45.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.979+0000 7f5c602fb640 1 --2- 172.21.15.42:0/1429842977 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5c5811c9f0 0x7f5c58137330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:11:45.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.979+0000 7f5c602fb640 10 monclient(hunting): start opening mon connection 2024-09-16T04:11:45.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.979+0000 7f5c602fb640 1 -- 172.21.15.42:0/1429842977 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f5c5811cdd0 con 0x7f5c5811c9f0 2024-09-16T04:11:45.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.979+0000 7f5c5e097640 1 --2- 172.21.15.42:0/1429842977 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5c5811c9f0 0x7f5c58137330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:11:45.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.979+0000 7f5c5e097640 10 monclient: get_auth_request con 0x7f5c5811c9f0 auth_method 0 2024-09-16T04:11:45.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.979+0000 7f5c5e097640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:11:45.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.979+0000 7f5c5e097640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:45.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.979+0000 7f5c5e097640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:45.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.979+0000 7f5c5e097640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:11:45.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.979+0000 7f5c5e097640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:11:45.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.979+0000 7f5c5e097640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:11:45.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.979+0000 7f5c5e097640 10 monclient(hunting): handle_auth_done global_id 1014554 payload 824 2024-09-16T04:11:45.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.979+0000 7f5c5e097640 1 -- 172.21.15.42:0/1429842977 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f5c58069040 con 0x7f5c5811c9f0 2024-09-16T04:11:45.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.979+0000 7f5c4effd640 10 client.1014548.objecter ms_handle_connect 0x7f5c5811c9f0 2024-09-16T04:11:45.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.979+0000 7f5c5e097640 1 --2- 172.21.15.42:0/1429842977 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5c5811c9f0 0x7f5c58137330 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f5c4801cda0 tx=0x7f5c580a22d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:11:45.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.979+0000 7f5c4effd640 10 client.1014548.objecter resend_mon_ops 2024-09-16T04:11:45.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.979+0000 7f5c4effd640 1 -- 172.21.15.42:0/1429842977 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c480ba040 con 0x7f5c5811c9f0 2024-09-16T04:11:45.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.979+0000 7f5c4effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:45.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.979+0000 7f5c4effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:11:45.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.979+0000 7f5c4effd640 10 monclient: dump: 2024-09-16T04:11:45.007 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:11:45.008 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:11:45.008 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:11:45.008 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:11:45.008 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:45.008 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:11:45.008 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:11:45.008 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:11:45.008 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:11:45.008 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:11:45.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.979+0000 7f5c4effd640 1 -- 172.21.15.42:0/1429842977 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4713 (secure 0 0 0) 0x7f5c480c0ce0 con 0x7f5c5811c9f0 2024-09-16T04:11:45.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.979+0000 7f5c4effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:11:45.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.979+0000 7f5c4effd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:11:45.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.979+0000 7f5c4effd640 1 -- 172.21.15.42:0/1429842977 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5c5811c9f0 msgr2=0x7f5c58137330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:45.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.979+0000 7f5c4effd640 1 --2- 172.21.15.42:0/1429842977 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5c5811c9f0 0x7f5c58137330 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f5c4801cda0 tx=0x7f5c580a22d0 comp rx=0 tx=0).stop 2024-09-16T04:11:45.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.979+0000 7f5c4effd640 1 -- 172.21.15.42:0/1429842977 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5c5811c9f0 msgr2=0x7f5c58137330 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:11:45.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.979+0000 7f5c4effd640 1 --2- 172.21.15.42:0/1429842977 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5c5811c9f0 0x7f5c58137330 secure :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f5c4801cda0 tx=0x7f5c580a22d0 comp rx=0 tx=0).stop 2024-09-16T04:11:45.009 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T04:11:45.009 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:11:45.009 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:11:45.009 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T04:11:45.009 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 268, 2024-09-16T04:11:45.009 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 300181, 2024-09-16T04:11:45.009 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T04:11:45.009 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T04:11:45.010 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T04:11:45.010 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T04:11:45.010 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:11:45.010 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 3, 2024-09-16T04:11:45.010 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:11:45.010 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T04:11:45.010 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T04:11:45.010 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:11:45.010 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:11:45.010 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:11:45.010 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:11:45.010 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:11:45.010 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:11:45.011 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:11:45.011 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:11:45.011 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:11:45.011 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:11:45.011 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:11:45.011 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:11:45.011 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:11:45.012 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T04:11:45.012 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:11:45.012 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:11:45.012 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:11:45.012 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:11:45.012 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:11:45.012 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:11:45.012 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:11:45.012 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:11:45.012 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:11:45.012 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:11:45.012 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:11:45.013 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:11:45.013 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:11:45.013 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T04:11:45.013 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T04:11:45.013 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T04:11:45.013 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T04:11:45.013 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T04:11:45.013 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T04:11:45.013 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:11:45.013 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:11:45.013 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T04:11:45.013 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T04:11:45.013 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T04:11:45.014 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T04:11:45.014 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T04:11:45.014 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T04:11:45.014 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T04:11:45.014 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:11:45.014 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T04:11:45.014 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:11:45.014 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:11:45.014 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:11:45.014 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:11:45.014 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:11:45.014 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:11:45.014 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:11:45.014 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:11:45.015 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:11:45.015 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:11:45.015 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:11:45.015 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:11:45.015 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T04:11:45.015 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:11:45.015 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T04:11:45.015 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:11:45.015 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:11:45.015 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:11:45.015 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:11:45.015 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:11:45.015 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:11:45.016 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:11:45.016 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T04:11:45.016 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:11:45.016 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:11:45.016 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:11:45.016 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:11:45.016 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T04:11:45.016 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:11:45.016 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:11:45.016 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:11:45.016 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:11:45.016 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T04:11:45.016 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T04:11:45.017 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:11:45.017 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:11:45.017 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:11:45.017 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:11:45.017 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:11:45.017 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:11:45.017 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:11:45.017 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:11:45.017 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:11:45.017 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:11:45.017 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:11:45.017 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T04:11:45.017 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:11:45.018 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:11:45.018 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:11:45.018 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:11:45.018 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T04:11:45.018 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:11:45.018 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:11:45.018 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:11:45.018 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:11:45.018 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T04:11:45.018 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T04:11:45.018 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:11:45.019 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:11:45.019 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:11:45.019 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:11:45.019 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:11:45.019 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:11:45.019 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:11:45.019 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:11:45.019 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:11:45.019 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:11:45.019 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:11:45.019 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T04:11:45.019 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:11:45.020 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:11:45.020 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:11:45.020 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:11:45.020 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T04:11:45.020 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:11:45.020 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:11:45.020 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:11:45.020 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:11:45.020 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T04:11:45.020 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T04:11:45.020 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:11:45.020 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:11:45.020 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:11:45.021 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:11:45.021 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:11:45.021 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:11:45.021 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T04:11:45.021 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T04:11:45.021 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:11:45.021 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:11:45.021 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:11:45.021 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:11:45.022 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:11:45.022 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:11:45.022 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T04:11:45.022 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:11:45.022 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:11:45.022 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:11:45.022 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 4 2024-09-16T04:11:45.022 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:11:45.022 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:11:45.022 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T04:11:45.022 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:11:45.022 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:11:45.023 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:11:45.023 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 4 2024-09-16T04:11:45.023 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:11:45.023 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:11:45.023 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:11:45.023 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T04:11:45.023 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T04:11:45.025 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.983+0000 7f5c602fb640 10 librados: watch_flush enter 2024-09-16T04:11:45.025 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.983+0000 7f5c602fb640 10 librados: watch_flush exit 2024-09-16T04:11:45.025 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.983+0000 7f5c602fb640 20 client.1014548.objecter shutdown clearing up homeless session... 2024-09-16T04:11:45.025 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.983+0000 7f5c602fb640 10 client.1014548.objecter successfully canceled tick 2024-09-16T04:11:45.025 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.983+0000 7f5c602fb640 1 -- 172.21.15.42:0/1429842977 >> v2:172.21.15.154:6812/254845342 conn(0x7f5c3c037810 msgr2=0x7f5c3c039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:45.025 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.983+0000 7f5c602fb640 1 --2- 172.21.15.42:0/1429842977 >> v2:172.21.15.154:6812/254845342 conn(0x7f5c3c037810 0x7f5c3c039cd0 secure :-1 s=READY pgs=3954 cs=0 l=1 rev1=1 crypto rx=0x7f5c50007fd0 tx=0x7f5c50007840 comp rx=0 tx=0).stop 2024-09-16T04:11:45.025 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.983+0000 7f5c602fb640 10 monclient: shutdown 2024-09-16T04:11:45.025 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.983+0000 7f5c602fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:11:45.025 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.983+0000 7f5c602fb640 1 -- 172.21.15.42:0/1429842977 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5c58135100 msgr2=0x7f5c58134520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:45.025 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.983+0000 7f5c602fb640 1 --2- 172.21.15.42:0/1429842977 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5c58135100 0x7f5c58134520 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f5c5400e870 tx=0x7f5c5400ed30 comp rx=0 tx=0).stop 2024-09-16T04:11:45.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.983+0000 7f5c5e898640 1 -- 172.21.15.42:0/1429842977 reap_dead start 2024-09-16T04:11:45.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.983+0000 7f5c5e898640 1 -- 172.21.15.42:0/1429842977 reap_dead start 2024-09-16T04:11:45.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.983+0000 7f5c602fb640 1 -- 172.21.15.42:0/1429842977 shutdown_connections 2024-09-16T04:11:45.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.983+0000 7f5c602fb640 1 -- 172.21.15.42:0/1429842977 >> 172.21.15.42:0/1429842977 conn(0x7f5c580a2420 msgr2=0x7f5c580a3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:11:45.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.983+0000 7f5c602fb640 1 -- 172.21.15.42:0/1429842977 shutdown_connections 2024-09-16T04:11:45.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.983+0000 7f5c602fb640 1 -- 172.21.15.42:0/1429842977 wait complete. 2024-09-16T04:11:45.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:44.983+0000 7f5c602fb640 1 librados: shutdown 2024-09-16T04:11:45.026 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:11:45.026 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T04:11:45.026 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T04:11:45.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.151+0000 7f3d7f0a6640 10 monclient: get_monmap_and_config 2024-09-16T04:11:45.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.151+0000 7f3d7f0a6640 10 monclient: build_initial_monmap 2024-09-16T04:11:45.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.151+0000 7f3d7f0a6640 10 monclient: monmap: 2024-09-16T04:11:45.154 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:11:45.154 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:11:45.154 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:11:45.152689+0000 2024-09-16T04:11:45.154 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:11:45.152689+0000 2024-09-16T04:11:45.154 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:11:45.154 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:11:45.154 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:11:45.154 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:11:45.155 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:11:45.155 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:11:45.155 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:11:45.155 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:11:45.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.151+0000 7f3d7f0a6640 1 Processor -- start 2024-09-16T04:11:45.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.151+0000 7f3d7f0a6640 1 -- start start 2024-09-16T04:11:45.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.151+0000 7f3d7f0a6640 10 monclient: init 2024-09-16T04:11:45.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.151+0000 7f3d7f0a6640 10 monclient: _reopen_session rank -1 2024-09-16T04:11:45.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.151+0000 7f3d7f0a6640 10 monclient: _add_conns ranks=[2,4,3,1,0] 2024-09-16T04:11:45.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.151+0000 7f3d7f0a6640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3d780a67e0 0x7f3d780a0b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:11:45.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.151+0000 7f3d7f0a6640 10 monclient(hunting): picked mon.noname-b con 0x7f3d780a67e0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:11:45.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.151+0000 7f3d7f0a6640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f3d780a00d0 0x7f3d780a6bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:11:45.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.151+0000 7f3d7f0a6640 10 monclient(hunting): picked mon.noname-c con 0x7f3d780a00d0 addr v2:172.21.15.154:3300/0 2024-09-16T04:11:45.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.151+0000 7f3d7f0a6640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f3d780a7100 0x7f3d781370b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:11:45.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.151+0000 7f3d7f0a6640 10 monclient(hunting): picked mon.noname-a con 0x7f3d780a7100 addr v2:172.21.15.42:3300/0 2024-09-16T04:11:45.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.151+0000 7f3d7f0a6640 10 monclient(hunting): start opening mon connection 2024-09-16T04:11:45.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.151+0000 7f3d7f0a6640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f3d780a1040 con 0x7f3d780a7100 2024-09-16T04:11:45.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.151+0000 7f3d7f0a6640 10 monclient(hunting): start opening mon connection 2024-09-16T04:11:45.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.151+0000 7f3d7f0a6640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f3d780a1a50 con 0x7f3d780a00d0 2024-09-16T04:11:45.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.151+0000 7f3d7f0a6640 10 monclient(hunting): start opening mon connection 2024-09-16T04:11:45.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.151+0000 7f3d7f0a6640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f3d780a16e0 con 0x7f3d780a67e0 2024-09-16T04:11:45.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.151+0000 7f3d7f0a6640 10 monclient(hunting): _renew_subs 2024-09-16T04:11:45.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.151+0000 7f3d7f0a6640 10 monclient(hunting): authenticate will time out at 2752.122556s 2024-09-16T04:11:45.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.151+0000 7f3d7d643640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f3d780a7100 0x7f3d781370b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:11:45.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.151+0000 7f3d7ce42640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3d780a67e0 0x7f3d780a0b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:11:45.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.151+0000 7f3d7d643640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f3d780a7100 0x7f3d781370b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:35482/0 (socket says 172.21.15.42:35482) 2024-09-16T04:11:45.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.151+0000 7f3d7d643640 1 -- 172.21.15.42:0/207266963 learned_addr learned my addr 172.21.15.42:0/207266963 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:11:45.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.151+0000 7f3d7d643640 10 monclient(hunting): get_auth_request con 0x7f3d780a7100 auth_method 0 2024-09-16T04:11:45.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.151+0000 7f3d7ce42640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3d780a67e0 0x7f3d780a0b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:51690/0 (socket says 172.21.15.42:51690) 2024-09-16T04:11:45.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.151+0000 7f3d7d643640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:11:45.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.151+0000 7f3d7d643640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:45.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.151+0000 7f3d7d643640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:45.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.151+0000 7f3d7ce42640 10 monclient(hunting): get_auth_request con 0x7f3d780a67e0 auth_method 0 2024-09-16T04:11:45.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.151+0000 7f3d7ce42640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:11:45.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.151+0000 7f3d7ce42640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:45.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.151+0000 7f3d7ce42640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:45.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.151+0000 7f3d7d643640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:11:45.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.151+0000 7f3d7d643640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:11:45.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.151+0000 7f3d7d643640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:11:45.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.151+0000 7f3d7ce42640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:11:45.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.151+0000 7f3d7ce42640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:11:45.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.151+0000 7f3d7ce42640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:11:45.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.151+0000 7f3d7d643640 10 monclient(hunting): handle_auth_done global_id 1034463 payload 274 2024-09-16T04:11:45.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.151+0000 7f3d7d643640 1 -- 172.21.15.42:0/207266963 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3d780a67e0 msgr2=0x7f3d780a0b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:11:45.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.151+0000 7f3d7d643640 1 --2- 172.21.15.42:0/207266963 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3d780a67e0 0x7f3d780a0b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:11:45.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.151+0000 7f3d77fff640 1 --2- 172.21.15.42:0/207266963 >> v2:172.21.15.154:3300/0 conn(0x7f3d780a00d0 0x7f3d780a6bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:11:45.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.151+0000 7f3d7d643640 1 -- 172.21.15.42:0/207266963 >> v2:172.21.15.154:3300/0 conn(0x7f3d780a00d0 msgr2=0x7f3d780a6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:11:45.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.151+0000 7f3d7d643640 1 --2- 172.21.15.42:0/207266963 >> v2:172.21.15.154:3300/0 conn(0x7f3d780a00d0 0x7f3d780a6bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:11:45.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.151+0000 7f3d7d643640 10 monclient: _finish_hunting 0 2024-09-16T04:11:45.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.151+0000 7f3d7d643640 1 monclient: found mon.noname-a 2024-09-16T04:11:45.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.151+0000 7f3d7d643640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:11:45.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.151+0000 7f3d7d643640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:11:45.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.151+0000 7f3d7d643640 1 -- 172.21.15.42:0/207266963 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d780a7570 con 0x7f3d780a7100 2024-09-16T04:11:45.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.151+0000 7f3d77fff640 10 monclient: get_auth_request con 0x7f3d780a00d0 auth_method 0 2024-09-16T04:11:45.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.151+0000 7f3d7ce42640 1 --2- 172.21.15.42:0/207266963 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3d780a67e0 0x7f3d780a0b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:11:45.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.151+0000 7f3d77fff640 1 --2- 172.21.15.42:0/207266963 >> v2:172.21.15.154:3300/0 conn(0x7f3d780a00d0 0x7f3d780a6bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T04:11:45.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.155+0000 7f3d7d643640 1 --2- 172.21.15.42:0/207266963 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3d780a7100 0x7f3d781370b0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f3d780a00a0 tx=0x7f3d7001c170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f4112af956141a7 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:11:45.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.155+0000 7f3d777fe640 1 -- 172.21.15.42:0/207266963 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d700b0070 con 0x7f3d780a7100 2024-09-16T04:11:45.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.155+0000 7f3d777fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:45.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.155+0000 7f3d777fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:11:45.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.155+0000 7f3d777fe640 10 monclient: dump: 2024-09-16T04:11:45.170 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:11:45.170 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:11:45.170 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:11:45.170 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:11:45.170 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:45.170 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:11:45.170 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:11:45.170 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:11:45.170 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:11:45.170 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:11:45.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.155+0000 7f3d777fe640 10 monclient: _finish_auth 0 2024-09-16T04:11:45.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.155+0000 7f3d777fe640 10 monclient: _check_auth_tickets 2024-09-16T04:11:45.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.155+0000 7f3d777fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:11:45.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.155+0000 7f3d7f0a6640 5 monclient: authenticate success, global_id 1034463 2024-09-16T04:11:45.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.155+0000 7f3d7f0a6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:11:45.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.155+0000 7f3d777fe640 1 -- 172.21.15.42:0/207266963 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d700ab600 con 0x7f3d780a7100 2024-09-16T04:11:45.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.155+0000 7f3d777fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:11:45.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.155+0000 7f3d7f0a6640 10 monclient: get_monmap_and_config success 2024-09-16T04:11:45.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.155+0000 7f3d7f0a6640 4 set_mon_vals no callback set 2024-09-16T04:11:45.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.155+0000 7f3d777fe640 1 -- 172.21.15.42:0/207266963 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d700ab9d0 con 0x7f3d780a7100 2024-09-16T04:11:45.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.155+0000 7f3d777fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:45.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.155+0000 7f3d777fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:11:45.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.155+0000 7f3d777fe640 10 monclient: dump: 2024-09-16T04:11:45.171 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:11:45.172 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:11:45.172 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:11:45.172 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:11:45.172 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:45.172 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:11:45.172 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:11:45.172 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:11:45.172 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:11:45.172 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:11:45.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.155+0000 7f3d7f0a6640 10 monclient: shutdown 2024-09-16T04:11:45.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.155+0000 7f3d7f0a6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:11:45.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.155+0000 7f3d7f0a6640 1 -- 172.21.15.42:0/207266963 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3d780a7100 msgr2=0x7f3d781370b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:45.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.155+0000 7f3d7f0a6640 1 --2- 172.21.15.42:0/207266963 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3d780a7100 0x7f3d781370b0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f3d780a00a0 tx=0x7f3d7001c170 comp rx=0 tx=0).stop 2024-09-16T04:11:45.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.155+0000 7f3d7f0a6640 1 -- 172.21.15.42:0/207266963 shutdown_connections 2024-09-16T04:11:45.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.155+0000 7f3d7f0a6640 1 --2- 172.21.15.42:0/207266963 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3d780a7100 0x7f3d781370b0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:11:45.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.155+0000 7f3d7f0a6640 1 --2- 172.21.15.42:0/207266963 >> v2:172.21.15.154:3300/0 conn(0x7f3d780a00d0 0x7f3d780a6bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:11:45.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.155+0000 7f3d7f0a6640 1 --2- 172.21.15.42:0/207266963 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3d780a67e0 0x7f3d780a0b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:11:45.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.155+0000 7f3d7f0a6640 1 -- 172.21.15.42:0/207266963 >> 172.21.15.42:0/207266963 conn(0x7f3d780a7ed0 msgr2=0x7f3d780a43d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:11:45.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.155+0000 7f3d7f0a6640 1 -- 172.21.15.42:0/207266963 shutdown_connections 2024-09-16T04:11:45.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.155+0000 7f3d7f0a6640 1 -- 172.21.15.42:0/207266963 wait complete. 2024-09-16T04:11:45.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.155+0000 7f3d7f0a6640 10 monclient: build_initial_monmap 2024-09-16T04:11:45.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.155+0000 7f3d7f0a6640 10 monclient: monmap: 2024-09-16T04:11:45.173 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:11:45.173 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:11:45.173 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:11:45.173 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:11:45.173 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:11:45.174 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:11:45.174 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:11:45.174 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:11:45.174 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:11:45.174 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:11:45.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.155+0000 7f3d7f0a6640 1 librados: starting msgr at 2024-09-16T04:11:45.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.155+0000 7f3d7f0a6640 1 librados: starting objecter 2024-09-16T04:11:45.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.155+0000 7f3d7f0a6640 1 Processor -- start 2024-09-16T04:11:45.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.155+0000 7f3d7f0a6640 1 -- start start 2024-09-16T04:11:45.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.155+0000 7f3d7f0a6640 1 librados: setting wanted keys 2024-09-16T04:11:45.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.155+0000 7f3d7f0a6640 1 librados: calling monclient init 2024-09-16T04:11:45.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.155+0000 7f3d7f0a6640 10 monclient: init 2024-09-16T04:11:45.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.155+0000 7f3d7f0a6640 10 monclient: _reopen_session rank -1 2024-09-16T04:11:45.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.155+0000 7f3d7f0a6640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:11:45.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.155+0000 7f3d7f0a6640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3d780a00d0 0x7f3d781ceef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:11:45.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.155+0000 7f3d7f0a6640 10 monclient(hunting): picked mon.noname-a con 0x7f3d780a00d0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:11:45.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.155+0000 7f3d7f0a6640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3d780a67e0 0x7f3d781cf430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:11:45.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.155+0000 7f3d7f0a6640 10 monclient(hunting): picked mon.noname-c con 0x7f3d780a67e0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:11:45.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.155+0000 7f3d7f0a6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d780a7100 0x7f3d781cf970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:11:45.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.155+0000 7f3d7f0a6640 10 monclient(hunting): picked mon.noname-b con 0x7f3d780a7100 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:11:45.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.155+0000 7f3d7f0a6640 10 monclient(hunting): start opening mon connection 2024-09-16T04:11:45.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.155+0000 7f3d7f0a6640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f3d781408b0 con 0x7f3d780a00d0 2024-09-16T04:11:45.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.155+0000 7f3d7f0a6640 10 monclient(hunting): start opening mon connection 2024-09-16T04:11:45.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.155+0000 7f3d7f0a6640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f3d78140730 con 0x7f3d780a7100 2024-09-16T04:11:45.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.155+0000 7f3d7f0a6640 10 monclient(hunting): start opening mon connection 2024-09-16T04:11:45.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.155+0000 7f3d7f0a6640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f3d78140a30 con 0x7f3d780a67e0 2024-09-16T04:11:45.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.155+0000 7f3d7f0a6640 10 monclient(hunting): _renew_subs 2024-09-16T04:11:45.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.155+0000 7f3d7f0a6640 10 monclient(hunting): authenticate will time out at 2752.127606s 2024-09-16T04:11:45.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.155+0000 7f3d7ce42640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3d780a00d0 0x7f3d781ceef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:11:45.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.155+0000 7f3d77fff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3d780a67e0 0x7f3d781cf430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:11:45.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.155+0000 7f3d7ce42640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3d780a00d0 0x7f3d781ceef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:35492/0 (socket says 172.21.15.42:35492) 2024-09-16T04:11:45.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.155+0000 7f3d7ce42640 1 -- 172.21.15.42:0/3344819892 learned_addr learned my addr 172.21.15.42:0/3344819892 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:11:45.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.155+0000 7f3d7ce42640 10 monclient(hunting): get_auth_request con 0x7f3d780a00d0 auth_method 0 2024-09-16T04:11:45.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.155+0000 7f3d7ce42640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:11:45.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.155+0000 7f3d7ce42640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:45.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.155+0000 7f3d7ce42640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:45.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.155+0000 7f3d77fff640 10 monclient(hunting): get_auth_request con 0x7f3d780a67e0 auth_method 0 2024-09-16T04:11:45.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.155+0000 7f3d77fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:11:45.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.155+0000 7f3d77fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:45.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.155+0000 7f3d77fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:45.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.155+0000 7f3d7ce42640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:11:45.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.155+0000 7f3d7ce42640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:11:45.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.155+0000 7f3d7ce42640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:11:45.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.155+0000 7f3d77fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:11:45.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.155+0000 7f3d77fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:11:45.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.155+0000 7f3d77fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:11:45.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.155+0000 7f3d7d643640 1 --2- 172.21.15.42:0/3344819892 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d780a7100 0x7f3d781cf970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:11:45.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.155+0000 7f3d7ce42640 10 monclient(hunting): handle_auth_done global_id 1034466 payload 824 2024-09-16T04:11:45.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.159+0000 7f3d7ce42640 1 -- 172.21.15.42:0/3344819892 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3d780a67e0 msgr2=0x7f3d781cf430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:45.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.159+0000 7f3d7ce42640 1 --2- 172.21.15.42:0/3344819892 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3d780a67e0 0x7f3d781cf430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:11:45.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.159+0000 7f3d7ce42640 1 -- 172.21.15.42:0/3344819892 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d780a7100 msgr2=0x7f3d781cf970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:45.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.159+0000 7f3d7ce42640 1 --2- 172.21.15.42:0/3344819892 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d780a7100 0x7f3d781cf970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:11:45.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.159+0000 7f3d7ce42640 10 monclient: _finish_hunting 0 2024-09-16T04:11:45.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.159+0000 7f3d7ce42640 1 monclient: found mon.noname-a 2024-09-16T04:11:45.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.159+0000 7f3d7ce42640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:11:45.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.159+0000 7f3d7ce42640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:11:45.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.159+0000 7f3d7ce42640 1 -- 172.21.15.42:0/3344819892 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d7813be10 con 0x7f3d780a00d0 2024-09-16T04:11:45.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.159+0000 7f3d7d643640 10 monclient: get_auth_request con 0x7f3d780a7100 auth_method 0 2024-09-16T04:11:45.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.159+0000 7f3d7d643640 1 --2- 172.21.15.42:0/3344819892 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d780a7100 0x7f3d781cf970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T04:11:45.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.159+0000 7f3d77fff640 1 --2- 172.21.15.42:0/3344819892 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3d780a67e0 0x7f3d781cf430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:11:45.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.159+0000 7f3d7ce42640 1 --2- 172.21.15.42:0/3344819892 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3d780a00d0 0x7f3d781ceef0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f3d6800efc0 tx=0x7f3d6800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:11:45.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.159+0000 7f3d757fa640 10 client.?.objecter ms_handle_connect 0x7f3d780a00d0 2024-09-16T04:11:45.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.159+0000 7f3d757fa640 10 client.?.objecter resend_mon_ops 2024-09-16T04:11:45.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.159+0000 7f3d757fa640 1 -- 172.21.15.42:0/3344819892 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d68018020 con 0x7f3d780a00d0 2024-09-16T04:11:45.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.159+0000 7f3d757fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:45.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.159+0000 7f3d757fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:11:45.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.159+0000 7f3d757fa640 10 monclient: dump: 2024-09-16T04:11:45.178 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:11:45.178 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:11:45.179 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:11:45.179 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:11:45.179 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:45.179 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:11:45.179 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:11:45.179 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:11:45.179 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:11:45.179 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:11:45.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.159+0000 7f3d757fa640 10 monclient: _finish_auth 0 2024-09-16T04:11:45.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.159+0000 7f3d757fa640 10 monclient: _check_auth_tickets 2024-09-16T04:11:45.180 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.159+0000 7f3d757fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:11:45.180 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.159+0000 7f3d757fa640 1 -- 172.21.15.42:0/3344819892 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d68014680 con 0x7f3d780a00d0 2024-09-16T04:11:45.180 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.159+0000 7f3d757fa640 10 monclient: handle_config config(0 keys) 2024-09-16T04:11:45.180 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.159+0000 7f3d757fa640 1 -- 172.21.15.42:0/3344819892 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d68014a50 con 0x7f3d780a00d0 2024-09-16T04:11:45.180 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.159+0000 7f3d7f0a6640 5 monclient: authenticate success, global_id 1034466 2024-09-16T04:11:45.180 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.159+0000 7f3d7f0a6640 10 monclient: _renew_subs 2024-09-16T04:11:45.180 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.159+0000 7f3d7f0a6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:11:45.180 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.159+0000 7f3d7f0a6640 1 -- 172.21.15.42:0/3344819892 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d7813c0f0 con 0x7f3d780a00d0 2024-09-16T04:11:45.180 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.159+0000 7f3d76ffd640 4 set_mon_vals no callback set 2024-09-16T04:11:45.180 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.159+0000 7f3d757fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:45.180 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.159+0000 7f3d757fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:11:45.180 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.159+0000 7f3d757fa640 10 monclient: dump: 2024-09-16T04:11:45.180 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:11:45.181 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:11:45.181 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:11:45.181 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:11:45.181 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:45.181 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:11:45.181 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:11:45.181 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:11:45.181 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:11:45.181 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:11:45.181 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.159+0000 7f3d7f0a6640 10 client.1034466.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:11:45.181 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.159+0000 7f3d7f0a6640 10 monclient: _renew_subs 2024-09-16T04:11:45.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.159+0000 7f3d7f0a6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:11:45.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.159+0000 7f3d7f0a6640 1 -- 172.21.15.42:0/3344819892 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d7813d530 con 0x7f3d780a00d0 2024-09-16T04:11:45.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.159+0000 7f3d7f0a6640 1 librados: init done 2024-09-16T04:11:45.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.159+0000 7f3d7f0a6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:45.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.159+0000 7f3d7f0a6640 1 --2- 172.21.15.42:0/3344819892 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d78131050 0x7f3d78131430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:11:45.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.159+0000 7f3d7f0a6640 10 monclient(hunting): start opening mon connection 2024-09-16T04:11:45.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.159+0000 7f3d7f0a6640 1 -- 172.21.15.42:0/3344819892 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f3d7806df20 con 0x7f3d78131050 2024-09-16T04:11:45.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.159+0000 7f3d757fa640 1 -- 172.21.15.42:0/3344819892 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f3d6801b020 con 0x7f3d780a00d0 2024-09-16T04:11:45.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.159+0000 7f3d757fa640 1 --2- 172.21.15.42:0/3344819892 >> v2:172.21.15.154:6812/254845342 conn(0x7f3d64037810 0x7f3d64039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:11:45.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.159+0000 7f3d757fa640 1 -- 172.21.15.42:0/3344819892 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(706..706 src has 451..706) ==== 5040+0+0 (secure 0 0 0) 0x7f3d680499c0 con 0x7f3d780a00d0 2024-09-16T04:11:45.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.159+0000 7f3d757fa640 10 client.1034466.objecter ms_dispatch 0x7f3d78000ba0 osd_map(706..706 src has 451..706) 2024-09-16T04:11:45.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.159+0000 7f3d757fa640 3 client.1034466.objecter handle_osd_map got epochs [706,706] > 0 2024-09-16T04:11:45.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.159+0000 7f3d757fa640 3 client.1034466.objecter handle_osd_map decoding full epoch 706 2024-09-16T04:11:45.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.159+0000 7f3d77fff640 1 --2- 172.21.15.42:0/3344819892 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d78131050 0x7f3d78131430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:11:45.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.159+0000 7f3d77fff640 10 monclient: get_auth_request con 0x7f3d78131050 auth_method 0 2024-09-16T04:11:45.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.159+0000 7f3d77fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:11:45.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.159+0000 7f3d77fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:45.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.159+0000 7f3d77fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:45.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.159+0000 7f3d757fa640 20 client.1034466.objecter dump_active .. 0 homeless 2024-09-16T04:11:45.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.159+0000 7f3d77fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:11:45.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.159+0000 7f3d77fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:11:45.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.159+0000 7f3d77fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:11:45.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.159+0000 7f3d7d643640 1 --2- 172.21.15.42:0/3344819892 >> v2:172.21.15.154:6812/254845342 conn(0x7f3d64037810 0x7f3d64039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:11:45.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.159+0000 7f3d7d643640 10 monclient: get_auth_request con 0x7f3d64037810 auth_method 0 2024-09-16T04:11:45.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.159+0000 7f3d77fff640 10 monclient(hunting): handle_auth_done global_id 1074142 payload 824 2024-09-16T04:11:45.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.159+0000 7f3d77fff640 1 -- 172.21.15.42:0/3344819892 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3d78073970 con 0x7f3d78131050 2024-09-16T04:11:45.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.159+0000 7f3d757fa640 10 client.1034466.objecter ms_handle_connect 0x7f3d78131050 2024-09-16T04:11:45.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.159+0000 7f3d77fff640 1 --2- 172.21.15.42:0/3344819892 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d78131050 0x7f3d78131430 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f3d7806a570 tx=0x7f3d6c00d590 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:11:45.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.159+0000 7f3d757fa640 10 client.1034466.objecter resend_mon_ops 2024-09-16T04:11:45.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.159+0000 7f3d757fa640 10 client.1034466.objecter ms_handle_connect 0x7f3d64037810 2024-09-16T04:11:45.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.159+0000 7f3d7d643640 1 --2- 172.21.15.42:0/3344819892 >> v2:172.21.15.154:6812/254845342 conn(0x7f3d64037810 0x7f3d64039cd0 secure :-1 s=READY pgs=3955 cs=0 l=1 rev1=1 crypto rx=0x7f3d700ad000 tx=0x7f3d700a6000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:11:45.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.163+0000 7f3d757fa640 1 -- 172.21.15.42:0/3344819892 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d6c00dea0 con 0x7f3d78131050 2024-09-16T04:11:45.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.163+0000 7f3d757fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:45.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.163+0000 7f3d757fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:11:45.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.163+0000 7f3d757fa640 10 monclient: dump: 2024-09-16T04:11:45.184 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:11:45.185 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:11:45.185 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:11:45.185 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:11:45.185 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:45.185 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:11:45.185 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:11:45.185 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:11:45.185 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:11:45.185 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:11:45.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.163+0000 7f3d757fa640 1 -- 172.21.15.42:0/3344819892 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f3d6c004220 con 0x7f3d78131050 2024-09-16T04:11:45.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.163+0000 7f3d757fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:11:45.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.163+0000 7f3d757fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:11:45.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.163+0000 7f3d757fa640 1 -- 172.21.15.42:0/3344819892 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d78131050 msgr2=0x7f3d78131430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:45.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.163+0000 7f3d757fa640 1 --2- 172.21.15.42:0/3344819892 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d78131050 0x7f3d78131430 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f3d7806a570 tx=0x7f3d6c00d590 comp rx=0 tx=0).stop 2024-09-16T04:11:45.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.163+0000 7f3d757fa640 1 -- 172.21.15.42:0/3344819892 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d78131050 msgr2=0x7f3d78131430 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:11:45.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.163+0000 7f3d757fa640 1 --2- 172.21.15.42:0/3344819892 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d78131050 0x7f3d78131430 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:11:45.188 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.171+0000 7f3d7f0a6640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:11:45.188 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.171+0000 7f3d7f0a6640 1 --2- 172.21.15.42:0/3344819892 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d78131970 0x7f3d78132160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:11:45.188 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.171+0000 7f3d7f0a6640 10 monclient(hunting): start opening mon connection 2024-09-16T04:11:45.188 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.171+0000 7f3d7f0a6640 1 -- 172.21.15.42:0/3344819892 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f3d780a0b00 con 0x7f3d78131970 2024-09-16T04:11:45.188 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.171+0000 7f3d77fff640 1 --2- 172.21.15.42:0/3344819892 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d78131970 0x7f3d78132160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:11:45.188 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.171+0000 7f3d77fff640 10 monclient: get_auth_request con 0x7f3d78131970 auth_method 0 2024-09-16T04:11:45.189 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.171+0000 7f3d77fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:11:45.189 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.171+0000 7f3d77fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:11:45.189 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.171+0000 7f3d77fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:11:45.189 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.171+0000 7f3d77fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:11:45.189 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.171+0000 7f3d77fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:11:45.189 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.171+0000 7f3d77fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:11:45.189 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.171+0000 7f3d77fff640 10 monclient(hunting): handle_auth_done global_id 1074145 payload 824 2024-09-16T04:11:45.189 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.171+0000 7f3d77fff640 1 -- 172.21.15.42:0/3344819892 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f3d7813c490 con 0x7f3d78131970 2024-09-16T04:11:45.189 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.171+0000 7f3d77fff640 1 --2- 172.21.15.42:0/3344819892 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d78131970 0x7f3d78132160 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f3d6c000c10 tx=0x7f3d6c0046d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:11:45.190 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.171+0000 7f3d757fa640 10 client.1034466.objecter ms_handle_connect 0x7f3d78131970 2024-09-16T04:11:45.190 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.171+0000 7f3d757fa640 10 client.1034466.objecter resend_mon_ops 2024-09-16T04:11:45.190 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.175+0000 7f3d757fa640 1 -- 172.21.15.42:0/3344819892 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d6c003e30 con 0x7f3d78131970 2024-09-16T04:11:45.190 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.175+0000 7f3d757fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:11:45.190 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.175+0000 7f3d757fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:11:45.190 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.175+0000 7f3d757fa640 10 monclient: dump: 2024-09-16T04:11:45.190 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:11:45.190 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:11:45.190 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:11:45.190 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:11:45.191 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:11:45.191 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:11:45.191 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:11:45.191 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:11:45.191 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:11:45.191 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:11:45.191 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.175+0000 7f3d757fa640 1 -- 172.21.15.42:0/3344819892 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4545 (secure 0 0 0) 0x7f3d6c003e30 con 0x7f3d78131970 2024-09-16T04:11:45.191 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.175+0000 7f3d757fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:11:45.191 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.175+0000 7f3d757fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:11:45.191 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.175+0000 7f3d757fa640 1 -- 172.21.15.42:0/3344819892 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d78131970 msgr2=0x7f3d78132160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:45.191 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.175+0000 7f3d757fa640 1 --2- 172.21.15.42:0/3344819892 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d78131970 0x7f3d78132160 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f3d6c000c10 tx=0x7f3d6c0046d0 comp rx=0 tx=0).stop 2024-09-16T04:11:45.191 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.175+0000 7f3d757fa640 1 -- 172.21.15.42:0/3344819892 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d78131970 msgr2=0x7f3d78132160 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:11:45.191 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.175+0000 7f3d757fa640 1 --2- 172.21.15.42:0/3344819892 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3d78131970 0x7f3d78132160 secure :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f3d6c000c10 tx=0x7f3d6c0046d0 comp rx=0 tx=0).stop 2024-09-16T04:11:45.192 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T04:11:45.192 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:11:45.192 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:11:45.192 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T04:11:45.192 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 268, 2024-09-16T04:11:45.192 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 18347, 2024-09-16T04:11:45.192 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T04:11:45.192 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T04:11:45.192 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T04:11:45.192 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T04:11:45.192 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:11:45.193 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 3, 2024-09-16T04:11:45.193 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:11:45.193 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T04:11:45.193 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T04:11:45.193 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:11:45.193 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:11:45.193 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:11:45.193 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:11:45.193 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:11:45.193 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:11:45.193 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:11:45.193 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:11:45.193 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:11:45.194 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:11:45.194 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:11:45.194 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:11:45.194 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:11:45.194 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T04:11:45.194 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:11:45.194 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:11:45.194 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:11:45.194 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:11:45.195 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:11:45.195 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:11:45.195 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:11:45.195 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:11:45.195 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:11:45.195 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:11:45.195 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:11:45.195 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:11:45.195 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:11:45.195 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T04:11:45.195 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T04:11:45.195 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T04:11:45.196 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T04:11:45.196 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T04:11:45.196 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T04:11:45.196 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:11:45.196 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:11:45.196 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T04:11:45.196 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T04:11:45.196 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T04:11:45.196 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T04:11:45.196 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T04:11:45.197 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T04:11:45.197 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T04:11:45.197 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:11:45.197 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T04:11:45.197 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:11:45.197 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:11:45.197 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:11:45.197 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:11:45.197 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:11:45.197 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:11:45.197 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:11:45.197 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:11:45.197 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:11:45.197 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:11:45.198 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:11:45.198 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:11:45.198 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T04:11:45.198 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:11:45.198 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T04:11:45.198 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:11:45.198 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:11:45.198 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:11:45.198 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:11:45.198 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:11:45.198 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:11:45.198 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:11:45.198 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T04:11:45.198 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:11:45.199 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:11:45.199 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:11:45.199 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:11:45.199 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T04:11:45.199 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:11:45.199 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:11:45.199 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:11:45.199 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:11:45.199 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T04:11:45.199 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T04:11:45.199 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:11:45.199 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:11:45.199 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:11:45.199 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:11:45.200 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:11:45.200 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:11:45.200 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:11:45.200 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:11:45.200 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:11:45.200 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:11:45.200 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:11:45.200 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T04:11:45.201 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:11:45.201 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:11:45.201 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:11:45.201 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:11:45.201 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T04:11:45.201 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:11:45.201 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:11:45.201 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:11:45.201 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:11:45.201 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T04:11:45.201 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T04:11:45.201 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:11:45.201 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:11:45.201 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:11:45.202 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:11:45.202 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:11:45.202 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:11:45.202 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:11:45.202 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:11:45.202 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:11:45.202 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:11:45.202 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:11:45.202 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T04:11:45.202 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:11:45.202 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:11:45.202 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:11:45.202 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:11:45.203 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T04:11:45.203 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:11:45.203 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:11:45.203 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:11:45.203 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:11:45.203 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T04:11:45.203 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T04:11:45.203 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:11:45.203 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:11:45.203 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:11:45.203 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:11:45.203 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:11:45.203 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:11:45.203 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T04:11:45.204 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T04:11:45.204 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:11:45.204 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:11:45.204 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:11:45.204 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:11:45.204 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:11:45.204 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:11:45.204 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T04:11:45.204 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:11:45.204 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:11:45.204 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:11:45.204 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:11:45.204 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:11:45.204 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:11:45.204 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:11:45.205 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T04:11:45.205 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T04:11:45.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.175+0000 7f3d7f0a6640 10 librados: watch_flush enter 2024-09-16T04:11:45.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.175+0000 7f3d7f0a6640 10 librados: watch_flush exit 2024-09-16T04:11:45.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.175+0000 7f3d7f0a6640 20 client.1034466.objecter shutdown clearing up homeless session... 2024-09-16T04:11:45.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.175+0000 7f3d7f0a6640 10 client.1034466.objecter successfully canceled tick 2024-09-16T04:11:45.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.175+0000 7f3d7f0a6640 1 -- 172.21.15.42:0/3344819892 >> v2:172.21.15.154:6812/254845342 conn(0x7f3d64037810 msgr2=0x7f3d64039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:45.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.175+0000 7f3d7f0a6640 1 --2- 172.21.15.42:0/3344819892 >> v2:172.21.15.154:6812/254845342 conn(0x7f3d64037810 0x7f3d64039cd0 secure :-1 s=READY pgs=3955 cs=0 l=1 rev1=1 crypto rx=0x7f3d700ad000 tx=0x7f3d700a6000 comp rx=0 tx=0).stop 2024-09-16T04:11:45.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.175+0000 7f3d7f0a6640 10 monclient: shutdown 2024-09-16T04:11:45.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.175+0000 7f3d7f0a6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:11:45.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.175+0000 7f3d7f0a6640 1 -- 172.21.15.42:0/3344819892 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3d780a00d0 msgr2=0x7f3d781ceef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:11:45.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.175+0000 7f3d7f0a6640 1 --2- 172.21.15.42:0/3344819892 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3d780a00d0 0x7f3d781ceef0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f3d6800efc0 tx=0x7f3d6800c490 comp rx=0 tx=0).stop 2024-09-16T04:11:45.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.175+0000 7f3d7d643640 1 -- 172.21.15.42:0/3344819892 reap_dead start 2024-09-16T04:11:45.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.175+0000 7f3d7f0a6640 1 -- 172.21.15.42:0/3344819892 shutdown_connections 2024-09-16T04:11:45.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.175+0000 7f3d7f0a6640 1 --2- 172.21.15.42:0/3344819892 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3d780a00d0 0x7f3d781ceef0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:11:45.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.175+0000 7f3d7f0a6640 1 -- 172.21.15.42:0/3344819892 >> 172.21.15.42:0/3344819892 conn(0x7f3d780a7ed0 msgr2=0x7f3d780a31d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:11:45.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.175+0000 7f3d7f0a6640 1 -- 172.21.15.42:0/3344819892 shutdown_connections 2024-09-16T04:11:45.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.175+0000 7f3d7f0a6640 1 -- 172.21.15.42:0/3344819892 wait complete. 2024-09-16T04:11:45.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:11:45.175+0000 7f3d7f0a6640 1 librados: shutdown 2024-09-16T04:11:45.208 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:11:45.208 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-09-16T04:11:45.208 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['a'] 2024-09-16T04:11:45.208 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-09-16T04:11:45.208 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-16T04:11:45.209 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-16T04:11:45.209 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T04:11:45.228 INFO:tasks.ceph.mon.c:Stopped 2024-09-16T04:11:45.228 INFO:tasks.mon_thrash:Sending STOP to mon a 2024-09-16T04:11:45.228 INFO:tasks.ceph.mon.a:Sent signal 19 2024-09-16T04:11:45.228 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-16T04:12:00.229 INFO:tasks.mon_thrash:Sending CONT to mon a 2024-09-16T04:12:00.266 INFO:tasks.ceph.mon.a:Sent signal 18 2024-09-16T04:12:00.267 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T04:12:00.267 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:12:00.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:00.395+0000 7f09cdb37640 10 monclient: get_monmap_and_config 2024-09-16T04:12:00.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:00.395+0000 7f09cdb37640 10 monclient: build_initial_monmap 2024-09-16T04:12:00.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:00.395+0000 7f09cdb37640 10 monclient: monmap: 2024-09-16T04:12:00.400 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:12:00.400 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:12:00.400 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:12:00.397619+0000 2024-09-16T04:12:00.400 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:12:00.397619+0000 2024-09-16T04:12:00.400 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:12:00.400 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:12:00.401 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:12:00.401 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:12:00.401 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:12:00.401 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:12:00.401 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:12:00.402 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:12:00.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:00.395+0000 7f09cdb37640 1 Processor -- start 2024-09-16T04:12:00.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:00.395+0000 7f09cdb37640 1 -- start start 2024-09-16T04:12:00.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:00.395+0000 7f09cdb37640 10 monclient: init 2024-09-16T04:12:00.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:00.395+0000 7f09cdb37640 10 monclient: _reopen_session rank -1 2024-09-16T04:12:00.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:00.395+0000 7f09cdb37640 10 monclient: _add_conns ranks=[2,0,1,4,3] 2024-09-16T04:12:00.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:00.395+0000 7f09cdb37640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f09c80a1330 0x7f09c80a1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:12:00.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:00.395+0000 7f09cdb37640 10 monclient(hunting): picked mon.noname-b con 0x7f09c80a1330 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:12:00.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:00.395+0000 7f09cdb37640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f09c809c920 addr v1:172.21.15.42:6789/0 2024-09-16T04:12:00.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:00.395+0000 7f09cdb37640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f09c8072d40 addr v1:172.21.15.154:6789/0 2024-09-16T04:12:00.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:00.395+0000 7f09cdb37640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f09c8137f40 con 0x7f09c809c920 2024-09-16T04:12:00.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:00.395+0000 7f09cdb37640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f09c8072350 con 0x7f09c8072d40 2024-09-16T04:12:00.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:00.395+0000 7f09cdb37640 10 monclient(hunting): start opening mon connection 2024-09-16T04:12:00.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:00.395+0000 7f09cdb37640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f09c8072b50 con 0x7f09c80a1330 2024-09-16T04:12:00.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:00.395+0000 7f09cdb37640 10 monclient(hunting): _renew_subs 2024-09-16T04:12:00.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:00.395+0000 7f09cdb37640 10 monclient(hunting): authenticate will time out at 2767.367063s 2024-09-16T04:12:00.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:00.395+0000 7f09c77fe640 1 -- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f09c80a1330 msgr2=0x7f09c80a1710 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T04:12:00.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:00.395+0000 7f09c77fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f09c80a1330 0x7f09c80a1710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:12:00.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:00.395+0000 7f09c6ffd640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f09c809c920 0x7f09c8137120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:34146/0 (socket says 172.21.15.42:34146) 2024-09-16T04:12:00.407 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:00.395+0000 7f09c6ffd640 1 -- 172.21.15.42:0/743329656 learned_addr learned my addr 172.21.15.42:0/743329656 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T04:12:00.601 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:00.599+0000 7f09c77fe640 1 -- 172.21.15.42:0/743329656 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f09c80a1330 msgr2=0x7f09c80a1710 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T04:12:00.601 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:00.599+0000 7f09c77fe640 1 --2- 172.21.15.42:0/743329656 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f09c80a1330 0x7f09c80a1710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T04:12:01.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:01.003+0000 7f09c77fe640 1 -- 172.21.15.42:0/743329656 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f09c80a1330 msgr2=0x7f09c80a1710 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T04:12:01.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:01.003+0000 7f09c77fe640 1 --2- 172.21.15.42:0/743329656 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f09c80a1330 0x7f09c80a1710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T04:12:01.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:01.807+0000 7f09c77fe640 1 -- 172.21.15.42:0/743329656 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f09c80a1330 msgr2=0x7f09c80a1710 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T04:12:01.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:01.807+0000 7f09c77fe640 1 --2- 172.21.15.42:0/743329656 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f09c80a1330 0x7f09c80a1710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T04:12:03.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:03.395+0000 7f09c57fa640 10 monclient(hunting): tick 2024-09-16T04:12:03.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:03.395+0000 7f09c57fa640 10 monclient(hunting): _check_auth_tickets 2024-09-16T04:12:03.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:03.395+0000 7f09c57fa640 1 monclient(hunting): continuing hunt 2024-09-16T04:12:03.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:03.395+0000 7f09c57fa640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T04:12:03.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:03.395+0000 7f09c57fa640 1 -- 172.21.15.42:0/743329656 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f09c80a1330 msgr2=0x7f09c80a1710 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:12:03.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:03.395+0000 7f09c57fa640 1 --2- 172.21.15.42:0/743329656 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f09c80a1330 0x7f09c80a1710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:12:03.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:03.395+0000 7f09c57fa640 1 -- 172.21.15.42:0/743329656 >> v1:172.21.15.154:6789/0 conn(0x7f09c8072d40 legacy=0x7f09c8137830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:03.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:03.395+0000 7f09c57fa640 1 -- 172.21.15.42:0/743329656 >> v1:172.21.15.42:6789/0 conn(0x7f09c809c920 legacy=0x7f09c8137120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:03.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:03.395+0000 7f09c57fa640 10 monclient: _add_conns ranks=[4,3] 2024-09-16T04:12:03.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:03.395+0000 7f09c57fa640 1 --2- 172.21.15.42:0/743329656 >> v2:172.21.15.154:3300/0 conn(0x7f09a8003720 0x7f09a8005ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:12:03.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:03.395+0000 7f09c57fa640 10 monclient(hunting): picked mon.noname-c con 0x7f09a8003720 addr v2:172.21.15.154:3300/0 2024-09-16T04:12:03.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:03.395+0000 7f09c57fa640 1 --2- 172.21.15.42:0/743329656 >> v2:172.21.15.42:3300/0 conn(0x7f09a8006290 0x7f09a8008710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:12:03.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:03.395+0000 7f09c57fa640 10 monclient(hunting): picked mon.noname-a con 0x7f09a8006290 addr v2:172.21.15.42:3300/0 2024-09-16T04:12:03.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:03.395+0000 7f09c57fa640 10 monclient(hunting): start opening mon connection 2024-09-16T04:12:03.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:03.395+0000 7f09c57fa640 1 -- 172.21.15.42:0/743329656 --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f09c8072b50 con 0x7f09a8006290 2024-09-16T04:12:03.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:03.395+0000 7f09c57fa640 10 monclient(hunting): start opening mon connection 2024-09-16T04:12:03.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:03.395+0000 7f09c57fa640 1 -- 172.21.15.42:0/743329656 --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f09a8008dd0 con 0x7f09a8003720 2024-09-16T04:12:03.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:03.395+0000 7f09c57fa640 10 monclient(hunting): _renew_subs 2024-09-16T04:12:03.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:03.395+0000 7f09c6ffd640 1 --2- 172.21.15.42:0/743329656 >> v2:172.21.15.42:3300/0 conn(0x7f09a8006290 0x7f09a8008710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:12:03.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:03.395+0000 7f09c6ffd640 10 monclient(hunting): get_auth_request con 0x7f09a8006290 auth_method 0 2024-09-16T04:12:03.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:03.395+0000 7f09c6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:12:03.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:03.395+0000 7f09c6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:03.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:03.395+0000 7f09c6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:03.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:03.395+0000 7f09c77fe640 1 --2- 172.21.15.42:0/743329656 >> v2:172.21.15.154:3300/0 conn(0x7f09a8003720 0x7f09a8005ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:12:03.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:03.395+0000 7f09c77fe640 10 monclient(hunting): get_auth_request con 0x7f09a8003720 auth_method 0 2024-09-16T04:12:03.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:03.395+0000 7f09c77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:12:03.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:03.395+0000 7f09c77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:03.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:03.395+0000 7f09c77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:03.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:03.395+0000 7f09c6ffd640 1 -- 172.21.15.42:0/743329656 >> v2:172.21.15.42:3300/0 conn(0x7f09a8006290 msgr2=0x7f09a8008710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:12:03.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:03.395+0000 7f09c6ffd640 1 -- 172.21.15.42:0/743329656 >> v2:172.21.15.42:3300/0 conn(0x7f09a8006290 msgr2=0x7f09a8008710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:12:03.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:03.399+0000 7f09c6ffd640 1 --2- 172.21.15.42:0/743329656 >> v2:172.21.15.42:3300/0 conn(0x7f09a8006290 0x7f09a8008710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:12:03.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:03.399+0000 7f09c6ffd640 1 --2- 172.21.15.42:0/743329656 >> v2:172.21.15.42:3300/0 conn(0x7f09a8006290 0x7f09a8008710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:12:03.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:03.399+0000 7f09c77fe640 1 -- 172.21.15.42:0/743329656 >> v2:172.21.15.154:3300/0 conn(0x7f09a8003720 msgr2=0x7f09a8005ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:12:03.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:03.399+0000 7f09c77fe640 1 -- 172.21.15.42:0/743329656 >> v2:172.21.15.154:3300/0 conn(0x7f09a8003720 msgr2=0x7f09a8005ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:12:03.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:03.399+0000 7f09c77fe640 1 --2- 172.21.15.42:0/743329656 >> v2:172.21.15.154:3300/0 conn(0x7f09a8003720 0x7f09a8005ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:12:03.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:03.399+0000 7f09c77fe640 1 --2- 172.21.15.42:0/743329656 >> v2:172.21.15.154:3300/0 conn(0x7f09a8003720 0x7f09a8005ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:12:03.606 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:03.603+0000 7f09c6ffd640 1 --2- 172.21.15.42:0/743329656 >> v2:172.21.15.42:3300/0 conn(0x7f09a8006290 0x7f09a8008710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:12:03.606 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:03.603+0000 7f09c6ffd640 10 monclient(hunting): get_auth_request con 0x7f09a8006290 auth_method 0 2024-09-16T04:12:03.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:03.603+0000 7f09c6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:12:03.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:03.603+0000 7f09c6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:03.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:03.603+0000 7f09c6ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:12:03.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:03.603+0000 7f09c77fe640 1 --2- 172.21.15.42:0/743329656 >> v2:172.21.15.154:3300/0 conn(0x7f09a8003720 0x7f09a8005ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:12:03.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:03.603+0000 7f09c6ffd640 1 -- 172.21.15.42:0/743329656 >> v2:172.21.15.42:3300/0 conn(0x7f09a8006290 msgr2=0x7f09a8008710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:12:03.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:03.603+0000 7f09c6ffd640 1 -- 172.21.15.42:0/743329656 >> v2:172.21.15.42:3300/0 conn(0x7f09a8006290 msgr2=0x7f09a8008710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:12:03.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:03.603+0000 7f09c77fe640 10 monclient(hunting): get_auth_request con 0x7f09a8003720 auth_method 0 2024-09-16T04:12:03.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:03.603+0000 7f09c77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:12:03.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:03.603+0000 7f09c77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:03.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:03.603+0000 7f09c6ffd640 1 --2- 172.21.15.42:0/743329656 >> v2:172.21.15.42:3300/0 conn(0x7f09a8006290 0x7f09a8008710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:12:03.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:03.603+0000 7f09c77fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:12:03.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:03.603+0000 7f09c6ffd640 1 --2- 172.21.15.42:0/743329656 >> v2:172.21.15.42:3300/0 conn(0x7f09a8006290 0x7f09a8008710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T04:12:03.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:03.603+0000 7f09c77fe640 1 -- 172.21.15.42:0/743329656 >> v2:172.21.15.154:3300/0 conn(0x7f09a8003720 msgr2=0x7f09a8005ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:12:03.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:03.603+0000 7f09c77fe640 1 -- 172.21.15.42:0/743329656 >> v2:172.21.15.154:3300/0 conn(0x7f09a8003720 msgr2=0x7f09a8005ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:12:03.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:03.603+0000 7f09c77fe640 1 --2- 172.21.15.42:0/743329656 >> v2:172.21.15.154:3300/0 conn(0x7f09a8003720 0x7f09a8005ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:12:03.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:03.603+0000 7f09c77fe640 1 --2- 172.21.15.42:0/743329656 >> v2:172.21.15.154:3300/0 conn(0x7f09a8003720 0x7f09a8005ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T04:12:04.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:04.007+0000 7f09c6ffd640 1 --2- 172.21.15.42:0/743329656 >> v2:172.21.15.42:3300/0 conn(0x7f09a8006290 0x7f09a8008710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:12:04.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:04.007+0000 7f09c6ffd640 10 monclient(hunting): get_auth_request con 0x7f09a8006290 auth_method 0 2024-09-16T04:12:04.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:04.007+0000 7f09c6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:12:04.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:04.007+0000 7f09c6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:04.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:04.007+0000 7f09c6ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:12:04.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:04.007+0000 7f09c77fe640 1 --2- 172.21.15.42:0/743329656 >> v2:172.21.15.154:3300/0 conn(0x7f09a8003720 0x7f09a8005ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:12:04.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:04.007+0000 7f09c77fe640 10 monclient(hunting): get_auth_request con 0x7f09a8003720 auth_method 0 2024-09-16T04:12:04.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:04.007+0000 7f09c77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:12:04.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:04.007+0000 7f09c77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:04.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:04.007+0000 7f09c77fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:12:04.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:04.007+0000 7f09c6ffd640 1 -- 172.21.15.42:0/743329656 >> v2:172.21.15.42:3300/0 conn(0x7f09a8006290 msgr2=0x7f09a8008710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:12:04.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:04.007+0000 7f09c6ffd640 1 -- 172.21.15.42:0/743329656 >> v2:172.21.15.42:3300/0 conn(0x7f09a8006290 msgr2=0x7f09a8008710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:12:04.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:04.007+0000 7f09c6ffd640 1 --2- 172.21.15.42:0/743329656 >> v2:172.21.15.42:3300/0 conn(0x7f09a8006290 0x7f09a8008710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:12:04.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:04.007+0000 7f09c6ffd640 1 --2- 172.21.15.42:0/743329656 >> v2:172.21.15.42:3300/0 conn(0x7f09a8006290 0x7f09a8008710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T04:12:04.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:04.007+0000 7f09c77fe640 1 -- 172.21.15.42:0/743329656 >> v2:172.21.15.154:3300/0 conn(0x7f09a8003720 msgr2=0x7f09a8005ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:12:04.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:04.007+0000 7f09c77fe640 1 -- 172.21.15.42:0/743329656 >> v2:172.21.15.154:3300/0 conn(0x7f09a8003720 msgr2=0x7f09a8005ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:12:04.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:04.007+0000 7f09c77fe640 1 --2- 172.21.15.42:0/743329656 >> v2:172.21.15.154:3300/0 conn(0x7f09a8003720 0x7f09a8005ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:12:04.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:04.007+0000 7f09c77fe640 1 --2- 172.21.15.42:0/743329656 >> v2:172.21.15.154:3300/0 conn(0x7f09a8003720 0x7f09a8005ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T04:12:04.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:04.811+0000 7f09c6ffd640 1 --2- 172.21.15.42:0/743329656 >> v2:172.21.15.42:3300/0 conn(0x7f09a8006290 0x7f09a8008710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:12:04.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:04.811+0000 7f09c6ffd640 10 monclient(hunting): get_auth_request con 0x7f09a8006290 auth_method 0 2024-09-16T04:12:04.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:04.811+0000 7f09c6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:12:04.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:04.811+0000 7f09c6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:04.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:04.811+0000 7f09c6ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:12:04.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:04.811+0000 7f09c77fe640 1 --2- 172.21.15.42:0/743329656 >> v2:172.21.15.154:3300/0 conn(0x7f09a8003720 0x7f09a8005ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:12:04.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:04.811+0000 7f09c77fe640 10 monclient(hunting): get_auth_request con 0x7f09a8003720 auth_method 0 2024-09-16T04:12:04.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:04.811+0000 7f09c6ffd640 1 -- 172.21.15.42:0/743329656 >> v2:172.21.15.42:3300/0 conn(0x7f09a8006290 msgr2=0x7f09a8008710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:12:04.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:04.811+0000 7f09c77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:12:04.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:04.811+0000 7f09c77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:04.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:04.811+0000 7f09c77fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:12:04.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:04.811+0000 7f09c6ffd640 1 -- 172.21.15.42:0/743329656 >> v2:172.21.15.42:3300/0 conn(0x7f09a8006290 msgr2=0x7f09a8008710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:12:04.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:04.811+0000 7f09c6ffd640 1 --2- 172.21.15.42:0/743329656 >> v2:172.21.15.42:3300/0 conn(0x7f09a8006290 0x7f09a8008710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:12:04.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:04.811+0000 7f09c6ffd640 1 --2- 172.21.15.42:0/743329656 >> v2:172.21.15.42:3300/0 conn(0x7f09a8006290 0x7f09a8008710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T04:12:04.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:04.811+0000 7f09c77fe640 1 -- 172.21.15.42:0/743329656 >> v2:172.21.15.154:3300/0 conn(0x7f09a8003720 msgr2=0x7f09a8005ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:12:04.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:04.811+0000 7f09c77fe640 1 -- 172.21.15.42:0/743329656 >> v2:172.21.15.154:3300/0 conn(0x7f09a8003720 msgr2=0x7f09a8005ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:12:04.816 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:04.811+0000 7f09c77fe640 1 --2- 172.21.15.42:0/743329656 >> v2:172.21.15.154:3300/0 conn(0x7f09a8003720 0x7f09a8005ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:12:04.816 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:04.811+0000 7f09c77fe640 1 --2- 172.21.15.42:0/743329656 >> v2:172.21.15.154:3300/0 conn(0x7f09a8003720 0x7f09a8005ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T04:12:06.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.398+0000 7f09c57fa640 10 monclient(hunting): tick 2024-09-16T04:12:06.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.398+0000 7f09c57fa640 10 monclient(hunting): _check_auth_tickets 2024-09-16T04:12:06.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.398+0000 7f09c57fa640 1 monclient(hunting): continuing hunt 2024-09-16T04:12:06.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.398+0000 7f09c57fa640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T04:12:06.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.398+0000 7f09c57fa640 1 -- 172.21.15.42:0/743329656 >> v2:172.21.15.154:3300/0 conn(0x7f09a8003720 msgr2=0x7f09a8005ba0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:12:06.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.398+0000 7f09c57fa640 1 --2- 172.21.15.42:0/743329656 >> v2:172.21.15.154:3300/0 conn(0x7f09a8003720 0x7f09a8005ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:12:06.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.398+0000 7f09c57fa640 1 -- 172.21.15.42:0/743329656 >> v2:172.21.15.42:3300/0 conn(0x7f09a8006290 msgr2=0x7f09a8008710 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:12:06.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.398+0000 7f09c57fa640 1 --2- 172.21.15.42:0/743329656 >> v2:172.21.15.42:3300/0 conn(0x7f09a8006290 0x7f09a8008710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:12:06.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.398+0000 7f09c57fa640 10 monclient: _add_conns ranks=[4,0,1,2,3] 2024-09-16T04:12:06.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.398+0000 7f09c7fff640 1 -- 172.21.15.42:0/743329656 reap_dead start 2024-09-16T04:12:06.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.398+0000 7f09c57fa640 1 --2- 172.21.15.42:0/743329656 >> v2:172.21.15.154:3300/0 conn(0x7f09a8003720 0x7f09a8005ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:12:06.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.398+0000 7f09c57fa640 10 monclient(hunting): picked mon.noname-c con 0x7f09a8003720 addr v2:172.21.15.154:3300/0 2024-09-16T04:12:06.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.398+0000 7f09c57fa640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f09c809c920 addr v1:172.21.15.42:6789/0 2024-09-16T04:12:06.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.398+0000 7f09c57fa640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f09a8006700 addr v1:172.21.15.154:6789/0 2024-09-16T04:12:06.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.398+0000 7f09c57fa640 1 -- 172.21.15.42:0/743329656 --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f09c809bc60 con 0x7f09c809c920 2024-09-16T04:12:06.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.398+0000 7f09c57fa640 1 -- 172.21.15.42:0/743329656 --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f09a800f020 con 0x7f09a8006700 2024-09-16T04:12:06.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.398+0000 7f09c57fa640 10 monclient(hunting): start opening mon connection 2024-09-16T04:12:06.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.398+0000 7f09c57fa640 1 -- 172.21.15.42:0/743329656 --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f09c8072b50 con 0x7f09a8003720 2024-09-16T04:12:06.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.398+0000 7f09c57fa640 10 monclient(hunting): _renew_subs 2024-09-16T04:12:06.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.398+0000 7f09c77fe640 1 --2- 172.21.15.42:0/743329656 >> v2:172.21.15.154:3300/0 conn(0x7f09a8003720 0x7f09a8005ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:12:06.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.398+0000 7f09c77fe640 10 monclient(hunting): get_auth_request con 0x7f09a8003720 auth_method 0 2024-09-16T04:12:06.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.398+0000 7f09c77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:12:06.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.398+0000 7f09c77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:06.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.398+0000 7f09c77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:06.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.398+0000 7f09c77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:12:06.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.398+0000 7f09c77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:12:06.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.398+0000 7f09c77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:12:06.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.398+0000 7f09c67fc640 1 -- 172.21.15.42:0/743329656 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2425253051 0 0) 0x7f09c809bc60 con 0x7f09c809c920 2024-09-16T04:12:06.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.398+0000 7f09c67fc640 10 monclient(hunting): _negotiate 2024-09-16T04:12:06.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.398+0000 7f09c67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:06.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.398+0000 7f09c67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:06.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.398+0000 7f09c67fc640 10 monclient(hunting): my global_id is 1034469 2024-09-16T04:12:06.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.398+0000 7f09c67fc640 1 -- 172.21.15.42:0/743329656 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f09b4003530 con 0x7f09c809c920 2024-09-16T04:12:06.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.398+0000 7f09c77fe640 10 monclient(hunting): handle_auth_done global_id 1074148 payload 274 2024-09-16T04:12:06.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.398+0000 7f09c77fe640 1 -- 172.21.15.42:0/743329656 >> v1:172.21.15.154:6789/0 conn(0x7f09a8006700 legacy=0x7f09a800c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:06.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.398+0000 7f09c77fe640 1 -- 172.21.15.42:0/743329656 >> v1:172.21.15.42:6789/0 conn(0x7f09c809c920 legacy=0x7f09a800b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:06.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.398+0000 7f09c67fc640 1 -- 172.21.15.42:0/743329656 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1893578783 0 0) 0x7f09a800f020 con 0x7f09a8006700 2024-09-16T04:12:06.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.398+0000 7f09c77fe640 10 monclient: _finish_hunting 0 2024-09-16T04:12:06.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.398+0000 7f09c77fe640 1 monclient: found mon.noname-c 2024-09-16T04:12:06.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.398+0000 7f09c77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:12:06.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.398+0000 7f09c77fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.154:3300/0 2024-09-16T04:12:06.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.398+0000 7f09c77fe640 1 -- 172.21.15.42:0/743329656 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09a8010250 con 0x7f09a8003720 2024-09-16T04:12:06.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.398+0000 7f09c67fc640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T04:12:06.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.402+0000 7f09c77fe640 1 --2- 172.21.15.42:0/743329656 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f09a8003720 0x7f09a8005ba0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f09b0003990 tx=0x7f09b001d560 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5eca1d4c536c0571 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:12:06.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.402+0000 7f09c67fc640 1 -- 172.21.15.42:0/743329656 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09b00ad430 con 0x7f09a8003720 2024-09-16T04:12:06.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.402+0000 7f09c67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:06.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.402+0000 7f09c67fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:12:06.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.402+0000 7f09c67fc640 10 monclient: dump: 2024-09-16T04:12:06.409 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:12:06.409 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:12:06.409 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:06.409 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:06.409 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:06.409 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:12:06.409 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:12:06.410 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:12:06.410 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:12:06.410 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:12:06.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.402+0000 7f09c67fc640 10 monclient: _finish_auth 0 2024-09-16T04:12:06.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.402+0000 7f09c67fc640 10 monclient: _check_auth_tickets 2024-09-16T04:12:06.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.402+0000 7f09c67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:12:06.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.402+0000 7f09c67fc640 1 -- 172.21.15.42:0/743329656 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f09b00b2070 con 0x7f09a8003720 2024-09-16T04:12:06.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.402+0000 7f09c67fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:12:06.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.402+0000 7f09c67fc640 1 -- 172.21.15.42:0/743329656 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09b00ad810 con 0x7f09a8003720 2024-09-16T04:12:06.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.402+0000 7f09c67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:06.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.402+0000 7f09c67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:12:06.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.402+0000 7f09c67fc640 10 monclient: dump: 2024-09-16T04:12:06.410 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:12:06.410 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:12:06.411 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:06.411 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:06.411 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:06.411 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:12:06.411 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:12:06.411 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:12:06.411 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:12:06.412 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:12:06.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.402+0000 7f09cdb37640 5 monclient: authenticate success, global_id 1074148 2024-09-16T04:12:06.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.402+0000 7f09cdb37640 10 monclient: get_monmap_and_config success 2024-09-16T04:12:06.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.402+0000 7f09cdb37640 4 set_mon_vals no callback set 2024-09-16T04:12:06.418 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.406+0000 7f09cdb37640 10 monclient: shutdown 2024-09-16T04:12:06.418 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.406+0000 7f09cdb37640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:12:06.418 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.406+0000 7f09cdb37640 1 -- 172.21.15.42:0/743329656 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f09a8003720 msgr2=0x7f09a8005ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:06.419 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.406+0000 7f09cdb37640 1 --2- 172.21.15.42:0/743329656 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f09a8003720 0x7f09a8005ba0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f09b0003990 tx=0x7f09b001d560 comp rx=0 tx=0).stop 2024-09-16T04:12:06.419 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.406+0000 7f09cdb37640 1 -- 172.21.15.42:0/743329656 shutdown_connections 2024-09-16T04:12:06.419 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.406+0000 7f09cdb37640 1 --2- 172.21.15.42:0/743329656 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f09a8003720 0x7f09a8005ba0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:12:06.419 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.406+0000 7f09cdb37640 1 -- 172.21.15.42:0/743329656 >> 172.21.15.42:0/743329656 conn(0x7f09c80a2420 msgr2=0x7f09c80a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:12:06.419 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.406+0000 7f09cdb37640 1 -- 172.21.15.42:0/743329656 shutdown_connections 2024-09-16T04:12:06.419 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.406+0000 7f09cdb37640 1 -- 172.21.15.42:0/743329656 wait complete. 2024-09-16T04:12:06.419 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.406+0000 7f09cdb37640 10 monclient: build_initial_monmap 2024-09-16T04:12:06.419 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.406+0000 7f09cdb37640 10 monclient: monmap: 2024-09-16T04:12:06.419 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:12:06.420 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:12:06.420 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:12:06.420 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:12:06.420 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:12:06.420 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:12:06.420 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:12:06.420 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:12:06.420 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:12:06.420 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:12:06.420 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.406+0000 7f09cdb37640 1 librados: starting msgr at 2024-09-16T04:12:06.420 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.406+0000 7f09cdb37640 1 librados: starting objecter 2024-09-16T04:12:06.421 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.406+0000 7f09cdb37640 1 Processor -- start 2024-09-16T04:12:06.421 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.406+0000 7f09cdb37640 1 -- start start 2024-09-16T04:12:06.421 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.406+0000 7f09cdb37640 1 librados: setting wanted keys 2024-09-16T04:12:06.421 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.406+0000 7f09cdb37640 1 librados: calling monclient init 2024-09-16T04:12:06.421 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.406+0000 7f09cdb37640 10 monclient: init 2024-09-16T04:12:06.421 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.406+0000 7f09cdb37640 10 monclient: _reopen_session rank -1 2024-09-16T04:12:06.421 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.406+0000 7f09cdb37640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:12:06.421 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.406+0000 7f09cdb37640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f09a8003720 0x7f09c81cf5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:12:06.421 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.406+0000 7f09cdb37640 10 monclient(hunting): picked mon.noname-a con 0x7f09a8003720 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:12:06.421 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.406+0000 7f09cdb37640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f09a8006700 0x7f09c81cfb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:12:06.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.406+0000 7f09cdb37640 10 monclient(hunting): picked mon.noname-b con 0x7f09a8006700 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:12:06.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.406+0000 7f09cdb37640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f09c809c920 0x7f09c81228e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:12:06.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.406+0000 7f09cdb37640 10 monclient(hunting): picked mon.noname-c con 0x7f09c809c920 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:12:06.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.406+0000 7f09cdb37640 10 monclient(hunting): start opening mon connection 2024-09-16T04:12:06.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.406+0000 7f09cdb37640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f09c81372a0 con 0x7f09a8003720 2024-09-16T04:12:06.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.406+0000 7f09cdb37640 10 monclient(hunting): start opening mon connection 2024-09-16T04:12:06.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.406+0000 7f09cdb37640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f09c8137120 con 0x7f09a8006700 2024-09-16T04:12:06.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.406+0000 7f09cdb37640 10 monclient(hunting): start opening mon connection 2024-09-16T04:12:06.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.406+0000 7f09cdb37640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f09c8137420 con 0x7f09c809c920 2024-09-16T04:12:06.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.406+0000 7f09cdb37640 10 monclient(hunting): _renew_subs 2024-09-16T04:12:06.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.406+0000 7f09cdb37640 10 monclient(hunting): authenticate will time out at 2773.379291s 2024-09-16T04:12:06.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.406+0000 7f09c7fff640 1 -- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f09c809c920 msgr2=0x7f09c81228e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T04:12:06.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.406+0000 7f09c7fff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f09c809c920 0x7f09c81228e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:12:06.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.406+0000 7f09c77fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f09a8003720 0x7f09c81cf5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:12:06.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.406+0000 7f09c77fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f09a8003720 0x7f09c81cf5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:38976/0 (socket says 172.21.15.42:38976) 2024-09-16T04:12:06.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.406+0000 7f09c77fe640 1 -- 172.21.15.42:0/3983403402 learned_addr learned my addr 172.21.15.42:0/3983403402 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:12:06.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.406+0000 7f09c77fe640 10 monclient(hunting): get_auth_request con 0x7f09a8003720 auth_method 0 2024-09-16T04:12:06.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.406+0000 7f09c77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:12:06.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.406+0000 7f09c77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:06.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.406+0000 7f09c77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:06.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.406+0000 7f09c6ffd640 1 --2- 172.21.15.42:0/3983403402 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f09a8006700 0x7f09c81cfb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:12:06.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.406+0000 7f09c77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:12:06.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.406+0000 7f09c77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:12:06.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.406+0000 7f09c77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:12:06.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.406+0000 7f09c6ffd640 10 monclient(hunting): get_auth_request con 0x7f09a8006700 auth_method 0 2024-09-16T04:12:06.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.406+0000 7f09c6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:12:06.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.406+0000 7f09c6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:06.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.406+0000 7f09c6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:06.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.406+0000 7f09c6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:12:06.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.406+0000 7f09c6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:12:06.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.406+0000 7f09c6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:12:06.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.406+0000 7f09c77fe640 10 monclient(hunting): handle_auth_done global_id 1034472 payload 824 2024-09-16T04:12:06.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.410+0000 7f09c77fe640 1 -- 172.21.15.42:0/3983403402 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f09c809c920 msgr2=0x7f09c81228e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:12:06.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.410+0000 7f09c77fe640 1 --2- 172.21.15.42:0/3983403402 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f09c809c920 0x7f09c81228e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:12:06.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.410+0000 7f09c77fe640 1 -- 172.21.15.42:0/3983403402 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f09a8006700 msgr2=0x7f09c81cfb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:06.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.410+0000 7f09c77fe640 1 --2- 172.21.15.42:0/3983403402 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f09a8006700 0x7f09c81cfb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:12:06.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.410+0000 7f09c77fe640 10 monclient: _finish_hunting 0 2024-09-16T04:12:06.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.410+0000 7f09c77fe640 1 monclient: found mon.noname-a 2024-09-16T04:12:06.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.410+0000 7f09c77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:12:06.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.410+0000 7f09c77fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:12:06.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.410+0000 7f09c77fe640 1 -- 172.21.15.42:0/3983403402 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09c8126870 con 0x7f09a8003720 2024-09-16T04:12:06.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.410+0000 7f09c77fe640 1 --2- 172.21.15.42:0/3983403402 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f09a8003720 0x7f09c81cf5d0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f09b001a170 tx=0x7f09b001a300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:12:06.425 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.410+0000 7f09bffff640 10 client.?.objecter ms_handle_connect 0x7f09a8003720 2024-09-16T04:12:06.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.410+0000 7f09bffff640 10 client.?.objecter resend_mon_ops 2024-09-16T04:12:06.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.410+0000 7f09bffff640 1 -- 172.21.15.42:0/3983403402 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09b00ad630 con 0x7f09a8003720 2024-09-16T04:12:06.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.410+0000 7f09bffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:06.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.410+0000 7f09bffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:12:06.426 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.410+0000 7f09bffff640 10 monclient: dump: 2024-09-16T04:12:06.426 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:12:06.426 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:12:06.426 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:06.426 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:06.426 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:06.426 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:12:06.427 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:12:06.427 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:12:06.427 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:12:06.427 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:12:06.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.410+0000 7f09bffff640 10 monclient: _finish_auth 0 2024-09-16T04:12:06.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.410+0000 7f09bffff640 10 monclient: _check_auth_tickets 2024-09-16T04:12:06.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.410+0000 7f09bffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:12:06.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.410+0000 7f09bffff640 1 -- 172.21.15.42:0/3983403402 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f09b00ad7d0 con 0x7f09a8003720 2024-09-16T04:12:06.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.410+0000 7f09cdb37640 5 monclient: authenticate success, global_id 1034472 2024-09-16T04:12:06.427 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.410+0000 7f09cdb37640 10 monclient: _renew_subs 2024-09-16T04:12:06.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.410+0000 7f09cdb37640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:12:06.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.410+0000 7f09cdb37640 1 -- 172.21.15.42:0/3983403402 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09c81362f0 con 0x7f09a8003720 2024-09-16T04:12:06.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.410+0000 7f09bffff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:12:06.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.410+0000 7f09bffff640 1 -- 172.21.15.42:0/3983403402 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09b00adbc0 con 0x7f09a8003720 2024-09-16T04:12:06.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.410+0000 7f09bffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:06.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.410+0000 7f09bffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:12:06.428 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.410+0000 7f09bffff640 10 monclient: dump: 2024-09-16T04:12:06.428 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:12:06.428 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:12:06.428 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:06.428 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:06.429 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:06.429 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:12:06.429 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:12:06.429 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:12:06.429 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:12:06.429 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:12:06.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.410+0000 7f09c57fa640 4 set_mon_vals no callback set 2024-09-16T04:12:06.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.410+0000 7f09cdb37640 10 client.1034472.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:12:06.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.410+0000 7f09cdb37640 10 monclient: _renew_subs 2024-09-16T04:12:06.429 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.410+0000 7f09cdb37640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:12:06.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.410+0000 7f09cdb37640 1 -- 172.21.15.42:0/3983403402 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09c81262b0 con 0x7f09a8003720 2024-09-16T04:12:06.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.410+0000 7f09cdb37640 1 librados: init done 2024-09-16T04:12:06.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.410+0000 7f09cdb37640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:12:06.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.410+0000 7f09cdb37640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:12:06.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.410+0000 7f09cdb37640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:12:06.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.410+0000 7f09cdb37640 1 -- 172.21.15.42:0/3983403402 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f09c8137880 con 0x7f09a8003720 2024-09-16T04:12:06.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.410+0000 7f09bffff640 1 -- 172.21.15.42:0/3983403402 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f09b00aa070 con 0x7f09a8003720 2024-09-16T04:12:06.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.410+0000 7f09bffff640 1 --2- 172.21.15.42:0/3983403402 >> v2:172.21.15.154:6812/254845342 conn(0x7f09a80389d0 0x7f09a803ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:12:06.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.410+0000 7f09bffff640 1 -- 172.21.15.42:0/3983403402 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(706..706 src has 451..706) ==== 5040+0+0 (secure 0 0 0) 0x7f09b00e3b00 con 0x7f09a8003720 2024-09-16T04:12:06.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.410+0000 7f09bffff640 10 client.1034472.objecter ms_dispatch 0x7f09c8000ba0 osd_map(706..706 src has 451..706) 2024-09-16T04:12:06.430 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.410+0000 7f09bffff640 3 client.1034472.objecter handle_osd_map got epochs [706,706] > 0 2024-09-16T04:12:06.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.410+0000 7f09bffff640 3 client.1034472.objecter handle_osd_map decoding full epoch 706 2024-09-16T04:12:06.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.410+0000 7f09bffff640 20 client.1034472.objecter dump_active .. 0 homeless 2024-09-16T04:12:06.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.410+0000 7f09c6ffd640 1 --2- 172.21.15.42:0/3983403402 >> v2:172.21.15.154:6812/254845342 conn(0x7f09a80389d0 0x7f09a803ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:12:06.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.410+0000 7f09c6ffd640 10 monclient: get_auth_request con 0x7f09a80389d0 auth_method 0 2024-09-16T04:12:06.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.410+0000 7f09c6ffd640 1 --2- 172.21.15.42:0/3983403402 >> v2:172.21.15.154:6812/254845342 conn(0x7f09a80389d0 0x7f09a803ae90 secure :-1 s=READY pgs=3956 cs=0 l=1 rev1=1 crypto rx=0x7f09c81261e0 tx=0x7f09b8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:12:06.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.410+0000 7f09bffff640 10 client.1034472.objecter ms_handle_connect 0x7f09a80389d0 2024-09-16T04:12:06.432 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.418+0000 7f09bffff640 1 -- 172.21.15.42:0/3983403402 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f09b00175a0 con 0x7f09a8003720 2024-09-16T04:12:06.432 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.418+0000 7f09bffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:12:06.432 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.418+0000 7f09bffff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:12:06.649 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.646+0000 7f09cdb37640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:12:06.649 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.646+0000 7f09cdb37640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:12:06.649 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.646+0000 7f09cdb37640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:12:06.650 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.646+0000 7f09cdb37640 1 -- 172.21.15.42:0/3983403402 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f09c8069040 con 0x7f09a8003720 2024-09-16T04:12:06.650 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.646+0000 7f09bffff640 1 -- 172.21.15.42:0/3983403402 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1555 (secure 0 0 0) 0x7f09b00c0400 con 0x7f09a8003720 2024-09-16T04:12:06.650 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.646+0000 7f09bffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:12:06.650 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.646+0000 7f09bffff640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:12:06.650 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":272,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:12:06.653 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.646+0000 7f09cdb37640 10 librados: watch_flush enter 2024-09-16T04:12:06.653 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.646+0000 7f09cdb37640 10 librados: watch_flush exit 2024-09-16T04:12:06.653 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.646+0000 7f09cdb37640 20 client.1034472.objecter shutdown clearing up homeless session... 2024-09-16T04:12:06.653 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.646+0000 7f09cdb37640 10 client.1034472.objecter successfully canceled tick 2024-09-16T04:12:06.653 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.646+0000 7f09cdb37640 1 -- 172.21.15.42:0/3983403402 >> v2:172.21.15.154:6812/254845342 conn(0x7f09a80389d0 msgr2=0x7f09a803ae90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:06.653 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.646+0000 7f09cdb37640 1 --2- 172.21.15.42:0/3983403402 >> v2:172.21.15.154:6812/254845342 conn(0x7f09a80389d0 0x7f09a803ae90 secure :-1 s=READY pgs=3956 cs=0 l=1 rev1=1 crypto rx=0x7f09c81261e0 tx=0x7f09b8009040 comp rx=0 tx=0).stop 2024-09-16T04:12:06.653 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.646+0000 7f09cdb37640 10 monclient: shutdown 2024-09-16T04:12:06.653 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.646+0000 7f09cdb37640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:12:06.653 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.646+0000 7f09cdb37640 1 -- 172.21.15.42:0/3983403402 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f09a8003720 msgr2=0x7f09c81cf5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:06.654 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.646+0000 7f09cdb37640 1 --2- 172.21.15.42:0/3983403402 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f09a8003720 0x7f09c81cf5d0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f09b001a170 tx=0x7f09b001a300 comp rx=0 tx=0).stop 2024-09-16T04:12:06.654 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.646+0000 7f09cdb37640 1 -- 172.21.15.42:0/3983403402 shutdown_connections 2024-09-16T04:12:06.654 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.646+0000 7f09cdb37640 1 --2- 172.21.15.42:0/3983403402 >> v2:172.21.15.154:6812/254845342 conn(0x7f09a80389d0 0x7f09a803ae90 unknown :-1 s=CLOSED pgs=3956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:12:06.654 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.646+0000 7f09cdb37640 1 --2- 172.21.15.42:0/3983403402 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f09c809c920 0x7f09c81228e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:12:06.654 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.646+0000 7f09cdb37640 1 --2- 172.21.15.42:0/3983403402 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f09a8006700 0x7f09c81cfb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:12:06.654 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.646+0000 7f09cdb37640 1 --2- 172.21.15.42:0/3983403402 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f09a8003720 0x7f09c81cf5d0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:12:06.654 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.646+0000 7f09cdb37640 1 -- 172.21.15.42:0/3983403402 >> 172.21.15.42:0/3983403402 conn(0x7f09c80a2420 msgr2=0x7f09c81359e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:12:06.654 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.646+0000 7f09cdb37640 1 -- 172.21.15.42:0/3983403402 shutdown_connections 2024-09-16T04:12:06.654 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.646+0000 7f09cdb37640 1 -- 172.21.15.42:0/3983403402 wait complete. 2024-09-16T04:12:06.654 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.646+0000 7f09cdb37640 1 librados: shutdown 2024-09-16T04:12:06.669 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T04:12:06.670 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-16T04:12:06.670 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T04:12:06.789 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.786+0000 7fe01e1a6640 10 monclient: get_monmap_and_config 2024-09-16T04:12:06.790 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.786+0000 7fe01e1a6640 10 monclient: build_initial_monmap 2024-09-16T04:12:06.790 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.786+0000 7fe01e1a6640 10 monclient: monmap: 2024-09-16T04:12:06.790 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:12:06.790 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:12:06.790 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:12:06.788502+0000 2024-09-16T04:12:06.790 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:12:06.788502+0000 2024-09-16T04:12:06.790 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:12:06.790 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:12:06.790 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:12:06.791 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:12:06.791 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:12:06.791 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:12:06.791 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:12:06.791 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:12:06.794 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.786+0000 7fe01e1a6640 1 Processor -- start 2024-09-16T04:12:06.794 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.786+0000 7fe01e1a6640 1 -- start start 2024-09-16T04:12:06.795 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.786+0000 7fe01e1a6640 10 monclient: init 2024-09-16T04:12:06.795 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.786+0000 7fe01e1a6640 10 monclient: _reopen_session rank -1 2024-09-16T04:12:06.795 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.786+0000 7fe01e1a6640 10 monclient: _add_conns ranks=[2,3,4,0,1] 2024-09-16T04:12:06.795 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.786+0000 7fe01e1a6640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe0180a39b0 0x7fe018133b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:12:06.796 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.786+0000 7fe01e1a6640 10 monclient(hunting): picked mon.noname-b con 0x7fe0180a39b0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:12:06.796 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.786+0000 7fe01e1a6640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe0180a3d90 0x7fe018137000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:12:06.796 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.786+0000 7fe01e1a6640 10 monclient(hunting): picked mon.noname-a con 0x7fe0180a3d90 addr v2:172.21.15.42:3300/0 2024-09-16T04:12:06.796 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.786+0000 7fe01e1a6640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fe018137540 0x7fe018137950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:12:06.796 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.786+0000 7fe01e1a6640 10 monclient(hunting): picked mon.noname-c con 0x7fe018137540 addr v2:172.21.15.154:3300/0 2024-09-16T04:12:06.796 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.786+0000 7fe01e1a6640 10 monclient(hunting): start opening mon connection 2024-09-16T04:12:06.796 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.786+0000 7fe01e1a6640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7fe018137e90 con 0x7fe0180a3d90 2024-09-16T04:12:06.797 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.786+0000 7fe01e1a6640 10 monclient(hunting): start opening mon connection 2024-09-16T04:12:06.797 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.786+0000 7fe01e1a6640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fe01809c140 con 0x7fe018137540 2024-09-16T04:12:06.797 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.786+0000 7fe01e1a6640 10 monclient(hunting): start opening mon connection 2024-09-16T04:12:06.797 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.786+0000 7fe01e1a6640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fe01809c420 con 0x7fe0180a39b0 2024-09-16T04:12:06.797 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.786+0000 7fe01e1a6640 10 monclient(hunting): _renew_subs 2024-09-16T04:12:06.797 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.786+0000 7fe01e1a6640 10 monclient(hunting): authenticate will time out at 2773.757944s 2024-09-16T04:12:06.797 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.786+0000 7fe0177fe640 1 -- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe0180a39b0 msgr2=0x7fe018133b80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T04:12:06.797 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.786+0000 7fe0177fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe0180a39b0 0x7fe018133b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:12:06.797 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.786+0000 7fe016ffd640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe0180a3d90 0x7fe018137000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:12:06.797 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.786+0000 7fe016ffd640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe0180a3d90 0x7fe018137000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:38982/0 (socket says 172.21.15.42:38982) 2024-09-16T04:12:06.797 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.786+0000 7fe016ffd640 1 -- 172.21.15.42:0/1988446104 learned_addr learned my addr 172.21.15.42:0/1988446104 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:12:06.797 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.786+0000 7fe016ffd640 10 monclient(hunting): get_auth_request con 0x7fe0180a3d90 auth_method 0 2024-09-16T04:12:06.797 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.786+0000 7fe016ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:12:06.798 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.786+0000 7fe016ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:06.798 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.786+0000 7fe016ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:06.798 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.786+0000 7fe017fff640 1 --2- 172.21.15.42:0/1988446104 >> v2:172.21.15.154:3300/0 conn(0x7fe018137540 0x7fe018137950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:12:06.798 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.786+0000 7fe016ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:12:06.798 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.786+0000 7fe016ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:12:06.798 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.786+0000 7fe016ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:12:06.798 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.786+0000 7fe017fff640 10 monclient(hunting): get_auth_request con 0x7fe018137540 auth_method 0 2024-09-16T04:12:06.798 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.786+0000 7fe017fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:12:06.798 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.786+0000 7fe017fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:06.798 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.786+0000 7fe017fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:06.798 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.786+0000 7fe016ffd640 10 monclient(hunting): handle_auth_done global_id 1034475 payload 274 2024-09-16T04:12:06.798 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.786+0000 7fe016ffd640 1 -- 172.21.15.42:0/1988446104 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe0180a39b0 msgr2=0x7fe018133b80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:12:06.798 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.786+0000 7fe016ffd640 1 --2- 172.21.15.42:0/1988446104 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe0180a39b0 0x7fe018133b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:12:06.799 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.786+0000 7fe016ffd640 1 -- 172.21.15.42:0/1988446104 >> v2:172.21.15.154:3300/0 conn(0x7fe018137540 msgr2=0x7fe018137950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:12:06.799 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.786+0000 7fe016ffd640 1 --2- 172.21.15.42:0/1988446104 >> v2:172.21.15.154:3300/0 conn(0x7fe018137540 0x7fe018137950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:12:06.799 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.786+0000 7fe016ffd640 10 monclient: _finish_hunting 0 2024-09-16T04:12:06.799 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.786+0000 7fe016ffd640 1 monclient: found mon.noname-a 2024-09-16T04:12:06.799 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.786+0000 7fe016ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:12:06.799 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.786+0000 7fe016ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:12:06.799 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.786+0000 7fe016ffd640 1 -- 172.21.15.42:0/1988446104 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe01806dd60 con 0x7fe0180a3d90 2024-09-16T04:12:06.799 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.786+0000 7fe017fff640 1 --2- 172.21.15.42:0/1988446104 >> v2:172.21.15.154:3300/0 conn(0x7fe018137540 0x7fe018137950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T04:12:06.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.790+0000 7fe016ffd640 1 --2- 172.21.15.42:0/1988446104 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe0180a3d90 0x7fe018137000 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fe0180a56c0 tx=0x7fe00001c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dff3ad536e254ad8 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:12:06.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.790+0000 7fe0167fc640 1 -- 172.21.15.42:0/1988446104 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0000b0070 con 0x7fe0180a3d90 2024-09-16T04:12:06.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.790+0000 7fe0167fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:06.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.790+0000 7fe0167fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:12:06.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.790+0000 7fe0167fc640 10 monclient: dump: 2024-09-16T04:12:06.806 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:12:06.806 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:12:06.806 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:06.806 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:06.806 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:06.806 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:12:06.806 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:12:06.806 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:12:06.806 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:12:06.806 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:12:06.806 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.790+0000 7fe0167fc640 10 monclient: _finish_auth 0 2024-09-16T04:12:06.806 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.790+0000 7fe0167fc640 10 monclient: _check_auth_tickets 2024-09-16T04:12:06.806 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.790+0000 7fe0167fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:12:06.807 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.790+0000 7fe0167fc640 1 -- 172.21.15.42:0/1988446104 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe0000a4e50 con 0x7fe0180a3d90 2024-09-16T04:12:06.807 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.790+0000 7fe01e1a6640 5 monclient: authenticate success, global_id 1034475 2024-09-16T04:12:06.807 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.790+0000 7fe01e1a6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:12:06.807 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.790+0000 7fe0167fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:12:06.807 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.790+0000 7fe01e1a6640 10 monclient: get_monmap_and_config success 2024-09-16T04:12:06.807 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.790+0000 7fe01e1a6640 4 set_mon_vals no callback set 2024-09-16T04:12:06.807 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.790+0000 7fe0167fc640 1 -- 172.21.15.42:0/1988446104 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0000ab820 con 0x7fe0180a3d90 2024-09-16T04:12:06.807 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.790+0000 7fe0167fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:06.807 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.790+0000 7fe0167fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:12:06.807 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.790+0000 7fe0167fc640 10 monclient: dump: 2024-09-16T04:12:06.807 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:12:06.808 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:12:06.808 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:06.808 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:06.808 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:06.808 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:12:06.808 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:12:06.808 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:12:06.808 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:12:06.808 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:12:06.808 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.790+0000 7fe01e1a6640 10 monclient: shutdown 2024-09-16T04:12:06.808 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.790+0000 7fe01e1a6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:12:06.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.790+0000 7fe01e1a6640 1 -- 172.21.15.42:0/1988446104 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe0180a3d90 msgr2=0x7fe018137000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:06.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.790+0000 7fe01e1a6640 1 --2- 172.21.15.42:0/1988446104 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe0180a3d90 0x7fe018137000 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fe0180a56c0 tx=0x7fe00001c400 comp rx=0 tx=0).stop 2024-09-16T04:12:06.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.790+0000 7fe01e1a6640 1 -- 172.21.15.42:0/1988446104 shutdown_connections 2024-09-16T04:12:06.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.790+0000 7fe01e1a6640 1 --2- 172.21.15.42:0/1988446104 >> v2:172.21.15.154:3300/0 conn(0x7fe018137540 0x7fe018137950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:12:06.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.790+0000 7fe01e1a6640 1 --2- 172.21.15.42:0/1988446104 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe0180a3d90 0x7fe018137000 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:12:06.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.790+0000 7fe01e1a6640 1 --2- 172.21.15.42:0/1988446104 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe0180a39b0 0x7fe018133b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:12:06.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.790+0000 7fe01e1a6640 1 -- 172.21.15.42:0/1988446104 >> 172.21.15.42:0/1988446104 conn(0x7fe0180a9fb0 msgr2=0x7fe0180aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:12:06.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.790+0000 7fe01e1a6640 1 -- 172.21.15.42:0/1988446104 shutdown_connections 2024-09-16T04:12:06.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.790+0000 7fe01e1a6640 1 -- 172.21.15.42:0/1988446104 wait complete. 2024-09-16T04:12:06.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.794+0000 7fe01e1a6640 10 monclient: build_initial_monmap 2024-09-16T04:12:06.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.794+0000 7fe01e1a6640 10 monclient: monmap: 2024-09-16T04:12:06.809 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:12:06.809 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:12:06.809 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:12:06.810 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:12:06.810 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:12:06.810 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:12:06.810 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:12:06.810 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:12:06.810 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:12:06.810 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:12:06.810 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.794+0000 7fe01e1a6640 1 librados: starting msgr at 2024-09-16T04:12:06.810 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.794+0000 7fe01e1a6640 1 librados: starting objecter 2024-09-16T04:12:06.810 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.794+0000 7fe01e1a6640 1 Processor -- start 2024-09-16T04:12:06.810 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.794+0000 7fe01e1a6640 1 -- start start 2024-09-16T04:12:06.810 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.794+0000 7fe01e1a6640 1 librados: setting wanted keys 2024-09-16T04:12:06.810 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.794+0000 7fe01e1a6640 1 librados: calling monclient init 2024-09-16T04:12:06.810 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.794+0000 7fe01e1a6640 10 monclient: init 2024-09-16T04:12:06.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.794+0000 7fe01e1a6640 10 monclient: _reopen_session rank -1 2024-09-16T04:12:06.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.794+0000 7fe01e1a6640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:12:06.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.794+0000 7fe01e1a6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe0180a39b0 0x7fe0181cf3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:12:06.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.794+0000 7fe01e1a6640 10 monclient(hunting): picked mon.noname-b con 0x7fe0180a39b0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:12:06.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.794+0000 7fe01e1a6640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe0180a3d90 0x7fe0180a2d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:12:06.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.794+0000 7fe01e1a6640 10 monclient(hunting): picked mon.noname-c con 0x7fe0180a3d90 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:12:06.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.794+0000 7fe01e1a6640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe018137540 0x7fe0180a32b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:12:06.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.794+0000 7fe01e1a6640 10 monclient(hunting): picked mon.noname-a con 0x7fe018137540 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:12:06.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.794+0000 7fe01e1a6640 10 monclient(hunting): start opening mon connection 2024-09-16T04:12:06.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.794+0000 7fe01e1a6640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fe018140dc0 con 0x7fe018137540 2024-09-16T04:12:06.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.794+0000 7fe01e1a6640 10 monclient(hunting): start opening mon connection 2024-09-16T04:12:06.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.794+0000 7fe01e1a6640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe018140f40 con 0x7fe0180a39b0 2024-09-16T04:12:06.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.794+0000 7fe01e1a6640 10 monclient(hunting): start opening mon connection 2024-09-16T04:12:06.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.794+0000 7fe01e1a6640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fe0181410c0 con 0x7fe0180a3d90 2024-09-16T04:12:06.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.794+0000 7fe01e1a6640 10 monclient(hunting): _renew_subs 2024-09-16T04:12:06.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.794+0000 7fe01e1a6640 10 monclient(hunting): authenticate will time out at 2773.765005s 2024-09-16T04:12:06.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.794+0000 7fe016ffd640 1 -- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe0180a3d90 msgr2=0x7fe0180a2d70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T04:12:06.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.794+0000 7fe016ffd640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe0180a3d90 0x7fe0180a2d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:12:06.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.794+0000 7fe017fff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe018137540 0x7fe0180a32b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:12:06.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.794+0000 7fe017fff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe018137540 0x7fe0180a32b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:38986/0 (socket says 172.21.15.42:38986) 2024-09-16T04:12:06.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.794+0000 7fe017fff640 1 -- 172.21.15.42:0/1125980072 learned_addr learned my addr 172.21.15.42:0/1125980072 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:12:06.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.794+0000 7fe017fff640 10 monclient(hunting): get_auth_request con 0x7fe018137540 auth_method 0 2024-09-16T04:12:06.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.794+0000 7fe017fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:12:06.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.794+0000 7fe017fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:06.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.794+0000 7fe017fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:06.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.794+0000 7fe017fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:12:06.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.794+0000 7fe017fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:12:06.813 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.794+0000 7fe017fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:12:06.813 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.794+0000 7fe0177fe640 1 --2- 172.21.15.42:0/1125980072 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe0180a39b0 0x7fe0181cf3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:12:06.813 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.794+0000 7fe017fff640 10 monclient(hunting): handle_auth_done global_id 1034478 payload 824 2024-09-16T04:12:06.813 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.794+0000 7fe017fff640 1 -- 172.21.15.42:0/1125980072 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe0180a3d90 msgr2=0x7fe0180a2d70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:12:06.813 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.794+0000 7fe017fff640 1 --2- 172.21.15.42:0/1125980072 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe0180a3d90 0x7fe0180a2d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:12:06.813 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.794+0000 7fe017fff640 1 -- 172.21.15.42:0/1125980072 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe0180a39b0 msgr2=0x7fe0181cf3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:06.813 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.794+0000 7fe017fff640 1 --2- 172.21.15.42:0/1125980072 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe0180a39b0 0x7fe0181cf3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:12:06.813 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.794+0000 7fe017fff640 10 monclient: _finish_hunting 0 2024-09-16T04:12:06.813 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.794+0000 7fe017fff640 1 monclient: found mon.noname-a 2024-09-16T04:12:06.813 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.794+0000 7fe017fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:12:06.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.794+0000 7fe017fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:12:06.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.794+0000 7fe017fff640 1 -- 172.21.15.42:0/1125980072 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe01813d400 con 0x7fe018137540 2024-09-16T04:12:06.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.794+0000 7fe0177fe640 10 monclient: get_auth_request con 0x7fe0180a39b0 auth_method 0 2024-09-16T04:12:06.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.794+0000 7fe0177fe640 1 --2- 172.21.15.42:0/1125980072 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe0180a39b0 0x7fe0181cf3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T04:12:06.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.794+0000 7fe017fff640 1 --2- 172.21.15.42:0/1125980072 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe018137540 0x7fe0180a32b0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fe00800e810 tx=0x7fe00800ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:12:06.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.794+0000 7fe00ffff640 10 client.?.objecter ms_handle_connect 0x7fe018137540 2024-09-16T04:12:06.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.794+0000 7fe00ffff640 10 client.?.objecter resend_mon_ops 2024-09-16T04:12:06.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.794+0000 7fe00ffff640 1 -- 172.21.15.42:0/1125980072 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe008017070 con 0x7fe018137540 2024-09-16T04:12:06.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.794+0000 7fe00ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:06.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.794+0000 7fe00ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:12:06.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.794+0000 7fe00ffff640 10 monclient: dump: 2024-09-16T04:12:06.814 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:12:06.814 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:12:06.814 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:06.815 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:06.815 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:06.815 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:12:06.815 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:12:06.815 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:12:06.815 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:12:06.815 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:12:06.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.794+0000 7fe00ffff640 10 monclient: _finish_auth 0 2024-09-16T04:12:06.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.794+0000 7fe00ffff640 10 monclient: _check_auth_tickets 2024-09-16T04:12:06.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.794+0000 7fe00ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:12:06.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.794+0000 7fe01e1a6640 5 monclient: authenticate success, global_id 1034478 2024-09-16T04:12:06.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.794+0000 7fe00ffff640 1 -- 172.21.15.42:0/1125980072 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe00800c8d0 con 0x7fe018137540 2024-09-16T04:12:06.816 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.794+0000 7fe01e1a6640 10 monclient: _renew_subs 2024-09-16T04:12:06.816 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.794+0000 7fe01e1a6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:12:06.816 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.794+0000 7fe01e1a6640 1 -- 172.21.15.42:0/1125980072 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe018132c20 con 0x7fe018137540 2024-09-16T04:12:06.816 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.794+0000 7fe00ffff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:12:06.816 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.794+0000 7fe00ffff640 1 -- 172.21.15.42:0/1125980072 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe00800f040 con 0x7fe018137540 2024-09-16T04:12:06.816 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.794+0000 7fe00ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:06.816 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.794+0000 7fe00ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:12:06.816 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.794+0000 7fe015ffb640 4 set_mon_vals no callback set 2024-09-16T04:12:06.816 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.794+0000 7fe00ffff640 10 monclient: dump: 2024-09-16T04:12:06.816 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:12:06.816 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:12:06.816 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:06.816 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:06.816 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:06.817 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:12:06.817 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:12:06.817 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:12:06.817 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:12:06.817 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:12:06.817 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.794+0000 7fe01e1a6640 10 client.1034478.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:12:06.817 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.794+0000 7fe01e1a6640 10 monclient: _renew_subs 2024-09-16T04:12:06.817 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.794+0000 7fe01e1a6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:12:06.817 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.794+0000 7fe01e1a6640 1 -- 172.21.15.42:0/1125980072 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe01809d240 con 0x7fe018137540 2024-09-16T04:12:06.817 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.794+0000 7fe01e1a6640 1 librados: init done 2024-09-16T04:12:06.817 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.794+0000 7fe01e1a6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:12:06.817 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.794+0000 7fe01e1a6640 1 --2- 172.21.15.42:0/1125980072 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe01813b2c0 0x7fe01813b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:12:06.817 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.794+0000 7fe01e1a6640 10 monclient(hunting): start opening mon connection 2024-09-16T04:12:06.818 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.794+0000 7fe01e1a6640 1 -- 172.21.15.42:0/1125980072 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fe0181342d0 con 0x7fe01813b2c0 2024-09-16T04:12:06.818 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.794+0000 7fe0177fe640 1 --2- 172.21.15.42:0/1125980072 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe01813b2c0 0x7fe01813b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:12:06.818 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.794+0000 7fe0177fe640 10 monclient: get_auth_request con 0x7fe01813b2c0 auth_method 0 2024-09-16T04:12:06.818 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.794+0000 7fe0177fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:12:06.818 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.794+0000 7fe0177fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:06.818 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.794+0000 7fe0177fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:06.818 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.794+0000 7fe0177fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:12:06.818 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.794+0000 7fe0177fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:12:06.818 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.794+0000 7fe0177fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:12:06.818 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.794+0000 7fe00ffff640 1 -- 172.21.15.42:0/1125980072 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fe00801a070 con 0x7fe018137540 2024-09-16T04:12:06.818 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.794+0000 7fe0177fe640 10 monclient(hunting): handle_auth_done global_id 1034481 payload 824 2024-09-16T04:12:06.818 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.794+0000 7fe0177fe640 1 -- 172.21.15.42:0/1125980072 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe018137000 con 0x7fe01813b2c0 2024-09-16T04:12:06.818 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.794+0000 7fe00ffff640 1 --2- 172.21.15.42:0/1125980072 >> v2:172.21.15.154:6812/254845342 conn(0x7fdff80377f0 0x7fdff8039cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:12:06.818 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.794+0000 7fe00ffff640 1 -- 172.21.15.42:0/1125980072 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(706..706 src has 451..706) ==== 5040+0+0 (secure 0 0 0) 0x7fe0080488d0 con 0x7fe018137540 2024-09-16T04:12:06.819 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.794+0000 7fe00ffff640 10 client.1034478.objecter ms_dispatch 0x7fe018000ba0 osd_map(706..706 src has 451..706) 2024-09-16T04:12:06.819 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.794+0000 7fe00ffff640 3 client.1034478.objecter handle_osd_map got epochs [706,706] > 0 2024-09-16T04:12:06.819 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.794+0000 7fe00ffff640 3 client.1034478.objecter handle_osd_map decoding full epoch 706 2024-09-16T04:12:06.819 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.794+0000 7fe0177fe640 1 --2- 172.21.15.42:0/1125980072 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe01813b2c0 0x7fe01813b6a0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fe01000ccd0 tx=0x7fe01000a5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:12:06.819 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.794+0000 7fe00ffff640 20 client.1034478.objecter dump_active .. 0 homeless 2024-09-16T04:12:06.819 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.794+0000 7fe00ffff640 10 client.1034478.objecter ms_handle_connect 0x7fe01813b2c0 2024-09-16T04:12:06.819 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.794+0000 7fe00ffff640 10 client.1034478.objecter resend_mon_ops 2024-09-16T04:12:06.819 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.794+0000 7fe00ffff640 1 -- 172.21.15.42:0/1125980072 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe010013070 con 0x7fe01813b2c0 2024-09-16T04:12:06.819 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.794+0000 7fe00ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:06.819 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.794+0000 7fe00ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:12:06.819 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.794+0000 7fe00ffff640 10 monclient: dump: 2024-09-16T04:12:06.819 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:12:06.819 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:12:06.819 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:06.820 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:06.820 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:06.820 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:12:06.820 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:12:06.820 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:12:06.820 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:12:06.820 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:12:06.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.794+0000 7fe016ffd640 1 --2- 172.21.15.42:0/1125980072 >> v2:172.21.15.154:6812/254845342 conn(0x7fdff80377f0 0x7fdff8039cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:12:06.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.794+0000 7fe016ffd640 10 monclient: get_auth_request con 0x7fdff80377f0 auth_method 0 2024-09-16T04:12:06.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.794+0000 7fe00ffff640 1 -- 172.21.15.42:0/1125980072 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fe01000ae90 con 0x7fe01813b2c0 2024-09-16T04:12:06.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.794+0000 7fe00ffff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:12:06.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.794+0000 7fe00ffff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:12:06.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.794+0000 7fe00ffff640 1 -- 172.21.15.42:0/1125980072 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe01813b2c0 msgr2=0x7fe01813b6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:06.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.794+0000 7fe00ffff640 1 --2- 172.21.15.42:0/1125980072 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe01813b2c0 0x7fe01813b6a0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fe01000ccd0 tx=0x7fe01000a5f0 comp rx=0 tx=0).stop 2024-09-16T04:12:06.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.794+0000 7fe00ffff640 1 -- 172.21.15.42:0/1125980072 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe01813b2c0 msgr2=0x7fe01813b6a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:12:06.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.794+0000 7fe00ffff640 1 --2- 172.21.15.42:0/1125980072 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe01813b2c0 0x7fe01813b6a0 secure :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fe01000ccd0 tx=0x7fe01000a5f0 comp rx=0 tx=0).stop 2024-09-16T04:12:06.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.798+0000 7fe016ffd640 1 --2- 172.21.15.42:0/1125980072 >> v2:172.21.15.154:6812/254845342 conn(0x7fdff80377f0 0x7fdff8039cb0 secure :-1 s=READY pgs=3957 cs=0 l=1 rev1=1 crypto rx=0x7fe0000ad000 tx=0x7fe000008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:12:06.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.798+0000 7fe00ffff640 10 client.1034478.objecter ms_handle_connect 0x7fdff80377f0 2024-09-16T04:12:06.823 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.802+0000 7fe01e1a6640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:12:06.823 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.802+0000 7fe01e1a6640 1 --2- 172.21.15.42:0/1125980072 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe01813bbe0 0x7fe018138130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:12:06.823 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.802+0000 7fe01e1a6640 10 monclient(hunting): start opening mon connection 2024-09-16T04:12:06.823 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.802+0000 7fe01e1a6640 1 -- 172.21.15.42:0/1125980072 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fe0180a5750 con 0x7fe01813bbe0 2024-09-16T04:12:06.823 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.802+0000 7fe0177fe640 1 --2- 172.21.15.42:0/1125980072 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe01813bbe0 0x7fe018138130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:12:06.823 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.802+0000 7fe0177fe640 10 monclient: get_auth_request con 0x7fe01813bbe0 auth_method 0 2024-09-16T04:12:06.823 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.802+0000 7fe0177fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:12:06.824 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.802+0000 7fe0177fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:06.824 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.802+0000 7fe0177fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:06.824 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.802+0000 7fe0177fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:12:06.824 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.802+0000 7fe0177fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:12:06.824 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.802+0000 7fe0177fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:12:06.824 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.802+0000 7fe0177fe640 10 monclient(hunting): handle_auth_done global_id 1034484 payload 824 2024-09-16T04:12:06.824 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.802+0000 7fe0177fe640 1 -- 172.21.15.42:0/1125980072 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fe0181cf930 con 0x7fe01813bbe0 2024-09-16T04:12:06.824 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.802+0000 7fe0177fe640 1 --2- 172.21.15.42:0/1125980072 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe01813bbe0 0x7fe018138130 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fe01000ccd0 tx=0x7fe010005d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:12:06.824 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.802+0000 7fe00ffff640 10 client.1034478.objecter ms_handle_connect 0x7fe01813bbe0 2024-09-16T04:12:06.824 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.802+0000 7fe00ffff640 10 client.1034478.objecter resend_mon_ops 2024-09-16T04:12:06.824 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.802+0000 7fe00ffff640 1 -- 172.21.15.42:0/1125980072 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe010018860 con 0x7fe01813bbe0 2024-09-16T04:12:06.824 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.802+0000 7fe00ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:06.825 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.802+0000 7fe00ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:12:06.825 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.802+0000 7fe00ffff640 10 monclient: dump: 2024-09-16T04:12:06.825 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:12:06.825 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:12:06.825 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:06.825 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:06.825 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:06.825 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:12:06.825 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:12:06.825 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:12:06.825 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:12:06.825 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:12:06.825 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.802+0000 7fe00ffff640 1 -- 172.21.15.42:0/1125980072 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4871 (secure 0 0 0) 0x7fe010018860 con 0x7fe01813bbe0 2024-09-16T04:12:06.825 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.802+0000 7fe00ffff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:12:06.826 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.802+0000 7fe00ffff640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:12:06.826 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.802+0000 7fe00ffff640 1 -- 172.21.15.42:0/1125980072 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe01813bbe0 msgr2=0x7fe018138130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:06.826 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.802+0000 7fe00ffff640 1 --2- 172.21.15.42:0/1125980072 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe01813bbe0 0x7fe018138130 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fe01000ccd0 tx=0x7fe010005d60 comp rx=0 tx=0).stop 2024-09-16T04:12:06.826 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.802+0000 7fe00ffff640 1 -- 172.21.15.42:0/1125980072 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe01813bbe0 msgr2=0x7fe018138130 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:12:06.826 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.802+0000 7fe00ffff640 1 --2- 172.21.15.42:0/1125980072 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe01813bbe0 0x7fe018138130 secure :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fe01000ccd0 tx=0x7fe010005d60 comp rx=0 tx=0).stop 2024-09-16T04:12:06.826 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T04:12:06.827 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:12:06.827 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:12:06.827 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "leader", 2024-09-16T04:12:06.827 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 272, 2024-09-16T04:12:06.827 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 206738, 2024-09-16T04:12:06.827 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T04:12:06.827 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T04:12:06.827 INFO:teuthology.orchestra.run.smithi042.stdout: 1 2024-09-16T04:12:06.827 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:12:06.827 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 1, 2024-09-16T04:12:06.828 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:12:06.828 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T04:12:06.828 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T04:12:06.828 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:12:06.828 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:12:06.828 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:12:06.828 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:12:06.828 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:12:06.828 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:12:06.828 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:12:06.828 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:12:06.828 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:12:06.828 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:12:06.829 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:12:06.829 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:12:06.829 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:12:06.829 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T04:12:06.829 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:12:06.829 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:12:06.829 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:12:06.829 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:12:06.829 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:12:06.829 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:12:06.829 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:12:06.829 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:12:06.829 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:12:06.830 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:12:06.830 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:12:06.830 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:12:06.830 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:12:06.830 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T04:12:06.830 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T04:12:06.830 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T04:12:06.830 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T04:12:06.830 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T04:12:06.830 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T04:12:06.830 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:12:06.830 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:12:06.830 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T04:12:06.830 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T04:12:06.831 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T04:12:06.831 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T04:12:06.831 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T04:12:06.831 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T04:12:06.831 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T04:12:06.831 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:12:06.831 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T04:12:06.831 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:12:06.831 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:12:06.831 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:12:06.831 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:12:06.831 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:12:06.831 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:12:06.831 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:12:06.832 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:12:06.832 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:12:06.832 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:12:06.832 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:12:06.832 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:12:06.832 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T04:12:06.832 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:12:06.832 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T04:12:06.832 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:12:06.832 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:12:06.832 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:12:06.832 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:12:06.833 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:12:06.833 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:12:06.833 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:12:06.833 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T04:12:06.833 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:12:06.833 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:12:06.833 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:12:06.833 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:12:06.833 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T04:12:06.833 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:12:06.833 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:12:06.833 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:12:06.833 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:12:06.833 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T04:12:06.834 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T04:12:06.834 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:12:06.834 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:12:06.834 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:12:06.834 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:12:06.834 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:12:06.834 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:12:06.834 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:12:06.834 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:12:06.834 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:12:06.834 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:12:06.834 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:12:06.834 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T04:12:06.834 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:12:06.835 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:12:06.835 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:12:06.835 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:12:06.835 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T04:12:06.835 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:12:06.835 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:12:06.835 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:12:06.835 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:12:06.835 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T04:12:06.835 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T04:12:06.835 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:12:06.835 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:12:06.835 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:12:06.836 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:12:06.836 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:12:06.836 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:12:06.836 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:12:06.836 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:12:06.836 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:12:06.836 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:12:06.836 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:12:06.836 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T04:12:06.836 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:12:06.836 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:12:06.836 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:12:06.836 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:12:06.837 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T04:12:06.837 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:12:06.837 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:12:06.837 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:12:06.837 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:12:06.837 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T04:12:06.837 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T04:12:06.837 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:12:06.837 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:12:06.837 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:12:06.837 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:12:06.837 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:12:06.837 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:12:06.837 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T04:12:06.838 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T04:12:06.838 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:12:06.838 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:12:06.838 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:12:06.838 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:12:06.838 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:12:06.838 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:12:06.838 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T04:12:06.838 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:12:06.838 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:12:06.838 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:12:06.838 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 6 2024-09-16T04:12:06.839 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:12:06.839 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:12:06.839 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T04:12:06.839 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:12:06.839 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:12:06.839 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:12:06.839 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 6 2024-09-16T04:12:06.839 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:12:06.839 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:12:06.839 INFO:teuthology.orchestra.run.smithi042.stdout: "mgr": [ 2024-09-16T04:12:06.839 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:12:06.839 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:12:06.839 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:12:06.840 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:12:06.840 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:12:06.840 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:12:06.840 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:12:06.840 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T04:12:06.840 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T04:12:06.841 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.802+0000 7fe01e1a6640 10 librados: watch_flush enter 2024-09-16T04:12:06.842 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.802+0000 7fe01e1a6640 10 librados: watch_flush exit 2024-09-16T04:12:06.842 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.802+0000 7fe01e1a6640 20 client.1034478.objecter shutdown clearing up homeless session... 2024-09-16T04:12:06.842 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.802+0000 7fe01e1a6640 10 client.1034478.objecter successfully canceled tick 2024-09-16T04:12:06.842 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.802+0000 7fe01e1a6640 1 -- 172.21.15.42:0/1125980072 >> v2:172.21.15.154:6812/254845342 conn(0x7fdff80377f0 msgr2=0x7fdff8039cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:06.842 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.802+0000 7fe01e1a6640 1 --2- 172.21.15.42:0/1125980072 >> v2:172.21.15.154:6812/254845342 conn(0x7fdff80377f0 0x7fdff8039cb0 secure :-1 s=READY pgs=3957 cs=0 l=1 rev1=1 crypto rx=0x7fe0000ad000 tx=0x7fe000008040 comp rx=0 tx=0).stop 2024-09-16T04:12:06.842 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.802+0000 7fe01e1a6640 10 monclient: shutdown 2024-09-16T04:12:06.842 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.802+0000 7fe01e1a6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:12:06.842 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.802+0000 7fe01e1a6640 1 -- 172.21.15.42:0/1125980072 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe018137540 msgr2=0x7fe0180a32b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:06.842 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.802+0000 7fe01e1a6640 1 --2- 172.21.15.42:0/1125980072 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe018137540 0x7fe0180a32b0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fe00800e810 tx=0x7fe00800ecd0 comp rx=0 tx=0).stop 2024-09-16T04:12:06.843 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.802+0000 7fe017fff640 1 -- 172.21.15.42:0/1125980072 reap_dead start 2024-09-16T04:12:06.843 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.802+0000 7fe017fff640 1 -- 172.21.15.42:0/1125980072 reap_dead start 2024-09-16T04:12:06.843 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.802+0000 7fe01e1a6640 1 -- 172.21.15.42:0/1125980072 shutdown_connections 2024-09-16T04:12:06.843 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.802+0000 7fe01e1a6640 1 -- 172.21.15.42:0/1125980072 >> 172.21.15.42:0/1125980072 conn(0x7fe0180a9fb0 msgr2=0x7fe0180a8010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:12:06.843 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.802+0000 7fe01e1a6640 1 -- 172.21.15.42:0/1125980072 shutdown_connections 2024-09-16T04:12:06.843 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.802+0000 7fe01e1a6640 1 -- 172.21.15.42:0/1125980072 wait complete. 2024-09-16T04:12:06.843 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.802+0000 7fe01e1a6640 1 librados: shutdown 2024-09-16T04:12:06.843 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T04:12:06.844 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-16T04:12:06.844 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T04:12:06.962 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.958+0000 7fdbe6706640 10 monclient: get_monmap_and_config 2024-09-16T04:12:06.962 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.958+0000 7fdbe6706640 10 monclient: build_initial_monmap 2024-09-16T04:12:06.962 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.958+0000 7fdbe6706640 10 monclient: monmap: 2024-09-16T04:12:06.963 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:12:06.963 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:12:06.963 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:12:06.961313+0000 2024-09-16T04:12:06.963 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:12:06.961313+0000 2024-09-16T04:12:06.963 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:12:06.963 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:12:06.963 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:12:06.963 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:12:06.963 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:12:06.963 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:12:06.963 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:12:06.963 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:12:06.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.958+0000 7fdbe6706640 1 Processor -- start 2024-09-16T04:12:06.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.958+0000 7fdbe6706640 1 -- start start 2024-09-16T04:12:06.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.958+0000 7fdbe6706640 10 monclient: init 2024-09-16T04:12:06.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.958+0000 7fdbe6706640 10 monclient: _reopen_session rank -1 2024-09-16T04:12:06.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.958+0000 7fdbe6706640 10 monclient: _add_conns ranks=[3,4,1,0,2] 2024-09-16T04:12:06.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.958+0000 7fdbe6706640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fdbe00a1330 0x7fdbe00a1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:12:06.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.958+0000 7fdbe6706640 10 monclient(hunting): picked mon.noname-a con 0x7fdbe00a1330 addr v2:172.21.15.42:3300/0 2024-09-16T04:12:06.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.958+0000 7fdbe6706640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fdbe006dac0 0x7fdbe00a1c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:12:06.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.958+0000 7fdbe6706640 10 monclient(hunting): picked mon.noname-c con 0x7fdbe006dac0 addr v2:172.21.15.154:3300/0 2024-09-16T04:12:06.967 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.958+0000 7fdbe6706640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fdbe0135100 addr v1:172.21.15.154:6789/0 2024-09-16T04:12:06.967 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.958+0000 7fdbe6706640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdbe009c7d0 con 0x7fdbe0135100 2024-09-16T04:12:06.967 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.958+0000 7fdbe6706640 10 monclient(hunting): start opening mon connection 2024-09-16T04:12:06.967 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.958+0000 7fdbe6706640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7fdbe009c090 con 0x7fdbe00a1330 2024-09-16T04:12:06.967 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.958+0000 7fdbe6706640 10 monclient(hunting): start opening mon connection 2024-09-16T04:12:06.967 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.958+0000 7fdbe6706640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fdbe009f610 con 0x7fdbe006dac0 2024-09-16T04:12:06.967 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.958+0000 7fdbe6706640 10 monclient(hunting): _renew_subs 2024-09-16T04:12:06.967 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.958+0000 7fdbe6706640 10 monclient(hunting): authenticate will time out at 2773.930696s 2024-09-16T04:12:06.967 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.958+0000 7fdbdffff640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fdbe00a1330 0x7fdbe00a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:12:06.967 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.958+0000 7fdbdffff640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fdbe00a1330 0x7fdbe00a1710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:39012/0 (socket says 172.21.15.42:39012) 2024-09-16T04:12:06.967 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.958+0000 7fdbdffff640 1 -- 172.21.15.42:0/1866698641 learned_addr learned my addr 172.21.15.42:0/1866698641 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:12:06.967 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.958+0000 7fdbdffff640 10 monclient(hunting): get_auth_request con 0x7fdbe00a1330 auth_method 0 2024-09-16T04:12:06.968 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.958+0000 7fdbdffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:12:06.968 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.958+0000 7fdbdffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:06.968 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.958+0000 7fdbdffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:06.968 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.958+0000 7fdbdffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:12:06.968 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.958+0000 7fdbdffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:12:06.968 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.958+0000 7fdbdffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:12:06.968 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.958+0000 7fdbdffff640 10 monclient(hunting): handle_auth_done global_id 1034487 payload 274 2024-09-16T04:12:06.968 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.958+0000 7fdbdffff640 1 -- 172.21.15.42:0/1866698641 >> v2:172.21.15.154:3300/0 conn(0x7fdbe006dac0 msgr2=0x7fdbe00a1c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:12:06.968 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.958+0000 7fdbdffff640 1 --2- 172.21.15.42:0/1866698641 >> v2:172.21.15.154:3300/0 conn(0x7fdbe006dac0 0x7fdbe00a1c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:12:06.968 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.958+0000 7fdbdffff640 1 -- 172.21.15.42:0/1866698641 >> v1:172.21.15.154:6789/0 conn(0x7fdbe0135100 legacy=0x7fdbe0135550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:12:06.968 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.958+0000 7fdbdffff640 10 monclient: _finish_hunting 0 2024-09-16T04:12:06.968 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.958+0000 7fdbdffff640 1 monclient: found mon.noname-a 2024-09-16T04:12:06.968 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.958+0000 7fdbdffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:12:06.969 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.958+0000 7fdbdffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:12:06.969 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.958+0000 7fdbdffff640 1 -- 172.21.15.42:0/1866698641 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdbe009f8f0 con 0x7fdbe00a1330 2024-09-16T04:12:06.969 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.962+0000 7fdbdffff640 1 --2- 172.21.15.42:0/1866698641 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdbe00a1330 0x7fdbe00a1710 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fdbe006da10 tx=0x7fdbd401c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12c257eddf2c89b1 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:12:06.975 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.962+0000 7fdbdeffd640 1 -- 172.21.15.42:0/1866698641 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbd40aa610 con 0x7fdbe00a1330 2024-09-16T04:12:06.976 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.962+0000 7fdbdeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:06.976 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.962+0000 7fdbdeffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:12:06.976 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.962+0000 7fdbdeffd640 10 monclient: dump: 2024-09-16T04:12:06.976 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:12:06.976 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:12:06.976 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:06.976 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:06.976 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:06.976 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:12:06.976 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:12:06.976 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:12:06.976 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:12:06.977 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:12:06.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.962+0000 7fdbdeffd640 10 monclient: _finish_auth 0 2024-09-16T04:12:06.977 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.962+0000 7fdbdeffd640 10 monclient: _check_auth_tickets 2024-09-16T04:12:06.978 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.962+0000 7fdbdeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:12:06.978 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.962+0000 7fdbdeffd640 1 -- 172.21.15.42:0/1866698641 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdbd40af070 con 0x7fdbe00a1330 2024-09-16T04:12:06.978 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.962+0000 7fdbe6706640 5 monclient: authenticate success, global_id 1034487 2024-09-16T04:12:06.978 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.962+0000 7fdbe6706640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:12:06.978 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.962+0000 7fdbdeffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:12:06.978 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.962+0000 7fdbe6706640 10 monclient: get_monmap_and_config success 2024-09-16T04:12:06.978 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.962+0000 7fdbdeffd640 1 -- 172.21.15.42:0/1866698641 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbd40aa9d0 con 0x7fdbe00a1330 2024-09-16T04:12:06.978 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.962+0000 7fdbe6706640 4 set_mon_vals no callback set 2024-09-16T04:12:06.978 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.962+0000 7fdbdeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:06.978 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.962+0000 7fdbdeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:12:06.978 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.962+0000 7fdbdeffd640 10 monclient: dump: 2024-09-16T04:12:06.978 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:12:06.978 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:12:06.979 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:06.979 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:06.979 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:06.979 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:12:06.979 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:12:06.979 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:12:06.979 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:12:06.979 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:12:06.979 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.962+0000 7fdbe6706640 10 monclient: shutdown 2024-09-16T04:12:06.979 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.962+0000 7fdbe6706640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:12:06.979 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.962+0000 7fdbe6706640 1 -- 172.21.15.42:0/1866698641 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdbe00a1330 msgr2=0x7fdbe00a1710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:06.979 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.962+0000 7fdbe6706640 1 --2- 172.21.15.42:0/1866698641 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdbe00a1330 0x7fdbe00a1710 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fdbe006da10 tx=0x7fdbd401c490 comp rx=0 tx=0).stop 2024-09-16T04:12:06.979 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.962+0000 7fdbe6706640 1 -- 172.21.15.42:0/1866698641 shutdown_connections 2024-09-16T04:12:06.980 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.962+0000 7fdbe6706640 1 --2- 172.21.15.42:0/1866698641 >> v2:172.21.15.154:3300/0 conn(0x7fdbe006dac0 0x7fdbe00a1c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:12:06.980 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.962+0000 7fdbe6706640 1 --2- 172.21.15.42:0/1866698641 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdbe00a1330 0x7fdbe00a1710 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:12:06.980 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.962+0000 7fdbe6706640 1 -- 172.21.15.42:0/1866698641 >> 172.21.15.42:0/1866698641 conn(0x7fdbe00a2420 msgr2=0x7fdbe00a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:12:06.980 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.962+0000 7fdbe6706640 1 -- 172.21.15.42:0/1866698641 shutdown_connections 2024-09-16T04:12:06.980 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.962+0000 7fdbe6706640 1 -- 172.21.15.42:0/1866698641 wait complete. 2024-09-16T04:12:06.980 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.962+0000 7fdbe6706640 10 monclient: build_initial_monmap 2024-09-16T04:12:06.980 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.962+0000 7fdbe6706640 10 monclient: monmap: 2024-09-16T04:12:06.980 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:12:06.980 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:12:06.980 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:12:06.980 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:12:06.980 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:12:06.980 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:12:06.980 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:12:06.981 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:12:06.981 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:12:06.981 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:12:06.981 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.962+0000 7fdbe6706640 1 librados: starting msgr at 2024-09-16T04:12:06.981 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.962+0000 7fdbe6706640 1 librados: starting objecter 2024-09-16T04:12:06.981 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.962+0000 7fdbe6706640 1 Processor -- start 2024-09-16T04:12:06.981 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.962+0000 7fdbe6706640 1 -- start start 2024-09-16T04:12:06.981 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.962+0000 7fdbe6706640 1 librados: setting wanted keys 2024-09-16T04:12:06.981 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.962+0000 7fdbe6706640 1 librados: calling monclient init 2024-09-16T04:12:06.981 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.962+0000 7fdbe6706640 10 monclient: init 2024-09-16T04:12:06.981 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.962+0000 7fdbe6706640 10 monclient: _reopen_session rank -1 2024-09-16T04:12:06.982 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.962+0000 7fdbe6706640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:12:06.982 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.962+0000 7fdbe6706640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdbe00a1330 0x7fdbe01cf1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:12:06.982 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.962+0000 7fdbe6706640 10 monclient(hunting): picked mon.noname-a con 0x7fdbe00a1330 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:12:06.982 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.962+0000 7fdbe6706640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fdbe0135100 0x7fdbe01334c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:12:06.982 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.962+0000 7fdbe6706640 10 monclient(hunting): picked mon.noname-c con 0x7fdbe0135100 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:12:06.982 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.962+0000 7fdbe6706640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdbe0133a00 0x7fdbe011bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:12:06.982 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.962+0000 7fdbe6706640 10 monclient(hunting): picked mon.noname-b con 0x7fdbe0133a00 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:12:06.982 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.962+0000 7fdbe6706640 10 monclient(hunting): start opening mon connection 2024-09-16T04:12:06.982 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.962+0000 7fdbe6706640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fdbe013fb80 con 0x7fdbe00a1330 2024-09-16T04:12:06.982 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.962+0000 7fdbe6706640 10 monclient(hunting): start opening mon connection 2024-09-16T04:12:06.982 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.962+0000 7fdbe6706640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fdbe009ef80 con 0x7fdbe0133a00 2024-09-16T04:12:06.982 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.962+0000 7fdbe6706640 10 monclient(hunting): start opening mon connection 2024-09-16T04:12:06.982 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.962+0000 7fdbe6706640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fdbe013fd00 con 0x7fdbe0135100 2024-09-16T04:12:06.982 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.962+0000 7fdbe6706640 10 monclient(hunting): _renew_subs 2024-09-16T04:12:06.983 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.962+0000 7fdbe6706640 10 monclient(hunting): authenticate will time out at 2773.935506s 2024-09-16T04:12:06.983 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.962+0000 7fdbdf7fe640 1 -- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fdbe0135100 msgr2=0x7fdbe01334c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T04:12:06.983 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.962+0000 7fdbdf7fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fdbe0135100 0x7fdbe01334c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:12:06.983 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.962+0000 7fdbdffff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdbe00a1330 0x7fdbe01cf1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:12:06.983 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.962+0000 7fdbdffff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdbe00a1330 0x7fdbe01cf1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:39028/0 (socket says 172.21.15.42:39028) 2024-09-16T04:12:06.983 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.962+0000 7fdbdffff640 1 -- 172.21.15.42:0/3736097204 learned_addr learned my addr 172.21.15.42:0/3736097204 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:12:06.983 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.962+0000 7fdbdffff640 10 monclient(hunting): get_auth_request con 0x7fdbe00a1330 auth_method 0 2024-09-16T04:12:06.983 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.962+0000 7fdbdffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:12:06.983 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.962+0000 7fdbdffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:06.983 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.962+0000 7fdbdffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:06.983 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.962+0000 7fdbdffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:12:06.983 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.962+0000 7fdbdffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:12:06.983 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.962+0000 7fdbdffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:12:06.984 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.962+0000 7fdbe4ca3640 1 --2- 172.21.15.42:0/3736097204 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdbe0133a00 0x7fdbe011bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:12:06.984 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.962+0000 7fdbdffff640 10 monclient(hunting): handle_auth_done global_id 1034490 payload 824 2024-09-16T04:12:06.984 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.962+0000 7fdbdffff640 1 -- 172.21.15.42:0/3736097204 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fdbe0135100 msgr2=0x7fdbe01334c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:12:06.984 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.962+0000 7fdbdffff640 1 --2- 172.21.15.42:0/3736097204 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fdbe0135100 0x7fdbe01334c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:12:06.984 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.962+0000 7fdbdffff640 1 -- 172.21.15.42:0/3736097204 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdbe0133a00 msgr2=0x7fdbe011bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:06.984 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.962+0000 7fdbdffff640 1 --2- 172.21.15.42:0/3736097204 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdbe0133a00 0x7fdbe011bf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:12:06.984 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.962+0000 7fdbdffff640 10 monclient: _finish_hunting 0 2024-09-16T04:12:06.984 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.962+0000 7fdbdffff640 1 monclient: found mon.noname-a 2024-09-16T04:12:06.984 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.962+0000 7fdbdffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:12:06.984 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.962+0000 7fdbdffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:12:06.984 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.962+0000 7fdbdffff640 1 -- 172.21.15.42:0/3736097204 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdbe011e020 con 0x7fdbe00a1330 2024-09-16T04:12:06.984 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.962+0000 7fdbe4ca3640 10 monclient: get_auth_request con 0x7fdbe0133a00 auth_method 0 2024-09-16T04:12:06.984 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.962+0000 7fdbe4ca3640 1 --2- 172.21.15.42:0/3736097204 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdbe0133a00 0x7fdbe011bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T04:12:06.985 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.966+0000 7fdbdffff640 1 --2- 172.21.15.42:0/3736097204 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdbe00a1330 0x7fdbe01cf1c0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fdbd40ac040 tx=0x7fdbd401acd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:12:06.985 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.966+0000 7fdbdcff9640 10 client.?.objecter ms_handle_connect 0x7fdbe00a1330 2024-09-16T04:12:06.985 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.966+0000 7fdbdcff9640 10 client.?.objecter resend_mon_ops 2024-09-16T04:12:06.985 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.966+0000 7fdbdcff9640 1 -- 172.21.15.42:0/3736097204 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbd40b3800 con 0x7fdbe00a1330 2024-09-16T04:12:06.985 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.966+0000 7fdbdcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:06.985 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.966+0000 7fdbdcff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:12:06.985 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.966+0000 7fdbdcff9640 10 monclient: dump: 2024-09-16T04:12:06.985 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:12:06.985 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:12:06.985 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:06.985 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:06.985 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:06.985 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:12:06.985 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:12:06.986 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:12:06.986 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:12:06.986 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:12:06.986 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.966+0000 7fdbdcff9640 10 monclient: _finish_auth 0 2024-09-16T04:12:06.986 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.966+0000 7fdbdcff9640 10 monclient: _check_auth_tickets 2024-09-16T04:12:06.986 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.966+0000 7fdbdcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:12:06.986 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.966+0000 7fdbdcff9640 1 -- 172.21.15.42:0/3736097204 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdbd40b39c0 con 0x7fdbe00a1330 2024-09-16T04:12:06.986 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.966+0000 7fdbe6706640 5 monclient: authenticate success, global_id 1034490 2024-09-16T04:12:06.986 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.966+0000 7fdbe6706640 10 monclient: _renew_subs 2024-09-16T04:12:06.986 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.966+0000 7fdbe6706640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:12:06.986 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.966+0000 7fdbe6706640 1 -- 172.21.15.42:0/3736097204 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdbe011e6e0 con 0x7fdbe00a1330 2024-09-16T04:12:06.986 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.966+0000 7fdbdcff9640 10 monclient: handle_config config(0 keys) 2024-09-16T04:12:06.986 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.966+0000 7fdbdcff9640 1 -- 172.21.15.42:0/3736097204 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbd40b3db0 con 0x7fdbe00a1330 2024-09-16T04:12:06.987 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.966+0000 7fdbdcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:06.987 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.966+0000 7fdbdcff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:12:06.987 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.966+0000 7fdbdcff9640 10 monclient: dump: 2024-09-16T04:12:06.987 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:12:06.987 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:12:06.987 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:06.987 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:06.987 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:06.987 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:12:06.987 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:12:06.987 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:12:06.987 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:12:06.987 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:12:06.987 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.966+0000 7fdbe6706640 10 client.1034490.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:12:06.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.966+0000 7fdbe6706640 10 monclient: _renew_subs 2024-09-16T04:12:06.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.966+0000 7fdbe6706640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:12:06.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.966+0000 7fdbe6706640 1 -- 172.21.15.42:0/3736097204 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdbe00a5360 con 0x7fdbe00a1330 2024-09-16T04:12:06.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.966+0000 7fdbde7fc640 4 set_mon_vals no callback set 2024-09-16T04:12:06.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.966+0000 7fdbe6706640 1 librados: init done 2024-09-16T04:12:06.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.966+0000 7fdbe6706640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:12:06.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.966+0000 7fdbe6706640 1 --2- 172.21.15.42:0/3736097204 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdbe011f740 0x7fdbe002c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:12:06.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.966+0000 7fdbe6706640 10 monclient(hunting): start opening mon connection 2024-09-16T04:12:06.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.966+0000 7fdbe6706640 1 -- 172.21.15.42:0/3736097204 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fdbe009e2e0 con 0x7fdbe011f740 2024-09-16T04:12:06.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.966+0000 7fdbdcff9640 1 -- 172.21.15.42:0/3736097204 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fdbd40c6440 con 0x7fdbe00a1330 2024-09-16T04:12:06.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.966+0000 7fdbdcff9640 1 --2- 172.21.15.42:0/3736097204 >> v2:172.21.15.154:6812/254845342 conn(0x7fdbcc0377f0 0x7fdbcc039cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:12:06.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.966+0000 7fdbdf7fe640 1 --2- 172.21.15.42:0/3736097204 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdbe011f740 0x7fdbe002c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:12:06.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.966+0000 7fdbdcff9640 1 -- 172.21.15.42:0/3736097204 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(706..706 src has 451..706) ==== 5040+0+0 (secure 0 0 0) 0x7fdbd40e3760 con 0x7fdbe00a1330 2024-09-16T04:12:06.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.966+0000 7fdbdcff9640 10 client.1034490.objecter ms_dispatch 0x7fdbe0000ba0 osd_map(706..706 src has 451..706) 2024-09-16T04:12:06.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.966+0000 7fdbdf7fe640 10 monclient: get_auth_request con 0x7fdbe011f740 auth_method 0 2024-09-16T04:12:06.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.966+0000 7fdbdf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:12:06.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.966+0000 7fdbdcff9640 3 client.1034490.objecter handle_osd_map got epochs [706,706] > 0 2024-09-16T04:12:06.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.966+0000 7fdbdf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:06.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.966+0000 7fdbdf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:06.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.966+0000 7fdbdcff9640 3 client.1034490.objecter handle_osd_map decoding full epoch 706 2024-09-16T04:12:06.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.966+0000 7fdbdcff9640 20 client.1034490.objecter dump_active .. 0 homeless 2024-09-16T04:12:06.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.966+0000 7fdbdf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:12:06.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.966+0000 7fdbdf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:12:06.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.966+0000 7fdbdf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:12:06.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.966+0000 7fdbe4ca3640 1 --2- 172.21.15.42:0/3736097204 >> v2:172.21.15.154:6812/254845342 conn(0x7fdbcc0377f0 0x7fdbcc039cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:12:06.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.966+0000 7fdbe4ca3640 10 monclient: get_auth_request con 0x7fdbcc0377f0 auth_method 0 2024-09-16T04:12:06.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.966+0000 7fdbdf7fe640 10 monclient(hunting): handle_auth_done global_id 1074160 payload 824 2024-09-16T04:12:06.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.966+0000 7fdbdf7fe640 1 -- 172.21.15.42:0/3736097204 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fdbe0073a30 con 0x7fdbe011f740 2024-09-16T04:12:06.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.966+0000 7fdbdcff9640 10 client.1034490.objecter ms_handle_connect 0x7fdbe011f740 2024-09-16T04:12:06.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.966+0000 7fdbdf7fe640 1 --2- 172.21.15.42:0/3736097204 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdbe011f740 0x7fdbe002c650 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fdbd0007fd0 tx=0x7fdbd000c810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:12:06.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.966+0000 7fdbdcff9640 10 client.1034490.objecter resend_mon_ops 2024-09-16T04:12:06.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.966+0000 7fdbdcff9640 10 client.1034490.objecter ms_handle_connect 0x7fdbcc0377f0 2024-09-16T04:12:06.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.966+0000 7fdbe4ca3640 1 --2- 172.21.15.42:0/3736097204 >> v2:172.21.15.154:6812/254845342 conn(0x7fdbcc0377f0 0x7fdbcc039cb0 secure :-1 s=READY pgs=3958 cs=0 l=1 rev1=1 crypto rx=0x7fdbd8007fd0 tx=0x7fdbd8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:12:06.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.966+0000 7fdbdcff9640 1 -- 172.21.15.42:0/3736097204 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbd0017070 con 0x7fdbe011f740 2024-09-16T04:12:06.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.966+0000 7fdbdcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:06.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.966+0000 7fdbdcff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:12:06.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.966+0000 7fdbdcff9640 10 monclient: dump: 2024-09-16T04:12:06.990 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:12:06.990 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:12:06.990 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:06.991 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:06.991 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:06.991 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:12:06.991 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:12:06.991 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:12:06.991 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:12:06.991 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:12:06.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.966+0000 7fdbdcff9640 1 -- 172.21.15.42:0/3736097204 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fdbd0005d70 con 0x7fdbe011f740 2024-09-16T04:12:06.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.966+0000 7fdbdcff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:12:06.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.966+0000 7fdbdcff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:12:06.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.966+0000 7fdbdcff9640 1 -- 172.21.15.42:0/3736097204 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdbe011f740 msgr2=0x7fdbe002c650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:06.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.966+0000 7fdbdcff9640 1 --2- 172.21.15.42:0/3736097204 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdbe011f740 0x7fdbe002c650 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fdbd0007fd0 tx=0x7fdbd000c810 comp rx=0 tx=0).stop 2024-09-16T04:12:06.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.966+0000 7fdbdcff9640 1 -- 172.21.15.42:0/3736097204 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdbe011f740 msgr2=0x7fdbe002c650 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:12:06.992 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.966+0000 7fdbdcff9640 1 --2- 172.21.15.42:0/3736097204 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdbe011f740 0x7fdbe002c650 secure :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fdbd0007fd0 tx=0x7fdbd000c810 comp rx=0 tx=0).stop 2024-09-16T04:12:06.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.974+0000 7fdbe6706640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:12:06.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.974+0000 7fdbe6706640 1 --2- 172.21.15.42:0/3736097204 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdbe002cb90 0x7fdbe002cf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:12:06.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.974+0000 7fdbe6706640 10 monclient(hunting): start opening mon connection 2024-09-16T04:12:06.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.974+0000 7fdbe6706640 1 -- 172.21.15.42:0/3736097204 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fdbe0135550 con 0x7fdbe002cb90 2024-09-16T04:12:06.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.974+0000 7fdbdf7fe640 1 --2- 172.21.15.42:0/3736097204 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdbe002cb90 0x7fdbe002cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:12:06.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.974+0000 7fdbdf7fe640 10 monclient: get_auth_request con 0x7fdbe002cb90 auth_method 0 2024-09-16T04:12:06.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.974+0000 7fdbdf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:12:06.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.974+0000 7fdbdf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:06.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.974+0000 7fdbdf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:06.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.974+0000 7fdbdf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:12:06.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.974+0000 7fdbdf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:12:06.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.974+0000 7fdbdf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:12:06.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.974+0000 7fdbdf7fe640 10 monclient(hunting): handle_auth_done global_id 1074163 payload 824 2024-09-16T04:12:06.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.974+0000 7fdbdf7fe640 1 -- 172.21.15.42:0/3736097204 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fdbe011e460 con 0x7fdbe002cb90 2024-09-16T04:12:06.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.974+0000 7fdbdf7fe640 1 --2- 172.21.15.42:0/3736097204 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdbe002cb90 0x7fdbe002cf70 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fdbd0014000 tx=0x7fdbd0000f40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:12:06.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.974+0000 7fdbdcff9640 10 client.1034490.objecter ms_handle_connect 0x7fdbe002cb90 2024-09-16T04:12:06.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.974+0000 7fdbdcff9640 10 client.1034490.objecter resend_mon_ops 2024-09-16T04:12:06.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.974+0000 7fdbdcff9640 1 -- 172.21.15.42:0/3736097204 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbd0017040 con 0x7fdbe002cb90 2024-09-16T04:12:06.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.974+0000 7fdbdcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:06.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.974+0000 7fdbdcff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:12:06.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.974+0000 7fdbdcff9640 10 monclient: dump: 2024-09-16T04:12:06.995 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:12:06.995 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:12:06.995 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:06.995 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:06.995 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:06.995 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:12:06.995 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:12:06.995 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:12:06.995 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:12:06.996 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:12:06.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.974+0000 7fdbdcff9640 1 -- 172.21.15.42:0/3736097204 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4534 (secure 0 0 0) 0x7fdbd0004050 con 0x7fdbe002cb90 2024-09-16T04:12:06.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.974+0000 7fdbdcff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:12:06.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.974+0000 7fdbdcff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:12:06.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.974+0000 7fdbdcff9640 1 -- 172.21.15.42:0/3736097204 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdbe002cb90 msgr2=0x7fdbe002cf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:06.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.974+0000 7fdbdcff9640 1 --2- 172.21.15.42:0/3736097204 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdbe002cb90 0x7fdbe002cf70 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fdbd0014000 tx=0x7fdbd0000f40 comp rx=0 tx=0).stop 2024-09-16T04:12:06.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.974+0000 7fdbdcff9640 1 -- 172.21.15.42:0/3736097204 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdbe002cb90 msgr2=0x7fdbe002cf70 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:12:06.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.974+0000 7fdbdcff9640 1 --2- 172.21.15.42:0/3736097204 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdbe002cb90 0x7fdbe002cf70 secure :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fdbd0014000 tx=0x7fdbd0000f40 comp rx=0 tx=0).stop 2024-09-16T04:12:06.996 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T04:12:06.997 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:12:06.997 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:12:06.997 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T04:12:06.997 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 272, 2024-09-16T04:12:06.997 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 40146, 2024-09-16T04:12:06.997 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T04:12:06.997 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T04:12:06.997 INFO:teuthology.orchestra.run.smithi042.stdout: 1 2024-09-16T04:12:06.997 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:12:06.997 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 1, 2024-09-16T04:12:06.997 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:12:06.997 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T04:12:06.997 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T04:12:06.998 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:12:06.998 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:12:06.998 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:12:06.998 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:12:06.998 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:12:06.998 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:12:06.998 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:12:06.998 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:12:06.998 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:12:06.998 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:12:06.998 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:12:06.998 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:12:06.998 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:12:06.999 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T04:12:06.999 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:12:06.999 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:12:06.999 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:12:06.999 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:12:06.999 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:12:06.999 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:12:06.999 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:12:06.999 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:12:06.999 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:12:06.999 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:12:06.999 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:12:06.999 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:12:07.000 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:12:07.000 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T04:12:07.000 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T04:12:07.000 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T04:12:07.000 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T04:12:07.000 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T04:12:07.000 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T04:12:07.000 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:12:07.000 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:12:07.000 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T04:12:07.000 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T04:12:07.000 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T04:12:07.000 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T04:12:07.000 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T04:12:07.001 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T04:12:07.001 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T04:12:07.001 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:12:07.001 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T04:12:07.001 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:12:07.001 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:12:07.001 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:12:07.001 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:12:07.002 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:12:07.002 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:12:07.002 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:12:07.002 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:12:07.002 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:12:07.002 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:12:07.002 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:12:07.002 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:12:07.002 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T04:12:07.002 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:12:07.002 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T04:12:07.002 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:12:07.002 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:12:07.002 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:12:07.003 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:12:07.003 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:12:07.003 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:12:07.003 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:12:07.003 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T04:12:07.003 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:12:07.003 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:12:07.003 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:12:07.003 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:12:07.003 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T04:12:07.003 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:12:07.003 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:12:07.003 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:12:07.004 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:12:07.004 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T04:12:07.004 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T04:12:07.004 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:12:07.004 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:12:07.004 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:12:07.004 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:12:07.004 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:12:07.004 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:12:07.004 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:12:07.004 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:12:07.004 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:12:07.004 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:12:07.005 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:12:07.005 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T04:12:07.005 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:12:07.005 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:12:07.005 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:12:07.005 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:12:07.005 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T04:12:07.005 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:12:07.005 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:12:07.005 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:12:07.005 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:12:07.005 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T04:12:07.005 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T04:12:07.005 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:12:07.006 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:12:07.006 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:12:07.006 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:12:07.006 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:12:07.006 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:12:07.006 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:12:07.006 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:12:07.006 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:12:07.006 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:12:07.006 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:12:07.006 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T04:12:07.006 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:12:07.006 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:12:07.007 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:12:07.007 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:12:07.007 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T04:12:07.007 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:12:07.007 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:12:07.007 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:12:07.007 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:12:07.007 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T04:12:07.007 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T04:12:07.007 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:12:07.007 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:12:07.007 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:12:07.007 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:12:07.008 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:12:07.008 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:12:07.008 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T04:12:07.008 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T04:12:07.008 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:12:07.008 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:12:07.008 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:12:07.008 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:12:07.008 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:12:07.008 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:12:07.008 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T04:12:07.008 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:12:07.008 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:12:07.009 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:12:07.009 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 2 2024-09-16T04:12:07.009 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:12:07.009 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:12:07.009 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:12:07.009 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T04:12:07.009 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T04:12:07.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.974+0000 7fdbe6706640 10 librados: watch_flush enter 2024-09-16T04:12:07.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.974+0000 7fdbe6706640 10 librados: watch_flush exit 2024-09-16T04:12:07.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.974+0000 7fdbe6706640 20 client.1034490.objecter shutdown clearing up homeless session... 2024-09-16T04:12:07.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.974+0000 7fdbe6706640 10 client.1034490.objecter successfully canceled tick 2024-09-16T04:12:07.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.974+0000 7fdbe6706640 1 -- 172.21.15.42:0/3736097204 >> v2:172.21.15.154:6812/254845342 conn(0x7fdbcc0377f0 msgr2=0x7fdbcc039cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:07.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.974+0000 7fdbe6706640 1 --2- 172.21.15.42:0/3736097204 >> v2:172.21.15.154:6812/254845342 conn(0x7fdbcc0377f0 0x7fdbcc039cb0 secure :-1 s=READY pgs=3958 cs=0 l=1 rev1=1 crypto rx=0x7fdbd8007fd0 tx=0x7fdbd8009040 comp rx=0 tx=0).stop 2024-09-16T04:12:07.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.974+0000 7fdbe6706640 10 monclient: shutdown 2024-09-16T04:12:07.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.974+0000 7fdbe6706640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:12:07.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.974+0000 7fdbe6706640 1 -- 172.21.15.42:0/3736097204 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdbe00a1330 msgr2=0x7fdbe01cf1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:07.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.974+0000 7fdbe6706640 1 --2- 172.21.15.42:0/3736097204 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdbe00a1330 0x7fdbe01cf1c0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fdbd40ac040 tx=0x7fdbd401acd0 comp rx=0 tx=0).stop 2024-09-16T04:12:07.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.974+0000 7fdbe4ca3640 1 -- 172.21.15.42:0/3736097204 reap_dead start 2024-09-16T04:12:07.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.974+0000 7fdbe4ca3640 1 -- 172.21.15.42:0/3736097204 reap_dead start 2024-09-16T04:12:07.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.974+0000 7fdbe6706640 1 -- 172.21.15.42:0/3736097204 shutdown_connections 2024-09-16T04:12:07.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.974+0000 7fdbe6706640 1 -- 172.21.15.42:0/3736097204 >> 172.21.15.42:0/3736097204 conn(0x7fdbe00a2420 msgr2=0x7fdbe0134300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:12:07.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.974+0000 7fdbe6706640 1 -- 172.21.15.42:0/3736097204 shutdown_connections 2024-09-16T04:12:07.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.974+0000 7fdbe6706640 1 -- 172.21.15.42:0/3736097204 wait complete. 2024-09-16T04:12:07.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:06.974+0000 7fdbe6706640 1 librados: shutdown 2024-09-16T04:12:07.012 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T04:12:07.012 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-16T04:12:27.013 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-16T04:12:27.013 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-09-16T04:12:27.013 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-16T04:12:27.014 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-16T04:12:27.016 INFO:tasks.ceph.mon.c:Started 2024-09-16T04:12:27.016 INFO:tasks.mon_thrash:Sending STOP to mon a 2024-09-16T04:12:27.017 INFO:tasks.ceph.mon.a:Sent signal 19 2024-09-16T04:12:27.017 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-16T04:12:42.017 INFO:tasks.mon_thrash:Sending CONT to mon a 2024-09-16T04:12:42.018 INFO:tasks.ceph.mon.a:Sent signal 18 2024-09-16T04:12:42.018 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T04:12:42.018 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:12:42.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.150+0000 7ff5a7d64640 10 monclient: get_monmap_and_config 2024-09-16T04:12:42.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.150+0000 7ff5a7d64640 10 monclient: build_initial_monmap 2024-09-16T04:12:42.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.150+0000 7ff5a7d64640 10 monclient: monmap: 2024-09-16T04:12:42.155 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:12:42.156 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:12:42.156 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:12:42.154140+0000 2024-09-16T04:12:42.156 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:12:42.154140+0000 2024-09-16T04:12:42.156 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:12:42.156 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:12:42.156 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:12:42.156 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:12:42.156 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:12:42.156 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:12:42.156 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:12:42.156 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:12:42.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.150+0000 7ff5a7d64640 1 Processor -- start 2024-09-16T04:12:42.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.150+0000 7ff5a7d64640 1 -- start start 2024-09-16T04:12:42.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.150+0000 7ff5a7d64640 10 monclient: init 2024-09-16T04:12:42.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.150+0000 7ff5a7d64640 10 monclient: _reopen_session rank -1 2024-09-16T04:12:42.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.150+0000 7ff5a7d64640 10 monclient: _add_conns ranks=[3,0,1,2,4] 2024-09-16T04:12:42.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.150+0000 7ff5a7d64640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff5a009fb00 0x7ff5a009ba50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:12:42.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.150+0000 7ff5a7d64640 10 monclient(hunting): picked mon.noname-a con 0x7ff5a009fb00 addr v2:172.21.15.42:3300/0 2024-09-16T04:12:42.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.150+0000 7ff5a7d64640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff5a009bf90 addr v1:172.21.15.42:6789/0 2024-09-16T04:12:42.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.150+0000 7ff5a7d64640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff5a009c400 addr v1:172.21.15.154:6789/0 2024-09-16T04:12:42.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.150+0000 7ff5a7d64640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff5a00a1e20 con 0x7ff5a009bf90 2024-09-16T04:12:42.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.150+0000 7ff5a7d64640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff5a009c850 con 0x7ff5a009c400 2024-09-16T04:12:42.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.150+0000 7ff5a7d64640 10 monclient(hunting): start opening mon connection 2024-09-16T04:12:42.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.150+0000 7ff5a7d64640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7ff5a009e250 con 0x7ff5a009fb00 2024-09-16T04:12:42.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.150+0000 7ff5a7d64640 10 monclient(hunting): _renew_subs 2024-09-16T04:12:42.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.150+0000 7ff5a7d64640 10 monclient(hunting): authenticate will time out at 2809.123777s 2024-09-16T04:12:42.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.150+0000 7ff5a5b00640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff5a009fb00 0x7ff5a009ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:12:42.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.150+0000 7ff5a52ff640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7ff5a009bf90 0x7ff5a0138110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:39004/0 (socket says 172.21.15.42:39004) 2024-09-16T04:12:42.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.150+0000 7ff5a52ff640 1 -- 172.21.15.42:0/3015523748 learned_addr learned my addr 172.21.15.42:0/3015523748 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T04:12:42.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.150+0000 7ff5a5b00640 10 monclient(hunting): get_auth_request con 0x7ff5a009fb00 auth_method 0 2024-09-16T04:12:42.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.150+0000 7ff5a5b00640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:12:42.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.150+0000 7ff5a5b00640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:42.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.150+0000 7ff5a5b00640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:42.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.154+0000 7ff5a5b00640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:12:42.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.154+0000 7ff5a5b00640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:12:42.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.154+0000 7ff5a5b00640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:12:42.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.154+0000 7ff5a4afe640 1 -- 172.21.15.42:0/3015523748 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1874890066 0 0) 0x7ff5a00a1e20 con 0x7ff5a009bf90 2024-09-16T04:12:42.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.154+0000 7ff5a4afe640 10 monclient(hunting): _negotiate 2024-09-16T04:12:42.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.154+0000 7ff5a4afe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:42.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.154+0000 7ff5a4afe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:42.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.154+0000 7ff5a4afe640 10 monclient(hunting): my global_id is 1034496 2024-09-16T04:12:42.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.154+0000 7ff5a4afe640 1 -- 172.21.15.42:0/3015523748 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff58c003530 con 0x7ff5a009bf90 2024-09-16T04:12:42.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.154+0000 7ff5a5b00640 10 monclient(hunting): handle_auth_done global_id 1034493 payload 274 2024-09-16T04:12:42.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.154+0000 7ff5a5b00640 1 -- 172.21.15.42:0/3015523748 >> v1:172.21.15.154:6789/0 conn(0x7ff5a009c400 legacy=0x7ff5a013b840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:12:42.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.154+0000 7ff5a5b00640 1 -- 172.21.15.42:0/3015523748 >> v1:172.21.15.42:6789/0 conn(0x7ff5a009bf90 legacy=0x7ff5a0138110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:42.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.154+0000 7ff5a5b00640 10 monclient: _finish_hunting 0 2024-09-16T04:12:42.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.154+0000 7ff5a5b00640 1 monclient: found mon.noname-a 2024-09-16T04:12:42.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.154+0000 7ff5a5b00640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:12:42.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.154+0000 7ff5a5b00640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:12:42.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.154+0000 7ff5a5b00640 1 -- 172.21.15.42:0/3015523748 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5a009ea30 con 0x7ff5a009fb00 2024-09-16T04:12:42.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.154+0000 7ff5a5b00640 1 --2- 172.21.15.42:0/3015523748 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff5a009fb00 0x7ff5a009ba50 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7ff59c00a960 tx=0x7ff59c01d6d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f03d56a2d51f439 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:12:42.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.154+0000 7ff5a4afe640 1 -- 172.21.15.42:0/3015523748 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff59c00b040 con 0x7ff5a009fb00 2024-09-16T04:12:42.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.154+0000 7ff5a4afe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:42.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.154+0000 7ff5a4afe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:12:42.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.154+0000 7ff5a4afe640 10 monclient: dump: 2024-09-16T04:12:42.167 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:12:42.168 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:12:42.168 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:42.168 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:42.168 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:42.168 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:12:42.168 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:12:42.168 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:12:42.168 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:12:42.168 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:12:42.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.154+0000 7ff5a4afe640 10 monclient: _finish_auth 0 2024-09-16T04:12:42.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.154+0000 7ff5a4afe640 10 monclient: _check_auth_tickets 2024-09-16T04:12:42.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.154+0000 7ff5a4afe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:12:42.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.154+0000 7ff5a7d64640 5 monclient: authenticate success, global_id 1034493 2024-09-16T04:12:42.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.154+0000 7ff5a7d64640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:12:42.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.154+0000 7ff5a4afe640 1 -- 172.21.15.42:0/3015523748 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff59c0a7710 con 0x7ff5a009fb00 2024-09-16T04:12:42.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.154+0000 7ff5a4afe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:12:42.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.154+0000 7ff5a7d64640 10 monclient: get_monmap_and_config success 2024-09-16T04:12:42.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.154+0000 7ff5a7d64640 4 set_mon_vals no callback set 2024-09-16T04:12:42.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.154+0000 7ff5a4afe640 1 -- 172.21.15.42:0/3015523748 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff59c0a7ae0 con 0x7ff5a009fb00 2024-09-16T04:12:42.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.154+0000 7ff5a4afe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:42.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.154+0000 7ff5a4afe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:12:42.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.154+0000 7ff5a4afe640 10 monclient: dump: 2024-09-16T04:12:42.169 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:12:42.169 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:12:42.169 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:42.169 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:42.170 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:42.170 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:12:42.170 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:12:42.170 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:12:42.170 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:12:42.170 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:12:42.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.154+0000 7ff5a7d64640 10 monclient: shutdown 2024-09-16T04:12:42.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.154+0000 7ff5a7d64640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:12:42.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.154+0000 7ff5a7d64640 1 -- 172.21.15.42:0/3015523748 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff5a009fb00 msgr2=0x7ff5a009ba50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:42.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.154+0000 7ff5a7d64640 1 --2- 172.21.15.42:0/3015523748 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff5a009fb00 0x7ff5a009ba50 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7ff59c00a960 tx=0x7ff59c01d6d0 comp rx=0 tx=0).stop 2024-09-16T04:12:42.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.154+0000 7ff5a7d64640 1 -- 172.21.15.42:0/3015523748 shutdown_connections 2024-09-16T04:12:42.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.154+0000 7ff5a7d64640 1 --2- 172.21.15.42:0/3015523748 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff5a009fb00 0x7ff5a009ba50 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:12:42.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.154+0000 7ff5a7d64640 1 -- 172.21.15.42:0/3015523748 >> 172.21.15.42:0/3015523748 conn(0x7ff5a00a2420 msgr2=0x7ff5a00a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:12:42.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.154+0000 7ff5a7d64640 1 -- 172.21.15.42:0/3015523748 shutdown_connections 2024-09-16T04:12:42.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.154+0000 7ff5a7d64640 1 -- 172.21.15.42:0/3015523748 wait complete. 2024-09-16T04:12:42.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.154+0000 7ff5a7d64640 10 monclient: build_initial_monmap 2024-09-16T04:12:42.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.154+0000 7ff5a7d64640 10 monclient: monmap: 2024-09-16T04:12:42.171 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:12:42.171 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:12:42.171 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:12:42.171 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:12:42.171 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:12:42.171 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:12:42.171 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:12:42.171 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:12:42.171 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:12:42.172 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:12:42.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.154+0000 7ff5a7d64640 1 librados: starting msgr at 2024-09-16T04:12:42.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.154+0000 7ff5a7d64640 1 librados: starting objecter 2024-09-16T04:12:42.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.158+0000 7ff5a7d64640 1 Processor -- start 2024-09-16T04:12:42.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.158+0000 7ff5a7d64640 1 -- start start 2024-09-16T04:12:42.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.158+0000 7ff5a7d64640 1 librados: setting wanted keys 2024-09-16T04:12:42.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.158+0000 7ff5a7d64640 1 librados: calling monclient init 2024-09-16T04:12:42.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.158+0000 7ff5a7d64640 10 monclient: init 2024-09-16T04:12:42.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.158+0000 7ff5a7d64640 10 monclient: _reopen_session rank -1 2024-09-16T04:12:42.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.158+0000 7ff5a7d64640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:12:42.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.158+0000 7ff5a7d64640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff5a009bf90 0x7ff5a01ceda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:12:42.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.158+0000 7ff5a7d64640 10 monclient(hunting): picked mon.noname-a con 0x7ff5a009bf90 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:12:42.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.158+0000 7ff5a7d64640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff5a009c400 0x7ff5a01353a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:12:42.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.158+0000 7ff5a7d64640 10 monclient(hunting): picked mon.noname-c con 0x7ff5a009c400 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:12:42.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.158+0000 7ff5a7d64640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff5a009fb00 0x7ff5a01358e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:12:42.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.158+0000 7ff5a7d64640 10 monclient(hunting): picked mon.noname-b con 0x7ff5a009fb00 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:12:42.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.158+0000 7ff5a7d64640 10 monclient(hunting): start opening mon connection 2024-09-16T04:12:42.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.158+0000 7ff5a7d64640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7ff5a00a3170 con 0x7ff5a009bf90 2024-09-16T04:12:42.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.158+0000 7ff5a7d64640 10 monclient(hunting): start opening mon connection 2024-09-16T04:12:42.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.158+0000 7ff5a7d64640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff5a009d0b0 con 0x7ff5a009fb00 2024-09-16T04:12:42.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.158+0000 7ff5a7d64640 10 monclient(hunting): start opening mon connection 2024-09-16T04:12:42.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.158+0000 7ff5a7d64640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7ff5a00a32f0 con 0x7ff5a009c400 2024-09-16T04:12:42.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.158+0000 7ff5a7d64640 10 monclient(hunting): _renew_subs 2024-09-16T04:12:42.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.158+0000 7ff5a7d64640 10 monclient(hunting): authenticate will time out at 2809.128650s 2024-09-16T04:12:42.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.158+0000 7ff5a5b00640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff5a009bf90 0x7ff5a01ceda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:12:42.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.158+0000 7ff5a5b00640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff5a009bf90 0x7ff5a01ceda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:39360/0 (socket says 172.21.15.42:39360) 2024-09-16T04:12:42.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.158+0000 7ff5a5b00640 1 -- 172.21.15.42:0/1847339213 learned_addr learned my addr 172.21.15.42:0/1847339213 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:12:42.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.158+0000 7ff5a5b00640 10 monclient(hunting): get_auth_request con 0x7ff5a009bf90 auth_method 0 2024-09-16T04:12:42.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.158+0000 7ff5a5b00640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:12:42.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.158+0000 7ff5a5b00640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:42.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.158+0000 7ff5a5b00640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:42.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.158+0000 7ff5a52ff640 1 --2- 172.21.15.42:0/1847339213 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff5a009c400 0x7ff5a01353a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:12:42.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.158+0000 7ff5a52ff640 10 monclient(hunting): get_auth_request con 0x7ff5a009c400 auth_method 0 2024-09-16T04:12:42.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.158+0000 7ff5a52ff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:12:42.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.158+0000 7ff5a52ff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:42.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.158+0000 7ff5a52ff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:42.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.158+0000 7ff5a5b00640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:12:42.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.158+0000 7ff5a5b00640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:12:42.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.158+0000 7ff5a5b00640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:12:42.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.158+0000 7ff5a52ff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:12:42.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.158+0000 7ff5a52ff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:12:42.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.158+0000 7ff5a52ff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:12:42.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.158+0000 7ff5a5b00640 10 monclient(hunting): handle_auth_done global_id 1034499 payload 824 2024-09-16T04:12:42.178 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.158+0000 7ff5a5b00640 1 -- 172.21.15.42:0/1847339213 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff5a009c400 msgr2=0x7ff5a01353a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:42.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.158+0000 7ff5a5b00640 1 --2- 172.21.15.42:0/1847339213 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff5a009c400 0x7ff5a01353a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:12:42.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.158+0000 7ff5a5b00640 1 -- 172.21.15.42:0/1847339213 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff5a009fb00 msgr2=0x7ff5a01358e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:42.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.158+0000 7ff5a6301640 1 --2- 172.21.15.42:0/1847339213 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff5a009fb00 0x7ff5a01358e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:12:42.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.158+0000 7ff5a5b00640 1 --2- 172.21.15.42:0/1847339213 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff5a009fb00 0x7ff5a01358e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:12:42.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.158+0000 7ff5a5b00640 10 monclient: _finish_hunting 0 2024-09-16T04:12:42.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.158+0000 7ff5a5b00640 1 monclient: found mon.noname-a 2024-09-16T04:12:42.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.158+0000 7ff5a5b00640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:12:42.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.158+0000 7ff5a5b00640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:12:42.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.158+0000 7ff5a5b00640 1 -- 172.21.15.42:0/1847339213 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5a0122d60 con 0x7ff5a009bf90 2024-09-16T04:12:42.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.158+0000 7ff5a6301640 10 monclient: get_auth_request con 0x7ff5a009fb00 auth_method 0 2024-09-16T04:12:42.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.158+0000 7ff5a6301640 1 --2- 172.21.15.42:0/1847339213 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff5a009fb00 0x7ff5a01358e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T04:12:42.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.158+0000 7ff5a5b00640 1 --2- 172.21.15.42:0/1847339213 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff5a009bf90 0x7ff5a01ceda0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7ff59c0b1040 tx=0x7ff59c01a470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:12:42.179 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.158+0000 7ff5967fc640 10 client.?.objecter ms_handle_connect 0x7ff5a009bf90 2024-09-16T04:12:42.180 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.158+0000 7ff5967fc640 10 client.?.objecter resend_mon_ops 2024-09-16T04:12:42.180 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.158+0000 7ff5967fc640 1 -- 172.21.15.42:0/1847339213 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff59c0bd020 con 0x7ff5a009bf90 2024-09-16T04:12:42.180 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.158+0000 7ff5967fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:42.180 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.158+0000 7ff5967fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:12:42.180 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.158+0000 7ff5967fc640 10 monclient: dump: 2024-09-16T04:12:42.180 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:12:42.180 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:12:42.180 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:42.180 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:42.180 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:42.180 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:12:42.180 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:12:42.181 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:12:42.181 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:12:42.181 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:12:42.181 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.158+0000 7ff5967fc640 10 monclient: _finish_auth 0 2024-09-16T04:12:42.181 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.158+0000 7ff5967fc640 10 monclient: _check_auth_tickets 2024-09-16T04:12:42.181 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.158+0000 7ff5967fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:12:42.181 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.158+0000 7ff5967fc640 1 -- 172.21.15.42:0/1847339213 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff59c0b8650 con 0x7ff5a009bf90 2024-09-16T04:12:42.181 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.158+0000 7ff5a7d64640 5 monclient: authenticate success, global_id 1034499 2024-09-16T04:12:42.181 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.158+0000 7ff5a7d64640 10 monclient: _renew_subs 2024-09-16T04:12:42.181 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.158+0000 7ff5a7d64640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:12:42.181 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.158+0000 7ff5a7d64640 1 -- 172.21.15.42:0/1847339213 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5a006f530 con 0x7ff5a009bf90 2024-09-16T04:12:42.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.158+0000 7ff5967fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:12:42.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.158+0000 7ff5967fc640 1 -- 172.21.15.42:0/1847339213 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff59c0b8a00 con 0x7ff5a009bf90 2024-09-16T04:12:42.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.158+0000 7ff5967fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:42.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.158+0000 7ff5967fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:12:42.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.158+0000 7ff5967fc640 10 monclient: dump: 2024-09-16T04:12:42.182 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:12:42.182 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:12:42.182 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:42.182 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:42.182 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:42.182 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:12:42.182 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:12:42.182 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:12:42.182 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:12:42.183 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:12:42.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.158+0000 7ff5a7d64640 10 client.1034499.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:12:42.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.158+0000 7ff597fff640 4 set_mon_vals no callback set 2024-09-16T04:12:42.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.158+0000 7ff5a7d64640 10 monclient: _renew_subs 2024-09-16T04:12:42.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.158+0000 7ff5a7d64640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:12:42.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.158+0000 7ff5a7d64640 1 -- 172.21.15.42:0/1847339213 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5a0073a30 con 0x7ff5a009bf90 2024-09-16T04:12:42.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.158+0000 7ff5a7d64640 1 librados: init done 2024-09-16T04:12:42.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.158+0000 7ff5967fc640 1 -- 172.21.15.42:0/1847339213 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7ff59c0c0020 con 0x7ff5a009bf90 2024-09-16T04:12:42.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.158+0000 7ff5a7d64640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:12:42.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.158+0000 7ff5a7d64640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:12:42.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.158+0000 7ff5a7d64640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:12:42.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.158+0000 7ff5a7d64640 1 -- 172.21.15.42:0/1847339213 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff5a009ba50 con 0x7ff5a009bf90 2024-09-16T04:12:42.183 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.158+0000 7ff5967fc640 1 --2- 172.21.15.42:0/1847339213 >> v2:172.21.15.154:6812/254845342 conn(0x7ff584037810 0x7ff584039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:12:42.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.158+0000 7ff5967fc640 1 -- 172.21.15.42:0/1847339213 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(706..706 src has 681..706) ==== 5040+0+0 (secure 0 0 0) 0x7ff59c0e3a00 con 0x7ff5a009bf90 2024-09-16T04:12:42.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.158+0000 7ff5967fc640 10 client.1034499.objecter ms_dispatch 0x7ff5a0000ba0 osd_map(706..706 src has 681..706) 2024-09-16T04:12:42.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.158+0000 7ff5967fc640 3 client.1034499.objecter handle_osd_map got epochs [706,706] > 0 2024-09-16T04:12:42.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.158+0000 7ff5967fc640 3 client.1034499.objecter handle_osd_map decoding full epoch 706 2024-09-16T04:12:42.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.158+0000 7ff5967fc640 20 client.1034499.objecter dump_active .. 0 homeless 2024-09-16T04:12:42.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.158+0000 7ff5a52ff640 1 --2- 172.21.15.42:0/1847339213 >> v2:172.21.15.154:6812/254845342 conn(0x7ff584037810 0x7ff584039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:12:42.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.158+0000 7ff5a52ff640 10 monclient: get_auth_request con 0x7ff584037810 auth_method 0 2024-09-16T04:12:42.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.162+0000 7ff5a52ff640 1 --2- 172.21.15.42:0/1847339213 >> v2:172.21.15.154:6812/254845342 conn(0x7ff584037810 0x7ff584039cd0 secure :-1 s=READY pgs=3966 cs=0 l=1 rev1=1 crypto rx=0x7ff590007fd0 tx=0x7ff590009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:12:42.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.162+0000 7ff5967fc640 10 client.1034499.objecter ms_handle_connect 0x7ff584037810 2024-09-16T04:12:42.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.162+0000 7ff5967fc640 1 -- 172.21.15.42:0/1847339213 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7ff59c0c7030 con 0x7ff5a009bf90 2024-09-16T04:12:42.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.162+0000 7ff5967fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:12:42.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.162+0000 7ff5967fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:12:42.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.394+0000 7ff5a7d64640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:12:42.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.394+0000 7ff5a7d64640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:12:42.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.394+0000 7ff5a7d64640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:12:42.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.394+0000 7ff5a7d64640 1 -- 172.21.15.42:0/1847339213 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7ff5a0069040 con 0x7ff5a009bf90 2024-09-16T04:12:42.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.394+0000 7ff5967fc640 1 -- 172.21.15.42:0/1847339213 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7ff59c0173e0 con 0x7ff5a009bf90 2024-09-16T04:12:42.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.394+0000 7ff5967fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:12:42.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.394+0000 7ff5967fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:12:42.400 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":276,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:12:42.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.398+0000 7ff5a7d64640 10 librados: watch_flush enter 2024-09-16T04:12:42.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.398+0000 7ff5a7d64640 10 librados: watch_flush exit 2024-09-16T04:12:42.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.398+0000 7ff5a7d64640 20 client.1034499.objecter shutdown clearing up homeless session... 2024-09-16T04:12:42.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.398+0000 7ff5a7d64640 10 client.1034499.objecter successfully canceled tick 2024-09-16T04:12:42.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.398+0000 7ff5a7d64640 1 -- 172.21.15.42:0/1847339213 >> v2:172.21.15.154:6812/254845342 conn(0x7ff584037810 msgr2=0x7ff584039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:42.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.398+0000 7ff5a7d64640 1 --2- 172.21.15.42:0/1847339213 >> v2:172.21.15.154:6812/254845342 conn(0x7ff584037810 0x7ff584039cd0 secure :-1 s=READY pgs=3966 cs=0 l=1 rev1=1 crypto rx=0x7ff590007fd0 tx=0x7ff590009040 comp rx=0 tx=0).stop 2024-09-16T04:12:42.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.398+0000 7ff5a7d64640 10 monclient: shutdown 2024-09-16T04:12:42.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.398+0000 7ff5a7d64640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:12:42.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.398+0000 7ff5a7d64640 1 -- 172.21.15.42:0/1847339213 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff5a009bf90 msgr2=0x7ff5a01ceda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:42.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.398+0000 7ff5a7d64640 1 --2- 172.21.15.42:0/1847339213 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff5a009bf90 0x7ff5a01ceda0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7ff59c0b1040 tx=0x7ff59c01a470 comp rx=0 tx=0).stop 2024-09-16T04:12:42.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.398+0000 7ff5a7d64640 1 -- 172.21.15.42:0/1847339213 shutdown_connections 2024-09-16T04:12:42.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.398+0000 7ff5a7d64640 1 --2- 172.21.15.42:0/1847339213 >> v2:172.21.15.154:6812/254845342 conn(0x7ff584037810 0x7ff584039cd0 unknown :-1 s=CLOSED pgs=3966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:12:42.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.398+0000 7ff5a7d64640 1 --2- 172.21.15.42:0/1847339213 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff5a009fb00 0x7ff5a01358e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:12:42.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.398+0000 7ff5a7d64640 1 --2- 172.21.15.42:0/1847339213 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff5a009c400 0x7ff5a01353a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:12:42.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.398+0000 7ff5a7d64640 1 --2- 172.21.15.42:0/1847339213 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff5a009bf90 0x7ff5a01ceda0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:12:42.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.398+0000 7ff5a7d64640 1 -- 172.21.15.42:0/1847339213 >> 172.21.15.42:0/1847339213 conn(0x7ff5a00a2420 msgr2=0x7ff5a0070540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:12:42.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.398+0000 7ff5a7d64640 1 -- 172.21.15.42:0/1847339213 shutdown_connections 2024-09-16T04:12:42.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.398+0000 7ff5a7d64640 1 -- 172.21.15.42:0/1847339213 wait complete. 2024-09-16T04:12:42.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.398+0000 7ff5a7d64640 1 librados: shutdown 2024-09-16T04:12:42.421 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T04:12:42.421 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T04:12:42.421 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T04:12:42.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.542+0000 7f4c663a0640 10 monclient: get_monmap_and_config 2024-09-16T04:12:42.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.542+0000 7f4c663a0640 10 monclient: build_initial_monmap 2024-09-16T04:12:42.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.542+0000 7f4c663a0640 10 monclient: monmap: 2024-09-16T04:12:42.549 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:12:42.549 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:12:42.549 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:12:42.548070+0000 2024-09-16T04:12:42.550 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:12:42.548070+0000 2024-09-16T04:12:42.550 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:12:42.550 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:12:42.550 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:12:42.550 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:12:42.550 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:12:42.550 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:12:42.550 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:12:42.550 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:12:42.552 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.546+0000 7f4c663a0640 1 Processor -- start 2024-09-16T04:12:42.552 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.546+0000 7f4c663a0640 1 -- start start 2024-09-16T04:12:42.552 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.546+0000 7f4c663a0640 10 monclient: init 2024-09-16T04:12:42.552 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.546+0000 7f4c663a0640 10 monclient: _reopen_session rank -1 2024-09-16T04:12:42.552 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.546+0000 7f4c663a0640 10 monclient: _add_conns ranks=[3,0,1,4,2] 2024-09-16T04:12:42.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.546+0000 7f4c663a0640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f4c600a1330 0x7f4c600a1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:12:42.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.546+0000 7f4c663a0640 10 monclient(hunting): picked mon.noname-a con 0x7f4c600a1330 addr v2:172.21.15.42:3300/0 2024-09-16T04:12:42.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.546+0000 7f4c663a0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4c6006dac0 addr v1:172.21.15.42:6789/0 2024-09-16T04:12:42.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.546+0000 7f4c663a0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4c600a1c50 addr v1:172.21.15.154:6789/0 2024-09-16T04:12:42.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.546+0000 7f4c663a0640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4c60135f20 con 0x7f4c6006dac0 2024-09-16T04:12:42.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.546+0000 7f4c663a0640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4c6009c090 con 0x7f4c600a1c50 2024-09-16T04:12:42.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.546+0000 7f4c663a0640 10 monclient(hunting): start opening mon connection 2024-09-16T04:12:42.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.546+0000 7f4c663a0640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f4c6009f610 con 0x7f4c600a1330 2024-09-16T04:12:42.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.546+0000 7f4c663a0640 10 monclient(hunting): _renew_subs 2024-09-16T04:12:42.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.546+0000 7f4c663a0640 10 monclient(hunting): authenticate will time out at 2809.517495s 2024-09-16T04:12:42.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.546+0000 7f4c5f7fe640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f4c6006dac0 0x7f4c60135100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:39016/0 (socket says 172.21.15.42:39016) 2024-09-16T04:12:42.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.546+0000 7f4c5f7fe640 1 -- 172.21.15.42:0/1549558156 learned_addr learned my addr 172.21.15.42:0/1549558156 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T04:12:42.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.546+0000 7f4c5ffff640 1 --2- 172.21.15.42:0/1549558156 >> v2:172.21.15.42:3300/0 conn(0x7f4c600a1330 0x7f4c600a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:12:42.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.546+0000 7f4c5ffff640 10 monclient(hunting): get_auth_request con 0x7f4c600a1330 auth_method 0 2024-09-16T04:12:42.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.546+0000 7f4c5ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:12:42.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.546+0000 7f4c5ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:42.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.546+0000 7f4c5ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:42.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.546+0000 7f4c5ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:12:42.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.546+0000 7f4c5ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:12:42.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.546+0000 7f4c5ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:12:42.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.546+0000 7f4c5effd640 1 -- 172.21.15.42:0/1549558156 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2857325160 0 0) 0x7f4c60135f20 con 0x7f4c6006dac0 2024-09-16T04:12:42.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.546+0000 7f4c5effd640 10 monclient(hunting): _negotiate 2024-09-16T04:12:42.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.546+0000 7f4c5effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:42.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.546+0000 7f4c5effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:42.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.546+0000 7f4c5effd640 10 monclient(hunting): my global_id is 1034505 2024-09-16T04:12:42.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.546+0000 7f4c5effd640 1 -- 172.21.15.42:0/1549558156 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4c40003530 con 0x7f4c6006dac0 2024-09-16T04:12:42.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.546+0000 7f4c5ffff640 10 monclient(hunting): handle_auth_done global_id 1034502 payload 274 2024-09-16T04:12:42.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.546+0000 7f4c5ffff640 1 -- 172.21.15.42:0/1549558156 >> v1:172.21.15.154:6789/0 conn(0x7f4c600a1c50 legacy=0x7f4c60135810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:42.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.546+0000 7f4c5ffff640 1 -- 172.21.15.42:0/1549558156 >> v1:172.21.15.42:6789/0 conn(0x7f4c6006dac0 legacy=0x7f4c60135100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:42.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.546+0000 7f4c5ffff640 10 monclient: _finish_hunting 0 2024-09-16T04:12:42.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.546+0000 7f4c5ffff640 1 monclient: found mon.noname-a 2024-09-16T04:12:42.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.546+0000 7f4c5ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:12:42.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.546+0000 7f4c5ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:12:42.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.546+0000 7f4c5ffff640 1 -- 172.21.15.42:0/1549558156 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c6009f8f0 con 0x7f4c600a1330 2024-09-16T04:12:42.562 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.550+0000 7f4c5ffff640 1 --2- 172.21.15.42:0/1549558156 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4c600a1330 0x7f4c600a1710 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f4c580097c0 tx=0x7f4c5801c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34bafe6aee41129 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:12:42.562 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.550+0000 7f4c5effd640 1 -- 172.21.15.42:0/1549558156 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c580b1070 con 0x7f4c600a1330 2024-09-16T04:12:42.562 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.550+0000 7f4c5effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:42.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.550+0000 7f4c5effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:12:42.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.550+0000 7f4c5effd640 10 monclient: dump: 2024-09-16T04:12:42.563 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:12:42.563 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:12:42.563 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:42.563 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:42.563 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:42.563 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:12:42.563 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:12:42.563 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:12:42.563 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:12:42.563 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:12:42.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.550+0000 7f4c5effd640 10 monclient: _finish_auth 0 2024-09-16T04:12:42.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.550+0000 7f4c5effd640 10 monclient: _check_auth_tickets 2024-09-16T04:12:42.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.550+0000 7f4c5effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:12:42.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.550+0000 7f4c5effd640 1 -- 172.21.15.42:0/1549558156 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4c580a8050 con 0x7f4c600a1330 2024-09-16T04:12:42.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.550+0000 7f4c5effd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:12:42.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.550+0000 7f4c5effd640 1 -- 172.21.15.42:0/1549558156 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c580ac860 con 0x7f4c600a1330 2024-09-16T04:12:42.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.550+0000 7f4c5effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:42.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.550+0000 7f4c5effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:12:42.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.550+0000 7f4c5effd640 10 monclient: dump: 2024-09-16T04:12:42.564 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:12:42.564 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:12:42.564 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:42.564 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:42.564 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:42.565 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:12:42.565 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:12:42.565 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:12:42.565 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:12:42.565 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:12:42.565 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.550+0000 7f4c663a0640 5 monclient: authenticate success, global_id 1034502 2024-09-16T04:12:42.565 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.550+0000 7f4c663a0640 10 monclient: get_monmap_and_config success 2024-09-16T04:12:42.565 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.550+0000 7f4c663a0640 4 set_mon_vals no callback set 2024-09-16T04:12:42.565 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.550+0000 7f4c663a0640 10 monclient: shutdown 2024-09-16T04:12:42.565 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.550+0000 7f4c663a0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:12:42.565 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.550+0000 7f4c663a0640 1 -- 172.21.15.42:0/1549558156 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4c600a1330 msgr2=0x7f4c600a1710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:42.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.550+0000 7f4c663a0640 1 --2- 172.21.15.42:0/1549558156 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4c600a1330 0x7f4c600a1710 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f4c580097c0 tx=0x7f4c5801c3c0 comp rx=0 tx=0).stop 2024-09-16T04:12:42.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.550+0000 7f4c663a0640 1 -- 172.21.15.42:0/1549558156 shutdown_connections 2024-09-16T04:12:42.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.550+0000 7f4c663a0640 1 --2- 172.21.15.42:0/1549558156 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4c600a1330 0x7f4c600a1710 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:12:42.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.550+0000 7f4c663a0640 1 -- 172.21.15.42:0/1549558156 >> 172.21.15.42:0/1549558156 conn(0x7f4c600a2420 msgr2=0x7f4c600a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:12:42.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.550+0000 7f4c663a0640 1 -- 172.21.15.42:0/1549558156 shutdown_connections 2024-09-16T04:12:42.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.550+0000 7f4c663a0640 1 -- 172.21.15.42:0/1549558156 wait complete. 2024-09-16T04:12:42.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.550+0000 7f4c663a0640 10 monclient: build_initial_monmap 2024-09-16T04:12:42.566 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.550+0000 7f4c663a0640 10 monclient: monmap: 2024-09-16T04:12:42.566 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:12:42.566 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:12:42.566 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:12:42.566 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:12:42.566 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:12:42.567 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:12:42.567 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:12:42.567 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:12:42.567 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:12:42.567 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:12:42.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.550+0000 7f4c663a0640 1 librados: starting msgr at 2024-09-16T04:12:42.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.550+0000 7f4c663a0640 1 librados: starting objecter 2024-09-16T04:12:42.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.550+0000 7f4c663a0640 1 Processor -- start 2024-09-16T04:12:42.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.550+0000 7f4c663a0640 1 -- start start 2024-09-16T04:12:42.567 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.550+0000 7f4c663a0640 1 librados: setting wanted keys 2024-09-16T04:12:42.568 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.550+0000 7f4c663a0640 1 librados: calling monclient init 2024-09-16T04:12:42.568 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.550+0000 7f4c663a0640 10 monclient: init 2024-09-16T04:12:42.568 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.550+0000 7f4c663a0640 10 monclient: _reopen_session rank -1 2024-09-16T04:12:42.568 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.550+0000 7f4c663a0640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:12:42.568 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.550+0000 7f4c663a0640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4c600a1330 0x7f4c60135260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:12:42.568 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.550+0000 7f4c663a0640 10 monclient(hunting): picked mon.noname-a con 0x7f4c600a1330 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:12:42.568 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.550+0000 7f4c663a0640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4c600a1c50 0x7f4c6009cb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:12:42.568 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.550+0000 7f4c663a0640 10 monclient(hunting): picked mon.noname-b con 0x7f4c600a1c50 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:12:42.568 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.550+0000 7f4c663a0640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4c6009de60 0x7f4c6009d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:12:42.568 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.550+0000 7f4c663a0640 10 monclient(hunting): picked mon.noname-c con 0x7f4c6009de60 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:12:42.568 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.550+0000 7f4c663a0640 10 monclient(hunting): start opening mon connection 2024-09-16T04:12:42.568 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.550+0000 7f4c663a0640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f4c6009eef0 con 0x7f4c600a1330 2024-09-16T04:12:42.568 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.550+0000 7f4c663a0640 10 monclient(hunting): start opening mon connection 2024-09-16T04:12:42.569 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.550+0000 7f4c663a0640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f4c6009ed70 con 0x7f4c600a1c50 2024-09-16T04:12:42.569 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.550+0000 7f4c663a0640 10 monclient(hunting): start opening mon connection 2024-09-16T04:12:42.569 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.550+0000 7f4c663a0640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f4c60140b90 con 0x7f4c6009de60 2024-09-16T04:12:42.569 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.550+0000 7f4c663a0640 10 monclient(hunting): _renew_subs 2024-09-16T04:12:42.569 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.550+0000 7f4c663a0640 10 monclient(hunting): authenticate will time out at 2809.523081s 2024-09-16T04:12:42.569 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.550+0000 7f4c5ffff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4c600a1330 0x7f4c60135260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:12:42.569 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.550+0000 7f4c5ffff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4c600a1330 0x7f4c60135260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:39370/0 (socket says 172.21.15.42:39370) 2024-09-16T04:12:42.569 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.550+0000 7f4c5ffff640 1 -- 172.21.15.42:0/982485791 learned_addr learned my addr 172.21.15.42:0/982485791 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:12:42.569 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.550+0000 7f4c5ffff640 10 monclient(hunting): get_auth_request con 0x7f4c600a1330 auth_method 0 2024-09-16T04:12:42.569 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.550+0000 7f4c5ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:12:42.569 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.550+0000 7f4c5ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:42.570 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.550+0000 7f4c5ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:42.570 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.550+0000 7f4c6493d640 1 --2- 172.21.15.42:0/982485791 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4c6009de60 0x7f4c6009d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:12:42.570 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.550+0000 7f4c5ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:12:42.570 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.550+0000 7f4c5ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:12:42.570 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.550+0000 7f4c5ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:12:42.570 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.550+0000 7f4c6493d640 10 monclient(hunting): get_auth_request con 0x7f4c6009de60 auth_method 0 2024-09-16T04:12:42.570 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.550+0000 7f4c6493d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:12:42.570 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.550+0000 7f4c6493d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:42.570 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.550+0000 7f4c6493d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:42.570 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.550+0000 7f4c6493d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:12:42.570 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.550+0000 7f4c6493d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:12:42.570 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.550+0000 7f4c6493d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:12:42.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.550+0000 7f4c5ffff640 10 monclient(hunting): handle_auth_done global_id 1034508 payload 824 2024-09-16T04:12:42.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.550+0000 7f4c5ffff640 1 -- 172.21.15.42:0/982485791 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4c6009de60 msgr2=0x7f4c6009d0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:42.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.550+0000 7f4c5ffff640 1 --2- 172.21.15.42:0/982485791 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f4c6009de60 0x7f4c6009d0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:12:42.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.550+0000 7f4c5ffff640 1 -- 172.21.15.42:0/982485791 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4c600a1c50 msgr2=0x7f4c6009cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:42.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.550+0000 7f4c5ffff640 1 --2- 172.21.15.42:0/982485791 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4c600a1c50 0x7f4c6009cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:12:42.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.550+0000 7f4c5ffff640 10 monclient: _finish_hunting 0 2024-09-16T04:12:42.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.550+0000 7f4c5ffff640 1 monclient: found mon.noname-a 2024-09-16T04:12:42.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.550+0000 7f4c5ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:12:42.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.550+0000 7f4c5ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:12:42.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.550+0000 7f4c5ffff640 1 -- 172.21.15.42:0/982485791 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c60071300 con 0x7f4c600a1330 2024-09-16T04:12:42.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.550+0000 7f4c5ffff640 1 --2- 172.21.15.42:0/982485791 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4c600a1330 0x7f4c60135260 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f4c580af040 tx=0x7f4c58019160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:12:42.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.550+0000 7f4c5cff9640 10 client.?.objecter ms_handle_connect 0x7f4c600a1330 2024-09-16T04:12:42.571 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.550+0000 7f4c5cff9640 10 client.?.objecter resend_mon_ops 2024-09-16T04:12:42.572 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.550+0000 7f4c5cff9640 1 -- 172.21.15.42:0/982485791 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c580b1040 con 0x7f4c600a1330 2024-09-16T04:12:42.572 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.550+0000 7f4c5cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:42.572 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.550+0000 7f4c5cff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:12:42.572 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.550+0000 7f4c5cff9640 10 monclient: dump: 2024-09-16T04:12:42.572 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:12:42.572 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:12:42.572 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:42.572 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:42.572 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:42.572 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:12:42.572 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:12:42.572 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:12:42.572 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:12:42.572 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:12:42.573 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.550+0000 7f4c5cff9640 10 monclient: _finish_auth 0 2024-09-16T04:12:42.573 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.550+0000 7f4c5cff9640 10 monclient: _check_auth_tickets 2024-09-16T04:12:42.573 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.550+0000 7f4c5cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:12:42.573 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.550+0000 7f4c5cff9640 1 -- 172.21.15.42:0/982485791 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4c580ba070 con 0x7f4c600a1330 2024-09-16T04:12:42.573 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.550+0000 7f4c5cff9640 10 monclient: handle_config config(0 keys) 2024-09-16T04:12:42.573 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.550+0000 7f4c5cff9640 1 -- 172.21.15.42:0/982485791 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c580b5890 con 0x7f4c600a1330 2024-09-16T04:12:42.573 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.550+0000 7f4c5cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:42.573 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.550+0000 7f4c5cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:12:42.573 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.550+0000 7f4c5cff9640 10 monclient: dump: 2024-09-16T04:12:42.573 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:12:42.573 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:12:42.573 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:42.573 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:42.573 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:42.574 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:12:42.574 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:12:42.574 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:12:42.574 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:12:42.574 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:12:42.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.550+0000 7f4c663a0640 5 monclient: authenticate success, global_id 1034508 2024-09-16T04:12:42.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.550+0000 7f4c663a0640 10 monclient: _renew_subs 2024-09-16T04:12:42.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.550+0000 7f4c663a0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:12:42.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.550+0000 7f4c663a0640 1 -- 172.21.15.42:0/982485791 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c6006e780 con 0x7f4c600a1330 2024-09-16T04:12:42.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.550+0000 7f4c5e7fc640 4 set_mon_vals no callback set 2024-09-16T04:12:42.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.554+0000 7f4c663a0640 10 client.1034508.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:12:42.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.554+0000 7f4c663a0640 10 monclient: _renew_subs 2024-09-16T04:12:42.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.554+0000 7f4c663a0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:12:42.574 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.554+0000 7f4c663a0640 1 -- 172.21.15.42:0/982485791 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c60120380 con 0x7f4c600a1330 2024-09-16T04:12:42.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.554+0000 7f4c663a0640 1 librados: init done 2024-09-16T04:12:42.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.554+0000 7f4c663a0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:12:42.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.554+0000 7f4c663a0640 1 --2- 172.21.15.42:0/982485791 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4c6006dac0 0x7f4c6002d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:12:42.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.554+0000 7f4c663a0640 10 monclient(hunting): start opening mon connection 2024-09-16T04:12:42.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.554+0000 7f4c663a0640 1 -- 172.21.15.42:0/982485791 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f4c601206d0 con 0x7f4c6006dac0 2024-09-16T04:12:42.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.554+0000 7f4c5cff9640 1 -- 172.21.15.42:0/982485791 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f4c580b5a30 con 0x7f4c600a1330 2024-09-16T04:12:42.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.554+0000 7f4c5f7fe640 1 --2- 172.21.15.42:0/982485791 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4c6006dac0 0x7f4c6002d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:12:42.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.554+0000 7f4c5f7fe640 10 monclient: get_auth_request con 0x7f4c6006dac0 auth_method 0 2024-09-16T04:12:42.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.554+0000 7f4c5f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:12:42.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.554+0000 7f4c5f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:42.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.554+0000 7f4c5f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:42.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.554+0000 7f4c5f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:12:42.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.554+0000 7f4c5f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:12:42.575 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.554+0000 7f4c5f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:12:42.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.554+0000 7f4c5cff9640 1 --2- 172.21.15.42:0/982485791 >> v2:172.21.15.154:6812/254845342 conn(0x7f4c44037810 0x7f4c44039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:12:42.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.554+0000 7f4c5cff9640 1 -- 172.21.15.42:0/982485791 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(706..706 src has 681..706) ==== 5040+0+0 (secure 0 0 0) 0x7f4c580e2e90 con 0x7f4c600a1330 2024-09-16T04:12:42.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.554+0000 7f4c5cff9640 10 client.1034508.objecter ms_dispatch 0x7f4c60000ba0 osd_map(706..706 src has 681..706) 2024-09-16T04:12:42.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.554+0000 7f4c5cff9640 3 client.1034508.objecter handle_osd_map got epochs [706,706] > 0 2024-09-16T04:12:42.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.554+0000 7f4c5cff9640 3 client.1034508.objecter handle_osd_map decoding full epoch 706 2024-09-16T04:12:42.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.554+0000 7f4c5f7fe640 10 monclient(hunting): handle_auth_done global_id 1034511 payload 824 2024-09-16T04:12:42.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.554+0000 7f4c5f7fe640 1 -- 172.21.15.42:0/982485791 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4c600a1710 con 0x7f4c6006dac0 2024-09-16T04:12:42.576 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.554+0000 7f4c5cff9640 20 client.1034508.objecter dump_active .. 0 homeless 2024-09-16T04:12:42.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.554+0000 7f4c5cff9640 10 client.1034508.objecter ms_handle_connect 0x7f4c6006dac0 2024-09-16T04:12:42.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.554+0000 7f4c5f7fe640 1 --2- 172.21.15.42:0/982485791 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4c6006dac0 0x7f4c6002d7d0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f4c6013d6b0 tx=0x7f4c4800ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:12:42.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.554+0000 7f4c5cff9640 10 client.1034508.objecter resend_mon_ops 2024-09-16T04:12:42.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.554+0000 7f4c6493d640 1 --2- 172.21.15.42:0/982485791 >> v2:172.21.15.154:6812/254845342 conn(0x7f4c44037810 0x7f4c44039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:12:42.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.554+0000 7f4c6493d640 10 monclient: get_auth_request con 0x7f4c44037810 auth_method 0 2024-09-16T04:12:42.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.554+0000 7f4c5cff9640 1 -- 172.21.15.42:0/982485791 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c48017070 con 0x7f4c6006dac0 2024-09-16T04:12:42.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.554+0000 7f4c5cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:42.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.554+0000 7f4c5cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:12:42.577 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.554+0000 7f4c5cff9640 10 monclient: dump: 2024-09-16T04:12:42.577 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:12:42.577 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:12:42.577 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:42.577 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:42.578 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:42.578 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:12:42.578 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:12:42.578 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:12:42.578 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:12:42.578 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:12:42.578 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.554+0000 7f4c5cff9640 1 -- 172.21.15.42:0/982485791 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f4c4800ab30 con 0x7f4c6006dac0 2024-09-16T04:12:42.578 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.554+0000 7f4c5cff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:12:42.578 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.554+0000 7f4c5cff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:12:42.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.554+0000 7f4c5cff9640 1 -- 172.21.15.42:0/982485791 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4c6006dac0 msgr2=0x7f4c6002d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:42.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.554+0000 7f4c5cff9640 1 --2- 172.21.15.42:0/982485791 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4c6006dac0 0x7f4c6002d7d0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f4c6013d6b0 tx=0x7f4c4800ce60 comp rx=0 tx=0).stop 2024-09-16T04:12:42.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.554+0000 7f4c5cff9640 1 -- 172.21.15.42:0/982485791 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4c6006dac0 msgr2=0x7f4c6002d7d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:12:42.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.554+0000 7f4c5cff9640 1 --2- 172.21.15.42:0/982485791 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4c6006dac0 0x7f4c6002d7d0 secure :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f4c6013d6b0 tx=0x7f4c4800ce60 comp rx=0 tx=0).stop 2024-09-16T04:12:42.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.554+0000 7f4c5cff9640 10 client.1034508.objecter ms_handle_connect 0x7f4c44037810 2024-09-16T04:12:42.579 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.554+0000 7f4c6493d640 1 --2- 172.21.15.42:0/982485791 >> v2:172.21.15.154:6812/254845342 conn(0x7f4c44037810 0x7f4c44039cd0 secure :-1 s=READY pgs=3967 cs=0 l=1 rev1=1 crypto rx=0x7f4c50007fd0 tx=0x7f4c50009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:12:42.580 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.562+0000 7f4c663a0640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:12:42.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.562+0000 7f4c663a0640 1 --2- 172.21.15.42:0/982485791 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4c601357a0 0x7f4c6009d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:12:42.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.562+0000 7f4c663a0640 10 monclient(hunting): start opening mon connection 2024-09-16T04:12:42.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.562+0000 7f4c663a0640 1 -- 172.21.15.42:0/982485791 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f4c60139a40 con 0x7f4c601357a0 2024-09-16T04:12:42.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.562+0000 7f4c5f7fe640 1 --2- 172.21.15.42:0/982485791 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4c601357a0 0x7f4c6009d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:12:42.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.562+0000 7f4c5f7fe640 10 monclient: get_auth_request con 0x7f4c601357a0 auth_method 0 2024-09-16T04:12:42.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.562+0000 7f4c5f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:12:42.581 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.562+0000 7f4c5f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:42.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.562+0000 7f4c5f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:42.584 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.562+0000 7f4c5f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:12:42.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.562+0000 7f4c5f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:12:42.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.562+0000 7f4c5f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:12:42.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.562+0000 7f4c5f7fe640 10 monclient(hunting): handle_auth_done global_id 1034514 payload 824 2024-09-16T04:12:42.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.562+0000 7f4c5f7fe640 1 -- 172.21.15.42:0/982485791 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f4c60069040 con 0x7f4c601357a0 2024-09-16T04:12:42.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.562+0000 7f4c5f7fe640 1 --2- 172.21.15.42:0/982485791 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4c601357a0 0x7f4c6009d5e0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f4c48005c10 tx=0x7f4c480014e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:12:42.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.562+0000 7f4c5cff9640 10 client.1034508.objecter ms_handle_connect 0x7f4c601357a0 2024-09-16T04:12:42.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.562+0000 7f4c5cff9640 10 client.1034508.objecter resend_mon_ops 2024-09-16T04:12:42.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.562+0000 7f4c5cff9640 1 -- 172.21.15.42:0/982485791 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c48017040 con 0x7f4c601357a0 2024-09-16T04:12:42.585 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.562+0000 7f4c5cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:42.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.562+0000 7f4c5cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:12:42.586 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.562+0000 7f4c5cff9640 10 monclient: dump: 2024-09-16T04:12:42.586 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:12:42.586 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:12:42.586 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:42.586 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:42.586 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:42.586 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:12:42.586 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:12:42.586 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:12:42.586 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:12:42.587 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:12:42.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.562+0000 7f4c5cff9640 1 -- 172.21.15.42:0/982485791 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4715 (secure 0 0 0) 0x7f4c4800adc0 con 0x7f4c601357a0 2024-09-16T04:12:42.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.562+0000 7f4c5cff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:12:42.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.562+0000 7f4c5cff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:12:42.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.562+0000 7f4c5cff9640 1 -- 172.21.15.42:0/982485791 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4c601357a0 msgr2=0x7f4c6009d5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:42.587 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.562+0000 7f4c5cff9640 1 --2- 172.21.15.42:0/982485791 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4c601357a0 0x7f4c6009d5e0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f4c48005c10 tx=0x7f4c480014e0 comp rx=0 tx=0).stop 2024-09-16T04:12:42.588 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.562+0000 7f4c5cff9640 1 -- 172.21.15.42:0/982485791 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4c601357a0 msgr2=0x7f4c6009d5e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:12:42.588 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.562+0000 7f4c5cff9640 1 --2- 172.21.15.42:0/982485791 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4c601357a0 0x7f4c6009d5e0 secure :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f4c48005c10 tx=0x7f4c480014e0 comp rx=0 tx=0).stop 2024-09-16T04:12:42.588 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T04:12:42.588 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:12:42.589 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:12:42.589 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "leader", 2024-09-16T04:12:42.589 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 276, 2024-09-16T04:12:42.589 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 242497, 2024-09-16T04:12:42.589 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T04:12:42.589 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T04:12:42.589 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T04:12:42.589 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T04:12:42.589 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:12:42.589 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 0, 2024-09-16T04:12:42.589 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:12:42.589 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T04:12:42.589 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T04:12:42.590 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:12:42.590 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:12:42.590 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:12:42.590 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:12:42.590 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:12:42.590 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:12:42.590 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:12:42.590 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:12:42.590 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:12:42.590 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:12:42.590 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:12:42.590 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:12:42.590 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:12:42.591 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T04:12:42.591 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:12:42.591 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:12:42.591 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:12:42.591 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:12:42.591 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:12:42.591 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:12:42.591 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:12:42.591 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:12:42.591 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:12:42.591 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:12:42.591 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:12:42.591 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:12:42.591 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:12:42.592 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T04:12:42.592 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T04:12:42.592 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T04:12:42.592 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T04:12:42.592 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T04:12:42.592 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T04:12:42.592 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:12:42.592 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:12:42.592 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T04:12:42.592 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T04:12:42.592 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T04:12:42.592 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T04:12:42.592 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T04:12:42.593 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T04:12:42.593 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T04:12:42.593 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:12:42.593 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T04:12:42.593 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:12:42.593 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:12:42.593 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:12:42.593 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:12:42.593 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:12:42.593 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:12:42.593 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:12:42.593 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:12:42.593 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:12:42.594 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:12:42.594 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:12:42.594 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:12:42.594 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T04:12:42.594 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:12:42.594 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T04:12:42.594 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:12:42.594 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:12:42.594 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:12:42.594 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:12:42.594 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:12:42.594 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:12:42.594 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:12:42.594 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T04:12:42.595 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:12:42.595 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:12:42.595 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:12:42.595 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:12:42.595 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T04:12:42.595 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:12:42.595 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:12:42.595 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:12:42.595 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:12:42.595 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T04:12:42.595 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T04:12:42.595 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:12:42.595 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:12:42.595 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:12:42.596 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:12:42.596 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:12:42.596 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:12:42.596 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:12:42.596 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:12:42.596 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:12:42.596 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:12:42.596 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:12:42.596 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T04:12:42.596 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:12:42.596 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:12:42.596 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:12:42.596 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:12:42.596 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T04:12:42.597 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:12:42.597 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:12:42.597 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:12:42.597 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:12:42.597 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T04:12:42.597 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T04:12:42.597 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:12:42.597 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:12:42.597 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:12:42.597 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:12:42.597 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:12:42.597 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:12:42.597 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:12:42.597 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:12:42.598 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:12:42.599 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:12:42.599 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:12:42.599 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T04:12:42.599 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:12:42.599 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:12:42.599 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:12:42.599 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:12:42.599 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T04:12:42.599 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:12:42.599 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:12:42.599 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:12:42.600 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:12:42.600 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T04:12:42.600 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T04:12:42.600 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:12:42.600 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:12:42.600 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:12:42.600 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:12:42.601 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:12:42.601 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:12:42.601 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T04:12:42.601 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T04:12:42.601 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:12:42.601 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:12:42.601 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:12:42.601 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:12:42.601 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:12:42.601 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:12:42.601 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T04:12:42.601 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:12:42.602 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:12:42.602 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:12:42.602 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 6 2024-09-16T04:12:42.602 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:12:42.602 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:12:42.602 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T04:12:42.603 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:12:42.603 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:12:42.603 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:12:42.603 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 6 2024-09-16T04:12:42.603 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:12:42.603 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:12:42.603 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:12:42.603 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T04:12:42.603 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T04:12:42.605 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.562+0000 7f4c663a0640 10 librados: watch_flush enter 2024-09-16T04:12:42.606 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.562+0000 7f4c663a0640 10 librados: watch_flush exit 2024-09-16T04:12:42.606 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.562+0000 7f4c663a0640 20 client.1034508.objecter shutdown clearing up homeless session... 2024-09-16T04:12:42.606 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.562+0000 7f4c663a0640 10 client.1034508.objecter successfully canceled tick 2024-09-16T04:12:42.606 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.562+0000 7f4c663a0640 1 -- 172.21.15.42:0/982485791 >> v2:172.21.15.154:6812/254845342 conn(0x7f4c44037810 msgr2=0x7f4c44039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:42.606 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.562+0000 7f4c663a0640 1 --2- 172.21.15.42:0/982485791 >> v2:172.21.15.154:6812/254845342 conn(0x7f4c44037810 0x7f4c44039cd0 secure :-1 s=READY pgs=3967 cs=0 l=1 rev1=1 crypto rx=0x7f4c50007fd0 tx=0x7f4c50009040 comp rx=0 tx=0).stop 2024-09-16T04:12:42.606 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.562+0000 7f4c663a0640 10 monclient: shutdown 2024-09-16T04:12:42.606 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.562+0000 7f4c663a0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:12:42.606 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.562+0000 7f4c663a0640 1 -- 172.21.15.42:0/982485791 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4c600a1330 msgr2=0x7f4c60135260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:42.606 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.562+0000 7f4c663a0640 1 --2- 172.21.15.42:0/982485791 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f4c600a1330 0x7f4c60135260 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f4c580af040 tx=0x7f4c58019160 comp rx=0 tx=0).stop 2024-09-16T04:12:42.606 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.562+0000 7f4c6493d640 1 -- 172.21.15.42:0/982485791 reap_dead start 2024-09-16T04:12:42.606 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.562+0000 7f4c6493d640 1 -- 172.21.15.42:0/982485791 reap_dead start 2024-09-16T04:12:42.606 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.562+0000 7f4c663a0640 1 -- 172.21.15.42:0/982485791 shutdown_connections 2024-09-16T04:12:42.606 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.562+0000 7f4c663a0640 1 -- 172.21.15.42:0/982485791 >> 172.21.15.42:0/982485791 conn(0x7f4c600a2420 msgr2=0x7f4c6006fd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:12:42.606 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.562+0000 7f4c663a0640 1 -- 172.21.15.42:0/982485791 shutdown_connections 2024-09-16T04:12:42.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.562+0000 7f4c663a0640 1 -- 172.21.15.42:0/982485791 wait complete. 2024-09-16T04:12:42.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.562+0000 7f4c663a0640 1 librados: shutdown 2024-09-16T04:12:42.607 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:12:42.607 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T04:12:42.607 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T04:12:42.725 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.718+0000 7fe0206a7640 10 monclient: get_monmap_and_config 2024-09-16T04:12:42.725 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.718+0000 7fe0206a7640 10 monclient: build_initial_monmap 2024-09-16T04:12:42.725 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.718+0000 7fe0206a7640 10 monclient: monmap: 2024-09-16T04:12:42.725 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:12:42.725 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:12:42.725 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:12:42.724040+0000 2024-09-16T04:12:42.725 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:12:42.724040+0000 2024-09-16T04:12:42.726 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:12:42.726 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:12:42.726 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:12:42.726 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:12:42.726 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:12:42.726 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:12:42.726 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:12:42.726 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:12:42.738 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.722+0000 7fe0206a7640 1 Processor -- start 2024-09-16T04:12:42.738 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.722+0000 7fe0206a7640 1 -- start start 2024-09-16T04:12:42.738 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.722+0000 7fe0206a7640 10 monclient: init 2024-09-16T04:12:42.738 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.722+0000 7fe0206a7640 10 monclient: _reopen_session rank -1 2024-09-16T04:12:42.738 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.722+0000 7fe0206a7640 10 monclient: _add_conns ranks=[4,3,1,0,2] 2024-09-16T04:12:42.738 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.722+0000 7fe0206a7640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fe0180a1330 0x7fe0180a1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:12:42.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.722+0000 7fe0206a7640 10 monclient(hunting): picked mon.noname-c con 0x7fe0180a1330 addr v2:172.21.15.154:3300/0 2024-09-16T04:12:42.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.722+0000 7fe0206a7640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe01806dac0 0x7fe0180a1c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:12:42.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.722+0000 7fe0206a7640 10 monclient(hunting): picked mon.noname-a con 0x7fe01806dac0 addr v2:172.21.15.42:3300/0 2024-09-16T04:12:42.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.722+0000 7fe0206a7640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe018135100 addr v1:172.21.15.154:6789/0 2024-09-16T04:12:42.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.722+0000 7fe0206a7640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe01809c7d0 con 0x7fe018135100 2024-09-16T04:12:42.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.722+0000 7fe0206a7640 10 monclient(hunting): start opening mon connection 2024-09-16T04:12:42.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.722+0000 7fe0206a7640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7fe01809c090 con 0x7fe01806dac0 2024-09-16T04:12:42.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.722+0000 7fe0206a7640 10 monclient(hunting): start opening mon connection 2024-09-16T04:12:42.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.722+0000 7fe0206a7640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fe01809f610 con 0x7fe0180a1330 2024-09-16T04:12:42.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.722+0000 7fe0206a7640 10 monclient(hunting): _renew_subs 2024-09-16T04:12:42.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.722+0000 7fe0206a7640 10 monclient(hunting): authenticate will time out at 2809.693480s 2024-09-16T04:12:42.739 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.722+0000 7fe01dc42640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe01806dac0 0x7fe0180a1c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:12:42.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.722+0000 7fe01dc42640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fe01806dac0 0x7fe0180a1c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:39406/0 (socket says 172.21.15.42:39406) 2024-09-16T04:12:42.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.722+0000 7fe01dc42640 1 -- 172.21.15.42:0/3838317228 learned_addr learned my addr 172.21.15.42:0/3838317228 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:12:42.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.722+0000 7fe01dc42640 10 monclient(hunting): get_auth_request con 0x7fe01806dac0 auth_method 0 2024-09-16T04:12:42.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.722+0000 7fe01dc42640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:12:42.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.722+0000 7fe01dc42640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:42.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.722+0000 7fe01dc42640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:42.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.722+0000 7fe01dc42640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:12:42.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.722+0000 7fe01dc42640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:12:42.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.722+0000 7fe01e443640 1 --2- 172.21.15.42:0/3838317228 >> v2:172.21.15.154:3300/0 conn(0x7fe0180a1330 0x7fe0180a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:12:42.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.722+0000 7fe01dc42640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:12:42.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.722+0000 7fe01e443640 10 monclient(hunting): get_auth_request con 0x7fe0180a1330 auth_method 0 2024-09-16T04:12:42.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.722+0000 7fe01e443640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:12:42.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.722+0000 7fe01e443640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:42.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.722+0000 7fe01e443640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:42.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.722+0000 7fe01dc42640 10 monclient(hunting): handle_auth_done global_id 1034517 payload 274 2024-09-16T04:12:42.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.722+0000 7fe01dc42640 1 -- 172.21.15.42:0/3838317228 >> v2:172.21.15.154:3300/0 conn(0x7fe0180a1330 msgr2=0x7fe0180a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:12:42.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.722+0000 7fe01dc42640 1 --2- 172.21.15.42:0/3838317228 >> v2:172.21.15.154:3300/0 conn(0x7fe0180a1330 0x7fe0180a1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:12:42.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.722+0000 7fe01dc42640 1 -- 172.21.15.42:0/3838317228 >> v1:172.21.15.154:6789/0 conn(0x7fe018135100 legacy=0x7fe018135550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:42.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.722+0000 7fe01dc42640 10 monclient: _finish_hunting 0 2024-09-16T04:12:42.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.722+0000 7fe01dc42640 1 monclient: found mon.noname-a 2024-09-16T04:12:42.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.722+0000 7fe01dc42640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:12:42.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.722+0000 7fe01dc42640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:12:42.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.722+0000 7fe01dc42640 1 -- 172.21.15.42:0/3838317228 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe01809f8f0 con 0x7fe01806dac0 2024-09-16T04:12:42.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.722+0000 7fe01e443640 1 --2- 172.21.15.42:0/3838317228 >> v2:172.21.15.154:3300/0 conn(0x7fe0180a1330 0x7fe0180a1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T04:12:42.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.726+0000 7fe01dc42640 1 --2- 172.21.15.42:0/3838317228 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe01806dac0 0x7fe0180a1c50 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fe01806add0 tx=0x7fe01401c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a469d2f6ad8dd497 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:12:42.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.726+0000 7fe01d441640 1 -- 172.21.15.42:0/3838317228 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0140aa650 con 0x7fe01806dac0 2024-09-16T04:12:42.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.726+0000 7fe01d441640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:42.742 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.726+0000 7fe01d441640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:12:42.742 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.726+0000 7fe01d441640 10 monclient: dump: 2024-09-16T04:12:42.742 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:12:42.742 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:12:42.742 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:42.742 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:42.742 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:42.742 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:12:42.742 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:12:42.743 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:12:42.743 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:12:42.743 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:12:42.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.726+0000 7fe01d441640 10 monclient: _finish_auth 0 2024-09-16T04:12:42.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.726+0000 7fe01d441640 10 monclient: _check_auth_tickets 2024-09-16T04:12:42.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.726+0000 7fe01d441640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:12:42.744 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.726+0000 7fe01d441640 1 -- 172.21.15.42:0/3838317228 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe0140af070 con 0x7fe01806dac0 2024-09-16T04:12:42.744 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.726+0000 7fe0206a7640 5 monclient: authenticate success, global_id 1034517 2024-09-16T04:12:42.744 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.726+0000 7fe01d441640 10 monclient: handle_config config(0 keys) 2024-09-16T04:12:42.744 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.726+0000 7fe0206a7640 10 monclient: get_monmap_and_config success 2024-09-16T04:12:42.744 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.726+0000 7fe01d441640 1 -- 172.21.15.42:0/3838317228 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0140aaa10 con 0x7fe01806dac0 2024-09-16T04:12:42.744 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.726+0000 7fe0206a7640 4 set_mon_vals no callback set 2024-09-16T04:12:42.744 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.726+0000 7fe01d441640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:42.744 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.726+0000 7fe01d441640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:12:42.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.726+0000 7fe01d441640 10 monclient: dump: 2024-09-16T04:12:42.745 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:12:42.745 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:12:42.745 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:42.745 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:42.745 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:42.745 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:12:42.745 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:12:42.745 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:12:42.745 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:12:42.746 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:12:42.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.726+0000 7fe0206a7640 10 monclient: shutdown 2024-09-16T04:12:42.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.726+0000 7fe0206a7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:12:42.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.726+0000 7fe0206a7640 1 -- 172.21.15.42:0/3838317228 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe01806dac0 msgr2=0x7fe0180a1c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:42.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.726+0000 7fe0206a7640 1 --2- 172.21.15.42:0/3838317228 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe01806dac0 0x7fe0180a1c50 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fe01806add0 tx=0x7fe01401c4d0 comp rx=0 tx=0).stop 2024-09-16T04:12:42.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.726+0000 7fe0206a7640 1 -- 172.21.15.42:0/3838317228 shutdown_connections 2024-09-16T04:12:42.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.726+0000 7fe0206a7640 1 --2- 172.21.15.42:0/3838317228 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe01806dac0 0x7fe0180a1c50 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:12:42.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.726+0000 7fe0206a7640 1 --2- 172.21.15.42:0/3838317228 >> v2:172.21.15.154:3300/0 conn(0x7fe0180a1330 0x7fe0180a1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:12:42.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.726+0000 7fe0206a7640 1 -- 172.21.15.42:0/3838317228 >> 172.21.15.42:0/3838317228 conn(0x7fe0180a2420 msgr2=0x7fe0180a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:12:42.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.726+0000 7fe0206a7640 1 -- 172.21.15.42:0/3838317228 shutdown_connections 2024-09-16T04:12:42.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.726+0000 7fe0206a7640 1 -- 172.21.15.42:0/3838317228 wait complete. 2024-09-16T04:12:42.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.726+0000 7fe0206a7640 10 monclient: build_initial_monmap 2024-09-16T04:12:42.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.726+0000 7fe0206a7640 10 monclient: monmap: 2024-09-16T04:12:42.746 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:12:42.747 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:12:42.747 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:12:42.747 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:12:42.747 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:12:42.747 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:12:42.747 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:12:42.747 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:12:42.747 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:12:42.747 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:12:42.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.726+0000 7fe0206a7640 1 librados: starting msgr at 2024-09-16T04:12:42.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.726+0000 7fe0206a7640 1 librados: starting objecter 2024-09-16T04:12:42.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.726+0000 7fe0206a7640 1 Processor -- start 2024-09-16T04:12:42.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.726+0000 7fe0206a7640 1 -- start start 2024-09-16T04:12:42.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.726+0000 7fe0206a7640 1 librados: setting wanted keys 2024-09-16T04:12:42.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.726+0000 7fe0206a7640 1 librados: calling monclient init 2024-09-16T04:12:42.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.726+0000 7fe0206a7640 10 monclient: init 2024-09-16T04:12:42.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.726+0000 7fe0206a7640 10 monclient: _reopen_session rank -1 2024-09-16T04:12:42.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.726+0000 7fe0206a7640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:12:42.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.726+0000 7fe0206a7640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe0180a1330 0x7fe01811bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:12:42.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.726+0000 7fe0206a7640 10 monclient(hunting): picked mon.noname-c con 0x7fe0180a1330 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:12:42.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.726+0000 7fe0206a7640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe018135100 0x7fe01811c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:12:42.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.726+0000 7fe0206a7640 10 monclient(hunting): picked mon.noname-b con 0x7fe018135100 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:12:42.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.726+0000 7fe0206a7640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe01811c9f0 0x7fe018133bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:12:42.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.726+0000 7fe0206a7640 10 monclient(hunting): picked mon.noname-a con 0x7fe01811c9f0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:12:42.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.726+0000 7fe0206a7640 10 monclient(hunting): start opening mon connection 2024-09-16T04:12:42.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.726+0000 7fe0206a7640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fe01813fb80 con 0x7fe01811c9f0 2024-09-16T04:12:42.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.726+0000 7fe0206a7640 10 monclient(hunting): start opening mon connection 2024-09-16T04:12:42.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.726+0000 7fe0206a7640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fe01809ef80 con 0x7fe018135100 2024-09-16T04:12:42.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.726+0000 7fe0206a7640 10 monclient(hunting): start opening mon connection 2024-09-16T04:12:42.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.726+0000 7fe0206a7640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fe01813fd00 con 0x7fe0180a1330 2024-09-16T04:12:42.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.726+0000 7fe0206a7640 10 monclient(hunting): _renew_subs 2024-09-16T04:12:42.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.726+0000 7fe0206a7640 10 monclient(hunting): authenticate will time out at 2809.699504s 2024-09-16T04:12:42.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.726+0000 7fe01e443640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe0180a1330 0x7fe01811bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:12:42.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.726+0000 7fe01e443640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe0180a1330 0x7fe01811bf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:40726/0 (socket says 172.21.15.42:40726) 2024-09-16T04:12:42.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.726+0000 7fe01ec44640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe01811c9f0 0x7fe018133bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:12:42.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.726+0000 7fe01e443640 1 -- 172.21.15.42:0/1561790824 learned_addr learned my addr 172.21.15.42:0/1561790824 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:12:42.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.726+0000 7fe01e443640 10 monclient(hunting): get_auth_request con 0x7fe0180a1330 auth_method 0 2024-09-16T04:12:42.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.726+0000 7fe01e443640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:12:42.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.726+0000 7fe01e443640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:42.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.726+0000 7fe01e443640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:42.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.726+0000 7fe01ec44640 10 monclient(hunting): get_auth_request con 0x7fe01811c9f0 auth_method 0 2024-09-16T04:12:42.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.726+0000 7fe01ec44640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:12:42.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.726+0000 7fe01ec44640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:42.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.726+0000 7fe01ec44640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:42.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.726+0000 7fe01ec44640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:12:42.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.726+0000 7fe01ec44640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:12:42.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.726+0000 7fe01dc42640 1 --2- 172.21.15.42:0/1561790824 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe018135100 0x7fe01811c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:12:42.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.726+0000 7fe01ec44640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:12:42.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.726+0000 7fe01e443640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:12:42.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.726+0000 7fe01e443640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:12:42.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.726+0000 7fe01e443640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:12:42.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.726+0000 7fe01dc42640 10 monclient(hunting): get_auth_request con 0x7fe018135100 auth_method 0 2024-09-16T04:12:42.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.726+0000 7fe01dc42640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:12:42.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.726+0000 7fe01dc42640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:42.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.726+0000 7fe01dc42640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:42.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.726+0000 7fe01e443640 10 monclient(hunting): handle_auth_done global_id 1084106 payload 824 2024-09-16T04:12:42.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.730+0000 7fe01e443640 1 -- 172.21.15.42:0/1561790824 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe018135100 msgr2=0x7fe01811c4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:42.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.730+0000 7fe01e443640 1 --2- 172.21.15.42:0/1561790824 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe018135100 0x7fe01811c4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:12:42.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.730+0000 7fe01e443640 1 -- 172.21.15.42:0/1561790824 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe01811c9f0 msgr2=0x7fe018133bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:42.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.730+0000 7fe01e443640 1 --2- 172.21.15.42:0/1561790824 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe01811c9f0 0x7fe018133bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:12:42.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.730+0000 7fe01e443640 10 monclient: _finish_hunting 0 2024-09-16T04:12:42.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.730+0000 7fe01e443640 1 monclient: found mon.noname-c 2024-09-16T04:12:42.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.730+0000 7fe01e443640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:12:42.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.730+0000 7fe01e443640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T04:12:42.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.730+0000 7fe01e443640 1 -- 172.21.15.42:0/1561790824 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe018071ac0 con 0x7fe0180a1330 2024-09-16T04:12:42.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.730+0000 7fe01ec44640 1 --2- 172.21.15.42:0/1561790824 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fe01811c9f0 0x7fe018133bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:12:42.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.730+0000 7fe01dc42640 1 --2- 172.21.15.42:0/1561790824 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fe018135100 0x7fe01811c4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T04:12:42.753 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.730+0000 7fe01e443640 1 --2- 172.21.15.42:0/1561790824 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe0180a1330 0x7fe01811bf70 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fe00800e840 tx=0x7fe00800ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:12:42.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.730+0000 7fe00effd640 10 client.?.objecter ms_handle_connect 0x7fe0180a1330 2024-09-16T04:12:42.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.730+0000 7fe00effd640 10 client.?.objecter resend_mon_ops 2024-09-16T04:12:42.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.730+0000 7fe00effd640 1 -- 172.21.15.42:0/1561790824 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe008017070 con 0x7fe0180a1330 2024-09-16T04:12:42.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.730+0000 7fe00effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:42.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.730+0000 7fe00effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:12:42.754 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.730+0000 7fe00effd640 10 monclient: dump: 2024-09-16T04:12:42.754 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:12:42.754 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:12:42.754 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:42.754 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:42.754 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:42.755 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:12:42.755 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:12:42.755 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:12:42.755 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:12:42.755 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:12:42.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.730+0000 7fe00effd640 10 monclient: _finish_auth 0 2024-09-16T04:12:42.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.730+0000 7fe00effd640 10 monclient: _check_auth_tickets 2024-09-16T04:12:42.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.730+0000 7fe00effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:12:42.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.730+0000 7fe00effd640 1 -- 172.21.15.42:0/1561790824 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe00800c8d0 con 0x7fe0180a1330 2024-09-16T04:12:42.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.730+0000 7fe00effd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:12:42.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.730+0000 7fe00effd640 1 -- 172.21.15.42:0/1561790824 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe00800f040 con 0x7fe0180a1330 2024-09-16T04:12:42.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.730+0000 7fe0206a7640 5 monclient: authenticate success, global_id 1084106 2024-09-16T04:12:42.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.730+0000 7fe0206a7640 10 monclient: _renew_subs 2024-09-16T04:12:42.755 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.730+0000 7fe0206a7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:12:42.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.730+0000 7fe0206a7640 1 -- 172.21.15.42:0/1561790824 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe01811e960 con 0x7fe0180a1330 2024-09-16T04:12:42.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.730+0000 7fe01cc40640 4 set_mon_vals no callback set 2024-09-16T04:12:42.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.730+0000 7fe00effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:42.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.730+0000 7fe00effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:12:42.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.730+0000 7fe00effd640 10 monclient: dump: 2024-09-16T04:12:42.756 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:12:42.756 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:12:42.756 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:42.756 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:42.756 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:42.756 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:12:42.756 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:12:42.757 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:12:42.757 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:12:42.757 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:12:42.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.730+0000 7fe0206a7640 10 client.1084106.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:12:42.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.730+0000 7fe0206a7640 10 monclient: _renew_subs 2024-09-16T04:12:42.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.730+0000 7fe0206a7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:12:42.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.730+0000 7fe0206a7640 1 -- 172.21.15.42:0/1561790824 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fe01811d6d0 con 0x7fe0180a1330 2024-09-16T04:12:42.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.730+0000 7fe0206a7640 1 librados: init done 2024-09-16T04:12:42.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.730+0000 7fe0206a7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:12:42.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.730+0000 7fe0206a7640 1 --2- 172.21.15.42:0/1561790824 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe01806dac0 0x7fe01802c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:12:42.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.730+0000 7fe0206a7640 10 monclient(hunting): start opening mon connection 2024-09-16T04:12:42.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.730+0000 7fe0206a7640 1 -- 172.21.15.42:0/1561790824 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fe018073a30 con 0x7fe01806dac0 2024-09-16T04:12:42.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.730+0000 7fe01dc42640 1 --2- 172.21.15.42:0/1561790824 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe01806dac0 0x7fe01802c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:12:42.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.730+0000 7fe00effd640 1 -- 172.21.15.42:0/1561790824 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fe00801a070 con 0x7fe0180a1330 2024-09-16T04:12:42.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.730+0000 7fe01dc42640 10 monclient: get_auth_request con 0x7fe01806dac0 auth_method 0 2024-09-16T04:12:42.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.730+0000 7fe01dc42640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:12:42.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.730+0000 7fe01dc42640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:42.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.730+0000 7fe01dc42640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:42.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.730+0000 7fe01dc42640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:12:42.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.730+0000 7fe01dc42640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:12:42.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.730+0000 7fe01dc42640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:12:42.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.730+0000 7fe00effd640 1 --2- 172.21.15.42:0/1561790824 >> v2:172.21.15.154:6812/254845342 conn(0x7fdffc037810 0x7fdffc039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:12:42.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.730+0000 7fe00effd640 1 -- 172.21.15.42:0/1561790824 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(706..706 src has 681..706) ==== 5040+0+0 (secure 0 0 0) 0x7fe008048860 con 0x7fe0180a1330 2024-09-16T04:12:42.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.730+0000 7fe00effd640 10 client.1084106.objecter ms_dispatch 0x7fe018000ba0 osd_map(706..706 src has 681..706) 2024-09-16T04:12:42.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.730+0000 7fe00effd640 3 client.1084106.objecter handle_osd_map got epochs [706,706] > 0 2024-09-16T04:12:42.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.730+0000 7fe00effd640 3 client.1084106.objecter handle_osd_map decoding full epoch 706 2024-09-16T04:12:42.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.730+0000 7fe01dc42640 10 monclient(hunting): handle_auth_done global_id 1084109 payload 824 2024-09-16T04:12:42.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.730+0000 7fe01dc42640 1 -- 172.21.15.42:0/1561790824 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe018135550 con 0x7fe01806dac0 2024-09-16T04:12:42.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.730+0000 7fe00effd640 20 client.1084106.objecter dump_active .. 0 homeless 2024-09-16T04:12:42.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.730+0000 7fe00effd640 10 client.1084106.objecter ms_handle_connect 0x7fe01806dac0 2024-09-16T04:12:42.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.730+0000 7fe01dc42640 1 --2- 172.21.15.42:0/1561790824 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe01806dac0 0x7fe01802c650 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fe0180a21c0 tx=0x7fe014019270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:12:42.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.730+0000 7fe00effd640 10 client.1084106.objecter resend_mon_ops 2024-09-16T04:12:42.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.730+0000 7fe01ec44640 1 --2- 172.21.15.42:0/1561790824 >> v2:172.21.15.154:6812/254845342 conn(0x7fdffc037810 0x7fdffc039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:12:42.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.730+0000 7fe01ec44640 10 monclient: get_auth_request con 0x7fdffc037810 auth_method 0 2024-09-16T04:12:42.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.730+0000 7fe00effd640 1 -- 172.21.15.42:0/1561790824 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0140b9070 con 0x7fe01806dac0 2024-09-16T04:12:42.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.730+0000 7fe00effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:42.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.730+0000 7fe00effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:12:42.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.730+0000 7fe00effd640 10 monclient: dump: 2024-09-16T04:12:42.760 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:12:42.760 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:12:42.760 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:42.760 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:42.760 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:42.760 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:12:42.760 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:12:42.760 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:12:42.760 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:12:42.760 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:12:42.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.730+0000 7fe00effd640 1 -- 172.21.15.42:0/1561790824 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fe0140af040 con 0x7fe01806dac0 2024-09-16T04:12:42.761 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.730+0000 7fe00effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:12:42.761 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.730+0000 7fe00effd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:12:42.761 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.730+0000 7fe00effd640 1 -- 172.21.15.42:0/1561790824 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe01806dac0 msgr2=0x7fe01802c650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:42.761 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.730+0000 7fe00effd640 1 --2- 172.21.15.42:0/1561790824 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe01806dac0 0x7fe01802c650 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fe0180a21c0 tx=0x7fe014019270 comp rx=0 tx=0).stop 2024-09-16T04:12:42.761 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.730+0000 7fe00effd640 1 -- 172.21.15.42:0/1561790824 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe01806dac0 msgr2=0x7fe01802c650 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:12:42.761 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.730+0000 7fe00effd640 1 --2- 172.21.15.42:0/1561790824 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe01806dac0 0x7fe01802c650 secure :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fe0180a21c0 tx=0x7fe014019270 comp rx=0 tx=0).stop 2024-09-16T04:12:42.761 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.730+0000 7fe01ec44640 1 --2- 172.21.15.42:0/1561790824 >> v2:172.21.15.154:6812/254845342 conn(0x7fdffc037810 0x7fdffc039cd0 secure :-1 s=READY pgs=3968 cs=0 l=1 rev1=1 crypto rx=0x7fe0100012f0 tx=0x7fe010009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:12:42.761 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.730+0000 7fe00effd640 10 client.1084106.objecter ms_handle_connect 0x7fdffc037810 2024-09-16T04:12:42.763 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.738+0000 7fe0206a7640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:12:42.763 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.738+0000 7fe0206a7640 1 --2- 172.21.15.42:0/1561790824 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe01802cb90 0x7fe01802cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:12:42.764 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.738+0000 7fe0206a7640 10 monclient(hunting): start opening mon connection 2024-09-16T04:12:42.764 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.738+0000 7fe0206a7640 1 -- 172.21.15.42:0/1561790824 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fe018138a40 con 0x7fe01802cb90 2024-09-16T04:12:42.764 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.738+0000 7fe01dc42640 1 --2- 172.21.15.42:0/1561790824 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe01802cb90 0x7fe01802cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:12:42.764 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.738+0000 7fe01dc42640 10 monclient: get_auth_request con 0x7fe01802cb90 auth_method 0 2024-09-16T04:12:42.764 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.738+0000 7fe01dc42640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:12:42.764 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.738+0000 7fe01dc42640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:42.764 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.738+0000 7fe01dc42640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:42.764 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.738+0000 7fe01dc42640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:12:42.765 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.738+0000 7fe01dc42640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:12:42.765 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.738+0000 7fe01dc42640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:12:42.765 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.738+0000 7fe01dc42640 10 monclient(hunting): handle_auth_done global_id 1084112 payload 824 2024-09-16T04:12:42.765 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.738+0000 7fe01dc42640 1 -- 172.21.15.42:0/1561790824 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fe01813c300 con 0x7fe01802cb90 2024-09-16T04:12:42.765 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.738+0000 7fe01dc42640 1 --2- 172.21.15.42:0/1561790824 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe01802cb90 0x7fe01802cfa0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fe0140190e0 tx=0x7fe014009240 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:12:42.765 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.738+0000 7fe00effd640 10 client.1084106.objecter ms_handle_connect 0x7fe01802cb90 2024-09-16T04:12:42.766 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.738+0000 7fe00effd640 10 client.1084106.objecter resend_mon_ops 2024-09-16T04:12:42.766 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.738+0000 7fe00effd640 1 -- 172.21.15.42:0/1561790824 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe014003bb0 con 0x7fe01802cb90 2024-09-16T04:12:42.766 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.738+0000 7fe00effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:42.766 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.738+0000 7fe00effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:12:42.766 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.738+0000 7fe00effd640 10 monclient: dump: 2024-09-16T04:12:42.767 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:12:42.767 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:12:42.767 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:42.767 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:42.767 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:42.767 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:12:42.767 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:12:42.767 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:12:42.767 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:12:42.768 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:12:42.768 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.738+0000 7fe00effd640 1 -- 172.21.15.42:0/1561790824 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4545 (secure 0 0 0) 0x7fe0140a6040 con 0x7fe01802cb90 2024-09-16T04:12:42.768 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.738+0000 7fe00effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:12:42.768 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.738+0000 7fe00effd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:12:42.768 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.738+0000 7fe00effd640 1 -- 172.21.15.42:0/1561790824 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe01802cb90 msgr2=0x7fe01802cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:42.769 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.738+0000 7fe00effd640 1 --2- 172.21.15.42:0/1561790824 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe01802cb90 0x7fe01802cfa0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fe0140190e0 tx=0x7fe014009240 comp rx=0 tx=0).stop 2024-09-16T04:12:42.769 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.738+0000 7fe00effd640 1 -- 172.21.15.42:0/1561790824 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe01802cb90 msgr2=0x7fe01802cfa0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:12:42.769 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.738+0000 7fe00effd640 1 --2- 172.21.15.42:0/1561790824 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe01802cb90 0x7fe01802cfa0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:12:42.769 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T04:12:42.770 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:12:42.770 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:12:42.770 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T04:12:42.770 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 276, 2024-09-16T04:12:42.770 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 15639, 2024-09-16T04:12:42.771 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T04:12:42.771 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T04:12:42.771 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T04:12:42.771 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T04:12:42.771 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:12:42.771 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 0, 2024-09-16T04:12:42.771 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:12:42.771 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T04:12:42.771 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T04:12:42.771 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:12:42.771 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:12:42.772 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:12:42.772 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:12:42.772 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:12:42.772 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:12:42.772 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:12:42.772 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:12:42.772 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:12:42.772 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:12:42.772 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:12:42.772 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:12:42.772 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:12:42.772 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T04:12:42.772 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:12:42.773 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:12:42.773 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:12:42.773 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:12:42.773 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:12:42.773 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:12:42.773 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:12:42.773 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:12:42.773 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:12:42.773 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:12:42.773 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:12:42.774 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:12:42.774 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:12:42.774 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T04:12:42.774 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T04:12:42.774 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T04:12:42.774 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T04:12:42.775 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T04:12:42.775 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T04:12:42.775 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:12:42.775 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:12:42.775 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T04:12:42.775 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T04:12:42.776 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T04:12:42.776 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T04:12:42.776 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T04:12:42.776 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T04:12:42.776 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T04:12:42.777 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:12:42.777 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T04:12:42.777 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:12:42.777 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:12:42.777 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:12:42.777 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:12:42.777 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:12:42.777 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:12:42.778 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:12:42.778 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:12:42.778 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:12:42.778 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:12:42.778 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:12:42.778 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:12:42.778 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T04:12:42.778 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:12:42.778 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T04:12:42.778 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:12:42.778 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:12:42.778 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:12:42.778 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:12:42.779 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:12:42.779 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:12:42.779 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:12:42.779 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T04:12:42.779 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:12:42.779 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:12:42.779 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:12:42.779 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:12:42.780 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T04:12:42.780 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:12:42.780 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:12:42.780 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:12:42.780 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:12:42.780 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T04:12:42.780 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T04:12:42.781 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:12:42.781 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:12:42.781 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:12:42.781 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:12:42.781 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:12:42.781 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:12:42.781 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:12:42.781 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:12:42.781 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:12:42.781 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:12:42.782 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:12:42.782 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T04:12:42.782 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:12:42.782 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:12:42.782 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:12:42.782 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:12:42.782 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T04:12:42.782 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:12:42.782 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:12:42.782 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:12:42.782 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:12:42.783 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T04:12:42.783 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T04:12:42.783 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:12:42.783 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:12:42.783 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:12:42.783 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:12:42.784 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:12:42.784 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:12:42.784 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:12:42.784 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:12:42.784 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:12:42.784 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:12:42.784 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:12:42.785 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T04:12:42.785 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:12:42.785 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:12:42.785 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:12:42.785 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:12:42.785 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T04:12:42.785 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:12:42.785 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:12:42.786 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:12:42.786 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:12:42.786 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T04:12:42.786 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T04:12:42.786 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:12:42.786 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:12:42.786 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:12:42.786 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:12:42.786 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:12:42.786 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:12:42.786 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T04:12:42.786 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T04:12:42.786 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:12:42.786 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:12:42.787 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:12:42.787 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:12:42.787 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:12:42.787 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:12:42.787 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T04:12:42.787 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:12:42.787 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:12:42.787 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:12:42.787 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 2 2024-09-16T04:12:42.787 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:12:42.787 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:12:42.787 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:12:42.787 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T04:12:42.787 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T04:12:42.792 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.738+0000 7fe0206a7640 10 librados: watch_flush enter 2024-09-16T04:12:42.793 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.738+0000 7fe0206a7640 10 librados: watch_flush exit 2024-09-16T04:12:42.793 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.738+0000 7fe0206a7640 20 client.1084106.objecter shutdown clearing up homeless session... 2024-09-16T04:12:42.793 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.738+0000 7fe0206a7640 10 client.1084106.objecter successfully canceled tick 2024-09-16T04:12:42.793 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.738+0000 7fe0206a7640 1 -- 172.21.15.42:0/1561790824 >> v2:172.21.15.154:6812/254845342 conn(0x7fdffc037810 msgr2=0x7fdffc039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:42.793 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.738+0000 7fe0206a7640 1 --2- 172.21.15.42:0/1561790824 >> v2:172.21.15.154:6812/254845342 conn(0x7fdffc037810 0x7fdffc039cd0 secure :-1 s=READY pgs=3968 cs=0 l=1 rev1=1 crypto rx=0x7fe0100012f0 tx=0x7fe010009040 comp rx=0 tx=0).stop 2024-09-16T04:12:42.793 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.738+0000 7fe0206a7640 10 monclient: shutdown 2024-09-16T04:12:42.793 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.738+0000 7fe0206a7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:12:42.793 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.738+0000 7fe0206a7640 1 -- 172.21.15.42:0/1561790824 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe0180a1330 msgr2=0x7fe01811bf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:42.794 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.738+0000 7fe0206a7640 1 --2- 172.21.15.42:0/1561790824 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fe0180a1330 0x7fe01811bf70 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fe00800e840 tx=0x7fe00800ed00 comp rx=0 tx=0).stop 2024-09-16T04:12:42.794 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.738+0000 7fe01ec44640 1 -- 172.21.15.42:0/1561790824 reap_dead start 2024-09-16T04:12:42.794 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.738+0000 7fe01ec44640 1 -- 172.21.15.42:0/1561790824 reap_dead start 2024-09-16T04:12:42.794 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.738+0000 7fe0206a7640 1 -- 172.21.15.42:0/1561790824 shutdown_connections 2024-09-16T04:12:42.794 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.738+0000 7fe0206a7640 1 -- 172.21.15.42:0/1561790824 >> 172.21.15.42:0/1561790824 conn(0x7fe0180a2420 msgr2=0x7fe0180a4040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:12:42.794 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.738+0000 7fe0206a7640 1 -- 172.21.15.42:0/1561790824 shutdown_connections 2024-09-16T04:12:42.794 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.738+0000 7fe0206a7640 1 -- 172.21.15.42:0/1561790824 wait complete. 2024-09-16T04:12:42.794 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.738+0000 7fe0206a7640 1 librados: shutdown 2024-09-16T04:12:42.795 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:12:42.795 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T04:12:42.795 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T04:12:42.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.918+0000 7f111f7d6640 10 monclient: get_monmap_and_config 2024-09-16T04:12:42.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.918+0000 7f111f7d6640 10 monclient: build_initial_monmap 2024-09-16T04:12:42.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.918+0000 7f111f7d6640 10 monclient: monmap: 2024-09-16T04:12:42.922 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:12:42.922 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:12:42.922 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:12:42.921010+0000 2024-09-16T04:12:42.923 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:12:42.921010+0000 2024-09-16T04:12:42.923 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:12:42.923 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:12:42.923 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:12:42.923 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:12:42.923 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:12:42.923 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:12:42.923 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:12:42.923 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:12:42.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.918+0000 7f111f7d6640 1 Processor -- start 2024-09-16T04:12:42.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.918+0000 7f111f7d6640 1 -- start start 2024-09-16T04:12:42.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.918+0000 7f111f7d6640 10 monclient: init 2024-09-16T04:12:42.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.918+0000 7f111f7d6640 10 monclient: _reopen_session rank -1 2024-09-16T04:12:42.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.918+0000 7f111f7d6640 10 monclient: _add_conns ranks=[1,2,4,3,0] 2024-09-16T04:12:42.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.918+0000 7f111f7d6640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f11180a1330 addr v1:172.21.15.154:6789/0 2024-09-16T04:12:42.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.918+0000 7f111f7d6640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f11180a1710 0x7f11180a1b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:12:42.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.918+0000 7f111f7d6640 10 monclient(hunting): picked mon.noname-b con 0x7f11180a1710 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:12:42.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.918+0000 7f111f7d6640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f11181393a0 0x7f111813b790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:12:42.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.918+0000 7f111f7d6640 10 monclient(hunting): picked mon.noname-c con 0x7f11181393a0 addr v2:172.21.15.154:3300/0 2024-09-16T04:12:42.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.918+0000 7f111f7d6640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f111809e4c0 con 0x7f11180a1330 2024-09-16T04:12:42.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.918+0000 7f111f7d6640 10 monclient(hunting): start opening mon connection 2024-09-16T04:12:42.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.918+0000 7f111f7d6640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f111809edf0 con 0x7f11181393a0 2024-09-16T04:12:42.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.918+0000 7f111f7d6640 10 monclient(hunting): start opening mon connection 2024-09-16T04:12:42.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.918+0000 7f111f7d6640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f111809cd60 con 0x7f11180a1710 2024-09-16T04:12:42.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.918+0000 7f111f7d6640 10 monclient(hunting): _renew_subs 2024-09-16T04:12:42.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.918+0000 7f111f7d6640 10 monclient(hunting): authenticate will time out at 2809.890521s 2024-09-16T04:12:42.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.918+0000 7f111cd71640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f11180a1710 0x7f11180a1b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:12:42.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.918+0000 7f111cd71640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f11180a1710 0x7f11180a1b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:40760/0 (socket says 172.21.15.42:40760) 2024-09-16T04:12:42.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.918+0000 7f111cd71640 1 -- 172.21.15.42:0/2718349886 learned_addr learned my addr 172.21.15.42:0/2718349886 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:12:42.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.918+0000 7f111cd71640 10 monclient(hunting): get_auth_request con 0x7f11180a1710 auth_method 0 2024-09-16T04:12:42.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.918+0000 7f111cd71640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:12:42.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.918+0000 7f111cd71640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:42.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.918+0000 7f111cd71640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:42.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.918+0000 7f111dd73640 1 --2- 172.21.15.42:0/2718349886 >> v2:172.21.15.154:3300/0 conn(0x7f11181393a0 0x7f111813b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:12:42.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.918+0000 7f111cd71640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:12:42.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.918+0000 7f111cd71640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:12:42.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.918+0000 7f111cd71640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:12:42.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.918+0000 7f111dd73640 10 monclient(hunting): get_auth_request con 0x7f11181393a0 auth_method 0 2024-09-16T04:12:42.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.918+0000 7f111dd73640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:12:42.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.918+0000 7f111dd73640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:42.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.918+0000 7f111dd73640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:42.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.918+0000 7f111cd71640 10 monclient(hunting): handle_auth_done global_id 1084115 payload 274 2024-09-16T04:12:42.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.918+0000 7f111cd71640 1 -- 172.21.15.42:0/2718349886 >> v2:172.21.15.154:3300/0 conn(0x7f11181393a0 msgr2=0x7f111813b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:12:42.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.918+0000 7f111cd71640 1 --2- 172.21.15.42:0/2718349886 >> v2:172.21.15.154:3300/0 conn(0x7f11181393a0 0x7f111813b790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:12:42.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.918+0000 7f111cd71640 1 -- 172.21.15.42:0/2718349886 >> v1:172.21.15.154:6789/0 conn(0x7f11180a1330 legacy=0x7f111809f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:42.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.918+0000 7f111cd71640 10 monclient: _finish_hunting 0 2024-09-16T04:12:42.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.918+0000 7f111cd71640 1 monclient: found mon.noname-b 2024-09-16T04:12:42.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.918+0000 7f111cd71640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:12:42.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.918+0000 7f111cd71640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T04:12:42.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.918+0000 7f111cd71640 1 -- 172.21.15.42:0/2718349886 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11180a2050 con 0x7f11180a1710 2024-09-16T04:12:42.929 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.918+0000 7f111dd73640 1 --2- 172.21.15.42:0/2718349886 >> v2:172.21.15.154:3300/0 conn(0x7f11181393a0 0x7f111813b790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T04:12:42.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.922+0000 7f111cd71640 1 --2- 172.21.15.42:0/2718349886 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f11180a1710 0x7f11180a1b10 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f110c00d150 tx=0x7f110c01eaf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5b3a9b4ddc6b32b9 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:12:42.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.922+0000 7f110bfff640 1 -- 172.21.15.42:0/2718349886 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f110c00b2d0 con 0x7f11180a1710 2024-09-16T04:12:42.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.922+0000 7f110bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:42.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.922+0000 7f110bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:12:42.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.922+0000 7f110bfff640 10 monclient: dump: 2024-09-16T04:12:42.935 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:12:42.936 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:12:42.936 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:42.936 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:42.936 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:42.936 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:12:42.936 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:12:42.936 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:12:42.936 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:12:42.936 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:12:42.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.922+0000 7f110bfff640 10 monclient: _finish_auth 0 2024-09-16T04:12:42.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.922+0000 7f110bfff640 10 monclient: _check_auth_tickets 2024-09-16T04:12:42.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.922+0000 7f110bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:12:42.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.922+0000 7f110bfff640 1 -- 172.21.15.42:0/2718349886 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f110c00b470 con 0x7f11180a1710 2024-09-16T04:12:42.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.922+0000 7f110bfff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:12:42.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.922+0000 7f110bfff640 1 -- 172.21.15.42:0/2718349886 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f110c0a7af0 con 0x7f11180a1710 2024-09-16T04:12:42.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.922+0000 7f111f7d6640 5 monclient: authenticate success, global_id 1084115 2024-09-16T04:12:42.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.922+0000 7f111f7d6640 10 monclient: get_monmap_and_config success 2024-09-16T04:12:42.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.922+0000 7f111f7d6640 4 set_mon_vals no callback set 2024-09-16T04:12:42.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.922+0000 7f110bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:42.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.922+0000 7f110bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:12:42.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.922+0000 7f110bfff640 10 monclient: dump: 2024-09-16T04:12:42.937 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:12:42.937 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:12:42.937 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:42.937 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:42.937 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:42.938 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:12:42.938 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:12:42.938 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:12:42.938 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:12:42.938 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:12:42.938 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.922+0000 7f111f7d6640 10 monclient: shutdown 2024-09-16T04:12:42.938 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.922+0000 7f111f7d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:12:42.938 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.922+0000 7f111f7d6640 1 -- 172.21.15.42:0/2718349886 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f11180a1710 msgr2=0x7f11180a1b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:42.938 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.922+0000 7f111f7d6640 1 --2- 172.21.15.42:0/2718349886 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f11180a1710 0x7f11180a1b10 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f110c00d150 tx=0x7f110c01eaf0 comp rx=0 tx=0).stop 2024-09-16T04:12:42.938 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.922+0000 7f111f7d6640 1 -- 172.21.15.42:0/2718349886 shutdown_connections 2024-09-16T04:12:42.938 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.922+0000 7f111f7d6640 1 --2- 172.21.15.42:0/2718349886 >> v2:172.21.15.154:3300/0 conn(0x7f11181393a0 0x7f111813b790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:12:42.938 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.922+0000 7f111f7d6640 1 --2- 172.21.15.42:0/2718349886 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f11180a1710 0x7f11180a1b10 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:12:42.938 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.922+0000 7f111f7d6640 1 -- 172.21.15.42:0/2718349886 >> 172.21.15.42:0/2718349886 conn(0x7f11180a2420 msgr2=0x7f11180a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:12:42.939 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.922+0000 7f111f7d6640 1 -- 172.21.15.42:0/2718349886 shutdown_connections 2024-09-16T04:12:42.939 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.922+0000 7f111f7d6640 1 -- 172.21.15.42:0/2718349886 wait complete. 2024-09-16T04:12:42.939 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.922+0000 7f111f7d6640 10 monclient: build_initial_monmap 2024-09-16T04:12:42.939 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.922+0000 7f111f7d6640 10 monclient: monmap: 2024-09-16T04:12:42.939 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:12:42.939 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:12:42.939 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:12:42.939 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:12:42.939 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:12:42.939 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:12:42.939 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:12:42.939 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:12:42.939 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:12:42.940 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:12:42.940 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.922+0000 7f111f7d6640 1 librados: starting msgr at 2024-09-16T04:12:42.940 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.922+0000 7f111f7d6640 1 librados: starting objecter 2024-09-16T04:12:42.940 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.922+0000 7f111f7d6640 1 Processor -- start 2024-09-16T04:12:42.940 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.922+0000 7f111f7d6640 1 -- start start 2024-09-16T04:12:42.940 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.922+0000 7f111f7d6640 1 librados: setting wanted keys 2024-09-16T04:12:42.940 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.922+0000 7f111f7d6640 1 librados: calling monclient init 2024-09-16T04:12:42.940 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.922+0000 7f111f7d6640 10 monclient: init 2024-09-16T04:12:42.940 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.922+0000 7f111f7d6640 10 monclient: _reopen_session rank -1 2024-09-16T04:12:42.940 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.922+0000 7f111f7d6640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:12:42.941 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.922+0000 7f111f7d6640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f11180a1330 0x7f11181cef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:12:42.941 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.922+0000 7f111f7d6640 10 monclient(hunting): picked mon.noname-a con 0x7f11180a1330 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:12:42.941 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.922+0000 7f111f7d6640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f11180a1710 0x7f11181325c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:12:42.941 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.926+0000 7f111f7d6640 10 monclient(hunting): picked mon.noname-c con 0x7f11180a1710 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:12:42.941 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.926+0000 7f111f7d6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f11181393a0 0x7f1118132b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:12:42.941 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.926+0000 7f111f7d6640 10 monclient(hunting): picked mon.noname-b con 0x7f11181393a0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:12:42.941 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.926+0000 7f111f7d6640 10 monclient(hunting): start opening mon connection 2024-09-16T04:12:42.941 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.926+0000 7f111f7d6640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f111809c5f0 con 0x7f11180a1330 2024-09-16T04:12:42.941 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.926+0000 7f111f7d6640 10 monclient(hunting): start opening mon connection 2024-09-16T04:12:42.941 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.926+0000 7f111f7d6640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f111809c470 con 0x7f11181393a0 2024-09-16T04:12:42.941 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.926+0000 7f111f7d6640 10 monclient(hunting): start opening mon connection 2024-09-16T04:12:42.941 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.926+0000 7f111f7d6640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f111809c770 con 0x7f11180a1710 2024-09-16T04:12:42.942 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.926+0000 7f111f7d6640 10 monclient(hunting): _renew_subs 2024-09-16T04:12:42.942 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.926+0000 7f111f7d6640 10 monclient(hunting): authenticate will time out at 2809.896071s 2024-09-16T04:12:42.942 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.926+0000 7f111cd71640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f11180a1710 0x7f11181325c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:12:42.942 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.926+0000 7f111cd71640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f11180a1710 0x7f11181325c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:40768/0 (socket says 172.21.15.42:40768) 2024-09-16T04:12:42.942 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.926+0000 7f111d572640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f11180a1330 0x7f11181cef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:12:42.942 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.926+0000 7f111cd71640 1 -- 172.21.15.42:0/2396123631 learned_addr learned my addr 172.21.15.42:0/2396123631 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:12:42.942 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.926+0000 7f111cd71640 10 monclient(hunting): get_auth_request con 0x7f11180a1710 auth_method 0 2024-09-16T04:12:42.942 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.926+0000 7f111cd71640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:12:42.942 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.926+0000 7f111cd71640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:42.942 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.926+0000 7f111cd71640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:42.942 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.926+0000 7f111d572640 10 monclient(hunting): get_auth_request con 0x7f11180a1330 auth_method 0 2024-09-16T04:12:42.942 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.926+0000 7f111d572640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:12:42.942 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.926+0000 7f111d572640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:42.943 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.926+0000 7f111d572640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:42.943 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.926+0000 7f111cd71640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:12:42.943 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.926+0000 7f111cd71640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:12:42.943 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.926+0000 7f111cd71640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:12:42.943 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.926+0000 7f111d572640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:12:42.943 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.926+0000 7f111d572640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:12:42.943 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.926+0000 7f111d572640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:12:42.943 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.926+0000 7f111dd73640 1 --2- 172.21.15.42:0/2396123631 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f11181393a0 0x7f1118132b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:12:42.943 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.926+0000 7f111cd71640 10 monclient(hunting): handle_auth_done global_id 1084118 payload 824 2024-09-16T04:12:42.943 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.926+0000 7f111cd71640 1 -- 172.21.15.42:0/2396123631 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f11181393a0 msgr2=0x7f1118132b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:42.943 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.926+0000 7f111cd71640 1 --2- 172.21.15.42:0/2396123631 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f11181393a0 0x7f1118132b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:12:42.943 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.926+0000 7f111cd71640 1 -- 172.21.15.42:0/2396123631 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f11180a1330 msgr2=0x7f11181cef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:42.943 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.926+0000 7f111cd71640 1 --2- 172.21.15.42:0/2396123631 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f11180a1330 0x7f11181cef10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:12:42.944 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.926+0000 7f111cd71640 10 monclient: _finish_hunting 0 2024-09-16T04:12:42.944 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.926+0000 7f111cd71640 1 monclient: found mon.noname-c 2024-09-16T04:12:42.944 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.926+0000 7f111cd71640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:12:42.944 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.926+0000 7f111cd71640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T04:12:42.944 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.926+0000 7f111cd71640 1 -- 172.21.15.42:0/2396123631 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1118073a30 con 0x7f11180a1710 2024-09-16T04:12:42.944 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.926+0000 7f111cd71640 1 --2- 172.21.15.42:0/2396123631 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f11180a1710 0x7f11181325c0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f110c01c720 tx=0x7f110c01b890 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:12:42.944 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.926+0000 7f1109ffb640 10 client.?.objecter ms_handle_connect 0x7f11180a1710 2024-09-16T04:12:42.944 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.926+0000 7f1109ffb640 10 client.?.objecter resend_mon_ops 2024-09-16T04:12:42.944 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.926+0000 7f1109ffb640 1 -- 172.21.15.42:0/2396123631 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f110c009600 con 0x7f11180a1710 2024-09-16T04:12:42.944 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.926+0000 7f1109ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:42.944 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.926+0000 7f1109ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:12:42.944 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.926+0000 7f1109ffb640 10 monclient: dump: 2024-09-16T04:12:42.944 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:12:42.945 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:12:42.945 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:42.945 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:42.945 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:42.945 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:12:42.945 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:12:42.945 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:12:42.945 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:12:42.945 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:12:42.945 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.926+0000 7f1109ffb640 10 monclient: _finish_auth 0 2024-09-16T04:12:42.945 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.926+0000 7f1109ffb640 10 monclient: _check_auth_tickets 2024-09-16T04:12:42.945 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.926+0000 7f1109ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:12:42.945 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.926+0000 7f111f7d6640 5 monclient: authenticate success, global_id 1084118 2024-09-16T04:12:42.946 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.926+0000 7f111f7d6640 10 monclient: _renew_subs 2024-09-16T04:12:42.946 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.926+0000 7f1109ffb640 1 -- 172.21.15.42:0/2396123631 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f110c0097e0 con 0x7f11180a1710 2024-09-16T04:12:42.946 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.926+0000 7f111f7d6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:12:42.946 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.926+0000 7f111f7d6640 1 -- 172.21.15.42:0/2396123631 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1118072030 con 0x7f11180a1710 2024-09-16T04:12:42.946 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.926+0000 7f1109ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:12:42.946 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.926+0000 7f1109ffb640 1 -- 172.21.15.42:0/2396123631 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f110c009bd0 con 0x7f11180a1710 2024-09-16T04:12:42.946 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.926+0000 7f1109ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:42.946 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.926+0000 7f1109ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:12:42.946 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.926+0000 7f1109ffb640 10 monclient: dump: 2024-09-16T04:12:42.946 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:12:42.946 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:12:42.946 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:42.946 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:42.946 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:42.947 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:12:42.947 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:12:42.947 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:12:42.947 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:12:42.947 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:12:42.947 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.926+0000 7f111f7d6640 10 client.1084118.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:12:42.947 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.926+0000 7f110b7fe640 4 set_mon_vals no callback set 2024-09-16T04:12:42.947 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.926+0000 7f111f7d6640 10 monclient: _renew_subs 2024-09-16T04:12:42.947 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.926+0000 7f111f7d6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:12:42.947 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.926+0000 7f111f7d6640 1 -- 172.21.15.42:0/2396123631 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f11181cfd70 con 0x7f11180a1710 2024-09-16T04:12:42.947 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.926+0000 7f111f7d6640 1 librados: init done 2024-09-16T04:12:42.947 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.926+0000 7f111f7d6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:12:42.947 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.926+0000 7f111f7d6640 1 --2- 172.21.15.42:0/2396123631 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f111811cfc0 0x7f111811d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:12:42.947 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.926+0000 7f111f7d6640 10 monclient(hunting): start opening mon connection 2024-09-16T04:12:42.948 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.926+0000 7f111f7d6640 1 -- 172.21.15.42:0/2396123631 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f111813ba90 con 0x7f111811cfc0 2024-09-16T04:12:42.948 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.926+0000 7f1109ffb640 1 -- 172.21.15.42:0/2396123631 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f110c0c0020 con 0x7f11180a1710 2024-09-16T04:12:42.948 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.926+0000 7f1109ffb640 1 --2- 172.21.15.42:0/2396123631 >> v2:172.21.15.154:6812/254845342 conn(0x7f10fc037810 0x7f10fc039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:12:42.948 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.926+0000 7f1109ffb640 1 -- 172.21.15.42:0/2396123631 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(706..706 src has 681..706) ==== 5040+0+0 (secure 0 0 0) 0x7f110c0e3700 con 0x7f11180a1710 2024-09-16T04:12:42.948 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.926+0000 7f1109ffb640 10 client.1084118.objecter ms_dispatch 0x7f1118000ba0 osd_map(706..706 src has 681..706) 2024-09-16T04:12:42.948 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.926+0000 7f1109ffb640 3 client.1084118.objecter handle_osd_map got epochs [706,706] > 0 2024-09-16T04:12:42.948 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.926+0000 7f1109ffb640 3 client.1084118.objecter handle_osd_map decoding full epoch 706 2024-09-16T04:12:42.948 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.926+0000 7f1109ffb640 20 client.1084118.objecter dump_active .. 0 homeless 2024-09-16T04:12:42.948 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.926+0000 7f111d572640 1 --2- 172.21.15.42:0/2396123631 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f111811cfc0 0x7f111811d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:12:42.948 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.926+0000 7f111d572640 10 monclient: get_auth_request con 0x7f111811cfc0 auth_method 0 2024-09-16T04:12:42.948 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.926+0000 7f111d572640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:12:42.948 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.926+0000 7f111d572640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:42.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.926+0000 7f111d572640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:42.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.926+0000 7f111dd73640 1 --2- 172.21.15.42:0/2396123631 >> v2:172.21.15.154:6812/254845342 conn(0x7f10fc037810 0x7f10fc039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:12:42.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.926+0000 7f111d572640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:12:42.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.926+0000 7f111d572640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:12:42.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.926+0000 7f111d572640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:12:42.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.926+0000 7f111dd73640 10 monclient: get_auth_request con 0x7f10fc037810 auth_method 0 2024-09-16T04:12:42.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.926+0000 7f111d572640 10 monclient(hunting): handle_auth_done global_id 1074178 payload 824 2024-09-16T04:12:42.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.926+0000 7f111d572640 1 -- 172.21.15.42:0/2396123631 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f11180a1b10 con 0x7f111811cfc0 2024-09-16T04:12:42.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.926+0000 7f111dd73640 1 --2- 172.21.15.42:0/2396123631 >> v2:172.21.15.154:6812/254845342 conn(0x7f10fc037810 0x7f10fc039cd0 secure :-1 s=READY pgs=3969 cs=0 l=1 rev1=1 crypto rx=0x7f111811f470 tx=0x7f1110008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:12:42.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.926+0000 7f1109ffb640 10 client.1084118.objecter ms_handle_connect 0x7f10fc037810 2024-09-16T04:12:42.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.926+0000 7f1109ffb640 10 client.1084118.objecter ms_handle_connect 0x7f111811cfc0 2024-09-16T04:12:42.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.926+0000 7f1109ffb640 10 client.1084118.objecter resend_mon_ops 2024-09-16T04:12:42.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.926+0000 7f111d572640 1 --2- 172.21.15.42:0/2396123631 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f111811cfc0 0x7f111811d3a0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f111400f540 tx=0x7f111400fa00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:12:42.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.926+0000 7f1109ffb640 1 -- 172.21.15.42:0/2396123631 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1114019070 con 0x7f111811cfc0 2024-09-16T04:12:42.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.926+0000 7f1109ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:42.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.926+0000 7f1109ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:12:42.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.926+0000 7f1109ffb640 10 monclient: dump: 2024-09-16T04:12:42.950 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:12:42.950 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:12:42.950 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:42.950 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:42.950 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:42.950 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:12:42.950 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:12:42.950 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:12:42.950 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:12:42.951 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:12:42.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.930+0000 7f1109ffb640 1 -- 172.21.15.42:0/2396123631 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f1114005320 con 0x7f111811cfc0 2024-09-16T04:12:42.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.930+0000 7f1109ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:12:42.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.930+0000 7f1109ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:12:42.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.930+0000 7f1109ffb640 1 -- 172.21.15.42:0/2396123631 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f111811cfc0 msgr2=0x7f111811d3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:42.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.930+0000 7f1109ffb640 1 --2- 172.21.15.42:0/2396123631 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f111811cfc0 0x7f111811d3a0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f111400f540 tx=0x7f111400fa00 comp rx=0 tx=0).stop 2024-09-16T04:12:42.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.930+0000 7f1109ffb640 1 -- 172.21.15.42:0/2396123631 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f111811cfc0 msgr2=0x7f111811d3a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:12:42.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.930+0000 7f1109ffb640 1 --2- 172.21.15.42:0/2396123631 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f111811cfc0 0x7f111811d3a0 secure :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f111400f540 tx=0x7f111400fa00 comp rx=0 tx=0).stop 2024-09-16T04:12:42.953 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.934+0000 7f111f7d6640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:12:42.953 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.934+0000 7f111f7d6640 1 --2- 172.21.15.42:0/2396123631 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f111811d8e0 0x7f1118137390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:12:42.953 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.934+0000 7f111f7d6640 10 monclient(hunting): start opening mon connection 2024-09-16T04:12:42.953 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.934+0000 7f111f7d6640 1 -- 172.21.15.42:0/2396123631 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f1118069040 con 0x7f111811d8e0 2024-09-16T04:12:42.954 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.934+0000 7f111d572640 1 --2- 172.21.15.42:0/2396123631 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f111811d8e0 0x7f1118137390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:12:42.954 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.934+0000 7f111d572640 10 monclient: get_auth_request con 0x7f111811d8e0 auth_method 0 2024-09-16T04:12:42.954 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.934+0000 7f111d572640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:12:42.954 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.934+0000 7f111d572640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:42.954 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.934+0000 7f111d572640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:42.954 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.934+0000 7f111d572640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:12:42.954 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.934+0000 7f111d572640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:12:42.954 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.934+0000 7f111d572640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:12:42.954 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.934+0000 7f111d572640 10 monclient(hunting): handle_auth_done global_id 1074181 payload 824 2024-09-16T04:12:42.954 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.934+0000 7f111d572640 1 -- 172.21.15.42:0/2396123631 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f111811dcc0 con 0x7f111811d8e0 2024-09-16T04:12:42.954 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.938+0000 7f1109ffb640 10 client.1084118.objecter ms_handle_connect 0x7f111811d8e0 2024-09-16T04:12:42.954 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.938+0000 7f1109ffb640 10 client.1084118.objecter resend_mon_ops 2024-09-16T04:12:42.954 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.938+0000 7f111d572640 1 --2- 172.21.15.42:0/2396123631 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f111811d8e0 0x7f1118137390 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f111400f540 tx=0x7f1114007f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:12:42.955 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.938+0000 7f1109ffb640 1 -- 172.21.15.42:0/2396123631 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1114019070 con 0x7f111811d8e0 2024-09-16T04:12:42.955 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.938+0000 7f1109ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:42.955 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.938+0000 7f1109ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:12:42.955 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.938+0000 7f1109ffb640 10 monclient: dump: 2024-09-16T04:12:42.955 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:12:42.955 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:12:42.955 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:42.955 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:42.955 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:42.955 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:12:42.955 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:12:42.955 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:12:42.955 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:12:42.956 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:12:42.956 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.938+0000 7f1109ffb640 1 -- 172.21.15.42:0/2396123631 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4712 (secure 0 0 0) 0x7f11140073a0 con 0x7f111811d8e0 2024-09-16T04:12:42.956 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.938+0000 7f1109ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:12:42.956 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.938+0000 7f1109ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:12:42.956 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.938+0000 7f1109ffb640 1 -- 172.21.15.42:0/2396123631 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f111811d8e0 msgr2=0x7f1118137390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:42.956 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.938+0000 7f1109ffb640 1 --2- 172.21.15.42:0/2396123631 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f111811d8e0 0x7f1118137390 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f111400f540 tx=0x7f1114007f90 comp rx=0 tx=0).stop 2024-09-16T04:12:42.956 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.938+0000 7f1109ffb640 1 -- 172.21.15.42:0/2396123631 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f111811d8e0 msgr2=0x7f1118137390 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:12:42.956 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.938+0000 7f1109ffb640 1 --2- 172.21.15.42:0/2396123631 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f111811d8e0 0x7f1118137390 secure :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f111400f540 tx=0x7f1114007f90 comp rx=0 tx=0).stop 2024-09-16T04:12:42.956 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T04:12:42.956 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:12:42.957 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:12:42.957 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T04:12:42.957 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 276, 2024-09-16T04:12:42.957 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 76109, 2024-09-16T04:12:42.957 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T04:12:42.957 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T04:12:42.957 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T04:12:42.957 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T04:12:42.957 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:12:42.957 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 0, 2024-09-16T04:12:42.957 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:12:42.957 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T04:12:42.958 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T04:12:42.958 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:12:42.958 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:12:42.958 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:12:42.958 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:12:42.958 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:12:42.958 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:12:42.958 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:12:42.958 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:12:42.958 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:12:42.958 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:12:42.958 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:12:42.958 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:12:42.959 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:12:42.959 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T04:12:42.959 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:12:42.959 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:12:42.959 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:12:42.959 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:12:42.959 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:12:42.959 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:12:42.959 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:12:42.959 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:12:42.959 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:12:42.959 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:12:42.959 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:12:42.959 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:12:42.960 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:12:42.960 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T04:12:42.960 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T04:12:42.960 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T04:12:42.960 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T04:12:42.960 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T04:12:42.960 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T04:12:42.960 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:12:42.960 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:12:42.960 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T04:12:42.960 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T04:12:42.960 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T04:12:42.960 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T04:12:42.961 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T04:12:42.961 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T04:12:42.961 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T04:12:42.961 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:12:42.961 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T04:12:42.961 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:12:42.961 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:12:42.961 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:12:42.961 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:12:42.961 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:12:42.962 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:12:42.962 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:12:42.962 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:12:42.962 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:12:42.962 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:12:42.962 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:12:42.962 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:12:42.962 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T04:12:42.962 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:12:42.962 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T04:12:42.962 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:12:42.962 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:12:42.963 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:12:42.963 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:12:42.963 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:12:42.963 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:12:42.963 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:12:42.963 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T04:12:42.963 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:12:42.963 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:12:42.963 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:12:42.963 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:12:42.963 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T04:12:42.963 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:12:42.963 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:12:42.964 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:12:42.964 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:12:42.964 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T04:12:42.964 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T04:12:42.964 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:12:42.964 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:12:42.964 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:12:42.964 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:12:42.964 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:12:42.964 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:12:42.964 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:12:42.964 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:12:42.964 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:12:42.964 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:12:42.965 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:12:42.965 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T04:12:42.965 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:12:42.965 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:12:42.965 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:12:42.965 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:12:42.965 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T04:12:42.965 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:12:42.965 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:12:42.965 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:12:42.965 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:12:42.965 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T04:12:42.965 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T04:12:42.965 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:12:42.966 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:12:42.966 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:12:42.966 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:12:42.966 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:12:42.966 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:12:42.966 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:12:42.966 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:12:42.966 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:12:42.966 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:12:42.966 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:12:42.966 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T04:12:42.966 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:12:42.966 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:12:42.967 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:12:42.967 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:12:42.967 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T04:12:42.967 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:12:42.967 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:12:42.967 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:12:42.967 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:12:42.967 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T04:12:42.967 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T04:12:42.967 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:12:42.967 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:12:42.967 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:12:42.967 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:12:42.967 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:12:42.968 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:12:42.968 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T04:12:42.968 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T04:12:42.968 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:12:42.968 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:12:42.968 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:12:42.968 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:12:42.968 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:12:42.968 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:12:42.968 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T04:12:42.968 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:12:42.968 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:12:42.968 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:12:42.969 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 2 2024-09-16T04:12:42.969 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:12:42.969 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:12:42.969 INFO:teuthology.orchestra.run.smithi042.stdout: "mgr": [ 2024-09-16T04:12:42.969 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:12:42.969 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:12:42.969 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:12:42.969 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:12:42.969 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:12:42.969 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:12:42.969 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:12:42.969 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T04:12:42.969 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T04:12:42.971 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.938+0000 7f111f7d6640 10 librados: watch_flush enter 2024-09-16T04:12:42.972 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.938+0000 7f111f7d6640 10 librados: watch_flush exit 2024-09-16T04:12:42.972 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.938+0000 7f111f7d6640 20 client.1084118.objecter shutdown clearing up homeless session... 2024-09-16T04:12:42.972 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.938+0000 7f111f7d6640 10 client.1084118.objecter successfully canceled tick 2024-09-16T04:12:42.972 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.938+0000 7f111f7d6640 1 -- 172.21.15.42:0/2396123631 >> v2:172.21.15.154:6812/254845342 conn(0x7f10fc037810 msgr2=0x7f10fc039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:42.972 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.938+0000 7f111f7d6640 1 --2- 172.21.15.42:0/2396123631 >> v2:172.21.15.154:6812/254845342 conn(0x7f10fc037810 0x7f10fc039cd0 secure :-1 s=READY pgs=3969 cs=0 l=1 rev1=1 crypto rx=0x7f111811f470 tx=0x7f1110008040 comp rx=0 tx=0).stop 2024-09-16T04:12:42.972 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.938+0000 7f111f7d6640 10 monclient: shutdown 2024-09-16T04:12:42.972 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.938+0000 7f111f7d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:12:42.972 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.938+0000 7f111f7d6640 1 -- 172.21.15.42:0/2396123631 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f11180a1710 msgr2=0x7f11181325c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:42.972 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.938+0000 7f111f7d6640 1 --2- 172.21.15.42:0/2396123631 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f11180a1710 0x7f11181325c0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f110c01c720 tx=0x7f110c01b890 comp rx=0 tx=0).stop 2024-09-16T04:12:42.972 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.938+0000 7f111dd73640 1 -- 172.21.15.42:0/2396123631 reap_dead start 2024-09-16T04:12:42.972 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.938+0000 7f111f7d6640 1 -- 172.21.15.42:0/2396123631 shutdown_connections 2024-09-16T04:12:42.972 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.938+0000 7f111f7d6640 1 --2- 172.21.15.42:0/2396123631 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f11180a1710 0x7f11181325c0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:12:42.973 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.938+0000 7f111f7d6640 1 -- 172.21.15.42:0/2396123631 >> 172.21.15.42:0/2396123631 conn(0x7f11180a2420 msgr2=0x7f1118134590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:12:42.973 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.938+0000 7f111f7d6640 1 -- 172.21.15.42:0/2396123631 shutdown_connections 2024-09-16T04:12:42.973 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.938+0000 7f111f7d6640 1 -- 172.21.15.42:0/2396123631 wait complete. 2024-09-16T04:12:42.973 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:42.938+0000 7f111f7d6640 1 librados: shutdown 2024-09-16T04:12:42.973 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:12:42.973 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T04:12:42.973 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T04:12:43.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.082+0000 7f9a935a6640 10 monclient: get_monmap_and_config 2024-09-16T04:12:43.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.082+0000 7f9a935a6640 10 monclient: build_initial_monmap 2024-09-16T04:12:43.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.082+0000 7f9a935a6640 10 monclient: monmap: 2024-09-16T04:12:43.089 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:12:43.089 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:12:43.089 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:12:43.087250+0000 2024-09-16T04:12:43.089 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:12:43.087250+0000 2024-09-16T04:12:43.089 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:12:43.089 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:12:43.089 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:12:43.089 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:12:43.089 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:12:43.089 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:12:43.089 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:12:43.089 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:12:43.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.086+0000 7f9a935a6640 1 Processor -- start 2024-09-16T04:12:43.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.086+0000 7f9a935a6640 1 -- start start 2024-09-16T04:12:43.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.086+0000 7f9a935a6640 10 monclient: init 2024-09-16T04:12:43.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.086+0000 7f9a935a6640 10 monclient: _reopen_session rank -1 2024-09-16T04:12:43.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.086+0000 7f9a935a6640 10 monclient: _add_conns ranks=[0,4,1,2,3] 2024-09-16T04:12:43.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.086+0000 7f9a935a6640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9a8c0a1330 addr v1:172.21.15.42:6789/0 2024-09-16T04:12:43.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.086+0000 7f9a935a6640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f9a8c06dac0 0x7f9a8c13a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:12:43.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.086+0000 7f9a935a6640 10 monclient(hunting): picked mon.noname-c con 0x7f9a8c06dac0 addr v2:172.21.15.154:3300/0 2024-09-16T04:12:43.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.086+0000 7f9a935a6640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9a8c0a1e20 addr v1:172.21.15.154:6789/0 2024-09-16T04:12:43.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.086+0000 7f9a935a6640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9a8c09c090 con 0x7f9a8c0a1330 2024-09-16T04:12:43.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.086+0000 7f9a935a6640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9a8c09f610 con 0x7f9a8c0a1e20 2024-09-16T04:12:43.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.086+0000 7f9a935a6640 10 monclient(hunting): start opening mon connection 2024-09-16T04:12:43.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.086+0000 7f9a935a6640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f9a8c09f8f0 con 0x7f9a8c06dac0 2024-09-16T04:12:43.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.086+0000 7f9a935a6640 10 monclient(hunting): _renew_subs 2024-09-16T04:12:43.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.086+0000 7f9a935a6640 10 monclient(hunting): authenticate will time out at 2810.057265s 2024-09-16T04:12:43.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.086+0000 7f9a91342640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f9a8c0a1330 0x7f9a8c0a1710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:39026/0 (socket says 172.21.15.42:39026) 2024-09-16T04:12:43.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.086+0000 7f9a91342640 1 -- 172.21.15.42:0/4097177580 learned_addr learned my addr 172.21.15.42:0/4097177580 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T04:12:43.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.086+0000 7f9a90b41640 1 --2- 172.21.15.42:0/4097177580 >> v2:172.21.15.154:3300/0 conn(0x7f9a8c06dac0 0x7f9a8c13a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:12:43.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.086+0000 7f9a90b41640 10 monclient(hunting): get_auth_request con 0x7f9a8c06dac0 auth_method 0 2024-09-16T04:12:43.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.086+0000 7f9a90b41640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:12:43.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.086+0000 7f9a90b41640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:43.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.086+0000 7f9a90b41640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:43.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.086+0000 7f9a7ffff640 1 -- 172.21.15.42:0/4097177580 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2992773651 0 0) 0x7f9a8c09c090 con 0x7f9a8c0a1330 2024-09-16T04:12:43.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.086+0000 7f9a7ffff640 10 monclient(hunting): _negotiate 2024-09-16T04:12:43.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.086+0000 7f9a7ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:43.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.086+0000 7f9a7ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:43.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.086+0000 7f9a7ffff640 10 monclient(hunting): my global_id is 1034526 2024-09-16T04:12:43.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.086+0000 7f9a7ffff640 1 -- 172.21.15.42:0/4097177580 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9a78003530 con 0x7f9a8c0a1330 2024-09-16T04:12:43.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.086+0000 7f9a90b41640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:12:43.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.086+0000 7f9a90b41640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:12:43.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.086+0000 7f9a90b41640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:12:43.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.086+0000 7f9a7ffff640 1 -- 172.21.15.42:0/4097177580 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3809942027 0 0) 0x7f9a78003530 con 0x7f9a8c0a1330 2024-09-16T04:12:43.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.086+0000 7f9a7ffff640 1 -- 172.21.15.42:0/4097177580 >> v2:172.21.15.154:3300/0 conn(0x7f9a8c06dac0 msgr2=0x7f9a8c13a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:12:43.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.086+0000 7f9a7ffff640 1 --2- 172.21.15.42:0/4097177580 >> v2:172.21.15.154:3300/0 conn(0x7f9a8c06dac0 0x7f9a8c13a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:12:43.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.086+0000 7f9a7ffff640 1 -- 172.21.15.42:0/4097177580 >> v1:172.21.15.154:6789/0 conn(0x7f9a8c0a1e20 legacy=0x7f9a8c13c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:43.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.086+0000 7f9a7ffff640 10 monclient: _finish_hunting 0 2024-09-16T04:12:43.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.086+0000 7f9a7ffff640 1 monclient: found mon.noname-a-legacy 2024-09-16T04:12:43.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.086+0000 7f9a7ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:12:43.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.086+0000 7f9a7ffff640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.42:6789/0 2024-09-16T04:12:43.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.086+0000 7f9a7ffff640 1 -- 172.21.15.42:0/4097177580 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a8c09fbd0 con 0x7f9a8c0a1330 2024-09-16T04:12:43.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.086+0000 7f9a7ffff640 10 monclient: _finish_auth 0 2024-09-16T04:12:43.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.086+0000 7f9a7ffff640 10 monclient: _check_auth_tickets 2024-09-16T04:12:43.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.086+0000 7f9a7ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:12:43.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.086+0000 7f9a7ffff640 1 -- 172.21.15.42:0/4097177580 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2177413734 0 0) 0x7f9a88001680 con 0x7f9a8c0a1330 2024-09-16T04:12:43.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.086+0000 7f9a935a6640 5 monclient: authenticate success, global_id 1034526 2024-09-16T04:12:43.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.086+0000 7f9a935a6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:12:43.098 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.086+0000 7f9a7ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:43.098 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.086+0000 7f9a7ffff640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T04:12:43.098 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.086+0000 7f9a7ffff640 10 monclient: dump: 2024-09-16T04:12:43.098 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:12:43.098 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:12:43.098 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:43.098 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:43.098 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:43.098 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:12:43.098 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:12:43.098 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:12:43.098 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:12:43.098 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:12:43.098 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.086+0000 7f9a7ffff640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] but i'm connected to v1:172.21.15.42:6789/0, reconnecting 2024-09-16T04:12:43.099 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.086+0000 7f9a7ffff640 10 monclient: _reopen_session rank -1 2024-09-16T04:12:43.099 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.086+0000 7f9a7ffff640 1 -- 172.21.15.42:0/4097177580 >> v1:172.21.15.42:6789/0 conn(0x7f9a8c0a1330 legacy=0x7f9a8c0a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:43.099 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.086+0000 7f9a7ffff640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:12:43.099 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.086+0000 7f9a7ffff640 1 --2- 172.21.15.42:0/4097177580 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9a78003a10 0x7f9a78003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:12:43.099 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.086+0000 7f9a7ffff640 10 monclient(hunting): picked mon.b con 0x7f9a78003a10 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:12:43.099 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.086+0000 7f9a7ffff640 1 --2- 172.21.15.42:0/4097177580 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9a78004810 0x7f9a78004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:12:43.099 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.086+0000 7f9a7ffff640 10 monclient(hunting): picked mon.a con 0x7f9a78004810 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:12:43.099 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.086+0000 7f9a7ffff640 1 --2- 172.21.15.42:0/4097177580 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9a78005220 0x7f9a7800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:12:43.100 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.086+0000 7f9a7ffff640 10 monclient(hunting): picked mon.c con 0x7f9a78005220 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:12:43.100 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.086+0000 7f9a7ffff640 10 monclient(hunting): start opening mon connection 2024-09-16T04:12:43.100 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.086+0000 7f9a7ffff640 1 -- 172.21.15.42:0/4097177580 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f9a8c09f8f0 con 0x7f9a78004810 2024-09-16T04:12:43.100 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.086+0000 7f9a7ffff640 10 monclient(hunting): start opening mon connection 2024-09-16T04:12:43.100 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.086+0000 7f9a7ffff640 1 -- 172.21.15.42:0/4097177580 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f9a7800c170 con 0x7f9a78003a10 2024-09-16T04:12:43.100 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.086+0000 7f9a7ffff640 10 monclient(hunting): start opening mon connection 2024-09-16T04:12:43.100 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.086+0000 7f9a7ffff640 1 -- 172.21.15.42:0/4097177580 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f9a7800c370 con 0x7f9a78005220 2024-09-16T04:12:43.100 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.086+0000 7f9a7ffff640 10 monclient(hunting): _renew_subs 2024-09-16T04:12:43.100 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.086+0000 7f9a7ffff640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:12:43.100 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.086+0000 7f9a935a6640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-16T04:12:43.100 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.086+0000 7f9a90b41640 1 --2- 172.21.15.42:0/4097177580 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9a78004810 0x7f9a78004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:12:43.100 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.086+0000 7f9a90b41640 10 monclient(hunting): get_auth_request con 0x7f9a78004810 auth_method 0 2024-09-16T04:12:43.101 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.086+0000 7f9a90b41640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:12:43.101 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.086+0000 7f9a90b41640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:43.101 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.086+0000 7f9a90b41640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:12:43.101 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.086+0000 7f9a91b43640 1 --2- 172.21.15.42:0/4097177580 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9a78005220 0x7f9a7800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:12:43.101 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.086+0000 7f9a91b43640 10 monclient(hunting): get_auth_request con 0x7f9a78005220 auth_method 0 2024-09-16T04:12:43.101 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.086+0000 7f9a91b43640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:12:43.101 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.086+0000 7f9a91b43640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:43.101 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.086+0000 7f9a91b43640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:12:43.101 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.086+0000 7f9a90b41640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:12:43.101 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.086+0000 7f9a90b41640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:12:43.101 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.086+0000 7f9a90b41640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:12:43.101 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.086+0000 7f9a91342640 1 --2- 172.21.15.42:0/4097177580 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9a78003a10 0x7f9a78003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:12:43.101 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.086+0000 7f9a91342640 10 monclient(hunting): get_auth_request con 0x7f9a78003a10 auth_method 0 2024-09-16T04:12:43.101 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.086+0000 7f9a91342640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:12:43.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.086+0000 7f9a91342640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:43.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.086+0000 7f9a91342640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:12:43.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.086+0000 7f9a91b43640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:12:43.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.086+0000 7f9a91b43640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:12:43.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.086+0000 7f9a91b43640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:12:43.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.086+0000 7f9a91342640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:12:43.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.086+0000 7f9a91342640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:12:43.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.086+0000 7f9a91342640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:12:43.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.086+0000 7f9a90b41640 10 monclient(hunting): handle_auth_done global_id 1034526 payload 293 2024-09-16T04:12:43.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.086+0000 7f9a90b41640 1 -- 172.21.15.42:0/4097177580 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9a78005220 msgr2=0x7f9a7800b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:12:43.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.086+0000 7f9a90b41640 1 --2- 172.21.15.42:0/4097177580 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9a78005220 0x7f9a7800b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:12:43.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.086+0000 7f9a90b41640 1 -- 172.21.15.42:0/4097177580 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9a78003a10 msgr2=0x7f9a78003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:12:43.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.086+0000 7f9a90b41640 1 --2- 172.21.15.42:0/4097177580 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9a78003a10 0x7f9a78003e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:12:43.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.086+0000 7f9a90b41640 10 monclient: _finish_hunting 0 2024-09-16T04:12:43.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.086+0000 7f9a90b41640 1 monclient: found mon.a 2024-09-16T04:12:43.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.086+0000 7f9a90b41640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:12:43.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.086+0000 7f9a90b41640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:12:43.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.086+0000 7f9a90b41640 1 -- 172.21.15.42:0/4097177580 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a78003530 con 0x7f9a78004810 2024-09-16T04:12:43.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.086+0000 7f9a90b41640 10 monclient: _finish_auth 0 2024-09-16T04:12:43.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.086+0000 7f9a90b41640 10 monclient: _check_auth_tickets 2024-09-16T04:12:43.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.086+0000 7f9a90b41640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:12:43.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.086+0000 7f9a91b43640 1 -- 172.21.15.42:0/4097177580 reap_dead start 2024-09-16T04:12:43.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.090+0000 7f9a90b41640 1 --2- 172.21.15.42:0/4097177580 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9a78004810 0x7f9a78004c90 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f9a80009a90 tx=0x7f9a8001ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d01a73ed30fb2f75 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:12:43.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.090+0000 7f9a7ffff640 1 -- 172.21.15.42:0/4097177580 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a80009690 con 0x7f9a78004810 2024-09-16T04:12:43.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.090+0000 7f9a7ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:43.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.090+0000 7f9a7ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:12:43.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.090+0000 7f9a7ffff640 10 monclient: dump: 2024-09-16T04:12:43.104 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:12:43.104 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:12:43.104 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:43.104 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:43.104 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:43.104 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:12:43.104 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:12:43.104 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:12:43.104 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:12:43.104 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:12:43.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.090+0000 7f9a7ffff640 1 -- 172.21.15.42:0/4097177580 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a800a8450 con 0x7f9a78004810 2024-09-16T04:12:43.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.090+0000 7f9a935a6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:12:43.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.090+0000 7f9a7ffff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:12:43.105 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.090+0000 7f9a935a6640 10 monclient: get_monmap_and_config success 2024-09-16T04:12:43.105 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.090+0000 7f9a935a6640 4 set_mon_vals no callback set 2024-09-16T04:12:43.105 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.090+0000 7f9a7ffff640 1 -- 172.21.15.42:0/4097177580 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a800a8820 con 0x7f9a78004810 2024-09-16T04:12:43.105 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.090+0000 7f9a7ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:43.105 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.090+0000 7f9a7ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:12:43.105 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.090+0000 7f9a7ffff640 10 monclient: dump: 2024-09-16T04:12:43.105 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:12:43.105 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:12:43.105 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:43.105 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:43.105 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:43.106 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:12:43.106 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:12:43.106 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:12:43.106 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:12:43.106 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:12:43.111 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.090+0000 7f9a935a6640 10 monclient: shutdown 2024-09-16T04:12:43.111 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.090+0000 7f9a935a6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:12:43.111 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.090+0000 7f9a935a6640 1 -- 172.21.15.42:0/4097177580 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9a78004810 msgr2=0x7f9a78004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:43.111 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.090+0000 7f9a935a6640 1 --2- 172.21.15.42:0/4097177580 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9a78004810 0x7f9a78004c90 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f9a80009a90 tx=0x7f9a8001ca10 comp rx=0 tx=0).stop 2024-09-16T04:12:43.111 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.090+0000 7f9a935a6640 1 -- 172.21.15.42:0/4097177580 shutdown_connections 2024-09-16T04:12:43.111 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.090+0000 7f9a935a6640 1 --2- 172.21.15.42:0/4097177580 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9a78004810 0x7f9a78004c90 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:12:43.111 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.090+0000 7f9a935a6640 1 -- 172.21.15.42:0/4097177580 >> 172.21.15.42:0/4097177580 conn(0x7f9a8c0a2420 msgr2=0x7f9a8c0a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:12:43.112 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.090+0000 7f9a935a6640 1 -- 172.21.15.42:0/4097177580 shutdown_connections 2024-09-16T04:12:43.112 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.090+0000 7f9a935a6640 1 -- 172.21.15.42:0/4097177580 wait complete. 2024-09-16T04:12:43.112 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.090+0000 7f9a935a6640 10 monclient: build_initial_monmap 2024-09-16T04:12:43.112 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.090+0000 7f9a935a6640 10 monclient: monmap: 2024-09-16T04:12:43.112 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:12:43.112 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:12:43.112 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:12:43.112 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:12:43.112 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:12:43.112 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:12:43.112 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:12:43.112 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:12:43.112 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:12:43.113 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:12:43.113 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.090+0000 7f9a935a6640 1 librados: starting msgr at 2024-09-16T04:12:43.113 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.090+0000 7f9a935a6640 1 librados: starting objecter 2024-09-16T04:12:43.113 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.090+0000 7f9a935a6640 1 Processor -- start 2024-09-16T04:12:43.113 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.090+0000 7f9a935a6640 1 -- start start 2024-09-16T04:12:43.113 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.090+0000 7f9a935a6640 1 librados: setting wanted keys 2024-09-16T04:12:43.113 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.090+0000 7f9a935a6640 1 librados: calling monclient init 2024-09-16T04:12:43.113 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.090+0000 7f9a935a6640 10 monclient: init 2024-09-16T04:12:43.114 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.090+0000 7f9a935a6640 10 monclient: _reopen_session rank -1 2024-09-16T04:12:43.114 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.090+0000 7f9a935a6640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:12:43.114 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.090+0000 7f9a935a6640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9a78004810 0x7f9a8c1cd7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:12:43.114 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.090+0000 7f9a935a6640 10 monclient(hunting): picked mon.noname-a con 0x7f9a78004810 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:12:43.114 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.090+0000 7f9a935a6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9a8c1cdd30 0x7f9a8c122c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:12:43.114 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.090+0000 7f9a935a6640 10 monclient(hunting): picked mon.noname-b con 0x7f9a8c1cdd30 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:12:43.114 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.090+0000 7f9a935a6640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9a8c123190 0x7f9a8c123610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:12:43.114 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.090+0000 7f9a935a6640 10 monclient(hunting): picked mon.noname-c con 0x7f9a8c123190 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:12:43.114 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.090+0000 7f9a935a6640 10 monclient(hunting): start opening mon connection 2024-09-16T04:12:43.114 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.090+0000 7f9a935a6640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f9a8c135910 con 0x7f9a78004810 2024-09-16T04:12:43.114 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.090+0000 7f9a935a6640 10 monclient(hunting): start opening mon connection 2024-09-16T04:12:43.114 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.090+0000 7f9a935a6640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f9a8c09ef80 con 0x7f9a8c1cdd30 2024-09-16T04:12:43.114 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.090+0000 7f9a935a6640 10 monclient(hunting): start opening mon connection 2024-09-16T04:12:43.115 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.090+0000 7f9a935a6640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f9a8c135a90 con 0x7f9a8c123190 2024-09-16T04:12:43.115 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.090+0000 7f9a935a6640 10 monclient(hunting): _renew_subs 2024-09-16T04:12:43.115 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.090+0000 7f9a935a6640 10 monclient(hunting): authenticate will time out at 2810.063396s 2024-09-16T04:12:43.115 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.090+0000 7f9a91342640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9a78004810 0x7f9a8c1cd7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:12:43.115 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.090+0000 7f9a91342640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9a78004810 0x7f9a8c1cd7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:39444/0 (socket says 172.21.15.42:39444) 2024-09-16T04:12:43.115 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.090+0000 7f9a91b43640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9a8c123190 0x7f9a8c123610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:12:43.115 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.090+0000 7f9a91342640 1 -- 172.21.15.42:0/2966370596 learned_addr learned my addr 172.21.15.42:0/2966370596 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:12:43.115 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.090+0000 7f9a91342640 10 monclient(hunting): get_auth_request con 0x7f9a78004810 auth_method 0 2024-09-16T04:12:43.115 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.090+0000 7f9a91342640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:12:43.115 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.090+0000 7f9a91342640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:43.115 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.090+0000 7f9a91342640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:43.115 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.090+0000 7f9a91b43640 10 monclient(hunting): get_auth_request con 0x7f9a8c123190 auth_method 0 2024-09-16T04:12:43.115 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.090+0000 7f9a91b43640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:12:43.115 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.090+0000 7f9a91b43640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:43.116 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.090+0000 7f9a91b43640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:43.116 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.090+0000 7f9a91342640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:12:43.116 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.090+0000 7f9a91342640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:12:43.116 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.090+0000 7f9a91342640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:12:43.116 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.090+0000 7f9a91b43640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:12:43.116 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.090+0000 7f9a91b43640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:12:43.116 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.090+0000 7f9a91b43640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:12:43.116 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.090+0000 7f9a90b41640 1 --2- 172.21.15.42:0/2966370596 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9a8c1cdd30 0x7f9a8c122c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:12:43.116 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.090+0000 7f9a90b41640 10 monclient(hunting): get_auth_request con 0x7f9a8c1cdd30 auth_method 0 2024-09-16T04:12:43.116 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.090+0000 7f9a90b41640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:12:43.116 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.090+0000 7f9a90b41640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:43.116 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.090+0000 7f9a90b41640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:43.116 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.090+0000 7f9a91342640 10 monclient(hunting): handle_auth_done global_id 1034529 payload 824 2024-09-16T04:12:43.117 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.090+0000 7f9a91342640 1 -- 172.21.15.42:0/2966370596 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9a8c123190 msgr2=0x7f9a8c123610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:43.117 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.090+0000 7f9a91342640 1 --2- 172.21.15.42:0/2966370596 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9a8c123190 0x7f9a8c123610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:12:43.117 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.090+0000 7f9a91342640 1 -- 172.21.15.42:0/2966370596 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9a8c1cdd30 msgr2=0x7f9a8c122c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:43.117 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.090+0000 7f9a91342640 1 --2- 172.21.15.42:0/2966370596 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9a8c1cdd30 0x7f9a8c122c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:12:43.117 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.090+0000 7f9a91342640 10 monclient: _finish_hunting 0 2024-09-16T04:12:43.117 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.090+0000 7f9a91342640 1 monclient: found mon.noname-a 2024-09-16T04:12:43.117 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.090+0000 7f9a91342640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:12:43.117 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.090+0000 7f9a91342640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:12:43.117 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.090+0000 7f9a91342640 1 -- 172.21.15.42:0/2966370596 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a8c126860 con 0x7f9a78004810 2024-09-16T04:12:43.117 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.094+0000 7f9a91342640 1 --2- 172.21.15.42:0/2966370596 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9a78004810 0x7f9a8c1cd7f0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f9a880072f0 tx=0x7f9a880077b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:12:43.118 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.094+0000 7f9a7dffb640 10 client.?.objecter ms_handle_connect 0x7f9a78004810 2024-09-16T04:12:43.118 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.094+0000 7f9a7dffb640 10 client.?.objecter resend_mon_ops 2024-09-16T04:12:43.118 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.094+0000 7f9a7dffb640 1 -- 172.21.15.42:0/2966370596 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a8801a070 con 0x7f9a78004810 2024-09-16T04:12:43.118 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.094+0000 7f9a7dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:43.118 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.094+0000 7f9a7dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:12:43.118 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.094+0000 7f9a7dffb640 10 monclient: dump: 2024-09-16T04:12:43.118 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:12:43.118 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:12:43.118 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:43.118 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:43.118 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:43.119 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:12:43.119 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:12:43.119 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:12:43.119 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:12:43.119 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:12:43.119 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.094+0000 7f9a7dffb640 10 monclient: _finish_auth 0 2024-09-16T04:12:43.119 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.094+0000 7f9a7dffb640 10 monclient: _check_auth_tickets 2024-09-16T04:12:43.119 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.094+0000 7f9a7dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:12:43.119 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.094+0000 7f9a7dffb640 1 -- 172.21.15.42:0/2966370596 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a8800f9c0 con 0x7f9a78004810 2024-09-16T04:12:43.119 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.094+0000 7f9a935a6640 5 monclient: authenticate success, global_id 1034529 2024-09-16T04:12:43.119 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.094+0000 7f9a935a6640 10 monclient: _renew_subs 2024-09-16T04:12:43.119 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.094+0000 7f9a935a6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:12:43.119 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.094+0000 7f9a935a6640 1 -- 172.21.15.42:0/2966370596 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a8c124cf0 con 0x7f9a78004810 2024-09-16T04:12:43.119 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.094+0000 7f9a7dffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:12:43.119 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.094+0000 7f9a7dffb640 1 -- 172.21.15.42:0/2966370596 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a8800fd90 con 0x7f9a78004810 2024-09-16T04:12:43.120 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.094+0000 7f9a7dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:43.120 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.094+0000 7f9a7dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:12:43.120 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.094+0000 7f9a7dffb640 10 monclient: dump: 2024-09-16T04:12:43.120 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:12:43.120 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:12:43.120 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:43.120 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:43.120 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:43.120 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:12:43.120 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:12:43.120 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:12:43.120 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:12:43.120 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:12:43.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.094+0000 7f9a7f7fe640 4 set_mon_vals no callback set 2024-09-16T04:12:43.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.094+0000 7f9a935a6640 10 client.1034529.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:12:43.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.094+0000 7f9a935a6640 10 monclient: _renew_subs 2024-09-16T04:12:43.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.094+0000 7f9a935a6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:12:43.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.094+0000 7f9a935a6640 1 -- 172.21.15.42:0/2966370596 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a8c071240 con 0x7f9a78004810 2024-09-16T04:12:43.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.094+0000 7f9a935a6640 1 librados: init done 2024-09-16T04:12:43.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.094+0000 7f9a935a6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:12:43.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.094+0000 7f9a935a6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:12:43.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.094+0000 7f9a935a6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:12:43.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.094+0000 7f9a935a6640 1 -- 172.21.15.42:0/2966370596 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a8c135e20 con 0x7f9a78004810 2024-09-16T04:12:43.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.094+0000 7f9a7dffb640 1 -- 172.21.15.42:0/2966370596 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f9a8801d070 con 0x7f9a78004810 2024-09-16T04:12:43.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.094+0000 7f9a7dffb640 1 --2- 172.21.15.42:0/2966370596 >> v2:172.21.15.154:6812/254845342 conn(0x7f9a70037810 0x7f9a70039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:12:43.121 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.094+0000 7f9a7dffb640 1 -- 172.21.15.42:0/2966370596 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(706..706 src has 681..706) ==== 5040+0+0 (secure 0 0 0) 0x7f9a880485b0 con 0x7f9a78004810 2024-09-16T04:12:43.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.094+0000 7f9a7dffb640 10 client.1034529.objecter ms_dispatch 0x7f9a8c000ba0 osd_map(706..706 src has 681..706) 2024-09-16T04:12:43.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.094+0000 7f9a7dffb640 3 client.1034529.objecter handle_osd_map got epochs [706,706] > 0 2024-09-16T04:12:43.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.094+0000 7f9a7dffb640 3 client.1034529.objecter handle_osd_map decoding full epoch 706 2024-09-16T04:12:43.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.094+0000 7f9a7dffb640 20 client.1034529.objecter dump_active .. 0 homeless 2024-09-16T04:12:43.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.094+0000 7f9a90b41640 1 --2- 172.21.15.42:0/2966370596 >> v2:172.21.15.154:6812/254845342 conn(0x7f9a70037810 0x7f9a70039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:12:43.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.094+0000 7f9a90b41640 10 monclient: get_auth_request con 0x7f9a70037810 auth_method 0 2024-09-16T04:12:43.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.094+0000 7f9a7dffb640 10 client.1034529.objecter ms_handle_connect 0x7f9a70037810 2024-09-16T04:12:43.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.094+0000 7f9a90b41640 1 --2- 172.21.15.42:0/2966370596 >> v2:172.21.15.154:6812/254845342 conn(0x7f9a70037810 0x7f9a70039cd0 secure :-1 s=READY pgs=3970 cs=0 l=1 rev1=1 crypto rx=0x7f9a800af000 tx=0x7f9a800046d0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:12:43.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.098+0000 7f9a7dffb640 1 -- 172.21.15.42:0/2966370596 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f9a880133b0 con 0x7f9a78004810 2024-09-16T04:12:43.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.098+0000 7f9a7dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:12:43.122 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.098+0000 7f9a7dffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:12:43.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.322+0000 7f9a935a6640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T04:12:43.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.322+0000 7f9a935a6640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T04:12:43.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.322+0000 7f9a935a6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:12:43.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.322+0000 7f9a935a6640 1 -- 172.21.15.42:0/2966370596 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f9a8c071cd0 con 0x7f9a78004810 2024-09-16T04:12:43.329 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.326+0000 7f9a7dffb640 1 -- 172.21.15.42:0/2966370596 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f9a88012da0 con 0x7f9a78004810 2024-09-16T04:12:43.329 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.326+0000 7f9a7dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T04:12:43.329 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.326+0000 7f9a7dffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:12:43.330 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.326+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.330 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.326+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=24,config=2,health=11,kv=10,logm=53} crc {auth=1297191310,config=692619927,health=4124707665,kv=3446188498,logm=3886933498}) 2024-09-16T04:12:43.330 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.326+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=24,config=2,health=11,kv=10,logm=53} crc {auth=862366300,config=692619927,health=2189798410,kv=3446188498,logm=120895834}) 2024-09-16T04:12:43.330 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.326+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.330 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.326+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=24,config=2,health=11,kv=10,logm=53} crc {auth=1297191310,config=692619927,health=4124707665,kv=3446188498,logm=3886933498}) 2024-09-16T04:12:43.330 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.326+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=27,config=2,health=10,kv=10,logm=51} crc {auth=2023711222,config=692619927,health=4114171652,kv=3446188498,logm=62625780}) 2024-09-16T04:12:43.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.326+0000 7f9a935a6640 10 librados: watch_flush enter 2024-09-16T04:12:43.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.326+0000 7f9a935a6640 10 librados: watch_flush exit 2024-09-16T04:12:43.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.326+0000 7f9a935a6640 20 client.1034529.objecter shutdown clearing up homeless session... 2024-09-16T04:12:43.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.326+0000 7f9a935a6640 10 client.1034529.objecter successfully canceled tick 2024-09-16T04:12:43.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.326+0000 7f9a935a6640 1 -- 172.21.15.42:0/2966370596 >> v2:172.21.15.154:6812/254845342 conn(0x7f9a70037810 msgr2=0x7f9a70039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:43.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.326+0000 7f9a935a6640 1 --2- 172.21.15.42:0/2966370596 >> v2:172.21.15.154:6812/254845342 conn(0x7f9a70037810 0x7f9a70039cd0 secure :-1 s=READY pgs=3970 cs=0 l=1 rev1=1 crypto rx=0x7f9a800af000 tx=0x7f9a800046d0 comp rx=0 tx=0).stop 2024-09-16T04:12:43.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.326+0000 7f9a935a6640 10 monclient: shutdown 2024-09-16T04:12:43.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.326+0000 7f9a935a6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:12:43.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.326+0000 7f9a935a6640 1 -- 172.21.15.42:0/2966370596 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9a78004810 msgr2=0x7f9a8c1cd7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:43.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.326+0000 7f9a935a6640 1 --2- 172.21.15.42:0/2966370596 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9a78004810 0x7f9a8c1cd7f0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f9a880072f0 tx=0x7f9a880077b0 comp rx=0 tx=0).stop 2024-09-16T04:12:43.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.326+0000 7f9a935a6640 1 -- 172.21.15.42:0/2966370596 shutdown_connections 2024-09-16T04:12:43.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.326+0000 7f9a935a6640 1 --2- 172.21.15.42:0/2966370596 >> v2:172.21.15.154:6812/254845342 conn(0x7f9a70037810 0x7f9a70039cd0 unknown :-1 s=CLOSED pgs=3970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:12:43.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.326+0000 7f9a935a6640 1 --2- 172.21.15.42:0/2966370596 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9a8c123190 0x7f9a8c123610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:12:43.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.326+0000 7f9a935a6640 1 --2- 172.21.15.42:0/2966370596 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9a8c1cdd30 0x7f9a8c122c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:12:43.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.326+0000 7f9a935a6640 1 --2- 172.21.15.42:0/2966370596 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9a78004810 0x7f9a8c1cd7f0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:12:43.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.326+0000 7f9a935a6640 1 -- 172.21.15.42:0/2966370596 >> 172.21.15.42:0/2966370596 conn(0x7f9a8c0a2420 msgr2=0x7f9a8c06f8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:12:43.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.326+0000 7f9a935a6640 1 -- 172.21.15.42:0/2966370596 shutdown_connections 2024-09-16T04:12:43.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.326+0000 7f9a935a6640 1 -- 172.21.15.42:0/2966370596 wait complete. 2024-09-16T04:12:43.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:43.326+0000 7f9a935a6640 1 librados: shutdown 2024-09-16T04:12:43.334 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.326+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.334 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.326+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4217683738}) 2024-09-16T04:12:43.334 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.330+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1708314235}) 2024-09-16T04:12:43.334 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.330+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.334 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.330+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4217683738}) 2024-09-16T04:12:43.334 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.330+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=338178402}) 2024-09-16T04:12:43.336 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.330+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.336 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.330+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2656961329}) 2024-09-16T04:12:43.336 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.330+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=408377299}) 2024-09-16T04:12:43.336 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.330+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.336 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.330+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2656961329}) 2024-09-16T04:12:43.336 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.330+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3187510638}) 2024-09-16T04:12:43.338 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.334+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.339 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.334+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3708734157}) 2024-09-16T04:12:43.339 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.334+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=529709096}) 2024-09-16T04:12:43.339 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.334+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.339 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.334+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3708734157}) 2024-09-16T04:12:43.339 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.334+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2850662157}) 2024-09-16T04:12:43.342 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.338+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.342 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.338+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=251569596}) 2024-09-16T04:12:43.342 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.338+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=818063939}) 2024-09-16T04:12:43.342 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.338+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.342 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.338+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=251569596}) 2024-09-16T04:12:43.342 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.338+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3269610590}) 2024-09-16T04:12:43.344 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.338+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.344 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.338+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3013333054}) 2024-09-16T04:12:43.345 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.338+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1011060746}) 2024-09-16T04:12:43.345 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.338+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.345 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.338+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3013333054}) 2024-09-16T04:12:43.345 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.338+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3057620097}) 2024-09-16T04:12:43.346 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.342+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.346 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.342+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1307883391}) 2024-09-16T04:12:43.346 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.342+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2373872586}) 2024-09-16T04:12:43.346 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.342+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.346 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.342+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1307883391}) 2024-09-16T04:12:43.346 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.342+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1825225039}) 2024-09-16T04:12:43.348 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.342+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.348 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.342+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=864752260}) 2024-09-16T04:12:43.348 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.342+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3656753323}) 2024-09-16T04:12:43.348 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.342+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.348 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.342+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=864752260}) 2024-09-16T04:12:43.348 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.342+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2482383299}) 2024-09-16T04:12:43.349 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.346+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.350 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.346+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1452341141}) 2024-09-16T04:12:43.350 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.346+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2905252920}) 2024-09-16T04:12:43.350 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.346+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.350 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.346+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1452341141}) 2024-09-16T04:12:43.350 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.346+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3035533432}) 2024-09-16T04:12:43.351 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.346+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.351 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.346+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=66290816}) 2024-09-16T04:12:43.351 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.346+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3958060484}) 2024-09-16T04:12:43.351 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.346+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.352 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.346+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=66290816}) 2024-09-16T04:12:43.352 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.346+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3873018274}) 2024-09-16T04:12:43.353 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.350+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.353 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.350+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2043066389}) 2024-09-16T04:12:43.353 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.350+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2148528583}) 2024-09-16T04:12:43.353 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.350+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.353 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.350+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2043066389}) 2024-09-16T04:12:43.353 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.350+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3795097457}) 2024-09-16T04:12:43.354 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T04:12:43.355 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.350+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.355 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.350+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1075425523}) 2024-09-16T04:12:43.355 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.350+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3982441841}) 2024-09-16T04:12:43.355 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.350+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.355 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.350+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1075425523}) 2024-09-16T04:12:43.355 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.350+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1907803206}) 2024-09-16T04:12:43.356 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.350+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.356 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.350+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3566362370}) 2024-09-16T04:12:43.357 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.350+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=605507284}) 2024-09-16T04:12:43.357 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.350+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.357 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.350+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3566362370}) 2024-09-16T04:12:43.357 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.350+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2450145887}) 2024-09-16T04:12:43.358 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.354+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.358 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.354+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1204375024}) 2024-09-16T04:12:43.358 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.354+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3314735563}) 2024-09-16T04:12:43.358 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.354+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.358 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.354+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1204375024}) 2024-09-16T04:12:43.358 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.354+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=386291965}) 2024-09-16T04:12:43.360 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.354+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.360 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.354+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2382719544}) 2024-09-16T04:12:43.360 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.354+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=757582210}) 2024-09-16T04:12:43.360 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.354+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.361 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.354+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2382719544}) 2024-09-16T04:12:43.361 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.354+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3888642433}) 2024-09-16T04:12:43.362 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.358+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.362 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.358+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=139396437}) 2024-09-16T04:12:43.363 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.358+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2859440075}) 2024-09-16T04:12:43.363 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.358+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.363 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.358+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=139396437}) 2024-09-16T04:12:43.363 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.358+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2803205734}) 2024-09-16T04:12:43.364 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.358+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.364 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.358+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3520091472}) 2024-09-16T04:12:43.365 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.358+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1256953175}) 2024-09-16T04:12:43.365 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.358+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.365 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.358+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3520091472}) 2024-09-16T04:12:43.365 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.358+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1332785577}) 2024-09-16T04:12:43.366 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.362+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.366 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.362+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1939132177}) 2024-09-16T04:12:43.366 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.362+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1281949358}) 2024-09-16T04:12:43.366 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.362+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.366 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.362+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1939132177}) 2024-09-16T04:12:43.366 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.362+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=26155533}) 2024-09-16T04:12:43.368 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.362+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.368 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.362+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4218207947}) 2024-09-16T04:12:43.368 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.362+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3080036112}) 2024-09-16T04:12:43.368 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.362+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.368 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.362+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4218207947}) 2024-09-16T04:12:43.368 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.362+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2274474817}) 2024-09-16T04:12:43.369 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.366+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.369 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.366+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3247498001}) 2024-09-16T04:12:43.370 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.366+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1410809703}) 2024-09-16T04:12:43.370 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.366+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.370 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.366+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3247498001}) 2024-09-16T04:12:43.370 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.366+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=776759540}) 2024-09-16T04:12:43.372 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.366+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.372 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.366+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=774622608}) 2024-09-16T04:12:43.372 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.366+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4166257975}) 2024-09-16T04:12:43.372 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.366+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.372 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.366+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=774622608}) 2024-09-16T04:12:43.372 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.366+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4085677662}) 2024-09-16T04:12:43.373 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.366+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.373 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.366+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=718153985}) 2024-09-16T04:12:43.373 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.366+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2633677469}) 2024-09-16T04:12:43.374 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.366+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.374 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.366+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=718153985}) 2024-09-16T04:12:43.374 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.366+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1789919419}) 2024-09-16T04:12:43.375 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.370+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.375 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.370+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=520991071}) 2024-09-16T04:12:43.375 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.370+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=680011978}) 2024-09-16T04:12:43.375 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.370+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.375 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.370+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=520991071}) 2024-09-16T04:12:43.375 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.370+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3367571506}) 2024-09-16T04:12:43.376 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.370+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.377 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.370+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=15191237}) 2024-09-16T04:12:43.377 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.370+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=616552649}) 2024-09-16T04:12:43.377 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.370+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.377 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.370+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=15191237}) 2024-09-16T04:12:43.377 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.370+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2545733381}) 2024-09-16T04:12:43.378 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.374+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.378 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.374+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2914034969}) 2024-09-16T04:12:43.379 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.374+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2766567365}) 2024-09-16T04:12:43.379 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.374+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.379 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.374+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2914034969}) 2024-09-16T04:12:43.379 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.374+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1267750676}) 2024-09-16T04:12:43.380 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.374+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.380 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.374+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3413363713}) 2024-09-16T04:12:43.380 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.374+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3817324519}) 2024-09-16T04:12:43.380 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.374+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.381 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.374+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3413363713}) 2024-09-16T04:12:43.381 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.374+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2119173154}) 2024-09-16T04:12:43.382 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.378+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.382 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.378+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1778513313}) 2024-09-16T04:12:43.382 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.378+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2915326597}) 2024-09-16T04:12:43.382 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.378+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.382 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.378+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1778513313}) 2024-09-16T04:12:43.383 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.378+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3301213449}) 2024-09-16T04:12:43.384 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.378+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.384 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.378+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3317369475}) 2024-09-16T04:12:43.384 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.378+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4258077638}) 2024-09-16T04:12:43.384 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.378+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.384 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.378+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3317369475}) 2024-09-16T04:12:43.384 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.378+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2307912621}) 2024-09-16T04:12:43.385 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.382+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.386 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.382+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4252657667}) 2024-09-16T04:12:43.386 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.382+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4039948942}) 2024-09-16T04:12:43.386 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.382+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.386 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.382+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4252657667}) 2024-09-16T04:12:43.386 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.382+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1811685352}) 2024-09-16T04:12:43.387 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.382+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.387 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.382+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=529215038}) 2024-09-16T04:12:43.388 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.382+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1774465692}) 2024-09-16T04:12:43.388 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.382+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.388 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.382+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=529215038}) 2024-09-16T04:12:43.388 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.382+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1771093440}) 2024-09-16T04:12:43.389 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.386+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.389 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.386+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=478201095}) 2024-09-16T04:12:43.389 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.386+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1552412633}) 2024-09-16T04:12:43.389 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.386+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.389 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.386+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=478201095}) 2024-09-16T04:12:43.389 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.386+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2124393879}) 2024-09-16T04:12:43.391 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.386+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.391 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.386+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3659133160}) 2024-09-16T04:12:43.391 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.386+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=89479391}) 2024-09-16T04:12:43.391 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.386+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.391 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.386+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3659133160}) 2024-09-16T04:12:43.391 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.386+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3566374991}) 2024-09-16T04:12:43.393 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.386+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.393 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.386+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3430169618}) 2024-09-16T04:12:43.393 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.386+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4147089210}) 2024-09-16T04:12:43.393 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.386+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.393 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.386+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3430169618}) 2024-09-16T04:12:43.393 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.386+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1469241933}) 2024-09-16T04:12:43.394 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.390+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.395 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.390+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3056041793}) 2024-09-16T04:12:43.395 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.390+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1297105688}) 2024-09-16T04:12:43.395 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.390+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.395 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.390+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3056041793}) 2024-09-16T04:12:43.395 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.390+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2237373066}) 2024-09-16T04:12:43.396 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.390+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.396 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.390+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2596961745}) 2024-09-16T04:12:43.396 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.390+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=823283908}) 2024-09-16T04:12:43.396 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.390+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.397 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.390+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2596961745}) 2024-09-16T04:12:43.397 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.390+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2595652339}) 2024-09-16T04:12:43.398 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.394+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.398 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.394+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4128075696}) 2024-09-16T04:12:43.398 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.394+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2582699927}) 2024-09-16T04:12:43.398 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.394+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.398 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.394+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4128075696}) 2024-09-16T04:12:43.398 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.394+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1681008376}) 2024-09-16T04:12:43.400 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.394+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.400 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.394+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2508917701}) 2024-09-16T04:12:43.400 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.394+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2851867330}) 2024-09-16T04:12:43.400 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.394+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.400 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.394+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2508917701}) 2024-09-16T04:12:43.400 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.394+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1556597528}) 2024-09-16T04:12:43.401 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.398+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.402 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.398+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1430021346}) 2024-09-16T04:12:43.402 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.398+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4221199861}) 2024-09-16T04:12:43.402 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.398+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.402 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.398+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1430021346}) 2024-09-16T04:12:43.402 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.398+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=900227207}) 2024-09-16T04:12:43.403 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.398+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.403 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.398+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1717632650}) 2024-09-16T04:12:43.403 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.398+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3910227460}) 2024-09-16T04:12:43.404 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.398+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.404 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.398+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1717632650}) 2024-09-16T04:12:43.404 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.398+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1274634639}) 2024-09-16T04:12:43.405 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.398+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.408 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.398+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2173233259}) 2024-09-16T04:12:43.408 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.398+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4172845455}) 2024-09-16T04:12:43.409 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.398+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.409 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.402+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2173233259}) 2024-09-16T04:12:43.409 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.402+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2243707474}) 2024-09-16T04:12:43.410 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.402+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.410 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.402+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1636300516}) 2024-09-16T04:12:43.410 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.402+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4175793742}) 2024-09-16T04:12:43.410 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.402+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.410 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.402+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1636300516}) 2024-09-16T04:12:43.410 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.402+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3697238186}) 2024-09-16T04:12:43.410 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.402+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.411 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.402+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=838859212}) 2024-09-16T04:12:43.411 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.402+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3940429145}) 2024-09-16T04:12:43.411 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.402+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.411 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.402+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=838859212}) 2024-09-16T04:12:43.411 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.402+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=655667028}) 2024-09-16T04:12:43.411 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.406+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.411 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.406+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=481484403}) 2024-09-16T04:12:43.411 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.406+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4018684275}) 2024-09-16T04:12:43.411 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.406+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.411 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.406+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=481484403}) 2024-09-16T04:12:43.411 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.406+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=288258317}) 2024-09-16T04:12:43.412 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.406+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.412 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.406+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2406199650}) 2024-09-16T04:12:43.412 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.406+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=829067140}) 2024-09-16T04:12:43.412 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.406+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.412 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.406+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2406199650}) 2024-09-16T04:12:43.412 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.406+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3231661228}) 2024-09-16T04:12:43.413 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.410+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.413 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.410+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2137954712}) 2024-09-16T04:12:43.413 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.410+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3536305302}) 2024-09-16T04:12:43.413 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.410+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.414 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.410+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2137954712}) 2024-09-16T04:12:43.414 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.410+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1304275575}) 2024-09-16T04:12:43.415 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.410+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.415 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.410+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3362298167}) 2024-09-16T04:12:43.415 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.410+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2662597467}) 2024-09-16T04:12:43.415 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.410+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.415 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.410+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3362298167}) 2024-09-16T04:12:43.416 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.410+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3447862642}) 2024-09-16T04:12:43.417 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.410+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.417 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.410+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1017471055}) 2024-09-16T04:12:43.417 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.410+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=102467684}) 2024-09-16T04:12:43.417 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.410+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.417 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.410+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1017471055}) 2024-09-16T04:12:43.417 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.410+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=991060748}) 2024-09-16T04:12:43.419 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.414+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.419 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.414+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1265577444}) 2024-09-16T04:12:43.419 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.414+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4265070852}) 2024-09-16T04:12:43.420 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.414+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.420 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.414+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1265577444}) 2024-09-16T04:12:43.420 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.414+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3532152882}) 2024-09-16T04:12:43.421 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.414+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.421 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.414+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1527566248}) 2024-09-16T04:12:43.422 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.414+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2528231236}) 2024-09-16T04:12:43.422 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.414+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.422 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.414+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1527566248}) 2024-09-16T04:12:43.422 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.414+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2356949441}) 2024-09-16T04:12:43.424 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.418+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.424 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.418+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2305292860}) 2024-09-16T04:12:43.424 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.418+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=246638009}) 2024-09-16T04:12:43.424 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.418+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.424 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.418+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2305292860}) 2024-09-16T04:12:43.424 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.418+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1785480504}) 2024-09-16T04:12:43.424 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.418+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.424 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.418+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2983514140}) 2024-09-16T04:12:43.424 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.418+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2532551097}) 2024-09-16T04:12:43.425 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.418+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.425 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.418+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2983514140}) 2024-09-16T04:12:43.425 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.418+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1072977392}) 2024-09-16T04:12:43.425 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.422+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.425 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.422+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1641212698}) 2024-09-16T04:12:43.425 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.422+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=635479148}) 2024-09-16T04:12:43.426 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.422+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.426 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.422+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1641212698}) 2024-09-16T04:12:43.426 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.422+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1063597033}) 2024-09-16T04:12:43.427 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.422+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.427 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.422+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1776689466}) 2024-09-16T04:12:43.427 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.422+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2686629220}) 2024-09-16T04:12:43.427 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.422+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.427 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.422+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1776689466}) 2024-09-16T04:12:43.427 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.422+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4125364038}) 2024-09-16T04:12:43.429 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.422+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.429 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.422+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=908426159}) 2024-09-16T04:12:43.429 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.422+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2323161075}) 2024-09-16T04:12:43.429 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.422+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.430 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.422+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=908426159}) 2024-09-16T04:12:43.430 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.422+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2211699627}) 2024-09-16T04:12:43.431 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.426+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.431 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.426+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=307978880}) 2024-09-16T04:12:43.431 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.426+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3734466039}) 2024-09-16T04:12:43.431 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.426+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.431 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.426+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=307978880}) 2024-09-16T04:12:43.431 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.426+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=955254361}) 2024-09-16T04:12:43.432 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.426+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.432 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.426+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1033538587}) 2024-09-16T04:12:43.432 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.426+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=652108916}) 2024-09-16T04:12:43.432 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.426+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.432 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.426+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1033538587}) 2024-09-16T04:12:43.432 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.426+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3146453564}) 2024-09-16T04:12:43.434 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.430+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.434 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.430+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=323532366}) 2024-09-16T04:12:43.434 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.430+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3733807904}) 2024-09-16T04:12:43.434 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.430+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.434 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.430+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=323532366}) 2024-09-16T04:12:43.434 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.430+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2473475063}) 2024-09-16T04:12:43.435 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.430+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.435 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.430+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2811853518}) 2024-09-16T04:12:43.435 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.430+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1603175091}) 2024-09-16T04:12:43.436 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.430+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.436 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.430+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2811853518}) 2024-09-16T04:12:43.436 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.430+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1585213932}) 2024-09-16T04:12:43.437 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.434+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.437 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.434+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=916560461}) 2024-09-16T04:12:43.437 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.434+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2481778490}) 2024-09-16T04:12:43.437 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.434+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.437 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.434+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=916560461}) 2024-09-16T04:12:43.437 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.434+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=341350347}) 2024-09-16T04:12:43.439 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.434+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.439 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.434+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2367871926}) 2024-09-16T04:12:43.439 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.434+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2750914282}) 2024-09-16T04:12:43.439 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.434+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.439 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.434+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2367871926}) 2024-09-16T04:12:43.439 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.434+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1266867873}) 2024-09-16T04:12:43.441 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.434+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.441 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.434+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=413497870}) 2024-09-16T04:12:43.441 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.434+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1466490130}) 2024-09-16T04:12:43.441 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.434+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.441 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.434+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=413497870}) 2024-09-16T04:12:43.441 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.434+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2823524449}) 2024-09-16T04:12:43.442 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.438+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.442 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.438+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2253477429}) 2024-09-16T04:12:43.442 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.438+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3935166806}) 2024-09-16T04:12:43.443 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.438+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.443 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.438+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2253477429}) 2024-09-16T04:12:43.443 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.438+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2464241893}) 2024-09-16T04:12:43.444 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.438+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.444 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.438+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3933354320}) 2024-09-16T04:12:43.444 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.438+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1507938783}) 2024-09-16T04:12:43.444 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.438+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.444 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.438+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3933354320}) 2024-09-16T04:12:43.444 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.438+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3503519987}) 2024-09-16T04:12:43.446 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.442+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.446 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.442+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=970574295}) 2024-09-16T04:12:43.446 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.442+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4287759202}) 2024-09-16T04:12:43.446 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.442+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.446 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.442+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=970574295}) 2024-09-16T04:12:43.446 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.442+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1434691270}) 2024-09-16T04:12:43.447 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.442+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.448 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.442+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2325351039}) 2024-09-16T04:12:43.448 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.442+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1867817884}) 2024-09-16T04:12:43.448 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.442+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.448 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.442+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2325351039}) 2024-09-16T04:12:43.448 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.442+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1245318444}) 2024-09-16T04:12:43.449 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.446+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.449 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.446+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3003583670}) 2024-09-16T04:12:43.449 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.446+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1169722879}) 2024-09-16T04:12:43.449 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.446+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.449 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.446+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3003583670}) 2024-09-16T04:12:43.449 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.446+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2020665962}) 2024-09-16T04:12:43.451 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.446+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.451 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.446+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3955792787}) 2024-09-16T04:12:43.451 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.446+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=28300084}) 2024-09-16T04:12:43.451 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.446+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.451 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.446+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3955792787}) 2024-09-16T04:12:43.451 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.446+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=383076772}) 2024-09-16T04:12:43.452 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.446+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.453 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.446+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=824892860}) 2024-09-16T04:12:43.453 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.446+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3032512846}) 2024-09-16T04:12:43.453 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.446+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.453 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.446+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=824892860}) 2024-09-16T04:12:43.453 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.446+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3982849112}) 2024-09-16T04:12:43.454 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.450+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.455 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.450+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2718090244}) 2024-09-16T04:12:43.455 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.450+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=878855653}) 2024-09-16T04:12:43.455 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.450+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.455 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.450+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2718090244}) 2024-09-16T04:12:43.455 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.450+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1111971337}) 2024-09-16T04:12:43.456 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.450+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.456 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.450+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2596113071}) 2024-09-16T04:12:43.456 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.450+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1237625739}) 2024-09-16T04:12:43.457 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.450+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.457 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.450+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2596113071}) 2024-09-16T04:12:43.457 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.450+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4042670926}) 2024-09-16T04:12:43.458 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.454+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.458 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.454+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2052861021}) 2024-09-16T04:12:43.458 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.454+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=935014203}) 2024-09-16T04:12:43.458 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.454+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.458 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.454+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2052861021}) 2024-09-16T04:12:43.458 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.454+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4200035930}) 2024-09-16T04:12:43.460 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.454+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.460 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.454+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2527927681}) 2024-09-16T04:12:43.460 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.454+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=202708611}) 2024-09-16T04:12:43.460 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.454+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.460 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.454+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2527927681}) 2024-09-16T04:12:43.460 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.454+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=525157076}) 2024-09-16T04:12:43.461 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.458+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.462 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.458+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1404167808}) 2024-09-16T04:12:43.462 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.458+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1625664521}) 2024-09-16T04:12:43.462 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.458+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.462 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.458+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1404167808}) 2024-09-16T04:12:43.462 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.458+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2219439436}) 2024-09-16T04:12:43.463 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.458+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.463 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.458+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2665544370}) 2024-09-16T04:12:43.463 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.458+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2744988301}) 2024-09-16T04:12:43.464 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.458+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.464 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.458+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2665544370}) 2024-09-16T04:12:43.464 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.458+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3073946709}) 2024-09-16T04:12:43.465 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.458+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.465 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.458+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4116397510}) 2024-09-16T04:12:43.465 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.458+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2717470239}) 2024-09-16T04:12:43.465 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.458+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.465 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.458+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4116397510}) 2024-09-16T04:12:43.465 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.458+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3229236267}) 2024-09-16T04:12:43.466 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.462+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.467 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.462+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2674938981}) 2024-09-16T04:12:43.467 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.462+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1273821195}) 2024-09-16T04:12:43.467 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.462+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.467 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.462+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2674938981}) 2024-09-16T04:12:43.467 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.462+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=477729821}) 2024-09-16T04:12:43.469 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.462+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.469 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.462+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=663025118}) 2024-09-16T04:12:43.469 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.462+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=489521097}) 2024-09-16T04:12:43.469 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.462+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.469 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.462+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=663025118}) 2024-09-16T04:12:43.469 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.462+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4180257966}) 2024-09-16T04:12:43.470 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.466+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.470 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.466+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4011111522}) 2024-09-16T04:12:43.470 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.466+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3276625992}) 2024-09-16T04:12:43.470 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.466+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.470 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.466+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4011111522}) 2024-09-16T04:12:43.470 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.466+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3804552927}) 2024-09-16T04:12:43.472 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.466+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.472 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.466+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1701703888}) 2024-09-16T04:12:43.472 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.466+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=431466273}) 2024-09-16T04:12:43.472 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.466+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.473 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.466+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1701703888}) 2024-09-16T04:12:43.473 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.466+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1608828379}) 2024-09-16T04:12:43.474 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.470+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.475 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.470+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1526347121}) 2024-09-16T04:12:43.475 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.470+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=696541599}) 2024-09-16T04:12:43.475 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.470+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.475 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.470+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1526347121}) 2024-09-16T04:12:43.475 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.470+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2329219444}) 2024-09-16T04:12:43.476 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.470+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.476 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.470+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=382029136}) 2024-09-16T04:12:43.476 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.470+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3322068023}) 2024-09-16T04:12:43.477 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.470+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.477 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.470+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=382029136}) 2024-09-16T04:12:43.477 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.470+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2355310351}) 2024-09-16T04:12:43.478 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.474+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.478 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.474+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=681088954}) 2024-09-16T04:12:43.478 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.474+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3890161716}) 2024-09-16T04:12:43.478 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.474+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.478 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.474+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=681088954}) 2024-09-16T04:12:43.478 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.474+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3556319347}) 2024-09-16T04:12:43.480 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.474+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.480 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.474+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1181865444}) 2024-09-16T04:12:43.480 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.474+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1573503040}) 2024-09-16T04:12:43.480 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.474+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.481 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.474+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1181865444}) 2024-09-16T04:12:43.481 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.474+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2497718191}) 2024-09-16T04:12:43.482 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.478+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.482 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.478+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3327155507}) 2024-09-16T04:12:43.482 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.478+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3123574200}) 2024-09-16T04:12:43.482 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.478+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.482 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.478+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3327155507}) 2024-09-16T04:12:43.482 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.478+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=522275687}) 2024-09-16T04:12:43.484 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.478+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.484 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.478+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1594199268}) 2024-09-16T04:12:43.484 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.478+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1966010721}) 2024-09-16T04:12:43.484 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.478+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.485 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.478+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1594199268}) 2024-09-16T04:12:43.485 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.478+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2717122261}) 2024-09-16T04:12:43.486 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.482+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.486 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.482+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1218126364}) 2024-09-16T04:12:43.486 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.482+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2880610422}) 2024-09-16T04:12:43.486 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.482+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.487 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.482+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1218126364}) 2024-09-16T04:12:43.487 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.482+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2351121446}) 2024-09-16T04:12:43.488 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.482+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.488 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.482+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=419815839}) 2024-09-16T04:12:43.488 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.482+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=479022009}) 2024-09-16T04:12:43.488 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.482+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.488 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.482+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=419815839}) 2024-09-16T04:12:43.488 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.482+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1305306572}) 2024-09-16T04:12:43.490 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.486+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.490 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.486+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=80445576}) 2024-09-16T04:12:43.490 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.486+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2372007744}) 2024-09-16T04:12:43.490 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.486+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.490 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.486+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=80445576}) 2024-09-16T04:12:43.490 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.486+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4123350900}) 2024-09-16T04:12:43.492 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.486+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.492 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.486+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2396504993}) 2024-09-16T04:12:43.492 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.486+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=320128487}) 2024-09-16T04:12:43.492 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.486+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.492 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.486+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2396504993}) 2024-09-16T04:12:43.492 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.486+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3753264555}) 2024-09-16T04:12:43.494 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.490+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.494 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.490+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2611191425}) 2024-09-16T04:12:43.494 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.490+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1982237576}) 2024-09-16T04:12:43.494 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.490+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.494 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.490+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2611191425}) 2024-09-16T04:12:43.494 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.490+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=70929374}) 2024-09-16T04:12:43.496 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.490+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.496 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.490+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4216476937}) 2024-09-16T04:12:43.496 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.490+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3603100089}) 2024-09-16T04:12:43.496 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.490+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.496 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.490+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4216476937}) 2024-09-16T04:12:43.496 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.490+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=96171248}) 2024-09-16T04:12:43.497 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.494+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.497 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.494+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3104978569}) 2024-09-16T04:12:43.498 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.494+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=133685530}) 2024-09-16T04:12:43.498 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.494+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.499 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.494+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3104978569}) 2024-09-16T04:12:43.499 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.494+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=642495807}) 2024-09-16T04:12:43.500 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.494+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.500 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.494+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1710047935}) 2024-09-16T04:12:43.500 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.494+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3499529183}) 2024-09-16T04:12:43.500 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.494+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.500 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.494+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1710047935}) 2024-09-16T04:12:43.500 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.494+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1435601740}) 2024-09-16T04:12:43.501 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.498+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.501 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.498+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2008747821}) 2024-09-16T04:12:43.501 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.498+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2915621780}) 2024-09-16T04:12:43.501 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.498+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.501 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.498+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2008747821}) 2024-09-16T04:12:43.502 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.498+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4237492856}) 2024-09-16T04:12:43.503 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.498+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.503 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.498+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2027811947}) 2024-09-16T04:12:43.503 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.498+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1655786429}) 2024-09-16T04:12:43.504 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.498+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.504 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.498+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2027811947}) 2024-09-16T04:12:43.504 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.498+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1437226744}) 2024-09-16T04:12:43.505 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.498+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.505 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.498+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3738901031}) 2024-09-16T04:12:43.505 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.502+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4136561638}) 2024-09-16T04:12:43.505 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.502+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.505 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.502+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3738901031}) 2024-09-16T04:12:43.506 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.502+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3396488599}) 2024-09-16T04:12:43.507 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.502+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.507 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.502+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2569772132}) 2024-09-16T04:12:43.507 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.502+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=333551458}) 2024-09-16T04:12:43.507 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.502+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.507 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.502+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2569772132}) 2024-09-16T04:12:43.507 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.502+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3893093102}) 2024-09-16T04:12:43.509 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.502+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.509 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.502+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1931904548}) 2024-09-16T04:12:43.509 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.502+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=709509137}) 2024-09-16T04:12:43.509 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.502+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.509 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.502+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1931904548}) 2024-09-16T04:12:43.509 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.502+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2085670312}) 2024-09-16T04:12:43.510 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.506+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.511 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.506+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1600954873}) 2024-09-16T04:12:43.511 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.506+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2388662613}) 2024-09-16T04:12:43.511 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.506+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.511 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.506+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1600954873}) 2024-09-16T04:12:43.511 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.506+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3776551170}) 2024-09-16T04:12:43.513 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.506+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.513 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.506+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3692853791}) 2024-09-16T04:12:43.513 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.506+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2131934799}) 2024-09-16T04:12:43.513 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.506+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.513 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.506+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3692853791}) 2024-09-16T04:12:43.513 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.506+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2913231018}) 2024-09-16T04:12:43.514 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.510+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.514 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.510+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2760264315}) 2024-09-16T04:12:43.515 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.510+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2346650034}) 2024-09-16T04:12:43.515 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.510+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.515 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.510+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2760264315}) 2024-09-16T04:12:43.515 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.510+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=210969033}) 2024-09-16T04:12:43.517 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.510+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.518 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.510+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4253833106}) 2024-09-16T04:12:43.518 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.510+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=710883866}) 2024-09-16T04:12:43.518 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.510+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.518 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.510+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4253833106}) 2024-09-16T04:12:43.518 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.510+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=80961041}) 2024-09-16T04:12:43.519 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.514+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.519 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.514+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1174102672}) 2024-09-16T04:12:43.519 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.514+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3247577072}) 2024-09-16T04:12:43.519 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.514+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.519 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.514+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1174102672}) 2024-09-16T04:12:43.519 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.514+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1284063073}) 2024-09-16T04:12:43.521 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.514+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.521 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.514+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2145320371}) 2024-09-16T04:12:43.521 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.514+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=816330200}) 2024-09-16T04:12:43.521 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.514+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.521 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.514+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2145320371}) 2024-09-16T04:12:43.521 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.514+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1756936532}) 2024-09-16T04:12:43.523 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.518+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.523 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.518+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4151227910}) 2024-09-16T04:12:43.523 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.518+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2882199388}) 2024-09-16T04:12:43.523 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.518+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.524 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.518+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4151227910}) 2024-09-16T04:12:43.524 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.518+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4092874300}) 2024-09-16T04:12:43.525 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.518+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.525 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.518+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3364508090}) 2024-09-16T04:12:43.525 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.518+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1752878253}) 2024-09-16T04:12:43.525 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.518+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.525 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.518+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3364508090}) 2024-09-16T04:12:43.525 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.518+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3703821181}) 2024-09-16T04:12:43.527 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.522+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.527 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.522+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=506313275}) 2024-09-16T04:12:43.527 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.522+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2197015647}) 2024-09-16T04:12:43.527 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.522+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.527 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.522+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=506313275}) 2024-09-16T04:12:43.527 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.522+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3815809312}) 2024-09-16T04:12:43.529 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.522+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.529 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.522+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1284456733}) 2024-09-16T04:12:43.529 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.522+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1204139264}) 2024-09-16T04:12:43.529 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.522+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.529 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.522+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1284456733}) 2024-09-16T04:12:43.529 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.522+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=165572531}) 2024-09-16T04:12:43.531 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.526+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.531 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.526+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=600949636}) 2024-09-16T04:12:43.531 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.526+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4105783919}) 2024-09-16T04:12:43.531 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.526+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.531 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.526+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=600949636}) 2024-09-16T04:12:43.531 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.526+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3052814983}) 2024-09-16T04:12:43.533 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.526+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.533 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.526+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3249951032}) 2024-09-16T04:12:43.533 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.526+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3955304513}) 2024-09-16T04:12:43.533 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.526+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.533 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.526+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3249951032}) 2024-09-16T04:12:43.533 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.526+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3267533009}) 2024-09-16T04:12:43.535 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.530+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.535 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.530+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2896927029}) 2024-09-16T04:12:43.535 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.530+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=922656584}) 2024-09-16T04:12:43.535 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.530+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.535 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.530+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2896927029}) 2024-09-16T04:12:43.536 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.530+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1715370859}) 2024-09-16T04:12:43.537 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.530+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.537 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.530+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1461349927}) 2024-09-16T04:12:43.537 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.530+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1707923592}) 2024-09-16T04:12:43.537 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.530+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.538 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.530+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1461349927}) 2024-09-16T04:12:43.538 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.530+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1574091428}) 2024-09-16T04:12:43.539 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.534+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.539 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.534+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2449358590}) 2024-09-16T04:12:43.539 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.534+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1490815442}) 2024-09-16T04:12:43.539 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.534+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.539 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.534+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2449358590}) 2024-09-16T04:12:43.539 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.534+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3315625197}) 2024-09-16T04:12:43.541 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.534+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.541 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.534+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=826305047}) 2024-09-16T04:12:43.541 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.534+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=126772943}) 2024-09-16T04:12:43.541 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.534+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.541 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.534+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=826305047}) 2024-09-16T04:12:43.542 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.534+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4171807174}) 2024-09-16T04:12:43.543 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.538+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.543 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.538+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=912493040}) 2024-09-16T04:12:43.543 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.538+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3431602025}) 2024-09-16T04:12:43.544 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.538+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.544 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.538+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=912493040}) 2024-09-16T04:12:43.544 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.538+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2617523168}) 2024-09-16T04:12:43.545 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.538+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.545 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.538+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3769439357}) 2024-09-16T04:12:43.545 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.538+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=921295118}) 2024-09-16T04:12:43.545 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.538+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.545 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.538+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3769439357}) 2024-09-16T04:12:43.545 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.538+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1796857563}) 2024-09-16T04:12:43.546 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.542+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.547 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.542+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1188160024}) 2024-09-16T04:12:43.547 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.542+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2716212059}) 2024-09-16T04:12:43.547 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.542+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.547 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.542+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1188160024}) 2024-09-16T04:12:43.547 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.542+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1009634516}) 2024-09-16T04:12:43.549 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.542+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.549 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.542+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1491093968}) 2024-09-16T04:12:43.549 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.542+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4106278313}) 2024-09-16T04:12:43.549 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.542+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.549 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.542+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1491093968}) 2024-09-16T04:12:43.549 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.542+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1204168229}) 2024-09-16T04:12:43.551 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.546+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.551 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.546+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1057499189}) 2024-09-16T04:12:43.551 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.546+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4281121939}) 2024-09-16T04:12:43.551 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.546+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.551 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.546+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1057499189}) 2024-09-16T04:12:43.551 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.546+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1519828474}) 2024-09-16T04:12:43.553 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.550+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.553 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.550+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2178384286}) 2024-09-16T04:12:43.553 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.550+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=916139177}) 2024-09-16T04:12:43.553 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.550+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.553 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.550+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2178384286}) 2024-09-16T04:12:43.554 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.550+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4207326292}) 2024-09-16T04:12:43.555 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.550+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.555 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.550+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=226150850}) 2024-09-16T04:12:43.555 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.550+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3141895390}) 2024-09-16T04:12:43.556 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.550+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.556 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.550+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=226150850}) 2024-09-16T04:12:43.556 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.550+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1599526822}) 2024-09-16T04:12:43.557 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.554+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.558 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.554+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3192319885}) 2024-09-16T04:12:43.558 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.554+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1810218753}) 2024-09-16T04:12:43.558 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.554+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.558 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.554+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3192319885}) 2024-09-16T04:12:43.558 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.554+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=366153692}) 2024-09-16T04:12:43.560 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.554+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.560 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.554+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=70920994}) 2024-09-16T04:12:43.560 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.554+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3269511744}) 2024-09-16T04:12:43.560 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.554+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.560 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.554+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=70920994}) 2024-09-16T04:12:43.560 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.554+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3050973763}) 2024-09-16T04:12:43.562 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.558+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.562 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.558+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3329922513}) 2024-09-16T04:12:43.562 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.558+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1233865818}) 2024-09-16T04:12:43.562 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.558+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.562 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.558+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3329922513}) 2024-09-16T04:12:43.562 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.558+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2452121734}) 2024-09-16T04:12:43.564 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.558+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.564 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.558+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2329066595}) 2024-09-16T04:12:43.564 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.558+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4222847800}) 2024-09-16T04:12:43.564 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.558+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.564 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.558+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2329066595}) 2024-09-16T04:12:43.564 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.558+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3121995421}) 2024-09-16T04:12:43.566 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.562+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.566 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.562+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2579486566}) 2024-09-16T04:12:43.566 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.562+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1016598771}) 2024-09-16T04:12:43.567 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.562+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.567 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.562+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2579486566}) 2024-09-16T04:12:43.567 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.562+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=780297054}) 2024-09-16T04:12:43.568 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.562+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.568 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.562+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=809260327}) 2024-09-16T04:12:43.569 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.562+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2545482540}) 2024-09-16T04:12:43.569 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.562+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.569 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.562+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=809260327}) 2024-09-16T04:12:43.569 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.562+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3758334812}) 2024-09-16T04:12:43.571 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.566+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.571 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.566+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2045919015}) 2024-09-16T04:12:43.571 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.566+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=893705420}) 2024-09-16T04:12:43.571 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.566+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.571 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.566+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2045919015}) 2024-09-16T04:12:43.571 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.566+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1020590640}) 2024-09-16T04:12:43.574 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.566+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.574 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.566+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=276162150}) 2024-09-16T04:12:43.574 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.566+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=721349446}) 2024-09-16T04:12:43.574 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.566+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.574 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.566+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=276162150}) 2024-09-16T04:12:43.574 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.566+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=166102261}) 2024-09-16T04:12:43.575 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.570+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.575 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.570+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=497173466}) 2024-09-16T04:12:43.575 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.570+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=53475491}) 2024-09-16T04:12:43.575 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.570+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.575 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.570+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=497173466}) 2024-09-16T04:12:43.575 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.570+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4047234883}) 2024-09-16T04:12:43.577 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.570+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.577 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.570+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1913266496}) 2024-09-16T04:12:43.577 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.570+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2710660254}) 2024-09-16T04:12:43.577 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.570+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.577 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.570+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1913266496}) 2024-09-16T04:12:43.577 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.570+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1864764655}) 2024-09-16T04:12:43.579 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.574+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.579 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.574+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3230685086}) 2024-09-16T04:12:43.579 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.574+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2384141000}) 2024-09-16T04:12:43.579 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.574+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.579 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.574+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3230685086}) 2024-09-16T04:12:43.579 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.574+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2829201278}) 2024-09-16T04:12:43.581 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.578+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.581 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.578+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3902826729}) 2024-09-16T04:12:43.581 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.578+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3971286810}) 2024-09-16T04:12:43.581 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.578+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.582 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.578+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3902826729}) 2024-09-16T04:12:43.582 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.578+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1940967149}) 2024-09-16T04:12:43.584 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.578+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.584 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.578+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=83,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=4} crc {logm=1155419239,mdsmap=4088524775,mgr=3915430446,mgr_command_descs=2107673600,mgr_metadata=1367272202,mgrstat=1656977504}) 2024-09-16T04:12:43.584 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.578+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=81,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=6} crc {logm=4175345634,mdsmap=4088524775,mgr=460109777,mgr_command_descs=2107673600,mgr_metadata=1367272202,mgrstat=672438184}) 2024-09-16T04:12:43.584 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.578+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.584 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.578+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=83,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=4} crc {logm=1155419239,mdsmap=4088524775,mgr=3915430446,mgr_command_descs=2107673600,mgr_metadata=1367272202,mgrstat=1656977504}) 2024-09-16T04:12:43.584 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.578+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=83,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=4} crc {logm=1746592887,mdsmap=4088524775,mgr=460109776,mgr_command_descs=2107673600,mgr_metadata=1367272202,mgrstat=1656977504}) 2024-09-16T04:12:43.587 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.582+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.587 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.582+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgrstat=6,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=41,osdmap=32} crc {mgrstat=729942468,mon_config_key=85057112,monmap=3937513981,osd_metadata=4182959540,osd_pg_creating=783066558,osd_snap=3781450228,osdmap=1791221832}) 2024-09-16T04:12:43.587 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.582+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mgrstat=6,mon_config_key=10,monmap=2,osd_metadata=6,osd_pg_creating=1,osd_snap=40,osdmap=35} crc {mgrstat=729942468,mon_config_key=1530417952,monmap=3140514009,osd_metadata=4182959540,osd_pg_creating=783066558,osd_snap=947927903,osdmap=841184757}) 2024-09-16T04:12:43.587 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.582+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.587 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.582+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgrstat=6,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=41,osdmap=32} crc {mgrstat=729942468,mon_config_key=85057112,monmap=3937513981,osd_metadata=4182959540,osd_pg_creating=783066558,osd_snap=3781450228,osdmap=1791221832}) 2024-09-16T04:12:43.587 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.582+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mgrstat=6,mon_config_key=11,monmap=3,osd_metadata=4,osd_pg_creating=1,osd_snap=40,osdmap=35} crc {mgrstat=729942468,mon_config_key=85057112,monmap=3937513981,osd_metadata=3306448908,osd_pg_creating=783066558,osd_snap=1824431148,osdmap=863361503}) 2024-09-16T04:12:43.598 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.594+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.599 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.594+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=19} crc {osdmap=1297452719}) 2024-09-16T04:12:43.599 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.594+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=20} crc {osdmap=1781542705}) 2024-09-16T04:12:43.599 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.594+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:12:43.599 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.594+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=19} crc {osdmap=1297452719}) 2024-09-16T04:12:43.599 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:12:43.594+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=20} crc {osdmap=1781542705}) 2024-09-16T04:12:44.355 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T04:12:44.355 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:12:44.490 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.486+0000 7f2f1628b640 10 monclient: get_monmap_and_config 2024-09-16T04:12:44.490 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.486+0000 7f2f1628b640 10 monclient: build_initial_monmap 2024-09-16T04:12:44.490 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.486+0000 7f2f1628b640 10 monclient: monmap: 2024-09-16T04:12:44.491 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:12:44.491 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:12:44.491 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:12:44.489266+0000 2024-09-16T04:12:44.491 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:12:44.489266+0000 2024-09-16T04:12:44.491 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:12:44.491 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:12:44.491 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:12:44.491 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:12:44.491 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:12:44.491 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:12:44.491 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:12:44.491 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:12:44.494 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.486+0000 7f2f1628b640 1 Processor -- start 2024-09-16T04:12:44.494 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.486+0000 7f2f1628b640 1 -- start start 2024-09-16T04:12:44.494 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.486+0000 7f2f1628b640 10 monclient: init 2024-09-16T04:12:44.494 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.486+0000 7f2f1628b640 10 monclient: _reopen_session rank -1 2024-09-16T04:12:44.494 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.486+0000 7f2f1628b640 10 monclient: _add_conns ranks=[4,0,2,1,3] 2024-09-16T04:12:44.494 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.486+0000 7f2f1628b640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f2f1009fb00 0x7f2f1009ba50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:12:44.494 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.486+0000 7f2f1628b640 10 monclient(hunting): picked mon.noname-c con 0x7f2f1009fb00 addr v2:172.21.15.154:3300/0 2024-09-16T04:12:44.494 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.486+0000 7f2f1628b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2f1009bf90 addr v1:172.21.15.42:6789/0 2024-09-16T04:12:44.494 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.486+0000 7f2f1628b640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2f1009c400 0x7f2f1013b840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:12:44.495 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.486+0000 7f2f1628b640 10 monclient(hunting): picked mon.noname-b con 0x7f2f1009c400 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:12:44.495 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.486+0000 7f2f1628b640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2f1009c870 con 0x7f2f1009bf90 2024-09-16T04:12:44.495 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.486+0000 7f2f1628b640 10 monclient(hunting): start opening mon connection 2024-09-16T04:12:44.495 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.486+0000 7f2f1628b640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f2f1009e250 con 0x7f2f1009fb00 2024-09-16T04:12:44.495 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.486+0000 7f2f1628b640 10 monclient(hunting): start opening mon connection 2024-09-16T04:12:44.495 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.486+0000 7f2f1628b640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f2f100a1e20 con 0x7f2f1009c400 2024-09-16T04:12:44.495 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.486+0000 7f2f1628b640 10 monclient(hunting): _renew_subs 2024-09-16T04:12:44.495 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.486+0000 7f2f1628b640 10 monclient(hunting): authenticate will time out at 2811.458847s 2024-09-16T04:12:44.495 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.486+0000 7f2f14828640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2f1009c400 0x7f2f1013b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:12:44.495 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.486+0000 7f2f14828640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2f1009c400 0x7f2f1013b840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:36740/0 (socket says 172.21.15.42:36740) 2024-09-16T04:12:44.495 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.486+0000 7f2f14828640 1 -- 172.21.15.42:0/2691246798 learned_addr learned my addr 172.21.15.42:0/2691246798 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:12:44.495 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.486+0000 7f2f14828640 10 monclient(hunting): get_auth_request con 0x7f2f1009c400 auth_method 0 2024-09-16T04:12:44.495 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.486+0000 7f2f14828640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:12:44.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.486+0000 7f2f14828640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:44.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.486+0000 7f2f14828640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:44.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.486+0000 7f2f14828640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:12:44.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.486+0000 7f2f14828640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:12:44.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.486+0000 7f2f14828640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:12:44.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.486+0000 7f2f0ffff640 1 --2- 172.21.15.42:0/2691246798 >> v2:172.21.15.154:3300/0 conn(0x7f2f1009fb00 0x7f2f1009ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:12:44.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.486+0000 7f2f0ffff640 10 monclient(hunting): get_auth_request con 0x7f2f1009fb00 auth_method 0 2024-09-16T04:12:44.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.486+0000 7f2f0ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:12:44.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.486+0000 7f2f0ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:44.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.486+0000 7f2f0ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:44.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.486+0000 7f2f14828640 10 monclient(hunting): handle_auth_done global_id 1084124 payload 274 2024-09-16T04:12:44.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.486+0000 7f2f14828640 1 -- 172.21.15.42:0/2691246798 >> v2:172.21.15.154:3300/0 conn(0x7f2f1009fb00 msgr2=0x7f2f1009ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:12:44.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.486+0000 7f2f14828640 1 --2- 172.21.15.42:0/2691246798 >> v2:172.21.15.154:3300/0 conn(0x7f2f1009fb00 0x7f2f1009ba50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:12:44.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.486+0000 7f2f14828640 1 -- 172.21.15.42:0/2691246798 >> v1:172.21.15.42:6789/0 conn(0x7f2f1009bf90 legacy=0x7f2f10138110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:44.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.486+0000 7f2f14828640 10 monclient: _finish_hunting 0 2024-09-16T04:12:44.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.486+0000 7f2f14828640 1 monclient: found mon.noname-b 2024-09-16T04:12:44.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.486+0000 7f2f0effd640 1 -- 172.21.15.42:0/2691246798 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2417154485 0 0) 0x7f2f1009c870 con 0x7f2f1009bf90 2024-09-16T04:12:44.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.486+0000 7f2f14828640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:12:44.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.486+0000 7f2f14828640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T04:12:44.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.486+0000 7f2f14828640 1 -- 172.21.15.42:0/2691246798 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f1009ea30 con 0x7f2f1009c400 2024-09-16T04:12:44.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.486+0000 7f2f0ffff640 1 --2- 172.21.15.42:0/2691246798 >> v2:172.21.15.154:3300/0 conn(0x7f2f1009fb00 0x7f2f1009ba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T04:12:44.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.486+0000 7f2f0effd640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T04:12:44.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.490+0000 7f2f14828640 1 --2- 172.21.15.42:0/2691246798 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2f1009c400 0x7f2f1013b840 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f2f0800bb10 tx=0x7f2f0801eaf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bf629b0747d8782c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:12:44.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.490+0000 7f2f0effd640 1 -- 172.21.15.42:0/2691246798 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f080b4070 con 0x7f2f1009c400 2024-09-16T04:12:44.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.490+0000 7f2f0effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:44.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.490+0000 7f2f0effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:12:44.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.490+0000 7f2f0effd640 10 monclient: dump: 2024-09-16T04:12:44.503 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:12:44.503 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:12:44.503 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:44.503 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:44.503 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:44.503 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:12:44.503 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:12:44.504 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:12:44.504 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:12:44.504 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:12:44.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.490+0000 7f2f0effd640 10 monclient: _finish_auth 0 2024-09-16T04:12:44.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.490+0000 7f2f0effd640 10 monclient: _check_auth_tickets 2024-09-16T04:12:44.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.490+0000 7f2f0effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:12:44.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.490+0000 7f2f1628b640 5 monclient: authenticate success, global_id 1084124 2024-09-16T04:12:44.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.490+0000 7f2f0effd640 1 -- 172.21.15.42:0/2691246798 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2f080a7b40 con 0x7f2f1009c400 2024-09-16T04:12:44.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.490+0000 7f2f0effd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:12:44.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.490+0000 7f2f1628b640 10 monclient: get_monmap_and_config success 2024-09-16T04:12:44.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.490+0000 7f2f1628b640 4 set_mon_vals no callback set 2024-09-16T04:12:44.505 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.490+0000 7f2f0effd640 1 -- 172.21.15.42:0/2691246798 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f080af440 con 0x7f2f1009c400 2024-09-16T04:12:44.505 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.490+0000 7f2f0effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:44.505 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.490+0000 7f2f0effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:12:44.505 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.490+0000 7f2f0effd640 10 monclient: dump: 2024-09-16T04:12:44.505 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:12:44.505 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:12:44.505 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:44.505 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:44.505 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:44.505 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:12:44.505 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:12:44.505 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:12:44.506 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:12:44.506 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:12:44.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.490+0000 7f2f1628b640 10 monclient: shutdown 2024-09-16T04:12:44.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.490+0000 7f2f1628b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:12:44.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.490+0000 7f2f1628b640 1 -- 172.21.15.42:0/2691246798 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2f1009c400 msgr2=0x7f2f1013b840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:44.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.490+0000 7f2f1628b640 1 --2- 172.21.15.42:0/2691246798 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2f1009c400 0x7f2f1013b840 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f2f0800bb10 tx=0x7f2f0801eaf0 comp rx=0 tx=0).stop 2024-09-16T04:12:44.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.490+0000 7f2f1628b640 1 -- 172.21.15.42:0/2691246798 shutdown_connections 2024-09-16T04:12:44.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.490+0000 7f2f1628b640 1 --2- 172.21.15.42:0/2691246798 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2f1009c400 0x7f2f1013b840 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:12:44.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.490+0000 7f2f1628b640 1 --2- 172.21.15.42:0/2691246798 >> v2:172.21.15.154:3300/0 conn(0x7f2f1009fb00 0x7f2f1009ba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:12:44.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.490+0000 7f2f1628b640 1 -- 172.21.15.42:0/2691246798 >> 172.21.15.42:0/2691246798 conn(0x7f2f100a2420 msgr2=0x7f2f100a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:12:44.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.490+0000 7f2f1628b640 1 -- 172.21.15.42:0/2691246798 shutdown_connections 2024-09-16T04:12:44.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.490+0000 7f2f1628b640 1 -- 172.21.15.42:0/2691246798 wait complete. 2024-09-16T04:12:44.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.490+0000 7f2f1628b640 10 monclient: build_initial_monmap 2024-09-16T04:12:44.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.490+0000 7f2f1628b640 10 monclient: monmap: 2024-09-16T04:12:44.507 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:12:44.507 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:12:44.507 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:12:44.507 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:12:44.507 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:12:44.507 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:12:44.507 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:12:44.507 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:12:44.507 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:12:44.507 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:12:44.508 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.490+0000 7f2f1628b640 1 librados: starting msgr at 2024-09-16T04:12:44.508 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.490+0000 7f2f1628b640 1 librados: starting objecter 2024-09-16T04:12:44.508 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.490+0000 7f2f1628b640 1 Processor -- start 2024-09-16T04:12:44.508 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.490+0000 7f2f1628b640 1 -- start start 2024-09-16T04:12:44.508 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.490+0000 7f2f1628b640 1 librados: setting wanted keys 2024-09-16T04:12:44.508 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.490+0000 7f2f1628b640 1 librados: calling monclient init 2024-09-16T04:12:44.508 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.490+0000 7f2f1628b640 10 monclient: init 2024-09-16T04:12:44.508 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.494+0000 7f2f1628b640 10 monclient: _reopen_session rank -1 2024-09-16T04:12:44.508 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.494+0000 7f2f1628b640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:12:44.508 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.494+0000 7f2f1628b640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2f1009bf90 0x7f2f10133180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:12:44.508 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.494+0000 7f2f1628b640 10 monclient(hunting): picked mon.noname-a con 0x7f2f1009bf90 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:12:44.508 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.494+0000 7f2f1628b640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2f1009c400 0x7f2f101336c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:12:44.508 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.494+0000 7f2f1628b640 10 monclient(hunting): picked mon.noname-b con 0x7f2f1009c400 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:12:44.509 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.494+0000 7f2f1628b640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2f1009fb00 0x7f2f101390e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:12:44.509 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.494+0000 7f2f1628b640 10 monclient(hunting): picked mon.noname-c con 0x7f2f1009fb00 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:12:44.509 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.494+0000 7f2f1628b640 10 monclient(hunting): start opening mon connection 2024-09-16T04:12:44.509 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.494+0000 7f2f1628b640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f2f100a32f0 con 0x7f2f1009bf90 2024-09-16T04:12:44.509 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.494+0000 7f2f1628b640 10 monclient(hunting): start opening mon connection 2024-09-16T04:12:44.509 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.494+0000 7f2f1628b640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f2f100a3170 con 0x7f2f1009c400 2024-09-16T04:12:44.509 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.494+0000 7f2f1628b640 10 monclient(hunting): start opening mon connection 2024-09-16T04:12:44.509 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.494+0000 7f2f1628b640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f2f1009d380 con 0x7f2f1009fb00 2024-09-16T04:12:44.509 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.494+0000 7f2f1628b640 10 monclient(hunting): _renew_subs 2024-09-16T04:12:44.509 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.494+0000 7f2f1628b640 10 monclient(hunting): authenticate will time out at 2811.464099s 2024-09-16T04:12:44.509 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.494+0000 7f2f14828640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2f1009fb00 0x7f2f101390e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:12:44.509 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.494+0000 7f2f0ffff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2f1009bf90 0x7f2f10133180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:12:44.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.494+0000 7f2f14828640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2f1009fb00 0x7f2f101390e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:36744/0 (socket says 172.21.15.42:36744) 2024-09-16T04:12:44.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.494+0000 7f2f14828640 1 -- 172.21.15.42:0/4089596164 learned_addr learned my addr 172.21.15.42:0/4089596164 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:12:44.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.494+0000 7f2f14828640 10 monclient(hunting): get_auth_request con 0x7f2f1009fb00 auth_method 0 2024-09-16T04:12:44.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.494+0000 7f2f14828640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:12:44.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.494+0000 7f2f14828640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:44.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.494+0000 7f2f14828640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:44.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.494+0000 7f2f0ffff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2f1009bf90 0x7f2f10133180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:46728/0 (socket says 172.21.15.42:46728) 2024-09-16T04:12:44.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.494+0000 7f2f0ffff640 10 monclient(hunting): get_auth_request con 0x7f2f1009bf90 auth_method 0 2024-09-16T04:12:44.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.494+0000 7f2f0ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:12:44.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.494+0000 7f2f0ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:44.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.494+0000 7f2f0ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:44.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.494+0000 7f2f14828640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:12:44.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.494+0000 7f2f14828640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:12:44.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.494+0000 7f2f14828640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:12:44.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.494+0000 7f2f0ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:12:44.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.494+0000 7f2f0ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:12:44.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.494+0000 7f2f0ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:12:44.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.494+0000 7f2f14828640 10 monclient(hunting): handle_auth_done global_id 1084127 payload 824 2024-09-16T04:12:44.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.494+0000 7f2f0f7fe640 1 --2- 172.21.15.42:0/4089596164 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2f1009c400 0x7f2f101336c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:12:44.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.494+0000 7f2f14828640 1 -- 172.21.15.42:0/4089596164 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2f1009c400 msgr2=0x7f2f101336c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:44.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.494+0000 7f2f14828640 1 --2- 172.21.15.42:0/4089596164 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2f1009c400 0x7f2f101336c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:12:44.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.494+0000 7f2f14828640 1 -- 172.21.15.42:0/4089596164 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2f1009bf90 msgr2=0x7f2f10133180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:44.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.494+0000 7f2f14828640 1 --2- 172.21.15.42:0/4089596164 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2f1009bf90 0x7f2f10133180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:12:44.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.494+0000 7f2f14828640 10 monclient: _finish_hunting 0 2024-09-16T04:12:44.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.494+0000 7f2f14828640 1 monclient: found mon.noname-c 2024-09-16T04:12:44.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.494+0000 7f2f14828640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:12:44.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.494+0000 7f2f14828640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T04:12:44.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.494+0000 7f2f14828640 1 -- 172.21.15.42:0/4089596164 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f1006dd00 con 0x7f2f1009fb00 2024-09-16T04:12:44.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.494+0000 7f2f0f7fe640 10 monclient: get_auth_request con 0x7f2f1009c400 auth_method 0 2024-09-16T04:12:44.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.494+0000 7f2f0ffff640 1 --2- 172.21.15.42:0/4089596164 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2f1009bf90 0x7f2f10133180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:12:44.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.494+0000 7f2f0f7fe640 1 --2- 172.21.15.42:0/4089596164 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2f1009c400 0x7f2f101336c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T04:12:44.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.494+0000 7f2f14828640 1 --2- 172.21.15.42:0/4089596164 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2f1009fb00 0x7f2f101390e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f2f0801c720 tx=0x7f2f0801b890 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:12:44.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.494+0000 7f2f0cff9640 10 client.?.objecter ms_handle_connect 0x7f2f1009fb00 2024-09-16T04:12:44.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.494+0000 7f2f0cff9640 10 client.?.objecter resend_mon_ops 2024-09-16T04:12:44.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.494+0000 7f2f0cff9640 1 -- 172.21.15.42:0/4089596164 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f080b4030 con 0x7f2f1009fb00 2024-09-16T04:12:44.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.494+0000 7f2f0cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:44.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.494+0000 7f2f0cff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:12:44.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.494+0000 7f2f0cff9640 10 monclient: dump: 2024-09-16T04:12:44.512 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:12:44.512 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:12:44.513 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:44.513 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:44.513 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:44.513 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:12:44.513 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:12:44.513 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:12:44.513 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:12:44.513 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:12:44.513 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.494+0000 7f2f0cff9640 10 monclient: _finish_auth 0 2024-09-16T04:12:44.513 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.494+0000 7f2f0cff9640 10 monclient: _check_auth_tickets 2024-09-16T04:12:44.513 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.494+0000 7f2f0cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:12:44.513 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.494+0000 7f2f0cff9640 1 -- 172.21.15.42:0/4089596164 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2f080b6070 con 0x7f2f1009fb00 2024-09-16T04:12:44.513 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.494+0000 7f2f1628b640 5 monclient: authenticate success, global_id 1084127 2024-09-16T04:12:44.513 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.494+0000 7f2f0cff9640 10 monclient: handle_config config(0 keys) 2024-09-16T04:12:44.514 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.494+0000 7f2f0cff9640 1 -- 172.21.15.42:0/4089596164 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f080ad550 con 0x7f2f1009fb00 2024-09-16T04:12:44.514 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.494+0000 7f2f1628b640 10 monclient: _renew_subs 2024-09-16T04:12:44.514 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.494+0000 7f2f1628b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:12:44.514 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.494+0000 7f2f1628b640 1 -- 172.21.15.42:0/4089596164 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f10121870 con 0x7f2f1009fb00 2024-09-16T04:12:44.514 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.494+0000 7f2f0cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:44.514 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.494+0000 7f2f0cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:12:44.514 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.494+0000 7f2f0e7fc640 4 set_mon_vals no callback set 2024-09-16T04:12:44.514 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.494+0000 7f2f0cff9640 10 monclient: dump: 2024-09-16T04:12:44.514 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:12:44.514 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:12:44.514 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:44.514 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:44.514 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:44.514 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:12:44.515 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:12:44.515 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:12:44.515 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:12:44.515 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:12:44.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.494+0000 7f2f1628b640 10 client.1084127.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:12:44.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.494+0000 7f2f1628b640 10 monclient: _renew_subs 2024-09-16T04:12:44.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.494+0000 7f2f1628b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:12:44.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.494+0000 7f2f1628b640 1 -- 172.21.15.42:0/4089596164 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f10073a30 con 0x7f2f1009fb00 2024-09-16T04:12:44.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.494+0000 7f2f1628b640 1 librados: init done 2024-09-16T04:12:44.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.494+0000 7f2f0cff9640 1 -- 172.21.15.42:0/4089596164 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f2f080c1060 con 0x7f2f1009fb00 2024-09-16T04:12:44.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.494+0000 7f2f1628b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:12:44.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.494+0000 7f2f1628b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:12:44.516 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.494+0000 7f2f1628b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:12:44.516 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.494+0000 7f2f1628b640 1 -- 172.21.15.42:0/4089596164 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2f1009ba50 con 0x7f2f1009fb00 2024-09-16T04:12:44.516 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.494+0000 7f2f0cff9640 1 --2- 172.21.15.42:0/4089596164 >> v2:172.21.15.154:6812/254845342 conn(0x7f2efc037810 0x7f2efc039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:12:44.516 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.494+0000 7f2f0cff9640 1 -- 172.21.15.42:0/4089596164 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(706..706 src has 681..706) ==== 5040+0+0 (secure 0 0 0) 0x7f2f080e3080 con 0x7f2f1009fb00 2024-09-16T04:12:44.516 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.494+0000 7f2f0cff9640 10 client.1084127.objecter ms_dispatch 0x7f2f10000ba0 osd_map(706..706 src has 681..706) 2024-09-16T04:12:44.516 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.494+0000 7f2f0cff9640 3 client.1084127.objecter handle_osd_map got epochs [706,706] > 0 2024-09-16T04:12:44.516 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.494+0000 7f2f0cff9640 3 client.1084127.objecter handle_osd_map decoding full epoch 706 2024-09-16T04:12:44.516 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.494+0000 7f2f0cff9640 20 client.1084127.objecter dump_active .. 0 homeless 2024-09-16T04:12:44.516 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.494+0000 7f2f0ffff640 1 --2- 172.21.15.42:0/4089596164 >> v2:172.21.15.154:6812/254845342 conn(0x7f2efc037810 0x7f2efc039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:12:44.516 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.494+0000 7f2f0ffff640 10 monclient: get_auth_request con 0x7f2efc037810 auth_method 0 2024-09-16T04:12:44.516 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.494+0000 7f2f0ffff640 1 --2- 172.21.15.42:0/4089596164 >> v2:172.21.15.154:6812/254845342 conn(0x7f2efc037810 0x7f2efc039cd0 secure :-1 s=READY pgs=3971 cs=0 l=1 rev1=1 crypto rx=0x7f2f00003f10 tx=0x7f2f00009890 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:12:44.516 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.494+0000 7f2f0cff9640 10 client.1084127.objecter ms_handle_connect 0x7f2efc037810 2024-09-16T04:12:44.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.498+0000 7f2f0cff9640 1 -- 172.21.15.42:0/4089596164 <== mon.2 v2:172.21.15.42:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f2f1006dd00 con 0x7f2f1009fb00 2024-09-16T04:12:44.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.498+0000 7f2f0cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:12:44.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.498+0000 7f2f0cff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:12:44.737 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.734+0000 7f2f1628b640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:12:44.738 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.734+0000 7f2f1628b640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:12:44.738 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.734+0000 7f2f1628b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:12:44.738 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.734+0000 7f2f1628b640 1 -- 172.21.15.42:0/4089596164 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f2f1013ad80 con 0x7f2f1009fb00 2024-09-16T04:12:44.738 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.734+0000 7f2f0cff9640 1 -- 172.21.15.42:0/4089596164 <== mon.2 v2:172.21.15.42:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7f2f08018800 con 0x7f2f1009fb00 2024-09-16T04:12:44.738 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.734+0000 7f2f0cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:12:44.738 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.734+0000 7f2f0cff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:12:44.739 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":276,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:12:44.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.734+0000 7f2f1628b640 10 librados: watch_flush enter 2024-09-16T04:12:44.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.734+0000 7f2f1628b640 10 librados: watch_flush exit 2024-09-16T04:12:44.742 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.734+0000 7f2f1628b640 20 client.1084127.objecter shutdown clearing up homeless session... 2024-09-16T04:12:44.742 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.734+0000 7f2f1628b640 10 client.1084127.objecter successfully canceled tick 2024-09-16T04:12:44.742 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.734+0000 7f2f1628b640 1 -- 172.21.15.42:0/4089596164 >> v2:172.21.15.154:6812/254845342 conn(0x7f2efc037810 msgr2=0x7f2efc039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:44.742 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.734+0000 7f2f1628b640 1 --2- 172.21.15.42:0/4089596164 >> v2:172.21.15.154:6812/254845342 conn(0x7f2efc037810 0x7f2efc039cd0 secure :-1 s=READY pgs=3971 cs=0 l=1 rev1=1 crypto rx=0x7f2f00003f10 tx=0x7f2f00009890 comp rx=0 tx=0).stop 2024-09-16T04:12:44.742 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.734+0000 7f2f1628b640 10 monclient: shutdown 2024-09-16T04:12:44.742 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.734+0000 7f2f1628b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:12:44.742 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.734+0000 7f2f1628b640 1 -- 172.21.15.42:0/4089596164 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2f1009fb00 msgr2=0x7f2f101390e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:44.742 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.734+0000 7f2f1628b640 1 --2- 172.21.15.42:0/4089596164 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2f1009fb00 0x7f2f101390e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f2f0801c720 tx=0x7f2f0801b890 comp rx=0 tx=0).stop 2024-09-16T04:12:44.742 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.734+0000 7f2f1628b640 1 -- 172.21.15.42:0/4089596164 shutdown_connections 2024-09-16T04:12:44.742 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.734+0000 7f2f1628b640 1 --2- 172.21.15.42:0/4089596164 >> v2:172.21.15.154:6812/254845342 conn(0x7f2efc037810 0x7f2efc039cd0 unknown :-1 s=CLOSED pgs=3971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:12:44.742 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.734+0000 7f2f1628b640 1 --2- 172.21.15.42:0/4089596164 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2f1009fb00 0x7f2f101390e0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:12:44.742 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.734+0000 7f2f1628b640 1 --2- 172.21.15.42:0/4089596164 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2f1009c400 0x7f2f101336c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:12:44.742 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.734+0000 7f2f1628b640 1 --2- 172.21.15.42:0/4089596164 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2f1009bf90 0x7f2f10133180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:12:44.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.734+0000 7f2f1628b640 1 -- 172.21.15.42:0/4089596164 >> 172.21.15.42:0/4089596164 conn(0x7f2f100a2420 msgr2=0x7f2f100a46b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:12:44.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.738+0000 7f2f1628b640 1 -- 172.21.15.42:0/4089596164 shutdown_connections 2024-09-16T04:12:44.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.738+0000 7f2f1628b640 1 -- 172.21.15.42:0/4089596164 wait complete. 2024-09-16T04:12:44.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.738+0000 7f2f1628b640 1 librados: shutdown 2024-09-16T04:12:44.758 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T04:12:44.758 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T04:12:44.758 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T04:12:44.759 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T04:12:44.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.874+0000 7fefe0f3a640 10 monclient: get_monmap_and_config 2024-09-16T04:12:44.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.874+0000 7fefe0f3a640 10 monclient: build_initial_monmap 2024-09-16T04:12:44.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.874+0000 7fefe0f3a640 10 monclient: monmap: 2024-09-16T04:12:44.880 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:12:44.880 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:12:44.880 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:12:44.878761+0000 2024-09-16T04:12:44.880 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:12:44.878761+0000 2024-09-16T04:12:44.880 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:12:44.881 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:12:44.881 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:12:44.881 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:12:44.881 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:12:44.881 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:12:44.881 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:12:44.881 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:12:44.883 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.874+0000 7fefe0f3a640 1 Processor -- start 2024-09-16T04:12:44.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.874+0000 7fefe0f3a640 1 -- start start 2024-09-16T04:12:44.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.874+0000 7fefe0f3a640 10 monclient: init 2024-09-16T04:12:44.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.878+0000 7fefe0f3a640 10 monclient: _reopen_session rank -1 2024-09-16T04:12:44.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.878+0000 7fefe0f3a640 10 monclient: _add_conns ranks=[1,2,4,3,0] 2024-09-16T04:12:44.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.878+0000 7fefe0f3a640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fefdc0a1330 addr v1:172.21.15.154:6789/0 2024-09-16T04:12:44.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.878+0000 7fefe0f3a640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fefdc06dac0 0x7fefdc13a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:12:44.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.878+0000 7fefe0f3a640 10 monclient(hunting): picked mon.noname-b con 0x7fefdc06dac0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:12:44.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.878+0000 7fefe0f3a640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fefdc0a1e20 0x7fefdc13c890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:12:44.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.878+0000 7fefe0f3a640 10 monclient(hunting): picked mon.noname-c con 0x7fefdc0a1e20 addr v2:172.21.15.154:3300/0 2024-09-16T04:12:44.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.878+0000 7fefe0f3a640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fefdc09c7d0 con 0x7fefdc0a1330 2024-09-16T04:12:44.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.878+0000 7fefe0f3a640 10 monclient(hunting): start opening mon connection 2024-09-16T04:12:44.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.878+0000 7fefe0f3a640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fefdc09c090 con 0x7fefdc0a1e20 2024-09-16T04:12:44.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.878+0000 7fefe0f3a640 10 monclient(hunting): start opening mon connection 2024-09-16T04:12:44.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.878+0000 7fefe0f3a640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fefdc09f610 con 0x7fefdc06dac0 2024-09-16T04:12:44.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.878+0000 7fefe0f3a640 10 monclient(hunting): _renew_subs 2024-09-16T04:12:44.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.878+0000 7fefe0f3a640 10 monclient(hunting): authenticate will time out at 2811.848247s 2024-09-16T04:12:44.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.878+0000 7fefd9d9b640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fefdc06dac0 0x7fefdc13a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:12:44.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.878+0000 7fefd9d9b640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fefdc06dac0 0x7fefdc13a340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:36746/0 (socket says 172.21.15.42:36746) 2024-09-16T04:12:44.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.878+0000 7fefd9d9b640 1 -- 172.21.15.42:0/1029717906 learned_addr learned my addr 172.21.15.42:0/1029717906 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:12:44.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.878+0000 7fefd9d9b640 10 monclient(hunting): get_auth_request con 0x7fefdc06dac0 auth_method 0 2024-09-16T04:12:44.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.878+0000 7fefd9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:12:44.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.878+0000 7fefd9d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:44.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.878+0000 7fefd9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:44.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.878+0000 7fefd9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:12:44.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.878+0000 7fefd9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:12:44.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.878+0000 7fefd9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:12:44.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.878+0000 7fefdad9d640 1 --2- 172.21.15.42:0/1029717906 >> v2:172.21.15.154:3300/0 conn(0x7fefdc0a1e20 0x7fefdc13c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:12:44.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.878+0000 7fefdad9d640 10 monclient(hunting): get_auth_request con 0x7fefdc0a1e20 auth_method 0 2024-09-16T04:12:44.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.878+0000 7fefdad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:12:44.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.878+0000 7fefdad9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:44.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.878+0000 7fefdad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:44.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.878+0000 7fefd9d9b640 10 monclient(hunting): handle_auth_done global_id 1084130 payload 274 2024-09-16T04:12:44.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.878+0000 7fefd9d9b640 1 -- 172.21.15.42:0/1029717906 >> v2:172.21.15.154:3300/0 conn(0x7fefdc0a1e20 msgr2=0x7fefdc13c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:12:44.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.878+0000 7fefd9d9b640 1 --2- 172.21.15.42:0/1029717906 >> v2:172.21.15.154:3300/0 conn(0x7fefdc0a1e20 0x7fefdc13c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:12:44.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.878+0000 7fefd9d9b640 1 -- 172.21.15.42:0/1029717906 >> v1:172.21.15.154:6789/0 conn(0x7fefdc0a1330 legacy=0x7fefdc0a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:12:44.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.878+0000 7fefd9d9b640 10 monclient: _finish_hunting 0 2024-09-16T04:12:44.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.878+0000 7fefd9d9b640 1 monclient: found mon.noname-b 2024-09-16T04:12:44.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.878+0000 7fefd9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:12:44.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.878+0000 7fefd9d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T04:12:44.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.878+0000 7fefd9d9b640 1 -- 172.21.15.42:0/1029717906 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fefdc09f8f0 con 0x7fefdc06dac0 2024-09-16T04:12:44.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.878+0000 7fefd9d9b640 1 --2- 172.21.15.42:0/1029717906 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fefdc06dac0 0x7fefdc13a340 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fefd0009870 tx=0x7fefd001c4d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=92777b22fa8ee2e6 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:12:44.894 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.878+0000 7fefd959a640 1 -- 172.21.15.42:0/1029717906 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefd00aa610 con 0x7fefdc06dac0 2024-09-16T04:12:44.894 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.878+0000 7fefd959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:44.894 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.878+0000 7fefd959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:12:44.894 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.878+0000 7fefd959a640 10 monclient: dump: 2024-09-16T04:12:44.894 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:12:44.894 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:12:44.894 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:44.895 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:44.895 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:44.895 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:12:44.895 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:12:44.895 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:12:44.895 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:12:44.895 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:12:44.895 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.878+0000 7fefd959a640 10 monclient: _finish_auth 0 2024-09-16T04:12:44.895 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.878+0000 7fefd959a640 10 monclient: _check_auth_tickets 2024-09-16T04:12:44.895 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.878+0000 7fefd959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:12:44.895 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.878+0000 7fefd959a640 1 -- 172.21.15.42:0/1029717906 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fefd00af070 con 0x7fefdc06dac0 2024-09-16T04:12:44.895 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.878+0000 7fefe0f3a640 5 monclient: authenticate success, global_id 1084130 2024-09-16T04:12:44.895 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.878+0000 7fefe0f3a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:12:44.896 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.878+0000 7fefd959a640 10 monclient: handle_config config(0 keys) 2024-09-16T04:12:44.896 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.878+0000 7fefe0f3a640 10 monclient: get_monmap_and_config success 2024-09-16T04:12:44.896 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.878+0000 7fefd959a640 1 -- 172.21.15.42:0/1029717906 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefd00aa9d0 con 0x7fefdc06dac0 2024-09-16T04:12:44.896 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.878+0000 7fefd959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:44.896 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.878+0000 7fefe0f3a640 4 set_mon_vals no callback set 2024-09-16T04:12:44.896 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.878+0000 7fefd959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:12:44.896 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.878+0000 7fefd959a640 10 monclient: dump: 2024-09-16T04:12:44.896 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:12:44.896 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:12:44.896 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:44.897 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:44.897 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:44.897 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:12:44.897 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:12:44.897 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:12:44.897 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:12:44.897 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:12:44.897 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.878+0000 7fefe0f3a640 10 monclient: shutdown 2024-09-16T04:12:44.897 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.878+0000 7fefe0f3a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:12:44.897 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.878+0000 7fefe0f3a640 1 -- 172.21.15.42:0/1029717906 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fefdc06dac0 msgr2=0x7fefdc13a340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:44.897 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.878+0000 7fefe0f3a640 1 --2- 172.21.15.42:0/1029717906 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fefdc06dac0 0x7fefdc13a340 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fefd0009870 tx=0x7fefd001c4d0 comp rx=0 tx=0).stop 2024-09-16T04:12:44.897 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.878+0000 7fefe0f3a640 1 -- 172.21.15.42:0/1029717906 shutdown_connections 2024-09-16T04:12:44.897 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.878+0000 7fefe0f3a640 1 --2- 172.21.15.42:0/1029717906 >> v2:172.21.15.154:3300/0 conn(0x7fefdc0a1e20 0x7fefdc13c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:12:44.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.878+0000 7fefe0f3a640 1 --2- 172.21.15.42:0/1029717906 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fefdc06dac0 0x7fefdc13a340 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:12:44.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.878+0000 7fefe0f3a640 1 -- 172.21.15.42:0/1029717906 >> 172.21.15.42:0/1029717906 conn(0x7fefdc0a2420 msgr2=0x7fefdc0a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:12:44.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.878+0000 7fefe0f3a640 1 -- 172.21.15.42:0/1029717906 shutdown_connections 2024-09-16T04:12:44.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.878+0000 7fefe0f3a640 1 -- 172.21.15.42:0/1029717906 wait complete. 2024-09-16T04:12:44.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.882+0000 7fefe0f3a640 10 monclient: build_initial_monmap 2024-09-16T04:12:44.898 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.882+0000 7fefe0f3a640 10 monclient: monmap: 2024-09-16T04:12:44.898 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:12:44.898 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:12:44.898 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:12:44.898 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:12:44.898 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:12:44.898 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:12:44.898 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:12:44.899 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:12:44.899 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:12:44.899 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:12:44.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.882+0000 7fefe0f3a640 1 librados: starting msgr at 2024-09-16T04:12:44.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.882+0000 7fefe0f3a640 1 librados: starting objecter 2024-09-16T04:12:44.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.882+0000 7fefe0f3a640 1 Processor -- start 2024-09-16T04:12:44.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.882+0000 7fefe0f3a640 1 -- start start 2024-09-16T04:12:44.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.882+0000 7fefe0f3a640 1 librados: setting wanted keys 2024-09-16T04:12:44.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.882+0000 7fefe0f3a640 1 librados: calling monclient init 2024-09-16T04:12:44.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.882+0000 7fefe0f3a640 10 monclient: init 2024-09-16T04:12:44.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.882+0000 7fefe0f3a640 10 monclient: _reopen_session rank -1 2024-09-16T04:12:44.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.882+0000 7fefe0f3a640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:12:44.899 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.882+0000 7fefe0f3a640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fefdc0a1330 0x7fefdc1ceda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:12:44.900 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.882+0000 7fefe0f3a640 10 monclient(hunting): picked mon.noname-b con 0x7fefdc0a1330 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:12:44.900 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.882+0000 7fefe0f3a640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fefdc0a1e20 0x7fefdc1355e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:12:44.900 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.882+0000 7fefe0f3a640 10 monclient(hunting): picked mon.noname-c con 0x7fefdc0a1e20 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:12:44.900 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.882+0000 7fefe0f3a640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fefdc135b20 0x7fefdc09cb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:12:44.900 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.882+0000 7fefe0f3a640 10 monclient(hunting): picked mon.noname-a con 0x7fefdc135b20 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:12:44.900 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.882+0000 7fefe0f3a640 10 monclient(hunting): start opening mon connection 2024-09-16T04:12:44.900 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.882+0000 7fefe0f3a640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fefdc13f770 con 0x7fefdc135b20 2024-09-16T04:12:44.900 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.882+0000 7fefe0f3a640 10 monclient(hunting): start opening mon connection 2024-09-16T04:12:44.900 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.882+0000 7fefe0f3a640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fefdc13f5f0 con 0x7fefdc0a1330 2024-09-16T04:12:44.900 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.882+0000 7fefe0f3a640 10 monclient(hunting): start opening mon connection 2024-09-16T04:12:44.900 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.882+0000 7fefe0f3a640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fefdc13f8f0 con 0x7fefdc0a1e20 2024-09-16T04:12:44.900 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.882+0000 7fefe0f3a640 10 monclient(hunting): _renew_subs 2024-09-16T04:12:44.900 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.882+0000 7fefe0f3a640 10 monclient(hunting): authenticate will time out at 2811.853313s 2024-09-16T04:12:44.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.882+0000 7fefd9d9b640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fefdc0a1e20 0x7fefdc1355e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:12:44.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.882+0000 7fefd9d9b640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fefdc0a1e20 0x7fefdc1355e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:36756/0 (socket says 172.21.15.42:36756) 2024-09-16T04:12:44.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.882+0000 7fefd9d9b640 1 -- 172.21.15.42:0/714418510 learned_addr learned my addr 172.21.15.42:0/714418510 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:12:44.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.882+0000 7fefd9d9b640 10 monclient(hunting): get_auth_request con 0x7fefdc0a1e20 auth_method 0 2024-09-16T04:12:44.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.882+0000 7fefd9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:12:44.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.882+0000 7fefd9d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:44.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.882+0000 7fefd9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:44.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.882+0000 7fefdad9d640 1 --2- 172.21.15.42:0/714418510 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fefdc135b20 0x7fefdc09cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:12:44.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.882+0000 7fefdad9d640 10 monclient(hunting): get_auth_request con 0x7fefdc135b20 auth_method 0 2024-09-16T04:12:44.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.882+0000 7fefdad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:12:44.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.882+0000 7fefdad9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:44.901 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.882+0000 7fefdad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:44.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.882+0000 7fefd9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:12:44.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.882+0000 7fefd9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:12:44.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.882+0000 7fefd9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:12:44.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.882+0000 7fefdad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:12:44.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.882+0000 7fefdad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:12:44.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.882+0000 7fefdad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:12:44.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.882+0000 7fefda59c640 1 --2- 172.21.15.42:0/714418510 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fefdc0a1330 0x7fefdc1ceda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:12:44.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.882+0000 7fefda59c640 10 monclient(hunting): get_auth_request con 0x7fefdc0a1330 auth_method 0 2024-09-16T04:12:44.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.882+0000 7fefda59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:12:44.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.882+0000 7fefda59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:44.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.882+0000 7fefda59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:44.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.882+0000 7fefd9d9b640 10 monclient(hunting): handle_auth_done global_id 1084133 payload 824 2024-09-16T04:12:44.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.882+0000 7fefd9d9b640 1 -- 172.21.15.42:0/714418510 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fefdc0a1330 msgr2=0x7fefdc1ceda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:44.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.882+0000 7fefd9d9b640 1 --2- 172.21.15.42:0/714418510 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fefdc0a1330 0x7fefdc1ceda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:12:44.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.882+0000 7fefd9d9b640 1 -- 172.21.15.42:0/714418510 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fefdc135b20 msgr2=0x7fefdc09cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:44.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.882+0000 7fefd9d9b640 1 --2- 172.21.15.42:0/714418510 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fefdc135b20 0x7fefdc09cb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:12:44.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.882+0000 7fefd9d9b640 10 monclient: _finish_hunting 0 2024-09-16T04:12:44.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.882+0000 7fefd9d9b640 1 monclient: found mon.noname-c 2024-09-16T04:12:44.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.882+0000 7fefd9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:12:44.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.882+0000 7fefd9d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T04:12:44.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.882+0000 7fefd9d9b640 1 -- 172.21.15.42:0/714418510 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fefdc09e680 con 0x7fefdc0a1e20 2024-09-16T04:12:44.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.882+0000 7fefd9d9b640 1 --2- 172.21.15.42:0/714418510 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fefdc0a1e20 0x7fefdc1355e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fefd00ac040 tx=0x7fefd001ad10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:12:44.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.882+0000 7fefcaffd640 10 client.?.objecter ms_handle_connect 0x7fefdc0a1e20 2024-09-16T04:12:44.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.882+0000 7fefcaffd640 10 client.?.objecter resend_mon_ops 2024-09-16T04:12:44.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.882+0000 7fefcaffd640 1 -- 172.21.15.42:0/714418510 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefd00bd5f0 con 0x7fefdc0a1e20 2024-09-16T04:12:44.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.882+0000 7fefcaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:44.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.882+0000 7fefcaffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:12:44.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.882+0000 7fefcaffd640 10 monclient: dump: 2024-09-16T04:12:44.904 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:12:44.904 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:12:44.904 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:44.904 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:44.904 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:44.904 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:12:44.904 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:12:44.904 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:12:44.904 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:12:44.904 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:12:44.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.882+0000 7fefcaffd640 10 monclient: _finish_auth 0 2024-09-16T04:12:44.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.882+0000 7fefcaffd640 10 monclient: _check_auth_tickets 2024-09-16T04:12:44.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.882+0000 7fefcaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:12:44.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.882+0000 7fefe0f3a640 5 monclient: authenticate success, global_id 1084133 2024-09-16T04:12:44.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.882+0000 7fefcaffd640 1 -- 172.21.15.42:0/714418510 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fefd00bd7b0 con 0x7fefdc0a1e20 2024-09-16T04:12:44.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.882+0000 7fefcaffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:12:44.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.882+0000 7fefe0f3a640 10 monclient: _renew_subs 2024-09-16T04:12:44.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.882+0000 7fefcaffd640 1 -- 172.21.15.42:0/714418510 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefd00bdba0 con 0x7fefdc0a1e20 2024-09-16T04:12:44.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.882+0000 7fefe0f3a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:12:44.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.882+0000 7fefe0f3a640 1 -- 172.21.15.42:0/714418510 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fefdc06e090 con 0x7fefdc0a1e20 2024-09-16T04:12:44.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.882+0000 7fefd8d99640 4 set_mon_vals no callback set 2024-09-16T04:12:44.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.882+0000 7fefcaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:44.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.882+0000 7fefcaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:12:44.905 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.882+0000 7fefcaffd640 10 monclient: dump: 2024-09-16T04:12:44.905 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:12:44.905 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:12:44.906 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:44.906 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:44.906 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:44.906 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:12:44.906 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:12:44.906 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:12:44.906 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:12:44.906 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:12:44.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.882+0000 7fefe0f3a640 10 client.1084133.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:12:44.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.882+0000 7fefe0f3a640 10 monclient: _renew_subs 2024-09-16T04:12:44.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.882+0000 7fefe0f3a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:12:44.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.882+0000 7fefe0f3a640 1 -- 172.21.15.42:0/714418510 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fefdc121cb0 con 0x7fefdc0a1e20 2024-09-16T04:12:44.906 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.882+0000 7fefe0f3a640 1 librados: init done 2024-09-16T04:12:44.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.882+0000 7fefe0f3a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:12:44.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.882+0000 7fefe0f3a640 1 --2- 172.21.15.42:0/714418510 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fefdc06dac0 0x7fefdc09d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:12:44.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.882+0000 7fefe0f3a640 10 monclient(hunting): start opening mon connection 2024-09-16T04:12:44.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.882+0000 7fefe0f3a640 1 -- 172.21.15.42:0/714418510 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fefdc0a1710 con 0x7fefdc06dac0 2024-09-16T04:12:44.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.882+0000 7fefda59c640 1 --2- 172.21.15.42:0/714418510 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fefdc06dac0 0x7fefdc09d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:12:44.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.882+0000 7fefda59c640 10 monclient: get_auth_request con 0x7fefdc06dac0 auth_method 0 2024-09-16T04:12:44.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.882+0000 7fefda59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:12:44.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.882+0000 7fefda59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:44.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.882+0000 7fefda59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:44.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.882+0000 7fefcaffd640 1 -- 172.21.15.42:0/714418510 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fefd00b33f0 con 0x7fefdc0a1e20 2024-09-16T04:12:44.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.882+0000 7fefcaffd640 1 --2- 172.21.15.42:0/714418510 >> v2:172.21.15.154:6812/254845342 conn(0x7fefbc037840 0x7fefbc039d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:12:44.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.882+0000 7fefcaffd640 1 -- 172.21.15.42:0/714418510 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(706..706 src has 681..706) ==== 5040+0+0 (secure 0 0 0) 0x7fefd00e3aa0 con 0x7fefdc0a1e20 2024-09-16T04:12:44.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.882+0000 7fefcaffd640 10 client.1084133.objecter ms_dispatch 0x7fefdc000ba0 osd_map(706..706 src has 681..706) 2024-09-16T04:12:44.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.882+0000 7fefcaffd640 3 client.1084133.objecter handle_osd_map got epochs [706,706] > 0 2024-09-16T04:12:44.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.882+0000 7fefcaffd640 3 client.1084133.objecter handle_osd_map decoding full epoch 706 2024-09-16T04:12:44.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.882+0000 7fefcaffd640 20 client.1084133.objecter dump_active .. 0 homeless 2024-09-16T04:12:44.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.882+0000 7fefda59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:12:44.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.882+0000 7fefda59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:12:44.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.882+0000 7fefda59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:12:44.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.882+0000 7fefda59c640 10 monclient(hunting): handle_auth_done global_id 1034541 payload 824 2024-09-16T04:12:44.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.882+0000 7fefdad9d640 1 --2- 172.21.15.42:0/714418510 >> v2:172.21.15.154:6812/254845342 conn(0x7fefbc037840 0x7fefbc039d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:12:44.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.882+0000 7fefda59c640 1 -- 172.21.15.42:0/714418510 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fefdc120380 con 0x7fefdc06dac0 2024-09-16T04:12:44.908 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.882+0000 7fefdad9d640 10 monclient: get_auth_request con 0x7fefbc037840 auth_method 0 2024-09-16T04:12:44.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.882+0000 7fefcaffd640 10 client.1084133.objecter ms_handle_connect 0x7fefdc06dac0 2024-09-16T04:12:44.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.882+0000 7fefcaffd640 10 client.1084133.objecter resend_mon_ops 2024-09-16T04:12:44.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.882+0000 7fefda59c640 1 --2- 172.21.15.42:0/714418510 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fefdc06dac0 0x7fefdc09d0a0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fefcc00e450 tx=0x7fefcc00e910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:12:44.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.886+0000 7fefcaffd640 1 -- 172.21.15.42:0/714418510 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefcc007240 con 0x7fefdc06dac0 2024-09-16T04:12:44.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.886+0000 7fefcaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:44.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.886+0000 7fefcaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:12:44.909 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.886+0000 7fefcaffd640 10 monclient: dump: 2024-09-16T04:12:44.909 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:12:44.909 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:12:44.909 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:44.909 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:44.909 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:44.909 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:12:44.910 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:12:44.910 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:12:44.910 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:12:44.910 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:12:44.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.886+0000 7fefcaffd640 1 -- 172.21.15.42:0/714418510 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fefcc007240 con 0x7fefdc06dac0 2024-09-16T04:12:44.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.886+0000 7fefcaffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:12:44.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.886+0000 7fefcaffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:12:44.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.886+0000 7fefcaffd640 1 -- 172.21.15.42:0/714418510 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fefdc06dac0 msgr2=0x7fefdc09d0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:44.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.886+0000 7fefcaffd640 1 --2- 172.21.15.42:0/714418510 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fefdc06dac0 0x7fefdc09d0a0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fefcc00e450 tx=0x7fefcc00e910 comp rx=0 tx=0).stop 2024-09-16T04:12:44.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.886+0000 7fefcaffd640 1 -- 172.21.15.42:0/714418510 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fefdc06dac0 msgr2=0x7fefdc09d0a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:12:44.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.886+0000 7fefcaffd640 1 --2- 172.21.15.42:0/714418510 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fefdc06dac0 0x7fefdc09d0a0 secure :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fefcc00e450 tx=0x7fefcc00e910 comp rx=0 tx=0).stop 2024-09-16T04:12:44.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.886+0000 7fefdad9d640 1 --2- 172.21.15.42:0/714418510 >> v2:172.21.15.154:6812/254845342 conn(0x7fefbc037840 0x7fefbc039d00 secure :-1 s=READY pgs=3972 cs=0 l=1 rev1=1 crypto rx=0x7fefd4003ab0 tx=0x7fefd4009820 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:12:44.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.886+0000 7fefcaffd640 10 client.1084133.objecter ms_handle_connect 0x7fefbc037840 2024-09-16T04:12:44.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.890+0000 7fefe0f3a640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:12:44.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.890+0000 7fefe0f3a640 1 --2- 172.21.15.42:0/714418510 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fefdc09d5e0 0x7fefdc138330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:12:44.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.890+0000 7fefe0f3a640 10 monclient(hunting): start opening mon connection 2024-09-16T04:12:44.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.890+0000 7fefe0f3a640 1 -- 172.21.15.42:0/714418510 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fefdc09d9c0 con 0x7fefdc09d5e0 2024-09-16T04:12:44.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.890+0000 7fefda59c640 1 --2- 172.21.15.42:0/714418510 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fefdc09d5e0 0x7fefdc138330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:12:44.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.890+0000 7fefda59c640 10 monclient: get_auth_request con 0x7fefdc09d5e0 auth_method 0 2024-09-16T04:12:44.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.890+0000 7fefda59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:12:44.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.890+0000 7fefda59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:44.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.890+0000 7fefda59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:44.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.890+0000 7fefda59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:12:44.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.890+0000 7fefda59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:12:44.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.890+0000 7fefda59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:12:44.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.890+0000 7fefda59c640 10 monclient(hunting): handle_auth_done global_id 1034544 payload 824 2024-09-16T04:12:44.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.890+0000 7fefda59c640 1 -- 172.21.15.42:0/714418510 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fefdc069040 con 0x7fefdc09d5e0 2024-09-16T04:12:44.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.890+0000 7fefcaffd640 10 client.1084133.objecter ms_handle_connect 0x7fefdc09d5e0 2024-09-16T04:12:44.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.890+0000 7fefda59c640 1 --2- 172.21.15.42:0/714418510 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fefdc09d5e0 0x7fefdc138330 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fefcc00e450 tx=0x7fefcc003770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:12:44.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.890+0000 7fefcaffd640 10 client.1084133.objecter resend_mon_ops 2024-09-16T04:12:44.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.890+0000 7fefcaffd640 1 -- 172.21.15.42:0/714418510 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefcc01ea20 con 0x7fefdc09d5e0 2024-09-16T04:12:44.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.890+0000 7fefcaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:44.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.890+0000 7fefcaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:12:44.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.890+0000 7fefcaffd640 10 monclient: dump: 2024-09-16T04:12:44.914 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:12:44.914 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:12:44.914 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:44.914 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:44.914 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:44.914 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:12:44.914 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:12:44.914 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:12:44.915 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:12:44.915 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:12:44.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.890+0000 7fefcaffd640 1 -- 172.21.15.42:0/714418510 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4715 (secure 0 0 0) 0x7fefcc01ea20 con 0x7fefdc09d5e0 2024-09-16T04:12:44.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.890+0000 7fefcaffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:12:44.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.890+0000 7fefcaffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:12:44.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.890+0000 7fefcaffd640 1 -- 172.21.15.42:0/714418510 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fefdc09d5e0 msgr2=0x7fefdc138330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:44.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.890+0000 7fefcaffd640 1 --2- 172.21.15.42:0/714418510 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fefdc09d5e0 0x7fefdc138330 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fefcc00e450 tx=0x7fefcc003770 comp rx=0 tx=0).stop 2024-09-16T04:12:44.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.890+0000 7fefcaffd640 1 -- 172.21.15.42:0/714418510 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fefdc09d5e0 msgr2=0x7fefdc138330 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:12:44.915 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.890+0000 7fefcaffd640 1 --2- 172.21.15.42:0/714418510 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fefdc09d5e0 0x7fefdc138330 secure :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fefcc00e450 tx=0x7fefcc003770 comp rx=0 tx=0).stop 2024-09-16T04:12:44.915 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T04:12:44.915 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:12:44.916 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:12:44.916 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "leader", 2024-09-16T04:12:44.916 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 276, 2024-09-16T04:12:44.916 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 244825, 2024-09-16T04:12:44.916 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T04:12:44.916 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T04:12:44.916 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T04:12:44.916 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T04:12:44.916 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:12:44.916 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 2, 2024-09-16T04:12:44.916 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:12:44.916 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T04:12:44.916 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T04:12:44.917 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:12:44.917 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:12:44.917 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:12:44.917 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:12:44.917 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:12:44.917 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:12:44.917 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:12:44.917 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:12:44.918 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:12:44.918 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:12:44.918 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:12:44.918 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:12:44.918 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:12:44.918 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T04:12:44.918 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:12:44.918 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:12:44.918 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:12:44.918 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:12:44.918 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:12:44.918 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:12:44.918 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:12:44.919 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:12:44.919 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:12:44.919 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:12:44.919 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:12:44.919 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:12:44.919 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:12:44.919 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T04:12:44.919 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T04:12:44.919 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T04:12:44.919 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T04:12:44.919 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T04:12:44.919 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T04:12:44.919 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:12:44.920 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:12:44.920 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T04:12:44.920 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T04:12:44.920 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T04:12:44.920 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T04:12:44.920 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T04:12:44.920 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T04:12:44.920 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T04:12:44.920 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:12:44.920 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T04:12:44.920 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:12:44.920 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:12:44.920 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:12:44.920 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:12:44.921 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:12:44.921 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:12:44.921 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:12:44.921 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:12:44.921 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:12:44.921 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:12:44.921 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:12:44.921 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:12:44.921 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T04:12:44.921 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:12:44.921 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T04:12:44.921 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:12:44.921 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:12:44.922 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:12:44.922 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:12:44.922 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:12:44.922 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:12:44.922 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:12:44.922 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T04:12:44.922 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:12:44.922 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:12:44.922 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:12:44.922 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:12:44.922 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T04:12:44.922 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:12:44.922 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:12:44.922 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:12:44.923 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:12:44.923 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T04:12:44.923 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T04:12:44.923 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:12:44.923 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:12:44.923 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:12:44.923 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:12:44.923 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:12:44.923 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:12:44.923 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:12:44.923 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:12:44.923 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:12:44.923 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:12:44.923 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:12:44.923 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T04:12:44.923 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:12:44.924 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:12:44.924 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:12:44.924 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:12:44.924 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T04:12:44.924 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:12:44.924 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:12:44.924 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:12:44.924 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:12:44.924 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T04:12:44.924 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T04:12:44.924 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:12:44.924 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:12:44.924 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:12:44.925 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:12:44.925 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:12:44.925 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:12:44.925 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:12:44.925 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:12:44.925 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:12:44.925 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:12:44.925 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:12:44.925 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T04:12:44.925 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:12:44.925 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:12:44.925 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:12:44.925 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:12:44.926 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T04:12:44.926 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:12:44.926 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:12:44.926 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:12:44.926 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:12:44.926 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T04:12:44.926 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T04:12:44.926 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:12:44.926 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:12:44.926 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:12:44.926 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:12:44.926 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:12:44.926 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:12:44.926 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T04:12:44.927 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T04:12:44.927 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:12:44.927 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:12:44.927 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:12:44.927 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:12:44.927 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:12:44.927 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:12:44.927 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T04:12:44.927 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:12:44.927 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:12:44.927 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:12:44.927 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 6 2024-09-16T04:12:44.927 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:12:44.927 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:12:44.928 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T04:12:44.928 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:12:44.928 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:12:44.928 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:12:44.928 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 5 2024-09-16T04:12:44.928 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:12:44.928 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:12:44.928 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:12:44.928 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T04:12:44.928 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T04:12:44.930 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.890+0000 7fefe0f3a640 10 librados: watch_flush enter 2024-09-16T04:12:44.930 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.890+0000 7fefe0f3a640 10 librados: watch_flush exit 2024-09-16T04:12:44.930 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.890+0000 7fefe0f3a640 20 client.1084133.objecter shutdown clearing up homeless session... 2024-09-16T04:12:44.930 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.890+0000 7fefe0f3a640 10 client.1084133.objecter successfully canceled tick 2024-09-16T04:12:44.930 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.890+0000 7fefe0f3a640 1 -- 172.21.15.42:0/714418510 >> v2:172.21.15.154:6812/254845342 conn(0x7fefbc037840 msgr2=0x7fefbc039d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:44.930 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.890+0000 7fefe0f3a640 1 --2- 172.21.15.42:0/714418510 >> v2:172.21.15.154:6812/254845342 conn(0x7fefbc037840 0x7fefbc039d00 secure :-1 s=READY pgs=3972 cs=0 l=1 rev1=1 crypto rx=0x7fefd4003ab0 tx=0x7fefd4009820 comp rx=0 tx=0).stop 2024-09-16T04:12:44.930 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.890+0000 7fefe0f3a640 10 monclient: shutdown 2024-09-16T04:12:44.930 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.890+0000 7fefe0f3a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:12:44.930 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.890+0000 7fefe0f3a640 1 -- 172.21.15.42:0/714418510 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fefdc0a1e20 msgr2=0x7fefdc1355e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:44.930 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.890+0000 7fefe0f3a640 1 --2- 172.21.15.42:0/714418510 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fefdc0a1e20 0x7fefdc1355e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fefd00ac040 tx=0x7fefd001ad10 comp rx=0 tx=0).stop 2024-09-16T04:12:44.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.890+0000 7fefdad9d640 1 -- 172.21.15.42:0/714418510 reap_dead start 2024-09-16T04:12:44.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.894+0000 7fefe0f3a640 1 -- 172.21.15.42:0/714418510 shutdown_connections 2024-09-16T04:12:44.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.894+0000 7fefe0f3a640 1 --2- 172.21.15.42:0/714418510 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fefdc0a1e20 0x7fefdc1355e0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:12:44.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.894+0000 7fefe0f3a640 1 -- 172.21.15.42:0/714418510 >> 172.21.15.42:0/714418510 conn(0x7fefdc0a2420 msgr2=0x7fefdc0a45b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:12:44.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.894+0000 7fefe0f3a640 1 -- 172.21.15.42:0/714418510 shutdown_connections 2024-09-16T04:12:44.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.894+0000 7fefe0f3a640 1 -- 172.21.15.42:0/714418510 wait complete. 2024-09-16T04:12:44.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:44.894+0000 7fefe0f3a640 1 librados: shutdown 2024-09-16T04:12:44.931 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:12:44.931 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T04:12:44.932 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T04:12:45.045 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.038+0000 7f09a20da640 10 monclient: get_monmap_and_config 2024-09-16T04:12:45.045 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.038+0000 7f09a20da640 10 monclient: build_initial_monmap 2024-09-16T04:12:45.045 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.038+0000 7f09a20da640 10 monclient: monmap: 2024-09-16T04:12:45.045 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:12:45.045 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:12:45.045 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:12:45.043806+0000 2024-09-16T04:12:45.045 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:12:45.043806+0000 2024-09-16T04:12:45.045 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:12:45.045 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:12:45.046 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:12:45.046 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:12:45.046 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:12:45.046 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:12:45.046 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:12:45.046 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:12:45.048 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.042+0000 7f09a20da640 1 Processor -- start 2024-09-16T04:12:45.048 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.042+0000 7f09a20da640 1 -- start start 2024-09-16T04:12:45.048 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.042+0000 7f09a20da640 10 monclient: init 2024-09-16T04:12:45.048 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.042+0000 7f09a20da640 10 monclient: _reopen_session rank -1 2024-09-16T04:12:45.048 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.042+0000 7f09a20da640 10 monclient: _add_conns ranks=[4,1,0,2,3] 2024-09-16T04:12:45.048 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.042+0000 7f09a20da640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f099c06db90 0x7f099c1340c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:12:45.048 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.042+0000 7f09a20da640 10 monclient(hunting): picked mon.noname-c con 0x7f099c06db90 addr v2:172.21.15.154:3300/0 2024-09-16T04:12:45.048 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.042+0000 7f09a20da640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f099c0a37f0 addr v1:172.21.15.154:6789/0 2024-09-16T04:12:45.048 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.042+0000 7f09a20da640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f099c0a3c00 addr v1:172.21.15.42:6789/0 2024-09-16T04:12:45.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.042+0000 7f09a20da640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f099c06ece0 con 0x7f099c0a3c00 2024-09-16T04:12:45.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.042+0000 7f09a20da640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f099c09bf80 con 0x7f099c0a37f0 2024-09-16T04:12:45.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.042+0000 7f09a20da640 10 monclient(hunting): start opening mon connection 2024-09-16T04:12:45.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.042+0000 7f09a20da640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f099c06df70 con 0x7f099c06db90 2024-09-16T04:12:45.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.042+0000 7f09a20da640 10 monclient(hunting): _renew_subs 2024-09-16T04:12:45.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.042+0000 7f09a20da640 10 monclient(hunting): authenticate will time out at 2812.013318s 2024-09-16T04:12:45.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.042+0000 7f099bfff640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f099c0a3c00 0x7f099c137520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:34580/0 (socket says 172.21.15.42:34580) 2024-09-16T04:12:45.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.042+0000 7f099bfff640 1 -- 172.21.15.42:0/3038021366 learned_addr learned my addr 172.21.15.42:0/3038021366 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T04:12:45.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.042+0000 7f099b7fe640 1 --2- 172.21.15.42:0/3038021366 >> v2:172.21.15.154:3300/0 conn(0x7f099c06db90 0x7f099c1340c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:12:45.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.042+0000 7f099b7fe640 10 monclient(hunting): get_auth_request con 0x7f099c06db90 auth_method 0 2024-09-16T04:12:45.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.042+0000 7f099b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:12:45.050 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.042+0000 7f099b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:45.050 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.042+0000 7f099b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:45.050 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.042+0000 7f099b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:12:45.050 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.042+0000 7f099b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:12:45.050 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.042+0000 7f099b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:12:45.050 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.042+0000 7f099a7fc640 1 -- 172.21.15.42:0/3038021366 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3779677905 0 0) 0x7f099c06ece0 con 0x7f099c0a3c00 2024-09-16T04:12:45.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.042+0000 7f099a7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:12:45.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.042+0000 7f099a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:45.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.042+0000 7f099a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:45.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.042+0000 7f099a7fc640 10 monclient(hunting): my global_id is 1034547 2024-09-16T04:12:45.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.042+0000 7f099a7fc640 1 -- 172.21.15.42:0/3038021366 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0984003530 con 0x7f099c0a3c00 2024-09-16T04:12:45.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.042+0000 7f099b7fe640 10 monclient(hunting): handle_auth_done global_id 1074202 payload 274 2024-09-16T04:12:45.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.042+0000 7f099b7fe640 1 -- 172.21.15.42:0/3038021366 >> v1:172.21.15.154:6789/0 conn(0x7f099c0a37f0 legacy=0x7f099c136e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:45.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.042+0000 7f099b7fe640 1 -- 172.21.15.42:0/3038021366 >> v1:172.21.15.42:6789/0 conn(0x7f099c0a3c00 legacy=0x7f099c137520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:45.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.042+0000 7f099b7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:12:45.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.042+0000 7f099b7fe640 1 monclient: found mon.noname-c 2024-09-16T04:12:45.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.042+0000 7f099b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:12:45.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.042+0000 7f099b7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.154:3300/0 2024-09-16T04:12:45.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.042+0000 7f099b7fe640 1 -- 172.21.15.42:0/3038021366 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f099c137c30 con 0x7f099c06db90 2024-09-16T04:12:45.059 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.046+0000 7f099b7fe640 1 --2- 172.21.15.42:0/3038021366 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f099c06db90 0x7f099c1340c0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f098c0097c0 tx=0x7f098c01c360 comp rx=0 tx=0).ready entity=mon.1 client_cookie=40a78c70ed4d72ef server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:12:45.059 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.046+0000 7f099a7fc640 1 -- 172.21.15.42:0/3038021366 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f098c0b2070 con 0x7f099c06db90 2024-09-16T04:12:45.060 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.046+0000 7f099a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:45.060 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.046+0000 7f099a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:12:45.060 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.046+0000 7f099a7fc640 10 monclient: dump: 2024-09-16T04:12:45.060 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:12:45.060 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:12:45.060 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:45.060 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:45.060 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:45.060 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:12:45.060 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:12:45.060 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:12:45.060 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:12:45.061 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:12:45.061 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.046+0000 7f099a7fc640 10 monclient: _finish_auth 0 2024-09-16T04:12:45.061 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.046+0000 7f099a7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:12:45.061 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.046+0000 7f099a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:12:45.061 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.046+0000 7f099a7fc640 1 -- 172.21.15.42:0/3038021366 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f098c0a9070 con 0x7f099c06db90 2024-09-16T04:12:45.061 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.046+0000 7f099a7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:12:45.061 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.046+0000 7f099a7fc640 1 -- 172.21.15.42:0/3038021366 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f098c0ad860 con 0x7f099c06db90 2024-09-16T04:12:45.061 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.046+0000 7f09a20da640 5 monclient: authenticate success, global_id 1074202 2024-09-16T04:12:45.061 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.046+0000 7f09a20da640 10 monclient: get_monmap_and_config success 2024-09-16T04:12:45.061 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.046+0000 7f09a20da640 4 set_mon_vals no callback set 2024-09-16T04:12:45.061 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.046+0000 7f099a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:45.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.046+0000 7f099a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:12:45.062 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.046+0000 7f099a7fc640 10 monclient: dump: 2024-09-16T04:12:45.062 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:12:45.062 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:12:45.062 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:45.062 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:45.062 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:45.062 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:12:45.062 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:12:45.062 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:12:45.063 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:12:45.063 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:12:45.063 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.046+0000 7f09a20da640 10 monclient: shutdown 2024-09-16T04:12:45.063 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.046+0000 7f09a20da640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:12:45.063 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.046+0000 7f09a20da640 1 -- 172.21.15.42:0/3038021366 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f099c06db90 msgr2=0x7f099c1340c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:45.063 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.046+0000 7f09a20da640 1 --2- 172.21.15.42:0/3038021366 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f099c06db90 0x7f099c1340c0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f098c0097c0 tx=0x7f098c01c360 comp rx=0 tx=0).stop 2024-09-16T04:12:45.063 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.046+0000 7f09a20da640 1 -- 172.21.15.42:0/3038021366 shutdown_connections 2024-09-16T04:12:45.063 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.046+0000 7f09a20da640 1 --2- 172.21.15.42:0/3038021366 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f099c06db90 0x7f099c1340c0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:12:45.063 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.046+0000 7f09a20da640 1 -- 172.21.15.42:0/3038021366 >> 172.21.15.42:0/3038021366 conn(0x7f099c0a9fb0 msgr2=0x7f099c0aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:12:45.063 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.046+0000 7f09a20da640 1 -- 172.21.15.42:0/3038021366 shutdown_connections 2024-09-16T04:12:45.063 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.046+0000 7f09a20da640 1 -- 172.21.15.42:0/3038021366 wait complete. 2024-09-16T04:12:45.063 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.046+0000 7f09a20da640 10 monclient: build_initial_monmap 2024-09-16T04:12:45.063 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.046+0000 7f09a20da640 10 monclient: monmap: 2024-09-16T04:12:45.064 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:12:45.064 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:12:45.064 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:12:45.064 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:12:45.064 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:12:45.064 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:12:45.064 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:12:45.064 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:12:45.064 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:12:45.064 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:12:45.064 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.046+0000 7f09a20da640 1 librados: starting msgr at 2024-09-16T04:12:45.064 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.046+0000 7f09a20da640 1 librados: starting objecter 2024-09-16T04:12:45.065 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.046+0000 7f09a20da640 1 Processor -- start 2024-09-16T04:12:45.065 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.046+0000 7f09a20da640 1 -- start start 2024-09-16T04:12:45.065 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.046+0000 7f09a20da640 1 librados: setting wanted keys 2024-09-16T04:12:45.065 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.046+0000 7f09a20da640 1 librados: calling monclient init 2024-09-16T04:12:45.065 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.046+0000 7f09a20da640 10 monclient: init 2024-09-16T04:12:45.065 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.046+0000 7f09a20da640 10 monclient: _reopen_session rank -1 2024-09-16T04:12:45.065 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.046+0000 7f09a20da640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:12:45.065 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.046+0000 7f09a20da640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f099c06db90 0x7f099c1309b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:12:45.065 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.046+0000 7f09a20da640 10 monclient(hunting): picked mon.noname-b con 0x7f099c06db90 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:12:45.065 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.046+0000 7f09a20da640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f099c0a37f0 0x7f099c0a1590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:12:45.065 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.046+0000 7f09a20da640 10 monclient(hunting): picked mon.noname-a con 0x7f099c0a37f0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:12:45.065 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.046+0000 7f09a20da640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f099c0a3c00 0x7f099c0a1ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:12:45.065 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.046+0000 7f09a20da640 10 monclient(hunting): picked mon.noname-c con 0x7f099c0a3c00 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:12:45.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.046+0000 7f09a20da640 10 monclient(hunting): start opening mon connection 2024-09-16T04:12:45.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.046+0000 7f09a20da640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f099c142b60 con 0x7f099c0a37f0 2024-09-16T04:12:45.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.046+0000 7f09a20da640 10 monclient(hunting): start opening mon connection 2024-09-16T04:12:45.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.046+0000 7f09a20da640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f099c1429e0 con 0x7f099c06db90 2024-09-16T04:12:45.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.046+0000 7f09a20da640 10 monclient(hunting): start opening mon connection 2024-09-16T04:12:45.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.046+0000 7f09a20da640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f099c142ce0 con 0x7f099c0a3c00 2024-09-16T04:12:45.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.046+0000 7f09a20da640 10 monclient(hunting): _renew_subs 2024-09-16T04:12:45.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.046+0000 7f09a20da640 10 monclient(hunting): authenticate will time out at 2812.019227s 2024-09-16T04:12:45.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.046+0000 7f099affd640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f099c0a37f0 0x7f099c0a1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:12:45.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.046+0000 7f099bfff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f099c0a3c00 0x7f099c0a1ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:12:45.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.046+0000 7f099affd640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f099c0a37f0 0x7f099c0a1590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:46782/0 (socket says 172.21.15.42:46782) 2024-09-16T04:12:45.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.046+0000 7f099bfff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f099c0a3c00 0x7f099c0a1ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:36770/0 (socket says 172.21.15.42:36770) 2024-09-16T04:12:45.066 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.046+0000 7f099affd640 1 -- 172.21.15.42:0/4595208 learned_addr learned my addr 172.21.15.42:0/4595208 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:12:45.067 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.046+0000 7f099affd640 10 monclient(hunting): get_auth_request con 0x7f099c0a37f0 auth_method 0 2024-09-16T04:12:45.067 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.046+0000 7f099affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:12:45.067 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.046+0000 7f099affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:45.067 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.046+0000 7f099affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:45.067 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.046+0000 7f099bfff640 10 monclient(hunting): get_auth_request con 0x7f099c0a3c00 auth_method 0 2024-09-16T04:12:45.067 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.046+0000 7f099bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:12:45.067 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.046+0000 7f099bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:45.067 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.046+0000 7f099bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:45.067 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.046+0000 7f099b7fe640 1 --2- 172.21.15.42:0/4595208 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f099c06db90 0x7f099c1309b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:12:45.067 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.046+0000 7f099b7fe640 10 monclient(hunting): get_auth_request con 0x7f099c06db90 auth_method 0 2024-09-16T04:12:45.067 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.046+0000 7f099b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:12:45.067 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.046+0000 7f099b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:45.067 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.046+0000 7f099b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:45.068 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.046+0000 7f099affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:12:45.068 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.046+0000 7f099affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:12:45.068 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.046+0000 7f099affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:12:45.068 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.046+0000 7f099bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:12:45.068 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.046+0000 7f099bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:12:45.068 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.046+0000 7f099bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:12:45.068 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.046+0000 7f099b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:12:45.068 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.046+0000 7f099b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:12:45.068 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.046+0000 7f099b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:12:45.068 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.046+0000 7f099affd640 10 monclient(hunting): handle_auth_done global_id 1034550 payload 824 2024-09-16T04:12:45.068 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.046+0000 7f099affd640 1 -- 172.21.15.42:0/4595208 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f099c0a3c00 msgr2=0x7f099c0a1ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:45.068 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.046+0000 7f099affd640 1 --2- 172.21.15.42:0/4595208 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f099c0a3c00 0x7f099c0a1ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:12:45.068 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.046+0000 7f099affd640 1 -- 172.21.15.42:0/4595208 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f099c06db90 msgr2=0x7f099c1309b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:45.069 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.046+0000 7f099affd640 1 --2- 172.21.15.42:0/4595208 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f099c06db90 0x7f099c1309b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:12:45.069 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.046+0000 7f099affd640 10 monclient: _finish_hunting 0 2024-09-16T04:12:45.069 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.046+0000 7f099affd640 1 monclient: found mon.noname-a 2024-09-16T04:12:45.069 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.046+0000 7f099affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:12:45.069 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.046+0000 7f099affd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:12:45.069 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.046+0000 7f099affd640 1 -- 172.21.15.42:0/4595208 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f099c133110 con 0x7f099c0a37f0 2024-09-16T04:12:45.069 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.046+0000 7f099bfff640 1 --2- 172.21.15.42:0/4595208 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f099c0a3c00 0x7f099c0a1ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:12:45.069 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.046+0000 7f099affd640 1 --2- 172.21.15.42:0/4595208 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f099c0a37f0 0x7f099c0a1590 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f099000c420 tx=0x7f099000c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:12:45.069 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.050+0000 7f0983fff640 10 client.?.objecter ms_handle_connect 0x7f099c0a37f0 2024-09-16T04:12:45.069 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.050+0000 7f0983fff640 10 client.?.objecter resend_mon_ops 2024-09-16T04:12:45.069 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.050+0000 7f0983fff640 1 -- 172.21.15.42:0/4595208 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09900052d0 con 0x7f099c0a37f0 2024-09-16T04:12:45.069 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.050+0000 7f0983fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:45.070 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.050+0000 7f0983fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:12:45.070 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.050+0000 7f0983fff640 10 monclient: dump: 2024-09-16T04:12:45.070 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:12:45.070 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:12:45.070 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:45.070 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:45.070 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:45.070 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:12:45.070 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:12:45.070 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:12:45.070 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:12:45.070 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:12:45.070 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.050+0000 7f0983fff640 10 monclient: _finish_auth 0 2024-09-16T04:12:45.070 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.050+0000 7f0983fff640 10 monclient: _check_auth_tickets 2024-09-16T04:12:45.071 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.050+0000 7f0983fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:12:45.071 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.050+0000 7f0983fff640 1 -- 172.21.15.42:0/4595208 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0990005470 con 0x7f099c0a37f0 2024-09-16T04:12:45.071 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.050+0000 7f0983fff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:12:45.071 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.050+0000 7f0983fff640 1 -- 172.21.15.42:0/4595208 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0990007260 con 0x7f099c0a37f0 2024-09-16T04:12:45.071 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.050+0000 7f09a20da640 5 monclient: authenticate success, global_id 1034550 2024-09-16T04:12:45.071 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.050+0000 7f09a20da640 10 monclient: _renew_subs 2024-09-16T04:12:45.071 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.050+0000 7f09a20da640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:12:45.071 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.050+0000 7f09a20da640 1 -- 172.21.15.42:0/4595208 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f099c06f6a0 con 0x7f099c0a37f0 2024-09-16T04:12:45.071 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.050+0000 7f0983fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:45.071 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.050+0000 7f0983fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:12:45.071 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.050+0000 7f0983fff640 10 monclient: dump: 2024-09-16T04:12:45.071 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:12:45.072 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:12:45.072 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:45.072 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:45.072 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:45.072 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:12:45.072 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:12:45.072 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:12:45.072 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:12:45.072 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:12:45.072 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.050+0000 7f09997fa640 4 set_mon_vals no callback set 2024-09-16T04:12:45.072 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.050+0000 7f09a20da640 10 client.1034550.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:12:45.072 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.050+0000 7f09a20da640 10 monclient: _renew_subs 2024-09-16T04:12:45.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.050+0000 7f09a20da640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:12:45.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.050+0000 7f09a20da640 1 -- 172.21.15.42:0/4595208 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f099c13a530 con 0x7f099c0a37f0 2024-09-16T04:12:45.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.050+0000 7f09a20da640 1 librados: init done 2024-09-16T04:12:45.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.050+0000 7f09a20da640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:12:45.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.050+0000 7f09a20da640 1 --2- 172.21.15.42:0/4595208 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f099c130ef0 0x7f099c0a2010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:12:45.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.050+0000 7f09a20da640 10 monclient(hunting): start opening mon connection 2024-09-16T04:12:45.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.050+0000 7f09a20da640 1 -- 172.21.15.42:0/4595208 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f099c1340c0 con 0x7f099c130ef0 2024-09-16T04:12:45.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.050+0000 7f0983fff640 1 -- 172.21.15.42:0/4595208 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f0990007400 con 0x7f099c0a37f0 2024-09-16T04:12:45.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.050+0000 7f099b7fe640 1 --2- 172.21.15.42:0/4595208 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f099c130ef0 0x7f099c0a2010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:12:45.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.050+0000 7f099b7fe640 10 monclient: get_auth_request con 0x7f099c130ef0 auth_method 0 2024-09-16T04:12:45.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.050+0000 7f099b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:12:45.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.050+0000 7f099b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:45.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.050+0000 7f099b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:45.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.050+0000 7f099b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:12:45.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.050+0000 7f099b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:12:45.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.050+0000 7f099b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:12:45.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.050+0000 7f0983fff640 1 --2- 172.21.15.42:0/4595208 >> v2:172.21.15.154:6812/254845342 conn(0x7f0988037840 0x7f0988039d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:12:45.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.050+0000 7f0983fff640 1 -- 172.21.15.42:0/4595208 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(706..706 src has 681..706) ==== 5040+0+0 (secure 0 0 0) 0x7f09900481a0 con 0x7f099c0a37f0 2024-09-16T04:12:45.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.050+0000 7f0983fff640 10 client.1034550.objecter ms_dispatch 0x7f099c000ba0 osd_map(706..706 src has 681..706) 2024-09-16T04:12:45.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.050+0000 7f0983fff640 3 client.1034550.objecter handle_osd_map got epochs [706,706] > 0 2024-09-16T04:12:45.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.050+0000 7f0983fff640 3 client.1034550.objecter handle_osd_map decoding full epoch 706 2024-09-16T04:12:45.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.050+0000 7f099b7fe640 10 monclient(hunting): handle_auth_done global_id 1084139 payload 824 2024-09-16T04:12:45.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.050+0000 7f099b7fe640 1 -- 172.21.15.42:0/4595208 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f099c0a3230 con 0x7f099c130ef0 2024-09-16T04:12:45.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.050+0000 7f0983fff640 20 client.1034550.objecter dump_active .. 0 homeless 2024-09-16T04:12:45.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.050+0000 7f099bfff640 1 --2- 172.21.15.42:0/4595208 >> v2:172.21.15.154:6812/254845342 conn(0x7f0988037840 0x7f0988039d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:12:45.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.050+0000 7f099bfff640 10 monclient: get_auth_request con 0x7f0988037840 auth_method 0 2024-09-16T04:12:45.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.050+0000 7f099b7fe640 1 --2- 172.21.15.42:0/4595208 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f099c130ef0 0x7f099c0a2010 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f099c09bbd0 tx=0x7f098c019100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:12:45.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.050+0000 7f0983fff640 10 client.1034550.objecter ms_handle_connect 0x7f099c130ef0 2024-09-16T04:12:45.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.050+0000 7f0983fff640 10 client.1034550.objecter resend_mon_ops 2024-09-16T04:12:45.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.050+0000 7f0983fff640 1 -- 172.21.15.42:0/4595208 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f098c0bd030 con 0x7f099c130ef0 2024-09-16T04:12:45.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.050+0000 7f0983fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:45.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.050+0000 7f0983fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:12:45.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.050+0000 7f0983fff640 10 monclient: dump: 2024-09-16T04:12:45.075 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:12:45.075 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:12:45.075 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:45.075 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:45.075 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:45.075 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:12:45.075 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:12:45.076 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:12:45.076 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:12:45.076 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:12:45.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.050+0000 7f0983fff640 1 -- 172.21.15.42:0/4595208 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f098c0a9040 con 0x7f099c130ef0 2024-09-16T04:12:45.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.050+0000 7f0983fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:12:45.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.050+0000 7f0983fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:12:45.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.050+0000 7f0983fff640 1 -- 172.21.15.42:0/4595208 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f099c130ef0 msgr2=0x7f099c0a2010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:45.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.050+0000 7f0983fff640 1 --2- 172.21.15.42:0/4595208 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f099c130ef0 0x7f099c0a2010 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f099c09bbd0 tx=0x7f098c019100 comp rx=0 tx=0).stop 2024-09-16T04:12:45.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.050+0000 7f0983fff640 1 -- 172.21.15.42:0/4595208 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f099c130ef0 msgr2=0x7f099c0a2010 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:12:45.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.050+0000 7f0983fff640 1 --2- 172.21.15.42:0/4595208 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f099c130ef0 0x7f099c0a2010 secure :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f099c09bbd0 tx=0x7f098c019100 comp rx=0 tx=0).stop 2024-09-16T04:12:45.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.050+0000 7f099bfff640 1 --2- 172.21.15.42:0/4595208 >> v2:172.21.15.154:6812/254845342 conn(0x7f0988037840 0x7f0988039d00 secure :-1 s=READY pgs=3973 cs=0 l=1 rev1=1 crypto rx=0x7f0994001570 tx=0x7f0994001a20 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:12:45.077 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.050+0000 7f0983fff640 10 client.1034550.objecter ms_handle_connect 0x7f0988037840 2024-09-16T04:12:45.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.058+0000 7f09a20da640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:12:45.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.058+0000 7f09a20da640 1 --2- 172.21.15.42:0/4595208 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f099c0a2550 0x7f099c09d770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:12:45.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.058+0000 7f09a20da640 10 monclient(hunting): start opening mon connection 2024-09-16T04:12:45.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.058+0000 7f09a20da640 1 -- 172.21.15.42:0/4595208 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f099c139370 con 0x7f099c0a2550 2024-09-16T04:12:45.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.058+0000 7f099b7fe640 1 --2- 172.21.15.42:0/4595208 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f099c0a2550 0x7f099c09d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:12:45.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.058+0000 7f099b7fe640 10 monclient: get_auth_request con 0x7f099c0a2550 auth_method 0 2024-09-16T04:12:45.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.058+0000 7f099b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:12:45.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.058+0000 7f099b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:45.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.058+0000 7f099b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:45.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.058+0000 7f099b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:12:45.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.058+0000 7f099b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:12:45.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.058+0000 7f099b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:12:45.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.058+0000 7f099b7fe640 10 monclient(hunting): handle_auth_done global_id 1084142 payload 824 2024-09-16T04:12:45.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.058+0000 7f099b7fe640 1 -- 172.21.15.42:0/4595208 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f099c068e40 con 0x7f099c0a2550 2024-09-16T04:12:45.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.058+0000 7f0983fff640 10 client.1034550.objecter ms_handle_connect 0x7f099c0a2550 2024-09-16T04:12:45.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.058+0000 7f0983fff640 10 client.1034550.objecter resend_mon_ops 2024-09-16T04:12:45.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.058+0000 7f099b7fe640 1 --2- 172.21.15.42:0/4595208 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f099c0a2550 0x7f099c09d770 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f098c01c3a0 tx=0x7f098c009190 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:12:45.080 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.058+0000 7f0983fff640 1 -- 172.21.15.42:0/4595208 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f098c0bd040 con 0x7f099c0a2550 2024-09-16T04:12:45.080 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.058+0000 7f0983fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:45.080 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.058+0000 7f0983fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:12:45.080 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.058+0000 7f0983fff640 10 monclient: dump: 2024-09-16T04:12:45.080 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:12:45.080 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:12:45.080 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:45.080 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:45.080 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:45.080 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:12:45.080 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:12:45.080 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:12:45.081 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:12:45.081 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:12:45.081 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.058+0000 7f0983fff640 1 -- 172.21.15.42:0/4595208 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4545 (secure 0 0 0) 0x7f098c0b3040 con 0x7f099c0a2550 2024-09-16T04:12:45.081 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.058+0000 7f0983fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:12:45.081 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.058+0000 7f0983fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:12:45.081 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.058+0000 7f0983fff640 1 -- 172.21.15.42:0/4595208 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f099c0a2550 msgr2=0x7f099c09d770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:45.081 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.058+0000 7f0983fff640 1 --2- 172.21.15.42:0/4595208 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f099c0a2550 0x7f099c09d770 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f098c01c3a0 tx=0x7f098c009190 comp rx=0 tx=0).stop 2024-09-16T04:12:45.081 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.058+0000 7f0983fff640 1 -- 172.21.15.42:0/4595208 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f099c0a2550 msgr2=0x7f099c09d770 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:12:45.081 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.058+0000 7f0983fff640 1 --2- 172.21.15.42:0/4595208 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f099c0a2550 0x7f099c09d770 secure :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f098c01c3a0 tx=0x7f098c009190 comp rx=0 tx=0).stop 2024-09-16T04:12:45.081 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T04:12:45.081 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:12:45.081 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:12:45.082 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T04:12:45.082 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 276, 2024-09-16T04:12:45.082 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 17959, 2024-09-16T04:12:45.082 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T04:12:45.082 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T04:12:45.082 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T04:12:45.082 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T04:12:45.082 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:12:45.082 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 3, 2024-09-16T04:12:45.082 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:12:45.082 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T04:12:45.082 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T04:12:45.082 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:12:45.083 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:12:45.083 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:12:45.083 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:12:45.083 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:12:45.083 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:12:45.083 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:12:45.083 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:12:45.083 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:12:45.083 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:12:45.083 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:12:45.083 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:12:45.083 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:12:45.083 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T04:12:45.084 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:12:45.084 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:12:45.084 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:12:45.084 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:12:45.084 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:12:45.084 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:12:45.084 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:12:45.084 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:12:45.084 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:12:45.084 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:12:45.084 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:12:45.084 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:12:45.084 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:12:45.085 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T04:12:45.085 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T04:12:45.085 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T04:12:45.085 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T04:12:45.085 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T04:12:45.085 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T04:12:45.085 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:12:45.085 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:12:45.085 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T04:12:45.085 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T04:12:45.085 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T04:12:45.086 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T04:12:45.086 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T04:12:45.086 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T04:12:45.086 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T04:12:45.086 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:12:45.086 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T04:12:45.086 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:12:45.086 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:12:45.086 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:12:45.086 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:12:45.086 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:12:45.086 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:12:45.086 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:12:45.086 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:12:45.087 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:12:45.087 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:12:45.087 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:12:45.087 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:12:45.087 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T04:12:45.087 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:12:45.087 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T04:12:45.087 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:12:45.087 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:12:45.087 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:12:45.087 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:12:45.087 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:12:45.087 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:12:45.087 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:12:45.087 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T04:12:45.088 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:12:45.088 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:12:45.088 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:12:45.088 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:12:45.088 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T04:12:45.088 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:12:45.088 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:12:45.088 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:12:45.088 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:12:45.088 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T04:12:45.088 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T04:12:45.088 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:12:45.089 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:12:45.089 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:12:45.089 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:12:45.089 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:12:45.089 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:12:45.089 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:12:45.089 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:12:45.089 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:12:45.089 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:12:45.089 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:12:45.089 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T04:12:45.089 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:12:45.089 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:12:45.090 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:12:45.090 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:12:45.090 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T04:12:45.090 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:12:45.090 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:12:45.090 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:12:45.090 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:12:45.090 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T04:12:45.090 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T04:12:45.090 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:12:45.090 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:12:45.091 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:12:45.091 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:12:45.091 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:12:45.091 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:12:45.091 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:12:45.091 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:12:45.091 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:12:45.091 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:12:45.091 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:12:45.091 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T04:12:45.091 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:12:45.091 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:12:45.092 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:12:45.092 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:12:45.092 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T04:12:45.092 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:12:45.092 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:12:45.092 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:12:45.092 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:12:45.092 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T04:12:45.092 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T04:12:45.092 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:12:45.092 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:12:45.092 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:12:45.093 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:12:45.093 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:12:45.093 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:12:45.093 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T04:12:45.093 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T04:12:45.093 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:12:45.093 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:12:45.093 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:12:45.093 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:12:45.093 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:12:45.093 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:12:45.093 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T04:12:45.093 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:12:45.093 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:12:45.094 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:12:45.094 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:12:45.094 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:12:45.094 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:12:45.094 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:12:45.094 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T04:12:45.094 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T04:12:45.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.058+0000 7f09a20da640 10 librados: watch_flush enter 2024-09-16T04:12:45.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.058+0000 7f09a20da640 10 librados: watch_flush exit 2024-09-16T04:12:45.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.058+0000 7f09a20da640 20 client.1034550.objecter shutdown clearing up homeless session... 2024-09-16T04:12:45.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.058+0000 7f09a20da640 10 client.1034550.objecter successfully canceled tick 2024-09-16T04:12:45.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.058+0000 7f09a20da640 1 -- 172.21.15.42:0/4595208 >> v2:172.21.15.154:6812/254845342 conn(0x7f0988037840 msgr2=0x7f0988039d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:45.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.058+0000 7f09a20da640 1 --2- 172.21.15.42:0/4595208 >> v2:172.21.15.154:6812/254845342 conn(0x7f0988037840 0x7f0988039d00 secure :-1 s=READY pgs=3973 cs=0 l=1 rev1=1 crypto rx=0x7f0994001570 tx=0x7f0994001a20 comp rx=0 tx=0).stop 2024-09-16T04:12:45.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.058+0000 7f09a20da640 10 monclient: shutdown 2024-09-16T04:12:45.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.058+0000 7f09a20da640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:12:45.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.058+0000 7f09a20da640 1 -- 172.21.15.42:0/4595208 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f099c0a37f0 msgr2=0x7f099c0a1590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:45.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.058+0000 7f09a20da640 1 --2- 172.21.15.42:0/4595208 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f099c0a37f0 0x7f099c0a1590 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f099000c420 tx=0x7f099000c8e0 comp rx=0 tx=0).stop 2024-09-16T04:12:45.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.058+0000 7f099bfff640 1 -- 172.21.15.42:0/4595208 reap_dead start 2024-09-16T04:12:45.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.058+0000 7f099bfff640 1 -- 172.21.15.42:0/4595208 reap_dead start 2024-09-16T04:12:45.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.058+0000 7f09a20da640 1 -- 172.21.15.42:0/4595208 shutdown_connections 2024-09-16T04:12:45.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.058+0000 7f09a20da640 1 -- 172.21.15.42:0/4595208 >> 172.21.15.42:0/4595208 conn(0x7f099c0a9fb0 msgr2=0x7f099c0a7ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:12:45.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.058+0000 7f09a20da640 1 -- 172.21.15.42:0/4595208 shutdown_connections 2024-09-16T04:12:45.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.058+0000 7f09a20da640 1 -- 172.21.15.42:0/4595208 wait complete. 2024-09-16T04:12:45.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.058+0000 7f09a20da640 1 librados: shutdown 2024-09-16T04:12:45.097 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:12:45.097 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T04:12:45.097 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T04:12:45.214 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.210+0000 7fd9d9ab8640 10 monclient: get_monmap_and_config 2024-09-16T04:12:45.214 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.210+0000 7fd9d9ab8640 10 monclient: build_initial_monmap 2024-09-16T04:12:45.214 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.210+0000 7fd9d9ab8640 10 monclient: monmap: 2024-09-16T04:12:45.215 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:12:45.215 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:12:45.215 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:12:45.212843+0000 2024-09-16T04:12:45.215 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:12:45.212843+0000 2024-09-16T04:12:45.215 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:12:45.215 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:12:45.215 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:12:45.216 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:12:45.216 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:12:45.216 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:12:45.216 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:12:45.216 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:12:45.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.210+0000 7fd9d9ab8640 1 Processor -- start 2024-09-16T04:12:45.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.210+0000 7fd9d9ab8640 1 -- start start 2024-09-16T04:12:45.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.210+0000 7fd9d9ab8640 10 monclient: init 2024-09-16T04:12:45.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.210+0000 7fd9d9ab8640 10 monclient: _reopen_session rank -1 2024-09-16T04:12:45.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.210+0000 7fd9d9ab8640 10 monclient: _add_conns ranks=[2,1,3,0,4] 2024-09-16T04:12:45.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.210+0000 7fd9d9ab8640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd9d40a39b0 0x7fd9d4133b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:12:45.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.210+0000 7fd9d9ab8640 10 monclient(hunting): picked mon.noname-b con 0x7fd9d40a39b0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:12:45.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.210+0000 7fd9d9ab8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd9d40a3d90 addr v1:172.21.15.154:6789/0 2024-09-16T04:12:45.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.210+0000 7fd9d9ab8640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd9d4137710 0x7fd9d413f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:12:45.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.210+0000 7fd9d9ab8640 10 monclient(hunting): picked mon.noname-a con 0x7fd9d4137710 addr v2:172.21.15.42:3300/0 2024-09-16T04:12:45.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.210+0000 7fd9d9ab8640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd9d409c140 con 0x7fd9d40a3d90 2024-09-16T04:12:45.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.210+0000 7fd9d9ab8640 10 monclient(hunting): start opening mon connection 2024-09-16T04:12:45.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.210+0000 7fd9d9ab8640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7fd9d409c420 con 0x7fd9d4137710 2024-09-16T04:12:45.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.210+0000 7fd9d9ab8640 10 monclient(hunting): start opening mon connection 2024-09-16T04:12:45.220 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.210+0000 7fd9d9ab8640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fd9d406dd60 con 0x7fd9d40a39b0 2024-09-16T04:12:45.220 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.210+0000 7fd9d9ab8640 10 monclient(hunting): _renew_subs 2024-09-16T04:12:45.220 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.210+0000 7fd9d9ab8640 10 monclient(hunting): authenticate will time out at 2812.182426s 2024-09-16T04:12:45.220 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.210+0000 7fd9d37fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd9d40a39b0 0x7fd9d4133b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:12:45.220 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.210+0000 7fd9d3fff640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd9d4137710 0x7fd9d413f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:12:45.220 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.210+0000 7fd9d3fff640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd9d4137710 0x7fd9d413f3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:46786/0 (socket says 172.21.15.42:46786) 2024-09-16T04:12:45.220 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.210+0000 7fd9d3fff640 1 -- 172.21.15.42:0/1505182728 learned_addr learned my addr 172.21.15.42:0/1505182728 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:12:45.221 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.210+0000 7fd9d3fff640 10 monclient(hunting): get_auth_request con 0x7fd9d4137710 auth_method 0 2024-09-16T04:12:45.221 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.210+0000 7fd9d37fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd9d40a39b0 0x7fd9d4133b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:36798/0 (socket says 172.21.15.42:36798) 2024-09-16T04:12:45.221 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.210+0000 7fd9d3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:12:45.221 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.210+0000 7fd9d3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:45.221 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.210+0000 7fd9d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:45.221 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.210+0000 7fd9d37fe640 10 monclient(hunting): get_auth_request con 0x7fd9d40a39b0 auth_method 0 2024-09-16T04:12:45.221 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.210+0000 7fd9d37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:12:45.221 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.210+0000 7fd9d37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:45.221 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.210+0000 7fd9d37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:45.222 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.210+0000 7fd9d3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:12:45.222 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.210+0000 7fd9d3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:12:45.222 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.210+0000 7fd9d3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:12:45.222 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.210+0000 7fd9d37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:12:45.222 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.210+0000 7fd9d37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:12:45.222 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.210+0000 7fd9d37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:12:45.222 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.210+0000 7fd9d37fe640 10 monclient(hunting): handle_auth_done global_id 1084145 payload 274 2024-09-16T04:12:45.222 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.210+0000 7fd9d37fe640 1 -- 172.21.15.42:0/1505182728 >> v2:172.21.15.42:3300/0 conn(0x7fd9d4137710 msgr2=0x7fd9d413f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:12:45.222 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.210+0000 7fd9d37fe640 1 --2- 172.21.15.42:0/1505182728 >> v2:172.21.15.42:3300/0 conn(0x7fd9d4137710 0x7fd9d413f3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:12:45.222 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.210+0000 7fd9d37fe640 1 -- 172.21.15.42:0/1505182728 >> v1:172.21.15.154:6789/0 conn(0x7fd9d40a3d90 legacy=0x7fd9d4137000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:45.222 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.210+0000 7fd9d37fe640 10 monclient: _finish_hunting 0 2024-09-16T04:12:45.222 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.210+0000 7fd9d37fe640 1 monclient: found mon.noname-b 2024-09-16T04:12:45.223 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.210+0000 7fd9d37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:12:45.223 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.210+0000 7fd9d37fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T04:12:45.223 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.210+0000 7fd9d37fe640 1 -- 172.21.15.42:0/1505182728 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9d40a5e20 con 0x7fd9d40a39b0 2024-09-16T04:12:45.223 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.210+0000 7fd9d3fff640 1 --2- 172.21.15.42:0/1505182728 >> v2:172.21.15.42:3300/0 conn(0x7fd9d4137710 0x7fd9d413f3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:12:45.230 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.214+0000 7fd9d37fe640 1 --2- 172.21.15.42:0/1505182728 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd9d40a39b0 0x7fd9d4133b80 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fd9bc0097c0 tx=0x7fd9bc01c360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=85a023b0128fb57c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:12:45.230 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.214+0000 7fd9d27fc640 1 -- 172.21.15.42:0/1505182728 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9bc0b0070 con 0x7fd9d40a39b0 2024-09-16T04:12:45.230 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.214+0000 7fd9d27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:45.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.214+0000 7fd9d27fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:12:45.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.214+0000 7fd9d27fc640 10 monclient: dump: 2024-09-16T04:12:45.231 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:12:45.231 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:12:45.231 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:45.231 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:45.231 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:45.232 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:12:45.232 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:12:45.232 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:12:45.232 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:12:45.232 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:12:45.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.214+0000 7fd9d27fc640 10 monclient: _finish_auth 0 2024-09-16T04:12:45.233 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.214+0000 7fd9d27fc640 10 monclient: _check_auth_tickets 2024-09-16T04:12:45.233 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.214+0000 7fd9d27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:12:45.233 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.214+0000 7fd9d27fc640 1 -- 172.21.15.42:0/1505182728 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd9bc0ab600 con 0x7fd9d40a39b0 2024-09-16T04:12:45.233 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.214+0000 7fd9d9ab8640 5 monclient: authenticate success, global_id 1084145 2024-09-16T04:12:45.233 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.214+0000 7fd9d9ab8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:12:45.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.214+0000 7fd9d27fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:12:45.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.214+0000 7fd9d9ab8640 10 monclient: get_monmap_and_config success 2024-09-16T04:12:45.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.214+0000 7fd9d9ab8640 4 set_mon_vals no callback set 2024-09-16T04:12:45.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.214+0000 7fd9d27fc640 1 -- 172.21.15.42:0/1505182728 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9bc0ab9d0 con 0x7fd9d40a39b0 2024-09-16T04:12:45.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.214+0000 7fd9d27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:45.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.214+0000 7fd9d27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:12:45.235 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.214+0000 7fd9d27fc640 10 monclient: dump: 2024-09-16T04:12:45.235 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:12:45.235 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:12:45.235 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:45.235 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:45.235 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:45.236 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:12:45.236 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:12:45.236 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:12:45.236 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:12:45.236 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:12:45.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.214+0000 7fd9d9ab8640 10 monclient: shutdown 2024-09-16T04:12:45.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.214+0000 7fd9d9ab8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:12:45.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.214+0000 7fd9d9ab8640 1 -- 172.21.15.42:0/1505182728 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd9d40a39b0 msgr2=0x7fd9d4133b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:45.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.214+0000 7fd9d9ab8640 1 --2- 172.21.15.42:0/1505182728 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd9d40a39b0 0x7fd9d4133b80 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fd9bc0097c0 tx=0x7fd9bc01c360 comp rx=0 tx=0).stop 2024-09-16T04:12:45.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.214+0000 7fd9d9ab8640 1 -- 172.21.15.42:0/1505182728 shutdown_connections 2024-09-16T04:12:45.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.214+0000 7fd9d9ab8640 1 --2- 172.21.15.42:0/1505182728 >> v2:172.21.15.42:3300/0 conn(0x7fd9d4137710 0x7fd9d413f3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:12:45.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.214+0000 7fd9d9ab8640 1 --2- 172.21.15.42:0/1505182728 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd9d40a39b0 0x7fd9d4133b80 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:12:45.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.214+0000 7fd9d9ab8640 1 -- 172.21.15.42:0/1505182728 >> 172.21.15.42:0/1505182728 conn(0x7fd9d40a9fb0 msgr2=0x7fd9d40aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:12:45.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.214+0000 7fd9d9ab8640 1 -- 172.21.15.42:0/1505182728 shutdown_connections 2024-09-16T04:12:45.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.214+0000 7fd9d9ab8640 1 -- 172.21.15.42:0/1505182728 wait complete. 2024-09-16T04:12:45.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.218+0000 7fd9d9ab8640 10 monclient: build_initial_monmap 2024-09-16T04:12:45.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.218+0000 7fd9d9ab8640 10 monclient: monmap: 2024-09-16T04:12:45.239 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:12:45.239 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:12:45.239 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:12:45.239 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:12:45.239 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:12:45.240 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:12:45.240 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:12:45.240 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:12:45.240 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:12:45.240 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:12:45.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.218+0000 7fd9d9ab8640 1 librados: starting msgr at 2024-09-16T04:12:45.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.218+0000 7fd9d9ab8640 1 librados: starting objecter 2024-09-16T04:12:45.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.218+0000 7fd9d9ab8640 1 Processor -- start 2024-09-16T04:12:45.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.218+0000 7fd9d9ab8640 1 -- start start 2024-09-16T04:12:45.241 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.218+0000 7fd9d9ab8640 1 librados: setting wanted keys 2024-09-16T04:12:45.242 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.218+0000 7fd9d9ab8640 1 librados: calling monclient init 2024-09-16T04:12:45.242 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.218+0000 7fd9d9ab8640 10 monclient: init 2024-09-16T04:12:45.242 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.218+0000 7fd9d9ab8640 10 monclient: _reopen_session rank -1 2024-09-16T04:12:45.242 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.218+0000 7fd9d9ab8640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:12:45.242 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.218+0000 7fd9d9ab8640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd9d40a39b0 0x7fd9d41cfba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:12:45.242 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.218+0000 7fd9d9ab8640 10 monclient(hunting): picked mon.noname-a con 0x7fd9d40a39b0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:12:45.243 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.218+0000 7fd9d9ab8640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd9d40a3d90 0x7fd9d41d00e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:12:45.243 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.218+0000 7fd9d9ab8640 10 monclient(hunting): picked mon.noname-c con 0x7fd9d40a3d90 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:12:45.243 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.218+0000 7fd9d9ab8640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd9d4137710 0x7fd9d41d0620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:12:45.243 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.218+0000 7fd9d9ab8640 10 monclient(hunting): picked mon.noname-b con 0x7fd9d4137710 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:12:45.243 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.218+0000 7fd9d9ab8640 10 monclient(hunting): start opening mon connection 2024-09-16T04:12:45.243 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.218+0000 7fd9d9ab8640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fd9d4142310 con 0x7fd9d40a39b0 2024-09-16T04:12:45.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.218+0000 7fd9d9ab8640 10 monclient(hunting): start opening mon connection 2024-09-16T04:12:45.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.218+0000 7fd9d9ab8640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd9d4142490 con 0x7fd9d4137710 2024-09-16T04:12:45.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.218+0000 7fd9d9ab8640 10 monclient(hunting): start opening mon connection 2024-09-16T04:12:45.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.218+0000 7fd9d9ab8640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fd9d40a2bc0 con 0x7fd9d40a3d90 2024-09-16T04:12:45.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.218+0000 7fd9d9ab8640 10 monclient(hunting): _renew_subs 2024-09-16T04:12:45.244 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.218+0000 7fd9d9ab8640 10 monclient(hunting): authenticate will time out at 2812.189267s 2024-09-16T04:12:45.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.218+0000 7fd9d37fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd9d40a39b0 0x7fd9d41cfba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:12:45.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.218+0000 7fd9d2ffd640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd9d40a3d90 0x7fd9d41d00e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:12:45.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.218+0000 7fd9d2ffd640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd9d40a3d90 0x7fd9d41d00e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:36804/0 (socket says 172.21.15.42:36804) 2024-09-16T04:12:45.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.218+0000 7fd9d37fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd9d40a39b0 0x7fd9d41cfba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:46794/0 (socket says 172.21.15.42:46794) 2024-09-16T04:12:45.245 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.218+0000 7fd9d2ffd640 1 -- 172.21.15.42:0/4158887530 learned_addr learned my addr 172.21.15.42:0/4158887530 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:12:45.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.218+0000 7fd9d2ffd640 10 monclient(hunting): get_auth_request con 0x7fd9d40a3d90 auth_method 0 2024-09-16T04:12:45.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.218+0000 7fd9d2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:12:45.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.218+0000 7fd9d2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:45.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.218+0000 7fd9d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:45.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.218+0000 7fd9d37fe640 10 monclient(hunting): get_auth_request con 0x7fd9d40a39b0 auth_method 0 2024-09-16T04:12:45.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.218+0000 7fd9d37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:12:45.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.218+0000 7fd9d37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:45.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.218+0000 7fd9d37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:45.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.218+0000 7fd9d2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:12:45.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.218+0000 7fd9d2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:12:45.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.218+0000 7fd9d2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:12:45.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.218+0000 7fd9d37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:12:45.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.218+0000 7fd9d37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:12:45.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.218+0000 7fd9d37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:12:45.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.218+0000 7fd9d3fff640 1 --2- 172.21.15.42:0/4158887530 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd9d4137710 0x7fd9d41d0620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:12:45.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.218+0000 7fd9d37fe640 10 monclient(hunting): handle_auth_done global_id 1034556 payload 824 2024-09-16T04:12:45.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.218+0000 7fd9d37fe640 1 -- 172.21.15.42:0/4158887530 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd9d40a3d90 msgr2=0x7fd9d41d00e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:45.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.218+0000 7fd9d37fe640 1 --2- 172.21.15.42:0/4158887530 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd9d40a3d90 0x7fd9d41d00e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:12:45.249 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.218+0000 7fd9d37fe640 1 -- 172.21.15.42:0/4158887530 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd9d4137710 msgr2=0x7fd9d41d0620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:45.249 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.218+0000 7fd9d37fe640 1 --2- 172.21.15.42:0/4158887530 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd9d4137710 0x7fd9d41d0620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:12:45.249 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.218+0000 7fd9d37fe640 10 monclient: _finish_hunting 0 2024-09-16T04:12:45.249 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.218+0000 7fd9d37fe640 1 monclient: found mon.noname-a 2024-09-16T04:12:45.249 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.218+0000 7fd9d37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:12:45.250 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.218+0000 7fd9d37fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:12:45.250 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.218+0000 7fd9d37fe640 1 -- 172.21.15.42:0/4158887530 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9d413f380 con 0x7fd9d40a39b0 2024-09-16T04:12:45.250 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.218+0000 7fd9d3fff640 10 monclient: get_auth_request con 0x7fd9d4137710 auth_method 0 2024-09-16T04:12:45.250 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.218+0000 7fd9d2ffd640 1 --2- 172.21.15.42:0/4158887530 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd9d40a3d90 0x7fd9d41d00e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:12:45.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.218+0000 7fd9d3fff640 1 --2- 172.21.15.42:0/4158887530 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd9d4137710 0x7fd9d41d0620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T04:12:45.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.218+0000 7fd9d37fe640 1 --2- 172.21.15.42:0/4158887530 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd9d40a39b0 0x7fd9d41cfba0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fd9bc0ab000 tx=0x7fd9bc019100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:12:45.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.218+0000 7fd9cbfff640 10 client.?.objecter ms_handle_connect 0x7fd9d40a39b0 2024-09-16T04:12:45.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.218+0000 7fd9cbfff640 10 client.?.objecter resend_mon_ops 2024-09-16T04:12:45.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.218+0000 7fd9cbfff640 1 -- 172.21.15.42:0/4158887530 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9bc0b8020 con 0x7fd9d40a39b0 2024-09-16T04:12:45.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.218+0000 7fd9cbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:45.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.218+0000 7fd9cbfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:12:45.252 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.218+0000 7fd9cbfff640 10 monclient: dump: 2024-09-16T04:12:45.253 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:12:45.253 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:12:45.253 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:45.253 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:45.253 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:45.253 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:12:45.253 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:12:45.253 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:12:45.253 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:12:45.253 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:12:45.253 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.218+0000 7fd9cbfff640 10 monclient: _finish_auth 0 2024-09-16T04:12:45.253 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.218+0000 7fd9cbfff640 10 monclient: _check_auth_tickets 2024-09-16T04:12:45.253 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.218+0000 7fd9cbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:12:45.254 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.218+0000 7fd9d9ab8640 5 monclient: authenticate success, global_id 1034556 2024-09-16T04:12:45.254 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.218+0000 7fd9cbfff640 1 -- 172.21.15.42:0/4158887530 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd9bc0b4410 con 0x7fd9d40a39b0 2024-09-16T04:12:45.254 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.218+0000 7fd9d9ab8640 10 monclient: _renew_subs 2024-09-16T04:12:45.254 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.218+0000 7fd9d9ab8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:12:45.254 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.218+0000 7fd9d9ab8640 1 -- 172.21.15.42:0/4158887530 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9d406f680 con 0x7fd9d40a39b0 2024-09-16T04:12:45.254 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.218+0000 7fd9cbfff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:12:45.254 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.218+0000 7fd9cbfff640 1 -- 172.21.15.42:0/4158887530 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9bc0b0040 con 0x7fd9d40a39b0 2024-09-16T04:12:45.254 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.218+0000 7fd9cbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:45.254 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.218+0000 7fd9cbfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:12:45.254 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.218+0000 7fd9cbfff640 10 monclient: dump: 2024-09-16T04:12:45.254 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:12:45.254 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:12:45.254 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:45.255 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:45.255 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:45.255 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:12:45.255 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:12:45.255 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:12:45.255 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:12:45.255 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:12:45.255 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.218+0000 7fd9d17fa640 4 set_mon_vals no callback set 2024-09-16T04:12:45.255 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.218+0000 7fd9d9ab8640 10 client.1034556.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:12:45.255 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.218+0000 7fd9d9ab8640 10 monclient: _renew_subs 2024-09-16T04:12:45.255 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.218+0000 7fd9d9ab8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:12:45.256 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.218+0000 7fd9d9ab8640 1 -- 172.21.15.42:0/4158887530 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9d40a28e0 con 0x7fd9d40a39b0 2024-09-16T04:12:45.256 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.218+0000 7fd9d9ab8640 1 librados: init done 2024-09-16T04:12:45.256 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.218+0000 7fd9d9ab8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:12:45.256 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.218+0000 7fd9d9ab8640 1 --2- 172.21.15.42:0/4158887530 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd9d40a3340 0x7fd9d412d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:12:45.256 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.218+0000 7fd9d9ab8640 10 monclient(hunting): start opening mon connection 2024-09-16T04:12:45.256 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.218+0000 7fd9d9ab8640 1 -- 172.21.15.42:0/4158887530 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd9d41342d0 con 0x7fd9d40a3340 2024-09-16T04:12:45.256 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.218+0000 7fd9cbfff640 1 -- 172.21.15.42:0/4158887530 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fd9bc0bb020 con 0x7fd9d40a39b0 2024-09-16T04:12:45.256 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.218+0000 7fd9cbfff640 1 --2- 172.21.15.42:0/4158887530 >> v2:172.21.15.154:6812/254845342 conn(0x7fd9b4037810 0x7fd9b4039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:12:45.256 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.218+0000 7fd9cbfff640 1 -- 172.21.15.42:0/4158887530 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(706..706 src has 681..706) ==== 5040+0+0 (secure 0 0 0) 0x7fd9bc0e2680 con 0x7fd9d40a39b0 2024-09-16T04:12:45.257 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.218+0000 7fd9cbfff640 10 client.1034556.objecter ms_dispatch 0x7fd9d4000ba0 osd_map(706..706 src has 681..706) 2024-09-16T04:12:45.257 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.218+0000 7fd9cbfff640 3 client.1034556.objecter handle_osd_map got epochs [706,706] > 0 2024-09-16T04:12:45.257 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.218+0000 7fd9cbfff640 3 client.1034556.objecter handle_osd_map decoding full epoch 706 2024-09-16T04:12:45.257 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.218+0000 7fd9cbfff640 20 client.1034556.objecter dump_active .. 0 homeless 2024-09-16T04:12:45.257 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.218+0000 7fd9d2ffd640 1 --2- 172.21.15.42:0/4158887530 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd9d40a3340 0x7fd9d412d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:12:45.257 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.218+0000 7fd9d2ffd640 10 monclient: get_auth_request con 0x7fd9d40a3340 auth_method 0 2024-09-16T04:12:45.257 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.218+0000 7fd9d2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:12:45.257 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.218+0000 7fd9d2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:45.257 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.218+0000 7fd9d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:45.257 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.218+0000 7fd9d3fff640 1 --2- 172.21.15.42:0/4158887530 >> v2:172.21.15.154:6812/254845342 conn(0x7fd9b4037810 0x7fd9b4039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:12:45.257 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.218+0000 7fd9d3fff640 10 monclient: get_auth_request con 0x7fd9b4037810 auth_method 0 2024-09-16T04:12:45.257 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.218+0000 7fd9d2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:12:45.258 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.218+0000 7fd9d2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:12:45.258 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.218+0000 7fd9d2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:12:45.258 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.222+0000 7fd9d2ffd640 10 monclient(hunting): handle_auth_done global_id 1074211 payload 824 2024-09-16T04:12:45.258 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.222+0000 7fd9d2ffd640 1 -- 172.21.15.42:0/4158887530 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd9d4133b80 con 0x7fd9d40a3340 2024-09-16T04:12:45.258 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.222+0000 7fd9d3fff640 1 --2- 172.21.15.42:0/4158887530 >> v2:172.21.15.154:6812/254845342 conn(0x7fd9b4037810 0x7fd9b4039cd0 secure :-1 s=READY pgs=3974 cs=0 l=1 rev1=1 crypto rx=0x7fd9cc009240 tx=0x7fd9cc00b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:12:45.258 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.222+0000 7fd9cbfff640 10 client.1034556.objecter ms_handle_connect 0x7fd9b4037810 2024-09-16T04:12:45.258 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.222+0000 7fd9cbfff640 10 client.1034556.objecter ms_handle_connect 0x7fd9d40a3340 2024-09-16T04:12:45.258 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.222+0000 7fd9cbfff640 10 client.1034556.objecter resend_mon_ops 2024-09-16T04:12:45.258 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.222+0000 7fd9d2ffd640 1 --2- 172.21.15.42:0/4158887530 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd9d40a3340 0x7fd9d412d2d0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fd9d40a03f0 tx=0x7fd9c400ff70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:12:45.258 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.222+0000 7fd9cbfff640 1 -- 172.21.15.42:0/4158887530 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9c40052e0 con 0x7fd9d40a3340 2024-09-16T04:12:45.258 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.222+0000 7fd9cbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:45.258 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.222+0000 7fd9cbfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:12:45.258 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.222+0000 7fd9cbfff640 10 monclient: dump: 2024-09-16T04:12:45.258 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:12:45.259 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:12:45.259 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:45.259 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:45.259 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:45.259 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:12:45.259 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:12:45.259 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:12:45.259 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:12:45.259 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:12:45.259 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.222+0000 7fd9cbfff640 1 -- 172.21.15.42:0/4158887530 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fd9c4018070 con 0x7fd9d40a3340 2024-09-16T04:12:45.259 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.222+0000 7fd9cbfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:12:45.259 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.222+0000 7fd9cbfff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:12:45.259 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.222+0000 7fd9cbfff640 1 -- 172.21.15.42:0/4158887530 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd9d40a3340 msgr2=0x7fd9d412d2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:45.259 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.222+0000 7fd9cbfff640 1 --2- 172.21.15.42:0/4158887530 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd9d40a3340 0x7fd9d412d2d0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fd9d40a03f0 tx=0x7fd9c400ff70 comp rx=0 tx=0).stop 2024-09-16T04:12:45.260 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.222+0000 7fd9cbfff640 1 -- 172.21.15.42:0/4158887530 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd9d40a3340 msgr2=0x7fd9d412d2d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:12:45.260 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.222+0000 7fd9cbfff640 1 --2- 172.21.15.42:0/4158887530 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd9d40a3340 0x7fd9d412d2d0 secure :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fd9d40a03f0 tx=0x7fd9c400ff70 comp rx=0 tx=0).stop 2024-09-16T04:12:45.262 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.226+0000 7fd9d9ab8640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:12:45.262 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.226+0000 7fd9d9ab8640 1 --2- 172.21.15.42:0/4158887530 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd9d412d810 0x7fd9d412dc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:12:45.262 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.226+0000 7fd9d9ab8640 10 monclient(hunting): start opening mon connection 2024-09-16T04:12:45.262 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.226+0000 7fd9d9ab8640 1 -- 172.21.15.42:0/4158887530 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd9d412e160 con 0x7fd9d412d810 2024-09-16T04:12:45.262 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.230+0000 7fd9d2ffd640 1 --2- 172.21.15.42:0/4158887530 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd9d412d810 0x7fd9d412dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:12:45.262 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.230+0000 7fd9d2ffd640 10 monclient: get_auth_request con 0x7fd9d412d810 auth_method 0 2024-09-16T04:12:45.263 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.230+0000 7fd9d2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:12:45.263 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.230+0000 7fd9d2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:45.263 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.230+0000 7fd9d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:45.263 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.230+0000 7fd9d2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:12:45.263 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.230+0000 7fd9d2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:12:45.263 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.230+0000 7fd9d2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:12:45.263 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.230+0000 7fd9d2ffd640 10 monclient(hunting): handle_auth_done global_id 1074214 payload 824 2024-09-16T04:12:45.263 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.230+0000 7fd9d2ffd640 1 -- 172.21.15.42:0/4158887530 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fd9d406dee0 con 0x7fd9d412d810 2024-09-16T04:12:45.263 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.230+0000 7fd9d2ffd640 1 --2- 172.21.15.42:0/4158887530 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd9d412d810 0x7fd9d412dc20 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fd9d40a02e0 tx=0x7fd9c4000ee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:12:45.263 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.230+0000 7fd9cbfff640 10 client.1034556.objecter ms_handle_connect 0x7fd9d412d810 2024-09-16T04:12:45.263 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.230+0000 7fd9cbfff640 10 client.1034556.objecter resend_mon_ops 2024-09-16T04:12:45.263 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.230+0000 7fd9cbfff640 1 -- 172.21.15.42:0/4158887530 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9c401ed30 con 0x7fd9d412d810 2024-09-16T04:12:45.263 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.230+0000 7fd9cbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:45.264 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.230+0000 7fd9cbfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:12:45.264 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.230+0000 7fd9cbfff640 10 monclient: dump: 2024-09-16T04:12:45.264 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:12:45.264 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:12:45.264 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:45.264 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:45.264 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:45.264 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:12:45.264 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:12:45.264 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:12:45.265 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:12:45.265 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:12:45.265 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.230+0000 7fd9cbfff640 1 -- 172.21.15.42:0/4158887530 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4712 (secure 0 0 0) 0x7fd9c4003600 con 0x7fd9d412d810 2024-09-16T04:12:45.265 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.230+0000 7fd9cbfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:12:45.265 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.230+0000 7fd9cbfff640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:12:45.265 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.230+0000 7fd9cbfff640 1 -- 172.21.15.42:0/4158887530 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd9d412d810 msgr2=0x7fd9d412dc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:45.265 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.230+0000 7fd9cbfff640 1 --2- 172.21.15.42:0/4158887530 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd9d412d810 0x7fd9d412dc20 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fd9d40a02e0 tx=0x7fd9c4000ee0 comp rx=0 tx=0).stop 2024-09-16T04:12:45.265 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.230+0000 7fd9cbfff640 1 -- 172.21.15.42:0/4158887530 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd9d412d810 msgr2=0x7fd9d412dc20 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:12:45.265 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.230+0000 7fd9cbfff640 1 --2- 172.21.15.42:0/4158887530 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd9d412d810 0x7fd9d412dc20 secure :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fd9d40a02e0 tx=0x7fd9c4000ee0 comp rx=0 tx=0).stop 2024-09-16T04:12:45.265 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T04:12:45.266 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:12:45.266 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:12:45.266 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T04:12:45.266 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 276, 2024-09-16T04:12:45.266 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 78401, 2024-09-16T04:12:45.266 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T04:12:45.266 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T04:12:45.266 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T04:12:45.266 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T04:12:45.267 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:12:45.267 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 3, 2024-09-16T04:12:45.267 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:12:45.267 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T04:12:45.267 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T04:12:45.267 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:12:45.267 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:12:45.267 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:12:45.267 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:12:45.267 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:12:45.267 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:12:45.267 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:12:45.268 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:12:45.268 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:12:45.268 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:12:45.268 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:12:45.268 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:12:45.268 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:12:45.268 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T04:12:45.268 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:12:45.268 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:12:45.268 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:12:45.268 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:12:45.269 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:12:45.269 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:12:45.269 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:12:45.269 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:12:45.269 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:12:45.269 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:12:45.269 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:12:45.269 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:12:45.269 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:12:45.269 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T04:12:45.269 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T04:12:45.269 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T04:12:45.269 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T04:12:45.270 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T04:12:45.270 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T04:12:45.270 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:12:45.270 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:12:45.270 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T04:12:45.270 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T04:12:45.270 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T04:12:45.270 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T04:12:45.270 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T04:12:45.270 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T04:12:45.270 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T04:12:45.270 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:12:45.270 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T04:12:45.270 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:12:45.271 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:12:45.271 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:12:45.271 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:12:45.271 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:12:45.271 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:12:45.271 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:12:45.271 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:12:45.271 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:12:45.271 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:12:45.271 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:12:45.271 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:12:45.271 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T04:12:45.271 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:12:45.272 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T04:12:45.272 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:12:45.272 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:12:45.272 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:12:45.272 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:12:45.272 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:12:45.272 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:12:45.272 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:12:45.272 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T04:12:45.272 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:12:45.272 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:12:45.272 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:12:45.273 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:12:45.273 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T04:12:45.273 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:12:45.273 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:12:45.273 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:12:45.273 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:12:45.273 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T04:12:45.273 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T04:12:45.273 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:12:45.273 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:12:45.273 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:12:45.274 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:12:45.274 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:12:45.274 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:12:45.274 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:12:45.274 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:12:45.274 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:12:45.274 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:12:45.274 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:12:45.274 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T04:12:45.274 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:12:45.274 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:12:45.274 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:12:45.274 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:12:45.275 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T04:12:45.275 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:12:45.275 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:12:45.275 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:12:45.275 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:12:45.275 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T04:12:45.275 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T04:12:45.275 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:12:45.275 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:12:45.275 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:12:45.275 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:12:45.275 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:12:45.275 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:12:45.276 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:12:45.276 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:12:45.276 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:12:45.276 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:12:45.276 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:12:45.276 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T04:12:45.276 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:12:45.276 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:12:45.276 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:12:45.276 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:12:45.276 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T04:12:45.276 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:12:45.276 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:12:45.277 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:12:45.277 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:12:45.277 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T04:12:45.277 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T04:12:45.277 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:12:45.277 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:12:45.277 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:12:45.277 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:12:45.277 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:12:45.277 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:12:45.277 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T04:12:45.278 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T04:12:45.278 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:12:45.278 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:12:45.278 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:12:45.278 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:12:45.278 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:12:45.278 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:12:45.278 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T04:12:45.278 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:12:45.278 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:12:45.278 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:12:45.278 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 2 2024-09-16T04:12:45.278 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:12:45.279 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:12:45.279 INFO:teuthology.orchestra.run.smithi042.stdout: "mgr": [ 2024-09-16T04:12:45.279 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:12:45.279 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:12:45.279 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:12:45.279 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:12:45.279 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:12:45.279 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:12:45.279 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:12:45.279 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T04:12:45.279 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T04:12:45.281 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.230+0000 7fd9d9ab8640 10 librados: watch_flush enter 2024-09-16T04:12:45.281 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.230+0000 7fd9d9ab8640 10 librados: watch_flush exit 2024-09-16T04:12:45.281 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.230+0000 7fd9d9ab8640 20 client.1034556.objecter shutdown clearing up homeless session... 2024-09-16T04:12:45.281 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.230+0000 7fd9d9ab8640 10 client.1034556.objecter successfully canceled tick 2024-09-16T04:12:45.281 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.230+0000 7fd9d9ab8640 1 -- 172.21.15.42:0/4158887530 >> v2:172.21.15.154:6812/254845342 conn(0x7fd9b4037810 msgr2=0x7fd9b4039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:45.281 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.230+0000 7fd9d9ab8640 1 --2- 172.21.15.42:0/4158887530 >> v2:172.21.15.154:6812/254845342 conn(0x7fd9b4037810 0x7fd9b4039cd0 secure :-1 s=READY pgs=3974 cs=0 l=1 rev1=1 crypto rx=0x7fd9cc009240 tx=0x7fd9cc00b040 comp rx=0 tx=0).stop 2024-09-16T04:12:45.281 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.230+0000 7fd9d9ab8640 10 monclient: shutdown 2024-09-16T04:12:45.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.230+0000 7fd9d9ab8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:12:45.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.230+0000 7fd9d9ab8640 1 -- 172.21.15.42:0/4158887530 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd9d40a39b0 msgr2=0x7fd9d41cfba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:45.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.230+0000 7fd9d9ab8640 1 --2- 172.21.15.42:0/4158887530 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd9d40a39b0 0x7fd9d41cfba0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fd9bc0ab000 tx=0x7fd9bc019100 comp rx=0 tx=0).stop 2024-09-16T04:12:45.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.230+0000 7fd9d3fff640 1 -- 172.21.15.42:0/4158887530 reap_dead start 2024-09-16T04:12:45.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.230+0000 7fd9d3fff640 1 -- 172.21.15.42:0/4158887530 reap_dead start 2024-09-16T04:12:45.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.230+0000 7fd9d9ab8640 1 -- 172.21.15.42:0/4158887530 shutdown_connections 2024-09-16T04:12:45.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.230+0000 7fd9d9ab8640 1 -- 172.21.15.42:0/4158887530 >> 172.21.15.42:0/4158887530 conn(0x7fd9d40a9fb0 msgr2=0x7fd9d4131980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:12:45.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.230+0000 7fd9d9ab8640 1 -- 172.21.15.42:0/4158887530 shutdown_connections 2024-09-16T04:12:45.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.230+0000 7fd9d9ab8640 1 -- 172.21.15.42:0/4158887530 wait complete. 2024-09-16T04:12:45.282 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.230+0000 7fd9d9ab8640 1 librados: shutdown 2024-09-16T04:12:45.283 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:12:45.283 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-09-16T04:12:45.283 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-16T04:12:45.283 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-09-16T04:12:45.283 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-16T04:12:45.284 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-16T04:12:45.284 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T04:12:45.296 INFO:tasks.ceph.mon.c:Stopped 2024-09-16T04:12:45.296 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T04:12:45.296 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:12:45.463 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.458+0000 7fd8a59d8640 10 monclient: get_monmap_and_config 2024-09-16T04:12:45.463 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.458+0000 7fd8a59d8640 10 monclient: build_initial_monmap 2024-09-16T04:12:45.463 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.458+0000 7fd8a59d8640 10 monclient: monmap: 2024-09-16T04:12:45.463 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:12:45.463 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:12:45.463 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:12:45.462000+0000 2024-09-16T04:12:45.463 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:12:45.462000+0000 2024-09-16T04:12:45.464 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:12:45.464 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:12:45.464 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:12:45.464 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:12:45.464 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:12:45.464 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:12:45.464 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:12:45.464 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:12:45.469 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.458+0000 7fd8a59d8640 1 Processor -- start 2024-09-16T04:12:45.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.458+0000 7fd8a59d8640 1 -- start start 2024-09-16T04:12:45.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.458+0000 7fd8a59d8640 10 monclient: init 2024-09-16T04:12:45.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.462+0000 7fd8a59d8640 10 monclient: _reopen_session rank -1 2024-09-16T04:12:45.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.462+0000 7fd8a59d8640 10 monclient: _add_conns ranks=[0,2,4,3,1] 2024-09-16T04:12:45.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.462+0000 7fd8a59d8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd8a006db90 addr v1:172.21.15.42:6789/0 2024-09-16T04:12:45.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.462+0000 7fd8a59d8640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd8a01340c0 0x7fd8a013c190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:12:45.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.462+0000 7fd8a59d8640 10 monclient(hunting): picked mon.noname-b con 0x7fd8a01340c0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:12:45.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.462+0000 7fd8a59d8640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fd8a013c6d0 0x7fd8a013eac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:12:45.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.462+0000 7fd8a59d8640 10 monclient(hunting): picked mon.noname-c con 0x7fd8a013c6d0 addr v2:172.21.15.154:3300/0 2024-09-16T04:12:45.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.462+0000 7fd8a59d8640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd8a006ece0 con 0x7fd8a006db90 2024-09-16T04:12:45.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.462+0000 7fd8a59d8640 10 monclient(hunting): start opening mon connection 2024-09-16T04:12:45.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.462+0000 7fd8a59d8640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fd8a006df70 con 0x7fd8a013c6d0 2024-09-16T04:12:45.471 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.462+0000 7fd8a59d8640 10 monclient(hunting): start opening mon connection 2024-09-16T04:12:45.471 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.462+0000 7fd8a59d8640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fd8a009bf80 con 0x7fd8a01340c0 2024-09-16T04:12:45.471 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.462+0000 7fd8a59d8640 10 monclient(hunting): _renew_subs 2024-09-16T04:12:45.471 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.462+0000 7fd8a59d8640 10 monclient(hunting): authenticate will time out at 2812.432263s 2024-09-16T04:12:45.471 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.462+0000 7fd89e7fc640 1 -- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd8a01340c0 msgr2=0x7fd8a013c190 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T04:12:45.471 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.462+0000 7fd89e7fc640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd8a01340c0 0x7fd8a013c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:12:45.471 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.462+0000 7fd89effd640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fd8a006db90 0x7fd8a00a37f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:34592/0 (socket says 172.21.15.42:34592) 2024-09-16T04:12:45.471 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.462+0000 7fd89effd640 1 -- 172.21.15.42:0/3416701705 learned_addr learned my addr 172.21.15.42:0/3416701705 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T04:12:45.471 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.462+0000 7fd89f7fe640 1 --2- 172.21.15.42:0/3416701705 >> v2:172.21.15.154:3300/0 conn(0x7fd8a013c6d0 0x7fd8a013eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:12:45.471 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.462+0000 7fd89dffb640 1 -- 172.21.15.42:0/3416701705 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1980058753 0 0) 0x7fd8a006ece0 con 0x7fd8a006db90 2024-09-16T04:12:45.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.462+0000 7fd89dffb640 10 monclient(hunting): _negotiate 2024-09-16T04:12:45.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.462+0000 7fd89dffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:45.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.462+0000 7fd89dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:45.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.462+0000 7fd89dffb640 10 monclient(hunting): my global_id is 1034559 2024-09-16T04:12:45.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.462+0000 7fd89dffb640 1 -- 172.21.15.42:0/3416701705 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd888003530 con 0x7fd8a006db90 2024-09-16T04:12:45.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.462+0000 7fd89f7fe640 10 monclient(hunting): get_auth_request con 0x7fd8a013c6d0 auth_method 0 2024-09-16T04:12:45.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.462+0000 7fd89f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:12:45.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.462+0000 7fd89f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:45.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.462+0000 7fd89f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:45.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.462+0000 7fd89f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:12:45.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.462+0000 7fd89f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:12:45.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.462+0000 7fd89f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:12:45.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.462+0000 7fd89dffb640 1 -- 172.21.15.42:0/3416701705 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 957665711 0 0) 0x7fd888003530 con 0x7fd8a006db90 2024-09-16T04:12:45.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.462+0000 7fd89dffb640 1 -- 172.21.15.42:0/3416701705 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd8a01340c0 msgr2=0x7fd8a013c190 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:12:45.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.462+0000 7fd89dffb640 1 --2- 172.21.15.42:0/3416701705 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd8a01340c0 0x7fd8a013c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:12:45.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.462+0000 7fd89dffb640 1 -- 172.21.15.42:0/3416701705 >> v2:172.21.15.154:3300/0 conn(0x7fd8a013c6d0 msgr2=0x7fd8a013eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:12:45.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.462+0000 7fd89dffb640 1 --2- 172.21.15.42:0/3416701705 >> v2:172.21.15.154:3300/0 conn(0x7fd8a013c6d0 0x7fd8a013eac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:12:45.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.462+0000 7fd89dffb640 10 monclient: _finish_hunting 0 2024-09-16T04:12:45.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.462+0000 7fd89dffb640 1 monclient: found mon.noname-a-legacy 2024-09-16T04:12:45.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.462+0000 7fd89dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:12:45.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.462+0000 7fd89dffb640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.42:6789/0 2024-09-16T04:12:45.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.462+0000 7fd89dffb640 1 -- 172.21.15.42:0/3416701705 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8a01344d0 con 0x7fd8a006db90 2024-09-16T04:12:45.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.462+0000 7fd89dffb640 10 monclient: _finish_auth 0 2024-09-16T04:12:45.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.462+0000 7fd89dffb640 10 monclient: _check_auth_tickets 2024-09-16T04:12:45.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.462+0000 7fd89dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:12:45.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.462+0000 7fd8a59d8640 5 monclient: authenticate success, global_id 1034559 2024-09-16T04:12:45.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.462+0000 7fd8a59d8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:12:45.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.462+0000 7fd89dffb640 1 -- 172.21.15.42:0/3416701705 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2177413734 0 0) 0x7fd890001680 con 0x7fd8a006db90 2024-09-16T04:12:45.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.462+0000 7fd89dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:45.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.462+0000 7fd89dffb640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T04:12:45.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.462+0000 7fd89dffb640 10 monclient: dump: 2024-09-16T04:12:45.474 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:12:45.475 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:12:45.475 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:45.475 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:45.475 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:45.475 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:12:45.475 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:12:45.475 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:12:45.475 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:12:45.475 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:12:45.475 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.462+0000 7fd89dffb640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] but i'm connected to v1:172.21.15.42:6789/0, reconnecting 2024-09-16T04:12:45.476 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.462+0000 7fd89dffb640 10 monclient: _reopen_session rank -1 2024-09-16T04:12:45.476 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.462+0000 7fd89dffb640 1 -- 172.21.15.42:0/3416701705 >> v1:172.21.15.42:6789/0 conn(0x7fd8a006db90 legacy=0x7fd8a00a37f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:45.476 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.462+0000 7fd89dffb640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:12:45.476 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.462+0000 7fd89dffb640 1 --2- 172.21.15.42:0/3416701705 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd8880039c0 0x7fd888003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:12:45.476 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.462+0000 7fd89dffb640 10 monclient(hunting): picked mon.b con 0x7fd8880039c0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:12:45.476 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.462+0000 7fd89dffb640 1 --2- 172.21.15.42:0/3416701705 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd8a01340c0 0x7fd888004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:12:45.476 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.462+0000 7fd89dffb640 10 monclient(hunting): picked mon.c con 0x7fd8a01340c0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:12:45.476 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.462+0000 7fd89dffb640 1 --2- 172.21.15.42:0/3416701705 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd888004d50 0x7fd8880051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:12:45.476 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.462+0000 7fd89dffb640 10 monclient(hunting): picked mon.a con 0x7fd888004d50 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:12:45.476 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.462+0000 7fd89dffb640 10 monclient(hunting): start opening mon connection 2024-09-16T04:12:45.477 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.462+0000 7fd89dffb640 1 -- 172.21.15.42:0/3416701705 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fd8a006df70 con 0x7fd888004d50 2024-09-16T04:12:45.477 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.462+0000 7fd89dffb640 10 monclient(hunting): start opening mon connection 2024-09-16T04:12:45.477 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.462+0000 7fd89dffb640 1 -- 172.21.15.42:0/3416701705 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd8a009bf80 con 0x7fd8880039c0 2024-09-16T04:12:45.477 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.462+0000 7fd89dffb640 10 monclient(hunting): start opening mon connection 2024-09-16T04:12:45.477 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.462+0000 7fd89dffb640 1 -- 172.21.15.42:0/3416701705 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fd88800bc10 con 0x7fd8a01340c0 2024-09-16T04:12:45.477 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.462+0000 7fd89dffb640 10 monclient(hunting): _renew_subs 2024-09-16T04:12:45.477 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.462+0000 7fd89dffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:12:45.477 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.462+0000 7fd89e7fc640 1 -- 172.21.15.42:0/3416701705 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd8a01340c0 msgr2=0x7fd888004810 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T04:12:45.477 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.462+0000 7fd8a59d8640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-16T04:12:45.477 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.462+0000 7fd89e7fc640 1 --2- 172.21.15.42:0/3416701705 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd8a01340c0 0x7fd888004810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:12:45.477 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.462+0000 7fd89f7fe640 1 --2- 172.21.15.42:0/3416701705 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd888004d50 0x7fd8880051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:12:45.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.462+0000 7fd89f7fe640 10 monclient(hunting): get_auth_request con 0x7fd888004d50 auth_method 0 2024-09-16T04:12:45.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.462+0000 7fd89f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:12:45.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.462+0000 7fd89f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:45.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.462+0000 7fd89f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:12:45.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.462+0000 7fd89effd640 1 --2- 172.21.15.42:0/3416701705 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd8880039c0 0x7fd888003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:12:45.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.462+0000 7fd89f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:12:45.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.462+0000 7fd89f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:12:45.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.462+0000 7fd89f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:12:45.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.462+0000 7fd89effd640 10 monclient(hunting): get_auth_request con 0x7fd8880039c0 auth_method 0 2024-09-16T04:12:45.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.462+0000 7fd89effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:12:45.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.462+0000 7fd89effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:45.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.462+0000 7fd89effd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:12:45.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.462+0000 7fd89effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:12:45.479 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.462+0000 7fd89effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:12:45.479 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.462+0000 7fd89effd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:12:45.479 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.462+0000 7fd89f7fe640 10 monclient(hunting): handle_auth_done global_id 1034559 payload 293 2024-09-16T04:12:45.479 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.462+0000 7fd89f7fe640 1 -- 172.21.15.42:0/3416701705 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd8a01340c0 msgr2=0x7fd888004810 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:12:45.479 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.462+0000 7fd89f7fe640 1 --2- 172.21.15.42:0/3416701705 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd8a01340c0 0x7fd888004810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:12:45.479 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.462+0000 7fd89f7fe640 1 -- 172.21.15.42:0/3416701705 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd8880039c0 msgr2=0x7fd888003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:12:45.479 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.462+0000 7fd89f7fe640 1 --2- 172.21.15.42:0/3416701705 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd8880039c0 0x7fd888003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:12:45.479 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.462+0000 7fd89f7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:12:45.479 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.462+0000 7fd89f7fe640 1 monclient: found mon.a 2024-09-16T04:12:45.479 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.462+0000 7fd89f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:12:45.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.462+0000 7fd89f7fe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:12:45.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.462+0000 7fd89f7fe640 1 -- 172.21.15.42:0/3416701705 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd888003530 con 0x7fd888004d50 2024-09-16T04:12:45.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.462+0000 7fd89f7fe640 10 monclient: _finish_auth 0 2024-09-16T04:12:45.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.462+0000 7fd89f7fe640 10 monclient: _check_auth_tickets 2024-09-16T04:12:45.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.462+0000 7fd89f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:12:45.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.462+0000 7fd89f7fe640 1 --2- 172.21.15.42:0/3416701705 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd888004d50 0x7fd8880051d0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fd898009f90 tx=0x7fd89801cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa055a7982dc2637 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:12:45.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.462+0000 7fd89dffb640 1 -- 172.21.15.42:0/3416701705 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8980b2070 con 0x7fd888004d50 2024-09-16T04:12:45.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.462+0000 7fd89dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:45.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.462+0000 7fd89dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:12:45.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.462+0000 7fd89dffb640 10 monclient: dump: 2024-09-16T04:12:45.480 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:12:45.480 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:12:45.481 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:45.481 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:45.481 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:45.481 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:12:45.481 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:12:45.481 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:12:45.481 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:12:45.481 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:12:45.481 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.466+0000 7fd8a59d8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:12:45.481 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.466+0000 7fd89dffb640 1 -- 172.21.15.42:0/3416701705 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd8980a5c50 con 0x7fd888004d50 2024-09-16T04:12:45.481 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.466+0000 7fd89dffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:12:45.481 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.466+0000 7fd8a59d8640 10 monclient: get_monmap_and_config success 2024-09-16T04:12:45.482 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.466+0000 7fd8a59d8640 4 set_mon_vals no callback set 2024-09-16T04:12:45.482 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.466+0000 7fd89dffb640 1 -- 172.21.15.42:0/3416701705 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd898009c10 con 0x7fd888004d50 2024-09-16T04:12:45.482 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.466+0000 7fd89dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:45.482 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.466+0000 7fd89dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:12:45.482 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.466+0000 7fd89dffb640 10 monclient: dump: 2024-09-16T04:12:45.482 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:12:45.482 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:12:45.483 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:45.483 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:45.483 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:45.483 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:12:45.483 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:12:45.483 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:12:45.483 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:12:45.483 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:12:45.483 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.466+0000 7fd8a59d8640 10 monclient: shutdown 2024-09-16T04:12:45.483 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.466+0000 7fd8a59d8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:12:45.484 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.466+0000 7fd8a59d8640 1 -- 172.21.15.42:0/3416701705 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd888004d50 msgr2=0x7fd8880051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:45.484 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.466+0000 7fd8a59d8640 1 --2- 172.21.15.42:0/3416701705 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd888004d50 0x7fd8880051d0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fd898009f90 tx=0x7fd89801cc60 comp rx=0 tx=0).stop 2024-09-16T04:12:45.484 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.466+0000 7fd89f7fe640 1 -- 172.21.15.42:0/3416701705 reap_dead start 2024-09-16T04:12:45.484 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.466+0000 7fd8a59d8640 1 -- 172.21.15.42:0/3416701705 shutdown_connections 2024-09-16T04:12:45.484 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.466+0000 7fd8a59d8640 1 -- 172.21.15.42:0/3416701705 >> 172.21.15.42:0/3416701705 conn(0x7fd8a00a9fb0 msgr2=0x7fd8a00aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:12:45.484 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.466+0000 7fd8a59d8640 1 -- 172.21.15.42:0/3416701705 shutdown_connections 2024-09-16T04:12:45.484 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.466+0000 7fd8a59d8640 1 -- 172.21.15.42:0/3416701705 wait complete. 2024-09-16T04:12:45.489 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.466+0000 7fd8a59d8640 10 monclient: build_initial_monmap 2024-09-16T04:12:45.489 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.466+0000 7fd8a59d8640 10 monclient: monmap: 2024-09-16T04:12:45.489 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:12:45.490 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:12:45.490 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:12:45.490 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:12:45.490 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:12:45.490 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:12:45.490 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:12:45.490 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:12:45.490 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:12:45.490 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:12:45.490 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.466+0000 7fd8a59d8640 1 librados: starting msgr at 2024-09-16T04:12:45.490 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.466+0000 7fd8a59d8640 1 librados: starting objecter 2024-09-16T04:12:45.490 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.466+0000 7fd8a59d8640 1 Processor -- start 2024-09-16T04:12:45.490 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.466+0000 7fd8a59d8640 1 -- start start 2024-09-16T04:12:45.490 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.466+0000 7fd8a59d8640 1 librados: setting wanted keys 2024-09-16T04:12:45.490 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.466+0000 7fd8a59d8640 1 librados: calling monclient init 2024-09-16T04:12:45.491 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.466+0000 7fd8a59d8640 10 monclient: init 2024-09-16T04:12:45.491 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.466+0000 7fd8a59d8640 10 monclient: _reopen_session rank -1 2024-09-16T04:12:45.491 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.466+0000 7fd8a59d8640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:12:45.491 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.466+0000 7fd8a59d8640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd8a00a3230 0x7fd8a009cb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:12:45.491 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.466+0000 7fd8a59d8640 10 monclient(hunting): picked mon.noname-c con 0x7fd8a00a3230 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:12:45.491 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.466+0000 7fd8a59d8640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd8a009d050 0x7fd8a009d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:12:45.491 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.466+0000 7fd8a59d8640 10 monclient(hunting): picked mon.noname-b con 0x7fd8a009d050 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:12:45.491 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.466+0000 7fd8a59d8640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd8a01d3920 0x7fd8a01d5d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:12:45.491 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.466+0000 7fd8a59d8640 10 monclient(hunting): picked mon.noname-a con 0x7fd8a01d3920 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:12:45.491 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.466+0000 7fd8a59d8640 10 monclient(hunting): start opening mon connection 2024-09-16T04:12:45.491 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.466+0000 7fd8a59d8640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fd8a00a4080 con 0x7fd8a01d3920 2024-09-16T04:12:45.491 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.466+0000 7fd8a59d8640 10 monclient(hunting): start opening mon connection 2024-09-16T04:12:45.492 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.466+0000 7fd8a59d8640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd8a00a3f00 con 0x7fd8a009d050 2024-09-16T04:12:45.492 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.466+0000 7fd8a59d8640 10 monclient(hunting): start opening mon connection 2024-09-16T04:12:45.492 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.466+0000 7fd8a59d8640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fd8a013f810 con 0x7fd8a00a3230 2024-09-16T04:12:45.492 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.466+0000 7fd8a59d8640 10 monclient(hunting): _renew_subs 2024-09-16T04:12:45.492 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.466+0000 7fd89effd640 1 -- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd8a00a3230 msgr2=0x7fd8a009cb10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T04:12:45.492 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.466+0000 7fd8a59d8640 10 monclient(hunting): authenticate will time out at 2812.438197s 2024-09-16T04:12:45.492 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.466+0000 7fd89effd640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd8a00a3230 0x7fd8a009cb10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:12:45.492 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.466+0000 7fd89f7fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd8a01d3920 0x7fd8a01d5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:12:45.493 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.466+0000 7fd89f7fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd8a01d3920 0x7fd8a01d5d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:46820/0 (socket says 172.21.15.42:46820) 2024-09-16T04:12:45.493 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.466+0000 7fd89f7fe640 1 -- 172.21.15.42:0/3497011943 learned_addr learned my addr 172.21.15.42:0/3497011943 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:12:45.493 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.466+0000 7fd89f7fe640 10 monclient(hunting): get_auth_request con 0x7fd8a01d3920 auth_method 0 2024-09-16T04:12:45.493 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.466+0000 7fd89f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:12:45.493 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.466+0000 7fd89f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:45.493 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.466+0000 7fd89f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:45.494 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.466+0000 7fd89f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:12:45.494 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.466+0000 7fd89f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:12:45.494 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.466+0000 7fd89f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:12:45.494 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.466+0000 7fd89e7fc640 1 --2- 172.21.15.42:0/3497011943 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd8a009d050 0x7fd8a009d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:12:45.494 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.466+0000 7fd89e7fc640 10 monclient(hunting): get_auth_request con 0x7fd8a009d050 auth_method 0 2024-09-16T04:12:45.494 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.466+0000 7fd89e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:12:45.494 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.466+0000 7fd89e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:45.494 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.466+0000 7fd89e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:45.494 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.466+0000 7fd89f7fe640 10 monclient(hunting): handle_auth_done global_id 1034562 payload 824 2024-09-16T04:12:45.494 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.466+0000 7fd89f7fe640 1 -- 172.21.15.42:0/3497011943 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd8a00a3230 msgr2=0x7fd8a009cb10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:12:45.494 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.466+0000 7fd89f7fe640 1 --2- 172.21.15.42:0/3497011943 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd8a00a3230 0x7fd8a009cb10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:12:45.494 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.466+0000 7fd89f7fe640 1 -- 172.21.15.42:0/3497011943 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd8a009d050 msgr2=0x7fd8a009d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:45.494 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.466+0000 7fd89f7fe640 1 --2- 172.21.15.42:0/3497011943 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd8a009d050 0x7fd8a009d4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:12:45.495 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.466+0000 7fd89f7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:12:45.495 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.466+0000 7fd89f7fe640 1 monclient: found mon.noname-a 2024-09-16T04:12:45.495 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.466+0000 7fd89f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:12:45.495 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.466+0000 7fd89f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:12:45.495 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.466+0000 7fd89f7fe640 1 -- 172.21.15.42:0/3497011943 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8a00a07c0 con 0x7fd8a01d3920 2024-09-16T04:12:45.495 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.466+0000 7fd89f7fe640 1 --2- 172.21.15.42:0/3497011943 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd8a01d3920 0x7fd8a01d5d10 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fd898009f90 tx=0x7fd898009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:12:45.495 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.466+0000 7fd887fff640 10 client.?.objecter ms_handle_connect 0x7fd8a01d3920 2024-09-16T04:12:45.495 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.466+0000 7fd887fff640 10 client.?.objecter resend_mon_ops 2024-09-16T04:12:45.495 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.466+0000 7fd887fff640 1 -- 172.21.15.42:0/3497011943 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8980b2040 con 0x7fd8a01d3920 2024-09-16T04:12:45.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.466+0000 7fd887fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:45.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.466+0000 7fd887fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:12:45.496 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.466+0000 7fd887fff640 10 monclient: dump: 2024-09-16T04:12:45.496 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:12:45.496 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:12:45.496 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:45.496 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:45.496 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:45.496 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:12:45.496 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:12:45.496 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:12:45.496 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:12:45.496 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:12:45.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.466+0000 7fd887fff640 10 monclient: _finish_auth 0 2024-09-16T04:12:45.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.466+0000 7fd887fff640 10 monclient: _check_auth_tickets 2024-09-16T04:12:45.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.466+0000 7fd887fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:12:45.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.466+0000 7fd887fff640 1 -- 172.21.15.42:0/3497011943 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd8980a5a50 con 0x7fd8a01d3920 2024-09-16T04:12:45.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.466+0000 7fd8a59d8640 5 monclient: authenticate success, global_id 1034562 2024-09-16T04:12:45.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.466+0000 7fd8a59d8640 10 monclient: _renew_subs 2024-09-16T04:12:45.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.466+0000 7fd8a59d8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:12:45.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.466+0000 7fd8a59d8640 1 -- 172.21.15.42:0/3497011943 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8a009f090 con 0x7fd8a01d3920 2024-09-16T04:12:45.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.466+0000 7fd887fff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:12:45.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.466+0000 7fd887fff640 1 -- 172.21.15.42:0/3497011943 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd898009c10 con 0x7fd8a01d3920 2024-09-16T04:12:45.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.466+0000 7fd887fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:45.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.466+0000 7fd887fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:12:45.497 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.466+0000 7fd887fff640 10 monclient: dump: 2024-09-16T04:12:45.498 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:12:45.498 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:12:45.498 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:45.498 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:45.498 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:45.498 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:12:45.498 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:12:45.498 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:12:45.498 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:12:45.498 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:12:45.498 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.466+0000 7fd8a59d8640 10 client.1034562.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:12:45.498 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.466+0000 7fd8a59d8640 10 monclient: _renew_subs 2024-09-16T04:12:45.498 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.466+0000 7fd8a59d8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:12:45.498 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.466+0000 7fd8a59d8640 1 -- 172.21.15.42:0/3497011943 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8a00a9200 con 0x7fd8a01d3920 2024-09-16T04:12:45.498 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.466+0000 7fd8a59d8640 1 librados: init done 2024-09-16T04:12:45.499 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.466+0000 7fd89d7fa640 4 set_mon_vals no callback set 2024-09-16T04:12:45.499 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.466+0000 7fd8a59d8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:12:45.499 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.466+0000 7fd8a59d8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:12:45.499 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.466+0000 7fd8a59d8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:12:45.499 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.466+0000 7fd8a59d8640 1 -- 172.21.15.42:0/3497011943 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd8a0132830 con 0x7fd8a01d3920 2024-09-16T04:12:45.499 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.466+0000 7fd887fff640 1 -- 172.21.15.42:0/3497011943 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fd8980be020 con 0x7fd8a01d3920 2024-09-16T04:12:45.499 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.470+0000 7fd887fff640 1 --2- 172.21.15.42:0/3497011943 >> v2:172.21.15.154:6812/254845342 conn(0x7fd88c0377c0 0x7fd88c039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:12:45.499 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.470+0000 7fd887fff640 1 -- 172.21.15.42:0/3497011943 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(706..706 src has 681..706) ==== 5040+0+0 (secure 0 0 0) 0x7fd8980e2830 con 0x7fd8a01d3920 2024-09-16T04:12:45.499 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.470+0000 7fd887fff640 10 client.1034562.objecter ms_dispatch 0x7fd8a0000ba0 osd_map(706..706 src has 681..706) 2024-09-16T04:12:45.499 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.470+0000 7fd887fff640 3 client.1034562.objecter handle_osd_map got epochs [706,706] > 0 2024-09-16T04:12:45.499 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.470+0000 7fd887fff640 3 client.1034562.objecter handle_osd_map decoding full epoch 706 2024-09-16T04:12:45.499 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.470+0000 7fd887fff640 20 client.1034562.objecter dump_active .. 0 homeless 2024-09-16T04:12:45.500 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.470+0000 7fd89effd640 1 --2- 172.21.15.42:0/3497011943 >> v2:172.21.15.154:6812/254845342 conn(0x7fd88c0377c0 0x7fd88c039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:12:45.500 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.470+0000 7fd89effd640 10 monclient: get_auth_request con 0x7fd88c0377c0 auth_method 0 2024-09-16T04:12:45.500 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.470+0000 7fd887fff640 10 client.1034562.objecter ms_handle_connect 0x7fd88c0377c0 2024-09-16T04:12:45.500 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.470+0000 7fd89effd640 1 --2- 172.21.15.42:0/3497011943 >> v2:172.21.15.154:6812/254845342 conn(0x7fd88c0377c0 0x7fd88c039c80 secure :-1 s=READY pgs=3975 cs=0 l=1 rev1=1 crypto rx=0x7fd8900011f0 tx=0x7fd8900010c0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:12:45.500 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.474+0000 7fd887fff640 1 -- 172.21.15.42:0/3497011943 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7fd8980c5020 con 0x7fd8a01d3920 2024-09-16T04:12:45.500 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.474+0000 7fd887fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:12:45.500 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.474+0000 7fd887fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:12:45.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.690+0000 7fd8a59d8640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:12:45.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.690+0000 7fd8a59d8640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:12:45.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.690+0000 7fd8a59d8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:12:45.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.690+0000 7fd8a59d8640 1 -- 172.21.15.42:0/3497011943 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fd8a00a37f0 con 0x7fd8a01d3920 2024-09-16T04:12:45.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.690+0000 7fd887fff640 1 -- 172.21.15.42:0/3497011943 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7fd898016970 con 0x7fd8a01d3920 2024-09-16T04:12:45.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.690+0000 7fd887fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:12:45.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.690+0000 7fd887fff640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:12:45.696 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":276,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:12:45.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.690+0000 7fd8a59d8640 10 librados: watch_flush enter 2024-09-16T04:12:45.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.690+0000 7fd8a59d8640 10 librados: watch_flush exit 2024-09-16T04:12:45.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.690+0000 7fd8a59d8640 20 client.1034562.objecter shutdown clearing up homeless session... 2024-09-16T04:12:45.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.690+0000 7fd8a59d8640 10 client.1034562.objecter successfully canceled tick 2024-09-16T04:12:45.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.690+0000 7fd8a59d8640 1 -- 172.21.15.42:0/3497011943 >> v2:172.21.15.154:6812/254845342 conn(0x7fd88c0377c0 msgr2=0x7fd88c039c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:45.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.690+0000 7fd8a59d8640 1 --2- 172.21.15.42:0/3497011943 >> v2:172.21.15.154:6812/254845342 conn(0x7fd88c0377c0 0x7fd88c039c80 secure :-1 s=READY pgs=3975 cs=0 l=1 rev1=1 crypto rx=0x7fd8900011f0 tx=0x7fd8900010c0 comp rx=0 tx=0).stop 2024-09-16T04:12:45.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.690+0000 7fd8a59d8640 10 monclient: shutdown 2024-09-16T04:12:45.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.690+0000 7fd8a59d8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:12:45.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.690+0000 7fd8a59d8640 1 -- 172.21.15.42:0/3497011943 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd8a01d3920 msgr2=0x7fd8a01d5d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:45.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.690+0000 7fd8a59d8640 1 --2- 172.21.15.42:0/3497011943 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd8a01d3920 0x7fd8a01d5d10 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fd898009f90 tx=0x7fd898009750 comp rx=0 tx=0).stop 2024-09-16T04:12:45.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.690+0000 7fd8a59d8640 1 -- 172.21.15.42:0/3497011943 shutdown_connections 2024-09-16T04:12:45.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.690+0000 7fd8a59d8640 1 --2- 172.21.15.42:0/3497011943 >> v2:172.21.15.154:6812/254845342 conn(0x7fd88c0377c0 0x7fd88c039c80 unknown :-1 s=CLOSED pgs=3975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:12:45.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.690+0000 7fd8a59d8640 1 --2- 172.21.15.42:0/3497011943 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd8a01d3920 0x7fd8a01d5d10 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:12:45.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.690+0000 7fd8a59d8640 1 --2- 172.21.15.42:0/3497011943 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd8a009d050 0x7fd8a009d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:12:45.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.690+0000 7fd8a59d8640 1 --2- 172.21.15.42:0/3497011943 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd8a00a3230 0x7fd8a009cb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:12:45.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.690+0000 7fd8a59d8640 1 -- 172.21.15.42:0/3497011943 >> 172.21.15.42:0/3497011943 conn(0x7fd8a00a9fb0 msgr2=0x7fd8a01397a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:12:45.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.694+0000 7fd8a59d8640 1 -- 172.21.15.42:0/3497011943 shutdown_connections 2024-09-16T04:12:45.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.694+0000 7fd8a59d8640 1 -- 172.21.15.42:0/3497011943 wait complete. 2024-09-16T04:12:45.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:45.694+0000 7fd8a59d8640 1 librados: shutdown 2024-09-16T04:12:48.714 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:12:48.847 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.842+0000 7feb0015f640 10 monclient: get_monmap_and_config 2024-09-16T04:12:48.847 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.842+0000 7feb0015f640 10 monclient: build_initial_monmap 2024-09-16T04:12:48.847 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.842+0000 7feb0015f640 10 monclient: monmap: 2024-09-16T04:12:48.847 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:12:48.847 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:12:48.847 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:12:48.845814+0000 2024-09-16T04:12:48.847 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:12:48.845814+0000 2024-09-16T04:12:48.847 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:12:48.848 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:12:48.848 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:12:48.848 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:12:48.848 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:12:48.848 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:12:48.848 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:12:48.848 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:12:48.861 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.842+0000 7feb0015f640 1 Processor -- start 2024-09-16T04:12:48.861 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.842+0000 7feb0015f640 1 -- start start 2024-09-16T04:12:48.862 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.842+0000 7feb0015f640 10 monclient: init 2024-09-16T04:12:48.862 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.842+0000 7feb0015f640 10 monclient: _reopen_session rank -1 2024-09-16T04:12:48.862 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.842+0000 7feb0015f640 10 monclient: _add_conns ranks=[2,4,0,1,3] 2024-09-16T04:12:48.862 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.842+0000 7feb0015f640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7feaf80a1330 0x7feaf80a1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:12:48.862 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.842+0000 7feb0015f640 10 monclient(hunting): picked mon.noname-b con 0x7feaf80a1330 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:12:48.862 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.842+0000 7feb0015f640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7feaf806dac0 0x7feaf80a1c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:12:48.862 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.842+0000 7feb0015f640 10 monclient(hunting): picked mon.noname-c con 0x7feaf806dac0 addr v2:172.21.15.154:3300/0 2024-09-16T04:12:48.862 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.842+0000 7feb0015f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7feaf8135100 addr v1:172.21.15.42:6789/0 2024-09-16T04:12:48.862 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.842+0000 7feb0015f640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feaf809c7d0 con 0x7feaf8135100 2024-09-16T04:12:48.862 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.842+0000 7feb0015f640 10 monclient(hunting): start opening mon connection 2024-09-16T04:12:48.862 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.842+0000 7feb0015f640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7feaf809c090 con 0x7feaf806dac0 2024-09-16T04:12:48.862 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.842+0000 7feb0015f640 10 monclient(hunting): start opening mon connection 2024-09-16T04:12:48.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.842+0000 7feb0015f640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7feaf809f610 con 0x7feaf80a1330 2024-09-16T04:12:48.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.842+0000 7feb0015f640 10 monclient(hunting): _renew_subs 2024-09-16T04:12:48.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.842+0000 7feb0015f640 10 monclient(hunting): authenticate will time out at 2815.815384s 2024-09-16T04:12:48.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.842+0000 7feafdefb640 1 -- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7feaf80a1330 msgr2=0x7feaf80a1710 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T04:12:48.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.842+0000 7feafdefb640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7feaf80a1330 0x7feaf80a1710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:12:48.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.842+0000 7feafe6fc640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7feaf8135100 0x7feaf8135550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:34608/0 (socket says 172.21.15.42:34608) 2024-09-16T04:12:48.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.842+0000 7feafe6fc640 1 -- 172.21.15.42:0/2200502453 learned_addr learned my addr 172.21.15.42:0/2200502453 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T04:12:48.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.842+0000 7feafcef9640 1 -- 172.21.15.42:0/2200502453 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3197269074 0 0) 0x7feaf809c7d0 con 0x7feaf8135100 2024-09-16T04:12:48.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.842+0000 7feafcef9640 10 monclient(hunting): _negotiate 2024-09-16T04:12:48.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.842+0000 7feafcef9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:48.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.842+0000 7feafcef9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:48.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.842+0000 7feafcef9640 10 monclient(hunting): my global_id is 1034565 2024-09-16T04:12:48.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.842+0000 7feafd6fa640 1 --2- 172.21.15.42:0/2200502453 >> v2:172.21.15.154:3300/0 conn(0x7feaf806dac0 0x7feaf80a1c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:12:48.863 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.842+0000 7feafcef9640 1 -- 172.21.15.42:0/2200502453 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feae4003530 con 0x7feaf8135100 2024-09-16T04:12:48.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.842+0000 7feafd6fa640 10 monclient(hunting): get_auth_request con 0x7feaf806dac0 auth_method 0 2024-09-16T04:12:48.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.842+0000 7feafd6fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:12:48.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.842+0000 7feafd6fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:48.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.842+0000 7feafd6fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:48.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.846+0000 7feafd6fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:12:48.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.846+0000 7feafd6fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:12:48.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.846+0000 7feafd6fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:12:48.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.846+0000 7feafcef9640 1 -- 172.21.15.42:0/2200502453 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2306604538 0 0) 0x7feae4003530 con 0x7feaf8135100 2024-09-16T04:12:48.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.846+0000 7feafcef9640 1 -- 172.21.15.42:0/2200502453 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7feaf80a1330 msgr2=0x7feaf80a1710 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:12:48.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.846+0000 7feafcef9640 1 --2- 172.21.15.42:0/2200502453 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7feaf80a1330 0x7feaf80a1710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:12:48.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.846+0000 7feafcef9640 1 -- 172.21.15.42:0/2200502453 >> v2:172.21.15.154:3300/0 conn(0x7feaf806dac0 msgr2=0x7feaf80a1c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:12:48.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.846+0000 7feafcef9640 1 --2- 172.21.15.42:0/2200502453 >> v2:172.21.15.154:3300/0 conn(0x7feaf806dac0 0x7feaf80a1c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:12:48.864 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.846+0000 7feafcef9640 10 monclient: _finish_hunting 0 2024-09-16T04:12:48.865 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.846+0000 7feafcef9640 1 monclient: found mon.noname-a-legacy 2024-09-16T04:12:48.865 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.846+0000 7feafcef9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:12:48.865 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.846+0000 7feafcef9640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.42:6789/0 2024-09-16T04:12:48.865 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.846+0000 7feafcef9640 1 -- 172.21.15.42:0/2200502453 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feaf809f8f0 con 0x7feaf8135100 2024-09-16T04:12:48.865 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.846+0000 7feafcef9640 10 monclient: _finish_auth 0 2024-09-16T04:12:48.865 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.846+0000 7feafcef9640 10 monclient: _check_auth_tickets 2024-09-16T04:12:48.865 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.846+0000 7feafcef9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:12:48.865 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.846+0000 7feafcef9640 1 -- 172.21.15.42:0/2200502453 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2177413734 0 0) 0x7feaf0001680 con 0x7feaf8135100 2024-09-16T04:12:48.865 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.846+0000 7feafcef9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:48.865 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.846+0000 7feafcef9640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T04:12:48.865 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.846+0000 7feafcef9640 10 monclient: dump: 2024-09-16T04:12:48.865 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:12:48.865 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:12:48.865 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:48.866 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:48.866 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:48.866 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:12:48.866 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:12:48.866 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:12:48.866 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:12:48.866 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:12:48.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.846+0000 7feafcef9640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] but i'm connected to v1:172.21.15.42:6789/0, reconnecting 2024-09-16T04:12:48.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.846+0000 7feafcef9640 10 monclient: _reopen_session rank -1 2024-09-16T04:12:48.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.846+0000 7feafcef9640 1 -- 172.21.15.42:0/2200502453 >> v1:172.21.15.42:6789/0 conn(0x7feaf8135100 legacy=0x7feaf8135550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:48.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.846+0000 7feafcef9640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:12:48.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.846+0000 7feafcef9640 1 --2- 172.21.15.42:0/2200502453 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7feaf80a1330 0x7feae40039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:12:48.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.846+0000 7feafcef9640 10 monclient(hunting): picked mon.c con 0x7feaf80a1330 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:12:48.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.846+0000 7feafcef9640 1 --2- 172.21.15.42:0/2200502453 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feae4003f60 0x7feae4004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:12:48.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.846+0000 7feafcef9640 10 monclient(hunting): picked mon.a con 0x7feae4003f60 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:12:48.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.846+0000 7feafcef9640 1 --2- 172.21.15.42:0/2200502453 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feae4004d50 0x7feae40051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:12:48.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.846+0000 7feafcef9640 10 monclient(hunting): picked mon.b con 0x7feae4004d50 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:12:48.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.846+0000 7feafcef9640 10 monclient(hunting): start opening mon connection 2024-09-16T04:12:48.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.846+0000 7feafcef9640 1 -- 172.21.15.42:0/2200502453 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7feaf809c090 con 0x7feae4003f60 2024-09-16T04:12:48.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.846+0000 7feafcef9640 10 monclient(hunting): start opening mon connection 2024-09-16T04:12:48.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.846+0000 7feafcef9640 1 -- 172.21.15.42:0/2200502453 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7feaf809f610 con 0x7feae4004d50 2024-09-16T04:12:48.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.846+0000 7feafcef9640 10 monclient(hunting): start opening mon connection 2024-09-16T04:12:48.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.846+0000 7feafcef9640 1 -- 172.21.15.42:0/2200502453 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7feae400bb90 con 0x7feaf80a1330 2024-09-16T04:12:48.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.846+0000 7feafcef9640 10 monclient(hunting): _renew_subs 2024-09-16T04:12:48.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.846+0000 7feafdefb640 1 -- 172.21.15.42:0/2200502453 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7feaf80a1330 msgr2=0x7feae40039e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T04:12:48.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.846+0000 7feafcef9640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:12:48.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.846+0000 7feafdefb640 1 --2- 172.21.15.42:0/2200502453 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7feaf80a1330 0x7feae40039e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:12:48.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.846+0000 7feafd6fa640 1 --2- 172.21.15.42:0/2200502453 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feae4003f60 0x7feae4004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:12:48.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.846+0000 7feafd6fa640 10 monclient(hunting): get_auth_request con 0x7feae4003f60 auth_method 0 2024-09-16T04:12:48.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.846+0000 7feafd6fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:12:48.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.846+0000 7feafd6fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:48.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.846+0000 7feafd6fa640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:12:48.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.846+0000 7feafd6fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:12:48.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.846+0000 7feafd6fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:12:48.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.846+0000 7feafd6fa640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:12:48.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.846+0000 7feafe6fc640 1 --2- 172.21.15.42:0/2200502453 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feae4004d50 0x7feae40051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:12:48.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.846+0000 7feafe6fc640 10 monclient(hunting): get_auth_request con 0x7feae4004d50 auth_method 0 2024-09-16T04:12:48.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.846+0000 7feafe6fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:12:48.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.846+0000 7feafe6fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:48.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.846+0000 7feafe6fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:12:48.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.846+0000 7feafd6fa640 10 monclient(hunting): handle_auth_done global_id 1034565 payload 293 2024-09-16T04:12:48.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.846+0000 7feafd6fa640 1 -- 172.21.15.42:0/2200502453 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7feaf80a1330 msgr2=0x7feae40039e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:12:48.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.846+0000 7feafd6fa640 1 --2- 172.21.15.42:0/2200502453 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7feaf80a1330 0x7feae40039e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:12:48.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.846+0000 7feafd6fa640 1 -- 172.21.15.42:0/2200502453 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feae4004d50 msgr2=0x7feae40051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:12:48.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.846+0000 7feafd6fa640 1 --2- 172.21.15.42:0/2200502453 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feae4004d50 0x7feae40051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:12:48.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.846+0000 7feafd6fa640 10 monclient: _finish_hunting 0 2024-09-16T04:12:48.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.846+0000 7feafd6fa640 1 monclient: found mon.a 2024-09-16T04:12:48.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.846+0000 7feafd6fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:12:48.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.846+0000 7feafd6fa640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:12:48.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.846+0000 7feafd6fa640 1 -- 172.21.15.42:0/2200502453 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feae4003530 con 0x7feae4003f60 2024-09-16T04:12:48.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.846+0000 7feafd6fa640 10 monclient: _finish_auth 0 2024-09-16T04:12:48.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.846+0000 7feafd6fa640 10 monclient: _check_auth_tickets 2024-09-16T04:12:48.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.846+0000 7feafd6fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:12:48.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.846+0000 7feb0015f640 5 monclient: authenticate success, global_id 1034565 2024-09-16T04:12:48.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.846+0000 7feafe6fc640 1 --2- 172.21.15.42:0/2200502453 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feae4004d50 0x7feae40051d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T04:12:48.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.846+0000 7feb0015f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:12:48.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.846+0000 7feafd6fa640 1 --2- 172.21.15.42:0/2200502453 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feae4003f60 0x7feae4004810 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7feae8009ae0 tx=0x7feae801ca60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=835460a723dbb787 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:12:48.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.846+0000 7feafcef9640 1 -- 172.21.15.42:0/2200502453 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feae80b2070 con 0x7feae4003f60 2024-09-16T04:12:48.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.846+0000 7feafcef9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:48.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.846+0000 7feafcef9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:12:48.870 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.846+0000 7feafcef9640 10 monclient: dump: 2024-09-16T04:12:48.870 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:12:48.870 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:12:48.870 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:48.870 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:48.870 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:48.870 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:12:48.870 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:12:48.871 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:12:48.871 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:12:48.871 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:12:48.871 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.846+0000 7feafcef9640 1 -- 172.21.15.42:0/2200502453 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feae80a5650 con 0x7feae4003f60 2024-09-16T04:12:48.871 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.846+0000 7feafcef9640 10 monclient: handle_config config(0 keys) 2024-09-16T04:12:48.871 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.846+0000 7feafcef9640 1 -- 172.21.15.42:0/2200502453 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feae80096b0 con 0x7feae4003f60 2024-09-16T04:12:48.871 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.846+0000 7feafcef9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:48.871 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.846+0000 7feafcef9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:12:48.871 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.846+0000 7feafcef9640 10 monclient: dump: 2024-09-16T04:12:48.871 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:12:48.871 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:12:48.871 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:48.871 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:48.871 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:48.872 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:12:48.872 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:12:48.872 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:12:48.872 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:12:48.872 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:12:48.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.846+0000 7feb0015f640 10 monclient: get_monmap_and_config success 2024-09-16T04:12:48.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.846+0000 7feb0015f640 4 set_mon_vals no callback set 2024-09-16T04:12:48.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.846+0000 7feb0015f640 10 monclient: shutdown 2024-09-16T04:12:48.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.846+0000 7feb0015f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:12:48.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.846+0000 7feb0015f640 1 -- 172.21.15.42:0/2200502453 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feae4003f60 msgr2=0x7feae4004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:48.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.846+0000 7feb0015f640 1 --2- 172.21.15.42:0/2200502453 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feae4003f60 0x7feae4004810 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7feae8009ae0 tx=0x7feae801ca60 comp rx=0 tx=0).stop 2024-09-16T04:12:48.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.846+0000 7feafe6fc640 1 -- 172.21.15.42:0/2200502453 reap_dead start 2024-09-16T04:12:48.872 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.846+0000 7feb0015f640 1 -- 172.21.15.42:0/2200502453 shutdown_connections 2024-09-16T04:12:48.873 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.846+0000 7feb0015f640 1 -- 172.21.15.42:0/2200502453 >> 172.21.15.42:0/2200502453 conn(0x7feaf80a2420 msgr2=0x7feaf80a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:12:48.873 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.846+0000 7feb0015f640 1 -- 172.21.15.42:0/2200502453 shutdown_connections 2024-09-16T04:12:48.873 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.846+0000 7feb0015f640 1 -- 172.21.15.42:0/2200502453 wait complete. 2024-09-16T04:12:48.873 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.850+0000 7feb0015f640 10 monclient: build_initial_monmap 2024-09-16T04:12:48.873 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.850+0000 7feb0015f640 10 monclient: monmap: 2024-09-16T04:12:48.873 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:12:48.873 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:12:48.873 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:12:48.873 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:12:48.873 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:12:48.873 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:12:48.873 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:12:48.873 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:12:48.874 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:12:48.874 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:12:48.874 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.850+0000 7feb0015f640 1 librados: starting msgr at 2024-09-16T04:12:48.874 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.850+0000 7feb0015f640 1 librados: starting objecter 2024-09-16T04:12:48.874 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.850+0000 7feb0015f640 1 Processor -- start 2024-09-16T04:12:48.874 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.850+0000 7feb0015f640 1 -- start start 2024-09-16T04:12:48.874 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.850+0000 7feb0015f640 1 librados: setting wanted keys 2024-09-16T04:12:48.874 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.850+0000 7feb0015f640 1 librados: calling monclient init 2024-09-16T04:12:48.874 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.850+0000 7feb0015f640 10 monclient: init 2024-09-16T04:12:48.874 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.850+0000 7feb0015f640 10 monclient: _reopen_session rank -1 2024-09-16T04:12:48.874 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.850+0000 7feb0015f640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:12:48.874 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.850+0000 7feb0015f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feae4003f60 0x7feaf809cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:12:48.874 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.850+0000 7feb0015f640 10 monclient(hunting): picked mon.noname-b con 0x7feae4003f60 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:12:48.874 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.850+0000 7feb0015f640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feaf809d0d0 0x7feaf809d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:12:48.875 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.850+0000 7feb0015f640 10 monclient(hunting): picked mon.noname-a con 0x7feaf809d0d0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:12:48.875 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.850+0000 7feb0015f640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7feaf812f7a0 0x7feaf812fc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:12:48.875 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.850+0000 7feb0015f640 10 monclient(hunting): picked mon.noname-c con 0x7feaf812f7a0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:12:48.875 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.850+0000 7feb0015f640 10 monclient(hunting): start opening mon connection 2024-09-16T04:12:48.875 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.850+0000 7feb0015f640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7feaf813db70 con 0x7feaf809d0d0 2024-09-16T04:12:48.875 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.850+0000 7feb0015f640 10 monclient(hunting): start opening mon connection 2024-09-16T04:12:48.875 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.850+0000 7feb0015f640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7feaf809ef80 con 0x7feae4003f60 2024-09-16T04:12:48.875 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.850+0000 7feb0015f640 10 monclient(hunting): start opening mon connection 2024-09-16T04:12:48.875 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.850+0000 7feb0015f640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7feaf813dcf0 con 0x7feaf812f7a0 2024-09-16T04:12:48.875 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.850+0000 7feb0015f640 10 monclient(hunting): _renew_subs 2024-09-16T04:12:48.875 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.850+0000 7feb0015f640 10 monclient(hunting): authenticate will time out at 2815.821489s 2024-09-16T04:12:48.875 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.850+0000 7feafe6fc640 1 -- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7feaf812f7a0 msgr2=0x7feaf812fc50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T04:12:48.875 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.850+0000 7feafe6fc640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7feaf812f7a0 0x7feaf812fc50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:12:48.875 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.850+0000 7feafd6fa640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feaf809d0d0 0x7feaf809d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:12:48.876 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.850+0000 7feafd6fa640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feaf809d0d0 0x7feaf809d510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:46842/0 (socket says 172.21.15.42:46842) 2024-09-16T04:12:48.876 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.850+0000 7feafd6fa640 1 -- 172.21.15.42:0/1028248068 learned_addr learned my addr 172.21.15.42:0/1028248068 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:12:48.876 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.850+0000 7feafd6fa640 10 monclient(hunting): get_auth_request con 0x7feaf809d0d0 auth_method 0 2024-09-16T04:12:48.876 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.850+0000 7feafd6fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:12:48.876 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.850+0000 7feafd6fa640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:48.876 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.850+0000 7feafd6fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:48.876 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.850+0000 7feafd6fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:12:48.876 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.850+0000 7feafd6fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:12:48.876 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.850+0000 7feafd6fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:12:48.876 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.850+0000 7feafdefb640 1 --2- 172.21.15.42:0/1028248068 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feae4003f60 0x7feaf809cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:12:48.876 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.850+0000 7feafdefb640 10 monclient(hunting): get_auth_request con 0x7feae4003f60 auth_method 0 2024-09-16T04:12:48.876 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.850+0000 7feafdefb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:12:48.876 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.850+0000 7feafdefb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:48.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.850+0000 7feafdefb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:48.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.850+0000 7feafd6fa640 10 monclient(hunting): handle_auth_done global_id 1034568 payload 824 2024-09-16T04:12:48.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.850+0000 7feafd6fa640 1 -- 172.21.15.42:0/1028248068 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7feaf812f7a0 msgr2=0x7feaf812fc50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:12:48.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.850+0000 7feafd6fa640 1 --2- 172.21.15.42:0/1028248068 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7feaf812f7a0 0x7feaf812fc50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:12:48.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.850+0000 7feafd6fa640 1 -- 172.21.15.42:0/1028248068 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feae4003f60 msgr2=0x7feaf809cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:48.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.850+0000 7feafd6fa640 1 --2- 172.21.15.42:0/1028248068 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feae4003f60 0x7feaf809cb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:12:48.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.850+0000 7feafd6fa640 10 monclient: _finish_hunting 0 2024-09-16T04:12:48.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.850+0000 7feafd6fa640 1 monclient: found mon.noname-a 2024-09-16T04:12:48.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.850+0000 7feafd6fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:12:48.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.850+0000 7feafd6fa640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:12:48.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.850+0000 7feafd6fa640 1 -- 172.21.15.42:0/1028248068 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feaf813b590 con 0x7feaf809d0d0 2024-09-16T04:12:48.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.850+0000 7feafd6fa640 1 --2- 172.21.15.42:0/1028248068 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feaf809d0d0 0x7feaf809d510 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7feae80b0040 tx=0x7feae4005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:12:48.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.850+0000 7feaee7fc640 10 client.?.objecter ms_handle_connect 0x7feaf809d0d0 2024-09-16T04:12:48.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.850+0000 7feaee7fc640 10 client.?.objecter resend_mon_ops 2024-09-16T04:12:48.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.850+0000 7feaee7fc640 1 -- 172.21.15.42:0/1028248068 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feae80bc030 con 0x7feaf809d0d0 2024-09-16T04:12:48.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.850+0000 7feaee7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:48.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.850+0000 7feaee7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:12:48.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.850+0000 7feaee7fc640 10 monclient: dump: 2024-09-16T04:12:48.878 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:12:48.878 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:12:48.879 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:48.879 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:48.879 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:48.879 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:12:48.879 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:12:48.879 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:12:48.879 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:12:48.879 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:12:48.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.850+0000 7feaee7fc640 10 monclient: _finish_auth 0 2024-09-16T04:12:48.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.850+0000 7feaee7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:12:48.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.850+0000 7feaee7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:12:48.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.850+0000 7feaee7fc640 1 -- 172.21.15.42:0/1028248068 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feae80a5e20 con 0x7feaf809d0d0 2024-09-16T04:12:48.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.850+0000 7feb0015f640 5 monclient: authenticate success, global_id 1034568 2024-09-16T04:12:48.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.850+0000 7feb0015f640 10 monclient: _renew_subs 2024-09-16T04:12:48.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.850+0000 7feb0015f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:12:48.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.850+0000 7feb0015f640 1 -- 172.21.15.42:0/1028248068 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feaf813acc0 con 0x7feaf809d0d0 2024-09-16T04:12:48.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.850+0000 7feaee7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:12:48.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.850+0000 7feaee7fc640 1 -- 172.21.15.42:0/1028248068 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feae80b2040 con 0x7feaf809d0d0 2024-09-16T04:12:48.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.850+0000 7feaee7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:48.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.850+0000 7feaee7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:12:48.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.850+0000 7feaee7fc640 10 monclient: dump: 2024-09-16T04:12:48.880 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:12:48.881 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:12:48.881 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:48.881 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:48.881 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:48.881 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:12:48.881 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:12:48.881 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:12:48.881 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:12:48.881 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:12:48.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.850+0000 7feaeffff640 4 set_mon_vals no callback set 2024-09-16T04:12:48.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.850+0000 7feb0015f640 10 client.1034568.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:12:48.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.850+0000 7feb0015f640 10 monclient: _renew_subs 2024-09-16T04:12:48.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.850+0000 7feb0015f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:12:48.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.850+0000 7feb0015f640 1 -- 172.21.15.42:0/1028248068 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feaf8073a30 con 0x7feaf809d0d0 2024-09-16T04:12:48.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.850+0000 7feb0015f640 1 librados: init done 2024-09-16T04:12:48.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.850+0000 7feb0015f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:12:48.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.850+0000 7feb0015f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:12:48.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.850+0000 7feb0015f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:12:48.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.850+0000 7feaee7fc640 1 -- 172.21.15.42:0/1028248068 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7feae80bf070 con 0x7feaf809d0d0 2024-09-16T04:12:48.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.850+0000 7feb0015f640 1 -- 172.21.15.42:0/1028248068 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feaf80a1c50 con 0x7feaf809d0d0 2024-09-16T04:12:48.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.850+0000 7feaee7fc640 1 --2- 172.21.15.42:0/1028248068 >> v2:172.21.15.154:6812/254845342 conn(0x7feadc0377c0 0x7feadc039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:12:48.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.850+0000 7feaee7fc640 1 -- 172.21.15.42:0/1028248068 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(706..706 src has 681..706) ==== 5040+0+0 (secure 0 0 0) 0x7feae80e2860 con 0x7feaf809d0d0 2024-09-16T04:12:48.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.850+0000 7feaee7fc640 10 client.1034568.objecter ms_dispatch 0x7feaf8000ba0 osd_map(706..706 src has 681..706) 2024-09-16T04:12:48.883 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.850+0000 7feaee7fc640 3 client.1034568.objecter handle_osd_map got epochs [706,706] > 0 2024-09-16T04:12:48.883 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.850+0000 7feaee7fc640 3 client.1034568.objecter handle_osd_map decoding full epoch 706 2024-09-16T04:12:48.883 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.850+0000 7feaee7fc640 20 client.1034568.objecter dump_active .. 0 homeless 2024-09-16T04:12:48.883 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.850+0000 7feafdefb640 1 --2- 172.21.15.42:0/1028248068 >> v2:172.21.15.154:6812/254845342 conn(0x7feadc0377c0 0x7feadc039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:12:48.883 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.850+0000 7feafdefb640 10 monclient: get_auth_request con 0x7feadc0377c0 auth_method 0 2024-09-16T04:12:48.883 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.854+0000 7feafdefb640 1 --2- 172.21.15.42:0/1028248068 >> v2:172.21.15.154:6812/254845342 conn(0x7feadc0377c0 0x7feadc039c80 secure :-1 s=READY pgs=3976 cs=0 l=1 rev1=1 crypto rx=0x7feaf4003d10 tx=0x7feaf40094d0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:12:48.883 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.854+0000 7feaee7fc640 10 client.1034568.objecter ms_handle_connect 0x7feadc0377c0 2024-09-16T04:12:48.883 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.858+0000 7feaee7fc640 1 -- 172.21.15.42:0/1028248068 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7feae80167a0 con 0x7feaf809d0d0 2024-09-16T04:12:48.883 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.858+0000 7feaee7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:12:48.883 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:48.858+0000 7feaee7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:12:49.085 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:49.082+0000 7feb0015f640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:12:49.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:49.082+0000 7feb0015f640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:12:49.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:49.082+0000 7feb0015f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:12:49.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:49.082+0000 7feb0015f640 1 -- 172.21.15.42:0/1028248068 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7feaf8130d00 con 0x7feaf809d0d0 2024-09-16T04:12:49.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:49.082+0000 7feaee7fc640 1 -- 172.21.15.42:0/1028248068 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7feae80c1ca0 con 0x7feaf809d0d0 2024-09-16T04:12:49.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:49.082+0000 7feaee7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:12:49.087 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:49.082+0000 7feaee7fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:12:49.087 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":276,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:12:49.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:49.082+0000 7feb0015f640 10 librados: watch_flush enter 2024-09-16T04:12:49.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:49.082+0000 7feb0015f640 10 librados: watch_flush exit 2024-09-16T04:12:49.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:49.082+0000 7feb0015f640 20 client.1034568.objecter shutdown clearing up homeless session... 2024-09-16T04:12:49.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:49.082+0000 7feb0015f640 10 client.1034568.objecter successfully canceled tick 2024-09-16T04:12:49.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:49.082+0000 7feb0015f640 1 -- 172.21.15.42:0/1028248068 >> v2:172.21.15.154:6812/254845342 conn(0x7feadc0377c0 msgr2=0x7feadc039c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:49.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:49.082+0000 7feb0015f640 1 --2- 172.21.15.42:0/1028248068 >> v2:172.21.15.154:6812/254845342 conn(0x7feadc0377c0 0x7feadc039c80 secure :-1 s=READY pgs=3976 cs=0 l=1 rev1=1 crypto rx=0x7feaf4003d10 tx=0x7feaf40094d0 comp rx=0 tx=0).stop 2024-09-16T04:12:49.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:49.082+0000 7feb0015f640 10 monclient: shutdown 2024-09-16T04:12:49.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:49.082+0000 7feb0015f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:12:49.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:49.082+0000 7feb0015f640 1 -- 172.21.15.42:0/1028248068 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feaf809d0d0 msgr2=0x7feaf809d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:49.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:49.082+0000 7feb0015f640 1 --2- 172.21.15.42:0/1028248068 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feaf809d0d0 0x7feaf809d510 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7feae80b0040 tx=0x7feae4005840 comp rx=0 tx=0).stop 2024-09-16T04:12:49.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:49.082+0000 7feb0015f640 1 -- 172.21.15.42:0/1028248068 shutdown_connections 2024-09-16T04:12:49.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:49.082+0000 7feb0015f640 1 --2- 172.21.15.42:0/1028248068 >> v2:172.21.15.154:6812/254845342 conn(0x7feadc0377c0 0x7feadc039c80 unknown :-1 s=CLOSED pgs=3976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:12:49.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:49.082+0000 7feb0015f640 1 --2- 172.21.15.42:0/1028248068 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7feaf812f7a0 0x7feaf812fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:12:49.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:49.082+0000 7feb0015f640 1 --2- 172.21.15.42:0/1028248068 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feaf809d0d0 0x7feaf809d510 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:12:49.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:49.082+0000 7feb0015f640 1 --2- 172.21.15.42:0/1028248068 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feae4003f60 0x7feaf809cb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:12:49.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:49.082+0000 7feb0015f640 1 -- 172.21.15.42:0/1028248068 >> 172.21.15.42:0/1028248068 conn(0x7feaf80a2420 msgr2=0x7feaf8134590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:12:49.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:49.082+0000 7feb0015f640 1 -- 172.21.15.42:0/1028248068 shutdown_connections 2024-09-16T04:12:49.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:49.082+0000 7feb0015f640 1 -- 172.21.15.42:0/1028248068 wait complete. 2024-09-16T04:12:49.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:49.082+0000 7feb0015f640 1 librados: shutdown 2024-09-16T04:12:52.108 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:12:52.242 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.238+0000 7f08897f6640 10 monclient: get_monmap_and_config 2024-09-16T04:12:52.242 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.238+0000 7f08897f6640 10 monclient: build_initial_monmap 2024-09-16T04:12:52.242 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.238+0000 7f08897f6640 10 monclient: monmap: 2024-09-16T04:12:52.242 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:12:52.242 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:12:52.243 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:12:52.241118+0000 2024-09-16T04:12:52.243 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:12:52.241118+0000 2024-09-16T04:12:52.243 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:12:52.243 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:12:52.243 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:12:52.243 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:12:52.243 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:12:52.243 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:12:52.243 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:12:52.243 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:12:52.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.238+0000 7f08897f6640 1 Processor -- start 2024-09-16T04:12:52.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.238+0000 7f08897f6640 1 -- start start 2024-09-16T04:12:52.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.238+0000 7f08897f6640 10 monclient: init 2024-09-16T04:12:52.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.238+0000 7f08897f6640 10 monclient: _reopen_session rank -1 2024-09-16T04:12:52.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.238+0000 7f08897f6640 10 monclient: _add_conns ranks=[4,2,0,3,1] 2024-09-16T04:12:52.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.238+0000 7f08897f6640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f08841353a0 0x7f0884135780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:12:52.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.238+0000 7f08897f6640 10 monclient(hunting): picked mon.noname-c con 0x7f08841353a0 addr v2:172.21.15.154:3300/0 2024-09-16T04:12:52.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.238+0000 7f08897f6640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f08840a0a40 0x7f08840a0e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:12:52.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.238+0000 7f08897f6640 10 monclient(hunting): picked mon.noname-b con 0x7f08840a0a40 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:12:52.246 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.238+0000 7f08897f6640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f08840a13a0 addr v1:172.21.15.42:6789/0 2024-09-16T04:12:52.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.238+0000 7f08897f6640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f08840a36b0 con 0x7f08840a13a0 2024-09-16T04:12:52.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.238+0000 7f08897f6640 10 monclient(hunting): start opening mon connection 2024-09-16T04:12:52.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.238+0000 7f08897f6640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f0884134e50 con 0x7f08841353a0 2024-09-16T04:12:52.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.238+0000 7f08897f6640 10 monclient(hunting): start opening mon connection 2024-09-16T04:12:52.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.238+0000 7f08897f6640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f0884075ce0 con 0x7f08840a0a40 2024-09-16T04:12:52.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.238+0000 7f08897f6640 10 monclient(hunting): _renew_subs 2024-09-16T04:12:52.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.238+0000 7f08897f6640 10 monclient(hunting): authenticate will time out at 2819.211046s 2024-09-16T04:12:52.247 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.238+0000 7f08827fc640 1 -- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f08840a0a40 msgr2=0x7f08840a0e60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T04:12:52.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.238+0000 7f08827fc640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f08840a0a40 0x7f08840a0e60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:12:52.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.238+0000 7f08837fe640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f08840a13a0 0x7f088413c180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:34614/0 (socket says 172.21.15.42:34614) 2024-09-16T04:12:52.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.238+0000 7f08837fe640 1 -- 172.21.15.42:0/1755747382 learned_addr learned my addr 172.21.15.42:0/1755747382 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T04:12:52.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.238+0000 7f0882ffd640 1 --2- 172.21.15.42:0/1755747382 >> v2:172.21.15.154:3300/0 conn(0x7f08841353a0 0x7f0884135780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:12:52.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.238+0000 7f0882ffd640 10 monclient(hunting): get_auth_request con 0x7f08841353a0 auth_method 0 2024-09-16T04:12:52.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.238+0000 7f0882ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:12:52.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.238+0000 7f0882ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:52.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.238+0000 7f0882ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:52.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.238+0000 7f0882ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:12:52.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.238+0000 7f0882ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:12:52.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.238+0000 7f0882ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:12:52.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.238+0000 7f0882ffd640 10 monclient(hunting): handle_auth_done global_id 1074229 payload 274 2024-09-16T04:12:52.248 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.242+0000 7f0882ffd640 1 -- 172.21.15.42:0/1755747382 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f08840a0a40 msgr2=0x7f08840a0e60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:12:52.249 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.242+0000 7f0882ffd640 1 --2- 172.21.15.42:0/1755747382 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f08840a0a40 0x7f08840a0e60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:12:52.249 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.242+0000 7f0882ffd640 1 -- 172.21.15.42:0/1755747382 >> v1:172.21.15.42:6789/0 conn(0x7f08840a13a0 legacy=0x7f088413c180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:52.249 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.242+0000 7f0882ffd640 10 monclient: _finish_hunting 0 2024-09-16T04:12:52.249 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.242+0000 7f0882ffd640 1 monclient: found mon.noname-c 2024-09-16T04:12:52.249 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.242+0000 7f0882ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:12:52.249 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.242+0000 7f0882ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.154:3300/0 2024-09-16T04:12:52.249 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.242+0000 7f0882ffd640 1 -- 172.21.15.42:0/1755747382 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08840a0260 con 0x7f08841353a0 2024-09-16T04:12:52.255 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.242+0000 7f0882ffd640 1 --2- 172.21.15.42:0/1755747382 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f08841353a0 0x7f0884135780 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f08740097c0 tx=0x7f087401cde0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4b1d25f2a0d9633e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:12:52.255 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.242+0000 7f0881ffb640 1 -- 172.21.15.42:0/1755747382 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08740b0070 con 0x7f08841353a0 2024-09-16T04:12:52.255 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.242+0000 7f0881ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:52.255 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.242+0000 7f0881ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:12:52.255 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.242+0000 7f0881ffb640 10 monclient: dump: 2024-09-16T04:12:52.256 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:12:52.256 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:12:52.256 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:52.256 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:52.256 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:52.256 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:12:52.256 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:12:52.256 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:12:52.256 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:12:52.256 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:12:52.256 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.242+0000 7f0881ffb640 10 monclient: _finish_auth 0 2024-09-16T04:12:52.256 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.242+0000 7f0881ffb640 10 monclient: _check_auth_tickets 2024-09-16T04:12:52.256 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.242+0000 7f0881ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:12:52.257 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.242+0000 7f08897f6640 5 monclient: authenticate success, global_id 1074229 2024-09-16T04:12:52.257 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.242+0000 7f08897f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:12:52.257 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.242+0000 7f0881ffb640 1 -- 172.21.15.42:0/1755747382 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f08740ab600 con 0x7f08841353a0 2024-09-16T04:12:52.257 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.242+0000 7f0881ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:12:52.257 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.242+0000 7f08897f6640 10 monclient: get_monmap_and_config success 2024-09-16T04:12:52.257 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.242+0000 7f08897f6640 4 set_mon_vals no callback set 2024-09-16T04:12:52.257 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.242+0000 7f0881ffb640 1 -- 172.21.15.42:0/1755747382 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08740ab9d0 con 0x7f08841353a0 2024-09-16T04:12:52.257 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.242+0000 7f0881ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:52.257 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.242+0000 7f0881ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:12:52.257 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.242+0000 7f0881ffb640 10 monclient: dump: 2024-09-16T04:12:52.257 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:12:52.257 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:12:52.257 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:52.258 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:52.258 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:52.258 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:12:52.258 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:12:52.258 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:12:52.258 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:12:52.258 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:12:52.258 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.242+0000 7f08897f6640 10 monclient: shutdown 2024-09-16T04:12:52.258 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.242+0000 7f08897f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:12:52.258 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.242+0000 7f08897f6640 1 -- 172.21.15.42:0/1755747382 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f08841353a0 msgr2=0x7f0884135780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:52.258 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.242+0000 7f08897f6640 1 --2- 172.21.15.42:0/1755747382 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f08841353a0 0x7f0884135780 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f08740097c0 tx=0x7f087401cde0 comp rx=0 tx=0).stop 2024-09-16T04:12:52.258 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.242+0000 7f08897f6640 1 -- 172.21.15.42:0/1755747382 shutdown_connections 2024-09-16T04:12:52.259 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.242+0000 7f08897f6640 1 --2- 172.21.15.42:0/1755747382 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f08840a0a40 0x7f08840a0e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:12:52.259 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.242+0000 7f08897f6640 1 --2- 172.21.15.42:0/1755747382 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f08841353a0 0x7f0884135780 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:12:52.259 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.242+0000 7f08897f6640 1 -- 172.21.15.42:0/1755747382 >> 172.21.15.42:0/1755747382 conn(0x7f08840a6640 msgr2=0x7f08840a2b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:12:52.259 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.242+0000 7f08897f6640 1 -- 172.21.15.42:0/1755747382 shutdown_connections 2024-09-16T04:12:52.259 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.242+0000 7f08897f6640 1 -- 172.21.15.42:0/1755747382 wait complete. 2024-09-16T04:12:52.259 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.246+0000 7f08897f6640 10 monclient: build_initial_monmap 2024-09-16T04:12:52.259 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.246+0000 7f08897f6640 10 monclient: monmap: 2024-09-16T04:12:52.259 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:12:52.259 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:12:52.259 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:12:52.259 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:12:52.259 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:12:52.259 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:12:52.260 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:12:52.260 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:12:52.260 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:12:52.260 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:12:52.260 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.246+0000 7f08897f6640 1 librados: starting msgr at 2024-09-16T04:12:52.260 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.246+0000 7f08897f6640 1 librados: starting objecter 2024-09-16T04:12:52.260 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.246+0000 7f08897f6640 1 Processor -- start 2024-09-16T04:12:52.260 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.246+0000 7f08897f6640 1 -- start start 2024-09-16T04:12:52.260 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.246+0000 7f08897f6640 1 librados: setting wanted keys 2024-09-16T04:12:52.260 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.246+0000 7f08897f6640 1 librados: calling monclient init 2024-09-16T04:12:52.260 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.246+0000 7f08897f6640 10 monclient: init 2024-09-16T04:12:52.260 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.246+0000 7f08897f6640 10 monclient: _reopen_session rank -1 2024-09-16T04:12:52.260 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.246+0000 7f08897f6640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:12:52.261 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.246+0000 7f08897f6640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f08840a0a40 0x7f088412eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:12:52.261 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.246+0000 7f08897f6640 10 monclient(hunting): picked mon.noname-a con 0x7f08840a0a40 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:12:52.261 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.246+0000 7f08897f6640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f08840a13a0 0x7f088412f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:12:52.261 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.246+0000 7f08897f6640 10 monclient(hunting): picked mon.noname-b con 0x7f08840a13a0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:12:52.261 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.246+0000 7f08897f6640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f08841353a0 0x7f08841cf050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:12:52.261 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.246+0000 7f08897f6640 10 monclient(hunting): picked mon.noname-c con 0x7f08841353a0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:12:52.261 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.246+0000 7f08897f6640 10 monclient(hunting): start opening mon connection 2024-09-16T04:12:52.261 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.246+0000 7f08897f6640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f088409ccc0 con 0x7f08840a0a40 2024-09-16T04:12:52.261 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.246+0000 7f08897f6640 10 monclient(hunting): start opening mon connection 2024-09-16T04:12:52.261 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.246+0000 7f08897f6640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f08840a1870 con 0x7f08840a13a0 2024-09-16T04:12:52.261 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.246+0000 7f08897f6640 10 monclient(hunting): start opening mon connection 2024-09-16T04:12:52.261 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.246+0000 7f08897f6640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f088409ce40 con 0x7f08841353a0 2024-09-16T04:12:52.261 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.246+0000 7f08897f6640 10 monclient(hunting): _renew_subs 2024-09-16T04:12:52.261 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.246+0000 7f08897f6640 10 monclient(hunting): authenticate will time out at 2819.217353s 2024-09-16T04:12:52.262 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.246+0000 7f08837fe640 1 -- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f08841353a0 msgr2=0x7f08841cf050 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T04:12:52.262 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.246+0000 7f08837fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f08841353a0 0x7f08841cf050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:12:52.262 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.246+0000 7f0882ffd640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f08840a0a40 0x7f088412eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:12:52.262 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.246+0000 7f0882ffd640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f08840a0a40 0x7f088412eb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:46858/0 (socket says 172.21.15.42:46858) 2024-09-16T04:12:52.262 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.246+0000 7f0882ffd640 1 -- 172.21.15.42:0/1483313289 learned_addr learned my addr 172.21.15.42:0/1483313289 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:12:52.262 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.246+0000 7f0882ffd640 10 monclient(hunting): get_auth_request con 0x7f08840a0a40 auth_method 0 2024-09-16T04:12:52.262 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.246+0000 7f0882ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:12:52.262 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.246+0000 7f0882ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:52.262 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.246+0000 7f0882ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:52.262 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.246+0000 7f0882ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:12:52.262 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.246+0000 7f0882ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:12:52.262 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.246+0000 7f0882ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:12:52.262 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.246+0000 7f08827fc640 1 --2- 172.21.15.42:0/1483313289 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f08840a13a0 0x7f088412f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:12:52.263 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.246+0000 7f0882ffd640 10 monclient(hunting): handle_auth_done global_id 1034571 payload 824 2024-09-16T04:12:52.263 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.246+0000 7f0882ffd640 1 -- 172.21.15.42:0/1483313289 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f08841353a0 msgr2=0x7f08841cf050 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:12:52.263 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.246+0000 7f0882ffd640 1 --2- 172.21.15.42:0/1483313289 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f08841353a0 0x7f08841cf050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:12:52.263 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.246+0000 7f0882ffd640 1 -- 172.21.15.42:0/1483313289 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f08840a13a0 msgr2=0x7f088412f0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:52.263 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.246+0000 7f0882ffd640 1 --2- 172.21.15.42:0/1483313289 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f08840a13a0 0x7f088412f0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:12:52.263 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.246+0000 7f0882ffd640 10 monclient: _finish_hunting 0 2024-09-16T04:12:52.263 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.246+0000 7f0882ffd640 1 monclient: found mon.noname-a 2024-09-16T04:12:52.263 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.246+0000 7f0882ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:12:52.263 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.246+0000 7f0882ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:12:52.263 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.246+0000 7f0882ffd640 1 -- 172.21.15.42:0/1483313289 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f088413bc60 con 0x7f08840a0a40 2024-09-16T04:12:52.263 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.246+0000 7f08827fc640 10 monclient: get_auth_request con 0x7f08840a13a0 auth_method 0 2024-09-16T04:12:52.263 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.246+0000 7f08827fc640 1 --2- 172.21.15.42:0/1483313289 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f08840a13a0 0x7f088412f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T04:12:52.263 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.246+0000 7f0882ffd640 1 --2- 172.21.15.42:0/1483313289 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f08840a0a40 0x7f088412eb90 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f08740ae040 tx=0x7f087401abc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:12:52.264 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.246+0000 7f087b7fe640 10 client.?.objecter ms_handle_connect 0x7f08840a0a40 2024-09-16T04:12:52.264 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.246+0000 7f087b7fe640 10 client.?.objecter resend_mon_ops 2024-09-16T04:12:52.264 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.246+0000 7f087b7fe640 1 -- 172.21.15.42:0/1483313289 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08740b0050 con 0x7f08840a0a40 2024-09-16T04:12:52.264 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.246+0000 7f087b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:52.264 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.246+0000 7f087b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:12:52.264 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.246+0000 7f087b7fe640 10 monclient: dump: 2024-09-16T04:12:52.264 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:12:52.264 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:12:52.264 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:52.264 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:52.264 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:52.264 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:12:52.264 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:12:52.264 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:12:52.265 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:12:52.265 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:12:52.265 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.246+0000 7f087b7fe640 10 monclient: _finish_auth 0 2024-09-16T04:12:52.265 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.246+0000 7f087b7fe640 10 monclient: _check_auth_tickets 2024-09-16T04:12:52.265 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.246+0000 7f087b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:12:52.265 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.246+0000 7f087b7fe640 1 -- 172.21.15.42:0/1483313289 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f08740bc070 con 0x7f08840a0a40 2024-09-16T04:12:52.265 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.246+0000 7f08897f6640 5 monclient: authenticate success, global_id 1034571 2024-09-16T04:12:52.265 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.246+0000 7f08897f6640 10 monclient: _renew_subs 2024-09-16T04:12:52.265 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.246+0000 7f08897f6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:12:52.265 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.246+0000 7f08897f6640 1 -- 172.21.15.42:0/1483313289 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f088413bf40 con 0x7f08840a0a40 2024-09-16T04:12:52.265 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.246+0000 7f087b7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:12:52.265 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.246+0000 7f087b7fe640 1 -- 172.21.15.42:0/1483313289 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08740b5810 con 0x7f08840a0a40 2024-09-16T04:12:52.265 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.246+0000 7f087b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:12:52.265 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.246+0000 7f087b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:12:52.266 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.246+0000 7f087b7fe640 10 monclient: dump: 2024-09-16T04:12:52.266 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:12:52.266 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:12:52.266 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:52.266 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:12:52.266 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:12:52.266 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:12:52.266 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:12:52.266 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:12:52.266 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:12:52.266 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:12:52.266 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.246+0000 7f0880ff9640 4 set_mon_vals no callback set 2024-09-16T04:12:52.266 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.246+0000 7f08897f6640 10 client.1034571.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:12:52.266 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.246+0000 7f08897f6640 10 monclient: _renew_subs 2024-09-16T04:12:52.267 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.246+0000 7f08897f6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:12:52.267 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.246+0000 7f08897f6640 1 -- 172.21.15.42:0/1483313289 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08840a40f0 con 0x7f08840a0a40 2024-09-16T04:12:52.267 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.246+0000 7f08897f6640 1 librados: init done 2024-09-16T04:12:52.267 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.246+0000 7f08897f6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:12:52.267 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.246+0000 7f08897f6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:12:52.267 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.246+0000 7f08897f6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:12:52.267 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.246+0000 7f08897f6640 1 -- 172.21.15.42:0/1483313289 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f08840a0e60 con 0x7f08840a0a40 2024-09-16T04:12:52.267 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.246+0000 7f087b7fe640 1 -- 172.21.15.42:0/1483313289 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f08740b59b0 con 0x7f08840a0a40 2024-09-16T04:12:52.267 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.246+0000 7f087b7fe640 1 --2- 172.21.15.42:0/1483313289 >> v2:172.21.15.154:6812/254845342 conn(0x7f08640377c0 0x7f0864039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:12:52.267 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.246+0000 7f087b7fe640 1 -- 172.21.15.42:0/1483313289 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(706..706 src has 681..706) ==== 5040+0+0 (secure 0 0 0) 0x7f08740e34d0 con 0x7f08840a0a40 2024-09-16T04:12:52.267 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.246+0000 7f087b7fe640 10 client.1034571.objecter ms_dispatch 0x7f0884000ba0 osd_map(706..706 src has 681..706) 2024-09-16T04:12:52.267 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.246+0000 7f087b7fe640 3 client.1034571.objecter handle_osd_map got epochs [706,706] > 0 2024-09-16T04:12:52.267 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.246+0000 7f087b7fe640 3 client.1034571.objecter handle_osd_map decoding full epoch 706 2024-09-16T04:12:52.268 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.246+0000 7f087b7fe640 20 client.1034571.objecter dump_active .. 0 homeless 2024-09-16T04:12:52.268 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.246+0000 7f08827fc640 1 --2- 172.21.15.42:0/1483313289 >> v2:172.21.15.154:6812/254845342 conn(0x7f08640377c0 0x7f0864039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:12:52.268 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.246+0000 7f08827fc640 10 monclient: get_auth_request con 0x7f08640377c0 auth_method 0 2024-09-16T04:12:52.268 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.250+0000 7f087b7fe640 10 client.1034571.objecter ms_handle_connect 0x7f08640377c0 2024-09-16T04:12:52.268 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.250+0000 7f08827fc640 1 --2- 172.21.15.42:0/1483313289 >> v2:172.21.15.154:6812/254845342 conn(0x7f08640377c0 0x7f0864039c80 secure :-1 s=READY pgs=3977 cs=0 l=1 rev1=1 crypto rx=0x7f088406fd10 tx=0x7f086c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:12:52.268 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.254+0000 7f087b7fe640 1 -- 172.21.15.42:0/1483313289 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f0874016090 con 0x7f08840a0a40 2024-09-16T04:12:52.268 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.254+0000 7f087b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:12:52.268 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.254+0000 7f087b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:12:52.477 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.470+0000 7f08897f6640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:12:52.477 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.470+0000 7f08897f6640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:12:52.477 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.474+0000 7f08897f6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:12:52.477 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.474+0000 7f08897f6640 1 -- 172.21.15.42:0/1483313289 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f08840a3450 con 0x7f08840a0a40 2024-09-16T04:12:52.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.474+0000 7f087b7fe640 1 -- 172.21.15.42:0/1483313289 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1562 (secure 0 0 0) 0x7f08740b4e20 con 0x7f08840a0a40 2024-09-16T04:12:52.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.474+0000 7f087b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:12:52.478 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.474+0000 7f087b7fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:12:52.478 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":276,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:12:52.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.474+0000 7f08897f6640 10 librados: watch_flush enter 2024-09-16T04:12:52.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.474+0000 7f08897f6640 10 librados: watch_flush exit 2024-09-16T04:12:52.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.474+0000 7f08897f6640 20 client.1034571.objecter shutdown clearing up homeless session... 2024-09-16T04:12:52.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.474+0000 7f08897f6640 10 client.1034571.objecter successfully canceled tick 2024-09-16T04:12:52.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.474+0000 7f08897f6640 1 -- 172.21.15.42:0/1483313289 >> v2:172.21.15.154:6812/254845342 conn(0x7f08640377c0 msgr2=0x7f0864039c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:52.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.474+0000 7f08897f6640 1 --2- 172.21.15.42:0/1483313289 >> v2:172.21.15.154:6812/254845342 conn(0x7f08640377c0 0x7f0864039c80 secure :-1 s=READY pgs=3977 cs=0 l=1 rev1=1 crypto rx=0x7f088406fd10 tx=0x7f086c008040 comp rx=0 tx=0).stop 2024-09-16T04:12:52.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.474+0000 7f08897f6640 10 monclient: shutdown 2024-09-16T04:12:52.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.474+0000 7f08897f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:12:52.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.474+0000 7f08897f6640 1 -- 172.21.15.42:0/1483313289 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f08840a0a40 msgr2=0x7f088412eb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:52.481 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.474+0000 7f08897f6640 1 --2- 172.21.15.42:0/1483313289 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f08840a0a40 0x7f088412eb90 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f08740ae040 tx=0x7f087401abc0 comp rx=0 tx=0).stop 2024-09-16T04:12:52.481 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.474+0000 7f08897f6640 1 -- 172.21.15.42:0/1483313289 shutdown_connections 2024-09-16T04:12:52.481 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.474+0000 7f08897f6640 1 --2- 172.21.15.42:0/1483313289 >> v2:172.21.15.154:6812/254845342 conn(0x7f08640377c0 0x7f0864039c80 unknown :-1 s=CLOSED pgs=3977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:12:52.481 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.474+0000 7f08897f6640 1 --2- 172.21.15.42:0/1483313289 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f08841353a0 0x7f08841cf050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:12:52.481 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.474+0000 7f08897f6640 1 --2- 172.21.15.42:0/1483313289 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f08840a13a0 0x7f088412f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:12:52.481 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.474+0000 7f08897f6640 1 --2- 172.21.15.42:0/1483313289 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f08840a0a40 0x7f088412eb90 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:12:52.481 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.474+0000 7f08897f6640 1 -- 172.21.15.42:0/1483313289 >> 172.21.15.42:0/1483313289 conn(0x7f08840a6640 msgr2=0x7f08840a4640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:12:52.481 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.474+0000 7f08897f6640 1 -- 172.21.15.42:0/1483313289 shutdown_connections 2024-09-16T04:12:52.481 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.474+0000 7f08897f6640 1 -- 172.21.15.42:0/1483313289 wait complete. 2024-09-16T04:12:52.481 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:52.474+0000 7f08897f6640 1 librados: shutdown 2024-09-16T04:12:55.497 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:12:55.630 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:55.626+0000 7f1b9a484640 10 monclient: get_monmap_and_config 2024-09-16T04:12:55.630 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:55.626+0000 7f1b9a484640 10 monclient: build_initial_monmap 2024-09-16T04:12:55.630 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:55.626+0000 7f1b9a484640 10 monclient: monmap: 2024-09-16T04:12:55.630 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:12:55.630 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:12:55.630 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:12:55.629114+0000 2024-09-16T04:12:55.631 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:12:55.629114+0000 2024-09-16T04:12:55.631 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:12:55.631 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:12:55.631 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:12:55.631 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:12:55.631 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:12:55.631 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:12:55.631 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:12:55.631 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:12:55.633 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:55.626+0000 7f1b9a484640 1 Processor -- start 2024-09-16T04:12:55.633 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:55.626+0000 7f1b9a484640 1 -- start start 2024-09-16T04:12:55.633 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:55.626+0000 7f1b9a484640 10 monclient: init 2024-09-16T04:12:55.633 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:55.626+0000 7f1b9a484640 10 monclient: _reopen_session rank -1 2024-09-16T04:12:55.633 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:55.626+0000 7f1b9a484640 10 monclient: _add_conns ranks=[4,2,0,3,1] 2024-09-16T04:12:55.633 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:55.626+0000 7f1b9a484640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f1b940a1330 0x7f1b940a1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:12:55.633 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:55.626+0000 7f1b9a484640 10 monclient(hunting): picked mon.noname-c con 0x7f1b940a1330 addr v2:172.21.15.154:3300/0 2024-09-16T04:12:55.633 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:55.626+0000 7f1b9a484640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1b9406dac0 0x7f1b940a1c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:12:55.633 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:55.626+0000 7f1b9a484640 10 monclient(hunting): picked mon.noname-b con 0x7f1b9406dac0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:12:55.634 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:55.626+0000 7f1b9a484640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1b94135100 addr v1:172.21.15.42:6789/0 2024-09-16T04:12:55.634 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:55.626+0000 7f1b9a484640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1b9409c7d0 con 0x7f1b94135100 2024-09-16T04:12:55.634 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:55.626+0000 7f1b9a484640 10 monclient(hunting): start opening mon connection 2024-09-16T04:12:55.634 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:55.626+0000 7f1b9a484640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f1b9409c090 con 0x7f1b940a1330 2024-09-16T04:12:55.634 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:55.626+0000 7f1b9a484640 10 monclient(hunting): start opening mon connection 2024-09-16T04:12:55.634 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:55.626+0000 7f1b9a484640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f1b9409f610 con 0x7f1b9406dac0 2024-09-16T04:12:55.634 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:55.626+0000 7f1b9a484640 10 monclient(hunting): _renew_subs 2024-09-16T04:12:55.634 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:55.626+0000 7f1b9a484640 10 monclient(hunting): authenticate will time out at 2822.598579s 2024-09-16T04:12:55.634 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:55.626+0000 7f1b937fe640 1 -- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1b9406dac0 msgr2=0x7f1b940a1c50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T04:12:55.634 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:55.626+0000 7f1b937fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1b9406dac0 0x7f1b940a1c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:12:55.634 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:55.626+0000 7f1b98a21640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f1b94135100 0x7f1b94135550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:57606/0 (socket says 172.21.15.42:57606) 2024-09-16T04:12:55.634 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:55.626+0000 7f1b98a21640 1 -- 172.21.15.42:0/2037284165 learned_addr learned my addr 172.21.15.42:0/2037284165 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T04:12:55.634 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:55.626+0000 7f1b93fff640 1 --2- 172.21.15.42:0/2037284165 >> v2:172.21.15.154:3300/0 conn(0x7f1b940a1330 0x7f1b940a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:12:55.635 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:55.626+0000 7f1b93fff640 10 monclient(hunting): get_auth_request con 0x7f1b940a1330 auth_method 0 2024-09-16T04:12:55.635 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:55.626+0000 7f1b93fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:12:55.635 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:55.626+0000 7f1b93fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:55.635 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:55.626+0000 7f1b93fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:55.635 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:55.626+0000 7f1b93fff640 1 -- 172.21.15.42:0/2037284165 >> v2:172.21.15.154:3300/0 conn(0x7f1b940a1330 msgr2=0x7f1b940a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:12:55.635 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:55.626+0000 7f1b93fff640 1 -- 172.21.15.42:0/2037284165 >> v2:172.21.15.154:3300/0 conn(0x7f1b940a1330 msgr2=0x7f1b940a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:12:55.635 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:55.626+0000 7f1b93fff640 1 --2- 172.21.15.42:0/2037284165 >> v2:172.21.15.154:3300/0 conn(0x7f1b940a1330 0x7f1b940a1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:12:55.635 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:55.626+0000 7f1b93fff640 1 --2- 172.21.15.42:0/2037284165 >> v2:172.21.15.154:3300/0 conn(0x7f1b940a1330 0x7f1b940a1710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:12:55.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:55.830+0000 7f1b937fe640 1 -- 172.21.15.42:0/2037284165 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1b9406dac0 msgr2=0x7f1b940a1c50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T04:12:55.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:55.830+0000 7f1b937fe640 1 --2- 172.21.15.42:0/2037284165 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1b9406dac0 0x7f1b940a1c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T04:12:55.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:55.830+0000 7f1b93fff640 1 --2- 172.21.15.42:0/2037284165 >> v2:172.21.15.154:3300/0 conn(0x7f1b940a1330 0x7f1b940a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:12:55.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:55.830+0000 7f1b93fff640 10 monclient(hunting): get_auth_request con 0x7f1b940a1330 auth_method 0 2024-09-16T04:12:55.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:55.830+0000 7f1b93fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:12:55.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:55.830+0000 7f1b93fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:55.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:55.830+0000 7f1b93fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:12:55.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:55.830+0000 7f1b93fff640 1 -- 172.21.15.42:0/2037284165 >> v2:172.21.15.154:3300/0 conn(0x7f1b940a1330 msgr2=0x7f1b940a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:12:55.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:55.830+0000 7f1b93fff640 1 -- 172.21.15.42:0/2037284165 >> v2:172.21.15.154:3300/0 conn(0x7f1b940a1330 msgr2=0x7f1b940a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:12:55.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:55.830+0000 7f1b93fff640 1 --2- 172.21.15.42:0/2037284165 >> v2:172.21.15.154:3300/0 conn(0x7f1b940a1330 0x7f1b940a1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:12:55.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:55.830+0000 7f1b93fff640 1 --2- 172.21.15.42:0/2037284165 >> v2:172.21.15.154:3300/0 conn(0x7f1b940a1330 0x7f1b940a1710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T04:12:56.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:56.234+0000 7f1b937fe640 1 -- 172.21.15.42:0/2037284165 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1b9406dac0 msgr2=0x7f1b940a1c50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T04:12:56.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:56.234+0000 7f1b937fe640 1 --2- 172.21.15.42:0/2037284165 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1b9406dac0 0x7f1b940a1c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T04:12:56.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:56.234+0000 7f1b93fff640 1 --2- 172.21.15.42:0/2037284165 >> v2:172.21.15.154:3300/0 conn(0x7f1b940a1330 0x7f1b940a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:12:56.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:56.234+0000 7f1b93fff640 10 monclient(hunting): get_auth_request con 0x7f1b940a1330 auth_method 0 2024-09-16T04:12:56.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:56.234+0000 7f1b93fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:12:56.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:56.234+0000 7f1b93fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:56.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:56.234+0000 7f1b93fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:12:56.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:56.234+0000 7f1b93fff640 1 -- 172.21.15.42:0/2037284165 >> v2:172.21.15.154:3300/0 conn(0x7f1b940a1330 msgr2=0x7f1b940a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:12:56.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:56.234+0000 7f1b93fff640 1 -- 172.21.15.42:0/2037284165 >> v2:172.21.15.154:3300/0 conn(0x7f1b940a1330 msgr2=0x7f1b940a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:12:56.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:56.234+0000 7f1b93fff640 1 --2- 172.21.15.42:0/2037284165 >> v2:172.21.15.154:3300/0 conn(0x7f1b940a1330 0x7f1b940a1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:12:56.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:56.234+0000 7f1b93fff640 1 --2- 172.21.15.42:0/2037284165 >> v2:172.21.15.154:3300/0 conn(0x7f1b940a1330 0x7f1b940a1710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T04:12:57.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:57.038+0000 7f1b937fe640 1 -- 172.21.15.42:0/2037284165 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1b9406dac0 msgr2=0x7f1b940a1c50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T04:12:57.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:57.038+0000 7f1b937fe640 1 --2- 172.21.15.42:0/2037284165 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1b9406dac0 0x7f1b940a1c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T04:12:57.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:57.038+0000 7f1b93fff640 1 --2- 172.21.15.42:0/2037284165 >> v2:172.21.15.154:3300/0 conn(0x7f1b940a1330 0x7f1b940a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:12:57.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:57.038+0000 7f1b93fff640 10 monclient(hunting): get_auth_request con 0x7f1b940a1330 auth_method 0 2024-09-16T04:12:57.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:57.038+0000 7f1b93fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:12:57.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:57.038+0000 7f1b93fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:57.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:57.038+0000 7f1b93fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:12:57.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:57.038+0000 7f1b93fff640 1 -- 172.21.15.42:0/2037284165 >> v2:172.21.15.154:3300/0 conn(0x7f1b940a1330 msgr2=0x7f1b940a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:12:57.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:57.038+0000 7f1b93fff640 1 -- 172.21.15.42:0/2037284165 >> v2:172.21.15.154:3300/0 conn(0x7f1b940a1330 msgr2=0x7f1b940a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:12:57.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:57.038+0000 7f1b93fff640 1 --2- 172.21.15.42:0/2037284165 >> v2:172.21.15.154:3300/0 conn(0x7f1b940a1330 0x7f1b940a1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:12:57.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:57.038+0000 7f1b93fff640 1 --2- 172.21.15.42:0/2037284165 >> v2:172.21.15.154:3300/0 conn(0x7f1b940a1330 0x7f1b940a1710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T04:12:58.632 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:58.630+0000 7f1b91ffb640 10 monclient(hunting): tick 2024-09-16T04:12:58.632 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:58.630+0000 7f1b91ffb640 10 monclient(hunting): _check_auth_tickets 2024-09-16T04:12:58.632 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:58.630+0000 7f1b91ffb640 1 monclient(hunting): continuing hunt 2024-09-16T04:12:58.633 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:58.630+0000 7f1b91ffb640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T04:12:58.633 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:58.630+0000 7f1b91ffb640 1 -- 172.21.15.42:0/2037284165 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1b9406dac0 msgr2=0x7f1b940a1c50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:12:58.633 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:58.630+0000 7f1b91ffb640 1 --2- 172.21.15.42:0/2037284165 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1b9406dac0 0x7f1b940a1c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:12:58.633 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:58.630+0000 7f1b91ffb640 1 -- 172.21.15.42:0/2037284165 >> v2:172.21.15.154:3300/0 conn(0x7f1b940a1330 msgr2=0x7f1b940a1710 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:12:58.633 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:58.630+0000 7f1b91ffb640 1 --2- 172.21.15.42:0/2037284165 >> v2:172.21.15.154:3300/0 conn(0x7f1b940a1330 0x7f1b940a1710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:12:58.633 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:58.630+0000 7f1b91ffb640 1 -- 172.21.15.42:0/2037284165 >> v1:172.21.15.42:6789/0 conn(0x7f1b94135100 legacy=0x7f1b94135550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:12:58.633 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:58.630+0000 7f1b91ffb640 10 monclient: _add_conns ranks=[1,3] 2024-09-16T04:12:58.633 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:58.630+0000 7f1b91ffb640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1b780036f0 addr v1:172.21.15.154:6789/0 2024-09-16T04:12:58.633 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:58.630+0000 7f1b91ffb640 1 --2- 172.21.15.42:0/2037284165 >> v2:172.21.15.42:3300/0 conn(0x7f1b78007380 0x7f1b78009840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:12:58.633 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:58.630+0000 7f1b91ffb640 10 monclient(hunting): picked mon.noname-a con 0x7f1b78007380 addr v2:172.21.15.42:3300/0 2024-09-16T04:12:58.633 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:58.630+0000 7f1b91ffb640 1 -- 172.21.15.42:0/2037284165 --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1b9409f8f0 con 0x7f1b780036f0 2024-09-16T04:12:58.633 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:58.630+0000 7f1b91ffb640 10 monclient(hunting): start opening mon connection 2024-09-16T04:12:58.633 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:58.630+0000 7f1b91ffb640 1 -- 172.21.15.42:0/2037284165 --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f1b9409c090 con 0x7f1b78007380 2024-09-16T04:12:58.633 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:58.630+0000 7f1b91ffb640 10 monclient(hunting): _renew_subs 2024-09-16T04:12:58.634 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:58.630+0000 7f1b937fe640 1 --2- 172.21.15.42:0/2037284165 >> v2:172.21.15.42:3300/0 conn(0x7f1b78007380 0x7f1b78009840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:12:58.634 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:58.630+0000 7f1b937fe640 10 monclient(hunting): get_auth_request con 0x7f1b78007380 auth_method 0 2024-09-16T04:12:58.634 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:58.630+0000 7f1b937fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:12:58.634 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:58.630+0000 7f1b937fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:58.634 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:58.630+0000 7f1b937fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:12:58.634 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:58.630+0000 7f1b937fe640 1 -- 172.21.15.42:0/2037284165 >> v2:172.21.15.42:3300/0 conn(0x7f1b78007380 msgr2=0x7f1b78009840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:12:58.634 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:58.630+0000 7f1b937fe640 1 -- 172.21.15.42:0/2037284165 >> v2:172.21.15.42:3300/0 conn(0x7f1b78007380 msgr2=0x7f1b78009840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:12:58.634 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:58.630+0000 7f1b937fe640 1 --2- 172.21.15.42:0/2037284165 >> v2:172.21.15.42:3300/0 conn(0x7f1b78007380 0x7f1b78009840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:12:58.634 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:58.630+0000 7f1b937fe640 1 --2- 172.21.15.42:0/2037284165 >> v2:172.21.15.42:3300/0 conn(0x7f1b78007380 0x7f1b78009840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:12:58.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:58.834+0000 7f1b937fe640 1 --2- 172.21.15.42:0/2037284165 >> v2:172.21.15.42:3300/0 conn(0x7f1b78007380 0x7f1b78009840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:12:58.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:58.834+0000 7f1b937fe640 10 monclient(hunting): get_auth_request con 0x7f1b78007380 auth_method 0 2024-09-16T04:12:58.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:58.834+0000 7f1b937fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:12:58.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:58.834+0000 7f1b937fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:58.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:58.834+0000 7f1b937fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:12:58.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:58.834+0000 7f1b937fe640 1 -- 172.21.15.42:0/2037284165 >> v2:172.21.15.42:3300/0 conn(0x7f1b78007380 msgr2=0x7f1b78009840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:12:58.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:58.834+0000 7f1b937fe640 1 -- 172.21.15.42:0/2037284165 >> v2:172.21.15.42:3300/0 conn(0x7f1b78007380 msgr2=0x7f1b78009840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:12:58.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:58.834+0000 7f1b937fe640 1 --2- 172.21.15.42:0/2037284165 >> v2:172.21.15.42:3300/0 conn(0x7f1b78007380 0x7f1b78009840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:12:58.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:58.834+0000 7f1b937fe640 1 --2- 172.21.15.42:0/2037284165 >> v2:172.21.15.42:3300/0 conn(0x7f1b78007380 0x7f1b78009840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T04:12:59.242 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:59.238+0000 7f1b937fe640 1 --2- 172.21.15.42:0/2037284165 >> v2:172.21.15.42:3300/0 conn(0x7f1b78007380 0x7f1b78009840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:12:59.242 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:59.238+0000 7f1b937fe640 10 monclient(hunting): get_auth_request con 0x7f1b78007380 auth_method 0 2024-09-16T04:12:59.242 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:59.238+0000 7f1b937fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:12:59.242 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:59.238+0000 7f1b937fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:12:59.242 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:59.238+0000 7f1b937fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:12:59.242 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:59.238+0000 7f1b937fe640 1 -- 172.21.15.42:0/2037284165 >> v2:172.21.15.42:3300/0 conn(0x7f1b78007380 msgr2=0x7f1b78009840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:12:59.242 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:59.238+0000 7f1b937fe640 1 -- 172.21.15.42:0/2037284165 >> v2:172.21.15.42:3300/0 conn(0x7f1b78007380 msgr2=0x7f1b78009840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:12:59.242 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:59.238+0000 7f1b937fe640 1 --2- 172.21.15.42:0/2037284165 >> v2:172.21.15.42:3300/0 conn(0x7f1b78007380 0x7f1b78009840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:12:59.242 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:12:59.238+0000 7f1b937fe640 1 --2- 172.21.15.42:0/2037284165 >> v2:172.21.15.42:3300/0 conn(0x7f1b78007380 0x7f1b78009840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T04:13:00.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.042+0000 7f1b937fe640 1 --2- 172.21.15.42:0/2037284165 >> v2:172.21.15.42:3300/0 conn(0x7f1b78007380 0x7f1b78009840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:13:00.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.042+0000 7f1b937fe640 10 monclient(hunting): get_auth_request con 0x7f1b78007380 auth_method 0 2024-09-16T04:13:00.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.042+0000 7f1b937fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:13:00.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.042+0000 7f1b937fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:00.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.042+0000 7f1b937fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:13:00.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.042+0000 7f1b937fe640 1 -- 172.21.15.42:0/2037284165 >> v2:172.21.15.42:3300/0 conn(0x7f1b78007380 msgr2=0x7f1b78009840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:13:00.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.042+0000 7f1b937fe640 1 -- 172.21.15.42:0/2037284165 >> v2:172.21.15.42:3300/0 conn(0x7f1b78007380 msgr2=0x7f1b78009840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:13:00.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.042+0000 7f1b937fe640 1 --2- 172.21.15.42:0/2037284165 >> v2:172.21.15.42:3300/0 conn(0x7f1b78007380 0x7f1b78009840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:13:00.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.042+0000 7f1b937fe640 1 --2- 172.21.15.42:0/2037284165 >> v2:172.21.15.42:3300/0 conn(0x7f1b78007380 0x7f1b78009840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T04:13:00.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.206+0000 7f1b92ffd640 1 -- 172.21.15.42:0/2037284165 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1578846285 0 0) 0x7f1b9409f8f0 con 0x7f1b780036f0 2024-09-16T04:13:00.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.206+0000 7f1b92ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:13:00.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.206+0000 7f1b92ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:00.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.206+0000 7f1b92ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:13:00.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.206+0000 7f1b92ffd640 10 monclient(hunting): my global_id is 1074232 2024-09-16T04:13:00.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.206+0000 7f1b92ffd640 1 -- 172.21.15.42:0/2037284165 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1b74003530 con 0x7f1b780036f0 2024-09-16T04:13:00.214 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.206+0000 7f1b92ffd640 1 -- 172.21.15.42:0/2037284165 <== mon.1 v1:172.21.15.154:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1508256698 0 0) 0x7f1b74003530 con 0x7f1b780036f0 2024-09-16T04:13:00.214 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.206+0000 7f1b92ffd640 1 -- 172.21.15.42:0/2037284165 >> v2:172.21.15.42:3300/0 conn(0x7f1b78007380 msgr2=0x7f1b78009840 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:13:00.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.206+0000 7f1b92ffd640 1 --2- 172.21.15.42:0/2037284165 >> v2:172.21.15.42:3300/0 conn(0x7f1b78007380 0x7f1b78009840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:13:00.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.206+0000 7f1b92ffd640 10 monclient: _finish_hunting 0 2024-09-16T04:13:00.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.206+0000 7f1b92ffd640 1 monclient: found mon.noname-c-legacy 2024-09-16T04:13:00.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.206+0000 7f1b92ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:13:00.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.206+0000 7f1b92ffd640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.154:6789/0 2024-09-16T04:13:00.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.206+0000 7f1b92ffd640 1 -- 172.21.15.42:0/2037284165 --> v1:172.21.15.154:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b7800afb0 con 0x7f1b780036f0 2024-09-16T04:13:00.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.206+0000 7f1b92ffd640 10 monclient: _finish_auth 0 2024-09-16T04:13:00.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.206+0000 7f1b92ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:13:00.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.206+0000 7f1b92ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:13:00.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.206+0000 7f1b92ffd640 1 -- 172.21.15.42:0/2037284165 <== mon.1 v1:172.21.15.154:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2177413734 0 0) 0x7f1b84009a00 con 0x7f1b780036f0 2024-09-16T04:13:00.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.206+0000 7f1b9a484640 5 monclient: authenticate success, global_id 1074232 2024-09-16T04:13:00.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.206+0000 7f1b92ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:13:00.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.206+0000 7f1b92ffd640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-16T04:13:00.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.206+0000 7f1b92ffd640 10 monclient: dump: 2024-09-16T04:13:00.216 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:13:00.216 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:13:00.216 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:00.216 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:00.216 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:00.216 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:13:00.216 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:13:00.216 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:13:00.216 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:13:00.216 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:13:00.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.206+0000 7f1b92ffd640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] but i'm connected to v1:172.21.15.154:6789/0, reconnecting 2024-09-16T04:13:00.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.206+0000 7f1b92ffd640 10 monclient: _reopen_session rank -1 2024-09-16T04:13:00.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.206+0000 7f1b92ffd640 1 -- 172.21.15.42:0/2037284165 >> v1:172.21.15.154:6789/0 conn(0x7f1b780036f0 legacy=0x7f1b78005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:00.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.206+0000 7f1b98a21640 1 -- 172.21.15.42:0/2037284165 reap_dead start 2024-09-16T04:13:00.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.206+0000 7f1b92ffd640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:13:00.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.206+0000 7f1b92ffd640 1 --2- 172.21.15.42:0/2037284165 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1b740039c0 0x7f1b74003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:13:00.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.206+0000 7f1b92ffd640 10 monclient(hunting): picked mon.b con 0x7f1b740039c0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:13:00.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.206+0000 7f1b92ffd640 1 --2- 172.21.15.42:0/2037284165 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1b74004810 0x7f1b74004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:13:00.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.206+0000 7f1b92ffd640 10 monclient(hunting): picked mon.c con 0x7f1b74004810 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:13:00.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.206+0000 7f1b92ffd640 1 --2- 172.21.15.42:0/2037284165 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1b74005220 0x7f1b7400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:13:00.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.206+0000 7f1b92ffd640 10 monclient(hunting): picked mon.a con 0x7f1b74005220 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:13:00.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.206+0000 7f1b92ffd640 10 monclient(hunting): start opening mon connection 2024-09-16T04:13:00.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.206+0000 7f1b92ffd640 1 -- 172.21.15.42:0/2037284165 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f1b9409c090 con 0x7f1b74005220 2024-09-16T04:13:00.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.206+0000 7f1b92ffd640 10 monclient(hunting): start opening mon connection 2024-09-16T04:13:00.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.206+0000 7f1b92ffd640 1 -- 172.21.15.42:0/2037284165 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f1b7400c150 con 0x7f1b740039c0 2024-09-16T04:13:00.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.206+0000 7f1b92ffd640 10 monclient(hunting): start opening mon connection 2024-09-16T04:13:00.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.206+0000 7f1b937fe640 1 -- 172.21.15.42:0/2037284165 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1b74004810 msgr2=0x7f1b74004c90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T04:13:00.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.206+0000 7f1b92ffd640 1 -- 172.21.15.42:0/2037284165 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f1b7400c350 con 0x7f1b74004810 2024-09-16T04:13:00.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.206+0000 7f1b937fe640 1 --2- 172.21.15.42:0/2037284165 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1b74004810 0x7f1b74004c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:13:00.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.206+0000 7f1b92ffd640 10 monclient(hunting): _renew_subs 2024-09-16T04:13:00.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.206+0000 7f1b92ffd640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:13:00.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.210+0000 7f1b9a484640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-16T04:13:00.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.210+0000 7f1b98a21640 1 --2- 172.21.15.42:0/2037284165 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1b74005220 0x7f1b7400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:13:00.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.210+0000 7f1b93fff640 1 --2- 172.21.15.42:0/2037284165 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1b740039c0 0x7f1b74003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:13:00.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.210+0000 7f1b93fff640 10 monclient(hunting): get_auth_request con 0x7f1b740039c0 auth_method 0 2024-09-16T04:13:00.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.210+0000 7f1b93fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:13:00.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.210+0000 7f1b93fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:00.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.210+0000 7f1b93fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:13:00.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.210+0000 7f1b98a21640 10 monclient(hunting): get_auth_request con 0x7f1b74005220 auth_method 0 2024-09-16T04:13:00.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.210+0000 7f1b98a21640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:13:00.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.210+0000 7f1b98a21640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:00.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.210+0000 7f1b98a21640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:13:00.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.210+0000 7f1b93fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:13:00.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.210+0000 7f1b93fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:13:00.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.210+0000 7f1b93fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:13:00.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.210+0000 7f1b98a21640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:13:00.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.210+0000 7f1b98a21640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:13:00.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.210+0000 7f1b98a21640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:13:00.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.210+0000 7f1b93fff640 10 monclient(hunting): handle_auth_done global_id 1074232 payload 293 2024-09-16T04:13:00.220 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.210+0000 7f1b93fff640 1 -- 172.21.15.42:0/2037284165 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1b74004810 msgr2=0x7f1b74004c90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:13:00.220 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.210+0000 7f1b93fff640 1 --2- 172.21.15.42:0/2037284165 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1b74004810 0x7f1b74004c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:13:00.220 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.210+0000 7f1b93fff640 1 -- 172.21.15.42:0/2037284165 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1b74005220 msgr2=0x7f1b7400b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:13:00.220 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.210+0000 7f1b93fff640 1 --2- 172.21.15.42:0/2037284165 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1b74005220 0x7f1b7400b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:13:00.220 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.210+0000 7f1b93fff640 10 monclient: _finish_hunting 0 2024-09-16T04:13:00.220 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.210+0000 7f1b93fff640 1 monclient: found mon.b 2024-09-16T04:13:00.220 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.210+0000 7f1b93fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:13:00.220 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.210+0000 7f1b93fff640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.154:3300/0 2024-09-16T04:13:00.220 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.210+0000 7f1b93fff640 1 -- 172.21.15.42:0/2037284165 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b74003530 con 0x7f1b740039c0 2024-09-16T04:13:00.220 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.210+0000 7f1b93fff640 10 monclient: _finish_auth 0 2024-09-16T04:13:00.220 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.210+0000 7f1b93fff640 10 monclient: _check_auth_tickets 2024-09-16T04:13:00.220 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.210+0000 7f1b93fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:13:00.220 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.210+0000 7f1b98a21640 1 --2- 172.21.15.42:0/2037284165 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1b74005220 0x7f1b7400b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:13:00.222 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.214+0000 7f1b93fff640 1 --2- 172.21.15.42:0/2037284165 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1b740039c0 0x7f1b74003dc0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f1b84009610 tx=0x7f1b840201e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=45a66bc74b434a90 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:13:00.222 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.214+0000 7f1b92ffd640 1 -- 172.21.15.42:0/2037284165 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b840af070 con 0x7f1b740039c0 2024-09-16T04:13:00.222 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.214+0000 7f1b92ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:13:00.222 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.214+0000 7f1b92ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:13:00.222 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.214+0000 7f1b92ffd640 10 monclient: dump: 2024-09-16T04:13:00.222 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:13:00.222 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:13:00.222 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:00.222 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:00.222 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:00.222 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:13:00.223 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:13:00.223 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:13:00.223 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:13:00.223 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:13:00.223 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.214+0000 7f1b92ffd640 1 -- 172.21.15.42:0/2037284165 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1b840b3400 con 0x7f1b740039c0 2024-09-16T04:13:00.223 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.214+0000 7f1b92ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:13:00.223 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.214+0000 7f1b92ffd640 1 -- 172.21.15.42:0/2037284165 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b84003bf0 con 0x7f1b740039c0 2024-09-16T04:13:00.223 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.214+0000 7f1b9a484640 10 monclient: get_monmap_and_config success 2024-09-16T04:13:00.223 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.214+0000 7f1b9a484640 4 set_mon_vals no callback set 2024-09-16T04:13:00.223 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.214+0000 7f1b92ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:13:00.223 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.214+0000 7f1b92ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:13:00.223 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.214+0000 7f1b92ffd640 10 monclient: dump: 2024-09-16T04:13:00.223 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:13:00.223 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:13:00.224 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:00.224 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:00.224 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:00.224 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:13:00.224 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:13:00.224 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:13:00.224 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:13:00.224 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:13:00.224 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.214+0000 7f1b9a484640 10 monclient: shutdown 2024-09-16T04:13:00.224 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.214+0000 7f1b9a484640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:13:00.224 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.214+0000 7f1b9a484640 1 -- 172.21.15.42:0/2037284165 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1b740039c0 msgr2=0x7f1b74003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:00.224 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.214+0000 7f1b9a484640 1 --2- 172.21.15.42:0/2037284165 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1b740039c0 0x7f1b74003dc0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f1b84009610 tx=0x7f1b840201e0 comp rx=0 tx=0).stop 2024-09-16T04:13:00.224 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.214+0000 7f1b9a484640 1 -- 172.21.15.42:0/2037284165 shutdown_connections 2024-09-16T04:13:00.225 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.214+0000 7f1b9a484640 1 --2- 172.21.15.42:0/2037284165 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1b74005220 0x7f1b7400b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:13:00.225 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.214+0000 7f1b9a484640 1 --2- 172.21.15.42:0/2037284165 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1b74004810 0x7f1b74004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:13:00.225 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.214+0000 7f1b9a484640 1 --2- 172.21.15.42:0/2037284165 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1b740039c0 0x7f1b74003dc0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:13:00.225 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.214+0000 7f1b9a484640 1 -- 172.21.15.42:0/2037284165 >> 172.21.15.42:0/2037284165 conn(0x7f1b940a2420 msgr2=0x7f1b940a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:13:00.225 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.214+0000 7f1b9a484640 1 -- 172.21.15.42:0/2037284165 shutdown_connections 2024-09-16T04:13:00.225 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.214+0000 7f1b9a484640 1 -- 172.21.15.42:0/2037284165 wait complete. 2024-09-16T04:13:00.225 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.218+0000 7f1b9a484640 10 monclient: build_initial_monmap 2024-09-16T04:13:00.225 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.218+0000 7f1b9a484640 10 monclient: monmap: 2024-09-16T04:13:00.225 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:13:00.225 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:13:00.225 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:13:00.225 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:13:00.225 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:13:00.225 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:13:00.226 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:13:00.226 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:13:00.226 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:13:00.226 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:13:00.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.218+0000 7f1b9a484640 1 librados: starting msgr at 2024-09-16T04:13:00.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.218+0000 7f1b9a484640 1 librados: starting objecter 2024-09-16T04:13:00.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.218+0000 7f1b9a484640 1 Processor -- start 2024-09-16T04:13:00.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.218+0000 7f1b9a484640 1 -- start start 2024-09-16T04:13:00.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.218+0000 7f1b9a484640 1 librados: setting wanted keys 2024-09-16T04:13:00.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.218+0000 7f1b9a484640 1 librados: calling monclient init 2024-09-16T04:13:00.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.218+0000 7f1b9a484640 10 monclient: init 2024-09-16T04:13:00.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.218+0000 7f1b9a484640 10 monclient: _reopen_session rank -1 2024-09-16T04:13:00.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.218+0000 7f1b9a484640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:13:00.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.218+0000 7f1b9a484640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1b740039c0 0x7f1b94133d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:13:00.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.218+0000 7f1b9a484640 10 monclient(hunting): picked mon.noname-b con 0x7f1b740039c0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:13:00.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.218+0000 7f1b9a484640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1b74004810 0x7f1b941204d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:13:00.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.218+0000 7f1b9a484640 10 monclient(hunting): picked mon.noname-a con 0x7f1b74004810 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:13:00.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.218+0000 7f1b9a484640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1b74005220 0x7f1b94120a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:13:00.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.218+0000 7f1b9a484640 10 monclient(hunting): picked mon.noname-c con 0x7f1b74005220 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:13:00.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.218+0000 7f1b9a484640 10 monclient(hunting): start opening mon connection 2024-09-16T04:13:00.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.218+0000 7f1b9a484640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f1b940a32f0 con 0x7f1b74004810 2024-09-16T04:13:00.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.218+0000 7f1b9a484640 10 monclient(hunting): start opening mon connection 2024-09-16T04:13:00.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.218+0000 7f1b9a484640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f1b940a3170 con 0x7f1b740039c0 2024-09-16T04:13:00.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.218+0000 7f1b9a484640 10 monclient(hunting): start opening mon connection 2024-09-16T04:13:00.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.218+0000 7f1b9a484640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f1b9409ed70 con 0x7f1b74005220 2024-09-16T04:13:00.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.218+0000 7f1b9a484640 10 monclient(hunting): _renew_subs 2024-09-16T04:13:00.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.218+0000 7f1b9a484640 10 monclient(hunting): authenticate will time out at 2827.191061s 2024-09-16T04:13:00.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.218+0000 7f1b98a21640 1 -- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1b74005220 msgr2=0x7f1b94120a10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T04:13:00.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.218+0000 7f1b937fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1b74004810 0x7f1b941204d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:13:00.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.218+0000 7f1b98a21640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1b74005220 0x7f1b94120a10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:13:00.233 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.218+0000 7f1b937fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1b74004810 0x7f1b941204d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:50178/0 (socket says 172.21.15.42:50178) 2024-09-16T04:13:00.233 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.218+0000 7f1b937fe640 1 -- 172.21.15.42:0/1258437645 learned_addr learned my addr 172.21.15.42:0/1258437645 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:13:00.233 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.218+0000 7f1b937fe640 10 monclient(hunting): get_auth_request con 0x7f1b74004810 auth_method 0 2024-09-16T04:13:00.233 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.218+0000 7f1b937fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:13:00.233 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.218+0000 7f1b937fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:00.233 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.218+0000 7f1b937fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:13:00.233 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.218+0000 7f1b937fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:13:00.233 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.218+0000 7f1b937fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:13:00.233 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.218+0000 7f1b937fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:13:00.233 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.218+0000 7f1b93fff640 1 --2- 172.21.15.42:0/1258437645 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1b740039c0 0x7f1b94133d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:13:00.233 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.218+0000 7f1b93fff640 10 monclient(hunting): get_auth_request con 0x7f1b740039c0 auth_method 0 2024-09-16T04:13:00.233 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.218+0000 7f1b93fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:13:00.233 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.218+0000 7f1b93fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:00.233 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.218+0000 7f1b93fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:13:00.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.218+0000 7f1b937fe640 10 monclient(hunting): handle_auth_done global_id 1034574 payload 824 2024-09-16T04:13:00.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.218+0000 7f1b937fe640 1 -- 172.21.15.42:0/1258437645 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1b74005220 msgr2=0x7f1b94120a10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:13:00.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.218+0000 7f1b937fe640 1 --2- 172.21.15.42:0/1258437645 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1b74005220 0x7f1b94120a10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:13:00.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.218+0000 7f1b937fe640 1 -- 172.21.15.42:0/1258437645 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1b740039c0 msgr2=0x7f1b94133d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:00.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.218+0000 7f1b937fe640 1 --2- 172.21.15.42:0/1258437645 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1b740039c0 0x7f1b94133d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:13:00.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.218+0000 7f1b937fe640 10 monclient: _finish_hunting 0 2024-09-16T04:13:00.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.218+0000 7f1b937fe640 1 monclient: found mon.noname-a 2024-09-16T04:13:00.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.218+0000 7f1b937fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:13:00.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.218+0000 7f1b937fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:13:00.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.218+0000 7f1b937fe640 1 -- 172.21.15.42:0/1258437645 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b94073a30 con 0x7f1b74004810 2024-09-16T04:13:00.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.218+0000 7f1b93fff640 1 --2- 172.21.15.42:0/1258437645 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1b740039c0 0x7f1b94133d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T04:13:00.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.222+0000 7f1b937fe640 1 --2- 172.21.15.42:0/1258437645 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1b74004810 0x7f1b941204d0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f1b7c00ba60 tx=0x7f1b7c00bf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:13:00.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.222+0000 7f1b90ff9640 10 client.?.objecter ms_handle_connect 0x7f1b74004810 2024-09-16T04:13:00.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.222+0000 7f1b90ff9640 10 client.?.objecter resend_mon_ops 2024-09-16T04:13:00.235 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.222+0000 7f1b90ff9640 1 -- 172.21.15.42:0/1258437645 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b7c005a20 con 0x7f1b74004810 2024-09-16T04:13:00.235 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.222+0000 7f1b90ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:13:00.235 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.222+0000 7f1b90ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:13:00.235 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.222+0000 7f1b90ff9640 10 monclient: dump: 2024-09-16T04:13:00.235 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:13:00.235 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:13:00.235 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:00.235 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:00.235 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:00.235 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:13:00.235 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:13:00.235 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:13:00.235 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:13:00.235 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:13:00.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.222+0000 7f1b90ff9640 10 monclient: _finish_auth 0 2024-09-16T04:13:00.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.222+0000 7f1b90ff9640 10 monclient: _check_auth_tickets 2024-09-16T04:13:00.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.222+0000 7f1b90ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:13:00.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.222+0000 7f1b90ff9640 1 -- 172.21.15.42:0/1258437645 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1b7c01c070 con 0x7f1b74004810 2024-09-16T04:13:00.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.222+0000 7f1b90ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T04:13:00.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.222+0000 7f1b90ff9640 1 -- 172.21.15.42:0/1258437645 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b7c017630 con 0x7f1b74004810 2024-09-16T04:13:00.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.222+0000 7f1b90ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:13:00.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.222+0000 7f1b90ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:13:00.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.222+0000 7f1b90ff9640 10 monclient: dump: 2024-09-16T04:13:00.236 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:13:00.236 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:13:00.236 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:00.236 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:00.237 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:00.237 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:13:00.237 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:13:00.237 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:13:00.237 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:13:00.237 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:13:00.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.222+0000 7f1b927fc640 4 set_mon_vals no callback set 2024-09-16T04:13:00.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.222+0000 7f1b9a484640 5 monclient: authenticate success, global_id 1034574 2024-09-16T04:13:00.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.222+0000 7f1b9a484640 10 monclient: _renew_subs 2024-09-16T04:13:00.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.222+0000 7f1b9a484640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:13:00.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.222+0000 7f1b9a484640 1 -- 172.21.15.42:0/1258437645 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b94124180 con 0x7f1b74004810 2024-09-16T04:13:00.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.222+0000 7f1b9a484640 10 client.1034574.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:13:00.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.222+0000 7f1b9a484640 10 monclient: _renew_subs 2024-09-16T04:13:00.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.222+0000 7f1b9a484640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:13:00.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.222+0000 7f1b9a484640 1 -- 172.21.15.42:0/1258437645 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b94122cd0 con 0x7f1b74004810 2024-09-16T04:13:00.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.222+0000 7f1b9a484640 1 librados: init done 2024-09-16T04:13:00.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.222+0000 7f1b90ff9640 1 -- 172.21.15.42:0/1258437645 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f1b7c0177d0 con 0x7f1b74004810 2024-09-16T04:13:00.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.222+0000 7f1b9a484640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:13:00.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.222+0000 7f1b9a484640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:13:00.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.222+0000 7f1b9a484640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:13:00.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.222+0000 7f1b9a484640 1 -- 172.21.15.42:0/1258437645 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b940a1710 con 0x7f1b74004810 2024-09-16T04:13:00.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.222+0000 7f1b90ff9640 1 --2- 172.21.15.42:0/1258437645 >> v2:172.21.15.154:6812/254845342 conn(0x7f1b78038000 0x7f1b7803a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:13:00.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.222+0000 7f1b90ff9640 1 -- 172.21.15.42:0/1258437645 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(706..706 src has 681..706) ==== 5040+0+0 (secure 0 0 0) 0x7f1b7c049880 con 0x7f1b74004810 2024-09-16T04:13:00.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.222+0000 7f1b90ff9640 10 client.1034574.objecter ms_dispatch 0x7f1b94000ba0 osd_map(706..706 src has 681..706) 2024-09-16T04:13:00.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.222+0000 7f1b90ff9640 3 client.1034574.objecter handle_osd_map got epochs [706,706] > 0 2024-09-16T04:13:00.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.222+0000 7f1b90ff9640 3 client.1034574.objecter handle_osd_map decoding full epoch 706 2024-09-16T04:13:00.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.222+0000 7f1b90ff9640 20 client.1034574.objecter dump_active .. 0 homeless 2024-09-16T04:13:00.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.222+0000 7f1b93fff640 1 --2- 172.21.15.42:0/1258437645 >> v2:172.21.15.154:6812/254845342 conn(0x7f1b78038000 0x7f1b7803a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:13:00.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.222+0000 7f1b93fff640 10 monclient: get_auth_request con 0x7f1b78038000 auth_method 0 2024-09-16T04:13:00.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.226+0000 7f1b90ff9640 10 client.1034574.objecter ms_handle_connect 0x7f1b78038000 2024-09-16T04:13:00.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.226+0000 7f1b93fff640 1 --2- 172.21.15.42:0/1258437645 >> v2:172.21.15.154:6812/254845342 conn(0x7f1b78038000 0x7f1b7803a4c0 secure :-1 s=READY pgs=3978 cs=0 l=1 rev1=1 crypto rx=0x7f1b740057f0 tx=0x7f1b84004ac0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:13:00.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.226+0000 7f1b90ff9640 1 -- 172.21.15.42:0/1258437645 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f1b7c026020 con 0x7f1b74004810 2024-09-16T04:13:00.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.226+0000 7f1b90ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:13:00.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.226+0000 7f1b90ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:13:00.479 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.474+0000 7f1b9a484640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:13:00.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.474+0000 7f1b9a484640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:13:00.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.474+0000 7f1b9a484640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:13:00.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.474+0000 7f1b9a484640 1 -- 172.21.15.42:0/1258437645 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f1b94124520 con 0x7f1b74004810 2024-09-16T04:13:00.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.474+0000 7f1b90ff9640 1 -- 172.21.15.42:0/1258437645 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1555 (secure 0 0 0) 0x7f1b94073a30 con 0x7f1b74004810 2024-09-16T04:13:00.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.474+0000 7f1b90ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:13:00.480 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.474+0000 7f1b90ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:13:00.481 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":280,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:13:00.483 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.478+0000 7f1b9a484640 10 librados: watch_flush enter 2024-09-16T04:13:00.483 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.478+0000 7f1b9a484640 10 librados: watch_flush exit 2024-09-16T04:13:00.483 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.478+0000 7f1b9a484640 20 client.1034574.objecter shutdown clearing up homeless session... 2024-09-16T04:13:00.483 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.478+0000 7f1b9a484640 10 client.1034574.objecter successfully canceled tick 2024-09-16T04:13:00.483 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.478+0000 7f1b9a484640 1 -- 172.21.15.42:0/1258437645 >> v2:172.21.15.154:6812/254845342 conn(0x7f1b78038000 msgr2=0x7f1b7803a4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:00.483 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.478+0000 7f1b9a484640 1 --2- 172.21.15.42:0/1258437645 >> v2:172.21.15.154:6812/254845342 conn(0x7f1b78038000 0x7f1b7803a4c0 secure :-1 s=READY pgs=3978 cs=0 l=1 rev1=1 crypto rx=0x7f1b740057f0 tx=0x7f1b84004ac0 comp rx=0 tx=0).stop 2024-09-16T04:13:00.483 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.478+0000 7f1b9a484640 10 monclient: shutdown 2024-09-16T04:13:00.483 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.478+0000 7f1b9a484640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:13:00.483 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.478+0000 7f1b9a484640 1 -- 172.21.15.42:0/1258437645 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1b74004810 msgr2=0x7f1b941204d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:00.483 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.478+0000 7f1b9a484640 1 --2- 172.21.15.42:0/1258437645 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1b74004810 0x7f1b941204d0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f1b7c00ba60 tx=0x7f1b7c00bf20 comp rx=0 tx=0).stop 2024-09-16T04:13:00.483 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.478+0000 7f1b9a484640 1 -- 172.21.15.42:0/1258437645 shutdown_connections 2024-09-16T04:13:00.483 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.478+0000 7f1b9a484640 1 --2- 172.21.15.42:0/1258437645 >> v2:172.21.15.154:6812/254845342 conn(0x7f1b78038000 0x7f1b7803a4c0 unknown :-1 s=CLOSED pgs=3978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:13:00.483 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.478+0000 7f1b9a484640 1 --2- 172.21.15.42:0/1258437645 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f1b74005220 0x7f1b94120a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:13:00.484 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.478+0000 7f1b9a484640 1 --2- 172.21.15.42:0/1258437645 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f1b74004810 0x7f1b941204d0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:13:00.484 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.478+0000 7f1b9a484640 1 --2- 172.21.15.42:0/1258437645 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f1b740039c0 0x7f1b94133d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:13:00.484 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.478+0000 7f1b9a484640 1 -- 172.21.15.42:0/1258437645 >> 172.21.15.42:0/1258437645 conn(0x7f1b940a2420 msgr2=0x7f1b94071490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:13:00.484 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.478+0000 7f1b9a484640 1 -- 172.21.15.42:0/1258437645 shutdown_connections 2024-09-16T04:13:00.484 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.478+0000 7f1b9a484640 1 -- 172.21.15.42:0/1258437645 wait complete. 2024-09-16T04:13:00.484 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.478+0000 7f1b9a484640 1 librados: shutdown 2024-09-16T04:13:00.500 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T04:13:00.500 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-16T04:13:00.500 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T04:13:00.626 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.622+0000 7f936e0c0640 10 monclient: get_monmap_and_config 2024-09-16T04:13:00.626 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.622+0000 7f936e0c0640 10 monclient: build_initial_monmap 2024-09-16T04:13:00.626 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.622+0000 7f936e0c0640 10 monclient: monmap: 2024-09-16T04:13:00.626 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:13:00.626 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:13:00.626 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:13:00.624840+0000 2024-09-16T04:13:00.626 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:13:00.624840+0000 2024-09-16T04:13:00.626 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:13:00.626 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:13:00.627 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:13:00.627 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:13:00.627 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:13:00.627 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:13:00.627 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:13:00.627 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:13:00.629 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.622+0000 7f936e0c0640 1 Processor -- start 2024-09-16T04:13:00.629 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.622+0000 7f936e0c0640 1 -- start start 2024-09-16T04:13:00.629 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.622+0000 7f936e0c0640 10 monclient: init 2024-09-16T04:13:00.629 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.622+0000 7f936e0c0640 10 monclient: _reopen_session rank -1 2024-09-16T04:13:00.629 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.622+0000 7f936e0c0640 10 monclient: _add_conns ranks=[1,2,4,0,3] 2024-09-16T04:13:00.629 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.622+0000 7f936e0c0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f93680a1330 addr v1:172.21.15.154:6789/0 2024-09-16T04:13:00.629 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.622+0000 7f936e0c0640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f936806dac0 0x7f936813a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:13:00.629 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.622+0000 7f936e0c0640 10 monclient(hunting): picked mon.noname-b con 0x7f936806dac0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:13:00.629 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.622+0000 7f936e0c0640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f93680a1e20 0x7f936813c890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:13:00.629 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.622+0000 7f936e0c0640 10 monclient(hunting): picked mon.noname-c con 0x7f93680a1e20 addr v2:172.21.15.154:3300/0 2024-09-16T04:13:00.630 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.622+0000 7f936e0c0640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f936809c7d0 con 0x7f93680a1330 2024-09-16T04:13:00.630 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.622+0000 7f936e0c0640 10 monclient(hunting): start opening mon connection 2024-09-16T04:13:00.630 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.622+0000 7f936e0c0640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f936809c090 con 0x7f93680a1e20 2024-09-16T04:13:00.630 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.622+0000 7f936e0c0640 10 monclient(hunting): start opening mon connection 2024-09-16T04:13:00.630 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.622+0000 7f936e0c0640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f936809f610 con 0x7f936806dac0 2024-09-16T04:13:00.630 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.622+0000 7f936e0c0640 10 monclient(hunting): _renew_subs 2024-09-16T04:13:00.630 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.622+0000 7f936e0c0640 10 monclient(hunting): authenticate will time out at 2827.594300s 2024-09-16T04:13:00.630 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.622+0000 7f9366ffd640 1 -- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f936806dac0 msgr2=0x7f936813a340 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T04:13:00.630 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.622+0000 7f9366ffd640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f936806dac0 0x7f936813a340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:13:00.630 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.622+0000 7f9367fff640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f93680a1e20 0x7f936813c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:13:00.630 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.622+0000 7f9367fff640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f93680a1e20 0x7f936813c890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.42:53362/0 (socket says 172.21.15.42:53362) 2024-09-16T04:13:00.630 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.622+0000 7f9367fff640 1 -- 172.21.15.42:0/3350898634 learned_addr learned my addr 172.21.15.42:0/3350898634 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:13:00.631 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.622+0000 7f9367fff640 10 monclient(hunting): get_auth_request con 0x7f93680a1e20 auth_method 0 2024-09-16T04:13:00.631 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.622+0000 7f9367fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:13:00.631 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.622+0000 7f9367fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:00.631 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.622+0000 7f9367fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:13:00.631 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.622+0000 7f9367fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:13:00.631 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.622+0000 7f9367fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:13:00.631 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.622+0000 7f9367fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:13:00.631 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.622+0000 7f9367fff640 10 monclient(hunting): handle_auth_done global_id 1074238 payload 274 2024-09-16T04:13:00.631 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.622+0000 7f9367fff640 1 -- 172.21.15.42:0/3350898634 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f936806dac0 msgr2=0x7f936813a340 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:13:00.632 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.622+0000 7f9367fff640 1 --2- 172.21.15.42:0/3350898634 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f936806dac0 0x7f936813a340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:13:00.632 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.622+0000 7f9367fff640 1 -- 172.21.15.42:0/3350898634 >> v1:172.21.15.154:6789/0 conn(0x7f93680a1330 legacy=0x7f93680a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:00.632 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.622+0000 7f9367fff640 10 monclient: _finish_hunting 0 2024-09-16T04:13:00.632 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.622+0000 7f9367fff640 1 monclient: found mon.noname-c 2024-09-16T04:13:00.632 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.622+0000 7f9367fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:13:00.632 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.622+0000 7f9367fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.154:3300/0 2024-09-16T04:13:00.632 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.622+0000 7f9367fff640 1 -- 172.21.15.42:0/3350898634 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f936809f8f0 con 0x7f93680a1e20 2024-09-16T04:13:00.633 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.626+0000 7f9367fff640 1 --2- 172.21.15.42:0/3350898634 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f93680a1e20 0x7f936813c890 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f9358009870 tx=0x7f935801c530 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2480449db03116c4 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:13:00.634 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.626+0000 7f93667fc640 1 -- 172.21.15.42:0/3350898634 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93580a9610 con 0x7f93680a1e20 2024-09-16T04:13:00.634 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.626+0000 7f93667fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:13:00.634 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.626+0000 7f93667fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:13:00.634 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.626+0000 7f93667fc640 10 monclient: dump: 2024-09-16T04:13:00.634 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:13:00.634 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:13:00.634 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:00.634 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:00.634 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:00.634 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:13:00.634 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:13:00.634 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:13:00.634 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:13:00.635 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:13:00.635 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.626+0000 7f93667fc640 10 monclient: _finish_auth 0 2024-09-16T04:13:00.635 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.626+0000 7f93667fc640 10 monclient: _check_auth_tickets 2024-09-16T04:13:00.635 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.626+0000 7f93667fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:13:00.635 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.626+0000 7f93667fc640 1 -- 172.21.15.42:0/3350898634 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f93580a97b0 con 0x7f93680a1e20 2024-09-16T04:13:00.635 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.626+0000 7f93667fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:13:00.635 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.626+0000 7f93667fc640 1 -- 172.21.15.42:0/3350898634 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93580a9b80 con 0x7f93680a1e20 2024-09-16T04:13:00.635 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.626+0000 7f93667fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:13:00.635 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.626+0000 7f93667fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:13:00.635 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.626+0000 7f93667fc640 10 monclient: dump: 2024-09-16T04:13:00.635 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:13:00.635 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:13:00.635 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:00.635 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:00.636 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:00.636 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:13:00.636 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:13:00.636 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:13:00.636 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:13:00.636 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:13:00.636 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.626+0000 7f936e0c0640 5 monclient: authenticate success, global_id 1074238 2024-09-16T04:13:00.636 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.626+0000 7f936e0c0640 10 monclient: get_monmap_and_config success 2024-09-16T04:13:00.636 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.626+0000 7f936e0c0640 4 set_mon_vals no callback set 2024-09-16T04:13:00.636 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.630+0000 7f936e0c0640 10 monclient: shutdown 2024-09-16T04:13:00.636 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.630+0000 7f936e0c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:13:00.636 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.630+0000 7f936e0c0640 1 -- 172.21.15.42:0/3350898634 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f93680a1e20 msgr2=0x7f936813c890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:00.636 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.630+0000 7f936e0c0640 1 --2- 172.21.15.42:0/3350898634 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f93680a1e20 0x7f936813c890 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f9358009870 tx=0x7f935801c530 comp rx=0 tx=0).stop 2024-09-16T04:13:00.636 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.630+0000 7f936e0c0640 1 -- 172.21.15.42:0/3350898634 shutdown_connections 2024-09-16T04:13:00.637 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.630+0000 7f936e0c0640 1 --2- 172.21.15.42:0/3350898634 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f93680a1e20 0x7f936813c890 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:13:00.637 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.630+0000 7f936e0c0640 1 --2- 172.21.15.42:0/3350898634 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f936806dac0 0x7f936813a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:13:00.637 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.630+0000 7f936e0c0640 1 -- 172.21.15.42:0/3350898634 >> 172.21.15.42:0/3350898634 conn(0x7f93680a2420 msgr2=0x7f93680a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:13:00.637 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.630+0000 7f936e0c0640 1 -- 172.21.15.42:0/3350898634 shutdown_connections 2024-09-16T04:13:00.637 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.630+0000 7f936e0c0640 1 -- 172.21.15.42:0/3350898634 wait complete. 2024-09-16T04:13:00.642 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.630+0000 7f936e0c0640 10 monclient: build_initial_monmap 2024-09-16T04:13:00.642 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.630+0000 7f936e0c0640 10 monclient: monmap: 2024-09-16T04:13:00.642 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:13:00.642 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:13:00.642 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:13:00.642 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:13:00.642 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:13:00.643 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:13:00.643 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:13:00.643 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:13:00.643 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:13:00.643 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:13:00.643 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.630+0000 7f936e0c0640 1 librados: starting msgr at 2024-09-16T04:13:00.643 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.630+0000 7f936e0c0640 1 librados: starting objecter 2024-09-16T04:13:00.643 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.630+0000 7f936e0c0640 1 Processor -- start 2024-09-16T04:13:00.643 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.630+0000 7f936e0c0640 1 -- start start 2024-09-16T04:13:00.643 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.630+0000 7f936e0c0640 1 librados: setting wanted keys 2024-09-16T04:13:00.643 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.630+0000 7f936e0c0640 1 librados: calling monclient init 2024-09-16T04:13:00.643 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.630+0000 7f936e0c0640 10 monclient: init 2024-09-16T04:13:00.643 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.630+0000 7f936e0c0640 10 monclient: _reopen_session rank -1 2024-09-16T04:13:00.644 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.630+0000 7f936e0c0640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:13:00.644 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.630+0000 7f936e0c0640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f93680a1330 0x7f93681ccba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:13:00.644 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.630+0000 7f936e0c0640 10 monclient(hunting): picked mon.noname-c con 0x7f93680a1330 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:13:00.644 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.630+0000 7f936e0c0640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f93680a1e20 0x7f9368135370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:13:00.644 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.630+0000 7f936e0c0640 10 monclient(hunting): picked mon.noname-b con 0x7f93680a1e20 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:13:00.644 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.630+0000 7f936e0c0640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f93681358b0 0x7f936811bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:13:00.644 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.630+0000 7f936e0c0640 10 monclient(hunting): picked mon.noname-a con 0x7f93681358b0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:13:00.644 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.630+0000 7f936e0c0640 10 monclient(hunting): start opening mon connection 2024-09-16T04:13:00.644 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.630+0000 7f936e0c0640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f936813f770 con 0x7f93681358b0 2024-09-16T04:13:00.644 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.630+0000 7f936e0c0640 10 monclient(hunting): start opening mon connection 2024-09-16T04:13:00.644 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.630+0000 7f936e0c0640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f936813f5f0 con 0x7f93680a1e20 2024-09-16T04:13:00.644 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.630+0000 7f936e0c0640 10 monclient(hunting): start opening mon connection 2024-09-16T04:13:00.644 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.630+0000 7f936e0c0640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f936813f8f0 con 0x7f93680a1330 2024-09-16T04:13:00.644 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.630+0000 7f936e0c0640 10 monclient(hunting): _renew_subs 2024-09-16T04:13:00.645 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.630+0000 7f936e0c0640 10 monclient(hunting): authenticate will time out at 2827.602352s 2024-09-16T04:13:00.645 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.630+0000 7f93677fe640 1 -- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f93680a1330 msgr2=0x7f93681ccba0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T04:13:00.645 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.630+0000 7f93677fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f93680a1330 0x7f93681ccba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:13:00.645 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.630+0000 7f9367fff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f93681358b0 0x7f936811bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:13:00.645 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.630+0000 7f9367fff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f93681358b0 0x7f936811bf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:50190/0 (socket says 172.21.15.42:50190) 2024-09-16T04:13:00.645 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.630+0000 7f9367fff640 1 -- 172.21.15.42:0/1020604661 learned_addr learned my addr 172.21.15.42:0/1020604661 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:13:00.645 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.630+0000 7f9367fff640 10 monclient(hunting): get_auth_request con 0x7f93681358b0 auth_method 0 2024-09-16T04:13:00.645 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.630+0000 7f9367fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:13:00.645 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.630+0000 7f9367fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:00.645 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.630+0000 7f9367fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:13:00.645 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.630+0000 7f9367fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:13:00.645 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.630+0000 7f9367fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:13:00.645 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.630+0000 7f9367fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:13:00.646 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.630+0000 7f9366ffd640 1 --2- 172.21.15.42:0/1020604661 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f93680a1e20 0x7f9368135370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:13:00.646 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.630+0000 7f9366ffd640 10 monclient(hunting): get_auth_request con 0x7f93680a1e20 auth_method 0 2024-09-16T04:13:00.646 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.630+0000 7f9366ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:13:00.646 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.630+0000 7f9366ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:00.646 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.630+0000 7f9366ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:13:00.646 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.630+0000 7f9367fff640 10 monclient(hunting): handle_auth_done global_id 1034577 payload 824 2024-09-16T04:13:00.646 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.630+0000 7f9367fff640 1 -- 172.21.15.42:0/1020604661 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f93680a1330 msgr2=0x7f93681ccba0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:13:00.646 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.630+0000 7f9367fff640 1 --2- 172.21.15.42:0/1020604661 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f93680a1330 0x7f93681ccba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:13:00.646 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.630+0000 7f9367fff640 1 -- 172.21.15.42:0/1020604661 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f93680a1e20 msgr2=0x7f9368135370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:00.646 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.630+0000 7f9367fff640 1 --2- 172.21.15.42:0/1020604661 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f93680a1e20 0x7f9368135370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:13:00.646 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.630+0000 7f9367fff640 10 monclient: _finish_hunting 0 2024-09-16T04:13:00.646 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.630+0000 7f9367fff640 1 monclient: found mon.noname-a 2024-09-16T04:13:00.646 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.630+0000 7f9367fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:13:00.646 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.630+0000 7f9367fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:13:00.647 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.630+0000 7f9367fff640 1 -- 172.21.15.42:0/1020604661 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f936806e280 con 0x7f93681358b0 2024-09-16T04:13:00.647 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.630+0000 7f9366ffd640 1 --2- 172.21.15.42:0/1020604661 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f93680a1e20 0x7f9368135370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T04:13:00.647 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.630+0000 7f9367fff640 1 --2- 172.21.15.42:0/1020604661 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f93681358b0 0x7f936811bf70 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f935801a160 tx=0x7f935801ad70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:13:00.647 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.630+0000 7f934ffff640 10 client.?.objecter ms_handle_connect 0x7f93681358b0 2024-09-16T04:13:00.647 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.630+0000 7f934ffff640 10 client.?.objecter resend_mon_ops 2024-09-16T04:13:00.647 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.630+0000 7f934ffff640 1 -- 172.21.15.42:0/1020604661 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93580bc650 con 0x7f93681358b0 2024-09-16T04:13:00.647 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.630+0000 7f934ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:13:00.647 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.630+0000 7f934ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:13:00.647 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.630+0000 7f934ffff640 10 monclient: dump: 2024-09-16T04:13:00.647 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:13:00.647 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:13:00.647 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:00.647 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:00.647 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:00.648 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:13:00.648 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:13:00.648 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:13:00.648 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:13:00.648 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:13:00.648 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.630+0000 7f934ffff640 10 monclient: _finish_auth 0 2024-09-16T04:13:00.648 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.630+0000 7f934ffff640 10 monclient: _check_auth_tickets 2024-09-16T04:13:00.648 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.630+0000 7f934ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:13:00.648 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.630+0000 7f936e0c0640 5 monclient: authenticate success, global_id 1034577 2024-09-16T04:13:00.648 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.630+0000 7f934ffff640 1 -- 172.21.15.42:0/1020604661 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f93580bc7f0 con 0x7f93681358b0 2024-09-16T04:13:00.648 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.630+0000 7f934ffff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:13:00.648 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.630+0000 7f934ffff640 1 -- 172.21.15.42:0/1020604661 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93580bcbc0 con 0x7f93681358b0 2024-09-16T04:13:00.648 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.630+0000 7f934ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:13:00.648 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.630+0000 7f934ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:13:00.649 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.630+0000 7f934ffff640 10 monclient: dump: 2024-09-16T04:13:00.649 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:13:00.649 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:13:00.649 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:00.649 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:00.649 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:00.649 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:13:00.649 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:13:00.649 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:13:00.649 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:13:00.649 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:13:00.649 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.630+0000 7f936e0c0640 10 monclient: _renew_subs 2024-09-16T04:13:00.649 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.630+0000 7f936e0c0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:13:00.649 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.630+0000 7f936e0c0640 1 -- 172.21.15.42:0/1020604661 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f936813cb20 con 0x7f93681358b0 2024-09-16T04:13:00.650 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.630+0000 7f93657fa640 4 set_mon_vals no callback set 2024-09-16T04:13:00.650 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.630+0000 7f936e0c0640 10 client.1034577.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:13:00.650 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.630+0000 7f936e0c0640 10 monclient: _renew_subs 2024-09-16T04:13:00.650 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.630+0000 7f936e0c0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:13:00.650 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.630+0000 7f936e0c0640 1 -- 172.21.15.42:0/1020604661 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f936811fa20 con 0x7f93681358b0 2024-09-16T04:13:00.650 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.630+0000 7f936e0c0640 1 librados: init done 2024-09-16T04:13:00.650 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.634+0000 7f936e0c0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:13:00.650 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.634+0000 7f936e0c0640 1 --2- 172.21.15.42:0/1020604661 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f936806dac0 0x7f936811c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:13:00.650 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.634+0000 7f936e0c0640 10 monclient(hunting): start opening mon connection 2024-09-16T04:13:00.650 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.634+0000 7f936e0c0640 1 -- 172.21.15.42:0/1020604661 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f936811f6a0 con 0x7f936806dac0 2024-09-16T04:13:00.650 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.634+0000 7f934ffff640 1 -- 172.21.15.42:0/1020604661 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f93580b13f0 con 0x7f93681358b0 2024-09-16T04:13:00.650 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.634+0000 7f93677fe640 1 --2- 172.21.15.42:0/1020604661 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f936806dac0 0x7f936811c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:13:00.650 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.634+0000 7f93677fe640 10 monclient: get_auth_request con 0x7f936806dac0 auth_method 0 2024-09-16T04:13:00.651 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.634+0000 7f93677fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:13:00.651 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.634+0000 7f93677fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:00.651 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.634+0000 7f93677fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:13:00.651 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.634+0000 7f93677fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:13:00.651 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.634+0000 7f93677fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:13:00.651 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.634+0000 7f93677fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:13:00.651 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.634+0000 7f934ffff640 1 --2- 172.21.15.42:0/1020604661 >> v2:172.21.15.154:6812/254845342 conn(0x7f93540377f0 0x7f9354039cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:13:00.651 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.634+0000 7f934ffff640 1 -- 172.21.15.42:0/1020604661 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(706..706 src has 681..706) ==== 5040+0+0 (secure 0 0 0) 0x7f93580e3a30 con 0x7f93681358b0 2024-09-16T04:13:00.651 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.634+0000 7f934ffff640 10 client.1034577.objecter ms_dispatch 0x7f9368000ba0 osd_map(706..706 src has 681..706) 2024-09-16T04:13:00.651 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.634+0000 7f934ffff640 3 client.1034577.objecter handle_osd_map got epochs [706,706] > 0 2024-09-16T04:13:00.651 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.634+0000 7f934ffff640 3 client.1034577.objecter handle_osd_map decoding full epoch 706 2024-09-16T04:13:00.651 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.634+0000 7f93677fe640 10 monclient(hunting): handle_auth_done global_id 1034580 payload 824 2024-09-16T04:13:00.651 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.634+0000 7f93677fe640 1 -- 172.21.15.42:0/1020604661 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f93680a1710 con 0x7f936806dac0 2024-09-16T04:13:00.651 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.634+0000 7f934ffff640 20 client.1034577.objecter dump_active .. 0 homeless 2024-09-16T04:13:00.652 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.634+0000 7f934ffff640 10 client.1034577.objecter ms_handle_connect 0x7f936806dac0 2024-09-16T04:13:00.652 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.634+0000 7f934ffff640 10 client.1034577.objecter resend_mon_ops 2024-09-16T04:13:00.652 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.634+0000 7f93677fe640 1 --2- 172.21.15.42:0/1020604661 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f936806dac0 0x7f936811c4b0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f93600013c0 tx=0x7f936000cb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:13:00.652 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.634+0000 7f934ffff640 1 -- 172.21.15.42:0/1020604661 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9360005d50 con 0x7f936806dac0 2024-09-16T04:13:00.652 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.634+0000 7f934ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:13:00.652 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.634+0000 7f934ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:13:00.652 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.634+0000 7f9366ffd640 1 --2- 172.21.15.42:0/1020604661 >> v2:172.21.15.154:6812/254845342 conn(0x7f93540377f0 0x7f9354039cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:13:00.652 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.634+0000 7f934ffff640 10 monclient: dump: 2024-09-16T04:13:00.652 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:13:00.652 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:13:00.652 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:00.652 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:00.652 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:00.652 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:13:00.653 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:13:00.653 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:13:00.653 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:13:00.653 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:13:00.653 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.634+0000 7f9366ffd640 10 monclient: get_auth_request con 0x7f93540377f0 auth_method 0 2024-09-16T04:13:00.653 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.634+0000 7f934ffff640 1 -- 172.21.15.42:0/1020604661 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f9360005d50 con 0x7f936806dac0 2024-09-16T04:13:00.653 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.634+0000 7f934ffff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:13:00.653 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.634+0000 7f934ffff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:13:00.653 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.634+0000 7f934ffff640 1 -- 172.21.15.42:0/1020604661 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f936806dac0 msgr2=0x7f936811c4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:00.653 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.634+0000 7f934ffff640 1 --2- 172.21.15.42:0/1020604661 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f936806dac0 0x7f936811c4b0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f93600013c0 tx=0x7f936000cb80 comp rx=0 tx=0).stop 2024-09-16T04:13:00.653 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.634+0000 7f934ffff640 1 -- 172.21.15.42:0/1020604661 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f936806dac0 msgr2=0x7f936811c4b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:13:00.653 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.634+0000 7f934ffff640 1 --2- 172.21.15.42:0/1020604661 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f936806dac0 0x7f936811c4b0 secure :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f93600013c0 tx=0x7f936000cb80 comp rx=0 tx=0).stop 2024-09-16T04:13:00.653 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.634+0000 7f934ffff640 10 client.1034577.objecter ms_handle_connect 0x7f93540377f0 2024-09-16T04:13:00.654 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.634+0000 7f9366ffd640 1 --2- 172.21.15.42:0/1020604661 >> v2:172.21.15.154:6812/254845342 conn(0x7f93540377f0 0x7f9354039cb0 secure :-1 s=READY pgs=3979 cs=0 l=1 rev1=1 crypto rx=0x7f936811d610 tx=0x7f935c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:13:00.655 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.638+0000 7f936e0c0640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:13:00.655 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.638+0000 7f936e0c0640 1 --2- 172.21.15.42:0/1020604661 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f936811c9f0 0x7f9368138330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:13:00.655 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.638+0000 7f936e0c0640 10 monclient(hunting): start opening mon connection 2024-09-16T04:13:00.655 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.638+0000 7f936e0c0640 1 -- 172.21.15.42:0/1020604661 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f936811ce00 con 0x7f936811c9f0 2024-09-16T04:13:00.655 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.638+0000 7f93677fe640 1 --2- 172.21.15.42:0/1020604661 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f936811c9f0 0x7f9368138330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:13:00.655 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.638+0000 7f93677fe640 10 monclient: get_auth_request con 0x7f936811c9f0 auth_method 0 2024-09-16T04:13:00.656 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.638+0000 7f93677fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:13:00.656 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.638+0000 7f93677fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:00.656 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.638+0000 7f93677fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:13:00.656 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.638+0000 7f93677fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:13:00.656 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.638+0000 7f93677fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:13:00.656 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.638+0000 7f93677fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:13:00.656 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.638+0000 7f93677fe640 10 monclient(hunting): handle_auth_done global_id 1034583 payload 824 2024-09-16T04:13:00.656 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.638+0000 7f93677fe640 1 -- 172.21.15.42:0/1020604661 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f9368069040 con 0x7f936811c9f0 2024-09-16T04:13:00.656 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.642+0000 7f93677fe640 1 --2- 172.21.15.42:0/1020604661 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f936811c9f0 0x7f9368138330 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f9360013000 tx=0x7f9360001260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:13:00.656 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.642+0000 7f934ffff640 10 client.1034577.objecter ms_handle_connect 0x7f936811c9f0 2024-09-16T04:13:00.656 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.642+0000 7f934ffff640 10 client.1034577.objecter resend_mon_ops 2024-09-16T04:13:00.656 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.642+0000 7f934ffff640 1 -- 172.21.15.42:0/1020604661 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f936000ab60 con 0x7f936811c9f0 2024-09-16T04:13:00.656 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.642+0000 7f934ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:13:00.657 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.642+0000 7f934ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:13:00.657 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.642+0000 7f934ffff640 10 monclient: dump: 2024-09-16T04:13:00.657 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:13:00.657 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:13:00.657 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:00.657 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:00.657 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:00.657 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:13:00.657 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:13:00.657 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:13:00.657 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:13:00.657 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:13:00.657 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.642+0000 7f934ffff640 1 -- 172.21.15.42:0/1020604661 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4704 (secure 0 0 0) 0x7f936000ab60 con 0x7f936811c9f0 2024-09-16T04:13:00.657 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.642+0000 7f934ffff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:13:00.658 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.642+0000 7f934ffff640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:13:00.658 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.642+0000 7f934ffff640 1 -- 172.21.15.42:0/1020604661 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f936811c9f0 msgr2=0x7f9368138330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:00.658 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.642+0000 7f934ffff640 1 --2- 172.21.15.42:0/1020604661 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f936811c9f0 0x7f9368138330 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f9360013000 tx=0x7f9360001260 comp rx=0 tx=0).stop 2024-09-16T04:13:00.658 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.642+0000 7f934ffff640 1 -- 172.21.15.42:0/1020604661 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f936811c9f0 msgr2=0x7f9368138330 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:13:00.658 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.642+0000 7f934ffff640 1 --2- 172.21.15.42:0/1020604661 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f936811c9f0 0x7f9368138330 secure :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f9360013000 tx=0x7f9360001260 comp rx=0 tx=0).stop 2024-09-16T04:13:00.658 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T04:13:00.658 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:13:00.658 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:13:00.658 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "leader", 2024-09-16T04:13:00.658 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 280, 2024-09-16T04:13:00.658 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 260577, 2024-09-16T04:13:00.659 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T04:13:00.659 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T04:13:00.659 INFO:teuthology.orchestra.run.smithi042.stdout: 1 2024-09-16T04:13:00.659 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:13:00.659 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 0, 2024-09-16T04:13:00.659 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:13:00.659 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T04:13:00.659 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T04:13:00.659 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:13:00.659 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:13:00.659 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:13:00.659 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:13:00.659 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:13:00.659 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:13:00.660 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:13:00.660 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:13:00.660 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:13:00.660 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:13:00.660 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:13:00.660 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:13:00.660 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:13:00.660 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T04:13:00.660 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:13:00.660 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:13:00.660 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:13:00.660 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:13:00.660 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:13:00.661 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:13:00.661 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:13:00.661 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:13:00.661 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:13:00.661 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:13:00.661 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:13:00.661 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:13:00.661 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:13:00.661 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T04:13:00.661 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T04:13:00.661 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T04:13:00.661 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T04:13:00.661 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T04:13:00.661 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T04:13:00.662 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:13:00.662 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:13:00.662 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T04:13:00.662 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T04:13:00.662 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T04:13:00.662 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T04:13:00.662 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T04:13:00.662 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T04:13:00.662 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T04:13:00.662 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:13:00.662 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T04:13:00.662 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:13:00.662 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:13:00.662 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:13:00.662 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:13:00.663 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:13:00.663 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:13:00.663 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:13:00.663 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:13:00.663 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:13:00.663 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:13:00.663 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:13:00.663 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:13:00.663 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T04:13:00.663 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:13:00.663 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T04:13:00.663 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:13:00.663 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:13:00.663 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:13:00.664 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:13:00.664 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:13:00.664 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:13:00.664 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:13:00.664 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T04:13:00.664 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:13:00.664 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:13:00.664 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:13:00.664 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:13:00.664 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T04:13:00.664 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:13:00.664 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:13:00.664 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:13:00.664 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:13:00.665 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T04:13:00.665 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T04:13:00.665 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:13:00.665 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:13:00.665 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:13:00.665 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:13:00.665 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:13:00.665 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:13:00.665 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:13:00.665 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:13:00.665 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:13:00.665 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:13:00.665 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:13:00.665 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T04:13:00.666 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:13:00.666 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:13:00.666 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:13:00.666 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:13:00.666 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T04:13:00.666 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:13:00.666 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:13:00.666 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:13:00.666 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:13:00.666 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T04:13:00.666 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T04:13:00.666 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:13:00.666 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:13:00.666 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:13:00.667 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:13:00.667 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:13:00.667 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:13:00.667 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:13:00.667 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:13:00.667 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:13:00.667 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:13:00.667 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:13:00.667 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T04:13:00.667 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:13:00.667 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:13:00.667 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:13:00.667 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:13:00.667 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T04:13:00.668 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:13:00.668 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:13:00.668 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:13:00.668 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:13:00.668 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T04:13:00.668 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T04:13:00.668 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:13:00.668 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:13:00.668 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:13:00.668 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:13:00.668 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:13:00.668 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:13:00.668 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T04:13:00.669 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T04:13:00.669 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:13:00.669 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:13:00.669 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:13:00.669 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:13:00.669 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:13:00.669 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:13:00.669 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T04:13:00.669 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:13:00.669 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:13:00.669 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:13:00.669 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 6 2024-09-16T04:13:00.669 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:13:00.669 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:13:00.669 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T04:13:00.670 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:13:00.670 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:13:00.670 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:13:00.670 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 6 2024-09-16T04:13:00.670 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:13:00.670 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:13:00.670 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:13:00.670 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T04:13:00.670 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T04:13:00.672 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.642+0000 7f936e0c0640 10 librados: watch_flush enter 2024-09-16T04:13:00.672 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.642+0000 7f936e0c0640 10 librados: watch_flush exit 2024-09-16T04:13:00.672 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.642+0000 7f936e0c0640 20 client.1034577.objecter shutdown clearing up homeless session... 2024-09-16T04:13:00.672 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.642+0000 7f936e0c0640 10 client.1034577.objecter successfully canceled tick 2024-09-16T04:13:00.672 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.642+0000 7f936e0c0640 1 -- 172.21.15.42:0/1020604661 >> v2:172.21.15.154:6812/254845342 conn(0x7f93540377f0 msgr2=0x7f9354039cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:00.672 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.642+0000 7f936e0c0640 1 --2- 172.21.15.42:0/1020604661 >> v2:172.21.15.154:6812/254845342 conn(0x7f93540377f0 0x7f9354039cb0 secure :-1 s=READY pgs=3979 cs=0 l=1 rev1=1 crypto rx=0x7f936811d610 tx=0x7f935c008040 comp rx=0 tx=0).stop 2024-09-16T04:13:00.672 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.642+0000 7f936e0c0640 10 monclient: shutdown 2024-09-16T04:13:00.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.642+0000 7f936e0c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:13:00.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.642+0000 7f936e0c0640 1 -- 172.21.15.42:0/1020604661 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f93681358b0 msgr2=0x7f936811bf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:00.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.642+0000 7f936e0c0640 1 --2- 172.21.15.42:0/1020604661 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f93681358b0 0x7f936811bf70 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f935801a160 tx=0x7f935801ad70 comp rx=0 tx=0).stop 2024-09-16T04:13:00.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.642+0000 7f9367fff640 1 -- 172.21.15.42:0/1020604661 reap_dead start 2024-09-16T04:13:00.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.642+0000 7f9367fff640 1 -- 172.21.15.42:0/1020604661 reap_dead start 2024-09-16T04:13:00.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.642+0000 7f936e0c0640 1 -- 172.21.15.42:0/1020604661 shutdown_connections 2024-09-16T04:13:00.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.642+0000 7f936e0c0640 1 -- 172.21.15.42:0/1020604661 >> 172.21.15.42:0/1020604661 conn(0x7f93680a2420 msgr2=0x7f93680a46b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:13:00.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.642+0000 7f936e0c0640 1 -- 172.21.15.42:0/1020604661 shutdown_connections 2024-09-16T04:13:00.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.642+0000 7f936e0c0640 1 -- 172.21.15.42:0/1020604661 wait complete. 2024-09-16T04:13:00.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.642+0000 7f936e0c0640 1 librados: shutdown 2024-09-16T04:13:00.673 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T04:13:00.674 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-16T04:13:00.674 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T04:13:00.791 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.786+0000 7f230841a640 10 monclient: get_monmap_and_config 2024-09-16T04:13:00.791 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.786+0000 7f230841a640 10 monclient: build_initial_monmap 2024-09-16T04:13:00.791 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.786+0000 7f230841a640 10 monclient: monmap: 2024-09-16T04:13:00.791 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:13:00.791 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:13:00.791 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:13:00.789763+0000 2024-09-16T04:13:00.791 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:13:00.789763+0000 2024-09-16T04:13:00.791 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:13:00.791 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:13:00.792 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:13:00.792 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:13:00.792 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:13:00.792 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:13:00.792 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:13:00.792 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:13:00.794 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.786+0000 7f230841a640 1 Processor -- start 2024-09-16T04:13:00.794 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.786+0000 7f230841a640 1 -- start start 2024-09-16T04:13:00.794 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.786+0000 7f230841a640 10 monclient: init 2024-09-16T04:13:00.794 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.786+0000 7f230841a640 10 monclient: _reopen_session rank -1 2024-09-16T04:13:00.794 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.786+0000 7f230841a640 10 monclient: _add_conns ranks=[1,2,4,3,0] 2024-09-16T04:13:00.795 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.786+0000 7f230841a640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f230009fb00 addr v1:172.21.15.154:6789/0 2024-09-16T04:13:00.795 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.786+0000 7f230841a640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f230009c160 0x7f2300139120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:13:00.795 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.786+0000 7f230841a640 10 monclient(hunting): picked mon.noname-b con 0x7f230009c160 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:13:00.795 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.786+0000 7f230841a640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f230009c5a0 0x7f230013b670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:13:00.795 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.786+0000 7f230841a640 10 monclient(hunting): picked mon.noname-c con 0x7f230009c5a0 addr v2:172.21.15.154:3300/0 2024-09-16T04:13:00.795 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.786+0000 7f230841a640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f23000a1e20 con 0x7f230009fb00 2024-09-16T04:13:00.795 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.786+0000 7f230841a640 10 monclient(hunting): start opening mon connection 2024-09-16T04:13:00.795 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.786+0000 7f230841a640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f230009e250 con 0x7f230009c5a0 2024-09-16T04:13:00.796 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.786+0000 7f230841a640 10 monclient(hunting): start opening mon connection 2024-09-16T04:13:00.796 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.786+0000 7f230841a640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f230009ea30 con 0x7f230009c160 2024-09-16T04:13:00.796 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.786+0000 7f230841a640 10 monclient(hunting): _renew_subs 2024-09-16T04:13:00.796 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.786+0000 7f230841a640 10 monclient(hunting): authenticate will time out at 2827.759730s 2024-09-16T04:13:00.796 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.786+0000 7f23059b5640 1 -- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f230009c160 msgr2=0x7f2300139120 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T04:13:00.796 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.786+0000 7f23059b5640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f230009c160 0x7f2300139120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:13:00.796 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.790+0000 7f23069b7640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f230009c5a0 0x7f230013b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:13:00.796 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.790+0000 7f23069b7640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f230009c5a0 0x7f230013b670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.42:53368/0 (socket says 172.21.15.42:53368) 2024-09-16T04:13:00.796 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.790+0000 7f23069b7640 1 -- 172.21.15.42:0/3991711562 learned_addr learned my addr 172.21.15.42:0/3991711562 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:13:00.796 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.790+0000 7f23069b7640 10 monclient(hunting): get_auth_request con 0x7f230009c5a0 auth_method 0 2024-09-16T04:13:00.796 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.790+0000 7f23069b7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:13:00.796 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.790+0000 7f23069b7640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:00.797 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.790+0000 7f23069b7640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:13:00.797 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.790+0000 7f23069b7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:13:00.797 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.790+0000 7f23069b7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:13:00.797 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.790+0000 7f23069b7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:13:00.797 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.790+0000 7f23069b7640 10 monclient(hunting): handle_auth_done global_id 1074247 payload 274 2024-09-16T04:13:00.797 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.790+0000 7f23069b7640 1 -- 172.21.15.42:0/3991711562 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f230009c160 msgr2=0x7f2300139120 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:13:00.797 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.790+0000 7f23069b7640 1 --2- 172.21.15.42:0/3991711562 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f230009c160 0x7f2300139120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:13:00.797 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.790+0000 7f23069b7640 1 -- 172.21.15.42:0/3991711562 >> v1:172.21.15.154:6789/0 conn(0x7f230009fb00 legacy=0x7f230009ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:00.797 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.790+0000 7f23069b7640 10 monclient: _finish_hunting 0 2024-09-16T04:13:00.797 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.790+0000 7f23069b7640 1 monclient: found mon.noname-c 2024-09-16T04:13:00.797 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.790+0000 7f23069b7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:13:00.797 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.790+0000 7f23069b7640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.154:3300/0 2024-09-16T04:13:00.798 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.790+0000 7f23069b7640 1 -- 172.21.15.42:0/3991711562 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f230009fee0 con 0x7f230009c5a0 2024-09-16T04:13:00.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.790+0000 7f23069b7640 1 --2- 172.21.15.42:0/3991711562 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f230009c5a0 0x7f230013b670 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f22f8009e50 tx=0x7f22f801da20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c7550ae3a32aed37 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:13:00.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.790+0000 7f23051b4640 1 -- 172.21.15.42:0/3991711562 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22f80a6860 con 0x7f230009c5a0 2024-09-16T04:13:00.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.790+0000 7f23051b4640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:13:00.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.790+0000 7f23051b4640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:13:00.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.790+0000 7f23051b4640 10 monclient: dump: 2024-09-16T04:13:00.806 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:13:00.806 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:13:00.806 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:00.806 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:00.806 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:00.806 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:13:00.806 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:13:00.806 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:13:00.806 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:13:00.806 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:13:00.807 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.790+0000 7f23051b4640 10 monclient: _finish_auth 0 2024-09-16T04:13:00.807 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.790+0000 7f23051b4640 10 monclient: _check_auth_tickets 2024-09-16T04:13:00.807 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.790+0000 7f23051b4640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:13:00.807 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.790+0000 7f23051b4640 1 -- 172.21.15.42:0/3991711562 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f22f80a6a00 con 0x7f230009c5a0 2024-09-16T04:13:00.807 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.790+0000 7f23051b4640 10 monclient: handle_config config(0 keys) 2024-09-16T04:13:00.807 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.790+0000 7f23051b4640 1 -- 172.21.15.42:0/3991711562 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22f80a6dd0 con 0x7f230009c5a0 2024-09-16T04:13:00.807 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.790+0000 7f23051b4640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:13:00.807 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.790+0000 7f23051b4640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:13:00.807 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.790+0000 7f23051b4640 10 monclient: dump: 2024-09-16T04:13:00.807 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:13:00.807 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:13:00.807 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:00.807 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:00.808 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:00.808 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:13:00.808 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:13:00.808 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:13:00.808 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:13:00.808 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:13:00.808 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.790+0000 7f230841a640 5 monclient: authenticate success, global_id 1074247 2024-09-16T04:13:00.808 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.790+0000 7f230841a640 10 monclient: get_monmap_and_config success 2024-09-16T04:13:00.808 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.790+0000 7f230841a640 4 set_mon_vals no callback set 2024-09-16T04:13:00.808 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.794+0000 7f230841a640 10 monclient: shutdown 2024-09-16T04:13:00.808 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.794+0000 7f230841a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:13:00.808 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.794+0000 7f230841a640 1 -- 172.21.15.42:0/3991711562 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f230009c5a0 msgr2=0x7f230013b670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:00.808 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.794+0000 7f230841a640 1 --2- 172.21.15.42:0/3991711562 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f230009c5a0 0x7f230013b670 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f22f8009e50 tx=0x7f22f801da20 comp rx=0 tx=0).stop 2024-09-16T04:13:00.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.794+0000 7f230841a640 1 -- 172.21.15.42:0/3991711562 shutdown_connections 2024-09-16T04:13:00.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.794+0000 7f230841a640 1 --2- 172.21.15.42:0/3991711562 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f230009c5a0 0x7f230013b670 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:13:00.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.794+0000 7f230841a640 1 --2- 172.21.15.42:0/3991711562 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f230009c160 0x7f2300139120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:13:00.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.794+0000 7f230841a640 1 -- 172.21.15.42:0/3991711562 >> 172.21.15.42:0/3991711562 conn(0x7f23000a2420 msgr2=0x7f23000a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:13:00.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.794+0000 7f230841a640 1 -- 172.21.15.42:0/3991711562 shutdown_connections 2024-09-16T04:13:00.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.794+0000 7f230841a640 1 -- 172.21.15.42:0/3991711562 wait complete. 2024-09-16T04:13:00.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.794+0000 7f230841a640 10 monclient: build_initial_monmap 2024-09-16T04:13:00.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.794+0000 7f230841a640 10 monclient: monmap: 2024-09-16T04:13:00.809 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:13:00.809 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:13:00.809 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:13:00.809 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:13:00.809 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:13:00.809 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:13:00.810 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:13:00.810 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:13:00.810 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:13:00.810 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:13:00.810 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.794+0000 7f230841a640 1 librados: starting msgr at 2024-09-16T04:13:00.810 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.794+0000 7f230841a640 1 librados: starting objecter 2024-09-16T04:13:00.810 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.794+0000 7f230841a640 1 Processor -- start 2024-09-16T04:13:00.810 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.794+0000 7f230841a640 1 -- start start 2024-09-16T04:13:00.810 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.794+0000 7f230841a640 1 librados: setting wanted keys 2024-09-16T04:13:00.810 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.794+0000 7f230841a640 1 librados: calling monclient init 2024-09-16T04:13:00.810 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.794+0000 7f230841a640 10 monclient: init 2024-09-16T04:13:00.810 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.794+0000 7f230841a640 10 monclient: _reopen_session rank -1 2024-09-16T04:13:00.810 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.794+0000 7f230841a640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:13:00.810 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.794+0000 7f230841a640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f230009c160 0x7f23001cf090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:13:00.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.794+0000 7f230841a640 10 monclient(hunting): picked mon.noname-c con 0x7f230009c160 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:13:00.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.794+0000 7f230841a640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f230009c5a0 0x7f2300137900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:13:00.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.794+0000 7f230841a640 10 monclient(hunting): picked mon.noname-a con 0x7f230009c5a0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:13:00.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.794+0000 7f230841a640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f230009fb00 0x7f2300137e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:13:00.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.794+0000 7f230841a640 10 monclient(hunting): picked mon.noname-b con 0x7f230009fb00 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:13:00.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.794+0000 7f230841a640 10 monclient(hunting): start opening mon connection 2024-09-16T04:13:00.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.794+0000 7f230841a640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f230009d380 con 0x7f230009c5a0 2024-09-16T04:13:00.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.794+0000 7f230841a640 10 monclient(hunting): start opening mon connection 2024-09-16T04:13:00.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.794+0000 7f230841a640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f230009d140 con 0x7f230009fb00 2024-09-16T04:13:00.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.794+0000 7f230841a640 10 monclient(hunting): start opening mon connection 2024-09-16T04:13:00.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.794+0000 7f230841a640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f230009d500 con 0x7f230009c160 2024-09-16T04:13:00.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.794+0000 7f230841a640 10 monclient(hunting): _renew_subs 2024-09-16T04:13:00.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.794+0000 7f230841a640 10 monclient(hunting): authenticate will time out at 2827.766295s 2024-09-16T04:13:00.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.794+0000 7f23061b6640 1 -- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f230009c160 msgr2=0x7f23001cf090 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.42:3301/0 2024-09-16T04:13:00.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.794+0000 7f23061b6640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f230009c160 0x7f23001cf090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:13:00.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.794+0000 7f23059b5640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f230009c5a0 0x7f2300137900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:13:00.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.794+0000 7f23059b5640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f230009c5a0 0x7f2300137900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:50226/0 (socket says 172.21.15.42:50226) 2024-09-16T04:13:00.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.794+0000 7f23059b5640 1 -- 172.21.15.42:0/1771597125 learned_addr learned my addr 172.21.15.42:0/1771597125 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:13:00.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.794+0000 7f23059b5640 10 monclient(hunting): get_auth_request con 0x7f230009c5a0 auth_method 0 2024-09-16T04:13:00.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.794+0000 7f23059b5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:13:00.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.794+0000 7f23059b5640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:00.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.794+0000 7f23059b5640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:13:00.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.794+0000 7f23059b5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:13:00.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.794+0000 7f23059b5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:13:00.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.794+0000 7f23059b5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:13:00.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.794+0000 7f23069b7640 1 --2- 172.21.15.42:0/1771597125 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f230009fb00 0x7f2300137e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:13:00.813 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.794+0000 7f23069b7640 10 monclient(hunting): get_auth_request con 0x7f230009fb00 auth_method 0 2024-09-16T04:13:00.813 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.794+0000 7f23069b7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:13:00.813 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.794+0000 7f23069b7640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:00.813 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.794+0000 7f23069b7640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:13:00.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.794+0000 7f23059b5640 10 monclient(hunting): handle_auth_done global_id 1034586 payload 824 2024-09-16T04:13:00.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.794+0000 7f23059b5640 1 -- 172.21.15.42:0/1771597125 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f230009c160 msgr2=0x7f23001cf090 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:13:00.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.794+0000 7f23059b5640 1 --2- 172.21.15.42:0/1771597125 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f230009c160 0x7f23001cf090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:13:00.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.794+0000 7f23059b5640 1 -- 172.21.15.42:0/1771597125 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f230009fb00 msgr2=0x7f2300137e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:00.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.794+0000 7f23059b5640 1 --2- 172.21.15.42:0/1771597125 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f230009fb00 0x7f2300137e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:13:00.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.794+0000 7f23059b5640 10 monclient: _finish_hunting 0 2024-09-16T04:13:00.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.794+0000 7f23059b5640 1 monclient: found mon.noname-a 2024-09-16T04:13:00.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.794+0000 7f23059b5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:13:00.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.794+0000 7f23059b5640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:13:00.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.794+0000 7f23059b5640 1 -- 172.21.15.42:0/1771597125 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23001262d0 con 0x7f230009c5a0 2024-09-16T04:13:00.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.794+0000 7f23059b5640 1 --2- 172.21.15.42:0/1771597125 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f230009c5a0 0x7f2300137900 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f22f000b530 tx=0x7f22f000b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:13:00.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.794+0000 7f22f6ffd640 10 client.?.objecter ms_handle_connect 0x7f230009c5a0 2024-09-16T04:13:00.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.794+0000 7f22f6ffd640 10 client.?.objecter resend_mon_ops 2024-09-16T04:13:00.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.794+0000 7f22f6ffd640 1 -- 172.21.15.42:0/1771597125 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22f0013020 con 0x7f230009c5a0 2024-09-16T04:13:00.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.794+0000 7f22f6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:13:00.815 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.794+0000 7f22f6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:13:00.816 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.794+0000 7f22f6ffd640 10 monclient: dump: 2024-09-16T04:13:00.816 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:13:00.816 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:13:00.816 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:00.816 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:00.816 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:00.816 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:13:00.816 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:13:00.816 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:13:00.816 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:13:00.816 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:13:00.816 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.794+0000 7f22f6ffd640 10 monclient: _finish_auth 0 2024-09-16T04:13:00.816 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.794+0000 7f22f6ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:13:00.817 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.794+0000 7f22f6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:13:00.817 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.794+0000 7f230841a640 5 monclient: authenticate success, global_id 1034586 2024-09-16T04:13:00.817 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.794+0000 7f22f6ffd640 1 -- 172.21.15.42:0/1771597125 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f22f0003df0 con 0x7f230009c5a0 2024-09-16T04:13:00.817 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.794+0000 7f230841a640 10 monclient: _renew_subs 2024-09-16T04:13:00.817 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.794+0000 7f230841a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:13:00.817 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.794+0000 7f230841a640 1 -- 172.21.15.42:0/1771597125 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2300073a30 con 0x7f230009c5a0 2024-09-16T04:13:00.817 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.794+0000 7f22f6ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:13:00.817 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.794+0000 7f22f6ffd640 1 -- 172.21.15.42:0/1771597125 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22f00052b0 con 0x7f230009c5a0 2024-09-16T04:13:00.817 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.794+0000 7f22f6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:13:00.817 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.794+0000 7f22f6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:13:00.817 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.794+0000 7f22f6ffd640 10 monclient: dump: 2024-09-16T04:13:00.818 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:13:00.818 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:13:00.818 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:00.818 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:00.818 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:00.818 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:13:00.818 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:13:00.818 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:13:00.818 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:13:00.819 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:13:00.819 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.794+0000 7f23049b3640 4 set_mon_vals no callback set 2024-09-16T04:13:00.819 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.794+0000 7f230841a640 10 client.1034586.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:13:00.819 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.794+0000 7f230841a640 10 monclient: _renew_subs 2024-09-16T04:13:00.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.794+0000 7f230841a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:13:00.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.794+0000 7f230841a640 1 -- 172.21.15.42:0/1771597125 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23001248c0 con 0x7f230009c5a0 2024-09-16T04:13:00.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.794+0000 7f230841a640 1 librados: init done 2024-09-16T04:13:00.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.798+0000 7f230841a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:13:00.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.798+0000 7f230841a640 1 --2- 172.21.15.42:0/1771597125 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2300122340 0x7f2300122720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:13:00.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.798+0000 7f230841a640 10 monclient(hunting): start opening mon connection 2024-09-16T04:13:00.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.798+0000 7f230841a640 1 -- 172.21.15.42:0/1771597125 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f230006ddb0 con 0x7f2300122340 2024-09-16T04:13:00.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.798+0000 7f22f6ffd640 1 -- 172.21.15.42:0/1771597125 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f22f0016020 con 0x7f230009c5a0 2024-09-16T04:13:00.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.798+0000 7f22f6ffd640 1 --2- 172.21.15.42:0/1771597125 >> v2:172.21.15.154:6812/254845342 conn(0x7f22e40377c0 0x7f22e4039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:13:00.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.798+0000 7f22f6ffd640 1 -- 172.21.15.42:0/1771597125 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(706..706 src has 681..706) ==== 5040+0+0 (secure 0 0 0) 0x7f22f0048870 con 0x7f230009c5a0 2024-09-16T04:13:00.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.798+0000 7f22f6ffd640 10 client.1034586.objecter ms_dispatch 0x7f2300000ba0 osd_map(706..706 src has 681..706) 2024-09-16T04:13:00.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.798+0000 7f22f6ffd640 3 client.1034586.objecter handle_osd_map got epochs [706,706] > 0 2024-09-16T04:13:00.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.798+0000 7f22f6ffd640 3 client.1034586.objecter handle_osd_map decoding full epoch 706 2024-09-16T04:13:00.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.798+0000 7f22f6ffd640 20 client.1034586.objecter dump_active .. 0 homeless 2024-09-16T04:13:00.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.798+0000 7f23061b6640 1 --2- 172.21.15.42:0/1771597125 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2300122340 0x7f2300122720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:13:00.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.798+0000 7f23061b6640 10 monclient: get_auth_request con 0x7f2300122340 auth_method 0 2024-09-16T04:13:00.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.798+0000 7f23061b6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:13:00.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.798+0000 7f23061b6640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:00.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.798+0000 7f23061b6640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:13:00.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.798+0000 7f23069b7640 1 --2- 172.21.15.42:0/1771597125 >> v2:172.21.15.154:6812/254845342 conn(0x7f22e40377c0 0x7f22e4039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:13:00.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.798+0000 7f23061b6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:13:00.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.798+0000 7f23061b6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:13:00.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.798+0000 7f23061b6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:13:00.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.798+0000 7f23069b7640 10 monclient: get_auth_request con 0x7f22e40377c0 auth_method 0 2024-09-16T04:13:00.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.798+0000 7f23061b6640 10 monclient(hunting): handle_auth_done global_id 1074256 payload 824 2024-09-16T04:13:00.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.798+0000 7f23061b6640 1 -- 172.21.15.42:0/1771597125 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f230009ba50 con 0x7f2300122340 2024-09-16T04:13:00.823 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.798+0000 7f22f6ffd640 10 client.1034586.objecter ms_handle_connect 0x7f22e40377c0 2024-09-16T04:13:00.823 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.798+0000 7f23069b7640 1 --2- 172.21.15.42:0/1771597125 >> v2:172.21.15.154:6812/254845342 conn(0x7f22e40377c0 0x7f22e4039c80 secure :-1 s=READY pgs=3980 cs=0 l=1 rev1=1 crypto rx=0x7f22f80af000 tx=0x7f22f80a8070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:13:00.823 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.798+0000 7f22f6ffd640 10 client.1034586.objecter ms_handle_connect 0x7f2300122340 2024-09-16T04:13:00.824 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.798+0000 7f22f6ffd640 10 client.1034586.objecter resend_mon_ops 2024-09-16T04:13:00.824 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.798+0000 7f23061b6640 1 --2- 172.21.15.42:0/1771597125 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2300122340 0x7f2300122720 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f22fc001040 tx=0x7f22fc00c8e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:13:00.824 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.798+0000 7f22f6ffd640 1 -- 172.21.15.42:0/1771597125 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22fc0074e0 con 0x7f2300122340 2024-09-16T04:13:00.824 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.798+0000 7f22f6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:13:00.824 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.798+0000 7f22f6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:13:00.825 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.798+0000 7f22f6ffd640 10 monclient: dump: 2024-09-16T04:13:00.825 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:13:00.828 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:13:00.828 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:00.829 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:00.829 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:00.829 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:13:00.829 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:13:00.829 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:13:00.829 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:13:00.830 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:13:00.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.798+0000 7f22f6ffd640 1 -- 172.21.15.42:0/1771597125 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f22fc0074e0 con 0x7f2300122340 2024-09-16T04:13:00.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.798+0000 7f22f6ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:13:00.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.798+0000 7f22f6ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:13:00.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.798+0000 7f22f6ffd640 1 -- 172.21.15.42:0/1771597125 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2300122340 msgr2=0x7f2300122720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:00.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.798+0000 7f22f6ffd640 1 --2- 172.21.15.42:0/1771597125 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2300122340 0x7f2300122720 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f22fc001040 tx=0x7f22fc00c8e0 comp rx=0 tx=0).stop 2024-09-16T04:13:00.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.798+0000 7f22f6ffd640 1 -- 172.21.15.42:0/1771597125 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2300122340 msgr2=0x7f2300122720 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:13:00.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.798+0000 7f22f6ffd640 1 --2- 172.21.15.42:0/1771597125 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2300122340 0x7f2300122720 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:13:00.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.806+0000 7f230841a640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:13:00.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.806+0000 7f230841a640 1 --2- 172.21.15.42:0/1771597125 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2300122c60 0x7f2300139740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:13:00.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.806+0000 7f230841a640 10 monclient(hunting): start opening mon connection 2024-09-16T04:13:00.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.806+0000 7f230841a640 1 -- 172.21.15.42:0/1771597125 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f2300123070 con 0x7f2300122c60 2024-09-16T04:13:00.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.806+0000 7f23061b6640 1 --2- 172.21.15.42:0/1771597125 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2300122c60 0x7f2300139740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:13:00.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.806+0000 7f23061b6640 10 monclient: get_auth_request con 0x7f2300122c60 auth_method 0 2024-09-16T04:13:00.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.806+0000 7f23061b6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:13:00.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.806+0000 7f23061b6640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:00.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.806+0000 7f23061b6640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:13:00.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.806+0000 7f23061b6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:13:00.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.806+0000 7f23061b6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:13:00.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.806+0000 7f23061b6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:13:00.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.806+0000 7f23061b6640 10 monclient(hunting): handle_auth_done global_id 1074259 payload 824 2024-09-16T04:13:00.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.806+0000 7f23061b6640 1 -- 172.21.15.42:0/1771597125 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f2300125b10 con 0x7f2300122c60 2024-09-16T04:13:00.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.806+0000 7f23061b6640 1 --2- 172.21.15.42:0/1771597125 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2300122c60 0x7f2300139740 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f22fc001040 tx=0x7f22fc001160 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:13:00.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.806+0000 7f22f6ffd640 10 client.1034586.objecter ms_handle_connect 0x7f2300122c60 2024-09-16T04:13:00.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.806+0000 7f22f6ffd640 10 client.1034586.objecter resend_mon_ops 2024-09-16T04:13:00.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.806+0000 7f22f6ffd640 1 -- 172.21.15.42:0/1771597125 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22fc01ad70 con 0x7f2300122c60 2024-09-16T04:13:00.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.806+0000 7f22f6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:13:00.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.806+0000 7f22f6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:13:00.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.806+0000 7f22f6ffd640 10 monclient: dump: 2024-09-16T04:13:00.837 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:13:00.837 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:13:00.837 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:00.837 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:00.838 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:00.838 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:13:00.838 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:13:00.838 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:13:00.838 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:13:00.838 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:13:00.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.806+0000 7f22f6ffd640 1 -- 172.21.15.42:0/1771597125 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4701 (secure 0 0 0) 0x7f22fc012400 con 0x7f2300122c60 2024-09-16T04:13:00.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.806+0000 7f22f6ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:13:00.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.806+0000 7f22f6ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:13:00.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.806+0000 7f22f6ffd640 1 -- 172.21.15.42:0/1771597125 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2300122c60 msgr2=0x7f2300139740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:00.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.806+0000 7f22f6ffd640 1 --2- 172.21.15.42:0/1771597125 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2300122c60 0x7f2300139740 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f22fc001040 tx=0x7f22fc001160 comp rx=0 tx=0).stop 2024-09-16T04:13:00.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.806+0000 7f22f6ffd640 1 -- 172.21.15.42:0/1771597125 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2300122c60 msgr2=0x7f2300139740 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:13:00.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.806+0000 7f22f6ffd640 1 --2- 172.21.15.42:0/1771597125 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2300122c60 0x7f2300139740 secure :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f22fc001040 tx=0x7f22fc001160 comp rx=0 tx=0).stop 2024-09-16T04:13:00.839 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T04:13:00.839 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:13:00.839 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:13:00.839 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T04:13:00.839 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 280, 2024-09-16T04:13:00.839 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 93976, 2024-09-16T04:13:00.839 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T04:13:00.839 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T04:13:00.840 INFO:teuthology.orchestra.run.smithi042.stdout: 1 2024-09-16T04:13:00.840 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:13:00.840 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 0, 2024-09-16T04:13:00.840 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:13:00.840 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T04:13:00.840 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T04:13:00.841 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:13:00.841 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:13:00.841 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:13:00.841 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:13:00.841 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:13:00.842 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:13:00.842 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:13:00.842 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:13:00.842 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:13:00.842 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:13:00.843 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:13:00.843 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:13:00.843 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:13:00.843 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T04:13:00.843 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:13:00.844 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:13:00.844 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:13:00.844 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:13:00.844 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:13:00.844 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:13:00.844 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:13:00.845 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:13:00.845 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:13:00.845 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:13:00.845 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:13:00.845 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:13:00.845 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:13:00.845 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T04:13:00.846 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T04:13:00.846 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T04:13:00.846 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T04:13:00.846 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T04:13:00.846 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T04:13:00.846 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:13:00.846 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:13:00.847 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T04:13:00.847 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T04:13:00.847 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T04:13:00.847 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T04:13:00.847 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T04:13:00.848 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T04:13:00.848 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T04:13:00.848 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:13:00.848 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T04:13:00.848 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:13:00.848 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:13:00.848 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:13:00.849 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:13:00.849 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:13:00.849 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:13:00.849 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:13:00.849 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:13:00.849 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:13:00.850 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:13:00.850 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:13:00.850 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:13:00.850 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T04:13:00.850 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:13:00.850 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T04:13:00.850 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:13:00.850 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:13:00.851 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:13:00.851 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:13:00.851 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:13:00.851 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:13:00.851 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:13:00.851 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T04:13:00.851 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:13:00.851 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:13:00.851 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:13:00.852 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:13:00.852 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T04:13:00.852 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:13:00.852 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:13:00.852 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:13:00.852 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:13:00.852 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T04:13:00.852 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T04:13:00.852 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:13:00.852 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:13:00.852 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:13:00.852 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:13:00.853 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:13:00.853 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:13:00.853 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:13:00.853 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:13:00.853 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:13:00.853 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:13:00.853 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:13:00.853 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T04:13:00.853 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:13:00.853 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:13:00.853 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:13:00.853 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:13:00.853 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T04:13:00.853 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:13:00.854 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:13:00.854 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:13:00.854 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:13:00.854 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T04:13:00.854 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T04:13:00.854 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:13:00.854 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:13:00.854 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:13:00.854 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:13:00.854 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:13:00.854 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:13:00.854 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:13:00.854 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:13:00.854 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:13:00.855 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:13:00.855 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:13:00.855 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T04:13:00.855 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:13:00.855 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:13:00.855 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:13:00.855 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:13:00.855 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T04:13:00.855 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:13:00.855 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:13:00.855 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:13:00.856 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:13:00.856 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T04:13:00.856 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T04:13:00.856 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:13:00.856 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:13:00.856 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:13:00.857 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:13:00.857 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:13:00.857 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:13:00.857 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T04:13:00.857 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T04:13:00.857 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:13:00.858 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:13:00.858 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:13:00.858 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:13:00.858 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:13:00.858 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:13:00.858 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T04:13:00.859 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:13:00.859 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:13:00.859 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:13:00.859 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 2 2024-09-16T04:13:00.859 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:13:00.859 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:13:00.860 INFO:teuthology.orchestra.run.smithi042.stdout: "mgr": [ 2024-09-16T04:13:00.860 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:13:00.860 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:13:00.860 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:13:00.861 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:13:00.861 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:13:00.861 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:13:00.861 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:13:00.861 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T04:13:00.861 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T04:13:00.865 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.806+0000 7f230841a640 10 librados: watch_flush enter 2024-09-16T04:13:00.865 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.806+0000 7f230841a640 10 librados: watch_flush exit 2024-09-16T04:13:00.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.806+0000 7f230841a640 20 client.1034586.objecter shutdown clearing up homeless session... 2024-09-16T04:13:00.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.806+0000 7f230841a640 10 client.1034586.objecter successfully canceled tick 2024-09-16T04:13:00.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.806+0000 7f230841a640 1 -- 172.21.15.42:0/1771597125 >> v2:172.21.15.154:6812/254845342 conn(0x7f22e40377c0 msgr2=0x7f22e4039c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:00.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.806+0000 7f230841a640 1 --2- 172.21.15.42:0/1771597125 >> v2:172.21.15.154:6812/254845342 conn(0x7f22e40377c0 0x7f22e4039c80 secure :-1 s=READY pgs=3980 cs=0 l=1 rev1=1 crypto rx=0x7f22f80af000 tx=0x7f22f80a8070 comp rx=0 tx=0).stop 2024-09-16T04:13:00.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.806+0000 7f230841a640 10 monclient: shutdown 2024-09-16T04:13:00.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.806+0000 7f230841a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:13:00.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.806+0000 7f230841a640 1 -- 172.21.15.42:0/1771597125 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f230009c5a0 msgr2=0x7f2300137900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:00.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.806+0000 7f230841a640 1 --2- 172.21.15.42:0/1771597125 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f230009c5a0 0x7f2300137900 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f22f000b530 tx=0x7f22f000b9f0 comp rx=0 tx=0).stop 2024-09-16T04:13:00.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.806+0000 7f23069b7640 1 -- 172.21.15.42:0/1771597125 reap_dead start 2024-09-16T04:13:00.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.806+0000 7f230841a640 1 -- 172.21.15.42:0/1771597125 shutdown_connections 2024-09-16T04:13:00.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.806+0000 7f230841a640 1 --2- 172.21.15.42:0/1771597125 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f230009c5a0 0x7f2300137900 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:13:00.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.806+0000 7f230841a640 1 -- 172.21.15.42:0/1771597125 >> 172.21.15.42:0/1771597125 conn(0x7f23000a2420 msgr2=0x7f2300070fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:13:00.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.806+0000 7f230841a640 1 -- 172.21.15.42:0/1771597125 shutdown_connections 2024-09-16T04:13:00.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.806+0000 7f230841a640 1 -- 172.21.15.42:0/1771597125 wait complete. 2024-09-16T04:13:00.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:00.806+0000 7f230841a640 1 librados: shutdown 2024-09-16T04:13:00.868 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T04:13:00.869 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-16T04:13:20.870 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-16T04:13:20.870 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-09-16T04:13:20.870 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-16T04:13:20.871 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-16T04:13:20.874 INFO:tasks.ceph.mon.c:Started 2024-09-16T04:13:20.874 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T04:13:20.874 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:13:20.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.990+0000 7fd6ad5aa640 10 monclient: get_monmap_and_config 2024-09-16T04:13:20.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.990+0000 7fd6ad5aa640 10 monclient: build_initial_monmap 2024-09-16T04:13:20.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.990+0000 7fd6ad5aa640 10 monclient: monmap: 2024-09-16T04:13:20.994 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:13:20.994 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:13:20.994 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:13:20.992848+0000 2024-09-16T04:13:20.994 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:13:20.992848+0000 2024-09-16T04:13:20.995 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:13:20.995 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:13:20.995 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:13:20.995 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:13:20.995 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:13:20.995 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:13:20.995 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:13:20.995 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:13:21.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.990+0000 7fd6ad5aa640 1 Processor -- start 2024-09-16T04:13:21.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.990+0000 7fd6ad5aa640 1 -- start start 2024-09-16T04:13:21.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.990+0000 7fd6ad5aa640 10 monclient: init 2024-09-16T04:13:21.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.990+0000 7fd6ad5aa640 10 monclient: _reopen_session rank -1 2024-09-16T04:13:21.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.990+0000 7fd6ad5aa640 10 monclient: _add_conns ranks=[2,0,1,3,4] 2024-09-16T04:13:21.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.990+0000 7fd6ad5aa640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd6a80a39b0 0x7fd6a8133b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:13:21.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.990+0000 7fd6ad5aa640 10 monclient(hunting): picked mon.noname-b con 0x7fd6a80a39b0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:13:21.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.990+0000 7fd6ad5aa640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd6a80a3d90 addr v1:172.21.15.42:6789/0 2024-09-16T04:13:21.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.990+0000 7fd6ad5aa640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd6a8137710 addr v1:172.21.15.154:6789/0 2024-09-16T04:13:21.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.990+0000 7fd6ad5aa640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd6a809c140 con 0x7fd6a80a3d90 2024-09-16T04:13:21.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.990+0000 7fd6ad5aa640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd6a809c420 con 0x7fd6a8137710 2024-09-16T04:13:21.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.990+0000 7fd6ad5aa640 10 monclient(hunting): start opening mon connection 2024-09-16T04:13:21.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.990+0000 7fd6ad5aa640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fd6a806dd60 con 0x7fd6a80a39b0 2024-09-16T04:13:21.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.990+0000 7fd6ad5aa640 10 monclient(hunting): _renew_subs 2024-09-16T04:13:21.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.990+0000 7fd6ad5aa640 10 monclient(hunting): authenticate will time out at 2847.961932s 2024-09-16T04:13:21.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.990+0000 7fd6a6ffd640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd6a80a39b0 0x7fd6a8133b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:13:21.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.990+0000 7fd6a67fc640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fd6a80a3d90 0x7fd6a8137000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:56210/0 (socket says 172.21.15.42:56210) 2024-09-16T04:13:21.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.990+0000 7fd6a67fc640 1 -- 172.21.15.42:0/3808282238 learned_addr learned my addr 172.21.15.42:0/3808282238 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T04:13:21.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.990+0000 7fd6a6ffd640 10 monclient(hunting): get_auth_request con 0x7fd6a80a39b0 auth_method 0 2024-09-16T04:13:21.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.990+0000 7fd6a6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:13:21.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.990+0000 7fd6a6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:21.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.990+0000 7fd6a6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:13:21.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.990+0000 7fd6a6ffd640 1 -- 172.21.15.42:0/3808282238 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd6a80a39b0 msgr2=0x7fd6a8133b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:13:21.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.990+0000 7fd6a6ffd640 1 -- 172.21.15.42:0/3808282238 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd6a80a39b0 msgr2=0x7fd6a8133b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:13:21.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.990+0000 7fd6a6ffd640 1 --2- 172.21.15.42:0/3808282238 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd6a80a39b0 0x7fd6a8133b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:13:21.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.990+0000 7fd6a6ffd640 1 --2- 172.21.15.42:0/3808282238 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd6a80a39b0 0x7fd6a8133b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:13:21.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.990+0000 7fd6a5ffb640 1 -- 172.21.15.42:0/3808282238 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3901726221 0 0) 0x7fd6a809c140 con 0x7fd6a80a3d90 2024-09-16T04:13:21.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.990+0000 7fd6a5ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:13:21.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.990+0000 7fd6a5ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:21.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.990+0000 7fd6a5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:13:21.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.990+0000 7fd6a5ffb640 10 monclient(hunting): my global_id is 1034589 2024-09-16T04:13:21.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.990+0000 7fd6a5ffb640 1 -- 172.21.15.42:0/3808282238 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd694003530 con 0x7fd6a80a3d90 2024-09-16T04:13:21.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.990+0000 7fd6a5ffb640 1 -- 172.21.15.42:0/3808282238 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2956477974 0 0) 0x7fd694003530 con 0x7fd6a80a3d90 2024-09-16T04:13:21.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.990+0000 7fd6a5ffb640 1 -- 172.21.15.42:0/3808282238 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd6a80a39b0 msgr2=0x7fd6a8133b80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:13:21.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.990+0000 7fd6a5ffb640 1 --2- 172.21.15.42:0/3808282238 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd6a80a39b0 0x7fd6a8133b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:13:21.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.990+0000 7fd6a5ffb640 1 -- 172.21.15.42:0/3808282238 >> v1:172.21.15.154:6789/0 conn(0x7fd6a8137710 legacy=0x7fd6a813f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:21.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.990+0000 7fd6a5ffb640 10 monclient: _finish_hunting 0 2024-09-16T04:13:21.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.990+0000 7fd6a5ffb640 1 monclient: found mon.noname-a-legacy 2024-09-16T04:13:21.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.990+0000 7fd6a5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:13:21.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.990+0000 7fd6a5ffb640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.42:6789/0 2024-09-16T04:13:21.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.990+0000 7fd6a5ffb640 1 -- 172.21.15.42:0/3808282238 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6a80a5e20 con 0x7fd6a80a3d90 2024-09-16T04:13:21.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.990+0000 7fd6a5ffb640 10 monclient: _finish_auth 0 2024-09-16T04:13:21.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.990+0000 7fd6a5ffb640 10 monclient: _check_auth_tickets 2024-09-16T04:13:21.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.990+0000 7fd6a5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:13:21.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.990+0000 7fd6a5ffb640 1 -- 172.21.15.42:0/3808282238 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2177413734 0 0) 0x7fd698001680 con 0x7fd6a80a3d90 2024-09-16T04:13:21.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.990+0000 7fd6a5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:13:21.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.990+0000 7fd6a5ffb640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T04:13:21.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.990+0000 7fd6a5ffb640 10 monclient: dump: 2024-09-16T04:13:21.009 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:13:21.009 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:13:21.009 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:21.009 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:21.009 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:21.009 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:13:21.009 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:13:21.009 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:13:21.009 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:13:21.009 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:13:21.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.990+0000 7fd6a5ffb640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] but i'm connected to v1:172.21.15.42:6789/0, reconnecting 2024-09-16T04:13:21.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.990+0000 7fd6a5ffb640 10 monclient: _reopen_session rank -1 2024-09-16T04:13:21.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.990+0000 7fd6a5ffb640 1 -- 172.21.15.42:0/3808282238 >> v1:172.21.15.42:6789/0 conn(0x7fd6a80a3d90 legacy=0x7fd6a8137000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:21.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.990+0000 7fd6a5ffb640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:13:21.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.990+0000 7fd6a5ffb640 1 --2- 172.21.15.42:0/3808282238 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd6940039f0 0x7fd694003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:13:21.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.990+0000 7fd6a5ffb640 10 monclient(hunting): picked mon.b con 0x7fd6940039f0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:13:21.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.990+0000 7fd6a5ffb640 1 --2- 172.21.15.42:0/3808282238 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd6a80a39b0 0x7fd694004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:13:21.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.990+0000 7fd6a5ffb640 10 monclient(hunting): picked mon.c con 0x7fd6a80a39b0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:13:21.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.990+0000 7fd6a5ffb640 1 --2- 172.21.15.42:0/3808282238 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd694004da0 0x7fd694005220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:13:21.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.990+0000 7fd6a5ffb640 10 monclient(hunting): picked mon.a con 0x7fd694004da0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:13:21.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.990+0000 7fd6a5ffb640 10 monclient(hunting): start opening mon connection 2024-09-16T04:13:21.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.990+0000 7fd6a5ffb640 1 -- 172.21.15.42:0/3808282238 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fd6a806dd60 con 0x7fd694004da0 2024-09-16T04:13:21.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.990+0000 7fd6a5ffb640 10 monclient(hunting): start opening mon connection 2024-09-16T04:13:21.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.990+0000 7fd6a5ffb640 1 -- 172.21.15.42:0/3808282238 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd69400bc50 con 0x7fd6940039f0 2024-09-16T04:13:21.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.990+0000 7fd6a5ffb640 10 monclient(hunting): start opening mon connection 2024-09-16T04:13:21.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.990+0000 7fd6a5ffb640 1 -- 172.21.15.42:0/3808282238 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fd69400be50 con 0x7fd6a80a39b0 2024-09-16T04:13:21.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.990+0000 7fd6a5ffb640 10 monclient(hunting): _renew_subs 2024-09-16T04:13:21.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.990+0000 7fd6a5ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:13:21.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.990+0000 7fd6a67fc640 1 --2- 172.21.15.42:0/3808282238 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd6a80a39b0 0x7fd694004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:13:21.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.990+0000 7fd6a77fe640 1 --2- 172.21.15.42:0/3808282238 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd694004da0 0x7fd694005220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:13:21.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.990+0000 7fd6a67fc640 10 monclient(hunting): get_auth_request con 0x7fd6a80a39b0 auth_method 0 2024-09-16T04:13:21.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.990+0000 7fd6a67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:13:21.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.990+0000 7fd6a67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:21.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.990+0000 7fd6a67fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:13:21.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.990+0000 7fd6a77fe640 10 monclient(hunting): get_auth_request con 0x7fd694004da0 auth_method 0 2024-09-16T04:13:21.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.990+0000 7fd6a77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:13:21.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.990+0000 7fd6a77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:21.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.990+0000 7fd6a77fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:13:21.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.990+0000 7fd6a67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:13:21.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.990+0000 7fd6a67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:13:21.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.990+0000 7fd6a67fc640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:13:21.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.990+0000 7fd6a77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:13:21.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.990+0000 7fd6a77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:13:21.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.990+0000 7fd6a77fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:13:21.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.990+0000 7fd6a6ffd640 1 --2- 172.21.15.42:0/3808282238 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd6940039f0 0x7fd694003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:13:21.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.990+0000 7fd6a6ffd640 10 monclient(hunting): get_auth_request con 0x7fd6940039f0 auth_method 0 2024-09-16T04:13:21.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.990+0000 7fd6a6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:13:21.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.990+0000 7fd6a6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:21.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.990+0000 7fd6a6ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:13:21.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.990+0000 7fd6a77fe640 10 monclient(hunting): handle_auth_done global_id 1034589 payload 293 2024-09-16T04:13:21.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.990+0000 7fd6a77fe640 1 -- 172.21.15.42:0/3808282238 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd6a80a39b0 msgr2=0x7fd694004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:13:21.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.990+0000 7fd6a77fe640 1 --2- 172.21.15.42:0/3808282238 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd6a80a39b0 0x7fd694004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:13:21.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.990+0000 7fd6a77fe640 1 -- 172.21.15.42:0/3808282238 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd6940039f0 msgr2=0x7fd694003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:13:21.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.990+0000 7fd6a77fe640 1 --2- 172.21.15.42:0/3808282238 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd6940039f0 0x7fd694003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:13:21.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.990+0000 7fd6a77fe640 10 monclient: _finish_hunting 0 2024-09-16T04:13:21.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.990+0000 7fd6a77fe640 1 monclient: found mon.a 2024-09-16T04:13:21.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.990+0000 7fd6a77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:13:21.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.990+0000 7fd6a77fe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:13:21.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.990+0000 7fd6a77fe640 1 -- 172.21.15.42:0/3808282238 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd694003530 con 0x7fd694004da0 2024-09-16T04:13:21.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.990+0000 7fd6a77fe640 10 monclient: _finish_auth 0 2024-09-16T04:13:21.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.990+0000 7fd6a77fe640 10 monclient: _check_auth_tickets 2024-09-16T04:13:21.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.990+0000 7fd6a77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:13:21.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.990+0000 7fd6ad5aa640 5 monclient: authenticate success, global_id 1034589 2024-09-16T04:13:21.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.990+0000 7fd6ad5aa640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:13:21.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.990+0000 7fd6a67fc640 1 --2- 172.21.15.42:0/3808282238 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd6a80a39b0 0x7fd694004810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:13:21.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.994+0000 7fd6a77fe640 1 --2- 172.21.15.42:0/3808282238 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd694004da0 0x7fd694005220 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fd6a0001e40 tx=0x7fd6a001c720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=731027ad06fabd47 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:13:21.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.994+0000 7fd6a5ffb640 1 -- 172.21.15.42:0/3808282238 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6a00b0070 con 0x7fd694004da0 2024-09-16T04:13:21.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.994+0000 7fd6a5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:13:21.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.994+0000 7fd6a5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:13:21.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.994+0000 7fd6a5ffb640 10 monclient: dump: 2024-09-16T04:13:21.014 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:13:21.014 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:13:21.014 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:21.014 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:21.014 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:21.014 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:13:21.015 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:13:21.015 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:13:21.015 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:13:21.015 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:13:21.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.994+0000 7fd6ad5aa640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:13:21.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.994+0000 7fd6a5ffb640 1 -- 172.21.15.42:0/3808282238 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd6a0003b80 con 0x7fd694004da0 2024-09-16T04:13:21.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.994+0000 7fd6a5ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:13:21.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.994+0000 7fd6ad5aa640 10 monclient: get_monmap_and_config success 2024-09-16T04:13:21.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.994+0000 7fd6ad5aa640 4 set_mon_vals no callback set 2024-09-16T04:13:21.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.994+0000 7fd6a5ffb640 1 -- 172.21.15.42:0/3808282238 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6a0001a90 con 0x7fd694004da0 2024-09-16T04:13:21.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.994+0000 7fd6a5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:13:21.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.994+0000 7fd6a5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:13:21.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.994+0000 7fd6a5ffb640 10 monclient: dump: 2024-09-16T04:13:21.016 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:13:21.016 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:13:21.016 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:21.016 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:21.016 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:21.016 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:13:21.016 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:13:21.016 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:13:21.016 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:13:21.016 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:13:21.016 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.994+0000 7fd6ad5aa640 10 monclient: shutdown 2024-09-16T04:13:21.016 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.994+0000 7fd6ad5aa640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:13:21.016 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.994+0000 7fd6ad5aa640 1 -- 172.21.15.42:0/3808282238 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd694004da0 msgr2=0x7fd694005220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:21.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.994+0000 7fd6ad5aa640 1 --2- 172.21.15.42:0/3808282238 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd694004da0 0x7fd694005220 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fd6a0001e40 tx=0x7fd6a001c720 comp rx=0 tx=0).stop 2024-09-16T04:13:21.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.994+0000 7fd6a77fe640 1 -- 172.21.15.42:0/3808282238 reap_dead start 2024-09-16T04:13:21.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.994+0000 7fd6ad5aa640 1 -- 172.21.15.42:0/3808282238 shutdown_connections 2024-09-16T04:13:21.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.994+0000 7fd6ad5aa640 1 -- 172.21.15.42:0/3808282238 >> 172.21.15.42:0/3808282238 conn(0x7fd6a80a9fb0 msgr2=0x7fd6a80aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:13:21.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.994+0000 7fd6ad5aa640 1 -- 172.21.15.42:0/3808282238 shutdown_connections 2024-09-16T04:13:21.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.994+0000 7fd6ad5aa640 1 -- 172.21.15.42:0/3808282238 wait complete. 2024-09-16T04:13:21.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.994+0000 7fd6ad5aa640 10 monclient: build_initial_monmap 2024-09-16T04:13:21.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.994+0000 7fd6ad5aa640 10 monclient: monmap: 2024-09-16T04:13:21.017 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:13:21.017 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:13:21.017 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:13:21.017 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:13:21.017 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:13:21.018 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:13:21.018 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:13:21.018 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:13:21.018 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:13:21.018 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:13:21.018 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.994+0000 7fd6ad5aa640 1 librados: starting msgr at 2024-09-16T04:13:21.018 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.994+0000 7fd6ad5aa640 1 librados: starting objecter 2024-09-16T04:13:21.018 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.994+0000 7fd6ad5aa640 1 Processor -- start 2024-09-16T04:13:21.018 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.994+0000 7fd6ad5aa640 1 -- start start 2024-09-16T04:13:21.019 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.994+0000 7fd6ad5aa640 1 librados: setting wanted keys 2024-09-16T04:13:21.019 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.994+0000 7fd6ad5aa640 1 librados: calling monclient init 2024-09-16T04:13:21.019 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.994+0000 7fd6ad5aa640 10 monclient: init 2024-09-16T04:13:21.019 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.994+0000 7fd6ad5aa640 10 monclient: _reopen_session rank -1 2024-09-16T04:13:21.019 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.994+0000 7fd6ad5aa640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:13:21.019 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.994+0000 7fd6ad5aa640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd694004da0 0x7fd6a813d850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:13:21.019 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.994+0000 7fd6ad5aa640 10 monclient(hunting): picked mon.noname-a con 0x7fd694004da0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:13:21.019 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.994+0000 7fd6ad5aa640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd6a813dd90 0x7fd6a813e1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:13:21.019 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.994+0000 7fd6ad5aa640 10 monclient(hunting): picked mon.noname-c con 0x7fd6a813dd90 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:13:21.019 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.994+0000 7fd6ad5aa640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd6a81cf4d0 0x7fd6a81cf950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:13:21.020 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.994+0000 7fd6ad5aa640 10 monclient(hunting): picked mon.noname-b con 0x7fd6a81cf4d0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:13:21.020 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.994+0000 7fd6ad5aa640 10 monclient(hunting): start opening mon connection 2024-09-16T04:13:21.020 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.994+0000 7fd6ad5aa640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fd6a8140cd0 con 0x7fd694004da0 2024-09-16T04:13:21.020 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.994+0000 7fd6ad5aa640 10 monclient(hunting): start opening mon connection 2024-09-16T04:13:21.020 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.994+0000 7fd6ad5aa640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd6a806df20 con 0x7fd6a81cf4d0 2024-09-16T04:13:21.020 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.994+0000 7fd6ad5aa640 10 monclient(hunting): start opening mon connection 2024-09-16T04:13:21.020 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.994+0000 7fd6ad5aa640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fd6a8140e50 con 0x7fd6a813dd90 2024-09-16T04:13:21.020 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.994+0000 7fd6ad5aa640 10 monclient(hunting): _renew_subs 2024-09-16T04:13:21.020 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.994+0000 7fd6ad5aa640 10 monclient(hunting): authenticate will time out at 2847.967819s 2024-09-16T04:13:21.021 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.994+0000 7fd6a67fc640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd6a813dd90 0x7fd6a813e1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:13:21.021 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.994+0000 7fd6a6ffd640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd694004da0 0x7fd6a813d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:13:21.021 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.994+0000 7fd6a67fc640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd6a813dd90 0x7fd6a813e1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:41040/0 (socket says 172.21.15.42:41040) 2024-09-16T04:13:21.021 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.994+0000 7fd6a67fc640 1 -- 172.21.15.42:0/401141881 learned_addr learned my addr 172.21.15.42:0/401141881 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:13:21.021 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.994+0000 7fd6a67fc640 10 monclient(hunting): get_auth_request con 0x7fd6a813dd90 auth_method 0 2024-09-16T04:13:21.021 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.994+0000 7fd6a67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:13:21.021 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.994+0000 7fd6a67fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:21.021 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.994+0000 7fd6a67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:13:21.021 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.994+0000 7fd6a6ffd640 10 monclient(hunting): get_auth_request con 0x7fd694004da0 auth_method 0 2024-09-16T04:13:21.021 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.994+0000 7fd6a6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:13:21.021 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.994+0000 7fd6a6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:21.021 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.994+0000 7fd6a6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:13:21.021 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.998+0000 7fd6a67fc640 1 -- 172.21.15.42:0/401141881 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd6a813dd90 msgr2=0x7fd6a813e1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-16T04:13:21.022 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.998+0000 7fd6a67fc640 1 -- 172.21.15.42:0/401141881 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd6a813dd90 msgr2=0x7fd6a813e1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:13:21.022 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.998+0000 7fd6a67fc640 1 --2- 172.21.15.42:0/401141881 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd6a813dd90 0x7fd6a813e1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:13:21.022 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.998+0000 7fd6a6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:13:21.022 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.998+0000 7fd6a6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:13:21.022 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.998+0000 7fd6a6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:13:21.022 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.998+0000 7fd6a67fc640 1 --2- 172.21.15.42:0/401141881 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd6a813dd90 0x7fd6a813e1d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:13:21.022 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.998+0000 7fd6a77fe640 1 --2- 172.21.15.42:0/401141881 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd6a81cf4d0 0x7fd6a81cf950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:13:21.022 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.998+0000 7fd6a6ffd640 10 monclient(hunting): handle_auth_done global_id 1034592 payload 824 2024-09-16T04:13:21.022 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.998+0000 7fd6a6ffd640 1 -- 172.21.15.42:0/401141881 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd6a813dd90 msgr2=0x7fd6a813e1d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:13:21.022 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.998+0000 7fd6a6ffd640 1 --2- 172.21.15.42:0/401141881 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd6a813dd90 0x7fd6a813e1d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:13:21.022 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.998+0000 7fd6a6ffd640 1 -- 172.21.15.42:0/401141881 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd6a81cf4d0 msgr2=0x7fd6a81cf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:21.022 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.998+0000 7fd6a6ffd640 1 --2- 172.21.15.42:0/401141881 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd6a81cf4d0 0x7fd6a81cf950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:13:21.022 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.998+0000 7fd6a6ffd640 10 monclient: _finish_hunting 0 2024-09-16T04:13:21.023 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.998+0000 7fd6a6ffd640 1 monclient: found mon.noname-a 2024-09-16T04:13:21.023 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.998+0000 7fd6a6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:13:21.023 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.998+0000 7fd6a6ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:13:21.023 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.998+0000 7fd6a6ffd640 1 -- 172.21.15.42:0/401141881 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6a8132e00 con 0x7fd694004da0 2024-09-16T04:13:21.023 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.998+0000 7fd6a77fe640 10 monclient: get_auth_request con 0x7fd6a81cf4d0 auth_method 0 2024-09-16T04:13:21.023 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.998+0000 7fd6a77fe640 1 --2- 172.21.15.42:0/401141881 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd6a81cf4d0 0x7fd6a81cf950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T04:13:21.023 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.998+0000 7fd6a6ffd640 1 --2- 172.21.15.42:0/401141881 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd694004da0 0x7fd6a813d850 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fd690010860 tx=0x7fd690010d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:13:21.023 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.998+0000 7fd69f7fe640 10 client.?.objecter ms_handle_connect 0x7fd694004da0 2024-09-16T04:13:21.023 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.998+0000 7fd69f7fe640 10 client.?.objecter resend_mon_ops 2024-09-16T04:13:21.023 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.998+0000 7fd69f7fe640 1 -- 172.21.15.42:0/401141881 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd69001a070 con 0x7fd694004da0 2024-09-16T04:13:21.023 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.998+0000 7fd69f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:13:21.023 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.998+0000 7fd69f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:13:21.023 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.998+0000 7fd69f7fe640 10 monclient: dump: 2024-09-16T04:13:21.024 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:13:21.024 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:13:21.024 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:21.024 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:21.024 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:21.024 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:13:21.024 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:13:21.024 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:13:21.024 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:13:21.024 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:13:21.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.998+0000 7fd69f7fe640 10 monclient: _finish_auth 0 2024-09-16T04:13:21.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.998+0000 7fd69f7fe640 10 monclient: _check_auth_tickets 2024-09-16T04:13:21.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.998+0000 7fd69f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:13:21.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.998+0000 7fd69f7fe640 1 -- 172.21.15.42:0/401141881 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd690004220 con 0x7fd694004da0 2024-09-16T04:13:21.025 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.998+0000 7fd69f7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:13:21.025 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.998+0000 7fd69f7fe640 1 -- 172.21.15.42:0/401141881 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd69000f040 con 0x7fd694004da0 2024-09-16T04:13:21.025 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.998+0000 7fd6ad5aa640 5 monclient: authenticate success, global_id 1034592 2024-09-16T04:13:21.025 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.998+0000 7fd6ad5aa640 10 monclient: _renew_subs 2024-09-16T04:13:21.025 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.998+0000 7fd6ad5aa640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:13:21.025 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.998+0000 7fd6ad5aa640 1 -- 172.21.15.42:0/401141881 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6a809db00 con 0x7fd694004da0 2024-09-16T04:13:21.025 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.998+0000 7fd6a57fa640 4 set_mon_vals no callback set 2024-09-16T04:13:21.025 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.998+0000 7fd69f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:13:21.025 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.998+0000 7fd69f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:13:21.025 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.998+0000 7fd69f7fe640 10 monclient: dump: 2024-09-16T04:13:21.025 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:13:21.025 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:13:21.026 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:21.026 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:21.026 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:21.026 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:13:21.026 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:13:21.026 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:13:21.026 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:13:21.026 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:13:21.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.998+0000 7fd6ad5aa640 10 client.1034592.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:13:21.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.998+0000 7fd6ad5aa640 10 monclient: _renew_subs 2024-09-16T04:13:21.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.998+0000 7fd6ad5aa640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:13:21.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.998+0000 7fd6ad5aa640 1 -- 172.21.15.42:0/401141881 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6a806f680 con 0x7fd694004da0 2024-09-16T04:13:21.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.998+0000 7fd6ad5aa640 1 librados: init done 2024-09-16T04:13:21.026 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.998+0000 7fd69f7fe640 1 -- 172.21.15.42:0/401141881 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fd69000c730 con 0x7fd694004da0 2024-09-16T04:13:21.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.998+0000 7fd6ad5aa640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:13:21.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.998+0000 7fd6ad5aa640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:13:21.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.998+0000 7fd6ad5aa640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:13:21.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.998+0000 7fd6ad5aa640 1 -- 172.21.15.42:0/401141881 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd6a8137000 con 0x7fd694004da0 2024-09-16T04:13:21.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.998+0000 7fd69f7fe640 1 --2- 172.21.15.42:0/401141881 >> v2:172.21.15.154:6812/254845342 conn(0x7fd688037810 0x7fd688039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:13:21.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.998+0000 7fd69f7fe640 1 -- 172.21.15.42:0/401141881 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(706..706 src has 681..706) ==== 5040+0+0 (secure 0 0 0) 0x7fd690048ff0 con 0x7fd694004da0 2024-09-16T04:13:21.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.998+0000 7fd69f7fe640 10 client.1034592.objecter ms_dispatch 0x7fd6a8000ba0 osd_map(706..706 src has 681..706) 2024-09-16T04:13:21.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.998+0000 7fd69f7fe640 3 client.1034592.objecter handle_osd_map got epochs [706,706] > 0 2024-09-16T04:13:21.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.998+0000 7fd69f7fe640 3 client.1034592.objecter handle_osd_map decoding full epoch 706 2024-09-16T04:13:21.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.998+0000 7fd69f7fe640 20 client.1034592.objecter dump_active .. 0 homeless 2024-09-16T04:13:21.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.998+0000 7fd6a67fc640 1 --2- 172.21.15.42:0/401141881 >> v2:172.21.15.154:6812/254845342 conn(0x7fd688037810 0x7fd688039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:13:21.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.998+0000 7fd6a67fc640 10 monclient: get_auth_request con 0x7fd688037810 auth_method 0 2024-09-16T04:13:21.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.998+0000 7fd69f7fe640 10 client.1034592.objecter ms_handle_connect 0x7fd688037810 2024-09-16T04:13:21.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:20.998+0000 7fd6a67fc640 1 --2- 172.21.15.42:0/401141881 >> v2:172.21.15.154:6812/254845342 conn(0x7fd688037810 0x7fd688039cd0 secure :-1 s=READY pgs=3982 cs=0 l=1 rev1=1 crypto rx=0x7fd6980010f0 tx=0x7fd698009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:13:21.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:21.002+0000 7fd69f7fe640 1 -- 172.21.15.42:0/401141881 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7fd6a8132e00 con 0x7fd694004da0 2024-09-16T04:13:21.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:21.002+0000 7fd69f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:13:21.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:21.002+0000 7fd69f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:13:21.235 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:21.230+0000 7fd6ad5aa640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:13:21.235 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:21.230+0000 7fd6ad5aa640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:13:21.235 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:21.230+0000 7fd6ad5aa640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:13:21.235 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:21.230+0000 7fd6ad5aa640 1 -- 172.21.15.42:0/401141881 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fd6a80a5750 con 0x7fd694004da0 2024-09-16T04:13:21.235 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:21.230+0000 7fd69f7fe640 1 -- 172.21.15.42:0/401141881 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7fd6900039a0 con 0x7fd694004da0 2024-09-16T04:13:21.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:21.230+0000 7fd69f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:13:21.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:21.230+0000 7fd69f7fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:13:21.236 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":280,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":21,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:13:21.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:21.234+0000 7fd6ad5aa640 10 librados: watch_flush enter 2024-09-16T04:13:21.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:21.234+0000 7fd6ad5aa640 10 librados: watch_flush exit 2024-09-16T04:13:21.238 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:21.234+0000 7fd6ad5aa640 20 client.1034592.objecter shutdown clearing up homeless session... 2024-09-16T04:13:21.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:21.234+0000 7fd6ad5aa640 10 client.1034592.objecter successfully canceled tick 2024-09-16T04:13:21.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:21.234+0000 7fd6ad5aa640 1 -- 172.21.15.42:0/401141881 >> v2:172.21.15.154:6812/254845342 conn(0x7fd688037810 msgr2=0x7fd688039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:21.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:21.234+0000 7fd6ad5aa640 1 --2- 172.21.15.42:0/401141881 >> v2:172.21.15.154:6812/254845342 conn(0x7fd688037810 0x7fd688039cd0 secure :-1 s=READY pgs=3982 cs=0 l=1 rev1=1 crypto rx=0x7fd6980010f0 tx=0x7fd698009040 comp rx=0 tx=0).stop 2024-09-16T04:13:21.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:21.234+0000 7fd6ad5aa640 10 monclient: shutdown 2024-09-16T04:13:21.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:21.234+0000 7fd6ad5aa640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:13:21.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:21.234+0000 7fd6ad5aa640 1 -- 172.21.15.42:0/401141881 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd694004da0 msgr2=0x7fd6a813d850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:21.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:21.234+0000 7fd6ad5aa640 1 --2- 172.21.15.42:0/401141881 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd694004da0 0x7fd6a813d850 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fd690010860 tx=0x7fd690010d20 comp rx=0 tx=0).stop 2024-09-16T04:13:21.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:21.234+0000 7fd6ad5aa640 1 -- 172.21.15.42:0/401141881 shutdown_connections 2024-09-16T04:13:21.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:21.234+0000 7fd6ad5aa640 1 --2- 172.21.15.42:0/401141881 >> v2:172.21.15.154:6812/254845342 conn(0x7fd688037810 0x7fd688039cd0 unknown :-1 s=CLOSED pgs=3982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:13:21.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:21.234+0000 7fd6ad5aa640 1 --2- 172.21.15.42:0/401141881 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd6a81cf4d0 0x7fd6a81cf950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:13:21.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:21.234+0000 7fd6ad5aa640 1 --2- 172.21.15.42:0/401141881 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd6a813dd90 0x7fd6a813e1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:13:21.239 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:21.234+0000 7fd6ad5aa640 1 --2- 172.21.15.42:0/401141881 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd694004da0 0x7fd6a813d850 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:13:21.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:21.234+0000 7fd6ad5aa640 1 -- 172.21.15.42:0/401141881 >> 172.21.15.42:0/401141881 conn(0x7fd6a80a9fb0 msgr2=0x7fd6a80a8360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:13:21.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:21.234+0000 7fd6ad5aa640 1 -- 172.21.15.42:0/401141881 shutdown_connections 2024-09-16T04:13:21.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:21.234+0000 7fd6ad5aa640 1 -- 172.21.15.42:0/401141881 wait complete. 2024-09-16T04:13:21.240 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:21.234+0000 7fd6ad5aa640 1 librados: shutdown 2024-09-16T04:13:24.257 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:13:24.389 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:24.382+0000 7f5a93b7a640 10 monclient: get_monmap_and_config 2024-09-16T04:13:24.389 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:24.382+0000 7f5a93b7a640 10 monclient: build_initial_monmap 2024-09-16T04:13:24.389 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:24.382+0000 7f5a93b7a640 10 monclient: monmap: 2024-09-16T04:13:24.389 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:13:24.389 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:13:24.389 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:13:24.387695+0000 2024-09-16T04:13:24.389 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:13:24.387695+0000 2024-09-16T04:13:24.389 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:13:24.389 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:13:24.389 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:13:24.390 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:13:24.390 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:13:24.390 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:13:24.390 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:13:24.390 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:13:24.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:24.386+0000 7f5a93b7a640 1 Processor -- start 2024-09-16T04:13:24.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:24.386+0000 7f5a93b7a640 1 -- start start 2024-09-16T04:13:24.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:24.386+0000 7f5a93b7a640 10 monclient: init 2024-09-16T04:13:24.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:24.386+0000 7f5a93b7a640 10 monclient: _reopen_session rank -1 2024-09-16T04:13:24.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:24.386+0000 7f5a93b7a640 10 monclient: _add_conns ranks=[0,2,1,4,3] 2024-09-16T04:13:24.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:24.386+0000 7f5a93b7a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5a8c0a39b0 addr v1:172.21.15.42:6789/0 2024-09-16T04:13:24.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:24.386+0000 7f5a93b7a640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5a8c0a3d90 0x7f5a8c13c380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:13:24.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:24.386+0000 7f5a93b7a640 10 monclient(hunting): picked mon.noname-b con 0x7f5a8c0a3d90 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:13:24.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:24.386+0000 7f5a93b7a640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5a8c13c8c0 addr v1:172.21.15.154:6789/0 2024-09-16T04:13:24.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:24.386+0000 7f5a93b7a640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5a8c134290 con 0x7f5a8c0a39b0 2024-09-16T04:13:24.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:24.386+0000 7f5a93b7a640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5a8c09c140 con 0x7f5a8c13c8c0 2024-09-16T04:13:24.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:24.386+0000 7f5a93b7a640 10 monclient(hunting): start opening mon connection 2024-09-16T04:13:24.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:24.386+0000 7f5a93b7a640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f5a8c09c420 con 0x7f5a8c0a3d90 2024-09-16T04:13:24.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:24.386+0000 7f5a93b7a640 10 monclient(hunting): _renew_subs 2024-09-16T04:13:24.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:24.386+0000 7f5a93b7a640 10 monclient(hunting): authenticate will time out at 2851.357253s 2024-09-16T04:13:24.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:24.386+0000 7f5a91115640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5a8c0a3d90 0x7f5a8c13c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:13:24.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:24.386+0000 7f5a91916640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f5a8c0a39b0 0x7f5a8c133b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:35742/0 (socket says 172.21.15.42:35742) 2024-09-16T04:13:24.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:24.386+0000 7f5a91916640 1 -- 172.21.15.42:0/2893455814 learned_addr learned my addr 172.21.15.42:0/2893455814 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T04:13:24.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:24.386+0000 7f5a91115640 10 monclient(hunting): get_auth_request con 0x7f5a8c0a3d90 auth_method 0 2024-09-16T04:13:24.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:24.386+0000 7f5a91115640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:13:24.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:24.386+0000 7f5a91115640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:24.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:24.386+0000 7f5a91115640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:13:24.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:24.386+0000 7f5a91115640 1 -- 172.21.15.42:0/2893455814 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5a8c0a3d90 msgr2=0x7f5a8c13c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T04:13:24.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:24.386+0000 7f5a91115640 1 -- 172.21.15.42:0/2893455814 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5a8c0a3d90 msgr2=0x7f5a8c13c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:13:24.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:24.386+0000 7f5a91115640 1 --2- 172.21.15.42:0/2893455814 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5a8c0a3d90 0x7f5a8c13c380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:13:24.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:24.386+0000 7f5a91115640 1 --2- 172.21.15.42:0/2893455814 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5a8c0a3d90 0x7f5a8c13c380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:13:24.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:24.590+0000 7f5a91115640 1 --2- 172.21.15.42:0/2893455814 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5a8c0a3d90 0x7f5a8c13c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:13:24.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:24.590+0000 7f5a91115640 10 monclient(hunting): get_auth_request con 0x7f5a8c0a3d90 auth_method 0 2024-09-16T04:13:24.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:24.590+0000 7f5a91115640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:13:24.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:24.590+0000 7f5a91115640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:24.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:24.590+0000 7f5a91115640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:13:24.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:24.590+0000 7f5a91115640 1 -- 172.21.15.42:0/2893455814 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5a8c0a3d90 msgr2=0x7f5a8c13c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T04:13:24.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:24.590+0000 7f5a91115640 1 -- 172.21.15.42:0/2893455814 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5a8c0a3d90 msgr2=0x7f5a8c13c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:13:24.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:24.590+0000 7f5a91115640 1 --2- 172.21.15.42:0/2893455814 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5a8c0a3d90 0x7f5a8c13c380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:13:24.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:24.590+0000 7f5a91115640 1 --2- 172.21.15.42:0/2893455814 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5a8c0a3d90 0x7f5a8c13c380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T04:13:24.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:24.994+0000 7f5a91115640 1 --2- 172.21.15.42:0/2893455814 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5a8c0a3d90 0x7f5a8c13c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:13:24.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:24.994+0000 7f5a91115640 10 monclient(hunting): get_auth_request con 0x7f5a8c0a3d90 auth_method 0 2024-09-16T04:13:24.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:24.994+0000 7f5a91115640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:13:24.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:24.994+0000 7f5a91115640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:24.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:24.994+0000 7f5a91115640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:13:24.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:24.994+0000 7f5a91115640 1 -- 172.21.15.42:0/2893455814 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5a8c0a3d90 msgr2=0x7f5a8c13c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T04:13:24.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:24.994+0000 7f5a91115640 1 -- 172.21.15.42:0/2893455814 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5a8c0a3d90 msgr2=0x7f5a8c13c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:13:24.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:24.994+0000 7f5a91115640 1 --2- 172.21.15.42:0/2893455814 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5a8c0a3d90 0x7f5a8c13c380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:13:24.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:24.994+0000 7f5a91115640 1 --2- 172.21.15.42:0/2893455814 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5a8c0a3d90 0x7f5a8c13c380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T04:13:25.802 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:25.798+0000 7f5a91115640 1 --2- 172.21.15.42:0/2893455814 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5a8c0a3d90 0x7f5a8c13c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:13:25.802 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:25.798+0000 7f5a91115640 10 monclient(hunting): get_auth_request con 0x7f5a8c0a3d90 auth_method 0 2024-09-16T04:13:25.802 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:25.798+0000 7f5a91115640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:13:25.802 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:25.798+0000 7f5a91115640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:25.802 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:25.798+0000 7f5a91115640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:13:25.802 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:25.798+0000 7f5a91115640 1 -- 172.21.15.42:0/2893455814 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5a8c0a3d90 msgr2=0x7f5a8c13c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T04:13:25.803 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:25.798+0000 7f5a91115640 1 -- 172.21.15.42:0/2893455814 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5a8c0a3d90 msgr2=0x7f5a8c13c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:13:25.803 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:25.798+0000 7f5a91115640 1 --2- 172.21.15.42:0/2893455814 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5a8c0a3d90 0x7f5a8c13c380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:13:25.803 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:25.798+0000 7f5a91115640 1 --2- 172.21.15.42:0/2893455814 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5a8c0a3d90 0x7f5a8c13c380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T04:13:27.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:27.386+0000 7f5a7f7fe640 10 monclient(hunting): tick 2024-09-16T04:13:27.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:27.386+0000 7f5a7f7fe640 10 monclient(hunting): _check_auth_tickets 2024-09-16T04:13:27.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:27.386+0000 7f5a7f7fe640 1 monclient(hunting): continuing hunt 2024-09-16T04:13:27.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:27.386+0000 7f5a7f7fe640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T04:13:27.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:27.386+0000 7f5a7f7fe640 1 -- 172.21.15.42:0/2893455814 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5a8c0a3d90 msgr2=0x7f5a8c13c380 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:13:27.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:27.386+0000 7f5a7f7fe640 1 --2- 172.21.15.42:0/2893455814 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5a8c0a3d90 0x7f5a8c13c380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:13:27.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:27.386+0000 7f5a7f7fe640 1 -- 172.21.15.42:0/2893455814 >> v1:172.21.15.154:6789/0 conn(0x7f5a8c13c8c0 legacy=0x7f5a8c13ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:27.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:27.386+0000 7f5a7f7fe640 1 -- 172.21.15.42:0/2893455814 >> v1:172.21.15.42:6789/0 conn(0x7f5a8c0a39b0 legacy=0x7f5a8c133b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:27.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:27.386+0000 7f5a7f7fe640 10 monclient: _add_conns ranks=[4,3] 2024-09-16T04:13:27.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:27.386+0000 7f5a7f7fe640 1 --2- 172.21.15.42:0/2893455814 >> v2:172.21.15.154:3300/0 conn(0x7f5a70003720 0x7f5a70005b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:13:27.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:27.386+0000 7f5a7f7fe640 10 monclient(hunting): picked mon.noname-c con 0x7f5a70003720 addr v2:172.21.15.154:3300/0 2024-09-16T04:13:27.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:27.386+0000 7f5a7f7fe640 1 --2- 172.21.15.42:0/2893455814 >> v2:172.21.15.42:3300/0 conn(0x7f5a70006270 0x7f5a70008730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:13:27.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:27.386+0000 7f5a7f7fe640 10 monclient(hunting): picked mon.noname-a con 0x7f5a70006270 addr v2:172.21.15.42:3300/0 2024-09-16T04:13:27.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:27.386+0000 7f5a7f7fe640 10 monclient(hunting): start opening mon connection 2024-09-16T04:13:27.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:27.386+0000 7f5a7f7fe640 1 -- 172.21.15.42:0/2893455814 --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f5a8c09c420 con 0x7f5a70006270 2024-09-16T04:13:27.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:27.386+0000 7f5a7f7fe640 10 monclient(hunting): start opening mon connection 2024-09-16T04:13:27.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:27.386+0000 7f5a7f7fe640 1 -- 172.21.15.42:0/2893455814 --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f5a70008df0 con 0x7f5a70003720 2024-09-16T04:13:27.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:27.386+0000 7f5a7f7fe640 10 monclient(hunting): _renew_subs 2024-09-16T04:13:27.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:27.386+0000 7f5a91115640 1 --2- 172.21.15.42:0/2893455814 >> v2:172.21.15.42:3300/0 conn(0x7f5a70006270 0x7f5a70008730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:13:27.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:27.386+0000 7f5a91916640 1 --2- 172.21.15.42:0/2893455814 >> v2:172.21.15.154:3300/0 conn(0x7f5a70003720 0x7f5a70005b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:13:27.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:27.386+0000 7f5a91115640 10 monclient(hunting): get_auth_request con 0x7f5a70006270 auth_method 0 2024-09-16T04:13:27.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:27.386+0000 7f5a91115640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:13:27.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:27.386+0000 7f5a91115640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:27.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:27.386+0000 7f5a91115640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:13:27.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:27.386+0000 7f5a91916640 10 monclient(hunting): get_auth_request con 0x7f5a70003720 auth_method 0 2024-09-16T04:13:27.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:27.386+0000 7f5a91916640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:13:27.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:27.386+0000 7f5a91916640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:27.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:27.386+0000 7f5a91916640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:13:27.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:27.386+0000 7f5a91916640 1 -- 172.21.15.42:0/2893455814 >> v2:172.21.15.154:3300/0 conn(0x7f5a70003720 msgr2=0x7f5a70005b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:13:27.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:27.386+0000 7f5a91916640 1 -- 172.21.15.42:0/2893455814 >> v2:172.21.15.154:3300/0 conn(0x7f5a70003720 msgr2=0x7f5a70005b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:13:27.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:27.386+0000 7f5a91916640 1 --2- 172.21.15.42:0/2893455814 >> v2:172.21.15.154:3300/0 conn(0x7f5a70003720 0x7f5a70005b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:13:27.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:27.386+0000 7f5a91916640 1 --2- 172.21.15.42:0/2893455814 >> v2:172.21.15.154:3300/0 conn(0x7f5a70003720 0x7f5a70005b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:13:27.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:27.386+0000 7f5a91115640 1 -- 172.21.15.42:0/2893455814 >> v2:172.21.15.42:3300/0 conn(0x7f5a70006270 msgr2=0x7f5a70008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:13:27.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:27.386+0000 7f5a91115640 1 -- 172.21.15.42:0/2893455814 >> v2:172.21.15.42:3300/0 conn(0x7f5a70006270 msgr2=0x7f5a70008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:13:27.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:27.386+0000 7f5a91115640 1 --2- 172.21.15.42:0/2893455814 >> v2:172.21.15.42:3300/0 conn(0x7f5a70006270 0x7f5a70008730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:13:27.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:27.386+0000 7f5a91115640 1 --2- 172.21.15.42:0/2893455814 >> v2:172.21.15.42:3300/0 conn(0x7f5a70006270 0x7f5a70008730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:13:27.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:27.590+0000 7f5a91115640 1 --2- 172.21.15.42:0/2893455814 >> v2:172.21.15.42:3300/0 conn(0x7f5a70006270 0x7f5a70008730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:13:27.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:27.590+0000 7f5a91115640 10 monclient(hunting): get_auth_request con 0x7f5a70006270 auth_method 0 2024-09-16T04:13:27.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:27.590+0000 7f5a91115640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:13:27.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:27.590+0000 7f5a91115640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:27.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:27.590+0000 7f5a91115640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:13:27.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:27.590+0000 7f5a91916640 1 --2- 172.21.15.42:0/2893455814 >> v2:172.21.15.154:3300/0 conn(0x7f5a70003720 0x7f5a70005b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:13:27.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:27.590+0000 7f5a91916640 10 monclient(hunting): get_auth_request con 0x7f5a70003720 auth_method 0 2024-09-16T04:13:27.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:27.590+0000 7f5a91916640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:13:27.593 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:27.590+0000 7f5a91916640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:27.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:27.590+0000 7f5a91916640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:13:27.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:27.590+0000 7f5a91115640 1 -- 172.21.15.42:0/2893455814 >> v2:172.21.15.42:3300/0 conn(0x7f5a70006270 msgr2=0x7f5a70008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:13:27.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:27.590+0000 7f5a91115640 1 -- 172.21.15.42:0/2893455814 >> v2:172.21.15.42:3300/0 conn(0x7f5a70006270 msgr2=0x7f5a70008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:13:27.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:27.590+0000 7f5a91115640 1 --2- 172.21.15.42:0/2893455814 >> v2:172.21.15.42:3300/0 conn(0x7f5a70006270 0x7f5a70008730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:13:27.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:27.590+0000 7f5a91115640 1 --2- 172.21.15.42:0/2893455814 >> v2:172.21.15.42:3300/0 conn(0x7f5a70006270 0x7f5a70008730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T04:13:27.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:27.590+0000 7f5a91916640 1 -- 172.21.15.42:0/2893455814 >> v2:172.21.15.154:3300/0 conn(0x7f5a70003720 msgr2=0x7f5a70005b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:13:27.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:27.590+0000 7f5a91916640 1 -- 172.21.15.42:0/2893455814 >> v2:172.21.15.154:3300/0 conn(0x7f5a70003720 msgr2=0x7f5a70005b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:13:27.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:27.590+0000 7f5a91916640 1 --2- 172.21.15.42:0/2893455814 >> v2:172.21.15.154:3300/0 conn(0x7f5a70003720 0x7f5a70005b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:13:27.594 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:27.590+0000 7f5a91916640 1 --2- 172.21.15.42:0/2893455814 >> v2:172.21.15.154:3300/0 conn(0x7f5a70003720 0x7f5a70005b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T04:13:27.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:27.994+0000 7f5a91115640 1 --2- 172.21.15.42:0/2893455814 >> v2:172.21.15.42:3300/0 conn(0x7f5a70006270 0x7f5a70008730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:13:27.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:27.994+0000 7f5a91115640 10 monclient(hunting): get_auth_request con 0x7f5a70006270 auth_method 0 2024-09-16T04:13:27.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:27.994+0000 7f5a91115640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:13:27.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:27.994+0000 7f5a91115640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:27.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:27.994+0000 7f5a91115640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:13:27.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:27.994+0000 7f5a91916640 1 --2- 172.21.15.42:0/2893455814 >> v2:172.21.15.154:3300/0 conn(0x7f5a70003720 0x7f5a70005b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:13:27.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:27.994+0000 7f5a91916640 10 monclient(hunting): get_auth_request con 0x7f5a70003720 auth_method 0 2024-09-16T04:13:27.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:27.994+0000 7f5a91916640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:13:27.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:27.994+0000 7f5a91916640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:27.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:27.994+0000 7f5a91916640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:13:27.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:27.994+0000 7f5a91115640 1 -- 172.21.15.42:0/2893455814 >> v2:172.21.15.42:3300/0 conn(0x7f5a70006270 msgr2=0x7f5a70008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:13:27.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:27.994+0000 7f5a91115640 1 -- 172.21.15.42:0/2893455814 >> v2:172.21.15.42:3300/0 conn(0x7f5a70006270 msgr2=0x7f5a70008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:13:27.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:27.994+0000 7f5a91115640 1 --2- 172.21.15.42:0/2893455814 >> v2:172.21.15.42:3300/0 conn(0x7f5a70006270 0x7f5a70008730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:13:27.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:27.994+0000 7f5a91115640 1 --2- 172.21.15.42:0/2893455814 >> v2:172.21.15.42:3300/0 conn(0x7f5a70006270 0x7f5a70008730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T04:13:27.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:27.994+0000 7f5a91916640 1 -- 172.21.15.42:0/2893455814 >> v2:172.21.15.154:3300/0 conn(0x7f5a70003720 msgr2=0x7f5a70005b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:13:27.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:27.994+0000 7f5a91916640 1 -- 172.21.15.42:0/2893455814 >> v2:172.21.15.154:3300/0 conn(0x7f5a70003720 msgr2=0x7f5a70005b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:13:27.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:27.994+0000 7f5a91916640 1 --2- 172.21.15.42:0/2893455814 >> v2:172.21.15.154:3300/0 conn(0x7f5a70003720 0x7f5a70005b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:13:27.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:27.994+0000 7f5a91916640 1 --2- 172.21.15.42:0/2893455814 >> v2:172.21.15.154:3300/0 conn(0x7f5a70003720 0x7f5a70005b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T04:13:28.802 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:28.798+0000 7f5a91115640 1 --2- 172.21.15.42:0/2893455814 >> v2:172.21.15.42:3300/0 conn(0x7f5a70006270 0x7f5a70008730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:13:28.802 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:28.798+0000 7f5a91115640 10 monclient(hunting): get_auth_request con 0x7f5a70006270 auth_method 0 2024-09-16T04:13:28.802 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:28.798+0000 7f5a91115640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:13:28.802 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:28.798+0000 7f5a91115640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:28.802 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:28.798+0000 7f5a91115640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:13:28.802 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:28.798+0000 7f5a91916640 1 --2- 172.21.15.42:0/2893455814 >> v2:172.21.15.154:3300/0 conn(0x7f5a70003720 0x7f5a70005b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:13:28.803 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:28.798+0000 7f5a91115640 1 -- 172.21.15.42:0/2893455814 >> v2:172.21.15.42:3300/0 conn(0x7f5a70006270 msgr2=0x7f5a70008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:13:28.803 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:28.798+0000 7f5a91115640 1 -- 172.21.15.42:0/2893455814 >> v2:172.21.15.42:3300/0 conn(0x7f5a70006270 msgr2=0x7f5a70008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:13:28.803 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:28.798+0000 7f5a91115640 1 --2- 172.21.15.42:0/2893455814 >> v2:172.21.15.42:3300/0 conn(0x7f5a70006270 0x7f5a70008730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:13:28.803 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:28.798+0000 7f5a91916640 10 monclient(hunting): get_auth_request con 0x7f5a70003720 auth_method 0 2024-09-16T04:13:28.803 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:28.798+0000 7f5a91916640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:13:28.803 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:28.798+0000 7f5a91916640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:28.803 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:28.798+0000 7f5a91916640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:13:28.803 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:28.798+0000 7f5a91115640 1 --2- 172.21.15.42:0/2893455814 >> v2:172.21.15.42:3300/0 conn(0x7f5a70006270 0x7f5a70008730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T04:13:28.803 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:28.798+0000 7f5a91916640 1 -- 172.21.15.42:0/2893455814 >> v2:172.21.15.154:3300/0 conn(0x7f5a70003720 msgr2=0x7f5a70005b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:13:28.803 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:28.798+0000 7f5a91916640 1 -- 172.21.15.42:0/2893455814 >> v2:172.21.15.154:3300/0 conn(0x7f5a70003720 msgr2=0x7f5a70005b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:13:28.803 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:28.798+0000 7f5a91916640 1 --2- 172.21.15.42:0/2893455814 >> v2:172.21.15.154:3300/0 conn(0x7f5a70003720 0x7f5a70005b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:13:28.803 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:28.798+0000 7f5a91916640 1 --2- 172.21.15.42:0/2893455814 >> v2:172.21.15.154:3300/0 conn(0x7f5a70003720 0x7f5a70005b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T04:13:30.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.386+0000 7f5a7f7fe640 10 monclient(hunting): tick 2024-09-16T04:13:30.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.386+0000 7f5a7f7fe640 10 monclient(hunting): _check_auth_tickets 2024-09-16T04:13:30.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.386+0000 7f5a7f7fe640 1 monclient(hunting): continuing hunt 2024-09-16T04:13:30.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.386+0000 7f5a7f7fe640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T04:13:30.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.386+0000 7f5a7f7fe640 1 -- 172.21.15.42:0/2893455814 >> v2:172.21.15.154:3300/0 conn(0x7f5a70003720 msgr2=0x7f5a70005b80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:13:30.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.386+0000 7f5a7f7fe640 1 --2- 172.21.15.42:0/2893455814 >> v2:172.21.15.154:3300/0 conn(0x7f5a70003720 0x7f5a70005b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:13:30.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.386+0000 7f5a7f7fe640 1 -- 172.21.15.42:0/2893455814 >> v2:172.21.15.42:3300/0 conn(0x7f5a70006270 msgr2=0x7f5a70008730 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:13:30.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.386+0000 7f5a7f7fe640 1 --2- 172.21.15.42:0/2893455814 >> v2:172.21.15.42:3300/0 conn(0x7f5a70006270 0x7f5a70008730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:13:30.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.386+0000 7f5a7f7fe640 10 monclient: _add_conns ranks=[1,4,0,3,2] 2024-09-16T04:13:30.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.386+0000 7f5a92117640 1 -- 172.21.15.42:0/2893455814 reap_dead start 2024-09-16T04:13:30.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.386+0000 7f5a7f7fe640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5a8c13c8c0 addr v1:172.21.15.154:6789/0 2024-09-16T04:13:30.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.386+0000 7f5a7f7fe640 1 --2- 172.21.15.42:0/2893455814 >> v2:172.21.15.154:3300/0 conn(0x7f5a70003720 0x7f5a70005b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:13:30.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.386+0000 7f5a7f7fe640 10 monclient(hunting): picked mon.noname-c con 0x7f5a70003720 addr v2:172.21.15.154:3300/0 2024-09-16T04:13:30.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.386+0000 7f5a7f7fe640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5a8c0a39b0 addr v1:172.21.15.42:6789/0 2024-09-16T04:13:30.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.386+0000 7f5a7f7fe640 1 -- 172.21.15.42:0/2893455814 --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5a8c06dd60 con 0x7f5a8c0a39b0 2024-09-16T04:13:30.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.386+0000 7f5a7f7fe640 1 -- 172.21.15.42:0/2893455814 --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5a70006720 con 0x7f5a8c13c8c0 2024-09-16T04:13:30.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.386+0000 7f5a7f7fe640 10 monclient(hunting): start opening mon connection 2024-09-16T04:13:30.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.386+0000 7f5a7f7fe640 1 -- 172.21.15.42:0/2893455814 --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f5a8c09c420 con 0x7f5a70003720 2024-09-16T04:13:30.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.386+0000 7f5a7f7fe640 10 monclient(hunting): _renew_subs 2024-09-16T04:13:30.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.386+0000 7f5a91115640 1 --2- 172.21.15.42:0/2893455814 >> v2:172.21.15.154:3300/0 conn(0x7f5a70003720 0x7f5a70005b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:13:30.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.386+0000 7f5a91115640 10 monclient(hunting): get_auth_request con 0x7f5a70003720 auth_method 0 2024-09-16T04:13:30.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.386+0000 7f5a91115640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:13:30.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.386+0000 7f5a91115640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:30.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.386+0000 7f5a91115640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:13:30.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.386+0000 7f5a91115640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:13:30.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.386+0000 7f5a91115640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:13:30.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.386+0000 7f5a91115640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:13:30.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.386+0000 7f5a90914640 1 -- 172.21.15.42:0/2893455814 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3495285123 0 0) 0x7f5a8c06dd60 con 0x7f5a8c0a39b0 2024-09-16T04:13:30.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.386+0000 7f5a90914640 10 monclient(hunting): _negotiate 2024-09-16T04:13:30.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.386+0000 7f5a90914640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:30.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.386+0000 7f5a90914640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:13:30.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.386+0000 7f5a90914640 10 monclient(hunting): my global_id is 1034595 2024-09-16T04:13:30.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.386+0000 7f5a90914640 1 -- 172.21.15.42:0/2893455814 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5a78003530 con 0x7f5a8c0a39b0 2024-09-16T04:13:30.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.386+0000 7f5a91115640 10 monclient(hunting): handle_auth_done global_id 1074265 payload 274 2024-09-16T04:13:30.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.386+0000 7f5a91115640 1 -- 172.21.15.42:0/2893455814 >> v1:172.21.15.154:6789/0 conn(0x7f5a8c13c8c0 legacy=0x7f5a7000b1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:30.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.386+0000 7f5a91115640 1 -- 172.21.15.42:0/2893455814 >> v1:172.21.15.42:6789/0 conn(0x7f5a8c0a39b0 legacy=0x7f5a7000e940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:30.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.386+0000 7f5a90914640 1 -- 172.21.15.42:0/2893455814 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1087667898 0 0) 0x7f5a70006720 con 0x7f5a8c13c8c0 2024-09-16T04:13:30.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.386+0000 7f5a91115640 10 monclient: _finish_hunting 0 2024-09-16T04:13:30.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.386+0000 7f5a91115640 1 monclient: found mon.noname-c 2024-09-16T04:13:30.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.386+0000 7f5a91115640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:13:30.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.386+0000 7f5a91115640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.154:3300/0 2024-09-16T04:13:30.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.386+0000 7f5a91115640 1 -- 172.21.15.42:0/2893455814 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a70007950 con 0x7f5a70003720 2024-09-16T04:13:30.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.386+0000 7f5a90914640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T04:13:30.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.394+0000 7f5a91115640 1 --2- 172.21.15.42:0/2893455814 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5a70003720 0x7f5a70005b80 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f5a88000ee0 tx=0x7f5a8801dd80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=82582f69cdc3023d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:13:30.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.394+0000 7f5a90914640 1 -- 172.21.15.42:0/2893455814 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a880b3070 con 0x7f5a70003720 2024-09-16T04:13:30.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.394+0000 7f5a90914640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:13:30.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.394+0000 7f5a90914640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:13:30.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.394+0000 7f5a90914640 10 monclient: dump: 2024-09-16T04:13:30.399 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:13:30.399 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:13:30.399 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:30.399 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:30.399 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:30.399 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:13:30.399 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:13:30.399 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:13:30.399 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:13:30.399 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:13:30.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.394+0000 7f5a90914640 10 monclient: _finish_auth 0 2024-09-16T04:13:30.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.394+0000 7f5a90914640 10 monclient: _check_auth_tickets 2024-09-16T04:13:30.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.394+0000 7f5a90914640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:13:30.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.394+0000 7f5a93b7a640 5 monclient: authenticate success, global_id 1074265 2024-09-16T04:13:30.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.394+0000 7f5a90914640 1 -- 172.21.15.42:0/2893455814 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5a880a6b60 con 0x7f5a70003720 2024-09-16T04:13:30.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.394+0000 7f5a93b7a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:13:30.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.394+0000 7f5a90914640 10 monclient: handle_config config(0 keys) 2024-09-16T04:13:30.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.394+0000 7f5a93b7a640 10 monclient: get_monmap_and_config success 2024-09-16T04:13:30.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.394+0000 7f5a93b7a640 4 set_mon_vals no callback set 2024-09-16T04:13:30.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.394+0000 7f5a90914640 1 -- 172.21.15.42:0/2893455814 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a8800f040 con 0x7f5a70003720 2024-09-16T04:13:30.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.394+0000 7f5a90914640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:13:30.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.394+0000 7f5a90914640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:13:30.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.394+0000 7f5a90914640 10 monclient: dump: 2024-09-16T04:13:30.400 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:13:30.401 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:13:30.401 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:30.401 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:30.401 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:30.401 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:13:30.401 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:13:30.401 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:13:30.401 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:13:30.401 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:13:30.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.394+0000 7f5a93b7a640 10 monclient: shutdown 2024-09-16T04:13:30.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.394+0000 7f5a93b7a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:13:30.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.394+0000 7f5a93b7a640 1 -- 172.21.15.42:0/2893455814 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5a70003720 msgr2=0x7f5a70005b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:30.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.394+0000 7f5a93b7a640 1 --2- 172.21.15.42:0/2893455814 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5a70003720 0x7f5a70005b80 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f5a88000ee0 tx=0x7f5a8801dd80 comp rx=0 tx=0).stop 2024-09-16T04:13:30.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.394+0000 7f5a93b7a640 1 -- 172.21.15.42:0/2893455814 shutdown_connections 2024-09-16T04:13:30.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.394+0000 7f5a93b7a640 1 --2- 172.21.15.42:0/2893455814 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5a70003720 0x7f5a70005b80 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:13:30.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.394+0000 7f5a93b7a640 1 -- 172.21.15.42:0/2893455814 >> 172.21.15.42:0/2893455814 conn(0x7f5a8c0a9fb0 msgr2=0x7f5a8c0aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:13:30.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.394+0000 7f5a93b7a640 1 -- 172.21.15.42:0/2893455814 shutdown_connections 2024-09-16T04:13:30.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.394+0000 7f5a93b7a640 1 -- 172.21.15.42:0/2893455814 wait complete. 2024-09-16T04:13:30.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.398+0000 7f5a93b7a640 10 monclient: build_initial_monmap 2024-09-16T04:13:30.403 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.398+0000 7f5a93b7a640 10 monclient: monmap: 2024-09-16T04:13:30.403 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:13:30.403 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:13:30.403 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:13:30.403 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:13:30.403 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:13:30.404 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:13:30.404 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:13:30.404 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:13:30.404 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:13:30.404 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:13:30.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.398+0000 7f5a93b7a640 1 librados: starting msgr at 2024-09-16T04:13:30.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.398+0000 7f5a93b7a640 1 librados: starting objecter 2024-09-16T04:13:30.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.398+0000 7f5a93b7a640 1 Processor -- start 2024-09-16T04:13:30.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.398+0000 7f5a93b7a640 1 -- start start 2024-09-16T04:13:30.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.398+0000 7f5a93b7a640 1 librados: setting wanted keys 2024-09-16T04:13:30.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.398+0000 7f5a93b7a640 1 librados: calling monclient init 2024-09-16T04:13:30.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.398+0000 7f5a93b7a640 10 monclient: init 2024-09-16T04:13:30.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.398+0000 7f5a93b7a640 10 monclient: _reopen_session rank -1 2024-09-16T04:13:30.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.398+0000 7f5a93b7a640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:13:30.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.398+0000 7f5a93b7a640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5a70003720 0x7f5a8c1cfcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:13:30.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.398+0000 7f5a93b7a640 10 monclient(hunting): picked mon.noname-b con 0x7f5a70003720 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:13:30.409 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.398+0000 7f5a93b7a640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5a8c0a39b0 0x7f5a8c1d0230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:13:30.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.398+0000 7f5a93b7a640 10 monclient(hunting): picked mon.noname-a con 0x7f5a8c0a39b0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:13:30.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.398+0000 7f5a93b7a640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5a8c13c8c0 0x7f5a8c0a01c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:13:30.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.398+0000 7f5a93b7a640 10 monclient(hunting): picked mon.noname-c con 0x7f5a8c13c8c0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:13:30.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.398+0000 7f5a93b7a640 10 monclient(hunting): start opening mon connection 2024-09-16T04:13:30.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.398+0000 7f5a93b7a640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f5a8c137be0 con 0x7f5a8c0a39b0 2024-09-16T04:13:30.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.398+0000 7f5a93b7a640 10 monclient(hunting): start opening mon connection 2024-09-16T04:13:30.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.398+0000 7f5a93b7a640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f5a8c137a60 con 0x7f5a70003720 2024-09-16T04:13:30.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.398+0000 7f5a93b7a640 10 monclient(hunting): start opening mon connection 2024-09-16T04:13:30.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.398+0000 7f5a93b7a640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f5a8c137d60 con 0x7f5a8c13c8c0 2024-09-16T04:13:30.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.398+0000 7f5a93b7a640 10 monclient(hunting): _renew_subs 2024-09-16T04:13:30.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.398+0000 7f5a93b7a640 10 monclient(hunting): authenticate will time out at 2857.370414s 2024-09-16T04:13:30.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.398+0000 7f5a91115640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5a8c0a39b0 0x7f5a8c1d0230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:13:30.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.398+0000 7f5a92117640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5a8c13c8c0 0x7f5a8c0a01c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:13:30.411 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.398+0000 7f5a91115640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5a8c0a39b0 0x7f5a8c1d0230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:51976/0 (socket says 172.21.15.42:51976) 2024-09-16T04:13:30.411 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.398+0000 7f5a91115640 1 -- 172.21.15.42:0/2755014488 learned_addr learned my addr 172.21.15.42:0/2755014488 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:13:30.411 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.398+0000 7f5a91115640 10 monclient(hunting): get_auth_request con 0x7f5a8c0a39b0 auth_method 0 2024-09-16T04:13:30.411 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.398+0000 7f5a91115640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:13:30.411 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.398+0000 7f5a91115640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:30.411 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.398+0000 7f5a91115640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:13:30.411 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.398+0000 7f5a92117640 10 monclient(hunting): get_auth_request con 0x7f5a8c13c8c0 auth_method 0 2024-09-16T04:13:30.411 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.398+0000 7f5a92117640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:13:30.411 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.398+0000 7f5a92117640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:30.411 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.398+0000 7f5a92117640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:13:30.411 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.398+0000 7f5a92117640 1 -- 172.21.15.42:0/2755014488 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5a8c13c8c0 msgr2=0x7f5a8c0a01c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-16T04:13:30.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.398+0000 7f5a91115640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:13:30.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.398+0000 7f5a92117640 1 -- 172.21.15.42:0/2755014488 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5a8c13c8c0 msgr2=0x7f5a8c0a01c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:13:30.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.398+0000 7f5a91115640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:13:30.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.398+0000 7f5a92117640 1 --2- 172.21.15.42:0/2755014488 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5a8c13c8c0 0x7f5a8c0a01c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:13:30.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.398+0000 7f5a91115640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:13:30.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.398+0000 7f5a92117640 1 --2- 172.21.15.42:0/2755014488 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5a8c13c8c0 0x7f5a8c0a01c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:13:30.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.398+0000 7f5a91916640 1 --2- 172.21.15.42:0/2755014488 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5a70003720 0x7f5a8c1cfcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:13:30.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.398+0000 7f5a91916640 10 monclient(hunting): get_auth_request con 0x7f5a70003720 auth_method 0 2024-09-16T04:13:30.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.398+0000 7f5a91916640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:13:30.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.398+0000 7f5a91916640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:30.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.398+0000 7f5a91916640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:13:30.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.398+0000 7f5a91115640 10 monclient(hunting): handle_auth_done global_id 1034598 payload 824 2024-09-16T04:13:30.412 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.398+0000 7f5a91115640 1 -- 172.21.15.42:0/2755014488 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5a8c13c8c0 msgr2=0x7f5a8c0a01c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:13:30.413 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.398+0000 7f5a91115640 1 --2- 172.21.15.42:0/2755014488 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5a8c13c8c0 0x7f5a8c0a01c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:13:30.413 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.398+0000 7f5a91115640 1 -- 172.21.15.42:0/2755014488 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5a70003720 msgr2=0x7f5a8c1cfcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:30.413 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.398+0000 7f5a91115640 1 --2- 172.21.15.42:0/2755014488 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5a70003720 0x7f5a8c1cfcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:13:30.413 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.398+0000 7f5a91115640 10 monclient: _finish_hunting 0 2024-09-16T04:13:30.413 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.398+0000 7f5a91115640 1 monclient: found mon.noname-a 2024-09-16T04:13:30.413 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.398+0000 7f5a91115640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:13:30.413 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.398+0000 7f5a91115640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:13:30.413 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.398+0000 7f5a91115640 1 -- 172.21.15.42:0/2755014488 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a8c1206c0 con 0x7f5a8c0a39b0 2024-09-16T04:13:30.413 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.398+0000 7f5a91916640 1 --2- 172.21.15.42:0/2755014488 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5a70003720 0x7f5a8c1cfcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T04:13:30.413 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.398+0000 7f5a91115640 1 --2- 172.21.15.42:0/2755014488 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5a8c0a39b0 0x7f5a8c1d0230 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f5a88000ee0 tx=0x7f5a8801b9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:13:30.413 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.398+0000 7f5a7e7fc640 10 client.?.objecter ms_handle_connect 0x7f5a8c0a39b0 2024-09-16T04:13:30.413 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.398+0000 7f5a7e7fc640 10 client.?.objecter resend_mon_ops 2024-09-16T04:13:30.414 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.398+0000 7f5a7e7fc640 1 -- 172.21.15.42:0/2755014488 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a880be020 con 0x7f5a8c0a39b0 2024-09-16T04:13:30.414 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.398+0000 7f5a7e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:13:30.414 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.398+0000 7f5a7e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:13:30.414 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.398+0000 7f5a7e7fc640 10 monclient: dump: 2024-09-16T04:13:30.414 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:13:30.414 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:13:30.414 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:30.414 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:30.414 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:30.415 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:13:30.415 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:13:30.415 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:13:30.415 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:13:30.415 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:13:30.415 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.398+0000 7f5a7e7fc640 10 monclient: _finish_auth 0 2024-09-16T04:13:30.415 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.398+0000 7f5a7e7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:13:30.415 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.398+0000 7f5a7e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:13:30.415 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.398+0000 7f5a7e7fc640 1 -- 172.21.15.42:0/2755014488 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5a880aebd0 con 0x7f5a8c0a39b0 2024-09-16T04:13:30.415 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.398+0000 7f5a93b7a640 5 monclient: authenticate success, global_id 1034598 2024-09-16T04:13:30.415 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.398+0000 7f5a93b7a640 10 monclient: _renew_subs 2024-09-16T04:13:30.415 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.398+0000 7f5a93b7a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:13:30.415 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.398+0000 7f5a93b7a640 1 -- 172.21.15.42:0/2755014488 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a8c0a1e60 con 0x7f5a8c0a39b0 2024-09-16T04:13:30.416 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.398+0000 7f5a93b7a640 10 client.1034598.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:13:30.416 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.398+0000 7f5a93b7a640 10 monclient: _renew_subs 2024-09-16T04:13:30.416 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.398+0000 7f5a93b7a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:13:30.416 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.398+0000 7f5a93b7a640 1 -- 172.21.15.42:0/2755014488 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a8c0a2300 con 0x7f5a8c0a39b0 2024-09-16T04:13:30.416 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.398+0000 7f5a93b7a640 1 librados: init done 2024-09-16T04:13:30.416 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.398+0000 7f5a7e7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:13:30.416 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.398+0000 7f5a7e7fc640 1 -- 172.21.15.42:0/2755014488 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a880b3040 con 0x7f5a8c0a39b0 2024-09-16T04:13:30.416 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.398+0000 7f5a7e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:13:30.416 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.402+0000 7f5a7e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:13:30.416 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.398+0000 7f5a7ffff640 4 set_mon_vals no callback set 2024-09-16T04:13:30.416 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.402+0000 7f5a7e7fc640 10 monclient: dump: 2024-09-16T04:13:30.416 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:13:30.416 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:13:30.417 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:30.417 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:30.417 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:30.417 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:13:30.417 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:13:30.417 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:13:30.417 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:13:30.417 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:13:30.417 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.402+0000 7f5a93b7a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:13:30.417 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.402+0000 7f5a93b7a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:13:30.417 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.402+0000 7f5a93b7a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:13:30.417 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.402+0000 7f5a93b7a640 1 -- 172.21.15.42:0/2755014488 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5a74005090 con 0x7f5a8c0a39b0 2024-09-16T04:13:30.417 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.402+0000 7f5a7e7fc640 1 -- 172.21.15.42:0/2755014488 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f5a880c1020 con 0x7f5a8c0a39b0 2024-09-16T04:13:30.417 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.402+0000 7f5a7e7fc640 1 --2- 172.21.15.42:0/2755014488 >> v2:172.21.15.154:6812/254845342 conn(0x7f5a700386a0 0x7f5a7003ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:13:30.418 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.402+0000 7f5a7e7fc640 1 -- 172.21.15.42:0/2755014488 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(706..706 src has 681..706) ==== 5040+0+0 (secure 0 0 0) 0x7f5a880e4df0 con 0x7f5a8c0a39b0 2024-09-16T04:13:30.418 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.402+0000 7f5a7e7fc640 10 client.1034598.objecter ms_dispatch 0x7f5a8c000ba0 osd_map(706..706 src has 681..706) 2024-09-16T04:13:30.418 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.402+0000 7f5a7e7fc640 3 client.1034598.objecter handle_osd_map got epochs [706,706] > 0 2024-09-16T04:13:30.418 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.402+0000 7f5a7e7fc640 3 client.1034598.objecter handle_osd_map decoding full epoch 706 2024-09-16T04:13:30.418 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.402+0000 7f5a7e7fc640 20 client.1034598.objecter dump_active .. 0 homeless 2024-09-16T04:13:30.418 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.402+0000 7f5a91916640 1 --2- 172.21.15.42:0/2755014488 >> v2:172.21.15.154:6812/254845342 conn(0x7f5a700386a0 0x7f5a7003ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:13:30.418 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.402+0000 7f5a91916640 10 monclient: get_auth_request con 0x7f5a700386a0 auth_method 0 2024-09-16T04:13:30.418 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.402+0000 7f5a91916640 1 --2- 172.21.15.42:0/2755014488 >> v2:172.21.15.154:6812/254845342 conn(0x7f5a700386a0 0x7f5a7003ab60 secure :-1 s=READY pgs=3984 cs=0 l=1 rev1=1 crypto rx=0x7f5a80000c10 tx=0x7f5a80009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:13:30.418 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.402+0000 7f5a7e7fc640 10 client.1034598.objecter ms_handle_connect 0x7f5a700386a0 2024-09-16T04:13:30.419 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.406+0000 7f5a7e7fc640 1 -- 172.21.15.42:0/2755014488 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f5a880c8070 con 0x7f5a8c0a39b0 2024-09-16T04:13:30.419 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.406+0000 7f5a7e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:13:30.419 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.406+0000 7f5a7e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:13:30.643 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.638+0000 7f5a93b7a640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:13:30.644 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.638+0000 7f5a93b7a640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:13:30.644 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.638+0000 7f5a93b7a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:13:30.644 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.638+0000 7f5a93b7a640 1 -- 172.21.15.42:0/2755014488 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f5a74005380 con 0x7f5a8c0a39b0 2024-09-16T04:13:30.644 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.638+0000 7f5a7e7fc640 1 -- 172.21.15.42:0/2755014488 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7f5a880088b0 con 0x7f5a8c0a39b0 2024-09-16T04:13:30.644 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.638+0000 7f5a7e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:13:30.644 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.638+0000 7f5a7e7fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:13:30.645 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":286,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:13:30.646 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.642+0000 7f5a93b7a640 10 librados: watch_flush enter 2024-09-16T04:13:30.646 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.642+0000 7f5a93b7a640 10 librados: watch_flush exit 2024-09-16T04:13:30.647 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.642+0000 7f5a93b7a640 20 client.1034598.objecter shutdown clearing up homeless session... 2024-09-16T04:13:30.647 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.642+0000 7f5a93b7a640 10 client.1034598.objecter successfully canceled tick 2024-09-16T04:13:30.647 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.642+0000 7f5a93b7a640 1 -- 172.21.15.42:0/2755014488 >> v2:172.21.15.154:6812/254845342 conn(0x7f5a700386a0 msgr2=0x7f5a7003ab60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:30.647 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.642+0000 7f5a93b7a640 1 --2- 172.21.15.42:0/2755014488 >> v2:172.21.15.154:6812/254845342 conn(0x7f5a700386a0 0x7f5a7003ab60 secure :-1 s=READY pgs=3984 cs=0 l=1 rev1=1 crypto rx=0x7f5a80000c10 tx=0x7f5a80009040 comp rx=0 tx=0).stop 2024-09-16T04:13:30.647 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.642+0000 7f5a93b7a640 10 monclient: shutdown 2024-09-16T04:13:30.647 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.642+0000 7f5a93b7a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:13:30.647 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.642+0000 7f5a93b7a640 1 -- 172.21.15.42:0/2755014488 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5a8c0a39b0 msgr2=0x7f5a8c1d0230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:30.647 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.642+0000 7f5a93b7a640 1 --2- 172.21.15.42:0/2755014488 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5a8c0a39b0 0x7f5a8c1d0230 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f5a88000ee0 tx=0x7f5a8801b9e0 comp rx=0 tx=0).stop 2024-09-16T04:13:30.647 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.642+0000 7f5a93b7a640 1 -- 172.21.15.42:0/2755014488 shutdown_connections 2024-09-16T04:13:30.647 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.642+0000 7f5a93b7a640 1 --2- 172.21.15.42:0/2755014488 >> v2:172.21.15.154:6812/254845342 conn(0x7f5a700386a0 0x7f5a7003ab60 unknown :-1 s=CLOSED pgs=3984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:13:30.647 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.642+0000 7f5a93b7a640 1 --2- 172.21.15.42:0/2755014488 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5a8c13c8c0 0x7f5a8c0a01c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:13:30.647 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.642+0000 7f5a93b7a640 1 --2- 172.21.15.42:0/2755014488 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5a8c0a39b0 0x7f5a8c1d0230 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:13:30.647 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.642+0000 7f5a93b7a640 1 --2- 172.21.15.42:0/2755014488 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5a70003720 0x7f5a8c1cfcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:13:30.648 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.642+0000 7f5a93b7a640 1 -- 172.21.15.42:0/2755014488 >> 172.21.15.42:0/2755014488 conn(0x7f5a8c0a9fb0 msgr2=0x7f5a8c131980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:13:30.648 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.642+0000 7f5a93b7a640 1 -- 172.21.15.42:0/2755014488 shutdown_connections 2024-09-16T04:13:30.648 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.642+0000 7f5a93b7a640 1 -- 172.21.15.42:0/2755014488 wait complete. 2024-09-16T04:13:30.648 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.642+0000 7f5a93b7a640 1 librados: shutdown 2024-09-16T04:13:30.664 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T04:13:30.664 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T04:13:30.664 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T04:13:30.792 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.786+0000 7fa83b000640 10 monclient: get_monmap_and_config 2024-09-16T04:13:30.792 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.786+0000 7fa83b000640 10 monclient: build_initial_monmap 2024-09-16T04:13:30.792 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.786+0000 7fa83b000640 10 monclient: monmap: 2024-09-16T04:13:30.792 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:13:30.792 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:13:30.792 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:13:30.790886+0000 2024-09-16T04:13:30.792 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:13:30.790886+0000 2024-09-16T04:13:30.792 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:13:30.793 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:13:30.793 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:13:30.793 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:13:30.793 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:13:30.793 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:13:30.793 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:13:30.793 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:13:30.803 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.786+0000 7fa83b000640 1 Processor -- start 2024-09-16T04:13:30.803 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.786+0000 7fa83b000640 1 -- start start 2024-09-16T04:13:30.803 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.786+0000 7fa83b000640 10 monclient: init 2024-09-16T04:13:30.804 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.790+0000 7fa83b000640 10 monclient: _reopen_session rank -1 2024-09-16T04:13:30.804 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.790+0000 7fa83b000640 10 monclient: _add_conns ranks=[1,3,0,2,4] 2024-09-16T04:13:30.804 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.790+0000 7fa83b000640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa8340a1330 addr v1:172.21.15.154:6789/0 2024-09-16T04:13:30.804 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.790+0000 7fa83b000640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fa83406dac0 0x7fa83413a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:13:30.804 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.790+0000 7fa83b000640 10 monclient(hunting): picked mon.noname-a con 0x7fa83406dac0 addr v2:172.21.15.42:3300/0 2024-09-16T04:13:30.804 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.790+0000 7fa83b000640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa8340a1e20 addr v1:172.21.15.42:6789/0 2024-09-16T04:13:30.804 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.790+0000 7fa83b000640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa83409c090 con 0x7fa8340a1e20 2024-09-16T04:13:30.804 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.790+0000 7fa83b000640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa83409f610 con 0x7fa8340a1330 2024-09-16T04:13:30.804 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.790+0000 7fa83b000640 10 monclient(hunting): start opening mon connection 2024-09-16T04:13:30.804 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.790+0000 7fa83b000640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7fa83409f8f0 con 0x7fa83406dac0 2024-09-16T04:13:30.804 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.790+0000 7fa83b000640 10 monclient(hunting): _renew_subs 2024-09-16T04:13:30.804 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.790+0000 7fa83b000640 10 monclient(hunting): authenticate will time out at 2857.760364s 2024-09-16T04:13:30.804 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.790+0000 7fa833fff640 1 -- >> v2:172.21.15.42:3300/0 conn(0x7fa83406dac0 msgr2=0x7fa83413a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:13:30.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.790+0000 7fa833fff640 1 -- >> v2:172.21.15.42:3300/0 conn(0x7fa83406dac0 msgr2=0x7fa83413a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:13:30.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.790+0000 7fa833fff640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fa83406dac0 0x7fa83413a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner read peer banner failed r=-1 ((1) Operation not permitted) 2024-09-16T04:13:30.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.790+0000 7fa833fff640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fa83406dac0 0x7fa83413a340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:13:30.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.790+0000 7fa83959d640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fa8340a1e20 0x7fa83413c890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:35762/0 (socket says 172.21.15.42:35762) 2024-09-16T04:13:30.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.790+0000 7fa83959d640 1 -- 172.21.15.42:0/4013459837 learned_addr learned my addr 172.21.15.42:0/4013459837 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T04:13:30.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.790+0000 7fa8337fe640 1 -- 172.21.15.42:0/4013459837 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1592861126 0 0) 0x7fa83409c090 con 0x7fa8340a1e20 2024-09-16T04:13:30.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.790+0000 7fa8337fe640 10 monclient(hunting): _negotiate 2024-09-16T04:13:30.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.790+0000 7fa8337fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:30.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.790+0000 7fa8337fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:13:30.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.790+0000 7fa8337fe640 10 monclient(hunting): my global_id is 1034601 2024-09-16T04:13:30.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.790+0000 7fa8337fe640 1 -- 172.21.15.42:0/4013459837 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa820003530 con 0x7fa8340a1e20 2024-09-16T04:13:30.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.790+0000 7fa8337fe640 1 -- 172.21.15.42:0/4013459837 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1135971837 0 0) 0x7fa820003530 con 0x7fa8340a1e20 2024-09-16T04:13:30.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.790+0000 7fa8337fe640 1 -- 172.21.15.42:0/4013459837 >> v2:172.21.15.42:3300/0 conn(0x7fa83406dac0 msgr2=0x7fa83413a340 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:13:30.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.790+0000 7fa8337fe640 1 --2- 172.21.15.42:0/4013459837 >> v2:172.21.15.42:3300/0 conn(0x7fa83406dac0 0x7fa83413a340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:13:30.806 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.790+0000 7fa8337fe640 1 -- 172.21.15.42:0/4013459837 >> v1:172.21.15.154:6789/0 conn(0x7fa8340a1330 legacy=0x7fa8340a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:30.806 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.790+0000 7fa8337fe640 10 monclient: _finish_hunting 0 2024-09-16T04:13:30.806 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.790+0000 7fa8337fe640 1 monclient: found mon.noname-a-legacy 2024-09-16T04:13:30.806 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.790+0000 7fa8337fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:13:30.806 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.790+0000 7fa8337fe640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.42:6789/0 2024-09-16T04:13:30.806 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.790+0000 7fa8337fe640 1 -- 172.21.15.42:0/4013459837 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa83409fbd0 con 0x7fa8340a1e20 2024-09-16T04:13:30.806 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.790+0000 7fa8337fe640 10 monclient: _finish_auth 0 2024-09-16T04:13:30.806 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.790+0000 7fa8337fe640 10 monclient: _check_auth_tickets 2024-09-16T04:13:30.806 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.790+0000 7fa8337fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:13:30.806 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.790+0000 7fa8337fe640 1 -- 172.21.15.42:0/4013459837 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2177413734 0 0) 0x7fa824001680 con 0x7fa8340a1e20 2024-09-16T04:13:30.806 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.790+0000 7fa83b000640 5 monclient: authenticate success, global_id 1034601 2024-09-16T04:13:30.806 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.790+0000 7fa8337fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:13:30.806 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.790+0000 7fa8337fe640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T04:13:30.806 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.790+0000 7fa8337fe640 10 monclient: dump: 2024-09-16T04:13:30.807 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:13:30.807 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:13:30.807 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:30.807 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:30.807 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:30.807 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:13:30.807 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:13:30.807 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:13:30.807 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:13:30.807 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:13:30.807 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.790+0000 7fa8337fe640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] but i'm connected to v1:172.21.15.42:6789/0, reconnecting 2024-09-16T04:13:30.807 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.790+0000 7fa8337fe640 10 monclient: _reopen_session rank -1 2024-09-16T04:13:30.807 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.790+0000 7fa8337fe640 1 -- 172.21.15.42:0/4013459837 >> v1:172.21.15.42:6789/0 conn(0x7fa8340a1e20 legacy=0x7fa83413c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:30.807 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.790+0000 7fa8337fe640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:13:30.807 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.790+0000 7fa8337fe640 1 --2- 172.21.15.42:0/4013459837 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa8200039c0 0x7fa820003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:13:30.808 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.790+0000 7fa8337fe640 10 monclient(hunting): picked mon.b con 0x7fa8200039c0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:13:30.808 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.790+0000 7fa8337fe640 1 --2- 172.21.15.42:0/4013459837 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa820004810 0x7fa820004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:13:30.808 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.790+0000 7fa8337fe640 10 monclient(hunting): picked mon.c con 0x7fa820004810 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:13:30.808 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.790+0000 7fa8337fe640 1 --2- 172.21.15.42:0/4013459837 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa8200053d0 0x7fa82000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:13:30.808 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.790+0000 7fa8337fe640 10 monclient(hunting): picked mon.a con 0x7fa8200053d0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:13:30.808 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.790+0000 7fa8337fe640 10 monclient(hunting): start opening mon connection 2024-09-16T04:13:30.808 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.790+0000 7fa8337fe640 1 -- 172.21.15.42:0/4013459837 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fa83409f8f0 con 0x7fa8200053d0 2024-09-16T04:13:30.808 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.790+0000 7fa8337fe640 10 monclient(hunting): start opening mon connection 2024-09-16T04:13:30.808 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.790+0000 7fa8337fe640 1 -- 172.21.15.42:0/4013459837 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa82000c290 con 0x7fa8200039c0 2024-09-16T04:13:30.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.790+0000 7fa8337fe640 10 monclient(hunting): start opening mon connection 2024-09-16T04:13:30.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.790+0000 7fa8337fe640 1 -- 172.21.15.42:0/4013459837 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fa82000c490 con 0x7fa820004810 2024-09-16T04:13:30.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.790+0000 7fa8337fe640 10 monclient(hunting): _renew_subs 2024-09-16T04:13:30.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.790+0000 7fa8337fe640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:13:30.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.790+0000 7fa83b000640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-16T04:13:30.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.790+0000 7fa833fff640 1 --2- 172.21.15.42:0/4013459837 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa820004810 0x7fa820004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:13:30.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.790+0000 7fa83959d640 1 --2- 172.21.15.42:0/4013459837 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa8200053d0 0x7fa82000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:13:30.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.790+0000 7fa833fff640 10 monclient(hunting): get_auth_request con 0x7fa820004810 auth_method 0 2024-09-16T04:13:30.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.790+0000 7fa833fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:13:30.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.790+0000 7fa833fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:30.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.790+0000 7fa833fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:13:30.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.790+0000 7fa83959d640 10 monclient(hunting): get_auth_request con 0x7fa8200053d0 auth_method 0 2024-09-16T04:13:30.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.790+0000 7fa83959d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:13:30.809 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.790+0000 7fa83959d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:30.810 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.790+0000 7fa83959d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:13:30.810 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.790+0000 7fa838d9c640 1 --2- 172.21.15.42:0/4013459837 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa8200039c0 0x7fa820003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:13:30.810 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.790+0000 7fa838d9c640 10 monclient(hunting): get_auth_request con 0x7fa8200039c0 auth_method 0 2024-09-16T04:13:30.810 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.790+0000 7fa838d9c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:13:30.810 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.790+0000 7fa838d9c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:30.810 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.790+0000 7fa838d9c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:13:30.810 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.790+0000 7fa83959d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:13:30.810 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.790+0000 7fa83959d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:13:30.810 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.790+0000 7fa83959d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:13:30.810 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.790+0000 7fa833fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:13:30.810 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.790+0000 7fa833fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:13:30.810 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.790+0000 7fa833fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:13:30.810 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.790+0000 7fa838d9c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:13:30.810 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.790+0000 7fa838d9c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:13:30.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.790+0000 7fa838d9c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:13:30.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.790+0000 7fa83959d640 10 monclient(hunting): handle_auth_done global_id 1034601 payload 293 2024-09-16T04:13:30.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.790+0000 7fa83959d640 1 -- 172.21.15.42:0/4013459837 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa820004810 msgr2=0x7fa820004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:13:30.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.790+0000 7fa83959d640 1 --2- 172.21.15.42:0/4013459837 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa820004810 0x7fa820004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:13:30.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.790+0000 7fa83959d640 1 -- 172.21.15.42:0/4013459837 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa8200039c0 msgr2=0x7fa820003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:13:30.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.790+0000 7fa83959d640 1 --2- 172.21.15.42:0/4013459837 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa8200039c0 0x7fa820003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:13:30.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.790+0000 7fa83959d640 10 monclient: _finish_hunting 0 2024-09-16T04:13:30.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.790+0000 7fa83959d640 1 monclient: found mon.a 2024-09-16T04:13:30.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.790+0000 7fa83959d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:13:30.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.790+0000 7fa83959d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:13:30.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.790+0000 7fa83959d640 1 -- 172.21.15.42:0/4013459837 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa820003530 con 0x7fa8200053d0 2024-09-16T04:13:30.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.790+0000 7fa83959d640 10 monclient: _finish_auth 0 2024-09-16T04:13:30.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.790+0000 7fa83959d640 10 monclient: _check_auth_tickets 2024-09-16T04:13:30.811 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.790+0000 7fa83959d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:13:30.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.790+0000 7fa833fff640 1 --2- 172.21.15.42:0/4013459837 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa820004810 0x7fa820004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:13:30.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.790+0000 7fa83959d640 1 -- 172.21.15.42:0/4013459837 reap_dead start 2024-09-16T04:13:30.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.790+0000 7fa83959d640 1 --2- 172.21.15.42:0/4013459837 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa8200053d0 0x7fa82000b9d0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fa82400c240 tx=0x7fa82401f3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ef825720ee5e306 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:13:30.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.794+0000 7fa8337fe640 1 -- 172.21.15.42:0/4013459837 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa81c0094c0 con 0x7fa8200053d0 2024-09-16T04:13:30.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.794+0000 7fa8337fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:13:30.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.794+0000 7fa8337fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:13:30.812 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.794+0000 7fa8337fe640 10 monclient: dump: 2024-09-16T04:13:30.812 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:13:30.812 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:13:30.812 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:30.813 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:30.813 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:30.813 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:13:30.813 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:13:30.813 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:13:30.813 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:13:30.813 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:13:30.813 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.794+0000 7fa8337fe640 1 -- 172.21.15.42:0/4013459837 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa824005ae0 con 0x7fa8200053d0 2024-09-16T04:13:30.813 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.794+0000 7fa8337fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:13:30.813 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.794+0000 7fa83b000640 10 monclient: get_monmap_and_config success 2024-09-16T04:13:30.813 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.794+0000 7fa83b000640 4 set_mon_vals no callback set 2024-09-16T04:13:30.813 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.794+0000 7fa8337fe640 1 -- 172.21.15.42:0/4013459837 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa824007d20 con 0x7fa8200053d0 2024-09-16T04:13:30.813 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.794+0000 7fa8337fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:13:30.813 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.794+0000 7fa8337fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:13:30.814 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.794+0000 7fa8337fe640 10 monclient: dump: 2024-09-16T04:13:30.814 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:13:30.814 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:13:30.815 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:30.815 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:30.815 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:30.815 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:13:30.815 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:13:30.816 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:13:30.816 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:13:30.816 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:13:30.816 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.794+0000 7fa83b000640 10 monclient: shutdown 2024-09-16T04:13:30.816 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.794+0000 7fa83b000640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:13:30.816 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.794+0000 7fa83b000640 1 -- 172.21.15.42:0/4013459837 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa8200053d0 msgr2=0x7fa82000b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:30.816 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.794+0000 7fa83b000640 1 --2- 172.21.15.42:0/4013459837 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa8200053d0 0x7fa82000b9d0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fa82400c240 tx=0x7fa82401f3d0 comp rx=0 tx=0).stop 2024-09-16T04:13:30.816 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.794+0000 7fa83b000640 1 -- 172.21.15.42:0/4013459837 shutdown_connections 2024-09-16T04:13:30.816 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.794+0000 7fa83b000640 1 --2- 172.21.15.42:0/4013459837 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa8200053d0 0x7fa82000b9d0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:13:30.817 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.794+0000 7fa83b000640 1 -- 172.21.15.42:0/4013459837 >> 172.21.15.42:0/4013459837 conn(0x7fa8340a2420 msgr2=0x7fa8340a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:13:30.817 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.794+0000 7fa83b000640 1 -- 172.21.15.42:0/4013459837 shutdown_connections 2024-09-16T04:13:30.817 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.794+0000 7fa83b000640 1 -- 172.21.15.42:0/4013459837 wait complete. 2024-09-16T04:13:30.817 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.794+0000 7fa83b000640 10 monclient: build_initial_monmap 2024-09-16T04:13:30.817 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.794+0000 7fa83b000640 10 monclient: monmap: 2024-09-16T04:13:30.817 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:13:30.817 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:13:30.818 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:13:30.818 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:13:30.818 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:13:30.818 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:13:30.818 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:13:30.818 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:13:30.818 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:13:30.818 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:13:30.819 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.794+0000 7fa83b000640 1 librados: starting msgr at 2024-09-16T04:13:30.819 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.794+0000 7fa83b000640 1 librados: starting objecter 2024-09-16T04:13:30.819 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.794+0000 7fa83b000640 1 Processor -- start 2024-09-16T04:13:30.819 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.794+0000 7fa83b000640 1 -- start start 2024-09-16T04:13:30.819 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.794+0000 7fa83b000640 1 librados: setting wanted keys 2024-09-16T04:13:30.819 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.794+0000 7fa83b000640 1 librados: calling monclient init 2024-09-16T04:13:30.819 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.794+0000 7fa83b000640 10 monclient: init 2024-09-16T04:13:30.819 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.794+0000 7fa83b000640 10 monclient: _reopen_session rank -1 2024-09-16T04:13:30.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.794+0000 7fa83b000640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:13:30.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.794+0000 7fa83b000640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa8200053d0 0x7fa834121a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:13:30.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.794+0000 7fa83b000640 10 monclient(hunting): picked mon.noname-b con 0x7fa8200053d0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:13:30.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.794+0000 7fa83b000640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa834121fd0 0x7fa8341223f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:13:30.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.794+0000 7fa83b000640 10 monclient(hunting): picked mon.noname-c con 0x7fa834121fd0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:13:30.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.794+0000 7fa83b000640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa834122930 0x7fa8341d0150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:13:30.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.794+0000 7fa83b000640 10 monclient(hunting): picked mon.noname-a con 0x7fa834122930 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:13:30.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.794+0000 7fa83b000640 10 monclient(hunting): start opening mon connection 2024-09-16T04:13:30.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.794+0000 7fa83b000640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fa83409eef0 con 0x7fa834122930 2024-09-16T04:13:30.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.794+0000 7fa83b000640 10 monclient(hunting): start opening mon connection 2024-09-16T04:13:30.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.794+0000 7fa83b000640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa83409ed70 con 0x7fa8200053d0 2024-09-16T04:13:30.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.794+0000 7fa83b000640 10 monclient(hunting): start opening mon connection 2024-09-16T04:13:30.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.794+0000 7fa83b000640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fa8340a1710 con 0x7fa834121fd0 2024-09-16T04:13:30.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.794+0000 7fa83b000640 10 monclient(hunting): _renew_subs 2024-09-16T04:13:30.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.794+0000 7fa83b000640 10 monclient(hunting): authenticate will time out at 2857.766317s 2024-09-16T04:13:30.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.794+0000 7fa833fff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa834121fd0 0x7fa8341223f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:13:30.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.794+0000 7fa83959d640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa834122930 0x7fa8341d0150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:13:30.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.794+0000 7fa833fff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa834121fd0 0x7fa8341223f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:48938/0 (socket says 172.21.15.42:48938) 2024-09-16T04:13:30.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.794+0000 7fa833fff640 1 -- 172.21.15.42:0/495616148 learned_addr learned my addr 172.21.15.42:0/495616148 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:13:30.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.794+0000 7fa833fff640 10 monclient(hunting): get_auth_request con 0x7fa834121fd0 auth_method 0 2024-09-16T04:13:30.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.794+0000 7fa833fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:13:30.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.794+0000 7fa833fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:30.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.794+0000 7fa833fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:13:30.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.794+0000 7fa83959d640 10 monclient(hunting): get_auth_request con 0x7fa834122930 auth_method 0 2024-09-16T04:13:30.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.794+0000 7fa83959d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:13:30.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.794+0000 7fa83959d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:30.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.794+0000 7fa83959d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:13:30.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.794+0000 7fa83959d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:13:30.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.794+0000 7fa83959d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:13:30.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.794+0000 7fa833fff640 1 -- 172.21.15.42:0/495616148 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa834121fd0 msgr2=0x7fa8341223f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-16T04:13:30.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.794+0000 7fa833fff640 1 -- 172.21.15.42:0/495616148 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa834121fd0 msgr2=0x7fa8341223f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:13:30.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.794+0000 7fa83959d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:13:30.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.794+0000 7fa833fff640 1 --2- 172.21.15.42:0/495616148 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa834121fd0 0x7fa8341223f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:13:30.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.794+0000 7fa833fff640 1 --2- 172.21.15.42:0/495616148 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa834121fd0 0x7fa8341223f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:13:30.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.794+0000 7fa838d9c640 1 --2- 172.21.15.42:0/495616148 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa8200053d0 0x7fa834121a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:13:30.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.794+0000 7fa83959d640 10 monclient(hunting): handle_auth_done global_id 1034604 payload 824 2024-09-16T04:13:30.823 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.794+0000 7fa83959d640 1 -- 172.21.15.42:0/495616148 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa834121fd0 msgr2=0x7fa8341223f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:13:30.823 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.794+0000 7fa83959d640 1 --2- 172.21.15.42:0/495616148 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa834121fd0 0x7fa8341223f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:13:30.823 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.794+0000 7fa83959d640 1 -- 172.21.15.42:0/495616148 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa8200053d0 msgr2=0x7fa834121a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:30.823 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.794+0000 7fa83959d640 1 --2- 172.21.15.42:0/495616148 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa8200053d0 0x7fa834121a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:13:30.823 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.794+0000 7fa83959d640 10 monclient: _finish_hunting 0 2024-09-16T04:13:30.823 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.794+0000 7fa83959d640 1 monclient: found mon.noname-a 2024-09-16T04:13:30.823 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.794+0000 7fa83959d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:13:30.823 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.794+0000 7fa83959d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:13:30.823 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.794+0000 7fa83959d640 1 -- 172.21.15.42:0/495616148 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa834123ef0 con 0x7fa834122930 2024-09-16T04:13:30.824 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.794+0000 7fa83959d640 1 --2- 172.21.15.42:0/495616148 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa834122930 0x7fa8341d0150 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fa8340a22f0 tx=0x7fa820005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:13:30.824 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.794+0000 7fa8317fa640 10 client.?.objecter ms_handle_connect 0x7fa834122930 2024-09-16T04:13:30.824 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.794+0000 7fa8317fa640 10 client.?.objecter resend_mon_ops 2024-09-16T04:13:30.824 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.794+0000 7fa8317fa640 1 -- 172.21.15.42:0/495616148 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa824007d20 con 0x7fa834122930 2024-09-16T04:13:30.824 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.794+0000 7fa8317fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:13:30.824 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.794+0000 7fa8317fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:13:30.824 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.794+0000 7fa8317fa640 10 monclient: dump: 2024-09-16T04:13:30.824 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:13:30.824 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:13:30.824 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:30.824 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:30.824 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:30.824 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:13:30.825 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:13:30.825 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:13:30.825 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:13:30.825 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:13:30.825 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.794+0000 7fa8317fa640 10 monclient: _finish_auth 0 2024-09-16T04:13:30.825 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.794+0000 7fa8317fa640 10 monclient: _check_auth_tickets 2024-09-16T04:13:30.825 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.794+0000 7fa8317fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:13:30.825 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.794+0000 7fa8317fa640 1 -- 172.21.15.42:0/495616148 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa824007070 con 0x7fa834122930 2024-09-16T04:13:30.825 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.794+0000 7fa8317fa640 10 monclient: handle_config config(0 keys) 2024-09-16T04:13:30.825 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.794+0000 7fa8317fa640 1 -- 172.21.15.42:0/495616148 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa824007420 con 0x7fa834122930 2024-09-16T04:13:30.825 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.794+0000 7fa83b000640 5 monclient: authenticate success, global_id 1034604 2024-09-16T04:13:30.825 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.794+0000 7fa83b000640 10 monclient: _renew_subs 2024-09-16T04:13:30.825 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.794+0000 7fa83b000640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:13:30.825 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.794+0000 7fa83b000640 1 -- 172.21.15.42:0/495616148 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8341256f0 con 0x7fa834122930 2024-09-16T04:13:30.826 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.794+0000 7fa8317fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:13:30.826 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.794+0000 7fa8327fc640 4 set_mon_vals no callback set 2024-09-16T04:13:30.826 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.794+0000 7fa8317fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:13:30.826 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.794+0000 7fa8317fa640 10 monclient: dump: 2024-09-16T04:13:30.826 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:13:30.826 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:13:30.826 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:30.826 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:30.826 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:30.826 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:13:30.826 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:13:30.826 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:13:30.826 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:13:30.826 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:13:30.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.794+0000 7fa83b000640 10 client.1034604.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:13:30.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.794+0000 7fa83b000640 10 monclient: _renew_subs 2024-09-16T04:13:30.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.794+0000 7fa83b000640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:13:30.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.794+0000 7fa83b000640 1 -- 172.21.15.42:0/495616148 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa83406e3f0 con 0x7fa834122930 2024-09-16T04:13:30.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.794+0000 7fa83b000640 1 librados: init done 2024-09-16T04:13:30.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.794+0000 7fa83b000640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:13:30.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.794+0000 7fa83b000640 1 --2- 172.21.15.42:0/495616148 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa83402d7d0 0x7fa834124f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:13:30.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.794+0000 7fa83b000640 10 monclient(hunting): start opening mon connection 2024-09-16T04:13:30.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.794+0000 7fa83b000640 1 -- 172.21.15.42:0/495616148 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fa834134f50 con 0x7fa83402d7d0 2024-09-16T04:13:30.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.798+0000 7fa838d9c640 1 --2- 172.21.15.42:0/495616148 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa83402d7d0 0x7fa834124f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:13:30.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.798+0000 7fa838d9c640 10 monclient: get_auth_request con 0x7fa83402d7d0 auth_method 0 2024-09-16T04:13:30.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.798+0000 7fa838d9c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:13:30.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.798+0000 7fa838d9c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:30.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.798+0000 7fa838d9c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:13:30.828 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.798+0000 7fa8317fa640 1 -- 172.21.15.42:0/495616148 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fa8240075c0 con 0x7fa834122930 2024-09-16T04:13:30.828 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.798+0000 7fa838d9c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:13:30.828 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.798+0000 7fa838d9c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:13:30.828 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.798+0000 7fa838d9c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:13:30.828 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.798+0000 7fa8317fa640 1 --2- 172.21.15.42:0/495616148 >> v2:172.21.15.154:6812/254845342 conn(0x7fa814037810 0x7fa814039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:13:30.828 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.798+0000 7fa838d9c640 10 monclient(hunting): handle_auth_done global_id 1034607 payload 824 2024-09-16T04:13:30.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.798+0000 7fa8317fa640 1 -- 172.21.15.42:0/495616148 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(706..706 src has 681..706) ==== 5040+0+0 (secure 0 0 0) 0x7fa8240e5ad0 con 0x7fa834122930 2024-09-16T04:13:30.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.798+0000 7fa8317fa640 10 client.1034604.objecter ms_dispatch 0x7fa834000ba0 osd_map(706..706 src has 681..706) 2024-09-16T04:13:30.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.798+0000 7fa8317fa640 3 client.1034604.objecter handle_osd_map got epochs [706,706] > 0 2024-09-16T04:13:30.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.798+0000 7fa8317fa640 3 client.1034604.objecter handle_osd_map decoding full epoch 706 2024-09-16T04:13:30.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.798+0000 7fa838d9c640 1 -- 172.21.15.42:0/495616148 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa83402dbb0 con 0x7fa83402d7d0 2024-09-16T04:13:30.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.798+0000 7fa8317fa640 20 client.1034604.objecter dump_active .. 0 homeless 2024-09-16T04:13:30.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.798+0000 7fa8317fa640 10 client.1034604.objecter ms_handle_connect 0x7fa83402d7d0 2024-09-16T04:13:30.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.798+0000 7fa8317fa640 10 client.1034604.objecter resend_mon_ops 2024-09-16T04:13:30.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.798+0000 7fa838d9c640 1 --2- 172.21.15.42:0/495616148 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa83402d7d0 0x7fa834124f90 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fa82c002130 tx=0x7fa82c005420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:13:30.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.798+0000 7fa8317fa640 1 -- 172.21.15.42:0/495616148 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa82c018030 con 0x7fa83402d7d0 2024-09-16T04:13:30.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.798+0000 7fa8317fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:13:30.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.798+0000 7fa8317fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:13:30.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.798+0000 7fa8317fa640 10 monclient: dump: 2024-09-16T04:13:30.830 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:13:30.830 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:13:30.830 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:30.830 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:30.830 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:30.830 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:13:30.830 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:13:30.830 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:13:30.830 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:13:30.831 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:13:30.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.798+0000 7fa833fff640 1 --2- 172.21.15.42:0/495616148 >> v2:172.21.15.154:6812/254845342 conn(0x7fa814037810 0x7fa814039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:13:30.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.798+0000 7fa833fff640 10 monclient: get_auth_request con 0x7fa814037810 auth_method 0 2024-09-16T04:13:30.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.798+0000 7fa8317fa640 1 -- 172.21.15.42:0/495616148 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fa82c014610 con 0x7fa83402d7d0 2024-09-16T04:13:30.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.798+0000 7fa8317fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:13:30.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.798+0000 7fa8317fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:13:30.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.798+0000 7fa8317fa640 1 -- 172.21.15.42:0/495616148 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa83402d7d0 msgr2=0x7fa834124f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:30.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.798+0000 7fa8317fa640 1 --2- 172.21.15.42:0/495616148 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa83402d7d0 0x7fa834124f90 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fa82c002130 tx=0x7fa82c005420 comp rx=0 tx=0).stop 2024-09-16T04:13:30.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.798+0000 7fa8317fa640 1 -- 172.21.15.42:0/495616148 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa83402d7d0 msgr2=0x7fa834124f90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:13:30.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.798+0000 7fa8317fa640 1 --2- 172.21.15.42:0/495616148 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa83402d7d0 0x7fa834124f90 secure :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fa82c002130 tx=0x7fa82c005420 comp rx=0 tx=0).stop 2024-09-16T04:13:30.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.798+0000 7fa8317fa640 10 client.1034604.objecter ms_handle_connect 0x7fa814037810 2024-09-16T04:13:30.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.798+0000 7fa833fff640 1 --2- 172.21.15.42:0/495616148 >> v2:172.21.15.154:6812/254845342 conn(0x7fa814037810 0x7fa814039cd0 secure :-1 s=READY pgs=3985 cs=0 l=1 rev1=1 crypto rx=0x7fa834072900 tx=0x7fa81c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:13:30.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.802+0000 7fa83b000640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:13:30.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.802+0000 7fa83b000640 1 --2- 172.21.15.42:0/495616148 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa834062ea0 0x7fa8340632b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:13:30.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.802+0000 7fa83b000640 10 monclient(hunting): start opening mon connection 2024-09-16T04:13:30.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.802+0000 7fa83b000640 1 -- 172.21.15.42:0/495616148 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fa834123390 con 0x7fa834062ea0 2024-09-16T04:13:30.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.802+0000 7fa838d9c640 1 --2- 172.21.15.42:0/495616148 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa834062ea0 0x7fa8340632b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:13:30.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.802+0000 7fa838d9c640 10 monclient: get_auth_request con 0x7fa834062ea0 auth_method 0 2024-09-16T04:13:30.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.802+0000 7fa838d9c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:13:30.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.802+0000 7fa838d9c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:30.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.802+0000 7fa838d9c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:13:30.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.802+0000 7fa838d9c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:13:30.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.802+0000 7fa838d9c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:13:30.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.802+0000 7fa838d9c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:13:30.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.802+0000 7fa838d9c640 10 monclient(hunting): handle_auth_done global_id 1034610 payload 824 2024-09-16T04:13:30.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.802+0000 7fa838d9c640 1 -- 172.21.15.42:0/495616148 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fa8341254d0 con 0x7fa834062ea0 2024-09-16T04:13:30.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.802+0000 7fa838d9c640 1 --2- 172.21.15.42:0/495616148 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa834062ea0 0x7fa8340632b0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fa82c002130 tx=0x7fa82c0014b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:13:30.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.802+0000 7fa8317fa640 10 client.1034604.objecter ms_handle_connect 0x7fa834062ea0 2024-09-16T04:13:30.834 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.802+0000 7fa8317fa640 10 client.1034604.objecter resend_mon_ops 2024-09-16T04:13:30.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.802+0000 7fa8317fa640 1 -- 172.21.15.42:0/495616148 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa82c01ea10 con 0x7fa834062ea0 2024-09-16T04:13:30.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.802+0000 7fa8317fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:13:30.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.802+0000 7fa8317fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:13:30.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.802+0000 7fa8317fa640 10 monclient: dump: 2024-09-16T04:13:30.835 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:13:30.835 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:13:30.835 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:30.835 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:30.835 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:30.835 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:13:30.835 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:13:30.835 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:13:30.835 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:13:30.835 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:13:30.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.806+0000 7fa8317fa640 1 -- 172.21.15.42:0/495616148 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4715 (secure 0 0 0) 0x7fa82c01ea10 con 0x7fa834062ea0 2024-09-16T04:13:30.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.806+0000 7fa8317fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:13:30.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.806+0000 7fa8317fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:13:30.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.806+0000 7fa8317fa640 1 -- 172.21.15.42:0/495616148 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa834062ea0 msgr2=0x7fa8340632b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:30.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.806+0000 7fa8317fa640 1 --2- 172.21.15.42:0/495616148 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa834062ea0 0x7fa8340632b0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fa82c002130 tx=0x7fa82c0014b0 comp rx=0 tx=0).stop 2024-09-16T04:13:30.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.806+0000 7fa8317fa640 1 -- 172.21.15.42:0/495616148 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa834062ea0 msgr2=0x7fa8340632b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:13:30.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.806+0000 7fa8317fa640 1 --2- 172.21.15.42:0/495616148 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa834062ea0 0x7fa8340632b0 secure :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fa82c002130 tx=0x7fa82c0014b0 comp rx=0 tx=0).stop 2024-09-16T04:13:30.836 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T04:13:30.836 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:13:30.837 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:13:30.837 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "leader", 2024-09-16T04:13:30.837 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 286, 2024-09-16T04:13:30.837 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 290737, 2024-09-16T04:13:30.837 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T04:13:30.837 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T04:13:30.837 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T04:13:30.837 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T04:13:30.837 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:13:30.837 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 0, 2024-09-16T04:13:30.837 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:13:30.837 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T04:13:30.837 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T04:13:30.837 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:13:30.838 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:13:30.838 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:13:30.838 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:13:30.838 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:13:30.838 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:13:30.838 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:13:30.838 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:13:30.838 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:13:30.838 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:13:30.838 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:13:30.838 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:13:30.838 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:13:30.838 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T04:13:30.838 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:13:30.839 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:13:30.839 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:13:30.839 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:13:30.839 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:13:30.840 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:13:30.840 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:13:30.841 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:13:30.841 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:13:30.842 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:13:30.842 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:13:30.842 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:13:30.842 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:13:30.842 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T04:13:30.842 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T04:13:30.842 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T04:13:30.842 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T04:13:30.843 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T04:13:30.843 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T04:13:30.843 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:13:30.843 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:13:30.843 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T04:13:30.843 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T04:13:30.843 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T04:13:30.843 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T04:13:30.843 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T04:13:30.844 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T04:13:30.844 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T04:13:30.844 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:13:30.844 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T04:13:30.844 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:13:30.844 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:13:30.844 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:13:30.844 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:13:30.844 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:13:30.844 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:13:30.844 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:13:30.844 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:13:30.844 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:13:30.844 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:13:30.845 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:13:30.845 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:13:30.845 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T04:13:30.845 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:13:30.845 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T04:13:30.845 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:13:30.845 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:13:30.845 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:13:30.845 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:13:30.845 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:13:30.845 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:13:30.845 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:13:30.845 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T04:13:30.846 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:13:30.846 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:13:30.846 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:13:30.846 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:13:30.846 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T04:13:30.846 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:13:30.846 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:13:30.846 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:13:30.846 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:13:30.846 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T04:13:30.846 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T04:13:30.846 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:13:30.846 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:13:30.846 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:13:30.847 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:13:30.847 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:13:30.847 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:13:30.847 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:13:30.847 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:13:30.847 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:13:30.847 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:13:30.847 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:13:30.847 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T04:13:30.847 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:13:30.847 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:13:30.847 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:13:30.847 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:13:30.847 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T04:13:30.848 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:13:30.848 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:13:30.848 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:13:30.848 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:13:30.848 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T04:13:30.848 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T04:13:30.848 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:13:30.848 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:13:30.848 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:13:30.848 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:13:30.848 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:13:30.848 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:13:30.848 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:13:30.849 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:13:30.849 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:13:30.849 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:13:30.849 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:13:30.850 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T04:13:30.850 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:13:30.850 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:13:30.850 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:13:30.850 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:13:30.850 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T04:13:30.850 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:13:30.850 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:13:30.850 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:13:30.850 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:13:30.850 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T04:13:30.850 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T04:13:30.850 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:13:30.850 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:13:30.851 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:13:30.851 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:13:30.851 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:13:30.851 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:13:30.851 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T04:13:30.851 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T04:13:30.851 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:13:30.851 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:13:30.851 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:13:30.851 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:13:30.851 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:13:30.851 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:13:30.851 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T04:13:30.851 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:13:30.852 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:13:30.852 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:13:30.852 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 6 2024-09-16T04:13:30.852 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:13:30.852 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:13:30.852 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T04:13:30.852 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:13:30.852 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:13:30.852 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:13:30.852 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 6 2024-09-16T04:13:30.852 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:13:30.852 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:13:30.852 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:13:30.853 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T04:13:30.853 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T04:13:30.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.806+0000 7fa83b000640 10 librados: watch_flush enter 2024-09-16T04:13:30.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.806+0000 7fa83b000640 10 librados: watch_flush exit 2024-09-16T04:13:30.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.806+0000 7fa83b000640 20 client.1034604.objecter shutdown clearing up homeless session... 2024-09-16T04:13:30.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.806+0000 7fa83b000640 10 client.1034604.objecter successfully canceled tick 2024-09-16T04:13:30.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.806+0000 7fa83b000640 1 -- 172.21.15.42:0/495616148 >> v2:172.21.15.154:6812/254845342 conn(0x7fa814037810 msgr2=0x7fa814039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:30.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.806+0000 7fa83b000640 1 --2- 172.21.15.42:0/495616148 >> v2:172.21.15.154:6812/254845342 conn(0x7fa814037810 0x7fa814039cd0 secure :-1 s=READY pgs=3985 cs=0 l=1 rev1=1 crypto rx=0x7fa834072900 tx=0x7fa81c008040 comp rx=0 tx=0).stop 2024-09-16T04:13:30.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.806+0000 7fa83b000640 10 monclient: shutdown 2024-09-16T04:13:30.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.806+0000 7fa83b000640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:13:30.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.806+0000 7fa83b000640 1 -- 172.21.15.42:0/495616148 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa834122930 msgr2=0x7fa8341d0150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:30.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.806+0000 7fa83b000640 1 --2- 172.21.15.42:0/495616148 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa834122930 0x7fa8341d0150 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fa8340a22f0 tx=0x7fa820005840 comp rx=0 tx=0).stop 2024-09-16T04:13:30.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.806+0000 7fa83959d640 1 -- 172.21.15.42:0/495616148 reap_dead start 2024-09-16T04:13:30.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.806+0000 7fa83959d640 1 -- 172.21.15.42:0/495616148 reap_dead start 2024-09-16T04:13:30.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.806+0000 7fa83b000640 1 -- 172.21.15.42:0/495616148 shutdown_connections 2024-09-16T04:13:30.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.806+0000 7fa83b000640 1 -- 172.21.15.42:0/495616148 >> 172.21.15.42:0/495616148 conn(0x7fa8340a2420 msgr2=0x7fa8340718a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:13:30.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.806+0000 7fa83b000640 1 -- 172.21.15.42:0/495616148 shutdown_connections 2024-09-16T04:13:30.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.806+0000 7fa83b000640 1 -- 172.21.15.42:0/495616148 wait complete. 2024-09-16T04:13:30.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.806+0000 7fa83b000640 1 librados: shutdown 2024-09-16T04:13:30.856 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:13:30.856 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T04:13:30.857 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T04:13:30.978 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.974+0000 7fd20926c640 10 monclient: get_monmap_and_config 2024-09-16T04:13:30.978 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.974+0000 7fd20926c640 10 monclient: build_initial_monmap 2024-09-16T04:13:30.978 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.974+0000 7fd20926c640 10 monclient: monmap: 2024-09-16T04:13:30.979 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:13:30.979 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:13:30.979 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:13:30.976854+0000 2024-09-16T04:13:30.979 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:13:30.976854+0000 2024-09-16T04:13:30.979 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:13:30.979 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:13:30.980 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:13:30.980 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:13:30.980 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:13:30.980 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:13:30.980 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:13:30.980 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:13:30.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.974+0000 7fd20926c640 1 Processor -- start 2024-09-16T04:13:30.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.974+0000 7fd20926c640 1 -- start start 2024-09-16T04:13:30.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.974+0000 7fd20926c640 10 monclient: init 2024-09-16T04:13:30.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.974+0000 7fd20926c640 10 monclient: _reopen_session rank -1 2024-09-16T04:13:30.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.974+0000 7fd20926c640 10 monclient: _add_conns ranks=[0,1,2,4,3] 2024-09-16T04:13:30.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.974+0000 7fd20926c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd2040a1330 addr v1:172.21.15.42:6789/0 2024-09-16T04:13:30.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.974+0000 7fd20926c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd20406dac0 addr v1:172.21.15.154:6789/0 2024-09-16T04:13:30.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.974+0000 7fd20926c640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd2040a1e20 0x7fd20413da70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:13:30.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.974+0000 7fd20926c640 10 monclient(hunting): picked mon.noname-b con 0x7fd2040a1e20 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:13:30.992 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.974+0000 7fd20926c640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd20409c090 con 0x7fd2040a1330 2024-09-16T04:13:30.992 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.974+0000 7fd20926c640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd20409f610 con 0x7fd20406dac0 2024-09-16T04:13:30.992 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.974+0000 7fd20926c640 10 monclient(hunting): start opening mon connection 2024-09-16T04:13:30.992 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.974+0000 7fd20926c640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fd20409c890 con 0x7fd2040a1e20 2024-09-16T04:13:30.992 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.974+0000 7fd20926c640 10 monclient(hunting): _renew_subs 2024-09-16T04:13:30.992 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.974+0000 7fd20926c640 10 monclient(hunting): authenticate will time out at 2857.946322s 2024-09-16T04:13:30.992 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.974+0000 7fd2037fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd2040a1e20 0x7fd20413da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:13:30.992 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.974+0000 7fd2037fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd2040a1e20 0x7fd20413da70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:48954/0 (socket says 172.21.15.42:48954) 2024-09-16T04:13:30.992 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.974+0000 7fd2037fe640 1 -- 172.21.15.42:0/3274479431 learned_addr learned my addr 172.21.15.42:0/3274479431 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:13:30.992 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.974+0000 7fd2037fe640 10 monclient(hunting): get_auth_request con 0x7fd2040a1e20 auth_method 0 2024-09-16T04:13:30.992 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.974+0000 7fd2037fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:13:30.992 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.974+0000 7fd2037fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:30.992 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.974+0000 7fd2037fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:13:30.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.974+0000 7fd2037fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:13:30.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.974+0000 7fd2037fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:13:30.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.974+0000 7fd2037fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:13:30.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.974+0000 7fd2037fe640 10 monclient(hunting): handle_auth_done global_id 1094099 payload 274 2024-09-16T04:13:30.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.974+0000 7fd2037fe640 1 -- 172.21.15.42:0/3274479431 >> v1:172.21.15.154:6789/0 conn(0x7fd20406dac0 legacy=0x7fd20413a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:13:30.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.974+0000 7fd2037fe640 1 -- 172.21.15.42:0/3274479431 >> v1:172.21.15.42:6789/0 conn(0x7fd2040a1330 legacy=0x7fd2040a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:30.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.974+0000 7fd2037fe640 10 monclient: _finish_hunting 0 2024-09-16T04:13:30.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.974+0000 7fd2037fe640 1 monclient: found mon.noname-b 2024-09-16T04:13:30.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.974+0000 7fd201ffb640 1 -- 172.21.15.42:0/3274479431 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3464564879 0 0) 0x7fd20409c090 con 0x7fd2040a1330 2024-09-16T04:13:30.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.974+0000 7fd2037fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:13:30.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.974+0000 7fd2037fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T04:13:30.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.974+0000 7fd2037fe640 1 -- 172.21.15.42:0/3274479431 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd20409f8f0 con 0x7fd2040a1e20 2024-09-16T04:13:30.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.974+0000 7fd201ffb640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T04:13:30.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.978+0000 7fd2037fe640 1 --2- 172.21.15.42:0/3274479431 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd2040a1e20 0x7fd20413da70 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fd1fc0098a0 tx=0x7fd1fc01c460 comp rx=0 tx=0).ready entity=mon.2 client_cookie=835c0aee923586ba server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:13:30.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.978+0000 7fd201ffb640 1 -- 172.21.15.42:0/3274479431 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1fc0ab680 con 0x7fd2040a1e20 2024-09-16T04:13:30.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.978+0000 7fd201ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:13:30.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.978+0000 7fd201ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:13:30.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.978+0000 7fd201ffb640 10 monclient: dump: 2024-09-16T04:13:30.994 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:13:30.994 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:13:30.994 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:30.994 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:30.994 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:30.994 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:13:30.994 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:13:30.994 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:13:30.994 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:13:30.994 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:13:30.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.978+0000 7fd201ffb640 10 monclient: _finish_auth 0 2024-09-16T04:13:30.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.978+0000 7fd201ffb640 10 monclient: _check_auth_tickets 2024-09-16T04:13:30.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.978+0000 7fd201ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:13:30.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.978+0000 7fd201ffb640 1 -- 172.21.15.42:0/3274479431 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd1fc0b0070 con 0x7fd2040a1e20 2024-09-16T04:13:30.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.978+0000 7fd201ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:13:30.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.978+0000 7fd201ffb640 1 -- 172.21.15.42:0/3274479431 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1fc0aba40 con 0x7fd2040a1e20 2024-09-16T04:13:30.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.978+0000 7fd20926c640 5 monclient: authenticate success, global_id 1094099 2024-09-16T04:13:30.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.978+0000 7fd20926c640 10 monclient: get_monmap_and_config success 2024-09-16T04:13:30.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.978+0000 7fd20926c640 4 set_mon_vals no callback set 2024-09-16T04:13:30.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.978+0000 7fd201ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:13:30.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.978+0000 7fd201ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:13:30.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.978+0000 7fd201ffb640 10 monclient: dump: 2024-09-16T04:13:30.995 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:13:30.995 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:13:30.996 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:30.996 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:30.996 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:30.996 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:13:30.996 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:13:30.996 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:13:30.996 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:13:30.996 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:13:30.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.978+0000 7fd20926c640 10 monclient: shutdown 2024-09-16T04:13:30.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.978+0000 7fd20926c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:13:30.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.978+0000 7fd20926c640 1 -- 172.21.15.42:0/3274479431 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd2040a1e20 msgr2=0x7fd20413da70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:30.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.978+0000 7fd20926c640 1 --2- 172.21.15.42:0/3274479431 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd2040a1e20 0x7fd20413da70 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fd1fc0098a0 tx=0x7fd1fc01c460 comp rx=0 tx=0).stop 2024-09-16T04:13:30.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.978+0000 7fd20926c640 1 -- 172.21.15.42:0/3274479431 shutdown_connections 2024-09-16T04:13:30.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.978+0000 7fd20926c640 1 --2- 172.21.15.42:0/3274479431 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd2040a1e20 0x7fd20413da70 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:13:30.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.978+0000 7fd20926c640 1 -- 172.21.15.42:0/3274479431 >> 172.21.15.42:0/3274479431 conn(0x7fd2040a2420 msgr2=0x7fd2040a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:13:30.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.978+0000 7fd20926c640 1 -- 172.21.15.42:0/3274479431 shutdown_connections 2024-09-16T04:13:30.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.978+0000 7fd20926c640 1 -- 172.21.15.42:0/3274479431 wait complete. 2024-09-16T04:13:30.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.978+0000 7fd20926c640 10 monclient: build_initial_monmap 2024-09-16T04:13:30.997 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.978+0000 7fd20926c640 10 monclient: monmap: 2024-09-16T04:13:30.997 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:13:30.997 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:13:30.997 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:13:30.997 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:13:30.997 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:13:30.997 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:13:30.997 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:13:30.997 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:13:30.998 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:13:30.998 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:13:30.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.978+0000 7fd20926c640 1 librados: starting msgr at 2024-09-16T04:13:30.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.978+0000 7fd20926c640 1 librados: starting objecter 2024-09-16T04:13:30.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.978+0000 7fd20926c640 1 Processor -- start 2024-09-16T04:13:30.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.978+0000 7fd20926c640 1 -- start start 2024-09-16T04:13:30.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.978+0000 7fd20926c640 1 librados: setting wanted keys 2024-09-16T04:13:30.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.978+0000 7fd20926c640 1 librados: calling monclient init 2024-09-16T04:13:30.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.978+0000 7fd20926c640 10 monclient: init 2024-09-16T04:13:30.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.978+0000 7fd20926c640 10 monclient: _reopen_session rank -1 2024-09-16T04:13:30.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.978+0000 7fd20926c640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:13:30.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.978+0000 7fd20926c640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd2040a1330 0x7fd204134d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:13:30.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.978+0000 7fd20926c640 10 monclient(hunting): picked mon.noname-c con 0x7fd2040a1330 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:13:30.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.978+0000 7fd20926c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd2040a1e20 0x7fd2041352a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:13:30.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.978+0000 7fd20926c640 10 monclient(hunting): picked mon.noname-b con 0x7fd2040a1e20 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:13:30.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.978+0000 7fd20926c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd20411cb10 0x7fd20411cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:13:30.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.978+0000 7fd20926c640 10 monclient(hunting): picked mon.noname-a con 0x7fd20411cb10 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:13:30.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.978+0000 7fd20926c640 10 monclient(hunting): start opening mon connection 2024-09-16T04:13:30.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.978+0000 7fd20926c640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fd204140950 con 0x7fd20411cb10 2024-09-16T04:13:30.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.978+0000 7fd20926c640 10 monclient(hunting): start opening mon connection 2024-09-16T04:13:30.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.978+0000 7fd20926c640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd2041407d0 con 0x7fd2040a1e20 2024-09-16T04:13:30.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.978+0000 7fd20926c640 10 monclient(hunting): start opening mon connection 2024-09-16T04:13:30.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.978+0000 7fd20926c640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fd204140ad0 con 0x7fd2040a1330 2024-09-16T04:13:30.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.978+0000 7fd20926c640 10 monclient(hunting): _renew_subs 2024-09-16T04:13:30.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.978+0000 7fd20926c640 10 monclient(hunting): authenticate will time out at 2857.951383s 2024-09-16T04:13:30.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.978+0000 7fd202ffd640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd2040a1330 0x7fd204134d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:13:30.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.978+0000 7fd2037fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd20411cb10 0x7fd20411cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:13:31.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.978+0000 7fd202ffd640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd2040a1330 0x7fd204134d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:48966/0 (socket says 172.21.15.42:48966) 2024-09-16T04:13:31.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.978+0000 7fd202ffd640 1 -- 172.21.15.42:0/365298843 learned_addr learned my addr 172.21.15.42:0/365298843 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:13:31.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.978+0000 7fd2037fe640 10 monclient(hunting): get_auth_request con 0x7fd20411cb10 auth_method 0 2024-09-16T04:13:31.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.978+0000 7fd2037fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:13:31.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.978+0000 7fd2037fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:31.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.978+0000 7fd2037fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:13:31.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.978+0000 7fd202ffd640 10 monclient(hunting): get_auth_request con 0x7fd2040a1330 auth_method 0 2024-09-16T04:13:31.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.978+0000 7fd202ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:13:31.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.978+0000 7fd202ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:31.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.978+0000 7fd202ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:13:31.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.978+0000 7fd2037fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:13:31.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.978+0000 7fd2037fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:13:31.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.978+0000 7fd2037fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:13:31.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.978+0000 7fd202ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:13:31.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.978+0000 7fd202ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:13:31.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.978+0000 7fd202ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:13:31.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.978+0000 7fd2037fe640 10 monclient(hunting): handle_auth_done global_id 1034616 payload 824 2024-09-16T04:13:31.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.978+0000 7fd2027fc640 1 --2- 172.21.15.42:0/365298843 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd2040a1e20 0x7fd2041352a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:13:31.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.978+0000 7fd2037fe640 1 -- 172.21.15.42:0/365298843 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd2040a1330 msgr2=0x7fd204134d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:31.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.978+0000 7fd2037fe640 1 --2- 172.21.15.42:0/365298843 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd2040a1330 0x7fd204134d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:13:31.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.978+0000 7fd2037fe640 1 -- 172.21.15.42:0/365298843 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd2040a1e20 msgr2=0x7fd2041352a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:31.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.978+0000 7fd2037fe640 1 --2- 172.21.15.42:0/365298843 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd2040a1e20 0x7fd2041352a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:13:31.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.978+0000 7fd2037fe640 10 monclient: _finish_hunting 0 2024-09-16T04:13:31.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.978+0000 7fd2037fe640 1 monclient: found mon.noname-a 2024-09-16T04:13:31.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.978+0000 7fd2037fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:13:31.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.978+0000 7fd2037fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:13:31.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.978+0000 7fd2037fe640 1 -- 172.21.15.42:0/365298843 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd204120200 con 0x7fd20411cb10 2024-09-16T04:13:31.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.978+0000 7fd202ffd640 1 --2- 172.21.15.42:0/365298843 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd2040a1330 0x7fd204134d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:13:31.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.982+0000 7fd2037fe640 1 --2- 172.21.15.42:0/365298843 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd20411cb10 0x7fd20411cfa0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fd1fc0ac000 tx=0x7fd1fc019200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:13:31.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.982+0000 7fd1fb7fe640 10 client.?.objecter ms_handle_connect 0x7fd20411cb10 2024-09-16T04:13:31.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.982+0000 7fd1fb7fe640 10 client.?.objecter resend_mon_ops 2024-09-16T04:13:31.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.982+0000 7fd1fb7fe640 1 -- 172.21.15.42:0/365298843 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1fc0b5810 con 0x7fd20411cb10 2024-09-16T04:13:31.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.982+0000 7fd1fb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:13:31.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.982+0000 7fd1fb7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:13:31.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.982+0000 7fd1fb7fe640 10 monclient: dump: 2024-09-16T04:13:31.002 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:13:31.002 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:13:31.003 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:31.003 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:31.003 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:31.003 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:13:31.003 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:13:31.003 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:13:31.003 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:13:31.003 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:13:31.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.982+0000 7fd1fb7fe640 10 monclient: _finish_auth 0 2024-09-16T04:13:31.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.982+0000 7fd1fb7fe640 10 monclient: _check_auth_tickets 2024-09-16T04:13:31.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.982+0000 7fd1fb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:13:31.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.982+0000 7fd20926c640 5 monclient: authenticate success, global_id 1034616 2024-09-16T04:13:31.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.982+0000 7fd1fb7fe640 1 -- 172.21.15.42:0/365298843 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd1fc0a7070 con 0x7fd20411cb10 2024-09-16T04:13:31.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.982+0000 7fd1fb7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:13:31.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.982+0000 7fd20926c640 10 monclient: _renew_subs 2024-09-16T04:13:31.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.982+0000 7fd1fb7fe640 1 -- 172.21.15.42:0/365298843 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1fc0b5bb0 con 0x7fd20411cb10 2024-09-16T04:13:31.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.982+0000 7fd20926c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:13:31.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.982+0000 7fd20926c640 1 -- 172.21.15.42:0/365298843 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2041207c0 con 0x7fd20411cb10 2024-09-16T04:13:31.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.982+0000 7fd1fb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:13:31.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.982+0000 7fd1fb7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:13:31.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.982+0000 7fd1fb7fe640 10 monclient: dump: 2024-09-16T04:13:31.004 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:13:31.004 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:13:31.004 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:31.004 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:31.004 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:31.004 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:13:31.005 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:13:31.005 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:13:31.005 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:13:31.005 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:13:31.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.982+0000 7fd2017fa640 4 set_mon_vals no callback set 2024-09-16T04:13:31.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.982+0000 7fd20926c640 10 client.1034616.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:13:31.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.982+0000 7fd20926c640 10 monclient: _renew_subs 2024-09-16T04:13:31.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.982+0000 7fd20926c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:13:31.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.982+0000 7fd20926c640 1 -- 172.21.15.42:0/365298843 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd20411f310 con 0x7fd20411cb10 2024-09-16T04:13:31.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.982+0000 7fd20926c640 1 librados: init done 2024-09-16T04:13:31.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.982+0000 7fd20926c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:13:31.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.982+0000 7fd20926c640 1 --2- 172.21.15.42:0/365298843 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd20406dac0 0x7fd20411d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:13:31.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.982+0000 7fd20926c640 10 monclient(hunting): start opening mon connection 2024-09-16T04:13:31.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.982+0000 7fd20926c640 1 -- 172.21.15.42:0/365298843 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fd2040a1710 con 0x7fd20406dac0 2024-09-16T04:13:31.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.982+0000 7fd1fb7fe640 1 -- 172.21.15.42:0/365298843 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fd1fc0bd020 con 0x7fd20411cb10 2024-09-16T04:13:31.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.982+0000 7fd202ffd640 1 --2- 172.21.15.42:0/365298843 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd20406dac0 0x7fd20411d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:13:31.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.982+0000 7fd202ffd640 10 monclient: get_auth_request con 0x7fd20406dac0 auth_method 0 2024-09-16T04:13:31.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.982+0000 7fd202ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:13:31.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.982+0000 7fd202ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:31.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.982+0000 7fd202ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:13:31.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.982+0000 7fd202ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:13:31.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.982+0000 7fd202ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:13:31.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.982+0000 7fd202ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:13:31.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.982+0000 7fd1fb7fe640 1 --2- 172.21.15.42:0/365298843 >> v2:172.21.15.154:6812/254845342 conn(0x7fd1e4037810 0x7fd1e4039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:13:31.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.982+0000 7fd202ffd640 10 monclient(hunting): handle_auth_done global_id 1094105 payload 824 2024-09-16T04:13:31.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.982+0000 7fd202ffd640 1 -- 172.21.15.42:0/365298843 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd20411e7b0 con 0x7fd20406dac0 2024-09-16T04:13:31.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.982+0000 7fd1fb7fe640 1 -- 172.21.15.42:0/365298843 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(706..706 src has 681..706) ==== 5040+0+0 (secure 0 0 0) 0x7fd1fc0e3590 con 0x7fd20411cb10 2024-09-16T04:13:31.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.982+0000 7fd1fb7fe640 10 client.1034616.objecter ms_dispatch 0x7fd204000ba0 osd_map(706..706 src has 681..706) 2024-09-16T04:13:31.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.982+0000 7fd1fb7fe640 3 client.1034616.objecter handle_osd_map got epochs [706,706] > 0 2024-09-16T04:13:31.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.982+0000 7fd1fb7fe640 3 client.1034616.objecter handle_osd_map decoding full epoch 706 2024-09-16T04:13:31.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.982+0000 7fd1fb7fe640 20 client.1034616.objecter dump_active .. 0 homeless 2024-09-16T04:13:31.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.982+0000 7fd1fb7fe640 10 client.1034616.objecter ms_handle_connect 0x7fd20406dac0 2024-09-16T04:13:31.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.982+0000 7fd1fb7fe640 10 client.1034616.objecter resend_mon_ops 2024-09-16T04:13:31.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.982+0000 7fd202ffd640 1 --2- 172.21.15.42:0/365298843 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd20406dac0 0x7fd20411d4e0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fd1f4007fd0 tx=0x7fd1f400d520 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:13:31.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.982+0000 7fd2027fc640 1 --2- 172.21.15.42:0/365298843 >> v2:172.21.15.154:6812/254845342 conn(0x7fd1e4037810 0x7fd1e4039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:13:31.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.982+0000 7fd2027fc640 10 monclient: get_auth_request con 0x7fd1e4037810 auth_method 0 2024-09-16T04:13:31.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.982+0000 7fd1fb7fe640 1 -- 172.21.15.42:0/365298843 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1f4003980 con 0x7fd20406dac0 2024-09-16T04:13:31.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.982+0000 7fd1fb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:13:31.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.982+0000 7fd1fb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:13:31.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.982+0000 7fd1fb7fe640 10 monclient: dump: 2024-09-16T04:13:31.007 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:13:31.008 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:13:31.008 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:31.008 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:31.008 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:31.008 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:13:31.008 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:13:31.008 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:13:31.008 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:13:31.008 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:13:31.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.982+0000 7fd1fb7fe640 1 -- 172.21.15.42:0/365298843 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fd1f4018070 con 0x7fd20406dac0 2024-09-16T04:13:31.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.982+0000 7fd1fb7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:13:31.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.982+0000 7fd1fb7fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:13:31.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.982+0000 7fd1fb7fe640 1 -- 172.21.15.42:0/365298843 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd20406dac0 msgr2=0x7fd20411d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:31.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.982+0000 7fd1fb7fe640 1 --2- 172.21.15.42:0/365298843 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd20406dac0 0x7fd20411d4e0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fd1f4007fd0 tx=0x7fd1f400d520 comp rx=0 tx=0).stop 2024-09-16T04:13:31.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.982+0000 7fd1fb7fe640 1 -- 172.21.15.42:0/365298843 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd20406dac0 msgr2=0x7fd20411d4e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:13:31.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.982+0000 7fd1fb7fe640 1 --2- 172.21.15.42:0/365298843 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd20406dac0 0x7fd20411d4e0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:13:31.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.982+0000 7fd1fb7fe640 10 client.1034616.objecter ms_handle_connect 0x7fd1e4037810 2024-09-16T04:13:31.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.982+0000 7fd2027fc640 1 --2- 172.21.15.42:0/365298843 >> v2:172.21.15.154:6812/254845342 conn(0x7fd1e4037810 0x7fd1e4039cd0 secure :-1 s=READY pgs=3986 cs=0 l=1 rev1=1 crypto rx=0x7fd20411f030 tx=0x7fd1ec009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:13:31.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.990+0000 7fd20926c640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:13:31.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.990+0000 7fd20926c640 1 --2- 172.21.15.42:0/365298843 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd20411da20 0x7fd20413a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:13:31.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.990+0000 7fd20926c640 10 monclient(hunting): start opening mon connection 2024-09-16T04:13:31.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.990+0000 7fd20926c640 1 -- 172.21.15.42:0/365298843 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fd204069040 con 0x7fd20411da20 2024-09-16T04:13:31.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.990+0000 7fd202ffd640 1 --2- 172.21.15.42:0/365298843 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd20411da20 0x7fd20413a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:13:31.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.990+0000 7fd202ffd640 10 monclient: get_auth_request con 0x7fd20411da20 auth_method 0 2024-09-16T04:13:31.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.990+0000 7fd202ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:13:31.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.990+0000 7fd202ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:31.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.990+0000 7fd202ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:13:31.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.990+0000 7fd202ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:13:31.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.990+0000 7fd202ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:13:31.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.990+0000 7fd202ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:13:31.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.990+0000 7fd202ffd640 1 -- 172.21.15.42:0/365298843 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd20411da20 msgr2=0x7fd20413a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-16T04:13:31.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.990+0000 7fd202ffd640 1 -- 172.21.15.42:0/365298843 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd20411da20 msgr2=0x7fd20413a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:13:31.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.990+0000 7fd202ffd640 1 --2- 172.21.15.42:0/365298843 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd20411da20 0x7fd20413a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:13:31.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:30.990+0000 7fd202ffd640 1 --2- 172.21.15.42:0/365298843 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd20411da20 0x7fd20413a730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:13:31.198 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.194+0000 7fd202ffd640 1 --2- 172.21.15.42:0/365298843 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd20411da20 0x7fd20413a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:13:31.198 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.194+0000 7fd202ffd640 10 monclient: get_auth_request con 0x7fd20411da20 auth_method 0 2024-09-16T04:13:31.198 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.194+0000 7fd202ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:13:31.198 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.194+0000 7fd202ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:31.198 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.194+0000 7fd202ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:13:31.198 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.194+0000 7fd202ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:13:31.198 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.194+0000 7fd202ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:13:31.198 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.194+0000 7fd202ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:13:31.198 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.194+0000 7fd202ffd640 10 monclient(hunting): handle_auth_done global_id 1094111 payload 824 2024-09-16T04:13:31.199 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.194+0000 7fd202ffd640 1 -- 172.21.15.42:0/365298843 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fd20411de00 con 0x7fd20411da20 2024-09-16T04:13:31.199 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.194+0000 7fd202ffd640 1 --2- 172.21.15.42:0/365298843 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd20411da20 0x7fd20413a730 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fd1f4007fd0 tx=0x7fd1f4001410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:13:31.199 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.194+0000 7fd1fb7fe640 10 client.1034616.objecter ms_handle_connect 0x7fd20411da20 2024-09-16T04:13:31.199 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.194+0000 7fd1fb7fe640 10 client.1034616.objecter resend_mon_ops 2024-09-16T04:13:31.199 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.194+0000 7fd1fb7fe640 1 -- 172.21.15.42:0/365298843 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1f40054e0 con 0x7fd20411da20 2024-09-16T04:13:31.199 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.194+0000 7fd1fb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:13:31.200 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.194+0000 7fd1fb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:13:31.200 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.194+0000 7fd1fb7fe640 10 monclient: dump: 2024-09-16T04:13:31.200 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:13:31.200 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:13:31.200 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:31.200 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:31.200 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:31.200 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:13:31.200 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:13:31.200 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:13:31.200 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:13:31.201 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:13:31.201 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.194+0000 7fd1fb7fe640 1 -- 172.21.15.42:0/365298843 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4545 (secure 0 0 0) 0x7fd1f40054e0 con 0x7fd20411da20 2024-09-16T04:13:31.201 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.194+0000 7fd1fb7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:13:31.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.194+0000 7fd1fb7fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:13:31.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.194+0000 7fd1fb7fe640 1 -- 172.21.15.42:0/365298843 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd20411da20 msgr2=0x7fd20413a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:31.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.194+0000 7fd1fb7fe640 1 --2- 172.21.15.42:0/365298843 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd20411da20 0x7fd20413a730 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fd1f4007fd0 tx=0x7fd1f4001410 comp rx=0 tx=0).stop 2024-09-16T04:13:31.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.194+0000 7fd1fb7fe640 1 -- 172.21.15.42:0/365298843 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd20411da20 msgr2=0x7fd20413a730 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:13:31.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.194+0000 7fd1fb7fe640 1 --2- 172.21.15.42:0/365298843 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd20411da20 0x7fd20413a730 secure :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fd1f4007fd0 tx=0x7fd1f4001410 comp rx=0 tx=0).stop 2024-09-16T04:13:31.202 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T04:13:31.202 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:13:31.202 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:13:31.202 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T04:13:31.202 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 286, 2024-09-16T04:13:31.203 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 10235, 2024-09-16T04:13:31.203 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T04:13:31.203 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T04:13:31.203 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T04:13:31.203 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T04:13:31.203 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:13:31.203 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 1, 2024-09-16T04:13:31.203 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:13:31.203 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T04:13:31.203 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T04:13:31.203 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:13:31.203 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:13:31.204 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:13:31.204 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:13:31.204 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:13:31.204 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:13:31.204 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:13:31.204 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:13:31.204 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:13:31.204 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:13:31.204 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:13:31.204 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:13:31.204 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:13:31.204 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T04:13:31.205 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:13:31.205 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:13:31.205 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:13:31.205 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:13:31.205 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:13:31.205 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:13:31.205 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:13:31.205 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:13:31.205 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:13:31.205 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:13:31.205 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:13:31.205 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:13:31.205 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:13:31.206 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T04:13:31.206 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T04:13:31.206 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T04:13:31.206 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T04:13:31.206 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T04:13:31.206 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T04:13:31.206 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:13:31.206 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:13:31.206 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T04:13:31.206 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T04:13:31.206 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T04:13:31.206 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T04:13:31.206 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T04:13:31.206 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T04:13:31.207 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T04:13:31.207 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:13:31.207 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T04:13:31.207 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:13:31.207 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:13:31.207 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:13:31.207 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:13:31.207 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:13:31.207 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:13:31.207 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:13:31.208 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:13:31.208 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:13:31.208 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:13:31.208 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:13:31.208 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:13:31.208 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T04:13:31.208 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:13:31.208 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T04:13:31.208 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:13:31.208 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:13:31.208 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:13:31.208 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:13:31.208 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:13:31.209 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:13:31.209 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:13:31.209 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T04:13:31.209 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:13:31.209 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:13:31.209 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:13:31.209 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:13:31.209 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T04:13:31.209 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:13:31.209 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:13:31.209 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:13:31.209 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:13:31.209 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T04:13:31.210 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T04:13:31.210 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:13:31.210 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:13:31.210 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:13:31.210 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:13:31.210 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:13:31.210 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:13:31.210 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:13:31.210 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:13:31.210 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:13:31.210 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:13:31.210 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:13:31.210 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T04:13:31.210 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:13:31.211 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:13:31.211 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:13:31.211 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:13:31.211 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T04:13:31.211 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:13:31.211 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:13:31.211 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:13:31.211 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:13:31.211 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T04:13:31.212 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T04:13:31.212 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:13:31.212 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:13:31.212 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:13:31.212 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:13:31.212 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:13:31.212 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:13:31.212 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:13:31.212 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:13:31.212 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:13:31.212 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:13:31.212 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:13:31.212 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T04:13:31.213 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:13:31.213 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:13:31.213 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:13:31.213 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:13:31.213 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T04:13:31.213 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:13:31.213 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:13:31.213 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:13:31.213 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:13:31.213 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T04:13:31.213 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T04:13:31.213 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:13:31.213 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:13:31.213 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:13:31.214 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:13:31.214 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:13:31.214 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:13:31.214 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T04:13:31.214 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T04:13:31.214 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:13:31.214 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:13:31.214 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:13:31.214 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:13:31.214 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:13:31.214 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:13:31.214 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T04:13:31.214 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:13:31.214 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:13:31.215 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:13:31.215 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:13:31.215 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:13:31.215 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:13:31.215 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:13:31.215 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T04:13:31.215 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T04:13:31.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.194+0000 7fd20926c640 10 librados: watch_flush enter 2024-09-16T04:13:31.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.194+0000 7fd20926c640 10 librados: watch_flush exit 2024-09-16T04:13:31.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.194+0000 7fd20926c640 20 client.1034616.objecter shutdown clearing up homeless session... 2024-09-16T04:13:31.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.194+0000 7fd20926c640 10 client.1034616.objecter successfully canceled tick 2024-09-16T04:13:31.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.194+0000 7fd20926c640 1 -- 172.21.15.42:0/365298843 >> v2:172.21.15.154:6812/254845342 conn(0x7fd1e4037810 msgr2=0x7fd1e4039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:31.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.194+0000 7fd20926c640 1 --2- 172.21.15.42:0/365298843 >> v2:172.21.15.154:6812/254845342 conn(0x7fd1e4037810 0x7fd1e4039cd0 secure :-1 s=READY pgs=3986 cs=0 l=1 rev1=1 crypto rx=0x7fd20411f030 tx=0x7fd1ec009040 comp rx=0 tx=0).stop 2024-09-16T04:13:31.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.194+0000 7fd20926c640 10 monclient: shutdown 2024-09-16T04:13:31.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.194+0000 7fd20926c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:13:31.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.194+0000 7fd20926c640 1 -- 172.21.15.42:0/365298843 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd20411cb10 msgr2=0x7fd20411cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:31.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.194+0000 7fd20926c640 1 --2- 172.21.15.42:0/365298843 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd20411cb10 0x7fd20411cfa0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fd1fc0ac000 tx=0x7fd1fc019200 comp rx=0 tx=0).stop 2024-09-16T04:13:31.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.194+0000 7fd2037fe640 1 -- 172.21.15.42:0/365298843 reap_dead start 2024-09-16T04:13:31.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.194+0000 7fd2037fe640 1 -- 172.21.15.42:0/365298843 reap_dead start 2024-09-16T04:13:31.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.198+0000 7fd20926c640 1 -- 172.21.15.42:0/365298843 shutdown_connections 2024-09-16T04:13:31.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.198+0000 7fd20926c640 1 -- 172.21.15.42:0/365298843 >> 172.21.15.42:0/365298843 conn(0x7fd2040a2420 msgr2=0x7fd20409cb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:13:31.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.198+0000 7fd20926c640 1 -- 172.21.15.42:0/365298843 shutdown_connections 2024-09-16T04:13:31.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.198+0000 7fd20926c640 1 -- 172.21.15.42:0/365298843 wait complete. 2024-09-16T04:13:31.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.198+0000 7fd20926c640 1 librados: shutdown 2024-09-16T04:13:31.226 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:13:31.226 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T04:13:31.227 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T04:13:31.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.354+0000 7f2cb0e44640 10 monclient: get_monmap_and_config 2024-09-16T04:13:31.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.354+0000 7f2cb0e44640 10 monclient: build_initial_monmap 2024-09-16T04:13:31.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.354+0000 7f2cb0e44640 10 monclient: monmap: 2024-09-16T04:13:31.359 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:13:31.359 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:13:31.359 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:13:31.357310+0000 2024-09-16T04:13:31.359 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:13:31.357310+0000 2024-09-16T04:13:31.359 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:13:31.359 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:13:31.359 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:13:31.359 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:13:31.359 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:13:31.359 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:13:31.359 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:13:31.359 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:13:31.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.354+0000 7f2cb0e44640 1 Processor -- start 2024-09-16T04:13:31.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.354+0000 7f2cb0e44640 1 -- start start 2024-09-16T04:13:31.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.354+0000 7f2cb0e44640 10 monclient: init 2024-09-16T04:13:31.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.354+0000 7f2cb0e44640 10 monclient: _reopen_session rank -1 2024-09-16T04:13:31.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.354+0000 7f2cb0e44640 10 monclient: _add_conns ranks=[1,3,2,4,0] 2024-09-16T04:13:31.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.354+0000 7f2cb0e44640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2cac0a1330 addr v1:172.21.15.154:6789/0 2024-09-16T04:13:31.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.354+0000 7f2cb0e44640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f2cac0a1710 0x7f2cac0a1b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:13:31.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.354+0000 7f2cb0e44640 10 monclient(hunting): picked mon.noname-a con 0x7f2cac0a1710 addr v2:172.21.15.42:3300/0 2024-09-16T04:13:31.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.354+0000 7f2cb0e44640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2cac1393a0 0x7f2cac13b790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:13:31.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.354+0000 7f2cb0e44640 10 monclient(hunting): picked mon.noname-b con 0x7f2cac1393a0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:13:31.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.354+0000 7f2cb0e44640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2cac09e4c0 con 0x7f2cac0a1330 2024-09-16T04:13:31.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.354+0000 7f2cb0e44640 10 monclient(hunting): start opening mon connection 2024-09-16T04:13:31.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.354+0000 7f2cb0e44640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f2cac09edf0 con 0x7f2cac0a1710 2024-09-16T04:13:31.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.354+0000 7f2cb0e44640 10 monclient(hunting): start opening mon connection 2024-09-16T04:13:31.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.354+0000 7f2cb0e44640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f2cac09cd60 con 0x7f2cac1393a0 2024-09-16T04:13:31.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.354+0000 7f2cb0e44640 10 monclient(hunting): _renew_subs 2024-09-16T04:13:31.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.354+0000 7f2cb0e44640 10 monclient(hunting): authenticate will time out at 2858.326920s 2024-09-16T04:13:31.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.354+0000 7f2caad9d640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2cac1393a0 0x7f2cac13b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:13:31.364 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.354+0000 7f2ca9d9b640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f2cac0a1710 0x7f2cac0a1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:13:31.364 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.354+0000 7f2caad9d640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2cac1393a0 0x7f2cac13b790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:48992/0 (socket says 172.21.15.42:48992) 2024-09-16T04:13:31.364 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.354+0000 7f2caad9d640 1 -- 172.21.15.42:0/2136304408 learned_addr learned my addr 172.21.15.42:0/2136304408 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:13:31.364 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.354+0000 7f2caad9d640 10 monclient(hunting): get_auth_request con 0x7f2cac1393a0 auth_method 0 2024-09-16T04:13:31.364 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.354+0000 7f2caa59c640 1 --1- 172.21.15.42:0/2136304408 >> v1:172.21.15.154:6789/0 conn(0x7f2cac0a1330 0x7f2cac09f180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.42:49032/0 (socket says 172.21.15.42:49032) 2024-09-16T04:13:31.364 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.354+0000 7f2caad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:13:31.364 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.354+0000 7f2caad9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:31.364 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.354+0000 7f2caad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:13:31.364 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.354+0000 7f2ca9d9b640 10 monclient(hunting): get_auth_request con 0x7f2cac0a1710 auth_method 0 2024-09-16T04:13:31.364 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.354+0000 7f2ca9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:13:31.364 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.354+0000 7f2ca9d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:31.364 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.354+0000 7f2ca9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:13:31.365 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.354+0000 7f2caad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:13:31.365 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.354+0000 7f2caad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:13:31.365 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.354+0000 7f2caad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:13:31.365 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.354+0000 7f2ca9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:13:31.365 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.354+0000 7f2ca9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:13:31.365 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.354+0000 7f2ca9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:13:31.365 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.354+0000 7f2caad9d640 10 monclient(hunting): handle_auth_done global_id 1094114 payload 274 2024-09-16T04:13:31.365 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.354+0000 7f2caad9d640 1 -- 172.21.15.42:0/2136304408 >> v2:172.21.15.42:3300/0 conn(0x7f2cac0a1710 msgr2=0x7f2cac0a1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:13:31.365 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.354+0000 7f2caad9d640 1 --2- 172.21.15.42:0/2136304408 >> v2:172.21.15.42:3300/0 conn(0x7f2cac0a1710 0x7f2cac0a1b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:13:31.365 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.354+0000 7f2caad9d640 1 -- 172.21.15.42:0/2136304408 >> v1:172.21.15.154:6789/0 conn(0x7f2cac0a1330 legacy=0x7f2cac09f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:31.365 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.354+0000 7f2caad9d640 10 monclient: _finish_hunting 0 2024-09-16T04:13:31.365 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.354+0000 7f2caad9d640 1 monclient: found mon.noname-b 2024-09-16T04:13:31.366 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.354+0000 7f2caad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:13:31.366 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.354+0000 7f2caad9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T04:13:31.366 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.354+0000 7f2caad9d640 1 -- 172.21.15.42:0/2136304408 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2cac0a2080 con 0x7f2cac1393a0 2024-09-16T04:13:31.366 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.354+0000 7f2ca9d9b640 1 --2- 172.21.15.42:0/2136304408 >> v2:172.21.15.42:3300/0 conn(0x7f2cac0a1710 0x7f2cac0a1b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:13:31.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.358+0000 7f2caad9d640 1 --2- 172.21.15.42:0/2136304408 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2cac1393a0 0x7f2cac13b790 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f2ca4009e50 tx=0x7f2ca401c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=11929c041a3778d5 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:13:31.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.358+0000 7f2ca959a640 1 -- 172.21.15.42:0/2136304408 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ca40a56c0 con 0x7f2cac1393a0 2024-09-16T04:13:31.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.358+0000 7f2ca959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:13:31.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.358+0000 7f2ca959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:13:31.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.358+0000 7f2ca959a640 10 monclient: dump: 2024-09-16T04:13:31.376 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:13:31.376 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:13:31.376 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:31.376 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:31.376 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:31.376 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:13:31.376 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:13:31.377 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:13:31.380 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:13:31.381 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:13:31.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.358+0000 7f2ca959a640 10 monclient: _finish_auth 0 2024-09-16T04:13:31.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.358+0000 7f2ca959a640 10 monclient: _check_auth_tickets 2024-09-16T04:13:31.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.358+0000 7f2ca959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:13:31.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.358+0000 7f2ca959a640 1 -- 172.21.15.42:0/2136304408 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ca40a5860 con 0x7f2cac1393a0 2024-09-16T04:13:31.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.358+0000 7f2cb0e44640 5 monclient: authenticate success, global_id 1094114 2024-09-16T04:13:31.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.358+0000 7f2cb0e44640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:13:31.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.358+0000 7f2ca959a640 10 monclient: handle_config config(0 keys) 2024-09-16T04:13:31.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.358+0000 7f2cb0e44640 10 monclient: get_monmap_and_config success 2024-09-16T04:13:31.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.358+0000 7f2cb0e44640 4 set_mon_vals no callback set 2024-09-16T04:13:31.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.358+0000 7f2ca959a640 1 -- 172.21.15.42:0/2136304408 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ca40a5c30 con 0x7f2cac1393a0 2024-09-16T04:13:31.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.358+0000 7f2ca959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:13:31.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.358+0000 7f2ca959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:13:31.382 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.358+0000 7f2ca959a640 10 monclient: dump: 2024-09-16T04:13:31.382 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:13:31.382 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:13:31.382 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:31.382 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:31.382 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:31.382 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:13:31.382 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:13:31.382 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:13:31.382 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:13:31.382 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:13:31.382 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.358+0000 7f2cb0e44640 10 monclient: shutdown 2024-09-16T04:13:31.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.358+0000 7f2cb0e44640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:13:31.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.358+0000 7f2cb0e44640 1 -- 172.21.15.42:0/2136304408 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2cac1393a0 msgr2=0x7f2cac13b790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:31.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.358+0000 7f2cb0e44640 1 --2- 172.21.15.42:0/2136304408 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2cac1393a0 0x7f2cac13b790 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f2ca4009e50 tx=0x7f2ca401c8f0 comp rx=0 tx=0).stop 2024-09-16T04:13:31.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.358+0000 7f2cb0e44640 1 -- 172.21.15.42:0/2136304408 shutdown_connections 2024-09-16T04:13:31.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.358+0000 7f2cb0e44640 1 --2- 172.21.15.42:0/2136304408 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2cac1393a0 0x7f2cac13b790 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:13:31.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.358+0000 7f2cb0e44640 1 --2- 172.21.15.42:0/2136304408 >> v2:172.21.15.42:3300/0 conn(0x7f2cac0a1710 0x7f2cac0a1b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:13:31.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.358+0000 7f2cb0e44640 1 -- 172.21.15.42:0/2136304408 >> 172.21.15.42:0/2136304408 conn(0x7f2cac0a2420 msgr2=0x7f2cac0a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:13:31.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.358+0000 7f2cb0e44640 1 -- 172.21.15.42:0/2136304408 shutdown_connections 2024-09-16T04:13:31.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.358+0000 7f2cb0e44640 1 -- 172.21.15.42:0/2136304408 wait complete. 2024-09-16T04:13:31.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.358+0000 7f2cb0e44640 10 monclient: build_initial_monmap 2024-09-16T04:13:31.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.358+0000 7f2cb0e44640 10 monclient: monmap: 2024-09-16T04:13:31.383 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:13:31.383 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:13:31.384 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:13:31.384 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:13:31.384 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:13:31.384 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:13:31.384 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:13:31.384 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:13:31.384 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:13:31.384 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:13:31.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.358+0000 7f2cb0e44640 1 librados: starting msgr at 2024-09-16T04:13:31.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.358+0000 7f2cb0e44640 1 librados: starting objecter 2024-09-16T04:13:31.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.358+0000 7f2cb0e44640 1 Processor -- start 2024-09-16T04:13:31.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.358+0000 7f2cb0e44640 1 -- start start 2024-09-16T04:13:31.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.358+0000 7f2cb0e44640 1 librados: setting wanted keys 2024-09-16T04:13:31.385 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.358+0000 7f2cb0e44640 1 librados: calling monclient init 2024-09-16T04:13:31.385 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.358+0000 7f2cb0e44640 10 monclient: init 2024-09-16T04:13:31.385 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2cb0e44640 10 monclient: _reopen_session rank -1 2024-09-16T04:13:31.385 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2cb0e44640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:13:31.385 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2cb0e44640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2cac0a1330 0x7f2cac13e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:13:31.385 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2cb0e44640 10 monclient(hunting): picked mon.noname-a con 0x7f2cac0a1330 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:13:31.385 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2cb0e44640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2cac0a1710 0x7f2cac13ea30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:13:31.385 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2cb0e44640 10 monclient(hunting): picked mon.noname-c con 0x7f2cac0a1710 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:13:31.385 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2cb0e44640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2cac1393a0 0x7f2cac13ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:13:31.385 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2cb0e44640 10 monclient(hunting): picked mon.noname-b con 0x7f2cac1393a0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:13:31.385 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2cb0e44640 10 monclient(hunting): start opening mon connection 2024-09-16T04:13:31.385 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2cb0e44640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f2cac09be60 con 0x7f2cac0a1330 2024-09-16T04:13:31.385 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2cb0e44640 10 monclient(hunting): start opening mon connection 2024-09-16T04:13:31.386 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2cb0e44640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f2cac09bce0 con 0x7f2cac1393a0 2024-09-16T04:13:31.386 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2cb0e44640 10 monclient(hunting): start opening mon connection 2024-09-16T04:13:31.386 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2cb0e44640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f2cac09bfe0 con 0x7f2cac0a1710 2024-09-16T04:13:31.386 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2cb0e44640 10 monclient(hunting): _renew_subs 2024-09-16T04:13:31.386 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2cb0e44640 10 monclient(hunting): authenticate will time out at 2858.332355s 2024-09-16T04:13:31.386 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2ca9d9b640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2cac0a1710 0x7f2cac13ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:13:31.386 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2caa59c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2cac0a1330 0x7f2cac13e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:13:31.386 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2ca9d9b640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2cac0a1710 0x7f2cac13ea30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:49004/0 (socket says 172.21.15.42:49004) 2024-09-16T04:13:31.386 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2ca9d9b640 1 -- 172.21.15.42:0/2104271530 learned_addr learned my addr 172.21.15.42:0/2104271530 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:13:31.386 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2ca9d9b640 10 monclient(hunting): get_auth_request con 0x7f2cac0a1710 auth_method 0 2024-09-16T04:13:31.386 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2ca9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:13:31.387 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2ca9d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:31.387 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2ca9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:13:31.387 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2caa59c640 10 monclient(hunting): get_auth_request con 0x7f2cac0a1330 auth_method 0 2024-09-16T04:13:31.387 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2caa59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:13:31.387 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2caa59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:31.387 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2caa59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:13:31.387 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2ca9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:13:31.387 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2ca9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:13:31.387 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2ca9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:13:31.387 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2caa59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:13:31.387 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2caa59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:13:31.388 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2caa59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:13:31.388 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2caad9d640 1 --2- 172.21.15.42:0/2104271530 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2cac1393a0 0x7f2cac13ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:13:31.388 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2caad9d640 10 monclient(hunting): get_auth_request con 0x7f2cac1393a0 auth_method 0 2024-09-16T04:13:31.388 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2caad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:13:31.388 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2caad9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:31.388 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2caad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:13:31.388 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2ca9d9b640 10 monclient(hunting): handle_auth_done global_id 1094117 payload 824 2024-09-16T04:13:31.388 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2ca9d9b640 1 -- 172.21.15.42:0/2104271530 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2cac1393a0 msgr2=0x7f2cac13ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:31.388 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2ca9d9b640 1 --2- 172.21.15.42:0/2104271530 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2cac1393a0 0x7f2cac13ef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:13:31.388 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2ca9d9b640 1 -- 172.21.15.42:0/2104271530 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2cac0a1330 msgr2=0x7f2cac13e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:31.388 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2ca9d9b640 1 --2- 172.21.15.42:0/2104271530 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2cac0a1330 0x7f2cac13e4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:13:31.389 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2ca9d9b640 10 monclient: _finish_hunting 0 2024-09-16T04:13:31.389 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2ca9d9b640 1 monclient: found mon.noname-c 2024-09-16T04:13:31.389 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2ca9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:13:31.389 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2ca9d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T04:13:31.389 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2ca9d9b640 1 -- 172.21.15.42:0/2104271530 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2cac132710 con 0x7f2cac0a1710 2024-09-16T04:13:31.389 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2ca9d9b640 1 --2- 172.21.15.42:0/2104271530 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2cac0a1710 0x7f2cac13ea30 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f2c9c00ebe0 tx=0x7f2c9c00c520 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:13:31.389 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2ca2ffd640 10 client.?.objecter ms_handle_connect 0x7f2cac0a1710 2024-09-16T04:13:31.389 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2ca2ffd640 10 client.?.objecter resend_mon_ops 2024-09-16T04:13:31.389 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2ca9d9b640 1 -- 172.21.15.42:0/2104271530 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2cac0a1710 msgr2=0x7f2cac13ea30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-16T04:13:31.389 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2ca9d9b640 1 -- 172.21.15.42:0/2104271530 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2cac0a1710 msgr2=0x7f2cac13ea30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:13:31.389 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2ca2ffd640 1 -- 172.21.15.42:0/2104271530 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c9c017070 con 0x7f2cac0a1710 2024-09-16T04:13:31.389 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2ca9d9b640 1 --2- 172.21.15.42:0/2104271530 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2cac0a1710 0x7f2cac13ea30 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f2c9c00ebe0 tx=0x7f2c9c00c520 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:13:31.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2ca2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:13:31.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2ca9d9b640 1 --2- 172.21.15.42:0/2104271530 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2cac0a1710 0x7f2cac13ea30 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f2c9c00ebe0 tx=0x7f2c9c00c520 comp rx=0 tx=0).stop 2024-09-16T04:13:31.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2ca2ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:13:31.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2ca2ffd640 10 monclient: dump: 2024-09-16T04:13:31.390 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:13:31.390 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:13:31.390 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:31.390 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:31.390 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:31.390 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:13:31.391 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:13:31.391 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:13:31.391 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:13:31.391 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:13:31.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2ca2ffd640 10 monclient: _finish_auth 0 2024-09-16T04:13:31.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2ca2ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:13:31.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2ca2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:13:31.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2ca2ffd640 10 monclient: ms_handle_reset current mon [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:13:31.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2ca2ffd640 10 monclient: _reopen_session rank -1 2024-09-16T04:13:31.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2ca2ffd640 1 -- 172.21.15.42:0/2104271530 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2cac0a1710 msgr2=0x7f2cac13ea30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:13:31.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2ca2ffd640 1 --2- 172.21.15.42:0/2104271530 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2cac0a1710 0x7f2cac13ea30 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:13:31.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2ca2ffd640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:13:31.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2ca2ffd640 1 --2- 172.21.15.42:0/2104271530 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2c8c005080 0x7f2c8c0074b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:13:31.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2ca2ffd640 10 monclient(hunting): picked mon.c con 0x7f2c8c005080 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:13:31.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2ca2ffd640 1 --2- 172.21.15.42:0/2104271530 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2cac1393a0 0x7f2c8c009fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:13:31.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2ca2ffd640 10 monclient(hunting): picked mon.b con 0x7f2cac1393a0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:13:31.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2ca2ffd640 1 --2- 172.21.15.42:0/2104271530 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2cac0a1330 0x7f2c8c00ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:13:31.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2ca2ffd640 10 monclient(hunting): picked mon.a con 0x7f2cac0a1330 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:13:31.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2ca2ffd640 10 monclient(hunting): start opening mon connection 2024-09-16T04:13:31.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2ca2ffd640 1 -- 172.21.15.42:0/2104271530 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f2c8c00d460 con 0x7f2cac0a1330 2024-09-16T04:13:31.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2ca2ffd640 10 monclient(hunting): start opening mon connection 2024-09-16T04:13:31.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2ca2ffd640 1 -- 172.21.15.42:0/2104271530 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f2c8c00d5e0 con 0x7f2cac1393a0 2024-09-16T04:13:31.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2ca2ffd640 10 monclient(hunting): start opening mon connection 2024-09-16T04:13:31.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2ca2ffd640 1 -- 172.21.15.42:0/2104271530 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f2c8c00d760 con 0x7f2c8c005080 2024-09-16T04:13:31.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2ca2ffd640 10 monclient(hunting): _renew_subs 2024-09-16T04:13:31.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2caa59c640 1 --2- 172.21.15.42:0/2104271530 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2c8c005080 0x7f2c8c0074b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:13:31.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2caa59c640 10 monclient(hunting): get_auth_request con 0x7f2c8c005080 auth_method 0 2024-09-16T04:13:31.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2caa59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:13:31.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2caa59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:31.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2caa59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:13:31.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2caad9d640 1 --2- 172.21.15.42:0/2104271530 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2cac0a1330 0x7f2c8c00ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:13:31.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2caad9d640 10 monclient(hunting): get_auth_request con 0x7f2cac0a1330 auth_method 0 2024-09-16T04:13:31.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2caad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:13:31.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2caad9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:31.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2caad9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:13:31.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2caa59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:13:31.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2caa59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:13:31.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2caa59c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:13:31.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2caad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:13:31.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2caad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:13:31.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2caad9d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:13:31.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2ca9d9b640 1 --2- 172.21.15.42:0/2104271530 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2cac1393a0 0x7f2c8c009fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:13:31.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2ca9d9b640 10 monclient(hunting): get_auth_request con 0x7f2cac1393a0 auth_method 0 2024-09-16T04:13:31.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2ca9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:13:31.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2ca9d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:31.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2ca9d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:13:31.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2caa59c640 10 monclient(hunting): handle_auth_done global_id 1094117 payload 293 2024-09-16T04:13:31.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2caa59c640 1 -- 172.21.15.42:0/2104271530 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2cac1393a0 msgr2=0x7f2c8c009fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:31.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2caa59c640 1 --2- 172.21.15.42:0/2104271530 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2cac1393a0 0x7f2c8c009fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:13:31.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2caa59c640 1 -- 172.21.15.42:0/2104271530 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2cac0a1330 msgr2=0x7f2c8c00ca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:31.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2caa59c640 1 --2- 172.21.15.42:0/2104271530 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2cac0a1330 0x7f2c8c00ca50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:13:31.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2caa59c640 10 monclient: _finish_hunting 0 2024-09-16T04:13:31.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2caa59c640 1 monclient: found mon.c 2024-09-16T04:13:31.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2caa59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:13:31.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2caa59c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:13:31.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2caa59c640 1 -- 172.21.15.42:0/2104271530 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=2+}) -- 0x7f2c8c00d9d0 con 0x7f2c8c005080 2024-09-16T04:13:31.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2caa59c640 10 monclient: _finish_auth 0 2024-09-16T04:13:31.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2caa59c640 10 monclient: _check_auth_tickets 2024-09-16T04:13:31.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2caa59c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:13:31.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2cb0e44640 5 monclient: authenticate success, global_id 1094117 2024-09-16T04:13:31.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2cb0e44640 10 monclient: _renew_subs 2024-09-16T04:13:31.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2caad9d640 1 --2- 172.21.15.42:0/2104271530 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2cac0a1330 0x7f2c8c00ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:13:31.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2cb0e44640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:13:31.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2cb0e44640 1 -- 172.21.15.42:0/2104271530 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2cac0a5210 con 0x7f2c8c005080 2024-09-16T04:13:31.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2cb0e44640 10 client.1094117.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:13:31.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2cb0e44640 10 monclient: _renew_subs 2024-09-16T04:13:31.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2cb0e44640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:13:31.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2cb0e44640 1 -- 172.21.15.42:0/2104271530 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f2cac073a30 con 0x7f2c8c005080 2024-09-16T04:13:31.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2cb0e44640 1 librados: init done 2024-09-16T04:13:31.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2ca2ffd640 10 client.1094117.objecter ms_handle_connect 0x7f2c8c005080 2024-09-16T04:13:31.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2caa59c640 1 --2- 172.21.15.42:0/2104271530 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2c8c005080 0x7f2c8c0074b0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f2cac072390 tx=0x7f2c9c009190 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:13:31.396 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2ca2ffd640 10 client.1094117.objecter resend_mon_ops 2024-09-16T04:13:31.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2ca2ffd640 1 -- 172.21.15.42:0/2104271530 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c94004a80 con 0x7f2c8c005080 2024-09-16T04:13:31.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2ca2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:13:31.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2ca2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:13:31.397 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2ca2ffd640 10 monclient: dump: 2024-09-16T04:13:31.397 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:13:31.397 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:13:31.397 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:31.397 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:31.397 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:31.397 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:13:31.397 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:13:31.397 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:13:31.397 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:13:31.397 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:13:31.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2ca2ffd640 1 -- 172.21.15.42:0/2104271530 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2c94018070 con 0x7f2c8c005080 2024-09-16T04:13:31.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2ca2ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:13:31.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2ca8d99640 4 set_mon_vals no callback set 2024-09-16T04:13:31.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2cb0e44640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:13:31.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2cb0e44640 1 --2- 172.21.15.42:0/2104271530 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2c90002890 0x7f2c90004d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:13:31.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2cb0e44640 10 monclient(hunting): start opening mon connection 2024-09-16T04:13:31.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2cb0e44640 1 -- 172.21.15.42:0/2104271530 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f2c90007cc0 con 0x7f2c90002890 2024-09-16T04:13:31.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2ca2ffd640 1 -- 172.21.15.42:0/2104271530 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f2c940018a0 con 0x7f2c8c005080 2024-09-16T04:13:31.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2ca9d9b640 1 --2- 172.21.15.42:0/2104271530 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2c90002890 0x7f2c90004d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:13:31.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2ca2ffd640 1 --2- 172.21.15.42:0/2104271530 >> v2:172.21.15.154:6812/254845342 conn(0x7f2cac0a1710 0x7f2c8c042070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:13:31.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2ca9d9b640 10 monclient: get_auth_request con 0x7f2c90002890 auth_method 0 2024-09-16T04:13:31.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2ca9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:13:31.398 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2ca9d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:31.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2ca9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:13:31.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2ca2ffd640 1 -- 172.21.15.42:0/2104271530 <== mon.2 v2:172.21.15.42:3301/0 4 ==== osd_map(706..706 src has 681..706) ==== 5040+0+0 (secure 0 0 0) 0x7f2c94047640 con 0x7f2c8c005080 2024-09-16T04:13:31.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2ca2ffd640 10 client.1094117.objecter ms_dispatch 0x7f2cac000ba0 osd_map(706..706 src has 681..706) 2024-09-16T04:13:31.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2ca2ffd640 3 client.1094117.objecter handle_osd_map got epochs [706,706] > 0 2024-09-16T04:13:31.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2ca2ffd640 3 client.1094117.objecter handle_osd_map decoding full epoch 706 2024-09-16T04:13:31.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2ca2ffd640 20 client.1094117.objecter dump_active .. 0 homeless 2024-09-16T04:13:31.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2ca9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:13:31.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2ca9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:13:31.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2ca9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:13:31.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2caad9d640 1 --2- 172.21.15.42:0/2104271530 >> v2:172.21.15.154:6812/254845342 conn(0x7f2cac0a1710 0x7f2c8c042070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:13:31.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2caad9d640 10 monclient: get_auth_request con 0x7f2cac0a1710 auth_method 0 2024-09-16T04:13:31.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.362+0000 7f2ca9d9b640 10 monclient(hunting): handle_auth_done global_id 1074283 payload 824 2024-09-16T04:13:31.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.366+0000 7f2ca9d9b640 1 -- 172.21.15.42:0/2104271530 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2c90007ec0 con 0x7f2c90002890 2024-09-16T04:13:31.399 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.366+0000 7f2ca2ffd640 10 client.1094117.objecter ms_handle_connect 0x7f2c90002890 2024-09-16T04:13:31.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.366+0000 7f2ca9d9b640 1 --2- 172.21.15.42:0/2104271530 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2c90002890 0x7f2c90004d30 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f2cac134cb0 tx=0x7f2c9c004740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:13:31.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.366+0000 7f2ca2ffd640 10 client.1094117.objecter resend_mon_ops 2024-09-16T04:13:31.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.366+0000 7f2ca2ffd640 10 client.1094117.objecter ms_handle_connect 0x7f2cac0a1710 2024-09-16T04:13:31.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.366+0000 7f2caad9d640 1 --2- 172.21.15.42:0/2104271530 >> v2:172.21.15.154:6812/254845342 conn(0x7f2cac0a1710 0x7f2c8c042070 secure :-1 s=READY pgs=3987 cs=0 l=1 rev1=1 crypto rx=0x7f2ca40ae000 tx=0x7f2ca40b8040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:13:31.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.366+0000 7f2ca2ffd640 1 -- 172.21.15.42:0/2104271530 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c9c009400 con 0x7f2c90002890 2024-09-16T04:13:31.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.366+0000 7f2ca2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:13:31.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.366+0000 7f2ca2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:13:31.400 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.366+0000 7f2ca2ffd640 10 monclient: dump: 2024-09-16T04:13:31.400 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:13:31.400 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:13:31.401 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:31.401 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:31.401 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:31.401 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:13:31.401 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:13:31.401 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:13:31.401 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:13:31.401 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:13:31.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.366+0000 7f2ca2ffd640 1 -- 172.21.15.42:0/2104271530 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f2c9c00cc30 con 0x7f2c90002890 2024-09-16T04:13:31.401 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.366+0000 7f2ca2ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:13:31.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.366+0000 7f2ca2ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:13:31.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.366+0000 7f2ca2ffd640 1 -- 172.21.15.42:0/2104271530 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2c90002890 msgr2=0x7f2c90004d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:31.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.366+0000 7f2ca2ffd640 1 --2- 172.21.15.42:0/2104271530 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2c90002890 0x7f2c90004d30 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f2cac134cb0 tx=0x7f2c9c004740 comp rx=0 tx=0).stop 2024-09-16T04:13:31.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.366+0000 7f2ca2ffd640 1 -- 172.21.15.42:0/2104271530 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2c90002890 msgr2=0x7f2c90004d30 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:13:31.402 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.366+0000 7f2ca2ffd640 1 --2- 172.21.15.42:0/2104271530 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2c90002890 0x7f2c90004d30 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:13:31.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.370+0000 7f2cb0e44640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:13:31.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.370+0000 7f2cb0e44640 1 --2- 172.21.15.42:0/2104271530 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2c90006d30 0x7f2c900071c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:13:31.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.370+0000 7f2cb0e44640 10 monclient(hunting): start opening mon connection 2024-09-16T04:13:31.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.370+0000 7f2cb0e44640 1 -- 172.21.15.42:0/2104271530 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f2c9000a200 con 0x7f2c90006d30 2024-09-16T04:13:31.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.374+0000 7f2ca9d9b640 1 --2- 172.21.15.42:0/2104271530 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2c90006d30 0x7f2c900071c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:13:31.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.374+0000 7f2ca9d9b640 10 monclient: get_auth_request con 0x7f2c90006d30 auth_method 0 2024-09-16T04:13:31.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.374+0000 7f2ca9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:13:31.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.374+0000 7f2ca9d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:31.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.374+0000 7f2ca9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:13:31.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.374+0000 7f2ca9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:13:31.404 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.374+0000 7f2ca9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:13:31.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.374+0000 7f2ca9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:13:31.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.374+0000 7f2ca9d9b640 10 monclient(hunting): handle_auth_done global_id 1074286 payload 824 2024-09-16T04:13:31.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.374+0000 7f2ca9d9b640 1 -- 172.21.15.42:0/2104271530 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f2c90007700 con 0x7f2c90006d30 2024-09-16T04:13:31.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.374+0000 7f2ca2ffd640 10 client.1094117.objecter ms_handle_connect 0x7f2c90006d30 2024-09-16T04:13:31.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.374+0000 7f2ca2ffd640 10 client.1094117.objecter resend_mon_ops 2024-09-16T04:13:31.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.374+0000 7f2ca9d9b640 1 --2- 172.21.15.42:0/2104271530 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2c90006d30 0x7f2c900071c0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f2c9c009750 tx=0x7f2c9c014000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:13:31.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.374+0000 7f2ca2ffd640 1 -- 172.21.15.42:0/2104271530 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c9c0044c0 con 0x7f2c90006d30 2024-09-16T04:13:31.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.374+0000 7f2ca2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:13:31.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.374+0000 7f2ca2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:13:31.405 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.374+0000 7f2ca2ffd640 10 monclient: dump: 2024-09-16T04:13:31.405 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:13:31.405 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:13:31.405 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:31.406 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:31.406 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:31.406 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:13:31.406 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:13:31.406 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:13:31.406 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:13:31.406 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:13:31.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.374+0000 7f2ca2ffd640 1 -- 172.21.15.42:0/2104271530 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4713 (secure 0 0 0) 0x7f2c9c0044c0 con 0x7f2c90006d30 2024-09-16T04:13:31.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.374+0000 7f2ca2ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:13:31.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.374+0000 7f2ca2ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:13:31.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.374+0000 7f2ca2ffd640 1 -- 172.21.15.42:0/2104271530 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2c90006d30 msgr2=0x7f2c900071c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:31.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.374+0000 7f2ca2ffd640 1 --2- 172.21.15.42:0/2104271530 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2c90006d30 0x7f2c900071c0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f2c9c009750 tx=0x7f2c9c014000 comp rx=0 tx=0).stop 2024-09-16T04:13:31.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.374+0000 7f2ca2ffd640 1 -- 172.21.15.42:0/2104271530 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2c90006d30 msgr2=0x7f2c900071c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:13:31.406 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.374+0000 7f2ca2ffd640 1 --2- 172.21.15.42:0/2104271530 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2c90006d30 0x7f2c900071c0 secure :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f2c9c009750 tx=0x7f2c9c014000 comp rx=0 tx=0).stop 2024-09-16T04:13:31.407 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T04:13:31.407 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:13:31.407 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:13:31.407 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T04:13:31.407 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 286, 2024-09-16T04:13:31.407 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 124543, 2024-09-16T04:13:31.407 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T04:13:31.407 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T04:13:31.407 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T04:13:31.407 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T04:13:31.408 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:13:31.408 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 1, 2024-09-16T04:13:31.408 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:13:31.408 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T04:13:31.408 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T04:13:31.408 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:13:31.408 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:13:31.408 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:13:31.408 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:13:31.408 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:13:31.408 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:13:31.408 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:13:31.409 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:13:31.409 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:13:31.409 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:13:31.409 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:13:31.409 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:13:31.409 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:13:31.409 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T04:13:31.409 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:13:31.409 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:13:31.409 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:13:31.409 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:13:31.409 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:13:31.409 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:13:31.410 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:13:31.410 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:13:31.410 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:13:31.410 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:13:31.410 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:13:31.410 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:13:31.410 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:13:31.410 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T04:13:31.410 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T04:13:31.410 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T04:13:31.410 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T04:13:31.410 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T04:13:31.410 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T04:13:31.410 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:13:31.411 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:13:31.411 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T04:13:31.411 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T04:13:31.411 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T04:13:31.411 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T04:13:31.411 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T04:13:31.411 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T04:13:31.411 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T04:13:31.411 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:13:31.412 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T04:13:31.412 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:13:31.412 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:13:31.412 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:13:31.412 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:13:31.412 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:13:31.412 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:13:31.413 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:13:31.413 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:13:31.413 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:13:31.413 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:13:31.413 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:13:31.413 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:13:31.413 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T04:13:31.413 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:13:31.413 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T04:13:31.413 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:13:31.413 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:13:31.413 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:13:31.413 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:13:31.413 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:13:31.414 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:13:31.414 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:13:31.414 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T04:13:31.414 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:13:31.414 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:13:31.414 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:13:31.414 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:13:31.414 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T04:13:31.414 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:13:31.414 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:13:31.414 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:13:31.414 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:13:31.415 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T04:13:31.415 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T04:13:31.415 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:13:31.415 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:13:31.415 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:13:31.415 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:13:31.415 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:13:31.415 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:13:31.415 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:13:31.415 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:13:31.415 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:13:31.415 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:13:31.415 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:13:31.416 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T04:13:31.416 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:13:31.416 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:13:31.416 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:13:31.416 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:13:31.416 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T04:13:31.416 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:13:31.416 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:13:31.416 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:13:31.416 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:13:31.416 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T04:13:31.416 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T04:13:31.416 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:13:31.417 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:13:31.417 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:13:31.417 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:13:31.417 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:13:31.417 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:13:31.417 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:13:31.417 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:13:31.417 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:13:31.417 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:13:31.417 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:13:31.417 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T04:13:31.417 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:13:31.417 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:13:31.418 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:13:31.418 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:13:31.418 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T04:13:31.418 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:13:31.418 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:13:31.418 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:13:31.418 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:13:31.418 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T04:13:31.418 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T04:13:31.418 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:13:31.418 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:13:31.418 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:13:31.418 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:13:31.419 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:13:31.419 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:13:31.419 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T04:13:31.419 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T04:13:31.419 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:13:31.419 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:13:31.419 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:13:31.419 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:13:31.419 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:13:31.419 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:13:31.419 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T04:13:31.419 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:13:31.419 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:13:31.419 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:13:31.420 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 2 2024-09-16T04:13:31.420 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:13:31.420 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:13:31.420 INFO:teuthology.orchestra.run.smithi042.stdout: "mgr": [ 2024-09-16T04:13:31.420 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:13:31.420 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:13:31.420 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:13:31.420 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:13:31.420 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:13:31.420 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:13:31.420 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:13:31.420 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T04:13:31.420 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T04:13:31.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.374+0000 7f2cb0e44640 10 librados: watch_flush enter 2024-09-16T04:13:31.422 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.374+0000 7f2cb0e44640 10 librados: watch_flush exit 2024-09-16T04:13:31.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.374+0000 7f2cb0e44640 20 client.1094117.objecter shutdown clearing up homeless session... 2024-09-16T04:13:31.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.374+0000 7f2cb0e44640 10 client.1094117.objecter successfully canceled tick 2024-09-16T04:13:31.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.374+0000 7f2cb0e44640 1 -- 172.21.15.42:0/2104271530 >> v2:172.21.15.154:6812/254845342 conn(0x7f2cac0a1710 msgr2=0x7f2c8c042070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:31.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.374+0000 7f2cb0e44640 1 --2- 172.21.15.42:0/2104271530 >> v2:172.21.15.154:6812/254845342 conn(0x7f2cac0a1710 0x7f2c8c042070 secure :-1 s=READY pgs=3987 cs=0 l=1 rev1=1 crypto rx=0x7f2ca40ae000 tx=0x7f2ca40b8040 comp rx=0 tx=0).stop 2024-09-16T04:13:31.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.374+0000 7f2cb0e44640 10 monclient: shutdown 2024-09-16T04:13:31.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.374+0000 7f2cb0e44640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:13:31.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.374+0000 7f2cb0e44640 1 -- 172.21.15.42:0/2104271530 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2c8c005080 msgr2=0x7f2c8c0074b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:31.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.374+0000 7f2cb0e44640 1 --2- 172.21.15.42:0/2104271530 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2c8c005080 0x7f2c8c0074b0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f2cac072390 tx=0x7f2c9c009190 comp rx=0 tx=0).stop 2024-09-16T04:13:31.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.374+0000 7f2caad9d640 1 -- 172.21.15.42:0/2104271530 reap_dead start 2024-09-16T04:13:31.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.374+0000 7f2caad9d640 1 -- 172.21.15.42:0/2104271530 reap_dead start 2024-09-16T04:13:31.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.374+0000 7f2cb0e44640 1 -- 172.21.15.42:0/2104271530 shutdown_connections 2024-09-16T04:13:31.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.374+0000 7f2cb0e44640 1 -- 172.21.15.42:0/2104271530 >> 172.21.15.42:0/2104271530 conn(0x7f2cac0a2420 msgr2=0x7f2cac1302b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:13:31.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.374+0000 7f2cb0e44640 1 -- 172.21.15.42:0/2104271530 shutdown_connections 2024-09-16T04:13:31.423 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.374+0000 7f2cb0e44640 1 -- 172.21.15.42:0/2104271530 wait complete. 2024-09-16T04:13:31.424 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.374+0000 7f2cb0e44640 1 librados: shutdown 2024-09-16T04:13:31.424 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:13:31.424 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T04:13:31.424 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T04:13:31.541 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.534+0000 7f2a9e99c640 10 monclient: get_monmap_and_config 2024-09-16T04:13:31.541 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.534+0000 7f2a9e99c640 10 monclient: build_initial_monmap 2024-09-16T04:13:31.541 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.534+0000 7f2a9e99c640 10 monclient: monmap: 2024-09-16T04:13:31.542 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:13:31.542 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:13:31.542 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:13:31.539352+0000 2024-09-16T04:13:31.542 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:13:31.539352+0000 2024-09-16T04:13:31.542 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:13:31.542 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:13:31.542 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:13:31.542 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:13:31.542 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:13:31.542 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:13:31.542 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:13:31.542 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:13:31.542 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.538+0000 7f2a9e99c640 1 Processor -- start 2024-09-16T04:13:31.542 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.538+0000 7f2a9e99c640 1 -- start start 2024-09-16T04:13:31.543 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.538+0000 7f2a9e99c640 10 monclient: init 2024-09-16T04:13:31.545 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.538+0000 7f2a9e99c640 10 monclient: _reopen_session rank -1 2024-09-16T04:13:31.545 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.538+0000 7f2a9e99c640 10 monclient: _add_conns ranks=[0,3,1,2,4] 2024-09-16T04:13:31.545 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.538+0000 7f2a9e99c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2a980a39b0 addr v1:172.21.15.42:6789/0 2024-09-16T04:13:31.545 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.538+0000 7f2a9e99c640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f2a980a3d90 0x7f2a9813c380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:13:31.545 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.538+0000 7f2a9e99c640 10 monclient(hunting): picked mon.noname-a con 0x7f2a980a3d90 addr v2:172.21.15.42:3300/0 2024-09-16T04:13:31.545 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.538+0000 7f2a9e99c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2a9813c8c0 addr v1:172.21.15.154:6789/0 2024-09-16T04:13:31.545 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.538+0000 7f2a9e99c640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2a98134290 con 0x7f2a980a39b0 2024-09-16T04:13:31.545 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.538+0000 7f2a9e99c640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2a9809c140 con 0x7f2a9813c8c0 2024-09-16T04:13:31.545 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.538+0000 7f2a9e99c640 10 monclient(hunting): start opening mon connection 2024-09-16T04:13:31.545 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.538+0000 7f2a9e99c640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f2a9809c420 con 0x7f2a980a3d90 2024-09-16T04:13:31.545 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.538+0000 7f2a9e99c640 10 monclient(hunting): _renew_subs 2024-09-16T04:13:31.545 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.538+0000 7f2a9e99c640 10 monclient(hunting): authenticate will time out at 2858.509007s 2024-09-16T04:13:31.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.538+0000 7f2a8ffff640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f2a980a3d90 0x7f2a9813c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:13:31.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.538+0000 7f2a8ffff640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f2a980a3d90 0x7f2a9813c380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:52066/0 (socket says 172.21.15.42:52066) 2024-09-16T04:13:31.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.538+0000 7f2a8ffff640 1 -- 172.21.15.42:0/1362766227 learned_addr learned my addr 172.21.15.42:0/1362766227 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:13:31.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.538+0000 7f2a8ffff640 10 monclient(hunting): get_auth_request con 0x7f2a980a3d90 auth_method 0 2024-09-16T04:13:31.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.538+0000 7f2a8ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:13:31.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.538+0000 7f2a8ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:31.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.538+0000 7f2a8ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:13:31.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.538+0000 7f2a8ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:13:31.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.538+0000 7f2a8ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:13:31.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.538+0000 7f2a8ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:13:31.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.538+0000 7f2a8ffff640 10 monclient(hunting): handle_auth_done global_id 1034625 payload 274 2024-09-16T04:13:31.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.538+0000 7f2a8ffff640 1 -- 172.21.15.42:0/1362766227 >> v1:172.21.15.154:6789/0 conn(0x7f2a9813c8c0 legacy=0x7f2a9813ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:13:31.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.538+0000 7f2a8ffff640 1 -- 172.21.15.42:0/1362766227 >> v1:172.21.15.42:6789/0 conn(0x7f2a980a39b0 legacy=0x7f2a98133b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:31.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.538+0000 7f2a8ffff640 10 monclient: _finish_hunting 0 2024-09-16T04:13:31.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.538+0000 7f2a8ffff640 1 monclient: found mon.noname-a 2024-09-16T04:13:31.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.538+0000 7f2a8ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:13:31.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.538+0000 7f2a8ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:13:31.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.538+0000 7f2a8ffff640 1 -- 172.21.15.42:0/1362766227 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a9806dd60 con 0x7f2a980a3d90 2024-09-16T04:13:31.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.538+0000 7f2a8ffff640 1 --2- 172.21.15.42:0/1362766227 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2a980a3d90 0x7f2a9813c380 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f2a88009870 tx=0x7f2a8801c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df41a3161f711d80 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:13:31.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.538+0000 7f2a977fe640 1 -- 172.21.15.42:0/1362766227 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a880b0070 con 0x7f2a980a3d90 2024-09-16T04:13:31.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.538+0000 7f2a977fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:13:31.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.538+0000 7f2a977fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:13:31.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.538+0000 7f2a977fe640 10 monclient: dump: 2024-09-16T04:13:31.547 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:13:31.547 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:13:31.547 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:31.547 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:31.548 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:31.548 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:13:31.548 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:13:31.548 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:13:31.548 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:13:31.548 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:13:31.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.538+0000 7f2a977fe640 10 monclient: _finish_auth 0 2024-09-16T04:13:31.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.538+0000 7f2a977fe640 10 monclient: _check_auth_tickets 2024-09-16T04:13:31.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.538+0000 7f2a977fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:13:31.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.538+0000 7f2a977fe640 1 -- 172.21.15.42:0/1362766227 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a880ab600 con 0x7f2a980a3d90 2024-09-16T04:13:31.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.538+0000 7f2a977fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:13:31.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.538+0000 7f2a977fe640 1 -- 172.21.15.42:0/1362766227 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a880ab9d0 con 0x7f2a980a3d90 2024-09-16T04:13:31.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.538+0000 7f2a977fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:13:31.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.538+0000 7f2a977fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:13:31.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.538+0000 7f2a977fe640 10 monclient: dump: 2024-09-16T04:13:31.549 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:13:31.549 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:13:31.549 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:31.549 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:31.549 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:31.549 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:13:31.549 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:13:31.549 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:13:31.549 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:13:31.549 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:13:31.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.538+0000 7f2a9e99c640 5 monclient: authenticate success, global_id 1034625 2024-09-16T04:13:31.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.538+0000 7f2a9e99c640 10 monclient: get_monmap_and_config success 2024-09-16T04:13:31.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.538+0000 7f2a9e99c640 4 set_mon_vals no callback set 2024-09-16T04:13:31.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.542+0000 7f2a9e99c640 10 monclient: shutdown 2024-09-16T04:13:31.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.542+0000 7f2a9e99c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:13:31.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.542+0000 7f2a9e99c640 1 -- 172.21.15.42:0/1362766227 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2a980a3d90 msgr2=0x7f2a9813c380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:31.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.542+0000 7f2a9e99c640 1 --2- 172.21.15.42:0/1362766227 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2a980a3d90 0x7f2a9813c380 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f2a88009870 tx=0x7f2a8801c3e0 comp rx=0 tx=0).stop 2024-09-16T04:13:31.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.542+0000 7f2a9e99c640 1 -- 172.21.15.42:0/1362766227 shutdown_connections 2024-09-16T04:13:31.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.542+0000 7f2a9e99c640 1 --2- 172.21.15.42:0/1362766227 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2a980a3d90 0x7f2a9813c380 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:13:31.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.542+0000 7f2a9e99c640 1 -- 172.21.15.42:0/1362766227 >> 172.21.15.42:0/1362766227 conn(0x7f2a980a9fb0 msgr2=0x7f2a980aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:13:31.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.542+0000 7f2a9e99c640 1 -- 172.21.15.42:0/1362766227 shutdown_connections 2024-09-16T04:13:31.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.542+0000 7f2a9e99c640 1 -- 172.21.15.42:0/1362766227 wait complete. 2024-09-16T04:13:31.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.542+0000 7f2a9e99c640 10 monclient: build_initial_monmap 2024-09-16T04:13:31.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.542+0000 7f2a9e99c640 10 monclient: monmap: 2024-09-16T04:13:31.555 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:13:31.555 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:13:31.555 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:13:31.555 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:13:31.555 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:13:31.555 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:13:31.555 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:13:31.555 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:13:31.555 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:13:31.555 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:13:31.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.542+0000 7f2a9e99c640 1 librados: starting msgr at 2024-09-16T04:13:31.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.542+0000 7f2a9e99c640 1 librados: starting objecter 2024-09-16T04:13:31.556 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.542+0000 7f2a9e99c640 1 Processor -- start 2024-09-16T04:13:31.556 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.542+0000 7f2a9e99c640 1 -- start start 2024-09-16T04:13:31.556 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.542+0000 7f2a9e99c640 1 librados: setting wanted keys 2024-09-16T04:13:31.556 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.542+0000 7f2a9e99c640 1 librados: calling monclient init 2024-09-16T04:13:31.556 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.542+0000 7f2a9e99c640 10 monclient: init 2024-09-16T04:13:31.556 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.542+0000 7f2a9e99c640 10 monclient: _reopen_session rank -1 2024-09-16T04:13:31.556 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.542+0000 7f2a9e99c640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:13:31.556 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.542+0000 7f2a9e99c640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2a980a39b0 0x7f2a981391a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:13:31.556 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.542+0000 7f2a9e99c640 10 monclient(hunting): picked mon.noname-c con 0x7f2a980a39b0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:13:31.556 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.542+0000 7f2a9e99c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2a980a3d90 0x7f2a981396e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:13:31.556 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.542+0000 7f2a9e99c640 10 monclient(hunting): picked mon.noname-b con 0x7f2a980a3d90 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:13:31.556 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.542+0000 7f2a9e99c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2a9813c8c0 0x7f2a981cf290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:13:31.556 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.542+0000 7f2a9e99c640 10 monclient(hunting): picked mon.noname-a con 0x7f2a9813c8c0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:13:31.556 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.542+0000 7f2a9e99c640 10 monclient(hunting): start opening mon connection 2024-09-16T04:13:31.557 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.542+0000 7f2a9e99c640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f2a98142f80 con 0x7f2a9813c8c0 2024-09-16T04:13:31.557 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.542+0000 7f2a9e99c640 10 monclient(hunting): start opening mon connection 2024-09-16T04:13:31.557 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.542+0000 7f2a9e99c640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f2a98142e00 con 0x7f2a980a3d90 2024-09-16T04:13:31.557 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.542+0000 7f2a9e99c640 10 monclient(hunting): start opening mon connection 2024-09-16T04:13:31.557 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.542+0000 7f2a9e99c640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f2a98143100 con 0x7f2a980a39b0 2024-09-16T04:13:31.557 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.542+0000 7f2a9e99c640 10 monclient(hunting): _renew_subs 2024-09-16T04:13:31.557 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.542+0000 7f2a9e99c640 10 monclient(hunting): authenticate will time out at 2858.514757s 2024-09-16T04:13:31.557 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.542+0000 7f2a97fff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2a980a39b0 0x7f2a981391a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:13:31.557 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.542+0000 7f2a97fff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2a980a39b0 0x7f2a981391a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:49012/0 (socket says 172.21.15.42:49012) 2024-09-16T04:13:31.557 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.542+0000 7f2a97fff640 1 -- 172.21.15.42:0/3583789686 learned_addr learned my addr 172.21.15.42:0/3583789686 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:13:31.557 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.542+0000 7f2a97fff640 10 monclient(hunting): get_auth_request con 0x7f2a980a39b0 auth_method 0 2024-09-16T04:13:31.557 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.542+0000 7f2a97fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:13:31.557 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.542+0000 7f2a97fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:31.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.542+0000 7f2a97fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:13:31.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.542+0000 7f2a9cf39640 1 --2- 172.21.15.42:0/3583789686 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2a9813c8c0 0x7f2a981cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:13:31.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.542+0000 7f2a9cf39640 10 monclient(hunting): get_auth_request con 0x7f2a9813c8c0 auth_method 0 2024-09-16T04:13:31.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.542+0000 7f2a9cf39640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:13:31.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.542+0000 7f2a9cf39640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:31.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.542+0000 7f2a9cf39640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:13:31.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.542+0000 7f2a97fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:13:31.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.542+0000 7f2a97fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:13:31.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.542+0000 7f2a97fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:13:31.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.542+0000 7f2a9cf39640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:13:31.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.542+0000 7f2a9cf39640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:13:31.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.542+0000 7f2a9cf39640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:13:31.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.542+0000 7f2a8ffff640 1 --2- 172.21.15.42:0/3583789686 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2a980a3d90 0x7f2a981396e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:13:31.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.542+0000 7f2a8ffff640 10 monclient(hunting): get_auth_request con 0x7f2a980a3d90 auth_method 0 2024-09-16T04:13:31.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.542+0000 7f2a8ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:13:31.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.542+0000 7f2a8ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:31.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.542+0000 7f2a8ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:13:31.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.542+0000 7f2a97fff640 10 monclient(hunting): handle_auth_done global_id 1094120 payload 824 2024-09-16T04:13:31.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.542+0000 7f2a97fff640 1 -- 172.21.15.42:0/3583789686 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2a980a3d90 msgr2=0x7f2a981396e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:31.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.542+0000 7f2a97fff640 1 --2- 172.21.15.42:0/3583789686 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2a980a3d90 0x7f2a981396e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:13:31.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.542+0000 7f2a97fff640 1 -- 172.21.15.42:0/3583789686 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2a9813c8c0 msgr2=0x7f2a981cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:31.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.542+0000 7f2a97fff640 1 --2- 172.21.15.42:0/3583789686 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2a9813c8c0 0x7f2a981cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:13:31.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.542+0000 7f2a97fff640 10 monclient: _finish_hunting 0 2024-09-16T04:13:31.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.542+0000 7f2a97fff640 1 monclient: found mon.noname-c 2024-09-16T04:13:31.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.542+0000 7f2a97fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:13:31.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.542+0000 7f2a97fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T04:13:31.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.542+0000 7f2a97fff640 1 -- 172.21.15.42:0/3583789686 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a9813c170 con 0x7f2a980a39b0 2024-09-16T04:13:31.560 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.542+0000 7f2a97fff640 1 --2- 172.21.15.42:0/3583789686 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2a980a39b0 0x7f2a981391a0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f2a8000c3a0 tx=0x7f2a8000c860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:13:31.560 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.542+0000 7f2a957fa640 10 client.?.objecter ms_handle_connect 0x7f2a980a39b0 2024-09-16T04:13:31.560 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.542+0000 7f2a957fa640 10 client.?.objecter resend_mon_ops 2024-09-16T04:13:31.560 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.542+0000 7f2a957fa640 1 -- 172.21.15.42:0/3583789686 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a800052d0 con 0x7f2a980a39b0 2024-09-16T04:13:31.560 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.542+0000 7f2a957fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:13:31.560 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.542+0000 7f2a957fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:13:31.560 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.542+0000 7f2a957fa640 10 monclient: dump: 2024-09-16T04:13:31.560 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:13:31.560 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:13:31.560 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:31.560 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:31.560 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:31.560 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:13:31.560 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:13:31.561 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:13:31.561 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:13:31.561 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:13:31.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.542+0000 7f2a957fa640 10 monclient: _finish_auth 0 2024-09-16T04:13:31.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.542+0000 7f2a957fa640 10 monclient: _check_auth_tickets 2024-09-16T04:13:31.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.542+0000 7f2a957fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:13:31.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.542+0000 7f2a957fa640 1 -- 172.21.15.42:0/3583789686 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a80005470 con 0x7f2a980a39b0 2024-09-16T04:13:31.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.542+0000 7f2a9e99c640 5 monclient: authenticate success, global_id 1094120 2024-09-16T04:13:31.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.542+0000 7f2a9e99c640 10 monclient: _renew_subs 2024-09-16T04:13:31.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.542+0000 7f2a9e99c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:13:31.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.542+0000 7f2a9e99c640 1 -- 172.21.15.42:0/3583789686 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a9813fcd0 con 0x7f2a980a39b0 2024-09-16T04:13:31.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.542+0000 7f2a957fa640 10 monclient: handle_config config(0 keys) 2024-09-16T04:13:31.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.542+0000 7f2a957fa640 1 -- 172.21.15.42:0/3583789686 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a80007260 con 0x7f2a980a39b0 2024-09-16T04:13:31.562 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.542+0000 7f2a957fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:13:31.562 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.542+0000 7f2a957fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:13:31.562 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.542+0000 7f2a957fa640 10 monclient: dump: 2024-09-16T04:13:31.562 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:13:31.562 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:13:31.562 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:31.562 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:31.562 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:31.562 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:13:31.562 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:13:31.562 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:13:31.562 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:13:31.562 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:13:31.562 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.542+0000 7f2a96ffd640 4 set_mon_vals no callback set 2024-09-16T04:13:31.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.542+0000 7f2a9e99c640 10 client.1094120.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:13:31.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.542+0000 7f2a9e99c640 10 monclient: _renew_subs 2024-09-16T04:13:31.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.542+0000 7f2a9e99c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:13:31.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.542+0000 7f2a9e99c640 1 -- 172.21.15.42:0/3583789686 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a9812df60 con 0x7f2a980a39b0 2024-09-16T04:13:31.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.542+0000 7f2a9e99c640 1 librados: init done 2024-09-16T04:13:31.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.546+0000 7f2a9e99c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:13:31.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.546+0000 7f2a9e99c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:13:31.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.546+0000 7f2a9e99c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:13:31.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.546+0000 7f2a9e99c640 1 -- 172.21.15.42:0/3583789686 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2a98133b80 con 0x7f2a980a39b0 2024-09-16T04:13:31.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.546+0000 7f2a957fa640 1 -- 172.21.15.42:0/3583789686 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f2a80007400 con 0x7f2a980a39b0 2024-09-16T04:13:31.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.546+0000 7f2a957fa640 1 --2- 172.21.15.42:0/3583789686 >> v2:172.21.15.154:6812/254845342 conn(0x7f2a78037840 0x7f2a78039d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:13:31.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.546+0000 7f2a957fa640 1 -- 172.21.15.42:0/3583789686 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(706..706 src has 681..706) ==== 5040+0+0 (secure 0 0 0) 0x7f2a800481e0 con 0x7f2a980a39b0 2024-09-16T04:13:31.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.546+0000 7f2a957fa640 10 client.1094120.objecter ms_dispatch 0x7f2a98000ba0 osd_map(706..706 src has 681..706) 2024-09-16T04:13:31.563 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.546+0000 7f2a957fa640 3 client.1094120.objecter handle_osd_map got epochs [706,706] > 0 2024-09-16T04:13:31.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.546+0000 7f2a957fa640 3 client.1094120.objecter handle_osd_map decoding full epoch 706 2024-09-16T04:13:31.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.546+0000 7f2a957fa640 20 client.1094120.objecter dump_active .. 0 homeless 2024-09-16T04:13:31.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.546+0000 7f2a8ffff640 1 --2- 172.21.15.42:0/3583789686 >> v2:172.21.15.154:6812/254845342 conn(0x7f2a78037840 0x7f2a78039d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:13:31.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.546+0000 7f2a8ffff640 10 monclient: get_auth_request con 0x7f2a78037840 auth_method 0 2024-09-16T04:13:31.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.546+0000 7f2a957fa640 10 client.1094120.objecter ms_handle_connect 0x7f2a78037840 2024-09-16T04:13:31.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.546+0000 7f2a8ffff640 1 --2- 172.21.15.42:0/3583789686 >> v2:172.21.15.154:6812/254845342 conn(0x7f2a78037840 0x7f2a78039d00 secure :-1 s=READY pgs=3989 cs=0 l=1 rev1=1 crypto rx=0x7f2a880ad000 tx=0x7f2a88008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:13:31.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.550+0000 7f2a957fa640 1 -- 172.21.15.42:0/3583789686 <== mon.2 v2:172.21.15.42:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f2a800067e0 con 0x7f2a980a39b0 2024-09-16T04:13:31.564 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.550+0000 7f2a957fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:13:31.565 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.550+0000 7f2a957fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:13:31.782 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.778+0000 7f2a9e99c640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T04:13:31.782 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.778+0000 7f2a9e99c640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T04:13:31.782 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.778+0000 7f2a9e99c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:13:31.783 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.778+0000 7f2a9e99c640 1 -- 172.21.15.42:0/3583789686 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f2a980a5750 con 0x7f2a980a39b0 2024-09-16T04:13:31.785 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.782+0000 7f2a957fa640 1 -- 172.21.15.42:0/3583789686 <== mon.2 v2:172.21.15.42:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f2a80003600 con 0x7f2a980a39b0 2024-09-16T04:13:31.785 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.782+0000 7f2a957fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T04:13:31.785 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.782+0000 7f2a957fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:13:31.788 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.782+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.788 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.782+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=26,config=2,health=12,kv=8,logm=52} crc {auth=2783416142,config=692619927,health=3684352996,kv=2236749221,logm=1693804759}) 2024-09-16T04:13:31.788 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.782+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=28,config=2,health=12,kv=10,logm=48} crc {auth=2878102095,config=692619927,health=3684352996,kv=3446188498,logm=3936170917}) 2024-09-16T04:13:31.789 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.782+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.789 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.782+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=26,config=2,health=12,kv=8,logm=52} crc {auth=2783416142,config=692619927,health=3684352996,kv=2236749221,logm=1693804759}) 2024-09-16T04:13:31.789 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.782+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=25,config=2,health=11,kv=10,logm=52} crc {auth=3029484991,config=692619927,health=1585242637,kv=3446188498,logm=4188795548}) 2024-09-16T04:13:31.790 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.782+0000 7f2a9e99c640 10 librados: watch_flush enter 2024-09-16T04:13:31.790 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.782+0000 7f2a9e99c640 10 librados: watch_flush exit 2024-09-16T04:13:31.790 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.782+0000 7f2a9e99c640 20 client.1094120.objecter shutdown clearing up homeless session... 2024-09-16T04:13:31.790 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.782+0000 7f2a9e99c640 10 client.1094120.objecter successfully canceled tick 2024-09-16T04:13:31.790 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.782+0000 7f2a9e99c640 1 -- 172.21.15.42:0/3583789686 >> v2:172.21.15.154:6812/254845342 conn(0x7f2a78037840 msgr2=0x7f2a78039d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:31.790 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.782+0000 7f2a9e99c640 1 --2- 172.21.15.42:0/3583789686 >> v2:172.21.15.154:6812/254845342 conn(0x7f2a78037840 0x7f2a78039d00 secure :-1 s=READY pgs=3989 cs=0 l=1 rev1=1 crypto rx=0x7f2a880ad000 tx=0x7f2a88008040 comp rx=0 tx=0).stop 2024-09-16T04:13:31.790 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.782+0000 7f2a9e99c640 10 monclient: shutdown 2024-09-16T04:13:31.790 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.782+0000 7f2a9e99c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:13:31.790 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.782+0000 7f2a9e99c640 1 -- 172.21.15.42:0/3583789686 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2a980a39b0 msgr2=0x7f2a981391a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:31.790 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.782+0000 7f2a9e99c640 1 --2- 172.21.15.42:0/3583789686 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2a980a39b0 0x7f2a981391a0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f2a8000c3a0 tx=0x7f2a8000c860 comp rx=0 tx=0).stop 2024-09-16T04:13:31.791 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.782+0000 7f2a9e99c640 1 -- 172.21.15.42:0/3583789686 shutdown_connections 2024-09-16T04:13:31.791 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.782+0000 7f2a9e99c640 1 --2- 172.21.15.42:0/3583789686 >> v2:172.21.15.154:6812/254845342 conn(0x7f2a78037840 0x7f2a78039d00 unknown :-1 s=CLOSED pgs=3989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:13:31.791 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.782+0000 7f2a9e99c640 1 --2- 172.21.15.42:0/3583789686 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2a9813c8c0 0x7f2a981cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:13:31.791 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.782+0000 7f2a9e99c640 1 --2- 172.21.15.42:0/3583789686 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2a980a3d90 0x7f2a981396e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:13:31.791 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.782+0000 7f2a9e99c640 1 --2- 172.21.15.42:0/3583789686 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2a980a39b0 0x7f2a981391a0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:13:31.791 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.782+0000 7f2a9e99c640 1 -- 172.21.15.42:0/3583789686 >> 172.21.15.42:0/3583789686 conn(0x7f2a980a9fb0 msgr2=0x7f2a9813ecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:13:31.791 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.782+0000 7f2a9e99c640 1 -- 172.21.15.42:0/3583789686 shutdown_connections 2024-09-16T04:13:31.791 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.782+0000 7f2a9e99c640 1 -- 172.21.15.42:0/3583789686 wait complete. 2024-09-16T04:13:31.791 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:31.782+0000 7f2a9e99c640 1 librados: shutdown 2024-09-16T04:13:31.792 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.782+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.792 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.782+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=250128795}) 2024-09-16T04:13:31.792 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.782+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2728185276}) 2024-09-16T04:13:31.792 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.782+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.792 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.782+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=250128795}) 2024-09-16T04:13:31.792 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.782+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=24710633}) 2024-09-16T04:13:31.792 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.786+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.793 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.786+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3307475039}) 2024-09-16T04:13:31.793 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.786+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3119137603}) 2024-09-16T04:13:31.793 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.786+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.793 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.786+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3307475039}) 2024-09-16T04:13:31.793 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.786+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=919385546}) 2024-09-16T04:13:31.794 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.790+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.794 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.790+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3375958955}) 2024-09-16T04:13:31.794 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.790+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4274606904}) 2024-09-16T04:13:31.794 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.790+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.794 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.790+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3375958955}) 2024-09-16T04:13:31.794 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.790+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1115417907}) 2024-09-16T04:13:31.796 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.790+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.797 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.790+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1598802156}) 2024-09-16T04:13:31.797 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.790+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4182579268}) 2024-09-16T04:13:31.797 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.790+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.797 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.790+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1598802156}) 2024-09-16T04:13:31.797 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.790+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1337922010}) 2024-09-16T04:13:31.798 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.794+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.799 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.794+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=431215678}) 2024-09-16T04:13:31.799 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.794+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3192270668}) 2024-09-16T04:13:31.799 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.794+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.799 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.794+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=431215678}) 2024-09-16T04:13:31.799 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.794+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3205039746}) 2024-09-16T04:13:31.800 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.794+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.801 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.794+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2255505948}) 2024-09-16T04:13:31.801 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.794+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1832743603}) 2024-09-16T04:13:31.801 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.794+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.801 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.794+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2255505948}) 2024-09-16T04:13:31.801 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.794+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=153309973}) 2024-09-16T04:13:31.802 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.798+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.802 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.798+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2828932469}) 2024-09-16T04:13:31.802 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.798+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2184933918}) 2024-09-16T04:13:31.802 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.798+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.802 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.798+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2828932469}) 2024-09-16T04:13:31.802 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.798+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2049317769}) 2024-09-16T04:13:31.804 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.798+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.804 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.798+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2918363791}) 2024-09-16T04:13:31.804 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.798+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2958034311}) 2024-09-16T04:13:31.804 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.798+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.804 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.798+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2918363791}) 2024-09-16T04:13:31.804 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.798+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=820355873}) 2024-09-16T04:13:31.806 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.802+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.806 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.802+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2087551818}) 2024-09-16T04:13:31.806 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.802+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1695071603}) 2024-09-16T04:13:31.806 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.802+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.806 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.802+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2087551818}) 2024-09-16T04:13:31.806 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.802+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2594209786}) 2024-09-16T04:13:31.807 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.802+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.808 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.802+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2878327704}) 2024-09-16T04:13:31.808 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.802+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=726614773}) 2024-09-16T04:13:31.808 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.802+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.808 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.802+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2878327704}) 2024-09-16T04:13:31.808 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.802+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4052667959}) 2024-09-16T04:13:31.809 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T04:13:31.810 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.806+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.810 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.806+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=291560516}) 2024-09-16T04:13:31.810 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.806+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=344357174}) 2024-09-16T04:13:31.810 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.806+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.810 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.806+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=291560516}) 2024-09-16T04:13:31.810 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.806+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3198913676}) 2024-09-16T04:13:31.811 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.806+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.811 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.806+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3637426390}) 2024-09-16T04:13:31.811 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.806+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=797901973}) 2024-09-16T04:13:31.811 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.806+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.811 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.806+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3637426390}) 2024-09-16T04:13:31.811 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.806+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=563171961}) 2024-09-16T04:13:31.813 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.806+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.813 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.806+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1163241482}) 2024-09-16T04:13:31.813 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.806+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1117189330}) 2024-09-16T04:13:31.813 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.806+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.813 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.806+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1163241482}) 2024-09-16T04:13:31.813 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.806+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4028006594}) 2024-09-16T04:13:31.814 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.810+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.814 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.810+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2938595219}) 2024-09-16T04:13:31.815 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.810+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4190972305}) 2024-09-16T04:13:31.815 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.810+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.815 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.810+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2938595219}) 2024-09-16T04:13:31.815 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.810+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3223468787}) 2024-09-16T04:13:31.816 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.810+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.816 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.810+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3330725007}) 2024-09-16T04:13:31.816 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.810+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2723946113}) 2024-09-16T04:13:31.817 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.810+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.817 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.810+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3330725007}) 2024-09-16T04:13:31.817 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.810+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3657732243}) 2024-09-16T04:13:31.818 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.814+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.818 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.814+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1408027915}) 2024-09-16T04:13:31.818 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.814+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2792876955}) 2024-09-16T04:13:31.818 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.814+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.818 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.814+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1408027915}) 2024-09-16T04:13:31.818 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.814+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2915446262}) 2024-09-16T04:13:31.820 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.814+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.820 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.814+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2780343548}) 2024-09-16T04:13:31.820 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.814+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1068845470}) 2024-09-16T04:13:31.820 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.814+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.820 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.814+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2780343548}) 2024-09-16T04:13:31.820 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.814+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2744140578}) 2024-09-16T04:13:31.822 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.818+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.822 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.818+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2323966511}) 2024-09-16T04:13:31.822 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.818+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=289884426}) 2024-09-16T04:13:31.822 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.818+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.822 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.818+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2323966511}) 2024-09-16T04:13:31.822 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.818+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4014712102}) 2024-09-16T04:13:31.824 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.818+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.824 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.818+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=925873528}) 2024-09-16T04:13:31.824 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.818+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3698817236}) 2024-09-16T04:13:31.825 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.818+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.825 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.818+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=925873528}) 2024-09-16T04:13:31.825 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.818+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1830638175}) 2024-09-16T04:13:31.826 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.822+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.826 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.822+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1587161495}) 2024-09-16T04:13:31.826 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.822+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3879989168}) 2024-09-16T04:13:31.826 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.822+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.826 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.822+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1587161495}) 2024-09-16T04:13:31.826 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.822+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=126964061}) 2024-09-16T04:13:31.828 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.822+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.828 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.822+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=460650272}) 2024-09-16T04:13:31.828 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.822+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2969479101}) 2024-09-16T04:13:31.828 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.822+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.828 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.822+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=460650272}) 2024-09-16T04:13:31.828 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.822+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1757336350}) 2024-09-16T04:13:31.830 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.826+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.830 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.826+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=441619785}) 2024-09-16T04:13:31.830 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.826+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=525904926}) 2024-09-16T04:13:31.831 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.826+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.831 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.826+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=441619785}) 2024-09-16T04:13:31.831 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.826+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1902395317}) 2024-09-16T04:13:31.832 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.826+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.832 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.826+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3391026604}) 2024-09-16T04:13:31.832 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.826+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1719078653}) 2024-09-16T04:13:31.832 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.826+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.832 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.826+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3391026604}) 2024-09-16T04:13:31.832 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.826+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2140829622}) 2024-09-16T04:13:31.833 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.830+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.834 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.830+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2980237891}) 2024-09-16T04:13:31.834 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.830+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1365933629}) 2024-09-16T04:13:31.834 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.830+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.834 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.830+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2980237891}) 2024-09-16T04:13:31.834 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.830+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=945325267}) 2024-09-16T04:13:31.835 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.830+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.835 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.830+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1693873395}) 2024-09-16T04:13:31.836 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.830+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1010590113}) 2024-09-16T04:13:31.836 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.830+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.836 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.830+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1693873395}) 2024-09-16T04:13:31.836 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.830+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3295033708}) 2024-09-16T04:13:31.837 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.834+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.837 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.834+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2572923837}) 2024-09-16T04:13:31.838 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.834+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2116560436}) 2024-09-16T04:13:31.838 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.834+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.838 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.834+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2572923837}) 2024-09-16T04:13:31.838 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.834+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=970945487}) 2024-09-16T04:13:31.839 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.834+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.839 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.834+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4061844494}) 2024-09-16T04:13:31.840 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.834+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2410729692}) 2024-09-16T04:13:31.840 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.834+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.840 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.834+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4061844494}) 2024-09-16T04:13:31.840 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.834+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=532763247}) 2024-09-16T04:13:31.842 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.838+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.842 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.838+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1534792709}) 2024-09-16T04:13:31.842 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.838+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=757211320}) 2024-09-16T04:13:31.842 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.838+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.842 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.838+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1534792709}) 2024-09-16T04:13:31.842 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.838+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1672961901}) 2024-09-16T04:13:31.843 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.838+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.844 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.838+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3466462518}) 2024-09-16T04:13:31.844 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.838+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1619909953}) 2024-09-16T04:13:31.844 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.838+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.844 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.838+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3466462518}) 2024-09-16T04:13:31.844 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.838+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=402975670}) 2024-09-16T04:13:31.845 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.842+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.845 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.842+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=189686395}) 2024-09-16T04:13:31.845 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.842+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4084446695}) 2024-09-16T04:13:31.845 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.842+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.846 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.842+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=189686395}) 2024-09-16T04:13:31.846 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.842+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1854563068}) 2024-09-16T04:13:31.847 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.842+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.847 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.842+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2171237833}) 2024-09-16T04:13:31.847 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.842+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4267375736}) 2024-09-16T04:13:31.847 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.842+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.848 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.842+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2171237833}) 2024-09-16T04:13:31.848 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.842+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=973034866}) 2024-09-16T04:13:31.849 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.846+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.849 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.846+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2416771912}) 2024-09-16T04:13:31.849 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.846+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1558452169}) 2024-09-16T04:13:31.849 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.846+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.849 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.846+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2416771912}) 2024-09-16T04:13:31.850 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.846+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2721690640}) 2024-09-16T04:13:31.851 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.846+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.852 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.846+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2970785145}) 2024-09-16T04:13:31.852 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.846+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3656542129}) 2024-09-16T04:13:31.852 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.846+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.852 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.846+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2970785145}) 2024-09-16T04:13:31.852 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.846+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2022295675}) 2024-09-16T04:13:31.854 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.850+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.854 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.850+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1348846853}) 2024-09-16T04:13:31.854 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.850+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2385522991}) 2024-09-16T04:13:31.854 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.850+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.854 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.850+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1348846853}) 2024-09-16T04:13:31.854 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.850+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2153282729}) 2024-09-16T04:13:31.858 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.850+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.858 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.850+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=697314322}) 2024-09-16T04:13:31.858 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.850+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1920261392}) 2024-09-16T04:13:31.858 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.850+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.858 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.850+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=697314322}) 2024-09-16T04:13:31.859 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.850+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=493385378}) 2024-09-16T04:13:31.859 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.854+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.859 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.854+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1324451143}) 2024-09-16T04:13:31.859 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.854+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2908306415}) 2024-09-16T04:13:31.859 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.854+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.859 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.854+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1324451143}) 2024-09-16T04:13:31.859 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.854+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4030451198}) 2024-09-16T04:13:31.861 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.854+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.862 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.854+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=884899762}) 2024-09-16T04:13:31.862 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.854+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2257905726}) 2024-09-16T04:13:31.862 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.854+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.862 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.854+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=884899762}) 2024-09-16T04:13:31.862 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.854+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=35476797}) 2024-09-16T04:13:31.863 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.854+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.863 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.854+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3205896821}) 2024-09-16T04:13:31.863 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.854+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1782713944}) 2024-09-16T04:13:31.863 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.858+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.863 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.858+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3205896821}) 2024-09-16T04:13:31.863 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.858+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1472502684}) 2024-09-16T04:13:31.866 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.858+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.866 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.858+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=678690029}) 2024-09-16T04:13:31.866 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.858+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2030837170}) 2024-09-16T04:13:31.866 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.858+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.867 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.858+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=678690029}) 2024-09-16T04:13:31.867 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.858+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1429111362}) 2024-09-16T04:13:31.867 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.858+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.867 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.858+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2103580123}) 2024-09-16T04:13:31.867 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.858+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2291387506}) 2024-09-16T04:13:31.867 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.858+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.868 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.858+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2103580123}) 2024-09-16T04:13:31.868 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.858+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=346694717}) 2024-09-16T04:13:31.870 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.862+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.870 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.862+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=556120427}) 2024-09-16T04:13:31.870 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.862+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=633634039}) 2024-09-16T04:13:31.870 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.862+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.870 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.862+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=556120427}) 2024-09-16T04:13:31.870 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.862+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1442761172}) 2024-09-16T04:13:31.871 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.866+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.871 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.866+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2169029794}) 2024-09-16T04:13:31.871 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.866+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3875455681}) 2024-09-16T04:13:31.871 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.866+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.871 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.866+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2169029794}) 2024-09-16T04:13:31.872 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.866+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1012347411}) 2024-09-16T04:13:31.873 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.866+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.873 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.866+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1859043269}) 2024-09-16T04:13:31.873 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.866+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4155602619}) 2024-09-16T04:13:31.873 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.866+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.873 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.866+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1859043269}) 2024-09-16T04:13:31.873 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.866+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2525493371}) 2024-09-16T04:13:31.875 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.870+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.875 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.870+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2372517511}) 2024-09-16T04:13:31.875 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.870+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2277507499}) 2024-09-16T04:13:31.875 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.870+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.876 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.870+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2372517511}) 2024-09-16T04:13:31.876 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.870+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1848994683}) 2024-09-16T04:13:31.878 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.870+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.878 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.870+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4007918450}) 2024-09-16T04:13:31.879 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.870+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2157364605}) 2024-09-16T04:13:31.879 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.870+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.879 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.870+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4007918450}) 2024-09-16T04:13:31.879 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.870+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3708910461}) 2024-09-16T04:13:31.879 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.874+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.879 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.874+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4102027542}) 2024-09-16T04:13:31.880 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.874+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2195575946}) 2024-09-16T04:13:31.880 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.874+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.880 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.874+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4102027542}) 2024-09-16T04:13:31.880 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.874+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4269173704}) 2024-09-16T04:13:31.881 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.874+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.881 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.874+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1982848480}) 2024-09-16T04:13:31.881 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.874+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2035173780}) 2024-09-16T04:13:31.882 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.874+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.882 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.874+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1982848480}) 2024-09-16T04:13:31.882 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.874+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1100531760}) 2024-09-16T04:13:31.882 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.878+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.883 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.878+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4190779789}) 2024-09-16T04:13:31.883 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.878+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=982263360}) 2024-09-16T04:13:31.883 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.878+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.883 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.878+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4190779789}) 2024-09-16T04:13:31.883 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.878+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4081163115}) 2024-09-16T04:13:31.884 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.878+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.884 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.878+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=503467625}) 2024-09-16T04:13:31.884 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.878+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=736548430}) 2024-09-16T04:13:31.884 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.878+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.885 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.878+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=503467625}) 2024-09-16T04:13:31.885 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.878+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=340272321}) 2024-09-16T04:13:31.886 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.882+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.887 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.882+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3537949791}) 2024-09-16T04:13:31.887 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.882+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3653762658}) 2024-09-16T04:13:31.887 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.882+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.887 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.882+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3537949791}) 2024-09-16T04:13:31.887 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.882+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1462437752}) 2024-09-16T04:13:31.890 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.882+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.890 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.882+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4263217724}) 2024-09-16T04:13:31.890 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.882+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2578566806}) 2024-09-16T04:13:31.890 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.882+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.890 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.882+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4263217724}) 2024-09-16T04:13:31.890 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.882+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1292938081}) 2024-09-16T04:13:31.891 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.886+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.891 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.886+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2467833820}) 2024-09-16T04:13:31.891 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.886+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3659264002}) 2024-09-16T04:13:31.891 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.886+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.891 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.886+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2467833820}) 2024-09-16T04:13:31.891 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.886+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2647342200}) 2024-09-16T04:13:31.893 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.886+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.893 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.886+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=487159846}) 2024-09-16T04:13:31.893 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.886+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=45302559}) 2024-09-16T04:13:31.893 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.886+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.893 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.886+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=487159846}) 2024-09-16T04:13:31.893 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.886+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2740929785}) 2024-09-16T04:13:31.895 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.890+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.895 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.890+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2224358570}) 2024-09-16T04:13:31.895 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.890+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2890627437}) 2024-09-16T04:13:31.895 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.890+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.895 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.890+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2224358570}) 2024-09-16T04:13:31.896 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.890+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1212315873}) 2024-09-16T04:13:31.897 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.890+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.898 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.890+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3776845216}) 2024-09-16T04:13:31.898 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.890+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=699648370}) 2024-09-16T04:13:31.898 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.890+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.898 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.890+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3776845216}) 2024-09-16T04:13:31.898 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.890+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=546519383}) 2024-09-16T04:13:31.900 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.894+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.900 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.894+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=314017083}) 2024-09-16T04:13:31.900 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.894+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3067502724}) 2024-09-16T04:13:31.900 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.894+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.900 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.894+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=314017083}) 2024-09-16T04:13:31.900 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.894+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3564025760}) 2024-09-16T04:13:31.902 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.898+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.902 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.898+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3264183824}) 2024-09-16T04:13:31.902 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.898+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3936890954}) 2024-09-16T04:13:31.902 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.898+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.902 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.898+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3264183824}) 2024-09-16T04:13:31.902 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.898+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3230239167}) 2024-09-16T04:13:31.903 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.898+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.904 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.898+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2171619125}) 2024-09-16T04:13:31.904 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.898+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4104376695}) 2024-09-16T04:13:31.904 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.898+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.904 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.898+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2171619125}) 2024-09-16T04:13:31.904 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.898+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3439407444}) 2024-09-16T04:13:31.906 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.902+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.906 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.902+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2472939002}) 2024-09-16T04:13:31.906 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.902+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=695142515}) 2024-09-16T04:13:31.906 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.902+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.906 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.902+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2472939002}) 2024-09-16T04:13:31.907 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.902+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1380851377}) 2024-09-16T04:13:31.908 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.902+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.908 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.902+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1898559258}) 2024-09-16T04:13:31.908 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.902+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2009068187}) 2024-09-16T04:13:31.908 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.902+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.909 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.902+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1898559258}) 2024-09-16T04:13:31.909 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.902+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2952237428}) 2024-09-16T04:13:31.910 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.906+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.911 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.906+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=491960819}) 2024-09-16T04:13:31.911 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.906+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2133579343}) 2024-09-16T04:13:31.911 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.906+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.911 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.906+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=491960819}) 2024-09-16T04:13:31.911 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.906+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2367075834}) 2024-09-16T04:13:31.912 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.906+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.912 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.906+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=793392529}) 2024-09-16T04:13:31.913 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.906+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4022035024}) 2024-09-16T04:13:31.913 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.906+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.913 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.906+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=793392529}) 2024-09-16T04:13:31.913 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.906+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3698642909}) 2024-09-16T04:13:31.915 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.910+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.915 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.910+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=939104732}) 2024-09-16T04:13:31.915 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.910+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=817920823}) 2024-09-16T04:13:31.915 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.910+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.915 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.910+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=939104732}) 2024-09-16T04:13:31.915 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.910+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2430856081}) 2024-09-16T04:13:31.917 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.910+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.917 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.910+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=853180693}) 2024-09-16T04:13:31.917 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.910+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2248779330}) 2024-09-16T04:13:31.917 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.910+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.917 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.910+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=853180693}) 2024-09-16T04:13:31.917 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.910+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=358294136}) 2024-09-16T04:13:31.919 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.914+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.919 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.914+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2479303772}) 2024-09-16T04:13:31.919 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.914+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4166285848}) 2024-09-16T04:13:31.919 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.914+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.919 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.914+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2479303772}) 2024-09-16T04:13:31.919 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.914+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2576080956}) 2024-09-16T04:13:31.921 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.914+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.921 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.914+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=329089875}) 2024-09-16T04:13:31.921 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.914+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2299380761}) 2024-09-16T04:13:31.921 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.914+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.921 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.914+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=329089875}) 2024-09-16T04:13:31.921 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.914+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=750247844}) 2024-09-16T04:13:31.923 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.918+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.923 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.918+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=99334867}) 2024-09-16T04:13:31.923 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.918+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3467405022}) 2024-09-16T04:13:31.923 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.918+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.923 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.918+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=99334867}) 2024-09-16T04:13:31.923 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.918+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2865079013}) 2024-09-16T04:13:31.925 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.918+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.925 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.918+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1080359493}) 2024-09-16T04:13:31.925 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.918+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1949398741}) 2024-09-16T04:13:31.925 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.918+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.925 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.918+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1080359493}) 2024-09-16T04:13:31.925 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.918+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2627669092}) 2024-09-16T04:13:31.927 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.922+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.927 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.922+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2456680714}) 2024-09-16T04:13:31.927 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.922+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1106703334}) 2024-09-16T04:13:31.927 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.922+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.927 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.922+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2456680714}) 2024-09-16T04:13:31.927 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.922+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1150400915}) 2024-09-16T04:13:31.929 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.922+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.929 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.922+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2298678394}) 2024-09-16T04:13:31.929 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.922+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1818795194}) 2024-09-16T04:13:31.929 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.922+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.929 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.922+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2298678394}) 2024-09-16T04:13:31.929 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.926+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4117317512}) 2024-09-16T04:13:31.931 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.926+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.931 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.926+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3020941373}) 2024-09-16T04:13:31.931 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.926+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2533807448}) 2024-09-16T04:13:31.931 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.926+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.931 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.926+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3020941373}) 2024-09-16T04:13:31.931 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.926+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=591304714}) 2024-09-16T04:13:31.933 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.930+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.933 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.930+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=856157567}) 2024-09-16T04:13:31.933 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.930+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2213768829}) 2024-09-16T04:13:31.933 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.930+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.934 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.930+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=856157567}) 2024-09-16T04:13:31.934 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.930+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2004912797}) 2024-09-16T04:13:31.935 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.930+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.936 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.930+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2748131567}) 2024-09-16T04:13:31.936 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.930+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=326885684}) 2024-09-16T04:13:31.936 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.930+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.936 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.930+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2748131567}) 2024-09-16T04:13:31.936 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.930+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3408680816}) 2024-09-16T04:13:31.937 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.934+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.938 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.934+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3131911282}) 2024-09-16T04:13:31.938 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.934+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1757925717}) 2024-09-16T04:13:31.938 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.934+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.938 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.934+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3131911282}) 2024-09-16T04:13:31.938 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.934+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=414587618}) 2024-09-16T04:13:31.940 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.934+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.940 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.934+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=86965691}) 2024-09-16T04:13:31.940 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.934+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=399990476}) 2024-09-16T04:13:31.940 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.934+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.940 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.934+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=86965691}) 2024-09-16T04:13:31.940 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.934+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3143600698}) 2024-09-16T04:13:31.942 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.938+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.942 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.938+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2301474926}) 2024-09-16T04:13:31.943 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.938+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=128661988}) 2024-09-16T04:13:31.943 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.938+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.943 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.938+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2301474926}) 2024-09-16T04:13:31.943 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.938+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3466818694}) 2024-09-16T04:13:31.945 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.938+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.945 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.938+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1561785294}) 2024-09-16T04:13:31.945 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.938+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1066478048}) 2024-09-16T04:13:31.945 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.938+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.945 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.938+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1561785294}) 2024-09-16T04:13:31.946 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.938+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4128632296}) 2024-09-16T04:13:31.949 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.942+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.949 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.942+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1280187680}) 2024-09-16T04:13:31.949 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.942+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2319837550}) 2024-09-16T04:13:31.949 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.942+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.949 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.942+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1280187680}) 2024-09-16T04:13:31.949 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.942+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1092357676}) 2024-09-16T04:13:31.950 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.942+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.950 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.942+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3120941039}) 2024-09-16T04:13:31.950 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.942+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2526329622}) 2024-09-16T04:13:31.950 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.942+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.950 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.942+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3120941039}) 2024-09-16T04:13:31.950 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.942+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3713433851}) 2024-09-16T04:13:31.953 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.946+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.953 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.946+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3193809892}) 2024-09-16T04:13:31.953 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.946+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1252961323}) 2024-09-16T04:13:31.953 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.946+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.954 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.946+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3193809892}) 2024-09-16T04:13:31.954 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.946+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1901348223}) 2024-09-16T04:13:31.954 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.946+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.954 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.946+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3646032884}) 2024-09-16T04:13:31.954 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.946+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1893679982}) 2024-09-16T04:13:31.954 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.946+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.955 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.946+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3646032884}) 2024-09-16T04:13:31.955 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.946+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=702191681}) 2024-09-16T04:13:31.957 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.950+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.957 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.950+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3373039489}) 2024-09-16T04:13:31.958 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.950+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=587931679}) 2024-09-16T04:13:31.958 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.950+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.958 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.950+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3373039489}) 2024-09-16T04:13:31.958 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.950+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2895699443}) 2024-09-16T04:13:31.958 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.950+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.958 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.950+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1068937092}) 2024-09-16T04:13:31.959 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.950+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1939802467}) 2024-09-16T04:13:31.959 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.950+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.959 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.950+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1068937092}) 2024-09-16T04:13:31.959 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.950+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1435846937}) 2024-09-16T04:13:31.960 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.954+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.960 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.954+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=313126080}) 2024-09-16T04:13:31.960 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.954+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=64079840}) 2024-09-16T04:13:31.961 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.954+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.961 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.954+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=313126080}) 2024-09-16T04:13:31.961 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.954+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4096433024}) 2024-09-16T04:13:31.962 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.954+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.962 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.954+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2138645409}) 2024-09-16T04:13:31.962 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.954+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=306847961}) 2024-09-16T04:13:31.963 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.954+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.963 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.954+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2138645409}) 2024-09-16T04:13:31.963 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.954+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=421427063}) 2024-09-16T04:13:31.965 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.958+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.965 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.958+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2455281036}) 2024-09-16T04:13:31.966 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.958+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3766046454}) 2024-09-16T04:13:31.966 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.958+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.966 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.958+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2455281036}) 2024-09-16T04:13:31.966 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.958+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1115224858}) 2024-09-16T04:13:31.966 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.958+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.967 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.958+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2000580798}) 2024-09-16T04:13:31.967 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.958+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2168488816}) 2024-09-16T04:13:31.967 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.958+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.967 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.958+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2000580798}) 2024-09-16T04:13:31.967 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.958+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1666522537}) 2024-09-16T04:13:31.968 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.962+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.968 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.962+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3145426245}) 2024-09-16T04:13:31.969 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.962+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1434007629}) 2024-09-16T04:13:31.969 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.962+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.969 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.962+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3145426245}) 2024-09-16T04:13:31.969 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.962+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1245831748}) 2024-09-16T04:13:31.970 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.966+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.970 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.966+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2946938390}) 2024-09-16T04:13:31.970 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.966+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2376274688}) 2024-09-16T04:13:31.970 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.966+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.971 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.966+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2946938390}) 2024-09-16T04:13:31.971 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.966+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2980379362}) 2024-09-16T04:13:31.972 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.966+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.972 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.966+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1735745086}) 2024-09-16T04:13:31.972 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.966+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3134705708}) 2024-09-16T04:13:31.972 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.966+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.972 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.966+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1735745086}) 2024-09-16T04:13:31.972 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.966+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=456233850}) 2024-09-16T04:13:31.973 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.966+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.973 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.966+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2620648128}) 2024-09-16T04:13:31.973 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.966+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1923152970}) 2024-09-16T04:13:31.973 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.970+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.973 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.970+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2620648128}) 2024-09-16T04:13:31.974 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.970+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2807838101}) 2024-09-16T04:13:31.975 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.970+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.975 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.970+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=374146107}) 2024-09-16T04:13:31.975 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.970+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1287591138}) 2024-09-16T04:13:31.975 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.970+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.975 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.970+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=374146107}) 2024-09-16T04:13:31.975 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.970+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3964453109}) 2024-09-16T04:13:31.977 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.970+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.977 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.970+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1663981808}) 2024-09-16T04:13:31.977 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.970+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1759970087}) 2024-09-16T04:13:31.977 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.970+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.977 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.970+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1663981808}) 2024-09-16T04:13:31.977 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.970+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=43484451}) 2024-09-16T04:13:31.979 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.974+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.979 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.974+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3783733064}) 2024-09-16T04:13:31.979 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.974+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3993864999}) 2024-09-16T04:13:31.979 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.974+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.979 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.974+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3783733064}) 2024-09-16T04:13:31.979 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.974+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1456741025}) 2024-09-16T04:13:31.980 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.974+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.980 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.974+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3376090089}) 2024-09-16T04:13:31.981 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.974+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2552517951}) 2024-09-16T04:13:31.981 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.974+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.981 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.974+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3376090089}) 2024-09-16T04:13:31.981 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.974+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3637125207}) 2024-09-16T04:13:31.982 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.978+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.982 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.978+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1914199595}) 2024-09-16T04:13:31.982 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.978+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3245411274}) 2024-09-16T04:13:31.983 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.978+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.983 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.978+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1914199595}) 2024-09-16T04:13:31.983 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.978+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4110049815}) 2024-09-16T04:13:31.984 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.978+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.984 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.978+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2479723766}) 2024-09-16T04:13:31.985 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.978+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2513645211}) 2024-09-16T04:13:31.985 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.978+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.985 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.978+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2479723766}) 2024-09-16T04:13:31.985 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.978+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4217408814}) 2024-09-16T04:13:31.986 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.982+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.986 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.982+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2654523923}) 2024-09-16T04:13:31.987 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.982+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4082512327}) 2024-09-16T04:13:31.987 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.982+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.987 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.982+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2654523923}) 2024-09-16T04:13:31.987 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.982+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2516421364}) 2024-09-16T04:13:31.989 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.982+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.989 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.982+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1195656572}) 2024-09-16T04:13:31.989 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.982+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=18555904}) 2024-09-16T04:13:31.992 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.982+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.993 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.982+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1195656572}) 2024-09-16T04:13:31.993 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.982+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2811797826}) 2024-09-16T04:13:31.994 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.986+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.994 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.986+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=779455961}) 2024-09-16T04:13:31.994 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.986+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2465651967}) 2024-09-16T04:13:31.994 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.986+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.994 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.986+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=779455961}) 2024-09-16T04:13:31.994 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.986+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2498277015}) 2024-09-16T04:13:31.994 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.986+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.994 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.986+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4027560989}) 2024-09-16T04:13:31.994 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.986+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1971929046}) 2024-09-16T04:13:31.994 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.986+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.994 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.986+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4027560989}) 2024-09-16T04:13:31.994 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.986+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2001278534}) 2024-09-16T04:13:31.995 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.990+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.995 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.990+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1906350364}) 2024-09-16T04:13:31.995 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.990+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=841514303}) 2024-09-16T04:13:31.995 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.990+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.996 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.990+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1906350364}) 2024-09-16T04:13:31.996 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.990+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1287096215}) 2024-09-16T04:13:31.997 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.990+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.997 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.990+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4187759053}) 2024-09-16T04:13:31.997 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.990+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=322836530}) 2024-09-16T04:13:31.997 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.990+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.997 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.990+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4187759053}) 2024-09-16T04:13:31.997 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.990+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3440129676}) 2024-09-16T04:13:31.998 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.994+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.998 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.994+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=37775503}) 2024-09-16T04:13:31.999 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.994+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2442499290}) 2024-09-16T04:13:31.999 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.994+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:31.999 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.994+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=37775503}) 2024-09-16T04:13:31.999 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.994+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3387343934}) 2024-09-16T04:13:32.000 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.994+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:32.000 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.994+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1981264956}) 2024-09-16T04:13:32.000 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.994+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2853502291}) 2024-09-16T04:13:32.000 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.994+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:32.000 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.994+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1981264956}) 2024-09-16T04:13:32.000 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.994+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3697349055}) 2024-09-16T04:13:32.002 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.998+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:32.002 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.998+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3754942319}) 2024-09-16T04:13:32.002 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.998+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1096326019}) 2024-09-16T04:13:32.002 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.998+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:32.002 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.998+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3754942319}) 2024-09-16T04:13:32.002 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.998+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4010329726}) 2024-09-16T04:13:32.004 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.998+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:32.004 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.998+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2666555024}) 2024-09-16T04:13:32.004 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.998+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4249508407}) 2024-09-16T04:13:32.004 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.998+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:32.004 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.998+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2666555024}) 2024-09-16T04:13:32.005 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:31.998+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1754072351}) 2024-09-16T04:13:32.006 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.002+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:32.006 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.002+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1281596537}) 2024-09-16T04:13:32.006 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.002+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1922373916}) 2024-09-16T04:13:32.006 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.002+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:32.006 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.002+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1281596537}) 2024-09-16T04:13:32.006 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.002+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3568529139}) 2024-09-16T04:13:32.008 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.002+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:32.008 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.002+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2134945813}) 2024-09-16T04:13:32.008 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.002+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3373229773}) 2024-09-16T04:13:32.008 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.002+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:32.008 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.002+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2134945813}) 2024-09-16T04:13:32.008 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.002+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=656800783}) 2024-09-16T04:13:32.010 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.006+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:32.010 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.006+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1708543987}) 2024-09-16T04:13:32.010 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.006+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3631020088}) 2024-09-16T04:13:32.010 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.006+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:32.010 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.006+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1708543987}) 2024-09-16T04:13:32.010 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.006+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1080178346}) 2024-09-16T04:13:32.012 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.006+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:32.012 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.006+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1140832067}) 2024-09-16T04:13:32.012 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.006+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4019695812}) 2024-09-16T04:13:32.012 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.006+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:32.012 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.006+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1140832067}) 2024-09-16T04:13:32.012 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.006+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1639282893}) 2024-09-16T04:13:32.014 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.010+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:32.014 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.010+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3423710894}) 2024-09-16T04:13:32.014 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.010+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3521400480}) 2024-09-16T04:13:32.014 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.010+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:32.014 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.010+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3423710894}) 2024-09-16T04:13:32.014 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.010+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1051626651}) 2024-09-16T04:13:32.016 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.010+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:32.016 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.010+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3099996050}) 2024-09-16T04:13:32.016 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.010+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2939051367}) 2024-09-16T04:13:32.016 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.010+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:32.016 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.010+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3099996050}) 2024-09-16T04:13:32.016 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.010+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4042619198}) 2024-09-16T04:13:32.018 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.014+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:32.018 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.014+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=236723259}) 2024-09-16T04:13:32.018 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.014+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2504143506}) 2024-09-16T04:13:32.018 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.014+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:32.018 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.014+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=236723259}) 2024-09-16T04:13:32.018 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.014+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1570150974}) 2024-09-16T04:13:32.020 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.014+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:32.020 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.014+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1444283112}) 2024-09-16T04:13:32.020 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.014+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2993255619}) 2024-09-16T04:13:32.020 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.014+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:32.020 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.014+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1444283112}) 2024-09-16T04:13:32.020 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.014+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2559195527}) 2024-09-16T04:13:32.022 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.018+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:32.022 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.018+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1064547644}) 2024-09-16T04:13:32.022 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.018+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3077754347}) 2024-09-16T04:13:32.022 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.018+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:32.022 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.018+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1064547644}) 2024-09-16T04:13:32.022 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.018+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2378739143}) 2024-09-16T04:13:32.024 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.018+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:32.024 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.018+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3789892894}) 2024-09-16T04:13:32.025 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.018+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1601214779}) 2024-09-16T04:13:32.025 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.018+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:32.025 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.018+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3789892894}) 2024-09-16T04:13:32.025 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.018+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3107587895}) 2024-09-16T04:13:32.026 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.022+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:32.027 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.022+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2913606011}) 2024-09-16T04:13:32.027 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.022+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=223154844}) 2024-09-16T04:13:32.027 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.022+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:32.027 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.022+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2913606011}) 2024-09-16T04:13:32.027 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.022+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1672809520}) 2024-09-16T04:13:32.029 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.026+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:32.029 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.026+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=380769953}) 2024-09-16T04:13:32.029 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.026+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2907605375}) 2024-09-16T04:13:32.029 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.026+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:32.029 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.026+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=380769953}) 2024-09-16T04:13:32.029 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.026+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=207865465}) 2024-09-16T04:13:32.034 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.030+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:32.035 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.030+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1782508467}) 2024-09-16T04:13:32.035 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.030+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2005948468}) 2024-09-16T04:13:32.035 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.030+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:32.035 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.030+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1782508467}) 2024-09-16T04:13:32.035 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.030+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1907672283}) 2024-09-16T04:13:32.040 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.034+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:32.040 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.034+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3504032617}) 2024-09-16T04:13:32.040 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.034+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2137097848}) 2024-09-16T04:13:32.040 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.034+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:32.040 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.034+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3504032617}) 2024-09-16T04:13:32.040 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.034+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2358984639}) 2024-09-16T04:13:32.042 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.038+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:32.043 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.038+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=619627754}) 2024-09-16T04:13:32.043 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.038+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2897879824}) 2024-09-16T04:13:32.043 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.038+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:32.043 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.038+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=619627754}) 2024-09-16T04:13:32.043 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.038+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1945490469}) 2024-09-16T04:13:32.044 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.038+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:32.044 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.038+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4161111067}) 2024-09-16T04:13:32.044 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.038+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=987129515}) 2024-09-16T04:13:32.045 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.038+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:32.045 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.038+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4161111067}) 2024-09-16T04:13:32.045 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.038+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3329392624}) 2024-09-16T04:13:32.046 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.042+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:32.047 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.042+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1645796011}) 2024-09-16T04:13:32.047 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.042+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=43261650}) 2024-09-16T04:13:32.047 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.042+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:32.048 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.042+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1645796011}) 2024-09-16T04:13:32.048 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.042+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1365400438}) 2024-09-16T04:13:32.049 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.042+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:32.049 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.042+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=309889696}) 2024-09-16T04:13:32.050 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.042+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=670850210}) 2024-09-16T04:13:32.050 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.042+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:32.050 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.042+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=309889696}) 2024-09-16T04:13:32.050 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.042+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1400580676}) 2024-09-16T04:13:32.051 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.046+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:32.052 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.046+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1213040120}) 2024-09-16T04:13:32.052 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.046+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=521504145}) 2024-09-16T04:13:32.052 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.046+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:32.052 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.046+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1213040120}) 2024-09-16T04:13:32.052 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.046+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1846603725}) 2024-09-16T04:13:32.053 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.050+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:32.054 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.050+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1749826877}) 2024-09-16T04:13:32.054 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.050+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1715188778}) 2024-09-16T04:13:32.054 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.050+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:32.054 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.050+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1749826877}) 2024-09-16T04:13:32.054 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.050+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1252835114}) 2024-09-16T04:13:32.056 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.050+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:32.056 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.050+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1443858628}) 2024-09-16T04:13:32.056 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.050+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3574054096}) 2024-09-16T04:13:32.056 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.050+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:32.057 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.050+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1443858628}) 2024-09-16T04:13:32.057 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.050+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4294083362}) 2024-09-16T04:13:32.058 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.054+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:32.058 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.054+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2953974408}) 2024-09-16T04:13:32.058 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.054+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1161943777}) 2024-09-16T04:13:32.058 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.054+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:32.058 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.054+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2953974408}) 2024-09-16T04:13:32.059 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.054+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2273579316}) 2024-09-16T04:13:32.060 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.054+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:32.061 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.054+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1446036468}) 2024-09-16T04:13:32.061 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.054+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2832325730}) 2024-09-16T04:13:32.061 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.054+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:32.061 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.054+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1446036468}) 2024-09-16T04:13:32.061 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.054+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1935151104}) 2024-09-16T04:13:32.063 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.058+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:32.063 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.058+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=154879862}) 2024-09-16T04:13:32.063 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.058+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1546960313}) 2024-09-16T04:13:32.063 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.058+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:32.063 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.058+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=154879862}) 2024-09-16T04:13:32.063 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.058+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3914078296}) 2024-09-16T04:13:32.065 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.058+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:32.065 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.062+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1942865638}) 2024-09-16T04:13:32.065 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.062+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3201039317}) 2024-09-16T04:13:32.065 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.062+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:32.065 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.062+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1942865638}) 2024-09-16T04:13:32.065 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.062+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=700895350}) 2024-09-16T04:13:32.068 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.062+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:32.068 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.062+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=72,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=8} crc {logm=946813321,mdsmap=3762917140,mgr=1483503728,mgr_command_descs=2107673600,mgr_metadata=1367272202,mgrstat=3186113285,mon_config_key=3573239904}) 2024-09-16T04:13:32.068 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.062+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=76,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=7,mon_config_key=4} crc {logm=2411433938,mdsmap=4088524775,mgr=460109776,mgr_command_descs=2107673600,mgr_metadata=1367272202,mgrstat=2061723722,mon_config_key=2532506080}) 2024-09-16T04:13:32.068 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.062+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:32.068 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.062+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=72,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=8} crc {logm=946813321,mdsmap=3762917140,mgr=1483503728,mgr_command_descs=2107673600,mgr_metadata=1367272202,mgrstat=3186113285,mon_config_key=3573239904}) 2024-09-16T04:13:32.068 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.062+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=72,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=7} crc {logm=3881798151,mdsmap=4088524775,mgr=460109776,mgr_command_descs=2107673600,mgr_metadata=1367272202,mgrstat=3186113285,mon_config_key=3867845802}) 2024-09-16T04:13:32.070 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.066+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:32.071 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.066+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mon_config_key=3,monmap=2,osd_metadata=6,osd_snap=42,osdmap=47} crc {mon_config_key=2278410813,monmap=3140514009,osd_metadata=4182959540,osd_snap=2456176894,osdmap=3628527254}) 2024-09-16T04:13:32.071 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.066+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mon_config_key=3,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=43,osdmap=44} crc {mon_config_key=2278410813,monmap=3937513981,osd_metadata=4182959540,osd_pg_creating=783066558,osd_snap=2967940269,osdmap=2224540171}) 2024-09-16T04:13:32.071 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.066+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:13:32.071 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.066+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mon_config_key=3,monmap=2,osd_metadata=6,osd_snap=42,osdmap=47} crc {mon_config_key=2278410813,monmap=3140514009,osd_metadata=4182959540,osd_snap=2456176894,osdmap=3628527254}) 2024-09-16T04:13:32.071 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:13:32.066+0000 7f124770c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mon_config_key=3,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=41,osdmap=46} crc {mon_config_key=2278410813,monmap=3937513981,osd_metadata=4182959540,osd_pg_creating=783066558,osd_snap=768091511,osdmap=2040793995}) 2024-09-16T04:13:32.812 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T04:13:32.813 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:13:32.946 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.942+0000 7fabe889c640 10 monclient: get_monmap_and_config 2024-09-16T04:13:32.947 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.942+0000 7fabe889c640 10 monclient: build_initial_monmap 2024-09-16T04:13:32.947 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.942+0000 7fabe889c640 10 monclient: monmap: 2024-09-16T04:13:32.947 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:13:32.947 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:13:32.947 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:13:32.945713+0000 2024-09-16T04:13:32.947 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:13:32.945713+0000 2024-09-16T04:13:32.947 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:13:32.947 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:13:32.947 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:13:32.947 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:13:32.947 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:13:32.947 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:13:32.947 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:13:32.948 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:13:32.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.942+0000 7fabe889c640 1 Processor -- start 2024-09-16T04:13:32.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.942+0000 7fabe889c640 1 -- start start 2024-09-16T04:13:32.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.942+0000 7fabe889c640 10 monclient: init 2024-09-16T04:13:32.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.942+0000 7fabe889c640 10 monclient: _reopen_session rank -1 2024-09-16T04:13:32.949 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.942+0000 7fabe889c640 10 monclient: _add_conns ranks=[1,0,2,4,3] 2024-09-16T04:13:32.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.942+0000 7fabe889c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fabe40a39b0 addr v1:172.21.15.154:6789/0 2024-09-16T04:13:32.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.942+0000 7fabe889c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fabe40a3d90 addr v1:172.21.15.42:6789/0 2024-09-16T04:13:32.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.942+0000 7fabe889c640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fabe413daa0 0x7fabe413fe90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:13:32.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.942+0000 7fabe889c640 10 monclient(hunting): picked mon.noname-b con 0x7fabe413daa0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:13:32.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.942+0000 7fabe889c640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fabe4134290 con 0x7fabe40a3d90 2024-09-16T04:13:32.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.942+0000 7fabe889c640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fabe409c140 con 0x7fabe40a39b0 2024-09-16T04:13:32.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.942+0000 7fabe889c640 10 monclient(hunting): start opening mon connection 2024-09-16T04:13:32.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.942+0000 7fabe889c640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fabe409c420 con 0x7fabe413daa0 2024-09-16T04:13:32.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.942+0000 7fabe889c640 10 monclient(hunting): _renew_subs 2024-09-16T04:13:32.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.942+0000 7fabe889c640 10 monclient(hunting): authenticate will time out at 2859.915155s 2024-09-16T04:13:32.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.942+0000 7fabe1d9b640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fabe40a3d90 0x7fabe413c380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:35768/0 (socket says 172.21.15.42:35768) 2024-09-16T04:13:32.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.942+0000 7fabe1d9b640 1 -- 172.21.15.42:0/3402077092 learned_addr learned my addr 172.21.15.42:0/3402077092 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T04:13:32.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.942+0000 7fabe2d9d640 1 --2- 172.21.15.42:0/3402077092 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fabe413daa0 0x7fabe413fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:13:32.950 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.942+0000 7fabe2d9d640 10 monclient(hunting): get_auth_request con 0x7fabe413daa0 auth_method 0 2024-09-16T04:13:32.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.942+0000 7fabe2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:13:32.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.942+0000 7fabe2d9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:32.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.942+0000 7fabe2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:13:32.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.942+0000 7fabe2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:13:32.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.942+0000 7fabe2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:13:32.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.942+0000 7fabe2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:13:32.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.942+0000 7fabe159a640 1 -- 172.21.15.42:0/3402077092 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 472208732 0 0) 0x7fabe4134290 con 0x7fabe40a3d90 2024-09-16T04:13:32.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.942+0000 7fabe159a640 10 monclient(hunting): _negotiate 2024-09-16T04:13:32.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.942+0000 7fabe159a640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:32.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.942+0000 7fabe159a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:13:32.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.942+0000 7fabe159a640 10 monclient(hunting): my global_id is 1034631 2024-09-16T04:13:32.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.942+0000 7fabe159a640 1 -- 172.21.15.42:0/3402077092 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fabcc003530 con 0x7fabe40a3d90 2024-09-16T04:13:32.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.942+0000 7fabe2d9d640 10 monclient(hunting): handle_auth_done global_id 1094123 payload 274 2024-09-16T04:13:32.951 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.942+0000 7fabe2d9d640 1 -- 172.21.15.42:0/3402077092 >> v1:172.21.15.154:6789/0 conn(0x7fabe40a39b0 legacy=0x7fabe4133b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:32.952 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.942+0000 7fabe2d9d640 1 -- 172.21.15.42:0/3402077092 >> v1:172.21.15.42:6789/0 conn(0x7fabe40a3d90 legacy=0x7fabe413c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:32.952 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.942+0000 7fabe2d9d640 10 monclient: _finish_hunting 0 2024-09-16T04:13:32.952 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.942+0000 7fabe2d9d640 1 monclient: found mon.noname-b 2024-09-16T04:13:32.952 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.942+0000 7fabe2d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:13:32.952 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.942+0000 7fabe2d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T04:13:32.952 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.942+0000 7fabe2d9d640 1 -- 172.21.15.42:0/3402077092 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fabe406dd60 con 0x7fabe413daa0 2024-09-16T04:13:32.958 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.946+0000 7fabe2d9d640 1 --2- 172.21.15.42:0/3402077092 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fabe413daa0 0x7fabe413fe90 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fabdc0097f0 tx=0x7fabdc01c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=82dfc5b056da86a9 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:13:32.958 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.946+0000 7fabe159a640 1 -- 172.21.15.42:0/3402077092 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabdc0b0070 con 0x7fabe413daa0 2024-09-16T04:13:32.959 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.946+0000 7fabe159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:13:32.959 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.946+0000 7fabe159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:13:32.959 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.946+0000 7fabe159a640 10 monclient: dump: 2024-09-16T04:13:32.959 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:13:32.959 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:13:32.959 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:32.959 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:32.959 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:32.959 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:13:32.959 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:13:32.959 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:13:32.959 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:13:32.959 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:13:32.960 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.946+0000 7fabe159a640 10 monclient: _finish_auth 0 2024-09-16T04:13:32.960 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.946+0000 7fabe159a640 10 monclient: _check_auth_tickets 2024-09-16T04:13:32.960 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.946+0000 7fabe159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:13:32.960 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.946+0000 7fabe889c640 5 monclient: authenticate success, global_id 1094123 2024-09-16T04:13:32.960 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.946+0000 7fabe889c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:13:32.960 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.946+0000 7fabe159a640 1 -- 172.21.15.42:0/3402077092 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fabdc0ab600 con 0x7fabe413daa0 2024-09-16T04:13:32.960 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.946+0000 7fabe159a640 10 monclient: handle_config config(0 keys) 2024-09-16T04:13:32.960 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.946+0000 7fabe889c640 10 monclient: get_monmap_and_config success 2024-09-16T04:13:32.960 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.946+0000 7fabe889c640 4 set_mon_vals no callback set 2024-09-16T04:13:32.961 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.946+0000 7fabe159a640 1 -- 172.21.15.42:0/3402077092 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabdc0ab9b0 con 0x7fabe413daa0 2024-09-16T04:13:32.961 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.946+0000 7fabe159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:13:32.961 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.946+0000 7fabe159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:13:32.961 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.946+0000 7fabe159a640 10 monclient: dump: 2024-09-16T04:13:32.961 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:13:32.961 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:13:32.961 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:32.961 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:32.961 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:32.961 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:13:32.961 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:13:32.961 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:13:32.961 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:13:32.962 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:13:32.962 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.950+0000 7fabe889c640 10 monclient: shutdown 2024-09-16T04:13:32.962 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.950+0000 7fabe889c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:13:32.962 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.950+0000 7fabe889c640 1 -- 172.21.15.42:0/3402077092 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fabe413daa0 msgr2=0x7fabe413fe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:32.962 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.950+0000 7fabe889c640 1 --2- 172.21.15.42:0/3402077092 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fabe413daa0 0x7fabe413fe90 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fabdc0097f0 tx=0x7fabdc01c3a0 comp rx=0 tx=0).stop 2024-09-16T04:13:32.962 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.950+0000 7fabe889c640 1 -- 172.21.15.42:0/3402077092 shutdown_connections 2024-09-16T04:13:32.962 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.950+0000 7fabe889c640 1 --2- 172.21.15.42:0/3402077092 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fabe413daa0 0x7fabe413fe90 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:13:32.962 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.950+0000 7fabe889c640 1 -- 172.21.15.42:0/3402077092 >> 172.21.15.42:0/3402077092 conn(0x7fabe40a9fb0 msgr2=0x7fabe40aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:13:32.962 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.950+0000 7fabe889c640 1 -- 172.21.15.42:0/3402077092 shutdown_connections 2024-09-16T04:13:32.962 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.950+0000 7fabe889c640 1 -- 172.21.15.42:0/3402077092 wait complete. 2024-09-16T04:13:32.962 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.950+0000 7fabe889c640 10 monclient: build_initial_monmap 2024-09-16T04:13:32.962 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.950+0000 7fabe889c640 10 monclient: monmap: 2024-09-16T04:13:32.962 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:13:32.963 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:13:32.963 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:13:32.963 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:13:32.963 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:13:32.963 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:13:32.963 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:13:32.963 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:13:32.963 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:13:32.964 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:13:32.964 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.950+0000 7fabe889c640 1 librados: starting msgr at 2024-09-16T04:13:32.964 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.950+0000 7fabe889c640 1 librados: starting objecter 2024-09-16T04:13:32.964 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.950+0000 7fabe889c640 1 Processor -- start 2024-09-16T04:13:32.964 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.950+0000 7fabe889c640 1 -- start start 2024-09-16T04:13:32.964 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.950+0000 7fabe889c640 1 librados: setting wanted keys 2024-09-16T04:13:32.964 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.950+0000 7fabe889c640 1 librados: calling monclient init 2024-09-16T04:13:32.964 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.950+0000 7fabe889c640 10 monclient: init 2024-09-16T04:13:32.964 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.950+0000 7fabe889c640 10 monclient: _reopen_session rank -1 2024-09-16T04:13:32.964 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.950+0000 7fabe889c640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:13:32.964 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.950+0000 7fabe889c640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fabe40a39b0 0x7fabe4072e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:13:32.964 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.950+0000 7fabe889c640 10 monclient(hunting): picked mon.noname-c con 0x7fabe40a39b0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:13:32.964 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.950+0000 7fabe889c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fabe40a3d90 0x7fabe413a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:13:32.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.950+0000 7fabe889c640 10 monclient(hunting): picked mon.noname-a con 0x7fabe40a3d90 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:13:32.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.950+0000 7fabe889c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fabe413daa0 0x7fabe413a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:13:32.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.950+0000 7fabe889c640 10 monclient(hunting): picked mon.noname-b con 0x7fabe413daa0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:13:32.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.950+0000 7fabe889c640 10 monclient(hunting): start opening mon connection 2024-09-16T04:13:32.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.950+0000 7fabe889c640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fabe4142f80 con 0x7fabe40a3d90 2024-09-16T04:13:32.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.950+0000 7fabe889c640 10 monclient(hunting): start opening mon connection 2024-09-16T04:13:32.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.950+0000 7fabe889c640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fabe4142e00 con 0x7fabe413daa0 2024-09-16T04:13:32.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.950+0000 7fabe889c640 10 monclient(hunting): start opening mon connection 2024-09-16T04:13:32.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.950+0000 7fabe889c640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fabe4143100 con 0x7fabe40a39b0 2024-09-16T04:13:32.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.950+0000 7fabe889c640 10 monclient(hunting): _renew_subs 2024-09-16T04:13:32.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.950+0000 7fabe889c640 10 monclient(hunting): authenticate will time out at 2859.922009s 2024-09-16T04:13:32.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.950+0000 7fabe259c640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fabe40a39b0 0x7fabe4072e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:13:32.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.950+0000 7fabe259c640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fabe40a39b0 0x7fabe4072e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:49032/0 (socket says 172.21.15.42:49032) 2024-09-16T04:13:32.965 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.950+0000 7fabe259c640 1 -- 172.21.15.42:0/1978569367 learned_addr learned my addr 172.21.15.42:0/1978569367 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:13:32.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.950+0000 7fabe259c640 10 monclient(hunting): get_auth_request con 0x7fabe40a39b0 auth_method 0 2024-09-16T04:13:32.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.950+0000 7fabe259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:13:32.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.950+0000 7fabe259c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:32.966 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.950+0000 7fabe1d9b640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fabe40a3d90 0x7fabe413a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:13:32.967 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.950+0000 7fabe259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:13:32.967 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.950+0000 7fabe1d9b640 10 monclient(hunting): get_auth_request con 0x7fabe40a3d90 auth_method 0 2024-09-16T04:13:32.967 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.950+0000 7fabe1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:13:32.967 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.950+0000 7fabe1d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:32.967 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.950+0000 7fabe1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:13:32.967 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.950+0000 7fabe259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:13:32.967 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.950+0000 7fabe259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:13:32.967 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.950+0000 7fabe259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:13:32.967 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.950+0000 7fabe1d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:13:32.967 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.950+0000 7fabe1d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:13:32.967 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.950+0000 7fabe1d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:13:32.967 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.950+0000 7fabe259c640 10 monclient(hunting): handle_auth_done global_id 1094126 payload 824 2024-09-16T04:13:32.968 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.950+0000 7fabe259c640 1 -- 172.21.15.42:0/1978569367 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fabe413daa0 msgr2=0x7fabe413a6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:32.968 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.950+0000 7fabe259c640 1 --2- 172.21.15.42:0/1978569367 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fabe413daa0 0x7fabe413a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:13:32.968 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.950+0000 7fabe259c640 1 -- 172.21.15.42:0/1978569367 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fabe40a3d90 msgr2=0x7fabe413a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:32.968 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.950+0000 7fabe259c640 1 --2- 172.21.15.42:0/1978569367 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fabe40a3d90 0x7fabe413a1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:13:32.968 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.950+0000 7fabe259c640 10 monclient: _finish_hunting 0 2024-09-16T04:13:32.968 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.950+0000 7fabe259c640 1 monclient: found mon.noname-c 2024-09-16T04:13:32.968 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.950+0000 7fabe259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:13:32.968 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.950+0000 7fabe259c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T04:13:32.968 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.950+0000 7fabe259c640 1 -- 172.21.15.42:0/1978569367 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fabe4132b20 con 0x7fabe40a39b0 2024-09-16T04:13:32.968 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.950+0000 7fabe259c640 1 --2- 172.21.15.42:0/1978569367 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fabe40a39b0 0x7fabe4072e10 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fabd400c790 tx=0x7fabd400cc50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:13:32.968 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.950+0000 7fabd2ffd640 10 client.?.objecter ms_handle_connect 0x7fabe40a39b0 2024-09-16T04:13:32.968 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.950+0000 7fabd2ffd640 10 client.?.objecter resend_mon_ops 2024-09-16T04:13:32.968 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.950+0000 7fabd2ffd640 1 -- 172.21.15.42:0/1978569367 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabd4016020 con 0x7fabe40a39b0 2024-09-16T04:13:32.969 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.950+0000 7fabd2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:13:32.969 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.950+0000 7fabd2ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:13:32.969 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.950+0000 7fabd2ffd640 10 monclient: dump: 2024-09-16T04:13:32.969 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:13:32.969 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:13:32.969 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:32.969 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:32.969 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:32.969 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:13:32.969 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:13:32.969 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:13:32.969 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:13:32.969 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:13:32.969 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.950+0000 7fabd2ffd640 10 monclient: _finish_auth 0 2024-09-16T04:13:32.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.950+0000 7fabd2ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:13:32.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.950+0000 7fabd2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:13:32.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.950+0000 7fabd2ffd640 1 -- 172.21.15.42:0/1978569367 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fabd4006d60 con 0x7fabe40a39b0 2024-09-16T04:13:32.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.950+0000 7fabe889c640 5 monclient: authenticate success, global_id 1094126 2024-09-16T04:13:32.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.950+0000 7fabe889c640 10 monclient: _renew_subs 2024-09-16T04:13:32.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.950+0000 7fabe889c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:13:32.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.950+0000 7fabe889c640 1 -- 172.21.15.42:0/1978569367 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fabe406f680 con 0x7fabe40a39b0 2024-09-16T04:13:32.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.950+0000 7fabd2ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:13:32.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.950+0000 7fabd2ffd640 1 -- 172.21.15.42:0/1978569367 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabd4003bc0 con 0x7fabe40a39b0 2024-09-16T04:13:32.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.950+0000 7fabd2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:13:32.970 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.950+0000 7fabd2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:13:32.971 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.950+0000 7fabd2ffd640 10 monclient: dump: 2024-09-16T04:13:32.971 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:13:32.971 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:13:32.971 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:32.971 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:32.971 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:32.971 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:13:32.971 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:13:32.971 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:13:32.971 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:13:32.971 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:13:32.971 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.950+0000 7fabd3fff640 4 set_mon_vals no callback set 2024-09-16T04:13:32.972 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.950+0000 7fabe889c640 10 client.1094126.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:13:32.972 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.950+0000 7fabe889c640 10 monclient: _renew_subs 2024-09-16T04:13:32.972 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.950+0000 7fabe889c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:13:32.972 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.950+0000 7fabe889c640 1 -- 172.21.15.42:0/1978569367 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fabe413c340 con 0x7fabe40a39b0 2024-09-16T04:13:32.972 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.950+0000 7fabe889c640 1 librados: init done 2024-09-16T04:13:32.972 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.950+0000 7fabe889c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:13:32.972 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.950+0000 7fabe889c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:13:32.972 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.950+0000 7fabe889c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:13:32.972 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.950+0000 7fabd2ffd640 1 -- 172.21.15.42:0/1978569367 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fabd400a850 con 0x7fabe40a39b0 2024-09-16T04:13:32.972 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.950+0000 7fabe889c640 1 -- 172.21.15.42:0/1978569367 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fabe4133b80 con 0x7fabe40a39b0 2024-09-16T04:13:32.972 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.954+0000 7fabd2ffd640 1 --2- 172.21.15.42:0/1978569367 >> v2:172.21.15.154:6812/254845342 conn(0x7fabc4037810 0x7fabc4039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:13:32.972 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.954+0000 7fabd2ffd640 1 -- 172.21.15.42:0/1978569367 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(706..706 src has 681..706) ==== 5040+0+0 (secure 0 0 0) 0x7fabd4048970 con 0x7fabe40a39b0 2024-09-16T04:13:32.972 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.954+0000 7fabd2ffd640 10 client.1094126.objecter ms_dispatch 0x7fabe4000ba0 osd_map(706..706 src has 681..706) 2024-09-16T04:13:32.973 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.954+0000 7fabd2ffd640 3 client.1094126.objecter handle_osd_map got epochs [706,706] > 0 2024-09-16T04:13:32.973 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.954+0000 7fabd2ffd640 3 client.1094126.objecter handle_osd_map decoding full epoch 706 2024-09-16T04:13:32.973 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.954+0000 7fabd2ffd640 20 client.1094126.objecter dump_active .. 0 homeless 2024-09-16T04:13:32.973 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.954+0000 7fabe1d9b640 1 --2- 172.21.15.42:0/1978569367 >> v2:172.21.15.154:6812/254845342 conn(0x7fabc4037810 0x7fabc4039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:13:32.973 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.954+0000 7fabe1d9b640 10 monclient: get_auth_request con 0x7fabc4037810 auth_method 0 2024-09-16T04:13:32.973 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.954+0000 7fabd2ffd640 10 client.1094126.objecter ms_handle_connect 0x7fabc4037810 2024-09-16T04:13:32.973 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.954+0000 7fabe1d9b640 1 --2- 172.21.15.42:0/1978569367 >> v2:172.21.15.154:6812/254845342 conn(0x7fabc4037810 0x7fabc4039cd0 secure :-1 s=READY pgs=3990 cs=0 l=1 rev1=1 crypto rx=0x7fabd80011f0 tx=0x7fabd8007d40 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:13:32.973 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.958+0000 7fabd2ffd640 1 -- 172.21.15.42:0/1978569367 <== mon.2 v2:172.21.15.42:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7fabd4020020 con 0x7fabe40a39b0 2024-09-16T04:13:32.974 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.958+0000 7fabd2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:13:32.974 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:32.958+0000 7fabd2ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:13:33.191 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.186+0000 7fabe889c640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:13:33.192 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.186+0000 7fabe889c640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:13:33.192 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.186+0000 7fabe889c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:13:33.192 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.186+0000 7fabe889c640 1 -- 172.21.15.42:0/1978569367 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fabe40a5750 con 0x7fabe40a39b0 2024-09-16T04:13:33.192 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.186+0000 7fabd2ffd640 1 -- 172.21.15.42:0/1978569367 <== mon.2 v2:172.21.15.42:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7fabd400ae20 con 0x7fabe40a39b0 2024-09-16T04:13:33.192 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.186+0000 7fabd2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:13:33.192 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.186+0000 7fabd2ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:13:33.193 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":286,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:13:33.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.190+0000 7fabe889c640 10 librados: watch_flush enter 2024-09-16T04:13:33.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.190+0000 7fabe889c640 10 librados: watch_flush exit 2024-09-16T04:13:33.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.190+0000 7fabe889c640 20 client.1094126.objecter shutdown clearing up homeless session... 2024-09-16T04:13:33.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.190+0000 7fabe889c640 10 client.1094126.objecter successfully canceled tick 2024-09-16T04:13:33.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.190+0000 7fabe889c640 1 -- 172.21.15.42:0/1978569367 >> v2:172.21.15.154:6812/254845342 conn(0x7fabc4037810 msgr2=0x7fabc4039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:33.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.190+0000 7fabe889c640 1 --2- 172.21.15.42:0/1978569367 >> v2:172.21.15.154:6812/254845342 conn(0x7fabc4037810 0x7fabc4039cd0 secure :-1 s=READY pgs=3990 cs=0 l=1 rev1=1 crypto rx=0x7fabd80011f0 tx=0x7fabd8007d40 comp rx=0 tx=0).stop 2024-09-16T04:13:33.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.190+0000 7fabe889c640 10 monclient: shutdown 2024-09-16T04:13:33.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.190+0000 7fabe889c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:13:33.195 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.190+0000 7fabe889c640 1 -- 172.21.15.42:0/1978569367 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fabe40a39b0 msgr2=0x7fabe4072e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:33.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.190+0000 7fabe889c640 1 --2- 172.21.15.42:0/1978569367 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fabe40a39b0 0x7fabe4072e10 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fabd400c790 tx=0x7fabd400cc50 comp rx=0 tx=0).stop 2024-09-16T04:13:33.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.190+0000 7fabe889c640 1 -- 172.21.15.42:0/1978569367 shutdown_connections 2024-09-16T04:13:33.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.190+0000 7fabe889c640 1 --2- 172.21.15.42:0/1978569367 >> v2:172.21.15.154:6812/254845342 conn(0x7fabc4037810 0x7fabc4039cd0 unknown :-1 s=CLOSED pgs=3990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:13:33.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.190+0000 7fabe889c640 1 --2- 172.21.15.42:0/1978569367 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fabe413daa0 0x7fabe413a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:13:33.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.190+0000 7fabe889c640 1 --2- 172.21.15.42:0/1978569367 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fabe40a3d90 0x7fabe413a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:13:33.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.190+0000 7fabe889c640 1 --2- 172.21.15.42:0/1978569367 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fabe40a39b0 0x7fabe4072e10 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:13:33.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.190+0000 7fabe889c640 1 -- 172.21.15.42:0/1978569367 >> 172.21.15.42:0/1978569367 conn(0x7fabe40a9fb0 msgr2=0x7fabe40a8850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:13:33.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.190+0000 7fabe889c640 1 -- 172.21.15.42:0/1978569367 shutdown_connections 2024-09-16T04:13:33.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.190+0000 7fabe889c640 1 -- 172.21.15.42:0/1978569367 wait complete. 2024-09-16T04:13:33.196 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.190+0000 7fabe889c640 1 librados: shutdown 2024-09-16T04:13:33.212 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T04:13:33.213 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-16T04:13:33.213 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T04:13:33.213 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T04:13:33.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.334+0000 7f834dcb1640 10 monclient: get_monmap_and_config 2024-09-16T04:13:33.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.334+0000 7f834dcb1640 10 monclient: build_initial_monmap 2024-09-16T04:13:33.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.334+0000 7f834dcb1640 10 monclient: monmap: 2024-09-16T04:13:33.338 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:13:33.338 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:13:33.338 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:13:33.336974+0000 2024-09-16T04:13:33.338 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:13:33.336974+0000 2024-09-16T04:13:33.338 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:13:33.338 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:13:33.339 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:13:33.339 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:13:33.339 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:13:33.339 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:13:33.339 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:13:33.339 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:13:33.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.334+0000 7f834dcb1640 1 Processor -- start 2024-09-16T04:13:33.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.334+0000 7f834dcb1640 1 -- start start 2024-09-16T04:13:33.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.334+0000 7f834dcb1640 10 monclient: init 2024-09-16T04:13:33.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.334+0000 7f834dcb1640 10 monclient: _reopen_session rank -1 2024-09-16T04:13:33.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.334+0000 7f834dcb1640 10 monclient: _add_conns ranks=[3,1,4,0,2] 2024-09-16T04:13:33.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.334+0000 7f834dcb1640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f83480a1330 0x7f83480a1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:13:33.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.334+0000 7f834dcb1640 10 monclient(hunting): picked mon.noname-a con 0x7f83480a1330 addr v2:172.21.15.42:3300/0 2024-09-16T04:13:33.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.334+0000 7f834dcb1640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f834806dac0 addr v1:172.21.15.154:6789/0 2024-09-16T04:13:33.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.334+0000 7f834dcb1640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f83480a1c50 0x7f8348135810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:13:33.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.334+0000 7f834dcb1640 10 monclient(hunting): picked mon.noname-c con 0x7f83480a1c50 addr v2:172.21.15.154:3300/0 2024-09-16T04:13:33.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.334+0000 7f834dcb1640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f834809c090 con 0x7f834806dac0 2024-09-16T04:13:33.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.334+0000 7f834dcb1640 10 monclient(hunting): start opening mon connection 2024-09-16T04:13:33.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.334+0000 7f834dcb1640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f834809f610 con 0x7f83480a1330 2024-09-16T04:13:33.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.334+0000 7f834dcb1640 10 monclient(hunting): start opening mon connection 2024-09-16T04:13:33.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.334+0000 7f834dcb1640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f834809f8f0 con 0x7f83480a1c50 2024-09-16T04:13:33.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.334+0000 7f834dcb1640 10 monclient(hunting): _renew_subs 2024-09-16T04:13:33.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.334+0000 7f834dcb1640 10 monclient(hunting): authenticate will time out at 2860.306453s 2024-09-16T04:13:33.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.334+0000 7f83477fe640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f83480a1330 0x7f83480a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:13:33.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.334+0000 7f83477fe640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f83480a1330 0x7f83480a1710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:52078/0 (socket says 172.21.15.42:52078) 2024-09-16T04:13:33.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.334+0000 7f83477fe640 1 -- 172.21.15.42:0/676468124 learned_addr learned my addr 172.21.15.42:0/676468124 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:13:33.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.334+0000 7f83477fe640 10 monclient(hunting): get_auth_request con 0x7f83480a1330 auth_method 0 2024-09-16T04:13:33.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.334+0000 7f83477fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:13:33.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.334+0000 7f83477fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:33.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.334+0000 7f83477fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:13:33.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.334+0000 7f83477fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:13:33.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.334+0000 7f83477fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:13:33.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.334+0000 7f83477fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:13:33.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.334+0000 7f8347fff640 1 --2- 172.21.15.42:0/676468124 >> v2:172.21.15.154:3300/0 conn(0x7f83480a1c50 0x7f8348135810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:13:33.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.334+0000 7f8347fff640 1 -- 172.21.15.42:0/676468124 >> v2:172.21.15.154:3300/0 conn(0x7f83480a1c50 msgr2=0x7f8348135810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T04:13:33.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.334+0000 7f8347fff640 1 -- 172.21.15.42:0/676468124 >> v2:172.21.15.154:3300/0 conn(0x7f83480a1c50 msgr2=0x7f8348135810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:13:33.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.334+0000 7f8347fff640 1 --2- 172.21.15.42:0/676468124 >> v2:172.21.15.154:3300/0 conn(0x7f83480a1c50 0x7f8348135810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:13:33.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.334+0000 7f8347fff640 1 --2- 172.21.15.42:0/676468124 >> v2:172.21.15.154:3300/0 conn(0x7f83480a1c50 0x7f8348135810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:13:33.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.334+0000 7f83477fe640 10 monclient(hunting): handle_auth_done global_id 1034637 payload 274 2024-09-16T04:13:33.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.334+0000 7f83477fe640 1 -- 172.21.15.42:0/676468124 >> v2:172.21.15.154:3300/0 conn(0x7f83480a1c50 msgr2=0x7f8348135810 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:13:33.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.334+0000 7f83477fe640 1 --2- 172.21.15.42:0/676468124 >> v2:172.21.15.154:3300/0 conn(0x7f83480a1c50 0x7f8348135810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:13:33.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.334+0000 7f83477fe640 1 -- 172.21.15.42:0/676468124 >> v1:172.21.15.154:6789/0 conn(0x7f834806dac0 legacy=0x7f8348135100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:33.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.334+0000 7f83477fe640 10 monclient: _finish_hunting 0 2024-09-16T04:13:33.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.334+0000 7f83477fe640 1 monclient: found mon.noname-a 2024-09-16T04:13:33.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.334+0000 7f83477fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:13:33.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.334+0000 7f83477fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:13:33.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.334+0000 7f83477fe640 1 -- 172.21.15.42:0/676468124 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f834809fbd0 con 0x7f83480a1330 2024-09-16T04:13:33.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.338+0000 7f83477fe640 1 --2- 172.21.15.42:0/676468124 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f83480a1330 0x7f83480a1710 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f834809c800 tx=0x7f834001c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89414537934bc0e0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:13:33.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.338+0000 7f83467fc640 1 -- 172.21.15.42:0/676468124 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83400a5790 con 0x7f83480a1330 2024-09-16T04:13:33.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.338+0000 7f83467fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:13:33.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.338+0000 7f83467fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:13:33.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.338+0000 7f83467fc640 10 monclient: dump: 2024-09-16T04:13:33.351 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:13:33.351 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:13:33.351 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:33.351 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:33.351 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:33.351 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:13:33.351 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:13:33.352 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:13:33.352 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:13:33.352 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:13:33.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.338+0000 7f83467fc640 10 monclient: _finish_auth 0 2024-09-16T04:13:33.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.338+0000 7f83467fc640 10 monclient: _check_auth_tickets 2024-09-16T04:13:33.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.338+0000 7f83467fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:13:33.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.338+0000 7f834dcb1640 5 monclient: authenticate success, global_id 1034637 2024-09-16T04:13:33.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.338+0000 7f83467fc640 1 -- 172.21.15.42:0/676468124 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f83400a5930 con 0x7f83480a1330 2024-09-16T04:13:33.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.338+0000 7f834dcb1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:13:33.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.338+0000 7f83467fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:13:33.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.338+0000 7f834dcb1640 10 monclient: get_monmap_and_config success 2024-09-16T04:13:33.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.338+0000 7f834dcb1640 4 set_mon_vals no callback set 2024-09-16T04:13:33.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.338+0000 7f83467fc640 1 -- 172.21.15.42:0/676468124 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83400a5d00 con 0x7f83480a1330 2024-09-16T04:13:33.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.338+0000 7f83467fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:13:33.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.338+0000 7f83467fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:13:33.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.338+0000 7f83467fc640 10 monclient: dump: 2024-09-16T04:13:33.353 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:13:33.353 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:13:33.353 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:33.353 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:33.353 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:33.353 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:13:33.353 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:13:33.353 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:13:33.353 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:13:33.353 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:13:33.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.338+0000 7f834dcb1640 10 monclient: shutdown 2024-09-16T04:13:33.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.338+0000 7f834dcb1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:13:33.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.338+0000 7f834dcb1640 1 -- 172.21.15.42:0/676468124 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f83480a1330 msgr2=0x7f83480a1710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:33.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.338+0000 7f834dcb1640 1 --2- 172.21.15.42:0/676468124 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f83480a1330 0x7f83480a1710 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f834809c800 tx=0x7f834001c8b0 comp rx=0 tx=0).stop 2024-09-16T04:13:33.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.338+0000 7f834dcb1640 1 -- 172.21.15.42:0/676468124 shutdown_connections 2024-09-16T04:13:33.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.338+0000 7f834dcb1640 1 --2- 172.21.15.42:0/676468124 >> v2:172.21.15.154:3300/0 conn(0x7f83480a1c50 0x7f8348135810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:13:33.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.338+0000 7f834dcb1640 1 --2- 172.21.15.42:0/676468124 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f83480a1330 0x7f83480a1710 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:13:33.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.338+0000 7f834dcb1640 1 -- 172.21.15.42:0/676468124 >> 172.21.15.42:0/676468124 conn(0x7f83480a2420 msgr2=0x7f83480a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:13:33.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.338+0000 7f834dcb1640 1 -- 172.21.15.42:0/676468124 shutdown_connections 2024-09-16T04:13:33.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.338+0000 7f834dcb1640 1 -- 172.21.15.42:0/676468124 wait complete. 2024-09-16T04:13:33.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.338+0000 7f834dcb1640 10 monclient: build_initial_monmap 2024-09-16T04:13:33.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.338+0000 7f834dcb1640 10 monclient: monmap: 2024-09-16T04:13:33.354 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:13:33.354 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:13:33.355 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:13:33.355 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:13:33.355 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:13:33.355 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:13:33.355 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:13:33.355 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:13:33.355 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:13:33.355 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:13:33.355 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.338+0000 7f834dcb1640 1 librados: starting msgr at 2024-09-16T04:13:33.355 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.338+0000 7f834dcb1640 1 librados: starting objecter 2024-09-16T04:13:33.355 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.342+0000 7f834dcb1640 1 Processor -- start 2024-09-16T04:13:33.355 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.342+0000 7f834dcb1640 1 -- start start 2024-09-16T04:13:33.355 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.342+0000 7f834dcb1640 1 librados: setting wanted keys 2024-09-16T04:13:33.355 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.342+0000 7f834dcb1640 1 librados: calling monclient init 2024-09-16T04:13:33.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.342+0000 7f834dcb1640 10 monclient: init 2024-09-16T04:13:33.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.342+0000 7f834dcb1640 10 monclient: _reopen_session rank -1 2024-09-16T04:13:33.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.342+0000 7f834dcb1640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:13:33.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.342+0000 7f834dcb1640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f83480a1330 0x7f83481ccc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:13:33.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.342+0000 7f834dcb1640 10 monclient(hunting): picked mon.noname-b con 0x7f83480a1330 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:13:33.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.342+0000 7f834dcb1640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f83480a1c50 0x7f8348134f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:13:33.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.342+0000 7f834dcb1640 10 monclient(hunting): picked mon.noname-c con 0x7f83480a1c50 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:13:33.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.342+0000 7f834dcb1640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8348135490 0x7f834811bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:13:33.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.342+0000 7f834dcb1640 10 monclient(hunting): picked mon.noname-a con 0x7f8348135490 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:13:33.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.342+0000 7f834dcb1640 10 monclient(hunting): start opening mon connection 2024-09-16T04:13:33.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.342+0000 7f834dcb1640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f8348135d50 con 0x7f8348135490 2024-09-16T04:13:33.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.342+0000 7f834dcb1640 10 monclient(hunting): start opening mon connection 2024-09-16T04:13:33.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.342+0000 7f834dcb1640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f834809ef80 con 0x7f83480a1330 2024-09-16T04:13:33.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.342+0000 7f834dcb1640 10 monclient(hunting): start opening mon connection 2024-09-16T04:13:33.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.342+0000 7f834dcb1640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f8348135ed0 con 0x7f83480a1c50 2024-09-16T04:13:33.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.342+0000 7f834dcb1640 10 monclient(hunting): _renew_subs 2024-09-16T04:13:33.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.342+0000 7f834dcb1640 10 monclient(hunting): authenticate will time out at 2860.312623s 2024-09-16T04:13:33.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.342+0000 7f8347fff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8348135490 0x7f834811bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:13:33.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.342+0000 7f8347fff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8348135490 0x7f834811bf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:52092/0 (socket says 172.21.15.42:52092) 2024-09-16T04:13:33.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.342+0000 7f8347fff640 1 -- 172.21.15.42:0/1914486117 learned_addr learned my addr 172.21.15.42:0/1914486117 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:13:33.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.342+0000 7f8347fff640 10 monclient(hunting): get_auth_request con 0x7f8348135490 auth_method 0 2024-09-16T04:13:33.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.342+0000 7f8347fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:13:33.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.342+0000 7f8347fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:33.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.342+0000 7f8347fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:13:33.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.342+0000 7f8346ffd640 1 --2- 172.21.15.42:0/1914486117 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f83480a1c50 0x7f8348134f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:13:33.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.342+0000 7f8346ffd640 10 monclient(hunting): get_auth_request con 0x7f83480a1c50 auth_method 0 2024-09-16T04:13:33.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.342+0000 7f8346ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:13:33.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.342+0000 7f8346ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:33.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.342+0000 7f8346ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:13:33.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.342+0000 7f8347fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:13:33.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.342+0000 7f8347fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:13:33.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.342+0000 7f8347fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:13:33.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.342+0000 7f8346ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:13:33.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.342+0000 7f8346ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:13:33.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.342+0000 7f8346ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:13:33.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.342+0000 7f83477fe640 1 --2- 172.21.15.42:0/1914486117 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f83480a1330 0x7f83481ccc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:13:33.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.342+0000 7f83477fe640 10 monclient(hunting): get_auth_request con 0x7f83480a1330 auth_method 0 2024-09-16T04:13:33.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.342+0000 7f83477fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:13:33.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.342+0000 7f83477fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:33.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.342+0000 7f83477fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:13:33.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.342+0000 7f8347fff640 10 monclient(hunting): handle_auth_done global_id 1034640 payload 824 2024-09-16T04:13:33.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.342+0000 7f8347fff640 1 -- 172.21.15.42:0/1914486117 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f83480a1c50 msgr2=0x7f8348134f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:33.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.342+0000 7f8347fff640 1 --2- 172.21.15.42:0/1914486117 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f83480a1c50 0x7f8348134f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:13:33.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.342+0000 7f8347fff640 1 -- 172.21.15.42:0/1914486117 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f83480a1330 msgr2=0x7f83481ccc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:33.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.342+0000 7f8347fff640 1 --2- 172.21.15.42:0/1914486117 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f83480a1330 0x7f83481ccc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:13:33.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.342+0000 7f8347fff640 10 monclient: _finish_hunting 0 2024-09-16T04:13:33.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.342+0000 7f8347fff640 1 monclient: found mon.noname-a 2024-09-16T04:13:33.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.342+0000 7f8347fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:13:33.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.342+0000 7f8347fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:13:33.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.342+0000 7f8347fff640 1 -- 172.21.15.42:0/1914486117 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8348073a30 con 0x7f8348135490 2024-09-16T04:13:33.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.342+0000 7f8347fff640 1 --2- 172.21.15.42:0/1914486117 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8348135490 0x7f834811bf70 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f833800c590 tx=0x7f833800ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:13:33.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.342+0000 7f833ffff640 10 client.?.objecter ms_handle_connect 0x7f8348135490 2024-09-16T04:13:33.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.342+0000 7f833ffff640 10 client.?.objecter resend_mon_ops 2024-09-16T04:13:33.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.342+0000 7f833ffff640 1 -- 172.21.15.42:0/1914486117 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8338015070 con 0x7f8348135490 2024-09-16T04:13:33.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.342+0000 7f833ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:13:33.360 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.342+0000 7f833ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:13:33.360 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.342+0000 7f833ffff640 10 monclient: dump: 2024-09-16T04:13:33.360 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:13:33.360 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:13:33.360 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:33.360 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:33.360 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:33.360 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:13:33.360 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:13:33.360 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:13:33.360 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:13:33.360 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:13:33.360 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.342+0000 7f833ffff640 10 monclient: _finish_auth 0 2024-09-16T04:13:33.360 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.342+0000 7f833ffff640 10 monclient: _check_auth_tickets 2024-09-16T04:13:33.361 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.342+0000 7f833ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:13:33.361 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.342+0000 7f834dcb1640 5 monclient: authenticate success, global_id 1034640 2024-09-16T04:13:33.361 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.342+0000 7f833ffff640 1 -- 172.21.15.42:0/1914486117 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8338011820 con 0x7f8348135490 2024-09-16T04:13:33.361 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.342+0000 7f834dcb1640 10 monclient: _renew_subs 2024-09-16T04:13:33.361 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.342+0000 7f834dcb1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:13:33.361 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.342+0000 7f834dcb1640 1 -- 172.21.15.42:0/1914486117 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f834806e7c0 con 0x7f8348135490 2024-09-16T04:13:33.361 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.342+0000 7f833ffff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:13:33.361 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.342+0000 7f833ffff640 1 -- 172.21.15.42:0/1914486117 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8338011c10 con 0x7f8348135490 2024-09-16T04:13:33.361 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.342+0000 7f833ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:13:33.361 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.342+0000 7f833ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:13:33.361 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.342+0000 7f833ffff640 10 monclient: dump: 2024-09-16T04:13:33.361 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:13:33.361 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:13:33.362 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:33.362 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:33.362 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:33.362 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:13:33.362 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:13:33.362 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:13:33.362 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:13:33.362 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:13:33.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.342+0000 7f83457fa640 4 set_mon_vals no callback set 2024-09-16T04:13:33.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.342+0000 7f834dcb1640 10 client.1034640.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:13:33.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.342+0000 7f834dcb1640 10 monclient: _renew_subs 2024-09-16T04:13:33.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.342+0000 7f834dcb1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:13:33.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.342+0000 7f834dcb1640 1 -- 172.21.15.42:0/1914486117 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f834811f600 con 0x7f8348135490 2024-09-16T04:13:33.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.342+0000 7f834dcb1640 1 librados: init done 2024-09-16T04:13:33.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.342+0000 7f834dcb1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:13:33.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.342+0000 7f834dcb1640 1 --2- 172.21.15.42:0/1914486117 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f834806dac0 0x7f834802c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:13:33.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.342+0000 7f834dcb1640 10 monclient(hunting): start opening mon connection 2024-09-16T04:13:33.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.342+0000 7f834dcb1640 1 -- 172.21.15.42:0/1914486117 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f83480a1710 con 0x7f834806dac0 2024-09-16T04:13:33.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.342+0000 7f83477fe640 1 --2- 172.21.15.42:0/1914486117 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f834806dac0 0x7f834802c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:13:33.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.342+0000 7f83477fe640 10 monclient: get_auth_request con 0x7f834806dac0 auth_method 0 2024-09-16T04:13:33.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.342+0000 7f83477fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:13:33.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.342+0000 7f83477fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:33.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.342+0000 7f83477fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:13:33.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.342+0000 7f833ffff640 1 -- 172.21.15.42:0/1914486117 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f8338018070 con 0x7f8348135490 2024-09-16T04:13:33.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.342+0000 7f83477fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:13:33.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.342+0000 7f83477fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:13:33.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.342+0000 7f83477fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:13:33.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.342+0000 7f83477fe640 10 monclient(hunting): handle_auth_done global_id 1034643 payload 824 2024-09-16T04:13:33.364 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.342+0000 7f83477fe640 1 -- 172.21.15.42:0/1914486117 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f834811ecb0 con 0x7f834806dac0 2024-09-16T04:13:33.364 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.342+0000 7f833ffff640 1 --2- 172.21.15.42:0/1914486117 >> v2:172.21.15.154:6812/254845342 conn(0x7f832c037810 0x7f832c039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:13:33.364 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.342+0000 7f833ffff640 1 -- 172.21.15.42:0/1914486117 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(706..706 src has 681..706) ==== 5040+0+0 (secure 0 0 0) 0x7f8338048b20 con 0x7f8348135490 2024-09-16T04:13:33.364 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.342+0000 7f833ffff640 10 client.1034640.objecter ms_dispatch 0x7f8348000ba0 osd_map(706..706 src has 681..706) 2024-09-16T04:13:33.364 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.342+0000 7f833ffff640 3 client.1034640.objecter handle_osd_map got epochs [706,706] > 0 2024-09-16T04:13:33.364 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.342+0000 7f833ffff640 3 client.1034640.objecter handle_osd_map decoding full epoch 706 2024-09-16T04:13:33.364 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.342+0000 7f833ffff640 20 client.1034640.objecter dump_active .. 0 homeless 2024-09-16T04:13:33.364 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.342+0000 7f83477fe640 1 --2- 172.21.15.42:0/1914486117 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f834806dac0 0x7f834802c650 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f834806da10 tx=0x7f8340019650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:13:33.364 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.342+0000 7f833ffff640 10 client.1034640.objecter ms_handle_connect 0x7f834806dac0 2024-09-16T04:13:33.364 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.342+0000 7f833ffff640 10 client.1034640.objecter resend_mon_ops 2024-09-16T04:13:33.364 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.342+0000 7f833ffff640 1 -- 172.21.15.42:0/1914486117 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83400bd070 con 0x7f834806dac0 2024-09-16T04:13:33.364 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.342+0000 7f833ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:13:33.364 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.342+0000 7f833ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:13:33.365 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.342+0000 7f833ffff640 10 monclient: dump: 2024-09-16T04:13:33.365 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:13:33.365 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:13:33.365 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:33.365 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:33.365 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:33.365 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:13:33.365 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:13:33.365 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:13:33.365 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:13:33.365 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:13:33.365 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.342+0000 7f8346ffd640 1 --2- 172.21.15.42:0/1914486117 >> v2:172.21.15.154:6812/254845342 conn(0x7f832c037810 0x7f832c039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:13:33.365 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.342+0000 7f8346ffd640 10 monclient: get_auth_request con 0x7f832c037810 auth_method 0 2024-09-16T04:13:33.365 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.342+0000 7f833ffff640 1 -- 172.21.15.42:0/1914486117 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f83400b6620 con 0x7f834806dac0 2024-09-16T04:13:33.366 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.342+0000 7f833ffff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:13:33.366 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.342+0000 7f833ffff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:13:33.366 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.342+0000 7f833ffff640 1 -- 172.21.15.42:0/1914486117 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f834806dac0 msgr2=0x7f834802c650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:33.366 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.342+0000 7f833ffff640 1 --2- 172.21.15.42:0/1914486117 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f834806dac0 0x7f834802c650 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f834806da10 tx=0x7f8340019650 comp rx=0 tx=0).stop 2024-09-16T04:13:33.366 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.342+0000 7f833ffff640 1 -- 172.21.15.42:0/1914486117 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f834806dac0 msgr2=0x7f834802c650 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:13:33.366 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.342+0000 7f833ffff640 1 --2- 172.21.15.42:0/1914486117 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f834806dac0 0x7f834802c650 secure :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f834806da10 tx=0x7f8340019650 comp rx=0 tx=0).stop 2024-09-16T04:13:33.366 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.346+0000 7f8346ffd640 1 --2- 172.21.15.42:0/1914486117 >> v2:172.21.15.154:6812/254845342 conn(0x7f832c037810 0x7f832c039cd0 secure :-1 s=READY pgs=3991 cs=0 l=1 rev1=1 crypto rx=0x7f834806e570 tx=0x7f8330009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:13:33.366 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.346+0000 7f833ffff640 10 client.1034640.objecter ms_handle_connect 0x7f832c037810 2024-09-16T04:13:33.368 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.350+0000 7f834dcb1640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:13:33.368 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.350+0000 7f834dcb1640 1 --2- 172.21.15.42:0/1914486117 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f834802cb90 0x7f834802cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:13:33.368 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.350+0000 7f834dcb1640 10 monclient(hunting): start opening mon connection 2024-09-16T04:13:33.368 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.350+0000 7f834dcb1640 1 -- 172.21.15.42:0/1914486117 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f834811df80 con 0x7f834802cb90 2024-09-16T04:13:33.368 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.350+0000 7f83477fe640 1 --2- 172.21.15.42:0/1914486117 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f834802cb90 0x7f834802cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:13:33.368 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.350+0000 7f83477fe640 10 monclient: get_auth_request con 0x7f834802cb90 auth_method 0 2024-09-16T04:13:33.368 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.350+0000 7f83477fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:13:33.368 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.350+0000 7f83477fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:33.368 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.350+0000 7f83477fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:13:33.368 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.350+0000 7f83477fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:13:33.369 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.350+0000 7f83477fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:13:33.369 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.350+0000 7f83477fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:13:33.369 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.350+0000 7f83477fe640 10 monclient(hunting): handle_auth_done global_id 1034646 payload 824 2024-09-16T04:13:33.369 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.350+0000 7f83477fe640 1 -- 172.21.15.42:0/1914486117 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f8348139a40 con 0x7f834802cb90 2024-09-16T04:13:33.369 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.350+0000 7f83477fe640 1 --2- 172.21.15.42:0/1914486117 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f834802cb90 0x7f834802cfa0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f834001c8f0 tx=0x7f83400a7000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:13:33.369 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.350+0000 7f833ffff640 10 client.1034640.objecter ms_handle_connect 0x7f834802cb90 2024-09-16T04:13:33.369 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.350+0000 7f833ffff640 10 client.1034640.objecter resend_mon_ops 2024-09-16T04:13:33.369 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.350+0000 7f833ffff640 1 -- 172.21.15.42:0/1914486117 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83400c3b50 con 0x7f834802cb90 2024-09-16T04:13:33.369 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.350+0000 7f833ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:13:33.369 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.350+0000 7f833ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:13:33.369 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.350+0000 7f833ffff640 10 monclient: dump: 2024-09-16T04:13:33.369 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:13:33.369 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:13:33.370 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:33.370 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:33.370 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:33.370 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:13:33.370 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:13:33.370 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:13:33.370 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:13:33.370 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:13:33.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.350+0000 7f833ffff640 1 -- 172.21.15.42:0/1914486117 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4715 (secure 0 0 0) 0x7f83400b67d0 con 0x7f834802cb90 2024-09-16T04:13:33.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.350+0000 7f833ffff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:13:33.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.350+0000 7f833ffff640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:13:33.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.350+0000 7f833ffff640 1 -- 172.21.15.42:0/1914486117 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f834802cb90 msgr2=0x7f834802cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:33.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.350+0000 7f833ffff640 1 --2- 172.21.15.42:0/1914486117 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f834802cb90 0x7f834802cfa0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f834001c8f0 tx=0x7f83400a7000 comp rx=0 tx=0).stop 2024-09-16T04:13:33.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.350+0000 7f833ffff640 1 -- 172.21.15.42:0/1914486117 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f834802cb90 msgr2=0x7f834802cfa0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:13:33.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.350+0000 7f833ffff640 1 --2- 172.21.15.42:0/1914486117 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f834802cb90 0x7f834802cfa0 secure :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f834001c8f0 tx=0x7f83400a7000 comp rx=0 tx=0).stop 2024-09-16T04:13:33.371 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T04:13:33.371 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:13:33.371 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:13:33.371 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "leader", 2024-09-16T04:13:33.371 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 286, 2024-09-16T04:13:33.371 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 293285, 2024-09-16T04:13:33.371 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T04:13:33.371 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T04:13:33.372 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T04:13:33.372 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T04:13:33.372 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:13:33.372 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 3, 2024-09-16T04:13:33.372 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:13:33.372 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T04:13:33.372 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T04:13:33.372 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:13:33.372 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:13:33.372 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:13:33.372 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:13:33.372 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:13:33.372 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:13:33.373 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:13:33.373 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:13:33.373 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:13:33.373 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:13:33.373 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:13:33.373 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:13:33.373 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:13:33.373 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T04:13:33.373 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:13:33.373 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:13:33.373 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:13:33.373 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:13:33.374 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:13:33.374 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:13:33.374 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:13:33.374 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:13:33.374 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:13:33.374 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:13:33.374 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:13:33.374 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:13:33.374 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:13:33.374 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T04:13:33.374 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T04:13:33.374 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T04:13:33.374 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T04:13:33.374 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T04:13:33.375 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T04:13:33.375 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:13:33.375 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:13:33.375 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T04:13:33.375 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T04:13:33.375 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T04:13:33.375 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T04:13:33.375 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T04:13:33.375 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T04:13:33.375 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T04:13:33.375 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:13:33.375 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T04:13:33.376 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:13:33.376 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:13:33.376 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:13:33.376 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:13:33.376 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:13:33.376 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:13:33.376 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:13:33.376 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:13:33.376 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:13:33.376 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:13:33.376 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:13:33.376 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:13:33.376 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T04:13:33.376 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:13:33.377 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T04:13:33.377 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:13:33.377 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:13:33.377 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:13:33.377 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:13:33.377 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:13:33.377 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:13:33.377 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:13:33.377 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T04:13:33.377 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:13:33.377 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:13:33.377 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:13:33.377 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:13:33.378 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T04:13:33.378 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:13:33.378 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:13:33.378 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:13:33.378 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:13:33.378 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T04:13:33.378 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T04:13:33.378 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:13:33.378 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:13:33.378 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:13:33.378 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:13:33.378 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:13:33.378 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:13:33.378 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:13:33.379 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:13:33.379 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:13:33.379 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:13:33.379 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:13:33.379 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T04:13:33.379 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:13:33.379 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:13:33.379 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:13:33.379 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:13:33.379 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T04:13:33.379 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:13:33.379 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:13:33.379 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:13:33.379 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:13:33.380 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T04:13:33.380 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T04:13:33.380 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:13:33.380 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:13:33.380 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:13:33.380 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:13:33.380 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:13:33.380 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:13:33.380 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:13:33.380 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:13:33.380 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:13:33.380 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:13:33.380 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:13:33.381 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T04:13:33.381 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:13:33.381 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:13:33.381 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:13:33.381 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:13:33.381 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T04:13:33.381 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:13:33.381 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:13:33.381 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:13:33.381 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:13:33.381 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T04:13:33.381 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T04:13:33.381 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:13:33.381 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:13:33.382 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:13:33.382 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:13:33.382 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:13:33.382 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:13:33.382 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T04:13:33.382 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T04:13:33.382 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:13:33.382 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:13:33.382 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:13:33.382 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:13:33.382 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:13:33.382 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:13:33.382 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T04:13:33.382 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:13:33.383 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:13:33.383 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:13:33.383 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 6 2024-09-16T04:13:33.383 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:13:33.383 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:13:33.383 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T04:13:33.383 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:13:33.383 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:13:33.383 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:13:33.383 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 6 2024-09-16T04:13:33.383 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:13:33.383 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:13:33.383 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:13:33.383 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T04:13:33.384 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T04:13:33.385 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.350+0000 7f834dcb1640 10 librados: watch_flush enter 2024-09-16T04:13:33.385 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.350+0000 7f834dcb1640 10 librados: watch_flush exit 2024-09-16T04:13:33.385 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.350+0000 7f834dcb1640 20 client.1034640.objecter shutdown clearing up homeless session... 2024-09-16T04:13:33.385 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.350+0000 7f834dcb1640 10 client.1034640.objecter successfully canceled tick 2024-09-16T04:13:33.386 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.350+0000 7f834dcb1640 1 -- 172.21.15.42:0/1914486117 >> v2:172.21.15.154:6812/254845342 conn(0x7f832c037810 msgr2=0x7f832c039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:33.386 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.350+0000 7f834dcb1640 1 --2- 172.21.15.42:0/1914486117 >> v2:172.21.15.154:6812/254845342 conn(0x7f832c037810 0x7f832c039cd0 secure :-1 s=READY pgs=3991 cs=0 l=1 rev1=1 crypto rx=0x7f834806e570 tx=0x7f8330009040 comp rx=0 tx=0).stop 2024-09-16T04:13:33.386 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.350+0000 7f834dcb1640 10 monclient: shutdown 2024-09-16T04:13:33.386 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.350+0000 7f834dcb1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:13:33.386 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.350+0000 7f834dcb1640 1 -- 172.21.15.42:0/1914486117 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8348135490 msgr2=0x7f834811bf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:33.386 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.350+0000 7f834dcb1640 1 --2- 172.21.15.42:0/1914486117 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8348135490 0x7f834811bf70 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f833800c590 tx=0x7f833800ca50 comp rx=0 tx=0).stop 2024-09-16T04:13:33.386 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.350+0000 7f8347fff640 1 -- 172.21.15.42:0/1914486117 reap_dead start 2024-09-16T04:13:33.386 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.350+0000 7f8347fff640 1 -- 172.21.15.42:0/1914486117 reap_dead start 2024-09-16T04:13:33.386 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.350+0000 7f834dcb1640 1 -- 172.21.15.42:0/1914486117 shutdown_connections 2024-09-16T04:13:33.386 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.350+0000 7f834dcb1640 1 -- 172.21.15.42:0/1914486117 >> 172.21.15.42:0/1914486117 conn(0x7f83480a2420 msgr2=0x7f834806fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:13:33.386 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.350+0000 7f834dcb1640 1 -- 172.21.15.42:0/1914486117 shutdown_connections 2024-09-16T04:13:33.386 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.350+0000 7f834dcb1640 1 -- 172.21.15.42:0/1914486117 wait complete. 2024-09-16T04:13:33.386 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.350+0000 7f834dcb1640 1 librados: shutdown 2024-09-16T04:13:33.387 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:13:33.387 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T04:13:33.387 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T04:13:33.505 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.498+0000 7f0aac99c640 10 monclient: get_monmap_and_config 2024-09-16T04:13:33.505 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.498+0000 7f0aac99c640 10 monclient: build_initial_monmap 2024-09-16T04:13:33.505 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.502+0000 7f0aac99c640 10 monclient: monmap: 2024-09-16T04:13:33.506 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:13:33.506 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:13:33.506 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:13:33.504136+0000 2024-09-16T04:13:33.506 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:13:33.504136+0000 2024-09-16T04:13:33.506 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:13:33.506 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:13:33.506 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:13:33.506 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:13:33.506 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:13:33.506 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:13:33.506 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:13:33.506 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:13:33.509 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.502+0000 7f0aac99c640 1 Processor -- start 2024-09-16T04:13:33.509 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.502+0000 7f0aac99c640 1 -- start start 2024-09-16T04:13:33.509 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.502+0000 7f0aac99c640 10 monclient: init 2024-09-16T04:13:33.509 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.502+0000 7f0aac99c640 10 monclient: _reopen_session rank -1 2024-09-16T04:13:33.509 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.502+0000 7f0aac99c640 10 monclient: _add_conns ranks=[4,3,0,2,1] 2024-09-16T04:13:33.509 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.502+0000 7f0aac99c640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f0aa80a1330 0x7f0aa80a1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:13:33.509 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.502+0000 7f0aac99c640 10 monclient(hunting): picked mon.noname-c con 0x7f0aa80a1330 addr v2:172.21.15.154:3300/0 2024-09-16T04:13:33.509 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.502+0000 7f0aac99c640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f0aa806dac0 0x7f0aa80a1c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:13:33.509 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.502+0000 7f0aac99c640 10 monclient(hunting): picked mon.noname-a con 0x7f0aa806dac0 addr v2:172.21.15.42:3300/0 2024-09-16T04:13:33.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.502+0000 7f0aac99c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0aa8135100 addr v1:172.21.15.42:6789/0 2024-09-16T04:13:33.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.502+0000 7f0aac99c640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0aa809c7d0 con 0x7f0aa8135100 2024-09-16T04:13:33.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.502+0000 7f0aac99c640 10 monclient(hunting): start opening mon connection 2024-09-16T04:13:33.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.502+0000 7f0aac99c640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f0aa809c090 con 0x7f0aa806dac0 2024-09-16T04:13:33.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.502+0000 7f0aac99c640 10 monclient(hunting): start opening mon connection 2024-09-16T04:13:33.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.502+0000 7f0aac99c640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f0aa809f610 con 0x7f0aa80a1330 2024-09-16T04:13:33.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.502+0000 7f0aac99c640 10 monclient(hunting): _renew_subs 2024-09-16T04:13:33.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.502+0000 7f0aac99c640 10 monclient(hunting): authenticate will time out at 2860.473602s 2024-09-16T04:13:33.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.502+0000 7f0aa5d9b640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f0aa806dac0 0x7f0aa80a1c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:13:33.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.502+0000 7f0aa5d9b640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f0aa806dac0 0x7f0aa80a1c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:52120/0 (socket says 172.21.15.42:52120) 2024-09-16T04:13:33.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.502+0000 7f0aa5d9b640 1 -- 172.21.15.42:0/1720303766 learned_addr learned my addr 172.21.15.42:0/1720303766 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:13:33.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.502+0000 7f0aa5d9b640 10 monclient(hunting): get_auth_request con 0x7f0aa806dac0 auth_method 0 2024-09-16T04:13:33.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.502+0000 7f0aa5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:13:33.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.502+0000 7f0aa5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:33.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.502+0000 7f0aa5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:13:33.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.502+0000 7f0aa5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:13:33.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.502+0000 7f0aa5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:13:33.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.502+0000 7f0aa5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:13:33.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.502+0000 7f0aa659c640 1 --2- 172.21.15.42:0/1720303766 >> v2:172.21.15.154:3300/0 conn(0x7f0aa80a1330 0x7f0aa80a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:13:33.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.502+0000 7f0aa659c640 10 monclient(hunting): get_auth_request con 0x7f0aa80a1330 auth_method 0 2024-09-16T04:13:33.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.502+0000 7f0aa659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:13:33.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.502+0000 7f0aa659c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:33.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.502+0000 7f0aa659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:13:33.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.502+0000 7f0aa5d9b640 10 monclient(hunting): handle_auth_done global_id 1034649 payload 274 2024-09-16T04:13:33.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.502+0000 7f0aa5d9b640 1 -- 172.21.15.42:0/1720303766 >> v2:172.21.15.154:3300/0 conn(0x7f0aa80a1330 msgr2=0x7f0aa80a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:13:33.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.502+0000 7f0aa5d9b640 1 --2- 172.21.15.42:0/1720303766 >> v2:172.21.15.154:3300/0 conn(0x7f0aa80a1330 0x7f0aa80a1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:13:33.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.502+0000 7f0aa5d9b640 1 -- 172.21.15.42:0/1720303766 >> v1:172.21.15.42:6789/0 conn(0x7f0aa8135100 legacy=0x7f0aa8135550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:33.513 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.502+0000 7f0aa5d9b640 10 monclient: _finish_hunting 0 2024-09-16T04:13:33.513 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.502+0000 7f0aa5d9b640 1 monclient: found mon.noname-a 2024-09-16T04:13:33.513 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.502+0000 7f0aa5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:13:33.513 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.502+0000 7f0aa5d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:13:33.513 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.502+0000 7f0aa5d9b640 1 -- 172.21.15.42:0/1720303766 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0aa809f8f0 con 0x7f0aa806dac0 2024-09-16T04:13:33.513 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.506+0000 7f0aa5d9b640 1 --2- 172.21.15.42:0/1720303766 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0aa806dac0 0x7f0aa80a1c50 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f0aa806add0 tx=0x7f0a9801c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8af2c36e64306383 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:13:33.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.506+0000 7f0aa559a640 1 -- 172.21.15.42:0/1720303766 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a980aa650 con 0x7f0aa806dac0 2024-09-16T04:13:33.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.506+0000 7f0aa559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:13:33.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.506+0000 7f0aa559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:13:33.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.506+0000 7f0aa559a640 10 monclient: dump: 2024-09-16T04:13:33.520 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:13:33.520 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:13:33.520 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:33.520 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:33.520 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:33.520 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:13:33.520 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:13:33.520 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:13:33.521 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:13:33.521 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:13:33.521 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.506+0000 7f0aa559a640 10 monclient: _finish_auth 0 2024-09-16T04:13:33.521 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.506+0000 7f0aa559a640 10 monclient: _check_auth_tickets 2024-09-16T04:13:33.521 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.506+0000 7f0aa559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:13:33.521 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.506+0000 7f0aa559a640 1 -- 172.21.15.42:0/1720303766 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a980af070 con 0x7f0aa806dac0 2024-09-16T04:13:33.521 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.506+0000 7f0aac99c640 5 monclient: authenticate success, global_id 1034649 2024-09-16T04:13:33.521 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.506+0000 7f0aac99c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:13:33.521 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.506+0000 7f0aa559a640 10 monclient: handle_config config(0 keys) 2024-09-16T04:13:33.521 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.506+0000 7f0aac99c640 10 monclient: get_monmap_and_config success 2024-09-16T04:13:33.521 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.506+0000 7f0aa559a640 1 -- 172.21.15.42:0/1720303766 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a980aaa10 con 0x7f0aa806dac0 2024-09-16T04:13:33.521 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.506+0000 7f0aa559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:13:33.521 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.506+0000 7f0aac99c640 4 set_mon_vals no callback set 2024-09-16T04:13:33.521 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.506+0000 7f0aa559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:13:33.522 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.506+0000 7f0aa559a640 10 monclient: dump: 2024-09-16T04:13:33.522 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:13:33.522 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:13:33.522 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:33.522 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:33.522 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:33.522 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:13:33.522 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:13:33.522 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:13:33.522 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:13:33.522 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:13:33.522 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.506+0000 7f0aac99c640 10 monclient: shutdown 2024-09-16T04:13:33.522 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.506+0000 7f0aac99c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:13:33.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.506+0000 7f0aac99c640 1 -- 172.21.15.42:0/1720303766 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0aa806dac0 msgr2=0x7f0aa80a1c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:33.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.506+0000 7f0aac99c640 1 --2- 172.21.15.42:0/1720303766 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0aa806dac0 0x7f0aa80a1c50 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f0aa806add0 tx=0x7f0a9801c4d0 comp rx=0 tx=0).stop 2024-09-16T04:13:33.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.506+0000 7f0aac99c640 1 -- 172.21.15.42:0/1720303766 shutdown_connections 2024-09-16T04:13:33.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.506+0000 7f0aac99c640 1 --2- 172.21.15.42:0/1720303766 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0aa806dac0 0x7f0aa80a1c50 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:13:33.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.506+0000 7f0aac99c640 1 --2- 172.21.15.42:0/1720303766 >> v2:172.21.15.154:3300/0 conn(0x7f0aa80a1330 0x7f0aa80a1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:13:33.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.506+0000 7f0aac99c640 1 -- 172.21.15.42:0/1720303766 >> 172.21.15.42:0/1720303766 conn(0x7f0aa80a2420 msgr2=0x7f0aa80a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:13:33.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.506+0000 7f0aac99c640 1 -- 172.21.15.42:0/1720303766 shutdown_connections 2024-09-16T04:13:33.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.506+0000 7f0aac99c640 1 -- 172.21.15.42:0/1720303766 wait complete. 2024-09-16T04:13:33.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.506+0000 7f0aac99c640 10 monclient: build_initial_monmap 2024-09-16T04:13:33.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.506+0000 7f0aac99c640 10 monclient: monmap: 2024-09-16T04:13:33.523 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:13:33.523 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:13:33.523 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:13:33.523 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:13:33.524 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:13:33.524 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:13:33.524 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:13:33.524 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:13:33.524 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:13:33.524 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:13:33.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.506+0000 7f0aac99c640 1 librados: starting msgr at 2024-09-16T04:13:33.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.506+0000 7f0aac99c640 1 librados: starting objecter 2024-09-16T04:13:33.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.506+0000 7f0aac99c640 1 Processor -- start 2024-09-16T04:13:33.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.506+0000 7f0aac99c640 1 -- start start 2024-09-16T04:13:33.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.506+0000 7f0aac99c640 1 librados: setting wanted keys 2024-09-16T04:13:33.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.506+0000 7f0aac99c640 1 librados: calling monclient init 2024-09-16T04:13:33.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.506+0000 7f0aac99c640 10 monclient: init 2024-09-16T04:13:33.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.506+0000 7f0aac99c640 10 monclient: _reopen_session rank -1 2024-09-16T04:13:33.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.506+0000 7f0aac99c640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:13:33.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.506+0000 7f0aac99c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0aa80a1330 0x7f0aa813c100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:13:33.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.506+0000 7f0aac99c640 10 monclient(hunting): picked mon.noname-b con 0x7f0aa80a1330 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:13:33.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.506+0000 7f0aac99c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0aa8135100 0x7f0aa8134530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:13:33.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.506+0000 7f0aac99c640 10 monclient(hunting): picked mon.noname-a con 0x7f0aa8135100 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:13:33.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.506+0000 7f0aac99c640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0aa8134a70 0x7f0aa809cb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:13:33.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.506+0000 7f0aac99c640 10 monclient(hunting): picked mon.noname-c con 0x7f0aa8134a70 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:13:33.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.506+0000 7f0aac99c640 10 monclient(hunting): start opening mon connection 2024-09-16T04:13:33.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.506+0000 7f0aac99c640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f0aa813fb80 con 0x7f0aa8135100 2024-09-16T04:13:33.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.506+0000 7f0aac99c640 10 monclient(hunting): start opening mon connection 2024-09-16T04:13:33.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.506+0000 7f0aac99c640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f0aa809ef80 con 0x7f0aa80a1330 2024-09-16T04:13:33.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.506+0000 7f0aac99c640 10 monclient(hunting): start opening mon connection 2024-09-16T04:13:33.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.506+0000 7f0aac99c640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f0aa813fd00 con 0x7f0aa8134a70 2024-09-16T04:13:33.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.506+0000 7f0aac99c640 10 monclient(hunting): _renew_subs 2024-09-16T04:13:33.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.506+0000 7f0aac99c640 10 monclient(hunting): authenticate will time out at 2860.478694s 2024-09-16T04:13:33.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.506+0000 7f0aa5d9b640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0aa8135100 0x7f0aa8134530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:13:33.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.506+0000 7f0aa5d9b640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0aa8135100 0x7f0aa8134530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:52132/0 (socket says 172.21.15.42:52132) 2024-09-16T04:13:33.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.506+0000 7f0aa5d9b640 1 -- 172.21.15.42:0/2346172697 learned_addr learned my addr 172.21.15.42:0/2346172697 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:13:33.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.506+0000 7f0aa5d9b640 10 monclient(hunting): get_auth_request con 0x7f0aa8135100 auth_method 0 2024-09-16T04:13:33.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.506+0000 7f0aa5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:13:33.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.506+0000 7f0aa5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:33.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.506+0000 7f0aa5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:13:33.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.506+0000 7f0aa6d9d640 1 --2- 172.21.15.42:0/2346172697 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0aa8134a70 0x7f0aa809cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:13:33.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.506+0000 7f0aa6d9d640 10 monclient(hunting): get_auth_request con 0x7f0aa8134a70 auth_method 0 2024-09-16T04:13:33.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.506+0000 7f0aa6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:13:33.526 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.506+0000 7f0aa6d9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:33.527 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.506+0000 7f0aa6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:13:33.527 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.506+0000 7f0aa5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:13:33.527 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.506+0000 7f0aa5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:13:33.527 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.506+0000 7f0aa5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:13:33.527 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.506+0000 7f0aa6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:13:33.527 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.506+0000 7f0aa6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:13:33.527 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.506+0000 7f0aa6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:13:33.527 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.506+0000 7f0aa5d9b640 10 monclient(hunting): handle_auth_done global_id 1034652 payload 824 2024-09-16T04:13:33.527 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.506+0000 7f0aa5d9b640 1 -- 172.21.15.42:0/2346172697 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0aa8134a70 msgr2=0x7f0aa809cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:33.527 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.506+0000 7f0aa5d9b640 1 --2- 172.21.15.42:0/2346172697 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0aa8134a70 0x7f0aa809cb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:13:33.527 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.506+0000 7f0aa659c640 1 --2- 172.21.15.42:0/2346172697 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0aa80a1330 0x7f0aa813c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:13:33.527 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.506+0000 7f0aa5d9b640 1 -- 172.21.15.42:0/2346172697 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0aa80a1330 msgr2=0x7f0aa813c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:33.527 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.506+0000 7f0aa5d9b640 1 --2- 172.21.15.42:0/2346172697 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0aa80a1330 0x7f0aa813c100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:13:33.528 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.506+0000 7f0aa5d9b640 10 monclient: _finish_hunting 0 2024-09-16T04:13:33.528 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.506+0000 7f0aa5d9b640 1 monclient: found mon.noname-a 2024-09-16T04:13:33.528 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.506+0000 7f0aa5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:13:33.528 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.506+0000 7f0aa5d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:13:33.528 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.506+0000 7f0aa5d9b640 1 -- 172.21.15.42:0/2346172697 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0aa81cf690 con 0x7f0aa8135100 2024-09-16T04:13:33.528 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.506+0000 7f0aa6d9d640 1 --2- 172.21.15.42:0/2346172697 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0aa8134a70 0x7f0aa809cb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:13:33.528 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.506+0000 7f0aa5d9b640 1 --2- 172.21.15.42:0/2346172697 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0aa8135100 0x7f0aa8134530 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f0a980190e0 tx=0x7f0a9801ad10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:13:33.528 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.506+0000 7f0a96ffd640 10 client.?.objecter ms_handle_connect 0x7f0aa8135100 2024-09-16T04:13:33.528 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.506+0000 7f0a96ffd640 10 client.?.objecter resend_mon_ops 2024-09-16T04:13:33.528 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.506+0000 7f0a96ffd640 1 -- 172.21.15.42:0/2346172697 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a980bd5f0 con 0x7f0aa8135100 2024-09-16T04:13:33.528 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.506+0000 7f0a96ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:13:33.528 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.506+0000 7f0a96ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:13:33.528 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.506+0000 7f0a96ffd640 10 monclient: dump: 2024-09-16T04:13:33.529 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:13:33.529 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:13:33.529 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:33.529 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:33.529 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:33.529 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:13:33.529 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:13:33.529 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:13:33.529 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:13:33.529 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:13:33.530 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.506+0000 7f0a96ffd640 10 monclient: _finish_auth 0 2024-09-16T04:13:33.530 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.506+0000 7f0a96ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:13:33.530 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.506+0000 7f0a96ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:13:33.530 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.506+0000 7f0a96ffd640 1 -- 172.21.15.42:0/2346172697 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a980a6040 con 0x7f0aa8135100 2024-09-16T04:13:33.530 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.506+0000 7f0a96ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:13:33.530 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.506+0000 7f0a96ffd640 1 -- 172.21.15.42:0/2346172697 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a980bd990 con 0x7f0aa8135100 2024-09-16T04:13:33.530 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.506+0000 7f0aac99c640 5 monclient: authenticate success, global_id 1034652 2024-09-16T04:13:33.530 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.506+0000 7f0aac99c640 10 monclient: _renew_subs 2024-09-16T04:13:33.530 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.506+0000 7f0aac99c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:13:33.530 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.506+0000 7f0aac99c640 1 -- 172.21.15.42:0/2346172697 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0aa80a50c0 con 0x7f0aa8135100 2024-09-16T04:13:33.530 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.506+0000 7f0a97fff640 4 set_mon_vals no callback set 2024-09-16T04:13:33.530 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.506+0000 7f0a96ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:13:33.530 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.506+0000 7f0a96ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:13:33.530 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.506+0000 7f0a96ffd640 10 monclient: dump: 2024-09-16T04:13:33.531 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:13:33.531 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:13:33.531 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:33.531 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:33.531 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:33.531 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:13:33.531 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:13:33.531 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:13:33.531 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:13:33.531 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:13:33.531 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.506+0000 7f0aac99c640 10 client.1034652.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:13:33.531 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.506+0000 7f0aac99c640 10 monclient: _renew_subs 2024-09-16T04:13:33.531 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.506+0000 7f0aac99c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:13:33.531 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.506+0000 7f0aac99c640 1 -- 172.21.15.42:0/2346172697 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0aa8073a30 con 0x7f0aa8135100 2024-09-16T04:13:33.532 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.506+0000 7f0aac99c640 1 librados: init done 2024-09-16T04:13:33.532 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.510+0000 7f0aac99c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:13:33.532 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.510+0000 7f0aac99c640 1 --2- 172.21.15.42:0/2346172697 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0aa806dac0 0x7f0aa8120740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:13:33.532 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.510+0000 7f0aac99c640 10 monclient(hunting): start opening mon connection 2024-09-16T04:13:33.532 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.510+0000 7f0aac99c640 1 -- 172.21.15.42:0/2346172697 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f0aa8135550 con 0x7f0aa806dac0 2024-09-16T04:13:33.532 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.510+0000 7f0a96ffd640 1 -- 172.21.15.42:0/2346172697 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f0a980c6440 con 0x7f0aa8135100 2024-09-16T04:13:33.532 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.510+0000 7f0aa659c640 1 --2- 172.21.15.42:0/2346172697 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0aa806dac0 0x7f0aa8120740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:13:33.532 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.510+0000 7f0aa659c640 10 monclient: get_auth_request con 0x7f0aa806dac0 auth_method 0 2024-09-16T04:13:33.532 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.510+0000 7f0aa659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:13:33.532 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.510+0000 7f0aa659c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:33.532 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.510+0000 7f0aa659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:13:33.532 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.510+0000 7f0aa659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:13:33.532 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.510+0000 7f0aa659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:13:33.533 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.510+0000 7f0aa659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:13:33.533 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.510+0000 7f0a96ffd640 1 --2- 172.21.15.42:0/2346172697 >> v2:172.21.15.154:6812/254845342 conn(0x7f0a88037870 0x7f0a88039d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:13:33.533 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.510+0000 7f0a96ffd640 1 -- 172.21.15.42:0/2346172697 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(706..706 src has 681..706) ==== 5040+0+0 (secure 0 0 0) 0x7f0a980e3500 con 0x7f0aa8135100 2024-09-16T04:13:33.533 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.510+0000 7f0a96ffd640 10 client.1034652.objecter ms_dispatch 0x7f0aa8000ba0 osd_map(706..706 src has 681..706) 2024-09-16T04:13:33.533 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.510+0000 7f0a96ffd640 3 client.1034652.objecter handle_osd_map got epochs [706,706] > 0 2024-09-16T04:13:33.533 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.510+0000 7f0a96ffd640 3 client.1034652.objecter handle_osd_map decoding full epoch 706 2024-09-16T04:13:33.533 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.510+0000 7f0aa659c640 10 monclient(hunting): handle_auth_done global_id 1094135 payload 824 2024-09-16T04:13:33.533 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.510+0000 7f0aa659c640 1 -- 172.21.15.42:0/2346172697 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0aa8120380 con 0x7f0aa806dac0 2024-09-16T04:13:33.533 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.510+0000 7f0a96ffd640 20 client.1034652.objecter dump_active .. 0 homeless 2024-09-16T04:13:33.533 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.510+0000 7f0a96ffd640 10 client.1034652.objecter ms_handle_connect 0x7f0aa806dac0 2024-09-16T04:13:33.533 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.510+0000 7f0a96ffd640 10 client.1034652.objecter resend_mon_ops 2024-09-16T04:13:33.533 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.510+0000 7f0aa659c640 1 --2- 172.21.15.42:0/2346172697 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0aa806dac0 0x7f0aa8120740 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f0a9c00e710 tx=0x7f0a9c00ebd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:13:33.534 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.510+0000 7f0a96ffd640 1 -- 172.21.15.42:0/2346172697 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a9c017070 con 0x7f0aa806dac0 2024-09-16T04:13:33.534 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.510+0000 7f0a96ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:13:33.534 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.510+0000 7f0a96ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:13:33.534 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.510+0000 7f0aa6d9d640 1 --2- 172.21.15.42:0/2346172697 >> v2:172.21.15.154:6812/254845342 conn(0x7f0a88037870 0x7f0a88039d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:13:33.534 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.510+0000 7f0a96ffd640 10 monclient: dump: 2024-09-16T04:13:33.534 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:13:33.534 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:13:33.534 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:33.534 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:33.534 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:33.534 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:13:33.534 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:13:33.534 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:13:33.534 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:13:33.535 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:13:33.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.510+0000 7f0aa6d9d640 10 monclient: get_auth_request con 0x7f0a88037870 auth_method 0 2024-09-16T04:13:33.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.510+0000 7f0a96ffd640 1 -- 172.21.15.42:0/2346172697 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f0a9c00c740 con 0x7f0aa806dac0 2024-09-16T04:13:33.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.510+0000 7f0a96ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:13:33.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.510+0000 7f0a96ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:13:33.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.510+0000 7f0a96ffd640 1 -- 172.21.15.42:0/2346172697 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0aa806dac0 msgr2=0x7f0aa8120740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:33.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.510+0000 7f0a96ffd640 1 --2- 172.21.15.42:0/2346172697 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0aa806dac0 0x7f0aa8120740 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f0a9c00e710 tx=0x7f0a9c00ebd0 comp rx=0 tx=0).stop 2024-09-16T04:13:33.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.510+0000 7f0a96ffd640 1 -- 172.21.15.42:0/2346172697 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0aa806dac0 msgr2=0x7f0aa8120740 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:13:33.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.510+0000 7f0a96ffd640 1 --2- 172.21.15.42:0/2346172697 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0aa806dac0 0x7f0aa8120740 secure :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f0a9c00e710 tx=0x7f0a9c00ebd0 comp rx=0 tx=0).stop 2024-09-16T04:13:33.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.510+0000 7f0a96ffd640 10 client.1034652.objecter ms_handle_connect 0x7f0a88037870 2024-09-16T04:13:33.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.510+0000 7f0aa6d9d640 1 --2- 172.21.15.42:0/2346172697 >> v2:172.21.15.154:6812/254845342 conn(0x7f0a88037870 0x7f0a88039d30 secure :-1 s=READY pgs=3992 cs=0 l=1 rev1=1 crypto rx=0x7f0aa0002210 tx=0x7f0aa0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:13:33.537 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.514+0000 7f0aac99c640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:13:33.537 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.514+0000 7f0aac99c640 1 --2- 172.21.15.42:0/2346172697 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0aa802c650 0x7f0aa802ca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:13:33.537 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.514+0000 7f0aac99c640 10 monclient(hunting): start opening mon connection 2024-09-16T04:13:33.538 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.514+0000 7f0aac99c640 1 -- 172.21.15.42:0/2346172697 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f0aa8138a40 con 0x7f0aa802c650 2024-09-16T04:13:33.538 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.514+0000 7f0aa659c640 1 --2- 172.21.15.42:0/2346172697 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0aa802c650 0x7f0aa802ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:13:33.538 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.514+0000 7f0aa659c640 10 monclient: get_auth_request con 0x7f0aa802c650 auth_method 0 2024-09-16T04:13:33.538 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.514+0000 7f0aa659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:13:33.538 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.514+0000 7f0aa659c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:33.538 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.514+0000 7f0aa659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:13:33.538 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.514+0000 7f0aa659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:13:33.538 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.514+0000 7f0aa659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:13:33.538 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.514+0000 7f0aa659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:13:33.538 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.518+0000 7f0aa659c640 10 monclient(hunting): handle_auth_done global_id 1094138 payload 824 2024-09-16T04:13:33.538 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.518+0000 7f0aa659c640 1 -- 172.21.15.42:0/2346172697 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f0aa8120c80 con 0x7f0aa802c650 2024-09-16T04:13:33.538 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.518+0000 7f0aa659c640 1 --2- 172.21.15.42:0/2346172697 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0aa802c650 0x7f0aa802ca30 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f0a9c008010 tx=0x7f0a9c004740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:13:33.538 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.518+0000 7f0a96ffd640 10 client.1034652.objecter ms_handle_connect 0x7f0aa802c650 2024-09-16T04:13:33.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.518+0000 7f0a96ffd640 10 client.1034652.objecter resend_mon_ops 2024-09-16T04:13:33.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.518+0000 7f0a96ffd640 1 -- 172.21.15.42:0/2346172697 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a9c00f040 con 0x7f0aa802c650 2024-09-16T04:13:33.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.518+0000 7f0a96ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:13:33.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.518+0000 7f0a96ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:13:33.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.518+0000 7f0a96ffd640 10 monclient: dump: 2024-09-16T04:13:33.539 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:13:33.539 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:13:33.539 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:33.539 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:33.539 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:33.539 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:13:33.539 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:13:33.539 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:13:33.539 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:13:33.540 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:13:33.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.518+0000 7f0a96ffd640 1 -- 172.21.15.42:0/2346172697 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4545 (secure 0 0 0) 0x7f0a9c013410 con 0x7f0aa802c650 2024-09-16T04:13:33.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.518+0000 7f0a96ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:13:33.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.518+0000 7f0a96ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:13:33.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.518+0000 7f0a96ffd640 1 -- 172.21.15.42:0/2346172697 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0aa802c650 msgr2=0x7f0aa802ca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:33.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.518+0000 7f0a96ffd640 1 --2- 172.21.15.42:0/2346172697 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0aa802c650 0x7f0aa802ca30 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f0a9c008010 tx=0x7f0a9c004740 comp rx=0 tx=0).stop 2024-09-16T04:13:33.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.518+0000 7f0a96ffd640 1 -- 172.21.15.42:0/2346172697 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0aa802c650 msgr2=0x7f0aa802ca30 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:13:33.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.518+0000 7f0a96ffd640 1 --2- 172.21.15.42:0/2346172697 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0aa802c650 0x7f0aa802ca30 secure :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f0a9c008010 tx=0x7f0a9c004740 comp rx=0 tx=0).stop 2024-09-16T04:13:33.540 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T04:13:33.541 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:13:33.541 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:13:33.541 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T04:13:33.541 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 286, 2024-09-16T04:13:33.541 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 12559, 2024-09-16T04:13:33.541 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T04:13:33.541 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T04:13:33.541 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T04:13:33.541 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T04:13:33.541 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:13:33.541 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 3, 2024-09-16T04:13:33.541 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:13:33.541 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T04:13:33.542 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T04:13:33.542 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:13:33.542 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:13:33.542 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:13:33.542 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:13:33.542 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:13:33.542 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:13:33.542 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:13:33.542 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:13:33.542 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:13:33.542 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:13:33.542 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:13:33.542 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:13:33.543 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:13:33.543 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T04:13:33.543 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:13:33.543 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:13:33.543 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:13:33.543 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:13:33.543 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:13:33.543 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:13:33.543 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:13:33.543 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:13:33.543 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:13:33.543 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:13:33.544 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:13:33.544 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:13:33.544 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:13:33.544 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T04:13:33.544 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T04:13:33.544 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T04:13:33.544 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T04:13:33.544 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T04:13:33.544 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T04:13:33.544 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:13:33.544 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:13:33.544 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T04:13:33.545 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T04:13:33.545 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T04:13:33.545 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T04:13:33.546 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T04:13:33.546 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T04:13:33.546 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T04:13:33.547 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:13:33.547 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T04:13:33.547 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:13:33.547 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:13:33.547 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:13:33.547 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:13:33.547 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:13:33.547 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:13:33.547 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:13:33.547 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:13:33.547 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:13:33.547 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:13:33.547 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:13:33.548 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:13:33.548 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T04:13:33.548 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:13:33.548 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T04:13:33.548 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:13:33.548 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:13:33.548 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:13:33.548 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:13:33.548 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:13:33.548 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:13:33.548 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:13:33.548 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T04:13:33.549 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:13:33.549 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:13:33.549 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:13:33.549 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:13:33.549 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T04:13:33.549 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:13:33.549 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:13:33.549 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:13:33.549 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:13:33.549 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T04:13:33.549 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T04:13:33.549 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:13:33.550 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:13:33.550 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:13:33.550 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:13:33.550 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:13:33.550 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:13:33.550 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:13:33.550 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:13:33.550 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:13:33.550 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:13:33.550 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:13:33.550 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T04:13:33.550 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:13:33.550 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:13:33.551 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:13:33.551 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:13:33.551 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T04:13:33.551 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:13:33.551 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:13:33.551 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:13:33.551 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:13:33.551 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T04:13:33.551 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T04:13:33.551 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:13:33.551 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:13:33.551 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:13:33.551 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:13:33.551 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:13:33.552 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:13:33.552 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:13:33.552 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:13:33.552 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:13:33.552 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:13:33.552 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:13:33.552 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T04:13:33.552 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:13:33.552 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:13:33.552 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:13:33.552 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:13:33.552 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T04:13:33.552 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:13:33.552 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:13:33.553 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:13:33.553 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:13:33.553 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T04:13:33.553 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T04:13:33.553 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:13:33.553 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:13:33.553 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:13:33.553 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:13:33.553 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:13:33.553 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:13:33.553 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T04:13:33.553 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T04:13:33.553 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:13:33.554 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:13:33.555 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:13:33.555 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:13:33.555 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:13:33.555 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:13:33.556 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T04:13:33.556 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:13:33.556 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:13:33.556 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:13:33.556 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:13:33.557 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:13:33.557 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:13:33.557 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:13:33.557 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T04:13:33.557 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T04:13:33.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.518+0000 7f0aac99c640 10 librados: watch_flush enter 2024-09-16T04:13:33.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.518+0000 7f0aac99c640 10 librados: watch_flush exit 2024-09-16T04:13:33.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.518+0000 7f0aac99c640 20 client.1034652.objecter shutdown clearing up homeless session... 2024-09-16T04:13:33.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.518+0000 7f0aac99c640 10 client.1034652.objecter successfully canceled tick 2024-09-16T04:13:33.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.518+0000 7f0aac99c640 1 -- 172.21.15.42:0/2346172697 >> v2:172.21.15.154:6812/254845342 conn(0x7f0a88037870 msgr2=0x7f0a88039d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:33.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.518+0000 7f0aac99c640 1 --2- 172.21.15.42:0/2346172697 >> v2:172.21.15.154:6812/254845342 conn(0x7f0a88037870 0x7f0a88039d30 secure :-1 s=READY pgs=3992 cs=0 l=1 rev1=1 crypto rx=0x7f0aa0002210 tx=0x7f0aa0009040 comp rx=0 tx=0).stop 2024-09-16T04:13:33.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.518+0000 7f0aac99c640 10 monclient: shutdown 2024-09-16T04:13:33.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.518+0000 7f0aac99c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:13:33.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.518+0000 7f0aac99c640 1 -- 172.21.15.42:0/2346172697 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0aa8135100 msgr2=0x7f0aa8134530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:33.560 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.518+0000 7f0aac99c640 1 --2- 172.21.15.42:0/2346172697 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0aa8135100 0x7f0aa8134530 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f0a980190e0 tx=0x7f0a9801ad10 comp rx=0 tx=0).stop 2024-09-16T04:13:33.560 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.518+0000 7f0aa6d9d640 1 -- 172.21.15.42:0/2346172697 reap_dead start 2024-09-16T04:13:33.560 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.518+0000 7f0aac99c640 1 -- 172.21.15.42:0/2346172697 shutdown_connections 2024-09-16T04:13:33.560 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.518+0000 7f0aac99c640 1 --2- 172.21.15.42:0/2346172697 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0aa8135100 0x7f0aa8134530 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:13:33.560 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.518+0000 7f0aac99c640 1 -- 172.21.15.42:0/2346172697 >> 172.21.15.42:0/2346172697 conn(0x7f0aa80a2420 msgr2=0x7f0aa80a4140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:13:33.560 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.518+0000 7f0aac99c640 1 -- 172.21.15.42:0/2346172697 shutdown_connections 2024-09-16T04:13:33.560 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.518+0000 7f0aac99c640 1 -- 172.21.15.42:0/2346172697 wait complete. 2024-09-16T04:13:33.560 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.518+0000 7f0aac99c640 1 librados: shutdown 2024-09-16T04:13:33.560 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:13:33.561 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T04:13:33.561 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T04:13:33.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.678+0000 7f3ebbaae640 10 monclient: get_monmap_and_config 2024-09-16T04:13:33.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.678+0000 7f3ebbaae640 10 monclient: build_initial_monmap 2024-09-16T04:13:33.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.678+0000 7f3ebbaae640 10 monclient: monmap: 2024-09-16T04:13:33.682 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:13:33.682 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:13:33.682 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:13:33.680527+0000 2024-09-16T04:13:33.682 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:13:33.680527+0000 2024-09-16T04:13:33.682 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:13:33.682 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:13:33.682 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:13:33.682 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:13:33.682 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:13:33.682 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:13:33.683 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:13:33.683 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:13:33.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.678+0000 7f3ebbaae640 1 Processor -- start 2024-09-16T04:13:33.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.678+0000 7f3ebbaae640 1 -- start start 2024-09-16T04:13:33.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.678+0000 7f3ebbaae640 10 monclient: init 2024-09-16T04:13:33.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.678+0000 7f3ebbaae640 10 monclient: _reopen_session rank -1 2024-09-16T04:13:33.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.678+0000 7f3ebbaae640 10 monclient: _add_conns ranks=[3,4,2,1,0] 2024-09-16T04:13:33.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.678+0000 7f3ebbaae640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f3eb40a39b0 0x7f3eb4133b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:13:33.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.678+0000 7f3ebbaae640 10 monclient(hunting): picked mon.noname-a con 0x7f3eb40a39b0 addr v2:172.21.15.42:3300/0 2024-09-16T04:13:33.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.678+0000 7f3ebbaae640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f3eb40a3d90 0x7f3eb4137000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:13:33.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.678+0000 7f3ebbaae640 10 monclient(hunting): picked mon.noname-c con 0x7f3eb40a3d90 addr v2:172.21.15.154:3300/0 2024-09-16T04:13:33.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.678+0000 7f3ebbaae640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3eb4137540 0x7f3eb4137990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:13:33.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.678+0000 7f3ebbaae640 10 monclient(hunting): picked mon.noname-b con 0x7f3eb4137540 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:13:33.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.678+0000 7f3ebbaae640 10 monclient(hunting): start opening mon connection 2024-09-16T04:13:33.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.678+0000 7f3ebbaae640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f3eb409c140 con 0x7f3eb40a39b0 2024-09-16T04:13:33.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.678+0000 7f3ebbaae640 10 monclient(hunting): start opening mon connection 2024-09-16T04:13:33.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.678+0000 7f3ebbaae640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f3eb409c420 con 0x7f3eb40a3d90 2024-09-16T04:13:33.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.678+0000 7f3ebbaae640 10 monclient(hunting): start opening mon connection 2024-09-16T04:13:33.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.678+0000 7f3ebbaae640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f3eb406dd60 con 0x7f3eb4137540 2024-09-16T04:13:33.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.678+0000 7f3ebbaae640 10 monclient(hunting): _renew_subs 2024-09-16T04:13:33.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.678+0000 7f3ebbaae640 10 monclient(hunting): authenticate will time out at 2860.650574s 2024-09-16T04:13:33.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.678+0000 7f3eb984a640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f3eb40a39b0 0x7f3eb4133b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:13:33.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.678+0000 7f3eba04b640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3eb4137540 0x7f3eb4137990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:13:33.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.678+0000 7f3eb984a640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f3eb40a39b0 0x7f3eb4133b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:52136/0 (socket says 172.21.15.42:52136) 2024-09-16T04:13:33.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.678+0000 7f3eb984a640 1 -- 172.21.15.42:0/1361228050 learned_addr learned my addr 172.21.15.42:0/1361228050 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:13:33.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.678+0000 7f3eb984a640 10 monclient(hunting): get_auth_request con 0x7f3eb40a39b0 auth_method 0 2024-09-16T04:13:33.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.678+0000 7f3eb984a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:13:33.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.678+0000 7f3eb984a640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:33.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.678+0000 7f3eb984a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:13:33.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.678+0000 7f3eba04b640 10 monclient(hunting): get_auth_request con 0x7f3eb4137540 auth_method 0 2024-09-16T04:13:33.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.678+0000 7f3eba04b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:13:33.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.678+0000 7f3eba04b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:33.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.678+0000 7f3eba04b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:13:33.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.678+0000 7f3eb984a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:13:33.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.678+0000 7f3eb984a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:13:33.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.678+0000 7f3eb984a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:13:33.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.678+0000 7f3eb9049640 1 --2- 172.21.15.42:0/1361228050 >> v2:172.21.15.154:3300/0 conn(0x7f3eb40a3d90 0x7f3eb4137000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:13:33.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.678+0000 7f3eba04b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:13:33.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.678+0000 7f3eba04b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:13:33.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.678+0000 7f3eba04b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:13:33.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.678+0000 7f3eb9049640 10 monclient(hunting): get_auth_request con 0x7f3eb40a3d90 auth_method 0 2024-09-16T04:13:33.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.678+0000 7f3eb9049640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:13:33.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.678+0000 7f3eb9049640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:33.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.678+0000 7f3eb9049640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:13:33.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.678+0000 7f3eb984a640 10 monclient(hunting): handle_auth_done global_id 1034655 payload 274 2024-09-16T04:13:33.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.678+0000 7f3eb984a640 1 -- 172.21.15.42:0/1361228050 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3eb4137540 msgr2=0x7f3eb4137990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:13:33.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.678+0000 7f3eb984a640 1 --2- 172.21.15.42:0/1361228050 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3eb4137540 0x7f3eb4137990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:13:33.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.678+0000 7f3eb984a640 1 -- 172.21.15.42:0/1361228050 >> v2:172.21.15.154:3300/0 conn(0x7f3eb40a3d90 msgr2=0x7f3eb4137000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:13:33.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.678+0000 7f3eb984a640 1 --2- 172.21.15.42:0/1361228050 >> v2:172.21.15.154:3300/0 conn(0x7f3eb40a3d90 0x7f3eb4137000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:13:33.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.678+0000 7f3eb984a640 10 monclient: _finish_hunting 0 2024-09-16T04:13:33.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.678+0000 7f3eb984a640 1 monclient: found mon.noname-a 2024-09-16T04:13:33.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.678+0000 7f3eb984a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:13:33.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.678+0000 7f3eb984a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:13:33.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.678+0000 7f3eb984a640 1 -- 172.21.15.42:0/1361228050 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3eb40a5e20 con 0x7f3eb40a39b0 2024-09-16T04:13:33.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.682+0000 7f3eb984a640 1 --2- 172.21.15.42:0/1361228050 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3eb40a39b0 0x7f3eb4133b80 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f3eb40a56c0 tx=0x7f3eb001c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8951855768ceaa31 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:13:33.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.682+0000 7f3eb8848640 1 -- 172.21.15.42:0/1361228050 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3eb00b0070 con 0x7f3eb40a39b0 2024-09-16T04:13:33.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.682+0000 7f3eb8848640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:13:33.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.682+0000 7f3eb8848640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:13:33.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.682+0000 7f3eb8848640 10 monclient: dump: 2024-09-16T04:13:33.696 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:13:33.696 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:13:33.696 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:33.696 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:33.696 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:33.696 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:13:33.696 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:13:33.696 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:13:33.696 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:13:33.696 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:13:33.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.682+0000 7f3eb8848640 10 monclient: _finish_auth 0 2024-09-16T04:13:33.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.682+0000 7f3eb8848640 10 monclient: _check_auth_tickets 2024-09-16T04:13:33.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.682+0000 7f3eb8848640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:13:33.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.682+0000 7f3eb8848640 1 -- 172.21.15.42:0/1361228050 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3eb00ab600 con 0x7f3eb40a39b0 2024-09-16T04:13:33.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.682+0000 7f3eb8848640 10 monclient: handle_config config(0 keys) 2024-09-16T04:13:33.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.682+0000 7f3eb8848640 1 -- 172.21.15.42:0/1361228050 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3eb00ab9d0 con 0x7f3eb40a39b0 2024-09-16T04:13:33.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.682+0000 7f3eb8848640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:13:33.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.682+0000 7f3eb8848640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:13:33.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.682+0000 7f3eb8848640 10 monclient: dump: 2024-09-16T04:13:33.697 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:13:33.697 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:13:33.697 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:33.697 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:33.697 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:33.698 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:13:33.698 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:13:33.698 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:13:33.698 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:13:33.698 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:13:33.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.682+0000 7f3ebbaae640 5 monclient: authenticate success, global_id 1034655 2024-09-16T04:13:33.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.682+0000 7f3ebbaae640 10 monclient: get_monmap_and_config success 2024-09-16T04:13:33.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.682+0000 7f3ebbaae640 4 set_mon_vals no callback set 2024-09-16T04:13:33.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.682+0000 7f3ebbaae640 10 monclient: shutdown 2024-09-16T04:13:33.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.682+0000 7f3ebbaae640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:13:33.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.682+0000 7f3ebbaae640 1 -- 172.21.15.42:0/1361228050 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3eb40a39b0 msgr2=0x7f3eb4133b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:33.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.682+0000 7f3ebbaae640 1 --2- 172.21.15.42:0/1361228050 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3eb40a39b0 0x7f3eb4133b80 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f3eb40a56c0 tx=0x7f3eb001c3e0 comp rx=0 tx=0).stop 2024-09-16T04:13:33.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.682+0000 7f3ebbaae640 1 -- 172.21.15.42:0/1361228050 shutdown_connections 2024-09-16T04:13:33.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.682+0000 7f3ebbaae640 1 --2- 172.21.15.42:0/1361228050 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3eb4137540 0x7f3eb4137990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:13:33.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.682+0000 7f3ebbaae640 1 --2- 172.21.15.42:0/1361228050 >> v2:172.21.15.154:3300/0 conn(0x7f3eb40a3d90 0x7f3eb4137000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:13:33.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.682+0000 7f3ebbaae640 1 --2- 172.21.15.42:0/1361228050 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3eb40a39b0 0x7f3eb4133b80 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:13:33.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.682+0000 7f3ebbaae640 1 -- 172.21.15.42:0/1361228050 >> 172.21.15.42:0/1361228050 conn(0x7f3eb40a9fb0 msgr2=0x7f3eb40aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:13:33.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.682+0000 7f3ebbaae640 1 -- 172.21.15.42:0/1361228050 shutdown_connections 2024-09-16T04:13:33.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.682+0000 7f3ebbaae640 1 -- 172.21.15.42:0/1361228050 wait complete. 2024-09-16T04:13:33.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.682+0000 7f3ebbaae640 10 monclient: build_initial_monmap 2024-09-16T04:13:33.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.682+0000 7f3ebbaae640 10 monclient: monmap: 2024-09-16T04:13:33.699 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:13:33.699 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:13:33.699 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:13:33.699 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:13:33.699 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:13:33.699 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:13:33.700 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:13:33.700 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:13:33.700 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:13:33.700 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:13:33.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.682+0000 7f3ebbaae640 1 librados: starting msgr at 2024-09-16T04:13:33.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.682+0000 7f3ebbaae640 1 librados: starting objecter 2024-09-16T04:13:33.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.682+0000 7f3ebbaae640 1 Processor -- start 2024-09-16T04:13:33.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.682+0000 7f3ebbaae640 1 -- start start 2024-09-16T04:13:33.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.682+0000 7f3ebbaae640 1 librados: setting wanted keys 2024-09-16T04:13:33.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.682+0000 7f3ebbaae640 1 librados: calling monclient init 2024-09-16T04:13:33.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.682+0000 7f3ebbaae640 10 monclient: init 2024-09-16T04:13:33.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.682+0000 7f3ebbaae640 10 monclient: _reopen_session rank -1 2024-09-16T04:13:33.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.682+0000 7f3ebbaae640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:13:33.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.682+0000 7f3ebbaae640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3eb40a39b0 0x7f3eb413b370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:13:33.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.682+0000 7f3ebbaae640 10 monclient(hunting): picked mon.noname-a con 0x7f3eb40a39b0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:13:33.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.682+0000 7f3ebbaae640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3eb40a3d90 0x7f3eb413b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:13:33.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.682+0000 7f3ebbaae640 10 monclient(hunting): picked mon.noname-b con 0x7f3eb40a3d90 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:13:33.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.682+0000 7f3ebbaae640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3eb4137540 0x7f3eb41cf060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:13:33.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.682+0000 7f3ebbaae640 10 monclient(hunting): picked mon.noname-c con 0x7f3eb4137540 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:13:33.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.682+0000 7f3ebbaae640 10 monclient(hunting): start opening mon connection 2024-09-16T04:13:33.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.682+0000 7f3ebbaae640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f3eb4140dc0 con 0x7f3eb40a39b0 2024-09-16T04:13:33.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.682+0000 7f3ebbaae640 10 monclient(hunting): start opening mon connection 2024-09-16T04:13:33.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.682+0000 7f3ebbaae640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f3eb4140f40 con 0x7f3eb40a3d90 2024-09-16T04:13:33.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.682+0000 7f3ebbaae640 10 monclient(hunting): start opening mon connection 2024-09-16T04:13:33.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.682+0000 7f3ebbaae640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f3eb409d560 con 0x7f3eb4137540 2024-09-16T04:13:33.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.682+0000 7f3ebbaae640 10 monclient(hunting): _renew_subs 2024-09-16T04:13:33.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.682+0000 7f3ebbaae640 10 monclient(hunting): authenticate will time out at 2860.655724s 2024-09-16T04:13:33.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.682+0000 7f3eb984a640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3eb40a39b0 0x7f3eb413b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:13:33.702 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.682+0000 7f3eba04b640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3eb4137540 0x7f3eb41cf060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:13:33.702 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.682+0000 7f3eb984a640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3eb40a39b0 0x7f3eb413b370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:52142/0 (socket says 172.21.15.42:52142) 2024-09-16T04:13:33.702 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.682+0000 7f3eb984a640 1 -- 172.21.15.42:0/3699453526 learned_addr learned my addr 172.21.15.42:0/3699453526 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:13:33.702 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.682+0000 7f3eb984a640 10 monclient(hunting): get_auth_request con 0x7f3eb40a39b0 auth_method 0 2024-09-16T04:13:33.702 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.682+0000 7f3eb984a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:13:33.702 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.682+0000 7f3eb984a640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:33.702 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.682+0000 7f3eb984a640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:13:33.702 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.682+0000 7f3eba04b640 10 monclient(hunting): get_auth_request con 0x7f3eb4137540 auth_method 0 2024-09-16T04:13:33.702 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.682+0000 7f3eba04b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:13:33.702 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.682+0000 7f3eba04b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:33.702 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.682+0000 7f3eba04b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:13:33.702 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.682+0000 7f3eb984a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:13:33.702 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.682+0000 7f3eb984a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:13:33.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.682+0000 7f3eb984a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:13:33.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.682+0000 7f3eba04b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:13:33.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.682+0000 7f3eba04b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:13:33.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.686+0000 7f3eba04b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:13:33.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.686+0000 7f3eb9049640 1 --2- 172.21.15.42:0/3699453526 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3eb40a3d90 0x7f3eb413b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:13:33.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.686+0000 7f3eb984a640 10 monclient(hunting): handle_auth_done global_id 1034658 payload 824 2024-09-16T04:13:33.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.686+0000 7f3eb984a640 1 -- 172.21.15.42:0/3699453526 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3eb4137540 msgr2=0x7f3eb41cf060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:33.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.686+0000 7f3eb984a640 1 --2- 172.21.15.42:0/3699453526 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3eb4137540 0x7f3eb41cf060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:13:33.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.686+0000 7f3eb984a640 1 -- 172.21.15.42:0/3699453526 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3eb40a3d90 msgr2=0x7f3eb413b8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:33.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.686+0000 7f3eb984a640 1 --2- 172.21.15.42:0/3699453526 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3eb40a3d90 0x7f3eb413b8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:13:33.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.686+0000 7f3eb984a640 10 monclient: _finish_hunting 0 2024-09-16T04:13:33.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.686+0000 7f3eb984a640 1 monclient: found mon.noname-a 2024-09-16T04:13:33.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.686+0000 7f3eb984a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:13:33.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.686+0000 7f3eb984a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:13:33.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.686+0000 7f3eb984a640 1 -- 172.21.15.42:0/3699453526 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3eb409c920 con 0x7f3eb40a39b0 2024-09-16T04:13:33.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.686+0000 7f3eb9049640 10 monclient: get_auth_request con 0x7f3eb40a3d90 auth_method 0 2024-09-16T04:13:33.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.686+0000 7f3eba04b640 1 --2- 172.21.15.42:0/3699453526 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3eb4137540 0x7f3eb41cf060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:13:33.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.686+0000 7f3eb9049640 1 --2- 172.21.15.42:0/3699453526 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3eb40a3d90 0x7f3eb413b8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T04:13:33.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.686+0000 7f3eb984a640 1 --2- 172.21.15.42:0/3699453526 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3eb40a39b0 0x7f3eb413b370 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f3eb00ae040 tx=0x7f3eb0019180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:13:33.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.686+0000 7f3eaa7fc640 10 client.?.objecter ms_handle_connect 0x7f3eb40a39b0 2024-09-16T04:13:33.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.686+0000 7f3eaa7fc640 10 client.?.objecter resend_mon_ops 2024-09-16T04:13:33.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.686+0000 7f3eaa7fc640 1 -- 172.21.15.42:0/3699453526 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3eb00b0040 con 0x7f3eb40a39b0 2024-09-16T04:13:33.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.686+0000 7f3eaa7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:13:33.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.686+0000 7f3eaa7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:13:33.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.686+0000 7f3eaa7fc640 10 monclient: dump: 2024-09-16T04:13:33.704 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:13:33.704 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:13:33.705 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:33.705 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:33.705 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:33.705 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:13:33.705 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:13:33.705 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:13:33.705 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:13:33.705 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:13:33.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.686+0000 7f3eaa7fc640 10 monclient: _finish_auth 0 2024-09-16T04:13:33.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.686+0000 7f3eaa7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:13:33.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.686+0000 7f3eaa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:13:33.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.686+0000 7f3eaa7fc640 1 -- 172.21.15.42:0/3699453526 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3eb00b9070 con 0x7f3eb40a39b0 2024-09-16T04:13:33.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.686+0000 7f3ebbaae640 5 monclient: authenticate success, global_id 1034658 2024-09-16T04:13:33.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.686+0000 7f3ebbaae640 10 monclient: _renew_subs 2024-09-16T04:13:33.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.686+0000 7f3ebbaae640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:13:33.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.686+0000 7f3ebbaae640 1 -- 172.21.15.42:0/3699453526 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3eb4137120 con 0x7f3eb40a39b0 2024-09-16T04:13:33.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.686+0000 7f3eaa7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:13:33.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.686+0000 7f3eaa7fc640 1 -- 172.21.15.42:0/3699453526 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3eb00b4860 con 0x7f3eb40a39b0 2024-09-16T04:13:33.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.686+0000 7f3eaa7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:13:33.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.686+0000 7f3eaa7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:13:33.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.686+0000 7f3eaa7fc640 10 monclient: dump: 2024-09-16T04:13:33.706 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:13:33.706 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:13:33.706 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:33.706 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:33.706 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:33.706 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:13:33.707 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:13:33.707 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:13:33.707 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:13:33.707 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:13:33.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.686+0000 7f3eabfff640 4 set_mon_vals no callback set 2024-09-16T04:13:33.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.686+0000 7f3ebbaae640 10 client.1034658.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:13:33.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.686+0000 7f3ebbaae640 10 monclient: _renew_subs 2024-09-16T04:13:33.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.686+0000 7f3ebbaae640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:13:33.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.686+0000 7f3ebbaae640 1 -- 172.21.15.42:0/3699453526 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3eb413cce0 con 0x7f3eb40a39b0 2024-09-16T04:13:33.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.686+0000 7f3ebbaae640 1 librados: init done 2024-09-16T04:13:33.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.686+0000 7f3ebbaae640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:13:33.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.686+0000 7f3ebbaae640 1 --2- 172.21.15.42:0/3699453526 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3eb409cf00 0x7f3eb41cf5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:13:33.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.686+0000 7f3ebbaae640 10 monclient(hunting): start opening mon connection 2024-09-16T04:13:33.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.686+0000 7f3ebbaae640 1 -- 172.21.15.42:0/3699453526 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f3eb413d8d0 con 0x7f3eb409cf00 2024-09-16T04:13:33.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.686+0000 7f3eaa7fc640 1 -- 172.21.15.42:0/3699453526 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f3eb00b4a00 con 0x7f3eb40a39b0 2024-09-16T04:13:33.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.686+0000 7f3eb9049640 1 --2- 172.21.15.42:0/3699453526 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3eb409cf00 0x7f3eb41cf5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:13:33.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.686+0000 7f3eaa7fc640 1 --2- 172.21.15.42:0/3699453526 >> v2:172.21.15.154:6812/254845342 conn(0x7f3e98037810 0x7f3e98039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:13:33.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.686+0000 7f3eb9049640 10 monclient: get_auth_request con 0x7f3eb409cf00 auth_method 0 2024-09-16T04:13:33.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.686+0000 7f3eaa7fc640 1 -- 172.21.15.42:0/3699453526 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(706..706 src has 681..706) ==== 5040+0+0 (secure 0 0 0) 0x7f3eb00e2a90 con 0x7f3eb40a39b0 2024-09-16T04:13:33.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.686+0000 7f3eb9049640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:13:33.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.686+0000 7f3eb9049640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:33.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.686+0000 7f3eb9049640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:13:33.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.686+0000 7f3eaa7fc640 10 client.1034658.objecter ms_dispatch 0x7f3eb4000ba0 osd_map(706..706 src has 681..706) 2024-09-16T04:13:33.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.686+0000 7f3eaa7fc640 3 client.1034658.objecter handle_osd_map got epochs [706,706] > 0 2024-09-16T04:13:33.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.686+0000 7f3eaa7fc640 3 client.1034658.objecter handle_osd_map decoding full epoch 706 2024-09-16T04:13:33.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.686+0000 7f3eaa7fc640 20 client.1034658.objecter dump_active .. 0 homeless 2024-09-16T04:13:33.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.686+0000 7f3eb9049640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:13:33.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.686+0000 7f3eb9049640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:13:33.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.686+0000 7f3eb9049640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:13:33.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.686+0000 7f3eba04b640 1 --2- 172.21.15.42:0/3699453526 >> v2:172.21.15.154:6812/254845342 conn(0x7f3e98037810 0x7f3e98039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:13:33.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.686+0000 7f3eba04b640 10 monclient: get_auth_request con 0x7f3e98037810 auth_method 0 2024-09-16T04:13:33.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.686+0000 7f3eb9049640 10 monclient(hunting): handle_auth_done global_id 1074301 payload 824 2024-09-16T04:13:33.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.686+0000 7f3eb9049640 1 -- 172.21.15.42:0/3699453526 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3eb4133b80 con 0x7f3eb409cf00 2024-09-16T04:13:33.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.686+0000 7f3eb9049640 1 --2- 172.21.15.42:0/3699453526 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3eb409cf00 0x7f3eb41cf5a0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f3eb413c8e0 tx=0x7f3ea400eba0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:13:33.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.686+0000 7f3eaa7fc640 10 client.1034658.objecter ms_handle_connect 0x7f3eb409cf00 2024-09-16T04:13:33.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.686+0000 7f3eaa7fc640 10 client.1034658.objecter resend_mon_ops 2024-09-16T04:13:33.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.686+0000 7f3eaa7fc640 1 -- 172.21.15.42:0/3699453526 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ea4017070 con 0x7f3eb409cf00 2024-09-16T04:13:33.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.686+0000 7f3eaa7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:13:33.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.686+0000 7f3eaa7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:13:33.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.686+0000 7f3eaa7fc640 10 monclient: dump: 2024-09-16T04:13:33.710 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:13:33.710 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:13:33.710 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:33.710 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:33.710 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:33.710 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:13:33.710 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:13:33.710 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:13:33.710 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:13:33.710 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:13:33.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.686+0000 7f3eaa7fc640 10 client.1034658.objecter ms_handle_connect 0x7f3e98037810 2024-09-16T04:13:33.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.686+0000 7f3eba04b640 1 --2- 172.21.15.42:0/3699453526 >> v2:172.21.15.154:6812/254845342 conn(0x7f3e98037810 0x7f3e98039cd0 secure :-1 s=READY pgs=3993 cs=0 l=1 rev1=1 crypto rx=0x7f3eac00e740 tx=0x7f3eac00b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:13:33.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.690+0000 7f3eaa7fc640 1 -- 172.21.15.42:0/3699453526 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f3ea4004020 con 0x7f3eb409cf00 2024-09-16T04:13:33.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.690+0000 7f3eaa7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:13:33.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.690+0000 7f3eaa7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:13:33.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.690+0000 7f3eaa7fc640 1 -- 172.21.15.42:0/3699453526 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3eb409cf00 msgr2=0x7f3eb41cf5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:33.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.690+0000 7f3eaa7fc640 1 --2- 172.21.15.42:0/3699453526 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3eb409cf00 0x7f3eb41cf5a0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f3eb413c8e0 tx=0x7f3ea400eba0 comp rx=0 tx=0).stop 2024-09-16T04:13:33.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.690+0000 7f3eaa7fc640 1 -- 172.21.15.42:0/3699453526 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3eb409cf00 msgr2=0x7f3eb41cf5a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:13:33.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.690+0000 7f3eaa7fc640 1 --2- 172.21.15.42:0/3699453526 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3eb409cf00 0x7f3eb41cf5a0 secure :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f3eb413c8e0 tx=0x7f3ea400eba0 comp rx=0 tx=0).stop 2024-09-16T04:13:33.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.694+0000 7f3ebbaae640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:13:33.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.694+0000 7f3ebbaae640 1 --2- 172.21.15.42:0/3699453526 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3eb41cfae0 0x7f3eb4135bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:13:33.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.694+0000 7f3ebbaae640 10 monclient(hunting): start opening mon connection 2024-09-16T04:13:33.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.694+0000 7f3ebbaae640 1 -- 172.21.15.42:0/3699453526 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f3eb41342d0 con 0x7f3eb41cfae0 2024-09-16T04:13:33.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.694+0000 7f3eb9049640 1 --2- 172.21.15.42:0/3699453526 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3eb41cfae0 0x7f3eb4135bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:13:33.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.694+0000 7f3eb9049640 10 monclient: get_auth_request con 0x7f3eb41cfae0 auth_method 0 2024-09-16T04:13:33.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.694+0000 7f3eb9049640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:13:33.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.694+0000 7f3eb9049640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:33.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.694+0000 7f3eb9049640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:13:33.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.694+0000 7f3eb9049640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:13:33.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.694+0000 7f3eb9049640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:13:33.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.694+0000 7f3eb9049640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:13:33.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.694+0000 7f3eb9049640 10 monclient(hunting): handle_auth_done global_id 1074304 payload 824 2024-09-16T04:13:33.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.694+0000 7f3eb9049640 1 -- 172.21.15.42:0/3699453526 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f3eb40a5750 con 0x7f3eb41cfae0 2024-09-16T04:13:33.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.694+0000 7f3eb9049640 1 --2- 172.21.15.42:0/3699453526 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3eb41cfae0 0x7f3eb4135bc0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f3eb413c8e0 tx=0x7f3ea40046d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:13:33.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.694+0000 7f3eaa7fc640 10 client.1034658.objecter ms_handle_connect 0x7f3eb41cfae0 2024-09-16T04:13:33.714 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.694+0000 7f3eaa7fc640 10 client.1034658.objecter resend_mon_ops 2024-09-16T04:13:33.714 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.694+0000 7f3eaa7fc640 1 -- 172.21.15.42:0/3699453526 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ea4029020 con 0x7f3eb41cfae0 2024-09-16T04:13:33.714 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.694+0000 7f3eaa7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:13:33.714 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.694+0000 7f3eaa7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:13:33.714 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.694+0000 7f3eaa7fc640 10 monclient: dump: 2024-09-16T04:13:33.714 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:13:33.714 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:13:33.714 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:33.714 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:33.714 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:33.714 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:13:33.714 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:13:33.714 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:13:33.714 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:13:33.715 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:13:33.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.698+0000 7f3eaa7fc640 1 -- 172.21.15.42:0/3699453526 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4713 (secure 0 0 0) 0x7f3ea4003a30 con 0x7f3eb41cfae0 2024-09-16T04:13:33.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.698+0000 7f3eaa7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:13:33.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.698+0000 7f3eaa7fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:13:33.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.698+0000 7f3eaa7fc640 1 -- 172.21.15.42:0/3699453526 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3eb41cfae0 msgr2=0x7f3eb4135bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:33.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.698+0000 7f3eaa7fc640 1 --2- 172.21.15.42:0/3699453526 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3eb41cfae0 0x7f3eb4135bc0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f3eb413c8e0 tx=0x7f3ea40046d0 comp rx=0 tx=0).stop 2024-09-16T04:13:33.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.698+0000 7f3eaa7fc640 1 -- 172.21.15.42:0/3699453526 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3eb41cfae0 msgr2=0x7f3eb4135bc0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:13:33.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.698+0000 7f3eaa7fc640 1 --2- 172.21.15.42:0/3699453526 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3eb41cfae0 0x7f3eb4135bc0 secure :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f3eb413c8e0 tx=0x7f3ea40046d0 comp rx=0 tx=0).stop 2024-09-16T04:13:33.715 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T04:13:33.715 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:13:33.715 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:13:33.716 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T04:13:33.716 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 286, 2024-09-16T04:13:33.716 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 126867, 2024-09-16T04:13:33.716 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T04:13:33.716 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T04:13:33.716 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T04:13:33.716 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T04:13:33.716 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:13:33.716 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 3, 2024-09-16T04:13:33.716 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:13:33.716 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T04:13:33.716 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T04:13:33.716 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:13:33.717 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:13:33.717 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:13:33.717 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:13:33.717 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:13:33.717 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:13:33.717 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:13:33.717 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:13:33.717 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:13:33.717 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:13:33.717 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:13:33.717 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:13:33.717 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:13:33.717 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T04:13:33.717 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:13:33.718 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:13:33.718 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:13:33.718 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:13:33.718 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:13:33.718 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:13:33.718 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:13:33.718 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:13:33.718 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:13:33.718 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:13:33.718 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:13:33.718 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:13:33.718 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:13:33.719 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T04:13:33.719 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T04:13:33.719 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T04:13:33.719 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T04:13:33.719 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T04:13:33.719 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T04:13:33.719 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:13:33.719 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:13:33.719 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T04:13:33.720 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T04:13:33.720 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T04:13:33.720 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T04:13:33.720 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T04:13:33.720 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T04:13:33.720 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T04:13:33.720 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:13:33.720 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T04:13:33.720 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:13:33.720 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:13:33.720 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:13:33.720 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:13:33.720 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:13:33.721 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:13:33.721 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:13:33.721 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:13:33.721 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:13:33.721 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:13:33.721 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:13:33.721 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:13:33.721 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T04:13:33.721 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:13:33.721 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T04:13:33.721 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:13:33.721 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:13:33.721 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:13:33.721 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:13:33.722 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:13:33.722 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:13:33.722 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:13:33.722 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T04:13:33.722 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:13:33.722 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:13:33.722 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:13:33.722 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:13:33.722 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T04:13:33.722 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:13:33.722 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:13:33.722 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:13:33.722 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:13:33.722 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T04:13:33.723 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T04:13:33.723 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:13:33.723 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:13:33.723 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:13:33.723 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:13:33.723 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:13:33.723 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:13:33.723 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:13:33.723 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:13:33.723 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:13:33.723 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:13:33.723 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:13:33.723 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T04:13:33.723 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:13:33.724 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:13:33.724 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:13:33.724 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:13:33.724 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T04:13:33.724 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:13:33.724 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:13:33.724 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:13:33.724 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:13:33.724 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T04:13:33.724 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T04:13:33.724 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:13:33.724 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:13:33.724 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:13:33.724 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:13:33.725 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:13:33.725 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:13:33.725 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:13:33.725 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:13:33.725 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:13:33.725 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:13:33.725 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:13:33.725 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T04:13:33.725 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:13:33.725 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:13:33.726 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:13:33.726 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:13:33.726 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T04:13:33.726 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:13:33.726 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:13:33.726 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:13:33.726 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:13:33.726 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T04:13:33.726 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T04:13:33.726 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:13:33.726 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:13:33.726 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:13:33.726 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:13:33.726 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:13:33.727 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:13:33.727 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T04:13:33.727 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T04:13:33.727 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:13:33.727 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:13:33.727 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:13:33.727 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:13:33.727 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:13:33.727 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:13:33.727 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T04:13:33.727 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:13:33.727 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:13:33.727 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:13:33.727 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 2 2024-09-16T04:13:33.728 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:13:33.728 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:13:33.728 INFO:teuthology.orchestra.run.smithi042.stdout: "mgr": [ 2024-09-16T04:13:33.728 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:13:33.728 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:13:33.728 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:13:33.728 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:13:33.728 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:13:33.728 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:13:33.728 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:13:33.728 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T04:13:33.728 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T04:13:33.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.698+0000 7f3ebbaae640 10 librados: watch_flush enter 2024-09-16T04:13:33.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.698+0000 7f3ebbaae640 10 librados: watch_flush exit 2024-09-16T04:13:33.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.698+0000 7f3ebbaae640 20 client.1034658.objecter shutdown clearing up homeless session... 2024-09-16T04:13:33.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.698+0000 7f3ebbaae640 10 client.1034658.objecter successfully canceled tick 2024-09-16T04:13:33.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.698+0000 7f3ebbaae640 1 -- 172.21.15.42:0/3699453526 >> v2:172.21.15.154:6812/254845342 conn(0x7f3e98037810 msgr2=0x7f3e98039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:33.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.698+0000 7f3ebbaae640 1 --2- 172.21.15.42:0/3699453526 >> v2:172.21.15.154:6812/254845342 conn(0x7f3e98037810 0x7f3e98039cd0 secure :-1 s=READY pgs=3993 cs=0 l=1 rev1=1 crypto rx=0x7f3eac00e740 tx=0x7f3eac00b040 comp rx=0 tx=0).stop 2024-09-16T04:13:33.730 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.698+0000 7f3ebbaae640 10 monclient: shutdown 2024-09-16T04:13:33.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.698+0000 7f3ebbaae640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:13:33.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.698+0000 7f3ebbaae640 1 -- 172.21.15.42:0/3699453526 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3eb40a39b0 msgr2=0x7f3eb413b370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:33.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.698+0000 7f3ebbaae640 1 --2- 172.21.15.42:0/3699453526 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3eb40a39b0 0x7f3eb413b370 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f3eb00ae040 tx=0x7f3eb0019180 comp rx=0 tx=0).stop 2024-09-16T04:13:33.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.698+0000 7f3eba04b640 1 -- 172.21.15.42:0/3699453526 reap_dead start 2024-09-16T04:13:33.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.698+0000 7f3eba04b640 1 -- 172.21.15.42:0/3699453526 reap_dead start 2024-09-16T04:13:33.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.698+0000 7f3ebbaae640 1 -- 172.21.15.42:0/3699453526 shutdown_connections 2024-09-16T04:13:33.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.698+0000 7f3ebbaae640 1 -- 172.21.15.42:0/3699453526 >> 172.21.15.42:0/3699453526 conn(0x7f3eb40a9fb0 msgr2=0x7f3eb40a8450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:13:33.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.698+0000 7f3ebbaae640 1 -- 172.21.15.42:0/3699453526 shutdown_connections 2024-09-16T04:13:33.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.698+0000 7f3ebbaae640 1 -- 172.21.15.42:0/3699453526 wait complete. 2024-09-16T04:13:33.731 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.698+0000 7f3ebbaae640 1 librados: shutdown 2024-09-16T04:13:33.731 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:13:33.732 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-09-16T04:13:33.732 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-16T04:13:33.732 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-09-16T04:13:33.732 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-16T04:13:33.732 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-16T04:13:33.732 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T04:13:33.751 INFO:tasks.ceph.mon.a:Stopped 2024-09-16T04:13:33.751 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-16T04:13:33.751 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:13:33.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.902+0000 7f90b86e2640 10 monclient: get_monmap_and_config 2024-09-16T04:13:33.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.902+0000 7f90b86e2640 10 monclient: build_initial_monmap 2024-09-16T04:13:33.907 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.902+0000 7f90b86e2640 10 monclient: monmap: 2024-09-16T04:13:33.907 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:13:33.907 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:13:33.907 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:13:33.905844+0000 2024-09-16T04:13:33.907 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:13:33.905844+0000 2024-09-16T04:13:33.907 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:13:33.907 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:13:33.908 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:13:33.908 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:13:33.908 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:13:33.908 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:13:33.908 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:13:33.908 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:13:33.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.902+0000 7f90b86e2640 1 Processor -- start 2024-09-16T04:13:33.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.902+0000 7f90b86e2640 1 -- start start 2024-09-16T04:13:33.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.902+0000 7f90b86e2640 10 monclient: init 2024-09-16T04:13:33.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.902+0000 7f90b86e2640 10 monclient: _reopen_session rank -1 2024-09-16T04:13:33.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.902+0000 7f90b86e2640 10 monclient: _add_conns ranks=[0,3,4,1,2] 2024-09-16T04:13:33.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.902+0000 7f90b86e2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f90b00a1330 addr v1:172.21.15.42:6789/0 2024-09-16T04:13:33.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.902+0000 7f90b86e2640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f90b00a1710 0x7f90b00a1b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:13:33.910 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.902+0000 7f90b86e2640 10 monclient(hunting): picked mon.noname-a con 0x7f90b00a1710 addr v2:172.21.15.42:3300/0 2024-09-16T04:13:33.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.902+0000 7f90b86e2640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f90b01393a0 0x7f90b013b790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:13:33.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.902+0000 7f90b86e2640 10 monclient(hunting): picked mon.noname-c con 0x7f90b01393a0 addr v2:172.21.15.154:3300/0 2024-09-16T04:13:33.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.902+0000 7f90b86e2640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f90b009e4c0 con 0x7f90b00a1330 2024-09-16T04:13:33.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.902+0000 7f90b647e640 1 -- >> v1:172.21.15.42:6789/0 conn(0x7f90b00a1330 legacy=0x7f90b009f180 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.42:6789/0 2024-09-16T04:13:33.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.902+0000 7f90b86e2640 10 monclient(hunting): start opening mon connection 2024-09-16T04:13:33.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.902+0000 7f90b86e2640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f90b009edf0 con 0x7f90b00a1710 2024-09-16T04:13:33.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.902+0000 7f90b86e2640 10 monclient(hunting): start opening mon connection 2024-09-16T04:13:33.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.902+0000 7f90b86e2640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f90b009cd60 con 0x7f90b01393a0 2024-09-16T04:13:33.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.902+0000 7f90b86e2640 10 monclient(hunting): _renew_subs 2024-09-16T04:13:33.911 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.902+0000 7f90b86e2640 10 monclient(hunting): authenticate will time out at 2860.875400s 2024-09-16T04:13:33.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.902+0000 7f90b5c7d640 1 -- >> v2:172.21.15.42:3300/0 conn(0x7f90b00a1710 msgr2=0x7f90b00a1b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3300/0 2024-09-16T04:13:33.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.902+0000 7f90b5c7d640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f90b00a1710 0x7f90b00a1b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:13:33.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.902+0000 7f90b6c7f640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f90b01393a0 0x7f90b013b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:13:33.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.902+0000 7f90b6c7f640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f90b01393a0 0x7f90b013b790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.154:3300/0 says I am v2:172.21.15.42:40530/0 (socket says 172.21.15.42:40530) 2024-09-16T04:13:33.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.902+0000 7f90b6c7f640 1 -- 172.21.15.42:0/859792278 learned_addr learned my addr 172.21.15.42:0/859792278 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:13:33.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.902+0000 7f90b6c7f640 10 monclient(hunting): get_auth_request con 0x7f90b01393a0 auth_method 0 2024-09-16T04:13:33.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.902+0000 7f90b6c7f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:13:33.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.902+0000 7f90b6c7f640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:33.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.902+0000 7f90b6c7f640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:13:33.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.906+0000 7f90b6c7f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:13:33.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.906+0000 7f90b6c7f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:13:33.912 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.906+0000 7f90b6c7f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:13:33.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.906+0000 7f90b6c7f640 10 monclient(hunting): handle_auth_done global_id 1074307 payload 274 2024-09-16T04:13:33.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.906+0000 7f90b6c7f640 1 -- 172.21.15.42:0/859792278 >> v2:172.21.15.42:3300/0 conn(0x7f90b00a1710 msgr2=0x7f90b00a1b40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:13:33.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.906+0000 7f90b6c7f640 1 --2- 172.21.15.42:0/859792278 >> v2:172.21.15.42:3300/0 conn(0x7f90b00a1710 0x7f90b00a1b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:13:33.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.906+0000 7f90b6c7f640 1 -- 172.21.15.42:0/859792278 >> v1:172.21.15.42:6789/0 conn(0x7f90b00a1330 legacy=0x7f90b009f180 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:13:33.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.906+0000 7f90b6c7f640 10 monclient: _finish_hunting 0 2024-09-16T04:13:33.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.906+0000 7f90b6c7f640 1 monclient: found mon.noname-c 2024-09-16T04:13:33.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.906+0000 7f90b6c7f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:13:33.913 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.906+0000 7f90b6c7f640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.154:3300/0 2024-09-16T04:13:33.914 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.906+0000 7f90b6c7f640 1 -- 172.21.15.42:0/859792278 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90b00a2080 con 0x7f90b01393a0 2024-09-16T04:13:33.918 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.906+0000 7f90b6c7f640 1 --2- 172.21.15.42:0/859792278 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f90b01393a0 0x7f90b013b790 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f90b009df50 tx=0x7f90a801c890 comp rx=0 tx=0).ready entity=mon.1 client_cookie=83add2918301a52c server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:13:33.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.906+0000 7f90b547c640 1 -- 172.21.15.42:0/859792278 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90a80a5550 con 0x7f90b01393a0 2024-09-16T04:13:33.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.906+0000 7f90b547c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:13:33.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.906+0000 7f90b547c640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:13:33.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.906+0000 7f90b547c640 10 monclient: dump: 2024-09-16T04:13:33.919 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:13:33.920 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:13:33.920 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:33.920 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:33.920 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:33.920 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:13:33.920 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:13:33.920 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:13:33.920 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:13:33.920 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:13:33.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.906+0000 7f90b547c640 10 monclient: _finish_auth 0 2024-09-16T04:13:33.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.906+0000 7f90b547c640 10 monclient: _check_auth_tickets 2024-09-16T04:13:33.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.906+0000 7f90b547c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:13:33.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.906+0000 7f90b547c640 1 -- 172.21.15.42:0/859792278 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f90a80a56f0 con 0x7f90b01393a0 2024-09-16T04:13:33.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.906+0000 7f90b547c640 10 monclient: handle_config config(0 keys) 2024-09-16T04:13:33.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.906+0000 7f90b547c640 1 -- 172.21.15.42:0/859792278 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90a80a5ac0 con 0x7f90b01393a0 2024-09-16T04:13:33.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.906+0000 7f90b547c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:13:33.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.906+0000 7f90b547c640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:13:33.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.906+0000 7f90b547c640 10 monclient: dump: 2024-09-16T04:13:33.921 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:13:33.921 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:13:33.921 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:33.921 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:33.921 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:33.922 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:13:33.922 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:13:33.922 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:13:33.922 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:13:33.922 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:13:33.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.906+0000 7f90b86e2640 5 monclient: authenticate success, global_id 1074307 2024-09-16T04:13:33.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.906+0000 7f90b86e2640 10 monclient: get_monmap_and_config success 2024-09-16T04:13:33.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.906+0000 7f90b86e2640 4 set_mon_vals no callback set 2024-09-16T04:13:33.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.906+0000 7f90b86e2640 10 monclient: shutdown 2024-09-16T04:13:33.922 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.906+0000 7f90b86e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:13:33.923 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.906+0000 7f90b86e2640 1 -- 172.21.15.42:0/859792278 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f90b01393a0 msgr2=0x7f90b013b790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:33.923 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.906+0000 7f90b86e2640 1 --2- 172.21.15.42:0/859792278 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f90b01393a0 0x7f90b013b790 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f90b009df50 tx=0x7f90a801c890 comp rx=0 tx=0).stop 2024-09-16T04:13:33.923 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.906+0000 7f90b86e2640 1 -- 172.21.15.42:0/859792278 shutdown_connections 2024-09-16T04:13:33.923 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.906+0000 7f90b86e2640 1 --2- 172.21.15.42:0/859792278 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f90b01393a0 0x7f90b013b790 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:13:33.923 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.906+0000 7f90b86e2640 1 --2- 172.21.15.42:0/859792278 >> v2:172.21.15.42:3300/0 conn(0x7f90b00a1710 0x7f90b00a1b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:13:33.923 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.906+0000 7f90b86e2640 1 -- 172.21.15.42:0/859792278 >> 172.21.15.42:0/859792278 conn(0x7f90b00a2420 msgr2=0x7f90b00a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:13:33.923 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.906+0000 7f90b86e2640 1 -- 172.21.15.42:0/859792278 shutdown_connections 2024-09-16T04:13:33.923 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.906+0000 7f90b86e2640 1 -- 172.21.15.42:0/859792278 wait complete. 2024-09-16T04:13:33.924 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.910+0000 7f90b86e2640 10 monclient: build_initial_monmap 2024-09-16T04:13:33.924 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.910+0000 7f90b86e2640 10 monclient: monmap: 2024-09-16T04:13:33.924 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:13:33.924 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:13:33.924 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:13:33.924 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:13:33.924 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:13:33.924 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:13:33.924 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:13:33.924 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:13:33.924 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:13:33.924 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:13:33.924 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.910+0000 7f90b86e2640 1 librados: starting msgr at 2024-09-16T04:13:33.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.910+0000 7f90b86e2640 1 librados: starting objecter 2024-09-16T04:13:33.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.910+0000 7f90b86e2640 1 Processor -- start 2024-09-16T04:13:33.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.910+0000 7f90b86e2640 1 -- start start 2024-09-16T04:13:33.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.910+0000 7f90b86e2640 1 librados: setting wanted keys 2024-09-16T04:13:33.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.910+0000 7f90b86e2640 1 librados: calling monclient init 2024-09-16T04:13:33.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.910+0000 7f90b86e2640 10 monclient: init 2024-09-16T04:13:33.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.910+0000 7f90b86e2640 10 monclient: _reopen_session rank -1 2024-09-16T04:13:33.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.910+0000 7f90b86e2640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:13:33.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.910+0000 7f90b86e2640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f90b00a1330 0x7f90b01ccf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:13:33.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.910+0000 7f90b86e2640 10 monclient(hunting): picked mon.noname-c con 0x7f90b00a1330 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:13:33.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.910+0000 7f90b86e2640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f90b00a1710 0x7f90b01344f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:13:33.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.910+0000 7f90b86e2640 10 monclient(hunting): picked mon.noname-b con 0x7f90b00a1710 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:13:33.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.910+0000 7f90b86e2640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f90b01393a0 0x7f90b0134a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:13:33.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.910+0000 7f90b86e2640 10 monclient(hunting): picked mon.noname-a con 0x7f90b01393a0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:13:33.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.910+0000 7f90b86e2640 10 monclient(hunting): start opening mon connection 2024-09-16T04:13:33.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.910+0000 7f90b86e2640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f90b013c8f0 con 0x7f90b01393a0 2024-09-16T04:13:33.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.910+0000 7f90b86e2640 10 monclient(hunting): start opening mon connection 2024-09-16T04:13:33.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.910+0000 7f90b86e2640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f90b013c770 con 0x7f90b00a1710 2024-09-16T04:13:33.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.910+0000 7f90b86e2640 10 monclient(hunting): start opening mon connection 2024-09-16T04:13:33.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.910+0000 7f90b86e2640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f90b013ca70 con 0x7f90b00a1330 2024-09-16T04:13:33.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.910+0000 7f90b86e2640 10 monclient(hunting): _renew_subs 2024-09-16T04:13:33.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.910+0000 7f90b86e2640 10 monclient(hunting): authenticate will time out at 2860.882035s 2024-09-16T04:13:33.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.910+0000 7f90b647e640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f90b00a1330 0x7f90b01ccf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:13:33.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.910+0000 7f90b6c7f640 1 -- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f90b01393a0 msgr2=0x7f90b0134a30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.42:3300/0 2024-09-16T04:13:33.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.910+0000 7f90b647e640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f90b00a1330 0x7f90b01ccf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:49112/0 (socket says 172.21.15.42:49112) 2024-09-16T04:13:33.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.910+0000 7f90b647e640 1 -- 172.21.15.42:0/3665469488 learned_addr learned my addr 172.21.15.42:0/3665469488 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:13:33.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.910+0000 7f90b647e640 10 monclient(hunting): get_auth_request con 0x7f90b00a1330 auth_method 0 2024-09-16T04:13:33.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.910+0000 7f90b647e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:13:33.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.910+0000 7f90b647e640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:33.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.910+0000 7f90b647e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:13:33.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.910+0000 7f90b6c7f640 1 --2- 172.21.15.42:0/3665469488 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f90b01393a0 0x7f90b0134a30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:13:33.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.910+0000 7f90b647e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:13:33.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.910+0000 7f90b647e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:13:33.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.910+0000 7f90b647e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:13:33.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.910+0000 7f90b5c7d640 1 --2- 172.21.15.42:0/3665469488 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f90b00a1710 0x7f90b01344f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:13:33.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.910+0000 7f90b5c7d640 10 monclient(hunting): get_auth_request con 0x7f90b00a1710 auth_method 0 2024-09-16T04:13:33.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.910+0000 7f90b5c7d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:13:33.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.910+0000 7f90b5c7d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:33.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.910+0000 7f90b5c7d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:13:33.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.910+0000 7f90b647e640 10 monclient(hunting): handle_auth_done global_id 1094147 payload 824 2024-09-16T04:13:33.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.910+0000 7f90b647e640 1 -- 172.21.15.42:0/3665469488 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f90b00a1710 msgr2=0x7f90b01344f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:33.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.910+0000 7f90b647e640 1 --2- 172.21.15.42:0/3665469488 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f90b00a1710 0x7f90b01344f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:13:33.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.910+0000 7f90b647e640 1 -- 172.21.15.42:0/3665469488 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f90b01393a0 msgr2=0x7f90b0134a30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:13:33.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.910+0000 7f90b647e640 1 --2- 172.21.15.42:0/3665469488 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f90b01393a0 0x7f90b0134a30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:13:33.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.910+0000 7f90b647e640 10 monclient: _finish_hunting 0 2024-09-16T04:13:33.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.910+0000 7f90b647e640 1 monclient: found mon.noname-c 2024-09-16T04:13:33.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.910+0000 7f90b647e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:13:33.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.910+0000 7f90b647e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T04:13:33.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.910+0000 7f90b647e640 1 -- 172.21.15.42:0/3665469488 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90b011de70 con 0x7f90b00a1330 2024-09-16T04:13:33.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.910+0000 7f90b5c7d640 1 --2- 172.21.15.42:0/3665469488 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f90b00a1710 0x7f90b01344f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T04:13:33.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.910+0000 7f90b647e640 1 --2- 172.21.15.42:0/3665469488 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f90b00a1330 0x7f90b01ccf00 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f90ac00b560 tx=0x7f90ac00ba20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:13:33.929 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.910+0000 7f90a6ffd640 10 client.?.objecter ms_handle_connect 0x7f90b00a1330 2024-09-16T04:13:33.929 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.910+0000 7f90a6ffd640 10 client.?.objecter resend_mon_ops 2024-09-16T04:13:33.929 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.910+0000 7f90a6ffd640 1 -- 172.21.15.42:0/3665469488 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90ac013020 con 0x7f90b00a1330 2024-09-16T04:13:33.929 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.910+0000 7f90a6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:13:33.929 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.910+0000 7f90a6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:13:33.929 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.910+0000 7f90a6ffd640 10 monclient: dump: 2024-09-16T04:13:33.929 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:13:33.929 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:13:33.929 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:33.929 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:33.929 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:33.929 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:13:33.929 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:13:33.930 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:13:33.930 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:13:33.930 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:13:33.930 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.910+0000 7f90a6ffd640 10 monclient: _finish_auth 0 2024-09-16T04:13:33.930 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.910+0000 7f90a6ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:13:33.930 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.910+0000 7f90a6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:13:33.930 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.910+0000 7f90a6ffd640 1 -- 172.21.15.42:0/3665469488 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f90ac003df0 con 0x7f90b00a1330 2024-09-16T04:13:33.930 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.910+0000 7f90a6ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:13:33.930 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.910+0000 7f90a6ffd640 1 -- 172.21.15.42:0/3665469488 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90ac0052b0 con 0x7f90b00a1330 2024-09-16T04:13:33.930 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.910+0000 7f90b86e2640 5 monclient: authenticate success, global_id 1094147 2024-09-16T04:13:33.930 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.910+0000 7f90b86e2640 10 monclient: _renew_subs 2024-09-16T04:13:33.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.910+0000 7f90b86e2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:13:33.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.910+0000 7f90b86e2640 1 -- 172.21.15.42:0/3665469488 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90b013b840 con 0x7f90b00a1330 2024-09-16T04:13:33.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.910+0000 7f90a7fff640 4 set_mon_vals no callback set 2024-09-16T04:13:33.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.910+0000 7f90a6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:13:33.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.910+0000 7f90a6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:13:33.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.910+0000 7f90a6ffd640 10 monclient: dump: 2024-09-16T04:13:33.931 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:13:33.931 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:13:33.931 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:33.931 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:33.931 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:33.931 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:13:33.932 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:13:33.932 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:13:33.932 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:13:33.932 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:13:33.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.910+0000 7f90b86e2640 10 client.1094147.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:13:33.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.910+0000 7f90b86e2640 10 monclient: _renew_subs 2024-09-16T04:13:33.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.910+0000 7f90b86e2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:13:33.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.910+0000 7f90b86e2640 1 -- 172.21.15.42:0/3665469488 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f90b00a43c0 con 0x7f90b00a1330 2024-09-16T04:13:33.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.910+0000 7f90b86e2640 1 librados: init done 2024-09-16T04:13:33.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.910+0000 7f90a6ffd640 1 -- 172.21.15.42:0/3665469488 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f90ac016020 con 0x7f90b00a1330 2024-09-16T04:13:33.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.910+0000 7f90b86e2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:13:33.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.910+0000 7f90b86e2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:13:33.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.910+0000 7f90b86e2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:13:33.932 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.910+0000 7f90a6ffd640 1 --2- 172.21.15.42:0/3665469488 >> v2:172.21.15.154:6812/254845342 conn(0x7f90940377c0 0x7f9094039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:13:33.933 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.910+0000 7f90b86e2640 1 -- 172.21.15.42:0/3665469488 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f90b00724f0 con 0x7f90b00a1330 2024-09-16T04:13:33.933 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.914+0000 7f90a6ffd640 1 -- 172.21.15.42:0/3665469488 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(706..706 src has 681..706) ==== 5040+0+0 (secure 0 0 0) 0x7f90ac048ad0 con 0x7f90b00a1330 2024-09-16T04:13:33.933 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.914+0000 7f90a6ffd640 10 client.1094147.objecter ms_dispatch 0x7f90b0000ba0 osd_map(706..706 src has 681..706) 2024-09-16T04:13:33.933 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.914+0000 7f90a6ffd640 3 client.1094147.objecter handle_osd_map got epochs [706,706] > 0 2024-09-16T04:13:33.933 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.914+0000 7f90a6ffd640 3 client.1094147.objecter handle_osd_map decoding full epoch 706 2024-09-16T04:13:33.933 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.914+0000 7f90a6ffd640 20 client.1094147.objecter dump_active .. 0 homeless 2024-09-16T04:13:33.933 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.914+0000 7f90b5c7d640 1 --2- 172.21.15.42:0/3665469488 >> v2:172.21.15.154:6812/254845342 conn(0x7f90940377c0 0x7f9094039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:13:33.933 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.914+0000 7f90b5c7d640 10 monclient: get_auth_request con 0x7f90940377c0 auth_method 0 2024-09-16T04:13:33.933 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.914+0000 7f90a6ffd640 10 client.1094147.objecter ms_handle_connect 0x7f90940377c0 2024-09-16T04:13:33.933 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.914+0000 7f90b5c7d640 1 --2- 172.21.15.42:0/3665469488 >> v2:172.21.15.154:6812/254845342 conn(0x7f90940377c0 0x7f9094039c80 secure :-1 s=READY pgs=3994 cs=0 l=1 rev1=1 crypto rx=0x7f90a0006fd0 tx=0x7f90a0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:13:33.934 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.918+0000 7f90a6ffd640 1 -- 172.21.15.42:0/3665469488 <== mon.2 v2:172.21.15.42:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f90ac01d020 con 0x7f90b00a1330 2024-09-16T04:13:33.934 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.918+0000 7f90a6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:13:33.934 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:33.918+0000 7f90a6ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:13:34.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:34.142+0000 7f90b86e2640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:13:34.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:34.142+0000 7f90b86e2640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:13:34.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:34.142+0000 7f90b86e2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:13:34.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:34.142+0000 7f90b86e2640 1 -- 172.21.15.42:0/3665469488 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f90b00a1b20 con 0x7f90b00a1330 2024-09-16T04:13:34.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:34.142+0000 7f90a6ffd640 1 -- 172.21.15.42:0/3665469488 <== mon.2 v2:172.21.15.42:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7f90ac020df0 con 0x7f90b00a1330 2024-09-16T04:13:34.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:34.142+0000 7f90a6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:13:34.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:34.142+0000 7f90a6ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:13:34.147 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":286,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":4,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:13:34.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:34.142+0000 7f90b86e2640 10 librados: watch_flush enter 2024-09-16T04:13:34.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:34.142+0000 7f90b86e2640 10 librados: watch_flush exit 2024-09-16T04:13:34.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:34.142+0000 7f90b86e2640 20 client.1094147.objecter shutdown clearing up homeless session... 2024-09-16T04:13:34.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:34.142+0000 7f90b86e2640 10 client.1094147.objecter successfully canceled tick 2024-09-16T04:13:34.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:34.146+0000 7f90b86e2640 1 -- 172.21.15.42:0/3665469488 >> v2:172.21.15.154:6812/254845342 conn(0x7f90940377c0 msgr2=0x7f9094039c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:34.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:34.146+0000 7f90b86e2640 1 --2- 172.21.15.42:0/3665469488 >> v2:172.21.15.154:6812/254845342 conn(0x7f90940377c0 0x7f9094039c80 secure :-1 s=READY pgs=3994 cs=0 l=1 rev1=1 crypto rx=0x7f90a0006fd0 tx=0x7f90a0008040 comp rx=0 tx=0).stop 2024-09-16T04:13:34.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:34.146+0000 7f90b86e2640 10 monclient: shutdown 2024-09-16T04:13:34.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:34.146+0000 7f90b86e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:13:34.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:34.146+0000 7f90b86e2640 1 -- 172.21.15.42:0/3665469488 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f90b00a1330 msgr2=0x7f90b01ccf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:34.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:34.146+0000 7f90b86e2640 1 --2- 172.21.15.42:0/3665469488 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f90b00a1330 0x7f90b01ccf00 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f90ac00b560 tx=0x7f90ac00ba20 comp rx=0 tx=0).stop 2024-09-16T04:13:34.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:34.146+0000 7f90b86e2640 1 -- 172.21.15.42:0/3665469488 shutdown_connections 2024-09-16T04:13:34.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:34.146+0000 7f90b86e2640 1 --2- 172.21.15.42:0/3665469488 >> v2:172.21.15.154:6812/254845342 conn(0x7f90940377c0 0x7f9094039c80 unknown :-1 s=CLOSED pgs=3994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:13:34.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:34.146+0000 7f90b86e2640 1 --2- 172.21.15.42:0/3665469488 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f90b01393a0 0x7f90b0134a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:13:34.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:34.146+0000 7f90b86e2640 1 --2- 172.21.15.42:0/3665469488 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f90b00a1710 0x7f90b01344f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:13:34.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:34.146+0000 7f90b86e2640 1 --2- 172.21.15.42:0/3665469488 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f90b00a1330 0x7f90b01ccf00 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:13:34.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:34.146+0000 7f90b86e2640 1 -- 172.21.15.42:0/3665469488 >> 172.21.15.42:0/3665469488 conn(0x7f90b00a2420 msgr2=0x7f90b013a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:13:34.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:34.146+0000 7f90b86e2640 1 -- 172.21.15.42:0/3665469488 shutdown_connections 2024-09-16T04:13:34.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:34.146+0000 7f90b86e2640 1 -- 172.21.15.42:0/3665469488 wait complete. 2024-09-16T04:13:34.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:34.146+0000 7f90b86e2640 1 librados: shutdown 2024-09-16T04:13:37.168 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:13:37.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.298+0000 7f876fd6c640 10 monclient: get_monmap_and_config 2024-09-16T04:13:37.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.298+0000 7f876fd6c640 10 monclient: build_initial_monmap 2024-09-16T04:13:37.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.298+0000 7f876fd6c640 10 monclient: monmap: 2024-09-16T04:13:37.303 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:13:37.303 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:13:37.303 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:13:37.301318+0000 2024-09-16T04:13:37.303 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:13:37.301318+0000 2024-09-16T04:13:37.303 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:13:37.303 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:13:37.303 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:13:37.303 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:13:37.303 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:13:37.303 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:13:37.303 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:13:37.303 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:13:37.306 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.298+0000 7f876fd6c640 1 Processor -- start 2024-09-16T04:13:37.306 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.298+0000 7f876fd6c640 1 -- start start 2024-09-16T04:13:37.306 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.298+0000 7f876fd6c640 10 monclient: init 2024-09-16T04:13:37.306 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.298+0000 7f876fd6c640 10 monclient: _reopen_session rank -1 2024-09-16T04:13:37.306 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.298+0000 7f876fd6c640 10 monclient: _add_conns ranks=[4,2,3,0,1] 2024-09-16T04:13:37.306 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.298+0000 7f876fd6c640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f876806db90 0x7f87681340c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:13:37.307 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.298+0000 7f876fd6c640 10 monclient(hunting): picked mon.noname-c con 0x7f876806db90 addr v2:172.21.15.154:3300/0 2024-09-16T04:13:37.307 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.298+0000 7f876fd6c640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f87680a37f0 0x7f87680a3c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:13:37.307 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.298+0000 7f876fd6c640 10 monclient(hunting): picked mon.noname-b con 0x7f87680a37f0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:13:37.307 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.298+0000 7f876fd6c640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f8768136e10 0x7f8768137240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:13:37.307 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.298+0000 7f876fd6c640 10 monclient(hunting): picked mon.noname-a con 0x7f8768136e10 addr v2:172.21.15.42:3300/0 2024-09-16T04:13:37.307 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.298+0000 7f876fd6c640 10 monclient(hunting): start opening mon connection 2024-09-16T04:13:37.307 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.298+0000 7f876fd6c640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f876806df70 con 0x7f8768136e10 2024-09-16T04:13:37.307 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.298+0000 7f876fd6c640 10 monclient(hunting): start opening mon connection 2024-09-16T04:13:37.307 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.298+0000 7f876fd6c640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f876806ece0 con 0x7f876806db90 2024-09-16T04:13:37.307 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.298+0000 7f876fd6c640 10 monclient(hunting): start opening mon connection 2024-09-16T04:13:37.307 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.298+0000 7f876fd6c640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f876809bf80 con 0x7f87680a37f0 2024-09-16T04:13:37.307 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.298+0000 7f876fd6c640 10 monclient(hunting): _renew_subs 2024-09-16T04:13:37.307 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.298+0000 7f876fd6c640 10 monclient(hunting): authenticate will time out at 2864.271041s 2024-09-16T04:13:37.307 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.298+0000 7f876e309640 1 -- >> v2:172.21.15.42:3300/0 conn(0x7f8768136e10 msgr2=0x7f8768137240 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3300/0 2024-09-16T04:13:37.308 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.298+0000 7f876e309640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f8768136e10 0x7f8768137240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:13:37.308 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.298+0000 7f876d307640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f87680a37f0 0x7f87680a3c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:13:37.308 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.298+0000 7f876d307640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f87680a37f0 0x7f87680a3c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:46306/0 (socket says 172.21.15.42:46306) 2024-09-16T04:13:37.308 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.298+0000 7f876d307640 1 -- 172.21.15.42:0/1172247994 learned_addr learned my addr 172.21.15.42:0/1172247994 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:13:37.308 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.298+0000 7f876d307640 10 monclient(hunting): get_auth_request con 0x7f87680a37f0 auth_method 0 2024-09-16T04:13:37.308 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.298+0000 7f876d307640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:13:37.308 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.298+0000 7f876d307640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:37.308 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.298+0000 7f876d307640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:13:37.308 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.298+0000 7f876db08640 1 --2- 172.21.15.42:0/1172247994 >> v2:172.21.15.154:3300/0 conn(0x7f876806db90 0x7f87681340c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:13:37.308 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.298+0000 7f876db08640 10 monclient(hunting): get_auth_request con 0x7f876806db90 auth_method 0 2024-09-16T04:13:37.309 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.298+0000 7f876db08640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:13:37.309 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.298+0000 7f876db08640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:37.309 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.298+0000 7f876db08640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:13:37.309 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.298+0000 7f876d307640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:13:37.309 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.298+0000 7f876d307640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:13:37.309 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.298+0000 7f876d307640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:13:37.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.298+0000 7f876db08640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:13:37.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.298+0000 7f876db08640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:13:37.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.298+0000 7f876db08640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:13:37.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.298+0000 7f876d307640 10 monclient(hunting): handle_auth_done global_id 1094150 payload 274 2024-09-16T04:13:37.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.298+0000 7f876d307640 1 -- 172.21.15.42:0/1172247994 >> v2:172.21.15.154:3300/0 conn(0x7f876806db90 msgr2=0x7f87681340c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:13:37.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.298+0000 7f876d307640 1 --2- 172.21.15.42:0/1172247994 >> v2:172.21.15.154:3300/0 conn(0x7f876806db90 0x7f87681340c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:13:37.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.298+0000 7f876d307640 1 -- 172.21.15.42:0/1172247994 >> v2:172.21.15.42:3300/0 conn(0x7f8768136e10 msgr2=0x7f8768137240 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:13:37.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.298+0000 7f876d307640 1 --2- 172.21.15.42:0/1172247994 >> v2:172.21.15.42:3300/0 conn(0x7f8768136e10 0x7f8768137240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:13:37.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.298+0000 7f876d307640 10 monclient: _finish_hunting 0 2024-09-16T04:13:37.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.298+0000 7f876d307640 1 monclient: found mon.noname-b 2024-09-16T04:13:37.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.298+0000 7f876d307640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:13:37.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.298+0000 7f876d307640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T04:13:37.310 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.298+0000 7f876d307640 1 -- 172.21.15.42:0/1172247994 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8768134780 con 0x7f87680a37f0 2024-09-16T04:13:37.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.302+0000 7f876d307640 1 --2- 172.21.15.42:0/1172247994 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f87680a37f0 0x7f87680a3c00 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f87680a76f0 tx=0x7f875801c400 comp rx=0 tx=0).ready entity=mon.2 client_cookie=71964ff1c05daeb6 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:13:37.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.302+0000 7f876cb06640 1 -- 172.21.15.42:0/1172247994 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87580b0070 con 0x7f87680a37f0 2024-09-16T04:13:37.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.302+0000 7f876cb06640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:13:37.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.302+0000 7f876cb06640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:13:37.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.302+0000 7f876cb06640 10 monclient: dump: 2024-09-16T04:13:37.316 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:13:37.316 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:13:37.316 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:37.316 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:37.317 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:37.317 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:13:37.317 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:13:37.317 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:13:37.317 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:13:37.317 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:13:37.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.302+0000 7f876cb06640 10 monclient: _finish_auth 0 2024-09-16T04:13:37.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.302+0000 7f876cb06640 10 monclient: _check_auth_tickets 2024-09-16T04:13:37.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.302+0000 7f876cb06640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:13:37.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.302+0000 7f876cb06640 1 -- 172.21.15.42:0/1172247994 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f87580a4e50 con 0x7f87680a37f0 2024-09-16T04:13:37.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.302+0000 7f876fd6c640 5 monclient: authenticate success, global_id 1094150 2024-09-16T04:13:37.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.302+0000 7f876fd6c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:13:37.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.302+0000 7f876cb06640 10 monclient: handle_config config(0 keys) 2024-09-16T04:13:37.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.302+0000 7f876fd6c640 10 monclient: get_monmap_and_config success 2024-09-16T04:13:37.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.302+0000 7f876fd6c640 4 set_mon_vals no callback set 2024-09-16T04:13:37.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.302+0000 7f876cb06640 1 -- 172.21.15.42:0/1172247994 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87580ab820 con 0x7f87680a37f0 2024-09-16T04:13:37.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.302+0000 7f876cb06640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:13:37.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.302+0000 7f876cb06640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:13:37.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.302+0000 7f876cb06640 10 monclient: dump: 2024-09-16T04:13:37.318 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:13:37.318 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:13:37.318 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:37.318 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:37.318 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:37.318 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:13:37.318 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:13:37.318 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:13:37.319 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:13:37.319 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:13:37.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.302+0000 7f876fd6c640 10 monclient: shutdown 2024-09-16T04:13:37.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.302+0000 7f876fd6c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:13:37.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.302+0000 7f876fd6c640 1 -- 172.21.15.42:0/1172247994 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f87680a37f0 msgr2=0x7f87680a3c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:37.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.302+0000 7f876fd6c640 1 --2- 172.21.15.42:0/1172247994 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f87680a37f0 0x7f87680a3c00 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f87680a76f0 tx=0x7f875801c400 comp rx=0 tx=0).stop 2024-09-16T04:13:37.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.302+0000 7f876fd6c640 1 -- 172.21.15.42:0/1172247994 shutdown_connections 2024-09-16T04:13:37.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.302+0000 7f876fd6c640 1 --2- 172.21.15.42:0/1172247994 >> v2:172.21.15.42:3300/0 conn(0x7f8768136e10 0x7f8768137240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:13:37.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.302+0000 7f876fd6c640 1 --2- 172.21.15.42:0/1172247994 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f87680a37f0 0x7f87680a3c00 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:13:37.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.302+0000 7f876fd6c640 1 --2- 172.21.15.42:0/1172247994 >> v2:172.21.15.154:3300/0 conn(0x7f876806db90 0x7f87681340c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:13:37.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.302+0000 7f876fd6c640 1 -- 172.21.15.42:0/1172247994 >> 172.21.15.42:0/1172247994 conn(0x7f87680a9fb0 msgr2=0x7f87680aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:13:37.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.302+0000 7f876fd6c640 1 -- 172.21.15.42:0/1172247994 shutdown_connections 2024-09-16T04:13:37.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.302+0000 7f876fd6c640 1 -- 172.21.15.42:0/1172247994 wait complete. 2024-09-16T04:13:37.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.302+0000 7f876fd6c640 10 monclient: build_initial_monmap 2024-09-16T04:13:37.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.302+0000 7f876fd6c640 10 monclient: monmap: 2024-09-16T04:13:37.320 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:13:37.320 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:13:37.320 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:13:37.320 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:13:37.320 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:13:37.320 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:13:37.320 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:13:37.320 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:13:37.320 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:13:37.320 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:13:37.320 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.302+0000 7f876fd6c640 1 librados: starting msgr at 2024-09-16T04:13:37.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.302+0000 7f876fd6c640 1 librados: starting objecter 2024-09-16T04:13:37.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.302+0000 7f876fd6c640 1 Processor -- start 2024-09-16T04:13:37.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.302+0000 7f876fd6c640 1 -- start start 2024-09-16T04:13:37.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.302+0000 7f876fd6c640 1 librados: setting wanted keys 2024-09-16T04:13:37.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.302+0000 7f876fd6c640 1 librados: calling monclient init 2024-09-16T04:13:37.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.302+0000 7f876fd6c640 10 monclient: init 2024-09-16T04:13:37.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.306+0000 7f876fd6c640 10 monclient: _reopen_session rank -1 2024-09-16T04:13:37.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.306+0000 7f876fd6c640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:13:37.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.306+0000 7f876fd6c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f876806db90 0x7f876813d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:13:37.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.306+0000 7f876fd6c640 10 monclient(hunting): picked mon.noname-a con 0x7f876806db90 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:13:37.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.306+0000 7f876fd6c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f87680a37f0 0x7f87681cc3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:13:37.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.306+0000 7f876fd6c640 10 monclient(hunting): picked mon.noname-b con 0x7f87680a37f0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:13:37.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.306+0000 7f876fd6c640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f8768136e10 0x7f87681cc8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:13:37.321 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.306+0000 7f876fd6c640 10 monclient(hunting): picked mon.noname-c con 0x7f8768136e10 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:13:37.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.306+0000 7f876fd6c640 10 monclient(hunting): start opening mon connection 2024-09-16T04:13:37.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.306+0000 7f876fd6c640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f8768140bd0 con 0x7f876806db90 2024-09-16T04:13:37.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.306+0000 7f876fd6c640 10 monclient(hunting): start opening mon connection 2024-09-16T04:13:37.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.306+0000 7f876fd6c640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f8768137c80 con 0x7f87680a37f0 2024-09-16T04:13:37.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.306+0000 7f876fd6c640 10 monclient(hunting): start opening mon connection 2024-09-16T04:13:37.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.306+0000 7f876fd6c640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f8768140d50 con 0x7f8768136e10 2024-09-16T04:13:37.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.306+0000 7f876fd6c640 10 monclient(hunting): _renew_subs 2024-09-16T04:13:37.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.306+0000 7f876fd6c640 10 monclient(hunting): authenticate will time out at 2864.276155s 2024-09-16T04:13:37.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.306+0000 7f876db08640 1 -- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f876806db90 msgr2=0x7f876813d1d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.42:3300/0 2024-09-16T04:13:37.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.306+0000 7f876db08640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f876806db90 0x7f876813d1d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:13:37.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.306+0000 7f876e309640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f8768136e10 0x7f87681cc8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:13:37.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.306+0000 7f876e309640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f8768136e10 0x7f87681cc8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:46316/0 (socket says 172.21.15.42:46316) 2024-09-16T04:13:37.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.306+0000 7f876e309640 1 -- 172.21.15.42:0/3527882181 learned_addr learned my addr 172.21.15.42:0/3527882181 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:13:37.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.306+0000 7f876e309640 10 monclient(hunting): get_auth_request con 0x7f8768136e10 auth_method 0 2024-09-16T04:13:37.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.306+0000 7f876e309640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:13:37.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.306+0000 7f876e309640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:37.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.306+0000 7f876e309640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:13:37.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.306+0000 7f876e309640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:13:37.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.306+0000 7f876e309640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:13:37.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.306+0000 7f876e309640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:13:37.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.306+0000 7f876d307640 1 --2- 172.21.15.42:0/3527882181 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f87680a37f0 0x7f87681cc3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:13:37.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.306+0000 7f876e309640 10 monclient(hunting): handle_auth_done global_id 1094153 payload 824 2024-09-16T04:13:37.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.306+0000 7f876e309640 1 -- 172.21.15.42:0/3527882181 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f87680a37f0 msgr2=0x7f87681cc3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:37.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.306+0000 7f876e309640 1 --2- 172.21.15.42:0/3527882181 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f87680a37f0 0x7f87681cc3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:13:37.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.306+0000 7f876e309640 1 -- 172.21.15.42:0/3527882181 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f876806db90 msgr2=0x7f876813d1d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:13:37.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.306+0000 7f876e309640 1 --2- 172.21.15.42:0/3527882181 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f876806db90 0x7f876813d1d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:13:37.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.306+0000 7f876e309640 10 monclient: _finish_hunting 0 2024-09-16T04:13:37.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.306+0000 7f876e309640 1 monclient: found mon.noname-c 2024-09-16T04:13:37.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.306+0000 7f876e309640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:13:37.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.306+0000 7f876e309640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T04:13:37.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.306+0000 7f876e309640 1 -- 172.21.15.42:0/3527882181 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f876809f5f0 con 0x7f8768136e10 2024-09-16T04:13:37.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.306+0000 7f876e309640 1 --2- 172.21.15.42:0/3527882181 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f8768136e10 0x7f87681cc8f0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f876000b410 tx=0x7f876000b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:13:37.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.306+0000 7f875e7fc640 10 client.?.objecter ms_handle_connect 0x7f8768136e10 2024-09-16T04:13:37.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.306+0000 7f875e7fc640 10 client.?.objecter resend_mon_ops 2024-09-16T04:13:37.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.306+0000 7f875e7fc640 1 -- 172.21.15.42:0/3527882181 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8760017070 con 0x7f8768136e10 2024-09-16T04:13:37.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.306+0000 7f875e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:13:37.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.306+0000 7f875e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:13:37.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.306+0000 7f875e7fc640 10 monclient: dump: 2024-09-16T04:13:37.324 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:13:37.324 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:13:37.324 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:37.325 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:37.325 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:37.325 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:13:37.325 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:13:37.325 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:13:37.325 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:13:37.325 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:13:37.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.306+0000 7f875e7fc640 10 monclient: _finish_auth 0 2024-09-16T04:13:37.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.306+0000 7f875e7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:13:37.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.306+0000 7f875e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:13:37.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.306+0000 7f875e7fc640 1 -- 172.21.15.42:0/3527882181 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8760003e10 con 0x7f8768136e10 2024-09-16T04:13:37.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.306+0000 7f876fd6c640 5 monclient: authenticate success, global_id 1094153 2024-09-16T04:13:37.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.306+0000 7f876fd6c640 10 monclient: _renew_subs 2024-09-16T04:13:37.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.306+0000 7f876fd6c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:13:37.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.306+0000 7f876fd6c640 1 -- 172.21.15.42:0/3527882181 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f876809f8d0 con 0x7f8768136e10 2024-09-16T04:13:37.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.306+0000 7f875e7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:13:37.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.306+0000 7f875e7fc640 1 -- 172.21.15.42:0/3527882181 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8760006310 con 0x7f8768136e10 2024-09-16T04:13:37.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.306+0000 7f875e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:13:37.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.306+0000 7f875e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:13:37.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.306+0000 7f875e7fc640 10 monclient: dump: 2024-09-16T04:13:37.326 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:13:37.326 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:13:37.326 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:37.326 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:37.326 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:37.326 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:13:37.326 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:13:37.326 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:13:37.327 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:13:37.327 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:13:37.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.306+0000 7f875ffff640 4 set_mon_vals no callback set 2024-09-16T04:13:37.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.306+0000 7f876fd6c640 10 client.1094153.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:13:37.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.306+0000 7f876fd6c640 10 monclient: _renew_subs 2024-09-16T04:13:37.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.306+0000 7f876fd6c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:13:37.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.306+0000 7f876fd6c640 1 -- 172.21.15.42:0/3527882181 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f87680a8a00 con 0x7f8768136e10 2024-09-16T04:13:37.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.306+0000 7f876fd6c640 1 librados: init done 2024-09-16T04:13:37.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.306+0000 7f876fd6c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:13:37.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.306+0000 7f876fd6c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:13:37.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.306+0000 7f876fd6c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:13:37.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.306+0000 7f876fd6c640 1 -- 172.21.15.42:0/3527882181 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f87680a3c00 con 0x7f8768136e10 2024-09-16T04:13:37.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.306+0000 7f875e7fc640 1 -- 172.21.15.42:0/3527882181 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f876001a070 con 0x7f8768136e10 2024-09-16T04:13:37.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.306+0000 7f875e7fc640 1 --2- 172.21.15.42:0/3527882181 >> v2:172.21.15.154:6812/254845342 conn(0x7f874c0377c0 0x7f874c039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:13:37.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.306+0000 7f875e7fc640 1 -- 172.21.15.42:0/3527882181 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(706..706 src has 681..706) ==== 5040+0+0 (secure 0 0 0) 0x7f8760047ff0 con 0x7f8768136e10 2024-09-16T04:13:37.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.306+0000 7f875e7fc640 10 client.1094153.objecter ms_dispatch 0x7f8768000ba0 osd_map(706..706 src has 681..706) 2024-09-16T04:13:37.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.306+0000 7f875e7fc640 3 client.1094153.objecter handle_osd_map got epochs [706,706] > 0 2024-09-16T04:13:37.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.306+0000 7f875e7fc640 3 client.1094153.objecter handle_osd_map decoding full epoch 706 2024-09-16T04:13:37.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.306+0000 7f875e7fc640 20 client.1094153.objecter dump_active .. 0 homeless 2024-09-16T04:13:37.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.306+0000 7f876db08640 1 --2- 172.21.15.42:0/3527882181 >> v2:172.21.15.154:6812/254845342 conn(0x7f874c0377c0 0x7f874c039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:13:37.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.306+0000 7f876db08640 10 monclient: get_auth_request con 0x7f874c0377c0 auth_method 0 2024-09-16T04:13:37.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.306+0000 7f876db08640 1 --2- 172.21.15.42:0/3527882181 >> v2:172.21.15.154:6812/254845342 conn(0x7f874c0377c0 0x7f874c039c80 secure :-1 s=READY pgs=3995 cs=0 l=1 rev1=1 crypto rx=0x7f8764009ac0 tx=0x7f876400b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:13:37.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.306+0000 7f875e7fc640 10 client.1094153.objecter ms_handle_connect 0x7f874c0377c0 2024-09-16T04:13:37.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.310+0000 7f875e7fc640 1 -- 172.21.15.42:0/3527882181 <== mon.2 v2:172.21.15.42:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f8760019e10 con 0x7f8768136e10 2024-09-16T04:13:37.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.310+0000 7f875e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:13:37.328 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.310+0000 7f875e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:13:37.543 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.538+0000 7f876fd6c640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:13:37.543 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.538+0000 7f876fd6c640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:13:37.543 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.538+0000 7f876fd6c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:13:37.544 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.538+0000 7f876fd6c640 1 -- 172.21.15.42:0/3527882181 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f876809de80 con 0x7f8768136e10 2024-09-16T04:13:37.544 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.538+0000 7f875e7fc640 1 -- 172.21.15.42:0/3527882181 <== mon.2 v2:172.21.15.42:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7f876000fb90 con 0x7f8768136e10 2024-09-16T04:13:37.544 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.538+0000 7f875e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:13:37.544 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.538+0000 7f875e7fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:13:37.544 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":286,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:13:37.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.542+0000 7f876fd6c640 10 librados: watch_flush enter 2024-09-16T04:13:37.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.542+0000 7f876fd6c640 10 librados: watch_flush exit 2024-09-16T04:13:37.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.542+0000 7f876fd6c640 20 client.1094153.objecter shutdown clearing up homeless session... 2024-09-16T04:13:37.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.542+0000 7f876fd6c640 10 client.1094153.objecter successfully canceled tick 2024-09-16T04:13:37.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.542+0000 7f876fd6c640 1 -- 172.21.15.42:0/3527882181 >> v2:172.21.15.154:6812/254845342 conn(0x7f874c0377c0 msgr2=0x7f874c039c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:37.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.542+0000 7f876fd6c640 1 --2- 172.21.15.42:0/3527882181 >> v2:172.21.15.154:6812/254845342 conn(0x7f874c0377c0 0x7f874c039c80 secure :-1 s=READY pgs=3995 cs=0 l=1 rev1=1 crypto rx=0x7f8764009ac0 tx=0x7f876400b040 comp rx=0 tx=0).stop 2024-09-16T04:13:37.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.542+0000 7f876fd6c640 10 monclient: shutdown 2024-09-16T04:13:37.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.542+0000 7f876fd6c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:13:37.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.542+0000 7f876fd6c640 1 -- 172.21.15.42:0/3527882181 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f8768136e10 msgr2=0x7f87681cc8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:37.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.542+0000 7f876fd6c640 1 --2- 172.21.15.42:0/3527882181 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f8768136e10 0x7f87681cc8f0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f876000b410 tx=0x7f876000b8d0 comp rx=0 tx=0).stop 2024-09-16T04:13:37.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.542+0000 7f876fd6c640 1 -- 172.21.15.42:0/3527882181 shutdown_connections 2024-09-16T04:13:37.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.542+0000 7f876fd6c640 1 --2- 172.21.15.42:0/3527882181 >> v2:172.21.15.154:6812/254845342 conn(0x7f874c0377c0 0x7f874c039c80 unknown :-1 s=CLOSED pgs=3995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:13:37.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.542+0000 7f876fd6c640 1 --2- 172.21.15.42:0/3527882181 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f8768136e10 0x7f87681cc8f0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:13:37.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.542+0000 7f876fd6c640 1 --2- 172.21.15.42:0/3527882181 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f87680a37f0 0x7f87681cc3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:13:37.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.542+0000 7f876fd6c640 1 --2- 172.21.15.42:0/3527882181 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f876806db90 0x7f876813d1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:13:37.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.542+0000 7f876fd6c640 1 -- 172.21.15.42:0/3527882181 >> 172.21.15.42:0/3527882181 conn(0x7f87680a9fb0 msgr2=0x7f87680a8210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:13:37.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.542+0000 7f876fd6c640 1 -- 172.21.15.42:0/3527882181 shutdown_connections 2024-09-16T04:13:37.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.542+0000 7f876fd6c640 1 -- 172.21.15.42:0/3527882181 wait complete. 2024-09-16T04:13:37.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:37.542+0000 7f876fd6c640 1 librados: shutdown 2024-09-16T04:13:40.562 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:13:40.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.686+0000 7ff4c9a73640 10 monclient: get_monmap_and_config 2024-09-16T04:13:40.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.686+0000 7ff4c9a73640 10 monclient: build_initial_monmap 2024-09-16T04:13:40.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.686+0000 7ff4c9a73640 10 monclient: monmap: 2024-09-16T04:13:40.691 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:13:40.691 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:13:40.691 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:13:40.689784+0000 2024-09-16T04:13:40.691 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:13:40.689784+0000 2024-09-16T04:13:40.691 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:13:40.691 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:13:40.691 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:13:40.692 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:13:40.692 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:13:40.692 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:13:40.692 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:13:40.692 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:13:40.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.686+0000 7ff4c9a73640 1 Processor -- start 2024-09-16T04:13:40.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.686+0000 7ff4c9a73640 1 -- start start 2024-09-16T04:13:40.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.686+0000 7ff4c9a73640 10 monclient: init 2024-09-16T04:13:40.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.686+0000 7ff4c9a73640 10 monclient: _reopen_session rank -1 2024-09-16T04:13:40.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.686+0000 7ff4c9a73640 10 monclient: _add_conns ranks=[4,3,2,0,1] 2024-09-16T04:13:40.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.686+0000 7ff4c9a73640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7ff4c40a1330 0x7ff4c40a1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:13:40.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.686+0000 7ff4c9a73640 10 monclient(hunting): picked mon.noname-c con 0x7ff4c40a1330 addr v2:172.21.15.154:3300/0 2024-09-16T04:13:40.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.686+0000 7ff4c9a73640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff4c406dac0 0x7ff4c40a1c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:13:40.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.686+0000 7ff4c9a73640 10 monclient(hunting): picked mon.noname-a con 0x7ff4c406dac0 addr v2:172.21.15.42:3300/0 2024-09-16T04:13:40.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.686+0000 7ff4c9a73640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff4c4135100 0x7ff4c4135550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:13:40.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.686+0000 7ff4c9a73640 10 monclient(hunting): picked mon.noname-b con 0x7ff4c4135100 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:13:40.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.686+0000 7ff4c9a73640 10 monclient(hunting): start opening mon connection 2024-09-16T04:13:40.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.686+0000 7ff4c9a73640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7ff4c409c7d0 con 0x7ff4c406dac0 2024-09-16T04:13:40.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.686+0000 7ff4c9a73640 10 monclient(hunting): start opening mon connection 2024-09-16T04:13:40.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.686+0000 7ff4c9a73640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7ff4c409c090 con 0x7ff4c40a1330 2024-09-16T04:13:40.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.686+0000 7ff4c9a73640 10 monclient(hunting): start opening mon connection 2024-09-16T04:13:40.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.686+0000 7ff4c9a73640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7ff4c409f610 con 0x7ff4c4135100 2024-09-16T04:13:40.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.686+0000 7ff4c9a73640 10 monclient(hunting): _renew_subs 2024-09-16T04:13:40.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.686+0000 7ff4c9a73640 10 monclient(hunting): authenticate will time out at 2867.659205s 2024-09-16T04:13:40.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.686+0000 7ff4c2ffd640 1 -- >> v2:172.21.15.42:3300/0 conn(0x7ff4c406dac0 msgr2=0x7ff4c40a1c50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3300/0 2024-09-16T04:13:40.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.686+0000 7ff4c2ffd640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7ff4c406dac0 0x7ff4c40a1c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:13:40.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.686+0000 7ff4c3fff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff4c4135100 0x7ff4c4135550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:13:40.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.686+0000 7ff4c3fff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff4c4135100 0x7ff4c4135550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:46326/0 (socket says 172.21.15.42:46326) 2024-09-16T04:13:40.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.686+0000 7ff4c3fff640 1 -- 172.21.15.42:0/1301650732 learned_addr learned my addr 172.21.15.42:0/1301650732 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:13:40.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.686+0000 7ff4c3fff640 10 monclient(hunting): get_auth_request con 0x7ff4c4135100 auth_method 0 2024-09-16T04:13:40.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.686+0000 7ff4c3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:13:40.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.686+0000 7ff4c3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:40.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.686+0000 7ff4c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:13:40.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.686+0000 7ff4c3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:13:40.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.686+0000 7ff4c3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:13:40.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.686+0000 7ff4c3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:13:40.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.686+0000 7ff4c37fe640 1 --2- 172.21.15.42:0/1301650732 >> v2:172.21.15.154:3300/0 conn(0x7ff4c40a1330 0x7ff4c40a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:13:40.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.686+0000 7ff4c37fe640 10 monclient(hunting): get_auth_request con 0x7ff4c40a1330 auth_method 0 2024-09-16T04:13:40.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.686+0000 7ff4c37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:13:40.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.686+0000 7ff4c37fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:40.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.686+0000 7ff4c37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:13:40.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.686+0000 7ff4c3fff640 10 monclient(hunting): handle_auth_done global_id 1094156 payload 274 2024-09-16T04:13:40.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.686+0000 7ff4c3fff640 1 -- 172.21.15.42:0/1301650732 >> v2:172.21.15.154:3300/0 conn(0x7ff4c40a1330 msgr2=0x7ff4c40a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:13:40.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.686+0000 7ff4c3fff640 1 --2- 172.21.15.42:0/1301650732 >> v2:172.21.15.154:3300/0 conn(0x7ff4c40a1330 0x7ff4c40a1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:13:40.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.686+0000 7ff4c3fff640 1 -- 172.21.15.42:0/1301650732 >> v2:172.21.15.42:3300/0 conn(0x7ff4c406dac0 msgr2=0x7ff4c40a1c50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:13:40.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.686+0000 7ff4c3fff640 1 --2- 172.21.15.42:0/1301650732 >> v2:172.21.15.42:3300/0 conn(0x7ff4c406dac0 0x7ff4c40a1c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:13:40.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.686+0000 7ff4c3fff640 10 monclient: _finish_hunting 0 2024-09-16T04:13:40.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.686+0000 7ff4c3fff640 1 monclient: found mon.noname-b 2024-09-16T04:13:40.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.686+0000 7ff4c3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:13:40.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.686+0000 7ff4c3fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T04:13:40.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.686+0000 7ff4c3fff640 1 -- 172.21.15.42:0/1301650732 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4c409f8f0 con 0x7ff4c4135100 2024-09-16T04:13:40.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.690+0000 7ff4c3fff640 1 --2- 172.21.15.42:0/1301650732 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff4c4135100 0x7ff4c4135550 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7ff4c406add0 tx=0x7ff4bc01c930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ad0f6670e7e3b85 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:13:40.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.690+0000 7ff4c27fc640 1 -- 172.21.15.42:0/1301650732 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4bc0a5770 con 0x7ff4c4135100 2024-09-16T04:13:40.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.690+0000 7ff4c27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:13:40.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.690+0000 7ff4c27fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:13:40.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.690+0000 7ff4c27fc640 10 monclient: dump: 2024-09-16T04:13:40.705 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:13:40.705 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:13:40.705 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:40.705 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:40.705 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:40.705 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:13:40.706 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:13:40.706 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:13:40.706 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:13:40.706 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:13:40.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.690+0000 7ff4c27fc640 10 monclient: _finish_auth 0 2024-09-16T04:13:40.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.690+0000 7ff4c27fc640 10 monclient: _check_auth_tickets 2024-09-16T04:13:40.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.690+0000 7ff4c27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:13:40.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.690+0000 7ff4c9a73640 5 monclient: authenticate success, global_id 1094156 2024-09-16T04:13:40.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.690+0000 7ff4c9a73640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:13:40.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.690+0000 7ff4c27fc640 1 -- 172.21.15.42:0/1301650732 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4bc0a5910 con 0x7ff4c4135100 2024-09-16T04:13:40.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.690+0000 7ff4c27fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:13:40.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.690+0000 7ff4c9a73640 10 monclient: get_monmap_and_config success 2024-09-16T04:13:40.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.690+0000 7ff4c9a73640 4 set_mon_vals no callback set 2024-09-16T04:13:40.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.690+0000 7ff4c27fc640 1 -- 172.21.15.42:0/1301650732 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4bc0a5ce0 con 0x7ff4c4135100 2024-09-16T04:13:40.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.690+0000 7ff4c27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:13:40.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.690+0000 7ff4c27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:13:40.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.690+0000 7ff4c27fc640 10 monclient: dump: 2024-09-16T04:13:40.707 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:13:40.707 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:13:40.707 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:40.707 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:40.707 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:40.707 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:13:40.707 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:13:40.707 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:13:40.707 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:13:40.707 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:13:40.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.690+0000 7ff4c9a73640 10 monclient: shutdown 2024-09-16T04:13:40.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.690+0000 7ff4c9a73640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:13:40.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.690+0000 7ff4c9a73640 1 -- 172.21.15.42:0/1301650732 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff4c4135100 msgr2=0x7ff4c4135550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:40.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.690+0000 7ff4c9a73640 1 --2- 172.21.15.42:0/1301650732 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff4c4135100 0x7ff4c4135550 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7ff4c406add0 tx=0x7ff4bc01c930 comp rx=0 tx=0).stop 2024-09-16T04:13:40.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.690+0000 7ff4c9a73640 1 -- 172.21.15.42:0/1301650732 shutdown_connections 2024-09-16T04:13:40.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.690+0000 7ff4c9a73640 1 --2- 172.21.15.42:0/1301650732 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff4c4135100 0x7ff4c4135550 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:13:40.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.690+0000 7ff4c9a73640 1 --2- 172.21.15.42:0/1301650732 >> v2:172.21.15.42:3300/0 conn(0x7ff4c406dac0 0x7ff4c40a1c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:13:40.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.690+0000 7ff4c9a73640 1 --2- 172.21.15.42:0/1301650732 >> v2:172.21.15.154:3300/0 conn(0x7ff4c40a1330 0x7ff4c40a1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:13:40.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.690+0000 7ff4c9a73640 1 -- 172.21.15.42:0/1301650732 >> 172.21.15.42:0/1301650732 conn(0x7ff4c40a2420 msgr2=0x7ff4c40a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:13:40.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.690+0000 7ff4c9a73640 1 -- 172.21.15.42:0/1301650732 shutdown_connections 2024-09-16T04:13:40.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.690+0000 7ff4c9a73640 1 -- 172.21.15.42:0/1301650732 wait complete. 2024-09-16T04:13:40.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.690+0000 7ff4c9a73640 10 monclient: build_initial_monmap 2024-09-16T04:13:40.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.690+0000 7ff4c9a73640 10 monclient: monmap: 2024-09-16T04:13:40.708 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:13:40.709 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:13:40.709 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:13:40.709 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:13:40.709 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:13:40.709 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:13:40.709 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:13:40.709 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:13:40.709 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:13:40.709 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:13:40.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.694+0000 7ff4c9a73640 1 librados: starting msgr at 2024-09-16T04:13:40.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.694+0000 7ff4c9a73640 1 librados: starting objecter 2024-09-16T04:13:40.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.694+0000 7ff4c9a73640 1 Processor -- start 2024-09-16T04:13:40.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.694+0000 7ff4c9a73640 1 -- start start 2024-09-16T04:13:40.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.694+0000 7ff4c9a73640 1 librados: setting wanted keys 2024-09-16T04:13:40.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.694+0000 7ff4c9a73640 1 librados: calling monclient init 2024-09-16T04:13:40.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.694+0000 7ff4c9a73640 10 monclient: init 2024-09-16T04:13:40.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.694+0000 7ff4c9a73640 10 monclient: _reopen_session rank -1 2024-09-16T04:13:40.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.694+0000 7ff4c9a73640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:13:40.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.694+0000 7ff4c9a73640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff4c40a1330 0x7ff4c41cef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:13:40.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.694+0000 7ff4c9a73640 10 monclient(hunting): picked mon.noname-b con 0x7ff4c40a1330 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:13:40.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.694+0000 7ff4c9a73640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff4c4135100 0x7ff4c4134590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:13:40.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.694+0000 7ff4c9a73640 10 monclient(hunting): picked mon.noname-a con 0x7ff4c4135100 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:13:40.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.694+0000 7ff4c9a73640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff4c4134ad0 0x7ff4c409cb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:13:40.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.694+0000 7ff4c9a73640 10 monclient(hunting): picked mon.noname-c con 0x7ff4c4134ad0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:13:40.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.694+0000 7ff4c9a73640 10 monclient(hunting): start opening mon connection 2024-09-16T04:13:40.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.694+0000 7ff4c9a73640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7ff4c409eef0 con 0x7ff4c4135100 2024-09-16T04:13:40.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.694+0000 7ff4c9a73640 10 monclient(hunting): start opening mon connection 2024-09-16T04:13:40.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.694+0000 7ff4c9a73640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7ff4c409ed70 con 0x7ff4c40a1330 2024-09-16T04:13:40.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.694+0000 7ff4c9a73640 10 monclient(hunting): start opening mon connection 2024-09-16T04:13:40.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.694+0000 7ff4c9a73640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7ff4c413eb70 con 0x7ff4c4134ad0 2024-09-16T04:13:40.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.694+0000 7ff4c9a73640 10 monclient(hunting): _renew_subs 2024-09-16T04:13:40.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.694+0000 7ff4c9a73640 10 monclient(hunting): authenticate will time out at 2867.664683s 2024-09-16T04:13:40.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.694+0000 7ff4c2ffd640 1 -- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff4c4135100 msgr2=0x7ff4c4134590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.42:3300/0 2024-09-16T04:13:40.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.694+0000 7ff4c2ffd640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff4c4135100 0x7ff4c4134590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:13:40.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.694+0000 7ff4c3fff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff4c4134ad0 0x7ff4c409cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:13:40.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.694+0000 7ff4c3fff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff4c4134ad0 0x7ff4c409cb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:46334/0 (socket says 172.21.15.42:46334) 2024-09-16T04:13:40.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.694+0000 7ff4c3fff640 1 -- 172.21.15.42:0/439437260 learned_addr learned my addr 172.21.15.42:0/439437260 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:13:40.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.694+0000 7ff4c3fff640 10 monclient(hunting): get_auth_request con 0x7ff4c4134ad0 auth_method 0 2024-09-16T04:13:40.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.694+0000 7ff4c3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:13:40.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.694+0000 7ff4c3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:40.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.694+0000 7ff4c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:13:40.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.694+0000 7ff4c3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:13:40.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.694+0000 7ff4c3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:13:40.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.694+0000 7ff4c3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:13:40.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.694+0000 7ff4c3fff640 10 monclient(hunting): handle_auth_done global_id 1094159 payload 824 2024-09-16T04:13:40.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.694+0000 7ff4c37fe640 1 --2- 172.21.15.42:0/439437260 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff4c40a1330 0x7ff4c41cef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:13:40.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.694+0000 7ff4c3fff640 1 -- 172.21.15.42:0/439437260 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff4c40a1330 msgr2=0x7ff4c41cef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:40.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.694+0000 7ff4c3fff640 1 --2- 172.21.15.42:0/439437260 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff4c40a1330 0x7ff4c41cef50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:13:40.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.694+0000 7ff4c3fff640 1 -- 172.21.15.42:0/439437260 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff4c4135100 msgr2=0x7ff4c4134590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:13:40.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.694+0000 7ff4c3fff640 1 --2- 172.21.15.42:0/439437260 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff4c4135100 0x7ff4c4134590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:13:40.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.694+0000 7ff4c3fff640 10 monclient: _finish_hunting 0 2024-09-16T04:13:40.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.694+0000 7ff4c3fff640 1 monclient: found mon.noname-c 2024-09-16T04:13:40.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.694+0000 7ff4c3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:13:40.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.694+0000 7ff4c3fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T04:13:40.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.694+0000 7ff4c3fff640 1 -- 172.21.15.42:0/439437260 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4c4071af0 con 0x7ff4c4134ad0 2024-09-16T04:13:40.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.694+0000 7ff4c3fff640 1 --2- 172.21.15.42:0/439437260 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff4c4134ad0 0x7ff4c409cb60 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7ff4bc01afe0 tx=0x7ff4bc0196d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:13:40.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.694+0000 7ff4bbfff640 10 client.?.objecter ms_handle_connect 0x7ff4c4134ad0 2024-09-16T04:13:40.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.694+0000 7ff4bbfff640 10 client.?.objecter resend_mon_ops 2024-09-16T04:13:40.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.694+0000 7ff4bbfff640 1 -- 172.21.15.42:0/439437260 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4bc0bc070 con 0x7ff4c4134ad0 2024-09-16T04:13:40.714 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.694+0000 7ff4bbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:13:40.714 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.694+0000 7ff4bbfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:13:40.714 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.694+0000 7ff4bbfff640 10 monclient: dump: 2024-09-16T04:13:40.714 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:13:40.714 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:13:40.714 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:40.714 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:40.714 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:40.714 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:13:40.714 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:13:40.714 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:13:40.714 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:13:40.714 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:13:40.714 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.694+0000 7ff4bbfff640 10 monclient: _finish_auth 0 2024-09-16T04:13:40.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.694+0000 7ff4bbfff640 10 monclient: _check_auth_tickets 2024-09-16T04:13:40.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.694+0000 7ff4bbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:13:40.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.694+0000 7ff4bbfff640 1 -- 172.21.15.42:0/439437260 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4bc0b8640 con 0x7ff4c4134ad0 2024-09-16T04:13:40.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.694+0000 7ff4bbfff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:13:40.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.694+0000 7ff4bbfff640 1 -- 172.21.15.42:0/439437260 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4bc0b89f0 con 0x7ff4c4134ad0 2024-09-16T04:13:40.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.694+0000 7ff4bbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:13:40.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.694+0000 7ff4bbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:13:40.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.694+0000 7ff4bbfff640 10 monclient: dump: 2024-09-16T04:13:40.715 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:13:40.715 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:13:40.715 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:40.715 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:40.715 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:40.715 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:13:40.716 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:13:40.716 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:13:40.716 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:13:40.716 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:13:40.716 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.694+0000 7ff4c9a73640 5 monclient: authenticate success, global_id 1094159 2024-09-16T04:13:40.716 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.694+0000 7ff4c9a73640 10 monclient: _renew_subs 2024-09-16T04:13:40.716 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.694+0000 7ff4c1ffb640 4 set_mon_vals no callback set 2024-09-16T04:13:40.716 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.694+0000 7ff4c9a73640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:13:40.716 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.694+0000 7ff4c9a73640 1 -- 172.21.15.42:0/439437260 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4c409e600 con 0x7ff4c4134ad0 2024-09-16T04:13:40.716 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.694+0000 7ff4c9a73640 10 client.1094159.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:13:40.716 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.694+0000 7ff4c9a73640 10 monclient: _renew_subs 2024-09-16T04:13:40.716 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.694+0000 7ff4c9a73640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:13:40.716 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.694+0000 7ff4c9a73640 1 -- 172.21.15.42:0/439437260 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4c4121f50 con 0x7ff4c4134ad0 2024-09-16T04:13:40.716 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.694+0000 7ff4c9a73640 1 librados: init done 2024-09-16T04:13:40.717 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.694+0000 7ff4c9a73640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:13:40.717 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.694+0000 7ff4c9a73640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:13:40.717 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.694+0000 7ff4c9a73640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:13:40.717 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.694+0000 7ff4bbfff640 1 -- 172.21.15.42:0/439437260 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7ff4bc0bf070 con 0x7ff4c4134ad0 2024-09-16T04:13:40.717 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.694+0000 7ff4c9a73640 1 -- 172.21.15.42:0/439437260 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff4c4135550 con 0x7ff4c4134ad0 2024-09-16T04:13:40.717 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.694+0000 7ff4bbfff640 1 --2- 172.21.15.42:0/439437260 >> v2:172.21.15.154:6812/254845342 conn(0x7ff4a40377c0 0x7ff4a4039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:13:40.717 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.694+0000 7ff4bbfff640 1 -- 172.21.15.42:0/439437260 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(706..706 src has 681..706) ==== 5040+0+0 (secure 0 0 0) 0x7ff4bc0e3940 con 0x7ff4c4134ad0 2024-09-16T04:13:40.717 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.694+0000 7ff4bbfff640 10 client.1094159.objecter ms_dispatch 0x7ff4c4000ba0 osd_map(706..706 src has 681..706) 2024-09-16T04:13:40.717 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.694+0000 7ff4bbfff640 3 client.1094159.objecter handle_osd_map got epochs [706,706] > 0 2024-09-16T04:13:40.717 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.694+0000 7ff4bbfff640 3 client.1094159.objecter handle_osd_map decoding full epoch 706 2024-09-16T04:13:40.717 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.694+0000 7ff4bbfff640 20 client.1094159.objecter dump_active .. 0 homeless 2024-09-16T04:13:40.717 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.694+0000 7ff4c37fe640 1 --2- 172.21.15.42:0/439437260 >> v2:172.21.15.154:6812/254845342 conn(0x7ff4a40377c0 0x7ff4a4039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:13:40.717 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.694+0000 7ff4c37fe640 10 monclient: get_auth_request con 0x7ff4a40377c0 auth_method 0 2024-09-16T04:13:40.717 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.698+0000 7ff4bbfff640 10 client.1094159.objecter ms_handle_connect 0x7ff4a40377c0 2024-09-16T04:13:40.718 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.698+0000 7ff4c37fe640 1 --2- 172.21.15.42:0/439437260 >> v2:172.21.15.154:6812/254845342 conn(0x7ff4a40377c0 0x7ff4a4039c80 secure :-1 s=READY pgs=3996 cs=0 l=1 rev1=1 crypto rx=0x7ff4ac009790 tx=0x7ff4ac008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:13:40.718 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.698+0000 7ff4bbfff640 1 -- 172.21.15.42:0/439437260 <== mon.2 v2:172.21.15.42:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7ff4bc0c6070 con 0x7ff4c4134ad0 2024-09-16T04:13:40.718 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.698+0000 7ff4bbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:13:40.718 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.698+0000 7ff4bbfff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:13:40.925 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.922+0000 7ff4c9a73640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:13:40.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.922+0000 7ff4c9a73640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:13:40.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.922+0000 7ff4c9a73640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:13:40.926 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.922+0000 7ff4c9a73640 1 -- 172.21.15.42:0/439437260 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7ff4c40008e0 con 0x7ff4c4134ad0 2024-09-16T04:13:40.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.922+0000 7ff4bbfff640 1 -- 172.21.15.42:0/439437260 <== mon.2 v2:172.21.15.42:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1562 (secure 0 0 0) 0x7ff4bc0165e0 con 0x7ff4c4134ad0 2024-09-16T04:13:40.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.922+0000 7ff4bbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:13:40.927 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.922+0000 7ff4bbfff640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:13:40.927 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":286,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:13:40.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.922+0000 7ff4c9a73640 10 librados: watch_flush enter 2024-09-16T04:13:40.928 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.922+0000 7ff4c9a73640 10 librados: watch_flush exit 2024-09-16T04:13:40.929 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.922+0000 7ff4c9a73640 20 client.1094159.objecter shutdown clearing up homeless session... 2024-09-16T04:13:40.929 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.922+0000 7ff4c9a73640 10 client.1094159.objecter successfully canceled tick 2024-09-16T04:13:40.929 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.922+0000 7ff4c9a73640 1 -- 172.21.15.42:0/439437260 >> v2:172.21.15.154:6812/254845342 conn(0x7ff4a40377c0 msgr2=0x7ff4a4039c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:40.929 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.922+0000 7ff4c9a73640 1 --2- 172.21.15.42:0/439437260 >> v2:172.21.15.154:6812/254845342 conn(0x7ff4a40377c0 0x7ff4a4039c80 secure :-1 s=READY pgs=3996 cs=0 l=1 rev1=1 crypto rx=0x7ff4ac009790 tx=0x7ff4ac008040 comp rx=0 tx=0).stop 2024-09-16T04:13:40.929 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.922+0000 7ff4c9a73640 10 monclient: shutdown 2024-09-16T04:13:40.930 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.922+0000 7ff4c9a73640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:13:40.930 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.922+0000 7ff4c9a73640 1 -- 172.21.15.42:0/439437260 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff4c4134ad0 msgr2=0x7ff4c409cb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:40.930 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.922+0000 7ff4c9a73640 1 --2- 172.21.15.42:0/439437260 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff4c4134ad0 0x7ff4c409cb60 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7ff4bc01afe0 tx=0x7ff4bc0196d0 comp rx=0 tx=0).stop 2024-09-16T04:13:40.930 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.922+0000 7ff4c9a73640 1 -- 172.21.15.42:0/439437260 shutdown_connections 2024-09-16T04:13:40.930 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.922+0000 7ff4c9a73640 1 --2- 172.21.15.42:0/439437260 >> v2:172.21.15.154:6812/254845342 conn(0x7ff4a40377c0 0x7ff4a4039c80 unknown :-1 s=CLOSED pgs=3996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:13:40.930 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.922+0000 7ff4c9a73640 1 --2- 172.21.15.42:0/439437260 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7ff4c4134ad0 0x7ff4c409cb60 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:13:40.930 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.922+0000 7ff4c9a73640 1 --2- 172.21.15.42:0/439437260 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7ff4c4135100 0x7ff4c4134590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:13:40.930 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.922+0000 7ff4c9a73640 1 --2- 172.21.15.42:0/439437260 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7ff4c40a1330 0x7ff4c41cef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:13:40.930 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.922+0000 7ff4c9a73640 1 -- 172.21.15.42:0/439437260 >> 172.21.15.42:0/439437260 conn(0x7ff4c40a2420 msgr2=0x7ff4c40a3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:13:40.930 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.922+0000 7ff4c9a73640 1 -- 172.21.15.42:0/439437260 shutdown_connections 2024-09-16T04:13:40.930 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.922+0000 7ff4c9a73640 1 -- 172.21.15.42:0/439437260 wait complete. 2024-09-16T04:13:40.930 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:40.922+0000 7ff4c9a73640 1 librados: shutdown 2024-09-16T04:13:43.945 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:13:44.080 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:44.074+0000 7fd5206b0640 10 monclient: get_monmap_and_config 2024-09-16T04:13:44.080 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:44.074+0000 7fd5206b0640 10 monclient: build_initial_monmap 2024-09-16T04:13:44.080 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:44.074+0000 7fd5206b0640 10 monclient: monmap: 2024-09-16T04:13:44.080 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:13:44.080 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:13:44.080 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:13:44.078916+0000 2024-09-16T04:13:44.081 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:13:44.078916+0000 2024-09-16T04:13:44.081 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:13:44.081 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:13:44.081 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:13:44.081 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:13:44.081 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:13:44.081 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:13:44.081 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:13:44.081 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:13:44.083 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:44.074+0000 7fd5206b0640 1 Processor -- start 2024-09-16T04:13:44.083 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:44.074+0000 7fd5206b0640 1 -- start start 2024-09-16T04:13:44.083 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:44.074+0000 7fd5206b0640 10 monclient: init 2024-09-16T04:13:44.083 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:44.078+0000 7fd5206b0640 10 monclient: _reopen_session rank -1 2024-09-16T04:13:44.083 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:44.078+0000 7fd5206b0640 10 monclient: _add_conns ranks=[1,2,4,3,0] 2024-09-16T04:13:44.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:44.078+0000 7fd5206b0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd5180a1330 addr v1:172.21.15.154:6789/0 2024-09-16T04:13:44.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:44.078+0000 7fd5206b0640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd51806dac0 0x7fd51813a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:13:44.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:44.078+0000 7fd5206b0640 10 monclient(hunting): picked mon.noname-b con 0x7fd51806dac0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:13:44.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:44.078+0000 7fd5206b0640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fd5180a1e20 0x7fd51813c890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:13:44.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:44.078+0000 7fd5206b0640 10 monclient(hunting): picked mon.noname-c con 0x7fd5180a1e20 addr v2:172.21.15.154:3300/0 2024-09-16T04:13:44.085 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:44.078+0000 7fd5206b0640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd51809c7d0 con 0x7fd5180a1330 2024-09-16T04:13:44.085 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:44.078+0000 7fd5206b0640 10 monclient(hunting): start opening mon connection 2024-09-16T04:13:44.085 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:44.078+0000 7fd5206b0640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fd51809c090 con 0x7fd5180a1e20 2024-09-16T04:13:44.085 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:44.078+0000 7fd5206b0640 10 monclient(hunting): start opening mon connection 2024-09-16T04:13:44.085 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:44.078+0000 7fd5206b0640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fd51809f610 con 0x7fd51806dac0 2024-09-16T04:13:44.085 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:44.078+0000 7fd5206b0640 10 monclient(hunting): _renew_subs 2024-09-16T04:13:44.085 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:44.078+0000 7fd5206b0640 10 monclient(hunting): authenticate will time out at 2871.048603s 2024-09-16T04:13:44.085 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:44.078+0000 7fd51dc4b640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd51806dac0 0x7fd51813a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:13:44.085 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:44.078+0000 7fd51dc4b640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd51806dac0 0x7fd51813a340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:60412/0 (socket says 172.21.15.42:60412) 2024-09-16T04:13:44.085 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:44.078+0000 7fd51dc4b640 1 -- 172.21.15.42:0/3404522325 learned_addr learned my addr 172.21.15.42:0/3404522325 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:13:44.085 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:44.078+0000 7fd51dc4b640 10 monclient(hunting): get_auth_request con 0x7fd51806dac0 auth_method 0 2024-09-16T04:13:44.085 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:44.078+0000 7fd51dc4b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:13:44.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:44.078+0000 7fd51dc4b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:44.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:44.078+0000 7fd51dc4b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:13:44.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:44.078+0000 7fd51dc4b640 1 -- 172.21.15.42:0/3404522325 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd51806dac0 msgr2=0x7fd51813a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:13:44.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:44.078+0000 7fd51dc4b640 1 -- 172.21.15.42:0/3404522325 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd51806dac0 msgr2=0x7fd51813a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:13:44.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:44.078+0000 7fd51dc4b640 1 --2- 172.21.15.42:0/3404522325 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd51806dac0 0x7fd51813a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:13:44.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:44.078+0000 7fd51ec4d640 1 --2- 172.21.15.42:0/3404522325 >> v2:172.21.15.154:3300/0 conn(0x7fd5180a1e20 0x7fd51813c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:13:44.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:44.078+0000 7fd51ec4d640 10 monclient(hunting): get_auth_request con 0x7fd5180a1e20 auth_method 0 2024-09-16T04:13:44.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:44.078+0000 7fd51dc4b640 1 --2- 172.21.15.42:0/3404522325 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd51806dac0 0x7fd51813a340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:13:44.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:44.078+0000 7fd51ec4d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:13:44.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:44.078+0000 7fd51ec4d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:44.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:44.078+0000 7fd51ec4d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:13:44.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:44.078+0000 7fd51ec4d640 1 -- 172.21.15.42:0/3404522325 >> v2:172.21.15.154:3300/0 conn(0x7fd5180a1e20 msgr2=0x7fd51813c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T04:13:44.087 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:44.078+0000 7fd51ec4d640 1 -- 172.21.15.42:0/3404522325 >> v2:172.21.15.154:3300/0 conn(0x7fd5180a1e20 msgr2=0x7fd51813c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:13:44.087 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:44.078+0000 7fd51ec4d640 1 --2- 172.21.15.42:0/3404522325 >> v2:172.21.15.154:3300/0 conn(0x7fd5180a1e20 0x7fd51813c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:13:44.087 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:44.078+0000 7fd51ec4d640 1 --2- 172.21.15.42:0/3404522325 >> v2:172.21.15.154:3300/0 conn(0x7fd5180a1e20 0x7fd51813c890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:13:44.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:44.282+0000 7fd51dc4b640 1 --2- 172.21.15.42:0/3404522325 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd51806dac0 0x7fd51813a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:13:44.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:44.282+0000 7fd51dc4b640 10 monclient(hunting): get_auth_request con 0x7fd51806dac0 auth_method 0 2024-09-16T04:13:44.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:44.282+0000 7fd51dc4b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:13:44.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:44.282+0000 7fd51dc4b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:44.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:44.282+0000 7fd51dc4b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:13:44.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:44.282+0000 7fd51ec4d640 1 --2- 172.21.15.42:0/3404522325 >> v2:172.21.15.154:3300/0 conn(0x7fd5180a1e20 0x7fd51813c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:13:44.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:44.282+0000 7fd51ec4d640 10 monclient(hunting): get_auth_request con 0x7fd5180a1e20 auth_method 0 2024-09-16T04:13:44.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:44.282+0000 7fd51ec4d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:13:44.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:44.282+0000 7fd51ec4d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:44.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:44.282+0000 7fd51ec4d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:13:44.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:44.282+0000 7fd51dc4b640 1 -- 172.21.15.42:0/3404522325 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd51806dac0 msgr2=0x7fd51813a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T04:13:44.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:44.282+0000 7fd51dc4b640 1 -- 172.21.15.42:0/3404522325 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd51806dac0 msgr2=0x7fd51813a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:13:44.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:44.282+0000 7fd51dc4b640 1 --2- 172.21.15.42:0/3404522325 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd51806dac0 0x7fd51813a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:13:44.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:44.282+0000 7fd51ec4d640 1 -- 172.21.15.42:0/3404522325 >> v2:172.21.15.154:3300/0 conn(0x7fd5180a1e20 msgr2=0x7fd51813c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:13:44.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:44.282+0000 7fd51dc4b640 1 --2- 172.21.15.42:0/3404522325 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd51806dac0 0x7fd51813a340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T04:13:44.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:44.282+0000 7fd51ec4d640 1 -- 172.21.15.42:0/3404522325 >> v2:172.21.15.154:3300/0 conn(0x7fd5180a1e20 msgr2=0x7fd51813c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:13:44.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:44.282+0000 7fd51ec4d640 1 --2- 172.21.15.42:0/3404522325 >> v2:172.21.15.154:3300/0 conn(0x7fd5180a1e20 0x7fd51813c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:13:44.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:44.282+0000 7fd51ec4d640 1 --2- 172.21.15.42:0/3404522325 >> v2:172.21.15.154:3300/0 conn(0x7fd5180a1e20 0x7fd51813c890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T04:13:44.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:44.686+0000 7fd51dc4b640 1 --2- 172.21.15.42:0/3404522325 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd51806dac0 0x7fd51813a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:13:44.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:44.686+0000 7fd51dc4b640 10 monclient(hunting): get_auth_request con 0x7fd51806dac0 auth_method 0 2024-09-16T04:13:44.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:44.686+0000 7fd51dc4b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:13:44.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:44.686+0000 7fd51dc4b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:44.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:44.686+0000 7fd51dc4b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:13:44.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:44.686+0000 7fd51ec4d640 1 --2- 172.21.15.42:0/3404522325 >> v2:172.21.15.154:3300/0 conn(0x7fd5180a1e20 0x7fd51813c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:13:44.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:44.686+0000 7fd51ec4d640 10 monclient(hunting): get_auth_request con 0x7fd5180a1e20 auth_method 0 2024-09-16T04:13:44.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:44.686+0000 7fd51ec4d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:13:44.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:44.686+0000 7fd51ec4d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:44.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:44.686+0000 7fd51ec4d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:13:44.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:44.686+0000 7fd51dc4b640 1 -- 172.21.15.42:0/3404522325 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd51806dac0 msgr2=0x7fd51813a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:13:44.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:44.686+0000 7fd51dc4b640 1 -- 172.21.15.42:0/3404522325 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd51806dac0 msgr2=0x7fd51813a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:13:44.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:44.686+0000 7fd51dc4b640 1 --2- 172.21.15.42:0/3404522325 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd51806dac0 0x7fd51813a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:13:44.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:44.686+0000 7fd51ec4d640 1 -- 172.21.15.42:0/3404522325 >> v2:172.21.15.154:3300/0 conn(0x7fd5180a1e20 msgr2=0x7fd51813c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T04:13:44.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:44.686+0000 7fd51dc4b640 1 --2- 172.21.15.42:0/3404522325 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd51806dac0 0x7fd51813a340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T04:13:44.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:44.686+0000 7fd51ec4d640 1 -- 172.21.15.42:0/3404522325 >> v2:172.21.15.154:3300/0 conn(0x7fd5180a1e20 msgr2=0x7fd51813c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:13:44.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:44.686+0000 7fd51ec4d640 1 --2- 172.21.15.42:0/3404522325 >> v2:172.21.15.154:3300/0 conn(0x7fd5180a1e20 0x7fd51813c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:13:44.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:44.686+0000 7fd51ec4d640 1 --2- 172.21.15.42:0/3404522325 >> v2:172.21.15.154:3300/0 conn(0x7fd5180a1e20 0x7fd51813c890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T04:13:45.494 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:45.490+0000 7fd51dc4b640 1 --2- 172.21.15.42:0/3404522325 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd51806dac0 0x7fd51813a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:13:45.494 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:45.490+0000 7fd51dc4b640 10 monclient(hunting): get_auth_request con 0x7fd51806dac0 auth_method 0 2024-09-16T04:13:45.494 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:45.490+0000 7fd51dc4b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:13:45.494 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:45.490+0000 7fd51dc4b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:45.494 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:45.490+0000 7fd51dc4b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:13:45.494 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:45.490+0000 7fd51ec4d640 1 --2- 172.21.15.42:0/3404522325 >> v2:172.21.15.154:3300/0 conn(0x7fd5180a1e20 0x7fd51813c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:13:45.495 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:45.490+0000 7fd51ec4d640 10 monclient(hunting): get_auth_request con 0x7fd5180a1e20 auth_method 0 2024-09-16T04:13:45.495 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:45.490+0000 7fd51ec4d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:13:45.495 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:45.490+0000 7fd51ec4d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:45.495 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:45.490+0000 7fd51ec4d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:13:45.495 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:45.490+0000 7fd51ec4d640 1 -- 172.21.15.42:0/3404522325 >> v2:172.21.15.154:3300/0 conn(0x7fd5180a1e20 msgr2=0x7fd51813c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-16T04:13:45.495 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:45.490+0000 7fd51ec4d640 1 -- 172.21.15.42:0/3404522325 >> v2:172.21.15.154:3300/0 conn(0x7fd5180a1e20 msgr2=0x7fd51813c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:13:45.495 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:45.490+0000 7fd51dc4b640 1 -- 172.21.15.42:0/3404522325 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd51806dac0 msgr2=0x7fd51813a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-16T04:13:45.495 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:45.490+0000 7fd51ec4d640 1 --2- 172.21.15.42:0/3404522325 >> v2:172.21.15.154:3300/0 conn(0x7fd5180a1e20 0x7fd51813c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:13:45.495 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:45.490+0000 7fd51dc4b640 1 -- 172.21.15.42:0/3404522325 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd51806dac0 msgr2=0x7fd51813a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:13:45.495 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:45.490+0000 7fd51dc4b640 1 --2- 172.21.15.42:0/3404522325 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd51806dac0 0x7fd51813a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:13:45.495 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:45.490+0000 7fd51ec4d640 1 --2- 172.21.15.42:0/3404522325 >> v2:172.21.15.154:3300/0 conn(0x7fd5180a1e20 0x7fd51813c890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T04:13:45.495 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:45.490+0000 7fd51dc4b640 1 --2- 172.21.15.42:0/3404522325 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd51806dac0 0x7fd51813a340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T04:13:47.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:47.078+0000 7fd50ffff640 10 monclient(hunting): tick 2024-09-16T04:13:47.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:47.078+0000 7fd50ffff640 10 monclient(hunting): _check_auth_tickets 2024-09-16T04:13:47.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:47.078+0000 7fd50ffff640 1 monclient(hunting): continuing hunt 2024-09-16T04:13:47.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:47.078+0000 7fd50ffff640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T04:13:47.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:47.078+0000 7fd50ffff640 1 -- 172.21.15.42:0/3404522325 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd51806dac0 msgr2=0x7fd51813a340 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:13:47.083 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:47.078+0000 7fd50ffff640 1 --2- 172.21.15.42:0/3404522325 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd51806dac0 0x7fd51813a340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:13:47.083 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:47.078+0000 7fd50ffff640 1 -- 172.21.15.42:0/3404522325 >> v2:172.21.15.154:3300/0 conn(0x7fd5180a1e20 msgr2=0x7fd51813c890 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:13:47.083 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:47.078+0000 7fd50ffff640 1 --2- 172.21.15.42:0/3404522325 >> v2:172.21.15.154:3300/0 conn(0x7fd5180a1e20 0x7fd51813c890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:13:47.083 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:47.078+0000 7fd50ffff640 1 -- 172.21.15.42:0/3404522325 >> v1:172.21.15.154:6789/0 conn(0x7fd5180a1330 legacy=0x7fd5180a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:47.083 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:47.078+0000 7fd50ffff640 10 monclient: _add_conns ranks=[0,3] 2024-09-16T04:13:47.083 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:47.078+0000 7fd50ffff640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd5040036b0 addr v1:172.21.15.42:6789/0 2024-09-16T04:13:47.083 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:47.078+0000 7fd50ffff640 1 --2- 172.21.15.42:0/3404522325 >> v2:172.21.15.42:3300/0 conn(0x7fd504007360 0x7fd504009820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:13:47.083 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:47.078+0000 7fd50ffff640 10 monclient(hunting): picked mon.noname-a con 0x7fd504007360 addr v2:172.21.15.42:3300/0 2024-09-16T04:13:47.083 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:47.078+0000 7fd50ffff640 1 -- 172.21.15.42:0/3404522325 --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd51809f8f0 con 0x7fd5040036b0 2024-09-16T04:13:47.083 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:47.078+0000 7fd50ffff640 10 monclient(hunting): start opening mon connection 2024-09-16T04:13:47.083 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:47.078+0000 7fd50ffff640 1 -- 172.21.15.42:0/3404522325 --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7fd51809c090 con 0x7fd504007360 2024-09-16T04:13:47.083 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:47.078+0000 7fd50ffff640 10 monclient(hunting): _renew_subs 2024-09-16T04:13:47.083 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:47.078+0000 7fd51dc4b640 1 -- 172.21.15.42:0/3404522325 >> v2:172.21.15.42:3300/0 conn(0x7fd504007360 msgr2=0x7fd504009820 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3300/0 2024-09-16T04:13:47.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:47.078+0000 7fd51e44c640 1 -- 172.21.15.42:0/3404522325 >> v1:172.21.15.42:6789/0 conn(0x7fd5040036b0 legacy=0x7fd504005b30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.42:6789/0 2024-09-16T04:13:47.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:47.078+0000 7fd51dc4b640 1 --2- 172.21.15.42:0/3404522325 >> v2:172.21.15.42:3300/0 conn(0x7fd504007360 0x7fd504009820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:13:47.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:47.282+0000 7fd51dc4b640 1 -- 172.21.15.42:0/3404522325 >> v2:172.21.15.42:3300/0 conn(0x7fd504007360 msgr2=0x7fd504009820 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3300/0 2024-09-16T04:13:47.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:47.282+0000 7fd51e44c640 1 -- 172.21.15.42:0/3404522325 >> v1:172.21.15.42:6789/0 conn(0x7fd5040036b0 legacy=0x7fd504005b30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.42:6789/0 2024-09-16T04:13:47.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:47.282+0000 7fd51dc4b640 1 --2- 172.21.15.42:0/3404522325 >> v2:172.21.15.42:3300/0 conn(0x7fd504007360 0x7fd504009820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-16T04:13:47.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:47.686+0000 7fd51dc4b640 1 -- 172.21.15.42:0/3404522325 >> v2:172.21.15.42:3300/0 conn(0x7fd504007360 msgr2=0x7fd504009820 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3300/0 2024-09-16T04:13:47.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:47.686+0000 7fd51e44c640 1 -- 172.21.15.42:0/3404522325 >> v1:172.21.15.42:6789/0 conn(0x7fd5040036b0 legacy=0x7fd504005b30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.42:6789/0 2024-09-16T04:13:47.689 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:47.686+0000 7fd51dc4b640 1 --2- 172.21.15.42:0/3404522325 >> v2:172.21.15.42:3300/0 conn(0x7fd504007360 0x7fd504009820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-16T04:13:48.493 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:48.490+0000 7fd51e44c640 1 -- 172.21.15.42:0/3404522325 >> v1:172.21.15.42:6789/0 conn(0x7fd5040036b0 legacy=0x7fd504005b30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.42:6789/0 2024-09-16T04:13:48.493 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:48.490+0000 7fd51dc4b640 1 -- 172.21.15.42:0/3404522325 >> v2:172.21.15.42:3300/0 conn(0x7fd504007360 msgr2=0x7fd504009820 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3300/0 2024-09-16T04:13:48.494 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:48.490+0000 7fd51dc4b640 1 --2- 172.21.15.42:0/3404522325 >> v2:172.21.15.42:3300/0 conn(0x7fd504007360 0x7fd504009820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-16T04:13:50.083 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.078+0000 7fd50ffff640 10 monclient(hunting): tick 2024-09-16T04:13:50.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.078+0000 7fd50ffff640 10 monclient(hunting): _check_auth_tickets 2024-09-16T04:13:50.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.078+0000 7fd50ffff640 1 monclient(hunting): continuing hunt 2024-09-16T04:13:50.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.078+0000 7fd50ffff640 10 monclient(hunting): _reopen_session rank -1 2024-09-16T04:13:50.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.078+0000 7fd50ffff640 1 -- 172.21.15.42:0/3404522325 >> v2:172.21.15.42:3300/0 conn(0x7fd504007360 msgr2=0x7fd504009820 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:13:50.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.078+0000 7fd50ffff640 1 --2- 172.21.15.42:0/3404522325 >> v2:172.21.15.42:3300/0 conn(0x7fd504007360 0x7fd504009820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:13:50.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.078+0000 7fd50ffff640 1 -- 172.21.15.42:0/3404522325 >> v1:172.21.15.42:6789/0 conn(0x7fd5040036b0 legacy=0x7fd504005b30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:13:50.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.078+0000 7fd51ec4d640 1 -- 172.21.15.42:0/3404522325 reap_dead start 2024-09-16T04:13:50.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.078+0000 7fd50ffff640 10 monclient: _add_conns ranks=[2,0,4,3,1] 2024-09-16T04:13:50.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.078+0000 7fd50ffff640 1 --2- 172.21.15.42:0/3404522325 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd504009ec0 0x7fd504009820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:13:50.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.078+0000 7fd50ffff640 10 monclient(hunting): picked mon.noname-b con 0x7fd504009ec0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:13:50.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.078+0000 7fd50ffff640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd50400a3b0 addr v1:172.21.15.42:6789/0 2024-09-16T04:13:50.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.078+0000 7fd50ffff640 1 --2- 172.21.15.42:0/3404522325 >> v2:172.21.15.154:3300/0 conn(0x7fd50400a8f0 0x7fd50400d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:13:50.085 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.078+0000 7fd50ffff640 10 monclient(hunting): picked mon.noname-c con 0x7fd50400a8f0 addr v2:172.21.15.154:3300/0 2024-09-16T04:13:50.085 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.078+0000 7fd50ffff640 1 -- 172.21.15.42:0/3404522325 --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd50400af80 con 0x7fd50400a3b0 2024-09-16T04:13:50.085 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.078+0000 7fd50ffff640 10 monclient(hunting): start opening mon connection 2024-09-16T04:13:50.085 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.078+0000 7fd50ffff640 1 -- 172.21.15.42:0/3404522325 --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fd51809c090 con 0x7fd50400a8f0 2024-09-16T04:13:50.085 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.078+0000 7fd50ffff640 10 monclient(hunting): start opening mon connection 2024-09-16T04:13:50.085 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.078+0000 7fd50ffff640 1 -- 172.21.15.42:0/3404522325 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fd51809f610 con 0x7fd504009ec0 2024-09-16T04:13:50.085 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.078+0000 7fd50ffff640 10 monclient(hunting): _renew_subs 2024-09-16T04:13:50.085 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.078+0000 7fd51dc4b640 1 -- 172.21.15.42:0/3404522325 >> v1:172.21.15.42:6789/0 conn(0x7fd50400a3b0 legacy=0x7fd504005aa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.42:6789/0 2024-09-16T04:13:50.085 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.078+0000 7fd51e44c640 1 --2- 172.21.15.42:0/3404522325 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd504009ec0 0x7fd504009820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:13:50.085 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.078+0000 7fd51e44c640 10 monclient(hunting): get_auth_request con 0x7fd504009ec0 auth_method 0 2024-09-16T04:13:50.085 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.078+0000 7fd51e44c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:13:50.085 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.078+0000 7fd51e44c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:50.085 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.078+0000 7fd51e44c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:13:50.085 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.078+0000 7fd51e44c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:13:50.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.078+0000 7fd51e44c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:13:50.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.078+0000 7fd51e44c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:13:50.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.078+0000 7fd51ec4d640 1 --2- 172.21.15.42:0/3404522325 >> v2:172.21.15.154:3300/0 conn(0x7fd50400a8f0 0x7fd50400d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:13:50.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.078+0000 7fd51ec4d640 10 monclient(hunting): get_auth_request con 0x7fd50400a8f0 auth_method 0 2024-09-16T04:13:50.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.078+0000 7fd51ec4d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:13:50.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.078+0000 7fd51ec4d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:50.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.078+0000 7fd51ec4d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:13:50.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.078+0000 7fd51e44c640 10 monclient(hunting): handle_auth_done global_id 1094165 payload 274 2024-09-16T04:13:50.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.078+0000 7fd51e44c640 1 -- 172.21.15.42:0/3404522325 >> v2:172.21.15.154:3300/0 conn(0x7fd50400a8f0 msgr2=0x7fd50400d3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:13:50.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.078+0000 7fd51e44c640 1 --2- 172.21.15.42:0/3404522325 >> v2:172.21.15.154:3300/0 conn(0x7fd50400a8f0 0x7fd50400d3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:13:50.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.078+0000 7fd51e44c640 1 -- 172.21.15.42:0/3404522325 >> v1:172.21.15.42:6789/0 conn(0x7fd50400a3b0 legacy=0x7fd504005aa0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:13:50.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.078+0000 7fd51e44c640 10 monclient: _finish_hunting 0 2024-09-16T04:13:50.087 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.078+0000 7fd51e44c640 1 monclient: found mon.noname-b 2024-09-16T04:13:50.087 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.078+0000 7fd51e44c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:13:50.087 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.078+0000 7fd51e44c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T04:13:50.087 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.078+0000 7fd51e44c640 1 -- 172.21.15.42:0/3404522325 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd51809f8f0 con 0x7fd504009ec0 2024-09-16T04:13:50.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.086+0000 7fd51e44c640 1 --2- 172.21.15.42:0/3404522325 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd504009ec0 0x7fd504009820 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fd508001160 tx=0x7fd50801c510 comp rx=0 tx=0).ready entity=mon.2 client_cookie=246b6777fcfef958 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:13:50.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.086+0000 7fd51d44a640 1 -- 172.21.15.42:0/3404522325 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd508004220 con 0x7fd504009ec0 2024-09-16T04:13:50.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.086+0000 7fd51d44a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:13:50.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.086+0000 7fd51d44a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:13:50.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.086+0000 7fd51d44a640 10 monclient: dump: 2024-09-16T04:13:50.091 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:13:50.091 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:13:50.091 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:50.091 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:50.091 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:50.091 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:13:50.091 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:13:50.091 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:13:50.091 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:13:50.091 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:13:50.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.086+0000 7fd51d44a640 10 monclient: _finish_auth 0 2024-09-16T04:13:50.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.086+0000 7fd51d44a640 10 monclient: _check_auth_tickets 2024-09-16T04:13:50.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.086+0000 7fd51d44a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:13:50.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.086+0000 7fd51d44a640 1 -- 172.21.15.42:0/3404522325 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd5080a6040 con 0x7fd504009ec0 2024-09-16T04:13:50.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.086+0000 7fd5206b0640 5 monclient: authenticate success, global_id 1094165 2024-09-16T04:13:50.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.086+0000 7fd5206b0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:13:50.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.086+0000 7fd51d44a640 10 monclient: handle_config config(0 keys) 2024-09-16T04:13:50.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.086+0000 7fd5206b0640 10 monclient: get_monmap_and_config success 2024-09-16T04:13:50.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.086+0000 7fd5206b0640 4 set_mon_vals no callback set 2024-09-16T04:13:50.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.086+0000 7fd51d44a640 1 -- 172.21.15.42:0/3404522325 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd508007070 con 0x7fd504009ec0 2024-09-16T04:13:50.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.086+0000 7fd51d44a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:13:50.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.086+0000 7fd51d44a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:13:50.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.086+0000 7fd51d44a640 10 monclient: dump: 2024-09-16T04:13:50.092 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:13:50.092 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:13:50.092 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:50.093 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:50.093 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:50.093 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:13:50.093 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:13:50.093 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:13:50.093 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:13:50.093 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:13:50.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.086+0000 7fd5206b0640 10 monclient: shutdown 2024-09-16T04:13:50.098 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.086+0000 7fd5206b0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:13:50.098 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.086+0000 7fd5206b0640 1 -- 172.21.15.42:0/3404522325 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd504009ec0 msgr2=0x7fd504009820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:50.098 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.086+0000 7fd5206b0640 1 --2- 172.21.15.42:0/3404522325 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd504009ec0 0x7fd504009820 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fd508001160 tx=0x7fd50801c510 comp rx=0 tx=0).stop 2024-09-16T04:13:50.098 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.086+0000 7fd5206b0640 1 -- 172.21.15.42:0/3404522325 shutdown_connections 2024-09-16T04:13:50.098 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.086+0000 7fd5206b0640 1 --2- 172.21.15.42:0/3404522325 >> v2:172.21.15.154:3300/0 conn(0x7fd50400a8f0 0x7fd50400d3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:13:50.098 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.086+0000 7fd5206b0640 1 --2- 172.21.15.42:0/3404522325 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd504009ec0 0x7fd504009820 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:13:50.098 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.086+0000 7fd5206b0640 1 -- 172.21.15.42:0/3404522325 >> 172.21.15.42:0/3404522325 conn(0x7fd5180a2420 msgr2=0x7fd5180a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:13:50.098 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.086+0000 7fd5206b0640 1 -- 172.21.15.42:0/3404522325 shutdown_connections 2024-09-16T04:13:50.098 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.086+0000 7fd5206b0640 1 -- 172.21.15.42:0/3404522325 wait complete. 2024-09-16T04:13:50.098 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.086+0000 7fd5206b0640 10 monclient: build_initial_monmap 2024-09-16T04:13:50.098 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.086+0000 7fd5206b0640 10 monclient: monmap: 2024-09-16T04:13:50.098 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:13:50.098 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:13:50.099 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:13:50.099 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:13:50.099 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:13:50.099 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:13:50.099 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:13:50.099 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:13:50.099 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:13:50.099 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:13:50.099 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.086+0000 7fd5206b0640 1 librados: starting msgr at 2024-09-16T04:13:50.099 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.086+0000 7fd5206b0640 1 librados: starting objecter 2024-09-16T04:13:50.099 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.086+0000 7fd5206b0640 1 Processor -- start 2024-09-16T04:13:50.099 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.086+0000 7fd5206b0640 1 -- start start 2024-09-16T04:13:50.099 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.086+0000 7fd5206b0640 1 librados: setting wanted keys 2024-09-16T04:13:50.099 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.086+0000 7fd5206b0640 1 librados: calling monclient init 2024-09-16T04:13:50.100 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.086+0000 7fd5206b0640 10 monclient: init 2024-09-16T04:13:50.100 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.090+0000 7fd5206b0640 10 monclient: _reopen_session rank -1 2024-09-16T04:13:50.100 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.090+0000 7fd5206b0640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:13:50.100 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.090+0000 7fd5206b0640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd504009ec0 0x7fd5181cc3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:13:50.100 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.090+0000 7fd5206b0640 10 monclient(hunting): picked mon.noname-c con 0x7fd504009ec0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:13:50.100 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.090+0000 7fd5206b0640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd50400a3b0 0x7fd5181cc8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:13:50.100 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.090+0000 7fd5206b0640 10 monclient(hunting): picked mon.noname-a con 0x7fd50400a3b0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:13:50.100 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.090+0000 7fd5206b0640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd50400a8f0 0x7fd518124a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:13:50.100 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.090+0000 7fd5206b0640 10 monclient(hunting): picked mon.noname-b con 0x7fd50400a8f0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:13:50.101 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.090+0000 7fd5206b0640 10 monclient(hunting): start opening mon connection 2024-09-16T04:13:50.101 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.090+0000 7fd5206b0640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fd5180a32f0 con 0x7fd50400a3b0 2024-09-16T04:13:50.101 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.090+0000 7fd5206b0640 10 monclient(hunting): start opening mon connection 2024-09-16T04:13:50.101 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.090+0000 7fd5206b0640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd5180a3170 con 0x7fd50400a8f0 2024-09-16T04:13:50.101 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.090+0000 7fd5206b0640 10 monclient(hunting): start opening mon connection 2024-09-16T04:13:50.101 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.090+0000 7fd5206b0640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fd51809ed70 con 0x7fd504009ec0 2024-09-16T04:13:50.101 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.090+0000 7fd5206b0640 10 monclient(hunting): _renew_subs 2024-09-16T04:13:50.101 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.090+0000 7fd5206b0640 10 monclient(hunting): authenticate will time out at 2877.060796s 2024-09-16T04:13:50.101 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.090+0000 7fd51dc4b640 1 -- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd50400a3b0 msgr2=0x7fd5181cc8e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.42:3300/0 2024-09-16T04:13:50.101 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.090+0000 7fd51dc4b640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd50400a3b0 0x7fd5181cc8e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:13:50.101 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.090+0000 7fd51e44c640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd504009ec0 0x7fd5181cc3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:13:50.101 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.090+0000 7fd51e44c640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd504009ec0 0x7fd5181cc3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:60462/0 (socket says 172.21.15.42:60462) 2024-09-16T04:13:50.101 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.090+0000 7fd51e44c640 1 -- 172.21.15.42:0/1649646901 learned_addr learned my addr 172.21.15.42:0/1649646901 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:13:50.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.090+0000 7fd51e44c640 10 monclient(hunting): get_auth_request con 0x7fd504009ec0 auth_method 0 2024-09-16T04:13:50.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.090+0000 7fd51e44c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:13:50.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.090+0000 7fd51e44c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:50.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.090+0000 7fd51e44c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:13:50.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.090+0000 7fd51e44c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:13:50.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.090+0000 7fd51e44c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:13:50.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.090+0000 7fd51e44c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:13:50.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.090+0000 7fd51ec4d640 1 --2- 172.21.15.42:0/1649646901 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd50400a8f0 0x7fd518124a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:13:50.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.090+0000 7fd51ec4d640 10 monclient(hunting): get_auth_request con 0x7fd50400a8f0 auth_method 0 2024-09-16T04:13:50.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.090+0000 7fd51ec4d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:13:50.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.090+0000 7fd51ec4d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:50.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.090+0000 7fd51ec4d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:13:50.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.090+0000 7fd51e44c640 10 monclient(hunting): handle_auth_done global_id 1094168 payload 824 2024-09-16T04:13:50.102 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.090+0000 7fd51e44c640 1 -- 172.21.15.42:0/1649646901 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd50400a8f0 msgr2=0x7fd518124a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:50.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.090+0000 7fd51e44c640 1 --2- 172.21.15.42:0/1649646901 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd50400a8f0 0x7fd518124a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:13:50.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.090+0000 7fd51e44c640 1 -- 172.21.15.42:0/1649646901 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd50400a3b0 msgr2=0x7fd5181cc8e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:13:50.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.090+0000 7fd51e44c640 1 --2- 172.21.15.42:0/1649646901 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd50400a3b0 0x7fd5181cc8e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:13:50.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.090+0000 7fd51e44c640 10 monclient: _finish_hunting 0 2024-09-16T04:13:50.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.090+0000 7fd51e44c640 1 monclient: found mon.noname-c 2024-09-16T04:13:50.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.090+0000 7fd51e44c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:13:50.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.090+0000 7fd51e44c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T04:13:50.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.090+0000 7fd51e44c640 1 -- 172.21.15.42:0/1649646901 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd518128460 con 0x7fd504009ec0 2024-09-16T04:13:50.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.090+0000 7fd51e44c640 1 --2- 172.21.15.42:0/1649646901 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd504009ec0 0x7fd5181cc3a0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fd50801c9f0 tx=0x7fd5080192b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:13:50.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.090+0000 7fd50effd640 10 client.?.objecter ms_handle_connect 0x7fd504009ec0 2024-09-16T04:13:50.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.090+0000 7fd50effd640 10 client.?.objecter resend_mon_ops 2024-09-16T04:13:50.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.090+0000 7fd50effd640 1 -- 172.21.15.42:0/1649646901 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5080af040 con 0x7fd504009ec0 2024-09-16T04:13:50.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.090+0000 7fd50effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:13:50.103 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.090+0000 7fd50effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:13:50.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.090+0000 7fd50effd640 10 monclient: dump: 2024-09-16T04:13:50.104 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:13:50.104 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:13:50.104 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:50.104 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:50.104 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:50.104 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:13:50.104 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:13:50.104 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:13:50.104 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:13:50.104 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:13:50.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.090+0000 7fd50effd640 10 monclient: _finish_auth 0 2024-09-16T04:13:50.104 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.090+0000 7fd50effd640 10 monclient: _check_auth_tickets 2024-09-16T04:13:50.105 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.090+0000 7fd50effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:13:50.105 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.090+0000 7fd50effd640 1 -- 172.21.15.42:0/1649646901 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd5080b9070 con 0x7fd504009ec0 2024-09-16T04:13:50.105 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.090+0000 7fd5206b0640 5 monclient: authenticate success, global_id 1094168 2024-09-16T04:13:50.105 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.090+0000 7fd5206b0640 10 monclient: _renew_subs 2024-09-16T04:13:50.105 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.090+0000 7fd5206b0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:13:50.105 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.090+0000 7fd5206b0640 1 -- 172.21.15.42:0/1649646901 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd518128740 con 0x7fd504009ec0 2024-09-16T04:13:50.105 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.090+0000 7fd50effd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:13:50.105 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.090+0000 7fd50effd640 1 -- 172.21.15.42:0/1649646901 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd508000e50 con 0x7fd504009ec0 2024-09-16T04:13:50.105 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.090+0000 7fd50effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:13:50.105 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.090+0000 7fd50effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:13:50.105 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.090+0000 7fd50effd640 10 monclient: dump: 2024-09-16T04:13:50.105 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:13:50.105 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:13:50.105 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:50.106 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:50.106 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:50.106 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:13:50.106 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:13:50.106 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:13:50.106 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:13:50.106 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:13:50.106 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.090+0000 7fd51cc49640 4 set_mon_vals no callback set 2024-09-16T04:13:50.106 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.090+0000 7fd5206b0640 10 client.1094168.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:13:50.106 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.090+0000 7fd5206b0640 10 monclient: _renew_subs 2024-09-16T04:13:50.106 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.090+0000 7fd5206b0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:13:50.106 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.090+0000 7fd5206b0640 1 -- 172.21.15.42:0/1649646901 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd518128a20 con 0x7fd504009ec0 2024-09-16T04:13:50.106 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.090+0000 7fd5206b0640 1 librados: init done 2024-09-16T04:13:50.106 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.090+0000 7fd5206b0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:13:50.107 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.090+0000 7fd5206b0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:13:50.107 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.090+0000 7fd5206b0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:13:50.107 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.090+0000 7fd5206b0640 1 -- 172.21.15.42:0/1649646901 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd5180a17a0 con 0x7fd504009ec0 2024-09-16T04:13:50.107 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.090+0000 7fd50effd640 1 -- 172.21.15.42:0/1649646901 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fd508005bd0 con 0x7fd504009ec0 2024-09-16T04:13:50.107 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.090+0000 7fd50effd640 1 --2- 172.21.15.42:0/1649646901 >> v2:172.21.15.154:6812/254845342 conn(0x7fd50403bab0 0x7fd50403df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:13:50.107 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.090+0000 7fd50effd640 1 -- 172.21.15.42:0/1649646901 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(708..708 src has 681..708) ==== 5048+0+0 (secure 0 0 0) 0x7fd5080e2f90 con 0x7fd504009ec0 2024-09-16T04:13:50.107 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.090+0000 7fd50effd640 10 client.1094168.objecter ms_dispatch 0x7fd518000ba0 osd_map(708..708 src has 681..708) 2024-09-16T04:13:50.107 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.090+0000 7fd50effd640 3 client.1094168.objecter handle_osd_map got epochs [708,708] > 0 2024-09-16T04:13:50.107 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.090+0000 7fd50effd640 3 client.1094168.objecter handle_osd_map decoding full epoch 708 2024-09-16T04:13:50.107 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.090+0000 7fd50effd640 20 client.1094168.objecter dump_active .. 0 homeless 2024-09-16T04:13:50.107 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.094+0000 7fd51dc4b640 1 --2- 172.21.15.42:0/1649646901 >> v2:172.21.15.154:6812/254845342 conn(0x7fd50403bab0 0x7fd50403df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:13:50.107 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.094+0000 7fd51dc4b640 10 monclient: get_auth_request con 0x7fd50403bab0 auth_method 0 2024-09-16T04:13:50.108 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.094+0000 7fd50effd640 10 client.1094168.objecter ms_handle_connect 0x7fd50403bab0 2024-09-16T04:13:50.108 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.094+0000 7fd51dc4b640 1 --2- 172.21.15.42:0/1649646901 >> v2:172.21.15.154:6812/254845342 conn(0x7fd50403bab0 0x7fd50403df70 secure :-1 s=READY pgs=3999 cs=0 l=1 rev1=1 crypto rx=0x7fd518126330 tx=0x7fd514009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:13:50.108 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.098+0000 7fd50effd640 1 -- 172.21.15.42:0/1649646901 <== mon.2 v2:172.21.15.42:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7fd508016190 con 0x7fd504009ec0 2024-09-16T04:13:50.108 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.098+0000 7fd50effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:13:50.108 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.098+0000 7fd50effd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:13:50.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.346+0000 7fd5206b0640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:13:50.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.346+0000 7fd5206b0640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:13:50.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.346+0000 7fd5206b0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:13:50.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.346+0000 7fd5206b0640 1 -- 172.21.15.42:0/1649646901 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fd518127010 con 0x7fd504009ec0 2024-09-16T04:13:50.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.350+0000 7fd50effd640 1 -- 172.21.15.42:0/1649646901 <== mon.2 v2:172.21.15.42:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1555 (secure 0 0 0) 0x7fd5080b4810 con 0x7fd504009ec0 2024-09-16T04:13:50.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.350+0000 7fd50effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:13:50.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.350+0000 7fd50effd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:13:50.354 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":288,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":2,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:13:50.355 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.350+0000 7fd5206b0640 10 librados: watch_flush enter 2024-09-16T04:13:50.355 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.350+0000 7fd5206b0640 10 librados: watch_flush exit 2024-09-16T04:13:50.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.350+0000 7fd5206b0640 20 client.1094168.objecter shutdown clearing up homeless session... 2024-09-16T04:13:50.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.350+0000 7fd5206b0640 10 client.1094168.objecter successfully canceled tick 2024-09-16T04:13:50.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.350+0000 7fd5206b0640 1 -- 172.21.15.42:0/1649646901 >> v2:172.21.15.154:6812/254845342 conn(0x7fd50403bab0 msgr2=0x7fd50403df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:50.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.350+0000 7fd5206b0640 1 --2- 172.21.15.42:0/1649646901 >> v2:172.21.15.154:6812/254845342 conn(0x7fd50403bab0 0x7fd50403df70 secure :-1 s=READY pgs=3999 cs=0 l=1 rev1=1 crypto rx=0x7fd518126330 tx=0x7fd514009040 comp rx=0 tx=0).stop 2024-09-16T04:13:50.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.350+0000 7fd5206b0640 10 monclient: shutdown 2024-09-16T04:13:50.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.350+0000 7fd5206b0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:13:50.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.350+0000 7fd5206b0640 1 -- 172.21.15.42:0/1649646901 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd504009ec0 msgr2=0x7fd5181cc3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:50.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.350+0000 7fd5206b0640 1 --2- 172.21.15.42:0/1649646901 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd504009ec0 0x7fd5181cc3a0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fd50801c9f0 tx=0x7fd5080192b0 comp rx=0 tx=0).stop 2024-09-16T04:13:50.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.350+0000 7fd5206b0640 1 -- 172.21.15.42:0/1649646901 shutdown_connections 2024-09-16T04:13:50.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.350+0000 7fd5206b0640 1 --2- 172.21.15.42:0/1649646901 >> v2:172.21.15.154:6812/254845342 conn(0x7fd50403bab0 0x7fd50403df70 unknown :-1 s=CLOSED pgs=3999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:13:50.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.350+0000 7fd5206b0640 1 --2- 172.21.15.42:0/1649646901 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd50400a8f0 0x7fd518124a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:13:50.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.350+0000 7fd5206b0640 1 --2- 172.21.15.42:0/1649646901 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd50400a3b0 0x7fd5181cc8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:13:50.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.350+0000 7fd5206b0640 1 --2- 172.21.15.42:0/1649646901 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd504009ec0 0x7fd5181cc3a0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:13:50.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.350+0000 7fd5206b0640 1 -- 172.21.15.42:0/1649646901 >> 172.21.15.42:0/1649646901 conn(0x7fd5180a2420 msgr2=0x7fd51806f890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:13:50.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.350+0000 7fd5206b0640 1 -- 172.21.15.42:0/1649646901 shutdown_connections 2024-09-16T04:13:50.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.350+0000 7fd5206b0640 1 -- 172.21.15.42:0/1649646901 wait complete. 2024-09-16T04:13:50.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.350+0000 7fd5206b0640 1 librados: shutdown 2024-09-16T04:13:50.372 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-16T04:13:50.372 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-16T04:13:50.372 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T04:13:50.489 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.486+0000 7f045a408640 10 monclient: get_monmap_and_config 2024-09-16T04:13:50.489 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.486+0000 7f045a408640 10 monclient: build_initial_monmap 2024-09-16T04:13:50.489 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.486+0000 7f045a408640 10 monclient: monmap: 2024-09-16T04:13:50.489 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:13:50.489 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:13:50.489 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:13:50.488169+0000 2024-09-16T04:13:50.490 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:13:50.488169+0000 2024-09-16T04:13:50.490 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:13:50.490 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:13:50.490 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:13:50.490 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:13:50.490 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:13:50.490 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:13:50.490 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:13:50.490 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:13:50.492 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.486+0000 7f045a408640 1 Processor -- start 2024-09-16T04:13:50.492 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.486+0000 7f045a408640 1 -- start start 2024-09-16T04:13:50.492 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.486+0000 7f045a408640 10 monclient: init 2024-09-16T04:13:50.492 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.486+0000 7f045a408640 10 monclient: _reopen_session rank -1 2024-09-16T04:13:50.492 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.486+0000 7f045a408640 10 monclient: _add_conns ranks=[0,1,2,4,3] 2024-09-16T04:13:50.492 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.486+0000 7f045a408640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f04540a1330 addr v1:172.21.15.42:6789/0 2024-09-16T04:13:50.492 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.486+0000 7f045a408640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f04540a1710 addr v1:172.21.15.154:6789/0 2024-09-16T04:13:50.492 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.486+0000 7f045a408640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f045413a3b0 0x7f045413c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:13:50.493 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.486+0000 7f045a408640 10 monclient(hunting): picked mon.noname-b con 0x7f045413a3b0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:13:50.493 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.486+0000 7f045a408640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f045409e4c0 con 0x7f04540a1330 2024-09-16T04:13:50.493 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.486+0000 7f045a408640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f04540a3170 con 0x7f04540a1710 2024-09-16T04:13:50.493 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.486+0000 7f045a408640 10 monclient(hunting): start opening mon connection 2024-09-16T04:13:50.493 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.486+0000 7f045a408640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f045409edf0 con 0x7f045413a3b0 2024-09-16T04:13:50.493 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.486+0000 7f045a408640 10 monclient(hunting): _renew_subs 2024-09-16T04:13:50.493 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.486+0000 7f045a408640 10 monclient(hunting): authenticate will time out at 2877.457626s 2024-09-16T04:13:50.494 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.486+0000 7f0453fff640 1 -- >> v1:172.21.15.42:6789/0 conn(0x7f04540a1330 legacy=0x7f045409f180 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.42:6789/0 2024-09-16T04:13:50.494 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.486+0000 7f04589a5640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f045413a3b0 0x7f045413c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:13:50.494 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.486+0000 7f04589a5640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f045413a3b0 0x7f045413c7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:60478/0 (socket says 172.21.15.42:60478) 2024-09-16T04:13:50.494 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.486+0000 7f04589a5640 1 -- 172.21.15.42:0/2277067164 learned_addr learned my addr 172.21.15.42:0/2277067164 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:13:50.494 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.486+0000 7f04589a5640 10 monclient(hunting): get_auth_request con 0x7f045413a3b0 auth_method 0 2024-09-16T04:13:50.494 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.486+0000 7f04589a5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:13:50.494 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.486+0000 7f04589a5640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:50.494 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.486+0000 7f04589a5640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:13:50.494 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.486+0000 7f04589a5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:13:50.494 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.486+0000 7f04589a5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:13:50.494 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.486+0000 7f04589a5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:13:50.494 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.486+0000 7f04589a5640 10 monclient(hunting): handle_auth_done global_id 1094171 payload 274 2024-09-16T04:13:50.494 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.486+0000 7f04589a5640 1 -- 172.21.15.42:0/2277067164 >> v1:172.21.15.154:6789/0 conn(0x7f04540a1710 legacy=0x7f04540a1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:13:50.495 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.486+0000 7f04589a5640 1 -- 172.21.15.42:0/2277067164 >> v1:172.21.15.42:6789/0 conn(0x7f04540a1330 legacy=0x7f045409f180 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:13:50.495 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.486+0000 7f04589a5640 10 monclient: _finish_hunting 0 2024-09-16T04:13:50.495 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.486+0000 7f04589a5640 1 monclient: found mon.noname-b 2024-09-16T04:13:50.495 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.486+0000 7f04589a5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:13:50.495 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.486+0000 7f04589a5640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T04:13:50.495 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.486+0000 7f04589a5640 1 -- 172.21.15.42:0/2277067164 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f045409d030 con 0x7f045413a3b0 2024-09-16T04:13:50.502 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.490+0000 7f04589a5640 1 --2- 172.21.15.42:0/2277067164 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f045413a3b0 0x7f045413c7a0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f0454073490 tx=0x7f044c01c8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=45f291ff58d8963 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:13:50.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.490+0000 7f0452ffd640 1 -- 172.21.15.42:0/2277067164 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f044c0a56a0 con 0x7f045413a3b0 2024-09-16T04:13:50.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.490+0000 7f0452ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:13:50.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.490+0000 7f0452ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:13:50.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.490+0000 7f0452ffd640 10 monclient: dump: 2024-09-16T04:13:50.503 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:13:50.503 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:13:50.503 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:50.503 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:50.503 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:50.503 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:13:50.503 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:13:50.503 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:13:50.504 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:13:50.504 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:13:50.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.490+0000 7f0452ffd640 10 monclient: _finish_auth 0 2024-09-16T04:13:50.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.490+0000 7f0452ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:13:50.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.490+0000 7f0452ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:13:50.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.490+0000 7f0452ffd640 1 -- 172.21.15.42:0/2277067164 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f044c0a5840 con 0x7f045413a3b0 2024-09-16T04:13:50.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.490+0000 7f045a408640 5 monclient: authenticate success, global_id 1094171 2024-09-16T04:13:50.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.490+0000 7f045a408640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:13:50.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.490+0000 7f0452ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:13:50.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.490+0000 7f045a408640 10 monclient: get_monmap_and_config success 2024-09-16T04:13:50.505 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.490+0000 7f0452ffd640 1 -- 172.21.15.42:0/2277067164 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f044c0a5c10 con 0x7f045413a3b0 2024-09-16T04:13:50.505 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.490+0000 7f0452ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:13:50.505 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.490+0000 7f045a408640 4 set_mon_vals no callback set 2024-09-16T04:13:50.505 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.490+0000 7f0452ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:13:50.505 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.490+0000 7f0452ffd640 10 monclient: dump: 2024-09-16T04:13:50.505 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:13:50.505 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:13:50.505 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:50.505 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:50.505 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:50.505 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:13:50.506 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:13:50.506 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:13:50.506 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:13:50.507 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:13:50.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.490+0000 7f045a408640 10 monclient: shutdown 2024-09-16T04:13:50.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.490+0000 7f045a408640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:13:50.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.490+0000 7f045a408640 1 -- 172.21.15.42:0/2277067164 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f045413a3b0 msgr2=0x7f045413c7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:50.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.490+0000 7f045a408640 1 --2- 172.21.15.42:0/2277067164 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f045413a3b0 0x7f045413c7a0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f0454073490 tx=0x7f044c01c8d0 comp rx=0 tx=0).stop 2024-09-16T04:13:50.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.490+0000 7f045a408640 1 -- 172.21.15.42:0/2277067164 shutdown_connections 2024-09-16T04:13:50.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.490+0000 7f045a408640 1 --2- 172.21.15.42:0/2277067164 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f045413a3b0 0x7f045413c7a0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:13:50.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.490+0000 7f045a408640 1 -- 172.21.15.42:0/2277067164 >> 172.21.15.42:0/2277067164 conn(0x7f04540a2420 msgr2=0x7f04540a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:13:50.508 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.490+0000 7f045a408640 1 -- 172.21.15.42:0/2277067164 shutdown_connections 2024-09-16T04:13:50.508 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.490+0000 7f045a408640 1 -- 172.21.15.42:0/2277067164 wait complete. 2024-09-16T04:13:50.508 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.490+0000 7f045a408640 10 monclient: build_initial_monmap 2024-09-16T04:13:50.508 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.490+0000 7f045a408640 10 monclient: monmap: 2024-09-16T04:13:50.508 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:13:50.508 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:13:50.508 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:13:50.508 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:13:50.508 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:13:50.508 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:13:50.508 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:13:50.508 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:13:50.509 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:13:50.509 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:13:50.509 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.490+0000 7f045a408640 1 librados: starting msgr at 2024-09-16T04:13:50.509 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.490+0000 7f045a408640 1 librados: starting objecter 2024-09-16T04:13:50.509 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.490+0000 7f045a408640 1 Processor -- start 2024-09-16T04:13:50.509 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.490+0000 7f045a408640 1 -- start start 2024-09-16T04:13:50.509 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.490+0000 7f045a408640 1 librados: setting wanted keys 2024-09-16T04:13:50.509 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.490+0000 7f045a408640 1 librados: calling monclient init 2024-09-16T04:13:50.509 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.490+0000 7f045a408640 10 monclient: init 2024-09-16T04:13:50.509 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.490+0000 7f045a408640 10 monclient: _reopen_session rank -1 2024-09-16T04:13:50.509 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.490+0000 7f045a408640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:13:50.509 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.490+0000 7f045a408640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f04540a1330 0x7f0454071da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:13:50.509 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.490+0000 7f045a408640 10 monclient(hunting): picked mon.noname-c con 0x7f04540a1330 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:13:50.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.490+0000 7f045a408640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f04540a1710 0x7f04541cbfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:13:50.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.490+0000 7f045a408640 10 monclient(hunting): picked mon.noname-a con 0x7f04540a1710 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:13:50.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.490+0000 7f045a408640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f045413a3b0 0x7f04541cc520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:13:50.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.490+0000 7f045a408640 10 monclient(hunting): picked mon.noname-b con 0x7f045413a3b0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:13:50.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.490+0000 7f045a408640 10 monclient(hunting): start opening mon connection 2024-09-16T04:13:50.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.490+0000 7f045a408640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f045409bea0 con 0x7f04540a1710 2024-09-16T04:13:50.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.490+0000 7f045a408640 10 monclient(hunting): start opening mon connection 2024-09-16T04:13:50.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.490+0000 7f045a408640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f045409bd20 con 0x7f045413a3b0 2024-09-16T04:13:50.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.490+0000 7f045a408640 10 monclient(hunting): start opening mon connection 2024-09-16T04:13:50.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.490+0000 7f045a408640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f045409c020 con 0x7f04540a1330 2024-09-16T04:13:50.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.490+0000 7f045a408640 10 monclient(hunting): _renew_subs 2024-09-16T04:13:50.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.490+0000 7f045a408640 10 monclient(hunting): authenticate will time out at 2877.463048s 2024-09-16T04:13:50.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.490+0000 7f04537fe640 1 -- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f04540a1710 msgr2=0x7f04541cbfe0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.42:3300/0 2024-09-16T04:13:50.510 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.490+0000 7f04537fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f04540a1710 0x7f04541cbfe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:13:50.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.490+0000 7f0453fff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f04540a1330 0x7f0454071da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:13:50.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.490+0000 7f0453fff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f04540a1330 0x7f0454071da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:60486/0 (socket says 172.21.15.42:60486) 2024-09-16T04:13:50.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.490+0000 7f0453fff640 1 -- 172.21.15.42:0/1722908999 learned_addr learned my addr 172.21.15.42:0/1722908999 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:13:50.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.490+0000 7f0453fff640 10 monclient(hunting): get_auth_request con 0x7f04540a1330 auth_method 0 2024-09-16T04:13:50.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.490+0000 7f0453fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:13:50.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.490+0000 7f0453fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:50.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.490+0000 7f0453fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:13:50.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.490+0000 7f0453fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:13:50.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.490+0000 7f0453fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:13:50.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.490+0000 7f0453fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:13:50.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.490+0000 7f04589a5640 1 --2- 172.21.15.42:0/1722908999 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f045413a3b0 0x7f04541cc520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:13:50.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.490+0000 7f04589a5640 10 monclient(hunting): get_auth_request con 0x7f045413a3b0 auth_method 0 2024-09-16T04:13:50.511 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.490+0000 7f04589a5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:13:50.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.490+0000 7f04589a5640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:50.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.490+0000 7f04589a5640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:13:50.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.490+0000 7f0453fff640 10 monclient(hunting): handle_auth_done global_id 1094174 payload 824 2024-09-16T04:13:50.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.490+0000 7f0453fff640 1 -- 172.21.15.42:0/1722908999 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f045413a3b0 msgr2=0x7f04541cc520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:50.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.490+0000 7f0453fff640 1 --2- 172.21.15.42:0/1722908999 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f045413a3b0 0x7f04541cc520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:13:50.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.490+0000 7f0453fff640 1 -- 172.21.15.42:0/1722908999 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f04540a1710 msgr2=0x7f04541cbfe0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:13:50.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.490+0000 7f0453fff640 1 --2- 172.21.15.42:0/1722908999 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f04540a1710 0x7f04541cbfe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:13:50.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.490+0000 7f0453fff640 10 monclient: _finish_hunting 0 2024-09-16T04:13:50.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.490+0000 7f0453fff640 1 monclient: found mon.noname-c 2024-09-16T04:13:50.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.490+0000 7f0453fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:13:50.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.490+0000 7f0453fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T04:13:50.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.490+0000 7f0453fff640 1 -- 172.21.15.42:0/1722908999 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0454073a30 con 0x7f04540a1330 2024-09-16T04:13:50.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.490+0000 7f0453fff640 1 --2- 172.21.15.42:0/1722908999 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f04540a1330 0x7f0454071da0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f044400b560 tx=0x7f044400ba20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:13:50.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.490+0000 7f0450ff9640 10 client.?.objecter ms_handle_connect 0x7f04540a1330 2024-09-16T04:13:50.513 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.490+0000 7f0450ff9640 10 client.?.objecter resend_mon_ops 2024-09-16T04:13:50.513 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.490+0000 7f0450ff9640 1 -- 172.21.15.42:0/1722908999 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0444013020 con 0x7f04540a1330 2024-09-16T04:13:50.513 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.490+0000 7f0450ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:13:50.513 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.490+0000 7f0450ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:13:50.513 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.490+0000 7f0450ff9640 10 monclient: dump: 2024-09-16T04:13:50.513 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:13:50.513 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:13:50.513 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:50.513 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:50.513 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:50.513 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:13:50.513 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:13:50.513 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:13:50.513 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:13:50.514 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:13:50.514 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.490+0000 7f0450ff9640 10 monclient: _finish_auth 0 2024-09-16T04:13:50.514 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.490+0000 7f0450ff9640 10 monclient: _check_auth_tickets 2024-09-16T04:13:50.514 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.490+0000 7f0450ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:13:50.514 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.490+0000 7f0450ff9640 1 -- 172.21.15.42:0/1722908999 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0444003df0 con 0x7f04540a1330 2024-09-16T04:13:50.514 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.490+0000 7f045a408640 5 monclient: authenticate success, global_id 1094174 2024-09-16T04:13:50.514 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.490+0000 7f045a408640 10 monclient: _renew_subs 2024-09-16T04:13:50.514 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.490+0000 7f045a408640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:13:50.514 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.490+0000 7f045a408640 1 -- 172.21.15.42:0/1722908999 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0454132280 con 0x7f04540a1330 2024-09-16T04:13:50.514 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.490+0000 7f0450ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T04:13:50.514 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.494+0000 7f0450ff9640 1 -- 172.21.15.42:0/1722908999 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04440052b0 con 0x7f04540a1330 2024-09-16T04:13:50.514 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.494+0000 7f0450ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:13:50.514 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.494+0000 7f0450ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:13:50.515 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.494+0000 7f0450ff9640 10 monclient: dump: 2024-09-16T04:13:50.515 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:13:50.515 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:13:50.515 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:50.515 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:50.515 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:50.515 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:13:50.516 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:13:50.516 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:13:50.516 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:13:50.516 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:13:50.516 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.494+0000 7f045a408640 10 client.1094174.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:13:50.516 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.494+0000 7f04527fc640 4 set_mon_vals no callback set 2024-09-16T04:13:50.516 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.494+0000 7f045a408640 10 monclient: _renew_subs 2024-09-16T04:13:50.516 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.494+0000 7f045a408640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:13:50.516 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.494+0000 7f045a408640 1 -- 172.21.15.42:0/1722908999 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f0454133270 con 0x7f04540a1330 2024-09-16T04:13:50.516 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.494+0000 7f045a408640 1 librados: init done 2024-09-16T04:13:50.516 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.494+0000 7f045a408640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:13:50.516 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.494+0000 7f045a408640 1 --2- 172.21.15.42:0/1722908999 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0454141760 0x7f045413fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:13:50.516 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.494+0000 7f045a408640 10 monclient(hunting): start opening mon connection 2024-09-16T04:13:50.516 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.494+0000 7f045a408640 1 -- 172.21.15.42:0/1722908999 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f04540a4830 con 0x7f0454141760 2024-09-16T04:13:50.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.494+0000 7f04537fe640 1 --2- 172.21.15.42:0/1722908999 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0454141760 0x7f045413fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:13:50.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.494+0000 7f0450ff9640 1 -- 172.21.15.42:0/1722908999 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f0444016020 con 0x7f04540a1330 2024-09-16T04:13:50.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.494+0000 7f04537fe640 10 monclient: get_auth_request con 0x7f0454141760 auth_method 0 2024-09-16T04:13:50.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.494+0000 7f04537fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:13:50.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.494+0000 7f04537fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:50.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.494+0000 7f04537fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:13:50.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.494+0000 7f04537fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:13:50.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.494+0000 7f04537fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:13:50.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.494+0000 7f04537fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:13:50.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.494+0000 7f0450ff9640 1 --2- 172.21.15.42:0/1722908999 >> v2:172.21.15.154:6812/254845342 conn(0x7f04400377c0 0x7f0440039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:13:50.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.494+0000 7f04537fe640 10 monclient(hunting): handle_auth_done global_id 1094177 payload 824 2024-09-16T04:13:50.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.494+0000 7f04537fe640 1 -- 172.21.15.42:0/1722908999 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f045409f180 con 0x7f0454141760 2024-09-16T04:13:50.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.494+0000 7f0450ff9640 1 -- 172.21.15.42:0/1722908999 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(709..709 src has 681..709) ==== 5048+0+0 (secure 0 0 0) 0x7f0444048b70 con 0x7f04540a1330 2024-09-16T04:13:50.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.494+0000 7f0450ff9640 10 client.1094174.objecter ms_dispatch 0x7f0454000ba0 osd_map(709..709 src has 681..709) 2024-09-16T04:13:50.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.494+0000 7f0450ff9640 3 client.1094174.objecter handle_osd_map got epochs [709,709] > 0 2024-09-16T04:13:50.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.494+0000 7f0450ff9640 3 client.1094174.objecter handle_osd_map decoding full epoch 709 2024-09-16T04:13:50.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.494+0000 7f0450ff9640 20 client.1094174.objecter dump_active .. 0 homeless 2024-09-16T04:13:50.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.494+0000 7f0450ff9640 10 client.1094174.objecter ms_handle_connect 0x7f0454141760 2024-09-16T04:13:50.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.494+0000 7f04537fe640 1 --2- 172.21.15.42:0/1722908999 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0454141760 0x7f045413fb20 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f0448000c10 tx=0x7f044800c830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:13:50.518 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.494+0000 7f0450ff9640 10 client.1094174.objecter resend_mon_ops 2024-09-16T04:13:50.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.494+0000 7f04589a5640 1 --2- 172.21.15.42:0/1722908999 >> v2:172.21.15.154:6812/254845342 conn(0x7f04400377c0 0x7f0440039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:13:50.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.494+0000 7f0450ff9640 1 -- 172.21.15.42:0/1722908999 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04480074e0 con 0x7f0454141760 2024-09-16T04:13:50.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.494+0000 7f0450ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:13:50.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.494+0000 7f0450ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:13:50.519 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.494+0000 7f0450ff9640 10 monclient: dump: 2024-09-16T04:13:50.519 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:13:50.519 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:13:50.519 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:50.519 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:50.519 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:50.519 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:13:50.519 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:13:50.519 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:13:50.520 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:13:50.520 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:13:50.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.494+0000 7f04589a5640 10 monclient: get_auth_request con 0x7f04400377c0 auth_method 0 2024-09-16T04:13:50.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.494+0000 7f0450ff9640 1 -- 172.21.15.42:0/1722908999 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f04480074e0 con 0x7f0454141760 2024-09-16T04:13:50.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.494+0000 7f0450ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:13:50.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.494+0000 7f0450ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:13:50.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.494+0000 7f0450ff9640 1 -- 172.21.15.42:0/1722908999 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0454141760 msgr2=0x7f045413fb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:50.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.494+0000 7f0450ff9640 1 --2- 172.21.15.42:0/1722908999 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0454141760 0x7f045413fb20 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f0448000c10 tx=0x7f044800c830 comp rx=0 tx=0).stop 2024-09-16T04:13:50.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.494+0000 7f0450ff9640 1 -- 172.21.15.42:0/1722908999 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0454141760 msgr2=0x7f045413fb20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:13:50.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.494+0000 7f0450ff9640 1 --2- 172.21.15.42:0/1722908999 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0454141760 0x7f045413fb20 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:13:50.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.494+0000 7f0450ff9640 10 client.1094174.objecter ms_handle_connect 0x7f04400377c0 2024-09-16T04:13:50.520 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.494+0000 7f04589a5640 1 --2- 172.21.15.42:0/1722908999 >> v2:172.21.15.154:6812/254845342 conn(0x7f04400377c0 0x7f0440039c80 secure :-1 s=READY pgs=4000 cs=0 l=1 rev1=1 crypto rx=0x7f044c0ae000 tx=0x7f044c0a7070 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:13:50.522 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.498+0000 7f045a408640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:13:50.522 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.498+0000 7f045a408640 1 --2- 172.21.15.42:0/1722908999 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0454140060 0x7f0454140470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:13:50.522 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.498+0000 7f045a408640 10 monclient(hunting): start opening mon connection 2024-09-16T04:13:50.522 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.498+0000 7f045a408640 1 -- 172.21.15.42:0/1722908999 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f04541409b0 con 0x7f0454140060 2024-09-16T04:13:50.522 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.498+0000 7f04537fe640 1 --2- 172.21.15.42:0/1722908999 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0454140060 0x7f0454140470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:13:50.522 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.502+0000 7f04537fe640 10 monclient: get_auth_request con 0x7f0454140060 auth_method 0 2024-09-16T04:13:50.522 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.502+0000 7f04537fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:13:50.522 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.502+0000 7f04537fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:50.522 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.502+0000 7f04537fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:13:50.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.502+0000 7f04537fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:13:50.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.502+0000 7f04537fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:13:50.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.502+0000 7f04537fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:13:50.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.502+0000 7f04537fe640 10 monclient(hunting): handle_auth_done global_id 1094180 payload 824 2024-09-16T04:13:50.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.502+0000 7f04537fe640 1 -- 172.21.15.42:0/1722908999 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f0454069040 con 0x7f0454140060 2024-09-16T04:13:50.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.502+0000 7f04537fe640 1 --2- 172.21.15.42:0/1722908999 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0454140060 0x7f0454140470 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f045409ff60 tx=0x7f0448000ee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:13:50.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.502+0000 7f0450ff9640 10 client.1094174.objecter ms_handle_connect 0x7f0454140060 2024-09-16T04:13:50.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.502+0000 7f0450ff9640 10 client.1094174.objecter resend_mon_ops 2024-09-16T04:13:50.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.502+0000 7f0450ff9640 1 -- 172.21.15.42:0/1722908999 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0448022410 con 0x7f0454140060 2024-09-16T04:13:50.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.502+0000 7f0450ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:13:50.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.502+0000 7f0450ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:13:50.523 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.502+0000 7f0450ff9640 10 monclient: dump: 2024-09-16T04:13:50.523 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:13:50.523 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:13:50.524 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:50.524 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:50.524 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:50.524 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:13:50.524 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:13:50.524 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:13:50.524 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:13:50.524 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:13:50.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.502+0000 7f0450ff9640 1 -- 172.21.15.42:0/1722908999 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4701 (secure 0 0 0) 0x7f0448022410 con 0x7f0454140060 2024-09-16T04:13:50.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.502+0000 7f0450ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:13:50.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.502+0000 7f0450ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:13:50.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.502+0000 7f0450ff9640 1 -- 172.21.15.42:0/1722908999 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0454140060 msgr2=0x7f0454140470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:50.524 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.502+0000 7f0450ff9640 1 --2- 172.21.15.42:0/1722908999 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0454140060 0x7f0454140470 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f045409ff60 tx=0x7f0448000ee0 comp rx=0 tx=0).stop 2024-09-16T04:13:50.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.502+0000 7f0450ff9640 1 -- 172.21.15.42:0/1722908999 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0454140060 msgr2=0x7f0454140470 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:13:50.525 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.502+0000 7f0450ff9640 1 --2- 172.21.15.42:0/1722908999 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0454140060 0x7f0454140470 secure :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f045409ff60 tx=0x7f0448000ee0 comp rx=0 tx=0).stop 2024-09-16T04:13:50.525 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T04:13:50.525 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:13:50.526 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:13:50.526 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T04:13:50.526 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 288, 2024-09-16T04:13:50.526 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 29543, 2024-09-16T04:13:50.526 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T04:13:50.526 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T04:13:50.526 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T04:13:50.526 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:13:50.526 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 2, 2024-09-16T04:13:50.526 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:13:50.526 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T04:13:50.526 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T04:13:50.526 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:13:50.526 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:13:50.527 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:13:50.527 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:13:50.527 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:13:50.527 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:13:50.527 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:13:50.527 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:13:50.527 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:13:50.527 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:13:50.527 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:13:50.527 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:13:50.527 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:13:50.528 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T04:13:50.528 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:13:50.528 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:13:50.528 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:13:50.528 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:13:50.528 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:13:50.528 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:13:50.528 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:13:50.528 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:13:50.528 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:13:50.529 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:13:50.529 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:13:50.529 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:13:50.529 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:13:50.529 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T04:13:50.529 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T04:13:50.529 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T04:13:50.529 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T04:13:50.529 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T04:13:50.529 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T04:13:50.529 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:13:50.529 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:13:50.529 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T04:13:50.530 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T04:13:50.530 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T04:13:50.530 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T04:13:50.530 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T04:13:50.530 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T04:13:50.530 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T04:13:50.530 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:13:50.530 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T04:13:50.530 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:13:50.530 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:13:50.530 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:13:50.530 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:13:50.530 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:13:50.530 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:13:50.531 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:13:50.531 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:13:50.531 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:13:50.531 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:13:50.531 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:13:50.531 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:13:50.531 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T04:13:50.531 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:13:50.531 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T04:13:50.531 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:13:50.531 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:13:50.531 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:13:50.531 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:13:50.531 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:13:50.532 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:13:50.532 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:13:50.532 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T04:13:50.532 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:13:50.532 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:13:50.532 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:13:50.532 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:13:50.532 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T04:13:50.532 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:13:50.532 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:13:50.532 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:13:50.532 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:13:50.532 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T04:13:50.533 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T04:13:50.533 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:13:50.533 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:13:50.533 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:13:50.533 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:13:50.533 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:13:50.533 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:13:50.533 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:13:50.533 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:13:50.533 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:13:50.533 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:13:50.533 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:13:50.533 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T04:13:50.533 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:13:50.534 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:13:50.534 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:13:50.534 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:13:50.534 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T04:13:50.534 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:13:50.534 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:13:50.534 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:13:50.534 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:13:50.534 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T04:13:50.534 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T04:13:50.534 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:13:50.534 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:13:50.534 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:13:50.534 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:13:50.535 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:13:50.535 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:13:50.535 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:13:50.535 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:13:50.535 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:13:50.535 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:13:50.535 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:13:50.535 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T04:13:50.535 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:13:50.536 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:13:50.536 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:13:50.536 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:13:50.536 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T04:13:50.536 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:13:50.536 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:13:50.536 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:13:50.536 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:13:50.536 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T04:13:50.536 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T04:13:50.536 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:13:50.537 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:13:50.537 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:13:50.537 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:13:50.537 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:13:50.537 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:13:50.537 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T04:13:50.537 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T04:13:50.537 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:13:50.537 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:13:50.537 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:13:50.537 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:13:50.538 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:13:50.538 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:13:50.538 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T04:13:50.538 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:13:50.538 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:13:50.538 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:13:50.538 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 5 2024-09-16T04:13:50.538 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:13:50.538 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:13:50.539 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T04:13:50.539 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:13:50.539 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:13:50.539 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:13:50.539 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 4 2024-09-16T04:13:50.539 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:13:50.539 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:13:50.539 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:13:50.539 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T04:13:50.539 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T04:13:50.541 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.502+0000 7f045a408640 10 librados: watch_flush enter 2024-09-16T04:13:50.542 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.502+0000 7f045a408640 10 librados: watch_flush exit 2024-09-16T04:13:50.542 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.502+0000 7f045a408640 20 client.1094174.objecter shutdown clearing up homeless session... 2024-09-16T04:13:50.542 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.502+0000 7f045a408640 10 client.1094174.objecter successfully canceled tick 2024-09-16T04:13:50.542 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.502+0000 7f045a408640 1 -- 172.21.15.42:0/1722908999 >> v2:172.21.15.154:6812/254845342 conn(0x7f04400377c0 msgr2=0x7f0440039c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:50.542 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.502+0000 7f045a408640 1 --2- 172.21.15.42:0/1722908999 >> v2:172.21.15.154:6812/254845342 conn(0x7f04400377c0 0x7f0440039c80 secure :-1 s=READY pgs=4000 cs=0 l=1 rev1=1 crypto rx=0x7f044c0ae000 tx=0x7f044c0a7070 comp rx=0 tx=0).stop 2024-09-16T04:13:50.543 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.502+0000 7f045a408640 10 monclient: shutdown 2024-09-16T04:13:50.543 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.502+0000 7f045a408640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:13:50.543 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.502+0000 7f045a408640 1 -- 172.21.15.42:0/1722908999 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f04540a1330 msgr2=0x7f0454071da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:50.543 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.502+0000 7f045a408640 1 --2- 172.21.15.42:0/1722908999 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f04540a1330 0x7f0454071da0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f044400b560 tx=0x7f044400ba20 comp rx=0 tx=0).stop 2024-09-16T04:13:50.543 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.502+0000 7f04589a5640 1 -- 172.21.15.42:0/1722908999 reap_dead start 2024-09-16T04:13:50.543 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.502+0000 7f04589a5640 1 -- 172.21.15.42:0/1722908999 reap_dead start 2024-09-16T04:13:50.543 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.502+0000 7f045a408640 1 -- 172.21.15.42:0/1722908999 shutdown_connections 2024-09-16T04:13:50.543 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.502+0000 7f045a408640 1 -- 172.21.15.42:0/1722908999 >> 172.21.15.42:0/1722908999 conn(0x7f04540a2420 msgr2=0x7f04540a3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:13:50.544 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.502+0000 7f045a408640 1 -- 172.21.15.42:0/1722908999 shutdown_connections 2024-09-16T04:13:50.544 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.502+0000 7f045a408640 1 -- 172.21.15.42:0/1722908999 wait complete. 2024-09-16T04:13:50.544 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.502+0000 7f045a408640 1 librados: shutdown 2024-09-16T04:13:50.544 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T04:13:50.544 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-16T04:13:50.544 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T04:13:50.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.662+0000 7f2989953640 10 monclient: get_monmap_and_config 2024-09-16T04:13:50.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.662+0000 7f2989953640 10 monclient: build_initial_monmap 2024-09-16T04:13:50.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.662+0000 7f2989953640 10 monclient: monmap: 2024-09-16T04:13:50.667 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:13:50.667 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:13:50.667 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:13:50.665680+0000 2024-09-16T04:13:50.667 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:13:50.665680+0000 2024-09-16T04:13:50.667 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:13:50.668 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:13:50.668 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:13:50.668 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:13:50.668 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:13:50.668 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:13:50.668 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:13:50.668 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:13:50.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.662+0000 7f2989953640 1 Processor -- start 2024-09-16T04:13:50.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.662+0000 7f2989953640 1 -- start start 2024-09-16T04:13:50.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.662+0000 7f2989953640 10 monclient: init 2024-09-16T04:13:50.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.662+0000 7f2989953640 10 monclient: _reopen_session rank -1 2024-09-16T04:13:50.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.662+0000 7f2989953640 10 monclient: _add_conns ranks=[0,3,1,2,4] 2024-09-16T04:13:50.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.662+0000 7f2989953640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f298406db90 addr v1:172.21.15.42:6789/0 2024-09-16T04:13:50.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.662+0000 7f2989953640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f29841340c0 0x7f298413c190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:13:50.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.662+0000 7f2989953640 10 monclient(hunting): picked mon.noname-a con 0x7f29841340c0 addr v2:172.21.15.42:3300/0 2024-09-16T04:13:50.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.662+0000 7f2989953640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f298413c6d0 addr v1:172.21.15.154:6789/0 2024-09-16T04:13:50.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.662+0000 7f2989953640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f298406ece0 con 0x7f298406db90 2024-09-16T04:13:50.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.662+0000 7f2989953640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f298409bf80 con 0x7f298413c6d0 2024-09-16T04:13:50.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.662+0000 7f2989953640 10 monclient(hunting): start opening mon connection 2024-09-16T04:13:50.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.662+0000 7f2989953640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f298406df70 con 0x7f29841340c0 2024-09-16T04:13:50.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.662+0000 7f2989953640 10 monclient(hunting): _renew_subs 2024-09-16T04:13:50.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.662+0000 7f2989953640 10 monclient(hunting): authenticate will time out at 2877.634995s 2024-09-16T04:13:50.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.662+0000 7f2982ffd640 1 -- >> v1:172.21.15.42:6789/0 conn(0x7f298406db90 legacy=0x7f29840a37f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.42:6789/0 2024-09-16T04:13:50.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.662+0000 7f29827fc640 1 -- >> v2:172.21.15.42:3300/0 conn(0x7f29841340c0 msgr2=0x7f298413c190 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3300/0 2024-09-16T04:13:50.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.662+0000 7f29827fc640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f29841340c0 0x7f298413c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:13:50.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.662+0000 7f29837fe640 1 --1- >> v1:172.21.15.154:6789/0 conn(0x7f298413c6d0 0x7f298413eac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.154:6789/0 says I am v1:172.21.15.42:52722/0 (socket says 172.21.15.42:52722) 2024-09-16T04:13:50.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.662+0000 7f29837fe640 1 -- 172.21.15.42:0/644559028 learned_addr learned my addr 172.21.15.42:0/644559028 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T04:13:50.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.662+0000 7f2981ffb640 1 -- 172.21.15.42:0/644559028 <== mon.1 v1:172.21.15.154:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2757594202 0 0) 0x7f297c0014a0 con 0x7f298413c6d0 2024-09-16T04:13:50.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.662+0000 7f2981ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:13:50.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.662+0000 7f2981ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:50.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.662+0000 7f2981ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:13:50.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.662+0000 7f2981ffb640 10 monclient(hunting): my global_id is 1074337 2024-09-16T04:13:50.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.662+0000 7f2981ffb640 1 -- 172.21.15.42:0/644559028 --> v1:172.21.15.154:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f296c003530 con 0x7f298413c6d0 2024-09-16T04:13:50.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.666+0000 7f2981ffb640 1 -- 172.21.15.42:0/644559028 <== mon.1 v1:172.21.15.154:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4125908373 0 0) 0x7f296c003530 con 0x7f298413c6d0 2024-09-16T04:13:50.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.666+0000 7f2981ffb640 1 -- 172.21.15.42:0/644559028 >> v2:172.21.15.42:3300/0 conn(0x7f29841340c0 msgr2=0x7f298413c190 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:13:50.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.666+0000 7f2981ffb640 1 --2- 172.21.15.42:0/644559028 >> v2:172.21.15.42:3300/0 conn(0x7f29841340c0 0x7f298413c190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:13:50.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.666+0000 7f2981ffb640 1 -- 172.21.15.42:0/644559028 >> v1:172.21.15.42:6789/0 conn(0x7f298406db90 legacy=0x7f29840a37f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:13:50.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.666+0000 7f2981ffb640 10 monclient: _finish_hunting 0 2024-09-16T04:13:50.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.666+0000 7f2981ffb640 1 monclient: found mon.noname-c-legacy 2024-09-16T04:13:50.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.666+0000 7f2981ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:13:50.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.666+0000 7f2981ffb640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.154:6789/0 2024-09-16T04:13:50.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.666+0000 7f2981ffb640 1 -- 172.21.15.42:0/644559028 --> v1:172.21.15.154:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29841344d0 con 0x7f298413c6d0 2024-09-16T04:13:50.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.666+0000 7f2981ffb640 10 monclient: _finish_auth 0 2024-09-16T04:13:50.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.666+0000 7f2981ffb640 10 monclient: _check_auth_tickets 2024-09-16T04:13:50.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.666+0000 7f2981ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:13:50.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.666+0000 7f2981ffb640 1 -- 172.21.15.42:0/644559028 <== mon.1 v1:172.21.15.154:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2177413734 0 0) 0x7f297c001bd0 con 0x7f298413c6d0 2024-09-16T04:13:50.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.666+0000 7f2989953640 5 monclient: authenticate success, global_id 1074337 2024-09-16T04:13:50.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.666+0000 7f2989953640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:13:50.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.666+0000 7f2981ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:13:50.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.666+0000 7f2981ffb640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-16T04:13:50.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.666+0000 7f2981ffb640 10 monclient: dump: 2024-09-16T04:13:50.688 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:13:50.688 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:13:50.688 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:50.689 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:50.689 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:50.690 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:13:50.690 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:13:50.690 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:13:50.690 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:13:50.690 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:13:50.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.666+0000 7f2981ffb640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] but i'm connected to v1:172.21.15.154:6789/0, reconnecting 2024-09-16T04:13:50.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.666+0000 7f2981ffb640 10 monclient: _reopen_session rank -1 2024-09-16T04:13:50.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.666+0000 7f2981ffb640 1 -- 172.21.15.42:0/644559028 >> v1:172.21.15.154:6789/0 conn(0x7f298413c6d0 legacy=0x7f298413eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:50.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.666+0000 7f2981ffb640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:13:50.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.666+0000 7f2981ffb640 1 --2- 172.21.15.42:0/644559028 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f296c0039c0 0x7f296c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:13:50.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.666+0000 7f2981ffb640 10 monclient(hunting): picked mon.c con 0x7f296c0039c0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:13:50.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.666+0000 7f2981ffb640 1 --2- 172.21.15.42:0/644559028 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f296c004810 0x7f296c004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:13:50.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.666+0000 7f2981ffb640 10 monclient(hunting): picked mon.a con 0x7f296c004810 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:13:50.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.666+0000 7f2981ffb640 1 --2- 172.21.15.42:0/644559028 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f296c0053d0 0x7f296c00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:13:50.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.666+0000 7f2981ffb640 10 monclient(hunting): picked mon.b con 0x7f296c0053d0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:13:50.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.666+0000 7f2981ffb640 10 monclient(hunting): start opening mon connection 2024-09-16T04:13:50.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.666+0000 7f2981ffb640 1 -- 172.21.15.42:0/644559028 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f298406df70 con 0x7f296c004810 2024-09-16T04:13:50.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.666+0000 7f2981ffb640 10 monclient(hunting): start opening mon connection 2024-09-16T04:13:50.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.666+0000 7f2981ffb640 1 -- 172.21.15.42:0/644559028 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f296c00c270 con 0x7f296c0053d0 2024-09-16T04:13:50.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.666+0000 7f2981ffb640 10 monclient(hunting): start opening mon connection 2024-09-16T04:13:50.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.666+0000 7f2981ffb640 1 -- 172.21.15.42:0/644559028 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f296c00c470 con 0x7f296c0039c0 2024-09-16T04:13:50.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.666+0000 7f2981ffb640 10 monclient(hunting): _renew_subs 2024-09-16T04:13:50.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.666+0000 7f2981ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:13:50.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.666+0000 7f2989953640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-16T04:13:50.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.666+0000 7f29827fc640 1 -- 172.21.15.42:0/644559028 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f296c004810 msgr2=0x7f296c004c90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.42:3300/0 2024-09-16T04:13:50.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.666+0000 7f29827fc640 1 --2- 172.21.15.42:0/644559028 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f296c004810 0x7f296c004c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:13:50.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.666+0000 7f2982ffd640 1 --2- 172.21.15.42:0/644559028 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f296c0039c0 0x7f296c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:13:50.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.666+0000 7f2982ffd640 10 monclient(hunting): get_auth_request con 0x7f296c0039c0 auth_method 0 2024-09-16T04:13:50.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.666+0000 7f2982ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:13:50.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.666+0000 7f2982ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:50.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.666+0000 7f2982ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:13:50.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.666+0000 7f2982ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:13:50.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.666+0000 7f2982ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:13:50.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.666+0000 7f2982ffd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:13:50.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.666+0000 7f29837fe640 1 --2- 172.21.15.42:0/644559028 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f296c0053d0 0x7f296c00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:13:50.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.666+0000 7f29837fe640 10 monclient(hunting): get_auth_request con 0x7f296c0053d0 auth_method 0 2024-09-16T04:13:50.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.666+0000 7f29837fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:13:50.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.666+0000 7f29837fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:50.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.666+0000 7f29837fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:13:50.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.666+0000 7f2982ffd640 10 monclient(hunting): handle_auth_done global_id 1074337 payload 293 2024-09-16T04:13:50.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.666+0000 7f2982ffd640 1 -- 172.21.15.42:0/644559028 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f296c0053d0 msgr2=0x7f296c00b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:13:50.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.666+0000 7f2982ffd640 1 --2- 172.21.15.42:0/644559028 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f296c0053d0 0x7f296c00b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:13:50.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.666+0000 7f2982ffd640 1 -- 172.21.15.42:0/644559028 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f296c004810 msgr2=0x7f296c004c90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:13:50.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.666+0000 7f2982ffd640 1 --2- 172.21.15.42:0/644559028 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f296c004810 0x7f296c004c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:13:50.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.666+0000 7f2982ffd640 10 monclient: _finish_hunting 0 2024-09-16T04:13:50.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.666+0000 7f2982ffd640 1 monclient: found mon.c 2024-09-16T04:13:50.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.666+0000 7f2982ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:13:50.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.666+0000 7f2982ffd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:13:50.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.666+0000 7f2982ffd640 1 -- 172.21.15.42:0/644559028 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f296c003530 con 0x7f296c0039c0 2024-09-16T04:13:50.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.666+0000 7f2982ffd640 10 monclient: _finish_auth 0 2024-09-16T04:13:50.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.666+0000 7f2982ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:13:50.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.666+0000 7f2982ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:13:50.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.666+0000 7f29837fe640 1 -- 172.21.15.42:0/644559028 reap_dead start 2024-09-16T04:13:50.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.666+0000 7f2982ffd640 1 --2- 172.21.15.42:0/644559028 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f296c0039c0 0x7f296c003dc0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f2974003990 tx=0x7f297401c420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cc41cee637e8e27d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:13:50.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.666+0000 7f2981ffb640 1 -- 172.21.15.42:0/644559028 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2974009520 con 0x7f296c0039c0 2024-09-16T04:13:50.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.666+0000 7f2981ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:13:50.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.666+0000 7f2981ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:13:50.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.666+0000 7f2981ffb640 10 monclient: dump: 2024-09-16T04:13:50.695 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:13:50.695 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:13:50.695 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:50.695 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:50.695 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:50.695 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:13:50.695 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:13:50.695 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:13:50.695 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:13:50.696 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:13:50.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.666+0000 7f2981ffb640 1 -- 172.21.15.42:0/644559028 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2974003bc0 con 0x7f296c0039c0 2024-09-16T04:13:50.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.666+0000 7f2989953640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:13:50.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.666+0000 7f2981ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:13:50.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.666+0000 7f2989953640 10 monclient: get_monmap_and_config success 2024-09-16T04:13:50.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.666+0000 7f2989953640 4 set_mon_vals no callback set 2024-09-16T04:13:50.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.666+0000 7f2981ffb640 1 -- 172.21.15.42:0/644559028 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2974005070 con 0x7f296c0039c0 2024-09-16T04:13:50.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.666+0000 7f2981ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:13:50.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.666+0000 7f2981ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:13:50.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.666+0000 7f2981ffb640 10 monclient: dump: 2024-09-16T04:13:50.697 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:13:50.697 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:13:50.697 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:50.697 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:50.697 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:50.697 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:13:50.697 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:13:50.698 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:13:50.698 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:13:50.698 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:13:50.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.666+0000 7f2989953640 10 monclient: shutdown 2024-09-16T04:13:50.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.666+0000 7f2989953640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:13:50.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.666+0000 7f2989953640 1 -- 172.21.15.42:0/644559028 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f296c0039c0 msgr2=0x7f296c003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:50.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.666+0000 7f2989953640 1 --2- 172.21.15.42:0/644559028 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f296c0039c0 0x7f296c003dc0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f2974003990 tx=0x7f297401c420 comp rx=0 tx=0).stop 2024-09-16T04:13:50.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.666+0000 7f2989953640 1 -- 172.21.15.42:0/644559028 shutdown_connections 2024-09-16T04:13:50.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.666+0000 7f2989953640 1 --2- 172.21.15.42:0/644559028 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f296c0039c0 0x7f296c003dc0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:13:50.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.666+0000 7f2989953640 1 -- 172.21.15.42:0/644559028 >> 172.21.15.42:0/644559028 conn(0x7f29840a9fb0 msgr2=0x7f29840aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:13:50.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.666+0000 7f2989953640 1 -- 172.21.15.42:0/644559028 shutdown_connections 2024-09-16T04:13:50.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.666+0000 7f2989953640 1 -- 172.21.15.42:0/644559028 wait complete. 2024-09-16T04:13:50.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.670+0000 7f2989953640 10 monclient: build_initial_monmap 2024-09-16T04:13:50.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.670+0000 7f2989953640 10 monclient: monmap: 2024-09-16T04:13:50.700 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:13:50.700 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:13:50.701 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:13:50.701 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:13:50.701 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:13:50.701 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:13:50.701 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:13:50.701 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:13:50.702 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:13:50.702 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:13:50.702 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.670+0000 7f2989953640 1 librados: starting msgr at 2024-09-16T04:13:50.702 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.670+0000 7f2989953640 1 librados: starting objecter 2024-09-16T04:13:50.702 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.670+0000 7f2989953640 1 Processor -- start 2024-09-16T04:13:50.702 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.670+0000 7f2989953640 1 -- start start 2024-09-16T04:13:50.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.670+0000 7f2989953640 1 librados: setting wanted keys 2024-09-16T04:13:50.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.670+0000 7f2989953640 1 librados: calling monclient init 2024-09-16T04:13:50.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.670+0000 7f2989953640 10 monclient: init 2024-09-16T04:13:50.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.670+0000 7f2989953640 10 monclient: _reopen_session rank -1 2024-09-16T04:13:50.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.670+0000 7f2989953640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:13:50.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.670+0000 7f2989953640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f296c0039c0 0x7f29840a30c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:13:50.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.670+0000 7f2989953640 10 monclient(hunting): picked mon.noname-b con 0x7f296c0039c0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:13:50.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.670+0000 7f2989953640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f29841cb940 0x7f29841cbdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:13:50.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.670+0000 7f2989953640 10 monclient(hunting): picked mon.noname-c con 0x7f29841cb940 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:13:50.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.670+0000 7f2989953640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f29841cc300 0x7f298412f3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:13:50.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.670+0000 7f2989953640 10 monclient(hunting): picked mon.noname-a con 0x7f29841cc300 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:13:50.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.670+0000 7f2989953640 10 monclient(hunting): start opening mon connection 2024-09-16T04:13:50.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.670+0000 7f2989953640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f2984137200 con 0x7f29841cc300 2024-09-16T04:13:50.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.670+0000 7f2989953640 10 monclient(hunting): start opening mon connection 2024-09-16T04:13:50.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.670+0000 7f2989953640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f2984137080 con 0x7f296c0039c0 2024-09-16T04:13:50.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.670+0000 7f2989953640 10 monclient(hunting): start opening mon connection 2024-09-16T04:13:50.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.670+0000 7f2989953640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f2984137380 con 0x7f29841cb940 2024-09-16T04:13:50.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.670+0000 7f2989953640 10 monclient(hunting): _renew_subs 2024-09-16T04:13:50.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.670+0000 7f2989953640 10 monclient(hunting): authenticate will time out at 2877.641349s 2024-09-16T04:13:50.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.670+0000 7f29837fe640 1 -- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f29841cc300 msgr2=0x7f298412f3f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.42:3300/0 2024-09-16T04:13:50.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.670+0000 7f29837fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f29841cc300 0x7f298412f3f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:13:50.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.670+0000 7f29827fc640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f29841cb940 0x7f29841cbdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:13:50.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.670+0000 7f29827fc640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f29841cb940 0x7f29841cbdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:60514/0 (socket says 172.21.15.42:60514) 2024-09-16T04:13:50.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.670+0000 7f29827fc640 1 -- 172.21.15.42:0/2727416163 learned_addr learned my addr 172.21.15.42:0/2727416163 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:13:50.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.670+0000 7f29827fc640 10 monclient(hunting): get_auth_request con 0x7f29841cb940 auth_method 0 2024-09-16T04:13:50.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.670+0000 7f29827fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:13:50.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.670+0000 7f29827fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:50.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.670+0000 7f29827fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:13:50.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.670+0000 7f2982ffd640 1 --2- 172.21.15.42:0/2727416163 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f296c0039c0 0x7f29840a30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:13:50.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.670+0000 7f29827fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:13:50.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.670+0000 7f29827fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:13:50.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.670+0000 7f29827fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:13:50.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.670+0000 7f2982ffd640 10 monclient(hunting): get_auth_request con 0x7f296c0039c0 auth_method 0 2024-09-16T04:13:50.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.670+0000 7f2982ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:13:50.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.670+0000 7f2982ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:50.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.670+0000 7f2982ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:13:50.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.670+0000 7f29827fc640 10 monclient(hunting): handle_auth_done global_id 1094183 payload 824 2024-09-16T04:13:50.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.670+0000 7f29827fc640 1 -- 172.21.15.42:0/2727416163 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f296c0039c0 msgr2=0x7f29840a30c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:50.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.670+0000 7f29827fc640 1 --2- 172.21.15.42:0/2727416163 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f296c0039c0 0x7f29840a30c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:13:50.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.670+0000 7f29827fc640 1 -- 172.21.15.42:0/2727416163 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f29841cc300 msgr2=0x7f298412f3f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:13:50.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.670+0000 7f29827fc640 1 --2- 172.21.15.42:0/2727416163 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f29841cc300 0x7f298412f3f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:13:50.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.670+0000 7f29827fc640 10 monclient: _finish_hunting 0 2024-09-16T04:13:50.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.670+0000 7f29827fc640 1 monclient: found mon.noname-c 2024-09-16T04:13:50.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.670+0000 7f29827fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:13:50.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.670+0000 7f29827fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T04:13:50.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.670+0000 7f29827fc640 1 -- 172.21.15.42:0/2727416163 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29840a20a0 con 0x7f29841cb940 2024-09-16T04:13:50.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.670+0000 7f2982ffd640 1 --2- 172.21.15.42:0/2727416163 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f296c0039c0 0x7f29840a30c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T04:13:50.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.670+0000 7f296bfff640 10 client.?.objecter ms_handle_connect 0x7f29841cb940 2024-09-16T04:13:50.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.670+0000 7f296bfff640 10 client.?.objecter resend_mon_ops 2024-09-16T04:13:50.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.670+0000 7f29827fc640 1 --2- 172.21.15.42:0/2727416163 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f29841cb940 0x7f29841cbdc0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f297800b580 tx=0x7f297800ba40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:13:50.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.670+0000 7f296bfff640 1 -- 172.21.15.42:0/2727416163 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2978016070 con 0x7f29841cb940 2024-09-16T04:13:50.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.670+0000 7f296bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:13:50.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.670+0000 7f296bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:13:50.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.670+0000 7f296bfff640 10 monclient: dump: 2024-09-16T04:13:50.707 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:13:50.707 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:13:50.707 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:50.707 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:50.707 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:50.708 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:13:50.708 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:13:50.708 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:13:50.708 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:13:50.708 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:13:50.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.670+0000 7f296bfff640 10 monclient: _finish_auth 0 2024-09-16T04:13:50.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.670+0000 7f296bfff640 10 monclient: _check_auth_tickets 2024-09-16T04:13:50.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.670+0000 7f296bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:13:50.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.670+0000 7f296bfff640 1 -- 172.21.15.42:0/2727416163 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2978003df0 con 0x7f29841cb940 2024-09-16T04:13:50.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.670+0000 7f296bfff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:13:50.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.670+0000 7f296bfff640 1 -- 172.21.15.42:0/2727416163 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2978005310 con 0x7f29841cb940 2024-09-16T04:13:50.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.670+0000 7f2989953640 5 monclient: authenticate success, global_id 1094183 2024-09-16T04:13:50.708 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.670+0000 7f2989953640 10 monclient: _renew_subs 2024-09-16T04:13:50.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.670+0000 7f2989953640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:13:50.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.670+0000 7f2989953640 1 -- 172.21.15.42:0/2727416163 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f298406f6a0 con 0x7f29841cb940 2024-09-16T04:13:50.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.670+0000 7f2980ff9640 4 set_mon_vals no callback set 2024-09-16T04:13:50.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.670+0000 7f296bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:13:50.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.670+0000 7f296bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:13:50.709 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.670+0000 7f296bfff640 10 monclient: dump: 2024-09-16T04:13:50.709 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:13:50.709 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:13:50.709 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:50.709 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:50.710 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:50.710 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:13:50.710 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:13:50.710 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:13:50.710 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:13:50.710 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:13:50.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.670+0000 7f2989953640 10 client.1094183.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:13:50.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.670+0000 7f2989953640 10 monclient: _renew_subs 2024-09-16T04:13:50.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.670+0000 7f2989953640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:13:50.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.670+0000 7f2989953640 1 -- 172.21.15.42:0/2727416163 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f29840a0650 con 0x7f29841cb940 2024-09-16T04:13:50.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.670+0000 7f2989953640 1 librados: init done 2024-09-16T04:13:50.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.670+0000 7f2989953640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:13:50.710 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.670+0000 7f2989953640 1 --2- 172.21.15.42:0/2727416163 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f298402c650 0x7f298402ca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:13:50.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.670+0000 7f2989953640 10 monclient(hunting): start opening mon connection 2024-09-16T04:13:50.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.670+0000 7f2989953640 1 -- 172.21.15.42:0/2727416163 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f2984137710 con 0x7f298402c650 2024-09-16T04:13:50.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.670+0000 7f296bfff640 1 -- 172.21.15.42:0/2727416163 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f2978019070 con 0x7f29841cb940 2024-09-16T04:13:50.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.670+0000 7f2982ffd640 1 --2- 172.21.15.42:0/2727416163 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f298402c650 0x7f298402ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:13:50.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.670+0000 7f2982ffd640 10 monclient: get_auth_request con 0x7f298402c650 auth_method 0 2024-09-16T04:13:50.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.670+0000 7f2982ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:13:50.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.670+0000 7f2982ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:50.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.670+0000 7f2982ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:13:50.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.670+0000 7f296bfff640 1 --2- 172.21.15.42:0/2727416163 >> v2:172.21.15.154:6812/254845342 conn(0x7f29700377c0 0x7f2970039c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:13:50.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.670+0000 7f296bfff640 1 -- 172.21.15.42:0/2727416163 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(709..709 src has 681..709) ==== 5048+0+0 (secure 0 0 0) 0x7f2978047f70 con 0x7f29841cb940 2024-09-16T04:13:50.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.670+0000 7f296bfff640 10 client.1094183.objecter ms_dispatch 0x7f2984000ba0 osd_map(709..709 src has 681..709) 2024-09-16T04:13:50.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.670+0000 7f296bfff640 3 client.1094183.objecter handle_osd_map got epochs [709,709] > 0 2024-09-16T04:13:50.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.670+0000 7f296bfff640 3 client.1094183.objecter handle_osd_map decoding full epoch 709 2024-09-16T04:13:50.711 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.670+0000 7f2982ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:13:50.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.670+0000 7f2982ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:13:50.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.670+0000 7f2982ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:13:50.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.670+0000 7f296bfff640 20 client.1094183.objecter dump_active .. 0 homeless 2024-09-16T04:13:50.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.670+0000 7f29837fe640 1 --2- 172.21.15.42:0/2727416163 >> v2:172.21.15.154:6812/254845342 conn(0x7f29700377c0 0x7f2970039c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:13:50.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.670+0000 7f29837fe640 10 monclient: get_auth_request con 0x7f29700377c0 auth_method 0 2024-09-16T04:13:50.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.674+0000 7f2982ffd640 10 monclient(hunting): handle_auth_done global_id 1074343 payload 824 2024-09-16T04:13:50.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.674+0000 7f2982ffd640 1 -- 172.21.15.42:0/2727416163 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f29841360b0 con 0x7f298402c650 2024-09-16T04:13:50.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.674+0000 7f2982ffd640 1 --2- 172.21.15.42:0/2727416163 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f298402c650 0x7f298402ca30 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f2974006fd0 tx=0x7f29740191c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:13:50.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.674+0000 7f296bfff640 10 client.1094183.objecter ms_handle_connect 0x7f298402c650 2024-09-16T04:13:50.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.674+0000 7f296bfff640 10 client.1094183.objecter resend_mon_ops 2024-09-16T04:13:50.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.674+0000 7f296bfff640 10 client.1094183.objecter ms_handle_connect 0x7f29700377c0 2024-09-16T04:13:50.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.674+0000 7f29837fe640 1 --2- 172.21.15.42:0/2727416163 >> v2:172.21.15.154:6812/254845342 conn(0x7f29700377c0 0x7f2970039c80 secure :-1 s=READY pgs=4001 cs=0 l=1 rev1=1 crypto rx=0x7f29841344a0 tx=0x7f297c0018d0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:13:50.712 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.674+0000 7f296bfff640 1 -- 172.21.15.42:0/2727416163 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29740bc070 con 0x7f298402c650 2024-09-16T04:13:50.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.674+0000 7f296bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:13:50.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.674+0000 7f296bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:13:50.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.674+0000 7f296bfff640 10 monclient: dump: 2024-09-16T04:13:50.713 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:13:50.713 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:13:50.713 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:50.713 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:50.713 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:50.713 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:13:50.713 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:13:50.713 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:13:50.713 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:13:50.713 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:13:50.713 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.674+0000 7f296bfff640 1 -- 172.21.15.42:0/2727416163 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f29740052b0 con 0x7f298402c650 2024-09-16T04:13:50.714 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.674+0000 7f296bfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:13:50.714 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.674+0000 7f296bfff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:13:50.714 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.674+0000 7f296bfff640 1 -- 172.21.15.42:0/2727416163 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f298402c650 msgr2=0x7f298402ca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:50.714 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.674+0000 7f296bfff640 1 --2- 172.21.15.42:0/2727416163 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f298402c650 0x7f298402ca30 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f2974006fd0 tx=0x7f29740191c0 comp rx=0 tx=0).stop 2024-09-16T04:13:50.714 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.674+0000 7f296bfff640 1 -- 172.21.15.42:0/2727416163 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f298402c650 msgr2=0x7f298402ca30 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:13:50.714 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.674+0000 7f296bfff640 1 --2- 172.21.15.42:0/2727416163 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f298402c650 0x7f298402ca30 secure :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f2974006fd0 tx=0x7f29740191c0 comp rx=0 tx=0).stop 2024-09-16T04:13:50.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.678+0000 7f2989953640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:13:50.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.678+0000 7f2989953640 1 --2- 172.21.15.42:0/2727416163 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f298402cf70 0x7f298402d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:13:50.715 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.678+0000 7f2989953640 10 monclient(hunting): start opening mon connection 2024-09-16T04:13:50.716 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.678+0000 7f2989953640 1 -- 172.21.15.42:0/2727416163 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f29840a13f0 con 0x7f298402cf70 2024-09-16T04:13:50.716 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.682+0000 7f2982ffd640 1 --2- 172.21.15.42:0/2727416163 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f298402cf70 0x7f298402d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:13:50.716 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.682+0000 7f2982ffd640 10 monclient: get_auth_request con 0x7f298402cf70 auth_method 0 2024-09-16T04:13:50.716 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.682+0000 7f2982ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:13:50.716 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.682+0000 7f2982ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:13:50.716 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.682+0000 7f2982ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:13:50.716 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.682+0000 7f2982ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:13:50.716 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.682+0000 7f2982ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:13:50.716 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.682+0000 7f2982ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:13:50.716 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.682+0000 7f2982ffd640 10 monclient(hunting): handle_auth_done global_id 1074346 payload 824 2024-09-16T04:13:50.716 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.682+0000 7f2982ffd640 1 -- 172.21.15.42:0/2727416163 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f29841300f0 con 0x7f298402cf70 2024-09-16T04:13:50.716 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.682+0000 7f2982ffd640 1 --2- 172.21.15.42:0/2727416163 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f298402cf70 0x7f298402d380 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f297401c250 tx=0x7f296c005840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:13:50.717 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.682+0000 7f296bfff640 10 client.1094183.objecter ms_handle_connect 0x7f298402cf70 2024-09-16T04:13:50.717 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.682+0000 7f296bfff640 10 client.1094183.objecter resend_mon_ops 2024-09-16T04:13:50.717 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.682+0000 7f296bfff640 1 -- 172.21.15.42:0/2727416163 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29740bc050 con 0x7f298402cf70 2024-09-16T04:13:50.717 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.682+0000 7f296bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:13:50.717 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.682+0000 7f296bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:13:50.717 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.682+0000 7f296bfff640 10 monclient: dump: 2024-09-16T04:13:50.717 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:13:50.717 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:13:50.717 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:50.717 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:13:50.717 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:13:50.717 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:13:50.717 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:13:50.718 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:13:50.718 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:13:50.718 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:13:50.718 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.682+0000 7f296bfff640 1 -- 172.21.15.42:0/2727416163 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4871 (secure 0 0 0) 0x7f29740bd040 con 0x7f298402cf70 2024-09-16T04:13:50.718 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.682+0000 7f296bfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:13:50.718 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.682+0000 7f296bfff640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:13:50.718 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.682+0000 7f296bfff640 1 -- 172.21.15.42:0/2727416163 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f298402cf70 msgr2=0x7f298402d380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:50.718 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.682+0000 7f296bfff640 1 --2- 172.21.15.42:0/2727416163 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f298402cf70 0x7f298402d380 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f297401c250 tx=0x7f296c005840 comp rx=0 tx=0).stop 2024-09-16T04:13:50.718 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.682+0000 7f296bfff640 1 -- 172.21.15.42:0/2727416163 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f298402cf70 msgr2=0x7f298402d380 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:13:50.718 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.682+0000 7f296bfff640 1 --2- 172.21.15.42:0/2727416163 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f298402cf70 0x7f298402d380 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:13:50.718 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T04:13:50.719 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:13:50.719 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:13:50.719 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "leader", 2024-09-16T04:13:50.719 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 288, 2024-09-16T04:13:50.719 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 143855, 2024-09-16T04:13:50.719 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T04:13:50.719 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T04:13:50.719 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T04:13:50.719 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:13:50.719 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 2, 2024-09-16T04:13:50.719 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:13:50.719 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T04:13:50.719 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T04:13:50.720 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:13:50.720 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:13:50.720 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:13:50.720 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:13:50.720 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:13:50.720 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:13:50.720 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:13:50.720 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:13:50.720 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:13:50.720 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:13:50.720 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:13:50.720 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:13:50.720 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:13:50.721 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T04:13:50.721 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:13:50.721 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:13:50.721 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:13:50.721 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:13:50.721 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:13:50.721 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:13:50.721 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:13:50.721 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:13:50.721 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:13:50.721 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:13:50.721 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:13:50.721 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:13:50.721 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:13:50.722 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T04:13:50.722 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T04:13:50.722 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T04:13:50.722 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T04:13:50.722 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T04:13:50.722 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T04:13:50.722 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:13:50.722 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:13:50.722 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T04:13:50.722 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T04:13:50.722 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T04:13:50.722 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T04:13:50.722 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T04:13:50.722 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T04:13:50.723 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T04:13:50.723 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:13:50.723 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T04:13:50.723 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:13:50.723 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:13:50.723 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:13:50.723 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:13:50.723 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:13:50.723 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:13:50.723 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:13:50.723 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:13:50.723 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:13:50.723 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:13:50.723 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:13:50.724 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:13:50.724 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T04:13:50.724 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:13:50.724 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T04:13:50.724 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:13:50.724 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:13:50.724 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:13:50.724 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:13:50.724 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:13:50.724 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:13:50.724 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:13:50.724 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T04:13:50.724 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:13:50.724 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:13:50.725 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:13:50.725 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:13:50.725 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T04:13:50.725 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:13:50.725 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:13:50.725 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:13:50.725 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:13:50.725 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T04:13:50.725 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T04:13:50.725 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:13:50.725 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:13:50.725 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:13:50.725 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:13:50.725 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:13:50.726 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:13:50.726 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:13:50.726 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:13:50.726 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:13:50.726 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:13:50.726 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:13:50.726 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T04:13:50.726 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:13:50.726 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:13:50.726 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:13:50.726 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:13:50.726 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T04:13:50.726 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:13:50.726 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:13:50.727 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:13:50.727 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:13:50.727 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T04:13:50.727 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T04:13:50.727 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:13:50.727 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:13:50.727 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:13:50.727 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:13:50.727 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:13:50.727 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:13:50.727 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:13:50.727 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:13:50.727 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:13:50.727 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:13:50.728 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:13:50.728 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T04:13:50.728 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:13:50.728 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:13:50.728 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:13:50.728 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:13:50.728 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T04:13:50.728 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:13:50.728 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:13:50.728 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:13:50.728 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:13:50.728 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T04:13:50.728 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T04:13:50.728 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:13:50.729 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:13:50.729 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:13:50.729 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:13:50.729 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:13:50.729 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:13:50.729 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T04:13:50.729 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T04:13:50.729 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:13:50.729 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:13:50.729 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:13:50.729 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:13:50.729 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:13:50.729 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:13:50.729 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T04:13:50.730 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:13:50.730 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:13:50.730 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:13:50.730 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:13:50.730 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:13:50.730 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:13:50.730 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T04:13:50.730 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:13:50.730 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:13:50.730 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:13:50.730 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 4 2024-09-16T04:13:50.730 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:13:50.730 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:13:50.730 INFO:teuthology.orchestra.run.smithi042.stdout: "mgr": [ 2024-09-16T04:13:50.731 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:13:50.731 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:13:50.731 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:13:50.731 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:13:50.731 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:13:50.731 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:13:50.731 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:13:50.731 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T04:13:50.731 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T04:13:50.733 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.682+0000 7f2989953640 10 librados: watch_flush enter 2024-09-16T04:13:50.733 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.682+0000 7f2989953640 10 librados: watch_flush exit 2024-09-16T04:13:50.733 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.682+0000 7f2989953640 20 client.1094183.objecter shutdown clearing up homeless session... 2024-09-16T04:13:50.733 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.682+0000 7f2989953640 10 client.1094183.objecter successfully canceled tick 2024-09-16T04:13:50.733 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.682+0000 7f2989953640 1 -- 172.21.15.42:0/2727416163 >> v2:172.21.15.154:6812/254845342 conn(0x7f29700377c0 msgr2=0x7f2970039c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:50.733 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.682+0000 7f2989953640 1 --2- 172.21.15.42:0/2727416163 >> v2:172.21.15.154:6812/254845342 conn(0x7f29700377c0 0x7f2970039c80 secure :-1 s=READY pgs=4001 cs=0 l=1 rev1=1 crypto rx=0x7f29841344a0 tx=0x7f297c0018d0 comp rx=0 tx=0).stop 2024-09-16T04:13:50.733 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.682+0000 7f2989953640 10 monclient: shutdown 2024-09-16T04:13:50.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.682+0000 7f2989953640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:13:50.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.682+0000 7f2989953640 1 -- 172.21.15.42:0/2727416163 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f29841cb940 msgr2=0x7f29841cbdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:13:50.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.682+0000 7f2989953640 1 --2- 172.21.15.42:0/2727416163 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f29841cb940 0x7f29841cbdc0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f297800b580 tx=0x7f297800ba40 comp rx=0 tx=0).stop 2024-09-16T04:13:50.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.682+0000 7f29837fe640 1 -- 172.21.15.42:0/2727416163 reap_dead start 2024-09-16T04:13:50.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.682+0000 7f29837fe640 1 -- 172.21.15.42:0/2727416163 reap_dead start 2024-09-16T04:13:50.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.682+0000 7f2989953640 1 -- 172.21.15.42:0/2727416163 shutdown_connections 2024-09-16T04:13:50.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.682+0000 7f2989953640 1 -- 172.21.15.42:0/2727416163 >> 172.21.15.42:0/2727416163 conn(0x7f29840a9fb0 msgr2=0x7f29841307a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:13:50.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.682+0000 7f2989953640 1 -- 172.21.15.42:0/2727416163 shutdown_connections 2024-09-16T04:13:50.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.682+0000 7f2989953640 1 -- 172.21.15.42:0/2727416163 wait complete. 2024-09-16T04:13:50.734 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:13:50.682+0000 7f2989953640 1 librados: shutdown 2024-09-16T04:13:50.734 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-16T04:13:50.735 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-16T04:13:51.310 INFO:tasks.workunit.client.0.smithi154.stdout: cls: Running main() from gmock_main.cc 2024-09-16T04:13:51.311 INFO:tasks.workunit.client.0.smithi154.stdout: cls: [==========] Running 2 tests from 1 test suite. 2024-09-16T04:13:51.311 INFO:tasks.workunit.client.0.smithi154.stdout: cls: [----------] Global test environment set-up. 2024-09-16T04:13:51.311 INFO:tasks.workunit.client.0.smithi154.stdout: cls: [----------] 2 tests from NeoRadosCls 2024-09-16T04:13:51.311 INFO:tasks.workunit.client.0.smithi154.stdout: cls: [ RUN ] NeoRadosCls.DNE 2024-09-16T04:13:51.311 INFO:tasks.workunit.client.0.smithi154.stdout: cls: [ OK ] NeoRadosCls.DNE (1818532 ms) 2024-09-16T04:13:51.311 INFO:tasks.workunit.client.0.smithi154.stdout: cls: [ RUN ] NeoRadosCls.RemoteReads 2024-09-16T04:13:51.311 INFO:tasks.workunit.client.0.smithi154.stdout: cls: [ OK ] NeoRadosCls.RemoteReads (3028 ms) 2024-09-16T04:13:51.311 INFO:tasks.workunit.client.0.smithi154.stdout: cls: [----------] 2 tests from NeoRadosCls (1821560 ms total) 2024-09-16T04:13:51.311 INFO:tasks.workunit.client.0.smithi154.stdout: cls: 2024-09-16T04:13:51.311 INFO:tasks.workunit.client.0.smithi154.stdout: cls: [----------] Global test environment tear-down 2024-09-16T04:13:51.311 INFO:tasks.workunit.client.0.smithi154.stdout: cls: [==========] 2 tests from 1 test suite ran. (1821560 ms total) 2024-09-16T04:13:51.311 INFO:tasks.workunit.client.0.smithi154.stdout: cls: [ PASSED ] 2 tests. 2024-09-16T04:13:51.314 INFO:tasks.workunit.client.0.smithi154.stderr:+ for t in "${!pids[@]}" 2024-09-16T04:13:51.314 INFO:tasks.workunit.client.0.smithi154.stderr:+ pid=28399 2024-09-16T04:13:51.314 INFO:tasks.workunit.client.0.smithi154.stderr:+ wait 28399 2024-09-16T04:13:51.315 INFO:tasks.workunit.client.0.smithi154.stderr:+ for t in "${!pids[@]}" 2024-09-16T04:13:51.315 INFO:tasks.workunit.client.0.smithi154.stderr:+ pid=28094 2024-09-16T04:13:51.315 INFO:tasks.workunit.client.0.smithi154.stderr:+ wait 28094 2024-09-16T04:13:51.315 INFO:tasks.workunit.client.0.smithi154.stderr:+ for t in "${!pids[@]}" 2024-09-16T04:13:51.315 INFO:tasks.workunit.client.0.smithi154.stderr:+ pid=28126 2024-09-16T04:13:51.315 INFO:tasks.workunit.client.0.smithi154.stderr:+ wait 28126 2024-09-16T04:13:51.315 INFO:tasks.workunit.client.0.smithi154.stderr:+ for t in "${!pids[@]}" 2024-09-16T04:13:51.315 INFO:tasks.workunit.client.0.smithi154.stderr:+ pid=28404 2024-09-16T04:13:51.315 INFO:tasks.workunit.client.0.smithi154.stderr:+ wait 28404 2024-09-16T04:13:51.315 INFO:tasks.workunit.client.0.smithi154.stderr:+ for t in "${!pids[@]}" 2024-09-16T04:13:51.315 INFO:tasks.workunit.client.0.smithi154.stderr:+ pid=28086 2024-09-16T04:13:51.315 INFO:tasks.workunit.client.0.smithi154.stderr:+ wait 28086 2024-09-16T04:13:51.315 INFO:tasks.workunit.client.0.smithi154.stderr:+ for t in "${!pids[@]}" 2024-09-16T04:13:51.315 INFO:tasks.workunit.client.0.smithi154.stderr:+ pid=28102 2024-09-16T04:13:51.316 INFO:tasks.workunit.client.0.smithi154.stderr:+ wait 28102 2024-09-16T04:13:51.316 INFO:tasks.workunit.client.0.smithi154.stderr:+ for t in "${!pids[@]}" 2024-09-16T04:13:51.316 INFO:tasks.workunit.client.0.smithi154.stderr:+ pid=28615 2024-09-16T04:13:51.316 INFO:tasks.workunit.client.0.smithi154.stderr:+ wait 28615 2024-09-16T04:13:51.316 INFO:tasks.workunit.client.0.smithi154.stderr:+ for t in "${!pids[@]}" 2024-09-16T04:13:51.316 INFO:tasks.workunit.client.0.smithi154.stderr:+ pid=28235 2024-09-16T04:13:51.316 INFO:tasks.workunit.client.0.smithi154.stderr:+ wait 28235 2024-09-16T04:13:51.316 INFO:tasks.workunit.client.0.smithi154.stderr:+ for t in "${!pids[@]}" 2024-09-16T04:13:51.316 INFO:tasks.workunit.client.0.smithi154.stderr:+ pid=28637 2024-09-16T04:13:51.316 INFO:tasks.workunit.client.0.smithi154.stderr:+ wait 28637 2024-09-16T04:13:51.316 INFO:tasks.workunit.client.0.smithi154.stderr:+ for t in "${!pids[@]}" 2024-09-16T04:13:51.316 INFO:tasks.workunit.client.0.smithi154.stderr:+ pid=28269 2024-09-16T04:13:51.316 INFO:tasks.workunit.client.0.smithi154.stderr:+ wait 28269 2024-09-16T04:13:51.317 INFO:tasks.workunit.client.0.smithi154.stderr:+ for t in "${!pids[@]}" 2024-09-16T04:13:51.317 INFO:tasks.workunit.client.0.smithi154.stderr:+ pid=28521 2024-09-16T04:13:51.317 INFO:tasks.workunit.client.0.smithi154.stderr:+ wait 28521 2024-09-16T04:13:51.317 INFO:tasks.workunit.client.0.smithi154.stderr:+ for t in "${!pids[@]}" 2024-09-16T04:13:51.317 INFO:tasks.workunit.client.0.smithi154.stderr:+ pid=28669 2024-09-16T04:13:51.317 INFO:tasks.workunit.client.0.smithi154.stderr:+ wait 28669 2024-09-16T04:13:51.317 INFO:tasks.workunit.client.0.smithi154.stderr:+ for t in "${!pids[@]}" 2024-09-16T04:13:51.317 INFO:tasks.workunit.client.0.smithi154.stderr:+ pid=28243 2024-09-16T04:13:51.317 INFO:tasks.workunit.client.0.smithi154.stderr:+ wait 28243 2024-09-16T04:13:51.318 INFO:tasks.workunit.client.0.smithi154.stderr:+ for t in "${!pids[@]}" 2024-09-16T04:13:51.318 INFO:tasks.workunit.client.0.smithi154.stderr:+ pid=28415 2024-09-16T04:13:51.318 INFO:tasks.workunit.client.0.smithi154.stderr:+ wait 28415 2024-09-16T04:13:51.318 INFO:tasks.workunit.client.0.smithi154.stderr:+ for t in "${!pids[@]}" 2024-09-16T04:13:51.318 INFO:tasks.workunit.client.0.smithi154.stderr:+ pid=28597 2024-09-16T04:13:51.318 INFO:tasks.workunit.client.0.smithi154.stderr:+ wait 28597 2024-09-16T04:13:51.318 INFO:tasks.workunit.client.0.smithi154.stderr:+ for t in "${!pids[@]}" 2024-09-16T04:13:51.318 INFO:tasks.workunit.client.0.smithi154.stderr:+ pid=28496 2024-09-16T04:13:51.318 INFO:tasks.workunit.client.0.smithi154.stderr:+ wait 28496 2024-09-16T04:13:51.318 INFO:tasks.workunit.client.0.smithi154.stderr:+ for t in "${!pids[@]}" 2024-09-16T04:13:51.318 INFO:tasks.workunit.client.0.smithi154.stderr:+ pid=28628 2024-09-16T04:13:51.318 INFO:tasks.workunit.client.0.smithi154.stderr:+ wait 28628 2024-09-16T04:13:51.318 INFO:tasks.workunit.client.0.smithi154.stderr:+ for t in "${!pids[@]}" 2024-09-16T04:13:51.318 INFO:tasks.workunit.client.0.smithi154.stderr:+ pid=28382 2024-09-16T04:13:51.319 INFO:tasks.workunit.client.0.smithi154.stderr:+ wait 28382 2024-09-16T04:13:51.319 INFO:tasks.workunit.client.0.smithi154.stderr:+ for t in "${!pids[@]}" 2024-09-16T04:13:51.319 INFO:tasks.workunit.client.0.smithi154.stderr:+ pid=28219 2024-09-16T04:13:51.319 INFO:tasks.workunit.client.0.smithi154.stderr:+ wait 28219 2024-09-16T04:13:51.319 INFO:tasks.workunit.client.0.smithi154.stderr:+ for t in "${!pids[@]}" 2024-09-16T04:13:51.319 INFO:tasks.workunit.client.0.smithi154.stderr:+ pid=28648 2024-09-16T04:13:51.319 INFO:tasks.workunit.client.0.smithi154.stderr:+ wait 28648 2024-09-16T04:13:51.319 INFO:tasks.workunit.client.0.smithi154.stderr:+ for t in "${!pids[@]}" 2024-09-16T04:13:51.319 INFO:tasks.workunit.client.0.smithi154.stderr:+ pid=28356 2024-09-16T04:13:51.319 INFO:tasks.workunit.client.0.smithi154.stderr:+ wait 28356 2024-09-16T04:13:51.319 INFO:tasks.workunit.client.0.smithi154.stderr:+ for t in "${!pids[@]}" 2024-09-16T04:13:51.319 INFO:tasks.workunit.client.0.smithi154.stderr:+ pid=28078 2024-09-16T04:13:51.319 INFO:tasks.workunit.client.0.smithi154.stderr:+ wait 28078 2024-09-16T04:13:51.320 INFO:tasks.workunit.client.0.smithi154.stderr:+ for t in "${!pids[@]}" 2024-09-16T04:13:51.320 INFO:tasks.workunit.client.0.smithi154.stderr:+ pid=28118 2024-09-16T04:13:51.320 INFO:tasks.workunit.client.0.smithi154.stderr:+ wait 28118 2024-09-16T04:13:51.320 INFO:tasks.workunit.client.0.smithi154.stderr:+ for t in "${!pids[@]}" 2024-09-16T04:13:51.320 INFO:tasks.workunit.client.0.smithi154.stderr:+ pid=28322 2024-09-16T04:13:51.320 INFO:tasks.workunit.client.0.smithi154.stderr:+ wait 28322 2024-09-16T04:13:51.320 INFO:tasks.workunit.client.0.smithi154.stderr:+ for t in "${!pids[@]}" 2024-09-16T04:13:51.320 INFO:tasks.workunit.client.0.smithi154.stderr:+ pid=28151 2024-09-16T04:13:51.320 INFO:tasks.workunit.client.0.smithi154.stderr:+ wait 28151 2024-09-16T04:13:51.320 INFO:tasks.workunit.client.0.smithi154.stderr:+ for t in "${!pids[@]}" 2024-09-16T04:13:51.320 INFO:tasks.workunit.client.0.smithi154.stderr:+ pid=28364 2024-09-16T04:13:51.320 INFO:tasks.workunit.client.0.smithi154.stderr:+ wait 28364 2024-09-16T04:13:51.321 INFO:tasks.workunit.client.0.smithi154.stderr:+ for t in "${!pids[@]}" 2024-09-16T04:13:51.321 INFO:tasks.workunit.client.0.smithi154.stderr:+ pid=28372 2024-09-16T04:13:51.321 INFO:tasks.workunit.client.0.smithi154.stderr:+ wait 28372 2024-09-16T04:13:51.321 INFO:tasks.workunit.client.0.smithi154.stderr:+ for t in "${!pids[@]}" 2024-09-16T04:13:51.321 INFO:tasks.workunit.client.0.smithi154.stderr:+ pid=28586 2024-09-16T04:13:51.321 INFO:tasks.workunit.client.0.smithi154.stderr:+ wait 28586 2024-09-16T04:13:51.321 INFO:tasks.workunit.client.0.smithi154.stderr:+ for t in "${!pids[@]}" 2024-09-16T04:13:51.322 INFO:tasks.workunit.client.0.smithi154.stderr:+ pid=28691 2024-09-16T04:13:51.322 INFO:tasks.workunit.client.0.smithi154.stderr:+ wait 28691 2024-09-16T04:13:51.322 INFO:tasks.workunit.client.0.smithi154.stderr:+ exit 0 2024-09-16T04:13:51.322 INFO:tasks.workunit.client.0.smithi154.stderr:+ cleanup 2024-09-16T04:13:51.322 INFO:tasks.workunit.client.0.smithi154.stderr:+ pkill -P 28072 2024-09-16T04:13:51.394 INFO:tasks.workunit.client.0.smithi154.stderr:+ true 2024-09-16T04:13:51.394 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-16T04:13:51.395 DEBUG:teuthology.orchestra.run.smithi154:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-16T04:13:51.451 INFO:tasks.workunit:Stopping ['rados/test.sh unit_test_scan'] on client.0... 2024-09-16T04:13:51.451 DEBUG:teuthology.orchestra.run.smithi154:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-09-16T04:13:51.811 DEBUG:teuthology.parallel:result is None 2024-09-16T04:13:51.811 DEBUG:teuthology.orchestra.run.smithi154:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-09-16T04:13:51.825 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-09-16T04:13:51.825 DEBUG:teuthology.orchestra.run.smithi154:> rmdir -- /home/ubuntu/cephtest/mnt.0 2024-09-16T04:13:51.886 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2024-09-16T04:13:51.887 DEBUG:teuthology.run_tasks:Unwinding manager mon_thrash 2024-09-16T04:13:51.901 INFO:tasks.mon_thrash:joining mon_thrasher 2024-09-16T04:14:10.736 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-16T04:14:10.736 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-09-16T04:14:10.736 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-16T04:14:10.736 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-16T04:14:10.739 INFO:tasks.ceph.mon.a:Started 2024-09-16T04:14:10.739 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T04:14:10.740 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:14:10.862 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.857+0000 7f2c217a4640 10 monclient: get_monmap_and_config 2024-09-16T04:14:10.862 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.857+0000 7f2c217a4640 10 monclient: build_initial_monmap 2024-09-16T04:14:10.862 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.857+0000 7f2c217a4640 10 monclient: monmap: 2024-09-16T04:14:10.862 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:14:10.862 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:14:10.862 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:14:10.860992+0000 2024-09-16T04:14:10.862 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:14:10.860992+0000 2024-09-16T04:14:10.862 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:14:10.863 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:10.863 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:14:10.863 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:14:10.863 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:14:10.863 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:14:10.863 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:14:10.863 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:10.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.857+0000 7f2c217a4640 1 Processor -- start 2024-09-16T04:14:10.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.857+0000 7f2c217a4640 1 -- start start 2024-09-16T04:14:10.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.857+0000 7f2c217a4640 10 monclient: init 2024-09-16T04:14:10.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.857+0000 7f2c217a4640 10 monclient: _reopen_session rank -1 2024-09-16T04:14:10.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.857+0000 7f2c217a4640 10 monclient: _add_conns ranks=[3,4,1,2,0] 2024-09-16T04:14:10.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.857+0000 7f2c217a4640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f2c1c0a39b0 0x7f2c1c133b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:10.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.857+0000 7f2c217a4640 10 monclient(hunting): picked mon.noname-a con 0x7f2c1c0a39b0 addr v2:172.21.15.42:3300/0 2024-09-16T04:14:10.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.857+0000 7f2c217a4640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f2c1c0a3d90 0x7f2c1c137000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:10.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.857+0000 7f2c217a4640 10 monclient(hunting): picked mon.noname-c con 0x7f2c1c0a3d90 addr v2:172.21.15.154:3300/0 2024-09-16T04:14:10.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.857+0000 7f2c217a4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2c1c137540 addr v1:172.21.15.154:6789/0 2024-09-16T04:14:10.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.857+0000 7f2c217a4640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2c1c09c140 con 0x7f2c1c137540 2024-09-16T04:14:10.866 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.857+0000 7f2c217a4640 10 monclient(hunting): start opening mon connection 2024-09-16T04:14:10.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.857+0000 7f2c217a4640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f2c1c09c420 con 0x7f2c1c0a39b0 2024-09-16T04:14:10.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.857+0000 7f2c217a4640 10 monclient(hunting): start opening mon connection 2024-09-16T04:14:10.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.857+0000 7f2c217a4640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f2c1c06dd60 con 0x7f2c1c0a3d90 2024-09-16T04:14:10.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.857+0000 7f2c217a4640 10 monclient(hunting): _renew_subs 2024-09-16T04:14:10.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.857+0000 7f2c217a4640 10 monclient(hunting): authenticate will time out at 2897.830278s 2024-09-16T04:14:10.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.857+0000 7f2c1affd640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f2c1c0a39b0 0x7f2c1c133b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:10.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.857+0000 7f2c1affd640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f2c1c0a39b0 0x7f2c1c133b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:56022/0 (socket says 172.21.15.42:56022) 2024-09-16T04:14:10.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.857+0000 7f2c1affd640 1 -- 172.21.15.42:0/4257071571 learned_addr learned my addr 172.21.15.42:0/4257071571 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:14:10.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.857+0000 7f2c1affd640 10 monclient(hunting): get_auth_request con 0x7f2c1c0a39b0 auth_method 0 2024-09-16T04:14:10.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.857+0000 7f2c1affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:14:10.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.857+0000 7f2c1affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:10.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.857+0000 7f2c1affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:10.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.857+0000 7f2c1affd640 1 -- 172.21.15.42:0/4257071571 >> v2:172.21.15.42:3300/0 conn(0x7f2c1c0a39b0 msgr2=0x7f2c1c133b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:14:10.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.857+0000 7f2c1affd640 1 -- 172.21.15.42:0/4257071571 >> v2:172.21.15.42:3300/0 conn(0x7f2c1c0a39b0 msgr2=0x7f2c1c133b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:14:10.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.857+0000 7f2c1affd640 1 --2- 172.21.15.42:0/4257071571 >> v2:172.21.15.42:3300/0 conn(0x7f2c1c0a39b0 0x7f2c1c133b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:14:10.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.857+0000 7f2c1affd640 1 --2- 172.21.15.42:0/4257071571 >> v2:172.21.15.42:3300/0 conn(0x7f2c1c0a39b0 0x7f2c1c133b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:14:10.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.857+0000 7f2c1a7fc640 1 --2- 172.21.15.42:0/4257071571 >> v2:172.21.15.154:3300/0 conn(0x7f2c1c0a3d90 0x7f2c1c137000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:10.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.857+0000 7f2c1a7fc640 10 monclient(hunting): get_auth_request con 0x7f2c1c0a3d90 auth_method 0 2024-09-16T04:14:10.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.857+0000 7f2c1a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:14:10.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.857+0000 7f2c1a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:10.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.857+0000 7f2c1a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:10.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.857+0000 7f2c1a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:14:10.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.857+0000 7f2c1a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:14:10.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.857+0000 7f2c1a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:14:10.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.857+0000 7f2c1a7fc640 10 monclient(hunting): handle_auth_done global_id 1074349 payload 274 2024-09-16T04:14:10.868 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.857+0000 7f2c1a7fc640 1 -- 172.21.15.42:0/4257071571 >> v2:172.21.15.42:3300/0 conn(0x7f2c1c0a39b0 msgr2=0x7f2c1c133b80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:14:10.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.857+0000 7f2c1a7fc640 1 --2- 172.21.15.42:0/4257071571 >> v2:172.21.15.42:3300/0 conn(0x7f2c1c0a39b0 0x7f2c1c133b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:10.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.857+0000 7f2c1a7fc640 1 -- 172.21.15.42:0/4257071571 >> v1:172.21.15.154:6789/0 conn(0x7f2c1c137540 legacy=0x7f2c1c13e390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:10.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.857+0000 7f2c1a7fc640 10 monclient: _finish_hunting 0 2024-09-16T04:14:10.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.857+0000 7f2c1a7fc640 1 monclient: found mon.noname-c 2024-09-16T04:14:10.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.857+0000 7f2c1a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:14:10.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.857+0000 7f2c1a7fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.154:3300/0 2024-09-16T04:14:10.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.857+0000 7f2c1a7fc640 1 -- 172.21.15.42:0/4257071571 --> v2:172.21.15.154:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c1c0a5e20 con 0x7f2c1c0a3d90 2024-09-16T04:14:10.876 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.861+0000 7f2c1a7fc640 1 --2- 172.21.15.42:0/4257071571 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2c1c0a3d90 0x7f2c1c137000 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f2c1c0a56c0 tx=0x7f2c1001c340 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bd51a387804a6b0d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:14:10.876 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.861+0000 7f2c19ffb640 1 -- 172.21.15.42:0/4257071571 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c100b0070 con 0x7f2c1c0a3d90 2024-09-16T04:14:10.876 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.861+0000 7f2c19ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:10.876 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.861+0000 7f2c19ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:14:10.876 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.861+0000 7f2c19ffb640 10 monclient: dump: 2024-09-16T04:14:10.876 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:14:10.876 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:14:10.876 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:10.876 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:10.876 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:10.876 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:10.877 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:14:10.877 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:14:10.877 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:14:10.877 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:10.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.861+0000 7f2c19ffb640 10 monclient: _finish_auth 0 2024-09-16T04:14:10.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.861+0000 7f2c19ffb640 10 monclient: _check_auth_tickets 2024-09-16T04:14:10.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.861+0000 7f2c19ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:14:10.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.861+0000 7f2c19ffb640 1 -- 172.21.15.42:0/4257071571 <== mon.1 v2:172.21.15.154:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2c100ab600 con 0x7f2c1c0a3d90 2024-09-16T04:14:10.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.861+0000 7f2c19ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:14:10.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.861+0000 7f2c19ffb640 1 -- 172.21.15.42:0/4257071571 <== mon.1 v2:172.21.15.154:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c100ab9d0 con 0x7f2c1c0a3d90 2024-09-16T04:14:10.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.861+0000 7f2c217a4640 5 monclient: authenticate success, global_id 1074349 2024-09-16T04:14:10.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.861+0000 7f2c217a4640 10 monclient: get_monmap_and_config success 2024-09-16T04:14:10.877 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.861+0000 7f2c217a4640 4 set_mon_vals no callback set 2024-09-16T04:14:10.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.861+0000 7f2c19ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:10.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.861+0000 7f2c19ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:14:10.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.861+0000 7f2c19ffb640 10 monclient: dump: 2024-09-16T04:14:10.878 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:14:10.878 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:14:10.878 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:10.878 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:10.878 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:10.878 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:10.878 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:14:10.878 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:14:10.878 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:14:10.878 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:10.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.861+0000 7f2c217a4640 10 monclient: shutdown 2024-09-16T04:14:10.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.861+0000 7f2c217a4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:14:10.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.861+0000 7f2c217a4640 1 -- 172.21.15.42:0/4257071571 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2c1c0a3d90 msgr2=0x7f2c1c137000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:10.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.861+0000 7f2c217a4640 1 --2- 172.21.15.42:0/4257071571 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2c1c0a3d90 0x7f2c1c137000 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f2c1c0a56c0 tx=0x7f2c1001c340 comp rx=0 tx=0).stop 2024-09-16T04:14:10.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.861+0000 7f2c217a4640 1 -- 172.21.15.42:0/4257071571 shutdown_connections 2024-09-16T04:14:10.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.861+0000 7f2c217a4640 1 --2- 172.21.15.42:0/4257071571 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2c1c0a3d90 0x7f2c1c137000 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:10.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.861+0000 7f2c217a4640 1 --2- 172.21.15.42:0/4257071571 >> v2:172.21.15.42:3300/0 conn(0x7f2c1c0a39b0 0x7f2c1c133b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:10.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.861+0000 7f2c217a4640 1 -- 172.21.15.42:0/4257071571 >> 172.21.15.42:0/4257071571 conn(0x7f2c1c0a9fb0 msgr2=0x7f2c1c0aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:14:10.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.861+0000 7f2c217a4640 1 -- 172.21.15.42:0/4257071571 shutdown_connections 2024-09-16T04:14:10.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.861+0000 7f2c217a4640 1 -- 172.21.15.42:0/4257071571 wait complete. 2024-09-16T04:14:10.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.861+0000 7f2c217a4640 10 monclient: build_initial_monmap 2024-09-16T04:14:10.879 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.861+0000 7f2c217a4640 10 monclient: monmap: 2024-09-16T04:14:10.879 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:14:10.879 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:14:10.880 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:14:10.880 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:14:10.880 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:14:10.880 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:10.880 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:14:10.880 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:14:10.880 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:14:10.880 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:10.880 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.861+0000 7f2c217a4640 1 librados: starting msgr at 2024-09-16T04:14:10.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.861+0000 7f2c217a4640 1 librados: starting objecter 2024-09-16T04:14:10.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.861+0000 7f2c217a4640 1 Processor -- start 2024-09-16T04:14:10.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.861+0000 7f2c217a4640 1 -- start start 2024-09-16T04:14:10.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.861+0000 7f2c217a4640 1 librados: setting wanted keys 2024-09-16T04:14:10.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.861+0000 7f2c217a4640 1 librados: calling monclient init 2024-09-16T04:14:10.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.861+0000 7f2c217a4640 10 monclient: init 2024-09-16T04:14:10.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.865+0000 7f2c217a4640 10 monclient: _reopen_session rank -1 2024-09-16T04:14:10.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.865+0000 7f2c217a4640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:14:10.881 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.865+0000 7f2c217a4640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2c1c0a39b0 0x7f2c1c13d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:10.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.865+0000 7f2c217a4640 10 monclient(hunting): picked mon.noname-b con 0x7f2c1c0a39b0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:14:10.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.865+0000 7f2c217a4640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2c1c0a3d90 0x7f2c1c13d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:10.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.865+0000 7f2c217a4640 10 monclient(hunting): picked mon.noname-a con 0x7f2c1c0a3d90 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:14:10.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.865+0000 7f2c217a4640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2c1c137540 0x7f2c1c13df00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:10.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.865+0000 7f2c217a4640 10 monclient(hunting): picked mon.noname-c con 0x7f2c1c137540 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:14:10.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.865+0000 7f2c217a4640 10 monclient(hunting): start opening mon connection 2024-09-16T04:14:10.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.865+0000 7f2c217a4640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f2c1c1424e0 con 0x7f2c1c0a3d90 2024-09-16T04:14:10.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.865+0000 7f2c217a4640 10 monclient(hunting): start opening mon connection 2024-09-16T04:14:10.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.865+0000 7f2c217a4640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f2c1c142660 con 0x7f2c1c0a39b0 2024-09-16T04:14:10.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.865+0000 7f2c217a4640 10 monclient(hunting): start opening mon connection 2024-09-16T04:14:10.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.865+0000 7f2c217a4640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f2c1c1427e0 con 0x7f2c1c137540 2024-09-16T04:14:10.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.865+0000 7f2c217a4640 10 monclient(hunting): _renew_subs 2024-09-16T04:14:10.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.865+0000 7f2c217a4640 10 monclient(hunting): authenticate will time out at 2897.836317s 2024-09-16T04:14:10.882 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.865+0000 7f2c1a7fc640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2c1c0a3d90 0x7f2c1c13d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:10.883 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.865+0000 7f2c1a7fc640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2c1c0a3d90 0x7f2c1c13d9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:56028/0 (socket says 172.21.15.42:56028) 2024-09-16T04:14:10.883 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.865+0000 7f2c1a7fc640 1 -- 172.21.15.42:0/1982312341 learned_addr learned my addr 172.21.15.42:0/1982312341 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:14:10.883 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.865+0000 7f2c1a7fc640 10 monclient(hunting): get_auth_request con 0x7f2c1c0a3d90 auth_method 0 2024-09-16T04:14:10.883 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.865+0000 7f2c1a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:14:10.883 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.865+0000 7f2c1a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:10.883 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.865+0000 7f2c1a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:10.883 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.865+0000 7f2c1b7fe640 1 --2- 172.21.15.42:0/1982312341 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2c1c137540 0x7f2c1c13df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:10.883 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.865+0000 7f2c1b7fe640 10 monclient(hunting): get_auth_request con 0x7f2c1c137540 auth_method 0 2024-09-16T04:14:10.883 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.865+0000 7f2c1b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:14:10.883 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.865+0000 7f2c1b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:10.883 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.865+0000 7f2c1b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:10.883 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.865+0000 7f2c1a7fc640 1 -- 172.21.15.42:0/1982312341 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2c1c0a3d90 msgr2=0x7f2c1c13d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-16T04:14:10.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.865+0000 7f2c1a7fc640 1 -- 172.21.15.42:0/1982312341 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2c1c0a3d90 msgr2=0x7f2c1c13d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:14:10.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.865+0000 7f2c1a7fc640 1 --2- 172.21.15.42:0/1982312341 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2c1c0a3d90 0x7f2c1c13d9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:14:10.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.865+0000 7f2c1b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:14:10.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.865+0000 7f2c1b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:14:10.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.865+0000 7f2c1a7fc640 1 --2- 172.21.15.42:0/1982312341 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2c1c0a3d90 0x7f2c1c13d9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:14:10.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.865+0000 7f2c1b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:14:10.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.865+0000 7f2c1affd640 1 --2- 172.21.15.42:0/1982312341 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2c1c0a39b0 0x7f2c1c13d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:10.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.865+0000 7f2c1affd640 10 monclient(hunting): get_auth_request con 0x7f2c1c0a39b0 auth_method 0 2024-09-16T04:14:10.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.865+0000 7f2c1affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:14:10.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.865+0000 7f2c1affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:10.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.865+0000 7f2c1affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:10.884 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.865+0000 7f2c1b7fe640 10 monclient(hunting): handle_auth_done global_id 1094186 payload 824 2024-09-16T04:14:10.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.865+0000 7f2c1b7fe640 1 -- 172.21.15.42:0/1982312341 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2c1c0a39b0 msgr2=0x7f2c1c13d480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:10.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.865+0000 7f2c1b7fe640 1 --2- 172.21.15.42:0/1982312341 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2c1c0a39b0 0x7f2c1c13d480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:10.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.865+0000 7f2c1b7fe640 1 -- 172.21.15.42:0/1982312341 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2c1c0a3d90 msgr2=0x7f2c1c13d9c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:14:10.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.865+0000 7f2c1b7fe640 1 --2- 172.21.15.42:0/1982312341 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2c1c0a3d90 0x7f2c1c13d9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:10.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.865+0000 7f2c1b7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:14:10.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.865+0000 7f2c1b7fe640 1 monclient: found mon.noname-c 2024-09-16T04:14:10.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.865+0000 7f2c1b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:14:10.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.865+0000 7f2c1b7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T04:14:10.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.865+0000 7f2c1b7fe640 1 -- 172.21.15.42:0/1982312341 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c1c09e7a0 con 0x7f2c1c137540 2024-09-16T04:14:10.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.865+0000 7f2c1b7fe640 1 --2- 172.21.15.42:0/1982312341 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2c1c137540 0x7f2c1c13df00 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f2c1400c700 tx=0x7f2c1400cbc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:14:10.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.865+0000 7f2c037fe640 10 client.?.objecter ms_handle_connect 0x7f2c1c137540 2024-09-16T04:14:10.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.865+0000 7f2c037fe640 10 client.?.objecter resend_mon_ops 2024-09-16T04:14:10.885 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.865+0000 7f2c037fe640 1 -- 172.21.15.42:0/1982312341 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c14003b80 con 0x7f2c1c137540 2024-09-16T04:14:10.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.865+0000 7f2c037fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:10.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.865+0000 7f2c037fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:14:10.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.865+0000 7f2c037fe640 10 monclient: dump: 2024-09-16T04:14:10.886 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:14:10.886 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:14:10.886 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:10.886 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:10.886 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:10.886 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:10.886 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:14:10.886 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:14:10.886 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:14:10.886 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:10.886 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.865+0000 7f2c037fe640 10 monclient: _finish_auth 0 2024-09-16T04:14:10.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.865+0000 7f2c037fe640 10 monclient: _check_auth_tickets 2024-09-16T04:14:10.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.865+0000 7f2c037fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:14:10.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.865+0000 7f2c037fe640 1 -- 172.21.15.42:0/1982312341 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2c14003d20 con 0x7f2c1c137540 2024-09-16T04:14:10.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.865+0000 7f2c037fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:14:10.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.865+0000 7f2c037fe640 1 -- 172.21.15.42:0/1982312341 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c1400aa30 con 0x7f2c1c137540 2024-09-16T04:14:10.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.865+0000 7f2c037fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:10.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.865+0000 7f2c037fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:14:10.887 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.865+0000 7f2c037fe640 10 monclient: dump: 2024-09-16T04:14:10.887 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:14:10.887 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:14:10.887 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:10.887 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:10.888 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:10.888 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:10.888 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:14:10.888 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:14:10.888 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:14:10.888 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:10.888 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.865+0000 7f2c217a4640 5 monclient: authenticate success, global_id 1094186 2024-09-16T04:14:10.888 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.865+0000 7f2c217a4640 10 monclient: _renew_subs 2024-09-16T04:14:10.888 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.865+0000 7f2c217a4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:14:10.888 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.865+0000 7f2c217a4640 1 -- 172.21.15.42:0/1982312341 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c1c09d510 con 0x7f2c1c137540 2024-09-16T04:14:10.888 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.865+0000 7f2c18ff9640 4 set_mon_vals no callback set 2024-09-16T04:14:10.888 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.865+0000 7f2c217a4640 10 client.1094186.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:14:10.888 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.865+0000 7f2c217a4640 10 monclient: _renew_subs 2024-09-16T04:14:10.889 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.865+0000 7f2c217a4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:14:10.889 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.865+0000 7f2c217a4640 1 -- 172.21.15.42:0/1982312341 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c1c136e00 con 0x7f2c1c137540 2024-09-16T04:14:10.889 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.865+0000 7f2c217a4640 1 librados: init done 2024-09-16T04:14:10.889 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.865+0000 7f2c217a4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:14:10.889 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.865+0000 7f2c217a4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:14:10.889 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.865+0000 7f2c217a4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:14:10.889 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.865+0000 7f2c217a4640 1 -- 172.21.15.42:0/1982312341 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2c1c133b80 con 0x7f2c1c137540 2024-09-16T04:14:10.889 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.865+0000 7f2c037fe640 1 -- 172.21.15.42:0/1982312341 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f2c1400abd0 con 0x7f2c1c137540 2024-09-16T04:14:10.889 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.865+0000 7f2c037fe640 1 --2- 172.21.15.42:0/1982312341 >> v2:172.21.15.154:6812/254845342 conn(0x7f2c08037810 0x7f2c08039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:10.889 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.865+0000 7f2c037fe640 1 -- 172.21.15.42:0/1982312341 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(710..710 src has 681..710) ==== 4681+0+0 (secure 0 0 0) 0x7f2c140489f0 con 0x7f2c1c137540 2024-09-16T04:14:10.889 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.865+0000 7f2c037fe640 10 client.1094186.objecter ms_dispatch 0x7f2c1c000ba0 osd_map(710..710 src has 681..710) 2024-09-16T04:14:10.889 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.865+0000 7f2c037fe640 3 client.1094186.objecter handle_osd_map got epochs [710,710] > 0 2024-09-16T04:14:10.890 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.865+0000 7f2c037fe640 3 client.1094186.objecter handle_osd_map decoding full epoch 710 2024-09-16T04:14:10.890 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.865+0000 7f2c037fe640 20 client.1094186.objecter dump_active .. 0 homeless 2024-09-16T04:14:10.890 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.865+0000 7f2c1affd640 1 --2- 172.21.15.42:0/1982312341 >> v2:172.21.15.154:6812/254845342 conn(0x7f2c08037810 0x7f2c08039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:10.890 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.865+0000 7f2c1affd640 10 monclient: get_auth_request con 0x7f2c08037810 auth_method 0 2024-09-16T04:14:10.890 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.869+0000 7f2c1affd640 1 --2- 172.21.15.42:0/1982312341 >> v2:172.21.15.154:6812/254845342 conn(0x7f2c08037810 0x7f2c08039cd0 secure :-1 s=READY pgs=4003 cs=0 l=1 rev1=1 crypto rx=0x7f2c0c009100 tx=0x7f2c0c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:14:10.890 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.869+0000 7f2c037fe640 10 client.1094186.objecter ms_handle_connect 0x7f2c08037810 2024-09-16T04:14:10.890 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.869+0000 7f2c037fe640 1 -- 172.21.15.42:0/1982312341 <== mon.2 v2:172.21.15.42:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f2c1404ae20 con 0x7f2c1c137540 2024-09-16T04:14:10.891 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.869+0000 7f2c037fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:14:10.891 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:10.869+0000 7f2c037fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:14:11.106 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:11.101+0000 7f2c217a4640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:14:11.107 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:11.101+0000 7f2c217a4640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:14:11.107 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:11.101+0000 7f2c217a4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:14:11.107 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:11.101+0000 7f2c217a4640 1 -- 172.21.15.42:0/1982312341 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f2c1c0a5750 con 0x7f2c1c137540 2024-09-16T04:14:11.107 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:11.101+0000 7f2c037fe640 1 -- 172.21.15.42:0/1982312341 <== mon.2 v2:172.21.15.42:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7f2c1400ce10 con 0x7f2c1c137540 2024-09-16T04:14:11.107 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:11.101+0000 7f2c037fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:14:11.108 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:11.101+0000 7f2c037fe640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:14:11.108 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":288,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":22,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:14:11.110 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:11.105+0000 7f2c217a4640 10 librados: watch_flush enter 2024-09-16T04:14:11.110 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:11.105+0000 7f2c217a4640 10 librados: watch_flush exit 2024-09-16T04:14:11.110 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:11.105+0000 7f2c217a4640 20 client.1094186.objecter shutdown clearing up homeless session... 2024-09-16T04:14:11.110 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:11.105+0000 7f2c217a4640 10 client.1094186.objecter successfully canceled tick 2024-09-16T04:14:11.110 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:11.105+0000 7f2c217a4640 1 -- 172.21.15.42:0/1982312341 >> v2:172.21.15.154:6812/254845342 conn(0x7f2c08037810 msgr2=0x7f2c08039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:11.110 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:11.105+0000 7f2c217a4640 1 --2- 172.21.15.42:0/1982312341 >> v2:172.21.15.154:6812/254845342 conn(0x7f2c08037810 0x7f2c08039cd0 secure :-1 s=READY pgs=4003 cs=0 l=1 rev1=1 crypto rx=0x7f2c0c009100 tx=0x7f2c0c008040 comp rx=0 tx=0).stop 2024-09-16T04:14:11.110 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:11.105+0000 7f2c217a4640 10 monclient: shutdown 2024-09-16T04:14:11.110 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:11.105+0000 7f2c217a4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:14:11.110 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:11.105+0000 7f2c217a4640 1 -- 172.21.15.42:0/1982312341 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2c1c137540 msgr2=0x7f2c1c13df00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:11.110 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:11.105+0000 7f2c217a4640 1 --2- 172.21.15.42:0/1982312341 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2c1c137540 0x7f2c1c13df00 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f2c1400c700 tx=0x7f2c1400cbc0 comp rx=0 tx=0).stop 2024-09-16T04:14:11.111 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:11.105+0000 7f2c217a4640 1 -- 172.21.15.42:0/1982312341 shutdown_connections 2024-09-16T04:14:11.111 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:11.105+0000 7f2c217a4640 1 --2- 172.21.15.42:0/1982312341 >> v2:172.21.15.154:6812/254845342 conn(0x7f2c08037810 0x7f2c08039cd0 unknown :-1 s=CLOSED pgs=4003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:11.111 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:11.105+0000 7f2c217a4640 1 --2- 172.21.15.42:0/1982312341 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f2c1c137540 0x7f2c1c13df00 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:11.111 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:11.105+0000 7f2c217a4640 1 --2- 172.21.15.42:0/1982312341 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f2c1c0a3d90 0x7f2c1c13d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:11.111 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:11.105+0000 7f2c217a4640 1 --2- 172.21.15.42:0/1982312341 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f2c1c0a39b0 0x7f2c1c13d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:11.111 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:11.105+0000 7f2c217a4640 1 -- 172.21.15.42:0/1982312341 >> 172.21.15.42:0/1982312341 conn(0x7f2c1c0a9fb0 msgr2=0x7f2c1c131980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:14:11.111 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:11.105+0000 7f2c217a4640 1 -- 172.21.15.42:0/1982312341 shutdown_connections 2024-09-16T04:14:11.111 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:11.105+0000 7f2c217a4640 1 -- 172.21.15.42:0/1982312341 wait complete. 2024-09-16T04:14:11.111 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:11.105+0000 7f2c217a4640 1 librados: shutdown 2024-09-16T04:14:14.127 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:14:14.263 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.257+0000 7fa70c4fc640 10 monclient: get_monmap_and_config 2024-09-16T04:14:14.263 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.257+0000 7fa70c4fc640 10 monclient: build_initial_monmap 2024-09-16T04:14:14.263 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.257+0000 7fa70c4fc640 10 monclient: monmap: 2024-09-16T04:14:14.263 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:14:14.263 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:14:14.263 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:14:14.261874+0000 2024-09-16T04:14:14.263 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:14:14.261874+0000 2024-09-16T04:14:14.263 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:14:14.264 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:14.264 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:14:14.264 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:14:14.264 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:14:14.264 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:14:14.264 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:14:14.264 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:14.266 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.257+0000 7fa70c4fc640 1 Processor -- start 2024-09-16T04:14:14.266 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.257+0000 7fa70c4fc640 1 -- start start 2024-09-16T04:14:14.266 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.257+0000 7fa70c4fc640 10 monclient: init 2024-09-16T04:14:14.266 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.257+0000 7fa70c4fc640 10 monclient: _reopen_session rank -1 2024-09-16T04:14:14.266 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.257+0000 7fa70c4fc640 10 monclient: _add_conns ranks=[1,0,2,3,4] 2024-09-16T04:14:14.266 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.257+0000 7fa70c4fc640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa70409fb00 addr v1:172.21.15.154:6789/0 2024-09-16T04:14:14.267 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.257+0000 7fa70c4fc640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa70409c160 addr v1:172.21.15.42:6789/0 2024-09-16T04:14:14.267 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.257+0000 7fa70c4fc640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa70409c5d0 0x7fa70413c850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:14.267 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.257+0000 7fa70c4fc640 10 monclient(hunting): picked mon.noname-b con 0x7fa70409c5d0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:14:14.268 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.257+0000 7fa70c4fc640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa7040a1e20 con 0x7fa70409c160 2024-09-16T04:14:14.268 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.257+0000 7fa70c4fc640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa70409ea30 con 0x7fa70409fb00 2024-09-16T04:14:14.268 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.257+0000 7fa70c4fc640 10 monclient(hunting): start opening mon connection 2024-09-16T04:14:14.268 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.257+0000 7fa70c4fc640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fa70409e250 con 0x7fa70409c5d0 2024-09-16T04:14:14.268 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.257+0000 7fa70c4fc640 10 monclient(hunting): _renew_subs 2024-09-16T04:14:14.268 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.257+0000 7fa70c4fc640 10 monclient(hunting): authenticate will time out at 2901.231891s 2024-09-16T04:14:14.268 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.261+0000 7fa70aa99640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa70409c5d0 0x7fa70413c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:14.268 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.261+0000 7fa709a97640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fa70409c160 0x7fa704139120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:52904/0 (socket says 172.21.15.42:52904) 2024-09-16T04:14:14.268 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.261+0000 7fa709a97640 1 -- 172.21.15.42:0/1043651609 learned_addr learned my addr 172.21.15.42:0/1043651609 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T04:14:14.268 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.261+0000 7fa70aa99640 10 monclient(hunting): get_auth_request con 0x7fa70409c5d0 auth_method 0 2024-09-16T04:14:14.268 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.261+0000 7fa70aa99640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:14:14.268 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.261+0000 7fa70aa99640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:14.269 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.261+0000 7fa70aa99640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:14.269 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.261+0000 7fa70aa99640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:14:14.269 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.261+0000 7fa70aa99640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:14:14.269 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.261+0000 7fa70aa99640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:14:14.269 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.261+0000 7fa70aa99640 10 monclient(hunting): handle_auth_done global_id 1094189 payload 274 2024-09-16T04:14:14.269 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.261+0000 7fa70aa99640 1 -- 172.21.15.42:0/1043651609 >> v1:172.21.15.154:6789/0 conn(0x7fa70409fb00 legacy=0x7fa70409ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:14.269 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.261+0000 7fa70aa99640 1 -- 172.21.15.42:0/1043651609 >> v1:172.21.15.42:6789/0 conn(0x7fa70409c160 legacy=0x7fa704139120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:14.269 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.261+0000 7fa70aa99640 10 monclient: _finish_hunting 0 2024-09-16T04:14:14.269 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.261+0000 7fa70aa99640 1 monclient: found mon.noname-b 2024-09-16T04:14:14.269 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.261+0000 7fa70aa99640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:14:14.269 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.261+0000 7fa70aa99640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T04:14:14.269 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.261+0000 7fa70aa99640 1 -- 172.21.15.42:0/1043651609 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa70409fee0 con 0x7fa70409c5d0 2024-09-16T04:14:14.275 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.261+0000 7fa70aa99640 1 --2- 172.21.15.42:0/1043651609 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa70409c5d0 0x7fa70413c850 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fa70409e3d0 tx=0x7fa6fc01c850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1f1cc3383636b966 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:14:14.275 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.261+0000 7fa709296640 1 -- 172.21.15.42:0/1043651609 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6fc0b2070 con 0x7fa70409c5d0 2024-09-16T04:14:14.275 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.261+0000 7fa709296640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:14.275 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.261+0000 7fa709296640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:14:14.276 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.261+0000 7fa709296640 10 monclient: dump: 2024-09-16T04:14:14.276 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:14:14.276 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:14:14.276 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:14.276 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:14.276 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:14.276 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:14.276 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:14:14.276 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:14:14.277 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:14:14.277 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:14.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.261+0000 7fa709296640 10 monclient: _finish_auth 0 2024-09-16T04:14:14.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.261+0000 7fa709296640 10 monclient: _check_auth_tickets 2024-09-16T04:14:14.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.261+0000 7fa709296640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:14:14.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.261+0000 7fa70c4fc640 5 monclient: authenticate success, global_id 1094189 2024-09-16T04:14:14.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.261+0000 7fa70c4fc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:14:14.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.261+0000 7fa709296640 1 -- 172.21.15.42:0/1043651609 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa6fc0a5910 con 0x7fa70409c5d0 2024-09-16T04:14:14.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.261+0000 7fa709296640 10 monclient: handle_config config(0 keys) 2024-09-16T04:14:14.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.261+0000 7fa709296640 1 -- 172.21.15.42:0/1043651609 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6fc0a5ce0 con 0x7fa70409c5d0 2024-09-16T04:14:14.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.261+0000 7fa709296640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:14.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.261+0000 7fa709296640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:14:14.277 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.261+0000 7fa709296640 10 monclient: dump: 2024-09-16T04:14:14.278 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:14:14.278 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:14:14.278 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:14.278 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:14.278 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:14.278 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:14.278 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:14:14.278 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:14:14.278 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:14:14.278 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:14.278 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.261+0000 7fa70c4fc640 10 monclient: get_monmap_and_config success 2024-09-16T04:14:14.278 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.261+0000 7fa70c4fc640 4 set_mon_vals no callback set 2024-09-16T04:14:14.278 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.261+0000 7fa70c4fc640 10 monclient: shutdown 2024-09-16T04:14:14.279 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.261+0000 7fa70c4fc640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:14:14.279 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.261+0000 7fa70c4fc640 1 -- 172.21.15.42:0/1043651609 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa70409c5d0 msgr2=0x7fa70413c850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:14.279 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.261+0000 7fa70c4fc640 1 --2- 172.21.15.42:0/1043651609 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa70409c5d0 0x7fa70413c850 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fa70409e3d0 tx=0x7fa6fc01c850 comp rx=0 tx=0).stop 2024-09-16T04:14:14.279 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.261+0000 7fa70c4fc640 1 -- 172.21.15.42:0/1043651609 shutdown_connections 2024-09-16T04:14:14.279 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.261+0000 7fa70c4fc640 1 --2- 172.21.15.42:0/1043651609 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa70409c5d0 0x7fa70413c850 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:14.279 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.261+0000 7fa70c4fc640 1 -- 172.21.15.42:0/1043651609 >> 172.21.15.42:0/1043651609 conn(0x7fa7040a2420 msgr2=0x7fa7040a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:14:14.279 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.261+0000 7fa70c4fc640 1 -- 172.21.15.42:0/1043651609 shutdown_connections 2024-09-16T04:14:14.279 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.261+0000 7fa70c4fc640 1 -- 172.21.15.42:0/1043651609 wait complete. 2024-09-16T04:14:14.279 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.265+0000 7fa70c4fc640 10 monclient: build_initial_monmap 2024-09-16T04:14:14.279 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.265+0000 7fa70c4fc640 10 monclient: monmap: 2024-09-16T04:14:14.279 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:14:14.279 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:14:14.279 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:14:14.280 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:14:14.280 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:14:14.280 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:14.280 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:14:14.280 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:14:14.280 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:14:14.280 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:14.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.265+0000 7fa70c4fc640 1 librados: starting msgr at 2024-09-16T04:14:14.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.265+0000 7fa70c4fc640 1 librados: starting objecter 2024-09-16T04:14:14.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.265+0000 7fa70c4fc640 1 Processor -- start 2024-09-16T04:14:14.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.265+0000 7fa70c4fc640 1 -- start start 2024-09-16T04:14:14.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.265+0000 7fa70c4fc640 1 librados: setting wanted keys 2024-09-16T04:14:14.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.265+0000 7fa70c4fc640 1 librados: calling monclient init 2024-09-16T04:14:14.280 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.265+0000 7fa70c4fc640 10 monclient: init 2024-09-16T04:14:14.281 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.265+0000 7fa70c4fc640 10 monclient: _reopen_session rank -1 2024-09-16T04:14:14.281 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.265+0000 7fa70c4fc640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:14:14.281 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.265+0000 7fa70c4fc640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa70409c160 0x7fa704122910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:14.281 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.265+0000 7fa70c4fc640 10 monclient(hunting): picked mon.noname-b con 0x7fa70409c160 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:14:14.281 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.265+0000 7fa70c4fc640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa70409c5d0 0x7fa704122e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:14.281 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.265+0000 7fa70c4fc640 10 monclient(hunting): picked mon.noname-a con 0x7fa70409c5d0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:14:14.281 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.265+0000 7fa70c4fc640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa70409fb00 0x7fa704123390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:14.281 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.265+0000 7fa70c4fc640 10 monclient(hunting): picked mon.noname-c con 0x7fa70409fb00 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:14:14.281 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.265+0000 7fa70c4fc640 10 monclient(hunting): start opening mon connection 2024-09-16T04:14:14.281 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.265+0000 7fa70c4fc640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fa70409d380 con 0x7fa70409c5d0 2024-09-16T04:14:14.281 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.265+0000 7fa70c4fc640 10 monclient(hunting): start opening mon connection 2024-09-16T04:14:14.281 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.265+0000 7fa70c4fc640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa70409d140 con 0x7fa70409c160 2024-09-16T04:14:14.281 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.265+0000 7fa70c4fc640 10 monclient(hunting): start opening mon connection 2024-09-16T04:14:14.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.265+0000 7fa70c4fc640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fa70409d500 con 0x7fa70409fb00 2024-09-16T04:14:14.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.265+0000 7fa70c4fc640 10 monclient(hunting): _renew_subs 2024-09-16T04:14:14.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.265+0000 7fa70c4fc640 10 monclient(hunting): authenticate will time out at 2901.237107s 2024-09-16T04:14:14.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.265+0000 7fa70aa99640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa70409fb00 0x7fa704123390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:14.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.265+0000 7fa709a97640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa70409c5d0 0x7fa704122e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:14.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.265+0000 7fa70aa99640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa70409fb00 0x7fa704123390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:51754/0 (socket says 172.21.15.42:51754) 2024-09-16T04:14:14.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.265+0000 7fa70aa99640 1 -- 172.21.15.42:0/1073276320 learned_addr learned my addr 172.21.15.42:0/1073276320 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:14:14.283 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.265+0000 7fa70aa99640 10 monclient(hunting): get_auth_request con 0x7fa70409fb00 auth_method 0 2024-09-16T04:14:14.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.265+0000 7fa70aa99640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:14:14.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.265+0000 7fa70aa99640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:14.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.265+0000 7fa709a97640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa70409c5d0 0x7fa704122e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:47814/0 (socket says 172.21.15.42:47814) 2024-09-16T04:14:14.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.265+0000 7fa70aa99640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:14.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.265+0000 7fa709a97640 10 monclient(hunting): get_auth_request con 0x7fa70409c5d0 auth_method 0 2024-09-16T04:14:14.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.265+0000 7fa709a97640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:14:14.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.265+0000 7fa709a97640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:14.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.265+0000 7fa709a97640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:14.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.265+0000 7fa70aa99640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:14:14.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.265+0000 7fa70aa99640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:14:14.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.265+0000 7fa70aa99640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:14:14.284 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.265+0000 7fa709a97640 1 -- 172.21.15.42:0/1073276320 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa70409c5d0 msgr2=0x7fa704122e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-16T04:14:14.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.265+0000 7fa709a97640 1 -- 172.21.15.42:0/1073276320 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa70409c5d0 msgr2=0x7fa704122e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-16T04:14:14.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.265+0000 7fa709a97640 1 --2- 172.21.15.42:0/1073276320 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa70409c5d0 0x7fa704122e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-16T04:14:14.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.265+0000 7fa709a97640 1 --2- 172.21.15.42:0/1073276320 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa70409c5d0 0x7fa704122e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-16T04:14:14.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.265+0000 7fa70a298640 1 --2- 172.21.15.42:0/1073276320 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa70409c160 0x7fa704122910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:14.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.265+0000 7fa70a298640 10 monclient(hunting): get_auth_request con 0x7fa70409c160 auth_method 0 2024-09-16T04:14:14.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.265+0000 7fa70a298640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:14:14.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.265+0000 7fa70a298640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:14.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.265+0000 7fa70a298640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:14.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.265+0000 7fa70aa99640 10 monclient(hunting): handle_auth_done global_id 1094192 payload 824 2024-09-16T04:14:14.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.265+0000 7fa70aa99640 1 -- 172.21.15.42:0/1073276320 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa70409c160 msgr2=0x7fa704122910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:14.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.265+0000 7fa70aa99640 1 --2- 172.21.15.42:0/1073276320 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa70409c160 0x7fa704122910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:14.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.265+0000 7fa70aa99640 1 -- 172.21.15.42:0/1073276320 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa70409c5d0 msgr2=0x7fa704122e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-16T04:14:14.285 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.265+0000 7fa70aa99640 1 --2- 172.21.15.42:0/1073276320 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa70409c5d0 0x7fa704122e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:14.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.265+0000 7fa70aa99640 10 monclient: _finish_hunting 0 2024-09-16T04:14:14.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.265+0000 7fa70aa99640 1 monclient: found mon.noname-c 2024-09-16T04:14:14.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.265+0000 7fa70aa99640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:14:14.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.265+0000 7fa70aa99640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T04:14:14.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.265+0000 7fa70aa99640 1 -- 172.21.15.42:0/1073276320 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa704124c30 con 0x7fa70409fb00 2024-09-16T04:14:14.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.265+0000 7fa70aa99640 1 --2- 172.21.15.42:0/1073276320 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa70409fb00 0x7fa704123390 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fa6fc0b0040 tx=0x7fa6fc0195f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:14:14.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.265+0000 7fa6faffd640 10 client.?.objecter ms_handle_connect 0x7fa70409fb00 2024-09-16T04:14:14.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.265+0000 7fa6faffd640 10 client.?.objecter resend_mon_ops 2024-09-16T04:14:14.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.265+0000 7fa6faffd640 1 -- 172.21.15.42:0/1073276320 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6fc0b2050 con 0x7fa70409fb00 2024-09-16T04:14:14.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.265+0000 7fa6faffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:14.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.265+0000 7fa6faffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:14:14.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.265+0000 7fa6faffd640 10 monclient: dump: 2024-09-16T04:14:14.286 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:14:14.287 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:14:14.287 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:14.287 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:14.287 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:14.287 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:14.287 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:14:14.287 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:14:14.287 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:14:14.287 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:14.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.265+0000 7fa6faffd640 10 monclient: _finish_auth 0 2024-09-16T04:14:14.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.265+0000 7fa6faffd640 10 monclient: _check_auth_tickets 2024-09-16T04:14:14.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.265+0000 7fa6faffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:14:14.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.265+0000 7fa70c4fc640 5 monclient: authenticate success, global_id 1094192 2024-09-16T04:14:14.287 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.265+0000 7fa6faffd640 1 -- 172.21.15.42:0/1073276320 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa6fc0be070 con 0x7fa70409fb00 2024-09-16T04:14:14.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.265+0000 7fa6faffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:14:14.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.265+0000 7fa6faffd640 1 -- 172.21.15.42:0/1073276320 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6fc0adbb0 con 0x7fa70409fb00 2024-09-16T04:14:14.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.265+0000 7fa70c4fc640 10 monclient: _renew_subs 2024-09-16T04:14:14.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.265+0000 7fa70c4fc640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:14:14.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.265+0000 7fa70c4fc640 1 -- 172.21.15.42:0/1073276320 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa704126640 con 0x7fa70409fb00 2024-09-16T04:14:14.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.265+0000 7fa708a95640 4 set_mon_vals no callback set 2024-09-16T04:14:14.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.265+0000 7fa6faffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:14.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.265+0000 7fa6faffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:14:14.288 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.265+0000 7fa6faffd640 10 monclient: dump: 2024-09-16T04:14:14.288 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:14:14.288 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:14:14.288 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:14.288 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:14.289 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:14.289 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:14.289 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:14:14.289 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:14:14.289 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:14:14.289 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:14.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.265+0000 7fa70c4fc640 10 client.1094192.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:14:14.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.265+0000 7fa70c4fc640 10 monclient: _renew_subs 2024-09-16T04:14:14.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.265+0000 7fa70c4fc640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:14:14.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.265+0000 7fa70c4fc640 1 -- 172.21.15.42:0/1073276320 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa704072540 con 0x7fa70409fb00 2024-09-16T04:14:14.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.265+0000 7fa70c4fc640 1 librados: init done 2024-09-16T04:14:14.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.265+0000 7fa70c4fc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:14:14.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.265+0000 7fa70c4fc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:14:14.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.265+0000 7fa70c4fc640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:14:14.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.265+0000 7fa6faffd640 1 -- 172.21.15.42:0/1073276320 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fa6fc0c3460 con 0x7fa70409fb00 2024-09-16T04:14:14.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.265+0000 7fa70c4fc640 1 -- 172.21.15.42:0/1073276320 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa704073a30 con 0x7fa70409fb00 2024-09-16T04:14:14.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.265+0000 7fa6faffd640 1 --2- 172.21.15.42:0/1073276320 >> v2:172.21.15.154:6812/254845342 conn(0x7fa6e8037810 0x7fa6e8039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:14.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.265+0000 7fa6faffd640 1 -- 172.21.15.42:0/1073276320 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(710..710 src has 681..710) ==== 4681+0+0 (secure 0 0 0) 0x7fa6fc0e3f70 con 0x7fa70409fb00 2024-09-16T04:14:14.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.265+0000 7fa6faffd640 10 client.1094192.objecter ms_dispatch 0x7fa704000ba0 osd_map(710..710 src has 681..710) 2024-09-16T04:14:14.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.265+0000 7fa6faffd640 3 client.1094192.objecter handle_osd_map got epochs [710,710] > 0 2024-09-16T04:14:14.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.265+0000 7fa6faffd640 3 client.1094192.objecter handle_osd_map decoding full epoch 710 2024-09-16T04:14:14.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.265+0000 7fa6faffd640 20 client.1094192.objecter dump_active .. 0 homeless 2024-09-16T04:14:14.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.265+0000 7fa70a298640 1 --2- 172.21.15.42:0/1073276320 >> v2:172.21.15.154:6812/254845342 conn(0x7fa6e8037810 0x7fa6e8039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:14.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.265+0000 7fa70a298640 10 monclient: get_auth_request con 0x7fa6e8037810 auth_method 0 2024-09-16T04:14:14.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.269+0000 7fa70a298640 1 --2- 172.21.15.42:0/1073276320 >> v2:172.21.15.154:6812/254845342 conn(0x7fa6e8037810 0x7fa6e8039cd0 secure :-1 s=READY pgs=4004 cs=0 l=1 rev1=1 crypto rx=0x7fa700000f70 tx=0x7fa700009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:14:14.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.269+0000 7fa6faffd640 10 client.1094192.objecter ms_handle_connect 0x7fa6e8037810 2024-09-16T04:14:14.291 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.269+0000 7fa6faffd640 1 -- 172.21.15.42:0/1073276320 <== mon.2 v2:172.21.15.42:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7fa6fc016560 con 0x7fa70409fb00 2024-09-16T04:14:14.291 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.269+0000 7fa6faffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:14:14.291 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.269+0000 7fa6faffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:14:14.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.497+0000 7fa70c4fc640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:14:14.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.497+0000 7fa70c4fc640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:14:14.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.497+0000 7fa70c4fc640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:14:14.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.497+0000 7fa70c4fc640 1 -- 172.21.15.42:0/1073276320 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fa70409ba50 con 0x7fa70409fb00 2024-09-16T04:14:14.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.497+0000 7fa6faffd640 1 -- 172.21.15.42:0/1073276320 <== mon.2 v2:172.21.15.42:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7fa6fc0ba030 con 0x7fa70409fb00 2024-09-16T04:14:14.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.497+0000 7fa6faffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:14:14.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.497+0000 7fa6faffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:14:14.504 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":288,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":26,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:14:14.506 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.501+0000 7fa70c4fc640 10 librados: watch_flush enter 2024-09-16T04:14:14.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.501+0000 7fa70c4fc640 10 librados: watch_flush exit 2024-09-16T04:14:14.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.501+0000 7fa70c4fc640 20 client.1094192.objecter shutdown clearing up homeless session... 2024-09-16T04:14:14.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.501+0000 7fa70c4fc640 10 client.1094192.objecter successfully canceled tick 2024-09-16T04:14:14.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.501+0000 7fa70c4fc640 1 -- 172.21.15.42:0/1073276320 >> v2:172.21.15.154:6812/254845342 conn(0x7fa6e8037810 msgr2=0x7fa6e8039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:14.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.501+0000 7fa70c4fc640 1 --2- 172.21.15.42:0/1073276320 >> v2:172.21.15.154:6812/254845342 conn(0x7fa6e8037810 0x7fa6e8039cd0 secure :-1 s=READY pgs=4004 cs=0 l=1 rev1=1 crypto rx=0x7fa700000f70 tx=0x7fa700009040 comp rx=0 tx=0).stop 2024-09-16T04:14:14.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.501+0000 7fa70c4fc640 10 monclient: shutdown 2024-09-16T04:14:14.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.501+0000 7fa70c4fc640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:14:14.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.501+0000 7fa70c4fc640 1 -- 172.21.15.42:0/1073276320 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa70409fb00 msgr2=0x7fa704123390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:14.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.501+0000 7fa70c4fc640 1 --2- 172.21.15.42:0/1073276320 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa70409fb00 0x7fa704123390 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fa6fc0b0040 tx=0x7fa6fc0195f0 comp rx=0 tx=0).stop 2024-09-16T04:14:14.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.501+0000 7fa70c4fc640 1 -- 172.21.15.42:0/1073276320 shutdown_connections 2024-09-16T04:14:14.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.501+0000 7fa70c4fc640 1 --2- 172.21.15.42:0/1073276320 >> v2:172.21.15.154:6812/254845342 conn(0x7fa6e8037810 0x7fa6e8039cd0 unknown :-1 s=CLOSED pgs=4004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:14.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.501+0000 7fa70c4fc640 1 --2- 172.21.15.42:0/1073276320 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa70409fb00 0x7fa704123390 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:14.507 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.501+0000 7fa70c4fc640 1 --2- 172.21.15.42:0/1073276320 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa70409c5d0 0x7fa704122e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:14.508 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.501+0000 7fa70c4fc640 1 --2- 172.21.15.42:0/1073276320 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa70409c160 0x7fa704122910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:14.508 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.501+0000 7fa70c4fc640 1 -- 172.21.15.42:0/1073276320 >> 172.21.15.42:0/1073276320 conn(0x7fa7040a2420 msgr2=0x7fa7040a37e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:14:14.508 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.501+0000 7fa70c4fc640 1 -- 172.21.15.42:0/1073276320 shutdown_connections 2024-09-16T04:14:14.508 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.501+0000 7fa70c4fc640 1 -- 172.21.15.42:0/1073276320 wait complete. 2024-09-16T04:14:14.508 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:14.501+0000 7fa70c4fc640 1 librados: shutdown 2024-09-16T04:14:17.524 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:14:17.657 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.653+0000 7f9bd277e640 10 monclient: get_monmap_and_config 2024-09-16T04:14:17.658 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.653+0000 7f9bd277e640 10 monclient: build_initial_monmap 2024-09-16T04:14:17.658 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.653+0000 7f9bd277e640 10 monclient: monmap: 2024-09-16T04:14:17.658 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:14:17.658 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:14:17.658 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:14:17.656699+0000 2024-09-16T04:14:17.658 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:14:17.656699+0000 2024-09-16T04:14:17.658 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:14:17.658 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:17.658 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:14:17.658 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:14:17.658 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:14:17.658 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:14:17.659 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:14:17.659 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:17.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.653+0000 7f9bd277e640 1 Processor -- start 2024-09-16T04:14:17.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.653+0000 7f9bd277e640 1 -- start start 2024-09-16T04:14:17.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.653+0000 7f9bd277e640 10 monclient: init 2024-09-16T04:14:17.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.653+0000 7f9bd277e640 10 monclient: _reopen_session rank -1 2024-09-16T04:14:17.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.653+0000 7f9bd277e640 10 monclient: _add_conns ranks=[4,0,3,1,2] 2024-09-16T04:14:17.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.653+0000 7f9bd277e640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f9bcc0a1330 0x7f9bcc0a1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:17.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.653+0000 7f9bd277e640 10 monclient(hunting): picked mon.noname-c con 0x7f9bcc0a1330 addr v2:172.21.15.154:3300/0 2024-09-16T04:14:17.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.653+0000 7f9bd277e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9bcc06dac0 addr v1:172.21.15.42:6789/0 2024-09-16T04:14:17.661 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.653+0000 7f9bd277e640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f9bcc0a1c50 0x7f9bcc135810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:17.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.653+0000 7f9bd277e640 10 monclient(hunting): picked mon.noname-a con 0x7f9bcc0a1c50 addr v2:172.21.15.42:3300/0 2024-09-16T04:14:17.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.653+0000 7f9bd277e640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9bcc09c090 con 0x7f9bcc06dac0 2024-09-16T04:14:17.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.653+0000 7f9bd277e640 10 monclient(hunting): start opening mon connection 2024-09-16T04:14:17.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.653+0000 7f9bd277e640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f9bcc09f610 con 0x7f9bcc0a1c50 2024-09-16T04:14:17.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.653+0000 7f9bd277e640 10 monclient(hunting): start opening mon connection 2024-09-16T04:14:17.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.653+0000 7f9bd277e640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f9bcc09f8f0 con 0x7f9bcc0a1330 2024-09-16T04:14:17.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.653+0000 7f9bd277e640 10 monclient(hunting): _renew_subs 2024-09-16T04:14:17.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.653+0000 7f9bd277e640 10 monclient(hunting): authenticate will time out at 2904.626163s 2024-09-16T04:14:17.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.653+0000 7f9bd0d1b640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f9bcc0a1c50 0x7f9bcc135810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:17.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.653+0000 7f9bcb7fe640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f9bcc06dac0 0x7f9bcc135100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:52910/0 (socket says 172.21.15.42:52910) 2024-09-16T04:14:17.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.653+0000 7f9bd0d1b640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f9bcc0a1c50 0x7f9bcc135810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:47820/0 (socket says 172.21.15.42:47820) 2024-09-16T04:14:17.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.653+0000 7f9bcb7fe640 1 -- 172.21.15.42:0/3135470072 learned_addr learned my addr 172.21.15.42:0/3135470072 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T04:14:17.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.653+0000 7f9bd0d1b640 10 monclient(hunting): get_auth_request con 0x7f9bcc0a1c50 auth_method 0 2024-09-16T04:14:17.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.653+0000 7f9bd0d1b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:14:17.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.653+0000 7f9bd0d1b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:17.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.653+0000 7f9bd0d1b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:17.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.653+0000 7f9bd0d1b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:14:17.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.653+0000 7f9bd0d1b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:14:17.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.653+0000 7f9bd0d1b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:14:17.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.653+0000 7f9bcbfff640 1 --2- 172.21.15.42:0/3135470072 >> v2:172.21.15.154:3300/0 conn(0x7f9bcc0a1330 0x7f9bcc0a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:17.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.653+0000 7f9bcbfff640 10 monclient(hunting): get_auth_request con 0x7f9bcc0a1330 auth_method 0 2024-09-16T04:14:17.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.653+0000 7f9bcbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:14:17.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.653+0000 7f9bcbfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:17.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.653+0000 7f9bcbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:17.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.653+0000 7f9bd0d1b640 10 monclient(hunting): handle_auth_done global_id 1104099 payload 274 2024-09-16T04:14:17.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.653+0000 7f9bd0d1b640 1 -- 172.21.15.42:0/3135470072 >> v2:172.21.15.154:3300/0 conn(0x7f9bcc0a1330 msgr2=0x7f9bcc0a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:14:17.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.653+0000 7f9bd0d1b640 1 --2- 172.21.15.42:0/3135470072 >> v2:172.21.15.154:3300/0 conn(0x7f9bcc0a1330 0x7f9bcc0a1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:17.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.653+0000 7f9bd0d1b640 1 -- 172.21.15.42:0/3135470072 >> v1:172.21.15.42:6789/0 conn(0x7f9bcc06dac0 legacy=0x7f9bcc135100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:17.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.653+0000 7f9bd0d1b640 10 monclient: _finish_hunting 0 2024-09-16T04:14:17.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.653+0000 7f9bd0d1b640 1 monclient: found mon.noname-a 2024-09-16T04:14:17.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.653+0000 7f9bcaffd640 1 -- 172.21.15.42:0/3135470072 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3392459054 0 0) 0x7f9bcc09c090 con 0x7f9bcc06dac0 2024-09-16T04:14:17.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.653+0000 7f9bd0d1b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:14:17.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.653+0000 7f9bd0d1b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:14:17.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.653+0000 7f9bd0d1b640 1 -- 172.21.15.42:0/3135470072 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9bcc09fbd0 con 0x7f9bcc0a1c50 2024-09-16T04:14:17.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.653+0000 7f9bcaffd640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T04:14:17.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.657+0000 7f9bd0d1b640 1 --2- 172.21.15.42:0/3135470072 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9bcc0a1c50 0x7f9bcc135810 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f9bcc06add0 tx=0x7f9bc401c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba4e959291440024 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:14:17.670 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.657+0000 7f9bcaffd640 1 -- 172.21.15.42:0/3135470072 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bc40b2070 con 0x7f9bcc0a1c50 2024-09-16T04:14:17.670 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.657+0000 7f9bcaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:17.670 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.657+0000 7f9bcaffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:14:17.670 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.657+0000 7f9bcaffd640 10 monclient: dump: 2024-09-16T04:14:17.670 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:14:17.670 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:14:17.670 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:17.670 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:17.670 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:17.670 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:17.670 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:14:17.670 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:14:17.670 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:14:17.670 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:17.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.657+0000 7f9bcaffd640 10 monclient: _finish_auth 0 2024-09-16T04:14:17.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.657+0000 7f9bcaffd640 10 monclient: _check_auth_tickets 2024-09-16T04:14:17.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.657+0000 7f9bcaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:14:17.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.657+0000 7f9bcaffd640 1 -- 172.21.15.42:0/3135470072 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9bc40a58d0 con 0x7f9bcc0a1c50 2024-09-16T04:14:17.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.657+0000 7f9bcaffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:14:17.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.657+0000 7f9bcaffd640 1 -- 172.21.15.42:0/3135470072 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bc40a5ca0 con 0x7f9bcc0a1c50 2024-09-16T04:14:17.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.657+0000 7f9bcaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:17.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.657+0000 7f9bcaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:14:17.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.657+0000 7f9bcaffd640 10 monclient: dump: 2024-09-16T04:14:17.671 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:14:17.671 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:14:17.671 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:17.671 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:17.671 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:17.672 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:17.672 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:14:17.672 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:14:17.672 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:14:17.672 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:17.672 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.657+0000 7f9bd277e640 5 monclient: authenticate success, global_id 1104099 2024-09-16T04:14:17.672 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.657+0000 7f9bd277e640 10 monclient: get_monmap_and_config success 2024-09-16T04:14:17.672 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.657+0000 7f9bd277e640 4 set_mon_vals no callback set 2024-09-16T04:14:17.672 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.657+0000 7f9bd277e640 10 monclient: shutdown 2024-09-16T04:14:17.672 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.657+0000 7f9bd277e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:14:17.672 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.657+0000 7f9bd277e640 1 -- 172.21.15.42:0/3135470072 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9bcc0a1c50 msgr2=0x7f9bcc135810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:17.672 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.657+0000 7f9bd277e640 1 --2- 172.21.15.42:0/3135470072 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9bcc0a1c50 0x7f9bcc135810 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f9bcc06add0 tx=0x7f9bc401c7f0 comp rx=0 tx=0).stop 2024-09-16T04:14:17.672 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.657+0000 7f9bd277e640 1 -- 172.21.15.42:0/3135470072 shutdown_connections 2024-09-16T04:14:17.672 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.657+0000 7f9bd277e640 1 --2- 172.21.15.42:0/3135470072 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9bcc0a1c50 0x7f9bcc135810 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:17.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.657+0000 7f9bd277e640 1 --2- 172.21.15.42:0/3135470072 >> v2:172.21.15.154:3300/0 conn(0x7f9bcc0a1330 0x7f9bcc0a1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:17.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.657+0000 7f9bd277e640 1 -- 172.21.15.42:0/3135470072 >> 172.21.15.42:0/3135470072 conn(0x7f9bcc0a2420 msgr2=0x7f9bcc0a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:14:17.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.657+0000 7f9bd277e640 1 -- 172.21.15.42:0/3135470072 shutdown_connections 2024-09-16T04:14:17.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.657+0000 7f9bd277e640 1 -- 172.21.15.42:0/3135470072 wait complete. 2024-09-16T04:14:17.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.657+0000 7f9bd277e640 10 monclient: build_initial_monmap 2024-09-16T04:14:17.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.657+0000 7f9bd277e640 10 monclient: monmap: 2024-09-16T04:14:17.673 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:14:17.673 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:14:17.673 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:14:17.673 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:14:17.673 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:14:17.673 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:17.673 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:14:17.673 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:14:17.674 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:14:17.674 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:17.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.657+0000 7f9bd277e640 1 librados: starting msgr at 2024-09-16T04:14:17.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.657+0000 7f9bd277e640 1 librados: starting objecter 2024-09-16T04:14:17.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.657+0000 7f9bd277e640 1 Processor -- start 2024-09-16T04:14:17.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.657+0000 7f9bd277e640 1 -- start start 2024-09-16T04:14:17.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.657+0000 7f9bd277e640 1 librados: setting wanted keys 2024-09-16T04:14:17.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.657+0000 7f9bd277e640 1 librados: calling monclient init 2024-09-16T04:14:17.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.657+0000 7f9bd277e640 10 monclient: init 2024-09-16T04:14:17.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.661+0000 7f9bd277e640 10 monclient: _reopen_session rank -1 2024-09-16T04:14:17.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.661+0000 7f9bd277e640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:14:17.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.661+0000 7f9bd277e640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9bcc0a1330 0x7f9bcc09cb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:17.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.661+0000 7f9bd277e640 10 monclient(hunting): picked mon.noname-c con 0x7f9bcc0a1330 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:14:17.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.661+0000 7f9bd277e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9bcc0a1c50 0x7f9bcc09d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:17.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.661+0000 7f9bd277e640 10 monclient(hunting): picked mon.noname-b con 0x7f9bcc0a1c50 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:14:17.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.661+0000 7f9bd277e640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9bcc09d5e0 0x7f9bcc1346f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:17.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.661+0000 7f9bd277e640 10 monclient(hunting): picked mon.noname-a con 0x7f9bcc09d5e0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:14:17.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.661+0000 7f9bd277e640 10 monclient(hunting): start opening mon connection 2024-09-16T04:14:17.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.661+0000 7f9bd277e640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f9bcc135d50 con 0x7f9bcc09d5e0 2024-09-16T04:14:17.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.661+0000 7f9bd277e640 10 monclient(hunting): start opening mon connection 2024-09-16T04:14:17.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.661+0000 7f9bd277e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f9bcc09ef80 con 0x7f9bcc0a1c50 2024-09-16T04:14:17.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.661+0000 7f9bd277e640 10 monclient(hunting): start opening mon connection 2024-09-16T04:14:17.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.661+0000 7f9bd277e640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f9bcc135ed0 con 0x7f9bcc0a1330 2024-09-16T04:14:17.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.661+0000 7f9bd277e640 10 monclient(hunting): _renew_subs 2024-09-16T04:14:17.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.661+0000 7f9bd277e640 10 monclient(hunting): authenticate will time out at 2904.632300s 2024-09-16T04:14:17.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.661+0000 7f9bcbfff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9bcc0a1330 0x7f9bcc09cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:17.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.661+0000 7f9bcbfff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9bcc0a1330 0x7f9bcc09cb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:51768/0 (socket says 172.21.15.42:51768) 2024-09-16T04:14:17.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.661+0000 7f9bd0d1b640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9bcc09d5e0 0x7f9bcc1346f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:17.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.661+0000 7f9bcbfff640 1 -- 172.21.15.42:0/924257238 learned_addr learned my addr 172.21.15.42:0/924257238 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:14:17.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.661+0000 7f9bcbfff640 10 monclient(hunting): get_auth_request con 0x7f9bcc0a1330 auth_method 0 2024-09-16T04:14:17.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.661+0000 7f9bcbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:14:17.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.661+0000 7f9bcbfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:17.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.661+0000 7f9bcbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:17.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.661+0000 7f9bd0d1b640 10 monclient(hunting): get_auth_request con 0x7f9bcc09d5e0 auth_method 0 2024-09-16T04:14:17.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.661+0000 7f9bd0d1b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:14:17.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.661+0000 7f9bd0d1b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:17.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.661+0000 7f9bd0d1b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:17.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.661+0000 7f9bcbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:14:17.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.661+0000 7f9bcbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:14:17.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.661+0000 7f9bcbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:14:17.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.661+0000 7f9bd0d1b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:14:17.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.661+0000 7f9bd0d1b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:14:17.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.661+0000 7f9bd0d1b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:14:17.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.661+0000 7f9bcb7fe640 1 --2- 172.21.15.42:0/924257238 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9bcc0a1c50 0x7f9bcc09d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:17.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.661+0000 7f9bcb7fe640 10 monclient(hunting): get_auth_request con 0x7f9bcc0a1c50 auth_method 0 2024-09-16T04:14:17.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.661+0000 7f9bcb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:14:17.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.661+0000 7f9bcb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:17.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.661+0000 7f9bcb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:17.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.661+0000 7f9bcbfff640 10 monclient(hunting): handle_auth_done global_id 1094195 payload 824 2024-09-16T04:14:17.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.661+0000 7f9bcbfff640 1 -- 172.21.15.42:0/924257238 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9bcc0a1c50 msgr2=0x7f9bcc09d0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:17.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.661+0000 7f9bcbfff640 1 --2- 172.21.15.42:0/924257238 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9bcc0a1c50 0x7f9bcc09d0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:17.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.661+0000 7f9bcbfff640 1 -- 172.21.15.42:0/924257238 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9bcc09d5e0 msgr2=0x7f9bcc1346f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:17.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.661+0000 7f9bcbfff640 1 --2- 172.21.15.42:0/924257238 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9bcc09d5e0 0x7f9bcc1346f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:17.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.661+0000 7f9bcbfff640 10 monclient: _finish_hunting 0 2024-09-16T04:14:17.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.661+0000 7f9bcbfff640 1 monclient: found mon.noname-c 2024-09-16T04:14:17.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.661+0000 7f9bcbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:14:17.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.661+0000 7f9bcbfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T04:14:17.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.661+0000 7f9bcbfff640 1 -- 172.21.15.42:0/924257238 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9bcc070fc0 con 0x7f9bcc0a1330 2024-09-16T04:14:17.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.661+0000 7f9bd0d1b640 1 --2- 172.21.15.42:0/924257238 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9bcc09d5e0 0x7f9bcc1346f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:14:17.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.661+0000 7f9bcbfff640 1 --2- 172.21.15.42:0/924257238 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9bcc0a1330 0x7f9bcc09cb60 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f9bb400d830 tx=0x7f9bb400dcf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:14:17.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.661+0000 7f9bc8ff9640 10 client.?.objecter ms_handle_connect 0x7f9bcc0a1330 2024-09-16T04:14:17.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.661+0000 7f9bc8ff9640 10 client.?.objecter resend_mon_ops 2024-09-16T04:14:17.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.661+0000 7f9bc8ff9640 1 -- 172.21.15.42:0/924257238 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bb400b6c0 con 0x7f9bcc0a1330 2024-09-16T04:14:17.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.661+0000 7f9bc8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:17.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.661+0000 7f9bc8ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:14:17.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.661+0000 7f9bc8ff9640 10 monclient: dump: 2024-09-16T04:14:17.678 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:14:17.678 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:14:17.678 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:17.679 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:17.679 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:17.679 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:17.679 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:14:17.679 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:14:17.679 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:14:17.679 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:17.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.661+0000 7f9bc8ff9640 10 monclient: _finish_auth 0 2024-09-16T04:14:17.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.661+0000 7f9bc8ff9640 10 monclient: _check_auth_tickets 2024-09-16T04:14:17.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.661+0000 7f9bc8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:14:17.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.661+0000 7f9bc8ff9640 1 -- 172.21.15.42:0/924257238 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9bb4019070 con 0x7f9bcc0a1330 2024-09-16T04:14:17.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.661+0000 7f9bc8ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T04:14:17.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.661+0000 7f9bc8ff9640 1 -- 172.21.15.42:0/924257238 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bb400bae0 con 0x7f9bcc0a1330 2024-09-16T04:14:17.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.661+0000 7f9bc8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:17.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.661+0000 7f9bc8ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:14:17.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.661+0000 7f9bc8ff9640 10 monclient: dump: 2024-09-16T04:14:17.680 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:14:17.680 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:14:17.680 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:17.680 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:17.680 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:17.680 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:17.680 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:14:17.680 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:14:17.680 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:14:17.681 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:17.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.661+0000 7f9bca7fc640 4 set_mon_vals no callback set 2024-09-16T04:14:17.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.661+0000 7f9bd277e640 5 monclient: authenticate success, global_id 1094195 2024-09-16T04:14:17.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.661+0000 7f9bd277e640 10 monclient: _renew_subs 2024-09-16T04:14:17.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.661+0000 7f9bd277e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:14:17.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.661+0000 7f9bd277e640 1 -- 172.21.15.42:0/924257238 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9bcc121310 con 0x7f9bcc0a1330 2024-09-16T04:14:17.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.661+0000 7f9bd277e640 10 client.1094195.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:14:17.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.661+0000 7f9bd277e640 10 monclient: _renew_subs 2024-09-16T04:14:17.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.661+0000 7f9bd277e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:14:17.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.661+0000 7f9bd277e640 1 -- 172.21.15.42:0/924257238 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f9bcc06e460 con 0x7f9bcc0a1330 2024-09-16T04:14:17.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.661+0000 7f9bd277e640 1 librados: init done 2024-09-16T04:14:17.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.661+0000 7f9bc8ff9640 1 -- 172.21.15.42:0/924257238 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f9bb401e460 con 0x7f9bcc0a1330 2024-09-16T04:14:17.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.661+0000 7f9bd277e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:14:17.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.661+0000 7f9bd277e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:14:17.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.661+0000 7f9bd277e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:14:17.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.661+0000 7f9bd277e640 1 -- 172.21.15.42:0/924257238 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9bcc073a30 con 0x7f9bcc0a1330 2024-09-16T04:14:17.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.661+0000 7f9bc8ff9640 1 --2- 172.21.15.42:0/924257238 >> v2:172.21.15.154:6812/254845342 conn(0x7f9bac037810 0x7f9bac039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:17.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.661+0000 7f9bc8ff9640 1 -- 172.21.15.42:0/924257238 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(710..710 src has 681..710) ==== 4681+0+0 (secure 0 0 0) 0x7f9bb40492b0 con 0x7f9bcc0a1330 2024-09-16T04:14:17.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.661+0000 7f9bc8ff9640 10 client.1094195.objecter ms_dispatch 0x7f9bcc000ba0 osd_map(710..710 src has 681..710) 2024-09-16T04:14:17.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.661+0000 7f9bc8ff9640 3 client.1094195.objecter handle_osd_map got epochs [710,710] > 0 2024-09-16T04:14:17.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.661+0000 7f9bc8ff9640 3 client.1094195.objecter handle_osd_map decoding full epoch 710 2024-09-16T04:14:17.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.661+0000 7f9bc8ff9640 20 client.1094195.objecter dump_active .. 0 homeless 2024-09-16T04:14:17.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.661+0000 7f9bcb7fe640 1 --2- 172.21.15.42:0/924257238 >> v2:172.21.15.154:6812/254845342 conn(0x7f9bac037810 0x7f9bac039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:17.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.661+0000 7f9bcb7fe640 10 monclient: get_auth_request con 0x7f9bac037810 auth_method 0 2024-09-16T04:14:17.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.665+0000 7f9bcb7fe640 1 --2- 172.21.15.42:0/924257238 >> v2:172.21.15.154:6812/254845342 conn(0x7f9bac037810 0x7f9bac039cd0 secure :-1 s=READY pgs=4005 cs=0 l=1 rev1=1 crypto rx=0x7f9bcc121800 tx=0x7f9bbc007b80 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:14:17.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.665+0000 7f9bc8ff9640 10 client.1094195.objecter ms_handle_connect 0x7f9bac037810 2024-09-16T04:14:17.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.665+0000 7f9bc8ff9640 1 -- 172.21.15.42:0/924257238 <== mon.2 v2:172.21.15.42:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f9bcc070fc0 con 0x7f9bcc0a1330 2024-09-16T04:14:17.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.665+0000 7f9bc8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:14:17.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.665+0000 7f9bc8ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:14:17.894 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.889+0000 7f9bd277e640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:14:17.894 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.889+0000 7f9bd277e640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:14:17.895 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.889+0000 7f9bd277e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:14:17.895 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.889+0000 7f9bd277e640 1 -- 172.21.15.42:0/924257238 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f9bcc09e830 con 0x7f9bcc0a1330 2024-09-16T04:14:17.895 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.889+0000 7f9bc8ff9640 1 -- 172.21.15.42:0/924257238 <== mon.2 v2:172.21.15.42:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7f9bb4014430 con 0x7f9bcc0a1330 2024-09-16T04:14:17.895 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.889+0000 7f9bc8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:14:17.895 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.889+0000 7f9bc8ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:14:17.896 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":292,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:14:17.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.893+0000 7f9bd277e640 10 librados: watch_flush enter 2024-09-16T04:14:17.902 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.893+0000 7f9bd277e640 10 librados: watch_flush exit 2024-09-16T04:14:17.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.893+0000 7f9bd277e640 20 client.1094195.objecter shutdown clearing up homeless session... 2024-09-16T04:14:17.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.893+0000 7f9bd277e640 10 client.1094195.objecter successfully canceled tick 2024-09-16T04:14:17.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.893+0000 7f9bd277e640 1 -- 172.21.15.42:0/924257238 >> v2:172.21.15.154:6812/254845342 conn(0x7f9bac037810 msgr2=0x7f9bac039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:17.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.893+0000 7f9bd277e640 1 --2- 172.21.15.42:0/924257238 >> v2:172.21.15.154:6812/254845342 conn(0x7f9bac037810 0x7f9bac039cd0 secure :-1 s=READY pgs=4005 cs=0 l=1 rev1=1 crypto rx=0x7f9bcc121800 tx=0x7f9bbc007b80 comp rx=0 tx=0).stop 2024-09-16T04:14:17.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.893+0000 7f9bd277e640 10 monclient: shutdown 2024-09-16T04:14:17.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.893+0000 7f9bd277e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:14:17.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.893+0000 7f9bd277e640 1 -- 172.21.15.42:0/924257238 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9bcc0a1330 msgr2=0x7f9bcc09cb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:17.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.893+0000 7f9bd277e640 1 --2- 172.21.15.42:0/924257238 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9bcc0a1330 0x7f9bcc09cb60 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f9bb400d830 tx=0x7f9bb400dcf0 comp rx=0 tx=0).stop 2024-09-16T04:14:17.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.893+0000 7f9bd277e640 1 -- 172.21.15.42:0/924257238 shutdown_connections 2024-09-16T04:14:17.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.893+0000 7f9bd277e640 1 --2- 172.21.15.42:0/924257238 >> v2:172.21.15.154:6812/254845342 conn(0x7f9bac037810 0x7f9bac039cd0 unknown :-1 s=CLOSED pgs=4005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:17.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.893+0000 7f9bd277e640 1 --2- 172.21.15.42:0/924257238 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9bcc09d5e0 0x7f9bcc1346f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:17.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.893+0000 7f9bd277e640 1 --2- 172.21.15.42:0/924257238 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9bcc0a1c50 0x7f9bcc09d0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:17.903 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.893+0000 7f9bd277e640 1 --2- 172.21.15.42:0/924257238 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9bcc0a1330 0x7f9bcc09cb60 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:17.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.893+0000 7f9bd277e640 1 -- 172.21.15.42:0/924257238 >> 172.21.15.42:0/924257238 conn(0x7f9bcc0a2420 msgr2=0x7f9bcc071790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:14:17.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.893+0000 7f9bd277e640 1 -- 172.21.15.42:0/924257238 shutdown_connections 2024-09-16T04:14:17.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.893+0000 7f9bd277e640 1 -- 172.21.15.42:0/924257238 wait complete. 2024-09-16T04:14:17.904 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:17.893+0000 7f9bd277e640 1 librados: shutdown 2024-09-16T04:14:17.914 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T04:14:17.914 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-16T04:14:17.915 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-16T04:14:18.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.017+0000 7f0a3097f640 10 monclient: get_monmap_and_config 2024-09-16T04:14:18.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.017+0000 7f0a3097f640 10 monclient: build_initial_monmap 2024-09-16T04:14:18.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.017+0000 7f0a3097f640 10 monclient: monmap: 2024-09-16T04:14:18.024 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:14:18.024 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:14:18.024 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:14:18.023114+0000 2024-09-16T04:14:18.025 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:14:18.023114+0000 2024-09-16T04:14:18.025 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:14:18.025 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:18.025 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:14:18.025 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:14:18.025 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:14:18.025 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:14:18.025 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:14:18.025 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:18.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.021+0000 7f0a3097f640 1 Processor -- start 2024-09-16T04:14:18.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.021+0000 7f0a3097f640 1 -- start start 2024-09-16T04:14:18.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.021+0000 7f0a3097f640 10 monclient: init 2024-09-16T04:14:18.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.021+0000 7f0a3097f640 10 monclient: _reopen_session rank -1 2024-09-16T04:14:18.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.021+0000 7f0a3097f640 10 monclient: _add_conns ranks=[0,3,4,1,2] 2024-09-16T04:14:18.027 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.021+0000 7f0a3097f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0a2c09fb00 addr v1:172.21.15.42:6789/0 2024-09-16T04:14:18.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.021+0000 7f0a3097f640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f0a2c137210 0x7f0a2c139600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:18.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.021+0000 7f0a3097f640 10 monclient(hunting): picked mon.noname-a con 0x7f0a2c137210 addr v2:172.21.15.42:3300/0 2024-09-16T04:14:18.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.021+0000 7f0a3097f640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f0a2c139b40 0x7f0a2c13bf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:18.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.021+0000 7f0a3097f640 10 monclient(hunting): picked mon.noname-c con 0x7f0a2c139b40 addr v2:172.21.15.154:3300/0 2024-09-16T04:14:18.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.021+0000 7f0a3097f640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0a2c0a1e20 con 0x7f0a2c09fb00 2024-09-16T04:14:18.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.021+0000 7f0a3097f640 10 monclient(hunting): start opening mon connection 2024-09-16T04:14:18.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.021+0000 7f0a3097f640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f0a2c0a1a40 con 0x7f0a2c137210 2024-09-16T04:14:18.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.021+0000 7f0a3097f640 10 monclient(hunting): start opening mon connection 2024-09-16T04:14:18.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.021+0000 7f0a3097f640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f0a2c09e250 con 0x7f0a2c139b40 2024-09-16T04:14:18.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.021+0000 7f0a3097f640 10 monclient(hunting): _renew_subs 2024-09-16T04:14:18.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.021+0000 7f0a3097f640 10 monclient(hunting): authenticate will time out at 2904.992984s 2024-09-16T04:14:18.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.021+0000 7f0a2a59c640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f0a2c09fb00 0x7f0a2c0a1330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:52926/0 (socket says 172.21.15.42:52926) 2024-09-16T04:14:18.028 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.021+0000 7f0a2a59c640 1 -- 172.21.15.42:0/3194962535 learned_addr learned my addr 172.21.15.42:0/3194962535 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T04:14:18.029 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.021+0000 7f0a29d9b640 1 --2- 172.21.15.42:0/3194962535 >> v2:172.21.15.42:3300/0 conn(0x7f0a2c137210 0x7f0a2c139600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:18.029 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.021+0000 7f0a2a59c640 1 -- 172.21.15.42:0/3194962535 >> v1:172.21.15.42:6789/0 conn(0x7f0a2c09fb00 legacy=0x7f0a2c0a1330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-16T04:14:18.029 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.021+0000 7f0a2a59c640 1 -- 172.21.15.42:0/3194962535 >> v1:172.21.15.42:6789/0 conn(0x7f0a2c09fb00 legacy=0x7f0a2c0a1330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-16T04:14:18.029 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.021+0000 7f0a2a59c640 1 --1- 172.21.15.42:0/3194962535 >> v1:172.21.15.42:6789/0 conn(0x7f0a2c09fb00 0x7f0a2c0a1330 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=0).handle_connect_reply_1 read connect reply failed 2024-09-16T04:14:18.029 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.021+0000 7f0a29d9b640 10 monclient(hunting): get_auth_request con 0x7f0a2c137210 auth_method 0 2024-09-16T04:14:18.029 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.021+0000 7f0a29d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:14:18.029 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.021+0000 7f0a29d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:18.029 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.021+0000 7f0a29d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:18.029 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.021+0000 7f0a2ad9d640 1 --2- 172.21.15.42:0/3194962535 >> v2:172.21.15.154:3300/0 conn(0x7f0a2c139b40 0x7f0a2c13bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:18.029 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.021+0000 7f0a29d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:14:18.029 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.021+0000 7f0a29d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:14:18.029 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.021+0000 7f0a29d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:14:18.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.021+0000 7f0a2ad9d640 10 monclient(hunting): get_auth_request con 0x7f0a2c139b40 auth_method 0 2024-09-16T04:14:18.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.021+0000 7f0a2ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:14:18.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.021+0000 7f0a2ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:18.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.021+0000 7f0a2ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:18.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.021+0000 7f0a29d9b640 10 monclient(hunting): handle_auth_done global_id 1104108 payload 274 2024-09-16T04:14:18.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.021+0000 7f0a29d9b640 1 -- 172.21.15.42:0/3194962535 >> v2:172.21.15.154:3300/0 conn(0x7f0a2c139b40 msgr2=0x7f0a2c13bf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:14:18.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.021+0000 7f0a29d9b640 1 --2- 172.21.15.42:0/3194962535 >> v2:172.21.15.154:3300/0 conn(0x7f0a2c139b40 0x7f0a2c13bf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:18.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.021+0000 7f0a29d9b640 1 -- 172.21.15.42:0/3194962535 >> v1:172.21.15.42:6789/0 conn(0x7f0a2c09fb00 legacy=0x7f0a2c0a1330 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-16T04:14:18.031 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.021+0000 7f0a29d9b640 10 monclient: _finish_hunting 0 2024-09-16T04:14:18.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.021+0000 7f0a29d9b640 1 monclient: found mon.noname-a 2024-09-16T04:14:18.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.021+0000 7f0a29d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:14:18.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.021+0000 7f0a29d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:14:18.032 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.021+0000 7f0a29d9b640 1 -- 172.21.15.42:0/3194962535 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a2c09ea30 con 0x7f0a2c137210 2024-09-16T04:14:18.039 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.025+0000 7f0a29d9b640 1 --2- 172.21.15.42:0/3194962535 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0a2c137210 0x7f0a2c139600 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f0a2c068f40 tx=0x7f0a2001c300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b62da5c00529d86 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:14:18.039 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.025+0000 7f0a2959a640 1 -- 172.21.15.42:0/3194962535 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a200b0070 con 0x7f0a2c137210 2024-09-16T04:14:18.039 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.025+0000 7f0a2959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:18.039 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.025+0000 7f0a2959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:14:18.039 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.025+0000 7f0a2959a640 10 monclient: dump: 2024-09-16T04:14:18.039 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:14:18.039 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:14:18.039 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:18.039 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:18.039 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:18.039 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:18.039 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:14:18.039 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:14:18.040 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:14:18.040 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:18.040 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.025+0000 7f0a2959a640 10 monclient: _finish_auth 0 2024-09-16T04:14:18.040 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.025+0000 7f0a2959a640 10 monclient: _check_auth_tickets 2024-09-16T04:14:18.040 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.025+0000 7f0a2959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:14:18.040 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.025+0000 7f0a2959a640 1 -- 172.21.15.42:0/3194962535 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a200ab600 con 0x7f0a2c137210 2024-09-16T04:14:18.040 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.025+0000 7f0a3097f640 5 monclient: authenticate success, global_id 1104108 2024-09-16T04:14:18.040 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.025+0000 7f0a3097f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:14:18.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.025+0000 7f0a2959a640 10 monclient: handle_config config(0 keys) 2024-09-16T04:14:18.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.025+0000 7f0a3097f640 10 monclient: get_monmap_and_config success 2024-09-16T04:14:18.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.025+0000 7f0a2959a640 1 -- 172.21.15.42:0/3194962535 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a200ab9d0 con 0x7f0a2c137210 2024-09-16T04:14:18.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.025+0000 7f0a3097f640 4 set_mon_vals no callback set 2024-09-16T04:14:18.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.025+0000 7f0a2959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:18.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.025+0000 7f0a2959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:14:18.041 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.025+0000 7f0a2959a640 10 monclient: dump: 2024-09-16T04:14:18.041 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:14:18.041 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:14:18.041 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:18.042 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:18.042 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:18.042 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:18.042 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:14:18.042 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:14:18.042 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:14:18.042 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:18.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.025+0000 7f0a3097f640 10 monclient: shutdown 2024-09-16T04:14:18.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.025+0000 7f0a3097f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:14:18.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.025+0000 7f0a3097f640 1 -- 172.21.15.42:0/3194962535 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0a2c137210 msgr2=0x7f0a2c139600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:18.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.025+0000 7f0a3097f640 1 --2- 172.21.15.42:0/3194962535 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0a2c137210 0x7f0a2c139600 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f0a2c068f40 tx=0x7f0a2001c300 comp rx=0 tx=0).stop 2024-09-16T04:14:18.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.025+0000 7f0a3097f640 1 -- 172.21.15.42:0/3194962535 shutdown_connections 2024-09-16T04:14:18.042 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.025+0000 7f0a3097f640 1 --2- 172.21.15.42:0/3194962535 >> v2:172.21.15.154:3300/0 conn(0x7f0a2c139b40 0x7f0a2c13bf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:18.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.025+0000 7f0a3097f640 1 --2- 172.21.15.42:0/3194962535 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0a2c137210 0x7f0a2c139600 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:18.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.025+0000 7f0a3097f640 1 -- 172.21.15.42:0/3194962535 >> 172.21.15.42:0/3194962535 conn(0x7f0a2c0a2420 msgr2=0x7f0a2c0a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:14:18.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.025+0000 7f0a3097f640 1 -- 172.21.15.42:0/3194962535 shutdown_connections 2024-09-16T04:14:18.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.025+0000 7f0a3097f640 1 -- 172.21.15.42:0/3194962535 wait complete. 2024-09-16T04:14:18.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.025+0000 7f0a3097f640 10 monclient: build_initial_monmap 2024-09-16T04:14:18.043 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.025+0000 7f0a3097f640 10 monclient: monmap: 2024-09-16T04:14:18.043 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:14:18.043 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:14:18.043 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:14:18.043 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:14:18.043 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:14:18.043 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:18.043 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:14:18.043 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:14:18.044 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:14:18.044 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:18.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.025+0000 7f0a3097f640 1 librados: starting msgr at 2024-09-16T04:14:18.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.025+0000 7f0a3097f640 1 librados: starting objecter 2024-09-16T04:14:18.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.025+0000 7f0a3097f640 1 Processor -- start 2024-09-16T04:14:18.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.025+0000 7f0a3097f640 1 -- start start 2024-09-16T04:14:18.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.025+0000 7f0a3097f640 1 librados: setting wanted keys 2024-09-16T04:14:18.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.025+0000 7f0a3097f640 1 librados: calling monclient init 2024-09-16T04:14:18.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.025+0000 7f0a3097f640 10 monclient: init 2024-09-16T04:14:18.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.025+0000 7f0a3097f640 10 monclient: _reopen_session rank -1 2024-09-16T04:14:18.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.025+0000 7f0a3097f640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:14:18.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.025+0000 7f0a3097f640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0a2c09fb00 0x7f0a2c132bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:18.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.025+0000 7f0a3097f640 10 monclient(hunting): picked mon.noname-c con 0x7f0a2c09fb00 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:14:18.044 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.025+0000 7f0a3097f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0a2c137210 0x7f0a2c1330f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:18.045 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.025+0000 7f0a3097f640 10 monclient(hunting): picked mon.noname-b con 0x7f0a2c137210 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:14:18.045 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.025+0000 7f0a3097f640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0a2c139b40 0x7f0a2c070090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:18.045 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.025+0000 7f0a3097f640 10 monclient(hunting): picked mon.noname-a con 0x7f0a2c139b40 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:14:18.045 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.025+0000 7f0a3097f640 10 monclient(hunting): start opening mon connection 2024-09-16T04:14:18.045 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.025+0000 7f0a3097f640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f0a2c09d380 con 0x7f0a2c139b40 2024-09-16T04:14:18.045 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.025+0000 7f0a3097f640 10 monclient(hunting): start opening mon connection 2024-09-16T04:14:18.045 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.025+0000 7f0a3097f640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f0a2c09d140 con 0x7f0a2c137210 2024-09-16T04:14:18.045 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.025+0000 7f0a3097f640 10 monclient(hunting): start opening mon connection 2024-09-16T04:14:18.045 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.025+0000 7f0a3097f640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f0a2c09d500 con 0x7f0a2c09fb00 2024-09-16T04:14:18.045 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.025+0000 7f0a3097f640 10 monclient(hunting): _renew_subs 2024-09-16T04:14:18.045 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.025+0000 7f0a3097f640 10 monclient(hunting): authenticate will time out at 2904.998473s 2024-09-16T04:14:18.045 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.025+0000 7f0a2a59c640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0a2c09fb00 0x7f0a2c132bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:18.045 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.025+0000 7f0a2ad9d640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0a2c139b40 0x7f0a2c070090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:18.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.025+0000 7f0a2a59c640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0a2c09fb00 0x7f0a2c132bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:51776/0 (socket says 172.21.15.42:51776) 2024-09-16T04:14:18.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.025+0000 7f0a2a59c640 1 -- 172.21.15.42:0/1455908778 learned_addr learned my addr 172.21.15.42:0/1455908778 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:14:18.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.025+0000 7f0a2a59c640 10 monclient(hunting): get_auth_request con 0x7f0a2c09fb00 auth_method 0 2024-09-16T04:14:18.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.025+0000 7f0a2a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:14:18.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.025+0000 7f0a2a59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:18.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.025+0000 7f0a2a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:18.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.025+0000 7f0a2ad9d640 10 monclient(hunting): get_auth_request con 0x7f0a2c139b40 auth_method 0 2024-09-16T04:14:18.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.025+0000 7f0a2ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:14:18.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.025+0000 7f0a2ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:18.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.025+0000 7f0a2ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:18.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.025+0000 7f0a2ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:14:18.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.025+0000 7f0a2ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:14:18.046 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.025+0000 7f0a2ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:14:18.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.025+0000 7f0a2a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:14:18.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.025+0000 7f0a2a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:14:18.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.025+0000 7f0a2a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:14:18.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.025+0000 7f0a29d9b640 1 --2- 172.21.15.42:0/1455908778 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0a2c137210 0x7f0a2c1330f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:18.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.025+0000 7f0a29d9b640 10 monclient(hunting): get_auth_request con 0x7f0a2c137210 auth_method 0 2024-09-16T04:14:18.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.025+0000 7f0a29d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:14:18.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.025+0000 7f0a29d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:18.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.025+0000 7f0a29d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:18.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.025+0000 7f0a2ad9d640 10 monclient(hunting): handle_auth_done global_id 1104111 payload 824 2024-09-16T04:14:18.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.025+0000 7f0a2ad9d640 1 -- 172.21.15.42:0/1455908778 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0a2c09fb00 msgr2=0x7f0a2c132bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:18.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.025+0000 7f0a2ad9d640 1 --2- 172.21.15.42:0/1455908778 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f0a2c09fb00 0x7f0a2c132bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:18.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.025+0000 7f0a2ad9d640 1 -- 172.21.15.42:0/1455908778 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0a2c137210 msgr2=0x7f0a2c1330f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:18.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.025+0000 7f0a2ad9d640 1 --2- 172.21.15.42:0/1455908778 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0a2c137210 0x7f0a2c1330f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:18.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.025+0000 7f0a2ad9d640 10 monclient: _finish_hunting 0 2024-09-16T04:14:18.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.025+0000 7f0a2ad9d640 1 monclient: found mon.noname-a 2024-09-16T04:14:18.048 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.025+0000 7f0a2ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:14:18.048 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.025+0000 7f0a2ad9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:14:18.048 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.025+0000 7f0a2ad9d640 1 -- 172.21.15.42:0/1455908778 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a2c139730 con 0x7f0a2c139b40 2024-09-16T04:14:18.048 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.025+0000 7f0a2ad9d640 1 --2- 172.21.15.42:0/1455908778 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0a2c139b40 0x7f0a2c070090 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f0a24009b60 tx=0x7f0a240085d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:14:18.048 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.025+0000 7f0a1affd640 10 client.?.objecter ms_handle_connect 0x7f0a2c139b40 2024-09-16T04:14:18.048 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.025+0000 7f0a1affd640 10 client.?.objecter resend_mon_ops 2024-09-16T04:14:18.048 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.025+0000 7f0a1affd640 1 -- 172.21.15.42:0/1455908778 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a24008d40 con 0x7f0a2c139b40 2024-09-16T04:14:18.048 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.025+0000 7f0a1affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:18.048 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.025+0000 7f0a1affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:14:18.048 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.025+0000 7f0a1affd640 10 monclient: dump: 2024-09-16T04:14:18.048 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:14:18.048 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:14:18.048 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:18.049 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:18.049 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:18.049 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:18.049 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:14:18.049 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:14:18.049 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:14:18.049 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:18.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.025+0000 7f0a1affd640 10 monclient: _finish_auth 0 2024-09-16T04:14:18.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.025+0000 7f0a1affd640 10 monclient: _check_auth_tickets 2024-09-16T04:14:18.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.025+0000 7f0a1affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:14:18.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.025+0000 7f0a1affd640 1 -- 172.21.15.42:0/1455908778 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a24009c50 con 0x7f0a2c139b40 2024-09-16T04:14:18.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.025+0000 7f0a1affd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:14:18.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.025+0000 7f0a1affd640 1 -- 172.21.15.42:0/1455908778 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a240039a0 con 0x7f0a2c139b40 2024-09-16T04:14:18.049 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.025+0000 7f0a3097f640 5 monclient: authenticate success, global_id 1104111 2024-09-16T04:14:18.050 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.025+0000 7f0a3097f640 10 monclient: _renew_subs 2024-09-16T04:14:18.050 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.025+0000 7f0a3097f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:14:18.050 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.025+0000 7f0a3097f640 1 -- 172.21.15.42:0/1455908778 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a2c13c060 con 0x7f0a2c139b40 2024-09-16T04:14:18.050 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.025+0000 7f0a28d99640 4 set_mon_vals no callback set 2024-09-16T04:14:18.050 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.025+0000 7f0a1affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:18.050 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.025+0000 7f0a1affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:14:18.050 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.025+0000 7f0a1affd640 10 monclient: dump: 2024-09-16T04:14:18.050 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:14:18.050 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:14:18.050 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:18.050 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:18.050 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:18.050 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:18.050 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:14:18.050 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:14:18.051 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:14:18.051 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:18.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.025+0000 7f0a3097f640 10 client.1104111.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:14:18.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.025+0000 7f0a3097f640 10 monclient: _renew_subs 2024-09-16T04:14:18.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.025+0000 7f0a3097f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:14:18.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.025+0000 7f0a3097f640 1 -- 172.21.15.42:0/1455908778 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a2c121980 con 0x7f0a2c139b40 2024-09-16T04:14:18.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.025+0000 7f0a3097f640 1 librados: init done 2024-09-16T04:14:18.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.025+0000 7f0a3097f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:14:18.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.025+0000 7f0a3097f640 1 --2- 172.21.15.42:0/1455908778 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0a2c0705d0 0x7f0a2c0709b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:18.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.029+0000 7f0a3097f640 10 monclient(hunting): start opening mon connection 2024-09-16T04:14:18.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.029+0000 7f0a3097f640 1 -- 172.21.15.42:0/1455908778 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f0a2c073a30 con 0x7f0a2c0705d0 2024-09-16T04:14:18.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.029+0000 7f0a2a59c640 1 --2- 172.21.15.42:0/1455908778 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0a2c0705d0 0x7f0a2c0709b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:18.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.029+0000 7f0a2a59c640 10 monclient: get_auth_request con 0x7f0a2c0705d0 auth_method 0 2024-09-16T04:14:18.051 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.029+0000 7f0a2a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:14:18.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.029+0000 7f0a2a59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:18.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.029+0000 7f0a2a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:18.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.029+0000 7f0a2a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:14:18.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.029+0000 7f0a2a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:14:18.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.029+0000 7f0a2a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:14:18.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.029+0000 7f0a1affd640 1 -- 172.21.15.42:0/1455908778 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f0a24016420 con 0x7f0a2c139b40 2024-09-16T04:14:18.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.029+0000 7f0a2a59c640 10 monclient(hunting): handle_auth_done global_id 1104114 payload 824 2024-09-16T04:14:18.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.029+0000 7f0a2a59c640 1 -- 172.21.15.42:0/1455908778 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0a2c134a20 con 0x7f0a2c0705d0 2024-09-16T04:14:18.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.029+0000 7f0a1affd640 1 --2- 172.21.15.42:0/1455908778 >> v2:172.21.15.154:6812/254845342 conn(0x7f0a0c037810 0x7f0a0c039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:18.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.029+0000 7f0a1affd640 1 -- 172.21.15.42:0/1455908778 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(710..710 src has 681..710) ==== 4681+0+0 (secure 0 0 0) 0x7f0a240488d0 con 0x7f0a2c139b40 2024-09-16T04:14:18.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.029+0000 7f0a1affd640 10 client.1104111.objecter ms_dispatch 0x7f0a2c000ba0 osd_map(710..710 src has 681..710) 2024-09-16T04:14:18.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.029+0000 7f0a1affd640 3 client.1104111.objecter handle_osd_map got epochs [710,710] > 0 2024-09-16T04:14:18.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.029+0000 7f0a1affd640 3 client.1104111.objecter handle_osd_map decoding full epoch 710 2024-09-16T04:14:18.052 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.029+0000 7f0a1affd640 20 client.1104111.objecter dump_active .. 0 homeless 2024-09-16T04:14:18.053 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.029+0000 7f0a1affd640 10 client.1104111.objecter ms_handle_connect 0x7f0a2c0705d0 2024-09-16T04:14:18.053 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.029+0000 7f0a1affd640 10 client.1104111.objecter resend_mon_ops 2024-09-16T04:14:18.053 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.029+0000 7f0a2a59c640 1 --2- 172.21.15.42:0/1455908778 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0a2c0705d0 0x7f0a2c0709b0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f0a1c00eaa0 tx=0x7f0a1c00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:14:18.053 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.029+0000 7f0a1affd640 1 -- 172.21.15.42:0/1455908778 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a1c01a070 con 0x7f0a2c0705d0 2024-09-16T04:14:18.053 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.029+0000 7f0a1affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:18.053 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.029+0000 7f0a1affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:14:18.053 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.029+0000 7f0a1affd640 10 monclient: dump: 2024-09-16T04:14:18.053 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:14:18.053 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:14:18.053 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:18.053 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:18.053 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:18.053 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:18.053 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:14:18.054 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:14:18.054 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:14:18.054 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:18.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.029+0000 7f0a29d9b640 1 --2- 172.21.15.42:0/1455908778 >> v2:172.21.15.154:6812/254845342 conn(0x7f0a0c037810 0x7f0a0c039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:18.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.029+0000 7f0a29d9b640 10 monclient: get_auth_request con 0x7f0a0c037810 auth_method 0 2024-09-16T04:14:18.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.029+0000 7f0a1affd640 1 -- 172.21.15.42:0/1455908778 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f0a1c00c990 con 0x7f0a2c0705d0 2024-09-16T04:14:18.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.029+0000 7f0a1affd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:14:18.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.029+0000 7f0a1affd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:14:18.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.029+0000 7f0a1affd640 1 -- 172.21.15.42:0/1455908778 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0a2c0705d0 msgr2=0x7f0a2c0709b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:18.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.029+0000 7f0a1affd640 1 --2- 172.21.15.42:0/1455908778 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0a2c0705d0 0x7f0a2c0709b0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f0a1c00eaa0 tx=0x7f0a1c00ef60 comp rx=0 tx=0).stop 2024-09-16T04:14:18.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.029+0000 7f0a1affd640 1 -- 172.21.15.42:0/1455908778 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0a2c0705d0 msgr2=0x7f0a2c0709b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:14:18.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.029+0000 7f0a1affd640 1 --2- 172.21.15.42:0/1455908778 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0a2c0705d0 0x7f0a2c0709b0 secure :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f0a1c00eaa0 tx=0x7f0a1c00ef60 comp rx=0 tx=0).stop 2024-09-16T04:14:18.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.029+0000 7f0a1affd640 10 client.1104111.objecter ms_handle_connect 0x7f0a0c037810 2024-09-16T04:14:18.054 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.029+0000 7f0a29d9b640 1 --2- 172.21.15.42:0/1455908778 >> v2:172.21.15.154:6812/254845342 conn(0x7f0a0c037810 0x7f0a0c039cd0 secure :-1 s=READY pgs=4006 cs=0 l=1 rev1=1 crypto rx=0x7f0a200ad000 tx=0x7f0a200a6000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:14:18.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.033+0000 7f0a3097f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:14:18.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.033+0000 7f0a3097f640 1 --2- 172.21.15.42:0/1455908778 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0a2c070ef0 0x7f0a2c1375f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:18.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.033+0000 7f0a3097f640 10 monclient(hunting): start opening mon connection 2024-09-16T04:14:18.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.033+0000 7f0a3097f640 1 -- 172.21.15.42:0/1455908778 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f0a2c133e70 con 0x7f0a2c070ef0 2024-09-16T04:14:18.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.033+0000 7f0a2a59c640 1 --2- 172.21.15.42:0/1455908778 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0a2c070ef0 0x7f0a2c1375f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:18.056 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.033+0000 7f0a2a59c640 10 monclient: get_auth_request con 0x7f0a2c070ef0 auth_method 0 2024-09-16T04:14:18.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.033+0000 7f0a2a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:14:18.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.033+0000 7f0a2a59c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:18.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.033+0000 7f0a2a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:18.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.033+0000 7f0a2a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:14:18.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.033+0000 7f0a2a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:14:18.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.033+0000 7f0a2a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:14:18.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.033+0000 7f0a2a59c640 10 monclient(hunting): handle_auth_done global_id 1104117 payload 824 2024-09-16T04:14:18.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.033+0000 7f0a2a59c640 1 -- 172.21.15.42:0/1455908778 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f0a2c069040 con 0x7f0a2c070ef0 2024-09-16T04:14:18.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.033+0000 7f0a2a59c640 1 --2- 172.21.15.42:0/1455908778 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0a2c070ef0 0x7f0a2c1375f0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f0a1c017000 tx=0x7f0a1c004980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:14:18.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.033+0000 7f0a1affd640 10 client.1104111.objecter ms_handle_connect 0x7f0a2c070ef0 2024-09-16T04:14:18.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.033+0000 7f0a1affd640 10 client.1104111.objecter resend_mon_ops 2024-09-16T04:14:18.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.033+0000 7f0a1affd640 1 -- 172.21.15.42:0/1455908778 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a1c01a040 con 0x7f0a2c070ef0 2024-09-16T04:14:18.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.033+0000 7f0a1affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:18.057 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.033+0000 7f0a1affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:14:18.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.033+0000 7f0a1affd640 10 monclient: dump: 2024-09-16T04:14:18.058 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:14:18.058 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:14:18.058 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:18.058 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:18.058 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:18.058 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:18.058 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:14:18.058 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:14:18.058 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:14:18.058 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:18.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.033+0000 7f0a1affd640 1 -- 172.21.15.42:0/1455908778 <== mon.0 v2:172.21.15.42:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4546 (secure 0 0 0) 0x7f0a1c027420 con 0x7f0a2c070ef0 2024-09-16T04:14:18.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.037+0000 7f0a1affd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:14:18.058 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.037+0000 7f0a1affd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:14:18.059 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.037+0000 7f0a1affd640 1 -- 172.21.15.42:0/1455908778 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0a2c070ef0 msgr2=0x7f0a2c1375f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:18.059 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.037+0000 7f0a1affd640 1 --2- 172.21.15.42:0/1455908778 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0a2c070ef0 0x7f0a2c1375f0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f0a1c017000 tx=0x7f0a1c004980 comp rx=0 tx=0).stop 2024-09-16T04:14:18.059 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.037+0000 7f0a1affd640 1 -- 172.21.15.42:0/1455908778 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0a2c070ef0 msgr2=0x7f0a2c1375f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:14:18.059 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.037+0000 7f0a1affd640 1 --2- 172.21.15.42:0/1455908778 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0a2c070ef0 0x7f0a2c1375f0 secure :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f0a1c017000 tx=0x7f0a1c004980 comp rx=0 tx=0).stop 2024-09-16T04:14:18.059 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T04:14:18.059 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:14:18.059 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:14:18.059 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "leader", 2024-09-16T04:14:18.059 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 292, 2024-09-16T04:14:18.059 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 7215, 2024-09-16T04:14:18.059 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T04:14:18.060 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T04:14:18.060 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T04:14:18.060 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T04:14:18.060 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:14:18.060 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 0, 2024-09-16T04:14:18.060 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:14:18.060 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T04:14:18.060 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T04:14:18.060 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:14:18.060 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:14:18.060 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:14:18.060 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:14:18.060 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:14:18.060 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:14:18.061 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:14:18.061 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:14:18.061 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:14:18.061 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:14:18.061 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:14:18.061 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:14:18.061 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:14:18.061 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T04:14:18.061 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:14:18.061 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:14:18.061 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:14:18.061 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:14:18.061 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:14:18.061 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:14:18.061 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:14:18.062 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:14:18.062 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:14:18.062 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:14:18.062 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:14:18.062 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:14:18.062 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:14:18.062 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T04:14:18.062 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T04:14:18.062 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T04:14:18.062 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T04:14:18.062 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T04:14:18.062 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T04:14:18.062 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:14:18.062 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:14:18.063 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T04:14:18.063 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T04:14:18.063 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T04:14:18.063 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T04:14:18.063 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T04:14:18.063 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T04:14:18.063 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T04:14:18.063 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:14:18.063 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T04:14:18.063 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:14:18.063 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:14:18.063 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:14:18.063 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:14:18.063 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:14:18.064 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:14:18.064 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:14:18.064 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:14:18.064 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:14:18.064 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:14:18.064 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:14:18.064 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:14:18.064 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T04:14:18.064 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:14:18.064 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T04:14:18.064 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:14:18.064 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:14:18.064 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:14:18.064 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:14:18.065 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:14:18.065 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:14:18.065 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:14:18.065 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T04:14:18.065 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:14:18.065 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:14:18.065 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:14:18.065 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:14:18.065 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T04:14:18.065 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:14:18.065 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:14:18.065 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:14:18.065 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:14:18.065 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T04:14:18.065 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T04:14:18.066 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:14:18.066 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:14:18.066 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:14:18.066 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:14:18.066 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:14:18.066 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:14:18.066 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:14:18.066 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:14:18.066 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:14:18.066 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:14:18.066 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:14:18.066 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T04:14:18.066 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:14:18.066 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:14:18.067 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:14:18.067 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:14:18.067 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T04:14:18.067 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:14:18.067 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:14:18.067 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:14:18.067 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:14:18.067 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T04:14:18.067 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T04:14:18.067 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:14:18.067 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:14:18.067 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:14:18.067 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:14:18.067 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:14:18.067 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:14:18.068 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:14:18.068 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:14:18.068 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:14:18.068 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:14:18.068 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:14:18.068 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T04:14:18.068 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:14:18.068 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:14:18.068 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:14:18.068 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:14:18.068 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T04:14:18.068 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:14:18.068 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:14:18.068 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:14:18.069 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:14:18.069 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T04:14:18.069 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T04:14:18.069 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:14:18.069 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:14:18.069 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:14:18.069 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:14:18.069 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:14:18.069 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:14:18.069 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T04:14:18.069 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T04:14:18.069 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:14:18.069 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:14:18.069 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:14:18.070 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:14:18.070 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:14:18.070 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:14:18.070 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T04:14:18.070 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:14:18.070 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:14:18.070 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:14:18.070 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 2 2024-09-16T04:14:18.070 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:14:18.070 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:14:18.070 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:14:18.070 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T04:14:18.070 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T04:14:18.072 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.037+0000 7f0a3097f640 10 librados: watch_flush enter 2024-09-16T04:14:18.072 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.037+0000 7f0a3097f640 10 librados: watch_flush exit 2024-09-16T04:14:18.072 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.037+0000 7f0a3097f640 20 client.1104111.objecter shutdown clearing up homeless session... 2024-09-16T04:14:18.072 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.037+0000 7f0a3097f640 10 client.1104111.objecter successfully canceled tick 2024-09-16T04:14:18.072 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.037+0000 7f0a3097f640 1 -- 172.21.15.42:0/1455908778 >> v2:172.21.15.154:6812/254845342 conn(0x7f0a0c037810 msgr2=0x7f0a0c039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:18.072 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.037+0000 7f0a3097f640 1 --2- 172.21.15.42:0/1455908778 >> v2:172.21.15.154:6812/254845342 conn(0x7f0a0c037810 0x7f0a0c039cd0 secure :-1 s=READY pgs=4006 cs=0 l=1 rev1=1 crypto rx=0x7f0a200ad000 tx=0x7f0a200a6000 comp rx=0 tx=0).stop 2024-09-16T04:14:18.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.037+0000 7f0a3097f640 10 monclient: shutdown 2024-09-16T04:14:18.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.037+0000 7f0a3097f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:14:18.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.037+0000 7f0a3097f640 1 -- 172.21.15.42:0/1455908778 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0a2c139b40 msgr2=0x7f0a2c070090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:18.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.037+0000 7f0a2ad9d640 1 -- 172.21.15.42:0/1455908778 reap_dead start 2024-09-16T04:14:18.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.037+0000 7f0a3097f640 1 --2- 172.21.15.42:0/1455908778 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0a2c139b40 0x7f0a2c070090 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f0a24009b60 tx=0x7f0a240085d0 comp rx=0 tx=0).stop 2024-09-16T04:14:18.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.037+0000 7f0a3097f640 1 -- 172.21.15.42:0/1455908778 shutdown_connections 2024-09-16T04:14:18.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.037+0000 7f0a3097f640 1 --2- 172.21.15.42:0/1455908778 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0a2c139b40 0x7f0a2c070090 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:18.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.037+0000 7f0a3097f640 1 -- 172.21.15.42:0/1455908778 >> 172.21.15.42:0/1455908778 conn(0x7f0a2c0a2420 msgr2=0x7f0a2c135a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:14:18.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.037+0000 7f0a3097f640 1 -- 172.21.15.42:0/1455908778 shutdown_connections 2024-09-16T04:14:18.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.037+0000 7f0a3097f640 1 -- 172.21.15.42:0/1455908778 wait complete. 2024-09-16T04:14:18.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.037+0000 7f0a3097f640 1 librados: shutdown 2024-09-16T04:14:18.073 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:14:18.074 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-16T04:14:18.074 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-16T04:14:18.187 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.181+0000 7fb9739ea640 10 monclient: get_monmap_and_config 2024-09-16T04:14:18.187 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.181+0000 7fb9739ea640 10 monclient: build_initial_monmap 2024-09-16T04:14:18.187 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.181+0000 7fb9739ea640 10 monclient: monmap: 2024-09-16T04:14:18.187 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:14:18.187 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:14:18.187 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:14:18.184780+0000 2024-09-16T04:14:18.187 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:14:18.184780+0000 2024-09-16T04:14:18.187 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:14:18.187 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:18.188 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:14:18.188 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:14:18.188 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:14:18.188 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:14:18.188 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:14:18.188 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:18.188 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.181+0000 7fb9739ea640 1 Processor -- start 2024-09-16T04:14:18.188 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.181+0000 7fb9739ea640 1 -- start start 2024-09-16T04:14:18.188 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.181+0000 7fb9739ea640 10 monclient: init 2024-09-16T04:14:18.190 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.181+0000 7fb9739ea640 10 monclient: _reopen_session rank -1 2024-09-16T04:14:18.190 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.181+0000 7fb9739ea640 10 monclient: _add_conns ranks=[2,3,4,1,0] 2024-09-16T04:14:18.190 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.181+0000 7fb9739ea640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb96c0a2340 0x7fb96c0a2720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:18.190 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.181+0000 7fb9739ea640 10 monclient(hunting): picked mon.noname-b con 0x7fb96c0a2340 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:14:18.190 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.181+0000 7fb9739ea640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb96c071510 0x7fb96c0a2c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:18.190 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.181+0000 7fb9739ea640 10 monclient(hunting): picked mon.noname-a con 0x7fb96c071510 addr v2:172.21.15.42:3300/0 2024-09-16T04:14:18.190 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.181+0000 7fb9739ea640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fb96c06f260 0x7fb96c1340f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:18.191 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.181+0000 7fb9739ea640 10 monclient(hunting): picked mon.noname-c con 0x7fb96c06f260 addr v2:172.21.15.154:3300/0 2024-09-16T04:14:18.191 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.181+0000 7fb9739ea640 10 monclient(hunting): start opening mon connection 2024-09-16T04:14:18.191 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.181+0000 7fb9739ea640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7fb96c0a31a0 con 0x7fb96c071510 2024-09-16T04:14:18.191 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.181+0000 7fb9739ea640 10 monclient(hunting): start opening mon connection 2024-09-16T04:14:18.191 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.181+0000 7fb9739ea640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fb96c070570 con 0x7fb96c06f260 2024-09-16T04:14:18.191 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.181+0000 7fb9739ea640 10 monclient(hunting): start opening mon connection 2024-09-16T04:14:18.191 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.181+0000 7fb9739ea640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fb96c06fa60 con 0x7fb96c0a2340 2024-09-16T04:14:18.191 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.181+0000 7fb9739ea640 10 monclient(hunting): _renew_subs 2024-09-16T04:14:18.191 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.181+0000 7fb9739ea640 10 monclient(hunting): authenticate will time out at 2905.154926s 2024-09-16T04:14:18.191 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.181+0000 7fb970f85640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb96c071510 0x7fb96c0a2c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:18.191 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.181+0000 7fb971786640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb96c0a2340 0x7fb96c0a2720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:18.191 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.181+0000 7fb970f85640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb96c071510 0x7fb96c0a2c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:47872/0 (socket says 172.21.15.42:47872) 2024-09-16T04:14:18.191 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.181+0000 7fb970f85640 1 -- 172.21.15.42:0/2500454961 learned_addr learned my addr 172.21.15.42:0/2500454961 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:14:18.192 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.181+0000 7fb970f85640 10 monclient(hunting): get_auth_request con 0x7fb96c071510 auth_method 0 2024-09-16T04:14:18.192 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.181+0000 7fb971786640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb96c0a2340 0x7fb96c0a2720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:51794/0 (socket says 172.21.15.42:51794) 2024-09-16T04:14:18.192 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.181+0000 7fb970f85640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:14:18.192 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.181+0000 7fb970f85640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:18.192 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.181+0000 7fb970f85640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:18.192 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.181+0000 7fb971786640 10 monclient(hunting): get_auth_request con 0x7fb96c0a2340 auth_method 0 2024-09-16T04:14:18.192 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.181+0000 7fb971786640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:14:18.192 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.181+0000 7fb971786640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:18.192 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.181+0000 7fb971786640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:18.192 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.181+0000 7fb970f85640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:14:18.192 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.181+0000 7fb970f85640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:14:18.192 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.181+0000 7fb970f85640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:14:18.192 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.181+0000 7fb971786640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:14:18.193 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.181+0000 7fb971786640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:14:18.193 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.181+0000 7fb971786640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:14:18.193 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.181+0000 7fb971f87640 1 --2- 172.21.15.42:0/2500454961 >> v2:172.21.15.154:3300/0 conn(0x7fb96c06f260 0x7fb96c1340f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:18.193 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.181+0000 7fb971f87640 10 monclient(hunting): get_auth_request con 0x7fb96c06f260 auth_method 0 2024-09-16T04:14:18.193 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.181+0000 7fb971f87640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:14:18.193 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.181+0000 7fb971f87640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:18.193 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.181+0000 7fb971f87640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:18.193 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.181+0000 7fb971786640 10 monclient(hunting): handle_auth_done global_id 1094201 payload 274 2024-09-16T04:14:18.193 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.181+0000 7fb971786640 1 -- 172.21.15.42:0/2500454961 >> v2:172.21.15.154:3300/0 conn(0x7fb96c06f260 msgr2=0x7fb96c1340f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:14:18.193 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.181+0000 7fb971786640 1 --2- 172.21.15.42:0/2500454961 >> v2:172.21.15.154:3300/0 conn(0x7fb96c06f260 0x7fb96c1340f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:18.193 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.181+0000 7fb971786640 1 -- 172.21.15.42:0/2500454961 >> v2:172.21.15.42:3300/0 conn(0x7fb96c071510 msgr2=0x7fb96c0a2c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:14:18.193 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.181+0000 7fb971786640 1 --2- 172.21.15.42:0/2500454961 >> v2:172.21.15.42:3300/0 conn(0x7fb96c071510 0x7fb96c0a2c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:18.193 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.181+0000 7fb971786640 10 monclient: _finish_hunting 0 2024-09-16T04:14:18.193 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.181+0000 7fb971786640 1 monclient: found mon.noname-b 2024-09-16T04:14:18.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.181+0000 7fb971786640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:14:18.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.181+0000 7fb971786640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T04:14:18.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.181+0000 7fb971786640 1 -- 172.21.15.42:0/2500454961 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb96c06efe0 con 0x7fb96c0a2340 2024-09-16T04:14:18.194 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.181+0000 7fb970f85640 1 --2- 172.21.15.42:0/2500454961 >> v2:172.21.15.42:3300/0 conn(0x7fb96c071510 0x7fb96c0a2c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:14:18.200 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.185+0000 7fb971786640 1 --2- 172.21.15.42:0/2500454961 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb96c0a2340 0x7fb96c0a2720 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fb96c06e810 tx=0x7fb96801c050 comp rx=0 tx=0).ready entity=mon.2 client_cookie=59ed8db78e314062 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:14:18.200 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.185+0000 7fb963fff640 1 -- 172.21.15.42:0/2500454961 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9680b0070 con 0x7fb96c0a2340 2024-09-16T04:14:18.200 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.185+0000 7fb963fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:18.200 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.185+0000 7fb963fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:14:18.200 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.185+0000 7fb963fff640 10 monclient: dump: 2024-09-16T04:14:18.200 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:14:18.201 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:14:18.201 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:18.201 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:18.201 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:18.201 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:18.201 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:14:18.201 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:14:18.201 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:14:18.201 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:18.201 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.185+0000 7fb963fff640 10 monclient: _finish_auth 0 2024-09-16T04:14:18.201 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.185+0000 7fb963fff640 10 monclient: _check_auth_tickets 2024-09-16T04:14:18.201 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.185+0000 7fb963fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:14:18.201 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.185+0000 7fb9739ea640 5 monclient: authenticate success, global_id 1094201 2024-09-16T04:14:18.201 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.185+0000 7fb9739ea640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:14:18.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.185+0000 7fb963fff640 1 -- 172.21.15.42:0/2500454961 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb9680ab600 con 0x7fb96c0a2340 2024-09-16T04:14:18.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.185+0000 7fb963fff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:14:18.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.185+0000 7fb9739ea640 10 monclient: get_monmap_and_config success 2024-09-16T04:14:18.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.185+0000 7fb963fff640 1 -- 172.21.15.42:0/2500454961 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9680ab9d0 con 0x7fb96c0a2340 2024-09-16T04:14:18.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.185+0000 7fb9739ea640 4 set_mon_vals no callback set 2024-09-16T04:14:18.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.185+0000 7fb963fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:18.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.185+0000 7fb963fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:14:18.202 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.185+0000 7fb963fff640 10 monclient: dump: 2024-09-16T04:14:18.202 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:14:18.202 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:14:18.202 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:18.202 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:18.202 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:18.202 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:18.203 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:14:18.203 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:14:18.203 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:14:18.203 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:18.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.185+0000 7fb9739ea640 10 monclient: shutdown 2024-09-16T04:14:18.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.185+0000 7fb9739ea640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:14:18.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.185+0000 7fb9739ea640 1 -- 172.21.15.42:0/2500454961 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb96c0a2340 msgr2=0x7fb96c0a2720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:18.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.185+0000 7fb9739ea640 1 --2- 172.21.15.42:0/2500454961 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb96c0a2340 0x7fb96c0a2720 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fb96c06e810 tx=0x7fb96801c050 comp rx=0 tx=0).stop 2024-09-16T04:14:18.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.185+0000 7fb9739ea640 1 -- 172.21.15.42:0/2500454961 shutdown_connections 2024-09-16T04:14:18.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.185+0000 7fb9739ea640 1 --2- 172.21.15.42:0/2500454961 >> v2:172.21.15.154:3300/0 conn(0x7fb96c06f260 0x7fb96c1340f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:18.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.185+0000 7fb9739ea640 1 --2- 172.21.15.42:0/2500454961 >> v2:172.21.15.42:3300/0 conn(0x7fb96c071510 0x7fb96c0a2c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:18.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.185+0000 7fb9739ea640 1 --2- 172.21.15.42:0/2500454961 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb96c0a2340 0x7fb96c0a2720 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:18.203 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.185+0000 7fb9739ea640 1 -- 172.21.15.42:0/2500454961 >> 172.21.15.42:0/2500454961 conn(0x7fb96c0a3430 msgr2=0x7fb96c0a3810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:14:18.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.185+0000 7fb9739ea640 1 -- 172.21.15.42:0/2500454961 shutdown_connections 2024-09-16T04:14:18.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.185+0000 7fb9739ea640 1 -- 172.21.15.42:0/2500454961 wait complete. 2024-09-16T04:14:18.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.185+0000 7fb9739ea640 10 monclient: build_initial_monmap 2024-09-16T04:14:18.204 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.185+0000 7fb9739ea640 10 monclient: monmap: 2024-09-16T04:14:18.204 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:14:18.204 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:14:18.204 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:14:18.204 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:14:18.204 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:14:18.204 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:18.204 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:14:18.204 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:14:18.204 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:14:18.205 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:18.205 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.185+0000 7fb9739ea640 1 librados: starting msgr at 2024-09-16T04:14:18.205 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.185+0000 7fb9739ea640 1 librados: starting objecter 2024-09-16T04:14:18.205 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.185+0000 7fb9739ea640 1 Processor -- start 2024-09-16T04:14:18.205 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.185+0000 7fb9739ea640 1 -- start start 2024-09-16T04:14:18.205 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.185+0000 7fb9739ea640 1 librados: setting wanted keys 2024-09-16T04:14:18.205 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.185+0000 7fb9739ea640 1 librados: calling monclient init 2024-09-16T04:14:18.205 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.185+0000 7fb9739ea640 10 monclient: init 2024-09-16T04:14:18.205 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.189+0000 7fb9739ea640 10 monclient: _reopen_session rank -1 2024-09-16T04:14:18.205 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.189+0000 7fb9739ea640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:14:18.205 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.189+0000 7fb9739ea640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb96c06f260 0x7fb96c1cef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:18.205 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.189+0000 7fb9739ea640 10 monclient(hunting): picked mon.noname-a con 0x7fb96c06f260 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:14:18.205 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.189+0000 7fb9739ea640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb96c071510 0x7fb96c1331b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:18.205 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.189+0000 7fb9739ea640 10 monclient(hunting): picked mon.noname-b con 0x7fb96c071510 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:14:18.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.189+0000 7fb9739ea640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb96c0a2340 0x7fb96c1336f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:18.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.189+0000 7fb9739ea640 10 monclient(hunting): picked mon.noname-c con 0x7fb96c0a2340 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:14:18.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.189+0000 7fb9739ea640 10 monclient(hunting): start opening mon connection 2024-09-16T04:14:18.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.189+0000 7fb9739ea640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fb96c13d6e0 con 0x7fb96c06f260 2024-09-16T04:14:18.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.189+0000 7fb9739ea640 10 monclient(hunting): start opening mon connection 2024-09-16T04:14:18.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.189+0000 7fb9739ea640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fb96c13d560 con 0x7fb96c071510 2024-09-16T04:14:18.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.189+0000 7fb9739ea640 10 monclient(hunting): start opening mon connection 2024-09-16T04:14:18.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.189+0000 7fb9739ea640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fb96c13d860 con 0x7fb96c0a2340 2024-09-16T04:14:18.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.189+0000 7fb9739ea640 10 monclient(hunting): _renew_subs 2024-09-16T04:14:18.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.189+0000 7fb9739ea640 10 monclient(hunting): authenticate will time out at 2905.160442s 2024-09-16T04:14:18.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.189+0000 7fb971786640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb96c06f260 0x7fb96c1cef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:18.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.189+0000 7fb971f87640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb96c0a2340 0x7fb96c1336f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:18.206 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.189+0000 7fb971786640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb96c06f260 0x7fb96c1cef00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:47874/0 (socket says 172.21.15.42:47874) 2024-09-16T04:14:18.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.189+0000 7fb971f87640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb96c0a2340 0x7fb96c1336f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:51802/0 (socket says 172.21.15.42:51802) 2024-09-16T04:14:18.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.189+0000 7fb971786640 1 -- 172.21.15.42:0/2424328004 learned_addr learned my addr 172.21.15.42:0/2424328004 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:14:18.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.189+0000 7fb971786640 10 monclient(hunting): get_auth_request con 0x7fb96c06f260 auth_method 0 2024-09-16T04:14:18.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.189+0000 7fb971786640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:14:18.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.189+0000 7fb971786640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:18.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.189+0000 7fb971786640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:18.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.189+0000 7fb971f87640 10 monclient(hunting): get_auth_request con 0x7fb96c0a2340 auth_method 0 2024-09-16T04:14:18.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.189+0000 7fb971f87640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:14:18.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.189+0000 7fb971f87640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:18.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.189+0000 7fb971f87640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:18.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.189+0000 7fb971786640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:14:18.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.189+0000 7fb971786640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:14:18.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.189+0000 7fb971786640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:14:18.207 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.189+0000 7fb971f87640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:14:18.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.189+0000 7fb971f87640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:14:18.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.189+0000 7fb971f87640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:14:18.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.189+0000 7fb970f85640 1 --2- 172.21.15.42:0/2424328004 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb96c071510 0x7fb96c1331b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:18.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.189+0000 7fb971786640 10 monclient(hunting): handle_auth_done global_id 1104123 payload 824 2024-09-16T04:14:18.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.189+0000 7fb971786640 1 -- 172.21.15.42:0/2424328004 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb96c0a2340 msgr2=0x7fb96c1336f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:18.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.189+0000 7fb971786640 1 --2- 172.21.15.42:0/2424328004 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb96c0a2340 0x7fb96c1336f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:18.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.189+0000 7fb971786640 1 -- 172.21.15.42:0/2424328004 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb96c071510 msgr2=0x7fb96c1331b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:18.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.189+0000 7fb971786640 1 --2- 172.21.15.42:0/2424328004 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb96c071510 0x7fb96c1331b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:18.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.189+0000 7fb971786640 10 monclient: _finish_hunting 0 2024-09-16T04:14:18.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.189+0000 7fb971786640 1 monclient: found mon.noname-a 2024-09-16T04:14:18.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.189+0000 7fb971786640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:14:18.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.189+0000 7fb971786640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:14:18.208 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.189+0000 7fb971786640 1 -- 172.21.15.42:0/2424328004 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb96c1205b0 con 0x7fb96c06f260 2024-09-16T04:14:18.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.189+0000 7fb970f85640 10 monclient: get_auth_request con 0x7fb96c071510 auth_method 0 2024-09-16T04:14:18.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.189+0000 7fb970f85640 1 --2- 172.21.15.42:0/2424328004 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb96c071510 0x7fb96c1331b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T04:14:18.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.189+0000 7fb971f87640 1 --2- 172.21.15.42:0/2424328004 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb96c0a2340 0x7fb96c1336f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:14:18.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.189+0000 7fb971786640 1 --2- 172.21.15.42:0/2424328004 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb96c06f260 0x7fb96c1cef00 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fb96801a700 tx=0x7fb968018df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:14:18.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.189+0000 7fb961ffb640 10 client.?.objecter ms_handle_connect 0x7fb96c06f260 2024-09-16T04:14:18.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.189+0000 7fb961ffb640 10 client.?.objecter resend_mon_ops 2024-09-16T04:14:18.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.189+0000 7fb961ffb640 1 -- 172.21.15.42:0/2424328004 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9680ba020 con 0x7fb96c06f260 2024-09-16T04:14:18.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.189+0000 7fb961ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:18.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.189+0000 7fb961ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:14:18.209 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.189+0000 7fb961ffb640 10 monclient: dump: 2024-09-16T04:14:18.209 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:14:18.209 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:14:18.209 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:18.209 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:18.210 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:18.210 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:18.210 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:14:18.210 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:14:18.210 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:14:18.210 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:18.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.189+0000 7fb961ffb640 10 monclient: _finish_auth 0 2024-09-16T04:14:18.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.189+0000 7fb961ffb640 10 monclient: _check_auth_tickets 2024-09-16T04:14:18.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.189+0000 7fb961ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:14:18.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.189+0000 7fb961ffb640 1 -- 172.21.15.42:0/2424328004 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb9680b5460 con 0x7fb96c06f260 2024-09-16T04:14:18.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.189+0000 7fb9739ea640 5 monclient: authenticate success, global_id 1104123 2024-09-16T04:14:18.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.189+0000 7fb9739ea640 10 monclient: _renew_subs 2024-09-16T04:14:18.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.189+0000 7fb9739ea640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:14:18.210 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.189+0000 7fb9739ea640 1 -- 172.21.15.42:0/2424328004 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb96c11f160 con 0x7fb96c06f260 2024-09-16T04:14:18.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.189+0000 7fb961ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:14:18.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.189+0000 7fb961ffb640 1 -- 172.21.15.42:0/2424328004 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9680b0040 con 0x7fb96c06f260 2024-09-16T04:14:18.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.189+0000 7fb961ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:18.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.189+0000 7fb961ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:14:18.211 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.189+0000 7fb961ffb640 10 monclient: dump: 2024-09-16T04:14:18.211 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:14:18.211 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:14:18.211 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:18.211 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:18.211 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:18.211 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:18.211 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:14:18.211 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:14:18.211 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:14:18.212 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:18.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.189+0000 7fb9739ea640 10 client.1104123.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:14:18.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.189+0000 7fb9739ea640 10 monclient: _renew_subs 2024-09-16T04:14:18.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.189+0000 7fb9739ea640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:14:18.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.189+0000 7fb9739ea640 1 -- 172.21.15.42:0/2424328004 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb96c120890 con 0x7fb96c06f260 2024-09-16T04:14:18.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.189+0000 7fb9637fe640 4 set_mon_vals no callback set 2024-09-16T04:14:18.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.189+0000 7fb9739ea640 1 librados: init done 2024-09-16T04:14:18.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.189+0000 7fb9739ea640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:14:18.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.189+0000 7fb9739ea640 1 --2- 172.21.15.42:0/2424328004 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb96c11cf80 0x7fb96c11d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:18.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.189+0000 7fb9739ea640 10 monclient(hunting): start opening mon connection 2024-09-16T04:14:18.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.189+0000 7fb9739ea640 1 -- 172.21.15.42:0/2424328004 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fb96c072690 con 0x7fb96c11cf80 2024-09-16T04:14:18.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.189+0000 7fb961ffb640 1 -- 172.21.15.42:0/2424328004 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fb9680bd020 con 0x7fb96c06f260 2024-09-16T04:14:18.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.189+0000 7fb970f85640 1 --2- 172.21.15.42:0/2424328004 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb96c11cf80 0x7fb96c11d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:18.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.189+0000 7fb970f85640 10 monclient: get_auth_request con 0x7fb96c11cf80 auth_method 0 2024-09-16T04:14:18.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.189+0000 7fb970f85640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:14:18.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.189+0000 7fb970f85640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:18.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.189+0000 7fb970f85640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:18.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.189+0000 7fb970f85640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:14:18.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.189+0000 7fb970f85640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:14:18.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.189+0000 7fb970f85640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:14:18.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.189+0000 7fb961ffb640 1 --2- 172.21.15.42:0/2424328004 >> v2:172.21.15.154:6812/254845342 conn(0x7fb950037810 0x7fb950039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:18.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.189+0000 7fb970f85640 10 monclient(hunting): handle_auth_done global_id 1094207 payload 824 2024-09-16T04:14:18.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.189+0000 7fb970f85640 1 -- 172.21.15.42:0/2424328004 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb96c0722d0 con 0x7fb96c11cf80 2024-09-16T04:14:18.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.189+0000 7fb961ffb640 1 -- 172.21.15.42:0/2424328004 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(710..710 src has 681..710) ==== 4681+0+0 (secure 0 0 0) 0x7fb9680e2e10 con 0x7fb96c06f260 2024-09-16T04:14:18.213 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.189+0000 7fb961ffb640 10 client.1104123.objecter ms_dispatch 0x7fb96c000ba0 osd_map(710..710 src has 681..710) 2024-09-16T04:14:18.214 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.189+0000 7fb961ffb640 3 client.1104123.objecter handle_osd_map got epochs [710,710] > 0 2024-09-16T04:14:18.214 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.189+0000 7fb961ffb640 3 client.1104123.objecter handle_osd_map decoding full epoch 710 2024-09-16T04:14:18.214 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.189+0000 7fb961ffb640 20 client.1104123.objecter dump_active .. 0 homeless 2024-09-16T04:14:18.214 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.189+0000 7fb961ffb640 10 client.1104123.objecter ms_handle_connect 0x7fb96c11cf80 2024-09-16T04:14:18.214 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.189+0000 7fb970f85640 1 --2- 172.21.15.42:0/2424328004 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb96c11cf80 0x7fb96c11d360 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fb96c1207c0 tx=0x7fb95c010c20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:14:18.214 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.189+0000 7fb961ffb640 10 client.1104123.objecter resend_mon_ops 2024-09-16T04:14:18.214 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.189+0000 7fb961ffb640 1 -- 172.21.15.42:0/2424328004 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb95c00f040 con 0x7fb96c11cf80 2024-09-16T04:14:18.214 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.189+0000 7fb961ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:18.214 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.189+0000 7fb961ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:14:18.214 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.189+0000 7fb961ffb640 10 monclient: dump: 2024-09-16T04:14:18.214 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:14:18.214 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:14:18.214 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:18.214 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:18.215 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:18.215 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:18.215 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:14:18.215 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:14:18.215 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:14:18.215 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:18.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.189+0000 7fb971f87640 1 --2- 172.21.15.42:0/2424328004 >> v2:172.21.15.154:6812/254845342 conn(0x7fb950037810 0x7fb950039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:18.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.189+0000 7fb971f87640 10 monclient: get_auth_request con 0x7fb950037810 auth_method 0 2024-09-16T04:14:18.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.189+0000 7fb961ffb640 1 -- 172.21.15.42:0/2424328004 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fb95c01c400 con 0x7fb96c11cf80 2024-09-16T04:14:18.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.189+0000 7fb961ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:14:18.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.189+0000 7fb961ffb640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:14:18.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.189+0000 7fb961ffb640 1 -- 172.21.15.42:0/2424328004 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb96c11cf80 msgr2=0x7fb96c11d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:18.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.189+0000 7fb961ffb640 1 --2- 172.21.15.42:0/2424328004 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb96c11cf80 0x7fb96c11d360 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fb96c1207c0 tx=0x7fb95c010c20 comp rx=0 tx=0).stop 2024-09-16T04:14:18.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.189+0000 7fb961ffb640 1 -- 172.21.15.42:0/2424328004 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb96c11cf80 msgr2=0x7fb96c11d360 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:14:18.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.189+0000 7fb961ffb640 1 --2- 172.21.15.42:0/2424328004 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb96c11cf80 0x7fb96c11d360 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:18.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.189+0000 7fb971f87640 1 --2- 172.21.15.42:0/2424328004 >> v2:172.21.15.154:6812/254845342 conn(0x7fb950037810 0x7fb950039cd0 secure :-1 s=READY pgs=4007 cs=0 l=1 rev1=1 crypto rx=0x7fb96c071ac0 tx=0x7fb964009820 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:14:18.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.189+0000 7fb961ffb640 10 client.1104123.objecter ms_handle_connect 0x7fb950037810 2024-09-16T04:14:18.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.197+0000 7fb9739ea640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:14:18.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.197+0000 7fb9739ea640 1 --2- 172.21.15.42:0/2424328004 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb96c11d8a0 0x7fb96c0a4a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:18.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.197+0000 7fb9739ea640 10 monclient(hunting): start opening mon connection 2024-09-16T04:14:18.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.197+0000 7fb9739ea640 1 -- 172.21.15.42:0/2424328004 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fb96c1340f0 con 0x7fb96c11d8a0 2024-09-16T04:14:18.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.197+0000 7fb970f85640 1 --2- 172.21.15.42:0/2424328004 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb96c11d8a0 0x7fb96c0a4a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:18.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.197+0000 7fb970f85640 10 monclient: get_auth_request con 0x7fb96c11d8a0 auth_method 0 2024-09-16T04:14:18.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.197+0000 7fb970f85640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:14:18.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.197+0000 7fb970f85640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:18.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.197+0000 7fb970f85640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:18.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.197+0000 7fb970f85640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:14:18.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.197+0000 7fb970f85640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:14:18.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.197+0000 7fb970f85640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:14:18.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.197+0000 7fb970f85640 10 monclient(hunting): handle_auth_done global_id 1094210 payload 824 2024-09-16T04:14:18.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.197+0000 7fb970f85640 1 -- 172.21.15.42:0/2424328004 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fb96c06b370 con 0x7fb96c11d8a0 2024-09-16T04:14:18.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.197+0000 7fb970f85640 1 --2- 172.21.15.42:0/2424328004 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb96c11d8a0 0x7fb96c0a4a30 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fb95c01b000 tx=0x7fb95c0046d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:14:18.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.197+0000 7fb961ffb640 10 client.1104123.objecter ms_handle_connect 0x7fb96c11d8a0 2024-09-16T04:14:18.219 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.197+0000 7fb961ffb640 10 client.1104123.objecter resend_mon_ops 2024-09-16T04:14:18.220 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.197+0000 7fb961ffb640 1 -- 172.21.15.42:0/2424328004 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb95c01c9c0 con 0x7fb96c11d8a0 2024-09-16T04:14:18.220 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.197+0000 7fb961ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:18.220 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.197+0000 7fb961ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:14:18.220 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.197+0000 7fb961ffb640 10 monclient: dump: 2024-09-16T04:14:18.220 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:14:18.220 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:14:18.220 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:18.220 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:18.220 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:18.220 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:18.220 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:14:18.220 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:14:18.220 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:14:18.220 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:18.221 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.197+0000 7fb961ffb640 1 -- 172.21.15.42:0/2424328004 <== mon.2 v2:172.21.15.42:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4712 (secure 0 0 0) 0x7fb95c021570 con 0x7fb96c11d8a0 2024-09-16T04:14:18.221 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.197+0000 7fb961ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:14:18.221 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.197+0000 7fb961ffb640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:14:18.221 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.197+0000 7fb961ffb640 1 -- 172.21.15.42:0/2424328004 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb96c11d8a0 msgr2=0x7fb96c0a4a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:18.221 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.197+0000 7fb961ffb640 1 --2- 172.21.15.42:0/2424328004 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb96c11d8a0 0x7fb96c0a4a30 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fb95c01b000 tx=0x7fb95c0046d0 comp rx=0 tx=0).stop 2024-09-16T04:14:18.221 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.197+0000 7fb961ffb640 1 -- 172.21.15.42:0/2424328004 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb96c11d8a0 msgr2=0x7fb96c0a4a30 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:14:18.221 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.197+0000 7fb961ffb640 1 --2- 172.21.15.42:0/2424328004 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb96c11d8a0 0x7fb96c0a4a30 secure :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fb95c01b000 tx=0x7fb95c0046d0 comp rx=0 tx=0).stop 2024-09-16T04:14:18.221 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T04:14:18.221 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:14:18.221 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:14:18.222 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T04:14:18.222 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 292, 2024-09-16T04:14:18.222 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 57239, 2024-09-16T04:14:18.222 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T04:14:18.222 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T04:14:18.222 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T04:14:18.222 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T04:14:18.222 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:14:18.222 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 1, 2024-09-16T04:14:18.222 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:14:18.222 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T04:14:18.222 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T04:14:18.222 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:14:18.222 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:14:18.223 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:14:18.223 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:14:18.223 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:14:18.223 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:14:18.223 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:14:18.223 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:14:18.223 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:14:18.223 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:14:18.223 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:14:18.223 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:14:18.223 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:14:18.223 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T04:14:18.223 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:14:18.223 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:14:18.223 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:14:18.224 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:14:18.224 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:14:18.224 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:14:18.224 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:14:18.224 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:14:18.224 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:14:18.224 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:14:18.224 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:14:18.224 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:14:18.224 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:14:18.224 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T04:14:18.224 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T04:14:18.224 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T04:14:18.224 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T04:14:18.225 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T04:14:18.225 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T04:14:18.225 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:14:18.225 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:14:18.225 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T04:14:18.225 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T04:14:18.225 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T04:14:18.225 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T04:14:18.225 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T04:14:18.225 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T04:14:18.225 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T04:14:18.225 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:14:18.225 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T04:14:18.225 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:14:18.226 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:14:18.226 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:14:18.226 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:14:18.226 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:14:18.226 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:14:18.226 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:14:18.226 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:14:18.226 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:14:18.226 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:14:18.226 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:14:18.226 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:14:18.226 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T04:14:18.226 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:14:18.226 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T04:14:18.227 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:14:18.227 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:14:18.227 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:14:18.227 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:14:18.227 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:14:18.227 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:14:18.227 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:14:18.227 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T04:14:18.227 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:14:18.227 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:14:18.227 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:14:18.227 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:14:18.227 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T04:14:18.227 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:14:18.228 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:14:18.228 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:14:18.228 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:14:18.228 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T04:14:18.228 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T04:14:18.228 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:14:18.228 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:14:18.228 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:14:18.228 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:14:18.228 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:14:18.228 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:14:18.228 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:14:18.228 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:14:18.228 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:14:18.229 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:14:18.229 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:14:18.229 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T04:14:18.229 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:14:18.229 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:14:18.229 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:14:18.229 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:14:18.229 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T04:14:18.229 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:14:18.229 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:14:18.229 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:14:18.230 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:14:18.230 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T04:14:18.230 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T04:14:18.230 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:14:18.230 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:14:18.230 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:14:18.230 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:14:18.230 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:14:18.230 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:14:18.230 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:14:18.230 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:14:18.230 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:14:18.230 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:14:18.230 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:14:18.231 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T04:14:18.231 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:14:18.231 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:14:18.231 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:14:18.231 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:14:18.231 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T04:14:18.231 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:14:18.231 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:14:18.231 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:14:18.231 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:14:18.231 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T04:14:18.232 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T04:14:18.232 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:14:18.232 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:14:18.232 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:14:18.232 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:14:18.232 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:14:18.232 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:14:18.232 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T04:14:18.232 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T04:14:18.233 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:14:18.233 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:14:18.233 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:14:18.233 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:14:18.233 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:14:18.233 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:14:18.233 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T04:14:18.233 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:14:18.233 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:14:18.233 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:14:18.233 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 5 2024-09-16T04:14:18.233 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:14:18.233 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:14:18.233 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T04:14:18.233 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:14:18.234 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:14:18.234 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:14:18.234 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 3 2024-09-16T04:14:18.234 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:14:18.234 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:14:18.234 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:14:18.234 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T04:14:18.234 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T04:14:18.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.197+0000 7fb9739ea640 10 librados: watch_flush enter 2024-09-16T04:14:18.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.197+0000 7fb9739ea640 10 librados: watch_flush exit 2024-09-16T04:14:18.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.197+0000 7fb9739ea640 20 client.1104123.objecter shutdown clearing up homeless session... 2024-09-16T04:14:18.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.197+0000 7fb9739ea640 10 client.1104123.objecter successfully canceled tick 2024-09-16T04:14:18.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.197+0000 7fb9739ea640 1 -- 172.21.15.42:0/2424328004 >> v2:172.21.15.154:6812/254845342 conn(0x7fb950037810 msgr2=0x7fb950039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:18.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.197+0000 7fb9739ea640 1 --2- 172.21.15.42:0/2424328004 >> v2:172.21.15.154:6812/254845342 conn(0x7fb950037810 0x7fb950039cd0 secure :-1 s=READY pgs=4007 cs=0 l=1 rev1=1 crypto rx=0x7fb96c071ac0 tx=0x7fb964009820 comp rx=0 tx=0).stop 2024-09-16T04:14:18.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.197+0000 7fb9739ea640 10 monclient: shutdown 2024-09-16T04:14:18.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.197+0000 7fb9739ea640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:14:18.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.197+0000 7fb9739ea640 1 -- 172.21.15.42:0/2424328004 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb96c06f260 msgr2=0x7fb96c1cef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:18.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.197+0000 7fb971f87640 1 -- 172.21.15.42:0/2424328004 reap_dead start 2024-09-16T04:14:18.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.197+0000 7fb9739ea640 1 --2- 172.21.15.42:0/2424328004 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb96c06f260 0x7fb96c1cef00 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fb96801a700 tx=0x7fb968018df0 comp rx=0 tx=0).stop 2024-09-16T04:14:18.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.197+0000 7fb9739ea640 1 -- 172.21.15.42:0/2424328004 shutdown_connections 2024-09-16T04:14:18.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.197+0000 7fb9739ea640 1 --2- 172.21.15.42:0/2424328004 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb96c06f260 0x7fb96c1cef00 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:18.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.197+0000 7fb9739ea640 1 -- 172.21.15.42:0/2424328004 >> 172.21.15.42:0/2424328004 conn(0x7fb96c0a3430 msgr2=0x7fb96c06bc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:14:18.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.197+0000 7fb9739ea640 1 -- 172.21.15.42:0/2424328004 shutdown_connections 2024-09-16T04:14:18.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.197+0000 7fb9739ea640 1 -- 172.21.15.42:0/2424328004 wait complete. 2024-09-16T04:14:18.237 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.197+0000 7fb9739ea640 1 librados: shutdown 2024-09-16T04:14:18.237 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:14:18.237 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-16T04:14:18.238 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-16T04:14:18.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.341+0000 7f9e7270c640 10 monclient: get_monmap_and_config 2024-09-16T04:14:18.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.341+0000 7f9e7270c640 10 monclient: build_initial_monmap 2024-09-16T04:14:18.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.341+0000 7f9e7270c640 10 monclient: monmap: 2024-09-16T04:14:18.346 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:14:18.346 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:14:18.346 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:14:18.344432+0000 2024-09-16T04:14:18.346 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:14:18.344432+0000 2024-09-16T04:14:18.346 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:14:18.346 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:18.346 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:14:18.346 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:14:18.346 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:14:18.346 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:14:18.346 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:14:18.346 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:18.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.341+0000 7f9e7270c640 1 Processor -- start 2024-09-16T04:14:18.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.341+0000 7f9e7270c640 1 -- start start 2024-09-16T04:14:18.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.341+0000 7f9e7270c640 10 monclient: init 2024-09-16T04:14:18.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.341+0000 7f9e7270c640 10 monclient: _reopen_session rank -1 2024-09-16T04:14:18.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.341+0000 7f9e7270c640 10 monclient: _add_conns ranks=[0,4,2,3,1] 2024-09-16T04:14:18.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.341+0000 7f9e7270c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9e6c0a1330 addr v1:172.21.15.42:6789/0 2024-09-16T04:14:18.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.341+0000 7f9e7270c640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f9e6c06dac0 0x7f9e6c13a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:18.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.341+0000 7f9e7270c640 10 monclient(hunting): picked mon.noname-c con 0x7f9e6c06dac0 addr v2:172.21.15.154:3300/0 2024-09-16T04:14:18.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.341+0000 7f9e7270c640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9e6c0a1e20 0x7f9e6c13c890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:18.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.341+0000 7f9e7270c640 10 monclient(hunting): picked mon.noname-b con 0x7f9e6c0a1e20 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:14:18.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.341+0000 7f9e7270c640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9e6c09c090 con 0x7f9e6c0a1330 2024-09-16T04:14:18.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.341+0000 7f9e7270c640 10 monclient(hunting): start opening mon connection 2024-09-16T04:14:18.349 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.341+0000 7f9e7270c640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f9e6c09c890 con 0x7f9e6c06dac0 2024-09-16T04:14:18.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.341+0000 7f9e7270c640 10 monclient(hunting): start opening mon connection 2024-09-16T04:14:18.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.341+0000 7f9e7270c640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f9e6c09f610 con 0x7f9e6c0a1e20 2024-09-16T04:14:18.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.341+0000 7f9e7270c640 10 monclient(hunting): _renew_subs 2024-09-16T04:14:18.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.341+0000 7f9e7270c640 10 monclient(hunting): authenticate will time out at 2905.313874s 2024-09-16T04:14:18.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.341+0000 7f9e6bfff640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f9e6c0a1330 0x7f9e6c0a1710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:52936/0 (socket says 172.21.15.42:52936) 2024-09-16T04:14:18.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.341+0000 7f9e6bfff640 1 -- 172.21.15.42:0/4249045651 learned_addr learned my addr 172.21.15.42:0/4249045651 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T04:14:18.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.341+0000 7f9e70ca9640 1 --2- 172.21.15.42:0/4249045651 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9e6c0a1e20 0x7f9e6c13c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:18.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.341+0000 7f9e70ca9640 10 monclient(hunting): get_auth_request con 0x7f9e6c0a1e20 auth_method 0 2024-09-16T04:14:18.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.341+0000 7f9e70ca9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:14:18.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.341+0000 7f9e70ca9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:18.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.341+0000 7f9e70ca9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:18.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.341+0000 7f9e6b7fe640 1 --2- 172.21.15.42:0/4249045651 >> v2:172.21.15.154:3300/0 conn(0x7f9e6c06dac0 0x7f9e6c13a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:18.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.341+0000 7f9e6b7fe640 10 monclient(hunting): get_auth_request con 0x7f9e6c06dac0 auth_method 0 2024-09-16T04:14:18.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.341+0000 7f9e6b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:14:18.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.341+0000 7f9e6b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:18.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.341+0000 7f9e6b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:18.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.341+0000 7f9e70ca9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:14:18.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.341+0000 7f9e70ca9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:14:18.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.341+0000 7f9e70ca9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:14:18.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.341+0000 7f9e6b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:14:18.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.341+0000 7f9e6b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:14:18.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.341+0000 7f9e6b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:14:18.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.341+0000 7f9e6affd640 1 -- 172.21.15.42:0/4249045651 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3479931032 0 0) 0x7f9e6c09c090 con 0x7f9e6c0a1330 2024-09-16T04:14:18.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.341+0000 7f9e6affd640 10 monclient(hunting): _negotiate 2024-09-16T04:14:18.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.341+0000 7f9e6affd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:18.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.341+0000 7f9e6affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:18.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.341+0000 7f9e6affd640 10 monclient(hunting): my global_id is 1104126 2024-09-16T04:14:18.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.341+0000 7f9e6affd640 1 -- 172.21.15.42:0/4249045651 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9e58003530 con 0x7f9e6c0a1330 2024-09-16T04:14:18.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.341+0000 7f9e70ca9640 10 monclient(hunting): handle_auth_done global_id 1094213 payload 274 2024-09-16T04:14:18.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.341+0000 7f9e70ca9640 1 -- 172.21.15.42:0/4249045651 >> v2:172.21.15.154:3300/0 conn(0x7f9e6c06dac0 msgr2=0x7f9e6c13a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:14:18.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.341+0000 7f9e70ca9640 1 --2- 172.21.15.42:0/4249045651 >> v2:172.21.15.154:3300/0 conn(0x7f9e6c06dac0 0x7f9e6c13a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:18.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.341+0000 7f9e70ca9640 1 -- 172.21.15.42:0/4249045651 >> v1:172.21.15.42:6789/0 conn(0x7f9e6c0a1330 legacy=0x7f9e6c0a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:18.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.341+0000 7f9e70ca9640 10 monclient: _finish_hunting 0 2024-09-16T04:14:18.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.341+0000 7f9e70ca9640 1 monclient: found mon.noname-b 2024-09-16T04:14:18.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.341+0000 7f9e70ca9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:14:18.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.341+0000 7f9e70ca9640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T04:14:18.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.341+0000 7f9e70ca9640 1 -- 172.21.15.42:0/4249045651 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e6c09f8f0 con 0x7f9e6c0a1e20 2024-09-16T04:14:18.361 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.345+0000 7f9e70ca9640 1 --2- 172.21.15.42:0/4249045651 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9e6c0a1e20 0x7f9e6c13c890 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f9e5c0097c0 tx=0x7f9e5c01c3c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ba273c6d63464c4a server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:14:18.361 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.345+0000 7f9e6affd640 1 -- 172.21.15.42:0/4249045651 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e5c0b1070 con 0x7f9e6c0a1e20 2024-09-16T04:14:18.361 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.345+0000 7f9e6affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:18.361 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.345+0000 7f9e6affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:14:18.361 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.345+0000 7f9e6affd640 10 monclient: dump: 2024-09-16T04:14:18.361 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:14:18.361 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:14:18.361 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:18.361 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:18.361 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:18.361 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:18.361 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:14:18.361 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:14:18.362 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:14:18.362 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:18.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.345+0000 7f9e6affd640 10 monclient: _finish_auth 0 2024-09-16T04:14:18.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.345+0000 7f9e6affd640 10 monclient: _check_auth_tickets 2024-09-16T04:14:18.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.345+0000 7f9e6affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:14:18.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.345+0000 7f9e6affd640 1 -- 172.21.15.42:0/4249045651 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9e5c0a8050 con 0x7f9e6c0a1e20 2024-09-16T04:14:18.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.345+0000 7f9e7270c640 5 monclient: authenticate success, global_id 1094213 2024-09-16T04:14:18.362 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.345+0000 7f9e7270c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:14:18.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.345+0000 7f9e6affd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:14:18.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.345+0000 7f9e7270c640 10 monclient: get_monmap_and_config success 2024-09-16T04:14:18.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.345+0000 7f9e6affd640 1 -- 172.21.15.42:0/4249045651 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e5c0ac860 con 0x7f9e6c0a1e20 2024-09-16T04:14:18.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.345+0000 7f9e6affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:18.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.345+0000 7f9e7270c640 4 set_mon_vals no callback set 2024-09-16T04:14:18.363 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.345+0000 7f9e6affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:14:18.364 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.345+0000 7f9e6affd640 10 monclient: dump: 2024-09-16T04:14:18.364 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:14:18.364 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:14:18.364 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:18.364 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:18.365 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:18.365 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:18.365 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:14:18.365 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:14:18.365 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:14:18.365 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:18.365 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.345+0000 7f9e7270c640 10 monclient: shutdown 2024-09-16T04:14:18.366 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.345+0000 7f9e7270c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:14:18.366 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.345+0000 7f9e7270c640 1 -- 172.21.15.42:0/4249045651 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9e6c0a1e20 msgr2=0x7f9e6c13c890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:18.366 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.345+0000 7f9e7270c640 1 --2- 172.21.15.42:0/4249045651 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9e6c0a1e20 0x7f9e6c13c890 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f9e5c0097c0 tx=0x7f9e5c01c3c0 comp rx=0 tx=0).stop 2024-09-16T04:14:18.366 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.345+0000 7f9e7270c640 1 -- 172.21.15.42:0/4249045651 shutdown_connections 2024-09-16T04:14:18.366 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.345+0000 7f9e7270c640 1 --2- 172.21.15.42:0/4249045651 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9e6c0a1e20 0x7f9e6c13c890 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:18.366 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.345+0000 7f9e7270c640 1 --2- 172.21.15.42:0/4249045651 >> v2:172.21.15.154:3300/0 conn(0x7f9e6c06dac0 0x7f9e6c13a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:18.366 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.345+0000 7f9e7270c640 1 -- 172.21.15.42:0/4249045651 >> 172.21.15.42:0/4249045651 conn(0x7f9e6c0a2420 msgr2=0x7f9e6c0a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:14:18.367 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.345+0000 7f9e7270c640 1 -- 172.21.15.42:0/4249045651 shutdown_connections 2024-09-16T04:14:18.367 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.345+0000 7f9e7270c640 1 -- 172.21.15.42:0/4249045651 wait complete. 2024-09-16T04:14:18.367 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.345+0000 7f9e7270c640 10 monclient: build_initial_monmap 2024-09-16T04:14:18.367 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.345+0000 7f9e7270c640 10 monclient: monmap: 2024-09-16T04:14:18.367 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:14:18.367 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:14:18.367 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:14:18.368 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:14:18.368 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:14:18.368 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:18.368 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:14:18.368 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:14:18.368 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:14:18.368 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:18.369 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.345+0000 7f9e7270c640 1 librados: starting msgr at 2024-09-16T04:14:18.369 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.345+0000 7f9e7270c640 1 librados: starting objecter 2024-09-16T04:14:18.369 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.345+0000 7f9e7270c640 1 Processor -- start 2024-09-16T04:14:18.369 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.345+0000 7f9e7270c640 1 -- start start 2024-09-16T04:14:18.369 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.345+0000 7f9e7270c640 1 librados: setting wanted keys 2024-09-16T04:14:18.369 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.345+0000 7f9e7270c640 1 librados: calling monclient init 2024-09-16T04:14:18.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.345+0000 7f9e7270c640 10 monclient: init 2024-09-16T04:14:18.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.349+0000 7f9e7270c640 10 monclient: _reopen_session rank -1 2024-09-16T04:14:18.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.349+0000 7f9e7270c640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:14:18.370 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.349+0000 7f9e7270c640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9e6c0a1330 0x7f9e6c1ccb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:18.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.349+0000 7f9e7270c640 10 monclient(hunting): picked mon.noname-c con 0x7f9e6c0a1330 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:14:18.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.349+0000 7f9e7270c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9e6c0a1e20 0x7f9e6c1352e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:18.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.349+0000 7f9e7270c640 10 monclient(hunting): picked mon.noname-a con 0x7f9e6c0a1e20 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:14:18.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.349+0000 7f9e7270c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9e6c135820 0x7f9e6c11bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:18.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.349+0000 7f9e7270c640 10 monclient(hunting): picked mon.noname-b con 0x7f9e6c135820 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:14:18.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.349+0000 7f9e7270c640 10 monclient(hunting): start opening mon connection 2024-09-16T04:14:18.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.349+0000 7f9e7270c640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f9e6c13f770 con 0x7f9e6c0a1e20 2024-09-16T04:14:18.371 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.349+0000 7f9e7270c640 10 monclient(hunting): start opening mon connection 2024-09-16T04:14:18.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.349+0000 7f9e7270c640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f9e6c13f5f0 con 0x7f9e6c135820 2024-09-16T04:14:18.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.349+0000 7f9e7270c640 10 monclient(hunting): start opening mon connection 2024-09-16T04:14:18.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.349+0000 7f9e7270c640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f9e6c13f8f0 con 0x7f9e6c0a1330 2024-09-16T04:14:18.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.349+0000 7f9e7270c640 10 monclient(hunting): _renew_subs 2024-09-16T04:14:18.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.349+0000 7f9e7270c640 10 monclient(hunting): authenticate will time out at 2905.320179s 2024-09-16T04:14:18.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.349+0000 7f9e6bfff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9e6c0a1330 0x7f9e6c1ccb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:18.372 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.349+0000 7f9e6b7fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9e6c0a1e20 0x7f9e6c1352e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:18.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.349+0000 7f9e6b7fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9e6c0a1e20 0x7f9e6c1352e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:47882/0 (socket says 172.21.15.42:47882) 2024-09-16T04:14:18.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.349+0000 7f9e6bfff640 10 monclient(hunting): get_auth_request con 0x7f9e6c0a1330 auth_method 0 2024-09-16T04:14:18.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.349+0000 7f9e6b7fe640 1 -- 172.21.15.42:0/3802209763 learned_addr learned my addr 172.21.15.42:0/3802209763 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:14:18.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.349+0000 7f9e6bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:14:18.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.349+0000 7f9e6bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:18.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.349+0000 7f9e6bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:18.373 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.349+0000 7f9e6b7fe640 10 monclient(hunting): get_auth_request con 0x7f9e6c0a1e20 auth_method 0 2024-09-16T04:14:18.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.349+0000 7f9e6b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:14:18.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.349+0000 7f9e6b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:18.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.349+0000 7f9e6b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:18.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.349+0000 7f9e6bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:14:18.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.349+0000 7f9e6bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:14:18.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.349+0000 7f9e6bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:14:18.374 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.349+0000 7f9e6b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:14:18.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.349+0000 7f9e6b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:14:18.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.349+0000 7f9e6b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:14:18.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.349+0000 7f9e70ca9640 1 --2- 172.21.15.42:0/3802209763 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9e6c135820 0x7f9e6c11bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:18.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.349+0000 7f9e6bfff640 10 monclient(hunting): handle_auth_done global_id 1094216 payload 824 2024-09-16T04:14:18.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.349+0000 7f9e6bfff640 1 -- 172.21.15.42:0/3802209763 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9e6c135820 msgr2=0x7f9e6c11bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:18.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.349+0000 7f9e6bfff640 1 --2- 172.21.15.42:0/3802209763 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9e6c135820 0x7f9e6c11bf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:18.375 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.349+0000 7f9e6bfff640 1 -- 172.21.15.42:0/3802209763 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9e6c0a1e20 msgr2=0x7f9e6c1352e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:18.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.349+0000 7f9e6bfff640 1 --2- 172.21.15.42:0/3802209763 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9e6c0a1e20 0x7f9e6c1352e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:18.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.349+0000 7f9e6bfff640 10 monclient: _finish_hunting 0 2024-09-16T04:14:18.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.349+0000 7f9e6bfff640 1 monclient: found mon.noname-c 2024-09-16T04:14:18.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.349+0000 7f9e6bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:14:18.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.349+0000 7f9e6bfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T04:14:18.376 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.349+0000 7f9e6bfff640 1 -- 172.21.15.42:0/3802209763 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e6c06e280 con 0x7f9e6c0a1330 2024-09-16T04:14:18.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.349+0000 7f9e70ca9640 10 monclient: get_auth_request con 0x7f9e6c135820 auth_method 0 2024-09-16T04:14:18.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.349+0000 7f9e70ca9640 1 --2- 172.21.15.42:0/3802209763 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9e6c135820 0x7f9e6c11bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T04:14:18.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.349+0000 7f9e6b7fe640 1 --2- 172.21.15.42:0/3802209763 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f9e6c0a1e20 0x7f9e6c1352e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:14:18.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.349+0000 7f9e6bfff640 1 --2- 172.21.15.42:0/3802209763 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9e6c0a1330 0x7f9e6c1ccb30 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f9e6400cb10 tx=0x7f9e6400a060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:14:18.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.349+0000 7f9e68ff9640 10 client.?.objecter ms_handle_connect 0x7f9e6c0a1330 2024-09-16T04:14:18.377 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.349+0000 7f9e68ff9640 10 client.?.objecter resend_mon_ops 2024-09-16T04:14:18.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.349+0000 7f9e68ff9640 1 -- 172.21.15.42:0/3802209763 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e64017070 con 0x7f9e6c0a1330 2024-09-16T04:14:18.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.349+0000 7f9e68ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:18.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.349+0000 7f9e68ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:14:18.378 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.349+0000 7f9e68ff9640 10 monclient: dump: 2024-09-16T04:14:18.378 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:14:18.379 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:14:18.379 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:18.379 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:18.379 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:18.379 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:18.379 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:14:18.380 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:14:18.380 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:14:18.380 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:18.380 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.349+0000 7f9e68ff9640 10 monclient: _finish_auth 0 2024-09-16T04:14:18.380 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.349+0000 7f9e68ff9640 10 monclient: _check_auth_tickets 2024-09-16T04:14:18.380 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.349+0000 7f9e68ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:14:18.380 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.349+0000 7f9e68ff9640 1 -- 172.21.15.42:0/3802209763 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9e6400ac10 con 0x7f9e6c0a1330 2024-09-16T04:14:18.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.349+0000 7f9e7270c640 5 monclient: authenticate success, global_id 1094216 2024-09-16T04:14:18.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.349+0000 7f9e7270c640 10 monclient: _renew_subs 2024-09-16T04:14:18.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.349+0000 7f9e7270c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:14:18.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.349+0000 7f9e7270c640 1 -- 172.21.15.42:0/3802209763 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e6c11f990 con 0x7f9e6c0a1330 2024-09-16T04:14:18.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.349+0000 7f9e68ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T04:14:18.381 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.349+0000 7f9e68ff9640 1 -- 172.21.15.42:0/3802209763 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e64012400 con 0x7f9e6c0a1330 2024-09-16T04:14:18.382 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.349+0000 7f9e68ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:18.382 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.349+0000 7f9e68ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:14:18.382 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.349+0000 7f9e68ff9640 10 monclient: dump: 2024-09-16T04:14:18.382 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:14:18.382 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:14:18.382 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:18.382 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:18.383 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:18.383 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:18.383 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:14:18.383 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:14:18.383 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:14:18.383 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:18.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.349+0000 7f9e6a7fc640 4 set_mon_vals no callback set 2024-09-16T04:14:18.383 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.349+0000 7f9e7270c640 10 client.1094216.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:14:18.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.349+0000 7f9e7270c640 10 monclient: _renew_subs 2024-09-16T04:14:18.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.349+0000 7f9e7270c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:14:18.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.349+0000 7f9e7270c640 1 -- 172.21.15.42:0/3802209763 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e6c13c380 con 0x7f9e6c0a1330 2024-09-16T04:14:18.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.349+0000 7f9e7270c640 1 librados: init done 2024-09-16T04:14:18.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.349+0000 7f9e7270c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:14:18.384 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.349+0000 7f9e7270c640 1 --2- 172.21.15.42:0/3802209763 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9e6c06dac0 0x7f9e6c02c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:18.385 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.349+0000 7f9e7270c640 10 monclient(hunting): start opening mon connection 2024-09-16T04:14:18.385 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.349+0000 7f9e7270c640 1 -- 172.21.15.42:0/3802209763 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f9e6c09c0f0 con 0x7f9e6c06dac0 2024-09-16T04:14:18.385 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.349+0000 7f9e68ff9640 1 -- 172.21.15.42:0/3802209763 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f9e6401a070 con 0x7f9e6c0a1330 2024-09-16T04:14:18.385 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.349+0000 7f9e68ff9640 1 --2- 172.21.15.42:0/3802209763 >> v2:172.21.15.154:6812/254845342 conn(0x7f9e4c037810 0x7f9e4c039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:18.385 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.349+0000 7f9e6b7fe640 1 --2- 172.21.15.42:0/3802209763 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9e6c06dac0 0x7f9e6c02c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:18.385 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.349+0000 7f9e6b7fe640 10 monclient: get_auth_request con 0x7f9e6c06dac0 auth_method 0 2024-09-16T04:14:18.386 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.349+0000 7f9e6b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:14:18.386 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.349+0000 7f9e6b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:18.386 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.349+0000 7f9e6b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:18.386 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.349+0000 7f9e68ff9640 1 -- 172.21.15.42:0/3802209763 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(710..710 src has 681..710) ==== 4681+0+0 (secure 0 0 0) 0x7f9e64048800 con 0x7f9e6c0a1330 2024-09-16T04:14:18.386 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.349+0000 7f9e68ff9640 10 client.1094216.objecter ms_dispatch 0x7f9e6c000ba0 osd_map(710..710 src has 681..710) 2024-09-16T04:14:18.386 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.349+0000 7f9e68ff9640 3 client.1094216.objecter handle_osd_map got epochs [710,710] > 0 2024-09-16T04:14:18.387 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.349+0000 7f9e68ff9640 3 client.1094216.objecter handle_osd_map decoding full epoch 710 2024-09-16T04:14:18.387 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.349+0000 7f9e68ff9640 20 client.1094216.objecter dump_active .. 0 homeless 2024-09-16T04:14:18.387 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.349+0000 7f9e6b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:14:18.387 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.349+0000 7f9e6b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:14:18.387 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.349+0000 7f9e6b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:14:18.387 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.349+0000 7f9e70ca9640 1 --2- 172.21.15.42:0/3802209763 >> v2:172.21.15.154:6812/254845342 conn(0x7f9e4c037810 0x7f9e4c039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:18.388 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.349+0000 7f9e70ca9640 10 monclient: get_auth_request con 0x7f9e4c037810 auth_method 0 2024-09-16T04:14:18.388 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.349+0000 7f9e6b7fe640 10 monclient(hunting): handle_auth_done global_id 1074376 payload 824 2024-09-16T04:14:18.388 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.349+0000 7f9e6b7fe640 1 -- 172.21.15.42:0/3802209763 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9e6c0a1710 con 0x7f9e6c06dac0 2024-09-16T04:14:18.388 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.349+0000 7f9e6b7fe640 1 --2- 172.21.15.42:0/3802209763 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9e6c06dac0 0x7f9e6c02c650 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f9e6c13c320 tx=0x7f9e5400f610 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:14:18.388 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.349+0000 7f9e68ff9640 10 client.1094216.objecter ms_handle_connect 0x7f9e6c06dac0 2024-09-16T04:14:18.388 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.349+0000 7f9e68ff9640 10 client.1094216.objecter resend_mon_ops 2024-09-16T04:14:18.388 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.349+0000 7f9e68ff9640 10 client.1094216.objecter ms_handle_connect 0x7f9e4c037810 2024-09-16T04:14:18.388 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.349+0000 7f9e70ca9640 1 --2- 172.21.15.42:0/3802209763 >> v2:172.21.15.154:6812/254845342 conn(0x7f9e4c037810 0x7f9e4c039cd0 secure :-1 s=READY pgs=4008 cs=0 l=1 rev1=1 crypto rx=0x7f9e5c0ae000 tx=0x7f9e5c0a7000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:14:18.388 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.349+0000 7f9e68ff9640 1 -- 172.21.15.42:0/3802209763 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e54004020 con 0x7f9e6c06dac0 2024-09-16T04:14:18.388 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.349+0000 7f9e68ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:18.388 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.349+0000 7f9e68ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:14:18.388 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.349+0000 7f9e68ff9640 10 monclient: dump: 2024-09-16T04:14:18.388 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:14:18.389 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:14:18.389 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:18.389 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:18.389 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:18.389 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:18.389 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:14:18.389 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:14:18.389 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:14:18.389 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:18.389 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.353+0000 7f9e68ff9640 1 -- 172.21.15.42:0/3802209763 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f9e54004020 con 0x7f9e6c06dac0 2024-09-16T04:14:18.389 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.353+0000 7f9e68ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:14:18.389 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.353+0000 7f9e68ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:14:18.389 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.353+0000 7f9e68ff9640 1 -- 172.21.15.42:0/3802209763 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9e6c06dac0 msgr2=0x7f9e6c02c650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:18.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.353+0000 7f9e68ff9640 1 --2- 172.21.15.42:0/3802209763 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9e6c06dac0 0x7f9e6c02c650 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f9e6c13c320 tx=0x7f9e5400f610 comp rx=0 tx=0).stop 2024-09-16T04:14:18.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.353+0000 7f9e68ff9640 1 -- 172.21.15.42:0/3802209763 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9e6c06dac0 msgr2=0x7f9e6c02c650 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:14:18.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.353+0000 7f9e68ff9640 1 --2- 172.21.15.42:0/3802209763 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9e6c06dac0 0x7f9e6c02c650 secure :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f9e6c13c320 tx=0x7f9e5400f610 comp rx=0 tx=0).stop 2024-09-16T04:14:18.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.357+0000 7f9e7270c640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-16T04:14:18.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.357+0000 7f9e7270c640 1 --2- 172.21.15.42:0/3802209763 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9e6c02cb90 0x7f9e6c02cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:18.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.357+0000 7f9e7270c640 10 monclient(hunting): start opening mon connection 2024-09-16T04:14:18.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.357+0000 7f9e7270c640 1 -- 172.21.15.42:0/3802209763 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f9e6c138a40 con 0x7f9e6c02cb90 2024-09-16T04:14:18.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.357+0000 7f9e6b7fe640 1 --2- 172.21.15.42:0/3802209763 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9e6c02cb90 0x7f9e6c02cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:18.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.357+0000 7f9e6b7fe640 10 monclient: get_auth_request con 0x7f9e6c02cb90 auth_method 0 2024-09-16T04:14:18.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.357+0000 7f9e6b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:14:18.392 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.357+0000 7f9e6b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:18.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.357+0000 7f9e6b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:18.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.357+0000 7f9e6b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:14:18.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.357+0000 7f9e6b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:14:18.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.357+0000 7f9e6b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:14:18.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.357+0000 7f9e6b7fe640 10 monclient(hunting): handle_auth_done global_id 1074379 payload 824 2024-09-16T04:14:18.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.357+0000 7f9e6b7fe640 1 -- 172.21.15.42:0/3802209763 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f9e6c069040 con 0x7f9e6c02cb90 2024-09-16T04:14:18.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.357+0000 7f9e68ff9640 10 client.1094216.objecter ms_handle_connect 0x7f9e6c02cb90 2024-09-16T04:14:18.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.357+0000 7f9e68ff9640 10 client.1094216.objecter resend_mon_ops 2024-09-16T04:14:18.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.357+0000 7f9e6b7fe640 1 --2- 172.21.15.42:0/3802209763 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9e6c02cb90 0x7f9e6c02cfa0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f9e6c13c320 tx=0x7f9e540046d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:14:18.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.357+0000 7f9e68ff9640 1 -- 172.21.15.42:0/3802209763 <== mon.1 v2:172.21.15.154:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e54003e80 con 0x7f9e6c02cb90 2024-09-16T04:14:18.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.357+0000 7f9e68ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:18.393 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.357+0000 7f9e68ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-16T04:14:18.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.357+0000 7f9e68ff9640 10 monclient: dump: 2024-09-16T04:14:18.394 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:14:18.394 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:14:18.394 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:18.394 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:18.394 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:18.394 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:18.394 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:14:18.394 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:14:18.394 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:14:18.394 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:18.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.361+0000 7f9e68ff9640 1 -- 172.21.15.42:0/3802209763 <== mon.1 v2:172.21.15.154:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4880 (secure 0 0 0) 0x7f9e540273a0 con 0x7f9e6c02cb90 2024-09-16T04:14:18.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.361+0000 7f9e68ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-16T04:14:18.394 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.361+0000 7f9e68ff9640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:14:18.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.361+0000 7f9e68ff9640 1 -- 172.21.15.42:0/3802209763 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9e6c02cb90 msgr2=0x7f9e6c02cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:18.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.361+0000 7f9e68ff9640 1 --2- 172.21.15.42:0/3802209763 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9e6c02cb90 0x7f9e6c02cfa0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f9e6c13c320 tx=0x7f9e540046d0 comp rx=0 tx=0).stop 2024-09-16T04:14:18.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.361+0000 7f9e68ff9640 1 -- 172.21.15.42:0/3802209763 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9e6c02cb90 msgr2=0x7f9e6c02cfa0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-16T04:14:18.395 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.361+0000 7f9e68ff9640 1 --2- 172.21.15.42:0/3802209763 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f9e6c02cb90 0x7f9e6c02cfa0 secure :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f9e6c13c320 tx=0x7f9e540046d0 comp rx=0 tx=0).stop 2024-09-16T04:14:18.395 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T04:14:18.395 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:14:18.395 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:14:18.395 INFO:teuthology.orchestra.run.smithi042.stdout: "state": "peon", 2024-09-16T04:14:18.395 INFO:teuthology.orchestra.run.smithi042.stdout: "election_epoch": 292, 2024-09-16T04:14:18.395 INFO:teuthology.orchestra.run.smithi042.stdout: "uptime": 171530, 2024-09-16T04:14:18.396 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum": [ 2024-09-16T04:14:18.396 INFO:teuthology.orchestra.run.smithi042.stdout: 0, 2024-09-16T04:14:18.396 INFO:teuthology.orchestra.run.smithi042.stdout: 1, 2024-09-16T04:14:18.396 INFO:teuthology.orchestra.run.smithi042.stdout: 2 2024-09-16T04:14:18.396 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:14:18.396 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_age": 1, 2024-09-16T04:14:18.396 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:14:18.396 INFO:teuthology.orchestra.run.smithi042.stdout: "required_con": "2450521708007866372", 2024-09-16T04:14:18.396 INFO:teuthology.orchestra.run.smithi042.stdout: "required_mon": [ 2024-09-16T04:14:18.396 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:14:18.396 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:14:18.396 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:14:18.396 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:14:18.396 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:14:18.397 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:14:18.397 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:14:18.397 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:14:18.397 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:14:18.397 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:14:18.397 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:14:18.397 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:14:18.397 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_con": "4540701547738038271", 2024-09-16T04:14:18.397 INFO:teuthology.orchestra.run.smithi042.stdout: "quorum_mon": [ 2024-09-16T04:14:18.397 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:14:18.397 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:14:18.397 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:14:18.398 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:14:18.398 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:14:18.398 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:14:18.398 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:14:18.398 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:14:18.398 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:14:18.399 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:14:18.399 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:14:18.399 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:14:18.399 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:14:18.399 INFO:teuthology.orchestra.run.smithi042.stdout: "outside_quorum": [], 2024-09-16T04:14:18.399 INFO:teuthology.orchestra.run.smithi042.stdout: "extra_probe_peers": [], 2024-09-16T04:14:18.399 INFO:teuthology.orchestra.run.smithi042.stdout: "sync_provider": [], 2024-09-16T04:14:18.399 INFO:teuthology.orchestra.run.smithi042.stdout: "monmap": { 2024-09-16T04:14:18.399 INFO:teuthology.orchestra.run.smithi042.stdout: "epoch": 1, 2024-09-16T04:14:18.399 INFO:teuthology.orchestra.run.smithi042.stdout: "fsid": "15853af6-d0b2-472c-b2ac-8d7ad428641f", 2024-09-16T04:14:18.399 INFO:teuthology.orchestra.run.smithi042.stdout: "modified": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:14:18.399 INFO:teuthology.orchestra.run.smithi042.stdout: "created": "2024-09-16T03:41:35.824079Z", 2024-09-16T04:14:18.399 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release": 19, 2024-09-16T04:14:18.400 INFO:teuthology.orchestra.run.smithi042.stdout: "min_mon_release_name": "squid", 2024-09-16T04:14:18.400 INFO:teuthology.orchestra.run.smithi042.stdout: "election_strategy": 1, 2024-09-16T04:14:18.400 INFO:teuthology.orchestra.run.smithi042.stdout: "disallowed_leaders: ": "", 2024-09-16T04:14:18.400 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false, 2024-09-16T04:14:18.400 INFO:teuthology.orchestra.run.smithi042.stdout: "tiebreaker_mon": "", 2024-09-16T04:14:18.400 INFO:teuthology.orchestra.run.smithi042.stdout: "removed_ranks: ": "", 2024-09-16T04:14:18.400 INFO:teuthology.orchestra.run.smithi042.stdout: "features": { 2024-09-16T04:14:18.400 INFO:teuthology.orchestra.run.smithi042.stdout: "persistent": [ 2024-09-16T04:14:18.400 INFO:teuthology.orchestra.run.smithi042.stdout: "kraken", 2024-09-16T04:14:18.400 INFO:teuthology.orchestra.run.smithi042.stdout: "luminous", 2024-09-16T04:14:18.400 INFO:teuthology.orchestra.run.smithi042.stdout: "mimic", 2024-09-16T04:14:18.400 INFO:teuthology.orchestra.run.smithi042.stdout: "osdmap-prune", 2024-09-16T04:14:18.400 INFO:teuthology.orchestra.run.smithi042.stdout: "nautilus", 2024-09-16T04:14:18.401 INFO:teuthology.orchestra.run.smithi042.stdout: "octopus", 2024-09-16T04:14:18.401 INFO:teuthology.orchestra.run.smithi042.stdout: "pacific", 2024-09-16T04:14:18.401 INFO:teuthology.orchestra.run.smithi042.stdout: "elector-pinging", 2024-09-16T04:14:18.401 INFO:teuthology.orchestra.run.smithi042.stdout: "quincy", 2024-09-16T04:14:18.401 INFO:teuthology.orchestra.run.smithi042.stdout: "reef", 2024-09-16T04:14:18.401 INFO:teuthology.orchestra.run.smithi042.stdout: "squid" 2024-09-16T04:14:18.401 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:14:18.401 INFO:teuthology.orchestra.run.smithi042.stdout: "optional": [] 2024-09-16T04:14:18.401 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:14:18.402 INFO:teuthology.orchestra.run.smithi042.stdout: "mons": [ 2024-09-16T04:14:18.402 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:14:18.402 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 0, 2024-09-16T04:14:18.402 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "a", 2024-09-16T04:14:18.402 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:14:18.402 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:14:18.403 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:14:18.403 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:14:18.403 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3300", 2024-09-16T04:14:18.403 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:14:18.403 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:14:18.403 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:14:18.404 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:14:18.404 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789", 2024-09-16T04:14:18.404 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:14:18.404 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:14:18.404 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:14:18.404 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:14:18.404 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6789/0", 2024-09-16T04:14:18.404 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6789/0", 2024-09-16T04:14:18.405 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:14:18.405 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:14:18.405 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:14:18.405 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:14:18.405 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:14:18.405 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 1, 2024-09-16T04:14:18.405 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "b", 2024-09-16T04:14:18.405 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:14:18.406 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:14:18.406 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:14:18.406 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:14:18.406 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:3300", 2024-09-16T04:14:18.406 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:14:18.406 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:14:18.406 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:14:18.406 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:14:18.407 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789", 2024-09-16T04:14:18.407 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:14:18.407 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:14:18.407 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:14:18.407 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:14:18.407 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.154:6789/0", 2024-09-16T04:14:18.407 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.154:6789/0", 2024-09-16T04:14:18.408 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:14:18.408 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:14:18.408 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:14:18.408 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:14:18.408 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:14:18.408 INFO:teuthology.orchestra.run.smithi042.stdout: "rank": 2, 2024-09-16T04:14:18.409 INFO:teuthology.orchestra.run.smithi042.stdout: "name": "c", 2024-09-16T04:14:18.409 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addrs": { 2024-09-16T04:14:18.409 INFO:teuthology.orchestra.run.smithi042.stdout: "addrvec": [ 2024-09-16T04:14:18.409 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:14:18.409 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v2", 2024-09-16T04:14:18.409 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:3301", 2024-09-16T04:14:18.410 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:14:18.410 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:14:18.410 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:14:18.410 INFO:teuthology.orchestra.run.smithi042.stdout: "type": "v1", 2024-09-16T04:14:18.410 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790", 2024-09-16T04:14:18.410 INFO:teuthology.orchestra.run.smithi042.stdout: "nonce": 0 2024-09-16T04:14:18.411 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:14:18.411 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:14:18.411 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:14:18.411 INFO:teuthology.orchestra.run.smithi042.stdout: "addr": "172.21.15.42:6790/0", 2024-09-16T04:14:18.412 INFO:teuthology.orchestra.run.smithi042.stdout: "public_addr": "172.21.15.42:6790/0", 2024-09-16T04:14:18.412 INFO:teuthology.orchestra.run.smithi042.stdout: "priority": 0, 2024-09-16T04:14:18.412 INFO:teuthology.orchestra.run.smithi042.stdout: "weight": 0, 2024-09-16T04:14:18.412 INFO:teuthology.orchestra.run.smithi042.stdout: "crush_location": "{}" 2024-09-16T04:14:18.412 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:14:18.412 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:14:18.413 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:14:18.413 INFO:teuthology.orchestra.run.smithi042.stdout: "feature_map": { 2024-09-16T04:14:18.413 INFO:teuthology.orchestra.run.smithi042.stdout: "mon": [ 2024-09-16T04:14:18.413 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:14:18.413 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:14:18.413 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:14:18.413 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:14:18.413 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:14:18.413 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:14:18.413 INFO:teuthology.orchestra.run.smithi042.stdout: "osd": [ 2024-09-16T04:14:18.413 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:14:18.413 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:14:18.413 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:14:18.413 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:14:18.414 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:14:18.414 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:14:18.414 INFO:teuthology.orchestra.run.smithi042.stdout: "client": [ 2024-09-16T04:14:18.414 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:14:18.414 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:14:18.414 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:14:18.414 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 3 2024-09-16T04:14:18.414 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:14:18.414 INFO:teuthology.orchestra.run.smithi042.stdout: ], 2024-09-16T04:14:18.414 INFO:teuthology.orchestra.run.smithi042.stdout: "mgr": [ 2024-09-16T04:14:18.414 INFO:teuthology.orchestra.run.smithi042.stdout: { 2024-09-16T04:14:18.414 INFO:teuthology.orchestra.run.smithi042.stdout: "features": "0x3f03cffffffdffff", 2024-09-16T04:14:18.414 INFO:teuthology.orchestra.run.smithi042.stdout: "release": "squid", 2024-09-16T04:14:18.414 INFO:teuthology.orchestra.run.smithi042.stdout: "num": 1 2024-09-16T04:14:18.414 INFO:teuthology.orchestra.run.smithi042.stdout: } 2024-09-16T04:14:18.415 INFO:teuthology.orchestra.run.smithi042.stdout: ] 2024-09-16T04:14:18.415 INFO:teuthology.orchestra.run.smithi042.stdout: }, 2024-09-16T04:14:18.415 INFO:teuthology.orchestra.run.smithi042.stdout: "stretch_mode": false 2024-09-16T04:14:18.415 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T04:14:18.417 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.361+0000 7f9e7270c640 10 librados: watch_flush enter 2024-09-16T04:14:18.417 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.361+0000 7f9e7270c640 10 librados: watch_flush exit 2024-09-16T04:14:18.417 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.361+0000 7f9e7270c640 20 client.1094216.objecter shutdown clearing up homeless session... 2024-09-16T04:14:18.417 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.361+0000 7f9e7270c640 10 client.1094216.objecter successfully canceled tick 2024-09-16T04:14:18.417 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.361+0000 7f9e7270c640 1 -- 172.21.15.42:0/3802209763 >> v2:172.21.15.154:6812/254845342 conn(0x7f9e4c037810 msgr2=0x7f9e4c039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:18.417 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.361+0000 7f9e7270c640 1 --2- 172.21.15.42:0/3802209763 >> v2:172.21.15.154:6812/254845342 conn(0x7f9e4c037810 0x7f9e4c039cd0 secure :-1 s=READY pgs=4008 cs=0 l=1 rev1=1 crypto rx=0x7f9e5c0ae000 tx=0x7f9e5c0a7000 comp rx=0 tx=0).stop 2024-09-16T04:14:18.417 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.361+0000 7f9e7270c640 10 monclient: shutdown 2024-09-16T04:14:18.417 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.361+0000 7f9e7270c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:14:18.417 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.361+0000 7f9e7270c640 1 -- 172.21.15.42:0/3802209763 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9e6c0a1330 msgr2=0x7f9e6c1ccb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:18.418 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.361+0000 7f9e7270c640 1 --2- 172.21.15.42:0/3802209763 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f9e6c0a1330 0x7f9e6c1ccb30 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f9e6400cb10 tx=0x7f9e6400a060 comp rx=0 tx=0).stop 2024-09-16T04:14:18.418 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.361+0000 7f9e70ca9640 1 -- 172.21.15.42:0/3802209763 reap_dead start 2024-09-16T04:14:18.418 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.361+0000 7f9e70ca9640 1 -- 172.21.15.42:0/3802209763 reap_dead start 2024-09-16T04:14:18.418 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.361+0000 7f9e7270c640 1 -- 172.21.15.42:0/3802209763 shutdown_connections 2024-09-16T04:14:18.418 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.361+0000 7f9e7270c640 1 -- 172.21.15.42:0/3802209763 >> 172.21.15.42:0/3802209763 conn(0x7f9e6c0a2420 msgr2=0x7f9e6c0a4ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:14:18.418 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.361+0000 7f9e7270c640 1 -- 172.21.15.42:0/3802209763 shutdown_connections 2024-09-16T04:14:18.418 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.361+0000 7f9e7270c640 1 -- 172.21.15.42:0/3802209763 wait complete. 2024-09-16T04:14:18.418 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.361+0000 7f9e7270c640 1 librados: shutdown 2024-09-16T04:14:18.419 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-16T04:14:18.419 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-16T04:14:18.419 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-16T04:14:18.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.529+0000 7fd0b316f640 10 monclient: get_monmap_and_config 2024-09-16T04:14:18.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.529+0000 7fd0b316f640 10 monclient: build_initial_monmap 2024-09-16T04:14:18.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.529+0000 7fd0b316f640 10 monclient: monmap: 2024-09-16T04:14:18.536 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:14:18.536 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:14:18.536 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:14:18.534303+0000 2024-09-16T04:14:18.536 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:14:18.534303+0000 2024-09-16T04:14:18.536 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:14:18.536 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:18.536 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:14:18.536 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:14:18.536 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:14:18.536 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:14:18.536 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:14:18.536 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:18.538 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.529+0000 7fd0b316f640 1 Processor -- start 2024-09-16T04:14:18.538 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.529+0000 7fd0b316f640 1 -- start start 2024-09-16T04:14:18.538 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.529+0000 7fd0b316f640 10 monclient: init 2024-09-16T04:14:18.538 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.529+0000 7fd0b316f640 10 monclient: _reopen_session rank -1 2024-09-16T04:14:18.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.529+0000 7fd0b316f640 10 monclient: _add_conns ranks=[0,3,2,1,4] 2024-09-16T04:14:18.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.529+0000 7fd0b316f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd0ac137500 addr v1:172.21.15.42:6789/0 2024-09-16T04:14:18.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.529+0000 7fd0b316f640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd0ac0a1f60 0x7fd0ac13c360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:18.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.529+0000 7fd0b316f640 10 monclient(hunting): picked mon.noname-a con 0x7fd0ac0a1f60 addr v2:172.21.15.42:3300/0 2024-09-16T04:14:18.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.529+0000 7fd0b316f640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd0ac06e8f0 0x7fd0ac13e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:18.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.529+0000 7fd0b316f640 10 monclient(hunting): picked mon.noname-b con 0x7fd0ac06e8f0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:14:18.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.529+0000 7fd0b316f640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd0ac0a2740 con 0x7fd0ac137500 2024-09-16T04:14:18.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.529+0000 7fd0b316f640 10 monclient(hunting): start opening mon connection 2024-09-16T04:14:18.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.529+0000 7fd0b316f640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7fd0ac0a2fe0 con 0x7fd0ac0a1f60 2024-09-16T04:14:18.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.529+0000 7fd0b316f640 10 monclient(hunting): start opening mon connection 2024-09-16T04:14:18.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.529+0000 7fd0b316f640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fd0ac0a2a20 con 0x7fd0ac06e8f0 2024-09-16T04:14:18.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.529+0000 7fd0b316f640 10 monclient(hunting): _renew_subs 2024-09-16T04:14:18.539 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.529+0000 7fd0b316f640 10 monclient(hunting): authenticate will time out at 2905.503777s 2024-09-16T04:14:18.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.529+0000 7fd0abfff640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd0ac0a1f60 0x7fd0ac13c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:18.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.529+0000 7fd0b170c640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd0ac06e8f0 0x7fd0ac13e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:18.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.533+0000 7fd0abfff640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fd0ac0a1f60 0x7fd0ac13c360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:47894/0 (socket says 172.21.15.42:47894) 2024-09-16T04:14:18.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.533+0000 7fd0abfff640 1 -- 172.21.15.42:0/2647310289 learned_addr learned my addr 172.21.15.42:0/2647310289 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:14:18.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.533+0000 7fd0abfff640 10 monclient(hunting): get_auth_request con 0x7fd0ac0a1f60 auth_method 0 2024-09-16T04:14:18.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.533+0000 7fd0abfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:14:18.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.533+0000 7fd0abfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:18.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.533+0000 7fd0abfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:18.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.533+0000 7fd0b170c640 10 monclient(hunting): get_auth_request con 0x7fd0ac06e8f0 auth_method 0 2024-09-16T04:14:18.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.533+0000 7fd0b170c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:14:18.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.533+0000 7fd0b170c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:18.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.533+0000 7fd0b170c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:18.540 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.533+0000 7fd0abfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:14:18.541 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.533+0000 7fd0abfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:14:18.541 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.533+0000 7fd0abfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:14:18.541 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.533+0000 7fd0b170c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:14:18.541 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.533+0000 7fd0b170c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:14:18.541 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.533+0000 7fd0b170c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:14:18.541 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.533+0000 7fd0abfff640 10 monclient(hunting): handle_auth_done global_id 1104132 payload 274 2024-09-16T04:14:18.541 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.533+0000 7fd0abfff640 1 -- 172.21.15.42:0/2647310289 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd0ac06e8f0 msgr2=0x7fd0ac13e8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:14:18.541 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.533+0000 7fd0abfff640 1 --2- 172.21.15.42:0/2647310289 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd0ac06e8f0 0x7fd0ac13e8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:18.541 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.533+0000 7fd0abfff640 1 -- 172.21.15.42:0/2647310289 >> v1:172.21.15.42:6789/0 conn(0x7fd0ac137500 legacy=0x7fd0ac1378e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:18.541 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.533+0000 7fd0abfff640 10 monclient: _finish_hunting 0 2024-09-16T04:14:18.541 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.533+0000 7fd0abfff640 1 monclient: found mon.noname-a 2024-09-16T04:14:18.541 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.533+0000 7fd0abfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:14:18.541 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.533+0000 7fd0abfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:14:18.541 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.533+0000 7fd0abfff640 1 -- 172.21.15.42:0/2647310289 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0ac0a2d00 con 0x7fd0ac0a1f60 2024-09-16T04:14:18.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.533+0000 7fd0abfff640 1 --2- 172.21.15.42:0/2647310289 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd0ac0a1f60 0x7fd0ac13c360 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fd09c009870 tx=0x7fd09c01c420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=840b5637f86a361 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:14:18.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.533+0000 7fd0ab7fe640 1 -- 172.21.15.42:0/2647310289 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd09c0b0070 con 0x7fd0ac0a1f60 2024-09-16T04:14:18.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.533+0000 7fd0ab7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:18.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.533+0000 7fd0ab7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:14:18.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.533+0000 7fd0ab7fe640 10 monclient: dump: 2024-09-16T04:14:18.548 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:14:18.548 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:14:18.548 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:18.548 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:18.548 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:18.548 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:18.548 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:14:18.548 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:14:18.548 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:14:18.548 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:18.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.533+0000 7fd0ab7fe640 10 monclient: _finish_auth 0 2024-09-16T04:14:18.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.533+0000 7fd0ab7fe640 10 monclient: _check_auth_tickets 2024-09-16T04:14:18.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.533+0000 7fd0ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:14:18.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.533+0000 7fd0b316f640 5 monclient: authenticate success, global_id 1104132 2024-09-16T04:14:18.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.533+0000 7fd0b316f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:14:18.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.533+0000 7fd0ab7fe640 1 -- 172.21.15.42:0/2647310289 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd09c0ab600 con 0x7fd0ac0a1f60 2024-09-16T04:14:18.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.533+0000 7fd0ab7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:14:18.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.533+0000 7fd0b316f640 10 monclient: get_monmap_and_config success 2024-09-16T04:14:18.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.533+0000 7fd0b316f640 4 set_mon_vals no callback set 2024-09-16T04:14:18.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.533+0000 7fd0ab7fe640 1 -- 172.21.15.42:0/2647310289 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd09c0ab9d0 con 0x7fd0ac0a1f60 2024-09-16T04:14:18.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.533+0000 7fd0ab7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:18.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.533+0000 7fd0ab7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:14:18.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.533+0000 7fd0ab7fe640 10 monclient: dump: 2024-09-16T04:14:18.550 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:14:18.550 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:14:18.550 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:18.550 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:18.550 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:18.550 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:18.550 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:14:18.550 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:14:18.550 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:14:18.550 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:18.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.533+0000 7fd0b316f640 10 monclient: shutdown 2024-09-16T04:14:18.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.533+0000 7fd0b316f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:14:18.551 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.533+0000 7fd0b316f640 1 -- 172.21.15.42:0/2647310289 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd0ac0a1f60 msgr2=0x7fd0ac13c360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:18.551 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.533+0000 7fd0b316f640 1 --2- 172.21.15.42:0/2647310289 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd0ac0a1f60 0x7fd0ac13c360 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fd09c009870 tx=0x7fd09c01c420 comp rx=0 tx=0).stop 2024-09-16T04:14:18.551 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.533+0000 7fd0b316f640 1 -- 172.21.15.42:0/2647310289 shutdown_connections 2024-09-16T04:14:18.551 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.533+0000 7fd0b316f640 1 --2- 172.21.15.42:0/2647310289 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd0ac06e8f0 0x7fd0ac13e8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:18.551 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.533+0000 7fd0b316f640 1 --2- 172.21.15.42:0/2647310289 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd0ac0a1f60 0x7fd0ac13c360 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:18.551 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.533+0000 7fd0b316f640 1 -- 172.21.15.42:0/2647310289 >> 172.21.15.42:0/2647310289 conn(0x7fd0ac075ae0 msgr2=0x7fd0ac075ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:14:18.551 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.533+0000 7fd0b316f640 1 -- 172.21.15.42:0/2647310289 shutdown_connections 2024-09-16T04:14:18.551 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.533+0000 7fd0b316f640 1 -- 172.21.15.42:0/2647310289 wait complete. 2024-09-16T04:14:18.551 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.537+0000 7fd0b316f640 10 monclient: build_initial_monmap 2024-09-16T04:14:18.551 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.537+0000 7fd0b316f640 10 monclient: monmap: 2024-09-16T04:14:18.551 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:14:18.551 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:14:18.551 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:14:18.552 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:14:18.552 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:14:18.552 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:18.552 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:14:18.552 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:14:18.552 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:14:18.552 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:18.552 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.537+0000 7fd0b316f640 1 librados: starting msgr at 2024-09-16T04:14:18.552 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.537+0000 7fd0b316f640 1 librados: starting objecter 2024-09-16T04:14:18.552 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.537+0000 7fd0b316f640 1 Processor -- start 2024-09-16T04:14:18.552 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.537+0000 7fd0b316f640 1 -- start start 2024-09-16T04:14:18.552 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.537+0000 7fd0b316f640 1 librados: setting wanted keys 2024-09-16T04:14:18.552 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.537+0000 7fd0b316f640 1 librados: calling monclient init 2024-09-16T04:14:18.552 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.537+0000 7fd0b316f640 10 monclient: init 2024-09-16T04:14:18.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.537+0000 7fd0b316f640 10 monclient: _reopen_session rank -1 2024-09-16T04:14:18.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.537+0000 7fd0b316f640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:14:18.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.537+0000 7fd0b316f640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd0ac06e8f0 0x7fd0ac136080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:18.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.537+0000 7fd0b316f640 10 monclient(hunting): picked mon.noname-c con 0x7fd0ac06e8f0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:14:18.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.537+0000 7fd0b316f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd0ac0a1f60 0x7fd0ac1365c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:18.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.537+0000 7fd0b316f640 10 monclient(hunting): picked mon.noname-b con 0x7fd0ac0a1f60 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:14:18.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.537+0000 7fd0b316f640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd0ac137500 0x7fd0ac11cf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:18.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.537+0000 7fd0b316f640 10 monclient(hunting): picked mon.noname-a con 0x7fd0ac137500 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:14:18.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.537+0000 7fd0b316f640 10 monclient(hunting): start opening mon connection 2024-09-16T04:14:18.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.537+0000 7fd0b316f640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fd0ac1419a0 con 0x7fd0ac137500 2024-09-16T04:14:18.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.537+0000 7fd0b316f640 10 monclient(hunting): start opening mon connection 2024-09-16T04:14:18.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.537+0000 7fd0b316f640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fd0ac141820 con 0x7fd0ac0a1f60 2024-09-16T04:14:18.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.537+0000 7fd0b316f640 10 monclient(hunting): start opening mon connection 2024-09-16T04:14:18.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.537+0000 7fd0b316f640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fd0ac141b20 con 0x7fd0ac06e8f0 2024-09-16T04:14:18.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.537+0000 7fd0b316f640 10 monclient(hunting): _renew_subs 2024-09-16T04:14:18.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.537+0000 7fd0b316f640 10 monclient(hunting): authenticate will time out at 2905.508926s 2024-09-16T04:14:18.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.537+0000 7fd0b0f0b640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd0ac06e8f0 0x7fd0ac136080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:18.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.537+0000 7fd0b170c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd0ac137500 0x7fd0ac11cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:18.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.537+0000 7fd0b170c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd0ac137500 0x7fd0ac11cf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:47902/0 (socket says 172.21.15.42:47902) 2024-09-16T04:14:18.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.537+0000 7fd0b0f0b640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd0ac06e8f0 0x7fd0ac136080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:51834/0 (socket says 172.21.15.42:51834) 2024-09-16T04:14:18.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.537+0000 7fd0b170c640 1 -- 172.21.15.42:0/2210135578 learned_addr learned my addr 172.21.15.42:0/2210135578 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:14:18.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.537+0000 7fd0b170c640 10 monclient(hunting): get_auth_request con 0x7fd0ac137500 auth_method 0 2024-09-16T04:14:18.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.537+0000 7fd0b170c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:14:18.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.537+0000 7fd0b170c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:18.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.537+0000 7fd0b170c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:18.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.537+0000 7fd0b0f0b640 10 monclient(hunting): get_auth_request con 0x7fd0ac06e8f0 auth_method 0 2024-09-16T04:14:18.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.537+0000 7fd0b0f0b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:14:18.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.537+0000 7fd0b0f0b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:18.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.537+0000 7fd0b0f0b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:18.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.537+0000 7fd0b170c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:14:18.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.537+0000 7fd0b170c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:14:18.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.537+0000 7fd0b170c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:14:18.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.537+0000 7fd0b0f0b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:14:18.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.537+0000 7fd0b0f0b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:14:18.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.537+0000 7fd0b0f0b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:14:18.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.537+0000 7fd0abfff640 1 --2- 172.21.15.42:0/2210135578 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd0ac0a1f60 0x7fd0ac1365c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:18.556 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.537+0000 7fd0b170c640 10 monclient(hunting): handle_auth_done global_id 1104138 payload 824 2024-09-16T04:14:18.556 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.537+0000 7fd0b170c640 1 -- 172.21.15.42:0/2210135578 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd0ac06e8f0 msgr2=0x7fd0ac136080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:18.556 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.537+0000 7fd0b170c640 1 --2- 172.21.15.42:0/2210135578 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd0ac06e8f0 0x7fd0ac136080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:18.556 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.537+0000 7fd0b170c640 1 -- 172.21.15.42:0/2210135578 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd0ac0a1f60 msgr2=0x7fd0ac1365c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:18.556 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.537+0000 7fd0b170c640 1 --2- 172.21.15.42:0/2210135578 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd0ac0a1f60 0x7fd0ac1365c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:18.556 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.537+0000 7fd0b170c640 10 monclient: _finish_hunting 0 2024-09-16T04:14:18.556 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.537+0000 7fd0b170c640 1 monclient: found mon.noname-a 2024-09-16T04:14:18.556 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.537+0000 7fd0b170c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:14:18.556 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.537+0000 7fd0b170c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:14:18.556 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.537+0000 7fd0b170c640 1 -- 172.21.15.42:0/2210135578 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0ac1208a0 con 0x7fd0ac137500 2024-09-16T04:14:18.556 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.537+0000 7fd0abfff640 10 monclient: get_auth_request con 0x7fd0ac0a1f60 auth_method 0 2024-09-16T04:14:18.556 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.537+0000 7fd0abfff640 1 --2- 172.21.15.42:0/2210135578 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd0ac0a1f60 0x7fd0ac1365c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T04:14:18.556 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.537+0000 7fd0b0f0b640 1 --2- 172.21.15.42:0/2210135578 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd0ac06e8f0 0x7fd0ac136080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:14:18.557 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.537+0000 7fd0b170c640 1 --2- 172.21.15.42:0/2210135578 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd0ac137500 0x7fd0ac11cf80 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fd0a400efc0 tx=0x7fd0a400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:14:18.557 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.537+0000 7fd0a97fa640 10 client.?.objecter ms_handle_connect 0x7fd0ac137500 2024-09-16T04:14:18.557 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.537+0000 7fd0a97fa640 10 client.?.objecter resend_mon_ops 2024-09-16T04:14:18.557 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.537+0000 7fd0a97fa640 1 -- 172.21.15.42:0/2210135578 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0a4018020 con 0x7fd0ac137500 2024-09-16T04:14:18.557 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.537+0000 7fd0a97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:18.557 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.537+0000 7fd0a97fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:14:18.557 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.537+0000 7fd0a97fa640 10 monclient: dump: 2024-09-16T04:14:18.557 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:14:18.557 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:14:18.557 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:18.557 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:18.557 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:18.557 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:18.558 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:14:18.558 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:14:18.558 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:14:18.558 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:18.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.537+0000 7fd0a97fa640 10 monclient: _finish_auth 0 2024-09-16T04:14:18.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.537+0000 7fd0a97fa640 10 monclient: _check_auth_tickets 2024-09-16T04:14:18.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.537+0000 7fd0a97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:14:18.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.537+0000 7fd0a97fa640 1 -- 172.21.15.42:0/2210135578 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd0a4014480 con 0x7fd0ac137500 2024-09-16T04:14:18.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.537+0000 7fd0a97fa640 10 monclient: handle_config config(0 keys) 2024-09-16T04:14:18.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.537+0000 7fd0a97fa640 1 -- 172.21.15.42:0/2210135578 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0a4014870 con 0x7fd0ac137500 2024-09-16T04:14:18.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.537+0000 7fd0a97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:18.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.537+0000 7fd0a97fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:14:18.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.537+0000 7fd0a97fa640 10 monclient: dump: 2024-09-16T04:14:18.558 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:14:18.559 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:14:18.559 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:18.559 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:18.559 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:18.559 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:18.559 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:14:18.559 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:14:18.559 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:14:18.559 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:18.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.537+0000 7fd0b316f640 5 monclient: authenticate success, global_id 1104138 2024-09-16T04:14:18.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.537+0000 7fd0b316f640 10 monclient: _renew_subs 2024-09-16T04:14:18.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.537+0000 7fd0b316f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:14:18.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.537+0000 7fd0aaffd640 4 set_mon_vals no callback set 2024-09-16T04:14:18.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.537+0000 7fd0b316f640 1 -- 172.21.15.42:0/2210135578 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0ac120b80 con 0x7fd0ac137500 2024-09-16T04:14:18.560 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.537+0000 7fd0b316f640 10 client.1104138.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:14:18.560 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.537+0000 7fd0b316f640 10 monclient: _renew_subs 2024-09-16T04:14:18.560 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.537+0000 7fd0b316f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:14:18.560 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.537+0000 7fd0b316f640 1 -- 172.21.15.42:0/2210135578 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0ac11f610 con 0x7fd0ac137500 2024-09-16T04:14:18.560 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.537+0000 7fd0b316f640 1 librados: init done 2024-09-16T04:14:18.560 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.537+0000 7fd0a97fa640 1 -- 172.21.15.42:0/2210135578 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fd0a401b020 con 0x7fd0ac137500 2024-09-16T04:14:18.560 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.537+0000 7fd0b316f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:14:18.560 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.537+0000 7fd0b316f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:14:18.560 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.537+0000 7fd0b316f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:14:18.560 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.537+0000 7fd0a97fa640 1 --2- 172.21.15.42:0/2210135578 >> v2:172.21.15.154:6812/254845342 conn(0x7fd098037840 0x7fd098039d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:18.560 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.537+0000 7fd0b316f640 1 -- 172.21.15.42:0/2210135578 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd0ac1378e0 con 0x7fd0ac137500 2024-09-16T04:14:18.560 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.537+0000 7fd0a97fa640 1 -- 172.21.15.42:0/2210135578 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(710..710 src has 681..710) ==== 4681+0+0 (secure 0 0 0) 0x7fd0a4049a30 con 0x7fd0ac137500 2024-09-16T04:14:18.560 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.537+0000 7fd0a97fa640 10 client.1104138.objecter ms_dispatch 0x7fd0ac000ba0 osd_map(710..710 src has 681..710) 2024-09-16T04:14:18.560 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.537+0000 7fd0a97fa640 3 client.1104138.objecter handle_osd_map got epochs [710,710] > 0 2024-09-16T04:14:18.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.537+0000 7fd0a97fa640 3 client.1104138.objecter handle_osd_map decoding full epoch 710 2024-09-16T04:14:18.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.537+0000 7fd0a97fa640 20 client.1104138.objecter dump_active .. 0 homeless 2024-09-16T04:14:18.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.537+0000 7fd0b0f0b640 1 --2- 172.21.15.42:0/2210135578 >> v2:172.21.15.154:6812/254845342 conn(0x7fd098037840 0x7fd098039d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:18.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.537+0000 7fd0b0f0b640 10 monclient: get_auth_request con 0x7fd098037840 auth_method 0 2024-09-16T04:14:18.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.541+0000 7fd0b0f0b640 1 --2- 172.21.15.42:0/2210135578 >> v2:172.21.15.154:6812/254845342 conn(0x7fd098037840 0x7fd098039d00 secure :-1 s=READY pgs=4009 cs=0 l=1 rev1=1 crypto rx=0x7fd0a0001c00 tx=0x7fd0a0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:14:18.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.541+0000 7fd0a97fa640 10 client.1104138.objecter ms_handle_connect 0x7fd098037840 2024-09-16T04:14:18.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.541+0000 7fd0a97fa640 1 -- 172.21.15.42:0/2210135578 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7fd0a4022030 con 0x7fd0ac137500 2024-09-16T04:14:18.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.541+0000 7fd0a97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:14:18.562 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.541+0000 7fd0a97fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:14:18.774 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.769+0000 7fd0b316f640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-16T04:14:18.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.769+0000 7fd0b316f640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-16T04:14:18.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.769+0000 7fd0b316f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:14:18.775 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.769+0000 7fd0b316f640 1 -- 172.21.15.42:0/2210135578 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7fd0ac11f170 con 0x7fd0ac137500 2024-09-16T04:14:18.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.769+0000 7fd0a97fa640 1 -- 172.21.15.42:0/2210135578 <== mon.0 v2:172.21.15.42:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7fd0a4010040 con 0x7fd0ac137500 2024-09-16T04:14:18.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.769+0000 7fd0a97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-16T04:14:18.776 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.769+0000 7fd0a97fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:14:18.777 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.773+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.778 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.773+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=29,config=2,health=12,kv=9,logm=48} crc {auth=35004643,config=692619927,health=553475474,kv=2818795783,logm=3917263541}) 2024-09-16T04:14:18.778 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.773+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=28,config=2,health=10,kv=9,logm=51} crc {auth=210773776,config=692619927,health=1027388545,kv=2236749221,logm=2310449580}) 2024-09-16T04:14:18.778 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.773+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.778 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.773+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=29,config=2,health=12,kv=9,logm=48} crc {auth=35004643,config=692619927,health=553475474,kv=2818795783,logm=3917263541}) 2024-09-16T04:14:18.778 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.773+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=28,config=2,health=12,kv=10,logm=48} crc {auth=4061701574,config=692619927,health=2196297382,kv=3446188498,logm=155374072}) 2024-09-16T04:14:18.779 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.773+0000 7fd0b316f640 10 librados: watch_flush enter 2024-09-16T04:14:18.779 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.773+0000 7fd0b316f640 10 librados: watch_flush exit 2024-09-16T04:14:18.779 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.773+0000 7fd0b316f640 20 client.1104138.objecter shutdown clearing up homeless session... 2024-09-16T04:14:18.779 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.773+0000 7fd0b316f640 10 client.1104138.objecter successfully canceled tick 2024-09-16T04:14:18.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.773+0000 7fd0b316f640 1 -- 172.21.15.42:0/2210135578 >> v2:172.21.15.154:6812/254845342 conn(0x7fd098037840 msgr2=0x7fd098039d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:18.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.773+0000 7fd0b316f640 1 --2- 172.21.15.42:0/2210135578 >> v2:172.21.15.154:6812/254845342 conn(0x7fd098037840 0x7fd098039d00 secure :-1 s=READY pgs=4009 cs=0 l=1 rev1=1 crypto rx=0x7fd0a0001c00 tx=0x7fd0a0009040 comp rx=0 tx=0).stop 2024-09-16T04:14:18.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.773+0000 7fd0b316f640 10 monclient: shutdown 2024-09-16T04:14:18.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.773+0000 7fd0b316f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:14:18.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.773+0000 7fd0b316f640 1 -- 172.21.15.42:0/2210135578 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd0ac137500 msgr2=0x7fd0ac11cf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:18.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.773+0000 7fd0b316f640 1 --2- 172.21.15.42:0/2210135578 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd0ac137500 0x7fd0ac11cf80 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fd0a400efc0 tx=0x7fd0a400c490 comp rx=0 tx=0).stop 2024-09-16T04:14:18.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.773+0000 7fd0b316f640 1 -- 172.21.15.42:0/2210135578 shutdown_connections 2024-09-16T04:14:18.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.773+0000 7fd0b316f640 1 --2- 172.21.15.42:0/2210135578 >> v2:172.21.15.154:6812/254845342 conn(0x7fd098037840 0x7fd098039d00 unknown :-1 s=CLOSED pgs=4009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:18.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.773+0000 7fd0b316f640 1 --2- 172.21.15.42:0/2210135578 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fd0ac137500 0x7fd0ac11cf80 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:18.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.773+0000 7fd0b316f640 1 --2- 172.21.15.42:0/2210135578 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fd0ac0a1f60 0x7fd0ac1365c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:18.780 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.773+0000 7fd0b316f640 1 --2- 172.21.15.42:0/2210135578 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fd0ac06e8f0 0x7fd0ac136080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:18.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.773+0000 7fd0b316f640 1 -- 172.21.15.42:0/2210135578 >> 172.21.15.42:0/2210135578 conn(0x7fd0ac075ae0 msgr2=0x7fd0ac0a5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:14:18.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.773+0000 7fd0b316f640 1 -- 172.21.15.42:0/2210135578 shutdown_connections 2024-09-16T04:14:18.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.773+0000 7fd0b316f640 1 -- 172.21.15.42:0/2210135578 wait complete. 2024-09-16T04:14:18.781 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:18.773+0000 7fd0b316f640 1 librados: shutdown 2024-09-16T04:14:18.782 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.773+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.782 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.773+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=505746725}) 2024-09-16T04:14:18.782 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.773+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2235318162}) 2024-09-16T04:14:18.782 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.773+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.782 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.773+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=505746725}) 2024-09-16T04:14:18.782 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.773+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1049633375}) 2024-09-16T04:14:18.784 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.777+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.784 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.777+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2365122778}) 2024-09-16T04:14:18.785 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.777+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=96705229}) 2024-09-16T04:14:18.785 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.777+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.785 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.777+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2365122778}) 2024-09-16T04:14:18.785 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.777+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3969241346}) 2024-09-16T04:14:18.788 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.781+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.788 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.781+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1911314824}) 2024-09-16T04:14:18.788 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.781+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2724289035}) 2024-09-16T04:14:18.789 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.781+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.789 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.781+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1911314824}) 2024-09-16T04:14:18.789 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.781+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3300363710}) 2024-09-16T04:14:18.792 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.785+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.792 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.785+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2818692773}) 2024-09-16T04:14:18.793 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.785+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3399159728}) 2024-09-16T04:14:18.793 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.785+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.793 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.785+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2818692773}) 2024-09-16T04:14:18.793 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.785+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2304445816}) 2024-09-16T04:14:18.795 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.789+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.795 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.789+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1860646192}) 2024-09-16T04:14:18.795 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.789+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2106228180}) 2024-09-16T04:14:18.795 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.789+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.795 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.789+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1860646192}) 2024-09-16T04:14:18.795 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.789+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=201435225}) 2024-09-16T04:14:18.797 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.789+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.797 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.789+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3831526161}) 2024-09-16T04:14:18.797 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.789+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3494380554}) 2024-09-16T04:14:18.797 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.789+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.797 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.789+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3831526161}) 2024-09-16T04:14:18.797 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.789+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=428685838}) 2024-09-16T04:14:18.798 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-16T04:14:18.799 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.793+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.799 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.793+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4147367796}) 2024-09-16T04:14:18.799 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.793+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2811909375}) 2024-09-16T04:14:18.799 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.793+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.799 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.793+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4147367796}) 2024-09-16T04:14:18.799 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.793+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1227952125}) 2024-09-16T04:14:18.801 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.793+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.801 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.793+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=72027860}) 2024-09-16T04:14:18.801 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.793+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3675408745}) 2024-09-16T04:14:18.801 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.793+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.801 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.793+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=72027860}) 2024-09-16T04:14:18.801 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.793+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3731596209}) 2024-09-16T04:14:18.802 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.797+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.803 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.797+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2547879386}) 2024-09-16T04:14:18.803 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.797+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2884727159}) 2024-09-16T04:14:18.803 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.797+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.803 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.797+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2547879386}) 2024-09-16T04:14:18.803 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.797+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4100951478}) 2024-09-16T04:14:18.805 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.797+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.805 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.797+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3614806655}) 2024-09-16T04:14:18.805 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.797+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1555892084}) 2024-09-16T04:14:18.805 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.797+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.805 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.797+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3614806655}) 2024-09-16T04:14:18.805 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.797+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=985584246}) 2024-09-16T04:14:18.806 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.801+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.806 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.801+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1100824435}) 2024-09-16T04:14:18.807 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.801+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3357572579}) 2024-09-16T04:14:18.807 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.801+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.807 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.801+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1100824435}) 2024-09-16T04:14:18.807 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.801+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1476488871}) 2024-09-16T04:14:18.808 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.801+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.808 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.801+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1386133963}) 2024-09-16T04:14:18.809 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.801+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3490408542}) 2024-09-16T04:14:18.809 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.801+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.809 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.801+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1386133963}) 2024-09-16T04:14:18.809 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.801+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3012179187}) 2024-09-16T04:14:18.810 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.805+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.811 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.805+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4211473692}) 2024-09-16T04:14:18.811 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.805+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2888751252}) 2024-09-16T04:14:18.811 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.805+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.811 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.805+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4211473692}) 2024-09-16T04:14:18.811 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.805+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1804788753}) 2024-09-16T04:14:18.812 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.805+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.812 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.805+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2938396771}) 2024-09-16T04:14:18.812 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.805+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3740590405}) 2024-09-16T04:14:18.812 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.805+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.812 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.805+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2938396771}) 2024-09-16T04:14:18.812 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.805+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1134167480}) 2024-09-16T04:14:18.814 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.809+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.814 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.809+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2561422639}) 2024-09-16T04:14:18.814 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.809+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3686832532}) 2024-09-16T04:14:18.814 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.809+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.814 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.809+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2561422639}) 2024-09-16T04:14:18.814 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.809+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4059337576}) 2024-09-16T04:14:18.816 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.809+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.817 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.809+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2861532387}) 2024-09-16T04:14:18.818 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.809+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=90139100}) 2024-09-16T04:14:18.818 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.809+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.818 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.809+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2861532387}) 2024-09-16T04:14:18.818 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.809+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2094464989}) 2024-09-16T04:14:18.818 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.813+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.818 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.813+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=259177998}) 2024-09-16T04:14:18.818 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.813+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=924901297}) 2024-09-16T04:14:18.819 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.813+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.819 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.813+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=259177998}) 2024-09-16T04:14:18.819 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.813+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1463692111}) 2024-09-16T04:14:18.820 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.813+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.820 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.813+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2111007669}) 2024-09-16T04:14:18.820 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.813+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3944439775}) 2024-09-16T04:14:18.820 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.813+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.820 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.813+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2111007669}) 2024-09-16T04:14:18.821 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.813+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3064680014}) 2024-09-16T04:14:18.822 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.817+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.822 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.817+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=54670817}) 2024-09-16T04:14:18.822 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.817+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1532659667}) 2024-09-16T04:14:18.822 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.817+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.822 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.817+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=54670817}) 2024-09-16T04:14:18.822 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.817+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3516502874}) 2024-09-16T04:14:18.824 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.817+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.824 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.817+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2533020763}) 2024-09-16T04:14:18.824 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.817+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4113492451}) 2024-09-16T04:14:18.824 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.817+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.824 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.817+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2533020763}) 2024-09-16T04:14:18.824 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.817+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1868366589}) 2024-09-16T04:14:18.826 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.821+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.826 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.821+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4278889336}) 2024-09-16T04:14:18.826 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.821+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3605374672}) 2024-09-16T04:14:18.826 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.821+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.826 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.821+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4278889336}) 2024-09-16T04:14:18.826 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.821+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3091967210}) 2024-09-16T04:14:18.827 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.821+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.828 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.821+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1504690849}) 2024-09-16T04:14:18.828 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.821+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1194839669}) 2024-09-16T04:14:18.828 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.821+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.828 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.821+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1504690849}) 2024-09-16T04:14:18.828 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.821+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3983455683}) 2024-09-16T04:14:18.830 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.825+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.830 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.825+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2816645342}) 2024-09-16T04:14:18.830 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.825+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=57354667}) 2024-09-16T04:14:18.830 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.825+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.830 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.825+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2816645342}) 2024-09-16T04:14:18.830 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.825+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3562588290}) 2024-09-16T04:14:18.832 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.825+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.832 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.825+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3636725116}) 2024-09-16T04:14:18.832 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.825+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2987551587}) 2024-09-16T04:14:18.832 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.825+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.832 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.825+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3636725116}) 2024-09-16T04:14:18.833 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.825+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1404399770}) 2024-09-16T04:14:18.834 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.829+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.835 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.829+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3462798615}) 2024-09-16T04:14:18.835 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.829+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=194763517}) 2024-09-16T04:14:18.835 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.829+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.835 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.829+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3462798615}) 2024-09-16T04:14:18.835 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.829+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=609333628}) 2024-09-16T04:14:18.836 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.829+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.836 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.829+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2149096497}) 2024-09-16T04:14:18.836 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.829+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=51652673}) 2024-09-16T04:14:18.837 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.829+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.837 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.829+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2149096497}) 2024-09-16T04:14:18.837 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.829+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1895521018}) 2024-09-16T04:14:18.838 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.833+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.838 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.833+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2677146134}) 2024-09-16T04:14:18.838 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.833+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4011404288}) 2024-09-16T04:14:18.838 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.833+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.838 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.833+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2677146134}) 2024-09-16T04:14:18.839 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.833+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1018391389}) 2024-09-16T04:14:18.839 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.833+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.840 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.833+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=314490512}) 2024-09-16T04:14:18.840 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.833+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1896534881}) 2024-09-16T04:14:18.840 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.833+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.840 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.833+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=314490512}) 2024-09-16T04:14:18.840 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.833+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=680665459}) 2024-09-16T04:14:18.841 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.837+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.841 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.837+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4181012681}) 2024-09-16T04:14:18.842 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.837+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1672822007}) 2024-09-16T04:14:18.842 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.837+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.842 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.837+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4181012681}) 2024-09-16T04:14:18.842 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.837+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=618124404}) 2024-09-16T04:14:18.843 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.837+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.843 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.837+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1490211498}) 2024-09-16T04:14:18.843 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.837+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2241530608}) 2024-09-16T04:14:18.843 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.837+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.844 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.837+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1490211498}) 2024-09-16T04:14:18.844 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.837+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3820111187}) 2024-09-16T04:14:18.845 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.837+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.845 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.837+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2792209581}) 2024-09-16T04:14:18.845 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.837+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2882783040}) 2024-09-16T04:14:18.845 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.837+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.845 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.837+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2792209581}) 2024-09-16T04:14:18.845 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.837+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1585095650}) 2024-09-16T04:14:18.846 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.841+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.847 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.841+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=232007071}) 2024-09-16T04:14:18.847 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.841+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2562868760}) 2024-09-16T04:14:18.847 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.841+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.847 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.841+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=232007071}) 2024-09-16T04:14:18.847 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.841+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3246865106}) 2024-09-16T04:14:18.849 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.841+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.849 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.841+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=506774121}) 2024-09-16T04:14:18.849 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.841+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=835807952}) 2024-09-16T04:14:18.849 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.841+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.850 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.841+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=506774121}) 2024-09-16T04:14:18.850 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.841+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2440070256}) 2024-09-16T04:14:18.851 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.845+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.851 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.845+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4172011285}) 2024-09-16T04:14:18.851 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.845+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4183534760}) 2024-09-16T04:14:18.851 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.845+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.851 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.845+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4172011285}) 2024-09-16T04:14:18.851 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.845+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3623362594}) 2024-09-16T04:14:18.853 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.845+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.853 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.845+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3233244770}) 2024-09-16T04:14:18.853 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.845+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1889616048}) 2024-09-16T04:14:18.853 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.845+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.853 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.845+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3233244770}) 2024-09-16T04:14:18.853 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.845+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1493545114}) 2024-09-16T04:14:18.854 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.849+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.854 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.849+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3937992838}) 2024-09-16T04:14:18.854 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.849+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4179744150}) 2024-09-16T04:14:18.854 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.849+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.854 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.849+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3937992838}) 2024-09-16T04:14:18.854 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.849+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1717362033}) 2024-09-16T04:14:18.856 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.849+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.856 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.849+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4137903552}) 2024-09-16T04:14:18.856 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.849+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2571265652}) 2024-09-16T04:14:18.856 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.849+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.856 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.849+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4137903552}) 2024-09-16T04:14:18.856 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.849+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2244599100}) 2024-09-16T04:14:18.858 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.853+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.858 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.853+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1728383641}) 2024-09-16T04:14:18.858 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.853+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1853047110}) 2024-09-16T04:14:18.858 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.853+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.858 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.853+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1728383641}) 2024-09-16T04:14:18.858 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.853+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3897793631}) 2024-09-16T04:14:18.859 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.853+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.860 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.853+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3900868747}) 2024-09-16T04:14:18.864 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.853+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=165794106}) 2024-09-16T04:14:18.864 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.853+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.864 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.853+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3900868747}) 2024-09-16T04:14:18.864 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.853+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2526114646}) 2024-09-16T04:14:18.864 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.857+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.864 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.857+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=554335452}) 2024-09-16T04:14:18.864 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.857+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3151366832}) 2024-09-16T04:14:18.864 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.857+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.864 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.857+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=554335452}) 2024-09-16T04:14:18.865 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.857+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1020780618}) 2024-09-16T04:14:18.865 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.857+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.865 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.857+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=293478713}) 2024-09-16T04:14:18.865 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.857+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1106358347}) 2024-09-16T04:14:18.865 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.857+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.865 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.857+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=293478713}) 2024-09-16T04:14:18.865 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.857+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2319018567}) 2024-09-16T04:14:18.866 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.857+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.866 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.857+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3654228811}) 2024-09-16T04:14:18.866 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.857+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=38625470}) 2024-09-16T04:14:18.866 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.857+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.866 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.857+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3654228811}) 2024-09-16T04:14:18.866 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.857+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2845125526}) 2024-09-16T04:14:18.866 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.861+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.867 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.861+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3293209640}) 2024-09-16T04:14:18.867 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.861+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3945184030}) 2024-09-16T04:14:18.867 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.861+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.867 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.861+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3293209640}) 2024-09-16T04:14:18.867 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.861+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4032843907}) 2024-09-16T04:14:18.868 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.861+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.869 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.861+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3916618432}) 2024-09-16T04:14:18.869 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.861+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4247686761}) 2024-09-16T04:14:18.869 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.861+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.869 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.861+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3916618432}) 2024-09-16T04:14:18.869 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.861+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=668503067}) 2024-09-16T04:14:18.871 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.865+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.871 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.865+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2234263054}) 2024-09-16T04:14:18.871 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.865+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3924403542}) 2024-09-16T04:14:18.871 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.865+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.871 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.865+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2234263054}) 2024-09-16T04:14:18.872 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.865+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=134892493}) 2024-09-16T04:14:18.873 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.865+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.873 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.865+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2275965525}) 2024-09-16T04:14:18.873 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.865+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4174369956}) 2024-09-16T04:14:18.873 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.865+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.874 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.865+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2275965525}) 2024-09-16T04:14:18.874 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.865+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=757702967}) 2024-09-16T04:14:18.874 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.869+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.874 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.869+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3765657877}) 2024-09-16T04:14:18.874 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.869+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1080004525}) 2024-09-16T04:14:18.874 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.869+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.874 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.869+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3765657877}) 2024-09-16T04:14:18.875 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.869+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1736354699}) 2024-09-16T04:14:18.876 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.869+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.876 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.869+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3534136385}) 2024-09-16T04:14:18.876 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.869+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2093284521}) 2024-09-16T04:14:18.876 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.869+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.876 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.869+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3534136385}) 2024-09-16T04:14:18.876 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.869+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=708974827}) 2024-09-16T04:14:18.878 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.873+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.878 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.873+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2061432307}) 2024-09-16T04:14:18.878 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.873+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1902380667}) 2024-09-16T04:14:18.878 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.873+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.878 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.873+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2061432307}) 2024-09-16T04:14:18.878 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.873+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4250053377}) 2024-09-16T04:14:18.880 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.873+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.880 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.873+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1689113305}) 2024-09-16T04:14:18.880 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.873+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2496027474}) 2024-09-16T04:14:18.880 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.873+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.880 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.873+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1689113305}) 2024-09-16T04:14:18.880 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.873+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3072956341}) 2024-09-16T04:14:18.882 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.877+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.882 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.877+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3893851189}) 2024-09-16T04:14:18.882 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.877+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3423696422}) 2024-09-16T04:14:18.882 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.877+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.882 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.877+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3893851189}) 2024-09-16T04:14:18.882 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.877+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=190412801}) 2024-09-16T04:14:18.884 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.877+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.884 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.877+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=652045251}) 2024-09-16T04:14:18.884 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.877+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2692033786}) 2024-09-16T04:14:18.884 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.877+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.884 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.877+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=652045251}) 2024-09-16T04:14:18.884 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.877+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1305071265}) 2024-09-16T04:14:18.886 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.881+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.886 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.881+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3531173434}) 2024-09-16T04:14:18.886 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.881+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1283936968}) 2024-09-16T04:14:18.886 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.881+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.886 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.881+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3531173434}) 2024-09-16T04:14:18.886 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.881+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4219862995}) 2024-09-16T04:14:18.887 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.881+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.887 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.881+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2700829168}) 2024-09-16T04:14:18.887 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.881+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1214083986}) 2024-09-16T04:14:18.887 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.881+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.887 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.881+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2700829168}) 2024-09-16T04:14:18.888 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.881+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3778547071}) 2024-09-16T04:14:18.889 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.885+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.889 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.885+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=497485140}) 2024-09-16T04:14:18.889 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.885+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1669473221}) 2024-09-16T04:14:18.889 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.885+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.889 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.885+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=497485140}) 2024-09-16T04:14:18.889 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.885+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2553947357}) 2024-09-16T04:14:18.891 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.885+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.891 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.885+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=135937767}) 2024-09-16T04:14:18.891 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.885+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2494875502}) 2024-09-16T04:14:18.891 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.885+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.891 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.885+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=135937767}) 2024-09-16T04:14:18.891 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.885+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3903683312}) 2024-09-16T04:14:18.893 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.889+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.893 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.889+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=768769868}) 2024-09-16T04:14:18.894 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.889+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=24705604}) 2024-09-16T04:14:18.894 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.889+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.894 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.889+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=768769868}) 2024-09-16T04:14:18.894 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.889+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=554568874}) 2024-09-16T04:14:18.895 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.889+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.896 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.889+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1289053295}) 2024-09-16T04:14:18.896 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.889+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3180069639}) 2024-09-16T04:14:18.896 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.889+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.896 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.889+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1289053295}) 2024-09-16T04:14:18.896 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.889+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4172429869}) 2024-09-16T04:14:18.897 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.889+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.897 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.889+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3521007656}) 2024-09-16T04:14:18.897 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.889+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2524874032}) 2024-09-16T04:14:18.898 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.893+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.898 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.893+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3521007656}) 2024-09-16T04:14:18.898 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.893+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=372758749}) 2024-09-16T04:14:18.899 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.893+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.899 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.893+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4234915948}) 2024-09-16T04:14:18.899 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.893+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=905509775}) 2024-09-16T04:14:18.899 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.893+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.899 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.893+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4234915948}) 2024-09-16T04:14:18.899 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.893+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2209487093}) 2024-09-16T04:14:18.900 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.893+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.901 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.893+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4117098958}) 2024-09-16T04:14:18.901 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.893+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1031362435}) 2024-09-16T04:14:18.901 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.893+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.901 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.893+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4117098958}) 2024-09-16T04:14:18.901 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.893+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4104146654}) 2024-09-16T04:14:18.902 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.897+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.902 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.897+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=27344657}) 2024-09-16T04:14:18.903 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.897+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=828623818}) 2024-09-16T04:14:18.903 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.897+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.903 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.897+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=27344657}) 2024-09-16T04:14:18.903 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.897+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=266212226}) 2024-09-16T04:14:18.904 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.897+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.904 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.897+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2606021618}) 2024-09-16T04:14:18.904 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.897+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3612859005}) 2024-09-16T04:14:18.904 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.897+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.905 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.897+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2606021618}) 2024-09-16T04:14:18.905 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.897+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2908402051}) 2024-09-16T04:14:18.907 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.901+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.907 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.901+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4253487938}) 2024-09-16T04:14:18.907 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.901+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=192079729}) 2024-09-16T04:14:18.907 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.901+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.907 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.901+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4253487938}) 2024-09-16T04:14:18.908 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.901+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=79278627}) 2024-09-16T04:14:18.908 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.901+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.908 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.901+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2151029143}) 2024-09-16T04:14:18.908 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.901+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4051357817}) 2024-09-16T04:14:18.909 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.901+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.909 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.901+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2151029143}) 2024-09-16T04:14:18.909 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.901+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=890419563}) 2024-09-16T04:14:18.910 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.905+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.910 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.905+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2094729059}) 2024-09-16T04:14:18.911 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.905+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1080792978}) 2024-09-16T04:14:18.911 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.905+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.911 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.905+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2094729059}) 2024-09-16T04:14:18.911 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.905+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3413248302}) 2024-09-16T04:14:18.912 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.905+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.912 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.905+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1084291859}) 2024-09-16T04:14:18.913 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.905+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=482986082}) 2024-09-16T04:14:18.913 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.905+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.913 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.905+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1084291859}) 2024-09-16T04:14:18.913 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.905+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3990281636}) 2024-09-16T04:14:18.914 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.909+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.915 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.909+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=379432444}) 2024-09-16T04:14:18.915 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.909+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=576417217}) 2024-09-16T04:14:18.915 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.909+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.915 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.909+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=379432444}) 2024-09-16T04:14:18.915 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.909+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3177517512}) 2024-09-16T04:14:18.916 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.909+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.917 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.909+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=41212462}) 2024-09-16T04:14:18.917 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.909+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2144544710}) 2024-09-16T04:14:18.917 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.909+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.918 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.909+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=41212462}) 2024-09-16T04:14:18.918 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.909+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2093411981}) 2024-09-16T04:14:18.920 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.913+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.920 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.913+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3763800594}) 2024-09-16T04:14:18.921 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.913+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1035854782}) 2024-09-16T04:14:18.921 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.913+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.921 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.913+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3763800594}) 2024-09-16T04:14:18.921 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.913+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1979931018}) 2024-09-16T04:14:18.921 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.913+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.921 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.913+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2724385410}) 2024-09-16T04:14:18.922 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.913+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=797901944}) 2024-09-16T04:14:18.922 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.913+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.922 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.913+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2724385410}) 2024-09-16T04:14:18.922 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.913+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1643620389}) 2024-09-16T04:14:18.925 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.917+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.925 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.917+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1079029561}) 2024-09-16T04:14:18.925 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.917+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1799064198}) 2024-09-16T04:14:18.926 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.917+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.926 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.917+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1079029561}) 2024-09-16T04:14:18.926 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.917+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3291540965}) 2024-09-16T04:14:18.926 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.917+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.926 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.917+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1603347414}) 2024-09-16T04:14:18.927 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.917+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=103035039}) 2024-09-16T04:14:18.927 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.917+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.927 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.917+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1603347414}) 2024-09-16T04:14:18.927 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.917+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=168980569}) 2024-09-16T04:14:18.927 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.917+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.927 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.917+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1802623816}) 2024-09-16T04:14:18.928 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.917+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1457587340}) 2024-09-16T04:14:18.928 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.917+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.928 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.921+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1802623816}) 2024-09-16T04:14:18.928 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.921+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2508054631}) 2024-09-16T04:14:18.933 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.921+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.933 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.921+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3306777797}) 2024-09-16T04:14:18.933 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.921+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1622816131}) 2024-09-16T04:14:18.933 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.921+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.934 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.921+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3306777797}) 2024-09-16T04:14:18.934 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.921+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2525691520}) 2024-09-16T04:14:18.934 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.921+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.934 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.921+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=562512982}) 2024-09-16T04:14:18.934 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.921+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=415104070}) 2024-09-16T04:14:18.934 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.921+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.934 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.921+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=562512982}) 2024-09-16T04:14:18.934 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.921+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2169932881}) 2024-09-16T04:14:18.934 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.925+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.934 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.925+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3697771056}) 2024-09-16T04:14:18.934 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.925+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=672195278}) 2024-09-16T04:14:18.935 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.925+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.935 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.925+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3697771056}) 2024-09-16T04:14:18.935 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.925+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3651982135}) 2024-09-16T04:14:18.935 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.925+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.935 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.925+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3507656515}) 2024-09-16T04:14:18.935 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.925+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1033541670}) 2024-09-16T04:14:18.935 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.925+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.935 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.925+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3507656515}) 2024-09-16T04:14:18.935 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.925+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=216693831}) 2024-09-16T04:14:18.936 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.929+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.936 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.929+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=991037801}) 2024-09-16T04:14:18.937 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.929+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2398417204}) 2024-09-16T04:14:18.937 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.929+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.937 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.929+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=991037801}) 2024-09-16T04:14:18.937 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.929+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3310926412}) 2024-09-16T04:14:18.937 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.929+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.937 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.929+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=150823216}) 2024-09-16T04:14:18.937 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.929+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=837648920}) 2024-09-16T04:14:18.937 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.929+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.937 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.929+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=150823216}) 2024-09-16T04:14:18.937 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.929+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=582138707}) 2024-09-16T04:14:18.938 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.933+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.938 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.933+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1066125837}) 2024-09-16T04:14:18.938 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.933+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4003177524}) 2024-09-16T04:14:18.938 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.933+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.939 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.933+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1066125837}) 2024-09-16T04:14:18.939 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.933+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=366451015}) 2024-09-16T04:14:18.939 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.933+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.940 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.933+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1631133445}) 2024-09-16T04:14:18.940 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.933+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1287069631}) 2024-09-16T04:14:18.940 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.933+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.940 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.933+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1631133445}) 2024-09-16T04:14:18.940 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.933+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3786894378}) 2024-09-16T04:14:18.941 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.937+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.941 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.937+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1354909897}) 2024-09-16T04:14:18.941 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.937+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2451269406}) 2024-09-16T04:14:18.942 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.937+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.942 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.937+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1354909897}) 2024-09-16T04:14:18.942 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.937+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3109229343}) 2024-09-16T04:14:18.943 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.937+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.943 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.937+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3694232518}) 2024-09-16T04:14:18.943 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.937+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=951080137}) 2024-09-16T04:14:18.943 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.937+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.943 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.937+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3694232518}) 2024-09-16T04:14:18.944 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.937+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3967667191}) 2024-09-16T04:14:18.945 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.937+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.945 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.941+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1121781522}) 2024-09-16T04:14:18.945 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.941+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2674185950}) 2024-09-16T04:14:18.945 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.941+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.945 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.941+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1121781522}) 2024-09-16T04:14:18.945 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.941+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2212492360}) 2024-09-16T04:14:18.947 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.941+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.947 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.941+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2919281050}) 2024-09-16T04:14:18.947 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.941+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3024635869}) 2024-09-16T04:14:18.947 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.941+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.947 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.941+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2919281050}) 2024-09-16T04:14:18.947 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.941+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2543889580}) 2024-09-16T04:14:18.949 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.945+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.949 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.945+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2242018330}) 2024-09-16T04:14:18.949 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.945+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2025299252}) 2024-09-16T04:14:18.950 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.945+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.950 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.945+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2242018330}) 2024-09-16T04:14:18.950 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.945+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=361463218}) 2024-09-16T04:14:18.952 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.945+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.952 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.945+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3890808507}) 2024-09-16T04:14:18.952 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.945+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3464366888}) 2024-09-16T04:14:18.952 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.945+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.952 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.945+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3890808507}) 2024-09-16T04:14:18.952 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.945+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3773092952}) 2024-09-16T04:14:18.954 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.949+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.954 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.949+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1801233274}) 2024-09-16T04:14:18.954 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.949+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1878929620}) 2024-09-16T04:14:18.954 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.949+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.954 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.949+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1801233274}) 2024-09-16T04:14:18.954 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.949+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1855776412}) 2024-09-16T04:14:18.956 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.949+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.956 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.949+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=501878276}) 2024-09-16T04:14:18.956 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.949+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3455317981}) 2024-09-16T04:14:18.956 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.949+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.956 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.949+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=501878276}) 2024-09-16T04:14:18.956 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.949+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2546756866}) 2024-09-16T04:14:18.958 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.953+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.958 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.953+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4010145645}) 2024-09-16T04:14:18.958 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.953+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1607371469}) 2024-09-16T04:14:18.958 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.953+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.958 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.953+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4010145645}) 2024-09-16T04:14:18.958 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.953+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3860908713}) 2024-09-16T04:14:18.960 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.953+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.960 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.953+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3184419570}) 2024-09-16T04:14:18.960 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.953+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2377382213}) 2024-09-16T04:14:18.960 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.953+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.960 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.953+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3184419570}) 2024-09-16T04:14:18.960 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.953+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1417626815}) 2024-09-16T04:14:18.962 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.957+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.962 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.957+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3668362844}) 2024-09-16T04:14:18.962 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.957+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2400572887}) 2024-09-16T04:14:18.962 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.957+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.962 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.957+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3668362844}) 2024-09-16T04:14:18.962 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.957+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1312901325}) 2024-09-16T04:14:18.964 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.957+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.964 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.957+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3658242906}) 2024-09-16T04:14:18.964 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.957+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=61896557}) 2024-09-16T04:14:18.965 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.957+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.965 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.957+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3658242906}) 2024-09-16T04:14:18.965 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.957+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4185854580}) 2024-09-16T04:14:18.967 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.961+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.967 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.961+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2169807478}) 2024-09-16T04:14:18.967 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.961+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=819082873}) 2024-09-16T04:14:18.967 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.961+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.967 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.961+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2169807478}) 2024-09-16T04:14:18.967 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.961+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1938195929}) 2024-09-16T04:14:18.969 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.961+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.969 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.961+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=255620176}) 2024-09-16T04:14:18.969 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.961+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2155908664}) 2024-09-16T04:14:18.969 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.961+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.969 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.961+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=255620176}) 2024-09-16T04:14:18.969 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.961+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1257583500}) 2024-09-16T04:14:18.971 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.965+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.971 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.965+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2198177032}) 2024-09-16T04:14:18.971 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.965+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1778968965}) 2024-09-16T04:14:18.971 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.965+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.971 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.965+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2198177032}) 2024-09-16T04:14:18.971 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.965+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3225808663}) 2024-09-16T04:14:18.973 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.965+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.973 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.965+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=229668885}) 2024-09-16T04:14:18.973 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.965+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4054611669}) 2024-09-16T04:14:18.973 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.969+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.973 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.969+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=229668885}) 2024-09-16T04:14:18.973 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.969+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2663813787}) 2024-09-16T04:14:18.975 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.969+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.975 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.969+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=702717637}) 2024-09-16T04:14:18.975 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.969+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4020208922}) 2024-09-16T04:14:18.975 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.969+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.975 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.969+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=702717637}) 2024-09-16T04:14:18.975 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.969+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=459940031}) 2024-09-16T04:14:18.977 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.969+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.977 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.969+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3693029615}) 2024-09-16T04:14:18.977 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.969+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2370739776}) 2024-09-16T04:14:18.977 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.969+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.977 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.969+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3693029615}) 2024-09-16T04:14:18.977 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.969+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1210523646}) 2024-09-16T04:14:18.978 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.973+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.979 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.973+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=171364261}) 2024-09-16T04:14:18.979 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.973+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2303433570}) 2024-09-16T04:14:18.979 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.973+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.979 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.973+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=171364261}) 2024-09-16T04:14:18.979 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.973+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1563167275}) 2024-09-16T04:14:18.981 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.973+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.981 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.973+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4149526266}) 2024-09-16T04:14:18.981 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.973+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=648955691}) 2024-09-16T04:14:18.981 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.973+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.981 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.973+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4149526266}) 2024-09-16T04:14:18.981 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.973+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1051493099}) 2024-09-16T04:14:18.982 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.977+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.982 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.977+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=340851084}) 2024-09-16T04:14:18.983 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.977+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2810611703}) 2024-09-16T04:14:18.983 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.977+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.983 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.977+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=340851084}) 2024-09-16T04:14:18.983 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.977+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=659518431}) 2024-09-16T04:14:18.984 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.977+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.984 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.977+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=919659289}) 2024-09-16T04:14:18.985 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.977+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=435937456}) 2024-09-16T04:14:18.985 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.977+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.985 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.977+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=919659289}) 2024-09-16T04:14:18.985 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.977+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4007396629}) 2024-09-16T04:14:18.986 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.981+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.986 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.981+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2645149376}) 2024-09-16T04:14:18.986 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.981+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3665931481}) 2024-09-16T04:14:18.987 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.981+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.987 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.981+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2645149376}) 2024-09-16T04:14:18.987 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.981+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1840695132}) 2024-09-16T04:14:18.989 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.981+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.989 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.981+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=698545394}) 2024-09-16T04:14:18.989 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.981+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3709629030}) 2024-09-16T04:14:18.989 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.981+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.989 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.981+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=698545394}) 2024-09-16T04:14:18.989 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.981+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3309947167}) 2024-09-16T04:14:18.990 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.985+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.990 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.985+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2433854877}) 2024-09-16T04:14:18.990 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.985+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3903490590}) 2024-09-16T04:14:18.991 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.985+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.991 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.985+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2433854877}) 2024-09-16T04:14:18.991 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.985+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1098141031}) 2024-09-16T04:14:18.992 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.985+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.992 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.985+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=74161943}) 2024-09-16T04:14:18.992 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.985+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3529960389}) 2024-09-16T04:14:18.992 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.985+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.993 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.985+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=74161943}) 2024-09-16T04:14:18.993 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.985+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1661710113}) 2024-09-16T04:14:18.995 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.989+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.995 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.989+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3459082240}) 2024-09-16T04:14:18.995 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.989+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1559829138}) 2024-09-16T04:14:18.995 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.989+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.995 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.989+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3459082240}) 2024-09-16T04:14:18.995 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.989+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1269395645}) 2024-09-16T04:14:18.997 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.989+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.997 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.989+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3510777991}) 2024-09-16T04:14:18.997 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.989+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=942197307}) 2024-09-16T04:14:18.997 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.993+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.997 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.993+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3510777991}) 2024-09-16T04:14:18.997 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.993+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3339565283}) 2024-09-16T04:14:18.999 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.993+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:18.999 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.993+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2309513263}) 2024-09-16T04:14:18.999 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.993+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2110024361}) 2024-09-16T04:14:18.999 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.993+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:19.000 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.993+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2309513263}) 2024-09-16T04:14:19.000 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.993+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=777484930}) 2024-09-16T04:14:19.002 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.997+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:19.002 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.997+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=219885024}) 2024-09-16T04:14:19.002 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.997+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3224718086}) 2024-09-16T04:14:19.002 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.997+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:19.002 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.997+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=219885024}) 2024-09-16T04:14:19.002 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.997+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2265433984}) 2024-09-16T04:14:19.003 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.997+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:19.003 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.997+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1993547060}) 2024-09-16T04:14:19.004 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.997+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=96220630}) 2024-09-16T04:14:19.004 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.997+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:19.004 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.997+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1993547060}) 2024-09-16T04:14:19.004 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:18.997+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1967531015}) 2024-09-16T04:14:19.006 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:19.001+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:19.006 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:19.001+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2495760718}) 2024-09-16T04:14:19.006 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:19.001+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4192124514}) 2024-09-16T04:14:19.006 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:19.001+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:19.006 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:19.001+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2495760718}) 2024-09-16T04:14:19.006 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:19.001+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1219542201}) 2024-09-16T04:14:19.008 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:19.001+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:19.008 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:19.001+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3801720101}) 2024-09-16T04:14:19.008 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:19.001+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3282253104}) 2024-09-16T04:14:19.008 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:19.001+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:19.008 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:19.001+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3801720101}) 2024-09-16T04:14:19.008 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:19.001+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2712650553}) 2024-09-16T04:14:19.010 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:19.005+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:19.010 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:19.005+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1140378217}) 2024-09-16T04:14:19.010 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:19.005+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2939808773}) 2024-09-16T04:14:19.010 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:19.005+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:19.010 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:19.005+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1140378217}) 2024-09-16T04:14:19.010 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:19.005+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1540983149}) 2024-09-16T04:14:19.012 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:19.005+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:19.012 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:19.005+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2683888869}) 2024-09-16T04:14:19.012 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:19.005+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3775810467}) 2024-09-16T04:14:19.013 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:19.005+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:19.013 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:19.005+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2683888869}) 2024-09-16T04:14:19.013 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:19.005+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1943536617}) 2024-09-16T04:14:19.015 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:19.009+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:19.015 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:19.009+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=235452613}) 2024-09-16T04:14:19.015 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:19.009+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2946381319}) 2024-09-16T04:14:19.016 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:19.009+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:19.016 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:19.009+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=235452613}) 2024-09-16T04:14:19.016 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:19.009+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=986034850}) 2024-09-16T04:14:19.017 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:19.009+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:19.018 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:19.009+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3778642070}) 2024-09-16T04:14:19.018 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:19.009+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4232657060}) 2024-09-16T04:14:19.018 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:19.009+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:19.018 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:19.009+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3778642070}) 2024-09-16T04:14:19.018 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:19.009+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2823997194}) 2024-09-16T04:14:19.019 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:19.013+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:19.019 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:19.013+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1462281675}) 2024-09-16T04:14:19.020 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:19.013+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=629762449}) 2024-09-16T04:14:19.020 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:19.013+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:19.020 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:19.013+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1462281675}) 2024-09-16T04:14:19.020 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:19.013+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1124756443}) 2024-09-16T04:14:19.021 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:19.013+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:19.021 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:19.013+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=544711395}) 2024-09-16T04:14:19.021 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:19.013+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2727311152}) 2024-09-16T04:14:19.021 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:19.013+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:19.022 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:19.013+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=544711395}) 2024-09-16T04:14:19.022 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:19.013+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3348632760}) 2024-09-16T04:14:19.023 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:19.017+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:19.023 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:19.017+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=659570768}) 2024-09-16T04:14:19.023 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:19.017+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2147858141}) 2024-09-16T04:14:19.023 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:19.017+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:19.023 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:19.017+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=659570768}) 2024-09-16T04:14:19.023 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:19.017+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1323691776}) 2024-09-16T04:14:19.024 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:19.017+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:19.024 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:19.017+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=821616213}) 2024-09-16T04:14:19.025 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:19.017+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=409595149}) 2024-09-16T04:14:19.025 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:19.017+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:19.025 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:19.017+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=821616213}) 2024-09-16T04:14:19.025 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:19.017+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1433177652}) 2024-09-16T04:14:19.026 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:19.021+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:19.026 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:19.021+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=538440888}) 2024-09-16T04:14:19.026 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:19.021+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2930364408}) 2024-09-16T04:14:19.027 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:19.021+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:19.027 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:19.021+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=538440888}) 2024-09-16T04:14:19.027 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:19.021+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3771083271}) 2024-09-16T04:14:19.029 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:19.021+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:19.029 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:19.021+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3741598728}) 2024-09-16T04:14:19.029 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:19.021+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3767232482}) 2024-09-16T04:14:19.029 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:19.021+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:19.029 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:19.021+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3741598728}) 2024-09-16T04:14:19.029 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:19.021+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2333332351}) 2024-09-16T04:14:19.030 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:19.025+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:19.030 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:19.025+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=468006106}) 2024-09-16T04:14:19.031 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:19.025+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3268826704}) 2024-09-16T04:14:19.031 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:19.025+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:19.031 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:19.025+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=468006106}) 2024-09-16T04:14:19.031 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:19.025+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2821234461}) 2024-09-16T04:14:19.033 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:19.025+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:19.033 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:19.025+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4273463708}) 2024-09-16T04:14:19.033 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:19.025+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2871777452}) 2024-09-16T04:14:19.033 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:19.025+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:19.033 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:19.025+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4273463708}) 2024-09-16T04:14:19.033 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:19.025+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1429364637}) 2024-09-16T04:14:19.035 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:19.029+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:19.035 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:19.029+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=770194962}) 2024-09-16T04:14:19.035 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:19.029+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2405724135}) 2024-09-16T04:14:19.035 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:19.029+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:19.035 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:19.029+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=770194962}) 2024-09-16T04:14:19.035 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:19.029+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3482721828}) 2024-09-16T04:14:19.037 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:19.029+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:19.037 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:19.029+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1503184339}) 2024-09-16T04:14:19.037 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:19.029+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=564179949}) 2024-09-16T04:14:19.037 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:19.029+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:19.037 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:19.029+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1503184339}) 2024-09-16T04:14:19.037 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:19.029+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3694050760}) 2024-09-16T04:14:19.039 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:19.033+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:19.039 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:19.033+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=971669820}) 2024-09-16T04:14:19.039 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:19.033+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4019414751}) 2024-09-16T04:14:19.039 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:19.033+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:19.039 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:19.033+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=971669820}) 2024-09-16T04:14:19.039 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:19.033+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=547049397}) 2024-09-16T04:14:19.041 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:19.037+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:19.041 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:19.037+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1069880852}) 2024-09-16T04:14:19.041 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:19.037+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4145710592}) 2024-09-16T04:14:19.041 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:19.037+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:19.041 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:19.037+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1069880852}) 2024-09-16T04:14:19.041 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:19.037+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2973193835}) 2024-09-16T04:14:19.043 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:19.037+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:19.043 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:19.037+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4249608706}) 2024-09-16T04:14:19.043 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:19.037+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=302902719}) 2024-09-16T04:14:19.043 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:19.037+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:19.043 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:19.037+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4249608706}) 2024-09-16T04:14:19.043 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:19.037+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=303978841}) 2024-09-16T04:14:19.046 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:19.041+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:19.046 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:19.041+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=90,mdsmap=3,mgr=7} crc {logm=860148239,mdsmap=4088524775,mgr=390803354}) 2024-09-16T04:14:19.046 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:19.041+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=85,mdsmap=2,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=4} crc {logm=1762001788,mdsmap=210808725,mgr=390803354,mgr_command_descs=2107673600,mgr_metadata=1367272202,mgrstat=1321109519}) 2024-09-16T04:14:19.046 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:19.041+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:19.046 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:19.041+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=90,mdsmap=3,mgr=7} crc {logm=860148239,mdsmap=4088524775,mgr=390803354}) 2024-09-16T04:14:19.046 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:19.041+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=86,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=1} crc {logm=1665897338,mdsmap=3762917140,mgr=460109776,mgr_command_descs=2107673600,mgr_metadata=1367272202,mgrstat=3194923591}) 2024-09-16T04:14:19.048 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:19.041+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:19.048 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:19.041+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgr=1,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=11,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=41,osdmap=26} crc {mgr=1747058506,mgr_command_descs=2107673600,mgr_metadata=1367272202,mgrstat=1976238721,mon_config_key=85057112,monmap=415087619,osd_metadata=3450683029,osd_pg_creating=1182525037,osd_snap=3010516726,osdmap=2496131089}) 2024-09-16T04:14:19.049 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:19.041+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mgr=1,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=10,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=41,osdmap=27} crc {mgr=1747058506,mgr_command_descs=2107673600,mgr_metadata=1367272202,mgrstat=295396518,mon_config_key=3448559827,monmap=3937513981,osd_metadata=4182959540,osd_pg_creating=1182525037,osd_snap=3864227582,osdmap=4258563141}) 2024-09-16T04:14:19.049 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:19.041+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:19.049 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:19.041+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgr=1,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=11,monmap=3,osd_metadata=5,osd_pg_creating=1,osd_snap=41,osdmap=26} crc {mgr=1747058506,mgr_command_descs=2107673600,mgr_metadata=1367272202,mgrstat=1976238721,mon_config_key=85057112,monmap=415087619,osd_metadata=3450683029,osd_pg_creating=1182525037,osd_snap=3010516726,osdmap=2496131089}) 2024-09-16T04:14:19.049 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:19.041+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mgr=1,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=11,monmap=3,osd_metadata=6,osd_pg_creating=1,osd_snap=41,osdmap=25} crc {mgr=1747058506,mgr_command_descs=2107673600,mgr_metadata=1367272202,mgrstat=3084660914,mon_config_key=85057112,monmap=3937513981,osd_metadata=4182959540,osd_pg_creating=1182525037,osd_snap=938030321,osdmap=4045426086}) 2024-09-16T04:14:19.063 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:19.057+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:19.064 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:19.057+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=36} crc {osdmap=2655254459}) 2024-09-16T04:14:19.064 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:19.057+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=37} crc {osdmap=1613243850}) 2024-09-16T04:14:19.064 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:19.057+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-16T04:14:19.064 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:19.057+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=36} crc {osdmap=2655254459}) 2024-09-16T04:14:19.064 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-09-16T04:14:19.057+0000 7f82cd97c640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=36} crc {osdmap=1607467609}) 2024-09-16T04:14:19.799 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-16T04:14:19.799 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-16T04:14:19.930 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.925+0000 7fb09b1b7640 10 monclient: get_monmap_and_config 2024-09-16T04:14:19.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.925+0000 7fb09b1b7640 10 monclient: build_initial_monmap 2024-09-16T04:14:19.931 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.925+0000 7fb09b1b7640 10 monclient: monmap: 2024-09-16T04:14:19.931 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:14:19.931 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:14:19.931 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:14:19.929703+0000 2024-09-16T04:14:19.931 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:14:19.929703+0000 2024-09-16T04:14:19.931 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:14:19.931 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:19.931 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:14:19.931 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:14:19.932 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:14:19.932 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:14:19.932 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:14:19.932 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:19.934 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.925+0000 7fb09b1b7640 1 Processor -- start 2024-09-16T04:14:19.934 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.925+0000 7fb09b1b7640 1 -- start start 2024-09-16T04:14:19.934 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.925+0000 7fb09b1b7640 10 monclient: init 2024-09-16T04:14:19.934 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.925+0000 7fb09b1b7640 10 monclient: _reopen_session rank -1 2024-09-16T04:14:19.934 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.925+0000 7fb09b1b7640 10 monclient: _add_conns ranks=[3,1,2,4,0] 2024-09-16T04:14:19.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.925+0000 7fb09b1b7640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb0940a1330 0x7fb0940a1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:19.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.925+0000 7fb09b1b7640 10 monclient(hunting): picked mon.noname-a con 0x7fb0940a1330 addr v2:172.21.15.42:3300/0 2024-09-16T04:14:19.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.925+0000 7fb09b1b7640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb09406dac0 addr v1:172.21.15.154:6789/0 2024-09-16T04:14:19.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.925+0000 7fb09b1b7640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb0940a1c50 0x7fb094135810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:19.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.925+0000 7fb09b1b7640 10 monclient(hunting): picked mon.noname-b con 0x7fb0940a1c50 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:14:19.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.925+0000 7fb09b1b7640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb09409c090 con 0x7fb09406dac0 2024-09-16T04:14:19.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.925+0000 7fb09b1b7640 10 monclient(hunting): start opening mon connection 2024-09-16T04:14:19.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.925+0000 7fb09b1b7640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7fb09409f610 con 0x7fb0940a1330 2024-09-16T04:14:19.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.925+0000 7fb09b1b7640 10 monclient(hunting): start opening mon connection 2024-09-16T04:14:19.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.925+0000 7fb09b1b7640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fb09409f8f0 con 0x7fb0940a1c50 2024-09-16T04:14:19.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.925+0000 7fb09b1b7640 10 monclient(hunting): _renew_subs 2024-09-16T04:14:19.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.925+0000 7fb09b1b7640 10 monclient(hunting): authenticate will time out at 2906.899184s 2024-09-16T04:14:19.935 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.925+0000 7fb098f53640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb0940a1330 0x7fb0940a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:19.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.925+0000 7fb099754640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb0940a1c50 0x7fb094135810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:19.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.925+0000 7fb099754640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb0940a1c50 0x7fb094135810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:51838/0 (socket says 172.21.15.42:51838) 2024-09-16T04:14:19.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.925+0000 7fb098f53640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fb0940a1330 0x7fb0940a1710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:47914/0 (socket says 172.21.15.42:47914) 2024-09-16T04:14:19.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.925+0000 7fb099754640 1 -- 172.21.15.42:0/2333724426 learned_addr learned my addr 172.21.15.42:0/2333724426 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:14:19.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.925+0000 7fb099754640 10 monclient(hunting): get_auth_request con 0x7fb0940a1c50 auth_method 0 2024-09-16T04:14:19.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.925+0000 7fb099754640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:14:19.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.925+0000 7fb099754640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:19.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.925+0000 7fb099754640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:19.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.925+0000 7fb098f53640 10 monclient(hunting): get_auth_request con 0x7fb0940a1330 auth_method 0 2024-09-16T04:14:19.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.925+0000 7fb098f53640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:14:19.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.925+0000 7fb098f53640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:19.936 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.925+0000 7fb098f53640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:19.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.925+0000 7fb099754640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:14:19.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.925+0000 7fb099754640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:14:19.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.925+0000 7fb099754640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:14:19.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.925+0000 7fb098f53640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:14:19.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.925+0000 7fb098f53640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:14:19.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.925+0000 7fb098f53640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:14:19.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.925+0000 7fb099754640 10 monclient(hunting): handle_auth_done global_id 1094225 payload 274 2024-09-16T04:14:19.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.925+0000 7fb099754640 1 -- 172.21.15.42:0/2333724426 >> v2:172.21.15.42:3300/0 conn(0x7fb0940a1330 msgr2=0x7fb0940a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:14:19.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.925+0000 7fb099754640 1 --2- 172.21.15.42:0/2333724426 >> v2:172.21.15.42:3300/0 conn(0x7fb0940a1330 0x7fb0940a1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:19.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.925+0000 7fb099754640 1 -- 172.21.15.42:0/2333724426 >> v1:172.21.15.154:6789/0 conn(0x7fb09406dac0 legacy=0x7fb094135100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:14:19.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.925+0000 7fb099754640 10 monclient: _finish_hunting 0 2024-09-16T04:14:19.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.925+0000 7fb099754640 1 monclient: found mon.noname-b 2024-09-16T04:14:19.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.925+0000 7fb099754640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:14:19.937 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.925+0000 7fb099754640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T04:14:19.938 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.925+0000 7fb099754640 1 -- 172.21.15.42:0/2333724426 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb09409fbd0 con 0x7fb0940a1c50 2024-09-16T04:14:19.938 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.925+0000 7fb098f53640 1 --2- 172.21.15.42:0/2333724426 >> v2:172.21.15.42:3300/0 conn(0x7fb0940a1330 0x7fb0940a1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:14:19.947 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.929+0000 7fb099754640 1 --2- 172.21.15.42:0/2333724426 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb0940a1c50 0x7fb094135810 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fb084009870 tx=0x7fb08401c470 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c3132e7c34583c6e server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:14:19.948 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.929+0000 7fb0937fe640 1 -- 172.21.15.42:0/2333724426 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0840b0070 con 0x7fb0940a1c50 2024-09-16T04:14:19.948 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.929+0000 7fb0937fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:19.948 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.929+0000 7fb0937fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:14:19.948 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.929+0000 7fb0937fe640 10 monclient: dump: 2024-09-16T04:14:19.948 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:14:19.948 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:14:19.949 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:19.949 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:19.949 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:19.949 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:19.949 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:14:19.949 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:14:19.952 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:14:19.952 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:19.952 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.929+0000 7fb0937fe640 10 monclient: _finish_auth 0 2024-09-16T04:14:19.952 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.929+0000 7fb0937fe640 10 monclient: _check_auth_tickets 2024-09-16T04:14:19.952 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.929+0000 7fb0937fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:14:19.952 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.929+0000 7fb09b1b7640 5 monclient: authenticate success, global_id 1094225 2024-09-16T04:14:19.953 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.929+0000 7fb0937fe640 1 -- 172.21.15.42:0/2333724426 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb0840ab600 con 0x7fb0940a1c50 2024-09-16T04:14:19.953 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.929+0000 7fb09b1b7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:14:19.953 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.929+0000 7fb0937fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:14:19.953 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.929+0000 7fb09b1b7640 10 monclient: get_monmap_and_config success 2024-09-16T04:14:19.953 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.929+0000 7fb09b1b7640 4 set_mon_vals no callback set 2024-09-16T04:14:19.953 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.929+0000 7fb0937fe640 1 -- 172.21.15.42:0/2333724426 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0840ab9d0 con 0x7fb0940a1c50 2024-09-16T04:14:19.953 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.929+0000 7fb0937fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:19.953 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.929+0000 7fb0937fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:14:19.953 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.929+0000 7fb0937fe640 10 monclient: dump: 2024-09-16T04:14:19.953 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:14:19.953 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:14:19.953 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:19.953 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:19.954 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:19.954 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:19.954 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:14:19.954 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:14:19.954 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:14:19.954 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:19.954 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.929+0000 7fb09b1b7640 10 monclient: shutdown 2024-09-16T04:14:19.954 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.929+0000 7fb09b1b7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:14:19.954 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.929+0000 7fb09b1b7640 1 -- 172.21.15.42:0/2333724426 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb0940a1c50 msgr2=0x7fb094135810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:19.954 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.929+0000 7fb09b1b7640 1 --2- 172.21.15.42:0/2333724426 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb0940a1c50 0x7fb094135810 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fb084009870 tx=0x7fb08401c470 comp rx=0 tx=0).stop 2024-09-16T04:14:19.954 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.929+0000 7fb09b1b7640 1 -- 172.21.15.42:0/2333724426 shutdown_connections 2024-09-16T04:14:19.954 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.929+0000 7fb09b1b7640 1 --2- 172.21.15.42:0/2333724426 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb0940a1c50 0x7fb094135810 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:19.954 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.929+0000 7fb09b1b7640 1 --2- 172.21.15.42:0/2333724426 >> v2:172.21.15.42:3300/0 conn(0x7fb0940a1330 0x7fb0940a1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:19.954 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.929+0000 7fb09b1b7640 1 -- 172.21.15.42:0/2333724426 >> 172.21.15.42:0/2333724426 conn(0x7fb0940a2420 msgr2=0x7fb0940a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:14:19.955 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.929+0000 7fb09b1b7640 1 -- 172.21.15.42:0/2333724426 shutdown_connections 2024-09-16T04:14:19.955 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.929+0000 7fb09b1b7640 1 -- 172.21.15.42:0/2333724426 wait complete. 2024-09-16T04:14:19.955 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.929+0000 7fb09b1b7640 10 monclient: build_initial_monmap 2024-09-16T04:14:19.955 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.929+0000 7fb09b1b7640 10 monclient: monmap: 2024-09-16T04:14:19.955 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:14:19.955 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:14:19.955 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:14:19.955 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:14:19.955 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:14:19.955 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:19.955 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:14:19.955 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:14:19.955 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:14:19.956 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:19.956 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.929+0000 7fb09b1b7640 1 librados: starting msgr at 2024-09-16T04:14:19.956 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.929+0000 7fb09b1b7640 1 librados: starting objecter 2024-09-16T04:14:19.956 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.929+0000 7fb09b1b7640 1 Processor -- start 2024-09-16T04:14:19.956 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.929+0000 7fb09b1b7640 1 -- start start 2024-09-16T04:14:19.956 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.929+0000 7fb09b1b7640 1 librados: setting wanted keys 2024-09-16T04:14:19.956 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.929+0000 7fb09b1b7640 1 librados: calling monclient init 2024-09-16T04:14:19.956 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.929+0000 7fb09b1b7640 10 monclient: init 2024-09-16T04:14:19.956 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.933+0000 7fb09b1b7640 10 monclient: _reopen_session rank -1 2024-09-16T04:14:19.956 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.933+0000 7fb09b1b7640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:14:19.956 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.933+0000 7fb09b1b7640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb0940a1330 0x7fb094135320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:19.956 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.933+0000 7fb09b1b7640 10 monclient(hunting): picked mon.noname-c con 0x7fb0940a1330 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:14:19.956 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.933+0000 7fb09b1b7640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb0940a1c50 0x7fb0941cf130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:19.956 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.933+0000 7fb09b1b7640 10 monclient(hunting): picked mon.noname-a con 0x7fb0940a1c50 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:14:19.957 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.933+0000 7fb09b1b7640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb094135860 0x7fb0941cf670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:19.957 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.933+0000 7fb09b1b7640 10 monclient(hunting): picked mon.noname-b con 0x7fb094135860 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:14:19.957 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.933+0000 7fb09b1b7640 10 monclient(hunting): start opening mon connection 2024-09-16T04:14:19.957 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.933+0000 7fb09b1b7640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fb094135d50 con 0x7fb0940a1c50 2024-09-16T04:14:19.957 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.933+0000 7fb09b1b7640 10 monclient(hunting): start opening mon connection 2024-09-16T04:14:19.957 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.933+0000 7fb09b1b7640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fb09409ef80 con 0x7fb094135860 2024-09-16T04:14:19.957 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.933+0000 7fb09b1b7640 10 monclient(hunting): start opening mon connection 2024-09-16T04:14:19.957 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.933+0000 7fb09b1b7640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fb094135ed0 con 0x7fb0940a1330 2024-09-16T04:14:19.957 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.933+0000 7fb09b1b7640 10 monclient(hunting): _renew_subs 2024-09-16T04:14:19.957 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.933+0000 7fb09b1b7640 10 monclient(hunting): authenticate will time out at 2906.904416s 2024-09-16T04:14:19.957 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.933+0000 7fb098f53640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb0940a1330 0x7fb094135320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:19.957 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.933+0000 7fb093fff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb0940a1c50 0x7fb0941cf130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:19.957 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.933+0000 7fb098f53640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb0940a1330 0x7fb094135320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:51854/0 (socket says 172.21.15.42:51854) 2024-09-16T04:14:19.957 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.933+0000 7fb098f53640 1 -- 172.21.15.42:0/3195604170 learned_addr learned my addr 172.21.15.42:0/3195604170 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:14:19.958 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.933+0000 7fb098f53640 10 monclient(hunting): get_auth_request con 0x7fb0940a1330 auth_method 0 2024-09-16T04:14:19.958 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.933+0000 7fb098f53640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:14:19.958 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.933+0000 7fb098f53640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:19.958 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.933+0000 7fb098f53640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:19.958 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.933+0000 7fb093fff640 10 monclient(hunting): get_auth_request con 0x7fb0940a1c50 auth_method 0 2024-09-16T04:14:19.958 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.933+0000 7fb093fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:14:19.958 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.933+0000 7fb093fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:19.958 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.933+0000 7fb093fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:19.958 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.933+0000 7fb099754640 1 --2- 172.21.15.42:0/3195604170 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb094135860 0x7fb0941cf670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:19.958 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.933+0000 7fb098f53640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:14:19.958 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.933+0000 7fb098f53640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:14:19.958 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.933+0000 7fb098f53640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:14:19.958 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.933+0000 7fb099754640 10 monclient(hunting): get_auth_request con 0x7fb094135860 auth_method 0 2024-09-16T04:14:19.958 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.933+0000 7fb099754640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:14:19.959 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.933+0000 7fb099754640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:19.959 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.933+0000 7fb099754640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:19.959 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.933+0000 7fb093fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:14:19.959 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.933+0000 7fb093fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:14:19.959 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.933+0000 7fb093fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:14:19.959 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.933+0000 7fb098f53640 10 monclient(hunting): handle_auth_done global_id 1094228 payload 824 2024-09-16T04:14:19.959 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.933+0000 7fb098f53640 1 -- 172.21.15.42:0/3195604170 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb094135860 msgr2=0x7fb0941cf670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:19.959 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.933+0000 7fb098f53640 1 --2- 172.21.15.42:0/3195604170 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb094135860 0x7fb0941cf670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:19.959 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.933+0000 7fb098f53640 1 -- 172.21.15.42:0/3195604170 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb0940a1c50 msgr2=0x7fb0941cf130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:19.959 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.933+0000 7fb098f53640 1 --2- 172.21.15.42:0/3195604170 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb0940a1c50 0x7fb0941cf130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:19.959 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.933+0000 7fb098f53640 10 monclient: _finish_hunting 0 2024-09-16T04:14:19.959 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.933+0000 7fb098f53640 1 monclient: found mon.noname-c 2024-09-16T04:14:19.959 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.933+0000 7fb098f53640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:14:19.959 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.933+0000 7fb098f53640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T04:14:19.960 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.933+0000 7fb098f53640 1 -- 172.21.15.42:0/3195604170 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb094071390 con 0x7fb0940a1330 2024-09-16T04:14:19.960 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.933+0000 7fb099754640 1 --2- 172.21.15.42:0/3195604170 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb094135860 0x7fb0941cf670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T04:14:19.960 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.933+0000 7fb098f53640 1 --2- 172.21.15.42:0/3195604170 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb0940a1330 0x7fb094135320 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fb08c00efc0 tx=0x7fb08c00c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:14:19.960 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.933+0000 7fb0917fa640 10 client.?.objecter ms_handle_connect 0x7fb0940a1330 2024-09-16T04:14:19.960 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.933+0000 7fb0917fa640 10 client.?.objecter resend_mon_ops 2024-09-16T04:14:19.960 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.933+0000 7fb0917fa640 1 -- 172.21.15.42:0/3195604170 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb08c018020 con 0x7fb0940a1330 2024-09-16T04:14:19.960 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.933+0000 7fb0917fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:19.960 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.933+0000 7fb0917fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:14:19.960 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.933+0000 7fb0917fa640 10 monclient: dump: 2024-09-16T04:14:19.960 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:14:19.960 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:14:19.960 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:19.960 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:19.961 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:19.961 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:19.961 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:14:19.961 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:14:19.961 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:14:19.961 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:19.961 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.933+0000 7fb0917fa640 10 monclient: _finish_auth 0 2024-09-16T04:14:19.961 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.933+0000 7fb0917fa640 10 monclient: _check_auth_tickets 2024-09-16T04:14:19.961 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.933+0000 7fb0917fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:14:19.961 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.933+0000 7fb0917fa640 1 -- 172.21.15.42:0/3195604170 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb08c014680 con 0x7fb0940a1330 2024-09-16T04:14:19.961 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.933+0000 7fb09b1b7640 5 monclient: authenticate success, global_id 1094228 2024-09-16T04:14:19.961 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.933+0000 7fb09b1b7640 10 monclient: _renew_subs 2024-09-16T04:14:19.961 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.933+0000 7fb09b1b7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:14:19.961 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.933+0000 7fb09b1b7640 1 -- 172.21.15.42:0/3195604170 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb094073a30 con 0x7fb0940a1330 2024-09-16T04:14:19.962 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.933+0000 7fb0917fa640 10 monclient: handle_config config(0 keys) 2024-09-16T04:14:19.962 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.933+0000 7fb0917fa640 1 -- 172.21.15.42:0/3195604170 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb08c014a50 con 0x7fb0940a1330 2024-09-16T04:14:19.962 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.933+0000 7fb0917fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:19.962 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.933+0000 7fb0917fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:14:19.962 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.933+0000 7fb0917fa640 10 monclient: dump: 2024-09-16T04:14:19.962 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:14:19.962 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:14:19.962 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:19.962 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:19.962 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:19.962 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:19.962 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:14:19.962 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:14:19.962 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:14:19.963 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:19.963 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.933+0000 7fb092ffd640 4 set_mon_vals no callback set 2024-09-16T04:14:19.963 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.933+0000 7fb09b1b7640 10 client.1094228.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:14:19.963 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.933+0000 7fb09b1b7640 10 monclient: _renew_subs 2024-09-16T04:14:19.963 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.933+0000 7fb09b1b7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:14:19.963 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.933+0000 7fb09b1b7640 1 -- 172.21.15.42:0/3195604170 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0941cbc30 con 0x7fb0940a1330 2024-09-16T04:14:19.963 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.933+0000 7fb09b1b7640 1 librados: init done 2024-09-16T04:14:19.963 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.933+0000 7fb09b1b7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:14:19.963 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.933+0000 7fb09b1b7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:14:19.963 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.933+0000 7fb09b1b7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:14:19.963 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.933+0000 7fb09b1b7640 1 -- 172.21.15.42:0/3195604170 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb0940a1710 con 0x7fb0940a1330 2024-09-16T04:14:19.963 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.933+0000 7fb0917fa640 1 -- 172.21.15.42:0/3195604170 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fb08c01b020 con 0x7fb0940a1330 2024-09-16T04:14:19.963 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.933+0000 7fb0917fa640 1 --2- 172.21.15.42:0/3195604170 >> v2:172.21.15.154:6812/254845342 conn(0x7fb074037870 0x7fb074039d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:19.963 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.933+0000 7fb0917fa640 1 -- 172.21.15.42:0/3195604170 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(710..710 src has 681..710) ==== 4681+0+0 (secure 0 0 0) 0x7fb08c0499a0 con 0x7fb0940a1330 2024-09-16T04:14:19.964 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.933+0000 7fb0917fa640 10 client.1094228.objecter ms_dispatch 0x7fb094000ba0 osd_map(710..710 src has 681..710) 2024-09-16T04:14:19.964 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.933+0000 7fb0917fa640 3 client.1094228.objecter handle_osd_map got epochs [710,710] > 0 2024-09-16T04:14:19.964 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.933+0000 7fb0917fa640 3 client.1094228.objecter handle_osd_map decoding full epoch 710 2024-09-16T04:14:19.964 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.933+0000 7fb0917fa640 20 client.1094228.objecter dump_active .. 0 homeless 2024-09-16T04:14:19.964 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.933+0000 7fb093fff640 1 --2- 172.21.15.42:0/3195604170 >> v2:172.21.15.154:6812/254845342 conn(0x7fb074037870 0x7fb074039d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:19.964 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.933+0000 7fb093fff640 10 monclient: get_auth_request con 0x7fb074037870 auth_method 0 2024-09-16T04:14:19.964 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.933+0000 7fb093fff640 1 --2- 172.21.15.42:0/3195604170 >> v2:172.21.15.154:6812/254845342 conn(0x7fb074037870 0x7fb074039d30 secure :-1 s=READY pgs=4010 cs=0 l=1 rev1=1 crypto rx=0x7fb09409e220 tx=0x7fb07c007e40 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:14:19.964 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.933+0000 7fb0917fa640 10 client.1094228.objecter ms_handle_connect 0x7fb074037870 2024-09-16T04:14:19.964 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.937+0000 7fb0917fa640 1 -- 172.21.15.42:0/3195604170 <== mon.2 v2:172.21.15.42:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7fb08c022030 con 0x7fb0940a1330 2024-09-16T04:14:19.964 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.937+0000 7fb0917fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:14:19.964 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:19.937+0000 7fb0917fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:14:20.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.165+0000 7fb09b1b7640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-16T04:14:20.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.165+0000 7fb09b1b7640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-16T04:14:20.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.165+0000 7fb09b1b7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:14:20.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.165+0000 7fb09b1b7640 1 -- 172.21.15.42:0/3195604170 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fb09406f460 con 0x7fb0940a1330 2024-09-16T04:14:20.172 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.165+0000 7fb0917fa640 1 -- 172.21.15.42:0/3195604170 <== mon.2 v2:172.21.15.42:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7fb08c010040 con 0x7fb0940a1330 2024-09-16T04:14:20.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.165+0000 7fb0917fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-16T04:14:20.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.165+0000 7fb0917fa640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:14:20.173 INFO:teuthology.orchestra.run.smithi042.stdout:{"election_epoch":292,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","modified":"2024-09-16T03:41:35.824079Z","created":"2024-09-16T03:41:35.824079Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3300","nonce":0},{"type":"v1","addr":"172.21.15.42:6789","nonce":0}]},"addr":"172.21.15.42:6789/0","public_addr":"172.21.15.42:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:3301","nonce":0},{"type":"v1","addr":"172.21.15.42:6790","nonce":0}]},"addr":"172.21.15.42:6790/0","public_addr":"172.21.15.42:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-16T04:14:20.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.169+0000 7fb09b1b7640 10 librados: watch_flush enter 2024-09-16T04:14:20.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.169+0000 7fb09b1b7640 10 librados: watch_flush exit 2024-09-16T04:14:20.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.169+0000 7fb09b1b7640 20 client.1094228.objecter shutdown clearing up homeless session... 2024-09-16T04:14:20.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.169+0000 7fb09b1b7640 10 client.1094228.objecter successfully canceled tick 2024-09-16T04:14:20.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.169+0000 7fb09b1b7640 1 -- 172.21.15.42:0/3195604170 >> v2:172.21.15.154:6812/254845342 conn(0x7fb074037870 msgr2=0x7fb074039d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:20.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.169+0000 7fb09b1b7640 1 --2- 172.21.15.42:0/3195604170 >> v2:172.21.15.154:6812/254845342 conn(0x7fb074037870 0x7fb074039d30 secure :-1 s=READY pgs=4010 cs=0 l=1 rev1=1 crypto rx=0x7fb09409e220 tx=0x7fb07c007e40 comp rx=0 tx=0).stop 2024-09-16T04:14:20.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.169+0000 7fb09b1b7640 10 monclient: shutdown 2024-09-16T04:14:20.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.169+0000 7fb09b1b7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:14:20.175 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.169+0000 7fb09b1b7640 1 -- 172.21.15.42:0/3195604170 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb0940a1330 msgr2=0x7fb094135320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:20.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.169+0000 7fb09b1b7640 1 --2- 172.21.15.42:0/3195604170 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb0940a1330 0x7fb094135320 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fb08c00efc0 tx=0x7fb08c00c490 comp rx=0 tx=0).stop 2024-09-16T04:14:20.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.169+0000 7fb09b1b7640 1 -- 172.21.15.42:0/3195604170 shutdown_connections 2024-09-16T04:14:20.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.169+0000 7fb09b1b7640 1 --2- 172.21.15.42:0/3195604170 >> v2:172.21.15.154:6812/254845342 conn(0x7fb074037870 0x7fb074039d30 unknown :-1 s=CLOSED pgs=4010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:20.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.169+0000 7fb09b1b7640 1 --2- 172.21.15.42:0/3195604170 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fb094135860 0x7fb0941cf670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:20.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.169+0000 7fb09b1b7640 1 --2- 172.21.15.42:0/3195604170 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fb0940a1c50 0x7fb0941cf130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:20.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.169+0000 7fb09b1b7640 1 --2- 172.21.15.42:0/3195604170 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fb0940a1330 0x7fb094135320 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:20.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.169+0000 7fb09b1b7640 1 -- 172.21.15.42:0/3195604170 >> 172.21.15.42:0/3195604170 conn(0x7fb0940a2420 msgr2=0x7fb09406f980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:14:20.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.169+0000 7fb09b1b7640 1 -- 172.21.15.42:0/3195604170 shutdown_connections 2024-09-16T04:14:20.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.169+0000 7fb09b1b7640 1 -- 172.21.15.42:0/3195604170 wait complete. 2024-09-16T04:14:20.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.169+0000 7fb09b1b7640 1 librados: shutdown 2024-09-16T04:14:20.191 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-16T04:14:20.191 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-09-16T04:14:20.200 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-16T04:14:20.200 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:14:20.326 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.321+0000 7f8b65b3f640 10 monclient: get_monmap_and_config 2024-09-16T04:14:20.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.321+0000 7f8b65b3f640 10 monclient: build_initial_monmap 2024-09-16T04:14:20.327 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.321+0000 7f8b65b3f640 10 monclient: monmap: 2024-09-16T04:14:20.327 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:14:20.327 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:14:20.327 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:14:20.325561+0000 2024-09-16T04:14:20.327 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:14:20.325561+0000 2024-09-16T04:14:20.327 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:14:20.327 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:20.327 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:14:20.327 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:14:20.327 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:14:20.327 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:14:20.327 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:14:20.328 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:20.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.321+0000 7f8b65b3f640 1 Processor -- start 2024-09-16T04:14:20.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.321+0000 7f8b65b3f640 1 -- start start 2024-09-16T04:14:20.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.321+0000 7f8b65b3f640 10 monclient: init 2024-09-16T04:14:20.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.321+0000 7f8b65b3f640 10 monclient: _reopen_session rank -1 2024-09-16T04:14:20.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.321+0000 7f8b65b3f640 10 monclient: _add_conns ranks=[1,4,0,3,2] 2024-09-16T04:14:20.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.321+0000 7f8b65b3f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8b6009ca60 addr v1:172.21.15.154:6789/0 2024-09-16T04:14:20.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.321+0000 7f8b65b3f640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f8b6009f240 0x7f8b6009db70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:20.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.321+0000 7f8b65b3f640 10 monclient(hunting): picked mon.noname-c con 0x7f8b6009f240 addr v2:172.21.15.154:3300/0 2024-09-16T04:14:20.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.321+0000 7f8b65b3f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8b60139760 addr v1:172.21.15.42:6789/0 2024-09-16T04:14:20.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.321+0000 7f8b65b3f640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8b6009f650 con 0x7f8b60139760 2024-09-16T04:14:20.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.321+0000 7f8b65b3f640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8b6009e580 con 0x7f8b6009ca60 2024-09-16T04:14:20.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.321+0000 7f8b65b3f640 10 monclient(hunting): start opening mon connection 2024-09-16T04:14:20.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.321+0000 7f8b65b3f640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f8b6009ce40 con 0x7f8b6009f240 2024-09-16T04:14:20.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.321+0000 7f8b65b3f640 10 monclient(hunting): _renew_subs 2024-09-16T04:14:20.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.321+0000 7f8b65b3f640 10 monclient(hunting): authenticate will time out at 2907.295670s 2024-09-16T04:14:20.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.321+0000 7f8b5ffff640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f8b60139760 0x7f8b6013bb50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:52956/0 (socket says 172.21.15.42:52956) 2024-09-16T04:14:20.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.321+0000 7f8b5ffff640 1 -- 172.21.15.42:0/1772972928 learned_addr learned my addr 172.21.15.42:0/1772972928 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T04:14:20.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.325+0000 7f8b5effd640 1 --2- 172.21.15.42:0/1772972928 >> v2:172.21.15.154:3300/0 conn(0x7f8b6009f240 0x7f8b6009db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:20.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.325+0000 7f8b5effd640 10 monclient(hunting): get_auth_request con 0x7f8b6009f240 auth_method 0 2024-09-16T04:14:20.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.325+0000 7f8b5e7fc640 1 -- 172.21.15.42:0/1772972928 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 857518232 0 0) 0x7f8b6009f650 con 0x7f8b60139760 2024-09-16T04:14:20.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.325+0000 7f8b5effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:14:20.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.325+0000 7f8b5effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:20.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.325+0000 7f8b5effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:20.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.325+0000 7f8b5e7fc640 10 monclient(hunting): _negotiate 2024-09-16T04:14:20.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.325+0000 7f8b5e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:20.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.325+0000 7f8b5e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:20.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.325+0000 7f8b5e7fc640 10 monclient(hunting): my global_id is 1104147 2024-09-16T04:14:20.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.325+0000 7f8b5e7fc640 1 -- 172.21.15.42:0/1772972928 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8b48003530 con 0x7f8b60139760 2024-09-16T04:14:20.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.325+0000 7f8b5effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:14:20.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.325+0000 7f8b5effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:14:20.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.325+0000 7f8b5effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:14:20.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.325+0000 7f8b5e7fc640 1 -- 172.21.15.42:0/1772972928 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3792023897 0 0) 0x7f8b48003530 con 0x7f8b60139760 2024-09-16T04:14:20.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.325+0000 7f8b5e7fc640 1 -- 172.21.15.42:0/1772972928 >> v2:172.21.15.154:3300/0 conn(0x7f8b6009f240 msgr2=0x7f8b6009db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:14:20.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.325+0000 7f8b5e7fc640 1 --2- 172.21.15.42:0/1772972928 >> v2:172.21.15.154:3300/0 conn(0x7f8b6009f240 0x7f8b6009db70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:20.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.325+0000 7f8b5e7fc640 1 -- 172.21.15.42:0/1772972928 >> v1:172.21.15.154:6789/0 conn(0x7f8b6009ca60 legacy=0x7f8b6009d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:14:20.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.325+0000 7f8b5e7fc640 10 monclient: _finish_hunting 0 2024-09-16T04:14:20.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.325+0000 7f8b5e7fc640 1 monclient: found mon.noname-a-legacy 2024-09-16T04:14:20.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.325+0000 7f8b5e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:14:20.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.325+0000 7f8b5e7fc640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.42:6789/0 2024-09-16T04:14:20.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.325+0000 7f8b5e7fc640 1 -- 172.21.15.42:0/1772972928 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b6009eb40 con 0x7f8b60139760 2024-09-16T04:14:20.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.325+0000 7f8b5e7fc640 10 monclient: _finish_auth 0 2024-09-16T04:14:20.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.325+0000 7f8b5e7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:14:20.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.325+0000 7f8b5e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:14:20.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.325+0000 7f8b65b3f640 5 monclient: authenticate success, global_id 1104147 2024-09-16T04:14:20.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.325+0000 7f8b5e7fc640 1 -- 172.21.15.42:0/1772972928 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2177413734 0 0) 0x7f8b58001a00 con 0x7f8b60139760 2024-09-16T04:14:20.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.325+0000 7f8b65b3f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:14:20.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.325+0000 7f8b5e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:20.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.325+0000 7f8b5e7fc640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T04:14:20.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.325+0000 7f8b5e7fc640 10 monclient: dump: 2024-09-16T04:14:20.337 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:14:20.337 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:14:20.337 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:20.337 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:20.337 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:20.337 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:20.338 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:14:20.338 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:14:20.338 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:14:20.338 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:20.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.325+0000 7f8b5e7fc640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] but i'm connected to v1:172.21.15.42:6789/0, reconnecting 2024-09-16T04:14:20.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.325+0000 7f8b5e7fc640 10 monclient: _reopen_session rank -1 2024-09-16T04:14:20.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.325+0000 7f8b5e7fc640 1 -- 172.21.15.42:0/1772972928 >> v1:172.21.15.42:6789/0 conn(0x7f8b60139760 legacy=0x7f8b6013bb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:20.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.325+0000 7f8b5e7fc640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:14:20.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.325+0000 7f8b5e7fc640 1 --2- 172.21.15.42:0/1772972928 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8b480039c0 0x7f8b48003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:20.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.325+0000 7f8b5e7fc640 10 monclient(hunting): picked mon.a con 0x7f8b480039c0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:14:20.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.325+0000 7f8b5e7fc640 1 --2- 172.21.15.42:0/1772972928 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f8b48004810 0x7f8b48004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:20.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.325+0000 7f8b5e7fc640 10 monclient(hunting): picked mon.c con 0x7f8b48004810 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:14:20.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.325+0000 7f8b5e7fc640 1 --2- 172.21.15.42:0/1772972928 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8b480051d0 0x7f8b4800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:20.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.325+0000 7f8b5e7fc640 10 monclient(hunting): picked mon.b con 0x7f8b480051d0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:14:20.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.325+0000 7f8b5e7fc640 10 monclient(hunting): start opening mon connection 2024-09-16T04:14:20.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.325+0000 7f8b5e7fc640 1 -- 172.21.15.42:0/1772972928 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f8b6009ce40 con 0x7f8b480039c0 2024-09-16T04:14:20.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.325+0000 7f8b5e7fc640 10 monclient(hunting): start opening mon connection 2024-09-16T04:14:20.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.325+0000 7f8b5e7fc640 1 -- 172.21.15.42:0/1772972928 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f8b4800c170 con 0x7f8b480051d0 2024-09-16T04:14:20.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.325+0000 7f8b5e7fc640 10 monclient(hunting): start opening mon connection 2024-09-16T04:14:20.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.325+0000 7f8b5e7fc640 1 -- 172.21.15.42:0/1772972928 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f8b4800c370 con 0x7f8b48004810 2024-09-16T04:14:20.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.325+0000 7f8b5e7fc640 10 monclient(hunting): _renew_subs 2024-09-16T04:14:20.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.325+0000 7f8b5e7fc640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:14:20.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.325+0000 7f8b65b3f640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-16T04:14:20.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.325+0000 7f8b5f7fe640 1 --2- 172.21.15.42:0/1772972928 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8b480039c0 0x7f8b48003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:20.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.325+0000 7f8b5effd640 1 --2- 172.21.15.42:0/1772972928 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f8b48004810 0x7f8b48004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:20.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.325+0000 7f8b5f7fe640 10 monclient(hunting): get_auth_request con 0x7f8b480039c0 auth_method 0 2024-09-16T04:14:20.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.325+0000 7f8b5f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:14:20.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.325+0000 7f8b5f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:20.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.325+0000 7f8b5f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:14:20.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.325+0000 7f8b5effd640 10 monclient(hunting): get_auth_request con 0x7f8b48004810 auth_method 0 2024-09-16T04:14:20.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.325+0000 7f8b5effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:14:20.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.325+0000 7f8b5effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:20.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.325+0000 7f8b5effd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:14:20.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.325+0000 7f8b5f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:14:20.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.325+0000 7f8b5f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:14:20.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.325+0000 7f8b5f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:14:20.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.325+0000 7f8b5effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:14:20.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.325+0000 7f8b5effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:14:20.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.325+0000 7f8b5effd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:14:20.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.325+0000 7f8b5ffff640 1 --2- 172.21.15.42:0/1772972928 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8b480051d0 0x7f8b4800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:20.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.325+0000 7f8b5effd640 10 monclient(hunting): handle_auth_done global_id 1104147 payload 293 2024-09-16T04:14:20.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.325+0000 7f8b5effd640 1 -- 172.21.15.42:0/1772972928 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8b480051d0 msgr2=0x7f8b4800b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:14:20.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.325+0000 7f8b5effd640 1 --2- 172.21.15.42:0/1772972928 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8b480051d0 0x7f8b4800b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:20.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.325+0000 7f8b5effd640 1 -- 172.21.15.42:0/1772972928 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8b480039c0 msgr2=0x7f8b48003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:14:20.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.325+0000 7f8b5effd640 1 --2- 172.21.15.42:0/1772972928 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8b480039c0 0x7f8b48003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:20.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.325+0000 7f8b5effd640 10 monclient: _finish_hunting 0 2024-09-16T04:14:20.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.325+0000 7f8b5effd640 1 monclient: found mon.c 2024-09-16T04:14:20.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.325+0000 7f8b5effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:14:20.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.325+0000 7f8b5effd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:14:20.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.325+0000 7f8b5effd640 1 -- 172.21.15.42:0/1772972928 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b48003530 con 0x7f8b48004810 2024-09-16T04:14:20.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.325+0000 7f8b5effd640 10 monclient: _finish_auth 0 2024-09-16T04:14:20.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.325+0000 7f8b5effd640 10 monclient: _check_auth_tickets 2024-09-16T04:14:20.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.325+0000 7f8b5effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:14:20.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.325+0000 7f8b5ffff640 10 monclient: get_auth_request con 0x7f8b480051d0 auth_method 0 2024-09-16T04:14:20.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.325+0000 7f8b5f7fe640 1 --2- 172.21.15.42:0/1772972928 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8b480039c0 0x7f8b48003de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:14:20.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.325+0000 7f8b5ffff640 1 --2- 172.21.15.42:0/1772972928 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8b480051d0 0x7f8b4800b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T04:14:20.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.325+0000 7f8b5ffff640 1 -- 172.21.15.42:0/1772972928 reap_dead start 2024-09-16T04:14:20.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.325+0000 7f8b5effd640 1 --2- 172.21.15.42:0/1772972928 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f8b48004810 0x7f8b48004c90 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f8b54009ae0 tx=0x7f8b5401ca50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8a4506ef4a82a775 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:14:20.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.325+0000 7f8b5e7fc640 1 -- 172.21.15.42:0/1772972928 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b540096b0 con 0x7f8b48004810 2024-09-16T04:14:20.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.325+0000 7f8b5e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:20.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.325+0000 7f8b5e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:14:20.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.325+0000 7f8b5e7fc640 10 monclient: dump: 2024-09-16T04:14:20.342 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:14:20.342 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:14:20.342 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:20.342 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:20.342 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:20.342 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:20.343 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:14:20.343 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:14:20.343 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:14:20.343 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:20.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.325+0000 7f8b5e7fc640 1 -- 172.21.15.42:0/1772972928 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8b540a6430 con 0x7f8b48004810 2024-09-16T04:14:20.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.325+0000 7f8b65b3f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:14:20.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.325+0000 7f8b5e7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:14:20.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.325+0000 7f8b5e7fc640 1 -- 172.21.15.42:0/1772972928 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b540a6820 con 0x7f8b48004810 2024-09-16T04:14:20.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.325+0000 7f8b65b3f640 10 monclient: get_monmap_and_config success 2024-09-16T04:14:20.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.325+0000 7f8b65b3f640 4 set_mon_vals no callback set 2024-09-16T04:14:20.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.325+0000 7f8b5e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:20.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.325+0000 7f8b5e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:14:20.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.325+0000 7f8b5e7fc640 10 monclient: dump: 2024-09-16T04:14:20.343 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:14:20.343 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:14:20.344 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:20.344 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:20.344 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:20.344 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:20.344 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:14:20.344 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:14:20.344 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:14:20.344 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:20.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.329+0000 7f8b65b3f640 10 monclient: shutdown 2024-09-16T04:14:20.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.329+0000 7f8b65b3f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:14:20.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.329+0000 7f8b65b3f640 1 -- 172.21.15.42:0/1772972928 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f8b48004810 msgr2=0x7f8b48004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:20.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.329+0000 7f8b65b3f640 1 --2- 172.21.15.42:0/1772972928 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f8b48004810 0x7f8b48004c90 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f8b54009ae0 tx=0x7f8b5401ca50 comp rx=0 tx=0).stop 2024-09-16T04:14:20.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.329+0000 7f8b65b3f640 1 -- 172.21.15.42:0/1772972928 shutdown_connections 2024-09-16T04:14:20.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.329+0000 7f8b65b3f640 1 --2- 172.21.15.42:0/1772972928 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f8b48004810 0x7f8b48004c90 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:20.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.329+0000 7f8b65b3f640 1 -- 172.21.15.42:0/1772972928 >> 172.21.15.42:0/1772972928 conn(0x7f8b6006bc40 msgr2=0x7f8b6006c050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:14:20.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.329+0000 7f8b65b3f640 1 -- 172.21.15.42:0/1772972928 shutdown_connections 2024-09-16T04:14:20.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.329+0000 7f8b65b3f640 1 -- 172.21.15.42:0/1772972928 wait complete. 2024-09-16T04:14:20.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.329+0000 7f8b65b3f640 10 monclient: build_initial_monmap 2024-09-16T04:14:20.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.329+0000 7f8b65b3f640 10 monclient: monmap: 2024-09-16T04:14:20.345 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:14:20.345 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:14:20.345 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:14:20.345 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:14:20.345 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:14:20.345 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:20.345 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:14:20.345 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:14:20.346 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:14:20.346 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:20.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.329+0000 7f8b65b3f640 1 librados: starting msgr at 2024-09-16T04:14:20.346 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.329+0000 7f8b65b3f640 1 librados: starting objecter 2024-09-16T04:14:20.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.329+0000 7f8b65b3f640 1 Processor -- start 2024-09-16T04:14:20.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.329+0000 7f8b65b3f640 1 -- start start 2024-09-16T04:14:20.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.329+0000 7f8b65b3f640 1 librados: setting wanted keys 2024-09-16T04:14:20.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.329+0000 7f8b65b3f640 1 librados: calling monclient init 2024-09-16T04:14:20.350 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.329+0000 7f8b65b3f640 10 monclient: init 2024-09-16T04:14:20.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.329+0000 7f8b65b3f640 10 monclient: _reopen_session rank -1 2024-09-16T04:14:20.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.329+0000 7f8b65b3f640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:14:20.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.329+0000 7f8b65b3f640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f8b48004810 0x7f8b601cef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:20.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.329+0000 7f8b65b3f640 10 monclient(hunting): picked mon.noname-c con 0x7f8b48004810 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:14:20.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.329+0000 7f8b65b3f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8b60133e00 0x7f8b601cba90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:20.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.329+0000 7f8b65b3f640 10 monclient(hunting): picked mon.noname-b con 0x7f8b60133e00 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:14:20.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.329+0000 7f8b65b3f640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8b60134280 0x7f8b601cbfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:20.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.329+0000 7f8b65b3f640 10 monclient(hunting): picked mon.noname-a con 0x7f8b60134280 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:14:20.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.329+0000 7f8b65b3f640 10 monclient(hunting): start opening mon connection 2024-09-16T04:14:20.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.329+0000 7f8b65b3f640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f8b6009e000 con 0x7f8b60134280 2024-09-16T04:14:20.351 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.329+0000 7f8b65b3f640 10 monclient(hunting): start opening mon connection 2024-09-16T04:14:20.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.329+0000 7f8b65b3f640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f8b6009de80 con 0x7f8b60133e00 2024-09-16T04:14:20.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.329+0000 7f8b65b3f640 10 monclient(hunting): start opening mon connection 2024-09-16T04:14:20.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.329+0000 7f8b65b3f640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f8b6009e180 con 0x7f8b48004810 2024-09-16T04:14:20.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.329+0000 7f8b65b3f640 10 monclient(hunting): _renew_subs 2024-09-16T04:14:20.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.329+0000 7f8b65b3f640 10 monclient(hunting): authenticate will time out at 2907.301885s 2024-09-16T04:14:20.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.329+0000 7f8b5f7fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f8b48004810 0x7f8b601cef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:20.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.329+0000 7f8b5f7fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f8b48004810 0x7f8b601cef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:51872/0 (socket says 172.21.15.42:51872) 2024-09-16T04:14:20.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.329+0000 7f8b5f7fe640 1 -- 172.21.15.42:0/3847410946 learned_addr learned my addr 172.21.15.42:0/3847410946 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:14:20.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.329+0000 7f8b5f7fe640 10 monclient(hunting): get_auth_request con 0x7f8b48004810 auth_method 0 2024-09-16T04:14:20.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.329+0000 7f8b5f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:14:20.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.329+0000 7f8b5f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:20.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.329+0000 7f8b5f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:20.352 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.329+0000 7f8b5ffff640 1 --2- 172.21.15.42:0/3847410946 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8b60134280 0x7f8b601cbfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:20.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.329+0000 7f8b5ffff640 10 monclient(hunting): get_auth_request con 0x7f8b60134280 auth_method 0 2024-09-16T04:14:20.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.329+0000 7f8b5ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:14:20.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.329+0000 7f8b5ffff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:20.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.329+0000 7f8b5ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:20.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.329+0000 7f8b5f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:14:20.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.329+0000 7f8b5f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:14:20.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.329+0000 7f8b5f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:14:20.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.329+0000 7f8b5ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:14:20.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.329+0000 7f8b5ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:14:20.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.329+0000 7f8b5ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:14:20.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.329+0000 7f8b5effd640 1 --2- 172.21.15.42:0/3847410946 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8b60133e00 0x7f8b601cba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:20.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.329+0000 7f8b5effd640 10 monclient(hunting): get_auth_request con 0x7f8b60133e00 auth_method 0 2024-09-16T04:14:20.353 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.329+0000 7f8b5effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:14:20.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.329+0000 7f8b5effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:20.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.329+0000 7f8b5effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:20.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.329+0000 7f8b5f7fe640 10 monclient(hunting): handle_auth_done global_id 1094231 payload 824 2024-09-16T04:14:20.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.329+0000 7f8b5f7fe640 1 -- 172.21.15.42:0/3847410946 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8b60133e00 msgr2=0x7f8b601cba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:20.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.329+0000 7f8b5f7fe640 1 --2- 172.21.15.42:0/3847410946 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8b60133e00 0x7f8b601cba90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:20.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.329+0000 7f8b5f7fe640 1 -- 172.21.15.42:0/3847410946 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8b60134280 msgr2=0x7f8b601cbfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:20.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.329+0000 7f8b5f7fe640 1 --2- 172.21.15.42:0/3847410946 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8b60134280 0x7f8b601cbfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:20.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.329+0000 7f8b5f7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:14:20.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.329+0000 7f8b5f7fe640 1 monclient: found mon.noname-c 2024-09-16T04:14:20.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.329+0000 7f8b5f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:14:20.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.329+0000 7f8b5f7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T04:14:20.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.329+0000 7f8b5f7fe640 1 -- 172.21.15.42:0/3847410946 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b60126550 con 0x7f8b48004810 2024-09-16T04:14:20.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.329+0000 7f8b5f7fe640 1 --2- 172.21.15.42:0/3847410946 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f8b48004810 0x7f8b601cef90 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f8b5000f7a0 tx=0x7f8b5000fc60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:14:20.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.329+0000 7f8b47fff640 10 client.?.objecter ms_handle_connect 0x7f8b48004810 2024-09-16T04:14:20.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.329+0000 7f8b47fff640 10 client.?.objecter resend_mon_ops 2024-09-16T04:14:20.354 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.329+0000 7f8b47fff640 1 -- 172.21.15.42:0/3847410946 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b500013a0 con 0x7f8b48004810 2024-09-16T04:14:20.355 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.329+0000 7f8b47fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:20.355 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.329+0000 7f8b47fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:14:20.355 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.329+0000 7f8b47fff640 10 monclient: dump: 2024-09-16T04:14:20.355 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:14:20.355 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:14:20.355 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:20.355 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:20.355 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:20.355 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:20.355 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:14:20.355 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:14:20.355 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:14:20.355 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:20.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.329+0000 7f8b47fff640 10 monclient: _finish_auth 0 2024-09-16T04:14:20.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.329+0000 7f8b47fff640 10 monclient: _check_auth_tickets 2024-09-16T04:14:20.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.329+0000 7f8b47fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:14:20.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.329+0000 7f8b47fff640 1 -- 172.21.15.42:0/3847410946 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8b50018070 con 0x7f8b48004810 2024-09-16T04:14:20.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.329+0000 7f8b47fff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:14:20.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.329+0000 7f8b47fff640 1 -- 172.21.15.42:0/3847410946 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b5000fe50 con 0x7f8b48004810 2024-09-16T04:14:20.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.329+0000 7f8b47fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:20.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.329+0000 7f8b47fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:14:20.356 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.329+0000 7f8b47fff640 10 monclient: dump: 2024-09-16T04:14:20.356 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:14:20.356 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:14:20.356 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:20.356 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:20.356 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:20.357 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:20.357 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:14:20.357 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:14:20.357 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:14:20.357 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:20.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.329+0000 7f8b65b3f640 5 monclient: authenticate success, global_id 1094231 2024-09-16T04:14:20.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.329+0000 7f8b65b3f640 10 monclient: _renew_subs 2024-09-16T04:14:20.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.329+0000 7f8b5dffb640 4 set_mon_vals no callback set 2024-09-16T04:14:20.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.329+0000 7f8b65b3f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:14:20.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.329+0000 7f8b65b3f640 1 -- 172.21.15.42:0/3847410946 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b601250a0 con 0x7f8b48004810 2024-09-16T04:14:20.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.329+0000 7f8b65b3f640 10 client.1094231.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:14:20.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.329+0000 7f8b65b3f640 10 monclient: _renew_subs 2024-09-16T04:14:20.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.329+0000 7f8b65b3f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:14:20.357 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.329+0000 7f8b65b3f640 1 -- 172.21.15.42:0/3847410946 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b60073080 con 0x7f8b48004810 2024-09-16T04:14:20.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.329+0000 7f8b65b3f640 1 librados: init done 2024-09-16T04:14:20.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.329+0000 7f8b65b3f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:14:20.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.329+0000 7f8b65b3f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:14:20.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.329+0000 7f8b65b3f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:14:20.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.329+0000 7f8b65b3f640 1 -- 172.21.15.42:0/3847410946 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b6009d150 con 0x7f8b48004810 2024-09-16T04:14:20.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.329+0000 7f8b47fff640 1 -- 172.21.15.42:0/3847410946 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f8b5000d6d0 con 0x7f8b48004810 2024-09-16T04:14:20.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.329+0000 7f8b47fff640 1 --2- 172.21.15.42:0/3847410946 >> v2:172.21.15.154:6812/254845342 conn(0x7f8b4c037840 0x7f8b4c039d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:20.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.329+0000 7f8b47fff640 1 -- 172.21.15.42:0/3847410946 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(710..710 src has 681..710) ==== 4681+0+0 (secure 0 0 0) 0x7f8b500487b0 con 0x7f8b48004810 2024-09-16T04:14:20.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.329+0000 7f8b47fff640 10 client.1094231.objecter ms_dispatch 0x7f8b60000ba0 osd_map(710..710 src has 681..710) 2024-09-16T04:14:20.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.329+0000 7f8b47fff640 3 client.1094231.objecter handle_osd_map got epochs [710,710] > 0 2024-09-16T04:14:20.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.329+0000 7f8b47fff640 3 client.1094231.objecter handle_osd_map decoding full epoch 710 2024-09-16T04:14:20.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.333+0000 7f8b47fff640 20 client.1094231.objecter dump_active .. 0 homeless 2024-09-16T04:14:20.358 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.333+0000 7f8b5effd640 1 --2- 172.21.15.42:0/3847410946 >> v2:172.21.15.154:6812/254845342 conn(0x7f8b4c037840 0x7f8b4c039d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:20.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.333+0000 7f8b5effd640 10 monclient: get_auth_request con 0x7f8b4c037840 auth_method 0 2024-09-16T04:14:20.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.333+0000 7f8b5effd640 1 --2- 172.21.15.42:0/3847410946 >> v2:172.21.15.154:6812/254845342 conn(0x7f8b4c037840 0x7f8b4c039d00 secure :-1 s=READY pgs=4011 cs=0 l=1 rev1=1 crypto rx=0x7f8b540ad000 tx=0x7f8b540b2040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:14:20.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.333+0000 7f8b47fff640 10 client.1094231.objecter ms_handle_connect 0x7f8b4c037840 2024-09-16T04:14:20.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.337+0000 7f8b47fff640 1 -- 172.21.15.42:0/3847410946 <== mon.2 v2:172.21.15.42:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f8b5000de30 con 0x7f8b48004810 2024-09-16T04:14:20.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.337+0000 7f8b47fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:14:20.359 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.337+0000 7f8b47fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:14:20.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.497+0000 7f8b65b3f640 1 -- 172.21.15.42:0/3847410946 --> v2:172.21.15.154:6812/254845342 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f8b60126950 con 0x7f8b4c037840 2024-09-16T04:14:20.512 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.505+0000 7f8b47fff640 1 -- 172.21.15.42:0/3847410946 <== mgr.4102 v2:172.21.15.154:6812/254845342 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+114174 (secure 0 0 0) 0x7f8b540a7410 con 0x7f8b4c037840 2024-09-16T04:14:20.514 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-09-16T04:14:20.514 INFO:teuthology.orchestra.run.smithi042.stderr:dumped all 2024-09-16T04:14:20.516 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.509+0000 7f8b65b3f640 10 librados: watch_flush enter 2024-09-16T04:14:20.516 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.509+0000 7f8b65b3f640 10 librados: watch_flush exit 2024-09-16T04:14:20.516 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.509+0000 7f8b65b3f640 20 client.1094231.objecter shutdown clearing up homeless session... 2024-09-16T04:14:20.516 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.509+0000 7f8b65b3f640 10 client.1094231.objecter successfully canceled tick 2024-09-16T04:14:20.516 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.509+0000 7f8b65b3f640 1 -- 172.21.15.42:0/3847410946 >> v2:172.21.15.154:6812/254845342 conn(0x7f8b4c037840 msgr2=0x7f8b4c039d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:20.516 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.509+0000 7f8b65b3f640 1 --2- 172.21.15.42:0/3847410946 >> v2:172.21.15.154:6812/254845342 conn(0x7f8b4c037840 0x7f8b4c039d00 secure :-1 s=READY pgs=4011 cs=0 l=1 rev1=1 crypto rx=0x7f8b540ad000 tx=0x7f8b540b2040 comp rx=0 tx=0).stop 2024-09-16T04:14:20.516 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.509+0000 7f8b65b3f640 10 monclient: shutdown 2024-09-16T04:14:20.516 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.509+0000 7f8b65b3f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:14:20.516 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.509+0000 7f8b65b3f640 1 -- 172.21.15.42:0/3847410946 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f8b48004810 msgr2=0x7f8b601cef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:20.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.509+0000 7f8b65b3f640 1 --2- 172.21.15.42:0/3847410946 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f8b48004810 0x7f8b601cef90 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f8b5000f7a0 tx=0x7f8b5000fc60 comp rx=0 tx=0).stop 2024-09-16T04:14:20.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.509+0000 7f8b65b3f640 1 -- 172.21.15.42:0/3847410946 shutdown_connections 2024-09-16T04:14:20.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.509+0000 7f8b65b3f640 1 --2- 172.21.15.42:0/3847410946 >> v2:172.21.15.154:6812/254845342 conn(0x7f8b4c037840 0x7f8b4c039d00 unknown :-1 s=CLOSED pgs=4011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:20.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.509+0000 7f8b65b3f640 1 --2- 172.21.15.42:0/3847410946 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8b60134280 0x7f8b601cbfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:20.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.509+0000 7f8b65b3f640 1 --2- 172.21.15.42:0/3847410946 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f8b60133e00 0x7f8b601cba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:20.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.509+0000 7f8b65b3f640 1 --2- 172.21.15.42:0/3847410946 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f8b48004810 0x7f8b601cef90 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:20.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.509+0000 7f8b65b3f640 1 -- 172.21.15.42:0/3847410946 >> 172.21.15.42:0/3847410946 conn(0x7f8b6006bc40 msgr2=0x7f8b600a47f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:14:20.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.509+0000 7f8b65b3f640 1 -- 172.21.15.42:0/3847410946 shutdown_connections 2024-09-16T04:14:20.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.509+0000 7f8b65b3f640 1 -- 172.21.15.42:0/3847410946 wait complete. 2024-09-16T04:14:20.517 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.509+0000 7f8b65b3f640 1 librados: shutdown 2024-09-16T04:14:20.533 INFO:teuthology.orchestra.run.smithi042.stdout:{"pg_ready":true,"pg_map":{"version":1653,"stamp":"2024-09-16T04:14:20.231345+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590392,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":122,"num_read_kb":106,"num_write":168,"num_write_kb":2425,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":127,"ondisk_log_size":127,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":431992,"kb_used_data":22680,"kb_used_omap":9,"kb_used_meta":407734,"kb_avail":565799048,"statfs":{"total":579820584960,"available":579378225152,"internally_reserved":0,"allocated":23224320,"data_stored":10083938,"data_compressed":18838,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":9567,"internal_metadata":417520289},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003822"},"pg_stats":[{"pgid":"31.f","version":"0'0","reported_seq":1350,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:53.282605+0000","last_change":"2024-09-16T03:43:34.400720+0000","last_active":"2024-09-16T04:13:53.282605+0000","last_peered":"2024-09-16T04:13:53.282605+0000","last_clean":"2024-09-16T04:13:53.282605+0000","last_became_active":"2024-09-16T03:43:34.399531+0000","last_became_peered":"2024-09-16T03:43:34.399531+0000","last_unstale":"2024-09-16T04:13:53.282605+0000","last_undegraded":"2024-09-16T04:13:53.282605+0000","last_fullsized":"2024-09-16T04:13:53.282605+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_clean_scrub_stamp":"2024-09-16T03:43:33.221564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:29:59.415591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.e","version":"0'0","reported_seq":1351,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:53.356452+0000","last_change":"2024-09-16T03:43:34.492536+0000","last_active":"2024-09-16T04:13:53.356452+0000","last_peered":"2024-09-16T04:13:53.356452+0000","last_clean":"2024-09-16T04:13:53.356452+0000","last_became_active":"2024-09-16T03:43:34.492249+0000","last_became_peered":"2024-09-16T03:43:34.492249+0000","last_unstale":"2024-09-16T04:13:53.356452+0000","last_undegraded":"2024-09-16T04:13:53.356452+0000","last_fullsized":"2024-09-16T04:13:53.356452+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_clean_scrub_stamp":"2024-09-16T03:43:33.221564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:15:41.168066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.d","version":"0'0","reported_seq":1373,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:51.310003+0000","last_change":"2024-09-16T03:43:34.414280+0000","last_active":"2024-09-16T04:13:51.310003+0000","last_peered":"2024-09-16T04:13:51.310003+0000","last_clean":"2024-09-16T04:13:51.310003+0000","last_became_active":"2024-09-16T03:43:34.399055+0000","last_became_peered":"2024-09-16T03:43:34.399055+0000","last_unstale":"2024-09-16T04:13:51.310003+0000","last_undegraded":"2024-09-16T04:13:51.310003+0000","last_fullsized":"2024-09-16T04:13:51.310003+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_clean_scrub_stamp":"2024-09-16T03:43:33.221564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:08:19.292854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.c","version":"0'0","reported_seq":1353,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:52.857977+0000","last_change":"2024-09-16T03:43:34.403649+0000","last_active":"2024-09-16T04:13:52.857977+0000","last_peered":"2024-09-16T04:13:52.857977+0000","last_clean":"2024-09-16T04:13:52.857977+0000","last_became_active":"2024-09-16T03:43:34.403187+0000","last_became_peered":"2024-09-16T03:43:34.403187+0000","last_unstale":"2024-09-16T04:13:52.857977+0000","last_undegraded":"2024-09-16T04:13:52.857977+0000","last_fullsized":"2024-09-16T04:13:52.857977+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_clean_scrub_stamp":"2024-09-16T03:43:33.221564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:21:54.130294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.13","version":"0'0","reported_seq":1354,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:51.300256+0000","last_change":"2024-09-16T03:43:34.382675+0000","last_active":"2024-09-16T04:13:51.300256+0000","last_peered":"2024-09-16T04:13:51.300256+0000","last_clean":"2024-09-16T04:13:51.300256+0000","last_became_active":"2024-09-16T03:43:34.382422+0000","last_became_peered":"2024-09-16T03:43:34.382422+0000","last_unstale":"2024-09-16T04:13:51.300256+0000","last_undegraded":"2024-09-16T04:13:51.300256+0000","last_fullsized":"2024-09-16T04:13:51.300256+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_clean_scrub_stamp":"2024-09-16T03:43:33.221564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:52:35.625157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.12","version":"0'0","reported_seq":1353,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:52.857683+0000","last_change":"2024-09-16T03:43:34.404613+0000","last_active":"2024-09-16T04:13:52.857683+0000","last_peered":"2024-09-16T04:13:52.857683+0000","last_clean":"2024-09-16T04:13:52.857683+0000","last_became_active":"2024-09-16T03:43:34.404299+0000","last_became_peered":"2024-09-16T03:43:34.404299+0000","last_unstale":"2024-09-16T04:13:52.857683+0000","last_undegraded":"2024-09-16T04:13:52.857683+0000","last_fullsized":"2024-09-16T04:13:52.857683+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_clean_scrub_stamp":"2024-09-16T03:43:33.221564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:25:52.933411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.11","version":"0'0","reported_seq":1354,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:51.299816+0000","last_change":"2024-09-16T03:43:34.355045+0000","last_active":"2024-09-16T04:13:51.299816+0000","last_peered":"2024-09-16T04:13:51.299816+0000","last_clean":"2024-09-16T04:13:51.299816+0000","last_became_active":"2024-09-16T03:43:34.354598+0000","last_became_peered":"2024-09-16T03:43:34.354598+0000","last_unstale":"2024-09-16T04:13:51.299816+0000","last_undegraded":"2024-09-16T04:13:51.299816+0000","last_fullsized":"2024-09-16T04:13:51.299816+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_clean_scrub_stamp":"2024-09-16T03:43:33.221564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:40:14.558362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.10","version":"0'0","reported_seq":1351,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:53.353474+0000","last_change":"2024-09-16T03:43:34.436288+0000","last_active":"2024-09-16T04:13:53.353474+0000","last_peered":"2024-09-16T04:13:53.353474+0000","last_clean":"2024-09-16T04:13:53.353474+0000","last_became_active":"2024-09-16T03:43:34.434548+0000","last_became_peered":"2024-09-16T03:43:34.434548+0000","last_unstale":"2024-09-16T04:13:53.353474+0000","last_undegraded":"2024-09-16T04:13:53.353474+0000","last_fullsized":"2024-09-16T04:13:53.353474+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_clean_scrub_stamp":"2024-09-16T03:43:33.221564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:03:15.848714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.17","version":"0'0","reported_seq":1349,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:53.282881+0000","last_change":"2024-09-16T03:43:34.456644+0000","last_active":"2024-09-16T04:13:53.282881+0000","last_peered":"2024-09-16T04:13:53.282881+0000","last_clean":"2024-09-16T04:13:53.282881+0000","last_became_active":"2024-09-16T03:43:34.454859+0000","last_became_peered":"2024-09-16T03:43:34.454859+0000","last_unstale":"2024-09-16T04:13:53.282881+0000","last_undegraded":"2024-09-16T04:13:53.282881+0000","last_fullsized":"2024-09-16T04:13:53.282881+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_clean_scrub_stamp":"2024-09-16T03:43:33.221564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:45:01.348488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.16","version":"0'0","reported_seq":1347,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:53.166394+0000","last_change":"2024-09-16T03:43:34.391035+0000","last_active":"2024-09-16T04:13:53.166394+0000","last_peered":"2024-09-16T04:13:53.166394+0000","last_clean":"2024-09-16T04:13:53.166394+0000","last_became_active":"2024-09-16T03:43:34.387629+0000","last_became_peered":"2024-09-16T03:43:34.387629+0000","last_unstale":"2024-09-16T04:13:53.166394+0000","last_undegraded":"2024-09-16T04:13:53.166394+0000","last_fullsized":"2024-09-16T04:13:53.166394+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_clean_scrub_stamp":"2024-09-16T03:43:33.221564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:11:59.627229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.15","version":"0'0","reported_seq":1352,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:52.859996+0000","last_change":"2024-09-16T03:43:34.387904+0000","last_active":"2024-09-16T04:13:52.859996+0000","last_peered":"2024-09-16T04:13:52.859996+0000","last_clean":"2024-09-16T04:13:52.859996+0000","last_became_active":"2024-09-16T03:43:34.387630+0000","last_became_peered":"2024-09-16T03:43:34.387630+0000","last_unstale":"2024-09-16T04:13:52.859996+0000","last_undegraded":"2024-09-16T04:13:52.859996+0000","last_fullsized":"2024-09-16T04:13:52.859996+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_clean_scrub_stamp":"2024-09-16T03:43:33.221564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:15:37.951806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.14","version":"0'0","reported_seq":1350,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:53.282153+0000","last_change":"2024-09-16T03:43:34.381624+0000","last_active":"2024-09-16T04:13:53.282153+0000","last_peered":"2024-09-16T04:13:53.282153+0000","last_clean":"2024-09-16T04:13:53.282153+0000","last_became_active":"2024-09-16T03:43:34.378385+0000","last_became_peered":"2024-09-16T03:43:34.378385+0000","last_unstale":"2024-09-16T04:13:53.282153+0000","last_undegraded":"2024-09-16T04:13:53.282153+0000","last_fullsized":"2024-09-16T04:13:53.282153+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_clean_scrub_stamp":"2024-09-16T03:43:33.221564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:20:14.916099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"14'117","reported_seq":1523,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:52.858864+0000","last_change":"2024-09-16T03:42:18.799294+0000","last_active":"2024-09-16T04:13:52.858864+0000","last_peered":"2024-09-16T04:13:52.858864+0000","last_clean":"2024-09-16T04:13:52.858864+0000","last_became_active":"2024-09-16T03:42:18.797239+0000","last_became_peered":"2024-09-16T03:42:18.797239+0000","last_unstale":"2024-09-16T04:13:52.858864+0000","last_undegraded":"2024-09-16T04:13:52.858864+0000","last_fullsized":"2024-09-16T04:13:52.858864+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:42:17.544237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:42:17.544237+0000","last_clean_scrub_stamp":"2024-09-16T03:42:17.544237+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:01:21.017593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":1374,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:51.300417+0000","last_change":"2024-09-16T03:42:27.630965+0000","last_active":"2024-09-16T04:13:51.300417+0000","last_peered":"2024-09-16T04:13:51.300417+0000","last_clean":"2024-09-16T04:13:51.300417+0000","last_became_active":"2024-09-16T03:42:25.615930+0000","last_became_peered":"2024-09-16T03:42:25.615930+0000","last_unstale":"2024-09-16T04:13:51.300417+0000","last_undegraded":"2024-09-16T04:13:51.300417+0000","last_fullsized":"2024-09-16T04:13:51.300417+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:42:24.583922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:42:24.583922+0000","last_clean_scrub_stamp":"2024-09-16T03:42:24.583922+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:59:59.653118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00099470900000000009,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.1e","version":"0'0","reported_seq":1355,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:51.302860+0000","last_change":"2024-09-16T03:43:34.401653+0000","last_active":"2024-09-16T04:13:51.302860+0000","last_peered":"2024-09-16T04:13:51.302860+0000","last_clean":"2024-09-16T04:13:51.302860+0000","last_became_active":"2024-09-16T03:43:34.397708+0000","last_became_peered":"2024-09-16T03:43:34.397708+0000","last_unstale":"2024-09-16T04:13:51.302860+0000","last_undegraded":"2024-09-16T04:13:51.302860+0000","last_fullsized":"2024-09-16T04:13:51.302860+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_clean_scrub_stamp":"2024-09-16T03:43:33.221564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:16:17.302952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1372,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:52.859154+0000","last_change":"2024-09-16T03:42:27.624504+0000","last_active":"2024-09-16T04:13:52.859154+0000","last_peered":"2024-09-16T04:13:52.859154+0000","last_clean":"2024-09-16T04:13:52.859154+0000","last_became_active":"2024-09-16T03:42:25.612386+0000","last_became_peered":"2024-09-16T03:42:25.612386+0000","last_unstale":"2024-09-16T04:13:52.859154+0000","last_undegraded":"2024-09-16T04:13:52.859154+0000","last_fullsized":"2024-09-16T04:13:52.859154+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:42:24.583922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:42:24.583922+0000","last_clean_scrub_stamp":"2024-09-16T03:42:24.583922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:00:49.446880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081399800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1d","version":"0'0","reported_seq":1372,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:51.310252+0000","last_change":"2024-09-16T03:43:34.408908+0000","last_active":"2024-09-16T04:13:51.310252+0000","last_peered":"2024-09-16T04:13:51.310252+0000","last_clean":"2024-09-16T04:13:51.310252+0000","last_became_active":"2024-09-16T03:43:34.399300+0000","last_became_peered":"2024-09-16T03:43:34.399300+0000","last_unstale":"2024-09-16T04:13:51.310252+0000","last_undegraded":"2024-09-16T04:13:51.310252+0000","last_fullsized":"2024-09-16T04:13:51.310252+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_clean_scrub_stamp":"2024-09-16T03:43:33.221564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:25:51.517103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1370,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:53.353598+0000","last_change":"2024-09-16T03:42:27.619252+0000","last_active":"2024-09-16T04:13:53.353598+0000","last_peered":"2024-09-16T04:13:53.353598+0000","last_clean":"2024-09-16T04:13:53.353598+0000","last_became_active":"2024-09-16T03:42:25.607533+0000","last_became_peered":"2024-09-16T03:42:25.607533+0000","last_unstale":"2024-09-16T04:13:53.353598+0000","last_undegraded":"2024-09-16T04:13:53.353598+0000","last_fullsized":"2024-09-16T04:13:53.353598+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:42:24.583922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:42:24.583922+0000","last_clean_scrub_stamp":"2024-09-16T03:42:24.583922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:25:58.536144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089083299999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1c","version":"0'0","reported_seq":1349,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:53.282306+0000","last_change":"2024-09-16T03:43:34.421587+0000","last_active":"2024-09-16T04:13:53.282306+0000","last_peered":"2024-09-16T04:13:53.282306+0000","last_clean":"2024-09-16T04:13:53.282306+0000","last_became_active":"2024-09-16T03:43:34.421320+0000","last_became_peered":"2024-09-16T03:43:34.421320+0000","last_unstale":"2024-09-16T04:13:53.282306+0000","last_undegraded":"2024-09-16T04:13:53.282306+0000","last_fullsized":"2024-09-16T04:13:53.282306+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_clean_scrub_stamp":"2024-09-16T03:43:33.221564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:47:25.449200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":1375,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:51.299718+0000","last_change":"2024-09-16T03:42:27.632050+0000","last_active":"2024-09-16T04:13:51.299718+0000","last_peered":"2024-09-16T04:13:51.299718+0000","last_clean":"2024-09-16T04:13:51.299718+0000","last_became_active":"2024-09-16T03:42:25.617896+0000","last_became_peered":"2024-09-16T03:42:25.617896+0000","last_unstale":"2024-09-16T04:13:51.299718+0000","last_undegraded":"2024-09-16T04:13:51.299718+0000","last_fullsized":"2024-09-16T04:13:51.299718+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:42:24.583922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:42:24.583922+0000","last_clean_scrub_stamp":"2024-09-16T03:42:24.583922+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:41:08.754274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021072529999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.1f","version":"0'0","reported_seq":1352,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:53.356655+0000","last_change":"2024-09-16T03:43:34.419004+0000","last_active":"2024-09-16T04:13:53.356655+0000","last_peered":"2024-09-16T04:13:53.356655+0000","last_clean":"2024-09-16T04:13:53.356655+0000","last_became_active":"2024-09-16T03:43:34.415747+0000","last_became_peered":"2024-09-16T03:43:34.415747+0000","last_unstale":"2024-09-16T04:13:53.356655+0000","last_undegraded":"2024-09-16T04:13:53.356655+0000","last_fullsized":"2024-09-16T04:13:53.356655+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_clean_scrub_stamp":"2024-09-16T03:43:33.221564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:01.644098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1367,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:53.166325+0000","last_change":"2024-09-16T03:42:27.631260+0000","last_active":"2024-09-16T04:13:53.166325+0000","last_peered":"2024-09-16T04:13:53.166325+0000","last_clean":"2024-09-16T04:13:53.166325+0000","last_became_active":"2024-09-16T03:42:25.609631+0000","last_became_peered":"2024-09-16T03:42:25.609631+0000","last_unstale":"2024-09-16T04:13:53.166325+0000","last_undegraded":"2024-09-16T04:13:53.166325+0000","last_fullsized":"2024-09-16T04:13:53.166325+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:42:24.583922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:42:24.583922+0000","last_clean_scrub_stamp":"2024-09-16T03:42:24.583922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:58:49.091085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001288897,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.19","version":"0'0","reported_seq":1347,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:53.166637+0000","last_change":"2024-09-16T03:43:34.341660+0000","last_active":"2024-09-16T04:13:53.166637+0000","last_peered":"2024-09-16T04:13:53.166637+0000","last_clean":"2024-09-16T04:13:53.166637+0000","last_became_active":"2024-09-16T03:43:34.332234+0000","last_became_peered":"2024-09-16T03:43:34.332234+0000","last_unstale":"2024-09-16T04:13:53.166637+0000","last_undegraded":"2024-09-16T04:13:53.166637+0000","last_fullsized":"2024-09-16T04:13:53.166637+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_clean_scrub_stamp":"2024-09-16T03:43:33.221564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:20:27.326212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1372,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:52.859726+0000","last_change":"2024-09-16T03:42:27.625091+0000","last_active":"2024-09-16T04:13:52.859726+0000","last_peered":"2024-09-16T04:13:52.859726+0000","last_clean":"2024-09-16T04:13:52.859726+0000","last_became_active":"2024-09-16T03:42:25.614458+0000","last_became_peered":"2024-09-16T03:42:25.614458+0000","last_unstale":"2024-09-16T04:13:52.859726+0000","last_undegraded":"2024-09-16T04:13:52.859726+0000","last_fullsized":"2024-09-16T04:13:52.859726+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:42:24.583922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:42:24.583922+0000","last_clean_scrub_stamp":"2024-09-16T03:42:24.583922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:26:14.681436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012701769999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.18","version":"0'0","reported_seq":1347,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:53.165619+0000","last_change":"2024-09-16T03:43:34.362758+0000","last_active":"2024-09-16T04:13:53.165619+0000","last_peered":"2024-09-16T04:13:53.165619+0000","last_clean":"2024-09-16T04:13:53.165619+0000","last_became_active":"2024-09-16T03:43:34.354705+0000","last_became_peered":"2024-09-16T03:43:34.354705+0000","last_unstale":"2024-09-16T04:13:53.165619+0000","last_undegraded":"2024-09-16T04:13:53.165619+0000","last_fullsized":"2024-09-16T04:13:53.165619+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_clean_scrub_stamp":"2024-09-16T03:43:33.221564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:16:36.345303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1366,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:53.166473+0000","last_change":"2024-09-16T03:42:27.631227+0000","last_active":"2024-09-16T04:13:53.166473+0000","last_peered":"2024-09-16T04:13:53.166473+0000","last_clean":"2024-09-16T04:13:53.166473+0000","last_became_active":"2024-09-16T03:42:25.608453+0000","last_became_peered":"2024-09-16T03:42:25.608453+0000","last_unstale":"2024-09-16T04:13:53.166473+0000","last_undegraded":"2024-09-16T04:13:53.166473+0000","last_fullsized":"2024-09-16T04:13:53.166473+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:42:24.583922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:42:24.583922+0000","last_clean_scrub_stamp":"2024-09-16T03:42:24.583922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:00:11.921493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001163533,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.1b","version":"0'0","reported_seq":1352,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:52.859563+0000","last_change":"2024-09-16T03:43:34.381629+0000","last_active":"2024-09-16T04:13:52.859563+0000","last_peered":"2024-09-16T04:13:52.859563+0000","last_clean":"2024-09-16T04:13:52.859563+0000","last_became_active":"2024-09-16T03:43:34.381335+0000","last_became_peered":"2024-09-16T03:43:34.381335+0000","last_unstale":"2024-09-16T04:13:52.859563+0000","last_undegraded":"2024-09-16T04:13:52.859563+0000","last_fullsized":"2024-09-16T04:13:52.859563+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_clean_scrub_stamp":"2024-09-16T03:43:33.221564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:30:50.930529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1371,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:52.861031+0000","last_change":"2024-09-16T03:42:27.624922+0000","last_active":"2024-09-16T04:13:52.861031+0000","last_peered":"2024-09-16T04:13:52.861031+0000","last_clean":"2024-09-16T04:13:52.861031+0000","last_became_active":"2024-09-16T03:42:25.615293+0000","last_became_peered":"2024-09-16T03:42:25.615293+0000","last_unstale":"2024-09-16T04:13:52.861031+0000","last_undegraded":"2024-09-16T04:13:52.861031+0000","last_fullsized":"2024-09-16T04:13:52.861031+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:42:24.583922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:42:24.583922+0000","last_clean_scrub_stamp":"2024-09-16T03:42:24.583922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:46:11.677251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010681690000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1a","version":"0'0","reported_seq":1349,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:53.281860+0000","last_change":"2024-09-16T03:43:34.382293+0000","last_active":"2024-09-16T04:13:53.281860+0000","last_peered":"2024-09-16T04:13:53.281860+0000","last_clean":"2024-09-16T04:13:53.281860+0000","last_became_active":"2024-09-16T03:43:34.372307+0000","last_became_peered":"2024-09-16T03:43:34.372307+0000","last_unstale":"2024-09-16T04:13:53.281860+0000","last_undegraded":"2024-09-16T04:13:53.281860+0000","last_fullsized":"2024-09-16T04:13:53.281860+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_clean_scrub_stamp":"2024-09-16T03:43:33.221564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:49:25.481308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.8","version":"0'0","reported_seq":1351,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:53.280246+0000","last_change":"2024-09-16T03:43:34.386011+0000","last_active":"2024-09-16T04:13:53.280246+0000","last_peered":"2024-09-16T04:13:53.280246+0000","last_clean":"2024-09-16T04:13:53.280246+0000","last_became_active":"2024-09-16T03:43:34.378483+0000","last_became_peered":"2024-09-16T03:43:34.378483+0000","last_unstale":"2024-09-16T04:13:53.280246+0000","last_undegraded":"2024-09-16T04:13:53.280246+0000","last_fullsized":"2024-09-16T04:13:53.280246+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_clean_scrub_stamp":"2024-09-16T03:43:33.221564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:25:42.568633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.9","version":"0'0","reported_seq":1374,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:51.306525+0000","last_change":"2024-09-16T03:43:34.389268+0000","last_active":"2024-09-16T04:13:51.306525+0000","last_peered":"2024-09-16T04:13:51.306525+0000","last_clean":"2024-09-16T04:13:51.306525+0000","last_became_active":"2024-09-16T03:43:34.378733+0000","last_became_peered":"2024-09-16T03:43:34.378733+0000","last_unstale":"2024-09-16T04:13:51.306525+0000","last_undegraded":"2024-09-16T04:13:51.306525+0000","last_fullsized":"2024-09-16T04:13:51.306525+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_clean_scrub_stamp":"2024-09-16T03:43:33.221564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:16:08.231273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.a","version":"0'0","reported_seq":1374,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:51.306820+0000","last_change":"2024-09-16T03:43:34.390824+0000","last_active":"2024-09-16T04:13:51.306820+0000","last_peered":"2024-09-16T04:13:51.306820+0000","last_clean":"2024-09-16T04:13:51.306820+0000","last_became_active":"2024-09-16T03:43:34.390049+0000","last_became_peered":"2024-09-16T03:43:34.390049+0000","last_unstale":"2024-09-16T04:13:51.306820+0000","last_undegraded":"2024-09-16T04:13:51.306820+0000","last_fullsized":"2024-09-16T04:13:51.306820+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_clean_scrub_stamp":"2024-09-16T03:43:33.221564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:48:31.039881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.b","version":"0'0","reported_seq":1348,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:53.165918+0000","last_change":"2024-09-16T03:43:34.332247+0000","last_active":"2024-09-16T04:13:53.165918+0000","last_peered":"2024-09-16T04:13:53.165918+0000","last_clean":"2024-09-16T04:13:53.165918+0000","last_became_active":"2024-09-16T03:43:34.328425+0000","last_became_peered":"2024-09-16T03:43:34.328425+0000","last_unstale":"2024-09-16T04:13:53.165918+0000","last_undegraded":"2024-09-16T04:13:53.165918+0000","last_fullsized":"2024-09-16T04:13:53.165918+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_clean_scrub_stamp":"2024-09-16T03:43:33.221564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:38:08.042571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.4","version":"0'0","reported_seq":1351,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:53.282653+0000","last_change":"2024-09-16T03:43:34.392184+0000","last_active":"2024-09-16T04:13:53.282653+0000","last_peered":"2024-09-16T04:13:53.282653+0000","last_clean":"2024-09-16T04:13:53.282653+0000","last_became_active":"2024-09-16T03:43:34.391868+0000","last_became_peered":"2024-09-16T03:43:34.391868+0000","last_unstale":"2024-09-16T04:13:53.282653+0000","last_undegraded":"2024-09-16T04:13:53.282653+0000","last_fullsized":"2024-09-16T04:13:53.282653+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_clean_scrub_stamp":"2024-09-16T03:43:33.221564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:02:40.567145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.5","version":"0'0","reported_seq":1351,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:53.280846+0000","last_change":"2024-09-16T03:43:34.419668+0000","last_active":"2024-09-16T04:13:53.280846+0000","last_peered":"2024-09-16T04:13:53.280846+0000","last_clean":"2024-09-16T04:13:53.280846+0000","last_became_active":"2024-09-16T03:43:34.419410+0000","last_became_peered":"2024-09-16T03:43:34.419410+0000","last_unstale":"2024-09-16T04:13:53.280846+0000","last_undegraded":"2024-09-16T04:13:53.280846+0000","last_fullsized":"2024-09-16T04:13:53.280846+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_clean_scrub_stamp":"2024-09-16T03:43:33.221564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:37:43.007953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.6","version":"0'0","reported_seq":1374,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:51.309271+0000","last_change":"2024-09-16T03:43:34.386607+0000","last_active":"2024-09-16T04:13:51.309271+0000","last_peered":"2024-09-16T04:13:51.309271+0000","last_clean":"2024-09-16T04:13:51.309271+0000","last_became_active":"2024-09-16T03:43:34.381834+0000","last_became_peered":"2024-09-16T03:43:34.381834+0000","last_unstale":"2024-09-16T04:13:51.309271+0000","last_undegraded":"2024-09-16T04:13:51.309271+0000","last_fullsized":"2024-09-16T04:13:51.309271+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_clean_scrub_stamp":"2024-09-16T03:43:33.221564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:06:41.696398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.7","version":"0'0","reported_seq":1349,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:53.164974+0000","last_change":"2024-09-16T03:43:34.296793+0000","last_active":"2024-09-16T04:13:53.164974+0000","last_peered":"2024-09-16T04:13:53.164974+0000","last_clean":"2024-09-16T04:13:53.164974+0000","last_became_active":"2024-09-16T03:43:34.292756+0000","last_became_peered":"2024-09-16T03:43:34.292756+0000","last_unstale":"2024-09-16T04:13:53.164974+0000","last_undegraded":"2024-09-16T04:13:53.164974+0000","last_fullsized":"2024-09-16T04:13:53.164974+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_clean_scrub_stamp":"2024-09-16T03:43:33.221564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:12:04.533479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.0","version":"0'0","reported_seq":1355,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:52.859564+0000","last_change":"2024-09-16T03:43:34.352792+0000","last_active":"2024-09-16T04:13:52.859564+0000","last_peered":"2024-09-16T04:13:52.859564+0000","last_clean":"2024-09-16T04:13:52.859564+0000","last_became_active":"2024-09-16T03:43:34.351273+0000","last_became_peered":"2024-09-16T03:43:34.351273+0000","last_unstale":"2024-09-16T04:13:52.859564+0000","last_undegraded":"2024-09-16T04:13:52.859564+0000","last_fullsized":"2024-09-16T04:13:52.859564+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_clean_scrub_stamp":"2024-09-16T03:43:33.221564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:50:21.469378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1","version":"22'7","reported_seq":1362,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:53.164484+0000","last_change":"2024-09-16T03:43:34.397561+0000","last_active":"2024-09-16T04:13:53.164484+0000","last_peered":"2024-09-16T04:13:53.164484+0000","last_clean":"2024-09-16T04:13:53.164484+0000","last_became_active":"2024-09-16T03:43:34.382744+0000","last_became_peered":"2024-09-16T03:43:34.382744+0000","last_unstale":"2024-09-16T04:13:53.164484+0000","last_undegraded":"2024-09-16T04:13:53.164484+0000","last_fullsized":"2024-09-16T04:13:53.164484+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_clean_scrub_stamp":"2024-09-16T03:43:33.221564+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:54:00.268175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.2","version":"0'0","reported_seq":1355,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:51.300709+0000","last_change":"2024-09-16T03:43:34.402175+0000","last_active":"2024-09-16T04:13:51.300709+0000","last_peered":"2024-09-16T04:13:51.300709+0000","last_clean":"2024-09-16T04:13:51.300709+0000","last_became_active":"2024-09-16T03:43:34.402014+0000","last_became_peered":"2024-09-16T03:43:34.402014+0000","last_unstale":"2024-09-16T04:13:51.300709+0000","last_undegraded":"2024-09-16T04:13:51.300709+0000","last_fullsized":"2024-09-16T04:13:51.300709+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_clean_scrub_stamp":"2024-09-16T03:43:33.221564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:46:39.366181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.3","version":"0'0","reported_seq":1374,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:51.306977+0000","last_change":"2024-09-16T03:43:34.380947+0000","last_active":"2024-09-16T04:13:51.306977+0000","last_peered":"2024-09-16T04:13:51.306977+0000","last_clean":"2024-09-16T04:13:51.306977+0000","last_became_active":"2024-09-16T03:43:34.379988+0000","last_became_peered":"2024-09-16T03:43:34.379988+0000","last_unstale":"2024-09-16T04:13:51.306977+0000","last_undegraded":"2024-09-16T04:13:51.306977+0000","last_fullsized":"2024-09-16T04:13:51.306977+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_clean_scrub_stamp":"2024-09-16T03:43:33.221564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:21:15.646033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":31,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":18838,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":12,"seq":51539607943,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":71596,"kb_used_data":3764,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94300244,"statfs":{"total":96636764160,"available":96563449856,"internally_reserved":0,"allocated":3854336,"data_stored":1504358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":11,"seq":47244640646,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":71228,"kb_used_data":3764,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94300612,"statfs":{"total":96636764160,"available":96563826688,"internally_reserved":0,"allocated":3854336,"data_stored":1504344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640643,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":67116,"kb_used_data":3812,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94304724,"statfs":{"total":96636764160,"available":96568037376,"internally_reserved":0,"allocated":3903488,"data_stored":2033267,"data_compressed":9419,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1593,"internal_metadata":64616903},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640644,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":71224,"kb_used_data":3760,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94300616,"statfs":{"total":96636764160,"available":96563830784,"internally_reserved":0,"allocated":3850240,"data_stored":1504339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":68876742},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640645,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":79552,"kb_used_data":3768,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94292288,"statfs":{"total":96636764160,"available":96555302912,"internally_reserved":0,"allocated":3858432,"data_stored":1504363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640644,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":71276,"kb_used_data":3812,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94300564,"statfs":{"total":96636764160,"available":96563777536,"internally_reserved":0,"allocated":3903488,"data_stored":2033267,"data_compressed":9419,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":9419,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":9419,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:14:20.535 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:14:20.660 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.653+0000 7fa8a5eba640 10 monclient: get_monmap_and_config 2024-09-16T04:14:20.660 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.653+0000 7fa8a5eba640 10 monclient: build_initial_monmap 2024-09-16T04:14:20.660 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.653+0000 7fa8a5eba640 10 monclient: monmap: 2024-09-16T04:14:20.660 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:14:20.660 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:14:20.660 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:14:20.659085+0000 2024-09-16T04:14:20.660 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:14:20.659085+0000 2024-09-16T04:14:20.661 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:14:20.661 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:20.661 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:14:20.661 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:14:20.661 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:14:20.661 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:14:20.661 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:14:20.661 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:20.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.653+0000 7fa8a5eba640 1 Processor -- start 2024-09-16T04:14:20.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.653+0000 7fa8a5eba640 1 -- start start 2024-09-16T04:14:20.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.653+0000 7fa8a5eba640 10 monclient: init 2024-09-16T04:14:20.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.657+0000 7fa8a5eba640 10 monclient: _reopen_session rank -1 2024-09-16T04:14:20.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.657+0000 7fa8a5eba640 10 monclient: _add_conns ranks=[0,3,4,2,1] 2024-09-16T04:14:20.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.657+0000 7fa8a5eba640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa8a00a1330 addr v1:172.21.15.42:6789/0 2024-09-16T04:14:20.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.657+0000 7fa8a5eba640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fa8a006dac0 0x7fa8a013a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:20.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.657+0000 7fa8a5eba640 10 monclient(hunting): picked mon.noname-a con 0x7fa8a006dac0 addr v2:172.21.15.42:3300/0 2024-09-16T04:14:20.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.657+0000 7fa8a5eba640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fa8a00a1e20 0x7fa8a013c890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:20.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.657+0000 7fa8a5eba640 10 monclient(hunting): picked mon.noname-c con 0x7fa8a00a1e20 addr v2:172.21.15.154:3300/0 2024-09-16T04:14:20.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.657+0000 7fa8a5eba640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa8a009c090 con 0x7fa8a00a1330 2024-09-16T04:14:20.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.657+0000 7fa8a5eba640 10 monclient(hunting): start opening mon connection 2024-09-16T04:14:20.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.657+0000 7fa8a5eba640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7fa8a009f610 con 0x7fa8a006dac0 2024-09-16T04:14:20.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.657+0000 7fa8a5eba640 10 monclient(hunting): start opening mon connection 2024-09-16T04:14:20.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.657+0000 7fa8a5eba640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fa8a009f8f0 con 0x7fa8a00a1e20 2024-09-16T04:14:20.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.657+0000 7fa8a5eba640 10 monclient(hunting): _renew_subs 2024-09-16T04:14:20.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.657+0000 7fa8a5eba640 10 monclient(hunting): authenticate will time out at 2907.628511s 2024-09-16T04:14:20.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.657+0000 7fa89effd640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fa8a006dac0 0x7fa8a013a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:20.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.657+0000 7fa89effd640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fa8a006dac0 0x7fa8a013a340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:47940/0 (socket says 172.21.15.42:47940) 2024-09-16T04:14:20.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.657+0000 7fa89effd640 1 -- 172.21.15.42:0/1563860180 learned_addr learned my addr 172.21.15.42:0/1563860180 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:14:20.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.657+0000 7fa89effd640 10 monclient(hunting): get_auth_request con 0x7fa8a006dac0 auth_method 0 2024-09-16T04:14:20.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.657+0000 7fa89effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:14:20.666 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.657+0000 7fa89effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:20.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.657+0000 7fa89effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:20.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.657+0000 7fa89effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:14:20.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.657+0000 7fa89effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:14:20.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.657+0000 7fa89effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:14:20.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.657+0000 7fa89ffff640 1 --2- 172.21.15.42:0/1563860180 >> v2:172.21.15.154:3300/0 conn(0x7fa8a00a1e20 0x7fa8a013c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:20.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.657+0000 7fa89effd640 10 monclient(hunting): handle_auth_done global_id 1104153 payload 274 2024-09-16T04:14:20.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.657+0000 7fa89effd640 1 -- 172.21.15.42:0/1563860180 >> v2:172.21.15.154:3300/0 conn(0x7fa8a00a1e20 msgr2=0x7fa8a013c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:14:20.667 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.657+0000 7fa89effd640 1 --2- 172.21.15.42:0/1563860180 >> v2:172.21.15.154:3300/0 conn(0x7fa8a00a1e20 0x7fa8a013c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:20.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.657+0000 7fa89effd640 1 -- 172.21.15.42:0/1563860180 >> v1:172.21.15.42:6789/0 conn(0x7fa8a00a1330 legacy=0x7fa8a00a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:20.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.657+0000 7fa89effd640 10 monclient: _finish_hunting 0 2024-09-16T04:14:20.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.657+0000 7fa89effd640 1 monclient: found mon.noname-a 2024-09-16T04:14:20.668 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.657+0000 7fa89effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:14:20.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.657+0000 7fa89effd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:14:20.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.657+0000 7fa89effd640 1 -- 172.21.15.42:0/1563860180 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8a009fbd0 con 0x7fa8a006dac0 2024-09-16T04:14:20.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.657+0000 7fa89ffff640 10 monclient: get_auth_request con 0x7fa8a00a1e20 auth_method 0 2024-09-16T04:14:20.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.657+0000 7fa89ffff640 1 --2- 172.21.15.42:0/1563860180 >> v2:172.21.15.154:3300/0 conn(0x7fa8a00a1e20 0x7fa8a013c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T04:14:20.669 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.657+0000 7fa89effd640 1 --2- 172.21.15.42:0/1563860180 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa8a006dac0 0x7fa8a013a340 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fa8a009c800 tx=0x7fa89401c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2993e2adb1befc99 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:14:20.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.657+0000 7fa89e7fc640 1 -- 172.21.15.42:0/1563860180 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8940b0070 con 0x7fa8a006dac0 2024-09-16T04:14:20.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.657+0000 7fa89e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:20.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.657+0000 7fa89e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:14:20.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.657+0000 7fa89e7fc640 10 monclient: dump: 2024-09-16T04:14:20.674 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:14:20.674 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:14:20.674 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:20.674 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:20.674 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:20.674 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:20.674 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:14:20.674 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:14:20.675 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:14:20.675 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:20.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.657+0000 7fa89e7fc640 10 monclient: _finish_auth 0 2024-09-16T04:14:20.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.657+0000 7fa89e7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:14:20.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.657+0000 7fa89e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:14:20.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.657+0000 7fa8a5eba640 5 monclient: authenticate success, global_id 1104153 2024-09-16T04:14:20.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.657+0000 7fa8a5eba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:14:20.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.657+0000 7fa89e7fc640 1 -- 172.21.15.42:0/1563860180 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa8940ab600 con 0x7fa8a006dac0 2024-09-16T04:14:20.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.657+0000 7fa89e7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:14:20.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.657+0000 7fa89e7fc640 1 -- 172.21.15.42:0/1563860180 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8940ab9d0 con 0x7fa8a006dac0 2024-09-16T04:14:20.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.657+0000 7fa89e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:20.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.657+0000 7fa89e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:14:20.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.657+0000 7fa89e7fc640 10 monclient: dump: 2024-09-16T04:14:20.675 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:14:20.676 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:14:20.676 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:20.676 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:20.676 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:20.676 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:20.676 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:14:20.676 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:14:20.676 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:14:20.676 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:20.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.657+0000 7fa8a5eba640 10 monclient: get_monmap_and_config success 2024-09-16T04:14:20.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.657+0000 7fa8a5eba640 4 set_mon_vals no callback set 2024-09-16T04:14:20.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.657+0000 7fa8a5eba640 10 monclient: shutdown 2024-09-16T04:14:20.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.657+0000 7fa8a5eba640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:14:20.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.657+0000 7fa8a5eba640 1 -- 172.21.15.42:0/1563860180 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa8a006dac0 msgr2=0x7fa8a013a340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:20.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.661+0000 7fa8a5eba640 1 --2- 172.21.15.42:0/1563860180 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa8a006dac0 0x7fa8a013a340 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fa8a009c800 tx=0x7fa89401c470 comp rx=0 tx=0).stop 2024-09-16T04:14:20.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.661+0000 7fa8a5eba640 1 -- 172.21.15.42:0/1563860180 shutdown_connections 2024-09-16T04:14:20.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.661+0000 7fa8a5eba640 1 --2- 172.21.15.42:0/1563860180 >> v2:172.21.15.154:3300/0 conn(0x7fa8a00a1e20 0x7fa8a013c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:20.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.661+0000 7fa8a5eba640 1 --2- 172.21.15.42:0/1563860180 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa8a006dac0 0x7fa8a013a340 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:20.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.661+0000 7fa8a5eba640 1 -- 172.21.15.42:0/1563860180 >> 172.21.15.42:0/1563860180 conn(0x7fa8a00a2420 msgr2=0x7fa8a00a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:14:20.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.661+0000 7fa8a5eba640 1 -- 172.21.15.42:0/1563860180 shutdown_connections 2024-09-16T04:14:20.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.661+0000 7fa8a5eba640 1 -- 172.21.15.42:0/1563860180 wait complete. 2024-09-16T04:14:20.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.661+0000 7fa8a5eba640 10 monclient: build_initial_monmap 2024-09-16T04:14:20.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.661+0000 7fa8a5eba640 10 monclient: monmap: 2024-09-16T04:14:20.677 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:14:20.677 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:14:20.677 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:14:20.678 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:14:20.678 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:14:20.678 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:20.678 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:14:20.678 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:14:20.678 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:14:20.678 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:20.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.661+0000 7fa8a5eba640 1 librados: starting msgr at 2024-09-16T04:14:20.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.661+0000 7fa8a5eba640 1 librados: starting objecter 2024-09-16T04:14:20.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.661+0000 7fa8a5eba640 1 Processor -- start 2024-09-16T04:14:20.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.661+0000 7fa8a5eba640 1 -- start start 2024-09-16T04:14:20.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.661+0000 7fa8a5eba640 1 librados: setting wanted keys 2024-09-16T04:14:20.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.661+0000 7fa8a5eba640 1 librados: calling monclient init 2024-09-16T04:14:20.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.661+0000 7fa8a5eba640 10 monclient: init 2024-09-16T04:14:20.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.661+0000 7fa8a5eba640 10 monclient: _reopen_session rank -1 2024-09-16T04:14:20.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.661+0000 7fa8a5eba640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:14:20.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.661+0000 7fa8a5eba640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa8a00a1330 0x7fa8a01cedb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:20.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.661+0000 7fa8a5eba640 10 monclient(hunting): picked mon.noname-a con 0x7fa8a00a1330 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:14:20.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.661+0000 7fa8a5eba640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa8a00a1e20 0x7fa8a0133710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:20.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.661+0000 7fa8a5eba640 10 monclient(hunting): picked mon.noname-c con 0x7fa8a00a1e20 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:14:20.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.661+0000 7fa8a5eba640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa8a0133c50 0x7fa8a011c1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:20.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.661+0000 7fa8a5eba640 10 monclient(hunting): picked mon.noname-b con 0x7fa8a0133c50 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:14:20.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.661+0000 7fa8a5eba640 10 monclient(hunting): start opening mon connection 2024-09-16T04:14:20.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.661+0000 7fa8a5eba640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fa8a013f800 con 0x7fa8a00a1330 2024-09-16T04:14:20.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.661+0000 7fa8a5eba640 10 monclient(hunting): start opening mon connection 2024-09-16T04:14:20.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.661+0000 7fa8a5eba640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fa8a009ef80 con 0x7fa8a0133c50 2024-09-16T04:14:20.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.661+0000 7fa8a5eba640 10 monclient(hunting): start opening mon connection 2024-09-16T04:14:20.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.661+0000 7fa8a5eba640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fa8a013f980 con 0x7fa8a00a1e20 2024-09-16T04:14:20.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.661+0000 7fa8a5eba640 10 monclient(hunting): _renew_subs 2024-09-16T04:14:20.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.661+0000 7fa8a5eba640 10 monclient(hunting): authenticate will time out at 2907.633750s 2024-09-16T04:14:20.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.661+0000 7fa89f7fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa8a00a1330 0x7fa8a01cedb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:20.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.661+0000 7fa89effd640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa8a00a1e20 0x7fa8a0133710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:20.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.661+0000 7fa89f7fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa8a00a1330 0x7fa8a01cedb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:47950/0 (socket says 172.21.15.42:47950) 2024-09-16T04:14:20.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.661+0000 7fa89f7fe640 1 -- 172.21.15.42:0/808387492 learned_addr learned my addr 172.21.15.42:0/808387492 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:14:20.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.661+0000 7fa89f7fe640 10 monclient(hunting): get_auth_request con 0x7fa8a00a1330 auth_method 0 2024-09-16T04:14:20.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.661+0000 7fa89f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:14:20.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.661+0000 7fa89f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:20.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.661+0000 7fa89f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:20.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.661+0000 7fa89effd640 10 monclient(hunting): get_auth_request con 0x7fa8a00a1e20 auth_method 0 2024-09-16T04:14:20.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.661+0000 7fa89effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:14:20.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.661+0000 7fa89effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:20.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.661+0000 7fa89effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:20.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.661+0000 7fa89f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:14:20.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.661+0000 7fa89f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:14:20.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.661+0000 7fa89f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:14:20.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.661+0000 7fa89effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:14:20.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.661+0000 7fa89effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:14:20.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.661+0000 7fa89effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:14:20.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.661+0000 7fa89f7fe640 10 monclient(hunting): handle_auth_done global_id 1104156 payload 824 2024-09-16T04:14:20.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.661+0000 7fa89ffff640 1 --2- 172.21.15.42:0/808387492 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa8a0133c50 0x7fa8a011c1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:20.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.661+0000 7fa89f7fe640 1 -- 172.21.15.42:0/808387492 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa8a00a1e20 msgr2=0x7fa8a0133710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:20.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.661+0000 7fa89f7fe640 1 --2- 172.21.15.42:0/808387492 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa8a00a1e20 0x7fa8a0133710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:20.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.661+0000 7fa89f7fe640 1 -- 172.21.15.42:0/808387492 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa8a0133c50 msgr2=0x7fa8a011c1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:20.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.661+0000 7fa89f7fe640 1 --2- 172.21.15.42:0/808387492 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa8a0133c50 0x7fa8a011c1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:20.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.661+0000 7fa89f7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:14:20.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.661+0000 7fa89f7fe640 1 monclient: found mon.noname-a 2024-09-16T04:14:20.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.661+0000 7fa89f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:14:20.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.661+0000 7fa89f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:14:20.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.661+0000 7fa89f7fe640 1 -- 172.21.15.42:0/808387492 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8a01cf2f0 con 0x7fa8a00a1330 2024-09-16T04:14:20.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.661+0000 7fa89effd640 1 --2- 172.21.15.42:0/808387492 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa8a00a1e20 0x7fa8a0133710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:14:20.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.661+0000 7fa89f7fe640 1 --2- 172.21.15.42:0/808387492 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa8a00a1330 0x7fa8a01cedb0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fa89000c450 tx=0x7fa89000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:14:20.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.661+0000 7fa887fff640 10 client.?.objecter ms_handle_connect 0x7fa8a00a1330 2024-09-16T04:14:20.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.661+0000 7fa887fff640 10 client.?.objecter resend_mon_ops 2024-09-16T04:14:20.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.661+0000 7fa887fff640 1 -- 172.21.15.42:0/808387492 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa890016020 con 0x7fa8a00a1330 2024-09-16T04:14:20.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.661+0000 7fa887fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:20.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.661+0000 7fa887fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:14:20.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.661+0000 7fa887fff640 10 monclient: dump: 2024-09-16T04:14:20.682 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:14:20.683 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:14:20.683 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:20.683 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:20.683 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:20.683 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:20.683 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:14:20.683 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:14:20.683 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:14:20.683 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:20.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.661+0000 7fa887fff640 10 monclient: _finish_auth 0 2024-09-16T04:14:20.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.661+0000 7fa887fff640 10 monclient: _check_auth_tickets 2024-09-16T04:14:20.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.661+0000 7fa887fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:14:20.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.661+0000 7fa8a5eba640 5 monclient: authenticate success, global_id 1104156 2024-09-16T04:14:20.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.661+0000 7fa887fff640 1 -- 172.21.15.42:0/808387492 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa8900052d0 con 0x7fa8a00a1330 2024-09-16T04:14:20.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.661+0000 7fa8a5eba640 10 monclient: _renew_subs 2024-09-16T04:14:20.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.661+0000 7fa8a5eba640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:14:20.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.661+0000 7fa8a5eba640 1 -- 172.21.15.42:0/808387492 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8a01200c0 con 0x7fa8a00a1330 2024-09-16T04:14:20.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.661+0000 7fa887fff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:14:20.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.661+0000 7fa887fff640 1 -- 172.21.15.42:0/808387492 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa890007260 con 0x7fa8a00a1330 2024-09-16T04:14:20.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.661+0000 7fa887fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:20.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.661+0000 7fa887fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:14:20.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.661+0000 7fa887fff640 10 monclient: dump: 2024-09-16T04:14:20.684 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:14:20.684 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:14:20.684 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:20.684 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:20.684 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:20.684 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:20.685 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:14:20.685 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:14:20.685 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:14:20.685 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:20.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.661+0000 7fa8a5eba640 10 client.1104156.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:14:20.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.661+0000 7fa8a5eba640 10 monclient: _renew_subs 2024-09-16T04:14:20.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.661+0000 7fa8a5eba640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:14:20.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.661+0000 7fa89dffb640 4 set_mon_vals no callback set 2024-09-16T04:14:20.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.661+0000 7fa8a5eba640 1 -- 172.21.15.42:0/808387492 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8a009cbd0 con 0x7fa8a00a1330 2024-09-16T04:14:20.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.661+0000 7fa8a5eba640 1 librados: init done 2024-09-16T04:14:20.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.661+0000 7fa8a5eba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:14:20.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.661+0000 7fa8a5eba640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:14:20.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.661+0000 7fa8a5eba640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:14:20.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.661+0000 7fa8a5eba640 1 -- 172.21.15.42:0/808387492 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa8a00a1710 con 0x7fa8a00a1330 2024-09-16T04:14:20.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.661+0000 7fa887fff640 1 -- 172.21.15.42:0/808387492 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fa890007400 con 0x7fa8a00a1330 2024-09-16T04:14:20.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.661+0000 7fa887fff640 1 --2- 172.21.15.42:0/808387492 >> v2:172.21.15.154:6812/254845342 conn(0x7fa88c037810 0x7fa88c039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:20.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.661+0000 7fa887fff640 1 -- 172.21.15.42:0/808387492 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(710..710 src has 681..710) ==== 4681+0+0 (secure 0 0 0) 0x7fa890048460 con 0x7fa8a00a1330 2024-09-16T04:14:20.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.661+0000 7fa887fff640 10 client.1104156.objecter ms_dispatch 0x7fa8a0000ba0 osd_map(710..710 src has 681..710) 2024-09-16T04:14:20.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.661+0000 7fa887fff640 3 client.1104156.objecter handle_osd_map got epochs [710,710] > 0 2024-09-16T04:14:20.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.661+0000 7fa887fff640 3 client.1104156.objecter handle_osd_map decoding full epoch 710 2024-09-16T04:14:20.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.661+0000 7fa887fff640 20 client.1104156.objecter dump_active .. 0 homeless 2024-09-16T04:14:20.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.665+0000 7fa89effd640 1 --2- 172.21.15.42:0/808387492 >> v2:172.21.15.154:6812/254845342 conn(0x7fa88c037810 0x7fa88c039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:20.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.665+0000 7fa89effd640 10 monclient: get_auth_request con 0x7fa88c037810 auth_method 0 2024-09-16T04:14:20.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.665+0000 7fa887fff640 10 client.1104156.objecter ms_handle_connect 0x7fa88c037810 2024-09-16T04:14:20.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.665+0000 7fa89effd640 1 --2- 172.21.15.42:0/808387492 >> v2:172.21.15.154:6812/254845342 conn(0x7fa88c037810 0x7fa88c039cd0 secure :-1 s=READY pgs=4012 cs=0 l=1 rev1=1 crypto rx=0x7fa8940ad000 tx=0x7fa894008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:14:20.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.669+0000 7fa887fff640 1 -- 172.21.15.42:0/808387492 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7fa890020020 con 0x7fa8a00a1330 2024-09-16T04:14:20.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.669+0000 7fa887fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:14:20.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.669+0000 7fa887fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:14:20.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.825+0000 7fa8a5eba640 1 -- 172.21.15.42:0/808387492 --> v2:172.21.15.154:6812/254845342 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fa8a011e600 con 0x7fa88c037810 2024-09-16T04:14:20.842 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.837+0000 7fa887fff640 1 -- 172.21.15.42:0/808387492 <== mgr.4102 v2:172.21.15.154:6812/254845342 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+114174 (secure 0 0 0) 0x7fa8a011e600 con 0x7fa88c037810 2024-09-16T04:14:20.843 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-09-16T04:14:20.844 INFO:teuthology.orchestra.run.smithi042.stderr:dumped all 2024-09-16T04:14:20.845 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.837+0000 7fa8a5eba640 10 librados: watch_flush enter 2024-09-16T04:14:20.845 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.837+0000 7fa8a5eba640 10 librados: watch_flush exit 2024-09-16T04:14:20.845 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.837+0000 7fa8a5eba640 20 client.1104156.objecter shutdown clearing up homeless session... 2024-09-16T04:14:20.845 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.837+0000 7fa8a5eba640 10 client.1104156.objecter successfully canceled tick 2024-09-16T04:14:20.845 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.837+0000 7fa8a5eba640 1 -- 172.21.15.42:0/808387492 >> v2:172.21.15.154:6812/254845342 conn(0x7fa88c037810 msgr2=0x7fa88c039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:20.845 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.837+0000 7fa8a5eba640 1 --2- 172.21.15.42:0/808387492 >> v2:172.21.15.154:6812/254845342 conn(0x7fa88c037810 0x7fa88c039cd0 secure :-1 s=READY pgs=4012 cs=0 l=1 rev1=1 crypto rx=0x7fa8940ad000 tx=0x7fa894008040 comp rx=0 tx=0).stop 2024-09-16T04:14:20.845 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.837+0000 7fa8a5eba640 10 monclient: shutdown 2024-09-16T04:14:20.845 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.837+0000 7fa8a5eba640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:14:20.845 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.837+0000 7fa8a5eba640 1 -- 172.21.15.42:0/808387492 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa8a00a1330 msgr2=0x7fa8a01cedb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:20.846 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.837+0000 7fa8a5eba640 1 --2- 172.21.15.42:0/808387492 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa8a00a1330 0x7fa8a01cedb0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fa89000c450 tx=0x7fa89000c910 comp rx=0 tx=0).stop 2024-09-16T04:14:20.846 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.837+0000 7fa8a5eba640 1 -- 172.21.15.42:0/808387492 shutdown_connections 2024-09-16T04:14:20.846 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.837+0000 7fa8a5eba640 1 --2- 172.21.15.42:0/808387492 >> v2:172.21.15.154:6812/254845342 conn(0x7fa88c037810 0x7fa88c039cd0 unknown :-1 s=CLOSED pgs=4012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:20.846 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.837+0000 7fa8a5eba640 1 --2- 172.21.15.42:0/808387492 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fa8a0133c50 0x7fa8a011c1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:20.846 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.837+0000 7fa8a5eba640 1 --2- 172.21.15.42:0/808387492 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fa8a00a1e20 0x7fa8a0133710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:20.846 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.837+0000 7fa8a5eba640 1 --2- 172.21.15.42:0/808387492 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fa8a00a1330 0x7fa8a01cedb0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:20.846 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.837+0000 7fa8a5eba640 1 -- 172.21.15.42:0/808387492 >> 172.21.15.42:0/808387492 conn(0x7fa8a00a2420 msgr2=0x7fa8a00a45b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:14:20.846 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.841+0000 7fa8a5eba640 1 -- 172.21.15.42:0/808387492 shutdown_connections 2024-09-16T04:14:20.846 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.841+0000 7fa8a5eba640 1 -- 172.21.15.42:0/808387492 wait complete. 2024-09-16T04:14:20.846 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.841+0000 7fa8a5eba640 1 librados: shutdown 2024-09-16T04:14:20.865 INFO:teuthology.orchestra.run.smithi042.stdout:{"pg_ready":true,"pg_map":{"version":1653,"stamp":"2024-09-16T04:14:20.231345+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590392,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":122,"num_read_kb":106,"num_write":168,"num_write_kb":2425,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":127,"ondisk_log_size":127,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":431992,"kb_used_data":22680,"kb_used_omap":9,"kb_used_meta":407734,"kb_avail":565799048,"statfs":{"total":579820584960,"available":579378225152,"internally_reserved":0,"allocated":23224320,"data_stored":10083938,"data_compressed":18838,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":9567,"internal_metadata":417520289},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003822"},"pg_stats":[{"pgid":"31.f","version":"0'0","reported_seq":1350,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:53.282605+0000","last_change":"2024-09-16T03:43:34.400720+0000","last_active":"2024-09-16T04:13:53.282605+0000","last_peered":"2024-09-16T04:13:53.282605+0000","last_clean":"2024-09-16T04:13:53.282605+0000","last_became_active":"2024-09-16T03:43:34.399531+0000","last_became_peered":"2024-09-16T03:43:34.399531+0000","last_unstale":"2024-09-16T04:13:53.282605+0000","last_undegraded":"2024-09-16T04:13:53.282605+0000","last_fullsized":"2024-09-16T04:13:53.282605+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_clean_scrub_stamp":"2024-09-16T03:43:33.221564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:29:59.415591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.e","version":"0'0","reported_seq":1351,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:53.356452+0000","last_change":"2024-09-16T03:43:34.492536+0000","last_active":"2024-09-16T04:13:53.356452+0000","last_peered":"2024-09-16T04:13:53.356452+0000","last_clean":"2024-09-16T04:13:53.356452+0000","last_became_active":"2024-09-16T03:43:34.492249+0000","last_became_peered":"2024-09-16T03:43:34.492249+0000","last_unstale":"2024-09-16T04:13:53.356452+0000","last_undegraded":"2024-09-16T04:13:53.356452+0000","last_fullsized":"2024-09-16T04:13:53.356452+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_clean_scrub_stamp":"2024-09-16T03:43:33.221564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:15:41.168066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.d","version":"0'0","reported_seq":1373,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:51.310003+0000","last_change":"2024-09-16T03:43:34.414280+0000","last_active":"2024-09-16T04:13:51.310003+0000","last_peered":"2024-09-16T04:13:51.310003+0000","last_clean":"2024-09-16T04:13:51.310003+0000","last_became_active":"2024-09-16T03:43:34.399055+0000","last_became_peered":"2024-09-16T03:43:34.399055+0000","last_unstale":"2024-09-16T04:13:51.310003+0000","last_undegraded":"2024-09-16T04:13:51.310003+0000","last_fullsized":"2024-09-16T04:13:51.310003+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_clean_scrub_stamp":"2024-09-16T03:43:33.221564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:08:19.292854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.c","version":"0'0","reported_seq":1353,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:52.857977+0000","last_change":"2024-09-16T03:43:34.403649+0000","last_active":"2024-09-16T04:13:52.857977+0000","last_peered":"2024-09-16T04:13:52.857977+0000","last_clean":"2024-09-16T04:13:52.857977+0000","last_became_active":"2024-09-16T03:43:34.403187+0000","last_became_peered":"2024-09-16T03:43:34.403187+0000","last_unstale":"2024-09-16T04:13:52.857977+0000","last_undegraded":"2024-09-16T04:13:52.857977+0000","last_fullsized":"2024-09-16T04:13:52.857977+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_clean_scrub_stamp":"2024-09-16T03:43:33.221564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:21:54.130294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.13","version":"0'0","reported_seq":1354,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:51.300256+0000","last_change":"2024-09-16T03:43:34.382675+0000","last_active":"2024-09-16T04:13:51.300256+0000","last_peered":"2024-09-16T04:13:51.300256+0000","last_clean":"2024-09-16T04:13:51.300256+0000","last_became_active":"2024-09-16T03:43:34.382422+0000","last_became_peered":"2024-09-16T03:43:34.382422+0000","last_unstale":"2024-09-16T04:13:51.300256+0000","last_undegraded":"2024-09-16T04:13:51.300256+0000","last_fullsized":"2024-09-16T04:13:51.300256+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_clean_scrub_stamp":"2024-09-16T03:43:33.221564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:52:35.625157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.12","version":"0'0","reported_seq":1353,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:52.857683+0000","last_change":"2024-09-16T03:43:34.404613+0000","last_active":"2024-09-16T04:13:52.857683+0000","last_peered":"2024-09-16T04:13:52.857683+0000","last_clean":"2024-09-16T04:13:52.857683+0000","last_became_active":"2024-09-16T03:43:34.404299+0000","last_became_peered":"2024-09-16T03:43:34.404299+0000","last_unstale":"2024-09-16T04:13:52.857683+0000","last_undegraded":"2024-09-16T04:13:52.857683+0000","last_fullsized":"2024-09-16T04:13:52.857683+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_clean_scrub_stamp":"2024-09-16T03:43:33.221564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:25:52.933411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.11","version":"0'0","reported_seq":1354,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:51.299816+0000","last_change":"2024-09-16T03:43:34.355045+0000","last_active":"2024-09-16T04:13:51.299816+0000","last_peered":"2024-09-16T04:13:51.299816+0000","last_clean":"2024-09-16T04:13:51.299816+0000","last_became_active":"2024-09-16T03:43:34.354598+0000","last_became_peered":"2024-09-16T03:43:34.354598+0000","last_unstale":"2024-09-16T04:13:51.299816+0000","last_undegraded":"2024-09-16T04:13:51.299816+0000","last_fullsized":"2024-09-16T04:13:51.299816+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_clean_scrub_stamp":"2024-09-16T03:43:33.221564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:40:14.558362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.10","version":"0'0","reported_seq":1351,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:53.353474+0000","last_change":"2024-09-16T03:43:34.436288+0000","last_active":"2024-09-16T04:13:53.353474+0000","last_peered":"2024-09-16T04:13:53.353474+0000","last_clean":"2024-09-16T04:13:53.353474+0000","last_became_active":"2024-09-16T03:43:34.434548+0000","last_became_peered":"2024-09-16T03:43:34.434548+0000","last_unstale":"2024-09-16T04:13:53.353474+0000","last_undegraded":"2024-09-16T04:13:53.353474+0000","last_fullsized":"2024-09-16T04:13:53.353474+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_clean_scrub_stamp":"2024-09-16T03:43:33.221564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:03:15.848714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.17","version":"0'0","reported_seq":1349,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:53.282881+0000","last_change":"2024-09-16T03:43:34.456644+0000","last_active":"2024-09-16T04:13:53.282881+0000","last_peered":"2024-09-16T04:13:53.282881+0000","last_clean":"2024-09-16T04:13:53.282881+0000","last_became_active":"2024-09-16T03:43:34.454859+0000","last_became_peered":"2024-09-16T03:43:34.454859+0000","last_unstale":"2024-09-16T04:13:53.282881+0000","last_undegraded":"2024-09-16T04:13:53.282881+0000","last_fullsized":"2024-09-16T04:13:53.282881+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_clean_scrub_stamp":"2024-09-16T03:43:33.221564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:45:01.348488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.16","version":"0'0","reported_seq":1347,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:53.166394+0000","last_change":"2024-09-16T03:43:34.391035+0000","last_active":"2024-09-16T04:13:53.166394+0000","last_peered":"2024-09-16T04:13:53.166394+0000","last_clean":"2024-09-16T04:13:53.166394+0000","last_became_active":"2024-09-16T03:43:34.387629+0000","last_became_peered":"2024-09-16T03:43:34.387629+0000","last_unstale":"2024-09-16T04:13:53.166394+0000","last_undegraded":"2024-09-16T04:13:53.166394+0000","last_fullsized":"2024-09-16T04:13:53.166394+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_clean_scrub_stamp":"2024-09-16T03:43:33.221564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:11:59.627229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.15","version":"0'0","reported_seq":1352,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:52.859996+0000","last_change":"2024-09-16T03:43:34.387904+0000","last_active":"2024-09-16T04:13:52.859996+0000","last_peered":"2024-09-16T04:13:52.859996+0000","last_clean":"2024-09-16T04:13:52.859996+0000","last_became_active":"2024-09-16T03:43:34.387630+0000","last_became_peered":"2024-09-16T03:43:34.387630+0000","last_unstale":"2024-09-16T04:13:52.859996+0000","last_undegraded":"2024-09-16T04:13:52.859996+0000","last_fullsized":"2024-09-16T04:13:52.859996+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_clean_scrub_stamp":"2024-09-16T03:43:33.221564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:15:37.951806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.14","version":"0'0","reported_seq":1350,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:53.282153+0000","last_change":"2024-09-16T03:43:34.381624+0000","last_active":"2024-09-16T04:13:53.282153+0000","last_peered":"2024-09-16T04:13:53.282153+0000","last_clean":"2024-09-16T04:13:53.282153+0000","last_became_active":"2024-09-16T03:43:34.378385+0000","last_became_peered":"2024-09-16T03:43:34.378385+0000","last_unstale":"2024-09-16T04:13:53.282153+0000","last_undegraded":"2024-09-16T04:13:53.282153+0000","last_fullsized":"2024-09-16T04:13:53.282153+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_clean_scrub_stamp":"2024-09-16T03:43:33.221564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:20:14.916099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"14'117","reported_seq":1523,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:52.858864+0000","last_change":"2024-09-16T03:42:18.799294+0000","last_active":"2024-09-16T04:13:52.858864+0000","last_peered":"2024-09-16T04:13:52.858864+0000","last_clean":"2024-09-16T04:13:52.858864+0000","last_became_active":"2024-09-16T03:42:18.797239+0000","last_became_peered":"2024-09-16T03:42:18.797239+0000","last_unstale":"2024-09-16T04:13:52.858864+0000","last_undegraded":"2024-09-16T04:13:52.858864+0000","last_fullsized":"2024-09-16T04:13:52.858864+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:42:17.544237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:42:17.544237+0000","last_clean_scrub_stamp":"2024-09-16T03:42:17.544237+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:01:21.017593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":1374,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:51.300417+0000","last_change":"2024-09-16T03:42:27.630965+0000","last_active":"2024-09-16T04:13:51.300417+0000","last_peered":"2024-09-16T04:13:51.300417+0000","last_clean":"2024-09-16T04:13:51.300417+0000","last_became_active":"2024-09-16T03:42:25.615930+0000","last_became_peered":"2024-09-16T03:42:25.615930+0000","last_unstale":"2024-09-16T04:13:51.300417+0000","last_undegraded":"2024-09-16T04:13:51.300417+0000","last_fullsized":"2024-09-16T04:13:51.300417+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:42:24.583922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:42:24.583922+0000","last_clean_scrub_stamp":"2024-09-16T03:42:24.583922+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:59:59.653118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00099470900000000009,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.1e","version":"0'0","reported_seq":1355,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:51.302860+0000","last_change":"2024-09-16T03:43:34.401653+0000","last_active":"2024-09-16T04:13:51.302860+0000","last_peered":"2024-09-16T04:13:51.302860+0000","last_clean":"2024-09-16T04:13:51.302860+0000","last_became_active":"2024-09-16T03:43:34.397708+0000","last_became_peered":"2024-09-16T03:43:34.397708+0000","last_unstale":"2024-09-16T04:13:51.302860+0000","last_undegraded":"2024-09-16T04:13:51.302860+0000","last_fullsized":"2024-09-16T04:13:51.302860+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_clean_scrub_stamp":"2024-09-16T03:43:33.221564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:16:17.302952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1372,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:52.859154+0000","last_change":"2024-09-16T03:42:27.624504+0000","last_active":"2024-09-16T04:13:52.859154+0000","last_peered":"2024-09-16T04:13:52.859154+0000","last_clean":"2024-09-16T04:13:52.859154+0000","last_became_active":"2024-09-16T03:42:25.612386+0000","last_became_peered":"2024-09-16T03:42:25.612386+0000","last_unstale":"2024-09-16T04:13:52.859154+0000","last_undegraded":"2024-09-16T04:13:52.859154+0000","last_fullsized":"2024-09-16T04:13:52.859154+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:42:24.583922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:42:24.583922+0000","last_clean_scrub_stamp":"2024-09-16T03:42:24.583922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:00:49.446880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081399800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1d","version":"0'0","reported_seq":1372,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:51.310252+0000","last_change":"2024-09-16T03:43:34.408908+0000","last_active":"2024-09-16T04:13:51.310252+0000","last_peered":"2024-09-16T04:13:51.310252+0000","last_clean":"2024-09-16T04:13:51.310252+0000","last_became_active":"2024-09-16T03:43:34.399300+0000","last_became_peered":"2024-09-16T03:43:34.399300+0000","last_unstale":"2024-09-16T04:13:51.310252+0000","last_undegraded":"2024-09-16T04:13:51.310252+0000","last_fullsized":"2024-09-16T04:13:51.310252+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_clean_scrub_stamp":"2024-09-16T03:43:33.221564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:25:51.517103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1370,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:53.353598+0000","last_change":"2024-09-16T03:42:27.619252+0000","last_active":"2024-09-16T04:13:53.353598+0000","last_peered":"2024-09-16T04:13:53.353598+0000","last_clean":"2024-09-16T04:13:53.353598+0000","last_became_active":"2024-09-16T03:42:25.607533+0000","last_became_peered":"2024-09-16T03:42:25.607533+0000","last_unstale":"2024-09-16T04:13:53.353598+0000","last_undegraded":"2024-09-16T04:13:53.353598+0000","last_fullsized":"2024-09-16T04:13:53.353598+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:42:24.583922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:42:24.583922+0000","last_clean_scrub_stamp":"2024-09-16T03:42:24.583922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:25:58.536144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089083299999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1c","version":"0'0","reported_seq":1349,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:53.282306+0000","last_change":"2024-09-16T03:43:34.421587+0000","last_active":"2024-09-16T04:13:53.282306+0000","last_peered":"2024-09-16T04:13:53.282306+0000","last_clean":"2024-09-16T04:13:53.282306+0000","last_became_active":"2024-09-16T03:43:34.421320+0000","last_became_peered":"2024-09-16T03:43:34.421320+0000","last_unstale":"2024-09-16T04:13:53.282306+0000","last_undegraded":"2024-09-16T04:13:53.282306+0000","last_fullsized":"2024-09-16T04:13:53.282306+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_clean_scrub_stamp":"2024-09-16T03:43:33.221564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:47:25.449200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":1375,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:51.299718+0000","last_change":"2024-09-16T03:42:27.632050+0000","last_active":"2024-09-16T04:13:51.299718+0000","last_peered":"2024-09-16T04:13:51.299718+0000","last_clean":"2024-09-16T04:13:51.299718+0000","last_became_active":"2024-09-16T03:42:25.617896+0000","last_became_peered":"2024-09-16T03:42:25.617896+0000","last_unstale":"2024-09-16T04:13:51.299718+0000","last_undegraded":"2024-09-16T04:13:51.299718+0000","last_fullsized":"2024-09-16T04:13:51.299718+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:42:24.583922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:42:24.583922+0000","last_clean_scrub_stamp":"2024-09-16T03:42:24.583922+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:41:08.754274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021072529999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.1f","version":"0'0","reported_seq":1352,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:53.356655+0000","last_change":"2024-09-16T03:43:34.419004+0000","last_active":"2024-09-16T04:13:53.356655+0000","last_peered":"2024-09-16T04:13:53.356655+0000","last_clean":"2024-09-16T04:13:53.356655+0000","last_became_active":"2024-09-16T03:43:34.415747+0000","last_became_peered":"2024-09-16T03:43:34.415747+0000","last_unstale":"2024-09-16T04:13:53.356655+0000","last_undegraded":"2024-09-16T04:13:53.356655+0000","last_fullsized":"2024-09-16T04:13:53.356655+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_clean_scrub_stamp":"2024-09-16T03:43:33.221564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:01.644098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1367,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:53.166325+0000","last_change":"2024-09-16T03:42:27.631260+0000","last_active":"2024-09-16T04:13:53.166325+0000","last_peered":"2024-09-16T04:13:53.166325+0000","last_clean":"2024-09-16T04:13:53.166325+0000","last_became_active":"2024-09-16T03:42:25.609631+0000","last_became_peered":"2024-09-16T03:42:25.609631+0000","last_unstale":"2024-09-16T04:13:53.166325+0000","last_undegraded":"2024-09-16T04:13:53.166325+0000","last_fullsized":"2024-09-16T04:13:53.166325+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:42:24.583922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:42:24.583922+0000","last_clean_scrub_stamp":"2024-09-16T03:42:24.583922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:58:49.091085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001288897,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.19","version":"0'0","reported_seq":1347,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:53.166637+0000","last_change":"2024-09-16T03:43:34.341660+0000","last_active":"2024-09-16T04:13:53.166637+0000","last_peered":"2024-09-16T04:13:53.166637+0000","last_clean":"2024-09-16T04:13:53.166637+0000","last_became_active":"2024-09-16T03:43:34.332234+0000","last_became_peered":"2024-09-16T03:43:34.332234+0000","last_unstale":"2024-09-16T04:13:53.166637+0000","last_undegraded":"2024-09-16T04:13:53.166637+0000","last_fullsized":"2024-09-16T04:13:53.166637+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_clean_scrub_stamp":"2024-09-16T03:43:33.221564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:20:27.326212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1372,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:52.859726+0000","last_change":"2024-09-16T03:42:27.625091+0000","last_active":"2024-09-16T04:13:52.859726+0000","last_peered":"2024-09-16T04:13:52.859726+0000","last_clean":"2024-09-16T04:13:52.859726+0000","last_became_active":"2024-09-16T03:42:25.614458+0000","last_became_peered":"2024-09-16T03:42:25.614458+0000","last_unstale":"2024-09-16T04:13:52.859726+0000","last_undegraded":"2024-09-16T04:13:52.859726+0000","last_fullsized":"2024-09-16T04:13:52.859726+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:42:24.583922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:42:24.583922+0000","last_clean_scrub_stamp":"2024-09-16T03:42:24.583922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:26:14.681436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012701769999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.18","version":"0'0","reported_seq":1347,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:53.165619+0000","last_change":"2024-09-16T03:43:34.362758+0000","last_active":"2024-09-16T04:13:53.165619+0000","last_peered":"2024-09-16T04:13:53.165619+0000","last_clean":"2024-09-16T04:13:53.165619+0000","last_became_active":"2024-09-16T03:43:34.354705+0000","last_became_peered":"2024-09-16T03:43:34.354705+0000","last_unstale":"2024-09-16T04:13:53.165619+0000","last_undegraded":"2024-09-16T04:13:53.165619+0000","last_fullsized":"2024-09-16T04:13:53.165619+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_clean_scrub_stamp":"2024-09-16T03:43:33.221564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:16:36.345303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1366,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:53.166473+0000","last_change":"2024-09-16T03:42:27.631227+0000","last_active":"2024-09-16T04:13:53.166473+0000","last_peered":"2024-09-16T04:13:53.166473+0000","last_clean":"2024-09-16T04:13:53.166473+0000","last_became_active":"2024-09-16T03:42:25.608453+0000","last_became_peered":"2024-09-16T03:42:25.608453+0000","last_unstale":"2024-09-16T04:13:53.166473+0000","last_undegraded":"2024-09-16T04:13:53.166473+0000","last_fullsized":"2024-09-16T04:13:53.166473+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:42:24.583922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:42:24.583922+0000","last_clean_scrub_stamp":"2024-09-16T03:42:24.583922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:00:11.921493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001163533,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.1b","version":"0'0","reported_seq":1352,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:52.859563+0000","last_change":"2024-09-16T03:43:34.381629+0000","last_active":"2024-09-16T04:13:52.859563+0000","last_peered":"2024-09-16T04:13:52.859563+0000","last_clean":"2024-09-16T04:13:52.859563+0000","last_became_active":"2024-09-16T03:43:34.381335+0000","last_became_peered":"2024-09-16T03:43:34.381335+0000","last_unstale":"2024-09-16T04:13:52.859563+0000","last_undegraded":"2024-09-16T04:13:52.859563+0000","last_fullsized":"2024-09-16T04:13:52.859563+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_clean_scrub_stamp":"2024-09-16T03:43:33.221564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:30:50.930529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1371,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:52.861031+0000","last_change":"2024-09-16T03:42:27.624922+0000","last_active":"2024-09-16T04:13:52.861031+0000","last_peered":"2024-09-16T04:13:52.861031+0000","last_clean":"2024-09-16T04:13:52.861031+0000","last_became_active":"2024-09-16T03:42:25.615293+0000","last_became_peered":"2024-09-16T03:42:25.615293+0000","last_unstale":"2024-09-16T04:13:52.861031+0000","last_undegraded":"2024-09-16T04:13:52.861031+0000","last_fullsized":"2024-09-16T04:13:52.861031+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:42:24.583922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:42:24.583922+0000","last_clean_scrub_stamp":"2024-09-16T03:42:24.583922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:46:11.677251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010681690000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1a","version":"0'0","reported_seq":1349,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:53.281860+0000","last_change":"2024-09-16T03:43:34.382293+0000","last_active":"2024-09-16T04:13:53.281860+0000","last_peered":"2024-09-16T04:13:53.281860+0000","last_clean":"2024-09-16T04:13:53.281860+0000","last_became_active":"2024-09-16T03:43:34.372307+0000","last_became_peered":"2024-09-16T03:43:34.372307+0000","last_unstale":"2024-09-16T04:13:53.281860+0000","last_undegraded":"2024-09-16T04:13:53.281860+0000","last_fullsized":"2024-09-16T04:13:53.281860+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_clean_scrub_stamp":"2024-09-16T03:43:33.221564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:49:25.481308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.8","version":"0'0","reported_seq":1351,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:53.280246+0000","last_change":"2024-09-16T03:43:34.386011+0000","last_active":"2024-09-16T04:13:53.280246+0000","last_peered":"2024-09-16T04:13:53.280246+0000","last_clean":"2024-09-16T04:13:53.280246+0000","last_became_active":"2024-09-16T03:43:34.378483+0000","last_became_peered":"2024-09-16T03:43:34.378483+0000","last_unstale":"2024-09-16T04:13:53.280246+0000","last_undegraded":"2024-09-16T04:13:53.280246+0000","last_fullsized":"2024-09-16T04:13:53.280246+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_clean_scrub_stamp":"2024-09-16T03:43:33.221564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:25:42.568633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.9","version":"0'0","reported_seq":1374,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:51.306525+0000","last_change":"2024-09-16T03:43:34.389268+0000","last_active":"2024-09-16T04:13:51.306525+0000","last_peered":"2024-09-16T04:13:51.306525+0000","last_clean":"2024-09-16T04:13:51.306525+0000","last_became_active":"2024-09-16T03:43:34.378733+0000","last_became_peered":"2024-09-16T03:43:34.378733+0000","last_unstale":"2024-09-16T04:13:51.306525+0000","last_undegraded":"2024-09-16T04:13:51.306525+0000","last_fullsized":"2024-09-16T04:13:51.306525+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_clean_scrub_stamp":"2024-09-16T03:43:33.221564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:16:08.231273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.a","version":"0'0","reported_seq":1374,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:51.306820+0000","last_change":"2024-09-16T03:43:34.390824+0000","last_active":"2024-09-16T04:13:51.306820+0000","last_peered":"2024-09-16T04:13:51.306820+0000","last_clean":"2024-09-16T04:13:51.306820+0000","last_became_active":"2024-09-16T03:43:34.390049+0000","last_became_peered":"2024-09-16T03:43:34.390049+0000","last_unstale":"2024-09-16T04:13:51.306820+0000","last_undegraded":"2024-09-16T04:13:51.306820+0000","last_fullsized":"2024-09-16T04:13:51.306820+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_clean_scrub_stamp":"2024-09-16T03:43:33.221564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:48:31.039881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.b","version":"0'0","reported_seq":1348,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:53.165918+0000","last_change":"2024-09-16T03:43:34.332247+0000","last_active":"2024-09-16T04:13:53.165918+0000","last_peered":"2024-09-16T04:13:53.165918+0000","last_clean":"2024-09-16T04:13:53.165918+0000","last_became_active":"2024-09-16T03:43:34.328425+0000","last_became_peered":"2024-09-16T03:43:34.328425+0000","last_unstale":"2024-09-16T04:13:53.165918+0000","last_undegraded":"2024-09-16T04:13:53.165918+0000","last_fullsized":"2024-09-16T04:13:53.165918+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_clean_scrub_stamp":"2024-09-16T03:43:33.221564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:38:08.042571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.4","version":"0'0","reported_seq":1351,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:53.282653+0000","last_change":"2024-09-16T03:43:34.392184+0000","last_active":"2024-09-16T04:13:53.282653+0000","last_peered":"2024-09-16T04:13:53.282653+0000","last_clean":"2024-09-16T04:13:53.282653+0000","last_became_active":"2024-09-16T03:43:34.391868+0000","last_became_peered":"2024-09-16T03:43:34.391868+0000","last_unstale":"2024-09-16T04:13:53.282653+0000","last_undegraded":"2024-09-16T04:13:53.282653+0000","last_fullsized":"2024-09-16T04:13:53.282653+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_clean_scrub_stamp":"2024-09-16T03:43:33.221564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:02:40.567145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.5","version":"0'0","reported_seq":1351,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:53.280846+0000","last_change":"2024-09-16T03:43:34.419668+0000","last_active":"2024-09-16T04:13:53.280846+0000","last_peered":"2024-09-16T04:13:53.280846+0000","last_clean":"2024-09-16T04:13:53.280846+0000","last_became_active":"2024-09-16T03:43:34.419410+0000","last_became_peered":"2024-09-16T03:43:34.419410+0000","last_unstale":"2024-09-16T04:13:53.280846+0000","last_undegraded":"2024-09-16T04:13:53.280846+0000","last_fullsized":"2024-09-16T04:13:53.280846+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_clean_scrub_stamp":"2024-09-16T03:43:33.221564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:37:43.007953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.6","version":"0'0","reported_seq":1374,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:51.309271+0000","last_change":"2024-09-16T03:43:34.386607+0000","last_active":"2024-09-16T04:13:51.309271+0000","last_peered":"2024-09-16T04:13:51.309271+0000","last_clean":"2024-09-16T04:13:51.309271+0000","last_became_active":"2024-09-16T03:43:34.381834+0000","last_became_peered":"2024-09-16T03:43:34.381834+0000","last_unstale":"2024-09-16T04:13:51.309271+0000","last_undegraded":"2024-09-16T04:13:51.309271+0000","last_fullsized":"2024-09-16T04:13:51.309271+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_clean_scrub_stamp":"2024-09-16T03:43:33.221564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:06:41.696398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.7","version":"0'0","reported_seq":1349,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:53.164974+0000","last_change":"2024-09-16T03:43:34.296793+0000","last_active":"2024-09-16T04:13:53.164974+0000","last_peered":"2024-09-16T04:13:53.164974+0000","last_clean":"2024-09-16T04:13:53.164974+0000","last_became_active":"2024-09-16T03:43:34.292756+0000","last_became_peered":"2024-09-16T03:43:34.292756+0000","last_unstale":"2024-09-16T04:13:53.164974+0000","last_undegraded":"2024-09-16T04:13:53.164974+0000","last_fullsized":"2024-09-16T04:13:53.164974+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_clean_scrub_stamp":"2024-09-16T03:43:33.221564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:12:04.533479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.0","version":"0'0","reported_seq":1355,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:52.859564+0000","last_change":"2024-09-16T03:43:34.352792+0000","last_active":"2024-09-16T04:13:52.859564+0000","last_peered":"2024-09-16T04:13:52.859564+0000","last_clean":"2024-09-16T04:13:52.859564+0000","last_became_active":"2024-09-16T03:43:34.351273+0000","last_became_peered":"2024-09-16T03:43:34.351273+0000","last_unstale":"2024-09-16T04:13:52.859564+0000","last_undegraded":"2024-09-16T04:13:52.859564+0000","last_fullsized":"2024-09-16T04:13:52.859564+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_clean_scrub_stamp":"2024-09-16T03:43:33.221564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:50:21.469378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1","version":"22'7","reported_seq":1362,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:53.164484+0000","last_change":"2024-09-16T03:43:34.397561+0000","last_active":"2024-09-16T04:13:53.164484+0000","last_peered":"2024-09-16T04:13:53.164484+0000","last_clean":"2024-09-16T04:13:53.164484+0000","last_became_active":"2024-09-16T03:43:34.382744+0000","last_became_peered":"2024-09-16T03:43:34.382744+0000","last_unstale":"2024-09-16T04:13:53.164484+0000","last_undegraded":"2024-09-16T04:13:53.164484+0000","last_fullsized":"2024-09-16T04:13:53.164484+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_clean_scrub_stamp":"2024-09-16T03:43:33.221564+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:54:00.268175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.2","version":"0'0","reported_seq":1355,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:51.300709+0000","last_change":"2024-09-16T03:43:34.402175+0000","last_active":"2024-09-16T04:13:51.300709+0000","last_peered":"2024-09-16T04:13:51.300709+0000","last_clean":"2024-09-16T04:13:51.300709+0000","last_became_active":"2024-09-16T03:43:34.402014+0000","last_became_peered":"2024-09-16T03:43:34.402014+0000","last_unstale":"2024-09-16T04:13:51.300709+0000","last_undegraded":"2024-09-16T04:13:51.300709+0000","last_fullsized":"2024-09-16T04:13:51.300709+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_clean_scrub_stamp":"2024-09-16T03:43:33.221564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:46:39.366181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.3","version":"0'0","reported_seq":1374,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:51.306977+0000","last_change":"2024-09-16T03:43:34.380947+0000","last_active":"2024-09-16T04:13:51.306977+0000","last_peered":"2024-09-16T04:13:51.306977+0000","last_clean":"2024-09-16T04:13:51.306977+0000","last_became_active":"2024-09-16T03:43:34.379988+0000","last_became_peered":"2024-09-16T03:43:34.379988+0000","last_unstale":"2024-09-16T04:13:51.306977+0000","last_undegraded":"2024-09-16T04:13:51.306977+0000","last_fullsized":"2024-09-16T04:13:51.306977+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_clean_scrub_stamp":"2024-09-16T03:43:33.221564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:21:15.646033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":31,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":18838,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":12,"seq":51539607943,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":71596,"kb_used_data":3764,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94300244,"statfs":{"total":96636764160,"available":96563449856,"internally_reserved":0,"allocated":3854336,"data_stored":1504358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":11,"seq":47244640646,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":71228,"kb_used_data":3764,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94300612,"statfs":{"total":96636764160,"available":96563826688,"internally_reserved":0,"allocated":3854336,"data_stored":1504344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640643,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":67116,"kb_used_data":3812,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94304724,"statfs":{"total":96636764160,"available":96568037376,"internally_reserved":0,"allocated":3903488,"data_stored":2033267,"data_compressed":9419,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1593,"internal_metadata":64616903},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640644,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":71224,"kb_used_data":3760,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94300616,"statfs":{"total":96636764160,"available":96563830784,"internally_reserved":0,"allocated":3850240,"data_stored":1504339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":68876742},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640645,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":79552,"kb_used_data":3768,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94292288,"statfs":{"total":96636764160,"available":96555302912,"internally_reserved":0,"allocated":3858432,"data_stored":1504363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640644,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":71276,"kb_used_data":3812,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94300564,"statfs":{"total":96636764160,"available":96563777536,"internally_reserved":0,"allocated":3903488,"data_stored":2033267,"data_compressed":9419,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":9419,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":9419,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:14:20.867 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-16T04:14:20.867 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:14:20.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.981+0000 7faede77c640 10 monclient: get_monmap_and_config 2024-09-16T04:14:20.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.981+0000 7faede77c640 10 monclient: build_initial_monmap 2024-09-16T04:14:20.988 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.981+0000 7faede77c640 10 monclient: monmap: 2024-09-16T04:14:20.988 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:14:20.989 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:14:20.989 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:14:20.987086+0000 2024-09-16T04:14:20.989 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:14:20.987086+0000 2024-09-16T04:14:20.989 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:14:20.989 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:20.989 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:14:20.989 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:14:20.989 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:14:20.989 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:14:20.989 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:14:20.989 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:20.992 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.981+0000 7faede77c640 1 Processor -- start 2024-09-16T04:14:20.992 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.981+0000 7faede77c640 1 -- start start 2024-09-16T04:14:20.992 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.981+0000 7faede77c640 10 monclient: init 2024-09-16T04:14:20.992 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.985+0000 7faede77c640 10 monclient: _reopen_session rank -1 2024-09-16T04:14:20.992 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.985+0000 7faede77c640 10 monclient: _add_conns ranks=[2,0,4,3,1] 2024-09-16T04:14:20.992 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.985+0000 7faede77c640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7faed80a1330 0x7faed809f180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:20.992 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.985+0000 7faede77c640 10 monclient(hunting): picked mon.noname-b con 0x7faed80a1330 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:14:20.992 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.985+0000 7faede77c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7faed80a1710 addr v1:172.21.15.42:6789/0 2024-09-16T04:14:20.992 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.985+0000 7faede77c640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7faed81393a0 0x7faed813b790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:20.992 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.985+0000 7faede77c640 10 monclient(hunting): picked mon.noname-c con 0x7faed81393a0 addr v2:172.21.15.154:3300/0 2024-09-16T04:14:20.992 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.985+0000 7faede77c640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faed809e4c0 con 0x7faed80a1710 2024-09-16T04:14:20.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.985+0000 7faede77c640 10 monclient(hunting): start opening mon connection 2024-09-16T04:14:20.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.985+0000 7faede77c640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7faed809edf0 con 0x7faed81393a0 2024-09-16T04:14:20.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.985+0000 7faede77c640 10 monclient(hunting): start opening mon connection 2024-09-16T04:14:20.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.985+0000 7faede77c640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7faed809cd60 con 0x7faed80a1330 2024-09-16T04:14:20.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.985+0000 7faede77c640 10 monclient(hunting): _renew_subs 2024-09-16T04:14:20.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.985+0000 7faede77c640 10 monclient(hunting): authenticate will time out at 2907.956523s 2024-09-16T04:14:20.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.985+0000 7faed7fff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7faed80a1330 0x7faed809f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:20.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.985+0000 7faed77fe640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7faed80a1710 0x7faed80a1b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:52980/0 (socket says 172.21.15.42:52980) 2024-09-16T04:14:20.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.985+0000 7faed77fe640 1 -- 172.21.15.42:0/221737642 learned_addr learned my addr 172.21.15.42:0/221737642 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T04:14:20.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.985+0000 7faed7fff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7faed80a1330 0x7faed809f180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:51892/0 (socket says 172.21.15.42:51892) 2024-09-16T04:14:20.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.985+0000 7faed7fff640 10 monclient(hunting): get_auth_request con 0x7faed80a1330 auth_method 0 2024-09-16T04:14:20.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.985+0000 7faed7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:14:20.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.985+0000 7faed7fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:20.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.985+0000 7faed7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:20.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.985+0000 7faed7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:14:20.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.985+0000 7faed7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:14:20.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.985+0000 7faed7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:14:20.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.985+0000 7faed6ffd640 1 -- 172.21.15.42:0/221737642 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4035403834 0 0) 0x7faed809e4c0 con 0x7faed80a1710 2024-09-16T04:14:20.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.985+0000 7faed6ffd640 10 monclient(hunting): _negotiate 2024-09-16T04:14:20.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.985+0000 7faed6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:20.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.985+0000 7faed6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:20.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.985+0000 7faed6ffd640 10 monclient(hunting): my global_id is 1104159 2024-09-16T04:14:20.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.985+0000 7faedcd19640 1 --2- 172.21.15.42:0/221737642 >> v2:172.21.15.154:3300/0 conn(0x7faed81393a0 0x7faed813b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:20.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.985+0000 7faed6ffd640 1 -- 172.21.15.42:0/221737642 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faec0003530 con 0x7faed80a1710 2024-09-16T04:14:20.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.985+0000 7faed7fff640 10 monclient(hunting): handle_auth_done global_id 1094237 payload 274 2024-09-16T04:14:20.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.985+0000 7faed7fff640 1 -- 172.21.15.42:0/221737642 >> v2:172.21.15.154:3300/0 conn(0x7faed81393a0 msgr2=0x7faed813b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:14:20.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.985+0000 7faed7fff640 1 --2- 172.21.15.42:0/221737642 >> v2:172.21.15.154:3300/0 conn(0x7faed81393a0 0x7faed813b790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:20.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.985+0000 7faed7fff640 1 -- 172.21.15.42:0/221737642 >> v1:172.21.15.42:6789/0 conn(0x7faed80a1710 legacy=0x7faed80a1b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:20.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.985+0000 7faed7fff640 10 monclient: _finish_hunting 0 2024-09-16T04:14:20.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.985+0000 7faed7fff640 1 monclient: found mon.noname-b 2024-09-16T04:14:20.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.985+0000 7faed7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:14:20.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.985+0000 7faed7fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T04:14:20.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.985+0000 7faed7fff640 1 -- 172.21.15.42:0/221737642 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faed809f6c0 con 0x7faed80a1330 2024-09-16T04:14:20.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.985+0000 7faedcd19640 10 monclient: get_auth_request con 0x7faed81393a0 auth_method 0 2024-09-16T04:14:20.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.985+0000 7faedcd19640 1 --2- 172.21.15.42:0/221737642 >> v2:172.21.15.154:3300/0 conn(0x7faed81393a0 0x7faed813b790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T04:14:20.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.985+0000 7faed7fff640 1 --2- 172.21.15.42:0/221737642 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7faed80a1330 0x7faed809f180 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7faec80097c0 tx=0x7faec801c360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d3bb15ddf04ae66 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:14:21.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.985+0000 7faed6ffd640 1 -- 172.21.15.42:0/221737642 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faec80b0070 con 0x7faed80a1330 2024-09-16T04:14:21.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.985+0000 7faed6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:21.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.985+0000 7faed6ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:14:21.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.985+0000 7faed6ffd640 10 monclient: dump: 2024-09-16T04:14:21.001 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:14:21.001 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:14:21.001 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:21.001 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:21.001 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:21.001 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:21.001 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:14:21.001 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:14:21.001 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:14:21.001 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:21.001 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.985+0000 7faed6ffd640 10 monclient: _finish_auth 0 2024-09-16T04:14:21.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.985+0000 7faed6ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:14:21.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.985+0000 7faed6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:14:21.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.985+0000 7faede77c640 5 monclient: authenticate success, global_id 1094237 2024-09-16T04:14:21.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.985+0000 7faede77c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:14:21.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.985+0000 7faed6ffd640 1 -- 172.21.15.42:0/221737642 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faec80ab600 con 0x7faed80a1330 2024-09-16T04:14:21.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.985+0000 7faed6ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:14:21.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.985+0000 7faede77c640 10 monclient: get_monmap_and_config success 2024-09-16T04:14:21.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.985+0000 7faede77c640 4 set_mon_vals no callback set 2024-09-16T04:14:21.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.985+0000 7faed6ffd640 1 -- 172.21.15.42:0/221737642 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faec80ab9b0 con 0x7faed80a1330 2024-09-16T04:14:21.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.985+0000 7faed6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:21.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.985+0000 7faed6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:14:21.002 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.985+0000 7faed6ffd640 10 monclient: dump: 2024-09-16T04:14:21.002 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:14:21.002 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:14:21.003 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:21.003 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:21.003 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:21.003 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:21.003 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:14:21.003 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:14:21.003 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:14:21.003 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:21.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.985+0000 7faede77c640 10 monclient: shutdown 2024-09-16T04:14:21.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.985+0000 7faede77c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:14:21.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.985+0000 7faede77c640 1 -- 172.21.15.42:0/221737642 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7faed80a1330 msgr2=0x7faed809f180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:21.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.985+0000 7faede77c640 1 --2- 172.21.15.42:0/221737642 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7faed80a1330 0x7faed809f180 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7faec80097c0 tx=0x7faec801c360 comp rx=0 tx=0).stop 2024-09-16T04:14:21.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.989+0000 7faede77c640 1 -- 172.21.15.42:0/221737642 shutdown_connections 2024-09-16T04:14:21.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.989+0000 7faede77c640 1 --2- 172.21.15.42:0/221737642 >> v2:172.21.15.154:3300/0 conn(0x7faed81393a0 0x7faed813b790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:21.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.989+0000 7faede77c640 1 --2- 172.21.15.42:0/221737642 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7faed80a1330 0x7faed809f180 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:21.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.989+0000 7faede77c640 1 -- 172.21.15.42:0/221737642 >> 172.21.15.42:0/221737642 conn(0x7faed80a2420 msgr2=0x7faed80a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:14:21.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.989+0000 7faede77c640 1 -- 172.21.15.42:0/221737642 shutdown_connections 2024-09-16T04:14:21.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.989+0000 7faede77c640 1 -- 172.21.15.42:0/221737642 wait complete. 2024-09-16T04:14:21.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.989+0000 7faede77c640 10 monclient: build_initial_monmap 2024-09-16T04:14:21.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.989+0000 7faede77c640 10 monclient: monmap: 2024-09-16T04:14:21.004 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:14:21.004 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:14:21.004 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:14:21.004 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:14:21.004 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:14:21.004 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:21.004 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:14:21.005 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:14:21.005 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:14:21.005 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:21.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.989+0000 7faede77c640 1 librados: starting msgr at 2024-09-16T04:14:21.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.989+0000 7faede77c640 1 librados: starting objecter 2024-09-16T04:14:21.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.989+0000 7faede77c640 1 Processor -- start 2024-09-16T04:14:21.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.989+0000 7faede77c640 1 -- start start 2024-09-16T04:14:21.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.989+0000 7faede77c640 1 librados: setting wanted keys 2024-09-16T04:14:21.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.989+0000 7faede77c640 1 librados: calling monclient init 2024-09-16T04:14:21.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.989+0000 7faede77c640 10 monclient: init 2024-09-16T04:14:21.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.989+0000 7faede77c640 10 monclient: _reopen_session rank -1 2024-09-16T04:14:21.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.989+0000 7faede77c640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:14:21.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.989+0000 7faede77c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faed80a1330 0x7faed81ccb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:21.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.989+0000 7faede77c640 10 monclient(hunting): picked mon.noname-b con 0x7faed80a1330 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:14:21.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.989+0000 7faede77c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faed80a1710 0x7faed8132230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:21.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.989+0000 7faede77c640 10 monclient(hunting): picked mon.noname-a con 0x7faed80a1710 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:14:21.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.989+0000 7faede77c640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7faed81393a0 0x7faed80706a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:21.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.989+0000 7faede77c640 10 monclient(hunting): picked mon.noname-c con 0x7faed81393a0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:14:21.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.989+0000 7faede77c640 10 monclient(hunting): start opening mon connection 2024-09-16T04:14:21.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.989+0000 7faede77c640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7faed813e880 con 0x7faed80a1710 2024-09-16T04:14:21.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.989+0000 7faede77c640 10 monclient(hunting): start opening mon connection 2024-09-16T04:14:21.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.989+0000 7faede77c640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7faed813e700 con 0x7faed80a1330 2024-09-16T04:14:21.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.989+0000 7faede77c640 10 monclient(hunting): start opening mon connection 2024-09-16T04:14:21.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.989+0000 7faede77c640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7faed813ea00 con 0x7faed81393a0 2024-09-16T04:14:21.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.989+0000 7faede77c640 10 monclient(hunting): _renew_subs 2024-09-16T04:14:21.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.989+0000 7faede77c640 10 monclient(hunting): authenticate will time out at 2907.961536s 2024-09-16T04:14:21.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.989+0000 7faedcd19640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7faed81393a0 0x7faed80706a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:21.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.989+0000 7faed77fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faed80a1710 0x7faed8132230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:21.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.989+0000 7faedcd19640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7faed81393a0 0x7faed80706a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:51898/0 (socket says 172.21.15.42:51898) 2024-09-16T04:14:21.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.989+0000 7faedcd19640 1 -- 172.21.15.42:0/2217608656 learned_addr learned my addr 172.21.15.42:0/2217608656 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:14:21.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.989+0000 7faedcd19640 10 monclient(hunting): get_auth_request con 0x7faed81393a0 auth_method 0 2024-09-16T04:14:21.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.989+0000 7faed77fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faed80a1710 0x7faed8132230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:47956/0 (socket says 172.21.15.42:47956) 2024-09-16T04:14:21.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.989+0000 7faedcd19640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:14:21.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.989+0000 7faedcd19640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:21.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.989+0000 7faedcd19640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:21.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.989+0000 7faed77fe640 10 monclient(hunting): get_auth_request con 0x7faed80a1710 auth_method 0 2024-09-16T04:14:21.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.989+0000 7faed77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:14:21.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.989+0000 7faed77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:21.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.989+0000 7faed77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:21.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.989+0000 7faedcd19640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:14:21.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.989+0000 7faedcd19640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:14:21.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.989+0000 7faedcd19640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:14:21.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.989+0000 7faed77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:14:21.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.989+0000 7faed77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:14:21.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.989+0000 7faed77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:14:21.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.989+0000 7faed7fff640 1 --2- 172.21.15.42:0/2217608656 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faed80a1330 0x7faed81ccb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:21.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.989+0000 7faedcd19640 10 monclient(hunting): handle_auth_done global_id 1094240 payload 824 2024-09-16T04:14:21.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.989+0000 7faedcd19640 1 -- 172.21.15.42:0/2217608656 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faed80a1330 msgr2=0x7faed81ccb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:21.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.989+0000 7faedcd19640 1 --2- 172.21.15.42:0/2217608656 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faed80a1330 0x7faed81ccb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:21.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.989+0000 7faedcd19640 1 -- 172.21.15.42:0/2217608656 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faed80a1710 msgr2=0x7faed8132230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:21.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.989+0000 7faedcd19640 1 --2- 172.21.15.42:0/2217608656 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faed80a1710 0x7faed8132230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:21.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.989+0000 7faedcd19640 10 monclient: _finish_hunting 0 2024-09-16T04:14:21.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.989+0000 7faedcd19640 1 monclient: found mon.noname-c 2024-09-16T04:14:21.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.989+0000 7faedcd19640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:14:21.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.989+0000 7faedcd19640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T04:14:21.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.989+0000 7faedcd19640 1 -- 172.21.15.42:0/2217608656 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faed811d610 con 0x7faed81393a0 2024-09-16T04:14:21.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.989+0000 7faedcd19640 1 --2- 172.21.15.42:0/2217608656 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7faed81393a0 0x7faed80706a0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7faed000cea0 tx=0x7faed000cf90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:14:21.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.989+0000 7faed4ff9640 10 client.?.objecter ms_handle_connect 0x7faed81393a0 2024-09-16T04:14:21.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.989+0000 7faed4ff9640 10 client.?.objecter resend_mon_ops 2024-09-16T04:14:21.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.989+0000 7faed4ff9640 1 -- 172.21.15.42:0/2217608656 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faed0003ba0 con 0x7faed81393a0 2024-09-16T04:14:21.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.989+0000 7faed4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:21.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.989+0000 7faed4ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:14:21.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.989+0000 7faed4ff9640 10 monclient: dump: 2024-09-16T04:14:21.010 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:14:21.010 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:14:21.010 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:21.010 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:21.010 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:21.010 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:21.010 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:14:21.010 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:14:21.010 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:14:21.010 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:21.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.989+0000 7faed4ff9640 10 monclient: _finish_auth 0 2024-09-16T04:14:21.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.989+0000 7faed4ff9640 10 monclient: _check_auth_tickets 2024-09-16T04:14:21.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.989+0000 7faed4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:14:21.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.989+0000 7faed4ff9640 1 -- 172.21.15.42:0/2217608656 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faed0003d60 con 0x7faed81393a0 2024-09-16T04:14:21.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.989+0000 7faed4ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T04:14:21.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.989+0000 7faed4ff9640 1 -- 172.21.15.42:0/2217608656 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faed000d880 con 0x7faed81393a0 2024-09-16T04:14:21.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.989+0000 7faed4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:21.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.989+0000 7faed4ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:14:21.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.989+0000 7faed4ff9640 10 monclient: dump: 2024-09-16T04:14:21.011 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:14:21.011 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:14:21.011 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:21.011 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:21.011 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:21.011 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:21.011 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:14:21.011 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:14:21.012 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:14:21.012 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:21.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.989+0000 7faede77c640 5 monclient: authenticate success, global_id 1094240 2024-09-16T04:14:21.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.989+0000 7faede77c640 10 monclient: _renew_subs 2024-09-16T04:14:21.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.989+0000 7faede77c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:14:21.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.989+0000 7faed67fc640 4 set_mon_vals no callback set 2024-09-16T04:14:21.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.989+0000 7faede77c640 1 -- 172.21.15.42:0/2217608656 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faed811d8f0 con 0x7faed81393a0 2024-09-16T04:14:21.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.989+0000 7faede77c640 10 client.1094240.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:14:21.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.989+0000 7faede77c640 10 monclient: _renew_subs 2024-09-16T04:14:21.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.989+0000 7faede77c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:14:21.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.989+0000 7faede77c640 1 -- 172.21.15.42:0/2217608656 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7faed8138af0 con 0x7faed81393a0 2024-09-16T04:14:21.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.989+0000 7faede77c640 1 librados: init done 2024-09-16T04:14:21.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.989+0000 7faed4ff9640 1 -- 172.21.15.42:0/2217608656 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7faed000da20 con 0x7faed81393a0 2024-09-16T04:14:21.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.989+0000 7faede77c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:14:21.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.989+0000 7faede77c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:14:21.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.989+0000 7faede77c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:14:21.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.989+0000 7faed4ff9640 1 --2- 172.21.15.42:0/2217608656 >> v2:172.21.15.154:6812/254845342 conn(0x7faec4037810 0x7faec4039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:21.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.989+0000 7faede77c640 1 -- 172.21.15.42:0/2217608656 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faed809f180 con 0x7faed81393a0 2024-09-16T04:14:21.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.989+0000 7faed4ff9640 1 -- 172.21.15.42:0/2217608656 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(710..710 src has 681..710) ==== 4681+0+0 (secure 0 0 0) 0x7faed00494f0 con 0x7faed81393a0 2024-09-16T04:14:21.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.989+0000 7faed4ff9640 10 client.1094240.objecter ms_dispatch 0x7faed8000ba0 osd_map(710..710 src has 681..710) 2024-09-16T04:14:21.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.989+0000 7faed4ff9640 3 client.1094240.objecter handle_osd_map got epochs [710,710] > 0 2024-09-16T04:14:21.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.989+0000 7faed4ff9640 3 client.1094240.objecter handle_osd_map decoding full epoch 710 2024-09-16T04:14:21.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.989+0000 7faed4ff9640 20 client.1094240.objecter dump_active .. 0 homeless 2024-09-16T04:14:21.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.989+0000 7faed7fff640 1 --2- 172.21.15.42:0/2217608656 >> v2:172.21.15.154:6812/254845342 conn(0x7faec4037810 0x7faec4039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:21.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.989+0000 7faed7fff640 10 monclient: get_auth_request con 0x7faec4037810 auth_method 0 2024-09-16T04:14:21.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.993+0000 7faed7fff640 1 --2- 172.21.15.42:0/2217608656 >> v2:172.21.15.154:6812/254845342 conn(0x7faec4037810 0x7faec4039cd0 secure :-1 s=READY pgs=4013 cs=0 l=1 rev1=1 crypto rx=0x7faec80ad000 tx=0x7faec80a6000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:14:21.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.993+0000 7faed4ff9640 10 client.1094240.objecter ms_handle_connect 0x7faec4037810 2024-09-16T04:14:21.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.997+0000 7faed4ff9640 1 -- 172.21.15.42:0/2217608656 <== mon.2 v2:172.21.15.42:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7faed0013340 con 0x7faed81393a0 2024-09-16T04:14:21.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.997+0000 7faed4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:14:21.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:20.997+0000 7faed4ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:14:21.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.157+0000 7faede77c640 1 -- 172.21.15.42:0/2217608656 --> v2:172.21.15.154:6812/254845342 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7faed80a0e80 con 0x7faec4037810 2024-09-16T04:14:21.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.165+0000 7faed4ff9640 1 -- 172.21.15.42:0/2217608656 <== mgr.4102 v2:172.21.15.154:6812/254845342 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+114174 (secure 0 0 0) 0x7faed80a0e80 con 0x7faec4037810 2024-09-16T04:14:21.173 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-09-16T04:14:21.175 INFO:teuthology.orchestra.run.smithi042.stderr:dumped all 2024-09-16T04:14:21.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.165+0000 7faede77c640 10 librados: watch_flush enter 2024-09-16T04:14:21.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.165+0000 7faede77c640 10 librados: watch_flush exit 2024-09-16T04:14:21.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.165+0000 7faede77c640 20 client.1094240.objecter shutdown clearing up homeless session... 2024-09-16T04:14:21.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.165+0000 7faede77c640 10 client.1094240.objecter successfully canceled tick 2024-09-16T04:14:21.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.169+0000 7faede77c640 1 -- 172.21.15.42:0/2217608656 >> v2:172.21.15.154:6812/254845342 conn(0x7faec4037810 msgr2=0x7faec4039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:21.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.169+0000 7faede77c640 1 --2- 172.21.15.42:0/2217608656 >> v2:172.21.15.154:6812/254845342 conn(0x7faec4037810 0x7faec4039cd0 secure :-1 s=READY pgs=4013 cs=0 l=1 rev1=1 crypto rx=0x7faec80ad000 tx=0x7faec80a6000 comp rx=0 tx=0).stop 2024-09-16T04:14:21.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.169+0000 7faede77c640 10 monclient: shutdown 2024-09-16T04:14:21.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.169+0000 7faede77c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:14:21.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.169+0000 7faede77c640 1 -- 172.21.15.42:0/2217608656 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7faed81393a0 msgr2=0x7faed80706a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:21.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.169+0000 7faede77c640 1 --2- 172.21.15.42:0/2217608656 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7faed81393a0 0x7faed80706a0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7faed000cea0 tx=0x7faed000cf90 comp rx=0 tx=0).stop 2024-09-16T04:14:21.176 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.169+0000 7faede77c640 1 -- 172.21.15.42:0/2217608656 shutdown_connections 2024-09-16T04:14:21.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.169+0000 7faede77c640 1 --2- 172.21.15.42:0/2217608656 >> v2:172.21.15.154:6812/254845342 conn(0x7faec4037810 0x7faec4039cd0 unknown :-1 s=CLOSED pgs=4013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:21.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.169+0000 7faede77c640 1 --2- 172.21.15.42:0/2217608656 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7faed81393a0 0x7faed80706a0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:21.177 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.169+0000 7faede77c640 1 --2- 172.21.15.42:0/2217608656 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7faed80a1710 0x7faed8132230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:21.180 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.169+0000 7faede77c640 1 --2- 172.21.15.42:0/2217608656 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7faed80a1330 0x7faed81ccb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:21.180 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.169+0000 7faede77c640 1 -- 172.21.15.42:0/2217608656 >> 172.21.15.42:0/2217608656 conn(0x7faed80a2420 msgr2=0x7faed80a4960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:14:21.180 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.169+0000 7faede77c640 1 -- 172.21.15.42:0/2217608656 shutdown_connections 2024-09-16T04:14:21.180 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.169+0000 7faede77c640 1 -- 172.21.15.42:0/2217608656 wait complete. 2024-09-16T04:14:21.180 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.169+0000 7faede77c640 1 librados: shutdown 2024-09-16T04:14:21.190 INFO:teuthology.orchestra.run.smithi042.stdout:{"pg_ready":true,"pg_map":{"version":1653,"stamp":"2024-09-16T04:14:20.231345+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590392,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":122,"num_read_kb":106,"num_write":168,"num_write_kb":2425,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":127,"ondisk_log_size":127,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":431992,"kb_used_data":22680,"kb_used_omap":9,"kb_used_meta":407734,"kb_avail":565799048,"statfs":{"total":579820584960,"available":579378225152,"internally_reserved":0,"allocated":23224320,"data_stored":10083938,"data_compressed":18838,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":9567,"internal_metadata":417520289},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003822"},"pg_stats":[{"pgid":"31.f","version":"0'0","reported_seq":1350,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:53.282605+0000","last_change":"2024-09-16T03:43:34.400720+0000","last_active":"2024-09-16T04:13:53.282605+0000","last_peered":"2024-09-16T04:13:53.282605+0000","last_clean":"2024-09-16T04:13:53.282605+0000","last_became_active":"2024-09-16T03:43:34.399531+0000","last_became_peered":"2024-09-16T03:43:34.399531+0000","last_unstale":"2024-09-16T04:13:53.282605+0000","last_undegraded":"2024-09-16T04:13:53.282605+0000","last_fullsized":"2024-09-16T04:13:53.282605+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_clean_scrub_stamp":"2024-09-16T03:43:33.221564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:29:59.415591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.e","version":"0'0","reported_seq":1351,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:53.356452+0000","last_change":"2024-09-16T03:43:34.492536+0000","last_active":"2024-09-16T04:13:53.356452+0000","last_peered":"2024-09-16T04:13:53.356452+0000","last_clean":"2024-09-16T04:13:53.356452+0000","last_became_active":"2024-09-16T03:43:34.492249+0000","last_became_peered":"2024-09-16T03:43:34.492249+0000","last_unstale":"2024-09-16T04:13:53.356452+0000","last_undegraded":"2024-09-16T04:13:53.356452+0000","last_fullsized":"2024-09-16T04:13:53.356452+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_clean_scrub_stamp":"2024-09-16T03:43:33.221564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:15:41.168066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.d","version":"0'0","reported_seq":1373,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:51.310003+0000","last_change":"2024-09-16T03:43:34.414280+0000","last_active":"2024-09-16T04:13:51.310003+0000","last_peered":"2024-09-16T04:13:51.310003+0000","last_clean":"2024-09-16T04:13:51.310003+0000","last_became_active":"2024-09-16T03:43:34.399055+0000","last_became_peered":"2024-09-16T03:43:34.399055+0000","last_unstale":"2024-09-16T04:13:51.310003+0000","last_undegraded":"2024-09-16T04:13:51.310003+0000","last_fullsized":"2024-09-16T04:13:51.310003+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_clean_scrub_stamp":"2024-09-16T03:43:33.221564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:08:19.292854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.c","version":"0'0","reported_seq":1353,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:52.857977+0000","last_change":"2024-09-16T03:43:34.403649+0000","last_active":"2024-09-16T04:13:52.857977+0000","last_peered":"2024-09-16T04:13:52.857977+0000","last_clean":"2024-09-16T04:13:52.857977+0000","last_became_active":"2024-09-16T03:43:34.403187+0000","last_became_peered":"2024-09-16T03:43:34.403187+0000","last_unstale":"2024-09-16T04:13:52.857977+0000","last_undegraded":"2024-09-16T04:13:52.857977+0000","last_fullsized":"2024-09-16T04:13:52.857977+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_clean_scrub_stamp":"2024-09-16T03:43:33.221564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:21:54.130294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.13","version":"0'0","reported_seq":1354,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:51.300256+0000","last_change":"2024-09-16T03:43:34.382675+0000","last_active":"2024-09-16T04:13:51.300256+0000","last_peered":"2024-09-16T04:13:51.300256+0000","last_clean":"2024-09-16T04:13:51.300256+0000","last_became_active":"2024-09-16T03:43:34.382422+0000","last_became_peered":"2024-09-16T03:43:34.382422+0000","last_unstale":"2024-09-16T04:13:51.300256+0000","last_undegraded":"2024-09-16T04:13:51.300256+0000","last_fullsized":"2024-09-16T04:13:51.300256+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_clean_scrub_stamp":"2024-09-16T03:43:33.221564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:52:35.625157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.12","version":"0'0","reported_seq":1353,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:52.857683+0000","last_change":"2024-09-16T03:43:34.404613+0000","last_active":"2024-09-16T04:13:52.857683+0000","last_peered":"2024-09-16T04:13:52.857683+0000","last_clean":"2024-09-16T04:13:52.857683+0000","last_became_active":"2024-09-16T03:43:34.404299+0000","last_became_peered":"2024-09-16T03:43:34.404299+0000","last_unstale":"2024-09-16T04:13:52.857683+0000","last_undegraded":"2024-09-16T04:13:52.857683+0000","last_fullsized":"2024-09-16T04:13:52.857683+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_clean_scrub_stamp":"2024-09-16T03:43:33.221564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:25:52.933411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.11","version":"0'0","reported_seq":1354,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:51.299816+0000","last_change":"2024-09-16T03:43:34.355045+0000","last_active":"2024-09-16T04:13:51.299816+0000","last_peered":"2024-09-16T04:13:51.299816+0000","last_clean":"2024-09-16T04:13:51.299816+0000","last_became_active":"2024-09-16T03:43:34.354598+0000","last_became_peered":"2024-09-16T03:43:34.354598+0000","last_unstale":"2024-09-16T04:13:51.299816+0000","last_undegraded":"2024-09-16T04:13:51.299816+0000","last_fullsized":"2024-09-16T04:13:51.299816+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_clean_scrub_stamp":"2024-09-16T03:43:33.221564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:40:14.558362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.10","version":"0'0","reported_seq":1351,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:53.353474+0000","last_change":"2024-09-16T03:43:34.436288+0000","last_active":"2024-09-16T04:13:53.353474+0000","last_peered":"2024-09-16T04:13:53.353474+0000","last_clean":"2024-09-16T04:13:53.353474+0000","last_became_active":"2024-09-16T03:43:34.434548+0000","last_became_peered":"2024-09-16T03:43:34.434548+0000","last_unstale":"2024-09-16T04:13:53.353474+0000","last_undegraded":"2024-09-16T04:13:53.353474+0000","last_fullsized":"2024-09-16T04:13:53.353474+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_clean_scrub_stamp":"2024-09-16T03:43:33.221564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:03:15.848714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.17","version":"0'0","reported_seq":1349,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:53.282881+0000","last_change":"2024-09-16T03:43:34.456644+0000","last_active":"2024-09-16T04:13:53.282881+0000","last_peered":"2024-09-16T04:13:53.282881+0000","last_clean":"2024-09-16T04:13:53.282881+0000","last_became_active":"2024-09-16T03:43:34.454859+0000","last_became_peered":"2024-09-16T03:43:34.454859+0000","last_unstale":"2024-09-16T04:13:53.282881+0000","last_undegraded":"2024-09-16T04:13:53.282881+0000","last_fullsized":"2024-09-16T04:13:53.282881+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_clean_scrub_stamp":"2024-09-16T03:43:33.221564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:45:01.348488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.16","version":"0'0","reported_seq":1347,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:53.166394+0000","last_change":"2024-09-16T03:43:34.391035+0000","last_active":"2024-09-16T04:13:53.166394+0000","last_peered":"2024-09-16T04:13:53.166394+0000","last_clean":"2024-09-16T04:13:53.166394+0000","last_became_active":"2024-09-16T03:43:34.387629+0000","last_became_peered":"2024-09-16T03:43:34.387629+0000","last_unstale":"2024-09-16T04:13:53.166394+0000","last_undegraded":"2024-09-16T04:13:53.166394+0000","last_fullsized":"2024-09-16T04:13:53.166394+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_clean_scrub_stamp":"2024-09-16T03:43:33.221564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:11:59.627229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.15","version":"0'0","reported_seq":1352,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:52.859996+0000","last_change":"2024-09-16T03:43:34.387904+0000","last_active":"2024-09-16T04:13:52.859996+0000","last_peered":"2024-09-16T04:13:52.859996+0000","last_clean":"2024-09-16T04:13:52.859996+0000","last_became_active":"2024-09-16T03:43:34.387630+0000","last_became_peered":"2024-09-16T03:43:34.387630+0000","last_unstale":"2024-09-16T04:13:52.859996+0000","last_undegraded":"2024-09-16T04:13:52.859996+0000","last_fullsized":"2024-09-16T04:13:52.859996+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_clean_scrub_stamp":"2024-09-16T03:43:33.221564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:15:37.951806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.14","version":"0'0","reported_seq":1350,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:53.282153+0000","last_change":"2024-09-16T03:43:34.381624+0000","last_active":"2024-09-16T04:13:53.282153+0000","last_peered":"2024-09-16T04:13:53.282153+0000","last_clean":"2024-09-16T04:13:53.282153+0000","last_became_active":"2024-09-16T03:43:34.378385+0000","last_became_peered":"2024-09-16T03:43:34.378385+0000","last_unstale":"2024-09-16T04:13:53.282153+0000","last_undegraded":"2024-09-16T04:13:53.282153+0000","last_fullsized":"2024-09-16T04:13:53.282153+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_clean_scrub_stamp":"2024-09-16T03:43:33.221564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:20:14.916099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"14'117","reported_seq":1523,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:52.858864+0000","last_change":"2024-09-16T03:42:18.799294+0000","last_active":"2024-09-16T04:13:52.858864+0000","last_peered":"2024-09-16T04:13:52.858864+0000","last_clean":"2024-09-16T04:13:52.858864+0000","last_became_active":"2024-09-16T03:42:18.797239+0000","last_became_peered":"2024-09-16T03:42:18.797239+0000","last_unstale":"2024-09-16T04:13:52.858864+0000","last_undegraded":"2024-09-16T04:13:52.858864+0000","last_fullsized":"2024-09-16T04:13:52.858864+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:42:17.544237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:42:17.544237+0000","last_clean_scrub_stamp":"2024-09-16T03:42:17.544237+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:01:21.017593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":1374,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:51.300417+0000","last_change":"2024-09-16T03:42:27.630965+0000","last_active":"2024-09-16T04:13:51.300417+0000","last_peered":"2024-09-16T04:13:51.300417+0000","last_clean":"2024-09-16T04:13:51.300417+0000","last_became_active":"2024-09-16T03:42:25.615930+0000","last_became_peered":"2024-09-16T03:42:25.615930+0000","last_unstale":"2024-09-16T04:13:51.300417+0000","last_undegraded":"2024-09-16T04:13:51.300417+0000","last_fullsized":"2024-09-16T04:13:51.300417+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:42:24.583922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:42:24.583922+0000","last_clean_scrub_stamp":"2024-09-16T03:42:24.583922+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:59:59.653118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00099470900000000009,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.1e","version":"0'0","reported_seq":1355,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:51.302860+0000","last_change":"2024-09-16T03:43:34.401653+0000","last_active":"2024-09-16T04:13:51.302860+0000","last_peered":"2024-09-16T04:13:51.302860+0000","last_clean":"2024-09-16T04:13:51.302860+0000","last_became_active":"2024-09-16T03:43:34.397708+0000","last_became_peered":"2024-09-16T03:43:34.397708+0000","last_unstale":"2024-09-16T04:13:51.302860+0000","last_undegraded":"2024-09-16T04:13:51.302860+0000","last_fullsized":"2024-09-16T04:13:51.302860+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_clean_scrub_stamp":"2024-09-16T03:43:33.221564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:16:17.302952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1372,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:52.859154+0000","last_change":"2024-09-16T03:42:27.624504+0000","last_active":"2024-09-16T04:13:52.859154+0000","last_peered":"2024-09-16T04:13:52.859154+0000","last_clean":"2024-09-16T04:13:52.859154+0000","last_became_active":"2024-09-16T03:42:25.612386+0000","last_became_peered":"2024-09-16T03:42:25.612386+0000","last_unstale":"2024-09-16T04:13:52.859154+0000","last_undegraded":"2024-09-16T04:13:52.859154+0000","last_fullsized":"2024-09-16T04:13:52.859154+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:42:24.583922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:42:24.583922+0000","last_clean_scrub_stamp":"2024-09-16T03:42:24.583922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:00:49.446880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081399800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1d","version":"0'0","reported_seq":1372,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:51.310252+0000","last_change":"2024-09-16T03:43:34.408908+0000","last_active":"2024-09-16T04:13:51.310252+0000","last_peered":"2024-09-16T04:13:51.310252+0000","last_clean":"2024-09-16T04:13:51.310252+0000","last_became_active":"2024-09-16T03:43:34.399300+0000","last_became_peered":"2024-09-16T03:43:34.399300+0000","last_unstale":"2024-09-16T04:13:51.310252+0000","last_undegraded":"2024-09-16T04:13:51.310252+0000","last_fullsized":"2024-09-16T04:13:51.310252+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_clean_scrub_stamp":"2024-09-16T03:43:33.221564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:25:51.517103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1370,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:53.353598+0000","last_change":"2024-09-16T03:42:27.619252+0000","last_active":"2024-09-16T04:13:53.353598+0000","last_peered":"2024-09-16T04:13:53.353598+0000","last_clean":"2024-09-16T04:13:53.353598+0000","last_became_active":"2024-09-16T03:42:25.607533+0000","last_became_peered":"2024-09-16T03:42:25.607533+0000","last_unstale":"2024-09-16T04:13:53.353598+0000","last_undegraded":"2024-09-16T04:13:53.353598+0000","last_fullsized":"2024-09-16T04:13:53.353598+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:42:24.583922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:42:24.583922+0000","last_clean_scrub_stamp":"2024-09-16T03:42:24.583922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:25:58.536144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089083299999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1c","version":"0'0","reported_seq":1349,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:53.282306+0000","last_change":"2024-09-16T03:43:34.421587+0000","last_active":"2024-09-16T04:13:53.282306+0000","last_peered":"2024-09-16T04:13:53.282306+0000","last_clean":"2024-09-16T04:13:53.282306+0000","last_became_active":"2024-09-16T03:43:34.421320+0000","last_became_peered":"2024-09-16T03:43:34.421320+0000","last_unstale":"2024-09-16T04:13:53.282306+0000","last_undegraded":"2024-09-16T04:13:53.282306+0000","last_fullsized":"2024-09-16T04:13:53.282306+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_clean_scrub_stamp":"2024-09-16T03:43:33.221564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:47:25.449200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":1375,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:51.299718+0000","last_change":"2024-09-16T03:42:27.632050+0000","last_active":"2024-09-16T04:13:51.299718+0000","last_peered":"2024-09-16T04:13:51.299718+0000","last_clean":"2024-09-16T04:13:51.299718+0000","last_became_active":"2024-09-16T03:42:25.617896+0000","last_became_peered":"2024-09-16T03:42:25.617896+0000","last_unstale":"2024-09-16T04:13:51.299718+0000","last_undegraded":"2024-09-16T04:13:51.299718+0000","last_fullsized":"2024-09-16T04:13:51.299718+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:42:24.583922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:42:24.583922+0000","last_clean_scrub_stamp":"2024-09-16T03:42:24.583922+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:41:08.754274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021072529999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.1f","version":"0'0","reported_seq":1352,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:53.356655+0000","last_change":"2024-09-16T03:43:34.419004+0000","last_active":"2024-09-16T04:13:53.356655+0000","last_peered":"2024-09-16T04:13:53.356655+0000","last_clean":"2024-09-16T04:13:53.356655+0000","last_became_active":"2024-09-16T03:43:34.415747+0000","last_became_peered":"2024-09-16T03:43:34.415747+0000","last_unstale":"2024-09-16T04:13:53.356655+0000","last_undegraded":"2024-09-16T04:13:53.356655+0000","last_fullsized":"2024-09-16T04:13:53.356655+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_clean_scrub_stamp":"2024-09-16T03:43:33.221564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:01.644098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1367,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:53.166325+0000","last_change":"2024-09-16T03:42:27.631260+0000","last_active":"2024-09-16T04:13:53.166325+0000","last_peered":"2024-09-16T04:13:53.166325+0000","last_clean":"2024-09-16T04:13:53.166325+0000","last_became_active":"2024-09-16T03:42:25.609631+0000","last_became_peered":"2024-09-16T03:42:25.609631+0000","last_unstale":"2024-09-16T04:13:53.166325+0000","last_undegraded":"2024-09-16T04:13:53.166325+0000","last_fullsized":"2024-09-16T04:13:53.166325+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:42:24.583922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:42:24.583922+0000","last_clean_scrub_stamp":"2024-09-16T03:42:24.583922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:58:49.091085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001288897,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.19","version":"0'0","reported_seq":1347,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:53.166637+0000","last_change":"2024-09-16T03:43:34.341660+0000","last_active":"2024-09-16T04:13:53.166637+0000","last_peered":"2024-09-16T04:13:53.166637+0000","last_clean":"2024-09-16T04:13:53.166637+0000","last_became_active":"2024-09-16T03:43:34.332234+0000","last_became_peered":"2024-09-16T03:43:34.332234+0000","last_unstale":"2024-09-16T04:13:53.166637+0000","last_undegraded":"2024-09-16T04:13:53.166637+0000","last_fullsized":"2024-09-16T04:13:53.166637+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_clean_scrub_stamp":"2024-09-16T03:43:33.221564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:20:27.326212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1372,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:52.859726+0000","last_change":"2024-09-16T03:42:27.625091+0000","last_active":"2024-09-16T04:13:52.859726+0000","last_peered":"2024-09-16T04:13:52.859726+0000","last_clean":"2024-09-16T04:13:52.859726+0000","last_became_active":"2024-09-16T03:42:25.614458+0000","last_became_peered":"2024-09-16T03:42:25.614458+0000","last_unstale":"2024-09-16T04:13:52.859726+0000","last_undegraded":"2024-09-16T04:13:52.859726+0000","last_fullsized":"2024-09-16T04:13:52.859726+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:42:24.583922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:42:24.583922+0000","last_clean_scrub_stamp":"2024-09-16T03:42:24.583922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:26:14.681436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012701769999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.18","version":"0'0","reported_seq":1347,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:53.165619+0000","last_change":"2024-09-16T03:43:34.362758+0000","last_active":"2024-09-16T04:13:53.165619+0000","last_peered":"2024-09-16T04:13:53.165619+0000","last_clean":"2024-09-16T04:13:53.165619+0000","last_became_active":"2024-09-16T03:43:34.354705+0000","last_became_peered":"2024-09-16T03:43:34.354705+0000","last_unstale":"2024-09-16T04:13:53.165619+0000","last_undegraded":"2024-09-16T04:13:53.165619+0000","last_fullsized":"2024-09-16T04:13:53.165619+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_clean_scrub_stamp":"2024-09-16T03:43:33.221564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:16:36.345303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1366,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:53.166473+0000","last_change":"2024-09-16T03:42:27.631227+0000","last_active":"2024-09-16T04:13:53.166473+0000","last_peered":"2024-09-16T04:13:53.166473+0000","last_clean":"2024-09-16T04:13:53.166473+0000","last_became_active":"2024-09-16T03:42:25.608453+0000","last_became_peered":"2024-09-16T03:42:25.608453+0000","last_unstale":"2024-09-16T04:13:53.166473+0000","last_undegraded":"2024-09-16T04:13:53.166473+0000","last_fullsized":"2024-09-16T04:13:53.166473+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:42:24.583922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:42:24.583922+0000","last_clean_scrub_stamp":"2024-09-16T03:42:24.583922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:00:11.921493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001163533,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.1b","version":"0'0","reported_seq":1352,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:52.859563+0000","last_change":"2024-09-16T03:43:34.381629+0000","last_active":"2024-09-16T04:13:52.859563+0000","last_peered":"2024-09-16T04:13:52.859563+0000","last_clean":"2024-09-16T04:13:52.859563+0000","last_became_active":"2024-09-16T03:43:34.381335+0000","last_became_peered":"2024-09-16T03:43:34.381335+0000","last_unstale":"2024-09-16T04:13:52.859563+0000","last_undegraded":"2024-09-16T04:13:52.859563+0000","last_fullsized":"2024-09-16T04:13:52.859563+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_clean_scrub_stamp":"2024-09-16T03:43:33.221564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:30:50.930529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1371,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:52.861031+0000","last_change":"2024-09-16T03:42:27.624922+0000","last_active":"2024-09-16T04:13:52.861031+0000","last_peered":"2024-09-16T04:13:52.861031+0000","last_clean":"2024-09-16T04:13:52.861031+0000","last_became_active":"2024-09-16T03:42:25.615293+0000","last_became_peered":"2024-09-16T03:42:25.615293+0000","last_unstale":"2024-09-16T04:13:52.861031+0000","last_undegraded":"2024-09-16T04:13:52.861031+0000","last_fullsized":"2024-09-16T04:13:52.861031+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:42:24.583922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:42:24.583922+0000","last_clean_scrub_stamp":"2024-09-16T03:42:24.583922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:46:11.677251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010681690000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1a","version":"0'0","reported_seq":1349,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:53.281860+0000","last_change":"2024-09-16T03:43:34.382293+0000","last_active":"2024-09-16T04:13:53.281860+0000","last_peered":"2024-09-16T04:13:53.281860+0000","last_clean":"2024-09-16T04:13:53.281860+0000","last_became_active":"2024-09-16T03:43:34.372307+0000","last_became_peered":"2024-09-16T03:43:34.372307+0000","last_unstale":"2024-09-16T04:13:53.281860+0000","last_undegraded":"2024-09-16T04:13:53.281860+0000","last_fullsized":"2024-09-16T04:13:53.281860+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_clean_scrub_stamp":"2024-09-16T03:43:33.221564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:49:25.481308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.8","version":"0'0","reported_seq":1351,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:53.280246+0000","last_change":"2024-09-16T03:43:34.386011+0000","last_active":"2024-09-16T04:13:53.280246+0000","last_peered":"2024-09-16T04:13:53.280246+0000","last_clean":"2024-09-16T04:13:53.280246+0000","last_became_active":"2024-09-16T03:43:34.378483+0000","last_became_peered":"2024-09-16T03:43:34.378483+0000","last_unstale":"2024-09-16T04:13:53.280246+0000","last_undegraded":"2024-09-16T04:13:53.280246+0000","last_fullsized":"2024-09-16T04:13:53.280246+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_clean_scrub_stamp":"2024-09-16T03:43:33.221564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:25:42.568633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.9","version":"0'0","reported_seq":1374,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:51.306525+0000","last_change":"2024-09-16T03:43:34.389268+0000","last_active":"2024-09-16T04:13:51.306525+0000","last_peered":"2024-09-16T04:13:51.306525+0000","last_clean":"2024-09-16T04:13:51.306525+0000","last_became_active":"2024-09-16T03:43:34.378733+0000","last_became_peered":"2024-09-16T03:43:34.378733+0000","last_unstale":"2024-09-16T04:13:51.306525+0000","last_undegraded":"2024-09-16T04:13:51.306525+0000","last_fullsized":"2024-09-16T04:13:51.306525+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_clean_scrub_stamp":"2024-09-16T03:43:33.221564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:16:08.231273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.a","version":"0'0","reported_seq":1374,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:51.306820+0000","last_change":"2024-09-16T03:43:34.390824+0000","last_active":"2024-09-16T04:13:51.306820+0000","last_peered":"2024-09-16T04:13:51.306820+0000","last_clean":"2024-09-16T04:13:51.306820+0000","last_became_active":"2024-09-16T03:43:34.390049+0000","last_became_peered":"2024-09-16T03:43:34.390049+0000","last_unstale":"2024-09-16T04:13:51.306820+0000","last_undegraded":"2024-09-16T04:13:51.306820+0000","last_fullsized":"2024-09-16T04:13:51.306820+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_clean_scrub_stamp":"2024-09-16T03:43:33.221564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:48:31.039881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.b","version":"0'0","reported_seq":1348,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:53.165918+0000","last_change":"2024-09-16T03:43:34.332247+0000","last_active":"2024-09-16T04:13:53.165918+0000","last_peered":"2024-09-16T04:13:53.165918+0000","last_clean":"2024-09-16T04:13:53.165918+0000","last_became_active":"2024-09-16T03:43:34.328425+0000","last_became_peered":"2024-09-16T03:43:34.328425+0000","last_unstale":"2024-09-16T04:13:53.165918+0000","last_undegraded":"2024-09-16T04:13:53.165918+0000","last_fullsized":"2024-09-16T04:13:53.165918+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_clean_scrub_stamp":"2024-09-16T03:43:33.221564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:38:08.042571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.4","version":"0'0","reported_seq":1351,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:53.282653+0000","last_change":"2024-09-16T03:43:34.392184+0000","last_active":"2024-09-16T04:13:53.282653+0000","last_peered":"2024-09-16T04:13:53.282653+0000","last_clean":"2024-09-16T04:13:53.282653+0000","last_became_active":"2024-09-16T03:43:34.391868+0000","last_became_peered":"2024-09-16T03:43:34.391868+0000","last_unstale":"2024-09-16T04:13:53.282653+0000","last_undegraded":"2024-09-16T04:13:53.282653+0000","last_fullsized":"2024-09-16T04:13:53.282653+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_clean_scrub_stamp":"2024-09-16T03:43:33.221564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:02:40.567145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.5","version":"0'0","reported_seq":1351,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:53.280846+0000","last_change":"2024-09-16T03:43:34.419668+0000","last_active":"2024-09-16T04:13:53.280846+0000","last_peered":"2024-09-16T04:13:53.280846+0000","last_clean":"2024-09-16T04:13:53.280846+0000","last_became_active":"2024-09-16T03:43:34.419410+0000","last_became_peered":"2024-09-16T03:43:34.419410+0000","last_unstale":"2024-09-16T04:13:53.280846+0000","last_undegraded":"2024-09-16T04:13:53.280846+0000","last_fullsized":"2024-09-16T04:13:53.280846+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_clean_scrub_stamp":"2024-09-16T03:43:33.221564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:37:43.007953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.6","version":"0'0","reported_seq":1374,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:51.309271+0000","last_change":"2024-09-16T03:43:34.386607+0000","last_active":"2024-09-16T04:13:51.309271+0000","last_peered":"2024-09-16T04:13:51.309271+0000","last_clean":"2024-09-16T04:13:51.309271+0000","last_became_active":"2024-09-16T03:43:34.381834+0000","last_became_peered":"2024-09-16T03:43:34.381834+0000","last_unstale":"2024-09-16T04:13:51.309271+0000","last_undegraded":"2024-09-16T04:13:51.309271+0000","last_fullsized":"2024-09-16T04:13:51.309271+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_clean_scrub_stamp":"2024-09-16T03:43:33.221564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:06:41.696398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.7","version":"0'0","reported_seq":1349,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:53.164974+0000","last_change":"2024-09-16T03:43:34.296793+0000","last_active":"2024-09-16T04:13:53.164974+0000","last_peered":"2024-09-16T04:13:53.164974+0000","last_clean":"2024-09-16T04:13:53.164974+0000","last_became_active":"2024-09-16T03:43:34.292756+0000","last_became_peered":"2024-09-16T03:43:34.292756+0000","last_unstale":"2024-09-16T04:13:53.164974+0000","last_undegraded":"2024-09-16T04:13:53.164974+0000","last_fullsized":"2024-09-16T04:13:53.164974+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_clean_scrub_stamp":"2024-09-16T03:43:33.221564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:12:04.533479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.0","version":"0'0","reported_seq":1355,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:52.859564+0000","last_change":"2024-09-16T03:43:34.352792+0000","last_active":"2024-09-16T04:13:52.859564+0000","last_peered":"2024-09-16T04:13:52.859564+0000","last_clean":"2024-09-16T04:13:52.859564+0000","last_became_active":"2024-09-16T03:43:34.351273+0000","last_became_peered":"2024-09-16T03:43:34.351273+0000","last_unstale":"2024-09-16T04:13:52.859564+0000","last_undegraded":"2024-09-16T04:13:52.859564+0000","last_fullsized":"2024-09-16T04:13:52.859564+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_clean_scrub_stamp":"2024-09-16T03:43:33.221564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:50:21.469378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1","version":"22'7","reported_seq":1362,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:53.164484+0000","last_change":"2024-09-16T03:43:34.397561+0000","last_active":"2024-09-16T04:13:53.164484+0000","last_peered":"2024-09-16T04:13:53.164484+0000","last_clean":"2024-09-16T04:13:53.164484+0000","last_became_active":"2024-09-16T03:43:34.382744+0000","last_became_peered":"2024-09-16T03:43:34.382744+0000","last_unstale":"2024-09-16T04:13:53.164484+0000","last_undegraded":"2024-09-16T04:13:53.164484+0000","last_fullsized":"2024-09-16T04:13:53.164484+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_clean_scrub_stamp":"2024-09-16T03:43:33.221564+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:54:00.268175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.2","version":"0'0","reported_seq":1355,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:51.300709+0000","last_change":"2024-09-16T03:43:34.402175+0000","last_active":"2024-09-16T04:13:51.300709+0000","last_peered":"2024-09-16T04:13:51.300709+0000","last_clean":"2024-09-16T04:13:51.300709+0000","last_became_active":"2024-09-16T03:43:34.402014+0000","last_became_peered":"2024-09-16T03:43:34.402014+0000","last_unstale":"2024-09-16T04:13:51.300709+0000","last_undegraded":"2024-09-16T04:13:51.300709+0000","last_fullsized":"2024-09-16T04:13:51.300709+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_clean_scrub_stamp":"2024-09-16T03:43:33.221564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:46:39.366181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.3","version":"0'0","reported_seq":1374,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:51.306977+0000","last_change":"2024-09-16T03:43:34.380947+0000","last_active":"2024-09-16T04:13:51.306977+0000","last_peered":"2024-09-16T04:13:51.306977+0000","last_clean":"2024-09-16T04:13:51.306977+0000","last_became_active":"2024-09-16T03:43:34.379988+0000","last_became_peered":"2024-09-16T03:43:34.379988+0000","last_unstale":"2024-09-16T04:13:51.306977+0000","last_undegraded":"2024-09-16T04:13:51.306977+0000","last_fullsized":"2024-09-16T04:13:51.306977+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_clean_scrub_stamp":"2024-09-16T03:43:33.221564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:21:15.646033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":31,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":18838,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":12,"seq":51539607943,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":71596,"kb_used_data":3764,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94300244,"statfs":{"total":96636764160,"available":96563449856,"internally_reserved":0,"allocated":3854336,"data_stored":1504358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":11,"seq":47244640646,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":71228,"kb_used_data":3764,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94300612,"statfs":{"total":96636764160,"available":96563826688,"internally_reserved":0,"allocated":3854336,"data_stored":1504344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640643,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":67116,"kb_used_data":3812,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94304724,"statfs":{"total":96636764160,"available":96568037376,"internally_reserved":0,"allocated":3903488,"data_stored":2033267,"data_compressed":9419,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1593,"internal_metadata":64616903},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640644,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":71224,"kb_used_data":3760,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94300616,"statfs":{"total":96636764160,"available":96563830784,"internally_reserved":0,"allocated":3850240,"data_stored":1504339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":68876742},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640645,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":79552,"kb_used_data":3768,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94292288,"statfs":{"total":96636764160,"available":96555302912,"internally_reserved":0,"allocated":3858432,"data_stored":1504363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640644,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":71276,"kb_used_data":3812,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94300564,"statfs":{"total":96636764160,"available":96563777536,"internally_reserved":0,"allocated":3903488,"data_stored":2033267,"data_compressed":9419,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":9419,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":9419,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:14:21.193 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T04:14:21.318 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.313+0000 7f5f880e8640 10 monclient: get_monmap_and_config 2024-09-16T04:14:21.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.313+0000 7f5f880e8640 10 monclient: build_initial_monmap 2024-09-16T04:14:21.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.313+0000 7f5f880e8640 10 monclient: monmap: 2024-09-16T04:14:21.319 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:14:21.319 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:14:21.319 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:14:21.317576+0000 2024-09-16T04:14:21.319 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:14:21.317576+0000 2024-09-16T04:14:21.319 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:14:21.319 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:21.319 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:14:21.319 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:14:21.319 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:14:21.320 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:14:21.320 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:14:21.320 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:21.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.313+0000 7f5f880e8640 1 Processor -- start 2024-09-16T04:14:21.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.313+0000 7f5f880e8640 1 -- start start 2024-09-16T04:14:21.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.313+0000 7f5f880e8640 10 monclient: init 2024-09-16T04:14:21.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.313+0000 7f5f880e8640 10 monclient: _reopen_session rank -1 2024-09-16T04:14:21.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.313+0000 7f5f880e8640 10 monclient: _add_conns ranks=[4,2,1,0,3] 2024-09-16T04:14:21.322 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.313+0000 7f5f880e8640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f5f8006db90 0x7f5f801340c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:21.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.313+0000 7f5f880e8640 10 monclient(hunting): picked mon.noname-c con 0x7f5f8006db90 addr v2:172.21.15.154:3300/0 2024-09-16T04:14:21.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.313+0000 7f5f880e8640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5f800a37f0 0x7f5f800a3c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:21.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.313+0000 7f5f880e8640 10 monclient(hunting): picked mon.noname-b con 0x7f5f800a37f0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:14:21.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.313+0000 7f5f880e8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5f80136e10 addr v1:172.21.15.154:6789/0 2024-09-16T04:14:21.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.313+0000 7f5f880e8640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5f8006ece0 con 0x7f5f80136e10 2024-09-16T04:14:21.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.313+0000 7f5f880e8640 10 monclient(hunting): start opening mon connection 2024-09-16T04:14:21.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.313+0000 7f5f880e8640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f5f8006df70 con 0x7f5f8006db90 2024-09-16T04:14:21.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.313+0000 7f5f880e8640 10 monclient(hunting): start opening mon connection 2024-09-16T04:14:21.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.313+0000 7f5f880e8640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f5f8009bf80 con 0x7f5f800a37f0 2024-09-16T04:14:21.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.313+0000 7f5f880e8640 10 monclient(hunting): _renew_subs 2024-09-16T04:14:21.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.313+0000 7f5f880e8640 10 monclient(hunting): authenticate will time out at 2908.287258s 2024-09-16T04:14:21.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.313+0000 7f5f85683640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5f800a37f0 0x7f5f800a3c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:21.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.313+0000 7f5f85683640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5f800a37f0 0x7f5f800a3c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:51914/0 (socket says 172.21.15.42:51914) 2024-09-16T04:14:21.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.313+0000 7f5f85683640 1 -- 172.21.15.42:0/2850924493 learned_addr learned my addr 172.21.15.42:0/2850924493 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:14:21.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.313+0000 7f5f85683640 10 monclient(hunting): get_auth_request con 0x7f5f800a37f0 auth_method 0 2024-09-16T04:14:21.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.313+0000 7f5f85683640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:14:21.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.313+0000 7f5f85683640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:21.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.313+0000 7f5f85683640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:21.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.313+0000 7f5f85683640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:14:21.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.313+0000 7f5f85683640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:14:21.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.313+0000 7f5f85683640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:14:21.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.313+0000 7f5f85e84640 1 --2- 172.21.15.42:0/2850924493 >> v2:172.21.15.154:3300/0 conn(0x7f5f8006db90 0x7f5f801340c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:21.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.313+0000 7f5f85e84640 10 monclient(hunting): get_auth_request con 0x7f5f8006db90 auth_method 0 2024-09-16T04:14:21.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.313+0000 7f5f85e84640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:14:21.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.313+0000 7f5f85e84640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:21.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.313+0000 7f5f85e84640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:21.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.313+0000 7f5f85683640 10 monclient(hunting): handle_auth_done global_id 1094243 payload 274 2024-09-16T04:14:21.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.313+0000 7f5f85683640 1 -- 172.21.15.42:0/2850924493 >> v2:172.21.15.154:3300/0 conn(0x7f5f8006db90 msgr2=0x7f5f801340c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:14:21.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.313+0000 7f5f85683640 1 --2- 172.21.15.42:0/2850924493 >> v2:172.21.15.154:3300/0 conn(0x7f5f8006db90 0x7f5f801340c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:21.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.313+0000 7f5f85683640 1 -- 172.21.15.42:0/2850924493 >> v1:172.21.15.154:6789/0 conn(0x7f5f80136e10 legacy=0x7f5f80137240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:14:21.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.313+0000 7f5f85683640 10 monclient: _finish_hunting 0 2024-09-16T04:14:21.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.313+0000 7f5f85683640 1 monclient: found mon.noname-b 2024-09-16T04:14:21.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.313+0000 7f5f85683640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:14:21.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.313+0000 7f5f85683640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T04:14:21.325 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.313+0000 7f5f85683640 1 -- 172.21.15.42:0/2850924493 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f80134780 con 0x7f5f800a37f0 2024-09-16T04:14:21.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.317+0000 7f5f85683640 1 --2- 172.21.15.42:0/2850924493 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5f800a37f0 0x7f5f800a3c00 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f5f74009870 tx=0x7f5f7401c3e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=63f906f0d48ac1cf server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:14:21.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.317+0000 7f5f84e82640 1 -- 172.21.15.42:0/2850924493 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f740b0070 con 0x7f5f800a37f0 2024-09-16T04:14:21.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.317+0000 7f5f84e82640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:21.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.317+0000 7f5f84e82640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:14:21.331 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.317+0000 7f5f84e82640 10 monclient: dump: 2024-09-16T04:14:21.331 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:14:21.331 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:14:21.331 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:21.331 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:21.331 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:21.331 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:21.331 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:14:21.332 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:14:21.332 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:14:21.332 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:21.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.317+0000 7f5f84e82640 10 monclient: _finish_auth 0 2024-09-16T04:14:21.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.317+0000 7f5f84e82640 10 monclient: _check_auth_tickets 2024-09-16T04:14:21.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.317+0000 7f5f84e82640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:14:21.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.317+0000 7f5f84e82640 1 -- 172.21.15.42:0/2850924493 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5f740ab600 con 0x7f5f800a37f0 2024-09-16T04:14:21.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.317+0000 7f5f880e8640 5 monclient: authenticate success, global_id 1094243 2024-09-16T04:14:21.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.317+0000 7f5f880e8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:14:21.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.317+0000 7f5f84e82640 10 monclient: handle_config config(0 keys) 2024-09-16T04:14:21.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.317+0000 7f5f84e82640 1 -- 172.21.15.42:0/2850924493 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f740ab9d0 con 0x7f5f800a37f0 2024-09-16T04:14:21.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.317+0000 7f5f84e82640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:21.332 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.317+0000 7f5f84e82640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:14:21.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.317+0000 7f5f84e82640 10 monclient: dump: 2024-09-16T04:14:21.333 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:14:21.333 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:14:21.333 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:21.333 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:21.333 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:21.333 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:21.333 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:14:21.333 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:14:21.333 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:14:21.333 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:21.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.317+0000 7f5f880e8640 10 monclient: get_monmap_and_config success 2024-09-16T04:14:21.333 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.317+0000 7f5f880e8640 4 set_mon_vals no callback set 2024-09-16T04:14:21.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.317+0000 7f5f880e8640 10 monclient: shutdown 2024-09-16T04:14:21.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.317+0000 7f5f880e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:14:21.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.317+0000 7f5f880e8640 1 -- 172.21.15.42:0/2850924493 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5f800a37f0 msgr2=0x7f5f800a3c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:21.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.317+0000 7f5f880e8640 1 --2- 172.21.15.42:0/2850924493 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5f800a37f0 0x7f5f800a3c00 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f5f74009870 tx=0x7f5f7401c3e0 comp rx=0 tx=0).stop 2024-09-16T04:14:21.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.317+0000 7f5f880e8640 1 -- 172.21.15.42:0/2850924493 shutdown_connections 2024-09-16T04:14:21.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.317+0000 7f5f880e8640 1 --2- 172.21.15.42:0/2850924493 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5f800a37f0 0x7f5f800a3c00 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:21.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.317+0000 7f5f880e8640 1 --2- 172.21.15.42:0/2850924493 >> v2:172.21.15.154:3300/0 conn(0x7f5f8006db90 0x7f5f801340c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:21.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.317+0000 7f5f880e8640 1 -- 172.21.15.42:0/2850924493 >> 172.21.15.42:0/2850924493 conn(0x7f5f800a9fb0 msgr2=0x7f5f800aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:14:21.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.317+0000 7f5f880e8640 1 -- 172.21.15.42:0/2850924493 shutdown_connections 2024-09-16T04:14:21.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.317+0000 7f5f880e8640 1 -- 172.21.15.42:0/2850924493 wait complete. 2024-09-16T04:14:21.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.317+0000 7f5f880e8640 10 monclient: build_initial_monmap 2024-09-16T04:14:21.334 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.317+0000 7f5f880e8640 10 monclient: monmap: 2024-09-16T04:14:21.334 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:14:21.334 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:14:21.334 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:14:21.335 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:14:21.335 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:14:21.335 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:21.335 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:14:21.335 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:14:21.335 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:14:21.335 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:21.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.317+0000 7f5f880e8640 1 librados: starting msgr at 2024-09-16T04:14:21.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.317+0000 7f5f880e8640 1 librados: starting objecter 2024-09-16T04:14:21.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.317+0000 7f5f880e8640 1 Processor -- start 2024-09-16T04:14:21.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.317+0000 7f5f880e8640 1 -- start start 2024-09-16T04:14:21.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.317+0000 7f5f880e8640 1 librados: setting wanted keys 2024-09-16T04:14:21.335 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.317+0000 7f5f880e8640 1 librados: calling monclient init 2024-09-16T04:14:21.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.317+0000 7f5f880e8640 10 monclient: init 2024-09-16T04:14:21.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.321+0000 7f5f880e8640 10 monclient: _reopen_session rank -1 2024-09-16T04:14:21.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.321+0000 7f5f880e8640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:14:21.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.321+0000 7f5f880e8640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5f8006db90 0x7f5f801cf5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:21.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.321+0000 7f5f880e8640 10 monclient(hunting): picked mon.noname-a con 0x7f5f8006db90 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:14:21.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.321+0000 7f5f880e8640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5f800a37f0 0x7f5f8013e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:21.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.321+0000 7f5f880e8640 10 monclient(hunting): picked mon.noname-c con 0x7f5f800a37f0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:14:21.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.321+0000 7f5f880e8640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5f80136e10 0x7f5f800a1870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:21.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.321+0000 7f5f880e8640 10 monclient(hunting): picked mon.noname-b con 0x7f5f80136e10 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:14:21.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.321+0000 7f5f880e8640 10 monclient(hunting): start opening mon connection 2024-09-16T04:14:21.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.321+0000 7f5f880e8640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f5f80141e40 con 0x7f5f8006db90 2024-09-16T04:14:21.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.321+0000 7f5f880e8640 10 monclient(hunting): start opening mon connection 2024-09-16T04:14:21.336 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.321+0000 7f5f880e8640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f5f80141cc0 con 0x7f5f80136e10 2024-09-16T04:14:21.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.321+0000 7f5f880e8640 10 monclient(hunting): start opening mon connection 2024-09-16T04:14:21.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.321+0000 7f5f880e8640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f5f80141fc0 con 0x7f5f800a37f0 2024-09-16T04:14:21.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.321+0000 7f5f880e8640 10 monclient(hunting): _renew_subs 2024-09-16T04:14:21.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.321+0000 7f5f880e8640 10 monclient(hunting): authenticate will time out at 2908.292315s 2024-09-16T04:14:21.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.321+0000 7f5f85683640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5f800a37f0 0x7f5f8013e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:21.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.321+0000 7f5f85e84640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5f8006db90 0x7f5f801cf5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:21.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.321+0000 7f5f85683640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5f800a37f0 0x7f5f8013e6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:51924/0 (socket says 172.21.15.42:51924) 2024-09-16T04:14:21.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.321+0000 7f5f85683640 1 -- 172.21.15.42:0/3287633244 learned_addr learned my addr 172.21.15.42:0/3287633244 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:14:21.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.321+0000 7f5f85683640 10 monclient(hunting): get_auth_request con 0x7f5f800a37f0 auth_method 0 2024-09-16T04:14:21.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.321+0000 7f5f85683640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:14:21.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.321+0000 7f5f85683640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:21.337 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.321+0000 7f5f85683640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:21.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.321+0000 7f5f85e84640 10 monclient(hunting): get_auth_request con 0x7f5f8006db90 auth_method 0 2024-09-16T04:14:21.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.321+0000 7f5f85e84640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:14:21.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.321+0000 7f5f85e84640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:21.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.321+0000 7f5f85e84640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:21.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.321+0000 7f5f85683640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:14:21.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.321+0000 7f5f85683640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:14:21.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.321+0000 7f5f85683640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:14:21.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.321+0000 7f5f85e84640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:14:21.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.321+0000 7f5f85e84640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:14:21.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.321+0000 7f5f85e84640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:14:21.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.321+0000 7f5f85683640 10 monclient(hunting): handle_auth_done global_id 1094246 payload 824 2024-09-16T04:14:21.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.321+0000 7f5f86685640 1 --2- 172.21.15.42:0/3287633244 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5f80136e10 0x7f5f800a1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:21.338 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.321+0000 7f5f85683640 1 -- 172.21.15.42:0/3287633244 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5f80136e10 msgr2=0x7f5f800a1870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:21.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.321+0000 7f5f85683640 1 --2- 172.21.15.42:0/3287633244 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5f80136e10 0x7f5f800a1870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:21.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.321+0000 7f5f85683640 1 -- 172.21.15.42:0/3287633244 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5f8006db90 msgr2=0x7f5f801cf5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:21.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.321+0000 7f5f85683640 1 --2- 172.21.15.42:0/3287633244 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5f8006db90 0x7f5f801cf5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:21.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.321+0000 7f5f85683640 10 monclient: _finish_hunting 0 2024-09-16T04:14:21.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.321+0000 7f5f85683640 1 monclient: found mon.noname-c 2024-09-16T04:14:21.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.321+0000 7f5f85683640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:14:21.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.321+0000 7f5f85683640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T04:14:21.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.321+0000 7f5f85683640 1 -- 172.21.15.42:0/3287633244 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f800a51b0 con 0x7f5f800a37f0 2024-09-16T04:14:21.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.321+0000 7f5f85e84640 1 --2- 172.21.15.42:0/3287633244 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5f8006db90 0x7f5f801cf5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:14:21.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.321+0000 7f5f85683640 1 --2- 172.21.15.42:0/3287633244 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5f800a37f0 0x7f5f8013e6e0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f5f740ae040 tx=0x7f5f74019180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:14:21.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.321+0000 7f5f7e7fc640 10 client.?.objecter ms_handle_connect 0x7f5f800a37f0 2024-09-16T04:14:21.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.321+0000 7f5f7e7fc640 10 client.?.objecter resend_mon_ops 2024-09-16T04:14:21.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.321+0000 7f5f7e7fc640 1 -- 172.21.15.42:0/3287633244 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f740b0040 con 0x7f5f800a37f0 2024-09-16T04:14:21.339 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.321+0000 7f5f7e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:21.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.321+0000 7f5f7e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:14:21.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.321+0000 7f5f7e7fc640 10 monclient: dump: 2024-09-16T04:14:21.340 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:14:21.340 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:14:21.340 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:21.340 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:21.340 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:21.340 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:21.340 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:14:21.340 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:14:21.340 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:14:21.340 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:21.340 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.321+0000 7f5f7e7fc640 10 monclient: _finish_auth 0 2024-09-16T04:14:21.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.321+0000 7f5f7e7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:14:21.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.321+0000 7f5f7e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:14:21.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.321+0000 7f5f7e7fc640 1 -- 172.21.15.42:0/3287633244 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5f740b9070 con 0x7f5f800a37f0 2024-09-16T04:14:21.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.321+0000 7f5f880e8640 5 monclient: authenticate success, global_id 1094246 2024-09-16T04:14:21.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.321+0000 7f5f880e8640 10 monclient: _renew_subs 2024-09-16T04:14:21.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.321+0000 7f5f880e8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:14:21.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.321+0000 7f5f880e8640 1 -- 172.21.15.42:0/3287633244 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f8006f6a0 con 0x7f5f800a37f0 2024-09-16T04:14:21.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.321+0000 7f5f7e7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:14:21.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.321+0000 7f5f7e7fc640 1 -- 172.21.15.42:0/3287633244 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f740b4680 con 0x7f5f800a37f0 2024-09-16T04:14:21.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.321+0000 7f5f7e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:21.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.321+0000 7f5f7e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:14:21.341 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.321+0000 7f5f7e7fc640 10 monclient: dump: 2024-09-16T04:14:21.341 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:14:21.341 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:14:21.342 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:21.342 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:21.342 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:21.342 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:21.342 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:14:21.342 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:14:21.342 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:14:21.342 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:21.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.321+0000 7f5f880e8640 10 client.1094246.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:14:21.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.321+0000 7f5f880e8640 10 monclient: _renew_subs 2024-09-16T04:14:21.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.321+0000 7f5f880e8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:14:21.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.321+0000 7f5f7f7fe640 4 set_mon_vals no callback set 2024-09-16T04:14:21.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.321+0000 7f5f880e8640 1 -- 172.21.15.42:0/3287633244 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f80139c70 con 0x7f5f800a37f0 2024-09-16T04:14:21.342 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.321+0000 7f5f880e8640 1 librados: init done 2024-09-16T04:14:21.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.321+0000 7f5f7e7fc640 1 -- 172.21.15.42:0/3287633244 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f5f740b4820 con 0x7f5f800a37f0 2024-09-16T04:14:21.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.321+0000 7f5f880e8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:14:21.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.321+0000 7f5f7e7fc640 1 --2- 172.21.15.42:0/3287633244 >> v2:172.21.15.154:6812/254845342 conn(0x7f5f6c037840 0x7f5f6c039d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:21.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.321+0000 7f5f880e8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:14:21.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.321+0000 7f5f880e8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:14:21.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.321+0000 7f5f880e8640 1 -- 172.21.15.42:0/3287633244 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5f801340c0 con 0x7f5f800a37f0 2024-09-16T04:14:21.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.321+0000 7f5f7e7fc640 1 -- 172.21.15.42:0/3287633244 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(710..710 src has 681..710) ==== 4681+0+0 (secure 0 0 0) 0x7f5f740e2870 con 0x7f5f800a37f0 2024-09-16T04:14:21.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.321+0000 7f5f7e7fc640 10 client.1094246.objecter ms_dispatch 0x7f5f80000ba0 osd_map(710..710 src has 681..710) 2024-09-16T04:14:21.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.321+0000 7f5f7e7fc640 3 client.1094246.objecter handle_osd_map got epochs [710,710] > 0 2024-09-16T04:14:21.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.321+0000 7f5f7e7fc640 3 client.1094246.objecter handle_osd_map decoding full epoch 710 2024-09-16T04:14:21.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.321+0000 7f5f7e7fc640 20 client.1094246.objecter dump_active .. 0 homeless 2024-09-16T04:14:21.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.321+0000 7f5f85e84640 1 --2- 172.21.15.42:0/3287633244 >> v2:172.21.15.154:6812/254845342 conn(0x7f5f6c037840 0x7f5f6c039d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:21.343 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.321+0000 7f5f85e84640 10 monclient: get_auth_request con 0x7f5f6c037840 auth_method 0 2024-09-16T04:14:21.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.321+0000 7f5f85e84640 1 --2- 172.21.15.42:0/3287633244 >> v2:172.21.15.154:6812/254845342 conn(0x7f5f6c037840 0x7f5f6c039d00 secure :-1 s=READY pgs=4014 cs=0 l=1 rev1=1 crypto rx=0x7f5f700077a0 tx=0x7f5f70009800 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:14:21.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.321+0000 7f5f7e7fc640 10 client.1094246.objecter ms_handle_connect 0x7f5f6c037840 2024-09-16T04:14:21.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.325+0000 7f5f7e7fc640 1 -- 172.21.15.42:0/3287633244 <== mon.2 v2:172.21.15.42:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f5f740160f0 con 0x7f5f800a37f0 2024-09-16T04:14:21.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.325+0000 7f5f7e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:14:21.344 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.325+0000 7f5f7e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:14:21.499 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.493+0000 7f5f880e8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-16T04:14:21.499 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.493+0000 7f5f880e8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T04:14:21.499 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.493+0000 7f5f880e8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:14:21.500 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.493+0000 7f5f880e8640 1 -- 172.21.15.42:0/3287633244 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5f801cfae0 con 0x7f5f800a37f0 2024-09-16T04:14:21.500 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.493+0000 7f5f7e7fc640 1 -- 172.21.15.42:0/3287633244 <== mon.2 v2:172.21.15.42:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v710) ==== 74+0+13949 (secure 0 0 0) 0x7f5f740b39f0 con 0x7f5f800a37f0 2024-09-16T04:14:21.500 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.493+0000 7f5f7e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-16T04:14:21.501 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.493+0000 7f5f7e7fc640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:14:21.501 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-09-16T04:14:21.501 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":710,"fsid":"15853af6-d0b2-472c-b2ac-8d7ad428641f","created":"2024-09-16T03:42:05.977277+0000","modified":"2024-09-16T04:13:51.289893+0000","last_up_change":"2024-09-16T03:42:17.544237+0000","last_in_change":"2024-09-16T03:42:08.437271+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":101,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":341,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T03:42:17.542498+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-16T03:42:23.677980+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":18,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"ceph_test_rados_api_asio","create_time":"2024-09-16T03:43:32.266113+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"183","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6b9fc058-2cff-46af-b95c-b78c72e1dc2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":708,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6808","nonce":1401549752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6809","nonce":1401549752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6811","nonce":1401549752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6810","nonce":1401549752}]},"public_addr":"172.21.15.42:6808/1401549752","cluster_addr":"172.21.15.42:6809/1401549752","heartbeat_back_addr":"172.21.15.42:6811/1401549752","heartbeat_front_addr":"172.21.15.42:6810/1401549752","state":["exists","up"]},{"osd":1,"uuid":"ee31f807-cb16-4c33-8e23-2573e1a6b5c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":708,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6800","nonce":4217863575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6801","nonce":4217863575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6803","nonce":4217863575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6802","nonce":4217863575}]},"public_addr":"172.21.15.42:6800/4217863575","cluster_addr":"172.21.15.42:6801/4217863575","heartbeat_back_addr":"172.21.15.42:6803/4217863575","heartbeat_front_addr":"172.21.15.42:6802/4217863575","state":["exists","up"]},{"osd":2,"uuid":"a26f3917-6702-43db-9355-e96da86a3365","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":708,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6804","nonce":263835041}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6805","nonce":263835041}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6807","nonce":263835041}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6806","nonce":263835041}]},"public_addr":"172.21.15.42:6804/263835041","cluster_addr":"172.21.15.42:6805/263835041","heartbeat_back_addr":"172.21.15.42:6807/263835041","heartbeat_front_addr":"172.21.15.42:6806/263835041","state":["exists","up"]},{"osd":3,"uuid":"4e809f2a-2ffd-4791-9f2d-18f1bfe7d2ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":708,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":4043102363}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6801","nonce":4043102363}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6803","nonce":4043102363}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":4043102363}]},"public_addr":"172.21.15.154:6800/4043102363","cluster_addr":"172.21.15.154:6801/4043102363","heartbeat_back_addr":"172.21.15.154:6803/4043102363","heartbeat_front_addr":"172.21.15.154:6802/4043102363","state":["exists","up"]},{"osd":4,"uuid":"dcf637ab-5f72-4ed2-a99b-69083f0b6b15","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":708,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":1038452759}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6805","nonce":1038452759}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6807","nonce":1038452759}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":1038452759}]},"public_addr":"172.21.15.154:6804/1038452759","cluster_addr":"172.21.15.154:6805/1038452759","heartbeat_back_addr":"172.21.15.154:6807/1038452759","heartbeat_front_addr":"172.21.15.154:6806/1038452759","state":["exists","up"]},{"osd":5,"uuid":"83c05e44-861b-4bb2-9657-19196e5ac1f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":708,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":3026127822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6809","nonce":3026127822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6811","nonce":3026127822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":3026127822}]},"public_addr":"172.21.15.154:6808/3026127822","cluster_addr":"172.21.15.154:6809/3026127822","heartbeat_back_addr":"172.21.15.154:6811/3026127822","heartbeat_front_addr":"172.21.15.154:6810/3026127822","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:42:15.086682+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:42:14.665142+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:42:14.860865+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:42:14.620446+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:42:15.163090+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T03:42:15.856205+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":3,"length":4}]},{"pool":7,"snaps":[{"begin":2,"length":3}]},{"pool":35,"snaps":[{"begin":1,"length":1}]},{"pool":75,"snaps":[{"begin":1,"length":1}]},{"pool":95,"snaps":[{"begin":4,"length":1},{"begin":7,"length":2}]},{"pool":106,"snaps":[{"begin":1,"length":1}]},{"pool":124,"snaps":[{"begin":1,"length":2}]},{"pool":136,"snaps":[{"begin":3,"length":1}]},{"pool":139,"snaps":[{"begin":2,"length":2}]},{"pool":148,"snaps":[{"begin":1,"length":4}]},{"pool":152,"snaps":[{"begin":2,"length":3}]},{"pool":156,"snaps":[{"begin":2,"length":1}]},{"pool":167,"snaps":[{"begin":2,"length":2}]},{"pool":172,"snaps":[{"begin":3,"length":4}]},{"pool":176,"snaps":[{"begin":2,"length":4}]},{"pool":188,"snaps":[{"begin":2,"length":1}]},{"pool":195,"snaps":[{"begin":2,"length":3}]},{"pool":206,"snaps":[{"begin":7,"length":1}]},{"pool":209,"snaps":[{"begin":4,"length":4}]},{"pool":229,"snaps":[{"begin":8,"length":1}]},{"pool":242,"snaps":[{"begin":10,"length":1}]},{"pool":247,"snaps":[{"begin":11,"length":2}]},{"pool":258,"snaps":[{"begin":14,"length":1}]},{"pool":321,"snaps":[{"begin":2,"length":1}]},{"pool":322,"snaps":[{"begin":3,"length":1}]},{"pool":325,"snaps":[{"begin":4,"length":1}]},{"pool":329,"snaps":[{"begin":6,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T04:14:21.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.497+0000 7f5f880e8640 10 librados: watch_flush enter 2024-09-16T04:14:21.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.497+0000 7f5f880e8640 10 librados: watch_flush exit 2024-09-16T04:14:21.503 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.497+0000 7f5f880e8640 20 client.1094246.objecter shutdown clearing up homeless session... 2024-09-16T04:14:21.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.497+0000 7f5f880e8640 10 client.1094246.objecter successfully canceled tick 2024-09-16T04:14:21.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.497+0000 7f5f880e8640 1 -- 172.21.15.42:0/3287633244 >> v2:172.21.15.154:6812/254845342 conn(0x7f5f6c037840 msgr2=0x7f5f6c039d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:21.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.497+0000 7f5f880e8640 1 --2- 172.21.15.42:0/3287633244 >> v2:172.21.15.154:6812/254845342 conn(0x7f5f6c037840 0x7f5f6c039d00 secure :-1 s=READY pgs=4014 cs=0 l=1 rev1=1 crypto rx=0x7f5f700077a0 tx=0x7f5f70009800 comp rx=0 tx=0).stop 2024-09-16T04:14:21.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.497+0000 7f5f880e8640 10 monclient: shutdown 2024-09-16T04:14:21.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.497+0000 7f5f880e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:14:21.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.497+0000 7f5f880e8640 1 -- 172.21.15.42:0/3287633244 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5f800a37f0 msgr2=0x7f5f8013e6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:21.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.497+0000 7f5f880e8640 1 --2- 172.21.15.42:0/3287633244 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5f800a37f0 0x7f5f8013e6e0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f5f740ae040 tx=0x7f5f74019180 comp rx=0 tx=0).stop 2024-09-16T04:14:21.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.497+0000 7f5f880e8640 1 -- 172.21.15.42:0/3287633244 shutdown_connections 2024-09-16T04:14:21.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.497+0000 7f5f880e8640 1 --2- 172.21.15.42:0/3287633244 >> v2:172.21.15.154:6812/254845342 conn(0x7f5f6c037840 0x7f5f6c039d00 unknown :-1 s=CLOSED pgs=4014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:21.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.497+0000 7f5f880e8640 1 --2- 172.21.15.42:0/3287633244 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5f80136e10 0x7f5f800a1870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:21.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.497+0000 7f5f880e8640 1 --2- 172.21.15.42:0/3287633244 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5f800a37f0 0x7f5f8013e6e0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:21.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.497+0000 7f5f880e8640 1 --2- 172.21.15.42:0/3287633244 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5f8006db90 0x7f5f801cf5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:21.504 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.497+0000 7f5f880e8640 1 -- 172.21.15.42:0/3287633244 >> 172.21.15.42:0/3287633244 conn(0x7f5f800a9fb0 msgr2=0x7f5f800a82a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:14:21.505 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.497+0000 7f5f880e8640 1 -- 172.21.15.42:0/3287633244 shutdown_connections 2024-09-16T04:14:21.505 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.497+0000 7f5f880e8640 1 -- 172.21.15.42:0/3287633244 wait complete. 2024-09-16T04:14:21.505 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:21.497+0000 7f5f880e8640 1 librados: shutdown 2024-09-16T04:14:22.521 INFO:tasks.ceph:Scrubbing osd.0 2024-09-16T04:14:22.521 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-09-16T04:14:22.659 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.653+0000 7f068048c640 10 monclient: get_monmap_and_config 2024-09-16T04:14:22.659 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.653+0000 7f068048c640 10 monclient: build_initial_monmap 2024-09-16T04:14:22.659 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.653+0000 7f068048c640 10 monclient: monmap: 2024-09-16T04:14:22.659 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:14:22.659 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:14:22.659 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:14:22.657955+0000 2024-09-16T04:14:22.660 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:14:22.657955+0000 2024-09-16T04:14:22.660 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:14:22.660 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:22.660 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:14:22.660 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:14:22.660 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:14:22.660 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:14:22.660 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:14:22.660 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:22.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.653+0000 7f068048c640 1 Processor -- start 2024-09-16T04:14:22.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.653+0000 7f068048c640 1 -- start start 2024-09-16T04:14:22.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.653+0000 7f068048c640 10 monclient: init 2024-09-16T04:14:22.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.653+0000 7f068048c640 10 monclient: _reopen_session rank -1 2024-09-16T04:14:22.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.653+0000 7f068048c640 10 monclient: _add_conns ranks=[2,1,0,3,4] 2024-09-16T04:14:22.662 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.653+0000 7f068048c640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f06780a2df0 0x7f0678134d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:22.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.653+0000 7f068048c640 10 monclient(hunting): picked mon.noname-b con 0x7f06780a2df0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:14:22.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.653+0000 7f068048c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0678072da0 addr v1:172.21.15.154:6789/0 2024-09-16T04:14:22.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.653+0000 7f068048c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0678073180 addr v1:172.21.15.42:6789/0 2024-09-16T04:14:22.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.653+0000 7f068048c640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f06780a56e0 con 0x7f0678073180 2024-09-16T04:14:22.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.653+0000 7f068048c640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f067809fd60 con 0x7f0678072da0 2024-09-16T04:14:22.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.653+0000 7f068048c640 10 monclient(hunting): start opening mon connection 2024-09-16T04:14:22.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.653+0000 7f068048c640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f067809ed70 con 0x7f06780a2df0 2024-09-16T04:14:22.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.653+0000 7f068048c640 10 monclient(hunting): _renew_subs 2024-09-16T04:14:22.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.653+0000 7f068048c640 10 monclient(hunting): authenticate will time out at 2909.627928s 2024-09-16T04:14:22.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.657+0000 7f067e228640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f06780a2df0 0x7f0678134d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:22.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.657+0000 7f067e228640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f06780a2df0 0x7f0678134d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:51928/0 (socket says 172.21.15.42:51928) 2024-09-16T04:14:22.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.657+0000 7f067e228640 1 -- 172.21.15.42:0/1691230271 learned_addr learned my addr 172.21.15.42:0/1691230271 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:14:22.663 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.657+0000 7f067e228640 10 monclient(hunting): get_auth_request con 0x7f06780a2df0 auth_method 0 2024-09-16T04:14:22.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.657+0000 7f067e228640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:14:22.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.657+0000 7f067e228640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:22.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.657+0000 7f067e228640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:22.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.657+0000 7f067e228640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:14:22.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.657+0000 7f067e228640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:14:22.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.657+0000 7f067e228640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:14:22.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.657+0000 7f067e228640 10 monclient(hunting): handle_auth_done global_id 1094249 payload 274 2024-09-16T04:14:22.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.657+0000 7f067e228640 1 -- 172.21.15.42:0/1691230271 >> v1:172.21.15.154:6789/0 conn(0x7f0678072da0 legacy=0x7f06780a48c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:14:22.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.657+0000 7f067e228640 1 -- 172.21.15.42:0/1691230271 >> v1:172.21.15.42:6789/0 conn(0x7f0678073180 legacy=0x7f06780a4fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:22.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.657+0000 7f067e228640 10 monclient: _finish_hunting 0 2024-09-16T04:14:22.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.657+0000 7f067e228640 1 monclient: found mon.noname-b 2024-09-16T04:14:22.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.657+0000 7f067e228640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:14:22.664 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.657+0000 7f067d226640 1 -- 172.21.15.42:0/1691230271 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2467992562 0 0) 0x7f06780a56e0 con 0x7f0678073180 2024-09-16T04:14:22.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.657+0000 7f067e228640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T04:14:22.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.657+0000 7f067e228640 1 -- 172.21.15.42:0/1691230271 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f067809f7a0 con 0x7f06780a2df0 2024-09-16T04:14:22.665 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.657+0000 7f067d226640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T04:14:22.671 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.657+0000 7f067e228640 1 --2- 172.21.15.42:0/1691230271 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f06780a2df0 0x7f0678134d00 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f06680098a0 tx=0x7f066801c400 comp rx=0 tx=0).ready entity=mon.2 client_cookie=786f42d83f5fdfdd server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:14:22.672 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.657+0000 7f067d226640 1 -- 172.21.15.42:0/1691230271 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06680b1070 con 0x7f06780a2df0 2024-09-16T04:14:22.672 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.657+0000 7f067d226640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:22.672 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.657+0000 7f067d226640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:14:22.672 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.657+0000 7f067d226640 10 monclient: dump: 2024-09-16T04:14:22.672 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:14:22.672 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:14:22.672 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:22.672 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:22.672 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:22.673 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:22.673 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:14:22.673 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:14:22.673 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:14:22.673 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:22.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.657+0000 7f067d226640 10 monclient: _finish_auth 0 2024-09-16T04:14:22.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.657+0000 7f067d226640 10 monclient: _check_auth_tickets 2024-09-16T04:14:22.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.657+0000 7f067d226640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:14:22.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.657+0000 7f068048c640 5 monclient: authenticate success, global_id 1094249 2024-09-16T04:14:22.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.657+0000 7f068048c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:14:22.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.657+0000 7f067d226640 1 -- 172.21.15.42:0/1691230271 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f06680ac640 con 0x7f06780a2df0 2024-09-16T04:14:22.673 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.657+0000 7f067d226640 10 monclient: handle_config config(0 keys) 2024-09-16T04:14:22.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.657+0000 7f067d226640 1 -- 172.21.15.42:0/1691230271 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06680aca10 con 0x7f06780a2df0 2024-09-16T04:14:22.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.657+0000 7f067d226640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:22.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.657+0000 7f067d226640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:14:22.674 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.657+0000 7f067d226640 10 monclient: dump: 2024-09-16T04:14:22.674 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:14:22.674 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:14:22.674 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:22.674 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:22.674 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:22.674 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:22.675 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:14:22.675 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:14:22.675 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:14:22.675 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:22.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.657+0000 7f068048c640 10 monclient: get_monmap_and_config success 2024-09-16T04:14:22.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.657+0000 7f068048c640 4 set_mon_vals no callback set 2024-09-16T04:14:22.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.661+0000 7f068048c640 10 monclient: shutdown 2024-09-16T04:14:22.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.661+0000 7f068048c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:14:22.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.661+0000 7f068048c640 1 -- 172.21.15.42:0/1691230271 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f06780a2df0 msgr2=0x7f0678134d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:22.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.661+0000 7f068048c640 1 --2- 172.21.15.42:0/1691230271 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f06780a2df0 0x7f0678134d00 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f06680098a0 tx=0x7f066801c400 comp rx=0 tx=0).stop 2024-09-16T04:14:22.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.661+0000 7f068048c640 1 -- 172.21.15.42:0/1691230271 shutdown_connections 2024-09-16T04:14:22.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.661+0000 7f068048c640 1 --2- 172.21.15.42:0/1691230271 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f06780a2df0 0x7f0678134d00 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:22.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.661+0000 7f068048c640 1 -- 172.21.15.42:0/1691230271 >> 172.21.15.42:0/1691230271 conn(0x7f06780a6300 msgr2=0x7f06780a24b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:14:22.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.661+0000 7f068048c640 1 -- 172.21.15.42:0/1691230271 shutdown_connections 2024-09-16T04:14:22.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.661+0000 7f068048c640 1 -- 172.21.15.42:0/1691230271 wait complete. 2024-09-16T04:14:22.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.661+0000 7f068048c640 10 monclient: build_initial_monmap 2024-09-16T04:14:22.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.661+0000 7f068048c640 10 monclient: monmap: 2024-09-16T04:14:22.676 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:14:22.676 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:14:22.676 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:14:22.676 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:14:22.676 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:14:22.676 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:22.676 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:14:22.676 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:14:22.676 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:14:22.676 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:22.676 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.661+0000 7f068048c640 1 librados: starting msgr at 2024-09-16T04:14:22.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.661+0000 7f068048c640 1 librados: starting objecter 2024-09-16T04:14:22.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.661+0000 7f068048c640 1 Processor -- start 2024-09-16T04:14:22.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.661+0000 7f068048c640 1 -- start start 2024-09-16T04:14:22.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.661+0000 7f068048c640 1 librados: setting wanted keys 2024-09-16T04:14:22.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.661+0000 7f068048c640 1 librados: calling monclient init 2024-09-16T04:14:22.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.661+0000 7f068048c640 10 monclient: init 2024-09-16T04:14:22.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.661+0000 7f068048c640 10 monclient: _reopen_session rank -1 2024-09-16T04:14:22.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.661+0000 7f068048c640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-16T04:14:22.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.661+0000 7f068048c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0678072da0 0x7f067812e380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:22.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.661+0000 7f068048c640 10 monclient(hunting): picked mon.noname-a con 0x7f0678072da0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:14:22.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.661+0000 7f068048c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0678073180 0x7f067812e8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:22.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.661+0000 7f068048c640 10 monclient(hunting): picked mon.noname-b con 0x7f0678073180 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:14:22.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.661+0000 7f068048c640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f06780a2df0 0x7f0678135d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:22.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.661+0000 7f068048c640 10 monclient(hunting): picked mon.noname-c con 0x7f06780a2df0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:14:22.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.661+0000 7f068048c640 10 monclient(hunting): start opening mon connection 2024-09-16T04:14:22.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.661+0000 7f068048c640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f0678140a50 con 0x7f0678072da0 2024-09-16T04:14:22.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.661+0000 7f068048c640 10 monclient(hunting): start opening mon connection 2024-09-16T04:14:22.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.661+0000 7f068048c640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f06780a31d0 con 0x7f0678073180 2024-09-16T04:14:22.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.661+0000 7f068048c640 10 monclient(hunting): start opening mon connection 2024-09-16T04:14:22.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.661+0000 7f068048c640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f0678140bd0 con 0x7f06780a2df0 2024-09-16T04:14:22.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.661+0000 7f068048c640 10 monclient(hunting): _renew_subs 2024-09-16T04:14:22.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.661+0000 7f068048c640 10 monclient(hunting): authenticate will time out at 2909.634289s 2024-09-16T04:14:22.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.661+0000 7f067e228640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0678072da0 0x7f067812e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:22.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.661+0000 7f067e228640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0678072da0 0x7f067812e380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:47968/0 (socket says 172.21.15.42:47968) 2024-09-16T04:14:22.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.661+0000 7f067e228640 1 -- 172.21.15.42:0/577447916 learned_addr learned my addr 172.21.15.42:0/577447916 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:14:22.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.661+0000 7f067e228640 10 monclient(hunting): get_auth_request con 0x7f0678072da0 auth_method 0 2024-09-16T04:14:22.678 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.661+0000 7f067e228640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:14:22.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.661+0000 7f067e228640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:22.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.661+0000 7f067ea29640 1 --2- 172.21.15.42:0/577447916 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f06780a2df0 0x7f0678135d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:22.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.661+0000 7f067e228640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:22.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.661+0000 7f067ea29640 10 monclient(hunting): get_auth_request con 0x7f06780a2df0 auth_method 0 2024-09-16T04:14:22.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.661+0000 7f067ea29640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:14:22.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.661+0000 7f067ea29640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:22.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.661+0000 7f067ea29640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:22.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.661+0000 7f067e228640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:14:22.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.661+0000 7f067e228640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:14:22.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.661+0000 7f067e228640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:14:22.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.661+0000 7f067ea29640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:14:22.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.661+0000 7f067ea29640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:14:22.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.661+0000 7f067ea29640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:14:22.679 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.661+0000 7f067da27640 1 --2- 172.21.15.42:0/577447916 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0678073180 0x7f067812e8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:22.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.661+0000 7f067e228640 10 monclient(hunting): handle_auth_done global_id 1104171 payload 824 2024-09-16T04:14:22.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.661+0000 7f067e228640 1 -- 172.21.15.42:0/577447916 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f06780a2df0 msgr2=0x7f0678135d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:22.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.661+0000 7f067e228640 1 --2- 172.21.15.42:0/577447916 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f06780a2df0 0x7f0678135d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:22.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.661+0000 7f067e228640 1 -- 172.21.15.42:0/577447916 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0678073180 msgr2=0x7f067812e8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:22.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.661+0000 7f067e228640 1 --2- 172.21.15.42:0/577447916 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0678073180 0x7f067812e8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:22.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.661+0000 7f067e228640 10 monclient: _finish_hunting 0 2024-09-16T04:14:22.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.661+0000 7f067e228640 1 monclient: found mon.noname-a 2024-09-16T04:14:22.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.661+0000 7f067e228640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:14:22.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.661+0000 7f067e228640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:14:22.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.661+0000 7f067e228640 1 -- 172.21.15.42:0/577447916 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06781381e0 con 0x7f0678072da0 2024-09-16T04:14:22.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.661+0000 7f067da27640 10 monclient: get_auth_request con 0x7f0678073180 auth_method 0 2024-09-16T04:14:22.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.661+0000 7f067da27640 1 --2- 172.21.15.42:0/577447916 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f0678073180 0x7f067812e8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T04:14:22.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.661+0000 7f067e228640 1 --2- 172.21.15.42:0/577447916 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0678072da0 0x7f067812e380 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f06680af040 tx=0x7f06680191a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:14:22.680 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.661+0000 7f066effd640 10 client.?.objecter ms_handle_connect 0x7f0678072da0 2024-09-16T04:14:22.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.661+0000 7f066effd640 10 client.?.objecter resend_mon_ops 2024-09-16T04:14:22.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.661+0000 7f066effd640 1 -- 172.21.15.42:0/577447916 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06680bc070 con 0x7f0678072da0 2024-09-16T04:14:22.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.661+0000 7f066effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:22.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.661+0000 7f066effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:14:22.681 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.661+0000 7f066effd640 10 monclient: dump: 2024-09-16T04:14:22.681 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:14:22.681 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:14:22.681 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:22.681 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:22.681 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:22.681 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:22.681 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:14:22.681 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:14:22.681 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:14:22.682 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:22.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.661+0000 7f066effd640 10 monclient: _finish_auth 0 2024-09-16T04:14:22.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.661+0000 7f066effd640 10 monclient: _check_auth_tickets 2024-09-16T04:14:22.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.661+0000 7f066effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:14:22.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.661+0000 7f066effd640 1 -- 172.21.15.42:0/577447916 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0668004020 con 0x7f0678072da0 2024-09-16T04:14:22.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.661+0000 7f068048c640 5 monclient: authenticate success, global_id 1104171 2024-09-16T04:14:22.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.661+0000 7f068048c640 10 monclient: _renew_subs 2024-09-16T04:14:22.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.661+0000 7f068048c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:14:22.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.661+0000 7f068048c640 1 -- 172.21.15.42:0/577447916 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06780a2850 con 0x7f0678072da0 2024-09-16T04:14:22.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.661+0000 7f066effd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:14:22.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.661+0000 7f066effd640 1 -- 172.21.15.42:0/577447916 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06680b1040 con 0x7f0678072da0 2024-09-16T04:14:22.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.661+0000 7f066effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:22.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.661+0000 7f066effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:14:22.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.661+0000 7f066effd640 10 monclient: dump: 2024-09-16T04:14:22.683 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:14:22.683 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:14:22.683 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:22.683 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:22.683 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:22.683 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:22.683 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:14:22.683 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:14:22.683 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:14:22.683 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:22.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.661+0000 7f067ca25640 4 set_mon_vals no callback set 2024-09-16T04:14:22.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.661+0000 7f068048c640 10 client.1104171.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:14:22.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.661+0000 7f068048c640 10 monclient: _renew_subs 2024-09-16T04:14:22.683 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.661+0000 7f068048c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:14:22.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.661+0000 7f068048c640 1 -- 172.21.15.42:0/577447916 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0678139690 con 0x7f0678072da0 2024-09-16T04:14:22.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.661+0000 7f068048c640 1 librados: init done 2024-09-16T04:14:22.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.665+0000 7f066effd640 1 -- 172.21.15.42:0/577447916 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f0668003620 con 0x7f0678072da0 2024-09-16T04:14:22.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.665+0000 7f066effd640 1 --2- 172.21.15.42:0/577447916 >> v2:172.21.15.154:6812/254845342 conn(0x7f065c037810 0x7f065c039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:22.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.665+0000 7f066effd640 1 -- 172.21.15.42:0/577447916 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(710..710 src has 681..710) ==== 4681+0+0 (secure 0 0 0) 0x7f06680e3190 con 0x7f0678072da0 2024-09-16T04:14:22.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.665+0000 7f066effd640 10 client.1104171.objecter ms_dispatch 0x7f0678000ba0 osd_map(710..710 src has 681..710) 2024-09-16T04:14:22.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.665+0000 7f066effd640 3 client.1104171.objecter handle_osd_map got epochs [710,710] > 0 2024-09-16T04:14:22.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.665+0000 7f066effd640 3 client.1104171.objecter handle_osd_map decoding full epoch 710 2024-09-16T04:14:22.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.665+0000 7f066effd640 20 client.1104171.objecter dump_active .. 0 homeless 2024-09-16T04:14:22.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.665+0000 7f068048c640 10 client.1104171.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:14:22.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.665+0000 7f068048c640 15 client.1104171.objecter _session_command_op_assign -1 1 2024-09-16T04:14:22.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.665+0000 7f068048c640 1 --2- 172.21.15.42:0/577447916 >> v2:172.21.15.42:6808/1401549752 conn(0x7f0678136780 0x7f0678136b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:22.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.665+0000 7f068048c640 20 client.1104171.objecter _get_session s=0x7f0678134d00 osd=0 3 2024-09-16T04:14:22.684 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.665+0000 7f068048c640 20 client.1104171.objecter put_session s=0x7f0678134d00 osd=0 3 2024-09-16T04:14:22.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.665+0000 7f068048c640 20 client.1104171.objecter _get_session s=0x7f0678134d00 osd=0 3 2024-09-16T04:14:22.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.665+0000 7f068048c640 15 client.1104171.objecter _session_command_op_remove -1 1 2024-09-16T04:14:22.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.665+0000 7f068048c640 20 client.1104171.objecter get_session s=0x7f0678134d00 osd=0 3 2024-09-16T04:14:22.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.665+0000 7f068048c640 15 client.1104171.objecter _session_command_op_assign 0 1 2024-09-16T04:14:22.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.665+0000 7f068048c640 20 client.1104171.objecter put_session s=0x7f0678134d00 osd=0 4 2024-09-16T04:14:22.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.665+0000 7f068048c640 10 client.1104171.objecter _send_command 1 2024-09-16T04:14:22.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.665+0000 7f068048c640 1 -- 172.21.15.42:0/577447916 --> v2:172.21.15.42:6808/1401549752 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0678137f60 con 0x7f0678136780 2024-09-16T04:14:22.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.665+0000 7f067ea29640 1 --2- 172.21.15.42:0/577447916 >> v2:172.21.15.42:6808/1401549752 conn(0x7f0678136780 0x7f0678136b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:22.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.665+0000 7f067da27640 1 --2- 172.21.15.42:0/577447916 >> v2:172.21.15.154:6812/254845342 conn(0x7f065c037810 0x7f065c039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:22.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.665+0000 7f067da27640 10 monclient: get_auth_request con 0x7f065c037810 auth_method 0 2024-09-16T04:14:22.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.665+0000 7f067ea29640 10 monclient: get_auth_request con 0x7f0678136780 auth_method 0 2024-09-16T04:14:22.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.665+0000 7f067ea29640 1 --2- 172.21.15.42:0/577447916 >> v2:172.21.15.42:6808/1401549752 conn(0x7f0678136780 0x7f0678136b60 crc :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:14:22.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.665+0000 7f066effd640 10 client.1104171.objecter ms_handle_connect 0x7f0678136780 2024-09-16T04:14:22.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.665+0000 7f066effd640 10 client.1104171.objecter ms_handle_connect 0x7f065c037810 2024-09-16T04:14:22.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.665+0000 7f067da27640 1 --2- 172.21.15.42:0/577447916 >> v2:172.21.15.154:6812/254845342 conn(0x7f065c037810 0x7f065c039cd0 secure :-1 s=READY pgs=4015 cs=0 l=1 rev1=1 crypto rx=0x7f0678072d40 tx=0x7f0674009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:14:22.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.665+0000 7f066effd640 1 -- 172.21.15.42:0/577447916 <== osd.0 v2:172.21.15.42:6808/1401549752 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f0678137f60 con 0x7f0678136780 2024-09-16T04:14:22.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.665+0000 7f066effd640 10 client.1104171.objecter ms_dispatch 0x7f0678000ba0 command_reply(tid 1: 0 ) 2024-09-16T04:14:22.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.665+0000 7f066effd640 10 client.1104171.objecter _finish_command 1 = system:0 2024-09-16T04:14:22.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.665+0000 7f066effd640 20 client.1104171.objecter put_session s=0x7f0678134d00 osd=0 4 2024-09-16T04:14:22.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.665+0000 7f066effd640 15 client.1104171.objecter _session_command_op_remove 0 1 2024-09-16T04:14:22.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.677+0000 7f068048c640 10 client.1104171.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-09-16T04:14:22.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.677+0000 7f068048c640 15 client.1104171.objecter _session_command_op_assign -1 2 2024-09-16T04:14:22.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.677+0000 7f068048c640 20 client.1104171.objecter _get_session s=0x7f0678134d00 osd=0 3 2024-09-16T04:14:22.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.677+0000 7f068048c640 20 client.1104171.objecter put_session s=0x7f0678134d00 osd=0 3 2024-09-16T04:14:22.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.677+0000 7f068048c640 20 client.1104171.objecter _get_session s=0x7f0678134d00 osd=0 3 2024-09-16T04:14:22.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.677+0000 7f068048c640 15 client.1104171.objecter _session_command_op_remove -1 2 2024-09-16T04:14:22.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.677+0000 7f068048c640 20 client.1104171.objecter get_session s=0x7f0678134d00 osd=0 3 2024-09-16T04:14:22.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.677+0000 7f068048c640 15 client.1104171.objecter _session_command_op_assign 0 2 2024-09-16T04:14:22.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.677+0000 7f068048c640 20 client.1104171.objecter put_session s=0x7f0678134d00 osd=0 4 2024-09-16T04:14:22.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.677+0000 7f068048c640 10 client.1104171.objecter _send_command 2 2024-09-16T04:14:22.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.677+0000 7f068048c640 1 -- 172.21.15.42:0/577447916 --> v2:172.21.15.42:6808/1401549752 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f06781370a0 con 0x7f0678136780 2024-09-16T04:14:22.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.677+0000 7f066effd640 1 -- 172.21.15.42:0/577447916 <== osd.0 v2:172.21.15.42:6808/1401549752 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f06781370a0 con 0x7f0678136780 2024-09-16T04:14:22.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.677+0000 7f066effd640 10 client.1104171.objecter ms_dispatch 0x7f0678000ba0 command_reply(tid 2: 0 ) 2024-09-16T04:14:22.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.677+0000 7f066effd640 10 client.1104171.objecter _finish_command 2 = system:0 2024-09-16T04:14:22.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.677+0000 7f066effd640 20 client.1104171.objecter put_session s=0x7f0678134d00 osd=0 4 2024-09-16T04:14:22.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.677+0000 7f066effd640 15 client.1104171.objecter _session_command_op_remove 0 2 2024-09-16T04:14:22.688 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T04:14:22.688 INFO:teuthology.orchestra.run.smithi042.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T04:14:22.688 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T04:14:22.690 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.681+0000 7f068048c640 10 librados: watch_flush enter 2024-09-16T04:14:22.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.681+0000 7f068048c640 10 librados: watch_flush exit 2024-09-16T04:14:22.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.681+0000 7f068048c640 10 client.1104171.objecter close_session for osd.0 2024-09-16T04:14:22.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.681+0000 7f068048c640 1 -- 172.21.15.42:0/577447916 >> v2:172.21.15.42:6808/1401549752 conn(0x7f0678136780 msgr2=0x7f0678136b60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:22.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.681+0000 7f068048c640 1 --2- 172.21.15.42:0/577447916 >> v2:172.21.15.42:6808/1401549752 conn(0x7f0678136780 0x7f0678136b60 crc :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:22.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.681+0000 7f068048c640 20 client.1104171.objecter put_session s=0x7f0678134d00 osd=0 1 2024-09-16T04:14:22.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.681+0000 7f068048c640 20 client.1104171.objecter shutdown clearing up homeless session... 2024-09-16T04:14:22.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.681+0000 7f068048c640 10 client.1104171.objecter successfully canceled tick 2024-09-16T04:14:22.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.681+0000 7f068048c640 1 -- 172.21.15.42:0/577447916 >> v2:172.21.15.154:6812/254845342 conn(0x7f065c037810 msgr2=0x7f065c039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:22.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.681+0000 7f068048c640 1 --2- 172.21.15.42:0/577447916 >> v2:172.21.15.154:6812/254845342 conn(0x7f065c037810 0x7f065c039cd0 secure :-1 s=READY pgs=4015 cs=0 l=1 rev1=1 crypto rx=0x7f0678072d40 tx=0x7f0674009040 comp rx=0 tx=0).stop 2024-09-16T04:14:22.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.681+0000 7f068048c640 10 monclient: shutdown 2024-09-16T04:14:22.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.681+0000 7f068048c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:14:22.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.681+0000 7f068048c640 1 -- 172.21.15.42:0/577447916 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0678072da0 msgr2=0x7f067812e380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:22.691 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.681+0000 7f068048c640 1 --2- 172.21.15.42:0/577447916 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f0678072da0 0x7f067812e380 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f06680af040 tx=0x7f06680191a0 comp rx=0 tx=0).stop 2024-09-16T04:14:22.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.681+0000 7f067ea29640 1 -- 172.21.15.42:0/577447916 reap_dead start 2024-09-16T04:14:22.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.681+0000 7f068048c640 1 -- 172.21.15.42:0/577447916 shutdown_connections 2024-09-16T04:14:22.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.681+0000 7f068048c640 1 -- 172.21.15.42:0/577447916 >> 172.21.15.42:0/577447916 conn(0x7f06780a6300 msgr2=0x7f0678133d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:14:22.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.681+0000 7f068048c640 1 -- 172.21.15.42:0/577447916 shutdown_connections 2024-09-16T04:14:22.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.681+0000 7f068048c640 1 -- 172.21.15.42:0/577447916 wait complete. 2024-09-16T04:14:22.692 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.681+0000 7f068048c640 1 librados: shutdown 2024-09-16T04:14:22.703 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-09-16T04:14:22.817 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.809+0000 7f5d55597640 10 monclient: get_monmap_and_config 2024-09-16T04:14:22.817 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.809+0000 7f5d55597640 10 monclient: build_initial_monmap 2024-09-16T04:14:22.817 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.809+0000 7f5d55597640 10 monclient: monmap: 2024-09-16T04:14:22.817 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:14:22.817 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:14:22.817 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:14:22.815709+0000 2024-09-16T04:14:22.817 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:14:22.815709+0000 2024-09-16T04:14:22.817 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:14:22.817 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:22.817 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:14:22.818 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:14:22.818 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:14:22.818 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:14:22.818 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:14:22.818 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:22.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.813+0000 7f5d55597640 1 Processor -- start 2024-09-16T04:14:22.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.813+0000 7f5d55597640 1 -- start start 2024-09-16T04:14:22.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.813+0000 7f5d55597640 10 monclient: init 2024-09-16T04:14:22.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.813+0000 7f5d55597640 10 monclient: _reopen_session rank -1 2024-09-16T04:14:22.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.813+0000 7f5d55597640 10 monclient: _add_conns ranks=[0,1,3,2,4] 2024-09-16T04:14:22.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.813+0000 7f5d55597640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5d5009fb00 addr v1:172.21.15.42:6789/0 2024-09-16T04:14:22.820 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.813+0000 7f5d55597640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5d5009c160 addr v1:172.21.15.154:6789/0 2024-09-16T04:14:22.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.813+0000 7f5d55597640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f5d5009c5d0 0x7f5d5013c850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:22.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.813+0000 7f5d55597640 10 monclient(hunting): picked mon.noname-a con 0x7f5d5009c5d0 addr v2:172.21.15.42:3300/0 2024-09-16T04:14:22.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.813+0000 7f5d55597640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5d500a1e20 con 0x7f5d5009fb00 2024-09-16T04:14:22.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.813+0000 7f5d55597640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5d5009ea30 con 0x7f5d5009c160 2024-09-16T04:14:22.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.813+0000 7f5d55597640 10 monclient(hunting): start opening mon connection 2024-09-16T04:14:22.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.813+0000 7f5d55597640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f5d5009e250 con 0x7f5d5009c5d0 2024-09-16T04:14:22.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.813+0000 7f5d55597640 10 monclient(hunting): _renew_subs 2024-09-16T04:14:22.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.813+0000 7f5d55597640 10 monclient(hunting): authenticate will time out at 2909.785229s 2024-09-16T04:14:22.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.813+0000 7f5d4f7fe640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f5d5009c5d0 0x7f5d5013c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:22.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.813+0000 7f5d4f7fe640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f5d5009c5d0 0x7f5d5013c850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:47976/0 (socket says 172.21.15.42:47976) 2024-09-16T04:14:22.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.813+0000 7f5d4f7fe640 1 -- 172.21.15.42:0/4263995807 learned_addr learned my addr 172.21.15.42:0/4263995807 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:14:22.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.813+0000 7f5d4f7fe640 10 monclient(hunting): get_auth_request con 0x7f5d5009c5d0 auth_method 0 2024-09-16T04:14:22.821 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.813+0000 7f5d4f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:14:22.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.813+0000 7f5d4f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:22.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.813+0000 7f5d4f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:22.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.813+0000 7f5d4f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:14:22.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.813+0000 7f5d4f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:14:22.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.813+0000 7f5d4f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:14:22.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.813+0000 7f5d4f7fe640 10 monclient(hunting): handle_auth_done global_id 1104174 payload 274 2024-09-16T04:14:22.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.813+0000 7f5d4f7fe640 1 -- 172.21.15.42:0/4263995807 >> v1:172.21.15.154:6789/0 conn(0x7f5d5009c160 legacy=0x7f5d50139120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:14:22.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.813+0000 7f5d4f7fe640 1 -- 172.21.15.42:0/4263995807 >> v1:172.21.15.42:6789/0 conn(0x7f5d5009fb00 legacy=0x7f5d5009ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:22.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.813+0000 7f5d4f7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:14:22.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.813+0000 7f5d4f7fe640 1 monclient: found mon.noname-a 2024-09-16T04:14:22.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.813+0000 7f5d4f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:14:22.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.813+0000 7f5d4f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:14:22.822 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.813+0000 7f5d4f7fe640 1 -- 172.21.15.42:0/4263995807 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d5009fee0 con 0x7f5d5009c5d0 2024-09-16T04:14:22.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.813+0000 7f5d4f7fe640 1 --2- 172.21.15.42:0/4263995807 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5d5009c5d0 0x7f5d5013c850 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f5d5009e3d0 tx=0x7f5d4801eaa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6359429165096f1 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:14:22.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.817+0000 7f5d4e7fc640 1 -- 172.21.15.42:0/4263995807 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d4800a2d0 con 0x7f5d5009c5d0 2024-09-16T04:14:22.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.817+0000 7f5d4e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:22.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.817+0000 7f5d4e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:14:22.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.817+0000 7f5d4e7fc640 10 monclient: dump: 2024-09-16T04:14:22.836 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:14:22.836 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:14:22.836 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:22.836 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:22.836 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:22.836 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:22.836 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:14:22.837 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:14:22.837 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:14:22.837 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:22.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.817+0000 7f5d4e7fc640 10 monclient: _finish_auth 0 2024-09-16T04:14:22.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.817+0000 7f5d4e7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:14:22.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.817+0000 7f5d4e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:14:22.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.817+0000 7f5d4e7fc640 1 -- 172.21.15.42:0/4263995807 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5d480a78f0 con 0x7f5d5009c5d0 2024-09-16T04:14:22.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.817+0000 7f5d4e7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:14:22.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.817+0000 7f5d4e7fc640 1 -- 172.21.15.42:0/4263995807 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d480a7cc0 con 0x7f5d5009c5d0 2024-09-16T04:14:22.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.817+0000 7f5d55597640 5 monclient: authenticate success, global_id 1104174 2024-09-16T04:14:22.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.817+0000 7f5d55597640 10 monclient: get_monmap_and_config success 2024-09-16T04:14:22.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.817+0000 7f5d55597640 4 set_mon_vals no callback set 2024-09-16T04:14:22.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.817+0000 7f5d4e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:22.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.817+0000 7f5d4e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:14:22.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.817+0000 7f5d4e7fc640 10 monclient: dump: 2024-09-16T04:14:22.838 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:14:22.838 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:14:22.838 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:22.838 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:22.838 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:22.838 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:22.838 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:14:22.838 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:14:22.838 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:14:22.838 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:22.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.817+0000 7f5d55597640 10 monclient: shutdown 2024-09-16T04:14:22.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.817+0000 7f5d55597640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:14:22.838 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.817+0000 7f5d55597640 1 -- 172.21.15.42:0/4263995807 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5d5009c5d0 msgr2=0x7f5d5013c850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:22.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.817+0000 7f5d55597640 1 --2- 172.21.15.42:0/4263995807 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5d5009c5d0 0x7f5d5013c850 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f5d5009e3d0 tx=0x7f5d4801eaa0 comp rx=0 tx=0).stop 2024-09-16T04:14:22.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.817+0000 7f5d55597640 1 -- 172.21.15.42:0/4263995807 shutdown_connections 2024-09-16T04:14:22.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.817+0000 7f5d55597640 1 --2- 172.21.15.42:0/4263995807 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5d5009c5d0 0x7f5d5013c850 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:22.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.817+0000 7f5d55597640 1 -- 172.21.15.42:0/4263995807 >> 172.21.15.42:0/4263995807 conn(0x7f5d500a2420 msgr2=0x7f5d500a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:14:22.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.817+0000 7f5d55597640 1 -- 172.21.15.42:0/4263995807 shutdown_connections 2024-09-16T04:14:22.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.817+0000 7f5d55597640 1 -- 172.21.15.42:0/4263995807 wait complete. 2024-09-16T04:14:22.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.817+0000 7f5d55597640 10 monclient: build_initial_monmap 2024-09-16T04:14:22.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.817+0000 7f5d55597640 10 monclient: monmap: 2024-09-16T04:14:22.839 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:14:22.839 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:14:22.839 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:14:22.839 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:14:22.839 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:14:22.839 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:22.839 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:14:22.840 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:14:22.840 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:14:22.840 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:22.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.817+0000 7f5d55597640 1 librados: starting msgr at 2024-09-16T04:14:22.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.817+0000 7f5d55597640 1 librados: starting objecter 2024-09-16T04:14:22.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.817+0000 7f5d55597640 1 Processor -- start 2024-09-16T04:14:22.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.817+0000 7f5d55597640 1 -- start start 2024-09-16T04:14:22.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.817+0000 7f5d55597640 1 librados: setting wanted keys 2024-09-16T04:14:22.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.817+0000 7f5d55597640 1 librados: calling monclient init 2024-09-16T04:14:22.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.817+0000 7f5d55597640 10 monclient: init 2024-09-16T04:14:22.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.817+0000 7f5d55597640 10 monclient: _reopen_session rank -1 2024-09-16T04:14:22.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.817+0000 7f5d55597640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:14:22.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.817+0000 7f5d55597640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5d5009c160 0x7f5d50132b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:22.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.817+0000 7f5d55597640 10 monclient(hunting): picked mon.noname-b con 0x7f5d5009c160 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:14:22.841 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.817+0000 7f5d55597640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5d5009c5d0 0x7f5d50133060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:22.841 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.817+0000 7f5d55597640 10 monclient(hunting): picked mon.noname-c con 0x7f5d5009c5d0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:14:22.841 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.817+0000 7f5d55597640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5d5009fb00 0x7f5d50139500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:22.841 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.817+0000 7f5d55597640 10 monclient(hunting): picked mon.noname-a con 0x7f5d5009fb00 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:14:22.841 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.817+0000 7f5d55597640 10 monclient(hunting): start opening mon connection 2024-09-16T04:14:22.841 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.817+0000 7f5d55597640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f5d5009d380 con 0x7f5d5009fb00 2024-09-16T04:14:22.841 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.817+0000 7f5d55597640 10 monclient(hunting): start opening mon connection 2024-09-16T04:14:22.841 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.817+0000 7f5d55597640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f5d5009d500 con 0x7f5d5009c160 2024-09-16T04:14:22.841 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.817+0000 7f5d55597640 10 monclient(hunting): start opening mon connection 2024-09-16T04:14:22.841 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.817+0000 7f5d55597640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f5d5013eda0 con 0x7f5d5009c5d0 2024-09-16T04:14:22.841 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.817+0000 7f5d55597640 10 monclient(hunting): _renew_subs 2024-09-16T04:14:22.841 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.817+0000 7f5d55597640 10 monclient(hunting): authenticate will time out at 2909.791006s 2024-09-16T04:14:22.841 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.817+0000 7f5d4f7fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5d5009fb00 0x7f5d50139500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:22.841 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.817+0000 7f5d47fff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5d5009c5d0 0x7f5d50133060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:22.841 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.817+0000 7f5d4f7fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5d5009fb00 0x7f5d50139500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:47990/0 (socket says 172.21.15.42:47990) 2024-09-16T04:14:22.842 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.817+0000 7f5d47fff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5d5009c5d0 0x7f5d50133060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:51956/0 (socket says 172.21.15.42:51956) 2024-09-16T04:14:22.842 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.817+0000 7f5d4f7fe640 1 -- 172.21.15.42:0/686528513 learned_addr learned my addr 172.21.15.42:0/686528513 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:14:22.842 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.817+0000 7f5d4f7fe640 10 monclient(hunting): get_auth_request con 0x7f5d5009fb00 auth_method 0 2024-09-16T04:14:22.842 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.817+0000 7f5d4f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:14:22.842 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.817+0000 7f5d4f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:22.842 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.817+0000 7f5d4f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:22.842 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.817+0000 7f5d47fff640 10 monclient(hunting): get_auth_request con 0x7f5d5009c5d0 auth_method 0 2024-09-16T04:14:22.842 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.817+0000 7f5d47fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:14:22.842 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.817+0000 7f5d47fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:22.842 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.817+0000 7f5d47fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:22.842 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.817+0000 7f5d4f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:14:22.842 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.817+0000 7f5d4f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:14:22.842 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.817+0000 7f5d4f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:14:22.843 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.817+0000 7f5d47fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:14:22.843 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.817+0000 7f5d47fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:14:22.843 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.817+0000 7f5d47fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:14:22.843 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.817+0000 7f5d4effd640 1 --2- 172.21.15.42:0/686528513 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5d5009c160 0x7f5d50132b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:22.843 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.817+0000 7f5d4f7fe640 10 monclient(hunting): handle_auth_done global_id 1104177 payload 824 2024-09-16T04:14:22.843 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.817+0000 7f5d4f7fe640 1 -- 172.21.15.42:0/686528513 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5d5009c5d0 msgr2=0x7f5d50133060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:22.843 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.817+0000 7f5d4f7fe640 1 --2- 172.21.15.42:0/686528513 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5d5009c5d0 0x7f5d50133060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:22.843 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.817+0000 7f5d4f7fe640 1 -- 172.21.15.42:0/686528513 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5d5009c160 msgr2=0x7f5d50132b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:22.843 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.817+0000 7f5d4f7fe640 1 --2- 172.21.15.42:0/686528513 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5d5009c160 0x7f5d50132b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:22.843 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.817+0000 7f5d4f7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:14:22.843 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.817+0000 7f5d4f7fe640 1 monclient: found mon.noname-a 2024-09-16T04:14:22.843 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.817+0000 7f5d4f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:14:22.843 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.817+0000 7f5d4f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:14:22.843 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.817+0000 7f5d4f7fe640 1 -- 172.21.15.42:0/686528513 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d5006dd90 con 0x7f5d5009fb00 2024-09-16T04:14:22.843 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.817+0000 7f5d4effd640 10 monclient: get_auth_request con 0x7f5d5009c160 auth_method 0 2024-09-16T04:14:22.844 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.817+0000 7f5d4effd640 1 --2- 172.21.15.42:0/686528513 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5d5009c160 0x7f5d50132b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T04:14:22.844 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.817+0000 7f5d4f7fe640 1 --2- 172.21.15.42:0/686528513 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5d5009fb00 0x7f5d50139500 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f5d4801b6b0 tx=0x7f5d4801b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:14:22.844 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.817+0000 7f5d477fe640 10 client.?.objecter ms_handle_connect 0x7f5d5009fb00 2024-09-16T04:14:22.844 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.817+0000 7f5d477fe640 10 client.?.objecter resend_mon_ops 2024-09-16T04:14:22.844 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.817+0000 7f5d477fe640 1 -- 172.21.15.42:0/686528513 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d480be020 con 0x7f5d5009fb00 2024-09-16T04:14:22.844 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.817+0000 7f5d477fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:22.844 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.817+0000 7f5d477fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:14:22.844 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.817+0000 7f5d477fe640 10 monclient: dump: 2024-09-16T04:14:22.844 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:14:22.844 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:14:22.844 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:22.844 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:22.844 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:22.844 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:22.845 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:14:22.845 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:14:22.845 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:14:22.845 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:22.845 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.817+0000 7f5d477fe640 10 monclient: _finish_auth 0 2024-09-16T04:14:22.845 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.817+0000 7f5d477fe640 10 monclient: _check_auth_tickets 2024-09-16T04:14:22.845 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.817+0000 7f5d477fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:14:22.845 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.817+0000 7f5d477fe640 1 -- 172.21.15.42:0/686528513 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5d48008710 con 0x7f5d5009fb00 2024-09-16T04:14:22.845 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.817+0000 7f5d477fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:14:22.845 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.817+0000 7f5d477fe640 1 -- 172.21.15.42:0/686528513 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d48008ae0 con 0x7f5d5009fb00 2024-09-16T04:14:22.845 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.817+0000 7f5d55597640 5 monclient: authenticate success, global_id 1104177 2024-09-16T04:14:22.845 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.817+0000 7f5d477fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:22.845 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.817+0000 7f5d477fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:14:22.845 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.817+0000 7f5d477fe640 10 monclient: dump: 2024-09-16T04:14:22.846 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:14:22.846 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:14:22.846 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:22.846 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:22.846 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:22.846 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:22.846 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:14:22.846 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:14:22.846 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:14:22.846 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:22.846 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.817+0000 7f5d4dffb640 4 set_mon_vals no callback set 2024-09-16T04:14:22.846 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.821+0000 7f5d55597640 10 monclient: _renew_subs 2024-09-16T04:14:22.846 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.821+0000 7f5d55597640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:14:22.846 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.821+0000 7f5d55597640 1 -- 172.21.15.42:0/686528513 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d50073a30 con 0x7f5d5009fb00 2024-09-16T04:14:22.846 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.821+0000 7f5d55597640 10 client.1104177.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:14:22.847 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.821+0000 7f5d55597640 10 monclient: _renew_subs 2024-09-16T04:14:22.847 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.821+0000 7f5d55597640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:14:22.847 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.821+0000 7f5d55597640 1 -- 172.21.15.42:0/686528513 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d5011ca60 con 0x7f5d5009fb00 2024-09-16T04:14:22.847 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.821+0000 7f5d55597640 1 librados: init done 2024-09-16T04:14:22.847 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.821+0000 7f5d477fe640 1 -- 172.21.15.42:0/686528513 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f5d480c1020 con 0x7f5d5009fb00 2024-09-16T04:14:22.847 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.821+0000 7f5d55597640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:14:22.847 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.821+0000 7f5d55597640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:14:22.847 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.821+0000 7f5d55597640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:14:22.847 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.821+0000 7f5d55597640 1 -- 172.21.15.42:0/686528513 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d5009ba50 con 0x7f5d5009fb00 2024-09-16T04:14:22.847 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.821+0000 7f5d477fe640 1 --2- 172.21.15.42:0/686528513 >> v2:172.21.15.154:6812/254845342 conn(0x7f5d34037810 0x7f5d34039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:22.847 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.821+0000 7f5d477fe640 1 -- 172.21.15.42:0/686528513 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(710..710 src has 681..710) ==== 4681+0+0 (secure 0 0 0) 0x7f5d480e5d20 con 0x7f5d5009fb00 2024-09-16T04:14:22.847 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.821+0000 7f5d477fe640 10 client.1104177.objecter ms_dispatch 0x7f5d50000ba0 osd_map(710..710 src has 681..710) 2024-09-16T04:14:22.847 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.821+0000 7f5d477fe640 3 client.1104177.objecter handle_osd_map got epochs [710,710] > 0 2024-09-16T04:14:22.847 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.821+0000 7f5d477fe640 3 client.1104177.objecter handle_osd_map decoding full epoch 710 2024-09-16T04:14:22.848 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.821+0000 7f5d477fe640 20 client.1104177.objecter dump_active .. 0 homeless 2024-09-16T04:14:22.848 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.821+0000 7f5d4effd640 1 --2- 172.21.15.42:0/686528513 >> v2:172.21.15.154:6812/254845342 conn(0x7f5d34037810 0x7f5d34039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:22.848 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.821+0000 7f5d4effd640 10 monclient: get_auth_request con 0x7f5d34037810 auth_method 0 2024-09-16T04:14:22.848 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.821+0000 7f5d477fe640 10 client.1104177.objecter ms_handle_connect 0x7f5d34037810 2024-09-16T04:14:22.848 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.821+0000 7f5d4effd640 1 --2- 172.21.15.42:0/686528513 >> v2:172.21.15.154:6812/254845342 conn(0x7f5d34037810 0x7f5d34039cd0 secure :-1 s=READY pgs=4016 cs=0 l=1 rev1=1 crypto rx=0x7f5d400016f0 tx=0x7f5d40009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:14:22.848 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.825+0000 7f5d477fe640 1 -- 172.21.15.42:0/686528513 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f5d480ea070 con 0x7f5d5009fb00 2024-09-16T04:14:22.848 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.825+0000 7f5d477fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:14:22.848 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.825+0000 7f5d477fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:14:22.999 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.993+0000 7f5d55597640 1 -- 172.21.15.42:0/686528513 --> v2:172.21.15.154:6812/254845342 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) -- 0x7f5d5009d170 con 0x7f5d34037810 2024-09-16T04:14:23.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.993+0000 7f5d477fe640 1 -- 172.21.15.42:0/686528513 <== mgr.4102 v2:172.21.15.154:6812/254845342 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f5d5009d170 con 0x7f5d34037810 2024-09-16T04:14:23.000 INFO:teuthology.orchestra.run.smithi042.stderr:instructed osd(s) 0 to deep-scrub 2024-09-16T04:14:23.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.997+0000 7f5d55597640 10 librados: watch_flush enter 2024-09-16T04:14:23.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.997+0000 7f5d55597640 10 librados: watch_flush exit 2024-09-16T04:14:23.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.997+0000 7f5d55597640 20 client.1104177.objecter shutdown clearing up homeless session... 2024-09-16T04:14:23.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.997+0000 7f5d55597640 10 client.1104177.objecter successfully canceled tick 2024-09-16T04:14:23.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.997+0000 7f5d55597640 1 -- 172.21.15.42:0/686528513 >> v2:172.21.15.154:6812/254845342 conn(0x7f5d34037810 msgr2=0x7f5d34039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:23.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.997+0000 7f5d55597640 1 --2- 172.21.15.42:0/686528513 >> v2:172.21.15.154:6812/254845342 conn(0x7f5d34037810 0x7f5d34039cd0 secure :-1 s=READY pgs=4016 cs=0 l=1 rev1=1 crypto rx=0x7f5d400016f0 tx=0x7f5d40009040 comp rx=0 tx=0).stop 2024-09-16T04:14:23.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.997+0000 7f5d55597640 10 monclient: shutdown 2024-09-16T04:14:23.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.997+0000 7f5d55597640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:14:23.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.997+0000 7f5d55597640 1 -- 172.21.15.42:0/686528513 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5d5009fb00 msgr2=0x7f5d50139500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:23.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.997+0000 7f5d55597640 1 --2- 172.21.15.42:0/686528513 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5d5009fb00 0x7f5d50139500 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f5d4801b6b0 tx=0x7f5d4801b840 comp rx=0 tx=0).stop 2024-09-16T04:14:23.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.997+0000 7f5d55597640 1 -- 172.21.15.42:0/686528513 shutdown_connections 2024-09-16T04:14:23.003 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.997+0000 7f5d55597640 1 --2- 172.21.15.42:0/686528513 >> v2:172.21.15.154:6812/254845342 conn(0x7f5d34037810 0x7f5d34039cd0 unknown :-1 s=CLOSED pgs=4016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:23.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.997+0000 7f5d55597640 1 --2- 172.21.15.42:0/686528513 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f5d5009fb00 0x7f5d50139500 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:23.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.997+0000 7f5d55597640 1 --2- 172.21.15.42:0/686528513 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f5d5009c5d0 0x7f5d50133060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:23.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.997+0000 7f5d55597640 1 --2- 172.21.15.42:0/686528513 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f5d5009c160 0x7f5d50132b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:23.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.997+0000 7f5d55597640 1 -- 172.21.15.42:0/686528513 >> 172.21.15.42:0/686528513 conn(0x7f5d500a2420 msgr2=0x7f5d50134320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:14:23.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.997+0000 7f5d55597640 1 -- 172.21.15.42:0/686528513 shutdown_connections 2024-09-16T04:14:23.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.997+0000 7f5d55597640 1 -- 172.21.15.42:0/686528513 wait complete. 2024-09-16T04:14:23.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:22.997+0000 7f5d55597640 1 librados: shutdown 2024-09-16T04:14:23.020 INFO:tasks.ceph:Scrubbing osd.1 2024-09-16T04:14:23.020 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-09-16T04:14:23.140 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.133+0000 7f432d21e640 10 monclient: get_monmap_and_config 2024-09-16T04:14:23.140 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.133+0000 7f432d21e640 10 monclient: build_initial_monmap 2024-09-16T04:14:23.141 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.133+0000 7f432d21e640 10 monclient: monmap: 2024-09-16T04:14:23.141 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:14:23.141 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:14:23.141 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:14:23.139331+0000 2024-09-16T04:14:23.141 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:14:23.139331+0000 2024-09-16T04:14:23.141 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:14:23.141 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:23.141 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:14:23.141 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:14:23.141 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:14:23.141 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:14:23.141 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:14:23.141 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:23.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.133+0000 7f432d21e640 1 Processor -- start 2024-09-16T04:14:23.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.137+0000 7f432d21e640 1 -- start start 2024-09-16T04:14:23.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.137+0000 7f432d21e640 10 monclient: init 2024-09-16T04:14:23.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.137+0000 7f432d21e640 10 monclient: _reopen_session rank -1 2024-09-16T04:14:23.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.137+0000 7f432d21e640 10 monclient: _add_conns ranks=[4,1,3,2,0] 2024-09-16T04:14:23.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.137+0000 7f432d21e640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f43280a2df0 0x7f4328134d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:23.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.137+0000 7f432d21e640 10 monclient(hunting): picked mon.noname-c con 0x7f43280a2df0 addr v2:172.21.15.154:3300/0 2024-09-16T04:14:23.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.137+0000 7f432d21e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4328072da0 addr v1:172.21.15.154:6789/0 2024-09-16T04:14:23.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.137+0000 7f432d21e640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f43280731b0 0x7f43280a4fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:23.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.137+0000 7f432d21e640 10 monclient(hunting): picked mon.noname-a con 0x7f43280731b0 addr v2:172.21.15.42:3300/0 2024-09-16T04:14:23.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.137+0000 7f432d21e640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f432809fd60 con 0x7f4328072da0 2024-09-16T04:14:23.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.137+0000 7f432d21e640 10 monclient(hunting): start opening mon connection 2024-09-16T04:14:23.144 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.137+0000 7f432d21e640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f4328073600 con 0x7f43280731b0 2024-09-16T04:14:23.145 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.137+0000 7f432d21e640 10 monclient(hunting): start opening mon connection 2024-09-16T04:14:23.145 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.137+0000 7f432d21e640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f432809ed70 con 0x7f43280a2df0 2024-09-16T04:14:23.145 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.137+0000 7f432d21e640 10 monclient(hunting): _renew_subs 2024-09-16T04:14:23.145 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.137+0000 7f432d21e640 10 monclient(hunting): authenticate will time out at 2910.108655s 2024-09-16T04:14:23.145 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.137+0000 7f432759e640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f43280731b0 0x7f43280a4fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:23.145 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.137+0000 7f432759e640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f43280731b0 0x7f43280a4fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:47996/0 (socket says 172.21.15.42:47996) 2024-09-16T04:14:23.145 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.137+0000 7f432759e640 1 -- 172.21.15.42:0/13822528 learned_addr learned my addr 172.21.15.42:0/13822528 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:14:23.145 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.137+0000 7f432759e640 10 monclient(hunting): get_auth_request con 0x7f43280731b0 auth_method 0 2024-09-16T04:14:23.145 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.137+0000 7f432759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:14:23.145 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.137+0000 7f432759e640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:23.145 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.137+0000 7f432759e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:23.145 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.137+0000 7f432759e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:14:23.145 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.137+0000 7f432759e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:14:23.145 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.137+0000 7f432759e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:14:23.145 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.137+0000 7f4326d9d640 1 --2- 172.21.15.42:0/13822528 >> v2:172.21.15.154:3300/0 conn(0x7f43280a2df0 0x7f4328134d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:23.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.137+0000 7f4326d9d640 10 monclient(hunting): get_auth_request con 0x7f43280a2df0 auth_method 0 2024-09-16T04:14:23.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.137+0000 7f4326d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:14:23.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.137+0000 7f4326d9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:23.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.137+0000 7f4326d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:23.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.137+0000 7f432759e640 10 monclient(hunting): handle_auth_done global_id 1104180 payload 274 2024-09-16T04:14:23.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.137+0000 7f432759e640 1 -- 172.21.15.42:0/13822528 >> v2:172.21.15.154:3300/0 conn(0x7f43280a2df0 msgr2=0x7f4328134d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:14:23.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.137+0000 7f432759e640 1 --2- 172.21.15.42:0/13822528 >> v2:172.21.15.154:3300/0 conn(0x7f43280a2df0 0x7f4328134d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:23.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.137+0000 7f432759e640 1 -- 172.21.15.42:0/13822528 >> v1:172.21.15.154:6789/0 conn(0x7f4328072da0 legacy=0x7f43280a48c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:14:23.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.137+0000 7f432759e640 10 monclient: _finish_hunting 0 2024-09-16T04:14:23.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.137+0000 7f432759e640 1 monclient: found mon.noname-a 2024-09-16T04:14:23.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.137+0000 7f432759e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:14:23.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.137+0000 7f432759e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:14:23.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.137+0000 7f432759e640 1 -- 172.21.15.42:0/13822528 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f432809f7a0 con 0x7f43280731b0 2024-09-16T04:14:23.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.137+0000 7f432759e640 1 --2- 172.21.15.42:0/13822528 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f43280731b0 0x7f43280a4fd0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f43280a33c0 tx=0x7f432001c870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fdee93c47cbd085 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:14:23.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.137+0000 7f4325d9b640 1 -- 172.21.15.42:0/13822528 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43200a5640 con 0x7f43280731b0 2024-09-16T04:14:23.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.137+0000 7f4325d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:23.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.137+0000 7f4325d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:14:23.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.137+0000 7f4325d9b640 10 monclient: dump: 2024-09-16T04:14:23.154 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:14:23.154 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:14:23.154 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:23.154 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:23.154 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:23.154 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:23.154 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:14:23.154 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:14:23.154 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:14:23.155 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:23.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.137+0000 7f4325d9b640 10 monclient: _finish_auth 0 2024-09-16T04:14:23.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.137+0000 7f4325d9b640 10 monclient: _check_auth_tickets 2024-09-16T04:14:23.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.137+0000 7f4325d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:14:23.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.137+0000 7f432d21e640 5 monclient: authenticate success, global_id 1104180 2024-09-16T04:14:23.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.137+0000 7f4325d9b640 1 -- 172.21.15.42:0/13822528 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f43200a57e0 con 0x7f43280731b0 2024-09-16T04:14:23.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.137+0000 7f432d21e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:14:23.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.137+0000 7f4325d9b640 10 monclient: handle_config config(0 keys) 2024-09-16T04:14:23.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.137+0000 7f432d21e640 10 monclient: get_monmap_and_config success 2024-09-16T04:14:23.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.137+0000 7f432d21e640 4 set_mon_vals no callback set 2024-09-16T04:14:23.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.137+0000 7f4325d9b640 1 -- 172.21.15.42:0/13822528 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43200a5bb0 con 0x7f43280731b0 2024-09-16T04:14:23.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.137+0000 7f4325d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:23.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.137+0000 7f4325d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:14:23.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.137+0000 7f4325d9b640 10 monclient: dump: 2024-09-16T04:14:23.156 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:14:23.156 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:14:23.156 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:23.156 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:23.156 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:23.156 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:23.156 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:14:23.156 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:14:23.156 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:14:23.156 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:23.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.141+0000 7f432d21e640 10 monclient: shutdown 2024-09-16T04:14:23.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.141+0000 7f432d21e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:14:23.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.141+0000 7f432d21e640 1 -- 172.21.15.42:0/13822528 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f43280731b0 msgr2=0x7f43280a4fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:23.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.141+0000 7f432d21e640 1 --2- 172.21.15.42:0/13822528 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f43280731b0 0x7f43280a4fd0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f43280a33c0 tx=0x7f432001c870 comp rx=0 tx=0).stop 2024-09-16T04:14:23.157 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.141+0000 7f432d21e640 1 -- 172.21.15.42:0/13822528 shutdown_connections 2024-09-16T04:14:23.157 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.141+0000 7f432d21e640 1 --2- 172.21.15.42:0/13822528 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f43280731b0 0x7f43280a4fd0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:23.157 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.141+0000 7f432d21e640 1 --2- 172.21.15.42:0/13822528 >> v2:172.21.15.154:3300/0 conn(0x7f43280a2df0 0x7f4328134d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:23.157 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.141+0000 7f432d21e640 1 -- 172.21.15.42:0/13822528 >> 172.21.15.42:0/13822528 conn(0x7f43280a6300 msgr2=0x7f43280a24b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:14:23.157 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.141+0000 7f432d21e640 1 -- 172.21.15.42:0/13822528 shutdown_connections 2024-09-16T04:14:23.157 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.141+0000 7f432d21e640 1 -- 172.21.15.42:0/13822528 wait complete. 2024-09-16T04:14:23.157 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.141+0000 7f432d21e640 10 monclient: build_initial_monmap 2024-09-16T04:14:23.157 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.141+0000 7f432d21e640 10 monclient: monmap: 2024-09-16T04:14:23.157 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:14:23.157 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:14:23.157 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:14:23.157 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:14:23.157 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:14:23.157 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:23.157 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:14:23.158 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:14:23.158 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:14:23.158 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:23.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.141+0000 7f432d21e640 1 librados: starting msgr at 2024-09-16T04:14:23.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.141+0000 7f432d21e640 1 librados: starting objecter 2024-09-16T04:14:23.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.141+0000 7f432d21e640 1 Processor -- start 2024-09-16T04:14:23.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.141+0000 7f432d21e640 1 -- start start 2024-09-16T04:14:23.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.141+0000 7f432d21e640 1 librados: setting wanted keys 2024-09-16T04:14:23.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.141+0000 7f432d21e640 1 librados: calling monclient init 2024-09-16T04:14:23.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.141+0000 7f432d21e640 10 monclient: init 2024-09-16T04:14:23.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.141+0000 7f432d21e640 10 monclient: _reopen_session rank -1 2024-09-16T04:14:23.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.141+0000 7f432d21e640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:14:23.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.141+0000 7f432d21e640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4328072da0 0x7f432812e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:23.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.141+0000 7f432d21e640 10 monclient(hunting): picked mon.noname-b con 0x7f4328072da0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:14:23.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.141+0000 7f432d21e640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f43280731b0 0x7f432812e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:23.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.141+0000 7f432d21e640 10 monclient(hunting): picked mon.noname-c con 0x7f43280731b0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:14:23.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.141+0000 7f432d21e640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f43280a2df0 0x7f43281cef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:23.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.141+0000 7f432d21e640 10 monclient(hunting): picked mon.noname-a con 0x7f43280a2df0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:14:23.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.141+0000 7f432d21e640 10 monclient(hunting): start opening mon connection 2024-09-16T04:14:23.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.141+0000 7f432d21e640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f432813f9b0 con 0x7f43280a2df0 2024-09-16T04:14:23.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.141+0000 7f432d21e640 10 monclient(hunting): start opening mon connection 2024-09-16T04:14:23.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.141+0000 7f432d21e640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f432813f830 con 0x7f4328072da0 2024-09-16T04:14:23.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.141+0000 7f432d21e640 10 monclient(hunting): start opening mon connection 2024-09-16T04:14:23.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.141+0000 7f432d21e640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f432813fb30 con 0x7f43280731b0 2024-09-16T04:14:23.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.141+0000 7f432d21e640 10 monclient(hunting): _renew_subs 2024-09-16T04:14:23.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.141+0000 7f432d21e640 10 monclient(hunting): authenticate will time out at 2910.113970s 2024-09-16T04:14:23.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.141+0000 7f432759e640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f43280a2df0 0x7f43281cef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:23.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.141+0000 7f432759e640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f43280a2df0 0x7f43281cef00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:48006/0 (socket says 172.21.15.42:48006) 2024-09-16T04:14:23.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.141+0000 7f432759e640 1 -- 172.21.15.42:0/310344368 learned_addr learned my addr 172.21.15.42:0/310344368 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:14:23.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.141+0000 7f432759e640 10 monclient(hunting): get_auth_request con 0x7f43280a2df0 auth_method 0 2024-09-16T04:14:23.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.141+0000 7f432759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:14:23.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.141+0000 7f432759e640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:23.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.141+0000 7f432759e640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:23.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.141+0000 7f432659c640 1 --2- 172.21.15.42:0/310344368 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f43280731b0 0x7f432812e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:23.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.141+0000 7f432659c640 10 monclient(hunting): get_auth_request con 0x7f43280731b0 auth_method 0 2024-09-16T04:14:23.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.141+0000 7f432659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:14:23.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.141+0000 7f432659c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:23.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.141+0000 7f432659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:23.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.141+0000 7f432759e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:14:23.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.141+0000 7f432759e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:14:23.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.141+0000 7f432759e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:14:23.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.141+0000 7f432659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:14:23.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.141+0000 7f432659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:14:23.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.141+0000 7f432659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:14:23.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.141+0000 7f4326d9d640 1 --2- 172.21.15.42:0/310344368 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4328072da0 0x7f432812e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:23.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.141+0000 7f4326d9d640 10 monclient(hunting): get_auth_request con 0x7f4328072da0 auth_method 0 2024-09-16T04:14:23.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.141+0000 7f4326d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:14:23.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.141+0000 7f4326d9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:23.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.141+0000 7f4326d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:23.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.141+0000 7f432659c640 10 monclient(hunting): handle_auth_done global_id 1094258 payload 824 2024-09-16T04:14:23.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.141+0000 7f432659c640 1 -- 172.21.15.42:0/310344368 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4328072da0 msgr2=0x7f432812e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:23.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.141+0000 7f432659c640 1 --2- 172.21.15.42:0/310344368 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f4328072da0 0x7f432812e470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:23.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.141+0000 7f432659c640 1 -- 172.21.15.42:0/310344368 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f43280a2df0 msgr2=0x7f43281cef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:23.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.141+0000 7f432659c640 1 --2- 172.21.15.42:0/310344368 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f43280a2df0 0x7f43281cef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:23.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.141+0000 7f432659c640 10 monclient: _finish_hunting 0 2024-09-16T04:14:23.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.141+0000 7f432659c640 1 monclient: found mon.noname-c 2024-09-16T04:14:23.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.141+0000 7f432659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:14:23.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.141+0000 7f432659c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T04:14:23.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.141+0000 7f432659c640 1 -- 172.21.15.42:0/310344368 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f432813bc50 con 0x7f43280731b0 2024-09-16T04:14:23.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.141+0000 7f432759e640 1 --2- 172.21.15.42:0/310344368 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f43280a2df0 0x7f43281cef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:14:23.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.141+0000 7f432659c640 1 --2- 172.21.15.42:0/310344368 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f43280731b0 0x7f432812e9b0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f431000c410 tx=0x7f431000c8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:14:23.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.141+0000 7f431f7fe640 10 client.?.objecter ms_handle_connect 0x7f43280731b0 2024-09-16T04:14:23.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.141+0000 7f431f7fe640 10 client.?.objecter resend_mon_ops 2024-09-16T04:14:23.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.141+0000 7f431f7fe640 1 -- 172.21.15.42:0/310344368 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4310017070 con 0x7f43280731b0 2024-09-16T04:14:23.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.141+0000 7f431f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:23.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.141+0000 7f431f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:14:23.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.141+0000 7f431f7fe640 10 monclient: dump: 2024-09-16T04:14:23.164 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:14:23.164 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:14:23.164 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:23.164 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:23.164 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:23.164 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:23.164 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:14:23.164 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:14:23.164 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:14:23.164 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:23.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.141+0000 7f431f7fe640 10 monclient: _finish_auth 0 2024-09-16T04:14:23.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.141+0000 7f431f7fe640 10 monclient: _check_auth_tickets 2024-09-16T04:14:23.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.141+0000 7f431f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:14:23.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.141+0000 7f431f7fe640 1 -- 172.21.15.42:0/310344368 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f43100069e0 con 0x7f43280731b0 2024-09-16T04:14:23.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.141+0000 7f432d21e640 5 monclient: authenticate success, global_id 1094258 2024-09-16T04:14:23.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.141+0000 7f432d21e640 10 monclient: _renew_subs 2024-09-16T04:14:23.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.141+0000 7f432d21e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:14:23.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.141+0000 7f432d21e640 1 -- 172.21.15.42:0/310344368 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4328073aa0 con 0x7f43280731b0 2024-09-16T04:14:23.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.141+0000 7f431f7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:14:23.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.141+0000 7f431f7fe640 1 -- 172.21.15.42:0/310344368 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4310006e10 con 0x7f43280731b0 2024-09-16T04:14:23.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.141+0000 7f431f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:23.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.141+0000 7f431f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:14:23.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.141+0000 7f431f7fe640 10 monclient: dump: 2024-09-16T04:14:23.165 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:14:23.166 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:14:23.166 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:23.166 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:23.166 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:23.166 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:23.166 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:14:23.166 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:14:23.166 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:14:23.166 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:23.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.141+0000 7f432559a640 4 set_mon_vals no callback set 2024-09-16T04:14:23.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.141+0000 7f432d21e640 10 client.1094258.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:14:23.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.141+0000 7f432d21e640 10 monclient: _renew_subs 2024-09-16T04:14:23.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.141+0000 7f432d21e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:14:23.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.141+0000 7f432d21e640 1 -- 172.21.15.42:0/310344368 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f432813b340 con 0x7f43280731b0 2024-09-16T04:14:23.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.141+0000 7f432d21e640 1 librados: init done 2024-09-16T04:14:23.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.141+0000 7f432d21e640 10 client.1094258.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:14:23.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.141+0000 7f432d21e640 15 client.1094258.objecter _session_command_op_assign -1 1 2024-09-16T04:14:23.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.141+0000 7f432d21e640 20 client.1094258.objecter _get_session osd=-1 returning homeless 2024-09-16T04:14:23.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.141+0000 7f432d21e640 10 client.1094258.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:14:23.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.141+0000 7f432d21e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:14:23.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.141+0000 7f432d21e640 1 -- 172.21.15.42:0/310344368 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f4328134d00 con 0x7f43280731b0 2024-09-16T04:14:23.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.141+0000 7f431f7fe640 1 -- 172.21.15.42:0/310344368 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f431001a070 con 0x7f43280731b0 2024-09-16T04:14:23.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.141+0000 7f431f7fe640 1 --2- 172.21.15.42:0/310344368 >> v2:172.21.15.154:6812/254845342 conn(0x7f4308037810 0x7f4308039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:23.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.145+0000 7f431f7fe640 1 -- 172.21.15.42:0/310344368 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(710..710 src has 681..710) ==== 4681+0+0 (secure 0 0 0) 0x7f4310047ea0 con 0x7f43280731b0 2024-09-16T04:14:23.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.145+0000 7f431f7fe640 10 client.1094258.objecter ms_dispatch 0x7f4328000ba0 osd_map(710..710 src has 681..710) 2024-09-16T04:14:23.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.145+0000 7f431f7fe640 3 client.1094258.objecter handle_osd_map got epochs [710,710] > 0 2024-09-16T04:14:23.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.145+0000 7f431f7fe640 3 client.1094258.objecter handle_osd_map decoding full epoch 710 2024-09-16T04:14:23.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.145+0000 7f431f7fe640 10 client.1094258.objecter checking command 1 2024-09-16T04:14:23.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.145+0000 7f431f7fe640 1 --2- 172.21.15.42:0/310344368 >> v2:172.21.15.42:6800/4217863575 conn(0x7f430803ec70 0x7f4308041110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:23.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.145+0000 7f431f7fe640 20 client.1094258.objecter _get_session s=0x7f430803e5c0 osd=1 3 2024-09-16T04:14:23.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.145+0000 7f431f7fe640 20 client.1094258.objecter put_session s=0x7f430803e5c0 osd=1 3 2024-09-16T04:14:23.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.145+0000 7f431f7fe640 15 client.1094258.objecter _session_command_op_remove -1 1 2024-09-16T04:14:23.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.145+0000 7f431f7fe640 20 client.1094258.objecter _get_session s=0x7f430803e5c0 osd=1 3 2024-09-16T04:14:23.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.145+0000 7f431f7fe640 20 client.1094258.objecter get_session s=0x7f430803e5c0 osd=1 3 2024-09-16T04:14:23.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.145+0000 7f431f7fe640 15 client.1094258.objecter _session_command_op_assign 1 1 2024-09-16T04:14:23.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.145+0000 7f431f7fe640 20 client.1094258.objecter put_session s=0x7f430803e5c0 osd=1 4 2024-09-16T04:14:23.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.145+0000 7f431f7fe640 10 client.1094258.objecter _send_command 1 2024-09-16T04:14:23.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.145+0000 7f431f7fe640 1 -- 172.21.15.42:0/310344368 --> v2:172.21.15.42:6800/4217863575 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f43080417a0 con 0x7f430803ec70 2024-09-16T04:14:23.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.145+0000 7f431f7fe640 20 client.1094258.objecter dump_active .. 0 homeless 2024-09-16T04:14:23.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.145+0000 7f431f7fe640 1 -- 172.21.15.42:0/310344368 <== mon.2 v2:172.21.15.42:3301/0 6 ==== mon_get_version_reply(handle=1 version=710) ==== 24+0+0 (secure 0 0 0) 0x7f431004a570 con 0x7f43280731b0 2024-09-16T04:14:23.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.145+0000 7f431f7fe640 10 monclient: handle_get_version_reply finishing 1 version 710 2024-09-16T04:14:23.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.145+0000 7f432759e640 1 --2- 172.21.15.42:0/310344368 >> v2:172.21.15.42:6800/4217863575 conn(0x7f430803ec70 0x7f4308041110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:23.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.145+0000 7f4326d9d640 1 --2- 172.21.15.42:0/310344368 >> v2:172.21.15.154:6812/254845342 conn(0x7f4308037810 0x7f4308039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:23.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.145+0000 7f432759e640 10 monclient: get_auth_request con 0x7f430803ec70 auth_method 0 2024-09-16T04:14:23.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.145+0000 7f4326d9d640 10 monclient: get_auth_request con 0x7f4308037810 auth_method 0 2024-09-16T04:14:23.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.145+0000 7f431f7fe640 10 client.1094258.objecter ms_handle_connect 0x7f430803ec70 2024-09-16T04:14:23.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.145+0000 7f432759e640 1 --2- 172.21.15.42:0/310344368 >> v2:172.21.15.42:6800/4217863575 conn(0x7f430803ec70 0x7f4308041110 crc :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:14:23.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.145+0000 7f431f7fe640 10 client.1094258.objecter ms_handle_connect 0x7f4308037810 2024-09-16T04:14:23.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.145+0000 7f4326d9d640 1 --2- 172.21.15.42:0/310344368 >> v2:172.21.15.154:6812/254845342 conn(0x7f4308037810 0x7f4308039cd0 secure :-1 s=READY pgs=4017 cs=0 l=1 rev1=1 crypto rx=0x7f431800b9d0 tx=0x7f4318008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:14:23.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.145+0000 7f431f7fe640 1 -- 172.21.15.42:0/310344368 <== osd.1 v2:172.21.15.42:6800/4217863575 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f43080417a0 con 0x7f430803ec70 2024-09-16T04:14:23.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.145+0000 7f431f7fe640 10 client.1094258.objecter ms_dispatch 0x7f4328000ba0 command_reply(tid 1: 0 ) 2024-09-16T04:14:23.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.145+0000 7f431f7fe640 10 client.1094258.objecter _finish_command 1 = system:0 2024-09-16T04:14:23.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.145+0000 7f431f7fe640 20 client.1094258.objecter put_session s=0x7f430803e5c0 osd=1 4 2024-09-16T04:14:23.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.145+0000 7f431f7fe640 15 client.1094258.objecter _session_command_op_remove 1 1 2024-09-16T04:14:23.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.157+0000 7f432d21e640 10 client.1094258.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-09-16T04:14:23.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.157+0000 7f432d21e640 15 client.1094258.objecter _session_command_op_assign -1 2 2024-09-16T04:14:23.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.157+0000 7f432d21e640 20 client.1094258.objecter _get_session s=0x7f430803e5c0 osd=1 3 2024-09-16T04:14:23.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.157+0000 7f432d21e640 20 client.1094258.objecter put_session s=0x7f430803e5c0 osd=1 3 2024-09-16T04:14:23.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.157+0000 7f432d21e640 20 client.1094258.objecter _get_session s=0x7f430803e5c0 osd=1 3 2024-09-16T04:14:23.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.157+0000 7f432d21e640 15 client.1094258.objecter _session_command_op_remove -1 2 2024-09-16T04:14:23.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.157+0000 7f432d21e640 20 client.1094258.objecter get_session s=0x7f430803e5c0 osd=1 3 2024-09-16T04:14:23.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.157+0000 7f432d21e640 15 client.1094258.objecter _session_command_op_assign 1 2 2024-09-16T04:14:23.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.157+0000 7f432d21e640 20 client.1094258.objecter put_session s=0x7f430803e5c0 osd=1 4 2024-09-16T04:14:23.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.157+0000 7f432d21e640 10 client.1094258.objecter _send_command 2 2024-09-16T04:14:23.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.157+0000 7f432d21e640 1 -- 172.21.15.42:0/310344368 --> v2:172.21.15.42:6800/4217863575 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f4328069040 con 0x7f430803ec70 2024-09-16T04:14:23.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.157+0000 7f431f7fe640 1 -- 172.21.15.42:0/310344368 <== osd.1 v2:172.21.15.42:6800/4217863575 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f4328069040 con 0x7f430803ec70 2024-09-16T04:14:23.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.157+0000 7f431f7fe640 10 client.1094258.objecter ms_dispatch 0x7f4328000ba0 command_reply(tid 2: 0 ) 2024-09-16T04:14:23.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.157+0000 7f431f7fe640 10 client.1094258.objecter _finish_command 2 = system:0 2024-09-16T04:14:23.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.157+0000 7f431f7fe640 20 client.1094258.objecter put_session s=0x7f430803e5c0 osd=1 4 2024-09-16T04:14:23.171 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.157+0000 7f431f7fe640 15 client.1094258.objecter _session_command_op_remove 1 2 2024-09-16T04:14:23.171 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T04:14:23.171 INFO:teuthology.orchestra.run.smithi042.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T04:14:23.172 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T04:14:23.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.157+0000 7f432d21e640 10 librados: watch_flush enter 2024-09-16T04:14:23.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.157+0000 7f432d21e640 10 librados: watch_flush exit 2024-09-16T04:14:23.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.157+0000 7f432d21e640 10 client.1094258.objecter close_session for osd.1 2024-09-16T04:14:23.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.157+0000 7f432d21e640 1 -- 172.21.15.42:0/310344368 >> v2:172.21.15.42:6800/4217863575 conn(0x7f430803ec70 msgr2=0x7f4308041110 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:23.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.157+0000 7f432d21e640 1 --2- 172.21.15.42:0/310344368 >> v2:172.21.15.42:6800/4217863575 conn(0x7f430803ec70 0x7f4308041110 crc :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:23.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.157+0000 7f432d21e640 20 client.1094258.objecter put_session s=0x7f430803e5c0 osd=1 1 2024-09-16T04:14:23.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.157+0000 7f432d21e640 20 client.1094258.objecter shutdown clearing up homeless session... 2024-09-16T04:14:23.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.157+0000 7f432d21e640 10 client.1094258.objecter successfully canceled tick 2024-09-16T04:14:23.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.157+0000 7f432d21e640 1 -- 172.21.15.42:0/310344368 >> v2:172.21.15.154:6812/254845342 conn(0x7f4308037810 msgr2=0x7f4308039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:23.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.157+0000 7f432d21e640 1 --2- 172.21.15.42:0/310344368 >> v2:172.21.15.154:6812/254845342 conn(0x7f4308037810 0x7f4308039cd0 secure :-1 s=READY pgs=4017 cs=0 l=1 rev1=1 crypto rx=0x7f431800b9d0 tx=0x7f4318008040 comp rx=0 tx=0).stop 2024-09-16T04:14:23.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.157+0000 7f432d21e640 10 monclient: shutdown 2024-09-16T04:14:23.173 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.157+0000 7f432d21e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:14:23.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.157+0000 7f432d21e640 1 -- 172.21.15.42:0/310344368 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f43280731b0 msgr2=0x7f432812e9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:23.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.157+0000 7f432d21e640 1 --2- 172.21.15.42:0/310344368 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f43280731b0 0x7f432812e9b0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f431000c410 tx=0x7f431000c8d0 comp rx=0 tx=0).stop 2024-09-16T04:14:23.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.157+0000 7f432759e640 1 -- 172.21.15.42:0/310344368 reap_dead start 2024-09-16T04:14:23.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.157+0000 7f432d21e640 1 -- 172.21.15.42:0/310344368 shutdown_connections 2024-09-16T04:14:23.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.157+0000 7f432d21e640 1 -- 172.21.15.42:0/310344368 >> 172.21.15.42:0/310344368 conn(0x7f43280a6300 msgr2=0x7f43281330c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:14:23.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.157+0000 7f432d21e640 1 -- 172.21.15.42:0/310344368 shutdown_connections 2024-09-16T04:14:23.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.157+0000 7f432d21e640 1 -- 172.21.15.42:0/310344368 wait complete. 2024-09-16T04:14:23.174 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.157+0000 7f432d21e640 1 librados: shutdown 2024-09-16T04:14:23.178 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-09-16T04:14:23.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.281+0000 7f568ff9c640 10 monclient: get_monmap_and_config 2024-09-16T04:14:23.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.281+0000 7f568ff9c640 10 monclient: build_initial_monmap 2024-09-16T04:14:23.286 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.281+0000 7f568ff9c640 10 monclient: monmap: 2024-09-16T04:14:23.286 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:14:23.286 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:14:23.286 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:14:23.284922+0000 2024-09-16T04:14:23.286 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:14:23.284922+0000 2024-09-16T04:14:23.286 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:14:23.286 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:23.286 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:14:23.287 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:14:23.287 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:14:23.287 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:14:23.287 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:14:23.287 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:23.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.281+0000 7f568ff9c640 1 Processor -- start 2024-09-16T04:14:23.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.281+0000 7f568ff9c640 1 -- start start 2024-09-16T04:14:23.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.281+0000 7f568ff9c640 10 monclient: init 2024-09-16T04:14:23.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.281+0000 7f568ff9c640 10 monclient: _reopen_session rank -1 2024-09-16T04:14:23.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.281+0000 7f568ff9c640 10 monclient: _add_conns ranks=[3,1,0,2,4] 2024-09-16T04:14:23.289 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.281+0000 7f568ff9c640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f56880a1330 0x7f568809f180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:23.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.281+0000 7f568ff9c640 10 monclient(hunting): picked mon.noname-a con 0x7f56880a1330 addr v2:172.21.15.42:3300/0 2024-09-16T04:14:23.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.281+0000 7f568ff9c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f56880a1710 addr v1:172.21.15.154:6789/0 2024-09-16T04:14:23.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.281+0000 7f568ff9c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f56881393a0 addr v1:172.21.15.42:6789/0 2024-09-16T04:14:23.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.281+0000 7f568ff9c640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f568809e4c0 con 0x7f56881393a0 2024-09-16T04:14:23.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.281+0000 7f568ff9c640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f568809f6c0 con 0x7f56880a1710 2024-09-16T04:14:23.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.281+0000 7f568ff9c640 10 monclient(hunting): start opening mon connection 2024-09-16T04:14:23.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.281+0000 7f568ff9c640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f568809edf0 con 0x7f56880a1330 2024-09-16T04:14:23.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.281+0000 7f568ff9c640 10 monclient(hunting): _renew_subs 2024-09-16T04:14:23.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.281+0000 7f568ff9c640 10 monclient(hunting): authenticate will time out at 2910.254331s 2024-09-16T04:14:23.290 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.281+0000 7f568dd38640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f56880a1330 0x7f568809f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:23.291 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.281+0000 7f568dd38640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f56880a1330 0x7f568809f180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:48008/0 (socket says 172.21.15.42:48008) 2024-09-16T04:14:23.291 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.281+0000 7f568dd38640 1 -- 172.21.15.42:0/673445188 learned_addr learned my addr 172.21.15.42:0/673445188 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:14:23.291 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.281+0000 7f568dd38640 10 monclient(hunting): get_auth_request con 0x7f56880a1330 auth_method 0 2024-09-16T04:14:23.291 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.281+0000 7f568dd38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:14:23.291 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.281+0000 7f568dd38640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:23.291 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.281+0000 7f568dd38640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:23.291 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.281+0000 7f568dd38640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:14:23.291 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.281+0000 7f568dd38640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:14:23.291 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.281+0000 7f568dd38640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:14:23.291 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.281+0000 7f568dd38640 10 monclient(hunting): handle_auth_done global_id 1104186 payload 274 2024-09-16T04:14:23.291 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.281+0000 7f568dd38640 1 -- 172.21.15.42:0/673445188 >> v1:172.21.15.154:6789/0 conn(0x7f56880a1710 legacy=0x7f56880a1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:14:23.292 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.281+0000 7f568dd38640 1 -- 172.21.15.42:0/673445188 >> v1:172.21.15.42:6789/0 conn(0x7f56881393a0 legacy=0x7f568813b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:23.292 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.281+0000 7f568dd38640 10 monclient: _finish_hunting 0 2024-09-16T04:14:23.292 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.281+0000 7f568dd38640 1 monclient: found mon.noname-a 2024-09-16T04:14:23.292 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.281+0000 7f568dd38640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:14:23.292 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.281+0000 7f568dd38640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:14:23.292 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.281+0000 7f568dd38640 1 -- 172.21.15.42:0/673445188 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56880a3170 con 0x7f56880a1330 2024-09-16T04:14:23.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.285+0000 7f568dd38640 1 --2- 172.21.15.42:0/673445188 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f56880a1330 0x7f568809f180 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f5680009870 tx=0x7f568001e5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28e77a9646a4de9d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:14:23.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.285+0000 7f568cd36640 1 -- 172.21.15.42:0/673445188 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56800ad610 con 0x7f56880a1330 2024-09-16T04:14:23.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.285+0000 7f568cd36640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:23.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.285+0000 7f568cd36640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:14:23.297 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.285+0000 7f568cd36640 10 monclient: dump: 2024-09-16T04:14:23.297 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:14:23.297 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:14:23.297 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:23.297 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:23.297 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:23.298 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:23.298 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:14:23.298 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:14:23.298 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:14:23.298 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:23.298 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.285+0000 7f568cd36640 10 monclient: _finish_auth 0 2024-09-16T04:14:23.298 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.285+0000 7f568cd36640 10 monclient: _check_auth_tickets 2024-09-16T04:14:23.298 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.285+0000 7f568cd36640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:14:23.298 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.285+0000 7f568ff9c640 5 monclient: authenticate success, global_id 1104186 2024-09-16T04:14:23.298 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.285+0000 7f568ff9c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:14:23.298 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.285+0000 7f568cd36640 1 -- 172.21.15.42:0/673445188 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f56800b2070 con 0x7f56880a1330 2024-09-16T04:14:23.298 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.285+0000 7f568cd36640 10 monclient: handle_config config(0 keys) 2024-09-16T04:14:23.298 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.285+0000 7f568cd36640 1 -- 172.21.15.42:0/673445188 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56800ad9d0 con 0x7f56880a1330 2024-09-16T04:14:23.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.285+0000 7f568cd36640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:23.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.285+0000 7f568cd36640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:14:23.299 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.285+0000 7f568cd36640 10 monclient: dump: 2024-09-16T04:14:23.299 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:14:23.299 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:14:23.299 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:23.299 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:23.299 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:23.299 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:23.299 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:14:23.299 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:14:23.299 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:14:23.299 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:23.300 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.285+0000 7f568ff9c640 10 monclient: get_monmap_and_config success 2024-09-16T04:14:23.300 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.285+0000 7f568ff9c640 4 set_mon_vals no callback set 2024-09-16T04:14:23.300 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.285+0000 7f568ff9c640 10 monclient: shutdown 2024-09-16T04:14:23.300 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.285+0000 7f568ff9c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:14:23.300 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.285+0000 7f568ff9c640 1 -- 172.21.15.42:0/673445188 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f56880a1330 msgr2=0x7f568809f180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:23.300 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.285+0000 7f568ff9c640 1 --2- 172.21.15.42:0/673445188 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f56880a1330 0x7f568809f180 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f5680009870 tx=0x7f568001e5f0 comp rx=0 tx=0).stop 2024-09-16T04:14:23.300 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.285+0000 7f568ff9c640 1 -- 172.21.15.42:0/673445188 shutdown_connections 2024-09-16T04:14:23.300 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.285+0000 7f568ff9c640 1 --2- 172.21.15.42:0/673445188 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f56880a1330 0x7f568809f180 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:23.300 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.285+0000 7f568ff9c640 1 -- 172.21.15.42:0/673445188 >> 172.21.15.42:0/673445188 conn(0x7f56880a2420 msgr2=0x7f56880a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:14:23.300 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.285+0000 7f568ff9c640 1 -- 172.21.15.42:0/673445188 shutdown_connections 2024-09-16T04:14:23.300 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.285+0000 7f568ff9c640 1 -- 172.21.15.42:0/673445188 wait complete. 2024-09-16T04:14:23.300 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.285+0000 7f568ff9c640 10 monclient: build_initial_monmap 2024-09-16T04:14:23.300 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.285+0000 7f568ff9c640 10 monclient: monmap: 2024-09-16T04:14:23.301 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:14:23.301 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:14:23.301 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:14:23.301 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:14:23.301 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:14:23.301 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:23.301 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:14:23.301 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:14:23.301 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:14:23.301 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:23.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.285+0000 7f568ff9c640 1 librados: starting msgr at 2024-09-16T04:14:23.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.285+0000 7f568ff9c640 1 librados: starting objecter 2024-09-16T04:14:23.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.285+0000 7f568ff9c640 1 Processor -- start 2024-09-16T04:14:23.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.285+0000 7f568ff9c640 1 -- start start 2024-09-16T04:14:23.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.285+0000 7f568ff9c640 1 librados: setting wanted keys 2024-09-16T04:14:23.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.285+0000 7f568ff9c640 1 librados: calling monclient init 2024-09-16T04:14:23.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.285+0000 7f568ff9c640 10 monclient: init 2024-09-16T04:14:23.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.285+0000 7f568ff9c640 10 monclient: _reopen_session rank -1 2024-09-16T04:14:23.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.285+0000 7f568ff9c640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:14:23.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.285+0000 7f568ff9c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f56880a1330 0x7f56881335c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:23.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.285+0000 7f568ff9c640 10 monclient(hunting): picked mon.noname-b con 0x7f56880a1330 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:14:23.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.285+0000 7f568ff9c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f56880a1710 0x7f5688133b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:23.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.285+0000 7f568ff9c640 10 monclient(hunting): picked mon.noname-a con 0x7f56880a1710 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:14:23.302 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.285+0000 7f568ff9c640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f56881393a0 0x7f5688134040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:23.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.285+0000 7f568ff9c640 10 monclient(hunting): picked mon.noname-c con 0x7f56881393a0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:14:23.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.285+0000 7f568ff9c640 10 monclient(hunting): start opening mon connection 2024-09-16T04:14:23.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.285+0000 7f568ff9c640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f568809c670 con 0x7f56880a1710 2024-09-16T04:14:23.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.285+0000 7f568ff9c640 10 monclient(hunting): start opening mon connection 2024-09-16T04:14:23.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.285+0000 7f568ff9c640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f568809c4f0 con 0x7f56880a1330 2024-09-16T04:14:23.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.285+0000 7f568ff9c640 10 monclient(hunting): start opening mon connection 2024-09-16T04:14:23.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.285+0000 7f568ff9c640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f568809c7f0 con 0x7f56881393a0 2024-09-16T04:14:23.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.285+0000 7f568ff9c640 10 monclient(hunting): _renew_subs 2024-09-16T04:14:23.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.285+0000 7f568ff9c640 10 monclient(hunting): authenticate will time out at 2910.259130s 2024-09-16T04:14:23.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.285+0000 7f568d537640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f56880a1710 0x7f5688133b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:23.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.285+0000 7f568d537640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f56880a1710 0x7f5688133b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:48018/0 (socket says 172.21.15.42:48018) 2024-09-16T04:14:23.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.285+0000 7f568d537640 1 -- 172.21.15.42:0/1081965412 learned_addr learned my addr 172.21.15.42:0/1081965412 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:14:23.303 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.285+0000 7f568d537640 10 monclient(hunting): get_auth_request con 0x7f56880a1710 auth_method 0 2024-09-16T04:14:23.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.285+0000 7f568d537640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:14:23.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.285+0000 7f568d537640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:23.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.285+0000 7f568d537640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:23.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.285+0000 7f568d537640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:14:23.304 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.285+0000 7f568d537640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:14:23.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.285+0000 7f568d537640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:14:23.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.285+0000 7f568e539640 1 --2- 172.21.15.42:0/1081965412 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f56881393a0 0x7f5688134040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:23.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.285+0000 7f568e539640 10 monclient(hunting): get_auth_request con 0x7f56881393a0 auth_method 0 2024-09-16T04:14:23.305 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.285+0000 7f568e539640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:14:23.306 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.285+0000 7f568e539640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:23.306 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.285+0000 7f568e539640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:23.306 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.285+0000 7f568dd38640 1 --2- 172.21.15.42:0/1081965412 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f56880a1330 0x7f56881335c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:23.306 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.285+0000 7f568d537640 10 monclient(hunting): handle_auth_done global_id 1104189 payload 824 2024-09-16T04:14:23.306 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.285+0000 7f568d537640 1 -- 172.21.15.42:0/1081965412 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f56881393a0 msgr2=0x7f5688134040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:23.306 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.285+0000 7f568d537640 1 --2- 172.21.15.42:0/1081965412 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f56881393a0 0x7f5688134040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:23.307 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.285+0000 7f568d537640 1 -- 172.21.15.42:0/1081965412 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f56880a1330 msgr2=0x7f56881335c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:23.307 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.285+0000 7f568d537640 1 --2- 172.21.15.42:0/1081965412 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f56880a1330 0x7f56881335c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:23.307 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.285+0000 7f568d537640 10 monclient: _finish_hunting 0 2024-09-16T04:14:23.307 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.285+0000 7f568d537640 1 monclient: found mon.noname-a 2024-09-16T04:14:23.307 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.285+0000 7f568d537640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:14:23.307 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.285+0000 7f568d537640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:14:23.307 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.285+0000 7f568d537640 1 -- 172.21.15.42:0/1081965412 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f568811d340 con 0x7f56880a1710 2024-09-16T04:14:23.308 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.285+0000 7f568dd38640 10 monclient: get_auth_request con 0x7f56880a1330 auth_method 0 2024-09-16T04:14:23.308 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.285+0000 7f568dd38640 1 --2- 172.21.15.42:0/1081965412 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f56880a1330 0x7f56881335c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T04:14:23.308 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.285+0000 7f568d537640 1 --2- 172.21.15.42:0/1081965412 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f56880a1710 0x7f5688133b00 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f567c00c7c0 tx=0x7f567c00cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:14:23.308 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.285+0000 7f56867fc640 10 client.?.objecter ms_handle_connect 0x7f56880a1710 2024-09-16T04:14:23.308 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.285+0000 7f56867fc640 10 client.?.objecter resend_mon_ops 2024-09-16T04:14:23.308 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.289+0000 7f56867fc640 1 -- 172.21.15.42:0/1081965412 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f567c016020 con 0x7f56880a1710 2024-09-16T04:14:23.308 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.289+0000 7f56867fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:23.308 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.289+0000 7f56867fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:14:23.309 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.289+0000 7f56867fc640 10 monclient: dump: 2024-09-16T04:14:23.309 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:14:23.309 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:14:23.309 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:23.309 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:23.310 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:23.310 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:23.310 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:14:23.310 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:14:23.310 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:14:23.310 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:23.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.289+0000 7f56867fc640 10 monclient: _finish_auth 0 2024-09-16T04:14:23.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.289+0000 7f56867fc640 10 monclient: _check_auth_tickets 2024-09-16T04:14:23.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.289+0000 7f56867fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:14:23.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.289+0000 7f56867fc640 1 -- 172.21.15.42:0/1081965412 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f567c00aa50 con 0x7f56880a1710 2024-09-16T04:14:23.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.289+0000 7f56867fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:14:23.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.289+0000 7f56867fc640 1 -- 172.21.15.42:0/1081965412 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f567c012420 con 0x7f56880a1710 2024-09-16T04:14:23.311 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.289+0000 7f56867fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:23.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.289+0000 7f56867fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:14:23.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.289+0000 7f56867fc640 10 monclient: dump: 2024-09-16T04:14:23.312 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:14:23.312 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:14:23.312 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:23.312 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:23.312 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:23.312 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:23.312 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:14:23.312 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:14:23.312 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:14:23.313 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:23.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.289+0000 7f56877fe640 4 set_mon_vals no callback set 2024-09-16T04:14:23.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.289+0000 7f568ff9c640 5 monclient: authenticate success, global_id 1104189 2024-09-16T04:14:23.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.289+0000 7f568ff9c640 10 monclient: _renew_subs 2024-09-16T04:14:23.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.289+0000 7f568ff9c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:14:23.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.289+0000 7f568ff9c640 1 -- 172.21.15.42:0/1081965412 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56880723e0 con 0x7f56880a1710 2024-09-16T04:14:23.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.289+0000 7f568ff9c640 10 client.1104189.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:14:23.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.289+0000 7f568ff9c640 10 monclient: _renew_subs 2024-09-16T04:14:23.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.289+0000 7f568ff9c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:14:23.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.289+0000 7f568ff9c640 1 -- 172.21.15.42:0/1081965412 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56880a53c0 con 0x7f56880a1710 2024-09-16T04:14:23.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.289+0000 7f568ff9c640 1 librados: init done 2024-09-16T04:14:23.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.289+0000 7f568ff9c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:14:23.314 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.289+0000 7f56867fc640 1 -- 172.21.15.42:0/1081965412 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f567c019020 con 0x7f56880a1710 2024-09-16T04:14:23.314 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.289+0000 7f568ff9c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:14:23.314 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.289+0000 7f568ff9c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:14:23.314 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.289+0000 7f568ff9c640 1 -- 172.21.15.42:0/1081965412 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f568809f180 con 0x7f56880a1710 2024-09-16T04:14:23.314 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.289+0000 7f56867fc640 1 --2- 172.21.15.42:0/1081965412 >> v2:172.21.15.154:6812/254845342 conn(0x7f5674037810 0x7f5674039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:23.314 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.289+0000 7f56867fc640 1 -- 172.21.15.42:0/1081965412 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(710..710 src has 681..710) ==== 4681+0+0 (secure 0 0 0) 0x7f567c048c60 con 0x7f56880a1710 2024-09-16T04:14:23.314 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.289+0000 7f56867fc640 10 client.1104189.objecter ms_dispatch 0x7f5688000ba0 osd_map(710..710 src has 681..710) 2024-09-16T04:14:23.314 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.289+0000 7f56867fc640 3 client.1104189.objecter handle_osd_map got epochs [710,710] > 0 2024-09-16T04:14:23.314 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.289+0000 7f56867fc640 3 client.1104189.objecter handle_osd_map decoding full epoch 710 2024-09-16T04:14:23.314 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.289+0000 7f56867fc640 20 client.1104189.objecter dump_active .. 0 homeless 2024-09-16T04:14:23.314 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.289+0000 7f568dd38640 1 --2- 172.21.15.42:0/1081965412 >> v2:172.21.15.154:6812/254845342 conn(0x7f5674037810 0x7f5674039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:23.314 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.289+0000 7f568dd38640 10 monclient: get_auth_request con 0x7f5674037810 auth_method 0 2024-09-16T04:14:23.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.289+0000 7f568dd38640 1 --2- 172.21.15.42:0/1081965412 >> v2:172.21.15.154:6812/254845342 conn(0x7f5674037810 0x7f5674039cd0 secure :-1 s=READY pgs=4018 cs=0 l=1 rev1=1 crypto rx=0x7f56800b1000 tx=0x7f5680008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:14:23.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.289+0000 7f56867fc640 10 client.1104189.objecter ms_handle_connect 0x7f5674037810 2024-09-16T04:14:23.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.293+0000 7f56867fc640 1 -- 172.21.15.42:0/1081965412 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f567c020030 con 0x7f56880a1710 2024-09-16T04:14:23.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.293+0000 7f56867fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:14:23.315 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.293+0000 7f56867fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:14:23.454 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.449+0000 7f568ff9c640 1 -- 172.21.15.42:0/1081965412 --> v2:172.21.15.154:6812/254845342 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) -- 0x7f5688072680 con 0x7f5674037810 2024-09-16T04:14:23.455 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.449+0000 7f56867fc640 1 -- 172.21.15.42:0/1081965412 <== mgr.4102 v2:172.21.15.154:6812/254845342 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f5688072680 con 0x7f5674037810 2024-09-16T04:14:23.455 INFO:teuthology.orchestra.run.smithi042.stderr:instructed osd(s) 1 to deep-scrub 2024-09-16T04:14:23.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.449+0000 7f568ff9c640 10 librados: watch_flush enter 2024-09-16T04:14:23.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.453+0000 7f568ff9c640 10 librados: watch_flush exit 2024-09-16T04:14:23.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.453+0000 7f568ff9c640 20 client.1104189.objecter shutdown clearing up homeless session... 2024-09-16T04:14:23.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.453+0000 7f568ff9c640 10 client.1104189.objecter successfully canceled tick 2024-09-16T04:14:23.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.453+0000 7f568ff9c640 1 -- 172.21.15.42:0/1081965412 >> v2:172.21.15.154:6812/254845342 conn(0x7f5674037810 msgr2=0x7f5674039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:23.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.453+0000 7f568ff9c640 1 --2- 172.21.15.42:0/1081965412 >> v2:172.21.15.154:6812/254845342 conn(0x7f5674037810 0x7f5674039cd0 secure :-1 s=READY pgs=4018 cs=0 l=1 rev1=1 crypto rx=0x7f56800b1000 tx=0x7f5680008040 comp rx=0 tx=0).stop 2024-09-16T04:14:23.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.453+0000 7f568ff9c640 10 monclient: shutdown 2024-09-16T04:14:23.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.453+0000 7f568ff9c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:14:23.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.453+0000 7f568ff9c640 1 -- 172.21.15.42:0/1081965412 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f56880a1710 msgr2=0x7f5688133b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:23.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.453+0000 7f568ff9c640 1 --2- 172.21.15.42:0/1081965412 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f56880a1710 0x7f5688133b00 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f567c00c7c0 tx=0x7f567c00cc80 comp rx=0 tx=0).stop 2024-09-16T04:14:23.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.453+0000 7f568ff9c640 1 -- 172.21.15.42:0/1081965412 shutdown_connections 2024-09-16T04:14:23.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.453+0000 7f568ff9c640 1 --2- 172.21.15.42:0/1081965412 >> v2:172.21.15.154:6812/254845342 conn(0x7f5674037810 0x7f5674039cd0 unknown :-1 s=CLOSED pgs=4018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:23.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.453+0000 7f568ff9c640 1 --2- 172.21.15.42:0/1081965412 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f56881393a0 0x7f5688134040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:23.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.453+0000 7f568ff9c640 1 --2- 172.21.15.42:0/1081965412 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f56880a1710 0x7f5688133b00 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:23.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.453+0000 7f568ff9c640 1 --2- 172.21.15.42:0/1081965412 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f56880a1330 0x7f56881335c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:23.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.453+0000 7f568ff9c640 1 -- 172.21.15.42:0/1081965412 >> 172.21.15.42:0/1081965412 conn(0x7f56880a2420 msgr2=0x7f568813a980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:14:23.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.453+0000 7f568ff9c640 1 -- 172.21.15.42:0/1081965412 shutdown_connections 2024-09-16T04:14:23.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.453+0000 7f568ff9c640 1 -- 172.21.15.42:0/1081965412 wait complete. 2024-09-16T04:14:23.459 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.453+0000 7f568ff9c640 1 librados: shutdown 2024-09-16T04:14:23.473 INFO:tasks.ceph:Scrubbing osd.2 2024-09-16T04:14:23.473 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-09-16T04:14:23.590 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.585+0000 7fbd62f4c640 10 monclient: get_monmap_and_config 2024-09-16T04:14:23.590 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.585+0000 7fbd62f4c640 10 monclient: build_initial_monmap 2024-09-16T04:14:23.590 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.585+0000 7fbd62f4c640 10 monclient: monmap: 2024-09-16T04:14:23.590 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:14:23.590 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:14:23.590 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:14:23.588808+0000 2024-09-16T04:14:23.591 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:14:23.588808+0000 2024-09-16T04:14:23.591 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:14:23.591 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:23.591 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:14:23.591 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:14:23.591 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:14:23.591 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:14:23.591 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:14:23.592 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:23.601 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.585+0000 7fbd62f4c640 1 Processor -- start 2024-09-16T04:14:23.601 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.585+0000 7fbd62f4c640 1 -- start start 2024-09-16T04:14:23.601 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.585+0000 7fbd62f4c640 10 monclient: init 2024-09-16T04:14:23.602 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.585+0000 7fbd62f4c640 10 monclient: _reopen_session rank -1 2024-09-16T04:14:23.602 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.585+0000 7fbd62f4c640 10 monclient: _add_conns ranks=[0,1,2,4,3] 2024-09-16T04:14:23.602 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.585+0000 7fbd62f4c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbd5c0a48c0 addr v1:172.21.15.42:6789/0 2024-09-16T04:14:23.602 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.585+0000 7fbd62f4c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbd5c0a53b0 addr v1:172.21.15.154:6789/0 2024-09-16T04:14:23.602 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.585+0000 7fbd62f4c640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbd5c09f3d0 0x7fbd5c13caf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:23.602 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.585+0000 7fbd62f4c640 10 monclient(hunting): picked mon.noname-b con 0x7fbd5c09f3d0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:14:23.602 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.585+0000 7fbd62f4c640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbd5c09d3a0 con 0x7fbd5c0a48c0 2024-09-16T04:14:23.602 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.585+0000 7fbd62f4c640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbd5c09e6a0 con 0x7fbd5c0a53b0 2024-09-16T04:14:23.602 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.585+0000 7fbd62f4c640 10 monclient(hunting): start opening mon connection 2024-09-16T04:14:23.602 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.585+0000 7fbd62f4c640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fbd5c09e980 con 0x7fbd5c09f3d0 2024-09-16T04:14:23.603 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.585+0000 7fbd62f4c640 10 monclient(hunting): _renew_subs 2024-09-16T04:14:23.603 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.585+0000 7fbd62f4c640 10 monclient(hunting): authenticate will time out at 2910.558086s 2024-09-16T04:14:23.603 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.585+0000 7fbd614e9640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbd5c09f3d0 0x7fbd5c13caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:23.603 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.585+0000 7fbd60ce8640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7fbd5c0a48c0 0x7fbd5c0a4ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:53024/0 (socket says 172.21.15.42:53024) 2024-09-16T04:14:23.603 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.585+0000 7fbd60ce8640 1 -- 172.21.15.42:0/826591894 learned_addr learned my addr 172.21.15.42:0/826591894 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T04:14:23.603 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.585+0000 7fbd614e9640 10 monclient(hunting): get_auth_request con 0x7fbd5c09f3d0 auth_method 0 2024-09-16T04:14:23.603 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.585+0000 7fbd614e9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:14:23.603 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.585+0000 7fbd614e9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:23.603 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.585+0000 7fbd614e9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:23.603 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.585+0000 7fbd614e9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:14:23.603 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.585+0000 7fbd614e9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:14:23.603 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.585+0000 7fbd614e9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:14:23.603 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.585+0000 7fbd614e9640 10 monclient(hunting): handle_auth_done global_id 1094264 payload 274 2024-09-16T04:14:23.604 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.585+0000 7fbd614e9640 1 -- 172.21.15.42:0/826591894 >> v1:172.21.15.154:6789/0 conn(0x7fbd5c0a53b0 legacy=0x7fbd5c1393c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:14:23.604 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.585+0000 7fbd5b7fe640 1 -- 172.21.15.42:0/826591894 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3149071827 0 0) 0x7fbd5c09d3a0 con 0x7fbd5c0a48c0 2024-09-16T04:14:23.604 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.585+0000 7fbd614e9640 1 -- 172.21.15.42:0/826591894 >> v1:172.21.15.42:6789/0 conn(0x7fbd5c0a48c0 legacy=0x7fbd5c0a4ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:23.604 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.585+0000 7fbd614e9640 10 monclient: _finish_hunting 0 2024-09-16T04:14:23.604 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.585+0000 7fbd614e9640 1 monclient: found mon.noname-b 2024-09-16T04:14:23.604 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.585+0000 7fbd614e9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:14:23.604 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.585+0000 7fbd614e9640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T04:14:23.604 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.585+0000 7fbd614e9640 1 -- 172.21.15.42:0/826591894 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd5c09ec60 con 0x7fbd5c09f3d0 2024-09-16T04:14:23.604 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.585+0000 7fbd5b7fe640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T04:14:23.604 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.589+0000 7fbd614e9640 1 --2- 172.21.15.42:0/826591894 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbd5c09f3d0 0x7fbd5c13caf0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fbd54009dc0 tx=0x7fbd5401d270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d6fb81b75817d28d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:14:23.604 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.589+0000 7fbd5b7fe640 1 -- 172.21.15.42:0/826591894 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd540b2070 con 0x7fbd5c09f3d0 2024-09-16T04:14:23.604 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.589+0000 7fbd5b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:23.604 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.589+0000 7fbd5b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:14:23.605 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.589+0000 7fbd5b7fe640 10 monclient: dump: 2024-09-16T04:14:23.605 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:14:23.605 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:14:23.605 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:23.605 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:23.605 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:23.605 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:23.605 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:14:23.605 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:14:23.605 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:14:23.605 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:23.605 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.589+0000 7fbd5b7fe640 10 monclient: _finish_auth 0 2024-09-16T04:14:23.605 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.589+0000 7fbd5b7fe640 10 monclient: _check_auth_tickets 2024-09-16T04:14:23.605 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.589+0000 7fbd5b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:14:23.606 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.589+0000 7fbd5b7fe640 1 -- 172.21.15.42:0/826591894 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbd540a58b0 con 0x7fbd5c09f3d0 2024-09-16T04:14:23.606 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.589+0000 7fbd62f4c640 5 monclient: authenticate success, global_id 1094264 2024-09-16T04:14:23.606 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.589+0000 7fbd62f4c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:14:23.606 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.589+0000 7fbd5b7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:14:23.606 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.589+0000 7fbd62f4c640 10 monclient: get_monmap_and_config success 2024-09-16T04:14:23.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.589+0000 7fbd62f4c640 4 set_mon_vals no callback set 2024-09-16T04:14:23.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.589+0000 7fbd5b7fe640 1 -- 172.21.15.42:0/826591894 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd540a5c80 con 0x7fbd5c09f3d0 2024-09-16T04:14:23.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.589+0000 7fbd5b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:23.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.589+0000 7fbd5b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:14:23.607 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.589+0000 7fbd5b7fe640 10 monclient: dump: 2024-09-16T04:14:23.607 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:14:23.607 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:14:23.607 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:23.607 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:23.607 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:23.607 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:23.607 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:14:23.607 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:14:23.607 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:14:23.608 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:23.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.589+0000 7fbd62f4c640 10 monclient: shutdown 2024-09-16T04:14:23.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.589+0000 7fbd62f4c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:14:23.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.589+0000 7fbd62f4c640 1 -- 172.21.15.42:0/826591894 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbd5c09f3d0 msgr2=0x7fbd5c13caf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:23.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.589+0000 7fbd62f4c640 1 --2- 172.21.15.42:0/826591894 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbd5c09f3d0 0x7fbd5c13caf0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fbd54009dc0 tx=0x7fbd5401d270 comp rx=0 tx=0).stop 2024-09-16T04:14:23.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.589+0000 7fbd62f4c640 1 -- 172.21.15.42:0/826591894 shutdown_connections 2024-09-16T04:14:23.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.589+0000 7fbd62f4c640 1 --2- 172.21.15.42:0/826591894 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbd5c09f3d0 0x7fbd5c13caf0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:23.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.589+0000 7fbd62f4c640 1 -- 172.21.15.42:0/826591894 >> 172.21.15.42:0/826591894 conn(0x7fbd5c0a6300 msgr2=0x7fbd5c0a24b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:14:23.608 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.589+0000 7fbd62f4c640 1 -- 172.21.15.42:0/826591894 shutdown_connections 2024-09-16T04:14:23.609 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.589+0000 7fbd62f4c640 1 -- 172.21.15.42:0/826591894 wait complete. 2024-09-16T04:14:23.609 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.589+0000 7fbd62f4c640 10 monclient: build_initial_monmap 2024-09-16T04:14:23.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.589+0000 7fbd62f4c640 10 monclient: monmap: 2024-09-16T04:14:23.610 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:14:23.610 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:14:23.610 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:14:23.610 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:14:23.610 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:14:23.610 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:23.610 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:14:23.610 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:14:23.610 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:14:23.610 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:23.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.589+0000 7fbd62f4c640 1 librados: starting msgr at 2024-09-16T04:14:23.610 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.589+0000 7fbd62f4c640 1 librados: starting objecter 2024-09-16T04:14:23.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.589+0000 7fbd62f4c640 1 Processor -- start 2024-09-16T04:14:23.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.589+0000 7fbd62f4c640 1 -- start start 2024-09-16T04:14:23.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.589+0000 7fbd62f4c640 1 librados: setting wanted keys 2024-09-16T04:14:23.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.589+0000 7fbd62f4c640 1 librados: calling monclient init 2024-09-16T04:14:23.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.589+0000 7fbd62f4c640 10 monclient: init 2024-09-16T04:14:23.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.589+0000 7fbd62f4c640 10 monclient: _reopen_session rank -1 2024-09-16T04:14:23.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.589+0000 7fbd62f4c640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:14:23.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.589+0000 7fbd62f4c640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbd5c09f3d0 0x7fbd5c11c000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:23.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.589+0000 7fbd62f4c640 10 monclient(hunting): picked mon.noname-b con 0x7fbd5c09f3d0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:14:23.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.589+0000 7fbd62f4c640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbd5c0a48c0 0x7fbd5c11c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:23.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.589+0000 7fbd62f4c640 10 monclient(hunting): picked mon.noname-c con 0x7fbd5c0a48c0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:14:23.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.589+0000 7fbd62f4c640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbd5c0a53b0 0x7fbd5c11ca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:23.611 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.589+0000 7fbd62f4c640 10 monclient(hunting): picked mon.noname-a con 0x7fbd5c0a53b0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:14:23.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.589+0000 7fbd62f4c640 10 monclient(hunting): start opening mon connection 2024-09-16T04:14:23.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.589+0000 7fbd62f4c640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fbd5c13f9d0 con 0x7fbd5c0a53b0 2024-09-16T04:14:23.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.589+0000 7fbd62f4c640 10 monclient(hunting): start opening mon connection 2024-09-16T04:14:23.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.589+0000 7fbd62f4c640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fbd5c13f850 con 0x7fbd5c09f3d0 2024-09-16T04:14:23.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.589+0000 7fbd62f4c640 10 monclient(hunting): start opening mon connection 2024-09-16T04:14:23.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.589+0000 7fbd62f4c640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fbd5c13fb50 con 0x7fbd5c0a48c0 2024-09-16T04:14:23.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.589+0000 7fbd62f4c640 10 monclient(hunting): _renew_subs 2024-09-16T04:14:23.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.589+0000 7fbd62f4c640 10 monclient(hunting): authenticate will time out at 2910.562916s 2024-09-16T04:14:23.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.589+0000 7fbd5bfff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbd5c0a48c0 0x7fbd5c11c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:23.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.589+0000 7fbd614e9640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbd5c0a53b0 0x7fbd5c11ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:23.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.589+0000 7fbd5bfff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbd5c0a48c0 0x7fbd5c11c540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:51984/0 (socket says 172.21.15.42:51984) 2024-09-16T04:14:23.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.589+0000 7fbd614e9640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbd5c0a53b0 0x7fbd5c11ca80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:48030/0 (socket says 172.21.15.42:48030) 2024-09-16T04:14:23.612 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.589+0000 7fbd5bfff640 1 -- 172.21.15.42:0/3937667968 learned_addr learned my addr 172.21.15.42:0/3937667968 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:14:23.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.589+0000 7fbd5bfff640 10 monclient(hunting): get_auth_request con 0x7fbd5c0a48c0 auth_method 0 2024-09-16T04:14:23.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.589+0000 7fbd5bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:14:23.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.589+0000 7fbd5bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:23.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.589+0000 7fbd5bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:23.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.589+0000 7fbd614e9640 10 monclient(hunting): get_auth_request con 0x7fbd5c0a53b0 auth_method 0 2024-09-16T04:14:23.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.589+0000 7fbd614e9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:14:23.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.589+0000 7fbd614e9640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:23.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.589+0000 7fbd614e9640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:23.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.589+0000 7fbd5bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:14:23.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.589+0000 7fbd5bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:14:23.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.589+0000 7fbd5bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:14:23.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.589+0000 7fbd614e9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:14:23.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.589+0000 7fbd614e9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:14:23.613 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.589+0000 7fbd614e9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:14:23.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.589+0000 7fbd60ce8640 1 --2- 172.21.15.42:0/3937667968 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbd5c09f3d0 0x7fbd5c11c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:23.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.589+0000 7fbd5bfff640 10 monclient(hunting): handle_auth_done global_id 1094267 payload 824 2024-09-16T04:14:23.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.589+0000 7fbd5bfff640 1 -- 172.21.15.42:0/3937667968 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbd5c09f3d0 msgr2=0x7fbd5c11c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:23.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.589+0000 7fbd5bfff640 1 --2- 172.21.15.42:0/3937667968 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbd5c09f3d0 0x7fbd5c11c000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:23.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.589+0000 7fbd5bfff640 1 -- 172.21.15.42:0/3937667968 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbd5c0a53b0 msgr2=0x7fbd5c11ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:23.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.589+0000 7fbd5bfff640 1 --2- 172.21.15.42:0/3937667968 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fbd5c0a53b0 0x7fbd5c11ca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:23.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.589+0000 7fbd5bfff640 10 monclient: _finish_hunting 0 2024-09-16T04:14:23.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.589+0000 7fbd5bfff640 1 monclient: found mon.noname-c 2024-09-16T04:14:23.614 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.589+0000 7fbd5bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:14:23.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.589+0000 7fbd5bfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T04:14:23.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.589+0000 7fbd5bfff640 1 -- 172.21.15.42:0/3937667968 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd5c11e3e0 con 0x7fbd5c0a48c0 2024-09-16T04:14:23.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.589+0000 7fbd60ce8640 10 monclient: get_auth_request con 0x7fbd5c09f3d0 auth_method 0 2024-09-16T04:14:23.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.589+0000 7fbd60ce8640 1 --2- 172.21.15.42:0/3937667968 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fbd5c09f3d0 0x7fbd5c11c000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T04:14:23.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.589+0000 7fbd5bfff640 1 --2- 172.21.15.42:0/3937667968 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbd5c0a48c0 0x7fbd5c11c540 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fbd4400c450 tx=0x7fbd4400c910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:14:23.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.589+0000 7fbd597fa640 10 client.?.objecter ms_handle_connect 0x7fbd5c0a48c0 2024-09-16T04:14:23.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.589+0000 7fbd597fa640 10 client.?.objecter resend_mon_ops 2024-09-16T04:14:23.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.589+0000 7fbd597fa640 1 -- 172.21.15.42:0/3937667968 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd44012830 con 0x7fbd5c0a48c0 2024-09-16T04:14:23.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.589+0000 7fbd597fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:23.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.589+0000 7fbd597fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:14:23.615 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.589+0000 7fbd597fa640 10 monclient: dump: 2024-09-16T04:14:23.615 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:14:23.615 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:14:23.615 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:23.616 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:23.616 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:23.616 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:23.616 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:14:23.616 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:14:23.616 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:14:23.616 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:23.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.589+0000 7fbd597fa640 10 monclient: _finish_auth 0 2024-09-16T04:14:23.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.589+0000 7fbd597fa640 10 monclient: _check_auth_tickets 2024-09-16T04:14:23.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.589+0000 7fbd597fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:14:23.616 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.589+0000 7fbd62f4c640 5 monclient: authenticate success, global_id 1094267 2024-09-16T04:14:23.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.589+0000 7fbd62f4c640 10 monclient: _renew_subs 2024-09-16T04:14:23.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.589+0000 7fbd597fa640 1 -- 172.21.15.42:0/3937667968 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbd44017070 con 0x7fbd5c0a48c0 2024-09-16T04:14:23.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.589+0000 7fbd62f4c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:14:23.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.589+0000 7fbd62f4c640 1 -- 172.21.15.42:0/3937667968 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd5c11e6c0 con 0x7fbd5c0a48c0 2024-09-16T04:14:23.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.589+0000 7fbd597fa640 10 monclient: handle_config config(0 keys) 2024-09-16T04:14:23.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.589+0000 7fbd597fa640 1 -- 172.21.15.42:0/3937667968 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd44012c30 con 0x7fbd5c0a48c0 2024-09-16T04:14:23.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.589+0000 7fbd597fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:23.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.589+0000 7fbd597fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:14:23.617 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.589+0000 7fbd597fa640 10 monclient: dump: 2024-09-16T04:14:23.618 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:14:23.618 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:14:23.618 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:23.618 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:23.618 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:23.618 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:23.618 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:14:23.618 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:14:23.618 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:14:23.618 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:23.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.589+0000 7fbd62f4c640 10 client.1094267.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:14:23.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.589+0000 7fbd5a7fc640 4 set_mon_vals no callback set 2024-09-16T04:14:23.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.589+0000 7fbd62f4c640 10 monclient: _renew_subs 2024-09-16T04:14:23.618 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.589+0000 7fbd62f4c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:14:23.619 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.589+0000 7fbd62f4c640 1 -- 172.21.15.42:0/3937667968 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd5c0a1eb0 con 0x7fbd5c0a48c0 2024-09-16T04:14:23.619 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.589+0000 7fbd62f4c640 1 librados: init done 2024-09-16T04:14:23.619 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.593+0000 7fbd62f4c640 10 client.1094267.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:14:23.619 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.593+0000 7fbd62f4c640 15 client.1094267.objecter _session_command_op_assign -1 1 2024-09-16T04:14:23.619 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.593+0000 7fbd62f4c640 20 client.1094267.objecter _get_session osd=-1 returning homeless 2024-09-16T04:14:23.619 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.593+0000 7fbd62f4c640 10 client.1094267.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:14:23.619 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.593+0000 7fbd62f4c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:14:23.619 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.593+0000 7fbd62f4c640 1 -- 172.21.15.42:0/3937667968 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fbd5c073aa0 con 0x7fbd5c0a48c0 2024-09-16T04:14:23.619 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.593+0000 7fbd597fa640 1 -- 172.21.15.42:0/3937667968 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fbd4401c460 con 0x7fbd5c0a48c0 2024-09-16T04:14:23.619 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.593+0000 7fbd597fa640 1 --2- 172.21.15.42:0/3937667968 >> v2:172.21.15.154:6812/254845342 conn(0x7fbd3c037810 0x7fbd3c039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:23.619 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.593+0000 7fbd597fa640 1 -- 172.21.15.42:0/3937667968 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(710..710 src has 681..710) ==== 4681+0+0 (secure 0 0 0) 0x7fbd44048830 con 0x7fbd5c0a48c0 2024-09-16T04:14:23.619 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.593+0000 7fbd597fa640 10 client.1094267.objecter ms_dispatch 0x7fbd5c000ba0 osd_map(710..710 src has 681..710) 2024-09-16T04:14:23.619 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.593+0000 7fbd597fa640 3 client.1094267.objecter handle_osd_map got epochs [710,710] > 0 2024-09-16T04:14:23.619 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.593+0000 7fbd597fa640 3 client.1094267.objecter handle_osd_map decoding full epoch 710 2024-09-16T04:14:23.620 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.593+0000 7fbd597fa640 10 client.1094267.objecter checking command 1 2024-09-16T04:14:23.620 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.593+0000 7fbd597fa640 1 --2- 172.21.15.42:0/3937667968 >> v2:172.21.15.42:6804/263835041 conn(0x7fbd3c03ec70 0x7fbd3c041110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:23.620 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.593+0000 7fbd597fa640 20 client.1094267.objecter _get_session s=0x7fbd3c03e5c0 osd=2 3 2024-09-16T04:14:23.620 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.593+0000 7fbd597fa640 20 client.1094267.objecter put_session s=0x7fbd3c03e5c0 osd=2 3 2024-09-16T04:14:23.620 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.593+0000 7fbd597fa640 15 client.1094267.objecter _session_command_op_remove -1 1 2024-09-16T04:14:23.620 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.593+0000 7fbd597fa640 20 client.1094267.objecter _get_session s=0x7fbd3c03e5c0 osd=2 3 2024-09-16T04:14:23.620 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.593+0000 7fbd597fa640 20 client.1094267.objecter get_session s=0x7fbd3c03e5c0 osd=2 3 2024-09-16T04:14:23.620 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.593+0000 7fbd597fa640 15 client.1094267.objecter _session_command_op_assign 2 1 2024-09-16T04:14:23.620 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.593+0000 7fbd597fa640 20 client.1094267.objecter put_session s=0x7fbd3c03e5c0 osd=2 4 2024-09-16T04:14:23.620 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.593+0000 7fbd597fa640 10 client.1094267.objecter _send_command 1 2024-09-16T04:14:23.620 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.593+0000 7fbd597fa640 1 -- 172.21.15.42:0/3937667968 --> v2:172.21.15.42:6804/263835041 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbd4400ae50 con 0x7fbd3c03ec70 2024-09-16T04:14:23.620 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.593+0000 7fbd597fa640 20 client.1094267.objecter dump_active .. 0 homeless 2024-09-16T04:14:23.620 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.593+0000 7fbd597fa640 1 -- 172.21.15.42:0/3937667968 <== mon.2 v2:172.21.15.42:3301/0 6 ==== mon_get_version_reply(handle=1 version=710) ==== 24+0+0 (secure 0 0 0) 0x7fbd4404d2c0 con 0x7fbd5c0a48c0 2024-09-16T04:14:23.620 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.593+0000 7fbd597fa640 10 monclient: handle_get_version_reply finishing 1 version 710 2024-09-16T04:14:23.621 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.593+0000 7fbd614e9640 1 --2- 172.21.15.42:0/3937667968 >> v2:172.21.15.42:6804/263835041 conn(0x7fbd3c03ec70 0x7fbd3c041110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:23.621 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.593+0000 7fbd614e9640 10 monclient: get_auth_request con 0x7fbd3c03ec70 auth_method 0 2024-09-16T04:14:23.621 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.593+0000 7fbd60ce8640 1 --2- 172.21.15.42:0/3937667968 >> v2:172.21.15.154:6812/254845342 conn(0x7fbd3c037810 0x7fbd3c039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:23.621 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.593+0000 7fbd60ce8640 10 monclient: get_auth_request con 0x7fbd3c037810 auth_method 0 2024-09-16T04:14:23.621 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.593+0000 7fbd597fa640 10 client.1094267.objecter ms_handle_connect 0x7fbd3c03ec70 2024-09-16T04:14:23.621 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.593+0000 7fbd614e9640 1 --2- 172.21.15.42:0/3937667968 >> v2:172.21.15.42:6804/263835041 conn(0x7fbd3c03ec70 0x7fbd3c041110 crc :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:14:23.621 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.593+0000 7fbd597fa640 10 client.1094267.objecter ms_handle_connect 0x7fbd3c037810 2024-09-16T04:14:23.621 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.593+0000 7fbd60ce8640 1 --2- 172.21.15.42:0/3937667968 >> v2:172.21.15.154:6812/254845342 conn(0x7fbd3c037810 0x7fbd3c039cd0 secure :-1 s=READY pgs=4019 cs=0 l=1 rev1=1 crypto rx=0x7fbd4c000f70 tx=0x7fbd4c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:14:23.621 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.593+0000 7fbd597fa640 1 -- 172.21.15.42:0/3937667968 <== osd.2 v2:172.21.15.42:6804/263835041 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fbd4400ae50 con 0x7fbd3c03ec70 2024-09-16T04:14:23.621 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.593+0000 7fbd597fa640 10 client.1094267.objecter ms_dispatch 0x7fbd5c000ba0 command_reply(tid 1: 0 ) 2024-09-16T04:14:23.621 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.593+0000 7fbd597fa640 10 client.1094267.objecter _finish_command 1 = system:0 2024-09-16T04:14:23.621 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.593+0000 7fbd597fa640 20 client.1094267.objecter put_session s=0x7fbd3c03e5c0 osd=2 4 2024-09-16T04:14:23.621 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.593+0000 7fbd597fa640 15 client.1094267.objecter _session_command_op_remove 2 1 2024-09-16T04:14:23.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.605+0000 7fbd62f4c640 10 client.1094267.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-09-16T04:14:23.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.605+0000 7fbd62f4c640 15 client.1094267.objecter _session_command_op_assign -1 2 2024-09-16T04:14:23.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.605+0000 7fbd62f4c640 20 client.1094267.objecter _get_session s=0x7fbd3c03e5c0 osd=2 3 2024-09-16T04:14:23.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.605+0000 7fbd62f4c640 20 client.1094267.objecter put_session s=0x7fbd3c03e5c0 osd=2 3 2024-09-16T04:14:23.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.605+0000 7fbd62f4c640 20 client.1094267.objecter _get_session s=0x7fbd3c03e5c0 osd=2 3 2024-09-16T04:14:23.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.605+0000 7fbd62f4c640 15 client.1094267.objecter _session_command_op_remove -1 2 2024-09-16T04:14:23.623 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.605+0000 7fbd62f4c640 20 client.1094267.objecter get_session s=0x7fbd3c03e5c0 osd=2 3 2024-09-16T04:14:23.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.605+0000 7fbd62f4c640 15 client.1094267.objecter _session_command_op_assign 2 2 2024-09-16T04:14:23.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.605+0000 7fbd62f4c640 20 client.1094267.objecter put_session s=0x7fbd3c03e5c0 osd=2 4 2024-09-16T04:14:23.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.605+0000 7fbd62f4c640 10 client.1094267.objecter _send_command 2 2024-09-16T04:14:23.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.605+0000 7fbd62f4c640 1 -- 172.21.15.42:0/3937667968 --> v2:172.21.15.42:6804/263835041 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fbd5c0a4ca0 con 0x7fbd3c03ec70 2024-09-16T04:14:23.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.605+0000 7fbd597fa640 1 -- 172.21.15.42:0/3937667968 <== osd.2 v2:172.21.15.42:6804/263835041 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fbd5c0a4ca0 con 0x7fbd3c03ec70 2024-09-16T04:14:23.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.605+0000 7fbd597fa640 10 client.1094267.objecter ms_dispatch 0x7fbd5c000ba0 command_reply(tid 2: 0 ) 2024-09-16T04:14:23.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.605+0000 7fbd597fa640 10 client.1094267.objecter _finish_command 2 = system:0 2024-09-16T04:14:23.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.605+0000 7fbd597fa640 20 client.1094267.objecter put_session s=0x7fbd3c03e5c0 osd=2 4 2024-09-16T04:14:23.624 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.605+0000 7fbd597fa640 15 client.1094267.objecter _session_command_op_remove 2 2 2024-09-16T04:14:23.625 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T04:14:23.625 INFO:teuthology.orchestra.run.smithi042.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T04:14:23.625 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T04:14:23.626 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.609+0000 7fbd62f4c640 10 librados: watch_flush enter 2024-09-16T04:14:23.626 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.609+0000 7fbd62f4c640 10 librados: watch_flush exit 2024-09-16T04:14:23.626 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.609+0000 7fbd62f4c640 10 client.1094267.objecter close_session for osd.2 2024-09-16T04:14:23.626 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.609+0000 7fbd62f4c640 1 -- 172.21.15.42:0/3937667968 >> v2:172.21.15.42:6804/263835041 conn(0x7fbd3c03ec70 msgr2=0x7fbd3c041110 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:23.626 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.609+0000 7fbd62f4c640 1 --2- 172.21.15.42:0/3937667968 >> v2:172.21.15.42:6804/263835041 conn(0x7fbd3c03ec70 0x7fbd3c041110 crc :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:23.627 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.609+0000 7fbd62f4c640 20 client.1094267.objecter put_session s=0x7fbd3c03e5c0 osd=2 1 2024-09-16T04:14:23.627 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.609+0000 7fbd62f4c640 20 client.1094267.objecter shutdown clearing up homeless session... 2024-09-16T04:14:23.627 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.609+0000 7fbd62f4c640 10 client.1094267.objecter successfully canceled tick 2024-09-16T04:14:23.627 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.609+0000 7fbd62f4c640 1 -- 172.21.15.42:0/3937667968 >> v2:172.21.15.154:6812/254845342 conn(0x7fbd3c037810 msgr2=0x7fbd3c039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:23.627 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.609+0000 7fbd62f4c640 1 --2- 172.21.15.42:0/3937667968 >> v2:172.21.15.154:6812/254845342 conn(0x7fbd3c037810 0x7fbd3c039cd0 secure :-1 s=READY pgs=4019 cs=0 l=1 rev1=1 crypto rx=0x7fbd4c000f70 tx=0x7fbd4c009040 comp rx=0 tx=0).stop 2024-09-16T04:14:23.627 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.609+0000 7fbd62f4c640 10 monclient: shutdown 2024-09-16T04:14:23.627 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.609+0000 7fbd62f4c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:14:23.627 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.609+0000 7fbd62f4c640 1 -- 172.21.15.42:0/3937667968 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbd5c0a48c0 msgr2=0x7fbd5c11c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:23.627 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.609+0000 7fbd62f4c640 1 --2- 172.21.15.42:0/3937667968 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fbd5c0a48c0 0x7fbd5c11c540 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fbd4400c450 tx=0x7fbd4400c910 comp rx=0 tx=0).stop 2024-09-16T04:14:23.627 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.609+0000 7fbd614e9640 1 -- 172.21.15.42:0/3937667968 reap_dead start 2024-09-16T04:14:23.628 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.609+0000 7fbd62f4c640 1 -- 172.21.15.42:0/3937667968 shutdown_connections 2024-09-16T04:14:23.628 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.609+0000 7fbd62f4c640 1 -- 172.21.15.42:0/3937667968 >> 172.21.15.42:0/3937667968 conn(0x7fbd5c0a6300 msgr2=0x7fbd5c0a22c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:14:23.628 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.609+0000 7fbd62f4c640 1 -- 172.21.15.42:0/3937667968 shutdown_connections 2024-09-16T04:14:23.628 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.609+0000 7fbd62f4c640 1 -- 172.21.15.42:0/3937667968 wait complete. 2024-09-16T04:14:23.628 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.609+0000 7fbd62f4c640 1 librados: shutdown 2024-09-16T04:14:23.629 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-09-16T04:14:23.740 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.733+0000 7f3cea934640 10 monclient: get_monmap_and_config 2024-09-16T04:14:23.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.733+0000 7f3cea934640 10 monclient: build_initial_monmap 2024-09-16T04:14:23.741 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.733+0000 7f3cea934640 10 monclient: monmap: 2024-09-16T04:14:23.741 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:14:23.741 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:14:23.741 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:14:23.739561+0000 2024-09-16T04:14:23.741 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:14:23.739561+0000 2024-09-16T04:14:23.741 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:14:23.741 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:23.741 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:14:23.741 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:14:23.741 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:14:23.741 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:14:23.742 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:14:23.742 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:23.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.737+0000 7f3cea934640 1 Processor -- start 2024-09-16T04:14:23.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.737+0000 7f3cea934640 1 -- start start 2024-09-16T04:14:23.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.737+0000 7f3cea934640 10 monclient: init 2024-09-16T04:14:23.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.737+0000 7f3cea934640 10 monclient: _reopen_session rank -1 2024-09-16T04:14:23.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.737+0000 7f3cea934640 10 monclient: _add_conns ranks=[2,4,3,0,1] 2024-09-16T04:14:23.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.737+0000 7f3cea934640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3ce40a1330 0x7f3ce409f180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:23.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.737+0000 7f3cea934640 10 monclient(hunting): picked mon.noname-b con 0x7f3ce40a1330 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:14:23.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.737+0000 7f3cea934640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f3ce40a1710 0x7f3ce40a1b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:23.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.737+0000 7f3cea934640 10 monclient(hunting): picked mon.noname-c con 0x7f3ce40a1710 addr v2:172.21.15.154:3300/0 2024-09-16T04:14:23.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.737+0000 7f3cea934640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f3ce41340f0 0x7f3ce41345a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:23.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.737+0000 7f3cea934640 10 monclient(hunting): picked mon.noname-a con 0x7f3ce41340f0 addr v2:172.21.15.42:3300/0 2024-09-16T04:14:23.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.737+0000 7f3cea934640 10 monclient(hunting): start opening mon connection 2024-09-16T04:14:23.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.737+0000 7f3cea934640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f3ce409edf0 con 0x7f3ce41340f0 2024-09-16T04:14:23.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.737+0000 7f3cea934640 10 monclient(hunting): start opening mon connection 2024-09-16T04:14:23.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.737+0000 7f3cea934640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f3ce409cd60 con 0x7f3ce40a1710 2024-09-16T04:14:23.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.737+0000 7f3cea934640 10 monclient(hunting): start opening mon connection 2024-09-16T04:14:23.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.737+0000 7f3cea934640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f3ce409e4c0 con 0x7f3ce40a1330 2024-09-16T04:14:23.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.737+0000 7f3cea934640 10 monclient(hunting): _renew_subs 2024-09-16T04:14:23.746 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.737+0000 7f3cea934640 10 monclient(hunting): authenticate will time out at 2910.708973s 2024-09-16T04:14:23.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.737+0000 7f3ce3fff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3ce40a1330 0x7f3ce409f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:23.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.737+0000 7f3ce3fff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3ce40a1330 0x7f3ce409f180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:51998/0 (socket says 172.21.15.42:51998) 2024-09-16T04:14:23.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.737+0000 7f3ce3fff640 1 -- 172.21.15.42:0/3908044189 learned_addr learned my addr 172.21.15.42:0/3908044189 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:14:23.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.737+0000 7f3ce3fff640 10 monclient(hunting): get_auth_request con 0x7f3ce40a1330 auth_method 0 2024-09-16T04:14:23.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.737+0000 7f3ce3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:14:23.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.737+0000 7f3ce3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:23.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.737+0000 7f3ce3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:23.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.737+0000 7f3ce8ed1640 1 --2- 172.21.15.42:0/3908044189 >> v2:172.21.15.42:3300/0 conn(0x7f3ce41340f0 0x7f3ce41345a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:23.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.737+0000 7f3ce8ed1640 10 monclient(hunting): get_auth_request con 0x7f3ce41340f0 auth_method 0 2024-09-16T04:14:23.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.737+0000 7f3ce8ed1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:14:23.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.737+0000 7f3ce8ed1640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:23.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.737+0000 7f3ce8ed1640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:23.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.737+0000 7f3ce3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:14:23.747 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.737+0000 7f3ce3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:14:23.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.737+0000 7f3ce3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:14:23.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.737+0000 7f3ce8ed1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:14:23.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.737+0000 7f3ce8ed1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:14:23.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.737+0000 7f3ce8ed1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:14:23.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.737+0000 7f3cdb7fe640 1 --2- 172.21.15.42:0/3908044189 >> v2:172.21.15.154:3300/0 conn(0x7f3ce40a1710 0x7f3ce40a1b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:23.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.737+0000 7f3ce3fff640 10 monclient(hunting): handle_auth_done global_id 1094270 payload 274 2024-09-16T04:14:23.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.737+0000 7f3ce3fff640 1 -- 172.21.15.42:0/3908044189 >> v2:172.21.15.154:3300/0 conn(0x7f3ce40a1710 msgr2=0x7f3ce40a1b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:14:23.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.737+0000 7f3ce3fff640 1 --2- 172.21.15.42:0/3908044189 >> v2:172.21.15.154:3300/0 conn(0x7f3ce40a1710 0x7f3ce40a1b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:23.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.737+0000 7f3ce3fff640 1 -- 172.21.15.42:0/3908044189 >> v2:172.21.15.42:3300/0 conn(0x7f3ce41340f0 msgr2=0x7f3ce41345a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:14:23.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.737+0000 7f3ce3fff640 1 --2- 172.21.15.42:0/3908044189 >> v2:172.21.15.42:3300/0 conn(0x7f3ce41340f0 0x7f3ce41345a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:23.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.737+0000 7f3ce3fff640 10 monclient: _finish_hunting 0 2024-09-16T04:14:23.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.737+0000 7f3ce3fff640 1 monclient: found mon.noname-b 2024-09-16T04:14:23.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.737+0000 7f3ce3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:14:23.748 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.737+0000 7f3ce3fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T04:14:23.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.737+0000 7f3ce3fff640 1 -- 172.21.15.42:0/3908044189 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ce40a2050 con 0x7f3ce40a1330 2024-09-16T04:14:23.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.737+0000 7f3cdb7fe640 10 monclient: get_auth_request con 0x7f3ce40a1710 auth_method 0 2024-09-16T04:14:23.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.737+0000 7f3cdb7fe640 1 --2- 172.21.15.42:0/3908044189 >> v2:172.21.15.154:3300/0 conn(0x7f3ce40a1710 0x7f3ce40a1b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T04:14:23.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.737+0000 7f3ce8ed1640 1 --2- 172.21.15.42:0/3908044189 >> v2:172.21.15.42:3300/0 conn(0x7f3ce41340f0 0x7f3ce41345a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:14:23.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.737+0000 7f3ce3fff640 1 --2- 172.21.15.42:0/3908044189 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3ce40a1330 0x7f3ce409f180 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f3ccc009870 tx=0x7f3ccc01c3e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5234aa0d363f29ad server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:14:23.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.737+0000 7f3ce37fe640 1 -- 172.21.15.42:0/3908044189 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ccc0b0070 con 0x7f3ce40a1330 2024-09-16T04:14:23.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.737+0000 7f3ce37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:23.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.737+0000 7f3ce37fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:14:23.749 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.737+0000 7f3ce37fe640 10 monclient: dump: 2024-09-16T04:14:23.749 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:14:23.749 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:14:23.749 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:23.749 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:23.749 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:23.750 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:23.750 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:14:23.750 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:14:23.750 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:14:23.750 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:23.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.737+0000 7f3ce37fe640 10 monclient: _finish_auth 0 2024-09-16T04:14:23.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.737+0000 7f3ce37fe640 10 monclient: _check_auth_tickets 2024-09-16T04:14:23.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.737+0000 7f3ce37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:14:23.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.737+0000 7f3ce37fe640 1 -- 172.21.15.42:0/3908044189 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3ccc0ab600 con 0x7f3ce40a1330 2024-09-16T04:14:23.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.737+0000 7f3ce37fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:14:23.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.737+0000 7f3ce37fe640 1 -- 172.21.15.42:0/3908044189 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ccc0ab9d0 con 0x7f3ce40a1330 2024-09-16T04:14:23.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.737+0000 7f3cea934640 5 monclient: authenticate success, global_id 1094270 2024-09-16T04:14:23.750 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.737+0000 7f3cea934640 10 monclient: get_monmap_and_config success 2024-09-16T04:14:23.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.737+0000 7f3cea934640 4 set_mon_vals no callback set 2024-09-16T04:14:23.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.737+0000 7f3ce37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:23.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.737+0000 7f3ce37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:14:23.751 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.737+0000 7f3ce37fe640 10 monclient: dump: 2024-09-16T04:14:23.751 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:14:23.751 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:14:23.751 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:23.751 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:23.751 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:23.751 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:23.751 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:14:23.751 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:14:23.751 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:14:23.751 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:23.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.741+0000 7f3cea934640 10 monclient: shutdown 2024-09-16T04:14:23.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.741+0000 7f3cea934640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:14:23.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.741+0000 7f3cea934640 1 -- 172.21.15.42:0/3908044189 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3ce40a1330 msgr2=0x7f3ce409f180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:23.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.741+0000 7f3cea934640 1 --2- 172.21.15.42:0/3908044189 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3ce40a1330 0x7f3ce409f180 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f3ccc009870 tx=0x7f3ccc01c3e0 comp rx=0 tx=0).stop 2024-09-16T04:14:23.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.741+0000 7f3cea934640 1 -- 172.21.15.42:0/3908044189 shutdown_connections 2024-09-16T04:14:23.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.741+0000 7f3cea934640 1 --2- 172.21.15.42:0/3908044189 >> v2:172.21.15.42:3300/0 conn(0x7f3ce41340f0 0x7f3ce41345a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:23.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.741+0000 7f3cea934640 1 --2- 172.21.15.42:0/3908044189 >> v2:172.21.15.154:3300/0 conn(0x7f3ce40a1710 0x7f3ce40a1b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:23.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.741+0000 7f3cea934640 1 --2- 172.21.15.42:0/3908044189 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3ce40a1330 0x7f3ce409f180 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:23.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.741+0000 7f3cea934640 1 -- 172.21.15.42:0/3908044189 >> 172.21.15.42:0/3908044189 conn(0x7f3ce40a2420 msgr2=0x7f3ce40a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:14:23.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.741+0000 7f3cea934640 1 -- 172.21.15.42:0/3908044189 shutdown_connections 2024-09-16T04:14:23.752 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.741+0000 7f3cea934640 1 -- 172.21.15.42:0/3908044189 wait complete. 2024-09-16T04:14:23.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.741+0000 7f3cea934640 10 monclient: build_initial_monmap 2024-09-16T04:14:23.756 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.741+0000 7f3cea934640 10 monclient: monmap: 2024-09-16T04:14:23.756 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:14:23.756 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:14:23.756 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:14:23.756 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:14:23.756 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:14:23.757 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:23.757 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:14:23.757 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:14:23.757 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:14:23.757 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:23.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.741+0000 7f3cea934640 1 librados: starting msgr at 2024-09-16T04:14:23.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.741+0000 7f3cea934640 1 librados: starting objecter 2024-09-16T04:14:23.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.741+0000 7f3cea934640 1 Processor -- start 2024-09-16T04:14:23.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.741+0000 7f3cea934640 1 -- start start 2024-09-16T04:14:23.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.741+0000 7f3cea934640 1 librados: setting wanted keys 2024-09-16T04:14:23.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.741+0000 7f3cea934640 1 librados: calling monclient init 2024-09-16T04:14:23.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.741+0000 7f3cea934640 10 monclient: init 2024-09-16T04:14:23.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.741+0000 7f3cea934640 10 monclient: _reopen_session rank -1 2024-09-16T04:14:23.757 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.741+0000 7f3cea934640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:14:23.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.741+0000 7f3cea934640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3ce40a1330 0x7f3ce4131450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:23.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.741+0000 7f3cea934640 10 monclient(hunting): picked mon.noname-a con 0x7f3ce40a1330 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:14:23.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.741+0000 7f3cea934640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3ce40a1710 0x7f3ce4131990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:23.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.741+0000 7f3cea934640 10 monclient(hunting): picked mon.noname-c con 0x7f3ce40a1710 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:14:23.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.741+0000 7f3cea934640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3ce41340f0 0x7f3ce409ba50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:23.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.741+0000 7f3cea934640 10 monclient(hunting): picked mon.noname-b con 0x7f3ce41340f0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:14:23.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.741+0000 7f3cea934640 10 monclient(hunting): start opening mon connection 2024-09-16T04:14:23.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.741+0000 7f3cea934640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f3ce409d1b0 con 0x7f3ce40a1330 2024-09-16T04:14:23.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.741+0000 7f3cea934640 10 monclient(hunting): start opening mon connection 2024-09-16T04:14:23.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.741+0000 7f3cea934640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f3ce409d030 con 0x7f3ce41340f0 2024-09-16T04:14:23.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.741+0000 7f3cea934640 10 monclient(hunting): start opening mon connection 2024-09-16T04:14:23.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.741+0000 7f3cea934640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f3ce413dbd0 con 0x7f3ce40a1710 2024-09-16T04:14:23.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.741+0000 7f3cea934640 10 monclient(hunting): _renew_subs 2024-09-16T04:14:23.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.741+0000 7f3cea934640 10 monclient(hunting): authenticate will time out at 2910.713847s 2024-09-16T04:14:23.758 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.741+0000 7f3ce3fff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3ce40a1330 0x7f3ce4131450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:23.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.741+0000 7f3ce3fff640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3ce40a1330 0x7f3ce4131450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:48048/0 (socket says 172.21.15.42:48048) 2024-09-16T04:14:23.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.741+0000 7f3ce3fff640 1 -- 172.21.15.42:0/3204232820 learned_addr learned my addr 172.21.15.42:0/3204232820 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:14:23.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.741+0000 7f3ce3fff640 10 monclient(hunting): get_auth_request con 0x7f3ce40a1330 auth_method 0 2024-09-16T04:14:23.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.741+0000 7f3ce3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:14:23.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.741+0000 7f3ce3fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:23.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.741+0000 7f3ce3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:23.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.741+0000 7f3cdb7fe640 1 --2- 172.21.15.42:0/3204232820 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3ce40a1710 0x7f3ce4131990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:23.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.741+0000 7f3cdb7fe640 10 monclient(hunting): get_auth_request con 0x7f3ce40a1710 auth_method 0 2024-09-16T04:14:23.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.741+0000 7f3cdb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:14:23.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.741+0000 7f3cdb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:23.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.741+0000 7f3cdb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:23.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.741+0000 7f3ce3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:14:23.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.741+0000 7f3ce3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:14:23.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.741+0000 7f3ce3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:14:23.759 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.741+0000 7f3cdb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:14:23.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.741+0000 7f3cdb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:14:23.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.741+0000 7f3cdb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:14:23.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.741+0000 7f3ce3fff640 10 monclient(hunting): handle_auth_done global_id 1104201 payload 824 2024-09-16T04:14:23.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.741+0000 7f3ce3fff640 1 -- 172.21.15.42:0/3204232820 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3ce40a1710 msgr2=0x7f3ce4131990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:23.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.741+0000 7f3ce3fff640 1 --2- 172.21.15.42:0/3204232820 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3ce40a1710 0x7f3ce4131990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:23.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.741+0000 7f3ce3fff640 1 -- 172.21.15.42:0/3204232820 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3ce41340f0 msgr2=0x7f3ce409ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:23.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.741+0000 7f3ce3fff640 1 --2- 172.21.15.42:0/3204232820 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3ce41340f0 0x7f3ce409ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:23.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.741+0000 7f3ce3fff640 10 monclient: _finish_hunting 0 2024-09-16T04:14:23.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.741+0000 7f3ce3fff640 1 monclient: found mon.noname-a 2024-09-16T04:14:23.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.741+0000 7f3ce3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:14:23.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.741+0000 7f3ce3fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:14:23.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.741+0000 7f3ce3fff640 1 -- 172.21.15.42:0/3204232820 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ce4073a30 con 0x7f3ce40a1330 2024-09-16T04:14:23.760 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.741+0000 7f3cdb7fe640 1 --2- 172.21.15.42:0/3204232820 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3ce40a1710 0x7f3ce4131990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:14:23.761 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.741+0000 7f3ce3fff640 1 --2- 172.21.15.42:0/3204232820 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3ce40a1330 0x7f3ce4131450 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f3ccc0ae040 tx=0x7f3ccc019180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:14:23.761 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.741+0000 7f3ce17fa640 10 client.?.objecter ms_handle_connect 0x7f3ce40a1330 2024-09-16T04:14:23.761 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.741+0000 7f3ce17fa640 10 client.?.objecter resend_mon_ops 2024-09-16T04:14:23.761 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.741+0000 7f3ce17fa640 1 -- 172.21.15.42:0/3204232820 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ccc0b8020 con 0x7f3ce40a1330 2024-09-16T04:14:23.761 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.741+0000 7f3ce17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:23.761 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.741+0000 7f3ce17fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:14:23.761 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.741+0000 7f3ce17fa640 10 monclient: dump: 2024-09-16T04:14:23.761 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:14:23.761 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:14:23.761 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:23.761 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:23.761 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:23.761 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:23.762 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:14:23.762 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:14:23.762 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:14:23.762 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:23.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.741+0000 7f3ce17fa640 10 monclient: _finish_auth 0 2024-09-16T04:14:23.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.741+0000 7f3ce17fa640 10 monclient: _check_auth_tickets 2024-09-16T04:14:23.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.741+0000 7f3ce17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:14:23.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.741+0000 7f3ce17fa640 1 -- 172.21.15.42:0/3204232820 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3ccc0b4410 con 0x7f3ce40a1330 2024-09-16T04:14:23.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.741+0000 7f3ce17fa640 10 monclient: handle_config config(0 keys) 2024-09-16T04:14:23.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.741+0000 7f3ce17fa640 1 -- 172.21.15.42:0/3204232820 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ccc0b0040 con 0x7f3ce40a1330 2024-09-16T04:14:23.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.741+0000 7f3cea934640 5 monclient: authenticate success, global_id 1104201 2024-09-16T04:14:23.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.741+0000 7f3cea934640 10 monclient: _renew_subs 2024-09-16T04:14:23.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.741+0000 7f3cea934640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:14:23.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.741+0000 7f3cea934640 1 -- 172.21.15.42:0/3204232820 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ce40a0c70 con 0x7f3ce40a1330 2024-09-16T04:14:23.762 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.741+0000 7f3ce2ffd640 4 set_mon_vals no callback set 2024-09-16T04:14:23.763 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.741+0000 7f3ce17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:23.763 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.741+0000 7f3ce17fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:14:23.763 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.741+0000 7f3ce17fa640 10 monclient: dump: 2024-09-16T04:14:23.763 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:14:23.763 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:14:23.763 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:23.763 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:23.763 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:23.763 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:23.763 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:14:23.763 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:14:23.763 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:14:23.763 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:23.763 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.741+0000 7f3cea934640 10 client.1104201.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:14:23.764 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.741+0000 7f3cea934640 10 monclient: _renew_subs 2024-09-16T04:14:23.764 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.741+0000 7f3cea934640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:14:23.764 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.741+0000 7f3cea934640 1 -- 172.21.15.42:0/3204232820 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ce413a1c0 con 0x7f3ce40a1330 2024-09-16T04:14:23.764 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.741+0000 7f3cea934640 1 librados: init done 2024-09-16T04:14:23.764 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.741+0000 7f3cea934640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:14:23.764 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.741+0000 7f3cea934640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:14:23.764 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.741+0000 7f3cea934640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:14:23.764 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.741+0000 7f3ce17fa640 1 -- 172.21.15.42:0/3204232820 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f3ccc0bb020 con 0x7f3ce40a1330 2024-09-16T04:14:23.764 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.741+0000 7f3cea934640 1 -- 172.21.15.42:0/3204232820 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ce40a1b10 con 0x7f3ce40a1330 2024-09-16T04:14:23.764 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.741+0000 7f3ce17fa640 1 --2- 172.21.15.42:0/3204232820 >> v2:172.21.15.154:6812/254845342 conn(0x7f3cc4037840 0x7f3cc4039d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:23.764 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.741+0000 7f3ce17fa640 1 -- 172.21.15.42:0/3204232820 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(710..710 src has 681..710) ==== 4681+0+0 (secure 0 0 0) 0x7f3ccc0e25d0 con 0x7f3ce40a1330 2024-09-16T04:14:23.764 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.741+0000 7f3ce17fa640 10 client.1104201.objecter ms_dispatch 0x7f3ce4000ba0 osd_map(710..710 src has 681..710) 2024-09-16T04:14:23.764 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.741+0000 7f3ce17fa640 3 client.1104201.objecter handle_osd_map got epochs [710,710] > 0 2024-09-16T04:14:23.764 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.741+0000 7f3ce17fa640 3 client.1104201.objecter handle_osd_map decoding full epoch 710 2024-09-16T04:14:23.765 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.741+0000 7f3ce17fa640 20 client.1104201.objecter dump_active .. 0 homeless 2024-09-16T04:14:23.765 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.745+0000 7f3cdb7fe640 1 --2- 172.21.15.42:0/3204232820 >> v2:172.21.15.154:6812/254845342 conn(0x7f3cc4037840 0x7f3cc4039d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:23.765 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.745+0000 7f3cdb7fe640 10 monclient: get_auth_request con 0x7f3cc4037840 auth_method 0 2024-09-16T04:14:23.765 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.745+0000 7f3cdb7fe640 1 --2- 172.21.15.42:0/3204232820 >> v2:172.21.15.154:6812/254845342 conn(0x7f3cc4037840 0x7f3cc4039d00 secure :-1 s=READY pgs=4020 cs=0 l=1 rev1=1 crypto rx=0x7f3ce413a030 tx=0x7f3cd4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:14:23.765 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.745+0000 7f3ce17fa640 10 client.1104201.objecter ms_handle_connect 0x7f3cc4037840 2024-09-16T04:14:23.765 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.749+0000 7f3ce17fa640 1 -- 172.21.15.42:0/3204232820 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f3ccc0c2030 con 0x7f3ce40a1330 2024-09-16T04:14:23.765 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.749+0000 7f3ce17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:14:23.765 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.749+0000 7f3ce17fa640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:14:23.916 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.909+0000 7f3cea934640 1 -- 172.21.15.42:0/3204232820 --> v2:172.21.15.154:6812/254845342 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) -- 0x7f3ce4069040 con 0x7f3cc4037840 2024-09-16T04:14:23.916 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.909+0000 7f3ce17fa640 1 -- 172.21.15.42:0/3204232820 <== mgr.4102 v2:172.21.15.154:6812/254845342 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f3ce4069040 con 0x7f3cc4037840 2024-09-16T04:14:23.916 INFO:teuthology.orchestra.run.smithi042.stderr:instructed osd(s) 2 to deep-scrub 2024-09-16T04:14:23.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.913+0000 7f3cea934640 10 librados: watch_flush enter 2024-09-16T04:14:23.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.913+0000 7f3cea934640 10 librados: watch_flush exit 2024-09-16T04:14:23.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.913+0000 7f3cea934640 20 client.1104201.objecter shutdown clearing up homeless session... 2024-09-16T04:14:23.919 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.913+0000 7f3cea934640 10 client.1104201.objecter successfully canceled tick 2024-09-16T04:14:23.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.913+0000 7f3cea934640 1 -- 172.21.15.42:0/3204232820 >> v2:172.21.15.154:6812/254845342 conn(0x7f3cc4037840 msgr2=0x7f3cc4039d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:23.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.913+0000 7f3cea934640 1 --2- 172.21.15.42:0/3204232820 >> v2:172.21.15.154:6812/254845342 conn(0x7f3cc4037840 0x7f3cc4039d00 secure :-1 s=READY pgs=4020 cs=0 l=1 rev1=1 crypto rx=0x7f3ce413a030 tx=0x7f3cd4008040 comp rx=0 tx=0).stop 2024-09-16T04:14:23.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.913+0000 7f3cea934640 10 monclient: shutdown 2024-09-16T04:14:23.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.913+0000 7f3cea934640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:14:23.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.913+0000 7f3cea934640 1 -- 172.21.15.42:0/3204232820 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3ce40a1330 msgr2=0x7f3ce4131450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:23.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.913+0000 7f3cea934640 1 --2- 172.21.15.42:0/3204232820 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3ce40a1330 0x7f3ce4131450 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f3ccc0ae040 tx=0x7f3ccc019180 comp rx=0 tx=0).stop 2024-09-16T04:14:23.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.913+0000 7f3cea934640 1 -- 172.21.15.42:0/3204232820 shutdown_connections 2024-09-16T04:14:23.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.913+0000 7f3cea934640 1 --2- 172.21.15.42:0/3204232820 >> v2:172.21.15.154:6812/254845342 conn(0x7f3cc4037840 0x7f3cc4039d00 unknown :-1 s=CLOSED pgs=4020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:23.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.913+0000 7f3cea934640 1 --2- 172.21.15.42:0/3204232820 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f3ce41340f0 0x7f3ce409ba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:23.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.913+0000 7f3cea934640 1 --2- 172.21.15.42:0/3204232820 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f3ce40a1710 0x7f3ce4131990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:23.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.913+0000 7f3cea934640 1 --2- 172.21.15.42:0/3204232820 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f3ce40a1330 0x7f3ce4131450 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:23.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.913+0000 7f3cea934640 1 -- 172.21.15.42:0/3204232820 >> 172.21.15.42:0/3204232820 conn(0x7f3ce40a2420 msgr2=0x7f3ce413aca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:14:23.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.913+0000 7f3cea934640 1 -- 172.21.15.42:0/3204232820 shutdown_connections 2024-09-16T04:14:23.920 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.913+0000 7f3cea934640 1 -- 172.21.15.42:0/3204232820 wait complete. 2024-09-16T04:14:23.921 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:23.913+0000 7f3cea934640 1 librados: shutdown 2024-09-16T04:14:23.935 INFO:tasks.ceph:Scrubbing osd.3 2024-09-16T04:14:23.935 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-09-16T04:14:24.061 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.057+0000 7f82792aa640 10 monclient: get_monmap_and_config 2024-09-16T04:14:24.061 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.057+0000 7f82792aa640 10 monclient: build_initial_monmap 2024-09-16T04:14:24.061 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.057+0000 7f82792aa640 10 monclient: monmap: 2024-09-16T04:14:24.062 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:14:24.062 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:14:24.062 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:14:24.060447+0000 2024-09-16T04:14:24.062 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:14:24.060447+0000 2024-09-16T04:14:24.062 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:14:24.062 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:24.062 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:14:24.062 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:14:24.062 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:14:24.062 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:14:24.062 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:14:24.062 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:24.072 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.057+0000 7f82792aa640 1 Processor -- start 2024-09-16T04:14:24.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.057+0000 7f82792aa640 1 -- start start 2024-09-16T04:14:24.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.057+0000 7f82792aa640 10 monclient: init 2024-09-16T04:14:24.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.057+0000 7f82792aa640 10 monclient: _reopen_session rank -1 2024-09-16T04:14:24.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.057+0000 7f82792aa640 10 monclient: _add_conns ranks=[4,0,1,3,2] 2024-09-16T04:14:24.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.057+0000 7f82792aa640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f82740a2df0 0x7f8274134d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:24.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.057+0000 7f82792aa640 10 monclient(hunting): picked mon.noname-c con 0x7f82740a2df0 addr v2:172.21.15.154:3300/0 2024-09-16T04:14:24.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.057+0000 7f82792aa640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8274072da0 addr v1:172.21.15.42:6789/0 2024-09-16T04:14:24.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.057+0000 7f82792aa640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f82740731b0 addr v1:172.21.15.154:6789/0 2024-09-16T04:14:24.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.057+0000 7f82792aa640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f82740a56e0 con 0x7f8274072da0 2024-09-16T04:14:24.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.057+0000 7f82792aa640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f827409fd60 con 0x7f82740731b0 2024-09-16T04:14:24.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.057+0000 7f82792aa640 10 monclient(hunting): start opening mon connection 2024-09-16T04:14:24.073 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.057+0000 7f82792aa640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f8274073600 con 0x7f82740a2df0 2024-09-16T04:14:24.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.057+0000 7f82792aa640 10 monclient(hunting): _renew_subs 2024-09-16T04:14:24.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.057+0000 7f82792aa640 10 monclient(hunting): authenticate will time out at 2911.029893s 2024-09-16T04:14:24.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.057+0000 7f82727fc640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f8274072da0 0x7f82740a48c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:55182/0 (socket says 172.21.15.42:55182) 2024-09-16T04:14:24.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.057+0000 7f82727fc640 1 -- 172.21.15.42:0/4067354131 learned_addr learned my addr 172.21.15.42:0/4067354131 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T04:14:24.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.057+0000 7f8272ffd640 1 --2- 172.21.15.42:0/4067354131 >> v2:172.21.15.154:3300/0 conn(0x7f82740a2df0 0x7f8274134d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:24.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.057+0000 7f8271ffb640 1 -- 172.21.15.42:0/4067354131 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2881817587 0 0) 0x7f82740a56e0 con 0x7f8274072da0 2024-09-16T04:14:24.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.057+0000 7f8271ffb640 10 monclient(hunting): _negotiate 2024-09-16T04:14:24.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.057+0000 7f8271ffb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:24.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.057+0000 7f8271ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:24.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.057+0000 7f8271ffb640 10 monclient(hunting): my global_id is 1104204 2024-09-16T04:14:24.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.057+0000 7f8271ffb640 1 -- 172.21.15.42:0/4067354131 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f825c003530 con 0x7f8274072da0 2024-09-16T04:14:24.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.057+0000 7f8272ffd640 10 monclient(hunting): get_auth_request con 0x7f82740a2df0 auth_method 0 2024-09-16T04:14:24.074 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.057+0000 7f8272ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:14:24.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.057+0000 7f8272ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:24.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.057+0000 7f8272ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:24.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.057+0000 7f8272ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:14:24.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.057+0000 7f8272ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:14:24.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.057+0000 7f8272ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:14:24.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.057+0000 7f8271ffb640 1 -- 172.21.15.42:0/4067354131 <== mon.0 v1:172.21.15.42:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 306188767 0 0) 0x7f825c003530 con 0x7f8274072da0 2024-09-16T04:14:24.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.057+0000 7f8271ffb640 1 -- 172.21.15.42:0/4067354131 >> v2:172.21.15.154:3300/0 conn(0x7f82740a2df0 msgr2=0x7f8274134d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:14:24.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.057+0000 7f8271ffb640 1 --2- 172.21.15.42:0/4067354131 >> v2:172.21.15.154:3300/0 conn(0x7f82740a2df0 0x7f8274134d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:24.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.057+0000 7f8271ffb640 1 -- 172.21.15.42:0/4067354131 >> v1:172.21.15.154:6789/0 conn(0x7f82740731b0 legacy=0x7f82740a4fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:24.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.057+0000 7f8271ffb640 10 monclient: _finish_hunting 0 2024-09-16T04:14:24.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.057+0000 7f8271ffb640 1 monclient: found mon.noname-a-legacy 2024-09-16T04:14:24.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.057+0000 7f8271ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:14:24.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.057+0000 7f8271ffb640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.42:6789/0 2024-09-16T04:14:24.075 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.057+0000 7f8271ffb640 1 -- 172.21.15.42:0/4067354131 --> v1:172.21.15.42:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f827409f7a0 con 0x7f8274072da0 2024-09-16T04:14:24.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.057+0000 7f8271ffb640 10 monclient: _finish_auth 0 2024-09-16T04:14:24.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.057+0000 7f8271ffb640 10 monclient: _check_auth_tickets 2024-09-16T04:14:24.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.057+0000 7f8271ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:14:24.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.057+0000 7f8271ffb640 1 -- 172.21.15.42:0/4067354131 <== mon.0 v1:172.21.15.42:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2177413734 0 0) 0x7f8268001680 con 0x7f8274072da0 2024-09-16T04:14:24.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.057+0000 7f82792aa640 5 monclient: authenticate success, global_id 1104204 2024-09-16T04:14:24.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.057+0000 7f82792aa640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:14:24.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.057+0000 7f8271ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:24.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.057+0000 7f8271ffb640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-16T04:14:24.076 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.057+0000 7f8271ffb640 10 monclient: dump: 2024-09-16T04:14:24.076 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:14:24.076 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:14:24.076 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:24.076 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:24.076 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:24.077 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:24.077 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:14:24.077 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:14:24.077 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:14:24.077 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:24.077 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.057+0000 7f8271ffb640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] but i'm connected to v1:172.21.15.42:6789/0, reconnecting 2024-09-16T04:14:24.077 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.057+0000 7f8271ffb640 10 monclient: _reopen_session rank -1 2024-09-16T04:14:24.077 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.057+0000 7f8271ffb640 1 -- 172.21.15.42:0/4067354131 >> v1:172.21.15.42:6789/0 conn(0x7f8274072da0 legacy=0x7f82740a48c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:24.077 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.057+0000 7f8271ffb640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:14:24.077 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.057+0000 7f8271ffb640 1 --2- 172.21.15.42:0/4067354131 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f825c0039c0 0x7f825c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:24.077 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.057+0000 7f8271ffb640 10 monclient(hunting): picked mon.c con 0x7f825c0039c0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:14:24.077 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.057+0000 7f8271ffb640 1 --2- 172.21.15.42:0/4067354131 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f825c004810 0x7f825c004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:24.077 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.057+0000 7f8271ffb640 10 monclient(hunting): picked mon.a con 0x7f825c004810 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:14:24.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.057+0000 7f8271ffb640 1 --2- 172.21.15.42:0/4067354131 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f825c0052f0 0x7f825c00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:24.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.057+0000 7f8271ffb640 10 monclient(hunting): picked mon.b con 0x7f825c0052f0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:14:24.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.057+0000 7f8271ffb640 10 monclient(hunting): start opening mon connection 2024-09-16T04:14:24.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.057+0000 7f8271ffb640 1 -- 172.21.15.42:0/4067354131 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f8274073600 con 0x7f825c004810 2024-09-16T04:14:24.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.057+0000 7f8271ffb640 10 monclient(hunting): start opening mon connection 2024-09-16T04:14:24.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.057+0000 7f8271ffb640 1 -- 172.21.15.42:0/4067354131 --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f825c00c1c0 con 0x7f825c0052f0 2024-09-16T04:14:24.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.057+0000 7f8271ffb640 10 monclient(hunting): start opening mon connection 2024-09-16T04:14:24.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.057+0000 7f8271ffb640 1 -- 172.21.15.42:0/4067354131 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f825c00c3c0 con 0x7f825c0039c0 2024-09-16T04:14:24.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.057+0000 7f8271ffb640 10 monclient(hunting): _renew_subs 2024-09-16T04:14:24.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.057+0000 7f8271ffb640 10 monclient(hunting): _finish_auth 0 2024-09-16T04:14:24.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.057+0000 7f82792aa640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-16T04:14:24.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.057+0000 7f8272ffd640 1 --2- 172.21.15.42:0/4067354131 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f825c0039c0 0x7f825c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:24.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.057+0000 7f82727fc640 1 --2- 172.21.15.42:0/4067354131 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f825c004810 0x7f825c004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:24.078 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.057+0000 7f82727fc640 10 monclient(hunting): get_auth_request con 0x7f825c004810 auth_method 0 2024-09-16T04:14:24.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.057+0000 7f82727fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:14:24.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.057+0000 7f82727fc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:24.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.057+0000 7f82727fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:14:24.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.057+0000 7f8272ffd640 10 monclient(hunting): get_auth_request con 0x7f825c0039c0 auth_method 0 2024-09-16T04:14:24.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.057+0000 7f8272ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:14:24.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.057+0000 7f8272ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:24.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.057+0000 7f8272ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:14:24.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.057+0000 7f82727fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:14:24.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.057+0000 7f82727fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:14:24.079 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.057+0000 7f82727fc640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:14:24.080 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.057+0000 7f8272ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:14:24.080 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.057+0000 7f8272ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:14:24.080 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.057+0000 7f8272ffd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-16T04:14:24.080 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.057+0000 7f82737fe640 1 --2- 172.21.15.42:0/4067354131 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f825c0052f0 0x7f825c00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:24.080 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.057+0000 7f82737fe640 10 monclient(hunting): get_auth_request con 0x7f825c0052f0 auth_method 0 2024-09-16T04:14:24.080 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.057+0000 7f82737fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:14:24.080 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.057+0000 7f82737fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:24.080 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.057+0000 7f82737fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-16T04:14:24.080 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.057+0000 7f82727fc640 10 monclient(hunting): handle_auth_done global_id 1104204 payload 293 2024-09-16T04:14:24.080 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.057+0000 7f82727fc640 1 -- 172.21.15.42:0/4067354131 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f825c0039c0 msgr2=0x7f825c003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:14:24.081 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.057+0000 7f82727fc640 1 --2- 172.21.15.42:0/4067354131 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f825c0039c0 0x7f825c003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:24.081 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.057+0000 7f82727fc640 1 -- 172.21.15.42:0/4067354131 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f825c0052f0 msgr2=0x7f825c00b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:14:24.081 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.057+0000 7f82727fc640 1 --2- 172.21.15.42:0/4067354131 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f825c0052f0 0x7f825c00b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:24.081 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.057+0000 7f82727fc640 10 monclient: _finish_hunting 0 2024-09-16T04:14:24.081 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.057+0000 7f82727fc640 1 monclient: found mon.a 2024-09-16T04:14:24.081 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.057+0000 7f82727fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:14:24.081 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.057+0000 7f82727fc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:14:24.081 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.057+0000 7f82727fc640 1 -- 172.21.15.42:0/4067354131 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f825c003530 con 0x7f825c004810 2024-09-16T04:14:24.081 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.057+0000 7f82727fc640 10 monclient: _finish_auth 0 2024-09-16T04:14:24.081 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.057+0000 7f82727fc640 10 monclient: _check_auth_tickets 2024-09-16T04:14:24.081 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.057+0000 7f82737fe640 1 -- 172.21.15.42:0/4067354131 reap_dead start 2024-09-16T04:14:24.081 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.057+0000 7f82727fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:14:24.081 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.057+0000 7f8272ffd640 1 --2- 172.21.15.42:0/4067354131 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f825c0039c0 0x7f825c003de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:14:24.081 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.061+0000 7f82727fc640 1 --2- 172.21.15.42:0/4067354131 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f825c004810 0x7f825c004c90 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f826800c210 tx=0x7f826801cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d28ddb78745382cc server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:14:24.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.061+0000 7f8271ffb640 1 -- 172.21.15.42:0/4067354131 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8268007d20 con 0x7f825c004810 2024-09-16T04:14:24.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.061+0000 7f8271ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:24.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.061+0000 7f8271ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:14:24.082 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.061+0000 7f8271ffb640 10 monclient: dump: 2024-09-16T04:14:24.082 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:14:24.082 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:14:24.082 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:24.082 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:24.082 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:24.082 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:24.082 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:14:24.082 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:14:24.082 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:14:24.082 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:24.083 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.061+0000 7f82792aa640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:14:24.083 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.061+0000 7f8271ffb640 1 -- 172.21.15.42:0/4067354131 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f82680a9070 con 0x7f825c004810 2024-09-16T04:14:24.083 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.061+0000 7f8271ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:14:24.083 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.061+0000 7f8271ffb640 1 -- 172.21.15.42:0/4067354131 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8268005da0 con 0x7f825c004810 2024-09-16T04:14:24.083 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.061+0000 7f8271ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:24.083 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.061+0000 7f8271ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:14:24.083 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.061+0000 7f8271ffb640 10 monclient: dump: 2024-09-16T04:14:24.083 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:14:24.083 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:14:24.083 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:24.083 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:24.083 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:24.083 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:24.084 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:14:24.084 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:14:24.084 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:14:24.084 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:24.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.061+0000 7f82792aa640 10 monclient: get_monmap_and_config success 2024-09-16T04:14:24.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.061+0000 7f82792aa640 4 set_mon_vals no callback set 2024-09-16T04:14:24.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.061+0000 7f82792aa640 10 monclient: shutdown 2024-09-16T04:14:24.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.061+0000 7f82792aa640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:14:24.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.061+0000 7f82792aa640 1 -- 172.21.15.42:0/4067354131 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f825c004810 msgr2=0x7f825c004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:24.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.061+0000 7f82792aa640 1 --2- 172.21.15.42:0/4067354131 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f825c004810 0x7f825c004c90 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f826800c210 tx=0x7f826801cc40 comp rx=0 tx=0).stop 2024-09-16T04:14:24.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.061+0000 7f82792aa640 1 -- 172.21.15.42:0/4067354131 shutdown_connections 2024-09-16T04:14:24.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.061+0000 7f82792aa640 1 --2- 172.21.15.42:0/4067354131 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f825c004810 0x7f825c004c90 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:24.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.061+0000 7f82792aa640 1 -- 172.21.15.42:0/4067354131 >> 172.21.15.42:0/4067354131 conn(0x7f82740a6300 msgr2=0x7f82740a24b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:14:24.084 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.061+0000 7f82792aa640 1 -- 172.21.15.42:0/4067354131 shutdown_connections 2024-09-16T04:14:24.085 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.061+0000 7f82792aa640 1 -- 172.21.15.42:0/4067354131 wait complete. 2024-09-16T04:14:24.085 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.061+0000 7f82792aa640 10 monclient: build_initial_monmap 2024-09-16T04:14:24.085 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.061+0000 7f82792aa640 10 monclient: monmap: 2024-09-16T04:14:24.085 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:14:24.085 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:14:24.085 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:14:24.085 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:14:24.085 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:14:24.085 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:24.085 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:14:24.085 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:14:24.085 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:14:24.085 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:24.085 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.061+0000 7f82792aa640 1 librados: starting msgr at 2024-09-16T04:14:24.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.061+0000 7f82792aa640 1 librados: starting objecter 2024-09-16T04:14:24.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.061+0000 7f82792aa640 1 Processor -- start 2024-09-16T04:14:24.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.061+0000 7f82792aa640 1 -- start start 2024-09-16T04:14:24.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.061+0000 7f82792aa640 1 librados: setting wanted keys 2024-09-16T04:14:24.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.061+0000 7f82792aa640 1 librados: calling monclient init 2024-09-16T04:14:24.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.061+0000 7f82792aa640 10 monclient: init 2024-09-16T04:14:24.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.061+0000 7f82792aa640 10 monclient: _reopen_session rank -1 2024-09-16T04:14:24.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.065+0000 7f82792aa640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:14:24.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.065+0000 7f82792aa640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f825c004810 0x7f82741cd520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:24.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.065+0000 7f82792aa640 10 monclient(hunting): picked mon.noname-c con 0x7f825c004810 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:14:24.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.065+0000 7f82792aa640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f827412e220 0x7f82741bde30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:24.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.065+0000 7f82792aa640 10 monclient(hunting): picked mon.noname-b con 0x7f827412e220 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:14:24.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.065+0000 7f82792aa640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f827412e660 0x7f82741be370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:24.086 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.065+0000 7f82792aa640 10 monclient(hunting): picked mon.noname-a con 0x7f827412e660 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:14:24.087 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.065+0000 7f82792aa640 10 monclient(hunting): start opening mon connection 2024-09-16T04:14:24.087 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.065+0000 7f82792aa640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f82740a54a0 con 0x7f827412e660 2024-09-16T04:14:24.087 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.065+0000 7f82792aa640 10 monclient(hunting): start opening mon connection 2024-09-16T04:14:24.087 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.065+0000 7f82792aa640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f82740a5320 con 0x7f827412e220 2024-09-16T04:14:24.087 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.065+0000 7f82792aa640 10 monclient(hunting): start opening mon connection 2024-09-16T04:14:24.087 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.065+0000 7f82792aa640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f8274137fe0 con 0x7f825c004810 2024-09-16T04:14:24.087 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.065+0000 7f82792aa640 10 monclient(hunting): _renew_subs 2024-09-16T04:14:24.087 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.065+0000 7f82792aa640 10 monclient(hunting): authenticate will time out at 2911.036144s 2024-09-16T04:14:24.087 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.065+0000 7f8272ffd640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f825c004810 0x7f82741cd520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:24.087 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.065+0000 7f8272ffd640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f825c004810 0x7f82741cd520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:33200/0 (socket says 172.21.15.42:33200) 2024-09-16T04:14:24.087 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.065+0000 7f82737fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f827412e660 0x7f82741be370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:24.087 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.065+0000 7f8272ffd640 1 -- 172.21.15.42:0/2270061752 learned_addr learned my addr 172.21.15.42:0/2270061752 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:14:24.087 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.065+0000 7f8272ffd640 10 monclient(hunting): get_auth_request con 0x7f825c004810 auth_method 0 2024-09-16T04:14:24.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.065+0000 7f8272ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:14:24.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.065+0000 7f8272ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:24.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.065+0000 7f8272ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:24.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.065+0000 7f82737fe640 10 monclient(hunting): get_auth_request con 0x7f827412e660 auth_method 0 2024-09-16T04:14:24.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.065+0000 7f82737fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:14:24.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.065+0000 7f82737fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:24.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.065+0000 7f82737fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:24.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.065+0000 7f8272ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:14:24.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.065+0000 7f8272ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:14:24.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.065+0000 7f8272ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:14:24.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.065+0000 7f82737fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:14:24.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.065+0000 7f82737fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:14:24.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.065+0000 7f82737fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:14:24.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.065+0000 7f82737fe640 10 monclient(hunting): handle_auth_done global_id 1104207 payload 824 2024-09-16T04:14:24.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.065+0000 7f82727fc640 1 --2- 172.21.15.42:0/2270061752 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f827412e220 0x7f82741bde30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:24.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.065+0000 7f82737fe640 1 -- 172.21.15.42:0/2270061752 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f825c004810 msgr2=0x7f82741cd520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:24.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.065+0000 7f82737fe640 1 --2- 172.21.15.42:0/2270061752 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f825c004810 0x7f82741cd520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:24.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.065+0000 7f82737fe640 1 -- 172.21.15.42:0/2270061752 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f827412e220 msgr2=0x7f82741bde30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:24.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.065+0000 7f82737fe640 1 --2- 172.21.15.42:0/2270061752 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f827412e220 0x7f82741bde30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:24.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.065+0000 7f82737fe640 10 monclient: _finish_hunting 0 2024-09-16T04:14:24.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.065+0000 7f82737fe640 1 monclient: found mon.noname-a 2024-09-16T04:14:24.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.065+0000 7f82737fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:14:24.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.065+0000 7f82737fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:14:24.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.065+0000 7f82737fe640 1 -- 172.21.15.42:0/2270061752 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82741c1dc0 con 0x7f827412e660 2024-09-16T04:14:24.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.065+0000 7f8272ffd640 1 --2- 172.21.15.42:0/2270061752 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f825c004810 0x7f82741cd520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:14:24.089 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.065+0000 7f82727fc640 10 monclient: get_auth_request con 0x7f827412e220 auth_method 0 2024-09-16T04:14:24.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.065+0000 7f82727fc640 1 --2- 172.21.15.42:0/2270061752 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f827412e220 0x7f82741bde30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T04:14:24.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.065+0000 7f82737fe640 1 --2- 172.21.15.42:0/2270061752 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f827412e660 0x7f82741be370 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f827406c7f0 tx=0x7f826c00e9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:14:24.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.065+0000 7f825b7fe640 10 client.?.objecter ms_handle_connect 0x7f827412e660 2024-09-16T04:14:24.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.065+0000 7f825b7fe640 10 client.?.objecter resend_mon_ops 2024-09-16T04:14:24.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.065+0000 7f825b7fe640 1 -- 172.21.15.42:0/2270061752 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f826c019070 con 0x7f827412e660 2024-09-16T04:14:24.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.065+0000 7f825b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:24.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.065+0000 7f825b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:14:24.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.065+0000 7f825b7fe640 10 monclient: dump: 2024-09-16T04:14:24.090 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:14:24.090 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:14:24.090 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:24.090 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:24.090 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:24.090 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:24.091 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:14:24.091 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:14:24.091 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:14:24.091 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:24.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.065+0000 7f825b7fe640 10 monclient: _finish_auth 0 2024-09-16T04:14:24.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.065+0000 7f825b7fe640 10 monclient: _check_auth_tickets 2024-09-16T04:14:24.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.065+0000 7f825b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:14:24.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.065+0000 7f825b7fe640 1 -- 172.21.15.42:0/2270061752 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f826c006e30 con 0x7f827412e660 2024-09-16T04:14:24.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.065+0000 7f825b7fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:14:24.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.065+0000 7f825b7fe640 1 -- 172.21.15.42:0/2270061752 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f826c003b80 con 0x7f827412e660 2024-09-16T04:14:24.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.065+0000 7f825b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:24.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.065+0000 7f825b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:14:24.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.065+0000 7f825b7fe640 10 monclient: dump: 2024-09-16T04:14:24.091 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:14:24.091 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:14:24.092 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:24.092 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:24.092 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:24.092 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:24.092 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:14:24.092 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:14:24.092 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:14:24.092 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:24.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.065+0000 7f82792aa640 5 monclient: authenticate success, global_id 1104207 2024-09-16T04:14:24.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.065+0000 7f82792aa640 10 monclient: _renew_subs 2024-09-16T04:14:24.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.065+0000 7f82792aa640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:14:24.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.065+0000 7f82792aa640 1 -- 172.21.15.42:0/2270061752 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82740a15f0 con 0x7f827412e660 2024-09-16T04:14:24.092 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.065+0000 7f82717fa640 4 set_mon_vals no callback set 2024-09-16T04:14:24.093 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.065+0000 7f82792aa640 10 client.1104207.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:14:24.093 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.065+0000 7f82792aa640 10 monclient: _renew_subs 2024-09-16T04:14:24.093 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.065+0000 7f82792aa640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:14:24.093 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.065+0000 7f82792aa640 1 -- 172.21.15.42:0/2270061752 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82741c1ae0 con 0x7f827412e660 2024-09-16T04:14:24.093 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.065+0000 7f82792aa640 1 librados: init done 2024-09-16T04:14:24.093 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.065+0000 7f82792aa640 10 client.1104207.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:14:24.093 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.065+0000 7f82792aa640 15 client.1104207.objecter _session_command_op_assign -1 1 2024-09-16T04:14:24.093 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.065+0000 7f82792aa640 20 client.1104207.objecter _get_session osd=-1 returning homeless 2024-09-16T04:14:24.093 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.065+0000 7f82792aa640 10 client.1104207.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:14:24.093 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.065+0000 7f82792aa640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:14:24.093 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.065+0000 7f82792aa640 1 -- 172.21.15.42:0/2270061752 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f8274073aa0 con 0x7f827412e660 2024-09-16T04:14:24.093 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.065+0000 7f825b7fe640 1 -- 172.21.15.42:0/2270061752 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f826c005110 con 0x7f827412e660 2024-09-16T04:14:24.093 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.065+0000 7f825b7fe640 1 --2- 172.21.15.42:0/2270061752 >> v2:172.21.15.154:6812/254845342 conn(0x7f8260037810 0x7f8260039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:24.093 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.065+0000 7f825b7fe640 1 -- 172.21.15.42:0/2270061752 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(710..710 src has 681..710) ==== 4681+0+0 (secure 0 0 0) 0x7f826c0480a0 con 0x7f827412e660 2024-09-16T04:14:24.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.065+0000 7f825b7fe640 10 client.1104207.objecter ms_dispatch 0x7f8274000ba0 osd_map(710..710 src has 681..710) 2024-09-16T04:14:24.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.065+0000 7f825b7fe640 3 client.1104207.objecter handle_osd_map got epochs [710,710] > 0 2024-09-16T04:14:24.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.065+0000 7f825b7fe640 3 client.1104207.objecter handle_osd_map decoding full epoch 710 2024-09-16T04:14:24.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.065+0000 7f825b7fe640 10 client.1104207.objecter checking command 1 2024-09-16T04:14:24.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.065+0000 7f825b7fe640 1 --2- 172.21.15.42:0/2270061752 >> v2:172.21.15.154:6800/4043102363 conn(0x7f826003ec70 0x7f8260041110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:24.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.065+0000 7f825b7fe640 20 client.1104207.objecter _get_session s=0x7f826003e5c0 osd=3 3 2024-09-16T04:14:24.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.065+0000 7f825b7fe640 20 client.1104207.objecter put_session s=0x7f826003e5c0 osd=3 3 2024-09-16T04:14:24.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.065+0000 7f825b7fe640 15 client.1104207.objecter _session_command_op_remove -1 1 2024-09-16T04:14:24.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.065+0000 7f825b7fe640 20 client.1104207.objecter _get_session s=0x7f826003e5c0 osd=3 3 2024-09-16T04:14:24.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.065+0000 7f825b7fe640 20 client.1104207.objecter get_session s=0x7f826003e5c0 osd=3 3 2024-09-16T04:14:24.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.065+0000 7f825b7fe640 15 client.1104207.objecter _session_command_op_assign 3 1 2024-09-16T04:14:24.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.065+0000 7f825b7fe640 20 client.1104207.objecter put_session s=0x7f826003e5c0 osd=3 4 2024-09-16T04:14:24.094 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.065+0000 7f825b7fe640 10 client.1104207.objecter _send_command 1 2024-09-16T04:14:24.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.065+0000 7f825b7fe640 1 -- 172.21.15.42:0/2270061752 --> v2:172.21.15.154:6800/4043102363 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f826c015e70 con 0x7f826003ec70 2024-09-16T04:14:24.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.065+0000 7f825b7fe640 20 client.1104207.objecter dump_active .. 0 homeless 2024-09-16T04:14:24.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.065+0000 7f825b7fe640 1 -- 172.21.15.42:0/2270061752 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_get_version_reply(handle=1 version=710) ==== 24+0+0 (secure 0 0 0) 0x7f826c04f050 con 0x7f827412e660 2024-09-16T04:14:24.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.065+0000 7f825b7fe640 10 monclient: handle_get_version_reply finishing 1 version 710 2024-09-16T04:14:24.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.065+0000 7f8272ffd640 1 --2- 172.21.15.42:0/2270061752 >> v2:172.21.15.154:6812/254845342 conn(0x7f8260037810 0x7f8260039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:24.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.065+0000 7f8272ffd640 10 monclient: get_auth_request con 0x7f8260037810 auth_method 0 2024-09-16T04:14:24.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.065+0000 7f82727fc640 1 --2- 172.21.15.42:0/2270061752 >> v2:172.21.15.154:6800/4043102363 conn(0x7f826003ec70 0x7f8260041110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:24.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.065+0000 7f82727fc640 10 monclient: get_auth_request con 0x7f826003ec70 auth_method 0 2024-09-16T04:14:24.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.065+0000 7f825b7fe640 10 client.1104207.objecter ms_handle_connect 0x7f8260037810 2024-09-16T04:14:24.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.065+0000 7f8272ffd640 1 --2- 172.21.15.42:0/2270061752 >> v2:172.21.15.154:6812/254845342 conn(0x7f8260037810 0x7f8260039cd0 secure :-1 s=READY pgs=4021 cs=0 l=1 rev1=1 crypto rx=0x7f8264009fd0 tx=0x7f8264009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:14:24.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.065+0000 7f825b7fe640 10 client.1104207.objecter ms_handle_connect 0x7f826003ec70 2024-09-16T04:14:24.095 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.065+0000 7f82727fc640 1 --2- 172.21.15.42:0/2270061752 >> v2:172.21.15.154:6800/4043102363 conn(0x7f826003ec70 0x7f8260041110 crc :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:14:24.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.069+0000 7f825b7fe640 1 -- 172.21.15.42:0/2270061752 <== osd.3 v2:172.21.15.154:6800/4043102363 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f826c015e70 con 0x7f826003ec70 2024-09-16T04:14:24.096 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.069+0000 7f825b7fe640 10 client.1104207.objecter ms_dispatch 0x7f8274000ba0 command_reply(tid 1: 0 ) 2024-09-16T04:14:24.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.069+0000 7f825b7fe640 10 client.1104207.objecter _finish_command 1 = system:0 2024-09-16T04:14:24.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.069+0000 7f825b7fe640 20 client.1104207.objecter put_session s=0x7f826003e5c0 osd=3 4 2024-09-16T04:14:24.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.069+0000 7f825b7fe640 15 client.1104207.objecter _session_command_op_remove 3 1 2024-09-16T04:14:24.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.081+0000 7f82792aa640 10 client.1104207.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-09-16T04:14:24.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.081+0000 7f82792aa640 15 client.1104207.objecter _session_command_op_assign -1 2 2024-09-16T04:14:24.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.081+0000 7f82792aa640 20 client.1104207.objecter _get_session s=0x7f826003e5c0 osd=3 3 2024-09-16T04:14:24.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.081+0000 7f82792aa640 20 client.1104207.objecter put_session s=0x7f826003e5c0 osd=3 3 2024-09-16T04:14:24.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.081+0000 7f82792aa640 20 client.1104207.objecter _get_session s=0x7f826003e5c0 osd=3 3 2024-09-16T04:14:24.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.081+0000 7f82792aa640 15 client.1104207.objecter _session_command_op_remove -1 2 2024-09-16T04:14:24.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.081+0000 7f82792aa640 20 client.1104207.objecter get_session s=0x7f826003e5c0 osd=3 3 2024-09-16T04:14:24.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.081+0000 7f82792aa640 15 client.1104207.objecter _session_command_op_assign 3 2 2024-09-16T04:14:24.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.081+0000 7f82792aa640 20 client.1104207.objecter put_session s=0x7f826003e5c0 osd=3 4 2024-09-16T04:14:24.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.081+0000 7f82792aa640 10 client.1104207.objecter _send_command 2 2024-09-16T04:14:24.097 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.081+0000 7f82792aa640 1 -- 172.21.15.42:0/2270061752 --> v2:172.21.15.154:6800/4043102363 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f82740a56b0 con 0x7f826003ec70 2024-09-16T04:14:24.098 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.081+0000 7f825b7fe640 1 -- 172.21.15.42:0/2270061752 <== osd.3 v2:172.21.15.154:6800/4043102363 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f82740a56b0 con 0x7f826003ec70 2024-09-16T04:14:24.098 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.081+0000 7f825b7fe640 10 client.1104207.objecter ms_dispatch 0x7f8274000ba0 command_reply(tid 2: 0 ) 2024-09-16T04:14:24.098 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.081+0000 7f825b7fe640 10 client.1104207.objecter _finish_command 2 = system:0 2024-09-16T04:14:24.098 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.081+0000 7f825b7fe640 20 client.1104207.objecter put_session s=0x7f826003e5c0 osd=3 4 2024-09-16T04:14:24.098 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.081+0000 7f825b7fe640 15 client.1104207.objecter _session_command_op_remove 3 2 2024-09-16T04:14:24.098 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T04:14:24.098 INFO:teuthology.orchestra.run.smithi042.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T04:14:24.098 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T04:14:24.099 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.081+0000 7f82792aa640 10 librados: watch_flush enter 2024-09-16T04:14:24.099 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.081+0000 7f82792aa640 10 librados: watch_flush exit 2024-09-16T04:14:24.099 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.081+0000 7f82792aa640 10 client.1104207.objecter close_session for osd.3 2024-09-16T04:14:24.100 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.081+0000 7f82792aa640 1 -- 172.21.15.42:0/2270061752 >> v2:172.21.15.154:6800/4043102363 conn(0x7f826003ec70 msgr2=0x7f8260041110 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:24.100 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.081+0000 7f82792aa640 1 --2- 172.21.15.42:0/2270061752 >> v2:172.21.15.154:6800/4043102363 conn(0x7f826003ec70 0x7f8260041110 crc :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:24.100 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.081+0000 7f82792aa640 20 client.1104207.objecter put_session s=0x7f826003e5c0 osd=3 1 2024-09-16T04:14:24.100 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.081+0000 7f82792aa640 20 client.1104207.objecter shutdown clearing up homeless session... 2024-09-16T04:14:24.100 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.081+0000 7f82792aa640 10 client.1104207.objecter successfully canceled tick 2024-09-16T04:14:24.100 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.081+0000 7f82792aa640 1 -- 172.21.15.42:0/2270061752 >> v2:172.21.15.154:6812/254845342 conn(0x7f8260037810 msgr2=0x7f8260039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:24.100 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.081+0000 7f82792aa640 1 --2- 172.21.15.42:0/2270061752 >> v2:172.21.15.154:6812/254845342 conn(0x7f8260037810 0x7f8260039cd0 secure :-1 s=READY pgs=4021 cs=0 l=1 rev1=1 crypto rx=0x7f8264009fd0 tx=0x7f8264009190 comp rx=0 tx=0).stop 2024-09-16T04:14:24.100 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.081+0000 7f82792aa640 10 monclient: shutdown 2024-09-16T04:14:24.100 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.081+0000 7f82792aa640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:14:24.100 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.081+0000 7f82792aa640 1 -- 172.21.15.42:0/2270061752 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f827412e660 msgr2=0x7f82741be370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:24.100 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.081+0000 7f82792aa640 1 --2- 172.21.15.42:0/2270061752 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f827412e660 0x7f82741be370 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f827406c7f0 tx=0x7f826c00e9e0 comp rx=0 tx=0).stop 2024-09-16T04:14:24.100 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.081+0000 7f82737fe640 1 -- 172.21.15.42:0/2270061752 reap_dead start 2024-09-16T04:14:24.100 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.081+0000 7f82792aa640 1 -- 172.21.15.42:0/2270061752 shutdown_connections 2024-09-16T04:14:24.101 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.081+0000 7f82792aa640 1 -- 172.21.15.42:0/2270061752 >> 172.21.15.42:0/2270061752 conn(0x7f82740a6300 msgr2=0x7f827412f8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:14:24.101 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.081+0000 7f82792aa640 1 -- 172.21.15.42:0/2270061752 shutdown_connections 2024-09-16T04:14:24.101 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.081+0000 7f82792aa640 1 -- 172.21.15.42:0/2270061752 wait complete. 2024-09-16T04:14:24.101 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.081+0000 7f82792aa640 1 librados: shutdown 2024-09-16T04:14:24.104 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-09-16T04:14:24.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.205+0000 7f7cb1db8640 10 monclient: get_monmap_and_config 2024-09-16T04:14:24.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.205+0000 7f7cb1db8640 10 monclient: build_initial_monmap 2024-09-16T04:14:24.212 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.205+0000 7f7cb1db8640 10 monclient: monmap: 2024-09-16T04:14:24.213 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:14:24.213 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:14:24.213 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:14:24.211522+0000 2024-09-16T04:14:24.213 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:14:24.211522+0000 2024-09-16T04:14:24.213 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:14:24.213 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:24.213 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:14:24.213 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:14:24.213 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:14:24.213 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:14:24.213 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:14:24.213 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:24.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.209+0000 7f7cb1db8640 1 Processor -- start 2024-09-16T04:14:24.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.209+0000 7f7cb1db8640 1 -- start start 2024-09-16T04:14:24.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.209+0000 7f7cb1db8640 10 monclient: init 2024-09-16T04:14:24.215 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.209+0000 7f7cb1db8640 10 monclient: _reopen_session rank -1 2024-09-16T04:14:24.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.209+0000 7f7cb1db8640 10 monclient: _add_conns ranks=[3,0,4,1,2] 2024-09-16T04:14:24.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.209+0000 7f7cb1db8640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f7cac0a2340 0x7f7cac072180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:24.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.209+0000 7f7cb1db8640 10 monclient(hunting): picked mon.noname-a con 0x7f7cac0a2340 addr v2:172.21.15.42:3300/0 2024-09-16T04:14:24.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.209+0000 7f7cb1db8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7cac0a2720 addr v1:172.21.15.42:6789/0 2024-09-16T04:14:24.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.209+0000 7f7cb1db8640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f7cac13a340 0x7f7cac13c730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:24.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.209+0000 7f7cb1db8640 10 monclient(hunting): picked mon.noname-c con 0x7f7cac13a340 addr v2:172.21.15.154:3300/0 2024-09-16T04:14:24.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.209+0000 7f7cb1db8640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7cac06cb00 con 0x7f7cac0a2720 2024-09-16T04:14:24.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.209+0000 7f7cb1db8640 10 monclient(hunting): start opening mon connection 2024-09-16T04:14:24.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.209+0000 7f7cb1db8640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f7cac06c450 con 0x7f7cac0a2340 2024-09-16T04:14:24.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.209+0000 7f7cb1db8640 10 monclient(hunting): start opening mon connection 2024-09-16T04:14:24.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.209+0000 7f7cb1db8640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f7cac070b90 con 0x7f7cac13a340 2024-09-16T04:14:24.216 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.209+0000 7f7cb1db8640 10 monclient(hunting): _renew_subs 2024-09-16T04:14:24.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.209+0000 7f7cb1db8640 10 monclient(hunting): authenticate will time out at 2911.180613s 2024-09-16T04:14:24.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.209+0000 7f7cab7fe640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f7cac0a2340 0x7f7cac072180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:24.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.209+0000 7f7cab7fe640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f7cac0a2340 0x7f7cac072180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:48544/0 (socket says 172.21.15.42:48544) 2024-09-16T04:14:24.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.209+0000 7f7cab7fe640 1 -- 172.21.15.42:0/713970769 learned_addr learned my addr 172.21.15.42:0/713970769 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:14:24.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.209+0000 7f7cab7fe640 10 monclient(hunting): get_auth_request con 0x7f7cac0a2340 auth_method 0 2024-09-16T04:14:24.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.209+0000 7f7cab7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:14:24.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.209+0000 7f7cab7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:24.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.209+0000 7f7cab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:24.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.209+0000 7f7cab7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:14:24.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.209+0000 7f7cab7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:14:24.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.209+0000 7f7cab7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:14:24.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.209+0000 7f7cab7fe640 10 monclient(hunting): handle_auth_done global_id 1104210 payload 274 2024-09-16T04:14:24.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.209+0000 7f7cabfff640 1 --2- 172.21.15.42:0/713970769 >> v2:172.21.15.154:3300/0 conn(0x7f7cac13a340 0x7f7cac13c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:24.217 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.209+0000 7f7cab7fe640 1 -- 172.21.15.42:0/713970769 >> v2:172.21.15.154:3300/0 conn(0x7f7cac13a340 msgr2=0x7f7cac13c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:14:24.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.209+0000 7f7cab7fe640 1 --2- 172.21.15.42:0/713970769 >> v2:172.21.15.154:3300/0 conn(0x7f7cac13a340 0x7f7cac13c730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:24.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.209+0000 7f7cab7fe640 1 -- 172.21.15.42:0/713970769 >> v1:172.21.15.42:6789/0 conn(0x7f7cac0a2720 legacy=0x7f7cac0a2b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:24.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.209+0000 7f7cab7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:14:24.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.209+0000 7f7cab7fe640 1 monclient: found mon.noname-a 2024-09-16T04:14:24.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.209+0000 7f7caa7fc640 1 -- 172.21.15.42:0/713970769 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 733321223 0 0) 0x7f7cac06cb00 con 0x7f7cac0a2720 2024-09-16T04:14:24.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.209+0000 7f7cab7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:14:24.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.209+0000 7f7cab7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:14:24.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.209+0000 7f7cab7fe640 1 -- 172.21.15.42:0/713970769 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7cac0726c0 con 0x7f7cac0a2340 2024-09-16T04:14:24.218 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.209+0000 7f7caa7fc640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-16T04:14:24.223 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.209+0000 7f7cab7fe640 1 --2- 172.21.15.42:0/713970769 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7cac0a2340 0x7f7cac072180 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f7cac06c260 tx=0x7f7ca401c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb6954b232ccf297 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:14:24.223 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.209+0000 7f7caa7fc640 1 -- 172.21.15.42:0/713970769 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ca40ac070 con 0x7f7cac0a2340 2024-09-16T04:14:24.223 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.209+0000 7f7caa7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:24.223 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.209+0000 7f7caa7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:14:24.223 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.209+0000 7f7caa7fc640 10 monclient: dump: 2024-09-16T04:14:24.223 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:14:24.223 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:14:24.223 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:24.224 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:24.224 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:24.224 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:24.224 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:14:24.224 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:14:24.224 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:14:24.224 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:24.224 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.209+0000 7f7caa7fc640 10 monclient: _finish_auth 0 2024-09-16T04:14:24.224 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.209+0000 7f7caa7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:14:24.224 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.209+0000 7f7caa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:14:24.224 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.209+0000 7f7cb1db8640 5 monclient: authenticate success, global_id 1104210 2024-09-16T04:14:24.224 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.209+0000 7f7cb1db8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:14:24.224 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.209+0000 7f7caa7fc640 1 -- 172.21.15.42:0/713970769 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ca40a6470 con 0x7f7cac0a2340 2024-09-16T04:14:24.224 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.209+0000 7f7caa7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:14:24.225 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.209+0000 7f7cb1db8640 10 monclient: get_monmap_and_config success 2024-09-16T04:14:24.225 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.209+0000 7f7caa7fc640 1 -- 172.21.15.42:0/713970769 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ca40a6840 con 0x7f7cac0a2340 2024-09-16T04:14:24.225 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.209+0000 7f7cb1db8640 4 set_mon_vals no callback set 2024-09-16T04:14:24.225 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.209+0000 7f7caa7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:24.225 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.209+0000 7f7caa7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:14:24.225 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.209+0000 7f7caa7fc640 10 monclient: dump: 2024-09-16T04:14:24.225 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:14:24.225 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:14:24.225 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:24.225 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:24.225 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:24.225 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:24.225 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:14:24.226 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:14:24.226 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:14:24.226 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:24.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.209+0000 7f7cb1db8640 10 monclient: shutdown 2024-09-16T04:14:24.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.209+0000 7f7cb1db8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:14:24.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.209+0000 7f7cb1db8640 1 -- 172.21.15.42:0/713970769 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7cac0a2340 msgr2=0x7f7cac072180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:24.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.209+0000 7f7cb1db8640 1 --2- 172.21.15.42:0/713970769 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7cac0a2340 0x7f7cac072180 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f7cac06c260 tx=0x7f7ca401c8b0 comp rx=0 tx=0).stop 2024-09-16T04:14:24.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.209+0000 7f7cb1db8640 1 -- 172.21.15.42:0/713970769 shutdown_connections 2024-09-16T04:14:24.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.209+0000 7f7cb1db8640 1 --2- 172.21.15.42:0/713970769 >> v2:172.21.15.154:3300/0 conn(0x7f7cac13a340 0x7f7cac13c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:24.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.209+0000 7f7cb1db8640 1 --2- 172.21.15.42:0/713970769 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7cac0a2340 0x7f7cac072180 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:24.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.209+0000 7f7cb1db8640 1 -- 172.21.15.42:0/713970769 >> 172.21.15.42:0/713970769 conn(0x7f7cac0a3430 msgr2=0x7f7cac0a3810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:14:24.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.209+0000 7f7cb1db8640 1 -- 172.21.15.42:0/713970769 shutdown_connections 2024-09-16T04:14:24.226 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.209+0000 7f7cb1db8640 1 -- 172.21.15.42:0/713970769 wait complete. 2024-09-16T04:14:24.227 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.213+0000 7f7cb1db8640 10 monclient: build_initial_monmap 2024-09-16T04:14:24.227 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.213+0000 7f7cb1db8640 10 monclient: monmap: 2024-09-16T04:14:24.227 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:14:24.227 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:14:24.227 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:14:24.227 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:14:24.227 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:14:24.227 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:24.227 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:14:24.227 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:14:24.227 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:14:24.227 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:24.227 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.213+0000 7f7cb1db8640 1 librados: starting msgr at 2024-09-16T04:14:24.227 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.213+0000 7f7cb1db8640 1 librados: starting objecter 2024-09-16T04:14:24.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.213+0000 7f7cb1db8640 1 Processor -- start 2024-09-16T04:14:24.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.213+0000 7f7cb1db8640 1 -- start start 2024-09-16T04:14:24.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.213+0000 7f7cb1db8640 1 librados: setting wanted keys 2024-09-16T04:14:24.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.213+0000 7f7cb1db8640 1 librados: calling monclient init 2024-09-16T04:14:24.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.213+0000 7f7cb1db8640 10 monclient: init 2024-09-16T04:14:24.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.213+0000 7f7cb1db8640 10 monclient: _reopen_session rank -1 2024-09-16T04:14:24.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.213+0000 7f7cb1db8640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:14:24.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.213+0000 7f7cb1db8640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7cac0a2340 0x7f7cac1ceb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:24.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.213+0000 7f7cb1db8640 10 monclient(hunting): picked mon.noname-a con 0x7f7cac0a2340 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:14:24.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.213+0000 7f7cb1db8640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7cac0a2720 0x7f7cac06fb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:24.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.213+0000 7f7cb1db8640 10 monclient(hunting): picked mon.noname-c con 0x7f7cac0a2720 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:14:24.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.213+0000 7f7cb1db8640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7cac13a340 0x7f7cac135260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:24.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.213+0000 7f7cb1db8640 10 monclient(hunting): picked mon.noname-b con 0x7f7cac13a340 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:14:24.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.213+0000 7f7cb1db8640 10 monclient(hunting): start opening mon connection 2024-09-16T04:14:24.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.213+0000 7f7cb1db8640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f7cac13f820 con 0x7f7cac0a2340 2024-09-16T04:14:24.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.213+0000 7f7cb1db8640 10 monclient(hunting): start opening mon connection 2024-09-16T04:14:24.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.213+0000 7f7cb1db8640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f7cac13f6a0 con 0x7f7cac13a340 2024-09-16T04:14:24.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.213+0000 7f7cb1db8640 10 monclient(hunting): start opening mon connection 2024-09-16T04:14:24.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.213+0000 7f7cb1db8640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f7cac13f9a0 con 0x7f7cac0a2720 2024-09-16T04:14:24.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.213+0000 7f7cb1db8640 10 monclient(hunting): _renew_subs 2024-09-16T04:14:24.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.213+0000 7f7cb1db8640 10 monclient(hunting): authenticate will time out at 2911.185145s 2024-09-16T04:14:24.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.213+0000 7f7cab7fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7cac0a2340 0x7f7cac1ceb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:24.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.213+0000 7f7cab7fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7cac0a2340 0x7f7cac1ceb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:48548/0 (socket says 172.21.15.42:48548) 2024-09-16T04:14:24.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.213+0000 7f7cab7fe640 1 -- 172.21.15.42:0/1463004751 learned_addr learned my addr 172.21.15.42:0/1463004751 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:14:24.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.213+0000 7f7cab7fe640 10 monclient(hunting): get_auth_request con 0x7f7cac0a2340 auth_method 0 2024-09-16T04:14:24.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.213+0000 7f7cab7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:14:24.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.213+0000 7f7cab7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:24.229 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.213+0000 7f7cab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:24.230 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.213+0000 7f7caaffd640 1 --2- 172.21.15.42:0/1463004751 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7cac0a2720 0x7f7cac06fb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:24.230 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.213+0000 7f7caaffd640 10 monclient(hunting): get_auth_request con 0x7f7cac0a2720 auth_method 0 2024-09-16T04:14:24.230 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.213+0000 7f7caaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:14:24.230 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.213+0000 7f7caaffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:24.230 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.213+0000 7f7caaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:24.230 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.213+0000 7f7cab7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:14:24.230 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.213+0000 7f7cab7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:14:24.230 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.213+0000 7f7cab7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:14:24.230 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.213+0000 7f7caaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:14:24.230 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.213+0000 7f7caaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:14:24.230 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.213+0000 7f7caaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:14:24.230 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.213+0000 7f7cab7fe640 10 monclient(hunting): handle_auth_done global_id 1104216 payload 824 2024-09-16T04:14:24.230 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.213+0000 7f7cab7fe640 1 -- 172.21.15.42:0/1463004751 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7cac0a2720 msgr2=0x7f7cac06fb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:24.230 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.213+0000 7f7cab7fe640 1 --2- 172.21.15.42:0/1463004751 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7cac0a2720 0x7f7cac06fb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:24.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.213+0000 7f7cab7fe640 1 -- 172.21.15.42:0/1463004751 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7cac13a340 msgr2=0x7f7cac135260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:24.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.213+0000 7f7cabfff640 1 --2- 172.21.15.42:0/1463004751 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7cac13a340 0x7f7cac135260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:24.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.213+0000 7f7cab7fe640 1 --2- 172.21.15.42:0/1463004751 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7cac13a340 0x7f7cac135260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:24.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.213+0000 7f7cab7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:14:24.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.213+0000 7f7cab7fe640 1 monclient: found mon.noname-a 2024-09-16T04:14:24.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.213+0000 7f7cab7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:14:24.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.213+0000 7f7cab7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:14:24.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.213+0000 7f7cab7fe640 1 -- 172.21.15.42:0/1463004751 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7cac073a10 con 0x7f7cac0a2340 2024-09-16T04:14:24.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.213+0000 7f7cab7fe640 1 --2- 172.21.15.42:0/1463004751 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7cac0a2340 0x7f7cac1ceb40 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f7ca401cd90 tx=0x7f7ca4019650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:14:24.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.213+0000 7f7c93fff640 10 client.?.objecter ms_handle_connect 0x7f7cac0a2340 2024-09-16T04:14:24.231 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.213+0000 7f7c93fff640 10 client.?.objecter resend_mon_ops 2024-09-16T04:14:24.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.213+0000 7f7c93fff640 1 -- 172.21.15.42:0/1463004751 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ca40ac040 con 0x7f7cac0a2340 2024-09-16T04:14:24.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.213+0000 7f7c93fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:24.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.213+0000 7f7c93fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:14:24.232 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.213+0000 7f7c93fff640 10 monclient: dump: 2024-09-16T04:14:24.232 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:14:24.232 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:14:24.232 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:24.232 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:24.232 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:24.232 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:24.232 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:14:24.232 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:14:24.233 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:14:24.233 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:24.233 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.213+0000 7f7c93fff640 10 monclient: _finish_auth 0 2024-09-16T04:14:24.233 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.213+0000 7f7c93fff640 10 monclient: _check_auth_tickets 2024-09-16T04:14:24.233 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.213+0000 7f7c93fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:14:24.233 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.213+0000 7f7c93fff640 1 -- 172.21.15.42:0/1463004751 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ca40b1650 con 0x7f7cac0a2340 2024-09-16T04:14:24.233 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.213+0000 7f7c93fff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:14:24.233 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.213+0000 7f7c93fff640 1 -- 172.21.15.42:0/1463004751 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ca40b1a00 con 0x7f7cac0a2340 2024-09-16T04:14:24.233 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.213+0000 7f7c93fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:24.233 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.213+0000 7f7c93fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:14:24.233 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.213+0000 7f7c93fff640 10 monclient: dump: 2024-09-16T04:14:24.233 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:14:24.233 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:14:24.233 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:24.233 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:24.234 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:24.234 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:24.234 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:14:24.234 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:14:24.234 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:14:24.234 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:24.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.213+0000 7f7cb1db8640 5 monclient: authenticate success, global_id 1104216 2024-09-16T04:14:24.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.213+0000 7f7cb1db8640 10 monclient: _renew_subs 2024-09-16T04:14:24.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.213+0000 7f7cb1db8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:14:24.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.213+0000 7f7cb1db8640 1 -- 172.21.15.42:0/1463004751 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7cac121390 con 0x7f7cac0a2340 2024-09-16T04:14:24.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.213+0000 7f7ca9ffb640 4 set_mon_vals no callback set 2024-09-16T04:14:24.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.213+0000 7f7cb1db8640 10 client.1104216.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:14:24.234 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.213+0000 7f7cb1db8640 10 monclient: _renew_subs 2024-09-16T04:14:24.235 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.213+0000 7f7cb1db8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:14:24.235 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.213+0000 7f7cb1db8640 1 -- 172.21.15.42:0/1463004751 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7cac1cf9b0 con 0x7f7cac0a2340 2024-09-16T04:14:24.235 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.213+0000 7f7cb1db8640 1 librados: init done 2024-09-16T04:14:24.235 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.213+0000 7f7c93fff640 1 -- 172.21.15.42:0/1463004751 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f7ca40af070 con 0x7f7cac0a2340 2024-09-16T04:14:24.235 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.213+0000 7f7c93fff640 1 --2- 172.21.15.42:0/1463004751 >> v2:172.21.15.154:6812/254845342 conn(0x7f7c98037840 0x7f7c98039d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:24.235 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.213+0000 7f7cb1db8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:14:24.235 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.213+0000 7f7cb1db8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:14:24.235 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.213+0000 7f7cb1db8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:14:24.235 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.213+0000 7f7c93fff640 1 -- 172.21.15.42:0/1463004751 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(710..710 src has 681..710) ==== 4681+0+0 (secure 0 0 0) 0x7f7ca40e2c30 con 0x7f7cac0a2340 2024-09-16T04:14:24.235 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.213+0000 7f7c93fff640 10 client.1104216.objecter ms_dispatch 0x7f7cac000ba0 osd_map(710..710 src has 681..710) 2024-09-16T04:14:24.235 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.213+0000 7f7c93fff640 3 client.1104216.objecter handle_osd_map got epochs [710,710] > 0 2024-09-16T04:14:24.235 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.213+0000 7f7c93fff640 3 client.1104216.objecter handle_osd_map decoding full epoch 710 2024-09-16T04:14:24.235 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.213+0000 7f7cb1db8640 1 -- 172.21.15.42:0/1463004751 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7cac072180 con 0x7f7cac0a2340 2024-09-16T04:14:24.235 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.213+0000 7f7c93fff640 20 client.1104216.objecter dump_active .. 0 homeless 2024-09-16T04:14:24.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.213+0000 7f7caaffd640 1 --2- 172.21.15.42:0/1463004751 >> v2:172.21.15.154:6812/254845342 conn(0x7f7c98037840 0x7f7c98039d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:24.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.213+0000 7f7caaffd640 10 monclient: get_auth_request con 0x7f7c98037840 auth_method 0 2024-09-16T04:14:24.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.213+0000 7f7caaffd640 1 --2- 172.21.15.42:0/1463004751 >> v2:172.21.15.154:6812/254845342 conn(0x7f7c98037840 0x7f7c98039d00 secure :-1 s=READY pgs=4022 cs=0 l=1 rev1=1 crypto rx=0x7f7cac13c1b0 tx=0x7f7ca0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:14:24.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.213+0000 7f7c93fff640 10 client.1104216.objecter ms_handle_connect 0x7f7c98037840 2024-09-16T04:14:24.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.217+0000 7f7c93fff640 1 -- 172.21.15.42:0/1463004751 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f7ca40165c0 con 0x7f7cac0a2340 2024-09-16T04:14:24.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.217+0000 7f7c93fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:14:24.236 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.217+0000 7f7c93fff640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:14:24.386 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.381+0000 7f7cb1db8640 1 -- 172.21.15.42:0/1463004751 --> v2:172.21.15.154:6812/254845342 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) -- 0x7f7cac13c200 con 0x7f7c98037840 2024-09-16T04:14:24.387 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.381+0000 7f7c93fff640 1 -- 172.21.15.42:0/1463004751 <== mgr.4102 v2:172.21.15.154:6812/254845342 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f7ca000f820 con 0x7f7c98037840 2024-09-16T04:14:24.387 INFO:teuthology.orchestra.run.smithi042.stderr:instructed osd(s) 3 to deep-scrub 2024-09-16T04:14:24.389 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.381+0000 7f7cb1db8640 10 librados: watch_flush enter 2024-09-16T04:14:24.389 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.381+0000 7f7cb1db8640 10 librados: watch_flush exit 2024-09-16T04:14:24.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.381+0000 7f7cb1db8640 20 client.1104216.objecter shutdown clearing up homeless session... 2024-09-16T04:14:24.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.381+0000 7f7cb1db8640 10 client.1104216.objecter successfully canceled tick 2024-09-16T04:14:24.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.385+0000 7f7cb1db8640 1 -- 172.21.15.42:0/1463004751 >> v2:172.21.15.154:6812/254845342 conn(0x7f7c98037840 msgr2=0x7f7c98039d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:24.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.385+0000 7f7cb1db8640 1 --2- 172.21.15.42:0/1463004751 >> v2:172.21.15.154:6812/254845342 conn(0x7f7c98037840 0x7f7c98039d00 secure :-1 s=READY pgs=4022 cs=0 l=1 rev1=1 crypto rx=0x7f7cac13c1b0 tx=0x7f7ca0009040 comp rx=0 tx=0).stop 2024-09-16T04:14:24.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.385+0000 7f7cb1db8640 10 monclient: shutdown 2024-09-16T04:14:24.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.385+0000 7f7cb1db8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:14:24.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.385+0000 7f7cb1db8640 1 -- 172.21.15.42:0/1463004751 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7cac0a2340 msgr2=0x7f7cac1ceb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:24.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.385+0000 7f7cb1db8640 1 --2- 172.21.15.42:0/1463004751 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7cac0a2340 0x7f7cac1ceb40 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f7ca401cd90 tx=0x7f7ca4019650 comp rx=0 tx=0).stop 2024-09-16T04:14:24.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.385+0000 7f7cb1db8640 1 -- 172.21.15.42:0/1463004751 shutdown_connections 2024-09-16T04:14:24.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.385+0000 7f7cb1db8640 1 --2- 172.21.15.42:0/1463004751 >> v2:172.21.15.154:6812/254845342 conn(0x7f7c98037840 0x7f7c98039d00 unknown :-1 s=CLOSED pgs=4022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:24.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.385+0000 7f7cb1db8640 1 --2- 172.21.15.42:0/1463004751 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7cac13a340 0x7f7cac135260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:24.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.385+0000 7f7cb1db8640 1 --2- 172.21.15.42:0/1463004751 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7cac0a2720 0x7f7cac06fb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:24.390 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.385+0000 7f7cb1db8640 1 --2- 172.21.15.42:0/1463004751 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7cac0a2340 0x7f7cac1ceb40 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:24.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.385+0000 7f7cb1db8640 1 -- 172.21.15.42:0/1463004751 >> 172.21.15.42:0/1463004751 conn(0x7f7cac0a3430 msgr2=0x7f7cac0a56c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:14:24.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.385+0000 7f7cb1db8640 1 -- 172.21.15.42:0/1463004751 shutdown_connections 2024-09-16T04:14:24.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.385+0000 7f7cb1db8640 1 -- 172.21.15.42:0/1463004751 wait complete. 2024-09-16T04:14:24.391 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.385+0000 7f7cb1db8640 1 librados: shutdown 2024-09-16T04:14:24.406 INFO:tasks.ceph:Scrubbing osd.4 2024-09-16T04:14:24.406 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-09-16T04:14:24.530 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.525+0000 7fdaada18640 10 monclient: get_monmap_and_config 2024-09-16T04:14:24.530 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.525+0000 7fdaada18640 10 monclient: build_initial_monmap 2024-09-16T04:14:24.530 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.525+0000 7fdaada18640 10 monclient: monmap: 2024-09-16T04:14:24.530 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:14:24.530 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:14:24.530 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:14:24.528900+0000 2024-09-16T04:14:24.530 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:14:24.528900+0000 2024-09-16T04:14:24.530 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:14:24.531 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:24.531 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:14:24.531 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:14:24.531 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:14:24.531 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:14:24.531 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:14:24.531 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:24.533 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.525+0000 7fdaada18640 1 Processor -- start 2024-09-16T04:14:24.533 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.525+0000 7fdaada18640 1 -- start start 2024-09-16T04:14:24.533 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.525+0000 7fdaada18640 10 monclient: init 2024-09-16T04:14:24.533 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.525+0000 7fdaada18640 10 monclient: _reopen_session rank -1 2024-09-16T04:14:24.534 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.525+0000 7fdaada18640 10 monclient: _add_conns ranks=[3,1,4,0,2] 2024-09-16T04:14:24.534 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.525+0000 7fdaada18640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fdaa81374e0 0x7fdaa80a2560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:24.534 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.525+0000 7fdaada18640 10 monclient(hunting): picked mon.noname-a con 0x7fdaa81374e0 addr v2:172.21.15.42:3300/0 2024-09-16T04:14:24.534 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.525+0000 7fdaada18640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fdaa81378c0 addr v1:172.21.15.154:6789/0 2024-09-16T04:14:24.534 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.525+0000 7fdaada18640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7fdaa813c720 0x7fdaa813eb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:24.534 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.525+0000 7fdaada18640 10 monclient(hunting): picked mon.noname-c con 0x7fdaa813c720 addr v2:172.21.15.154:3300/0 2024-09-16T04:14:24.534 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.525+0000 7fdaada18640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdaa80a75e0 con 0x7fdaa81378c0 2024-09-16T04:14:24.534 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.525+0000 7fdaada18640 10 monclient(hunting): start opening mon connection 2024-09-16T04:14:24.534 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.525+0000 7fdaada18640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7fdaa80a1910 con 0x7fdaa81374e0 2024-09-16T04:14:24.534 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.525+0000 7fdaada18640 10 monclient(hunting): start opening mon connection 2024-09-16T04:14:24.534 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.525+0000 7fdaada18640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7fdaa80a1350 con 0x7fdaa813c720 2024-09-16T04:14:24.534 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.525+0000 7fdaada18640 10 monclient(hunting): _renew_subs 2024-09-16T04:14:24.534 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.525+0000 7fdaada18640 10 monclient(hunting): authenticate will time out at 2911.498737s 2024-09-16T04:14:24.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.525+0000 7fdaa6ffd640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fdaa81374e0 0x7fdaa80a2560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:24.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.525+0000 7fdaa6ffd640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7fdaa81374e0 0x7fdaa80a2560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:48552/0 (socket says 172.21.15.42:48552) 2024-09-16T04:14:24.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.525+0000 7fdaa6ffd640 1 -- 172.21.15.42:0/3925683331 learned_addr learned my addr 172.21.15.42:0/3925683331 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:14:24.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.525+0000 7fdaa6ffd640 10 monclient(hunting): get_auth_request con 0x7fdaa81374e0 auth_method 0 2024-09-16T04:14:24.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.525+0000 7fdaa6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:14:24.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.525+0000 7fdaa6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:24.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.525+0000 7fdaa6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:24.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.525+0000 7fdaa6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:14:24.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.525+0000 7fdaa6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:14:24.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.525+0000 7fdaa6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:14:24.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.525+0000 7fdaa77fe640 1 --2- 172.21.15.42:0/3925683331 >> v2:172.21.15.154:3300/0 conn(0x7fdaa813c720 0x7fdaa813eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:24.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.525+0000 7fdaa77fe640 10 monclient(hunting): get_auth_request con 0x7fdaa813c720 auth_method 0 2024-09-16T04:14:24.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.525+0000 7fdaa77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:14:24.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.525+0000 7fdaa77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:24.536 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.525+0000 7fdaa77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:24.536 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.525+0000 7fdaa6ffd640 10 monclient(hunting): handle_auth_done global_id 1104219 payload 274 2024-09-16T04:14:24.536 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.525+0000 7fdaa6ffd640 1 -- 172.21.15.42:0/3925683331 >> v2:172.21.15.154:3300/0 conn(0x7fdaa813c720 msgr2=0x7fdaa813eb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:14:24.536 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.525+0000 7fdaa6ffd640 1 --2- 172.21.15.42:0/3925683331 >> v2:172.21.15.154:3300/0 conn(0x7fdaa813c720 0x7fdaa813eb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:24.536 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.525+0000 7fdaa6ffd640 1 -- 172.21.15.42:0/3925683331 >> v1:172.21.15.154:6789/0 conn(0x7fdaa81378c0 legacy=0x7fdaa8137cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:14:24.536 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.525+0000 7fdaa6ffd640 10 monclient: _finish_hunting 0 2024-09-16T04:14:24.536 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.525+0000 7fdaa6ffd640 1 monclient: found mon.noname-a 2024-09-16T04:14:24.536 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.525+0000 7fdaa6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:14:24.536 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.525+0000 7fdaa6ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:14:24.536 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.525+0000 7fdaa6ffd640 1 -- 172.21.15.42:0/3925683331 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdaa80a1630 con 0x7fdaa81374e0 2024-09-16T04:14:24.536 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.525+0000 7fdaa77fe640 1 --2- 172.21.15.42:0/3925683331 >> v2:172.21.15.154:3300/0 conn(0x7fdaa813c720 0x7fdaa813eb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-16T04:14:24.542 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.529+0000 7fdaa6ffd640 1 --2- 172.21.15.42:0/3925683331 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdaa81374e0 0x7fdaa80a2560 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fdaa80a20a0 tx=0x7fda9801c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7abdfa1431e48d93 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:14:24.542 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.529+0000 7fdaa5ffb640 1 -- 172.21.15.42:0/3925683331 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaa80a5840 con 0x7fdaa81374e0 2024-09-16T04:14:24.542 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.529+0000 7fdaa5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:24.542 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.529+0000 7fdaa5ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:14:24.542 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.529+0000 7fdaa5ffb640 10 monclient: dump: 2024-09-16T04:14:24.543 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:14:24.543 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:14:24.543 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:24.543 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:24.543 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:24.543 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:24.543 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:14:24.543 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:14:24.543 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:14:24.543 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:24.543 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.529+0000 7fdaa5ffb640 10 monclient: _finish_auth 0 2024-09-16T04:14:24.543 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.529+0000 7fdaa5ffb640 10 monclient: _check_auth_tickets 2024-09-16T04:14:24.543 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.529+0000 7fdaa5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:14:24.544 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.529+0000 7fdaada18640 5 monclient: authenticate success, global_id 1104219 2024-09-16T04:14:24.544 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.529+0000 7fdaa5ffb640 1 -- 172.21.15.42:0/3925683331 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda980b0070 con 0x7fdaa81374e0 2024-09-16T04:14:24.544 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.529+0000 7fdaada18640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:14:24.544 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.529+0000 7fdaa5ffb640 10 monclient: handle_config config(0 keys) 2024-09-16T04:14:24.544 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.529+0000 7fdaada18640 10 monclient: get_monmap_and_config success 2024-09-16T04:14:24.544 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.529+0000 7fdaada18640 4 set_mon_vals no callback set 2024-09-16T04:14:24.544 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.529+0000 7fdaa5ffb640 1 -- 172.21.15.42:0/3925683331 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda980ab820 con 0x7fdaa81374e0 2024-09-16T04:14:24.544 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.529+0000 7fdaa5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:24.544 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.529+0000 7fdaa5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:14:24.544 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.529+0000 7fdaa5ffb640 10 monclient: dump: 2024-09-16T04:14:24.544 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:14:24.544 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:14:24.544 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:24.544 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:24.545 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:24.545 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:24.545 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:14:24.545 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:14:24.545 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:14:24.545 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:24.545 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.529+0000 7fdaada18640 10 monclient: shutdown 2024-09-16T04:14:24.545 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.529+0000 7fdaada18640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:14:24.545 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.529+0000 7fdaada18640 1 -- 172.21.15.42:0/3925683331 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdaa81374e0 msgr2=0x7fdaa80a2560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:24.545 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.529+0000 7fdaada18640 1 --2- 172.21.15.42:0/3925683331 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdaa81374e0 0x7fdaa80a2560 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fdaa80a20a0 tx=0x7fda9801c3e0 comp rx=0 tx=0).stop 2024-09-16T04:14:24.545 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.529+0000 7fdaada18640 1 -- 172.21.15.42:0/3925683331 shutdown_connections 2024-09-16T04:14:24.545 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.529+0000 7fdaada18640 1 --2- 172.21.15.42:0/3925683331 >> v2:172.21.15.154:3300/0 conn(0x7fdaa813c720 0x7fdaa813eb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:24.545 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.529+0000 7fdaada18640 1 --2- 172.21.15.42:0/3925683331 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdaa81374e0 0x7fdaa80a2560 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:24.545 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.529+0000 7fdaada18640 1 -- 172.21.15.42:0/3925683331 >> 172.21.15.42:0/3925683331 conn(0x7fdaa80aa320 msgr2=0x7fdaa80a6820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:14:24.545 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.529+0000 7fdaada18640 1 -- 172.21.15.42:0/3925683331 shutdown_connections 2024-09-16T04:14:24.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.529+0000 7fdaada18640 1 -- 172.21.15.42:0/3925683331 wait complete. 2024-09-16T04:14:24.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.529+0000 7fdaada18640 10 monclient: build_initial_monmap 2024-09-16T04:14:24.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.529+0000 7fdaada18640 10 monclient: monmap: 2024-09-16T04:14:24.546 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:14:24.546 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:14:24.546 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:14:24.546 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:14:24.546 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:14:24.546 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:24.546 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:14:24.546 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:14:24.546 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:14:24.546 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:24.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.529+0000 7fdaada18640 1 librados: starting msgr at 2024-09-16T04:14:24.546 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.529+0000 7fdaada18640 1 librados: starting objecter 2024-09-16T04:14:24.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.529+0000 7fdaada18640 1 Processor -- start 2024-09-16T04:14:24.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.529+0000 7fdaada18640 1 -- start start 2024-09-16T04:14:24.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.529+0000 7fdaada18640 1 librados: setting wanted keys 2024-09-16T04:14:24.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.529+0000 7fdaada18640 1 librados: calling monclient init 2024-09-16T04:14:24.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.529+0000 7fdaada18640 10 monclient: init 2024-09-16T04:14:24.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.533+0000 7fdaada18640 10 monclient: _reopen_session rank -1 2024-09-16T04:14:24.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.533+0000 7fdaada18640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:14:24.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.533+0000 7fdaada18640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fdaa81374e0 0x7fdaa81cf150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:24.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.533+0000 7fdaada18640 10 monclient(hunting): picked mon.noname-c con 0x7fdaa81374e0 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:14:24.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.533+0000 7fdaada18640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdaa81378c0 0x7fdaa81cf880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:24.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.533+0000 7fdaada18640 10 monclient(hunting): picked mon.noname-b con 0x7fdaa81378c0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:14:24.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.533+0000 7fdaada18640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdaa813c720 0x7fdaa809dad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:24.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.533+0000 7fdaada18640 10 monclient(hunting): picked mon.noname-a con 0x7fdaa813c720 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:14:24.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.533+0000 7fdaada18640 10 monclient(hunting): start opening mon connection 2024-09-16T04:14:24.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.533+0000 7fdaada18640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7fdaa8141c00 con 0x7fdaa813c720 2024-09-16T04:14:24.547 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.533+0000 7fdaada18640 10 monclient(hunting): start opening mon connection 2024-09-16T04:14:24.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.533+0000 7fdaada18640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7fdaa8141a80 con 0x7fdaa81378c0 2024-09-16T04:14:24.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.533+0000 7fdaada18640 10 monclient(hunting): start opening mon connection 2024-09-16T04:14:24.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.533+0000 7fdaada18640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7fdaa8141d80 con 0x7fdaa81374e0 2024-09-16T04:14:24.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.533+0000 7fdaada18640 10 monclient(hunting): _renew_subs 2024-09-16T04:14:24.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.533+0000 7fdaada18640 10 monclient(hunting): authenticate will time out at 2911.504385s 2024-09-16T04:14:24.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.533+0000 7fdaa6ffd640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fdaa81374e0 0x7fdaa81cf150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:24.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.533+0000 7fdaa77fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdaa813c720 0x7fdaa809dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:24.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.533+0000 7fdaa6ffd640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fdaa81374e0 0x7fdaa81cf150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:33218/0 (socket says 172.21.15.42:33218) 2024-09-16T04:14:24.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.533+0000 7fdaa77fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdaa813c720 0x7fdaa809dad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:48566/0 (socket says 172.21.15.42:48566) 2024-09-16T04:14:24.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.533+0000 7fdaa6ffd640 1 -- 172.21.15.42:0/1813051148 learned_addr learned my addr 172.21.15.42:0/1813051148 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:14:24.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.533+0000 7fdaa6ffd640 10 monclient(hunting): get_auth_request con 0x7fdaa81374e0 auth_method 0 2024-09-16T04:14:24.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.533+0000 7fdaa6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:14:24.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.533+0000 7fdaa6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:24.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.533+0000 7fdaa6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:24.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.533+0000 7fdaa77fe640 10 monclient(hunting): get_auth_request con 0x7fdaa813c720 auth_method 0 2024-09-16T04:14:24.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.533+0000 7fdaa77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:14:24.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.533+0000 7fdaa77fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:24.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.533+0000 7fdaa77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:24.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.533+0000 7fdaa6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:14:24.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.533+0000 7fdaa6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:14:24.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.533+0000 7fdaa6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:14:24.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.533+0000 7fdaa77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:14:24.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.533+0000 7fdaa77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:14:24.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.533+0000 7fdaa77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:14:24.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.533+0000 7fdaa67fc640 1 --2- 172.21.15.42:0/1813051148 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdaa81378c0 0x7fdaa81cf880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:24.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.533+0000 7fdaa6ffd640 10 monclient(hunting): handle_auth_done global_id 1094282 payload 824 2024-09-16T04:14:24.549 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.533+0000 7fdaa6ffd640 1 -- 172.21.15.42:0/1813051148 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdaa81378c0 msgr2=0x7fdaa81cf880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:24.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.533+0000 7fdaa6ffd640 1 --2- 172.21.15.42:0/1813051148 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdaa81378c0 0x7fdaa81cf880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:24.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.533+0000 7fdaa6ffd640 1 -- 172.21.15.42:0/1813051148 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdaa813c720 msgr2=0x7fdaa809dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:24.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.533+0000 7fdaa6ffd640 1 --2- 172.21.15.42:0/1813051148 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdaa813c720 0x7fdaa809dad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:24.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.533+0000 7fdaa6ffd640 10 monclient: _finish_hunting 0 2024-09-16T04:14:24.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.533+0000 7fdaa6ffd640 1 monclient: found mon.noname-c 2024-09-16T04:14:24.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.533+0000 7fdaa6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:14:24.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.533+0000 7fdaa6ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T04:14:24.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.533+0000 7fdaa6ffd640 1 -- 172.21.15.42:0/1813051148 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdaa8121120 con 0x7fdaa81374e0 2024-09-16T04:14:24.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.533+0000 7fdaa67fc640 10 monclient: get_auth_request con 0x7fdaa81378c0 auth_method 0 2024-09-16T04:14:24.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.533+0000 7fdaa77fe640 1 --2- 172.21.15.42:0/1813051148 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7fdaa813c720 0x7fdaa809dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:14:24.551 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.533+0000 7fdaa67fc640 1 --2- 172.21.15.42:0/1813051148 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7fdaa81378c0 0x7fdaa81cf880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T04:14:24.551 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.533+0000 7fdaa6ffd640 1 --2- 172.21.15.42:0/1813051148 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fdaa81374e0 0x7fdaa81cf150 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fda9801a010 tx=0x7fda9801ac20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:14:24.551 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.533+0000 7fda8ffff640 10 client.?.objecter ms_handle_connect 0x7fdaa81374e0 2024-09-16T04:14:24.551 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.533+0000 7fda8ffff640 10 client.?.objecter resend_mon_ops 2024-09-16T04:14:24.551 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.533+0000 7fda8ffff640 1 -- 172.21.15.42:0/1813051148 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda980be5f0 con 0x7fdaa81374e0 2024-09-16T04:14:24.551 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.533+0000 7fda8ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:24.551 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.533+0000 7fda8ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:14:24.551 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.533+0000 7fda8ffff640 10 monclient: dump: 2024-09-16T04:14:24.551 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:14:24.551 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:14:24.551 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:24.551 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:24.551 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:24.551 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:24.552 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:14:24.552 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:14:24.552 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:14:24.552 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:24.552 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.533+0000 7fda8ffff640 10 monclient: _finish_auth 0 2024-09-16T04:14:24.552 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.533+0000 7fda8ffff640 10 monclient: _check_auth_tickets 2024-09-16T04:14:24.552 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.533+0000 7fda8ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:14:24.552 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.533+0000 7fda8ffff640 1 -- 172.21.15.42:0/1813051148 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda980be790 con 0x7fdaa81374e0 2024-09-16T04:14:24.552 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.533+0000 7fdaada18640 5 monclient: authenticate success, global_id 1094282 2024-09-16T04:14:24.552 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.533+0000 7fdaada18640 10 monclient: _renew_subs 2024-09-16T04:14:24.552 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.533+0000 7fdaada18640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:14:24.552 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.533+0000 7fdaada18640 1 -- 172.21.15.42:0/1813051148 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdaa8120370 con 0x7fdaa81374e0 2024-09-16T04:14:24.552 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.533+0000 7fda8ffff640 10 monclient: handle_config config(0 keys) 2024-09-16T04:14:24.552 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.533+0000 7fda8ffff640 1 -- 172.21.15.42:0/1813051148 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda980beb60 con 0x7fdaa81374e0 2024-09-16T04:14:24.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.533+0000 7fda8ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:24.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.533+0000 7fda8ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:14:24.553 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.533+0000 7fda8ffff640 10 monclient: dump: 2024-09-16T04:14:24.553 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:14:24.553 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:14:24.553 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:24.553 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:24.553 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:24.553 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:24.553 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:14:24.554 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:14:24.554 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:14:24.554 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:24.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.533+0000 7fdaa4ff9640 4 set_mon_vals no callback set 2024-09-16T04:14:24.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.533+0000 7fdaada18640 10 client.1094282.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:14:24.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.533+0000 7fdaada18640 10 monclient: _renew_subs 2024-09-16T04:14:24.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.533+0000 7fdaada18640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:14:24.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.533+0000 7fdaada18640 1 -- 172.21.15.42:0/1813051148 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fdaa8120e40 con 0x7fdaa81374e0 2024-09-16T04:14:24.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.533+0000 7fdaada18640 1 librados: init done 2024-09-16T04:14:24.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.533+0000 7fdaada18640 10 client.1094282.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:14:24.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.533+0000 7fdaada18640 15 client.1094282.objecter _session_command_op_assign -1 1 2024-09-16T04:14:24.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.533+0000 7fdaada18640 20 client.1094282.objecter _get_session osd=-1 returning homeless 2024-09-16T04:14:24.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.533+0000 7fdaada18640 10 client.1094282.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:14:24.554 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.533+0000 7fdaada18640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:14:24.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.533+0000 7fdaada18640 1 -- 172.21.15.42:0/1813051148 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fdaa80a9720 con 0x7fdaa81374e0 2024-09-16T04:14:24.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.533+0000 7fda8ffff640 1 -- 172.21.15.42:0/1813051148 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7fda980aa070 con 0x7fdaa81374e0 2024-09-16T04:14:24.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.533+0000 7fda8ffff640 1 --2- 172.21.15.42:0/1813051148 >> v2:172.21.15.154:6812/254845342 conn(0x7fda94037810 0x7fda94039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:24.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.533+0000 7fda8ffff640 1 -- 172.21.15.42:0/1813051148 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(710..710 src has 681..710) ==== 4681+0+0 (secure 0 0 0) 0x7fda980e3480 con 0x7fdaa81374e0 2024-09-16T04:14:24.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.533+0000 7fda8ffff640 10 client.1094282.objecter ms_dispatch 0x7fdaa8000ba0 osd_map(710..710 src has 681..710) 2024-09-16T04:14:24.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.533+0000 7fda8ffff640 3 client.1094282.objecter handle_osd_map got epochs [710,710] > 0 2024-09-16T04:14:24.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.533+0000 7fda8ffff640 3 client.1094282.objecter handle_osd_map decoding full epoch 710 2024-09-16T04:14:24.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.533+0000 7fda8ffff640 10 client.1094282.objecter checking command 1 2024-09-16T04:14:24.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.533+0000 7fda8ffff640 1 --2- 172.21.15.42:0/1813051148 >> v2:172.21.15.154:6804/1038452759 conn(0x7fda9403ec70 0x7fda94041110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:24.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.533+0000 7fda8ffff640 20 client.1094282.objecter _get_session s=0x7fda9403e5c0 osd=4 3 2024-09-16T04:14:24.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.533+0000 7fda8ffff640 20 client.1094282.objecter put_session s=0x7fda9403e5c0 osd=4 3 2024-09-16T04:14:24.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.533+0000 7fda8ffff640 15 client.1094282.objecter _session_command_op_remove -1 1 2024-09-16T04:14:24.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.533+0000 7fda8ffff640 20 client.1094282.objecter _get_session s=0x7fda9403e5c0 osd=4 3 2024-09-16T04:14:24.555 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.533+0000 7fda8ffff640 20 client.1094282.objecter get_session s=0x7fda9403e5c0 osd=4 3 2024-09-16T04:14:24.556 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.533+0000 7fda8ffff640 15 client.1094282.objecter _session_command_op_assign 4 1 2024-09-16T04:14:24.556 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.533+0000 7fda8ffff640 20 client.1094282.objecter put_session s=0x7fda9403e5c0 osd=4 4 2024-09-16T04:14:24.556 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.533+0000 7fda8ffff640 10 client.1094282.objecter _send_command 1 2024-09-16T04:14:24.556 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.533+0000 7fda8ffff640 1 -- 172.21.15.42:0/1813051148 --> v2:172.21.15.154:6804/1038452759 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fda980b0e70 con 0x7fda9403ec70 2024-09-16T04:14:24.556 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.533+0000 7fda8ffff640 20 client.1094282.objecter dump_active .. 0 homeless 2024-09-16T04:14:24.556 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.533+0000 7fda8ffff640 1 -- 172.21.15.42:0/1813051148 <== mon.2 v2:172.21.15.42:3301/0 6 ==== mon_get_version_reply(handle=1 version=710) ==== 24+0+0 (secure 0 0 0) 0x7fda980ea050 con 0x7fdaa81374e0 2024-09-16T04:14:24.556 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.533+0000 7fda8ffff640 10 monclient: handle_get_version_reply finishing 1 version 710 2024-09-16T04:14:24.556 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.533+0000 7fdaa67fc640 1 --2- 172.21.15.42:0/1813051148 >> v2:172.21.15.154:6812/254845342 conn(0x7fda94037810 0x7fda94039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:24.556 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.533+0000 7fdaa67fc640 10 monclient: get_auth_request con 0x7fda94037810 auth_method 0 2024-09-16T04:14:24.556 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.533+0000 7fdaa77fe640 1 --2- 172.21.15.42:0/1813051148 >> v2:172.21.15.154:6804/1038452759 conn(0x7fda9403ec70 0x7fda94041110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:24.556 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.533+0000 7fdaa77fe640 10 monclient: get_auth_request con 0x7fda9403ec70 auth_method 0 2024-09-16T04:14:24.556 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.533+0000 7fdaa67fc640 1 --2- 172.21.15.42:0/1813051148 >> v2:172.21.15.154:6812/254845342 conn(0x7fda94037810 0x7fda94039cd0 secure :-1 s=READY pgs=4023 cs=0 l=1 rev1=1 crypto rx=0x7fdaa809f5a0 tx=0x7fda9c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:14:24.556 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.533+0000 7fda8ffff640 10 client.1094282.objecter ms_handle_connect 0x7fda94037810 2024-09-16T04:14:24.556 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.533+0000 7fda8ffff640 10 client.1094282.objecter ms_handle_connect 0x7fda9403ec70 2024-09-16T04:14:24.557 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.533+0000 7fdaa77fe640 1 --2- 172.21.15.42:0/1813051148 >> v2:172.21.15.154:6804/1038452759 conn(0x7fda9403ec70 0x7fda94041110 crc :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:14:24.557 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.537+0000 7fda8ffff640 1 -- 172.21.15.42:0/1813051148 <== osd.4 v2:172.21.15.154:6804/1038452759 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fda980b0e70 con 0x7fda9403ec70 2024-09-16T04:14:24.557 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.537+0000 7fda8ffff640 10 client.1094282.objecter ms_dispatch 0x7fdaa8000ba0 command_reply(tid 1: 0 ) 2024-09-16T04:14:24.557 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.537+0000 7fda8ffff640 10 client.1094282.objecter _finish_command 1 = system:0 2024-09-16T04:14:24.557 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.537+0000 7fda8ffff640 20 client.1094282.objecter put_session s=0x7fda9403e5c0 osd=4 4 2024-09-16T04:14:24.557 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.537+0000 7fda8ffff640 15 client.1094282.objecter _session_command_op_remove 4 1 2024-09-16T04:14:24.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.549+0000 7fdaada18640 10 client.1094282.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-09-16T04:14:24.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.549+0000 7fdaada18640 15 client.1094282.objecter _session_command_op_assign -1 2 2024-09-16T04:14:24.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.549+0000 7fdaada18640 20 client.1094282.objecter _get_session s=0x7fda9403e5c0 osd=4 3 2024-09-16T04:14:24.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.549+0000 7fdaada18640 20 client.1094282.objecter put_session s=0x7fda9403e5c0 osd=4 3 2024-09-16T04:14:24.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.549+0000 7fdaada18640 20 client.1094282.objecter _get_session s=0x7fda9403e5c0 osd=4 3 2024-09-16T04:14:24.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.549+0000 7fdaada18640 15 client.1094282.objecter _session_command_op_remove -1 2 2024-09-16T04:14:24.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.549+0000 7fdaada18640 20 client.1094282.objecter get_session s=0x7fda9403e5c0 osd=4 3 2024-09-16T04:14:24.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.549+0000 7fdaada18640 15 client.1094282.objecter _session_command_op_assign 4 2 2024-09-16T04:14:24.558 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.549+0000 7fdaada18640 20 client.1094282.objecter put_session s=0x7fda9403e5c0 osd=4 4 2024-09-16T04:14:24.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.549+0000 7fdaada18640 10 client.1094282.objecter _send_command 2 2024-09-16T04:14:24.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.549+0000 7fdaada18640 1 -- 172.21.15.42:0/1813051148 --> v2:172.21.15.154:6804/1038452759 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fdaa80a2540 con 0x7fda9403ec70 2024-09-16T04:14:24.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.549+0000 7fda8ffff640 1 -- 172.21.15.42:0/1813051148 <== osd.4 v2:172.21.15.154:6804/1038452759 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fdaa80a2540 con 0x7fda9403ec70 2024-09-16T04:14:24.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.549+0000 7fda8ffff640 10 client.1094282.objecter ms_dispatch 0x7fdaa8000ba0 command_reply(tid 2: 0 ) 2024-09-16T04:14:24.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.549+0000 7fda8ffff640 10 client.1094282.objecter _finish_command 2 = system:0 2024-09-16T04:14:24.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.549+0000 7fda8ffff640 20 client.1094282.objecter put_session s=0x7fda9403e5c0 osd=4 4 2024-09-16T04:14:24.559 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.549+0000 7fda8ffff640 15 client.1094282.objecter _session_command_op_remove 4 2 2024-09-16T04:14:24.559 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T04:14:24.559 INFO:teuthology.orchestra.run.smithi042.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T04:14:24.559 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T04:14:24.560 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.553+0000 7fdaada18640 10 librados: watch_flush enter 2024-09-16T04:14:24.560 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.553+0000 7fdaada18640 10 librados: watch_flush exit 2024-09-16T04:14:24.560 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.553+0000 7fdaada18640 10 client.1094282.objecter close_session for osd.4 2024-09-16T04:14:24.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.553+0000 7fdaada18640 1 -- 172.21.15.42:0/1813051148 >> v2:172.21.15.154:6804/1038452759 conn(0x7fda9403ec70 msgr2=0x7fda94041110 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:24.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.553+0000 7fdaada18640 1 --2- 172.21.15.42:0/1813051148 >> v2:172.21.15.154:6804/1038452759 conn(0x7fda9403ec70 0x7fda94041110 crc :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:24.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.553+0000 7fdaada18640 20 client.1094282.objecter put_session s=0x7fda9403e5c0 osd=4 1 2024-09-16T04:14:24.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.553+0000 7fdaada18640 20 client.1094282.objecter shutdown clearing up homeless session... 2024-09-16T04:14:24.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.553+0000 7fdaada18640 10 client.1094282.objecter successfully canceled tick 2024-09-16T04:14:24.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.553+0000 7fdaada18640 1 -- 172.21.15.42:0/1813051148 >> v2:172.21.15.154:6812/254845342 conn(0x7fda94037810 msgr2=0x7fda94039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:24.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.553+0000 7fdaada18640 1 --2- 172.21.15.42:0/1813051148 >> v2:172.21.15.154:6812/254845342 conn(0x7fda94037810 0x7fda94039cd0 secure :-1 s=READY pgs=4023 cs=0 l=1 rev1=1 crypto rx=0x7fdaa809f5a0 tx=0x7fda9c009040 comp rx=0 tx=0).stop 2024-09-16T04:14:24.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.553+0000 7fdaada18640 10 monclient: shutdown 2024-09-16T04:14:24.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.553+0000 7fdaada18640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:14:24.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.553+0000 7fdaada18640 1 -- 172.21.15.42:0/1813051148 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fdaa81374e0 msgr2=0x7fdaa81cf150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:24.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.553+0000 7fdaada18640 1 --2- 172.21.15.42:0/1813051148 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7fdaa81374e0 0x7fdaa81cf150 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fda9801a010 tx=0x7fda9801ac20 comp rx=0 tx=0).stop 2024-09-16T04:14:24.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.553+0000 7fdaa77fe640 1 -- 172.21.15.42:0/1813051148 reap_dead start 2024-09-16T04:14:24.561 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.553+0000 7fdaada18640 1 -- 172.21.15.42:0/1813051148 shutdown_connections 2024-09-16T04:14:24.562 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.553+0000 7fdaada18640 1 -- 172.21.15.42:0/1813051148 >> 172.21.15.42:0/1813051148 conn(0x7fdaa80aa320 msgr2=0x7fdaa80a2ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:14:24.562 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.553+0000 7fdaada18640 1 -- 172.21.15.42:0/1813051148 shutdown_connections 2024-09-16T04:14:24.562 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.553+0000 7fdaada18640 1 -- 172.21.15.42:0/1813051148 wait complete. 2024-09-16T04:14:24.562 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.553+0000 7fdaada18640 1 librados: shutdown 2024-09-16T04:14:24.573 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-09-16T04:14:24.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.677+0000 7f6e8244d640 10 monclient: get_monmap_and_config 2024-09-16T04:14:24.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.677+0000 7f6e8244d640 10 monclient: build_initial_monmap 2024-09-16T04:14:24.682 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.677+0000 7f6e8244d640 10 monclient: monmap: 2024-09-16T04:14:24.682 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:14:24.682 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:14:24.682 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:14:24.680955+0000 2024-09-16T04:14:24.682 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:14:24.680955+0000 2024-09-16T04:14:24.682 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:14:24.682 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:24.682 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:14:24.683 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:14:24.683 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:14:24.683 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:14:24.683 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:14:24.683 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:24.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.677+0000 7f6e8244d640 1 Processor -- start 2024-09-16T04:14:24.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.677+0000 7f6e8244d640 1 -- start start 2024-09-16T04:14:24.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.677+0000 7f6e8244d640 10 monclient: init 2024-09-16T04:14:24.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.677+0000 7f6e8244d640 10 monclient: _reopen_session rank -1 2024-09-16T04:14:24.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.677+0000 7f6e8244d640 10 monclient: _add_conns ranks=[1,4,3,2,0] 2024-09-16T04:14:24.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.677+0000 7f6e8244d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6e7c09fe60 addr v1:172.21.15.154:6789/0 2024-09-16T04:14:24.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.677+0000 7f6e8244d640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f6e7c0a1fb0 0x7f6e7c13c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:24.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.677+0000 7f6e8244d640 10 monclient(hunting): picked mon.noname-c con 0x7f6e7c0a1fb0 addr v2:172.21.15.154:3300/0 2024-09-16T04:14:24.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.677+0000 7f6e8244d640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f6e7c0a0950 0x7f6e7c13eac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:24.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.677+0000 7f6e8244d640 10 monclient(hunting): picked mon.noname-a con 0x7f6e7c0a0950 addr v2:172.21.15.42:3300/0 2024-09-16T04:14:24.685 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.677+0000 7f6e8244d640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6e7c06caa0 con 0x7f6e7c09fe60 2024-09-16T04:14:24.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.677+0000 7f6e8244d640 10 monclient(hunting): start opening mon connection 2024-09-16T04:14:24.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.677+0000 7f6e8244d640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f6e7c06c7f0 con 0x7f6e7c0a0950 2024-09-16T04:14:24.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.677+0000 7f6e8244d640 10 monclient(hunting): start opening mon connection 2024-09-16T04:14:24.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.677+0000 7f6e8244d640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f6e7c0a1d80 con 0x7f6e7c0a1fb0 2024-09-16T04:14:24.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.677+0000 7f6e8244d640 10 monclient(hunting): _renew_subs 2024-09-16T04:14:24.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.677+0000 7f6e8244d640 10 monclient(hunting): authenticate will time out at 2911.650352s 2024-09-16T04:14:24.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.677+0000 7f6e809ea640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f6e7c0a0950 0x7f6e7c13eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:24.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.677+0000 7f6e809ea640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f6e7c0a0950 0x7f6e7c13eac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:48572/0 (socket says 172.21.15.42:48572) 2024-09-16T04:14:24.686 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.677+0000 7f6e809ea640 1 -- 172.21.15.42:0/3335186703 learned_addr learned my addr 172.21.15.42:0/3335186703 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:14:24.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.677+0000 7f6e809ea640 10 monclient(hunting): get_auth_request con 0x7f6e7c0a0950 auth_method 0 2024-09-16T04:14:24.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.677+0000 7f6e809ea640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:14:24.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.677+0000 7f6e809ea640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:24.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.677+0000 7f6e809ea640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:24.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.677+0000 7f6e809ea640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:14:24.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.677+0000 7f6e809ea640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:14:24.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.677+0000 7f6e809ea640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:14:24.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.677+0000 7f6e7b7fe640 1 --2- 172.21.15.42:0/3335186703 >> v2:172.21.15.154:3300/0 conn(0x7f6e7c0a1fb0 0x7f6e7c13c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:24.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.677+0000 7f6e7b7fe640 10 monclient(hunting): get_auth_request con 0x7f6e7c0a1fb0 auth_method 0 2024-09-16T04:14:24.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.677+0000 7f6e7b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:14:24.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.677+0000 7f6e7b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:24.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.677+0000 7f6e7b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:24.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.677+0000 7f6e809ea640 10 monclient(hunting): handle_auth_done global_id 1104225 payload 274 2024-09-16T04:14:24.687 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.677+0000 7f6e809ea640 1 -- 172.21.15.42:0/3335186703 >> v2:172.21.15.154:3300/0 conn(0x7f6e7c0a1fb0 msgr2=0x7f6e7c13c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:14:24.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.677+0000 7f6e809ea640 1 --2- 172.21.15.42:0/3335186703 >> v2:172.21.15.154:3300/0 conn(0x7f6e7c0a1fb0 0x7f6e7c13c570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:24.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.677+0000 7f6e809ea640 1 -- 172.21.15.42:0/3335186703 >> v1:172.21.15.154:6789/0 conn(0x7f6e7c09fe60 legacy=0x7f6e7c0a0240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:14:24.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.677+0000 7f6e809ea640 10 monclient: _finish_hunting 0 2024-09-16T04:14:24.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.677+0000 7f6e809ea640 1 monclient: found mon.noname-a 2024-09-16T04:14:24.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.677+0000 7f6e809ea640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:14:24.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.677+0000 7f6e809ea640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:14:24.688 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.677+0000 7f6e809ea640 1 -- 172.21.15.42:0/3335186703 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e7c0a2790 con 0x7f6e7c0a0950 2024-09-16T04:14:24.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.681+0000 7f6e809ea640 1 --2- 172.21.15.42:0/3335186703 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6e7c0a0950 0x7f6e7c13eac0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f6e7c0a2f30 tx=0x7f6e7401c380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc5e8d32f5994bdd server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:14:24.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.681+0000 7f6e7affd640 1 -- 172.21.15.42:0/3335186703 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e740b0070 con 0x7f6e7c0a0950 2024-09-16T04:14:24.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.681+0000 7f6e7affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:24.693 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.681+0000 7f6e7affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:14:24.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.681+0000 7f6e7affd640 10 monclient: dump: 2024-09-16T04:14:24.694 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:14:24.694 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:14:24.694 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:24.694 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:24.694 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:24.694 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:24.694 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:14:24.694 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:14:24.694 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:14:24.694 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:24.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.681+0000 7f6e7affd640 10 monclient: _finish_auth 0 2024-09-16T04:14:24.694 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.681+0000 7f6e7affd640 10 monclient: _check_auth_tickets 2024-09-16T04:14:24.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.681+0000 7f6e7affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:14:24.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.681+0000 7f6e7affd640 1 -- 172.21.15.42:0/3335186703 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6e740ab600 con 0x7f6e7c0a0950 2024-09-16T04:14:24.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.681+0000 7f6e8244d640 5 monclient: authenticate success, global_id 1104225 2024-09-16T04:14:24.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.681+0000 7f6e8244d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:14:24.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.681+0000 7f6e7affd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:14:24.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.681+0000 7f6e8244d640 10 monclient: get_monmap_and_config success 2024-09-16T04:14:24.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.681+0000 7f6e7affd640 1 -- 172.21.15.42:0/3335186703 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e740ab9d0 con 0x7f6e7c0a0950 2024-09-16T04:14:24.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.681+0000 7f6e7affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:24.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.681+0000 7f6e8244d640 4 set_mon_vals no callback set 2024-09-16T04:14:24.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.681+0000 7f6e7affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:14:24.695 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.681+0000 7f6e7affd640 10 monclient: dump: 2024-09-16T04:14:24.695 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:14:24.695 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:14:24.696 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:24.696 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:24.696 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:24.696 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:24.696 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:14:24.696 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:14:24.696 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:14:24.696 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:24.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.681+0000 7f6e8244d640 10 monclient: shutdown 2024-09-16T04:14:24.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.681+0000 7f6e8244d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:14:24.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.681+0000 7f6e8244d640 1 -- 172.21.15.42:0/3335186703 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6e7c0a0950 msgr2=0x7f6e7c13eac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:24.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.681+0000 7f6e8244d640 1 --2- 172.21.15.42:0/3335186703 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6e7c0a0950 0x7f6e7c13eac0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f6e7c0a2f30 tx=0x7f6e7401c380 comp rx=0 tx=0).stop 2024-09-16T04:14:24.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.681+0000 7f6e8244d640 1 -- 172.21.15.42:0/3335186703 shutdown_connections 2024-09-16T04:14:24.696 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.681+0000 7f6e8244d640 1 --2- 172.21.15.42:0/3335186703 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6e7c0a0950 0x7f6e7c13eac0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:24.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.681+0000 7f6e8244d640 1 --2- 172.21.15.42:0/3335186703 >> v2:172.21.15.154:3300/0 conn(0x7f6e7c0a1fb0 0x7f6e7c13c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:24.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.681+0000 7f6e8244d640 1 -- 172.21.15.42:0/3335186703 >> 172.21.15.42:0/3335186703 conn(0x7f6e7c075ae0 msgr2=0x7f6e7c075ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:14:24.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.681+0000 7f6e8244d640 1 -- 172.21.15.42:0/3335186703 shutdown_connections 2024-09-16T04:14:24.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.681+0000 7f6e8244d640 1 -- 172.21.15.42:0/3335186703 wait complete. 2024-09-16T04:14:24.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.681+0000 7f6e8244d640 10 monclient: build_initial_monmap 2024-09-16T04:14:24.697 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.681+0000 7f6e8244d640 10 monclient: monmap: 2024-09-16T04:14:24.697 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:14:24.697 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:14:24.697 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:14:24.697 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:14:24.697 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:14:24.697 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:24.697 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:14:24.697 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:14:24.698 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:14:24.698 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:24.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.681+0000 7f6e8244d640 1 librados: starting msgr at 2024-09-16T04:14:24.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.681+0000 7f6e8244d640 1 librados: starting objecter 2024-09-16T04:14:24.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.681+0000 7f6e8244d640 1 Processor -- start 2024-09-16T04:14:24.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.681+0000 7f6e8244d640 1 -- start start 2024-09-16T04:14:24.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.681+0000 7f6e8244d640 1 librados: setting wanted keys 2024-09-16T04:14:24.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.681+0000 7f6e8244d640 1 librados: calling monclient init 2024-09-16T04:14:24.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.681+0000 7f6e8244d640 10 monclient: init 2024-09-16T04:14:24.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.685+0000 7f6e8244d640 10 monclient: _reopen_session rank -1 2024-09-16T04:14:24.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.685+0000 7f6e8244d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-16T04:14:24.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.685+0000 7f6e8244d640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f6e7c09fe60 0x7f6e7c1cf080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:24.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.685+0000 7f6e8244d640 10 monclient(hunting): picked mon.noname-c con 0x7f6e7c09fe60 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:14:24.698 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.685+0000 7f6e8244d640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6e7c0a0950 0x7f6e7c134480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:24.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.685+0000 7f6e8244d640 10 monclient(hunting): picked mon.noname-a con 0x7f6e7c0a0950 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:14:24.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.685+0000 7f6e8244d640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6e7c0a1fb0 0x7f6e7c13c660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:24.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.685+0000 7f6e8244d640 10 monclient(hunting): picked mon.noname-b con 0x7f6e7c0a1fb0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:14:24.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.685+0000 7f6e8244d640 10 monclient(hunting): start opening mon connection 2024-09-16T04:14:24.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.685+0000 7f6e8244d640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f6e7c141a30 con 0x7f6e7c0a0950 2024-09-16T04:14:24.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.685+0000 7f6e8244d640 10 monclient(hunting): start opening mon connection 2024-09-16T04:14:24.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.685+0000 7f6e8244d640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f6e7c06eb00 con 0x7f6e7c0a1fb0 2024-09-16T04:14:24.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.685+0000 7f6e8244d640 10 monclient(hunting): start opening mon connection 2024-09-16T04:14:24.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.685+0000 7f6e8244d640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f6e7c141bb0 con 0x7f6e7c09fe60 2024-09-16T04:14:24.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.685+0000 7f6e8244d640 10 monclient(hunting): _renew_subs 2024-09-16T04:14:24.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.685+0000 7f6e8244d640 10 monclient(hunting): authenticate will time out at 2911.656181s 2024-09-16T04:14:24.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.685+0000 7f6e7bfff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f6e7c09fe60 0x7f6e7c1cf080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:24.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.685+0000 7f6e7b7fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6e7c0a0950 0x7f6e7c134480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:24.699 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.685+0000 7f6e7bfff640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f6e7c09fe60 0x7f6e7c1cf080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:33230/0 (socket says 172.21.15.42:33230) 2024-09-16T04:14:24.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.685+0000 7f6e7b7fe640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6e7c0a0950 0x7f6e7c134480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:48580/0 (socket says 172.21.15.42:48580) 2024-09-16T04:14:24.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.685+0000 7f6e7bfff640 1 -- 172.21.15.42:0/4223726169 learned_addr learned my addr 172.21.15.42:0/4223726169 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:14:24.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.685+0000 7f6e7bfff640 10 monclient(hunting): get_auth_request con 0x7f6e7c09fe60 auth_method 0 2024-09-16T04:14:24.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.685+0000 7f6e7bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:14:24.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.685+0000 7f6e7bfff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:24.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.685+0000 7f6e7bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:24.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.685+0000 7f6e7b7fe640 10 monclient(hunting): get_auth_request con 0x7f6e7c0a0950 auth_method 0 2024-09-16T04:14:24.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.685+0000 7f6e7b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:14:24.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.685+0000 7f6e7b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:24.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.685+0000 7f6e7b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:24.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.685+0000 7f6e7bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:14:24.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.685+0000 7f6e7bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:14:24.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.685+0000 7f6e7bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:14:24.700 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.685+0000 7f6e7b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:14:24.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.685+0000 7f6e7b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:14:24.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.685+0000 7f6e7b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:14:24.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.685+0000 7f6e809ea640 1 --2- 172.21.15.42:0/4223726169 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6e7c0a1fb0 0x7f6e7c13c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:24.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.685+0000 7f6e809ea640 10 monclient(hunting): get_auth_request con 0x7f6e7c0a1fb0 auth_method 0 2024-09-16T04:14:24.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.685+0000 7f6e809ea640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:14:24.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.685+0000 7f6e809ea640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:24.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.685+0000 7f6e809ea640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:24.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.685+0000 7f6e7b7fe640 10 monclient(hunting): handle_auth_done global_id 1104228 payload 824 2024-09-16T04:14:24.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.685+0000 7f6e7b7fe640 1 -- 172.21.15.42:0/4223726169 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f6e7c09fe60 msgr2=0x7f6e7c1cf080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:24.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.685+0000 7f6e7b7fe640 1 --2- 172.21.15.42:0/4223726169 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f6e7c09fe60 0x7f6e7c1cf080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:24.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.685+0000 7f6e7b7fe640 1 -- 172.21.15.42:0/4223726169 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6e7c0a1fb0 msgr2=0x7f6e7c13c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:24.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.685+0000 7f6e7b7fe640 1 --2- 172.21.15.42:0/4223726169 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6e7c0a1fb0 0x7f6e7c13c660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:24.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.685+0000 7f6e7b7fe640 10 monclient: _finish_hunting 0 2024-09-16T04:14:24.701 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.685+0000 7f6e7b7fe640 1 monclient: found mon.noname-a 2024-09-16T04:14:24.702 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.685+0000 7f6e7b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:14:24.702 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.685+0000 7f6e7b7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:14:24.702 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.685+0000 7f6e7b7fe640 1 -- 172.21.15.42:0/4223726169 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e7c06f190 con 0x7f6e7c0a0950 2024-09-16T04:14:24.702 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.685+0000 7f6e7bfff640 1 --2- 172.21.15.42:0/4223726169 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f6e7c09fe60 0x7f6e7c1cf080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:14:24.702 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.685+0000 7f6e7b7fe640 1 --2- 172.21.15.42:0/4223726169 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6e7c0a0950 0x7f6e7c134480 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f6e6400d5f0 tx=0x7f6e6400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:14:24.702 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.685+0000 7f6e78ff9640 10 client.?.objecter ms_handle_connect 0x7f6e7c0a0950 2024-09-16T04:14:24.702 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.685+0000 7f6e78ff9640 10 client.?.objecter resend_mon_ops 2024-09-16T04:14:24.702 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.685+0000 7f6e78ff9640 1 -- 172.21.15.42:0/4223726169 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e640125f0 con 0x7f6e7c0a0950 2024-09-16T04:14:24.702 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.685+0000 7f6e78ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:24.702 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.685+0000 7f6e78ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:14:24.702 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.685+0000 7f6e78ff9640 10 monclient: dump: 2024-09-16T04:14:24.702 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:14:24.702 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:14:24.702 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:24.703 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:24.703 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:24.703 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:24.703 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:14:24.703 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:14:24.703 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:14:24.703 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:24.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.685+0000 7f6e78ff9640 10 monclient: _finish_auth 0 2024-09-16T04:14:24.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.685+0000 7f6e78ff9640 10 monclient: _check_auth_tickets 2024-09-16T04:14:24.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.685+0000 7f6e78ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:14:24.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.685+0000 7f6e78ff9640 1 -- 172.21.15.42:0/4223726169 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6e64019070 con 0x7f6e7c0a0950 2024-09-16T04:14:24.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.685+0000 7f6e8244d640 5 monclient: authenticate success, global_id 1104228 2024-09-16T04:14:24.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.685+0000 7f6e8244d640 10 monclient: _renew_subs 2024-09-16T04:14:24.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.685+0000 7f6e8244d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:14:24.703 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.685+0000 7f6e8244d640 1 -- 172.21.15.42:0/4223726169 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e7c13eb80 con 0x7f6e7c0a0950 2024-09-16T04:14:24.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.685+0000 7f6e78ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T04:14:24.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.685+0000 7f6e78ff9640 1 -- 172.21.15.42:0/4223726169 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e640129b0 con 0x7f6e7c0a0950 2024-09-16T04:14:24.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.685+0000 7f6e78ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:24.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.685+0000 7f6e78ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:14:24.704 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.685+0000 7f6e78ff9640 10 monclient: dump: 2024-09-16T04:14:24.704 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:14:24.704 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:14:24.704 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:24.704 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:24.704 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:24.704 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:24.704 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:14:24.704 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:14:24.704 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:14:24.704 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:24.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.685+0000 7f6e7a7fc640 4 set_mon_vals no callback set 2024-09-16T04:14:24.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.685+0000 7f6e8244d640 10 client.1104228.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:14:24.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.685+0000 7f6e8244d640 10 monclient: _renew_subs 2024-09-16T04:14:24.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.685+0000 7f6e8244d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:14:24.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.685+0000 7f6e8244d640 1 -- 172.21.15.42:0/4223726169 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e7c11eb30 con 0x7f6e7c0a0950 2024-09-16T04:14:24.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.685+0000 7f6e8244d640 1 librados: init done 2024-09-16T04:14:24.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.685+0000 7f6e78ff9640 1 -- 172.21.15.42:0/4223726169 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f6e64012b50 con 0x7f6e7c0a0950 2024-09-16T04:14:24.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.685+0000 7f6e8244d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:14:24.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.685+0000 7f6e8244d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:14:24.705 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.685+0000 7f6e8244d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:14:24.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.685+0000 7f6e8244d640 1 -- 172.21.15.42:0/4223726169 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6e7c0a0240 con 0x7f6e7c0a0950 2024-09-16T04:14:24.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.685+0000 7f6e78ff9640 1 --2- 172.21.15.42:0/4223726169 >> v2:172.21.15.154:6812/254845342 conn(0x7f6e5c037840 0x7f6e5c039d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:24.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.685+0000 7f6e78ff9640 1 -- 172.21.15.42:0/4223726169 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(710..710 src has 681..710) ==== 4681+0+0 (secure 0 0 0) 0x7f6e64048e70 con 0x7f6e7c0a0950 2024-09-16T04:14:24.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.685+0000 7f6e78ff9640 10 client.1104228.objecter ms_dispatch 0x7f6e7c000ba0 osd_map(710..710 src has 681..710) 2024-09-16T04:14:24.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.685+0000 7f6e78ff9640 3 client.1104228.objecter handle_osd_map got epochs [710,710] > 0 2024-09-16T04:14:24.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.685+0000 7f6e78ff9640 3 client.1104228.objecter handle_osd_map decoding full epoch 710 2024-09-16T04:14:24.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.685+0000 7f6e78ff9640 20 client.1104228.objecter dump_active .. 0 homeless 2024-09-16T04:14:24.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.685+0000 7f6e7bfff640 1 --2- 172.21.15.42:0/4223726169 >> v2:172.21.15.154:6812/254845342 conn(0x7f6e5c037840 0x7f6e5c039d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:24.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.685+0000 7f6e7bfff640 10 monclient: get_auth_request con 0x7f6e5c037840 auth_method 0 2024-09-16T04:14:24.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.685+0000 7f6e78ff9640 10 client.1104228.objecter ms_handle_connect 0x7f6e5c037840 2024-09-16T04:14:24.706 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.685+0000 7f6e7bfff640 1 --2- 172.21.15.42:0/4223726169 >> v2:172.21.15.154:6812/254845342 conn(0x7f6e5c037840 0x7f6e5c039d00 secure :-1 s=READY pgs=4024 cs=0 l=1 rev1=1 crypto rx=0x7f6e6c00f3c0 tx=0x7f6e6c003e70 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:14:24.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.689+0000 7f6e78ff9640 1 -- 172.21.15.42:0/4223726169 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f6e6400bdf0 con 0x7f6e7c0a0950 2024-09-16T04:14:24.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.689+0000 7f6e78ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:14:24.707 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.689+0000 7f6e78ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:14:24.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.845+0000 7f6e8244d640 1 -- 172.21.15.42:0/4223726169 --> v2:172.21.15.154:6812/254845342 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) -- 0x7f6e7c06b3e0 con 0x7f6e5c037840 2024-09-16T04:14:24.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.845+0000 7f6e78ff9640 1 -- 172.21.15.42:0/4223726169 <== mgr.4102 v2:172.21.15.154:6812/254845342 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f6e7c06b3e0 con 0x7f6e5c037840 2024-09-16T04:14:24.853 INFO:teuthology.orchestra.run.smithi042.stderr:instructed osd(s) 4 to deep-scrub 2024-09-16T04:14:24.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.849+0000 7f6e8244d640 10 librados: watch_flush enter 2024-09-16T04:14:24.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.849+0000 7f6e8244d640 10 librados: watch_flush exit 2024-09-16T04:14:24.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.849+0000 7f6e8244d640 20 client.1104228.objecter shutdown clearing up homeless session... 2024-09-16T04:14:24.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.849+0000 7f6e8244d640 10 client.1104228.objecter successfully canceled tick 2024-09-16T04:14:24.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.849+0000 7f6e8244d640 1 -- 172.21.15.42:0/4223726169 >> v2:172.21.15.154:6812/254845342 conn(0x7f6e5c037840 msgr2=0x7f6e5c039d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:24.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.849+0000 7f6e8244d640 1 --2- 172.21.15.42:0/4223726169 >> v2:172.21.15.154:6812/254845342 conn(0x7f6e5c037840 0x7f6e5c039d00 secure :-1 s=READY pgs=4024 cs=0 l=1 rev1=1 crypto rx=0x7f6e6c00f3c0 tx=0x7f6e6c003e70 comp rx=0 tx=0).stop 2024-09-16T04:14:24.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.849+0000 7f6e8244d640 10 monclient: shutdown 2024-09-16T04:14:24.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.849+0000 7f6e8244d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:14:24.857 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.849+0000 7f6e8244d640 1 -- 172.21.15.42:0/4223726169 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6e7c0a0950 msgr2=0x7f6e7c134480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:24.857 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.849+0000 7f6e8244d640 1 --2- 172.21.15.42:0/4223726169 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6e7c0a0950 0x7f6e7c134480 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f6e6400d5f0 tx=0x7f6e6400dab0 comp rx=0 tx=0).stop 2024-09-16T04:14:24.857 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.849+0000 7f6e8244d640 1 -- 172.21.15.42:0/4223726169 shutdown_connections 2024-09-16T04:14:24.857 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.849+0000 7f6e8244d640 1 --2- 172.21.15.42:0/4223726169 >> v2:172.21.15.154:6812/254845342 conn(0x7f6e5c037840 0x7f6e5c039d00 unknown :-1 s=CLOSED pgs=4024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:24.857 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.849+0000 7f6e8244d640 1 --2- 172.21.15.42:0/4223726169 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f6e7c0a1fb0 0x7f6e7c13c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:24.857 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.849+0000 7f6e8244d640 1 --2- 172.21.15.42:0/4223726169 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f6e7c0a0950 0x7f6e7c134480 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:24.857 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.849+0000 7f6e8244d640 1 --2- 172.21.15.42:0/4223726169 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f6e7c09fe60 0x7f6e7c1cf080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:24.857 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.849+0000 7f6e8244d640 1 -- 172.21.15.42:0/4223726169 >> 172.21.15.42:0/4223726169 conn(0x7f6e7c075ae0 msgr2=0x7f6e7c071050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:14:24.857 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.849+0000 7f6e8244d640 1 -- 172.21.15.42:0/4223726169 shutdown_connections 2024-09-16T04:14:24.857 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.849+0000 7f6e8244d640 1 -- 172.21.15.42:0/4223726169 wait complete. 2024-09-16T04:14:24.857 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.849+0000 7f6e8244d640 1 librados: shutdown 2024-09-16T04:14:24.872 INFO:tasks.ceph:Scrubbing osd.5 2024-09-16T04:14:24.873 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-09-16T04:14:24.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.985+0000 7f855fb12640 10 monclient: get_monmap_and_config 2024-09-16T04:14:24.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.985+0000 7f855fb12640 10 monclient: build_initial_monmap 2024-09-16T04:14:24.991 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.985+0000 7f855fb12640 10 monclient: monmap: 2024-09-16T04:14:24.991 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:14:24.991 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:14:24.991 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:14:24.989804+0000 2024-09-16T04:14:24.991 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:14:24.989804+0000 2024-09-16T04:14:24.991 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:14:24.991 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:24.991 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:14:24.992 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:14:24.992 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:14:24.992 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:14:24.992 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:14:24.992 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:24.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.985+0000 7f855fb12640 1 Processor -- start 2024-09-16T04:14:24.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.985+0000 7f855fb12640 1 -- start start 2024-09-16T04:14:24.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.985+0000 7f855fb12640 10 monclient: init 2024-09-16T04:14:24.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.985+0000 7f855fb12640 10 monclient: _reopen_session rank -1 2024-09-16T04:14:24.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.985+0000 7f855fb12640 10 monclient: _add_conns ranks=[1,0,2,4,3] 2024-09-16T04:14:24.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.985+0000 7f855fb12640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f85581374e0 addr v1:172.21.15.154:6789/0 2024-09-16T04:14:24.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.985+0000 7f855fb12640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8558137fd0 addr v1:172.21.15.42:6789/0 2024-09-16T04:14:24.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.985+0000 7f855fb12640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f85580a2560 0x7f855813fe50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:24.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.985+0000 7f855fb12640 10 monclient(hunting): picked mon.noname-b con 0x7f85580a2560 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:14:24.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.985+0000 7f855fb12640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f85580a75e0 con 0x7f8558137fd0 2024-09-16T04:14:24.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.985+0000 7f855fb12640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f85580a1350 con 0x7f85581374e0 2024-09-16T04:14:24.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.985+0000 7f855fb12640 10 monclient(hunting): start opening mon connection 2024-09-16T04:14:24.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.985+0000 7f855fb12640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f85580a1910 con 0x7f85580a2560 2024-09-16T04:14:24.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.985+0000 7f855fb12640 10 monclient(hunting): _renew_subs 2024-09-16T04:14:24.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.985+0000 7f855fb12640 10 monclient(hunting): authenticate will time out at 2911.959567s 2024-09-16T04:14:24.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.985+0000 7f855d0ad640 1 --1- >> v1:172.21.15.42:6789/0 conn(0x7f8558137fd0 0x7f855813c720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.42:6789/0 says I am v1:172.21.15.42:55192/0 (socket says 172.21.15.42:55192) 2024-09-16T04:14:24.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.985+0000 7f855d0ad640 1 -- 172.21.15.42:0/1428306030 learned_addr learned my addr 172.21.15.42:0/1428306030 (peer_addr_for_me v1:172.21.15.42:0/0) 2024-09-16T04:14:24.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.985+0000 7f855e0af640 1 --2- 172.21.15.42:0/1428306030 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f85580a2560 0x7f855813fe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:24.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.985+0000 7f855e0af640 10 monclient(hunting): get_auth_request con 0x7f85580a2560 auth_method 0 2024-09-16T04:14:24.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.985+0000 7f855e0af640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:14:24.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.985+0000 7f855e0af640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:24.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.985+0000 7f855e0af640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:24.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.985+0000 7f855e0af640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:14:24.995 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.985+0000 7f855e0af640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:14:24.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.985+0000 7f855e0af640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:14:24.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.985+0000 7f855c8ac640 1 -- 172.21.15.42:0/1428306030 <== mon.0 v1:172.21.15.42:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2092310372 0 0) 0x7f85580a75e0 con 0x7f8558137fd0 2024-09-16T04:14:24.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.989+0000 7f855c8ac640 10 monclient(hunting): _negotiate 2024-09-16T04:14:24.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.989+0000 7f855c8ac640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:24.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.989+0000 7f855c8ac640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:24.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.989+0000 7f855c8ac640 10 monclient(hunting): my global_id is 1104231 2024-09-16T04:14:24.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.989+0000 7f855c8ac640 1 -- 172.21.15.42:0/1428306030 --> v1:172.21.15.42:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8540003530 con 0x7f8558137fd0 2024-09-16T04:14:24.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.989+0000 7f855e0af640 10 monclient(hunting): handle_auth_done global_id 1094288 payload 274 2024-09-16T04:14:24.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.989+0000 7f855e0af640 1 -- 172.21.15.42:0/1428306030 >> v1:172.21.15.154:6789/0 conn(0x7f85581374e0 legacy=0x7f85581378c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:14:24.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.989+0000 7f855e0af640 1 -- 172.21.15.42:0/1428306030 >> v1:172.21.15.42:6789/0 conn(0x7f8558137fd0 legacy=0x7f855813c720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:24.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.989+0000 7f855e0af640 10 monclient: _finish_hunting 0 2024-09-16T04:14:24.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.989+0000 7f855e0af640 1 monclient: found mon.noname-b 2024-09-16T04:14:24.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.989+0000 7f855e0af640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:14:24.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.989+0000 7f855e0af640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T04:14:24.996 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.989+0000 7f855e0af640 1 -- 172.21.15.42:0/1428306030 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85580a1630 con 0x7f85580a2560 2024-09-16T04:14:25.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.989+0000 7f855e0af640 1 --2- 172.21.15.42:0/1428306030 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f85580a2560 0x7f855813fe50 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f85480097f0 tx=0x7f854801c360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4601af6a994182b1 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:14:25.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.989+0000 7f855c8ac640 1 -- 172.21.15.42:0/1428306030 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85480b0070 con 0x7f85580a2560 2024-09-16T04:14:25.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.989+0000 7f855c8ac640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:25.004 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.989+0000 7f855c8ac640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:14:25.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.989+0000 7f855c8ac640 10 monclient: dump: 2024-09-16T04:14:25.005 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:14:25.005 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:14:25.005 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:25.005 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:25.005 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:25.005 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:25.005 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:14:25.005 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:14:25.005 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:14:25.005 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:25.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.989+0000 7f855c8ac640 10 monclient: _finish_auth 0 2024-09-16T04:14:25.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.989+0000 7f855c8ac640 10 monclient: _check_auth_tickets 2024-09-16T04:14:25.005 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.989+0000 7f855c8ac640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:14:25.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.989+0000 7f855c8ac640 1 -- 172.21.15.42:0/1428306030 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f85480ab600 con 0x7f85580a2560 2024-09-16T04:14:25.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.989+0000 7f855fb12640 5 monclient: authenticate success, global_id 1094288 2024-09-16T04:14:25.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.989+0000 7f855fb12640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-16T04:14:25.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.989+0000 7f855c8ac640 10 monclient: handle_config config(0 keys) 2024-09-16T04:14:25.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.989+0000 7f855fb12640 10 monclient: get_monmap_and_config success 2024-09-16T04:14:25.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.989+0000 7f855fb12640 4 set_mon_vals no callback set 2024-09-16T04:14:25.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.989+0000 7f855c8ac640 1 -- 172.21.15.42:0/1428306030 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85480ab9d0 con 0x7f85580a2560 2024-09-16T04:14:25.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.989+0000 7f855c8ac640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:25.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.989+0000 7f855c8ac640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:14:25.006 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.989+0000 7f855c8ac640 10 monclient: dump: 2024-09-16T04:14:25.006 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:14:25.006 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:14:25.006 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:25.007 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:25.007 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:25.007 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:25.007 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:14:25.007 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:14:25.007 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:14:25.007 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:25.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.989+0000 7f855fb12640 10 monclient: shutdown 2024-09-16T04:14:25.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.989+0000 7f855fb12640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:14:25.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.989+0000 7f855fb12640 1 -- 172.21.15.42:0/1428306030 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f85580a2560 msgr2=0x7f855813fe50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:25.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.989+0000 7f855fb12640 1 --2- 172.21.15.42:0/1428306030 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f85580a2560 0x7f855813fe50 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f85480097f0 tx=0x7f854801c360 comp rx=0 tx=0).stop 2024-09-16T04:14:25.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.989+0000 7f855fb12640 1 -- 172.21.15.42:0/1428306030 shutdown_connections 2024-09-16T04:14:25.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.989+0000 7f855fb12640 1 --2- 172.21.15.42:0/1428306030 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f85580a2560 0x7f855813fe50 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:25.007 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.989+0000 7f855fb12640 1 -- 172.21.15.42:0/1428306030 >> 172.21.15.42:0/1428306030 conn(0x7f85580aa320 msgr2=0x7f85580a6820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:14:25.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.989+0000 7f855fb12640 1 -- 172.21.15.42:0/1428306030 shutdown_connections 2024-09-16T04:14:25.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.989+0000 7f855fb12640 1 -- 172.21.15.42:0/1428306030 wait complete. 2024-09-16T04:14:25.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.989+0000 7f855fb12640 10 monclient: build_initial_monmap 2024-09-16T04:14:25.008 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.989+0000 7f855fb12640 10 monclient: monmap: 2024-09-16T04:14:25.008 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:14:25.008 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:14:25.008 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:14:25.008 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:14:25.008 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:14:25.008 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:25.008 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:14:25.008 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:14:25.008 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:14:25.009 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:25.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.989+0000 7f855fb12640 1 librados: starting msgr at 2024-09-16T04:14:25.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.989+0000 7f855fb12640 1 librados: starting objecter 2024-09-16T04:14:25.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.989+0000 7f855fb12640 1 Processor -- start 2024-09-16T04:14:25.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.989+0000 7f855fb12640 1 -- start start 2024-09-16T04:14:25.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.989+0000 7f855fb12640 1 librados: setting wanted keys 2024-09-16T04:14:25.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.989+0000 7f855fb12640 1 librados: calling monclient init 2024-09-16T04:14:25.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.989+0000 7f855fb12640 10 monclient: init 2024-09-16T04:14:25.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.993+0000 7f855fb12640 10 monclient: _reopen_session rank -1 2024-09-16T04:14:25.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.993+0000 7f855fb12640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-16T04:14:25.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.993+0000 7f855fb12640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f85580a2560 0x7f85581cf4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:25.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.993+0000 7f855fb12640 10 monclient(hunting): picked mon.noname-c con 0x7f85580a2560 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:14:25.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.993+0000 7f855fb12640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f85581374e0 0x7f8558136920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:25.009 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.993+0000 7f855fb12640 10 monclient(hunting): picked mon.noname-b con 0x7f85581374e0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:14:25.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.993+0000 7f855fb12640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8558137fd0 0x7f8558136e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:25.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.993+0000 7f855fb12640 10 monclient(hunting): picked mon.noname-a con 0x7f8558137fd0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:14:25.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.993+0000 7f855fb12640 10 monclient(hunting): start opening mon connection 2024-09-16T04:14:25.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.993+0000 7f855fb12640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f8558142f40 con 0x7f8558137fd0 2024-09-16T04:14:25.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.993+0000 7f855fb12640 10 monclient(hunting): start opening mon connection 2024-09-16T04:14:25.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.993+0000 7f855fb12640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f8558142dc0 con 0x7f85581374e0 2024-09-16T04:14:25.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.993+0000 7f855fb12640 10 monclient(hunting): start opening mon connection 2024-09-16T04:14:25.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.993+0000 7f855fb12640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f85581430c0 con 0x7f85580a2560 2024-09-16T04:14:25.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.993+0000 7f855fb12640 10 monclient(hunting): _renew_subs 2024-09-16T04:14:25.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.993+0000 7f855fb12640 10 monclient(hunting): authenticate will time out at 2911.964372s 2024-09-16T04:14:25.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.993+0000 7f855d8ae640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f85580a2560 0x7f85581cf4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:25.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.993+0000 7f855e0af640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8558137fd0 0x7f8558136e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:25.010 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.993+0000 7f855e0af640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8558137fd0 0x7f8558136e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:48582/0 (socket says 172.21.15.42:48582) 2024-09-16T04:14:25.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.993+0000 7f855d8ae640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f85580a2560 0x7f85581cf4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:33242/0 (socket says 172.21.15.42:33242) 2024-09-16T04:14:25.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.993+0000 7f855e0af640 1 -- 172.21.15.42:0/1459430434 learned_addr learned my addr 172.21.15.42:0/1459430434 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:14:25.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.993+0000 7f855e0af640 10 monclient(hunting): get_auth_request con 0x7f8558137fd0 auth_method 0 2024-09-16T04:14:25.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.993+0000 7f855e0af640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:14:25.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.993+0000 7f855e0af640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:25.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.993+0000 7f855e0af640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:25.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.993+0000 7f855d8ae640 10 monclient(hunting): get_auth_request con 0x7f85580a2560 auth_method 0 2024-09-16T04:14:25.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.993+0000 7f855d8ae640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:14:25.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.993+0000 7f855d8ae640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:25.011 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.993+0000 7f855d8ae640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:25.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.993+0000 7f855e0af640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:14:25.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.993+0000 7f855e0af640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:14:25.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.993+0000 7f855e0af640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:14:25.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.993+0000 7f855d8ae640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:14:25.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.993+0000 7f855d8ae640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:14:25.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.993+0000 7f855d8ae640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:14:25.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.993+0000 7f855d0ad640 1 --2- 172.21.15.42:0/1459430434 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f85581374e0 0x7f8558136920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:25.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.993+0000 7f855d8ae640 10 monclient(hunting): handle_auth_done global_id 1094291 payload 824 2024-09-16T04:14:25.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.993+0000 7f855d8ae640 1 -- 172.21.15.42:0/1459430434 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f85581374e0 msgr2=0x7f8558136920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:25.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.993+0000 7f855d8ae640 1 --2- 172.21.15.42:0/1459430434 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f85581374e0 0x7f8558136920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:25.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.993+0000 7f855d8ae640 1 -- 172.21.15.42:0/1459430434 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8558137fd0 msgr2=0x7f8558136e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:25.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.993+0000 7f855d8ae640 1 --2- 172.21.15.42:0/1459430434 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8558137fd0 0x7f8558136e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:25.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.993+0000 7f855d8ae640 10 monclient: _finish_hunting 0 2024-09-16T04:14:25.012 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.993+0000 7f855d8ae640 1 monclient: found mon.noname-c 2024-09-16T04:14:25.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.993+0000 7f855d8ae640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:14:25.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.993+0000 7f855d8ae640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T04:14:25.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.993+0000 7f855d8ae640 1 -- 172.21.15.42:0/1459430434 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f855806f830 con 0x7f85580a2560 2024-09-16T04:14:25.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.993+0000 7f855d0ad640 10 monclient: get_auth_request con 0x7f85581374e0 auth_method 0 2024-09-16T04:14:25.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.993+0000 7f855e0af640 1 --2- 172.21.15.42:0/1459430434 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f8558137fd0 0x7f8558136e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:14:25.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.993+0000 7f855d0ad640 1 --2- 172.21.15.42:0/1459430434 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f85581374e0 0x7f8558136920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T04:14:25.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.993+0000 7f855d8ae640 1 --2- 172.21.15.42:0/1459430434 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f85580a2560 0x7f85581cf4d0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f855000c410 tx=0x7f855000c8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:14:25.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.993+0000 7f854e7fc640 10 client.?.objecter ms_handle_connect 0x7f85580a2560 2024-09-16T04:14:25.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.993+0000 7f854e7fc640 10 client.?.objecter resend_mon_ops 2024-09-16T04:14:25.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.993+0000 7f854e7fc640 1 -- 172.21.15.42:0/1459430434 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8550017070 con 0x7f85580a2560 2024-09-16T04:14:25.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.993+0000 7f854e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:25.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.993+0000 7f854e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:14:25.013 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.993+0000 7f854e7fc640 10 monclient: dump: 2024-09-16T04:14:25.014 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:14:25.014 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:14:25.014 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:25.014 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:25.014 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:25.014 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:25.014 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:14:25.014 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:14:25.014 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:14:25.014 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:25.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.993+0000 7f854e7fc640 10 monclient: _finish_auth 0 2024-09-16T04:14:25.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.993+0000 7f854e7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:14:25.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.993+0000 7f854e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:14:25.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.993+0000 7f854e7fc640 1 -- 172.21.15.42:0/1459430434 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f85500069e0 con 0x7f85580a2560 2024-09-16T04:14:25.014 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.993+0000 7f854e7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:14:25.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.993+0000 7f854e7fc640 1 -- 172.21.15.42:0/1459430434 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8550006e10 con 0x7f85580a2560 2024-09-16T04:14:25.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.993+0000 7f854e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:25.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.993+0000 7f854e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:14:25.015 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.993+0000 7f854e7fc640 10 monclient: dump: 2024-09-16T04:14:25.015 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:14:25.015 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:14:25.015 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:25.015 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:25.015 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:25.015 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:25.015 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:14:25.015 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:14:25.015 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:14:25.015 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:25.016 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.993+0000 7f854f7fe640 4 set_mon_vals no callback set 2024-09-16T04:14:25.016 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.993+0000 7f855fb12640 5 monclient: authenticate success, global_id 1094291 2024-09-16T04:14:25.016 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.993+0000 7f855fb12640 10 monclient: _renew_subs 2024-09-16T04:14:25.016 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.993+0000 7f855fb12640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:14:25.016 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.993+0000 7f855fb12640 1 -- 172.21.15.42:0/1459430434 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f855813d460 con 0x7f85580a2560 2024-09-16T04:14:25.016 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.993+0000 7f855fb12640 10 client.1094291.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:14:25.016 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.993+0000 7f855fb12640 10 monclient: _renew_subs 2024-09-16T04:14:25.016 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.993+0000 7f855fb12640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:14:25.016 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.993+0000 7f855fb12640 1 -- 172.21.15.42:0/1459430434 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f855813ebd0 con 0x7f85580a2560 2024-09-16T04:14:25.016 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.993+0000 7f855fb12640 1 librados: init done 2024-09-16T04:14:25.016 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.993+0000 7f855fb12640 10 client.1094291.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:14:25.016 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.993+0000 7f855fb12640 15 client.1094291.objecter _session_command_op_assign -1 1 2024-09-16T04:14:25.016 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.993+0000 7f855fb12640 20 client.1094291.objecter _get_session osd=-1 returning homeless 2024-09-16T04:14:25.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.993+0000 7f855fb12640 10 client.1094291.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:14:25.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.993+0000 7f855fb12640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:14:25.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.993+0000 7f855fb12640 1 -- 172.21.15.42:0/1459430434 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f85581378c0 con 0x7f85580a2560 2024-09-16T04:14:25.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.993+0000 7f854e7fc640 1 -- 172.21.15.42:0/1459430434 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f855001a070 con 0x7f85580a2560 2024-09-16T04:14:25.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.993+0000 7f854e7fc640 1 --2- 172.21.15.42:0/1459430434 >> v2:172.21.15.154:6812/254845342 conn(0x7f8544037840 0x7f8544039d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:25.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.993+0000 7f854e7fc640 1 -- 172.21.15.42:0/1459430434 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(710..710 src has 681..710) ==== 4681+0+0 (secure 0 0 0) 0x7f8550047e50 con 0x7f85580a2560 2024-09-16T04:14:25.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.993+0000 7f854e7fc640 10 client.1094291.objecter ms_dispatch 0x7f8558000ba0 osd_map(710..710 src has 681..710) 2024-09-16T04:14:25.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.993+0000 7f854e7fc640 3 client.1094291.objecter handle_osd_map got epochs [710,710] > 0 2024-09-16T04:14:25.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.993+0000 7f854e7fc640 3 client.1094291.objecter handle_osd_map decoding full epoch 710 2024-09-16T04:14:25.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.993+0000 7f854e7fc640 10 client.1094291.objecter checking command 1 2024-09-16T04:14:25.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.993+0000 7f854e7fc640 1 --2- 172.21.15.42:0/1459430434 >> v2:172.21.15.154:6808/3026127822 conn(0x7f854403eca0 0x7f8544041140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:25.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.993+0000 7f854e7fc640 20 client.1094291.objecter _get_session s=0x7f854403e5f0 osd=5 3 2024-09-16T04:14:25.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.993+0000 7f854e7fc640 20 client.1094291.objecter put_session s=0x7f854403e5f0 osd=5 3 2024-09-16T04:14:25.017 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.993+0000 7f854e7fc640 15 client.1094291.objecter _session_command_op_remove -1 1 2024-09-16T04:14:25.018 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.993+0000 7f854e7fc640 20 client.1094291.objecter _get_session s=0x7f854403e5f0 osd=5 3 2024-09-16T04:14:25.018 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.993+0000 7f854e7fc640 20 client.1094291.objecter get_session s=0x7f854403e5f0 osd=5 3 2024-09-16T04:14:25.018 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.993+0000 7f854e7fc640 15 client.1094291.objecter _session_command_op_assign 5 1 2024-09-16T04:14:25.018 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.993+0000 7f854e7fc640 20 client.1094291.objecter put_session s=0x7f854403e5f0 osd=5 4 2024-09-16T04:14:25.018 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.993+0000 7f854e7fc640 10 client.1094291.objecter _send_command 1 2024-09-16T04:14:25.018 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.993+0000 7f854e7fc640 1 -- 172.21.15.42:0/1459430434 --> v2:172.21.15.154:6808/3026127822 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8550016ec0 con 0x7f854403eca0 2024-09-16T04:14:25.018 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.993+0000 7f854e7fc640 20 client.1094291.objecter dump_active .. 0 homeless 2024-09-16T04:14:25.018 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.993+0000 7f854e7fc640 1 -- 172.21.15.42:0/1459430434 <== mon.2 v2:172.21.15.42:3301/0 6 ==== mon_get_version_reply(handle=1 version=710) ==== 24+0+0 (secure 0 0 0) 0x7f855004a4c0 con 0x7f85580a2560 2024-09-16T04:14:25.018 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.993+0000 7f854e7fc640 10 monclient: handle_get_version_reply finishing 1 version 710 2024-09-16T04:14:25.018 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.993+0000 7f855e0af640 1 --2- 172.21.15.42:0/1459430434 >> v2:172.21.15.154:6808/3026127822 conn(0x7f854403eca0 0x7f8544041140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:25.018 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.993+0000 7f855d0ad640 1 --2- 172.21.15.42:0/1459430434 >> v2:172.21.15.154:6812/254845342 conn(0x7f8544037840 0x7f8544039d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:25.018 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.993+0000 7f855e0af640 10 monclient: get_auth_request con 0x7f854403eca0 auth_method 0 2024-09-16T04:14:25.018 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.993+0000 7f855d0ad640 10 monclient: get_auth_request con 0x7f8544037840 auth_method 0 2024-09-16T04:14:25.018 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.997+0000 7f855e0af640 1 --2- 172.21.15.42:0/1459430434 >> v2:172.21.15.154:6808/3026127822 conn(0x7f854403eca0 0x7f8544041140 crc :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:14:25.019 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.997+0000 7f854e7fc640 10 client.1094291.objecter ms_handle_connect 0x7f854403eca0 2024-09-16T04:14:25.019 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.997+0000 7f854e7fc640 10 client.1094291.objecter ms_handle_connect 0x7f8544037840 2024-09-16T04:14:25.019 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.997+0000 7f855d0ad640 1 --2- 172.21.15.42:0/1459430434 >> v2:172.21.15.154:6812/254845342 conn(0x7f8544037840 0x7f8544039d00 secure :-1 s=READY pgs=4025 cs=0 l=1 rev1=1 crypto rx=0x7f855813d060 tx=0x7f8554009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:14:25.019 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.997+0000 7f854e7fc640 1 -- 172.21.15.42:0/1459430434 <== osd.5 v2:172.21.15.154:6808/3026127822 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8550016ec0 con 0x7f854403eca0 2024-09-16T04:14:25.019 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.997+0000 7f854e7fc640 10 client.1094291.objecter ms_dispatch 0x7f8558000ba0 command_reply(tid 1: 0 ) 2024-09-16T04:14:25.019 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.997+0000 7f854e7fc640 10 client.1094291.objecter _finish_command 1 = system:0 2024-09-16T04:14:25.019 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.997+0000 7f854e7fc640 20 client.1094291.objecter put_session s=0x7f854403e5f0 osd=5 4 2024-09-16T04:14:25.019 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:24.997+0000 7f854e7fc640 15 client.1094291.objecter _session_command_op_remove 5 1 2024-09-16T04:14:25.020 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.009+0000 7f855fb12640 10 client.1094291.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-09-16T04:14:25.020 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.009+0000 7f855fb12640 15 client.1094291.objecter _session_command_op_assign -1 2 2024-09-16T04:14:25.020 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.009+0000 7f855fb12640 20 client.1094291.objecter _get_session s=0x7f854403e5f0 osd=5 3 2024-09-16T04:14:25.020 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.009+0000 7f855fb12640 20 client.1094291.objecter put_session s=0x7f854403e5f0 osd=5 3 2024-09-16T04:14:25.020 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.009+0000 7f855fb12640 20 client.1094291.objecter _get_session s=0x7f854403e5f0 osd=5 3 2024-09-16T04:14:25.020 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.009+0000 7f855fb12640 15 client.1094291.objecter _session_command_op_remove -1 2 2024-09-16T04:14:25.020 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.009+0000 7f855fb12640 20 client.1094291.objecter get_session s=0x7f854403e5f0 osd=5 3 2024-09-16T04:14:25.021 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.009+0000 7f855fb12640 15 client.1094291.objecter _session_command_op_assign 5 2 2024-09-16T04:14:25.021 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.009+0000 7f855fb12640 20 client.1094291.objecter put_session s=0x7f854403e5f0 osd=5 4 2024-09-16T04:14:25.021 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.009+0000 7f855fb12640 10 client.1094291.objecter _send_command 2 2024-09-16T04:14:25.021 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.009+0000 7f855fb12640 1 -- 172.21.15.42:0/1459430434 --> v2:172.21.15.154:6808/3026127822 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f855813e1e0 con 0x7f854403eca0 2024-09-16T04:14:25.021 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.009+0000 7f854e7fc640 1 -- 172.21.15.42:0/1459430434 <== osd.5 v2:172.21.15.154:6808/3026127822 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f855813e1e0 con 0x7f854403eca0 2024-09-16T04:14:25.021 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.009+0000 7f854e7fc640 10 client.1094291.objecter ms_dispatch 0x7f8558000ba0 command_reply(tid 2: 0 ) 2024-09-16T04:14:25.021 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.009+0000 7f854e7fc640 10 client.1094291.objecter _finish_command 2 = system:0 2024-09-16T04:14:25.021 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.009+0000 7f854e7fc640 20 client.1094291.objecter put_session s=0x7f854403e5f0 osd=5 4 2024-09-16T04:14:25.021 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.009+0000 7f854e7fc640 15 client.1094291.objecter _session_command_op_remove 5 2 2024-09-16T04:14:25.021 INFO:teuthology.orchestra.run.smithi042.stdout:{ 2024-09-16T04:14:25.021 INFO:teuthology.orchestra.run.smithi042.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T04:14:25.022 INFO:teuthology.orchestra.run.smithi042.stdout:} 2024-09-16T04:14:25.023 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.009+0000 7f855fb12640 10 librados: watch_flush enter 2024-09-16T04:14:25.023 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.009+0000 7f855fb12640 10 librados: watch_flush exit 2024-09-16T04:14:25.023 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.009+0000 7f855fb12640 10 client.1094291.objecter close_session for osd.5 2024-09-16T04:14:25.023 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.009+0000 7f855fb12640 1 -- 172.21.15.42:0/1459430434 >> v2:172.21.15.154:6808/3026127822 conn(0x7f854403eca0 msgr2=0x7f8544041140 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:25.023 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.009+0000 7f855fb12640 1 --2- 172.21.15.42:0/1459430434 >> v2:172.21.15.154:6808/3026127822 conn(0x7f854403eca0 0x7f8544041140 crc :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:25.023 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.009+0000 7f855fb12640 20 client.1094291.objecter put_session s=0x7f854403e5f0 osd=5 1 2024-09-16T04:14:25.023 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.009+0000 7f855fb12640 20 client.1094291.objecter shutdown clearing up homeless session... 2024-09-16T04:14:25.023 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.009+0000 7f855fb12640 10 client.1094291.objecter successfully canceled tick 2024-09-16T04:14:25.023 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.009+0000 7f855fb12640 1 -- 172.21.15.42:0/1459430434 >> v2:172.21.15.154:6812/254845342 conn(0x7f8544037840 msgr2=0x7f8544039d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:25.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.009+0000 7f855fb12640 1 --2- 172.21.15.42:0/1459430434 >> v2:172.21.15.154:6812/254845342 conn(0x7f8544037840 0x7f8544039d00 secure :-1 s=READY pgs=4025 cs=0 l=1 rev1=1 crypto rx=0x7f855813d060 tx=0x7f8554009040 comp rx=0 tx=0).stop 2024-09-16T04:14:25.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.009+0000 7f855fb12640 10 monclient: shutdown 2024-09-16T04:14:25.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.009+0000 7f855fb12640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:14:25.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.009+0000 7f855fb12640 1 -- 172.21.15.42:0/1459430434 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f85580a2560 msgr2=0x7f85581cf4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:25.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.009+0000 7f855fb12640 1 --2- 172.21.15.42:0/1459430434 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f85580a2560 0x7f85581cf4d0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f855000c410 tx=0x7f855000c8d0 comp rx=0 tx=0).stop 2024-09-16T04:14:25.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.009+0000 7f855e0af640 1 -- 172.21.15.42:0/1459430434 reap_dead start 2024-09-16T04:14:25.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.009+0000 7f855fb12640 1 -- 172.21.15.42:0/1459430434 shutdown_connections 2024-09-16T04:14:25.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.009+0000 7f855fb12640 1 -- 172.21.15.42:0/1459430434 >> 172.21.15.42:0/1459430434 conn(0x7f85580aa320 msgr2=0x7f855806d5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:14:25.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.009+0000 7f855fb12640 1 -- 172.21.15.42:0/1459430434 shutdown_connections 2024-09-16T04:14:25.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.009+0000 7f855fb12640 1 -- 172.21.15.42:0/1459430434 wait complete. 2024-09-16T04:14:25.024 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.009+0000 7f855fb12640 1 librados: shutdown 2024-09-16T04:14:25.031 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-09-16T04:14:25.145 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.137+0000 7f907a6d3640 10 monclient: get_monmap_and_config 2024-09-16T04:14:25.145 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.141+0000 7f907a6d3640 10 monclient: build_initial_monmap 2024-09-16T04:14:25.145 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.141+0000 7f907a6d3640 10 monclient: monmap: 2024-09-16T04:14:25.145 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:14:25.145 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:14:25.146 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:14:25.144157+0000 2024-09-16T04:14:25.146 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:14:25.144157+0000 2024-09-16T04:14:25.146 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:14:25.146 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:25.146 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:14:25.146 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:14:25.146 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:14:25.146 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:14:25.146 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:14:25.146 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:25.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.141+0000 7f907a6d3640 1 Processor -- start 2024-09-16T04:14:25.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.141+0000 7f907a6d3640 1 -- start start 2024-09-16T04:14:25.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.141+0000 7f907a6d3640 10 monclient: init 2024-09-16T04:14:25.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.141+0000 7f907a6d3640 10 monclient: _reopen_session rank -1 2024-09-16T04:14:25.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.141+0000 7f907a6d3640 10 monclient: _add_conns ranks=[1,2,4,0,3] 2024-09-16T04:14:25.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.141+0000 7f907a6d3640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f90740a1330 addr v1:172.21.15.154:6789/0 2024-09-16T04:14:25.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.141+0000 7f907a6d3640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f90740a1710 0x7f90740a1b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:25.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.141+0000 7f907a6d3640 10 monclient(hunting): picked mon.noname-b con 0x7f90740a1710 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:14:25.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.141+0000 7f907a6d3640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f90741393a0 0x7f907413b790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:25.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.141+0000 7f907a6d3640 10 monclient(hunting): picked mon.noname-c con 0x7f90741393a0 addr v2:172.21.15.154:3300/0 2024-09-16T04:14:25.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.141+0000 7f907a6d3640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f907409e4c0 con 0x7f90740a1330 2024-09-16T04:14:25.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.141+0000 7f907a6d3640 10 monclient(hunting): start opening mon connection 2024-09-16T04:14:25.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.141+0000 7f907a6d3640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f907409edf0 con 0x7f90741393a0 2024-09-16T04:14:25.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.141+0000 7f907a6d3640 10 monclient(hunting): start opening mon connection 2024-09-16T04:14:25.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.141+0000 7f907a6d3640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f907409cd60 con 0x7f90740a1710 2024-09-16T04:14:25.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.141+0000 7f907a6d3640 10 monclient(hunting): _renew_subs 2024-09-16T04:14:25.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.141+0000 7f907a6d3640 10 monclient(hunting): authenticate will time out at 2912.113514s 2024-09-16T04:14:25.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.141+0000 7f90737fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f90740a1710 0x7f90740a1b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:25.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.141+0000 7f90737fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f90740a1710 0x7f90740a1b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:33252/0 (socket says 172.21.15.42:33252) 2024-09-16T04:14:25.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.141+0000 7f90737fe640 1 -- 172.21.15.42:0/1752528483 learned_addr learned my addr 172.21.15.42:0/1752528483 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:14:25.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.141+0000 7f90737fe640 10 monclient(hunting): get_auth_request con 0x7f90740a1710 auth_method 0 2024-09-16T04:14:25.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.141+0000 7f90737fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:14:25.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.141+0000 7f90737fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:25.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.141+0000 7f90737fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:25.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.141+0000 7f90737fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:14:25.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.141+0000 7f90737fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:14:25.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.141+0000 7f90737fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:14:25.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.141+0000 7f9078c70640 1 --2- 172.21.15.42:0/1752528483 >> v2:172.21.15.154:3300/0 conn(0x7f90741393a0 0x7f907413b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:25.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.141+0000 7f90737fe640 10 monclient(hunting): handle_auth_done global_id 1094294 payload 274 2024-09-16T04:14:25.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.141+0000 7f90737fe640 1 -- 172.21.15.42:0/1752528483 >> v2:172.21.15.154:3300/0 conn(0x7f90741393a0 msgr2=0x7f907413b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:14:25.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.141+0000 7f90737fe640 1 --2- 172.21.15.42:0/1752528483 >> v2:172.21.15.154:3300/0 conn(0x7f90741393a0 0x7f907413b790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:25.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.141+0000 7f90737fe640 1 -- 172.21.15.42:0/1752528483 >> v1:172.21.15.154:6789/0 conn(0x7f90740a1330 legacy=0x7f907409f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:14:25.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.141+0000 7f90737fe640 10 monclient: _finish_hunting 0 2024-09-16T04:14:25.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.141+0000 7f90737fe640 1 monclient: found mon.noname-b 2024-09-16T04:14:25.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.141+0000 7f90737fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:14:25.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.141+0000 7f90737fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T04:14:25.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.141+0000 7f90737fe640 1 -- 172.21.15.42:0/1752528483 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90740a2050 con 0x7f90740a1710 2024-09-16T04:14:25.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.141+0000 7f9078c70640 10 monclient: get_auth_request con 0x7f90741393a0 auth_method 0 2024-09-16T04:14:25.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.141+0000 7f9078c70640 1 --2- 172.21.15.42:0/1752528483 >> v2:172.21.15.154:3300/0 conn(0x7f90741393a0 0x7f907413b790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T04:14:25.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.145+0000 7f90737fe640 1 --2- 172.21.15.42:0/1752528483 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f90740a1710 0x7f90740a1b10 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f9064009870 tx=0x7f906401c3e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8df61926f804976f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:14:25.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.145+0000 7f9072ffd640 1 -- 172.21.15.42:0/1752528483 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90640b0070 con 0x7f90740a1710 2024-09-16T04:14:25.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.145+0000 7f9072ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:25.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.145+0000 7f9072ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:14:25.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.145+0000 7f9072ffd640 10 monclient: dump: 2024-09-16T04:14:25.153 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:14:25.153 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:14:25.153 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:25.153 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:25.153 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:25.153 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:25.153 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:14:25.154 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:14:25.154 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:14:25.154 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:25.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.145+0000 7f9072ffd640 10 monclient: _finish_auth 0 2024-09-16T04:14:25.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.145+0000 7f9072ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:14:25.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.145+0000 7f9072ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:14:25.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.145+0000 7f9072ffd640 1 -- 172.21.15.42:0/1752528483 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f90640ab600 con 0x7f90740a1710 2024-09-16T04:14:25.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.145+0000 7f9072ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:14:25.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.145+0000 7f9072ffd640 1 -- 172.21.15.42:0/1752528483 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90640ab9d0 con 0x7f90740a1710 2024-09-16T04:14:25.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.145+0000 7f907a6d3640 5 monclient: authenticate success, global_id 1094294 2024-09-16T04:14:25.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.145+0000 7f907a6d3640 10 monclient: get_monmap_and_config success 2024-09-16T04:14:25.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.145+0000 7f907a6d3640 4 set_mon_vals no callback set 2024-09-16T04:14:25.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.145+0000 7f9072ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:25.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.145+0000 7f9072ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:14:25.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.145+0000 7f9072ffd640 10 monclient: dump: 2024-09-16T04:14:25.155 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:14:25.155 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:14:25.155 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:25.155 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:25.155 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:25.155 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:25.155 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:14:25.155 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:14:25.155 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:14:25.155 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:25.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.145+0000 7f907a6d3640 10 monclient: shutdown 2024-09-16T04:14:25.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.145+0000 7f907a6d3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:14:25.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.145+0000 7f907a6d3640 1 -- 172.21.15.42:0/1752528483 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f90740a1710 msgr2=0x7f90740a1b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:25.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.145+0000 7f907a6d3640 1 --2- 172.21.15.42:0/1752528483 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f90740a1710 0x7f90740a1b10 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f9064009870 tx=0x7f906401c3e0 comp rx=0 tx=0).stop 2024-09-16T04:14:25.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.145+0000 7f907a6d3640 1 -- 172.21.15.42:0/1752528483 shutdown_connections 2024-09-16T04:14:25.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.145+0000 7f907a6d3640 1 --2- 172.21.15.42:0/1752528483 >> v2:172.21.15.154:3300/0 conn(0x7f90741393a0 0x7f907413b790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:25.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.145+0000 7f907a6d3640 1 --2- 172.21.15.42:0/1752528483 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f90740a1710 0x7f90740a1b10 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:25.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.145+0000 7f907a6d3640 1 -- 172.21.15.42:0/1752528483 >> 172.21.15.42:0/1752528483 conn(0x7f90740a2420 msgr2=0x7f90740a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:14:25.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.145+0000 7f907a6d3640 1 -- 172.21.15.42:0/1752528483 shutdown_connections 2024-09-16T04:14:25.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.145+0000 7f907a6d3640 1 -- 172.21.15.42:0/1752528483 wait complete. 2024-09-16T04:14:25.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.145+0000 7f907a6d3640 10 monclient: build_initial_monmap 2024-09-16T04:14:25.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.145+0000 7f907a6d3640 10 monclient: monmap: 2024-09-16T04:14:25.161 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:14:25.161 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:14:25.161 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:14:25.161 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:14:25.161 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:14:25.161 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:25.161 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:14:25.161 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:14:25.161 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:14:25.161 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:25.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.145+0000 7f907a6d3640 1 librados: starting msgr at 2024-09-16T04:14:25.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.145+0000 7f907a6d3640 1 librados: starting objecter 2024-09-16T04:14:25.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.145+0000 7f907a6d3640 1 Processor -- start 2024-09-16T04:14:25.161 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.145+0000 7f907a6d3640 1 -- start start 2024-09-16T04:14:25.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.145+0000 7f907a6d3640 1 librados: setting wanted keys 2024-09-16T04:14:25.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.145+0000 7f907a6d3640 1 librados: calling monclient init 2024-09-16T04:14:25.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.145+0000 7f907a6d3640 10 monclient: init 2024-09-16T04:14:25.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.145+0000 7f907a6d3640 10 monclient: _reopen_session rank -1 2024-09-16T04:14:25.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.145+0000 7f907a6d3640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-16T04:14:25.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.145+0000 7f907a6d3640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f90740a1330 0x7f90740718d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:25.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.145+0000 7f907a6d3640 10 monclient(hunting): picked mon.noname-b con 0x7f90740a1330 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:14:25.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.145+0000 7f907a6d3640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f90740a1710 0x7f9074134560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:25.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.145+0000 7f907a6d3640 10 monclient(hunting): picked mon.noname-c con 0x7f90740a1710 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:14:25.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.145+0000 7f907a6d3640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f90741393a0 0x7f907409ba50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:25.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.145+0000 7f907a6d3640 10 monclient(hunting): picked mon.noname-a con 0x7f90741393a0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:14:25.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.145+0000 7f907a6d3640 10 monclient(hunting): start opening mon connection 2024-09-16T04:14:25.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.145+0000 7f907a6d3640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f907413e880 con 0x7f90741393a0 2024-09-16T04:14:25.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.145+0000 7f907a6d3640 10 monclient(hunting): start opening mon connection 2024-09-16T04:14:25.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.145+0000 7f907a6d3640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f907413e700 con 0x7f90740a1330 2024-09-16T04:14:25.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.145+0000 7f907a6d3640 10 monclient(hunting): start opening mon connection 2024-09-16T04:14:25.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.145+0000 7f907a6d3640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f907413ea00 con 0x7f90740a1710 2024-09-16T04:14:25.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.145+0000 7f907a6d3640 10 monclient(hunting): _renew_subs 2024-09-16T04:14:25.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.145+0000 7f907a6d3640 10 monclient(hunting): authenticate will time out at 2912.118190s 2024-09-16T04:14:25.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.145+0000 7f90737fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f90740a1710 0x7f9074134560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:25.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.145+0000 7f9078c70640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f90741393a0 0x7f907409ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:25.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.145+0000 7f90737fe640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f90740a1710 0x7f9074134560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:33264/0 (socket says 172.21.15.42:33264) 2024-09-16T04:14:25.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.145+0000 7f90737fe640 1 -- 172.21.15.42:0/590904192 learned_addr learned my addr 172.21.15.42:0/590904192 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:14:25.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.145+0000 7f9078c70640 10 monclient(hunting): get_auth_request con 0x7f90741393a0 auth_method 0 2024-09-16T04:14:25.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.145+0000 7f9078c70640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:14:25.163 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.145+0000 7f9078c70640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:25.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.145+0000 7f9078c70640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:25.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.145+0000 7f90737fe640 10 monclient(hunting): get_auth_request con 0x7f90740a1710 auth_method 0 2024-09-16T04:14:25.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.145+0000 7f90737fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:14:25.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.145+0000 7f90737fe640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:25.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.145+0000 7f90737fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:25.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.145+0000 7f9078c70640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:14:25.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.145+0000 7f9078c70640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:14:25.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.145+0000 7f9078c70640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:14:25.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.145+0000 7f90737fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:14:25.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.145+0000 7f90737fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:14:25.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.145+0000 7f90737fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:14:25.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.145+0000 7f9073fff640 1 --2- 172.21.15.42:0/590904192 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f90740a1330 0x7f90740718d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:25.164 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.145+0000 7f90737fe640 10 monclient(hunting): handle_auth_done global_id 1094297 payload 824 2024-09-16T04:14:25.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.145+0000 7f90737fe640 1 -- 172.21.15.42:0/590904192 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f90740a1330 msgr2=0x7f90740718d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:25.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.145+0000 7f90737fe640 1 --2- 172.21.15.42:0/590904192 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f90740a1330 0x7f90740718d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:25.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.145+0000 7f90737fe640 1 -- 172.21.15.42:0/590904192 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f90741393a0 msgr2=0x7f907409ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:25.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.145+0000 7f90737fe640 1 --2- 172.21.15.42:0/590904192 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f90741393a0 0x7f907409ba50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:25.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.145+0000 7f90737fe640 10 monclient: _finish_hunting 0 2024-09-16T04:14:25.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.145+0000 7f90737fe640 1 monclient: found mon.noname-c 2024-09-16T04:14:25.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.145+0000 7f90737fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:14:25.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.145+0000 7f90737fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T04:14:25.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.145+0000 7f90737fe640 1 -- 172.21.15.42:0/590904192 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f907411e770 con 0x7f90740a1710 2024-09-16T04:14:25.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.145+0000 7f9073fff640 10 monclient: get_auth_request con 0x7f90740a1330 auth_method 0 2024-09-16T04:14:25.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.145+0000 7f9078c70640 1 --2- 172.21.15.42:0/590904192 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f90741393a0 0x7f907409ba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:14:25.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.145+0000 7f9073fff640 1 --2- 172.21.15.42:0/590904192 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f90740a1330 0x7f90740718d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T04:14:25.165 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.145+0000 7f90737fe640 1 --2- 172.21.15.42:0/590904192 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f90740a1710 0x7f9074134560 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f90640ae040 tx=0x7f906401ac20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:14:25.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.145+0000 7f9070ff9640 10 client.?.objecter ms_handle_connect 0x7f90740a1710 2024-09-16T04:14:25.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.145+0000 7f9070ff9640 10 client.?.objecter resend_mon_ops 2024-09-16T04:14:25.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.145+0000 7f9070ff9640 1 -- 172.21.15.42:0/590904192 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90640ba020 con 0x7f90740a1710 2024-09-16T04:14:25.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.145+0000 7f9070ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:25.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.145+0000 7f9070ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:14:25.166 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.145+0000 7f9070ff9640 10 monclient: dump: 2024-09-16T04:14:25.166 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:14:25.166 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:14:25.166 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:25.166 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:25.166 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:25.166 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:25.166 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:14:25.166 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:14:25.166 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:14:25.167 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:25.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.145+0000 7f9070ff9640 10 monclient: _finish_auth 0 2024-09-16T04:14:25.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.145+0000 7f9070ff9640 10 monclient: _check_auth_tickets 2024-09-16T04:14:25.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.145+0000 7f9070ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:14:25.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.145+0000 7f9070ff9640 1 -- 172.21.15.42:0/590904192 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f90640b6640 con 0x7f90740a1710 2024-09-16T04:14:25.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.145+0000 7f907a6d3640 5 monclient: authenticate success, global_id 1094297 2024-09-16T04:14:25.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.145+0000 7f907a6d3640 10 monclient: _renew_subs 2024-09-16T04:14:25.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.145+0000 7f907a6d3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:14:25.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.145+0000 7f907a6d3640 1 -- 172.21.15.42:0/590904192 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f907411fc20 con 0x7f90740a1710 2024-09-16T04:14:25.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.145+0000 7f9070ff9640 10 monclient: handle_config config(0 keys) 2024-09-16T04:14:25.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.145+0000 7f9070ff9640 1 -- 172.21.15.42:0/590904192 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90640b0040 con 0x7f90740a1710 2024-09-16T04:14:25.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.145+0000 7f9070ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:25.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.145+0000 7f9070ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:14:25.167 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.145+0000 7f9070ff9640 10 monclient: dump: 2024-09-16T04:14:25.168 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:14:25.168 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:14:25.168 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:25.168 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:25.168 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:25.168 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:25.168 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:14:25.168 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:14:25.168 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:14:25.168 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:25.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.145+0000 7f907a6d3640 10 client.1094297.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:14:25.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.145+0000 7f907a6d3640 10 monclient: _renew_subs 2024-09-16T04:14:25.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.145+0000 7f90727fc640 4 set_mon_vals no callback set 2024-09-16T04:14:25.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.145+0000 7f907a6d3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:14:25.168 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.145+0000 7f907a6d3640 1 -- 172.21.15.42:0/590904192 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f907411f940 con 0x7f90740a1710 2024-09-16T04:14:25.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.145+0000 7f907a6d3640 1 librados: init done 2024-09-16T04:14:25.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.145+0000 7f907a6d3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:14:25.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.145+0000 7f907a6d3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:14:25.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.145+0000 7f907a6d3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:14:25.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.145+0000 7f907a6d3640 1 -- 172.21.15.42:0/590904192 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f90740a1b10 con 0x7f90740a1710 2024-09-16T04:14:25.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.145+0000 7f9070ff9640 1 -- 172.21.15.42:0/590904192 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f90640bd020 con 0x7f90740a1710 2024-09-16T04:14:25.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.149+0000 7f9070ff9640 1 --2- 172.21.15.42:0/590904192 >> v2:172.21.15.154:6812/254845342 conn(0x7f9054037810 0x7f9054039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:25.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.149+0000 7f9070ff9640 1 -- 172.21.15.42:0/590904192 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(710..710 src has 681..710) ==== 4681+0+0 (secure 0 0 0) 0x7f90640e2d30 con 0x7f90740a1710 2024-09-16T04:14:25.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.149+0000 7f9070ff9640 10 client.1094297.objecter ms_dispatch 0x7f9074000ba0 osd_map(710..710 src has 681..710) 2024-09-16T04:14:25.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.149+0000 7f9070ff9640 3 client.1094297.objecter handle_osd_map got epochs [710,710] > 0 2024-09-16T04:14:25.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.149+0000 7f9070ff9640 3 client.1094297.objecter handle_osd_map decoding full epoch 710 2024-09-16T04:14:25.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.149+0000 7f9070ff9640 20 client.1094297.objecter dump_active .. 0 homeless 2024-09-16T04:14:25.169 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.149+0000 7f9073fff640 1 --2- 172.21.15.42:0/590904192 >> v2:172.21.15.154:6812/254845342 conn(0x7f9054037810 0x7f9054039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:25.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.149+0000 7f9073fff640 10 monclient: get_auth_request con 0x7f9054037810 auth_method 0 2024-09-16T04:14:25.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.149+0000 7f9070ff9640 10 client.1094297.objecter ms_handle_connect 0x7f9054037810 2024-09-16T04:14:25.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.149+0000 7f9073fff640 1 --2- 172.21.15.42:0/590904192 >> v2:172.21.15.154:6812/254845342 conn(0x7f9054037810 0x7f9054039cd0 secure :-1 s=READY pgs=4026 cs=0 l=1 rev1=1 crypto rx=0x7f905c0016e0 tx=0x7f905c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:14:25.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.153+0000 7f9070ff9640 1 -- 172.21.15.42:0/590904192 <== mon.2 v2:172.21.15.42:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f90640c4030 con 0x7f90740a1710 2024-09-16T04:14:25.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.153+0000 7f9070ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:14:25.170 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.153+0000 7f9070ff9640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:14:25.312 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.305+0000 7f907a6d3640 1 -- 172.21.15.42:0/590904192 --> v2:172.21.15.154:6812/254845342 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) -- 0x7f9074069040 con 0x7f9054037810 2024-09-16T04:14:25.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.309+0000 7f9070ff9640 1 -- 172.21.15.42:0/590904192 <== mgr.4102 v2:172.21.15.154:6812/254845342 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f9074069040 con 0x7f9054037810 2024-09-16T04:14:25.313 INFO:teuthology.orchestra.run.smithi042.stderr:instructed osd(s) 5 to deep-scrub 2024-09-16T04:14:25.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.309+0000 7f907a6d3640 10 librados: watch_flush enter 2024-09-16T04:14:25.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.309+0000 7f907a6d3640 10 librados: watch_flush exit 2024-09-16T04:14:25.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.309+0000 7f907a6d3640 20 client.1094297.objecter shutdown clearing up homeless session... 2024-09-16T04:14:25.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.309+0000 7f907a6d3640 10 client.1094297.objecter successfully canceled tick 2024-09-16T04:14:25.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.309+0000 7f907a6d3640 1 -- 172.21.15.42:0/590904192 >> v2:172.21.15.154:6812/254845342 conn(0x7f9054037810 msgr2=0x7f9054039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:25.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.309+0000 7f907a6d3640 1 --2- 172.21.15.42:0/590904192 >> v2:172.21.15.154:6812/254845342 conn(0x7f9054037810 0x7f9054039cd0 secure :-1 s=READY pgs=4026 cs=0 l=1 rev1=1 crypto rx=0x7f905c0016e0 tx=0x7f905c009040 comp rx=0 tx=0).stop 2024-09-16T04:14:25.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.309+0000 7f907a6d3640 10 monclient: shutdown 2024-09-16T04:14:25.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.309+0000 7f907a6d3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:14:25.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.309+0000 7f907a6d3640 1 -- 172.21.15.42:0/590904192 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f90740a1710 msgr2=0x7f9074134560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:25.316 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.309+0000 7f907a6d3640 1 --2- 172.21.15.42:0/590904192 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f90740a1710 0x7f9074134560 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f90640ae040 tx=0x7f906401ac20 comp rx=0 tx=0).stop 2024-09-16T04:14:25.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.309+0000 7f907a6d3640 1 -- 172.21.15.42:0/590904192 shutdown_connections 2024-09-16T04:14:25.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.309+0000 7f907a6d3640 1 --2- 172.21.15.42:0/590904192 >> v2:172.21.15.154:6812/254845342 conn(0x7f9054037810 0x7f9054039cd0 unknown :-1 s=CLOSED pgs=4026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:25.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.309+0000 7f907a6d3640 1 --2- 172.21.15.42:0/590904192 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f90741393a0 0x7f907409ba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:25.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.309+0000 7f907a6d3640 1 --2- 172.21.15.42:0/590904192 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f90740a1710 0x7f9074134560 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:25.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.309+0000 7f907a6d3640 1 --2- 172.21.15.42:0/590904192 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f90740a1330 0x7f90740718d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:25.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.309+0000 7f907a6d3640 1 -- 172.21.15.42:0/590904192 >> 172.21.15.42:0/590904192 conn(0x7f90740a2420 msgr2=0x7f90740a3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:14:25.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.309+0000 7f907a6d3640 1 -- 172.21.15.42:0/590904192 shutdown_connections 2024-09-16T04:14:25.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.309+0000 7f907a6d3640 1 -- 172.21.15.42:0/590904192 wait complete. 2024-09-16T04:14:25.317 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.309+0000 7f907a6d3640 1 librados: shutdown 2024-09-16T04:14:25.332 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:14:25.452 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.445+0000 7f0945484640 10 monclient: get_monmap_and_config 2024-09-16T04:14:25.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.445+0000 7f0945484640 10 monclient: build_initial_monmap 2024-09-16T04:14:25.453 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.445+0000 7f0945484640 10 monclient: monmap: 2024-09-16T04:14:25.453 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:14:25.453 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:14:25.453 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:14:25.451706+0000 2024-09-16T04:14:25.453 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:14:25.451706+0000 2024-09-16T04:14:25.453 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:14:25.453 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:25.453 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:14:25.453 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:14:25.453 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:14:25.453 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:14:25.454 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:14:25.454 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:25.455 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.449+0000 7f0945484640 1 Processor -- start 2024-09-16T04:14:25.455 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.449+0000 7f0945484640 1 -- start start 2024-09-16T04:14:25.455 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.449+0000 7f0945484640 10 monclient: init 2024-09-16T04:14:25.455 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.449+0000 7f0945484640 10 monclient: _reopen_session rank -1 2024-09-16T04:14:25.455 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.449+0000 7f0945484640 10 monclient: _add_conns ranks=[3,0,4,1,2] 2024-09-16T04:14:25.456 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.449+0000 7f0945484640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f094009fb00 0x7f094009ba50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:25.456 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.449+0000 7f0945484640 10 monclient(hunting): picked mon.noname-a con 0x7f094009fb00 addr v2:172.21.15.42:3300/0 2024-09-16T04:14:25.456 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.449+0000 7f0945484640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f094009bf90 addr v1:172.21.15.42:6789/0 2024-09-16T04:14:25.456 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.449+0000 7f0945484640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f094009c400 0x7f094013b840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:25.456 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.449+0000 7f0945484640 10 monclient(hunting): picked mon.noname-c con 0x7f094009c400 addr v2:172.21.15.154:3300/0 2024-09-16T04:14:25.456 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.449+0000 7f0945484640 1 -- --> v1:172.21.15.42:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f09400a1e20 con 0x7f094009bf90 2024-09-16T04:14:25.456 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.449+0000 7f0945484640 10 monclient(hunting): start opening mon connection 2024-09-16T04:14:25.456 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.449+0000 7f0945484640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f094009e250 con 0x7f094009fb00 2024-09-16T04:14:25.456 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.449+0000 7f0945484640 10 monclient(hunting): start opening mon connection 2024-09-16T04:14:25.456 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.449+0000 7f0945484640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f094009c850 con 0x7f094009c400 2024-09-16T04:14:25.456 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.449+0000 7f0945484640 10 monclient(hunting): _renew_subs 2024-09-16T04:14:25.456 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.449+0000 7f0945484640 10 monclient(hunting): authenticate will time out at 2912.421111s 2024-09-16T04:14:25.456 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.449+0000 7f093effd640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f094009fb00 0x7f094009ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:25.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.449+0000 7f093effd640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f094009fb00 0x7f094009ba50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:48594/0 (socket says 172.21.15.42:48594) 2024-09-16T04:14:25.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.449+0000 7f093effd640 1 -- 172.21.15.42:0/2152426659 learned_addr learned my addr 172.21.15.42:0/2152426659 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:14:25.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.449+0000 7f093effd640 10 monclient(hunting): get_auth_request con 0x7f094009fb00 auth_method 0 2024-09-16T04:14:25.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.449+0000 7f093effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:14:25.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.449+0000 7f093effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:25.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.449+0000 7f093effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:25.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.449+0000 7f093effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:14:25.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.449+0000 7f093effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:14:25.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.449+0000 7f093effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:14:25.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.449+0000 7f093effd640 10 monclient(hunting): handle_auth_done global_id 1104240 payload 274 2024-09-16T04:14:25.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.449+0000 7f093effd640 1 -- 172.21.15.42:0/2152426659 >> v2:172.21.15.154:3300/0 conn(0x7f094009c400 msgr2=0x7f094013b840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:14:25.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.449+0000 7f093effd640 1 --2- 172.21.15.42:0/2152426659 >> v2:172.21.15.154:3300/0 conn(0x7f094009c400 0x7f094013b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:25.457 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.449+0000 7f093effd640 1 -- 172.21.15.42:0/2152426659 >> v1:172.21.15.42:6789/0 conn(0x7f094009bf90 legacy=0x7f0940138110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:25.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.449+0000 7f093effd640 10 monclient: _finish_hunting 0 2024-09-16T04:14:25.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.449+0000 7f093effd640 1 monclient: found mon.noname-a 2024-09-16T04:14:25.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.449+0000 7f093effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:14:25.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.449+0000 7f093effd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:14:25.458 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.449+0000 7f093effd640 1 -- 172.21.15.42:0/2152426659 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f094009ea30 con 0x7f094009fb00 2024-09-16T04:14:25.462 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.449+0000 7f093effd640 1 --2- 172.21.15.42:0/2152426659 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f094009fb00 0x7f094009ba50 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f09400a37b0 tx=0x7f093001eab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a306b9a090f17e4f server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:14:25.463 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.449+0000 7f093e7fc640 1 -- 172.21.15.42:0/2152426659 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09300b4070 con 0x7f094009fb00 2024-09-16T04:14:25.463 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.449+0000 7f093e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:25.463 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.449+0000 7f093e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:14:25.463 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.449+0000 7f093e7fc640 10 monclient: dump: 2024-09-16T04:14:25.463 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:14:25.463 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:14:25.463 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:25.463 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:25.463 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:25.463 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:25.463 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:14:25.463 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:14:25.463 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:14:25.464 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:25.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.453+0000 7f093e7fc640 10 monclient: _finish_auth 0 2024-09-16T04:14:25.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.453+0000 7f093e7fc640 10 monclient: _check_auth_tickets 2024-09-16T04:14:25.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.453+0000 7f093e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:14:25.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.453+0000 7f093e7fc640 1 -- 172.21.15.42:0/2152426659 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f09300a7b00 con 0x7f094009fb00 2024-09-16T04:14:25.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.453+0000 7f093e7fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:14:25.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.453+0000 7f093e7fc640 1 -- 172.21.15.42:0/2152426659 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09300af440 con 0x7f094009fb00 2024-09-16T04:14:25.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.453+0000 7f0945484640 5 monclient: authenticate success, global_id 1104240 2024-09-16T04:14:25.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.453+0000 7f0945484640 10 monclient: get_monmap_and_config success 2024-09-16T04:14:25.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.453+0000 7f0945484640 4 set_mon_vals no callback set 2024-09-16T04:14:25.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.453+0000 7f093e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:25.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.453+0000 7f093e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:14:25.464 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.453+0000 7f093e7fc640 10 monclient: dump: 2024-09-16T04:14:25.465 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:14:25.465 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:14:25.465 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:25.465 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:25.465 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:25.465 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:25.465 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:14:25.465 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:14:25.465 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:14:25.465 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:25.465 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.453+0000 7f0945484640 10 monclient: shutdown 2024-09-16T04:14:25.465 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.453+0000 7f0945484640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:14:25.465 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.453+0000 7f0945484640 1 -- 172.21.15.42:0/2152426659 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f094009fb00 msgr2=0x7f094009ba50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:25.466 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.453+0000 7f0945484640 1 --2- 172.21.15.42:0/2152426659 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f094009fb00 0x7f094009ba50 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f09400a37b0 tx=0x7f093001eab0 comp rx=0 tx=0).stop 2024-09-16T04:14:25.466 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.453+0000 7f0945484640 1 -- 172.21.15.42:0/2152426659 shutdown_connections 2024-09-16T04:14:25.466 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.453+0000 7f0945484640 1 --2- 172.21.15.42:0/2152426659 >> v2:172.21.15.154:3300/0 conn(0x7f094009c400 0x7f094013b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:25.466 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.453+0000 7f0945484640 1 --2- 172.21.15.42:0/2152426659 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f094009fb00 0x7f094009ba50 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:25.466 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.453+0000 7f0945484640 1 -- 172.21.15.42:0/2152426659 >> 172.21.15.42:0/2152426659 conn(0x7f09400a2420 msgr2=0x7f09400a2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:14:25.466 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.453+0000 7f0945484640 1 -- 172.21.15.42:0/2152426659 shutdown_connections 2024-09-16T04:14:25.466 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.453+0000 7f0945484640 1 -- 172.21.15.42:0/2152426659 wait complete. 2024-09-16T04:14:25.466 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.453+0000 7f0945484640 10 monclient: build_initial_monmap 2024-09-16T04:14:25.466 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.453+0000 7f0945484640 10 monclient: monmap: 2024-09-16T04:14:25.466 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:14:25.466 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:14:25.466 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:14:25.466 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:14:25.466 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:14:25.466 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:25.467 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:14:25.467 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:14:25.467 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:14:25.467 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:25.467 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.453+0000 7f0945484640 1 librados: starting msgr at 2024-09-16T04:14:25.467 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.453+0000 7f0945484640 1 librados: starting objecter 2024-09-16T04:14:25.467 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.453+0000 7f0945484640 1 Processor -- start 2024-09-16T04:14:25.467 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.453+0000 7f0945484640 1 -- start start 2024-09-16T04:14:25.467 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.453+0000 7f0945484640 1 librados: setting wanted keys 2024-09-16T04:14:25.467 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.453+0000 7f0945484640 1 librados: calling monclient init 2024-09-16T04:14:25.467 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.453+0000 7f0945484640 10 monclient: init 2024-09-16T04:14:25.467 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.453+0000 7f0945484640 10 monclient: _reopen_session rank -1 2024-09-16T04:14:25.467 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.453+0000 7f0945484640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:14:25.467 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.453+0000 7f0945484640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f094009bf90 0x7f09401cee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:25.467 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.453+0000 7f0945484640 10 monclient(hunting): picked mon.noname-a con 0x7f094009bf90 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:14:25.467 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.453+0000 7f0945484640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f094009c400 0x7f094013a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:25.468 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.453+0000 7f0945484640 10 monclient(hunting): picked mon.noname-c con 0x7f094009c400 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:14:25.468 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.453+0000 7f0945484640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f094009fb00 0x7f0940125330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:25.468 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.453+0000 7f0945484640 10 monclient(hunting): picked mon.noname-b con 0x7f094009fb00 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:14:25.468 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.453+0000 7f0945484640 10 monclient(hunting): start opening mon connection 2024-09-16T04:14:25.468 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.453+0000 7f0945484640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f09400a3170 con 0x7f094009bf90 2024-09-16T04:14:25.468 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.453+0000 7f0945484640 10 monclient(hunting): start opening mon connection 2024-09-16T04:14:25.468 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.453+0000 7f0945484640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f094009e000 con 0x7f094009fb00 2024-09-16T04:14:25.468 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.453+0000 7f0945484640 10 monclient(hunting): start opening mon connection 2024-09-16T04:14:25.468 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.453+0000 7f0945484640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f09400a32f0 con 0x7f094009c400 2024-09-16T04:14:25.468 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.453+0000 7f0945484640 10 monclient(hunting): _renew_subs 2024-09-16T04:14:25.468 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.453+0000 7f0945484640 10 monclient(hunting): authenticate will time out at 2912.426158s 2024-09-16T04:14:25.468 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.453+0000 7f093effd640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f094009bf90 0x7f09401cee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:25.468 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.453+0000 7f093effd640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f094009bf90 0x7f09401cee50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:48606/0 (socket says 172.21.15.42:48606) 2024-09-16T04:14:25.468 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.453+0000 7f093effd640 1 -- 172.21.15.42:0/4049373895 learned_addr learned my addr 172.21.15.42:0/4049373895 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:14:25.469 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.453+0000 7f093effd640 10 monclient(hunting): get_auth_request con 0x7f094009bf90 auth_method 0 2024-09-16T04:14:25.469 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.453+0000 7f093effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:14:25.469 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.453+0000 7f093effd640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:25.469 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.453+0000 7f093effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:25.469 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.453+0000 7f0937fff640 1 --2- 172.21.15.42:0/4049373895 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f094009c400 0x7f094013a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:25.469 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.453+0000 7f0937fff640 10 monclient(hunting): get_auth_request con 0x7f094009c400 auth_method 0 2024-09-16T04:14:25.469 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.453+0000 7f0937fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:14:25.469 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.453+0000 7f0937fff640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:25.469 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.453+0000 7f0937fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:25.469 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.453+0000 7f093effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:14:25.469 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.453+0000 7f093effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:14:25.469 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.453+0000 7f093effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:14:25.469 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.453+0000 7f0937fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:14:25.469 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.453+0000 7f0937fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:14:25.469 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.453+0000 7f0937fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:14:25.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.453+0000 7f093f7fe640 1 --2- 172.21.15.42:0/4049373895 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f094009fb00 0x7f0940125330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:25.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.453+0000 7f093effd640 10 monclient(hunting): handle_auth_done global_id 1104243 payload 824 2024-09-16T04:14:25.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.453+0000 7f093effd640 1 -- 172.21.15.42:0/4049373895 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f094009c400 msgr2=0x7f094013a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:25.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.453+0000 7f093effd640 1 --2- 172.21.15.42:0/4049373895 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f094009c400 0x7f094013a8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:25.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.453+0000 7f093effd640 1 -- 172.21.15.42:0/4049373895 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f094009fb00 msgr2=0x7f0940125330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:25.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.453+0000 7f093effd640 1 --2- 172.21.15.42:0/4049373895 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f094009fb00 0x7f0940125330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:25.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.453+0000 7f093effd640 10 monclient: _finish_hunting 0 2024-09-16T04:14:25.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.453+0000 7f093effd640 1 monclient: found mon.noname-a 2024-09-16T04:14:25.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.453+0000 7f093effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:14:25.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.453+0000 7f093effd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:14:25.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.453+0000 7f093effd640 1 -- 172.21.15.42:0/4049373895 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09401292c0 con 0x7f094009bf90 2024-09-16T04:14:25.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.453+0000 7f093f7fe640 10 monclient: get_auth_request con 0x7f094009fb00 auth_method 0 2024-09-16T04:14:25.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.453+0000 7f093f7fe640 1 --2- 172.21.15.42:0/4049373895 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f094009fb00 0x7f0940125330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T04:14:25.470 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.453+0000 7f093effd640 1 --2- 172.21.15.42:0/4049373895 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f094009bf90 0x7f09401cee50 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f093000b2c0 tx=0x7f093001d2f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:14:25.471 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.453+0000 7f09377fe640 10 client.?.objecter ms_handle_connect 0x7f094009bf90 2024-09-16T04:14:25.471 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.453+0000 7f09377fe640 10 client.?.objecter resend_mon_ops 2024-09-16T04:14:25.471 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.453+0000 7f09377fe640 1 -- 172.21.15.42:0/4049373895 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09300bf070 con 0x7f094009bf90 2024-09-16T04:14:25.471 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.453+0000 7f09377fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:25.471 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.453+0000 7f09377fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:14:25.471 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.453+0000 7f09377fe640 10 monclient: dump: 2024-09-16T04:14:25.471 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:14:25.471 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:14:25.471 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:25.471 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:25.471 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:25.471 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:25.471 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:14:25.471 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:14:25.472 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:14:25.472 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:25.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.453+0000 7f09377fe640 10 monclient: _finish_auth 0 2024-09-16T04:14:25.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.453+0000 7f09377fe640 10 monclient: _check_auth_tickets 2024-09-16T04:14:25.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.453+0000 7f09377fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:14:25.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.453+0000 7f09377fe640 1 -- 172.21.15.42:0/4049373895 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f09300ba640 con 0x7f094009bf90 2024-09-16T04:14:25.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.453+0000 7f09377fe640 10 monclient: handle_config config(0 keys) 2024-09-16T04:14:25.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.453+0000 7f09377fe640 1 -- 172.21.15.42:0/4049373895 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09300b4070 con 0x7f094009bf90 2024-09-16T04:14:25.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.453+0000 7f09377fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:25.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.453+0000 7f09377fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:14:25.472 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.453+0000 7f09377fe640 10 monclient: dump: 2024-09-16T04:14:25.472 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:14:25.472 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:14:25.472 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:25.473 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:25.473 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:25.473 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:25.473 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:14:25.473 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:14:25.473 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:14:25.473 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:25.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.453+0000 7f0945484640 5 monclient: authenticate success, global_id 1104243 2024-09-16T04:14:25.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.453+0000 7f0945484640 10 monclient: _renew_subs 2024-09-16T04:14:25.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.453+0000 7f0945484640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:14:25.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.453+0000 7f093dffb640 4 set_mon_vals no callback set 2024-09-16T04:14:25.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.453+0000 7f0945484640 1 -- 172.21.15.42:0/4049373895 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0940073a30 con 0x7f094009bf90 2024-09-16T04:14:25.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.453+0000 7f0945484640 10 client.1104243.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:14:25.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.453+0000 7f0945484640 10 monclient: _renew_subs 2024-09-16T04:14:25.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.453+0000 7f0945484640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:14:25.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.453+0000 7f0945484640 1 -- 172.21.15.42:0/4049373895 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0940126fd0 con 0x7f094009bf90 2024-09-16T04:14:25.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.453+0000 7f0945484640 1 librados: init done 2024-09-16T04:14:25.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.453+0000 7f09377fe640 1 -- 172.21.15.42:0/4049373895 <== mon.0 v2:172.21.15.42:3300/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f09300b93e0 con 0x7f094009bf90 2024-09-16T04:14:25.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.453+0000 7f0945484640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:14:25.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.453+0000 7f0945484640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:14:25.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.453+0000 7f0945484640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.42:3300/0 2024-09-16T04:14:25.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.453+0000 7f0945484640 1 -- 172.21.15.42:0/4049373895 --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f094009ba50 con 0x7f094009bf90 2024-09-16T04:14:25.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.457+0000 7f09377fe640 1 --2- 172.21.15.42:0/4049373895 >> v2:172.21.15.154:6812/254845342 conn(0x7f0920037840 0x7f0920039d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:25.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.457+0000 7f09377fe640 1 -- 172.21.15.42:0/4049373895 <== mon.0 v2:172.21.15.42:3300/0 5 ==== osd_map(710..710 src has 681..710) ==== 4681+0+0 (secure 0 0 0) 0x7f09300e3560 con 0x7f094009bf90 2024-09-16T04:14:25.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.457+0000 7f09377fe640 10 client.1104243.objecter ms_dispatch 0x7f0940000ba0 osd_map(710..710 src has 681..710) 2024-09-16T04:14:25.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.457+0000 7f09377fe640 3 client.1104243.objecter handle_osd_map got epochs [710,710] > 0 2024-09-16T04:14:25.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.457+0000 7f09377fe640 3 client.1104243.objecter handle_osd_map decoding full epoch 710 2024-09-16T04:14:25.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.457+0000 7f09377fe640 20 client.1104243.objecter dump_active .. 0 homeless 2024-09-16T04:14:25.474 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.457+0000 7f0937fff640 1 --2- 172.21.15.42:0/4049373895 >> v2:172.21.15.154:6812/254845342 conn(0x7f0920037840 0x7f0920039d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:25.475 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.457+0000 7f0937fff640 10 monclient: get_auth_request con 0x7f0920037840 auth_method 0 2024-09-16T04:14:25.475 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.457+0000 7f09377fe640 10 client.1104243.objecter ms_handle_connect 0x7f0920037840 2024-09-16T04:14:25.475 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.457+0000 7f0937fff640 1 --2- 172.21.15.42:0/4049373895 >> v2:172.21.15.154:6812/254845342 conn(0x7f0920037840 0x7f0920039d00 secure :-1 s=READY pgs=4027 cs=0 l=1 rev1=1 crypto rx=0x7f09401294d0 tx=0x7f0928009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:14:25.475 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.461+0000 7f09377fe640 1 -- 172.21.15.42:0/4049373895 <== mon.0 v2:172.21.15.42:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f09300187c0 con 0x7f094009bf90 2024-09-16T04:14:25.475 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.461+0000 7f09377fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:14:25.475 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.461+0000 7f09377fe640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:14:25.626 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.621+0000 7f0945484640 1 -- 172.21.15.42:0/4049373895 --> v2:172.21.15.154:6812/254845342 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f094013b260 con 0x7f0920037840 2024-09-16T04:14:25.634 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.629+0000 7f09377fe640 1 -- 172.21.15.42:0/4049373895 <== mgr.4102 v2:172.21.15.154:6812/254845342 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+114174 (secure 0 0 0) 0x7f094013b260 con 0x7f0920037840 2024-09-16T04:14:25.635 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-09-16T04:14:25.636 INFO:teuthology.orchestra.run.smithi042.stderr:dumped all 2024-09-16T04:14:25.638 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.629+0000 7f0945484640 10 librados: watch_flush enter 2024-09-16T04:14:25.638 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.629+0000 7f0945484640 10 librados: watch_flush exit 2024-09-16T04:14:25.638 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.629+0000 7f0945484640 20 client.1104243.objecter shutdown clearing up homeless session... 2024-09-16T04:14:25.638 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.629+0000 7f0945484640 10 client.1104243.objecter successfully canceled tick 2024-09-16T04:14:25.638 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.633+0000 7f0945484640 1 -- 172.21.15.42:0/4049373895 >> v2:172.21.15.154:6812/254845342 conn(0x7f0920037840 msgr2=0x7f0920039d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:25.638 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.633+0000 7f0945484640 1 --2- 172.21.15.42:0/4049373895 >> v2:172.21.15.154:6812/254845342 conn(0x7f0920037840 0x7f0920039d00 secure :-1 s=READY pgs=4027 cs=0 l=1 rev1=1 crypto rx=0x7f09401294d0 tx=0x7f0928009040 comp rx=0 tx=0).stop 2024-09-16T04:14:25.638 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.633+0000 7f0945484640 10 monclient: shutdown 2024-09-16T04:14:25.638 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.633+0000 7f0945484640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:14:25.638 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.633+0000 7f0945484640 1 -- 172.21.15.42:0/4049373895 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f094009bf90 msgr2=0x7f09401cee50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:25.638 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.633+0000 7f0945484640 1 --2- 172.21.15.42:0/4049373895 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f094009bf90 0x7f09401cee50 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f093000b2c0 tx=0x7f093001d2f0 comp rx=0 tx=0).stop 2024-09-16T04:14:25.638 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.633+0000 7f0945484640 1 -- 172.21.15.42:0/4049373895 shutdown_connections 2024-09-16T04:14:25.638 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.633+0000 7f0945484640 1 --2- 172.21.15.42:0/4049373895 >> v2:172.21.15.154:6812/254845342 conn(0x7f0920037840 0x7f0920039d00 unknown :-1 s=CLOSED pgs=4027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:25.639 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.633+0000 7f0945484640 1 --2- 172.21.15.42:0/4049373895 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f094009fb00 0x7f0940125330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:25.639 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.633+0000 7f0945484640 1 --2- 172.21.15.42:0/4049373895 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f094009c400 0x7f094013a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:25.639 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.633+0000 7f0945484640 1 --2- 172.21.15.42:0/4049373895 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f094009bf90 0x7f09401cee50 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:25.639 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.633+0000 7f0945484640 1 -- 172.21.15.42:0/4049373895 >> 172.21.15.42:0/4049373895 conn(0x7f09400a2420 msgr2=0x7f094006e430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:14:25.639 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.633+0000 7f0945484640 1 -- 172.21.15.42:0/4049373895 shutdown_connections 2024-09-16T04:14:25.639 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.633+0000 7f0945484640 1 -- 172.21.15.42:0/4049373895 wait complete. 2024-09-16T04:14:25.639 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:25.633+0000 7f0945484640 1 librados: shutdown 2024-09-16T04:14:25.655 INFO:teuthology.orchestra.run.smithi042.stdout:{"pg_ready":true,"pg_map":{"version":1655,"stamp":"2024-09-16T04:14:24.232433+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590392,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":122,"num_read_kb":106,"num_write":168,"num_write_kb":2425,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":127,"ondisk_log_size":127,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":431992,"kb_used_data":22680,"kb_used_omap":9,"kb_used_meta":407734,"kb_avail":565799048,"statfs":{"total":579820584960,"available":579378225152,"internally_reserved":0,"allocated":23224320,"data_stored":10083938,"data_compressed":18838,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":9567,"internal_metadata":417520289},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003620"},"pg_stats":[{"pgid":"31.f","version":"0'0","reported_seq":1350,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:53.282605+0000","last_change":"2024-09-16T03:43:34.400720+0000","last_active":"2024-09-16T04:13:53.282605+0000","last_peered":"2024-09-16T04:13:53.282605+0000","last_clean":"2024-09-16T04:13:53.282605+0000","last_became_active":"2024-09-16T03:43:34.399531+0000","last_became_peered":"2024-09-16T03:43:34.399531+0000","last_unstale":"2024-09-16T04:13:53.282605+0000","last_undegraded":"2024-09-16T04:13:53.282605+0000","last_fullsized":"2024-09-16T04:13:53.282605+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_clean_scrub_stamp":"2024-09-16T03:43:33.221564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:29:59.415591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.e","version":"0'0","reported_seq":1351,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:53.356452+0000","last_change":"2024-09-16T03:43:34.492536+0000","last_active":"2024-09-16T04:13:53.356452+0000","last_peered":"2024-09-16T04:13:53.356452+0000","last_clean":"2024-09-16T04:13:53.356452+0000","last_became_active":"2024-09-16T03:43:34.492249+0000","last_became_peered":"2024-09-16T03:43:34.492249+0000","last_unstale":"2024-09-16T04:13:53.356452+0000","last_undegraded":"2024-09-16T04:13:53.356452+0000","last_fullsized":"2024-09-16T04:13:53.356452+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_clean_scrub_stamp":"2024-09-16T03:43:33.221564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:15:41.168066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.d","version":"0'0","reported_seq":1373,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:51.310003+0000","last_change":"2024-09-16T03:43:34.414280+0000","last_active":"2024-09-16T04:13:51.310003+0000","last_peered":"2024-09-16T04:13:51.310003+0000","last_clean":"2024-09-16T04:13:51.310003+0000","last_became_active":"2024-09-16T03:43:34.399055+0000","last_became_peered":"2024-09-16T03:43:34.399055+0000","last_unstale":"2024-09-16T04:13:51.310003+0000","last_undegraded":"2024-09-16T04:13:51.310003+0000","last_fullsized":"2024-09-16T04:13:51.310003+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_clean_scrub_stamp":"2024-09-16T03:43:33.221564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:08:19.292854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.c","version":"0'0","reported_seq":1353,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:52.857977+0000","last_change":"2024-09-16T03:43:34.403649+0000","last_active":"2024-09-16T04:13:52.857977+0000","last_peered":"2024-09-16T04:13:52.857977+0000","last_clean":"2024-09-16T04:13:52.857977+0000","last_became_active":"2024-09-16T03:43:34.403187+0000","last_became_peered":"2024-09-16T03:43:34.403187+0000","last_unstale":"2024-09-16T04:13:52.857977+0000","last_undegraded":"2024-09-16T04:13:52.857977+0000","last_fullsized":"2024-09-16T04:13:52.857977+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_clean_scrub_stamp":"2024-09-16T03:43:33.221564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:21:54.130294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.13","version":"0'0","reported_seq":1354,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:51.300256+0000","last_change":"2024-09-16T03:43:34.382675+0000","last_active":"2024-09-16T04:13:51.300256+0000","last_peered":"2024-09-16T04:13:51.300256+0000","last_clean":"2024-09-16T04:13:51.300256+0000","last_became_active":"2024-09-16T03:43:34.382422+0000","last_became_peered":"2024-09-16T03:43:34.382422+0000","last_unstale":"2024-09-16T04:13:51.300256+0000","last_undegraded":"2024-09-16T04:13:51.300256+0000","last_fullsized":"2024-09-16T04:13:51.300256+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_clean_scrub_stamp":"2024-09-16T03:43:33.221564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:52:35.625157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.12","version":"0'0","reported_seq":1353,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:52.857683+0000","last_change":"2024-09-16T03:43:34.404613+0000","last_active":"2024-09-16T04:13:52.857683+0000","last_peered":"2024-09-16T04:13:52.857683+0000","last_clean":"2024-09-16T04:13:52.857683+0000","last_became_active":"2024-09-16T03:43:34.404299+0000","last_became_peered":"2024-09-16T03:43:34.404299+0000","last_unstale":"2024-09-16T04:13:52.857683+0000","last_undegraded":"2024-09-16T04:13:52.857683+0000","last_fullsized":"2024-09-16T04:13:52.857683+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_clean_scrub_stamp":"2024-09-16T03:43:33.221564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:25:52.933411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.11","version":"0'0","reported_seq":1354,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:51.299816+0000","last_change":"2024-09-16T03:43:34.355045+0000","last_active":"2024-09-16T04:13:51.299816+0000","last_peered":"2024-09-16T04:13:51.299816+0000","last_clean":"2024-09-16T04:13:51.299816+0000","last_became_active":"2024-09-16T03:43:34.354598+0000","last_became_peered":"2024-09-16T03:43:34.354598+0000","last_unstale":"2024-09-16T04:13:51.299816+0000","last_undegraded":"2024-09-16T04:13:51.299816+0000","last_fullsized":"2024-09-16T04:13:51.299816+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_clean_scrub_stamp":"2024-09-16T03:43:33.221564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:40:14.558362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.10","version":"0'0","reported_seq":1351,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:53.353474+0000","last_change":"2024-09-16T03:43:34.436288+0000","last_active":"2024-09-16T04:13:53.353474+0000","last_peered":"2024-09-16T04:13:53.353474+0000","last_clean":"2024-09-16T04:13:53.353474+0000","last_became_active":"2024-09-16T03:43:34.434548+0000","last_became_peered":"2024-09-16T03:43:34.434548+0000","last_unstale":"2024-09-16T04:13:53.353474+0000","last_undegraded":"2024-09-16T04:13:53.353474+0000","last_fullsized":"2024-09-16T04:13:53.353474+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_clean_scrub_stamp":"2024-09-16T03:43:33.221564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:03:15.848714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.17","version":"0'0","reported_seq":1349,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:53.282881+0000","last_change":"2024-09-16T03:43:34.456644+0000","last_active":"2024-09-16T04:13:53.282881+0000","last_peered":"2024-09-16T04:13:53.282881+0000","last_clean":"2024-09-16T04:13:53.282881+0000","last_became_active":"2024-09-16T03:43:34.454859+0000","last_became_peered":"2024-09-16T03:43:34.454859+0000","last_unstale":"2024-09-16T04:13:53.282881+0000","last_undegraded":"2024-09-16T04:13:53.282881+0000","last_fullsized":"2024-09-16T04:13:53.282881+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_clean_scrub_stamp":"2024-09-16T03:43:33.221564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:45:01.348488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.16","version":"0'0","reported_seq":1347,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:53.166394+0000","last_change":"2024-09-16T03:43:34.391035+0000","last_active":"2024-09-16T04:13:53.166394+0000","last_peered":"2024-09-16T04:13:53.166394+0000","last_clean":"2024-09-16T04:13:53.166394+0000","last_became_active":"2024-09-16T03:43:34.387629+0000","last_became_peered":"2024-09-16T03:43:34.387629+0000","last_unstale":"2024-09-16T04:13:53.166394+0000","last_undegraded":"2024-09-16T04:13:53.166394+0000","last_fullsized":"2024-09-16T04:13:53.166394+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_clean_scrub_stamp":"2024-09-16T03:43:33.221564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:11:59.627229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.15","version":"0'0","reported_seq":1352,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:52.859996+0000","last_change":"2024-09-16T03:43:34.387904+0000","last_active":"2024-09-16T04:13:52.859996+0000","last_peered":"2024-09-16T04:13:52.859996+0000","last_clean":"2024-09-16T04:13:52.859996+0000","last_became_active":"2024-09-16T03:43:34.387630+0000","last_became_peered":"2024-09-16T03:43:34.387630+0000","last_unstale":"2024-09-16T04:13:52.859996+0000","last_undegraded":"2024-09-16T04:13:52.859996+0000","last_fullsized":"2024-09-16T04:13:52.859996+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_clean_scrub_stamp":"2024-09-16T03:43:33.221564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:15:37.951806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.14","version":"0'0","reported_seq":1350,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:53.282153+0000","last_change":"2024-09-16T03:43:34.381624+0000","last_active":"2024-09-16T04:13:53.282153+0000","last_peered":"2024-09-16T04:13:53.282153+0000","last_clean":"2024-09-16T04:13:53.282153+0000","last_became_active":"2024-09-16T03:43:34.378385+0000","last_became_peered":"2024-09-16T03:43:34.378385+0000","last_unstale":"2024-09-16T04:13:53.282153+0000","last_undegraded":"2024-09-16T04:13:53.282153+0000","last_fullsized":"2024-09-16T04:13:53.282153+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_clean_scrub_stamp":"2024-09-16T03:43:33.221564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:20:14.916099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"14'117","reported_seq":1523,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:52.858864+0000","last_change":"2024-09-16T03:42:18.799294+0000","last_active":"2024-09-16T04:13:52.858864+0000","last_peered":"2024-09-16T04:13:52.858864+0000","last_clean":"2024-09-16T04:13:52.858864+0000","last_became_active":"2024-09-16T03:42:18.797239+0000","last_became_peered":"2024-09-16T03:42:18.797239+0000","last_unstale":"2024-09-16T04:13:52.858864+0000","last_undegraded":"2024-09-16T04:13:52.858864+0000","last_fullsized":"2024-09-16T04:13:52.858864+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:42:17.544237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:42:17.544237+0000","last_clean_scrub_stamp":"2024-09-16T03:42:17.544237+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:01:21.017593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":1374,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:51.300417+0000","last_change":"2024-09-16T03:42:27.630965+0000","last_active":"2024-09-16T04:13:51.300417+0000","last_peered":"2024-09-16T04:13:51.300417+0000","last_clean":"2024-09-16T04:13:51.300417+0000","last_became_active":"2024-09-16T03:42:25.615930+0000","last_became_peered":"2024-09-16T03:42:25.615930+0000","last_unstale":"2024-09-16T04:13:51.300417+0000","last_undegraded":"2024-09-16T04:13:51.300417+0000","last_fullsized":"2024-09-16T04:13:51.300417+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:42:24.583922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:42:24.583922+0000","last_clean_scrub_stamp":"2024-09-16T03:42:24.583922+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:59:59.653118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00099470900000000009,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.1e","version":"0'0","reported_seq":1355,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:51.302860+0000","last_change":"2024-09-16T03:43:34.401653+0000","last_active":"2024-09-16T04:13:51.302860+0000","last_peered":"2024-09-16T04:13:51.302860+0000","last_clean":"2024-09-16T04:13:51.302860+0000","last_became_active":"2024-09-16T03:43:34.397708+0000","last_became_peered":"2024-09-16T03:43:34.397708+0000","last_unstale":"2024-09-16T04:13:51.302860+0000","last_undegraded":"2024-09-16T04:13:51.302860+0000","last_fullsized":"2024-09-16T04:13:51.302860+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_clean_scrub_stamp":"2024-09-16T03:43:33.221564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:16:17.302952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1372,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:52.859154+0000","last_change":"2024-09-16T03:42:27.624504+0000","last_active":"2024-09-16T04:13:52.859154+0000","last_peered":"2024-09-16T04:13:52.859154+0000","last_clean":"2024-09-16T04:13:52.859154+0000","last_became_active":"2024-09-16T03:42:25.612386+0000","last_became_peered":"2024-09-16T03:42:25.612386+0000","last_unstale":"2024-09-16T04:13:52.859154+0000","last_undegraded":"2024-09-16T04:13:52.859154+0000","last_fullsized":"2024-09-16T04:13:52.859154+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:42:24.583922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:42:24.583922+0000","last_clean_scrub_stamp":"2024-09-16T03:42:24.583922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:00:49.446880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081399800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1d","version":"0'0","reported_seq":1372,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:51.310252+0000","last_change":"2024-09-16T03:43:34.408908+0000","last_active":"2024-09-16T04:13:51.310252+0000","last_peered":"2024-09-16T04:13:51.310252+0000","last_clean":"2024-09-16T04:13:51.310252+0000","last_became_active":"2024-09-16T03:43:34.399300+0000","last_became_peered":"2024-09-16T03:43:34.399300+0000","last_unstale":"2024-09-16T04:13:51.310252+0000","last_undegraded":"2024-09-16T04:13:51.310252+0000","last_fullsized":"2024-09-16T04:13:51.310252+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_clean_scrub_stamp":"2024-09-16T03:43:33.221564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:25:51.517103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1370,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:53.353598+0000","last_change":"2024-09-16T03:42:27.619252+0000","last_active":"2024-09-16T04:13:53.353598+0000","last_peered":"2024-09-16T04:13:53.353598+0000","last_clean":"2024-09-16T04:13:53.353598+0000","last_became_active":"2024-09-16T03:42:25.607533+0000","last_became_peered":"2024-09-16T03:42:25.607533+0000","last_unstale":"2024-09-16T04:13:53.353598+0000","last_undegraded":"2024-09-16T04:13:53.353598+0000","last_fullsized":"2024-09-16T04:13:53.353598+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:42:24.583922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:42:24.583922+0000","last_clean_scrub_stamp":"2024-09-16T03:42:24.583922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:25:58.536144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089083299999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1c","version":"0'0","reported_seq":1349,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:53.282306+0000","last_change":"2024-09-16T03:43:34.421587+0000","last_active":"2024-09-16T04:13:53.282306+0000","last_peered":"2024-09-16T04:13:53.282306+0000","last_clean":"2024-09-16T04:13:53.282306+0000","last_became_active":"2024-09-16T03:43:34.421320+0000","last_became_peered":"2024-09-16T03:43:34.421320+0000","last_unstale":"2024-09-16T04:13:53.282306+0000","last_undegraded":"2024-09-16T04:13:53.282306+0000","last_fullsized":"2024-09-16T04:13:53.282306+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_clean_scrub_stamp":"2024-09-16T03:43:33.221564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:47:25.449200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":1375,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:51.299718+0000","last_change":"2024-09-16T03:42:27.632050+0000","last_active":"2024-09-16T04:13:51.299718+0000","last_peered":"2024-09-16T04:13:51.299718+0000","last_clean":"2024-09-16T04:13:51.299718+0000","last_became_active":"2024-09-16T03:42:25.617896+0000","last_became_peered":"2024-09-16T03:42:25.617896+0000","last_unstale":"2024-09-16T04:13:51.299718+0000","last_undegraded":"2024-09-16T04:13:51.299718+0000","last_fullsized":"2024-09-16T04:13:51.299718+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:42:24.583922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:42:24.583922+0000","last_clean_scrub_stamp":"2024-09-16T03:42:24.583922+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:41:08.754274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021072529999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.1f","version":"0'0","reported_seq":1352,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:53.356655+0000","last_change":"2024-09-16T03:43:34.419004+0000","last_active":"2024-09-16T04:13:53.356655+0000","last_peered":"2024-09-16T04:13:53.356655+0000","last_clean":"2024-09-16T04:13:53.356655+0000","last_became_active":"2024-09-16T03:43:34.415747+0000","last_became_peered":"2024-09-16T03:43:34.415747+0000","last_unstale":"2024-09-16T04:13:53.356655+0000","last_undegraded":"2024-09-16T04:13:53.356655+0000","last_fullsized":"2024-09-16T04:13:53.356655+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_clean_scrub_stamp":"2024-09-16T03:43:33.221564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:01.644098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1367,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:53.166325+0000","last_change":"2024-09-16T03:42:27.631260+0000","last_active":"2024-09-16T04:13:53.166325+0000","last_peered":"2024-09-16T04:13:53.166325+0000","last_clean":"2024-09-16T04:13:53.166325+0000","last_became_active":"2024-09-16T03:42:25.609631+0000","last_became_peered":"2024-09-16T03:42:25.609631+0000","last_unstale":"2024-09-16T04:13:53.166325+0000","last_undegraded":"2024-09-16T04:13:53.166325+0000","last_fullsized":"2024-09-16T04:13:53.166325+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:42:24.583922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:42:24.583922+0000","last_clean_scrub_stamp":"2024-09-16T03:42:24.583922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:58:49.091085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001288897,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.19","version":"0'0","reported_seq":1347,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:53.166637+0000","last_change":"2024-09-16T03:43:34.341660+0000","last_active":"2024-09-16T04:13:53.166637+0000","last_peered":"2024-09-16T04:13:53.166637+0000","last_clean":"2024-09-16T04:13:53.166637+0000","last_became_active":"2024-09-16T03:43:34.332234+0000","last_became_peered":"2024-09-16T03:43:34.332234+0000","last_unstale":"2024-09-16T04:13:53.166637+0000","last_undegraded":"2024-09-16T04:13:53.166637+0000","last_fullsized":"2024-09-16T04:13:53.166637+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_clean_scrub_stamp":"2024-09-16T03:43:33.221564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:20:27.326212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1372,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:52.859726+0000","last_change":"2024-09-16T03:42:27.625091+0000","last_active":"2024-09-16T04:13:52.859726+0000","last_peered":"2024-09-16T04:13:52.859726+0000","last_clean":"2024-09-16T04:13:52.859726+0000","last_became_active":"2024-09-16T03:42:25.614458+0000","last_became_peered":"2024-09-16T03:42:25.614458+0000","last_unstale":"2024-09-16T04:13:52.859726+0000","last_undegraded":"2024-09-16T04:13:52.859726+0000","last_fullsized":"2024-09-16T04:13:52.859726+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:42:24.583922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:42:24.583922+0000","last_clean_scrub_stamp":"2024-09-16T03:42:24.583922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:26:14.681436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012701769999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.18","version":"0'0","reported_seq":1347,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:53.165619+0000","last_change":"2024-09-16T03:43:34.362758+0000","last_active":"2024-09-16T04:13:53.165619+0000","last_peered":"2024-09-16T04:13:53.165619+0000","last_clean":"2024-09-16T04:13:53.165619+0000","last_became_active":"2024-09-16T03:43:34.354705+0000","last_became_peered":"2024-09-16T03:43:34.354705+0000","last_unstale":"2024-09-16T04:13:53.165619+0000","last_undegraded":"2024-09-16T04:13:53.165619+0000","last_fullsized":"2024-09-16T04:13:53.165619+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_clean_scrub_stamp":"2024-09-16T03:43:33.221564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:16:36.345303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1366,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:53.166473+0000","last_change":"2024-09-16T03:42:27.631227+0000","last_active":"2024-09-16T04:13:53.166473+0000","last_peered":"2024-09-16T04:13:53.166473+0000","last_clean":"2024-09-16T04:13:53.166473+0000","last_became_active":"2024-09-16T03:42:25.608453+0000","last_became_peered":"2024-09-16T03:42:25.608453+0000","last_unstale":"2024-09-16T04:13:53.166473+0000","last_undegraded":"2024-09-16T04:13:53.166473+0000","last_fullsized":"2024-09-16T04:13:53.166473+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:42:24.583922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:42:24.583922+0000","last_clean_scrub_stamp":"2024-09-16T03:42:24.583922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:00:11.921493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001163533,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.1b","version":"0'0","reported_seq":1352,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:52.859563+0000","last_change":"2024-09-16T03:43:34.381629+0000","last_active":"2024-09-16T04:13:52.859563+0000","last_peered":"2024-09-16T04:13:52.859563+0000","last_clean":"2024-09-16T04:13:52.859563+0000","last_became_active":"2024-09-16T03:43:34.381335+0000","last_became_peered":"2024-09-16T03:43:34.381335+0000","last_unstale":"2024-09-16T04:13:52.859563+0000","last_undegraded":"2024-09-16T04:13:52.859563+0000","last_fullsized":"2024-09-16T04:13:52.859563+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_clean_scrub_stamp":"2024-09-16T03:43:33.221564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:30:50.930529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1371,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:52.861031+0000","last_change":"2024-09-16T03:42:27.624922+0000","last_active":"2024-09-16T04:13:52.861031+0000","last_peered":"2024-09-16T04:13:52.861031+0000","last_clean":"2024-09-16T04:13:52.861031+0000","last_became_active":"2024-09-16T03:42:25.615293+0000","last_became_peered":"2024-09-16T03:42:25.615293+0000","last_unstale":"2024-09-16T04:13:52.861031+0000","last_undegraded":"2024-09-16T04:13:52.861031+0000","last_fullsized":"2024-09-16T04:13:52.861031+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:42:24.583922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:42:24.583922+0000","last_clean_scrub_stamp":"2024-09-16T03:42:24.583922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:46:11.677251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010681690000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1a","version":"0'0","reported_seq":1349,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:53.281860+0000","last_change":"2024-09-16T03:43:34.382293+0000","last_active":"2024-09-16T04:13:53.281860+0000","last_peered":"2024-09-16T04:13:53.281860+0000","last_clean":"2024-09-16T04:13:53.281860+0000","last_became_active":"2024-09-16T03:43:34.372307+0000","last_became_peered":"2024-09-16T03:43:34.372307+0000","last_unstale":"2024-09-16T04:13:53.281860+0000","last_undegraded":"2024-09-16T04:13:53.281860+0000","last_fullsized":"2024-09-16T04:13:53.281860+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_clean_scrub_stamp":"2024-09-16T03:43:33.221564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:49:25.481308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.8","version":"0'0","reported_seq":1351,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:53.280246+0000","last_change":"2024-09-16T03:43:34.386011+0000","last_active":"2024-09-16T04:13:53.280246+0000","last_peered":"2024-09-16T04:13:53.280246+0000","last_clean":"2024-09-16T04:13:53.280246+0000","last_became_active":"2024-09-16T03:43:34.378483+0000","last_became_peered":"2024-09-16T03:43:34.378483+0000","last_unstale":"2024-09-16T04:13:53.280246+0000","last_undegraded":"2024-09-16T04:13:53.280246+0000","last_fullsized":"2024-09-16T04:13:53.280246+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_clean_scrub_stamp":"2024-09-16T03:43:33.221564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:25:42.568633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.9","version":"0'0","reported_seq":1374,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:51.306525+0000","last_change":"2024-09-16T03:43:34.389268+0000","last_active":"2024-09-16T04:13:51.306525+0000","last_peered":"2024-09-16T04:13:51.306525+0000","last_clean":"2024-09-16T04:13:51.306525+0000","last_became_active":"2024-09-16T03:43:34.378733+0000","last_became_peered":"2024-09-16T03:43:34.378733+0000","last_unstale":"2024-09-16T04:13:51.306525+0000","last_undegraded":"2024-09-16T04:13:51.306525+0000","last_fullsized":"2024-09-16T04:13:51.306525+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_clean_scrub_stamp":"2024-09-16T03:43:33.221564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:16:08.231273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.a","version":"0'0","reported_seq":1374,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:51.306820+0000","last_change":"2024-09-16T03:43:34.390824+0000","last_active":"2024-09-16T04:13:51.306820+0000","last_peered":"2024-09-16T04:13:51.306820+0000","last_clean":"2024-09-16T04:13:51.306820+0000","last_became_active":"2024-09-16T03:43:34.390049+0000","last_became_peered":"2024-09-16T03:43:34.390049+0000","last_unstale":"2024-09-16T04:13:51.306820+0000","last_undegraded":"2024-09-16T04:13:51.306820+0000","last_fullsized":"2024-09-16T04:13:51.306820+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_clean_scrub_stamp":"2024-09-16T03:43:33.221564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:48:31.039881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.b","version":"0'0","reported_seq":1348,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:53.165918+0000","last_change":"2024-09-16T03:43:34.332247+0000","last_active":"2024-09-16T04:13:53.165918+0000","last_peered":"2024-09-16T04:13:53.165918+0000","last_clean":"2024-09-16T04:13:53.165918+0000","last_became_active":"2024-09-16T03:43:34.328425+0000","last_became_peered":"2024-09-16T03:43:34.328425+0000","last_unstale":"2024-09-16T04:13:53.165918+0000","last_undegraded":"2024-09-16T04:13:53.165918+0000","last_fullsized":"2024-09-16T04:13:53.165918+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_clean_scrub_stamp":"2024-09-16T03:43:33.221564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:38:08.042571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.4","version":"0'0","reported_seq":1351,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:53.282653+0000","last_change":"2024-09-16T03:43:34.392184+0000","last_active":"2024-09-16T04:13:53.282653+0000","last_peered":"2024-09-16T04:13:53.282653+0000","last_clean":"2024-09-16T04:13:53.282653+0000","last_became_active":"2024-09-16T03:43:34.391868+0000","last_became_peered":"2024-09-16T03:43:34.391868+0000","last_unstale":"2024-09-16T04:13:53.282653+0000","last_undegraded":"2024-09-16T04:13:53.282653+0000","last_fullsized":"2024-09-16T04:13:53.282653+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_clean_scrub_stamp":"2024-09-16T03:43:33.221564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:02:40.567145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.5","version":"0'0","reported_seq":1351,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:53.280846+0000","last_change":"2024-09-16T03:43:34.419668+0000","last_active":"2024-09-16T04:13:53.280846+0000","last_peered":"2024-09-16T04:13:53.280846+0000","last_clean":"2024-09-16T04:13:53.280846+0000","last_became_active":"2024-09-16T03:43:34.419410+0000","last_became_peered":"2024-09-16T03:43:34.419410+0000","last_unstale":"2024-09-16T04:13:53.280846+0000","last_undegraded":"2024-09-16T04:13:53.280846+0000","last_fullsized":"2024-09-16T04:13:53.280846+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_clean_scrub_stamp":"2024-09-16T03:43:33.221564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:37:43.007953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.6","version":"0'0","reported_seq":1374,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:51.309271+0000","last_change":"2024-09-16T03:43:34.386607+0000","last_active":"2024-09-16T04:13:51.309271+0000","last_peered":"2024-09-16T04:13:51.309271+0000","last_clean":"2024-09-16T04:13:51.309271+0000","last_became_active":"2024-09-16T03:43:34.381834+0000","last_became_peered":"2024-09-16T03:43:34.381834+0000","last_unstale":"2024-09-16T04:13:51.309271+0000","last_undegraded":"2024-09-16T04:13:51.309271+0000","last_fullsized":"2024-09-16T04:13:51.309271+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_clean_scrub_stamp":"2024-09-16T03:43:33.221564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:06:41.696398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.7","version":"0'0","reported_seq":1349,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:53.164974+0000","last_change":"2024-09-16T03:43:34.296793+0000","last_active":"2024-09-16T04:13:53.164974+0000","last_peered":"2024-09-16T04:13:53.164974+0000","last_clean":"2024-09-16T04:13:53.164974+0000","last_became_active":"2024-09-16T03:43:34.292756+0000","last_became_peered":"2024-09-16T03:43:34.292756+0000","last_unstale":"2024-09-16T04:13:53.164974+0000","last_undegraded":"2024-09-16T04:13:53.164974+0000","last_fullsized":"2024-09-16T04:13:53.164974+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_clean_scrub_stamp":"2024-09-16T03:43:33.221564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:12:04.533479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.0","version":"0'0","reported_seq":1355,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:52.859564+0000","last_change":"2024-09-16T03:43:34.352792+0000","last_active":"2024-09-16T04:13:52.859564+0000","last_peered":"2024-09-16T04:13:52.859564+0000","last_clean":"2024-09-16T04:13:52.859564+0000","last_became_active":"2024-09-16T03:43:34.351273+0000","last_became_peered":"2024-09-16T03:43:34.351273+0000","last_unstale":"2024-09-16T04:13:52.859564+0000","last_undegraded":"2024-09-16T04:13:52.859564+0000","last_fullsized":"2024-09-16T04:13:52.859564+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_clean_scrub_stamp":"2024-09-16T03:43:33.221564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:50:21.469378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1","version":"22'7","reported_seq":1362,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:53.164484+0000","last_change":"2024-09-16T03:43:34.397561+0000","last_active":"2024-09-16T04:13:53.164484+0000","last_peered":"2024-09-16T04:13:53.164484+0000","last_clean":"2024-09-16T04:13:53.164484+0000","last_became_active":"2024-09-16T03:43:34.382744+0000","last_became_peered":"2024-09-16T03:43:34.382744+0000","last_unstale":"2024-09-16T04:13:53.164484+0000","last_undegraded":"2024-09-16T04:13:53.164484+0000","last_fullsized":"2024-09-16T04:13:53.164484+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_clean_scrub_stamp":"2024-09-16T03:43:33.221564+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:54:00.268175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.2","version":"0'0","reported_seq":1355,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:51.300709+0000","last_change":"2024-09-16T03:43:34.402175+0000","last_active":"2024-09-16T04:13:51.300709+0000","last_peered":"2024-09-16T04:13:51.300709+0000","last_clean":"2024-09-16T04:13:51.300709+0000","last_became_active":"2024-09-16T03:43:34.402014+0000","last_became_peered":"2024-09-16T03:43:34.402014+0000","last_unstale":"2024-09-16T04:13:51.300709+0000","last_undegraded":"2024-09-16T04:13:51.300709+0000","last_fullsized":"2024-09-16T04:13:51.300709+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_clean_scrub_stamp":"2024-09-16T03:43:33.221564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:46:39.366181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.3","version":"0'0","reported_seq":1374,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:13:51.306977+0000","last_change":"2024-09-16T03:43:34.380947+0000","last_active":"2024-09-16T04:13:51.306977+0000","last_peered":"2024-09-16T04:13:51.306977+0000","last_clean":"2024-09-16T04:13:51.306977+0000","last_became_active":"2024-09-16T03:43:34.379988+0000","last_became_peered":"2024-09-16T03:43:34.379988+0000","last_unstale":"2024-09-16T04:13:51.306977+0000","last_undegraded":"2024-09-16T04:13:51.306977+0000","last_fullsized":"2024-09-16T04:13:51.306977+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T03:43:33.221564+0000","last_clean_scrub_stamp":"2024-09-16T03:43:33.221564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:21:15.646033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":31,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":18838,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":12,"seq":51539607944,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":71596,"kb_used_data":3764,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94300244,"statfs":{"total":96636764160,"available":96563449856,"internally_reserved":0,"allocated":3854336,"data_stored":1504358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":11,"seq":47244640647,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":71228,"kb_used_data":3764,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94300612,"statfs":{"total":96636764160,"available":96563826688,"internally_reserved":0,"allocated":3854336,"data_stored":1504344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640644,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":67116,"kb_used_data":3812,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94304724,"statfs":{"total":96636764160,"available":96568037376,"internally_reserved":0,"allocated":3903488,"data_stored":2033267,"data_compressed":9419,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1593,"internal_metadata":64616903},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640645,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":71224,"kb_used_data":3760,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94300616,"statfs":{"total":96636764160,"available":96563830784,"internally_reserved":0,"allocated":3850240,"data_stored":1504339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":68876742},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640646,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":79552,"kb_used_data":3768,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94292288,"statfs":{"total":96636764160,"available":96555302912,"internally_reserved":0,"allocated":3858432,"data_stored":1504363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640645,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":71276,"kb_used_data":3812,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94300564,"statfs":{"total":96636764160,"available":96563777536,"internally_reserved":0,"allocated":3903488,"data_stored":2033267,"data_compressed":9419,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":9419,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":9419,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:14:25.660 INFO:tasks.ceph:pgid 31.f last_scrub_stamp 2024-09-16T03:43:33.221564+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=43, tm_sec=33, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=14, tm_sec=21, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T04:14:25.660 INFO:tasks.ceph:pgid 31.e last_scrub_stamp 2024-09-16T03:43:33.221564+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=43, tm_sec=33, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=14, tm_sec=21, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T04:14:25.661 INFO:tasks.ceph:pgid 31.d last_scrub_stamp 2024-09-16T03:43:33.221564+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=43, tm_sec=33, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=14, tm_sec=21, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T04:14:25.661 INFO:tasks.ceph:pgid 31.c last_scrub_stamp 2024-09-16T03:43:33.221564+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=43, tm_sec=33, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=14, tm_sec=21, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T04:14:25.661 INFO:tasks.ceph:pgid 31.13 last_scrub_stamp 2024-09-16T03:43:33.221564+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=43, tm_sec=33, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=14, tm_sec=21, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T04:14:25.665 INFO:tasks.ceph:pgid 31.12 last_scrub_stamp 2024-09-16T03:43:33.221564+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=43, tm_sec=33, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=14, tm_sec=21, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T04:14:25.665 INFO:tasks.ceph:pgid 31.11 last_scrub_stamp 2024-09-16T03:43:33.221564+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=43, tm_sec=33, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=14, tm_sec=21, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T04:14:25.665 INFO:tasks.ceph:pgid 31.10 last_scrub_stamp 2024-09-16T03:43:33.221564+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=43, tm_sec=33, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=14, tm_sec=21, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T04:14:25.665 INFO:tasks.ceph:pgid 31.17 last_scrub_stamp 2024-09-16T03:43:33.221564+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=43, tm_sec=33, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=14, tm_sec=21, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T04:14:25.665 INFO:tasks.ceph:pgid 31.16 last_scrub_stamp 2024-09-16T03:43:33.221564+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=43, tm_sec=33, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=14, tm_sec=21, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T04:14:25.665 INFO:tasks.ceph:pgid 31.15 last_scrub_stamp 2024-09-16T03:43:33.221564+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=43, tm_sec=33, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=14, tm_sec=21, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T04:14:25.665 INFO:tasks.ceph:pgid 31.14 last_scrub_stamp 2024-09-16T03:43:33.221564+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=43, tm_sec=33, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=14, tm_sec=21, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T04:14:25.666 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-09-16T03:42:17.544237+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=42, tm_sec=17, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=14, tm_sec=21, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T04:14:25.666 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-09-16T03:42:24.583922+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=42, tm_sec=24, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=14, tm_sec=21, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T04:14:25.666 INFO:tasks.ceph:pgid 31.1e last_scrub_stamp 2024-09-16T03:43:33.221564+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=43, tm_sec=33, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=14, tm_sec=21, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T04:14:25.666 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-09-16T03:42:24.583922+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=42, tm_sec=24, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=14, tm_sec=21, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T04:14:25.666 INFO:tasks.ceph:pgid 31.1d last_scrub_stamp 2024-09-16T03:43:33.221564+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=43, tm_sec=33, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=14, tm_sec=21, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T04:14:25.666 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-09-16T03:42:24.583922+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=42, tm_sec=24, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=14, tm_sec=21, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T04:14:25.666 INFO:tasks.ceph:pgid 31.1c last_scrub_stamp 2024-09-16T03:43:33.221564+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=43, tm_sec=33, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=14, tm_sec=21, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T04:14:25.666 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-09-16T03:42:24.583922+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=42, tm_sec=24, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=14, tm_sec=21, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T04:14:25.666 INFO:tasks.ceph:pgid 31.1f last_scrub_stamp 2024-09-16T03:43:33.221564+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=43, tm_sec=33, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=14, tm_sec=21, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T04:14:25.667 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-09-16T03:42:24.583922+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=42, tm_sec=24, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=14, tm_sec=21, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T04:14:25.667 INFO:tasks.ceph:pgid 31.19 last_scrub_stamp 2024-09-16T03:43:33.221564+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=43, tm_sec=33, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=14, tm_sec=21, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T04:14:25.667 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-09-16T03:42:24.583922+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=42, tm_sec=24, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=14, tm_sec=21, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T04:14:25.667 INFO:tasks.ceph:pgid 31.18 last_scrub_stamp 2024-09-16T03:43:33.221564+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=43, tm_sec=33, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=14, tm_sec=21, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T04:14:25.667 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-09-16T03:42:24.583922+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=42, tm_sec=24, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=14, tm_sec=21, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T04:14:25.667 INFO:tasks.ceph:pgid 31.1b last_scrub_stamp 2024-09-16T03:43:33.221564+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=43, tm_sec=33, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=14, tm_sec=21, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T04:14:25.667 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-09-16T03:42:24.583922+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=42, tm_sec=24, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=14, tm_sec=21, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T04:14:25.667 INFO:tasks.ceph:pgid 31.1a last_scrub_stamp 2024-09-16T03:43:33.221564+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=43, tm_sec=33, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=14, tm_sec=21, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T04:14:25.667 INFO:tasks.ceph:pgid 31.8 last_scrub_stamp 2024-09-16T03:43:33.221564+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=43, tm_sec=33, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=14, tm_sec=21, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T04:14:25.667 INFO:tasks.ceph:pgid 31.9 last_scrub_stamp 2024-09-16T03:43:33.221564+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=43, tm_sec=33, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=14, tm_sec=21, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T04:14:25.667 INFO:tasks.ceph:pgid 31.a last_scrub_stamp 2024-09-16T03:43:33.221564+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=43, tm_sec=33, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=14, tm_sec=21, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T04:14:25.668 INFO:tasks.ceph:pgid 31.b last_scrub_stamp 2024-09-16T03:43:33.221564+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=43, tm_sec=33, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=14, tm_sec=21, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T04:14:25.668 INFO:tasks.ceph:pgid 31.4 last_scrub_stamp 2024-09-16T03:43:33.221564+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=43, tm_sec=33, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=14, tm_sec=21, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T04:14:25.668 INFO:tasks.ceph:pgid 31.5 last_scrub_stamp 2024-09-16T03:43:33.221564+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=43, tm_sec=33, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=14, tm_sec=21, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T04:14:25.668 INFO:tasks.ceph:pgid 31.6 last_scrub_stamp 2024-09-16T03:43:33.221564+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=43, tm_sec=33, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=14, tm_sec=21, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T04:14:25.668 INFO:tasks.ceph:pgid 31.7 last_scrub_stamp 2024-09-16T03:43:33.221564+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=43, tm_sec=33, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=14, tm_sec=21, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T04:14:25.668 INFO:tasks.ceph:pgid 31.0 last_scrub_stamp 2024-09-16T03:43:33.221564+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=43, tm_sec=33, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=14, tm_sec=21, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T04:14:25.668 INFO:tasks.ceph:pgid 31.1 last_scrub_stamp 2024-09-16T03:43:33.221564+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=43, tm_sec=33, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=14, tm_sec=21, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T04:14:25.668 INFO:tasks.ceph:pgid 31.2 last_scrub_stamp 2024-09-16T03:43:33.221564+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=43, tm_sec=33, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=14, tm_sec=21, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T04:14:25.669 INFO:tasks.ceph:pgid 31.3 last_scrub_stamp 2024-09-16T03:43:33.221564+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=3, tm_min=43, tm_sec=33, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=4, tm_min=14, tm_sec=21, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T04:14:25.669 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-16T04:14:45.670 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T04:14:45.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.797+0000 7feab7e2f640 10 monclient: get_monmap_and_config 2024-09-16T04:14:45.805 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.797+0000 7feab7e2f640 10 monclient: build_initial_monmap 2024-09-16T04:14:45.806 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.797+0000 7feab7e2f640 10 monclient: monmap: 2024-09-16T04:14:45.806 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:14:45.806 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:14:45.806 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:14:45.803350+0000 2024-09-16T04:14:45.806 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:14:45.803350+0000 2024-09-16T04:14:45.806 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:14:45.807 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:45.807 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:14:45.807 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:14:45.807 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:14:45.807 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:14:45.807 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:14:45.808 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:45.808 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.801+0000 7feab7e2f640 1 Processor -- start 2024-09-16T04:14:45.808 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.801+0000 7feab7e2f640 1 -- start start 2024-09-16T04:14:45.808 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.801+0000 7feab7e2f640 10 monclient: init 2024-09-16T04:14:45.824 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.801+0000 7feab7e2f640 10 monclient: _reopen_session rank -1 2024-09-16T04:14:45.824 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.801+0000 7feab7e2f640 10 monclient: _add_conns ranks=[1,2,3,4,0] 2024-09-16T04:14:45.825 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.801+0000 7feab7e2f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7feab00a39b0 addr v1:172.21.15.154:6789/0 2024-09-16T04:14:45.825 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.801+0000 7feab7e2f640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7feab00a3d90 0x7feab013c380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:45.825 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.801+0000 7feab7e2f640 10 monclient(hunting): picked mon.noname-b con 0x7feab00a3d90 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:14:45.825 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.801+0000 7feab7e2f640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7feab013c8c0 0x7feab013ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:45.825 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.801+0000 7feab7e2f640 10 monclient(hunting): picked mon.noname-a con 0x7feab013c8c0 addr v2:172.21.15.42:3300/0 2024-09-16T04:14:45.826 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.801+0000 7feab7e2f640 1 -- --> v1:172.21.15.154:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feab0134290 con 0x7feab00a39b0 2024-09-16T04:14:45.826 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.801+0000 7feab7e2f640 10 monclient(hunting): start opening mon connection 2024-09-16T04:14:45.826 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.801+0000 7feab7e2f640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7feab009c140 con 0x7feab013c8c0 2024-09-16T04:14:45.826 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.801+0000 7feab7e2f640 10 monclient(hunting): start opening mon connection 2024-09-16T04:14:45.826 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.801+0000 7feab7e2f640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7feab009c420 con 0x7feab00a3d90 2024-09-16T04:14:45.826 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.801+0000 7feab7e2f640 10 monclient(hunting): _renew_subs 2024-09-16T04:14:45.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.801+0000 7feab7e2f640 10 monclient(hunting): authenticate will time out at 2932.773085s 2024-09-16T04:14:45.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.801+0000 7feab63cc640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7feab013c8c0 0x7feab013ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:45.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.801+0000 7feab53ca640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7feab00a3d90 0x7feab013c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:45.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.801+0000 7feab63cc640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7feab013c8c0 0x7feab013ecb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:52564/0 (socket says 172.21.15.42:52564) 2024-09-16T04:14:45.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.801+0000 7feab63cc640 1 -- 172.21.15.42:0/3057969259 learned_addr learned my addr 172.21.15.42:0/3057969259 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:14:45.827 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.801+0000 7feab63cc640 10 monclient(hunting): get_auth_request con 0x7feab013c8c0 auth_method 0 2024-09-16T04:14:45.828 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.801+0000 7feab63cc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:14:45.828 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.801+0000 7feab63cc640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:45.828 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.801+0000 7feab63cc640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:45.828 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.801+0000 7feab53ca640 10 monclient(hunting): get_auth_request con 0x7feab00a3d90 auth_method 0 2024-09-16T04:14:45.828 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.801+0000 7feab53ca640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:14:45.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.801+0000 7feab53ca640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:45.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.801+0000 7feab53ca640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:45.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.801+0000 7feab63cc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:14:45.829 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.801+0000 7feab63cc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:14:45.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.801+0000 7feab63cc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:14:45.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.801+0000 7feab53ca640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:14:45.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.801+0000 7feab53ca640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:14:45.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.801+0000 7feab53ca640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:14:45.830 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.801+0000 7feab53ca640 10 monclient(hunting): handle_auth_done global_id 1094303 payload 274 2024-09-16T04:14:45.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.801+0000 7feab53ca640 1 -- 172.21.15.42:0/3057969259 >> v2:172.21.15.42:3300/0 conn(0x7feab013c8c0 msgr2=0x7feab013ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:14:45.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.801+0000 7feab53ca640 1 --2- 172.21.15.42:0/3057969259 >> v2:172.21.15.42:3300/0 conn(0x7feab013c8c0 0x7feab013ecb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:45.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.801+0000 7feab53ca640 1 -- 172.21.15.42:0/3057969259 >> v1:172.21.15.154:6789/0 conn(0x7feab00a39b0 legacy=0x7feab0133b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:45.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.801+0000 7feab53ca640 10 monclient: _finish_hunting 0 2024-09-16T04:14:45.831 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.801+0000 7feab53ca640 1 monclient: found mon.noname-b 2024-09-16T04:14:45.832 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.801+0000 7feab53ca640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:14:45.832 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.801+0000 7feab53ca640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.42:3301/0 2024-09-16T04:14:45.832 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.801+0000 7feab53ca640 1 -- 172.21.15.42:0/3057969259 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feab006dd60 con 0x7feab00a3d90 2024-09-16T04:14:45.832 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.801+0000 7feab63cc640 1 --2- 172.21.15.42:0/3057969259 >> v2:172.21.15.42:3300/0 conn(0x7feab013c8c0 0x7feab013ecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:14:45.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.801+0000 7feab53ca640 1 --2- 172.21.15.42:0/3057969259 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7feab00a3d90 0x7feab013c380 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7feaa00097c0 tx=0x7feaa001c360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1f2e0f089502553d server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:14:45.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.805+0000 7feab4bc9640 1 -- 172.21.15.42:0/3057969259 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feaa00b0070 con 0x7feab00a3d90 2024-09-16T04:14:45.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.805+0000 7feab4bc9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:45.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.805+0000 7feab4bc9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-16T04:14:45.833 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.805+0000 7feab4bc9640 10 monclient: dump: 2024-09-16T04:14:45.833 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:14:45.834 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:14:45.834 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:45.834 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:45.834 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:45.834 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:45.834 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:14:45.835 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:14:45.835 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:14:45.835 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:45.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.805+0000 7feab4bc9640 10 monclient: _finish_auth 0 2024-09-16T04:14:45.835 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.805+0000 7feab4bc9640 10 monclient: _check_auth_tickets 2024-09-16T04:14:45.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.805+0000 7feab4bc9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:14:45.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.805+0000 7feab4bc9640 1 -- 172.21.15.42:0/3057969259 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feaa00ab600 con 0x7feab00a3d90 2024-09-16T04:14:45.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.805+0000 7feab4bc9640 10 monclient: handle_config config(0 keys) 2024-09-16T04:14:45.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.805+0000 7feab4bc9640 1 -- 172.21.15.42:0/3057969259 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feaa00ab9d0 con 0x7feab00a3d90 2024-09-16T04:14:45.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.805+0000 7feab7e2f640 5 monclient: authenticate success, global_id 1094303 2024-09-16T04:14:45.836 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.805+0000 7feab7e2f640 10 monclient: get_monmap_and_config success 2024-09-16T04:14:45.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.805+0000 7feab7e2f640 4 set_mon_vals no callback set 2024-09-16T04:14:45.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.805+0000 7feab4bc9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:45.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.805+0000 7feab4bc9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:14:45.837 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.805+0000 7feab4bc9640 10 monclient: dump: 2024-09-16T04:14:45.837 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:14:45.838 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:14:45.838 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:45.838 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:45.838 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:45.838 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:45.839 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:14:45.839 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:14:45.839 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:14:45.839 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:45.839 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.805+0000 7feab7e2f640 10 monclient: shutdown 2024-09-16T04:14:45.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.805+0000 7feab7e2f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:14:45.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.805+0000 7feab7e2f640 1 -- 172.21.15.42:0/3057969259 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7feab00a3d90 msgr2=0x7feab013c380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:45.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.805+0000 7feab7e2f640 1 --2- 172.21.15.42:0/3057969259 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7feab00a3d90 0x7feab013c380 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7feaa00097c0 tx=0x7feaa001c360 comp rx=0 tx=0).stop 2024-09-16T04:14:45.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.805+0000 7feab7e2f640 1 -- 172.21.15.42:0/3057969259 shutdown_connections 2024-09-16T04:14:45.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.805+0000 7feab7e2f640 1 --2- 172.21.15.42:0/3057969259 >> v2:172.21.15.42:3300/0 conn(0x7feab013c8c0 0x7feab013ecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:45.840 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.805+0000 7feab7e2f640 1 --2- 172.21.15.42:0/3057969259 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7feab00a3d90 0x7feab013c380 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:45.841 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.805+0000 7feab7e2f640 1 -- 172.21.15.42:0/3057969259 >> 172.21.15.42:0/3057969259 conn(0x7feab00a9fb0 msgr2=0x7feab00aa3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:14:45.841 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.805+0000 7feab7e2f640 1 -- 172.21.15.42:0/3057969259 shutdown_connections 2024-09-16T04:14:45.841 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.805+0000 7feab7e2f640 1 -- 172.21.15.42:0/3057969259 wait complete. 2024-09-16T04:14:45.841 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.805+0000 7feab7e2f640 10 monclient: build_initial_monmap 2024-09-16T04:14:45.841 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.805+0000 7feab7e2f640 10 monclient: monmap: 2024-09-16T04:14:45.842 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:14:45.842 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:14:45.842 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:14:45.842 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:14:45.842 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:14:45.842 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:45.843 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:14:45.843 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:14:45.843 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:14:45.843 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:45.843 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.805+0000 7feab7e2f640 1 librados: starting msgr at 2024-09-16T04:14:45.844 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.805+0000 7feab7e2f640 1 librados: starting objecter 2024-09-16T04:14:45.844 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.805+0000 7feab7e2f640 1 Processor -- start 2024-09-16T04:14:45.844 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.805+0000 7feab7e2f640 1 -- start start 2024-09-16T04:14:45.844 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.805+0000 7feab7e2f640 1 librados: setting wanted keys 2024-09-16T04:14:45.844 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.805+0000 7feab7e2f640 1 librados: calling monclient init 2024-09-16T04:14:45.844 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.805+0000 7feab7e2f640 10 monclient: init 2024-09-16T04:14:45.845 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.805+0000 7feab7e2f640 10 monclient: _reopen_session rank -1 2024-09-16T04:14:45.845 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.805+0000 7feab7e2f640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-16T04:14:45.845 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.805+0000 7feab7e2f640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feab00a39b0 0x7feab01cf1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:45.845 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.805+0000 7feab7e2f640 10 monclient(hunting): picked mon.noname-a con 0x7feab00a39b0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:14:45.845 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.805+0000 7feab7e2f640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7feab00a3d90 0x7feab01cf6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:45.846 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.805+0000 7feab7e2f640 10 monclient(hunting): picked mon.noname-c con 0x7feab00a3d90 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:14:45.846 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.805+0000 7feab7e2f640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feab013c8c0 0x7feab01cfc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:45.846 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.805+0000 7feab7e2f640 10 monclient(hunting): picked mon.noname-b con 0x7feab013c8c0 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:14:45.846 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.805+0000 7feab7e2f640 10 monclient(hunting): start opening mon connection 2024-09-16T04:14:45.846 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.805+0000 7feab7e2f640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7feab0141da0 con 0x7feab00a39b0 2024-09-16T04:14:45.847 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.805+0000 7feab7e2f640 10 monclient(hunting): start opening mon connection 2024-09-16T04:14:45.847 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.805+0000 7feab7e2f640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7feab0141c20 con 0x7feab013c8c0 2024-09-16T04:14:45.847 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.805+0000 7feab7e2f640 10 monclient(hunting): start opening mon connection 2024-09-16T04:14:45.847 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.805+0000 7feab7e2f640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7feab0141f20 con 0x7feab00a3d90 2024-09-16T04:14:45.847 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.805+0000 7feab7e2f640 10 monclient(hunting): _renew_subs 2024-09-16T04:14:45.847 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.805+0000 7feab7e2f640 10 monclient(hunting): authenticate will time out at 2932.778390s 2024-09-16T04:14:45.848 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.805+0000 7feab53ca640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7feab00a3d90 0x7feab01cf6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:45.848 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.805+0000 7feab5bcb640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feab00a39b0 0x7feab01cf1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:45.848 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.805+0000 7feab53ca640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7feab00a3d90 0x7feab01cf6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:46372/0 (socket says 172.21.15.42:46372) 2024-09-16T04:14:45.848 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.805+0000 7feab53ca640 1 -- 172.21.15.42:0/3550109999 learned_addr learned my addr 172.21.15.42:0/3550109999 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:14:45.848 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.805+0000 7feab5bcb640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feab00a39b0 0x7feab01cf1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:52572/0 (socket says 172.21.15.42:52572) 2024-09-16T04:14:45.849 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.805+0000 7feab53ca640 10 monclient(hunting): get_auth_request con 0x7feab00a3d90 auth_method 0 2024-09-16T04:14:45.849 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.805+0000 7feab53ca640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:14:45.849 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.805+0000 7feab53ca640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:45.849 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.805+0000 7feab53ca640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:45.849 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.805+0000 7feab5bcb640 10 monclient(hunting): get_auth_request con 0x7feab00a39b0 auth_method 0 2024-09-16T04:14:45.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.805+0000 7feab5bcb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:14:45.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.805+0000 7feab5bcb640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:45.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.805+0000 7feab5bcb640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:45.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.805+0000 7feab53ca640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:14:45.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.805+0000 7feab53ca640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:14:45.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.805+0000 7feab53ca640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:14:45.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.805+0000 7feab5bcb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:14:45.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.805+0000 7feab5bcb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:14:45.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.805+0000 7feab5bcb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:14:45.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.805+0000 7feab63cc640 1 --2- 172.21.15.42:0/3550109999 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feab013c8c0 0x7feab01cfc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:45.850 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.805+0000 7feab53ca640 10 monclient(hunting): handle_auth_done global_id 1094306 payload 824 2024-09-16T04:14:45.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.805+0000 7feab53ca640 1 -- 172.21.15.42:0/3550109999 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feab013c8c0 msgr2=0x7feab01cfc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:45.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.805+0000 7feab53ca640 1 --2- 172.21.15.42:0/3550109999 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feab013c8c0 0x7feab01cfc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:45.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.805+0000 7feab53ca640 1 -- 172.21.15.42:0/3550109999 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feab00a39b0 msgr2=0x7feab01cf1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:45.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.805+0000 7feab53ca640 1 --2- 172.21.15.42:0/3550109999 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feab00a39b0 0x7feab01cf1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:45.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.805+0000 7feab53ca640 10 monclient: _finish_hunting 0 2024-09-16T04:14:45.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.805+0000 7feab53ca640 1 monclient: found mon.noname-c 2024-09-16T04:14:45.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.805+0000 7feab53ca640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:14:45.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.805+0000 7feab53ca640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T04:14:45.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.805+0000 7feab53ca640 1 -- 172.21.15.42:0/3550109999 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feab006f680 con 0x7feab00a3d90 2024-09-16T04:14:45.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.805+0000 7feab53ca640 1 --2- 172.21.15.42:0/3550109999 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7feab00a3d90 0x7feab01cf6e0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7feaa00ae040 tx=0x7feaa0019100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:14:45.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.805+0000 7feaa67fc640 10 client.?.objecter ms_handle_connect 0x7feab00a3d90 2024-09-16T04:14:45.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.805+0000 7feaa67fc640 10 client.?.objecter resend_mon_ops 2024-09-16T04:14:45.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.805+0000 7feaa67fc640 1 -- 172.21.15.42:0/3550109999 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feaa00b0040 con 0x7feab00a3d90 2024-09-16T04:14:45.851 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.805+0000 7feaa67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:45.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.805+0000 7feaa67fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:14:45.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.805+0000 7feaa67fc640 10 monclient: dump: 2024-09-16T04:14:45.852 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:14:45.852 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:14:45.852 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:45.852 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:45.852 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:45.852 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:45.852 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:14:45.852 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:14:45.852 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:14:45.852 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:45.852 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.805+0000 7feaa67fc640 10 monclient: _finish_auth 0 2024-09-16T04:14:45.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.805+0000 7feaa67fc640 10 monclient: _check_auth_tickets 2024-09-16T04:14:45.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.805+0000 7feaa67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:14:45.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.805+0000 7feaa67fc640 1 -- 172.21.15.42:0/3550109999 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feaa00b43f0 con 0x7feab00a3d90 2024-09-16T04:14:45.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.805+0000 7feaa67fc640 10 monclient: handle_config config(0 keys) 2024-09-16T04:14:45.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.805+0000 7feaa67fc640 1 -- 172.21.15.42:0/3550109999 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feaa00b4800 con 0x7feab00a3d90 2024-09-16T04:14:45.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.805+0000 7feab7e2f640 5 monclient: authenticate success, global_id 1094306 2024-09-16T04:14:45.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.805+0000 7feab7e2f640 10 monclient: _renew_subs 2024-09-16T04:14:45.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.805+0000 7feab7e2f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:14:45.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.805+0000 7feab7e2f640 1 -- 172.21.15.42:0/3550109999 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feab013a9b0 con 0x7feab00a3d90 2024-09-16T04:14:45.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.805+0000 7feaa7fff640 4 set_mon_vals no callback set 2024-09-16T04:14:45.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.805+0000 7feaa67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:45.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.805+0000 7feaa67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:14:45.853 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.805+0000 7feaa67fc640 10 monclient: dump: 2024-09-16T04:14:45.854 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:14:45.854 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:14:45.854 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:45.854 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:45.854 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:45.854 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:45.854 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:14:45.854 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:14:45.854 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:14:45.854 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:45.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.805+0000 7feab7e2f640 10 client.1094306.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:14:45.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.805+0000 7feab7e2f640 10 monclient: _renew_subs 2024-09-16T04:14:45.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.805+0000 7feab7e2f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:14:45.854 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.805+0000 7feab7e2f640 1 -- 172.21.15.42:0/3550109999 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7feab01397e0 con 0x7feab00a3d90 2024-09-16T04:14:45.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.805+0000 7feab7e2f640 1 librados: init done 2024-09-16T04:14:45.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.809+0000 7feab7e2f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:14:45.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.809+0000 7feab7e2f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:14:45.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.809+0000 7feaa67fc640 1 -- 172.21.15.42:0/3550109999 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7feaa00b49a0 con 0x7feab00a3d90 2024-09-16T04:14:45.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.809+0000 7feab7e2f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:14:45.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.809+0000 7feab7e2f640 1 -- 172.21.15.42:0/3550109999 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feab0133b80 con 0x7feab00a3d90 2024-09-16T04:14:45.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.809+0000 7feaa67fc640 1 --2- 172.21.15.42:0/3550109999 >> v2:172.21.15.154:6812/254845342 conn(0x7fea94037840 0x7fea94039d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:45.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.809+0000 7feaa67fc640 1 -- 172.21.15.42:0/3550109999 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(710..710 src has 681..710) ==== 4681+0+0 (secure 0 0 0) 0x7feaa00e26c0 con 0x7feab00a3d90 2024-09-16T04:14:45.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.809+0000 7feaa67fc640 10 client.1094306.objecter ms_dispatch 0x7feab0000ba0 osd_map(710..710 src has 681..710) 2024-09-16T04:14:45.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.809+0000 7feaa67fc640 3 client.1094306.objecter handle_osd_map got epochs [710,710] > 0 2024-09-16T04:14:45.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.809+0000 7feaa67fc640 3 client.1094306.objecter handle_osd_map decoding full epoch 710 2024-09-16T04:14:45.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.809+0000 7feaa67fc640 20 client.1094306.objecter dump_active .. 0 homeless 2024-09-16T04:14:45.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.809+0000 7feab5bcb640 1 --2- 172.21.15.42:0/3550109999 >> v2:172.21.15.154:6812/254845342 conn(0x7fea94037840 0x7fea94039d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:45.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.809+0000 7feab5bcb640 10 monclient: get_auth_request con 0x7fea94037840 auth_method 0 2024-09-16T04:14:45.855 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.809+0000 7feaa67fc640 10 client.1094306.objecter ms_handle_connect 0x7fea94037840 2024-09-16T04:14:45.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.809+0000 7feab5bcb640 1 --2- 172.21.15.42:0/3550109999 >> v2:172.21.15.154:6812/254845342 conn(0x7fea94037840 0x7fea94039d00 secure :-1 s=READY pgs=4028 cs=0 l=1 rev1=1 crypto rx=0x7feaac001770 tx=0x7feaac009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:14:45.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.813+0000 7feaa67fc640 1 -- 172.21.15.42:0/3550109999 <== mon.2 v2:172.21.15.42:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7feab006f680 con 0x7feab00a3d90 2024-09-16T04:14:45.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.813+0000 7feaa67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:14:45.856 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.813+0000 7feaa67fc640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:14:45.978 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.973+0000 7feab7e2f640 1 -- 172.21.15.42:0/3550109999 --> v2:172.21.15.154:6812/254845342 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7feab013be00 con 0x7fea94037840 2024-09-16T04:14:45.986 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.981+0000 7feaa67fc640 1 -- 172.21.15.42:0/3550109999 <== mgr.4102 v2:172.21.15.154:6812/254845342 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+114198 (secure 0 0 0) 0x7feab013be00 con 0x7fea94037840 2024-09-16T04:14:45.988 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-09-16T04:14:45.989 INFO:teuthology.orchestra.run.smithi042.stderr:dumped all 2024-09-16T04:14:45.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.985+0000 7feab7e2f640 10 librados: watch_flush enter 2024-09-16T04:14:45.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.985+0000 7feab7e2f640 10 librados: watch_flush exit 2024-09-16T04:14:45.989 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.985+0000 7feab7e2f640 20 client.1094306.objecter shutdown clearing up homeless session... 2024-09-16T04:14:45.990 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.985+0000 7feab7e2f640 10 client.1094306.objecter successfully canceled tick 2024-09-16T04:14:45.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.985+0000 7feab7e2f640 1 -- 172.21.15.42:0/3550109999 >> v2:172.21.15.154:6812/254845342 conn(0x7fea94037840 msgr2=0x7fea94039d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:45.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.985+0000 7feab7e2f640 1 --2- 172.21.15.42:0/3550109999 >> v2:172.21.15.154:6812/254845342 conn(0x7fea94037840 0x7fea94039d00 secure :-1 s=READY pgs=4028 cs=0 l=1 rev1=1 crypto rx=0x7feaac001770 tx=0x7feaac009040 comp rx=0 tx=0).stop 2024-09-16T04:14:45.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.985+0000 7feab7e2f640 10 monclient: shutdown 2024-09-16T04:14:45.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.985+0000 7feab7e2f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:14:45.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.985+0000 7feab7e2f640 1 -- 172.21.15.42:0/3550109999 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7feab00a3d90 msgr2=0x7feab01cf6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:45.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.985+0000 7feab7e2f640 1 --2- 172.21.15.42:0/3550109999 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7feab00a3d90 0x7feab01cf6e0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7feaa00ae040 tx=0x7feaa0019100 comp rx=0 tx=0).stop 2024-09-16T04:14:45.993 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.985+0000 7feab7e2f640 1 -- 172.21.15.42:0/3550109999 shutdown_connections 2024-09-16T04:14:45.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.985+0000 7feab7e2f640 1 --2- 172.21.15.42:0/3550109999 >> v2:172.21.15.154:6812/254845342 conn(0x7fea94037840 0x7fea94039d00 unknown :-1 s=CLOSED pgs=4028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:45.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.985+0000 7feab7e2f640 1 --2- 172.21.15.42:0/3550109999 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7feab013c8c0 0x7feab01cfc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:45.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.985+0000 7feab7e2f640 1 --2- 172.21.15.42:0/3550109999 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7feab00a3d90 0x7feab01cf6e0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:45.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.985+0000 7feab7e2f640 1 --2- 172.21.15.42:0/3550109999 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7feab00a39b0 0x7feab01cf1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:45.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.985+0000 7feab7e2f640 1 -- 172.21.15.42:0/3550109999 >> 172.21.15.42:0/3550109999 conn(0x7feab00a9fb0 msgr2=0x7feab00a89d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:14:45.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.985+0000 7feab7e2f640 1 -- 172.21.15.42:0/3550109999 shutdown_connections 2024-09-16T04:14:45.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.985+0000 7feab7e2f640 1 -- 172.21.15.42:0/3550109999 wait complete. 2024-09-16T04:14:45.994 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:45.985+0000 7feab7e2f640 1 librados: shutdown 2024-09-16T04:14:46.005 INFO:teuthology.orchestra.run.smithi042.stdout:{"pg_ready":true,"pg_map":{"version":1665,"stamp":"2024-09-16T04:14:44.238757+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590392,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":122,"num_read_kb":106,"num_write":168,"num_write_kb":2425,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":132,"ondisk_log_size":132,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":430520,"kb_used_data":22680,"kb_used_omap":9,"kb_used_meta":407734,"kb_avail":565800520,"statfs":{"total":579820584960,"available":579379732480,"internally_reserved":0,"allocated":23224320,"data_stored":10083938,"data_compressed":18838,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":9567,"internal_metadata":417520289},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003766"},"pg_stats":[{"pgid":"31.f","version":"0'0","reported_seq":1359,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:14:27.404629+0000","last_change":"2024-09-16T04:14:27.404685+0000","last_active":"2024-09-16T04:14:27.404629+0000","last_peered":"2024-09-16T04:14:27.404629+0000","last_clean":"2024-09-16T04:14:27.404629+0000","last_became_active":"2024-09-16T03:43:34.399531+0000","last_became_peered":"2024-09-16T03:43:34.399531+0000","last_unstale":"2024-09-16T04:14:27.404629+0000","last_undegraded":"2024-09-16T04:14:27.404629+0000","last_fullsized":"2024-09-16T04:14:27.404629+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:14:27.404455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:14:27.404455+0000","last_clean_scrub_stamp":"2024-09-16T04:14:27.404455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:42:25.239301+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.e","version":"0'0","reported_seq":1360,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:14:26.894300+0000","last_change":"2024-09-16T04:14:26.894357+0000","last_active":"2024-09-16T04:14:26.894300+0000","last_peered":"2024-09-16T04:14:26.894300+0000","last_clean":"2024-09-16T04:14:26.894300+0000","last_became_active":"2024-09-16T03:43:34.492249+0000","last_became_peered":"2024-09-16T03:43:34.492249+0000","last_unstale":"2024-09-16T04:14:26.894300+0000","last_undegraded":"2024-09-16T04:14:26.894300+0000","last_fullsized":"2024-09-16T04:14:26.894300+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:14:26.894111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:14:26.894111+0000","last_clean_scrub_stamp":"2024-09-16T04:14:26.894111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:44:44.624870+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.d","version":"0'0","reported_seq":1382,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:14:30.098195+0000","last_change":"2024-09-16T04:14:30.098297+0000","last_active":"2024-09-16T04:14:30.098195+0000","last_peered":"2024-09-16T04:14:30.098195+0000","last_clean":"2024-09-16T04:14:30.098195+0000","last_became_active":"2024-09-16T03:43:34.399055+0000","last_became_peered":"2024-09-16T03:43:34.399055+0000","last_unstale":"2024-09-16T04:14:30.098195+0000","last_undegraded":"2024-09-16T04:14:30.098195+0000","last_fullsized":"2024-09-16T04:14:30.098195+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:14:30.097981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:14:30.097981+0000","last_clean_scrub_stamp":"2024-09-16T04:14:30.097981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:40:57.629609+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.c","version":"0'0","reported_seq":1362,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:14:31.951661+0000","last_change":"2024-09-16T04:14:31.951718+0000","last_active":"2024-09-16T04:14:31.951661+0000","last_peered":"2024-09-16T04:14:31.951661+0000","last_clean":"2024-09-16T04:14:31.951661+0000","last_became_active":"2024-09-16T03:43:34.403187+0000","last_became_peered":"2024-09-16T03:43:34.403187+0000","last_unstale":"2024-09-16T04:14:31.951661+0000","last_undegraded":"2024-09-16T04:14:31.951661+0000","last_fullsized":"2024-09-16T04:14:31.951661+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:14:31.951469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:14:31.951469+0000","last_clean_scrub_stamp":"2024-09-16T04:14:31.951469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:42:08.671599+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.13","version":"0'0","reported_seq":1363,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:14:28.351010+0000","last_change":"2024-09-16T04:14:28.351167+0000","last_active":"2024-09-16T04:14:28.351010+0000","last_peered":"2024-09-16T04:14:28.351010+0000","last_clean":"2024-09-16T04:14:28.351010+0000","last_became_active":"2024-09-16T03:43:34.382422+0000","last_became_peered":"2024-09-16T03:43:34.382422+0000","last_unstale":"2024-09-16T04:14:28.351010+0000","last_undegraded":"2024-09-16T04:14:28.351010+0000","last_fullsized":"2024-09-16T04:14:28.351010+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:14:28.350803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:14:28.350803+0000","last_clean_scrub_stamp":"2024-09-16T04:14:28.350803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:34:30.186749+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.12","version":"0'0","reported_seq":1362,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:14:32.921817+0000","last_change":"2024-09-16T04:14:32.921873+0000","last_active":"2024-09-16T04:14:32.921817+0000","last_peered":"2024-09-16T04:14:32.921817+0000","last_clean":"2024-09-16T04:14:32.921817+0000","last_became_active":"2024-09-16T03:43:34.404299+0000","last_became_peered":"2024-09-16T03:43:34.404299+0000","last_unstale":"2024-09-16T04:14:32.921817+0000","last_undegraded":"2024-09-16T04:14:32.921817+0000","last_fullsized":"2024-09-16T04:14:32.921817+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:14:32.921650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:14:32.921650+0000","last_clean_scrub_stamp":"2024-09-16T04:14:32.921650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:55:31.307282+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.11","version":"0'0","reported_seq":1363,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:14:27.373347+0000","last_change":"2024-09-16T04:14:27.373401+0000","last_active":"2024-09-16T04:14:27.373347+0000","last_peered":"2024-09-16T04:14:27.373347+0000","last_clean":"2024-09-16T04:14:27.373347+0000","last_became_active":"2024-09-16T03:43:34.354598+0000","last_became_peered":"2024-09-16T03:43:34.354598+0000","last_unstale":"2024-09-16T04:14:27.373347+0000","last_undegraded":"2024-09-16T04:14:27.373347+0000","last_fullsized":"2024-09-16T04:14:27.373347+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:14:27.373146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:14:27.373146+0000","last_clean_scrub_stamp":"2024-09-16T04:14:27.373146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:23:48.982619+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.10","version":"0'0","reported_seq":1360,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:14:25.905518+0000","last_change":"2024-09-16T04:14:25.905544+0000","last_active":"2024-09-16T04:14:25.905518+0000","last_peered":"2024-09-16T04:14:25.905518+0000","last_clean":"2024-09-16T04:14:25.905518+0000","last_became_active":"2024-09-16T03:43:34.434548+0000","last_became_peered":"2024-09-16T03:43:34.434548+0000","last_unstale":"2024-09-16T04:14:25.905518+0000","last_undegraded":"2024-09-16T04:14:25.905518+0000","last_fullsized":"2024-09-16T04:14:25.905518+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:14:25.905426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:14:25.905426+0000","last_clean_scrub_stamp":"2024-09-16T04:14:25.905426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:24:15.619236+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.17","version":"0'0","reported_seq":1358,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:14:30.376636+0000","last_change":"2024-09-16T04:14:30.376692+0000","last_active":"2024-09-16T04:14:30.376636+0000","last_peered":"2024-09-16T04:14:30.376636+0000","last_clean":"2024-09-16T04:14:30.376636+0000","last_became_active":"2024-09-16T03:43:34.454859+0000","last_became_peered":"2024-09-16T03:43:34.454859+0000","last_unstale":"2024-09-16T04:14:30.376636+0000","last_undegraded":"2024-09-16T04:14:30.376636+0000","last_fullsized":"2024-09-16T04:14:30.376636+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:14:30.376463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:14:30.376463+0000","last_clean_scrub_stamp":"2024-09-16T04:14:30.376463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:18:34.143490+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.16","version":"0'0","reported_seq":1356,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:14:30.293421+0000","last_change":"2024-09-16T04:14:30.293478+0000","last_active":"2024-09-16T04:14:30.293421+0000","last_peered":"2024-09-16T04:14:30.293421+0000","last_clean":"2024-09-16T04:14:30.293421+0000","last_became_active":"2024-09-16T03:43:34.387629+0000","last_became_peered":"2024-09-16T03:43:34.387629+0000","last_unstale":"2024-09-16T04:14:30.293421+0000","last_undegraded":"2024-09-16T04:14:30.293421+0000","last_fullsized":"2024-09-16T04:14:30.293421+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:14:30.293245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:14:30.293245+0000","last_clean_scrub_stamp":"2024-09-16T04:14:30.293245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:51:33.071787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.15","version":"0'0","reported_seq":1361,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:14:30.928205+0000","last_change":"2024-09-16T04:14:30.928264+0000","last_active":"2024-09-16T04:14:30.928205+0000","last_peered":"2024-09-16T04:14:30.928205+0000","last_clean":"2024-09-16T04:14:30.928205+0000","last_became_active":"2024-09-16T03:43:34.387630+0000","last_became_peered":"2024-09-16T03:43:34.387630+0000","last_unstale":"2024-09-16T04:14:30.928205+0000","last_undegraded":"2024-09-16T04:14:30.928205+0000","last_fullsized":"2024-09-16T04:14:30.928205+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:14:30.928002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:14:30.928002+0000","last_clean_scrub_stamp":"2024-09-16T04:14:30.928002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:40:59.329601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.14","version":"0'0","reported_seq":1359,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:14:25.404160+0000","last_change":"2024-09-16T04:14:25.404212+0000","last_active":"2024-09-16T04:14:25.404160+0000","last_peered":"2024-09-16T04:14:25.404160+0000","last_clean":"2024-09-16T04:14:25.404160+0000","last_became_active":"2024-09-16T03:43:34.378385+0000","last_became_peered":"2024-09-16T03:43:34.378385+0000","last_unstale":"2024-09-16T04:14:25.404160+0000","last_undegraded":"2024-09-16T04:14:25.404160+0000","last_fullsized":"2024-09-16T04:14:25.404160+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:14:25.404043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:14:25.404043+0000","last_clean_scrub_stamp":"2024-09-16T04:14:25.404043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:16:01.677636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"710'119","reported_seq":1535,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:14:24.915484+0000","last_change":"2024-09-16T04:14:24.915484+0000","last_active":"2024-09-16T04:14:24.915484+0000","last_peered":"2024-09-16T04:14:24.915484+0000","last_clean":"2024-09-16T04:14:24.915484+0000","last_became_active":"2024-09-16T03:42:18.797239+0000","last_became_peered":"2024-09-16T03:42:18.797239+0000","last_unstale":"2024-09-16T04:14:24.915484+0000","last_undegraded":"2024-09-16T04:14:24.915484+0000","last_fullsized":"2024-09-16T04:14:24.915484+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"710'119","last_scrub_stamp":"2024-09-16T04:14:24.915239+0000","last_deep_scrub":"710'119","last_deep_scrub_stamp":"2024-09-16T04:14:24.915239+0000","last_clean_scrub_stamp":"2024-09-16T04:14:24.915239+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:03:08.574566+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"710'2","reported_seq":1384,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:14:25.355847+0000","last_change":"2024-09-16T04:14:25.355917+0000","last_active":"2024-09-16T04:14:25.355847+0000","last_peered":"2024-09-16T04:14:25.355847+0000","last_clean":"2024-09-16T04:14:25.355847+0000","last_became_active":"2024-09-16T03:42:25.615930+0000","last_became_peered":"2024-09-16T03:42:25.615930+0000","last_unstale":"2024-09-16T04:14:25.355847+0000","last_undegraded":"2024-09-16T04:14:25.355847+0000","last_fullsized":"2024-09-16T04:14:25.355847+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"710'2","last_scrub_stamp":"2024-09-16T04:14:25.355660+0000","last_deep_scrub":"710'2","last_deep_scrub_stamp":"2024-09-16T04:14:25.355660+0000","last_clean_scrub_stamp":"2024-09-16T04:14:25.355660+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:48:49.022618+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00099470900000000009,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.1e","version":"0'0","reported_seq":1364,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:14:30.361030+0000","last_change":"2024-09-16T04:14:30.361088+0000","last_active":"2024-09-16T04:14:30.361030+0000","last_peered":"2024-09-16T04:14:30.361030+0000","last_clean":"2024-09-16T04:14:30.361030+0000","last_became_active":"2024-09-16T03:43:34.397708+0000","last_became_peered":"2024-09-16T03:43:34.397708+0000","last_unstale":"2024-09-16T04:14:30.361030+0000","last_undegraded":"2024-09-16T04:14:30.361030+0000","last_fullsized":"2024-09-16T04:14:30.361030+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:14:30.360853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:14:30.360853+0000","last_clean_scrub_stamp":"2024-09-16T04:14:30.360853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:12:05.191309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1382,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:14:25.950778+0000","last_change":"2024-09-16T04:14:25.950778+0000","last_active":"2024-09-16T04:14:25.950778+0000","last_peered":"2024-09-16T04:14:25.950778+0000","last_clean":"2024-09-16T04:14:25.950778+0000","last_became_active":"2024-09-16T03:42:25.612386+0000","last_became_peered":"2024-09-16T03:42:25.612386+0000","last_unstale":"2024-09-16T04:14:25.950778+0000","last_undegraded":"2024-09-16T04:14:25.950778+0000","last_fullsized":"2024-09-16T04:14:25.950778+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:14:25.950607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:14:25.950607+0000","last_clean_scrub_stamp":"2024-09-16T04:14:25.950607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:22:21.170788+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00081399800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1d","version":"0'0","reported_seq":1381,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:14:29.114625+0000","last_change":"2024-09-16T04:14:29.114726+0000","last_active":"2024-09-16T04:14:29.114625+0000","last_peered":"2024-09-16T04:14:29.114625+0000","last_clean":"2024-09-16T04:14:29.114625+0000","last_became_active":"2024-09-16T03:43:34.399300+0000","last_became_peered":"2024-09-16T03:43:34.399300+0000","last_unstale":"2024-09-16T04:14:29.114625+0000","last_undegraded":"2024-09-16T04:14:29.114625+0000","last_fullsized":"2024-09-16T04:14:29.114625+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:14:29.114391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:14:29.114391+0000","last_clean_scrub_stamp":"2024-09-16T04:14:29.114391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:28:06.531026+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1379,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:14:23.993466+0000","last_change":"2024-09-16T04:14:23.993487+0000","last_active":"2024-09-16T04:14:23.993466+0000","last_peered":"2024-09-16T04:14:23.993466+0000","last_clean":"2024-09-16T04:14:23.993466+0000","last_became_active":"2024-09-16T03:42:25.607533+0000","last_became_peered":"2024-09-16T03:42:25.607533+0000","last_unstale":"2024-09-16T04:14:23.993466+0000","last_undegraded":"2024-09-16T04:14:23.993466+0000","last_fullsized":"2024-09-16T04:14:23.993466+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:14:23.993392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:14:23.993392+0000","last_clean_scrub_stamp":"2024-09-16T04:14:23.993392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:08:00.260797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089083299999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1c","version":"0'0","reported_seq":1358,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:14:29.415656+0000","last_change":"2024-09-16T04:14:29.415761+0000","last_active":"2024-09-16T04:14:29.415656+0000","last_peered":"2024-09-16T04:14:29.415656+0000","last_clean":"2024-09-16T04:14:29.415656+0000","last_became_active":"2024-09-16T03:43:34.421320+0000","last_became_peered":"2024-09-16T03:43:34.421320+0000","last_unstale":"2024-09-16T04:14:29.415656+0000","last_undegraded":"2024-09-16T04:14:29.415656+0000","last_fullsized":"2024-09-16T04:14:29.415656+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:14:29.415443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:14:29.415443+0000","last_clean_scrub_stamp":"2024-09-16T04:14:29.415443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:02:14.694624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"710'3","reported_seq":1385,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:14:26.330994+0000","last_change":"2024-09-16T04:14:26.331110+0000","last_active":"2024-09-16T04:14:26.330994+0000","last_peered":"2024-09-16T04:14:26.330994+0000","last_clean":"2024-09-16T04:14:26.330994+0000","last_became_active":"2024-09-16T03:42:25.617896+0000","last_became_peered":"2024-09-16T03:42:25.617896+0000","last_unstale":"2024-09-16T04:14:26.330994+0000","last_undegraded":"2024-09-16T04:14:26.330994+0000","last_fullsized":"2024-09-16T04:14:26.330994+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"710'3","last_scrub_stamp":"2024-09-16T04:14:26.330786+0000","last_deep_scrub":"710'3","last_deep_scrub_stamp":"2024-09-16T04:14:26.330786+0000","last_clean_scrub_stamp":"2024-09-16T04:14:26.330786+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:11:26.230725+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0021072529999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.1f","version":"0'0","reported_seq":1361,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:14:24.946105+0000","last_change":"2024-09-16T04:14:24.946159+0000","last_active":"2024-09-16T04:14:24.946105+0000","last_peered":"2024-09-16T04:14:24.946105+0000","last_clean":"2024-09-16T04:14:24.946105+0000","last_became_active":"2024-09-16T03:43:34.415747+0000","last_became_peered":"2024-09-16T03:43:34.415747+0000","last_unstale":"2024-09-16T04:14:24.946105+0000","last_undegraded":"2024-09-16T04:14:24.946105+0000","last_fullsized":"2024-09-16T04:14:24.946105+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:14:24.945924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:14:24.945924+0000","last_clean_scrub_stamp":"2024-09-16T04:14:24.945924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:04:26.364453+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1376,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:14:25.325940+0000","last_change":"2024-09-16T04:14:25.326012+0000","last_active":"2024-09-16T04:14:25.325940+0000","last_peered":"2024-09-16T04:14:25.325940+0000","last_clean":"2024-09-16T04:14:25.325940+0000","last_became_active":"2024-09-16T03:42:25.609631+0000","last_became_peered":"2024-09-16T03:42:25.609631+0000","last_unstale":"2024-09-16T04:14:25.325940+0000","last_undegraded":"2024-09-16T04:14:25.325940+0000","last_fullsized":"2024-09-16T04:14:25.325940+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:14:25.325784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:14:25.325784+0000","last_clean_scrub_stamp":"2024-09-16T04:14:25.325784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:51:27.939663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001288897,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.19","version":"0'0","reported_seq":1356,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:14:28.338587+0000","last_change":"2024-09-16T04:14:28.338642+0000","last_active":"2024-09-16T04:14:28.338587+0000","last_peered":"2024-09-16T04:14:28.338587+0000","last_clean":"2024-09-16T04:14:28.338587+0000","last_became_active":"2024-09-16T03:43:34.332234+0000","last_became_peered":"2024-09-16T03:43:34.332234+0000","last_unstale":"2024-09-16T04:14:28.338587+0000","last_undegraded":"2024-09-16T04:14:28.338587+0000","last_fullsized":"2024-09-16T04:14:28.338587+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:14:28.338407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:14:28.338407+0000","last_clean_scrub_stamp":"2024-09-16T04:14:28.338407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:52:54.722361+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1381,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:14:26.903027+0000","last_change":"2024-09-16T04:14:26.903081+0000","last_active":"2024-09-16T04:14:26.903027+0000","last_peered":"2024-09-16T04:14:26.903027+0000","last_clean":"2024-09-16T04:14:26.903027+0000","last_became_active":"2024-09-16T03:42:25.614458+0000","last_became_peered":"2024-09-16T03:42:25.614458+0000","last_unstale":"2024-09-16T04:14:26.903027+0000","last_undegraded":"2024-09-16T04:14:26.903027+0000","last_fullsized":"2024-09-16T04:14:26.903027+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:14:26.902851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:14:26.902851+0000","last_clean_scrub_stamp":"2024-09-16T04:14:26.902851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T11:26:22.262679+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0012701769999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.18","version":"0'0","reported_seq":1356,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:14:29.325290+0000","last_change":"2024-09-16T04:14:29.325390+0000","last_active":"2024-09-16T04:14:29.325290+0000","last_peered":"2024-09-16T04:14:29.325290+0000","last_clean":"2024-09-16T04:14:29.325290+0000","last_became_active":"2024-09-16T03:43:34.354705+0000","last_became_peered":"2024-09-16T03:43:34.354705+0000","last_unstale":"2024-09-16T04:14:29.325290+0000","last_undegraded":"2024-09-16T04:14:29.325290+0000","last_fullsized":"2024-09-16T04:14:29.325290+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:14:29.325086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:14:29.325086+0000","last_clean_scrub_stamp":"2024-09-16T04:14:29.325086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:48:15.456536+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1375,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:14:24.293029+0000","last_change":"2024-09-16T04:14:24.293066+0000","last_active":"2024-09-16T04:14:24.293029+0000","last_peered":"2024-09-16T04:14:24.293029+0000","last_clean":"2024-09-16T04:14:24.293029+0000","last_became_active":"2024-09-16T03:42:25.608453+0000","last_became_peered":"2024-09-16T03:42:25.608453+0000","last_unstale":"2024-09-16T04:14:24.293029+0000","last_undegraded":"2024-09-16T04:14:24.293029+0000","last_fullsized":"2024-09-16T04:14:24.293029+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:14:24.292923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:14:24.292923+0000","last_clean_scrub_stamp":"2024-09-16T04:14:24.292923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:52:50.960136+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001163533,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.1b","version":"0'0","reported_seq":1361,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:14:29.884401+0000","last_change":"2024-09-16T04:14:29.884456+0000","last_active":"2024-09-16T04:14:29.884401+0000","last_peered":"2024-09-16T04:14:29.884401+0000","last_clean":"2024-09-16T04:14:29.884401+0000","last_became_active":"2024-09-16T03:43:34.381335+0000","last_became_peered":"2024-09-16T03:43:34.381335+0000","last_unstale":"2024-09-16T04:14:29.884401+0000","last_undegraded":"2024-09-16T04:14:29.884401+0000","last_fullsized":"2024-09-16T04:14:29.884401+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:14:29.884224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:14:29.884224+0000","last_clean_scrub_stamp":"2024-09-16T04:14:29.884224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:17:58.982998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1380,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:14:27.855027+0000","last_change":"2024-09-16T04:14:27.855078+0000","last_active":"2024-09-16T04:14:27.855027+0000","last_peered":"2024-09-16T04:14:27.855027+0000","last_clean":"2024-09-16T04:14:27.855027+0000","last_became_active":"2024-09-16T03:42:25.615293+0000","last_became_peered":"2024-09-16T03:42:25.615293+0000","last_unstale":"2024-09-16T04:14:27.855027+0000","last_undegraded":"2024-09-16T04:14:27.855027+0000","last_fullsized":"2024-09-16T04:14:27.855027+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:14:27.854861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:14:27.854861+0000","last_clean_scrub_stamp":"2024-09-16T04:14:27.854861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:33:12.804931+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0010681690000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1a","version":"0'0","reported_seq":1358,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:14:23.448842+0000","last_change":"2024-09-16T04:14:23.448878+0000","last_active":"2024-09-16T04:14:23.448842+0000","last_peered":"2024-09-16T04:14:23.448842+0000","last_clean":"2024-09-16T04:14:23.448842+0000","last_became_active":"2024-09-16T03:43:34.372307+0000","last_became_peered":"2024-09-16T03:43:34.372307+0000","last_unstale":"2024-09-16T04:14:23.448842+0000","last_undegraded":"2024-09-16T04:14:23.448842+0000","last_fullsized":"2024-09-16T04:14:23.448842+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:14:23.448741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:14:23.448741+0000","last_clean_scrub_stamp":"2024-09-16T04:14:23.448741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:06:06.891036+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.8","version":"0'0","reported_seq":1360,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:14:24.425854+0000","last_change":"2024-09-16T04:14:24.425885+0000","last_active":"2024-09-16T04:14:24.425854+0000","last_peered":"2024-09-16T04:14:24.425854+0000","last_clean":"2024-09-16T04:14:24.425854+0000","last_became_active":"2024-09-16T03:43:34.378483+0000","last_became_peered":"2024-09-16T03:43:34.378483+0000","last_unstale":"2024-09-16T04:14:24.425854+0000","last_undegraded":"2024-09-16T04:14:24.425854+0000","last_fullsized":"2024-09-16T04:14:24.425854+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:14:24.425749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:14:24.425749+0000","last_clean_scrub_stamp":"2024-09-16T04:14:24.425749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:26:47.010250+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.9","version":"0'0","reported_seq":1383,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:14:27.083005+0000","last_change":"2024-09-16T04:14:27.083055+0000","last_active":"2024-09-16T04:14:27.083005+0000","last_peered":"2024-09-16T04:14:27.083005+0000","last_clean":"2024-09-16T04:14:27.083005+0000","last_became_active":"2024-09-16T03:43:34.378733+0000","last_became_peered":"2024-09-16T03:43:34.378733+0000","last_unstale":"2024-09-16T04:14:27.083005+0000","last_undegraded":"2024-09-16T04:14:27.083005+0000","last_fullsized":"2024-09-16T04:14:27.083005+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:14:27.082842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:14:27.082842+0000","last_clean_scrub_stamp":"2024-09-16T04:14:27.082842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:22:50.422365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.a","version":"0'0","reported_seq":1383,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:14:28.086372+0000","last_change":"2024-09-16T04:14:28.086423+0000","last_active":"2024-09-16T04:14:28.086372+0000","last_peered":"2024-09-16T04:14:28.086372+0000","last_clean":"2024-09-16T04:14:28.086372+0000","last_became_active":"2024-09-16T03:43:34.390049+0000","last_became_peered":"2024-09-16T03:43:34.390049+0000","last_unstale":"2024-09-16T04:14:28.086372+0000","last_undegraded":"2024-09-16T04:14:28.086372+0000","last_fullsized":"2024-09-16T04:14:28.086372+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:14:28.086203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:14:28.086203+0000","last_clean_scrub_stamp":"2024-09-16T04:14:28.086203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:52:52.459860+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.b","version":"0'0","reported_seq":1357,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:14:27.333061+0000","last_change":"2024-09-16T04:14:27.333120+0000","last_active":"2024-09-16T04:14:27.333061+0000","last_peered":"2024-09-16T04:14:27.333061+0000","last_clean":"2024-09-16T04:14:27.333061+0000","last_became_active":"2024-09-16T03:43:34.328425+0000","last_became_peered":"2024-09-16T03:43:34.328425+0000","last_unstale":"2024-09-16T04:14:27.333061+0000","last_undegraded":"2024-09-16T04:14:27.333061+0000","last_fullsized":"2024-09-16T04:14:27.333061+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:14:27.332894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:14:27.332894+0000","last_clean_scrub_stamp":"2024-09-16T04:14:27.332894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:08:51.784987+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.4","version":"0'0","reported_seq":1360,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:14:26.419668+0000","last_change":"2024-09-16T04:14:26.419722+0000","last_active":"2024-09-16T04:14:26.419668+0000","last_peered":"2024-09-16T04:14:26.419668+0000","last_clean":"2024-09-16T04:14:26.419668+0000","last_became_active":"2024-09-16T03:43:34.391868+0000","last_became_peered":"2024-09-16T03:43:34.391868+0000","last_unstale":"2024-09-16T04:14:26.419668+0000","last_undegraded":"2024-09-16T04:14:26.419668+0000","last_fullsized":"2024-09-16T04:14:26.419668+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:14:26.419485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:14:26.419485+0000","last_clean_scrub_stamp":"2024-09-16T04:14:26.419485+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:47:38.149533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.5","version":"0'0","reported_seq":1360,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:14:28.435346+0000","last_change":"2024-09-16T04:14:28.435399+0000","last_active":"2024-09-16T04:14:28.435346+0000","last_peered":"2024-09-16T04:14:28.435346+0000","last_clean":"2024-09-16T04:14:28.435346+0000","last_became_active":"2024-09-16T03:43:34.419410+0000","last_became_peered":"2024-09-16T03:43:34.419410+0000","last_unstale":"2024-09-16T04:14:28.435346+0000","last_undegraded":"2024-09-16T04:14:28.435346+0000","last_fullsized":"2024-09-16T04:14:28.435346+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:14:28.435182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:14:28.435182+0000","last_clean_scrub_stamp":"2024-09-16T04:14:28.435182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:54:01.616842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.6","version":"0'0","reported_seq":1383,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:14:26.076410+0000","last_change":"2024-09-16T04:14:26.076463+0000","last_active":"2024-09-16T04:14:26.076410+0000","last_peered":"2024-09-16T04:14:26.076410+0000","last_clean":"2024-09-16T04:14:26.076410+0000","last_became_active":"2024-09-16T03:43:34.381834+0000","last_became_peered":"2024-09-16T03:43:34.381834+0000","last_unstale":"2024-09-16T04:14:26.076410+0000","last_undegraded":"2024-09-16T04:14:26.076410+0000","last_fullsized":"2024-09-16T04:14:26.076410+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:14:26.076222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:14:26.076222+0000","last_clean_scrub_stamp":"2024-09-16T04:14:26.076222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:04:12.276594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.7","version":"0'0","reported_seq":1358,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:14:26.291563+0000","last_change":"2024-09-16T04:14:26.291616+0000","last_active":"2024-09-16T04:14:26.291563+0000","last_peered":"2024-09-16T04:14:26.291563+0000","last_clean":"2024-09-16T04:14:26.291563+0000","last_became_active":"2024-09-16T03:43:34.292756+0000","last_became_peered":"2024-09-16T03:43:34.292756+0000","last_unstale":"2024-09-16T04:14:26.291563+0000","last_undegraded":"2024-09-16T04:14:26.291563+0000","last_fullsized":"2024-09-16T04:14:26.291563+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:14:26.291394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:14:26.291394+0000","last_clean_scrub_stamp":"2024-09-16T04:14:26.291394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:42:24.796438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.0","version":"0'0","reported_seq":1364,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:14:28.843538+0000","last_change":"2024-09-16T04:14:28.843604+0000","last_active":"2024-09-16T04:14:28.843538+0000","last_peered":"2024-09-16T04:14:28.843538+0000","last_clean":"2024-09-16T04:14:28.843538+0000","last_became_active":"2024-09-16T03:43:34.351273+0000","last_became_peered":"2024-09-16T03:43:34.351273+0000","last_unstale":"2024-09-16T04:14:28.843538+0000","last_undegraded":"2024-09-16T04:14:28.843538+0000","last_fullsized":"2024-09-16T04:14:28.843538+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:14:28.843342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:14:28.843342+0000","last_clean_scrub_stamp":"2024-09-16T04:14:28.843342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:21:11.435499+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1","version":"710'8","reported_seq":1372,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:14:31.311910+0000","last_change":"2024-09-16T04:14:31.311971+0000","last_active":"2024-09-16T04:14:31.311910+0000","last_peered":"2024-09-16T04:14:31.311910+0000","last_clean":"2024-09-16T04:14:31.311910+0000","last_became_active":"2024-09-16T03:43:34.382744+0000","last_became_peered":"2024-09-16T03:43:34.382744+0000","last_unstale":"2024-09-16T04:14:31.311910+0000","last_undegraded":"2024-09-16T04:14:31.311910+0000","last_fullsized":"2024-09-16T04:14:31.311910+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"710'8","last_scrub_stamp":"2024-09-16T04:14:31.311738+0000","last_deep_scrub":"710'8","last_deep_scrub_stamp":"2024-09-16T04:14:31.311738+0000","last_clean_scrub_stamp":"2024-09-16T04:14:31.311738+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T09:34:09.018710+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.2","version":"0'0","reported_seq":1364,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:14:29.356524+0000","last_change":"2024-09-16T04:14:29.356578+0000","last_active":"2024-09-16T04:14:29.356524+0000","last_peered":"2024-09-16T04:14:29.356524+0000","last_clean":"2024-09-16T04:14:29.356524+0000","last_became_active":"2024-09-16T03:43:34.402014+0000","last_became_peered":"2024-09-16T03:43:34.402014+0000","last_unstale":"2024-09-16T04:14:29.356524+0000","last_undegraded":"2024-09-16T04:14:29.356524+0000","last_fullsized":"2024-09-16T04:14:29.356524+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:14:29.356360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:14:29.356360+0000","last_clean_scrub_stamp":"2024-09-16T04:14:29.356360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T10:44:13.688133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.3","version":"0'0","reported_seq":1383,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-09-16T04:14:25.032038+0000","last_change":"2024-09-16T04:14:25.032090+0000","last_active":"2024-09-16T04:14:25.032038+0000","last_peered":"2024-09-16T04:14:25.032038+0000","last_clean":"2024-09-16T04:14:25.032038+0000","last_became_active":"2024-09-16T03:43:34.379988+0000","last_became_peered":"2024-09-16T03:43:34.379988+0000","last_unstale":"2024-09-16T04:14:25.032038+0000","last_undegraded":"2024-09-16T04:14:25.032038+0000","last_fullsized":"2024-09-16T04:14:25.032038+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T04:14:25.031862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T04:14:25.031862+0000","last_clean_scrub_stamp":"2024-09-16T04:14:25.031862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:35:24.329947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":31,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":64,"acting":64,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":1057856,"data_compressed":18838,"data_compressed_allocated":65536,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":12,"seq":51539607948,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":71044,"kb_used_data":3764,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94300796,"statfs":{"total":96636764160,"available":96564015104,"internally_reserved":0,"allocated":3854336,"data_stored":1504358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":11,"seq":47244640651,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":71044,"kb_used_data":3764,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94300796,"statfs":{"total":96636764160,"available":96564015104,"internally_reserved":0,"allocated":3854336,"data_stored":1504344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640648,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66932,"kb_used_data":3812,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94304908,"statfs":{"total":96636764160,"available":96568225792,"internally_reserved":0,"allocated":3903488,"data_stored":2033267,"data_compressed":9419,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1593,"internal_metadata":64616903},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640649,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":71040,"kb_used_data":3760,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94300800,"statfs":{"total":96636764160,"available":96564019200,"internally_reserved":0,"allocated":3850240,"data_stored":1504339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":68876742},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640650,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":79368,"kb_used_data":3768,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94292472,"statfs":{"total":96636764160,"available":96555491328,"internally_reserved":0,"allocated":3858432,"data_stored":1504363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640649,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":71092,"kb_used_data":3812,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94300748,"statfs":{"total":96636764160,"available":96563965952,"internally_reserved":0,"allocated":3903488,"data_stored":2033267,"data_compressed":9419,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":9419,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":528928,"data_compressed":9419,"data_compressed_allocated":32768,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T04:14:46.009 DEBUG:teuthology.orchestra.run.smithi042:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-09-16T04:14:46.133 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.125+0000 7f7e9cac4640 10 monclient: get_monmap_and_config 2024-09-16T04:14:46.133 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.125+0000 7f7e9cac4640 10 monclient: build_initial_monmap 2024-09-16T04:14:46.133 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.125+0000 7f7e9cac4640 10 monclient: monmap: 2024-09-16T04:14:46.133 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:14:46.133 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:14:46.133 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T04:14:46.131649+0000 2024-09-16T04:14:46.133 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T04:14:46.131649+0000 2024-09-16T04:14:46.133 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:14:46.133 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:46.133 INFO:teuthology.orchestra.run.smithi042.stderr:0: v1:172.21.15.42:6789/0 mon.noname-a-legacy 2024-09-16T04:14:46.133 INFO:teuthology.orchestra.run.smithi042.stderr:1: v1:172.21.15.154:6789/0 mon.noname-c-legacy 2024-09-16T04:14:46.134 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-b 2024-09-16T04:14:46.134 INFO:teuthology.orchestra.run.smithi042.stderr:3: v2:172.21.15.42:3300/0 mon.noname-a 2024-09-16T04:14:46.134 INFO:teuthology.orchestra.run.smithi042.stderr:4: v2:172.21.15.154:3300/0 mon.noname-c 2024-09-16T04:14:46.134 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:46.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.129+0000 7f7e9cac4640 1 Processor -- start 2024-09-16T04:14:46.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.129+0000 7f7e9cac4640 1 -- start start 2024-09-16T04:14:46.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.129+0000 7f7e9cac4640 10 monclient: init 2024-09-16T04:14:46.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.129+0000 7f7e9cac4640 10 monclient: _reopen_session rank -1 2024-09-16T04:14:46.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.129+0000 7f7e9cac4640 10 monclient: _add_conns ranks=[4,2,3,1,0] 2024-09-16T04:14:46.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.129+0000 7f7e9cac4640 1 --2- >> v2:172.21.15.154:3300/0 conn(0x7f7e980a2d60 0x7f7e98134d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:46.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.129+0000 7f7e9cac4640 10 monclient(hunting): picked mon.noname-c con 0x7f7e980a2d60 addr v2:172.21.15.154:3300/0 2024-09-16T04:14:46.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.129+0000 7f7e9cac4640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7e98072d70 0x7f7e980a1330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:46.136 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.129+0000 7f7e9cac4640 10 monclient(hunting): picked mon.noname-b con 0x7f7e98072d70 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:14:46.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.129+0000 7f7e9cac4640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f7e980731d0 0x7f7e980a1870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:46.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.129+0000 7f7e9cac4640 10 monclient(hunting): picked mon.noname-a con 0x7f7e980731d0 addr v2:172.21.15.42:3300/0 2024-09-16T04:14:46.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.129+0000 7f7e9cac4640 10 monclient(hunting): start opening mon connection 2024-09-16T04:14:46.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.129+0000 7f7e9cac4640 1 -- --> v2:172.21.15.42:3300/0 -- mon_getmap magic: 0 -- 0x7f7e9809ee90 con 0x7f7e980731d0 2024-09-16T04:14:46.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.129+0000 7f7e9cac4640 10 monclient(hunting): start opening mon connection 2024-09-16T04:14:46.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.129+0000 7f7e9cac4640 1 -- --> v2:172.21.15.154:3300/0 -- mon_getmap magic: 0 -- 0x7f7e980a3140 con 0x7f7e980a2d60 2024-09-16T04:14:46.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.129+0000 7f7e9cac4640 10 monclient(hunting): start opening mon connection 2024-09-16T04:14:46.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.129+0000 7f7e9cac4640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f7e98135390 con 0x7f7e98072d70 2024-09-16T04:14:46.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.129+0000 7f7e9cac4640 10 monclient(hunting): _renew_subs 2024-09-16T04:14:46.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.129+0000 7f7e9cac4640 10 monclient(hunting): authenticate will time out at 2933.101304s 2024-09-16T04:14:46.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.129+0000 7f7e96d9d640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f7e980731d0 0x7f7e980a1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:46.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.129+0000 7f7e95d9b640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7e98072d70 0x7f7e980a1330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:46.137 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.129+0000 7f7e96d9d640 1 --2- >> v2:172.21.15.42:3300/0 conn(0x7f7e980731d0 0x7f7e980a1870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:52578/0 (socket says 172.21.15.42:52578) 2024-09-16T04:14:46.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.129+0000 7f7e96d9d640 1 -- 172.21.15.42:0/2029159993 learned_addr learned my addr 172.21.15.42:0/2029159993 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:14:46.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.129+0000 7f7e95d9b640 10 monclient(hunting): get_auth_request con 0x7f7e98072d70 auth_method 0 2024-09-16T04:14:46.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.129+0000 7f7e95d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:14:46.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.129+0000 7f7e95d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:46.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.129+0000 7f7e95d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:46.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.129+0000 7f7e96d9d640 10 monclient(hunting): get_auth_request con 0x7f7e980731d0 auth_method 0 2024-09-16T04:14:46.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.129+0000 7f7e96d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:14:46.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.129+0000 7f7e96d9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:46.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.129+0000 7f7e96d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:46.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.129+0000 7f7e96d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:14:46.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.129+0000 7f7e96d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:14:46.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.129+0000 7f7e96d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:14:46.138 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.129+0000 7f7e95d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:14:46.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.129+0000 7f7e95d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:14:46.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.129+0000 7f7e95d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:14:46.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.129+0000 7f7e9659c640 1 --2- 172.21.15.42:0/2029159993 >> v2:172.21.15.154:3300/0 conn(0x7f7e980a2d60 0x7f7e98134d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:46.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.129+0000 7f7e9659c640 10 monclient(hunting): get_auth_request con 0x7f7e980a2d60 auth_method 0 2024-09-16T04:14:46.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.129+0000 7f7e9659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:14:46.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.129+0000 7f7e9659c640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:46.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.129+0000 7f7e9659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:46.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.129+0000 7f7e96d9d640 10 monclient(hunting): handle_auth_done global_id 1104252 payload 274 2024-09-16T04:14:46.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.129+0000 7f7e96d9d640 1 -- 172.21.15.42:0/2029159993 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7e98072d70 msgr2=0x7f7e980a1330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:14:46.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.129+0000 7f7e96d9d640 1 --2- 172.21.15.42:0/2029159993 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7e98072d70 0x7f7e980a1330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:46.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.129+0000 7f7e96d9d640 1 -- 172.21.15.42:0/2029159993 >> v2:172.21.15.154:3300/0 conn(0x7f7e980a2d60 msgr2=0x7f7e98134d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-16T04:14:46.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.129+0000 7f7e96d9d640 1 --2- 172.21.15.42:0/2029159993 >> v2:172.21.15.154:3300/0 conn(0x7f7e980a2d60 0x7f7e98134d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:46.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.129+0000 7f7e96d9d640 10 monclient: _finish_hunting 0 2024-09-16T04:14:46.139 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.129+0000 7f7e96d9d640 1 monclient: found mon.noname-a 2024-09-16T04:14:46.140 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.129+0000 7f7e96d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:14:46.140 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.129+0000 7f7e96d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.42:3300/0 2024-09-16T04:14:46.140 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.129+0000 7f7e96d9d640 1 -- 172.21.15.42:0/2029159993 --> v2:172.21.15.42:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e9809fa40 con 0x7f7e980731d0 2024-09-16T04:14:46.145 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.133+0000 7f7e96d9d640 1 --2- 172.21.15.42:0/2029159993 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7e980731d0 0x7f7e980a1870 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f7e980a3330 tx=0x7f7e9001c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d4727b51fe4e22 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:14:46.145 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.133+0000 7f7e9559a640 1 -- 172.21.15.42:0/2029159993 <== mon.0 v2:172.21.15.42:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e900a5730 con 0x7f7e980731d0 2024-09-16T04:14:46.145 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.133+0000 7f7e9559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:46.145 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.133+0000 7f7e9559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-16T04:14:46.145 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.133+0000 7f7e9559a640 10 monclient: dump: 2024-09-16T04:14:46.145 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:14:46.146 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:14:46.146 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:46.146 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:46.146 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:46.146 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:46.146 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:14:46.146 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:14:46.146 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:14:46.146 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:46.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.133+0000 7f7e9559a640 10 monclient: _finish_auth 0 2024-09-16T04:14:46.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.133+0000 7f7e9559a640 10 monclient: _check_auth_tickets 2024-09-16T04:14:46.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.133+0000 7f7e9559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:14:46.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.133+0000 7f7e9559a640 1 -- 172.21.15.42:0/2029159993 <== mon.0 v2:172.21.15.42:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e900a58d0 con 0x7f7e980731d0 2024-09-16T04:14:46.146 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.133+0000 7f7e9559a640 10 monclient: handle_config config(0 keys) 2024-09-16T04:14:46.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.133+0000 7f7e9559a640 1 -- 172.21.15.42:0/2029159993 <== mon.0 v2:172.21.15.42:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e900a5ca0 con 0x7f7e980731d0 2024-09-16T04:14:46.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.133+0000 7f7e9cac4640 5 monclient: authenticate success, global_id 1104252 2024-09-16T04:14:46.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.133+0000 7f7e9cac4640 10 monclient: get_monmap_and_config success 2024-09-16T04:14:46.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.133+0000 7f7e9cac4640 4 set_mon_vals no callback set 2024-09-16T04:14:46.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.133+0000 7f7e9559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:46.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.133+0000 7f7e9559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-16T04:14:46.147 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.133+0000 7f7e9559a640 10 monclient: dump: 2024-09-16T04:14:46.147 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:14:46.147 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:14:46.147 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:46.147 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:46.147 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:46.147 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:46.147 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:14:46.147 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:14:46.148 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:14:46.148 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:46.148 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.133+0000 7f7e9cac4640 10 monclient: shutdown 2024-09-16T04:14:46.148 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.133+0000 7f7e9cac4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:14:46.148 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.133+0000 7f7e9cac4640 1 -- 172.21.15.42:0/2029159993 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7e980731d0 msgr2=0x7f7e980a1870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:46.148 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.133+0000 7f7e9cac4640 1 --2- 172.21.15.42:0/2029159993 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7e980731d0 0x7f7e980a1870 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f7e980a3330 tx=0x7f7e9001c9a0 comp rx=0 tx=0).stop 2024-09-16T04:14:46.148 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.133+0000 7f7e9cac4640 1 -- 172.21.15.42:0/2029159993 shutdown_connections 2024-09-16T04:14:46.148 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.133+0000 7f7e9cac4640 1 --2- 172.21.15.42:0/2029159993 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7e980731d0 0x7f7e980a1870 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:46.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.133+0000 7f7e9cac4640 1 --2- 172.21.15.42:0/2029159993 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7e98072d70 0x7f7e980a1330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:46.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.133+0000 7f7e9cac4640 1 --2- 172.21.15.42:0/2029159993 >> v2:172.21.15.154:3300/0 conn(0x7f7e980a2d60 0x7f7e98134d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:46.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.133+0000 7f7e9cac4640 1 -- 172.21.15.42:0/2029159993 >> 172.21.15.42:0/2029159993 conn(0x7f7e980a62b0 msgr2=0x7f7e980a2420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:14:46.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.133+0000 7f7e9cac4640 1 -- 172.21.15.42:0/2029159993 shutdown_connections 2024-09-16T04:14:46.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.133+0000 7f7e9cac4640 1 -- 172.21.15.42:0/2029159993 wait complete. 2024-09-16T04:14:46.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.133+0000 7f7e9cac4640 10 monclient: build_initial_monmap 2024-09-16T04:14:46.149 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.133+0000 7f7e9cac4640 10 monclient: monmap: 2024-09-16T04:14:46.149 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 0 2024-09-16T04:14:46.149 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-16T04:14:46.149 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 0.000000 2024-09-16T04:14:46.149 INFO:teuthology.orchestra.run.smithi042.stderr:created 0.000000 2024-09-16T04:14:46.149 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 0 (unknown) 2024-09-16T04:14:46.149 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:46.150 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.noname-a 2024-09-16T04:14:46.150 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.noname-b 2024-09-16T04:14:46.150 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.noname-c 2024-09-16T04:14:46.150 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:46.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.133+0000 7f7e9cac4640 1 librados: starting msgr at 2024-09-16T04:14:46.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.133+0000 7f7e9cac4640 1 librados: starting objecter 2024-09-16T04:14:46.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.133+0000 7f7e9cac4640 1 Processor -- start 2024-09-16T04:14:46.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.133+0000 7f7e9cac4640 1 -- start start 2024-09-16T04:14:46.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.133+0000 7f7e9cac4640 1 librados: setting wanted keys 2024-09-16T04:14:46.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.133+0000 7f7e9cac4640 1 librados: calling monclient init 2024-09-16T04:14:46.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.133+0000 7f7e9cac4640 10 monclient: init 2024-09-16T04:14:46.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.133+0000 7f7e9cac4640 10 monclient: _reopen_session rank -1 2024-09-16T04:14:46.150 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.133+0000 7f7e9cac4640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-16T04:14:46.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.133+0000 7f7e9cac4640 1 --2- >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7e98072d70 0x7f7e98138940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:46.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.133+0000 7f7e9cac4640 10 monclient(hunting): picked mon.noname-b con 0x7f7e98072d70 addr [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] 2024-09-16T04:14:46.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.133+0000 7f7e9cac4640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7e980731d0 0x7f7e98138e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:46.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.133+0000 7f7e9cac4640 10 monclient(hunting): picked mon.noname-a con 0x7f7e980731d0 addr [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] 2024-09-16T04:14:46.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.133+0000 7f7e9cac4640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7e980a2d60 0x7f7e981393c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:46.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.133+0000 7f7e9cac4640 10 monclient(hunting): picked mon.noname-c con 0x7f7e980a2d60 addr [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] 2024-09-16T04:14:46.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.133+0000 7f7e9cac4640 10 monclient(hunting): start opening mon connection 2024-09-16T04:14:46.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.133+0000 7f7e9cac4640 1 -- --> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] -- mon_getmap magic: 0 -- 0x7f7e9813ebf0 con 0x7f7e980731d0 2024-09-16T04:14:46.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.133+0000 7f7e9cac4640 10 monclient(hunting): start opening mon connection 2024-09-16T04:14:46.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.133+0000 7f7e9cac4640 1 -- --> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] -- mon_getmap magic: 0 -- 0x7f7e9813ea70 con 0x7f7e98072d70 2024-09-16T04:14:46.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.133+0000 7f7e9cac4640 10 monclient(hunting): start opening mon connection 2024-09-16T04:14:46.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.133+0000 7f7e9cac4640 1 -- --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_getmap magic: 0 -- 0x7f7e9813ed70 con 0x7f7e980a2d60 2024-09-16T04:14:46.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.133+0000 7f7e9cac4640 10 monclient(hunting): _renew_subs 2024-09-16T04:14:46.151 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.133+0000 7f7e9cac4640 10 monclient(hunting): authenticate will time out at 2933.106912s 2024-09-16T04:14:46.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.133+0000 7f7e95d9b640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7e980731d0 0x7f7e98138e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:46.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.133+0000 7f7e96d9d640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7e980a2d60 0x7f7e981393c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:46.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.133+0000 7f7e95d9b640 1 --2- >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7e980731d0 0x7f7e98138e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3300/0 says I am v2:172.21.15.42:52594/0 (socket says 172.21.15.42:52594) 2024-09-16T04:14:46.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.133+0000 7f7e95d9b640 1 -- 172.21.15.42:0/4261418941 learned_addr learned my addr 172.21.15.42:0/4261418941 (peer_addr_for_me v2:172.21.15.42:0/0) 2024-09-16T04:14:46.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.133+0000 7f7e96d9d640 1 --2- >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7e980a2d60 0x7f7e981393c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.42:3301/0 says I am v2:172.21.15.42:46392/0 (socket says 172.21.15.42:46392) 2024-09-16T04:14:46.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.133+0000 7f7e95d9b640 10 monclient(hunting): get_auth_request con 0x7f7e980731d0 auth_method 0 2024-09-16T04:14:46.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.133+0000 7f7e95d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:14:46.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.133+0000 7f7e95d9b640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:46.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.133+0000 7f7e95d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:46.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.133+0000 7f7e96d9d640 10 monclient(hunting): get_auth_request con 0x7f7e980a2d60 auth_method 0 2024-09-16T04:14:46.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.133+0000 7f7e96d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-16T04:14:46.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.133+0000 7f7e96d9d640 10 monclient(hunting): _init_auth method 2 2024-09-16T04:14:46.152 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.133+0000 7f7e96d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-16T04:14:46.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.133+0000 7f7e96d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:14:46.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.133+0000 7f7e96d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:14:46.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.133+0000 7f7e96d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:14:46.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.133+0000 7f7e95d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-16T04:14:46.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.133+0000 7f7e95d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-16T04:14:46.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.133+0000 7f7e95d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-16T04:14:46.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.133+0000 7f7e9659c640 1 --2- 172.21.15.42:0/4261418941 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7e98072d70 0x7f7e98138940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:46.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.133+0000 7f7e96d9d640 10 monclient(hunting): handle_auth_done global_id 1094312 payload 824 2024-09-16T04:14:46.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.133+0000 7f7e96d9d640 1 -- 172.21.15.42:0/4261418941 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7e98072d70 msgr2=0x7f7e98138940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:46.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.133+0000 7f7e96d9d640 1 --2- 172.21.15.42:0/4261418941 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7e98072d70 0x7f7e98138940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:46.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.133+0000 7f7e96d9d640 1 -- 172.21.15.42:0/4261418941 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7e980731d0 msgr2=0x7f7e98138e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:46.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.133+0000 7f7e96d9d640 1 --2- 172.21.15.42:0/4261418941 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7e980731d0 0x7f7e98138e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:46.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.133+0000 7f7e96d9d640 10 monclient: _finish_hunting 0 2024-09-16T04:14:46.153 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.133+0000 7f7e96d9d640 1 monclient: found mon.noname-c 2024-09-16T04:14:46.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.133+0000 7f7e96d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-16T04:14:46.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.133+0000 7f7e96d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.42:3301/0 2024-09-16T04:14:46.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.133+0000 7f7e96d9d640 1 -- 172.21.15.42:0/4261418941 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e9813aee0 con 0x7f7e980a2d60 2024-09-16T04:14:46.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.133+0000 7f7e9659c640 10 monclient: get_auth_request con 0x7f7e98072d70 auth_method 0 2024-09-16T04:14:46.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.133+0000 7f7e9659c640 1 --2- 172.21.15.42:0/4261418941 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7e98072d70 0x7f7e98138940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-16T04:14:46.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.133+0000 7f7e95d9b640 1 --2- 172.21.15.42:0/4261418941 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7e980731d0 0x7f7e98138e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-16T04:14:46.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.133+0000 7f7e96d9d640 1 --2- 172.21.15.42:0/4261418941 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7e980a2d60 0x7f7e981393c0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f7e900ab000 tx=0x7f7e90019740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:14:46.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.133+0000 7f7e86ffd640 10 client.?.objecter ms_handle_connect 0x7f7e980a2d60 2024-09-16T04:14:46.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.133+0000 7f7e86ffd640 10 client.?.objecter resend_mon_ops 2024-09-16T04:14:46.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.133+0000 7f7e86ffd640 1 -- 172.21.15.42:0/4261418941 <== mon.2 v2:172.21.15.42:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e900bc030 con 0x7f7e980a2d60 2024-09-16T04:14:46.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.133+0000 7f7e86ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:46.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.133+0000 7f7e86ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-16T04:14:46.154 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.133+0000 7f7e86ffd640 10 monclient: dump: 2024-09-16T04:14:46.155 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:14:46.155 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:14:46.155 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:46.155 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:46.155 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:46.155 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:46.155 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:14:46.155 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:14:46.155 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:14:46.155 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:46.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.133+0000 7f7e86ffd640 10 monclient: _finish_auth 0 2024-09-16T04:14:46.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.133+0000 7f7e86ffd640 10 monclient: _check_auth_tickets 2024-09-16T04:14:46.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.133+0000 7f7e86ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-16T04:14:46.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.133+0000 7f7e86ffd640 1 -- 172.21.15.42:0/4261418941 <== mon.2 v2:172.21.15.42:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e900b8470 con 0x7f7e980a2d60 2024-09-16T04:14:46.155 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.133+0000 7f7e9cac4640 5 monclient: authenticate success, global_id 1094312 2024-09-16T04:14:46.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.133+0000 7f7e9cac4640 10 monclient: _renew_subs 2024-09-16T04:14:46.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.133+0000 7f7e9cac4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:14:46.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.133+0000 7f7e9cac4640 1 -- 172.21.15.42:0/4261418941 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e9811c710 con 0x7f7e980a2d60 2024-09-16T04:14:46.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.133+0000 7f7e86ffd640 10 monclient: handle_config config(0 keys) 2024-09-16T04:14:46.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.133+0000 7f7e86ffd640 1 -- 172.21.15.42:0/4261418941 <== mon.2 v2:172.21.15.42:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e90009510 con 0x7f7e980a2d60 2024-09-16T04:14:46.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.133+0000 7f7e86ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-16T04:14:46.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.133+0000 7f7e86ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-16T04:14:46.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.133+0000 7f7e9cac4640 10 client.1094312.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-16T04:14:46.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.133+0000 7f7e94d99640 4 set_mon_vals no callback set 2024-09-16T04:14:46.156 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.133+0000 7f7e86ffd640 10 monclient: dump: 2024-09-16T04:14:46.156 INFO:teuthology.orchestra.run.smithi042.stderr:epoch 1 2024-09-16T04:14:46.156 INFO:teuthology.orchestra.run.smithi042.stderr:fsid 15853af6-d0b2-472c-b2ac-8d7ad428641f 2024-09-16T04:14:46.156 INFO:teuthology.orchestra.run.smithi042.stderr:last_changed 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:46.156 INFO:teuthology.orchestra.run.smithi042.stderr:created 2024-09-16T03:41:35.824079+0000 2024-09-16T04:14:46.157 INFO:teuthology.orchestra.run.smithi042.stderr:min_mon_release 19 (squid) 2024-09-16T04:14:46.157 INFO:teuthology.orchestra.run.smithi042.stderr:election_strategy: 1 2024-09-16T04:14:46.157 INFO:teuthology.orchestra.run.smithi042.stderr:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-09-16T04:14:46.157 INFO:teuthology.orchestra.run.smithi042.stderr:1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-16T04:14:46.157 INFO:teuthology.orchestra.run.smithi042.stderr:2: [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] mon.c 2024-09-16T04:14:46.157 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:46.157 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.133+0000 7f7e9cac4640 10 monclient: _renew_subs 2024-09-16T04:14:46.157 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.133+0000 7f7e9cac4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:14:46.157 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.133+0000 7f7e9cac4640 1 -- 172.21.15.42:0/4261418941 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e9811c150 con 0x7f7e980a2d60 2024-09-16T04:14:46.157 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.133+0000 7f7e9cac4640 1 librados: init done 2024-09-16T04:14:46.157 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.137+0000 7f7e9cac4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-16T04:14:46.157 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.137+0000 7f7e86ffd640 1 -- 172.21.15.42:0/4261418941 <== mon.2 v2:172.21.15.42:3301/0 4 ==== mgrmap(e 5) ==== 44018+0+0 (secure 0 0 0) 0x7f7e900bf070 con 0x7f7e980a2d60 2024-09-16T04:14:46.157 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.137+0000 7f7e9cac4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:14:46.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.137+0000 7f7e9cac4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:14:46.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.137+0000 7f7e9cac4640 1 -- 172.21.15.42:0/4261418941 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e98134d10 con 0x7f7e980a2d60 2024-09-16T04:14:46.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.137+0000 7f7e86ffd640 1 --2- 172.21.15.42:0/4261418941 >> v2:172.21.15.154:6812/254845342 conn(0x7f7e78037810 0x7f7e78039cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-16T04:14:46.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.137+0000 7f7e86ffd640 1 -- 172.21.15.42:0/4261418941 <== mon.2 v2:172.21.15.42:3301/0 5 ==== osd_map(710..710 src has 681..710) ==== 4681+0+0 (secure 0 0 0) 0x7f7e900e3160 con 0x7f7e980a2d60 2024-09-16T04:14:46.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.137+0000 7f7e86ffd640 10 client.1094312.objecter ms_dispatch 0x7f7e98000ba0 osd_map(710..710 src has 681..710) 2024-09-16T04:14:46.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.137+0000 7f7e86ffd640 3 client.1094312.objecter handle_osd_map got epochs [710,710] > 0 2024-09-16T04:14:46.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.137+0000 7f7e86ffd640 3 client.1094312.objecter handle_osd_map decoding full epoch 710 2024-09-16T04:14:46.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.137+0000 7f7e86ffd640 20 client.1094312.objecter dump_active .. 0 homeless 2024-09-16T04:14:46.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.137+0000 7f7e9659c640 1 --2- 172.21.15.42:0/4261418941 >> v2:172.21.15.154:6812/254845342 conn(0x7f7e78037810 0x7f7e78039cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-16T04:14:46.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.137+0000 7f7e9659c640 10 monclient: get_auth_request con 0x7f7e78037810 auth_method 0 2024-09-16T04:14:46.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.137+0000 7f7e86ffd640 10 client.1094312.objecter ms_handle_connect 0x7f7e78037810 2024-09-16T04:14:46.158 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.137+0000 7f7e9659c640 1 --2- 172.21.15.42:0/4261418941 >> v2:172.21.15.154:6812/254845342 conn(0x7f7e78037810 0x7f7e78039cd0 secure :-1 s=READY pgs=4029 cs=0 l=1 rev1=1 crypto rx=0x7f7e880097c0 tx=0x7f7e88008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-16T04:14:46.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.141+0000 7f7e86ffd640 1 -- 172.21.15.42:0/4261418941 <== mon.2 v2:172.21.15.42:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+180250 (secure 0 0 0) 0x7f7e900c7020 con 0x7f7e980a2d60 2024-09-16T04:14:46.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.141+0000 7f7e86ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-16T04:14:46.159 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.141+0000 7f7e86ffd640 10 monclient: _finish_command 1 = system:0 2024-09-16T04:14:46.313 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.309+0000 7f7e9cac4640 10 start_mon_command cmd=[{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-09-16T04:14:46.314 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.309+0000 7f7e9cac4640 10 monclient: _send_command 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-09-16T04:14:46.314 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.309+0000 7f7e9cac4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.42:3301/0 2024-09-16T04:14:46.314 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.309+0000 7f7e9cac4640 1 -- 172.21.15.42:0/4261418941 --> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) -- 0x7f7e98131e30 con 0x7f7e980a2d60 2024-09-16T04:14:46.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.313+0000 7f7e86ffd640 1 -- 172.21.15.42:0/4261418941 <== mon.2 v2:172.21.15.42:3301/0 7 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f7e900ad5b0 con 0x7f7e980a2d60 2024-09-16T04:14:46.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.313+0000 7f7e86ffd640 10 monclient: handle_config config(1 keys) 2024-09-16T04:14:46.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.313+0000 7f7e94d99640 4 set_mon_vals no callback set 2024-09-16T04:14:46.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.313+0000 7f7e94d99640 10 set_mon_vals mon_health_to_clog = false 2024-09-16T04:14:46.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.313+0000 7f7e86ffd640 1 -- 172.21.15.42:0/4261418941 <== mon.2 v2:172.21.15.42:3301/0 8 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v4)=0 v4) ==== 123+0+0 (secure 0 0 0) 0x7f7e900165c0 con 0x7f7e980a2d60 2024-09-16T04:14:46.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.313+0000 7f7e86ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-09-16T04:14:46.319 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.313+0000 7f7e86ffd640 10 monclient: _finish_command 2 = system:0 2024-09-16T04:14:46.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.317+0000 7f7e9cac4640 10 librados: watch_flush enter 2024-09-16T04:14:46.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.317+0000 7f7e9cac4640 10 librados: watch_flush exit 2024-09-16T04:14:46.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.317+0000 7f7e9cac4640 20 client.1094312.objecter shutdown clearing up homeless session... 2024-09-16T04:14:46.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.317+0000 7f7e9cac4640 10 client.1094312.objecter successfully canceled tick 2024-09-16T04:14:46.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.317+0000 7f7e9cac4640 1 -- 172.21.15.42:0/4261418941 >> v2:172.21.15.154:6812/254845342 conn(0x7f7e78037810 msgr2=0x7f7e78039cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:46.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.317+0000 7f7e9cac4640 1 --2- 172.21.15.42:0/4261418941 >> v2:172.21.15.154:6812/254845342 conn(0x7f7e78037810 0x7f7e78039cd0 secure :-1 s=READY pgs=4029 cs=0 l=1 rev1=1 crypto rx=0x7f7e880097c0 tx=0x7f7e88008040 comp rx=0 tx=0).stop 2024-09-16T04:14:46.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.317+0000 7f7e9cac4640 10 monclient: shutdown 2024-09-16T04:14:46.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.317+0000 7f7e9cac4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-16T04:14:46.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.317+0000 7f7e9cac4640 1 -- 172.21.15.42:0/4261418941 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7e980a2d60 msgr2=0x7f7e981393c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-16T04:14:46.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.317+0000 7f7e9cac4640 1 --2- 172.21.15.42:0/4261418941 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7e980a2d60 0x7f7e981393c0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f7e900ab000 tx=0x7f7e90019740 comp rx=0 tx=0).stop 2024-09-16T04:14:46.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.317+0000 7f7e9cac4640 1 -- 172.21.15.42:0/4261418941 shutdown_connections 2024-09-16T04:14:46.323 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.317+0000 7f7e9cac4640 1 --2- 172.21.15.42:0/4261418941 >> v2:172.21.15.154:6812/254845342 conn(0x7f7e78037810 0x7f7e78039cd0 unknown :-1 s=CLOSED pgs=4029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:46.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.317+0000 7f7e9cac4640 1 --2- 172.21.15.42:0/4261418941 >> [v2:172.21.15.42:3301/0,v1:172.21.15.42:6790/0] conn(0x7f7e980a2d60 0x7f7e981393c0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:46.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.317+0000 7f7e9cac4640 1 --2- 172.21.15.42:0/4261418941 >> [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] conn(0x7f7e980731d0 0x7f7e98138e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:46.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.317+0000 7f7e9cac4640 1 --2- 172.21.15.42:0/4261418941 >> [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] conn(0x7f7e98072d70 0x7f7e98138940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-16T04:14:46.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.317+0000 7f7e9cac4640 1 -- 172.21.15.42:0/4261418941 >> 172.21.15.42:0/4261418941 conn(0x7f7e980a62b0 msgr2=0x7f7e980720c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-16T04:14:46.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.317+0000 7f7e9cac4640 1 -- 172.21.15.42:0/4261418941 shutdown_connections 2024-09-16T04:14:46.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.317+0000 7f7e9cac4640 1 -- 172.21.15.42:0/4261418941 wait complete. 2024-09-16T04:14:46.324 INFO:teuthology.orchestra.run.smithi042.stderr:2024-09-16T04:14:46.317+0000 7f7e9cac4640 1 librados: shutdown 2024-09-16T04:14:46.339 INFO:teuthology.misc:Shutting down mds daemons... 2024-09-16T04:14:46.339 INFO:teuthology.misc:Shutting down osd daemons... 2024-09-16T04:14:46.339 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-09-16T04:14:46.339 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T04:14:46.409 INFO:tasks.ceph.osd.0:Stopped 2024-09-16T04:14:46.409 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-09-16T04:14:46.409 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T04:14:46.489 INFO:tasks.ceph.osd.1:Stopped 2024-09-16T04:14:46.490 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-09-16T04:14:46.490 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T04:14:46.570 INFO:tasks.ceph.osd.2:Stopped 2024-09-16T04:14:46.571 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-09-16T04:14:46.571 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T04:14:46.671 INFO:tasks.ceph.osd.3:Stopped 2024-09-16T04:14:46.672 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-09-16T04:14:46.672 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T04:14:50.983 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~0s 2024-09-16T04:14:52.774 INFO:tasks.ceph.osd.4:Stopped 2024-09-16T04:14:52.774 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-09-16T04:14:52.774 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T04:14:52.862 INFO:tasks.ceph.osd.5:Stopped 2024-09-16T04:14:52.862 INFO:teuthology.misc:Shutting down mgr daemons... 2024-09-16T04:14:52.862 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-09-16T04:14:52.862 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T04:14:52.893 INFO:tasks.ceph.mgr.x:Stopped 2024-09-16T04:14:52.893 INFO:teuthology.misc:Shutting down mon daemons... 2024-09-16T04:14:52.893 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-16T04:14:52.893 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T04:14:52.920 INFO:tasks.ceph.mon.a:Stopped 2024-09-16T04:14:52.921 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-16T04:14:52.921 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T04:14:52.987 INFO:tasks.ceph.mon.c:Stopped 2024-09-16T04:14:52.987 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-16T04:14:52.988 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T04:14:53.013 INFO:tasks.ceph.mon.b:Stopped 2024-09-16T04:14:53.014 INFO:tasks.ceph:Checking cluster log for badness... 2024-09-16T04:14:53.014 DEBUG:teuthology.orchestra.run.smithi042:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v ScrubResult | egrep -v 'scrub mismatch' | egrep -v 'overall HEALTH_' | egrep -v '\(MGR_DOWN\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'overall HEALTH_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(MON_DOWN\)' | egrep -v '\(PG_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v 'slow request' | head -n 1 2024-09-16T04:14:53.055 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi042.front.sepia.ceph.com 2024-09-16T04:14:53.056 DEBUG:teuthology.orchestra.run.smithi042:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-09-16T04:14:53.283 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi042.front.sepia.ceph.com 2024-09-16T04:14:53.283 DEBUG:teuthology.orchestra.run.smithi042:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-09-16T04:14:53.510 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi042.front.sepia.ceph.com 2024-09-16T04:14:53.510 DEBUG:teuthology.orchestra.run.smithi042:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-09-16T04:14:53.702 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi154.front.sepia.ceph.com 2024-09-16T04:14:53.702 DEBUG:teuthology.orchestra.run.smithi154:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-09-16T04:14:53.864 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi154.front.sepia.ceph.com 2024-09-16T04:14:53.864 DEBUG:teuthology.orchestra.run.smithi154:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-09-16T04:14:54.033 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi154.front.sepia.ceph.com 2024-09-16T04:14:54.033 DEBUG:teuthology.orchestra.run.smithi154:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-09-16T04:14:54.242 INFO:tasks.ceph:Archiving mon data... 2024-09-16T04:14:54.243 DEBUG:teuthology.misc:Transferring archived files from smithi042:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7905897/data/mon.a.tgz 2024-09-16T04:14:54.243 DEBUG:teuthology.orchestra.run.smithi042:> mktemp 2024-09-16T04:14:54.249 INFO:teuthology.orchestra.run.smithi042.stdout:/tmp/tmp.dOD1DEpVrE 2024-09-16T04:14:54.250 DEBUG:teuthology.orchestra.run.smithi042:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.dOD1DEpVrE 2024-09-16T04:14:54.508 DEBUG:teuthology.orchestra.run.smithi042:> sudo chmod 0666 /tmp/tmp.dOD1DEpVrE 2024-09-16T04:14:54.574 DEBUG:teuthology.orchestra.remote:smithi042:/tmp/tmp.dOD1DEpVrE is 1MB 2024-09-16T04:14:54.652 DEBUG:teuthology.orchestra.run.smithi042:> rm -fr /tmp/tmp.dOD1DEpVrE 2024-09-16T04:14:54.658 DEBUG:teuthology.misc:Transferring archived files from smithi042:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7905897/data/mon.c.tgz 2024-09-16T04:14:54.658 DEBUG:teuthology.orchestra.run.smithi042:> mktemp 2024-09-16T04:14:54.706 INFO:teuthology.orchestra.run.smithi042.stdout:/tmp/tmp.3yrkP2mBxT 2024-09-16T04:14:54.706 DEBUG:teuthology.orchestra.run.smithi042:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.3yrkP2mBxT 2024-09-16T04:14:54.974 DEBUG:teuthology.orchestra.run.smithi042:> sudo chmod 0666 /tmp/tmp.3yrkP2mBxT 2024-09-16T04:14:55.042 DEBUG:teuthology.orchestra.remote:smithi042:/tmp/tmp.3yrkP2mBxT is 1MB 2024-09-16T04:14:55.112 DEBUG:teuthology.orchestra.run.smithi042:> rm -fr /tmp/tmp.3yrkP2mBxT 2024-09-16T04:14:55.118 DEBUG:teuthology.misc:Transferring archived files from smithi154:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7905897/data/mon.b.tgz 2024-09-16T04:14:55.118 DEBUG:teuthology.orchestra.run.smithi154:> mktemp 2024-09-16T04:14:55.124 INFO:teuthology.orchestra.run.smithi154.stdout:/tmp/tmp.AKahuZT2ee 2024-09-16T04:14:55.124 DEBUG:teuthology.orchestra.run.smithi154:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.AKahuZT2ee 2024-09-16T04:14:55.373 DEBUG:teuthology.orchestra.run.smithi154:> sudo chmod 0666 /tmp/tmp.AKahuZT2ee 2024-09-16T04:14:55.437 DEBUG:teuthology.orchestra.remote:smithi154:/tmp/tmp.AKahuZT2ee is 1MB 2024-09-16T04:14:55.507 DEBUG:teuthology.orchestra.run.smithi154:> rm -fr /tmp/tmp.AKahuZT2ee 2024-09-16T04:14:55.513 INFO:tasks.ceph:Cleaning ceph cluster... 2024-09-16T04:14:55.514 DEBUG:teuthology.orchestra.run.smithi042:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-16T04:14:55.516 DEBUG:teuthology.orchestra.run.smithi154:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-16T04:14:55.577 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-16T04:14:55.618 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-16T04:14:55.618 INFO:tasks.ceph:Archiving crash dumps... 2024-09-16T04:14:55.620 DEBUG:teuthology.misc:Transferring archived files from smithi042:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7905897/remote/smithi042/crash 2024-09-16T04:14:55.621 DEBUG:teuthology.orchestra.run.smithi042:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-16T04:14:55.635 DEBUG:teuthology.misc:Transferring archived files from smithi154:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7905897/remote/smithi154/crash 2024-09-16T04:14:55.636 DEBUG:teuthology.orchestra.run.smithi154:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-16T04:14:55.673 INFO:tasks.ceph:Compressing logs... 2024-09-16T04:14:55.673 DEBUG:teuthology.orchestra.run.smithi042:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-16T04:14:55.682 DEBUG:teuthology.orchestra.run.smithi154:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-16T04:14:55.709 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41507.log 2024-09-16T04:14:55.709 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43050.log 2024-09-16T04:14:55.709 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.41507.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28834.log 2024-09-16T04:14:55.710 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.43050.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30604.log 2024-09-16T04:14:55.710 INFO:teuthology.orchestra.run.smithi042.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.41507.log.gz 2024-09-16T04:14:55.711 INFO:teuthology.orchestra.run.smithi042.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.43050.log.gz 2024-09-16T04:14:55.711 INFO:teuthology.orchestra.run.smithi042.stderr:gzip/var/log/ceph/ceph-client.admin.28834.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.30832.log 2024-09-16T04:14:55.711 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.30604.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39935.log 2024-09-16T04:14:55.712 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.30832.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35395.log 2024-09-16T04:14:55.712 INFO:teuthology.orchestra.run.smithi042.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.30604.log.gz 2024-09-16T04:14:55.712 INFO:teuthology.orchestra.run.smithi042.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.28834.log.gz 2024-09-16T04:14:55.712 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38644.log 2024-09-16T04:14:55.714 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.39935.log: /var/log/ceph/ceph-client.admin.35395.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44742.log 2024-09-16T04:14:55.714 INFO:teuthology.orchestra.run.smithi042.stderr: 91.9% -- replaced with /var/log/ceph/ceph-client.admin.30832.log.gz 2024-09-16T04:14:55.714 INFO:teuthology.orchestra.run.smithi042.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.39935.log.gz 2024-09-16T04:14:55.714 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.38644.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.35395.log.gz 2024-09-16T04:14:55.714 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41299.log 2024-09-16T04:14:55.714 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.44742.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28419.log 2024-09-16T04:14:55.714 INFO:teuthology.orchestra.run.smithi042.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.38644.log.gz 2024-09-16T04:14:55.714 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.41299.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39031.log 2024-09-16T04:14:55.716 INFO:teuthology.orchestra.run.smithi042.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.44742.log.gz/var/log/ceph/ceph-client.admin.28419.log: 2024-09-16T04:14:55.717 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39313.log 2024-09-16T04:14:55.717 INFO:teuthology.orchestra.run.smithi042.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.41299.log.gz 2024-09-16T04:14:55.717 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.39031.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39803.log 2024-09-16T04:14:55.717 INFO:teuthology.orchestra.run.smithi042.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.28419.log.gz 2024-09-16T04:14:55.717 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.39313.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42972.log 2024-09-16T04:14:55.717 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.39803.log: gzip 88.3% -5 -- replaced with /var/log/ceph/ceph-client.admin.39031.log.gz --verbose 2024-09-16T04:14:55.717 INFO:teuthology.orchestra.run.smithi042.stderr: -- /var/log/ceph/ceph-client.admin.35015.log 2024-09-16T04:14:55.717 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.42972.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35921.log 2024-09-16T04:14:55.717 INFO:teuthology.orchestra.run.smithi042.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.39313.log.gz 2024-09-16T04:14:55.717 INFO:teuthology.orchestra.run.smithi042.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.39803.log.gz 2024-09-16T04:14:55.717 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.35015.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28029.log 2024-09-16T04:14:55.717 INFO:teuthology.orchestra.run.smithi042.stderr: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.42972.log.gz 2024-09-16T04:14:55.717 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.35921.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38358.log 2024-09-16T04:14:55.717 INFO:teuthology.orchestra.run.smithi042.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.35015.log.gz 2024-09-16T04:14:55.721 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32828.log 2024-09-16T04:14:55.721 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.28029.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.35921.log.gz 2024-09-16T04:14:55.721 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.38358.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32682.log 2024-09-16T04:14:55.721 INFO:teuthology.orchestra.run.smithi042.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.28029.log.gz 2024-09-16T04:14:55.721 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44794.log/var/log/ceph/ceph-client.admin.32828.log: 2024-09-16T04:14:55.722 INFO:teuthology.orchestra.run.smithi042.stderr: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.38358.log.gz 2024-09-16T04:14:55.722 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.32682.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34254.log 2024-09-16T04:14:55.722 INFO:teuthology.orchestra.run.smithi042.stderr:gzip/var/log/ceph/ceph-client.admin.44794.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.39909.log 2024-09-16T04:14:55.722 INFO:teuthology.orchestra.run.smithi042.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.32682.log.gz 2024-09-16T04:14:55.722 INFO:teuthology.orchestra.run.smithi042.stderr: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.32828.log.gz 2024-09-16T04:14:55.722 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.34254.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31550.log 2024-09-16T04:14:55.722 INFO:teuthology.orchestra.run.smithi042.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.44794.log.gzgzip 2024-09-16T04:14:55.722 INFO:teuthology.orchestra.run.smithi042.stderr: -5 --verbose -- 87.8% /var/log/ceph/ceph-client.admin.40229.log -- replaced with /var/log/ceph/ceph-client.admin.34254.log.gz 2024-09-16T04:14:55.722 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:55.722 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.39909.log: /var/log/ceph/ceph-client.admin.31550.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42218.log 2024-09-16T04:14:55.722 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38410.log 2024-09-16T04:14:55.722 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.40229.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.39909.log.gz 2024-09-16T04:14:55.722 INFO:teuthology.orchestra.run.smithi042.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.31550.log.gz 2024-09-16T04:14:55.722 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.42218.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38072.log 2024-09-16T04:14:55.723 INFO:teuthology.orchestra.run.smithi042.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.40229.log.gz 2024-09-16T04:14:55.723 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.38410.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41195.log 2024-09-16T04:14:55.723 INFO:teuthology.orchestra.run.smithi042.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.42218.log.gz 2024-09-16T04:14:55.723 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.38072.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37137.log 2024-09-16T04:14:55.723 INFO:teuthology.orchestra.run.smithi042.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.38410.log.gz 2024-09-16T04:14:55.731 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37305.log 2024-09-16T04:14:55.732 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.41195.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.38072.log.gz 2024-09-16T04:14:55.732 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.37137.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31075.log 2024-09-16T04:14:55.732 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29969.log 2024-09-16T04:14:55.732 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.37305.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.37137.log.gz 2024-09-16T04:14:55.732 INFO:teuthology.orchestra.run.smithi042.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.41195.log.gz 2024-09-16T04:14:55.732 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.31075.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43589.log 2024-09-16T04:14:55.732 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.29969.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.37305.log.gz 2024-09-16T04:14:55.732 INFO:teuthology.orchestra.run.smithi042.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.31075.log.gz 2024-09-16T04:14:55.732 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34150.log 2024-09-16T04:14:55.732 INFO:teuthology.orchestra.run.smithi042.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.29969.log.gz 2024-09-16T04:14:55.732 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27847.log/var/log/ceph/ceph-client.admin.43589.log: 2024-09-16T04:14:55.732 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.34150.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33013.log 2024-09-16T04:14:55.733 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.27847.log: gzip 88.9% -5 -- replaced with /var/log/ceph/ceph-client.admin.43589.log.gz --verbose 2024-09-16T04:14:55.733 INFO:teuthology.orchestra.run.smithi042.stderr: -- /var/log/ceph/ceph-client.admin.37683.log 2024-09-16T04:14:55.733 INFO:teuthology.orchestra.run.smithi042.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.34150.log.gz 2024-09-16T04:14:55.733 INFO:teuthology.orchestra.run.smithi042.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.27847.log.gz 2024-09-16T04:14:55.733 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.33013.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34496.log 2024-09-16T04:14:55.733 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.37683.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43285.log 2024-09-16T04:14:55.733 INFO:teuthology.orchestra.run.smithi042.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.37683.log.gz 2024-09-16T04:14:55.733 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.34496.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40864.log 2024-09-16T04:14:55.733 INFO:teuthology.orchestra.run.smithi042.stderr: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.33013.log.gz 2024-09-16T04:14:55.733 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.43285.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40195.log 2024-09-16T04:14:55.733 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.40864.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38098.log 2024-09-16T04:14:55.733 INFO:teuthology.orchestra.run.smithi042.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.34496.log.gz 2024-09-16T04:14:55.733 INFO:teuthology.orchestra.run.smithi042.stderr: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.43285.log.gz 2024-09-16T04:14:55.733 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.40195.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34334.log 2024-09-16T04:14:55.734 INFO:teuthology.orchestra.run.smithi042.stderr: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.40864.log.gz 2024-09-16T04:14:55.734 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38436.log 2024-09-16T04:14:55.734 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.38098.log: /var/log/ceph/ceph-client.admin.34334.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.40195.log.gz 2024-09-16T04:14:55.734 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39601.log 2024-09-16T04:14:55.734 INFO:teuthology.orchestra.run.smithi042.stderr: 89.1%/var/log/ceph/ceph-client.admin.38436.log: -- replaced with /var/log/ceph/ceph-client.admin.34334.log.gz 2024-09-16T04:14:55.734 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29999.log 2024-09-16T04:14:55.734 INFO:teuthology.orchestra.run.smithi042.stderr: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.38098.log.gz 2024-09-16T04:14:55.734 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34072.log 2024-09-16T04:14:55.734 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.39601.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.38436.log.gz 2024-09-16T04:14:55.734 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.29999.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32298.log 2024-09-16T04:14:55.734 INFO:teuthology.orchestra.run.smithi042.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.29999.log.gz 2024-09-16T04:14:55.734 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.34072.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28733.log 2024-09-16T04:14:55.734 INFO:teuthology.orchestra.run.smithi042.stderr: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.39601.log.gz 2024-09-16T04:14:55.734 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.32298.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35687.log 2024-09-16T04:14:55.735 INFO:teuthology.orchestra.run.smithi042.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.34072.log.gz 2024-09-16T04:14:55.735 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.28733.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42582.log 2024-09-16T04:14:55.735 INFO:teuthology.orchestra.run.smithi042.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.32298.log.gz 2024-09-16T04:14:55.735 INFO:teuthology.orchestra.run.smithi042.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.28733.log.gz 2024-09-16T04:14:55.735 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.35687.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30766.log 2024-09-16T04:14:55.735 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.42582.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36568.log 2024-09-16T04:14:55.735 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.30766.log: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.35687.log.gz 2024-09-16T04:14:55.735 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42166.log 2024-09-16T04:14:55.735 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.36568.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.42582.log.gz 2024-09-16T04:14:55.735 INFO:teuthology.orchestra.run.smithi042.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.30766.log.gz 2024-09-16T04:14:55.735 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30364.log 2024-09-16T04:14:55.758 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29286.log 2024-09-16T04:14:55.758 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28123.log 2024-09-16T04:14:55.758 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28160.log 2024-09-16T04:14:55.758 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.29286.log: /var/log/ceph/ceph-client.admin.28123.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28466.log 2024-09-16T04:14:55.758 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.28160.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28396.log 2024-09-16T04:14:55.758 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.28466.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29416.log 2024-09-16T04:14:55.758 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.28396.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29235.log 2024-09-16T04:14:55.758 INFO:teuthology.orchestra.run.smithi154.stderr:gzip/var/log/ceph/ceph-client.admin.29416.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.29233.log 2024-09-16T04:14:55.759 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.29235.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28537.log 2024-09-16T04:14:55.759 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.29233.log: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.29286.log.gz 2024-09-16T04:14:55.759 INFO:teuthology.orchestra.run.smithi154.stderr: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.28466.log.gz 2024-09-16T04:14:55.759 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28692.log 2024-09-16T04:14:55.759 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.28537.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29336.log 2024-09-16T04:14:55.759 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.28692.log: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.28160.log.gz 2024-09-16T04:14:55.759 INFO:teuthology.orchestra.run.smithi154.stderr: 91.6% -- replaced with /var/log/ceph/ceph-client.admin.29235.log.gz 91.1% -- replaced with /var/log/ceph/ceph-client.admin.29416.log.gz 2024-09-16T04:14:55.759 INFO:teuthology.orchestra.run.smithi154.stderr: 2024-09-16T04:14:55.759 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28469.log 2024-09-16T04:14:55.759 INFO:teuthology.orchestra.run.smithi154.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.28537.log.gz 2024-09-16T04:14:55.759 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.29336.log: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.29233.log.gz 2024-09-16T04:14:55.759 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25140.log 2024-09-16T04:14:55.759 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.28469.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25260.log 2024-09-16T04:14:55.759 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.25140.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29285.log 2024-09-16T04:14:55.760 INFO:teuthology.orchestra.run.smithi154.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.28469.log.gz/var/log/ceph/ceph-client.admin.25260.log: 2024-09-16T04:14:55.760 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28673.log 2024-09-16T04:14:55.760 INFO:teuthology.orchestra.run.smithi154.stderr: 87.8%/var/log/ceph/ceph-client.admin.29285.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.25140.log.gz 2024-09-16T04:14:55.760 INFO:teuthology.orchestra.run.smithi154.stderr: -- replaced with /var/log/ceph/ceph-client.admin.29336.log.gz 2024-09-16T04:14:55.760 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 88.3% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.25260.log.gz /var/log/ceph/ceph-client.admin.28353.log 2024-09-16T04:14:55.760 INFO:teuthology.orchestra.run.smithi154.stderr: 2024-09-16T04:14:55.760 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.28673.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28611.log 2024-09-16T04:14:55.760 INFO:teuthology.orchestra.run.smithi154.stderr: 94.0% -- replaced with /var/log/ceph/ceph-client.admin.28396.log.gz 2024-09-16T04:14:55.760 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.28353.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28620.log 2024-09-16T04:14:55.760 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.28611.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28091.log 2024-09-16T04:14:55.760 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.28620.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29338.log 2024-09-16T04:14:55.760 INFO:teuthology.orchestra.run.smithi154.stderr: 91.6% -- replaced with /var/log/ceph/ceph-client.admin.29285.log.gz 2024-09-16T04:14:55.761 INFO:teuthology.orchestra.run.smithi154.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.28673.log.gz 2024-09-16T04:14:55.761 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.28091.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29284.log 2024-09-16T04:14:55.761 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.29338.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28277.log 2024-09-16T04:14:55.761 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.29284.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.29338.log.gz 2024-09-16T04:14:55.761 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28563.log 2024-09-16T04:14:55.761 INFO:teuthology.orchestra.run.smithi154.stderr: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.29284.log.gz 2024-09-16T04:14:55.761 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.28277.log: 92.3% -- replaced with /var/log/ceph/ceph-client.admin.28353.log.gz 2024-09-16T04:14:55.761 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-09-16T04:14:55.761 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.28563.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-09-16T04:14:55.767 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-osd.3.log: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.28563.log.gz 2024-09-16T04:14:55.767 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28487.log 2024-09-16T04:14:55.767 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-osd.5.log: 92.2% -- replaced with /var/log/ceph/ceph-client.admin.28620.log.gz 2024-09-16T04:14:55.767 INFO:teuthology.orchestra.run.smithi154.stderr: 91.6% -- replaced with /var/log/ceph/ceph-client.admin.28277.log.gz 2024-09-16T04:14:55.767 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28198.log 2024-09-16T04:14:55.767 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.28487.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29715.log 2024-09-16T04:14:55.767 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.28198.log: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.28487.log.gz 2024-09-16T04:14:55.767 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28107.log 2024-09-16T04:14:55.775 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.29715.log: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.29715.log.gz 2024-09-16T04:14:55.775 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28369.log 2024-09-16T04:14:55.786 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.28107.log: 92.6% -- replaced with /var/log/ceph/ceph-client.admin.28611.log.gz 2024-09-16T04:14:55.786 INFO:teuthology.orchestra.run.smithi154.stderr: 92.7% -- replaced with /var/log/ceph/ceph-client.admin.28107.log.gz 2024-09-16T04:14:55.787 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25212.log 2024-09-16T04:14:55.787 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.28369.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29714.log 2024-09-16T04:14:55.787 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.25212.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.25212.log.gz 2024-09-16T04:14:55.787 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29411.log 2024-09-16T04:14:55.787 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.29714.log: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.28369.log.gz 2024-09-16T04:14:55.787 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28187.log 2024-09-16T04:14:55.787 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.29411.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.29411.log.gz 2024-09-16T04:14:55.787 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28359.log 2024-09-16T04:14:55.787 INFO:teuthology.orchestra.run.smithi154.stderr: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.29714.log.gz 2024-09-16T04:14:55.787 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.28187.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-09-16T04:14:55.797 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.28359.log: 92.9% -- replaced with /var/log/ceph/ceph-client.admin.28692.log.gz 2024-09-16T04:14:55.797 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28083.log 2024-09-16T04:14:55.797 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-mgr.x.log: 91.5% -- replaced with /var/log/ceph/ceph-client.admin.28359.log.gz 2024-09-16T04:14:55.797 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25188.log 2024-09-16T04:14:55.803 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.28083.log: 93.5% -- replaced with /var/log/ceph/ceph-client.admin.28123.log.gz 2024-09-16T04:14:55.803 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-16T04:14:55.810 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.25188.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.25188.log.gz 2024-09-16T04:14:55.810 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28240.log 2024-09-16T04:14:55.816 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.28091.log.gz 2024-09-16T04:14:55.816 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28443.log 2024-09-16T04:14:55.826 INFO:teuthology.orchestra.run.smithi042.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.36568.log.gz 2024-09-16T04:14:55.826 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-09-16T04:14:55.826 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.42166.log: /var/log/ceph/ceph-client.admin.30364.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44638.log 2024-09-16T04:14:55.826 INFO:teuthology.orchestra.run.smithi042.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.30364.log.gz 2024-09-16T04:14:55.826 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43076.log 2024-09-16T04:14:55.826 INFO:teuthology.orchestra.run.smithi042.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.42166.log.gz 2024-09-16T04:14:55.826 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.44638.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37942.log 2024-09-16T04:14:55.826 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.43076.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.44638.log.gz 2024-09-16T04:14:55.826 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43233.log 2024-09-16T04:14:55.826 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32934.log 2024-09-16T04:14:55.827 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.37942.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.43076.log.gz 2024-09-16T04:14:55.827 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.43233.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43433.log 2024-09-16T04:14:55.827 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.32934.log: 89.7%gzip -- replaced with /var/log/ceph/ceph-client.admin.43233.log.gz 2024-09-16T04:14:55.827 INFO:teuthology.orchestra.run.smithi042.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.36462.log 2024-09-16T04:14:55.827 INFO:teuthology.orchestra.run.smithi042.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.37942.log.gz 2024-09-16T04:14:55.827 INFO:teuthology.orchestra.run.smithi042.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.32934.log.gz 2024-09-16T04:14:55.827 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.43433.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33737.log 2024-09-16T04:14:55.827 INFO:teuthology.orchestra.run.smithi042.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.43433.log.gz 2024-09-16T04:14:55.827 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29834.log 2024-09-16T04:14:55.827 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.36462.log: /var/log/ceph/ceph-client.admin.33737.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40483.log 2024-09-16T04:14:55.827 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.29834.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35551.log 2024-09-16T04:14:55.827 INFO:teuthology.orchestra.run.smithi042.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.36462.log.gz 2024-09-16T04:14:55.827 INFO:teuthology.orchestra.run.smithi042.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.33737.log.gz 2024-09-16T04:14:55.827 INFO:teuthology.orchestra.run.smithi042.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.29834.log.gz 2024-09-16T04:14:55.828 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39110.log 2024-09-16T04:14:55.828 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.40483.log: /var/log/ceph/ceph-client.admin.35551.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35975.log 2024-09-16T04:14:55.828 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.39110.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33605.log 2024-09-16T04:14:55.828 INFO:teuthology.orchestra.run.smithi042.stderr: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.35551.log.gz 2024-09-16T04:14:55.828 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.35975.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.40483.log.gz 2024-09-16T04:14:55.828 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32100.log 2024-09-16T04:14:55.828 INFO:teuthology.orchestra.run.smithi042.stderr: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.39110.log.gz 2024-09-16T04:14:55.828 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.33605.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44534.log 2024-09-16T04:14:55.828 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.32100.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31834.log 2024-09-16T04:14:55.828 INFO:teuthology.orchestra.run.smithi042.stderr: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.35975.log.gz 2024-09-16T04:14:55.828 INFO:teuthology.orchestra.run.smithi042.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.33605.log.gz 2024-09-16T04:14:55.828 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.44534.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43485.log 2024-09-16T04:14:55.828 INFO:teuthology.orchestra.run.smithi042.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.32100.log.gz 2024-09-16T04:14:55.828 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.31834.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39393.log 2024-09-16T04:14:55.829 INFO:teuthology.orchestra.run.smithi042.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.44534.log.gz 2024-09-16T04:14:55.829 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.43485.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42140.log 2024-09-16T04:14:55.829 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.39393.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.31834.log.gz 2024-09-16T04:14:55.829 INFO:teuthology.orchestra.run.smithi042.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.43485.log.gz 2024-09-16T04:14:55.829 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34885.log 2024-09-16T04:14:55.829 INFO:teuthology.orchestra.run.smithi042.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.39393.log.gz 2024-09-16T04:14:55.829 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.42140.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41273.log 2024-09-16T04:14:55.829 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.34885.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32747.log 2024-09-16T04:14:55.829 INFO:teuthology.orchestra.run.smithi042.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.42140.log.gz 2024-09-16T04:14:55.829 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.41273.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41627.log 2024-09-16T04:14:55.829 INFO:teuthology.orchestra.run.smithi042.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.34885.log.gz 2024-09-16T04:14:55.829 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.32747.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28523.log 2024-09-16T04:14:55.829 INFO:teuthology.orchestra.run.smithi042.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.41273.log.gz 2024-09-16T04:14:55.829 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.41627.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34859.log 2024-09-16T04:14:55.830 INFO:teuthology.orchestra.run.smithi042.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.32747.log.gz 2024-09-16T04:14:55.830 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.28523.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32776.log 2024-09-16T04:14:55.830 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.34859.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.41627.log.gz 2024-09-16T04:14:55.830 INFO:teuthology.orchestra.run.smithi042.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.28523.log.gz 2024-09-16T04:14:55.830 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42400.log 2024-09-16T04:14:55.830 INFO:teuthology.orchestra.run.smithi042.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.34859.log.gz 2024-09-16T04:14:55.830 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.32776.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41654.log 2024-09-16T04:14:55.830 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.42400.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42114.log 2024-09-16T04:14:55.830 INFO:teuthology.orchestra.run.smithi042.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.32776.log.gz 2024-09-16T04:14:55.830 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.41654.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31312.log 2024-09-16T04:14:55.830 INFO:teuthology.orchestra.run.smithi042.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.41654.log.gz 2024-09-16T04:14:55.830 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.42114.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.42400.log.gz 2024-09-16T04:14:55.830 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30578.log 2024-09-16T04:14:55.830 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33482.log 2024-09-16T04:14:55.831 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.31312.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.42114.log.gz 2024-09-16T04:14:55.831 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.30578.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34911.log 2024-09-16T04:14:55.831 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.33482.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.31312.log.gz 2024-09-16T04:14:55.831 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30860.log 2024-09-16T04:14:55.831 INFO:teuthology.orchestra.run.smithi042.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.30578.log.gz 2024-09-16T04:14:55.831 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.34911.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.33482.log.gz 2024-09-16T04:14:55.831 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35341.log 2024-09-16T04:14:55.831 INFO:teuthology.orchestra.run.smithi042.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.34911.log.gz 2024-09-16T04:14:55.831 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39523.log 2024-09-16T04:14:55.831 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.30860.log: /var/log/ceph/ceph-client.admin.35341.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36406.log 2024-09-16T04:14:55.831 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.39523.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.35341.log.gz 2024-09-16T04:14:55.831 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33143.log 2024-09-16T04:14:55.831 INFO:teuthology.orchestra.run.smithi042.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.30860.log.gz 2024-09-16T04:14:55.831 INFO:teuthology.orchestra.run.smithi042.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.39523.log.gz 2024-09-16T04:14:55.832 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27775.log 2024-09-16T04:14:55.832 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.36406.log: /var/log/ceph/ceph-client.admin.33143.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28202.log 2024-09-16T04:14:55.832 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.27775.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.33143.log.gz 2024-09-16T04:14:55.832 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33815.log 2024-09-16T04:14:55.832 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41020.log 2024-09-16T04:14:55.832 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.28202.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.36406.log.gz 2024-09-16T04:14:55.832 INFO:teuthology.orchestra.run.smithi042.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.27775.log.gz 2024-09-16T04:14:55.832 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.33815.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35895.log 2024-09-16T04:14:55.832 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose 87.9% --/var/log/ceph/ceph-client.admin.41020.log: -- replaced with /var/log/ceph/ceph-client.admin.28202.log.gz 2024-09-16T04:14:55.832 INFO:teuthology.orchestra.run.smithi042.stderr: /var/log/ceph/ceph-client.admin.37111.log 2024-09-16T04:14:55.832 INFO:teuthology.orchestra.run.smithi042.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.33815.log.gz 2024-09-16T04:14:55.832 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.35895.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36189.log 2024-09-16T04:14:55.832 INFO:teuthology.orchestra.run.smithi042.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.41020.log.gz 2024-09-16T04:14:55.832 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.37111.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39339.log 2024-09-16T04:14:55.833 INFO:teuthology.orchestra.run.smithi042.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.35895.log.gz 2024-09-16T04:14:55.833 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32612.log 2024-09-16T04:14:55.833 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.36189.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.37111.log.gz 2024-09-16T04:14:55.833 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.39339.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40785.log 2024-09-16T04:14:55.833 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.32612.log: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.39339.log.gz 2024-09-16T04:14:55.833 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42322.log 2024-09-16T04:14:55.833 INFO:teuthology.orchestra.run.smithi042.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.36189.log.gz 2024-09-16T04:14:55.833 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.40785.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.32612.log.gz 2024-09-16T04:14:55.833 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34522.log 2024-09-16T04:14:55.833 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.42322.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28575.log 2024-09-16T04:14:55.833 INFO:teuthology.orchestra.run.smithi042.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.40785.log.gz 2024-09-16T04:14:55.833 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.34522.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43996.log 2024-09-16T04:14:55.833 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.28575.log: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.42322.log.gz 2024-09-16T04:14:55.834 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43311.log 2024-09-16T04:14:55.834 INFO:teuthology.orchestra.run.smithi042.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.34522.log.gz 2024-09-16T04:14:55.834 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.43996.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30025.log 2024-09-16T04:14:55.834 INFO:teuthology.orchestra.run.smithi042.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.28575.log.gz 2024-09-16T04:14:55.834 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.43311.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.43996.log.gz 2024-09-16T04:14:55.834 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36321.log 2024-09-16T04:14:55.834 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.30025.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43641.log 2024-09-16T04:14:55.834 INFO:teuthology.orchestra.run.smithi042.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.43311.log.gz 2024-09-16T04:14:55.834 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.36321.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43206.log 2024-09-16T04:14:55.834 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.43641.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.30025.log.gz 2024-09-16T04:14:55.834 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34392.log 2024-09-16T04:14:55.834 INFO:teuthology.orchestra.run.smithi042.stderr: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.36321.log.gz 2024-09-16T04:14:55.834 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.43206.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42790.log 2024-09-16T04:14:55.834 INFO:teuthology.orchestra.run.smithi042.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.43641.log.gz 2024-09-16T04:14:55.835 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 89.2% 2024-09-16T04:14:55.835 INFO:teuthology.orchestra.run.smithi042.stderr: -- replaced with /var/log/ceph/ceph-client.admin.43206.log.gz 2024-09-16T04:14:55.835 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.34392.log: /var/log/ceph/ceph-client.admin.42790.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37968.log 2024-09-16T04:14:55.835 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-osd.2.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.34392.log.gz 2024-09-16T04:14:55.835 INFO:teuthology.orchestra.run.smithi042.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.42790.log.gz 2024-09-16T04:14:55.835 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32152.log 2024-09-16T04:14:55.835 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39857.log 2024-09-16T04:14:55.835 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.37968.log: /var/log/ceph/ceph-client.admin.32152.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32073.log 2024-09-16T04:14:55.835 INFO:teuthology.orchestra.run.smithi042.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.37968.log.gz 2024-09-16T04:14:55.835 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.39857.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40691.log 2024-09-16T04:14:55.835 INFO:teuthology.orchestra.run.smithi042.stderr: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.32152.log.gz 2024-09-16T04:14:55.835 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.32073.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36488.log 2024-09-16T04:14:55.835 INFO:teuthology.orchestra.run.smithi042.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.39857.log.gz 2024-09-16T04:14:55.835 INFO:teuthology.orchestra.run.smithi042.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.32073.log.gz 2024-09-16T04:14:55.836 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.40691.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35843.log 2024-09-16T04:14:55.836 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40535.log 2024-09-16T04:14:55.836 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.36488.log: /var/log/ceph/ceph-client.admin.35843.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40509.log 2024-09-16T04:14:55.836 INFO:teuthology.orchestra.run.smithi042.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.40691.log.gz 2024-09-16T04:14:55.836 INFO:teuthology.orchestra.run.smithi042.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.35843.log.gz 2024-09-16T04:14:55.836 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.40535.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36698.log 2024-09-16T04:14:55.836 INFO:teuthology.orchestra.run.smithi042.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.36488.log.gz 2024-09-16T04:14:55.836 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.40509.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40039.log 2024-09-16T04:14:55.836 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.36698.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35183.log 2024-09-16T04:14:55.836 INFO:teuthology.orchestra.run.smithi042.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.40535.log.gz 2024-09-16T04:14:55.836 INFO:teuthology.orchestra.run.smithi042.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.40509.log.gz 2024-09-16T04:14:55.836 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.40039.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28341.log 2024-09-16T04:14:55.836 INFO:teuthology.orchestra.run.smithi042.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.36698.log.gz 2024-09-16T04:14:55.837 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.35183.log: gzip -5 --verbose 89.7% -- -- replaced with /var/log/ceph/ceph-client.admin.40039.log.gz /var/log/ceph/ceph-client.admin.33169.log 2024-09-16T04:14:55.837 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:55.837 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36514.log 2024-09-16T04:14:55.837 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.28341.log: /var/log/ceph/ceph-client.admin.33169.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28679.log 2024-09-16T04:14:55.837 INFO:teuthology.orchestra.run.smithi042.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.35183.log.gz 2024-09-16T04:14:55.837 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.36514.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.28341.log.gz 2024-09-16T04:14:55.837 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32534.log 2024-09-16T04:14:55.837 INFO:teuthology.orchestra.run.smithi042.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.33169.log.gz 2024-09-16T04:14:55.837 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.28679.log: 88.5%gzip -- replaced with /var/log/ceph/ceph-client.admin.36514.log.gz 2024-09-16T04:14:55.837 INFO:teuthology.orchestra.run.smithi042.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.37605.log 2024-09-16T04:14:55.837 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25120.log 2024-09-16T04:14:55.837 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.32534.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.28679.log.gz 2024-09-16T04:14:55.837 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.37605.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29917.log 2024-09-16T04:14:55.837 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.25120.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.32534.log.gz 2024-09-16T04:14:55.838 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42062.log 2024-09-16T04:14:55.838 INFO:teuthology.orchestra.run.smithi042.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.37605.log.gz 2024-09-16T04:14:55.838 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.29917.log: gzip -5 --verbose 88.0% -- /var/log/ceph/ceph-client.admin.27878.log -- replaced with /var/log/ceph/ceph-client.admin.25120.log.gz 2024-09-16T04:14:55.838 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:55.838 INFO:teuthology.orchestra.run.smithi042.stderr:gzip/var/log/ceph/ceph-client.admin.42062.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.30737.log 2024-09-16T04:14:55.838 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.27878.log: gzip -5 89.5% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.29917.log.gz 2024-09-16T04:14:55.838 INFO:teuthology.orchestra.run.smithi042.stderr: /var/log/ceph/ceph-client.admin.32586.log 2024-09-16T04:14:55.838 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.30737.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.42062.log.gz 2024-09-16T04:14:55.838 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41403.log 2024-09-16T04:14:55.838 INFO:teuthology.orchestra.run.smithi042.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.27878.log.gz 2024-09-16T04:14:55.838 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.32586.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31576.log 2024-09-16T04:14:55.838 INFO:teuthology.orchestra.run.smithi042.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.30737.log.gz 2024-09-16T04:14:55.838 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.41403.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33789.log 2024-09-16T04:14:55.838 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.31576.log: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.32586.log.gz 2024-09-16T04:14:55.839 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28549.log 2024-09-16T04:14:55.839 INFO:teuthology.orchestra.run.smithi042.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.41403.log.gz 2024-09-16T04:14:55.839 INFO:teuthology.orchestra.run.smithi042.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.31576.log.gz 2024-09-16T04:14:55.839 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.33789.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-09-16T04:14:55.839 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.28549.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31284.log 2024-09-16T04:14:55.839 INFO:teuthology.orchestra.run.smithi042.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.33789.log.gz 2024-09-16T04:14:55.839 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35713.log 2024-09-16T04:14:55.839 INFO:teuthology.orchestra.run.smithi042.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.28549.log.gz 2024-09-16T04:14:55.839 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.31284.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32560.log 2024-09-16T04:14:55.839 INFO:teuthology.orchestra.run.smithi042.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.31284.log.gz 2024-09-16T04:14:55.839 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.35713.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43459.log 2024-09-16T04:14:55.839 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32960.log 2024-09-16T04:14:55.839 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.32560.log: /var/log/ceph/ceph-client.admin.43459.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32986.log 2024-09-16T04:14:55.839 INFO:teuthology.orchestra.run.smithi042.stderr: 87.6% 89.8% -- replaced with /var/log/ceph/ceph-client.admin.35713.log.gz 2024-09-16T04:14:55.840 INFO:teuthology.orchestra.run.smithi042.stderr: -- replaced with /var/log/ceph/ceph-client.admin.32560.log.gz 2024-09-16T04:14:55.840 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.32960.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33403.log 2024-09-16T04:14:55.840 INFO:teuthology.orchestra.run.smithi042.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.43459.log.gz 2024-09-16T04:14:55.840 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.32986.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35525.log 2024-09-16T04:14:55.840 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.33403.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.32960.log.gz 2024-09-16T04:14:55.840 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42660.log 2024-09-16T04:14:55.840 INFO:teuthology.orchestra.run.smithi042.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.32986.log.gz 2024-09-16T04:14:55.840 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.35525.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.33403.log.gz 2024-09-16T04:14:55.840 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35869.log 2024-09-16T04:14:55.840 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.42660.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32378.log 2024-09-16T04:14:55.840 INFO:teuthology.orchestra.run.smithi042.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.35525.log.gz 2024-09-16T04:14:55.840 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.35869.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39961.log 2024-09-16T04:14:55.840 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.32378.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.42660.log.gz 2024-09-16T04:14:55.841 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31797.log 2024-09-16T04:14:55.841 INFO:teuthology.orchestra.run.smithi042.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.35869.log.gz 2024-09-16T04:14:55.841 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.39961.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36776.log 2024-09-16T04:14:55.841 INFO:teuthology.orchestra.run.smithi042.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.32378.log.gz 2024-09-16T04:14:55.841 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.31797.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.39961.log.gz 2024-09-16T04:14:55.841 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33273.log 2024-09-16T04:14:55.841 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.36776.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34574.log 2024-09-16T04:14:55.841 INFO:teuthology.orchestra.run.smithi042.stderr: 88.2%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30239.log 2024-09-16T04:14:55.841 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.33273.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.36776.log.gz 2024-09-16T04:14:55.841 INFO:teuthology.orchestra.run.smithi042.stderr: -- replaced with /var/log/ceph/ceph-client.admin.31797.log.gz 2024-09-16T04:14:55.841 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.34574.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38849.log 2024-09-16T04:14:55.841 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35150.log 2024-09-16T04:14:55.841 INFO:teuthology.orchestra.run.smithi042.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.33273.log.gz 2024-09-16T04:14:55.841 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.30239.log: /var/log/ceph/ceph-client.admin.38849.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.34574.log.gz 2024-09-16T04:14:55.842 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39575.log 2024-09-16T04:14:55.842 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43615.log 2024-09-16T04:14:55.842 INFO:teuthology.orchestra.run.smithi042.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.38849.log.gz 2024-09-16T04:14:55.842 INFO:teuthology.orchestra.run.smithi042.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.30239.log.gz 2024-09-16T04:14:55.842 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.35150.log: /var/log/ceph/ceph-client.admin.39575.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.35150.log.gz 2024-09-16T04:14:55.842 INFO:teuthology.orchestra.run.smithi042.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.39575.log.gz 2024-09-16T04:14:55.842 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44950.log 2024-09-16T04:14:55.842 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.43615.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30630.log 2024-09-16T04:14:55.842 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.44950.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29606.log 2024-09-16T04:14:55.842 INFO:teuthology.orchestra.run.smithi042.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.43615.log.gz 2024-09-16T04:14:55.842 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29356.log 2024-09-16T04:14:55.842 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.30630.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.44950.log.gz 2024-09-16T04:14:55.842 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.29606.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38875.log 2024-09-16T04:14:55.843 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.29356.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.30630.log.gz 2024-09-16T04:14:55.843 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28778.log 2024-09-16T04:14:55.843 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42504.log 2024-09-16T04:14:55.843 INFO:teuthology.orchestra.run.smithi042.stderr: 88.5%/var/log/ceph/ceph-client.admin.38875.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.29356.log.gz 2024-09-16T04:14:55.843 INFO:teuthology.orchestra.run.smithi042.stderr: -- replaced with /var/log/ceph/ceph-client.admin.29606.log.gz 2024-09-16T04:14:55.843 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.28778.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44612.log 2024-09-16T04:14:55.843 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.42504.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34652.log 2024-09-16T04:14:55.843 INFO:teuthology.orchestra.run.smithi042.stderr: 89.6% 89.0% -- replaced with /var/log/ceph/ceph-client.admin.28778.log.gz 2024-09-16T04:14:55.843 INFO:teuthology.orchestra.run.smithi042.stderr: -- replaced with /var/log/ceph/ceph-client.admin.38875.log.gz 2024-09-16T04:14:55.843 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.44612.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41680.log 2024-09-16T04:14:55.843 INFO:teuthology.orchestra.run.smithi042.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.42504.log.gz 2024-09-16T04:14:55.843 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34098.log 2024-09-16T04:14:55.843 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.34652.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.44612.log.gz 2024-09-16T04:14:55.843 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.41680.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34046.log 2024-09-16T04:14:55.844 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.34098.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38126.log 2024-09-16T04:14:55.844 INFO:teuthology.orchestra.run.smithi042.stderr: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.34652.log.gz 2024-09-16T04:14:55.844 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.34046.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.41680.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33299.log 2024-09-16T04:14:55.844 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:55.844 INFO:teuthology.orchestra.run.smithi042.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34098.log.gz 2024-09-16T04:14:55.844 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.38126.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30079.log 2024-09-16T04:14:55.844 INFO:teuthology.orchestra.run.smithi042.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.34046.log.gz 2024-09-16T04:14:55.844 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43154.log 2024-09-16T04:14:55.844 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.33299.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.38126.log.gz 2024-09-16T04:14:55.844 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42608.log 2024-09-16T04:14:55.844 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.30079.log: /var/log/ceph/ceph-client.admin.43154.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.33299.log.gz 2024-09-16T04:14:55.844 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31390.log 2024-09-16T04:14:55.844 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.42608.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28367.log 2024-09-16T04:14:55.845 INFO:teuthology.orchestra.run.smithi042.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.43154.log.gz 2024-09-16T04:14:55.845 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.31390.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28906.log 2024-09-16T04:14:55.845 INFO:teuthology.orchestra.run.smithi042.stderr: 87.6% 88.8% -- replaced with /var/log/ceph/ceph-client.admin.30079.log.gz 2024-09-16T04:14:55.845 INFO:teuthology.orchestra.run.smithi042.stderr: -- replaced with /var/log/ceph/ceph-client.admin.42608.log.gz 2024-09-16T04:14:55.845 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.28367.log: -- /var/log/ceph/ceph-client.admin.33711.log 2024-09-16T04:14:55.845 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.28906.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38953.log 2024-09-16T04:14:55.845 INFO:teuthology.orchestra.run.smithi042.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.31390.log.gz 2024-09-16T04:14:55.845 INFO:teuthology.orchestra.run.smithi042.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.28367.log.gz/var/log/ceph/ceph-client.admin.33711.log: 2024-09-16T04:14:55.845 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32508.log 2024-09-16T04:14:55.845 INFO:teuthology.orchestra.run.smithi042.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.28906.log.gz 2024-09-16T04:14:55.845 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34176.log 2024-09-16T04:14:55.845 INFO:teuthology.orchestra.run.smithi042.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.33711.log.gz 2024-09-16T04:14:55.845 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.38953.log: /var/log/ceph/ceph-client.admin.32508.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39627.log 2024-09-16T04:14:55.845 INFO:teuthology.orchestra.run.smithi042.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.38953.log.gz 2024-09-16T04:14:55.846 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.34176.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38592.log 2024-09-16T04:14:55.846 INFO:teuthology.orchestra.run.smithi042.stderr: 88.7%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27892.log 2024-09-16T04:14:55.846 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.39627.log: -- replaced with /var/log/ceph/ceph-client.admin.32508.log.gz 2024-09-16T04:14:55.846 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.38592.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36374.log 2024-09-16T04:14:55.846 INFO:teuthology.orchestra.run.smithi042.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.34176.log.gz 2024-09-16T04:14:55.846 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.21403.log 2024-09-16T04:14:55.846 INFO:teuthology.orchestra.run.smithi042.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.39627.log.gz 2024-09-16T04:14:55.846 INFO:teuthology.orchestra.run.smithi042.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.38592.log.gz 2024-09-16T04:14:55.846 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.27892.log: /var/log/ceph/ceph-client.admin.36374.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.27892.log.gz 2024-09-16T04:14:55.846 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30912.log 2024-09-16T04:14:55.846 INFO:teuthology.orchestra.run.smithi042.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.36374.log.gz 2024-09-16T04:14:55.846 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph.tmp-client.admin.21403.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42868.log 2024-09-16T04:14:55.846 INFO:teuthology.orchestra.run.smithi042.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.21403.log.gz 2024-09-16T04:14:55.846 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.30912.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28244.log 2024-09-16T04:14:55.847 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.42868.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37890.log 2024-09-16T04:14:55.847 INFO:teuthology.orchestra.run.smithi042.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.30912.log.gz 2024-09-16T04:14:55.847 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.28244.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36085.log 2024-09-16T04:14:55.847 INFO:teuthology.orchestra.run.smithi042.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.42868.log.gz 2024-09-16T04:14:55.847 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.37890.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38462.log 2024-09-16T04:14:55.847 INFO:teuthology.orchestra.run.smithi042.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.28244.log.gz 2024-09-16T04:14:55.847 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.36085.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31994.log 2024-09-16T04:14:55.847 INFO:teuthology.orchestra.run.smithi042.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.37890.log.gz 2024-09-16T04:14:55.847 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.38462.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42270.log 2024-09-16T04:14:55.847 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.31994.log: 91.1%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42036.log 2024-09-16T04:14:55.847 INFO:teuthology.orchestra.run.smithi042.stderr: -- replaced with /var/log/ceph/ceph-client.admin.36085.log.gz 2024-09-16T04:14:55.847 INFO:teuthology.orchestra.run.smithi042.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.38462.log.gz 2024-09-16T04:14:55.847 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.42270.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39058.log 2024-09-16T04:14:55.847 INFO:teuthology.orchestra.run.smithi042.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.31994.log.gz 2024-09-16T04:14:55.848 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36955.log 2024-09-16T04:14:55.848 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.42036.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.42270.log.gz 2024-09-16T04:14:55.848 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33351.log 2024-09-16T04:14:55.848 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.39058.log: /var/log/ceph/ceph-client.admin.36955.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40379.log 2024-09-16T04:14:55.848 INFO:teuthology.orchestra.run.smithi042.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.42036.log.gz 2024-09-16T04:14:55.848 INFO:teuthology.orchestra.run.smithi042.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.39058.log.gz 2024-09-16T04:14:55.848 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29502.log 2024-09-16T04:14:55.848 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.33351.log: /var/log/ceph/ceph-client.admin.40379.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28939.log 2024-09-16T04:14:55.848 INFO:teuthology.orchestra.run.smithi042.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.36955.log.gz 2024-09-16T04:14:55.848 INFO:teuthology.orchestra.run.smithi042.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.33351.log.gz 2024-09-16T04:14:55.848 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.29502.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41914.log 2024-09-16T04:14:55.848 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44048.log 2024-09-16T04:14:55.848 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.28939.log: 90.5% 87.5% -- replaced with /var/log/ceph/ceph-client.admin.29502.log.gz 2024-09-16T04:14:55.849 INFO:teuthology.orchestra.run.smithi042.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40379.log.gz/var/log/ceph/ceph-client.admin.41914.log: 2024-09-16T04:14:55.849 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34989.log 2024-09-16T04:14:55.849 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28601.log 2024-09-16T04:14:55.849 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.44048.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.28939.log.gz 2024-09-16T04:14:55.849 INFO:teuthology.orchestra.run.smithi042.stderr: 89.4%/var/log/ceph/ceph-client.admin.34989.log: -- replaced with /var/log/ceph/ceph-client.admin.41914.log.gz 2024-09-16T04:14:55.849 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42192.log 2024-09-16T04:14:55.849 INFO:teuthology.orchestra.run.smithi042.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.44048.log.gz 2024-09-16T04:14:55.849 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.28601.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41562.log 2024-09-16T04:14:55.849 INFO:teuthology.orchestra.run.smithi042.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.34989.log.gz 2024-09-16T04:14:55.849 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.42192.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43407.log 2024-09-16T04:14:55.849 INFO:teuthology.orchestra.run.smithi042.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.28601.log.gz 2024-09-16T04:14:55.849 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.41562.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42530.log 2024-09-16T04:14:55.849 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.43407.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.42192.log.gz 2024-09-16T04:14:55.849 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38927.log 2024-09-16T04:14:55.850 INFO:teuthology.orchestra.run.smithi042.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.41562.log.gz 2024-09-16T04:14:55.850 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.42530.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35421.log 2024-09-16T04:14:55.850 INFO:teuthology.orchestra.run.smithi042.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.43407.log.gz 2024-09-16T04:14:55.850 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.38927.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31046.log 2024-09-16T04:14:55.850 INFO:teuthology.orchestra.run.smithi042.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.42530.log.gz 2024-09-16T04:14:55.850 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.35421.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28445.log 2024-09-16T04:14:55.850 INFO:teuthology.orchestra.run.smithi042.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.38927.log.gz 2024-09-16T04:14:55.850 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.31046.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35473.log 2024-09-16T04:14:55.850 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.28445.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.35421.log.gz 2024-09-16T04:14:55.850 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41481.log 2024-09-16T04:14:55.850 INFO:teuthology.orchestra.run.smithi042.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.31046.log.gz 2024-09-16T04:14:55.850 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.35473.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37553.log 2024-09-16T04:14:55.850 INFO:teuthology.orchestra.run.smithi042.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.28445.log.gz 2024-09-16T04:14:55.850 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.41481.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37085.log 2024-09-16T04:14:55.850 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32882.log 2024-09-16T04:14:55.851 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.37553.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.35473.log.gz 2024-09-16T04:14:55.851 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.37085.log: gzip 90.2% -5 -- replaced with /var/log/ceph/ceph-client.admin.41481.log.gz --verbose 2024-09-16T04:14:55.851 INFO:teuthology.orchestra.run.smithi042.stderr: -- /var/log/ceph/ceph-client.admin.36724.log 2024-09-16T04:14:55.851 INFO:teuthology.orchestra.run.smithi042.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.37553.log.gz/var/log/ceph/ceph-client.admin.32882.log: 2024-09-16T04:14:55.851 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34364.log 2024-09-16T04:14:55.851 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.36724.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.37085.log.gz 2024-09-16T04:14:55.851 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33659.log 2024-09-16T04:14:55.851 INFO:teuthology.orchestra.run.smithi042.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.32882.log.gz 2024-09-16T04:14:55.851 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.34364.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37033.log 2024-09-16T04:14:55.851 INFO:teuthology.orchestra.run.smithi042.stderr: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.36724.log.gz 2024-09-16T04:14:55.851 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.33659.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37475.log 2024-09-16T04:14:55.851 INFO:teuthology.orchestra.run.smithi042.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.34364.log.gz 2024-09-16T04:14:55.851 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.37033.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29769.log 2024-09-16T04:14:55.851 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.37475.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41429.log 2024-09-16T04:14:55.852 INFO:teuthology.orchestra.run.smithi042.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.33659.log.gz 2024-09-16T04:14:55.852 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.29769.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.37033.log.gz 2024-09-16T04:14:55.852 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34807.log 2024-09-16T04:14:55.852 INFO:teuthology.orchestra.run.smithi042.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.37475.log.gz 2024-09-16T04:14:55.852 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29464.log 2024-09-16T04:14:55.852 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.41429.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.29769.log.gz 2024-09-16T04:14:55.852 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29736.log 2024-09-16T04:14:55.852 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.34807.log: /var/log/ceph/ceph-client.admin.29464.log: 88.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42478.log 2024-09-16T04:14:55.852 INFO:teuthology.orchestra.run.smithi042.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41429.log.gz 2024-09-16T04:14:55.852 INFO:teuthology.orchestra.run.smithi042.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.34807.log.gz 2024-09-16T04:14:55.852 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.29736.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37501.log 2024-09-16T04:14:55.852 INFO:teuthology.orchestra.run.smithi042.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.29464.log.gz 2024-09-16T04:14:55.852 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.42478.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32126.log 2024-09-16T04:14:55.852 INFO:teuthology.orchestra.run.smithi042.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.29736.log.gz 2024-09-16T04:14:55.853 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.37501.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41533.log 2024-09-16T04:14:55.853 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.32126.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.42478.log.gz 2024-09-16T04:14:55.853 INFO:teuthology.orchestra.run.smithi042.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.37501.log.gz 2024-09-16T04:14:55.853 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36750.log 2024-09-16T04:14:55.853 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.41533.log: gzip 87.8% -5 -- replaced with /var/log/ceph/ceph-client.admin.32126.log.gz --verbose 2024-09-16T04:14:55.853 INFO:teuthology.orchestra.run.smithi042.stderr: -- /var/log/ceph/ceph-client.admin.30053.log 2024-09-16T04:14:55.853 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.36750.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38979.log 2024-09-16T04:14:55.853 INFO:teuthology.orchestra.run.smithi042.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.41533.log.gz 2024-09-16T04:14:55.853 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.30053.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29438.log 2024-09-16T04:14:55.853 INFO:teuthology.orchestra.run.smithi042.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.36750.log.gz 2024-09-16T04:14:55.853 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.38979.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34937.log 2024-09-16T04:14:55.853 INFO:teuthology.orchestra.run.smithi042.stderr: 89.4%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43337.log 2024-09-16T04:14:55.853 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.29438.log: -- replaced with /var/log/ceph/ceph-client.admin.30053.log.gz 2024-09-16T04:14:55.853 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37758.log 2024-09-16T04:14:55.854 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.34937.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.38979.log.gz 2024-09-16T04:14:55.854 INFO:teuthology.orchestra.run.smithi042.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.29438.log.gz 2024-09-16T04:14:55.854 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.43337.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.34937.log.gz 2024-09-16T04:14:55.854 INFO:teuthology.orchestra.run.smithi042.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.43337.log.gz 2024-09-16T04:14:55.854 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44820.log 2024-09-16T04:14:55.854 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.37758.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34280.log 2024-09-16T04:14:55.854 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.44820.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28217.log 2024-09-16T04:14:55.854 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.34280.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34020.log 2024-09-16T04:14:55.854 INFO:teuthology.orchestra.run.smithi042.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.44820.log.gz 2024-09-16T04:14:55.854 INFO:teuthology.orchestra.run.smithi042.stderr: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.37758.log.gz 2024-09-16T04:14:55.854 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.28217.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41247.log 2024-09-16T04:14:55.854 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.34020.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39497.log 2024-09-16T04:14:55.854 INFO:teuthology.orchestra.run.smithi042.stderr: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.34280.log.gz 2024-09-16T04:14:55.854 INFO:teuthology.orchestra.run.smithi042.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.28217.log.gz 2024-09-16T04:14:55.855 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.41247.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36672.log 2024-09-16T04:14:55.855 INFO:teuthology.orchestra.run.smithi042.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.34020.log.gz 2024-09-16T04:14:55.855 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30416.log 2024-09-16T04:14:55.855 INFO:teuthology.orchestra.run.smithi042.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.41247.log.gz 2024-09-16T04:14:55.855 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.39497.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41046.log 2024-09-16T04:14:55.855 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.36672.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36620.log 2024-09-16T04:14:55.855 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.30416.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.39497.log.gz 2024-09-16T04:14:55.855 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.41046.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43024.log 2024-09-16T04:14:55.855 INFO:teuthology.orchestra.run.smithi042.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.36672.log.gz 2024-09-16T04:14:55.855 INFO:teuthology.orchestra.run.smithi042.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.30416.log.gz 2024-09-16T04:14:55.855 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.36620.log: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.41046.log.gz 2024-09-16T04:14:55.855 INFO:teuthology.orchestra.run.smithi042.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.36620.log.gz 2024-09-16T04:14:55.855 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42712.log 2024-09-16T04:14:55.855 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.43024.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31628.log 2024-09-16T04:14:55.855 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.42712.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36594.log 2024-09-16T04:14:55.856 INFO:teuthology.orchestra.run.smithi042.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.43024.log.gz 2024-09-16T04:14:55.856 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.31628.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40720.log 2024-09-16T04:14:55.856 INFO:teuthology.orchestra.run.smithi042.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.42712.log.gz 2024-09-16T04:14:55.856 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.36594.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40353.log 2024-09-16T04:14:55.856 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.40720.log: gzip -5 90.6% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.31628.log.gz /var/log/ceph/ceph-client.admin.36436.log 2024-09-16T04:14:55.856 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:55.856 INFO:teuthology.orchestra.run.smithi042.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.36594.log.gz/var/log/ceph/ceph-client.admin.40353.log: 2024-09-16T04:14:55.856 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28497.log 2024-09-16T04:14:55.856 INFO:teuthology.orchestra.run.smithi042.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.40720.log.gz 2024-09-16T04:14:55.856 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.36436.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39005.log 2024-09-16T04:14:55.856 INFO:teuthology.orchestra.run.smithi042.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.40353.log.gz 2024-09-16T04:14:55.856 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.28497.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38332.log 2024-09-16T04:14:55.856 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.39005.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44100.log 2024-09-16T04:14:55.857 INFO:teuthology.orchestra.run.smithi042.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.36436.log.gz 2024-09-16T04:14:55.857 INFO:teuthology.orchestra.run.smithi042.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.28497.log.gz 2024-09-16T04:14:55.857 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.38332.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42348.log 2024-09-16T04:14:55.857 INFO:teuthology.orchestra.run.smithi042.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.39005.log.gz 2024-09-16T04:14:55.857 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.44100.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32456.log 2024-09-16T04:14:55.857 INFO:teuthology.orchestra.run.smithi042.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.38332.log.gz 2024-09-16T04:14:55.857 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.42348.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44430.log 2024-09-16T04:14:55.857 INFO:teuthology.orchestra.run.smithi042.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.44100.log.gz 2024-09-16T04:14:55.857 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.32456.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41169.log 2024-09-16T04:14:55.857 INFO:teuthology.orchestra.run.smithi042.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.42348.log.gz 2024-09-16T04:14:55.857 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.44430.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28705.log 2024-09-16T04:14:55.857 INFO:teuthology.orchestra.run.smithi042.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.32456.log.gz 2024-09-16T04:14:55.857 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.41169.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36059.log 2024-09-16T04:14:55.857 INFO:teuthology.orchestra.run.smithi042.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.44430.log.gz 2024-09-16T04:14:55.857 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.28705.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39419.log 2024-09-16T04:14:55.858 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.36059.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.41169.log.gz 2024-09-16T04:14:55.858 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43771.log 2024-09-16T04:14:55.858 INFO:teuthology.orchestra.run.smithi042.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.28705.log.gz 2024-09-16T04:14:55.858 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.39419.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34444.log 2024-09-16T04:14:55.858 INFO:teuthology.orchestra.run.smithi042.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.36059.log.gz 2024-09-16T04:14:55.858 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.43771.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31208.log 2024-09-16T04:14:55.858 INFO:teuthology.orchestra.run.smithi042.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.39419.log.gz 2024-09-16T04:14:55.858 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.34444.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43719.log 2024-09-16T04:14:55.858 INFO:teuthology.orchestra.run.smithi042.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.43771.log.gz 2024-09-16T04:14:55.858 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.31208.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44690.log 2024-09-16T04:14:55.858 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.43719.log: 89.3%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33763.log 2024-09-16T04:14:55.858 INFO:teuthology.orchestra.run.smithi042.stderr: -- replaced with /var/log/ceph/ceph-client.admin.34444.log.gz 2024-09-16T04:14:55.858 INFO:teuthology.orchestra.run.smithi042.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.31208.log.gz 2024-09-16T04:14:55.858 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.44690.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43128.log 2024-09-16T04:14:55.859 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.33763.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-09-16T04:14:55.859 INFO:teuthology.orchestra.run.smithi042.stderr: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.43719.log.gz 2024-09-16T04:14:55.859 INFO:teuthology.orchestra.run.smithi042.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.44690.log.gz 2024-09-16T04:14:55.859 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.43128.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44560.log 2024-09-16T04:14:55.859 INFO:teuthology.orchestra.run.smithi042.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.33763.log.gz 2024-09-16T04:14:55.859 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39682.log 2024-09-16T04:14:55.859 INFO:teuthology.orchestra.run.smithi042.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.43128.log.gz 2024-09-16T04:14:55.859 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5/var/log/ceph/ceph-client.admin.44560.log: --verbose -- /var/log/ceph/ceph-client.admin.40457.log 2024-09-16T04:14:55.859 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.39682.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41221.log 2024-09-16T04:14:55.859 INFO:teuthology.orchestra.run.smithi042.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.44560.log.gz/var/log/ceph/ceph-client.admin.40457.log: 2024-09-16T04:14:55.859 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35635.log 2024-09-16T04:14:55.859 INFO:teuthology.orchestra.run.smithi042.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.39682.log.gz 2024-09-16T04:14:55.859 INFO:teuthology.orchestra.run.smithi042.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.40457.log.gz 2024-09-16T04:14:55.880 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.28240.log: 91.7%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28632.log 2024-09-16T04:14:55.880 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.28443.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.28443.log.gz 2024-09-16T04:14:55.880 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28224.log 2024-09-16T04:14:55.880 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.28632.log: -- replaced with /var/log/ceph/ceph-client.admin.28240.log.gz 2024-09-16T04:14:55.880 INFO:teuthology.orchestra.run.smithi154.stderr: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.28083.log.gz 2024-09-16T04:14:55.880 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28479.log 2024-09-16T04:14:55.880 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.28224.log: 91.7% -- replaced with /var/log/ceph/ceph-client.admin.28632.log.gz 2024-09-16T04:14:55.880 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28249.log 2024-09-16T04:14:55.880 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.28479.log: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.28224.log.gz 2024-09-16T04:14:55.880 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28499.log 2024-09-16T04:14:55.881 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.28249.log: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.28249.log.gz 2024-09-16T04:14:55.881 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29339.log 2024-09-16T04:14:55.881 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.28499.log: 95.2% -- replaced with /var/log/ceph/ceph-client.admin.28198.log.gz 2024-09-16T04:14:55.881 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28286.log 2024-09-16T04:14:55.881 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.29339.log: 91.5% -- replaced with /var/log/ceph/ceph-client.admin.29339.log.gz 2024-09-16T04:14:55.881 INFO:teuthology.orchestra.run.smithi154.stderr: 92.1% -- replaced with /var/log/ceph/ceph-client.admin.28499.log.gz 2024-09-16T04:14:55.881 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28649.log 2024-09-16T04:14:55.881 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.28286.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29716.log 2024-09-16T04:14:55.881 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.28649.log: 87.6% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-16T04:14:55.881 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29410.log 2024-09-16T04:14:55.881 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.29716.log: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.28286.log.gz 2024-09-16T04:14:55.881 INFO:teuthology.orchestra.run.smithi154.stderr: 93.3% -- replaced with /var/log/ceph/ceph-client.admin.28479.log.gz 2024-09-16T04:14:55.881 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28115.log 2024-09-16T04:14:55.881 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28489.log 2024-09-16T04:14:55.882 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.29410.log: /var/log/ceph/ceph-client.admin.28115.log: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.29716.log.gz 2024-09-16T04:14:55.882 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-16T04:14:55.882 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.28489.log: 91.5% -- replaced with /var/log/ceph/ceph-client.admin.28115.log.gz 2024-09-16T04:14:55.882 INFO:teuthology.orchestra.run.smithi154.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.28489.log.gz 2024-09-16T04:14:55.882 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28653.log 2024-09-16T04:14:55.882 INFO:teuthology.orchestra.run.smithi154.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.29410.log.gz 2024-09-16T04:14:55.882 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28553.log 2024-09-16T04:14:55.882 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph.audit.log: /var/log/ceph/ceph-client.admin.28653.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28591.log 2024-09-16T04:14:55.882 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.28553.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.28649.log.gz 2024-09-16T04:14:55.882 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28468.log 2024-09-16T04:14:55.882 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.28591.log: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.28653.log.gz 2024-09-16T04:14:55.882 INFO:teuthology.orchestra.run.smithi154.stderr: 92.0% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-16T04:14:55.882 INFO:teuthology.orchestra.run.smithi154.stderr: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.28553.log.gz 2024-09-16T04:14:55.882 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28099.log 2024-09-16T04:14:55.883 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.28468.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28128.log 2024-09-16T04:14:55.883 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.28099.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28216.log 2024-09-16T04:14:55.883 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.28128.log: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.28468.log.gz 2024-09-16T04:14:55.883 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25236.log 2024-09-16T04:14:55.883 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.28216.log: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.28128.log.gz 2024-09-16T04:14:55.883 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28467.log 2024-09-16T04:14:55.883 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.25236.log: 92.1% -- replaced with /var/log/ceph/ceph-client.admin.28099.log.gz 2024-09-16T04:14:55.883 INFO:teuthology.orchestra.run.smithi154.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.25236.log.gz 2024-09-16T04:14:55.883 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-09-16T04:14:55.883 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29335.log 2024-09-16T04:14:55.883 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.28467.log: /var/log/ceph/ceph-mon.b.log: 91.7% -- replaced with /var/log/ceph/ceph-client.admin.28467.log.gz 2024-09-16T04:14:55.883 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29337.log 2024-09-16T04:14:55.887 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.29335.log: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.29335.log.gz 2024-09-16T04:14:55.887 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-09-16T04:14:55.887 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.29337.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.29337.log.gz 2024-09-16T04:14:55.887 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25164.log 2024-09-16T04:14:55.892 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-osd.4.log: 92.9% -- replaced with /var/log/ceph/ceph-client.admin.28591.log.gz 2024-09-16T04:14:55.907 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28425.log 2024-09-16T04:14:55.907 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.25164.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.25164.log.gz 2024-09-16T04:14:55.907 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28345.log 2024-09-16T04:14:55.907 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.28425.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.28425.log.gz 2024-09-16T04:14:55.907 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29412.log 2024-09-16T04:14:55.912 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.28345.log: 92.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28301.log 2024-09-16T04:14:55.919 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.29412.log: 91.7% -- replaced with /var/log/ceph/ceph-client.admin.28345.log.gz 2024-09-16T04:14:55.919 INFO:teuthology.orchestra.run.smithi154.stderr: -- replaced with /var/log/ceph/ceph-client.admin.29412.log.gz 2024-09-16T04:14:55.919 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29234.log 2024-09-16T04:14:55.926 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.28301.log: 92.2%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28381.log 2024-09-16T04:14:55.926 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.29234.log: 91.7% -- replaced with /var/log/ceph/ceph-client.admin.28301.log.gz 2024-09-16T04:14:55.926 INFO:teuthology.orchestra.run.smithi154.stderr: -- replaced with /var/log/ceph/ceph-client.admin.29234.log.gz 2024-09-16T04:14:55.927 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28232.log 2024-09-16T04:14:55.930 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.41221.log: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.41221.log.gz 2024-09-16T04:14:55.930 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37864.log 2024-09-16T04:14:55.930 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.35635.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40561.log 2024-09-16T04:14:55.930 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.37864.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35817.log 2024-09-16T04:14:55.931 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.40561.log: 88.9%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41351.log 2024-09-16T04:14:55.931 INFO:teuthology.orchestra.run.smithi042.stderr: -- replaced with /var/log/ceph/ceph-client.admin.35635.log.gz 2024-09-16T04:14:55.931 INFO:teuthology.orchestra.run.smithi042.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.37864.log.gz 2024-09-16T04:14:55.931 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.35817.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.40561.log.gz 2024-09-16T04:14:55.931 INFO:teuthology.orchestra.run.smithi042.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.35817.log.gz 2024-09-16T04:14:55.931 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37579.log 2024-09-16T04:14:55.931 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.41351.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36029.log 2024-09-16T04:14:55.931 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.37579.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37812.log 2024-09-16T04:14:55.931 INFO:teuthology.orchestra.run.smithi042.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.41351.log.gz 2024-09-16T04:14:55.931 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.36029.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40091.log 2024-09-16T04:14:55.931 INFO:teuthology.orchestra.run.smithi042.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.37579.log.gz/var/log/ceph/ceph-client.admin.37812.log: 2024-09-16T04:14:55.931 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30656.log 2024-09-16T04:14:55.931 INFO:teuthology.orchestra.run.smithi042.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.36029.log.gz 2024-09-16T04:14:55.931 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.40091.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.37812.log.gz 2024-09-16T04:14:55.932 INFO:teuthology.orchestra.run.smithi042.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.40091.log.gz 2024-09-16T04:14:55.932 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39445.log 2024-09-16T04:14:55.932 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.30656.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35041.log 2024-09-16T04:14:55.932 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.39445.log: 88.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39549.log 2024-09-16T04:14:55.932 INFO:teuthology.orchestra.run.smithi042.stderr: -- replaced with /var/log/ceph/ceph-client.admin.30656.log.gz 2024-09-16T04:14:55.932 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.35041.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39747.log 2024-09-16T04:14:55.932 INFO:teuthology.orchestra.run.smithi042.stderr: 89.6%gzip -- replaced with /var/log/ceph/ceph-client.admin.39445.log.gz -5 --verbose -- /var/log/ceph/ceph-client.admin.38384.log 2024-09-16T04:14:55.932 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:55.932 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.39549.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.35041.log.gz 2024-09-16T04:14:55.932 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.39747.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.39549.log.gz 2024-09-16T04:14:55.932 INFO:teuthology.orchestra.run.smithi042.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.39747.log.gz 2024-09-16T04:14:55.932 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25195.log 2024-09-16T04:14:55.932 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.38384.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27887.log 2024-09-16T04:14:55.933 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.25195.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33539.log 2024-09-16T04:14:55.933 INFO:teuthology.orchestra.run.smithi042.stderr: 89.3%/var/log/ceph/ceph-client.admin.27887.log: -- replaced with /var/log/ceph/ceph-client.admin.38384.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30329.log 2024-09-16T04:14:55.933 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:55.933 INFO:teuthology.orchestra.run.smithi042.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.25195.log.gz 2024-09-16T04:14:55.933 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.33539.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.27887.log.gz 2024-09-16T04:14:55.933 INFO:teuthology.orchestra.run.smithi042.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.33539.log.gz 2024-09-16T04:14:55.933 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41784.log 2024-09-16T04:14:55.933 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.30329.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30390.log 2024-09-16T04:14:55.933 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.41784.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44974.log 2024-09-16T04:14:55.933 INFO:teuthology.orchestra.run.smithi042.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.30329.log.gz 2024-09-16T04:14:55.933 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.30390.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.41784.log.gz 2024-09-16T04:14:55.933 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31886.log 2024-09-16T04:14:55.933 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.44974.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28393.log 2024-09-16T04:14:55.933 INFO:teuthology.orchestra.run.smithi042.stderr: 89.8%/var/log/ceph/ceph-client.admin.31886.log: -- replaced with /var/log/ceph/ceph-client.admin.30390.log.gz 2024-09-16T04:14:55.934 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33994.log 2024-09-16T04:14:55.934 INFO:teuthology.orchestra.run.smithi042.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.44974.log.gz 2024-09-16T04:14:55.934 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43259.log 2024-09-16T04:14:55.934 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.28393.log: /var/log/ceph/ceph-client.admin.33994.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31416.log 2024-09-16T04:14:55.934 INFO:teuthology.orchestra.run.smithi042.stderr: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.31886.log.gz 2024-09-16T04:14:55.934 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.43259.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.33994.log.gz 2024-09-16T04:14:55.934 INFO:teuthology.orchestra.run.smithi042.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.28393.log.gz 2024-09-16T04:14:55.934 INFO:teuthology.orchestra.run.smithi042.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.43259.log.gz 2024-09-16T04:14:55.934 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32206.log 2024-09-16T04:14:55.934 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.31416.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31256.log 2024-09-16T04:14:55.934 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.32206.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34600.log 2024-09-16T04:14:55.934 INFO:teuthology.orchestra.run.smithi042.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.31416.log.gz 2024-09-16T04:14:55.934 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.31256.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30711.log 2024-09-16T04:14:55.935 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30968.log 2024-09-16T04:14:55.935 INFO:teuthology.orchestra.run.smithi042.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.32206.log.gz 2024-09-16T04:14:55.935 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.34600.log: /var/log/ceph/ceph-client.admin.30711.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.31256.log.gz 2024-09-16T04:14:55.935 INFO:teuthology.orchestra.run.smithi042.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.34600.log.gz 2024-09-16T04:14:55.935 INFO:teuthology.orchestra.run.smithi042.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.30711.log.gz 2024-09-16T04:14:55.935 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40916.log 2024-09-16T04:14:55.935 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.30968.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32242.log 2024-09-16T04:14:55.935 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.40916.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42842.log 2024-09-16T04:14:55.935 INFO:teuthology.orchestra.run.smithi042.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.30968.log.gz/var/log/ceph/ceph-client.admin.32242.log: 2024-09-16T04:14:55.935 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30185.log 2024-09-16T04:14:55.935 INFO:teuthology.orchestra.run.smithi042.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.40916.log.gz 2024-09-16T04:14:55.935 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.42842.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.32242.log.gz 2024-09-16T04:14:55.935 INFO:teuthology.orchestra.run.smithi042.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.42842.log.gz 2024-09-16T04:14:55.935 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40665.log 2024-09-16T04:14:55.936 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.30185.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29865.log 2024-09-16T04:14:55.936 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37189.log 2024-09-16T04:14:55.936 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.40665.log: /var/log/ceph/ceph-client.admin.29865.log: gzip 90.1% -- replaced with /var/log/ceph/ceph-client.admin.30185.log.gz -5 2024-09-16T04:14:55.936 INFO:teuthology.orchestra.run.smithi042.stderr: --verbose -- /var/log/ceph/ceph-client.admin.40587.log 2024-09-16T04:14:55.936 INFO:teuthology.orchestra.run.smithi042.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.29865.log.gz 2024-09-16T04:14:55.936 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.37189.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.40665.log.gz 2024-09-16T04:14:55.936 INFO:teuthology.orchestra.run.smithi042.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.37189.log.gz 2024-09-16T04:14:55.936 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37786.log 2024-09-16T04:14:55.936 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.40587.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29077.log 2024-09-16T04:14:55.936 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.37786.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32270.log 2024-09-16T04:14:55.936 INFO:teuthology.orchestra.run.smithi042.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.40587.log.gz 2024-09-16T04:14:55.936 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.29077.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28189.log 2024-09-16T04:14:55.936 INFO:teuthology.orchestra.run.smithi042.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.37786.log.gz 2024-09-16T04:14:55.936 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.32270.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31446.log 2024-09-16T04:14:55.937 INFO:teuthology.orchestra.run.smithi042.stderr: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.29077.log.gz 2024-09-16T04:14:55.937 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.28189.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.32270.log.gz 2024-09-16T04:14:55.937 INFO:teuthology.orchestra.run.smithi042.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.28189.log.gz 2024-09-16T04:14:55.937 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36243.log 2024-09-16T04:14:55.937 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.31446.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36163.log 2024-09-16T04:14:55.937 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.36243.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31912.log 2024-09-16T04:14:55.937 INFO:teuthology.orchestra.run.smithi042.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.31446.log.gz 2024-09-16T04:14:55.937 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.36163.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27893.log 2024-09-16T04:14:55.937 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.31912.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.36243.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35499.log 2024-09-16T04:14:55.937 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:55.937 INFO:teuthology.orchestra.run.smithi042.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.36163.log.gz 2024-09-16T04:14:55.937 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.27893.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.31912.log.gz 2024-09-16T04:14:55.937 INFO:teuthology.orchestra.run.smithi042.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.27893.log.gz 2024-09-16T04:14:55.937 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29386.log 2024-09-16T04:14:55.938 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.35499.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34626.log 2024-09-16T04:14:55.938 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.29386.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34963.log 2024-09-16T04:14:55.938 INFO:teuthology.orchestra.run.smithi042.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.35499.log.gz/var/log/ceph/ceph-client.admin.34626.log: 2024-09-16T04:14:55.938 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44898.log 2024-09-16T04:14:55.938 INFO:teuthology.orchestra.run.smithi042.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.29386.log.gz 2024-09-16T04:14:55.938 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.34963.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.34626.log.gz 2024-09-16T04:14:55.938 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35263.log 2024-09-16T04:14:55.938 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35577.log 2024-09-16T04:14:55.938 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.44898.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.34963.log.gz 2024-09-16T04:14:55.938 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.35263.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30299.log 2024-09-16T04:14:55.938 INFO:teuthology.orchestra.run.smithi042.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.44898.log.gz 2024-09-16T04:14:55.938 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37241.log 2024-09-16T04:14:55.938 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.35577.log: /var/log/ceph/ceph-client.admin.30299.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.35263.log.gz 2024-09-16T04:14:55.938 INFO:teuthology.orchestra.run.smithi042.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.30299.log.gz 2024-09-16T04:14:55.939 INFO:teuthology.orchestra.run.smithi042.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.35577.log.gz 2024-09-16T04:14:55.939 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41940.log 2024-09-16T04:14:55.939 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.37241.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40942.log 2024-09-16T04:14:55.939 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.41940.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35369.log 2024-09-16T04:14:55.939 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.40942.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.37241.log.gz 2024-09-16T04:14:55.939 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44352.log 2024-09-16T04:14:55.939 INFO:teuthology.orchestra.run.smithi042.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.41940.log.gz 2024-09-16T04:14:55.939 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38046.log 2024-09-16T04:14:55.939 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.35369.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.40942.log.gz 2024-09-16T04:14:55.939 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.44352.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.35369.log.gz 2024-09-16T04:14:55.939 INFO:teuthology.orchestra.run.smithi042.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.44352.log.gz 2024-09-16T04:14:55.939 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43918.log 2024-09-16T04:14:55.939 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.38046.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33429.log 2024-09-16T04:14:55.939 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.43918.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-16T04:14:55.939 INFO:teuthology.orchestra.run.smithi042.stderr: 87.3%/var/log/ceph/ceph-client.admin.33429.log: -- replaced with /var/log/ceph/ceph-client.admin.38046.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39136.log 2024-09-16T04:14:55.940 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:55.940 INFO:teuthology.orchestra.run.smithi042.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.43918.log.gz 2024-09-16T04:14:55.940 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.33429.log.gz 2024-09-16T04:14:55.940 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27795.log 2024-09-16T04:14:55.940 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.39136.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41758.log 2024-09-16T04:14:55.940 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.27795.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33968.log 2024-09-16T04:14:55.940 INFO:teuthology.orchestra.run.smithi042.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.39136.log.gz 2024-09-16T04:14:55.940 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.41758.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.27795.log.gz 2024-09-16T04:14:55.940 INFO:teuthology.orchestra.run.smithi042.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.41758.log.gz 2024-09-16T04:14:55.940 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44508.log 2024-09-16T04:14:55.940 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.33968.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42556.log 2024-09-16T04:14:55.940 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.44508.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29580.log 2024-09-16T04:14:55.940 INFO:teuthology.orchestra.run.smithi042.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.33968.log.gz 2024-09-16T04:14:55.941 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.42556.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.44508.log.gz 2024-09-16T04:14:55.941 INFO:teuthology.orchestra.run.smithi042.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.42556.log.gz 2024-09-16T04:14:55.941 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30938.log 2024-09-16T04:14:55.941 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.29580.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30470.log 2024-09-16T04:14:55.941 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.30938.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38186.log 2024-09-16T04:14:55.941 INFO:teuthology.orchestra.run.smithi042.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.29580.log.gz 2024-09-16T04:14:55.941 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.30470.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.30938.log.gz 2024-09-16T04:14:55.941 INFO:teuthology.orchestra.run.smithi042.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.30470.log.gz 2024-09-16T04:14:55.941 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41888.log 2024-09-16T04:14:55.941 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.38186.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36929.log 2024-09-16T04:14:55.941 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.41888.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30106.log 2024-09-16T04:14:55.941 INFO:teuthology.orchestra.run.smithi042.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.38186.log.gz 2024-09-16T04:14:55.941 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.36929.log: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.41888.log.gz 2024-09-16T04:14:55.941 INFO:teuthology.orchestra.run.smithi042.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.36929.log.gz 2024-09-16T04:14:55.942 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35791.log 2024-09-16T04:14:55.942 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.30106.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41810.log 2024-09-16T04:14:55.942 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.35791.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40296.log 2024-09-16T04:14:55.942 INFO:teuthology.orchestra.run.smithi042.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.30106.log.gz 2024-09-16T04:14:55.942 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.41810.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.35791.log.gz 2024-09-16T04:14:55.942 INFO:teuthology.orchestra.run.smithi042.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.41810.log.gz 2024-09-16T04:14:55.942 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27669.log 2024-09-16T04:14:55.942 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.40296.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31338.log 2024-09-16T04:14:55.942 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.27669.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42946.log 2024-09-16T04:14:55.942 INFO:teuthology.orchestra.run.smithi042.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.40296.log.gz 2024-09-16T04:14:55.942 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.31338.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.27669.log.gz 2024-09-16T04:14:55.942 INFO:teuthology.orchestra.run.smithi042.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.31338.log.gz 2024-09-16T04:14:55.942 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30133.log 2024-09-16T04:14:55.942 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.42946.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32482.log 2024-09-16T04:14:55.943 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.30133.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34228.log 2024-09-16T04:14:55.943 INFO:teuthology.orchestra.run.smithi042.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.42946.log.gz 2024-09-16T04:14:55.943 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.32482.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.30133.log.gz 2024-09-16T04:14:55.943 INFO:teuthology.orchestra.run.smithi042.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.32482.log.gz 2024-09-16T04:14:55.943 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39367.log 2024-09-16T04:14:55.943 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.34228.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44178.log 2024-09-16T04:14:55.943 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.39367.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31472.log 2024-09-16T04:14:55.943 INFO:teuthology.orchestra.run.smithi042.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.34228.log.gz 2024-09-16T04:14:55.943 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.44178.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.39367.log.gz 2024-09-16T04:14:55.943 INFO:teuthology.orchestra.run.smithi042.stderr: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.44178.log.gz 2024-09-16T04:14:55.943 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32856.log 2024-09-16T04:14:55.943 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.31472.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33247.log 2024-09-16T04:14:55.943 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.32856.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36811.log 2024-09-16T04:14:55.943 INFO:teuthology.orchestra.run.smithi042.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.31472.log.gz 2024-09-16T04:14:55.944 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.33247.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.32856.log.gz 2024-09-16T04:14:55.944 INFO:teuthology.orchestra.run.smithi042.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.33247.log.gz 2024-09-16T04:14:55.944 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31128.log 2024-09-16T04:14:55.944 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.36811.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32802.log 2024-09-16T04:14:55.944 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.31128.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30524.log 2024-09-16T04:14:55.944 INFO:teuthology.orchestra.run.smithi042.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.36811.log.gz 2024-09-16T04:14:55.944 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.32802.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.31128.log.gz 2024-09-16T04:14:55.944 INFO:teuthology.orchestra.run.smithi042.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.32802.log.gz 2024-09-16T04:14:55.944 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44260.log 2024-09-16T04:14:55.944 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.30524.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33685.log 2024-09-16T04:14:55.944 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.44260.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35093.log 2024-09-16T04:14:55.944 INFO:teuthology.orchestra.run.smithi042.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.30524.log.gz 2024-09-16T04:14:55.944 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.33685.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.44260.log.gz 2024-09-16T04:14:55.944 INFO:teuthology.orchestra.run.smithi042.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.33685.log.gz 2024-09-16T04:14:55.945 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29943.log 2024-09-16T04:14:55.945 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.35093.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29045.log 2024-09-16T04:14:55.945 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.29943.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39162.log 2024-09-16T04:14:55.945 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.29045.log: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.35093.log.gz 2024-09-16T04:14:55.945 INFO:teuthology.orchestra.run.smithi042.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.29943.log.gz 2024-09-16T04:14:55.945 INFO:teuthology.orchestra.run.smithi042.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.29045.log.gz 2024-09-16T04:14:55.945 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25001.log 2024-09-16T04:14:55.945 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.39162.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38020.log 2024-09-16T04:14:55.945 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.25001.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40169.log 2024-09-16T04:14:55.945 INFO:teuthology.orchestra.run.smithi042.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.39162.log.gz 2024-09-16T04:14:55.945 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.38020.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.25001.log.gz 2024-09-16T04:14:55.945 INFO:teuthology.orchestra.run.smithi042.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.38020.log.gz 2024-09-16T04:14:55.945 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27725.log 2024-09-16T04:14:55.946 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.40169.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33899.log 2024-09-16T04:14:55.946 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.27725.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29143.log 2024-09-16T04:14:55.946 INFO:teuthology.orchestra.run.smithi042.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.40169.log.gz 2024-09-16T04:14:55.946 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.33899.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.27725.log.gz 2024-09-16T04:14:55.946 INFO:teuthology.orchestra.run.smithi042.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.33899.log.gz 2024-09-16T04:14:55.946 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37215.log 2024-09-16T04:14:55.946 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.29143.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36217.log 2024-09-16T04:14:55.946 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.37215.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28970.log 2024-09-16T04:14:55.946 INFO:teuthology.orchestra.run.smithi042.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.29143.log.gz 2024-09-16T04:14:55.946 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.36217.log: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.37215.log.gz 2024-09-16T04:14:55.946 INFO:teuthology.orchestra.run.smithi042.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.36217.log.gz 2024-09-16T04:14:55.946 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-09-16T04:14:55.946 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.28970.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43511.log 2024-09-16T04:14:55.946 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28048.log 2024-09-16T04:14:55.947 INFO:teuthology.orchestra.run.smithi042.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.28970.log.gz 2024-09-16T04:14:55.947 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.43511.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.43511.log.gz 2024-09-16T04:14:55.947 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30552.log 2024-09-16T04:14:55.947 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41732.log 2024-09-16T04:14:55.947 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.28048.log: /var/log/ceph/ceph-client.admin.30552.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.28048.log.gz 2024-09-16T04:14:55.947 INFO:teuthology.orchestra.run.smithi042.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.30552.log.gz 2024-09-16T04:14:55.947 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42426.log 2024-09-16T04:14:55.947 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.41732.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.41732.log.gz 2024-09-16T04:14:55.947 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37527.log 2024-09-16T04:14:55.947 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34704.log 2024-09-16T04:14:55.947 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.42426.log: /var/log/ceph/ceph-client.admin.37527.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.42426.log.gz 2024-09-16T04:14:55.947 INFO:teuthology.orchestra.run.smithi042.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.37527.log.gzgzip 2024-09-16T04:14:55.947 INFO:teuthology.orchestra.run.smithi042.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.39987.log 2024-09-16T04:14:55.947 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.34704.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.34704.log.gz 2024-09-16T04:14:55.948 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41098.log 2024-09-16T04:14:55.948 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29554.log 2024-09-16T04:14:55.948 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.39987.log: /var/log/ceph/ceph-client.admin.41098.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.39987.log.gz 2024-09-16T04:14:55.948 INFO:teuthology.orchestra.run.smithi042.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.41098.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34124.log 2024-09-16T04:14:55.948 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:55.948 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.29554.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.29554.log.gz 2024-09-16T04:14:55.948 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44404.log 2024-09-16T04:14:55.948 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42816.log 2024-09-16T04:14:55.948 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.34124.log: /var/log/ceph/ceph-client.admin.44404.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.34124.log.gz 2024-09-16T04:14:55.948 INFO:teuthology.orchestra.run.smithi042.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.44404.log.gz 2024-09-16T04:14:55.948 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40405.log 2024-09-16T04:14:55.948 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37370.log 2024-09-16T04:14:55.948 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.42816.log: /var/log/ceph/ceph-client.admin.40405.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.42816.log.gz 2024-09-16T04:14:55.948 INFO:teuthology.orchestra.run.smithi042.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.40405.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27750.log 2024-09-16T04:14:55.949 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:55.949 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.37370.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.37370.log.gz 2024-09-16T04:14:55.949 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44768.log 2024-09-16T04:14:55.949 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29891.log 2024-09-16T04:14:55.949 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.27750.log: /var/log/ceph/ceph-client.admin.44768.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.27750.log.gz 2024-09-16T04:14:55.949 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43667.log 2024-09-16T04:14:55.949 INFO:teuthology.orchestra.run.smithi042.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.44768.log.gz 2024-09-16T04:14:55.949 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.29891.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.29891.log.gz 2024-09-16T04:14:55.949 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44872.log 2024-09-16T04:14:55.949 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27821.log 2024-09-16T04:14:55.949 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.43667.log: /var/log/ceph/ceph-client.admin.44872.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.43667.log.gz 2024-09-16T04:14:55.949 INFO:teuthology.orchestra.run.smithi042.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.44872.log.gz 2024-09-16T04:14:55.949 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38514.log 2024-09-16T04:14:55.949 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38251.log 2024-09-16T04:14:55.950 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.27821.log: /var/log/ceph/ceph-client.admin.38514.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.27821.log.gz 2024-09-16T04:14:55.950 INFO:teuthology.orchestra.run.smithi042.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.38514.log.gz 2024-09-16T04:14:55.950 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40812.log 2024-09-16T04:14:55.950 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.38251.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.38251.log.gz 2024-09-16T04:14:55.950 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44664.log 2024-09-16T04:14:55.950 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37657.log 2024-09-16T04:14:55.950 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.40812.log: /var/log/ceph/ceph-client.admin.44664.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.40812.log.gz 2024-09-16T04:14:55.950 INFO:teuthology.orchestra.run.smithi042.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.44664.log.gz 2024-09-16T04:14:55.950 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35236.log 2024-09-16T04:14:55.950 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.37657.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.37657.log.gz 2024-09-16T04:14:55.950 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44378.log 2024-09-16T04:14:55.950 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32430.log 2024-09-16T04:14:55.950 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.35236.log: /var/log/ceph/ceph-client.admin.44378.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.35236.log.gz 2024-09-16T04:14:55.950 INFO:teuthology.orchestra.run.smithi042.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.44378.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44074.log 2024-09-16T04:14:55.951 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:55.951 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.32430.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.32430.log.gz 2024-09-16T04:14:55.951 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44204.log 2024-09-16T04:14:55.951 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42894.log 2024-09-16T04:14:55.951 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.44074.log: /var/log/ceph/ceph-client.admin.44204.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.44074.log.gz 2024-09-16T04:14:55.951 INFO:teuthology.orchestra.run.smithi042.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.44204.log.gz 2024-09-16T04:14:55.951 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43944.log 2024-09-16T04:14:55.951 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.42894.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.42894.log.gz 2024-09-16T04:14:55.951 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31730.log 2024-09-16T04:14:55.951 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34418.log 2024-09-16T04:14:55.951 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.43944.log: /var/log/ceph/ceph-client.admin.31730.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.31730.log.gz 90.1% -- replaced with /var/log/ceph/ceph-client.admin.43944.log.gz 2024-09-16T04:14:55.951 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:55.951 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36111.log 2024-09-16T04:14:55.951 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28185.log 2024-09-16T04:14:55.951 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.34418.log: /var/log/ceph/ceph-client.admin.36111.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.34418.log.gz 2024-09-16T04:14:55.952 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40890.log 2024-09-16T04:14:55.952 INFO:teuthology.orchestra.run.smithi042.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.36111.log.gz 2024-09-16T04:14:55.953 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.28381.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29417.log 2024-09-16T04:14:55.954 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.28232.log: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.28232.log.gz 2024-09-16T04:14:55.965 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/ceph-client.admin.29417.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.29417.log.gz 2024-09-16T04:14:55.968 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.28185.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.28185.log.gz 2024-09-16T04:14:55.968 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43693.log 2024-09-16T04:14:55.968 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37267.log 2024-09-16T04:14:55.968 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.40890.log: /var/log/ceph/ceph-client.admin.43693.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.40890.log.gz 2024-09-16T04:14:55.968 INFO:teuthology.orchestra.run.smithi042.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.43693.log.gz 2024-09-16T04:14:55.968 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44586.log 2024-09-16T04:14:55.968 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.37267.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.37267.log.gz 2024-09-16T04:14:55.968 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31654.log 2024-09-16T04:14:55.968 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42088.log 2024-09-16T04:14:55.968 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.44586.log: /var/log/ceph/ceph-client.admin.31654.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.44586.log.gz 2024-09-16T04:14:55.968 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-16T04:14:55.968 INFO:teuthology.orchestra.run.smithi042.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.31654.log.gz 2024-09-16T04:14:55.969 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.42088.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.42088.log.gz 2024-09-16T04:14:55.969 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32020.log 2024-09-16T04:14:55.969 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28653.log 2024-09-16T04:14:55.969 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph.audit.log: /var/log/ceph/ceph-client.admin.32020.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.32020.log.gz 2024-09-16T04:14:55.969 INFO:teuthology.orchestra.run.smithi042.stderr: 90.1% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-16T04:14:55.969 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31860.log 2024-09-16T04:14:55.969 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.28653.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29169.log 2024-09-16T04:14:55.969 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.31860.log: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.28653.log.gz 2024-09-16T04:14:55.969 INFO:teuthology.orchestra.run.smithi042.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.31860.log.gz 2024-09-16T04:14:55.969 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38540.log 2024-09-16T04:14:55.969 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29658.log 2024-09-16T04:14:55.969 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.29169.log: /var/log/ceph/ceph-client.admin.38540.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.29169.log.gz 2024-09-16T04:14:55.969 INFO:teuthology.orchestra.run.smithi042.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.38540.log.gz 2024-09-16T04:14:55.969 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41325.log 2024-09-16T04:14:55.970 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.29658.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.29658.log.gz 2024-09-16T04:14:55.970 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30498.log 2024-09-16T04:14:55.970 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30159.log 2024-09-16T04:14:55.970 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.41325.log: /var/log/ceph/ceph-client.admin.30498.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.41325.log.gz 2024-09-16T04:14:55.970 INFO:teuthology.orchestra.run.smithi042.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.30498.log.gz 2024-09-16T04:14:55.970 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31182.log 2024-09-16T04:14:55.970 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31602.log 2024-09-16T04:14:55.970 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.30159.log: /var/log/ceph/ceph-client.admin.31182.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.30159.log.gz 2024-09-16T04:14:55.970 INFO:teuthology.orchestra.run.smithi042.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.31182.log.gz 2024-09-16T04:14:55.970 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31364.log 2024-09-16T04:14:55.970 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.31602.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.31602.log.gz 2024-09-16T04:14:55.970 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38901.log 2024-09-16T04:14:55.970 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36903.log 2024-09-16T04:14:55.970 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.31364.log: /var/log/ceph/ceph-client.admin.38901.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.31364.log.gz 2024-09-16T04:14:55.970 INFO:teuthology.orchestra.run.smithi042.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.38901.log.gz 2024-09-16T04:14:55.971 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34202.log 2024-09-16T04:14:55.971 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37423.log 2024-09-16T04:14:55.971 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.36903.log: /var/log/ceph/ceph-client.admin.34202.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.36903.log.gz 2024-09-16T04:14:55.971 INFO:teuthology.orchestra.run.smithi042.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34202.log.gz 2024-09-16T04:14:55.971 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38703.log 2024-09-16T04:14:55.971 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41377.log 2024-09-16T04:14:55.971 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.37423.log: /var/log/ceph/ceph-client.admin.38703.log: 94.8% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-16T04:14:55.971 INFO:teuthology.orchestra.run.smithi042.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.38703.log.gz 2024-09-16T04:14:55.971 INFO:teuthology.orchestra.run.smithi042.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.37423.log.gz 2024-09-16T04:14:55.971 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36269.log 2024-09-16T04:14:55.971 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.41377.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38152.log 2024-09-16T04:14:55.971 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.36269.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28868.log 2024-09-16T04:14:55.971 INFO:teuthology.orchestra.run.smithi042.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.41377.log.gz 2024-09-16T04:14:55.971 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.38152.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.36269.log.gz 2024-09-16T04:14:55.972 INFO:teuthology.orchestra.run.smithi042.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.38152.log.gz 2024-09-16T04:14:55.972 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41455.log 2024-09-16T04:14:55.972 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.28868.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42686.log 2024-09-16T04:14:55.972 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.41455.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43537.log 2024-09-16T04:14:55.972 INFO:teuthology.orchestra.run.smithi042.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.28868.log.gz 2024-09-16T04:14:55.972 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.42686.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.41455.log.gz 2024-09-16T04:14:55.972 INFO:teuthology.orchestra.run.smithi042.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.42686.log.gz 2024-09-16T04:14:55.972 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28471.log 2024-09-16T04:14:55.972 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.43537.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40325.log 2024-09-16T04:14:55.972 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.28471.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29016.log 2024-09-16T04:14:55.972 INFO:teuthology.orchestra.run.smithi042.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.43537.log.gz 2024-09-16T04:14:55.972 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.40325.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.28471.log.gz 2024-09-16T04:14:55.972 INFO:teuthology.orchestra.run.smithi042.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.40325.log.gz 2024-09-16T04:14:55.972 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36295.log 2024-09-16T04:14:55.973 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.29016.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35289.log 2024-09-16T04:14:55.973 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.36295.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39653.log 2024-09-16T04:14:55.973 INFO:teuthology.orchestra.run.smithi042.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.29016.log.gz 2024-09-16T04:14:55.973 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.35289.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.36295.log.gz 2024-09-16T04:14:55.973 INFO:teuthology.orchestra.run.smithi042.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.35289.log.gz 2024-09-16T04:14:55.973 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38306.log 2024-09-16T04:14:55.973 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.39653.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42634.log 2024-09-16T04:14:55.973 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.38306.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41862.log 2024-09-16T04:14:55.973 INFO:teuthology.orchestra.run.smithi042.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.39653.log.gz 2024-09-16T04:14:55.973 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.42634.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.38306.log.gz 2024-09-16T04:14:55.973 INFO:teuthology.orchestra.run.smithi042.stderr: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.42634.log.gz 2024-09-16T04:14:55.973 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42920.log 2024-09-16T04:14:55.973 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.41862.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37994.log 2024-09-16T04:14:55.973 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.42920.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28033.log 2024-09-16T04:14:55.974 INFO:teuthology.orchestra.run.smithi042.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.41862.log.gz 2024-09-16T04:14:55.974 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.37994.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.42920.log.gz 2024-09-16T04:14:55.974 INFO:teuthology.orchestra.run.smithi042.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.37994.log.gz 2024-09-16T04:14:55.974 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37449.log 2024-09-16T04:14:55.974 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.28033.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32908.log 2024-09-16T04:14:55.981 INFO:teuthology.orchestra.run.smithi154.stderr: 92.8% -- replaced with /var/log/ceph/ceph-client.admin.28381.log.gz 2024-09-16T04:14:55.981 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.37449.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37397.log 2024-09-16T04:14:55.982 INFO:teuthology.orchestra.run.smithi042.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.28033.log.gz 2024-09-16T04:14:55.982 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.32908.log: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.37449.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33221.log 2024-09-16T04:14:55.982 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:55.982 INFO:teuthology.orchestra.run.smithi042.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.32908.log.gz 2024-09-16T04:14:55.982 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.37397.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.37397.log.gz 2024-09-16T04:14:55.982 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35739.log 2024-09-16T04:14:55.982 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.33221.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32324.log 2024-09-16T04:14:55.982 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.35739.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35124.log 2024-09-16T04:14:55.982 INFO:teuthology.orchestra.run.smithi042.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.33221.log.gz 2024-09-16T04:14:55.982 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.32324.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.35739.log.gz 2024-09-16T04:14:55.982 INFO:teuthology.orchestra.run.smithi042.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.32324.log.gz 2024-09-16T04:14:55.982 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40838.log 2024-09-16T04:14:55.983 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.35124.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31020.log 2024-09-16T04:14:55.983 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.40838.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31524.log 2024-09-16T04:14:55.983 INFO:teuthology.orchestra.run.smithi042.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.35124.log.gz 2024-09-16T04:14:55.983 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.31020.log: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.40838.log.gz 2024-09-16T04:14:55.983 INFO:teuthology.orchestra.run.smithi042.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.31020.log.gz 2024-09-16T04:14:55.983 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44325.log 2024-09-16T04:14:55.983 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.31524.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37059.log 2024-09-16T04:14:55.983 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.44325.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33040.log 2024-09-16T04:14:55.983 INFO:teuthology.orchestra.run.smithi042.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.31524.log.gz 2024-09-16T04:14:55.983 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.37059.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.44325.log.gz 2024-09-16T04:14:55.983 INFO:teuthology.orchestra.run.smithi042.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.37059.log.gz 2024-09-16T04:14:55.983 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41836.log 2024-09-16T04:14:55.983 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.33040.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37631.log 2024-09-16T04:14:55.983 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.41836.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37163.log 2024-09-16T04:14:55.984 INFO:teuthology.orchestra.run.smithi042.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.33040.log.gz 2024-09-16T04:14:55.984 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.37631.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.41836.log.gz 2024-09-16T04:14:55.984 INFO:teuthology.orchestra.run.smithi042.stderr: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.37631.log.gz 2024-09-16T04:14:55.984 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33066.log 2024-09-16T04:14:55.984 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.37163.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33508.log 2024-09-16T04:14:55.984 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.33066.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30994.log 2024-09-16T04:14:55.984 INFO:teuthology.orchestra.run.smithi042.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.37163.log.gz 2024-09-16T04:14:55.984 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.33508.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.33066.log.gz 2024-09-16T04:14:55.984 INFO:teuthology.orchestra.run.smithi042.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.33508.log.gz 2024-09-16T04:14:55.984 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44846.log 2024-09-16T04:14:55.984 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.30994.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38823.log 2024-09-16T04:14:55.984 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.44846.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33455.log 2024-09-16T04:14:55.984 INFO:teuthology.orchestra.run.smithi042.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.30994.log.gz 2024-09-16T04:14:55.984 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.38823.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.44846.log.gz 2024-09-16T04:14:55.985 INFO:teuthology.orchestra.run.smithi042.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.38823.log.gz 2024-09-16T04:14:55.985 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36003.log 2024-09-16T04:14:55.985 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 has been restored 2024-09-16T04:14:55.988 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.33455.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40613.log 2024-09-16T04:14:55.988 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.36003.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32638.log 2024-09-16T04:14:55.988 INFO:teuthology.orchestra.run.smithi042.stderr: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.33455.log.gz/var/log/ceph/ceph-client.admin.40613.log: 2024-09-16T04:14:55.988 INFO:teuthology.orchestra.run.smithi042.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.36003.log.gz 2024-09-16T04:14:55.988 INFO:teuthology.orchestra.run.smithi042.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.40613.log.gz 2024-09-16T04:14:55.988 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40994.log 2024-09-16T04:14:55.988 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.32638.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38278.log 2024-09-16T04:14:55.988 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.40994.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29528.log 2024-09-16T04:14:55.989 INFO:teuthology.orchestra.run.smithi042.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.32638.log.gz 2024-09-16T04:14:55.989 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.38278.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.40994.log.gz 2024-09-16T04:14:55.989 INFO:teuthology.orchestra.run.smithi042.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.38278.log.gz 2024-09-16T04:14:55.989 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33873.log 2024-09-16T04:14:55.989 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.29528.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44230.log 2024-09-16T04:14:55.989 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.33873.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35947.log 2024-09-16T04:14:55.989 INFO:teuthology.orchestra.run.smithi042.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.29528.log.gz 2024-09-16T04:14:55.989 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.44230.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.33873.log.gz 2024-09-16T04:14:55.989 INFO:teuthology.orchestra.run.smithi042.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.44230.log.gz 2024-09-16T04:14:55.989 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40431.log 2024-09-16T04:14:55.989 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.35947.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44126.log 2024-09-16T04:14:55.991 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.40431.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30886.log 2024-09-16T04:14:55.991 INFO:teuthology.orchestra.run.smithi042.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.35947.log.gz 2024-09-16T04:14:55.991 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.44126.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.40431.log.gz 2024-09-16T04:14:55.991 INFO:teuthology.orchestra.run.smithi042.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.44126.log.gz 2024-09-16T04:14:55.991 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40117.log 2024-09-16T04:14:55.992 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.30886.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38797.log 2024-09-16T04:14:55.992 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.40117.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32180.log 2024-09-16T04:14:55.992 INFO:teuthology.orchestra.run.smithi042.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.30886.log.gz 2024-09-16T04:14:55.992 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.38797.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.40117.log.gz 2024-09-16T04:14:55.992 INFO:teuthology.orchestra.run.smithi042.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.38797.log.gz 2024-09-16T04:14:55.992 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39195.log 2024-09-16T04:14:55.994 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.32180.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39774.log 2024-09-16T04:14:55.994 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.39195.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31966.log 2024-09-16T04:14:55.994 INFO:teuthology.orchestra.run.smithi042.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.32180.log.gz 2024-09-16T04:14:55.994 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.39774.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.39195.log.gz 2024-09-16T04:14:55.994 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28049.log 2024-09-16T04:14:55.994 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34308.log 2024-09-16T04:14:55.994 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.31966.log: /var/log/ceph/ceph-client.admin.28049.log: 92.2% -- replaced with /var/log/ceph/ceph-client.admin.39774.log.gz 2024-09-16T04:14:55.994 INFO:teuthology.orchestra.run.smithi042.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.31966.log.gz 2024-09-16T04:14:55.996 INFO:teuthology.orchestra.run.smithi042.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.28049.log.gz 2024-09-16T04:14:55.996 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36981.log 2024-09-16T04:14:55.996 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.34308.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28047.log 2024-09-16T04:14:55.996 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.36981.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40143.log 2024-09-16T04:14:55.996 INFO:teuthology.orchestra.run.smithi042.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.34308.log.gz 2024-09-16T04:14:55.996 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.28047.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.36981.log.gz 2024-09-16T04:14:55.996 INFO:teuthology.orchestra.run.smithi042.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.28047.log.gz 2024-09-16T04:14:55.997 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43970.log 2024-09-16T04:14:55.997 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.40143.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35765.log 2024-09-16T04:14:55.997 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.43970.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30213.log 2024-09-16T04:14:55.998 INFO:teuthology.orchestra.run.smithi042.stderr: 90.2%/var/log/ceph/ceph-client.admin.35765.log: -- replaced with /var/log/ceph/ceph-client.admin.40143.log.gz 2024-09-16T04:14:55.998 INFO:teuthology.orchestra.run.smithi042.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.43970.log.gz 2024-09-16T04:14:55.998 INFO:teuthology.orchestra.run.smithi042.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.35765.log.gz 2024-09-16T04:14:55.999 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44152.log 2024-09-16T04:14:55.999 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.30213.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28222.log 2024-09-16T04:14:55.999 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.44152.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28627.log 2024-09-16T04:14:56.000 INFO:teuthology.orchestra.run.smithi042.stderr: 89.5%/var/log/ceph/ceph-client.admin.28222.log: -- replaced with /var/log/ceph/ceph-client.admin.30213.log.gz 2024-09-16T04:14:56.000 INFO:teuthology.orchestra.run.smithi042.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.44152.log.gz 2024-09-16T04:14:56.000 INFO:teuthology.orchestra.run.smithi042.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.28222.log.gz 2024-09-16T04:14:56.002 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44456.log 2024-09-16T04:14:56.002 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.28627.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37838.log 2024-09-16T04:14:56.002 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.44456.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40065.log 2024-09-16T04:14:56.002 INFO:teuthology.orchestra.run.smithi042.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.28627.log.gz 2024-09-16T04:14:56.002 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.37838.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.44456.log.gz 2024-09-16T04:14:56.002 INFO:teuthology.orchestra.run.smithi042.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.37838.log.gz 2024-09-16T04:14:56.003 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43563.log 2024-09-16T04:14:56.003 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.40065.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29684.log 2024-09-16T04:14:56.003 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.43563.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31154.log 2024-09-16T04:14:56.004 INFO:teuthology.orchestra.run.smithi042.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.40065.log.gz/var/log/ceph/ceph-client.admin.29684.log: 2024-09-16T04:14:56.004 INFO:teuthology.orchestra.run.smithi042.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.43563.log.gz 2024-09-16T04:14:56.004 INFO:teuthology.orchestra.run.smithi042.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.29684.log.gz 2024-09-16T04:14:56.005 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39471.log 2024-09-16T04:14:56.006 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.31154.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42998.log 2024-09-16T04:14:56.006 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.39471.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39260.log 2024-09-16T04:14:56.006 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.42998.log: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.31154.log.gz 2024-09-16T04:14:56.006 INFO:teuthology.orchestra.run.smithi042.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.39471.log.gz 2024-09-16T04:14:56.007 INFO:teuthology.orchestra.run.smithi042.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.42998.log.gz 2024-09-16T04:14:56.007 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33845.log 2024-09-16T04:14:56.008 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.39260.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34833.log 2024-09-16T04:14:56.008 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.33845.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43745.log 2024-09-16T04:14:56.008 INFO:teuthology.orchestra.run.smithi042.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.39260.log.gz/var/log/ceph/ceph-client.admin.34833.log: 2024-09-16T04:14:56.009 INFO:teuthology.orchestra.run.smithi042.stderr: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.33845.log.gz 2024-09-16T04:14:56.009 INFO:teuthology.orchestra.run.smithi042.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.34833.log.gz 2024-09-16T04:14:56.009 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34781.log 2024-09-16T04:14:56.010 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.43745.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39084.log 2024-09-16T04:14:56.011 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.34781.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40639.log 2024-09-16T04:14:56.011 INFO:teuthology.orchestra.run.smithi042.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.43745.log.gz 2024-09-16T04:14:56.011 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.39084.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.34781.log.gz 2024-09-16T04:14:56.011 INFO:teuthology.orchestra.run.smithi042.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.39084.log.gz 2024-09-16T04:14:56.011 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34470.log 2024-09-16T04:14:56.012 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.40639.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33195.log 2024-09-16T04:14:56.012 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.34470.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38618.log 2024-09-16T04:14:56.013 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.33195.log: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.40639.log.gz 89.1% -- replaced with /var/log/ceph/ceph-client.admin.34470.log.gz 2024-09-16T04:14:56.013 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:56.013 INFO:teuthology.orchestra.run.smithi042.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.33195.log.gz 2024-09-16T04:14:56.013 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41072.log 2024-09-16T04:14:56.014 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.38618.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40013.log 2024-09-16T04:14:56.014 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.41072.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43892.log 2024-09-16T04:14:56.015 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.40013.log: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.38618.log.gz 2024-09-16T04:14:56.015 INFO:teuthology.orchestra.run.smithi042.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.41072.log.gz 2024-09-16T04:14:56.015 INFO:teuthology.orchestra.run.smithi042.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.40013.log.gz 2024-09-16T04:14:56.016 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37916.log 2024-09-16T04:14:56.016 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.43892.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42764.log 2024-09-16T04:14:56.017 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.37916.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32404.log 2024-09-16T04:14:56.017 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.42764.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.43892.log.gz 2024-09-16T04:14:56.017 INFO:teuthology.orchestra.run.smithi042.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.37916.log.gz 2024-09-16T04:14:56.017 INFO:teuthology.orchestra.run.smithi042.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.42764.log.gz 2024-09-16T04:14:56.017 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33325.log 2024-09-16T04:14:56.019 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.32404.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31498.log 2024-09-16T04:14:56.019 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.33325.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42452.log 2024-09-16T04:14:56.019 INFO:teuthology.orchestra.run.smithi042.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.32404.log.gz 2024-09-16T04:14:56.019 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.31498.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.33325.log.gz 2024-09-16T04:14:56.019 INFO:teuthology.orchestra.run.smithi042.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.31498.log.gz 2024-09-16T04:14:56.020 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32046.log 2024-09-16T04:14:56.020 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.42452.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44482.log 2024-09-16T04:14:56.021 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.32046.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42296.log 2024-09-16T04:14:56.021 INFO:teuthology.orchestra.run.smithi042.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.42452.log.gz 2024-09-16T04:14:56.021 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.44482.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.32046.log.gz 2024-09-16T04:14:56.021 INFO:teuthology.orchestra.run.smithi042.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.44482.log.gz 2024-09-16T04:14:56.022 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29412.log 2024-09-16T04:14:56.026 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.42296.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43102.log 2024-09-16T04:14:56.026 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.29412.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35067.log 2024-09-16T04:14:56.026 INFO:teuthology.orchestra.run.smithi042.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.42296.log.gz 2024-09-16T04:14:56.026 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.43102.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.29412.log.gz 2024-09-16T04:14:56.026 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42010.log 2024-09-16T04:14:56.026 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.35067.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.43102.log.gz 2024-09-16T04:14:56.026 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39287.log 2024-09-16T04:14:56.026 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.42010.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28046.log 2024-09-16T04:14:56.026 INFO:teuthology.orchestra.run.smithi042.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.35067.log.gz 2024-09-16T04:14:56.026 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.39287.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.42010.log.gz 2024-09-16T04:14:56.026 INFO:teuthology.orchestra.run.smithi042.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.39287.log.gz 2024-09-16T04:14:56.027 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27886.log 2024-09-16T04:14:56.027 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.28046.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29710.log 2024-09-16T04:14:56.028 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.27886.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31938.log 2024-09-16T04:14:56.028 INFO:teuthology.orchestra.run.smithi042.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.28046.log.gz/var/log/ceph/ceph-client.admin.29710.log: 2024-09-16T04:14:56.029 INFO:teuthology.orchestra.run.smithi042.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.27886.log.gz 2024-09-16T04:14:56.029 INFO:teuthology.orchestra.run.smithi042.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.29710.log.gz 2024-09-16T04:14:56.029 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35603.log 2024-09-16T04:14:56.029 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.31938.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42244.log 2024-09-16T04:14:56.029 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.35603.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37007.log 2024-09-16T04:14:56.029 INFO:teuthology.orchestra.run.smithi042.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.31938.log.gz 2024-09-16T04:14:56.029 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.42244.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.35603.log.gz 2024-09-16T04:14:56.029 INFO:teuthology.orchestra.run.smithi042.stderr: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.42244.log.gz 2024-09-16T04:14:56.030 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35315.log 2024-09-16T04:14:56.030 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.37007.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35661.log 2024-09-16T04:14:56.031 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.35315.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33633.log 2024-09-16T04:14:56.031 INFO:teuthology.orchestra.run.smithi042.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.37007.log.gz 2024-09-16T04:14:56.031 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.35661.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.35315.log.gz 2024-09-16T04:14:56.032 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30683.log 2024-09-16T04:14:56.032 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.33633.log: 90.8%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38488.log 2024-09-16T04:14:56.032 INFO:teuthology.orchestra.run.smithi042.stderr: -- replaced with /var/log/ceph/ceph-client.admin.35661.log.gz 2024-09-16T04:14:56.032 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.30683.log: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.33633.log.gz 2024-09-16T04:14:56.033 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44716.log 2024-09-16T04:14:56.034 INFO:teuthology.orchestra.run.smithi042.stderr: 91.4%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38566.log 2024-09-16T04:14:56.034 INFO:teuthology.orchestra.run.smithi042.stderr: -- replaced with /var/log/ceph/ceph-client.admin.30683.log.gz 2024-09-16T04:14:56.034 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.38488.log: /var/log/ceph/ceph-client.admin.44716.log: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.38488.log.gz 2024-09-16T04:14:56.034 INFO:teuthology.orchestra.run.smithi042.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.44716.log.gz 2024-09-16T04:14:56.034 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43180.log 2024-09-16T04:14:56.035 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.38566.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39831.log 2024-09-16T04:14:56.035 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.43180.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36542.log 2024-09-16T04:14:56.036 INFO:teuthology.orchestra.run.smithi042.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.38566.log.gz 2024-09-16T04:14:56.036 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.39831.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39883.log 2024-09-16T04:14:56.036 INFO:teuthology.orchestra.run.smithi042.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.43180.log.gz 2024-09-16T04:14:56.036 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.36542.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.39831.log.gz 2024-09-16T04:14:56.036 INFO:teuthology.orchestra.run.smithi042.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.36542.log.gz 2024-09-16T04:14:56.037 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35209.log 2024-09-16T04:14:56.037 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.39883.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32350.log 2024-09-16T04:14:56.037 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.35209.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44022.log 2024-09-16T04:14:56.039 INFO:teuthology.orchestra.run.smithi042.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.39883.log.gz 2024-09-16T04:14:56.039 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.32350.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.35209.log.gz 2024-09-16T04:14:56.039 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33377.log 2024-09-16T04:14:56.039 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.44022.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38670.log 2024-09-16T04:14:56.039 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.33377.log: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.32350.log.gz 2024-09-16T04:14:56.041 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36137.log 2024-09-16T04:14:56.042 INFO:teuthology.orchestra.run.smithi042.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.44022.log.gz 2024-09-16T04:14:56.042 INFO:teuthology.orchestra.run.smithi042.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.33377.log.gz 2024-09-16T04:14:56.042 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.38670.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35447.log 2024-09-16T04:14:56.042 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.36137.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36876.log 2024-09-16T04:14:56.042 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.35447.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.38670.log.gz 2024-09-16T04:14:56.042 INFO:teuthology.orchestra.run.smithi042.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.36137.log.gzgzip 2024-09-16T04:14:56.042 INFO:teuthology.orchestra.run.smithi042.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.31680.log 2024-09-16T04:14:56.042 INFO:teuthology.orchestra.run.smithi042.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.35447.log.gz 2024-09-16T04:14:56.042 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42374.log 2024-09-16T04:14:56.042 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.36876.log: /var/log/ceph/ceph-client.admin.31680.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34678.log 2024-09-16T04:14:56.044 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.42374.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.36876.log.gz 2024-09-16T04:14:56.044 INFO:teuthology.orchestra.run.smithi042.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.31680.log.gz 2024-09-16T04:14:56.045 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose 89.3% -- -- replaced with /var/log/ceph/ceph-client.admin.42374.log.gz /var/log/ceph/ceph-client.admin.43800.log 2024-09-16T04:14:56.045 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:14:56.045 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.34678.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27885.log 2024-09-16T04:14:56.045 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.43800.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41706.log 2024-09-16T04:14:56.045 INFO:teuthology.orchestra.run.smithi042.stderr: 89.4%/var/log/ceph/ceph-client.admin.27885.log: -- replaced with /var/log/ceph/ceph-client.admin.34678.log.gz 2024-09-16T04:14:56.045 INFO:teuthology.orchestra.run.smithi042.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.43800.log.gz 2024-09-16T04:14:56.045 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30442.log 2024-09-16T04:14:56.045 INFO:teuthology.orchestra.run.smithi042.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.27885.log.gz 2024-09-16T04:14:56.047 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.41706.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38768.log 2024-09-16T04:14:56.047 INFO:teuthology.orchestra.run.smithi042.stderr: 89.1%/var/log/ceph/ceph-client.admin.30442.log: -- replaced with /var/log/ceph/ceph-client.admin.41706.log.gz 2024-09-16T04:14:56.047 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44924.log 2024-09-16T04:14:56.047 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29632.log 2024-09-16T04:14:56.047 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.38768.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.30442.log.gz 2024-09-16T04:14:56.047 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.44924.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.38768.log.gz 2024-09-16T04:14:56.047 INFO:teuthology.orchestra.run.smithi042.stderr: 87.8%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31102.log 2024-09-16T04:14:56.047 INFO:teuthology.orchestra.run.smithi042.stderr: -- replaced with /var/log/ceph/ceph-client.admin.44924.log.gz 2024-09-16T04:14:56.048 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42738.log 2024-09-16T04:14:56.048 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.29632.log: /var/log/ceph/ceph-client.admin.31102.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34548.log 2024-09-16T04:14:56.049 INFO:teuthology.orchestra.run.smithi042.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.29632.log.gz 2024-09-16T04:14:56.049 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.42738.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.31102.log.gz 2024-09-16T04:14:56.049 INFO:teuthology.orchestra.run.smithi042.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.42738.log.gz 2024-09-16T04:14:56.050 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40968.log 2024-09-16T04:14:56.050 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.34548.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43866.log 2024-09-16T04:14:56.050 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.40968.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36348.log 2024-09-16T04:14:56.050 INFO:teuthology.orchestra.run.smithi042.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34548.log.gz 2024-09-16T04:14:56.051 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.43866.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.40968.log.gz 2024-09-16T04:14:56.051 INFO:teuthology.orchestra.run.smithi042.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.43866.log.gz 2024-09-16T04:14:56.051 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36646.log 2024-09-16T04:14:56.051 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.36348.log: /var/log/ceph/ceph-client.admin.36646.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.36348.log.gz 2024-09-16T04:14:56.052 INFO:teuthology.orchestra.run.smithi042.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.36646.log.gz 2024-09-16T04:14:56.335 INFO:teuthology.orchestra.run.smithi154.stderr: 92.2% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-09-16T04:14:56.363 INFO:teuthology.orchestra.run.smithi154.stderr: 93.6% -- replaced with /var/log/ceph/ceph-client.admin.28216.log.gz 2024-09-16T04:14:56.925 INFO:teuthology.orchestra.run.smithi154.stderr: 92.2% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-09-16T04:14:56.948 INFO:teuthology.orchestra.run.smithi042.stderr: 92.6% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-09-16T04:14:57.064 INFO:teuthology.orchestra.run.smithi154.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.28187.log.gz 2024-09-16T04:14:58.547 INFO:teuthology.orchestra.run.smithi042.stderr: 93.1% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-09-16T04:15:03.003 INFO:teuthology.orchestra.run.smithi154.stderr: 92.5% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-09-16T04:15:03.253 INFO:teuthology.orchestra.run.smithi042.stderr: 92.6% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-09-16T04:15:03.309 INFO:teuthology.orchestra.run.smithi154.stderr: 92.6% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-09-16T04:15:03.509 INFO:teuthology.orchestra.run.smithi042.stderr: 92.5% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-09-16T04:15:03.723 INFO:teuthology.orchestra.run.smithi154.stderr: 92.7% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-09-16T04:15:03.724 INFO:teuthology.orchestra.run.smithi154.stderr: 2024-09-16T04:15:03.724 INFO:teuthology.orchestra.run.smithi154.stderr:real 0m8.003s 2024-09-16T04:15:03.724 INFO:teuthology.orchestra.run.smithi154.stderr:user 0m25.825s 2024-09-16T04:15:03.724 INFO:teuthology.orchestra.run.smithi154.stderr:sys 0m0.907s 2024-09-16T04:15:04.911 INFO:teuthology.orchestra.run.smithi042.stderr: 92.6% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-09-16T04:15:04.912 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-09-16T04:15:04.912 INFO:teuthology.orchestra.run.smithi042.stderr:real 0m9.228s 2024-09-16T04:15:04.912 INFO:teuthology.orchestra.run.smithi042.stderr:user 0m28.175s 2024-09-16T04:15:04.912 INFO:teuthology.orchestra.run.smithi042.stderr:sys 0m1.016s 2024-09-16T04:15:04.914 INFO:tasks.ceph:Archiving logs... 2024-09-16T04:15:04.914 DEBUG:teuthology.misc:Transferring archived files from smithi042:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7905897/remote/smithi042/log 2024-09-16T04:15:04.915 DEBUG:teuthology.orchestra.run.smithi042:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-16T04:15:09.012 DEBUG:teuthology.misc:Transferring archived files from smithi154:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7905897/remote/smithi154/log 2024-09-16T04:15:09.013 DEBUG:teuthology.orchestra.run.smithi154:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-16T04:15:12.456 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-16T04:15:12.465 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-16T04:15:12.466 DEBUG:teuthology.orchestra.run.smithi042:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-16T04:15:12.468 DEBUG:teuthology.orchestra.run.smithi154:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-16T04:15:12.551 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-16T04:15:12.559 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-16T04:15:12.559 DEBUG:teuthology.orchestra.run.smithi042:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T04:15:12.562 DEBUG:teuthology.orchestra.run.smithi154:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T04:15:12.573 INFO:teuthology.orchestra.run.smithi042.stdout: remote refid st t when poll reach delay offset jitter 2024-09-16T04:15:12.573 INFO:teuthology.orchestra.run.smithi042.stdout:============================================================================== 2024-09-16T04:15:12.573 INFO:teuthology.orchestra.run.smithi042.stdout:*hv01.front.sepi 67.205.162.81 3 u 18 64 377 0.106 +0.713 0.147 2024-09-16T04:15:12.573 INFO:teuthology.orchestra.run.smithi042.stdout:+hv02.front.sepi 63.231.80.2 3 u 28 64 377 0.096 -0.915 0.073 2024-09-16T04:15:12.574 INFO:teuthology.orchestra.run.smithi042.stdout:+hv03.front.sepi 74.6.168.72 3 u 15 64 377 0.088 +0.928 0.028 2024-09-16T04:15:12.574 INFO:teuthology.orchestra.run.smithi042.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 +0.000 0.000 2024-09-16T04:15:12.576 INFO:teuthology.orchestra.run.smithi154.stdout: remote refid st t when poll reach delay offset jitter 2024-09-16T04:15:12.576 INFO:teuthology.orchestra.run.smithi154.stdout:============================================================================== 2024-09-16T04:15:12.576 INFO:teuthology.orchestra.run.smithi154.stdout:*hv01.front.sepi 67.205.162.81 3 u 32 64 377 0.084 -6.000 1.108 2024-09-16T04:15:12.576 INFO:teuthology.orchestra.run.smithi154.stdout:+hv02.front.sepi 63.231.80.2 3 u 21 64 377 0.109 -7.977 1.449 2024-09-16T04:15:12.576 INFO:teuthology.orchestra.run.smithi154.stdout:+hv03.front.sepi 74.6.168.72 3 u 17 64 377 0.087 -3.984 1.575 2024-09-16T04:15:12.576 INFO:teuthology.orchestra.run.smithi154.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 +0.000 0.000 2024-09-16T04:15:12.577 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-16T04:15:12.586 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-16T04:15:12.587 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-16T04:15:12.620 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-16T04:15:12.655 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-16T04:15:12.705 INFO:teuthology.task.internal:Duration was 2593.157910 seconds 2024-09-16T04:15:12.706 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-16T04:15:12.738 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-16T04:15:12.738 DEBUG:teuthology.orchestra.run.smithi042:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-16T04:15:12.741 DEBUG:teuthology.orchestra.run.smithi154:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-16T04:15:12.778 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-16T04:15:12.778 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi042.front.sepia.ceph.com 2024-09-16T04:15:12.779 DEBUG:teuthology.orchestra.run.smithi042:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-16T04:15:12.830 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi154.front.sepia.ceph.com 2024-09-16T04:15:12.830 DEBUG:teuthology.orchestra.run.smithi154:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-16T04:15:12.840 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-16T04:15:12.840 DEBUG:teuthology.orchestra.run.smithi042:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-16T04:15:12.874 DEBUG:teuthology.orchestra.run.smithi154:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-16T04:15:12.912 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-16T04:15:12.912 DEBUG:teuthology.orchestra.run.smithi042:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-16T04:15:12.946 DEBUG:teuthology.orchestra.run.smithi154:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-16T04:15:13.020 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-16T04:15:13.030 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-16T04:15:13.030 DEBUG:teuthology.orchestra.run.smithi042:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-16T04:15:13.045 DEBUG:teuthology.orchestra.run.smithi154:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-16T04:15:13.057 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-16T04:15:13.080 DEBUG:teuthology.orchestra.run.smithi042:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-16T04:15:13.090 DEBUG:teuthology.orchestra.run.smithi154:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-16T04:15:13.100 INFO:teuthology.orchestra.run.smithi042.stdout:kernel.core_pattern = core 2024-09-16T04:15:13.110 INFO:teuthology.orchestra.run.smithi154.stdout:kernel.core_pattern = core 2024-09-16T04:15:13.127 DEBUG:teuthology.orchestra.run.smithi042:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-16T04:15:13.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T04:15:13.173 DEBUG:teuthology.orchestra.run.smithi154:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-16T04:15:13.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T04:15:13.178 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-16T04:15:13.188 INFO:teuthology.task.internal:Transferring archived files... 2024-09-16T04:15:13.189 DEBUG:teuthology.misc:Transferring archived files from smithi042:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7905897/remote/smithi042 2024-09-16T04:15:13.189 DEBUG:teuthology.orchestra.run.smithi042:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-16T04:15:13.239 DEBUG:teuthology.misc:Transferring archived files from smithi154:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-15_21:00:02-rados-squid-distro-default-smithi/7905897/remote/smithi154 2024-09-16T04:15:13.239 DEBUG:teuthology.orchestra.run.smithi154:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-16T04:15:13.282 INFO:teuthology.task.internal:Removing archive directory... 2024-09-16T04:15:13.282 DEBUG:teuthology.orchestra.run.smithi042:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-16T04:15:13.285 DEBUG:teuthology.orchestra.run.smithi154:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-16T04:15:13.310 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-16T04:15:13.323 INFO:teuthology.task.internal:Not uploading archives. 2024-09-16T04:15:13.324 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-16T04:15:13.356 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-16T04:15:13.356 DEBUG:teuthology.orchestra.run.smithi042:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-16T04:15:13.359 DEBUG:teuthology.orchestra.run.smithi154:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-16T04:15:13.362 INFO:teuthology.orchestra.run.smithi042.stdout: 658642 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 16 04:15 /home/ubuntu/cephtest 2024-09-16T04:15:13.366 INFO:teuthology.orchestra.run.smithi154.stdout: 658615 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 16 04:15 /home/ubuntu/cephtest 2024-09-16T04:15:13.367 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-16T04:15:13.407 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-16T04:15:13.508 INFO:teuthology.run:Summary data: description: rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} thrashers/one workloads/rados_api_tests} duration: 2593.1579101085663 flavor: default owner: scheduled_teuthology@teuthology success: true 2024-09-16T04:15:13.509 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-16T04:15:13.584 INFO:teuthology.run:pass